demiller9
2nd November 2011 15:58 UTC

Setup crashes - Stack Overflow
When I create a custom page and then decide not to display it I called Abort. The following page (Finish) crashes. The crash is a Stack Overflow, and the stack is full of 4 function calls: nsDialogs.dll!10001441() / USER32.dll!_UserCallWinProc@20() / USER32.dll!_CallWindowProcAorW@24() / USER32.dll!_CallWindowProcA@20().

Standard NSIS 2.46 using MUI2, WinXP SP3 (and Win2000).

I rewrote my installer to decide if the page will be shown before calling Create, so I don't have the problem anymore, but this smells like a bug.

Don

OutFile Test.exe
!include "MUI2.nsh"
!insertmacro MUI_PAGE_WELCOME
Page custom MyPage
!define MUI_PAGE_CUSTOMFUNCTION_PRE Finish.Pre
!define MUI_PAGE_CUSTOMFUNCTION_SHOW Finish.Show
!insertmacro MUI_PAGE_FINISH
!insertmacro MUI_LANGUAGE "English"
Section
SectionEnd
Function MyPage
${IfCmd} MessageBox MB_OKCANCEL "OK = Call nsDialogs::Create$\nCancel = Abort the custom page" IDCANCEL ${||} Abort ${|}
nsDialogs::Create 1018
Pop $0
${IfCmd} MessageBox MB_OKCANCEL "OK = Show the custom page$\nCancel = Abort the custom page / Crash" IDCANCEL ${||} Abort ${|}
!insertmacro MUI_HEADER_TEXT "Custom page" "Aborting a custom page after the inner dialog is created crashes the installer."
nsDialogs::Show
FunctionEnd
Function Finish.Pre
MessageBox MB_OK "This is the Pre callback"
FunctionEnd
Function Finish.Show
MessageBox MB_OK "This is the Show callback"
FunctionEnd

Afrow UK
2nd November 2011 16:03 UTC

Calling Abort after creating the custom page will no doubt cause unpredictable behaviour as it is not something you should be doing.

Stu

demiller9
2nd November 2011 16:44 UTC

Stu
I agree that Abort is something to avoid after calling Create, but the 'bug' still happens - without calling Abort - if I simply skip the nsDialogs::Show function.

This version doesn't call Abort after calling Create. Crashes when asked to not show the custom page.
Don

OutFile Test_1.exe
!include "MUI2.nsh"
!insertmacro MUI_PAGE_WELCOME
Page custom MyPage
!define MUI_PAGE_CUSTOMFUNCTION_PRE Finish.Pre
!define MUI_PAGE_CUSTOMFUNCTION_SHOW Finish.Show
!insertmacro MUI_PAGE_FINISH
!insertmacro MUI_LANGUAGE "English"
Section
SectionEnd
Function MyPage
${IfCmd} MessageBox MB_OKCANCEL "OK = Call nsDialogs::Create$\nCancel = Abort the custom page" IDCANCEL ${||} Abort ${|}
nsDialogs::Create 1018
Pop $0
!insertmacro MUI_HEADER_TEXT "Custom page" "Aborting a custom page after the inner dialog is created crashes the installer."
${IfCmd} MessageBox MB_OKCANCEL "OK = Show the custom page$\nCancel = Don$\'t show the custom page / Crash" IDOK ${||} nsDialogs::Show ${|}
FunctionEnd
Function Finish.Pre
MessageBox MB_OK "This is the Pre callback"
FunctionEnd
Function Finish.Show
MessageBox MB_OK "This is the Show callback"
FunctionEnd

Afrow UK
2nd November 2011 18:05 UTC

The fact that you are skipping Show is what is causing the crash. It has nothing to do with calling Abort as such. It is because you are creating the dialog but never showing it thus allowing the installer to load the next page without allowing the plug-in to cleanup and de-subclass any window procedures. The plug-in could be fixed to not crash, but why add the extra C code when the scripter shouldn't be doing this in the first place.

Edit: Basically the 'bug' is with your code, not the plug-ins'.

Stu

T.Slappy
3rd November 2011 07:30 UTC

Simply to say: you use nsDialogs::Create without nsDialogs::Show, that is causing all issues.

demiller9
3rd November 2011 14:37 UTC

I know that's what causes it; I think that it shouldn't cause a stack overflow. Who knows - maybe a one line code change could avoid the crash.

My installer no longer does anything like this, so I won't see it crash. I started the thread to see if anyone was interested in fixing the root cause of the crash.

Anders
8th November 2011 01:10 UTC

With the custom page plugins; once you say A you have to say B :) (There is probably some subclassing going on to catch the page change messages)

anandsunku
16th November 2011 12:23 UTC

crash dump analysis
if the installer crashes after installation is complete due to some reason, how can we do the crash dump analysis. like we do for the c++ code with windbg. like collecting symbols, loading symbols , etc.,