Setup crashes - Stack Overflow

When I create a custom page and then decide not to display it I called Abort. The following page (Finish) crashes. The crash is a Stack Overflow, and the stack is full of 4 function calls: nsDialogs.dll!10001441() / USER32.dll!_UserCallWinProc@20() / USER32.dll!_CallWindowProcAorW@24() / USER32.dll!_CallWindowProcA@20().

Standard NSIS 2.46 using MUI2, WinXP SP3 (and Win2000).

I rewrote my installer to decide if the page will be shown before calling Create, so I don't have the problem anymore, but this smells like a bug.

Don

OutFile Test.exe
!include "MUI2.nsh"
!insertmacro MUI_PAGE_WELCOME
Page custom MyPage
!define MUI_PAGE_CUSTOMFUNCTION_PRE Finish.Pre
!define MUI_PAGE_CUSTOMFUNCTION_SHOW Finish.Show
!insertmacro MUI_PAGE_FINISH
!insertmacro MUI_LANGUAGE "English"
Section
SectionEnd
Function MyPage
${IfCmd} MessageBox MB_OKCANCEL "OK = Call nsDialogs::Create$\nCancel = Abort the custom page" IDCANCEL ${||} Abort ${|}
nsDialogs::Create 1018
Pop $0
${IfCmd} MessageBox MB_OKCANCEL "OK = Show the custom page$\nCancel = Abort the custom page / Crash" IDCANCEL ${||} Abort ${|}
!insertmacro MUI_HEADER_TEXT "Custom page" "Aborting a custom page after the inner dialog is created crashes the installer."
nsDialogs::Show
FunctionEnd
Function Finish.Pre
MessageBox MB_OK "This is the Pre callback"
FunctionEnd
Function Finish.Show
MessageBox MB_OK "This is the Show callback"
FunctionEnd

Calling Abort after creating the custom page will no doubt cause unpredictable behaviour as it is not something you should be doing.

Stu

Stu
I agree that Abort is something to avoid after calling Create, but the 'bug' still happens - without calling Abort - if I simply skip the nsDialogs::Show function.

This version doesn't call Abort after calling Create. Crashes when asked to not show the custom page.
Don

OutFile Test_1.exe
!include "MUI2.nsh"
!insertmacro MUI_PAGE_WELCOME
Page custom MyPage
!define MUI_PAGE_CUSTOMFUNCTION_PRE Finish.Pre
!define MUI_PAGE_CUSTOMFUNCTION_SHOW Finish.Show
!insertmacro MUI_PAGE_FINISH
!insertmacro MUI_LANGUAGE "English"
Section
SectionEnd
Function MyPage
${IfCmd} MessageBox MB_OKCANCEL "OK = Call nsDialogs::Create$\nCancel = Abort the custom page" IDCANCEL ${||} Abort ${|}
nsDialogs::Create 1018
Pop $0
!insertmacro MUI_HEADER_TEXT "Custom page" "Aborting a custom page after the inner dialog is created crashes the installer."
${IfCmd} MessageBox MB_OKCANCEL "OK = Show the custom page$\nCancel = Don$\'t show the custom page / Crash" IDOK ${||} nsDialogs::Show ${|}
FunctionEnd
Function Finish.Pre
MessageBox MB_OK "This is the Pre callback"
FunctionEnd
Function Finish.Show
MessageBox MB_OK "This is the Show callback"
FunctionEnd

The fact that you are skipping Show is what is causing the crash. It has nothing to do with calling Abort as such. It is because you are creating the dialog but never showing it thus allowing the installer to load the next page without allowing the plug-in to cleanup and de-subclass any window procedures. The plug-in could be fixed to not crash, but why add the extra C code when the scripter shouldn't be doing this in the first place.

Edit: Basically the 'bug' is with your code, not the plug-ins'.

Stu

Simply to say: you use nsDialogs::Create without nsDialogs::Show, that is causing all issues.

I know that's what causes it; I think that it shouldn't cause a stack overflow. Who knows - maybe a one line code change could avoid the crash.

My installer no longer does anything like this, so I won't see it crash. I started the thread to see if anyone was interested in fixing the root cause of the crash.

With the custom page plugins; once you say A you have to say B 🙂 (There is probably some subclassing going on to catch the page change messages)

crash dump analysis

if the installer crashes after installation is complete due to some reason, how can we do the crash dump analysis. like we do for the c++ code with windbg. like collecting symbols, loading symbols , etc.,