Necessary conditions include:
- custom page
- ExecToStack is executed in the custom page show function
- the executed program broadcasts a windows SetNonClientMetrics message.
Below you can find two scripts that reproduce this problem.
The second script is written in AutoIt3, it just broadcasts the message.
The first script is NSIS, it execs once in .onInit, which demonstrates that there is no failure (because it's outside the custom page). Then it execs in the custom page show function where it hangs.
You should be seeing exactly four message boxes, but you will actually see only three, one before and one after exec in .onInit, and one before exec in the show function. Then you need to kill _prog.exe which hangs in the background.
_bug.nsi
!include "MUI2.nsh"
AutoCloseWindow false
WindowIcon on
XPSTYLE on
Name "_bug"
OutFile "_bug.exe"
Caption "_bug"
VIProductVersion "1.0"
Var program
Page custom Show Leave ""
!insertmacro MUI_LANGUAGE "English"
!macro _RunProgram
MessageBox mb_ok "in function ${__FUNCTION__} line ${__LINE__}..."
nsExec::ExecToStack $program
Pop $0
MessageBox mb_ok "in function ${__FUNCTION__} line ${__LINE__} exec status was $0"
!macroend
!define RunProgram `!insertmacro _RunProgram`
Function Show
${RunProgram}
FunctionEnd
Function Leave
FunctionEnd
Function .onInit
InitPluginsDir
strcpy $program "$EXEDIR\\_prog.exe"
${RunProgram}
FunctionEnd
Section
SectionEnd _prog.au3Global Const $WM_SETTINGCHANGE = 0x1A
Global Const $HWND_BROADCAST = 0xFFFF
Global Const $SPI_SETNONCLIENTMETRICS = 42
DllCall("user32.dll", "int", "SendMessage", _
"hwnd", $HWND_BROADCAST, _
"uint", $WM_SETTINGCHANGE, _
"wparam", $SPI_SETNONCLIENTMETRICS, _
"str", "WindowMetrics" _
)