pabs
31st July 2005 14:38 UTC

debugging InstallOptions problem [linux]
Hi all,

When I compile the 2.08 InstallOptions plugin on debian, and use it in the WordFunc.nsi example, it fails to show the dialog due to the call to CreateDialog returning NULL. The error returned from GetLastError is 0 "the operation completed successfully" :/

The weird thing is that the makefile system I hacked up for 2.06 works, while the Scons build stuff in 2.08 does not. The resulting installer segfaults, but the same happens with installers that use the windows-compiled plugin, so thats a separate issue.

Anyone have any idea what might be wrong with the Scons stuff? I tried adding -mwindows and a couple of other things, no change though.

kichik
2nd August 2005 20:41 UTC

Something very similar happened here. The generated installer failed on CreateDialog in exactly the same way. I think it might be a problem with the dialog resource, but I've never really researched it deeply enough.

pabs
3rd August 2005 07:13 UTC

Hmm, thing is that the 2.06 compile works, but 2.08 does not.

Also, when I dump the resources from each dll using ResHacker, the text version is exactly the same.

Hmm, perhaps the mingw-exehead incorrectly decompresses the plugin file?

pabs
4th August 2005 13:13 UTC

I noticed that, of the example nsi files, only a couple actually work with the scons built 2.08 debian package.

I tested them all under cygwin gdb on winxp and got the following backtraces (many were similar/the same):

#0  0x77f5ac87 in ntdll!RtlUnicodeToMultiByteSize () from ntdll.dll
#1  0x77f5abb5 in ntdll!RtlUnicodeStringToCountedOemString () from ntdll.dll
#2  0x77e712d2 in KERNEL32!GetTempFileNameA () from /c/WINDOWS/system32/kernel32.dll
#3  0x00405e1b in ?? ()
-------------------------------
#0  0x77e727aa in lstrcpynA () from /c/WINDOWS/system32/kernel32.dll
#1  0x0025ad49 in ?? ()
-------------------------------
#0  0x77d4fe10 in wsprintfA () from /c/WINDOWS/system32/user32.dll
#1  0x00465000 in ?? ()
-------------------------------
#0  0x00401e1f in ?? ()
-------------------------------
#0  0x77f75a59 in ntdll!DbgUiConnectToDbg () from ntdll.dll
#1  0x77f9cb5e in ntdll!RtlpNtEnumerateSubKey () from ntdll.dll
Previous frame inner to this frame (corrupt stack?)

kichik
4th August 2005 22:32 UTC

Can you attach an example installer? If it's too big, use stashbox.

pabs
5th August 2005 04:56 UTC

I've uploaded all the examples built with the debian nsis 2.08 package to here:

http://bonedaddy.net/pabs3/files/tmp/examples.zip (1.4M)

Where the example uses System.dll, I used the one from the windows package of nsis 2.08.

I've also uploaded the stubs and plugins to here:

http://bonedaddy.net/pabs3/files/tmp/stubs-plugins.zip (222K)

kichik
5th August 2005 12:37 UTC

I've managed to get the installer not to segfault (attached). It has the exact same problem as in the other thread. I've checked out 2.06's deb, and it also contains bad section order and was probably working just by chance. It seems like the linker script never really worked.

onad
5th August 2005 13:11 UTC

Good to know I'm not stupid anyhow :-) , building on FreeBSD is difficult and challenging to say the least. Did not have much time last weeks but as soon as possible I'll bug the forum again with problem when buiding on other OS's. I also will be procative as soon as I have ONCE a succesful build on FreeBSD via SCons. A Wiki page will be extended explaining what to do to create a successful build on Linux, FreeBSD, and if possible a MAC Intel.

kichik
5th August 2005 14:18 UTC

InstallOptions fails because MinGW decided DllMain isn't the main function. It's set its own entry point and ignored DllMain. Because of this, m_hInstance is not defined properly and CreateDialog can't find the appropriate resource. The problem is the entry flag for ld which points to _DllMain@16 instead of _DllMain@12. ld doesn't complain and so the wrong entry point is used. I've fixed this in the latest CVS version. I'll upload a patch once I find a fix for the first problem too.

kichik
5th August 2005 16:43 UTC

I've ended up using ld's default linker script. It's probably version specific and might not work that well with other versions, but it's better than the last one. I've also fixed its usage. Previously, it was passed to ld using -Wl and not -T as it should be.

Attached is a patch genereated with:

cvs -q patch -R -u -D yesterday -D now . > deb.patch.txt

pabs
6th August 2005 11:22 UTC

The patch worked perfectly, all of the installers worked fine and didn't show any strange problems. I've applied this patch to debian nsis 2.08-3 and will upload it to debian eperimental soon.

One exception was BgImage Test.exe, which segfaulted. The backtrace was something like this:

ntdll
ntdll
SetBg
SetBg
?
?
?
...

pabs
7th August 2005 10:19 UTC

Ok, the problem with BgImage.dll is that the critical section object is not initialised, because DllMain is not entered. According to objectdump -x and the MSVC dependency walker, BgImage.dll doesn't even export a DllMain. The windows nsis 2.08 Plugins don't seem to export the DllMain either.

This is with the patch. I've attached the BgImage plugin (with debugging messageboxes).

pabs
7th August 2005 11:01 UTC

OK, I tested the InstallOptions plugin, and the DllMain from it also does not get called. However, the Banner plugin does get it's DllMain call. After testing some more of the plugins, it seems like the C++ plugins are the ones that are broken, the C ones are fine.

kichik
9th August 2005 17:06 UTC

Uff, MinGW is so annoying. Try adding extern "C" to DllMain.

pabs
10th August 2005 11:31 UTC

Thanks, that fixes it. Patch uploaded to sourceforge. New package soon to be in debian.