RegDLL crashes installer

I have an installer that needs to register several existing dlls. I have tested it on different versions of Windows and it always crashes on RegDLL. I have tried registering one of the four dlls, and it doesn't matter which one I pick it still crashes. If I use ExecWait Regsvr32 it works fine.

!include "MUI.nsh"



Name "DLL Registration"

OutFile "DLL_setup.exe"

InstallDir "C:\MYAPP"



!define MUI_ABORTWARNING

!insertmacro MUI_PAGE_WELCOME

!insertmacro MUI_PAGE_DIRECTORY

!insertmacro MUI_PAGE_INSTFILES

!insertmacro MUI_PAGE_FINISH

!insertmacro MUI_LANGUAGE "English"



Section "Register DLLs" SecRegisterDLL



  SetOutPath $INSTDIR



;  ExecWait "regsvr32 /s $INSTDIR\MYAPP-2.dll "





  RegDLL $INSTDIR\MYAPP-1.dll

;  RegDLL $INSTDIR\MYAPP-2.dll

;  RegDLL $INSTDIR\MYAPP-3.dll

;  RegDLL $INSTDIR\MYAPP-4.dll

 

SectionEnd

Are you sure regsvr32 really succeeds? Try without the /S flag. Also try registering the same DLL with regsvr32 and RegDLL.

I once had an installer crash when registering a DLL. It was happening because the DLL had a dependency that was missing. You can run http://www.dependencywalker.com/ to see what dependencies the DLL has, and then have your installer install those before registering your DLL. Hopefully, that's what's causing it.

Sorry for the delay in replying.

The DLLs do indeed register fine when using regsvr32 with or without the "/s" parameter. I tested it manually from a command prompt and also using ExecWait from inside an installer and in both cases the registration succeeds.

If I switch my installer over to using RegDLL, it still consistently crashes the installer with "Not Responding" in Task Manager.

Dependency Walker indicates "Error opening file" MSJAVA.DLL. This dll is probably related in some way to the COBOL run-time that my DLLs are dependent upon. I was unaware of this particular dependency and it has not created any problem in the past. The application works fine when I register my DLLs manually or in a batch file.

At this point I am planning on using RegSvr32, but I am curious why my installer crashes as opposed to just returning an error code.

Does it actually crash or just hangs? In any case, I'll need the DLL files to figure out why they do that, unless you can provide debugging information like a crash dump or at least a stack trace.

Crashing is probably not the correct term. It hangs or gets locked up. The only way to recover is to End Task in Windows Task Manager.

I could send you the dlls, but they are completely dependent on the COBOL runtime, so you would have to have the installed first.

How do I send you a crash dump or stack trace?

The DLL files themselves would be better than a crash dump or a stack trace. So if you can send those, it'd be best. I'll handle the COBOL runtime.

Thanks kichik. Attached is one of the DLLs that is causing the problem.

I got the runtime DLL files from Futijsu, but all I could get them to do is kill every process they were loaded from with a message saying I didn't set them up correctly. Do you know where I can get a non-MSM installer that's not Futisju? I get around 1kb/s from that website...

Alternatively, that crash dump seems more and more compelling... To generate it, run windbg, attach it to the frozen installer and write .dump /ma C:\crash.dump in its command line.

I'm pretty sure you have to use Fujitsu's proprietary COBOL runtime to make it work.

I ran windbg and created the crash dump as you recommended. However, the file is very large 30mb+ unzipped, about 11mb zipped. Couldn't attach it to this forum message. You can download it here.

The COBOL runtimes don't seem to like the GUI. When registered, it tries to activate the runtime COM object using CoGetClassObject but not before it locks a critical section. COM tries to load the requested class in the GUI thread which in turn tries to lock the same critical section and freezes. The COM objects shouldn't lock critical sections this way in single-threaded apartments as COM should handle that internally.

I've created a little program that should be able to reproduce it. Put lucis-2.dll next to it and run it. It should also freeze in the same manner. If it doesn't, let me know. If it does, you should post that as part of a support request to Fujitsu, or some COBOL forum.

Thanks for all your help kickik. I did test using your cobol.exe and it did indeed lockup. I have a limited understanding of COM so I don't understand why the locking is causing a problem, but will pursue it with Fujitsu.

I am curious why the problem does not manifest itself when using regsvr32 which is what I am using to work around the issue.

regsvr32 doesn't have two threads. NSIS creates one thread for the GUI and another thread for executing the sections. The second thread calls RegDLL, loads the DLL and calls DllRegisterServer. The DLL, in turn, enters the critical section and tries to get an interface to the runtime using CoGetClassObject. COM does all the magic of loading the other DLL and getting the interface behind the scenes. Part of that magic is loading the DLL in the GUI thread using some windows messages. When the DLL is loaded in the GUI thread, it tries to enter the same critical sections the other thread already has locked. That's when the deadlock appears.

As COM handles all the synchronization for single-threaded apartment, the critical section usage only gets in the way and results in this deadlock.