njshaw2
25th May 2010 15:22 UTC

Problem using COM in System::Call DLL with NSIS CreateShortCut
Hi,

In the last week, some update to Microsoft Windows has caused this problem; I'm not sure of any way around it. Prior to last week, my installer worked fine, but now when it runs it fails to create any shortcuts with CreateShortCut.

Here's what I'm doing:
0) Set up installer sections, MUI, etc.
1) Load my own DLL using System::Call to call a function that initialises COM using COINIT_MULTITHREADED.
2) Do stuff with the response to the DLL call.
3) Install my files.
4) Create shortcuts.

If I don't call the DLL at all, the shortcuts get created.

If I initialise COM using COINIT_APARTMENTTHREADED, all the shotcuts also get created fine.

BUT if I initialise COM using COINIT_MULTITHREADED, the shortcuts subsequently fail to be created. The return code from my call to CoInitializeEx() is RPC_E_CHANGED_MODE, meaning that the installer has initialised COM in the COINIT_APARTMENTTHREADED mode, presumably only for the shortcut creation (nothing else seems to be affected in the installer).

Any ideas why this is going on (and why it's only just started happening last week)? And is there any way I can work around it in the installer? I'd prefer not to have to change my entire codebase to use APARTMENTTHREADED if I can avoid it. :)

Thanks,
Nick.

njshaw2
25th May 2010 16:04 UTC

I've solved this myself (sorry for bothering y'all!). To explain why it was doing it:

The installer uses OleInitalize() for its plugins, and OleInitialize() calls CoInitializeEx() using COINIT_APARTMENTTHREADED, so my call to CoInitializeEx() with MULTITHREADED was returning RPC_E_CHANGED_MODE. I was then happily using the existing COM instance from OleInitialize() and incorrectly then closing COM in my DLL with CoUninitialize() which killed off the installer's plugins' OleInitialize().

I've fixed the above now. Again, apologies for the bother! :D

Afrow UK
25th May 2010 17:59 UTC

No it's fine - that's valuable information for others.

Stu