bob32
11th February 2005 19:32 UTC

Need help with System:Call
I am having problems when trying to call a DLL with System::Call.
I know for certain that the DLL is being loaded when I attempt to call it (I verified that by using FileMon by SysInternals).
It is the call itself that is failing. I am going crazy trying to figure out what I am doing wrong. The only thing I know for certain is that the DLL *is* in an accessible location and that it is being loaded when I issue the System:Call. Any help would be very appreciated.

Here is the prototype for the DLL Function:
typedef BOOL (__stdcall *TCC_FN)(DWORD*);
BOOL __stdcall MyFunc(DWORD *results);

here is the function I wrote:
;--------------------------------------------------
SetPluginUnload alwaysoff
Function CallMyFunc
SetOutPath $PLUGINSDIR
System::Call "MyDLL::MyFunc(*i .r1)i.r2"
MessageBox MB_OK 'Results: $1'
MessageBox MB_OK 'Return Value: $2'

; last plugin call must not have /NOUNLOAD so NSIS will be able to delete
; the temporary DLL
SetPluginUnload manual
; do nothing (but let the installer unload the System dll)
System::Free 0
FunctionEnd
;--------------------------------------------------
The messageboxes that result read as follows:
Results: 0
Return Value: error

Valid return values for "Results" ($1) are 0, 1, 2, 4, 8, 16, and 32
Valid result values for "Return Value" ($2) are 1 and 0

kichik
11th February 2005 20:07 UTC

If the DLL is surely loaded but the function is not called, it's probably a name mangling problem. Open your DLL with Dependency Walker and make sure the function is really named MyFunc. I guess it's actually named _MyFunc.

bob32
11th February 2005 21:05 UTC

I can't claim to be 100% sure it's loading. I just know that when the call to System::Call is issued, FileMon shows that my setup is doing /something/ with MyDLL.dll. I interepreted that as loading, but I guess it could be /trying/ to load the dll and failing. Is there any way to know if that's the case?

kichik
11th February 2005 21:12 UTC

You can use another SysInternal tool called DebugView. In your plug-in's DllMain use OutputDebugString and use DebugView to see if DllMain is really called.

Have you tried Dependency Walker?

bob32
12th February 2005 01:49 UTC

Okay, I figured it out.
Posting my solution here in case another bone headed novice does the same thing I was doing.
Due to my unfamiliarity with the NSIS installer, I was just piecing code together from the samples without a 100% understanding of what it's actually doing. Anyway, I was doing the following
;---------------------------
...
ReserveFile "MyDLL.DLL"
...
!insertmacro MUI_INSTALLOPTIONS_EXTRACT "MyDLL.DLL"
...
;---------------------------

For some reason, extracting my DLL in that manner was causing my DLL to become corrupted (or, for some reason, simply unusable - I haven't had time to fully investigate). Once I changed the code to be a simple
File MyDLL.DLL
The function worked perfectly.

Thank you for your help, by the way, kichik.

kichik
12th February 2005 10:54 UTC

MUI_INSTALLOPTIONS_EXTRACT treats the extracted file as an INI file and uses WriteINIStr to write something to it. That's why it corrupts DLLs.