njshaw2
2nd March 2009 17:11 UTC

System::Call not working with custom DLL
Hi,

I've written a function in my own DLL to find out if the user has permission to install the software version (based on a hardware dongle setting, but that detail is irrelevant). My function, in C, is something like this:

int WINAPI CanUserInstall(void)
{
int Ret = FALSE;

// do various checks
// if user is allowed to install, set Ret to TRUE;

return Ret;
}

This function is being exported by the DLL correctly (according to Microsoft's Dependency Walker app).

In my NSIS script, I have the following code:

SetOutPath "$TEMP"
File "installer.dll"
System::Call 'installer.dll::CanUserInstall(v) i.R0 ?e'
Pop $0
MessageBox MB_OK "Return value = $R0, lasterr = $0"
IntCmp $R0 1 OkToInstall CancelInstall

CancelInstall:
Abort "Not allowed to install"

OkToInstall:
; ... do the install

Firstly, the messagebox returns this:
"Return value = error, lasterr = 0"
Then the IntCmp always goes to the CancelInstall as $R0 always equals 0.

Any ideas where I'm going wrong? The error return implies it either can't load the DLL, or can't find the function in it, but I can't see where I've gone wrong.

Thanks for any help.
Nick.

Anders
2nd March 2009 17:27 UTC

you don't need ".dll". you don't need v, just () is fine, other than that it looks ok to me. And I'm sure you know, in this example, ?e is pointless since you don't call SetLastError or other windows api's

AxelMock
2nd March 2009 17:34 UTC

Re: System::Call not working with custom DLL

Originally posted by njshaw2

[B]Hi,

I've written a function in my own DLL to find out if the user has permission to install the software version (based

Anders
2nd March 2009 17:51 UTC

AxelMock, dude, you know nsis already has .ini support?

and a define will not help at all, thats all precompiler magic and does not change the end result

njshaw2
2nd March 2009 18:49 UTC

Thanks for the suggestions. I've already tried without the ".dll", without the (v) and without the ?e - all give the same result.

I'm at the stage where I can't see the DLL is actually loaded - I've put OutputDebugStrings in the DLL's DllMain() which are never being called, so either NSIS is failing to load the DLL, or the DLL is not loading properly. Does the DLL need any specific calling convention or build property? I'm using WINAPI for the functions, and the project is built with Multi-threaded DLL support. It's also being built in VS2008, can't see why that could be an issue, but just in case.

It looks, from the replies, like I'm doing all the right stuff in NSIS (I can do calls to Win32 APIs fine, so I know the system.dll is working and my syntax is pretty much ok) - back to the old VS2008 debugging, it looks like.

Anders
2nd March 2009 20:56 UTC

Process Monitor from sysinternals should be able to help out. You can see both the filesystem operations to make sure it finds the file, and possible errors when the loader tries to map the dll

AxelMock
3rd March 2009 07:36 UTC

Originally posted by Anders
AxelMock, dude, you know nsis already has .ini support?

njshaw2
16th March 2009 14:48 UTC

I've found the problem I was having. Subtle. You need to include the path of your dll in the System::Call() information if your dll is not in the system path. The documentation doesn't specify this; it implies it will load it from the last configured "SetOutPath". Obviously not.

So I used:

SetOutPath "$TEMP"
File "Data\MyFile.dll"
System:Call "$TEMP\MyFile.dll:MyFunc() i.r0"

and from then on it was happy. :)

Afrow UK
16th March 2009 15:52 UTC

I think if you extract your dll to the same folder as the System.dll then it will work (i.e. $PLUGINSDIR).

Stu