blh83
18th January 2011 01:24 UTC

Error spawning a command prompt
I am seeing some odd behavior when spawning a command prompt from an NSIS installer. Specifically, I am trying to run the following command:

rundll32.exe setupapi.dll,InstallHInfSection DefaultInstall 132 myInf.inf

This will install a filter driver on Windows. If I run this command from a command prompt without using the installer, everything works just fine (it creates the necessary registry entries from the .inf file and copies my .sys file to \Windows\System32\drivers). However, if I try and run this from the installer using something like

nsExec::Exec '"rundll32.exe" setupapi.dll,InstallHInfSection DefaultInstall 132 myInf.inf'

it kind of works. I say 'kind of' because only the registry entries are created but the .sys file is not being copied (and no error is returned). This seems to indicate an error registering the driver even though no error is returned. I have made sure that I am running this command from the directory which contains 'myInf.inf'.

As an experiment, I ran the following:

ReadEnvStr $R0 COMSPEC
Exec '"$R0" start cmd.exe'

This opens up a command prompt that I can now interact with. If I type the rundll32 command in here directly, I still see the odd behavior that my .sys file is not being copied but no error is being returned. Is there some issue I don't know about when spawning a command prompt from the installer? This is incredibly frustrating and seemingly inexplicable. The fact that my registry entries are being created from the .inf file and no errors are returned, seems to suggest that it should work even though it isn't. Any input is appreciated.

Anders
18th January 2011 01:39 UTC

Try using SetOutPath before the exec call and set it to the directory where your .sys file is.

Also, I don't think this is really the best way to install a driver these days, but without knowing what kind of driver it is I can't really offer a better solution other than trying to read about installing drivers on MSDN and http://www.osronline.com/showlists.cfm?list=ntdev

blh83
18th January 2011 16:24 UTC

Thanks for your suggestions Anders. I had tried setting the out path and it didn't help and also have made sure that I have elevated privileges. I have to install the driver this way because it is a filter driver (not a normal device driver) and I am doing it via command line. It may be that I have to write my own dll to call some DIFx functions but was hoping to avoid it.

I just find it very odd that a command line spawned from the installer using Windows "start" function doesn't spawn a command prompt that seems to have full functionality.