Sam_Engenuity
16th March 2005 14:15 UTC

EXE Encription and DLL security issue
Hey all

This is my first post on this forum, i have 2 issues that have arised while i was creating installers for Engenuity's products.

My first question is, Is the compiled EXE file encripted? is it secure? Is there a way to penetrate in the exe and be able to extract files that you normaly don't want the user to extract?

My second question is, One of my installers has to call a dll function using the system::call however, i dont want the user to, see the dll... or be able to copy it, access it, hack it and change the current dll with the modified one that he created. The dll basically checks to see if the user is licensed to install the product or not. Is there a way to hide the dll, or to encrypt it somehow...? this is actually a big security issue for the company since they are using Installshield and they want to move with NSIS but just because of that, they might not.

Thank you very much for your answers
Samuel Godbout
Engenuity Technologies .inc
http://www.engenuitytech.com/

Afrow UK
16th March 2005 16:09 UTC

I heard that there's software that can decompress files in an NSIS installer, but obviously from the system that has the installer on it (not remotely).

As for using that dll, all I can think of is to extract it, call it, and delete it immediately. I'm not sure how to hide it (other than using file flags)

-Stu

Sam_Engenuity
16th March 2005 18:18 UTC

Thanks for your reply Afrow UK

So you are saying that yes it is possible to decompress an NSIS Installer and be able to hack into it?
In a way the EXE is not encripted at all....

And for the DLL i tried doing what you said... I extracted the DLL in the $PLUGINDIR called it using system::call however when i try to delete it after, it doesnt allow me since, it says that it is in use. All of the code is written in Function .oninit

What can i do?
Thx for the help

Pidgeot
16th March 2005 19:36 UTC

While it's impossible to guarantee that no one will ever see your DLL, you can still take various steps to keep as few people as possible from trying to crack your product(s).

Some of the things you can do are:
*Making the DLL file hidden upon extraction
*Extract as late as possible, delete as soon as possible
*Overwrite the file contents repeatedly to prevent undeletion
*Use various checks in the installer to make sure the file has not been tampered with
*Use various checks in the DLL to try to make sure the file has not been tampered with
*Make the DLL code long and complicated by inserting a bunch of useless, but seemingly relevant code right around the verification part, thus making disassembly very tiresome
*Instead of packing the DLL as-is, include an encrypted version and a decrypter in your uninstaller to run just before using the DLL
*Create a hacked version of NSIS that uses a more secure checksum, such as MD5 or SHA1, to prevent tampering
*Download the DLL instead of extracting it, failing the installation if anything goes wrong during the download (would be a great inconvenience if user doesn't have internet access on the target machine)

Of course, not all of these things could be done in .onInit, but the code could be adjusted accordingly.

FWIW, InstallShield wouldn't be all that much better, since you would still have to take similar steps in an attempt to guard your files from tampering. That should not be the argument for not switching, since it holds true for any installer - since the installer can do all of it on its own, a determined person could perform disassembly to tamper with the files, or learn how to extract single files.

Afrow UK
16th March 2005 20:35 UTC

Hmm thinking about it anyway, as long as you re-extract the DLL in .onInit whenever the installer is ran then how can they get a tampered DLL to be used in the installer?

I haven't checked if this works, but you could try using the NSIS MD5 plugin on itself!
Edit: No that won't work because you have to have a copy of the checksum in your installer to begin with lol

-Stu

Comm@nder21
16th March 2005 20:47 UTC

yep, but as long as it's nearly impossible to reverse md5 encryption, this version is safe:
- include the checksum in your installer
- include the md5sum plugin in your installer
- get a string/integer from the user, encode it (with just one call of md5 plugin) and compare it with saved string (just another line: StrCmp instruction) and then deal with the result.

Sam_Engenuity
16th March 2005 20:47 UTC

Thanks for your reply Pidgeot

*Extract as late as possible, delete as soon as possible

I understand.. however i have the same problem as before. I extract the file call it through system::Call and then when i try to delete it, the installer tells me that the file is use. I can't delete it until the installer ends and closes. I can't overwrite the file either since its in "use".

Is there a way to delete it or overwrite it somehow??
Thanks again

kichik
17th March 2005 17:48 UTC

The System plug-in doesn't unload the DLL after it uses it. You can force it to unload it by using ?u. For example:

System::Call "mydll::myfunc() ?u"