How to add to DEP opt-out list programmatically.

touchring
24th March 2009 06:50 UTC

How to add to DEP opt-out list programmatically.
Does anyone knows how to add to DEP exclusion list programmatically other than via the sysdm.cpl or control panel?

Like many, I've been using asprotect and aspack for my program and recently i discovered that windows vista ultimate and windows 2008 server no longer exempt asprotect packed apps from DEP. Asprotect packed apps crash when DEP is activated, which is almost every windows ultimate and windows 2008 server system since all programs have DEP activated by default, unlike XP.

I've done some research and many articles point to adding a registry key like the following but it does not work (in that the application still crashes when run) although you can actually see the entry in the DEP list if you run sysdm.cpl.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers]
"C:\\path\\myapplication.exe"="EnableNXShowUI"

If i add it manually via the sysdm.cpl GUI, my application does not crash any more. I've tried using processexplorer to see where else Windows would write to the registry but no clue. The topics about editing boot.ini is no help either as it will entirely deactivate DEP, which is not what i want to do.

Someone suggested broadcasting WM_SETTINGCHANGE, so i tried the following after adding the EXE path to AppCompatFlags but no luck also.

SendMessage ${HWND_BROADCAST} ${WM_SETTINGCHANGE} 0 "STR:Environment" /TIMEOUT=0

Hope any of you out there with some clues may help. :|

Anders
24th March 2009 11:10 UTC

I don't know if WM_SETTINGSCHANGE can help, but I know that the STR:Environment part is wrong. You could try AppCompatFlags, but for most stuff like that, you should use the full registry "path".

But, I'm hoping that this is by design and that only the user can add items to the DEP list. You should contact the Asprotect people and get them to give you a version that works with DEP on (Remember, DEP has a mode where it does not use that list, but forces DEP on everything)

Or maybe you could switch to UPX

edit:
try setting __COMPAT_LAYER=EnableNXShowUI as a environment variable in your installer before starting your program (not system wide, just for your installer, that should allow you to start it from the installer) See http://www.microsoft.com/windowsxp/u.../layertip.mspx

touchring
24th March 2009 14:39 UTC

You should contact the Asprotect people and get them to give you a version that works with DEP on (Remember, DEP has a mode where it does not use that list, but forces DEP on everything)

ASProtect support has practically stopped ever since starforce bought over the company. Their latest version demo would not even work on my XP.

edit:
try setting __COMPAT_LAYER=EnableNXShowUI as a environment variable in your installer before starting your program (not system wide, just for your installer, that should allow you to start it from the installer) See http://www.microsoft.com/windowsxp/u.../layertip.mspx [/B]

Thanks for mentioning this option, but that won't work as the application is actually the application that NSIS installer is suppose to install. :eek:

Anders
24th March 2009 15:14 UTC

I don't know why it does not work for you...it works fine for me when testing the XP stuff


CopyFiles "$sysdir\calc.exe" $temp
System::Call 'kernel32::GetLongPathName(t "$temp\calc.exe", t .r1, i ${NSIS_MAX_STRLEN})i'
WriteRegStr HKLM "Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" $1 "WIN95 DISABLETHEMES"
Exec '"$temp\calc.exe"'

Note that I had to call GetLongPathName to get the correct path. You should make sure you are using the correct full path

edit:
and just doing


CopyFiles "$sysdir\calc.exe" $temp
System::Call 'Kernel32::SetEnvironmentVariableA(t, t) i("__COMPAT_LAYER", "DISABLETHEMES").r0'
Exec '"$temp\calc.exe"'

also works, but thats a one time thing only when started from your installer

DrO
25th March 2009 00:24 UTC

not knowing further details of what's going on, etc but it's possible to disable a process from DEP as shown here though that is mainly aimed at coders but it may be off some help (or more likely not but mentioning it whilst i remember).

-daz

Anders
25th March 2009 13:50 UTC

ah, yes, I had forgotten about those API's (it also exists on XP SP2, but with a different name, and it's undocumented, IIRC google chrome uses it, check its source)

But these API's have the same problem as setting the registry option, if DEP is always on, they have no effect.

So why can you not switch to a different packer? UPX?

DrO
25th March 2009 14:18 UTC

wasn't sure if it'd work if it's been forced on. but yeah would make more sense to switch to a different packer (if that really is something that is needed to be done).

-daz

touchring
25th March 2009 20:53 UTC

Thanks for your suggestions, the registry method doesn't work, even when using full path. There must be an additional step.

Switching to a different packer will be the last resort as my whole registration system, which includes an online component, is built on asprotect.

Someone gave me a link - blogs.msdn.com/gauravb/archive/2008/09/23/disable-dep-on-applications.aspx

But i'm not sure how i can use it since asprotect runs before my application does.