Gato Muerto
19th June 2007 02:51 UTC

Problem with WriteRegStr, HKLM, and AppCompatFlags
Program requires compatibility mode (Win98/ME) to run properly in Vista. To streamline this, I have added some generic code to a series of installer scripts in order to "Plug and Run". Unfortunately, the script is not writing the required registry entry. I am likely missing something quite obvious, but was wondering if anyone else has an idea. Here is the important code:

!define APPCOMPATMODE "WIN98"
  # Registry entry in order to enable compatibility mode settings for the application.
  # Put a space between each mode selected.
  # The following modes are available:
  #    {WIN95 WIN98 NT4SP5 WIN2000 WINXPSP2}
  #    256COLOR
  #    640X480
  #    DISABLETHEMES
  #    DISABLEDWM
  #    HIGHDPIAWARE
  #    RUNASADMIN
#snip loads of code...

Section "Base Components" BASECOMPONENTS
#snip loads of code...

  !ifdef APPCOMPATMODE
    WriteRegStr HKLM "Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" "$INSTDIR\${PROGEXE}" "${APPCOMPATMODE}"
  !endif
#snip loads of code...

SectionEnd

The odd thing is that if I change HKLM to HKCU, the code works perfectly, the registry entry appears correctly (in CU), and the program works (for that one user). Perhaps I am missing something obvious, but what is wrong here?

Anders
19th June 2007 04:49 UTC

your installer needs admin rights to write to HKLM, use "RequestExecutionLevel admin" in your script

Gato Muerto
19th June 2007 17:48 UTC

Thanks. Like I said, was missing something rather obvious. :P
Actually thought that since Admin rights are needed to execute anyways, the installer was already running with Admin rights.

Gato Muerto
11th July 2007 02:29 UTC

Thanks for the help. Have a very similar issue, and believe reposting in the same thread would be better all around.

I am reusing the same code for a new installer and running into another almost identical problem. Run the code on Vista, everything is great, works fine. Move to a machine running the 64 bit version of Vista, and no joy. Again, HKLM fails, HLCU works, which leads to believe that the ReqExeLev isn't working for some reason. Same CDROM on both the 32bit and 64bit OS, so rather confused as to the cause here. Is there a difference for how the x64 version of Vista handles "RequestExecutionLevel"?

Anders
11th July 2007 12:57 UTC

Probably registry redirection. Try using SetRegView 64

Gato Muerto
11th July 2007 18:50 UTC

Thanks, that did it.
Does this setting do anything if the OS is 32-bit instead of 64-bit?

Anders
11th July 2007 20:44 UTC

Well, the extra flag for RegOpen/CreateKey is still there probably, but should not do any damage. You can use the RunningOnx64 test in x64.nsh and only use SetRegView on x64 if you want.

Gato Muerto
11th July 2007 20:47 UTC

Thanks. I have tested it on x32 machines, and couldn't see any difference, but that doesn't mean anything...

kichik
11th July 2007 20:54 UTC

The flag for the registry functions are passed regardless of the architecture, if SetRegView is used. It still works on XP and some other versions of Windows I've tested, but you can never know what an unsupported flag can do. It's best to only use SetRegView for Windows x64.

Gato Muerto
11th July 2007 21:04 UTC

Augh. More code to add and debug. >_<

Well, glad to know. Doing the work now could mean a lot less work in the future, when x64 architecture is more common.

Gato Muerto
14th August 2007 01:47 UTC

Going back over my list of "to be done" projects. Noted that the x64.nsh uses DisableX64FSRedirection & EnableX64FSRedirection. However, according to M$, this is no longer proper, and they recommend using the Wow64DisableWow64FsRedirection and Wow64RevertWow64FsRedirection functions.
http://msdn2.microsoft.com/en-us/library/aa365744.aspx

Should this be corrected?

Additionally, my education in programming stopped with grade-school BASIC and some minor self-taught Pascal. I am somewhat clueless regarding system:call and such, so forgive me if I am asking obvious questions.

It doesn't appear that x64.nsh/DisableX64FSRedirection checks to see if it is running on a x64 OS before attempting to run. Am I missing something, or was this an oversight?

Anders
14th August 2007 08:52 UTC

They say the problem is with nested calls, this probably does not apply to nsis. I would guess its more if you create a 3rd party library and then you don't know what state to revert to when you are done. Also, if we changed to the "new" functions (MSDN lists XPx64 support) we would have to remember the original state in a variable. But it guess kichik has to chime in with the final answer.

As for the lack of OS check, the system plugin checks with GetProcAddress if the function you want to call exists, if it doesnt, its no big deal, the call just doesnt take place.

kichik
14th August 2007 20:50 UTC

What Anders said + the new functions have higher version requirements.