Problem with WriteRegStr and AppCompatFlags

Program requires compatibility mode (Win98/ME) to run properly in Vista. To streamline this, I have added some generic code to a series of installer scripts in order to "Plug and Run". Unfortunately, the script is not writing the required registry entry. I am likely missing something quite obvious, but was wondering if anyone else has an idea. Here is the important code:

!define APPCOMPATMODE "WIN98"

  # Registry entry in order to enable compatibility mode settings for the application.

  # Put a space between each mode selected.

  # The following modes are available:

  #    {WIN95 WIN98 NT4SP5 WIN2000 WINXPSP2}

  #    256COLOR

  #    640X480

  #    DISABLETHEMES

  #    DISABLEDWM

  #    HIGHDPIAWARE

  #    RUNASADMIN

#snip loads of code...



Section "Base Components" BASECOMPONENTS

#snip loads of code...



  !ifdef APPCOMPATMODE

    WriteRegStr HKLM "Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" "$INSTDIR\${PROGEXE}" "${APPCOMPATMODE}"

  !endif

#snip loads of code...



SectionEnd

The odd thing is that if I change HKLM to HKCU, the code works perfectly, the registry entry appears correctly (in CU), and the program works (for that one user). Perhaps I am missing something obvious, but what is wrong here?

your installer needs admin rights to write to HKLM, use "RequestExecutionLevel admin" in your script

Thanks. Like I said, was missing something rather obvious. :P
Actually thought that since Admin rights are needed to execute anyways, the installer was already running with Admin rights.

Thanks for the help. Have a very similar issue, and believe reposting in the same thread would be better all around.

I am reusing the same code for a new installer and running into another almost identical problem. Run the code on Vista, everything is great, works fine. Move to a machine running the 64 bit version of Vista, and no joy. Again, HKLM fails, HLCU works, which leads to believe that the ReqExeLev isn't working for some reason. Same CDROM on both the 32bit and 64bit OS, so rather confused as to the cause here. Is there a difference for how the x64 version of Vista handles "RequestExecutionLevel"?

Probably registry redirection. Try using SetRegView 64

Thanks, that did it.
Does this setting do anything if the OS is 32-bit instead of 64-bit?

Well, the extra flag for RegOpen/CreateKey is still there probably, but should not do any damage. You can use the RunningOnx64 test in x64.nsh and only use SetRegView on x64 if you want.

Thanks. I have tested it on x32 machines, and couldn't see any difference, but that doesn't mean anything...

The flag for the registry functions are passed regardless of the architecture, if SetRegView is used. It still works on XP and some other versions of Windows I've tested, but you can never know what an unsupported flag can do. It's best to only use SetRegView for Windows x64.

Augh. More code to add and debug. >_<

Well, glad to know. Doing the work now could mean a lot less work in the future, when x64 architecture is more common.

Going back over my list of "to be done" projects. Noted that the x64.nsh uses DisableX64FSRedirection & EnableX64FSRedirection. However, according to M$, this is no longer proper, and they recommend using the Wow64DisableWow64FsRedirection and Wow64RevertWow64FsRedirection functions.


Should this be corrected?

Additionally, my education in programming stopped with grade-school BASIC and some minor self-taught Pascal. I am somewhat clueless regarding system:call and such, so forgive me if I am asking obvious questions.

It doesn't appear that x64.nsh/DisableX64FSRedirection checks to see if it is running on a x64 OS before attempting to run. Am I missing something, or was this an oversight?

They say the problem is with nested calls, this probably does not apply to nsis. I would guess its more if you create a 3rd party library and then you don't know what state to revert to when you are done. Also, if we changed to the "new" functions (MSDN lists XPx64 support) we would have to remember the original state in a variable. But it guess kichik has to chime in with the final answer.

As for the lack of OS check, the system plugin checks with GetProcAddress if the function you want to call exists, if it doesnt, its no big deal, the call just doesnt take place.

What Anders said + the new functions have higher version requirements.