Unicode

As I mentioned before, for the languages that have codepage support, I simply ran a a2u.exe to convert the ANSI script to the Unicode script. I did not hand edit all of those. I don't know 25 languages, after all. 🙂 So you could have one source file for each language and then GENERATE ANSI and Unicode versions of the example files. So in essence, you never have "oops" mistakes because there is only one source.

From your reply, Kichik, it seems as though there IS going to be a Unicode build of some form. Is this correct? Does that mean that this Unicode build is going to be integrated into the main release sometime soon?

If so, that will be great. I for one have need for an installer in one of the languages mentioned below (Georgian), and next year I may have need of one of the others. PLUS, the benefits of not having to switch code-pages in order to have the install show up with intelligible text is also very nice.

Additionally, how many people have tried this version of the installer? I'd like some idea of how stable it is and how it is supported on different OSes. My experience so far has been very good. I was able to build my install with the version made available here without any changes to my script (other than saving the file in Unicode), and it is a pretty large script (1K lines). I'm now in the process of localizing the Georgian version, and I'll make sure to post back with how successful the results were.

Unicode seems to be the wave of the future, and I'll be really happy to see NSIS make the path available for those of us who want it. After all, the product I'm shipping does not support any of the ANSI versions of Windows, and Microsoft doesn't support them anymore either.

KrisAster, there's no time frame and my bet won't be on soon.

For those using the Unicode version of NSIS, I took the latest 2.33 source and merged the changes onto the Unicode version. The new binaries and source are posted to http://www.scratchpaper.com. Let me know if you find any problems with this release. Also let me know if you find the Unicode version useful.

I modified the Unicode language files in the installer to contain vernacular names for the various languages. So you can do something like this:

Thanks for this work. the unicode build works well.
I'm chinese, and using japanese windows edition to study japanese. And really many chinese language only nsis setup bother me which can't display correctly on my system.
Really great, hope this Unicode support feature will be merged into the official NSIS.

Thanks for your patch, it seems to be working great.

I'd love to see this included in the official source.

Unicode NSIS has been updated to support Georgian (a Unicode only language). Updated Unicode Simplified Chinese language files. Also fixed some bugs in
Unicode Turkish language file and Unicode zip2exe. Please give it a whirl and let me know if you see any more problems. As always, the site is http://www.scratchpaper.com

I'm trying to use your Unicode version. It works well until the use of some plugins.
I tryed to convert the GetVersion plugin of AfroUK.
I put your exDLL and recompiled with no success: the compiler give me some errors. I think that the problem is the definition of _UNICODE but I don't know where to put it (I don't know C++ and its compiling process).
If someone tells me how to do it, I think I'm able to translate the others plugin I need for my installer.

Thank you

Well, it will be a bit difficult not knowing C++ if you want to convert other plugins. But you could try downloading the source from the NSIS branch, then download the source from my site. And you could compare the source files for a smaller plugin like LangDLL and see what the differences are.

BTW, other than the GetVersion plugin, what other plugins do you use?

Yes, C++ isn't the perfect language to make small change for a person doesn't know it.
Anyway I'll try to do it (is this the right time to study even the C++ 😁 ?).

These are the plugins that I use in my installer:
- GetVersion (to kown the Windows and IE versions and the OS architecture)
- GetSize (to know the size of the data folder)
- HwInfo (to know the processor name)
- nsProcess (to kill a process before uninstalling)

Bye

Yes, C++ isn't the perfect language to make small change for a person doesn't know it.

No language is.

Originally posted by CodeSquid
No language is.

You're right, but C++ isn't the easier one.

Originally posted by jimpark
Well, it will be a bit difficult not knowing C++ if you want to convert other plugins. But you could try downloading the source from the NSIS branch, then download the source from my site. And you could compare the source files for a smaller plugin like LangDLL and see what the differences are.

BTW, other than the GetVersion plugin, what other plugins do you use?

I converted GetVersion and works well only under ANSI version (like the original one). Can be the problem in the definition of _UNICODE (I don't find it)?

Thanks

Originally posted by prz
I converted GetVersion and works well only under ANSI version (like the original one). Can be the problem in the definition of _UNICODE (I don't find it)?

Thanks

I added the _UNICODE string to the pre-processor strings, but the output of the plugin is the same of the ANSI version.
something else to make a unicode version of the plugin?

Bye

Thanks for the great work on Unicode NSIS, and for hosting it. I just used it for an english/french/czech installer (and I should be adding more languages soon). It still needs to be more thoroughly tested but it seems to have worked fine. 🙂

A note though: I didn't convert the install script itself to UTF16-LE (I did at first but NIS Edit wouldn't display it correctly). It worked anyway. It was completely ANSI, but ANSI is not a subset of UTF16 so I'm (happily) surprised.

I consider Unicode as the way to go: displaying multiple or Unicode-only languages is more and more needed, and I'm really glad that NSIS is moving to complete Unicode support. 👍

Thank you for the kind words of encouragement. I've never used NIS Edit but I take your word for it that it doesn't understand UTF16. If you didn't convert the script to Unicode, perhaps the ANSI one was run and it created an ANSI installer? One thing you could do is check that the ANSI one works first, then convert it to Unicode using notepad or a2u.exe and then create the installer again using the Unicode version. If it worked for the ANSI it will most likely just work for the converted Unicode version (barring the use of 3rd party plugins that have not been converted to support Unicode.)

Actually, the ANSI script was compiled with classic NSIS until now. It works correctly. But I wondered if making a unicode installer was possible, and found out about Unicode NSIS.
I don't use any 3rd party plugin, only language files, which I converted to UTF16-LE. And I updated the link to the compiler in NIS Edit to Unicode NSIS. The only issue was that NIS Edit displays only the BOM of a unicode script.

Classic NSIS wouldn't compile even the ANSI script (using unicode language files), but Unicode NSIS produces a working installer.

However, I just compiled the Unicode script on Unicode NSIS, and did a byte-to-byte comparison of the 2 "unicode" .exe installer: they are indeed not exactly the same.
I couldn't spot any difference by running them though, even with czech texts (usually requiring CP1250 while I use CP1252 myself). I'll have to test with russian or chinese.

Anyway, I'll use the unicode script from now on to be on the safe side.
I described everything here in case it is of some help to you. I'll be happy to make some more tests or give some more details if you wish.

What about making a Unicode version that supports both Windows 95/98/Me as well as Windows 2000/XP/Vista? I think it should be possible to have scripts and internal NSIS strings in Unicode but convert the user interface strings to ANSI for 95/98/Me.

Because there is no simple one-to-one conversion of ANSI NSIS scripts to Unicode (multiple codepages can be used in a single file for e.g. LangStrings), we can just call it NSIS 3.0 and require scripts to be in Unicode.

Actually the problem is during the loading the exe and dynamic link time. By using Unicode-based Win32 calls, the program fails to even run. Not sure how to solve that except by distributing unicows.dll with the installer on Win9x. But seriously, I think we should just drop Win9x support soon. Looking at browser OS stats at http://www.w3schools.com/browsers/browsers_os.asp, only 1% of the web users use Win98. And the demographic for users of products that ship with NSIS installers are probably netizens. Every year that passes by, the need to support Win98 and therefore both ANSI and Unicode will dwindle. We will eventually only need Unicode. 😁

Merged in 2.34 changes into the Unicode version of NSIS. Go to the usual place: http://www.scratchpaper.com for the binaries and source.

Added Pashtu support. Thank you Zabeeh khan.

I've tried to compile your patch on Linux but it utterly failed.

Can you please update your patch so that it compiles on Linux?
If you don't have Linux, you can download it from http://www.ubuntu.com/

Originally posted by jimpark
Actually the problem is during the loading the exe and dynamic link time. By using Unicode-based Win32 calls, the program fails to even run.

It's possible to call these APIs on run-time depending on the Windows version. A framework for this method is already included in the NSIS source and is used for various functions that are not available in all Windows versions.

CodeSquid, it's not that I don't have Linux. I did this project because of a need for a Unicode installer on Windows for the project I'm working on. I'm giving the source back to the community as-is. I'm only updating the code so that my code base doesn't go out of sync with the latest changes. And I'm offering it to the public and especially the NSIS developer community so they can potentially use it to create an official Unicode version of NSIS. And while there isn't such, the developers who need a Unicode solution for their installer will have something that works right now. I could just update it internally and never let anyone else use it or see it but that would not be following the spirit of open source. I don't have the time or resources to make sure the source builds on Linux, Mac and whatever else NSIS usually can build on. So if it's important to someone that the Unicode source builds on Linux or on the Mac, I hope that someone steps up to the plate and makes it work for those OSes.

I'd very much like this in the official NSIS code. Unfortunately I don't have the time to work on this either. 🙁

I think it will require quite some effort to have combine a non-Unicode/Unicode version in the same source. This would also cause a lot of trouble with plug-ins. Having a major new release (NSIS 3.0) with only Unicode support and Unicode script files would probably be a better solution.

If only Windows 2000 and later need to be supported, the current Unicode version is mostly complete (expect for some issues like Linux compilation). Windows 95/98/Me cannot be supported forever. Microsoft has dropped support for all these versions some time ago, so there are also no more security fixes. The C++ runtime of Visual Studio 2008 doesn't even support them anymore.

So maybe this would be the time to remove legacy stuff and introduce some breaking changes.

I'd like to suggest support for UTF-8 scripts, with and without BOM. I exclusively do UTF-8 without a BOM, as I feel it hurts more than it helps: it only serves as identification (even though in almost all cases, you'd do just fine by assuming UTF-8, and using ANSI as a fallback), and excluding it allows for a neater handling by non-Unicode programs.

If the file is BOM-less and contains invalid UTF-8, I think it should be read as ANSI and internally converted to Unicode for the actual installer. Such behavior would help reduce the effort needed to migrate installers to Unicode, as it should allow most existing scripts to continue compiling with no changes (as long as they don't use any plugins not supplied with NSIS).

I do realize there's a potential problem when a file in a different codepage is included from the script. This could be solved in various ways, such as using a stack to keep track of codepage information. Whenever inclusion starts, the current info is pushed onto the stack. When inclusion stops, pop it back.

Although there is a risk of incorrectly interpreting an ANSI script as UTF-8, it is extremely slim, due to the fact that it requires some very specific byte patterns as the only non-ASCII text in the entire script.

The real crux of the problem is the conversion from ANSI to Unicode. What codepage should you use? Hopefully, the script has that information. But even if it does, it may mean you have to parse a significant portion of the scripts to figure out which code page the text is in which means writing a lot of code that is not there currently. Or I guess you could default to using the system codepage but that may not be the intent of the developer. If someone develops a Chinese installer and gives it to someone who uses an English codepage on their machine, the same installer would produce junk characters on its messages.

Originally posted by jimpark
Or I guess you could default to using the system codepage but that may not be the intent of the developer.

I'm not sure you understood me correctly, but my idea is to use the system codepage to convert it at compile-time; allowing us to minimize issues. It would likely be best to issue a warning about ANSI scripts being deprecated when it has to do this, so the developer knows to use UTF-8 or UTF-16 instead of relying on the system codepage.

If someone develops a Chinese installer and gives it to someone who uses an English codepage on their machine, the same installer would produce junk characters on its messages.

As long as they only get the compiled installer, no problem, as it would have been handled by then. If they distribute the actual ANSI script, without realizing this change had happened, then yes, they'd have a problem, but this should not happen very frequently - even less if we issue that warning - and the fix is simple; that particular script just needs to be converted into a Unicode format.

For installers creating a dynamic script, nothing prevents us from using UTF-8 as a default output codepage for file writing to minimize issues; as long as they are able to change it from the script (for when they need a specific codepage). Yes, that one would require a change, but you'd be affecting fewer scripts, thus easing migration.

Yes, I do understand you. I am talking about the developers sharing scripts with different codepages set on their machines. The problem you are trying to fix... which is converting their ANSI scripts to Unicode is trivial. Open it up in notepad.exe and save as Unicode. Or use the a2u.exe program I provided. All the proposed changes you are talking about only saves the little tiny headache of converting your script once to Unicode. The real problem is that people write scripts that are shared like libraries, like the language files. They all still need to be converted to Unicode, whether UTF-8 or UTF-16 before they are shareable with others. There's no way around that. We can do some intelligent guessing as the NSIS program parses the ANSI scripts but it may not always be correct.