Unicode

I don't know of a way to enumerate all the supported language scripts on a given user's computer. I can do it per locale or language but that's not sufficient since the user's computer is usually set to one language/locale but may be able to support scripts of other languages. If someone knows how to list all the scripts supported by a given system, I think it's just a matter of calling GetStringScripts() and VerifyScripts().

Anyway, this shouldn't be an issue--everyone should have the East Asian Language support turned on! 🙂

Unfortunately, those two methods don't exist on versions prior to Vista (and there, East Asian Language support is always turned on) - you can get support on XP SP2 and 2003 SP1, but that requires a separate download (and doesn't help with Win2000).

You can use EnumSystemLocales with LCID_INSTALLED to check which locales are "ready for use" (if you know specifically which one to check for, you can just use IsValidLocale), or you could check the possible languages groups with IsValidLanguageGroup and LGRPID_INSTALLED.

However, all of this requires that you have a mapping between the languages and the locales (or language group) - I can't remember if that information is currently available to NSIS.

I don't think it is available hence my initial thought to use GetStringScripts(). Which makes me think that may be the use of the native language to display the language name may not be a good idea. Or maybe I should combine them together so that there's a legible English version of the name in the drop down menu along with it in the native script.

Couldn't it be a missing font issue, rather than an unsupported language?

Technically, on NT-based OSes, it's ONLY a matter of a missing font (and to some degree, the version of Unicode they support), but that's why LCID_/LGRPID_INSTALLED should be used (instead of _SUPPORTED). Using that flag makes the operating system report if it has everything it needs to display a string based on a given locale.

The problem is, checking against a specific font name is less than optimal, since their names may be localized (the japanese fonts MS Gothic and MS Mincho do that, to give examples), and Windows might know of a font you don't. That's why you need to check on the locale or language group, if you're going to filter on it - but to the best of my knowledge, no WinAPI function exists to detect locale, language group, or script for a given string (except those Jim already mentioned, which are only built into Vista and require a separate download for XP and 2003, therefore not really being usable here).

Latest 7-zip alpha build supports opening Unicode NSIS installers 👍

Our unicode-nsis installer calls a java program using nsExec and all the output from the java program gets directed into the scrolling text box. If we choose English for the installation language, everything in the text box looks fine , but when an exception stack trace is printed, a part of the stack trace gets cut off. If we choose Japanese for the installation language, weird characters are displayed in place of forward slashes. What could be the reason?

I've hacked the RealProgress plugin source to get it working with the unicode-nsis. Could it be the reason for this behaviour? (seems unlikely though). I'm attaching a snapshot.
Look at the second line in the image. It is actually supposed to be C:/PROGRA~1/Java/.. so on. But all "/" are replaced with some weird character. Similarly, both the stack traces are only partially displayed.
Any help will be greatly appreciated. Thanks in advance.

I can't say why it cuts off part of it, but the reason \ is replaced by the ¥ sign is for historical reasons: Japan (and Korea) put their currency symbol in their code page back in the pre-Unicode days. When Unicode was introduced, it was believed that people were too used to seeing the currency symbol as the path separator, so they kept it that way. (There's no real "right" answer to that problem, so you can't really blame them, even if it does seem a bit foolish these days)

As a result, all Japanese fonts (MS Gothic, etc.) show a Yen sign where the backslash is supposed to be, just like Korean fonts (GulimChe, etc.) show a Won sign (â‚©).

sridhare, when an exception is thrown and the stack trace is printed by your Java app, there are potential problems with buffered IO that is probably causing the cutoff of the text output. I don't know how much control over the IO you have with the Java program but you might want it to be unbuffered when outputting or at least flush the IO when it exits. The other issue is that I think NSIS will also buffer the input as well so it will display everything if it thinks the IO has been closed or is flushed.

BTW, I have not been able to work on the updates to the Unicode NSIS for a while. I will be very busy until probably sometime in June. My apologies.

Thanks for all the response. This java exception stack trace cut off never happened when we were using (non-Unicode) NSIS. Are you saying that the buffering issue could arise in the Unicode NSIS ?

No, there isn't anything different about how NSIS handles the IO buffers with Unicode. When an exception happens, does the output look like junk? I don't know how it is in Java but in C++, the exception strings need to be ASCII and I wonder if something similar is happening in Java.

Originally posted by jimpark
No, there isn't anything different about how NSIS handles the IO buffers with Unicode. When an exception happens, does the output look like junk? I don't know how it is in Java but in C++, the exception strings need to be ASCII and I wonder if something similar is happening in Java.

No, the output doesn't look like junk, but it gets displayed incompletely - This is the actual exception message that used to get displayed fine in the log window when I was using the official NSIS :

INFO: Not started: http://localhost:80/identity

java.net.ConnectException: Connection refused: connect

	at java.net.PlainSocketImpl.socketConnect(Native Method)

	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)

	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)

	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)

	at java.net.Socket.connect(Socket.java:516)

	at sun.net.NetworkClient.doConnect(NetworkClient.java:152)

	at sun.net.www.http.HttpClient.openServer(HttpClient.java:365)

	at sun.net.www.http.HttpClient.openServer(HttpClient.java:477)

	at sun.net.www.http.HttpClient.<init>(HttpClient.java:214)

	at sun.net.www.http.HttpClient.New(HttpClient.java:287)

	at sun.net.www.http.HttpClient.New(HttpClient.java:299)

	at sun.net.https://www.protocol.http.HttpURLCon...tNewHttpClient(HttpURLConnection.java:795)

	at sun.net.https://www.protocol.http.HttpURLCon...n.plainConnect(HttpURLConnection.java:747)

	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:672)

	at com.pramati.bfly.install.InstallOperations.getStatus(InstallOperations.java:335)

	at com.pramati.bfly.install.InstallOperations.dasStatus(InstallOperations.java:190)

	at com.pramati.bfly.install.BaseInitializer.start(BaseInitializer.java:301)

	at com.pramati.bfly.install.offline.StandaloneInstaller.start(StandaloneInstaller.java:488)

	at com.pramati.bfly.install.offline.StandaloneInstaller.start(StandaloneInstaller.java:475)

	at com.pramati.izpack.DekohProcessPanel.invokeInstaller(DekohProcessPanel.java:479)

	at com.pramati.izpack.DekohProcessPanel.installDekoh(DekohProcessPanel.java:346)

	at com.pramati.izpack.DekohProcessPanel.access$300(DekohProcessPanel.java:43)

	at com.pramati.izpack.DekohProcessPanel$2.run(DekohProcessPanel.java:262)

Now, after switching to the Unicode NSIS, the same exception stack trace started appearing as follows!

INFO: Not started: http://localhost:80/identity

java.net.ConnectException: Connection refused: connect

        nnection.java:747)

Also, when this cut-off happens, it also swallows some other messages printed to the window after this stack trace. Having the full exception stack traces is very critical for debugging, it'd be great if you can be of help.

Just released Unicode NSIS 2.37 which merges in the changes from the trunk into the Unicode build. You can find it at the usual place: www.scratchpaper.com.

sridhare, try the new version to see if it fixes your issue. If not, I do have another idea that might fix it. So let me know.

sridhare, I found the bug and I've finally fixed it. Sorry it took so long. I didn't have any time to work on the Unicode NSIS until recently. I just released new binaries and fixed source at the usual place. http://www.scratchpaper.com The version is 2.37.1. Everyone should get this new one instead.

Just released 2.37.2 which has an initial Vietnamese support thanks to Clytie Siddall of the OpenOffice translation team.

Whats the status of this Unicode branch? Ready to be merged into trunk?

I ask because Debian would like to use it for our win32-loader program used to boot the Debian installer on Windows machines:

Because a Unicode build will break backwards compatibility with existing scripts, it will probably be a NSIS 3.0 release.

However, Win9x support is still not avaialble. It is possible to add support for these platforms (even with the internal installer data being in Unicode) by converting the data back to ANSI for the user interface.

The other option would be to completely drop support for Win9x in NSIS 3.x.

I would love to see the code get merged into the trunk as well, as I've mentioned many times already.

As for Win9x support, it is not on my list of priorities. I personally don't have a need for it since all the software we are shipping require Win2k+ anyway. I think most developers have similar requirements nowadays.

The "unofficial" status of the Unicode NSIS branch hasn't stopped high profile projects like Winamp, OpenOffice, Flickr, Filezilla, etc. from using it. And in the near future Mozilla's products like Firefox and Thunderbird will use it as well (Unicode NSIS is already incorporated into their build environment since MozillaBuild 1.2). It just goes to show that there is definitely a need for an Open Source Unicode Installer.

And I'll mention this point again as well. My codebase generates both the Unicode version and the ANSI version that DOES run on Win9x. So I really don't see what you can lose from adopting the codebase as-is and then working forward from there to make the Unicode version Win9x compliant via unicows.dll or some other method as mentioned by Joost.

I don't think it's a good idea to have an ANSI and a Unicode version at the same time. Many scripts will not be compatible with both versions, so this will cause a lot of confusion. And there is also the issue of plug-in compatibility.

Regarding Win9x, we cannot use Unicows or something like that because this file does not exist on all systems. The only option for real Win9x support would be to create ANSI dialogs and convert the intenal Unicode data to ANSI based on the current code page. In this case, scripts, examples, plug-ins etc. will all use Unicode internally and there will be no compatibility issues.

Personally I think it's OK to drop Win9x support in NSIS 3.0 (with NSIS 2.x still being available for Win9x users) and move to Unicode scripts. But I don't know whether other developers agree with this.

I'd definately like to have Unicode support in the official version.

I see three potential options other than the one I have which is two versions of NSIS, meaning two sets of plugins (but one codebase) and two sets of scripts, which I concede can be unwieldy.

1. One makensis.exe that can generate both an ANSI or a Unicode installer. This means that there will still need to be two sets of plugins and exeheads, but potentially just one set of scripts. If we go this route, I'd suggest that all scripts are Unicode. If we start with ANSI codepage scripts, mixed scripts in a single file becomes very difficult if not impossible to support. (Imagine a single string with mixed scripts.) And we would alienate Unicode-only languages.

2. We generate one exehead but with the ability to on-the-fly convert the internally stored Unicode strings to the ANSI codepage strings to call the ANSI Windows APIs on Win9x machines. This requires that we have a small but portable implementation of wide string to multi-byte string conversion function. Maybe we can leverage some code in the Wine project to do this. But it does mean making the exehead and therefore the final installer to be bigger. The plugins would also need to do this conversion as well since they would likely get the Unicode strings on the stack. So this method requires more complexity in the exehead and the plugins.

3. Drop ANSI support in NSIS 3.0 entirely. We have one version of everything going forward, but we lose the Win9x support. But as Joost mentioned, NSIS 2.0 can cover that small and rapidly shrinking market. (The Win9x crowd is ~0.2% of the web surfing public according to w3school: http://www.w3schools.com/browsers/browsers_os.asp)

I think the logical choice is 3. But it isn't my decision.

Originally posted by jimpark

2. ...This requires that we have a small but portable implementation of wide string to multi-byte string conversion function. Maybe we can leverage some code in the Wine project to do this

Win9x has support for a limited set of Wide API's out of the box, including WideCharToMultiByte IIRC

Originally posted by jimpark

3. ...according to w3school

That is a dev specific site, yes, the % of people using 9x is small, but not THAT small


I prefer option 1 myself, old scripts would stay in whatever ansi codepage they have always been in, unicode scripts would need a way to specify that they are unicode, with either a BOM or a nsis instruction at the top

You might be right about the percentage of people using 9x being too small but I would still argue that around 1% is right. Is there a different number you are aware of?

I wouldn't mind getting option 1, either. We might be able to do it by inserting both the Unicode and ANSI exehead into every installer (option controlled, of course). It also means two copies of every plugin as well. So your installer overhead would be 2x what is now if you wanted a "Universal installer."

Unfortunately, this change is not something I can do. Time constraints are a problem for one. But more importantly, it would fork the code too much and I wouldn't be able to maintain the code to be in sync with the main build. It could easily end up being an abandoned fork.

But we are all eager to see the main NSIS developers start work on an official Unicode version. Is there any work going underway for this support currently? If not, is there any tentative plans for Unicode support in the near future?

Personally, I see no reason to continue win9x development. I know that kichik and other NSIS developers disagree, but the userbase has dropped off dramatically. Based on winamp.com web stats, Winamp's win98/ME userbase went from 3% in 2006 to less than 1% by January 2007 which triggered us to drop support for the platform entirely.

I agree that developers should drop 9x support. It's like the idea of having 16bit/32bit software build few years before 😉

Keeping the compatibility would require too much effort, effort that could be used other things.

And if you don't trust you can make a survey, just don't forget to put the questions right 😉

If we want to continue Win9x support, the best option would be to have all scripts and plug-ins in Unicode but add support to the exehead to create ANSI dialogs on Win9x.

If this is not possible, the scripts should move to Unicode anyway but there will be two sets of plug-ins and exeheads.

However, it may indeed be better to drop Win9x support for NSIS 3.0.

the script should stay like they are, BOM for unicode files and no BOM means ansi, otherwise tricks played with !system that you include again will have to be unicode aswell, and the console is not unicode by default

But then you also need two sets of langauge files etc., which will all be way to complicated.

Originally posted by Joost Verburg
But then you also need two sets of langauge files etc., which will all be way to complicated.

You can't convert the lang file on the fly from ansi to unicode when you know the codepage ?