Unicode

I found this error on WinXP SP3. But in Vista SP1 the same error.

This is report:
Error name: APPCRASH
Application name: makensis.exe
Version: 0.0.0.0
Time stamp of application: 4885ef1f
File with error: makensis.exe
Version: 0.0.0.0
Time stamp of file: 4885ef1f
Code of exception: c0000005
Stack of exception: 00006c6b
OS version: 6.0.6001.2.1.0.768.2
Language: 1049
Advanced info1: 8a1e
Advanced info2: 9e3d13911997afb9af7c20e45db763b3
Advanced info3: 3aba
Advanced info4: 1beeb7f8f0c5e3f3f9c6c0f5ff97433e

1. In script: CRCCheck off
2. Generating installer with name 'example.exe' without code with UPX.
3. Command line: upx -9 example.exe
4. Installer is working! Files is copying to folders. But the error when uninstall.exe creating.

Is there a plugin you are using to get this UPX capability? Or are you calling the UPX tool straight as part of your script? I only support the default plugins that come with NSIS. If you are using a third-party plugin, you will need to contact them for help or convert the third-party plugin to interface with the Unicode NSIS using Unicode strings. It seems like either a plugin that has not been converted for Unicode support or an UPX issue. Either way, I don't think I can help you in this case. I apologize.

I calling the UPX tool straight as part of script.

See in file NSIS.chm:
"NSIS Users Manual" -> "Chapter 5: Compile Time Commands" -> "5.1 Compiler Utility Commands" -> "5.1.10 !packhdr"

Sorry, there is new code now. I will try the new code and will write the result here.

Same result.

Script file contents:

;...<code before>...

page licensepage

;...<pages>...



!packhdr ....



Section First

 ;Some data

SectionEnd

;...<code after>...

Output in makensis:

;...<code before>...



!packhdr



Section First

Its all! Why?
Application error at this place.

I'm an engineer on Second Life (www.secondlife.com). We've been using NSIS for years and love it.

Recently we've noticed that installers built with NSIS 2.39 (and earlier) fail to launch on east Asian systems when the path to the installer has a non-roman character in it. For example, if a Korean user has a Windows username with Korean characters in it (í•œêµ_ì–´), and the installer is on the desktop, it fails to launch with the error:

NSIS Error: Error launching installer

I've noticed that I don't get this error if I build the installer using NSIS Unicode.

Is this a known issue with the mainline NSIS? Do I have to switch to NSIS Unicode if I want my Asian users to be able to run the installer off their desktops?

Thanks,

James Cook
james@lindenlab.com

If the user had used MBCS (ANSI codepage) for their path names, it might have worked. But usually when Windows creates the User's documents and settings directory using the user name, that user name is going to be Unicode, not ANSI codepage, even if the operating system codepage is Korean in this case. So that's why ANSI NSIS has trouble with it. However, Unicode NSIS has no problems at all.

Not entirely true, Jim. In NTFS, all path names are Unicode (actually, this also applies to LFNs in FAT), so Windows will convert to ANSI when applications call the relevant *A-APIs.

However, the ANSI codepage used is dependant on the setting selected for non-Unicode applications in Regional and Language settings in the Control Panel. If you've selected a codepage that doesn't contain the required characters, it'll fail to make a proper conversion because the unmapped characters are converted to a ?, which in turn means the path won't be found.

Now, although I kind of doubt James posted the actual text in question, it would appear to contain 2 characters that aren't Korean (at least they appear as good old "I don't know what this character looks like" to me). That might indicate characters outside of CP949, the Korean codepage.

(There's also an underscore, but that's in the 0-127 area, where they've only changed \ to ₩ - just like CP932, Japanese, has ¥ at that position.)

Ah, thanks for the clarification Pidgeot. So the user name issue then must be that people like to have their user names in their native language which naturally causes Unicode characters to be included into the desktop path which is something like c:\documents and settings\<user name>\desktop or on Vista c:\Users\<user name>\desktop.

Now what if the system codepage is not Korean but the person still wanted to have a Korean user name? How does the system convert that to codepage for NSIS to manipulate and back to Unicode? I think that's also a legitimate issue.

So perhaps recently, there was a change in NSIS which required that it needed to know where itself was not just where the user wanted to install the program.

Just to elaborate how it all works with an example:

If the system codepage is not Korean, and you use ANSI functions to fetch a path containing Korean characters, you'll get a question mark in place of each non-convertable character.

Having your username in your native language is usually not an issue, because you'll usually have the same language set for non-Unicode applications, and when you aren't using that language in path names, it's usually English anyway, which can be expressed in all code pages.

To give an example, the issue can pop up if I (being a Danish user) wanted to run a Japanese non-Unicode program. Running it with the non-Unicode language set to Danish would result in mojibake for text, so I'll need to set it to Japanese to run that program.

(There's a tool by Microsoft, called AppLocale, which can set this for a single application - and because changing the global setting requires a reboot, that's a good thing - but that's a different story.)

If my username then included a Danish letter, such as Ã…, any non-Unicode application attempting to get the username would read that character as a question mark, because Ã… does not exist in the Japanese codepage. The folder "Ã…l" (Eel) would thus become "?l" - but because the folder is called "Ã…l" and not "?l", it can't access the folder.

You might argue that it could try to guess - after all, ? is a wildcard - however, there are issues with that:

1) If there are multiple matches (let's say we also have a folder named "Øl", meaning beer), you don't know which one you're supposed to get. Acting differently in the case of "only one match" is not consistent behavior, so that's not that good of an idea.
2) Strictly speaking, NTFS does allow a file to be called "?l" - the only character not really allowed is NUL, much like in Linux. Win32 adds several characters to the "disallowed" set due to backwards compatibility

Thanks for the details, guys. I agree the issue is people with paths to the installer like C:\Documents and Settings\<username-with-extended-chars>\Desktop. I presume this means I'm not doing anything wrong with my plain-old-NSIS installer?

I wish there was a workaround for plain-NSIS because we've got it installed on a plethora of developer and build machines.

Could it be because we're using CRC check?

Can you guys think of a workaround other than switching to NSIS Unicode?

James

(I attached our installer script below in case you're curious.)

I made a very small installer to test, and I get the same result with CRCCheck On and CRCCheck Off - so that's not it.

Now, this is just my opinion, but I would probably go Unicode now. It's not something you can really avoid, since the Unicode branch will eventually be moved into the main build, so not doing it now would only postpone the migration.

The alternative would probably be an addition to a readme or FAQ or whatever, with a guide on how to change to the appropriate codepage, or suggesting the workaround to place it in a path only containing ASCII.

I'd like to stress again: The problem is not with paths that contain characters <= U+007F, since those are unchanged in all Windows codepages (except the issue with \, but that one is handled gracefully). The problem is with characters that are >= U+0080, and are not representable in the user's codepage.

Thanks guys. I really appreciate the help. I think we'll switch the NSIS Unicode for the time being.

Enjoy the coffee, Jim! :-)

James

I'm having another problem...

An installer built with NSIS Unicode 2.38.1 does not show the "unpacking data: ##%" dialog during load. The same installer built with NSIS 2.39 does show the dialog.

My installer is 21 MB compressed with LZMA, so the unpack takes a noticeable amount of time. Is there a way to get that dialog back, or put up some other sort of progress display?

Thanks,

James

Does that happen with any compression setting? I'm imagining that it does. And also, does it print anything at all like "unpacking data:?" or is there no analogous printed line at all?

Jim,

Should I run the installer from the command line to look for output? I get no window open of any kind during unpack. The first window I get is language select (from our .onInit() function).

I've only tried it with SetCompressor /solid lzma

Tomorrow in the office I'll try it with other compression settings. Any particular ones to try?

James

(I attached the shell of our installer -- all the %%FOO%% sections are automatically populated with our lists of files.)

It doesn't look like installers print anything to the console -- running from the command line shows nothing.

Also, I have the same problem (no "unpacking" dialog) with:
SetCompressor zlib
SetCompressor /solid zlib
SetCompressor /solid bzip2

Hope that helps.

James

A workaround could be to add all files in the right order in the solid archive (using ReserveFile). Then there will be no need to extract a lot of data at startup.

Thanks for the suggestion, Joost. I ended up reordering my .nsi script so all the functions are at the start of the file, followed by all the files. We used to have the .onInit function at the end of the .nsi file, which I think was the cause of the slow unpack. This seems to eliminate the startup delay that caused the "unpacking data" dialog to appear and solves my problem.

I didn't see this documented anywhere -- does NSIS interleave the packed file data and the function definitions? If so, it seems important to put all your user-written functions before the first big list of files. Should that go in the docs somewhere?

Jim, just so you know, NSIS Unicode definitely doesn't show the "unpacking data" dialog, though, even on slow unpacks.

Thanks again, guys. James

2.40?

Are you going to update NSIS' Unicode version up to 2.40?
There are some important (for me) improvements and bug fixes.

Thank you.

I'm currently at the busiest time of the year for development. We have what we need right now as far as an installer is concerned so I'm not going to be able to update this any time soon. I'm hoping if the NSIS guys are really going to support Unicode, they pick it up soon.

kichik, whats the status of this Unicode support? Needs reviewing? Needs something fixed? Debian folks are wanting this too.

I've been looking at the code changes just now.

First thing that got me was all the ANSI/Unicode dirs all over the place with the files in UTF-16. I didn't think that is a good idea, so I converted all the Unicode directories into UTF-8 and looked at the differences between them.

Based on that, here are some suggestions:

Don't duplicate stuff that doesn't have non-ANSI characters in it.

Where the only change between Unicode and ANSI is to re-encode the © (Copyright) symbol, just replace it with the word "Copyright", since that is as legally valid as "©".

Where differing parameters to System::Call are needed, add an !ifdef UNICODE or something.

Some more comments:

Don't add/remove whitespace where it isn't needed.

Menu/images/Unicode/create_header.py isn't needed, Scripts/release.py does that stuff.

Please don't include proprietary formats like Paint.NET in nsis, standardised PNG, BMP or ICO are best.

For the languages, store everything in UTF-8 in the source code and convert to ANSI at makensis time if the user sets a specific code page (and error out if all the characters are not available in that CP). On Linux the iconv function can help and for people running makensis on Win9x the unicows library could be useful:



Switching to gettext style translations and the PO file format for translations might be useful too, dunno how well Windows supports it though.

Use ISO 639-1 codes for language type instead of English words for the languages.

For the Contrib/UIs, add ifdefs, compile them twice and install them into Unicode and ANSI directories. Alternatively makensis could detect a "RichEdit20" class and replace it with "RichEdit20A" or "RichEdit20W" as appropriate.

Examples/Unicode/bigtest.nsi doesn't need to use the ¢ character.

Examples/Unicode/makensis.nsi doesn't need ANSI/Unicode in the registry/etc does it? Just make one version of makensis that does both ANSI and Unicode depending on a command-line switch or a script command. Same for a lot of other stuff.

Will make some of these changes myself, look at the rest of the changes and add more comments.

I just filed issue 2230926 related to NSIS 2.40 (and earlier) installers throwing an error on startup if the path to the installer contains wide characters.



NSIS Unicode doesn't seem to have this problem. I wonder if the parts of the Unicode patch that fix this issue could be ported over separately, or as an initial step.

James
james@lindenlab.com

Hello, I was using ANSI version of NSIS until I faced the issue with unicode user name. Then I found unicode nsis and it solve the issue.

I just read though this entire thread, and now a bit confused. Looks like I was supposed to convert scripts to unicode?

I did not converted my scripts to unicode, they are still ANSI. But installer now works fine with UNICODE user names, so, definitely, I compiled unicode version.

Why I was able to compile ANSI scripts? Isnt it should be mandatory converted to UNICODE?

I did nothing else but reinstalled NSIS and recompiled it under unicode NSIS. Looks like everything is fine.
What I am missing?

What it does is try to convert the ANSI script to Unicode using the system's codepage. That's probably okay if you are the only developer and you never change your system codepage to something else but technically this is incorrect. In order to make your NSIS script portable, you should convert your script to UTF-16 (using notepad.exe, for instance). If you don't, you might find that later down the line, users report that your installer produces junk text and you might not remember why. It's simple to do, so just load it up with notepad and save as Unicode and save yourself future grief.

Thanks. One more question, if you please.
We are planning to do some localization.
Question regarding existing language files: I've noticed, that with unicoed nsis you ship non-unicode language files. Correct?

How non-unicode language files survive with unicode installer? What shell I do about it? We server like 9 languages, like Gernam, French, Spanish, Simplified Chinesse, Traditional Chiness, Korean, Dutch, Italian, Japanese, Portuguese.

No, the language files that come with Unicode NSIS are indeed Unicode. I'm not sure why you think that wasn't the case, unless you've been looking at the "regular" NSIS files.

In unicode NSIS installation files in ".\nsis\Contrib\Language files" are not unicode. See attahcment.

/edit: attachment deleted. I was looking into the wrong NSIS installation folder.

Those aren't the Unicode NSIS files. I've looked at the exact same files here (2.38.1-Unicode), and they are Unicode here.

Note that, by default, Unicode NSIS installs itself into Program Files\NSIS\Unicode - it does not overwrite the ANSI NSIS files (but the context menu is changed to point to the Unicode NSIS instead). The path you give suggest you're looking at the ANSI NSIS files.