Unicode

Let's be fair MSG. If the merging of my code into the trunk is really what's holding the new NSIS up, then it's done. It was done four years ago when I presented my work. The reality is they are reinventing the wheel. They are redoing my work, their way.

And frankly, in the last four years, there hasn't been much work being done on NSIS at all. The biggest being the Unicode port I did four years ago. Just check the code changes here. Apart from the copyright year updates, I never had to struggle very much to update my codebase to reflect the trunk's. Usually, the changes per release was just a few lines here and there.

And really what does it matter now anyway? I personally don't know a major project that uses the ANSI version. Do you? All the products I use, if they use NSIS, use my Unicode version. The only motivation I can think of to use the ANSI version is if I needed to support Windows 98/ME customers. But what are these people connecting to the internet with abandoned OSes anyway? They don't even get security updates from Microsoft!

But you might be right about me being able to do it alone. Perhaps I should ask for help from the people who have been offering. But so far, it hasn't been that much of a workload. That may of course change with the coming Windows 8 support.

I'm not a programmer, so I can't comment on why work in your branch is or isn't or should or shouldn't be compatible with work in the trunk. The fact remains that the progress in trunk is stalled because of unicode. It's not guaranteed that progress will continue if the stalling factor is removed, but it's pretty certain that progress will not continue as long as it's stalled...

As for the trunk unicodification, if they're redoing your work their way, then wouldn't it be beneficial to all of us if you would help in redoing it? You have the experience, after all. The alternative is either to wait till trunk unicode is finished (unlikely) or to drop trunk development completely.

The latter would be, in my opinion, a bad move. Your work, while cool and very very usable (kudos), is almost source-closing NSIS because it doesn't use the existing NSIS development infrastructure. Fragmentation destroys human & technical resources.

Well, open source also has a way of moving forward when the product gets neglected, stalled, etc. It gets forked and the original dies. I don't know if this is what's happening but with the dearth of activity (releases), it's a possibility.

If you are asking me to redo my work, then no. It could be that my approach of how it should be done is different from what they want to do. And in that case, they have to take the time to actually implement it. If you are asking me to help them do their approach when I believe the approach I took is better, again, no. There is no incompatibility with my approach since I've done it and it works. And I think the fact that it's taking more than two years to do what I've done four years ago should speak for itself.

The official NSIS needs to support other compilers (and POSIX support), not just VS2005+. We cannot rely on the CRT to provide stdio ANSI/UTF8/UTF16 conversion like the fork does...

The CRT does not provide UTF-8, UTF-16, UTF-32, and ANSI conversions. Hence, the NSIS Unicode does not rely on them. What you may be facing is the fact that wchar_t is defined as different widths for the different platforms. The CRT in Linux is going to be 32-bits. The CRT on Windows is going to be 16-bits. If you are trying to do cross-platform building of win32 programs, then you will need to make sure that the CRT that you link to will also be for UTF-16LE. Otherwise, all your wide char string functions will not work right because they want wchar_t to be 32-bits. Or you will have to link in ICU which is a huge library and then convert everything to the "native" wchar_t.

I say forget building NSIS on other platforms. Why bother? NSIS is only a Windows installer solution. Build it on Windows. As for other compilers on Windows, I'm sure with a little work people can get my NSIS Unicode to build on gcc, et al. Is this what you guys are stuck with? I don't have any expertise in cross-platform building and I don't have much interest (no need yet) so I won't be much of a help there.

Originally Posted by jimpark View Post

The CRT does not provide UTF-8, UTF-16, UTF-32, and ANSI conversions. Hence, the NSIS Unicode does not rely on them.

Then why are you passing "ccs=<encoding>" to fopen?

You are right, Anders. I do remember using that MS extension to fopen. It was very convenient at the time. Are you guys adamant about not linking to MS CRT? But you aren't adamant about calling Win32 API, right?

Otherwise, you might have to wrap a new class for Unicode text file and simply pass that object around instead of using CRT IO API on the FILE*. That is a more extensive change. But is that all you are stuck on or are there other issues?

Originally Posted by jimpark View Post

The CRT does not provide UTF-8, UTF-16, UTF-32, and ANSI conversions. Hence, the NSIS Unicode does not rely on them. What you may be facing is the fact that wchar_t is defined as different widths for the different platforms. The CRT in Linux is going to be 32-bits. The CRT on Windows is going to be 16-bits. If you are trying to do cross-platform building of win32 programs, then you will need to make sure that the CRT that you link to will also be for UTF-16LE. Otherwise, all your wide char string functions will not work right because they want wchar_t to be 32-bits. Or you will have to link in ICU which is a huge library and then convert everything to the "native" wchar_t.

Actually on Linux you would not use wchar_t at all, because on Linux all the API functions support Unicode via UTF-8 encoding, so you can use char all the way. On Windows this doesn't work. The ANSI functions in the Win32 API function, that are defined with the char type, do not support UTF-8 (and thus do not handle Unicode strings). And the Unicode-enabled functions in the Win32 API are defined with the wchar_t type and require an UTF-16 encoding. Everything could have been so easy, if M$ had decided to use UTF-8 in their API!

The general solution for cross-platform applications that build on Windows as well as on Linux is using macros. Instead of char or wchar_t, you use a TCHAR macro. This macro will expand to wchar_t on Windows and to char on Linux. Needless to say that all string manipulation functions need to be replaced with macros too, when going this route. It needs a lot of code change.

The other solution is: Use char with UTF-8 all the way - even on Windows. With that solution you can use the normal char-based string functions and don't need any macro "magic". However, on the Windows platform, you will need to convert from UTF-8 to UTF-16 before certain API functions. For example, on Windows, you can't pass the UTF-8 string to fopen(), but have to write your own fopen_utf8() that internally converts to UTF-16 and then calls wfopen(). That is what I did with a lot of code that had been ported from Linux to Windows, but did not support Unicode yet. The Win32 API provides the needed functions to convert between UTF-8 and UTF-16.


This is the code I use to enable Unicode support on Windows in applications that use char all the way*:

(* as is the case with most code ported from Linux or written by people who don't care/know about Unicode support)

All fopen()'s must be replaced by fopen_utf8(). Also the "main" function has to be wrapped like this:


Actually I saw that idea in the LAME code first 🙂

When I was referring to wchar_t width differences, I was referring to string functions found in <wchar.h>. http://pubs.opengroup.org/onlinepubs...h/wchar.h.html

I do think that _wfopen() not existing in Linux is an oversight, but one that can be easily reimplemented. Just convert your wide strings to UTF-8. Converting from UTF-8 to UTF-16 and vice versa is not hard. You can link to ICU, iconv or (if you have access to Windows API) use WideCharToMultiByte / MultiByteToWideChar using CP_UTF8 as your target. Or you could write your own from the Unicode specs which is also not hard -- I've done it before.

But since we are targeting Windows as the working platform, keep all the strings as UTF-16LE. We aren't really porting a Unix program into Windows, it's really the other way around. Also, if we restrict building of NSIS to Windows, then I don't think you have many problems. I think MinGW needs to link to MS CRT (from what I've heard), so you have all of Microsoft's extensions anyway.

I still don't really understand the motivation of trying to build NSIS on Linux. It's an exercise in cross-platform building and it might be interesting to some. But since NSIS itself needs to run on Windows, I don't really see how anyone would benefit from the exercise.

Originally Posted by jimpark View Post

I still don't really understand the motivation of trying to build NSIS on Linux. It's an exercise in cross-platform building and it might be interesting to some. But since NSIS itself needs to run on Windows, I don't really see how anyone would benefit from the exercise.

There are quite a few OpenSource projects that originate from the Linux world and the main developers don't care much about Windows. These people often support Windows ports, as long as it doesn't make too much trouble and as long as it can be done from their usual Linux build environment. So cross-compiling Win32 binaries from Linux is okay, having to set-up and use a Windows machine is not. That's why it may be convenient to have a "native" Linux port of MakeNSIS, even though the final installer will never run under Linux (outside of Wine).

Still I think when compiling the EXE headers on Linux one would have to use a cross-compiler anyway (because you need them to be compiled as Win32 binaries). And that cross-compiler would use 16-Bit wchar_t's, just like any C/C++ compiler on Windows...

Excellent points, Lord Mulder. But is that what we are aiming to provide, an MakeNSIS that runs natively on Linux? I've yet to see it. And if that's what we are aiming to do, I think that's admirable. But if we are just trying to build NSIS (including MakeNSIS) that only runs on Windows but built from Linux, who are we doing this work for? Ourselves, the builders of NSIS? But why? I have a Windows box. I need to actually run NSIS to build my setup and test my Windows products.

I agree that being able to cross-compile Win32 binaries of MakeNSIS from Linux isn't very useful. It certainly would be a nice addition, but IMHO it should be "low priority" feature. But then again you would be using the cross-compiler (not the native Linux compiler) to compile MakeNSIS for Win32 from Linux anyway. And again this eliminates most portability issues, I think. So I guess most problems you will be facing will be "MinGW/GCC -vs- MSVC" quirks rather than "Win32 -vs- Linux" issues...

(If something compiles with the MinGW/MSYS compiler on Win32, it probably compiles with the cross-compiler on Linux too)

i think the main point is that makensis is already provided with the ability to cross-compile, so dropping that probably isn't going to be liked by those who do rely on it. yes it doesn't quite make sense but it could be from part of an automated build system pulling things from a repository and building from there on a linux machine for example. obviously there was a reason why it was added as support so there must have been the demand.

as for the whole this build vs 'official' talk, i was under the impression when wizou was porting things back that it was generally a like for like thing. though that all seems to have stopped / lost any focus which probably isn't helping things. but i can see why people want to maintain some sort of legacy support as tbh, a unicode build doesn't offer that much advantage when you're only providing something for people where ansi will fit - i don't disagree that unicode is useful but there are reasons why some of us do still prefer to use the 'official' builds compared to the unicode ones.

-daz

First of all, if written properly, the very same code can be compiled as ANSI or as Unicode. So if you still want to support an ANSI version while offering a Unicode support, this is doable. It's not hard to do for new code, though it makes some work to "upgrade" existing code. But if you have to rewrite you code for Unicode support anyway, making it support Unicode and ANSI at the same time is minimal extra work.

But why would you want an ANSI version of NSIS nowadays? Every Windows system that is used nowadays does support Unicode. Even good old Windows 2000 (which is not provided with important security updates anymore for more than a year now!) did support Unicode just fine. I know that there are still a few people using Windows 2000, although nobody can seriously recommend this, but even older Windows versions, like 9x and NT 4.0, definitely are a lost case. Not worth bothering. Really! At the same time the lack of Unicode support causes serious problems on non-archaic systems: Can't install to directories that contain Unicode characters (and these may exist on every system!), can't access or create files whose name contains Unicode characters, can't display text that contains characters not available in the user's current ANSI Codepage (a huge limitation for NSIS' otherwise great translation system). And so on...

Why not make the cut:
Go for NSIS 3.xx to be fully Unicode and keep the 2.xx branch for backward-compatibility (for those who really care). It's not unusual at all in the software world to drop support for outdated OS in never versions and maintain the old version for backward-compatibility for a while.


(BTW: If current NSIS already supports cross-compiling, how will adding Unicode support destroy this?)

Originally Posted by jimpark View Post

You are right, Anders. I do remember using that MS extension to fopen. It was very convenient at the time. Are you guys adamant about not linking to MS CRT? But you aren't adamant about calling Win32 API, right?

Well, to maintain POSIX support we try not to use too much Win32 stuff in makensis.exe since it has to be in a #ifdef _WIN32 block with a #else for the POSIX analog.

Using the basic CRT stuff is not a problem (fopen etc), but using compiler extensions can be a problem.

So the only solution I see is some sort of wrapper around FILE* that reads ANSI/UTF8/UTF16 and spits out UTF16 on the other side. (I was hoping to add UTF8 support to the ansi version but I'm not sure if that is going to happen (I already have the code for .nlf loading, the normal source files is the problem))

Now I am confused 😕

Are we still talking about cross-compiling MakeNSIS.exe from a Linux system or are you talking about a native Linux port of MakeNSIS?

Originally Posted by LoRd_MuldeR View Post

Are we still talking about cross-compiling MakeNSIS.exe from a Linux system or are you talking about a native Linux port of MakeNSIS?

What would be the point of cross-compiling makensis and then having to use wine or whatever to run it? Even if you don't have a cross-compiler you can compile makensis on POSIX but you need to grab the stubs and plugins from somewhere else... (See "G.3 Building on POSIX" in the helpfile)

Originally Posted by Anders View Post

What would be the point of cross-compiling makensis and then having to use wine or whatever to run it?

Exactly my point 🙂

And that's why I was a bit confused about jimpark's statement "MakeNSIS that runs natively on Linux? I've yet to see it".

Still it is perfectly possible to write code that supports Unicode and compiles on POSIX (Linux, MacOS X) as well as on Windows.

Two common approaches have been mention here:


(So while retaining cross-platform compatibility makes things a bit more complicated, it shouldn't be a showstopper for Unicode NSIS)

Sorry, I did not mean to confuse but to really show the lack of utility in having MakeNSIS build on Linux. I think the only use is if someone really wanted to build their own toolchain, including NSIS, on Linux as some sort of a build process as DrO mentioned. I still don't know why anyone would do that. And if someone does do that, does he also cross-build GCC or some other compiler for Windows on Linux and then decide to use that on their Windows builds? The fact is, in order to create a Windows installer, they need to run MakeNSIS on Windows.

Why can't we just forget catering to these hypothetical people. It's not worth stalling the project over. They are getting a Windows setup making toolchain that needs to be built on Windows. They will still need to have a Windows box as part of their resources to actually run MakeNSIS and make their installer. And if a few of these hypothetical people exist, do they matter so much to stall the whole development?

Again, I would not be making this argument if MakeNSIS did run natively on Linux, making NSIS a cross-platform setup builder. But that is not the case.

Originally Posted by jimpark View Post

Again, I would not be making this argument if MakeNSIS did run natively on Linux, making NSIS a cross-platform setup builder. But that is not the case.

If I understand Anders correctly, then MakeNSIS already can be build as a native Linux application (only the EXE headers can't, for obvious reason) and so it does run natively on Linux. And that, indeed, makes it possible to build Windows applications on Linux (using the cross-compiler) and then package them in an NSIS installer with MakeNSIS for distribution. No Windows machine required.

But once again: That is not the reason that prevents MakeNSIS from supporting Unicode. It is possible to write code that builds natively on Windows, builds natively on Linux and supports Unicode on both platforms. See the link in my previous post, for example.

Originally Posted by jimpark View Post

And if a few of these hypothetical people exist, do they matter so much to stall the whole development?

It is in the Debian repository IIRC.

Also, we got a patch from the Fedora/RedHat downstream this month so they are not hypothetical.

Even if we dropped POSIX support, VC6, VS2003 and MinGW would still need fixing... (I know you don't care about VC6/VS2k3 but fixing MinGW would probably also fix the older VC's)

Borland C++ and Open Watcom C/C++ don't not have official support and I don't know how long it has been since anyone tried those but they might also have similar issues...

The point is, POSIX is not stalling anything, the usage of MS extensions is the problem.

To help me understand, they can't build exehead on Linux or do they using a cross-platform compiler and somehow with stubbed libs? If they need to build their setups on Linux, can they do it under Wine?

Anders, also not just MS extensions but also Win32 API calls, as well. Has anyone looked into using iconv or ICU for Unicode support? The standard runtime does not provide enough support to do what we want.

Originally Posted by jimpark View Post

If they need to build their setups on Linux, can they do it under Wine?

They could. But that's probably not what they want 😁 🙄

(Also Wine is limited to x86 systems by the way it works. Linux also runs on many other architectures)

Originally Posted by jimpark View Post

Anders, also not just MS extensions but also Win32 API calls, as well. Has anyone looked into using iconv or ICU for Unicode support? The standard runtime does not provide enough support to do what we want.

What exactly is missing?

makensis has been fully portable to POSIX since version 2.01. The stubs are either cross compiled with MinGW or copied from the Windows build. Since I've added this feature, I've seen it ported to numerous platforms, adopted by some Linux distributions, ported to Darwin and pop-up on one of the *BSD trees. I received help requests to get it working on AIX and other non-mainstream platforms. The Debian and Redhat ports in particular saw caring maintenance with patches, fixes and even upstream patches.

I never stopped to ask why. People just wanted it. I don't have statistics, so I can't give out exact numbers. But they exist. I think the strongest selling point is for projects mainly aimed at Linux. By giving them the option to build everything on Linux, we make the Windows porters job easier. They don't have to maintain their own systems to build the application and can streamline the Windows port into the main project build cycle.

In the past, MinGW was also the only decent free solution for building on Windows. Visual Studio Express didn't exist or wasn't good enough.

As for Win32 API being used in makensis, any of them that still exist in the code are implemented for non-Win32 platforms. I think the LZMA compression module still has some threading functions implemented using pthreads.

--

All this is just technical details. Regarding the future, I do agree Unicode is required.

Thanks Kichik for the info. As to Lord Mulder's question, the standard library does not have a way to convert from one form of Unicode encoding to another, not to mention the normalization of Unicode. For example, Windows wants UTF-16 little endian and likes precomposed characters. Linux's wide chars are UTF-32 little endian if running Intel but possibly big endian on other processors. Mac OS X uses UTF-8 fully decomposed characters. So now you see the extent of the problem with supporting Unicode and multiple platforms. If we want to use the wchar library in the C runtime, we have to encode the Unicode strings to what they want to see in that platform. If the strings are going to be stored in a binary format that will be used on Windows, such as the strings in the setup, they need to be encoded to UTF-16LE.

The new C++11 standard provides more Unicode support, including expressing Unicode string literals as 8 bit, 16 bit or 32 bit, but we will still need to write some code for byte ordering. But using the new C++ standars also seems to be out of the question if we need to support legacy compilers.

No wonder the project is stalled. It's very difficult to move forward with so many restrictions. If this is what needs to be done, then keep the strings in the preferred encoding on the platform so that all the wchar string functions available since C++03 standard can be used during the NSI script processing. Then when saving the string, save them as UTF-16LE so that the exehead doesn't have to convert the strings. So string saved to disk is in the native encoding of Windows, but the string in memory is what's expected by the wchar functions.

The other option is to stick to a single encoding even in memory but that would mean not being able to use the wchar functions in the C runtime and hence having to reimplement them on platforms if the chosen Unicode encoding isn't what the platform CRT likes.

(Call me an uninformed smartypants, but I see Jim say that the problem is very complicated, Mulder says that it's already been solved, and Anders states that the real problem lies elsewhere entirley (namely at ANSI .nsi reading). So while POSIX makensis has been cleared up, I'm wondering if there's still some confusion on what exactly needs to be figured out / agreed upon?)

Originally Posted by jimpark View Post

Thanks Kichik for the info. As to Lord Mulder's question, the standard library does not have a way to convert from one form of Unicode encoding to another, not to mention the normalization of Unicode. For example, Windows wants UTF-16 little endian and likes precomposed characters. Linux's wide chars are UTF-32 little endian if running Intel but possibly big endian on other processors. Mac OS X uses UTF-8 fully decomposed characters. So now you see the extent of the problem with supporting Unicode and multiple platforms. If we want to use the wchar library in the C runtime, we have to encode the Unicode strings to what they want to see in that platform. If the strings are going to be stored in a binary format that will be used on Windows, such as the strings in the setup, they need to be encoded to UTF-16LE.

I see that converting between different Unicode encodings might be a problem. On Windows there is a Win32 API function to convert UTF-16 to UTF-8 or some ANSI Codepage (and vice versa). I'm not sure if there is an equivalent on POSIX/Unix systems.

But: I still don't understand why these conversions are needed at all 😕

On Linux (and other POSIX systems too?) all the API/CRT functions use UTF-8. The NSIS source code files (.nsi) should be encoded in UTF-8 too - Unicode text files almost always are UTF-8 encoded. So on these systems we should be able to use the 'char' type with UFT-8 encoding all the way. And, as UTF-8 is a sequence of individual bytes, there should be no "endianess" problems either. No need to use the 'wchar_t' type with UTF-16 or UTF-32. And no conversions needed anywhere.

Now, on Windows, we can either stick with the 'char' type and use UTF-8 or we can use 'wchar_t' with UFT-16. In the former case we need minimal code change compared to the 'POSIX version', but before each Win32 API we need to convert the argument from UTF-8 to UTF-16 (not really a problem, because the Win32 API also has the required function to convert); it basically needs a simple UTF-8 wrapper function around each Win32 API function (at least around those that deal with strings). In the latter case we would have to replace any 'char' or 'wchar_t' with a TCHAR macro, that expands to 'char' on Linux/POSIX and to 'wchar_t' on Windows, in the code. Some string manipulation functions would have to be replaced by macros too. But generally you can simply put the required macros in a single header file (with a big "#ifdef _WIN32 .... #else .... #endif" around to switch between the different OS) once and include it everywhere...

Now remember that makensis also runs on Windows. In fact, most of the time it does. 🙂 And because all Unicode text must go through UTF-16LE conversion to display correctly, we have this problem. Well, maybe this isn't that bad if we say that makensis is UTF-8 and the Windows GUI shell simply converts UTF-8 to UTF-16LE. The only thing about UTF-8 is that it is so variable in length. MS tries to keep one character per 16 bits which is why it prefers completely precomposed characters, i.e. one code point that has both base and diacritic characters. But UTF-8 is very variable. Each character could be one, two, three, or four bytes longs, or longer in the case with diacritics. And then exehead would have to do the same conversion from UTF-8 to UTF-16. I don't like that the exehead would have to do extra work to deal with all the strings. But this is an option that might then push all the conversion stuff out of the POSIX code and into the Windows code. Anders, this is what you were thinking, right? What did you find?

Originally Posted by jimpark View Post

Now remember that makensis also runs on Windows. In fact, most of the time it does. 🙂 And because all Unicode text must go through UTF-16LE conversion to display correctly, we have this problem.

What problem?

On Windows a simple MultiByteToWideChar(CP_UTF8, 0, input, -1, Buffer, BuffSize) will do the conversion.

Originally Posted by jimpark View Post

Well, maybe this isn't that bad if we say that makensis is UTF-8 and the Windows GUI shell simply converts UTF-8 to UTF-16LE.

I think you would do the UTF-8 -> UTF-16 conversion yourself before each Win32 API call. The Shell API doesn't handle UTF-8.

Originally Posted by jimpark View Post

The only thing about UTF-8 is that it is so variable in length. MS tries to keep one character per 16 bits which is why it prefers completely precomposed characters, i.e. one code point that has both base and diacritic characters. But UTF-8 is very variable. Each character could be one, two, three, or four bytes longs, or longer in the case with diacritics.

Yes, in UTF-8 characters have variable length (in bytes). But that's not a problem. Each byte still "looks" like a normal ANSI character and at the end there always is a NULL terminator. So any function that was designed to handle ASCII characters will handle UTF-8 too. All ASCII characters are even identical between ASCII and UTF-8. Only problem I see with UTF-8 is that we need to allocate some "extra" space (if you want to ensure that n characters fit in the buffer, 4*n bytes are required). But on the other hand: If you allocate n bytes buffer, with UTF-16 you can store a maximum of n/2 characters. With UTF-8 you can store up to n characters, maybe fewer...

BTW: UTF-16 also may use pairs of consecutive 16-Bit words ("surrogate pair"), because 2^16 is not enough for all Unicode characters!

Originally Posted by jimpark View Post

And then exehead would have to do the same conversion from UTF-8 to UTF-16. I don't like that the exehead would have to do extra work to deal with all the strings. But this is an option that might then push all the conversion stuff out of the POSIX code and into the Windows code. Anders, this is what you were thinking, right? What did you find?

Now, if we talk about the EXE header again: Here we do not need to care about POSIX at all. It will never run anywhere else but on Windows (Wine). It will be compiled either on Windows or by a cross-compiler. Thus in the EXE header it is safe to write Windows-specific code, i.e. use 'wchar_t' with UTF-16 all the way and call Win32 API functions directly. The only conversions that will be needed is: Strings that have been generated by MakeNSIS and are stored in the installer EXE's "data" section need to be converted from UTF-8 to UTF-16 once. But I doubt that this causes too much performance overhead. And again MultiByteToWideChar() will do the job...