Skip to content
⌘ NSIS Forum Archive

Unicode

573 posts

jimpark#
Maybe "problem" was a strong word.

Correct me if I'm wrong but here's what I've gathered so far:
  1. makensis should only call standard library (no win32 calls if possible).
  2. if win32/posix is needed, then we will need to do #ifdef for win32 and posix.
  3. makensis should be able to read various Unicode encoded files. NSI, NLF, NSH etc. These files may be UTF-8, UTF-16LE/BE, UTF-32LE/BE and we should ideally support them all.
  4. makensisw (Windows GUI) should be able to read outputs from makensis and display the output text to the user.
  5. exehead will only use win32 and will not link to any external libraries including C runtime to reduce size.
  6. plugins will need to handle Unicode strings.


I think point 3 is why I think we should consider linking to iconv or ICU. Point 4, I think we can deal with by using MultiByteToWideChar since we should only ever see UTF-8 coming out of makensis. And once we have iconv or ICU, I think we should convert the strings to UTF-16LE when storing them in the data section so that exehead will see UTF-16LE and not have to convert. This also means that for point 6, the plugin writers will only have to deal with the native Windows Unicode encoding.

Also, as a side note, I think we should throw out the possibility of building ANSI while doing this work. There really isn't a good argument for keeping ANSI support and it's a simplification that can help everyone conceptually get their heads around the problems.
LoRd_MuldeR#
To #2
If not done already, it may be wise to refactor OS-specific stuff into some "OS Support" class/file (i.e. have a "os_support.h" with the function declarations and have several "os_support_win32.cpp", "os_support_linux.cpp", etc. files with their OS-specific implementations), rather than have a lot of #ifdef's all over the place...

To #3
Why is that? Why not say that all text files (NSI, NLF, NSH) have to be UTF-8 and that's it? If required, any encoding can be converted to UTF-8 easily with a command-line utility, like iconv. Anyway, I can't remember to see any "plain text" Unicode files with a different encoding than UTF-8 in the wild. Is this really used? And do we really need to support it ???

To #4
Does makensisw need to compile natively under POSIX too? I don't think so. People that use Linux & Co generally know how to use a console! Especially because NSIS is targeted at developers! Anyway, the one and only way, that I am aware of, to properly output Unicode characters to the Windows console is outputting UTF-8 to the STDOUT. However you must also call SetConsoleOutputCP() and enable UTF-8. output. There is no UTF-16 mode for SetConsoleOutputCP()! Last but not least, you must not use printf() and friends, because they screw up UTF-8 strings with their "translation" functions. Use WriteFile() instead and write to the handle you got via GetStandardHandle(). This of course means that the GUI applications, that redirects the console application's output, has to treat the text as UTF-8 too.

(BTW: You may think that you can write UTF-16 strings to the Windows console by using wprintf(), but that's not the case. It internally converts to the ANSI Codepage and replaces all characters by '?' that are not available in the current ANSI Codepage *d'oh*)
jimpark#
Yes, we need #3 because UTF-16LE is the default encoding for anything Unicode on Windows. When you opened up a resource file in Developer Studio, you will find that it is encoded in UTF-16LE by default. If you save your text file as Unicode in Windows notepad, it is UTF-16LE by default.

Also, we may need to access data from other Unicode resources like OpenType fonts. By default, for the Windows platform, the encoding of strings in the names section is UTF-16BE. (In fact, everything in an OpenType font file is big endian.) So if we need to get some data from other sources, likely, we will need to convert.
LoRd_MuldeR#
Originally Posted by jimpark View Post
Yes, we need #3 because UTF-16LE is the default encoding for anything Unicode on Windows. When you opened up a resource file in Developer Studio, you will find that it is encoded in UTF-16LE by default. If you save your text file as Unicode in Windows notepad, it is UTF-16LE by default.
We should think about this. Any halfway decent Text/Code editor can save as UTF-8. Even the Windows Notepad can 😉

And, as said before, it is easy to convert between the different encodings before handing the file to MakeNSIS.

Also, if you don't restrict NSI/NLF/NSH files to a specific encoding, how do you detect a file's actual encoding at runtime ???

(I know that you may be able to guess the encoding via BOM character, but the BOM may be missing)


Originally Posted by jimpark View Post
Also, we may need to access data from other Unicode resources like OpenType fonts. By default, for the Windows platform, the encoding of strings in the names section is UTF-16BE. (In fact, everything in an OpenType font file is big endian.) So if we need to get some data from other sources, likely, we will need to convert.
That could be an issue 🙁

BTW: Why does MakeNSIS have to deal with OpenType font files?

[EDIT]

After a quick Google search I found this:
jimpark#
Frankly, NSIS doesn't really need to handle OpenType files. But it would be nice to. I've gotten requests to get version information and font name information so that a font can be updated for install. It's a perfectly valid thing to install fonts. And so I've provided those capabilities in the last release of Unicode NSIS.

And all the existing users of Unicode NSIS probably have their NSI files as UTF-16LE.

Anyway, I have no qualms about linking to new libraries. I don't want to write Unicode conversion code myself, especially if I have to support Unicode normalization. If makensis runs on Mac OS X, for example, it decomposes all the Unicode strings so even if you convert the strings to UTF-16LE, the decomposed character may not render correctly on Windows and will probably fail string comparisons with those entered by the user since Windows prefers composed characters. So normalization is also an issue. This is a consequence of Unicode support plus multi-platform support.
LoRd_MuldeR#
Back to another topic for a moment:

Originally Posted by jimpark View Post
I just released a beta version of 2.46.3. I've still compiled it using MSVC 2010 and I did not need the EncoderPointer.lib since I do not link to the standard library in exehead. So NSIS itself still requires Windows XP+ to build installers but the installer it generates should be able to run under Windows 2000. Lord Mulder, can you check this when you have the time? I've also added GetFontName and GetFontNameLocal to round out the font commands.
Can you please also re-compile the plug-ins? I noticed the nsExec plug-in fails on Win2k.

By looking at the file dates, it seems the plug-ins included in v2.46.3 are the same as in v2.46.2.

I reverted to the 'nsExec.dll' from your v2.46.1 release and the problem is gone...

[EDIT]

Okay, it seems that 'nsExec.dll' creates a temporary executable that does the actual job.

While I don't understand why that is done, it explains the problem:

The temporary EXE won't run on Windows 2000 because its OperatingSystemVersion field is 5.1.

(...because it has been compiled by the VisualStudio 2010 compiler)
jimpark#
Thanks for the report, Lord Mulder. I thought I build everything with the correct SUBSYSTEM setting. I will check on that again. And I will clean out the objs. But you will have to wait until Monday for the new build.
jimpark#
I've updated the build and uploaded 2.46.3 Beta 2. Please let me know if you experience any problems.
Anders#
The next version of VS will drop XP support as well: http://connect.microsoft.com/VisualS...details/690617
LoRd_MuldeR#edited
Originally Posted by jimpark View Post
I've updated the build and uploaded 2.46.3 Beta 2. Please let me know if you experience any problems.
Thanks! I will give it a try, as soon as I have some spare time...

[EDIT] Just quick note: The 'uninst' stub still has a file date from 2002 [/EDIT]

Originally Posted by Anders View Post
The next version of VS will drop XP support as well: http://connect.microsoft.com/VisualS...details/690617
🧟

Too bad. This would make the next VS useless for most developers for a long time. While the market share of Win7 is growing, XP still has around 40% (and Vista never got any noteworthy market share). Unless, of course, there will be a workaround to fix XP-compatibility.
jimpark#
That's too bad. Visual Studio 2011 is supposed to support std::atomics and a lot of the std threading library. I was looking forward to that. No XP support would effectively kill it for us also. We need to support WinXP. Time to write to Microsoft.
mrjohn#
Originally Posted by LoRd_MuldeR View Post
Unicode Setup from this location is detected as Adware : 🙁

http://www.virustotal.com/file-scan/report.html?id=82b3056fbbcf76cc6e177a22f48d0f48aa46e769039495ca2651ac29aa5e8c0b-1317715970
mrjohn#
This is Avira response :

Dear Sir or Madam,thank you for your email to Avira's virus lab.
Tracking number: INC00846451.

We received the following archive files:
File ID Filename Size (Byte) Result
26326890 suspect_FALSE.zip 1.69 MB OK

A listing of files contained inside archives alongside their results can be found below:
File ID Filename Size (Byte) Result
26326891 nsis-2.46.3-Unico...up.exe 1.71 MB MALWARE


Please find a detailed report concerning each individual sample below:
Filename Result
nsis-2.46.3-Unico...up.exe MALWARE

The file 'nsis-2.46.3-Unicode-setup.exe' has been determined to be 'MALWARE'.Our analysts named the threat ADWARE/Adware.Gen.This file is detected by a special detection routine from the engine module.
Please note that Avira's proactive heuristic detection module AHeAD detected this threat up front without the latest VDF update as: ADWARE/Adware.Gen.
jimpark#
Originally Posted by mrjohn View Post
I've checked the link and saw that they had listed nsis-2.46.3-Unicode-setup.zip which is not a name of any file I've uploaded. The MD5 digest it has listed do not match any file I've uploaded either. So I can only conclude that whatever file they've tested is not mine.
LoRd_MuldeR#
I'd write a mail to virus_malware@avira.com in order to clearify that.

In my experience they are quite responsive...
Zinthose#
Can we get a direct link? I can't find the download link on the site.
Можем ли мы получить прямую ссылку? Я не могу найти ссылку на скачивание на сайте.
Yathosho#
Originally Posted by Zinthose View Post
Can we get a direct link? I can't find the download link on the site.
Можем ли мы получить прямую ссылку? Я не могу найти ссылку на скачивание на сайте.
Zinthose#
Originally Posted by Yathosho View Post
.... OPPS.... 😐

I meant to post this in another topic... CURSE you multi-tabbed browsing!!
Yathosho#
it seems obvious, but is that ANSI build fully compatible with the official nsis? just asking, cause i'd prefer to install it over my current installation and not in a seperate folder.
fhkd#
GetVersion.exe is not a valid Win32 application.

Hello,

I compiled following Code with Unicode NSIS 2.46.3:



!define File "program.exe"

OutFile "GetVersion.exe"



Function .onInit

 ## Get file version

 GetDllVersion "${File}" $R0 $R1

  IntOp $R2 $R0 / 0x00010000

  IntOp $R3 $R0 & 0x0000FFFF

  IntOp $R4 $R1 / 0x00010000

  IntOp $R5 $R1 & 0x0000FFFF

 StrCpy $R1 "$R2.$R3.$R4.$R5"



 ## Write it to a !define for use in main script

 FileOpen $R0 "DefineValues.txt" w

 FileWrite $R0 '!define PRODUCT_VERSION "$R1" $\n'

 FileClose $R0



 Abort

FunctionEnd



Section

SectionEnd
I get a GetVersion.exe, but if I try to start it from the Windows Explorer, I get the message

GetVersion.exe is not a valid Win32 application.
If I use NSIS 2.46, it works fine and I get DefinesValues.txt with the entry !define PRODUCT_VERSION ....

I know the post
Originally Posted by vcoder View Post
This script work well on ANSI version of NSIS and failed on Unicode version: ...
I use Windows 7 64 bit, and other more complex Setup-Skript get compiled well with Unicode NSIS.
LoRd_MuldeR#edited
I saw this once. Try to include at least one FILE in your installer. Can be any non-empty dummy file.

@jimpark: Any ideas?
fhkd#
Originally Posted by LoRd_MuldeR View Post
I saw this once. Try to include at least one FILE in your installer. Can be any non-empty dummy file.

@jimpark: Any ideas?
Thanks for the answer.

I tried first following solution:



Section

!tempfile DUMMYFILE

!appendfile "${DUMMYFILE}" "${DUMMYFILE}"

File "${DUMMYFILE}"

!delfile "${DUMMYFILE}"

!undef DUMMYFILE

SectionEnd
It doesn't help, also the dummyfile was packed.

But after I tried



Section

File "program.exe"

SectionEnd
it works, also it is not nice.

Thanks for the help!
fhkd#
Hello again!

While I tried to implement the solution with File "program.exe", I noticed the command !define /product_version in the NSIS User Manual.

Solutions like
Originally Posted by vcoder View Post
OutFile "GetVersion.exe"
with GetDLLVersion or GetDLLVersionLocal are for getting the version from a file on the building machine as a compiler constant during compile time, I think.

With the command !define /product_version it's quite easier.

So I write a little NSIS header GetVersionLocal.nsh with the macros GetFileVersionLocal and GetProductVersionLocal.
Now I can get a constant with the version number without makeing a dummy setup.



!insertmacro GetProductVersionLocal "$%windir%\system32\kernel32.dll" version

!echo "${version_0}.${version_1}.${version_2}.${version_3}"

!echo "${version}"
For a description see the header file.
Yathosho#
when trying to compile a script on windows 2003 server, i get this error:

"The procedure entry point EncodePointer could not be located in the dynamic link library KERNEL32.dll"

some seconds later a second message pops up:

"Unable to initialize MakeNSIS. Please verify that makensis.exe is in the same directory as makensisw.exe"

(it is in the same directory)
Afrow UK#
SP1 installed?
Minimum supported server
Windows Server 2008, Windows Server 2003 with SP1
Stu
LoRd_MuldeR#
Originally Posted by Yathosho View Post
when trying to compile a script on windows 2003 server, i get this error:

"The procedure entry point EncodePointer could not be located in the dynamic link library KERNEL32.dll"

some seconds later a second message pops up:

"Unable to initialize MakeNSIS. Please verify that makensis.exe is in the same directory as makensisw.exe"

(it is in the same directory)
Please see my post here:
Are you a developer who uses NSIS to distribute your application? Are you a Winamp plug-in developer who wants to use NSIS to distribute your plug-in? Have suggestions for other people like you? This is the place.


In short: Binaries compiled with VS2010 don't run on systems prior to WinXP with SP-2, unless countermeasures are taken.
(Probably not a big deal for MakeNSIS, which runs on developer machine only, but important for the EXE stubs)
Yathosho#
Originally Posted by Afrow UK View Post
Minimum supported server
Windows Server 2008, Windows Server 2003 with SP1
i wonder where you even found that, such things should be mentioned on the website. anyway, i'm only using win 2003 because i have no legit copy of windows xp. so there should be no troubles when using xp (sp3)?