Unicode

That's excellent news! Is there an ETA on when this will be complete?

No ETA yet.. I just started the SVN branch I will work on..
I guess the first merge/port will be easy especially as your work will greatly simplify the effort.
But then there will be a phase of discussion/arbitration/tweaking/testing as to what should be the user & developer experience regarding this new version.
I will keep you updated on my progress.

well, if we don't look for "functionW" before "function", we are going to need some kind of define that is "A" or "W" and maybe a list of API's where it is required. IMHO that is the wrong approach, and the reverse is needed, they need to look for the char type specific version first unless a (new) option is provided in the system call

my opinion is to check whether the parameter type contains a "t", then we know we are facing an API which has two variant..
and only in this case, we start checking for 'functionW' or 'functionA' variant first.. if entrypoint is not found, then we check for 'function'

@Wizou: the problem with that check is that if the user calls CreateFile() with 'w' in the Unicode version of NSIS, that's not wrong either. CreateFile should still resolve to CreateFileW.

Like I said, the only problems I've seen are the string functions that mimic standard library calls. There aren't that many and they are all lowercased like C standard library functions. If the person has programmed C at all, functions like strlen should jump out as taking char* and not wchar_t*. So it should raise a warning flag and the user should realize he should put a W at the end.

Windows programmers don't think about that, I'm so used to just calling lstr*, I don't think about the char type. Whats wrong by defaulting to the W versions and only falling back if W does not exists or a option to disable it was set?

Do you have a single example of a function that needs to be called without W when running as unicode (meaning foo should be called with WCHAR* even if fooW exists)?

Originally posted by jimpark
@Wizou: the problem with that check is that if the user calls CreateFile() with 'w' in the Unicode version of NSIS, that's not wrong either. CreateFile should still resolve to CreateFileW.

I don't agree with you. This is wrong.
If the user explicitely wants to call the Unicode version using 'w' as parameter type, he has to specify CreateFileW as the function name.

Summary:
* the user wants to call explicitely CreateFileA => he uses function name CreateFileA and parameter type 'm'
* the user wants to call explicitely CreateFileW => he uses function name CreateFileW and parameter type 'w'
* the user wants to call CreateFile whatever suits best according to NSIS variant => he uses function name CreateFile and parameter type 't'
* outside the presence of parameter type t, we can't suppose the existence of a A/W suffix, and it seems to me it would be wrong to try to add one (it would risk to reach another API which is unrelated)

Yeah, I agree with Wizou

That would be nice if everyone did that. But remember, when people read the Windows documentation, what do they read? The documentation says to use CreateFile(). It does not say to use CreateFileW() for Unicode. We're just lucky that Microsoft has been largely consistent with these W and A suffixes.

If you write C/C++, you will also be calling CreateFile() with wchar_t*. Because that's what the documentation tells you to do. You just have to make sure you define some macro like _UNICODE or UNICODE. If you read the fine print you get the "is implemented as CreateFileW() and CreateFileA()" but no example tells you to use these directly.

And remember, there's really no bearing on type-safety here. I can use any old function signature and as long as the name of the function matches, the System plugin will push your arguments, whatever types they may be, onto the stack and happily call the function. The arguments can be totally wrong types and have bogus values. So it's not even useful in that respects either.

To the best of my knowledge, there IS no CreateFile-function, only CreateFileA and CreateFileW. CreateFile is #defined to CreateFileA or CreateFileW in WinBase.h, depending on whether or not UNICODE is defined.

As such, if the Sytem plug-in allows you to just write CreateFile, then it must either already be checking for *A, or it has some predefined mappings.

Originally posted by Anders
Windows programmers don't think about that, I'm so used to just calling lstr*, I don't think about the char type. Whats wrong by defaulting to the W versions and only falling back if W does not exists or a option to disable it was set?

Do you have a single example of a function that needs to be called without W when running as unicode (meaning foo should be called with WCHAR* even if fooW exists)?

I believe there are calls in the Windows API where there is no ANSI counterparts so the "W" suffix version would fail. But I don't know of a case where the "W" suffix version would also be a different function. That doesn't mean that they don't exist or will never exist.

However, remember, this plugin architecture can call other libraries than those provided by Microsoft. And who knows what other people do.

Basically, as I mentioned before, it is wrong to second guess the user from the onset. Don't you hate it when your word processor "corrects" your typing when you type some acronym thinking it's SMARTER than you? I don't believe programs should behave that way. And I certainly don't want a programming tool to behave that way.

User: "I want to call Foo()."

Computer: "No, you don't. You want Bar(). Bar is nicer."

User: "No, I want Foo()!"

Computer: "Hey, I found Bar()! Look at that. It's a function that exists. Let's just call Bar()."

User: "I said, call Foo()!!!"

Computer: "Silly user. I called Bar() for you which is much nicer and that caused a booboo. You better go fix it."

What a nightmare!

@Pidgeot: Yes, you are right. There is no CreateFile() function in the library you link to. There's a macro. But the documentation tells you to call CreateFile().

Originally posted by jimpark
That would be nice if everyone did that. But remember, when people read the Windows documentation, what do they read? The documentation says to use CreateFile(). It does not say to use CreateFileW() for Unicode. We're just lucky that Microsoft has been largely consistent with these W and A suffixes.

If you write C/C++, you will also be calling CreateFile() with wchar_t*. Because that's what the documentation tells you to do. You just have to make sure you define some macro like _UNICODE or UNICODE. If you read the fine print you get the "is implemented as CreateFileW() and CreateFileA()" but no example tells you to use these directly.

You will not be using wchar_t*, you will be using LPTSTR. t is the system plugins version of LPTSTR. The minute you start using w / wchar*, you already know you are working with unicode only. In 99.99% of cases, if there is a t, you need to look for the W/A versions first. This is the most compatible approach (And the only way to achieve A/U compatibility with existing code)

@Anders: What you say is true. But if you are coding a Unicode only app, you will notice that people use wchar_t with CreateFile() and not CreateFileW().

Either way, I don't like the system to think it's smarter than the programmer.

but we sort of have to be, in windows/C world, this stuff happens at compile time, for us it happens at runtime, and so, t is our clue that the user trusts us and expects us to the right thing

(OT: Why don't you two hang out in IRC, this discussion would go a lot faster that way)

Unfortunately, my access to the internet is very limited. So I can't IRC. This is also what keeps me from being able to work on the NSIS repository directly (and hence that's why I haven't created a branch in the repository for my stuff here). So once the group actually takes my code into the repository, I can't work on it anymore. Anyway, that's my situation and will be for the foreseeable future.

Well, there are HTTP IRC "clients" that work just fine, try mibbit

You are both right...
The user doesn't want to care or know about CreateFileA or W, he will just use CreateFile... and parameter type 't', and there will be no problem.

I think the few users that care to write their own System::Call (rather than copy/pasting them from somewhere) should know about TCHAR/LPTSTR and A/W variant.

About recompiling plug-ins ?

Hi jimpark,

Lately, I've been studying some C, and got the hang of recompiling plug-ins. I've done 2 so far, and almost/basically got the registry plug-in done.
Using an NSIS .nsh script instead of the plug-ins just ended up being to slow

My question is about including <nsis/pluginapi.h>, if I just use this, the script is missing the pop- & push string/integer functions.

As I read topics about this issue in this forum, one stated I should include pluginapi.c, another stated I should use the pluginapi.lib.

Myself, I only got the scripts recompiled including <nsis/pluginapi.c>. Which includes pluginapi.h, anyway. And as I understand including Exdll.h, isn't necessary anymore?

Is this the way you advise me to compile Unicode NSIS plug-ins ???

Or should I implement pluginapi.lib some how ?
I only found this file in the installed version of Unicode NSIS, in the Examples\Plugins

I am using Microsoft Visual Studio 2008 express edition.

See my forum topic at : portableapps.com/node/21879
EnumINI.7z, is using the pluginapi.c
NewTextReplace, still uses it's own pop & push functions, but maybe I should change that one time.

It's very admirable of you to take the initiative. I'd advise that you download the source code for the Unicode NSIS and look at the Contrib folder. All of these are plugins. For example, a simple one to imitate is the Banner plugin and see what I did there to support Unicode.

Done all of that a while ago. I needed the examples.

As well I got pluginapi.lib linked by VC++ now. So it's all fine ! But including pluginapi.c worked as well.

Finally I got an reply of the developer of the MD5dll plug-in, he will do this plugin soon !

@Jim Park
Are you gonna compile NSIS 2.46 based Unicode build?

@All
Could someone with skills compile this ANSI NSIS plugins to UNICODE version?
1. CrcCheck.dll

2. Linker.dll


-Pawel

GJ with new version jimpark!

However I still get messages about BOMs with /V0 and /V1 switches.

Details here: http://forums.winamp.com/showthread....08#post2497708
Mr. kichik opinion: http://forums.winamp.com/showthread....55#post2498455

Hope it will be included some day 🙂

nsisunz+CallAnsiPlugin

@Anders:
I use your plugin for unzip file to folder, but it consumes 2 gb memory, without result.
Can you show me how to unzip file?

NSIS Unicode with &quot;m&quot; and &quot;t&quot; System types

Hi Jim, Anders and others,

I have a question about NSIS Unicode (I'm using v2.45.1):

I have a DLL that I need to call with System::Call and some of the function prototypes in that DLL have parameters that are explicitly "char*" because the DLL expects some strings in UTF-8 format and also sometime return strings in UTF-8 format.

From what I understand I have to use the "m" type in these System:Call prototypes to explicitly specify char* but what happens when I actually call them and pass a NSIS variable or register?

From my testing it just works (incidentally this is probably because the strings I pass in contain only english characters) but consider if I did the following:

StrCpy $0 "some chinese characters"
System::Call 'SomeDLL:SomeFunction(m)i(.r0).r1? c'

Does the Unicode string contained in $0 get converted in any way before the DLL function is called or is it just called with a typecast to (char *) with whatever data is in $0?

Since my calls are working, I suspect it's being converted to multibyte, but how and when ?

I'm asking because I'm having trouble with this when doing it the other way around: i.e. the DLL returns some UTF-8 to me that I need to display in NSIS.

For example if there is a function in that DLL defined as:

int SomeDLLGetMessage(char *buffer);

Which expects you to pass a pre-allocated char* buffer of at least 1024 bytes in order to place a utf8 string in it...

...and if I call this function this way:

System::Call 'SomeDLL::SomeDLLGetMessage(m)i(.r0).r1? c'

I find that it completes and that $0 contains something that I am able to display with a simple MessageBox MB_OK "$0".

Since the DLL call returns UTF-8 I would have expected $0 to contain single-byte data and therefore to be mangled when I attempt to display it. However it displays (the utf8 chars are displayed "as is" but it displays nonetheless).

Is there some kind of conversion happening for me inside NSIS ? What is the proper way of doing this?

I'd be more than happy to convert the UTF-8 data i get back to Unicode myself by calling MultiByteToWideChar but it doesn't seem to work. When I do this, I am able to convert $0 to Unicode but when I try to display it, NSIS only displays the first character (as if it was treated like a single byte string and stopped at the first 0 byte)...

I guess I am confused at how this needs to be handled. Is there a way to call a DLL, retrieve a UTF8 string and convert it to Unicode to display it ? or do I need to have the DLL modified to return Unicode data instead ?

Sorry for the long post :-(

Cheers,
Damien.

yes, m will convert the buffer. In your case, you should allocate 1024 bytes with System::Alloc and pass it to SomeDLLGetMessage as i, not m. When the call returns, you can pass it to MultiByteToWideChar yourself (Utf8 is only supported on XP+ in MultiByteToWideChar)

There is an implicit conversion happening with 'm'. In fact, it uses the system default codepage for the conversion. If the return value is also 'm', (ANSI codepage) it will convert it back to utf16.

Are you sure the API sets the buffer with a UTF8 string? Often times people mix up UTF8 and ANSI. UTF8 is a byte-based encoding of Unicode. UTF8, UTF16-LE, UTF16-BE, UTF32 are all Unicode but different encodings of it. ANSI is a different beast altogether requiring the codepage information to really figure out which 'character'* each of the codes represent. This means that 'm' does not do UTF8 to UTF16 conversion at all. This would ONLY work for English since the ASCII set is shared by all ANSI codepages and UTF8. So make sure that the API really does set the string with UTF8 encoding. (This may be unlikely since it requires work to actually get a UTF8 string in Windows.) But if it really is UTF8, what Anders suggests might be the way to do it. Otherwise, if it's ANSI, then 'm' is fine as long as you are okay with the assumption that the string returned is going to be in the system default codepage -- if it's not then it won't display correctly on the system anyway, so it's really not a big concern.

* not necessarily character but really 'code' since diacritics and other character modifying codes exist

Yes the DLL really does return UTF-8 but thanks to you guys I got it working !

I was breaking down the process in two functions, one to call the DLL and one to convert to Unicode and in the process the results were transiting into NSIS variables and therefore being converted all around. Now that I do both the DLL call and MultiByteToWideChar in the same block of code using allocated buffers and the 'i' type, everything is pretty :-)

Thank you guys you saved my friday ;-)

Cheers,
Damien.

Hi Jim, there seems to be an issue with the myatoi and myatou in the pluginapi library (and possibly the myatoi_or) too. If the plugin is built as Unicode then they all work fine but they fail to parse a valid integer for multi-byte builds. Sometimes they return 0 or they return some hideously large number (like an unsigned has been negatived). Same goes for popint().

I will probably look into fixing it myself but I'd rather you had a look. If you want me to test, let me know.

Stu

Hi Jim, there seems to be an issue with the myatoi and myatou in the pluginapi library (and possibly the myatoi_or) too. If the plugin is built as Unicode then they all work fine but they fail to parse a valid integer for multi-byte builds. Sometimes they return 0 or they return some hideously large number (like an unsigned has been negatived). Same goes for popint().

I will probably look into fixing it myself but I'd rather you had a look. If you want me to test, let me know.

Stu