Big problem with big files...

More interpretation

I've had more time to look at the logs

What I think this may be showing is that ZLIB uses the available physical memory and then automatically begins using available virtual memory. That is proven when you note that the ZLIB CSVs show available physical memory dropping to zero, hanging in the zero range for several minutes, while the processor interrupts and memory swapping takes care of memory's job. Which is why ZLIB compression works.

However, I note that the processor interrupts begin immediatly before the memory reaches 32MB (I can't prove that 32 is the point, but I think I can prove that the swapping begins before we run out of memory) anyway during BZ2 compression, I don't note any appreciable change in processor interrupts or swap file usage prior to running out of physical memory.

Now, with that said, I'm pretty sure that an application this "simple" (no disrespect intended) does not diffrentiate between physical memory and virtual memory. It would take a savy programmer to force the program to look only at certain ranges of memory. I haven't yet begun to look through the source myself (I want to duplicate and understand the problem first) so I can't state definitively (sp?) that "A genius programmer has added code to NSIS's available compression routines to cause only physical memory to be addressed." Realizing that I am a novice programmer, the person responsible for the addition of BZ2 should not take offense to my wording "genius" above for indubitably it requires more skill to differentiate (in code) between virtual and physical memory than to simply use virtual memory.

Therefore if BZ2 is loosly based on the Wheeler block sorting process, it might be that process itself which is responsible using only available physical memory.

Sincerely,
Jacob Metro

Just so you'll know I am not ignoring this, I am reading this, and I will have a look in all of your supplied details when I get enough free time for it.

Thanks

OK, done in latest CVS version, FINALLY! I have improved NSIS's file mapping so it won't map the entire file into memory (including the installer data block which is saved in a memory mapped file too). I have successfully compiled 3 installers of random movie trailers, 1GB total. Aside from my computer begging for mercy everything went fine 😉

23 minutes with no compression, 1 hour and 4 minutes with bzip2 compression and 28 minutes with zlib compression.

All installers installed all of the trailers with no problems and with the new extraction progress I could even see that it's not stuck 🙂

Operation choke computer completed successfully 😁

Almost forgot... Use FileBufSize to let NSIS feast on your 20TB 5GHz QUAD DDR-III 😉

I think that this improvement brings NSIS again a little bit closer to the top of most used IDK's.

Good work! 👍

Awesome

This is awesome. Thanks for the patience and hard work.

I still get this error when using XP on NTFS while compiling more than 2GB of many small files. I'm trying to use LZMA compression because I believe it will be able to compress the whole installer down below 700MB but I guess because it needs to create one large temp file before LZMA compression, and it is failing at the ~2GB limit of the temp file.

I get:

Internal compiler error #12345: error mapping file (2103166549, 30836736) is out of range.

Any way to increase this or work around it?

Also, I see in the source that ints were used for the file mapping and buffers instead of unsigned ints. Maybe someone 'in the know' could change these to unsigned and increase the abilities to at least 4GB.

Large file handling has been improved, but 2GB is still the current limit.

You'd better use different archives instead of one 2GB executable.

That's really a downer. With the LZMA support, it would compress down to fit on one CD. Now, with external archives, the whole thing looks very kludgy.

I've tried to edit the source myself but I don't have as good of an understanding of the whole project. Changing the offset in the GrowBuf and MMap objects would obviously need to be done. Other than that, just the external code that uses those objects have to understand unsigned offsets instead of signed ones. Why have signed offsets in a buffer or file anyway?

I hope this is strongly considered in a near future release.

Try the version without solid compression (NSIS 2.0 RC2):

I just saw that right before I replied. I'm going to give it a try shortly. Thanks.

Doesn't cut it. I need the solid compression to get the 2.2GB down to about 700MB.

So non-solid gives no error but solid does? Maybe the system for solid compression can be changed so it will be able to compress more than 2GB of data. Having a compressed installer larger than 2GB won't be useful because most Windows file systems have a 2GB file size limit.

I'm building it now but its taking quite a long time. I'm assuming that it will give no error since I can do this same process with zlib, but it won't generate a setup.exe that will fit on a CD. (Zlib ends up around 1GB IIRC) I know by using 7-zip manually that the 2.2GB of files will compress down to <700MB, so I know this would work if the datablock could grow beyond 2GB and then get compressed solidly via LZMA. I'm just at the mercy of the developers and if they feel this is an important enough feature to change/implement. I'm hoping they do.

LZMA non-solid is definately not the same as ZLIB. It will probably be a little larger than LZMA solid, but usually it won't make a huge difference.

I'll know in a couple of hours. Besides this 2GB issue, I think the rest of it is fantastic. Along with the HM NIS Editor, it is very nice. I'm trying to think of all the things I need to make installers for so I can use this more. Problem is, I have many other installs that also require 2GB of data files. 🙁

Here's the final report. It didn't quite make it.

EXE header size: 49664 / 34816 bytes
Install code: 19821 / 86427 bytes
Install data: 736102408 / -1999567682 bytes
CRC (0xA0163DCE): 4 / 4 bytes

Total size: 736171897 / -1999446435 bytes (-36.-8%)

Well, coverted to megabytes it's 702 MB.

Most 80 minute CD's have a capacity of about 703 MB, so it will probably work.

You can also increase the dictionary size a bit to make it fit.

Perhaps the install data size variable shouldn't be signed to avoid negative numbers?

Yes, it should be unsigned.

ok, i'm getting this error:

Processed 1 file, writing output:
Internal compiler error #12345: deflateInit() failed(-1).

Note: you may have one or two (large) stale temporary file(s)
left in your temporary directory (Generally this only happens on Windows 9x).

i'm compiling few files (about 20), each one below 1 kb (only dummyfiles for testing), and some gfx. the whole installer is about 500 kb, with lzma.
the installer compiles correctly, while SetCompressorDictSize isn't used. everytime i use this setting, the error occurs (with any dictsize).
the nsis-script is running on a FAT32 partition with 3gb free space.
the partition with windows-xp has about 5gb free disk space, and the partition expecially for temporary files (including virtualram/swap) has another 3gb space, about 2gb free.
i'm using latest cvs-version of nsis.
why does this error occur at this configuration?

I can't reproduce it, please attach a script including the files. Btw, this is not related to this topic.

Did you recompile makensis.exe? Note that the LZMA source code has been updated today (it's smaller and faster) and needs some testing.

When compiling installers to release we always recommend using a stable version, however we really appreciate it that you help testing the development version 🙂

i'm always using the latest cvs, because, when u look at the changelog, it seems to be more stable (and more up to date - look at the language files) than the "stable".

also, this problem exists with some versions before.

and, yes, i think this is related to this topic, because i found this topic (and read all posts) when i searched for error #12345.

i don't know how to recompile makensis.exe, sorry.

hmm, i'll see, if i can attach an example.

You must have used a dictionary size which your computer can not handle. Remeber that SetCompressorDictSize takes the number of mega bytes to use and not the number of bytes.

All errors related to memory or failed compression are marked as error #12345.

Because you did not recompile makensis it is indeed not related to the new CVS source. I missed that line about SetCompressorDictSize.

oh, damn, it's in mbyte?
i thought, that was kbyte. kbyte would make more sense, i think.
but, ok, then this is the misstake, i entered 4096 as dict-size 🙂

edit:
so, does this command support floating numbers?

edit2:
worked with 'SetCompressorDictSize 8' 🙂

8 is the default. It doesn't support floating point numbers. I see no real need for that. In 7-zip all of the options are in mega bytes too...

Hi all,

my problem is this:
i've got about 1.5 GByte of files to install (multiple files; biggest not larger than 200 MB). I can only compile using zlib compression. When i try to use bzip2 or lzma I get this error:

Using lzma (compress whole) compression.

EXE header size: 49152 / 35328 bytes
Install code: (38548 bytes)
Install data: (1444031845 bytes)
Uninstall code+data: (61041 bytes)
Compressed data: Error: deflateToFile: deflate()=-3
Error - aborting creation process

I tried it with bigger dictsizes, but this didn't work. These results were with a dictsize of 48 MB (was desperate 🙂.

ps I use Windows2000 SP4 and NTFS drive with about 10 GB free.

I used the latest NSIS build (non-CVS)

A larger dictionary size makes compression more complicated, a smaller size makes it faster and reduces the memory requirements.

You can also download a special build from the website that uses non-solid compression.

Thanks non-solid and smaller dictionary size did the trick...

Sometimes 🙁 Most of the time I get the same error with some of the files: deflate()=-3. The weird thing is if try compressing these files alone in NSIS I don't get an error message. Weirder still is that it's not always the same files. Sometimes the compiler stops on a file and sometimes not.
I'm not considering doing these files separate, I want one big file. Please help.

ps: I also tried compressing them outside of nsis with 7-zip and no problem