Hello1024
8th January 2007 23:32 UTC

Large installers truncate contents
Hi,

I posted this a while ago at the bottom of a similar thread, but since I got no reply I guess it was the wrong place to post.

Has anyone had any success with large files. I'm trying to use the LZMA compressor to compress a 10 GB file, but I know most of the file is empty space and it will compress down to 550-600MB.

Problem is whatever compression I use, it seems to stop after 705MB, as if it had reached the end of the file. No error is given. The file on disk isn't corrupted, because I tried an alternative file with the same results. It isn't a file system limitation, because I'm using an NTFS drive. It isn't a 32 bit limitation, because I'm nowhere near the 2/4GB limits of 32 bit systems (at least for the compressed size). I don't think it's a memory limitation, because I have easily enough virtual memory to store the entire file and more for compression space.

I can't use what others have suggested in this forum of splitting the installer and the data, because both need to be combined in one file for easier distribution.

At the moment, I make do with the following:
the installer extracts a .bz2 version of the data file, then it extrracts bunzip2.exe, a windows "unzipper". It then execwaits bunzip2 to do the decompression of the file.
The problem with this method is that it takes too much time, because as far as I can see, the compressed data is first extracted to a temp folder, then copied to the output folder. Then bz2 copies that to memory, and then decompresses it to the output folder. That process takes about 30 mins on my test machine, whereas simply allocating a 10GB file on the disk and writing 600MB or so of data to the begining of it should take more like 1 min.

Any ideas of a way I can solve the problem?

Thanks

n_baua
9th January 2007 06:01 UTC

Hi Hello1024,
I also had a simillar problem when I used the makensisw.exe to compile one of my installers which ran in to 2GB(Video and MP3 files no empty space), the compiler stopped ib between. I solved it by compiling the script file through command line and it worked out for me.

Give it a try, I also do not think it is the problem with VM or System.

Best of luck.

Hello1024
9th January 2007 18:01 UTC

Thanks for the advice - I'll have a go.

Hello1024
9th January 2007 20:38 UTC

Just tried that, and it didn't work. I've attached the entire log, but this is the line thats causing trouble:

File: "ubuntu.img"->"c:\ubuntu\ubuntu.img" [compress] 53135547/705032704 bytes

The file ubuntu.img is actually 5,000,000,000 bytes exactly. (~4.65GB)

Do you think it could be anything todo with the fact I'm compiling on a network drive? If it is, I MIGHT be able to move some hard disks around and get it compiling on the local computer.

n_baua
10th January 2007 07:30 UTC

Hi,

I never came against the single large file(4+GB) as you are trying to install, I guess this must produce the virtual memory problem and I also fear the network compilation doubt you have is right. just check it out.

I am not sure why you are using the single img file installation can you just split it up and merge it later(This would be easy with just one marco or batch statement) when setup is done with entire copying.

:confused:

Hello1024
10th January 2007 18:11 UTC

Yes, I can split and merge it, but I need to figure a way so that the extract and merge can be done in a single step, because I don't want it to be extracted in pieces, and then merged to a single file for two reasons - 1. it would take more time - the data is written to the disk, read from the disk, merged, and written to the disk. and 2. that would require my end user to have 20GB of disk space (10GB for the files to be merged, and 10GB for the resultant merged file before the files to be merged are deleted)

I'll see if I can have a go with the file on a ocal disk drive and see if that solves the problem.