99999999
18th August 2007 05:17 UTC
Pre Compression of files
Okay I have a system, which builds installers for around 2000 targets. The targets are independent installers, and each of them has different files, and they range from the typical of 20mb to 1.5gb.
The problem is that 99.9% of the build process, is putting the installer together. What I would prefer is if it would be possible to build a .zip file or use makensis, to compress the files ahead of time, so that I could include a zip file for these file each time, and only those files which changed each time, would be required to be recompressed.
The issue is that while I don't need to build these all everyday, I do need to build them in batches from time to time. As a result, it can take a couple days to complete all the builds on 4 super speedy boxes.
And I need to do the batched builds, quite a few times, as it usually includes testing of changes to the installer, and the installer builds.
So something of this nature would reduce the build time significantly, and would make my job much easier. Is there any method of doing this currently with NSIS?
michaelcsikos
18th August 2007 08:25 UTC
I don't know of a way of doing what you describe, but if you are mainly rebuilding the installer for testing only, just don't bother compressing (SetCompress off).
Otherwise, if the files for a particular installer have not changed, and the NSIS script has not changed since the last build, the installer does not need to be rebuilt. Perhaps you could write a clever script or program to detect this and skip those builds (if installer exe's modified date is later than the NSIS script and every file to be included). Obviously, this would require some work to begin with, but it could potentially save you many hours.
Are all the files local to the build machines?
99999999
19th August 2007 05:55 UTC
Originally posted by michaelcsikos
I don't know of a way of doing what you describe, but if you are mainly rebuilding the installer for testing only, just don't bother compressing (SetCompress off).
...
Are all the files local to the build machines?
Yeah I've tried it with compress off, however this still doesn't help a great deal, because so many of the targets have huge numbers of files, (80mb of 4k files sometimes, textures and such,) so really problem is reading, adding each of those each time.
Most of the time, the testing involved is not the build system, but the installers being built, and of course each one of them can be somewhat unique. Each one has a hooks file, which is tied into the main installer script, and allows for customization of the scripts.
The files are local to the build machine, after being checked out.
The ideal system, would include a method of pre-compression of these files, and then checking in that single pre-compressed block (and it would speed up the checkin/checkout,) then the tool we use to put targets into the system, would be responsible for doing the compression once.
This would also reduce threats of virus infecting those files.
michaelcsikos
20th August 2007 01:26 UTC
Have you tried pre-compressing using zip2exe in the NSIS\Bin directory? You could compress your textures and then uncompress them to the $INSTDIR silently in your main script.
99999999
20th August 2007 03:29 UTC
Originally posted by michaelcsikos
Have you tried pre-compressing using zip2exe in the NSIS\Bin directory? You could compress your textures and then uncompress them to the $INSTDIR silently in your main script.
No I hadn't thought of doing it that way, primarily because from what I could tell we would have to extrapolate the entire zip file container to disk, and then decompress the items from it. I'll look though and see if that is the way it works.