Wizou
8th July 2010 00:53 UTC

[Plugin] WPatch : Optimized incremental patch system
  I just released WPatch for wide public release, after having been extensively tested in production environment.
Web site: http://wiz0u.free.fr/prog/WPatch/

Presentation

WPatch is an optimized incremental patch system for NSIS (Nullsoft Install System).
This patch system consists of 3 tools:

• WGenPat: Compares 2 versions of the same file and creates a chunk of binary data describing the changes needed to transform the "old" version into the "new" one
• WGenPatDir: Compares 2 directories (using WGenPat eventually) and creates a NSIS script that can patch the first directory to transform it into the second directory
• WPatch: NSIS plugin that use this binary data to convert effectively an "old" version of the file into the "new" version of the file.

Original Features (and improvements over the VPatch system included in NSIS)

• In-place patching:
  WPatch is will patch files "in-place" in one pass. It does not create a temporary copy of the files to patch, and modifies the original file directly. This means the final user don't need extra free disk space, only the final file size is required.
• Fast and Precise mode:
  When enabled, for each files to patch, instead of scanning the whole patch database for the file signature, it will locate immediately a specific patch information (by offset). File signature is still verified to prevent patching a wrong version of the file. This mode also allows to patch two identical source files into different target files.
• Support for huge files:
  By default, a MD5 hash of the whole file is calculated as the file signature to verify that the file is really in the patch database. This can lead to a significant slowdown for huge files. For those, you can choose to only use the first & last 64K of the file as the signature. In this case, make sure that the beginning or end of the file contains data that are unique/specific for each patch (typically, I recommend placing a timestamp updated for each version of the huge file)

WPatch has been widely tested and performance improved in a real production environment.

Using the WPatch system is as simple as calling: (for example)

WGenPatDir.exe --precise --exclude *.tmp;.svn dir1 dir2

to establish the difference between dir1 and dir2 (your "old" and "new" version of your program's directory), and, in your NSIS installer script:

Section

    InitPluginsDir

    ;...

    ; $INSTDIR points to the directory dir1 that is going to be transformed into dir2

SectionEnd

>!include WGenPatDir.nsh

Section

    IfErrors 0+2

        MessageBox MB_OK "There has been some errors !"

    ;...

>SectionEnd 

>

MSG
8th July 2010 05:49 UTC

Great work, again! I'll try this out for our patching installer project, if we ever get around to making an update. :-)

zeeh3
8th July 2010 11:33 UTC

Cool plugin, thanks for sharing :)

820815
15th July 2010 10:17 UTC

Originally posted by Wizou
Support for huge files

4GB limit? :(

CrushBug
15th July 2010 20:47 UTC

It said huge, not gianormous. =P

Wizou
16th July 2010 02:49 UTC

Originally posted by 820815
4GB limit? :(

mmmh i'm not sure.. maybe it supports patching files that are up to 4 GB, you will have to test... (and tell me ^^)
I'm lazy about reviewing my code for this limit.. however it has been tested perfectly with 1.85 GB files..

Of course, we are talking about a *patching system* here, so this assumes the 4 GB file to be patched is already installed on the target system... (which can be difficult to achieve with NSIS in the first place)

820815
18th July 2010 10:51 UTC

I've tried before to ask, on 4.34GB file

[ChunkedFile] Filesize of 0 gives 0 chunks.
[ChunkedFile] Memory to be used by those chunks: 0 bytes...

Wizou
18th July 2010 11:26 UTC

Oh, not over 4GB, that's for sure.. (I didn't use int64)
But it might be possible the system supports files between 2GB & 4GB (unsigned int32)

brim4brim
18th July 2010 13:18 UTC

Hi,

This seems like the closest thing to what I'm looking to achieve. We are looking at moving NSIS installer at the moment in the company I work for and for one of our projects, we need to compare two xml files for user settings and if a line is missing from one then add it to the other file.

I'm not sure if its possible to do this with your plugin? Essentially they should have the same tags as the core files but if the XML values in the files are different, I don't want to overwrite them, I just want to add any new tags I'm adding to the software in the new release.

I hope that makes sense and that you can let me know if your plugin is suitable for this purpose.

Thanks for your help.

MSG
18th July 2010 14:58 UTC

Originally posted by brim4brim
I'm not sure if its possible to do this with your plugin? Essentially they should have the same tags as the core files but if the XML values in the files are different, I don't want to overwrite them, I just want to add any new tags I'm adding to the software in the new release.

No, this cannot be done with a binary differ, as they are designed to encompass all changes at once. What you want to do can be done entirely in native NSIS code, using FileRead, StrCpy and StrCmp (or LogicLib). Something like:

FileRead file1_line
FileRead file2_line
(some macro here to trim strings down to only the tag - maybe look for = character, or whatever)
if file1_tag != file2_tag
copy file1_tag to file2
(maybe add some default value to file2)
endif
loop.

brim4brim
18th July 2010 17:17 UTC

ah ok cool thank you for pointing me in right direction.

Wizou
18th July 2010 23:54 UTC

You seem to be looking for a 3-way merge = Take the difference between 2 files and incorporate those differences into a 3rd file, even if it contains other changes.

There are a few tools that can perform this... Typically diff3 from Unix...
For Windows, you may use something like http://kdiff3.sourceforge.net/

Or maybe, first establish a "text diff" file, and apply it to patch the 3rd file. For example, use http://gnuwin32.sourceforge.net/packages/patch.htm

brim4brim
19th July 2010 00:44 UTC

Even better from its description, it is exactly what I'm looking for :D

It is for 100% windows use, webservice running in IIS with web/desktop front end in C# and separate Windows Mobile application too.

Thanks a million :)

Wizou
19th July 2010 01:48 UTC

Be careful, those 3-way merge have their limits, they cannot always work if they can't find where to insert the difference (if the files have changed too much around the lines to modify)..
Maybe, another solution for you could be to merge the differences more manually, by manipulating directly the XML data in the target file, for example using the nsisXML plug-in.