[Plugin] WPatch : Optimized incremental patch system

I just released WPatch for wide public release, after having been extensively tested in production environment.
Web site: http://wiz0u.free.fr/prog/WPatch/

Presentation

WPatch is an optimized incremental patch system for NSIS (Nullsoft Install System).
This patch system consists of 3 tools:

• WGenPat: Compares 2 versions of the same file and creates a chunk of binary data describing the changes needed to transform the "old" version into the "new" one
• WGenPatDir: Compares 2 directories (using WGenPat eventually) and creates a NSIS script that can patch the first directory to transform it into the second directory
• WPatch: NSIS plugin that use this binary data to convert effectively an "old" version of the file into the "new" version of the file.

Original Features (and improvements over the VPatch system included in NSIS)

• In-place patching:
  WPatch is will patch files "in-place" in one pass. It does not create a temporary copy of the files to patch, and modifies the original file directly. This means the final user don't need extra free disk space, only the final file size is required.
• Fast and Precise mode:
  When enabled, for each files to patch, instead of scanning the whole patch database for the file signature, it will locate immediately a specific patch information (by offset). File signature is still verified to prevent patching a wrong version of the file. This mode also allows to patch two identical source files into different target files.
• Support for huge files:
  By default, a MD5 hash of the whole file is calculated as the file signature to verify that the file is really in the patch database. This can lead to a significant slowdown for huge files. For those, you can choose to only use the first & last 64K of the file as the signature. In this case, make sure that the beginning or end of the file contains data that are unique/specific for each patch (typically, I recommend placing a timestamp updated for each version of the huge file)

WPatch has been widely tested and performance improved in a real production environment.

Using the WPatch system is as simple as calling: (for example)

WGenPatDir.exe --precise --exclude *.tmp;.svn dir1 dir2

to establish the difference between dir1 and dir2 (your "old" and "new" version of your program's directory), and, in your NSIS installer script:

Section
    InitPluginsDir
    ;...
    ; $INSTDIR points to the directory dir1 that is going to be transformed into dir2
SectionEnd
!include WGenPatDir.nsh
Section
    IfErrors 0 +2
        MessageBox MB_OK "There has been some errors !"
    ;...
SectionEnd 

Great work, again! I'll try this out for our patching installer project, if we ever get around to making an update. :-)

Cool plugin, thanks for sharing 🙂

Originally Posted by Wizou View Post

Support for huge files

4GB limit? 🙁

It said huge, not gianormous. =P

Originally Posted by 820815 View Post

4GB limit? 🙁

mmmh i'm not sure.. maybe it supports patching files that are up to 4 GB, you will have to test... (and tell me ^^)
I'm lazy about reviewing my code for this limit.. however it has been tested perfectly with 1.85 GB files..

Of course, we are talking about a *patching system* here, so this assumes the 4 GB file to be patched is already installed on the target system... (which can be difficult to achieve with NSIS in the first place)

I've tried before to ask, on 4.34GB file

[ChunkedFile] Filesize of 0 gives 0 chunks.

[ChunkedFile] Memory to be used by those chunks: 0 bytes...

Oh, not over 4GB, that's for sure.. (I didn't use int64)
But it might be possible the system supports files between 2GB & 4GB (unsigned int32)

Hi,

This seems like the closest thing to what I'm looking to achieve. We are looking at moving NSIS installer at the moment in the company I work for and for one of our projects, we need to compare two xml files for user settings and if a line is missing from one then add it to the other file.

I'm not sure if its possible to do this with your plugin? Essentially they should have the same tags as the core files but if the XML values in the files are different, I don't want to overwrite them, I just want to add any new tags I'm adding to the software in the new release.

I hope that makes sense and that you can let me know if your plugin is suitable for this purpose.

Thanks for your help.

Originally Posted by brim4brim View Post

I'm not sure if its possible to do this with your plugin? Essentially they should have the same tags as the core files but if the XML values in the files are different, I don't want to overwrite them, I just want to add any new tags I'm adding to the software in the new release.

No, this cannot be done with a binary differ, as they are designed to encompass all changes at once. What you want to do can be done entirely in native NSIS code, using FileRead, StrCpy and StrCmp (or LogicLib). Something like:

FileRead file1_line
FileRead file2_line
(some macro here to trim strings down to only the tag - maybe look for = character, or whatever)
if file1_tag != file2_tag
copy file1_tag to file2
(maybe add some default value to file2)
endif
loop.

ah ok cool thank you for pointing me in right direction.

You seem to be looking for a 3-way merge = Take the difference between 2 files and incorporate those differences into a 3rd file, even if it contains other changes.

There are a few tools that can perform this... Typically diff3 from Unix...
For Windows, you may use something like http://kdiff3.sourceforge.net/

Or maybe, first establish a "text diff" file, and apply it to patch the 3rd file. For example, use http://gnuwin32.sourceforge.net/packages/patch.htm

Even better from its description, it is exactly what I'm looking for 😁

It is for 100% windows use, webservice running in IIS with web/desktop front end in C# and separate Windows Mobile application too.

Thanks a million 🙂

Be careful, those 3-way merge have their limits, they cannot always work if they can't find where to insert the difference (if the files have changed too much around the lines to modify)..
Maybe, another solution for you could be to merge the differences more manually, by manipulating directly the XML data in the target file, for example using the nsisXML plug-in.

I tried to use WGenPatDir, calling it like this:

WPatch/WGenPatDir.exe Project Project_0.2.0
[…Lots of files it successfully creates patches for.., until…]
WGenPat v4.00
[Source] Project\UDKGame\CookedPC\Textures.tfc
[Target] Project_0.2.0\UDKGame\CookedPC\Textures.tfc
[PatchFile] WGenPatDir.pat
Source and target file have equal CRCs!WGENPAT.EXE /H "Project\UDKGame\CookedPC\Textures.tfc" "Project_0.2.0\UDKGame\CookedPC\Textures.tfc" WGenPatDir.pat
returned an error 2


According to diff the files actually DO differ:

$ diff -r -q Project Project_0.2.0/ | grep -i textur
Files Project/UDKGame/CookedPC/Textures.tfc and Project_0.2.0/UDKGame/CookedPC/Textures.tfc differ


Any ideas?

This question was already discussed there :