VPatch 3.0 release - NSIS Forums

Koen van de Sande
29th August 2005 12:21 UTC

VPatch 3.0 release
VPatch is a binary patch generator, to be used in conjunction with NSIS for example. After 9 months of development and 8 release candidates (and a few beta's) there is now a stable version: 3.0.

For those of you who haven't used any release candidates:

Date/time is preserved on the file patched
MD5 is now used for checksums instead of CRC
Patch generator completely rewritten (faster and lower memory usage). Now handles 500MB files as well, for example.
Patch generator flags replacement of a patch inside a patch file (same checksum) as an error
Bugfixes in the runtimes, now share a single codebase

Compared to release candidate 8 the GUI has seen a bugfix and received a new icon. The whole package now has an installer again; the last time there was an installer was in 2.0.

You can find VPatch on TibEd.net.

Comm@nder21
29th August 2005 15:49 UTC

great work, man!
thanx in advance!

pabs
30th August 2005 03:00 UTC

Cool! Do the important parts of it compile on Linux/POSIX?

rsegal
30th August 2005 04:39 UTC

Koen you industrious Hollander. Fantastic work!

pabs
30th August 2005 10:11 UTC

posix portability stuff
Genpat3 and Linux/Unix:

* including windows.h
* TRUE/FALSE/BOOL/DWORD/BYTE/etc
* Win32 file functions
* FILETIME
* MAX_PATH
* GetTempFileName
* no makefile/build system
* the following errors occur a lot:

PatchGenerator.cpp:203: error: invalid conversion from â€˜unsigned char*â€™ to â€˜char*â€™
PatchGenerator.cpp:203: error: initializing argument 1 of â€˜std::basic_istream<_CharT, _Traits>& std::basic_istream<_CharT, _Traits>::read(_CharT*, std::streamsize) [with _CharT = char, _Traits = std::char_traits<char>]â€™

All these are probably easily fixed. If you don't mind depending on msvcrt, then the posix equivalents can be used instead otherwise you would have to use some wrapper class or something.

Koen van de Sande
2nd September 2005 12:24 UTC

It is a longterm goal to compile on POSIX. The actual patch generator was written with cross-platform in mind, but things like GetTempFilename... well I just did not know how to do them the POSIX way. The same goes for a function to obtain file size.
The compiler used is Borland C++. Kichik said he'd look into compiling GenPat with MSVC++ which should increase cross-platformness (I hope).
I welcome change suggestions to compile on POSIX, just show me before/after for pieces of code. :)

Koen van de Sande
3rd September 2005 16:26 UTC

I've just redone my entire website. The link has not changed, which I think is pretty neat. Primary difference for VPatch: changelog has its own page now, and there is a Paypal donation button. You never know who might me feeling generous :D

Koen van de Sande
3rd September 2005 17:49 UTC

Whoops. Distribution updated:
- one of the shortcuts to readme didn't work
- Genpat was still branded RC8
- added (as a comment) a way to do GetTempFileName the POSIX way

pabs
4th September 2005 12:16 UTC

In the manual page for tmpnam, it says the following:


BUGS
       Never use this function. Use tmpfile(3) instead.

IIRC, that is because using it causes security issues through the race conditions created by using it. I imagine the same issues occur GetTempFileName as well.

Here is the manual page for tmpfile:

tmpfile man.

I think you should use the way that NSIS works to compile different functions for different platforms. Myself and perhaps kichik can help you create replacement functions if you do not know how to do certain things in POSIX.

Koen van de Sande
7th September 2005 09:46 UTC

So now I know why I only put it in as a comment :D and not as replacement code. My POSIX/cross-platform question:
- how do I get a file date, and because I want to convert it to a FILETIME, how do I multiply two 32-bit integers and get a 64-bit integer as a result?
- what is a 64-bit integer type name?
- IIRC the read function of C++ streams returns char instead of unsigned char. Is it safe to cast it to unsigned char?

For those of you looking for a compiled version of the new NSIS plugin: you can now find it on the VPatch website.

pabs
7th September 2005 15:40 UTC

1.

#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>

int stat(const char *file_name, struct stat *buf);
int lstat(const char *file_name, struct stat *buf);

Pick one of the statstruct members st_atime (last access), st_mtime (last modification), st_ctime (creation time). All of these are of type time_t. On some platforms, this is 64-bit, on some it is 32. See this MSDN url for how to convert to FILETIME:

http://msdn.microsoft.com/library/de..._file_time.asp

As for multiplying 2 32 bit integers and getting a 64 bit one, just cast to 64 bit and multiply normally.

So, the time_t -> FILETIME conversion would look something like this:

void TimetToFileTime( time_t t, LPFILETIME pft )
{
    uint64_t ll = ((uint64_t)t) *10000000L) + 116444736000000000L;
    pft->dwLowDateTime = (DWORD)(ll & 0xFFFFFFFF);
    pft->dwHighDateTime = (DWORD)((ll>>32)&0xFFFFFFFF);
}

2.

#include <stdint.h>
int64_t intwith64ormorebits;
uint64_t unsignedintwith64ormorebits;

3. I'm not hugely familiar with C++, but I think that would be safe. In C, I'd just use (unsigned char)somechar, I think C++ uses reinterpret_cast or something (the C version would probably work though).

Koen van de Sande
7th September 2005 18:39 UTC

Wish I'd known about these integer types before. I've removed the windows.h dependency of VPatch by replacing the getFileTime function with a POSIX version.

The temporary file is a bit of a problem though. Basically I need a C++ ofstream to write out bits and pieces of the new patch. When it's done, it needs to be copied back into the filename specified by the user. I cannot use the user specified file right away, because sometimes I need to 'update' it and then I copy it to the temporary file first (with some changes). Ideas on this?

Koen van de Sande
8th September 2005 00:35 UTC

Here is my go at a POSIX version. Notes:
- includes a MinGW makefile
- will not link with MinGW 3.3.1
- works fine with Borland C++ compiler
- still has Windows platform dependency for temporary files

I don't see why this won't link right now... something with multiple definitions. My MinGW is really old (3.3.1) but I can't upgrade right now. I've included updated code.

pabs
8th September 2005 05:40 UTC

With the attached patch, I can build genpat3posix on linux (I assume on mingw32 too). I haven't tested if it works yet though. It gives the following warnings/errors during building:

pabs@chianamo:~/devel/debian/misc/nsis/genpat3posix$ make
g++ -Wall -c -o Checksums.o Checksums.cpp
g++ -Wall -c -o adler32.o adler32.cpp
g++ -Wall -c -o ChunkedFile.o ChunkedFile.cpp
g++ -Wall -c -o GlobalTypes.o GlobalTypes.cpp
g++ -Wall -c -o FileFormat1.o FileFormat1.cpp
g++ -Wall -c -o main.o main.cpp
PatchGenerator.h: In constructor â€˜PatchGenerator::PatchGenerator(bistream&, TFileOffset, bistream&, TFileOffset, bostream&)â€™:
PatchGenerator.h:60: warning: â€˜PatchGenerator::blockSizeâ€™ will be initialized after
PatchGenerator.h:51: warning: â€˜unsigned char* PatchGenerator::targetCDataâ€™
PatchGenerator.cpp:34: warning: when initialized here
cc -c -o md5.o md5.c
md5.c:56: warning: conflicting types for built-in function â€˜memcpyâ€™
g++ -Wall -c -o POSIXUtil.o POSIXUtil.cpp
g++ -o genpat Checksums.o adler32.o ChunkedFile.o GlobalTypes.o FileFormat1.o main.o md5.o POSIXUtil.o
main.o: In function `getTempFile()':
main.cpp:(.text+0x1c7): warning: the use of `tmpnam' is dangerous, better use `mkstemp'

here is the manpage of mkstemp: http://www.opengroup.org/onlinepubs/...s/mkstemp.html

The mkstemp manpage on linux also mentions that tmpfile should be used instead, but in your case its better to use mkstemp I think.

http://www.opengroup.org/onlinepubs/...s/tmpfile.html

pabs
8th September 2005 14:51 UTC

Did a quick test of my compiled version of genpat3posix, I had to change all the char* exceptions in main.cpp to const char* exceptions. I ran it like so:

genpat oldfile.txt newfile.txt posix.pat

After that I got a problem where patch.good() returns 0 at this statement in main.cpp:

  if(source.good() && target.good() && patch.good()) {

The cause of this is the call to removeExistingPatch in the preceeding loop. As far as I can tell, it enters this if, and returns from it:

TFileOffset removeExistingPatch(bistream& in, TFileOffset inSize,
                                bostream& out, TChecksum* removeCRC, 
                                bool existanceIsError) {
  // MD5 mode
  TFileOffset fileCount = 0x80000000;
  // empty file/does not yet exist
  if(in.bad() || in.eof() || (inSize == 0)) {
    writeDword(out,MAGIC_VPAT);
    writeDword(out,fileCount); // noFiles
    return fileCount;
  }

After that point, patch.good() returns false. Not sure how that breaks things, I don't know enuf C++.

Koen van de Sande
8th September 2005 18:54 UTC

I've been including your patch. Except for the checksums part, which I'm reimplementing using C++ streams. Note that if you break the checksum implementation, you might get these kinds of errors. At least, I'm happy with my test cases for VPatch know, because I was falling into the readsome trap of C++.

Can you maybe remove the manpage for tmpfile from this topic? The layout does not like it very much.

kichik
8th September 2005 19:07 UTC

Originally posted by Koen van de Sande
Can you maybe remove the manpage for tmpfile from this topic? The layout does not like it very much.

All posts edited to fit in one screen.

Koen van de Sande
8th September 2005 19:15 UTC

Thanks Kichik :)

Here's an updated version of the (work-in-progress) GenPat-POSIX. Notes:
- compiles all files on Borland C++, MinGW 3.3.1, GCC 3.3.5pre (SuSE)
- Needs more testcases. Not for production use, since checksum code has been rewritten and not all boundary cases have been checked.
- WILL NOT LINK. Only Borland C++ can link its object files into an executable. MinGW and GCC spit out a huge list of undefined references. I need help with this :confused:
- check out secure temp file creation

I'm quite curious how you got your code to link in the first place, 'cause I can't :D

Koen van de Sande
9th September 2005 00:16 UTC

This link error is freakin' brilliant. :(

Basically, my bifstream/bofstream implicitly construct a
std::char_traits<unsigned char>. And this does not exist for unsigned char, it's actually documented for GCC.
And the same for MSVC7.

This is turning into a huge problem. By design all my functions read in bytes from a stream... and this just works in Borland C++. But apparantly the route I chose with change the input/output stream to use the unsigned char type does not work, in fact. So I should revert back to casting everywhere!?

Well, at least I am not alone in this. Defining your own char_traits is considered bad style... so I guess I'll have to cast. :hang: :eek:

Koen van de Sande
9th September 2005 00:36 UTC

So I changed it all to casting. But still need to clean up code in places (ifstream and bifstream are the same now, but I changed it everywhere before). This one compiles on Linux. And I guess it should work, but I haven't tested if the patches are valid. But it runs at least :D

pabs
10th September 2005 03:45 UTC

Compiled fine here too - no warnings either :) - posix.pat (created from genpat compiled from genpat3posix3.zip) and patch.pat created from oldfile.txt and newfile.txt are exactly the same, apart from 8 bytes at byte 93. I'm guessing this is the timestamp field. Might be a good idea to run it over heaps of different combinations of files, hexdump all the .pat files and diff them just to be sure it works in lots of cases. I haven't tested if the decoder plugin understands posix.pat, but I assume it would.

Anyways, I think that once a new VPatch release is out with these changes included, then the next NSIS version should include the new VPatch. Yay, finally genpat can be included in the debian package :)

kichik
10th September 2005 08:49 UTC

Originally posted by pabs
Anyways, I think that once a new VPatch release is out with these changes included, then the next NSIS version should include the new VPatch.

Yep.

Koen van de Sande
10th September 2005 14:38 UTC

I'm currently spending some time on the test suite. Something came to mind: what if the source and target file for GenPat are the same? Right now it will create a patch which just copies the entire file. Applying the patch will then work just fine with no errors. Is this the right semantic? I think it is, but I thought I'd check :D

Koen van de Sande
10th September 2005 14:50 UTC

Here's an updated version to make it compile with MinGW also. Currently putting it through my test cases.

[edit]
The POSIX version uses a different function to retrieve timestamps from files. And the results are different: one of them does not respect time zones. VPatch 3.0 uses some Windows function, and retrieves: (2002, 2, 2, 4, 58, 22), while the POSIX stat function returns (2002, 2, 2, 5, 58, 22) which is off by one hour. Windows explorer shows 4:58 for the target file. Now what should I do about this :rolleyes:
This regression was caught by my new test suite, by the way :D
[/edit]

Koen van de Sande
11th September 2005 01:07 UTC

Now that I've reverted back to the Windows date/time extraction on the Win32 platform, I've encountered something in my test cases: a different patch. Visual C++ and MinGW generate different patches than the Borland C++ one... though they do work fine and have the exact same size. I feel I need to get to the bottom of this first before putting out a new version. It might just be a subtlety in the way the patch is generated, but better safe than sorry. At least, Visual C++ and MinGW do agree either on the patch :)

[edit]It's a subtle thing indeed: when multiple blocks can be used for a copy from the source file (with the same contents), they pick a different one[/edit].

Koen van de Sande
11th September 2005 12:23 UTC

Here's another new version. This one:
- complies to my tests (though MinGW/VC++6 have a different interpretation of patch making than BC++, they are the same size and give the correct result)
- compiles on Visual C++ 6, MinGW, GCC and Borland C++
- is considered a release candidate. If more people check if it compiles for them, then it's ready. It compiles for me everywhere :D

Note to MinGW people trying to compile: I don't know if __WIN32__ is defined automatically in MinGW. If you want to use the Windows GetTempFilename/GetFileTime functions (which are slightly better than the POSIX ones), then you should verify that it is defined, otherwise it will use POSIX functions.

pabs
13th September 2005 04:38 UTC

Compiles and works on Linux fine. You might consider using - instead of / as a command-line option delimiter on Linux, because on Linux / is part of filenames, and - is the commonly used option delimiter.

kichik
17th September 2005 10:27 UTC

VPatch 3.0 was committed into CVS. Currently, GenPat is not compiled natively but only cross-compiled. I need to make some changes to the build system to make it compile natively.

Thanks!

Koen van de Sande
17th September 2005 18:29 UTC

I've put out the GenPat POSIX version as VPatch 3.1. Compared to the version in CVS (which is based on the genpat3posix5.zip) the notable changes (files attached):
- main.cpp was changed to use "-" as a switch option character on non-Win32 instead of "/"
- Readme.html was changed to include changelog of 3.1 (POSIX compatibility)

Remarks on the current CVS version:
- VAppend was in with 2.1, only it's now been moved to a "VAppend" folder for 3.0, and it's no longer in CVS. However, this might not be such a bad thing because the NSIS version does not need it, it's only for all non-NSIS runtimes. So I think it is best to leave this out. :)
- VPatch_tests.py runs several functionality tests on GenPat (Windows only), perhaps this can be included. One of the tests will not work because of missing files (only I have the datafiles on my PC), but all others are self-contained and quite useful. I'll leave the decision on this up to Kichik :D
- VPatchLib.nsh makes more sense in the NSIS include folder (?) Or can it be included from where it is now as well?

You can find the ZIP/Installer distro of 3.1 on TibEd.net: http://www.tibed.net/vpatch.

kichik
17th September 2005 18:39 UTC

Thanks, uploaded.

The tests script fails without VAppend so I didn't include it yet.

The header file is located in the Contrib directory in CVS, but it's installed into the Include directory.

pabs
18th September 2005 09:19 UTC

Not sure if this is related to the conversion to native rather than cross-compiled, but I get this failure with g++ 4.0:

g++ -o build/release/VPatch/Source/GenPat/GenPat -s -Wl,--file-alignment,512 -Wl,-Map,build/release/VPatch/Source/GenPat/GenPat.map build/release/VPatch/Source/GenPat/adler32.o build/release/VPatch/Source/GenPat/Checksums.o build/release/VPatch/Source/GenPat/ChunkedFile.o build/release/VPatch/Source/GenPat/FileFormat1.o build/release/VPatch/Source/GenPat/GlobalTypes.o build/release/VPatch/Source/GenPat/main.o build/release/VPatch/Source/GenPat/md5.o build/release/VPatch/Source/GenPat/PatchGenerator.o build/release/VPatch/Source/GenPat/POSIXUtil.o
/usr/bin/ld: unrecognized option '--file-alignment'
/usr/bin/ld: use the --help option for usage information
collect2: ld returned 1 exit status
scons: *** [build/release/VPatch/Source/GenPat/GenPat] Error 1

Koen van de Sande
18th September 2005 09:41 UTC

Ah yes, the tests are useless without VAppend. So they do not need to be included with NSIS.

The GCC 4 error looks very much like an option to the compiler is not recognized. This might be a SCons thing.

pabs
18th September 2005 18:57 UTC

If I ignore that error using SKIPUTILS, I get some more build errors on the plugin and elsewhere, which can be fixed by the attached patch against CVS.

kichik
20th September 2005 18:41 UTC

Thanks, all fixed.

Koen van de Sande
26th September 2005 20:21 UTC

These changes will be added to the 'full' VPatch as well, but it will have to wait a while 'cause it's not that an important an update for the default distribution. It will be bundled as soon as there is also something else new that warrants a new version. :)

T.Slappy
28th June 2006 18:34 UTC

Vpatch use problem
Hi
I would like to use Vpatch for installing my mod.
Mod consist from 6 files (aprox 30 MB each)
This files should be inserted to archives (each file to separate archive, apr 600 MB), but I do not want the user to insert each file manually

I want to patch file with vpatch plugin.
I tried it in this way:

!insertmacro VPatchFile "m12_main.pat" "$INSTDIR\M12_main.zip"
(I think it means: patch "$INSTDIR\M12_main.zip" with file "m12_main.pat")

(before patching I unpacked m12_main.pat file to instdir)

The problem is, that I want the smallest installer and I can not extract those 600 MB archives to patch them (readme example says that I have to extract new file)

Out-dated game archives are present at target system and I want to patch them with patch files generated with genpat, so the installer size should be aprx 6x30MB (uncompressed)
Is there any way to do it?
thx

Joost Verburg
28th June 2006 18:50 UTC

Zip files have no solid compression, so you can patch them directly. You have to create a new zip file and create a patch from the old to the new archive.

T.Slappy
28th June 2006 19:26 UTC

vpatch
I used this script:
vpatch::vpatchfile $PLUGINSDIR\patch.pat" "$INSTDIR\M12_main.zip" "$INSTDIR\M12_main_new.zip"

(from vpach example)

M12_main.zip is present on target system, M12_main_new.zip no (I used it to generate patch with genpat.exe), so I got this error::
Result: Unable to open source file

So what to do?
(I do not want to extract whole M12_main.zip to target system, because it is 600MB big)

JasonFriday13
29th June 2006 00:11 UTC

You forgot a quote mark:

vpatch::vpatchfile "$PLUGINSDIR\patch.pat" "$INSTDIR\M12_main.zip" "$INSTDIR\M12_main_new.zip"

T.Slappy
30th June 2006 16:51 UTC

vpatch
JasonFriday13: Thanks, but it did not help

I always have error: cannot open output file
I tried
vpatch::vpatchfile "$PLUGINSDIR\patch.pat" "$INSTDIR\M12_main.zip" "$INSTDIR\M12_main.zip",

vpatch::vpatchfile "$INSTDIR\patch.pat" "$INSTDIR\M12_main.zip" "$INSTDIR\M12_main.zip",

vpatch::vpatchfile "$INSTDIR\patch.pat" "$INSTDIR\M12_main.zip"

too, but still the same error

I used the example to make it, but there are 3 files (new txt, old txt and patch) and I have only two (old and patch)
and it still does not work....