VPatch 3.0 release

VPatch is a binary patch generator, to be used in conjunction with NSIS for example. After 9 months of development and 8 release candidates (and a few beta's) there is now a stable version: 3.0.

For those of you who haven't used any release candidates:

• Date/time is preserved on the file patched
• MD5 is now used for checksums instead of CRC
• Patch generator completely rewritten (faster and lower memory usage). Now handles 500MB files as well, for example.
• Patch generator flags replacement of a patch inside a patch file (same checksum) as an error
• Bugfixes in the runtimes, now share a single codebase

Compared to release candidate 8 the GUI has seen a bugfix and received a new icon. The whole package now has an installer again; the last time there was an installer was in 2.0.

You can find VPatch on TibEd.net.

great work, man!
thanx in advance!

Cool! Do the important parts of it compile on Linux/POSIX?

Koen you industrious Hollander. Fantastic work!

posix portability stuff

Genpat3 and Linux/Unix:

* including windows.h
* TRUE/FALSE/BOOL/DWORD/BYTE/etc
* Win32 file functions
* FILETIME
* MAX_PATH
* GetTempFileName
* no makefile/build system
* the following errors occur a lot:

PatchGenerator.cpp:203: error: invalid conversion from ‘unsigned char*’ to ‘char*’
PatchGenerator.cpp:203: error: initializing argument 1 of ‘std::basic_istream<_CharT, _Traits>& std::basic_istream<_CharT, _Traits>::read(_CharT*, std::streamsize) [with _CharT = char, _Traits = std::char_traits<char>]’

All these are probably easily fixed. If you don't mind depending on msvcrt, then the posix equivalents can be used instead otherwise you would have to use some wrapper class or something.

It is a longterm goal to compile on POSIX. The actual patch generator was written with cross-platform in mind, but things like GetTempFilename... well I just did not know how to do them the POSIX way. The same goes for a function to obtain file size.
The compiler used is Borland C++. Kichik said he'd look into compiling GenPat with MSVC++ which should increase cross-platformness (I hope).
I welcome change suggestions to compile on POSIX, just show me before/after for pieces of code. 🙂

I've just redone my entire website. The link has not changed, which I think is pretty neat. Primary difference for VPatch: changelog has its own page now, and there is a Paypal donation button. You never know who might me feeling generous 😁

Whoops. Distribution updated:
- one of the shortcuts to readme didn't work
- Genpat was still branded RC8
- added (as a comment) a way to do GetTempFileName the POSIX way

In the manual page for tmpnam, it says the following:

BUGS

       Never use this function. Use tmpfile(3) instead.

IIRC, that is because using it causes security issues through the race conditions created by using it. I imagine the same issues occur GetTempFileName as well.

Here is the manual page for tmpfile:

tmpfile man.

I think you should use the way that NSIS works to compile different functions for different platforms. Myself and perhaps kichik can help you create replacement functions if you do not know how to do certain things in POSIX.

So now I know why I only put it in as a comment 😁 and not as replacement code. My POSIX/cross-platform question:
- how do I get a file date, and because I want to convert it to a FILETIME, how do I multiply two 32-bit integers and get a 64-bit integer as a result?
- what is a 64-bit integer type name?
- IIRC the read function of C++ streams returns char instead of unsigned char. Is it safe to cast it to unsigned char?

For those of you looking for a compiled version of the new NSIS plugin: you can now find it on the VPatch website.

1.

#include <sys/types.h>

#include <sys/stat.h>

#include <unistd.h>



int stat(const char *file_name, struct stat *buf);

int lstat(const char *file_name, struct stat *buf);

Pick one of the statstruct members st_atime (last access), st_mtime (last modification), st_ctime (creation time). All of these are of type time_t. On some platforms, this is 64-bit, on some it is 32. See this MSDN url for how to convert to FILETIME:



As for multiplying 2 32 bit integers and getting a 64 bit one, just cast to 64 bit and multiply normally.

So, the time_t -> FILETIME conversion would look something like this:

void TimetToFileTime( time_t t, LPFILETIME pft )

{

    uint64_t ll = ((uint64_t)t) *10000000L) + 116444736000000000L;

    pft->dwLowDateTime = (DWORD)(ll & 0xFFFFFFFF);

    pft->dwHighDateTime = (DWORD)((ll>>32)&0xFFFFFFFF);

}

2.

#include <stdint.h>

int64_t intwith64ormorebits;

uint64_t unsignedintwith64ormorebits;

3. I'm not hugely familiar with C++, but I think that would be safe. In C, I'd just use (unsigned char)somechar, I think C++ uses reinterpret_cast or something (the C version would probably work though).

Wish I'd known about these integer types before. I've removed the windows.h dependency of VPatch by replacing the getFileTime function with a POSIX version.

The temporary file is a bit of a problem though. Basically I need a C++ ofstream to write out bits and pieces of the new patch. When it's done, it needs to be copied back into the filename specified by the user. I cannot use the user specified file right away, because sometimes I need to 'update' it and then I copy it to the temporary file first (with some changes). Ideas on this?

Here is my go at a POSIX version. Notes:
- includes a MinGW makefile
- will not link with MinGW 3.3.1
- works fine with Borland C++ compiler
- still has Windows platform dependency for temporary files

I don't see why this won't link right now... something with multiple definitions. My MinGW is really old (3.3.1) but I can't upgrade right now. I've included updated code.

With the attached patch, I can build genpat3posix on linux (I assume on mingw32 too). I haven't tested if it works yet though. It gives the following warnings/errors during building:

pabs@chianamo:~/devel/debian/misc/nsis/genpat3posix$ make
g++ -Wall -c -o Checksums.o Checksums.cpp
g++ -Wall -c -o adler32.o adler32.cpp
g++ -Wall -c -o ChunkedFile.o ChunkedFile.cpp
g++ -Wall -c -o GlobalTypes.o GlobalTypes.cpp
g++ -Wall -c -o FileFormat1.o FileFormat1.cpp
g++ -Wall -c -o main.o main.cpp
PatchGenerator.h: In constructor ‘PatchGenerator::PatchGenerator(bistream&, TFileOffset, bistream&, TFileOffset, bostream&)’:
PatchGenerator.h:60: warning: ‘PatchGenerator::blockSize’ will be initialized after
PatchGenerator.h:51: warning: ‘unsigned char* PatchGenerator::targetCData’
PatchGenerator.cpp:34: warning: when initialized here
cc -c -o md5.o md5.c
md5.c:56: warning: conflicting types for built-in function ‘memcpy’
g++ -Wall -c -o POSIXUtil.o POSIXUtil.cpp
g++ -o genpat Checksums.o adler32.o ChunkedFile.o GlobalTypes.o FileFormat1.o main.o md5.o POSIXUtil.o
main.o: In function `getTempFile()':
main.cpp🙁.text+0x1c7): warning: the use of `tmpnam' is dangerous, better use `mkstemp'

here is the manpage of mkstemp: http://www.opengroup.org/onlinepubs/...s/mkstemp.html

The mkstemp manpage on linux also mentions that tmpfile should be used instead, but in your case its better to use mkstemp I think.

Did a quick test of my compiled version of genpat3posix, I had to change all the char* exceptions in main.cpp to const char* exceptions. I ran it like so:

genpat oldfile.txt newfile.txt posix.pat

After that I got a problem where patch.good() returns 0 at this statement in main.cpp:

  if(source.good() && target.good() && patch.good()) {

The cause of this is the call to removeExistingPatch in the preceeding loop. As far as I can tell, it enters this if, and returns from it:

TFileOffset removeExistingPatch(bistream& in, TFileOffset inSize,

                                bostream& out, TChecksum* removeCRC, 

				bool existanceIsError) {

  // MD5 mode

  TFileOffset fileCount = 0x80000000;

  // empty file/does not yet exist

  if(in.bad() || in.eof() || (inSize == 0)) {

    writeDword(out,MAGIC_VPAT);

    writeDword(out,fileCount); // noFiles

    return fileCount;

  }

After that point, patch.good() returns false. Not sure how that breaks things, I don't know enuf C++.

I've been including your patch. Except for the checksums part, which I'm reimplementing using C++ streams. Note that if you break the checksum implementation, you might get these kinds of errors. At least, I'm happy with my test cases for VPatch know, because I was falling into the readsome trap of C++.

Can you maybe remove the manpage for tmpfile from this topic? The layout does not like it very much.

Originally posted by Koen van de Sande
Can you maybe remove the manpage for tmpfile from this topic? The layout does not like it very much.

All posts edited to fit in one screen.

Thanks Kichik 🙂

Here's an updated version of the (work-in-progress) GenPat-POSIX. Notes:
- compiles all files on Borland C++, MinGW 3.3.1, GCC 3.3.5pre (SuSE)
- Needs more testcases. Not for production use, since checksum code has been rewritten and not all boundary cases have been checked.
- WILL NOT LINK. Only Borland C++ can link its object files into an executable. MinGW and GCC spit out a huge list of undefined references. I need help with this 😕
- check out secure temp file creation

I'm quite curious how you got your code to link in the first place, 'cause I can't 😁

This link error is freakin' brilliant. 🙁

Basically, my bifstream/bofstream implicitly construct a
std::char_traits<unsigned char>. And this does not exist for unsigned char, it's actually documented for GCC.
And the same for MSVC7.

This is turning into a huge problem. By design all my functions read in bytes from a stream... and this just works in Borland C++. But apparantly the route I chose with change the input/output stream to use the unsigned char type does not work, in fact. So I should revert back to casting everywhere!?

Well, at least I am not alone in this. Defining your own char_traits is considered bad style... so I guess I'll have to cast. 😔 😱

So I changed it all to casting. But still need to clean up code in places (ifstream and bifstream are the same now, but I changed it everywhere before). This one compiles on Linux. And I guess it should work, but I haven't tested if the patches are valid. But it runs at least 😁

Compiled fine here too - no warnings either 🙂 - posix.pat (created from genpat compiled from genpat3posix3.zip) and patch.pat created from oldfile.txt and newfile.txt are exactly the same, apart from 8 bytes at byte 93. I'm guessing this is the timestamp field. Might be a good idea to run it over heaps of different combinations of files, hexdump all the .pat files and diff them just to be sure it works in lots of cases. I haven't tested if the decoder plugin understands posix.pat, but I assume it would.

Anyways, I think that once a new VPatch release is out with these changes included, then the next NSIS version should include the new VPatch. Yay, finally genpat can be included in the debian package 🙂

Originally posted by pabs
Anyways, I think that once a new VPatch release is out with these changes included, then the next NSIS version should include the new VPatch.

Yep.

I'm currently spending some time on the test suite. Something came to mind: what if the source and target file for GenPat are the same? Right now it will create a patch which just copies the entire file. Applying the patch will then work just fine with no errors. Is this the right semantic? I think it is, but I thought I'd check 😁

Here's an updated version to make it compile with MinGW also. Currently putting it through my test cases.

[edit]
The POSIX version uses a different function to retrieve timestamps from files. And the results are different: one of them does not respect time zones. VPatch 3.0 uses some Windows function, and retrieves: (2002, 2, 2, 4, 58, 22), while the POSIX stat function returns (2002, 2, 2, 5, 58, 22) which is off by one hour. Windows explorer shows 4:58 for the target file. Now what should I do about this 🙄
This regression was caught by my new test suite, by the way 😁
[/edit]

Now that I've reverted back to the Windows date/time extraction on the Win32 platform, I've encountered something in my test cases: a different patch. Visual C++ and MinGW generate different patches than the Borland C++ one... though they do work fine and have the exact same size. I feel I need to get to the bottom of this first before putting out a new version. It might just be a subtlety in the way the patch is generated, but better safe than sorry. At least, Visual C++ and MinGW do agree either on the patch 🙂

[edit]It's a subtle thing indeed: when multiple blocks can be used for a copy from the source file (with the same contents), they pick a different one[/edit].

Here's another new version. This one:
- complies to my tests (though MinGW/VC++6 have a different interpretation of patch making than BC++, they are the same size and give the correct result)
- compiles on Visual C++ 6, MinGW, GCC and Borland C++
- is considered a release candidate. If more people check if it compiles for them, then it's ready. It compiles for me everywhere 😁

Note to MinGW people trying to compile: I don't know if __WIN32__ is defined automatically in MinGW. If you want to use the Windows GetTempFilename/GetFileTime functions (which are slightly better than the POSIX ones), then you should verify that it is defined, otherwise it will use POSIX functions.

Compiles and works on Linux fine. You might consider using - instead of / as a command-line option delimiter on Linux, because on Linux / is part of filenames, and - is the commonly used option delimiter.

VPatch 3.0 was committed into CVS. Currently, GenPat is not compiled natively but only cross-compiled. I need to make some changes to the build system to make it compile natively.

Thanks!

I've put out the GenPat POSIX version as VPatch 3.1. Compared to the version in CVS (which is based on the genpat3posix5.zip) the notable changes (files attached):
- main.cpp was changed to use "-" as a switch option character on non-Win32 instead of "/"
- Readme.html was changed to include changelog of 3.1 (POSIX compatibility)

Remarks on the current CVS version:
- VAppend was in with 2.1, only it's now been moved to a "VAppend" folder for 3.0, and it's no longer in CVS. However, this might not be such a bad thing because the NSIS version does not need it, it's only for all non-NSIS runtimes. So I think it is best to leave this out. 🙂
- VPatch_tests.py runs several functionality tests on GenPat (Windows only), perhaps this can be included. One of the tests will not work because of missing files (only I have the datafiles on my PC), but all others are self-contained and quite useful. I'll leave the decision on this up to Kichik 😁
- VPatchLib.nsh makes more sense in the NSIS include folder (?) Or can it be included from where it is now as well?

You can find the ZIP/Installer distro of 3.1 on TibEd.net: http://www.tibed.net/vpatch.

Thanks, uploaded.

The tests script fails without VAppend so I didn't include it yet.

The header file is located in the Contrib directory in CVS, but it's installed into the Include directory.