ImBcmDth
13th August 2007 22:39 UTC

NSISdl gets resume ability
I recently discovered to my horror that the NSISdl plugin has no concept of the HTTP 1.1 "Range:" header. This isn't much of a problem with small files but often on larger transfers a long-lived connection may be reset by an intervening router or proxy and firewall or antivirus software.

Once you get files that are greater than a gigabyte, you will go looking for a more robust downloading solution. I examined InetLoad but it seems a little buggy and is in development. NSISdl does nearly everything I wanted so I decided that I should try adding the resume feature to retry failed transfers.

CHANGES
Instead of immediately failing from a timeout, dropped connection, or bad server response the new code attempts to reestablish the download where it left off (up to "/RETRIES=" times.)

All of the changes except one are in the nsisdl.cpp file's download function. The only change outside of that file is to allow a response code of 206 (Partial Content) in httpget.cpp.

The code seems to work fine for me but I would love to get people to test it. I whipped up a quick downloader to fetch the latest OpenOffice and pulled the plug on my router several times during the transfer as a test. The download completed perfectly and the file was not corrupt.

NOTES
The code is ugly, the orginal NSISdl.cpp was in bad need of refactoring (any takers? hah!) and my additions make the problem even more obvious. The huge download function desperately needs to be broken up into more manageable pieces but I was more concerned last night with getting the code working order than fixing a pretty ugly function.

Attachment contains source code(only the two modified files), precompiled nsisdl.DLL (compiled for size), and an example OpenOffice downloader .nsi script.

Jon

ImBcmDth
14th August 2007 03:22 UTC

How do code fixes get added to the CVS? Should I submit this as a fix for a bug through the bug tracking system? Also are there any coding style guidelines I should follow?

Thanks in advance.

Jon

Anders
14th August 2007 08:56 UTC

Yeah, post on the patch tracker ( http://sourceforge.net/tracker/?grou...49&atid=373087 )

http://nsis.sourceforge.net/Inetc_plug-in already supports resume I think, and works better AFAIK

Takhir
14th August 2007 12:00 UTC

About InetLoad: it was created for re-get option :) Yes, we had information earlier about (seldom) crashes if plug-in was running in NSISdl mode (embedded progress bar), this happened in the part of code that came from NSISdl plug-in :). Last year I don't see any problems with InetLoad (might be fixed, or may be because inetc is now in use).
I left InetLoad "as it is" making critical updates only because too many people use it. Inetc is much better, it not uses NSISdl embeded progress bar :) that caused crashes (embedded borderless child dialog with cancel button appears now), it is stable and "under improvement" ("under development" sounds like "not finished" :) ), new features appear ... if requested and look reasonable :).
IMHO NSISdl is good in simple environment (no proxy, for example), on Win95 Gold (no IE) and if you want learn how all these network/http tricks can be implemented in C code. But for buseness applications I vote for WinInet base of inetload/inetc. Just because it was tested billions of times. And it supports a lot of add-ons, for example, I had request for redirections, but found that this was already implemented by MS in WinInet, so it works already.

ImBcmDth
14th August 2007 12:51 UTC

I didn't mean any disrespect when I referred to Inet* as in development Takhir. :D You are right though, a better term would be under improvement. (I like that, I'm stealing it.)

I guess I just liked the simplistic nature of NSISdl and its single minded purpose. I only wished it could be more bulletproof. I wanted to have a direct replacement for users who didn't have the time or knowledge to change their install scripts over to Inet*.

This way they can simply recompile their install script (if my changes ever makes it to the official release) and *poof* they would have a downloader that doesn't panic when the connection is reset for a variety of reasons. (I have a linksys router. :D)

Well, at worst it was a learning session. I have to admit, I didn't know about Inet* until after I started working on the changes to NSISdl. Makes you wonder what percentage of users use NSISdl just because its part of the default package and only search for an alternatives when it begins to show its age?

Jon

Backland
14th August 2007 14:24 UTC

Looking over the details, I'm not really sure why InetC isnt part of the default package. Might be a good idea to request its inclusion in the tracker.

ImBcmDth
15th August 2007 03:33 UTC

InetC has more dependencies then just using NSISdl. This isn't a problem if you are only targeting Windows 98 or Windows 2000+ but if you happen to have a desire to use a downloader on '95 or NT then you pretty much have to use NSISdl.

I am pretty sure that is why it remains the default internet plugin. The baseline NSIS has to work on as many Microsoft operating systems as possible. How can you tell someone they can't install My_Great_Utility on an '95 computer? I still know people who run Windows 95 because all they do with a computer is "check their email."

The libraries that are required for WinInet functionality are non-redistributable according to Microsoft.

I guess there is a case to be made for continued support of NSISdl. I was sorta getting discouraged in my efforts with NSISdl. Does anyone else use it still?

Jon

Backland
15th August 2007 07:42 UTC

I think it might be possible to code your installer to use NSISdl for win95 users while using InetC for those with WinInet available. A bit of extra work, but you can get the best of both.

Takhir
15th August 2007 08:32 UTC

http 1.1 chunked server reply handling would be a very good option for NSISdl (or this is done already? :) Forum search gives only "kichik: TRANSFER-ENCODING: chunked is a feature of HTTP/1.1. NSISdl uses HTTP/1.0.").
Code for this is very simple, few years ago I wrote socket based http client with this option (30 lines only) for live output in the java applet (because java httpconnection caches data).

ImBcmDth
19th August 2007 12:40 UTC

I just completed a pretty massive rewrite of nsisdl.cpp to clean up my contribution into something more suitable for wide distribution.

Where I am:

NSISdl.cpp
Revision 1.4 - 2007/8/16 - Jon-Carlos Rivera 
-Download function now attempts to resume a failed download
-Arguments to the download function are now order independent (ReadMe.txt updated to reflect this change)
-Translation now includes resume string 

ImBcmDth
19th August 2007 12:42 UTC

It seems that previewing a post results in attachments being lost.

Jon

kichik
19th August 2007 21:28 UTC

Originally posted by ImBcmDth I am considering adding the error strings (such as "Connection timeout." NOT the standard "success" and "cancel") to my new /translation3 function. Any reason I shouldn't do that?

Takhir
20th August 2007 06:44 UTC

Hey Takhir, I looked at adding chunked transfer decoding to NSISdl. I understand that the Content-Length header shouldn't be present in chunked cases meaning you don't know the length of the file you are downloading. If so, then how do you handle the progress bar during (large) chunked downloads?

ImBcmDth
20th August 2007 06:51 UTC

Thanks Takhir, I assumed that would be your answer. I was hoping that there was some super secret way to know the file length because some "bug-workaround" code in NSISdl depends on knowing that in advance.

This item is now in the tracker:
http://sourceforge.net/tracker/index...49&atid=373087

Jon

ImBcmDth
26th August 2007 03:58 UTC

Hey I'm back with progress.

Here is a tip for you: the code in a standards compliant "transfer-encoding: chunked" decoder (hey! say that ten times fast) is a nightmare to debug.

NSISdl CHANGES
-----------------
HTTP/1.1
-Chunked Transfers
-Supports (ignores) "100 Continue" response

More Robust
-bad-connection retry capability (/RETRIES=)
-File-based resuming (/RESUME)
-Script supplied file size (/FILESIZE=)

Less Demanding
-Order independent arguments
-Doesn't care at all about Content-Length tags anymore
-Doesn't delete incomplete downloads (/NODELETE)

Other
-Error message internationalization support (yell at your customers in their native tongue!)

Jon

Afrow UK
26th August 2007 10:59 UTC

Now if you could just add a switch to make NSISdl use 50% bandwidth (or less) :)

Stu

ImBcmDth
31st August 2007 04:04 UTC

I have been trying all week-end but I haven't figured out how to lower bandwidth utilization. :D

I did manage to make NSISdl exhibit no processor use (at least it registers as 0% load on my computer.) This is a huge step up from the near 100% cpu use I was seeing from the base NSISdl in 2.29!

This change doesn't require any new switch. ;)

Jon

ImBcmDth
31st August 2007 07:42 UTC

Originally posted by ImBcmDth
This is a huge step up from the near 100% cpu use I was seeing from the base NSISdl in 2.29!

NSISdl 1.4 - HTTP downloading plugin for NSIS
---------------------------------------------

Copyright (C) 2001-2002 Yaroslav Faybishenko & Justin Frankel

This plugin can be used from NSIS to download files via http.

To connect to the internet, use the Dialer plugin.

USAGE
-----

NSISdl::download [ARGUMENT 1]..[ARGUMENT X] <REMOTE URL> <LOCAL FILENAME>

..Or to download without the progress window appearing:

NSISdl::download_quiet [ARGUMENT 1]..[ARGUMENT X] <REMOTE URL> <LOCAL FILENAME>

The return value is pushed to the stack:

  "cancel" if cancelled
  "success" if success
  otherwise, an error string describing the error

Example of usage:

NSISdl::download http://www.domain.com/file localfile.exe
Pop $R0 ;Get the return value
  StrCmp $R0 "success" +3
    MessageBox MB_OK "Download failed: $R0"
    Quit

For another example, see waplugin.nsi in the examples directory.

ARGUMENTS
---------

Arguments passed can be any combination of these (a proper value in <..> is mandatory):
/TIMEOUT=<number of milliseconds>
/RETRIES=<number of retry attempts>
/FILESIZE=<file size in bytes>
/PROXY <proxy URL>
/RESUME 
/NODELETE 
/NOIEPROXY

The next three arguments are mutually exclusive. If more than one of the
translate options are used, the last one specified takes precedence:

/TRANSLATE3 <downloading> <connecting> <second> <minute> <hour> <seconds> <minutes> <hours> <progress> <resuming> <timeout_error> <resume_error> <file_error> <incomplete_error> <badresponse_error> <downloading_phrase> <connecting_phrase> <readingheaders_phrase>

/TRANSLATE2 <downloading> <connecting> <second> <minute> <hour> <seconds> <minutes> <hours> <progress>

/TRANSLATE <downloading> <connecting> <second> <minute> <hour> <plural> <progress> <remaining>


TIMEOUT
-------

You can pass /TIMEOUT to set the timeout in milliseconds:

NSISdl::download /TIMEOUT=30000 http://www.domain.com/file localfile.exe


RETRIES
-------

You can pass /RETRIES to set the number of times a file download will be
retried (using HTTP 1.1 Range header) before it finally returns a failure.
This should allow for small connection outages.

NSISdl::download /RETRIES=2 http://www.domain.com/file localfile.exe


RESUME & NODELETE
-----------------

NSISdl can resume downloading incomplete files with /RESUME switch. If the
switch is set and the output file exists, the file will be opened and the
existing file length read. The download will then continue downloading
from the end of the file.

NSISdl can also be set to ingore incomplete or failed downloads with the 
/RESUME. This feature gives the script writer more options when dealing with 
a partial download. Together with the /RESUME option, you can now continue 
downloads started during a previous installation session or work around
major connection issues.

This is one way to use /RESUME and /NODELETE in a script:

!define DL_FILE_URL "http://www.domain.com/file"
!define DL_FILE_SIZE 15218444

StrCpy $1 "$INSTDIR\download.tmp"

IfFileExists $1 +1 DoesntExist
  Push $1
  Call FileSizeNew ; FROM - http://nsis.sourceforge.net/Getting_File_Size
  Pop $2
  IntCmp $2 ${DL_FILE_SIZE} Success
  MessageBox MB_YESNO "The file $1 already exists.$\nDo you want to resume the download?" /SD IDYES IDYES ResumeDL IDNO DoesntExist
ResumeDL:
    NSISdl::download /TIMEOUT=25000 /NODELETE /RESUME /RETRIES=8 /FILESIZE=${DL_FILE_SIZE} ${DL_FILE_URL} "$1"
  Goto Done
DoesntExist:
    NSISdl::download /TIMEOUT=25000 /NODELETE /RETRIES=8 /FILESIZE=${DL_FILE_SIZE} ${DL_FILE_URL}  "$1"
Done:
  Pop $R0 ;Get the return value

  StrCmp $R0 "success" Success
    MessageBox MB_OK "Download failed: $R0"
  Quit
Success:


FILESIZE
--------

The plugin has support for script provided file sizes. This is primarily
for displaying a progress bar even when the server doesn't send a content-
length header or when the server uses chunked transfer encoding.

NSISdl::download /FILESIZE=1048576 http://www.domain.com/file localfile.exe


PROXIES
-------

NSISdl supports only basic configurations of proxies. It doesn't support
proxies which require authentication, automatic configuration script, etc.
NSISdl reads the proxy configuration from Internet Explorer's registry key
under HKLM\Software\Microsoft\Windows\CurrentVersion\Internet Settings. It
reads and parses ProxyEnable and ProxyServer.

If you don't want NSISdl to use Internet Explorer's settings, use the
/NOIEPROXY flag. For example:


NSISdl::download /NOIEPROXY http://www.domain.com/file localfile.exe
NSISdl::download /NOIEPROXY /TIMEOUT=30000 http://www.domain.com/file localfile.exe

If you want to specify a proxy on your own, use the /PROXY flag. For example:

NSISdl::download /PROXY proxy.whatever.com http://www.domain.com/file localfile.exe
NSISdl::download /RETRIES=3 /PROXY proxy.whatever.com:8080 http://www.domain.com/file localfile.exe


TRANSLATE
---------

To translate NSISdl add the following values to the call line:

/TRANSLATE3 <downloading> <connecting> <second> <minute> <hour> <seconds> <minutes> <hours> <progress> <resuming> <timeout_error> <resume_error> <file_error> <incomplete_error> <badresponse_error> <downloading_phrase> <connecting_phrase> <readingheaders_phrase>

Default values are:
 
  downloading - "Downloading %s"
  connecting - "Connecting ..."
  second - " (1 second remaining)"
  minute - " (1 minute remaining)"
  hour - " (1 hour remaining)"
  seconds - " (%u seconds remaining)"
  minutes - " (%u minutes remaining)"
  hours - " (%u hours remaining)"
  progress - "%skB (%d%%) of %skB @ %u.%01ukB/s"
  resuming - "Resuming interrupted download (%d of %d retries remaining)"
  timeout_error - "Timeout while %s."
  resume_error - "Server does not support resume."
  file_error - "Unable to open %s."
  incomplete_error - "Download incomplete."
  badresponse_error - "Bad response status."
  downloading_phrase - "downloading"
  connecting_phrase - "connecting"
  readingHeaders_phrase - "reading headers"

The old /TRANSLATE2 method still works for backward compatibility.

/TRANSLATE2 <downloading> <connecting> <second> <minute> <hour> <seconds> <minutes> <hours> <progress>

Default values are:
 
  downloading - "Downloading %s"
  connecting - "Connecting ..."
  second - " (1 second remaining)"
  minute - " (1 minute remaining)"
  hour - " (1 hour remaining)"
  seconds - " (%u seconds remaining)"
  minutes - " (%u minutes remaining)"
  hours - " (%u hours remaining)"
  progress - "%skB (%d%%) of %skB @ %u.%01ukB/s"

The old /TRANSLATE method still works for backward compatibility.

/TRANSLATE <downloading> <connecting> <second> <minute> <hour> <plural> <progress> <remaining>

Default values are:

  downloading - "Downloading %s"
  connecting - "Connecting ..."
  second - "second"
  minute - "minute"
  hour - "hour"
  plural - "s"
  progress - "%dkB (%d%%) of %ukB @ %d.%01dkB/s"
  remaining -  " (%d %s%s remaining)"

makhellion
6th September 2007 08:20 UTC

@ ImBcmDth

Hey man, I would like to thank you. I have no idea about what you did to bring me this working version of the nsisdl plugin ;) , but all I know - that is all I need to know - is my nsis packet is perfect now.
I had problems trying to download a php page with the original plugin, and successfully tried your nsisdl which could handle that page without any problem, and noticed no problem with the cpu utilization.
Thank you again!

Kar.ma aka makhellion