Johnm312
18th September 2008 03:56 UTC

Downloading Files from a list.
  I have a question.


I had a similar script for this in the past but it is not quite working for this install.

I have a list (actually a few). that are located on a server. that have a url followed by list of 100's of small files that I need to collect.

My old system had INI's where there were just a few files so it was easy to manage and download

the new server just has a text file with all the files listed that I need to grab.

***93;

>ww.someserver.com/files


>***91;Files***93;

>file1.dat

someotherfile.dat

randomename.dat

somthingorother.dat

etc.dat 

>

Anyone have any ideas on how to possibly grab the files?

QOAL
18th September 2008 13:51 UTC

You may be interested in a script (well two scripts) I wrote, it goes a bit further than you'd want I guess. (it crc checks the files on the users drive to check if they need updating or not.)

I'll attach them so you can have a look.
(I left them set up to create a list of files from a local SVN check out of a game I used to work on. And download the update file from localhost and the actual files from the actual SVN.)

My script uses 2 extra things:
CRCCheck.dll plugin form http://nsis.sourceforge.net/mediawik...d/CRCCheck.zip
RecFind.nsh from http://nsis.sourceforge.net/RecFind:...ext,_FindClose

Basically the Generate_Update_File.nsi script creates a file that lists, recursively, all the files in a given folder and their crc's.
The Updater.nsi downloads this file and checks the users local files to see what needs updating/deleting etc.

The code might need a bit of cleaning up, but it works.

Johnm312
29th September 2008 22:27 UTC

QOAL,

Thanks. I had had not had time to do a thing this last week. I just DL'd it and am going to take a look.

I appreciate the help.

Johnm312
30th September 2008 06:07 UTC

QOAL,

I just got time to test your script out.

Very nice! It is almost exactly what i had in mind. I am sure I could use this script with a few changes. I like the subdirectory searching.

I did find 2 bugs, 1 minor and 1 major.

#1 in the build script; It adds the text file itself to the directory list. if I don't manually remove it from the text file, the rest of the download fails after it hits that section.

Not a big deal, I just have to remember to edit the text file or maybe change the script to create the text file in the $exedir.

#2 In the download script; This is a bit of a problem. any source file or folder that has a space in it fails to copy.

The download name/path is correct in the log. but, i get failed download. I took a look all over the script and did not see how to fix that. I did try it on both a UNIX and a windows host to see if maybe it was the %20 missing.


2 other things that i did notice that would be of help;

1 the option to remove local files if they are not on the host. perhaps a flag to set in the text file.

2 this is probably not something that can be changed, all of my INI's (30+) and most of my documentation wants to re-download every time I run it. I can probably find a solution to this.

I am very impressed by this and I thank you for sharing it. If you have any ideas on the "space" issue, i would love some help.

QOAL
30th September 2008 11:43 UTC

Glad you like it. :)

Okay I've fixed those two bugs now:
It makes sure it doesn't add the updates file to the list
Replaces Spaces with %20, if any other characters need to be made web safe then take a look at line 206 of updater.nsi; copy and edit that line for extra chars.

To force it to download a file each time, even if it's the same as on disk, change the CRC value in the generated file to something other than the correct value. I have added a check for it being FORCE, which would be better to use since it should be a tiny amount faster and also easier to see what should be always downloaded.

To make it so you delete a file locally add the file name then 0 on the line below.
You could just make a empty file with the correct name and run the generator, has the same effect
To remove folders do folder name then RMDIR on the line below.

As for you last point, it should only download files if they have a different CRC than what's in the updates file (aka they have been modified), hrmm.
So maybe an option to just check if a file exists locally and if it does then don't download?
Okay I just added that, you'll have to change the CRC in the generated file to ONCE for each file that needs that.


I might be able to do something with arrays so you can list force download (, ignore?) & delete files in a separate file which the generator looks at and merges in to the update list for you. yay/nay? I'll have a play any way.

Any thing else? :)

Johnm312
30th September 2008 17:55 UTC

I just took a quick look at the update.

The generator works without a hitch.

The updater has a new bug. the word replace line for "/" "\" is now ignored, resulting in no downloaded files.
once I rem'd out the %20 line it was back to normal.


Something minor:
I noticed that the server path has a double // in it. This could have been in the last release and I overlooked it. I added a detailPrint line to show the server path to help debug the %20 error so now I see it.

Starting update process...
Downloading updates list file: updates.txt

Downloading new file: \USSE262A.dat
edited_servername.com/beta/test//USSE262A.dat
Folder: subfolder
Make: C:\Dat_Sync_Demo\subfolder
Create folder: C:\Dat_Sync_Demo\subfolder
Delete file: C:\Dat_Sync_Demo\updates.txt
Updating finished!
Completed

I think the extra backslash is created by the generator by adding a \ before the root files. so far my server could care less and still works.


Thanks for taking the time to update it.

QOAL
30th September 2008 19:16 UTC

Hrmm indeed

*one hour passes*

Should be fixed now

Johnm312
30th September 2008 20:26 UTC

Bloody Brilliant!

I tried most of the features in a single run, I than ran it a second time. It still has the issues with the INI files. I can live with that since several do get updated with each update.

The deletion feature works fine, File/Folder names are resolved that have spaces. the second "/" is fine.

Quote:

---

If I do go ahead with this script I'll make sure and credit you (Sorry no royalties, this is freeware).

You defiantly gave me a useful option and saved me days of pounding my head to a pulp on my desk.


[Afterhought]

I just had a thought about using your logic to create snapshot difference folder.

Take a snapshot of the virgin game folder; then add whatever user created mods/patches/updates. Take a new snapshot, copying whatever files/folders that are not CRC original to a new folder or zip. Thus creating a user created or custom patch that they can archive or re-apply later. Perhaps this might even save me time in creating my updates.


Anyways...

Thanks a ton! You defiantly made my day and I appreciate Your help.

John

Run#1: Output folder: C:\Dat_Sync_Demo Starting update process... Downloading updates list file: updates.txt Downloading new file: Text Document with Spaces.txt WebServer/beta/test/Text%20Document%20with%20Spaces.txt Downloading new file: USDDA40X.dat WebServer/beta/test/USDDA40X.dat Downloading new file: Version.ini WebServer/beta/test/Version.ini Delete file: C:\Dat_Sync_Demo\USGT45U2.dat Folder: subfolder_test Make: C:\Dat_Sync_Demo\subfolder_test Create folder: C:\Dat_Sync_Demo\subfolder_test Downloading new file: subfolder_test\USDD35.dat WebServer/beta/test/subfolder_test/USDD35.dat Folder: Folder with Spaces Make: C:\Dat_Sync_Demo\Folder with Spaces Create folder: C:\Dat_Sync_Demo\Folder with Spaces Downloading new file: Folder with Spaces\USGT45U2.dat WebServer/beta/test/Folder%20with%20Spaces/USGT45U2.dat Delete file: C:\Dat_Sync_Demo\updates.txt Updating finished! Completed Run#2: Output folder: C:\Dat_Sync_Demo Starting update process... Downloading updates list file: updates.txt Delete file: C:\Dat_Sync_Demo\Version.ini Downloading new file: Version.ini WebServer/beta/test/Version.ini Delete file: C:\Dat_Sync_Demo\updates.txt Updating finished! Completed

---