Dealing with extremely large file count question?

Hi Guys,
a question when dealing with a very large amount of file being installed...
(200k count atm later on ill need to do the same for 1.5mil!)

using the new code bit from earlier post i am able to run the setup itself fairly quickly,
now comes the tricky part; the uninstall,

in this step i have to specifically revert the install procedure;
which is returning each file to its original location,
with a large file count and no option to use wildcard the uninstall bit is well over 20mb,
this is due to specific instruction given to each file,
the process of building the installer due to the long uninstall list is taking forever (up to 8 hours) to build,

(this installer only moves existing files from multiple dir's to a single dir,
the uninstaller reverts that step; but i have to be specific in that step,)

i think i need to build an external exe that does these specific revert back steps (they don't change) so when i rebuild i don't have to wait 8 hours for each build to complete,

is there another way to speed this up?

TIA
Chris

Post a short snippet of code that shows what you are doing for the uninstall files, there might be a bottleneck that could be widened.

One way to speed it up is during the install make a list of all the files that were moved and save it in the install directory somewhere as a file. Then on uninstall, just loop through the same file and do your specific work on each one.

Thank you Jason 🙂
here is a small snippet from my uninstall script,
its more of a patch installer in this case where we don't install files just move existing files from one dir to another during install; in install step i can use a wild card when moving from multiple folders into one;
the uninstall is tricky without the use of wildcard,

i also know the file names and location in advance so this list doesn't need to be "generated" during setup,

Section Uninstall
SetDetailsPrint none
        SetOverwrite on
          rename "$INSTDIR\Patch\00000118.BMP" "$INSTDIR\Dir0\00000118.BMP"
          rename "$INSTDIR\Patch\00010119.BMP" "$INSTDIR\Dir1\00010119.BMP"
          rename "$INSTDIR\Patch\00010120.BMP" "$INSTDIR\Dir2\00010120.BMP"
          rename "$INSTDIR\Patch\00020121.BMP" "$INSTDIR\Dir3\00020121.BMP"
          rename "$INSTDIR\Patch\00020122.BMP" "$INSTDIR\Dir4\00020122.BMP"
          ..... 
          (the list continues with about 200k similar rename entries)
        SetAutoClose true
SectionEnd 

the uninstall bit just moves files from one dir to another, no new files added or removed,

TIA
Chris

You shouldn't be using Rename to move files around, use CopyFiles and Delete.

You know that you can keep the directory structure the same when you move the files over, this allows you to use the same wildcard loop for the installer and uninstaller and all you change are the directories it points to. ProcessTextFiles will have to be modified to take two directories instead of one.

On the other hand, if you know the files to be moved at compile time, you can use compile time commands to generate two nsis script files, one for the installer and one for the uninstaller. Then you just !include that file in the section instead of doing it at install time.

Thank you Jason 🙂
wouldn't copy then delete take longer?

ATB
Chris

I don't know if it will take longer, I haven't timed it.

Another option is using the system plugin to call MoveFileEx(), you will have to read the documentation on how to use it.

Thank you Jason 🙂
i will look into your suggestion,

since this is a "patch installer" we are not installing or removing files at any point,
we just move them within the same top folder from multiple dir to single dir (installer part)
and vice versa (uninstaller part) from single dir back to multi spesific dir location,

uninstall is a spesific revert off a list embedded in the uninstaller,
and thats where the issue of this long stall is happening,

there are no files to compress/decompress in the process,
its just move instructions on install with wildcard;
and than uninstaller spesific restore,

i thought renaming files makes the most sense in this situation since t s just a ptr record change,

TIA
Chris

here is what i came up with so far,
working off external list like so should speed things up,
since rename wont take one line operation i had to use two files;
one with a line containing the copy "from" path, and the other the copy "to" path,

this works fine in my standalone test with small number of files,
once in the full script it will stop looping after one loop,

what am i doing wrong here?

section
ReadRegStr $INSTDIR HKCU "Software\..." Path
Open:
FileOpen $1 "$INSTDIR\from.txt" r
FileOpen $2 "$INSTDIR\to.txt" r
IfErrors exit
loop:
FileRead $1 $3
FileRead $2 $4
IfErrors closelist
Rename $INSTDIR\$3 $INSTDIR\$4
Push $3
Push $4
Goto loop
closelist:
FileClose $1
FileClose $2
IntOp $5 $5 + 1
Goto exit
exit:
sectionend 

TIA
Chris

as usual as soon as i post for advice i find the answer a minute later,
i just had to clear errors first 😉

ATB
Chris

Interesting, you don't need 'push $3' and 'push $4', because you aren't popping them off anywhere. Same with IntOp $5 $5 + 1 ($5 isn't used anywhere). But I assume they are being used in your actual script?

Thank you Jason 🙂
$3 and $4 are the lines passed from reading each of the lists,
$5 is leftovers from my test; i was sure it increments my next line 🙄

Ive actually made few more adjustments but i seem to be stuck atm,
the approach i used above works with one condition; that all files be present,
if one file is missing the whole script exits in the middle of uninstall instruction with external list (not what we want); i need to keep parsing through the list to the end regardless of file present in origin or not,

my latest attempt is to pass a blank line ""/NULL as the returning value to exit the list by using a blank line in from.txt

but that not working for me atm 🙁

from.txt content file looks like this

CB\test1\image0.JPG
CB\test1\image1.JPG
CB\test1\image2.JPG
CB\test1\image3.JPG 

to.txt content looks like this

CB\test2\image0.JPG
CB\test3\image1.JPG
CB\test4\image2.JPG
CB\test5\image3.JPG 

section
ReadRegStr $INSTDIR HKCU "Software\..." Path
Open:
FileOpen $1 "$INSTDIR\from.txt" r
FileOpen $2 "$INSTDIR\to.txt" r
IfErrors exit
        loop:
        FileRead $1 $3
        FileRead $2 $4
    StrCmp $3 "" closelist
        Rename $INSTDIR\$3 $INSTDIR\$4
        Push $3
        Push $4
        Goto loop
        
        closelist:
        FileClose $1
        FileClose $2
        
        IntOp $5 $5 + 1
        Goto exit
        
        exit: 

ATB
Chris

i settled on this for now
in from.txt last line is "end"

        ReadRegStr $INSTDIR HKCU "Software\..." Path
        ClearErrors
        Open:
        FileOpen $1 "$INSTDIR\from.txt" r
        FileOpen $2 "$INSTDIR\to.txt" r
        IfErrors exit
        loop:
        FileRead $1 $3
        FileRead $2 $4
        StrCmp $3 "end" closelist
        Rename $INSTDIR\$3 $INSTDIR\$4
        Goto loop
        closelist:
        FileClose $1
        FileClose $2
        Goto exit
        exit: 

ATB
Chris