Going back to what Kichik already pointed out:
From Kichik:
You can't compress the installer before it's created. Even more, you can't use UPX to compress the complete installer because it'll fail the CRC check. The data after the header/stub is compressed anyway.
In other words, UPX can only compress the exe
header not the full EXE when used from within your script. And, if you try to compress the full EXE after it's created, then you'll run into CRC errors when you try to run it.
In my opinion, the savings you'd gain with UPX (or any other EXE compressor) would be minimal at best, which makes me wonder why you'd even want to bother. (Consider this: A barebones do nothing script compiled with NSIS compiles to about 34K. 34K is tiny!)
So my advice would be that if you want to compress the full EXE, then just specify the compression method as solid lzma in your script. (command: SetCompressor /SOLID lzma) You won't get much better compression than that. (If you do, it will be very minimal at best.)