When I use InstallLib with REBOOT_PROTECTED, it doesn't work at all. When the setup tries to replace the dll, the setup stops and I have to click on cancel.
Here is my code ($OUTDIR = $SYSDIR):
!insertmacro InstallLib REGDLL $0 REBOOT_PROTECTED "${LOCAL_PATH}\DLL\${FileName}" "$OUTDIR\${FileName}" "$OUTDIR"