RmDir "$INSTDIR" not removing directory

When I run RmDir "$INSTDIR", the directory remains. Everything within the directory structure is gone, but $INSTDIR remains. How do I get the directory to delete?

Tried
RmDir /r "$INSTDIR" ?

If that doesn't work then
Delete "$INSTDIR"

-Stu