--purge does not do anything with dependencies. You will need to explicitly remove all packages that are marked as installed manually, i. e. all packages that you pointed a package manager to install. If a DE was installed automatically by Debian installer, or if you installed it with apt install xfce4, the only manually installed component it the xfce4 metapackage, and using the --auto-remove flag will remove all its dependencies. But if you additionally installed any components or packages that depend on that components, you will also need to clean them up manually.