I think so, at least if they are the same release (e.g. both are gutsy or whatever).
First off, clean the apt-get archive.
This should make sure, I think, that you don't have any old sources when you later try to apt-get install something. Then add the xubuntu cd to your sources list using the command
Terminal will prompt you to pop in a CD. Do it, press enter and it should scan the CD. After that I think you should be able to install anything from the xubuntu CD by using the regular apt-get command. That should include the xfce desktop.
Hope it helps
/mojoman