Removing nvidia drivers without killing kubuntu-desktop (Kubuntu 12.04)

I am trying to install CUDA drivers on kubuntu 12.04, but in order to do so I need to remove nvidia drivers (which come in with the distro). The problem I have is that while doing so it also removes kubuntu-desktop, which is disasterous (though cuda installs ok). I had the same problem when trying Linux Mint 13 KDE.

I followed the link here:
http://sn0v.wordpress.com/2012/05/11/installing-cuda-on-ubuntu-12-04/
and noticed that some people have come across this problem as well, but no solution has been found yet - except for not installing the cuda, lol.

I also tried this:

sudo apt-get purge nvidia-common

only to get the same warning of being about to remove the desktop:

Reading package lists... Done
Building dependency tree   
Reading state information... Done
The following packages will be REMOVED:
    kubuntu-desktop nvidia-common
0 upgraded, 0 newly installed, 2 to remove and 435 not upgraded.
After this operation, 197 kB disk space will be freed.

Do you want to continue [Y/n]? n

My question is - can you remove the nvidia drivers in a way that does not remove kubuntu-desktop (or ubuntu-desktop) at the same time?
Or do I have to go with killing the desktop and then install the it again after cuda is installed? Wouldn't it affect other apps, which depend on kubuntu-desktop?

I need CUDA for my CG work and so any help would be really welcome, thanks!