You can't prise the switch out of the armrest..
I've recently (two days ago) replaced my other half's drivers side switch module.
Take a look at Sarge's thread on how to remove the door card that spike posted.
You can't directly remove the housing that the switches clip into, but you can gently pull the armrest out far enough to unclip the housing and thus remove it entirely from the door card.
Once done, it's fairly simple to unclip the switch and replace it. New switches are about £28 from Audi.
Be aware that when you've clipped the lot back in, plug back in the control lead (purple clip) and try the windows AND the mirrors before re-attaching the door card. You may want to have a small child to hand to hold the door card or operate the switches.
If they don't work, unplug the control lead and re-insert it firmly...
I spent about 30 minutes getting very worried cos the newly inserted switch didn't do jack! It took 3 unplugs/plugs to get it to respond properly :|