It is possible; DJ190 did mine.
After removing the door cards, you will see that the armrests are held on by plastic/nylon 'rivets' for want of a better description. Once you dremel off the heads of these rivets, the armrest is free. You can reattach the armrest to different door cards by substituting short thick screws in place of the rivets.
No damage was caused to anything on my A2 during the procedure.