My VW had a thin rubber coating on a bunch of the interior pieces and I also had the problem with it wearing off. But I managed to sand mine off. It was a pain and clogged up the paper a lot, but using a 220 grit paper seemed to work of then I just used a filler primer then lighter paper to make it smooth.
But for people who need to do this and sand paper isnt working i would suggest taking it to a local shop somewhere and have then lightly sand blast or glass bead it off for you. Ive done it a few times and it works great.