I've been doing exactly the same thing.
The screen isn't setting itself to 640x480, it's just reporting that it's displaying that. It lies. The jagginess you're seeing is due to the timings being wrong. So... we set the timings exactly as stated on the Xenarc FAQ page... but DisplayConfigX (DCX) won't let me do that - it changes some of the figures to something I don't want.
I emailed the author, and it's a bug in Mac OS X 10.3.x, and DCX prevents you entering values that'll screw up the OS. He said it'd be fixed in OS 10.4. Now, I have a developer subscription with Apple and receive developer builds through the mail, so I asked him to let me try it.
The latest version of 10.4 with the latest version of DCX now let me enter the values... and I get a blank screen. The screen switches itself off completely. Could be a bug in the prerelease version os OS X, or something else.
To this day, I haven't got 800x480 working properly on the Mac mini. But you're right, 800x600 looks nice!