Well... you have pretty much deduced the problem, so good job with that
Thing is it is an odd problem. Thicker cables are better (usually). Do all your vga cables (the thin working one versus the non-thin one) have the same number of pins? VGA cables dont have to have all the pins in there connected either so one side may not have all pins. SOme are used for data back from the monitor like type/resolution that sort of thing.
My only guess right now is that the thicker ones probably have all pins exposed so the monitor is sending back data that it is not 800x600 but 800x480 and the graphics chip can't handle that weird resolution... But the thin cable doesnt send it out, so the computer doesnt know or care and just blasts it with 800x600 which the monitor then internally scales down to 800x480 by itself.
Other than that, I am pretty much stumped.