Well, I finally hit on an idea that I should have earlier, but spaced on.
Like I said, the lilliput works fine if toggled on and off. So I looked at the menu and the entry where it displays the current mode info.
At 800x600. It listed horizontal at 46.9khz and the vertical at 75hz.
So I took a guess, I divided the horizontal by the vertical lines and came up with, approx 78.16
So plugged in 78.16 for HorizSync and voila! It works. Boots right into X without the screen going nuts.
One problem though, now it comes up into a lower refresh rate. 37.9khz x 60hz. Weird. But good enough for now. I'll have to play with it some more when I get a chance.