Well, I finally found the problem, and I'm ****ed at how simple of a problem it is and how much headache and time it costed me.
But at least it's working now. Believe or not, the interference is caused by the refresh rate of the LCD. When my LCD is set to 85Hz, it creates interference that can even be seen on the LCD sometime. But since it was really minor, I didn't pay too much attention to it. When I adjust the refresh rate down to 60 Hz, the interference is gone, and long and behold, the GPS signal is much much more stable. Why or why didn't I think of this earlier?
I think this explains why some other people who have the same hardware setup with me but no problem for them. So don't try to use the highest refresh rate on your Lilliput!! I hope this would be helpful to people that are having "interference" problems.