Look to me like a uncalibrated finger
I did a lot testing and trying myself on this an I figured, that it doesn't make sense to add any kind of offset to the mouse move because even 40 pixel offset dont do it driving through road holes.
What I'm actually doing I'm adding a time offset (which is 600 ms). If you hold it longer then this time period, it switches into the index scrolling mode (this btw. works only on the right half of the phonebook).
So, use the right half of the phonebook for selecting names and the left half for scrolling through the phonebook and everything will work perfect.