Yes the adapter could be bottlneck, but I think you have other issues.
That card has a max transfer read speed of 25Mb/s and max write of 20Mb/s.
ATA100 from before the new millenium is 100Mb/s, ATA133 is 133Mb/s. SATAI is 150Mb/s and SATAII is 300Mb/s.
I think there is your bottlneck.
A CF card is never for speed. It is for durability.