What kind of serial ports? Are they built in serial ports from the motherboard, or are they some sort of USB-serial adapter?
For troubleshooting, disconnect all other serial "stuff" so you can determine where the problem is. ie. Work with one serial device only, unplug the rest. This will eventually tell you where the problem lies.