The Y connectors for your audio signals are part of the problem, methinks.
You're splitting the signal in half for each line.
Two possible solutions:
1. Configure your sound card/chipset to output 5.1 and ignore the center channel. Then use headphone jacks into RCAs for each audio channel.
2. Put an EQ/preamp between the PC and the amps. Run simple L/R stereo into the EQ/pre-amp and let is split the signals into 4.1.
I use the second option. It makes for less headaches, especially when you get an MP3 that someone encoded for only 2 channels or something stupid.
You get the added benefit of higher audio signal voltage feeding the amps, which results in better overall sound quality.