In fact, many amplifiers include differential inputs and a high-level signal can be used on the RCA inputs. The input sensitivity control will have to be turned down, though.
Here's how to tell if your amp is capable:
1. look at the CEA 2006 rating for maximum input signal voltage. If it's better than about 5V, it'll be ok. 6V+ is better.
2. Use an ohm meter to be sure than none of the RCA shield connectors are continuous to each other or continuous to chassis ground.
If your amp meets these two criteria, you can safely connect the speaker outputs directly to the RCA inputs. The benefit of this arrangement is tht the signal will be completely isolated from ground and the input voltage is higher than using RCA outputs. That means NO NOISE.