the noise is definitely a ground loop. ground loops are caused when different devices have different resistances to passing voltage to ground. in a perfect world, this resistance would be zero-- which would mean that you wouldn't have problems...
as bad monkey said, a common fix is to try to connect everything to the same power points. the other common causes are poor grounds--either from a loose/corroded connection, or from a generally poor connection. if your using the factory ground connections for some devices, but sanded the paint off the area for other devices, the factory ground could be a poor connection.
or, as you started to suspect, a single devices poor power supply could cause noise issues as well.
another really easy thing to try is to connect a wire between the negative part of the rca, and ground. sometimes this will drain the noise off the line..(but it doesn't always work, or can make the noise louder-- so be prepared to take it off real fast)