You are overloading the 5v rail as you found... The thing is that transistors vary with temperature. So yes heat has to do with it. Not only does the power supply become less efficient (gives less current on the 5v rail), but the devices pull more power to cover the increased inefficiencies. It is a compounding problem.
Now this change is rather small. If it is enough to make your system instable, you are right on the edge of a dangerous cascading failure.
Maybe a simple 5v supply to power your peripherals is all you need, but I would definately not suggest you leave it as is.