I had over-heating issues when my last system was in the trunk as well. that's one of the reasons I designed my new setup with the PC under the passenger seat.
fans will certainly help, but as long as all of the air in your trunk is hot then that fan will just be blowing the hot air around.
you need a fan to suck in cooler air from the passenger cabin or exhaust the hot air from the trunk (or both). most cars have exhaust vents somewhere in the trunk body panels. if you pull up the carpeting in your trunk you might find the exhaust vents. maybe you could mount a fan on inside of the vents to blow the hot air out of the trunk.