vibration is almost inevitible, that's how sound is made. dampening the sound is the only way to get rid of the rattle. For my set-up I have a trunk pop motor that is rattling the hell out of my trunk. How did I find it, i got my @$$ in the trunk, and looked. Started touching things on the trunk lid, and found the motor was not just loose but designed to be that way. Dampened the sound with dynamat extreme. There's a little vibration left, but it's something I can live with.
Dumb question though:
Why do you care if your trunk rattles, when are you listening to your subs outside the car?
(I know it's a moot point though, because with proper damping comes better sound, and that's why I did dynamat.)
In theory, the other drivers should have to pay for the convenience of not having to hear the trunk rattle, rather than you spending a couple hundred for a dynamat trunk kit. (and time to install it)
If it's only to prevent the rattle, isolate it by looking in the trunk. with it closed and speakers pumping, press on different parts of the trunk and see what's making noise. sometimes it's the tail light just a little loose, or the license plate flapping away. simple fixes.
If you want to get better sound isolation out of the subs themselves, completely dampen the trunk with dynamat and that may in itself fix the rattle.
A combo of dampening and closed cell faom (ensolite) will work. You have to first determine what is rattling, then figure out if making it's one sound or rattling against something else. If it's making sound on its own (vibrating) then dynmat or similar material, if its rattling against another object, use closed cell foam or ensolite. Most of the time it's just your liscense plate.