polyfill isnt used too often in ported applications. Ive heard talk about people gluing sheets of polyfill to the walls, but at that point you may as well purchase eggcrate foam. I'm not so sure the difference would be noticible considering the noise floor in a car anyways.
I'd err on the lazy side myself, and not bother.
however, a hollow sound might be caused by box vibration. how well is your box braced? would it be very difficult to cut some wood and glue pieces inside the box now that its already together?