Honestly, I don't think it would be an issue.
The biggest concern I'd think would be the CPU cooler coming loose. If it's securely mounted, then that wouldn't be an issue.
With motherboards that require a gooldy amount of cooling and a larger HSF, I wouldn't suggest it. With anything that you'd put in a Voom case, I think it would be OK.