You may be right as many video cards have limited overlay support. In any case, RR does NOT use any overlay support directly. The only time RR could be using a video Overlay is when playing a video on the video player and/or DVD player, still this depends EXCLUSIVELY on the PLAYER used. The built-in player will use whatever defaults are set in windows -- this generally means using overlay when available. MPC can be set specifically NOT to use overlay if you want. I would assume that VLC may have similar options.
I would suggest YOU configure RR to use MPC and set it up to use VMR9 -- this should prevent it from using any overlay (just so you can use it outside).
What doesn't make sense to me in all this is why would you want to watch streamed tv while playing a video in RR -- if you're not playing a video in RR then your video overlay should be available to use in another player (and if this is the case and it's not working, then you may just be running out of video resourses -- you can try usedx=false in RR.INI and see).