Getting the output to turn on and off however will not be easy. Using the speaker into an input would require a filter of some kind and some sort of cap integrator to know when it is speaking. This will introduce lag time as well as this cap charges to make sure it is voice and not just noise. So you might miss the beginning part and that would be annoying.
It is probably best to have a software control that detects when there is output to the speakers, and then flip the FB output.