Hope this is the right forum for this!

I am after a bit of help with a wiring issue for my stereo.
I am currently running the stereo off the main power supply (main battery). What I'm trying to do is put in a 3-way (on-off-on) switch in the dash to flip between powering the stereo off the main battery (while driving) to the second leisure battery (when parked up, so as not to run the main battery down). I've put the switch wiring in, but the supply from the second battery is not good, and I think it's because of the way I split the power feed between the fuses/fuse box.

I have two fuse boxes (4 points on each) attached to a board, as well as a connector block for earthing. There's one big feed in to the first point on the fuse box and then I've daisy chained power to all the other fuse points. Also, I have one large earth cable that is connected to a 15amp connector block, also joined together.

The connector block that I've used to common up the earths is brand new, all connections are good on that front and showing 0 when doing a continuity test

So, when I want to add a device, I make sure I put in the appropriate fuse, plug the live into the top of that fuse point and the earth into the connector block.

The diagram shows how it's set up.

The problem:
When I flip the switch on the dash from battery 1 to battery 2 for the stereo, the display of the stereo is faintly flickering, as is the little green light that I have put inline (as an idicator that
I'm running off batt 2). I have put the multimeter on the fuse - 4-amp - one connecter at either end and if I'm reading the display right, it's showing 0.09 amps.

You'll see that the stereo is the 6th fuse along. None of the other devices are on when I'm testing the stereo.

Can anyone tell what it is that I've done wrong here? Is it the way that I'm trying to split the power from the feed from the battery