DIY speaker switch

I finally (2-3 month ago) got a new pair of monitors - Dynaudio bm6a. i still have my old NS10 and I wanted to be able to switch between the 2 sets. it suppose to be quiet simple, isn't it?
I got a simple switch with 12 legs, the 4 legs in the middle (off) coming from the mixer (L + R), one set of 4 legs (on) for each set of speakers.
It is generally works but when switching to the Dynaudio the NS10's making a noise.

What can it be? Where did i go wrong? Do I need to ground it?