Midi only uses pins 4 and 5 for signals, and pin 2 is connected at one end of the cable as a shield, so you can probably just connect pin 2 to a common point for all the sockets.
Normally i don't think you ground pin 2 of an input socket, but you do on an output socket. Having said that, a switcher will be acting as an extension of a midi lead, so unless someone knows better i'd say connect all the pin 2's together inside.
You can probably get by with a 2 pole rotary switch.