Initializing display parameters fails when using HDMI switch on computer output

 I've been using TeamViewer for the past year to control my media center computer (an Intel NUC running Ubuntu), which outputs to HDMI2 on my plasma TV.  The setup was very simple: on HDMI1 we watch cable, on HDMI2 we watch whatever is coming from the computer.  If we're watching cable, I can use TeamViewer to connect to and configure/update/play with my media center computer.

I want to add some additional HDMI components to my setup (a ChromeCast) so I added an HDMI switch between the media center computer and my TV (to HDMI2). However, TeamViewer now hangs and eventually fails (after 91 seconds) during the "Initializing Display Parameters" step if my TV isn't showing the output from the media center computer.  It doesn't matter which input is selected on the HDMI matrix switch, if the content isn't being shown on the TV then TeamViewer can't connect to it.  I would think that if I set the HDMI switch to the media center computer then that output would be sent to my TV's HDMI2 and I would be able to connect with TeamViewer even if I've set the TV to HDMI1, but that's not the case.  If I disconnect the HDMI cable from my media center computer then I CAN connect.  

Any idea why the HDMI switch could be interfering with TeamViewer's connection capabilities?

 Eric