RDP is disconnected when connecting to a Teamviewer session

I am running Teamviewer from my computer at work. When I log into this from home, I RDP to it through a VPN-tunnel.

Whenever I connect to a Teamviewer session the RDP connection breaks quickly and when it reconnects after a few seconds the PC I am trying to remote via Teamviewer is open. 

Anything I can do to stop the short RDP break?