Teamviewer & RDP conflict on remote PC



I've got a remote PC I'm connecting to using TeamViewer. At the same time there's another user sometimes connecting to that remote PC using RDP and Windows 10 standard remote desktop. 

When the second user is logged in via RDP and I connect to the PC with Teamviewer - I'm getting into his RDP session instead of the main session on that PC. 

I'd like to avoid this behaviour, always connecting to just the main session on remote PC and ignoring any RDP sessions that might be running - is it possible somehow?