How can I restrict access for TeamViewer connections to my computer?

Hello,
 Is it possible to set exclusive access to a PC for Remote Control & VPN?

Example, I only want myself to be able to remote control my own PC, whether that's from my phone or from my work. Because it's 100% personal, I only use it for myself to access personal data remotely, and surf from work to circumevent company Firewall.

VPN is only for family & friends, to access my shared network drives.
But I do not want them accidentally remoting into my Desktop.

Is there any possible way of splitting this up?