VM Guest access w/o allowing access to host

If I allow remote access to a Virtual Machine via TV, can I restrict access to the host for that remote viewer? Maybe this is not a TV-related setting?