Teamviewer Host not loading in Raspberry Pi Desktop
So I have a standard Raspbian Buster install on a Rasberry Pi 3B. It boots into the desktop. I installed the latest Teamviewer Host for RPi. It works fine. Yay.
But then I decide I don't want the Pi to boot to the desktop but just to the command line, and then I can start the desktop with "startx". So I change the raspi-config to boot to the command line and reboot.
I check Teamview from another computer and it is working fine, I can see the command line interface.
Then I start the desktop. Once the desktop is up, Teamviewer stops working. All I see on the remote computer is a frozen desktop screen. Locally, the desktop is working fine.
I tried running the host startup from the desktop menu, and it gives me the cursor busy signal for a few seconds then goes back to normal. But it still isn't working on the remote computer.
If I exit the desktop back to the command line, then TV host works again.
If I change the configuration to boot into the desktop directly again, then everything works fine in the desktop with the host.
Can anyone help?
Summary: RPI booting into desktop and Teamviwer GUI works.
RPI booting into command line, Teamviewer works until you start desktop then it stops.
Edited to add: If I log out of the Desktop session (it is still in the desktop GUI asking me to log back in) and then log back in, Teamviewer also stops working. On the remote computer it just shows that it is trying to connect (Initializing Display...) and then eventually fails and closed the connection window with no error showing up.