Since it was mentioned regarding macOS 10.13 High Sierra, you may wish to look through the new developer documentation for the new Metal screen capture API. It allows for capturing both image data and command queues. It may prove to be useful for countering the serious frame rate drops I currently experience when TeamViewer is capturing either of my screens, one of which is the 5k iMac's main 5120x2880 screen. Something to optionally support for newer systems, maybe?
Okay, so Metal capture is just for debugging purposes. Maybe AVCaptureScreenInput is what you're already using? Too bad it has such a high GPU utilization impact.
It would appear that TeamViewer uses the CGWindow API (CGWindowListCreateImage) for desktop capture on macOS, which was definitely not designed for high speed capture. There is a better API, AVCaptureScreenInput, which has been around since macOS 10.7.
Or you could use CGDisplayStream. 10.8+.