How Peering Streams Your Mac's Screen to Your iPhone
A remote desktop is a video stream with one unusual requirement: the person watching is also the person driving. Every design choice below follows from keeping that loop short.
Capture: ask the system for the picture
The Mac app uses Apple's ScreenCaptureKit to get frames. The source can be a whole display, every window of one app, or a single window, and you can switch between them without dropping the connection. This is also why macOS asks for the Screen Recording permission the first time: without it the Mac can only hand over a black picture.
Encode: low latency, no reordering
Frames go to Apple's VideoToolbox encoder on the Mac, as H.264 or HEVC, whichever the phone says it can decode. The settings are the opposite of what a streaming service uses:
- No B-frames. A B-frame needs a frame from the future, so using them means holding pictures back. Without them, decode order is display order and nothing waits.
- Low latency rate control. The encoder spends its bits as it goes instead of looking ahead.
- A keyframe at least every two seconds, and immediately when a new viewer joins, resumes or switches source, so a lost picture heals quickly.
- A bitrate that scales with the screen. By default it is the pixel count times the frame rate times 0.12, held between 4 and 30 Mbps, at 60 frames a second by default.
The decoder's setup information (the parameter sets) is sent once, as its own message, and not repeated in every frame. A phone that joins mid-stream receives that message and then a keyframe.
Send: droppable frames, protected keyframes
Each frame travels with a small 25 byte header: whether it is a keyframe, its pixel size and a presentation time. On a relayed or direct connection it is cut into datagrams of about 1,080 bytes, each one sealed with ChaCha20-Poly1305 using a key only the two devices know. A counter in every datagram feeds a 64 entry replay window, so a captured packet cannot be played back later.
The rule for congestion is blunt: video is droppable and keyframes are not. If a new frame is ready while an old one is still waiting to go out, the old one is replaced, except that an undelivered keyframe is never replaced by an ordinary frame. Showing the screen as it is now matters more than showing every frame that ever existed.
On a home network the picture travels over its own TCP connection instead, where the network
is reliable enough that the same rule rarely has to fire. Every socket sets TCP_NODELAY.
The default behaviour of batching small writes adds tens of milliseconds at each hop, which is
invisible in a download and very visible when you are moving a cursor.
Recover: pay for loss in pictures, not in latency
UDP does not retransmit, so a lost datagram means a damaged frame. The phone does not wait for it. It keeps decoding, and the damage clears at the next keyframe, at most two seconds away. If you notice a smear after a bad bit of Wi-Fi, that is what you are seeing. Pausing and resuming the picture forces a keyframe immediately.
If the fast path stops delivering altogether, the stream falls back to the control connection at a lower bitrate, as the connection post describes, and returns when the fast path recovers.
Show: fit the phone, not the Mac
The phone chooses a quality tier and can ask for fewer frames per second, which the Mac clamps to a sensible range. The app can fit the whole desktop on the screen or fill it, and even ask the Mac for a virtual display shaped like the phone, so a landscape monitor is not squeezed into a portrait screen. Taps on the picture travel the other way, which is the subject of how a tap becomes a click.
Questions
What video codec does Peering use?
H.264 or HEVC, encoded on the Mac with VideoToolbox, with no B-frames and low latency settings. The iPhone tells the Mac which it can decode.
Why does the picture sometimes smear for a moment?
A lost network packet damages a frame, and the picture heals at the next keyframe, which arrives at least every two seconds. Peering prefers a brief smear to waiting for a retransmission.