How a Tap on Your iPhone Becomes a Click on Your Mac
Sending a click is easy. Sending thousands of them over a bad connection, in order, without making the cursor lag, is where remote control gets interesting.
Positions are fractions, not pixels
When you touch the picture, the phone does not send "pixel 812, 640". It sends a point between 0 and 1 on each axis, relative to the source being shown. The Mac turns that into real coordinates for the display or window it is capturing. This is why a tap lands in the right place whether the picture is fit to the screen or zoomed, in portrait or landscape, on a Retina panel or a virtual display shaped like the phone. Neither side needs to agree on a resolution.
Not every message is equal
Input travels as small JSON commands on the encrypted control connection, and the app treats each kind differently:
- Mouse moves keep only the latest. If three moves queue up while the network stalls, the first two are worthless. The newest one replaces them, and moves are limited to about 60 a second.
- Scrolls add up. Two scroll events that queue together become one event with the sum, so no distance is lost.
- Clicks and keys are never dropped or reordered. A mouse up that arrives before its mouse down would be a bug you could see.
That small policy is most of the difference between a cursor that glides and one that lurches after a hiccup. It also means a bad connection degrades into a less smooth pointer and never into a wrong action.
Keys are macOS key codes
A key press is a macOS virtual key code plus modifier bits for shift, control, option and command. The Mac app converts it into a real keyboard event and posts it to the system, which is why it needs the Accessibility permission. Shortcuts follow the app you are in: the panel above shows the commands for the editor in front, not a fixed list.
Getting keys to behave like a physical keyboard has surprising edges. A real arrow key carries two extra flags, a function flag and a numeric pad flag, that a naive synthetic event leaves out. Without them, Control and Up arrow did nothing on the Mac instead of opening Mission Control, even though the right key code had arrived. Peering now sets both flags, as a real keyboard does.
Text is not key presses
Emoji, accents and non-Latin scripts do not map to key codes. For those the phone sends the characters themselves as text, and the Mac types them in. That is why the keyboard works in any language, and why a few Mac shortcuts do not work in that mode. It is a trade the app lets you switch in Settings.
Staying in sync
The phone pings the Mac every second, and the Mac answers by echoing the ping, which doubles as a live latency reading. If the phone goes to the background, it says so first, because iOS can freeze an app without closing its connections, and the Mac then stops sending video to it until it returns. A session can also be set to observe only, in which case the Mac ignores input while the picture keeps flowing.
The other direction of this loop is how the picture gets to the phone. Together they decide how responsive the whole thing feels.
Questions
Does Peering support emoji and non-English keyboards?
Yes. Text that is not a key press is sent as characters and typed on the Mac, so emoji and any language work. Mac shortcuts are only sent in the mode that sends key codes.
Why does the cursor stay smooth on a weak connection?
Mouse moves are coalesced so only the latest position is kept, and scrolls are summed. Clicks and keys are always delivered in order.