Screen & screenshots

Reading the screen back is how a test asserts on what the app showed. Two shapes: a one-shot PNG, and a continuous frame stream.

Screenshot

device-bridge screenshot [--out shot.png] [--display N] [--window ID]

wrote shot.png (3456x2234, 4812901 bytes)

Flag	Default	Meaning
--out	shot.png	output PNG path
--display	0	display index; 0 is the main display
--window	0	CoreGraphics window id; 0 means the whole display

Spike S5 measured a 1920×1080 capture at 110 ms. That is a cold one-shot and the cost is dominated by ScreenCaptureKit setup — if you are taking more than a couple, use the stream instead.

The video-out stream

device-bridge start video-out --port 9334 --token "$WINDOWCTL_TOKEN"

One persistent SCStream is started per connected client and torn down when that client disconnects. Every frame arrives as one binary WebSocket message: a 16-byte header then the raw pixels.

• Kind = 4 (screen)
• Format = 11 (packed BGRA — the native macOS capture format, no conversion)
• Length = payload bytes

Standalone defaults are the display’s native size at 10 fps. To pick a resolution and frame rate, go through the control plane and negotiate with VideoParams on start.

Measuring your own latency

stamp in the header is a producer monotonic millisecond counter. Subtract it from your own monotonic clock on receipt and you have the transport cost with no wall clock and no clock-sync assumption. That is exactly how spike S0 arrived at 5.6 ms p50 for a 3.6 MB 720p frame.

Bandwidth is the real limit, not latency

Raw BGRA is 4 bytes per pixel. 1080p60 is roughly 480 MB/s, which is past what a WebSocket on loopback should be asked to carry. ADR 0002 anticipated this and keeps two escape hatches: a shared-memory ring buffer for the same-machine fast path, and JPEG (Format = 12) for remote. At 30 fps and 720p, neither is needed.

Screen Recording permission

ScreenCaptureKit is gated behind Screen Recording in System Settings. The permission flow works and the error is explicit when the grant is missing. As with Accessibility, the grant follows the calling process.

Video-in is a different thing entirely

Capturing the screen is reading. Feeding a camera is writing, and that lane — video-in, the virtual camera — is not built. It needs a CoreMediaIO system extension (designed in ADR 0006, feasibility-confirmed in spike S4) plus a signing and user-approval ceremony. device-bridge install camera says so and exits non-zero rather than pretending.