Stop asking for prose when the machine needs a branch.
The typesafe-computer-use project reads a Mac screen, constructs candidate actions, asks a classifier to choose one, and uses a separate writing model only when a field needs free text. Its README says the loop stops on low confidence, repeated no-ops, a step ceiling, an explicit done state, Ctrl-C, or a mouse movement to the top-left corner.[1]
That split matters. A screenshot-to-prose model can invent an action, explain it, and blur uncertainty into a sentence. A typed chooser has to select from the options the surrounding program supplied. The program can reject low confidence and log the exact candidate set.
The benchmark is a lead, not a purchase order.
The repository author reports about $0.0002 per classifier decision, 0.13 to 0.38 seconds of model latency, and about 1.5 seconds for an end-to-end step with capture and OCR. The comparison uses one project, one task setup, and author-run measurements. The README also says deterministic date parsing had to replace reasoning that a larger model handled from pixels.[1]
So do not turn the numbers into a universal price sheet. Treat them as a useful experiment. Moving judgment into a narrow classifier can cut cost and delay, but the missing reasoning does not vanish. Engineers rebuild it as OCR, parsing, candidate generation, state tracking, and stop policy.
The model got smaller because the contract got sharper. The rest of the intelligence moved into code you can inspect.
Typed output is only one rail.
TypeSafe's own documentation describes its Jev model as a structured decision system. Its Choice primitive returns a selected option, probabilities, and confidence. The docs recommend atomic questions and tell developers to combine separate judgments in code instead of asking one prompt to weigh several concerns.[2]
That creates a better seam for testing, but it does not certify the screen reader, the candidate generator, the click target, or the goal. A perfectly typed wrong choice is still wrong. The useful unit is the full loop: observed state, allowed actions, decision, execution boundary, and stop receipt.
Screen permission is data permission.
Apple says users choose which apps and websites may record the screen and system audio. Apple also warns that information collected by third parties is governed by those parties' terms and privacy policies.[3] A local capture step does not make later processing local. A product should say what leaves the Mac, which service receives it, and what the no-permission path can still do.
The project README says the terminal needs Screen Recording and Accessibility permissions to capture and act. Those are separate powers. Keep them separate in the interface too. Let a dry run inspect and propose without clicking. Make live action a deliberate mode with a visible limit and an outside stop.
Build the loop as four replaceable parts.
- Observe: name the exact screen data collected and where it is processed.
- Constrain: generate a short, inspectable list of allowed actions.
- Gate: record probabilities, set a confidence floor, and define the low-confidence path.
- Stop: cap steps, detect no-ops, expose a deadman control, and preserve a receipt.
Now you can swap OCR without changing the action policy. You can test candidate generation without granting Accessibility. You can replay decisions without touching the desktop. That is a real control surface, not a chat bubble wearing driving gloves.