Computer Use & GUI Agents

Computer use is the capability that lets AI agents operate software with no integration at all: the model looks at a screenshot, decides where to click or what to type, and a runtime executes that action on a real desktop or browser. Anthropic shipped the first frontier version in October 2024, OpenAI followed with Operator in January 2025, and by mid-2026 the pattern has gone from demo to deployable for narrow workflows. It is the bluntest tool in the agent toolbox, and the right one exactly when nothing cleaner exists.

The screenshot, decide, act loop#

Every GUI agent runs the same cycle. Capture the screen as an image (sometimes plus an accessibility tree). Hand it to a vision-language model along with the goal and the action history. The model emits one concrete action: click at coordinates (412, 308), type "Q2 invoice", press Enter, scroll down. The runtime executes it, waits for the UI to settle, takes a fresh screenshot, and the loop repeats until the model declares the task done.

Walk through "file this expense in the internal portal." Screenshot one shows the dashboard; the model clicks the Expenses tab. Screenshot two shows a form; it clicks the amount field and types 142.50. Screenshot three reveals a validation error because the date is empty; the model reads the red text, clicks the date picker, fixes it, and submits. That self-correction from pixels is the whole trick, and also the whole cost: each step is a full model call on an image of roughly a thousand input tokens, so a 30-action task means 30 round trips and several minutes of wall-clock time. The same job through tool calling against an expenses API would be one call.

Why drive a GUI when APIs exist#

Because mostly they do not. The software economy runs on a long tail of systems with no public API: legacy ERPs, government filing portals, insurance and healthcare systems fronted by 20-year-old desktop clients, partner sites behind logins, anything served through Citrix. A consultancy rule of thumb: the workflows clients most want automated are precisely the ones stuck in this tail, which is why robotic process automation became a multibillion dollar industry before LLMs existed.

GUI agents differ from classic RPA in what breaks them. RPA scripts pin actions to brittle selectors and pixel offsets; move a button and the bot dies until a developer re-records it. A GUI agent reads the screen semantically, so a redesigned layout usually survives, but in exchange you give up determinism: the same task can take a different path on every run. The honest framing is that computer use trades RPA's fragility for stochasticity. A claims processor that re-keys data from PDFs into a mainframe-fronted desktop app is a good fit; a high-volume, never-changing form is still better served by a recorded script or, best of all, an actual API.

Reliability and latency, the honest numbers#

The standard yardstick is OSWorld: 369 real tasks on a live Ubuntu VM spanning LibreOffice, browsers, file managers, and multi-app workflows. Humans complete 72.36%. When the benchmark launched in April 2024, the best agent managed 12.24%. Claude 3.5 Sonnet hit 14.9% in the screenshot-only setting that October (22% with more attempts allowed); OpenAI's Computer-Using Agent reached about 38% at Operator's launch in January 2025. By mid-2026, frontier agents score in the low to high 70s on OSWorld-Verified, the tightened July 2025 revision that closed loopholes agents had learned to game. Claude Sonnet 4.6 reports 72.5%, essentially the human baseline, with newer flagship models claiming several points above it.

Matching humans on a benchmark does not mean matching them in production. The loop compounds error: at 97% per-step reliability, a 40-step task succeeds about 30% of the time. Latency stacks the same way; with a couple of seconds of model time plus UI settle time per action, real tasks run 2 to 15 minutes, and screenshots make every step expensive in image tokens. The practical consequence in 2026: computer use works for tasks measured in tens of actions with a clear success check at the end, not hundreds.

Safety: gates, sandboxes, and scoped credentials#

An agent with your mouse can do anything you can do, including the things you would never do: empty a folder, wire money, reply-all. Three controls have become table stakes. First, confirmation gates: classify actions by reversibility and pause for human approval before anything destructive, financial, or outbound (the gate in the diagram above). Second, sandboxing: run the agent in a dedicated VM or browser profile that is not your session, so a wrong click is contained; Anthropic's own docs tell you to assume this posture. Third, scoped credentials: give the agent its own low-privilege account with spending caps and a domain allowlist instead of your logged-in cookies.

The threat that makes this non-optional is prompt injection. Everything on screen is model input, so a malicious email or web page can carry instructions the agent may obey. The OS-Harm benchmark (150 tasks across misuse, injection, and misbehavior categories) found that frontier agents comply with many deliberate misuse requests outright and remain vulnerable to even static prompt injections. Treat screen content as untrusted, the same way you treat retrieved documents, and layer guardrails outside the model rather than trusting it to police itself.

The current landscape#

Three families dominate. Anthropic's computer use tool (beta since October 2024, enabled via a beta header) gives Claude screenshot, mouse, and keyboard primitives against any desktop, usually paired with bash and file tools; the reference setup is a containerized Linux VM. OpenAI's lineage ran from the CUA model behind Operator, through Operator's absorption into ChatGPT agent in July 2025, to computer use capabilities exposed through its developer platform; the standalone Operator surface shut down in August 2025. Browser agents narrow the scope to a single app, the browser, and exploit it: tools like the open source Browser Use framework (about 98k GitHub stars) combine screenshots with the DOM and accessibility data, which makes actions cheaper and grounding more reliable than raw pixels. The pattern generalizes: the narrower the surface, the better today's agents perform on it.

Practical takeaways#

Reach for computer use last, after confirming no API, export, or database path exists, because every alternative is faster, cheaper, and more deterministic. Scope tasks to tens of steps with a verifiable end state, and measure success rates on your own workflow rather than trusting OSWorld headlines. Run the agent in a sandbox with its own scoped, low-privilege credentials from day one. Gate irreversible actions on human confirmation and treat all screen content as untrusted input. And where the work lives in a browser, use a browser-native agent before a general desktop one; the smaller surface buys you real reliability.