Computer Use Agents (GUI Agents)

A computer use agent is an AI agent that operates software through its screen: the model looks at a screenshot, decides where to click or what to type, and a runtime carries out that action on a real desktop or browser. No integration is needed. Anthropic released the first frontier version in October 2024 and OpenAI followed with Operator in January 2025. It is the slowest and least predictable way for an agent to act, so we treat it as the option for software that offers nothing else.

The screenshot, decide, act loop#

Every GUI agent runs the same cycle. Capture the screen as an image, sometimes with the accessibility tree, the structured list of buttons and fields the operating system keeps for screen readers. Hand it to a vision-language model with the goal and the actions so far. The model returns one concrete action: click at coordinates (412, 308), type "Q2 invoice", press Enter, scroll down. The runtime performs it, waits for the screen to settle, takes a fresh screenshot, and the loop repeats until the model says the task is done.

Walk through "file this expense in the internal portal." Screenshot one shows the dashboard, and the model clicks the Expenses tab. Screenshot two shows a form; it clicks the amount field and types 142.50. Screenshot three shows a validation error because the date is empty, so the model reads the red text, opens the date picker, fixes it and submits. A recorded script cannot recover from a screen it did not expect, and an agent can, but every recovery costs another full step. Each step is a full model call, and as of October 2026 Anthropic's documentation puts one screenshot at roughly 1,000 to 1,800 input tokens, so a 30-action task means 30 round trips and minutes of wall-clock time. The same job through tool calling against an expenses API would be one call.

Why drive a screen when APIs exist#

Because a lot of the software people work in has none. Legacy finance and operations systems, government filing portals, insurance and healthcare systems behind old desktop clients, partner sites behind a login, anything served through a remote desktop: robotic process automation (RPA), recorded scripts that replay clicks, became a large industry on this software before language models existed. We hit the same wall building the executive operations agent. A large share of a leader's week happens in apps behind a company login and a second-factor check, with no usable API, and an agent limited to clean APIs can only finish the part of a routine that happens to be convenient. A routine left 70% done still has a person holding the sequence, so it saves almost nothing. That agent works inside those apps through authenticated browser sessions.

RPA and a GUI agent break in different ways. An RPA script pins each action to a selector or a pixel position, so a moved button stops the bot until someone records it again. A GUI agent reads the screen, so a redesigned layout usually survives, and in exchange the same task can take a different path on every run. A claims processor who re-keys data from PDFs into a desktop app with no integration is a good fit for an agent. A high-volume form that never changes is still better as a recorded script, and better still as a real API.

Benchmark scores and what they leave out#

OSWorld is the standard test: 369 real tasks on live Ubuntu virtual machines, across office apps, browsers, file managers and workflows that span several apps. People complete about 72% of them (72.36% in the paper). The best agent at the April 2024 launch managed 12.24%. Claude 3.5 Sonnet scored 14.9% screenshot-only that October, and 22.0% when given more steps. OpenAI's Computer-Using Agent reached 38.1% at Operator's launch. The maintainers released OSWorld-Verified in July 2025, fixing tasks the community had reported as broken and updating the results, and in February 2026 Anthropic's system card for Claude Sonnet 4.6 reported 72.5% on it, level with the human figure. In June 2026 they released OSWorld 2.0. As of October 2026 the published leaderboards disagree with each other and mix independently verified runs with self-reported ones.

Do not read a human-level score as human-level reliability on your workflow. The loop compounds error: at 97% reliability per step, a 40-step task succeeds about 30% of the time. Time stacks the same way, because every action pays for a model call and for the screen to settle. Scope a computer use task to tens of actions with a result you can check at the end, and measure the success rate on your own task, because none of these benchmarks contains your portal.

Safety: gates, sandboxes and credentials#

An agent with your mouse can do anything you can do, including emptying a folder, sending money or replying to everyone. As of October 2026 Anthropic's computer use documentation asks for a dedicated virtual machine or container with minimal privileges, internet access limited to a list of approved domains, no account logins handed to the model, and a person confirming anything with real-world consequences, such as a payment or accepting terms. We add one rule: sort actions by whether they can be undone, and pause before anything destructive, financial or outbound. That pause is the gate in the diagram above.

Prompt injection, text that smuggles instructions to the model, is why none of this is optional. Everything on the screen is model input, so a malicious email or web page can carry instructions the agent may follow. OS-Harm tested agents on 150 tasks across deliberate misuse, prompt injection and model misbehavior, and found they comply with many deliberate misuse requests and are relatively vulnerable to static prompt injections. Anthropic now runs classifiers that scan screenshots for injected instructions, and that helps, but treat screen content as untrusted the way you treat retrieved documents, and put guardrails outside the model.

Credentials need their own design. On the executive operations agent the passwords sit in an encrypted vault on the user's machine, and the tool layer injects a secret only when an action runs, so the secret never enters the prompt. The agent drives a logged-in session; it never learns a password. If a screen it reads is hostile, there is no credential in its context to leak.

Pick the narrowest surface#

A desktop agent that sees only pixels is the most general option and the least reliable. A browser agent narrows the surface to one application and can read the page's structure as well as its picture: the open source Browser Use library, for example, gives the model a numbered list of the page's interactive elements, so it chooses "click element 14" rather than guessing coordinates. When the work lives in a browser, start there. When a system has any API at all, even a partial one, call that for the steps it covers and keep the screen for the rest.

Check the application, not the agent's report#

A computer use agent decides for itself when the task is done, and nothing in the loop checks that decision unless you build the check. On the executive operations agent we compare the agent's claimed completion against the actual state of the target application, and that comparison catches an agent reporting success on work that did not land. The failures that agent records are not model failures: a login fails, the page has changed since the plan was made, an action is not supported, a tool returns a partial result, or the completion check fails. Each comes back to a person as the one unresolved step with the record of what the agent did attached, so they fix that step instead of auditing the whole run. Build that check before you widen what the agent may do, because it tells you whether the success rate you measured is real.