Building with the Claude API · lesson 8 of 17

Computer Use (Desktop Control)

Computer use is a beta tool that gives Claude screenshot capture, mouse control, and keyboard input over a desktop environment — Claude operates software the way a person does (look at the screen, move the cursor, click, type) instead of through that software's API. Claude 3.5 Sonnet was the first frontier model to offer it (public beta, October 2024). It is a beta feature and requires a beta header.

The agent loop

Computer use is regular tool-use with a screenshot feedback cycle. You never let Claude touch the machine directly — your application executes each action and returns the result:

1. Send the computer tool + a desktop task in the request
2. Claude replies with stop_reason: "tool_use" and an action (e.g. left_click at [x,y])
3. Your app runs the action on a VM/container, takes a screenshot
4. Reply with a tool_result block whose content includes that screenshot (image block)
5. Repeat until Claude answers with no tool_use (done) — or a max-iteration cap trips

Agent loop — the repeat-until-done cycle of executing Claude's tool_use request and returning a screenshot as a tool_result; needs a max-iteration cap to prevent runaway API cost.
Client-side tool — Claude never connects to the environment; your app implements screenshot capture and input and returns results. Because inputs/screenshots stay in your environment, computer use is ZDR-eligible.
Schema-less (trained-in) tool — you don't supply an input_schema; the schema is built into the model, which is why Claude calls it more reliably than a hand-rolled equivalent.

Versions, models, beta headers

Two versions are current; pass the matching beta header:

Tool `type`	Beta header	Models
`computer_20251124`	`computer-use-2025-11-24`	Opus 5, Sonnet 5, Opus 4.8, Opus 4.7, Opus 4.6, Sonnet 4.6, Opus 4.5
`computer_20250124`	`computer-use-2025-01-24`	Sonnet 4.5, Haiku 4.5, Opus 4.1, Sonnet 4, Opus 4

computer_20251124 — the current-generation computer tool; adds the zoom action.
computer_20250124 — the prior version; introduced the "enhanced" action set (scroll, drag, right/middle click, etc.).
The original October-2024 version (computer_20241022, with a cursor_position action) is superseded and no longer in the current docs.

The three tools

The reference setup composes three Anthropic-schema tools in one tools array; only the computer tool needs the beta header:

computer — screenshot + mouse + keyboard desktop control (the capability this note is about). Requires the beta header.
bash (bash_20250124) — run shell commands. GA, optional augmentation.
text_editor (text_editor_20250728, name str_replace_based_edit_tool; text_editor_20250124 for pre-Claude-4) — view/edit files. GA, optional augmentation.

Reach for bash/text_editor for terminal and file work instead of driving a terminal through the GUI — they are trained-in and more reliable for those operations.

Actions

Set on the computer tool via input.action; coordinates are coordinate: [x, y].

Basic (all versions) — screenshot, left_click, type, key, mouse_move.
Enhanced (computer_20250124+) — scroll (with scroll_direction/scroll_amount), left_click_drag, right_click, middle_click, double_click, triple_click, left_mouse_down, left_mouse_up, hold_key, wait.
zoom (computer_20251124 only) — inspect a region: [x1,y1,x2,y2] at full resolution; requires enable_zoom: true in the tool definition. Useful for small or illegible text.
Modifier keys — pass shift/ctrl/alt/super via a text param on the click/scroll action (distinct from hold_key, which holds a key for a duration).

Required tool params: type, name ("computer"), display_width_px, display_height_px. Optional: display_number (X11), enable_zoom.

Setting it up

Building your own environment needs four pieces (Anthropic ships a reference implementation, anthropic-quickstarts/computer-use-demo, that provides all four as a starting point — a Docker container with Xvfb/Mutter/Tint2, a loop.py agent loop, and a web UI):

Virtualized/containerized environment — a sandboxed desktop suitable for Claude to drive.
Tool implementation — at least one Anthropic-schema computer-use tool wired to real screenshot/input.
Agent loop — code that calls the API and executes returned tool_use results.
Entry point — an API or UI that starts the loop from user input.

Resolution, tokens, and cost levers

Accuracy and cost both hinge on screenshot size — the main levers a practitioner controls:

Resolution — use XGA 1024x768 or 1280x720 for general desktops (1280x800/1366x768 for web apps); avoid resolutions above 1920×1080 — higher res hurts accuracy and cost. Consistently poor click accuracy usually means resolution is too low.
Image caps & scaling — long-edge limit is 2576px (Opus 5, Sonnet 5, Opus 4.8/4.7) vs ~1568px / ~1.15MP on earlier models; resize oversized screenshots and scale Claude's returned coordinates back up. On Retina (device-pixel-ratio 2) displays, halve the coordinates or downscale 2×.
Token overhead — the beta adds ~466–499 system-prompt tokens plus ~735 tokens per tool definition (Claude 4.x); each screenshot costs ~1,000–1,800 input tokens. Prune old screenshots in batches (e.g. keep the last three) and use cache_control to preserve the cache prefix.
Pricing — computer use follows standard tool-use pricing: ordinary input/output tokens plus the overhead above. No surcharge.
Thinking effort — a cost/quality lever (extended-thinking); Anthropic's guidance is roughly high for Opus 4.7, medium for Sonnet 4.6 / Opus 4.6, and avoiding max (adds cost without improving UI-task accuracy).

When to use it (and when not to)

Prefer computer use when the target has no API or structured integration and only a GUI, for cross-application GUI workflows (read a spreadsheet, drive a browser, fill a form), automated software testing, or open-ended research/RPA — and speed is not critical and the environment is trusted.
Prefer a defined tool, a direct API, or an MCP server when you're calling into existing systems (databases, internal APIs, filesystems), need structured guaranteed-shape output, need reliability and graceful error recovery, need low latency, or the task demands perfect precision or touches sensitive data. A trained-in schema is more reliable than pixel-level control.
Availability: the computer-use beta runs on the Anthropic API, Amazon Bedrock, and Google Cloud Vertex AI.

Safety and prompt injection

Computer use carries risks "distinct from standard API features," heightened on the open internet:

Prompt injection — text on a webpage or inside a screenshot can override your instructions; Claude will sometimes follow content-embedded commands that conflict with yours.
Classifier defense — when computer-use tools are in play, classifiers automatically flag potential prompt injection in screenshots and steer the model to ask for user confirmation before the next action (opt-out via support for no-human-in-the-loop cases). It is a backstop, not a substitute: "these precautions remain important even with the classifier defense layer in place."
Precautions (from the docs) — run in a dedicated VM/container with minimal privileges; avoid giving the model sensitive data or login credentials; limit internet access to an allowlist of domains; and ask a human to confirm consequential or consent actions (accepting cookies, financial transactions, agreeing to terms).
Consent & guardrails — developers must inform end users and obtain consent; account creation and social-media posting are deliberately limited (human-impersonation guardrail).
Data handling — a client-side tool: screenshots/inputs/files stay in your environment, so it is ZDR-eligible: Yes but HIPAA-eligible: No (beta features aren't covered by the BAA). Flagged sessions may still be retained up to two years.

Limitations

Anthropic frames the feature as experimental — "at times cumbersome and error-prone":

Latency — the screenshot loop can be too slow for real-time human-AI interaction; best for non-time-critical tasks.
Coordinate hallucination — Claude may output wrong coordinates; reliability drops on niche apps or several apps at once.
Flaky actions — scrolling can silently fail (prefer keyboard alternatives); complex spreadsheet operations may need several attempts; the "flipbook" screenshot view can miss short-lived events.
Benchmark anchor — on OSWorld, Claude 3.5 Sonnet scored 14.9% screenshot-only (22.0% with more steps) vs 7.8% for the next-best system at launch; current models are substantially higher, but the exact figures live in the models' system cards.

Related but distinct

Claude for Chrome — a separate consumer browser extension (beta, Max plan) that acts in the user's own browser; the computer-use tool drives an isolated desktop VM via the API. Both operate a UI and share prompt-injection risk, but they are different surfaces.
The memory tool — another Anthropic-schema client-side tool (memory_20250818) for cross-session memory under /memories; same tool family as bash/text_editor, unrelated to desktop control.

Key terms

Computer use — the beta tool giving Claude screenshot + mouse + keyboard control of a desktop to operate software via its GUI instead of an API.
Agent loop — the execute-action → return-screenshot cycle that drives computer use, bounded by a max-iteration cap to cap cost.
Client-side tool — a tool your application executes (Claude never touches the environment); screenshots stay in your environment, making computer use ZDR-eligible.
Schema-less / trained-in tool — an Anthropic-schema tool (computer, bash, text_editor, memory) with no caller-supplied input schema; called more reliably because the schema is trained into the model.
Beta header — computer-use-2025-11-24 (for computer_20251124) or computer-use-2025-01-24 (for computer_20250124), required to use the computer tool.
Enhanced actions — the action set added in computer_20250124 (scroll, drag, right/middle/double/triple click, hold_key, wait) beyond the basic screenshot/click/type/key/mouse_move.
zoom action — a computer_20251124 action that inspects a screen region at full resolution; requires enable_zoom: true.
Prompt-injection classifier — the defense that flags injected instructions in screenshots and steers Claude to ask for user confirmation before acting.
Computer-use precautions — dedicated VM/min privileges, no sensitive credentials, domain allowlist, and human confirmation for consequential/consent actions.
OSWorld — the desktop-agent benchmark; Claude 3.5 Sonnet's launch scores (14.9% / 22.0%) are the grounded historical anchor for computer use.