How Claude’s Computer Use Tool Controls a Screen

The idea of an AI agent watching a screen and operating software the way a person does is no longer hypothetical. Anthropic ships a computer use tool that lets Claude take screenshots, move the cursor, click, and type. That same set of actions is what would be needed to drive any visual application, including a game that exposes no API of its own.

What the Tool Actually Does

According to Anthropic’s documentation at https://platform.claude.com/docs/en/docs/build-with-claude/computer-use, the computer use tool gives Claude four capabilities. It can capture screenshots to see what is currently displayed, control the mouse to click and drag and move the cursor, send keyboard input to type text and use shortcuts, and through those actions interact with any application or interface on the desktop.

The tool can be combined with other tools, such as a bash tool and a text editor tool, for broader automation workflows. On WebArena, a benchmark for autonomous web navigation across real websites, Anthropic reports that Claude achieves state-of-the-art results among single-agent systems on multi-step browser tasks.

This is the mechanism behind the broader notion of an AI that observes a screen and acts on it. Rather than reverse-engineering an application’s internals, the model perceives the visual interface and issues the same inputs a human operator would.

Beta Status and Model Support

Computer use is a beta feature and requires a beta header on each request. The documentation lists computer-use-2025-11-24 for newer Claude models including Claude Opus 4.8, and an earlier computer-use-2025-01-24 header for models such as Claude Sonnet 4.5 and Claude Haiku 4.5. The feature is also eligible for Zero Data Retention arrangements, meaning data sent through it is not stored after the API response is returned for organizations with that arrangement.

A request defines the computer tool with a display width, height, and display number so the model knows the dimensions of the screen it is operating. The documentation’s quick start shows the tool declared alongside optional text editor and bash tools, with a plain-language instruction such as saving a picture to the desktop driving the session.

The Agent Loop and Reference Implementation

Anthropic points developers to a computer use reference implementation hosted on GitHub. According to the documentation, it includes a web interface, a Docker container, example tool implementations, and an agent loop. The agent loop is the part that repeatedly feeds Claude the current screen state, receives the next action, executes it, and captures a fresh screenshot so the model can react to the result.

Running the tool inside a Docker container reflects the security posture Anthropic recommends. The documentation advises using a dedicated virtual machine or container with minimal privileges, avoiding access to sensitive data such as login credentials, limiting internet access to an allowlist of domains, and asking a human to confirm consequential actions.

Prompt Injection and Other Risks

Anthropic is direct about the limitations and dangers. Claude can sometimes follow instructions found in on-screen content, including text on web pages or inside images, even when those instructions conflict with the user’s request. To counter this, the model is trained to resist prompt injection, and classifiers automatically run on prompts to flag potential injection attempts. When a classifier detects a likely injection in a screenshot, it steers the model to ask for user confirmation before continuing.

These defenses do not remove the need for isolation and human oversight. The documentation frames computer use as carrying risks distinct from standard API features, heightened when the agent touches the internet, and recommends informing end users and obtaining consent before enabling it in a product.

The practical takeaway is that screen-driven automation with Claude is real and documented, but it is a beta capability meant to run in sandboxed environments with a human in the loop, not an unattended bot turned loose on arbitrary software.