Building a Computer Use Agent: A Practical SOP from Sandbox to Safety Guardrails

Traditional agents work by calling APIs -- you first build them tool interfaces. Computer Use takes a different path: the model takes a screenshot to "see" the current screen, then outputs mouse coordinates and keyboard commands to click buttons, fill forms, and switch windows, operating the graphical interface just like a human would. In theory, any website or desktop app you can use, the agent can use too.

This route was pushed forward by Anthropic's Computer Use capability in October 2025, and over half a year later it is no longer a single-vendor game (see the site's Computer Use Era Hotspot). But there is a gap between "can use" and "dare to use": the ability to operate a computer means the ability to cause harm. This SOP gives you a repeatable build flow: sandbox setup -> wire up Computer Use API -> main loop -> safety guardrails -> tuning, five steps. Code is verified against Anthropic's official documentation and the trycua/cua repository, current as of 2026-08-11, with APIs subject to the official source. It complements the batch's Computer Use Agent Comparison (which solution to pick) and cua Open Source Project Analysis (deep dive into sandbox internals): those answer "what and why," this one answers "how to build it."

1. Computer Use Is Not Tool Calling

Let us clear up a common confusion first. The site's Tool Calling SOP covers function calling / tool use -- the model produces structured JSON (which function to call, what parameters to pass), your code executes and feeds back the result. The prerequisite for that route: the software has an API exposed for you.

Computer Use needs no API. The model directly "looks at" a screen screenshot, understands the interface state, then outputs an action: move the mouse to coordinate (x, y), left-click, type text, press a keyboard shortcut. After execution, a new screenshot is taken, the model looks again and decides again, looping until the task is done or a guardrail fires. This is GUI grounding, not API calling.

Core differences:

In one sentence: tool calling is "the software paves the road for you," Computer Use is "no road, I blaze my own trail." The latter is more universal but slower, pricier, and harder to control. Which route to pick depends on whether the target system has an API.

2. The Five-Step SOP

Step 1: Environment and Sandbox Setup

First principle: never let a computer-use agent operate your production machine directly. The ability to operate a computer means the ability to cause harm. Isolate with a sandbox.

trycua/cua provides an out-of-the-box sandbox supporting Linux/macOS/Windows, locally via QEMU or in the cloud:

pip install cua  # requires Python 3.11+

from cua import Sandbox, Image

# Create an ephemeral Linux sandbox (disposable, isolated from host)
async with Sandbox.ephemeral(Image.linux()) as sb:  # also .macos() .windows()
    await sb.shell.run("echo hello")       # run command
    shot = await sb.screenshot()            # take screenshot
    await sb.mouse.click(100, 200)          # click coordinate
    await sb.keyboard.type("Hello!")        # type text

You can also spin up a Docker container with a VNC-enabled Linux desktop, or use cua's Lume component to run a macOS VM (based on Apple Virtualization.Framework). Core requirements: the sandbox has a graphical desktop, can take screenshots, can simulate mouse and keyboard, and is physically isolated from your real environment.

Step 2: Wire Up the Computer Use API

Anthropic's computer use goes through the beta channel. Three things: tool definition, beta flag, model selection.

import anthropic

client = anthropic.Anthropic()  # ANTHROPIC_API_KEY from environment variable

# Tool definition: tell the model "you have a 1024x768 computer to control"
tools = [{
    "type": "computer_20250124",       # tool type (versioned)
    "name": "computer",
    "display_width_px": 1024,          # screen width (keep <=1024, larger = costly and slow)
    "display_height_px": 768,
}]
BETAS = ["computer-use-2025-01-24"]    # beta flag

Key fields: type must be a versioned tool type (computer_20250124 or newer); display_width_px / display_height_px define the model's coordinate space -- coordinates the model outputs are based on this size; the official recommendation is not to exceed XGA (1024x768) or WXGA (1280x800), as higher resolutions increase the token cost per screenshot. The model is called via client.beta.messages.create() with the betas parameter. You must use a model version that supports computer use -- check the official docs.

Step 3: The Main Loop (Screenshot - Decide - Act)

This is the heart of the entire agent. The model takes a screenshot -> looks and decides -> outputs an action -> you execute -> take a new screenshot -> look again and decide again, looping until the task is complete or a guardrail triggers.

The actions the model outputs (the input field of the tool_use block) contain an action and accompanying parameters. Actions supported by computer_20250124: screenshot (take screenshot), mouse_move (move to coordinate), left_click / right_click / double_click / triple_click (click), left_click_drag (drag, requires start and end coordinates), type (input text), key (key combination like ctrl+c), scroll (scroll), wait (wait), cursor_position (get cursor position).

Loop logic: call API -> check stop_reason, if it is not "tool_use" the model considers the task done -> otherwise iterate tool_use blocks and execute actions -> feed results back as tool_result -> call API again. Full code in Section 3.

Step 4: Safety Guardrails (Whitelist + Human Confirmation)

An agent that can operate a computer must have guardrails. This is a red line, not a suggestion. Three lines of defense:

First line: action whitelist. Only allow the agent to execute a predefined set of safe actions. For example, restrict it to operating on a specific web page, only filling forms without submitting, only reading not writing. Enforce this in the execution layer (your code), do not rely on the model's self-discipline.

Second line: human confirmation for sensitive operations. Define a set of sensitive keywords (delete, rm, drop, payment, transfer, submit order). When the model's action parameters match a keyword, pause the loop and prompt for human confirmation. The agent should not delete files, spend money, or submit forms without your knowledge.

Third line: sandbox isolation. The agent only operates inside the sandbox, which is physically isolated from the production environment. Even if the agent goes rogue, the worst case is a ruined disposable sandbox, not your real systems. Do not touch production environments, do not touch real accounts, do not touch real money.

Step 5: Tuning

Once it runs, optimize three things:

Screenshot cost. Every step transmits a screenshot to the model, making it the biggest token consumer. Optimization: control resolution (1024x768 is enough), compress screenshots after capture, keep only the most recent N screenshots (drop old ones to reduce context). Community reference scale: a complete task may run 10-50 steps, each step consuming thousands to tens of thousands of tokens, with per-task cost in the range of a few cents to a few dollars (subject to actual API billing).

Coordinate accuracy. The model sometimes miscalculates button coordinates by a few pixels. Optimization: write the screen resolution and scale ratio in the system prompt, have the model screenshot before acting rather than clicking blind, add tolerance for critical clicks (if it misses, screenshot and retry).

Loop detection. The model may get stuck in a "click A -> screenshot -> nothing changed -> click A again" loop. Optimization: record the action sequence of the last N steps, detect repetitive patterns and force-break or switch strategy.

3. Minimal Runnable Example

The following Python strings the five steps together: cua sandbox + Anthropic Computer Use API + safety guardrails + main loop. Verified with ast.parse.

import anthropic
from cua import Sandbox, Image

client = anthropic.Anthropic()  # ANTHROPIC_API_KEY from environment variable

tools = [{
    "type": "computer_20250124",
    "name": "computer",
    "display_width_px": 1024,
    "display_height_px": 768,
}]
BETAS = ["computer-use-2025-01-24"]
MAX_STEPS = 15
SENSITIVE = ["delete", "rm ", "drop", "payment", "transfer", "submit order"]


def is_sensitive(inp: dict) -> bool:
    text = str(inp).lower()
    return any(k in text for k in SENSITIVE)


async def execute_action(sb, inp: dict) -> str:
    """Map model output actions to cua sandbox execution, return result text"""
    action = inp.get("action", "")
    if action == "screenshot":
        await sb.screenshot()
        return "screenshot taken"
    coord = inp.get("coordinate")
    if action == "mouse_move" and coord:
        return f"moved to {coord}"
    if action in ("left_click", "right_click", "double_click") and coord:
        await sb.mouse.click(coord[0], coord[1])
        return f"{action} at {coord}"
    if action == "type":
        await sb.keyboard.type(inp.get("text", ""))
        return f"typed: {inp.get('text', '')[:80]}"
    if action == "key":
        return f"key pressed: {inp.get('text', '')}"
    return f"action {action} executed"


async def run_agent(task: str):
    async with Sandbox.ephemeral(Image.linux()) as sb:
        messages = [{"role": "user", "content": task}]
        for step in range(MAX_STEPS):
            resp = client.beta.messages.create(
                model="claude-sonnet-4-20250514",
                max_tokens=4096,
                tools=tools,
                messages=messages,
                betas=BETAS,
            )
            # stop_reason not tool_use = model considers task done
            if resp.stop_reason != "tool_use":
                print(f"step {step}: task complete")
                return resp.content
            messages.append({"role": "assistant", "content": resp.content})
            for block in resp.content:
                if block.type != "tool_use":
                    continue
                # Safety guardrail: sensitive ops need human confirmation
                if is_sensitive(block.input):
                    ok = input(f"sensitive op {block.input.get('action')}, confirm? (y/n): ")
                    if ok.lower() != "y":
                        result = "user declined"
                    else:
                        result = await execute_action(sb, block.input)
                else:
                    result = await execute_action(sb, block.input)
                # Feed back tool result, tool_use_id must match block.id
                messages.append({
                    "role": "user",
                    "content": [{"type": "tool_result",
                                 "tool_use_id": block.id,
                                 "content": result}],
                })
        print(f"max steps {MAX_STEPS} reached, stopping")

Key points: resp.stop_reason != "tool_use" is the task completion signal (the model no longer requests a tool and instead outputs final text); block.input is the action parameter dict the model outputs (containing action, coordinate, text, etc.); when feeding back tool_result, the tool_use_id must correspond to block.id, or the API will error. Screenshots should actually return base64 image data to the model (simplified to text here; a full implementation converts sb.screenshot() output to base64 and fills it as an image block in the tool_result content field).

4. Five Pitfalls

Pitfall 1: Screenshots too large, token bill explodes Every step transmits a screenshot, and a high-resolution screenshot alone costs thousands of tokens. A 30-step task burns tens of thousands of tokens just on screenshots. Community reports from early testing mention "burning through ten dollars in ten minutes." Fix: compress resolution to 1024x768, reduce image quality after capture, keep only the most recent 3-5 screenshots and drop history. Set a MAX_STEPS ceiling so the agent cannot run forever.

Pitfall 2: Coordinate drift, clicking the wrong button The model miscalculates a button's coordinates by a few pixels and clicks the adjacent delete button instead of edit. This is a universal CUA problem. Fix: have the model screenshot before acting to see the screen clearly; write the screen resolution in the system prompt; screenshot after critical operations to verify results (e.g., check whether the URL changed, whether expected text appeared on the page).

Pitfall 3: Infinite loop, agent repeatedly clicking the same spot The model gets stuck in a "click -> screenshot shows no change -> click again" loop, wasting both tokens and time. Fix: record the action sequence of the last N steps, force-break when repetitive actions are detected; write in the system prompt "if the same operation has no effect twice in a row, try a different approach or report to the user."

Pitfall 4: No confirmation for sensitive operations, agent causes real damage The agent takes it upon itself to delete files, submit orders, or click "confirm payment." The ability to operate a computer means the ability to cause harm -- a CUA without human confirmation is a ticking bomb. Fix: add sensitive keyword detection in the execution layer, pause for human confirmation when triggered. Do not count on "please do not delete files" in the system prompt to stop the model -- that is a suggestion, not a constraint. It must be enforced in code.

Pitfall 5: Sandbox not properly isolated, agent escapes to the host Improper sandbox configuration allows the agent to access the host's real filesystem through shared directories, network interfaces, or the clipboard. Fix: do not mount host sensitive directories in the sandbox, isolate the network by default or route through a proxy, use disposable ephemeral sandboxes (cua's Sandbox.ephemeral() is designed for this). Never put real credentials in the sandbox.

5. FAQ

Q1: What is the difference between Computer Use and RPA (macro recorders)? RPA records mouse trajectories and keyboard inputs then plays them back -- it is a script that breaks when the screen layout changes. Computer Use is based on a vision model that looks at the screen, understands the interface state, and then decides, without preset coordinates, finding buttons even if they moved. RPA is precise but fragile; Computer Use is flexible but slow and expensive.

Q2: Can I run it directly on my computer without a sandbox? Technically yes, but strongly discouraged. An agent that can operate a computer can cause harm -- deleting files, sending emails, clicking buttons it should not. The cost of a sandbox (one extra VM/container) is far lower than the cost of one mistake. At minimum, use a Docker container for isolation; never let the agent touch the host machine directly.

Q3: Besides Anthropic, what other Computer Use solutions exist? OpenAI Operator (cloud browser), Google Project Mariner, Zhipu AutoGLM (mobile), Microsoft Copilot Studio Computer Use (Windows desktop), ByteDance UI-TARS (open source), and more. Each has different positioning and maturity -- see the batch's Computer Use Agent Comparison.

Q4: How much does a single task cost roughly? It depends on the number of steps and screenshot resolution. Community reference scale: a 10-30 step task, each step consuming roughly thousands of tokens for screenshot plus reasoning, with total cost in the range of a few cents to a few dollars. Complex tasks may cost more. Recommended: use low resolution plus a MAX_STEPS ceiling during development to control cost, and monitor per-task token consumption in production. Subject to actual API billing.

Q5: Can Computer Use completely replace tool calling? No, the two are complementary. Systems with API interfaces should use tool calling -- it is faster, cheaper, and more reliable. Computer Use is for legacy systems without APIs or arbitrary web pages. A mature agent typically uses both: call APIs where possible, operate the GUI only when there is no API. See the site's Tool Calling SOP.

References

• Anthropic Computer Use official docs (computer_20250124 tool type, beta flag, action list): https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool
• Anthropic computer-use-demo official repo (loop.py main loop, tools definition): https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo
• trycua/cua official repo (Sandbox API, Image, Lume virtualization): https://github.com/trycua/cua
• cua.ai official docs (sandbox SDK, driver, bench): https://cua.ai/docs
• This site's "Computer Use Era Hotspot": https://aiwebcool.com/en/computer-use-agent-era-hotspot
• This site's "Computer Use Agent Comparison": https://aiwebcool.com/en/ai-computer-use-agents-comparison-review
• This site's "cua Open Source Project Analysis": https://aiwebcool.com/en/cua-computer-use-resource
• This site's "AI Agent Tool Calling SOP": https://aiwebcool.com/en/ai-agent-tool-calling-sop