trycua/cua: open-source computer-use 2.0, AI drives your whole OS in the background

trycua/cua bets on the next layer of AI agents: not making models smarter, but letting a model operate an entire computer the way a human does-in the background, cross-platform, without stealing your mouse. GitHub 21,114 stars (real-time, per GitHub), 1,445 forks, MIT license, created 2025-01-31, last push today (2026-08-11), not archived, trending on Trendshift. Homepage cua.ai. The primary language is listed as HTML, but it is really a multi-component monorepo: the core driver is written in Rust with UniFFI bindings exposing Python and TypeScript SDKs, and the team includes ex-Microsoft engineers-which is why Windows support runs so deep. The differentiator in one line: browser-use only handles the browser, openclaw runs your email and calendar, cua handles the entire OS-any desktop app can be driven in the background, plus a full computer-use 2.0 stack of sandbox, benchmarks, and virtualization.

Scope: this article is based on the trycua/cua GitHub README, libs/cua-driver/README.md, and the in-repo blog post "Inside Windows computer-use." Facts as of 2026-08-11; star count fluctuates in real time. It complements but is independent of our computer-use agent era hotspot and AI computer-use agent comparison review.

1. Positioning: computer-use 2.0 is not "screenshot and click"

The first generation of computer use (Anthropic's early Computer Use API) was "look at the screen, click the mouse": screenshot fed to the model each step, model outputs coordinates, execute in the foreground. Slow, blind, and it hogs the foreground-while the AI operates, your mouse is hijacked. cua bets on 2.0: the AI drives multiple computers in the background at once, like a real employee working in parallel, without stealing your cursor or focus. The README positioning, verbatim: "Scale computer-use 2.0 with open-source drivers, cross-OS fleets, and benchmarks for training, evaluation, and data generation." Not another "AI operates a computer" demo, but the infrastructure for letting AI drive computers in the background-drivers, sandbox, benchmarks, and virtualization, four pieces, open-sourced together.

"Background" is the key word. The driver README states it plainly: "drives native macOS apps without stealing focus." While the AI operates, a synthetic cursor painted by the driver appears on screen, separate from your physical mouse-you do your work, the AI drives another set of apps, neither disturbs the other. This is the prerequisite for computer-use moving from "demo toy" to "parallel productivity."

2. Core components: four pieces, each its own job

Cua Drivers: give an agent background computer-use ability. The core. A background computer-use driver supporting macOS, Windows, and Linux (Linux via X11 and compositor-specific Wayland routes, with explicit limits on raw background input). The agent clicks, types, scrolls, and verifies through it, never stealing your cursor. Two interfaces: MCP over stdio (cua-driver mcp, for Claude Code, Codex, Cursor, OpenClaw to connect directly) and CLI (cua-driver call). Under the hood is a Rust runtime with UniFFI bindings exposing Python and TypeScript SDKs.

How does it "see" the screen? The in-repo blog says the driver gives the agent three things at once: window pixels (real-time screen image), UIA/MSAA accessibility trees (the UI structure an app exposes-what a button is called, what an input's label is), and an action layer (clicks, typing, scrolling, setting values, verification). Combined, the agent doesn't only "look" at the screen-it reads the interface's logical structure and operates precisely, faster and more accurate than pure vision. Why is Windows harder than macOS? The blog, verbatim: "one driver has to work across Win32, WPF, WinUI, UWP/WinRT, Electron, Chromium, legacy controls, and custom-rendered canvases." The solution is to make the driver a "router"-pixels, accessibility trees (UIA, falling back to MSAA for old apps), and actions each have multiple backend paths, chosen dynamically per target. The default background mode refuses to silently steal the foreground; targets that can't run in the background are returned as explicit errors instead of failing silently. That is a mark of engineering maturity.

Cua (Sandbox): give an agent an OS to work in. pip install cua (Python 3.11+), one API for any VM or container image, cloud or local. Supports Linux containers, Linux VMs, macOS, Windows, Android, and BYOI (bring your own .qcow2/.iso). Cloud via cua.ai, local via QEMU. The agent sees the screen, clicks buttons, runs commands, and completes tasks autonomously in an ephemeral environment.

Cua Bench: score computer-use ability. Benchmarks plus RL environments, evaluating agents on OSWorld, ScreenSpot, Windows Arena, and custom tasks, with trajectory export for training. An objective yardstick for researchers, a model-selection tool for everyone else. Registry at cuabench.ai.

Lume: macOS virtualization. Uses Apple's Virtualization.Framework to spin up macOS/Linux VMs on Apple Silicon with near-native performance. One command pulls an Apple restore image and starts a Tahoe or Sequoia VM; --unattended installs offline, configures SSH and autologin. Use it when an agent needs to run Xcode or Final Cut in an isolated macOS environment.

3. How it splits from openclaw, browser-use, and closed-source Operator

First, draw the territories. browser-use only handles the browser: vision-first, screenshots fed to the model which outputs click coordinates, bounded by the browser tab. openclaw is a personal assistant: runs on your machine and manages email, calendar, files, and terminal-its strength is "doing your chores" personal automation. cua's territory is "background control of the entire OS"-any desktop app (not just the browser) can be driven, plus sandbox, benchmarks, and virtualization. In one line: browser-use is "eyes that only watch the web," openclaw is "manages your daily chores," cua is "hands and eyes that drive your whole computer, plus a workstation and a scorecard." The three are not pure competitors: the cua-driver README lists OpenClaw as a supported client, and browser-use can run inside a cua sandbox-cua is more like the infrastructure layer underneath. Against closed-source OpenAI Operator, cua is open-source MIT, model-swappable, locally offline-capable, and embeddable in your own code, while Operator is a black-box SaaS. To build your own pipeline, see our computer-use agent build SOP.

4. Getting started: three shortest paths

Path A: give Claude Code background control (fastest). macOS/Linux: /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"; Windows: irm https://cua.ai/driver/install.ps1 | iex. Hook it into Claude Code:

claude mcp add --transport stdio cua-driver -- cua-driver mcp

Then tell Claude Code "open this app, extract the data, fill in the spreadsheet"-it actually operates the desktop app. Other clients use cua-driver mcp-config to print config; run cua-driver doctor to check the environment. Three permission modes: standard (default, no prompts), bounded (only allowlisted, manifest-reviewed tools), unrestricted (--dangerously-bypass-approvals, dangerous)-fixed at launch, changing requires restarting the daemon.

Path B: run an autonomous agent via Sandbox. pip install cua, spin up an ephemeral sandbox, and the agent sees the screen, clicks buttons, runs commands. Cloud via cua.ai, local via QEMU, Android supported too.

Path C: run benchmarks. Clone the repo, cd cua-bench, uv tool install -e ., cb image create linux-docker to build the image, cb run dataset datasets/cua-bench-basic --agent cua-agent --max-parallel 4 to score.

5. Barriers and cost: open source is not zero-cost

Four points. First, Windows compatibility still has edges: the blog admits minimized windows have no pixels for WGC to capture, that Chromium coordinate clicks, GTK buttons, and WPF drags must explicitly opt into foreground, and that some accessibility providers lie, hang, or expose less tree than the UI shows. cua returns these as explicit errors instead of swallowing them, but you have to know how to handle them. Second, the Session 0 trap: an agent launched from SSH, a service, or the wrong parent process lands in Session 0 (no interactive desktop), where EnumWindows, UIA, and screenshots all return empty. cua solves this with a daemon running in the user's interactive session, proxying tool calls over a named pipe-but it means you must run the daemon in the right place. Third, model costs are on you: the infrastructure is free, but the LLM behind the agent burns your API budget; computer-use feeds pixels and trees every step, so tokens burn faster than plain text. Fourth, maturity: 21k stars, pushed today, but still an infrastructure project in rapid iteration-interfaces and docs shift, it is not an out-of-the-box product. Run your own scenario before production.

Suited for: developers who want to give Claude Code or Codex "whole-computer control"; teams running automation on Windows/macOS desktop apps without foreground hijacking; researchers needing benchmarks and trajectories for computer-use training. Skip it if: you only want browser automation-browser-use is a better fit, and we have a browser-use resource deep-dive; you want an assistant for email and calendar-openclaw suits that; you want out-of-the-box with no config-closed-source Operator saves hassle.

6. FAQ

Q: What's the difference between cua and browser-use? A: browser-use only handles the browser, vision-first (screenshots fed to the model, which outputs coordinates). cua handles the entire OS-any desktop app can be driven in the background, via pixels plus accessibility tree plus action layer in parallel, without stealing your cursor. The two can also complement each other: browser-use can run inside a cua sandbox.

Q: How does "background run, no cursor stealing" actually work? A: The driver paints a synthetic cursor, separate from the physical mouse, with its own cursor_id per agent. The underlying action can be UIA, MSAA, PostMessage, or SendInput. The default background mode refuses to silently steal the foreground; specific cases explicitly opt into foreground. Source: the in-repo blog "Inside Windows computer-use."

Q: Is Linux supported? A: Yes, with limits. The driver on Linux uses X11 and compositor-specific Wayland routes, with explicit limits on raw background input; macOS and Windows are fully supported. The Sandbox layer supports all OSes.

Q: How do I install it, and does it cost anything? A: MIT, open-source, free. Driver install on macOS/Linux: curl https://cua.ai/driver/install.sh; Windows: irm https://cua.ai/driver/install.ps1 | iex. Sandbox: pip install cua (Python 3.11+). Cloud sandbox via cua.ai is pay-as-you-go; local QEMU is zero cloud cost; model fees are on you.

Q: 21k stars-is the project mature? A: Active but still in rapid iteration. Created 2025-01-31, pushed today, not archived. Interfaces and docs are still shifting, and the blog admits Windows compatibility edges. Run your own scenario before production.

References

• trycua/cua GitHub repo (21,114 stars / 1,445 forks, MIT, multi-language monorepo): https://github.com/trycua/cua
• cua homepage and docs: https://cua.ai | https://cua.ai/docs
• Cua Driver architecture (libs/cua-driver/README.md): https://github.com/trycua/cua/blob/main/libs/cua-driver/README.md
• In-repo blog "Inside Windows computer-use": https://github.com/trycua/cua/blob/main/blog/inside-windows-computer-use.md
• Related on this site: computer-use agent era hotspot | AI computer-use agent comparison review | computer-use agent build SOP | browser-use resource deep-dive