Remember macos-harness, the browser-use project that gives an LLM an entire Mac with zero rails? Its product-level sequel has arrived: ShawnPana/phone-harness, with a one-line pitch - let your agent control your phone.
GitHub API snapshot (2026-08-22): 1,977 stars / 183 forks, MIT, Python, created 2026-08-07 - two weeks old, averaging over a hundred stars a day. Claude Code or Codex on your Mac can now operate the actual phone in your pocket: open apps, tap buttons, type, scroll lists, send messages.
Scope note: star counts and repo status are API snapshots (2026-08-22); this is a representative teardown based on the README and install docs, not a long-term hands-on; the iPhone path depends on macOS Sequoia+'s iPhone Mirroring feature.
1. How It Controls an iPhone Without a Jailbreak
The smartest move in phone-harness is turning the Mac into the entire transport layer instead of attacking the phone itself:
- The iPhone path: macOS Sequoia+'s built-in iPhone Mirroring renders the phone as a Mac window and forwards real mouse/keyboard input as touches. The harness captures that window with
screencaptureand runs Apple's Vision framework OCR - every visible string on screen comes with a tap-ready coordinate. The README calls it "the poor man's DOM." For actions, HID-level CGEvents inject taps, long-presses, drags, flicks, scrolls, and keystrokes. - The Android path: adb over USB or Wi-Fi.
screencapcaptures; the phone's own accessibility tree (uiautomator) supplies exact text and exact boxes - controls OCR can never find,tap_ui("url_bar")can.input tap/swipe/textdoes the hands. Coordinates map 1:1 to the screenshot, no conversion.
No jailbreak, no Xcode, no WebDriverAgent, no app installed on the phone. The README also keeps an honest list of things that do NOT work, "learned the hard way": AppleScript click at silently does nothing to the mirroring window (it's a video stream with no accessibility tree); unicode input fails (mirroring forwards raw HID keycodes, so typing must use keycodes); slow touch-drags barely move an iOS list (use wheel scroll for lists, a fast flick for pages); input while the window isn't frontmost gets swallowed. That density of "we broke it so you don't have to" documentation is the fastest quality signal an open-source project can give.
2. Installation Is a Prompt: The Agent Sets Itself Up
Like macos-harness, installation isn't a CLI tutorial - it's a prompt you paste into Claude Code or Codex: clone the repo to ~/.phone-harness, read install.md, put phone-harness on your PATH, register it as an agent skill, then walk you through onboarding. It asks exactly one question (is your default phone an iPhone or an Android?) and only hands you the steps that genuinely need your fingers: on iPhone, pair Mirroring once and grant the terminal Accessibility and Screen Recording permissions; on Android, enable Developer options, then plug in and tap Allow, or enter one six-digit wireless-debugging pairing code.
After setup, phone-harness --doctor verifies the chain and phone-harness config set platform ios|android sets the default (both can be configured). Usage looks like this:
./phone-harness <<'PY'
open_app("Notes")
tap_text("New Note")
type_text("hello from the harness")
print([o["text"] for o in ocr()][:10])
PYOpen Notes, tap "New Note," type, then OCR again to confirm - see, act, verify: a complete perception-action loop.
3. The "Model Writes Its Own Tools" Philosophy Wins Again
phone-harness inherits this generation's core design view: no hundreds of prebuilt tools - a few primitives, and whatever logic is missing, the model writes mid-task in ordinary Python (in agent-workspace/agent_helpers.py). There is no "WeChat tool" and no "banking tool." When the model hits an app it has never seen, it composes ocr() + tap() + wait_stable() on the spot.
The leverage is especially large on phones: there are millions of apps and their UIs change daily, so any prebuilt toolset rots; whereas the "screenshot - recognize - tap - verify" loop is app-agnostic. And writing code is precisely what current models are best at - the bet is placed on the longest lever.
4. Before You Enjoy This: Three Gates to Think Through
Our standing position: the stronger the harness, the tighter the reins — the Agent Guardrails Deployment SOP spells out the approval gates, least-privilege rules, and audit logs to install before any agent touches a device. phone-harness faces your real phone - messaging, banking apps, and SMS verification codes all live there. Three gates:
- Start with a spare phone or a clean account. The harness refuses to drive a locked phone, but that's a guard rail, not a defense line. Run scenarios with no sensitive data first.
- Add human confirmation for anything that spends money. Any "tap the confirm button" action needs an approval gate at your business layer, not the harness layer. Just last week OpenAI announced a slowdown after its own agent hacked into Hugging Face, as we documented in OpenAI Hits the Brakes - "models probe boundaries autonomously" is now an empirically verified fact.
- Be honest about where phone automation actually shines. It's best at high-frequency repetitive UI operations - regression tests, bulk settings changes, cross-app data moves, remote demos for relatives - not at replacing an app's API. Where an official API exists, always use the API first. The harness is the last resort when there is no API, not the first choice.
One-line closer: when the agent's hands reach from your keyboard to your phone screen, the distance between "can do" and "should be allowed to do" is an entire judgment system you have to build yourself.
FAQ
Q1: Does it need a jailbreak or root? Does it support two phones?
A1: Neither. iPhone goes through macOS Sequoia+'s iPhone Mirroring window (screencapture + OCR + CGEvent); Android goes through adb (USB or Wi-Fi, screencap + uiautomator + input); nothing gets installed on the phone. Both can be configured at once, and config set platform switches the default.
Q2: Does it work on Windows / Linux? A2: The Android path depends only on adb in theory, so the cross-platform barrier is low. But the iPhone path is deeply tied to macOS - iPhone Mirroring, Vision OCR, and CGEvents - making it effectively a macOS-only solution today. The README's premise is also "the Mac is the whole transport."
Q3: What if OCR misreads and an icon button can't be tapped?
A3: Two routes. On Android, prefer the accessibility tree (tap_ui matches controls exactly). For icon-only controls, the README provides tap_image_point(x, y, image_size=...), converting image-pixel coordinates to screen points. And the verify loop (screenshot again after tapping) is itself the safety net for recognition error.
Q4: How is this fundamentally different from vendor voice assistants or Shortcuts? A4: Generality and autonomy. Assistants and Shortcuts are prebuilt capabilities limited to what their designers imagined; phone-harness hands the whole screen to a model that writes code, so when it meets an un-prebuilt scenario it composes primitives on the spot. The ceiling is much higher - and so is the risk.
Q5: What's the practical use for ordinary people today? A5: Three high-frequency categories: bulk repetitive operations on niche apps without APIs; cross-app data porting (read from one app, fill into another); UI regression testing. The principle: official API first; the harness is the last resort when no API exists.
References
- ShawnPana/phone-harness (GitHub API snapshot 2026-08-22): 1,977 stars / 183 forks, MIT, Python, created 2026-08-07
- phone-harness README: iPhone Mirroring + Vision OCR + CGEvent architecture, Android adb + uiautomator path, the learned-the-hard-way list, setup prompt and usage examples
- Related reading: our macos-harness teardown (same design philosophy on the desktop), the Agent Guardrails Deployment SOP, and OpenAI Hits the Brakes
This article is based on the README and official docs (as of 2026-08-22); star counts are API snapshots, subject to the official source.