Open-AutoGLM: Zhipu Open-Sourced a Phone Agent That Actually Clicks - "Open Xiaohongshu and Search for Food" and It Does It

For the past two years the agent battlefield has lived inside browsers and terminals: reading files, calling APIs, running tests. But where ordinary people actually spend five or six hours a day is the phone screen - and the "intelligence" on phones still tops out at a voice assistant launching an app for you. Zhipu's zai-org open-sourced Open-AutoGLM to attack exactly this hardest slice of GUI: a phone-side intelligent assistant framework built on AutoGLM. Say "Open Xiaohongshu and search for food" in natural language, and it takes its own screenshot, reads the interface itself, plans its own next step, and taps the screen itself, until the task is done.

GitHub API snapshot (2026-08-27): 26,104 stars / 4,019 forks, Apache-2.0, Python, created 2025-12-08. One caveat on scope: the repo's last push landed 2026-03-06 - nearly half a year of silence as of the snapshot - so the star count is accumulated history rather than direct evidence of current momentum.

Scope note first: star counts and repo status are API snapshots (2026-08-27); this is a representative teardown based on the official README (Chinese and English) plus repo metadata, not a long-term hands-on; "works as described" follows the official docs and was not verified device by device.

1. What It Solves: Turning "Talking" into "Doing"

Open-AutoGLM's positioning fits in one sentence: automation of phone GUI operations. The system controls the device via ADB (Android Debug Bridge), perceives the screen with a vision-language model (VLM), and generates and executes operation workflows through intelligent planning. The user describes the need in natural language; the Agent parses the intent, understands the current interface, plans the next action, and completes the whole flow.

Its run loop is compressed into a single line in the README (verbatim): screenshot -> vision model reads the interface -> outputs tap coordinates -> ADB executes the action -> loop. This "vision route" is the key to the project's value: it does not depend on accessibility interfaces, control trees, or any app's cooperation - just "look at the screen" and "tap coordinates" works on any application. The price is a VLM inference pass on every step, which costs more time and tokens than control-tree approaches, but nothing else matches its generality.

One sentence to separate it from the phone-harness teardown we covered in August: phone-harness "controls an iPhone through a Mac mirror," poking at iOS via the Mac's mouse and keyboard; Open-AutoGLM takes the mainstream road of "native Android ADB + VLM screen perception" and gets low-level device control directly. The contrast with the frameworks in our Agent Runtime five-way comparison is more fundamental: those runtimes live on servers and in sandboxes, operating on files and processes; Open-AutoGLM operates a physical phone - one of the rare agent projects that actually has fingers.

2. Core Design: a Perceive-Plan-Execute Loop with Two Safety Gates

Each step can only use a fixed set of ten action primitives:

Each step, the model outputs a reasoning trace plus one action JSON, all visible with verbose=True - you watch it write "currently on the system home screen, I need to launch Xiaohongshu first," then emit {"action": "Tap", "element": [500, 100]}, and move to the next step. Each task defaults to a 100-step cap (PHONE_AGENT_MAX_STEPS is adjustable), preventing runaway loops.

The loop can also be wrapped one level up by the Python API, turning "one sentence" into a function call inside your program:

from phone_agent import PhoneAgent
from phone_agent.model import ModelConfig

model_config = ModelConfig(
    base_url="http://localhost:8000/v1",
    model_name="autoglm-phone-9b",
)
agent = PhoneAgent(model_config=model_config)
result = agent.run("Open Taobao and search for wireless earphones")

So it is not just a command-line toy: you can wire flows like "order tonight's takeout" or "forward the articles in my favorites to File Transfer Assistant" into your own scheduling system. The framework (running on your computer, calling the model, parsing actions, controlling the phone) and the model service (remote API or local deployment) are two decoupled halves speaking an OpenAI-compatible interface. The model-side parameters are deliberate too: temperature 0.1, max_tokens 3000 - the low temperature is on purpose, because GUI operation wants determinism, not divergence.

The part of these bare primitives most worth singling out is the two safety gates - the dividing line between this and the crowd of GUI-agent demos:

• Sensitive-operation confirmation: a built-in confirmation mechanism with a confirmation_callback, so sensitive actions ask you before executing.
• Manual takeover: for login and captcha scenarios that machines should not touch, the Take_over primitive hands control back to a human, customizable via takeover_callback. Even finer is the black-screenshot detection - payment, password, and banking apps actively block screenshots, and when the Agent receives a black frame it recognizes "this is a sensitive page" and automatically requests manual takeover instead of tapping blindly.

Two more engineering details show this is not a toy. First, remote debugging: with the phone and computer on the same WiFi, adb connect 192.168.1.100:5555 controls the device wirelessly, and multiple devices are distinguished by --device-id. Second, the system prompt ships in both Chinese and English (prompts_zh.py / prompts_en.py) and is directly editable - the docs state explicitly that injecting app names can disable specific apps. Drawing fences around the agent has been made a configuration item.

3. Where the Model Comes From: One 9B Phone-Specialized Model, Three Ways to Get It

The framework itself contains no model; the model is an external service. What is open-sourced is AutoGLM-Phone-9B, in two editions:

The model architecture is identical to GLM-4.1V-9B-Thinking, and deployment follows the GLM-V repo. Three tiers of access:

The local launch parameters are strikingly specific: --max-model-len 25480, at most 10 images per prompt, image max_pixels of 5,000,000 - these are not decorative numbers, and deviating from them likely breaks the deployment. The README even ships a check_deployment_en.py script and warns: "if the reasoning chain is very short or contains garbled text, the model deployment has probably failed."

4. How to Install: the README Contains a Whole Chapter Written for an AI to Read

There are two install routes.

The lazy route: configure GLM Coding Plan with Claude Code, then enter one prompt:

Access the documentation and install AutoGLM for me
https://raw.githubusercontent.com/zai-org/Open-AutoGLM/refs/heads/main/README_en.md

This design is more interesting than it looks: at the end of the README sits an entire chapter titled "Automated Deployment Guide (for AI)," which states outright that "this section is designed for AI assistants such as Claude Code; if you are a human reader, skip it." In other words, this README is simultaneously documentation for humans and an installer for agents - it pre-packages a pre-deployment checklist (does the cable support data transfer, is USB debug security mode on), an exception-handling table, and an acceptance test ("Open WeChat, send a message to File Transfer Assistant: deployment successful"). Using an agent to install an agent is meta enough to count as a new 2026 pattern for project READMEs, converging with the docs-as-toolchain spirit of OpenAI's open-sourced Codex Harness.

The manual route: Python 3.10+, pip install -r requirements.txt && pip install -e ., then the Android-side preparation - enable developer mode and USB debugging (tap the build number in Settings about ten times to unlock developer mode, then tick USB debugging under developer options), install ADB Keyboard (required for text input; without it, Chinese input fails; HarmonyOS devices use the native input method and skip it), and confirm the connection with adb devices. Device requirements are Android 7.0+ or HarmonyOS NEXT and above (HarmonyOS goes through the HDC toolchain, switched via --device-type hdc), and there is a separate iOS setup doc (the WebDriverAgent route) for iPhone.

Once the phone connects, the official acceptance test is "Open WeChat, send a message to File Transfer Assistant: deployment successful" - the phone opens WeChat by itself, finds the contact by itself, and sends the message by itself; only when all three steps pass is the deployment done. It is a smartly designed test: it covers the three core primitives Launch, Tap, and Type without touching any sensitive permissions, and confirms in one minute that the whole chain (model service, ADB connection, action execution) is alive.

The official docs list the common traps; the three most likely to defeat you: a charge-only cable (empty adb devices list - swap the cable); some models requiring "USB debugging (security settings)" to be enabled before taps register (the symptom: apps launch but taps do nothing); and Windows GBK encoding errors (UnicodeEncodeError: gbk code - set PYTHONIOENCODING=utf-8 before running). Also, interactive mode does not work in a non-TTY environment (EOF when reading a line) - just pass the task description on the command line directly.

5. What It Can and Cannot Do

Coverage comes as hard official numbers: 50+ mainstream Chinese apps on Android, 60+ native and system apps on HarmonyOS:

What it can do: composite cross-app tasks (the README demo includes "compare this product's price on JD vs Taobao, then order on the cheaper one" - the model's reasoning chain decomposes it into launch JD, search, note the price, switch to Taobao, compare, order); routine batch chores such as the official sample tasks "open Meituan and search for nearby hotpot" or "open Douyin and browse videos," or scheduled forwarding of a few news items to a colleague; and, on the testing side, regression sweeps - throw the same task description at different model or prompt versions and watch whether it taps correctly. The environment variables also leave a door open for unattended use: PHONE_AGENT_BASE_URL, PHONE_AGENT_MODEL, PHONE_AGENT_DEVICE_ID, PHONE_AGENT_MAX_STEPS and friends can all be preset in the environment for scripted invocation.

What it cannot do, or does at a discount, stated plainly:

• Login and captcha require a human: by design, not defect - but it means this is not a fully unattended solution.
• Payment and banking pages are black boxes: screenshots are blocked by the system, and the Agent can only request takeover.
• Research and learning use only: the README terms say verbatim that it is "strictly prohibited" to use it for illegal information acquisition, system interference, or any illegal activities - so forget gray-market bulk automation, and do not treat it as production-ready either.
• Weigh the activity level yourself: under our snapshot the last push was 2026-03-06; there is no commitment on adapting to new phone models or app redesigns.

6. Ecosystem: Not Just One Repo, a Whole Product Line

Judged by the repo alone you would underrate it - Open-AutoGLM has several gears meshing around it:

• Two-way Midscene.js integration: that open-source, vision-model-driven UI automation SDK (JavaScript/YAML flow syntax) has completed its adaptation of the AutoGLM model, and the docs note it works on both iOS and Android - effectively piping in the JS/YAML ecosystem, so frontend developers can use it too.
• GLM Coding Plan linkage: the install entry plugs directly into Zhipu's own Claude Code subscription plan - using their own agent framework to install their own agent framework.
• The companion product Autotyper: Zhipu AI Input Method, the "command by voice" landing form (autoglm.zhipuai.cn/autotyper) - the open-source framework is the research entrance, the keyboard is the commercial exit.
• The "AutoGLM Shizhanshipai" incentive campaign: get it running or build a derivative, share a cash prize pool of tens of thousands of RMB - the project actively channels traffic toward derivatives.
• Two papers as lineage: AutoGLM (arXiv:2411.00820) and MobileRL (arXiv:2509.18119, online agentic reinforcement learning for mobile GUI agents) - the technical route has a publicly checkable provenance.

In the bigger picture this connects directly to GLM-5.3-Flash, which Zhipu open-sourced this same week - the first natively multimodal model in the GLM-5 line, whose Visual Coding capability lets a model coordinate across code, browser, and graphical interfaces with visual verification. GUI/vision agents are the lane Zhipu is explicitly betting on: GLM-5.3-Flash covers visual verification on the code and browser side, Open-AutoGLM covers actual operations on the phone side - one model's "eyes" plus one framework's "fingers." For the coding-side counterpart, see our Cline teardown; for the multimodal launch itself, see the sister piece Zhipu open-sources GLM-5.3-Flash.

7. License and Fit

Apache-2.0 - two words: use freely. None of OpenViking's AGPL virality concerns; the licensing bar for commercial derivative work is zero (the "research and learning only" terms still apply, of course - that is a product boundary, not a license boundary).

Who it is for: developers studying phone GUI agents (currently the most-starred and best-documented open-source Chinese phone agent implementation, with a fair share of those 26k stars channeled in by the incentive campaign), teams doing Android automation testing who want to retire brittle scripts (Python via the native API, YAML/JS via Midscene.js - both roads work), and independent developers building derivatives for the prize pool. Who it is not for: enterprise RPA customers wanting out-of-the-box, SLA-backed, unattended operation - logins need a human, payments are a black screen, and activity paused in March. Those three are enough.

One-line closer: while most agents are still reading and writing files on a server, Open-AutoGLM open-sourced the "fingers" - it may not be the strongest agent, but it is one of the very few that will actually open Xiaohongshu for you.

References

• zai-org/Open-AutoGLM (GitHub API snapshot 2026-08-27): 26,104 stars / 4,019 forks, Apache-2.0, Python, created 2025-12-08, last push 2026-03-06, official description "An Open Phone Agent Model & Framework. Unlocking the AI Phone for Everyone"
• Open-AutoGLM official README (Chinese/English), including model downloads, deployment parameters, supported app lists, and the AI-facing deployment guide: https://github.com/zai-org/Open-AutoGLM
• Model pages: AutoGLM-Phone-9B / AutoGLM-Phone-9B-Multilingual, https://huggingface.co/zai-org/AutoGLM-Phone-9B
• Companion product: Zhipu AI Input Method (Autotyper), https://autoglm.zhipuai.cn/autotyper/
• Papers: AutoGLM (arXiv:2411.00820), MobileRL (arXiv:2509.18119)
• Midscene.js integration guide (AutoGLM adaptation): https://midscenejs.com/model-common-config.html#auto-glm
• Related reading: our Zhipu GLM-5.3-Flash open-source hotspot (sister piece), phone-harness teardown (the iPhone mirror route), Agent Runtime five-way comparison, OpenAI open-sources Codex Harness, Cline coding agent teardown

This article is based on the official README and repo metadata (as of 2026-08-27); star counts are API snapshots, no long-term hands-on testing was done, and supported apps and capabilities are per the official docs.