emazaheri/ios-agent

**Give an AI agent an iPhone. It checks every step it takes.**
ios-agent lets an AI agent use an iOS Simulator or a real iPhone, over a cable
or Wi-Fi. Every action returns the screen it produced, so the agent can tell
when a tap did nothing or typed text did not land. Screens arrive as a few
hundred tokens rather than tens of thousands. Anything that sends, pays,
deletes or reaches another person asks first. It uses only Apple's public
APIs, so an Xcode update does not break it. Use it from the terminal, from
Claude Code or any MCP client, or as a Python library.
One goal, start to finish, at 2.5x. The agent deep-links into Maps for the
driving time, then taps through to walking and transit and scrolls to read the
detail: 4 actions, 1 observation, 2,767 device tokens, 49.8s of real time. The
terminal is the agent's own transcript; the phone is an iOS Simulator being
driven by it.
quickstart checks the toolchain, offers the repairs that are cheap enough to
be worth offering, builds WebDriverAgent if it is missing (about 20 seconds,
once), and drops you into manual mode, which drives the device by hand and
needs no API key. When you want the agent itself:
Contents
- What you can do with it · Features · Who it is for
- Requirements · Setup
- The terminal app · Connecting your own agent over MCP
- What the model sees · Safety
- Measured on real hardware
- Why it is built this way · Why only public APIs
- Tracing · Development · Contributing
What you can do with it
| You want to | Start with |
|---|---|
| Hand an agent a goal and watch it work | uv run ios-agent "turn on bold text", walked through in the quickstart |
| Check that a change to your own app works | docs/check-your-app.md |
| Give Claude Code, Cursor or any MCP client hands on a device | ios-mcp serve, one entry in .mcp.json |
| Run it against your own iPhone, over a cable or Wi-Fi | docs/real-device-setup.md, then uv run ios-agent --pick "..." |
| Drive a device by hand, with no API key, to see what an agent would see | uv run ios-agent manual |
| Build your own agent on top, without the protocol in between | await run_goal(session, "...") or IosSession directly, see docs/library.md |
Typical goals: check that a change you just made works in the running app,
change a setting, read an answer out of an app that has no API, or walk a flow
on a phone you cannot hand to a test suite.
docs/use-cases.md lists what works today, with the
evidence for each and what is not supported yet.
Features
It knows when an action did not work
The most common complaint about agent tools for phones is an action that
reports success while nothing happened. Here:
- **Every action returns the screen it produced**, with screen_changed, so a
tap that did nothing is visible at once. A real no-op reports
screen_changed=False on a physical iPhone too, not only on a simulator.
- **Typed text is read back from the field.** Text that did not land returns
ok: false with what the field shows, rather than being reported as typed.
- **The agent names elements, never coordinates.** It passes a ref like e2;
if the screen moved, the host re-finds the same element by identity through
six tiers, and refuses when a ref now points at something else.
- **Switches are set, not toggled.** Asking for on when a switch is already on
does nothing, instead of turning it off.
- **Retries are safe.** Idempotency keys mean an agent framework that replays a
step does not tap Send twice.
It works on a real iPhone, over a cable or Wi-Fi
- Verified on an iPhone 17 Pro Max on iOS 26.6, over USB and over Wi-Fi, at
the same action count as the simulator.
- ios-agent doctor checks the toolchain, the runtimes, the tunnel and the
runner's signing expiry, and gives a **remedy for each failure**.
- The device runner stops with the process that started it, a SIGKILL
included, so a crashed run does not hold the phone. ios-mcp reset clears
anything else.
- The device picker never pre-selects a physical phone, and /device switches
devices mid-session.
It is cheap on tokens
- Screens arrive as a compact digest, not accessibility XML: **251 raw nodes to
12 elements** on a real third-party screen, **50 to 474 tokens per step**
across thirteen golden flows.
- When the next screen is similar, an action returns only what changed.
- A 300-row Contacts list comes back in one observation of 438 tokens.
- ios_find searches the full tree for anything the digest left out, so
compaction never hides a control for good.