Perception-Dynamics-Inc/a11y-computer-use

> 0.4.3 captures 100x faster on macOS 26 (screencapture, per-window OCR that ignores covering apps). 0.4.2 closes the first three agent-filed issues (file dialogs by address, no whole-display OCR under an app's grant), refuses to fight the user for an app (user_active), and asks permission through a real macOS alert. 0.4.0: the user keeps working while the agent works (docs/coexist.md).
Install
Linux
An isolated venv cannot build the [agent,linux] extras. PyGObject is sdist-only and pycairo has no Linux wheel, so pip compiles both, and that compile needs pkg-config plus the cairo and girepository headers. Use the distro packages and a venv that can see them:
The other path is to install those build dependencies and then pip-install into an isolated venv. The package list is in docs/linux-port.md.
Add it to Claude Code, Claude Desktop, Cursor, or any MCP host:
For a browser tab instead of the desktop, start Chrome with --remote-debugging-port=9222 and set A11Y_COMPUTER_USE_DRIVER=browser.
What the model sees
Then it acts: click(ref="e3"), type("hello"), set_value(ref="e2", value="..."). Ref clicks go through the accessibility API and leave your pointer alone; typing addressed to an app (type(text, app=...)) leaves your screen alone too. A stale ref comes back as stale_ref with the nearest live candidates. Re-observing as a diff costs about 10 tokens. Apps with no accessibility tree (Telegram, canvases) get OCR refs o1..oN from on-device text recognition, and click(ref="o7") works the same way (docs/ocr-refs.md).
26 tools: desktop_snapshot, find, screen_text, screenshot, zoom, crop (PNG of one ref's on-screen bounds; no OCR), click, hover (Linux: move the pointer and deliver a hover with no click; elsewhere unsupported), type, key, scroll, drag (with waypoint paths for strokes), wait_for, wait_until, act, set_value, scroll_to_find, notes, menu, file_dialog, app, window, clipboard, request_permission, grant_app, report_issue (agents file the tool's own defects and slow calls as issues here, docs/reporting.md), plus console, network, and webmcp (tools a page registers through navigator.modelContext, listed as w1..wN, docs/webmcp.md) on the browser. Menus and file dialogs stay reachable even in custom-drawn apps (docs/macos-primitives.md). Details: docs/agent-loop.md and the tool docstrings.
Why not the alternatives
| | Approach | Native desktop | Any model | Embeddable |
|---|---|---|---|---|
| Claude Desktop built-in | pixel loop | ✅ | ❌ | ❌ |
| browser-use, Playwright MCP | a11y tree | ❌ browser only | ✅ | ✅ |
| UI-TARS-desktop | pixel / vision | ✅ | partial | app |
| Windows-MCP, Terminator | a11y (UIA) | Windows only | ✅ | ✅ |
| **a11y-computer-use** | a11y tree + vision fallback | ✅ macOS, Linux, Windows*, Chromium | ✅ local too | ✅ MCP, CLI, Python |
Same planner, 13 browser tasks, one round: refs finished 13/13 with 0 misclicks for $7.04; screenshot coordinates finished 7/13 with 27 misclicks for $11.74 (full results). Run it yourself with a11y-computer-use bench h2h.
Platforms
| | Observe | Ref actions | Type, keys | Coordinates, screenshot | Verified |
|---|---|---|---|---|---|
| macOS (AX) | ✅ | ✅ | ✅ | ✅ | live on a granted Mac |