Touchpoint-Labs/touchpoint

Give your AI agent eyes and hands on any desktop.
pip install touchpoint-py
AI agent researches data in Chrome, then creates a formatted Excel table — full task completed in ~12 minutes
---
Touchpoint is a **cross-platform Python library** for reading and interacting with desktop UI through native accessibility APIs. One import, one API — works on Linux, macOS, and Windows, with built-in support for Chromium and Electron apps via CDP (Chrome DevTools Protocol).
Instead of scraping pixels or running vision models, Touchpoint reads the real accessibility tree — structured names, roles, states, and positions for every element on screen. Fast and reliable, with **no vision model required**. Ships with an MCP server so LLM agents like Claude, Cursor, or any local model can control any desktop app out of the box.
Why Touchpoint?
| | Screenshot / vision | Browser automation | **Touchpoint** |
|---|---|---|---|
| Native desktop apps | ⚠️ inaccurate or slow | ❌ | ✅ |
| Browsers | ⚠️ inaccurate or slow | ✅ | ✅ via CDP |
| Electron apps (Slack, VS Code, ...) | ⚠️ inaccurate or slow | ⚠️ web content only | ✅ native + web |
| Structured element data | ❌ needs OCR/vision model | ✅ web only | ✅ names, roles, states, positions |
| Works with local / non-vision models | ❌ | ✅ web only | ✅ all apps |
| Works across Linux, macOS, Windows | ✅ | ✅ | ✅ |
---
Table of Contents
- Table of Contents
- Install
- Platform requirements
- Quick Start
- Element IDs
- Output formats
- MCP Server
- Tools
- Client setup
- Environment variables
- Browser \& Electron Apps (CDP)
- Setup
- API Reference
- Discovery
- Search \& Wait
- Actions
- Input
- Screenshot \& Config
- Architecture
- Configuration
- Development
- Status
- Known limitations
- License
---
Install
Requires **Python 3.10+**.
Everything is included: your platform's native backend, CDP support for browsers and Electron apps, the MCP server, and screenshot capabilities. Platform-specific dependencies are installed automatically via pip environment markers.
Platform requirements
| Platform | Backend | Requirement |
|----------|---------|-------------|
| **Linux** | AT-SPI2 | Install xdotool (required for input + minimize_window) and wmctrl (required for all window management — used for AT-SPI → X11 id mapping). Most desktops include python3-gi and gir1.2-atspi-2.0 — install them if missing. |
| **Windows** | UI Automation | None — uses built-in COM APIs |
| **macOS** | Accessibility (AX) | Grant permission: System Settings → Privacy & Security → Accessibility |
---
Quick Start
Element IDs
Every element has a unique ID like atspi:1234:1:2.0 or cdp:9222:TID:4. Action functions accept either an Element object or a bare ID string — useful for storing references across steps:
Output formats
Control how results are returned:
---
MCP Server
Touchpoint ships an MCP (Model Context Protocol) server ready for any MCP-compatible client. Use it to let LLM agents like Claude, Cursor, local models, or any tool that supports MCP control your desktop.
Two modes — vision and no-vision
Set TOUCHPOINT_MODE=no-vision (default: vision) to switch modes:
- **Vision mode** — agents use screenshot() to see the screen and interact by element ID or coordinates. Best for frontier models with strong vision capabilities.
- **No-vision mode** — agents use snapshot() to get a compact structured text tree of the active window, then act on element IDs directly. Works with **any model including local ones that have no vision capability**. Most action tools append auto-verify flags ((new window: ...), (focus moved), (no change detected)) so the agent can detect state changes without taking a screenshot.
Tools
| Category | Vision mode | No-vision mode |
|----------|------------|----------------|
| **Orient** | screenshot, snapshot, apps, windows | snapshot, diff_snapshot, apps, windows |
| **Find** | find, get_element | find |
| **Read** | read_text | read_text |
| **Actions** | click (element or coordinates), set_value, set_numeric_value, select_text, focus, action | click (element only), set_value, set_numeric_value, select_text, focus, action |