HoldMyBeer-gg/bad-ass-mcp

Every other tool needs a browser or a vision model. This one reads what the OS already knows.
A cross-platform MCP server for desktop GUI automation — driven by accessibility APIs, not screenshots.
Works with Claude, Codex, Gemini, Ollama, or any OpenAI-compatible client that supports MCP.
bad-ass-mcp recording itself interacting with MarkTheCrab
## Why
Most AI desktop automation tools work by taking screenshots and asking a vision model "what do you see?" That's slow, expensive, and fragile.
Every major OS exposes a structured accessibility tree — the same one used by screen readers — that describes every button, text field, combo box, and menu in every running application. bad-ass-mcp speaks that language directly.
macOS: AXUIElement via PyObjC
Linux: AT-SPI2 via gi.repository.Atspi
Windows: UI Automation via comtypes
Actions fire on control objects directly, so the target window doesn't need focus. The user can keep working while automation runs in the background.
## Tools
| Tool | Description |
| listwindows | List all visible application windows |
| gettree | Full accessibility tree for a window as nested JSON |
| findelements | Find elements by role and/or name |
| click | Click / invoke an element (foreground-independent); button="right", clicks=2; falls back to a real click at the element's center |
| clickat | Click at absolute screen coordinates — fallback for webview/canvas UI; takes button and clicks |
| drag | Drag-and-drop with real pointer events, between coordinates or element handles |
| typetext | Type text into a field (SetValue → EditableText → key injection) |
| selectoption | Select an option in a combo box or list (a11y items → SetValue → keystrokes), verified by reading the value back |
| getvalue | Get the current value or text of an element |
| waitforwindow | Wait until a window matching a pattern appears |
| waitforelement | Wait until an element exists and optionally has a given state |
| screenshot | Capture a PNG (write to outputpath or return base64) |
| startrecording | Start recording the screen (or a specific window) |
| stoprecording | Stop recording and export as GIF (short clips) or MP4 (long sessions, no re-encode) |
| learnlayout | Resolve semantic names → live handle IDs (one call per session) |
| runsequence | Execute a list of actions server-side — no per-action round-trips |
## Field notes: when the tree isn't enough
Observed driving large native apps (a game editor, a DAW) on Windows:
Context menus don't exist in the a11y tree until they're open, and nothing in UIA opens them. click(handle, button="right") or clickat(x, y, button="right") does; the menu is then a readable window.
Double-click targets (list rows that open an editor) often expose only a Select action, which highlights the row and stops. Use clicks=2.
Tab strips can behave the same way: a11y Select highlights the tab without switching the page. click now falls back to a real pointer click when no pattern takes, which switches it.
Stock Win32 combo boxes are unreliable through UIA: the item list only exists while dropped down, and SetValue can succeed on one combo and silently no-op on the next one in the same dialog. selectoption ends with a keystroke strategy (focus, type the item, Return) and reads the value back, so ok: false means it really didn't change.