woladi/macos-vision-mcp

Your documents and your screen stay on the machine. Only extracted text and structured JSON ever reach the model.
Local, private, offline OCR **and UI testing** for any MCP client — no API keys, no uploads.
Cut document token costs by ~97%, and let an agent see and click your Mac's UI without a single screenshot leaving the machine.
Pre-extracts text and image data locally before your AI ever sees it — cutting token usage by ~97% on real documents and returning structured paragraphs, lines, and bounding boxes so the model can reconstruct the document into Markdown, HTML, DOCX, or any other format. Files never leave your Mac: no cloud API, no API keys, no network requests.
> **How the ~97% is measured:** a 44-page scanned PDF sent as page images costs ~73,500 tokens; the same file run through analyze_document returns ~2,400 tokens of extracted text and structure (raw page-image tokens vs. extracted-text tokens). Your numbers vary with page density and tokenizer — treat 97% as the order of magnitude, not a guarantee.
**Contents:** Quick Start · What you get · What agents use this for · UI testing · Why it's different · Available Tools · Usage · Example workflows · Configuration · Privacy layer
What you get
- OCR for images and PDFs (JPG, PNG, HEIC, TIFF, multi-page PDF) via Apple Vision Framework.
- ~97% token reduction: a 44-page PDF costs ~2,400 tokens instead of ~73,500.
- Reading-order paragraphs + raw text blocks with bounding boxes — rich structure for the model to reconstruct the document into any output format (Markdown, HTML, DOCX, JSON), not a lossy plain-text dump.
- Face detection, barcode/QR reading, and image classification — all on-device.
- Full document pipeline: OCR + faces + barcodes + rectangles in a single tool call.
- Works with Claude Code, Claude Desktop, and Cursor — any MCP-compatible client.
- No files uploaded to any server — processing stays entirely on your Mac.
- **UI testing for agents**: screenshot a window locally, find an element by its visible text, get back click coordinates, and assert what's on screen — all without uploading the screenshot.
- 100% offline after npm install — powered by Apple Vision Framework, same engine as Live Text in Photos.app.
❌ Without / ✅ With
❌ **Without macos-vision-mcp:**
- Sending a 44-page PDF costs ~73,500 tokens
- Every image, invoice, or contract goes through a cloud API
- Sensitive documents leave your machine on every request
✅ **With macos-vision-mcp:**
- Local Apple Vision pre-extracts text before Claude ever sees it
- ~2,400 tokens for the same 44-page PDF — 97% fewer
- Files never leave your Mac
What agents use this for
Most work an agent does on a Mac needs no deep understanding of a layout. It needs to see what is
on screen, find the thing it is looking for, act on it, and confirm what happened. That loop —
list_windows → find_element → click (via any input driver) → assert_text — covers a lot,
and every step of it runs on the machine.
- **Drive an app that has no API.** Native tools, Electron apps, internal software, anything with
a GUI and no scripting interface. find_element("Export") returns the point to click.
- **Read what is on screen right now.** A dialog, an error banner, a notification, a progress
state — including a window sitting behind others, without bringing it to the front.
- **Confirm an action actually worked.** assert_text is string matching after unicode
normalisation, so it answers pass/fail the same way every time instead of asking a model to
judge a picture.
- **Pull data out of software that will not export it.** OCR a window, get the text, move on.
- **Audit accessibility.** ui_snapshot reports every piece of visible text the accessibility
tree does not account for — unlabelled controls and custom-drawn text, with coordinates.
- **Test a UI for regressions.** The case this started as, and still a good one — see the chapter
below.
- **Work through documents.** The original job: invoices, contracts, scans, PDFs.
Why doing it locally is better, not just different
**Cheaper.** A verdict costs ~240 tokens against ~6,900 for the screenshot it replaces — about
**29×**. Over a twenty-step task that is ~4,800 tokens instead of ~138,000. It is the difference
between an agent that can afford to check its work after every step and one that cannot.
**More private, and this is the part that never shows up on a bill.** A screenshot is not a neat
crop of the button you cared about. It carries whatever else was on screen: another window, a