V
vercel-labs/agent-desktop
Computer use CLI for AI agents
View on GitHub ↗Link checked 2026-11-12 · HTTP 200 · opens the vendor's own site
Install & config
Client config (built from the install command this repo gives)
{
"mcpServers": {
"vercel-labs-agent-desktop": {
"command": "npx",
"args": ["skills"]
}
}
}Parsed from the README we fetched on 2026-11-02; we have not executed it.
command/args come from the install command the repository states. When it is ambiguous, the project README wins.
VS Code (.vscode/mcp.json - the key is servers)
{
"servers": {
"vercel-labs-agent-desktop": {
"command": "npx",
"args": ["skills"]
}
}
}Same server, but VS Code keeps it under <code>servers</code>.
Claude Code (command line)
claude mcp add --transport stdio vercel-labs-agent-desktop -- npx skills
Claude Code does not read a config file; add the server with this command.
Key names differ by client: Clients & CLI
- Star count
- 23,229 ★ · vercel-labs/agent-desktop · read from the GitHub search API · (checked 2026-11-13 20:00:50)
- Source
- read directly from its GitHub page · type: MCP server
- Harvest history
- first seen 2026-11-03 · last seen 2026-11-14 · seen 9× · listed by 1 sources
- Install
npx skills
FAQ (answers come from our records)
How do I install it?
The repository provides this install command:
npx skills. It comes from the repository itself; we cannot guarantee it runs as-is in your client.Who is it for?
We have no tags recorded for it yet.
Who maintains it?
repository
vercel-labs/agent-desktop；first catalogued 2026-11-03；last seen 2026-11-14。There is no official certification or quality rating here — we record sources and dates only.# agent-desktop
以下摘录自该仓库的 README（片段） · 在 GitHub 读完整 README →
Computer use for AI agents: read and operate desktop apps through the OS accessibility tree.
A CLI and C library for macOS, Windows and Linux. A snapshot returns JSON with a ref for every element an agent can act on. The agent acts on a ref and snapshots again to check, with no pixel matching; the loop stays in your agent.
On macOS, CLI calls reuse a private command host for each snapshot namespace and binary build. It retains native accessibility objects so a removed row cannot silently resolve to a surviving row with the same identifier. The host exits after 15 idle minutes. TMPDIR must be owned by the current user with mode 0700; the standard macOS per-user temporary directory satisfies this requirement. Headless calls still prohibit foreground activation and synthesized input.
Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content. Keep them out of public logs and uploads. The optional JEV helper sends screen descriptions to a configured model provider; it is separate from the native CLI. See privacy and data handling.
## Installation
Install on macOS, Windows or Linux with npm:
bash
npm install -g agent-desktop
agent-desktop version
Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64.
For direct execution without a JavaScript launcher, download the matching native archive from
GitHub Releases and keep its helper beside the executable.
Official Homebrew and Scoop packages are being submitted; they are not yet available in the default catalogs.
The repository no longer maintains a custom Homebrew tap or Scoop bucket.
macOS needs Accessibility permission: agent-desktop permissions --guide.
Windows needs no grant for apps at your terminal's integrity level.
Linux needs the user's graphical session with at-spi2-core.
See installation and permissions for package-manager behavior, release archives, and building from source.
## Quick start
Observe, act on a ref, observe again. Any app name from agent-desktop list-apps works; Finder is the macOS example.
Each command prints one JSON envelope; a trimmed snapshot follows the commands.
bash
agent-desktop snapshot --app Finder -i # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i # observe again to verify
json
"version": "2.5",
"ok": true,
"command": "snapshot",
"data": {
"app": "Finder",
"window": { "id": "w-204749", "title": "Downloads" },
"snapshotid": "s8f3k2p9",
"refcount": 59,
"complete": true,
"tree": {
"role": "window",
"name": "Downloads",
"children": [
{ "role": "treeitem", "name": "Documents", "refid": "@s8f3k2p9:e8", "availableactions": ["Click"] }
Refs belong to their snapshot. A bare @e8 is rejected with INVALIDARGS. When the UI changes, an action
returns STALEREF or AMBIGUOUSTARGET: take a new snapshot and use its refs.
Several windows give AMBIGUOUSTARGET; snapshot lists the candidate window ids in error.details. Pass one
with --window-id.
Dense apps: map first with snapshot --skeleton --app Slack -i --compact, then drill into one region with
A CLI and C library for macOS, Windows and Linux. A snapshot returns JSON with a ref for every element an agent can act on. The agent acts on a ref and snapshots again to check, with no pixel matching; the loop stays in your agent.
On macOS, CLI calls reuse a private command host for each snapshot namespace and binary build. It retains native accessibility objects so a removed row cannot silently resolve to a surviving row with the same identifier. The host exits after 15 idle minutes. TMPDIR must be owned by the current user with mode 0700; the standard macOS per-user temporary directory satisfies this requirement. Headless calls still prohibit foreground activation and synthesized input.
Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content. Keep them out of public logs and uploads. The optional JEV helper sends screen descriptions to a configured model provider; it is separate from the native CLI. See privacy and data handling.
## Installation
Install on macOS, Windows or Linux with npm:
bash
npm install -g agent-desktop
agent-desktop version
Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64.
For direct execution without a JavaScript launcher, download the matching native archive from
GitHub Releases and keep its helper beside the executable.
Official Homebrew and Scoop packages are being submitted; they are not yet available in the default catalogs.
The repository no longer maintains a custom Homebrew tap or Scoop bucket.
macOS needs Accessibility permission: agent-desktop permissions --guide.
Windows needs no grant for apps at your terminal's integrity level.
Linux needs the user's graphical session with at-spi2-core.
See installation and permissions for package-manager behavior, release archives, and building from source.
## Quick start
Observe, act on a ref, observe again. Any app name from agent-desktop list-apps works; Finder is the macOS example.
Each command prints one JSON envelope; a trimmed snapshot follows the commands.
bash
agent-desktop snapshot --app Finder -i # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i # observe again to verify
json
"version": "2.5",
"ok": true,
"command": "snapshot",
"data": {
"app": "Finder",
"window": { "id": "w-204749", "title": "Downloads" },
"snapshotid": "s8f3k2p9",
"refcount": 59,
"complete": true,
"tree": {
"role": "window",
"name": "Downloads",
"children": [
{ "role": "treeitem", "name": "Documents", "refid": "@s8f3k2p9:e8", "availableactions": ["Click"] }
Refs belong to their snapshot. A bare @e8 is rejected with INVALIDARGS. When the UI changes, an action
returns STALEREF or AMBIGUOUSTARGET: take a new snapshot and use its refs.
Several windows give AMBIGUOUSTARGET; snapshot lists the candidate window ids in error.details. Pass one
with --window-id.
Dense apps: map first with snapshot --skeleton --app Slack -i --compact, then drill into one region with
Listed badge (optional)
Copy this line into the README if you want the listing mark. Optional, and it does not affect being listed.
[](https://anclum.com/en/s/vercel-labs-agent-desktop/)
- Source
- github · https://github.com/vercel-labs/agent-desktop
- Traction
- 23,229 ★
- added
- today