drknowhow/overshoulder

A desktop hand for AI agents on Windows. An agent leases one window (or a whole screen), sees its
controls, clicks and types inside it, and can record what it did as a tutorial with numbered steps.
Any agent that speaks MCP can use it: Claude Code, Codex, Gemini CLI, or a plain script.
An agent recorded this on its own, through MCP: "How to work out a 15% tip on a $48.50 bill in
Windows Calculator". Video, step images and steps.md are in docs/demo/calculator-tip.
Agents work on their window in the background, so the person at the desk keeps their own cursor and
keyboard. See Working beside you.
What agents learn about an app goes into its profile, and people share profiles at
overshoulder.yepgent.com. See Sharing a profile.
Status: phase 6. Windows 10/11 only.
## Install
Needs Python 3.11+ and ffmpeg on PATH (for recording).
powershell
git clone https://github.com/drknowhow/overshoulder
cd overshoulder
python -m venv .venv
.venv\Scripts\pip install -e .
Run Overshoulder from a normal user session. It refuses to start elevated.
## Quick start
Start the gateway and the watchdog, each in its own terminal:
powershell
overshoulder serve # MCP at http://127.0.0.1:47621/mcp
overshoulder watchdog # panic key, human-input pause, scope outline, agent pointer
To start both at every login instead (per-user scheduled tasks, not elevated, restarted if they
crash; logs in %LOCALAPPDATA%\Overshoulder\logs):
powershell
overshoulder autostart install # also: status, uninstall
Create a client for each agent. The token is printed once; only its hash is stored.
powershell
overshoulder client add claude-code
Register it with your agent, for example Claude Code:
powershell
claude mcp add --transport http overshoulder http://127.0.0.1:47621/mcp --header "Authorization: Bearer "
Then ask the agent something like "take the Notepad window as your scope and type a shopping list".
## What an agent can do
| Tool | Does |
| windows | List windows it could take as a scope |
| takescope | Lease window: , screen: or screen:all |
| observe, find, read | See the scope's controls (with refs like e12), filter them, read values; find painted buttons by their text or an icon by a description (below) |
| act | click, doubleclick, rightclick, drag, invoke, toggle, select, setvalue, type, hotkey, focus, activate, each verified afterwards |
| screenshot | Image of the scope |
| recordstart, recordstop | Record the scope and render a tutorial |
| redact | While recording, hide an element in every frame of the tutorial |
| annotate, render | Edit one of its recordings' annotation layer and render it again |
| profile | What earlier sessions learned about the scope's app; add a note for the next agent (below) |
| releasescope, status | Give the desktop back; see who is driving |
Agents should prefer element refs and invoke/setvalue. Coordinates work as well, for canvas
apps and other apps whose controls Windows can't see. act takes method. auto (the default) works
in the background and uses the person's mouse and keyboard only when nothing else reaches the app and
their settings allow it; background never uses them; input uses them straight away. Each result