mishan/shotbox

Eyes, ears and hands on a real graphical program, for an AI agent, a
script or CI.
An agent working on a program with a window can't see it or hear it.
shotbox runs the program in a sealed session of its own, X11 or Wayland,
and lets whatever is driving it look, listen and act: read the accessible
tree, take screenshots, record what it plays, click by a button's name,
type, hold keys and gamepad buttons, and wait until the program is
actually ready. Then measure what it finds: names a screen reader can't
say, contrast, the Tab order, pixels changed against a reference, where
it flickers, how fast it follows the pointer, where its sound drops out,
and what it used.
Nothing it runs reaches your desktop or speakers by accident, and with
--net loopback, not your network either.
Made with shotbox: each scene is a play script in a sealed
session, recorded by shotbox record. With sound.
claude mcp add shotbox -- shotbox mcp --allow ~/src/app/shotbox-programs
Without an agent, it's the same session from the command line:
shotbox shoot editor.png --window 'Text Editor' -- gnome-text-editor notes.md
Worked examples, from checking a change to a game's smoke test:
docs/recipes.md. How it differs from other tools that
drive a GUI: docs/comparison.md.
## For agents
shotbox mcp --allow PROGRAMS gives an agent sealed sessions over the
Model Context Protocol (stdio). It can start a session, spawn programs
from the list in PROGRAMS (NAME = COMMAND a line, and nothing else),
and then:
read the accessible tree, and take screenshots, which come back as
images;
click by a node's name, press keys, type and work a gamepad;
wait for windows, stillness and announcements;
lint, measure contrast and walk the Tab order;
record what a session plays, and hear where it dropped out.
That's enough for an agent to check its own change in a GUI, test a game
by playing it, audit what a screen reader hears, or reproduce a bug report
in a session that's the same each time. The accessible tree is the cheap
way to look, a thousand or two tokens of text; a screenshot is for what only
pixels say. An agent can't run anything off the list, and its sessions stop
when the server does.
What it's for, the tools, and how it's built: docs/mcp.md.
An agent that runs a program itself, from its shell, runs it on your
desktop: a window opens on your screen, a sound plays through your
speakers, a browser run under xvfb-run finds your Wayland display all
the same. To keep it in a session, put this in the project's CLAUDE.md
or AGENTS.md, or your own:
markdown
## Graphical programs, browsers and sound
Never run a graphical program, a browser (headless too) or anything that
makes sound straight on this machine: it reaches my desktop and speakers.
Run it in a shotbox session:
a picture: shotbox shoot OUT.png --window RE -- COMMAND
a script or a test suite with windows: shotbox run [--audio] -- COMMAND
with none, a headless browser say: shotbox exec [--audio] -- COMMAND;
the shotbox package's sealed() seals a browser's environment, but
not the speakers
a session to come back to: shotbox start --name NAME, then
shotbox spawn --session NAME -- COMMAND, and shotbox stop NAME
Not xvfb-run: it lets WAYLANDDISPLAY and the session bus through. A