iris-eval/mcp-server

**Iris scores every agent run for quality, safety, and cost — on your machine, with no SDK and no account.** Most agent projects check quality by running a few remembered prompts and eyeballing the output. Iris replaces that with numbers you can audit: your agent's runs land in a SQLite database on your disk, 26 built-in rules score them deterministically — PII, prompt injection, hallucination markers, cost thresholds, and the agent's own tool calls — free, with no LLM calls, and an optional LLM judge with a hard per-eval cost cap handles the semantic questions. Every rule is inspectable and editable, because a judge you can't audit is just vibes with a number on it. MIT licensed, no telemetry. Nothing leaves your machine unless you turn one of these on: an OpenTelemetry endpoint (IRIS_OTEL_ENDPOINT), which exports traces to the collector you name; the LLM judge with your own key, which sends the text it judges to that provider, and whose citation check fetches the pages an output cites; or a webhook, which posts ids, the verdict and rule names, never the text, to the address you set.
**Requires Node.js 22.13 or later.** Check with node --version.
The demo database, recorded by scripts/demo-media.mts; the source is demo.mp4. A still: dashboard-overview.png.
A failure on screen in 60 seconds
No agent wiring, no config — one command:
This seeds a demo database — five small agents, two weeks of runs, every verdict the engine's own — and serves the dashboard against it at **http://localhost:6920** (your browser opens automatically on first run). The dashboard lands on **Failures**: what failed, worst and newest first, each card naming the rule and its evidence. Worth clicking into — a PII leak caught by the safety rules, a hidden directive in a forum post that the summarizer complied with, a number the source document never said, two runs on the same twelve questions compared with an interval (**Runs**), a deployed custom rule and a paused one with their audit rows, and a failed LLM-judge score with its rationale.
Demo data lives in its own database (demo.db in your Iris home directory — ~/.iris on macOS/Linux, %USERPROFILE%\.iris on Windows) and never mixes with your real traces. Remove all of it with one command:
Hook up your own agent
First, prove the install works on this machine — it runs offline and opens nothing of yours:
Then add Iris to your MCP client. One command writes the client's own config file, keeps every other server in it, and pins the version you ran:
The clients: claude-code, claude-desktop, cursor, windsurf, continue, vscode, cline, zed, codex, gemini. install --list shows the ones found on this machine, the Iris each one runs and the file it reads; install --uninstall takes Iris out again. Every client shares one database, so after an upgrade move them all at once with install --upgrade (Updating). Restart the client to load it.
**Claude Desktop: one click.** Each release from 0.20.0 on attaches iris-eval.mcpb, an MCP Bundle: download the latest, open it, and Claude Desktop shows an install dialog. Nothing on it is required — an Anthropic or OpenAI key for the LLM judge is optional, and the dashboard is a switch that starts off. The bundle holds the npm package and its dependencies, so nothing else needs installing: Claude Desktop runs it under the Node it ships when that Node is 22.13 or newer (Claude Desktop 1.1.6679 ships 24.13), and Iris stores traces with Node's built-in SQLite, in the same ~/.iris every other install uses. The release notes show how to verify its signature and build attestation.
It runs in any MCP client, and every client it names has a row with what was actually checked. **Verified on every CI run: Claude Code, Gemini CLI** — the real client starts Iris from the config the installer wrote and reports it connected (claude mcp list, gemini mcp list), on Linux, macOS and Windows; Claude Code's capture plugin hooks are driven through the real scripts too. **Claimed from each client's own MCP documentation** — the installer writes the configuration shape the client documents, and that writer is tested on the shape; nobody on the Iris side has watched it connect: Claude Desktop, Cursor, Devin Desktop (Windsurf), Continue, VS Code, Cline, Zed, OpenAI Codex CLI. Every row with its source and the date it was read: https://iris-eval.com/clients. By hand instead, one block, dashboard included:
Your client lists Iris's twelve tools on connect, and the dashboard serves at **http://localhost:6920**. Now paste this to your agent: