inth3shadows/runecho

**Your AI agent calls a function that doesn't exist. RunEcho stops the edit before it's written** —
not after the build fails. It runs as a PreToolUse hook, checks every
Edit/Write against the symbols your code actually declares, and asks you
first when a reference has nothing behind it:
- **Deterministic.** A parse and a lookup, not a model: the same edit gets the
same verdict on every run, machine and agent. No LLM, no API keys, no network.
- **Free in context.** A clean check writes nothing; only an edit it stops costs
anything (~100 tokens). ~12 ms, no build, no language server.
- **Honest about its reach.** It reads bare calls, constants and type
references, which caught 4 of 9 real hallucinations in our benchmark; the
other 5 need receiver types and are out of scope by design. One layer, like a
type checker, not the whole answer.
How Well It Works
**The same code produces the same answer.** Every check is a parse and a lookup,
so the verdict is identical on every run, every machine, and every agent — there
is no model to sample from and nothing to re-roll. That is what makes it a gate
rather than an opinion: it is **model-free and vendor-neutral** — no LLM, no API
keys, no network, no build, no language server.
**The guard costs zero context tokens** — measured, not asserted: a clean check
writes nothing at all, and only an edit it actually stops costs anything (~100
tokens). It is a PreToolUse hook, so the agent never spends context deciding
whether to call it. The oracle MCP server is a separate surface and is *not*
free — its tool schemas cost ~968 tokens at session start, and structure
unscoped is expensive enough to be worth scoping. Every number, including the
unflattering ones, is in bench/TOKEN-COST.md.
**How often it's wrong, measured against git history, not approvals.** A
user's approve-anyway rate turned out to have no variance to measure — 308 of
308 ask-gated edits approved, zero denied, across 30 days of dogfood traffic.
So fpaudit judges each flagged symbol against **dated git history** instead:
was it defined at ask time, and is it defined now. Live reading across this
project's own dogfood corpus: **15.3% false-positive, 33.7% premature** (the
guard was *correct*, just fired before the symbol it flagged existed — an
agent writing a caller before its callee), **51.0% stands** (a real unbacked
reference). The full method, and what NOT to conclude from the 51%, is in
bench/FPAUDIT.md.
**The scope, stated up front.** RunEcho reads *unqualified* references — bare
calls, constant references, and type annotations. Measured against its own
corpus of real model hallucinations, that catches **4 of 9** (N=15 hand-verified
cases mined from live session transcripts, each backed by a compiler or runtime
error as independent ground truth). The other 5 are qualified positions —
df.groupby(…), tree.Root() — which need receiver-type resolution and are out
of scope by design. The numbers, including the misses, are in
bench/FINDINGS.md.
That is the honest shape of the thing: **one cheap layer against AI coding
mistakes — not the whole answer.** A narrow, fast, certain check that runs before
the write, not a system that makes your agent correct. Run it the way you run a
type checker — one layer that removes one class of mistake completely, alongside
the tests and review that catch the rest.
Why RunEcho Exists
Coding agents are useful, but they routinely make three kinds of mistakes:
- they refer to functions or types that do not exist
- they describe structural changes inaccurately
- they keep reasoning from stale repo state after the code has moved on
RunEcho exists to give those agents a **local source of truth** they can query
before they speak, edit, or commit.
Use it when you want:
- a deterministic answer to "does this symbol actually exist?"