harriswekselblatt/pocp

POCP checks that the tests a pull request was supposed to pass actually ran, on the exact code being merged, under rules the pull request can't change. When a pull request edits existing tests, it runs the old tests on the new code, and holds risky test edits for human review. An optional deep mode also compares the old and new tests against a sample of planted bugs. It works with any test command and any language, and needs only Node 20 or later and git.
It doesn't prove your code is correct. It's also early: the results below come from a scripted test suite and small pilots, not a completed evaluation. To install it, see How to try it.
## Demo
An agent edits a failing test until CI turns green. POCP runs the old test against the new code and routes the suspicious change to human review.
To run it from a clone: node examples/demo/demo.mjs. It takes a few seconds and shortens POCP's output; --keep leaves the scratch repository so you can see everything.
## Why this exists
I started building with AI coding agents without knowing much about software, and I still can't read the code they write, not at the scale they write it. That left one question I couldn't let go of: if I can't read it, how do I trust it? How do I know a change does what I meant, doesn't quietly break something, and doesn't fake its way past the tests?
POCP is my answer. Instead of trusting a sentence an agent typed ("tests pass"), it records proof: the checks that ran, on the exact code that was committed, under rules the change itself can't loosen. Then it checks that proof again in CI.
I'm testing POCP with separate benchmarks and a reference checker that shares no implementation code with POCP. So far, only pilots have run. No counted evaluation exists, and the pre-registration is unfinished. The results and their limits are linked below.
## What it does
A rule in .proof/rules.json says which files need which checks:
json
"pocp": 1,
"requirements": [
{ "id": "billing-tests", "globs": ["src/billing/"],
"obligation": { "kind": "test", "run": "node --test test/billing.test.js" } }
A check is any command, such as npm test, pytest, cargo test, go test ./... or make check.
Locally, a git hook blocks a commit that touches src/billing/ until a packet records that node --test test/billing.test.js passed on exactly the staged code. pocp fix runs the checks and writes the packet, and the commit gets a trailer that points to it:
Proof-Of-Change: p-2f007850 subject=sha256:2ad8c12023b7 policy=sha256:f21acbea749e proven=1/1 attested=0/0
In CI, pocp verify origin/main..HEAD --require-coverage --rerun (the base branch, then the pull request's head):
re-runs every required check itself, instead of trusting the packets, which are only pointers;
judges the pull request by the base branch's rules, so the pull request can't remove or loosen a rule or add its own signing key;
when the pull request edits existing tests or test setup, runs the test check once, on the pull request's net change, so a weakening split across commits or placed in a merge commit is judged the same way. It has two modes (below);
fails with a named reason, and says when only a maintainer can resolve it, for example by approving an intended test change with a label. On GitHub, a separate trusted workflow turns that into one commit status, pocp: green when verified, yellow while it waits for a maintainer or a check didn't finish, red on a real failure (docs/STATUS.md).