SLP-DEV1/claude-code-canary

GitHub Action ·
Plugin demo ·
Plugin suites ·
MCP contracts ·
Agent teams ·
Doctor ·
Distribution ·
Security
Claude Code Canary
**Deterministic regression testing and compatibility intelligence for Claude Code.**
Claude Code changes. Your CLAUDE.md, hooks, plugins, MCP servers and permission rules change too. Canary gives you a repeatable answer to the question that normally becomes guesswork:
> **What broke, and which Claude Code release first broke it?**
Canary runs the same scenario from the same Git commit in disposable worktrees, captures tool/token/reported-cost/duration metrics, checks deterministic assertions, compares releases, bisects regressions and builds full plugin compatibility matrices.
What v2 adds
Canary v2 turns those individual checks into a compatibility platform:
- first-class scenario suites with tags, affected-path selection, concurrency, deterministic sharding and run budgets;
- scheduled release watch with known-good state, regression detection and automatic first-bad-release bisection;
- deterministic failure fingerprints and flakiness analysis so noisy scenarios are not confused with upstream regressions;
- portable HTML, JUnit and SARIF reporting plus local historical trends;
- compatibility manifests, canary.lock, open registry aggregation, evidence-backed badges and inspectable scenario packs;
- permission-policy/trust regression coverage, isolated MCP fixtures, gateway matrices and signed/checksummed attestations;
- versioned public schemas plus compatibility query/explain/graph APIs for build tools and multi-project workspaces.
All existing v1 workflows remain available through the v2 CLI.
Start in 30 seconds
Install the published CLI and verify the host before spending tokens:
Already using GitHub Actions? Add Canary as a CI step and use the stable @v2 major channel:
That gives you the same scenario locally and in CI, with machine-readable results in .canary/results/ and a Markdown summary in GitHub Actions.
**Choose the workflow that matches the problem:**
| Goal | Command / Action mode |
| --- | --- |
| Catch a Claude Code release regression | compare |
| Find the first bad Claude Code release | bisect |
| Gate a repository pull request | pr-check |
| Keep recurring CI to one Claude run | baseline-check |
| Test an entire Claude Code plugin surface | plugin-suite |
| Detect MCP schema/capability drift | mcp-check |
| Run a deterministic scenario suite | suite |
| Guard against newly published Claude Code releases | watch |
| Measure scenario stability/noise | flake |
| Publish/query portable compatibility evidence | compat / lock |
| Generate interoperable local/CI reports | report / trend |
| Check host/plugin/MCP readiness without exposing secrets | doctor |
The 30-second demo
Plugin author? Turn your plugin surface into smoke tests, then test every generated scenario across recent Claude Code releases:
You get one report like this:
Need the exact regression boundary instead?
Why Canary is different
| Problem | Canary |
| --- | --- |
| "The new Claude release feels worse" | Run the same deterministic scenario on both releases. |
| "Which release broke us?" | Binary-search the real published Claude Code release range. |
| "Does this plugin still work?" | Generate smoke tests and run a release × component compatibility suite. |
| "Is my new CLAUDE.md actually better?" | Run interleaved A/B configuration experiments. |
| "This worked yesterday" | Record a good real task and replay it from the exact original commit. |
| "How do I report this bug safely?" | Export a bounded, redacted reproduction bundle. |
| "Did agent usage blow up?" | Track tool calls, tokens, duration and reported cost. |
| "Did this PR make the agent worse?" | Compare base vs head with the same Claude executable and fail on configured deltas. |
| "Can CI do this without paying for two runs every time?" | Commit a known-good metric baseline and execute only the candidate. |
| "Did my MCP server silently change?" | Snapshot tools/prompts/resources and fail on removed tools, schema changes or capability regressions. |