zencefilefendi/kadi

**A calibrated judge in front of your frontier model.**
Routine decisions are answered locally, with a distribution-free bound on how often they disagree with the oracle.
Everything else is escalated, and every escalation teaches the local judge.
Roadmap · Engineering report · Theory · Threat model · MCP server · Türkçe · TypeScript client
---
Abstract
LLM-based agents and applications make large numbers of small, schema-shaped decisions: route a ticket, flag spam, pick the next tool, approve a change. Sending each one to a frontier model is accurate but slow and expensive. Sending each one to a small model is cheap, but its error rate is unknown.
Kadı is a three-tier cascade: exact memory → local reflex → oracle. The middle tier answers **only when a statistical certificate allows it**: either a split-conformal prediction set with at most one label, or a Learn-then-Test threshold. Under exchangeability, this bounds the probability that a returned decision disagrees with the oracle by a user-chosen α. The bound can be marginal or per label (Mondrian). With Learn then Test it holds **with probability ≥ 1 − δ, uniformly over time**.
Doing this in an *online* system takes four mechanisms (v0.6):
- **Frozen model versions.** A live model learns from every oracle label, but only frozen snapshots answer, and each is certified on calibration samples drawn *after* its freeze. So the scored model is genuinely independent of its calibration data, with no re-scoring and no leakage.
- **Random calibration samples.** The sampling coin never depends on the input. Calibrating on escalated examples is never done; §5 of the theory notes shows how that shortcut breaks the bound.
- **Anytime-valid certification.** Betting e-processes with Ville's inequality, and an error budget δ shared across versions, so every certificate ever issued holds at every point in time.
- **An e-detector on the deployed risk.** It has a bounded false-alarm rate (ARL), and on an alarm the served version is re-certified.
Kadı adds a novelty guard against out-of-distribution and adversarial inputs, shadow-mode deployment, PII redaction, a tamper-evident ledger and a hardened HTTP/MCP service. It is written in pure standard-library Python with zero dependencies.
How it works
| Tier | Latency | Cost | Answers when |
|---|---|---|---|
| **Memory** | µs | $0 | the exact input was judged before |
| **Reflex** | ms | $0 | the served frozen version's certificate allows it (and the guards pass) |
| **Oracle** | s | $$ | anything else; its verdict calibrates frozen versions and trains the live model |
| **Fallback** | ms | $0 | the oracle is down and on_oracle_error = "best_guess"; flagged, no guarantee |
The guarantee in one line
$$
\Pr\big(\hat{y} \neq O(x)\big) \;\le\; \Pr\big(O(x) \notin C(x)\big) \;\le\; \alpha
$$
A disagreement requires a local answer, and a local answer requires |C(x)| ≤ 1 (an empty set answers the argmax, which is already counted as a miscoverage). The full statement, its assumptions, the Mondrian variant, the ACI bound and **what is not guaranteed** are in docs/THEORY.md.
Two policies
| policy | Answers locally when | Guarantee | Best for |
|---|---|---|---|
| "conformal" (default) | the conformal set has at most one label | Pr(disagree) ≤ α **in expectation**; ACI adapts to drift; optional **per-label** bound (conditional=true) | the most local answers at a given α; rare-but-critical labels |
| "ltt" | max p ≥ λ̂, where λ̂ is certified by **anytime-valid Learn then Test** | **with probability ≥ 1 − δ, at every point in time**, Pr(disagree) ≤ α | when you need a high-probability, time-uniform guarantee (audits, regulated use) |
Why not just a confidence threshold?
| | Confidence threshold / vendor router | Kadı |
|---|---|---|
| When it answers locally | Uncalibrated score > hand-tuned cut-off | A frozen model version's certificate allows it (conformal set ≤ 1 label, or LTT threshold) |
| Error statement | Benchmark accuracy on someone else's data | **Disagreement with *your* oracle ≤ α**, per question, optionally per label |
| Under drift | Silent degradation | ACI re-tunes α t ; an e-detector with bounded false-alarm rate re-certifies, escalates and alerts |
| Evidence | Offline eval | Continuous random audits on live traffic, shown on the dashboard |