U
Ustay-coder/verdict
Write the rubric once.
updated 2026-11-19
View on GitHub ↗Direct link · no redirect page
We have no copy-paste install command and no public remote endpoint, so we do not print a config. The step above is reference only, not a command we can vouch for. A made-up one is worse than none. How each client wires MCP up →
- Star count
- 0 ★ · Ustay-coder/verdict · read from the GitHub search API · (checked 2026-11-19 20:00:48)
- Source
- collected from the punkpeye/awesome-mcp-servers list · type: MCP server
- Harvest history
- first seen 2026-11-18 · last seen 2026-11-20 · seen 3× · listed by 3 sources
- Score
- score 300 · 3 sources · mcp
FAQ (answers come from our records)
Who is it for?
We filed it under these tags: MCP servers. The tags come from the entry itself, not a taxonomy we imposed.
Who maintains it?
repository
Ustay-coder/verdict;first catalogued 2026-11-18;last seen 2026-11-20。There is no official certification or quality rating here — we record sources and dates only.README excerpt (English, untranslated) · Read the full README on GitHub →
# Verdict — one rubric, every decision model
Excerpted from the repository README (abridged)
Write the rubric once. Run it on Jev, Clef, OpenAI's Decisions API, Gemini's and any other decision model, and see where they disagree on your own cases.
Decision models answer typed questions (yes or no, one of these, a level on a scale) with probabilities instead of prose, in a fraction of a second. Every vendor publishes benchmarks; none of them is about your cases, and each vendor wants its questions in its own shape. Verdict turns a rubric — criteria, scales, levels — into the questions each model takes, sends the same questions to all of them, and puts the answers side by side.
bash
npm install -g @verdict-ai/cli
verdict run -r examples/banking77/intent.yaml -d examples/banking77/cases.jsonl --models jev,clef,clef-flash,openrouter,openai,gemini
text
Transfer intent × cases.jsonl: 80 case(s), 6 model(s), the same questions to each
model answered by agreed with labels failed asked a human input tokens calls time
jev typesafe/jev-1.14.0 66/80 (83%) 0 0 44,560 80 2.1 s
clef cloudflare/clef 70/80 (88%) 0 0 31,259 80 6.9 s
clef-flash cloudflare/clef-flash 73/80 (91%) 0 0 31,259 80 4.5 s
openrouter openrouter/typesafe/jev-1.13-20260917 67/80 (84%) 0 0 44,560 80 3.9 s
openai openai/gpt-6-luna 69/80 (86%) 0 0 25,254 80 7.1 s
gemini gemini/gemini-decision-1 68/80 (85%) 0 0 25,364 80 7.1 s
the models gave one answer on 70 of 80 question(s) they all answered; they differ on 10:
A real run (2026-11-12) of eighty customer messages from a public dataset, on six models: examples/banking77 has the rubric, the cases, the whole output and how to read it. It is not a ranking: eighty cases separate the models by a handful of answers. The ten cases where they differ are the ones worth reading, and on your cases they will be different ones.
## Try it
Node 22 or newer. No key is needed for the first run:
bash
npm install -g @verdict-ai/cli
git clone https://github.com/Ustay-coder/verdict && cd verdict/examples/banking77
verdict run -r intent.yaml -d cases.jsonl --models mock # a stand-in model: shows the flow, not real answers
With a key for any one service, name its model instead (TYPESAFE_API_KEY for jev; CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN for clef; OPENROUTER_API_KEY; OPENAI_API_KEY). A model that cannot answer keeps its row and says why.
Then your own cases:
bash
verdict rubric init rubric.yaml # a starter rubric to edit
verdict run -r rubric.yaml -d cases.jsonl --models jev,clef # cases.jsonl: one {"id", "inputs", "expected"} per line
verdict diff --base run_… --head run_… # any two runs, case by case
The rubric file is not edited: --models re-points its decision judges, one run per model, and every run is kept. Getting started walks through it; Decision models explains the table.
## Models and inputs
| Name it | Model | Service | Checked against the live service | Pictures and PDFs |
| jev | Jev | TypeSafe | Yes | No, text only |
| clef | Clef | Cloudflare Workers AI | Yes | Pictures, up to 4 |
| clef-flash | Clef-flash | Cloudflare Workers AI | Yes | Pictures, up to 4 |
| openrouter: | For example typesafe/jev-1.13 | OpenRouter | Yes | No |
| openai: | OpenAI's Decisions API, for example gpt-6-luna | OpenAI | Yes | Pictures |
Decision models answer typed questions (yes or no, one of these, a level on a scale) with probabilities instead of prose, in a fraction of a second. Every vendor publishes benchmarks; none of them is about your cases, and each vendor wants its questions in its own shape. Verdict turns a rubric — criteria, scales, levels — into the questions each model takes, sends the same questions to all of them, and puts the answers side by side.
bash
npm install -g @verdict-ai/cli
verdict run -r examples/banking77/intent.yaml -d examples/banking77/cases.jsonl --models jev,clef,clef-flash,openrouter,openai,gemini
text
Transfer intent × cases.jsonl: 80 case(s), 6 model(s), the same questions to each
model answered by agreed with labels failed asked a human input tokens calls time
jev typesafe/jev-1.14.0 66/80 (83%) 0 0 44,560 80 2.1 s
clef cloudflare/clef 70/80 (88%) 0 0 31,259 80 6.9 s
clef-flash cloudflare/clef-flash 73/80 (91%) 0 0 31,259 80 4.5 s
openrouter openrouter/typesafe/jev-1.13-20260917 67/80 (84%) 0 0 44,560 80 3.9 s
openai openai/gpt-6-luna 69/80 (86%) 0 0 25,254 80 7.1 s
gemini gemini/gemini-decision-1 68/80 (85%) 0 0 25,364 80 7.1 s
the models gave one answer on 70 of 80 question(s) they all answered; they differ on 10:
A real run (2026-11-12) of eighty customer messages from a public dataset, on six models: examples/banking77 has the rubric, the cases, the whole output and how to read it. It is not a ranking: eighty cases separate the models by a handful of answers. The ten cases where they differ are the ones worth reading, and on your cases they will be different ones.
## Try it
Node 22 or newer. No key is needed for the first run:
bash
npm install -g @verdict-ai/cli
git clone https://github.com/Ustay-coder/verdict && cd verdict/examples/banking77
verdict run -r intent.yaml -d cases.jsonl --models mock # a stand-in model: shows the flow, not real answers
With a key for any one service, name its model instead (TYPESAFE_API_KEY for jev; CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN for clef; OPENROUTER_API_KEY; OPENAI_API_KEY). A model that cannot answer keeps its row and says why.
Then your own cases:
bash
verdict rubric init rubric.yaml # a starter rubric to edit
verdict run -r rubric.yaml -d cases.jsonl --models jev,clef # cases.jsonl: one {"id", "inputs", "expected"} per line
verdict diff --base run_… --head run_… # any two runs, case by case
The rubric file is not edited: --models re-points its decision judges, one run per model, and every run is kept. Getting started walks through it; Decision models explains the table.
## Models and inputs
| Name it | Model | Service | Checked against the live service | Pictures and PDFs |
| jev | Jev | TypeSafe | Yes | No, text only |
| clef | Clef | Cloudflare Workers AI | Yes | Pictures, up to 4 |
| clef-flash | Clef-flash | Cloudflare Workers AI | Yes | Pictures, up to 4 |
| openrouter: | For example typesafe/jev-1.13 | OpenRouter | Yes | No |
| openai: | OpenAI's Decisions API, for example gpt-6-luna | OpenAI | Yes | Pictures |
Listed badge (optional)
Copy this line into the README if you want the listing mark. Optional, and it does not affect being listed.
[](https://anclum.com/en/s/ustay-coder-verdict/)
- Source
- awesome · https://github.com/Ustay-coder/verdict
- Browse by field
- MCP servers
- Listed by
- 3 independent sources
- added
- today
More MCP servers
Filed under
Every tag opens the full list for that filter. Counts are the live catalogue figures.