W
WendellResearch-AI/benchwarmers
★ 1·updated 2026-11-19
View on GitHub ↗Direct link · no redirect page
We have no copy-paste install command and no public remote endpoint, so we do not print a config. The step above is reference only, not a command we can vouch for. A made-up one is worse than none. How each client wires MCP up →
- Star count
- 1 ★ · WendellResearch-AI/benchwarmers · read from the GitHub search API · (checked 2026-11-19 20:00:48)
- Source
- collected from the punkpeye/awesome-mcp-servers list · type: MCP server
- Harvest history
- first seen 2026-11-18 · last seen 2026-11-20 · seen 3× · listed by 3 sources
- Score
- score 300 · 1 ★ · 3 sources · mcp
FAQ (answers come from our records)
Who is it for?
We filed it under these tags: MCP servers. The tags come from the entry itself, not a taxonomy we imposed.
Who maintains it?
repository
WendellResearch-AI/benchwarmers;first catalogued 2026-11-18;last seen 2026-11-20。There is no official certification or quality rating here — we record sources and dates only.README excerpt (English, untranslated) · Read the full README on GitHub →
Put me in, coach. Every model on your bench gets its minutes.
Excerpted from the repository README (abridged)
Benchwarmers is a self-hosted panel gateway for LLMs. Your app asks for one
of your collections, and the whole panel answers: the models you picked
for that kind of work, plus explorer seats that rotate in the models you
have used least. Every answer comes back side by side, labelled by model and
seat. You check them against the real work and record a verdict, and those
verdicts decide who sits on the next panel.
What it is best at:
Asking several models at once. One request, in the OpenAI, Anthropic,
Gemini or Ollama format, a panel of answers. Where they disagree is where
to look.
Agents too. Claude Code and other tool-calling clients use a collection
through route mode: one member per turn, kept for the conversation, with
failover, and every call on the record.
Reviewing. Code review, claims about code, designs: different models
catch different things, and two models agreeing is not proof.
Playing to each model's strengths. A collection per kind of work
(reviewers, designers, enumerators), each with the models that do it best.
Giving every model playing time. Explorers cycle through the rest of
the bench, so new and free models get tried beside the ones you trust,
without you choosing them by hand.
Benchmarking on your own projects. A review suite seeded from your own
repositories, plus your verdicts on real work, instead of public leaderboards.
Knowing who to bench. Who is confidently wrong, who has stopped
answering (retired automatically), and who is resting after a rate limit.
Keeping free and cheap models useful. Health probes, rate limits taken
from what providers say and enforced, and one endpoint for every app, over
any OpenAI-compatible provider or hub, or Anthropic's and Gemini's own APIs.
| Related tools | Examples | What they do | What Benchwarmers does differently |
| Gateways and routers | LiteLLM, OpenRouter, Portkey | One request to one model, with fallback | Fans one prompt out to a panel and returns every answer; for agents, routes to one member by its record |
| Councils and ensembles | LLM Council, Mixture-of-Agents | Several models merged into one answer | Keeps every answer, because who said what is the evidence; one merged summary only if you ask |
| Side-by-side chat | Open WebUI, LMArena | Several answers shown together | Records a verdict per answer and uses them to pick the next panel |
| Evaluation harnesses | promptfoo, Inspect, SWE-bench | Score models offline on fixed test sets | Scores them on your real work and your own code, and keeps rotating new models in |
## Contents
Try it
How it works
Asking a panel
From the command line
From Claude Code and other agents
The suite
Docker and hosting
What leaves your machine
Limits
Security
Development
Licence
Day-to-day use (the dashboard's pages, connecting apps, onboarding in bulk) is
in USAGE.md. Running it on a server or a hosting platform is in
HOSTING.md. A coding agent setting it up for someone should start
with AGENTS.md.
## Try it
Pre-1.0, early. It runs with Python 3.11 or later on Windows, macOS and
Linux, from PyPI, npm, a checkout or Docker. Expect rough edges, and read the
code where these pages are thin.
Quickest, with Python 3.11+ installed, from PyPI:
of your collections, and the whole panel answers: the models you picked
for that kind of work, plus explorer seats that rotate in the models you
have used least. Every answer comes back side by side, labelled by model and
seat. You check them against the real work and record a verdict, and those
verdicts decide who sits on the next panel.
What it is best at:
Asking several models at once. One request, in the OpenAI, Anthropic,
Gemini or Ollama format, a panel of answers. Where they disagree is where
to look.
Agents too. Claude Code and other tool-calling clients use a collection
through route mode: one member per turn, kept for the conversation, with
failover, and every call on the record.
Reviewing. Code review, claims about code, designs: different models
catch different things, and two models agreeing is not proof.
Playing to each model's strengths. A collection per kind of work
(reviewers, designers, enumerators), each with the models that do it best.
Giving every model playing time. Explorers cycle through the rest of
the bench, so new and free models get tried beside the ones you trust,
without you choosing them by hand.
Benchmarking on your own projects. A review suite seeded from your own
repositories, plus your verdicts on real work, instead of public leaderboards.
Knowing who to bench. Who is confidently wrong, who has stopped
answering (retired automatically), and who is resting after a rate limit.
Keeping free and cheap models useful. Health probes, rate limits taken
from what providers say and enforced, and one endpoint for every app, over
any OpenAI-compatible provider or hub, or Anthropic's and Gemini's own APIs.
| Related tools | Examples | What they do | What Benchwarmers does differently |
| Gateways and routers | LiteLLM, OpenRouter, Portkey | One request to one model, with fallback | Fans one prompt out to a panel and returns every answer; for agents, routes to one member by its record |
| Councils and ensembles | LLM Council, Mixture-of-Agents | Several models merged into one answer | Keeps every answer, because who said what is the evidence; one merged summary only if you ask |
| Side-by-side chat | Open WebUI, LMArena | Several answers shown together | Records a verdict per answer and uses them to pick the next panel |
| Evaluation harnesses | promptfoo, Inspect, SWE-bench | Score models offline on fixed test sets | Scores them on your real work and your own code, and keeps rotating new models in |
## Contents
Try it
How it works
Asking a panel
From the command line
From Claude Code and other agents
The suite
Docker and hosting
What leaves your machine
Limits
Security
Development
Licence
Day-to-day use (the dashboard's pages, connecting apps, onboarding in bulk) is
in USAGE.md. Running it on a server or a hosting platform is in
HOSTING.md. A coding agent setting it up for someone should start
with AGENTS.md.
## Try it
Pre-1.0, early. It runs with Python 3.11 or later on Windows, macOS and
Linux, from PyPI, npm, a checkout or Docker. Expect rough edges, and read the
code where these pages are thin.
Quickest, with Python 3.11+ installed, from PyPI:
Listed badge (optional)
Copy this line into the README if you want the listing mark. Optional, and it does not affect being listed.
[](https://anclum.com/en/s/wendellresearch-ai-benchwarmers/)
- Source
- awesome · https://github.com/WendellResearch-AI/benchwarmers
- Browse by field
- MCP servers
- Traction
- 1 ★
- Listed by
- 3 independent sources
- added
- today
More MCP servers
Filed under
Every tag opens the full list for that filter. Counts are the live catalogue figures.