W
WendellResearch-AI/benchwarmers
★ 1·更新 2026-11-19
在 GitHub 查看 ↗直链 · 无中转页
我们没有可直接复制的安装命令,也没有公开的远端端点,所以不出配置。上面那条命令仅供参考,不是我们能担保的可复制命令——编一条出来比不给更糟。 各客户端的接法 →
- ★ 数取值
- 1 ★ · WendellResearch-AI/benchwarmers · 取自 GitHub 搜索 API · (2026-11-19 20:00:48 核对)
- 来源明细
- 收录自 punkpeye/awesome-mcp-servers 榜单(我们从中收集的条目之一) · 条目类型:MCP 服务
- 收录与活跃
- 收录于 2026-11-18 · 最近见到 2026-11-20 · 共见到 3 次 · 被 3 个来源列出
- 评分构成
- 综合分 300 · 1 ★ · 3 个来源 · mcp
常见问题(答案都来自我们的记录)
它适合谁 / 用来做什么?
我们把它归在这些标签下: MCP 服务。标签来自条目自身,不是我们强加的分类。
谁维护它、还在更新吗?
仓库
WendellResearch-AI/benchwarmers;我们收录于 2026-11-18;最近见到 2026-11-20。这里没有"官方认证"或"质量评级"——我们只记录来源与时间。README 原文(英文,未翻译) · 在 GitHub 读完整 README →
Put me in, coach. Every model on your bench gets its minutes.
以下摘录自该仓库的 README(片段)
Benchwarmers is a self-hosted panel gateway for LLMs. Your app asks for one
of your collections, and the whole panel answers: the models you picked
for that kind of work, plus explorer seats that rotate in the models you
have used least. Every answer comes back side by side, labelled by model and
seat. You check them against the real work and record a verdict, and those
verdicts decide who sits on the next panel.
What it is best at:
Asking several models at once. One request, in the OpenAI, Anthropic,
Gemini or Ollama format, a panel of answers. Where they disagree is where
to look.
Agents too. Claude Code and other tool-calling clients use a collection
through route mode: one member per turn, kept for the conversation, with
failover, and every call on the record.
Reviewing. Code review, claims about code, designs: different models
catch different things, and two models agreeing is not proof.
Playing to each model's strengths. A collection per kind of work
(reviewers, designers, enumerators), each with the models that do it best.
Giving every model playing time. Explorers cycle through the rest of
the bench, so new and free models get tried beside the ones you trust,
without you choosing them by hand.
Benchmarking on your own projects. A review suite seeded from your own
repositories, plus your verdicts on real work, instead of public leaderboards.
Knowing who to bench. Who is confidently wrong, who has stopped
answering (retired automatically), and who is resting after a rate limit.
Keeping free and cheap models useful. Health probes, rate limits taken
from what providers say and enforced, and one endpoint for every app, over
any OpenAI-compatible provider or hub, or Anthropic's and Gemini's own APIs.
| Related tools | Examples | What they do | What Benchwarmers does differently |
| Gateways and routers | LiteLLM, OpenRouter, Portkey | One request to one model, with fallback | Fans one prompt out to a panel and returns every answer; for agents, routes to one member by its record |
| Councils and ensembles | LLM Council, Mixture-of-Agents | Several models merged into one answer | Keeps every answer, because who said what is the evidence; one merged summary only if you ask |
| Side-by-side chat | Open WebUI, LMArena | Several answers shown together | Records a verdict per answer and uses them to pick the next panel |
| Evaluation harnesses | promptfoo, Inspect, SWE-bench | Score models offline on fixed test sets | Scores them on your real work and your own code, and keeps rotating new models in |
## Contents
Try it
How it works
Asking a panel
From the command line
From Claude Code and other agents
The suite
Docker and hosting
What leaves your machine
Limits
Security
Development
Licence
Day-to-day use (the dashboard's pages, connecting apps, onboarding in bulk) is
in USAGE.md. Running it on a server or a hosting platform is in
HOSTING.md. A coding agent setting it up for someone should start
with AGENTS.md.
## Try it
Pre-1.0, early. It runs with Python 3.11 or later on Windows, macOS and
Linux, from PyPI, npm, a checkout or Docker. Expect rough edges, and read the
code where these pages are thin.
Quickest, with Python 3.11+ installed, from PyPI:
of your collections, and the whole panel answers: the models you picked
for that kind of work, plus explorer seats that rotate in the models you
have used least. Every answer comes back side by side, labelled by model and
seat. You check them against the real work and record a verdict, and those
verdicts decide who sits on the next panel.
What it is best at:
Asking several models at once. One request, in the OpenAI, Anthropic,
Gemini or Ollama format, a panel of answers. Where they disagree is where
to look.
Agents too. Claude Code and other tool-calling clients use a collection
through route mode: one member per turn, kept for the conversation, with
failover, and every call on the record.
Reviewing. Code review, claims about code, designs: different models
catch different things, and two models agreeing is not proof.
Playing to each model's strengths. A collection per kind of work
(reviewers, designers, enumerators), each with the models that do it best.
Giving every model playing time. Explorers cycle through the rest of
the bench, so new and free models get tried beside the ones you trust,
without you choosing them by hand.
Benchmarking on your own projects. A review suite seeded from your own
repositories, plus your verdicts on real work, instead of public leaderboards.
Knowing who to bench. Who is confidently wrong, who has stopped
answering (retired automatically), and who is resting after a rate limit.
Keeping free and cheap models useful. Health probes, rate limits taken
from what providers say and enforced, and one endpoint for every app, over
any OpenAI-compatible provider or hub, or Anthropic's and Gemini's own APIs.
| Related tools | Examples | What they do | What Benchwarmers does differently |
| Gateways and routers | LiteLLM, OpenRouter, Portkey | One request to one model, with fallback | Fans one prompt out to a panel and returns every answer; for agents, routes to one member by its record |
| Councils and ensembles | LLM Council, Mixture-of-Agents | Several models merged into one answer | Keeps every answer, because who said what is the evidence; one merged summary only if you ask |
| Side-by-side chat | Open WebUI, LMArena | Several answers shown together | Records a verdict per answer and uses them to pick the next panel |
| Evaluation harnesses | promptfoo, Inspect, SWE-bench | Score models offline on fixed test sets | Scores them on your real work and your own code, and keeps rotating new models in |
## Contents
Try it
How it works
Asking a panel
From the command line
From Claude Code and other agents
The suite
Docker and hosting
What leaves your machine
Limits
Security
Development
Licence
Day-to-day use (the dashboard's pages, connecting apps, onboarding in bulk) is
in USAGE.md. Running it on a server or a hosting platform is in
HOSTING.md. A coding agent setting it up for someone should start
with AGENTS.md.
## Try it
Pre-1.0, early. It runs with Python 3.11 or later on Windows, macOS and
Linux, from PyPI, npm, a checkout or Docker. Expect rough edges, and read the
code where these pages are thin.
Quickest, with Python 3.11+ installed, from PyPI:
收录徽章(可选)
想把收录标记放进 README 就复制这行,可选,不影响收录。
[](https://anclum.com/zh/s/wendellresearch-ai-benchwarmers/)
- 来源
- awesome · https://github.com/WendellResearch-AI/benchwarmers
- 按领域浏览
- MCP 服务
- Traction
- 1 ★
- 收录来源
- 3 个独立来源
- 收录于
- 今天
更多 MCP 服务器
所属领域与标签
每个标签都能点进去看同类的全部条目。数字是当前库里的真实条数。