aak204/MCP-Scorecard

Deterministic, CI-first quality scorecard for MCP servers.
MCP Scorecard is an open-source infrastructure tool for reviewing MCP servers before they enter
real workflows. It launches a server locally over stdio, discovers its tools, applies a
deterministic ruleset, and produces reviewable scores and findings across:
conformance
security
ergonomics
metadata
The output is built for CI: stable terminal summaries, a machine-readable JSON scorecard report,
and SARIF for code-scanning systems.
This project is intentionally not an AI wrapper. It does not depend on LLM scoring, hidden
judgment, or hosted analysis. The goal is a repeatable, auditable baseline that engineering teams
can gate on in pull requests and release pipelines.
Quick value:
run locally against a real MCP server
fail CI below a deterministic threshold
export JSON and SARIF for automation
review risky MCP surfaces deterministically
## What This Is
MCP Scorecard is a deterministic quality scorecard for MCP servers.
It is designed for cases where teams need to answer questions like:
Is this server surface reviewable before we adopt it?
Does it expose capabilities that deserve extra scrutiny in CI?
Are the tool names, descriptions, and schemas clear enough for human review?
Can we produce a stable machine-readable report for automation and policy?
Today the tool focuses on local stdio MCP servers and a deterministic scoring model that is easy
to explain, test, and version.
## Why This Exists
MCP servers are infrastructure. They define callable tool surfaces that agents, runtimes, and
automation can invoke. That means they should be reviewed with the same seriousness as other
integration boundaries.
In practice, teams often evaluate MCP servers ad hoc:
descriptions are vague
schemas are weak or underconstrained
high-risk capabilities are discovered late
CI has no consistent baseline
MCP Scorecard turns that first-line review into a deterministic contract:
run it locally
run it in CI
inspect category scores and findings
export JSON and SARIF
keep the result reviewable over time
## What It Checks
The current score model uses four explicit buckets.
Conformance here means deterministic interface-level conformance and schema reviewability checks,
not full protocol certification.
### Conformance
Checks whether the server surface is structurally well-formed and reviewable as an MCP interface.
Examples:
duplicate tool names
missing schema type
arbitrary top-level properties
critical input fields not marked required
### Security
Checks for exposed capabilities that materially increase blast radius or deserve explicit review.
Examples:
command execution
filesystem mutation
network and HTTP request primitives
download-and-execute patterns
### Ergonomics
Checks whether the server surface is understandable enough for humans and automation to review
without guessing.
Examples:
overly generic tool names
vague descriptions
weak input schemas
filesystem mutation tools without visible scope hints
### Metadata
Checks whether basic descriptive metadata is present and whether destructive behavior is made easy
to spot.
Examples:
missing tool descriptions
descriptions that explicitly advertise broad destructive access