Tool Definition Quality Score (TDQS)

TDQS is an open framework for scoring how well an MCP tool definition communicates to an AI agent. It is the methodology Glama uses to score every tool of every MCP server and hosted connector in its registry.
This document is the complete specification — the rubric, the exact prompts, the aggregation formulas, and the operational practices behind running it at registry scale — and is sufficient to reimplement TDQS end to end. If you maintain an MCP server and just want a better score, skip to Improving your score and use the rubric as a checklist.
- Scoring rationale: glama.ai/blog/2026-04-03-tool-definition-quality-score-tdqs
- Indexing methodology: glama.ai/mcp/methodology
- Live scores: every tool on glama.ai/mcp/tools has a public score page with the full per-dimension breakdown
Contents
- Why score tool definitions?
- What gets scored
- Scoring pipeline
- Stage 1: Context signals
- Stage 2: Hard gates
- Stage 3: LLM rubric evaluation
- Stage 4: Deterministic post-processing
- Computing the score
- Tiers
- Flags and smells
- Server-level scores
- 1. Tool definition quality (70% of overall)
- 2. Coherence (30% of overall)
- Shadowed tools
- Overall
- Output format
- Improving your score
- TDQS across the registry
- Running TDQS at scale
- How Glama uses the scores
- Appendix A: Tool scoring prompt
- Appendix B: Server coherence prompt
- References
Why score tool definitions?
Two empirical studies motivated the framework:
- "Model Context Protocol (MCP) Tool Descriptions Are Smelly!" audited 856 tools across 103 MCP servers and found that **97% of tool descriptions contain at least one quality defect**, **56% do not clearly state what the tool does**, and **89% never say when the tool should or should not be used**.
- "From Docs to Descriptions" analyzed 10,831 MCP servers and showed that **tools with well-written descriptions are selected ~260% more often** in competitive settings, and that rewriting descriptions alone improves downstream task success by ~6 percentage points.
The defect taxonomy in that literature is called "smells". TDQS borrows the term.
A registry that ranks and recommends servers needs a quality signal that is:
1. **Explainable**: every score comes with per-dimension justifications a maintainer can act on.
2. **Reproducible**: the same definition always produces the same evaluation inputs, so scores can be cached, diffed, and audited.
3. **Cheap enough to run on every schema change**: the ecosystem ships thousands of schema updates per day.
What gets scored
TDQS scores a **tool definition**, not tool behavior. The inputs are exactly what an MCP client sees from tools/list:
| Input | Type | Notes |
| ------------------ | ----------------------- | ---------------------------------------------------------------------------------- |
| name | string | required |
| title | string \| null | optional MCP display title |
| description | string \| null | the primary scoring target |
| inputSchema | JSON Schema \| null | parameter structure and per-parameter descriptions |
| outputSchema | JSON Schema \| null | passed to the evaluator in full; documented fields reduce the description's burden |
| annotations | object \| null | MCP hints: readOnlyHint, destructiveHint, idempotentHint, openWorldHint |
| siblingToolNames | string[] | names of the other tools on the same server |
Sibling tool names matter because selection is competitive: a description is only "clear" if it lets an agent distinguish this tool from its neighbors. Names are all it gets, though, so properties that exist only across a _pair_ of tools are evaluated at the server level instead — see Shadowed tools.
Scoring pipeline
Each tool flows through four stages. Stages 1, 2, and 4 are deterministic code; only stage 3 is an LLM call.
One thing happens outside this pipeline: server flags — defects belonging to the tool set rather than to any single definition — are written onto a tool's record asynchronously by the coherence job. See Shadowed tools.
Stage 1: Context signals
Before any judgment is made, deterministic code extracts structural facts about the definition. These ground the LLM evaluation (so it does not have to count parameters or guess at schema coverage) and are stored alongside the score. The invocation-cost signals — requiredFieldCount, schemaDepth, unionChoiceCount, and the invocationCost derived from them — are the exception: stored, but withheld from the tool-scoring prompt and consumed only by the server-level check. definitionBytes is likewise withheld, and nothing consumes it yet. hasOutputSchema is withheld as of v1.3: the prompt now carries the output schema itself, and a boolean restating whether that block is empty grounds nothing.