mclenhard/mcp-evals

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring, with built-in observability support. This helps ensure your MCP server's tools are working correctly, performing well, and are fully observable with integrated monitoring and metrics.
## Installation
### As a Node.js Package
bash
npm install mcp-evals
### As a GitHub Action
Add the following to your workflow file:
yaml
name: Run MCP Evaluations
on:
pullrequest:
types: [opened, synchronize, reopened]
jobs:
evaluate:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
uses: actions/checkout@v4
name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
name: Install dependencies
run: npm install
name: Run MCP Evaluations
uses: mclenhard/mcp-evals@v1.0.9
with:
evalspath: 'src/evals/evals.ts' # Can also use .yaml files
serverpath: 'src/index.ts'
openaiapikey: ${{ secrets.OPENAIAPIKEY }}
model: 'gpt-4' # Optional, defaults to gpt-4
## Usage -- Evals
### 1. Create Your Evaluation File
You can create evaluation configurations in either TypeScript or YAML format.
#### Option A: TypeScript Configuration
Create a file (e.g., evals.ts) that exports your evaluation configuration:
typescript
import { EvalConfig } from 'mcp-evals';
import { openai } from "@ai-sdk/openai";
import { grade, EvalFunction} from "mcp-evals";
const weatherEval: EvalFunction = {
name: 'Weather Tool Evaluation',
description: 'Evaluates the accuracy and completeness of weather information retrieval',
run: async () => {
const result = await grade(openai("gpt-4"), "What is the weather in New York?");
return JSON.parse(result);
};
const config: EvalConfig = {
model: openai("gpt-4"),
evals: [weatherEval]
};
export default config;
export const evals = [
weatherEval,
// add other evals here
];
#### Option B: YAML Configuration
For simpler configuration, you can use YAML format (e.g., evals.yaml):
yaml
# Model configuration
model:
provider: openai # 'openai' or 'anthropic'
name: gpt-4o # Model name
# apikey: sk-... # Optional, uses OPENAIAPIKEY env var by default
# List of evaluations to run
evals:
name: weatherquerybasic
description: Test basic weather information retrieval
prompt: "What is the current weather in San Francisco?"
expectedresult: "Should return current weather data for San Francisco including temperature, conditions, etc."
name: weatherforecast
description: Test weather forecast functionality
prompt: "Can you give me the 3-day weather forecast for Seattle?"
expectedresult: "Should return a multi-day forecast for Seattle"
name: invalidlocation
description: Test handling of invalid location requests
prompt: "What's the weather in Atlantis?"
expectedresult: "Should handle invalid location gracefully with appropriate error message"
### 2. Run the Evaluations
#### As a Node.js Package
You can run the evaluations using the CLI with either TypeScript or YAML files:
bash
# Using TypeScript configuration
npx mcp-eval path/to/your/evals.ts path/to/your/server.ts
# Using YAML configuration
npx mcp-eval path/to/your/evals.yaml path/to/your/server.ts
#### As a GitHub Action
The action will automatically:
1. Run your evaluations