Jwrede/llmprobe

> Synthetic monitoring and CI smoke tests for LLM inference endpoints. Measure TTFT, latency, throughput, and errors. Single binary, zero SDKs.
llmprobe is a CLI tool for LLM serving reliability. It probes hosted APIs or
OpenAI-compatible inference servers, then reports the metrics that matter for
production user experience: time to first token (TTFT), total latency,
generation throughput (tokens/sec), and error rates.
Use it as a one-off health check, a continuous monitor, or a CI gate that
blocks deploys when your LLM provider is degraded.
Public benchmark
llm-bench uses llmprobe to run a
continuous public benchmark of major LLM APIs. It publishes a live dashboard at
bench.jonathanwrede.de and raw JSONL data in
Jwrede/llm-bench-data.
This is the intended use case: repeated synthetic probes that make LLM latency,
TTFT regressions, throughput drops, and provider degradation visible before
users report them.
Install
**Download a prebuilt binary** from the
latest release
(Linux, macOS, Windows; amd64 and arm64).
Or install from source:
Claude Code plugin
Install as a Claude Code plugin for /llmprobe skill and MCP tools:
Or register the MCP server directly:
llmprobe runs locally and only contacts LLM endpoints you configure.
See PRIVACY.md for details.
Quick start
llmprobe works with OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock, and
OpenAI-compatible endpoints such as vLLM, Ollama, OpenRouter, Groq, Together AI,
Fireworks, DeepSeek, and Mistral.
Create a probes.yml (or copy the included example):
Run a probe:
What it measures
| Metric | What it means |
|--------|--------------|
| **TTFT** | Time from request send to first content token. This is what users feel as "lag" before the response starts streaming. |
| **Latency** | Total time from request to stream close. |
| **Tok/s** | Generation throughput: tokens produced per second after the first token. Calculated as token_count / (latency - ttft). |
| **Tokens** | Total output tokens. Prefers provider usage metadata when available, falls back to SSE event counting. |
| **Status** | healthy if all thresholds pass, degraded if any threshold is exceeded, error if the request failed. |
Commands
llmprobe probe
One-off health check. Probes all configured endpoints and prints results.
Exit codes for CI:
| --fail-on | Exit 0 | Exit 1 |
|-------------|--------|--------|
| error (default) | healthy or degraded | any error |
| degraded | healthy only | degraded or error |
| none | always | never |
llmprobe watch
Continuous monitoring. Probes all endpoints on an interval and prints a
summary line per iteration.
The --tui flag launches a live terminal dashboard with a TTFT chart,
color legend, and statistics table. Use --load to import historical
JSONL data (from llmprobe watch -f json > data.jsonl).
llmprobe report
Generate a Markdown summary from JSONL probe data with p50/p95/p99
percentiles for TTFT, latency, and throughput per endpoint.
Output:
llmprobe baseline
Create a baseline file from historical JSONL data for regression detection.
Reference the baseline in your config to use multiplier-based thresholds:
This lets you detect regressions relative to your own historical data rather
than setting absolute thresholds.
llmprobe version
Print the installed binary version.
CI integration
Use llmprobe probe as a pre-deploy gate:
This blocks the deploy if any LLM provider is experiencing degraded
performance right now.
When a probe fails, the output shows only the failing endpoints:
MCP server
llmprobe includes a built-in Model Context Protocol
server, allowing Claude Code and other MCP hosts to check LLM API health