Akhilesh-Vangala/healthcare-evidence-agent

A governed AI agent that answers biotech commercial, medical and government-affairs questions over FDA labels, ClinicalTrials.gov and Medicare coverage decisions. It can only cite a document a tool retrieved in that run. Everything else is a grader's decision, not the model's:
Tools: six MCP tools over three public APIs.
Procedure: four Agent Skills, loaded only when needed.
Grader: accepts a claim only if its sourceid was retrieved in that run and its quote is a verbatim span of that document.
> Public data only. Not medical advice. No PHI.
Open the console ↗ · Study · Cross-runtime check · Methodology · Architecture · For reviewers
The console calls the agent's own tools live, runs a live citation check against the real document, replays recorded
real runs and shows the published study. Live answering runs only where a model key and rate and spend limits are
configured.
## At a glance
| What was measured | Result |
| Golden-set pass rate, tools + skills (240-run study, Claude Haiku 5.5) | 95% (95% CI 87–100%), vs 20% for a fixed pipeline |
| Effect of adding MCP tools · then Agent Skills (paired by task) | +57 pts [+33, +77] · +15 pts [+2, +30] |
| Invented clinical-trial IDs with tools | 0, vs 4 runs closed book |
| Same configuration on a second runtime (Anthropic Messages API, 28 tasks) | 92.9% vs 94.6%, paired −1.8 pts [−7.1, +3.6] |
| Cost per task with prompt caching · input read from cache | $0.0033 → $0.0019 · 63% |
| Claim precision on the API runtime | 97.9% |
| Red-team suite (injection, promotion, dosing, off-label, spoofing, comparison) | 10 of 12 runs held |
| Larger model (Sonnet 5.5) instead of Haiku | +1.7 pts, not significant, at about 16× the cost |
| Tests in CI | 47 pytest (tools mocked) · 20 Playwright + axe WCAG 2 AA · published analysis re-derived from committed reports |
## Stack
| Layer | Technology |
| Agent runtimes | Headless Claude Code (claude -p, MCP over stdio) and the Anthropic Python SDK (tool-use loop, prompt caching) |
| Tools and procedure | MCP server (mcp SDK) with six tools; four Agent Skills loaded on demand |
| Data sources | openFDA drug labels, ClinicalTrials.gov v2, CMS National Coverage Determinations (httpx, cached) |
| Evaluation | Deterministic grader, paired bootstrap confidence intervals, offline re-grading of saved runs |
| Service | FastAPI, server-sent events, Upstash Redis rate limits and daily USD cap that fail closed |
| Console | React 18, TypeScript, Vite; Playwright and axe tests |
| Infrastructure | Vercel (console and API as separate projects), GitHub Actions |
## Results in detail
### Published study: 240 runs on headless Claude Code (Claude Haiku 5.5)
| Configuration | Pass rate (95% CI) | Claims verified | Cost / task | Median latency |
| Fixed pipeline (no LLM) | 20% (7–37%) | n/a† | $0 | 0.1 s |
| A. closed book | 23% (10–40%) | 0% | $0.0014 | 10 s |
| B. + MCP tools | 80% (63–93%) | 89% | $0.0025 | 11 s |
| C. + tools + skills | 95% (87–100%) | 94% | $0.0033 | 14 s |
| D. C on Sonnet 5.5 | 97% (90–100%) | 94% | $0.0532 | 16 s |
† Baseline claims are fetched excerpts and verify by construction; the pipeline fails on product choice, refusal and not-found.