McCarryster/why_proebali_dota2

An AI agent that works out why a Dota 2 match was lost.
Give it a finished match and a question ("What were the key mistakes?"). It plans hypotheses, investigates each one in parallel with tools, checks every number it cites against the real match data, and writes a short report with a turning point and the evidence behind each finding.
This project is a showcase of AI agent engineering: the same agent architecture built twice, once from scratch on the Anthropic API and once with LangGraph, so the two approaches can be compared on identical tasks. Evaluation with RAGAS is planned.
## Status
| Piece | State |
| Baseline (single LLM call, no tools) | Done |
| From-scratch agent (planner, parallel investigators, reporter, orchestrator) | Done and working |
| LangGraph agent (same architecture, library version) | Planned |
| RAGAS evaluation across both systems | Planned |
## How the from-scratch agent works
mermaid
flowchart TD
Q[Question + match ID] --> P[Planner 1 to 5 testable hypotheses]
P --> I1[Investigator H1]
P --> I2[Investigator H2]
P --> I3[Investigator H3]
subgraph MCP[dota-tools MCP server]
T1[fetchmatch]
T2[getevents]
T3[verifyclaim]
end
I1 MCP
I2 MCP
I3 MCP
I1 --> S[Run state findings, dropped, tokens]
I2 --> S
I3 --> S
S --> D{Decide}
D -->|not enough evidence, budget left| P
D -->|finish| R[Reporter]
R --> OUT[Report: answer, turning point, verified findings, unresolved]
1. Planner reads a compact match digest and proposes 3 independent, falsifiable hypotheses (max 5), each a single sentence naming one cause.
2. Investigators run in parallel (asyncio.gather), one per hypothesis. Each is a tool-calling loop (up to 8 tool calls) that explores the match through MCP tools and ends by calling submitfinding.
3. Verification gate. Before a finding is accepted, every piece of evidence in it is re-checked by the deterministic verifyclaim tool. If a number or timestamp is wrong, the model gets the actual value back and gets a bounded retry. A finding that cannot be verified is dropped, never reported.
4. Orchestrator decision. After each round a plain-Python rule decides to finish or replan (asking the planner for new hypotheses, excluding ones already tried), based on evidence count, an iteration limit, and a token budget.
5. Reporter turns the verified findings into a 2-4 sentence answer, an optional turning point (only if a finding backs it), and a list of what remains unresolved.
### Design highlights
No LLM-as-judge for facts. Evidence is typed (stat, timeline, event) and checked against OpenDota data in code. Verification messages are written for the model to read, e.g. "goldadv at 1320s is -50352, not -30596", so retries fix the fact, not just the format.
Strict structured output. Pydantic schemas with cross-field validators (each evidence kind requires and forbids specific fields). Validation errors are fed back to the model for a retry. One validator exists because tool-call markup leaked into a claim string in a real run.
MCP for tools and prompts. A stdio MCP server (dota-tools) exposes three tools and the planner / investigator / reporter prompts. The agent talks to it through a small hand-written MCP bridge, the from-scratch counterpart of a library adapter.