amirsasani/incident-response-copilot

An evidence-based incident investigation agent built with LangChain and LangGraph. It correlates logs, metrics, distributed traces, deployments and operational runbooks, then produces a structured root-cause report with traceable evidence IDs.
The project demonstrates how an AI agent can support site reliability and platform teams while keeping conclusions grounded in observable system data.
## Current status
[x] Create a synthetic incident dataset
[x] Implement log search
[x] Implement metric search
[x] Implement deployment search
[x] Implement runbook search
[x] Define typed investigation state and report schemas
[x] Build a LangGraph tool-calling workflow
[x] Generate structured incident reports with Ollama
[x] Validate report evidence against collected tool results
[x] Add isolated tool and graph tests
[x] Add a five-incident labelled evaluation dataset
[x] Evaluate root-cause accuracy and evidence recall
[x] Import and investigate an external RCAEval RE2 case
[x] Evaluate a balanced 12-case RCAEval subset
[x] Add distributed-trace search
[ ] Expose investigations through a FastAPI service
[ ] Add human approval before remediation actions
[ ] Add tracing and workflow observability
## How it works
mermaid
flowchart LR
A[Incident record] --> B[Load incident]
B --> C[LangGraph investigator]
C --> D{Tool calls required?}
D -->|Yes| E[Logs]
D -->|Yes| F[Metrics]
D -->|Yes| G[Traces]
D -->|Yes| H[Deployments]
D -->|Yes| L[Runbooks]
E --> I[Collect evidence]
F --> I
G --> I
H --> I
L --> I
I --> C
D -->|No| J[Structured report]
J --> K[Evidence validation]
The investigator receives the incident metadata and decides which tools to call. LangGraph executes those tools, adds their results to the conversation and stores normalized evidence in the shared state. When the investigation is complete, a separate structured-output call creates an IncidentReport.
The final report is rejected if it references an evidence ID that was not returned by a tool.
## Example incident
The included INC-001 scenario represents an increase in checkout failures and latency. The available evidence includes:
database connection timeout logs;
database pool, HTTP error-rate and latency metrics;
a deployment that reduced DBPOOLMAXSIZE from 30 to 5;
a database connection troubleshooting runbook.
The agent correlates these sources and identifies database connection pool exhaustion as the likely root cause.
## Project structure
text
incident-response-copilot/
├── data/
│ ├── incidents/ # Incident descriptions
│ ├── logs/ # JSON Lines application logs
│ ├── metrics/ # Time-series metric samples
│ ├── external/ # Locally imported public benchmark data
│ ├── runbooks/ # Operational runbooks
│ └── deployments.json # Deployment history
├── evaluation/
│ ├── incidentcases.json # Synthetic labelled cases
│ ├── rcaevalcases.json # Balanced external benchmark manifest
│ └── rcaevalresults.json
├── scripts/
│ ├── importrcaeval.py # Case-at-a-time RCAEval importer
│ ├── importrcaevalbatch.py
│ ├── evaluatercaeval.py
│ └── investigate.py # CLI entry point
├── src/
│ ├── agent/
│ │ ├── graph.py # LangGraph workflow
│ │ ├── nodes.py # Investigation and report nodes
│ │ └── state.py # Typed state and report schemas