sk-20231/agentforge

> An AI agent with intent routing, semantic memory, RAG with citation guardrails, ReAct reasoning, tools over **MCP (Model Context Protocol)**, prompt-injection defences, CI eval gating, difficulty-based model routing, and full trace/cost observability. The core is built from primitives against the OpenAI API — so every moving part stays visible, debuggable, and tunable — then extended with the tools the industry actually runs on. Built to harden toward real, multi-user use.
---
Why this exists
Owning the core of an agent — instead of deferring it to a framework — is what keeps the parts that matter for **debugging, evaluation, and scale** in your hands. Reach for a high-level framework first and those parts stay hidden:
- What does retrieval actually return?
- Why did the model hallucinate that citation?
- Which prompt burned the token budget?
- How do you *know* the new embedding model didn't regress quality?
So AgentForge's core is built from the ground up — every decision (chunk size, similarity metric, routing logic, citation format, eval thresholds) is visible and tunable. **Then**, from that understanding, it integrates the tools the industry actually uses — starting with **MCP (Model Context Protocol)** for tool/resource interop, including consuming a third-party server behind a client-side security boundary. The point isn't to avoid frameworks; it's to understand *what's really happening* behind an agent **and** be fluent with the tooling teams ship on.
---
What it does
An interactive agent (CLI or Streamlit UI) that can:
- **Classify intent** and route each message to the right pipeline (remember, act, reason, search docs, etc.)
- **Remember things** using semantic memory (embeddings + cosine similarity)
- **Use tools over MCP** — Wikipedia, weather (open-meteo), and HackerNews run as our own MCP servers, plus a third-party fetch server consumed over the protocol. All tool output is sanitized and wrapped as untrusted data to defend against indirect prompt injection; untrusted third-party servers also get a client-side SSRF URL guard
- **Zero hardcoded tools** — both the ACT and ReAct pipelines discover and call every tool at runtime over MCP through a shared gateway; adding a tool means adding a server, not editing the agent
- **Multi-step reasoning** — ReAct (Reason + Act) loop for complex tasks
- **Route by difficulty** — reuse the intent label to send routine turns to a small model and escalate only multi-step reasoning to a frontier model (off by default; opt in per your bill)
- **Answer from documents (RAG)** — chunk, embed, retrieve, generate with a citation guardrail that strips hallucinated references
- **Personalize** answers with stored user facts
- **Rewrite follow-ups** into standalone queries so "How does it work?" retrieves correctly in a 10-turn conversation
- **Stream** tokens to the CLI and Streamlit UI as they arrive
- **Self-evaluate** — automated Recall@K retrieval tests + LLM-as-judge faithfulness scoring, gated in CI
- **Observability** — every run has a trace ID, per-span latency, P50/P95 percentiles, token counts, and USD cost
---
Architecture
Every LLM call goes through a Span context manager that records start/end timestamps, duration, and token usage. Traces are structured JSONL — grep-friendly, easy to aggregate.
Both tool pipelines reach their tools over MCP through one gateway, and both run as LangGraph graphs, so a gated tool call can **pause the turn for human approval** and resume later from a checkpoint (see decision #5).
---
AI concepts covered
| Concept | Where in the code | Why it matters |
|---|---|---|
| **Intent classification** | main.py — classify_intent | One input can mean many actions — specialize per intent |
| **Semantic memory** | memory/semantic.py | Long-term, queryable user context via embeddings |