elhamid/llm-council

Fork notes (elhamid)
This is a fork of karpathy/llm-council.
Update — 2025-12-19 (Stability + auditability pass)
New in this fork since the previously published state:
- **Conversation history is complete again:** storage loads older runs from backend/data/conversations.json and supports the legacy data/ layout (including data/conversations/ per-conversation JSON files when present), so older runs show up in the UI.
- **Sidebar no longer truncates early:** the frontend now requests /api/conversations?limit=500 so the conversation list reliably shows full history.
- **Stage 2 judge duplication fixed:** Stage 2 now dedupes judge models so evaluation does not silently double-count the same judge.
- **Titles persist correctly:** after Stage 3 completes, a title is derived and saved to the conversation record, so it sticks across refreshes and appears in history.
- **Tooling + reproducibility:** added Stage 2 smoke/quality scripts and supporting evaluation artifacts to make regressions repeatable.
- **Roles updated + clarified:** earlier fork notes described roles as **Analyst / Researcher / Critic / Provocateur**; the implemented role set is now **Builder / Reviewer / Synthesizer / Contrarian** (with a provider-default mapping in backend/roles.py). This keeps the “multi-perspective” intent, but with clearer, more actionable role behavior.
Value added — 2025-12-14 (decision-quality focused)
- **Decision-auditable runs:** every council response is saved with a compact decision trace (Stage 1 answers, Stage 2 rankings, and the Stage 2→model mapping), so you can inspect *why* the Chairman concluded what it did — not just the final text.
- **Reduced “model-brand” bias in judging:** Stage 2 rankings operate on anonymized responses (Response A/B/C/…), and the label→model mapping is preserved for post-hoc review. This keeps peer review focused on content quality rather than model identity.
- **Role-separated council behavior:** explicit role specs (Analyst / Researcher / Critic / Provocateur + Chairman) make the council behave more like a real review board: one pushes rigor, one hunts missing facts, one stress-tests, one challenges assumptions — then the Chairman synthesizes.
- **Repeatable scoring across runs:** aggregated ranks (average rank + count) are persisted so you can compare council behavior over time and across prompts, instead of treating each run as a one-off chat.
Implementation notes (supporting the above)
- Real SSE endpoint (text/event-stream) for /api/conversations/{id}/message/stream with incremental stage*_start/complete events.
- Persist meta/metadata (label_to_model, aggregate_rankings, model_roles) so Stage2 renders correctly and the run is reviewable later.
- Frontend Stage2 reads msg.meta || msg.metadata so fork/upstream payload shapes both render.
----
The idea of this repo is that instead of asking a question to your favorite LLM provider (e.g. OpenAI GPT 5.1, Google Gemini 3.0 Pro, Anthropic Claude Sonnet 4.5, xAI Grok 4, eg.c), you can group them into your "LLM Council". This repo is a simple, local web app that essentially looks like ChatGPT except it uses OpenRouter to send your query to multiple LLMs, it then asks them to review and rank each other's work, and finally a Chairman LLM produces the final response.