llm-eval-tracker

Live index of LLM evaluation tools and benchmarks, refreshed every 15 minutes from GitHub
**⭐ Star this repo to bookmark — fresh data every 15 minutes**
English · 中文 · 日本語 · 한국어 · Español · Português
---
💡 What is this?
Automatically discovers and indexes new LLM evaluation frameworks, benchmarks, and harnesses as they appear on GitHub. Generates a structured, searchable catalog with metadata like stars, activity, and category tags. Designed for ML engineers who need to stay current without manually scanning repositories.
This list is **auto-updated every 15 minutes** by a GitHub Actions cron.
Each commit reflects a real change in the upstream data source — new items added,
expired items removed — so you can rely on what you see being current.
---
📋 Current Items
> ⏰ Last updated: 2026-10-16 07:45 UTC
>
> Data source: GitHub Search API
>
> The table below is rewritten on every cron tick. Star the repo to bookmark.
| # | Name | ⭐ | Lang | Updated | Description |
|---|------|---|------|---------|-------------|
| 1 | promptfoo/promptfoo | 26039 | TypeScript | 2026-10-16 | Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, C |
| 2 | judgekeeper/judgekeeper | 0 | Python | 2026-10-16 | Check your LLM-as-a-judge: see how often it agrees with human labels, in one command. |
| 3 | Giskard-AI/giskard-oss | 5906 | Python | 2026-10-16 | 🐢 Open-Source Evaluation & Testing library for LLM Agents |
| 4 | Arize-ai/phoenix | 11869 | Python | 2026-10-16 | AI Observability & Evaluation |
| 5 | yukincom/llm-SugarScape | 7 | Python | 2026-10-16 | Multi-agent simulation using LLMs. Agents autonomously decide actions for survival, reproduction, and social behavior in |
| 6 | HLIntel/thea-software | 3 | Python | 2026-10-15 | AI coding-agent verification — a machine-readable rulebook (atlas.yaml) telling agents which commands prove a change; th |
| 7 | ManavPratapSingh/RAG-Stress | 0 | Python | 2026-10-15 | A deterministic evaluation harness stress-testing RAG pipelines against controlled distractors, contradictions, and posi |
| 8 | verifywise-ai/verifywise | 364 | TypeScript | 2026-10-16 | Complete AI governance and LLM Evals platform with support for EU AI Act, ISO 42001, NIST AI RMF and 20+ more AI framewo |
| 9 | NoesisVision/nasde-toolkit | 14 | Python | 2026-10-15 | CLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini ind |
| 10 | decimal-labs/decimalai-python | 1 | Python | 2026-10-15 | Python SDK for agent evals and skill routing — measure a skill's real lift before you trust it |
| 11 | FZF525/zhajinhua-ai-arena | 1 | TypeScript | 2026-10-15 | 纯积分制 5 张炸金花规则引擎 + AI 擂台 (zero-dependency TS engine for a 5-card Zha Jin Hua variant: 10 bots, LLM seats, paired evaluati |
| 12 | izam-mohammed/ragrank | 47 | Python | 2026-10-15 | 🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it understands context, its tone, an |
| 13 | open-compass/opencompass | 7520 | Python | 2026-10-16 | OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, |
| 14 | Shinibali/CuriosLM | 0 | Python | 2026-10-15 | GPT-2 (124M) → Gemma 3-style (113M), one architecture change at a time: training, evals, analysis, inference, context ex |
| 15 | homemade-software-inc/completion-kit | 3 | Ruby | 2026-10-14 | Your prompts need tests too. Run prompts against real datasets, score outputs with LLM judges, version everything, and c |
| 16 | lehigh-university-libraries/htr | 2 | Go | 2026-10-14 | Handwritten Text Recognition llm eval tool |
| 17 | Sans-cell-art/-Project-Phoenix-The-E-Waste-Supercomputer- | 0 | — | 2026-10-14 | ♻️ Transform e-waste into a powerful, low-cost cloud operating system, unlocking computing potential and promoting resou |