natea/harness-eval

An evaluation framework that definitively ranks agentic coding frameworks by
having each one build the same product from the same spec, with the same
harness and model, in isolated sandboxes — then grading every artifact on a
weighted, evidence-based rubric.
First question answered: which of Superpowers, Compound Engineering,
Agent Skills, and GSD builds the best implementation of the
Symphony Service Specification (an
issue-tracker-driven orchestration daemon) when driven by Claude Code +
Opus 4.6? The architecture deliberately holds every other variable fixed so
the framework is the only thing being measured — and swaps cleanly for phase
2 (other harnesses like OpenCode/Codex, other models like GLM via z.ai).
## Status
Built (merged or in review) — each capability was specified under
openspec/changes/
before implementation:
Core harness — candidate registry, orchestration with isolation +
budgets + provenance, headless build sessions, two-instrument grading
(evidence-based evaluator + blind code-quality judge), telemetry, and
results.json + scorecard.md. (setup-harness-eval-framework)
Results dashboard — cross-run leaderboard, per-run scorecards, trial
drill-downs with evidence, and live re-weighting. (add-results-dashboard)
PRD library + bring-your-own-PRD — a target abstraction spanning several
product shapes (symphony-daemon, cli-tool, plus rest-api and web-app
adapted from ViBench), an init --target … --spec scaffolder, and
freeze/attribution rules. (add-prd-library)
Pluggable models — a model-profile registry, worker/judge resolution,
GLM via z.ai validated end-to-end, a model probe connectivity check,
and cross-vendor-judge + cost-basis caveats in results/scorecards.
(add-pluggable-models)
Pluggable providers/harnesses — Docker, E2B, macOS Virtualization, and
worktree providers share the same sandbox contract; harness registry entries
are validated per implemented driver. See
docs/HARNESS-ONBOARDING.md.
In review on branches: the Eval Studio web UI (review + configure on
shadcn/ui — this branch). Not yet built: a language-effectiveness
evaluation mode. See
ROADMAP.md.
## How it works
registry (pinned frameworks) ──┐
eval target (PRD + test plan) ─┼─▶ orchestrator ─▶ per-trial sandbox ─▶ headless build session(s)
run config (budgets, weights) ─┘ │ │
▼ ▼
provenance + telemetry archived workspace + transcripts
│ │
└────────────▶ grading ◀────────────┘
evaluator (test plan, evidence)
blind judge (code quality)
results.json ─▶ scorecard.md + web dashboard
1. Candidate registry (config/registry.yaml) — each framework under
test is declared with version-pinned install commands, its prescribed
session script (e.g. GSD's /gsd-new-project phases vs Superpowers' single
prompt), and a content-free continuation allowlist. Fairness rules are
enforced: every candidate gets the identical rendered base prompt; only
framework-prescribed command wrappers differ.
2. Eval targets (targets/ /) — a target couples a PRD, a frozen
weighted test plan derived from it (content-hashed at run start; fatal
cold-start gates; optional bonus steps), and any fixtures the evaluator
needs (e.g. a mock Linear GraphQL API and a stub Codex app-server for the