KogenAI/kogen-bench

Benchmarks for Kogen, a coding-agent harness. The repository records comparisons with Codex and Grok Build, model and effort settings, harness designs, implementation languages, correctness, and reported usage costs. See the public glossary for arm, model, and analysis terms.
> Open source, closed contribution. Source code is available under the Apache License 2.0. This repository does not accept issues or pull requests and is not a public support channel. You may study and adapt it under the license.
## Start here
FINDINGS.md is the family-by-family summary of what the public evidence does and does not establish.
Hypothesis register lists H01–H153 and links each proposition to its family page.
Round register lists experiment pages and their status; each round page is the authority for that run's history and limits.
Publication validation records the final validator output, current blockers, and bounded claims safe to publish.
Public glossary defines arm, model, and analysis terms.
Mechanism inventory lists the 80 changeable parts of Kogen and its build loop, with their defaults and what the benchmarks have and have not measured for each part. Model-free profiles of run logs (checkpoint cost, provider retries, build round bounds, response durations, cut tool results, roles that end without submitting, acceptance-named tests, builder search tools) sit beside it.
Verification guide gives the commands and limits for recomputing published records and numbers.
Rerun guide explains task grading and the current state of the execution kit.
Ubuntu host setup is the one-command installer for a fresh Ubuntu 24.04 x86_64 worker. Fresh-machine setup was verified on 9 October 2026 on a host that has since been retired. See the rerun guide for the manual login and admission controls.
Round status summarizes current round states.
Credits records acknowledgements.
### Find an experiment
Start with the topic in FINDINGS.md, then follow its H IDs to the linked family page. The family page points to relevant round receipts. To look up a known round or study ID directly, use the round register or the machine-readable round index. Task definitions are in tasks/; official outcomes and captured deliveries are indexed separately in results/.
### For agents and maintainers
Use the hypothesis register for family ownership and evidence status, the family page for hypothesis-level synthesis, and each round page for run history. cells.jsonl contains official outcome rows; the run-record index lists one captured-delivery JSONL per round, including deliveries without an official grade, plus an unassigned.jsonl partition for records without an unambiguous round tag. Other large evidence families are also indexed per round. Schema 1.2 uses compact missing-reason codes listed in the legend. These exports answer different questions and must not be treated as interchangeable. Study design, execution, grading, and interpretation are documented in METHOD.md; required cell fields and their schema are in STANDARD.md.
### Reproduce
From the repository root, use the standard-library-only scripts in reproduce/README.md. The primary commands are: