PeterMakhnatch/eval-lab

Eval Lab runs coding agents on real tasks through Harbor
and checks whether the resulting score means anything.
The current focus is reward hacking: tasks and graders that pay an agent
for not doing the work, and agents that take that path. The lab finds these
holes, fixes the tasks, proves the fix, and detects cheating in trajectories
before any score or trajectory is reported or used for training.
## Why this matters
A pass is only evidence of skill if the task cannot be passed another way.
Published RL environments often can be. Xiaomi's MiMo-V2.6 RL code tasks, as
converted to Harbor, are the main case study here:
the upstream fix is still inside most task images (Vals found it in 67% of
code tasks), and the grader runs as root in the agent's own container;
a prompted cheating agent (Terminus-2, GLM-5.3-Flash) copied the fix on 6 of
8 published tasks;
on the hardened versions, two cheater models made 26 attempts and copied
nothing; two of those runs solved the task honestly instead.
Details: docs/mimo/README.md,
cheater-recall-v1.
## What the lab does
| Area | Question | Main tools | State |
| Task and grader integrity | Can this task be passed without doing the work? | hack scan, tasks admit, cheat run, hack mutate, vcheck, audit; hardening transforms (tasks derive, mimo-clean) | Applied to the 2,698 MiMo code tasks |
| Cheating detection | Did this trajectory cheat? | detectors scorecard, rules and LLM readers, watch (live alert and stop), Integrity dashboard, CheatBench port | Scored on labelled corpora |
| Execution and evidence | Run it once, keep the proof | run, matrix, submit/approve/tick; local Docker, Daytona, Modal; model-call capture; ATIF → Parquet/DuckDB | Daily use |
| Experiments on clean tasks | Does a model, prompt or training change help? | Baselines, SFT and GEPA pilots, synthetic benchmarks, Harbor ↔ Inspect ↔ Karotte interop | Small pilots |
Every task version is identified by a content digest. A fix is never an edit
in place: it is a new variant with a lineage record pointing to its parent
(docs/task-variants.md). Raw Harbor job directories
are the source of truth; databases and Parquet tables are rebuilt from them.
mermaid
flowchart LR
T[Task package pinned digest] --> G{Integrity gate scan, controls, cheat ladder, mutation}
G -- hole found --> V[Hardened variant new digest + lineage]
V --> G
G -- admitted --> R[Harbor runs spend-gated queue]
R --> E[Evidence job dirs, ATIF, Parquet]
E --> D[Cheating detectors and human verdicts]
D --> C[Counted score and training export]
A raw score is the verifier's reward. A counted score drops passes that
were copied, tampered with or broken by infrastructure. Results are reported
as counted, with the exclusions listed.
## Results so far (as of 2026-10-11)
| Result | Evidence |
| MiMo-V2.6 RL catalogued: 7,780 Harbor tasks (code 2,698, webdev 2,093, cyber 1,000, music 1,000, general 925, terminal 64), each with static findings | docs/mimo-task-catalog.md |
| Code tasks cleaned (mimo-clean-v5): 2,666 rebuilt, 32 discarded, 564 reference fixes recovered and replayed (fix scores 1, doing nothing scores 0) | docs/mimo/README.md |
| New grader (separate-verifier@8): runs in a fresh container and grades fail-to-pass against its own pristine run; eight versions in three days, each fixing a hole or a false 0 found by measurement | docs/mimo/separate-verifier.md |