reliable-era/andrej-karpathy-skills-jev

A Claude Code plugin that asks TypeSafe Jev to check whether the agent followed
Karpathy's coding guidelines.
It checks assumptions before editing, and simplicity, scope, and verification before finishing.
The plugin includes both the skill and the hooks that run these checks automatically.
Comparison · Install · The four principles · How it works · Results
Comparison with the Original Skill
| Aspect | Native agent | Baseline: original Karpathy skill | karpathy-jev |
|---|---|---|---|
| Added guidelines | None | Skill text | Skill text and router |
| Who checks compliance? | Agent | Agent | Jev judges evidence; code applies rules |
| Runs if the skill text is never opened? | No added check | No | Yes, with Claude Code hooks |
| Diff and command evidence collected by a router? | No | No | Yes |
| Hidden pipeline failures handled? | No added handling | No added handling | pipefail and result parsing |
| External judge required? | No | No | TypeSafe API key |
| Support outside Claude Code | Agent's own behaviour | Voluntary skill use | Explicit router calls |
| Claude Code / Sonnet 5: SWE-bench Verified successes | 129/150 | 130/150 | **127/150** |
| Codex / GPT-5.5: SWE-bench Verified successes | 121/150 | 118/150 | **117/150** |
| Pi / Qwen3.8-27B: SWE-bench Verified successes | 99/150 | 97/150 | **92/150** |
**Performance rows:** 10% checkpoint campaign, 50 SWE-bench Verified tasks × 3 attempts per method.
No difference is statistically significant, and karpathy-jev is not ahead in any harness.
See Results for Terminal-Bench, false “done” claims and caveats.
These are differences in the added mechanism. A native agent can still ask questions, make a small
patch, and run tests on its own.
The historical Sonnet comparison reported that the original skill was never opened (0/20), while
hooks ran this router automatically. It also reported 10/20 Jev runs ending with a block still open;
that count does not establish that every block was correct. See the
evaluation details for the reported outcomes and audit limits.
*Jev supplies judgments; code applies gates and thresholds. Claude Code invokes the router through
hooks, while other agents must call it explicitly. Finishing checks are limited to two reviews.*
Install
Requires **Python 3.6+** and a **TypeSafe API key**. Make the key available through TYPESAFE_API_KEY
or the private file ~/.karpathy-jev/key before starting Claude Code.
Option A: Claude Code plugin — recommended
From inside Claude Code **2.1.275 or later**, install with one command:
Choose the installation scope when prompted. This installs **the skill and its hooks together**;
there is no need to copy files or edit hook commands. Follow Claude Code's install summary if it
requests a reload.
From a terminal with Claude Code **2.1.292 or later**, the equivalent is:
See Claude Code's plugin installation guide
for the supported installation scopes.
Option B: Install from a local checkout
Run this once, replacing the path with your checkout:
To load the plugin for just one session, start Claude Code from your target project:
Both options load the bundled skill and hooks. Copying only SKILL.md or the skill folder does not
provide automatic checks.
The Problems
A coding agent can make an assumption without checking it, build more than the request needs, change
unrelated files, or say “done” without testing its final edit.
The original Karpathy skill gives the agent rules for avoiding these mistakes. This plugin adds
checks of the work the agent actually performed.
The Solution
The agent writes the code. Jev judges specific questions about the request, diff, and command results.
The router applies the configured rules and can ask the agent to revise its work.
| Principle | What the router checks |
|---|---|
| **Think Before Coding** | Does an unclear request need clarification before the first edit? |
| **Simplicity First** | Does the patch contain work the request does not need? |
| **Surgical Changes** | Are the changed hunks relevant to the request? |
| **Goal-Driven Execution** | Was the final change checked? For a bug fix, was the failure reproduced first? |