elberacasa/umbra

Umbra is a deterministic Trust Score (0–100) for AI-generated code: the vibe
coding security scanner that verifies what your agent shipped, not what it
claimed. One command, fully local, evidence for every finding.
**→ umbra-badge.umbrabadge.workers.dev — the landing, live badges and report pages**
Quickstart · Demo · The Audit · How it works · The Four Axes · FAQ · Roadmap · Contributing · Website
Why Umbra exists
Studies put exploitable vulnerabilities in 40 to 60 percent of AI-generated
code, and coding agents routinely claim "all tests pass" when three do. The
tooling for *writing* code with AI is a year ahead of the tooling for
*trusting* it. Umbra closes that gap: SAST rebuilt for how software gets
written now, plus sandboxed verification that catches what static rules
cannot.
One command scans any repo an agent produced (Claude Code, Cursor, Copilot,
Windsurf, Lovable) and returns a score with file:line evidence for every
finding. With --deep it goes further: Umbra builds and boots the repo in a
locked-down Docker sandbox, then replays the agent's own claims against
reality. If the agent is lying about tests, the score is capped below
passing, with receipts.
The audit: 61 vibe-coded repos, scanned
We ran Umbra over 61 public, actively-maintained AI-built repos and
published everything. The Vibe-Coding Security Audit:
| Finding | Repos hit |
|---|---:|
| Hardcoded-secret findings (committed .env, service keys in source) | **25%** |
| API routes with no auth check | **26%** |
| Injection sinks (SQL interpolation, unsafe HTML injection) | **49%** |
| Entire databases / SQL dumps committed to git | **13%** |
| At least one critical finding | **10%** |
| Zero scored findings (genuinely clean) | 7 of 61 |
Mean trust score: **74/100**. One in five repos fails outright. The full
report has per-class deep dives with representative snippets and fixes, the
complete per-repo table, and an honest methodology section — including the
false positives we found in our own rules while running it, and fixed
(rubric v4).
Quickstart
That's the whole interface. Three verbs: check, heal, protect.
**Using an AI coding agent?** Umbra is built to be driven by agents, not
just run by humans:
- **Any agent** — it reads this repo's AGENTS.md / llms.txt and knows what to do. Or tell yours: "check this repo with umbra."
- **Claude Code / Kimi Code** — --setup installs PreToolUse hooks so every file the agent writes is guarded before it lands.
- **Claude Code, Cursor, Copilot, Windsurf** — the trust-review skill makes the agent scan its own work before declaring done.
- **MCP-native agents** — add umbra-mcp (npx --yes -p @elberacasa/umbra umbra-mcp) and the agent gets scan_repo, guard_content, and get_score as tools.
Real output, scanning a typical vibe-coded Next.js app
(fixtures/bad-app in this repo, Trust Score **30/100**):
The exit code is **1** when the score is below 50, so CI can gate on it.
All commands and flags (the expert layer — most users never need these)
The canonical package is @elberacasa/umbra; umbra-scan is the short
alias. Same engine either way.
How it works
A leaked Stripe key or an alg: none JWT never reaches the file. The path
guard hard-blocks agent writes into .git/hooks and .git/config
(CVE-2026-26268,
the agent-planted git hook escape), and live credentials going into .env.
Blocking is reserved for high-confidence critical/high findings; everything
else warns, and every failure fails open. Verdicts land in ~0.2 ms, so the
guard never slows the agent down. Full story:
docs/immune-layer.md.
--deep: verify AI code, don't trust it
The fast scan is static. --deep is LLM code verification with evidence.
Umbra copies the repo into a throwaway Docker container (no network at
runtime, 512 MB / 1 CPU hard limits, 120-second kill switch), builds it,
boots it, HTTP-probes its endpoints, and replays every claim found in
READMEs and agent artifacts against what actually happens. Slower (minutes,