dmmdea/offload-harness

**Delegate the grunt work to a free local model — keep your cloud tokens for judgment.**
A local-first harness that offloads short-context, low-judgment work — **summarize · classify · extract · triage** (plus vision, OCR, transcription, and image/SVG generation) — to a free **Gemma-family cascade** served by llama.cpp. It runs as a Go CLI and as an **MCP server** for AI coding agents. It **never calls a cloud model**: when it can't do a task confidently, it returns a structured **defer** so your agent handles it.
---
What & why
When an agent (or you) needs to summarize a log, label a ticket, pull fields out of a document, or answer a yes/no question about some text, that work is **mechanical and low-judgment** — but it still costs context window and cloud tokens. offload-harness runs those tasks on a **free local model** so the bulk, low-value tokens never enter your expensive context. A built-in **ledger** reports exactly how many tokens you saved.
The design rule is simple: **the local model does grunt work; your agent keeps all judgment.** Every task either returns a verified, schema-valid result or **defers** — a structured {"deferred": true, ...} that tells the caller "do this one yourself." There is no cloud fallback inside the harness, and it holds no cloud credentials.
It's for anyone running an AI coding agent or pipeline who wants to **cut token spend on bulk text work** while keeping data on the box.
Features
- **Free & fully local** — all inference runs on your GPU via llama.cpp; no API keys, no metering, nothing leaves the machine.
- **Never calls cloud, always defers** — low confidence returns a structured defer instead of guessing. The core offload path has no cloud credentials by design (the opt-in nim remote tool below is the one explicit, deliberate exception).
- **Self-learning cascade** — fast tasks enter at a small tier and escalate to a larger model only when genuinely uncertain (logprob decision margin + self-reported confidence).
- **Reliable structured output** — enforces a generated **GBNF grammar** + Go schema validation, working around the model's JSON-schema crashes.
- **Single static binary** — one Go executable; CLI and MCP server in the same build.
- **MCP-native** — exposes 26 tools over stdio for any MCP client (Claude Code and friends), including a read-only local agent (agent_run). That is the count every box registers; a box with the delegator role enabled adds agent_delegate, and a box listing an accelerator adds its own — 11 for hailo-8l, 4 for coral-edgetpu, 3 for rknpu, shared names registered once (ADR 0037), and a box listing a device it lacks in fleet_accelerators adds that device's tools as forwarders to the fleet node that has it, image bytes travelling with the call (ADR 0038, 0.115.0) — read tools/list, never a written figure.
- **Beyond text** — local **vision** (VQA / OCR / image-field-extract / render-QA), **speech-to-text** (whisper.cpp), **image/audio/video generation** (HiDream-O1 / SDXL / Qwen-Image · Wan 2.2 · Chatterbox TTS · ACE-Step via ComfyUI), **three image-edit routes** (deterministic PIL ops · masked inpainting · maskless instruction edits), and a dependency-free **SVG data-viz kit**.
- **Optional remote escalation** — an explicit, opt-in nim tool reaches any OpenAI-compatible **NVIDIA NIM** endpoint (NVIDIA's hosted build.nvidia.com free-model catalog, or a self-hosted NIM) for the rare task that needs a frontier model the local GPU can't run. Key from env only; never ledgered; the local cascade is untouched.
- **Token ledger** — append-only JSONL accounting of every offloaded call and the cloud tokens it saved.
Quickstart
First output in under five commands. If the local model is unreachable or unsure, you'll get {"deferred": true, "reason": "..."} — that's expected, not a failure.
Set up with your AI agent
The fastest path on Windows is to let your coding agent install the whole stack for you. Point
**Claude Code** (or any capable agent) at this repo and say:
> *Follow setup/SETUP-AGENT.md and install the stack.*
That runbook is written **for an agent**: it runs three idempotent PowerShell scripts —
setup/detect.ps1 (pick the backend: CUDA / Vulkan / CPU) → setup/install.ps1 (pinned llama.cpp +
llama-swap binaries, the Gemma-4 models, the Go build) → setup/selftest.ps1 (a machine-readable
receipt with a deep-context canary) — and branches on the JSON each one emits. It never substitutes a
pinned asset and never installs CUDA/ROCm (the release binaries carry their own runtime; Vulkan uses