musharna/jobd

**A self-hostable, GPU-aware job broker for your own machines — with native MCP/agent integration.**
> Like task-spooler or pueue, but across *all* your machines — and VRAM-aware.
Demo recorded on jobd v0.5.30; output on later versions may differ.
You have a couple of boxes with GPUs — a workstation, a server, maybe a laptop — wired together over Tailscale or a LAN. You want to fire off training runs, data pipelines, and long batch jobs from anywhere, have them land on whichever machine actually has the VRAM free, survive across sessions, and get preempted cleanly when something more important shows up. You don't have a cloud, a Kubernetes cluster, or a Slurm install, and you don't want one.
jobd is that missing piece: a lightweight, single-process broker that turns a handful of personal machines into a single queue — and an LLM agent can drive it directly.
VRAM routing tracks one GPU per host: the worker reports free memory on GPU index 0 only.
Why it exists
Most schedulers assume a datacenter. The lightweight ones that don't (a bare nohup, a tmux session, an ssh-and-pray script) give you nothing: no queue, no VRAM-aware routing, no preemption, no record of what ran where. jobd fills the gap between "ssh in and run it" and "stand up Slurm":
- **VRAM-fit routing.** The broker matches each job against live worker capacity (free VRAM / RAM / CPUs, capability tags, arch/OS) and dispatches to a worker that actually fits — instead of you guessing which box is free. One GPU per host is tracked (GPU index 0).
- **Preempt + checkpoint window.** A higher-priority job can preempt a running one: the worker sends SIGTERM, the workload gets a grace window, then SIGKILL. jobd gives each job a per-job JOBD_CHECKPOINT_DIR to write into during that window; saving the checkpoint is the workload's job. A preempted job ends in the terminal preempted state and is not re-run automatically — to resume, you submit a new job pointed at the old checkpoint. (See docs/preemption.md.)
- **Survives sessions.** Submit, close your laptop, check back tomorrow. Jobs live in the broker, not your shell.
- **Agent-native.** Ships a first-class MCP server so an LLM agent (Claude Code, etc.) can submit, monitor, and babysit jobs as tool calls.
- **Yours.** One broker process you run on a machine you own. No accounts, no telemetry, no per-GPU-hour billing; the broker and worker talk only to each other and to your clients. The optional self-update scripts do go online to fetch releases: scripts/update-worker.sh (which job fleet add installs on a timer) installs from PyPI, and scripts/deploy-broker.sh queries the GitHub API and pulls the image from GHCR.
- **Loopback by default, tailnet-only beyond it.** The broker binds 127.0.0.1 unless you set JOBD_HOST. A request from any address that is neither loopback nor in Tailscale's CGNAT range (100.64.0.0/10) gets a 403 (JOBD_DISABLE_TAILNET_ACL=1 turns that check off).
Why not just use…?
| Tool | What it gives you | Why jobd instead |
| ------------------------------------------------------------------------------ | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| **nohup / tmux / ssh-and-pray** | Runs a command on one box | No queue, no VRAM-aware routing, no preemption, no record of what ran where |
| **task-spooler** | A real job queue — on a single machine | jobd queues across _all_ your machines and routes by live VRAM/CPU fit |
| **Pueue** | A mature single-machine command queue daemon | Pueue's own README declares distributed execution out of scope — jobd is that missing layer, plus GPU awareness |
| **HyperQueue** | Multi-machine task scheduling with HPC roots, single binary | HQ counts GPUs but doesn't track VRAM, and has no preemption/checkpoint contract or agent interface |
| **Slurm** | Datacenter-grade scheduling | Heavy to stand up and operate for 2–3 personal boxes; jobd is one process + a poller per host |
| **SkyPilot / dstack** | Provision and run on clouds + your own machines | SkyPilot's "existing machines" mode installs a Kubernetes cluster (k3s) on your boxes; dstack wants Docker + passwordless sudo on every host. jobd is one process + a poller — no containers, no sudo, no K8s |