QuietFlare/clew

Clew turns the lineage your workflow engine already writes into decisions
you can defend. When an input or container goes bad, it says what to do
about each output it reached. When the disk fills up, it names the work
directories that are safe to delete and shows why. When an incident
report arrives as a sentence, it turns the sentence into a question the
run can answer, or holds it for a person. Any of these decisions can be
sealed as evidence that an auditor checks offline.
It reads the record from Nextflow, Snakemake, Cromwell, Horus, DNAnexus
and Latch, and changes nothing in your pipeline. No model touches a
verdict: where a model is used, it proposes, versioned rules decide, and
the record says which model was asked. The examples are from genomics
because that is where it was first used. The graph underneath is tasks
that read and write files, whatever the field.
A clew is the ball of thread Ariadne gave Theseus. You follow it back out.
Try it in five minutes
**In a Codespace.** Clew, Nextflow and two recorded nf-core/demo runs
are in place when the terminal prints "Clew is installed". Say the seqtk
container turned out to have a defect, and ask what it reached:
Three tasks get a verdict, five are untouched, and the policy version
that decided is printed at the top. docs/try.md carries on
from there: drift between the two runs, reclaim, sealing the answer as
evidence, and the page.
**In Docker.** The page, the commands and the same two runs, in one image:
Open the link it prints. Browse on the page uploads a run from your
machine, or mount a launch folder with -v /path/to/launch:/runs/mine.
From a checkout, docker build -t clew . gives the same image.
**On your own machine.** Python 3.9 or later, no dependencies:
docs/try.md fetches the two recorded runs and takes it
from there.
The flow
Six steps, one command each. Every step reads what the one before wrote
and leaves a file the next one reads, so any step can be rerun or checked
on its own.
1. Extract
Point Clew at where the workflow was launched. The engine's own record
becomes graph.json: every task, every file it read and wrote, and the
content digests the engine recorded.
clew extract alone lists every engine installed. clew digest fills in
digests an engine did not record. clew stitch joins runs that consumed
each other's outputs. Sources says what each engine
records and what that limits.
2. Ask
Impact gives every affected task a verdict, re-run, quarantine, delete or
disclose, with the derivation chain as evidence and the policy version
that produced it. Reclaim proposes only the directories the graph proves
redundant and deletes nothing without --apply. Drift names the first
task on each chain whose outputs differ, and why. A verdict that depends
on a file existing is withheld, never guessed. Add --html report.html
to any of the three for a one-page report. More in
storage, reclaim and
drift.
3. Triage
A classifier picks one of the run's own triggers and says how sure it is.
Versioned settings then ask it, hold it for a person, or dismiss it. A
held incident waits until someone decides under their own name. More in
triage and triggers.
4. Seal
A bundle holds the plan, the policy it was computed under, the triage
record and decision, and the hashes of every input. verify recomputes
every verdict from the bundle alone, with no database and no
credentials. An optional event log on Postgres keeps what people decided,
append-only and hash-chained, and clew gate reads it before a run