rbmuller/scherlok

Zero-config anomaly detection for your database tables.
No YAML, no rules, no thresholds. Scherlok learns what "normal" looks like, then tells you when something changes.
No database handy? The demo seeds one, learns it, breaks it, and catches it, in about a second:
Works with **PostgreSQL, BigQuery, Snowflake, MySQL, DuckDB** and **dbt**. Alerts go to Slack, Discord, Teams, email, or your CI exit code.
---
The Problem
Every data team has the same nightmare:
> A source API silently changes from **dollars to cents**. Revenue dashboards show wrong numbers for **3 weeks** before anyone notices.
>
> A column starts returning **NULLs**. A table stops updating. Row counts drop **40% on a Tuesday**. Nobody knows until the CEO asks why the report looks weird.
Current tools (Great Expectations, Soda, dbt tests) require you to **define what "correct" looks like** before you can detect what's wrong. Hundreds of rules. Dozens of YAML files. And you still miss things — because you can't write rules for problems you haven't imagined yet.
What It Catches
| Anomaly | What Happened | Severity |
|---------|---------------|----------|
| **Volume drop** | Row count dropped 40% overnight | CRITICAL |
| **Volume spike** | 3x more rows than normal | WARNING |
| **Freshness alert** | Table hasn't updated in 12h (normally every 2h) | CRITICAL |
| **Schema drift** | Column removed or type changed | CRITICAL |
| **NULL surge** | NULL rate jumped from 2% to 45% | WARNING |
| **Distribution shift** | Column mean shifted 3+ standard deviations (Shewhart-style control limit) | INFO, WARNING above 5σ |
| **Cardinality explosion** | Status column went from 5 values to 500 | CRITICAL |
Every anomaly is auto-scored: **INFO**, **WARNING**, or **CRITICAL**. No thresholds to configure.
How It Works
Scherlok takes the opposite approach of rule-based tools: **learn first, then detect.**
Three commands. Five minutes. Done. (scherlok ci runs all three in one step for pipelines.)
After five valid profiles, Scherlok learns per-metric variability from the
latest 30 profiles using robust historical baselines for volume, numeric mean
shifts, NULL rates, and distinct counts. During cold start or when history is
not usable, it keeps the conservative fixed defaults.
1. investigate — Learn the patterns
Scherlok profiles every table: row counts, column types, NULL rates, value distributions, freshness cadence, cardinality. Stores everything locally in SQLite.
2. watch — Detect anomalies
3. Alert — Slack, CI/CD, or both
Auto-detects Slack, Discord, and Teams from the URL and formats the payload accordingly. Any other URL receives a generic JSON payload.
CI/CD Integration
Use Scherlok as a data quality gate. The ci command does it in one line:
If Scherlok detects a critical anomaly, the pipeline fails. Bad data never reaches production.
Works with dbt
Already running dbt? Scherlok complements dbt test with **automatic** anomaly detection — no rules to write.
Scherlok reads target/manifest.json, discovers every materialized model (table, incremental, view), auto-resolves the connection from your profiles.yml, and profiles each model:
Use it as a CI gate after dbt run:
Or collapse both steps into one with the wrapper:
The wrapper runs dbt run by default and uses the successful model nodes recorded in
target/run_results.json, so partial runs profile only what dbt actually built. Use
--build to run dbt build; successful models are still profiled when a test failure