aegis-dq/aegis-dq

**The open-source agentic data quality framework.** Point it at your policy docs and warehouse — it generates rules, validates your data, diagnoses every failure with LLM root-cause analysis, and proposes SQL fixes. Run from the CLI, Airflow, GitHub Actions, or conversationally via Hermes.
Real-world result: **12 AML policy docs → 55 rules generated → 11 BSA/OFAC violations detected → all diagnosed → $0.01 total LLM cost.**
- **31 rule types** — completeness, uniqueness, validity, referential integrity, statistical, ML anomaly detection
- **6 warehouse adapters** — DuckDB, Postgres/Redshift, BigQuery, Databricks, AWS Athena, Snowflake
- **Pluggable LLMs** — Anthropic Claude, OpenAI, Ollama (local), AWS Bedrock
- **Agentic pipeline** — plan → parallel validation → LLM diagnose → RCA → SQL remediate → report
- **Hermes + MCP** — run full pipelines conversationally; listed on Glama.ai
---
GitHub Actions — Quick Start
Add a data quality gate to any workflow in under 2 minutes:
The step **fails the job automatically** when any rules fail, blocking broken data from reaching production. Set fail-on-failure: 'false' to report without blocking.
**Offline mode (no API key required):**
Action inputs
| Input | Default | Description |
|---|---|---|
| rules-file | rules.yaml | Path to rules YAML |
| db | :memory: | DuckDB file path |
| warehouse | duckdb | duckdb · postgres · redshift |
| pg-dsn | — | PostgreSQL / Redshift connection DSN |
| no-llm | false | Skip LLM — free offline validation |
| llm | anthropic | anthropic · openai · ollama |
| llm-model | *(provider default)* | Override the default model |
| fail-on-failure | true | Fail the step when rules fail |
| version | *(latest)* | Pin a specific aegis-dq version |
| anthropic-api-key | — | Required when llm: anthropic |
| openai-api-key | — | Required when llm: openai |
Action outputs
| Output | Description |
|---|---|
| rules-checked | Total rules evaluated |
| passed | Rules that passed |
| failed | Rules that failed |
| pass-rate | Pass rate as a decimal (e.g. "91.67") |
| report-json | Absolute path to the full JSON report |
**Using outputs in downstream steps:**
---
Demo
---
Generate rules with the LLM
Instead of writing rules by hand, let Aegis introspect your table schema and generate a draft rules file:
Add a --kb document — any plain text or markdown file describing your business logic — and the LLM generates **business validation rules** alongside structural ones:
**What goes in a KB file?** Anything your team knows about the data:
The LLM turns these into accepted_values, sql_expression, between, and foreign_key rules automatically. Generated rules are stamped status: draft — review, promote to active, and commit.
**All aegis generate options:**
| Flag | Default | Description |
|---|---|---|
| --db | — | DuckDB file for schema introspection |
| --kb | — | Business-context file (text/markdown) |
| --output | rules.yaml | Output YAML file |
| --max-rules | 20 | Cap on number of rules generated |
| --no-verify | false | Skip SQL verification of generated rules |
| --save-versions | false | Persist rules to version store |
| --provider | anthropic | LLM provider |
| --model | *(default)* | Override model |