datanika-io/datanika-core

**Open-source data pipeline platform — Extract, Load, Transform, and Orchestrate from a single UI.**
Datanika combines dlt (extract + load) with dbt-core (transform) and adds visual pipeline management, scheduling, and monitoring — all in one Python-native platform.
> Think Airbyte + dbt Cloud + Airflow — in one tool, self-hostable with Docker Compose.
---
Features
🔌 **36 Connectors** — PostgreSQL, MySQL, Oracle, MongoDB, BigQuery, Snowflake, Stripe, HubSpot, Salesforce, Kafka, ClickHouse, and more
🔄 **dbt Transformations** — SQL models, tests, snapshots, packages, and source freshness built in
📊 **Visual Pipeline Builder** — DAG editor with dependency management
⏰ **Scheduling** — Cron-based with APScheduler, persistent across restarts
📈 **Monitoring** — Run history, streaming logs, and dashboard stats
🔐 **Enterprise Security** — RBAC, SSO (SAML/OIDC), audit logging, encrypted credentials
🌍 **9 Languages** — English, German, French, Spanish, Russian, Greek, Chinese, Arabic, Serbian
🔌 **REST API** — Full CRUD with OpenAPI/Swagger docs, rate limiting, and scoped API keys
🤖 **AI-Agent Ready** — hosted + local MCP server (25 tools), /llms.txt, agent-guide.md, 5-tier capability API, compile+preview validation, typed error codes, ?wait=true, Idempotency-Key, run cancellation
🚀 **Pipeline Templates** — One-click starter templates (Stripe→Postgres, Postgres→BigQuery, CSV→DuckDB) with prefilled connection configs
🔔 **Notifications** — Slack, Telegram, email, and webhook alerts on run completion, plus an in-app notification center
📦 **Self-Hostable** — Single docker compose up — no Kubernetes required
---
Quick Start
Docker (recommended)
App available at http://localhost:3000
Development
---
Releases & versioning
Datanika uses **0.x SemVer** with tagged releases (v0.1.0, …). While pre-1.0, breaking
changes bump the **minor**. There is no 1.0 yet — that will mean a committed, stable
public API contract.
**Self-hosting? Pin a release instead of tracking master.** master is continuously
deployed to our hosted app and moves several times a day.
> ⚠️ **There is no publicly pullable image yet.** ghcr.io/datanika-io/datanika-core
> exists but is **private**, so docker pull on any tag answers denied for everyone
> outside the org — this README told you otherwise until 2026-09-03. It stays private on
> purpose: the image grafts in our closed-source cloud plugin, so publishing it as built
> today would publish that too. **Pin the source tag and build**; that path is exercised
> by CI.
>
> **Building it yourself gives you the same artifact we would publish.** Since
> 2026-09-04 the Dockerfile carries a DATANIKA_IMAGE_EDITION build arg, and the
> core-only variant is built and asserted on every PR (core-only-image in CI) from a
> context that does not contain the cloud tree at all — which is the context you have.
>
> ⚠️ **The Dockerfile expects a monorepo-shaped context**: it does
> COPY datanika/pyproject.toml, so the build context is the **parent** directory and
> this checkout has to be the datanika/ inside it. Clone it under that name and build
> from one level up:
>
>
>
> The resulting image carries no /cloud tree and cannot import datanika_cloud.
> Leave DATANIKA_EDITION unset — its default is core. Setting it to cloud on this
> image fails immediately and loudly (ModuleNotFoundError: No module named
> 'datanika_cloud'), which is deliberate: the alternative is a service that looks
> healthy and enforces nothing. Publishing this under a public GHCR package is tracked