KrishivAI/krishiv-engine

Krishiv is a Rust-native, Apache Spark™-compatible compute engine we are building for the agentic era. Keep the PySpark
and Spark SQL you already write, and run it on a vectorized, Arrow-native engine that handles batch SQL,
streaming and incrementally maintained materialized views with the same semantics. Start in-process on a
laptop, and take the same code to a single server or a cluster.
> [!WARNING]
> Alpha (0.2.0-alpha): the embedded core runs, the rest is being built. You can run SQL and DataFrames
> in-process from Python today (see the Quickstart); there is no stable API yet, and nothing has been
> published to PyPI or crates.io. The sections after the Quickstart describe what we are building, not what exists
> today.
⭐ Star the repo and 👀 Watch → Custom → Releases to follow development from the first commit to 1.0.
## Why Krishiv
| | Design goal | What it means for you |
| 🦀 | Spark-compatible, no JVM | Keep your PySpark and Spark SQL code. Spark Connect clients connect directly; execution is Rust-native and vectorized. |
| 🔁 | One engine: batch, streaming, incremental | The same SQL and the same semantics everywhere. A batch query is just a bounded stream. |
| 📦 | Embedded → single-node → distributed | pip install krishiv or cargo add krishiv to start in-process, then scale out without rewriting your code. |
| ⚡ | Always-fresh materialized views | Views update incrementally as data changes (DBSP-inspired incremental computation), and streaming state is directly queryable. |
| 🤖 | Built for the agentic era | A built-in MCP server lets AI agents explore and query your data; Arrow-native, zero-copy interop with pandas, Polars and PyArrow. |
| ✅ | Open and correctness-first | Apache-2.0. Tested against real Spark behaviour with golden fixtures and differential testing. |
## How it fits
mermaid
flowchart LR
subgraph APIs["Your code"]
A1["PySpark via Spark Connect"]
A2["SQL"]
A3["Python DataFrames"]
A4["Rust"]
A5["MCP for AI agents"]
A6["Arrow Flight SQL"]
end
subgraph Engine["One Krishiv engine"]
E1["Batch"]
E2["Streaming"]
E3["Incremental views"]
end
subgraph Deploy["Run it anywhere"]
D1["Embedded"]
D2["Single-node"]
D3["Distributed"]
end
subgraph Lake["Your lakehouse"]
L1["Parquet"]
L2["Apache Iceberg"]
L3["Delta Lake"]
end
APIs --> Engine --> Deploy --> Lake
## Supported CPUs
The shipped x86-64 binaries and wheels are built for the x86-64-v3 level: AVX, AVX2, FMA, BMI1/2, F16C, LZCNT and
MOVBE, i.e. Intel Haswell (2013) / AMD Excavator and Zen or newer. The arm64 builds target an Armv8.2 baseline
(neoverse-n1: AWS Graviton2 or newer, Apple silicon). On an older CPU the main build refuses to start and says
which feature is missing, instead of crashing (the Python package raises an ImportError; on Linux only: the
macOS arm64 and Windows wheels are separate builds and are not checked).
For an older x86-64 CPU use the compat build, which targets the plain x86-64 baseline (a little slower on
vectorised workloads). It covers Linux x86-64 only: a manylinux x86_64 wheel and an x86-64 CLI binary. No arm64 or
musllinux compat build is provided:
Python: pip install krishiv-compat (it is the same package and imports as krishiv; install one of the two, not both).