cluaiz/cluaiz

Documentation  | 
Website  | 
Discord  | 
Reddit  | 
LinkedIn
> [!WARNING]
> Pre-release. v0.1.0 is the first public build. Expect breaking changes in settings, APIs and file layout
> until the stable release.
## About Cluaiz
Cluaiz is one engine process that runs local models and serves them to apps, scripts and agents:
GGUF models through a fork of llama.cpp linked into the engine:
chat, vision, audio input, embeddings.
ONNX models through ONNX Runtime: text-to-speech (Kokoro and others), speech-to-text (Whisper), embeddings
and vision encoders.
A placement planner that measures the free GPU memory and RAM on every load and chooses where the model
runs, how large the context is and which KV cache type fits. You can override any of it.
MoE models larger than the GPU, and larger than free RAM: attention stays on the GPU, the experts sit in
RAM, and whatever does not fit RAM is read from the SSD when it is used.
APIs that existing clients already speak (OpenAI-compatible and Ollama-style routes), plus CEP, one
JSON-RPC 2.0 protocol for everything engine-specific, over HTTP, MCP or a C library.
Tools for agents: skills, plugins and MCP servers, with permission levels, approvals and an audit log.
On the same model file, placement and settings, Cluaiz generates as fast as llama.cpp's own llama-bench, and
reads prompts up to 2x faster when a model is split across GPU and RAM (see Benchmarks).
> [!TIP]
> A 26B model on a 4 GB laptop GPU. Gemma 4 26B A4B (MoE, 13.3 GB file) on an RTX 3050 Laptop with 4 GB
> VRAM: 16–18 tokens/s with about 8 GB of free RAM, and still 7.4 tokens/s with only about 3 GB free,
> when more of the experts are read from the SSD. The planner picks this placement by itself (n_cpu_moe); no flags
> needed. Dense models that do not fit RAM + VRAM are refused with the reason instead of crawling from the disk.
> Details.
## Quick Start
1. Install (downloads the release for your system and the GPU pack for your GPU, and checks every file
against the release's checksums):
powershell
# Windows
powershell -c "irm https://cluaiz.com/install.ps1 | iex"
bash
# Linux and macOS
curl -fsSL https://cluaiz.com/install.sh | bash
Supported systems: Windows 10 / 11 x64; Linux x64 or ARM64 with glibc 2.38 or newer (Ubuntu 24.04, Debian 13,
Fedora 39 or newer; Ubuntu 22.04 and Debian 12 are not supported yet); macOS 14 or newer on Apple Silicon (M1 or
newer). Intel Macs and Windows on ARM are not supported yet.
Update later with cluaiz self update (models and settings stay); remove everything with cluaiz self uninstall
(--keep-models keeps the models).
2. Chat with a model (an installed model, a registry name or a Hugging Face repository; it is downloaded on
first use):
bash
cluaiz run unsloth/Qwen3-4B-GGUF
cluaiz run unsloth/Qwen3-4B-GGUF "Explain a KV cache in two sentences."
3. Let it work in a folder: the agent reads, searches and edits files and runs commands there, asking
before every change (guide):
bash
cd path/to/your/project
cluaiz agent -m qwen3-4b-gguf-Q4_K_M
4. See what the planner chooses on this machine before loading:
bash
cluaiz show unsloth/Qwen3-4B-GGUF
5. Serve it to other apps and call the OpenAI-compatible API: