bitnet.cpp

📰 News
07/23/2026: 📣 We released VibeASR.cpp — a real-time multilingual ASR inference engine on CPU using BitNet I2_S quantization, achieving RTF Code ] [ Models ] [ Report ]
07/20/2026: 📣 We released BitNet-embedding-0.6B and BitNet-embedding-270M on Hugging Face — the first 1-bit embedding models that deliver competitive embedding quality with significantly faster inference on CPUs.
- **1.42x to 2.28x speedup** over F16 on BitNet-embedding-0.6B prefill (8 threads)
- **1.32x to 1.74x speedup** over F16 on BitNet-embedding-270M prefill (8 threads)
- Supports I2_S conversion with optimized kernels on x86 CPUs
- Lossless inference with 2 bits per weight
07/16/2026: 📣 Released BitNet Embeddings 0.6B/270M: I2_S Conversion and Inference Optimization — detailed guide for converting and running BitNet embedding models with optimized I2_S kernels.
01/15/2026: 📣 Released BitNet CPU Inference Optimization — parallel kernel implementations with configurable tiling and embedding quantization support, achieving **1.15x to 2.1x** additional speedup over the original implementation.
05/20/2025: 📣 Released BitNet Official GPU inference kernel — extending 1-bit inference beyond CPUs.
04/14/2025: 📣 Released BitNet Official 2B Parameter Model on Hugging Face — the first official BitNet b1.58 model trained with 4T tokens.
02/18/2025: 📑 Bitnet.cpp: Efficient Edge Inference for Ternary LLMs — system-level paper on bitnet.cpp's architecture and design.
11/08/2024: 📑 BitNet a4.8: 4-bit Activations for 1-bit LLMs — enabling 4-bit activations for further efficiency gains.
10/21/2024: 📑 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs — the technical report behind bitnet.cpp.
10/17/2024: 📣 bitnet.cpp 1.0 released.
03/21/2024: 📑 The-Era-of-1-bit-LLMs: Training Tips, Code, FAQ
02/27/2024: 📑 The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits — the foundational paper introducing BitNet b1.58.
10/17/2023: 📑 BitNet: Scaling 1-bit Transformers for Large Language Models — the original BitNet paper.
Overview
bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels that support **fast** and **lossless** inference of 1.58-bit models on **CPU** and **GPU** (NPU support coming next).
Try it out via this online demo, or build and run it on your own CPU or GPU.
bitnet.cpp achieves speedups of **1.37x** to **5.07x** on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by **55.4%** to **70.0%**, further boosting overall efficiency. On x86 CPUs, speedups range from **2.37x** to **6.17x** with energy reductions between **71.9%** to **82.2%**. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. Please refer to the technical report for more details.
Model Releases
1. BitNet-b1.58-2B-4T - 1-bit Large Language Model
**BitNet-b1.58-2B-4T** is the first official BitNet b1.58 model with **2.4B parameters**, trained on **4 trillion tokens**. It is a ternary (1.58-bit) language model that delivers competitive performance with full-precision models of similar size while enabling significantly faster and more energy-efficient inference.
- **Fast CPU Inference**: Achieves up to **6.17x speedup** on x86 CPUs and **5.07x** on ARM CPUs compared to full-precision models.
- **Energy Efficient**: Reduces energy consumption by up to **82.2%** on x86 and **70.0%** on ARM.
- **GPU Support**: Official GPU inference kernel available for accelerated deployment.
- **Chat-Ready**: Supports conversational mode for interactive use.
🤗 Hugging Face | 🔗 Online Demo | 📄 Technical Report