GitHub

🚀 What is LocalGPT?
LocalGPT is a **fully private, on-premise Document Intelligence platform**. Ask questions, summarise, and uncover insights from your files with state-of-the-art AI—no data ever leaves your machine.
More than a traditional RAG (Retrieval-Augmented Generation) tool, LocalGPT features a **hybrid search engine** that fuses dense vector search with LanceDB's native full-text search, arbitrated by a **calibrated cross-encoder reranker**. A **smart router** picks between RAG and direct LLM answering for every query, while **contextual enrichment** and sentence-level Context Pruning surface only the most relevant content. Optional passes — Late Chunking, an independent answer **verification** step, and experimental multi-vector (late-interaction) retrieval — can be switched on per config; the defaults ship with exactly the components that earned their place in measured evaluations (see eval/decisions/).
The architecture is **modular and lightweight**—enable only the components you need. The RAG core is plain Python built on the standard library's HTTP server, with no web framework and no agent framework in the way.
▶️ Video
Watch this video to get started with LocalGPT.
| Home | Create Index | Chat |
|------|--------------|------|
| | | |
✨ Features
- **Utmost Privacy**: Your data remains on your computer, ensuring 100% security.
- **Versatile Model Support**: Swap generation models freely via Ollama.
- **Diverse Embeddings**: HuggingFace embedding models (harrier-oss-v1, the Qwen3-Embedding family) or any Ollama embedding tag.
- **Reuse Your LLM**: Once downloaded, reuse your LLM without the need for repeated downloads.
- **API**: A REST gateway on port 8000 and the RAG API on port 8001 for building your own applications.
- **CUDA, MPS & CPU**: Embedding and reranking pick CUDA, then Apple MPS, then CPU automatically.
📖 Document Processing
- **Formats**: PDF, DOCX, HTML/HTM, Markdown, and TXT, parsed by Docling
- **OCR fallback**: PDFs with no text layer are re-run through Docling's OCR pipeline; the engine is chosen from whatever is installed (OcrMac on macOS, then EasyOCR, RapidOCR, tesserocr, or the tesseract CLI)
- **Contextual Enrichment**: Chunk-level context generated by a small LLM, inspired by Contextual Retrieval
- **Late Chunking** (off by default): A second, document-level embedding pass stored in a companion _lc table. The 2026-08-18 component ablation measured its removal at the noise floor on single-turn quality while it doubles the vectors written per index, so it now ships disabled; one flag (retrieval.latechunk.enabled) re-enables both the index-time build and the query-time leg — multi-turn conversations with drifting phrasing are where it earns its cost
- **Document Overviews**: A short per-document summary written to index_store/overviews/ .jsonl and used by the router
🤖 AI-Powered Chat
- **Natural Language Queries**: Ask questions in plain English
- **Source Attribution**: Answers come back with the chunks they were grounded in
- **Smart Routing**: Chooses between RAG and a direct LLM answer per query
- **Query Decomposition**: Splits complex questions into sub-questions, retrieves per sub-question, then pools the candidates for one rerank and one synthesis pass (per-sub-answer composition is available as an option)
- **Reciprocal Rank Fusion**: Vector and full-text hits are fused with RRF — no weights to tune
- **Reranking**: A cross-encoder pass over the fused candidate set, on by default with calibrated score-based selection (eval/DECISIONS.md)
- **Sentence Pruning**: Optional Provence pruning drops irrelevant sentences from each chunk
- **Semantic Caching**: TTL cache with a 0.98 similarity threshold, scoped to the session
- **Answer Verification** (off by default): A second pass that appends [Confidence: N%] to the answer. Ablation measured zero verdict flips from disabling it — it annotates rather than changes answers — so it ships disabled; re-enable with verification.enabled
🛠️ Developer-Friendly
- **RESTful APIs**: Every UI action is a documented HTTP call
- **Streaming phases**: Server-Sent Events expose each pipeline stage as it runs
- **Flexible Configuration**: Models, chunk size, retrieval mode and toggles per request
- **One master config**: rag_system/main.py holds every default, overridable by environment variable