ndjordjevic/pinrag

Overview
PinRAG is for **when you want to learn about something and your materials are scattered**—PDFs and ebooks, GitHub repos, YouTube videos, Discord discussions, and plain notes. You index those materials into **one shared RAG index**, then **ask questions from Cursor, VS Code (GitHub Copilot), or any MCP-capable assistant** and get answers **with citations** pointing back to pages, timestamps, files, or threads.
Under the hood it is **Retrieval-Augmented Generation** built with **LangChain** and exposed as an **MCP (Model Context Protocol) server**: add documents from the editor, query with natural language, list or remove what you indexed. Supported inputs include PDFs, local text files and directories, Discord exports, YouTube (transcript from URL, playlist, or ID), and GitHub repo URLs. For YouTube you can optionally add **vision** so on-screen code, diagrams, and UI text are merged with the transcript in the same chunks—see YouTube vision enrichment.
Features
- **Multi-format indexing** — PDF (.pdf), local files or directories, plain text (.txt), Discord export (.txt), YouTube (video or playlist URL, or video ID), GitHub repo (URL), web documentation sites (URL)
- **Optional YouTube vision** — Off by default. When enabled, runs a vision model (OpenAI, Anthropic, or OpenRouter native video) and merges structured on-screen context with the transcript so RAG chunks carry searchable code names, labels, and diagrams—not speech alone. OpenRouter mode avoids local ffmpeg/video download; openai/anthropic use scene keyframes and require pinrag[vision] + **ffmpeg** (see YouTube vision enrichment)
- **RAG with citations** — Answers cite source context: PDF page, YouTube timestamp, document name for plain text and Discord, chunk index for GitHub repos, source URL for web documentation
- **Document tags** — Tag documents at index time (e.g. AMIGA, PI_PICO) for filtered search
- **Metadata filtering** — query_tool supports document_id, tag, document_type, PDF page_min/page_max, and response_style (thorough or concise)
- **MCP tools** — add_document_tool, query_tool, list_documents_tool, remove_document_tool, set_document_tag_tool, list_collections_tool; optional collection on tools overrides PINRAG_COLLECTION_NAME for that call
- **MCP resources** — pinrag://documents (indexed documents) and pinrag://server-config (env vars and config); click in Cursor’s MCP panel to view
- **MCP prompt** — use_pinrag (parameter: request) for querying, indexing, listing, or removing documents
- **Configurable LLM** — OpenRouter (default, free openrouter/free router), OpenAI, Anthropic, or Cerebras Inference (OpenAI-compatible API); set via PINRAG_LLM_PROVIDER and PINRAG_LLM_MODEL in MCP env or your shell
- **Local embeddings** — Nomic (PINRAG_EMBEDDING_MODEL, default nomic-embed-text-v1.5); no API key; first run downloads model weights (~270 MB, cached)
- **Retrieval & chunking options** — Structure-aware chunking (on by default); optional FlashRank re-ranking, multi-query expansion, and parent-child chunks for PDFs (see Configuration)
- **Observability** — MCP tool notifications (ctx.log) plus optional LangSmith tracing
- **Built with** — LangChain, Chroma; optional OpenRouter, OpenAI, Anthropic, FlashRank
Installation
Add PinRAG as an MCP server in your editor. Install **uv** and ensure uvx is on your PATH—that runs PinRAG from PyPI without a prior pip install.
**Cursor:** add this under mcpServers in ~/.cursor/mcp.json:
**VS Code (GitHub Copilot):** run **MCP: Open User Configuration** from the Command Palette (or add .vscode/mcp.json in a workspace), then merge this shape—top-level key is servers:
Quick Start
HTTP server mode
For clients that speak MCP over HTTP (e.g. **pinrag-cli** with --server), run:
This starts a **streamable-HTTP** MCP endpoint at http:// : /mcp. The default pinrag stdio command for editors is unchanged; pinrag server is additive. Connect pinrag-cli with --server http://127.0.0.1:8765/mcp.
Configure MCP server
Put API keys and any PinRAG settings in the MCP entry’s **env block**. The server does not load .env files when the editor launches it.
Use in chat
| Action | Tool |
|--------|------|
| Index files, directories, or URLs | add_document_tool — required **paths**: list of local paths (PDFs, plain or DiscordChatExporter .txt, directories) or URLs (YouTube videos, playlist URLs, GitHub repos, web documentation sites; bare YouTube video IDs allowed). Optional **tags** (one per path). For GitHub URLs only: **branch**, **include_patterns**, **exclude_patterns**. |