codingthefuturewithai/rag-retriever

A semantic search system that crawls websites, indexes content, and provides AI-powered search capabilities through an MCP server. Built with modular architecture using OpenAI embeddings and ChromaDB vector store.
## 🚀 Quick Start with AI Assistant
Let your AI coding assistant help you set up and use RAG Retriever:
Setup: Direct your AI assistant to SETUP_ASSISTANT_PROMPT.md
Usage: Direct your AI assistant to USAGE_ASSISTANT_PROMPT.md
CLI Operations: Direct your AI assistant to CLI_ASSISTANT_PROMPT.md
Administration: Direct your AI assistant to ADMIN_ASSISTANT_PROMPT.md
Advanced Content: Direct your AI assistant to ADVANCED_CONTENT_INGESTION_PROMPT.md
Troubleshooting: Direct your AI assistant to TROUBLESHOOTING_ASSISTANT_PROMPT.md
Quick Commands: See QUICKSTART.md for copy-paste installation commands.
These prompts provide comprehensive instructions for your AI assistant to walk you through setup, usage, and troubleshooting without needing to read through documentation.
## What RAG Retriever Does
RAG Retriever enhances your AI coding workflows by providing:
Website Crawling: Index documentation sites, blogs, and knowledge bases
Semantic Search: Find relevant information using natural language queries
Collection Management: Organize content into themed collections
MCP Integration: Direct access from Claude Code and other AI assistants
Fast Processing: 20x faster crawling with Crawl4AI option
Rich Metadata: Extract titles, descriptions, and source attribution
## Key Features
### 🌐 Advanced Web Crawling
Playwright: Reliable JavaScript-enabled crawling
Crawl4AI: High-performance crawling with content filtering
Configurable depth: Control how deep to crawl linked pages
Same-domain focus: Automatically stays within target sites
### 🔍 Semantic Search
OpenAI Embeddings: Uses text-embedding-3-large for high-quality vectors
Relevance Scoring: Configurable similarity thresholds
Cross-Collection Search: Search across all collections simultaneously
Source Attribution: Track where information comes from
### 📚 Collection Management
Named Collections: Organize content by topic, project, or source
Metadata Tracking: Creation dates, document counts, descriptions
Health Monitoring: Audit collections for quality and freshness
Easy Cleanup: Remove or rebuild collections as needed
### 🎯 Quality Management
Content Quality Assessment: Systematic evaluation of indexed content
AI-Powered Quality Review: Use AI to assess accuracy and completeness
Contradiction Detection: Find conflicting information across collections
Relevance Monitoring: Track search quality metrics over time
Best Practice Guidance: Comprehensive collection organization strategies
### 🤖 AI Integration
MCP Server: Direct integration with Claude Code
Custom Commands: Pre-built workflows for common tasks
Tool Descriptions: Clear interfaces for AI assistants
Permission Management: Secure access controls
## MCP vs CLI Capabilities
### MCP Server (Claude Code Integration)
The MCP server provides secure, AI-friendly access to core functionality:
Web Crawling: Index websites and documentation
Semantic Search: Search across collections with relevance scoring
Collection Discovery: List and explore available collections