SanggonBoy/PyreCrawl

**One command gives any AI agent the whole web.** Scrape, extract, crawl, map, and search —
self-hosted, no API keys, no rate limits, no subscription.
PyreCrawl speaks **MCP** (Model Context Protocol), the standard tool interface for Claude,
Cursor, VS Code, Codex, OpenCode, Hermes, and any MCP-compatible agent.
A **smart auto-fallback ladder** always picks the cheapest method that succeeds:
⚡ Tools exposed
| Tool | What it does |
|---|---|
| scrape(url, prefer="auto") | Single URL → LLM-ready markdown |
| extract(url, schema) | Scrape + structured extraction (JsonCss schema) |
| map_site(root, include_pattern=None, limit=200) | Enumerate all internal URLs |
| crawl(root, max_pages=5, prefer="auto", include_paths=None, exclude_paths=None, max_depth=0) | Multi-page crawl with path filters + true BFS depth |
| document(url) | PDF/DOCX/PPTX → markdown (no browser, optional [docs] extras) |
| search(query, limit=10) | Web search via DuckDuckGo HTML (no API key) |
| search_papers(query, limit=8, source="arxiv", category=None) | Academic search via arXiv + Crossref (no API key) — feed pdf_url into document |
| batch_scrape(urls[], ...) | Many URLs in ONE call — parallel, deduped, cache-aware |
| deep_research(query, limit=5, scrape_top=3) | Search → evidence pack with [n] citations (no LLM synthesis — your agent does that) |
| monitor(url, action, css_selector=None) | Change detection with persisted snapshots + unified diff |
| session(session, action, ...) | Persistent browser session (cookies kept) — login walls, multi-step flows, screenshots |
| cache(action) | Inspect/clear/enable/disable the HTTP response cache |
| health() | Versions + import sanity check |
**MCP Resources** (read-only state without a tool call):
pyrecrawl://cache/stats · pyrecrawl://sessions · pyrecrawl://monitors
**MCP Prompts** (ready-made playbooks): research(topic) · rag_ingest(site) · watch_page(url)
Env flags
| Variable | Default | Effect |
|---|---|---|
| PYRECRAWL_CACHE | off | 1 = in-memory LRU (128 pages), or a directory path (reserved for disk mode) |
| PYRECRAWL_CACHE_TTL | 900 | Cache entry lifetime in seconds |
| PYRECRAWL_MONITOR_DIR | ~/.pyrecrawl/monitors | Where monitor snapshots persist |
| PYRECRAWL_NO_TELEMETRY | off | 1 = disable the anonymous startup ping (also honors DO_NOT_TRACK=1) |
prefer options: "auto" (default ladder) · "fast" (HTTP only) · "stealth" (CF bypass) · "llm" (deep processing).
---
🚀 Install & Use (one-liner)
1. Install
UV (recommended — one command, zero Python setup)
UV is a fast Python package manager that handles Python itself —
no need to install Python separately. Get it once:
Learn more about UV →
Then run PyreCrawl directly — no venv, no pip install, no Python download:
Or via uv tool install (persistent, recommended for regular use)
Or via pipx (alternative)
Or via pip into a venv
2. One-time browser engines
This installs Chromium + stealth browser engines (~2 min, one-time).
3. Register with your AI agent
Supported agents: claude-desktop, claude-code, cursor, vscode, codex, opencode, hermes.
4. Start chatting
After installing + registering, **restart your agent** (or start a new session). Then ask:
> *"Scrape https://example.com and summarize it."*
Available tools:
| Tool | What it does |
|------|-------------|
| scrape | Fetch a single URL → markdown (auto-escalates past Cloudflare) |
| extract | Scrape + structured extraction via CSS schema → JSON |
| map_site | Enumerate all internal URLs from a root |
| crawl | Multi-page crawl: discover + scrape in bulk |
| batch_scrape | Fetch many URLs in one parallel call |
| search | Web search via DuckDuckGo with anti-bot bypass |
| search_papers | Academic paper search (arXiv / Crossref) |