AIMLPM/markcrawl

Turn any webpage or website into clean Markdown for LLM pipelines — in one command.
> **Latest:** v0.11.1 (2026-05-12) — default aggregator URL filter. See What's New below.
MarkCrawl is a crawl-and-structure engine. It fetches one page or crawls an entire website, strips navigation/scripts/boilerplate, and writes clean Markdown files with a structured JSONL index. Every page includes a citation with the access date. No API keys needed.
Everything else — LLM extraction, Supabase upload, MCP server, LangChain tools — is optional and installed separately.
> **Want a hosted API instead of running locally?** Join the waitlist — we're gauging interest.
**LLM agents:** Load docs/LLM_PROMPT.md as a system prompt to generate correct MarkCrawl commands automatically.
What's New
Install or upgrade with pip:
If markcrawl --help is missing flags you expect (e.g. --screenshot, --seed-file, --smart-sample, --download-images), your local install is stale. Run pip install --upgrade markcrawl against the same Python that owns the markcrawl binary on your PATH — head -1 $(which markcrawl) shows the right interpreter. PyPI is always the source of truth; see CHANGELOG.md for the full release history.
**v0.11 highlights** (changelog):
- **Aggregator URL filter** (default, v0.11.1) — rejects mdBook /print.html and Hugo /_print/ pages during crawl-time URL filtering. These bundle the entire docs tree on a single URL and otherwise dominate retrieval rankings on cosine similarity (markcrawl was returning them in 49% of rust-book and 39% of kubernetes-docs top-5 retrieval slots before the fix; competitors return 0%). Opt out via include_aggregator_pages=True / --include-aggregators.
- **Binary downloads** (v0.11.0) — new download_types=["pdf", "docx"] kwarg streams referenced files to /downloads/ with size + content-type guards. Pre-fetch download_filter callback receives URL + anchor text + parent-page context; reject candidates before any HTTP bytes transfer.
- **Local embedder is the default** since v0.10.1 — pip install markcrawl ships the full ML stack (torch + transformers + sentence-transformers). Zero API key required for embedding. Override with MARKCRAWL_EMBEDDER=text-embedding-3-small or the embedding_model kwarg if you want OpenAI back.
- **Tenacity-backed HTTP retry** — full-jitter exponential backoff (2 s → 30 s, 5 attempts) that honors the server's Retry-After header on 429s.
> **Where markcrawl stands on the public benchmark, honestly.** The independent llm-crawler-benchmarks v1.4 leaderboard measures 7 web crawlers on how well their output supports RAG. Markcrawl ranks **1st on cost** ($4,505/yr at 100,000-page scale) but **7th of 7 on answer quality** (3.77/5) and retrieval accuracy (MRR 0.341 vs leaders at 0.76). We're actively working to close that gap on three fronts:
>
> 1. **v0.11.1 (just shipped)** filters out /print.html and /_print/ "whole-book-on-one-page" URLs that were stealing 39–49% of markcrawl's top-5 retrieval slots on documentation sites. Competitors already filter these. Expected MRR improvement: **+0.02 to +0.04** on docs-heavy sites (formal measurement pending the next benchmark cycle).
> 2. **Upcoming releases** improve how markcrawl chooses *which* pages to crawl within its budget — markcrawl's deliberately-narrower crawl strategy (which keeps cost low and signal-to-noise high) is also the main cause of the retrieval gap.
> 3. **The benchmark itself is being improved** — v1.4's test questions were sampled from one specific crawler's output, which structurally penalizes any crawler whose discovery strategy differs from that anchor. The benchmark is being updated so each site's test questions come from the site's own sitemap, independent of any crawler. We expect this fix alone to surface **~5–10%** of markcrawl's current "misses" as actually correct answers at different URLs — work shown in our audit notes.
>
> **Goal for the next benchmark cycle:** move from 7th to mid-pack on retrieval (+0.10 to +0.20 MRR) and answer quality, while keeping the cost-efficiency lead. Honest, measured progress — we publish the numbers either way.
Quickstart (2 minutes)