alex-on-ai/WebReaper

AI-native web scraper. Single binary with a bundled Claude Code skill.
Built by HighCraft.io.
Install
**macOS / Linux (Homebrew):**
**Any POSIX shell (install.sh):**
**.NET library:**
Windows binaries are on the GitHub Releases page; winget and Scoop are on the v10.1 roadmap.
**Updating:** brew upgrade webreaper (Homebrew), or re-run the install.sh line with --upgrade appended (| sh -s -- --upgrade), or dotnet add package WebReaper (library). When a newer release exists, scrape / crawl / map print a one-line upgrade hint on stderr (interactive terminals only, never in a pipe or CI); disable that check with WEBREAPER_NO_UPDATE_CHECK=1.
30-second demo
After webreaper init, the next Claude Code session picks up the skill and routes scraping intents (*"give me the markdown of X"*, *"what blog posts are on Y"*, *"scrape the top 5 articles"*) to webreaper automatically.
Table of contents
- Why WebReaper
- Quick start
- Bot protection that just works
- Power your agent
- AI features
- Use cases
- Packages
- Compared to Firecrawl, Crawl4AI, and Crawlee
- API overview
- Architecture and interfaces
- Repository structure
- License
Why WebReaper
🪶 Drop on PATH, run. No Docker, no Postgres, no signup. ~12 MB binary.
🤖 AI-native by composition. Markdown by default. Schema extraction, LLM fallback, self-healing selectors, autonomous agent. Stack with .With…() .
🔌 Bring any LLM. OpenAI, Anthropic, Ollama, Azure OpenAI, llamafile, via Microsoft.Extensions.AI .
🛡 Bot-checks handled automatically. A blocked page climbs HTTP → browser → stealth on its own, per page and host-sticky. Challenge pages are dropped, never returned as data. No flag needed for the browser fallback.
📡 Distributed when needed. Swap scheduler, tracker, sink to Redis, MongoDB, SQLite, Azure Service Bus, Cosmos. Same code.
📜 MIT, not AGPL. Embed in commercial software, fork, modify, redistribute. Firecrawl's AGPL requires open-sourcing your service or paying for a commercial license.
Quick start
CLI
The CLI is built Native-AOT (ADR-0043), ships as a single binary on every tagged GitHub release across six RIDs (linux-x64, linux-arm64, osx-x64, osx-arm64, win-x64, win-arm64), and is block-aware with automatic browser/stealth escalation (ADR-0083). The macOS binaries are Apple codesigned and notarized (ADR-0071); Homebrew installs run without Gatekeeper warnings on a clean machine.
Library
That is HTTP-only, no extra packages, no schema. For structured fields, swap AsMarkdown() for Extract(schema); for JS-rendered pages, Crawl for CrawlWithBrowser plus a transport satellite (WebReaper.Playwright or WebReaper.Cdp). The full surface is in the API overview.
Bot protection that just works
Most scrapers make you opt into a browser up front and guess when a site is blocking you. WebReaper detects the block and escalates on its own, one page at a time:
- **Automatic.** A plain scrape or crawl starts on a fast HTTP fetch and climbs to a real browser only when a page looks blocked (a challenge status, response header, or body marker). No flag needed for the browser fallback.
- **Per page, host-sticky.** The first confirmed block on a host lifts that host's floor, so the rest of a whole-site crawl starts at the working tier instead of re-paying the failed one. You pay for the climb once, not per page.
- **No garbage in your data.** A page still blocked at the top tier is dropped, never written to your output, and the run exits non-zero so an unattended job knows. Clean data or a clear signal, never a challenge page masquerading as content.
- **You decide on stealth.** --stealth starts at the stealth backend; --auto-stealth (or WEBREAPER_AUTO_STEALTH=1) enables it unattended; --no-auto-stealth caps the climb at a vanilla browser. The ~220 MB stealth backend downloads only if you opt in.
Self-hosted, single binary, no cloud round-trip (ADR-0083).
Power your agent
Onboarding an agent from zero? Hand it docs/AI-ONBOARDING.md — install → verify → choose-your-path on one page, no credentials anywhere.
Claude Code skill
Writes a polished SKILL.md to .claude/skills/webreaper/. Claude Code loads it on the next session and routes scraping intents to the CLI: *"scrape the top 5 stories on HN and summarize each"*, *"give me the markdown of this article"*, *"this Cloudflare-protected site is blocking me"*. The skill describes when to prefer webreaper over the built-in WebFetch (artifacts and structured data go through webreaper; conversational answers stay on WebFetch).