andyliszewski/webcrawl-mcp

A lightweight MCP server that gives Claude Code (or any MCP client) the ability to scrape, search, map, and crawl the web — using free, open-source libraries. Firecrawl is supported as an **optional** fallback for JS-heavy sites when you have a key.
Why
Most scraping doesn't actually need a headless browser. trafilatura handles the ~80% case (articles, docs, blogs) locally, which is faster and keeps external API usage to a minimum. This server routes the easy stuff through local extraction and only falls back to Firecrawl when content quality is genuinely poor.
Tools
| Tool | Purpose |
|------|---------|
| webcrawl_scrape | Fetch a single URL → {content, source} (markdown + provenance) |
| webcrawl_search | DuckDuckGo search (optionally scrape results, each with provenance) |
| webcrawl_map | Discover same-domain URLs from a starting page |
| webcrawl_crawl | BFS crawl multiple pages (each result includes provenance) |
The source field on scraped content is one of static_http, static_http_retry, firecrawl_transport_fallback, or firecrawl_quality_fallback — see Fallback behavior.
Install
Requires Python 3.12+.
Quick smoke test (should print Webcrawl MCP server running then exit cleanly with Ctrl-C):
Install from source (for development)
Configure your MCP client
Claude Code
Create .mcp.json in your project root (or merge into ~/.claude/settings.json):
This uses uvx to run the package in a temporary environment — no manual install needed. If uvx is unavailable, install via pip install webcrawl-mcp and use "command": "webcrawl-mcp" with no args instead.
For a source checkout (development), point command at your venv's Python and use "args": ["-m", "webcrawl_mcp"].
Then in a Claude Code session run /mcp — you should see webcrawl listed with four tools.
Claude Desktop
Same JSON shape, placed in claude_desktop_config.json (see Anthropic's docs for the OS-specific path). Restart Claude Desktop after editing.
Other MCP clients (Cursor, Cline, Continue, Zed, etc.)
The command / args / env shape is standardized. Consult your client's MCP docs for where to put it.
Verify it's working
Ask your agent something like:
> Use the webcrawl_scrape tool to fetch https://docs.python.org/3/library/asyncio.html and summarize the first section.
You should see the tool invocation in the client UI, followed by a summary grounded in the live page content. If nothing happens, see Troubleshooting.
Environment variables
All optional:
| Variable | Default | Purpose |
|----------|---------|---------|
| USER_AGENT | Mozilla/5.0 (compatible; WebcrawlMCP/1.0; …) | HTTP User-Agent |
| REQUEST_TIMEOUT | 30 | Seconds before request timeout |
| FIRECRAWL_API_KEY | *(unset)* | If set, enables Firecrawl fallback for low-quality extractions. **Leave unset for a fully free setup.** |
| FIRECRAWL_API_URL | https://api.firecrawl.dev/v1 | Firecrawl endpoint |
| FALLBACK_ON_TRANSPORT_ERROR | false | If true **and** FIRECRAWL_API_KEY is set, route bot-blocked statuses (403, 429, 503) to Firecrawl instead of raising. Opt-in. |
| POLITE_MODE | true | On a 429 with a parseable Retry-After, retry the original request once after the indicated wait (capped at REQUEST_TIMEOUT) before falling through. |