crawlbase/crawlbase-mcp

Crawlbase MCP is a Model Context Protocol (MCP) server that bridges AI agents and the live web. Instead of relying on outdated training data, your LLMs can now fetch fresh, structured, real-time content — powered by Crawlbase’s proven crawling infrastructure trusted by 70,000+ developers worldwide.
It handles the complexity of scraping for you:
- JavaScript rendering for modern web apps
- Proxy rotation & anti-bot evasion
- Structured outputs (HTML, Markdown, screenshots)
How It Works
- Get Free Crawlbase Tokens → Sign up at Crawlbase ↗️, get free Normal, and JavaScript tokens.
- Set Up MCP Configuration → Configure the MCP server in your preferred client (Claude, Cursor, or Windsurf) by updating the MCP Servers settings.
- Start Crawling → Use commands like **crawl**, **crawl_markdown**, or **crawl_screenshot** to bring live web data into your AI agent.
Setup & Integration
Claude Desktop
1. Open Claude Desktop → Settings → Developer → Edit Config
2. Add to claude_desktop_config.json:
3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.
Claude Code
Add to your claude.json configuration:
Cursor IDE
1. Open Cursor IDE → File → Preferences → Cursor Settings → Tools and Integrations → Add Custom MCP
2. Add to mcp.json:
3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.
Windsurf IDE
1. Open WindSurf IDE → File → Preferences → WindSurf Settings → General → MCP Servers → Manage MCPs → View raw config
2. Add to mcp_config.json:
3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.
HTTP Transport Mode
For scenarios where you need a shared MCP server accessible over HTTP (e.g., multi-user environments, custom integrations), you can run the server in HTTP mode:
The server exposes:
- POST /mcp - MCP Streamable HTTP endpoint
- GET /health - Health check endpoint
Per-Request Token Authentication
HTTP mode supports per-request tokens via headers, allowing multiple users to share a single server:
**Headers:**
- X-Crawlbase-Token - Normal token for HTML requests
- X-Crawlbase-JS-Token - JavaScript token for JS-rendered pages/screenshots
Headers override environment variables when provided, enabling multi-tenant deployments.
🔑 Get your free tokens at Crawlbase ↗️.
Usage
Once configured, use these commands inside Claude, Cursor, or Windsurf:
- crawl → Fetch raw HTML
- crawl_markdown → Extract clean Markdown
- crawl_screenshot → Capture screenshots
Example prompts:
- “Crawl Hacker News and return top stories in markdown.”
- “Take a screenshot of TechCrunch homepage.”
- “Fetch Tesla investor relations page as HTML.”
Async Crawling with Cloud Storage
For larger jobs, Crawlbase MCP can push crawl results to Crawlbase Cloud Storage instead of returning them immediately. Your AI agent can then come back later to read, list, or clean up those pages — useful when crawling many URLs at once, revisiting a dataset across sessions, or keeping heavy HTML out of the chat until you actually need it.