laurentvv/crawl4ai-mcp-llm

A powerful web crawling tool that integrates with AI assistants via the MCP (Model Context Protocol). This project allows AI assistants to crawl websites, extract dynamic content, navigate through links, and save structured Markdown files directly.
## 📋 Features
Native integration with AI assistants via MCP
Return scraped Markdown content directly to the AI
Extracts and surfaces internal/external links for AI navigation
Website crawling with configurable depth and page limit
Detailed crawl result statistics, including the list of skipped pages
Live progress notifications during long crawls
Saved results exposed as MCP resources (crawl://results/...)
Error and not found page handling
Secure by default: only public http(s) URLs, JavaScript disabled unless you opt in
Advanced Scraping Capabilities:
Magic Mode: crawl4ai heuristics that simulate a real browser and help with some anti-bot protections (not a guaranteed bypass)
Targeted Extraction: Fetch only what you need using CSS selectors
Custom JavaScript (opt-in): Execute code before extraction (clicks, scrolls, form fills)
Persistent Sessions: One browser is shared across calls, so a session_id keeps cookies and state until it is closed (close_session) or stays idle too long
SPA Support: Wait for dynamic CSS selectors or set explicit pre-extraction delays
## 🚀 MCP Configuration
The simplest and recommended way to use this tool is via uvx. The @latest suffix makes uvx check PyPI at each start and run the newest published version (without it, uvx keeps reusing the version it cached the first time).
### Prerequisites
uv installed on your system.
### Setup for AI Assistants (e.g., Claude Desktop, Cline)
Add the following to your AI Assistant's MCP configuration file (e.g., cline_mcp_settings.json or claude_desktop_config.json):
> Note: Python 3.12 or 3.13 is required (crawl4ai does not support 3.14 yet). Specifying --python 3.13 is recommended, especially on Windows, to avoid compilation issues with certain dependencies.
From PyPI (recommended):
json
"mcpServers": {
"crawl": {
"command": "uvx",
"args": [
"--python",
"3.13",
"crawl4ai-mcp-llm@latest"
],
"disabled": false,
"autoApprove": [],
"timeout": 600
From GitHub (latest unreleased):
json
"mcpServers": {
"crawl": {
"command": "uvx",
"args": [
"--python",
"3.13",
"--from",
"git+https://github.com/laurentvv/crawl4ai-mcp-llm",
"crawl4ai-mcp-llm"
],
"disabled": false,
"autoApprove": [],
"timeout": 600
Claude Code:
bash
claude mcp add crawl -s user -- uvx --python 3.13 crawl4ai-mcp-llm@latest
### Important: Browser Installation
The crawler uses Playwright to handle dynamic content. Install Chromium once after setting up the tool:
bash
# When running the server with uvx (recommended setup)
uvx --python 3.13 --from crawl4ai-mcp-llm@latest playwright install chromium
# From a local clone
uv run playwright install chromium
## 🖥️ Usage
Once configured, you can use the crawler by asking your AI assistant to perform a crawl.
### Usage Examples with Claude/Cline
Single Page: "Fetch https://docs.python.org/3/library/re.html (just that page) and summarize it."
Simple Crawl: "Can you crawl the site example.com and give me a summary?"
Crawl with Options: "Can you crawl https://example.com with a depth of 3 and include external links?"