unclecode/crawl4ai

Latest: v0.9.5 (14 Oct 2026) · all releases →
Crawl4AI turns any website into clean, LLM-ready Markdown for RAG, AI agents and data pipelines. Run the open-source web crawler and scraper yourself, free forever, or use it hosted with one key: /scrape a page to Markdown, /extract it to JSON, /search the web, /serp for Google's results page as data, /answer a question with sources; MCP for your agent. Free: 1,000 credits every month, 10 a minute, 2 at a time. Paid: 120 a minute, 10 at a time, from $10 a month.
## Two ways to use Crawl4AI
### 🐍 Run it yourself: open source, forever
bash
pip install -U crawl4ai
crawl4ai-setup # installs the browser, once
python
import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://news.ycombinator.com")
print(result.markdown)
asyncio.run(main())
Docker server, CLI and every option: Installation · docs.crawl4ai.com
### ☁️ Or use the cloud: no browsers, no proxies
1.
Verify your email and 1,000 free credits are yours every month. No card.
2. Get any page as Markdown:
bash
curl -s https://api.crawl4ai.com/scrape \
-H "Authorization: Bearer $CRAWL4AIKEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown
The same key works for /extract, /search, /serp, /answer and many URLs at once (/scrape/batch, /scrape/jobs). Pay as you go: live prices.
3. Give it to your AI agent. Claude Code shown; Codex, Cursor and OpenCode →
bash
claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp \
--header "Authorization: Bearer $CRAWL4AIKEY"
### Which one?
| | 🐍 Library | 🐳 Your own server | ☁️ Crawl4AI Cloud |
| Runs the browsers | you, in your Python process | you, in Docker on your machine | we do |
| JS-heavy pages and bot walls | your settings, your proxies | your settings, your proxies | handled for you, automatically |
| Web search | – | – | /search, /serp and /answer |
| Price | free, forever | free (your hosting) | 1,000 free credits a month; plans from $10 |
🤓 My Personal Story
I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.
In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.
I made it open source for availability, anyone can use it without a gate. Now I’m building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.
That platform is live now: Crawl4AI Cloud.
Why developers pick Crawl4AI
LLM-ready output: smart Markdown with headings, tables, code and citation hints
Fast in practice: async browser pool, caching, minimal hops
Full control: sessions, proxies, cookies, user scripts, hooks
Adaptive intelligence: learns site patterns, explores only what matters
Deploy anywhere: no keys needed, CLI and Docker, or the hosted cloud