lucasmartins-ai/lookacrawler

**Free, open-source, token-efficient local alternative to Firecrawl with native Model Context Protocol (MCP) Server for LLMs.**
---
📌 Why LookaCrawler?
Web crawling for Large Language Models (LLMs) is broken by default: modern web pages contain massive HTML bloat (scripts, tracking pixels, nested divs, navigation headers, stylesheets), costing thousands of wasted tokens per page.
**LookaCrawler** is an open-source, token-optimized local crawler that cuts a median 90% of tokens on real pages (21–98% across 8 sites, see benchmark-live.ts), extracts clean Markdown, bypasses anti-bot barriers with stealth Playwright drivers, and exposes a native **Model Context Protocol (MCP) Server** ready for **Claude Desktop, Cursor, and Antigravity**.
---
🥊 Comparison: LookaCrawler vs Alternatives
| Feature | 🕷️ LookaCrawler | 🔥 Firecrawl (Cloud) | ⚡ Jina Reader |
| :--- | :---: | :---: | :---: |
| **Pricing / Cost** | **$0.00 (100% Free Open Source)** | $16 to $99+/month | Rate-limited API |
| **Token Reduction** | **Median 90% fewer tokens (real pages) + Footnotes** | Standard Markdown | Basic Markdown |
| **Autonomous Crawling** | **Native map & crawl (BFS + URL patterns)** | Cloud Crawler | Single-page only |
| **Pre-Crawl Actions** | **Native Playwright (click, scroll, fill)** | Paid Addon | None |
| **Link Formatting** | **Inline, References Footnotes, Strip** | Inline only | Inline only |
| **Data Privacy** | **100% Local (Zero Telemetry)** | Cloud Provider | Cloud API |
| **MCP Integration** | **Native Tools + Resources + Prompts** | Community Wrapper | None |
| **Stealth & Anti-Bot** | **Real Chrome + Stealth Fingerprint** | Cloud Proxies | Basic Headers |
| **Local SQLite Cache** | **Built-in (24h TTL cache)** | Redis / Paid Addon | None |
| **JS SPA Support** | **Playwright + Chrome Pool** | Cloud Headless | Headless |
---
🚀 Key Features
- **Token Economy First:** Automatically prunes scripts, styles, inline SVGs, tracking tags, navigations, footers, redundant forms, and boilerplate containers with high link density (>80%).
- **Advanced Link & Image Formatting:**
- link_format: Choose between inline (standard markdown), references (footnote citations [1], saving ~25% tokens on repetitive URLs), or strip (pure text).
- image_mode: Choose between ignore (zero tokens), alt_only (preserves semantic context without URL bloat), or markdown (full ).
- **Autonomous Mapping & Recursive Crawling:**
- map_website: Inspects /robots.txt, sitemaps, and root anchors to discover all pages in a domain.
- crawl_website: Breadth-first autonomous crawling with max depth, max pages, URL pattern filters (substring, * wildcard), and real-time token accounting.
- **Pre-Crawl Browser Actions:** Automate clicks, scrolls, typing, and waits in Playwright before extracting content (dismiss cookie banners, scroll for infinite loading, expand accordions).
- **Dual Hybrid Crawling Engine:**
- fast: Ultra-fast native HTTP GET with backoff. Auto-escalates to deep if an anti-bot challenge is encountered.
- deep: Headless Playwright engine launching real Google Chrome with stealth patches (navigator.webdriver cleared, WebGL spoofed, CDP leaks stripped) to transparently crawl Cloudflare/Turnstile-protected pages.
- **Native MCP Ecosystem:**
- **Tools:** extract_web_content, crawl_website, map_website, batch_extract_web_content, extract_structured_data, extract_contacts, search_places, discover_local_leads, deep_enrich_lead, lint_anti_slop, lookup_social_profile, search_social_intent, audit_company_website, analyze_competitors, generate_growth_plan, gtm_pipeline_run.
- **Resources:** Live telemetry at crawler://metrics and cache analytics at crawler://cache/stats.