FayAndXan/spectrawl

The unified web layer for AI agents. Search, browse, crawl, extract, and act on platforms — one package, self-hosted.
**5,000 free searches/month** via Gemini Grounded Search. Full page scraping, stealth browsing, multi-page crawling, structured extraction, AI browser agent, 24 platform adapters.
What It Does
AI agents need to interact with the web — searching, browsing pages, crawling sites, logging into platforms, posting content. Today you wire together Playwright + a search API + cookie managers + platform-specific scripts. Spectrawl is one package that does all of it.
How It Works
Spectrawl searches via Gemini Grounded Search (Google-quality results), scrapes the top pages for full content, and returns everything to your agent. Your agent's LLM reads the actual sources and forms its own answer — no pre-chewed summaries.
Quick Start
> **Why no summary by default?** Your agent already has an LLM. If we summarize AND your agent summarizes, you're paying two LLMs for one answer. We return rich sources — your agent does the rest.
Spectrawl vs Others
| | Tavily | Crawl4AI | Firecrawl | Stagehand | Spectrawl |
|---|---|---|---|---|---|
| Speed | ~2s | ~5s | ~3s | ~3s | ~6-10s |
| Free tier | 1,000/mo | Unlimited | 500/mo | None | 5,000/mo |
| Returns | Snippets + AI | Markdown | Markdown/JSON | Structured | Full page + structured |
| Self-hosted | No | Yes | Yes | Yes | Yes |
| Anti-detect | No | No | No | No | **Yes (Camoufox)** |
| Block detection | No | No | No | No | **8 services** |
| CAPTCHA solving | No | No | No | No | **Yes (Gemini Vision)** |
| Structured extraction | No | No | No | **Yes** | **Yes** |
| NL browser agent | No | No | No | **Yes** | **Yes** |
| Network capturing | No | Yes | No | No | **Yes** |
| Multi-page crawl | No | Yes | Yes | No | **Yes (+ sitemap)** |
| Platform posting | No | No | No | No | **24 adapters** |
| Auth management | No | No | No | No | **Cookie store + refresh** |
Search
Two modes: **basic search** and **deep search**.
Basic Search
Returns raw search results from the engine cascade. Fast, lightweight.
Deep Search
Full pipeline: query expansion → parallel search → merge/dedup → rerank → scrape top N → optional AI summary with citations.
Search Engine Cascade
Default cascade: **Gemini Grounded → Tavily → Brave**
Gemini Grounded Search gives you Google-quality results through the Gemini API. Free tier: 5,000 grounded queries/month.
| Engine | Free Tier | Key Required | Default |
|--------|-----------|-------------|---------|
| **Gemini Grounded** | 5,000/month | GEMINI_API_KEY | ✅ Primary |
| Tavily | 1,000/month | TAVILY_API_KEY | ✅ 1st fallback |
| Brave | 2,000/month | BRAVE_API_KEY | ✅ 2nd fallback |
| DuckDuckGo | Unlimited | None | Available |
| Bing | Unlimited | None | Available |
| Serper | 2,500 trial | SERPER_API_KEY | Available |
| Google CSE | 100/day | GOOGLE_CSE_KEY | Available |
| Jina Reader | Unlimited | None | Available |
| SearXNG | Unlimited | Self-hosted | Available |
Deep Search Pipeline
What you get without any keys
DDG-only search, raw results, no AI answer. Works from home IPs. Datacenter IPs get rate-limited by DDG — recommend at minimum a free Gemini key.
Browse
Stealth browsing with anti-detection. Three tiers (auto-escalation):
1. **Playwright + stealth plugin** — default, works immediately
2. **Camoufox binary** — engine-level anti-fingerprint (npx spectrawl install-stealth)
3. **Remote Camoufox** — for existing deployments
If tier 1 gets blocked, Spectrawl automatically escalates to tier 2 (if installed) or tier 3 (if configured). No manual intervention needed.
Browse Options
Browse Response
Block Page Detection
Spectrawl detects block/challenge pages from **8 anti-bot services** and reports them in the response instead of returning garbage HTML:
- **Cloudflare** (including RFC 9457 structured errors)
- **Akamai**
- **AWS WAF**
- **Imperva / Incapsula**
- **DataDome**
- **PerimeterX / HUMAN**
- **hCaptcha** challenges