Mingye-Lu/AgenticCrawler

██╔══██╗██╔════╝██╔══██╗██╔══██╗██║ ██║██║
███████║██║ ██████╔╝███████║██║ █╗ ██║██║
██╔══██║██║ ██╔══██╗██╔══██║██║███╗██║██║
██║ ██║╚██████╗██║ ██║██║ ██║╚███╔███╔╝███████╗
╚═╝ ╚═╝ ╚═════╝╚═╝ ╚═╝╚═╝ ╚═╝ ╚══╝╚══╝ ╚══════╝
Browser automation agent that acts and observes. Navigate, click, and extract — or inspect network traffic, debug console errors, profile performance, and audit accessibility. One Rust binary.
Single Rust binary. Full DevTools observability. 43 tools. 25 LLM providers. MCP server built-in.
---
Why acrawl?
Most browser agents stop at navigate and click. acrawl goes further: the agent can also inspect network requests, analyze console errors, measure page performance, audit accessibility, and intercept network calls — the full DevTools surface, available as first-class agent tools.
It ships as a single Rust binary. No Python runtime, no Node runtime, no Docker. Drop it into any server or CI pipeline and describe a goal; the agent figures out what to visit, what to click, what to inspect, and when it's done.
- **One binary, zero runtimes.** cargo build --release produces a self-contained executable. No Python, no Node runtime — just Rust and a Chromium download for browser automation.
- **Acts and observes.** The agent has the full DevTools surface as first-class tools: inspect network requests with timing, analyze and deduplicate console logs, stream WebSocket messages, measure page performance (TTFB, resource breakdown), audit cookies and browser storage, measure JS/CSS coverage, run axe-core WCAG accessibility audits, and intercept or mock network calls. No other agent framework exposes this.
- **Deterministic where you can, AI where you must.** Define loops, conditionals, and parallel branches as JSON scripts — executed without any LLM calls. Fall back to the agent when pages behave unexpectedly. Best of both worlds.
- **No code required.** Describe the goal in plain English. The agent plans, navigates, and extracts.
- **Smart fetching.** Static pages are served over HTTP (fast). When JavaScript or interaction is needed, acrawl detects JS framework markers (__next_data__, __nuxt, __vue, ng-app, React roots), auth redirects, and short bodies — then transparently escalates to a headless browser.
- **Sub-agent parallelism.** Fork child agents onto separate browser tabs with independent state and step budgets. A URL-claiming registry prevents siblings from crawling the same page twice.
- **MCP client and server.** Extend the agent with custom tools via Model Context Protocol servers. Or flip it: acrawl mcp exposes 38 browser and DevTools tools plus run_goal to Claude Code, Cursor, VS Code, Zed, and 13 other clients.
- **25 LLM providers.** Anthropic, OpenAI, Google Gemini, DeepSeek, AWS Bedrock, Azure OpenAI, Vertex AI, GitHub Copilot, Groq, Mistral, xAI, Cohere, Alibaba DashScope, OpenRouter, and more. Or bring your own via any OpenAI-compatible endpoint.
How does it compare?
vs. AI web agents and scraping tools
| | acrawl | browser-use | Stagehand | Skyvern | Firecrawl | Playwright MCP | Scrapy | Playwright scripts |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| No code needed | Yes | No | No | Partial | No | No | No | No |
| Single binary | Yes | No | No | No | No | No | No | No |
| JS rendering | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes |
| LLM-powered navigation | Yes | Yes | Yes | Yes | Limited | No | No | No |
| No Python / Node needed | Yes | No | No | No | No | No | No | No |
| Form filling / interaction | Yes | Yes | Yes | Yes | No | Yes | No | Yes |
| Sub-agent parallelism | Yes | No | No | Partial | Partial | No | Partial | No |
| 25 LLM providers | Yes | Via LiteLLM | Partial | Partial | N/A | N/A | N/A | N/A |
| MCP client (use tools) | Yes | No | No | No | No | No | No | No |
| MCP server (expose as tools) | Yes | No | No | No | Yes | Yes | No | No |
| Stealth browser built-in | Yes | Cloud only | Via Browserbase | Cloud only | No | No | No | No |
| DevTools observability (network, console, perf, a11y) | Yes | No | No | No | No | No | No | No |
| Deterministic script layer (zero LLM calls) | Yes | No | Partial | No | No | No | Yes | Yes |
| Open source | Yes | Yes (MIT) | Yes (MIT) | Yes (Apache) | Engine only | Yes (MIT) | Yes (BSD) | Yes (Apache) |
Notes:
- **browser-use** (85k+ GitHub stars): Python + Playwright, DOM + screenshots, supports GPT/Claude/Gemini/Ollama via LiteLLM, 89.1% WebVoyager. No single binary — requires Python and pip install. Every action calls an LLM: 2-5s/step, ~$0.02-0.30/task. Cloud tier adds stealth; self-hosted is bare Playwright.