0xMassi/webclaw

webclaw
Turn websites into clean markdown, JSON, and LLM-ready context.
CLI, MCP server, REST API, and SDKs for AI agents and RAG pipelines.
---
Most web scraping tools give your agent one of two bad outputs:
- a blocked page, login wall, or empty app shell
- raw HTML full of nav, scripts, styling, ads, and duplicated boilerplate
webclaw.io is the hosted web extraction API for webclaw. This repo contains the open-source CLI, MCP server, extraction engine, and self-hostable server.
webclaw turns a URL into clean content your tools can actually use.
Use it from the terminal, wire it into Claude/Cursor through MCP, call the hosted API from your app, or self-host the OSS server.
---
Install
Agent setup
The fastest way to connect webclaw to Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, and other MCP-compatible tools:
The installer detects supported clients and configures the MCP server for you.
Homebrew
Prebuilt binaries
Download macOS, Linux, and Windows binaries from GitHub Releases.
Windows (x64)
The prebuilt ZIP is the easiest installation and does not require Rust. On the
v0.6.22 release page,
download webclaw-v0.6.22-x86_64-pc-windows-msvc.zip. Extract it in File Explorer,
open PowerShell in the extracted folder containing webclaw.exe, and run:
PowerShell requires the .\ prefix for programs in the current directory.
Add that directory to your user PATH only if you want to run webclaw from
other folders. If Windows reports a missing Visual C++ runtime DLL, install the
Microsoft Visual C++ Redistributable for x64. The ZIP contains x64 binaries;
Windows ARM64 is not covered by the native installation check.
To build from source on Windows, install Rust with the x86_64-pc-windows-msvc
toolchain, Visual Studio Build Tools with **Desktop development with C++** and
the Windows SDK, Git, CMake 3.22 or newer, LLVM (including libclang.dll), and
NASM. Open a fresh Developer PowerShell for Visual Studio after installation.
Ensure CMake and NASM are on PATH and set LIBCLANG_PATH to your LLVM bin
directory if bindgen cannot find it, for example:
The Cargo package is webclaw-cli; the executable is webclaw. These packages
are installed from Git, not crates.io. A first source build can take several
minutes. The Windows CI workflow checks the Git installation and published ZIP
on a hosted runner with preinstalled build tools; it does not model a clean PC.
Linux containers, including Docker on macOS, do not validate Windows support.
Docker
Cargo
If building from source fails because native build tools are missing, install the platform prerequisites:
| OS | Command |
| --- | --- |
| Debian / Ubuntu | sudo apt install -y pkg-config libssl-dev cmake clang git build-essential |
| Fedora / RHEL | sudo dnf install -y pkg-config openssl-devel cmake clang git make gcc |
| Arch | sudo pacman -S pkg-config openssl cmake clang git base-devel |
| macOS | xcode-select --install |
---
Quick Start
Scrape one page
Return LLM-optimized text
Keep only the main content
Include or exclude selectors
Crawl a documentation site
Workflow examples
- HTML to Markdown for RAG
- Firecrawl-compatible API
- MCP web scraping
- Proxy-backed crawling with ColdProxy
- Cloudflare diagnostics
Extract brand assets
Compare a page over time
---
MCP Server
webclaw ships with an MCP server for AI agents.
Zero-install — point any MCP client at the npx launcher:
Or run npx create-webclaw to auto-detect your AI tools and write their configs for you.
Then ask your agent things like:
---
Use as an agent skill
Add webclaw to Claude Code, Cursor, Windsurf, and other MCP agents in one command:
Your agent gets scrape, crawl, map, extract, summarize, diff, brand, and search
as native tools. Most sites extract locally with no API key. Set WEBCLAW_API_KEY