77777R7/Octocrawl

One link. Web data, ready.
Markdown and data from web pages, for AI agents and RAG.
Fewer tokens than raw HTML. Every result records how it was read.
Website ·
Docs ·
Connect MCP ·
Blog ·
Changelog
Scrape pages. Crawl websites. Turn web content into usable data.
Octocrawl is an open-source web scraping and crawling toolkit for AI agents, RAG and research. Collect documentation for a knowledge base, extract product details for comparison, or turn web tables and listings into data for analysis. Each page comes with a source record.
Try a page in your browser, or connect your agent through hosted MCP or the REST API. No installation or API key is needed to try the hosted service.
## What can you do with Octocrawl?
| Capability | What you get |
| Scrape a page | Readable Markdown for LLM context, plus HTML, links, image URLs and page metadata. Request a screenshot on the browser lane. |
| Map a website | Discover URLs from sitemaps and page links, and filter them by a search term to find the pages you need. |
| Crawl a site or section | Follow links from a starting URL and collect each page's content, with path filters, depth and page limits. |
| Scrape a list of URLs | Submit a batch, retrieve results as they arrive, and keep the outcome of every URL, including failures. |
| Extract structured data | Get tables and repeated listings as JSON and CSV, or pass a JSON Schema for supported fields such as product names, prices and page labels. Extraction guide. |
| Read dynamic pages · local actions | Render JavaScript, click controls, scroll, load more items and follow pagination before extracting content. Browser actions. |
| Read PDF reports | Extract a PDF's text layer with page markers for citations. CSV, JSON and plain-text URLs also return their text. File support. |
| Revisit sources on a schedule · local workers | Run Monitors and send their events to an HTTPS endpoint with retried delivery. Monitor setup. |
Hosted Octocrawl offers scrape and map, and with an account crawl and batch jobs (up to 500 pages a crawl, 1,000 URLs a batch); keyless requests use HTTP, and an account key enables the browser lane. Run locally for larger jobs, page actions and Monitors; see Running Octocrawl for setup and self-hosting options. The web preview is a single-page trial, with five pages a day and up to 20 named fields.
## Why Octocrawl?
Web data inside your agent. Connect Claude Code, Cursor, OpenCode or Codex through MCP, or build with REST, TypeScript, Python and the Vercel AI SDK.
Structured fields without a model by default. Read supported values from the page itself, with field evidence and explicit missing values. Model fallback is opt-in.
Know when a scrape fell short. Checks flag challenge pages, wrong content, missing facts, truncation and too little content. Blocked and failed results carry a reason; fetched error pages keep their status and Markdown. Result states.
Trace results back to their sources. Each page carries an Evidence Record with its URL, fetch time, status and extraction evidence. Unobserved values stay null.
Run it where you need it. Use the hosted service, run with npx, or self-host with Docker Compose. The server is AGPL-3.0; the client libraries are MIT.