7
77777R7/Octocrawl
Octocrawl: readable Markdown and checked fields from public web pages, for AI agents, RAG and research. Failures come back with a reason. Self-host with REST, SDK or MCP.
no SECURITY.md
View on GitHub ↗Link checked 2026-11-12 · automated check blocked (HTTP 429), status unconfirmed
Install & config
Remote endpoint (mcpServers - Claude Desktop / Cursor / Windsurf)
{
"mcpServers": {
"77777r7-octocrawl": {
"url": "https://mcp.octocrawl.dev/mcp"
}
}
}Paste this into the mcpServers config of Claude Desktop or Cursor. The endpoint comes from public evidence; we do not install or host it. Official Registry record: this endpoint lists no request headers. That is not the same as key-free - the project docs decide.
VS Code (.vscode/mcp.json - the key is servers, remote needs type: http)
{
"servers": {
"77777r7-octocrawl": {
"type": "http",
"url": "https://mcp.octocrawl.dev/mcp"
}
}
}VS Code differs from the other three: the top-level key is <code>servers</code> and a remote endpoint also needs <code>"type": "http"</code>. Paste into .vscode/mcp.json.
Claude Code (command line)
claude mcp add --transport http 77777r7-octocrawl https://mcp.octocrawl.dev/mcp
Claude Code does not read a config file; add the server with this command.
Install command from the repository
claude mcp add octocrawl -- npx -y @octocrawl/mcp # the published stdio server, a client of that API
Parsed from the README we fetched on 2026-10-26; we have not executed it.
This is the install method the repository states, not an MCP client config, so no JSON is generated.
Key names differ by client: Clients & CLI
- Star count
- 6 ★ · 77777R7/Octocrawl · read from the GitHub search API · (checked 2026-11-13 20:00:50)
- Source
- collected from the punkpeye/awesome-mcp-servers list · type: MCP server
- Harvest history
- first seen 2026-10-23 · last seen 2026-11-14 · seen 26× · listed by 23 sources
- Score
- score 2,300 · 6 ★ · 23 sources · mcp
- Install
claude mcp add octocrawl -- npx -y @octocrawl/mcp # the published stdio server, a client of that API
FAQ (answers come from our records)
How do I install it?
The repository provides this install command:
claude mcp add octocrawl -- npx -y @octocrawl/mcp # the published stdio server, a client of that API. It comes from the repository itself; we cannot guarantee it runs as-is in your client.Who is it for?
We filed it under these tags: MCP 服务. The tags come from the entry itself, not a taxonomy we imposed.
Who maintains it?
repository
77777R7/Octocrawl；first catalogued 2026-10-23；last seen 2026-11-14。There is no official certification or quality rating here — we record sources and dates only.Octocrawl
以下摘录自该仓库的 README（片段） · 在 GitHub 读完整 README →
One link. Web data, ready.
Markdown and data from web pages, for AI agents and RAG.
Fewer tokens than raw HTML. Every result records how it was read.
Website ·
Docs ·
Connect MCP ·
Blog ·
Changelog
Scrape pages. Crawl websites. Turn web content into usable data.
Octocrawl is an open-source web scraping and crawling toolkit for AI agents, RAG and research. Collect documentation for a knowledge base, extract product details for comparison, or turn web tables and listings into data for analysis. Each page comes with a source record.
Try a page in your browser, or connect your agent through hosted MCP or the REST API. No installation or API key is needed to try the hosted service.
## What can you do with Octocrawl?
| Capability | What you get |
| Scrape a page | Readable Markdown for LLM context, plus HTML, links, image URLs and page metadata. Request a screenshot on the browser lane. |
| Map a website | Discover URLs from sitemaps and page links, and filter them by a search term to find the pages you need. |
| Crawl a site or section | Follow links from a starting URL and collect each page's content, with path filters, depth and page limits. |
| Scrape a list of URLs | Submit a batch, retrieve results as they arrive, and keep the outcome of every URL, including failures. |
| Extract structured data | Get tables and repeated listings as JSON and CSV, or pass a JSON Schema for supported fields such as product names, prices and page labels. Extraction guide. |
| Read dynamic pages · local actions | Render JavaScript, click controls, scroll, load more items and follow pagination before extracting content. Browser actions. |
| Read PDF reports | Extract a PDF's text layer with page markers for citations. CSV, JSON and plain-text URLs also return their text. File support. |
| Revisit sources on a schedule · local workers | Run Monitors and send their events to an HTTPS endpoint with retried delivery. Monitor setup. |
Hosted Octocrawl offers scrape and map, and with an account crawl and batch jobs (up to 500 pages a crawl, 1,000 URLs a batch); keyless requests use HTTP, and an account key enables the browser lane. Run locally for larger jobs, page actions and Monitors; see Running Octocrawl for setup and self-hosting options. The web preview is a single-page trial, with five pages a day and up to 20 named fields.
## Why Octocrawl?
Web data inside your agent. Connect Claude Code, Cursor, OpenCode or Codex through MCP, or build with REST, TypeScript, Python and the Vercel AI SDK.
Structured fields without a model by default. Read supported values from the page itself, with field evidence and explicit missing values. Model fallback is opt-in.
Know when a scrape fell short. Checks flag challenge pages, wrong content, missing facts, truncation and too little content. Blocked and failed results carry a reason; fetched error pages keep their status and Markdown. Result states.
Trace results back to their sources. Each page carries an Evidence Record with its URL, fetch time, status and extraction evidence. Unobserved values stay null.
Run it where you need it. Use the hosted service, run with npx, or self-host with Docker Compose. The server is AGPL-3.0; the client libraries are MIT.
Markdown and data from web pages, for AI agents and RAG.
Fewer tokens than raw HTML. Every result records how it was read.
Website ·
Docs ·
Connect MCP ·
Blog ·
Changelog
Scrape pages. Crawl websites. Turn web content into usable data.
Octocrawl is an open-source web scraping and crawling toolkit for AI agents, RAG and research. Collect documentation for a knowledge base, extract product details for comparison, or turn web tables and listings into data for analysis. Each page comes with a source record.
Try a page in your browser, or connect your agent through hosted MCP or the REST API. No installation or API key is needed to try the hosted service.
## What can you do with Octocrawl?
| Capability | What you get |
| Scrape a page | Readable Markdown for LLM context, plus HTML, links, image URLs and page metadata. Request a screenshot on the browser lane. |
| Map a website | Discover URLs from sitemaps and page links, and filter them by a search term to find the pages you need. |
| Crawl a site or section | Follow links from a starting URL and collect each page's content, with path filters, depth and page limits. |
| Scrape a list of URLs | Submit a batch, retrieve results as they arrive, and keep the outcome of every URL, including failures. |
| Extract structured data | Get tables and repeated listings as JSON and CSV, or pass a JSON Schema for supported fields such as product names, prices and page labels. Extraction guide. |
| Read dynamic pages · local actions | Render JavaScript, click controls, scroll, load more items and follow pagination before extracting content. Browser actions. |
| Read PDF reports | Extract a PDF's text layer with page markers for citations. CSV, JSON and plain-text URLs also return their text. File support. |
| Revisit sources on a schedule · local workers | Run Monitors and send their events to an HTTPS endpoint with retried delivery. Monitor setup. |
Hosted Octocrawl offers scrape and map, and with an account crawl and batch jobs (up to 500 pages a crawl, 1,000 URLs a batch); keyless requests use HTTP, and an account key enables the browser lane. Run locally for larger jobs, page actions and Monitors; see Running Octocrawl for setup and self-hosting options. The web preview is a single-page trial, with five pages a day and up to 20 named fields.
## Why Octocrawl?
Web data inside your agent. Connect Claude Code, Cursor, OpenCode or Codex through MCP, or build with REST, TypeScript, Python and the Vercel AI SDK.
Structured fields without a model by default. Read supported values from the page itself, with field evidence and explicit missing values. Model fallback is opt-in.
Know when a scrape fell short. Checks flag challenge pages, wrong content, missing facts, truncation and too little content. Blocked and failed results carry a reason; fetched error pages keep their status and Markdown. Result states.
Trace results back to their sources. Each page carries an Evidence Record with its URL, fetch time, status and extraction evidence. Unobserved values stay null.
Run it where you need it. Use the hosted service, run with npx, or self-host with Docker Compose. The server is AGPL-3.0; the client libraries are MIT.
Listed badge (optional)
Copy this line into the README if you want the listing mark. Optional, and it does not affect being listed.
[](https://anclum.com/en/s/77777r7-octocrawl/)
- Source
- awesome · https://github.com/77777R7/Octocrawl
- Browse by field
- MCP servers
- Traction
- 6 ★
- Listed by
- 23 independent sources
- added
- today
- Security policy
- no
SECURITY.mdat the repository root (which does not mean it is unsafe)
More like this
Filed under
Every tag opens the full list for that filter. Counts are the live catalogue figures.