pragmar/mcp-server-webcrawl

GitHub |
Docs |
PyPi
mcp-server-webcrawl
Advanced search and retrieval for web crawler data. With **mcp-server-webcrawl**, your AI client filters and analyzes web content under your direction or autonomously. The server includes a fulltext search interface with boolean support, and resource filtering by type, HTTP status, and more.
**mcp-server-webcrawl** provides the LLM a complete menu with which to search, and works with a variety of web crawlers:
| Crawler/Format | Description | Platforms | Setup Guide |
|---|---|---|---|
| [**ArchiveBox**][1] | Web archiving tool | macOS/Linux | [Setup Guide][8] |
| [**HTTrack**][2] | GUI mirroring tool | macOS/Windows/Linux | [Setup Guide][9] |
| [**InterroBot**][3] | GUI crawler and analyzer | macOS/Windows/Linux | [Setup Guide][10] |
| [**Katana**][4] | CLI security-focused crawler | macOS/Windows/Linux | [Setup Guide][11] |
| [**SiteOne**][5] | GUI crawler and analyzer | macOS/Windows/Linux | [Setup Guide][12] |
| [**WARC**][6] | Standard web archive format | varies by client | [Setup Guide][13] |
| [**wget**][7] | CLI website mirroring tool | macOS/Linux | [Setup Guide][14] |
[1]: https://archivebox.io
[2]: https://github.com/xroche/httrack
[3]: https://interro.bot
[4]: https://github.com/projectdiscovery/katana
[5]: https://crawler.siteone.io
[6]: https://en.wikipedia.org/wiki/WARC_(file_format)
[7]: https://en.wikipedia.org/wiki/Wget
[8]: https://pragmar.github.io/mcp-server-webcrawl/guides/archivebox.html
[9]: https://pragmar.github.io/mcp-server-webcrawl/guides/httrack.html
[10]: https://pragmar.github.io/mcp-server-webcrawl/guides/interrobot.html
[11]: https://pragmar.github.io/mcp-server-webcrawl/guides/katana.html
[12]: https://pragmar.github.io/mcp-server-webcrawl/guides/siteone.html
[13]: https://pragmar.github.io/mcp-server-webcrawl/guides/warc.html
[14]: https://pragmar.github.io/mcp-server-webcrawl/guides/wget.html
**mcp-server-webcrawl** is free and open source, and requires Claude Desktop and Python (>=3.10). It is installed on the command line, via pip install:
For step-by-step MCP server setup, refer to the Setup Guides.
Features
* Claude Desktop ready
* Multi-crawler compatible
* Filter by type, status, and more
* Boolean search support
* Support for Markdown and snippets
* Roll your own website knowledgebase
Prompt Routines
**mcp-server-webcrawl** provides the toolkit necessary to search web crawl data freestyle, figuring it out as you go, reacting to each query. This is what it was designed for.
It is also capable of running routines (as prompts). You can write these yourself, or use the ones provided. These prompts are **copy and paste**, and used as raw Markdown. They are enabled by the advanced search provided to the LLM; queries and logic can be embedded in a procedural set of instructions, or even an input loop as is the case with Gopher Service.
| Prompt | Download | Category | Description |
|--------|----------|----------|-------------|
|🔍 **SEO Audit** | auditseo.md | audit | Technical SEO (search engine optimization) analysis. Covers the basics, with options to dive deeper. |
|🔗 **404 Audit** | audit404.md | audit | Broken link detection and pattern analysis. Not only finds issues, but suggests fixes. |
|⚡ **Performance Audit** | auditperf.md | audit | Website speed and optimization analysis. Real talk. |
|📁 **File Audit** | auditfiles.md | audit | File organization and asset analysis. Discover the composition of your website. |
|🌐 **Gopher Interface** | gopher.md | interface | An old-fashioned search interface inspired by the Gopher clients of yesteryear. |
|⚙️ **Search Test** | testsearch.md | self-test | A battery of tests to check for Boolean logical inconsistencies in the search query parser and subsequent FTS5 conversion. |