just-every/mcp-read-website-fast

Fast, token-efficient web content extraction for AI agents - converts websites to clean Markdown.
Overview
Existing MCP web crawlers are slow and consume large quantities of tokens. This pauses the development process and provides incomplete results as LLMs need to parse whole web pages.
This MCP package fetches web pages locally, strips noise, and converts content to clean Markdown while preserving links. Designed for Claude Code, IDEs and LLM pipelines with minimal token footprint. Crawl sites locally with minimal dependencies.
**Note:** This package now uses @just-every/crawl for its core crawling and markdown conversion functionality.
Features
- **Fast startup** using official MCP SDK with lazy loading for optimal performance
- **Content extraction** using Mozilla Readability (same as Firefox Reader View)
- **HTML to Markdown** conversion with Turndown + GFM support
- **Smart caching** with SHA-256 hashed URLs
- **Polite crawling** with robots.txt support and rate limiting
- **Concurrent fetching** with configurable depth crawling
- **Stream-first design** for low memory usage
- **Link preservation** for knowledge graphs
- **Optional chunking** for downstream processing
Installation
Claude Code
VS Code
Cursor
JetBrains IDEs
Settings → Tools → AI Assistant → Model Context Protocol (MCP) → Add
Choose “As JSON” and paste:
Or, in the chat window, type /add and fill in the same JSON—both paths land the server in a single step. 
Raw JSON (works in any MCP client)
Drop this into your client’s mcp.json (e.g. .vscode/mcp.json, ~/.cursor/mcp.json, or .mcp.json for Claude).
Features
- **Fast startup** using official MCP SDK with lazy loading for optimal performance
- **Content extraction** using Mozilla Readability (same as Firefox Reader View)
- **HTML to Markdown** conversion with Turndown + GFM support
- **Smart caching** with SHA-256 hashed URLs
- **Polite crawling** with robots.txt support and rate limiting
- **Concurrent fetching** with configurable depth crawling
- **Stream-first design** for low memory usage
- **Link preservation** for knowledge graphs
- **Optional chunking** for downstream processing
Available Tools
- read_website - Fetches a webpage and converts it to clean markdown
- Parameters:
- url (required): The HTTP/HTTPS URL to fetch
- pages (optional): Maximum number of pages to crawl (default: 1, max: 100)
Available Resources
- read-website-fast://status - Get cache statistics
- read-website-fast://clear-cache - Clear the cache directory
Development Usage
Install
Single page fetch
Crawl with depth
Output formats
CLI Options
- -p, --pages - Maximum number of pages to crawl (default: 1)
- -c, --concurrency - Max concurrent requests (default: 3)
- --no-robots - Ignore robots.txt
- --all-origins - Allow cross-origin crawling
- -u, --user-agent - Custom user agent
- --cache-dir - Cache directory (default: .cache)