bighippoman/intercept-mcp

Give your AI the ability to read the web. One command, no API keys required.
Without it, your AI hits a URL and gets a 403, a wall, or a wall of raw HTML. With intercept, it almost always gets the content — clean markdown, ready to use.
Handles tweets, YouTube videos (with transcripts when available), arXiv papers, PDFs, Wikipedia articles, and GitHub repos. If the first strategy fails, it tries up to 14 more before giving up.
Works with any MCP client: Claude Code, Claude Desktop, Codex, Cursor, Windsurf, Cline, and more.
Install
Claude Code
Codex
Cursor
Settings → MCP → Add Server:
Windsurf
Settings → MCP → Add Server → same JSON config as above.
Claude Desktop
Add to your claude_desktop_config.json:
Other MCP clients
Any client that supports stdio MCP servers can run npx -y intercept-mcp.
No API keys needed for the fetch tool.
How it works
URLs are processed in four stages:
1. Site-specific handlers
Known URL patterns are routed to dedicated handlers before the fallback pipeline:
| Pattern | Handler | What you get |
|---------|---------|-------------|
| twitter.com/*/status/*, x.com/*/status/* | Twitter/X | Tweet text, author, media, engagement stats (via third-party APIs) |
| youtube.com/watch?v=*, youtu.be/* | YouTube | Title, channel, duration, views, description, transcript (when captions available) |
| arxiv.org/abs/*, arxiv.org/pdf/* | arXiv | Paper metadata, authors, abstract, categories |
| *.pdf | PDF | Extracted text (text-layer PDFs only) |
| *.wikipedia.org/wiki/* | Wikipedia | Clean article content via Wikimedia REST API |
| github.com/{owner}/{repo} | GitHub | Raw README.md content |
| github.com/{o}/{r}/blob/{ref}/{path} | GitHub | Raw file content, code-fenced by language |
| github.com/{o}/{r}/issues/{n}, /pull/{n} | GitHub | Issue/PR title, state, body, diff stats, comments (via GitHub API) |
| github.com/{o}/{r}/releases/tag/{t}, /releases/latest | GitHub | Release notes (via GitHub API) |
The GitHub API endpoints work unauthenticated (60 requests/hour). Set GITHUB_TOKEN to raise the limit.
2. Shared cache (agentsweb.org)
Before hitting any fetcher, every request checks agentsweb.org — a global shared markdown cache for AI agents backed by a 9-source parallel fetch pipeline with JS/SPA rendering (React, Vue, Angular via Cloudflare Browser Run). If another agent already fetched this URL, you get the result in under 50ms.
Every successful fetch contributes back automatically. Entries gain trust through a self-healing consensus model: when independent instances fetch the same URL and confirm the same content, confidence increases.
Opt out entirely with INTERCEPT_SHARED_CACHE=false, or use read-only mode (consume but never contribute) with INTERCEPT_CACHE_READ_ONLY=true.
agentsweb.org API
agentsweb.org also exposes standalone endpoints for direct use:
- **/web?q=** — search the web
- **/research?q=** — search + fetch + cache in one call
- **/fetch?url=** — fetch on demand, auto-cached
See agentsweb.org/docs for full API documentation.
3. Fallback pipeline
If no handler matches (or the handler returns nothing), the URL enters the multi-tier pipeline:
| Tier | Fetcher | Strategy |
|------|---------|----------|
| 0 | agentsweb.org | Global shared markdown cache — instant if another agent already fetched this URL |
| 1 | Cloudflare Browser Run | JS/SPA rendering + markdown extraction — also powers agentsweb.org (optional, needs API token) |
| 1 | Jina Reader | Clean markdown extraction service |
| 2 | Wayback Machine | Archived version from archive.org |
| 2 | Arquivo.pt | Portuguese web archive (broad international coverage) |
| 2 | Common Crawl | Petabyte web archive read from Common Crawl's index + S3 — not subject to the origin's rate limits, bot detection, or paywall |
| 2 | Codetabs | CORS proxy |
| 3 | Markdown endpoint | Asks the site for a native markdown version ( .md + Accept: text/markdown) |
| 3 | archive.ph | Archived snapshots via timemap API + stealth TLS fetch |
| 3 | Raw fetch | Direct GET with browser headers + Turndown markdown conversion |
| 3 | Stealth fetch | Browser TLS fingerprint impersonation via got-scraping (opt-in, see below) |
| 3 | FlareSolverr | Real-browser challenge solver for Cloudflare/DDoS-Guard (opt-in, needs a FlareSolverr instance) |