Grubbomatic/crawl-readiness-mcp

**An MCP server that checks a website's AI readiness: can ChatGPT, Claude, Perplexity, Google AI and 50+ other AI crawlers access it?** Then it generates the fixes (llms.txt, an AI-crawler robots.txt, JSON-LD structured data) and reads your LLM Monitor results, inside Claude Desktop, Cursor, Cline, Windsurf and other MCP clients.
Ask Claude "is my site AI-readable?" and it'll actually check — running a live audit against 50+ AI crawlers (ChatGPT, Claude, Perplexity, Google AI, and more). Then ask it to fix what's broken, and it can generate a proper llms.txt, robots.txt, or JSON-LD schema and write it straight into your project.
No more copy-paste between your editor and yet another SEO tool.
---
What it does
Eight tools your AI assistant can call:
**Audit (no signup):**
- check_ai_readiness — Score a site 0–100 against 50+ AI crawlers, with a prioritized fix list; extras that don't affect the score are marked as such
- validate_schema — Audit JSON-LD structured data on any URL (20+ types, per-type rules)
- validate_robots — Line-by-line robots.txt audit
- check_content_parity — Compare what humans see vs what AI crawlers see
**Generate (requires free API key):**
- generate_llms_txt — Auto-crawl a site and produce a properly formatted llms.txt
- generate_robots_txt — AI-crawler-aware robots.txt with preset policies
- generate_schema — Organization / WebSite / Article JSON-LD ready to paste
**Monitor (read-only, requires free API key):**
- get_monitor_trend — See whether ChatGPT, Claude, Perplexity & Google AI mention your brand vs competitors, and how that's trending week over week (reads your LLM Monitor projects; doesn't trigger runs)
---
Install
Claude Desktop, one click
Download crawl-readiness-mcp- .mcpb from the latest release and double-click it (or drag it into the Claude Desktop window). Claude Desktop shows what the extension does and asks for the optional API key. The bundle carries the server and its dependencies, so nothing else is installed.
Claude Code, as a plugin
This repository is also a Claude plugin: the .mcp.json at its root starts the server (pinned to this version), and the ai-readiness skill tells Claude when to use which tool and how to read the results. Add it from the Claude directory once it is listed there. To try it before then, clone the repository and start Claude Code with it loaded:
Claude Code asks for the optional API key when the plugin is enabled and keeps it in secure storage; the plugin's own files never contain it.
For Claude Desktop, by hand
Open your config file:
- **macOS:** ~/Library/Application Support/Claude/claude_desktop_config.json
- **Windows:** %APPDATA%\Claude\claude_desktop_config.json
Add this to the mcpServers block:
Restart Claude Desktop. You'll see "crawl-readiness" listed in the tools panel.
For Cursor
Open Cursor Settings → **Features** → **Model Context Protocol** → **Add new global MCP server**.
For Cline / Windsurf / Continue
Same shape — add crawl-readiness to your MCP servers list with npx -y crawl-readiness-mcp as the command.
---
Unlock the generator tools
The four audit tools work out of the box, no signup. The three generator tools need an API key — free at crawlreadiness.com/dashboard (50 checks/month on the free tier, no credit card).
Once you have a key, add it via env in the same config: