aparajithn/agent-scraper-mcp

**The #1 most requested utility for AI agents** — professional web scraping, screenshots, and content extraction via MCP + REST API.
Features
🌐 **Clean Content Extraction** — Extract readable text/markdown from any webpage (like Readability)
🎯 **Structured Scraping** — Extract specific data using CSS selectors
📸 **Screenshots** — Capture full-page or viewport screenshots with Playwright
🔗 **Link Extraction** — Get all links from a page with optional regex filtering
📋 **Metadata Extraction** — Extract title, description, Open Graph tags, favicon, etc
🔍 **Google Search** — Search Google and get results programmatically
Quick Start
MCP Configuration
Add to your MCP settings file (cline_mcp_settings.json or similar):
REST API
Base URL: https://agent-scraper-mcp.onrender.com
Scrape URL (Clean Content)
Response:
Scrape Structured Data
Response:
Screenshot URL
Response:
Extract Links
Extract Metadata
Search Google
Pricing
Free Tier
- **50 requests per IP per day**
- All tools included
- No credit card required
Paid Tier (HTTP 402 / x402 Payment)
After free tier exhausted:
- **Scraping tools**: $0.005/request (scrape_url, scrape_structured, extract_links, extract_meta, search_google)
- **Screenshot tool**: $0.01/request (higher due to compute cost)
Payment via the x402 protocol (HTTP 402) in USDC on Base:
- Wallet address (payTo): 0x8E844a7De89d7CfBFe9B4453E65935A22F146aBB
When you exceed the free tier, the server responds with HTTP 402 carrying
machine-readable payment requirements (x402 v1), so x402-aware clients pay and
retry automatically:
To pay manually, attach an X-Payment header containing a base64-encoded
JSON envelope with an EIP-3009 TransferWithAuthorization (USDC on Base)
signed with the payer's EIP-712 key for the exact price, addressed to the
wallet above, with an unused nonce. Payments are verified cryptographically
(signature recovery + amount/recipient/deadline/replay checks) before the
request is served — applies to both the REST API and MCP tools/call.
Collecting revenue (server operator)
With X402_SETTLER_PRIVATE_KEY set, every verified payment is executed
on-chain within seconds: the EIP-3009 transferWithAuthorization is submitted
to the USDC contract on Base and the USDC moves from the payer to
X402_WALLET_ADDRESS. The settler key does **not** need to be the receiving
wallet's key — any address may execute the authorization, so a dedicated
gas-only burner key can submit the transactions while funds land in the main
wallet. It only needs a small ETH balance on Base for gas. Additional env vars:
X402_RPC_URL (default https://mainnet.base.org) and
X402_SETTLE_MIN_MICROUSD (default 0 — settle each payment immediately;
raise it to batch small payments). Failed settlements are retried with
backoff and, if all retries fail, the full authorization is written to the
logs (CRITICAL) for manual redemption — no payment is silently lost.
Tools Reference
1. scrape_url
Extract clean, readable content from any webpage (like Readability).
**Parameters:**
- url (string, required): URL to scrape
- format (string, optional): Output format — text, markdown, or html (default: markdown)
**Returns:** {success, url, title, content, format}