webscraping-ai/webscraping-ai-mcp-server

> **Prefer zero setup?** Use the hosted remote MCP server:
> add https://mcp.webscraping.ai/mcp to your MCP client and sign in with your
> WebScraping.AI account — OAuth handles auth, no API key or local install
> needed. This repo is the open-source stdio version for self-hosting and
> customization.
A Model Context Protocol (MCP) server implementation that integrates with
WebScraping.AI for web data extraction capabilities —
Chromium JavaScript rendering, rotating datacenter/residential/stealth proxies,
and AI-powered question answering and structured field extraction on any page.
Sign up to get an API key — a free
trial, no credit card required. See the
API documentation for the full parameter reference.
Features
- Question answering about web page content
- Structured data extraction from web pages
- HTML content retrieval with JavaScript rendering
- Plain text extraction from web pages
- CSS selector-based content extraction
- Google search results (SERP) as parsed JSON
- Structured JSON for pages on supported sites (e.g. YouTube, TikTok, X, LinkedIn, Instagram, Reddit) from their normal URL
- Multiple proxy types (datacenter, residential, stealth) with country selection
- JavaScript rendering using headless Chrome/Chromium
- Concurrent request management with rate limiting
- Custom JavaScript execution on target pages
- Device emulation (desktop, mobile, tablet)
- Account usage monitoring
- Content sandboxing option - Wraps scraped content with security boundaries to help protect against prompt injection
Installation
Running with npx
Manual Installation
Configuring in Cursor
Note: Requires Cursor version 0.45.6+
The WebScraping.AI MCP server can be configured in two ways in Cursor:
1. **Project-specific Configuration** (recommended for team projects):
Create a .cursor/mcp.json file in your project directory:
2. **Global Configuration** (for personal use across all projects):
Create a ~/.cursor/mcp.json file in your home directory with the same configuration format as above.
> If you are using Windows and are running into issues, try using cmd /c "set WEBSCRAPING_AI_API_KEY=your-api-key && npx -y webscraping-ai-mcp" as the command.
This configuration will make the WebScraping.AI tools available to Cursor's AI agent automatically when relevant for web scraping tasks.
Running on Claude Desktop
Add this to your claude_desktop_config.json:
Configuration
Environment Variables
Required
- WEBSCRAPING_AI_API_KEY: Your WebScraping.AI API key
- Required for all operations
- Get your API key from WebScraping.AI
Optional Configuration
- WEBSCRAPING_AI_CONCURRENCY_LIMIT: Maximum number of concurrent requests (default: 5)
- WEBSCRAPING_AI_DEFAULT_PROXY_TYPE: Type of proxy to use (default: residential)
- WEBSCRAPING_AI_DEFAULT_JS_RENDERING: Enable/disable JavaScript rendering (default: true)
- WEBSCRAPING_AI_DEFAULT_TIMEOUT: Maximum web page retrieval time in ms (default: 15000, max: 30000)
- WEBSCRAPING_AI_DEFAULT_JS_TIMEOUT: Maximum JavaScript rendering time in ms (default: 2000)
Security Configuration
**Content Sandboxing** - Protect against indirect prompt injection attacks by wrapping scraped content with clear security boundaries.
- WEBSCRAPING_AI_ENABLE_CONTENT_SANDBOXING: Enable/disable content sandboxing (default: false)
- true: Wraps all scraped content (and search results) with security boundaries
- false: No sandboxing
When enabled, content is wrapped like this:
This helps modern LLMs understand that the content is external and should not be treated as system instructions.
Configuration Examples
For standard usage:
Tools
1. Question Tool (webscraping_ai_question)
Ask questions about web page content.
Example response:
2. Fields Tool (webscraping_ai_fields)
Extract structured data from web pages based on instructions.
Example response:
3. HTML Tool (webscraping_ai_html)
Get the full HTML of a web page with JavaScript rendering.