🐍 🏠 🍎 🪟 🐧 - Extract clean web content for RAG and answer questions about web pages
🔍 Web Analyzer MCP
以下摘录自该仓库的 README(片段) · 在 GitHub 读完整 README →
A powerful MCP (Model Context Protocol) server for intelligent web content analysis and summarization. Built with FastMCP, this server provides smart web scraping, content extraction, and AI-powered question-answering capabilities.
✨ Features
🎯 Core Tools
1. **url_to_markdown** - Extract and summarize key web page content
- Analyzes content importance using custom algorithms
- Removes ads, navigation, and irrelevant content
- Keeps only essential information (tables, images, key text)
- Outputs structured markdown optimized for analysis
2. **web_content_qna** - AI-powered Q&A about web content
- Extracts relevant content sections from web pages
- Uses intelligent chunking and relevance matching
- Answers questions using OpenAI GPT models
🚀 Key Features
- **Smart Content Ranking**: Algorithm-based content importance scoring
- **Essential Content Only**: Removes clutter, keeps what matters
- **Multi-IDE Support**: Works with Claude Desktop, Cursor, VS Code, PyCharm
- **Flexible Models**: Choose from GPT-3.5, GPT-4, GPT-4 Turbo, or GPT-5
📦 Installation
Prerequisites
- uv (Python package manager)
- Chrome/Chromium browser (for Selenium)
- OpenAI API key (for Q&A functionality)
🚀 Quick Start with uv (Recommended)
Installing via Smithery
To install web-analyzer-mcp for Claude Desktop automatically via Smithery:
IDE/Editor Integration
Install Claude Desktop
Add to your Claude Desktop_config.json file. See Claude Desktop MCP documentation for more details.
Install Claude Code (VS Code Extension)
Add the server using Claude Code CLI:
Install Cursor IDE
Add to your Cursor settings (File > Preferences > Settings > Extensions > MCP):
Install JetBrains AI Assistant
See JetBrains AI Assistant Documentation for more details.
1. In JetBrains IDEs go to **Settings** → **Tools** → **AI Assistant** → **Model Context Protocol (MCP)**
2. Click **+ Add**
3. Click on **Command** in the top-left corner of the dialog and select the **As JSON** option from the list
4. Add this configuration and click **OK**:
🎛️ Tool Descriptions
url_to_markdown
Converts web pages to clean markdown format with essential content extraction.
**Parameters:**
- url (string): The web page URL to analyze
**Returns:** Clean markdown content with structured data preservation
web_content_qna
Answers questions about web page content using intelligent content analysis.
**Parameters:**
- url (string): The web page URL to analyze
- question (string): Question about the page content
**Returns:** AI-generated answer based on page content
🏗️ Architecture
Content Extraction Pipeline
1. **URL Validation** - Ensures proper URL format
2. **HTML Fetching** - Uses Selenium for dynamic content
3. **Content Parsing** - BeautifulSoup for HTML processing
4. **Element Scoring** - Custom algorithm ranks content importance
5. **Content Filtering** - Removes duplicates and low-value content
6. **Markdown Conversion** - Structured output generation
Q&A Processing Pipeline