Apexflowlabsllc/apex-flow-web-scraper

An owned web extraction engine from Apex Flow Labs.
Collect public pages, keep extraction evidence, and work with durable jobs from
a CLI, a private local workspace or a compatible MCP assistant.
Release 0.2.0. No hosted scraping API or model key is required.
Software is MIT licensed; compute, bandwidth, storage and maintenance are not free.
## Hosted API for agents
Start a free browser trial for 1,000 credits. A Project Pass is $5 once for 5,000 credits valid for 30 days. Starter is $15/month for 25,000 credits. A successful plain page costs one credit and a rendered page costs three.
Connect a Streamable HTTP MCP client to https://webscraper.apexflowlabs.com/v1/mcp with a bearer API key. The six tools support page extraction, parallel batches, change checks, sitemap mapping, account-scoped search and credit balance. Local stdio clients can run the apex-scraper-mcp command with APEX_SCRAPER_API_KEY in their secret environment.
Agent setup · OpenAPI 3.1 · Benchmark and pricing comparison
## Quick start
Requires Python 3.11+. Tested on Windows and Linux. Install Brave separately if
you want JavaScript rendering; HTTP-only collection does not need a browser.
sh
git clone https://github.com/Apexflowlabsllc/apex-flow-web-scraper.git
cd apex-flow-web-scraper
python -m venv .venv
Activate with .venv\Scripts\Activate.ps1 in Windows PowerShell or
source .venv/bin/activate on Linux, then:
sh
python -m pip install -r requirements.lock
python -m pip install --no-deps .
rainbo-scrape health
rainbo-scrape scrape https://example.com --render never --pages 1
rainbo-scrape status JOB_ID
rainbo-scrape results JOB_ID
Submission returns a job ID. The service can start a background worker; for an
explicitly managed process run rainbo-scrape worker. Keep one worker per data
directory. RAINBO_SCRAPE_DATA selects the data directory; the default is
~/.local/share/rainbo-scrape. RAINBO_BRAVE_PATH can select an installed Brave
executable if the default locations are not suitable.
An operator may configure an HTTP proxy for public HTTP fetches and
guarded browser subresources with RAINBO_SCRAPE_PROXY_URL, for example
http://proxy.example:3128. Keep credentials
in the process environment; never put them in job arguments, recipes or source
control. The engine independently validates destination addresses before
forwarding. This routes through a proxy you control; it does not provide a proxy
fleet, residential IPs, geography guarantees, or access to protected sites.
Health reports only whether a proxy is configured.
## Private workspace
Run rainbo-scrape web --port 8840, then open http://127.0.0.1:8840/ on that
machine. Jobs, recipes, evidence, search, comparisons and exports stay in the
installation's data directory. The web interface binds to loopback and is not
a public multi-tenant hosting service. It has Host/Origin checks, not an account
and authorization system for internet exposure.
## Assistant connection (MCP)
Configure your client with an absolute path to the installed command:
json
"mcpServers": {
"apex-web-scraper": {
"command": "/absolute/path/to/.venv/bin/rainbo-scrape",
"args": ["mcp"],
"env": {"RAINBO_SCRAPE_DATA": "/absolute/path/to/private-data"}