sireggserroneous/scout-mcp

Scout is an MCP server for web crawling that improves with use. A capable model like Claude works a site out once.
Scout compiles what it learned into a **recipe** that a 4B model can run with one instruction.
Claude reaches the page, maps the site and finds the traps. It then hands back a one-line card that Qwen3.5-4B runs
correctly from then on:
Install
**Claude Code** (installs the MCP server and the /scout skill):
**Any MCP client.** Install uv, then add this to the client's MCP config:
**Pages built by JavaScript** need a headless browser. Add "--with", "playwright" to the uvx args, then run
uvx playwright install chromium once. If you already run Firecrawl or crawl4ai, set FIRECRAWL_URL or
CRAWL4AI_URL and Scout uses it as one more reader.
From a maze to one line
Getting MikroTik specs looks like a one-step job. Here is what Claude actually ran into on the way:
| Try | Result |
|---|---|
| guess mikrotik.com/product/{model} | RB433AH works. hAP ac3 is a 404: that page lives at hap_ac3. |
| lowercase it, turn spaces into _ | Now RB433AH is a 404: that one really is uppercase. |
| also turn + into plus | CRS354-48G-4S+2Q+RM works (crs354_48g_4splus2qplusrm)… |
| | …but RB5009UG+S+IN lives at rb5009ug_s_in. Same character, opposite rule. |
| **look the url up in the sitemap, comparing both sides with plus and punctuation removed** | **all four styles work** |
Then the data: 4 fields, each a label on one line and its value on a later line, on a page that also has a 40-row
throughput table, a documents list and a retailer map.
Written out as instructions, that is a 7-step program with a string-normalisation rule, a loop over 566 urls,
a branch for "not found", and four lookups in a 7,000-character page. That is too much for a 4B model. So it is
never handed to one. Claude compiles it instead:
compile runs the recipe on every example and **saves it only if all of them pass**. The two wrong guesses above
were caught this way: each one passed on some examples and failed on others. Once saved, every branch lives in code.
The small model gets one card and one tool, and its whole job is a single call:
| | the small model gets | the small model must |
|---|---|---|
| without a recipe | 7 numbered steps, site_map and reach | normalise strings, scan 566 urls, branch, read a 7k-char page, format a line |
| with a recipe | 1 line, the run tool | make one call, copy one line |
**Measured on Qwen3.5-4B** (Q4_K_M, llama.cpp on a laptop iGPU), on 5 MikroTik models, as an A/B:
| | exact answer | tool calls | wall time (median) |
|---|---|---|---|
| **A**: the 7 steps as instructions, site_map + reach | 4 / 5 | 2–4 per model, 15 in all (5 of them wrong turns) | 389–671 s (414 s) |
| **B**: the card, run | **5 / 5** | **1 per model** | **22–195 s (88 s)** |
In arm A, Qwen filtered the sitemap with the raw model name, built a url with a space in it, and once dropped the $
from a price. It reached the right page in the end, but it took 3 to 11 minutes each time, mostly spent reading 566
urls and a 7,000-character page. In arm B the model never sees any of that. One arm-A run did catch a bug in the
recipe: the RAM regex had matched a sentence in the product description, not the spec list. The fix (anchoring each
field to its list item) went in, the recipe was recompiled with that model as a fifth example, and the arm-B run was
repeated on the fixed recipe. Both arms used the same model and the same Scout. Harness: bench/qwen_ab.py.
If a page changes, run fails loudly with a code: NO_MATCH at step 1, EXTRACT_MISS at step 3. The recipe's
score drops, and recipes() marks it last run failed … recompile. The next big-model session fixes the recipe. The
small model's card stays the same.
The CLI form fits a small model with a shell. It prints one line and exits 1 on failure:
Recipe steps
| Step | Does |
|---|---|
| {"input": [[regex, repl], ...], "lower": true} | rewrites the input (slug rules) |
| {"map": url, "filter": regex, "same": [[regex, repl], ...]} | the site's own urls. same keeps the url whose last segment equals the input after both are rewritten. Last step: returns the list. Otherwise: the first url becomes {url}. |
| {"reach": "https://…/{input}" \| "{url}", "want": regex} | reads a page with the full reader ladder (below) |
| {"find": regex} | the first link on the page whose url or text matches becomes {url} |
| {"links": "/chapter/ /"} | a link family from families. Last step: every matching link on the page. Otherwise: the first becomes {url}. |