AntonG87/codearia-sieve

Parses a web page into the exact format a decision model needs.
Dates as dates. Numbers with units. Text in chunks that fit the model's window, each pointing back to where it came from.
One call — and the token bill, before and after.
Measured, not promised. A live run over random pages picked the same day — fresh news from RSS feeds in several languages, random Wikipedia articles, docs, blogs, government sites, recipes, shops. Every row is in bench/analytics/ , and npm run analytics reruns the whole thing. The output was then put in front of a decision model: see Checked by a judge .
What it does
An agent that needs a web page fetches the whole thing: navigation, cookie banner, footer, ad slots, a megabyte of framework markup. Then a model paid per token digs through the pile for one paragraph.
**codearia-sieve does the digging before the model sees anything** — and returns the page as *state*, not prose:
**Dates become dates**
"Published September 15, 2026" → "2026-09-15".
Read from JSON-LD, meta tags and first; from a byline only when the markup is silent, and never guessed.
**Numbers become facts**
"$42 per billion tokens" → { value: 42, unit: "USD_per_billion" }.
Works in nine languages; the decimal comma follows the page's language. A number without a unit is not a fact. A year is never a fact.
**Text becomes chunks that fit**
Each chunk knows its size in tokens and characters, the blocks it was built from, and the #anchor on the page where a decision can be checked.
Everything else — menus, footers, banners, tag rows, "read more" — is removed, and with trace: true you get the list of what was removed and why.
Who it is for
**People who build agents and have seen the bill.** Every fetched page costs tens of thousands of tokens before the agent has read a word of it. Median page in the sample: 53 718 tokens in, 1 106 out.
**People who run cheap decision models.** Classifiers, rankers, System One models like Jev that judge instead of write. They are nearly free and very fast, and they have hard edges: they cannot count, they read dates as text, and their accuracy drops as irrelevant material fills the context. Every "page to markdown" tool prepares input for a reader. This one prepares input for a judge.
**People who need answers they can check.** A verdict from scraped text is unprovable unless each piece points back to its source. Here every fact names its block and every chunk carries an anchor.
How it works
Eight steps of ordinary code. No model runs unless you plug one in. The same HTML gives the same JSON, byte for byte.
1. **Fetch** — robots.txt first; a refusal is reported, not bypassed. Plain HTTP, honest user agent.
2. **Parse** — HTML into a DOM with linkedom. No browser.
3. **Dates and ids from the untouched tree** — cleaning strips , bylines and attributes, so both are read before it runs.
4. **Clean** — Defuddle removes chrome; Readability takes over if it comes back empty. Inline elements get a space first, so 20 Sept 10 min never becomes 202610 min.
5. **Blocks** — headings, paragraphs, lists, tables, code, quotes, in order, with positional ids. Old pages set with become paragraphs too; table rows keep their column headers.
6. **Facts and anchors** — ids restored; numbers become facts only beside a unit; ranges keep both ends.
7. **Chunk** — greedy, in order, under two budgets at once: 20 000 tokens and 50 000 characters by default.
8. **Assemble** — state, markdown, usage, warnings, and the trace on request.