YOLOW92/pagesift

High-performance, memory-bounded web scraping and extraction engine built for AI agents and LLM pipelines.
Turn any website, document, or archive into clean Markdown and structured data — one Rust binary, native SDKs, dual Firecrawl v1/v2 & Jina Reader APIs, at a fraction of the memory.
English · 简体中文 · 日本語 · Español
## Performance at a glance (0.81)
PageSift 0.81 was re-measured on 2026-10-29; competitor numbers are the recorded 2026-10-21/22/27/28 data and were not re-run. One row per workload, best PageSift configuration; memory is a column, not a separate rank.
### Far ahead
| Domain | Workload | Metric | PageSift 0.81 | Fastest rival | Rival | Gap | Peak MB/worker (PageSift / rival) |
| PDF → Markdown | olmOCR-bench 1,269 pages, per page (single thread) | ms/page | 2.2 | pypdf | 50.0 | ×22.32 | — |
| HTML → Markdown | 600 generated documents, per page | ms/page | 0.24 | Turndown | 1.9 | ×8.24 | — |
| Main content | Firecrawl 597 pages, single thread | p50 ms/page | 1.0 | crw | 7.4 | ×7.23 | — |
| Publication dates | htmldate 1,000 pages, date search | ms/page | 0.31 | htmldate (fast) | 2.0 | ×6.45 | — |
| Office / e-books | B4 81 files, in-process | files/s | 94.1 | Kreuzberg | 19.5 | ×4.82 | 38 / 231 |
| RAG chunking | 5 corpora, 1.4 MB | total ms | 11 | chonkie recursive | 35 | ×3.24 | — |
| PDF → Markdown | B1 144 pages, 1 worker | files/s | 258.5 | pdfoxide | 100.6 | ×2.57 | 163 / 29 |
| RAG query filter | 472 questions, default settings | answer recall | 0.929 | Crawl4AI BM25 | 0.673 | +0.256 | — |
| Wild crawl | polite, 40 sites × 60 pages, coverage | coverage | 0.740 | Crawl4AI (HTTP) | 0.529 | +0.211 | — |
| Wild crawl | fast, 40 sites × 60 pages, coverage | coverage | 0.685 | Crawl4AI (HTTP) | 0.499 | +0.186 | — |
| Office / e-books | B4 hold-out 39 files, structure kept | structure kept | 0.988 | Kreuzberg | 0.816 | +0.172 | — |
| PDF → Markdown | olmOCR-bench 1,269 pages | official score | 0.628 | pymupdf4llm | 0.484 | +0.144 | — |
### Ahead
| Domain | Workload | Metric | PageSift 0.81 | Fastest rival | Rival | Gap | Peak MB/worker (PageSift / rival) |
| PDF → Markdown | B1 144 pages, 16 workers | files/s | 580.6 | pdfoxide | 501.7 | ×1.16 | 14 / 11 |
| Main content | A2 269 random pages, 16 workers | pages/s | 630.0 | Resiliparse | 559.3 | ×1.13 | 17 / 54 |
| HTML → Markdown | 600 generated documents, fidelity | mean F1 | 1.000 | markdownify | 0.936 | +0.064 | — |
| Main content | WCXB test split, 150 pages | F1 | 0.866 | Readability + Turndown | 0.848 | +0.018 | — |
A2 was re-fetched on 2026-10-29; its absolute times are not directly comparable with the 10-28 A2.
Full report
## How PageSift compares
Still behind: A2 replay, 269 pages, 1 worker 3.1 pages/s in the 0.81.2 local replay (2.4 in the 0.81 report) vs spider 156.6 pages/s; fast, 40 sites × 60 pages, time 16.6 seconds vs spider 1.4 seconds; B5 134 pages, completion 0.112 vs Docling (GPU) 1.000.
Full report.
## Scale: 200,000,000 requests on one machine (0.70)
🏁 200M saturation on 192 vCPU (synthetic local origin, no network — not a real-internet speed) — 200,000,000 / 200,000,000, 0 failures, 947,611 pages/s, 211.1 s wall, 888.1 MiB peak RSS. Source: 0.70200mregression.md.