raintree-technology/docpull

**Active open-source project · MIT License**
DocPull turns changing public web sources into cited, reproducible context for AI
agents and retrieval pipelines. Use it when your application needs to know which
sources it used, whether they changed, and how to rebuild the same context later.
Install and sync your first source
The project stores declared sources in docpull.yaml and resolved inputs in
.docpull/context.lock.json. Later syncs produce a hash-based diff while preserving
source URLs, content hashes, run IDs, citations, and export metadata.
docpull query retrieves deduplicated passages from locked archives with version
filters, a response budget, and exact source spans. The same query is available
through the Python SDK and MCP. See Query locked context
for budget semantics and queries across projects.
No account or paid API is required for this path. Direct fetching, discovery,
extraction, indexing, pack analysis, and diffs run locally.
Why use DocPull
- **Reproduce agent context.** Stable IDs, hashes, manifests, and lockfiles show which
source versions produced an answer or artifact.
- **Detect source drift.** Sync and diff documentation, product pages, policies,
feeds, repositories, packages, standards, and local documents.
- **Keep evidence inspectable.** Markdown, NDJSON, SQLite, citations, and provenance
sidecars remain readable without a hosted service.
- **Choose the downstream surface.** Export context for agent clients, vector import,
data workflows, or a versioned context-pack release.
- **Keep expensive routes explicit.** Browser and cloud rendering require an explicit
choice and can be blocked with a zero-dollar budget.
How it works
DocPull's v3 pack contract separates raw extraction, agent-ready context, and
eval-grade evidence. Validate the level a downstream system requires:
docpull ci checks freshness, citation coverage, pack quality, rights metadata, and
other configured gates. It writes context-ci.report.json and CONTEXT_CI.md, then
exits non-zero when a hard gate fails.
Supported surfaces
| Surface | Use it for | Start here |
| --- | --- | --- |
| CLI | Fetch, sync, diff, validate, and export | docs/cli-recipes.md |
| Python SDK | Embed acquisition in Python applications | docs/surface-contract.md |
| MCP server | Give local agent clients source tools | MCP server |
| TypeScript SDK | Read local packs and invoke the CLI from Node or Bun | sdk/js/README.md |
| Agent plugin | Install the supported MCP workflow in an agent client | plugin/README.md |
Common source shapes include static and server-rendered websites, OpenAPI documents,
feeds, papers, public GitHub repositories, npm and PyPI packages, standards, datasets,
transcripts, Wikimedia pages, product and policy pages, and local PDF or office files.
See context-pack workflows for the complete surface.
MCP server
Claude Code can register the same local server:
The Python stdio server is the supported MCP server.
Limits and security boundary
DocPull is an evidence-acquisition engine, not a hosted competitive-intelligence
product. It owns fetching, explicit rendering adapters, versioning, citations,