jackjin1997/markindex

Vectorless, structure-first retrieval over large libraries of long Markdown documents — for agents.
MarkIndex parses every Markdown file into a tree of its own sections, stores the trees, the original lines and a
mixed Chinese/English BM25 index in one SQLite file, and gives an agent five read-only tools to walk
"library → folder → document → section" and read the original text with line numbers. No embeddings, no vector
database, no chunking guesswork: the document's own structure is the index, and every piece of evidence can be
cited back as [docpath, L - ].
> Inspired by VectifyAI/PageIndex, MarkIndex is an independent,
> unofficial TypeScript implementation focused on Markdown. It is not affiliated with or endorsed by VectifyAI.
中文说明：README.zh-CN.md
## Why
Structure-first, not chunk-first. Long professional documents (filings, guidelines, laws, manuals) are
already organized by their authors. Navigating that outline keeps context intact and makes answers explainable
and citable; arbitrary chunks + vector similarity cut sections apart.
Deterministic and LLM-free indexing. Parsing (mdast, not regex), tree building and the token-cost tree
optimization never call a model; the same input always produces the same tree. LLMs are optional and only used
for node summaries, document descriptions and the built-in retrieve() loop.
Markdown done properly. Setext headings, preface text before the first heading, documents without any
heading, front matter, # inside code fences, skipped heading levels, BOM / CRLF — every source line belongs to
exactly one node, and oversized sections are split at block boundaries into chunk nodes.
Keyword fallback that works for Chinese. BM25 over Intl.Segmenter word segmentation (Chinese, English and
mixed text, identifiers like C++, ISO-27001, v2.1), plus a trigram substring index as a second pass.
Budgeted tool responses. Tools never throw; they return JSON envelopes with nextsteps guidance and stay
under a 95 000-character budget, paginating large outlines and sections.
## Install
bash
npm install markindex # library + CLI (Node.js >= 22.12)
npx markindex --help
From source:
bash
git clone https://github.com/jackjin1997/markindex.git && cd markindex
npm install && npm run build
node dist/cli.js --help # or npm link to get a markindex command
Optional, only for LLM features (summaries, retrieve(), agent-mode evaluation): install the
AI SDK provider you want, e.g. npm install @ai-sdk/anthropic, and set its API key
(ANTHROPICAPIKEY, …). With npx and no local install, put the provider in the same temporary environment:
bash
npx -p markindex -p @ai-sdk/anthropic markindex index ./docs --summary-model anthropic:claude-sonnet-5-5
## CLI
bash
# Build or incrementally update the index (stored in /.markindex/index.db)
markindex index ./docs
markindex index ./docs --summary-model anthropic:claude-sonnet-5-5 --estimate # cost estimate, no LLM calls
markindex index ./docs --summary-model anthropic:claude-sonnet-5-5 # with node summaries
markindex index ./docs --summary-model anthropic:claude-sonnet-5-5 --summary-rpm 50 --summary-tpm 40000 # rate-limited
markindex index ./docs --json # machine-readable report