Shweta-Mishra-ai/tokenmizer

Your AI forgets why. TokenMizer remembers.
An OpenAI-compatible proxy that builds a knowledge graph of your
session — decisions, files, errors, goals — and replays it when the
context window runs out. Not a summary: a queryable graph that knows
"we switched from MongoDB to PostgreSQL, and here is why."
One line to adopt · works with Claude, GPT, Gemini, Grok, DeepSeek, Mistral, Cohere, Ollama · MIT
Quick start ·
See it ·
Claude Code & MCP ·
Benchmarks ·
Architecture ·
Roadmap
Real run: 25-node graph, checkpoint ckpt_21a0959c3ddf , 233-token resume. Regenerate with python scripts/gen_demo_gif.py .
---
The problem
Every AI session has a context limit. When you hit it, the model forgets
every decision and every rationale built over hours of work, and you
spend the first ten minutes of the next session re-explaining the
project.
Summarising the history does not fix this. A summary tells you *what*
was decided; it loses *why*, and it loses what was rejected — so the
model happily re-proposes the thing you moved off three sessions ago.
Quick start
Then change one line in your client:
Everything else is unchanged: same request shape, same response shape,
plus a tokenmizer block reporting what was saved. Open
and the session is already there.
Windows, Ollama, Docker, and the full step-by-step
**Windows (PowerShell)**
**No API key?** Ollama runs locally and free:
**Docker**
Full installation notes, every provider's environment variable, and the
configuration reference are in
**docs/configuration.md** and
**docs/deployment.md**.
See it
Nothing here is a mockup. Every screenshot and the demo below are the
shipped UI rendering a session from the labelled corpus in
benchmarks/eval/corpus.
The session graph is a page you drive, not a picture
Radial → select a node → force → filter a type → timeline → light. Regenerate with python scripts/gen_graph_demo.py .
Select a node and you get its type, status, community, importance,
confidence, when it was first seen and every relation it carries — the
provenance behind a fact, not just the fact. Filter a type or a community
and the counts move with it. Three layouts, a light theme and a PNG
export are one click each.
It is **one self-contained HTML file with no external requests**, so it
opens offline, works from a file:// URL, and can be sent to someone who
has never installed TokenMizer.
The dashboard tells you what it actually knows
Your sessions, the live resume block each one would inject right now, and
its graph — not an example of one. The health pill reads /health, which
reports degraded with the counters behind it when a write has failed,
rather than saying ok whatever happened.
The session graph, grouped the way you would group it
Each node type gets its own arc of the circle, named on the ring, and
relations are drawn as chords bowed through the middle — so the shape of
the session is readable before you read a single label, and which type a
node is comes from *where it sits*, not from telling two hues apart. The
palette is checked with a validator, not by eye: every adjacent pair
clears the colour-blind separation floor, and the four types that carry
no meaning of their own share one neutral grey.
The panel beside it counts what the session knows and what is missing —
open issues, decisions changed, history gaps, unconnected nodes — then
lists the detected communities, the hotspots everything hangs off, and
which kinds of node point at which. Filter by type or community, search,
click a node for the supersession chain behind it.
The same session as a story
Timeline mode puts each node type in its own lane, ordered by when the
fact entered the session, with supersessions drawn as arcs. When a whole
transcript was checkpointed in one call every node shares a timestamp, so
the axis says so and falls back to the order the session stated things
rather than inventing dates.
How it works
TokenMizer is a local proxy between your app and any LLM. Every request
passes through a pipeline that builds a live knowledge graph, compresses
inputs, caches responses, and checkpoints before the context runs out.
The graph is not a summary. It is typed nodes and edges — decisions,
tasks, files, errors, goals — with a lifecycle, so a decision that gets