Jwrede/tokentoll

> Prevent LLM cost regressions before production.
tokentoll is a CI gate for LLM cost. It statically analyzes Python, JavaScript, and TypeScript for LLM API calls, scores every pull request against a policy you control, and posts a PASS/WARN/FAIL verdict directly on the PR. Optionally, it fails the workflow when the policy is violated, so cost regressions cannot be merged.
## Live demo
Jwrede/tokentoll-demo is a small polyglot LLM app (Python + TypeScript) wired up to the tokentoll cost gate. Two PRs are already open against it:
PR #1: Add Anthropic Haiku translation helper. New call site, well within budget. Verdict: PASS, workflow green.
PR #2: switch supportbot to gpt-4o. A model swap that trips two policy rules. Verdict: FAIL, workflow red.
Open each PR's conversation tab to see the verdict comment tokentoll actually posts.
## The verdict comment
When a PR violates your policy, tokentoll comments with a verdict and a blocking-findings list, then exits non-zero so the check fails. Example:
md
## tokentoll verdict: FAIL
Blocking findings (2):
src/agent.py:42 - per-call cost grew 15.0x (threshold 5x)
total monthly delta +$812.00 exceeds budget $250.00
> Required action: revert the regression, raise the threshold in .tokentoll.yml, or add an exemption.
When the PR is clean, the verdict is PASS and the comment shows only the cost delta table. When no policy is configured, tokentoll posts an informational delta comment with no verdict.
## Quick start (60 seconds)
Add .github/workflows/tokentoll.yml:
yaml
name: tokentoll
on:
pullrequest:
paths:
".py"
".ts"
".tsx"
".js"
".jsx"
permissions:
contents: read
pull-requests: write
jobs:
cost-gate:
runs-on: ubuntu-latest
steps:
uses: actions/checkout@v4
with:
fetch-depth: 0
uses: Jwrede/tokentoll@v0.7.0
with:
fail-on-policy-violation: true
Then add .tokentoll.yml to your repo root:
yaml
budgets:
maxmonthlydeltausd: 250
maxcallsitemonthlyusd: 100
maxrelativeincrease: 5.0
policies:
blockunknownmodels: true
failonpolicyviolation: true
Future PRs receive a verdict comment. PRs that exceed the thresholds fail the workflow.
For SHA-pinned installs and minimal-permissions setups, see docs/github-action.md. For the full policy schema, see docs/policy.md. For the security posture, see docs/security.md.
## What it detects
Python
| SDK | Patterns |
| OpenAI | chat.completions.create, responses.create |
| Anthropic | messages.create, messages.stream |
| Google GenAI | models.generatecontent |
| LiteLLM | completion, acompletion |
| LangChain | ChatOpenAI, ChatAnthropic, initchatmodel |
| Zhipu AI | ZhipuAiClient, ZhipuAI (GLM models) |
JavaScript / TypeScript (parsed via tree-sitter, handles .js, .jsx, .ts, .tsx)
| SDK | Patterns |
| OpenAI Node SDK | client.chat.completions.create, client.responses.create, client.embeddings.create |
| Anthropic SDK | client.messages.create, client.messages.stream |
| Vercel AI SDK | generateText, streamText, generateObject, streamObject, embed, embedMany |
| LangChain.js | new ChatOpenAI, new ChatAnthropic, new ChatGoogleGenerativeAI, ... |
| OpenAI-compatible | same shape as OpenAI Node SDK, picked up automatically |
## Policy rules
The policy block in .tokentoll.yml controls when a PR fails: