MonedaIO/benchmarks

By Moneda · Leaderboards: benchmarks.finance · Datasets: FilingsBench · PortfolioBench · CiteBench
Benchmarks for AI systems that answer questions about finance: reading company
filings, doing portfolio arithmetic, and citing sources that actually say what
the answer claims. Leaderboards, method and changelog are at
benchmarks.finance.
This repository holds what you need to rerun the public part of each benchmark
yourself: the harness, the graders, the judge prompts, the item format, each
benchmark's public sample, and the scripts that turn a graded run into a
leaderboard row.
> Provenance. Moneda Bench is created and maintained by Moneda, which also
> builds Numa 1, one of the systems it scores. Every system is run on the same
> items, the same window, the same grader and default settings, and the public
> splits can be rerun by anyone. Moneda is not a regulated firm and does not
> manage portfolios. A score does not certify compliance with any rule, and no
> result is investment advice or investment performance.
## The benchmarks
| Benchmark | Folder | What it tests | Public sample | Graded by | Leaderboard |
| Moneda FilingsBench | filingsbench/ | Can a system find the right SEC filing and period with no document attached, and read the figure correctly? | 44 questions | code against SEC XBRL as first filed; two judges for segment questions | link |
| Moneda PortfolioBench | portfoliobench/ | Portfolio decisions on hypothetical portfolios: rebalancing, volatility, drawdown, VaR, correlation, position sizing, constraints, fund choice | 16 tasks (Part A) | code | link |
| Moneda CiteBench | citebench/ | In a research note, does each linked source say what the claim next to it says? | 11 companies | code plus a two-judge panel per claim, human audit | link |
Coming later, each with its own folder when its first public sample is ready:
Moneda GroundBench (made-up live market figures), Moneda MarketBench
(market-level questions), Moneda KnowledgeBench (finance knowledge) and
Moneda AdviceBench (personal directives without over-refusing education).
In code each benchmark keeps a short key: filings, portfolio, and
verifiability for CiteBench.
## External track: other people's benchmarks, our grading
benchmarks.finance also shows scores Moneda ran on public benchmarks it did
not write, where the grading is Moneda's own rather than the authors'. Their
runners are in external/ so anyone can reproduce those scores.
Each folder holds an adapter that downloads the original data from its
official source at run time (pinned revision, cached outside the repo,
$MONEDABENCHDATADIR), our judge prompts and grading code, the settings of
the published runs, and a README on how the grading differs. None of their
items, answers or per-question files are in this repository, and
scripts/check-public.mjs fails if one is committed.
| Benchmark | Key | Data licence | What is ours | Comparable with the authors' board? |
| FinanceBench (Patronus AI, open sample) | financebench | CC BY-NC 4.0, non-commercial | no documents given; two LLM judges instead of human labels; split panel counts half | No |
| FinanceQA (AfterQuery) | financeqa | Apache-2.0 | code exact match, then two LLM judges instead of human annotators | No |