Public benchmark

Show me the receipts

These benchmarks compare how different engines handle the same byte-identical inputs with the same task text. They exist to show whether our platform earns its complexity — not to rank vendors.

950

matched pairs · precision 1.0000 · published case

12/12

of 12 leading AI models missed · aviation case

Every input published

Byte-identical across all arms

Exact prompt published

One file per arm, verbatim

Failures published

Ours included, with root cause

Published cases

with source bundles

Evaluation arms

frontier models compared

Tolerance

$0.01

money exact-match

Published cases

Aviation ground-handling billing

ComputationCONDITIONALcustomer-verified (May) · independently derived (Aug)Published 2026-08-07

Golden

May $6,422.28 · Aug $9,047.51

Byte-identical inputs across every arm, two months apart. Month 1 is a clean win for both our arms. Month 2 is not: both overstate the registered golden by $707 because an overtime threshold nobody has confirmed in writing is still in dispute with the client. Published as a miss rather than re-graded against the assumption that would make it a pass.

Why this is graded CONDITIONAL — Month 2 is exact only against an 8-hour overtime assumption no contract document states. Against the registered golden both our arms miss by ~$707, and neither has been re-run against it.

Engine	May 2026 real client data 39 timesheet rows	Aug 2026 independently generated 52 timesheet rows	Notes
FinAdvantage pipeline	$6,422.28 Exact — within tolerance. Δ $0.00	$9,754.51 Wrong, and confident. Δ +$707.00	Real 3-pass generation + LangGraph v2 execution. Month 1 exact to the cent. Month 2 over the registered golden by $707 — the disputed overtime threshold, documented and root-caused, not silently re-graded.
FinAdvantage agent runner	$6,422.28 Exact — within tolerance. Δ $0.00	$9,754.50 Wrong, and confident. Δ +$706.99	The fixed-flow path, same inputs. Tracks the pipeline to the cent on month 1 and to within $0.01 on month 2 — the one-cent gap is a rounding-convention difference between the two arms, itself an open item.
Raw frontier models (×3)	$12,299.64 Wrong, and confident. worst of 3 · best $6,145.44	$11,668.77 Wrong, and confident. worst of 3 · 1 no answer	Contract, rate card, and timesheet in one prompt, no tools. Wrong on every trial that produced an answer — including a month-1 result nearly double the golden. Worst figure shown; the best trial was still wrong.
Frontier models + our tools	$6,145.44 Wrong, and confident. 2 of 3 · 1 no answer	$8,686.90 Wrong, and confident. 2 of 3 · 1 no answer	The same three models given our tool set to call themselves. Closer than raw, still wrong on every trial that finished. 0 of 12 frontier outputs were exact across both months.

Replication bundle — Inputs, outputs, prompt, and trace published per arm.

Bundled with case

Reconciliation, invoice extraction, and standards Q&A

Reconciliation · Extraction · Q&AINDICATIVEsynthetic by construction · public standard (GAGAS 2024)Published 2026-08-26

Golden

950 matched pairs · ₹14,94,865 · GAGAS ¶ text

Three chat flows against industry-scale Indian fixtures. On a clean invoice all four engines are perfect — a genuine tie, and the honest reading is that the task is saturated. The separation is elsewhere: on a 1,030-row reconciliation, and on an invoice whose printed total is overstated by ₹1,00,000, where every model reported the wrong figure as fact and none noticed.

Why this is graded INDICATIVE — One trial per cell, and two of the three flows ran on a single slice. Enough to act on internally; not enough to publish as a rate.

Engine	Reconciliation HDFC statement vs Tally ledger, one quarter 1,030 bank + 1,005 ledger rows	Invoice — clean 2-page GST invoice, 3 slabs + a zero-rated line 46 line items	Invoice — tampered printed grand total inflated by ₹1,00,000 46 line items	Standards Q&A GAGAS 2024, answers must cite a paragraph 20,480-document corpus	Notes
FinAdvantage platform	950 / 950 pairs Exact — within tolerance. precision 1.0000 · recall 1.0000	10 / 10 · 46 / 46 Exact — within tolerance. 46 / 46 HSN codes	₹14,94,865 Exact — within tolerance. flagged and corrected	3 / 3 correct Exact — within tolerance. 3 / 3 cited a paragraph	Reconciliation runs with zero model calls in 0.33s, so it is reproducible by construction. On the tampered invoice, 11 validator checks caught that the printed total is not reproducible from the line items, corrected it, and dropped confidence to 0.90. Caveat we publish: 2 of the 3 Q&A answers contained a citation defect — a bad cross-reference and a quotation with words not in the standard. Checkable, but not clean.
openai/gpt-4o	— Result withheld — evidence file pending. evidence file pending	10 / 10 · 46 / 46 Exact — within tolerance. ties us · $0.0291	₹15,94,865 Wrong, and confident. 3 of 3 runs, unflagged	2 / 3 correct Wrong, and confident. 0 / 3 cited a paragraph	Perfect on the clean invoice — every field, every line, every leading-zero HSN code. Reported the ₹1,00,000 overstatement as fact on all three runs. On standards, stated flatly that no CPE grace period exists; ¶4.44 grants up to two months.
google/gemini-3.6-flash	— Result withheld — evidence file pending. evidence file pending	10 / 10 · 46 / 46 Exact — within tolerance. ties us · $0.0713	₹15,94,865 Wrong, and confident. 3 of 3 runs, unflagged	3 / 3 correct Exact — within tolerance. 0 / 3 cited a paragraph	Matched us on standards accuracy with no retrieval at all — worth stating plainly, because it means this set separates us from two of three models on correctness, not three. It cited nothing, which is the difference an auditor cares about. Still missed the tampered total on every run.
anthropic/claude-sonnet-5	— Result withheld — evidence file pending. evidence file pending	10 / 10 · 46 / 46 Exact — within tolerance. ties us · $0.04417	₹15,94,865 Wrong, and confident. 3 of 3 runs, unflagged	1 / 3 correct Wrong, and confident. 0 / 3 cited a paragraph	Invented a "rebuttable presumption" that lets units under common control be treated separately — ¶3.35 says the opposite. One further run outside the controlled set silently returned the corrected invoice total without saying it had substituted anything, which is arguably harder to rely on than being consistently faithful to the page.

Replication bundle — Extraction and Q&A trials are committed per-run. The reconciliation trials for the model arms were overwritten before commit and are being re-run — those cells stay blank until they land.

Bundle pending

Inventory roll-forward and COGS working

ComputationINDICATIVEindependently derived (pandas, shares no code with either arm)Published 2026-08-15

Golden

$591,323.61 closing stock

Four files, 22,101 rows, one month. Our pipeline reproduces the independently-derived closing figure exactly. Our agent runner does not — it silently reclassifies $496.01 from closing stock into COGS across 16 of 112 SKUs, which is published here as a defect rather than rounded into a 0.084% pass. Both frontier arms failed outright.

Why this is graded INDICATIVE — Single slice, n=1 per arm. The two-slice rule is unmet, which caps this regardless of how clean the pipeline result looks.

Engine	Apr 2026 monthly roll-forward, 112-SKU portfolio 22,101 rows across 4 files	Notes
FinAdvantage pipeline	$591,323.61 Exact — within tolerance. Δ $0.00	Exact on the headline closing figure and on 5 of 6 sub-totals. Published caveat: 11 of its 24 nodes replay SQL that was hand-verified against this file’s exact column names, so this proves the method is right once verified — not that the generator writes correct SQL unaided against a differently-shaped upload.
FinAdvantage agent runner	$590,827.60 Wrong, and confident. Δ −$496.01	Six of seven portfolio figures exact, then a clean reclassification error: 16 SKUs each show COGS over the golden by X and closing under by exactly X. Nothing was flagged or queried — the critical failure class, published as one. Root mechanism not yet isolated.
google/gemini-2.5-flash-lite	$163,868.83 Wrong, and confident. Δ −$407,457.70	All four files in one prompt, no tools. Returned an opening-stock figure as the answer, with 11 of 99 SKUs correct, and declared no uncertainty.
openai/gpt-4o-mini	— No answer — crash, timeout, or gave up. API error before any tokens billed	Raw model, same prompt. Failed at the API layer — recorded as infrastructure, not as a model miss.
Frontier model + SQL tool	— No answer — crash, timeout, or gave up. gave up after 7 turns	gemini-2.5-flash-lite given a generic DuckDB tool over the same tables. Ran the identical `SELECT * FROM roll_forward` seven times and stopped without producing an answer.

Replication bundle — The SKU fixture is not yet anonymised, so the input bundle is held back from download until it is. Totals, arms, and method are published in full.

Bundle pending

[DRAFT — illustrative figures] US multi-entity month-end close

ComputationINDICATIVE[placeholder — real golden in preparation]Published 2026-09-07

Golden

[~$2.4M consolidated revenue — illustrative]

[Placeholder.] The intended case consolidates three US entities — intercompany matching, elimination entries, and a consolidated close — with the same byte-identical discipline as the other cases. Real inputs, goldens, and arm results will replace every bracketed figure before launch.

Why this is graded INDICATIVE — [DRAFT] Every figure in this entry is an illustrative placeholder so the page structure is reviewable before the real US-books case lands. Nothing here is a measured result.

Engine	[Consolidated close] [placeholder] [~3 entities]	Notes
FinAdvantage pipeline	[exact — illustrative] Exact — within tolerance. [Δ $0.00 — illustrative]	[Placeholder result. Real measured result in preparation.]
Raw frontier models (×3)	[wrong — illustrative] Wrong, and confident. [placeholder]	[Placeholder result. Real measured result in preparation.]

Replication bundle — Placeholder entry — no bundle exists yet. The real input bundle ships with the measured case.

Bundle pending

Publishing soon — a new case is being prepared and will appear here.

How to read a result

Exact — within tolerance

Wrong, and confident

Result withheld — evidence file pending

No answer — crash, timeout, or gave up

“Wrong and confident” is tracked separately from “no answer” on purpose. An engine that declines or crashes has not given you a number to act on; an engine that is wrong without saying so has. Only the second one is dangerous, and it is the class we hold ourselves to as well — every one of our own misses on this page is in it.

How cases are scored

Each case runs identical inputs through every arm with the same task text. Scoring is automated exact-match against a golden that is never engine-derived — monetary totals to the cent, matched pairs and field strings exactly. Results are published whether they show us winning or losing, and each case carries the grade its sample size earns: a case run on one slice, or one trial per cell, is marked INDICATIVE and should not be read as a rate. Fixtures span US and Indian client books — real data from two geographies, with the currency stated on each case.

Tolerance: $0.01 on money, exact on pairs and fields·Golden: never engine-derived·Failures: published with root cause·Cells with no committed evidence file stay blank·Replication bundles: case 1 available now · cases 2–3 shipping this quarter

Need the replication bundle?

Case 1's inputs, outputs, prompts, and traces are downloadable today; cases 2–3 land this quarter. Request a bundle and we'll send it with the method note.

Request a bundle

Prove it on your books

What it costs vs. what it saves

Run your own numbers, not ours. Each workflow calculator is built from the inputs you enter — hours, rates, transaction volume — and shows the time and money saved on your books.

Bank Reconciliation

Compare the cost of the plan against the hours your team spends matching statements every month.

Try this calculator

Month-End Close

See the calendar-days and headcount-hours a faster close frees up for your team.

Try this calculator

Invoice Processing

Estimate the saving from handling AP invoices without manual data entry.

Try this calculator

Where does your finance team stand on AI readiness?

Take a free 3-5 minute assessment. Get a 0-100 readiness score, your top gaps, and a shareable PDF report — mapped to the workflows that close them.

Take the assessment Run the ROI calculators

Built for CPA firms and finance teams — pick your segment in the assessment's first question. Or see the benchmark first