LLMs know accounting formulas. FinIndices shows they can't apply them

Remove the formula sheet and the model stumbles. That is the bluntest way to read FinIndices, a new benchmark built from the actual financial statements of Chinese listed companies, with uncropped reports running up to 32,000 tokens. Given explicit formulas, strong LLMs handle single-indicator tasks reasonably well. Take the hints away and scores collapse. Gemini-3.1-Pro fell from 70.70% to 38.22% on table tasks, a swing the paper's authors attribute to "fragile pattern matching" rather than real understanding of accounting structure.

FinIndices, released on Hugging Face, is designed to test what existing financial benchmarks dodge: whether models can process real statements end to end instead of answering multiple choice questions over cropped tables. It comes in two task types. Single-Index asks for one financial indicator computed from long, distractor-rich statements. Table-Index asks for multiple indicators across multiple periods, delivered as structured HTML, JSON, or a table. The second task exists because real financial workflows end in comparison tables, not single numbers.

The knowledge bottleneck: memorized, not understood

The paper's first finding, the Knowledge Bottleneck, is that LLMs have absorbed accounting formulas during pretraining without internalizing when to use them. The giveaway is where failures appear: distinguishing interest-bearing from non-interest-bearing liabilities when computing invested capital, separating equity attributable to the parent company from total shareholders' equity when minority interests matter, and matching income-statement flow variables with average balance-sheet stock variables. Chinese Accounting Standards (CAS) add their own traps, with line items like credit impairment losses and non-operating gains or losses that models must map onto the right indicators.

Formulas in FinIndices are curated from authoritative sources, including CAS, financial statement analysis textbooks, and corporate finance references, and the test set was manually audited by domain experts for accounting validity and numerical correctness. With hints removed, temporal de-cumulation goes wrong and stock-flow caliber mismatches appear. The 32-point drop on table tasks is the visible symptom; the authors read it as evidence that benchmark-level financial competence is often shallow pattern matching, the same charge leveled at HumanEval, the coding test that never asked whether the code was real work.

The structural bottleneck: tables drain reasoning

The second finding is more surprising. Models that execute isolated derivations flawlessly regress to shallow heuristics when asked to fill multi-metric, multi-period tables. The paper calls this the Structural Bottleneck: output structure is a genuine load on reasoning capacity, not a formatting afterthought. The same split shows up in agent research, where PawBench's results point to the harness around the model rather than the brain.

Under that pressure, models fetch incorrect adjacent columns, substitute lazy literal arithmetic for deeper accounting adjustments, mix up reporting periods, and misuse beginning balances. The denser the output, the worse the thinking. For teams building tools that turn statements into audit-ready tables, that is the finding that matters most.

Hallucination is the default failure mode

FinIndices also includes adversarial cases where the required reports are missing entirely. The correct answer is "insufficient information," not a number. The failure mode the benchmark is designed to catch is the confident figure hallucinated anyway. For a financial agent, knowing when to say "I can't tell" is the whole job.

The same pattern shows up beyond finance

Finance is an unforgiving testbed, but not the only domain where this divide appears. SDABench, a capability-oriented benchmark spanning six scientific reasoning skills, found models strong on descriptive analysis but collapsing on inferential and causal tasks, a gap documented in detail in SDABench's findings. Even the integrity of older reasoning tests has come under scrutiny: audits of GSM8K, the grade-school math set, found error rates in its questions as high as 42%, per the audit behind that number.

High scores on controlled questions overstate what models can do in the wild, the same dynamic that hollowed out MMLU, where models eventually started acing the answer key, as the story of MMLU's decline recounts. FinIndices is engineered to be unkind to that overstatement.

Fine-tuning can restore some structure

The paper offers some good news for model builders. Supervised fine-tuning produced substantial zero-hint gains: +8.54% on Single-Index and +3.82% on Table-Index. Structured logic can be partially restored through data-centric alignment, even if the question the authors open with, whether LLMs possess genuine structural reasoning or only surface-level pattern matching, stays open.

A model that only performs with the formula sheet in front of it has not learned accounting; it has learned to recognize it. For teams shipping financial agents, that distinction is not academic.