Mistral's OCR 4 scores big, but its own audit shows why benchmark numbers don't tell the real story

Document extraction is the quiet workhorse of enterprise AI pipelines: unglamorous, often outsourced to legacy vendors, but critical for any system that ingests invoices, contracts, or technical reports. Mistral AI's OCR 4, released today, brings the first substantial structural leap the field has seen in years. Alongside extracted text, the model returns bounding boxes for every block, typed classification (title, table, equation, signature, and more), and per-word confidence scores. The next trillion-dollar bottleneck in AI isn't…

The numbers are strong: 85.20 on OlmOCRBench, 93.07 on OmniDocBench, and a 72% win rate in head-to-head human preference evaluations against every leading system tested. But the company's accompanying methodology disclosure may be the more significant contribution to the field. Ai2's olmo-eval gives LLM developers a microscope for…

When benchmarks penalize correct answers

Mistral audited the mismatches behind its benchmark scores and found most were not model errors but artifacts of how the evaluation scripts work. The recurring categories reveal an uncomfortable truth about the state of document AI benchmarking.

Ground-truth errors in the reference annotations themselves, like missing or extra text, transcriptions of redacted regions, and typos, mean the model correctly reads the source document but is marked wrong. Equivalent math notation in different LaTeX that renders identically is counted as a mismatch: the rendered equation is correct but the string comparison is not. Equation segmentation, whether an expression is emitted as a single LaTeX block or split across inline fragments, affects the match even when the rendered content is identical. Column-ordering assumptions in multi-column layouts cause correct extractions to be scored as reading-order failures. Block-type attribution issues arise when headers or footers are expected to be stripped, but the test then checks for a string that also appears as a page title that should be present, flagging it incorrectly. The verification horizon: why verifying coding agents…

These artifacts concentrate in mathematical, scientific, and multi-column documents, which happen to be the highest-value use cases for enterprise document extraction. They more often penalize correct output than reward incorrect output. The aggregate score, Mistral acknowledges, is directional rather than definitive.

The structural intelligence gain

Beyond the scoring debate, OCR 4's key product advance is that downstream systems now access not only what the document says but also where each element sits, what role it plays, and how confident the model is in each region. This opens three concrete use cases.

Semantic chunking for RAG becomes more reliable when blocks are pre-classified and localized. A table header should not be split from its rows by a naive character-count chunker. Agentic workflows can move from reading documents to acting on them: form filling, invoice processing, and compliance checks benefit from structural primitives rather than flat text. Confidence scores enable efficient human-in-the-loop verification, where only low-confidence regions require manual review rather than full page rechecks. Your AI search pipeline is broken. This open-source…

Economics and deployment

OCR 4 is compact enough to deploy on a single container for self-hosted environments, addressing data-sovereignty requirements that prevent many enterprises from sending documents to cloud APIs. The model costs $4 per 1,000 pages via API, with a 50% Batch-API discount reducing that to $2 per 1,000 pages. Document AI, which layers structured JSON output via a small model on top of the OCR result, costs $5 per 1,000 pages. Gemma 4 is not a chatbot, and that's the point

Early feedback from Rogo, an AI engineer working with financial documents, reported equivalent accuracy to leading agentic parsers at roughly 8x lower cost and 17x lower latency. For production use cases at scale, those deltas compound rapidly. Cognition's new coding agent scores near frontier…

The model supports 170 languages across 10 language groups, with the widest performance gap in specialized and low-resource languages: Hindi, Japanese, Georgian, Bengali, Armenian, Hebrew, Greek, Tamil, where competing systems degrade sharply. That multilingual range positions OCR 4 for global enterprise deployments that cannot afford to silo document processing by language.

The honest benchmark

Mistral's decision to publish a detailed audit of benchmark artifacts rather than only the headline scores signals a maturity the document-extraction industry has needed. Every vendor in this space faces the same evaluation problems. OCR 4 is the first to lay them out publicly alongside its own results. The practical recommendation for any buyer remains the same as always: evaluate on your own documents, not on benchmark aggregates.