Aleph Alpha needed 4 trillion German tokens. The open web offers 1.94.

A frontier language model is mostly a data problem before it is a modeling problem, and for German the data problem has a number. Aleph Alpha says a German-language training run needs at least 4 trillion German tokens, a floor its ablation studies set by showing a 20% German share performs best on its German benchmarks. The open web, as currently curated, offers about half of it. Data does more than fill a budget, which is why Ai2 let EMO's modularity emerge from its corpus rather than from human priors.

Six free datasets make up the bulk of what a German model builder can pull off the shelf.

Dataset	Size (billion tokens)	Latest source data
HPLT 3.0 (German portion)	609	2025
KletterMix (machine translation of ClimbMix)	500	2024
FineWeb2 (German portion)	378	2024
FinePDFs (German portion)	177	2025
German Commons	154	2025
MT-Nemotron-CC (machine translation of Nemotron-CC)	117	2024

Together they total roughly 1.94 trillion tokens, a figure Aleph Alpha reaches by dividing each dataset's uncompressed byte size by four so the counts are comparable. Training a top model today takes more than 20 trillion tokens, by the company's reckoning, citing Nemotron 3 Super. Token budgets are a live design lever elsewhere too, as Boris-2's lopsided allocation makes plain. Aleph Alpha also leans on Muennighoff et al. on repetition: reusing the same documents stops paying off after about four epochs, so the German target can draw on a smaller pool, but not an arbitrarily small one.

A ceiling, not a total

Two constraints lower the 1.94 trillion. The first is provenance: roughly 600 billion of it, split between KletterMix and MT-Nemotron-CC, comes from machine-translating English corpora rather than German-native text. The second is overlap: HPLT 3.0, FineWeb2 and German Commons all derive from Common Crawl, so their sizes do not simply add up once duplicates are removed.

Then there is freshness. Each free dataset is a snapshot, assembled from crawls up to a fixed date and then frozen; FineWeb2 contains nothing crawled after April 2024. Aleph Alpha's example of the failure is blunt. Trained without retrieval or tool access, a model built on that data will answer with confidence that Olaf Scholz is chancellor. Newer releases push the cutoff closer to the present, but by months, and only when the publisher ships a new version. The hazard is familiar from agent memory, where a stale fact is worse than a missing one.

The company built its own Common Crawl pipeline around that rhythm. It is incremental, so a new snapshot can be folded in without reprocessing what came before, and it is parameterized by language. More than 120 snapshots exist and about one arrives each month, each carrying roughly 100 million German documents, around 50 billion raw tokens. Deduplication and heuristics cut that hard, and Aleph Alpha states the trade-off plainly: every filter step buys quality at the cost of quantity.

Synthetic data multiplies, it does not add

Aleph Alpha uses synthetic data and still treats it as a supporting act. Reformulation, where an LLM rewrites existing German documents into encyclopedic entries, Q&A dialogues or textbook passages, produced about 1 billion tokens for its Aleph-Alpha-GermanWeb dataset, drawn from FineWeb2 German text. The method is cheap in the right way: the model needs good German, not deep German world knowledge, and it dresses the same facts in many phrasings. It cannot create knowledge that was not already there. The same pattern shows elsewhere: a Picbreeder rerun found that VLM-generated archives lacked the boldness of human ones.

Translation is the other half, and it imports a skew. Machine-translated German is usually grammatical but slips into what the company calls translationese, the habit of rendering idioms word for word: 'Drive safe!' becomes 'Fahre sicher!' rather than the idiomatic 'Komm gut an!' Distribution is the harder constraint. A corpus translated from English carries the geographic and institutional lean of the English web, so a model trained mostly on it learns to speak German about a world that looks American. Pleias and NVIDIA base their Nemotron-Personas datasets on national census statistics for the same reason, Aleph Alpha notes.

The counterargument gets a hearing. The authors of ÜberWeb argue that knowledge does transfer across languages. But Aleph Alpha cites both ÜberWeb and more recent work by Gaber et al. as suggesting that transferring well requires careful multilingual curation, and possibly targeted translation, to avoid hurting per-language performance. Neither approach replaces organic German web data, because both presuppose it.

One mis-tuned filter, seven in eight documents lost

The pipeline is the most transferable part. Thresholds, wordlists and classifiers are configurable, and German was the first language added after English. A word-length filter calibrated on English, where Gopher set bounds of 3 to 10 characters, throws out good German: official German prose averages 11.7 characters per word against 5.7 for its English translation. FineWeb2's recommended German interval of 0 to 15 cuts the rejection rate from 0.3% to 0.02%, which across roughly 100 million documents per crawl means about 300,000 ordinary documents kept instead of discarded.

The stopword filter is starker. Applied with the English Gopher list, it discards more than 87% of German documents reaching it, against 0.6% with the German list, or roughly 150 times as many.

Compliance runs through the same stages. Aleph Alpha filters URLs against the European Commission's Counterfeit and Piracy Watch List and the UT1 blacklist maintained by Université Toulouse Capitole, and it replaces personally identifiable information, including emails, phone numbers, IP addresses and bank account numbers, with special tokens. Obfuscated forms like 'name [at] domain' count too. The cost is measured: on its own PII benchmarks the model scores about 5 percentage points below one trained on the same data without PII replacement, while other benchmarks hold.

The bill for every non-English model

The German case reads less like a product announcement than a cost estimate for every language that is not English. Aleph Alpha says it has curated and generated more than 2 trillion high-quality German tokens so far, closing much of the gap between what the open web offers and what a frontier model needs. Whether the next language is cheaper depends less on better synthetic models than on how much organic text that language's web holds. Europe's other major sovereign-model effort has been trimming its lineup too, per the mid-2026 Mistral portfolio audit.