MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages

# Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages
Source: [https://arxiv.org/html/2607.00890](https://arxiv.org/html/2607.00890)
Maximilian Idahl1,2,Jörg Tiedemann3,Sampo Pyysalo4,David Salinas5,6,Tomasz Galica4, Shenbin Qian7,Tudor Nicolae Mateiu8,Zihao Li3,Anna Lokrantz9,Fedor Vitiugin4, André Martins10,11,12,Jenna Kanerva4,Filip Ginter4,Matthias Lindemann10,Tim Isbister9, Birger Moëll9,Jonas Lindh9,Jan Hajič13,Jenia Jitsev14,15,16,17,Andrey Kutuzov7, Stephan Oepen7,Gema Ramírez\-Sánchez8 1ellamind2Leibniz University Hannover3University of Helsinki4University of Turku 5ELLIS Institute Tübingen6Prior Labs7University of Oslo8Prompsit Language Engineering 9AI Sweden10Instituto de Telecomunicações11Instituto Superior Técnico12TransPerfect 13Charles University14Ontocord15LAION16Open\-Ψ\\Psi\(Open\-Sci\) Collective 17Juelich Supercomputing Center \(JSC\), Research Center Juelich \(FZJ\)

###### Abstract

Open web\-scale pre\-training corpora remain concentrated in English, limiting multilingual LLM development\. We introduce MultiSynt/MT, an open synthetic parallel corpus with approximately 4\.8 trillion target\-language tokens across 36 languages, produced by translating 100 billion high\-quality Nemotron\-CC tokens withTower\+and OPUS\-MT/HPLT\-MT systems\. For many medium\- and lower\-resource European languages, this is the largest openly available pre\-training resource\. On a broad multilingual benchmark suite, reference LLMs trained on MultiSynt/MT reach the final score of HPLT 2\.0, a native\-data baseline, using roughly 72% fewer pre\-training tokens, and outperform it by approximately 15% relative at a matched 100B\-token training budget\. Our analyses also identify evaluation blind spots: standard multiple\-choice benchmarks miss translation\-quality differences that a fluency\-sensitive LLM\-as\-judge evaluation cleanly recovers on the trained LLMs \(with no fluency deficit in MultiSynt itself\), and Norwegian idiomatic and culturally grounded tasks, for example, remain better served by native data\. We release the corpus, including row\-aligned translations from multiple systems, to support controlled research on multilingual pre\-training data and evaluation\.

MultiSynt/MT: Trillion\-Token Multi\-Parallel Pre\-Training Data Translated Across 36 Languages

Maximilian Idahl1,2, Jörg Tiedemann3, Sampo Pyysalo4, David Salinas5,6, Tomasz Galica4,Shenbin Qian7,Tudor Nicolae Mateiu8,Zihao Li3,Anna Lokrantz9,Fedor Vitiugin4,André Martins10,11,12,Jenna Kanerva4,Filip Ginter4,Matthias Lindemann10,Tim Isbister9,Birger Moëll9,Jonas Lindh9,Jan Hajič13,Jenia Jitsev14,15,16,17,Andrey Kutuzov7,Stephan Oepen7,Gema Ramírez\-Sánchez81ellamind2Leibniz University Hannover3University of Helsinki4University of Turku5ELLIS Institute Tübingen6Prior Labs7University of Oslo8Prompsit Language Engineering9AI Sweden10Instituto de Telecomunicações11Instituto Superior Técnico12TransPerfect13Charles University14Ontocord15LAION16Open\-Ψ\\Psi\(Open\-Sci\) Collective17Juelich Supercomputing Center \(JSC\), Research Center Juelich \(FZJ\)

## 1Introduction

Openly available pre\-training data at scale exist primarily for English\(Penedoet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib22); Suet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib18)\), with most other languages lacking volume or quality \(see Section[2](https://arxiv.org/html/2607.00890#S2)for details\)\. This shortfall has motivated growing interest in machine translation as a means of producing multilingual pre\-training data at scale, but the practice raises legitimate concerns: translated text can carry stylistic artifacts known as “translationese” and inherits the cultural reference frame of the source language, with names, places, idioms and culturally grounded knowledge in the English source remaining English\-anchored after translation\(Gellerstam,[1986](https://arxiv.org/html/2607.00890#bib.bib2); Rileyet al\.,[2020](https://arxiv.org/html/2607.00890#bib.bib83)\)\.

We address this resource limitation by introducingMultiSynt/MT, an open multilingual synthetic parallel corpus of approximately4\.8 trillion target\-language tokens covering 36 languages, produced by translating a 100B\-token sample of high\-quality web data from Nemotron\-CC with open translation models, includingTower\+9B and 72B\(Reiet al\.,[2026](https://arxiv.org/html/2607.00890#bib.bib4)\)and OPUS\-MT\(Tiedemannet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib90)\)\. MultiSynt/MT is the largest openly available pre\-training corpus to date for many European low\- and medium\-resource languages, exceeding the largest comparable native resource \(HPLT 3\.0;Oepenet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib21)\) by more than an order of magnitude in the lowest\-resource ones \(Figure[1](https://arxiv.org/html/2607.00890#S1.F1)\)\. For a subset of languages, we release row\-aligned parallel translations from multiple systems, enabling controlled comparison under matched source data, and the corpus is released under a permissive open license\.

![Refer to caption](https://arxiv.org/html/2607.00890v1/x1.png)Figure 1:Per\-language token counts in MultiSynt/MT and HPLT 3\.0 \(log scale, Gemma\-3 tokenization, 24\-language representative subset\)\. MultiSynt/MT is larger for 19 of 24 languages, by over an order of magnitude for the lowest\-resource ones \(e\.g\. mlt 182×\\times, gle 153×\\times, nno 79×\\times\); HPLT 3\.0 is larger only on deu and spa \(5–6×\\times\)\. The value above each group gives the MultiSynt/MT\-to\-HPLT 3\.0 token ratio\. Full statistics in Appendix[D](https://arxiv.org/html/2607.00890#A4)\.![Refer to caption](https://arxiv.org/html/2607.00890v1/x2.png)Figure 2:HPLT 2\.0 \(native\) vs MultiSynt/MT \(OPUS\-MT\) on the multilingual benchmark suite, averaged over 5 languages \(dan, nld, ita, por, swe\)\. MultiSynt/MT crosses the HPLT 2\.0 endpoint at∼\\sim28B training tokens and continues to improve through 100B tokens\. Per\-language/benchmark breakdowns are provided as supplementary data\.Alongside the corpus, we present a balanced empirical characterization reporting both where reference LLMs trained on MultiSynt/MT outperform native multilingual baselines and where they fall short, and drawing attention to evaluation practices that obscure these differences\. On standard multilingual base\-model benchmarks, reference LLMs pre\-trained on MultiSynt/MT reach the final score of a native\-data baseline \(HPLT 2\.0111Our pre\-training experiments began before the more recent HPLT 3\.0 release was available, so we use HPLT 2\.0 as the native baseline throughout Section[4](https://arxiv.org/html/2607.00890#S4)\. While the 3\.0 version is larger, the distributions of the two are closely similar\(Oepenet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib21)\); see also Figure[6](https://arxiv.org/html/2607.00890#S5.F6)\.\), using approximately 72% fewer training tokens, and improve over it by approximately 15% relative at a matched 100B\-token training budget \(Section[4](https://arxiv.org/html/2607.00890#S4)\)\. A fluency\-sensitive LLM\-as\-judge evaluation on the trained LLMs cleanly recovers an MT\-system quality ranking that standard multiple\-choice benchmarks fail to discriminate, exposing a blind spot in standard evaluation rather than a fluency deficit in MultiSynt \(Section[5\.1](https://arxiv.org/html/2607.00890#S5.SS1)\); a complementary embedding\-space diagnostic finds that standard benchmark items tend to have more MultiSynt/MT than HPLT 2\.0 documents among their nearest neighbors \(Section[5\.2](https://arxiv.org/html/2607.00890#S5.SS2)\)\. On Norwegian tasks targeting idiomatic and culturally grounded knowledge, models trained on native data outperform translated\-data models throughout training, while on Norwegian commonsense reasoning the translated data closes the gap and eventually overtakes the native baseline; direct evidence that translated text is not a substitute for native corpora on all phenomena \(Section[5\.3](https://arxiv.org/html/2607.00890#S5.SS3)\)\.

## 2Related Work

#### Multilingual pre\-training corpora\.

Open multilingual pre\-training corpora have evolved from filtered Common Crawl pipelines \(CCNet, mC4, OSCAR, ROOTS\) to larger and better\-filtered resources including MADLAD\-400, CulturaX, Glot500, HPLT, FineWeb2 and EMMA\-style adaptation\(Wenzeket al\.,[2020](https://arxiv.org/html/2607.00890#bib.bib100); Conneauet al\.,[2020](https://arxiv.org/html/2607.00890#bib.bib36); Xueet al\.,[2021](https://arxiv.org/html/2607.00890#bib.bib102); Abadjiet al\.,[2022](https://arxiv.org/html/2607.00890#bib.bib24); Laurençonet al\.,[2022](https://arxiv.org/html/2607.00890#bib.bib65); BigScience Workshop,[2022](https://arxiv.org/html/2607.00890#bib.bib86); Kuduguntaet al\.,[2023](https://arxiv.org/html/2607.00890#bib.bib63); Nguyenet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib75); ImaniGooghariet al\.,[2023](https://arxiv.org/html/2607.00890#bib.bib56); de Gibertet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib38); Burchellet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib20); Penedoet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib80); Jiet al\.,[2024a](https://arxiv.org/html/2607.00890#bib.bib58)\), but a persistent volume and quality imbalance remains for lower\-resource languages\(Kreutzeret al\.,[2022](https://arxiv.org/html/2607.00890#bib.bib62)\)\. MultiSynt/MT targets that part of the language distribution as a complement to these corpora; we use HPLT\(Oepenet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib21)\), the strongest openly available native multilingual baseline at our scale, as our principal point of comparison\.

#### High\-quality English source data\.

Translation\-based corpora inherit the quality distribution of their source: recent reference\-baseline work shows that Nemotron\-CC HQ\(Suet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib18)\)produces the strongest downstream performance across model and token scales, ahead of established English corpora including C4, The Pile, RefinedWeb, FineWeb\-Edu and DataComp\-LM\(Raffelet al\.,[2020](https://arxiv.org/html/2607.00890#bib.bib82); Gaoet al\.,[2021](https://arxiv.org/html/2607.00890#bib.bib48); Penedoet al\.,[2023](https://arxiv.org/html/2607.00890#bib.bib79),[2024](https://arxiv.org/html/2607.00890#bib.bib22); Liet al\.,[2024a](https://arxiv.org/html/2607.00890#bib.bib68); Nezhurinaet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib19)\), motivating its use as the MultiSynt/MT source distribution\.

#### Translated corpora for language\-model pre\-training\.

Studies on Basque, Indian languages, Arabic, Indonesian and Tamil consistently find that machine\-translated pre\-training data is competitive with native data on standard NLU but underperforms on culturally nuanced tasks\(Urbizuet al\.,[2023](https://arxiv.org/html/2607.00890#bib.bib92); Doshiet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib41); Boughorbelet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib29); Velasco and Roque,[2025](https://arxiv.org/html/2607.00890#bib.bib94)\)\. Closest in scale, TransWeb\-Edu / CuatroLLM and its extension TransWebLLM translate up to 1\.7T tokens from FineWeb\-Edu into nine languages\(Wanget al\.,[2024a](https://arxiv.org/html/2607.00890#bib.bib98),[2025](https://arxiv.org/html/2607.00890#bib.bib97)\), and the contemporaneous FineTranslations reverses the direction by translating 500\+ non\-English languages into English\(Penedoet al\.,[2026](https://arxiv.org/html/2607.00890#bib.bib81)\)\. A parallel line treats translation as an explicit pre\-training objective rather than a corpus\-construction mechanism\(Jiet al\.,[2024b](https://arxiv.org/html/2607.00890#bib.bib57); Liet al\.,[2024b](https://arxiv.org/html/2607.00890#bib.bib67); Jiet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib59); Liet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib69)\); we instead train standard next\-token language models on translated target\-language documents\. MultiSynt/MT extends this line in three ways: 36\-language coverage at approximately 4\.8T target\-language tokens, a Nemotron\-CC HQ rather than FineWeb\-Edu source, and row\-aligned outputs from multiple open MT systems \(translation\-specialized LLMs in the Tower family\(Alveset al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib5); Reiet al\.,[2026](https://arxiv.org/html/2607.00890#bib.bib4)\)and classical NMT systems\(Tiedemann,[2012](https://arxiv.org/html/2607.00890#bib.bib15); Tiedemannet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib90); Junczys\-Dowmuntet al\.,[2018](https://arxiv.org/html/2607.00890#bib.bib17)\)supporting controlled comparison under matched source data\.

#### Translationese, cultural grounding, and native\-authored evaluation\.

Translated text differs systematically from natively authored text\(Koppel and Ordan,[2011](https://arxiv.org/html/2607.00890#bib.bib61); Volanskyet al\.,[2015](https://arxiv.org/html/2607.00890#bib.bib96); Vanmassenhoveet al\.,[2021](https://arxiv.org/html/2607.00890#bib.bib93); Bizzoniet al\.,[2020](https://arxiv.org/html/2607.00890#bib.bib28)\), with downstream effects on cross\-lingual benchmark interpretation\(Grahamet al\.,[2019](https://arxiv.org/html/2607.00890#bib.bib50); Freitaget al\.,[2020](https://arxiv.org/html/2607.00890#bib.bib47); Artetxeet al\.,[2020](https://arxiv.org/html/2607.00890#bib.bib25); Rileyet al\.,[2020](https://arxiv.org/html/2607.00890#bib.bib83),[2021](https://arxiv.org/html/2607.00890#bib.bib84)\), and multilingual models in turn often lack the cultural and idiomatic knowledge of the target\-language community\(Hershcovichet al\.,[2022](https://arxiv.org/html/2607.00890#bib.bib54); Naouset al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib74); Liuet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib72); Pawaret al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib78); Yaoet al\.,[2024b](https://arxiv.org/html/2607.00890#bib.bib103); Etxanizet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib45)\)\. Multilingual evaluation is itself often translation\-derived \(Belebele, MMLU\-ProX, XNLI, XTREME\)\(Conneauet al\.,[2018](https://arxiv.org/html/2607.00890#bib.bib35); Huet al\.,[2020](https://arxiv.org/html/2607.00890#bib.bib55); Bandarkaret al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib26); Xuanet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib101)\); native\-authored alternatives such as TyDi QA, Global MMLU and NorEval are an essential complement\(Clarket al\.,[2020](https://arxiv.org/html/2607.00890#bib.bib33); Singhet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib89); Mikhailovet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib10); Romanouet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib85); Wuet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib32)\), and LLM\-as\-a\-judge protocols\(Zhenget al\.,[2023](https://arxiv.org/html/2607.00890#bib.bib105); Liuet al\.,[2023](https://arxiv.org/html/2607.00890#bib.bib71); Vergaet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib95); Guet al\.,[2026](https://arxiv.org/html/2607.00890#bib.bib52)\)provide a fluency\-sensitive complement that we exploit in Section[5](https://arxiv.org/html/2607.00890#S5)\.

## 3Building MultiSynt/MT

### 3\.1Source data: high\-quality English web text

We translate from the*actual high\-quality*split of Nemotron\-CC\(Suet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib18)\), which contains non\-synthetic Common Crawl documents receiving the highest scores from its ensemble quality classifier\. Recent reference\-baseline work\(Nezhurinaet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib19)\)finds that Nemotron\-CC HQ produces the strongest average downstream performance among established open English pre\-training corpora across multiple model and token scales\. We draw a uniform random sample of approximately 155 million documents \(on the order of 100 billion English tokens\), sized so that each per\-language translation yields outputs at the same order of magnitude, and apply no topical, length, or quality stratification beyond the underlying quality classifier\. Translations preserve a stable mapping to their source documents, so the resulting corpus is usable both as per\-language pre\-training data and as a multi\-way parallel resource for cross\-language and cross\-system experiments \(Section[3\.4](https://arxiv.org/html/2607.00890#S3.SS4)\)\.

### 3\.2Translation system pool and selection

We require open translation systems that combine strong target\-language quality with throughput feasible at the 100B\-token scale, and that together cover the languages targeted by MultiSynt/MT, including medium\- and lower\-resource European languages where openly available native pre\-training data is scarce\. No single family meets all three requirements: translation\-specialized LLMs in theTowerfamily\(Alveset al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib5); Reiet al\.,[2026](https://arxiv.org/html/2607.00890#bib.bib4)\)and general\-purpose open instruction\-tuned LLMs\(Martinset al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib3); Yanget al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib6); Gemma Team,[2025](https://arxiv.org/html/2607.00890#bib.bib7)\)concentrate on higher\-resource European languages, while OPUS\-MT and HPLT\-MT\(Tiedemannet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib90); de Gibertet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib38)\)extend coverage to lower\-resource languages\. We therefore use LLM\-based translation and OPUS\-MT/HPLT\-MT jointly as the production pool\.

To rank candidate systems, we ran a small human\-judged quality study covering seven languages \(Czech, French, Finnish, German, Italian, Polish, Spanish\)\. For each language, a single native speaker rated translations of the same 10 English source documents from each of six candidate systems on a fluency\-focused 1–5 scale, blind to system identity, additionally penalizing hallucinations, repetitions, and extra or missing text\. The full protocol, KEOPS annotation interface, and per\-language results are in Appendix[E](https://arxiv.org/html/2607.00890#A5)\.Tower\+\(we use the 72B model\) is rated highest in every language, the general\-purpose LLMs \(EuroLLM\-9B\-Instruct, Qwen3\-32B, Gemma\-3\-4b\-it, Mistral\-Small\-3\.2\-24B\-Instruct\) bunch well below it, and OPUS\-MT is close to the general\-purpose LLMs on average \(Figure[3](https://arxiv.org/html/2607.00890#S3.F3)\); the ranking is consistent with the publicTowerMT leaderboard and with per\-language Flores\-200 BLEU on the OPUS\-MT dashboard\.222[opus\.nlpl\.eu](https://opus.nlpl.eu/)Based on this ranking the general\-purpose LLMs were dropped from the production pool\.

![Refer to caption](https://arxiv.org/html/2607.00890v1/x3.png)Figure 3:Mean human quality rating per candidate MT system, averaged over the seven evaluated languages on a 1–5 Likert\-style scale\.Tower\+is rated highest overall\. See Appendix[E](https://arxiv.org/html/2607.00890#A5)for the protocol and Figure[8](https://arxiv.org/html/2607.00890#A5.F8)for the per\-language breakdown\.We ship translations fromTower\+9B \(Gemma\-based, 16 languages\)333[hf\.co/Unbabel/Tower\-Plus\-9B](https://hf.co/Unbabel/Tower-Plus-9B)andTower\+72B \(Qwen\-based, 5 languages: German, Finnish, Italian, Spanish, Swedish\)\.444[hf\.co/Unbabel/Tower\-Plus\-72B](https://hf.co/Unbabel/Tower-Plus-72B)A reference\-LLM comparison between the two on the 5\-language overlap \(Section[4](https://arxiv.org/html/2607.00890#S4)\) showed a small enough downstream gap that we treat 9B as the default LLM\-based system and retain 72B on the high\-resource subset where any residual gains can be isolated\. OPUS\-MT supplies the production models for 31 of our 36 target languages from the Tatoeba Translation Challenge collection\(Tiedemann,[2020](https://arxiv.org/html/2607.00890#bib.bib13)\), and HPLT\-MT supplies the remaining 5 \(Basque, Irish, Icelandic, Maltese, Albanian\) where no comparable OPUS\-MT model is available\. These NMT systems are orders of magnitude smaller than the LLM\-based candidates \(∼\\sim75M–230M parameters\) with corresponding throughput advantages, at the cost of a sentence\-level constraint addressed by the pipeline of Section[3\.3](https://arxiv.org/html/2607.00890#S3.SS3)\. The released corpus therefore contains translations fromTower\+9B \(16 languages\),Tower\+72B \(5 languages\), and OPUS\-MT/HPLT\-MT \(36 languages\); for the languages where multiple systems are shipped, the corpus supports controlled comparison of translation choices under matched source data, which we exploit in Sections[4](https://arxiv.org/html/2607.00890#S4)and[5](https://arxiv.org/html/2607.00890#S5)\.

### 3\.3Translation pipeline at scale

We run two parallel translation pipelines on the same 155 million source documents, both supporting per\-shard resumption and emitting Parquet files that preserve source document identifiers and order so that the released corpus remains a synchronized multi\-parallel resource\. The*LLM pipeline*deploysTower\+9B and 72B on Leonardo HPC cluster \(CINECA, NVIDIA A100 nodes\) with vLLM under a Slurm\-native orchestration toolkit released as open\-source code,555[github\.com/ellamind/inference\-hive](https://github.com/ellamind/inference-hive)consuming approximately 3\.1 million A100 GPU\-hours in aggregate\. The*NMT pipeline*decodes OPUS\-MT and HPLT\-MT models with Marian\-NMT\(Junczys\-Dowmuntet al\.,[2018](https://arxiv.org/html/2607.00890#bib.bib17)\)on LUMI cluster \(AMD MI250x\), with sentence\-splitting pre\-processing and document\-reassembly post\-processing that preserve both per\-sentence and per\-document alignments; the pipeline is released as open\-source code at[github\.com/Helsinki\-NLP/Opus\-MT](https://github.com/Helsinki-NLP/Opus-MT)\. Per\-shard throughput, decoding settings, energy, and carbon\-footprint estimates for both pipelines are reported in Appendix[A](https://arxiv.org/html/2607.00890#A1)\.

### 3\.4Resource overview

MultiSynt/MT contains approximately 4\.8 trillion target\-language tokens across 36 languages, derived from the single shared sample of 155M English documents introduced in Section[3\.1](https://arxiv.org/html/2607.00890#S3.SS1), and is the largest openly available pre\-training corpus to date for many European low\- and medium\-resource languages \(Figure[1](https://arxiv.org/html/2607.00890#S1.F1)\)\. Counting each translation system’s output separately, the released artifact totals approximately 7\.1T tokens \(1\.7T fromTower\+9B, 0\.5T fromTower\+72B, 4\.8T from OPUS\-MT/HPLT\-MT, all under Gemma\-3 tokenization\); per\-language and per\-system breakdowns are in Appendix[D](https://arxiv.org/html/2607.00890#A4)\.

#### Splits and schema\.

For each language we release three splits: The*parallel*split contains the documents translated by all systems released for that language, row\-aligned by index so that rowiiof any language file refers to the same source document, supporting cross\-language and cross\-system controlled experiments\. The*additional*split contains further translated documents that are not present for every language, enabling larger language\-specific volumes at the cost of cross\-language alignment, and the*all*split is the union of the two\. Each row contains the sourcewarc\_record\_id, the translated text, its token count, and the source partition identifier\.

#### Availability and license\.

## 4Effectiveness for Multilingual Pre\-training

### 4\.1Reference model setup

We evaluate MultiSynt/MT as pre\-training data by training reference language models on each MultiSynt/MT translation variant and on a native multilingual baseline dataset\. Across variants, we hold the model architecture, tokenizer, optimizer settings, and total token budget fixed; the variants differ in the pre\-training corpus itself—specifically, the source data and the translation system \(if any\)—and in the set of languages each corpus covers\.

Our reference architecture follows that ofPenedoet al\.\([2024](https://arxiv.org/html/2607.00890#bib.bib22)\): we use a dense Llama\-like transformer with 24 layers, hidden size 2048, 32 attention heads, RoPE positional embeddings, and RMSNorm, with 1\.61B parameters in transformer layers\. We use thegoogle/gemma\-3\-27b\-pttokenizer\(Gemma Team,[2025](https://arxiv.org/html/2607.00890#bib.bib7)\)with a vocabulary size of 262144 for its broad multilingual coverage across the languages in MultiSynt/MT\. We tie word embeddings, giving a total of 2\.15B parameters\. The optimization recipe uses Adam withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, weight decay 0\.05, gradient clipping at 1, global batch size 1024, and a linear WSD learning\-rate schedule with 10% warmup and 20% cooldown\. Full hyperparameters are reported in Appendix[B](https://arxiv.org/html/2607.00890#A2), Table[1](https://arxiv.org/html/2607.00890#A2.T1)\.

Training is conducted with Megatron\-LM\(Shoeybiet al\.,[2019](https://arxiv.org/html/2607.00890#bib.bib23)\)on the LUMI supercomputer, using 16 nodes with 4 AMD MI250x GPUs per node\. For every pre\-training data variant, we train a separate model on a budget of 100 billion tokens, saving checkpoints every 1,000 steps\. Each variant is trained once; we do not report variance over training seeds, and the curves shown in Section[4\.2](https://arxiv.org/html/2607.00890#S4.SS2)should be read as single\-seed point estimates\.

We compare four pre\-training data variants under this setup: HPLT 2\.0\(Burchellet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib20)\), a strong openly available native multilingual baseline at the relevant scale, and MultiSynt/MT translated byTower\+9B \(16 languages\),Tower\+72B \(5 languages\), and OPUS\-MT/HPLT\-MT \(36 languages\)\. Holding architecture, optimizer, and budget fixed lets us attribute downstream differences to the pre\-training corpus\. Within the MultiSynt/MT family, this isolates the choice of translation system; between MultiSynt/MT and HPLT 2\.0 the comparison additionally varies the source corpus and its quality\-filtering pipeline, a confounding factor we discuss in Section[6](https://arxiv.org/html/2607.00890#S6)\.

#### Evaluation\.

Downstream evaluation uses two open frameworks, LightEval\(Habibet al\.,[2023](https://arxiv.org/html/2607.00890#bib.bib8)\)and LM\-evaluation\-harness\(Gaoet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib9)\)\. We evaluate each checkpoint on a broad multilingual benchmark suite: belebeleBandarkaret al\.\([2024](https://arxiv.org/html/2607.00890#bib.bib26)\), arc:challengeClarket al\.\([2018](https://arxiv.org/html/2607.00890#bib.bib108)\); Dac Laiet al\.\([2023](https://arxiv.org/html/2607.00890#bib.bib110)\), hellaswagZellerset al\.\([2019](https://arxiv.org/html/2607.00890#bib.bib109)\); Dac Laiet al\.\([2023](https://arxiv.org/html/2607.00890#bib.bib110)\), goldenswagChizhovet al\.\([2025](https://arxiv.org/html/2607.00890#bib.bib111)\), xstory\-clozeLinet al\.\([2021](https://arxiv.org/html/2607.00890#bib.bib112)\), global\-mmluSinghet al\.\([2025](https://arxiv.org/html/2607.00890#bib.bib89)\), mmluHendryckset al\.\([2021](https://arxiv.org/html/2607.00890#bib.bib53)\); Dac Laiet al\.\([2023](https://arxiv.org/html/2607.00890#bib.bib110)\), examsHardalovet al\.\([2020](https://arxiv.org/html/2607.00890#bib.bib114)\), xcodahChenet al\.\([2019](https://arxiv.org/html/2607.00890#bib.bib115)\), xcsqaTalmoret al\.\([2019](https://arxiv.org/html/2607.00890#bib.bib116)\), MultiBlimpJumeletet al\.\([2026](https://arxiv.org/html/2607.00890#bib.bib117)\), enemNuneset al\.\([2023](https://arxiv.org/html/2607.00890#bib.bib118)\), faquadSayamaet al\.\([2019](https://arxiv.org/html/2607.00890#bib.bib123)\), oab\-examsPireset al\.\([2025](https://arxiv.org/html/2607.00890#bib.bib119)\), glianorexGriotet al\.\([2025](https://arxiv.org/html/2607.00890#bib.bib120)\), xnliConneauet al\.\([2018](https://arxiv.org/html/2607.00890#bib.bib35)\), pawsxYanget al\.\([2019](https://arxiv.org/html/2607.00890#bib.bib122)\), Finbenchkytöniemi2025finbenchv2unifiedrobustbenchmark\. We apply the same evaluation protocol across all variants\.

Per\-language scores are aggregated into a single mean across the languages covered by every variant under comparison; per\-language and per\-benchmark breakdowns are provided as supplementary data\.

#### Decontamination check\.

A controlled training experiment with vs\. without n\-gram decontamination of the Nemotron\-CC source rules out source\-document contamination as an explanation for the Section[4\.2](https://arxiv.org/html/2607.00890#S4.SS2)gains \(the released MultiSynt/MT corpus is itself not decontaminated due to the low risk of actual contamination; see Appendix[C](https://arxiv.org/html/2607.00890#A3)\)\.

### 4\.2Gains over native baselines

Figure[2](https://arxiv.org/html/2607.00890#S1.F2)reports aggregate benchmark performance over training tokens, comparing reference LLMs pre\-trained on the HPLT 2\.0 native baseline against reference LLMs pre\-trained on MultiSynt/MT OPUS\-MT, averaged over the five languages where both baselines have evaluation data \(Danish, Dutch, Italian, Portuguese, Swedish\)\. Models pre\-trained on MultiSynt/MT reach the HPLT 2\.0 endpoint score at approximately 28B training tokens, roughly72%72\\%fewer than the native baseline; at the matched 100B\-token training budget, MultiSynt/MT exceeds the HPLT 2\.0 endpoint by approximately 15% relative\.

Figure[4](https://arxiv.org/html/2607.00890#S4.F4)compares the three MultiSynt/MT variants \(OPUS\-MT,Tower\+9B,Tower\+72B\) on Swedish, the single language for which a reference LLM was trained on every variant\. The three MultiSynt/MT curves cluster tightly together throughout training and all sit well above the HPLT 2\.0 baseline, indicating that on these standard multilingual benchmarks the gain over native data is largely independent of the choice of MT system within MultiSynt/MT\. The downstream gap amongTower\+9B,Tower\+72B, and OPUS\-MT is therefore small enough that we treat the 9B variant as our default LLM\-based system and retain 72B only on the high\-resource subset where any residual gains can be isolated \(Section[4\.1](https://arxiv.org/html/2607.00890#S4.SS1)\)\.

Per\-language and per\-benchmark breakdowns are reported in Appendix[D](https://arxiv.org/html/2607.00890#A4)\.

![Refer to caption](https://arxiv.org/html/2607.00890v1/x4.png)Figure 4:MultiSynt/MT variants \(OPUS\-MT,Tower\+9B,Tower\+72B\) vs the HPLT 2\.0 native baseline on the Swedish multilingual benchmark suite \(the single language for which a reference LLM was trained on every variant\)\. The three MultiSynt/MT curves sit indistinguishably above HPLT 2\.0 throughout training: translated vs native dominates MT\-system choice\.

## 5Interpreting the Gains: Fluency, Benchmark Alignment, and Native\-Authored Tasks

Section[4](https://arxiv.org/html/2607.00890#S4)showed that training on MultiSynt/MT yields large gains over native data on standard multilingual benchmarks\. We now interpret those gains along three axes the standard benchmarks do not surface: an LLM\-judge fluency probe that exposes MT\-system differences flattened by multiple\-choice scoring \(Section[5\.1](https://arxiv.org/html/2607.00890#S5.SS1)\), an embedding\-space diagnostic of benchmark\-corpus alignment \(Section[5\.2](https://arxiv.org/html/2607.00890#S5.SS2)\), and a Norwegian case study on native\-authored idiomatic and culturally grounded tasks \(Section[5\.3](https://arxiv.org/html/2607.00890#S5.SS3)\)\. We draw the methodological consequences in Section[6](https://arxiv.org/html/2607.00890#S6)\.

### 5\.1What standard benchmarks miss about MT\-system choice

The MT\-variant indistinguishability of Figure[4](https://arxiv.org/html/2607.00890#S4.F4)is surprising: the same three translation systems are clearly separated by human raters \(Figure[3](https://arxiv.org/html/2607.00890#S3.F3)\) and by the publicTowerMT leaderboard\. A plausible explanation is that translationese artifacts in the MT output propagate to the trained LLM in a form the standard multilingual benchmarks of Section[4](https://arxiv.org/html/2607.00890#S4), predominantly multiple\-choice QA and selection, cannot detect: those benchmarks reward picking the right token under strong exploitable cues, while the surface fluency of the model’s own free\-form generations does not enter the score\.

To probe this hypothesis, we evaluate the pre\-trained models with an LLM\-as\-a\-judge protocol that scores fluency of free\-form generations\. For each of German, Spanish, Finnish, Italian and Swedish we collect 100 natural\-language cut\-out sentences \(e\.g\.,“Das Grundgesetz der Bundesrepublik Deutschland garantiert…”\)\. Each candidate pre\-trained model produces a continuation that DeepSeek V3\.1 then compares against a continuation from a monolingual HPLT 2\.0 1\.7B native\-data baseline; we report the candidate’s*winrate*averaged over prompts and languages\.

As a sanity check, the off\-the\-shelf Qwen 2\.5 series at five sizes from 0\.5B to 14B parameters shows winrate rising monotonically with model size against the HPLT 2\.0 baseline \(Appendix[F](https://arxiv.org/html/2607.00890#A6), Figure[9](https://arxiv.org/html/2607.00890#A6.F9)\), confirming the judge responds to generation quality\. The reference LLMs pre\-trained on MultiSynt/MT with matched data and budget produce a clear ranking,Tower\+72B≥\\geqTower\+9B\>\>OPUS\-MT \(Figure[5](https://arxiv.org/html/2607.00890#S5.F5)\), consistent with the human evaluation of the underlying MT outputs \(Figure[3](https://arxiv.org/html/2607.00890#S3.F3)\) and the publicTowerMT leaderboard\. All three MultiSynt/MT variants match or exceed the HPLT 2\.0 baseline on the judge, so MultiSynt/MT is not fluency\-deficient; the finding is that standard benchmarks are blind to translation\-quality differences the trained LLMs nonetheless inherit\.

![Refer to caption](https://arxiv.org/html/2607.00890v1/x5.png)Figure 5:Fluency LLM\-judge winrate of our reference 1\.7B LLMs pre\-trained on MultiSynt/MT against a monolingual HPLT 2\.0 1\.7B baseline\. The rankingTower\+72B≥\\geqTower\+9B\>\>OPUS\-MT matches the human MT evaluation \(Figure[3](https://arxiv.org/html/2607.00890#S3.F3)\) and the public Tower MT leaderboard\.
### 5\.2Benchmark\-corpus alignment in embedding space

As a complementary diagnostic, we compare the embedding neighborhoods of standard benchmark items against MultiSynt/MT and HPLT 2\.0 documents\. Across most benchmarks in the suite, benchmark items have more MultiSynt/MT documents among their nearest embedding neighbors, suggesting that translated high\-quality English\-source data provides strong coverage of many benchmark\-adjacent regions; this is not a causal test of the gains, but it indicates that benchmark\-corpus alignment is one reason those gains should be interpreted with care \(Appendix[G](https://arxiv.org/html/2607.00890#A7)\)\.

![Refer to caption](https://arxiv.org/html/2607.00890v1/x6.png)Figure 6:NorEval performance across 11 training checkpoints for reference LLMs pre\-trained on native Norwegian \(HPLT 2\.0, HPLT 3\.0\) vs MultiSynt/MT translated \(OPUS\-MT,Tower\+9B\) data\. Left: NorIdiom\-nob \(Bokmål idiom completion, F1\); native data dominate translated data throughout training\. Right: NorCommonsenseQA\-nob \(Bokmål commonsense MCQ, accuracy\); the four curves overlap, translated data is slightly ahead at 100B tokens\.
### 5\.3Where translated data falls short: a Norwegian case study

Named entities, idiomatic constructions, and culturally specific knowledge in English text remain English\-anchored after translation, so we expect translated data to underperform native data on target\-language tasks probing local idiomaticity or culture even when the translations are fluent\. We test this by training four reference LLMs sharing the architecture and recipe of Section[4](https://arxiv.org/html/2607.00890#S4)and differing only in Norwegian pre\-training data: HPLT 2\.0 \(native\), HPLT 3\.0 \(native\), MultiSynt/MT translated byTower\+9B, and MultiSynt/MT translated by OPUS\-MT, evaluated across 11 training checkpoints on two Bokmål tasks from NorEval777NorEval comprises 24 native\-authored Norwegian tasks covering both Bokmål and Nynorsk, the two written variants of Norwegian\. Aggregate benchmark results, obtained by normalizing and averaging scores across all tasks, show a clear advantage for models trained on native Norwegian data\.\(Mikhailovet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib10)\):NorIdiom\(idiomatic expressions\) andNorCommonsenseQA\(commonsense reasoning\), selected as illustrative examples following the criteria inOepenet al\.\([2025](https://arxiv.org/html/2607.00890#bib.bib21)\)\.

OnNorIdiom\(Figure[6](https://arxiv.org/html/2607.00890#S5.F6), left\), both native\-data models outperform both translated\-data models with a stable gap throughout training, consistent with idiomatic constructions being a long\-tail phenomenon of the target language that the translated corpus does not generate at the rate native text does\. OnNorCommonsenseQA\(Figure[6](https://arxiv.org/html/2607.00890#S5.F6), right\), native data leads in early\-to\-mid training but the translated\-data models close the gap and match or exceed the native models in later checkpoints; this is a complementary signal that translated data lacks local cultural anchoring but preserves the structured reasoning patterns of the high\-quality English source\(Kabraet al\.,[2026](https://arxiv.org/html/2607.00890#bib.bib11)\)\. Together, the two tasks suggest a content\-dependent picture: native data wins on tasks probing culturally or idiomatically local knowledge; on tasks probing transferable reasoning structure, models trained on MultiSynt/MT match or exceed those trained on native data at matched training budget\.

## 6Discussion and Recommendations

#### Treat translated data as a complement to native data, not a replacement\.

For high\-resource languages, where native corpora are already abundant, MultiSynt/MT is most useful as a domain\-coverage extension\. For medium\- and lower\-resource languages, translating a high\-quality source corpus is the primary vehicle for closing the high\-quality data\-scale gap, but it should still be paired with whatever native corpus is available to preserve idiomatic and cultural anchoring \(Section[5\.3](https://arxiv.org/html/2607.00890#S5.SS3)\)\.

#### The headline gain may reflect source quality alone, not translation as a beneficial operation\.

The Section[4\.2](https://arxiv.org/html/2607.00890#S4.SS2)comparison varies both the source corpus \(Nemotron\-CC HQ vs\. native HPLT 2\.0 web text\) and the translation step, so the gain cannot be attributed to translation*per se*: it is consistent with the net effect arising entirely from the higher quality of the English source, with translation contributing nothing of its own\. Disentangling the two would require a controlled source\-corpus comparison that we leave to future work\. The practical counterfactual is nonetheless not arbitrary translated versus arbitrary native text, but translated high\-quality source text at 100B\-token scale versus the much smaller native pool that would survive equivalently stringent quality filtering, which for medium\- and lower\-resource languages would shrink the already\-scarce native data to a small fraction of itself\.

#### Pair translation\-derived benchmarks with native\-authored ones, and complement multiple\-choice with quality\-sensitive measures\.

A large fraction of widely used multilingual benchmarks, including Belebele\(Bandarkaret al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib26)\), m\-ArenaHard\(Danget al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib37)\), MMMLU\(OpenAI,[2024](https://arxiv.org/html/2607.00890#bib.bib77)\)and MT\-Bench\-X\(Weberet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib99)\), are themselves produced by translating English sources, and our embedding\-space diagnostic shows that they tend to sit in neighborhoods with more MultiSynt/MT than HPLT 2\.0 documents \(Section[5\.2](https://arxiv.org/html/2607.00890#S5.SS2)\)\. A model trained on translated data shares the same source\-language frame as these benchmarks and is, in part, being evaluated on how well it has absorbed that frame; standard multiple\-choice scoring is additionally blind to fluency\-quality differences that the resulting LLMs nevertheless inherit \(Section[5\.1](https://arxiv.org/html/2607.00890#S5.SS1)\)\. Native\-authored benchmarks such as NorEval\(Mikhailovet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib10)\)expose gaps that translation\-derived benchmarks fail to surface \(Section[5\.3](https://arxiv.org/html/2607.00890#S5.SS3)\); free\-form quality\-sensitive measures such as the LLM\-judge protocol of Section[5\.1](https://arxiv.org/html/2607.00890#S5.SS1)expose differences that discriminative ones flatten\.

## 7Conclusion

MultiSynt/MT provides approximately 4\.8T target\-language tokens across 36 languages, the largest openly available pre\-training corpus to date for many European low\- and medium\-resource languages, and reference LLMs trained on it match a native\-data baseline \(HPLT 2\.0\) using roughly 72% fewer pre\-training tokens while exceeding it by approximately 15% relative at a matched 100B\-token budget\. The translated corpus nonetheless falls short on idiomatic and culturally grounded Norwegian tasks and is best understood as a complement to native data; we release it under CC0 with row\-aligned translations from multiple MT systems to support controlled follow\-up on translation\-system choice, scaling, native\-benchmark coverage, and source\-quality vs\. translation attribution\.

## Limitations

#### Corpus and translation biases\.

MultiSynt/MT inherits the topical, stylistic, and English\-centric biases of its Nemotron\-CC HQ source: a quality\-filtered Common Crawl pool that favours factual prose and formal registers over oral, vernacular, dialectal, and culturally local text, and that anchors named entities, idioms, and culturally specific knowledge to an English reference frame\. The chosen MT systems further introduce systematic translationese artifacts \(reduced lexical variety, calque\-driven syntax, lower idiomatic density\) that the standard multilingual benchmarks of Section[4](https://arxiv.org/html/2607.00890#S4)flatten but that surface under fluency\-sensitive evaluation \(Section[5\.1](https://arxiv.org/html/2607.00890#S5.SS1)\) and on natively authored idiomatic and culturally grounded tasks \(Section[5\.3](https://arxiv.org/html/2607.00890#S5.SS3)\)\. The headline comparison against HPLT 2\.0 \(Section[4\.2](https://arxiv.org/html/2607.00890#S4.SS2)\) additionally varies the source distribution and its quality\-filtering pipeline, so the gain cannot be attributed to translation alone; an isolating comparison against an equivalently stringently filtered native baseline is left to future work\.

#### Evaluation coverage, scale, and feedback risks\.

Our empirical analyses are concentrated on European languages in the high\- and medium\-resource range and are conducted at a single 1\.7B\-parameter scale with single\-seed pre\-training runs \(Section[4\.1](https://arxiv.org/html/2607.00890#S4.SS1)\), so behaviour on the lowest\-resource and non\-European languages in the released corpus, scaling trends to larger models, and seed\-level variance are not directly verified by our experiments\. The embedding\-space alignment diagnostic \(Section[5\.2](https://arxiv.org/html/2607.00890#S5.SS2)\) is descriptive rather than causal, the fluency LLM\-judge protocol \(Section[5\.1](https://arxiv.org/html/2607.00890#S5.SS1)\) inherits the biases of its judge and prompt set, and the human MT evaluation \(Section[3\.2](https://arxiv.org/html/2607.00890#S3.SS2)\) covers only seven languages with one annotator each\. Finally, releasing MultiSynt/MT openly creates a feedback path into future open MT systems trained on web crawls that will then ingest it, with the attendant long\-tail erosion risks of recursive synthetic\-data training\(Shumailovet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib12)\); we mark the corpus as synthetic in the released metadata and recommend treating it as a complement to native data rather than a replacement \(Section[6](https://arxiv.org/html/2607.00890#S6)\)\.

## Acknowledgments

We acknowledge EuroHPC JU for awarding the project ID EHPC\-AIF\-2025LS01\-028 access to the EuroHPC supercomputer LEONARDO hosted by CINECA \(Italy\) and the LEONARDO consortium\. We acknowledge EuroHPC JU for awarding the project ID EHPC\-AIF\-2025LS16\-024 access to the EuroHPC supercomputer MareNostrum 5 hosted by the Barcelona Supercomputing Center \(BSC\)\. We acknowledge EuroHPC JU for awarding the project ID HPC\-REG\-2024R02\-167 access to the EuroHPC supercomputer LUMI hosted by CSC \(Finland\) and the LUMI consortium\. Part of the computations were performed on resources provided by Sigma2 \- the National Infrastructure for High\-Performance Computing and Data Storage in Norway\. This research was supported by the OpenEuroLLM project, co\-funded by the Digital Europe Programme under GA no\. 101195233, and partially by DECOLLAGE \(ERC\-2022\-CoG 101088763\)\. This project is supported by the German Federal Ministry for Economic Affairs and Energy \(BMWE\) through EU\-SAI/SOOFI: Sovereign Open Source Foundation Models for European Intelligence \(grant number 13IPC040J\)\.

## References

- Towards a cleaner document\-oriented multilingual crawled corpus\.InProceedings of the Thirteenth Language Resources and Evaluation Conference,Marseille, France,pp\. 4344–4355\.External Links:[Link](https://aclanthology.org/2022.lrec-1.463/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- D\. M\. Alves, J\. Pombal, N\. M\. Guerreiro, P\. H\. Martins, J\. Alves, A\. Farajian, B\. Peters, R\. Rei, P\. Fernandes, S\. Agrawal, P\. Colombo, J\. G\. C\. de Souza, and A\. Martins \(2024\)Tower: an open multilingual large language model for translation\-related tasks\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=EHPns3hVkj)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2607.00890#S3.SS2.p1.1)\.
- M\. Artetxe, G\. Labaka, and E\. Agirre \(2020\)Translation artifacts in cross\-lingual transfer learning\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,Online,pp\. 7674–7684\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.618),[Link](https://aclanthology.org/2020.emnlp-main.618/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- L\. Bandarkar, D\. Liang, B\. Muller, M\. Artetxe, S\. N\. Shukla, D\. Husa, N\. Goyal, A\. Krishnan, L\. Zettlemoyer, and M\. Khabsa \(2024\)The belebele benchmark: a parallel reading comprehension dataset in 122 language variants\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 749–775\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.44),[Link](https://aclanthology.org/2024.acl-long.44/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2607.00890#S6.SS0.SSS0.Px3.p1.1)\.
- BigScience Workshop \(2022\)BLOOM: a 176b\-parameter open\-access multilingual language model\.CoRRabs/2211\.05100\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2211.05100),[Link](https://arxiv.org/abs/2211.05100)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Bizzoni, T\. S\. Juzek, C\. España\-Bonet, K\. Dutta Chowdhury, J\. van Genabith, and E\. Teich \(2020\)How human is machine translationese? comparing human and machine translations of text and speech\.InProceedings of the 17th International Conference on Spoken Language Translation,Online,pp\. 280–290\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.iwslt-1.34),[Link](https://aclanthology.org/2020.iwslt-1.34/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Boughorbel, M\. R\. Parvez, and M\. Hawasly \(2024\)Improving language models trained on translated data with continual pre\-training and dictionary learning analysis\.InProceedings of the Second Arabic Natural Language Processing Conference,Bangkok, Thailand,pp\. 73–88\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.arabicnlp-1.7),[Link](https://aclanthology.org/2024.arabicnlp-1.7/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1877–1901\.External Links:[Link](https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)Cited by:[Appendix C](https://arxiv.org/html/2607.00890#A3.SS0.SSS0.Px1.p1.1)\.
- L\. Burchell, O\. de Gibert, N\. Arefyev, M\. Aulamo, M\. Bañón, P\. Chen, M\. Fedorova, L\. Guillou, B\. Haddow, J\. Hajič, J\. Helcl, E\. Henriksson, M\. Klimaszewski, V\. Komulainen, A\. Kutuzov, J\. Kytöniemi, V\. Laippala, P\. Mæhlum, B\. Malik, F\. Mehryary, V\. Mikhailov, N\. Moghe, A\. Myntti, D\. O’Brien, S\. Oepen, P\. Pal, J\. Piha, S\. Pyysalo, G\. Ramírez\-Sánchez, D\. Samuel, P\. Stepachev, J\. Tiedemann, D\. Variš, T\. Vojtěchová, and J\. Zaragoza\-Bernabeu \(2025\)An expanded massive multilingual dataset for high\-performance language technologies \(HPLT\)\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 17452–17485\.Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.p4.1)\.
- M\. Chen, M\. D’Arcy, A\. Liu, J\. Fernandez, and D\. Downey \(2019\)CODAH: an adversarially\-authored question answering dataset for common sense\.InProceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP,Minneapolis, USA,pp\. 63–69\.External Links:[Document](https://dx.doi.org/10.18653/v1/W19-2008),[Link](https://www.aclweb.org/anthology/W19-2008)Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- P\. Chizhov, M\. Nee, P\. Langlais, and I\. P\. Yamshchikov \(2025\)What the hellaswag? on the validity of common\-sense reasoning benchmarks\.External Links:2504\.07825,[Link](https://arxiv.org/abs/2504.07825)Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- J\. H\. Clark, E\. Choi, M\. Collins, D\. Garrette, T\. Kwiatkowski, V\. Nikolaev, and J\. Palomaki \(2020\)TyDi QA: a benchmark for information\-seeking question answering in typologically diverse languages\.Transactions of the Association for Computational Linguistics8,pp\. 454–470\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00317),[Link](https://aclanthology.org/2020.tacl-1.30/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv:1803\.05457v1\.Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov \(2020\)Unsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp\. 8440–8451\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747),[Link](https://aclanthology.org/2020.acl-main.747/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Conneau, R\. Rinott, G\. Lample, A\. Williams, S\. R\. Bowman, H\. Schwenk, and V\. Stoyanov \(2018\)XNLI: evaluating cross\-lingual sentence representations\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Brussels, Belgium,pp\. 2475–2485\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1269),[Link](https://aclanthology.org/D18-1269/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- V\. Dac Lai, C\. Van Nguyen, N\. T\. Ngo, T\. Nguyen, F\. Dernoncourt, R\. A\. Rossi, and T\. H\. Nguyen \(2023\)Okapi: instruction\-tuned large language models in multiple languages with reinforcement learning from human feedback\.arXiv e\-prints,pp\. arXiv–2307\.Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Dang, S\. Singh, D\. D’souza, A\. Ahmadian, A\. Salamanca, M\. Smith, A\. Peppin, S\. Hong, M\. Govindassamy, T\. Zhao, S\. Kublik, M\. Amer, V\. Aryabumi, J\. A\. Campos, Y\. Tan, T\. Kocmi, F\. Strub, N\. Grinsztajn, Y\. Flet\-Berliac, A\. Locatelli, H\. Lin, D\. Talupuru, B\. Venkitesh, D\. Cairuz, B\. Yang, T\. Chung, W\. Ko, S\. S\. Shi, A\. Shukayev, S\. Bae, A\. Piktus, R\. Castagné, F\. Cruz\-Salinas, E\. Kim, L\. Crawhall\-Stein, A\. Morisot, S\. Roy, P\. Blunsom, I\. Zhang, A\. Gomez, N\. Frosst, M\. Fadaee, B\. Ermis, A\. Üstün, and S\. Hooker \(2024\)Aya expanse: combining research breakthroughs for a new multilingual frontier\.External Links:2412\.04261,[Link](https://arxiv.org/abs/2412.04261)Cited by:[§6](https://arxiv.org/html/2607.00890#S6.SS0.SSS0.Px3.p1.1)\.
- O\. de Gibert, G\. Nail, N\. Arefyev, M\. Bañón, J\. van der Linde, S\. Ji, J\. Zaragoza\-Bernabeu, M\. Aulamo, G\. Ramírez\-Sánchez, A\. Kutuzov, S\. Pyysalo, S\. Oepen, and J\. Tiedemann \(2024\)A new massive multilingual dataset for high\-performance language technologies\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),Torino, Italia,pp\. 1116–1128\.External Links:[Link](https://aclanthology.org/2024.lrec-main.100/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2607.00890#S3.SS2.p1.1)\.
- M\. Doshi, R\. Dabre, and P\. Bhattacharyya \(2024\)Pretraining language models using translationese\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 5843–5862\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.334),[Link](https://aclanthology.org/2024.emnlp-main.334/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Etxaniz, G\. Azkune, A\. Soroa, O\. Lopez de Lacalle, and M\. Artetxe \(2024\)BertaQA: how much do language models know about local culture?\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,External Links:[Link](https://arxiv.org/abs/2406.07302)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Freitag, D\. Grangier, and I\. Caswell \(2020\)BLEU might be guilty but references are not innocent\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Online,pp\. 61–71\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.5),[Link](https://aclanthology.org/2020.emnlp-main.5/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. Leahy \(2021\)The Pile: an 800GB dataset of diverse text for language modeling\.CoRRabs/2101\.00027\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2101.00027),[Link](https://arxiv.org/abs/2101.00027)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- M\. Gellerstam \(1986\)Translationese in swedish novels translated from english\.Translation studies in Scandinavia1,pp\. 88–95\.Cited by:[§1](https://arxiv.org/html/2607.00890#S1.p1.1)\.
- Gemma Team \(2025\)Gemma 3 technical report\.CoRRabs/2503\.19786\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2503.19786),[Link](https://arxiv.org/abs/2503.19786)Cited by:[§3\.2](https://arxiv.org/html/2607.00890#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.p2.2)\.
- Y\. Graham, B\. Haddow, and P\. Koehn \(2019\)Translationese in machine translation evaluation\.InProceedings of the Fourth Conference on Machine Translation,Florence, Italy,pp\. 72–81\.External Links:[Link](https://aclanthology.org/W19-5208/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Griot, J\. Vanderdonckt, D\. Yuksel, and C\. Hemptinne \(2025\)Pattern recognition or medical knowledge? the problem with multiple\-choice questions in medicine\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5321–5341\.External Links:[Link](http://dx.doi.org/10.18653/v1/2025.acl-long.266),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.266)Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu, S\. Wang, K\. Zhang, Z\. Lin, B\. Zhang, L\. Ni, W\. Gao, Y\. Wang, and J\. Guo \(2026\)A survey on LLM\-as\-a\-judge\.The Innovation,pp\. 101253\.External Links:[Document](https://dx.doi.org/10.1016/j.xinn.2025.101253),[Link](https://doi.org/10.1016/j.xinn.2025.101253)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- N\. Habib, C\. Fourrier, H\. Kydlíček, T\. Wolf, and L\. Tunstall \(2023\)LightEval: a lightweight framework for llm evaluation\.External Links:[Link](https://github.com/huggingface/lighteval)Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- M\. Hardalov, T\. Mihaylov, D\. Zlatkova, Y\. Dinkov, I\. Koychev, and P\. Nakov \(2020\)EXAMS: a multi\-subject high school examinations dataset for cross\-lingual and multilingual question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 5427–5444\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.438),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.438)Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1),[OpenAI \(2024\)](https://arxiv.org/html/2607.00890#bib.bib77)\.
- D\. Hershcovich, S\. Frank, H\. Lent, M\. de Lhoneux, M\. Abdou, S\. Brandl, E\. Bugliarello, L\. Cabello Piqueras, I\. Chalkidis, R\. Cui, C\. Fierro, K\. Margatina, P\. Rust, and A\. Søgaard \(2022\)Challenges and strategies in cross\-cultural NLP\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Dublin, Ireland,pp\. 6997–7013\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.482),[Link](https://aclanthology.org/2022.acl-long.482/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- J\. Hu, S\. Ruder, A\. Siddhant, G\. Neubig, O\. Firat, and M\. Johnson \(2020\)XTREME: a massively multilingual multi\-task benchmark for evaluating cross\-lingual generalisation\.InProceedings of the 37th International Conference on Machine Learning,pp\. 4411–4421\.External Links:[Link](https://proceedings.mlr.press/v119/hu20b.html)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- A\. ImaniGooghari, P\. Lin, A\. H\. Kargaran, S\. Severini, M\. Jalili Sabet, N\. Kassner, C\. Ma, H\. Schmid, A\. Martins, F\. Yvon, and H\. Schütze \(2023\)Glot500: scaling multilingual corpora and language models to 500 languages\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada,pp\. 1082–1117\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.61),[Link](https://aclanthology.org/2023.acl-long.61/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Ji, Z\. Li, J\. Paavola, I\. Paul, H\. Luo, and J\. Tiedemann \(2025\)Massively multilingual adaptation of large language models using bilingual translation data\.External Links:2506\.00469,[Link](https://arxiv.org/abs/2506.00469)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Ji, Z\. Li, I\. Paul, J\. Paavola, P\. Lin, P\. Chen, D\. O’Brien, H\. Luo, H\. Schütze, J\. Tiedemann, and B\. Haddow \(2024a\)EMMA\-500: enhancing massively multilingual adaptation of large language models\.External Links:2409\.17892,[Link](https://arxiv.org/abs/2409.17892)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Ji, T\. Mickus, V\. Segonne, and J\. Tiedemann \(2024b\)Can machine translation bridge multilingual pretraining and cross\-lingual transfer learning?\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation,Torino, Italy,pp\. 2809–2818\.External Links:[Link](https://aclanthology.org/2024.lrec-main.250/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Jumelet, L\. Weissweiler, J\. Nivre, and A\. Bisazza \(2026\)MultiBLiMP 1\.0: a massively multilingual benchmark of linguistic minimal pairs\.External Links:2504\.02768,[Link](https://arxiv.org/abs/2504.02768)Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- M\. Junczys\-Dowmunt, R\. Grundkiewicz, T\. Dwojak, H\. Hoang, K\. Heafield, T\. Neckermann, F\. Seide, U\. Germann, A\. F\. Aji, N\. Bogoychev, A\. F\. T\. Martins, and A\. Birch \(2018\)Marian: fast neural machine translation in C\+\+\.InProceedings of ACL 2018, System Demonstrations,Melbourne, Australia,pp\. 116–121\.External Links:[Link](https://aclanthology.org/P18-4020/),[Document](https://dx.doi.org/10.18653/v1/P18-4020)Cited by:[Appendix A](https://arxiv.org/html/2607.00890#A1.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1),[§3\.3](https://arxiv.org/html/2607.00890#S3.SS3.p1.1)\.
- A\. Kabra, Y\. Yin, A\. Gong, K\. Stankevičiūtė, D\. Go, J\. Lee, K\. Z\. Luo, C\. P\. Gomes, and K\. Q\. Weinberger \(2026\)Learning from synthetic data improves multi\-hop reasoning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=38nYZ5QBui)Cited by:[§5\.3](https://arxiv.org/html/2607.00890#S5.SS3.p2.1)\.
- T\. Kocmi, E\. Avramidis, R\. Bawden, O\. Bojar, K\. Dranch, A\. Dvorkovich, S\. Dukanov, N\. Fedorova, M\. Fishel, M\. Freitag, T\. Gowda, R\. Grundkiewicz, B\. Haddow, M\. Karpinska, P\. Koehn, H\. Lakougna, J\. Lundin, K\. Murray, M\. Nagata, S\. Perrella, L\. Proietti, M\. Popel, M\. Popović, P\. Riley, M\. Shmatova, S\. Steingrímsson, L\. Yankovskaya, and V\. Zouhar \(2025\)Preliminary ranking of WMT25 general machine translation systems\.External Links:2508\.14909,[Link](https://arxiv.org/abs/2508.14909)Cited by:[Appendix E](https://arxiv.org/html/2607.00890#A5.SS0.SSS0.Px3.p1.1)\.
- M\. Koppel and N\. Ordan \(2011\)Translationese and its dialects\.InProceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies,Portland, Oregon, USA,pp\. 1318–1326\.External Links:[Link](https://aclanthology.org/P11-1132/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- J\. Kreutzer, I\. Caswell, L\. Wang, A\. Wahab, D\. van Esch, N\. Ulzii\-Orshikh, A\. A\. Tapo, N\. Subramani, A\. Sokolov, C\. Sikasote, M\. Setyawan, S\. Sarin, S\. Samb, B\. Sagot, C\. Rivera, A\. Rios, I\. Papadimitriou, S\. Osei, P\. O\. Suarez, I\. Orife, K\. Ogueji, R\. A\. Niyongabo, T\. Q\. Nguyen, M\. Müller, A\. Müller, S\. H\. Muhammad, N\. Muhammad, A\. Mnyakeni, J\. Mirzakhalov, T\. Matangira, C\. Leong, N\. Lawson, S\. Kudugunta, Y\. Jernite, M\. Jenny, O\. Firat, B\. Dossou, S\. Dlamini, N\. de Silva, S\. Çabuk Ballı, S\. Biderman, A\. Battisti, A\. Baruwa, A\. Bapna, P\. Baljekar, I\. A\. Azime, A\. Awokoya, D\. Ataman, O\. Ahia, O\. Ahia, S\. Agrawal, and M\. Adeyemi \(2022\)Quality at a glance: an audit of web\-crawled multilingual datasets\.Transactions of the Association for Computational Linguistics10,pp\. 50–72\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00447),[Link](https://aclanthology.org/2022.tacl-1.4/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Kudugunta, I\. Caswell, B\. Zhang, X\. Garcia, D\. Xin, A\. Kusupati, R\. Stella, A\. Bapna, and O\. Firat \(2023\)MADLAD\-400: a multilingual and document\-level large audited dataset\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 67284–67296\.External Links:[Document](https://dx.doi.org/10.52202/075280-2940),[Link](https://papers.neurips.cc/paper_files/paper/2023/hash/d49042a5d49818711c401d34172f9900-Abstract-Datasets_and_Benchmarks.html)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles \(SOSP ’23\),External Links:[Link](https://arxiv.org/abs/2309.06180)Cited by:[Appendix A](https://arxiv.org/html/2607.00890#A1.SS0.SSS0.Px1.p1.1)\.
- H\. Laurençon, L\. Saulnier, T\. Wang, C\. Akiki, A\. V\. del Moral, T\. Le Scao, L\. Von Werra, C\. Mou, E\. González Ponferrada, H\. Nguyen, J\. Frohberg, and BigScience Workshop \(2022\)The BigScience ROOTS corpus: a 1\.6tb composite multilingual dataset\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/ce9e92e3de2372a4b93353eb7f3dc0bd-Abstract-Datasets_and_Benchmarks.html)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Lee, D\. Ippolito, A\. Nystrom, C\. Zhang, D\. Eck, C\. Callison\-Burch, and N\. Carlini \(2022\)Deduplicating training data makes language models better\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Dublin, Ireland,pp\. 8424–8445\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.577),[Link](https://aclanthology.org/2022.acl-long.577/)Cited by:[Appendix C](https://arxiv.org/html/2607.00890#A3.SS0.SSS0.Px1.p1.1)\.
- J\. Li, A\. Fang, G\. Smyrnis, M\. Ivgi, M\. Jordan, S\. Y\. Gadre, H\. Bansal, E\. K\. Guha, S\. S\. Keh, K\. Arora, S\. Garg, R\. Xin, N\. Muennighoff, R\. Heckel, J\. Mercat, M\. F\. Chen, S\. Gururangan, M\. Wortsman, A\. Albalak, Y\. Bitton, M\. Nezhurina, A\. Abbas, C\. Hsieh, D\. Ghosh, J\. Gardner, M\. Kilian, H\. Zhang, R\. Shao, S\. M\. Pratt, S\. Sanyal, G\. Ilharco, G\. Daras, K\. Marathe, A\. Gokaslan, J\. Zhang, K\. R\. Chandu, T\. Nguyen, I\. Vasiljevic, S\. M\. Kakade, S\. Song, S\. Sanghavi, F\. Faghri, S\. Oh, L\. Zettlemoyer, K\. Lo, A\. El\-Nouby, H\. Pouransari, A\. Toshev, S\. Wang, D\. Groeneveld, L\. Soldaini, P\. W\. Koh, J\. Jitsev, T\. Kollar, A\. Dimakis, Y\. Carmon, A\. Dave, L\. Schmidt, and V\. Shankar \(2024a\)DataComp\-LM: in search of the next generation of training sets for language models\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=CNWdWn47IE)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Li, S\. Ji, H\. Luo, and J\. Tiedemann \(2025\)Rethinking multilingual continual pretraining: data mixing for adapting LLMs across languages and resources\.External Links:2504\.04152,[Link](https://arxiv.org/abs/2504.04152)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Li, S\. Ji, T\. Mickus, V\. Segonne, and J\. Tiedemann \(2024b\)A comparison of language modeling and translation as multilingual pretraining objectives\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 15882–15894\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.888),[Link](https://aclanthology.org/2024.emnlp-main.888/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- X\. V\. Lin, T\. Mihaylov, M\. Artetxe, T\. Wang, S\. Chen, D\. Simig, M\. Ott, N\. Goyal, S\. Bhosale, J\. Du, R\. Pasunuru, S\. Shleifer, P\. S\. Koura, V\. Chaudhary, B\. O’Horo, J\. Wang, L\. Zettlemoyer, Z\. Kozareva, M\. T\. Diab, V\. Stoyanov, and X\. Li \(2021\)Few\-shot learning with multilingual language models\.CoRRabs/2112\.10668\.External Links:[Link](https://arxiv.org/abs/2112.10668),2112\.10668Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- C\. C\. Liu, F\. Koto, T\. Baldwin, and I\. Gurevych \(2024\)Are multilingual LLMs culturally\-diverse reasoners? an investigation into multicultural proverbs and sayings\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 2016–2039\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.112),[Link](https://aclanthology.org/2024.naacl-long.112/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 2511–2522\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153),[Link](https://aclanthology.org/2023.emnlp-main.153/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- P\. H\. Martins, J\. Alves, P\. Fernandes, N\. M\. Guerreiro, R\. Rei, M\. A\. Farajian, M\. Klimaszewski, D\. M\. Alves, J\. Pombal, N\. Boizard, M\. Faysse, P\. Colombo, F\. Yvon, B\. Haddow, J\. G\. C\. de Souza, A\. Birch, and A\. F\. T\. Martins \(2025\)EuroLLM\-9B: technical report\.CoRRabs/2506\.04079\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2506.04079),[Link](https://arxiv.org/abs/2506.04079)Cited by:[§3\.2](https://arxiv.org/html/2607.00890#S3.SS2.p1.1)\.
- V\. Mikhailov, T\. Enstad, D\. Samuel, H\. C\. Farsethås, A\. Kutuzov, E\. Velldal, and L\. Øvrelid \(2025\)NorEval: a Norwegian language understanding and generation evaluation benchmark\.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp\. 3495–3541\.External Links:[Link](https://aclanthology.org/2025.findings-acl.181/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.181)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1),[§5\.3](https://arxiv.org/html/2607.00890#S5.SS3.p1.1),[§6](https://arxiv.org/html/2607.00890#S6.SS0.SSS0.Px3.p1.1)\.
- T\. Naous, M\. J\. Ryan, A\. Ritter, and W\. Xu \(2024\)Having beer after prayer? measuring cultural bias in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 16366–16393\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.862),[Link](https://aclanthology.org/2024.acl-long.862/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Nezhurina, J\. Franke, T\. Nakamura, T\. Carstensen, N\. Ajroldi, V\. Komulainen, D\. Salinas, and J\. Jitsev \(2025\)Open\-sci\-ref\-0\.01: open and reproducible reference baselines for language model and dataset comparison\.CoRRabs/2509\.09009\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2509.09009),[Link](https://arxiv.org/abs/2509.09009)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2607.00890#S3.SS1.p1.1)\.
- T\. Nguyen, C\. V\. Nguyen, V\. D\. Lai, H\. Man, N\. T\. Ngo, F\. Dernoncourt, R\. A\. Rossi, and T\. H\. Nguyen \(2024\)CulturaX: a cleaned, enormous, and multilingual dataset for large language models in 167 languages\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),Torino, Italia,pp\. 4226–4237\.External Links:[Link](https://aclanthology.org/2024.lrec-main.377/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Nunes, R\. Primi, R\. Pires, R\. Lotufo, and R\. Nogueira \(2023\)Evaluating gpt\-3\.5 and gpt\-4 models on brazilian university admission exams\.External Links:2303\.17003Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- S\. Oepen, N\. Arefyev, M\. Aulamo, M\. Bañón, M\. Buljan, L\. Burchell, L\. Charpentier, P\. Chen, M\. Fedorova, O\. de Gibert, B\. Haddow, J\. Hajič, J\. Helcl, A\. Kutuzov, V\. Laippala, Z\. Li, R\. Luukkonen, B\. Malik, V\. Mikhailov, A\. Myntti, D\. O’Brien, L\. Poláková, S\. Pyysalo, G\. Ramírez\-Sánchez, J\. Siewert, P\. Stepachev, J\. Tiedemann, T\. Vahtola, D\. Variš, F\. Vitiugin, T\. Vojtěchová, and J\. Zaragoza \(2025\)HPLT 3\.0: very large\-scale multilingual resources for LLMs and MT\. mono\- and bi\-lingual data, multilingual evaluation, and pre\-trained models\.CoRRabs/2511\.01066\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2511.01066),[Link](https://arxiv.org/abs/2511.01066)Cited by:[§1](https://arxiv.org/html/2607.00890#S1.p2.1),[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1),[§5\.3](https://arxiv.org/html/2607.00890#S5.SS3.p1.1),[footnote 1](https://arxiv.org/html/2607.00890#footnote1)\.
- OpenAI \(2024\)MMMLU: multilingual massive multitask language understanding\.Note:[https://huggingface\.co/datasets/openai/MMMLU](https://huggingface.co/datasets/openai/MMMLU)Professionally translated test split of MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.00890#bib.bib53)\)across 14 languagesCited by:[§6](https://arxiv.org/html/2607.00890#S6.SS0.SSS0.Px3.p1.1)\.
- S\. Pawar, J\. Park, J\. Jin, A\. Arora, J\. Myung, S\. Yadav, F\. G\. Haznitrama, I\. Song, A\. Oh, and I\. Augenstein \(2025\)Survey of cultural awareness in language models: text and beyond\.Computational Linguistics51\(3\),pp\. 907–1004\.External Links:[Document](https://dx.doi.org/10.1162/coli.a.14),[Link](https://aclanthology.org/2025.cl-3.7/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. A\. Raffel, L\. von Werra, and T\. Wolf \(2024\)The FineWeb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 30811–30849\.External Links:[Document](https://dx.doi.org/10.52202/079017-0970),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§1](https://arxiv.org/html/2607.00890#S1.p1.1),[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.p2.2)\.
- G\. Penedo, H\. Kydlíček, A\. H\. Kargaran, and L\. von Werra \(2026\)FineTranslations\.Hugging Face\.Note:[https://huggingface\.co/datasets/HuggingFaceFW/finetranslations](https://huggingface.co/datasets/HuggingFaceFW/finetranslations)Dataset; non\-English FineWeb2 documents translated into English with Gemma3 27B; directionally opposite of MultiSynt/MT\.Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- G\. Penedo, H\. Kydlíček, V\. Sabolčec, B\. Messmer, N\. Foroutan, A\. H\. Kargaran, C\. Raffel, M\. Jaggi, L\. von Werra, and T\. Wolf \(2025\)FineWeb2: one pipeline to scale them all: adapting pre\-training data processing to every language\.CoRRabs/2506\.20920\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2506.20920),[Link](https://arxiv.org/abs/2506.20920)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Penedo, Q\. Malartic, D\. Hesslow, R\. Cojocaru, H\. Alobeidli, A\. Cappelli, B\. Pannier, E\. Almazrouei, and J\. Launay \(2023\)The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data only\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 79155–79172\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/fa3ed726cc5073b9c31e3e49a807789c-Abstract-Datasets_and_Benchmarks.html)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Pires, R\. Malaquias Junior, and R\. Nogueira \(2025\)Automatic legal writing evaluation of llms\.InProceedings of the International Conference on Artificial Intelligence and Law \(ICAIL\),Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of Machine Learning Research21\(140\),pp\. 1–67\.External Links:[Link](https://jmlr.org/papers/v21/20-074.html)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Rei, N\. M\. Guerreiro, J\. Pombal, J\. Alves, P\. Teixeirinha, A\. Farajian, and A\. F\. T\. Martins \(2026\)Tower\+: bridging generality and translation specialization in multilingual LLMs\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Note:To appearCited by:[§1](https://arxiv.org/html/2607.00890#S1.p2.1),[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2607.00890#S3.SS2.p1.1)\.
- P\. Riley, I\. Caswell, M\. Freitag, and D\. Grangier \(2020\)Translationese as a language in “multilingual” NMT\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp\. 7737–7746\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.691),[Link](https://aclanthology.org/2020.acl-main.691/)Cited by:[§1](https://arxiv.org/html/2607.00890#S1.p1.1),[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- P\. Riley, A\. Gokaslan, and N\. A\. Smith \(2021\)Sometimes we want translationese\.CoRRabs/2104\.07623\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2104.07623),[Link](https://arxiv.org/abs/2104.07623)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Romanou, N\. Foroutan, A\. Sotnikova, Z\. Chen, S\. H\. Nelaturu, S\. Singh, R\. Maheshwary, M\. Altomare, M\. A\. Haggag, I\. Schlag, M\. Fadaee, S\. Hooker, A\. Bosselut, S\. A, A\. Amayuelas, A\. H\. Amirudin, V\. Aryabumi, D\. Boiko, M\. Chang, J\. Chim, G\. Cohen, A\. K\. Dalmia, A\. Diress, S\. Duwal, D\. Dzenhaliou, D\. F\. E\. Florez, F\. Farestam, J\. M\. Imperial, S\. B\. Islam, P\. Isotalo, M\. Jabbarishiviari, B\. F\. Karlsson, E\. Khalilov, C\. Klamm, F\. Koto, D\. Krzeminski, G\. A\. de Melo, S\. Montariol, Y\. Nan, J\. Niklaus, J\. Novikova, J\. S\. Obando\-Ceron, D\. Paul, E\. Ploeger, J\. Purbey, S\. Rajwal, S\. S\. Ravi, S\. Rydell, R\. Santhosh, D\. Sharma, M\. P\. Skenduli, A\. Soltani Moakhar, B\. Soltani Moakhar, R\. Tamir, A\. K\. Tarun, A\. T\. Wasi, T\. O\. Weerasinghe, S\. Yilmaz, and M\. Zhang \(2025\)INCLUDE: evaluating multilingual language understanding with regional knowledge\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/ced46a50befedcb884ccf0cbe8c3ad23-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- H\. F\. Sayama, A\. V\. Araujo, and E\. R\. Fernandes \(2019\)FaQuAD: reading comprehension dataset in the domain of brazilian higher education\.In2019 8th Brazilian Conference on Intelligent Systems \(BRACIS\),Vol\.,pp\. 443–448\.External Links:[Document](https://dx.doi.org/10.1109/BRACIS.2019.00084)Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- M\. Shoeybi, M\. Patwary, R\. Puri, P\. LeGresley, J\. Casper, and B\. Catanzaro \(2019\)Megatron\-LM: training multi\-billion parameter language models using model parallelism\.CoRRabs/1909\.08053\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1909.08053),[Link](https://arxiv.org/abs/1909.08053)Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.p3.1)\.
- I\. Shumailov, Z\. Shumaylov, Y\. Zhao, N\. Papernot, R\. Anderson, and Y\. Gal \(2024\)AI models collapse when trained on recursively generated data\.Nature631\(8022\),pp\. 755–759\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07566-y),[Link](https://www.nature.com/articles/s41586-024-07566-y)Cited by:[Evaluation coverage, scale, and feedback risks\.](https://arxiv.org/html/2607.00890#Sx1.SS0.SSS0.Px2.p1.1)\.
- A\. K\. Singh, M\. Y\. Kocyigit, A\. Poulton, D\. Esiobu, M\. Lomeli, G\. Szilvasy, and D\. Hupkes \(2024\)Evaluation data contamination in LLMs: how do we measure it and when does it matter?\.External Links:2411\.03923,[Link](https://arxiv.org/abs/2411.03923)Cited by:[Appendix C](https://arxiv.org/html/2607.00890#A3.SS0.SSS0.Px1.p1.1)\.
- S\. Singh, A\. Romanou, C\. Fourrier, D\. I\. Adelani, J\. G\. Ngui, D\. Vila\-Suero, P\. Limkonchotiwat, K\. Marchisio, W\. Q\. Leong, Y\. Susanto, R\. Ng, S\. Longpre, S\. Ruder, W\. Ko, A\. Bosselut, A\. Oh, A\. Martins, L\. Choshen, D\. Ippolito, E\. Ferrante, M\. Fadaee, B\. Ermis, and S\. Hooker \(2025\)Global MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 18761–18799\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.919),[Link](https://aclanthology.org/2025.acl-long.919/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- D\. Su, K\. Kong, Y\. Lin, J\. Jennings, B\. Norick, M\. Kliegl, M\. Patwary, M\. Shoeybi, and B\. Catanzaro \(2025\)Nemotron\-CC: transforming Common Crawl into a refined long\-horizon pretraining dataset\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 2459–2475\.External Links:[Link](https://aclanthology.org/2025.acl-long.123/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.123)Cited by:[§1](https://arxiv.org/html/2607.00890#S1.p1.1),[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2607.00890#S3.SS1.p1.1)\.
- A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)CommonsenseQA: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),Minneapolis, Minnesota,pp\. 4149–4158\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1421),[Link](https://www.aclweb.org/anthology/N19-1421)Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Tiedemann, M\. Aulamo, D\. Bakshandaeva, M\. Boggia, S\. Grönroos, T\. Nieminen, A\. Raganato, Y\. Scherrer, R\. Vázquez, and S\. Virpioja \(2024\)Democratizing neural machine translation with OPUS\-MT\.Language Resources and Evaluation58,pp\. 713–755\.External Links:[Document](https://dx.doi.org/10.1007/s10579-023-09704-w),[Link](https://doi.org/10.1007/s10579-023-09704-w)Cited by:[§1](https://arxiv.org/html/2607.00890#S1.p2.1),[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2607.00890#S3.SS2.p1.1)\.
- J\. Tiedemann \(2009\)News from opus\-a collection of multilingual parallel corpora with tools and interfaces\.InRecent advances in natural language processing,Vol\.5,pp\. 237–248\.Cited by:[Appendix A](https://arxiv.org/html/2607.00890#A1.SS0.SSS0.Px4.p1.1)\.
- J\. Tiedemann \(2012\)Parallel data, tools and interfaces in opus\.\.InLrec,Vol\.2012,pp\. 2214–2218\.Cited by:[Appendix A](https://arxiv.org/html/2607.00890#A1.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Tiedemann \(2020\)The tatoeba translation challenge – realistic data sets for low resource and multilingual MT\.InProceedings of the Fifth Conference on Machine Translation,Online,pp\. 1174–1182\.External Links:[Link](https://aclanthology.org/2020.wmt-1.139/),[Document](https://dx.doi.org/10.18653/v1/2020.wmt-1.139)Cited by:[Appendix A](https://arxiv.org/html/2607.00890#A1.SS0.SSS0.Px4.p1.1),[§3\.2](https://arxiv.org/html/2607.00890#S3.SS2.p3.1)\.
- G\. Urbizu, I\. San Vicente, X\. Saralegi, and A\. Corral \(2023\)Not enough data to pre\-train your language model? MT to the rescue\!\.InFindings of the Association for Computational Linguistics: ACL 2023,Toronto, Canada,pp\. 3826–3836\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.235),[Link](https://aclanthology.org/2023.findings-acl.235/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- E\. Vanmassenhove, D\. Shterionov, and A\. Way \(2021\)Machine translationese: effects of algorithmic bias on linguistic complexity in machine translation\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,Online,pp\. 2203–2213\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.eacl-main.188),[Link](https://aclanthology.org/2021.eacl-main.188/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- D\. J\. Velasco and M\. T\. Roque \(2025\)Scaling, simplification, and adaptation: lessons from pretraining on machine\-translated text\.InProceedings of the 5th Workshop on Multilingual Representation Learning,Suzhou, China,pp\. 612–630\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.mrl-main.40),[Link](https://aclanthology.org/2025.mrl-main.40/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- P\. Verga, S\. Hofstatter, S\. Althammer, Y\. Su, A\. Piktus, A\. Arkhangorodsky, M\. Xu, N\. White, and P\. Lewis \(2024\)Replacing judges with juries: evaluating LLM generations with a panel of diverse models\.CoRRabs/2404\.18796\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2404.18796),[Link](https://arxiv.org/abs/2404.18796)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- V\. Volansky, N\. Ordan, and S\. Wintner \(2015\)On the features of translationese\.Digital Scholarship in the Humanities30\(1\),pp\. 98–118\.External Links:[Document](https://dx.doi.org/10.1093/llc/fqt031),[Link](https://academic.oup.com/dsh/article/30/1/98/350113)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- J\. Wang, Y\. Lu, M\. Weber, M\. Ryabinin, D\. I\. Adelani, Y\. Chen, R\. Tang, and P\. Stenetorp \(2025\)Multilingual language model pretraining using machine\-translated data\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 28087–28107\.Note:TransWebLLM / TransWeb\-Edu in 9 languages; EMNLP publication of arXiv:2502\.13252\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1426),[Link](https://aclanthology.org/2025.emnlp-main.1426/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Wang, Y\. Lu, M\. Weber, M\. Ryabinin, Y\. Chen, R\. Tang, and P\. Stenetorp \(2024a\)Multilingual pretraining using a large corpus machine\-translated from a single source language\.CoRRabs/2410\.23956\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2410.23956),[Link](https://arxiv.org/abs/2410.23956)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px3.p1.1)\.
- L\. Wang, N\. Yang, X\. Huang, L\. Yang, R\. Majumder, and F\. Wei \(2024b\)Multilingual e5 text embeddings: a technical report\.arXiv preprint arXiv:2402\.05672\.Cited by:[Appendix G](https://arxiv.org/html/2607.00890#A7.SS0.SSS0.Px1.p1.1)\.
- A\. A\. Weber, K\. Thellmann, J\. Ebert, N\. Flores\-Herr, J\. Lehmann, M\. Fromm, and M\. Ali \(2024\)Investigating multilingual instruction\-tuning: do polyglot models demand for multilingual instructions?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 20829–20855\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1159),[Link](https://aclanthology.org/2024.emnlp-main.1159/)Cited by:[§6](https://arxiv.org/html/2607.00890#S6.SS0.SSS0.Px3.p1.1)\.
- G\. Wenzek, M\. Lachaux, A\. Conneau, V\. Chaudhary, F\. Guzmán, A\. Joulin, and E\. Grave \(2020\)CCNet: extracting high quality monolingual datasets from web crawl data\.InProceedings of the Twelfth Language Resources and Evaluation Conference,Marseille, France,pp\. 4003–4012\.External Links:[Link](https://aclanthology.org/2020.lrec-1.494/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Wu, W\. Wang, S\. Liu, H\. Yin, X\. Wang, Y\. Zhao, C\. Lyu, L\. Wang, W\. Luo, and K\. Zhang \(2025\)The bitter lesson learned from 2,000\+ multilingual benchmarks\.CoRRabs/2504\.15521\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.15521),[Link](https://arxiv.org/abs/2504.15521)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- W\. Xuan, R\. Yang, H\. Qi, Q\. Zeng, Y\. Xiao, A\. Feng, D\. Liu, Y\. Xing, J\. Wang, F\. Gao, J\. Lu, Y\. Jiang, H\. Li, X\. Li, K\. Yu, R\. Dong, S\. Gu, Y\. Li, X\. Xie, F\. Juefei\-Xu, F\. Khomh, O\. Yoshie, Q\. Chen, D\. Teodoro, N\. Liu, R\. Goebel, L\. Ma, E\. Marrese\-Taylor, S\. Lu, Y\. Iwasawa, Y\. Matsuo, and I\. Li \(2025\)MMLU\-ProX: a multilingual benchmark for advanced large language model evaluation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 1513–1532\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.79),[Link](https://aclanthology.org/2025.emnlp-main.79/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- L\. Xue, N\. Constant, A\. Roberts, M\. Kale, R\. Al\-Rfou, A\. Siddhant, A\. Barua, and C\. Raffel \(2021\)MT5: a massively multilingual pre\-trained text\-to\-text transformer\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Online,pp\. 483–498\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.41),[Link](https://aclanthology.org/2021.naacl-main.41/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.CoRRabs/2505\.09388\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.09388),[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3\.2](https://arxiv.org/html/2607.00890#S3.SS2.p1.1)\.
- Y\. Yang, Y\. Zhang, C\. Tar, and J\. Baldridge \(2019\)PAWS\-X: A Cross\-lingual Adversarial Dataset for Paraphrase Identification\.InProc\. of EMNLP,Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- B\. Yao, Y\. Jiang, D\. Yang, and J\. Hu \(2024a\)Data contamination can cross language barriers\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 17864–17875\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.990),[Link](https://aclanthology.org/2024.emnlp-main.990/)Cited by:[Appendix C](https://arxiv.org/html/2607.00890#A3.SS0.SSS0.Px1.p1.1)\.
- B\. Yao, X\. Y\. Li, N\. Muennighoff, D\. Radev, P\. Viswanath, and D\. Kiela \(2024b\)Benchmarking machine translation with cultural awareness\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 13078–13096\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.726),[Link](https://aclanthology.org/2024.emnlp-main.726/)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Cited by:[§4\.1](https://arxiv.org/html/2607.00890#S4.SS1.SSS0.Px1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,External Links:[Link](https://arxiv.org/abs/2306.05685)Cited by:[§2](https://arxiv.org/html/2607.00890#S2.SS0.SSS0.Px4.p1.1)\.

## Appendix ATranslation Pipeline Details

#### Orchestration toolkit \(LLM pipeline\)\.

For the LLM\-based translation runs of Section[3\.3](https://arxiv.org/html/2607.00890#S3.SS3), we use an open\-source Slurm\-native orchestration toolkit888[github\.com/ellamind/inference\-hive](https://github.com/ellamind/inference-hive)that scales inference workloads across hundreds to thousands of GPUs while maintaining linear throughput\. The toolkit deploys multiple distributed inference servers in parallel: a single YAML configuration file specifies the cluster, inference\-server, and data settings, and the toolkit generates the required Slurm job scripts, manages resource allocation, and coordinates distributed execution\. It provides error handling with automatic retries and resumption, progress and throughput monitoring, built\-in support for large\-scale loading of Parquet input files, and a flexible inference server backend\. We ran all inference with vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2607.00890#bib.bib106)\)\.

#### LLM runs: per\-model throughput\.

Across batched production runs \(4 GPUs per inference server\),Tower\+9B sustained approximately 1,000 output tokens per second per GPU, andTower\+72B approximately 400 output tokens per second per GPU\. The 9B variant therefore produces output tokens approximately2\.6×2\.6\\timesfaster than the 72B variant per GPU\-second; the per\-request advantage is wider \(approximately3\.2×3\.2\\times\) because the 72B model supports a longer maximum sequence length, allowing us to feed longer source documents per request and yielding roughly 26% longer translated outputs on average\.

#### LLM runs: aggregate compute, energy and carbon footprint\.

The LLM\-based translation runs were carried out on the Leonardo \(CINECA\) supercomputer\. Aggregate GPU consumption is approximately 3\.1 million A100 GPU\-hours\. We estimate the corresponding energy and carbon footprint following the methodology used in CINECA project reporting: each A100 on Leonardo draws approximately 440 W under load, giving 1,379,638 kWh of consumed energy; applying the 2025 Italian electricity\-grid carbon intensity of 358 g CO2\-eq/kWh yields approximately 493\.9 t CO2\-eq\. For reference, this energy budget corresponds to roughly 16% of the annual generation of an average wind turbine \(3 MW nameplate capacity, 2,800 equivalent full\-load hours per year,≈\\approx8\.4 GWh/year\)\. These figures cover only the LLM\-based translation pipeline; the corresponding energy and carbon estimates for the OPUS\-MT/HPLT\-MT runs on LUMI \(AMD MI250x\) are reported alongside the NMT shard statistics below\.

#### NMT models: training data and inference setup\.

The OPUS\-MT and HPLT\-MT models we use are decoded with Marian\-NMT\(Junczys\-Dowmuntet al\.,[2018](https://arxiv.org/html/2607.00890#bib.bib17)\)at beam size 4 on single\-node jobs allocating two AMD MI250x GPUs each, on the LUMI supercomputer\. All models are trained on OPUS data\(Tiedemann,[2009](https://arxiv.org/html/2607.00890#bib.bib14),[2012](https://arxiv.org/html/2607.00890#bib.bib15)\)using the Tatoeba Translation Challenge compilation\(Tiedemann,[2020](https://arxiv.org/html/2607.00890#bib.bib13)\), augmented with back\-translations of Wikimedia content\. None of the NMT models are fine\-tuned for specific tasks; we use them off\-the\-shelf without further data normalisation or filtering\. The final list of selected NMT models is published together with the dataset card\. The sentence\-splitter pre\-processing step greedily merges segments shorter than 256 characters and splits sentences longer than 1024 characters before feeding them to the decoder; this length normalisation matters because NMT batching benefits sharply from uniform input length\.

#### NMT runs: sharding, runtimes, energy and per\-GPU power\.

The corpus was split into 100 shards per language of approximately 30 million sentence\-aligned lines \(roughly 700 million space\-separated tokens each\)\. Per\-shard runtime averaged about 23 hours for transformer\-base models \(∼\\sim75M parameters\) and about 30 hours for transformer\-big models \(∼\\sim230M parameters\), with sustained GPU utilisation between 85% and 95%\. Total energy consumption for one shard translated with transformer\-base models is roughly 12 kWh, at an average of 260–270 W per GPU\. For transformer\-big models, total energy consumption is roughly double at 24–26 kWh, at an average of 360–370 W per GPU\. The pattern suggests that further batching and inference optimisation could improve GPU utilisation for the smaller models and yield additional speed\-ups\.

## Appendix BReference Model Hyperparameters

#### Compute setup\.

All reference\-model training and evaluation runs were carried out on the LUMI supercomputer \(16 nodes of 4 AMD MI250x GPUs per node\), with one exception: the Nemotron\-CC decontamination control experiment of Appendix[C](https://arxiv.org/html/2607.00890#A3)was run on Leonardo \(CINECA\) with NVIDIA A100 nodes\.

Table 1:Reference\-model hyperparameters used in the main pre\-training runs\.

## Appendix CDecontamination: Methodology and Full Tables

#### Motivation and prior work\.

Large web\-derived corpora can overlap with evaluation benchmarks through exact duplicates, near\-duplicates, paraphrases, or translated variants\. The GPT\-3 family established n\-gram\-based decontamination as a standard pre\-training practice\(Brownet al\.,[2020](https://arxiv.org/html/2607.00890#bib.bib30)\), and deduplication has been shown to reduce memorization and improve downstream model quality\(Leeet al\.,[2022](https://arxiv.org/html/2607.00890#bib.bib66)\)\. Recent work emphasizes that contamination is difficult to define uniformly and that its impact depends on task, model, and training recipe\(Singhet al\.,[2024](https://arxiv.org/html/2607.00890#bib.bib88)\)\. For multilingual settings the problem is sharper: contamination can cross language boundaries, and translated benchmark items can evade simple text\-overlap detection while still inflating evaluation scores\(Yaoet al\.,[2024a](https://arxiv.org/html/2607.00890#bib.bib104)\)\. We therefore decontaminate the English source documents before translation, report overlap statistics per benchmark, and run a controlled training comparison to verify that removing flagged documents has no measurable downstream effect\.

#### Released corpus\.

The released MultiSynt/MT corpus itself is not decontaminated; the decontamination described here is applied to the English source documents used as input to the controlled training comparison below\.

#### Pipeline stages\.

We use the NeMo Curator decontamination pipeline, which constructs n\-gram representations of downstream task datasets and removes any matching content from the training corpus in three stages\.Indexing:each benchmark dataset is normalised and tokenised, and n\-grams of size 8–13 are extracted from the test split \(or the validation split, where no test split is available\), following the text formatting in lm\-evaluation\-harness, to build a compact n\-gram index\.Matching:each source document is processed through the same normalisation/tokenisation/n\-gram\-extraction pipeline and its n\-grams are compared against the benchmark index to identify overlaps\.Removal:source documents whose matched n\-grams occur fewer than 10 times in the corpus are treated as contamination signals and removed in full\.

The indexed evaluation suite \(see Table[2](https://arxiv.org/html/2607.00890#A3.T2)for details\) spans code, common sense reasoning, instruction following, language understanding, linguistic competence, mathematics, reasoning, translation, and world knowledge\.

#### Match counts and rates\.

After applying the pipeline to the source Nemotron\-CC documents used as input to translation, a total of 34,969 documents per language \(0\.0002%\) are flagged as contaminated\. Per\-benchmark match counts and contamination rates are reported in Table[2](https://arxiv.org/html/2607.00890#A3.T2)\. Contamination is unevenly distributed: the largest absolute contribution comes from BoolQ, followed at lower rates by HellaSwag, MMLU, and MTBench\. Even where the absolute number of matches is large, the corresponding percentages remain very low, indicating that contamination is sparse relative to corpus size\.

#### Controlled training experiment\.

To verify that the residual contamination has no measurable downstream effect, we train a 0\.6B\-parameter reference model on the chosen Nemotron\-CC source slice with and without decontamination, under identical hyperparameters and a 100B\-token budget, and evaluate 0\-shot on the Finetasks suite \(ARC\-easy, ARC\-challenge, PIQA, HellaSwag, OpenBookQA\)\. The two learning curves track each other throughout training \(Figure[7](https://arxiv.org/html/2607.00890#A3.F7)\), with no consistent advantage for either setting\.

![Refer to caption](https://arxiv.org/html/2607.00890v1/figures/nemotron-decont-1.png)Figure 7:Controlled training experiment on the Nemotron\-CC source data, with and without decontamination\. A 0\.6B\-parameter reference model is trained for 100B tokens and evaluated 0\-shot on the Finetasks suite \(ARC\-easy, ARC\-challenge, PIQA, HellaSwag, OpenBookQA\)\. Curves track each other throughout training, indicating no measurable downstream effect from decontamination\.TaskCategoryPassagesSize \(MB\)MatchesContamination%HumanevalCode32810\.6419201\.3e\-05MBPPCode8784\.8529322\.1e\-05ARC ChallengeCommon sense3,5161\.14151097\.0e\-05ARC EasyCommon sense7,1281\.99721571\.0e\-04BoolQCommon sense3,2702\.0204232751\.5e\-02CommonsenseQACommon sense1,2210\.263500COPACommon sense1,1000\.142200HellaSwagCommon sense10,04210\.669930342\.0e\-03LambadaCommon sense5,1531\.6316281\.8e\-05OpenBookQACommon sense2,0000\.385116\.5e\-07PIQACommon sense5,5121\.417421\.3e\-06XStoryClozeCommon sense7,5552\.540700XWinogradCommon sense2,6710\.3142312\.0e\-05AlpacaEvalInstruction1,5930\.17201701\.1e\-04do\_not\_answerInstruction9393\.816116\.5e\-07IFEvalInstruction5410\.2743593\.8e\-05m\-ArenaHardInstruction6,0003\.3170149\.0e\-06MT\-Bench\-XInstruction4000\.2720442\.8e\-05MTBenchInstruction1,3881\.800031272\.0e\-03Multi\-IFEvalInstruction22,50534\.377421\.3e\-06ToxigenInstruction9400\.3500161\.0e\-05PAWS\-XLang\. underst\.14,0003\.47581368\.8e\-05MultiBlimpLinguistic60,69392\.10141358\.7e\-05AIME 25Math300\.011800GSM8KMath1,3190\.6815117\.1e\-06Hendrycks\_MATHMath5,0032\.000010656\.9e\-04PolyMathMath3,0000\.6709644\.1e\-05XNLIReasoning27,4495\.2555312\.0e\-05FLORES\-200Translation37,44410\.3435513\.3e\-05BelebeleWorld Knowl\.29,70027\.51755133\.3e\-04GPQAWorld Knowl\.5464\.017900MMLUWorld Knowl\.14,0426\.651428411\.8e\-03Table 2:Benchmark evaluation suite grouped by capability category, showing approximate passage count and size of the splits used for decontamination, total number of contaminated documents containing at least one benchmark n\-gram that failed the frequency threshold and ratio of contaminated documents to the full source corpus per benchmark\.

## Appendix DPer\-Language Resource Statistics

Table[3](https://arxiv.org/html/2607.00890#A4.T3)reports per\-language Gemma\-3 token counts for each MultiSynt/MT translation system and for the HPLT 3\.0 native baseline, covering all 36 languages of the released corpus\.999Albanian is listed assqiin MultiSynt/MT \(the ISO 639\-3 macrolanguage code, matching OPUS\-MT/HPLT\-MT’s output language label\) and asals\_Latnin HPLT 3\.0 \(Tosk Albanian, the basis of Standard Albanian and the FLORES/NLLB convention for the language\)\. The two refer to the same standard written language in practice\.Figure[1](https://arxiv.org/html/2607.00890#S1.F1)in Section[3\.4](https://arxiv.org/html/2607.00890#S3.SS4)visualizes a 24\-language representative subset drawn from this table; the table itself gives the full picture, including languages where Tower\+ 9B or 72B was not run and languages without an HPLT 3\.0 baseline\. For consistency with Figure[1](https://arxiv.org/html/2607.00890#S1.F1), the MultiSynt/MT\-to\-HPLT 3\.0 ratio is computed against the per\-language maximum across MultiSynt/MT systems \(effectively OPUS\-MT/HPLT\-MT, whose output exceeds Tower\+ on every language\)\. Sums of the per\-system columns recover the corpus totals reported in Section[3\.4](https://arxiv.org/html/2607.00890#S3.SS4): 1\.7T from Tower\+ 9B \(16 languages\), 0\.5T from Tower\+ 72B \(5 languages\), 4\.8T from OPUS\-MT/HPLT\-MT \(36 languages\), for approximately 7\.1T tokens across all system outputs and approximately 4\.8T unique target\-language tokens\.

Table 3:Per\-language Gemma\-3 token counts \(billions\) for each MultiSynt/MT translation system and the HPLT 3\.0 native baseline, for the 36 languages of MultiSynt/MT\. Rows sorted by HPLT 3\.0 token count ascending\. The MS/HPLT column gives the ratio of the per\-language MultiSynt/MT maximum \(always OPUS\-MT/HPLT\-MT, whose output exceeds Tower\+ on every language\) to HPLT 3\.0\. The Total row sums each column; the Total MS/HPLT ratio is the sum of MultiSynt/MT maxima divided by the sum of HPLT 3\.0 tokens\. Tower\+ 9B was run on 16 languages, Tower\+ 72B on a 5\-language high\-resource subset, and OPUS\-MT/HPLT\-MT on all 36 languages\.
## Appendix EHuman MT evaluation: per\-language breakdown

Figure[8](https://arxiv.org/html/2607.00890#A5.F8)shows the per\-language breakdown of the human evaluation summarized in Section[3\.2](https://arxiv.org/html/2607.00890#S3.SS2)and Figure[3](https://arxiv.org/html/2607.00890#S3.F3)\.

#### Protocol\.

For each language, a single native speaker of the target language rated translations of the same 10 English source documents from each of the six candidate systems, blind to which system produced each translation; because the source documents are held constant across systems within a language, the within\-language comparison among systems is paired\. Annotators were instructed to focus primarily on target\-language fluency and to additionally penalize specific model failures \(hallucinations, repetitions, extra text not present in the source, and missing text\); machine\-translation outputs that were truncated due to inference length constraints were flagged in the guidelines and were exempted from penalty\. Annotators recorded each judgment on a continuous slider mapped to a 0–100 scale, with the endpoints anchored as*very poor translation*\(0\) and*excellent translation*\(100\); the recorded scores are aggregated into five 20\-point bins \(0–20, 21–40, 41–60, 61–80, 81–100\) and reported on a 1–5 Likert\-style scale \(1 = bad, 5 = excellent\)\.

#### Results\.

Tower\+is rated highest in each of the seven evaluated languages; the relative ordering of the other systems varies by language\. Czech and Finnish, the two morphologically richest languages in the evaluated set, receive the lowest scores across all systems, includingTower\+; on the higher\-resource Romance languages \(French, Italian, Spanish\) the gap betweenTower\+and the general\-purpose LLMs narrows substantially, while OPUS\-MT remains a clear step below the LLM\-based systems\.

#### Comparison to WMT25\.

Recent WMT25 rankings\(Kocmiet al\.,[2025](https://arxiv.org/html/2607.00890#bib.bib60)\)placeTower\+9B and 72B mid\-pack against frontier proprietary systems \(e\.g\., Gemini\-2\.5\-Pro, GPT\-4\.1, DeepSeek\-V3\) and against the strongest constrained submissions\. That evaluation, however, operates on paragraph\-sized segments of approximately 100 words drawn from news, social media, speech transcripts, and literary text, rather than on the full web documents that constitute the Nemotron\-CC HQ source distribution; several of the higher\-ranked systems are also closed\-weights or otherwise infeasible to deploy at the 100B\-token source scale on the open compute available for this work\.

![Refer to caption](https://arxiv.org/html/2607.00890v1/x7.png)Figure 8:Per\-language breakdown of the human evaluation of MT quality \(cf\. Figure[3](https://arxiv.org/html/2607.00890#S3.F3)\)\. Languages are arranged along the x\-axis by mean quality across all six systems \(hardest on the left, easiest on the right\); the per\-system overall mean across the seven languages is shown next to each legend entry\.Tower\+is rated highest in every language; per\-language deviations among the other systems are detailed in the bars\.

## Appendix FFluency LLM\-judge: sanity check

Figure[9](https://arxiv.org/html/2607.00890#A6.F9)reports the sanity check referenced in Section[5\.1](https://arxiv.org/html/2607.00890#S5.SS1): the off\-the\-shelf Qwen 2\.5 series at five sizes from 0\.5B to 14B parameters, evaluated by the same LLM\-judge protocol against the monolingual HPLT 2\.0 1\.7B baseline\. Winrate against the baseline rises monotonically with Qwen model size across all evaluated languages, confirming that the judge tracks generation quality and that the MT\-variant ranking reported in Figure[5](https://arxiv.org/html/2607.00890#S5.F5)reflects real differences rather than judge noise\.

![Refer to caption](https://arxiv.org/html/2607.00890v1/x8.png)Figure 9:Sanity check: fluency LLM\-judge winrate of the Qwen 2\.5 model series \(0\.5B–14B parameters\) against monolingual HPLT 2\.0 1\.7B baselines trained on native data\. Winrate rises monotonically with model size across all evaluated languages, confirming that the judge tracks generation quality\. The dashed line at 0\.5 marks parity with the native baseline\.
## Appendix GEmbedding\-space coverage analysis

This appendix reports the embedding\-space benchmark\-alignment diagnostic summarized in Section[5\.2](https://arxiv.org/html/2607.00890#S5.SS2)\. We use it as a descriptive check on benchmark\-adjacent coverage rather than as a causal test of the gains in Section[4\.2](https://arxiv.org/html/2607.00890#S4.SS2)\. The analysis measures which data source aligns more closely with benchmark regions in a shared embedding space\.

#### Embedding space\.

We map each document into an embedding space usingintfloat/multilingual\-e5\-large\-instruct\(Wanget al\.,[2024b](https://arxiv.org/html/2607.00890#bib.bib107)\), producing one vector per document\.

Figure[10](https://arxiv.org/html/2607.00890#A7.F10)illustrates benchmark\-adjacent coverage for Spanish ARC\-Challenge – the benchmark that deviates most from parity in the main kNN figure \(Figure[11](https://arxiv.org/html/2607.00890#A7.F11)\) – by partitioning the joint pre\-training embedding space intok=2,000k=2\{,\}000clusters using k\-means and assigning each benchmark example to its nearest cluster\. The two distributions partially overlap, but MultiSynt/MT covers several regions that are essentially unpopulated by HPLT 2\.0 \(and vice versa\); clusters that contain a benchmark item tend to lie in MultiSynt/MT\-denser regions, with a mean MultiSynt/MT share of0\.720\.72versus0\.470\.47for clusters that do not\. This is consistent with the per\-language kNN ratio of0\.800\.80for Spanish ARC\-Challenge in Figure[11](https://arxiv.org/html/2607.00890#A7.F11)\.

![Refer to caption](https://arxiv.org/html/2607.00890v1/x9.png)Figure 10:t\-SNE projection of 2,000 k\-means cluster centroids of the joint HPLT 2\.0 \+ MultiSynt/MT pre\-training embedding space for Spanish, colored by the per\-cluster share of MultiSynt/MT documents \(blue: HPLT 2\.0\-dominated; orange: MultiSynt/MT\-dominated\)\. Clusters that contain a Spanish ARC\-Challenge benchmark item are outlined in black and tend to concentrate in MultiSynt/MT\-denser regions of the space\.
#### kNN coverage\.

To quantify this effect across benchmarks and languages, for each benchmark example, we retrieve its top\-kknearest neighbors from the union of native \(HPLT 2\.0\) and synthetic \(MultiSynt/MT\) pre\-training documents under cosine similarity, and report the average fractionksynthetic/kk\_\{\\mathrm\{synthetic\}\}/kacross benchmark items fork=20k=20\. A value of0\.50\.5indicates equal local density from the two sources; values above0\.50\.5indicate denser synthetic coverage of benchmark\-adjacent regions\.

The detailed results are shown in Figure[11](https://arxiv.org/html/2607.00890#A7.F11)\. Across all benchmarks, the mean values over the available languages indicate that the nearest documents are more often from MultiSynt/MT than from the native HPLT 2\.0 data\. HellaSwag is the only benchmark for which some languages show a higher fraction of adjacent native documents, though its mean MultiSynt/MT fraction is still 0\.53\. The other five benchmarks show a higher MultiSynt/MT fraction for all available languages\. ARC\-Challenge deviates most from the parity line, with a MultiSynt/MT fraction of 0\.82\. These results suggest that translated data increases the coverage of benchmark\-adjacent regions, potentially bringing in content that is otherwise underrepresented in the native datasets\. However, we only conducted a descriptive study and did not account for causality of any potential gains\.

![Refer to caption](https://arxiv.org/html/2607.00890v1/x10.png)Figure 11:Embedding\-space coverage analysis\. For each benchmark, the fraction of MultiSynt/MT documents among the top\-kkembedding neighbors of benchmark items is shown for every available language \(one dot per language;nnin row labels\)\. Black vertical ticks mark the per\-benchmark mean across languages; the dashed line is parity at0\.50\.5\. Six of seven benchmarks sit fully right of parity: their benchmark items have more MultiSynt/MT documents than HPLT 2\.0 documents among their nearest embedding neighbors\. HellaSwag is the lone exception, with five Latin\-script languages on the HPLT 2\.0 side\.