当你不知道如何评估时，该如何评估？为基础语言模型基准测试选择可靠的评估配置

# How to Evaluate When You Don’t Know How to Evaluate?Choosing Reliable Setups for Base Language Model Benchmarking
Source: [https://arxiv.org/html/2610.19185](https://arxiv.org/html/2610.19185)
rmTeXGyreTermesX

David SamuelJindřich HelclAffiliation:Andrey KutuzovStephan OepenAffiliation:Department of InformaticsAffiliation:University of OsloAffiliation:\{shenbinq, davisamu, jindrich, andreku, oe\}@ifi\.uio\.no

###### Abstract

Evaluation of large language models \(LLMs\) remains largely pre\-scientific: it involves many design decisions, with little guidance on how to make them\. These choices can substantially affect measured performance and, consequently, decisions about subsequent pretraining\. To identify effective evaluation setups, we investigate 8 aspects, such as task formulation or the number of in\-context examples\. We compare the setups using three widely\-used signal criteria, repurposed as normative meta\-measures: monotonicity, non\-randomness, and ranking consistency of the evaluation scores across intermediate checkpoints\. In an early pretraining scenario, we evaluate these setups on 26 monolingual 2B\-parameter models trained on different types of data, using 80 tasks in four languages\. We also examine the setups on three larger LLMs \(≥\\geq7B\) in the same four languages and explore how the preferred setups change with model scale\. Our results yield concrete recommendations: some choices, such as the use of multiple prompts, are consistently useful, whereas others, such as task formulation, depend on model scale\. More generally, our framework serves as a recipe for the choice of evaluation setups in new scenarios\.

## 1Introduction

Figure 1:Average score of six Norwegian 2B models, pretrained on different corpora under two evaluation setups for NorEval\([Mikhailov et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib10)\)classification tasks\. Both use 5\-shot prompting and the maximum over multiple prompts\.Setup A: random\-baseline rescaling, mean over task categories, cloze formulation \(CF\), no log\-probability normalization, accuracy\.Setup B:z\-score rescaling, mean over tasks, multiple\-choice formulation \(MCF\), character normalization, answer probability\. At the end of training, the two setups rank the models largely in reverse order \(Kendall’sτ=−0\.6\\tau=\-0\.6\)\. MADLAD\-400 ranks highest under setup A but lowest under setup B, whereas the two machine\-translated corpora, OPUS and Tower\-9B, shift from the bottom to the top of the ranking\. The choice of evaluation setup can lead to different conclusions about which training corpus is preferable\.Designing evaluation of large language models \(LLMs\) involves a number of decisions about different aspects, such as the selection of benchmarks, prompting strategies, metrics, and their aggregation\([Liang et al\., 2023](https://arxiv.org/html/2610.19185#bib.bib32);[Alzahrani et al\., 2024](https://arxiv.org/html/2610.19185#bib.bib33)\)\. Moreover, these decisions often depend on various circumstances, such as the model architecture and size, its training data, as well as the specific stage of the model training\([Magnusson et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib34)\)\. The choices made in this process at a given stage of development, such as early pretraining, affect subsequent decisions in further model development \(e\.g\. the training data composition for the next training phase\), which in turn directly affect the final model performance\.

In this meta\-study, we look at such crossroads in LLM development and give advice on how to select an evaluation setup that extracts the most signal from the intermediate model results\. Early pretraining will serve as our example situation: consider a model developer deciding which dataset to use to train their model\. Relying on scaling laws\([Kaplan et al\., 2020](https://arxiv.org/html/2610.19185#bib.bib30)\), they train a set of small proxy models on the candidate datasets and pick the dataset whose proxy performs the best\.

The challenge lies in how we extract the signal from the model performance\. In two case studies, we demonstrate that seemingly minor changes to the evaluation setup can lead to very different development decisions as shown in[Figure1](https://arxiv.org/html/2610.19185#S1.F1)\. We investigate this issue during early pretraining using intermediate checkpoints of small models, and examine its generalizability to larger model scales using three pretrained multilingual LLMs\.

##### Contributions\.

First, building on techniques from prior work\([Madaan et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib2);[Gu et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib9);[Penedo et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib19)\), we develop a meta\-evaluation framework to identify useful evaluation setups\. The framework combines three normative meta\-measures, each of them quantifying one desirable property of the signal from running the evaluations:*monotonicity*\(performance improving with training progress\),*non\-randomness*\(clear separation from a random baseline\), and*ranking consistency*\(stable relative rankings of models in later training stages\)\.

Second, we conduct a systematic evaluation of eight aspects of the evaluation setup across checkpoints of 26 monolingual 2B models and 80 tasks in four languages \(Norwegian, Finnish, French, and Spanish\)\. As part of this study, we extend 14 Norwegian multiple\-choice tasks in NorEval\([Mikhailov et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib10)\)with additional task formulations, log\-likelihood normalization methods, and soft probabilistic metrics, while largely retaining existing task implementations for the other languages\.

Third, we identify evaluation choices that generalize across languages and model scales, as well as those that depend on the evaluation context\. Multiple prompts and 5\-shot prompting provide consistently informative signals, whereas the relative effectiveness of task formulations and metrics changes with model scale: cloze formulation and answer probability are preferred for smaller models, while multiple\-choice formulation and accuracy become increasingly competitive for larger models\. We validate these findings on three larger multilingual LLMs\. Our framework offers a systematic approach to selecting useful evaluation setups, enabling more reliable decisions during pretraining\.

## 2Related Work

##### Meta\-measures and task selection\.

To our best knowledge, the notion of “meta\-measuring” LLM evaluation was first used informally to select tasks for the DCLM and FineWeb suites\([Li et al\., 2024](https://arxiv.org/html/2610.19185#bib.bib7);[Penedo et al\., 2024](https://arxiv.org/html/2610.19185#bib.bib6)\)\. FineTasks\([Kydlíček et al\., 2024](https://arxiv.org/html/2610.19185#bib.bib5)\)subsequently formalized four meta\-measures:monotonicity, independently investigated around the same time by[Madaan et al\. \(2025\)](https://arxiv.org/html/2610.19185#bib.bib2),non\-randomness,ranking consistencyandsignal\-to\-noise ratio\. We adopt three of them, following their definitions in FineWeb2\([Penedo et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib19),§\\lx@sectionsignA\.5\.1\)\. These measures have been applied in several task\-selection studies\([Oepen et al\., 2026](https://arxiv.org/html/2610.19185#bib.bib14);[Kytöniemi et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib11);[Alrashed and Orabona, 2026](https://arxiv.org/html/2610.19185#bib.bib4);[Yagoubi et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib1)\)\. Unlike our work, they focus on selectingwhich tasksto include rather than systematically selecting the evaluation setup\.

##### Evaluation setups and benchmarks\.

OLMES\([Gu et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib9)\)standardizes formulation, normalization and few\-shot examples for English multiple\-choice tasks, but chooses them by convention and by the accuracy they yield, not by any explicit signal\-quality criteria\.[Madaan et al\. \(2025\)](https://arxiv.org/html/2610.19185#bib.bib2)and[Heineman et al\. \(2025a\)](https://arxiv.org/html/2610.19185#bib.bib3)investigate evaluation reliability, showing that continuous metrics, cloze formulation and checkpoint merging reduce noise, but primarily focus on individual evaluation setup choices\. In contrast, we compare complete setups, jointly over eight aspects and three meta\-measures\.

Such a comparison needs benchmarks that expose these choices, but most existing benchmarks provide limited variation in prompts, formulations, metrics, or decoding strategies\. NorEval\([Mikhailov et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib10)\)provides multiple prompts per task, and FIN\-bench\-v2\([Kytöniemi et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib11)\)in addition contains cloze and multiple\-choice formulations with five prompts each\. We extend NorEval with additional formulations, log\-probability normalizations, probabilistic metrics and sampling \(§\\lx@sectionsign[3\.3](https://arxiv.org/html/2610.19185#S3.SS3)\) to study all eight aspects\.

## 3Experimental Setup

This section introduces the aspects and options of the evaluation setup \(§\\lx@sectionsign[3\.1](https://arxiv.org/html/2610.19185#S3.SS1)\), the meta\-measures for evaluating the choice of evaluation setup \(§\\lx@sectionsign[3\.2](https://arxiv.org/html/2610.19185#S3.SS2)\), and the benchmarks \(§\\lx@sectionsign[3\.3](https://arxiv.org/html/2610.19185#S3.SS3)\) and models \(§\\lx@sectionsign[3\.4](https://arxiv.org/html/2610.19185#S3.SS4)\) we run\. Inference details are in Appendix[A](https://arxiv.org/html/2610.19185#A1)\.

### 3\.1Evaluation Setup

We study eight aspects of the evaluation setup\. As classification \(multiple\-choice questions\) and generation tasks are scored with different metrics, we evaluate and compare them separately\. Four aspects apply to both task types, three to classification tasks only, and one to generative tasks only\.

#### 3\.1\.1Task\-Agnostic Aspects

##### Number of shots\.

It can be difficult for a base model to infer the intended task from a short prompt alone \(*0\-shot*setting\)\. This prompt ambiguity can be resolved by including some solved examples of the same task in the context \(*5\-shot*evaluation in our case\), following the in\-context learning approach from[Brown et al\. \(2020\)](https://arxiv.org/html/2610.19185#bib.bib29)\.

##### Prompts\.

The issue with prompt ambiguity might be worse when a task relies only on a single prompt template\([Mizrahi et al\., 2024](https://arxiv.org/html/2610.19185#bib.bib27)\)\. We compare three options:*single*, which evaluates each task with its first \(or only\) template, and*multi\-mean*and*multi\-max*, which evaluate each task using all of its templates and combine the per\-prompt scores by their mean or maximum\.

##### Score rescaling\.

Tasks differ in their chance level and in the range their metric attains\. Therefore, simple averaging raw scores may let some tasks dominate the aggregate\. We compare four ways of putting the task scoresyyon a common footing before aggregating across tasks:

rbaseline​\(y\)\\displaystyle r\_\{\\text\{baseline\}\}\(y\)=max⁡\(0,y−bq1−bq\),\\displaystyle=\\max\\\!\\left\(0,\\ \\frac\{y\-b\_\{q\}\}\{1\-b\_\{q\}\}\\right\),\(1\)rz\-score​\(y\)\\displaystyle r\_\{\\text\{\{z\}\-score\}\}\(y\)=y−meanqsdq,\\displaystyle=\\frac\{y\-\\mathrm\{mean\}\_\{q\}\}\{\\mathrm\{sd\}\_\{q\}\},\(2\)rmin\-max​\(y\)\\displaystyle r\_\{\\text\{min\-max\}\}\(y\)=y−minqmaxq−minq,\\displaystyle=\\frac\{y\-\\min\_\{q\}\}\{\\max\_\{q\}\-\\min\_\{q\}\},\(3\)rnone​\(y\)\\displaystyle r\_\{\\text\{none\}\}\(y\)=y\.\\displaystyle=y\.\(4\)Here, all scoresyyare on a 0–1 scale, where 1 is the best possible score\.bqb\_\{q\}is the random baseline of taskqq, andmeanq\\mathrm\{mean\}\_\{q\},sdq\\mathrm\{sd\}\_\{q\},minq\\min\_\{q\}andmaxq\\max\_\{q\}are computed over the scores of taskqqfor all models and their checkpoints\. Baseline rescaling maps chance level to 0 and a perfect score to 1\([Srivastava et al\., 2023](https://arxiv.org/html/2610.19185#bib.bib23)\)\.Z\-score and min\-max rescaling align both the location and the spread of the tasks, but no longer have absolute interpretation\([Jain et al\., 2005](https://arxiv.org/html/2610.19185#bib.bib25)\)\.

##### Cross\-task aggregation\.

The task scores are combined into one score per checkpoint either by the*task\-level mean*over all tasks, or by the*category mean*, which first averages the tasks within each task category \(e\.g\., reading comprehension, machine translation\) and then averages those scores\. This equalizes the contribution of each category to the score, regardless of how many tasks it contains\.

#### 3\.1\.2Classification Task Aspects

For a multiple\-choice question with contextxxand answersa1,…,ana\_\{1\},\\dots,a\_\{n\}\(usually spanning more tokensaija\_\{i\}^\{j\}\), of whichai∗a\_\{i^\{\*\}\}is correct, the model scores each answer by its conditional log\-likelihood:

ℓi=∑aij∈ailog⁡Pθ​\(aij\|x,ai<j\)\.\\ell\_\{i\}=\\sum\_\{a\_\{i\}^\{j\}\\in a\_\{i\}\}\\log P\_\{\\theta\}\(a\_\{i\}^\{j\}\|x,a\_\{i\}^\{<j\}\)\.\(5\)The three aspects below concern how the question is presented to the model \(formulation\), how the option scoresℓi\\ell\_\{i\}are normalized \(log\-probability normalization\), and how the normalized scores are turned into a task score \(metric\)\.

##### Formulation\.

We consider three task formulations\([Robinson and Wingate, 2023](https://arxiv.org/html/2610.19185#bib.bib26);[Gu et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib9)\), illustrated in[Table1](https://arxiv.org/html/2610.19185#S3.T1)\. First, in*cloze formulation*\(CF\), the context contains only the passage and the question\. The optionsaia\_\{i\}are the full answer texts\. The model thus scores each answer text as a continuation without seeing the alternatives\. Second, in*multiple\-choice formulation*\(MCF\), the answer options are listed in the context with letter labels \(e\.g\., A, B, C, D\), and the optionsaia\_\{i\}are the letters themselves\. The model has to see all alternatives and map its choice to a label, which requires in\-context understanding of the multiple\-choice format\. Third, the*hybrid*formulation lists the answer options in the context, as in MCF, but without labels, and the optionsaia\_\{i\}are the full answer texts, as in CF\. The model sees all alternatives but does not need to produce a symbolic label\.

Table 1:Example prompt templates and answer options for the three task formulations of a multiple\-choice task\.
##### Log\-probability normalization\.

In practice, the raw log\-likelihoodℓi\\ell\_\{i\}can favor short options, since every additional token lowers it\. It also reflects how likely an option is regardless of the question\([Schaeffer et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib28)\)\. We compare three normalizations:

sinone\\displaystyle s\_\{i\}^\{\\text\{none\}\}=ℓi,\\displaystyle=\\ell\_\{i\},\(6\)sichar\\displaystyle s\_\{i\}^\{\\text\{char\}\}=ℓi/\|ai\|char,\\displaystyle=\\ell\_\{i\}\\,/\\,\|a\_\{i\}\|\_\{\\text\{char\}\},\(7\)siPMI\\displaystyle s\_\{i\}^\{\\text\{PMI\}\}=ℓi−log⁡Pθ​\(ai\),\\displaystyle=\\ell\_\{i\}\-\\log P\_\{\\theta\}\(a\_\{i\}\),\(8\)where\|ai\|char\|a\_\{i\}\|\_\{\\text\{char\}\}is the number of characters ofaia\_\{i\}, andPθ​\(ai\)P\_\{\\theta\}\(a\_\{i\}\)is the prior probability ofaia\_\{i\}\. Character\-length normalization removes the length bias, and point\-wise mutual information \(PMI\) removes the bias of the model for options that are likely, regardless of the question\.

##### Metrics\.

With the normalized scoress1,…,sns\_\{1\},\\dots,s\_\{n\}of the answers, we define three metrics, each averaged over the task examples:

Acc⁡\(si\)\\displaystyle\\mathrm\{Acc\(s\_\{i\}\)\}=𝟙\[argmaxisi=i∗\],\\displaystyle=\\mathbb\{1\}\\big\[\\arg\\max\_\{i\}s\_\{i\}=i^\{\*\}\\big\],\(9\)CondProb⁡\(si\)\\displaystyle\\mathrm\{CondProb\(s\_\{i\}\)\}=exp⁡\(si∗\)∑i=1nexp⁡\(si\),\\displaystyle=\\frac\{\\exp\(s\_\{i^\{\*\}\}\)\}\{\\sum\_\{i=1\}^\{n\}\\exp\(s\_\{i\}\)\},\(10\)AnsProb⁡\(si\)\\displaystyle\\mathrm\{AnsProb\(s\_\{i\}\)\}=exp⁡\(si∗\)\.\\displaystyle=\\exp\(s\_\{i^\{\*\}\}\)\.\(11\)These metrics are based on the earlier work by[Schaeffer et al\. \(2025\)](https://arxiv.org/html/2610.19185#bib.bib28)\.*Accuracy*\(Acc\) is discrete: each question contributes 0 or 1, so a model whose probability for the correct answer rises steadily during training gains nothing until that answer overtakes the others\. The two probability\-based metrics are continuous and can therefore reveal such gradual progress in early pretraining\. The*conditional answer probability*\(CondProb\) is the probability mass that a softmax over the normalized scores assigns to the correct option, i\.e\., the probability of the correct option conditional on the set of answer options\. The*answer probability*\(AnsProb\) is the probability of the correct option itself, without comparison to the alternatives\.111AnsProb and CondProb are true probabilities of the model only withsnones^\{\\text\{none\}\}; withsPMIs^\{\\text\{PMI\}\}, AnsProb can exceed 1\.

#### 3\.1\.3Generation Task Aspects

##### Decoding\.

We narrow our selection of decoding strategies for generation tasks to two options\. With*greedy decoding*, the model generates the most likely token at every step, which gives a single deterministic output per question\. With*random sampling*, tokens are sampled from the full model distribution \(temperature 1, no top\-kkor top\-pptruncation\), estimating the expected performance of the evaluated model\([Miller, 2024](https://arxiv.org/html/2610.19185#bib.bib24)\)\.

### 3\.2Meta\-Measures

We use three*meta\-measures*to assess how informative and diagnostic an evaluation setup is for tracking model development during pretraining\. Whereas an ordinary evaluation metric assigns a score to a model checkpoint on a task, a meta\-measure operates on a collection of such scores across checkpoints and models\. We define the three meta\-measures formally in the following subsections\. Appendix[B](https://arxiv.org/html/2610.19185#A2)describes how we compute them for all setups and languages\.

##### Notation\.

Letf∈ℱf\\in\\mathcal\{F\}index a model,t=1,…,Tft=1,\\dots,T\_\{f\}its checkpoints, andpf,tp\_\{f,t\}the number of training tokens at checkpointtt, so thatpf,Tfp\_\{f,T\_\{f\}\}is the token budget of a modelff\. An evaluation setupeeassigns one option to each aspect in§\\lx@sectionsign[3\.1](https://arxiv.org/html/2610.19185#S3.SS1)\. All setup analyses are carried out separately for each language and task type\. Under setupee, each checkpoint receives a single scoresf,ts\_\{f,t\}, which is computed from task scores with the metric \(e\.g\., accuracy\) ofee, aggregated over prompts, rescaled and aggregated over tasks, as prescribed byee\. A meta\-measure maps the scores of a setup over all models and their checkpoints to a single value\.222We note thatsf,ts\_\{f,t\}is a function ofee\. For readability, we omit this from the notation\.

##### Monotonicity\.

A useful setup should reflect that a model improves as it is trained on more data\. For each model, we compute the Spearman rank correlationρ\\rhobetween training progress and score over all of its checkpoints, and average it across models:

Mono⁡\(e\)=1\|ℱ\|​∑f∈ℱρ⁡\[\(pf,t\)t=1Tf,\(sf,t\)t=1Tf\]\.\\mathrm\{Mono\}\(e\)=\\frac\{1\}\{\|\\mathcal\{F\}\|\}\\\!\\sum\_\{f\\in\\mathcal\{F\}\}\\rho\\Big\[\\\!\(p\_\{f,t\}\)\_\{t=1\}^\{T\_\{f\}\},\(s\_\{f,t\}\)\_\{t=1\}^\{T\_\{f\}\}\\\!\\Big\]\\\!\.\\\!\(12\)

##### Non\-randomness\.

A setup should yield scores that are clearly distinguishable from chance, i\.e\., the gap between the best model and a random model should be large relative to the noise of the scores\. We therefore express the gap between the best final score and the random baseline in units of noise, following FineWeb2\([Penedo et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib19)\):

NonRand⁡\(e\)=maxf∈ℱ⁡sf,Tf−beσ^e,\\mathrm\{NonRand\}\(e\)=\\frac\{\\max\_\{f\\in\\mathcal\{F\}\}s\_\{f,T\_\{f\}\}\-b\_\{e\}\}\{\\hat\{\\sigma\}\_\{e\}\},\(13\)wherebeb\_\{e\}is the score of a random model, obtained by putting the random baselinebqb\_\{q\}of every task through the same rescaling and task aggregation as the model scores\. We use the variation of each model’s score at the end of training as a proxy for the noiseσ^e\\hat\{\\sigma\}\_\{e\}: the sample standard deviation over its lastK=5K=5checkpoints averaged across models,

σ^e=1\|ℱ\|​∑f∈ℱsd⁡\(sf,Tf−K\+1,…,sf,Tf\)\.\\hat\{\\sigma\}\_\{e\}=\\frac\{1\}\{\|\\mathcal\{F\}\|\}\\sum\_\{f\\in\\mathcal\{F\}\}\\operatorname\{sd\}\\big\(s\_\{f,T\_\{f\}\-K\+1\},\\dots,s\_\{f,T\_\{f\}\}\\big\)\.\(14\)Sincebeb\_\{e\}andσ^e\\hat\{\\sigma\}\_\{e\}are on the same scale as the scores,NonRand\\mathrm\{NonRand\}is unit\-free and comparable between setups with different score rescaling\.

##### Ranking consistency\.

A setup should rank models consistently, so that the comparison drawn at one checkpoint is not overturned at the next one\. Since models may differ in their token budgets \(e\.g\., the larger LLMs in§\\lx@sectionsign[3\.4](https://arxiv.org/html/2610.19185#S3.SS4)\), we align them by the number of training tokens, and split training into consecutive windows at half of the smallest budget and at the end of the training of each model\. A window is used only if at least two models are still training in it\. Following FineWeb2, we ignore the first window, where rankings are very inconsistent\.

Letπk,1,…,πk,mk\\pi\_\{k,1\},\\dots,\\pi\_\{k,m\_\{k\}\}be the rankings of the models at the successive checkpoints in windowkk\. Ranking consistency averages Kendall’sτ\\taubetween adjacent rankings within each window, and then over theNNremaining windows:

RankCons⁡\(e\)\\displaystyle\\mathrm\{RankCons\}\(e\)=∑k=1N∑i=1mk−1τ⁡\(πk,i,πk,i\+1\)N⁡\(mk−1\)\.\\displaystyle=\\sum\_\{k=1\}^\{N\}\\sum\_\{i=1\}^\{m\_\{k\}\-1\}\\frac\{\\tau\\big\(\\pi\_\{k,i\},\\pi\_\{k,i\+1\}\\big\)\}\{N\(m\_\{k\}\-1\)\}\.\(15\)

##### From meta\-measures to setup choices\.

A setup could be selected separately based on the highest score of each meta\-measure, but the three meta\-measures are on different scales and may favor different setups\. We therefore combine them using ranks\. For each language and task type, we compute an*overall score*\(00–11\) for each setup as the mean of its percentile ranks under the three meta\-measures\. We evaluate setups in three complementary ways\. The*best complete setup*is the setup with the highest overall score\. It captures interactions among aspects but may depend on small score differences\. The*option ranking*scores each option by the mean overall score of all setups containing it, averaging over all combinations of the other aspects, and thus measures its average effect\. The*per\-option win rate*compares pairs of setups that differ in only one aspect and records, for each meta\-measure, how often an option wins \(ties count as half\)\. We then average across the three meta\-measures\. This measures how consistently an option outperforms alternatives, regardless of the magnitude of the difference\. Further details are provided in Appendix[B](https://arxiv.org/html/2610.19185#A2)\.

### 3\.3Evaluation Benchmarks

We use an established multi\-task benchmark for each of the four languages\. The benchmarks differ in which evaluation setups they support\.[TableC\.1](https://arxiv.org/html/2610.19185#A3.T1)lists the number of tasks that support each aspect of the evaluation setup for each language\.

Figure 2:Per\-option win rates of the 2B models \(circle\) and larger LLMs \(diamond\) pooled over languages and task types; win rates are computed per meta\-measure, then averaged\. Higher win rate \(red\) is better\. The decoding aspect is only available for the 2B models\.##### NorEval\.

NorEval\([Mikhailov et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib10)\)is a Norwegian benchmark of 35 tasks, 17 multiple\-choice and 18 generative, covering Norwegian Bokmål, Nynorsk and Northern Sámi\. Its tasks span language understanding, linguistic knowledge, world knowledge and reasoning, generation and summarization, and machine translation\. Most of them come with four to six human\-written prompts\. Since the original benchmark does not support several of the setups studied in this work, we extend it in three ways:

- •Task formulations:every multiple\-choice task with a set of answer options \(14 out of 17\)333Pair\-wise comparison tasks such as NoCoLA\([Jentoft and Samuel, 2023](https://arxiv.org/html/2610.19185#bib.bib16)\)are excluded\.is available in CF, MCF and hybrid formulation, for all of its prompts\.
- •Log\-probability normalization and metrics:in addition to the raw accuracy and character\-length normalized accuracy, we compute CondProb and AnsProb and their character\- and PMI\-normalized counterparts as additional metrics\.
- •Random sampling:generative tasks drawKKsamples per question with temperature one, and report the mean score over samples\.

Together with the multiple prompts, our extension makes Norwegian the only language for which all eight aspects can be studied \([TableC\.1](https://arxiv.org/html/2610.19185#A3.T1)\)\.

##### FIN\-bench\-v2\.

For Finnish, we use FIN\-bench\-v2\([Kytöniemi et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib11)\), which combines Finnish versions of established English benchmarks with native\-authored tasks in Finnish\. We use its 14 tasks, 12 multiple\-choice and 2 generative, each with five prompts\. Ten of the multiple\-choice tasks come in both CF and MCF formulation, which allows us to compare these two formulations\.

##### FrenchBench\.

For French, we use FrenchBench\([Faysse et al\., 2024](https://arxiv.org/html/2610.19185#bib.bib12)\), and the French portions of Global\-MMLU\([Singh et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib17)\)and INCLUDE\([Romanou et al\., 2024](https://arxiv.org/html/2610.19185#bib.bib18)\), resulting in 16 tasks \(11 multiple\-choice, 5 generative\)\. Nine of the tasks are evaluated with the multi\-prompt version from HPLT\-E\([Oepen et al\., 2026](https://arxiv.org/html/2610.19185#bib.bib14)\)\. The remaining seven have the single prompt of its original implementation\. FrenchBench has no formulation variants and does not provide PMI\-normalized accuracy, so French is not included in the comparisons of formulation and log\-probability normalization\.

##### SpanishBench\.

For Spanish, we use SpanishBench from IberoBench\([Baucells et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib13)\)in its multi\-prompt version from HPLT\-E\. It comprises 16 tasks \(11 multiple\-choice, 5 generative\), each with three prompts\. There are no formulation variants for Spanish\. PMI\-normalized accuracy is available for only 5 of the multiple\-choice tasks\.

### 3\.4Models

##### Early pretraining models\.

Our main experiments use 26 monolingual 2B\-parameter models that share the same architecture and training setup and differ only in their pretraining data, each trained on 100B tokens of Norwegian, Finnish, French or Spanish\. Four models are trained on native web corpora, FineWeb2, HPLT 2\.0, HPLT 3\.0 and MADLAD\-400\([Oepen et al\., 2026](https://arxiv.org/html/2610.19185#bib.bib14)\), and three on synthetic data machine\-translated from English with Tower\-9B, Tower\-72B and OPUS\-MT, respectively\([Idahl et al\., 2026](https://arxiv.org/html/2610.19185#bib.bib20)\)\. Tower\-72B is available for Finnish and Spanish only, which gives seven models for these two languages and six for French and Norwegian\. We evaluate 11 checkpoints of each model, from 2B to 100B tokens\.

##### Larger\-scale LLMs\.

To examine how the preferred setups change with model scale, we evaluate the intermediate checkpoints of three open\-source larger multilingual LLMs that differ in their training budgets: 1\) OLMo 3 7B\([Team Olmo et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib22)\), with 10 checkpoints of its pretraining stage up to 6T tokens; 2\) Apertus 8B\([Hernández\-Cano et al\., 2026](https://arxiv.org/html/2610.19185#bib.bib21)\), with 14 checkpoints up to 15T tokens; 3\) Prelude 9B,444[https://huggingface\.co/openeurollm/prelude](https://huggingface.co/openeurollm/prelude)with 45 checkpoints up to 8T tokens\.

## 4Results

We first present the early pretraining models \(§\\lx@sectionsign[4\.1](https://arxiv.org/html/2610.19185#S4.SS1)\) and then compare them with the larger LLMs \(§\\lx@sectionsign[4\.2](https://arxiv.org/html/2610.19185#S4.SS2)\)\. The win rates of both model scales are summarized in[Figure2](https://arxiv.org/html/2610.19185#S3.F2)\. Language\-specific results of the three setup choice methods are in Appendix[D](https://arxiv.org/html/2610.19185#A4)\.

### 4\.1Early Pretraining Models

Based on the agreement of the three methods – best complete setup, option ranking and per\-option win rates – we group the evaluation aspects into a tiered recipe: high\-impact choices, low\-impact choices and context\-dependent choices in[Table2](https://arxiv.org/html/2610.19185#S4.T2)\.

##### High\-impact choices\.

Four choices are consistently selected by all three methods and in nearly every language for the 2B models\. These choices can yield drastically different conclusions about pretraining as shown in[Figure1](https://arxiv.org/html/2610.19185#S1.F1)\.

The*answer probability*is by far the most informative metric\. It wins 93% of its pairwise comparisons across all meta\-measures\. Accuracy is the least informative metric because its discrete scores increase variance without giving enough signal\.

*Character normalization*wins in every language except Spanish, although its win\-rate advantage is small \(55%\) and arises primarily from non\-randomness\. PMI has the highest non\-randomness \(66%\) but the lowest monotonicity \(18%\)\.

*5\-shot*prompting outperforms 0\-shot \(win rate 70%\), most clearly in monotonicity \(84%\)\.

Among the formulations,*CF*is selected by all methods \(win rate 76%\), whereas MCF is the least informative, consistent with the observation that small models have not yet learned the multiple\-choice format\([Gu et al\., 2025](https://arxiv.org/html/2610.19185#bib.bib9)\)\.

##### Low\-impact choices\.

There are two aspects that do not show a clear preference\.

Usingmultiple promptsis preferable to using a single prompt, but the choice between aggregating their scores by the maximum or the mean has little effect\. The two aggregation strategies differ primarily in the meta\-measure they favor: maximum aggregation yields higher non\-randomness, whereas mean aggregation yields higher monotonicity \(Appendix[E](https://arxiv.org/html/2610.19185#A5)for disagreement analysis\)\.

Similarly, averaging over all tasks and averaging over task\-category means yield comparable results, with win rates of 56% and 44%, respectively\. Only French favors the category mean for all three methods, whereas the other languages differ in choices\.

TierAspect \(top option\)2BLargerHigh\-impactMetric \(AnsProb\)4/43/4Logprob\. Norm\. \(char\)3/43/4Shots \(5\-shot\)4/44/4Formul\. \(CF\)2/21/2Low\-impactPrompts \(multi\-max\)0/41/4Task agg\. \(task\-level mean\)2/43/4Context\-dep\.Rescaling \(baseline / none\)0/40/4Decoding \(sampling\)0/1–

Table 2:Number of languages in which all three setup choice methods select the option with the highest pooled win rate \([Figure2](https://arxiv.org/html/2610.19185#S3.F2)\)\. For rescaling, the top option is baseline for the 2B models and none for the larger LLMs\. Formulation: Finnish and Norwegian; decoding: Norwegian, 2B models only\.
##### Context\-dependent choices\.

The most disagreement is in preferences of the following two aspects\.

Score rescaling exhibits the greatest disagreement among the three methods\. In no language do all three methods selectbaseline rescaling, and their preferred options are distributed across baseline, min\-max and no rescaling\. This disagreement is driven by an interaction with the metric\. For answer probability, whose chance level is 0, all rescaling methods yield nearly identical mean overall scores, and the best complete setups select min\-max rescaling by a small margin\. By contrast, for accuracy, baseline rescaling and no rescaling clearly outperform min\-max andz\-score rescaling\.

For the Norwegian generative tasks,samplingperforms better on average \(option\-ranking score 0\.56 vs\. 0\.45 in[TableD\.2](https://arxiv.org/html/2610.19185#A4.T2); win rate 64%\), although the best complete setup uses greedy decoding\.

##### The tiered recipe versus the best complete setup\.

Combining the highest\-ranked option for each aspect according to the option\-ranking analysis in[TableD\.2](https://arxiv.org/html/2610.19185#A4.T2)\(5\-shot, multi\-max, baseline rescaling, mean over tasks, CF, answer probability, and character normalization\) yields a setup that ranks 6th among the 264 setups shared across all languages, with an overall score of 0\.93, compared with 0\.95 for the best complete setup\. The latter differs only in its use of min\-max rescaling\. Thus, selecting options independently by aspect incurs only a small loss relative to jointly optimizing the complete setup, and all of the top five complete setups agree on thehigh\-impact choices\.

### 4\.2Larger\-Scale LLMs

We apply our approach to the three larger LLMs \(§\\lx@sectionsign[3\.4](https://arxiv.org/html/2610.19185#S3.SS4)\) to assess its robustness across model scales and training stages\. Specifically, we examine whether the identified recipe captures properties of the evaluation setups rather than characteristics specific to the 2B models: high\-impact choices are expected to remain stable across scales, whereas context\-dependent choices may vary\. We also assess which evaluation setups continue to provide informative signals later in pretraining, when model performance is well above chance\.[Figure2](https://arxiv.org/html/2610.19185#S3.F2)presents the win rates of smaller vs\. larger models\.

##### Findings that remain stable across scale\.

For the larger LLMs, all three methods continue to select answer probability with character normalization, 5\-shot prompting and aggregation by the mean over all tasks, consistent with the results for the 2B models\. Using multiple prompts also remains preferable to using a single prompt, with the single\-prompt option attaining a win rate of 35% for the larger LLMs, compared with 37% for the 2B models\. The three methods agree to a similar extent for the larger LLMs as shown in[Table2](https://arxiv.org/html/2610.19185#S4.T2)\. The exceptions are formulation, where all three methods select hybrid, and the metric, where the win rate favors accuracy for Norwegian \([TableD\.1](https://arxiv.org/html/2610.19185#A4.T1)\)\. These results indicate that the high\-impact choices identified for the 2B models are not specific to that model scale, providing further support for the use of the meta\-measures to select evaluation setups\.

##### Scale\-dependent changes\.

The main differences arise among choices whose impacts are high or context dependent\. These changes are consistent with known effects of model scale\. Accuracy becomes much more informative \(win rate 6%→\\rightarrow37%\), primarily due to improved monotonicity \(9%→\\rightarrow53%\), suggesting that it provides a more reliable signal of training progress once models perform well above chance\. Although answer probability remains the strongest metric, its advantage decreases \(93%→\\rightarrow63%\), both with monotonicity \(92%→\\rightarrow44%\) and ranking consistency \(92%→\\rightarrow52%\)\.

The advantage of CF over other task formulations diminishes for the larger LLMs\. Although CF remains the preferred formulation overall, its win rate decreases \(76%→\\rightarrow61%\), whereas MCF improves \(20%→\\rightarrow38%\), especially in ranking consistency \(33%→\\rightarrow71%\) and monotonicity \(2%→\\rightarrow30%\)\. For Norwegian, the option ranking and the win rate prefer the hybrid formulation \([TableD\.1](https://arxiv.org/html/2610.19185#A4.T1)\)\. PMI normalization shows a substantial decline in non\-randomness \(66%→\\rightarrow17%\), indicating that it becomes less effective, relative to other options, at distinguishing model performance from chance\. Preferences among rescaling methods also become less pronounced\.Z\-score rescaling gains \(39%→\\rightarrow49%\) and is used in the best complete setup, whereas option ranking and win rate slightly favor no rescaling \(56%\) over baseline rescaling \(54%\)\.

At the level of complete setups, the overall scores for the 2B and larger models are only moderately correlated \(Spearman’sρ\\rho= 0\.66 for Finnish, 0\.41 for Spanish, 0\.32 for French, and 0\.30 for Norwegian\), with high agreement among the highest\-ranked setups and substantial reordering in the middle of the ranking\.

##### Implications\.

The scale\-robust components of the recipe can be applied throughout pretraining, whereas context\-dependent choices should be reassessed as model scale increases\. During later stages of pretraining, accuracy and the MCF formulation become considerably more informative, making them increasingly viable alternatives to answer probability and CF, respectively\. However, because these results are based on only three models trained with different compute budgets, estimates of ranking consistency rely on relatively few comparisons\. We therefore interpret the observed scale\-dependent differences as suggestive rather than conclusive\.

## 5Conclusion

We studied how the choices in an evaluation setup affect the signal that benchmark results provide during pretraining\. We scored every combination of eight evaluation aspects on three meta\-measures \(monotonicity, non\-randomness and ranking consistency\), using 26 monolingual 2B\-parameter models in four languages and validating on three larger multilingual LLMs\.

Our results support a tiered recipe rather than a single optimal evaluation setup\. Several choices consistently yield more informative signals across model scales: answer probability with character normalization outperforms alternative metrics, while 5\-shot prompting and multiple prompts outperform 0\-shot and single\-prompt evaluation, respectively\. Other choices, such as prompt and task aggregation strategies, have relatively limited effects\. Some preferences can vary with model scale: answer probability and cloze formulation are preferred for the 2B models, whereas accuracy and multiple\-choice \(or hybrid\) formulation become increasingly informative as model performance grows\. These findings suggest prioritizing robust choices while reassessing context\-dependent ones as training progresses\. More broadly, our framework provides a systematic approach to assessing the reliability of evaluation setups before using their results to guide model development\.

## Limitations

There is no ground truth for which evaluation setup is best during pretraining\. We therefore use three meta\-measures as a proxy for a useful indicator\. These properties are desirable, but they do not guarantee that a setup captures everything that matters for pretraining decisions\. For instance, a setup can score high on all three meta\-measures while measuring a narrow capability, or while being insensitive to differences in data that matter downstream\. Our recipe should therefore be read as a selection of setups that give informative and diagnostic signals, not of setups that are necessarily the most valid measures of model quality\.

We combine the three meta\-measures with equal weight, as the mean of their percentile ranks\. Other combinations, such as different weights, requiring a minimum value on each meta\-measure, or Pareto\-optimal setups, could lead to different choices, in particular where the meta\-measures disagree\. Appendix[E](https://arxiv.org/html/2610.19185#A5)discusses such a case in more detail\. Other meta\-measures have been proposed, such as the signal\-to\-noise ratio of[Heineman et al\. \(2025b\)](https://arxiv.org/html/2610.19185#bib.bib8)\. We did not include them, since they are defined and estimated in different ways \(e\.g\., from seed runs or variation between nearby checkpoints\) that are not directly comparable with our setting\.

Our study covers four European languages, each with its own benchmark, and the available options differ between them\. Formulation variants exist only for Finnish and Norwegian\. The conditional answer probability and generative tasks are compared for Norwegian only\. Some findings may therefore reflect properties of individual benchmarks rather than of the evaluation setups\. The larger\-scale validation relies on three models with different architectures, training data and token budgets\. With three models, ranking consistency rests on few comparisons, so we consider the differences between the 2B and the larger models indicative rather than conclusive\.

## Ethical Considerations

This research relies exclusively on publicly accessible datasets, with all data utilization adhering to the license agreements specified by[Mikhailov et al\. \(2025\)](https://arxiv.org/html/2610.19185#bib.bib10);[Kytöniemi et al\. \(2025\)](https://arxiv.org/html/2610.19185#bib.bib11);[Faysse et al\. \(2024\)](https://arxiv.org/html/2610.19185#bib.bib12);[Singh et al\. \(2025\)](https://arxiv.org/html/2610.19185#bib.bib17);[Romanou et al\. \(2024\)](https://arxiv.org/html/2610.19185#bib.bib18);[Baucells et al\. \(2025\)](https://arxiv.org/html/2610.19185#bib.bib13)and[Oepen et al\. \(2026\)](https://arxiv.org/html/2610.19185#bib.bib14)\. It is presumed that these datasets contain no sensitive or personally identifiable information\. Consequently, their application in this study is deemed to have no significant ethical risks\.

All original ideas, analyses, and content were created by the authors\. AI tools were used only as supportive aids for improving writing quality and assisting with coding tasks\. The authors retain full responsibility for the intellectual content, analyses, and conclusions presented in this work\.

## Acknowledgments

This research was supported by the OpenEuroLLM project, co\-funded by the Digital Europe Programme under grant agreement No\. 101195233\. This research was also supported by the European Union’s Horizon Europe research and innovation programme under the Marie Skłodowska\-Curie grant agreement No\. 101126636\. We acknowledge Sigma2, Norway for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC \(Finland\) and the LUMI consortium\.

## References

- Alrashed and Orabona \(2026\)S\. Alrashed and F\. OrabonaMix, MinHash, and Match: cross\-source agreement for multilingual pretraining datasets\.External Links:2512\.18834,[Link](https://arxiv.org/abs/2512.18834)Cited by:[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px1.p1.1)\.
- Alzahraniet al\.\(2024\)N\. Alzahrani, H\. Alyahya, Y\. Alnumay, S\. AlRashed, S\. Alsubaie, Y\. Almushayqih, F\. Mirza, N\. Alotaibi, N\. Al\-Twairesh, A\. Alowisheq, M\. S\. Bari, and H\. KhanWhen benchmarks are targets: revealing the sensitivity of large language model leaderboards\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 13787–13805\.External Links:[Link](https://aclanthology.org/2024.acl-long.744/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.744)Cited by:[§1](https://arxiv.org/html/2610.19185#S1.p1.1)\.
- Baucellset al\.\(2025\)I\. Baucells, J\. Aula\-Blasco, I\. de\-Dios\-Flores, S\. Paniagua Suárez, N\. Perez, A\. Salles, S\. Sotelo Docio, J\. Falcão, J\. J\. Saiz, R\. Sepulveda Torres, J\. Barnes, P\. Gamallo, A\. Gonzalez\-Agirre, G\. Rigau, and M\. VillegasIberoBench: a benchmark for LLM evaluation in Iberian languages\.InProceedings of the 31st International Conference on Computational Linguistics,Abu Dhabi, UAE,pp\. 10491–10519\.External Links:[Link](https://aclanthology.org/2025.coling-main.699/)Cited by:[§3\.3](https://arxiv.org/html/2610.19185#S3.SS3.SSS0.Px4.p1.1),[Ethical Considerations](https://arxiv.org/html/2610.19185#Sx2.p1.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§3\.1\.1](https://arxiv.org/html/2610.19185#S3.SS1.SSS1.Px1.p1.1)\.
- Faysseet al\.\(2024\)M\. Faysse, P\. Fernandes, N\. M\. Guerreiro, A\. Loison, D\. M\. Alves, C\. Corro, N\. Boizard, J\. Alves, R\. Rei, P\. H\. Martins, A\. Bigata Casademunt, F\. Yvon, A\. F\. T\. Martins, G\. Viaud, C\. Hudelot, and P\. ColomboCroissantLLM: a truly bilingual French\-English language model\.External Links:2402\.00786Cited by:[§3\.3](https://arxiv.org/html/2610.19185#S3.SS3.SSS0.Px3.p1.1),[Ethical Considerations](https://arxiv.org/html/2610.19185#Sx2.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. ZouA framework for few\-shot language model evaluation\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.10256836),[Link](https://zenodo.org/records/10256836)Cited by:[Appendix A](https://arxiv.org/html/2610.19185#A1.p1.1)\.
- Guet al\.\(2025\)Y\. Gu, O\. Tafjord, B\. Kuehl, D\. Haddad, J\. Dodge, and H\. HajishirziOLMES: a standard for language model evaluations\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 5020–5048\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.282/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.282),ISBN 979\-8\-89176\-195\-7Cited by:[§1](https://arxiv.org/html/2610.19185#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px2.p1.1),[§3\.1\.2](https://arxiv.org/html/2610.19185#S3.SS1.SSS2.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.19185#S4.SS1.SSS0.Px1.p5.1)\.
- Heinemanet al\.\(2025a\)D\. Heineman, V\. Hofmann, I\. Magnusson, Y\. Gu, N\. A\. Smith, H\. Hajishirzi, K\. Lo, and J\. DodgeSignal and noise: a framework for reducing uncertainty in language model evaluation\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=sAFottNlra)Cited by:[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px2.p1.1)\.
- Heinemanet al\.\(2025b\)D\. Heineman, V\. Hofmann, I\. Magnusson, Y\. Gu, N\. A\. Smith, H\. Hajishirzi, K\. Lo, and J\. DodgeSignal and noise: a framework for reducing uncertainty in language model evaluation\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/18b3b2947174ae9b1a8fc91ff89f4eb6-Paper-Conference.pdf)Cited by:[Limitations](https://arxiv.org/html/2610.19185#Sx1.p2.1)\.
- Hernández\-Canoet al\.\(2026\)A\. Hernández\-Cano, A\. Hägele, A\. H\. Huang, A\. Romanou, A\. Solergibert, B\. Pásztor, B\. Messmer, D\. Garbaya, E\. F\. Ďurech, I\. Hakimi, J\. G\. Giraldo, M\. Ismayilzada, N\. Foroutan, S\. Moalla, T\. Chen, V\. Sabolčec, Y\. Xu, M\. Aerni, B\. AlKhamissi, I\. A\. Marinas, M\. H\. Amani, M\. Ansaripour, I\. Badanin, H\. Benoit, E\. Boros, N\. J\. Browning, F\. Bösch, M\. Böther, N\. Canova, C\. Challier, C\. Charmillot, J\. Coles, J\. M\. Deriu, A\. Devos, L\. Drescher, D\. Dzenhaliou, M\. Ehrmann, D\. Fan, S\. Fan, S\. Gao, M\. Gila, M\. Grandury, D\. Hashemi, A\. M\. Hoyle, J\. Jiang, M\. Klein, A\. Kucharavy, A\. Kucherenko, F\. Lübeck, R\. Machacek, T\. I\. Manitaras, A\. Marfurt, K\. Matoba, S\. Matrenok, H\. Mendonça, F\. R\. Mohamed, S\. Montariol, L\. Mouchel, S\. Najem\-Meyer, J\. Ni, G\. Oliva, M\. Pagliardini, E\. Palme, A\. Panferov, L\. Paoletti, M\. Passerini, I\. Pavlov, A\. Poiroux, K\. Ponkshe, N\. Ranchin, J\. Rando, M\. Sauser, J\. Saydaliev, M\. Sayfiddinov, M\. Schneider, S\. Schuppli, M\. Scialanga, A\. Semenov, K\. Shridhar, R\. Singhal, A\. Sotnikova, A\. Sternfeld, A\. K\. Tarun, P\. Teiletche, J\. Vamvas, X\. Yao, H\. Zhao, A\. Ilic, A\. Klimovic, A\. Krause, C\. Gulcehre, D\. Rosenthal, E\. Ash, F\. Tramèr, J\. VandeVondele, L\. Veraldi, M\. Rajman, T\. C\. Schulthess, T\. Hoefler, A\. Bosselut, M\. Jaggi, and I\. SchlagApertus: democratizing open and compliant LLMs for global language environments\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 46877–46955\.External Links:[Link](https://aclanthology.org/2026.acl-long.2172/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.2172),ISBN 979\-8\-89176\-390\-6Cited by:[§3\.4](https://arxiv.org/html/2610.19185#S3.SS4.SSS0.Px2.p1.1)\.
- Idahlet al\.\(2026\)M\. Idahl, J\. Tiedemann, S\. Pyysalo, D\. Salinas, T\. Galica, S\. Qian, T\. N\. Mateiu, Z\. Li, A\. Lokrantz, F\. Vitiugin, A\. F\. T\. Martins, J\. Kanerva, F\. Ginter, M\. Lindemann, T\. Isbister, B\. Moell, J\. Lindh, J\. Hajič, J\. Jitsev, A\. Kutuzov, S\. Oepen, and G\. Ramírez\-SánchezMultiSynt/MT: trillion\-token multi\-parallel pre\-training data translated across 36 languages\.External Links:2607\.00890,[Link](https://arxiv.org/abs/2607.00890)Cited by:[§3\.4](https://arxiv.org/html/2610.19185#S3.SS4.SSS0.Px1.p1.1)\.
- Jainet al\.\(2005\)A\. Jain, K\. Nandakumar, and A\. RossScore normalization in multimodal biometric systems\.Pattern Recogn\.38\(12\),pp\. 2270–2285\.External Links:ISSN 0031\-3203,[Link](https://doi.org/10.1016/j.patcog.2005.01.012),[Document](https://dx.doi.org/10.1016/j.patcog.2005.01.012)Cited by:[§3\.1\.1](https://arxiv.org/html/2610.19185#S3.SS1.SSS1.Px3.p1.2)\.
- Jentoft and Samuel \(2023\)M\. Jentoft and D\. SamuelNoCoLA: the Norwegian corpus of linguistic acceptability\.InProceedings of the 24th Nordic Conference on Computational Linguistics \(NoDaLiDa\),T\. Alumäe and M\. Fishel \(Eds\.\),Tórshavn, Faroe Islands,pp\. 610–617\.External Links:[Link](https://aclanthology.org/2023.nodalida-1.60/)Cited by:[footnote 3](https://arxiv.org/html/2610.19185#footnote3)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.External Links:2001\.08361,[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2610.19185#S1.p2.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles,SOSP ’23,New York, NY, USA,pp\. 611–626\.External Links:ISBN 9798400702297,[Link](https://doi.org/10.1145/3600006.3613165),[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[Appendix A](https://arxiv.org/html/2610.19185#A1.p1.1)\.
- Kydlíčeket al\.\(2024\)H\. Kydlíček, G\. Penedo, C\. Fourier, N\. Habib, and T\. WolfFineTasks: finding signal in a haystack of 200\+ multilingual tasks\.External Links:[Link](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fine-tasks)Cited by:[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px1.p1.1)\.
- Kytöniemiet al\.\(2025\)J\. Kytöniemi, J\. Piha, A\. Reunamo, F\. Vitiugin, F\. Mehryary, and S\. PyysaloFIN\-bench\-v2: a unified and robust benchmark suite for evaluating Finnish large language models\.External Links:2512\.13330,[Link](https://arxiv.org/abs/2512.13330)Cited by:[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px2.p2.1),[§3\.3](https://arxiv.org/html/2610.19185#S3.SS3.SSS0.Px2.p1.1),[Ethical Considerations](https://arxiv.org/html/2610.19185#Sx2.p1.1)\.
- Liet al\.\(2024\)J\. Li, A\. Fang, G\. Smyrnis, M\. Ivgi, M\. Jordan, S\. Y\. Gadre, H\. Bansal, E\. K\. Guha, S\. Keh, K\. Arora, S\. Garg, R\. Xin, N\. Muennighoff, R\. Heckel, J\. Mercat, M\. F\. Chen, S\. Gururangan, M\. Wortsman, A\. Albalak, Y\. Bitton, M\. Nezhurina, A\. K\. M\. Abbas, C\. Hsieh, D\. Ghosh, J\. P\. Gardner, M\. Kilian, H\. Zhang, R\. Shao, S\. M\. Pratt, S\. Sanyal, G\. Ilharco, G\. Daras, K\. Marathe, A\. Gokaslan, J\. Zhang, K\. Chandu, T\. Nguyen, I\. Vasiljevic, S\. M\. Kakade, S\. Song, S\. Sanghavi, F\. Faghri, S\. Oh, L\. Zettlemoyer, K\. Lo, A\. El\-Nouby, H\. Pouransari, A\. T\. Toshev, S\. Wang, D\. Groeneveld, L\. Soldaini, P\. W\. Koh, J\. Jitsev, T\. Kollar, A\. Dimakis, Y\. Carmon, A\. Dave, L\. Schmidt, and V\. ShankarDataComp\-LM: in search of the next generation of training sets for language models\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=CNWdWn47IE)Cited by:[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px1.p1.1)\.
- Lianget al\.\(2023\)P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar, B\. Newman, B\. Yuan, B\. Yan, C\. Zhang, C\. Cosgrove, C\. D\. Manning, C\. Re, D\. Acosta\-Navas, D\. A\. Hudson, E\. Zelikman, E\. Durmus, F\. Ladhak, F\. Rong, H\. Ren, H\. Yao, J\. Wang, K\. Santhanam, L\. Orr, L\. Zheng, M\. Yuksekgonul, M\. Suzgun, N\. Kim, N\. Guha, N\. S\. Chatterji, O\. Khattab, P\. Henderson, Q\. Huang, R\. A\. Chi, S\. M\. Xie, S\. Santurkar, S\. Ganguli, T\. Hashimoto, T\. Icard, T\. Zhang, V\. Chaudhary, W\. Wang, X\. Li, Y\. Mai, Y\. Zhang, and Y\. KoreedaHolistic Evaluation of Language Models\.Transactions on Machine Learning Research\.External Links:[Link](https://mlanthology.org/tmlr/2023/liang2023tmlr-holistic/)Cited by:[§1](https://arxiv.org/html/2610.19185#S1.p1.1)\.
- Madaanet al\.\(2025\)L\. Madaan, A\. K\. Singh, R\. Schaeffer, A\. Poulton, S\. Koyejo, P\. Stenetorp, S\. Narang, and D\. HupkesQuantifying variance in evaluation benchmarks\.External Links:[Link](https://openreview.net/forum?id=E2RyjrBMVZ)Cited by:[§1](https://arxiv.org/html/2610.19185#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px2.p1.1)\.
- Magnussonet al\.\(2025\)I\. Magnusson, N\. Tai, B\. Bogin, D\. Heineman, J\. D\. Hwang, L\. Soldaini, A\. Bhagia, J\. Liu, D\. Groeneveld, O\. Tafjord, N\. A\. Smith, P\. W\. Koh, and J\. DodgeDataDecide: how to predict best pretraining data with small experiments\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 42487–42502\.External Links:[Link](https://proceedings.mlr.press/v267/magnusson25a.html)Cited by:[§1](https://arxiv.org/html/2610.19185#S1.p1.1)\.
- Mikhailovet al\.\(2025\)V\. Mikhailov, T\. Enstad, D\. Samuel, H\. C\. Farsethås, A\. Kutuzov, E\. Velldal, and L\. ØvrelidNorEval: a Norwegian language understanding and generation evaluation benchmark\.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp\. 3495–3541\.External Links:[Link](https://aclanthology.org/2025.findings-acl.181/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.181)Cited by:[Figure 1](https://arxiv.org/html/2610.19185#S1.F1),[§1](https://arxiv.org/html/2610.19185#S1.SS0.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px2.p2.1),[§3\.3](https://arxiv.org/html/2610.19185#S3.SS3.SSS0.Px1.p1.1),[Ethical Considerations](https://arxiv.org/html/2610.19185#Sx2.p1.1)\.
- Miller \(2024\)E\. MillerAdding error bars to evals: a statistical approach to language model evaluations\.External Links:2411\.00640,[Link](https://arxiv.org/abs/2411.00640)Cited by:[§3\.1\.3](https://arxiv.org/html/2610.19185#S3.SS1.SSS3.Px1.p1.1)\.
- Mizrahiet al\.\(2024\)M\. Mizrahi, G\. Kaplan, D\. Malkin, R\. Dror, D\. Shahaf, and G\. StanovskyState of what art? a call for multi\-prompt LLM evaluation\.Transactions of the Association for Computational Linguistics12,pp\. 933–949\.External Links:[Link](https://aclanthology.org/2024.tacl-1.52/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00681)Cited by:[§3\.1\.1](https://arxiv.org/html/2610.19185#S3.SS1.SSS1.Px2.p1.1)\.
- Oepenet al\.\(2026\)S\. Oepen, N\. Arefyev, M\. Aulamo, M\. Bañón, M\. Buljan, L\. V\. Burchell, L\. G\. G\. Charpentier, P\. Chen, M\. Fedorova, O\. de Gibert, B\. Haddow, J\. Hajič, J\. Helcl, A\. Kutuzov, V\. Laippala, Z\. Li, B\. Malik, V\. Mikhailov, A\. Myntti, D\. O’Brien, L\. Polakova, G\. Ramírez\-Sánchez, J\. Siewert, P\. Stepachev, J\. Tiedemann, T\. Vahtola, D\. Varis, F\. Vitiugin, and J\. ZaragozaHPLT 3\.0: very large\-scale multilingual resources for LLMs and MT\. mono\- and bi\-lingual data, multilingual evaluation, and pre\-trained models\.InProceedings of the Fifteenth Language Resources and Evaluation Conference,S\. Piperidis, N\. Bel, H\. van den Heuvel, N\. Ide, S\. Krek, and A\. Toral \(Eds\.\),Palma de Mallorca, Spain,pp\. 1409–1434\.External Links:[Link](https://aclanthology.org/2026.lrec-1.110/),[Document](https://dx.doi.org/10.63317/25xbdofco9od)Cited by:[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2610.19185#S3.SS3.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2610.19185#S3.SS4.SSS0.Px1.p1.1),[Ethical Considerations](https://arxiv.org/html/2610.19185#Sx2.p1.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, L\. B\. Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, and T\. WolfThe FineWeb datasets: decanting the web for the finest text data at scale\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px1.p1.1)\.
- Penedoet al\.\(2025\)G\. Penedo, H\. Kydlíček, V\. Sabolčec, B\. Messmer, N\. Foroutan, A\. H\. Kargaran, C\. Raffel, M\. Jaggi, L\. Von Werra, and T\. WolfFineWeb2: one pipeline to scale them all – adapting pre\-training data processing to every language\.External Links:2506\.20920,[Link](https://arxiv.org/abs/2506.20920)Cited by:[§1](https://arxiv.org/html/2610.19185#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2610.19185#S3.SS2.SSS0.Px3.p1.1)\.
- Robinson and Wingate \(2023\)J\. Robinson and D\. WingateLeveraging large language models for multiple choice question answering\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=yKbprarjc5B)Cited by:[§3\.1\.2](https://arxiv.org/html/2610.19185#S3.SS1.SSS2.Px1.p1.1)\.
- Romanouet al\.\(2024\)A\. Romanou, N\. Foroutan, A\. Sotnikova, Z\. Chen, S\. H\. Nelaturu, S\. Singh, R\. Maheshwary, M\. Altomare, M\. A\. Haggag, A\. Snegha, A\. Amayuelas, A\. H\. Amirudin, V\. Aryabumi, D\. Boiko, M\. Chang, J\. Chim, G\. Cohen, A\. K\. Dalmia, A\. Diress, S\. Duwal, D\. Dzenhaliou, D\. F\. E\. Florez, F\. Farestam, J\. M\. Imperial, S\. B\. Islam, P\. Isotalo, M\. Jabbarishiviari, B\. F\. Karlsson, E\. Khalilov, C\. Klamm, F\. Koto, D\. Krzemiński, G\. A\. de Melo, S\. Montariol, Y\. Nan, J\. Niklaus, J\. Novikova, J\. S\. O\. Ceron, D\. Paul, E\. Ploeger, J\. Purbey, S\. Rajwal, S\. S\. Ravi, S\. Rydell, R\. Santhosh, D\. Sharma, M\. P\. Skenduli, A\. S\. Moakhar, B\. S\. Moakhar, R\. Tamir, A\. K\. Tarun, A\. T\. Wasi, T\. O\. Weerasinghe, S\. Yilmaz, M\. Zhang, I\. Schlag, M\. Fadaee, S\. Hooker, and A\. BosselutINCLUDE: evaluating multilingual language understanding with regional knowledge\.External Links:2411\.19799,[Link](https://arxiv.org/abs/2411.19799)Cited by:[§3\.3](https://arxiv.org/html/2610.19185#S3.SS3.SSS0.Px3.p1.1),[Ethical Considerations](https://arxiv.org/html/2610.19185#Sx2.p1.1)\.
- Schaefferet al\.\(2025\)R\. Schaeffer, H\. Schoelkopf, B\. Miranda, G\. Mukobi, V\. Madan, A\. Ibrahim, H\. Bradley, S\. Biderman, and S\. KoyejoWhy has predicting downstream capabilities of frontier AI models with scale remained elusive?\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=I1NtlLvJal)Cited by:[§3\.1\.2](https://arxiv.org/html/2610.19185#S3.SS1.SSS2.Px2.p1.1),[§3\.1\.2](https://arxiv.org/html/2610.19185#S3.SS1.SSS2.Px3.p1.2)\.
- Singhet al\.\(2025\)S\. Singh, A\. Romanou, C\. Fourrier, D\. I\. Adelani, J\. G\. Ngui, D\. Vila\-Suero, P\. Limkonchotiwat, K\. Marchisio, W\. Q\. Leong, Y\. Susanto, R\. Ng, S\. Longpre, S\. Ruder, W\. Ko, A\. Bosselut, A\. Oh, A\. Martins, L\. Choshen, D\. Ippolito, E\. Ferrante, M\. Fadaee, B\. Ermis, and S\. HookerGlobal MMLU: understanding and addressing cultural and linguistic biases in multilingual evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 18761–18799\.External Links:[Link](https://aclanthology.org/2025.acl-long.919/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.919),ISBN 979\-8\-89176\-251\-0Cited by:[§3\.3](https://arxiv.org/html/2610.19185#S3.SS3.SSS0.Px3.p1.1),[Ethical Considerations](https://arxiv.org/html/2610.19185#Sx2.p1.1)\.
- Srivastavaet al\.\(2023\)A\. Srivastava, A\. Rastogi, A\. Rao, A\. A\. M\. Shoeb, A\. Abid, A\. Fisch, A\. R\. Brown, A\. Santoro, A\. Gupta, A\. Garriga\-Alonso, A\. Kluska, A\. Lewkowycz, A\. Agarwal, A\. Power, A\. Ray, A\. Warstadt, A\. W\. Kocurek, A\. Safaya, A\. Tazarv, A\. Xiang, A\. Parrish, A\. Nie, A\. Hussain, A\. Askell, A\. Dsouza, A\. Slone, A\. Rahane, A\. S\. Iyer, A\. J\. Andreassen, A\. Madotto, A\. Santilli, A\. Stuhlmüller, A\. M\. Dai, A\. La, A\. K\. Lampinen, A\. Zou, A\. Jiang, A\. Chen, A\. Vuong, A\. Gupta, A\. Gottardi, A\. Norelli, A\. Venkatesh, A\. Gholamidavoodi, A\. Tabassum, A\. Menezes, A\. Kirubarajan, A\. Mullokandov, A\. Sabharwal, A\. Herrick, A\. Efrat, A\. Erdem, A\. Karakaş, B\. R\. Roberts, B\. S\. Loe, B\. Zoph, B\. Bojanowski, B\. Özyurt, B\. Hedayatnia, B\. Neyshabur, B\. Inden, B\. Stein, B\. Ekmekci, B\. Y\. Lin, B\. Howald, B\. Orinion, C\. Diao, C\. Dour, C\. Stinson, C\. Argueta, C\. Ferri, C\. Singh, C\. Rathkopf, C\. Meng, C\. Baral, C\. Wu, C\. Callison\-Burch, C\. Waites, C\. Voigt, C\. D\. Manning, C\. Potts, C\. Ramirez, C\. E\. Rivera, C\. Siro, C\. Raffel, C\. Ashcraft, C\. Garbacea, D\. Sileo, D\. Garrette, D\. Hendrycks, D\. Kilman, D\. Roth, C\. D\. Freeman, D\. Khashabi, D\. Levy, D\. M\. González, D\. Perszyk, D\. Hernandez, D\. Chen, D\. Ippolito, D\. Gilboa, D\. Dohan, D\. Drakard, D\. Jurgens, D\. Datta, D\. Ganguli, D\. Emelin, D\. Kleyko, D\. Yuret, D\. Chen, D\. Tam, D\. Hupkes, D\. Misra, D\. Buzan, D\. C\. Mollo, D\. Yang, D\. Lee, D\. Schrader, E\. Shutova, E\. D\. Cubuk, E\. Segal, E\. Hagerman, E\. Barnes, E\. Donoway, E\. Pavlick,et al\.Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=uyTL5Bvosj)Cited by:[§3\.1\.1](https://arxiv.org/html/2610.19185#S3.SS1.SSS1.Px3.p1.2)\.
- Team Olmoet al\.\(2025\)Team Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. Morrison, J\. Poznanski, K\. Lo, L\. Soldaini, M\. Jordan, M\. Chen, M\. Noukhovitch, N\. Lambert, P\. Walsh, P\. Dasigi, R\. Berry, S\. Malik, S\. Shah, S\. Geng, S\. Arora, S\. Gupta, T\. Anderson, T\. Xiao, T\. Murray, T\. Romero, V\. Graf, A\. Asai, A\. Bhagia, A\. Wettig, A\. Liu, A\. Rangapur, C\. Anastasiades, C\. Huang, D\. Schwenk, H\. Trivedi, I\. Magnusson, J\. Lochner, J\. Liu, L\. J\. V\. Miranda, M\. Sap, M\. Morgan, M\. Schmitz, M\. Guerquin, M\. Wilson, R\. Huff, R\. Le Bras, R\. Xin, R\. Shao, S\. Skjonsberg, S\. Z\. Shen, S\. S\. Li, T\. Wilde, V\. Pyatkin, W\. Merrill, Y\. Chang, Y\. Gu, Z\. Zeng, A\. Sabharwal, L\. Zettlemoyer, P\. W\. Koh, A\. Farhadi, N\. A\. Smith, and H\. HajishirziOlmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[§3\.4](https://arxiv.org/html/2610.19185#S3.SS4.SSS0.Px2.p1.1)\.
- Yagoubiet al\.\(2025\)M\. Yagoubi, Y\. Dahou, B\. Mokeddem, Y\. Belkada, P\. H\. Le\-Khac, B\. E\. A\. Boussaha, R\. Alami, J\. Zuo, D\. Marsili, M\. Farooq, M\. Lalmas, G\. Gkioxari, P\. Gallinari, P\. Torr, and H\. HacidNeurIPS 2025 E2LM competition : early training evaluation of language models\.External Links:2506\.07731,[Link](https://arxiv.org/abs/2506.07731)Cited by:[§2](https://arxiv.org/html/2610.19185#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AAppendix: Inference Details

We obtain the raw results using the LM Evaluation Harness\([Gao et al\., 2023](https://arxiv.org/html/2610.19185#bib.bib15)\)with vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.19185#bib.bib31)\)as the backend\. Each evaluation job runs one model checkpoint on a single GPU without tensor parallelism\. The 2B models are loaded in their default precision and the larger LLMs in bfloat16, using the intermediate checkpoints released by their developers\. For multiple\-choice tasks, the harness computes the log\-likelihood of every answer option given the context and, for PMI, given an empty context\. The metrics in §[3\.1\.2](https://arxiv.org/html/2610.19185#S3.SS1.SSS2.Px3)are derived from these log\-likelihoods\. Generative tasks are decoded greedily or by sampling \(§[3\.1](https://arxiv.org/html/2610.19185#S3.SS1)\)\. The few\-shot examples are drawn with a fixed seed, so that all checkpoints of all models see the same examples\. The harness writes one result file per model checkpoint, task, prompt, formulation and number of shots, from which we extract the metric values\.

All evaluations were run on NVIDIA GH200 GPUs \(120 GB\)\. Evaluating the 2B models took roughly 11,000 GPU hours, most of them for the Norwegian benchmark with its many prompts, formulations and decoding strategies\. Evaluating the three larger LLMs took roughly 7,000 GPU hours in total\. Computing the meta\-measure scores for all setups takes around 5 minutes on a CPU\.

## Appendix BAppendix: From Meta\-Measure Scores to Setup Choices

##### Meta\-measure matrices\.

For each language and task type, we enumerate all combinations of the options of the aspects \(§[3\.1](https://arxiv.org/html/2610.19185#S3.SS1)\) that are available in that language\. For each setupee, we compute the scoresf,ts\_\{f,t\}of every checkpoint as described in §[3\.2](https://arxiv.org/html/2610.19185#S3.SS2): we select the metric and normalization ofeefor every task the setup covers, apply baseline rescaling to the score of each prompt, aggregate over the prompts ofee\(the first template only, or the mean or maximum over all templates\), applyz\-score or min\-max rescaling to the task scores, and aggregate over tasks or task categories\. From these scores, we compute the three meta\-measures\. This yields a matrix of meta\-measure scoresMμ,l​\(e\)M\_\{\\mu,l\}\(e\)with one entry per meta\-measureμ\\mu, languagelland setupee\. An entry is undefined if languagelldoes not offer an option ofee\. For the 2B models, the classification matrix covers 1,488 setups and the generation matrix \(Norwegian\) 96 setups\.

##### Overall score\.

Within each language, we convert each meta\-measure to its percentile rank among the setups of that language, with ties receiving their average rank, and define the overall score of a setup as the mean of its three percentile ranks:

Ol​\(e\)=13​∑μpctl​\(Mμ,l​\(e\)\)\.O\_\{l\}\(e\)=\\frac\{1\}\{3\}\\sum\_\{\\mu\}\\mathrm\{pct\}\_\{l\}\\big\(M\_\{\\mu,l\}\(e\)\\big\)\.\(16\)

##### Best complete setup\.

Within a language, the best complete setup isarg⁡maxe​Ol​\(e\)\\arg\\max\_\{e\}O\_\{l\}\(e\)\. Pooled over languages, it maximizes the mean overall score1\|ℒ\|​∑lOl​\(e\)\\frac\{1\}\{\|\\mathcal\{L\}\|\}\\sum\_\{l\}O\_\{l\}\(e\)over the setups that every language has\. For languages without an option in an aspect \(e\.g\., French and Spanish for formulation\), that aspect is ignored: we let them contribute the same score to all of its options, so the aspect is decided by the other languages\.

##### Option ranking\.

For an aspectaaand one of its optionsoo, the option ranking averages the overall score over all setups in which aspectaatakes optionoo, i\.e\., over all combinations of the other aspects:

Rl\(a,o\)=meane:ea=oOl\(e\)\.R\_\{l\}\(a,o\)=\\operatorname\{mean\}\_\{e:\\,e\_\{a\}=o\}O\_\{l\}\(e\)\.\(17\)Pooled over languages,R⁡\(a,o\)R\(a,o\)is the mean ofRl​\(a,o\)R\_\{l\}\(a,o\)over the languagesℒa\\mathcal\{L\}\_\{a\}that offer an option in aspectaa\. Combining the best option of each aspect gives the setup reported as the option\-ranking\.

##### Per\-option win rate\.

For an aspectaaand two of its optionsooando′o^\{\\prime\}, a*context*ccis a combination of options of all other aspects, andecoe\_\{c\}^\{o\}andeco′e\_\{c\}^\{o^\{\\prime\}\}are the two setups that differ only in aspectaa\. Within a language and for each meta\-measure, optionoowins a context if it has the higher meta\-measure score, with ties counting half:

wμ,l​\(c,o,o′\)=\{1if​mco\>mco′,12if​mco=mco′,0otherwise,w\_\{\\mu,l\}\(c;o,o^\{\\prime\}\)=\\begin\{cases\}1&\\text\{if \}m\_\{c\}^\{o\}\>m\_\{c\}^\{o^\{\\prime\}\},\\\\ \\tfrac\{1\}\{2\}&\\text\{if \}m\_\{c\}^\{o\}=m\_\{c\}^\{o^\{\\prime\}\},\\\\ 0&\\text\{otherwise,\}\\end\{cases\}\(18\)withmco=Mμ,l​\(eco\)m\_\{c\}^\{o\}=M\_\{\\mu,l\}\(e\_\{c\}^\{o\}\)\. The win rate of optionoois the share of all its comparisons with the other options of aspectaathat it wins:

Wμ,l​\(a,o\)=∑o′≠o∑c∈Cl​\(o,o′\)wμ,l​\(c,o,o′\)∑o′≠o\|Cl​\(o,o′\)\|,W\_\{\\mu,l\}\(a,o\)=\\frac\{\\sum\_\{o^\{\\prime\}\\neq o\}\\sum\_\{c\\in C\_\{l\}\(o,o^\{\\prime\}\)\}w\_\{\\mu,l\}\(c;o,o^\{\\prime\}\)\}\{\\sum\_\{o^\{\\prime\}\\neq o\}\|C\_\{l\}\(o,o^\{\\prime\}\)\|\},\(19\)whereCl​\(o,o′\)C\_\{l\}\(o,o^\{\\prime\}\)is the set of contexts in which both options exist in languagell\. We averageWμ,l​\(a,o\)W\_\{\\mu,l\}\(a,o\)over the languages inℒa\\mathcal\{L\}\_\{a\}and over the three meta\-measures, and select the option with the highest win rate\. Unlike option ranking, which uses the overall score, the win rate compares the raw meta\-measure values and ignores the size of the differences\.

## Appendix CAppendix: Task Coverage of Evaluation Setups

[TableC\.1](https://arxiv.org/html/2610.19185#A3.T1)lists, for each aspect, the compared options and the number of tasks per language on which they are evaluated\. A setup covers the tasks for which all of its options are available, so the number of tasks can differ between the options of an aspect\.

Table C\.1:Options of each evaluation aspect inclassification tasksand the number of tasks per language on which they are evaluated \(–: unavailable\)\. Where the number of tasks differs between options, the numbers are separated by slashes in the order of the options\. Metric and log\-probability normalization are given as one row per metric \(Accuracy, CondProb, AnsProb\) with the three normalizations\. Finnish and Norwegian setups cover the multiple\-choice tasks with formulation variants \(10 of 12 and 14 of 17\), while French and Spanish have no formulation variants\.Generation tasks\(omitted in the table\) are compared for the 18 Norwegian tasks only, with both options of greedy decoding and random sampling\.
## Appendix DAppendix: Language\-Wise Results

[TableD\.1](https://arxiv.org/html/2610.19185#A4.T1)lists the options selected by the three setup choice methods for both model scales, pooled over languages and for each language\.[TableD\.2](https://arxiv.org/html/2610.19185#A4.T2)gives the option\-ranking scores for both model scales\.[FigureD\.1](https://arxiv.org/html/2610.19185#A4.F1)shows the per\-option win rates of both model scales under each meta\-measure\.

Table D\.1:Options selected by the three methods \(B: best complete setup, O: option ranking, W: per\-option win rate\) for classification tasks, pooled over languages and per language, for the 2B models and the larger LLMs\.Bold: the method selects a different option than the best complete setup\. Grey: the larger LLMs get a different option than the 2B models from the same method\. AP: answer probability, CP: conditional answer probability, Acc: accuracy; 5, 0: number of shots; max, mean, sgl: multi\-max, multi\-mean and single prompt; base, z, mm: baseline,z\-score and min\-max rescaling; mean: task\-level mean, cat: category mean; hyb: hybrid; –: no choice \(no formulation variants\)\.Table D\.2:Option\-ranking scores \(mean overall score of all setups with the option, over all combinations of the other aspects\), averaged over languages, for the 2B models and the larger LLMs\. Options are ordered by the 2B model scores\. Bold marks the highest score per aspect and model scale, before rounding into two decimals\.†Norwegian only\.‡Finnish and Norwegian\.Figure D\.1:Per\-option win rates of the 2B models \(circle\) and larger LLMs \(diamond\) pooled over languages and task types; win rates are computed for each meta\-measure are visualized in the three columns: monotonicity, non\-randomness and ranking consistency\.
## Appendix EAppendix: When the Meta\-Measures Disagree

AspectMonoNonRandRankConsClassification tasksFormulationCF 97CF 69CF 63MetricAnsProb 92AnsProb 95AnsProb 92Normalizationchar 64PMI 66none 57Shots5\-shot 845\-shot 615\-shot 63Promptsmulti\-mean 67multi\-max 82tie \(53\)Rescalingbaseline 65baseline 73tie \(52\)Task agg\.mean 60mean 65mean 53Generative tasks \(Norwegian\)Shots5\-shot 1000\-shot 675\-shot 86Promptsmulti\-max 52multi\-mean 67single 62Rescalingnone 80z\-score 81none 61Task agg\.mean 71category 98mean 59Decodingsampling 67sampling 56sampling 68

Table E\.1:Option with the highest win rate \(%\) per meta\-measure, pooled over languages, for the 2B models\.The overall score assigns equal weight to the three meta\-measures\. To examine disagreements among them, we determine for each aspect the option with the highest win rate under each meta\-measure separately \([TableE\.1](https://arxiv.org/html/2610.19185#A5.T1)\)\. For the 2B models, the three meta\-measures select the same option for formulation, metric, number of shots and task aggregation, i\.e\., for most of the aspects where all three setup choice methods agree \([TableD\.1](https://arxiv.org/html/2610.19185#A4.T1)\)\. They disagree on log\-probability normalization and prompt aggregation, while ranking consistency provides little separation among rescaling options\. Disagreement is more pronounced for Norwegian generative tasks, occurring for four of the five aspects\.

##### Example: maximum versus mean prompt aggregation\.

We compare 672 pairs of setups that differ only in whether scores across prompts are aggregated by the maximum or the mean\. Maximum aggregation yields higher non\-randomness in 80% of pairs, whereas mean aggregation yields higher monotonicity in 58% and slightly higher ranking consistency\. The non\-randomness advantage of maximum aggregation is driven primarily by its numerator: the gap between the best final score and chance\. This gap is larger under maximum aggregation in 94% of pairs and by 22% at the median, while the corresponding noise is only 7% larger\. However, this advantage partly results mechanically from taking a maximum\. Because scores vary across prompts, the maximum over several prompts exceeds their mean even when the prompts are equally informative, including for a model performing at chance\. At the first checkpoint \(2B tokens\), for example, maximum accuracy already exceeds mean accuracy by 3\.0 points on average, compared with a 5\.3\-point difference at the end of training\. Because non\-randomness compares this maximum with the chance level of a single prompt, part of the observed advantage reflects the aggregation procedure rather than additional signal\. Mean aggregation, by contrast, reduces prompt\-level variation and exhibits slightly stronger monotonicity\.

##### Implications\.

Agreement among the meta\-measures indicates that a recommendation is insensitive to their relative weighting, as is the case for most high\-impact choices\. When the meta\-measures disagree, the preferred option depends on the desired property\. For prompt aggregation, mean aggregation better captures monotonic training progress, whereas maximum aggregation yields greater separation from the single\-prompt chance baseline\. Such disagreements occur primarily for aspects whose options have similar overall scores \(§[4](https://arxiv.org/html/2610.19185#S4)\), helping explain their relatively low impact on average\. More generally, a meta\-measure can be inflated by a score transformation without a corresponding increase in useful signal\. Maximum aggregation provides one example\. Examining individual meta\-measures and for non\-randomness, its constituent terms can reveal such effects\. Finally, disagreements are stronger for the generative tasks\. For example, category\-mean aggregation wins 98% of comparisons under non\-randomness but only 29% under monotonicity\. These results motivate combining multiple meta\-measures rather than relying on any single criterion, while explicitly reporting cases where they disagree\.