When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

# When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Source: [https://arxiv.org/html/2602.16763](https://arxiv.org/html/2602.16763)
Anka ReuelPrajna SoniSanchit AhujaPawan Sasanka AmmanamanchiRuchit RawalVilém ZouharSrishti YadavChenxi WhitehouseDayeon KiJennifer MickelLeshem ChoshenMarek ŠuppaJan BatznerJenny ChimJeba SaniaYanan LongHossein A\. RahmaniChristina KnightYiyang NanJyoutir RajYu FanShubham SinghSubramanyam SahooEliya HabbaUsman GoharSiddhesh PawarRobert ScholzArjun SubramonianJingwei NiMykel J\. KochenderferSanmi KoyejoMrinmaya SachanStella BidermanZeerak TalatAvijit GhoshIrene Solaiman

###### Abstract

Artificial intelligence benchmarks are an important mechanism for measuring model progress and guiding deployment decisions\. However, benchmarks quickly “saturate”, making it difficult to differentiate models and diminishing their long\-term value\. In this study, we definebenchmark saturationand analyze it across 60 language model benchmarks using 14 properties that relate to saturation\. We find that nearly half of the our benchmarks exhibit saturation, with rates increasing with age\. Further, we find that resilience to saturation is impacted by expert\-curation, not by public test data\. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches\.111Data and code are available in the[Github repository](https://github.com/evaleval/benchmark-saturation)\.

Machine Learning, ICML

\* Lead authors⋄\\diamondTop contributors†\\daggerAdvisors

This project was completed as part of the Evaluating Evaluations \(EvalEval\) Coalition:![[Uncaptioned image]](https://arxiv.org/html/2602.16763v3/images/logo-square.png)[https://evalevalai\.com/](https://evalevalai.com/)

## 1Introduction

Artificial Intelligence \(AI\) benchmarks play a central role in measuring model progress, guiding deployment decisions, and informing policy and regulation\(Hardyet al\.,[2025](https://arxiv.org/html/2602.16763#bib.bib30),[2024](https://arxiv.org/html/2602.16763#bib.bib16); Alzahraniet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib14); Union,[2024](https://arxiv.org/html/2602.16763#bib.bib56)\)\. Their value depends on their ability to distinguish between models\. Yet many widely used benchmarks \(e\.g\.,HumanEval\(Chen,[2021](https://arxiv.org/html/2602.16763#bib.bib59)\)\) have rapidly “saturated”\(Maslejet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib57)\), with top\-performing systems achieving near\-identical scores\. When performance converges within a narrow range, benchmarks lose discriminative power and provide limited guidance for model comparison or selection\(Ottet al\.,[2022](https://arxiv.org/html/2602.16763#bib.bib32); Chenet al\.,[2025](https://arxiv.org/html/2602.16763#bib.bib58)\)\. Similar dynamics have been observed in other domains–for example, ImageNet\(Denget al\.,[2009](https://arxiv.org/html/2602.16763#bib.bib9)\)exhibits near\-ceiling performance for most new models\.222We want to emphasize that saturation of benchmarks is not always negative–if the benchmark was valid\(Salaudeenet al\.,[2025](https://arxiv.org/html/2602.16763#bib.bib136)\), saturation means that a task can be considered “solved”\.

Despite its importance,*benchmark saturation*has received limited systematic study\. Prior work often notes performance plateaus, increased robustness\(Ashury\-Tahanet al\.,[2026a](https://arxiv.org/html/2602.16763#bib.bib3)\)or introduces new benchmarks in response\(Wanget al\.,[2024b](https://arxiv.org/html/2602.16763#bib.bib45); Jimenezet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib75)\), but rarely analyzes the mechanisms driving saturation\. It remains unclear why some benchmarks saturate quickly while others retain discriminative power, and there is no agreed\-upon operational definition–whether saturation reflects near\-human performance, fixed ceilings, or the loss of statistical separability among state\-of\-the\-art models\. We address these gaps by defining saturation as the loss of reliable discriminative power among top\-performing models and operationalizing it through an uncertainty\-aware saturation index derived from leaderboard data\. Using this framework, we analyze 60 widely used text\-based LLM benchmarks across domains and evaluation settings, annotated along dimensions such as task design, linguistic scope, data construction, and accessibility to study factors associated with saturation\.

This paper makes the following contributions:

- •We define benchmark saturation as the loss of reliable discriminative power among state\-of\-the\-art models and introduce a reproducible, uncertainty\-aware saturation index derived from leaderboard data\.
- •We identify which benchmark properties are systematically associated with saturation based on an analysis of 60 benchmarks\. We find that commonly assumed safeguards, such as private test sets or closed\-ended formats, have limited impact on saturation, while benchmark age and scale strongly predict it\.
- •We derive practical recommendations for benchmark design and lifecycle management, including monitoring practices, uncertainty reporting, and criteria for benchmark retirement or revision\.

The remaining paper is organized as follows:[Section˜2](https://arxiv.org/html/2602.16763#S2)formalizes benchmark saturation and introduces our saturation index;[Section˜3](https://arxiv.org/html/2602.16763#S3)outlines benchmark collection and annotation;[Section˜4](https://arxiv.org/html/2602.16763#S4)presents the empirical analyses;[Section˜5](https://arxiv.org/html/2602.16763#S5)discusses implications and actionable recommendations;[Section˜6](https://arxiv.org/html/2602.16763#S6)concludes with limitations and future directions\.

#### Conflict of Interest Disclosure

This work was conducted as part of a research coalition, some of whose members \(including coauthors\) have contributed to models or reported evaluations analyzed in this study\. All artifacts were subject to the same inclusion and annotation procedure, regardless of author involvement\.

## 2Conceptualizing Benchmark Saturation

In this section, we formally define benchmark saturation, introduce our uncertainty\-aware saturation index, and analyze its robustness to key parameter choices\.

### 2\.1Definition and Scope

We define*benchmark saturation*as the loss of reliable discriminative power among top\-performing models under comparison\. A benchmark is saturated when top\-performing models cannot be statistically distinguished and performance approaches the empirically observed ceiling of the benchmark\. This notion corresponds to what prior work informally describes as*performance saturation*–a plateau where inter\-model differences become negligible\(Justen,[2025](https://arxiv.org/html/2602.16763#bib.bib23); Wanget al\.,[2024b](https://arxiv.org/html/2602.16763#bib.bib45); Ottet al\.,[2022](https://arxiv.org/html/2602.16763#bib.bib32)\)\.

#### Human performance ceiling\.

Unlike definitions based on reaching human\-level performance\(Guptaet al\.,[2025](https://arxiv.org/html/2602.16763#bib.bib19)\), our definition does not rely on human baselines, which are often impossible to comprehensively obtain, unavailable, or inconsistently measured\(Weiet al\.,[2025](https://arxiv.org/html/2602.16763#bib.bib33)\)\. Moreover, human\-level performance does not imply saturation, as models may still be statistically distinguishable even after reaching human\-level scores, allowing the benchmark to retain discriminative power\. Previous analyses describe saturation patterns descriptively\(Ottet al\.,[2022](https://arxiv.org/html/2602.16763#bib.bib32)\)or emphasize lifecycle management\(Hardyet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib16)\), but do not provide a quantitative criterion to determine saturation\.

#### Saturation vs\. stagnation\.

We therefore formalize saturation as a measurable property derived from leaderboard uncertainty\. We further distinguish*stagnation*from saturation: stagnation refers to statistical indistinguishability among top models, whereas saturation additionally requires that performance is near the empirical ceiling\. In practice, limited noise estimates blur the distinction between the two\.

Definition: Benchmark SaturationA benchmark is*saturated*if the evaluated models can not be reliably distinguished by their performance scores and any further improvements are not statistically distinguishable under the evaluation protocol\. Formally, saturation is characterized by: \(1\) statistically alike performance among different top\-performing models \(2\) top performing models are approaching the benchmark’s empirically inferred ceiling\.

If only condition \(1\) holds, we refer to the benchmark as being*stagnated*rather than saturated\. In this case, observed indistinguishability may arise from model\-level limitations, evaluation noise, insufficient benchmark sensitivity, or artifacts in the benchmark itself \(e\.g\., spurious correlations or repetitive patterns\) and may be overcome by future architectural, training, or evaluation advances\. It is often difficult to clearly distinguish stagnation from saturation, as reliable estimates of evaluation noise and benchmark ceilings are rarely available\.

Our operationalization should satisfy four desiderata:

1. 1\.Model\-relative:Defined with respect to top\-performing models at a given time\.
2. 2\.Metric\-agnostic:Applicable across common metrics \(accuracy, F1, BLEU\)\.
3. 3\.Data\-driven:Avoids reliance on externally curated performance ceilings\.
4. 4\.Reproducible:Produces identical decisions given the same leaderboard snapshot\.

To formalize this notion, we consider the performance of a set of top\-performing models on each benchmark\. For a given benchmark, lets1≥⋯≥sks\_\{1\}\\geq\\dots\\geq s\_\{k\}denote the scores of the topkkmodels \(defaultk=5k=5\)\. We introducekkas a general parameter to avoid fixing the number of models considered, and to flexibly define the set of top\-performing models used to assess saturation\. In our analysis, we fixk=5k=5to ensure comparability across benchmarks\. This choice reflects a practical trade\-off: smaller values ofkkcan lead to unstable estimates, while larger values risk mixing frontier models with older or less relevant ones, particularly given incomplete leaderboard coverage\. Empirically, most benchmarks in our dataset report on approximately 5–7 recent, highly capable models, makingk=5k=5a reasonable and consistent choice\.

### 2\.2Uncertainty\-Aware Saturation Measurement

#### Performance\-based evaluation\.

For accuracy\-like metrics that are averages over a fixed test set of sizenn, we approximate the standard error of a model scoressas

SE​\(s\)≈s​\(1−s\)neff\.\\mathrm\{SE\}\(s\)\\approx\\sqrt\{\\frac\{s\(1\-s\)\}\{n\_\{\\mathrm\{eff\}\}\}\}\.\(1\)where​neff=nα,α∈\[0,1\],default​α=0\.5,\\text\{where \}n\_\{\\mathrm\{eff\}\}=n^\{\\alpha\},\\qquad\\alpha\\in\[0,1\],\\ \\text\{default \}\\alpha=0\.5,
Note that accuracy\-like metrics, metrics computed as averages over a fixed set of test samples with bounded per\-sample contributions \(e\.g\., accuracy, F1, BLEU\) are broadly used\. In such metrics, uncertainty can be approximated from finite\-sample variability\. For other metric types \(e\.g\., Pass@k\), the same framework is applicable but requires a benchmark\-specific uncertainty estimate, such as bootstrap intervals or repeated\-evaluation variance\.

The effective test set sizeneff=nαn\_\{\\mathrm\{eff\}\}=n^\{\\alpha\}down weights the nominal test set sizennto avoid an overly strong dependence of the saturation calculation on test set size\. In our dataset, benchmark sizes vary substantially, ranging from a few dozen to several hundred thousand test samples, with a highly skewed distribution due to a small number of very large benchmarks\. Using the raw test set size n would therefore cause the uncertainty term to be dominated by these outliers, leading to disproportionately small standard errors and artificially low saturation estimates for large benchmarks\.

Thus, the standard error of the difference between the top model andkk\-th model is then

SEΔ≈s1​\(1−s1\)neff\+sk​\(1−sk\)neff\.\\mathrm\{SE\}\_\{\\Delta\}\\approx\\sqrt\{\\frac\{s\_\{1\}\(1\-s\_\{1\}\)\}\{n\_\{\\mathrm\{eff\}\}\}\+\\frac\{s\_\{k\}\(1\-s\_\{k\}\)\}\{n\_\{\\mathrm\{eff\}\}\}\}\.\(2\)LetΔ=s1−sk\\Delta=s\_\{1\}\-s\_\{k\}\. We consider the top models to be statistically similar in performance ifΔ≤z⋅SEΔ\\Delta\\leq z\\cdot\\mathrm\{SE\}\_\{\\Delta\}, wherezzis a standard normal quantile \(e\.g\.,z=1\.96z=1\.96for a 95% confidence level\)\. This criterion considers both dataset size and evaluation noise\. Evaluation uncertainty refers to the expected variability in leaderboard scores introduced by finite test set size and metric estimation noise\. We define this uncertainty through the standard error of model scores and their differences, and treat performance differences within this range as statistically indistinguishable\.

#### Score compression\.

To quantify to which degree performance scores at the top of the leaderboard are collapsing, we compute the normalized score range

Rnorm=s1−skSEΔ\.R\_\{\\mathrm\{norm\}\}=\\frac\{s\_\{1\}\-s\_\{k\}\}\{\\mathrm\{SE\}\_\{\\Delta\}\}\.\(3\)
RnormR\_\{\\mathrm\{norm\}\}can be interpreted as a signal\-to\-noise ratio, comparing observed top\-model score spread to expected evaluation uncertainty\. LowerRnormR\_\{\\mathrm\{norm\}\}indicates greater saturation, with top\-model differences falling within expected evaluation uncertainty and showing limited discrimination\.333In rare cases with near\-zero uncertainty \(e\.g\., deterministic near\-perfect scores\), we add a smallϵ\\epsilon\-stabilization in the denominator to avoid numerical instability\.

#### Empirical approximation of the noise ceiling\.

Rather than assuming a fixed or externally defined noise ceiling, we treat the highest observed model performance \(s1s\_\{1\}\) as an empirical proxy for the ceiling\. Saturation is therefore assessed relative to the distribution of observed model scores, rather than with respect to an absolute performance target such as perfect accuracy \(i\.e\., accuracy of 100%\)\.

Strong clustering of top models at a low performance level should not be interpreted as the task being solved\. Instead, such clustering indicatesmodel\-level saturation: the benchmark may no longer effectively distinguish between contemporary state\-of\-the\-art models\. However, as observed in prior benchmarks, this form of saturation reflects stagnation and does not preclude the benchmark from regaining discriminative power following paradigm shifts \(e\.g\.,introduction of reasoning\-centric or tool\-augmented models\)\(Cobbeet al\.,[2021](https://arxiv.org/html/2602.16763#bib.bib29); Lewkowyczet al\.,[2022](https://arxiv.org/html/2602.16763#bib.bib28)\)\.

#### Saturation index\.

To capture saturation as a graded phenomenon, we combine the above signals into a continuous saturation indexSindex∈\[0,1\]S\_\{\\text\{index\}\}\\in\[0,1\], which increases as top models become statistically indistinguishable\. Benchmarks with higher values ofSindexS\_\{\\text\{index\}\}show stronger saturation evidence\. We define the saturation index as

Sindex=exp⁡\(−Rnorm2\),S\_\{\\text\{index\}\}=\\exp\(\-R\_\{\\mathrm\{norm\}\}^\{2\}\),\(4\)which assigns high values when the performance differences are small relative to the evaluation uncertainty\. High values ofSindexS\_\{\\text\{index\}\}indicate benchmarks where top\-performing models are tightly clustered within evaluation noise, reflecting reduced discriminative power\.

For interpretability, we bucket benchmarks into five bins:*very low*\(<0\.01<0\.01\),*low*\(\[0\.01,0\.3\)\[0\.01,0\.3\)\),*moderate*\(\[0\.3,0\.7\)\[0\.3,0\.7\)\),*high*\(\[0\.7,0\.9\)\[0\.7,0\.9\)\), and*very high*saturation \(≥0\.9\\geq 0\.9\)\. Notably, high saturation may also occur at lower absolute performance levels, reflecting model\-level saturation rather than task\-level completion\. These bins are interpretable, empirically motivated ranges over a continuous score, intended to summarize broad saturation regimes rather than define strict thresholds\. They reflect the spread ofSi​n​d​e​xS\_\{index\}observed across benchmarks while preserving the index’s continuity\.

### 2\.3Sensitivity to Parameter Selection

Table 1:Sensitivity analysis of saturation index with respect tokkandα\\alpha\. We report Spearman rank correlation and the percentage of benchmarks assigned to the same saturation bin\.We conduct a sensitivity analysis with varyingk∈3,5k\\in\{3,5\}andα∈0,0\.5,1\\alpha\\in\{0,0\.5,1\}values\. Across these settings, the resulting saturation indices remain highly correlated, indicating that the relative ranking of benchmarks is preserved\.[Table˜1](https://arxiv.org/html/2602.16763#S2.T1)gives an overview of correlation and the fraction of benchmarks that remain in the same bins\. While we observe variation in bin assignments, most changes occur between neighbouring bins rather than large shifts, which suggests that the underlying signal is stable even when scores vary\. We further observe that a smaller k\-value \(e\.g\.,k=3k=3\) increases variance due to limited model coverage, while largerkkrisks mixing frontier and non\-frontier models given incomplete, static leaderboard data\. Similarly,α=1\\alpha=1leads to strong dependence on test set size, whereasα=0\\alpha=0ignores evaluation uncertainty\. The choiceα=0\.5\\alpha=0\.5is a balanced trade\-off, moderating dataset size effects while preserving uncertainty awareness\. Overall, absolute saturation values may shift slightly, but benchmark ordering remains stable\.

## 3Methodology

Table 2:Hypotheses on factors driving benchmark saturation\.![Refer to caption](https://arxiv.org/html/2602.16763v3/images/saturation_scores_scatter.png)Figure 1:Overview of saturation scores across all studied benchmarks, ranked by their saturation levels\.To study benchmark saturation, we combine structured benchmark annotations with leaderboard\-based analysis\.

### 3\.1Benchmark Collection and Annotation

#### \(1\) Initial benchmark selection\.

We used a three\-stage, criteria\-driven process to construct a representative benchmark set, focusing on benchmarks that \(i\) are actively used in contemporary LLM evaluation, \(ii\) provide sufficient longitudinal data, and \(iii\) vary along dimensions relevant to our hypotheses\.

We compiled candidate benchmarks from two sources:*1\. Evaluation reports from major model developers\.*We extracted benchmarks appearing in evaluation sections of official reports \(such as model cards or technical reports\) released between Jan 2022 and Nov 2025 by major developers, including OpenAI, Anthropic, Google, Meta, and Alibaba, to reflect real\-world evaluation practices and downstream adoption\. In total, we reviewed 61 documents and identified 190 benchmarks used in at least one report\.*2\. Highly\-cited benchmark papers\.*We additionally collected widely cited benchmarks via the Semantic Scholar API using keyword\-based search \(details in Appendix[C](https://arxiv.org/html/2602.16763#A3)\)\.

#### \(2\) Criteria\-based filtering\.

We filtered benchmarks to ensure suitability for analysis using the following criteria:*1\. Public documentation:*Benchmark documentation \(e\.g\.,paper, technical report, or website\) must be publicly available\.*2\. Sustained usage:*Benchmarks extracted from developer reports must appear in at least five distinct reports to ensure broader relevance\.*3\. Clear evaluation protocol:*Benchmarks with ambiguous scoring, inconsistent splits, or unclear evaluation procedures were excluded\.*4\. Text\-only scope:*We restricted our analysis to text\-based benchmarks, excluding multimodal datasets to isolate language\-related saturation effects\.*5\. Available leaderboard data:*We included only benchmarks with sufficiently up\-to\-date leaderboard data and multiple evaluated models, otherwise they were excluded \(e\.g\.,BIG\-Bench\(Srivastavaet al\.,[2023](https://arxiv.org/html/2602.16763#bib.bib121)\)\)\.

#### \(3\) Hypothesis\-driven refinement\.

We initially develop a set of hypotheses for potential causes of benchmark saturation \(see[Table˜2](https://arxiv.org/html/2602.16763#S3.T2)and[Appendix˜B](https://arxiv.org/html/2602.16763#A2)\)\. To ensure adequate sample sizes across hypotheses, we supplemented the filtered set with benchmarks filling gaps along key dimensions \(e\.g\.,multilingual, templated, open\-ended\)\. We conducted targeted Google Scholar searches using terms such asAI benchmark,leaderboard,evaluation, anddataset, combined with hypothesis\-specific keywords \(e\.g\.,multilingual,open\-ended generation\)\.

After filtering and refinement, the final dataset consists of60 benchmarks\. See[Table˜3](https://arxiv.org/html/2602.16763#A3.T3)for the full list\.

#### \(4\) Annotation protocol\.

To test the hypotheses in[Table˜2](https://arxiv.org/html/2602.16763#S3.T2), we annotated benchmarks according to the schema in[Table˜4](https://arxiv.org/html/2602.16763#A4.T4)\. Annotations capture: \(i\)*temporality*\(e\.g\.,release date\), \(ii\)*saturation metrics*\(e\.g\.,top\-5 model scores\), \(iii\)*data quality indicators*, \(iv\)*task structure*\(e\.g\.,input/output format\), and \(v\)*dataset properties*\(e\.g\.,curation strategy\)\. Annotations were collected through a structured protocol involving 23 researchers with expertise in dataset curation and evaluation\. Each benchmark was independently annotated and secondarily reviewed using a predefined schema, followed by a final cross\-benchmark consistency audit to resolve remaining ambiguities\.

#### Final benchmark set\.

Our benchmark selection spans a broad range of evaluation settings, including knowledge and reasoning tasks, multilingual, coding, long\-context and factuality benchmarks, and recent agentic tasks\. The benchmarks vary substantially in age \(between 1 and 114 months\), scale \(from a few to hundreds of thousands of test samples\), accessibility, output format, and construction style\. Overall, the set includes 56 public and 4 private benchmarks, 44 English\-only and 16 multilingual benchmarks, 28 closed\-ended and 31 open\-ended benchmarks, and 14 templated versus 46 non\-templated benchmarks \([Figure˜1](https://arxiv.org/html/2602.16763#S3.F1)\)\.

## 4Empirical Analysis of Benchmark Saturation

![Refer to caption](https://arxiv.org/html/2602.16763v3/x1.png)Figure 2:Analysis of benchmark saturation \(N=60N=60\)\.The figure is organized into two groups:Group A \(Left\)focuses on age\-balanced categories \(H1,H5,H6H\_\{1\},H\_\{5\},H\_\{6\}\), whileGroup B \(Right\)examines temporal dynamics \(H2,H3,H4H\_\{2\},H\_\{3\},H\_\{4\}\), revealing that performance gaps in these categories are often driven by benchmark maturity\. For each hypothesis, the first column of the group displays raw saturation rates\. In the scatter plots, point colors correspond to the categories defined in the adjacent bar plots \(legends omitted for brevity\);×\\timesdenotes saturated and∘\\circdenotes non\-saturated benchmarks\.We analyze saturation patterns across 60 text\-based LLM benchmarks spanning domains, task formats, and evaluation settings\. Using our saturation index \([Section˜2](https://arxiv.org/html/2602.16763#S2)\), we examine \(i\) saturation prevalence, \(ii\) temporal and exposure effects, and \(iii\) differences across benchmark properties\.

### 4\.1Hypotheses\-specific Analysis

We evaluate five hypotheses regarding potential drivers of saturation, grouped by accessibility \(H1H\_\{1\}\), linguistic scope \(H2H\_\{2\}\), data construction and quality \(H3H\_\{3\}\), task design \(H4H\_\{4\}\), popularity \(H5H\_\{5\}\), and template \(H6H\_\{6\}\); see App\.[B](https://arxiv.org/html/2602.16763#A2)for details\. Since benchmark age is itself positively associated with saturation and differs across several benchmark categories, age is an important cofounding factor in cross\-benchmark comparisons\. We therefore distinguish age\-balanced comparisons \(H1H\_\{1\},H5H\_\{5\},H6H\_\{6\}\), where groups have similar maturity, from age\-confounded comparisons \(H2H\_\{2\}–H4H\_\{4\}\) \([Figure˜2](https://arxiv.org/html/2602.16763#S4.F2)\)\.

#### Overall saturation patterns\.

Saturation is widespread\. Of the 60 benchmarks analyzed, 29 exhibit high or very high saturation \(Sindex≥0\.7S\_\{\\text\{index\}\}\\geq 0\.7\), out of which 14 fall into the very high category \(Sindex≥0\.9S\_\{\\text\{index\}\}\\geq 0\.9\)\. These benchmarks show strong compression among top\-performing models, indicating limited discriminative power at the frontier\. Across benchmarks, larger test sets are associated with lower saturation indices\. Benchmarks with more test items show less score compression among top models, consistent with lower evaluation uncertainty and higher resolution\. This relationship persists in joint regression \([Section˜4\.2](https://arxiv.org/html/2602.16763#S4.SS2)\), suggesting that measurement scale impacts discriminative power\.

#### Temporal and exposure effects\.

[Figure˜3](https://arxiv.org/html/2602.16763#S4.F3)shows that the average saturation index increases with benchmark age\. The proportion of saturated benchmarks rises from 42\.9% for benchmarks released within the past 24 months to 54\.5% for those older than 60 months, with corresponding meanSindexS\_\{\\text\{index\}\}values of 0\.51, 0\.52, and 0\.60 across age bins\. While the trend is modest and not statistically significant at conventional thresholds, it is directionally consistent: older benchmarks exhibit greater top\-score compression\. We evaluate benchmark adoption using citation counts and inclusion in industry model release reports\. Raw correlations show that benchmarks with higher citation counts tend to have higher mean saturation indices \([Figure˜4](https://arxiv.org/html/2602.16763#S4.F4)\)\. However, after controlling for benchmark age, citation counts are not significantly associated with saturation \(ρ=0\.22\\rho=0\.22,p=0\.12p=0\.12\)\. Citation growth rates \(ρ=0\.13\\rho=0\.13,p=0\.37p=0\.37\) and frequency of appearance in technical reports \(ρ=0\.05\\rho=0\.05,p=0\.73p=0\.73\) likewise show no significant association\. These results suggest that maturity and cumulative exposure over time, rather than adoption metrics alone, better explain saturation patterns\.

#### Accessibility and task design\.

Public \(N=56N=56\) and private \(N=4N=4\) benchmarks exhibit similar saturation distributions\. We find no statistically meaningful difference inSindexS\_\{\\text\{index\}\}between the two groups\. Hiding test data does not appear to prevent saturation once benchmarks are widely adopted, rejecting hypothesisH1H\_\{1\}\. Output format is age\-balanced \(p=0\.40p=0\.40\)\. We observe no meaningful difference between closed\-ended \(N=28N=28\) and open\-ended \(N=31N=31\) benchmarks, suggesting that generation\-based evaluation does not systematically preserve longer discriminative power\.

#### Benchmark composition and construction\.

English\-only benchmarks \(N=44N=44\) show higher raw saturation rates than multilingual ones \(N=16N=16\)\. However, benchmark age is a clear cofounding factor forH2H\_\{2\}: multilingual benchmarks in our dataset are substantially younger on average \(32\.9 vs\. 48\.9 months\)\. This indicates that the apparent robustness of multilingual benchmarks is largely explained by their young age rather than intrinsic resistance to saturation\. Accordingly, we do not find support forH2H\_\{2\}\. We further examine whether benchmark design choices influence saturation, specifically whether expert\- or human\-curated benchmarks are more robust than crowdsourced or synthetic ones \(H3H\_\{3\}\), and whether non\-templated benchmarks are more resistant than templated benchmarks \(H6H\_\{6\}\)\. Our analysis shows that curation categories differ significantly in age \(p=0\.0017p=0\.0017\)\. Crowdsourced benchmarks are older on average and exhibit higher saturation rates in raw comparisons\. Expert\-curated benchmarks show lower saturation at comparable ages, and several of these benchmarks \(e\.g\.,ARC\-AGI, BIG\-Bench Hard\) remain unsaturated despite prolonged exposure\. Furthermore, templated benchmarks \(N=14N=14\) do not differ significantly from non\-templated ones \(N=46N=46\) in saturation behaviour \(p=0\.10p=0\.10\)\. Literal diversity alone does not appear to determine longevity\. Fully synthetic benchmarks currently exhibit low saturation but are also relatively recent, limiting causal interpretation\. These results suggest that expert\-driven and adversarial design may improve robustness to saturation, though age remains a cofounding factor\.

Benchmarks with documented quality issues \(N=40N=40\) exhibit higher saturation rates than those without \(N=20N=20\), but they are also significantly older on average \(51\.5 vs\. 30\.9 months;p=0\.01p=0\.01\)\. In our annotations, documented quality issues include evidence of contamination, train\-test overlap, noisy or low\-quality examples, mislabeling or answer errors, documented demographic or linguistic imbalances, and other benchmark\-specific problems such as unstable evaluation setups, ambiguity, or missing context\. The association is consistent with multiple explanations: artifact exploitation, improved construction practices over time, or increased scrutiny of older benchmarks\. Observationally, we find correlation but cannot isolate directionality\.

![Refer to caption](https://arxiv.org/html/2602.16763v3/x2.png)Figure 3:Mean saturation index grouped by binned time since benchmark release \(in months\)\.Older benchmarks exhibit higher average saturation, reflecting increasing performance compression among state\-of\-the\-art models as benchmarks age\. Error bars denote one standard deviation within each bin\.![Refer to caption](https://arxiv.org/html/2602.16763v3/x3.png)Figure 4:Mean saturation index grouped by binned benchmark citation counts\.Benchmarks with higher citation counts exhibit higher saturation rates, suggesting that benchmark adoption and exposure are associated with reduced discriminative power over time\. Error bars denote one standard deviation within each bin\.

### 4\.2Joint Analysis of Saturation Factors

To quantify which benchmark properties jointly explain variation in saturation, we fit a Bayesian regression model predictingSindexS\_\{\\text\{index\}\}from benchmark age, test set size, adoption proxies, accessibility, output format, templating, language coverage, curation strategy, and documented quality issues\. The fitted model achievesRBayes2=0\.884±0\.012R^\{2\}\_\{\\text\{Bayes\}\}=0\.884\\pm 0\.012\.

Across specifications, benchmark age and test set size show the most consistent effects\. Adoption metrics contribute modestly but are not robust once age is included\. In contrast, accessibility \(public vs\. private\), output format, and templating do not exhibit reliable associations with saturation\. Overall, the results indicate that saturation is more strongly associated with maturity and measurement scale than with commonly assumed design safeguards\.444See[AppendixF](https://arxiv.org/html/2602.16763#A6)in the appendix for further details\.

## 5Synthesis and Implications

Our empirical analysis reveals a consistent pattern: benchmark saturation is primarily driven by structural exposure dynamics and measurement resolution limits, rather than by isolated design choices\. While contamination, overfitting, and ceiling effects have been discussed independently in prior work\(McCoyet al\.,[2019](https://arxiv.org/html/2602.16763#bib.bib25); Murahariet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib10); Schaeffer,[2023](https://arxiv.org/html/2602.16763#bib.bib137)\), our results clarify which factors systematically correlate with saturation across benchmarks\.

### 5\.1Saturation as a Structural Phenomenon

Our empirical results indicate that benchmark saturation is primarily a structural consequence of exposure dynamics and measurement resolution, rather than isolated design flaws\. Two variables emerge as the most consistent predictors: benchmark age and test set scale\.

#### Age and exposure\-driven compression\.

Older benchmarks exhibit higher saturation indices, even after controlling for adoption metrics such as citation counts or inclusion in technical reports\. Once age is accounted for, these popularity proxies no longer show associations with saturation, suggesting that cumulative exposure, not popularity alone, drives convergence\. Repeated optimization against a stable evaluation target progressively compresses performance differences among frontier models\. Our results are consistent with this interpretation, since older benchmarks exhibit higher saturation, although our analysis does not directly identify the causal mechanism\. Similar plateau dynamics have been discussed qualitatively in prior work\(Ottet al\.,[2022](https://arxiv.org/html/2602.16763#bib.bib32)\)\. This exposure effect is consistent with known risks of familiarity and memorization\. Publicly accessible benchmarks increase the possibility that evaluation data, or close variants, appear in training corpora \(seeH1H\_\{1\}and[Section˜4\.1](https://arxiv.org/html/2602.16763#S4.SS1)\), as well as findings in the literature\(Zhouet al\.,[2023b](https://arxiv.org/html/2602.16763#bib.bib107); Balloccuet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib108)\)\. However, in our analysis, private test sets do not systematically reduce saturation once benchmark age is considered, suggesting that privacy alone is not sufficient\. Even without explicit leakage or contamination, our finding that older benchmarks exhibit higher saturation supports the broader mechanism that repeated exposure to fixed evaluation formats encourages benchmark\-specific optimization, narrowing observable performance gaps over time\.

#### Test set scale and measurement resolution limits\.

In our empirical analysis \([Section˜4](https://arxiv.org/html/2602.16763#S4)\), larger evaluation sets are consistently associated with lower saturation indices\. This suggests that discriminative power depends critically on statistical resolution\. When evaluation uncertainty exceeds true performance gaps, top models become statistically indistinguishable even if substantive differences remain\. Smaller test sets accelerate this effect, as variance dominates observed score differences\. Moreover, reliance on coarse aggregate metrics \(e\.g\.,single accuracy scores\) can mask residual behavioral variation across subskills or input types\(Murahariet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib10); Saxonet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib42)\)\. Taken together with our finding that older benchmarks tend to be more saturated, these results suggest that saturation often reflects loss of*relative separability*among top\-performing models rather than complete task mastery \(which would be desirable; importantly, benchmark saturation is a neutral, not a negative phenomenon\. It only becomes an issue if saturation does not reflect task mastery\)\. Benchmark maturity increases optimization pressure, while finite evaluation resolution constrains the ability to detect incremental gains\. Saturation therefore emerges from the interaction between cumulative exposure and statistical measurement limits, even in the absence of explicit contamination or fundamental capability ceilings\.

### 5\.2Safeguards That Do Not Prevent Saturation

Although benchmark age emerges as the strongest factor and most consistently correlates with saturation, we test the remaining hypotheses to evaluate whether commonly assumed safeguards retain explanatory power once we take age into account\. Our results show that, these safeguards do not show robust associations with saturation in our data\.

#### Private test sets\.

Benchmark creators should not rely on private or held\-out test sets alone as a long\-term defense against saturation\.In our H1 analysis, we observe similar saturation distributions and no statistically meaningful difference inSindexS\_\{\\text\{index\}\}between public and private benchmarks\. While contamination and memorization are well\-documented risks\(Zhouet al\.,[2023b](https://arxiv.org/html/2602.16763#bib.bib107); Balloccuet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib108); Denget al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib127); Sainzet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib41)\), secrecy alone does not prevent compression once distributional characteristics become widely known\. Direct fine\-tuning on evaluation data can trivially inflate scores\(Schaeffer,[2023](https://arxiv.org/html/2602.16763#bib.bib137)\), but our results suggest that even without explicit leakage, prolonged exposure drives convergence\.

#### Open\-ended output formats\.

Benchmark creators should not assume that switching from multiple\-choice to open\-ended generation alone will meaningfully extend benchmark usefulness over time\.In evaluating hypothesis H4, we observe no meaningful difference in saturation distributions between closed\-ended \(N=28\) and open\-ended \(N=31\) benchmarks\. The output format comparison is age\-balanced \(p=0\.40\)\. Although multiple\-choice benchmarks may enable overfitting strategies\(Chandaket al\.,[2025](https://arxiv.org/html/2602.16763#bib.bib130)\), and models can exploit superficial cues\(McCoyet al\.,[2019](https://arxiv.org/html/2602.16763#bib.bib25); Pacchiardiet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib34)\), format alone does not determine longevity\. Compression seems to occur in both settings\.

#### Template diversity and multilinguality\.

Benchmark creators should prioritize refresh mechanisms, substantive difficulty and measurement resolution over surface\-level diversity features such as templating or multilingual scope alone\.In evaluating hypothesis H6, we find that templated benchmarks \(N=14\) do not differ significantly from non\-templated benchmarks \(N=46\) in saturation behaviour \(p=0\.10\), suggesting that template diversity alone does not delay saturation\. Multilingual benchmarks appear more robust in raw comparisons, but this effect is largely explained by recency\. In evaluating hypothesis H2, we find that multilingual benchmarks \(N=16\) show lower raw saturation rates than English\-only benchmarks \(N=44\), but this apparent advantage is confounded by benchmark maturity: multilingual benchmarks in our sample are substantially younger on average \(32\.9 vs\. 48\.9 months\)\. While English\-dominant pretraining corpora may accelerate ceiling effects on English\-only tasks\(Touvronet al\.,[2023](https://arxiv.org/html/2602.16763#bib.bib128); Wanget al\.,[2024a](https://arxiv.org/html/2602.16763#bib.bib122)\), age remains the dominant predictor\.

Recent evidence from SWE\-bench Verified also illustrates how plateaus can arise from evaluation artifacts rather than capability ceilings\.OpenAI \([2024b](https://arxiv.org/html/2602.16763#bib.bib20)\)report that many frequently\-failed tasks contain narrow or wide tests that reject functionally correct solutions and performance increasingly reflects training exposure to benchmark\-associated repositories rather than general coding ability\.

### 5\.3Structural Resistance to Saturation

A minority of benchmarks remain unsaturated despite substantial exposure\. Qualitatively, these benchmarks tend to share structural properties that alter one or both of the above mechanisms\. Benchmarks with adversarial or dynamic data collection \(e\.g\.,Dynabench\(Kielaet al\.,[2021](https://arxiv.org/html/2602.16763#bib.bib135)\)\) reduce optimization stability by continuously updating the evaluation distribution\. Broad, capability\-spanning initiatives such as BIG\-Bench\(Srivastavaet al\.,[2023](https://arxiv.org/html/2602.16763#bib.bib121)\)expand coverage and limit narrow over\-fitting\. Holistic evaluation frameworks that track multiple behavioural dimensions\(Lianget al\.,[2023](https://arxiv.org/html/2602.16763#bib.bib24)\)increase measurement granularity\.

### 5\.4Implications for Benchmark Lifecycle Management

Our findings suggest that sustainable evaluation requires monitoring benchmark’s discriminative power rather than relying on absolute score improvements\. Benchmarks should be treated as evolving measurement instruments whose usefulness can reduce as models adapt to them\.

#### Benchmark design considerations\.

Our findings suggest four actionable takeaways during benchmark design\.\(1\) Increase evaluation resolution\.Across our analyses, test set scale is one of the strongest predictors of lower saturation\. Benchmark designers should therefore make sure that score differences between models exceed expected evaluation uncertainty\. This can require larger test sets, harder examples, stratified reporting by subgroups of items \(e\.g\., according to subskills\), or multiple complementary metrics providing more insights into performance differences rather than a single aggregate score\.\(2\) Integrate dynamic benchmark updates\.Static benchmarks become easier optimization targets over time\. Periodic refreshes, adversarial data collection, rotating hidden subsets, or continuously updated evaluation pools can reduce benchmark convergence resulting from exposure and prolong benchmark usefulness\.\(3\) Report uncertainty\-aware statistics\.Integrate into leaderboard reporting confidence intervals, the spread of scores among top systems, and compression indicators in addition to aggregated peak scores\. Small improvements that fall within evaluation noise should not be interpreted as meaningful progress\.\(4\) Define criteria for lifecycle management\.Benchmarks creators should include explicit revision, expansion, or retirement procedures during benchmark design once frontier systems become statistically indistinguishable, as highlighted in lifecycle\-oriented evaluation frameworks\(Hardyet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib16)\)\.

#### When is saturation desirable?

Benchmark saturation is not inherently negative\. If a benchmark is well\-designed, valid, and measures a clearly defined capability, then convergence of top\-performing models new the benchmark’s ceiling may indicate genuine task mastery\. In such cases, saturation reflects substantive progress: models can reliably perform the task the benchmark was intended to measure\. However, saturation becomes problematic when it reflects loss of measurement resolution rather than capability completion\. If score compression arises because evaluation noise exceeds true performance gaps, or because the benchmark lacks sufficient depth or coverage to differentiate increasingly capable systems, then apparent convergence may mask unresolved weaknesses\. In this scenario, models may appear indistinguishable despite meaningful differences in robustness, calibration, or generalization\. The key distinction is whether saturation reflects true capability attainment or reduced discriminative power: the former signals progress, while the latter calls for revision or expansion\.

## 6Limitations and Future work

Our benchmark selection, though criteria\-driven, reflects current evaluation practices and may overrepresent widely\-adopted benchmarks\. Top\-N leaderboard snapshots may miss saturation dynamics for sparse or inconsistently evaluated benchmarks\. The saturation index further depends on currently available frontier model evaluations, which may be incomplete, selectively reported, or inconsistently updated\. We assume benchmark properties are time\-invariant, yet attributes like annotation diversity evolve post\-release\. Similarly, benchmarks themselves may change over time through revised splits, refreshed test sets, or updated protocols, which are not captured in our static annotations\.

Our analysis relies on publicly available leaderboard data, which posed several challenges\. Multiple leaderboards may exist for a benchmark, often differing in evaluation setups \(e\.g\., LLM\-judge prompts\) and scoring criteria\. Many leaderboards are not regularly updated and may omit newly released models\. We therefore prioritized leaderboards based on visibility, recency, and result verification, though inconsistencies remain\. Finally, our uncertainty estimates are designed for accuracy\-like metrics over fixed test sets; metrics such as Elo ratings, pass@k, or judge\-based evaluations require tailored variance estimates\.

Future work should incorporate continuous\-time leaderboard data and distinguish genuine saturation from temporary plateaus\. Longitudinal analysis and causal studies comparing different exposure patterns could further clarify the mechanisms driving saturation\. Studying performance shifts following major model innovations could clarify whether saturation is transient or persistent\.

## 7Conclusion

In this work, we present a systematic analysis of benchmark saturation\. By introducing an uncertainty\-aware saturation index and characterizing benchmarks across multiple design dimensions, we identify which properties are associated with saturation dynamics\. Our findings challenge common assumptions \(e\.g\. the protective role of private test sets\) and highlight the importance of benchmark design, scale, and lifecycle management\. This work provides a foundation for more robust and sustainable evaluation practices, designing benchmarks such that they remain informative over time\.

## Impact Statement

Benchmark scores increasingly shape public discourse, model deployment, investment, marketing, policy decisions, and resource allocation in AI development\. When saturated benchmarks are reported without appropriate context, they risk misinforming stakeholders about meaningful capability differences\. Our analysis demonstrates that near\-ceiling scores often fail to discriminate between models in ways that matter for downstream applications\. This work encourages more careful communication of evaluation results, particularly when such results inform decisions in high\-stakes domains such as healthcare, education, and public services\.

## Acknowledgements

We thank Siva Kantha Rao Vanama, Abhijit Ubale, Vijaya Kumar Reddy Palreddy, Shivaprasad Chitta, Sasikanth Kotti, Wm\. Matthew Kennedy, and Alexander Hoyle who supported this project through annotation efforts, discussions, and comments on the paper draft\.

Mubashara Akhtar was supported by the ETH AI Center through an ETH AI Center postdoctoral fellowship\. Hossein A\. Rahmani’s effort was supported by the Engineering and Physical Sciences Research Council \(EP/S021566/1\)\. Vilém Zouhar gratefully acknowledges the support of the Google PhD Fellowship\. Marek Suppa was funded by the EU NextGenerationEU through the Recovery and Resilience Plan for Slovakia under the project No\. 09I02\-03\-V01\-00029\. Jan Batzner was supported by the Federal Ministry of Research, Technology, and Space of Germany \[Grant Number 16DII131\]\. Yanan Long thanks the TPU Research Cloud for computational support\. Anka Reuel was supported by the Stanford Interdisciplinary Graduate Fellowship\. Sanmi Koyejo is partially supported by NSF 2046795 and 2205329, IES R305C240046, ARPA\-H, the MacArthur Foundation, Schmidt Sciences, Stanford HAI, RAISE Health, OpenAI, Microsoft, and Google\.

## References

- D\. I\. Adelani, H\. Liu, X\. Shen, N\. Vassilyev, J\. O\. Alabi, Y\. Mao, H\. Gao, and E\. A\. Lee \(2024\)SIB\-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200\+ Languages and Dialects\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),St\. Julian’s, Malta,pp\. 226–245\(en\)\.External Links:[Link](https://aclanthology.org/2024.eacl-long.14),[Document](https://dx.doi.org/10.18653/v1/2024.eacl-long.14)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.17.15.4)\.
- S\. AI \(2024a\)GSM8K\-Indic: A multilingual version of GSM8K for indian languages\.Hugging Face\.External Links:[Link](https://huggingface.co/datasets/sarvamai/gsm8k-indic)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.22.20.2)\.
- S\. AI \(2024b\)MMLU\-Indic: A multilingual version of MMLU for indian languages\.Hugging Face\.External Links:[Link](https://huggingface.co/datasets/sarvamai/mmlu-indic)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.5.3.4)\.
- N\. Alzahrani, H\. Alyahya, Y\. Alnumay, S\. AlRashed, S\. Alsubaie, Y\. Almushayqih, F\. Mirza, N\. Alotaibi, N\. Al\-Twairesh, A\. Alowisheq, M\. S\. Bari, and H\. Khan \(2024\)When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 13787–13805\(en\)\.External Links:[Link](https://aclanthology.org/2024.acl-long.744),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.744)Cited by:[§1](https://arxiv.org/html/2602.16763#S1.p1.1)\.
- S\. Ashury\-Tahan, A\. Gera, E\. Bandel, M\. Shmueli\-Scheuer, and L\. Choshen \(2026a\)Robustness as an Emergent Property of Task Performance\.arXiv\.Note:arXiv:2602\.03344 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2602.03344),[Document](https://dx.doi.org/10.48550/arXiv.2602.03344)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p2.1),[§1](https://arxiv.org/html/2602.16763#S1.p2.1)\.
- S\. Ashury\-Tahan, Y\. Mai, R\. C, A\. Gera, Y\. Perlitz, A\. Yehudai, E\. Bandel, L\. Choshen, E\. Shnarch, P\. Liang, and M\. Shmueli\-Scheuer \(2026b\)The Mighty ToRR: A Benchmark for Table Reasoning and Robustness\.arXiv\.Note:arXiv:2502\.19412 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2502.19412),[Document](https://dx.doi.org/10.48550/arXiv.2502.19412)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p2.1)\.
- S\. Balloccu, P\. Schmidtová, M\. Lango, and O\. Dusek \(2024\)Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed\-Source LLMs\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),St\. Julian’s, Malta,pp\. 67–93\(en\)\.External Links:[Link](https://aclanthology.org/2024.eacl-long.5),[Document](https://dx.doi.org/10.18653/v1/2024.eacl-long.5)Cited by:[Appendix B](https://arxiv.org/html/2602.16763#A2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2602.16763#S5.SS1.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2602.16763#S5.SS2.SSS0.Px1.p1.1)\.
- L\. Bandarkar, D\. Liang, B\. Muller, M\. Artetxe, S\. N\. Shukla, D\. Husa, N\. Goyal, A\. Krishnan, L\. Zettlemoyer, and M\. Khabsa \(2024\)The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 749–775\(en\)\.External Links:[Link](https://aclanthology.org/2024.acl-long.44),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.44)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.9.7.2)\.
- V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhan \(2025\)τ2\\tau^\{2\}\-Bench: Evaluating Conversational Agents in a Dual\-Control Environment\.arXiv\.Note:Version Number: 1External Links:[Link](https://arxiv.org/abs/2506.07982),[Document](https://dx.doi.org/10.48550/ARXIV.2506.07982)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.1.4)\.
- Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. Choi \(2020\)PIQA: Reasoning about Physical Commonsense in Natural Language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.External Links:ISSN 2374\-3468, 2159\-5399,[Link](https://ojs.aaai.org/index.php/AAAI/article/view/6239),[Document](https://dx.doi.org/10.1609/aaai.v34i05.6239)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.12.10.4)\.
- F\. Cassano, J\. Gouwar, D\. Nguyen, S\. Nguyen, L\. Phipps\-Costin, D\. Pinckney, M\. Yee, Y\. Zi, C\. J\. Anderson, M\. Q\. Feldman, A\. Guha, M\. Greenberg, and A\. Jangda \(2022\)MultiPL\-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation\.arXiv\.Note:Version Number: 4External Links:[Link](https://arxiv.org/abs/2208.08227),[Document](https://dx.doi.org/10.48550/ARXIV.2208.08227)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.9.7.4)\.
- N\. Chandak, S\. Goel, A\. Prabhu, M\. Hardt, and J\. Geiping \(2025\)Answer Matching Outperforms Multiple Choice for Language Model Evaluation\.arXiv\.Note:Version Number: 1Other 34 pages, Code is available at https://github\.com/nikhilchandak/answer\-matchingExternal Links:[Link](https://arxiv.org/abs/2507.02856),[Document](https://dx.doi.org/10.48550/ARXIV.2507.02856)Cited by:[Appendix B](https://arxiv.org/html/2602.16763#A2.SS0.SSS0.Px4.p1.1),[§5\.2](https://arxiv.org/html/2602.16763#S5.SS2.SSS0.Px2.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating Large Language Models Trained on Code\.arXiv\.Note:Version Number: 2Other corrected typos, added references, added authors, added acknowledgementsExternal Links:[Link](https://arxiv.org/abs/2107.03374),[Document](https://dx.doi.org/10.48550/ARXIV.2107.03374)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.24.22.2)\.
- M\. Chen \(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§1](https://arxiv.org/html/2602.16763#S1.p1.1)\.
- S\. Chen, Y\. Chen, Z\. Li, Y\. Jiang, Z\. Wan, Y\. He, D\. Ran, T\. Gu, H\. Li, T\. Xie, and B\. Ray \(2025\)Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 10091–10109\(en\)\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.511),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.511)Cited by:[§1](https://arxiv.org/html/2602.16763#S1.p1.1)\.
- E\. Choi, H\. He, M\. Iyyer, M\. Yatskar, W\. Yih, Y\. Choi, P\. Liang, and L\. Zettlemoyer \(2018\)QuAC: Question Answering in Context\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2174–2184\.External Links:[Link](https://aclanthology.org/D18-1241/),[Document](https://dx.doi.org/10.18653/v1/D18-1241)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.13.11.4)\.
- F\. Chollet, M\. Knoop, G\. Kamradt, B\. Landers, and H\. Pinkard \(2025\)ARC\-AGI\-2: A New Challenge for Frontier AI Reasoning Systems\.arXiv\.Note:Version Number: 2External Links:[Link](https://arxiv.org/abs/2505.11831),[Document](https://dx.doi.org/10.48550/ARXIV.2505.11831)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.7.5.2)\.
- F\. Chollet \(2019\)On the Measure of Intelligence\.arXiv\.Note:arXiv:1911\.01547 \[cs\.AI\]External Links:[Link](http://arxiv.org/abs/1911.01547),[Document](https://dx.doi.org/10.48550/arXiv.1911.01547)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.7.5.2)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 2924–2936\.External Links:[Link](https://aclanthology.org/N19-1300/),[Document](https://dx.doi.org/10.18653/v1/N19-1300)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.11.9.2)\.
- J\. H\. Clark, E\. Choi, M\. Collins, D\. Garrette, T\. Kwiatkowski, V\. Nikolaev, and J\. Palomaki \(2020\)T <span style="font\-variant:small\-caps;"\>y</span\> D <span style="font\-variant:small\-caps;"\>i</span\> QA: A Benchmark for Information\-Seeking Question Answering inTypologicallyDiverse Languages\.Transactions of the Association for Computational Linguistics8,pp\. 454–470\(en\)\.External Links:ISSN 2307\-387X,[Link](https://direct.mit.edu/tacl/article/96451),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00317)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.27.25.4)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training Verifiers to Solve Math Word Problems\.arXiv\.Note:Version Number: 2External Links:[Link](https://arxiv.org/abs/2110.14168),[Document](https://dx.doi.org/10.48550/ARXIV.2110.14168)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.21.19.2),[§2\.2](https://arxiv.org/html/2602.16763#S2.SS2.SSS0.Px3.p2.1)\.
- D\. Das, K\. De Langis, A\. Martin\-Boyle, J\. Kim, M\. Lee, Z\. M\. Kim, S\. A\. Hayati, R\. Owan, B\. Hu, R\. Parkar, R\. Koo, J\. Park, A\. Tyagi, L\. Ferland, S\. Roy, V\. Liu, and D\. Kang \(2024\)Under the Surface: Tracking the Artifactuality of LLM\-Generated Data\.arXiv\.Note:Version Number: 2Other Core Authors: Debarati Das, Karin De Langis, Anna Martin\-Boyle, Jaehyung Kim, Minhwa Lee and Zae Myung Kim \| Project lead : Debarati Das \| PI : Dongyeop KangExternal Links:[Link](https://arxiv.org/abs/2401.14698),[Document](https://dx.doi.org/10.48550/ARXIV.2401.14698)Cited by:[Appendix B](https://arxiv.org/html/2602.16763#A2.SS0.SSS0.Px3.p1.1)\.
- C\. Deng, Y\. Zhao, X\. Tang, M\. Gerstein, and A\. Cohan \(2024\)Investigating Data Contamination in Modern Benchmarks for Large Language Models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 8706–8719\(en\)\.External Links:[Link](https://aclanthology.org/2024.naacl-long.482),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.482)Cited by:[Appendix B](https://arxiv.org/html/2602.16763#A2.SS0.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2602.16763#S5.SS2.SSS0.Px1.p1.1)\.
- J\. Deng, W\. Dong, R\. Socher, L\. Li, Kai Li, and Li Fei\-Fei \(2009\)ImageNet: A large\-scale hierarchical image database\.In2009 IEEE Conference on Computer Vision and Pattern Recognition,Miami, FL,pp\. 248–255\.External Links:ISBN 978\-1\-4244\-3992\-8,[Link](https://ieeexplore.ieee.org/document/5206848/),[Document](https://dx.doi.org/10.1109/CVPR.2009.5206848)Cited by:[§1](https://arxiv.org/html/2602.16763#S1.p1.1)\.
- D\. Dua, Y\. Wang, P\. Dasigi, G\. Stanovsky, S\. Singh, and M\. Gardner \(2019\)DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 2368–2378\.External Links:[Link](https://aclanthology.org/N19-1246/),[Document](https://dx.doi.org/10.18653/v1/N19-1246)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.14.12.2)\.
- A\. P\. Gema, J\. O\. J\. Leang, G\. Hong, A\. Devoto, A\. C\. M\. Mancino, R\. Saxena, X\. He, Y\. Zhao, X\. Du, M\. R\. Ghasemi Madani, C\. Barale, R\. McHardy, J\. Harris, J\. Kaddour, E\. Van Krieken, and P\. Minervini \(2025\)Are We Done with MMLU?\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Albuquerque, New Mexico,pp\. 5069–5096\(en\)\.External Links:[Link](https://aclanthology.org/2025.naacl-long.262),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.262)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.7.5.4)\.
- A\. Gill, A\. Ravichander, and A\. Marasovic \(2025\)What Has Been Lost with Synthetic Evaluation?\.InFindings of the Association for Computational Linguistics: EMNLP 2025,Suzhou, China,pp\. 9902–9945\(en\)\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.526),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.526)Cited by:[Appendix B](https://arxiv.org/html/2602.16763#A2.SS0.SSS0.Px3.p1.1)\.
- E\. Glazer, E\. Erdil, T\. Besiroglu, D\. Chicharro, E\. Chen, A\. Gunning, C\. F\. Olsson, J\. Denain, A\. Ho, E\. d\. O\. Santos, O\. Järviniemi, M\. Barnett, R\. Sandler, M\. Vrzala, J\. Sevilla, Q\. Ren, E\. Pratt, L\. Levine, G\. Barkley, N\. Stewart, B\. Grechuk, T\. Grechuk, S\. V\. Enugandla, and M\. Wildon \(2024\)FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI\.arXiv\.Note:Version Number: 7External Links:[Link](https://arxiv.org/abs/2411.04872),[Document](https://dx.doi.org/10.48550/ARXIV.2411.04872)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.31.29.4)\.
- N\. Goyal, C\. Gao, V\. Chaudhary, P\. Chen, G\. Wenzek, D\. Ju, S\. Krishnan, M\. Ranzato, F\. Guzmán, and A\. Fan \(2022\)The <span style="font\-variant:small\-caps;"\>Flores\-101</span\> Evaluation Benchmark for Low\-Resource and Multilingual Machine Translation\.Transactions of the Association for Computational Linguistics10,pp\. 522–538\(en\)\.External Links:ISSN 2307\-387X,[Link](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00474/110993/The-Flores-101-Evaluation-Benchmark-for-Low),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00474)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.16.14.2)\.
- N\. Guha, J\. Nyarko, D\. E\. Ho, C\. Ré, A\. Chilton, A\. Narayana, A\. Chohlas\-Wood, A\. Peters, B\. Waldon, D\. N\. Rockmore, D\. Zambrano, D\. Talisman, E\. Hoque, F\. Surani, F\. Fagan, G\. Sarfaty, G\. M\. Dickinson, H\. Porat, J\. Hegland, J\. Wu, J\. Nudell, J\. Niklaus, J\. Nay, J\. H\. Choi, K\. Tobia, M\. Hagan, M\. Ma, M\. Livermore, N\. Rasumov\-Rahe, N\. Holzenberger, N\. Kolt, P\. Henderson, S\. Rehaag, S\. Goel, S\. Gao, S\. Williams, S\. Gandhi, T\. Zur, V\. Iyer, and Z\. Li \(2023\)LEGALBENCH: a collaboratively built benchmark for measuring legal reasoning in large language models\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.28.26.2)\.
- V\. Gupta, C\. Ross, D\. Pantoja, R\. J\. Passonneau, M\. Ung, and A\. Williams \(2025\)Improving Model Evaluation using SMART Filtering of Benchmark Datasets\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Albuquerque, New Mexico,pp\. 4595–4615\(en\)\.External Links:[Link](https://aclanthology.org/2025.naacl-long.235),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.235)Cited by:[§2\.1](https://arxiv.org/html/2602.16763#S2.SS1.SSS0.Px1.p1.1)\.
- L\. Haas, G\. Yona, G\. D’Antonio, S\. Goldshtein, and D\. Das \(2025\)SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge\.arXiv\.Note:Version Number: 2External Links:[Link](https://arxiv.org/abs/2509.07968),[Document](https://dx.doi.org/10.48550/ARXIV.2509.07968)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.18.16.4)\.
- E\. Habba, O\. Arviv, I\. Itzhak, Y\. Perlitz, E\. Bandel, L\. Choshen, M\. Shmueli\-Scheuer, and G\. Stanovsky \(2025\)DOVE: A Large\-Scale Multi\-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation\.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp\. 11744–11763\(en\)\.External Links:[Link](https://aclanthology.org/2025.findings-acl.611),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.611)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p2.1)\.
- A\. Hardy, M\. Hardy, M\. Kochenderfer, M\. Lamparth, A\. Reuel, and C\. Smith \(2024\)BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices\.InAdvances in Neural Information Processing Systems 37,Vancouver, BC, Canada,pp\. 21763–21813\.External Links:ISBN 979\-8\-3313\-1438\-5,[Link](http://www.proceedings.com/079017-0685.html),[Document](https://dx.doi.org/10.52202/079017-0685)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p1.1),[§1](https://arxiv.org/html/2602.16763#S1.p1.1),[§2\.1](https://arxiv.org/html/2602.16763#S2.SS1.SSS0.Px1.p1.1),[§5\.4](https://arxiv.org/html/2602.16763#S5.SS4.SSS0.Px1.p1.1)\.
- A\. Hardy, A\. Reuel, K\. Jafari Meimandi, L\. Soder, A\. Griffith, D\. M\. Asmar, S\. Koyejo, M\. S\. Bernstein, and M\. J\. Kochenderfer \(2025\)More than Marketing? On the Information Value of AI Benchmarks for Practitioners\.InProceedings of the 30th International Conference on Intelligent User Interfaces,Cagliari Italy,pp\. 1032–1047\(en\)\.External Links:ISBN 979\-8\-4007\-1306\-4,[Link](https://dl.acm.org/doi/10.1145/3708359.3712152),[Document](https://dx.doi.org/10.1145/3708359.3712152)Cited by:[§1](https://arxiv.org/html/2602.16763#S1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2020\)Measuring Massive Multitask Language Understanding\.arXiv\.Note:Version Number: 3Other ICLR 2021; the test and code is available at https://github\.com/hendrycks/testExternal Links:[Link](https://arxiv.org/abs/2009.03300),[Document](https://dx.doi.org/10.48550/ARXIV.2009.03300)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.4.2.4)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring Mathematical Problem Solving With the MATH Dataset\.arXiv\.Note:Version Number: 2Other NeurIPS 2021\. Code and the MATH dataset is available at https://github\.com/hendrycks/math/External Links:[Link](https://arxiv.org/abs/2103.03874),[Document](https://dx.doi.org/10.48550/ARXIV.2103.03874)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.31.29.2)\.
- J\. Hu, S\. Ruder, A\. Siddhant, G\. Neubig, O\. Firat, and M\. Johnson \(2020\)XTREME: a massively multilingual multi\-task benchmark for evaluating cross\-lingual generalization\.InProceedings of the 37th International Conference on Machine Learning,ICML’20\.Cited by:[Appendix B](https://arxiv.org/html/2602.16763#A2.SS0.SSS0.Px2.p1.1)\.
- Y\. Huang, Y\. Bai, Z\. Zhu, J\. Zhang, J\. Zhang, T\. Su, J\. Liu, C\. Lv, Y\. Zhang, j\. lei, Y\. Fu, M\. Sun, and J\. He \(2023\)C\-Eval: A Multi\-Level Multi\-Discipline Chinese Evaluation Suite for Foundation Models\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 62991–63010\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/c6ec1844bec96d6d32ae95ae694e23d8-Paper-Datasets_and_Benchmarks.pdf)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.12.10.2)\.
- A\. Jacovi, A\. Wang, C\. Alberti, C\. Tao, J\. Lipovetz, K\. Olszewska, L\. Haas, M\. Liu, N\. Keating, A\. Bloniarz, C\. Saroufim, C\. Fry, D\. Marcus, D\. Kukliansky, G\. S\. Tomar, J\. Swirhun, J\. Xing, L\. Wang, M\. Aaron, M\. Ambar, R\. Fellinger, R\. Wang, R\. Sims, Z\. Zhang, S\. Goldshtein, Y\. Matias, and D\. Das \(2024\)FACTS leaderboard\.Note:[https://kaggle\.com/facts\-leaderboard](https://kaggle.com/facts-leaderboard)Google DeepMind, Google Research, Google Cloud, KaggleExternal Links:[Link](https://www.kaggle.com/benchmarks/google/facts)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.15.13.2)\.
- A\. Jacovi, A\. Wang, C\. Alberti, C\. Tao, J\. Lipovetz, K\. Olszewska, L\. Haas, M\. Liu, N\. Keating, A\. Bloniarz, C\. Saroufim, C\. Fry, D\. Marcus, D\. Kukliansky, G\. S\. Tomar, J\. Swirhun, J\. Xing, L\. Wang, M\. Gurumurthy, M\. Aaron, M\. Ambar, R\. Fellinger, R\. Wang, Z\. Zhang, S\. Goldshtein, and D\. Das \(2025\)The FACTS Grounding Leaderboard: Benchmarking LLMs’ Ability to Ground Responses to Long\-Form Input\.arXiv\.Note:Version Number: 1External Links:[Link](https://arxiv.org/abs/2501.03200),[Document](https://dx.doi.org/10.48550/ARXIV.2501.03200)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.15.13.2)\.
- N\. Jain, Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2025\)LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 58791–58831\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/94074dd5a072d28ff75a76dabed43767-Paper-Conference.pdf)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.30.28.2)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2024\)SWE\-bench: Can Language Models Resolve Real\-world Github Issues?\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 54107–54157\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/edac78c3e300629acfe6cbe9ca88fb84-Paper-Conference.pdf)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.22.20.4),[§1](https://arxiv.org/html/2602.16763#S1.p2.1)\.
- D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. Szolovits \(2021\)What Disease Does This Patient Have? A Large\-Scale Open Domain Question Answering Dataset from Medical Exams\.Applied Sciences11\(14\),pp\. 6421\(en\)\.External Links:ISSN 2076\-3417,[Link](https://www.mdpi.com/2076-3417/11/14/6421),[Document](https://dx.doi.org/10.3390/app11146421)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.20.18.2)\.
- M\. Joshi, E\. Choi, D\. Weld, and L\. Zettlemoyer \(2017\)TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vancouver, Canada,pp\. 1601–1611\(en\)\.External Links:[Link](http://aclweb.org/anthology/P17-1147),[Document](https://dx.doi.org/10.18653/v1/P17-1147)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.25.23.4)\.
- L\. Justen \(2025\)LLMs Outperform Experts on Challenging Biology Benchmarks\.arXiv\.Note:Version Number: 3External Links:[Link](https://arxiv.org/abs/2505.06108),[Document](https://dx.doi.org/10.48550/ARXIV.2505.06108)Cited by:[§2\.1](https://arxiv.org/html/2602.16763#S2.SS1.p1.1)\.
- D\. Kiela, M\. Bartolo, Y\. Nie, D\. Kaushik, A\. Geiger, Z\. Wu, B\. Vidgen, G\. Prasad, A\. Singh, P\. Ringshia, Z\. Ma, T\. Thrush, S\. Riedel, Z\. Waseem, P\. Stenetorp, R\. Jia, M\. Bansal, C\. Potts, and A\. Williams \(2021\)Dynabench: Rethinking Benchmarking in NLP\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Online,pp\. 4110–4124\(en\)\.External Links:[Link](https://aclanthology.org/2021.naacl-main.324),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.324)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p1.1),[§5\.3](https://arxiv.org/html/2602.16763#S5.SS3.p1.1)\.
- T\. Kocmi, E\. Avramidis, R\. Bawden, O\. Bojar, A\. Dvorkovich, C\. Federmann, M\. Fishel, M\. Freitag, T\. Gowda, R\. Grundkiewicz, B\. Haddow, M\. Karpinska, P\. Koehn, B\. Marie, C\. Monz, K\. Murray, M\. Nagata, M\. Popel, M\. Popović, M\. Shmatova, S\. Steingrímsson, and V\. Zouhar \(2024\)Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet\.InProceedings of the Ninth Conference on Machine Translation,Miami, Florida, USA,pp\. 1–46\(en\)\.External Links:[Link](https://aclanthology.org/2024.wmt-1.1),[Document](https://dx.doi.org/10.18653/v1/2024.wmt-1.1)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.30.28.4)\.
- T\. Kocmi, E\. Avramidis, R\. Bawden, O\. Bojar, A\. Dvorkovich, C\. Federmann, M\. Fishel, M\. Freitag, T\. Gowda, R\. Grundkiewicz, B\. Haddow, P\. Koehn, B\. Marie, C\. Monz, M\. Morishita, K\. Murray, M\. Nagata, T\. Nakazawa, M\. Popel, M\. Popović, and M\. Shmatova \(2023\)Findings of the 2023 Conference on Machine Translation \(WMT23\): LLMs Are Here but Not Quite There Yet\.InProceedings of the Eighth Conference on Machine Translation,Singapore,pp\. 1–42\(en\)\.External Links:[Link](https://aclanthology.org/2023.wmt-1.1),[Document](https://dx.doi.org/10.18653/v1/2023.wmt-1.1)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.30.28.4)\.
- T\. Kocmi, R\. Bawden, O\. Bojar, A\. Dvorkovich, C\. Federmann, M\. Fishel, T\. Gowda, Y\. Graham, R\. Grundkiewicz, B\. Haddow, R\. Knowles, P\. Koehn, C\. Monz, M\. Morishita, M\. Nagata, T\. Nakazawa, M\. Novák, M\. Popel, and M\. Popović \(2022\)Findings of the 2022 Conference on Machine Translation \(WMT22\)\.InProceedings of the Seventh Conference on Machine Translation \(WMT\),Abu Dhabi, United Arab Emirates \(Hybrid\),pp\. 1–45\(en\)\.External Links:[Link](https://aclanthology.org/2022.wmt-1.1),[Document](https://dx.doi.org/10.18653/v1/2022.wmt-1.1)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.30.28.4)\.
- T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. Petrov \(2019\)Natural Questions: A Benchmark for Question Answering Research\.Transactions of the Association for Computational Linguistics7,pp\. 453–466\(en\)\.External Links:ISSN 2307\-387X,[Link](https://direct.mit.edu/tacl/article/43518),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.10.8.4)\.
- G\. Lai, Q\. Xie, H\. Liu, Y\. Yang, and E\. Hovy \(2017\)RACE: Large\-scale ReAding Comprehension Dataset From Examinations\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,Copenhagen, Denmark,pp\. 785–794\(en\)\.External Links:[Link](http://aclweb.org/anthology/D17-1082),[Document](https://dx.doi.org/10.18653/v1/D17-1082)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.15.13.4)\.
- N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi, N\. A\. Smith, and H\. Hajishirzi \(2025\)RewardBench: Evaluating Reward Models for Language Modeling\.Association for Computational Linguistics,Albuquerque, New Mexico\(en\)\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.96),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.96)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.16.14.4)\.
- A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo, Y\. Wu, B\. Neyshabur, G\. Gur\-Ari, and V\. Misra \(2022\)Solving Quantitative Reasoning Problems with Language Models\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 3843–3857\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/18abbeef8cfe9203fdf9053c9c4fe191-Paper-Conference.pdf)Cited by:[§2\.2](https://arxiv.org/html/2602.16763#S2.SS2.SSS0.Px3.p2.1)\.
- T\. Li\*, W\. Chiang\*, E\. Frick, L\. Dunlap, B\. Zhu, J\. E\. Gonzalez, and I\. Stoica \(2024\)From live data to high\-quality benchmarks: The arena\-hard pipeline\.External Links:[Link](https://lmsys.org/blog/2024-04-19-arena-hard/)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.8.6.2)\.
- P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar, B\. Newman, B\. Yuan, B\. Yan, C\. Zhang, C\. Cosgrove, C\. D\. Manning, C\. Re, D\. Acosta\-Navas, D\. A\. Hudson, E\. Zelikman, E\. Durmus, F\. Ladhak, F\. Rong, H\. Ren, H\. Yao, J\. Wang, K\. Santhanam, L\. Orr, L\. Zheng, M\. Yuksekgonul, M\. Suzgun, N\. Kim, N\. Guha, N\. S\. Chatterji, O\. Khattab, P\. Henderson, Q\. Huang, R\. A\. Chi, S\. M\. Xie, S\. Santurkar, S\. Ganguli, T\. Hashimoto, T\. Icard, T\. Zhang, V\. Chaudhary, W\. Wang, X\. Li, Y\. Mai, Y\. Zhang, and Y\. Koreeda \(2023\)Holistic Evaluation of Language Models\.Transactions on Machine Learning Research\.External Links:[Link](https://mlanthology.org/tmlr/2023/liang2023tmlr-holistic/)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p1.1),[Appendix A](https://arxiv.org/html/2602.16763#A1.p2.1),[§5\.3](https://arxiv.org/html/2602.16763#S5.SS3.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)TruthfulQA: Measuring How Models Mimic Human Falsehoods\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Dublin, Ireland,pp\. 3214–3252\(en\)\.External Links:[Link](https://aclanthology.org/2022.acl-long.229),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.229)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.26.24.4)\.
- Y\. L\. Liu, S\. L\. Blodgett, J\. C\. K\. Cheung, Q\. V\. Liao, A\. Olteanu, and Z\. Xiao \(2024\)ECBD: Evidence\-Centered Benchmark Design for NLP\.arXiv\.Note:Version Number: 1External Links:[Link](https://arxiv.org/abs/2406.08723),[Document](https://dx.doi.org/10.48550/ARXIV.2406.08723)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p1.1)\.
- N\. Maslej, L\. Fattorini, R\. Perrault, V\. Parli, A\. Reuel, E\. Brynjolfsson, J\. Etchemendy, K\. Ligett, T\. Lyons, J\. Manyika, J\. C\. Niebles, Y\. Shoham, R\. Wald, and J\. Clark \(2024\)Artificial Intelligence Index Report 2024\.arXiv\.Note:Version Number: 1External Links:[Link](https://arxiv.org/abs/2405.19522),[Document](https://dx.doi.org/10.48550/ARXIV.2405.19522)Cited by:[§1](https://arxiv.org/html/2602.16763#S1.p1.1)\.
- R\. T\. McCoy, E\. Pavlick, and T\. Linzen \(2019\)Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp\. 3428–3448\(en\)\.External Links:[Link](https://aclanthology.org/P19-1334),[Document](https://dx.doi.org/10.18653/v1/P19-1334)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p2.1),[§5\.2](https://arxiv.org/html/2602.16763#S5.SS2.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2602.16763#S5.p1.1)\.
- M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan, J\. Shen, G\. Ye, H\. Lin, J\. Poulos, M\. Wang, M\. Nezhurina, J\. Jitsev, D\. Lu, O\. M\. Mastromichalakis, Z\. Xu, Z\. Chen, Y\. Liu, R\. Zhang, L\. L\. Chen, A\. Kashyap, J\. Uslu, J\. Li, J\. Wu, M\. Yan, S\. Bian, V\. Sharma, K\. Sun, S\. Dillmann, A\. Anand, A\. Lanpouthakoun, B\. Koopah, C\. Hu, E\. Guha, G\. H\. S\. Dreiman, J\. Zhu, K\. Krauth, L\. Zhong, N\. Muennighoff, R\. Amanfu, S\. Tan, S\. Pimpalgaonkar, T\. Aggarwal, X\. Lin, X\. Lan, X\. Zhao, Y\. Liang, Y\. Wang, Z\. Wang, C\. Zhou, D\. Heineman, H\. Liu, H\. Trivedi, J\. Yang, J\. Lin, M\. Shetty, M\. Yang, N\. Omi, N\. Raoof, S\. Li, T\. Y\. Zhuo, W\. Lin, Y\. Dai, Y\. Wang, W\. Chai, S\. Zhou, D\. Wahdany, Z\. She, J\. Hu, Z\. Dong, Y\. Zhu, S\. Cui, A\. Saiyed, A\. Kolbeinsson, J\. Hu, C\. M\. Rytting, R\. Marten, Y\. Wang, A\. Dimakis, A\. Konwinski, and L\. Schmidt \(2026\)Terminal\-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces\.arXiv\.Note:Version Number: 1External Links:[Link](https://arxiv.org/abs/2601.11868),[Document](https://dx.doi.org/10.48550/ARXIV.2601.11868)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.24.22.4)\.
- T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal \(2018\)Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Brussels, Belgium,pp\. 2381–2391\(en\)\.External Links:[Link](http://aclweb.org/anthology/D18-1260),[Document](https://dx.doi.org/10.18653/v1/D18-1260)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.11.9.4)\.
- V\. Murahari, A\. Deshpande, P\. Clark, T\. Rajpurohit, A\. Sabharwal, K\. Narasimhan, and A\. Kalyan \(2024\)QualEval: Qualitative Evaluation for Model Improvement\.arXiv\.Note:arXiv:2311\.02807 \[cs\.LG\]Comment: NAACL 2024External Links:[Link](http://arxiv.org/abs/2311.02807),[Document](https://dx.doi.org/10.48550/arXiv.2311.02807)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p2.1),[§5\.1](https://arxiv.org/html/2602.16763#S5.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2602.16763#S5.p1.1)\.
- Y\. Nie, A\. Williams, E\. Dinan, M\. Bansal, J\. Weston, and D\. Kiela \(2020\)Adversarial NLI: A New Benchmark for Natural Language Understanding\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp\. 4885–4901\(en\)\.External Links:[Link](https://www.aclweb.org/anthology/2020.acl-main.441),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.441)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.6.4.2)\.
- OpenAI \(2020\)Dataset Language Statistics\.Technical reportOpenAI\.External Links:[Link](https://github.com/openai/gpt-3/tree/master/dataset_statistics)Cited by:[Appendix B](https://arxiv.org/html/2602.16763#A2.SS0.SSS0.Px2.p1.1)\.
- OpenAI \(2024a\)MMMLU: Multilingual massive multitask language understanding\.Hugging Face\.External Links:[Link](https://huggingface.co/datasets/openai/MMMLU)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.8.6.4)\.
- OpenAI \(2024b\)Why we no longer evaluate on swe\-bench verified\.Note:[https://openai\.com/index/why\-we\-no\-longer\-evaluate\-swe\-bench\-verified/](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/)Accessed: 2026\-05\-19Cited by:[§5\.2](https://arxiv.org/html/2602.16763#S5.SS2.SSS0.Px3.p2.1)\.
- S\. Ott, A\. Barbosa\-Silva, K\. Blagec, J\. Brauner, and M\. Samwald \(2022\)Mapping global dynamics of benchmark creation and saturation in artificial intelligence\.Nature Communications13\(1\),pp\. 6793\(en\)\.External Links:ISSN 2041\-1723,[Link](https://www.nature.com/articles/s41467-022-34591-0),[Document](https://dx.doi.org/10.1038/s41467-022-34591-0)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p2.1),[Appendix B](https://arxiv.org/html/2602.16763#A2.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2602.16763#S1.p1.1),[§2\.1](https://arxiv.org/html/2602.16763#S2.SS1.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2602.16763#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2602.16763#S5.SS1.SSS0.Px1.p1.1)\.
- L\. Pacchiardi, M\. Tesic, L\. G\. Cheke, and J\. Hernández\-Orallo \(2024\)Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers\.arXiv\.Note:Version Number: 1External Links:[Link](https://arxiv.org/abs/2410.11672),[Document](https://dx.doi.org/10.48550/ARXIV.2410.11672)Cited by:[§5\.2](https://arxiv.org/html/2602.16763#S5.SS2.SSS0.Px2.p1.1)\.
- R\. Y\. Pang, A\. Parrish, N\. Joshi, N\. Nangia, J\. Phang, A\. Chen, V\. Padmakumar, J\. Ma, J\. Thompson, H\. He, and S\. Bowman \(2022\)QuALITY: Question Answering with Long Input Texts, Yes\!\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Seattle, United States,pp\. 5336–5358\(en\)\.External Links:[Link](https://aclanthology.org/2022.naacl-main.391),[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.391)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.14.12.4)\.
- D\. Paperno, G\. Kruszewski, A\. Lazaridou, N\. Q\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. Fernandez \(2016\)The LAMBADA dataset: Word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Berlin, Germany,pp\. 1525–1534\(en\)\.External Links:[Link](http://aclweb.org/anthology/P16-1144),[Document](https://dx.doi.org/10.18653/v1/P16-1144)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.27.25.2)\.
- L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi, M\. Choi, A\. Agrawal, A\. Chopra, A\. Khoja, R\. Kim, R\. Ren, J\. Hausenloy, O\. Zhang, M\. Mazeika, D\. Dodonov, T\. Nguyen, J\. Lee, D\. Anderson, M\. Doroshenko, A\. C\. Stokes, M\. Mahmood, O\. Pokutnyi, O\. Iskra, J\. P\. Wang, J\. Levin, M\. Kazakov, F\. Feng, S\. Y\. Feng, H\. Zhao, M\. Yu, V\. Gangal, C\. Zou, Z\. Wang, S\. Popov, R\. Gerbicz, G\. Galgon, J\. Schmitt, W\. Yeadon, Y\. Lee, S\. Sauers, A\. Sanchez, F\. Giska, M\. Roth, S\. Riis, S\. Utpala, N\. Burns, G\. M\. Goshu, M\. M\. Naiya, C\. Agu, Z\. Giboney, A\. Cheatom, F\. Fournier\-Facio, S\. Crowson, L\. Finke, Z\. Cheng, J\. Zampese, R\. G\. Hoerr, M\. Nandor, H\. Park, T\. Gehrunger, J\. Cai, B\. McCarty, A\. C\. Garretson, E\. Taylor, D\. Sileo, Q\. Ren, U\. Qazi, L\. Li, J\. Nam, J\. B\. Wydallis, P\. Arkhipov, J\. W\. L\. Shi, A\. Bacho, C\. G\. Willcocks, H\. Cao, S\. Motwani, E\. d\. O\. Santos, J\. Veith, E\. Vendrow, D\. Cojoc, K\. Zenitani, J\. Robinson, L\. Tang, Y\. Li, J\. Vendrow, N\. W\. Fraga, V\. Kuchkin, A\. P\. Maksimov, P\. Marion, D\. Efremov, J\. Lynch, K\. Liang, A\. Mikov, A\. Gritsevskiy, J\. Guillod, G\. Demir, D\. Martinez, B\. Pageler, K\. Zhou, S\. Soori, O\. Press, H\. Tang, P\. Rissone, S\. R\. Green, L\. Brüssel, M\. Twayana, A\. Dieuleveut, J\. M\. Imperial, A\. Prabhu, J\. Yang, N\. Crispino, A\. Rao, D\. Zvonkine, G\. Loiseau, M\. Kalinin, M\. Lukas, C\. Manolescu, N\. Stambaugh, S\. Mishra, T\. Hogg, C\. Bosio, B\. P\. Coppola, J\. Salazar, J\. Jin, R\. Sayous, S\. Ivanov, P\. Schwaller, S\. Senthilkuma, A\. M\. Bran, A\. Algaba, K\. V\. d\. Houte, L\. Van Der Sypt, B\. Verbeken, D\. Noever, A\. Kopylov, B\. Myklebust, B\. Li, L\. Schut, E\. Zheltonozhskii, Q\. Yuan, D\. Lim, R\. Stanley, T\. Yang, J\. Maar, J\. Wykowski, M\. Oller, A\. Sahu, C\. G\. Ardito, Y\. Hu, A\. G\. K\. Kamdoum, A\. Jin, T\. G\. Vilchis, Y\. Zu, M\. Lackner, J\. Koppel, G\. Sun, D\. S\. Antonenko, S\. Chern, B\. Zhao, P\. Arsene, J\. M\. Cavanagh, D\. Li, J\. Shen, D\. Crisostomi, W\. Zhang, A\. Dehghan, S\. Ivanov, D\. Perrella, N\. Kaparov, A\. Zang, I\. Sucholutsky, A\. Kharlamova, D\. Orel, V\. Poritski, S\. Ben\-David, Z\. Berger, P\. Whitfill, M\. Foster, D\. Munro, L\. Ho, S\. Sivarajan, D\. B\. Hava, A\. Kuchkin, D\. Holmes, A\. Rodriguez\-Romero, F\. Sommerhage, A\. Zhang, R\. Moat, K\. Schneider, Z\. Kazibwe, D\. Clarke, D\. H\. Kim, F\. M\. Dias, S\. Fish, V\. Elser, T\. Kreiman, V\. E\. G\. Vilchis, I\. Klose, U\. Anantheswaran, A\. Zweiger, K\. Rawal, J\. Li, J\. Nguyen, N\. Daans, H\. Heidinger, M\. Radionov, V\. Rozhoň, V\. Ginis, C\. Stump, N\. Cohen, R\. Poświata, J\. Tkadlec, A\. Goldfarb, C\. Wang, P\. Padlewski, S\. Barzowski, K\. Montgomery, R\. Stendall, J\. Tucker\-Foltz, J\. Stade, T\. R\. Rogers, T\. Goertzen, D\. Grabb, A\. Shukla, A\. Givré, J\. A\. Ambay, A\. Sen, M\. F\. Aziz, M\. H\. Inlow, H\. He, L\. Zhang, Y\. Kaddar, I\. Ängquist, Y\. Chen, H\. K\. Wang, K\. Ramakrishnan, E\. Thornley, A\. Terpin, H\. Schoelkopf, E\. Zheng, A\. Carmi, E\. D\. L\. Brown, K\. Zhu, M\. Bartolo, R\. Wheeler, M\. Stehberger, P\. Bradshaw, J\. Heimonen, K\. Sridhar, I\. Akov, J\. Sandlin, Y\. Makarychev, J\. Tam, H\. Hoang, D\. M\. Cunningham, V\. Goryachev, D\. Patramanis, M\. Krause, A\. Redenti, D\. Aldous, J\. Lai, S\. Coleman, J\. Xu, S\. Lee, I\. Magoulas, S\. Zhao, N\. Tang, M\. K\. Cohen, O\. Paradise, J\. H\. Kirchner, M\. Ovchynnikov, J\. O\. Matos, A\. Shenoy, M\. Wang, Y\. Nie, A\. Sztyber\-Betley, P\. Faraboschi, R\. Riblet, J\. Crozier, S\. Halasyamani, S\. Verma, P\. Joshi, E\. Meril, Z\. Ma, J\. Andréoletti, R\. Singhal, J\. Platnick, V\. Nevirkovets, L\. Basler, A\. Ivanov, S\. Khoury, N\. Gustafsson, M\. Piccardo, H\. Mostaghimi, Q\. Chen, V\. Singh, T\. Q\. Khánh, P\. Rosu, H\. Szlyk, Z\. Brown, H\. Narayan, A\. Menezes, J\. Roberts, W\. Alley, K\. Sun, A\. Patel, M\. Lamparth, A\. Reuel, L\. Xin, H\. Xu, J\. Loader, F\. Martin, Z\. Wang, A\. Achilleos, T\. Preu, T\. Korbak, I\. Bosio, F\. Kazemi, Z\. Chen, B\. Bálint, E\. J\. Y\. Lo, J\. Wang, M\. I\. S\. Nunes, J\. Milbauer, M\. S\. Bari, Z\. Wang, B\. Ansarinejad, Y\. Sun, S\. Durand, H\. Elgnainy, G\. Douville, D\. Tordera, G\. Balabanian, H\. Wolff, L\. Kvistad, H\. Milliron, A\. Sakor, M\. Eron, A\. F\. D\. O\., S\. Shah, X\. Zhou, F\. Kamalov, S\. Abdoli, T\. Santens, S\. Barkan, A\. Tee, R\. Zhang, A\. Tomasiello, G\. B\. De Luca, S\. Looi, V\. Le, N\. Kolt, J\. Pan, E\. Rodman, J\. Drori, C\. J\. Fossum, N\. Muennighoff, M\. Jagota, R\. Pradeep, H\. Fan, J\. Eicher, M\. Chen, K\. Thaman, W\. Merrill, M\. Firsching, C\. Harris, S\. Ciobâcă, J\. Gross, R\. Pandey, I\. Gusev, A\. Jones, S\. Agnihotri, P\. Zhelnov, M\. Mofayezi, A\. Piperski, D\. K\. Zhang, K\. Dobarskyi, R\. Leventov, I\. Soroko, J\. Duersch, V\. Taamazyan, A\. Ho, W\. Ma, W\. Held, R\. Xian, A\. R\. Zebaze, M\. Mohamed, J\. N\. Leser, M\. X\. Yuan, L\. Yacar, J\. Lengler, K\. Olszewska, C\. Di Fratta, E\. Oliveira, J\. W\. Jackson, A\. Zou, M\. Chidambaram, T\. Manik, H\. Haffenden, D\. Stander, A\. Dasouqi, A\. Shen, B\. Golshani, D\. Stap, E\. Kretov, M\. Uzhou, A\. B\. Zhidkovskaya, N\. Winter, M\. O\. Rodriguez, R\. Lauff, D\. Wehr, C\. Tang, Z\. Hossain, S\. Phillips, F\. Samuele, F\. Ekström, A\. Hammon, O\. Patel, F\. Farhidi, G\. Medley, F\. Mohammadzadeh, M\. Peñaflor, H\. Kassahun, A\. Friedrich, R\. H\. Perez, D\. Pyda, T\. Sakal, O\. Dhamane, A\. K\. Mirabadi, E\. Hallman, K\. Okutsu, M\. Battaglia, M\. Maghsoudimehrabani, A\. Amit, D\. Hulbert, R\. Pereira, S\. Weber, Handoko, A\. Peristyy, S\. Malina, M\. Mehkary, R\. Aly, F\. Reidegeld, A\. Dick, C\. Friday, M\. Singh, H\. Shapourian, W\. Kim, M\. Costa, H\. Gurdogan, H\. Kumar, C\. Ceconello, C\. Zhuang, H\. Park, M\. Carroll, A\. R\. Tawfeek, S\. Steinerberger, D\. Aggarwal, M\. Kirchhof, L\. Dai, E\. Kim, J\. Ferret, J\. Shah, Y\. Wang, M\. Yan, K\. Burdzy, L\. Zhang, A\. Franca, D\. T\. Pham, K\. Y\. Loh, A\. Jackson, P\. Giordano, P\. Petersen, A\. Cosma, J\. Colino, C\. White, J\. Votava, V\. Vinnikov, E\. Delaney, P\. Spelda, V\. Stritecky, S\. M\. Shahid, J\. Mourrat, L\. Vetoshkin, K\. Sponselee, R\. Bacho, Z\. Yong, F\. de la Rosa, N\. Cho, X\. Li, G\. Malod, O\. Weller, G\. Albani, L\. Lang, J\. Laurendeau, D\. Kazakov, F\. Adesanya, J\. Portier, L\. Hollom, V\. Souza, Y\. A\. Zhou, J\. Degorre, Y\. Yalın, G\. D\. Obikoya, Rai, F\. Bigi, M\. C\. Boscá, O\. Shumar, K\. Bacho, G\. Recchia, M\. Popescu, N\. Shulga, N\. M\. Tanwie, T\. C\. H\. Lux, B\. Rank, C\. Ni, M\. Brooks, A\. Yakimchyk, Huanxu, Liu, S\. Cavalleri, O\. Häggström, E\. Verkama, J\. Newbould, H\. Gundlach, L\. Brito\-Santana, B\. Amaro, V\. Vajipey, R\. Grover, T\. Wang, Y\. Kratish, W\. Li, S\. Gopi, A\. Caciolai, C\. S\. de Witt, P\. Hernández\-Cámara, E\. Rodolà, J\. Robins, D\. Williamson, V\. Cheng, B\. Raynor, H\. Qi, B\. Segev, J\. Fan, S\. Martinson, E\. Y\. Wang, K\. Hausknecht, M\. P\. Brenner, M\. Mao, C\. Demian, P\. Kassani, X\. Zhang, D\. Avagian, E\. J\. Scipio, A\. Ragoler, J\. Tan, B\. Sims, R\. Plecnik, A\. Kirtland, O\. F\. Bodur, D\. P\. Shinde, Y\. C\. L\. Labrador, Z\. Adoul, M\. Zekry, A\. Karakoc, T\. C\. B\. Santos, S\. Shamseldeen, L\. Karim, A\. Liakhovitskaia, N\. Resman, N\. Farina, J\. C\. Gonzalez, G\. Maayan, E\. Anderson, R\. D\. O\. Pena, E\. Kelley, H\. Mariji, R\. Pouriamanesh, W\. Wu, R\. Finocchio, I\. Alarab, J\. Cole, D\. Ferreira, B\. Johnson, M\. Safdari, L\. Dai, S\. Arthornthurasuk, I\. C\. McAlister, A\. J\. Moyano, A\. Pronin, J\. Fan, A\. Ramirez\-Trinidad, Y\. Malysheva, D\. Pottmaier, O\. Taheri, S\. Stepanic, S\. Perry, L\. Askew, R\. A\. H\. Rodríguez, A\. M\. R\. Minissi, R\. Lorena, K\. Iyer, A\. A\. Fasiludeen, R\. Clark, J\. Ducey, M\. Piza, M\. Somrak, E\. Vergo, J\. Qin, B\. Borbás, E\. Chu, J\. Lindsey, A\. Jallon, I\. M\. J\. McInnis, E\. Chen, A\. Semler, L\. Gloor, T\. Shah, M\. Carauleanu, P\. Lauer, T\. \\\. Huy, H\. Shahrtash, E\. Duc, L\. Lewark, A\. Brown, S\. Albanie, B\. Weber, W\. S\. Vaz, P\. Clavier, Y\. Fan, G\. P\. R\. e\. Silva, Long, Lian, M\. Abramovitch, X\. Jiang, S\. Mendoza, M\. Islam, J\. Gonzalez, V\. Mavroudis, J\. Xu, P\. Kumar, L\. P\. Goswami, D\. Bugas, N\. Heydari, F\. Jeanplong, T\. Jansen, A\. Pinto, A\. Apronti, A\. Galal, N\. Ze\-An, A\. Singh, T\. Jiang, J\. o\. A\. Xavier, K\. P\. Agarwal, M\. Berkani, G\. Zhang, Z\. Du, B\. A\. d\. O\. Junior, D\. Malishev, N\. Remy, T\. D\. Hartman, T\. Tarver, S\. Mensah, G\. A\. Loume, W\. Morak, F\. Habibi, S\. Hoback, W\. Cai, J\. Gimenez, R\. G\. Montecillo, J\. Łucki, R\. Campbell, A\. Sharma, K\. Meer, S\. Gul, D\. E\. Gonzalez, X\. Alapont, A\. Hoover, G\. Chhablani, F\. Vargus, A\. Agarwal, Y\. Jiang, D\. Patil, D\. Outevsky, K\. J\. Scaria, R\. Maheshwari, A\. Dendane, P\. Shukla, A\. Cartwright, S\. Bogdanov, N\. Mündler, S\. Möller, L\. Arnaboldi, K\. Thaman, M\. R\. Siddiqi, P\. Saxena, H\. Gupta, T\. Fruhauff, G\. Sherman, M\. Vincze, S\. Usawasutsakorn, D\. Ler, A\. Radhakrishnan, I\. Enyekwe, S\. M\. Salauddin, J\. Muzhen, A\. Maksapetyan, V\. Rossbach, C\. Harjadi, M\. Bahaloohoreh, C\. Sparrow, J\. Sidhu, S\. Ali, S\. Bian, J\. Lai, E\. Singer, J\. L\. Uro, G\. Bateman, M\. Sayed, A\. Menshawy, D\. Duclosel, D\. Bezzi, Y\. Jain, A\. Aaron, M\. Tiryakioglu, S\. Siddh, K\. Krenek, I\. A\. Shah, J\. Jin, S\. Creighton, D\. Peskoff, Z\. EL\-Wasif, R\. P, M\. Richmond, J\. McGowan, T\. Patwardhan, H\. Sun, T\. Sun, N\. Zubić, S\. Sala, S\. Ebert, J\. Kaddour, M\. Schottdorf, D\. Wang, G\. Petruzella, A\. Meiburg, T\. Medved, A\. ElSheikh, S\. A\. Hebbar, L\. Vaquero, X\. Yang, J\. Poulos, V\. Zouhar, S\. Bogdanik, M\. Zhang, J\. Sanz\-Ros, D\. Anugraha, Y\. Dai, A\. N\. Nhu, X\. Wang, A\. A\. Demircali, Z\. Jia, Y\. Zhou, J\. Wu, M\. He, N\. Chandok, A\. Sinha, G\. Luo, L\. Le, M\. Noyé, M\. Perełkiewicz, I\. Pantidis, T\. Qi, S\. S\. Purohit, L\. Parcalabescu, T\. Nguyen, G\. I\. Winata, E\. M\. Ponti, H\. Li, K\. Dhole, J\. Park, D\. Abbondanza, Y\. Wang, A\. Nayak, D\. M\. Caetano, A\. A\. W\. L\. Wong, M\. del Rio\-Chanona, D\. Kondor, P\. Francois, E\. Chalstrey, J\. Zsambok, D\. Hoyer, J\. Reddish, J\. Hauser, F\. Rodrigo\-Ginés, S\. Datta, M\. Shepherd, T\. Kamphuis, Q\. Zhang, H\. Kim, R\. Sun, J\. Yao, F\. Dernoncourt, S\. Krishna, S\. Rismanchian, B\. Pu, F\. Pinto, Y\. Wang, K\. Shridhar, K\. J\. Overholt, G\. Briia, H\. Nguyen, David, S\. Bartomeu, T\. C\. Pang, A\. Wecker, Y\. Xiong, F\. Li, L\. S\. Huber, J\. Jaeger, R\. De Maddalena, X\. H\. Lù, Y\. Zhang, C\. Beger, P\. T\. J\. Kon, S\. Li, V\. Sanker, M\. Yin, Y\. Liang, X\. Zhang, A\. Agrawal, L\. S\. Yifei, Z\. Zhang, M\. Cai, Y\. Sonmez, C\. Cozianu, C\. Li, A\. Slen, S\. Yu, H\. K\. Park, G\. Sarti, M\. Briański, A\. Stolfo, T\. A\. Nguyen, M\. Zhang, Y\. Perlitz, J\. Hernandez\-Orallo, R\. Li, A\. Shabani, F\. Juefei\-Xu, S\. Dhingra, O\. Zohar, M\. C\. Nguyen, A\. Pondaven, A\. Yilmaz, X\. Zhao, C\. Jin, M\. Jiang, S\. Todoran, X\. Han, J\. Kreuer, B\. Rabern, A\. Plassart, M\. Maggetti, L\. Yap, R\. Geirhos, J\. Kean, D\. Wang, S\. Mollaei, C\. Sun, Y\. Yin, S\. Wang, R\. Li, Y\. Chang, A\. Wei, A\. Bizeul, X\. Wang, A\. O\. Arrais, K\. Mukherjee, J\. Chamorro\-Padial, J\. Liu, X\. Qu, J\. Guan, A\. Bouyamourn, S\. Wu, M\. Plomecka, J\. Chen, M\. Tang, J\. Deng, S\. Subramanian, H\. Xi, H\. Chen, W\. Zhang, Y\. Ren, H\. Tu, S\. Kim, Y\. Chen, S\. V\. Marjanović, J\. Ha, G\. Luczyna, J\. J\. Ma, Z\. Shen, D\. Song, C\. E\. Zhang, Z\. Wang, G\. Gendron, Y\. Xiao, L\. Smucker, E\. Weng, K\. H\. Lee, Z\. Ye, S\. Ermon, I\. D\. Lopez\-Miguel, T\. Knights, A\. Gitter, N\. Park, B\. Wei, H\. Chen, K\. Pai, A\. Elkhanany, H\. Lin, P\. D\. Siedler, J\. Fang, R\. Mishra, K\. Zsolnai\-Fehér, X\. Jiang, S\. Khan, J\. Yuan, R\. K\. Jain, X\. Lin, M\. Peterson, Z\. Wang, A\. Malusare, M\. Tang, I\. Gupta, I\. Fosin, T\. Kang, B\. Dworakowska, K\. Matsumoto, G\. Zheng, G\. Sewuster, J\. P\. Villanueva, I\. Rannev, I\. Chernyavsky, J\. Chen, D\. Banik, B\. Racz, W\. Dong, J\. Wang, L\. Bashmal, D\. V\. Gonçalves, W\. Hu, K\. Bar, O\. Bohdal, A\. S\. Patlan, S\. Dhuliawala, C\. Geirhos, J\. Wist, Y\. Kansal, B\. Chen, K\. Tire, A\. T\. Yücel, B\. Christof, V\. Singla, Z\. Song, S\. Chen, J\. Ge, K\. Ponkshe, I\. Park, T\. Shi, M\. Q\. Ma, J\. Mak, S\. Lai, A\. Moulin, Z\. Cheng, Z\. Zhu, Z\. Zhang, V\. Patil, K\. Jha, Q\. Men, J\. Wu, T\. Zhang, B\. H\. Vieira, A\. F\. Aji, J\. Chung, M\. Mahfoud, H\. T\. Hoang, M\. Sperzel, W\. Hao, K\. Meding, S\. Xu, V\. Kostakos, D\. Manini, Y\. Liu, C\. Toukmaji, J\. Paek, E\. Yu, A\. E\. Demircali, Z\. Sun, I\. Dewerpe, H\. Qin, R\. Pflugfelder, J\. Bailey, J\. Morris, V\. Heilala, S\. Rosset, Z\. Yu, P\. E\. Chen, W\. Yeo, E\. Jain, R\. Yang, S\. Chigurupati, J\. Chernyavsky, S\. P\. Reddy, S\. Venugopalan, H\. Batra, C\. F\. Park, H\. Tran, G\. Maximiano, G\. Zhang, Y\. Liang, H\. Shiyu, R\. Xu, R\. Pan, S\. Suresh, Z\. Liu, S\. Gulati, S\. Zhang, P\. Turchin, C\. W\. Bartlett, C\. R\. Scotese, P\. M\. Cao, B\. Wu, J\. Karwowski, D\. Scaramuzza, A\. Nattanmai, G\. McKellips, A\. Cheraku, A\. Suhail, E\. Luo, M\. Deng, J\. Luo, A\. Zhang, K\. Jindel, K\. Halevy, A\. Baranov, M\. Liu, A\. Avadhanam, D\. Zhang, B\. Ma, E\. Fu, L\. Do, J\. Lass, H\. Yang, S\. Sunkari, V\. Bharath, V\. Ai, J\. Leung, R\. Agrawal, A\. Zhou, K\. Chen, T\. Kalpathi, Z\. Xu, G\. Wang, T\. Xiao, E\. Maung, S\. Lee, R\. Yue, B\. Zhao, J\. Yoon, S\. Sun, A\. Singh, C\. Peng, T\. Osbey, T\. Wang, D\. Echeazu, T\. Wu, S\. Patel, V\. Kulkarni, V\. Sundarapandiyan, A\. Le, Z\. Nasim, S\. Yalam, R\. Kasamsetty, S\. Samal, D\. Sun, N\. Shah, A\. Saha, A\. Zhang, L\. Nguyen, L\. Nagumalli, K\. Wang, A\. Wu, A\. Telluri, S\. Dillmann, Z\. Wang, J\. Luo, H\. Lunn, A\. Gazizov, H\. Qiu, A\. G\. Hart, R\. B\. Gabrielsson, A\. Lukoianov, S\. Yue, A\. Wang, and D\. Hendrycks \(2025\)Humanity’s Last Exam\.arXiv\.Note:Version Number: 10Other 29 pages, 6 figuresExternal Links:[Link](https://arxiv.org/abs/2501.14249),[Document](https://dx.doi.org/10.48550/ARXIV.2501.14249)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.25.23.2)\.
- I\. D\. Raji, E\. M\. Bender, A\. Paullada, E\. Denton, and A\. Hanna \(2021\)AI and the Everything in the Whole Wide World Benchmark\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Note:arXiv: 2111\.15366Comment: Accepted in NeurIPS 2021 Benchmarks and Datasets trackExternal Links:[Link](https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/084b6fbb10729ed4da8c3d3f5a3ae7c9-Paper-round2.pdf)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p1.1)\.
- P\. Rajpurkar, R\. Jia, and P\. Liang \(2018\)Know What You Don’t Know: Unanswerable Questions for SQuAD\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),Melbourne, Australia,pp\. 784–789\(en\)\.External Links:[Link](http://aclweb.org/anthology/P18-2124),[Document](https://dx.doi.org/10.18653/v1/P18-2124)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.20.18.4)\.
- P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. Liang \(2016\)SQuAD: 100,000\+ Questions for Machine Comprehension of Text\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,Austin, Texas,pp\. 2383–2392\(en\)\.External Links:[Link](http://aclweb.org/anthology/D16-1264),[Document](https://dx.doi.org/10.18653/v1/D16-1264)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.20.18.4)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2023\)GPQA: A Graduate\-Level Google\-Proof Q&amp;A Benchmark\.arXiv\.Note:Version Number: 1Other 28 pages, 5 figures, 7 tablesExternal Links:[Link](https://arxiv.org/abs/2311.12022),[Document](https://dx.doi.org/10.48550/ARXIV.2311.12022)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.19.17.2)\.
- O\. Sainz, I\. García\-Ferrero, A\. Jacovi, J\. Ander Campos, Y\. Elazar, E\. Agirre, Y\. Goldberg, W\. Chen, J\. Chim, L\. Choshen, L\. D’Amico\-Wong, M\. Dell, R\. Fan, S\. Golchin, Y\. Li, P\. Liu, B\. Pahwa, A\. Prabhu, S\. Sharma, E\. Silcock, K\. Solonko, D\. Stap, M\. Surdeanu, Y\. Tseng, V\. Udandarao, Z\. Wang, R\. Xu, and J\. Yang \(2024\)Data Contamination Report from the 2024 CONDA Shared Task\.InProceedings of the 1st Workshop on Data Contamination \(CONDA\),Bangkok, Thailand,pp\. 41–56\(en\)\.External Links:[Link](https://aclanthology.org/2024.conda-1.4),[Document](https://dx.doi.org/10.18653/v1/2024.conda-1.4)Cited by:[§5\.2](https://arxiv.org/html/2602.16763#S5.SS2.SSS0.Px1.p1.1)\.
- K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. Choi \(2020\)WinoGrande: An Adversarial Winograd Schema Challenge at Scale\.Proceedings of the AAAI Conference on Artificial Intelligence34\(05\),pp\. 8732–8740\.External Links:ISSN 2374\-3468, 2159\-5399,[Link](https://ojs.aaai.org/index.php/AAAI/article/view/6399),[Document](https://dx.doi.org/10.1609/aaai.v34i05.6399)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.29.27.4)\.
- O\. E\. Salaudeen, A\. Reuel, A\. M\. Ahmed, S\. Bedi, Z\. Robertson, S\. Sundar, B\. W\. Domingue, A\. Wang, and S\. Koyejo \(2025\)Measurement to Meaning: A Validity\-Centered Framework for AI Evaluation\.InNeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling,External Links:[Link](https://openreview.net/forum?id=2Bw6uC49QF)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p1.1),[footnote 2](https://arxiv.org/html/2602.16763#footnote2)\.
- M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. Choi \(2019\)Social IQa: Commonsense Reasoning about Social Interactions\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),Hong Kong, China,pp\. 4462–4472\(en\)\.External Links:[Link](https://www.aclweb.org/anthology/D19-1454),[Document](https://dx.doi.org/10.18653/v1/D19-1454)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.19.17.4)\.
- M\. Saxon, A\. Holtzman, P\. West, W\. Y\. Wang, and N\. Saphra \(2024\)Benchmarks as Microscopes: A Call for Model Metrology\.arXiv\.Note:Version Number: 2Other Conference paper at COLM 2024External Links:[Link](https://arxiv.org/abs/2407.16711),[Document](https://dx.doi.org/10.48550/ARXIV.2407.16711)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p2.1),[§5\.1](https://arxiv.org/html/2602.16763#S5.SS1.SSS0.Px2.p1.1)\.
- R\. Schaeffer \(2023\)Pretraining on the Test Set Is All You Need\.arXiv\.Note:Version Number: 1Other 3 pages, satireExternal Links:[Link](https://arxiv.org/abs/2309.08632),[Document](https://dx.doi.org/10.48550/ARXIV.2309.08632)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p2.1),[§5\.2](https://arxiv.org/html/2602.16763#S5.SS2.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2602.16763#S5.p1.1)\.
- F\. Shi, M\. Suzgun, M\. Freitag, X\. Wang, S\. Srivats, S\. Vosoughi, H\. W\. Chung, Y\. Tay, S\. Ruder, D\. Zhou, D\. Das, and J\. Wei \(2023\)Language models are multilingual chain\-of\-thought reasoners\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=fR3wGCk-IXp)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.3.1.4)\.
- S\. Singh, A\. Romanou, C\. Fourrier, D\. I\. Adelani, J\. G\. Ngui, D\. Vila\-Suero, P\. Limkonchotiwat, K\. Marchisio, W\. Q\. Leong, Y\. Susanto, R\. Ng, S\. Longpre, S\. Ruder, W\. Ko, A\. Bosselut, A\. Oh, A\. Martins, L\. Choshen, D\. Ippolito, E\. Ferrante, M\. Fadaee, B\. Ermis, and S\. Hooker \(2025\)Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 18761–18799\(en\)\.External Links:[Link](https://aclanthology.org/2025.acl-long.919),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.919)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.17.15.2)\.
- G\. Son, J\. Hong, H\. Ko, and J\. Thorne \(2025\)Linguistic Generalizability of Test\-Time Scaling in Mathematical Reasoning\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 14333–14368\(en\)\.External Links:[Link](https://aclanthology.org/2025.acl-long.699),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.699)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.28.26.4)\.
- A\. Srivastava, A\. Rastogi, A\. Rao, A\. A\. M\. Shoeb, A\. Abid, A\. Fisch, A\. R\. Brown, A\. Santoro, A\. Gupta, A\. Garriga\-Alonso, A\. Kluska, A\. Lewkowycz, A\. Agarwal, A\. Power, A\. Ray, A\. Warstadt, A\. W\. Kocurek, A\. Safaya, A\. Tazarv, A\. Xiang, A\. Parrish, A\. Nie, A\. Hussain, A\. Askell, A\. Dsouza, A\. Slone, A\. Rahane, A\. S\. Iyer, A\. J\. Andreassen, A\. Madotto, A\. Santilli, A\. Stuhlmüller, A\. M\. Dai, A\. La, A\. K\. Lampinen, A\. Zou, A\. Jiang, A\. Chen, A\. Vuong, A\. Gupta, A\. Gottardi, A\. Norelli, A\. Venkatesh, A\. Gholamidavoodi, A\. Tabassum, A\. Menezes, A\. Kirubarajan, A\. Mullokandov, A\. Sabharwal, A\. Herrick, A\. Efrat, A\. Erdem, A\. Karakaş, B\. R\. Roberts, B\. S\. Loe, B\. Zoph, B\. Bojanowski, B\. Özyurt, B\. Hedayatnia, B\. Neyshabur, B\. Inden, B\. Stein, B\. Ekmekci, B\. Y\. Lin, B\. Howald, B\. Orinion, C\. Diao, C\. Dour, C\. Stinson, C\. Argueta, C\. Ferri, C\. Singh, C\. Rathkopf, C\. Meng, C\. Baral, C\. Wu, C\. Callison\-Burch, C\. Waites, C\. Voigt, C\. D\. Manning, C\. Potts, C\. Ramirez, C\. E\. Rivera, C\. Siro, C\. Raffel, C\. Ashcraft, C\. Garbacea, D\. Sileo, D\. Garrette, D\. Hendrycks, D\. Kilman, D\. Roth, C\. D\. Freeman, D\. Khashabi, D\. Levy, D\. M\. González, D\. Perszyk, D\. Hernandez, D\. Chen, D\. Ippolito, D\. Gilboa, D\. Dohan, D\. Drakard, D\. Jurgens, D\. Datta, D\. Ganguli, D\. Emelin, D\. Kleyko, D\. Yuret, D\. Chen, D\. Tam, D\. Hupkes, D\. Misra, D\. Buzan, D\. C\. Mollo, D\. Yang, D\. Lee, D\. Schrader, E\. Shutova, E\. D\. Cubuk, E\. Segal, E\. Hagerman, E\. Barnes, E\. Donoway, E\. Pavlick, E\. Rodolà, E\. Lam, E\. Chu, E\. Tang, E\. Erdem, E\. Chang, E\. A\. Chi, E\. Dyer, E\. Jerzak, E\. Kim, E\. E\. Manyasi, E\. Zheltonozhskii, F\. Xia, F\. Siar, F\. Martínez\-Plumed, F\. Happé, F\. Chollet, F\. Rong, G\. Mishra, G\. I\. Winata, G\. d\. Melo, G\. Kruszewski, G\. Parascandolo, G\. Mariani, G\. X\. Wang, G\. Jaimovitch\-Lopez, G\. Betz, G\. Gur\-Ari, H\. Galijasevic, H\. Kim, H\. Rashkin, H\. Hajishirzi, H\. Mehta, H\. Bogar, H\. F\. A\. Shevlin, H\. Schuetze, H\. Yakura, H\. Zhang, H\. M\. Wong, I\. Ng, I\. Noble, J\. Jumelet, J\. Geissinger, J\. Kernion, J\. Hilton, J\. Lee, J\. F\. Fisac, J\. B\. Simon, J\. Koppel, J\. Zheng, J\. Zou, J\. Kocon, J\. Thompson, J\. Wingfield, J\. Kaplan, J\. Radom, J\. Sohl\-Dickstein, J\. Phang, J\. Wei, J\. Yosinski, J\. Novikova, J\. Bosscher, J\. Marsh, J\. Kim, J\. Taal, J\. Engel, J\. Alabi, J\. Xu, J\. Song, J\. Tang, J\. Waweru, J\. Burden, J\. Miller, J\. U\. Balis, J\. Batchelder, J\. Berant, J\. Frohberg, J\. Rozen, J\. Hernandez\-Orallo, J\. Boudeman, J\. Guerr, J\. Jones, J\. B\. Tenenbaum, J\. S\. Rule, J\. Chua, K\. Kanclerz, K\. Livescu, K\. Krauth, K\. Gopalakrishnan, K\. Ignatyeva, K\. Markert, K\. Dhole, K\. Gimpel, K\. Omondi, K\. W\. Mathewson, K\. Chiafullo, K\. Shkaruta, K\. Shridhar, K\. McDonell, K\. Richardson, L\. Reynolds, L\. Gao, L\. Zhang, L\. Dugan, L\. Qin, L\. Contreras\-Ochando, L\. Morency, L\. Moschella, L\. Lam, L\. Noble, L\. Schmidt, L\. He, L\. Oliveros\-Colón, L\. Metz, L\. K\. Senel, M\. Bosma, M\. Sap, M\. T\. Hoeve, M\. Farooqi, M\. Faruqui, M\. Mazeika, M\. Baturan, M\. Marelli, M\. Maru, M\. J\. Ramirez\-Quintana, M\. Tolkiehn, M\. Giulianelli, M\. Lewis, M\. Potthast, M\. L\. Leavitt, M\. Hagen, M\. Schubert, M\. O\. Baitemirova, M\. Arnaud, M\. McElrath, M\. A\. Yee, M\. Cohen, M\. Gu, M\. Ivanitskiy, M\. Starritt, M\. Strube, M\. Swędrowski, M\. Bevilacqua, M\. Yasunaga, M\. Kale, M\. Cain, M\. Xu, M\. Suzgun, M\. Walker, M\. Tiwari, M\. Bansal, M\. Aminnaseri, M\. Geva, M\. Gheini, M\. V\. T, N\. Peng, N\. A\. Chi, N\. Lee, N\. G\. Krakover, N\. Cameron, N\. Roberts, N\. Doiron, N\. Martinez, N\. Nangia, N\. Deckers, N\. Muennighoff, N\. S\. Keskar, N\. S\. Iyer, N\. Constant, N\. Fiedel, N\. Wen, O\. Zhang, O\. Agha, O\. Elbaghdadi, O\. Levy, O\. Evans, P\. A\. M\. Casares, P\. Doshi, P\. Fung, P\. P\. Liang, P\. Vicol, P\. Alipoormolabashi, P\. Liao, P\. Liang, P\. W\. Chang, P\. Eckersley, P\. M\. Htut, P\. Hwang, P\. Miłkowski, P\. Patil, P\. Pezeshkpour, P\. Oli, Q\. Mei, Q\. Lyu, Q\. Chen, R\. Banjade, R\. E\. Rudolph, R\. Gabriel, R\. Habacker, R\. Risco, R\. Millière, R\. Garg, R\. Barnes, R\. A\. Saurous, R\. Arakawa, R\. Raymaekers, R\. Frank, R\. Sikand, R\. Novak, R\. Sitelew, R\. L\. Bras, R\. Liu, R\. Jacobs, R\. Zhang, R\. Salakhutdinov, R\. A\. Chi, S\. R\. Lee, R\. Stovall, R\. Teehan, R\. Yang, S\. Singh, S\. M\. Mohammad, S\. Anand, S\. Dillavou, S\. Shleifer, S\. Wiseman, S\. Gruetter, S\. R\. Bowman, S\. S\. Schoenholz, S\. Han, S\. Kwatra, S\. A\. Rous, S\. Ghazarian, S\. Ghosh, S\. Casey, S\. Bischoff, S\. Gehrmann, S\. Schuster, S\. Sadeghi, S\. Hamdan, S\. Zhou, S\. Srivastava, S\. Shi, S\. Singh, S\. Asaadi, S\. S\. Gu, S\. Pachchigar, S\. Toshniwal, S\. Upadhyay, S\. S\. Debnath, S\. Shakeri, S\. Thormeyer, S\. Melzi, S\. Reddy, S\. P\. Makini, S\. Lee, S\. Torene, S\. Hatwar, S\. Dehaene, S\. Divic, S\. Ermon, S\. Biderman, S\. Lin, S\. Prasad, S\. Piantadosi, S\. Shieber, S\. Misherghi, S\. Kiritchenko, S\. Mishra, T\. Linzen, T\. Schuster, T\. Li, T\. Yu, T\. Ali, T\. Hashimoto, T\. Wu, T\. Desbordes, T\. Rothschild, T\. Phan, T\. Wang, T\. Nkinyili, T\. Schick, T\. Kornev, T\. Tunduny, T\. Gerstenberg, T\. Chang, T\. Neeraj, T\. Khot, T\. Shultz, U\. Shaham, V\. Misra, V\. Demberg, V\. Nyamai, V\. Raunak, V\. V\. Ramasesh, v\. u\. prabhu, V\. Padmakumar, V\. Srikumar, W\. Fedus, W\. Saunders, W\. Zhang, W\. Vossen, X\. Ren, X\. Tong, X\. Zhao, X\. Wu, X\. Shen, Y\. Yaghoobzadeh, Y\. Lakretz, Y\. Song, Y\. Bahri, Y\. Choi, Y\. Yang, S\. Hao, Y\. Chen, Y\. Belinkov, Y\. Hou, Y\. Hou, Y\. Bai, Z\. Seid, Z\. Zhao, Z\. Wang, Z\. J\. Wang, Z\. Wang, and Z\. Wu \(2023\)Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models\.Transactions on Machine Learning Research\.Note:Featured CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=uyTL5Bvosj)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p1.1),[§3\.1](https://arxiv.org/html/2602.16763#S3.SS1.SSS0.Px2.p1.1),[§5\.3](https://arxiv.org/html/2602.16763#S5.SS3.p1.1)\.
- A\. Subramonian, X\. Yuan, H\. Daumé Iii, and S\. L\. Blodgett \(2023\)It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance\.InFindings of the Association for Computational Linguistics: ACL 2023,Toronto, Canada,pp\. 3234–3279\(en\)\.External Links:[Link](https://aclanthology.org/2023.findings-acl.202),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.202)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p1.1)\.
- M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou, and J\. Wei \(2023\)Challenging BIG\-Bench Tasks and Whether Chain\-of\-Thought Can Solve Them\.InFindings of the Association for Computational Linguistics: ACL 2023,Toronto, Canada,pp\. 13003–13051\(en\)\.External Links:[Link](https://aclanthology.org/2023.findings-acl.824),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.824)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.10.8.2)\.
- A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4149–4158\.External Links:[Link](https://aclanthology.org/N19-1421/),[Document](https://dx.doi.org/10.18653/v1/N19-1421)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.13.11.2)\.
- T\. T\. Team \(2025\)Terminal\-bench: A benchmark for AI agents in terminal environments\.External Links:[Link](https://github.com/laude-institute/terminal-bench)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.23.21.4)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. C\. Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. Scialom \(2023\)Llama 2: Open Foundation and Fine\-Tuned Chat Models\.arXiv\.Note:Version Number: 2External Links:[Link](https://arxiv.org/abs/2307.09288),[Document](https://dx.doi.org/10.48550/ARXIV.2307.09288)Cited by:[Appendix B](https://arxiv.org/html/2602.16763#A2.SS0.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2602.16763#S5.SS2.SSS0.Px3.p1.1)\.
- E\. Union \(2024\)Article 51: Classification of general\-purpose AI models as general\-purpose AI models with systemic risk\.Note:\[Accessed 12\-01\-2026\]External Links:[Link](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)Cited by:[§1](https://arxiv.org/html/2602.16763#S1.p1.1)\.
- D\. Vilares and C\. Gómez\-Rodríguez \(2019\)HEAD\-QA: A Healthcare Dataset for Complex Reasoning\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp\. 960–966\(en\)\.External Links:[Link](https://aclanthology.org/P19-1092),[Document](https://dx.doi.org/10.18653/v1/P19-1092)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.1.3)\.
- A\. Wang, Y\. Pruksachatkun, N\. Nangia, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman \(2019\)SuperGLUE: a stickier benchmark for general\-purpose language understanding systems\.InProceedings of the 33rd International Conference on Neural Information Processing Systems,Red Hook, NY, USA\.Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.21.19.4)\.
- A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. Bowman \(2018\)GLUE: A Multi\-Task Benchmark and Analysis Platform for Natural Language Understanding\.Association for Computational Linguistics,Brussels, Belgium\(en\)\.External Links:[Link](http://aclweb.org/anthology/W18-5446),[Document](https://dx.doi.org/10.18653/v1/W18-5446)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.18.16.2)\.
- W\. Wang, Z\. Tu, C\. Chen, Y\. Yuan, J\. Huang, W\. Jiao, and M\. Lyu \(2024a\)All Languages Matter: On the Multilingual Safety of LLMs\.InFindings of the Association for Computational Linguistics ACL 2024,Bangkok, Thailand and virtual meeting,pp\. 5865–5877\(en\)\.External Links:[Link](https://aclanthology.org/2024.findings-acl.349),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.349)Cited by:[Appendix B](https://arxiv.org/html/2602.16763#A2.SS0.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2602.16763#S5.SS2.SSS0.Px3.p1.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen \(2024b\)MMLU\-Pro: a more robust and challenging multi\-task language understanding benchmark\.NIPS ’24,Curran Associates Inc\.,Red Hook, NY, USA\.External Links:ISBN 979\-8\-3313\-1438\-5Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.6.4.4),[§1](https://arxiv.org/html/2602.16763#S1.p2.1),[§2\.1](https://arxiv.org/html/2602.16763#S2.SS1.p1.1)\.
- J\. Wei, N\. Karina, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. Fedus \(2024\)Measuring short\-form factuality in large language models\.arXiv\.Note:Version Number: 1Other Blog post: https://openai\.com/index/introducing\-simpleqa/External Links:[Link](https://arxiv.org/abs/2411.04368),[Document](https://dx.doi.org/10.48550/ARXIV.2411.04368)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.18.16.4)\.
- K\. Wei, P\. Paskov, S\. Dev, M\. J\. Byun, A\. Reuel, X\. Roberts\-Gaal, R\. Calcott, E\. Coxon, and C\. Deshpande \(2025\)Position: human baselines in model evaluations need rigor and transparency \(With recommendations & reporting checklist\)\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 82265–82325\.External Links:[Link](https://proceedings.mlr.press/v267/wei25s.html)Cited by:[§2\.1](https://arxiv.org/html/2602.16763#S2.SS1.SSS0.Px1.p1.1)\.
- C\. White, S\. Dooley, M\. Roberts, A\. Pal, B\. Feuer, S\. Jain, R\. Shwartz\-Ziv, N\. Jain, K\. Saifullah, S\. Dey, Shubh\-Agrawal, S\. Sandha, S\. Naidu, C\. Hegde, Y\. LeCun, T\. Goldstein, W\. Neiswanger, and M\. Goldblum \(2025\)LiveBench: A Challenging, Contamination\-Limited LLM Benchmark\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 91595–91631\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/e4a46394ba5378b3f9a186a5b4c650d1-Paper-Conference.pdf)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.29.27.2)\.
- Y\. Xue, X\. Cao, X\. Yang, Y\. Wang, R\. Wang, and J\. Li \(2023\)We Need to Talk About Reproducibility in NLP Model Comparison\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 9424–9434\(en\)\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.586),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.586)Cited by:[Appendix A](https://arxiv.org/html/2602.16763#A1.p2.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: Can a Machine Really Finish Your Sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp\. 4791–4800\(en\)\.External Links:[Link](https://aclanthology.org/P19-1472),[Document](https://dx.doi.org/10.18653/v1/P19-1472)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.23.21.2)\.
- Y\. Zeng, Y\. Yang, A\. Zhou, J\. Z\. Tan, Y\. Tu, Y\. Mai, K\. Klyman, M\. Pan, R\. Jia, D\. Song, P\. Liang, and B\. Li \(2024\)AIR\-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies\.arXiv\.Note:Version Number: 2External Links:[Link](https://arxiv.org/abs/2407.17436),[Document](https://dx.doi.org/10.48550/ARXIV.2407.17436)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.5.3.2)\.
- Z\. Zheng, Z\. Cheng, Z\. Shen, S\. Zhou, K\. Liu, H\. He, D\. Li, S\. Wei, H\. Hao, J\. Yao, P\. Sheng, Z\. Wang, W\. Chai, A\. Korolova, P\. Henderson, S\. Arora, P\. Viswanath, J\. Shang, and S\. Xie \(2025\)LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/9712b78386cebdc3db7f1a48c2d20edb-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.30.28.2)\.
- W\. Zhong, R\. Cui, Y\. Guo, Y\. Liang, S\. Lu, Y\. Wang, A\. Saied, W\. Chen, and N\. Duan \(2024\)AGIEval: A Human\-Centric Benchmark for Evaluating Foundation Models\.InFindings of the Association for Computational Linguistics: NAACL 2024,Mexico City, Mexico,pp\. 2299–2314\(en\)\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.149),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.149)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.3.1.2)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023a\)Instruction\-Following Evaluation for Large Language Models\.arXiv\.Note:Version Number: 1External Links:[Link](https://arxiv.org/abs/2311.07911),[Document](https://dx.doi.org/10.48550/ARXIV.2311.07911)Cited by:[Table 3](https://arxiv.org/html/2602.16763#A3.T3.1.26.24.2)\.
- K\. Zhou, Y\. Zhu, Z\. Chen, W\. Chen, W\. X\. Zhao, X\. Chen, Y\. Lin, J\. Wen, and J\. Han \(2023b\)Don’t Make Your LLM an Evaluation Benchmark Cheater\.arXiv\.Note:Version Number: 1Other 11 pagesExternal Links:[Link](https://arxiv.org/abs/2311.01964),[Document](https://dx.doi.org/10.48550/ARXIV.2311.01964)Cited by:[Appendix B](https://arxiv.org/html/2602.16763#A2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2602.16763#S5.SS1.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2602.16763#S5.SS2.SSS0.Px1.p1.1)\.

## Appendix ARelated Work

Benchmark Design & Evolution\.The development of AI benchmarks has evolved alongside advances in models, with an increasing focus on broad coverage and rigorous design\(Hardyet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib16); Liuet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib134)\)\. Recent efforts emphasize diversity of tasks and continuous updates: for example, the BIG\-Bench project crowdsourced hundreds of tasks to test language models’ breadth\(Srivastavaet al\.,[2023](https://arxiv.org/html/2602.16763#bib.bib121)\), and dynamic benchmarks like Dynabench introduced adversarial, ongoingly collected test data so that evaluation remains challenging as models improve\(Kielaet al\.,[2021](https://arxiv.org/html/2602.16763#bib.bib135)\)\. New benchmark paradigms also expand how performance is measured\. The Holistic Evaluation of Language Models initiative, for instance, treats evaluation as a “living” benchmark that is continuously updated and tracks multiple metrics \(accuracy, calibration, fairness, etc\.\) across many scenarios\(Lianget al\.,[2023](https://arxiv.org/html/2602.16763#bib.bib24)\)\. Additionally, researchers have proposed principled frameworks for benchmark construction to ensure that datasets, tasks, and metrics truly capture the targeted capabilities\(Salaudeenet al\.,[2025](https://arxiv.org/html/2602.16763#bib.bib136); Liuet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib134); Subramonianet al\.,[2023](https://arxiv.org/html/2602.16763#bib.bib139); Rajiet al\.,[2021](https://arxiv.org/html/2602.16763#bib.bib140)\)\.

Issues in AI Evaluations\.Despite continual benchmark innovation, significant challenges persist in how we evaluate AI systems\. Various works have highlighted fundamental evaluation pitfalls in AI evaluation: Data contamination, i\.e\., when test content appears in training, can artificially inflate scores\. For example,Schaeffer \([2023](https://arxiv.org/html/2602.16763#bib.bib137)\)demonstrated that directly fine\-tuning on a test set yields nearly perfect accuracy\. Gamability of benchmarks is another concern: models often exploit spurious correlations or annotation artifacts to get high accuracy without genuine understanding\. For instance,McCoyet al\.\([2019](https://arxiv.org/html/2602.16763#bib.bib25)\)have shown that models rely on superficial cues \(e\.g\.,lexical overlap or keyword hints\) instead of robust reasoning, achieving “right for the wrong reason” performance that fails on stress tests\. Finally, reproducibility remains a challenge in AI evaluation: Seemingly superior results frequently vanish under minor experimental changes \(i\.e\.,simply altering random seeds or dataset splits can yield statistically significant performance fluctuations and inconsistent evaluation protocols or opaque reporting have further complicated fair comparison of models\(Xueet al\.,[2023](https://arxiv.org/html/2602.16763#bib.bib138); Habbaet al\.,[2025](https://arxiv.org/html/2602.16763#bib.bib21); Ashury\-Tahanet al\.,[2026b](https://arxiv.org/html/2602.16763#bib.bib4)\)\)\. In parallel to this workAshury\-Tahanet al\.\([2026a](https://arxiv.org/html/2602.16763#bib.bib3)\), shows that this brittleness is highly reduced with saturation\. Finally,Ottet al\.\([2022](https://arxiv.org/html/2602.16763#bib.bib32)\)show that benchmark saturation is a common occurrence, potentially making them misleading indicators of progress once models overfit to test quirks rather than achieve substantive gains\. Yet,Ottet al\.\([2022](https://arxiv.org/html/2602.16763#bib.bib32)\)neither quantitatively define benchmark saturation nor do the authors analyze the causes of such saturation, two gaps we fill in this work\. Relatedly, there is increasing awareness that aggregate metrics such as accuracy, F1, or single\-scale scores often fail to capture nuanced model behavior\(Murahariet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib10)\)\. This coarseness can create a misleading sense of benchmark saturation – models may reach near\-ceiling aggregate scores while still exhibit substantial variation across subskills or input types\. This apparent saturation, driven by the insensitivity of aggregate metrics, obscures remaining weaknesses and limits the diagnostic value of benchmarks\. Consequently, several works advocate for holistic evaluation frameworks, including hybrid scoring schemes\(Lianget al\.,[2023](https://arxiv.org/html/2602.16763#bib.bib24)\)or even a new discipline of model metrology to formalize rigorous, fine\-grained measurement practices\(Saxonet al\.,[2024](https://arxiv.org/html/2602.16763#bib.bib42)\)\.

## Appendix BHypotheses

We investigate five hypotheses about factors driving benchmark saturation, grounded in prior literature and design challenges\. To support this analysis, we annotated 60 LLM benchmarks with related properties such as task format, data curation, and known quality issues\. These annotations, detailed in Sec\.[3](https://arxiv.org/html/2602.16763#S3), enable empirical testing of the hypotheses \(Sec\.[4](https://arxiv.org/html/2602.16763#S4)\)\.

#### \(H1\) Data Access and Test Set Exposure:

Public benchmarks saturate faster than private benchmarks with held\-out test sets\.When test questions are public, models often memorize or leak this content from their training corpora, yielding artificially high scores that do not reflect true generalization:Zhouet al\.\([2023b](https://arxiv.org/html/2602.16763#bib.bib107)\)demonstrate that if an LLM’s pre\-training data contains examples from an evaluation benchmark, the model’s score on that benchmark is significantly boosted\. Likewise,Balloccuet al\.\([2024](https://arxiv.org/html/2602.16763#bib.bib108)\)conducted a large\-scale analysis of GPT\-3\.5 and GPT\-4 and found they were exposed to approximately 4\.7 million benchmark samples during training, which may explain why these models quickly achieve near\-perfect scores on popular public tests\.Denget al\.\([2024](https://arxiv.org/html/2602.16763#bib.bib127)\)devised a protocol to probe contamination on knowledge benchmarks and found that GPT\-4 and Claude could fill in missing parts of real test questions with unnaturally high accuracy, implying the models had internalized those test items\. These findings support H1: because public benchmarks are easily scraped or overfit, top model scores on them often reflect memorization\.

#### \(H2\) Language Coverage:

English\-only benchmarks saturate faster than multilingual or mixed\-language benchmarks\.English dominates the pre\-training corpora of most models \(often \>85–90% of tokens\)\(Touvronet al\.,[2023](https://arxiv.org/html/2602.16763#bib.bib128); OpenAI,[2020](https://arxiv.org/html/2602.16763#bib.bib133)\)\.Wanget al\.\([2024a](https://arxiv.org/html/2602.16763#bib.bib122)\)observe that an LLM’s ranking across languages correlates strongly with the proportion of that language in its training data, where models consistently excel at English and a few other high\-resource languages, but struggle as one moves to less\-seen languages\. Consequently, an English\-only task can hit a performance ceiling quicker because the model’s exposure to English makes the task easier in distribution\. In contrast, a multilingual benchmark introduces linguistic diversity that challenges the model’s weaker languages and forces more robust generalization\(Huet al\.,[2020](https://arxiv.org/html/2602.16763#bib.bib129)\)\.

#### \(H3\) Data Curation Strategy:

Human\-authored benchmarks are more resistant to performance saturation than synthetic or hybrid ones\.Human\-curated evaluations typically span a richer diversity of problems and deeper conceptual challenges, often including intentionally difficult or adversarially crafted questions that thwart simple pattern\-matching:Daset al\.\([2024](https://arxiv.org/html/2602.16763#bib.bib132)\)found that LLM outputs risk repetitive formats and missing corner\-case reasoning\. Diversity and deliberate complexity introduced by humans make it harder for models to “solve” benchmark tasks by exploiting superficial regularities\(Gillet al\.,[2025](https://arxiv.org/html/2602.16763#bib.bib131)\)\. By contrast, LLM\-generated \(synthetic\) benchmarks tend to exhibit hidden structural patterns or stylistic biases that models quickly learn to exploit, yielding artificially high scores without commensurate gains in real capability\(Gillet al\.,[2025](https://arxiv.org/html/2602.16763#bib.bib131)\)\.

#### \(H4\) Task Output Format:

Benchmarks that use a closed\-ended response format \(e\.g\. multiple\-choice, true/false\) tend to saturate faster than those requiring open\-ended generation\.Closed\-ended tasks constrain the output space, making it easier for models to guess or recognize the correct answer without full understanding\. The underlying mechanism is that closed formats convert complex tasks into simpler classification problems: the model’s job is reduced to selecting one ofNNoptions, a setup amenable to elimination strategies, frequency biases, or even memorized question\-option pairs\. Moreover, closed\-ended benchmarks typically have an inherent guessing baseline \(e\.g\. 25% for 4\-choice questions\), so even an uninformed model starts at a higher performance floor\. Recent work demonstrated that some MCQ benchmarks enable overfitting of models such that they pick the right option without even reading the question\(Chandaket al\.,[2025](https://arxiv.org/html/2602.16763#bib.bib130)\)\. By contrast, open\-ended prompts \(where the model must generate a free\-form answer, explanation, or output\) vastly expand the solution space and typically require a deeper grasp of the problem\.

#### \(H5\) Benchmark Maturity and Popularity:

Benchmarks that are older and more widely adopted saturate faster than newer or less\-used benchmarks\.As benchmarks mature and become widely adopted by the research community, they are repeatedly used for model development, hyperparameter tuning, prompt engineering, and evaluation, increasing optimization pressure against the benchmark itself\. Prior work has noted that performance on popular benchmarks often improves rapidly shortly after release and then plateaus as models converge on similar solutions\(Ottet al\.,[2022](https://arxiv.org/html/2602.16763#bib.bib32)\)\. Moreover, widely adopted benchmarks are more likely to be included—directly or indirectly—in training data or evaluation pipelines, further accelerating score convergence\. As a result, benchmark age and popularity may jointly contribute to saturation by increasing exposure and targeted optimization, even when absolute task difficulty remains unchanged\.

#### \(H6\) Template vs Non\-Template:

Non\-templated benchmarks are more resistant to performance saturation than templated benchmarks\.Templated benchmarks generate data samples using predefined patterns, structures, or parameterized templates, often resulting in repeated surface forms with limited variation\. While such designs enable scalability and controlled coverage, they can introduce regularities that models quickly learn to exploit\. In contrast, non\-templated benchmarks consist of more diverse, free\-form instances that are less constrained by fixed generation patterns\. We therefore hypothesize that templated benchmarks, due to their structural regularities and reduced diversity, are more prone to faster saturation compared to non\-templated, free\-form benchmarks\.

## Appendix CSemantic Scholar Benchmark Collection

We retrieved all benchmarks appearing in the most\-cited research papers between 2022 and November 2025 using Semantic Scholar API and the following queries \(50 per keyword\):language model evaluation,LLM benchmark,foundation model benchmark,language model benchmarkandlanguage model evaluation benchmark\. The Semantic Scholar API retrieves the most relevant papers within a given time period\. We first retrieve 200 relevant papers per keyword and select the top 50 cited papers\. After merging all retrieved papers and deduplicating, we identified 186 papers using these keywords\. We excluded non\-text\-based benchmarks\. This keyword\-based search yielded 2 additional benchmarks that were previously absent from our collection\.

Table 3:Benchmarks included in our analysis \(N=60\)
## Appendix DField Definitions for Annotation and Examples

This appendix provides detailed tables describing the annotation schema and benchmark metadata used in our analysis \([Table˜4](https://arxiv.org/html/2602.16763#A4.T4)\), along with example rows illustrating the collected saturation metrics \([Table˜6](https://arxiv.org/html/2602.16763#A4.T6)\) and dataset properties \([Table˜5](https://arxiv.org/html/2602.16763#A4.T5)\)\.

Table 4:Benchmark Annotation Schema\. Each benchmark in our dataset is annotated with the following fields to enable systematic analysis of saturation dynamics\.Table 5:Benchmark Dataset Properties \(Example Rows\)Table 6:Benchmark Saturation Analysis \(Example Rows\)
## Appendix EBenchmark\-Level Saturation \- Overview and Case Studies

To complement our analysis, we provide benchmark\-level case studies in[Table˜7](https://arxiv.org/html/2602.16763#A5.T7)illustrating how the saturation index behaves across different benchmarks\. These examples highlight how score compression, evaluation uncertainty, and dataset properties jointly determine whether a benchmark is saturated, stagnated, or remains discriminative\. These examples show a range of saturation indices from fully saturated benchmarks, where evaluation noise obscures all meaningful differences, to unsaturated benchmarks that retain strong discriminative power\.

Math\-500 \(very high saturation,Sindex=0\.92S\_\{\\text\{index\}\}=0\.92\)\. The Math\-500 leaderboard shows that top\-performing models are tightly clustered within a 1\.0\-point range \(98\.2–99\.2\), which lies within the estimated evaluation uncertainty \(S​EΔ=0\.0338SE\_\{\\Delta\}=0\.0338\)\. This results in a low normalized range \(Rnorm=0\.30R\_\{\\text\{norm\}\}=0\.30\), indicating that performance differences are not statistically meaningful and the benchmark has lost discriminative power\.

LiveBench \(very high saturation,Sindex=0\.99S\_\{\\text\{index\}\}=0\.99\)\. Although designed to mitigate contamination through regular updates, LiveBench shows high score compression \(range = 1\.09\) relative to its uncertainty \(S​EΔ=0\.1028SE\_\{\\Delta\}=0\.1028\), resulting a very lowRnorm=0\.11R\_\{\\text\{norm\}\}=0\.11\. Notably, this occurs at moderate performance levels \( 79%\), suggesting model\-level stagnation rather than task completion\.

LiveCodeBench \(high saturation,Sindex=0\.77S\_\{\\text\{index\}\}=0\.77\)\. LiveCodeBench shows stronger separation among top models \(performance range = 3\.9\), which results in a higher normalized range \(Rnorm=0\.51R\_\{\\text\{norm\}\}=0\.51\)\. While still showing score compression, it demonstrates that also dynamically constructed benchmarks can saturate when evaluation resolution is limited\.

TruthfulQA \(moderate saturation,Sindex=0\.55S\_\{\\text\{index\}\}=0\.55\)\. The TruthfulQA leaderbaord shows a wider spread among top models \(range = 6\.7\), which exceeds evaluation uncertainty\. This leads to meaningful differentiation \(Rnorm=0\.78R\_\{\\text\{norm\}\}=0\.78\), but partial clustering indicates early signs of convergence, which is consistent with the benchmarks age and exposure\.

Humanity’s Last Exam \(low saturation,Sindex=0\.22S\_\{\\text\{index\}\}=0\.22\)\. This benchmark shows substantial separation among top models \(range = 11\.4\), which exceeds uncertainty \(Rnorm=1\.23R\_\{\\text\{norm\}\}=1\.23\)\. Combined with its large test set and recent release, it retains strong discriminative power and shows now clear sign of saturation\.

Table 7:Representative benchmarks illustrating different saturation regimes\.
## Appendix FFurther Saturation Analysis

This appendix presents additional results from the joint regression analysis, including posterior coefficient estimates \([Figure˜5](https://arxiv.org/html/2602.16763#A6.F5)\) and model performance, to provide a more detailed view of the factors associated with benchmark saturation \([Figure˜6](https://arxiv.org/html/2602.16763#A6.F6)\)\.

![Refer to caption](https://arxiv.org/html/2602.16763v3/x4.png)Figure 5:Forest plot of posterior regression coefficients from the joint interaction model predicting benchmark saturation\. Points denote posterior means, inner line segments show 50% highest posterior density intervals, and outer segments indicate 95% credible intervals\. Benchmark age and test set size exhibit the most consistent effects on saturation, while task format, literal diversity \(templating\), and their interactions show no strong effects after controlling for confounders\.![Refer to caption](https://arxiv.org/html/2602.16763v3/x5.png)Figure 6:Posterior distribution of the AUROC for the interaction model predicting benchmark saturation\. The distribution is tightly concentrated near high values \(median approx\. 0\.98\), indicating that the model distinguishes saturated from non\-saturated benchmarks across posterior samples\.
## Appendix GAuthor Contribution Statement

ConceptualizationM\. Akhtar, A\. ReuelData CurationM\. Akhtar, P\. Soni, S\. Ahuja, P\. Ammanamanchi, R\. Rawal, S\. Yadav, C\. Whitehouse, D\. Ki, J\. Mickel, M\. Šuppa, J\. Batzner, J\. Chim, J\. Sania, Y\. Long, H\. Rahmani, C\. Knight, Y\. Nan, J\. Raj, Y\. Fan, S\. Singh, S\. Sahoo, E\. Habba, S\. Pawar, R\. Scholz, A\. Subramanian, J\. Ni, L\. Struppek, A\. GhoshInvestigationM\. Akhtar, P\. Soni, S\. Ahuja, P\. Ammanamanchi, R\. Rawal, V\. Zouhar, S\. Yadav, C\. Whitehouse, D\. Ki, L\. Ibrahim, J\. Raj, Y\. Fan, L\. Struppek, U\. Gohar, J\. MickelMethodologyM\. Akhtar, A\. Reuel, P\. Soni, S\. Ahuja, P\. Ammanamanchi, R\. Rawal, V\. Zouhar, S\. Yadav, C\. Whitehouse, D\. KiSoftwareM\. Akhtar, P\. Soni, S\. Ahuja, P\. Ammanamanchi, R\. Rawal, V\. Zouhar, S\. Yadav, C\. Whitehouse, D\. Ki, J\. Raj, M\. ŠuppaFormal AnalysisM\. Akhtar, P\. Soni, S\. Ahuja, P\. Ammanamanchi, R\. Rawal, V\. Zouhar, S\. Yadav, C\. Whitehouse, D\. Ki, Y\. Long, J\. Chim, J\. Sania, M\. Šuppa, Y\. NanWriting \(Original Draft\)M\. Akhtar, A\. Reuel, P\. Soni, S\. Ahuja, P\. Ammanamanchi, R\. Rawal, V\. Zouhar, S\. Yadav, C\. Whitehouse, D\. Ki, R\. Scholz, L\. Ibrahim, Y\. FanWriting \(Review & Editing\)M\. Akhtar, A\. Reuel, P\. Soni, S\. Ahuja, P\. Ammanamanchi, R\. Rawal, V\. Zouhar, S\. Yadav, C\. Whitehouse, D\. Ki, J\. Mickel, L\. Choshen, M\. Šuppa, J\. Batzner, J\. Chim, J\. Sania, Y\. Long, H\. Rahmani, C\. Knight, Y\. Nan, J\. Raj, Y\. Fan, S\. Singh, S\. Sahoo, E\. Habba, U\. Gohar, S\. Pawar, R\. Scholz, A\. Subramanian, J\. Ni, L\. Struppek, L\. Ibrahim, M\. Kochenderfer, S\. Koyejo, M\. Sachan, S\. Biderman, Z\. Talat, A\. Ghosh, I\. SolaimanVisualizationM\. Akhtar, J\. Raj, V\. Zouhar, M\. Šuppa, C\. WhitehouseSupervisionM\. Akhtar, A\. Reuel, L\. Choshen, M\. Kochenderfer, S\. Koyejo, M\. Sachan, S\. Biderman, Z\. Talat, A\. Ghosh, I\. Solaiman