Position: Ideas Should be the Center of Machine Learning Research

# Position: Ideas Should be the Center of Machine Learning Research
Source: [https://arxiv.org/html/2605.15253](https://arxiv.org/html/2605.15253)
###### Abstract

Machine learning research increasingly bifurcates into two disconnected modes: benchmark\-driven engineering that prioritizes metrics over understanding, and idealized theory that often fails to transfer to modern systems\. In this position paper, we argue that the field focuses too heavily on these endpoints, neglecting the central scientific object: the idea\. We propose an Ideas First framework in which*ideas*are valued for the behavioral*signatures*they predict in modern models, and these signatures are tested through*tailored experiments*designed to detect the relevant patterns rather than to win leaderboards\. This shift not only bridges the gap between theory and practice but also promotes equity by removing the “complexity premium,” enabling rigorous scientific contributions from researchers with modest computational, financial, and human resources\. Ultimately, we advocate for a research culture centered on ideas, treating benchmarks and theorems as instruments for testing mechanistic hypotheses rather than as ends in themselves\.

Machine Learning, ICML

## 1Introduction

Machine learning research has converged on two dominant modes\. On the empirical side, shared benchmarks and leaderboards define success: contributions are judged primarily by their effect on a single number on a widely recognized test set\(Lipton and Steinhardt,[2018](https://arxiv.org/html/2605.15253#bib.bib1); Dehghaniet al\.,[2021](https://arxiv.org/html/2605.15253#bib.bib37)\)\. On the theoretical side, many results are evaluated by the strength of guarantees in highly idealized settings that only partially resemble contemporary overparameterized models\(Baraniuket al\.,[2020](https://arxiv.org/html/2605.15253#bib.bib54); Zhanget al\.,[2017](https://arxiv.org/html/2605.15253#bib.bib53)\)\. Both modes have delivered real progress, yet they also narrow what counts as a contribution and who can participate, particularly when large complex models, and extensive ablations become de facto requirements\(Ahmed and Wahed,[2020](https://arxiv.org/html/2605.15253#bib.bib80)\)\.

#### Our position\.

In this paper we argue that the central scientific object in machine learning is not a benchmark score or a theorem, but an*idea*: a hypothesis about how learning systems work, what kinds of structure they can exploit, or how they should be evaluated\. We call our perspective*Ideas First*\. Ideas gain scientific content when they give rise to*signatures*that we can look for in the behavior of modern models \(Figure[1](https://arxiv.org/html/2605.15253#S1.F1)\), such as specific patterns, characteristic failure modes, or qualitative changes in representation geometry\. Experiments are then designed to search for these signatures and rule out competing explanations, rather than primarily to maximize performance on a fixed leaderboard\. This reverses the usual order of justification: instead of starting from a large system, benchmark, or theory and asking what results can be extracted, we start from the idea and ask what behavior it predicts and how those predictions can be tested\. On this view, simple models, small\-scale experiments, and minimal theoretical analyses can be especially valuable when they isolate the mechanism of interest, while large benchmarks, complex systems, and sophisticated theorems remain important when they clarify, operationalize, or stress\-test the idea\. We are not against benchmarks or theory; we are in favor of placing ideas at the center so that strong ideas can later generate both informative benchmarks and illuminating theorems without being defined by them at birth\.

#### Equity and simplicity\.

This shift also puts equity and simplicity at its core\. When publication standards implicitly demand large models, complex implementations, and exhaustive ablations, researchers with substantial computational and engineering resources are favored, while simple but sharp ideas that can be tested in modest settings struggle to be seen\(Ahmed and Wahed,[2020](https://arxiv.org/html/2605.15253#bib.bib80); Strubellet al\.,[2019](https://arxiv.org/html/2605.15253#bib.bib81); Schwartzet al\.,[2020](https://arxiv.org/html/2605.15253#bib.bib106)\)\. Large labs can distribute work across many people, combining teams that run large scale experiments with teams that derive sophisticated theoretical results\. This capacity can and does lead to impressive contributions\. The problem arises when the same expectations are used as a gatekeeper: an idea can be rejected because it lacks a strong theorem in a highly idealized setting whose assumptions limit its relevance, or because it lacks large benchmark results that may themselves be vulnerable to overfitting and design artifacts\(Dehghaniet al\.,[2021](https://arxiv.org/html/2605.15253#bib.bib37); Bouthillieret al\.,[2021](https://arxiv.org/html/2605.15253#bib.bib38)\)\. History and sociology of science suggest that major advances often emerge from chains of small, conceptually clear steps rather than isolated breakthroughs\(Merton,[1973](https://arxiv.org/html/2605.15253#bib.bib121); Kuhn,[1962](https://arxiv.org/html/2605.15253#bib.bib122)\)\. A culture that filters out such steps because they are not wrapped in heavy systems or frontier benchmarks risks discarding the very ideas that future theory and benchmarks could build on\.

#### Change of culture\.

Our position is a call to return to hypothesis driven inquiry in the sense used in empirical sciences such as physics and biology\. This style of work already underlies much of the most insightful machine learning research \(see Section[4](https://arxiv.org/html/2605.15253#S4)\), but it is rarely foregrounded in how contributions are framed, evaluated, and rewarded\. We argue that the community should normalize the Ideas First perspective, especially for junior researchers, and align reviewing and publishing norms with it\. Doing so would deliberately shift incentives away from a status quo that unconsciously mimics the resource intensive priorities of large industrial labs toward a culture in which clear ideas, explicit signatures, and accessible experiments are recognized as core scientific contributions\.

#### Goals\.

The objectives of this position paper are threefold: \(i\) We propose a framework that links*ideas*to*predicted signatures*to*tailored experiments*for modern neural systems\. \(ii\) We develop a case in favor of idea centric evaluation, connecting it to work on benchmarks, theory, philosophy of science, and equity in AI\. \(iii\) We provide concrete examples of this framework in practice, together with practical notions for evaluating ideas\.

#### Structure of the paper\.

Section[2](https://arxiv.org/html/2605.15253#S2)introduces our critique of benchmark driven engineering and idealized theory\. Section[3](https://arxiv.org/html/2605.15253#S3)presents the Ideas First framework\. Section[4](https://arxiv.org/html/2605.15253#S4)revisits several deep learning results through this lens, and Section[5](https://arxiv.org/html/2605.15253#S5)offers practical guidance for authors and reviewers\. Section[6](https://arxiv.org/html/2605.15253#S6)gives a hypothetical case study that illustrates the full workflow\. Section[7](https://arxiv.org/html/2605.15253#S7)develops the broader case for idea centric evaluation and its implications\. Finally, we present alternative views in Section[8](https://arxiv.org/html/2605.15253#S8)\.

![Refer to caption](https://arxiv.org/html/2605.15253v1/x1.png)Figure 1:\(Top\) Current practice often bifurcates into*Benchmark\-Driven Engineering*\(optimizing a single metric on large systems, often obscuring mechanism\) and*Idealized Theory*\(rigorous proofs in simplified settings that may not transfer to modern models\)\. \(Bottom\) Our proposed framework links these worlds: An*Idea*generates a concrete*Signature*\(an observable behavioral commitment \)\. A Tailored*Experiment*is then designed specifically to detect this signature, prioritizing mechanistic clarity and accessible experimental design over state\-of\-the\-art leaderboard performance\.

## 2Standard modes

Before introducing our framework, we briefly characterize two standard modes that structure much of contemporary work on neural systems\. While we acknowledge that research methodologies are diverse, we argue that with sufficient flexibility, the incentives of the field tend to align the vast majority of current work with one of these two dominant archetypes\.

### Mode A: Benchmark\-driven engineering

This mode improves comparative performance on standard tasks by scaling models, adjusting training recipes, and modifying architectures and data\. The central artifact is a system together with transparent comparisons on leaderboards and ablation studies that indicate which choices influenced the metric\. Success is a higher score at fixed resources or at clearly reported increases in resources\. The unit of evidence is a demonstrated score gain supported by controlled ablations\. The strength of this mode is that it maps where and how well a method works across tasks and scales\. Its limitation is that it remains largely agnostic to mechanism and can confound the effects of scale, data, and compute unless coupled to analyses that target explanation\.

#### Illustrative papers\.

AlexNet\(Krizhevskyet al\.,[2012](https://arxiv.org/html/2605.15253#bib.bib20)\), ResNet\(Heet al\.,[2016](https://arxiv.org/html/2605.15253#bib.bib21)\), and EfficientNet\(Tan and Le,[2019](https://arxiv.org/html/2605.15253#bib.bib22)\); in multimodal and language, CLIP\(Radfordet al\.,[2021](https://arxiv.org/html/2605.15253#bib.bib23)\), GPT−3\(Brownet al\.,[2020](https://arxiv.org/html/2605.15253#bib.bib24)\), and Chinchilla\(Hoffmannet al\.,[2022](https://arxiv.org/html/2605.15253#bib.bib3)\)\.

### Mode B: Idealized theory

This mode derives provable statements in stylized or asymptotic regimes, for example width tending to infinity, infinitesimal step sizes, or noiseless labels\. The central artifacts are theorems, bounds, and limiting dynamics, and the objective is logical correctness under explicit assumptions\. The unit of evidence is a formal proof, sometimes accompanied by simulation in simplified settings\. The strength of this mode is conceptual precision: it produces definitions, constraints, and organizing principles\. Its limitation is that observable consequences for finite modern systems are often left implicit, measurable diagnostics are usually out of scope, and external validity remains unknown until tested\.

#### Illustrative papers\.

Spectrally normalized margin bounds\(Bartlettet al\.,[2017](https://arxiv.org/html/2605.15253#bib.bib25)\), norm\-based capacity and implicit regularization\(Neyshaburet al\.,[2015](https://arxiv.org/html/2605.15253#bib.bib26)\), exact learning dynamics in deep linear networks\(Saxeet al\.,[2014](https://arxiv.org/html/2605.15253#bib.bib29)\), and benign overfitting in linear regression\(Bartlettet al\.,[2020](https://arxiv.org/html/2605.15253#bib.bib30)\)\.

#### Takeaway\.

These modes answer different questions:*What works?*\(benchmarks\) and*What must be true under certain assumptions?*\(theory\)\. Our proposal in the rest of the paper is to foreground the missing question:*What should we see if a given idea about mechanism is right?*

### 2\.1What can go wrong?

When pursued in isolation these modes can lead to problems, either specific \(with direct impact on the current study\) or general \(with broad impact on the research community\):

#### Benchmark myopia \(Mode A\)\.

Optimizing a single score obscures mechanism and is inconclusive when multiple changes move the metric in offsetting ways\.*Consequence:*improvements are hard to attribute; negative or mechanism\-revealing results are underreported\.

#### Transfer gap \(Mode B\)\.

Theorems crafted in idealized limits rarely issue observable predictions for real models\.*Consequence:*theory informs intuitions but does not guide measurement; experiments cannot falsify the claims\.

#### Non\-cumulative findings \(within empirical practice\)\.

Exploratory ablations and robustness checks often document effects without anchoring them to hypotheses or controls predicted to be inert\.*Consequence:*results do not compose; later studies cannot easily reuse or contradict prior claims\.

#### Complexity premium\.

Reviewing often treats complexity and scale as a proxy for depth: large models, elaborate training pipelines, and intricate theory are seen as more serious than simple baselines and ideas that isolate a mechanism\.*Consequence:*incentives favor “bigger and more complex” over “cleaner and truer” and discourage small, sharp experiments or proofs that make precise, testable commitments\.

#### Resource asymmetry\.

State\-of\-the\-art \(SOTA\) incentives, meaning incentives to achieve or surpass the best reported performance on a benchmark or task, disproportionately reward compute\- and data\-heavy results\. This makes it costly to test ideas unless they are bundled with large\-scale experiments or elaborate systems\. Large labs can distribute work across many people and machines; smaller groups cannot\.*Consequence:*publication bias toward scale, underinvestment in mechanism\-driven tests, weaker reproducibility for labs without access to large budgets, and a higher chance that simple but sharp ideas never leave the drawing board\.

## 3Ideas First

Now that we have introduced the standard modes of research in neural models alongside possible problems that they face, we can sketch what research looks like when ideas—not just final metrics—do the heavy lifting\. In this view, an idea earns weight by making observable commitments \(signatures\) that experiments can seek and stress\-test\. Our proposed framework can be summarized as

i​d​e​a→s​i​g​n​a​t​u​r​e→t​a​i​l​o​r​e​d​e​x​p​e​r​i​m​e​n​t\.idea\\rightarrow signature\\rightarrow\{tailored\\ experiment\}\.
We treat progress as a chain from*ideas*to*experiments*, with*signatures*as the link\. An idea is useful when it makes something*visible*; an experiment is useful when it is*built to see that thing*\. Benchmarks remain valuable instruments for external validity and regressions, but*signatures*, not ranks, are our unit of explanation and value\. Let’s elaborate on our three linked notions of ideas, signatures and experiments\.

#### What is an idea?

An*idea*is a scope\-bearing claim about*how or why*a system behaves, usually established in a*simplified setting*\(e\.g\., single layer, tied weights, infinite width, synthetic data\) where analysis or controlled observation is feasible\. The idea names the mechanism or constraint, states where it is meant to apply and where it may fail, and issues*observable commitments*\. Its value is conceptual and predictive, not benchmark\-driven: it makes something precise enough that we can later look for it beyond the toy regime\.

#### What is a signature?

A*signature*is the concrete way an idea would*show up in a complex/modern model*\. It translates the idea’s commitments into measurable phenomena—geometry, dynamics, causal responses, invariances, thresholds, or characteristic error patterns—that should be detectable \(perhaps approximately or in restricted regimes\) when we move from the simplified setting to realistic architectures, scales, and data\. A good signature specifies*where*and*how*to look \(layers, training phases, sweeps\) and*what trend or boundary*to expect\.

#### What is a tailored experiment?

A*tailored experiment*is an empirical test in a complex model that is*designed to observe the signature*\. The idea determines the readouts, perturbations, controls, and the region of model/data space where detectability is highest\. Success is the clear visibility \(or principled absence\) of the predicted pattern, not a leaderboard delta\. Concretely: \(i\) define the signature as a statistic/visual you can resolve; \(ii\) choose instruments that expose it \(measurements, ablations/repairs, counterfactuals\); \(iii\) place the measurement where the idea predicts signal \(layers/training windows/scale\); \(iv\) sweep the knob the idea says should modulate the signature and include*negative controls*the idea says should not; \(v\) report the qualitative trend, thresholds, and failures that refine scope\.

Modern systems are noisy and heterogeneous; pointwise perfection is the wrong target\. We evaluate signatures*in expectation*or as*coarse*patterns \(e\.g\., “usually increasing”\), and we allow bounded exceptions\. What matters is that the predicted*shape*is visible with reasonable aggregation or replication, not that every unit or datapoint obeys it\.

#### Why this helps\.

Our position directly combats*benchmark myopia*by shifting the basis of evaluation from marginal score improvements to the explicit detection or falsification of predicted signatures\. By requiring that abstract ideas manifest as observable behaviors in finite models, we bridge the*transfer gap*, translating idealized theory into concrete measurements that experiments can use to refine or reject hypotheses\. Furthermore, anchoring ablations and robustness checks in specific predictions addresses the problem of*non\-cumulative findings*, transforming scattered exploratory results into hypothesis\-driven tests that future work can reliably replicate\. Emphasizing sharp experiments that make a signature visible regardless of system size counters the*complexity premium*, establishing conceptual clarity and testable commitments as the primary currency of evidence\. Finally, by legitimizing modest, carefully targeted experiments as sufficient to substantiate an idea, the framework mitigates*resource asymmetry*, empowering low\-compute groups to participate substantively in shaping the field\.

## 4Illustrative Examples

Here we present a list of three well\-known works where our approach is present:

### 4\.1Linearized training \(NTK\)

In the infinite\-width limit, training a network is equivalent to kernel gradient flow with a*fixed*Neural Tangent Kernel, yielding a linearized, feature\-frozen dynamics\(Jacotet al\.,[2018](https://arxiv.org/html/2605.15253#bib.bib4)\)\. This gives a crisp bridge from an idealized regime to modern practice: if the idea carries over at scale, early training of big models should visibly track their NTK linearizations before feature learning takes over\.

#### Idea:

In simplified settings \(wide limits; squared loss; isotropic inputs\), gradient descent follows kernel regression under the NTK fixed at initialization; predictions evolve linearly around the start point\(Jacotet al\.,[2018](https://arxiv.org/html/2605.15253#bib.bib4)\)\. Signature:In complex models, during early epochs the network’s predictions and loss trajectory closely match those of its NTK\-at\-init linearization; agreement improves with width and fades as features move\. Tailored experiment:Instantiate the linearized predictor and compare it to full training across widths/epochs, reading out prediction/loss alignment and its breakdown\.Leeet al\.\([2019](https://arxiv.org/html/2605.15253#bib.bib17)\)do exactly this for CNN/ResNet/WRN on CIFAR, directly exposing the early\-time linearization signature\.

### 4\.2Implicit bias to max\-margin

Gradient descent on separable problems implicitly prefers maximum\-margin solutions: a phenomenon proved for linear models with exponential\-tail losses and extended to homogeneous deep nets\(Soudryet al\.,[2018](https://arxiv.org/html/2605.15253#bib.bib18); Lyu and Li,[2020](https://arxiv.org/html/2605.15253#bib.bib19)\)\. The simplified analysis predicts a concrete directional trend that should leave traces in modern representations\.

#### Idea:

For linearly separable data, GD on logistic/exponential losses drives the weight direction toward the hard\-margin SVM solution; analogously holds for homogeneous networks\(Soudryet al\.,[2018](https://arxiv.org/html/2605.15253#bib.bib18); Lyu and Li,[2020](https://arxiv.org/html/2605.15253#bib.bib19)\)\. Signature:In complex models, once training error hits zero, the*normalized margin*in penultimate\-layer features continues to increase and the classifier direction aligns with a max\-margin solution\. Tailored experiment:Track normalized margins and alignment\-to\-SVM over epochs after interpolation\.Lyu and Li \([2020](https://arxiv.org/html/2605.15253#bib.bib19)\)report monotone \(smoothed\) margin growth and alignment trends on MNIST/CIFAR with MLPs/CNNs, making the max\-margin signature visible\.

### 4\.3Mixup \(heuristic\)

Training on convex combinations of inputs and labels encourages approximately linear behavior between examples\(Zhanget al\.,[2018](https://arxiv.org/html/2605.15253#bib.bib126)\)\. The heuristic is easy to see in toy 2D settings where decision boundaries straighten along line segments and scales to large vision/speech models, suggesting a clear signature to seek in modern networks\.

#### Idea:

In simplified settings, decision regions straighten along line segments between examples and logits vary smoothly with the mixing coefficient\(Zhanget al\.,[2018](https://arxiv.org/html/2605.15253#bib.bib126)\)\. Signature:In complex models, along interpolation pathsxλ=λ​xi\+\(1−λ\)​xjx\_\{\\lambda\}=\\lambda x\_\{i\}\+\(1\-\\lambda\)x\_\{j\}, predicted logits for the mixed label vary roughly linearly inλ\\lambda; memorization under label noise is reduced and gradients are smoother between samples\. Tailored experiment:Probe logits vs\. mixing coefficient on real datasets/models and stress\-test with corrupted labels; report interpolation\-linearity and reduced memorization alongside standard accuracy\(Zhanget al\.,[2018](https://arxiv.org/html/2605.15253#bib.bib126)\)\.

## 5Call to Action: A Field Guide

Now we provide a brief field guide for authors and reviewers that translates our proposal into concrete suggestions for writing and evaluating idea centric work\.

### 5\.1For authors

#### Specifying the idea\.

Aim to\(i\) state a one or two sentence claim, whether it concerns a concrete method, improvement or a stylized theoretical result; \(ii\) make the scope explicit in terms of architectures, data regimes, training setups, or the class of models and assumptions considered in the theory; and \(iii\) mention at least one plausible failure mode or limitation, for example regimes where you do not expect the claim to apply\.Avoid\(i\) presenting only a slogan or heuristic with no precise claim; and \(ii\) letting the idea be defined only via a specific experiment, benchmark gain, or theorem statement without intuitive interpretation and scope\.

#### Defining signatures\.

Aim to\(i\) introduce a small set of signatures that you reuse throughout the paper; \(ii\) for each signature, specify the measured or derived quantity, how it is computed, and the expected pattern; and \(iii\) state regimes where the signature should appear and where it should weaken or disappear\. Signatures can be empirical measurements, such as margin distributions or error patterns, or theoretical predictions, such as scaling laws, shapes of learning curves, or stability properties that could be tested\.Avoid\(i\) using generic predictions such as “better generalization” without concrete measurements or model level predictions and \(ii\) introducing many loosely related metrics, lemmas, or bounds whose relation to the idea is unclear\.

#### Designing tailored experiments\.

Aim to\(i\) make the primary outcome the presence, absence, or strength of your signatures, using empirical measurements or simulations; \(ii\) use the simplest models and datasets that remain faithful to the stated scope of the idea while plausibly exhibiting the same qualitative patterns as more complex state of the art systems; and \(iii\) include at least one stress test or boundary case where the idea predicts change, weakening, or failure, for example by relaxing a key assumption or moving outside the intended regime\.Avoid\(i\) treating benchmark gains or theorem strength as a substitute for testing signatures and \(ii\) scaling models and data sizes, or adding layers of technical assumptions, without a corresponding change in what the idea actually claims about mechanisms or behavior\.

### 5\.2For reviewers

#### Evaluating the idea\.

Aim to\(i\) judge the clarity, mechanism, and scope of the claim independently of raw performance numbers or formal sophistication; \(ii\) check that the idea is specific enough that data, simulations, or counterexamples could contradict it; and \(iii\) reward connections to and tensions with prior empirical and theoretical work, not only incremental gains\.Avoid\(i\) rejecting mainly because the paper does not present new state of the art benchmarks or fully general theorems and \(ii\) demanding large scale experiments when the main contribution is theoretical and/or conceptual, or demanding complete formalization when the stated contribution is mostly empirical\.

#### Evaluating signatures\.

Aim to\(i\) check that signatures are concrete, measurable, or otherwise operationalizable, and clearly derived from the idea or model; \(ii\) look for explicit regimes where signatures should appear and for proposed negative controls or assumptions under which they should fail; and \(iii\) encourage signatures that other groups could realistically test or extend, whether by running new experiments or by sharpening or generalizing the theory\.Avoid\(i\) asking primarily for more benchmarks or more theorems when the missing piece is clearer signatures and \(ii\) accepting vague or post hoc patterns, or technically involved results with no interpretable signature, as sufficient evidence for the idea\.

#### Evaluating tailored experiments\.

Aim to\(i\) evaluate how well the experiments and simulations align with the stated signatures and scope; \(ii\) prioritize the informativeness and design of these probes over sheer scale or technical difficulty; and \(iii\) value reporting of failures, anomalies, and boundary cases that refine the idea or expose limits of the formal model\.Avoid\(i\) automatically requesting larger models or datasets when the claim is not about that regime, or stronger and more general theorems when the idea is already well supported by partial results and empirical signatures, and \(ii\) treating small benchmark deltas or theorem strength alone as the main success criterion for an idea centric paper\.

## 6Hypothetical case study: Investigating topic inertia in LLMs

To illustrate the proposed framework in practice, we present ahypothetical case study, inspired byJia and Diaz\-Rodriguez \([2025](https://arxiv.org/html/2605.15253#bib.bib110)\), investigating how large language models \(LLMs\) maintain topic consistency as context grows\.

The idea\.Since the main components of LLMs are attention layers, we begin with a hypothesis derived from a simplified theoretical setting: a single\-layer self\-attention model\(Vaswaniet al\.,[2017](https://arxiv.org/html/2605.15253#bib.bib111)\)with a unified key\-query matrix\. Theoretical analysis in this idealized regime suggests that as the input sequence length increases, the attention mechanism becomes increasingly biased toward the dominant semantic cluster of the input\. We generalize this concept as “Topic Inertia”: the probability of an LLM adhering to the topic of the input sequence increases as the input length grows\.*Scope:*This claim is grounded in the mechanics of standard self\-attention and is not expected to extend to non\-attention architectures \(e\.g\., RNNs\) or complex agentic workflows involving multi\-step reasoning\.

The signature\.We translate this idea into a concrete, observable pattern for modern, deep LLMs\.*Predicted Signature:*The semantic similarity \(measured via cosine similarity of embeddings\) between the input prompt and the generated text should exhibit a generally increasing trend as the number of tokens in the prompt increases\. A flat or decreasing trend would falsify the hypothesis\.

The tailored experiment\.To detect this signature, we design a targeted experiment\. \(i\) We select a curated set of 200 documents from a private corpus unknown to public models to rule out memorization effects\. \(ii\) We sample text segments of varying lengths \(from 10 to 200 tokens\) from these documents and we use them as prompts to query multiple open\-weights models: LLaMA 2\(Touvronet al\.,[2023](https://arxiv.org/html/2605.15253#bib.bib112)\), GPT\-NeoX\-20B\(Blacket al\.,[2022](https://arxiv.org/html/2605.15253#bib.bib113)\), MPT\-7B\(MosaicML,[2023](https://arxiv.org/html/2605.15253#bib.bib114)\)\. As a negative control, we also generate with a simple RNN baseline \(which lacks the attention mechanism\)\. \(iii\) We compute the cosine similarity between SBERT embeddings\(Reimers and Gurevych,[2019](https://arxiv.org/html/2605.15253#bib.bib117)\)of the prompt and the generated text for varying prompt lengths\.

Hypothetical results\.In Figure[2](https://arxiv.org/html/2605.15253#S6.F2)we observe all tested LLMs display the predicted average upward trend in similarity as context length increases, while it is absent in the RNN baseline\. We conclude that the signature is visible, supporting the Topic Inertia hypothesis within the defined scope\. We explicitly note that these results apply to fixed\-size inputs and do not make claims regarding topic drift during long\-form generation \(increasing output length\)\.

#### Hypothetical Contributions\.

The hypothetical contributions of this work are: \(i\) the formalization of the*idea*of “Topic Inertia,” demonstrated theoretically in simplified self\-attention models; and \(ii\) the empirical observation of its predicted*signature*in complex LLMs through*tailored experiments*\. This demonstrates that the identified mechanism extends beyond theoretical results to accurately describe the behavior of complex Large Language Models\.

![Refer to caption](https://arxiv.org/html/2605.15253v1/x2.png)Figure 2:The framework on a hypothetical example\. The*Idea*\(Topic Inertia\) predicts a specific*Signature*: an upward trend in embedding similarity as prompt increases\. The*Tailored Experiment*confirms this signature in Transformer models, while the negative control \(RNN\) shows no such trend, validating the mechanism\.
### 6\.1Hypothetical Criticisms and the Defense of Scope

If this study were submitted under the current evaluation regime, it would likely face rejection for failing to meet the implicit requirements of benchmark\-driven engineering or idealized theory\. By anticipating these criticisms, we illustrate how our framework protects scientific value from being discarded due to structural biases\.

The critique of scale and SOTA \(Mode A\)\.A reviewer might object:*“Why stop at 200 tokens? Why evaluate on older open\-weights models instead of frontier better systems like GPT\-5\.1\(OpenAI,[2025](https://arxiv.org/html/2605.15253#bib.bib115)\)or Gemini 3\(Google DeepMind,[2025](https://arxiv.org/html/2605.15253#bib.bib116)\)? Does this insight improve perplexity on the LongBench\(Baiet al\.,[2024](https://arxiv.org/html/2605.15253#bib.bib118)\)leaderboard?”*Under our framework, these demands enforce aresource thresholdrather than ensuring scientific rigor\. If the signature of “Topic Inertia” is clearly resolvable at 200 tokens, scaling the experiment requires massive compute without adding mechanistic insight\. Furthermore, demanding evaluations on closed, proprietary APIs introduces reproducibility issues and financial barriers\. A tailored experiment maximizes the ratio of insight to compute; demanding scale for scale’s sake excludes researchers in low\-resource settings and perpetuates “Red AI” without refining the idea\.

The critique of exactness and ablation \(Mode B\)\.A reviewer might also ask:*Why does the trend fluctuate and dip rather than strictly increasing as the theorem suggests? Why did you only test Cosine Similarity and not sweep across all possible embedding metrics?”*This reflects a confusion between asignatureand a mathematical law\. Real\-world data is noisy; the scientific question is whether the mechanism is visible*in expectation*\. The “dips” in the graph represent natural variance, not a falsification of the trend\. Similarly, demanding exhaustive ablations across every possible metric is a form of thecomplexity premium\. Since the tailored experiment included a negative control that ruled out generic artifacts, additional metric sweeps serve only to increase the authors’ workload, not the validity of the claim\.

The “Ideas First” standard\.Ultimately, an “Ideas First” culture changes the burden of proof\. It allows authors to state:*“The idea is clearly visible via the predicted signature; optimization and scaling are future work\.”*If the mechanism is isolated and the signature is robustly detected against controls, the paper has succeeded\. The community must learn to accept clear, bounded interesting insights without penalizing them for not being total, SOTA systems\.

## 7Discussion

Our central premise is that the primary unit of scientific value in ML should be the*idea*, rather than a marginal leaderboard gain or an intricate theorem\. Currently, the field often inverts this priority, judging work by benchmark numbers or formal guarantees in idealized settings\. We argue that this regime is epistemically fragile and structurally biased\. An alternative culture centered on ideas and tailored experiments is both scientifically grounded and ethically preferable\.

### 7\.1Benchmark Centrism and Its Limits

The dominance of benchmarks encourages the use of complex notation to obfuscate rather than clarify, and a fixation on incremental SOTA results, often obscuring*what is actually being learned*\(Lipton and Steinhardt,[2018](https://arxiv.org/html/2605.15253#bib.bib1)\)\.Koch and Peterson \([2024](https://arxiv.org/html/2605.15253#bib.bib36)\)describe benchmarking as an “epistemic monoculture” that privileges specific tasks while marginalizing alternative goals\.

Empirically, benchmark conclusions are surprisingly fragile\. Model superiority often depends more on the specific test set than the method’s intrinsic properties \(the “Benchmark Lottery”\)\(Dehghaniet al\.,[2021](https://arxiv.org/html/2605.15253#bib.bib37)\), and small perturbations or variance in seeds can swamp reported gains\(Bouthillieret al\.,[2021](https://arxiv.org/html/2605.15253#bib.bib38); Alzahraniet al\.,[2024](https://arxiv.org/html/2605.15253#bib.bib39)\)\. Furthermore, standard evaluation pipelines can yield misleading conclusions about transferability\(Singhet al\.,[2025a](https://arxiv.org/html/2605.15253#bib.bib96)\)and ignore critical costs like energy and fairness\(Ethayarajhet al\.,[2020](https://arxiv.org/html/2605.15253#bib.bib43)\)\.

Leaderboards are not neutral scoreboards; they are socio\-technical artifacts that can entrench power imbalances\(Erikssonet al\.,[2025](https://arxiv.org/html/2605.15253#bib.bib40)\)and encourage overfitting to the evaluation environment itself\(Singhet al\.,[2025b](https://arxiv.org/html/2605.15253#bib.bib42)\)\. Specialized domains face similar issues, where widely used datasets can systematically bias conclusions in drug discovery and scientific ML\(Cieplinskiet al\.,[2023](https://arxiv.org/html/2605.15253#bib.bib98); Thiyagalingamet al\.,[2022](https://arxiv.org/html/2605.15253#bib.bib49)\)\. While benchmarks remain indispensable infrastructure\(Denget al\.,[2009](https://arxiv.org/html/2605.15253#bib.bib45); Wanget al\.,[2018](https://arxiv.org/html/2605.15253#bib.bib46); Walters,[2023](https://arxiv.org/html/2605.15253#bib.bib99)\), treating marginal improvements as the primary currency of research is at odds with robust science\.

### 7\.2Idealized Theory and Partial Understanding

Theoretical work faces a parallel transfer gap \(see Figure[1](https://arxiv.org/html/2605.15253#S1.F1)\)\. Classical frameworks often fail to explain the generalization of deep networks that fit random noise\(Zhanget al\.,[2017](https://arxiv.org/html/2605.15253#bib.bib53)\)\. While recent science of deep learning efforts are valuable\(Baraniuket al\.,[2020](https://arxiv.org/html/2605.15253#bib.bib54); Drori and others,[2022](https://arxiv.org/html/2605.15253#bib.bib58)\), they frequently rely on stylized models and asymptotic regimes that depart from frontier systems\.

Philosophers of science emphasize that laws and theories are typically partial, holding only in circumscribed situations\(Cartwright,[1983](https://arxiv.org/html/2605.15253#bib.bib59); da Costa and French,[2003](https://arxiv.org/html/2605.15253#bib.bib60)\)\. Scientific understanding often emerges from the interplay of models and exploratory experimentation rather than universal derivation\(Giere,[2019](https://arxiv.org/html/2605.15253#bib.bib61); Frigg and Hartmann,[2025](https://arxiv.org/html/2605.15253#bib.bib62)\)\. As in physics, where theory must remain aligned to experimental feedback\(Ellis and Silk,[2014](https://arxiv.org/html/2605.15253#bib.bib63); Hossenfelder,[2018](https://arxiv.org/html/2605.15253#bib.bib64)\), mechanistic understanding in ML requires probing trained models directly\(Räz and Beisbart,[2024](https://arxiv.org/html/2605.15253#bib.bib101)\)\. We advocate for a practice where theory is treated as partial and idealized, and experiments possess enough autonomy to stabilize phenomena independent of a single theoretical framework\(Hacking,[1983](https://arxiv.org/html/2605.15253#bib.bib65)\)\.

### 7\.3Resource Inequality and the Bias of Current Practice

Demanding large\-scale experiments and extensive ablations as a default requirement for publication creates a resource threshold for legitimacy\. This exacerbates the “compute divide”, concentrating research power in a few well\-resourced labs\(Ahmed and Wahed,[2020](https://arxiv.org/html/2605.15253#bib.bib80)\)\. “Red AI” practices that chase accuracy at any cost worsen environmental impacts and research inequality\(Strubellet al\.,[2019](https://arxiv.org/html/2605.15253#bib.bib81); Schwartzet al\.,[2020](https://arxiv.org/html/2605.15253#bib.bib106); Xuet al\.,[2021](https://arxiv.org/html/2605.15253#bib.bib83)\)\.

While recommendations for extensive reproducibility checks are well\-intentioned\(Gundersenet al\.,[2022](https://arxiv.org/html/2605.15253#bib.bib119); Nature Editorial,[2021](https://arxiv.org/html/2605.15253#bib.bib89)\), they structurally disadvantage researchers in the Global South and teaching\-focused institutions\(Mohamedet al\.,[2020](https://arxiv.org/html/2605.15253#bib.bib91); Chanet al\.,[2021](https://arxiv.org/html/2605.15253#bib.bib120)\)\. Current norms effectively assert that a contribution is only valuable if validated in a regime dominated by a small set of actors\(Wuet al\.,[2022](https://arxiv.org/html/2605.15253#bib.bib85)\)\. Positioning idea\-centric research as a methodology for “Frugal AI” where systems achieve high impact with minimal resources is therefore critical\(UNESCO,[2025](https://arxiv.org/html/2605.15253#bib.bib94)\)\. By validating “modest” experiments that clearly isolate a mechanism, the field can broaden participation and mitigate the colonial power relations currently reproduced by AI development\(Ayanaet al\.,[2024](https://arxiv.org/html/2605.15253#bib.bib92)\)\.

### 7\.4Simplicity and Cumulative Science

Scientific merit should not be conflated with model complexity\. Simple algorithms and conceptual proposals are often dismissed as preliminary if they lack large\-scale demonstrations, yet simplicity often increases evidential value by isolating the mechanism of interest\.

History suggests that major scientific advances emerge from chains of small, incremental contributions\(Merton,[1973](https://arxiv.org/html/2605.15253#bib.bib121); Kuhn,[1962](https://arxiv.org/html/2605.15253#bib.bib122)\)\. A culture that filters out implementable ideas because they lack heavy engineering risks damaging the field’s future\. Protecting the ecology of simple ideas allows for a cumulative process where insights can be tested and scaled by others\. A reviewing culture that values idea\-centric work would reduce the “fixed cost” of participation and prevent the loss of valuable concepts at the submission stage\.

### 7\.5Balancing Rigor, Benchmarks, and Idea\-Centric Evaluation

We do not argue against theoretical rigor or benchmarks, but for diversifying what counts as evidence\. Benchmarks should be viewed as evolving socio\-technical artifacts\(Hardt,[2025](https://arxiv.org/html/2605.15253#bib.bib100); Erikssonet al\.,[2025](https://arxiv.org/html/2605.15253#bib.bib40)\)rather than static scoreboards\. Theory should be acknowledged as partial and guided by exploratory experiment\(Steinle,[2002](https://arxiv.org/html/2605.15253#bib.bib67)\)\.

Ultimately, methodological choices are inseparable from questions of justice\(Buolamwini and Gebru,[2018](https://arxiv.org/html/2605.15253#bib.bib73); Noble,[2018](https://arxiv.org/html/2605.15253#bib.bib75)\)\. By valuing strong ideas supported by appropriately scaled experiments and negative controls, we can align ML research with the history of successful science while fostering a more equitable and robust community\.

## 8Alternative Views

Our proposal sits within a broader landscape of reasonable disagreement\. Many researchers accept that benchmarks, large\-scale ablations and strict reproducibility standards are imperfect yet still regard them and their leaderboards as indispensable coordination tools\. Similarly, idealized theory and the emerging science of deep learning are seen as the primary route to general principles, with transfer to contemporary systems treated as a long\-term goal\. We disagree to the extent that this stance treats benchmark gains and idealized theory as self\-justifying, rather than requiring a clear link to concrete ideas and model behavior\.

A second concern is that simple models and tailored experiments might be less trustworthy than benchmark evaluations, since targeted tests can be unconsciously tuned to confirm a preferred story\. From this perspective, strong benchmark improvements or theorems in clean settings are valued because they are harder to manipulate and easier to evaluate consistently\. We argue instead that explicit behavioral signatures, and negative controls can make targeted experiments at least as rigorous and reproducible as broad benchmarks\.

An alternative view on the role of equity in methodological choice is that resource inequality and the marginalization of small labs are serious social problems but should not shape epistemic standards\. If frontier applications require massive compute and complex systems, it is then acceptable that only a few institutions can fully participate, and equity should be addressed through funding and policy rather than by redefining good evidence\. We disagree because methodological standards and distributive justice are intertwined, and norms that systematically exclude many researchers are both epistemically and ethically costly\.

Another alternative view is that some theoretical research in machine learning should be valued in its own right, even when it is not primarily aimed at explaining the behavior of contemporary models\. Such work may advance the mathematics of machine learning or contribute to adjacent fields such as statistics and theoretical computer science\. We view this as an important and legitimate agenda\. Our aim here is narrower: to argue for a complementary program in the space between idealized theory and benchmark\-driven experimentation, where ideas are evaluated through observable signatures in modern systems\.

Finally, some see machine learning as closer to engineering than to basic science, where value is measured primarily by deployed systems, competitive performance, and real\-world impact\. Within this paradigm, theory, benchmarks, and large\-scale experimentation are evaluated mainly by their contribution to products and capabilities, and a program that foregrounds ideas and signatures may seem secondary\. We disagree because even in engineering\-dominated fields, durable progress depends on understanding mechanisms rather than relying only on short\-term performance gains\.

### Conclusion

We have argued that ML research is currently constrained by a dual fixation on leaderboard dominance and idealized theory, which excludes low\-resource researchers, particularly those in the Global South\. By adopting the Ideas First framework, the community can shift its focus to understanding behavior\. This approach restores the cumulative nature of scientific discovery by validating clear, scope\-limited hypotheses through accessible experiments\. Ultimately, prioritizing mechanistic clarity over SOTA metrics is not just a methodological correction; it is an ethical imperative that democratizes participation and fosters a more sustainable, scientifically rigorous future for AI\.

## Acknowledgments

This work was supported by the Natural Sciences and Engineering Research Council of Canada \(NSERC\) under grant DGECR\-2022\-04531\. The author thanks the anonymous reviewers, and the session chair for their valuable feedback and insightful suggestions, which greatly improved the quality of this work\.

## References

- N\. Ahmed and M\. Wahed \(2020\)The de\-democratization of AI: deep learning and the compute divide in artificial intelligence research\.arXiv preprint arXiv:2010\.15581\.Cited by:[§1](https://arxiv.org/html/2605.15253#S1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2605.15253#S1.p1.1),[§7\.3](https://arxiv.org/html/2605.15253#S7.SS3.p1.1)\.
- N\. Alzahrani, H\. Alyahya, Y\. Alnumay, S\. Alrashed, S\. Alsubaie, Y\. Almushayqih, F\. Mirza, N\. Alotaibi, N\. Al\-Twairesh, A\. Alowisheq,et al\.\(2024\)When benchmarks are targets: revealing the sensitivity of large language model leaderboards\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13787–13805\.Cited by:[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p2.1)\.
- G\. Ayana, K\. Dese, H\. Daba Nemomssa, B\. Habtamu, B\. Mellado, K\. Badu, E\. Yamba, S\. L\. Faye, M\. Ondua, D\. Nsagha,et al\.\(2024\)Decolonizing global ai governance: assessment of the state of decolonized ai governance in sub\-saharan africa\.Royal Society Open Science11\(8\)\.Cited by:[§7\.3](https://arxiv.org/html/2605.15253#S7.SS3.p2.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 3119–3137\.External Links:[Link](https://aclanthology.org/2024.acl-long.172/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172)Cited by:[§6\.1](https://arxiv.org/html/2605.15253#S6.SS1.p2.1.2)\.
- R\. Baraniuk, D\. Donoho, and M\. Gavish \(2020\)The science of deep learning\.Proceedings of the National Academy of Sciences117\(48\),pp\. 30029–30032\.Cited by:[§1](https://arxiv.org/html/2605.15253#S1.p1.1),[§7\.2](https://arxiv.org/html/2605.15253#S7.SS2.p1.1)\.
- P\. L\. Bartlett, D\. J\. Foster, and M\. J\. Telgarsky \(2017\)Spectrally\-normalized margin bounds for neural networks\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2](https://arxiv.org/html/2605.15253#S2.SSx2.SSS0.Px1.p1.1)\.
- P\. L\. Bartlett, P\. M\. Long, G\. Lugosi, and A\. Tsigler \(2020\)Benign overfitting in linear regression\.Proceedings of the National Academy of Sciences117\(48\),pp\. 30063–30070\.Cited by:[§2](https://arxiv.org/html/2605.15253#S2.SSx2.SSS0.Px1.p1.1)\.
- S\. Black, S\. Biderman, E\. Hallahan, Q\. Anthony, L\. Gao, L\. Golding, H\. He, C\. Leahy, K\. McDonell, J\. Phang, M\. Pieler, U\. S\. Prashanth, S\. Purohit, L\. Reynolds, J\. Tow, B\. Wang, and S\. Weinbach \(2022\)GPT\-neox\-20b: an open\-source autoregressive language model\.InProceedings of BigScience Episode \#5 – Workshop on Challenges & Perspectives in Creating Large Language Models,virtual\+Dublin,pp\. 95–136\.External Links:[Link](https://aclanthology.org/2022.bigscience-1.9/),[Document](https://dx.doi.org/10.18653/v1/2022.bigscience-1.9)Cited by:[§6](https://arxiv.org/html/2605.15253#S6.p4.1)\.
- X\. Bouthillier, P\. Delaunay, M\. Bronzi, A\. Trofimov, B\. Nichyporuk, J\. Szeto, N\. Mohammadi Sepahvand, E\. Raff, K\. Madan, V\. Voleti,et al\.\(2021\)Accounting for variance in machine learning benchmarks\.Proceedings of Machine Learning and Systems3,pp\. 747–769\.Cited by:[§1](https://arxiv.org/html/2605.15253#S1.SS0.SSS0.Px2.p1.1),[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p2.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§2](https://arxiv.org/html/2605.15253#S2.SSx1.SSS0.Px1.p1.1)\.
- J\. Buolamwini and T\. Gebru \(2018\)Gender shades: intersectional accuracy disparities in commercial gender classification\.InProceedings of the 1st Conference on Fairness, Accountability and Transparency \(FAT\*\),Proceedings of Machine Learning Research, Vol\.81,pp\. 77–91\.Cited by:[§7\.5](https://arxiv.org/html/2605.15253#S7.SS5.p2.1)\.
- N\. Cartwright \(1983\)How the laws of physics lie\.Oxford University Press\.Cited by:[§7\.2](https://arxiv.org/html/2605.15253#S7.SS2.p2.1)\.
- A\. Chan, C\. T\. Okolo, Z\. Terner, and A\. Wang \(2021\)The limits of global inclusion in ai development\.arXiv preprint arXiv:2102\.01265\.Cited by:[§7\.3](https://arxiv.org/html/2605.15253#S7.SS3.p2.1)\.
- T\. Cieplinski, T\. Danel, S\. Podlewska, and S\. Jastrzebski \(2023\)Generative models should at least be able to design molecules that dock well: a new benchmark\.Journal of Chemical Information and Modeling63\(11\),pp\. 3238–3247\.Cited by:[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p3.1)\.
- N\. C\. A\. da Costa and S\. French \(2003\)Science and partial truth: a unitary approach to models and scientific reasoning\.Oxford University Press\.Cited by:[§7\.2](https://arxiv.org/html/2605.15253#S7.SS2.p2.1)\.
- M\. Dehghani, Y\. Tay, F\. Diaz, and D\. Metzler \(2021\)The benchmark lottery\.InProceedings of the 9th International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2605.15253#S1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2605.15253#S1.p1.1),[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p2.1)\.
- J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-Fei \(2009\)ImageNet: a large\-scale hierarchical image database\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 248–255\.Cited by:[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p3.1)\.
- I\. Droriet al\.\(2022\)A neural network solves, explains, and generates university math problems by program synthesis and few\-shot learning at human level\.InProceedings of the National Academy of Sciences,Note:Representative of work framing deep learning as scientific discoveryCited by:[§7\.2](https://arxiv.org/html/2605.15253#S7.SS2.p1.1)\.
- G\. Ellis and J\. Silk \(2014\)Scientific method: defend the integrity of physics\.Nature516\(7531\),pp\. 321–323\.Cited by:[§7\.2](https://arxiv.org/html/2605.15253#S7.SS2.p2.1)\.
- M\. Eriksson, E\. Purificato, A\. Noroozian, J\. Vinagre, G\. Chaslot, E\. Gomez, and D\. Fernandez\-Llorca \(2025\)Can we trust ai benchmarks? an interdisciplinary review of current issues in ai evaluation\.InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society \(AIES\),Cited by:[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p3.1),[§7\.5](https://arxiv.org/html/2605.15253#S7.SS5.p1.1)\.
- K\. Ethayarajh, S\. Swayamdipta, and Y\. Choi \(2020\)Utility is in the eye of the user: a critique of NLP leaderboards\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4846–4856\.Cited by:[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p2.1)\.
- R\. Frigg and S\. Hartmann \(2025\)Models in science\.InThe Stanford Encyclopedia of Philosophy,E\. N\. Zalta and U\. Nodelman \(Eds\.\),Cited by:[§7\.2](https://arxiv.org/html/2605.15253#S7.SS2.p2.1)\.
- R\. N\. Giere \(2019\)Scientific perspectivism\.University of Chicago press\.Cited by:[§7\.2](https://arxiv.org/html/2605.15253#S7.SS2.p2.1)\.
- Google DeepMind \(2025\)Gemini 3 models\.Note:[https://deepmind\.google/models/gemini/](https://deepmind.google/models/gemini/)Google DeepMind model documentationCited by:[§6\.1](https://arxiv.org/html/2605.15253#S6.SS1.p2.1.2)\.
- O\. E\. Gundersen, K\. Coakley, C\. R\. Kirkpatrick, and Y\. Gil \(2022\)Sources of irreproducibility in machine learning: a review\.arXiv preprint arXiv:2204\.07610\.Cited by:[§7\.3](https://arxiv.org/html/2605.15253#S7.SS3.p2.1)\.
- I\. Hacking \(1983\)Representing and intervening: introductory topics in the philosophy of natural science\.Cambridge University Press\.Cited by:[§7\.2](https://arxiv.org/html/2605.15253#S7.SS2.p2.1)\.
- M\. Hardt \(2025\)The emerging science of machine learning benchmarks\.Note:Online at[https://mlbenchmarks\.org](https://mlbenchmarks.org/)ManuscriptCited by:[§7\.5](https://arxiv.org/html/2605.15253#S7.SS5.p1.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 770–778\.Cited by:[§2](https://arxiv.org/html/2605.15253#S2.SSx1.SSS0.Px1.p1.1)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.\(2022\)Training compute\-optimal large language models\.InProceedings of the 36th International Conference on Neural Information Processing Systems,pp\. 30016–30030\.Cited by:[§2](https://arxiv.org/html/2605.15253#S2.SSx1.SSS0.Px1.p1.1)\.
- S\. Hossenfelder \(2018\)Lost in math: how beauty leads physics astray\.Basic Books\.Cited by:[§7\.2](https://arxiv.org/html/2605.15253#S7.SS2.p2.1)\.
- A\. Jacot, F\. Gabriel, and C\. Hongler \(2018\)Neural tangent kernel: convergence and generalization in neural networks\.Advances in neural information processing systems\.Cited by:[§4\.1](https://arxiv.org/html/2605.15253#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2605.15253#S4.SS1.p1.1)\.
- M\. Jia and J\. Diaz\-Rodriguez \(2025\)Dynamics of spontaneous topic changes in next token prediction with self\-attention\.Advances in neural information processing systems39\.Cited by:[§6](https://arxiv.org/html/2605.15253#S6.p1.1)\.
- B\. J\. Koch and D\. Peterson \(2024\)From protoscience to epistemic monoculture: how benchmarking set the stage for the deep learning revolution\.arXiv preprint arXiv:2404\.06647\.Cited by:[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p1.1)\.
- A\. Krizhevsky, I\. Sutskever, and G\. E\. Hinton \(2012\)ImageNet classification with deep convolutional neural networks\.InAdvances in Neural Information Processing Systems,Vol\.25,pp\. 1097–1105\.Cited by:[§2](https://arxiv.org/html/2605.15253#S2.SSx1.SSS0.Px1.p1.1)\.
- T\. S\. Kuhn \(1962\)The structure of scientific revolutions\.University of Chicago Press\.Cited by:[§1](https://arxiv.org/html/2605.15253#S1.SS0.SSS0.Px2.p1.1),[§7\.4](https://arxiv.org/html/2605.15253#S7.SS4.p2.1)\.
- J\. Lee, L\. Xiao, S\. S\. Schoenholz, Y\. Bahri, R\. Novak, J\. Sohl\-Dickstein, and J\. Pennington \(2019\)Wide neural networks of any depth evolve as linear models under gradient descent\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§4\.1](https://arxiv.org/html/2605.15253#S4.SS1.SSS0.Px1.p1.1)\.
- Z\. C\. Lipton and J\. Steinhardt \(2018\)Troubling trends in machine learning scholarship\.arXiv:1807\.03341\.Cited by:[§1](https://arxiv.org/html/2605.15253#S1.p1.1),[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p1.1)\.
- K\. Lyu and J\. Li \(2020\)Gradient descent maximizes the margin of homogeneous neural networks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4\.2](https://arxiv.org/html/2605.15253#S4.SS2.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2605.15253#S4.SS2.p1.1)\.
- R\. K\. Merton \(1973\)The sociology of science: theoretical and empirical investigations\.University of Chicago Press\.Cited by:[§1](https://arxiv.org/html/2605.15253#S1.SS0.SSS0.Px2.p1.1),[§7\.4](https://arxiv.org/html/2605.15253#S7.SS4.p2.1)\.
- S\. Mohamed, M\. Png, and W\. Isaac \(2020\)Decolonial AI: decolonial theory as sociotechnical foresight in artificial intelligence\.Philosophy & Technology33\(4\),pp\. 659–684\.Cited by:[§7\.3](https://arxiv.org/html/2605.15253#S7.SS3.p2.1)\.
- MosaicML \(2023\)Introducing MPT\-7b: a new standard for open\-source, commercially usable LLMs\.Note:[https://www\.databricks\.com/blog/mpt\-7b](https://www.databricks.com/blog/mpt-7b)Databricks blogCited by:[§6](https://arxiv.org/html/2605.15253#S6.p4.1)\.
- Nature Editorial \(2021\)Moving towards reproducible machine learning\.Nature Computational Science1\(10\),pp\. 629–630\.Cited by:[§7\.3](https://arxiv.org/html/2605.15253#S7.SS3.p2.1)\.
- B\. Neyshabur, R\. Tomioka, and N\. Srebro \(2015\)Norm\-based capacity control in neural networks\.InProceedings of the 28th Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.40,pp\. 1376–1401\.Cited by:[§2](https://arxiv.org/html/2605.15253#S2.SSx2.SSS0.Px1.p1.1)\.
- S\. U\. Noble \(2018\)Algorithms of oppression: how search engines reinforce racism\.NYU Press\.Cited by:[§7\.5](https://arxiv.org/html/2605.15253#S7.SS5.p2.1)\.
- OpenAI \(2025\)GPT\-5\.1 models\.Note:[https://platform\.openai\.com/docs/models](https://platform.openai.com/docs/models)OpenAI model documentationCited by:[§6\.1](https://arxiv.org/html/2605.15253#S6.SS1.p2.1.2)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever \(2021\)Learning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 8748–8763\.Cited by:[§2](https://arxiv.org/html/2605.15253#S2.SSx1.SSS0.Px1.p1.1)\.
- T\. Räz and C\. Beisbart \(2024\)The importance of understanding deep learning\.Erkenntnis89\(5\),pp\. 1823–1840\.External Links:[Document](https://dx.doi.org/10.1007/s10670-022-00605-y)Cited by:[§7\.2](https://arxiv.org/html/2605.15253#S7.SS2.p2.1)\.
- N\. Reimers and I\. Gurevych \(2019\)Sentence\-BERT: sentence embeddings using Siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),Hong Kong, China,pp\. 3982–3992\.External Links:[Link](https://aclanthology.org/D19-1410/),[Document](https://dx.doi.org/10.18653/v1/D19-1410)Cited by:[§6](https://arxiv.org/html/2605.15253#S6.p4.1)\.
- A\. M\. Saxe, J\. L\. McClelland, and S\. Ganguli \(2014\)Exact solutions to the nonlinear dynamics of learning in deep linear neural networks\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.15253#S2.SSx2.SSS0.Px1.p1.1)\.
- R\. Schwartz, J\. Dodge, N\. A\. Smith, and O\. Etzioni \(2020\)Green AI\.Communications of the ACM63\(12\),pp\. 54–63\.External Links:[Document](https://dx.doi.org/10.1145/3381831)Cited by:[§1](https://arxiv.org/html/2605.15253#S1.SS0.SSS0.Px2.p1.1),[§7\.3](https://arxiv.org/html/2605.15253#S7.SS3.p1.1)\.
- P\. Singh, S\. Hess, and J\. Vanschoren \(2025a\)How not to benchmark your site metric: beyond static leaderboards and towards realistic evaluation\.arXiv preprint arXiv:2510\.06448\.Cited by:[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p2.1)\.
- S\. Singh, Y\. Nan, A\. Wang, D\. D’Souza, S\. Kapoor, A\. Üstün, S\. Koyejo, Y\. Deng, S\. Longpre, N\. Smith, B\. Ermis, M\. Fadaee, and S\. Hooker \(2025b\)The leaderboard illusion\.Advances in neural information processing systems39\.Cited by:[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p3.1)\.
- D\. Soudry, E\. Hoffer, M\. S\. Nacson, S\. Gunasekar, and N\. Srebro \(2018\)The implicit bias of gradient descent on separable data\.Journal of Machine Learning Research19\(70\),pp\. 1–57\.Cited by:[§4\.2](https://arxiv.org/html/2605.15253#S4.SS2.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2605.15253#S4.SS2.p1.1)\.
- F\. Steinle \(2002\)Experiments in history and philosophy of science\.InThe Philosophy of Scientific Experimentation,H\. Radder \(Ed\.\),Cited by:[§7\.5](https://arxiv.org/html/2605.15253#S7.SS5.p1.1)\.
- E\. Strubell, A\. Ganesh, and A\. McCallum \(2019\)Energy and policy considerations for deep learning in NLP\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 3645–3650\.Cited by:[§1](https://arxiv.org/html/2605.15253#S1.SS0.SSS0.Px2.p1.1),[§7\.3](https://arxiv.org/html/2605.15253#S7.SS3.p1.1)\.
- M\. Tan and Q\. V\. Le \(2019\)EfficientNet: rethinking model scaling for convolutional neural networks\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 6105–6114\.Cited by:[§2](https://arxiv.org/html/2605.15253#S2.SSx1.SSS0.Px1.p1.1)\.
- J\. Thiyagalingam, A\. Trefethen,et al\.\(2022\)Scientific machine learning benchmarks\.Philosophical Transactions of the Royal Society A380\(2229\),pp\. 20210336\.Cited by:[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p3.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2307.09288),[Link](https://arxiv.org/abs/2307.09288)Cited by:[§6](https://arxiv.org/html/2605.15253#S6.p4.1)\.
- UNESCO \(2025\)Smarter, smaller, stronger: resource\-efficient generative AI and the future of digital transformation\.Note:UNESCO reportExternal Links:[Link](https://unesdoc.unesco.org/ark:/48223/pf0000394521)Cited by:[§7\.3](https://arxiv.org/html/2605.15253#S7.SS3.p2.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§6](https://arxiv.org/html/2605.15253#S6.p2.1)\.
- P\. Walters \(2023\)We need better benchmarks for machine learning in drug discovery\.Note:[https://practicalcheminformatics\.blogspot\.com/2023/08/we\-need\-better\-benchmarks\-for\-machine\.html](https://practicalcheminformatics.blogspot.com/2023/08/we-need-better-benchmarks-for-machine.html)Blog postCited by:[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p3.1)\.
- A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman \(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP Workshop BlackboxNLP,pp\. 353–355\.Cited by:[§7\.1](https://arxiv.org/html/2605.15253#S7.SS1.p3.1)\.
- C\. Wu, R\. Raghavendra, U\. Gupta, B\. Acun,et al\.\(2022\)Sustainable AI: environmental implications, challenges and opportunities\.InProceedings of Machine Learning and Systems \(MLSys\),Cited by:[§7\.3](https://arxiv.org/html/2605.15253#S7.SS3.p2.1)\.
- J\. Xu, W\. Zhou, Z\. Fu, H\. Zhou, and L\. Li \(2021\)A survey on green deep learning\.arXiv preprint arXiv:2111\.05193\.Cited by:[§7\.3](https://arxiv.org/html/2605.15253#S7.SS3.p1.1)\.
- C\. Zhang, S\. Bengio, M\. Hardt, B\. Recht, and O\. Vinyals \(2017\)Understanding deep learning requires rethinking generalization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2605.15253#S1.p1.1),[§7\.2](https://arxiv.org/html/2605.15253#S7.SS2.p1.1)\.
- H\. Zhang, M\. Cisse, Y\. N\. Dauphin, and D\. Lopez\-Paz \(2018\)Mixup: beyond empirical risk minimization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4\.3](https://arxiv.org/html/2605.15253#S4.SS3.SSS0.Px1.p1.2),[§4\.3](https://arxiv.org/html/2605.15253#S4.SS3.p1.1)\.