Unified Neural Scaling Laws
Summary
Presents a unified neural scaling law that accurately models deep neural network scaling across multiple dimensions including parameters, dataset size, training steps, and compute, validated across diverse architectures and tasks.
View Cached Full Text
Cached at: 06/02/26, 03:34 PM
Paper page - Unified Neural Scaling Laws
Source: https://huggingface.co/papers/2605.26248
Abstract
A Unified Neural Scaling Law is presented that accurately models and extrapolates deep neural network scaling behaviors across multiple simultaneous dimensions including parameters, dataset size, training steps, and compute across diverse architectures and tasks.
We present a functional form (that we refer to as aUnified Neural Scaling Law(UNSL)) that accurately models and extrapolates thescaling behaviorsofdeep neural networksas multiple dimensions all vary simultaneously (i.e. how the evaluation metric of interest varies as one simultaneously varies the number ofmodel parameters,training dataset size, number oftraining steps, number ofinference steps, amount ofcompute, and varioushyperparameters) for variousarchitecturesand for each of various tasks within a varied set of upstream anddownstream tasks. This set includes large-scalevision,language,math, andreinforcement learning. When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.26248
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.26248 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.26248 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.26248 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Unified Neural Scaling Laws
This paper presents Unified Neural Scaling Laws (UNSL), a functional form that accurately models and extrapolates deep neural network scaling behaviors as multiple dimensions such as parameters, data, and steps vary simultaneously, improving over previous scaling laws.
Scaling laws for neural language models
Foundational empirical study demonstrating power-law scaling relationships between language model performance and model size, dataset size, and compute budget, with implications for optimal training allocation and sample efficiency.
How AI training scales
OpenAI researchers discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training across a wide range of tasks. They found that more complex tasks and more powerful models tolerate larger batch sizes, suggesting future AI systems can scale further through increased parallelization.
Prescriptive Scaling Laws for Data Constrained Training
A modified scaling law accounting for data repetition effects provides compute-optimal training strategies for data-constrained scenarios, showing that beyond a point further repetition is counterproductive and compute is better spent on model capacity.
Spectral Scaling Laws of Muon
This paper presents the first systematic study of singular value spectral behavior in Muon optimizer momentum matrices during LLM training, discovering clean power-law scaling relationships across model sizes (77M–2.8B parameters). The findings provide practitioners with principled, layer-aware guidelines for configuring Newton–Schulz iterations to maintain orthonormalization quality at frontier scale without unnecessary computation.