随机特征二元语法模型的精确缩放定律

# Exact Scaling Laws for a Random-Features Bigram Model
Source: [https://arxiv.org/html/2610.19281](https://arxiv.org/html/2610.19281)
\\jmlrissue\\jmlrpages

Julian Enoch1,2 Elvis Dohmatob1,2 1Concordia University 2Mila – Quebec AI Institute

###### Abstract

Understanding neural scaling laws is increasingly important for theory and practice, yet first\-principles explanations remain limited even in simple settings\. In this work, we study capacity\-limited scaling in a bottlenecked bigram model with power\-law \(Zipf\) token statistics, viewed as an analytically tractable toy model of a large language model\. Fixed random embeddings serve as a proxy for representational capacity and induce an explicit bottleneck of width, allowing us to derive scaling laws for prediction regret as a function of model width, vocabulary size, and the Zipf exponent\. Our results complement recent theoretical accounts based on optimization dynamics by isolating scaling due purely to representational constraints\. In the appropriate regime, the theory recovers the functional form of classical neural scaling laws and suggests that heavy\-tailed language statistics may underlie both optimization\- and capacity\-driven scaling\. Experiments natural language datasets \(WikiText2, OpenWebText\) closely match the predicted curves\.

††proceedings::## 1Introduction

Neural scaling laws describe systematic relationships between predictive performance, model size, training data, and compute, providing an empirical basis for allocating resources in language\-model training\([Kaplan et al\., 2020](https://arxiv.org/html/2610.19281#bib.bib1)\)\. Theoretical studies have connected these relationships to data distributions, model capacity, and optimization dynamics\([Maloney et al\., 2022](https://arxiv.org/html/2610.19281#bib.bib7);[Cabannes et al\., 2024](https://arxiv.org/html/2610.19281#bib.bib5);[Kunstner and Bach, 2025](https://arxiv.org/html/2610.19281#bib.bib4)\)\. An important challenge is to distinguish these mechanisms: similar learning curves can reflect insufficient observations, a restricted representation, or incomplete optimization\.

In language modeling, the highly uneven frequencies of tokens make the role of representational capacity particularly interesting\. An unconstrained bigram model assigns a separate next\-token distribution to every context, regardless of how frequently that context occurs\. Does accurate prediction require this degree of flexibility, or can a substantially smaller representation suffice? More broadly, how does the power\-law structure of language determine the cost of compressing its predictive information? Addressing these questions provides a way to connect empirical capacity scaling to explicit properties of the underlying data distribution\.

### 1\.1Contributions

Our main contributions can be summarized as follows\.

– A bottlenecked bigram model\.We introduce a capacity\-constrained bigram model with Zipfian token statistics as an analytically tractable toy model of a large language model\. Fixed random embeddings serve as a proxy for representational capacity, imposing a width\-NNbottleneck withd​NdNtrainable parameters\. By studying the optimally fitted predictor, we isolate the cost of limited capacity from optimization error\. This extends the unrestricted bigram setting of[Kunstner and Bach \(2025\)](https://arxiv.org/html/2610.19281#bib.bib4)by introducing an explicit representational constraint, complementing their analysis of scaling with the number of optimization steps\. In the summable Zipf regimeα\>1\\alpha\>1, our results also recover the associative\-memory scaling laws of[Cabannes et al\. \(2024\)](https://arxiv.org/html/2610.19281#bib.bib5)\.

– Exact scaling laws\.We obtain an exact representation of regret relative to the unconstrained empirical optimum and derive asymptotic scaling laws in vocabulary sizedd, sample sizenn, embedding dimensionNN, and the Zipf exponent\. The parameterizationN=γ​dβN=\\gamma d^\{\\beta\}reveals distinct sublinear and linear capacity regimes\. In the appropriate regime, the resulting laws recover the functional form of classical empirical neural scaling laws\([Kaplan et al\., 2020](https://arxiv.org/html/2610.19281#bib.bib1)\)\.

– An explicit account of finite\-sample effects\.Our theory incorporates vocabulary occupancy and variability in empirical transition probabilities, separating a capacity\-dependent contribution to regret from a finite\-sample correction\. This identifies when capacity constraints dominate and when limited observations materially affect the scaling behavior\.

– Empirical evaluation\.In Section[4](https://arxiv.org/html/2610.19281#S4), we present results of extensive experiments which validate our scaling laws both on real\-world natural language datasets, WikiText\-2\([Merity et al\., 2017](https://arxiv.org/html/2610.19281#bib.bib14)\)and OpenWebText2\([Gao et al\., 2020](https://arxiv.org/html/2610.19281#bib.bib15)\), support the predicted scaling curves across vocabulary sizesdd, sample sizesnn, and embedding dimensionsNN\. They examine both the decay of regret under sublinear capacity and its approach to zero as the bottleneck becomes sufficiently wide\.

### 1\.2Related work

The pioneering work of[Kaplan et al\. \(2020\)](https://arxiv.org/html/2610.19281#bib.bib1)established empirical relationships between language\-model loss, parameter count, training data, and compute, motivating the use of scaling laws for resource allocation\. Subsequent theoretical work has identified several mechanisms that can produce such behavior\.

Zipfian statistics, sampling, and optimization\.[Hutter \(2021\)](https://arxiv.org/html/2610.19281#bib.bib3)derive data\-dependent learning curves for a memorization\-based model, explaining how power\-law frequencies lead to power\-law decay of prediction error with sample size\. More recently,[Kunstner and Bach \(2025\)](https://arxiv.org/html/2610.19281#bib.bib4)analyze gradient descent and sign descent for a linear bigram model under Zipfian assumptions, characterizing how optimization depends on vocabulary size and the frequency exponent\. Their results emphasize the importance of the regime nearα=1\\alpha=1\. Neither analysis introduces a representation width that can be varied independently of vocabulary size: Hutter focuses on sample\-size scaling, while Kunstner and Bach study optimization of an unrestricted bigram parameterization\. Consequently, they do not characterize the approximation error caused by a separate embedding\-width budget\. We introduce that capacity axis through a fixed random embedding of widthNN, distinguishing representational limitations from finite\-data and optimization effects\.

Finite\-capacity associative memories\.[Cabannes et al\. \(2024\)](https://arxiv.org/html/2610.19281#bib.bib5)provide a closely related capacity\-based perspective, deriving sample\- and parameter\-scaling laws for associative memories built from outer products of random embeddings\. Their analysis explicitly addresses finite memory and the efficiency of different storage schemes\. The principal infinite\-vocabulary power\-law results concern the summable regimeα\>1\\alpha\>1\. In that regime, the Zipf normalizing constant has a finite limit as vocabulary size grows, allowing leading\-order laws that do not retain an explicit vocabulary cutoff\. This contrasts withα≤1\\alpha\\leq 1, where normalization remains vocabulary\-dependent\. Our analysis keeps vocabulary sizeddexplicit and studies its interaction with the bottleneck width, including sublinear and linear parameterizationsN=γ​dβN=\\gamma d^\{\\beta\}\. We also study squared\-loss regret for conditional next\-token probabilities rather than the association\-retrieval error analyzed in that work\.

Spectral and random\-feature theories\.Kernel and random\-feature models provide another route to analytical scaling laws\.[Cui et al\. \(2021\)](https://arxiv.org/html/2610.19281#bib.bib6)characterize how spectral decay, target regularity, noise, and regularization determine learning\-curve exponents, including the crossover between noiseless and noisy regimes\.[Maloney et al\. \(2022\)](https://arxiv.org/html/2610.19281#bib.bib7)develop a solvable generative and random\-feature model with joint dependence on dataset size and parameter count, and explain how a finite spectral range can interrupt power\-law scaling\. These works already incorporate statistical capacity constraints; our contribution is not to introduce capacity into scaling\-law theory in general\. Rather, we connect an explicit random\-embedding bottleneck to discrete Zipfian token statistics and characterize regret relative to the unconstrained empirical bigram optimum\. This distinction also matters when comparing our results with population generalization error or cross\-entropy scaling in other models\.

[Bahri et al\. \(2024\)](https://arxiv.org/html/2610.19281#bib.bib2)distinguish variance\-limited and resolution\-limited regimes and relate scaling exponents to spectral properties of the learning problem\. These results indicate that superficially similar power laws can arise from different statistical or computational limitations\. Our work studies one such mechanism in a discrete next\-token prediction setting: the error induced by a restricted representation, after optimizing the predictor within that representation\.

Theory of skill acquisition in language models\.[Michaud et al\. \(2023\)](https://arxiv.org/html/2610.19281#bib.bib8)propose that smooth aggregate scaling can emerge from learning discrete skills with a heavy\-tailed frequency distribution\.[Nam et al\. \(2024\)](https://arxiv.org/html/2610.19281#bib.bib9)develop an exactly solvable model for multitask sparse parity, connecting skill emergence with scaling in training time, data, and model size\. In a stochastic\-process formulation,[Dębowski \(2023\)](https://arxiv.org/html/2610.19281#bib.bib10)construct multiperiodic Santa Fe processes and memory\-based predictors exhibiting power\-law cross\-entropy decay with model and data resources\. These approaches explain scaling through the acquisition of discrete information\. Our model instead studies the approximation of empirical next\-token distributions through a shared, finite\-dimensional random representation, without postulating a sequence of separately acquired skills\.

Distributional tails and model collapse\.[Dohmatob et al\. \(2024\)](https://arxiv.org/html/2610.19281#bib.bib11)study how incorporating synthetic data into training distributions changes scaling laws and can lead to model collapse\([Shumailov et al\., 2024](https://arxiv.org/html/2610.19281#bib.bib16)\)\. Their analysis highlights the importance of preserving distributional tails: increasing data or model size need not yield the same improvements when rare events are suppressed\. This provides a complementary distributional perspective on scaling, whereas our work studies the regret induced by a representational bottleneck under a specified Zipfian distribution\.

Attention models andnn\-gram structure\.[Lyu et al\. \(2025\)](https://arxiv.org/html/2610.19281#bib.bib12)derive scaling laws for a solvable self\-attention model, characterizing dependencies on training time, model size, data, and context length\. From a complementary empirical perspective,[Nguyen \(2024\)](https://arxiv.org/html/2610.19281#bib.bib13)investigate how well rules based onnn\-gram statistics approximate transformer next\-token predictions\. These studies motivate simplified sequence\-prediction models as analytical testbeds, without equating them with full transformers\. Our bottlenecked bigram model isolates the interaction between Zipfian token statistics and finite representational capacity, rather than modeling attention or long\-context dependencies\.

## 2Theoretical Setup

Like[Kunstner and Bach \(2025\)](https://arxiv.org/html/2610.19281#bib.bib4), we consider a bigram model in which each word is associated with a unique integeri∈\[d\]i\\in\[d\], where\[d\]:=\{1,2,…,d\}\[d\]:=\\\{1,2,\\ldots,d\\\}is the vocabulary, andd≫d\\gg1 is the vocabulary size\. The frequency ofiith token/word ispip\_\{i\}, and conditioned onkkthe frequency of the next word beingjjisπj\|i\\pi\_\{j\\mid i\}\. This defines a distributionPPover word pairs\. We make the following assumption on this distribution\.

###### Assumption 1\.

\(i\)Zipf marginals\.There exists an exponentα\>0\\alpha\>0such that theiith word has frequency

pi=i−αHd,α,with​Hd,α:=∑k=1d1kα\.\\displaystyle p\_\{i\}=\\frac\{i^\{\-\\alpha\}\}\{H\_\{d,\\alpha\}\},\\text\{ with \}H\_\{d,\\alpha\}:=\\sum\_\{k=1\}^\{d\}\\frac\{1\}\{k^\{\\alpha\}\}\.\(1\)
\(ii\)Constant collision probability\.There exists a constantA∈\(0,1\]A\\in\(0,1\]such that for each wordi∈\[d\]i\\in\[d\],

∑j=1dπj\|i2=A\.\\displaystyle\\sum\_\{j=1\}^\{d\}\\pi\_\{j\\mid i\}^\{2\}=A\.\(2\)

The distributional constantsα\\alphaandAAwill play a crucial role in our theory\. A similar assumption was made in\([Cabannes et al\., 2024](https://arxiv.org/html/2610.19281#bib.bib5)\)\. Also,[Kunstner and Bach \(2025\)](https://arxiv.org/html/2610.19281#bib.bib4)considered a similar but strong similar assumption as above was also made in and\([Kunstner and Bach, 2025](https://arxiv.org/html/2610.19281#bib.bib4)\)\. Indeed, in place of the second part of the above assumption, they assumed that the next\-word frequencies also followed a Zipf law with exponentα′\>0\\alpha^\{\\prime\}\>0\. Such distributions verify the invariance \([2](https://arxiv.org/html/2610.19281#S2.E2)\) above withA=Hd,2​α′/Hd,α′2A=H\_\{d,2\\alpha^\{\\prime\}\}/H\_\{d,\\alpha^\{\\prime\}\}^\{2\}\.

Table 1:Global parameters and descriptive statistics for real\-world natural language datasets\. In our experiments, we sample subsets of these datasets, restricting both the vocabulary sizeddand the sample sizenn\.### 2\.1Unconstrained Bigram Model

Our proposed capacity\-constrained bigram model \(Section[2\.2](https://arxiv.org/html/2610.19281#S2.SS2)\) is motivated by the following full\-capacity bigram model considered by[Kunstner and Bach \(2025\)](https://arxiv.org/html/2610.19281#bib.bib4), which we now describe\.

Nature draws an i\.i\.d\. sample\{\(x1,y1\),…,\(xn,yn\)\}\\\{\(x\_\{1\},y\_\{1\}\),\\ldots,\(x\_\{n\},y\_\{n\}\)\\\}of sizennfromPP, where eachxtx\_\{t\}andyty\_\{t\}is the one\-hot encoding of the corresponding symbol in the alphabet\[d\]\[d\]\. Table[1](https://arxiv.org/html/2610.19281#S2.T1)lists the values for the sample sizennand the vocabulary sizennfor popular natural language datasets: WikiText\-2\([Merity et al\., 2017](https://arxiv.org/html/2610.19281#bib.bib14)\)and OpenWebText2\([Gao et al\., 2020](https://arxiv.org/html/2610.19281#bib.bib15)\)\. LetXXandYYbe then×dn\\times dbinary matrices whosettth rows arext⊤x\_\{t\}^\{\\top\}andyt⊤y\_\{t\}^\{\\top\}, respectively\.[Kunstner and Bach \(2025\)](https://arxiv.org/html/2610.19281#bib.bib4)considered the problem of approximatingYYbyX​WXWthrough minimization of the loss

Ln,d​\(W\):=n−1​‖X​W−Y‖Fro2,L\_\{n,d\}\(W\):=n^\{\-1\}\\\|XW\-Y\\\|\_\{\\mathrm\{Fro\}\}^\{2\},\(3\)over a parameter matrixW∈ℝd×dW\\in\\mathbb\{R\}^\{d\\times d\}\. Here,‖Z‖Fro2:=tr⁡Z⊤​Z\\\|Z\\\|\_\{\\mathrm\{Fro\}\}^\{2\}:=\\operatorname\{tr\}Z^\{\\top\}Zdenotes the squared Frobenius \(aka Hilbert–Schmidt\) norm of a matrixZZ\. The learnable parameters are the entries ofWW\. LetW∗W\_\{\*\}denote a minimizer ofLn,dL\_\{n,d\}, and define the followingd×dd\\times dmatrices

K:=n−1​X⊤​X,B:=n−1​X⊤​Y\.\\displaystyle K:=n^\{\-1\}X^\{\\top\}X,\\quad B:=n^\{\-1\}X^\{\\top\}Y\.\(4\)When the matrixKKis invertible \(which happens for largenn\), the normal equations giveW∗=K−1​BW\_\{\*\}=K^\{\-1\}B\.

For a fitted matrixWW, the corresponding downstream predictor isf^​\(i\):=arg⁡maxj∈\[d\]⁡\(W⊤​x\)j\\widehat\{f\}\(i\):=\\arg\\max\_\{j\\in\[d\]\}\(W^\{\\top\}x\)\_\{j\}, wherex=δi∈ℝdx=\\delta\_\{i\}\\in\\mathbb\{R\}^\{d\}is the one\-hot encoding ofii\. Predictive performance can also be assessed using top\-kkaccuracy, the proportion of examples for which the true next symbol appears among thekkhighest\-scoring predictions, withk=3k=3, for example\.

Scaling Laws for Gradient Descent\.Consider a sequence of iteratesW⁡\(t\)W\(t\)generated by gradient descent \(GD\) initialized at zero,W⁡\(0\)=0W\(0\)=0\. For invertibleKKin \([4](https://arxiv.org/html/2610.19281#S2.E4)\), the regret at initialization satisfies

Ln,d​\(0\)−Ln,d​\(W∗\)\\displaystyle L\_\{n,d\}\(0\)\-L\_\{n,d\}\(W\_\{\*\}\)=tr⁡\(W∗⊤​K​W∗\)=tr⁡\(B⊤​K−1​B\)≃∑i=1dpi​∑j=1dπj\|i2=A,\\displaystyle=\\operatorname\{tr\}\(W\_\{\*\}^\{\\top\}KW\_\{\*\}\)=\\operatorname\{tr\}\(B^\{\\top\}K^\{\-1\}B\)\\simeq\\sum\_\{i=1\}^\{d\}p\_\{i\}\\sum\_\{j=1\}^\{d\}\\pi\_\{j\\mid i\}^\{2\}=A,whereA∈\(0,1\]A\\in\(0,1\]is the constant defined in \([2](https://arxiv.org/html/2610.19281#S2.E2)\)\. The approximation replaces empirical frequencies by their population counterparts in the large\-sample regime\.

[Kunstner and Bach \(2025\)](https://arxiv.org/html/2610.19281#bib.bib4)studied as a function of optimization horizonTT, the normalized regret measure

rn,d​\(T\):=Ln,d​\(W⁡\(T\)\)−Ln,d​\(W∗\)Ln,d​\(0\)−Ln,d​\(W∗\)\.r\_\{n,d\}\(T\):=\\frac\{L\_\{n,d\}\(W\(T\)\)\-L\_\{n,d\}\(W\_\{\*\}\)\}\{L\_\{n,d\}\(0\)\-L\_\{n,d\}\(W\_\{\*\}\)\}\.and distinguished three regimes of the Zipf exponent in \([1](https://arxiv.org/html/2610.19281#S2.E1)\): subcritical \(α<1\\alpha<1\), critical \(α=1\\alpha=1\), and supercritical \(α\>1\\alpha\>1\)\. For each regime, they identified appropriate scalings of the iteration countTTwith vocabulary sizeddand derived explicit asymptotic laws for GD\. In particular, in the supercritical regime, their large\-vocabulary analysis yields the epoch\-wise scaling law

rn,d​\(T\)≍T−c,c:=1−1/α,r\_\{n,d\}\(T\)\\asymp T^\{\-c\},\\qquad c:=1\-1/\\alpha,for largeTT\. This exponent coincides with that obtained by[Hutter \(2021\)](https://arxiv.org/html/2610.19281#bib.bib3);[Cabannes et al\. \(2024\)](https://arxiv.org/html/2610.19281#bib.bib5)for sample\-size scaling, although the resource varied here is the number of optimization steps\. The same work also derives scaling laws for sign gradient descent \(sign\-GD\), showing how the choice of optimizer changes the dependence on vocabulary size\.

These results describe optimization\-time scaling under idealized Zipfian frequency assumptions\. Normalizing the regret factors out the initialization scaleAA, which can itself depend ondd, while the limiting curves express the remaining vocabulary\-size dependence through a rescaling of time\. Finite\-sample fluctuations are not tracked explicitly in this formulation\. Our results instead retain explicit dependence on vocabulary sizeddand sample sizenn, together with an independently adjustable embedding widthNN, to characterize regret under a representational constraint\.

### 2\.2A Capacity\-Constrained Biagram Model

In the model \([3](https://arxiv.org/html/2610.19281#S2.E3)\) studied in\([Kunstner and Bach, 2025](https://arxiv.org/html/2610.19281#bib.bib4)\)\. the number of trainable parameters equalsd2d^\{2\}, withdddenoting the vocabulary size\. However, this is unrealistic for the large\-vocabulary setting considered here, which is also typical of real language models\. To better reflect practice, we instead introduce a notion of capacity \(i\.e\., parameter count\) as follows\. Given an embedding matrixΩ∈ℝd×N\\Omega\\in\\mathbb\{R\}^\{d\\times N\}with embedding dimensionN≤1N\\leq 1, we assume the weights matrixWWadmits the following factorization:

W=Ω​Θ,\\displaystyle W=\\Omega\\Theta,\(5\)with learnable factorΘ∈ℝN×d\\Theta\\in\\mathbb\{R\}^\{N\\times d\}\. This is a capacity constraint, and reduced the number of learnable parameters fromd2d^\{2\}toN​d≤d2Nd\\leq d^\{2\}\. Consider the modified optimization problem

minimize⁡Ln,d​\(Ω​Θ\)​over​Θ∈ℝN×d,\\operatorname\{minimize\}L\_\{n,d\}\(\\Omega\\Theta\)\\text\{ over \}\\Theta\\in\\mathbb\{R\}^\{N\\times d\},\(6\)whereLn,dL\_\{n,d\}is the loss function defined in \([3](https://arxiv.org/html/2610.19281#S2.E3)\)\.

###### Assumption 2\(Random Embeddings\)\.

The entries of the embedding matrixΩ\\Omegaare sampled iid from a distribution with zero mean, variance1/d1/d, and with Gaussian concentration property\. For example, the Gaussian distribution itself𝒩⁡\(0,1/d\)\\mathcal\{N\}\(0,1/d\)\.

LetΘ^n,d,N∈ℝN×d\\hat\{\\Theta\}\_\{n,d,N\}\\in\\mathbb\{R\}^\{N\\times d\}be any minimizer in \([6](https://arxiv.org/html/2610.19281#S2.E6)\) and setW^n,d,N:=Ω​Θ^n,d,N∈ℝd×d\\hat\{W\}\_\{n,d,N\}:=\\Omega\\hat\{\\Theta\}\_\{n,d,N\}\\in\\mathbb\{R\}^\{d\\times d\}\. Our main focus will be the analysis of the f expected normalized regret:

rn,d​\(N\):=𝔼\[Rn,d\(N\)\],Rn,d\(N\):=Ln,d​\(W^n,d,N\)−Ln,d​\(W∗\)Ln,d​\(0\)−Ln,d​\(W∗\),\\begin\{split\}r\_\{n,d\}\(N\)&:=\\mathbb\{E\}\[R\_\{n,d\}\(N\)\],\\quad R\_\{n,d\}\(N\):=\\frac\{L\_\{n,d\}\(\\hat\{W\}\_\{n,d,N\}\)\-L\_\{n,d\}\(W\_\{\*\}\)\}\{L\_\{n,d\}\(0\)\-L\_\{n,d\}\(W\_\{\*\}\)\},\\end\{split\}\(7\)whereA∈\(0,1\]A\\in\(0,1\]is the constant in \([2](https://arxiv.org/html/2610.19281#S2.E2)\)\. We have made explicit the dependence on sample sizenn, vocabulary sizedd, and embedding dimensionNN, as these will appear explicitly in our scaling laws\.

## 3Main Results

### 3\.1Bias\-Variance Decomposition

Consider the vectorc=\(c1,…,cd\)∈ℝdc=\(c\_\{1\},\\ldots,c\_\{d\}\)\\in\\mathbb\{R\}^\{d\}defined by

ck:=\#⁡\{t∈\[n\]∣xt=ek\},\\displaystyle c\_\{k\}:=\\\#\\\{t\\in\[n\]\\mid x\_\{t\}=e\_\{k\}\\\},\(8\)the number of times thekkth vocabulary symbol appears in the dataset\. LetSSbe the number of unique vocabulary symbols present in the training dataset,

S:=\#​I,where​I:=\{k∈\[d\]∣ci\>0\}\.\\displaystyle S:=\\\#I,\\text\{ where \}I:=\\\{k\\in\[d\]\\mid c\_\{i\}\>0\\\}\.\(9\)Note that butSSandIIare deterministic functions of the vector of countscc\. It is clear that∑kck=n\\sum\_\{k\}c\_\{k\}=nandS≤dS\\leq d\. Finally, for anyk∈\[d\]k\\in\[d\], letp^k:=ck/n\\hat\{p\}\_\{k\}:=c\_\{k\}/nbe the empirical frequency of thekkth token\.

We start with the following general result\.

###### Proposition 1\.

Under Assumption[1](https://arxiv.org/html/2610.19281#S2.EGx2), the conditional expected regret admits the following bias\-variance decomposition:

𝔼\[Rn,d\(N\)∣c\]≃V\+B,B:=ANν,V:=\(1−A\)\(S−N\)\+n,\\boxed\{\\begin\{split\}&\\mathbb\{E\}\[R\_\{n,d\}\(N\)\\mid c\]\\simeq V\+B,\\quad B:=AN\\nu,\\quad V:=\(1\-A\)\\frac\{\(S\-N\)\_\{\+\}\}\{n\},\\end\{split\}\}\(10\)whereA∈\(0,1\)A\\in\(0,1\)is defined in \([2](https://arxiv.org/html/2610.19281#S2.E2)\),p^k:=ck/n\\hat\{p\}\_\{k\}:=c\_\{k\}/n, andν=0\\nu=0whenN≥SN\\geq S\. For0<N<S0<N<S,ν\\nuis the unique positive solution to the fixed\-point equation

N=∑k∈Ip^kp^k\+ν\.\\displaystyle N=\\sum\_\{k\\in I\}\\dfrac\{\\hat\{p\}\_\{k\}\}\{\\hat\{p\}\_\{k\}\+\\nu\}\.\(11\)

The second term,V:=\(1−A\)​\(S−N\)\+/nV:=\(1\-A\)\(S\-N\)\_\{\+\}/nin the decomposition \([10](https://arxiv.org/html/2610.19281#S3.E10)\) captures the finite\-sample variability of the empirical transition probabilities and can be interpreted as a*variance contribution*\. It is present only when the embedding widthNNis smaller than the numberSSof observed contexts, and vanishes onceN≥SN\\geq S\. Here,S=S⁡\(c\)≤min⁡\(d,n\)S=S\(c\)\\leq\\min\(d,n\)is the realized vocabulary occupancy defined in \([9](https://arxiv.org/html/2610.19281#S3.E9)\); its distribution depends on the sample size and the underlying context probabilities over the vocabulary\. In particular, the unconstrained bigram model hasN=d≥SN=d\\geq S, and soV=0V=0\.

In contrast, the first term,B:=A​N​νB:=AN\\nucaptures the*approximation cost*of the representational bottleneck and can be interpreted as a bias contribution\. This is reminiscent of model*bias*\. For0<N<S0<N<S, the fixed\-point equation gives the equivalent expressionB=A​∑k∈Ip^k​νp^k\+ν\.B=A\\sum\_\{k\\in I\}\\dfrac\{\\hat\{p\}\_\{k\}\\nu\}\{\\hat\{p\}\_\{k\}\+\\nu\}\.Thus, unlikeVV, this term depends on the full empirical context\-frequency profile, rather than only on the number of observed contexts\. For a fixed frequency profile, it decreases as the embedding width increases\. However, it need not vanish as the sample size grows whenN<dN<d: additional observations cannot by themselves remove the limitations of a fixed representation\.

Finally, whenN≥SN\\geq S, both terms in \([10](https://arxiv.org/html/2610.19281#S3.E10)\) vanish\. This agrees with the exact interpolation property: whenever the embedding matrix restricted to the observed contexts has full row rank, the student can reproduce all empirical transition rows, yieldingRn,d​\(N\)=0R\_\{n,d\}\(N\)=0\.

### 3\.2Sketch of Proof of Proposition[1](https://arxiv.org/html/2610.19281#Thmtheorem1)

We provide the high\-level ideas here\. The complete proof is provided in the supplemental \(appendix\)\.

Conditioning on the empirical counts vectorc=\(c1,…,cd\)∈ℝdc=\(c\_\{1\},\\ldots,c\_\{d\}\)\\in\\mathbb\{R\}^\{d\}, one first show the exact identity

𝔼Y,Ω​\[Rn,d​\(N\)∣c\]=1n​∑k∈Ick​‖Mk‖22​\(1−hk\),\\displaystyle\\mathbb\{E\}\_\{Y,\\Omega\}\[R\_\{n,d\}\(N\)\\mid c\]=\\frac\{1\}\{n\}\\sum\_\{k\\in I\}c\_\{k\}\\\|M\_\{k\}\\\|\_\{2\}^\{2\}\\,\(1\-h\_\{k\}\),\(12\)whereMkM\_\{k\}is thekkth row of the empirical matrixM∈ℝd×dM\\in\\mathbb\{R\}^\{d\\times d\}of transition frequencies, i\.e\.

Mk​k′:=1ck​∑i=1nxi​k​yi​k′,\\displaystyle M\_\{kk^\{\\prime\}\}:=\\frac\{1\}\{c\_\{k\}\}\\sum\_\{i=1\}^\{n\}x\_\{ik\}\\,y\_\{ik^\{\\prime\}\},andhk:=𝔼⁡\[Pk​k\]h\_\{k\}:=\\mathbb\{E\}\[P\_\{kk\}\]is the expectation of thekkth diagonal of entry of the matrixP∈ℝd×dP\\in\\mathbb\{R\}^\{d\\times d\}which is the orthogonal projection onto the column space ofDc1/2​ΩD\_\{c\}^\{1/2\}\\Omega, withDc:=Diag⁡\(c\)D\_\{c\}:=\\operatorname\{Diag\}\(c\)\. One then computes explicitly

ck​‖Mk‖2=ck​ak\+1−ak,with​ak:=∑k′=1dπ^k′\|k2\.c\_\{k\}\\\|M\_\{k\}\\\|^\{2\}=c\_\{k\}a\_\{k\}\+1\-a\_\{k\},\\text\{ with \}a\_\{k\}:=\\sum\_\{k^\{\\prime\}=1\}^\{d\}\\hat\{\\pi\}\_\{k^\{\\prime\}\\mid k\}^\{2\}\.By entry\-wise concentration ofMMto the next\-word transition matrixπ\\pi\(for example, see\([Berend and Kontorovich, 2012](https://arxiv.org/html/2610.19281#bib.bib17)\)\), we know thatak≃∑k′=1dπk′\|k2=Aa\_\{k\}\\simeq\\sum\_\{k^\{\\prime\}=1\}^\{d\}\\pi\_\{k^\{\\prime\}\\mid k\}^\{2\}=Auniformly onk∈\[d\]k\\in\[d\], with high probability\. whereA∈\(0,1\]A\\in\(0,1\]is the constant appearing in equation \([2](https://arxiv.org/html/2610.19281#S2.E2)\) of Assumption[1](https://arxiv.org/html/2610.19281#S2.EGx2)\. It follows from \([12](https://arxiv.org/html/2610.19281#S3.E12)\) that

𝔼⁡\[Rn,d​\(N\)∣c\]≃An​∑k∈I\(1−hk\)​ck\+1−An​∑k∈I\(1−hk\)\.\\mathbb\{E\}\[R\_\{n,d\}\(N\)\\mid c\]\\simeq\\frac\{A\}\{n\}\\sum\_\{k\\in I\}\(1\-h\_\{k\}\)c\_\{k\}\+\\frac\{1\-A\}\{n\}\\sum\_\{k\\in I\}\(1\-h\_\{k\}\)\.Finally, using random matrix theory, one shows thathk≃ck/\(ck\+N​ν\)h\_\{k\}\\simeq c\_\{k\}/\(c\_\{k\}\+N\\nu\)uniformly onkk, whereν\>0\\nu\>0is as defined in the proposition\. Plugging this estimate into the above equation and rearranging then gives the desired result\. ∎

### 3\.3Explicit Scaling Laws

Note that the RHS in \([10](https://arxiv.org/html/2610.19281#S3.E10)\) is random as it depends on the random countsc=\(c1,…,cd\)c=\(c\_\{1\},\\ldots,c\_\{d\}\)\. In order to de\-randomize the formula, we need the sample sizennto be sufficiently large as a function of the vocabulary sizedd\. Letα\\alphathe Zipfian exponent for the marginals as in Assumption[1](https://arxiv.org/html/2610.19281#S2.EGx2)\.

###### Theorem 2\.

Consider the sublinear regime where the random embedding dimensionN≤dN\\leq dgrows with the vocabulary sizedd\. Finally, assume sample sizennis sufficiently large in the sense thatn≫dα​Hd,αn\\gg d^\{\\alpha\}H\_\{d,\\alpha\}\. Then, we have the followin exact scaling law for the regret:

rn,d​\(N\)≃B\+V,with​B:=A​N​ν,and​V:=\(1−A\)​\{ϕ,if​N=o⁡\(d\),\(1−ψ\)​ϕ,if​N≍d,\\displaystyle\\boxed\{\\begin\{split\}r\_\{n,d\}\(N\)&\\simeq B\+V,\\text\{ with \}B:=AN\\nu,\\text\{ and \}V:=\(1\-A\)\\begin\{cases\}\\phi,&\\mbox\{ if \}N=o\(d\),\\\\ \(1\-\\psi\)\\phi,&\\mbox\{ if \}N\\asymp d,\\end\{cases\}\\end\{split\}\}\(13\)where the constantA∈\(0,1\]A\\in\(0,1\]is the statistic defined in \([2](https://arxiv.org/html/2610.19281#S2.E2)\), whileψ=ψ⁡\(N,d\):=N/d\\psi=\\psi\(N,d\):=N/d,ϕ=ϕ⁡\(n,d\):=d/n\\phi=\\phi\(n,d\):=d/n, andν=να​\(N,d\)\\nu=\\nu\_\{\\alpha\}\(N,d\)is the unique positive solution to the following fixed\-point equation

N=∑i=1dpipi\+ν\.\\displaystyle N=\\sum\_\{i=1\}^\{d\}\\dfrac\{p\_\{i\}\}\{p\_\{i\}\+\\nu\}\.\(14\)
Furthermore, we have the following breakdown:

- \(i\)\(Sub\-critical Zipf exponent\)\.If the exponent in \([1](https://arxiv.org/html/2610.19281#S2.E1)\) verifies0≤α<10\\leq\\alpha<1, then B≃A​\{1,if​N=o⁡\(d\),\(1−α\)​ψ/ρ,if​N≍d\\displaystyle B\\simeq A\\begin\{cases\}1,&\\mbox\{ if \}N=o\(d\),\\\\ \(1\-\\alpha\)\\psi/\\rho,&\\mbox\{ if \}N\\asymp d\\end\{cases\}\(15\)whereρ=ρ⁡\(α,ψ\)\\rho=\\rho\(\\alpha,\\psi\)is the unique positive solution to the fixed\-point equation ψ=∫01ρρ\+tα​𝑑t\.\\displaystyle\\psi=\\int\_\{0\}^\{1\}\\dfrac\{\\rho\}\{\\rho\+t^\{\\alpha\}\}\\mathrm\{d\}t\.\(16\)
- \(ii\)\(Critical Zipf exponent\)\.Ifα=1\\alpha=1, then B≃Alog⁡d​\{log⁡\(d/N\),if​N=o⁡\(d\),1/q,if​N≍d,\\displaystyle B\\simeq\\frac\{A\}\{\\log d\}\\begin\{cases\}\\log\(d/N\),&\\mbox\{ if \}N=o\(d\),\\\\ 1/q,&\\mbox\{ if \}N\\asymp d,\\end\{cases\}\(17\)where is the unique positive solution to the following fixed\-point equation: N=∑i=1dqq\+i\.\\displaystyle N=\\sum\_\{i=1\}^\{d\}\\dfrac\{q\}\{q\+i\}\.\(18\)
- \(iii\)\(Super\-critical Zipf exponent\)\.Ifα\>1\\alpha\>1, thenB≃Cα​N−\(α−1\)B\\simeq C\_\{\\alpha\}N^\{\-\(\\alpha\-1\)\}, whereCαC\_\{\\alpha\}is a positive constant only depending onα\\alpha\.

In the super\-critical regime \(α\>1\\alpha\>1\), theα−1\\alpha\-1exponent in our scaling lawB≃N−\(α−1\)B\\simeq N^\{\-\(\\alpha\-1\)\}the one reported in one of the main results in\([Cabannes et al\., 2024](https://arxiv.org/html/2610.19281#bib.bib5)\)\. In the critical regime \(α=1\\alpha=1\), letb=−𝒲−1​\(−ψ​e−ψ\)−ψb=\-\\mathcal\{W\}\_\{\-1\}\(\-\\psi e^\{\-\\psi\}\)\-\\psi, where𝒲−1\\mathcal\{W\}\_\{\-1\}denotes the−1\-1branch of the Lambert–W function\. Then the1/q1/qterm in \([17](https://arxiv.org/html/2610.19281#S3.E17)\) admits the following expansion:

1q≃b−b2b−1​12​N\.\\dfrac\{1\}\{q\}\\simeq b\-\\dfrac\{b^\{2\}\}\{b\-1\}\\,\\dfrac\{1\}\{2N\}\.
Note that the conditionn≫dα​Hd,αn\\gg d^\{\\alpha\}H\_\{d,\\alpha\}is very mild and in satisfied in language datasets like OpenWebText2\([Gao et al\., 2020](https://arxiv.org/html/2610.19281#bib.bib15)\), WikiText\-2\([Merity et al\., 2017](https://arxiv.org/html/2610.19281#bib.bib14)\), whereα<1\\alpha<1\(refer to Table[1](https://arxiv.org/html/2610.19281#S2.T1)\), and soHd,α=O⁡\(log⁡d\)H\_\{d,\\alpha\}=O\(\\log d\); the condition then simply requires the sample sizennto grow super\-linearly with vocabulary sizedd\. This condition ensures that the dataset contains enough information to allow for the estimation of the head of the marginal distribution of the tokens in the vocabulary\.

In the sub\-critical and critical regimes which better capture to natural language distribution, our theorem provides new scaling laws which better match the empirically observed laws of[Kaplan et al\. \(2020\)](https://arxiv.org/html/2610.19281#bib.bib1)\.

### 3\.4A Canonical Scaling for the Embedding Dimension

A reasonable embedding dimensionNNw\.r\.t the vocabulary sizeddis

N≃γ​dβ,\\displaystyle N\\simeq\\gamma d^\{\\beta\},\(19\)for constantsγ,β\>0\\gamma,\\beta\>0withβ≤1\\beta\\leq 1\. In particular,β=1\\beta=1captures the linear scaling regime whereN≍dN\\asymp d, whileβ∈\(0,1\)\\beta\\in\(0,1\)captures the sublinear regime whereN=o⁡\(d\)N=o\(d\)\. By fixingddand sample sizenn, we empirically trace the decay of the excess risk and overlay the exact closed\-form asymptotics derived from our random matrix theory approach\. For example, in the critical setting where the marginal Zipf exponent isα=1\\alpha=1, the above theorem predicts that provided the sample size is sufficiently large \(i\.e\.n≫d​log⁡dn\\gg d\\log d\), the quenched regret is given by

rn,d​\(N\)≃\{A⁡\(1−β\)\+\(1−A\)​ϕ,if​β<1,A/\(q​log⁡d\)\+\(1−A\)​\(1−γ\)​ϕ,if​β=1,\\displaystyle r\_\{n,d\}\(N\)\\simeq\\begin\{cases\}A\(1\-\\beta\)\+\(1\-A\)\\phi,&\\mbox\{ if \}\\beta<1,\\\\ A/\(q\\log d\)\+\(1\-A\)\(1\-\\gamma\)\\phi,&\\mbox\{ if \}\\beta=1,\\end\{cases\}\(20\)withϕ:=d/n\\phi:=d/n\.

## 4Empirical Validation

### 4\.1Experimental Setup

We validate the capacity–risk predictions from Section[3](https://arxiv.org/html/2610.19281#S3)in a setting where representational capacity can be tuned independently of corpus statistics\. We study the capacity\-constrained linear bigram model in which the target embeddings are restricted to a rank\-NNsubspace via a fixed random projection\. For each configuration, we optimize to convergence and report the normalized excess risk \(regret\)rn,d​\(N\)r\_\{n,d\}\(N\)relative to the unconstrained empirical optimum \(cf\. \([13](https://arxiv.org/html/2610.19281#S3.E13)\)\)\.

To connect with the broader scaling\-laws literature\([Kaplan et al\., 2020](https://arxiv.org/html/2610.19281#bib.bib1)\), we treatNNas an explicit capacity axis and couple it to vocabulary size throughN=γ​dβN=\\gamma d^\{\\beta\}\(\([19](https://arxiv.org/html/2610.19281#S3.E19)\)\), sweepingγ\\gammaandβ\\betato probe both the linear\-width regime \(β=1\\beta=1\) and sublinear\-width regime \(β<1\\beta<1\)\.

\(a\)Vocabulary sized=103d=10^\{3\}and sample sizen=105n=10^\{5\}\.
\(b\)Vocabulary sized=104d=10^\{4\}and sample sizen=106n=10^\{6\}\.

Figure 1:Capacity scaling on WikiText\-2\.Converged normalized excess risk \(regret\)rn,d​\(N\)r\_\{n,d\}\(N\)\([7](https://arxiv.org/html/2610.19281#S2.E7)\) for the random\-features bigram model, as the embedding width is scaled asN=γ​dβN=\\gamma d^\{\\beta\}\(\([19](https://arxiv.org/html/2610.19281#S3.E19)\)\)\.Left:vary the width multiplierγ\\gammaat fixed exponentsβ\\beta\.Right:vary the exponentβ\\betaat fixed multipliersγ\\gamma\. Solid curves \(markers\) show empirical means over random projections; shaded regions indicate variability across trials\. Dashed curves are the theoretical predictions from our Theorem[2](https://arxiv.org/html/2610.19281#Thmtheorem2)\. Notice the excellent match\.\(a\)Vocabulary sized=103d=10^\{3\}and sample sizen=105n=10^\{5\}\.
\(b\)Vocabulary sized=104d=10^\{4\}and sample sizen=106n=10^\{6\}\.

Figure 2:Capacity scaling on OpenWebText2\.Converged normalized excess risk \(regret\)rn,d​\(N\)r\_\{n,d\}\(N\)\([7](https://arxiv.org/html/2610.19281#S2.E7)\) for the random\-features bigram model with a rank\-NNembedding bottleneck, whereN=γ​dβN=\\gamma d^\{\\beta\}\(\([19](https://arxiv.org/html/2610.19281#S3.E19)\)\)\.Left:varyγ\\gammaat fixedβ\\beta\.Right:varyβ\\betaat fixedγ\\gamma\. Solid curves \(markers\) show empirical means over random projections; shaded regions show trial variability\. Dashed curves are the corresponding theoretical predictions from our Theorem[2](https://arxiv.org/html/2610.19281#Thmtheorem2)\. Notice the excellent match\.
### 4\.2Datasets

We run experiments across three distinct datasets: Synthetic data generated exactly from Zipfian distributions, and the empirical real\-world corpora WikiText\-2\([Merity et al\., 2017](https://arxiv.org/html/2610.19281#bib.bib14)\)and OpenWebText2\([Gao et al\., 2020](https://arxiv.org/html/2610.19281#bib.bib15)\)\. For the real\-world datasets, the vocabulary sizeddand the token countnnare calculated over the whole dataset\. The Zipfian exponentα\\alphaare extracted through log\-log regression over the corpus distribution\. Table[1](https://arxiv.org/html/2610.19281#S2.T1)provides a list of these descriptive statistics\.

### 4\.3Results

The results across all configurations showcase an alignment between the empirical excess risk of the capacity\-constrained models and our theoretical predictions\. Despite the noise inherent in real\-world distributions, the derived asymptotics capture the empirical behavior across all datasets\.

WikiText\-2\.Figure[1\(a\)](https://arxiv.org/html/2610.19281#S4.F1.sf1)and[1\(b\)](https://arxiv.org/html/2610.19281#S4.F1.sf2)demonstrate the capacity scaling laws for the WikiText\-2 dataset\([Merity et al\., 2017](https://arxiv.org/html/2610.19281#bib.bib14)\), for different values of the vocabulary sizeddand embedding dimensionNN, for different values ofγ\\gammaandβ\\betain \([19](https://arxiv.org/html/2610.19281#S3.E19)\)\. Notice how excellently our theory predicts the performance of the model\. From the figures, we see that increasing capacityNNincreases the model capacity and reduces regret in both settings\. In the linear scaling regime \(β=1\\beta=1\), the model reaches zero regret atγ=1\\gamma=1, while in the sublinear scaling regime \(β<1\\beta<1\) it retain nonzero regret floor\.

#### OpenWebText2\.

Figure[2](https://arxiv.org/html/2610.19281#S4.F2)shows the same capacity scaling analysis on OpenWebText2\([Gao et al\., 2020](https://arxiv.org/html/2610.19281#bib.bib15)\)for two vocabulary cutoffs\. Despite the substantially noisier empirical frequency profile of a web\-scale corpus \(and the resulting deviations from an ideal Zipf law at the head and tail\), the predicted capacity–risk tradeoff remains remarkably accurate\.

Qualitatively, we again observe the two regimes predicted by Theorem[2](https://arxiv.org/html/2610.19281#Thmtheorem2)\. First, in the linear\-width regime \(β=1\\beta=1\), increasing the width multiplierγ\\gammaproduces a sharp decrease in regret, approaching the unconstrained optimum onceNNis on the order ofdd\. Second, in the sublinear regime \(β<1\\beta<1\), improvements withγ\\gammadiminish and the curves flatten, indicating that performance becomes variance\-limited rather than approximation\-limited\. This saturation is consistent with the finite\-sample term in \([13](https://arxiv.org/html/2610.19281#S3.E13)\), which scales withd/nd/nand therefore persists even asNNincreases whenNNgrows too slowly withdd\.

Beyond confirming the asymptotics on cleaner benchmarks \(WikiText\-2\) and synthetic data, OpenWebText2 provides evidence that the scaling\-law picture is robust to realistic corpus artifacts\. In particular, the close match between empirical curves and the theorem\-derived predictions suggests that the dominant driver of the observed scaling is the interaction between heavy\-tailed token frequencies and the explicit embedding bottleneck, rather than details of text domain or preprocessing\.

## 5Concluding Remark

Our results isolate representational capacity as a source of scaling, complementing explanations based on optimization dynamics\. Keeping vocabulary size explicit distinguishes increasing the capacity available for a fixed prediction problem from enlarging the problem itself\. This distinction is particularly relevant near the critical Zipf exponentα=1\\alpha=1, where vocabulary\-dependent normalization affects the scaling behavior\. Together, these perspectives compare empirically supported believes that heavy\-tailed language statistics may underlie both capacity\- and optimization\-driven scaling\([Kaplan et al\., 2020](https://arxiv.org/html/2610.19281#bib.bib1)\)\.

Our analytically tractable setting provides a quantitative benchmark for capacity\-driven scaling, while experiments on synthetic and natural language data support the predicted dependence on model width\. This framework opens the way to studying how learned embeddings, longer contexts, and finite optimization budgets modify these laws\. Such extensions would help identify which scaling behaviors reflect the underlying language statistics and which depend on architectural and training choices\.

#### Future Directions\.

A natural extension is to replace fixed random embeddings with learned representations and investigate how adaptation to predictive structure changes the capacity scaling\.

Also, extending the analysis to higher\-order token dependencies and heterogeneous transition distributions would further reveal how context length and richer language statistics shape these tradeoffs\.

### Acknowledgement

We would like to acknowledge that the first authorJulian Enochis a PhD student funded by bursaries from Canada Research Chair grant number CRC\-2023\-00228 \(*Computing for the Greater Good*\) awarded toElvis Dohmatob\.

### AI use statement

In this work, we have made limited use of generative AI tools for spell\-checking, and also reformulating chunks of text in the introduction \(Section[1](https://arxiv.org/html/2610.19281#S1), specially[1\.2](https://arxiv.org/html/2610.19281#S1.SS2)\)\. Also, AI was used to help with the tedious verification / sanity\-checking of some of our proofs \(Section[A](https://arxiv.org/html/2610.19281#A1)\)\. No other use of AI was made in our work\. The ideas and execution involved in this work is ours\. We take full responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of AI\.

## References

- Bahriet al\.\(2024\)Y\. Bahri, E\. Dyer, J\. Kaplan, J\. Lee, and U\. SharmaExplaining neural scaling laws\.Proceedings of the National Academy of Sciences121\(27\)\.Cited by:[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p5.1)\.
- Berend and Kontorovich \(2012\)D\. Berend and A\. KontorovichOn the convergence of the empirical distribution\.Note:arXiv preprintExternal Links:1205\.6711Cited by:[§3\.2](https://arxiv.org/html/2610.19281#S3.SS2.p2.4)\.
- Cabanneset al\.\(2024\)V\. Cabannes, E\. Dohmatob, and A\. BiettiScaling laws for associative memories\.InInternational Conference on Learning Representations,Cited by:[§1\.1](https://arxiv.org/html/2610.19281#S1.SS1.p2.1),[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p3.1),[§1](https://arxiv.org/html/2610.19281#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.19281#S2.SS1.p5.3),[§2](https://arxiv.org/html/2610.19281#S2.p2.1),[§3\.3](https://arxiv.org/html/2610.19281#S3.SS3.p2.1)\.
- Cuiet al\.\(2021\)H\. Cui, B\. Loureiro, F\. Krzakala, and L\. ZdeborováGeneralization error rates in kernel regression: the crossover from the noiseless to noisy regime\.InAdvances in Neural Information Processing Systems,Vol\.34\.Cited by:[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p4.1)\.
- Dębowski \(2023\)Ł\. DębowskiA simplistic model of neural scaling laws: multiperiodic Santa Fe processes\.External Links:2302\.09049v1Cited by:[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p6.1)\.
- Dohmatobet al\.\(2024\)E\. Dohmatob, Y\. Feng, P\. Yang, F\. Charton, and J\. KempeA tale of tails: model collapse as a change of scaling laws\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235\.Cited by:[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p7.1)\.
- Gaoet al\.\(2020\)L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. LeahyThe Pile: an 800GB dataset of diverse text for language modeling\.arXiv preprint arXiv:2101\.00027\.Cited by:[§1\.1](https://arxiv.org/html/2610.19281#S1.SS1.p5.1),[§2\.1](https://arxiv.org/html/2610.19281#S2.SS1.p2.1),[§3\.3](https://arxiv.org/html/2610.19281#S3.SS3.p3.1),[§4\.2](https://arxiv.org/html/2610.19281#S4.SS2.p1.1),[§4\.3](https://arxiv.org/html/2610.19281#S4.SS3.SSS0.Px1.p1.1)\.
- Hutter \(2021\)M\. HutterLearning curve theory\.External Links:2102\.04074Cited by:[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p2.1),[§2\.1](https://arxiv.org/html/2610.19281#S2.SS1.p5.3)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.External Links:2001\.08361Cited by:[§1\.1](https://arxiv.org/html/2610.19281#S1.SS1.p3.1),[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p1.1),[§1](https://arxiv.org/html/2610.19281#S1.p1.1),[§3\.3](https://arxiv.org/html/2610.19281#S3.SS3.p4.1),[§4\.1](https://arxiv.org/html/2610.19281#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.19281#S5.p1.1)\.
- Kunstner and Bach \(2025\)F\. Kunstner and F\. BachScaling laws for gradient descent and sign descent for linear bigram models under Zipf’s law\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[§1\.1](https://arxiv.org/html/2610.19281#S1.SS1.p2.1),[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p2.1),[§1](https://arxiv.org/html/2610.19281#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.19281#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.19281#S2.SS1.p2.1),[§2\.1](https://arxiv.org/html/2610.19281#S2.SS1.p5.1),[§2\.2](https://arxiv.org/html/2610.19281#S2.SS2.p1.1),[§2](https://arxiv.org/html/2610.19281#S2.p1.1),[§2](https://arxiv.org/html/2610.19281#S2.p2.1)\.
- Lyuet al\.\(2025\)B\. Lyu, D\. Wang, and Z\. ZhuA solvable attention for neural scaling laws\.InInternational Conference on Learning Representations,Cited by:[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p8.1)\.
- Maloneyet al\.\(2022\)A\. Maloney, D\. A\. Roberts, and J\. SullyA solvable model of neural scaling laws\.External Links:2210\.16859Cited by:[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p4.1),[§1](https://arxiv.org/html/2610.19281#S1.p1.1)\.
- Merityet al\.\(2017\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. SocherPointer sentinel mixture models\.InInternational Conference on Learning Representations,Cited by:[§1\.1](https://arxiv.org/html/2610.19281#S1.SS1.p5.1),[§2\.1](https://arxiv.org/html/2610.19281#S2.SS1.p2.1),[§3\.3](https://arxiv.org/html/2610.19281#S3.SS3.p3.1),[§4\.2](https://arxiv.org/html/2610.19281#S4.SS2.p1.1),[§4\.3](https://arxiv.org/html/2610.19281#S4.SS3.p2.1)\.
- Michaudet al\.\(2023\)E\. Michaud, Z\. Liu, U\. Girit, and M\. TegmarkThe quantization model of neural scaling\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p6.1)\.
- Namet al\.\(2024\)Y\. Nam, N\. Fonseca, S\. H\. Lee, C\. Mingard, and A\. A\. LouisAn exactly solvable model for emergence and scaling laws in the multitask sparse parity problem\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p6.1)\.
- Nguyen \(2024\)T\. NguyenUnderstanding transformers via N\-gram statistics\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p8.1)\.
- Shumailovet al\.\(2024\)I\. Shumailov, Z\. Shumaylov, Y\. Zhao, Y\. Gal, N\. Papernot, and R\. AndersonAI models collapse when trained on recursively generated data\.Nature631,pp\. 755–759\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07566-y)Cited by:[§1\.2](https://arxiv.org/html/2610.19281#S1.SS2.p7.1)\.

Appendix

## Appendix AProof of Main Results

### A\.1Proof of Proposition[1](https://arxiv.org/html/2610.19281#Thmtheorem1)

As in the proposition, defineck:=\#⁡\{t∈\[n\]∣xt=ek\}c\_\{k\}:=\\\#\\\{t\\in\[n\]\\mid x\_\{t\}=e\_\{k\}\\\},I=\{k∈\[d\]∣ck\>0\}I=\\\{k\\in\[d\]\\mid c\_\{k\}\>0\\\}, andS:=\|I\|S:=\|I\|, wherextx\_\{t\}is thettth row of the design matrixX∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}\. For any candidate student modelW∈ℝd×dW\\in\\mathbb\{R\}^\{d\\times d\}, one can write the loss as

L⁡\(W\):=1n​‖X​W−Y‖F2=1n​∑k∈I∑xt=ek‖Wk−yt‖22\.\\displaystyle L\(W\):=\\frac\{1\}\{n\}\\\|XW\-Y\\\|\_\{F\}^\{2\}=\\frac\{1\}\{n\}\\sum\_\{k\\in I\}\\sum\_\{x\_\{t\}=e\_\{k\}\}\\\|W\_\{k\}\-y\_\{t\}\\\|\_\{2\}^\{2\}\.\(21\)
#### Step 1: A Formula for Regret of any Student\.

Define anS×dS\\times dmatrixMMby setting

Mk:=1ck​∑xt=ekyt,for all​k∈I\.\\displaystyle M\_\{k\}:=\\frac\{1\}\{c\_\{k\}\}\\sum\_\{x\_\{t\}=e\_\{k\}\}y\_\{t\},\\text\{ for all \}k\\in I\.\(22\)Thek′k^\{\\prime\}th entry ofMkM\_\{k\}is nothing but the \(empirical\) probability of observing thek′k^\{\\prime\}th token immediately after thekkth token\. We can writeWk−yt=\(Wk−Mk\)\+\(Mk−yt\)W\_\{k\}\-y\_\{t\}=\(W\_\{k\}\-M\_\{k\}\)\+\(M\_\{k\}\-y\_\{t\}\), and so

∑xt=ek‖Wk−yt‖22=ck​‖Wk−Mk‖22\+∑xt=ek‖Mk−yt‖22\+2​\(Wk−Mk\)⊤​∑xt=ek\(Mk−yt\)\.\\sum\_\{x\_\{t\}=e\_\{k\}\}\\\|W\_\{k\}\-y\_\{t\}\\\|\_\{2\}^\{2\}=c\_\{k\}\\\|W\_\{k\}\-M\_\{k\}\\\|\_\{2\}^\{2\}\+\\sum\_\{x\_\{t\}=e\_\{k\}\}\\\|M\_\{k\}\-y\_\{t\}\\\|\_\{2\}^\{2\}\+2\(W\_\{k\}\-M\_\{k\}\)^\{\\top\}\\sum\_\{x\_\{t\}=e\_\{k\}\}\(M\_\{k\}\-y\_\{t\}\)\.But,∑xt=ek\(Mk−yt\)=0\\sum\_\{x\_\{t\}=e\_\{k\}\}\(M\_\{k\}\-y\_\{t\}\)=0by definition ofMkM\_\{k\}\. We deduce that

L⁡\(W\)=1n​∑k∈Ick​‖Wk−Mk‖22\+1n​∑k∈I∑xt=ek‖Mk−yt‖22\.\\displaystyle L\(W\)=\\frac\{1\}\{n\}\\sum\_\{k\\in I\}c\_\{k\}\\\|W\_\{k\}\-M\_\{k\}\\\|\_\{2\}^\{2\}\+\\frac\{1\}\{n\}\\sum\_\{k\\in I\}\\sum\_\{x\_\{t\}=e\_\{k\}\}\\\|M\_\{k\}\-y\_\{t\}\\\|\_\{2\}^\{2\}\.Since an unconstrained minimizerW∗W\_\{\*\}satisfiesW∗,k=MkW\_\{\*,k\}=M\_\{k\}for allk∈Ik\\in I, we deduce that the regret of anyWWis given by the following exact formula:

R⁡\(W\):=L⁡\(W\)−L⁡\(W∗\)=1n​∑k∈Ick​‖Wk−Mk‖22\.\\displaystyle R\(W\):=L\(W\)\-L\(W\_\{\*\}\)=\\frac\{1\}\{n\}\\sum\_\{k\\in I\}c\_\{k\}\\\|W\_\{k\}\-M\_\{k\}\\\|\_\{2\}^\{2\}\.\(23\)

#### Step 2: Representation of the Optimal Student and Its Regret\.

We now restrict to students of the formW=Ω​ΘW=\\Omega\\Thetafor someΘ∈ℝN×d\\Theta\\in\\mathbb\{R\}^\{N\\times d\}\. LetΩI∈ℝS×d\\Omega\_\{I\}\\in\\mathbb\{R\}^\{S\\times d\}be the restriction ofΩ\\Omegato the rows corresponding to observed tokensk∈Ik\\in I\. Define

Dc:=diag⁡\(ck∣k∈I\),Ac:=Dc1/2​ΩI,Bc:=Dc1/2​M\.D\_\{c\}:=\\operatorname\{diag\}\(c\_\{k\}\\mid k\\in I\),\\quad A\_\{c\}:=D\_\{c\}^\{1/2\}\\Omega\_\{I\},\\quad B\_\{c\}:=D\_\{c\}^\{1/2\}M\.Then, the constrained estimator isW^=Ω​Θ^\\hat\{W\}=\\Omega\\hat\{\\Theta\}, whereΘ^\\hat\{\\Theta\}minimizes

1n​∑k∈Ick​‖\(Ω​Θ\)k−Mk‖22=1n​‖Ac​Θ−Bc‖F2\.\\frac\{1\}\{n\}\\sum\_\{k\\in I\}c\_\{k\}\\\|\(\\Omega\\Theta\)\_\{k\}\-M\_\{k\}\\\|\_\{2\}^\{2\}=\\frac\{1\}\{n\}\\\|A\_\{c\}\\Theta\-B\_\{c\}\\\|\_\{F\}^\{2\}\.The normal equations forΘ^\\hat\{\\Theta\}areAc⊤​Ac​Θ^=Ac⊤​BcA\_\{c\}^\{\\top\}A\_\{c\}\\hat\{\\Theta\}=A\_\{c\}^\{\\top\}B\_\{c\}\. This gives

Ac​Θ^=PΩI​Bc,with​PΩI:=Ac​\(Ac⊤​Ac\)†​Ac⊤=Dc1/2​ΩI​\(ΩI⊤​Dc​ΩI\)†​ΩI⊤​Dc1/2\.A\_\{c\}\\hat\{\\Theta\}=P\_\{\\Omega\_\{I\}\}B\_\{c\},\\text\{ with \}P\_\{\\Omega\_\{I\}\}:=A\_\{c\}\(A\_\{c\}^\{\\top\}A\_\{c\}\)^\{\\dagger\}A\_\{c\}^\{\\top\}=D\_\{c\}^\{1/2\}\\Omega\_\{I\}\(\\Omega\_\{I\}^\{\\top\}D\_\{c\}\\Omega\_\{I\}\)^\{\\dagger\}\\Omega\_\{I\}^\{\\top\}D\_\{c\}^\{1/2\}\.Note thatPΩIP\_\{\\Omega\_\{I\}\}is the orthogonal projection matrix onto the column space ofAc=Dc1/2​ΩIA\_\{c\}=D\_\{c\}^\{1/2\}\\Omega\_\{I\}\. DefiningPΩI⟂:=I−PΩIP\_\{\\Omega\_\{I\}\}^\{\\perp\}:=\\operatorname\{I\}\-P\_\{\\Omega\_\{I\}\}, one has

R⁡\(W\)=1n​‖PΩI⟂​Bc‖F2=1n​tr⁡Bc⊤​PΩI⟂​Bc=1n​tr⁡\(IS−PΩI\)​Bc​Bc⊤\.R\(W\)=\\frac\{1\}\{n\}\\\|P\_\{\\Omega\_\{I\}\}^\{\\perp\}B\_\{c\}\\\|\_\{F\}^\{2\}=\\frac\{1\}\{n\}\\operatorname\{tr\}B\_\{c\}^\{\\top\}P\_\{\\Omega\_\{I\}\}^\{\\perp\}B\_\{c\}=\\frac\{1\}\{n\}\\operatorname\{tr\}\(\\operatorname\{I\}\_\{S\}\-P\_\{\\Omega\_\{I\}\}\)B\_\{c\}B\_\{c\}^\{\\top\}\.We now marginalize ofΩ\\Omega\. We shall show that𝔼Y,Ω​\[PΩI\]\\mathbb\{E\}\_\{Y,\\Omega\}\[P\_\{\\Omega\_\{I\}\}\]is diagonal\. To this end, define

hk:=𝔼Y,Ω​\[\(PΩI\)k,k\]\.h\_\{k\}:=\\mathbb\{E\}\_\{Y,\\Omega\}\[\(P\_\{\\Omega\_\{I\}\}\)\_\{k,k\}\]\.These scalars should be reminiscent of*leverage scores*in sketching methods\. LetJ=diag⁡\(ε1,…,εS\)J=\\operatorname\{diag\}\(\\varepsilon\_\{1\},\\ldots,\\varepsilon\_\{S\}\), withε1,…,εS∈\{−1,1\}\\varepsilon\_\{1\},\\ldots,\\varepsilon\_\{S\}\\in\\\{\-1,1\\\}\. SinceJJcommutes withDcD\_\{c\}\(both being diagonal matrices\), one hasJ​Dc​J=J2​Dc=DcJD\_\{c\}J=J^\{2\}D\_\{c\}=D\_\{c\}and so

PJ​ΩI=Dc1/2​J​ΩI​\(ΩI⊤​J​Dc​J​ΩI\)†​ΩI⊤​J​Dc1/2=J​Dc1/2​ΩI​\(ΩI⊤​Dc​ΩI\)†​ΩI⊤​Dc1/2​J=J​PΩI​J\.P\_\{J\\Omega\_\{I\}\}=D\_\{c\}^\{1/2\}J\\Omega\_\{I\}\(\\Omega\_\{I\}^\{\\top\}JD\_\{c\}J\\Omega\_\{I\}\)^\{\\dagger\}\\Omega\_\{I\}^\{\\top\}JD\_\{c\}^\{1/2\}=JD\_\{c\}^\{1/2\}\\Omega\_\{I\}\(\\Omega\_\{I\}^\{\\top\}D\_\{c\}\\Omega\_\{I\}\)^\{\\dagger\}\\Omega\_\{I\}^\{\\top\}D\_\{c\}^\{1/2\}J=JP\_\{\\Omega\_\{I\}\}J\.On the other hand, due to the rotational invariance of the distribution ofΩ\\Omega, we know thatJ​ΩI=JJ\\Omega\_\{I\}=Jin distribution, and so

𝔼Y,Ω​\[PΩI\]=𝔼Y,Ω​\[PJ​ΩI\]=J​𝔼Y,Ω​\[PΩI\]​J\.\\mathbb\{E\}\_\{Y,\\Omega\}\[P\_\{\\Omega\_\{I\}\}\]=\\mathbb\{E\}\_\{Y,\\Omega\}\[P\_\{J\\Omega\_\{I\}\}\]=J\\,\\mathbb\{E\}\_\{Y,\\Omega\}\[P\_\{\\Omega\_\{I\}\}\]J\.The above identity holds for any diagonal matrix with diagonal entries±1\\pm 1\. For anyk≠k′k\\neq k^\{\\prime\}, chooseJJso that it flips the the signs of thekkth but not thek′k^\{\\prime\}th row ofPΩIP\_\{\\Omega\_\{I\}\}\. Then,\(𝔼Y,Ω​\[PΩI\]\)i​j=\(J​𝔼Y,Ω​\[PΩI\]​J\)i​j=−\(𝔼Y,Ω​\[PΩI\]\)i​j\(\\mathbb\{E\}\_\{Y,\\Omega\}\[P\_\{\\Omega\_\{I\}\}\]\)\_\{ij\}=\(J\\,\\mathbb\{E\}\_\{Y,\\Omega\}\[P\_\{\\Omega\_\{I\}\}\]J\)\_\{ij\}=\-\(\\mathbb\{E\}\_\{Y,\\Omega\}\[P\_\{\\Omega\_\{I\}\}\]\)\_\{ij\}, and so\(𝔼Y,Ω​\[PΩI\]\)i​j=0\(\\mathbb\{E\}\_\{Y,\\Omega\}\[P\_\{\\Omega\_\{I\}\}\]\)\_\{ij\}=0\. We conclude that

𝔼Ω​\[PΩI\]=diag⁡\(hk∣k∈I\)\.\\displaystyle\\mathbb\{E\}\_\{\\Omega\}\[P\_\{\\Omega\_\{I\}\}\]=\\operatorname\{diag\}\(h\_\{k\}\\mid k\\in I\)\.\(24\)This in turn gives

𝔼Y,Ω​\[rn,d​\(N\)\]=1n​tr⁡\(IS−𝔼Y,Ω​\[PΩI\]\)​Dc1/2​M​M⊤​Dc1/2=1n​∑k∈Ick​‖Mk‖22​\(1−hk\)\.\\mathbb\{E\}\_\{Y,\\Omega\}\[r\_\{n,d\}\(N\)\]=\\frac\{1\}\{n\}\\operatorname\{tr\}\(\\operatorname\{I\}\_\{S\}\-\\mathbb\{E\}\_\{Y,\\Omega\}\[P\_\{\\Omega\_\{I\}\}\]\)D\_\{c\}^\{1/2\}MM^\{\\top\}D\_\{c\}^\{1/2\}=\\frac\{1\}\{n\}\\sum\_\{k\\in I\}c\_\{k\}\\\|M\_\{k\}\\\|\_\{2\}^\{2\}\\,\(1\-h\_\{k\}\)\.Now, one computes

‖Mk‖2\\displaystyle\\\|M\_\{k\}\\\|^\{2\}=1ck2​\(∑t\|xt=ek𝔼⁡\[‖yt‖2\]\+∑t′≠t,xt′=xt=ek𝔼⁡\[yt′⊤​yt\]\)=1ck2​\(ck\+ck​\(ck−1\)​ak\)\\displaystyle=\\frac\{1\}\{c\_\{k\}^\{2\}\}\(\\sum\_\{t\|x\_\{t\}=e\_\{k\}\}\\mathbb\{E\}\[\\\|y\_\{t\}\\\|^\{2\}\]\+\\sum\_\{t^\{\\prime\}\\neq t,\\,x\_\{t^\{\\prime\}\}=x\_\{t\}=e\_\{k\}\}\\mathbb\{E\}\[y\_\{t^\{\\prime\}\}^\{\\top\}y\_\{t\}\]\)=\\frac\{1\}\{c\_\{k\}^\{2\}\}\(c\_\{k\}\+c\_\{k\}\(c\_\{k\}\-1\)a\_\{k\}\)=ak\+1−akck,\\displaystyle=a\_\{k\}\+\\frac\{1\-a\_\{k\}\}\{c\_\{k\}\},where we have used the fact that‖yt‖2=1\\\|y\_\{t\}\\\|^\{2\}=1for allttbecause theyty\_\{t\}’s are on\-hot encodings, and𝔼⁡\[yt′⊤​yt\]=𝔼​\[yt′\]⊤​𝔼​\[yt\]=ak:=‖𝔼⁡\[y∣x=ek\]‖2\\mathbb\{E\}\[y\_\{t^\{\\prime\}\}^\{\\top\}y\_\{t\}\]=\\mathbb\{E\}\[y\_\{t^\{\\prime\}\}\]^\{\\top\}\\mathbb\{E\}\[y\_\{t\}\]=a\_\{k\}:=\\\|\\mathbb\{E\}\[y\\mid x=e\_\{k\}\]\\\|^\{2\}fort′≠tt^\{\\prime\}\\neq tdue to independence\. under Assumption[1](https://arxiv.org/html/2610.19281#S2.EGx2), we haveak≡Aa\_\{k\}\\equiv A, and so

𝔼Y,Ω​\[rn,d​\(N\)∣c\]=∑k∈I‖Mk‖22​\(1−hk\)=An​∑k∈Ick​\(1−hk\)\+1−An​∑k∈I1−hk\.\\displaystyle\\mathbb\{E\}\_\{Y,\\Omega\}\[r\_\{n,d\}\(N\)\\mid c\]=\\sum\_\{k\\in I\}\\\|M\_\{k\}\\\|\_\{2\}^\{2\}\\,\(1\-h\_\{k\}\)=\\frac\{A\}\{n\}\\sum\_\{k\\in I\}c\_\{k\}\(1\-h\_\{k\}\)\+\\frac\{1\-A\}\{n\}\\sum\_\{k\\in I\}1\-h\_\{k\}\.Now,∑k∈Ihk\\sum\_\{k\\in I\}h\_\{k\}equals the rank ofPΩIP\_\{\\Omega\_\{I\}\}which ismin⁡\(S,N\)\\min\(S,N\)a\.s\. We deduce that∑k∈I\(1−hk\)=S−min⁡\(S,N\)=\(S−N\)\+\\sum\_\{k\\in I\}\(1\-h\_\{k\}\)=S\-\\min\(S,N\)=\(S\-N\)\_\{\+\}, and so

𝔼Y,Ω​\[rn,d​\(N\)\]=An​𝔼​∑k∈Ick​\(1−hk\)\+\(1−A\)​𝔼​\(S−N\)\+n\\mathbb\{E\}\_\{Y,\\Omega\}\[r\_\{n,d\}\(N\)\]=\\frac\{A\}\{n\}\\mathbb\{E\}\\sum\_\{k\\in I\}c\_\{k\}\\,\(1\-h\_\{k\}\)\+\(1\-A\)\\mathbb\{E\}\\frac\{\(S\-N\)\_\{\+\}\}\{n\}

#### Step 3: Random Matrix Theory\.

We now produce a deterministic equivalent for the random quantity∑k∈Ick​\(1−hk\)\\sum\_\{k\\in I\}c\_\{k\}\\,\(1\-h\_\{k\}\)\. Observe that ifN<S:=\|I\|N<S:=\|I\|, then the rank ofΩI\\Omega\_\{I\}isSS;G:=ΩI′​ΩIG:=\\Omega\_\{I\}^\{\\prime\}\\Omega\_\{I\}has full rankNN, and is therefore invertible a\.s\. Thus, for anyk∈Ik\\in I, one hashk=ck​wk′​G−1​wkh\_\{k\}=c\_\{k\}w\_\{k\}^\{\\prime\}G^\{\-1\}w\_\{k\}, wherewkw\_\{k\}is row ofΩ\\Omegacorresponding to thekkth token\.

We now use a leave\-one\-out \(LOO\) argument\. Decompose the covariance matrixG=ck​wk​wk⊤\+G−kG=c\_\{k\}w\_\{k\}w\_\{k\}^\{\\top\}\+G\_\{\-k\}, whereG−k:=∑ℓ≠kcℓ​wℓ​wℓ⊤G\_\{\-k\}:=\\sum\_\{\\ell\\neq k\}c\_\{\\ell\}w\_\{\\ell\}w\_\{\\ell\}^\{\\top\}\. The Sherman\-Morrison formula gives

G−1=G−k−1−ck1\+ck​wk⊤​G−k−1​wk​G−k−1​wk​wk⊤​G−k−1\.\\displaystyle G^\{\-1\}=G\_\{\-k\}^\{\-1\}\-\\frac\{c\_\{k\}\}\{1\+c\_\{k\}w\_\{k\}^\{\\top\}G\_\{\-k\}^\{\-1\}w\_\{k\}\}G\_\{\-k\}^\{\-1\}w\_\{k\}w\_\{k\}^\{\\top\}G\_\{\-k\}^\{\-1\}\.Plugging into the previous expression forℓi\\ell\_\{i\}and simplifying gives

1−hk=11\+ck​wk⊤​G−k−1​wk=11\+\(ck/k\)​tr⁡G−k−1\+o⁡\(1\)\.\\displaystyle 1\-h\_\{k\}=\\frac\{1\}\{1\+c\_\{k\}w\_\{k\}^\{\\top\}G\_\{\-k\}^\{\-1\}w\_\{k\}\}=\\frac\{1\}\{1\+\(c\_\{k\}/k\)\\operatorname\{tr\}G\_\{\-k\}^\{\-1\}\}\+o\(1\)\.The last step is becausewkw\_\{k\}andG−kG\_\{\-k\}are independent, so thatwk⊤​G−k−1​wk=d−1​tr⁡G−k−1\+o⁡\(1\)w\_\{k\}^\{\\top\}G\_\{\-k\}^\{\-1\}w\_\{k\}=d^\{\-1\}\\operatorname\{tr\}G\_\{\-k\}^\{\-1\}\+o\(1\)\. But\(1/d\)​tr⁡G−k−1=\(1\+o⁡\(1\)\)​d−1​tr⁡G−1\(1/d\)\\operatorname\{tr\}G\_\{\-k\}^\{\-1\}=\(1\+o\(1\)\)d^\{\-1\}\\operatorname\{tr\}G^\{\-1\}\(we have only removed one entry\), and so we get

1−hk=11\+ck​\(1/d\)​tr⁡G−1\+o⁡\(1\)\.\\displaystyle 1\-h\_\{k\}=\\frac\{1\}\{1\+c\_\{k\}\\,\(1/d\)\\operatorname\{tr\}G^\{\-1\}\}\+o\(1\)\.
Now, by classical random matrix theory \(RMT\), it holds thatd−1​tr⁡\(G−z​IN\)−1→m⁡\(z\)d^\{\-1\}\\operatorname\{tr\}\(G\-z\\mathrm\{I\}\_\{N\}\)^\{\-1\}\\to m\(z\)for anyz∈ℂ∖ℝ\+z\\in\\mathbb\{C\}\\setminus\\mathbb\{R\}\_\{\+\}, where the limiting Stieltjes transformmmsatisfies the fixed\-point equation

1m=−z\+1N​∑k∈Ick1\+ck​m\.\\displaystyle\\frac\{1\}\{m\}=\-z\+\\frac\{1\}\{N\}\\sum\_\{k\\in I\}\\frac\{c\_\{k\}\}\{1\+c\_\{k\}m\}\.\(25\)Definingκ=κ⁡\(z\):=1/m⁡\(z\)\\kappa=\\kappa\(z\):=1/m\(z\), we have the equivalent formulation

1N​∑k∈Ici​μci\+μ=μ\+z\.\\displaystyle\\frac\{1\}\{N\}\\sum\_\{k\\in I\}\\frac\{c\_\{i\}\\mu\}\{c\_\{i\}\+\\mu\}=\\mu\+z\.\(26\)In particular sinceN<SN<S, takingz→0z\\to 0withinℂ∖ℝ\+\\mathbb\{C\}\\setminus\\mathbb\{R\}\_\{\+\}, we get thatμ\\musatisfies the following fixed\-point equation

N=∑k∈Ickck\+μ\.\\displaystyle N=\\sum\_\{k\\in I\}\\frac\{c\_\{k\}\}\{c\_\{k\}\+\\mu\}\.This gives,hk≃ck/\(ck\+μ\)h\_\{k\}\\simeq c\_\{k\}/\(c\_\{k\}\+\\mu\),1−hk≃μ/\(ck\+μ\)1\-h\_\{k\}\\simeq\\mu/\(c\_\{k\}\+\\mu\)for allk∈Ik\\in I, and so

∑k∈Ick​ak​\(1−hk\)≃A​μ​∑k∈Ickck\+μ=A​N​μ=A​N​n​ν\.\\sum\_\{k\\in I\}c\_\{k\}a\_\{k\}\\,\(1\-h\_\{k\}\)\\simeq A\\mu\\sum\_\{k\\in I\}\\frac\{c\_\{k\}\}\{c\_\{k\}\+\\mu\}=AN\\mu=ANn\\nu\.which completes the proof of Proposition[1](https://arxiv.org/html/2610.19281#Thmtheorem1)\. ∎

### A\.2Proof of Theorem[2](https://arxiv.org/html/2610.19281#Thmtheorem2)

Proposition[1](https://arxiv.org/html/2610.19281#Thmtheorem1)already separates the contributions of limited representation and finite sampling\. The remaining task is to replace the empirical frequencies by their population counterparts and understand how the resulting fixed point depends on the Zipf exponent\.

To distinguish the two fixed points, writeν^\\widehat\{\\nu\}for the sample\-dependent quantity in Proposition[1](https://arxiv.org/html/2610.19281#Thmtheorem1), and letν\>0\\nu\>0solve

N=∑i=1dpipi\+ν,pi=i−αHd,α\.N=\\sum\_\{i=1\}^\{d\}\\frac\{p\_\{i\}\}\{p\_\{i\}\+\\nu\},\\quad p\_\{i\}=\\frac\{i^\{\-\\alpha\}\}\{H\_\{d,\\alpha\}\}\.\(27\)For0<N<d0<N<d, existence and uniqueness follow because the right\-hand side is continuous and strictly decreasing fromddto zero\. Throughout the proof,α\\alphais fixed,d,N→∞d,N\\to\\infty, and eitherN=o⁡\(d\)N=o\(d\)orN/d→ψ0∈\(0,1\)N/d\\to\\psi\_\{0\}\\in\(0,1\)\.

We first analyze the population fixed point\. Introduce

ψ:=Nd,ρd:=1Hd,α​dα​ν\.\\psi:=\\frac\{N\}\{d\},\\quad\\rho\_\{d\}:=\\frac\{1\}\{H\_\{d,\\alpha\}d^\{\\alpha\}\\nu\}\.Then \([27](https://arxiv.org/html/2610.19281#A1.E27)\) becomes

ψ=1d​∑i=1dρdρd\+\(i/d\)α\.\\psi=\\frac\{1\}\{d\}\\sum\_\{i=1\}^\{d\}\\frac\{\\rho\_\{d\}\}\{\\rho\_\{d\}\+\(i/d\)^\{\\alpha\}\}\.The useful observation is that this sum is a Riemann approximation to the scalar function

Gα​\(ρ\):=∫01ρρ\+tα​𝑑t\.G\_\{\\alpha\}\(\\rho\):=\\int\_\{0\}^\{1\}\\frac\{\\rho\}\{\\rho\+t^\{\\alpha\}\}\\,\\mathrm\{d\}t\.Since the integrand is nonincreasing intt,

0≤Gα​\(ρd\)−ψ≤1d,Gα​\(ρd\)=ψ⁡\(1\+O⁡\(N−1\)\)\.0\\leq G\_\{\\alpha\}\(\\rho\_\{d\}\)\-\\psi\\leq\\frac\{1\}\{d\},\\quad G\_\{\\alpha\}\(\\rho\_\{d\}\)=\\psi\\bigl\(1\+O\(N^\{\-1\}\)\\bigr\)\.\(28\)Meanwhile, the bias contribution has the exact representation

B=A​N​ν=A​d1−αHd,α​ψρd\.B=AN\\nu=A\\frac\{d^\{1\-\\alpha\}\}\{H\_\{d,\\alpha\}\}\\frac\{\\psi\}\{\\rho\_\{d\}\}\.\(29\)Thus all three scaling regimes follow from the behavior ofGαG\_\{\\alpha\}and the normalizationHd,αH\_\{d,\\alpha\}\.

When the width is proportional to the vocabulary size,ψ→ψ0∈\(0,1\)\\psi\\to\\psi\_\{0\}\\in\(0,1\), continuity and strict monotonicity imply

ρd∼ρ⁡\(α,ψ\),Gα​\(ρ⁡\(α,ψ\)\)=ψ\.\\rho\_\{d\}\\sim\\rho\(\\alpha,\\psi\),\\quad G\_\{\\alpha\}\\bigl\(\\rho\(\\alpha,\\psi\)\\bigr\)=\\psi\.For sublinear width,ψ→0\\psi\\to 0, and \([28](https://arxiv.org/html/2610.19281#A1.E28)\) instead reduces the problem to the behavior ofGα​\(ρ\)G\_\{\\alpha\}\(\\rho\)asρ↓0\\rho\\downarrow 0\.

For a subcritical exponent0≤α<10\\leq\\alpha<1, the functiont−αt^\{\-\\alpha\}is integrable on\(0,1\)\(0,1\)\. Dominated convergence therefore gives

Gα​\(ρ\)ρ=∫01d​tρ\+tα⟶11−α\.\\frac\{G\_\{\\alpha\}\(\\rho\)\}\{\\rho\}=\\int\_\{0\}^\{1\}\\frac\{\\mathrm\{d\}t\}\{\\rho\+t^\{\\alpha\}\}\\longrightarrow\\frac\{1\}\{1\-\\alpha\}\.Together withHd,α∼d1−α/\(1−α\)H\_\{d,\\alpha\}\\sim d^\{1\-\\alpha\}/\(1\-\\alpha\), this immediately yields both subcritical laws\. Indeed, ifN=o⁡\(d\)N=o\(d\), thenρd∼\(1−α\)​ψ\\rho\_\{d\}\\sim\(1\-\\alpha\)\\psi, and henceB∼AB\\sim A\. For proportional width, the integral fixed point remains nontrivial, giving

B∼A⁡\(1−α\)​ψρ⁡\(α,ψ\)\.B\\sim A\(1\-\\alpha\)\\frac\{\\psi\}\{\\rho\(\\alpha,\\psi\)\}\.
At the critical exponentα=1\\alpha=1, the integral can be evaluated explicitly:

G1​\(ρ\)=ρ​log⁡\(1\+1ρ\)\.G\_\{1\}\(\\rho\)=\\rho\\log\\left\(1\+\\frac\{1\}\{\\rho\}\\right\)\.\(30\)ForN=o⁡\(d\)N=o\(d\), equations \([28](https://arxiv.org/html/2610.19281#A1.E28)\) and \([30](https://arxiv.org/html/2610.19281#A1.E30)\) give

ψ∼ρd​log⁡\(1/ρd\)\.\\psi\\sim\\rho\_\{d\}\\log\(1/\\rho\_\{d\}\)\.The logarithmic factor changes more slowly thanρd\\rho\_\{d\}, so

log⁡\(1/ψ\)=log⁡\(1/ρd\)−log⁡log⁡\(1/ρd\)\+o⁡\(1\)∼log⁡\(1/ρd\)\.\\log\(1/\\psi\)=\\log\(1/\\rho\_\{d\}\)\-\\log\\log\(1/\\rho\_\{d\}\)\+o\(1\)\\sim\\log\(1/\\rho\_\{d\}\)\.Consequently,

ψρd∼log⁡\(1/ψ\)=log⁡\(d/N\)\.\\frac\{\\psi\}\{\\rho\_\{d\}\}\\sim\\log\(1/\\psi\)=\\log\(d/N\)\.UsingHd,1∼log⁡dH\_\{d,1\}\\sim\\log din \([29](https://arxiv.org/html/2610.19281#A1.E29)\), we obtain

B∼Alog⁡d​log⁡\(dN\)\.B\\sim\\frac\{A\}\{\\log d\}\\log\\left\(\\frac\{d\}\{N\}\\right\)\.
For proportional width, it is convenient to return to the discrete fixed point and set

q:=1Hd,1​ν\.q:=\\frac\{1\}\{H\_\{d,1\}\\nu\}\.This gives

N=∑i=1dqq\+i,B=A​NHd,1​q∼Alog⁡d​Nq\.N=\\sum\_\{i=1\}^\{d\}\\frac\{q\}\{q\+i\},\\quad B=\\frac\{AN\}\{H\_\{d,1\}q\}\\sim\\frac\{A\}\{\\log d\}\\frac\{N\}\{q\}\.In particular, the factor associated with this definition ofqqisN/qN/q\. An explicit expression is obtained by writingb:=ψ/ρb:=\\psi/\\rho, whereρ=ρ⁡\(1,ψ\)\\rho=\\rho\(1,\\psi\)\. The integral equation then reads

b=log⁡\(1\+bψ\),b=\\log\\left\(1\+\\frac\{b\}\{\\psi\}\\right\),whose positive solution is

b=−𝒲−1​\(−ψ​e−ψ\)−ψ\.b=\-\\mathcal\{W\}\_\{\-1\}\(\-\\psi e^\{\-\\psi\}\)\-\\psi\.Thus, in the proportional\-width regime,

B∼Alog⁡d​\[−𝒲−1​\(−ψ​e−ψ\)−ψ\]\.B\\sim\\frac\{A\}\{\\log d\}\\left\[\-\\mathcal\{W\}\_\{\-1\}\(\-\\psi e^\{\-\\psi\}\)\-\\psi\\right\]\.
The supercritical caseα\>1\\alpha\>1has a different small\-ρ\\rhobehavior\. Substitutingt=ρ1/α​ut=\\rho^\{1/\\alpha\}ugives

Gα\(ρ\)=ρ1/α∫0ρ−1/αd​u1\+uα\.G\_\{\\alpha\}\(\\rho\)=\\rho^\{1/\\alpha\}\\int\_\{0\}^\{\\rho^\{\-1/\\alpha\}\}\\frac\{\\mathrm\{d\}u\}\{1\+u^\{\\alpha\}\}\.The integral now converges asρ↓0\\rho\\downarrow 0, with

Iα:=∫0∞d​u1\+uα=πα​sin⁡\(π/α\)\.I\_\{\\alpha\}:=\\int\_\{0\}^\{\\infty\}\\frac\{\\mathrm\{d\}u\}\{1\+u^\{\\alpha\}\}=\\frac\{\\pi\}\{\\alpha\\sin\(\\pi/\\alpha\)\}\.Hence, forN=o⁡\(d\)N=o\(d\),

ψ∼Iα​ρd1/α,ρd∼\(ψIα\)α\.\\psi\\sim I\_\{\\alpha\}\\rho\_\{d\}^\{1/\\alpha\},\\quad\\rho\_\{d\}\\sim\\left\(\\frac\{\\psi\}\{I\_\{\\alpha\}\}\\right\)^\{\\alpha\}\.SinceHd,α→ζ⁡\(α\)H\_\{d,\\alpha\}\\to\\zeta\(\\alpha\), equation \([29](https://arxiv.org/html/2610.19281#A1.E29)\) becomes

B\\displaystyle B∼A​Iααζ⁡\(α\)​d1−α​ψ1−α\\displaystyle\\sim\\frac\{AI\_\{\\alpha\}^\{\\alpha\}\}\{\\zeta\(\\alpha\)\}d^\{1\-\\alpha\}\\psi^\{1\-\\alpha\}=A​Cα​N−\(α−1\),Cα:=1ζ⁡\(α\)​\[πα​sin⁡\(π/α\)\]α\.\\displaystyle=AC\_\{\\alpha\}N^\{\-\(\\alpha\-1\)\},\\quad C\_\{\\alpha\}:=\\frac\{1\}\{\\zeta\(\\alpha\)\}\\left\[\\frac\{\\pi\}\{\\alpha\\sin\(\\pi/\\alpha\)\}\\right\]^\{\\alpha\}\.This constant applies to sublinear width\. IfN/d→ψ0∈\(0,1\)N/d\\to\\psi\_\{0\}\\in\(0,1\), the vocabulary cutoff remains relevant, and the corresponding law is

B∼Aζ⁡\(α\)​d1−α​ψρ⁡\(α,ψ\)\.B\\sim\\frac\{A\}\{\\zeta\(\\alpha\)\}d^\{1\-\\alpha\}\\frac\{\\psi\}\{\\rho\(\\alpha,\\psi\)\}\.Equivalently, the dependence is still proportional toN−\(α−1\)N^\{\-\(\\alpha\-1\)\}, but its prefactor also depends onψ\\psi\.

It remains to justify replacing the empirical quantities in the proposition by those analyzed above\. The sample\-size assumption can be written as

λd:=n​pd=ndα​Hd,α⟶∞\.\\lambda\_\{d\}:=np\_\{d\}=\\frac\{n\}\{d^\{\\alpha\}H\_\{d,\\alpha\}\}\\longrightarrow\\infty\.Thus even the least frequent context has a diverging expected count\. In particular,

𝔼⁡\[d−S\]=∑i=1d\(1−pi\)n≤∑i=1de−n​pi≤d​e−λd\.\\mathbb\{E\}\[d\-S\]=\\sum\_\{i=1\}^\{d\}\(1\-p\_\{i\}\)^\{n\}\\leq\\sum\_\{i=1\}^\{d\}e^\{\-np\_\{i\}\}\\leq de^\{\-\\lambda\_\{d\}\}\.Sinced−N≥δ​dd\-N\\geq\\delta deventually for someδ\>0\\delta\>0in both width regimes under consideration, the inequality

0≤\(d−N\)−\(S−N\)\+≤d−S0\\leq\(d\-N\)\-\(S\-N\)\_\{\+\}\\leq d\-Simplies

𝔼​\|\(S−N\)\+d−N−1\|≤e−λdδ⟶0\.\\mathbb\{E\}\\left\|\\frac\{\(S\-N\)\_\{\+\}\}\{d\-N\}\-1\\right\|\\leq\\frac\{e^\{\-\\lambda\_\{d\}\}\}\{\\delta\}\\longrightarrow 0\.\(31\)The deterministic variance contribution is therefore

V:=1−An​\(d−N\)=\(1−A\)​\(1−ψ\)​ϕ,ϕ:=dn\.V:=\\frac\{1\-A\}\{n\}\(d\-N\)=\(1\-A\)\(1\-\\psi\)\\phi,\\quad\\phi:=\\frac\{d\}\{n\}\.For sublinear width, this reduces toV=\(1−A\)​ϕ​\(1\+o⁡\(1\)\)V=\(1\-A\)\\phi\(1\+o\(1\)\)\.

To handle the fixed point, define

F⁡\(t\):=∑i=1dpipi\+t,F^​\(t\):=∑i=1dp^ip^i\+t,p^i:=cin\.F\(t\):=\\sum\_\{i=1\}^\{d\}\\frac\{p\_\{i\}\}\{p\_\{i\}\+t\},\\quad\\widehat\{F\}\(t\):=\\sum\_\{i=1\}^\{d\}\\frac\{\\widehat\{p\}\_\{i\}\}\{\\widehat\{p\}\_\{i\}\+t\},\\quad\\widehat\{p\}\_\{i\}:=\\frac\{c\_\{i\}\}\{n\}\.Rather than requiring all empirical frequencies to concentrate uniformly, it is enough to control this aggregate\. For everyt\>0t\>0,

\|p^ip^i\+t−pipi\+t\|≤pipi\+t​\|p^i−pi\|pi\.\\left\|\\frac\{\\widehat\{p\}\_\{i\}\}\{\\widehat\{p\}\_\{i\}\+t\}\-\\frac\{p\_\{i\}\}\{p\_\{i\}\+t\}\\right\|\\leq\\frac\{p\_\{i\}\}\{p\_\{i\}\+t\}\\,\\frac\{\|\\widehat\{p\}\_\{i\}\-p\_\{i\}\|\}\{p\_\{i\}\}\.The binomial variance bound gives

𝔼​\|p^i−pi\|pi≤1n​pi≤1λd,\\mathbb\{E\}\\frac\{\|\\widehat\{p\}\_\{i\}\-p\_\{i\}\|\}\{p\_\{i\}\}\\leq\\frac\{1\}\{\\sqrt\{np\_\{i\}\}\}\\leq\\frac\{1\}\{\\sqrt\{\\lambda\_\{d\}\}\},and hence

𝔼​\|F^​\(t\)−F⁡\(t\)\|≤F⁡\(t\)λd\.\\mathbb\{E\}\|\\widehat\{F\}\(t\)\-F\(t\)\|\\leq\\frac\{F\(t\)\}\{\\sqrt\{\\lambda\_\{d\}\}\}\.\(32\)
The population asymptotics also show that the root is stable under these perturbations\. For any fixeda\>0a\>0,

F⁡\(a​ν\)N⟶\{a−1,N=o\(d\),0≤α≤1,a−1/α,N=o\(d\),α\>1,Gα​\(ρ0/a\)Gα​\(ρ0\),N/d→ψ0∈\(0,1\),\\frac\{F\(a\\nu\)\}\{N\}\\longrightarrow\\begin\{cases\}a^\{\-1\},&N=o\(d\),\\quad 0\\leq\\alpha\\leq 1,\\\\\[3\.0pt\] a^\{\-1/\\alpha\},&N=o\(d\),\\quad\\alpha\>1,\\\\\[6\.0pt\] \\displaystyle\\frac\{G\_\{\\alpha\}\(\\rho\_\{0\}/a\)\}\{G\_\{\\alpha\}\(\\rho\_\{0\}\)\},&N/d\\to\\psi\_\{0\}\\in\(0,1\),\\end\{cases\}whereGα​\(ρ0\)=ψ0G\_\{\\alpha\}\(\\rho\_\{0\}\)=\\psi\_\{0\}\. Each limit is greater than one fora<1a<1and smaller than one fora\>1a\>1\. For any fixedϵ∈\(0,1\)\\epsilon\\in\(0,1\), equation \([32](https://arxiv.org/html/2610.19281#A1.E32)\) therefore ensures that, with probability tending to one,

F^​\(\(1−ϵ\)​ν\)\>N,F^​\(\(1\+ϵ\)​ν\)<N\.\\widehat\{F\}\(\(1\-\\epsilon\)\\nu\)\>N,\\quad\\widehat\{F\}\(\(1\+\\epsilon\)\\nu\)<N\.Monotonicity traps the empirical root between these two arguments:

\(1−ϵ\)​ν≤ν^≤\(1\+ϵ\)​ν\.\(1\-\\epsilon\)\\nu\\leq\\widehat\{\\nu\}\\leq\(1\+\\epsilon\)\\nu\.Thus

ν^/ν→ℙ1\.\\widehat\{\\nu\}/\\nu\\xrightarrow\{\\mathbb\{P\}\}1\.\(33\)Here, as in the proposition,ν^=0\\widehat\{\\nu\}=0whenN≥SN\\geq S; the preceding argument also shows that this event has vanishing probability in the regimes considered\.

For an expectation over the sample, we need slightly more than convergence in probability\. A useful feature of the fixed point is that its solution is concave in the frequency vector\. Indeed, letνN​\(u\)\\nu\_\{N\}\(u\)denote the solution for a nonnegative vectoruu, setting it to zero whenuuhas at mostNNnonzero entries\. Fort≥0t\\geq 0,

t≤νN\(u\)⟺∑i=1dui​tui\+t−Nt≥0,t\\leq\\nu\_\{N\}\(u\)\\quad\\Longleftrightarrow\\quad\\sum\_\{i=1\}^\{d\}\\frac\{u\_\{i\}t\}\{u\_\{i\}\+t\}\-Nt\\geq 0,with the continuous extension atui=t=0u\_\{i\}=t=0\. Becauseui​t/\(ui\+t\)u\_\{i\}t/\(u\_\{i\}\+t\)is jointly concave, this describes a convex hypograph\. Jensen’s inequality consequently gives

𝔼⁡\[ν^\]=𝔼⁡\[νN​\(p^\)\]≤νN​\(𝔼⁡\[p^\]\)=ν\.\\mathbb\{E\}\[\\widehat\{\\nu\}\]=\\mathbb\{E\}\[\\nu\_\{N\}\(\\widehat\{p\}\)\]\\leq\\nu\_\{N\}\(\\mathbb\{E\}\[\\widehat\{p\}\]\)=\\nu\.Together with \([33](https://arxiv.org/html/2610.19281#A1.E33)\), this implies relativeL1L^\{1\}convergence\. To see this, putZd:=ν^/νZ\_\{d\}:=\\widehat\{\\nu\}/\\nu\. ThenZd≥0Z\_\{d\}\\geq 0,𝔼⁡\[Zd\]≤1\\mathbb\{E\}\[Z\_\{d\}\]\\leq 1, andZd→1Z\_\{d\}\\to 1in probability, so

𝔼​\|Zd−1\|=𝔼⁡\[Zd\]−1\+2​𝔼​\[\(1−Zd\)\+\]≤2​𝔼​\[\(1−Zd\)\+\]⟶0\.\\mathbb\{E\}\|Z\_\{d\}\-1\|=\\mathbb\{E\}\[Z\_\{d\}\]\-1\+2\\mathbb\{E\}\[\(1\-Z\_\{d\}\)\_\{\+\}\]\\leq 2\\mathbb\{E\}\[\(1\-Z\_\{d\}\)\_\{\+\}\]\\longrightarrow 0\.
We can now return to Proposition[1](https://arxiv.org/html/2610.19281#Thmtheorem1)\. Writing its remainder asε⁡\(c\)\\varepsilon\(c\), the conditional expected regret is

𝔼Y,Ω​\[Rn,d​\(N\)∣c\]=A​N​ν^\+1−An​\(S−N\)\+\+ε⁡\(c\),\\mathbb\{E\}\_\{Y,\\Omega\}\[R\_\{n,d\}\(N\)\\mid c\]=AN\\widehat\{\\nu\}\+\\frac\{1\-A\}\{n\}\(S\-N\)\_\{\+\}\+\\varepsilon\(c\),where the expectation includes the next\-token observations unless they have already been averaged in the definition of the regret\. The estimates above show that the two displayed main terms equal

B\+V\+oL1​\(B\+V\)\.B\+V\+o\_\{L^\{1\}\}\(B\+V\)\.Therefore, provided the proposition’s approximation error satisfies𝔼​\|ε⁡\(c\)\|=o⁡\(B\+V\)\\mathbb\{E\}\|\\varepsilon\(c\)\|=o\(B\+V\), we conclude that

𝔼X,Y,Ω​\[Rn,d​\(N\)\]∼B\+V\.\\mathbb\{E\}\_\{X,Y,\\Omega\}\[R\_\{n,d\}\(N\)\]\\sim B\+V\.The three forms ofBBfollow from the population fixed\-point analysis, and the stated variance terms follow from \([31](https://arxiv.org/html/2610.19281#A1.E31)\)\.

The restrictionψ0<1\\psi\_\{0\}<1is important: whenN/d→1N/d\\to 1, the quantityd−Nd\-Nmay be too small for the occupancy estimate above to give a relative approximation\. AtN=dN=d, however, the separate interpolation branch applies directly:ν=0\\nu=0, and both contributions vanish\. ∎