One Spectrum, Two Resources: Data-Memory Scaling in Autoregressive Prediction

# One Spectrum, Two Resources:Data–Memory Scaling in Autoregressive Prediction
Source: [https://arxiv.org/html/2609.13500](https://arxiv.org/html/2609.13500)
\\addtotheorempostheadhook

\[theorem\]\\addtotheorempostheadhook\[lemma\]\\addtotheorempostheadhook\[corollary\]\\addtotheorempostheadhook\[proposition\]

Xiaoyu Li††thanks:xiaoyu\.li2@unsw\.edu\.au\. University of New South Wales\.

###### Abstract

How much learned memory is needed to benefit from more data? We show that the two resources are governed by one predictive\-energy spectrum in a positive\-entropy autoregressive retrieval source\. Each coordinate contributes its query probability times the squared radius of its unknown logit\. Writingμ\\mufor the resulting energy spectrum, we prove the minimax lawℜvalue∗\(n,B\)≍RΦμ\(n−1\)\+Φμ\(τB\),Φμ\(t\)=∫min\{x,t\}μ\(dx\),\\mathfrak\{R\}^\{\*\}\_\{\\rm value\}\(n,B\)\\asymp\_\{R\}\\Phi\_\{\\mu\}\(n^\{\-1\}\)\+\\Phi\_\{\\mu\}\(\\tau\_\{B\}\),\\Phi\_\{\\mu\}\(t\)=\\int\\min\\\{x,t\\\}\\,\\mu\(\\mathrm\{d\}x\),fornnprediction blocks and a learned state with at most2B2^\{B\}values\. Data set the resolution1/n1/n; memory sets the levelτB\\tau\_\{B\}reached by optimal bit allocation\. The complete curve also recovers the positive spectrum\. Energy–dimension pairing is essential: two causal sources with identical block\-energy and block\-dimension marginals have different data and memory exponents\. A masked query–key attention head learns the route and values, realizing the law with explicit routing, format, and arithmetic errors\. Further results give exponent\-adaptive allocation, finite\-precision realization, and compute–precision laws under two\-sided arithmetic assumptions\. Experiments recover the data–memory collapse and coupling exponents, explain the routing and allocation mechanisms, and examine weight\-only quantization across six pretrained\-model scales\.

## 1Introduction

More data reveal more predictive structure, but a predictor benefits only from what its learned state can retain\. When should we collect more data, and when should we increase learned memory? Empirical scaling laws relate loss to data, model size, and compute\([Kaplan et al\., 2020](https://arxiv.org/html/2609.13500#bib.bib15);[Hoffmann et al\., 2022](https://arxiv.org/html/2609.13500#bib.bib17)\)\. To understand how these resources work together, we seek a property of the prediction problem that determines both the information available in finite data and the information preserved by a finite learned state\.

Consider a prediction task with many independently varying pieces of knowledge\. A piece that is queried often or strongly changes the next\-token probability deserves more learning and storage resources than one that is rarely used or has little predictive effect\. Our model expresses each piece as an unknown scalar prediction parameter, called a*coordinate*\. The key quantity is predictive energy per independent coordinate\. In our autoregressive source, coordinateiihas query probabilitypip\_\{i\}and an unknown logit, or log\-odds of a binary target, in\[−Ri,Ri\]\[\-R\_\{i\},R\_\{i\}\]\. Its predictive energyqi=pi​Ri2q\_\{i\}=p\_\{i\}R\_\{i\}^\{2\}measures its potential contribution to prediction error\. Collecting these energies, counting repeated values separately, gives the spectrum\. At a chosen error resolution, small coordinates contribute their whole energy and larger coordinates contribute only the unresolved portion\. Adding these contributions defines a spectral curve\. Finite data and finite learned memory read*the same spectral curve at different resolutions*\.

This connection gives a joint resource law\. Letnncount independent training blocks and letBBcount the bits retained after training, so the learner can finish in at most2B2^\{B\}different states\. We measure excess cross\-entropy above the loss of a predictor that knows the true conditional probabilities\. First consider value estimation with the relevant coordinate revealed\. Data set an error resolution of order1/n1/n; memory allocates bits until retained coordinates reach a common residual\-error level\. The smallest worst\-case expected excess loss is, up to constants, the sum of the spectral curve at these two resolutions\. This minimax result allows unrestricted training computation and joint coding across coordinates\. Section[3](https://arxiv.org/html/2609.13500#S3)defines the source, risk, and two resolutions before stating the full formula\.

The law makes the resource balance explicit\. List coordinate energies in decreasing order and supposeq\(i\)≍i−\(1\+χ\)q\_\{\(i\)\}\\asymp i^\{\-\(1\+\\chi\)\}, whereχ\>0\\chi\>0is their decay exponent and≍\\asympdenotes equality up to constant factors\. The two terms scale asn−χ/\(1\+χ\)\+B−χn^\{\-\\chi/\(1\+\\chi\)\}\+B^\{\-\\chi\}and balance atB≍n1/\(1\+χ\)B\\asymp n^\{1/\(1\+\\chi\)\}\. Below this memory scale, retaining learned information is the bottleneck; above it, observations matter more\. More generally, an energy group can spread its total energy across several independent coordinates\. Increasing that dimension produces more parameters to learn, each with less predictive energy\. The general scaling corollary quantifies this effect through the distribution of coordinate energies\. The full spectral curve applies beyond power laws and recovers every positive coordinate energy with its multiplicity\.

This perspective reveals information that separate scaling descriptors discard\. Concentrating energy in one coordinate makes it easier to learn and retain than spreading it across many independent coordinates\. We construct two causal sources with exactly the same block\-energy and block\-dimension multisets at every dyadic level\. Pairing high energy with small blocks or large blocks changes the coordinate spectrum and both resource exponents\. The pairing, not either marginal alone, determines how predictive information is distributed\.

The source also makes the learning mechanism explicit\. Each regeneration block containsKKcandidate tokens, a query, and a stochastic next token\. An unknown route selects the relevant candidate, whose bounded Bernoulli logit is also unknown\. Calibration blocks reveal the correct candidate; prediction blocks teach its value; evaluation uses independent held\-out blocks\. Fixed embeddings and projections contain neither unknown object\. A fixed\-key, trainable\-query attention head learns the route, followed by frequency\-preconditioned output\-parameter learning\. Its masked query–key scores and gradients match the construction in the proof, connecting the spectral law to a concrete two\-stage learner\.

The same resource picture organizes representation and computation\. A universal allocation schedule adapts to an unknown power\-law exponent when coordinate order is known\. Finite\-precision evaluation quantifies routing, storage, and arithmetic errors\. For the specified dense computational graph, operation counts and numerical assumptions connect the precision needed for a target error to bit complexity and conditional compute phases\. These results explain how statistical resolution becomes a representational and computational requirement\.

- •One spectrum, two resources\.An exact truncated\-energy curve expresses both terms of arbitrary\-state minimax risk, recovers the positive spectrum, and determines the data and learned\-memory exponents\. A masked\-attention learner realizes the spectral surface with explicit routing, format, and arithmetic errors; adaptive allocation and precision analysis connect this law to implementation\.
- •Why separate marginals fail\.Same\-marginal causal sources have different coordinate spectra and different data and memory exponents\. Their energy–dimension pairing identifies the information missing from separate scaling descriptors\.

Our experiments vary sample and state budgets on a fixed source, test same\-marginal separation, and compare learned, known, and frozen routing\. Interventions and perturbed energy profiles examine the mechanisms behind the measured curves\. Continued training tests the interaction between added data and deployed storage, while weight\-only post\-training quantization across six pretrained Transformer scales measures the effect of numerical representation at fixed checkpoints\.

## 2Related Work

Our question connects scaling laws, finite\-rate representation, and distribution\-specific learning: which source statistic determines the value of additional observations and additional learned memory? The following literatures provide the statistical and computational ingredients\.[Table1](https://arxiv.org/html/2609.13500#S2.T1)summarizes the closest theorem\-level connections\.

#### Empirical and mechanistic scaling laws\.

Empirical studies found extended power\-law regimes in neural prediction as data, model size, and computation grow\([Hestness et al\., 2017](https://arxiv.org/html/2609.13500#bib.bib14);[Kaplan et al\., 2020](https://arxiv.org/html/2609.13500#bib.bib15);[Henighan et al\., 2020](https://arxiv.org/html/2609.13500#bib.bib16)\)\. Compute\-optimal work asks how a training budget should be divided between parameters and tokens\([Hoffmann et al\., 2022](https://arxiv.org/html/2609.13500#bib.bib17)\), while routed models distinguish total parameters from parameters used per token\([Clark et al\., 2022](https://arxiv.org/html/2609.13500#bib.bib18)\)\. Extrapolation studies, broken\-power\-law fits, and replications show that fitted exponents depend on the model family, resource definition, and training prescription\([Alabdulmohsin et al\., 2022](https://arxiv.org/html/2609.13500#bib.bib19);[Caballero et al\., 2023](https://arxiv.org/html/2609.13500#bib.bib20);[Porian et al\., 2024](https://arxiv.org/html/2609.13500#bib.bib21)\)\. Theoretical accounts derive learning curves from task dimension, spectral decay, or simplified dynamics\([Sharma and Kaplan, 2020](https://arxiv.org/html/2609.13500#bib.bib22);[Bordelon et al\., 2020](https://arxiv.org/html/2609.13500#bib.bib5);[Bahri et al\., 2024](https://arxiv.org/html/2609.13500#bib.bib7)\); solvable random\-feature and dynamical models separate data\-, width\-, and time\-limited phases\([Maloney et al\., 2022](https://arxiv.org/html/2609.13500#bib.bib23);[Bordelon et al\., 2024](https://arxiv.org/html/2609.13500#bib.bib24);[Paquette et al\., 2024](https://arxiv.org/html/2609.13500#bib.bib25)\)\. A discrete\-skill model obtains aggregate power laws from decreasing skill frequencies\([Michaud et al\., 2023](https://arxiv.org/html/2609.13500#bib.bib26)\)\. Knowledge\-capacity studies instead translate factual prediction loss into retained information and estimate bits per parameter\([Allen\-Zhu and Li, 2025](https://arxiv.org/html/2609.13500#bib.bib59);[Morris et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib61)\)\. A compression model with frequency\-ranked knowledge derives both data and model scaling laws\([Pan et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib60)\), while an optimization analysis of multilayer Transformers derives resource\-dependent generalization phases\([Yang, 2025](https://arxiv.org/html/2609.13500#bib.bib62)\)\. We connect the data and learned\-memory axes through a common source spectrum\. Its tail determines both rates, and its construction identifies the energy–dimension pairing on which those rates depend\.

Table 1:Closest results and the resource question they answer\.Heremmis Gaussian sequence dimension,NcN\_\{c\}is codebook size, andbbis bits per Gaussian coordinate,BBis total retained bits, andnnis the number of training blocks in our source\. CE denotes cross\-entropy\. The last row identifies the observation model and finite\-resource conclusion needed here\.StudyObservations and parametersCode and riskConclusion[Zhu and Lafferty \(2014\)](https://arxiv.org/html/2609.13500#bib.bib1)Gaussian sequence; deterministic means in a Euclidean ballArbitrary codebook,m​bmbbits; normalized squared errorSharp asymptotic estimation–storage trade\-off asm→∞m\\to\\infty\.[Luschgy and Pagès \(2004\)](https://arxiv.org/html/2609.13500#bib.bib47)Gaussian random element; covariance eigenvaluesNcN\_\{c\}\-point quantizer; expected squared Hilbert errorHigh\-resolution distortion from eigenvalue decay;B=log2⁡NcB=\\log\_\{2\}N\_\{c\}\.[Allen\-Zhu and Li \(2025\)](https://arxiv.org/html/2609.13500#bib.bib59)Random discrete factual tuples; trained language model weightsFinite weight state; factual generation negative log\-likelihoodAchieved loss lower\-bounds the information that the weights must retain\.[Pan et al\. \(2025\)](https://arxiv.org/html/2609.13500#bib.bib60)Hierarchical syntax–knowledge mixture; frequency\-ranked knowledgeBayesian/universal code; predictive redundancy under data or model capacityPower\-law knowledge frequencies generate data and model scaling laws\.[Goel et al\. \(2026\)](https://arxiv.org/html/2609.13500#bib.bib58)Gaussian linear regression; softmax attentionExpected empirical\-gradient oracle; squared loss; no bit constraintGeometric optimization with a finite\-context statistical term\.This workRandom coordinate observations; independently bounded scalar logitsAny2B2^\{B\}\-state learner; held\-out excess Bernoulli CEUniform\(n,B\)\(n,B\)risk through one spectrum; pairing changes both exponents\.
#### Finite\-rate estimation and memory\.

Rate–distortion theory characterizes the smallest coding rate compatible with a prescribed distortion, and classical quantization theory develops the geometry of finite codebooks\([Shannon, 1959](https://arxiv.org/html/2609.13500#bib.bib27);[Gray and Neuhoff, 1998](https://arxiv.org/html/2609.13500#bib.bib28)\)\. Quantized Gaussian sequence estimation gives sharp arbitrary\-codebook storage–risk trade\-offs, smoothness adaptation over Sobolev classes, and multiple data–communication regimes\([Zhu and Lafferty, 2014](https://arxiv.org/html/2609.13500#bib.bib1);[Zhu and Lafferty, 2018b](https://arxiv.org/html/2609.13500#bib.bib2);[Zhu and Lafferty, 2018a](https://arxiv.org/html/2609.13500#bib.bib3)\)\. Related theories bound working memory, sequential estimator states, retained sample information, or Bayesian minimum excess risk\([Steinhardt and Duchi, 2015](https://arxiv.org/html/2609.13500#bib.bib49);[Berg et al\., 2021](https://arxiv.org/html/2609.13500#bib.bib51);[Feldman et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib52);[Hafez\-Kolahi et al\., 2021](https://arxiv.org/html/2609.13500#bib.bib50)\)\. WritingBBfor retained bits, we constrain the final learned state to at most2B2^\{B\}values while allowing unrestricted training computation\. Functional quantization is especially close to our spectrum calculation: regularly varying covariance eigenvalues determine sharp Hilbert\-space quantization rates, and product quantizers attain them\([Luschgy and Pagès, 2004](https://arxiv.org/html/2609.13500#bib.bib47);[Luschgy et al\., 2010](https://arxiv.org/html/2609.13500#bib.bib48)\)\. Our additional step is a uniform finite\-resource risk bound for random coordinate observations: each observation queries one unknown binary prediction parameter, and all logits vary independently within bounded intervals\. It accounts for coordinates with few or no visits while allowing any final2B2^\{B\}\-state encoder\. The same curve then expresses sample and retained\-state error\. The same\-marginal construction resolves an identification question: separate block\-energy and block\-dimension distributions do not determine the coordinate spectrum\.

#### Neural quantization and precision\.

Mixed\-precision training retains high precision where it is numerically useful\([Micikevicius et al\., 2018](https://arxiv.org/html/2609.13500#bib.bib11)\)\. Mean\-field, Hessian\-aware, and learned\-precision methods quantify depth, sensitivity, and heterogeneous allocation effects\([Blumenfeld et al\., 2019](https://arxiv.org/html/2609.13500#bib.bib29);[Dong et al\., 2020](https://arxiv.org/html/2609.13500#bib.bib6);[Savarese et al\., 2022](https://arxiv.org/html/2609.13500#bib.bib30)\)\. Per\-tensor analyses and integer programs contain the dynamic\-range\-squared times4−b4^\{\-b\}distortion law and resource\-constrained bit allocation\([Chen et al\., 2020](https://arxiv.org/html/2609.13500#bib.bib53);[Yao et al\., 2021](https://arxiv.org/html/2609.13500#bib.bib54)\)\. Practical large language model \(LLM\) quantization uses outlier handling, second\-order rounding, activation rescaling, and data\-dependent discrepancy control\([Dettmers et al\., 2022](https://arxiv.org/html/2609.13500#bib.bib31);[Frantar et al\., 2023](https://arxiv.org/html/2609.13500#bib.bib32);[Xiao et al\., 2023](https://arxiv.org/html/2609.13500#bib.bib33);[Chee et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib34)\)\. Precision\-aware scaling studies measure how these effects change with model size\([Dettmers and Zettlemoyer, 2023](https://arxiv.org/html/2609.13500#bib.bib10);[Kumar et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib4);[Sun et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib9);[Zhang et al\., 2026](https://arxiv.org/html/2609.13500#bib.bib8)\); kernel\-regime analysis establishes width\-dependent convergence for 1\-bit networks\([Daliri et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib63)\)\. Our finite\-precision analysis composes local numerical distortion with finite\-sample uncertainty and the learned\-state budget\. For the specified computational graph, it also quantifies the arithmetic cost associated with the required precision\.

#### Minimax generalization and attention optimization\.

Worst\-case neural\-network theory controls function classes through Vapnik–Chervonenkis dimension, pseudo\-dimension, norms, margins, and covering numbers\([Bartlett et al\., 2019](https://arxiv.org/html/2609.13500#bib.bib35);[Bartlett et al\., 2017](https://arxiv.org/html/2609.13500#bib.bib36)\)\. Compression and description\-length analyses relate concise predictors to generalization, including probably approximately correct \(PAC\)–Bayes bounds\([Arora et al\., 2018](https://arxiv.org/html/2609.13500#bib.bib38);[Daniely and Granot, 2019](https://arxiv.org/html/2609.13500#bib.bib37);[Lotfi et al\., 2022](https://arxiv.org/html/2609.13500#bib.bib39)\)\. Distribution\-specific minimax theory ties rates to compositional smoothness, Besov regularity, or intrinsic dimension\([Schmidt\-Hieber, 2020](https://arxiv.org/html/2609.13500#bib.bib40);[Suzuki, 2019](https://arxiv.org/html/2609.13500#bib.bib41);[Nakada and Imaizumi, 2020](https://arxiv.org/html/2609.13500#bib.bib42)\)\. Our statistical result follows this distribution\-specific approach, with held\-out autoregressive cross\-entropy and an arbitrary finite learned state\.

For sequence models, norm\-based Transformer bounds, analyses of Markov next\-token prediction, dependent\-token pretraining, and nearly tight sample complexity address generalization under architecture or dependence\([Trauger and Tewari, 2024](https://arxiv.org/html/2609.13500#bib.bib43);[Yüksel and Flammarion, 2025](https://arxiv.org/html/2609.13500#bib.bib44);[Li et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib45);[Yang et al\., 2026](https://arxiv.org/html/2609.13500#bib.bib46)\)\. Attention optimization studies max\-margin token selection, retrieval learning, and frequency\-dependent associative memory\([Ataee Tarzanagh et al\., 2023](https://arxiv.org/html/2609.13500#bib.bib55);[Li et al\., 2024](https://arxiv.org/html/2609.13500#bib.bib56);[Cabannes et al\., 2024](https://arxiv.org/html/2609.13500#bib.bib57)\)\. Structure\-aware preconditioning gives a data–compute law for Gaussian softmax regression under an expected empirical\-gradient oracle\([Goel et al\., 2026](https://arxiv.org/html/2609.13500#bib.bib58)\); other softmax\-regression objectives admit efficient gradient methods with structured output constraints\([Chu et al\., 2024](https://arxiv.org/html/2609.13500#bib.bib69)\)\. Theory for prefix tuning and next\-step forecasting identifies how attention training and its generalization depend on the learned representation\([Liang et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib64);[Ke et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib65)\)\. Complementary work asks which token state can be removed while preserving attention or long\-context prediction: sparse approximation controls attention\-matrix error, while parallel and value\-aware compressors allocate a limited context state\([Deng et al\., 2024](https://arxiv.org/html/2609.13500#bib.bib66);[Xiong et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib67);[Yang and Li, 2026](https://arxiv.org/html/2609.13500#bib.bib68)\)\. We count the samples used to learn the relevant candidate and its prediction parameter, then the bits used to retain the fitted state\. Appendix[A](https://arxiv.org/html/2609.13500#A1)gives the detailed comparison of the statistical, representational, and computational settings\.

## 3The Energy\-per\-Coordinate Spectrum: Minimax Risk and Identifiability

We first define the source and finite\-state value\-estimation problem; Section[4](https://arxiv.org/html/2609.13500#S4)additionally learns the relevant route\.

### Problem setup

#### Coordinates and unknown knowledge\.

A coordinate is an independent scalar prediction parameter; modejjgroupsdjd\_\{j\}coordinatesi=\(j,r\)i=\(j,r\),r∈\[dj\]r\\in\[d\_\{j\}\], where\[m\]=\{1,…,m\}\[m\]=\\\{1,\\ldots,m\\\}\. Separating energy from parameter count, the finite/countable setℐ\\mathcal\{I\}has public frequenciespi\>0p\_\{i\}\>0,∑ipi=1\\sum\_\{i\}p\_\{i\}=1, and radii0<Ri≤R<∞0<R\_\{i\}\\leq R<\\infty\. Unknown logitsθ∈Θ:=∏i∈ℐ\[−Ri,Ri\]\\theta\\in\\Theta:=\\prod\_\{i\\in\\mathcal\{I\}\}\[\-R\_\{i\},R\_\{i\}\]specify binary target probabilities\.

#### One causal prediction block\.

Sample candidatesC1,…,CKC\_\{1\},\\ldots,C\_\{K\}independently frompp, then an independent query rowQQ, uniform on\[K\]\[K\], followed by targetYY\. A fixed unknown matching mapπ:\[K\]→\[K\]\\pi:\[K\]\\to\[K\]selectsI=Cπ⁡\(Q\)I=C\_\{\\pi\(Q\)\}, withY\|I∼Bernoulli⁡\(σ⁡\(θI\)\)Y\\mid I\\sim\\operatorname\{Bernoulli\}\(\\sigma\(\\theta\_\{I\}\)\)\. Hereσ⁡\(t\)=\(1\+e−t\)−1\\sigma\(t\)=\(1\+e^\{\-t\}\)^\{\-1\}is the sigmoid, soθi\\theta\_\{i\}is the log\-odds atI=iI=i, andPr⁡\(I=i\)=pi\\Pr\(I=i\)=p\_\{i\}\. Delimiters separate independent, causally ordered blocks;nncounts blocks, not tokens\. Calibration reveals\(Q,π⁡\(Q\)\)\(Q,\\pi\(Q\)\), prediction targets teach logits, and testing uses independent blocks\. Revealed routes/coordinates give value data𝒟n=\(\(It,Yt\)\)t=1n\\mathcal\{D\}\_\{n\}=\(\(I\_\{t\},Y\_\{t\}\)\)\_\{t=1\}^\{n\}, separating estimation from retrieval\.

#### Loss and the retained\-state constraint\.

For coordinate predictionsf=\(fi\)f=\(f\_\{i\}\), defineℓ⁡\(y,z\)=−y​log⁡σ⁡\(z\)−\(1−y\)​log⁡\(1−σ⁡\(z\)\)\\ell\(y,z\)=\-y\\log\\sigma\(z\)\-\(1\-y\)\\log\(1\-\\sigma\(z\)\)andℒθ​\(f\)=𝔼θ​\[ℓ⁡\(Y,fI\)\]\\mathcal\{L\}\_\{\\theta\}\(f\)=\\mathbb\{E\}\_\{\\theta\}\[\\ell\(Y,f\_\{I\}\)\]\. Loss is in nats, storage in bits\. Bounded logits give positive conditional entropy, Bayes lossℒ∞=ℒθ​\(θ\)\>0\\mathcal\{L\}\_\{\\infty\}=\\mathcal\{L\}\_\{\\theta\}\(\\theta\)\>0, and excess loss comparable to frequency\-weighted squared logit error, withRR\-dependent constants\. A learner encodes𝒟n\\mathcal\{D\}\_\{n\}intoS=ϕ⁡\(𝒟n\)∈𝒮S=\\phi\(\\mathcal\{D\}\_\{n\}\)\\in\\mathcal\{S\},\|𝒮\|≤2B\|\\mathcal\{S\}\|\\leq 2^\{B\}, and predicts with a fixed decoderg⁡\(S,i\)∈\[−R,R\]g\(S,i\)\\in\[\-R,R\]\. Arbitrary computation and joint coordinate coding are allowed, with all data\-dependent output inSS\. For integerB≥0B\\geq 0, minimax excess risk is

ℜvalue∗\(n,B\)=infϕ,g:\|𝒮\|≤2Bsupθ∈Θ𝔼θ\[ℒθ\(g\(ϕ\(𝒟n\),⋅\)\)−ℒθ\(θ\)\]\.\\mathfrak\{R\}^\{\*\}\_\{\\rm value\}\(n,B\)=\\inf\_\{\\phi,g:\\,\|\\mathcal\{S\}\|\\leq 2^\{B\}\}\\;\\sup\_\{\\theta\\in\\Theta\}\\mathbb\{E\}\_\{\\theta\}\\\!\\left\[\\mathcal\{L\}\_\{\\theta\}\\bigl\(g\(\\phi\(\\mathcal\{D\}\_\{n\}\),\\cdot\)\\bigr\)\-\\mathcal\{L\}\_\{\\theta\}\(\\theta\)\\right\]\.Expectations average training/learner and independent test randomness\. Encoder/decoder side information is fixed and public; learned bits exclude fixed network storage\. Constants in≍R\\asymp\_\{R\}depend only onRR\.

#### Predictive energy and its spectrum\.

Defineqi=pi​Ri2q\_\{i\}=p\_\{i\}R\_\{i\}^\{2\}, with∑iqi<∞\\sum\_\{i\}q\_\{i\}<\\infty\. Doubling frequency doubles energy; halving logit radius quarters it\. Equal energies can have different frequencies and ranges\. Withδx\\delta\_\{x\}a unit point mass,

μ=∑iδqi,𝒩μ​\(t\)=μ⁡\(\(t,∞\)\),Φμ​\(t\)=∫min⁡\{x,t\}​μ​\(𝑑x\)=∫0t𝒩μ​\(s\)​𝑑s\.\\mu=\\sum\_\{i\}\\delta\_\{q\_\{i\}\},\\quad\\mathcal\{N\}\_\{\\mu\}\(t\)=\\mu\(\(t,\\infty\)\),\\quad\\Phi\_\{\\mu\}\(t\)=\\int\\min\\\{x,t\\\}\\,\\mu\(\\mathrm\{d\}x\)=\\int\_\{0\}^\{t\}\\mathcal\{N\}\_\{\\mu\}\(s\)\\,\\mathrm\{d\}s\.Thus𝒩μ​\(t\)\\mathcal\{N\}\_\{\\mu\}\(t\)counts energies abovett, including repeats;Φμ​\(t\)=∑imin⁡\{qi,t\}\\Phi\_\{\\mu\}\(t\)=\\sum\_\{i\}\\min\\\{q\_\{i\},t\\\}caps unresolved contributions\. For water levelτ\>0\\tau\>0, letℬμ​\(τ\)=12​∫log2⁡\(x/τ\)\+​μ​\(𝑑x\)\\mathcal\{B\}\_\{\\mu\}\(\\tau\)=\\frac\{1\}\{2\}\\int\\log\_\{2\}\(x/\\tau\)\_\{\+\}\\mu\(\\mathrm\{d\}x\)and defineτB\\tau\_\{B\}as the generalized solution ofℬμ​\(τB\)=B\\mathcal\{B\}\_\{\\mu\}\(\\tau\_\{B\}\)=B, withτ0=maxi⁡qi\\tau\_\{0\}=\\max\_\{i\}q\_\{i\}\. Here\(x\)\+=max⁡\{x,0\}\(x\)\_\{\+\}=\\max\\\{x,0\\\}applies after the logarithm: coordinates aboveτ\\taureceive12​log2⁡\(qi/τ\)\\frac\{1\}\{2\}\\log\_\{2\}\(q\_\{i\}/\\tau\)bits, others zero\.ℬμ\\mathcal\{B\}\_\{\\mu\}counts continuous bits needed to reachτ\\tau; finitely many energies exceed each positive level, even in countable sets\.

#### Energy groups and scaling profiles\.

An energy block is a coordinate group, distinct from a sampled token block\. For total energyaja\_\{j\}, dimensiondjd\_\{j\}, and equal\-energy coordinates,qj​r=aj/djq\_\{jr\}=a\_\{j\}/d\_\{j\}andμ=∑jdj​δaj/dj\\mu=\\sum\_\{j\}d\_\{j\}\\delta\_\{a\_\{j\}/d\_\{j\}\}: dimension sets both atom count and size\. Profilesaj≍j−\(1\+χ\)a\_\{j\}\\asymp j^\{\-\(1\+\\chi\)\},dj≍jγd\_\{j\}\\asymp j^\{\\gamma\}, withχ\>0\\chi\>0,γ≥0\\gamma\\geq 0, describe energy decay and dimension growth; the theorem itself needs no power law\.

### 3\.1One Spectrum Controls Data and Learned State

###### Theorem 3\.1\(One spectrum controls data and memory; informal version of[TheoremC\.1](https://arxiv.org/html/2609.13500#A3.Thmtheorem1)\)\.

Letqi\>0q\_\{i\}\>0,∑iqi<∞\\sum\_\{i\}q\_\{i\}<\\infty, and letμ\\mu,Φμ\\Phi\_\{\\mu\}, andτB\\tau\_\{B\}be defined above\. Then

𝒮q​\(n\)\\displaystyle\\mathcal\{S\}\_\{q\}\(n\):=∑imin⁡\{qi,n−1\}=Φμ​\(n−1\),\\displaystyle:=\\sum\_\{i\}\\min\\\{q\_\{i\},n^\{\-1\}\\\}=\\Phi\_\{\\mu\}\(n^\{\-1\}\),\(3\.1\)Dq​\(B\)\\displaystyle D\_\{q\}\(B\):=infbi≥0∑ibi≤B∑iqi​2−2​bi=Φμ​\(τB\)\.\\displaystyle:=\\inf\_\{\\begin\{subarray\}\{c\}b\_\{i\}\\geq 0\\\\ \\sum\_\{i\}b\_\{i\}\\leq B\\end\{subarray\}\}\\sum\_\{i\}q\_\{i\}2^\{\-2b\_\{i\}\}=\\Phi\_\{\\mu\}\(\\tau\_\{B\}\)\.\(3\.2\)For the bounded\-logit value experiment,

ℜ∗value\(n,B\)≍RΦμ\(n−1\)\+Φμ\(τB\)\.\\boxed\{\\mathfrak\{R\}^\{\*\}\_\{\\rm value\}\(n,B\)\\asymp\_\{R\}\\Phi\_\{\\mu\}\(n^\{\-1\}\)\+\\Phi\_\{\\mu\}\(\\tau\_\{B\}\)\.\}\(3\.3\)Moreover, for everyt\>0t\>0,

Φμ,\+′​\(t\)=μ⁡\(\(t,∞\)\),Φμ,−′​\(t\)=μ⁡\(\[t,∞\)\)\.\\Phi^\{\\prime\}\_\{\\mu,\+\}\(t\)=\\mu\(\(t,\\infty\)\),\\qquad\\Phi^\{\\prime\}\_\{\\mu,\-\}\(t\)=\\mu\(\[t,\\infty\)\)\.\(3\.4\)Hence the exact curveΦμ\\Phi\_\{\\mu\}recovers the positive spectrum, including atom multiplicities, up to coordinate relabeling\.

The statistical term limits what observations resolve; the representation term limits what any final state can preserve\. The latter is a continuous allocation problem; the finite\-state minimax comparison accounts for the rounding needed to implement an integer\-bit code\.

Binary testing lower\-bounds the sample errormin⁡\{qi,1/n\}\\min\\\{q\_\{i\},1/n\\\}; an entropy argument lower\-bounds representation error even for joint codes\. Estimation followed by quantization attains their sum up to constants\.

Geometrically,𝒩μ​\(t\)\\mathcal\{N\}\_\{\\mu\}\(t\)counts coordinates whose energy exceeds resolutiontt, whileΦμ​\(t\)\\Phi\_\{\\mu\}\(t\)accumulates the unresolved energy\. Data read this curve att=1/nt=1/n\. Storage pays12​log2⁡\(qi/τB\)\\frac\{1\}\{2\}\\log\_\{2\}\(q\_\{i\}/\\tau\_\{B\}\)bits for every coordinate above its water level and reads the same curve att=τBt=\\tau\_\{B\}\. For one coordinate, the two spectral terms aremin⁡\{q,1/n\}\\min\\\{q,1/n\\\}andq​2−2​Bq2^\{\-2B\}\. Water filling equalizes remaining error on coordinates receiving bits\. The derivatives recover every positive energy and its multiplicity\.

###### Corollary 3\.2\(Spectral scaling exponents; informal version of[CorollaryC\.2](https://arxiv.org/html/2609.13500#A3.Thmtheorem2)\)\.

If𝒩μ​\(t\)∼t−ρ​L​\(1/t\)\\mathcal\{N\}\_\{\\mu\}\(t\)\\sim t^\{\-\\rho\}L\(1/t\)for0<ρ<10<\\rho<1, whereLLis slowly varying, then

Φμ\(n−1\)∈RV−\(1−ρ\),Φμ\(τB\)∈RV−\(1−ρ\)/ρ\.\\Phi\_\{\\mu\}\(n^\{\-1\}\)\\in\\operatorname\{RV\}\_\{\-\(1\-\\rho\)\},\\qquad\\Phi\_\{\\mu\}\(\\tau\_\{B\}\)\\in\\operatorname\{RV\}\_\{\-\(1\-\\rho\)/\\rho\}\.Foraj≍j−\(1\+χ\)a\_\{j\}\\asymp j^\{\-\(1\+\\chi\)\}anddj≍jγd\_\{j\}\\asymp j^\{\\gamma\},ρ=\(1\+γ\)/\(1\+χ\+γ\)\\rho=\(1\+\\gamma\)/\(1\+\\chi\+\\gamma\), giving ratesn−χ/\(1\+χ\+γ\)n^\{\-\\chi/\(1\+\\chi\+\\gamma\)\}andB−χ/\(1\+γ\)B^\{\-\\chi/\(1\+\\gamma\)\}, up to slowly varying factors\.

HereRVα\\operatorname\{RV\}\_\{\\alpha\}denotes regular variation with indexα\\alpha: a functionggsatisfiesg⁡\(c​x\)/g⁡\(x\)→cαg\(cx\)/g\(x\)\\to c^\{\\alpha\}asx→∞x\\to\\infty, for everyc\>0c\>0\. Thus the density of small spectral atoms determines both exponents and relates them through one tail index\.

### 3\.2Why Separate Marginals Cannot Identify Scaling

A fixed amount of predictive energy is harder to learn and retain when it is spread across more independent coordinates\. Imagine two lists: the total energy assigned to each group and the number of independent coordinates in each group\. A marginal descriptor retains these lists but forgets which energy goes with which dimension\. We compare two assignments of the same lists: concentrate the high energies in one\-coordinate groups, or spread them across the larger groups\. The next theorem makes this comparison exact at every dyadic level, a level indexed by an integerℓ\\ellwith group counts proportional to2ℓ2^\{\\ell\}\.

###### Theorem 3\.3\(Same block marginals, different scaling laws; informal version of[TheoremC\.6](https://arxiv.org/html/2609.13500#A3.Thmtheorem6)\)\.

For everyχ,γ\>0\\chi,\\gamma\>0, there are two positive\-conditional\-entropy causal\-retrieval sources with exactly the same block\-energy multiset and the same block\-dimension multiset at every dyadic level\. They differ only in the pairing\. All displayed dimensions in this construction are rounded upward to integers\. With the common fixed\-KKroute revealed,

ℜeasy\(n,B\)≍n−χ/\(1\+χ\)\+B−χ,\\mathfrak\{R\}\_\{\\rm easy\}\(n,B\)\\asymp n^\{\-\\chi/\(1\+\\chi\)\}\+B^\{\-\\chi\},whereas

ℜhard\(n,B\)≍n−χ/\(1\+χ\+γ\)\+B−χ/\(1\+γ\)\.\\mathfrak\{R\}\_\{\\rm hard\}\(n,B\)\\asymp n^\{\-\\chi/\(1\+\\chi\+\\gamma\)\}\+B^\{\-\\chi/\(1\+\\gamma\)\}\.

At dyadic levelℓ\\ell, both sources have2ℓ2^\{\\ell\}high\- and low\-energy blocks with energies proportional to2−\(1\+χ\)​ℓ2^\{\-\(1\+\\chi\)\\ell\}and2−\(1\+χ\+δ\)​ℓ2^\{\-\(1\+\\chi\+\\delta\)\\ell\}, and2ℓ2^\{\\ell\}copies each of dimensions11and2γ​ℓ2^\{\\gamma\\ell\}\. Chooseδ\>γ​χ\\delta\>\\gamma\\chi\. The easy source pairs high energy with dimension one; the hard source pairs it with dimension2γ​ℓ2^\{\\gamma\\ell\}\. Thus both block marginals agree exactly, not merely asymptotically\.

A component with block\-energy exponentssand dimension exponentgghas spectral tail index\(1\+g\)/\(1\+s\+g\)\(1\+g\)/\(1\+s\+g\)\. The high\-energy component therefore has index1/\(1\+χ\)1/\(1\+\\chi\)in the easy source and\(1\+γ\)/\(1\+χ\+γ\)\(1\+\\gamma\)/\(1\+\\chi\+\\gamma\)in the hard source; the condition onδ\\deltamakes the low\-energy component lower order\. Applying[Corollary3\.2](https://arxiv.org/html/2609.13500#S3.Thmtheorem2)yields all four exponents\. The full construction and converse are in Appendix[I](https://arxiv.org/html/2609.13500#A9)\.

Pairing changes the coordinate\-energy distribution while preserving both block marginals\. This is the information a scaling descriptor must retain;[Equation3\.4](https://arxiv.org/html/2609.13500#S3.E4)recovers it from the integrated tail\.

## 4Learning the Causal Self\-Attention Predictor

The value theorem reveals the relevant coordinate\. We now learn how to select it from the candidate tokens, then learn its prediction parameter\. Training has two stages: route calibration \(Stage A\) followed by value learning \(Stage V\)\. Both use exact\-arithmetic stochastic gradient descent \(SGD\); their sample and state costs are accounted for separately\.

#### From a route table to query–key attention\.

Letxq,xs∈ℝdx\_\{q\},x\_\{s\}\\in\\mathbb\{R\}^\{d\}be the query token and candidate token at positionss\. Their fixed embeddings include orthogonal address blocks:PQ∈ℝK×dP\_\{Q\}\\in\\mathbb\{R\}^\{K\\times d\}extracts the query row andPC∈ℝK×dP\_\{C\}\\in\\mathbb\{R\}^\{K\\times d\}extracts the candidate position, soPQ​xq=eqP\_\{Q\}x\_\{q\}=e\_\{q\}andPC​xs=esP\_\{C\}x\_\{s\}=e\_\{s\}, whereese\_\{s\}is thess\-th standard basis vector\. Writedh≥Kd\_\{h\}\\geq Kfor the attention\-head dimension\. An isometric embeddingJ∈ℝdh×KJ\\in\\mathbb\{R\}^\{d\_\{h\}\\times K\}, satisfyingJ⊤​J=IKJ^\{\\top\}J=I\_\{K\}, places these addresses in the head\. The trainable matrixA∈ℝK×KA\\in\\mathbb\{R\}^\{K\\times K\}stores one score for each query–position pair\. Define the fixed key map and trainable query map by

WK=dh​J​PC,WQ​\(A\)=J​A⊤​PQ,⟨WQ​\(A\)​xq,WK​xs⟩dh=Aq​s\.W\_\{K\}=\\sqrt\{d\_\{h\}\}JP\_\{C\},\\qquad W\_\{Q\}\(A\)=JA^\{\\top\}P\_\{Q\},\\qquad\\frac\{\\langle W\_\{Q\}\(A\)x\_\{q\},W\_\{K\}x\_\{s\}\\rangle\}\{\\sqrt\{d\_\{h\}\}\}=A\_\{qs\}\.\(4\.1\)
###### Proposition 4\.1\(Exact query–key realization; informal version of[PropositionC\.3](https://arxiv.org/html/2609.13500#A3.Thmtheorem3)\)\.

With a strict causal mask over theKKpreceding candidates,[Equation4\.1](https://arxiv.org/html/2609.13500#S4.E1)exactly realizes the route scores and attention\. FixWKW\_\{K\}and train onlyWQW\_\{Q\}’s query\-address block\. Euclidean SGD then gives the sameAAupdate, attended feature, and output\-vector gradient\.

The fixed embedding, key projection, and value extraction contain neither the unknown route nor the unknown logits\. Those quantities enter only the trained query\-address block and output coordinates\.

#### Learning the route\.

For a fixed query row, calibration increases the matching score relative to its distractors\. Zero initialization keeps the distractor scores equal\. Their common gap from the matching score is the marginβt\\beta\_\{t\}afterttvisits to that row;δt\\delta\_\{t\}is the total attention probability on incorrect candidates\. The following result quantifies how calibration reduces this leakage\.

###### Proposition 4\.2\(Online routing dynamics; informal version of[PropositionC\.4](https://arxiv.org/html/2609.13500#A3.Thmtheorem4)\)\.

ForK≥2K\\geq 2, zero initialization and softmax cross\-entropy steps of size0<η≤10<\\eta\\leq 1give the margin and distractor mass afterttrow visits:

βt\+1=βt\+η​Keβt\+K−1,δt=K−1eβt\+K−1\.\\beta\_\{t\+1\}=\\beta\_\{t\}\+\\eta\\frac\{K\}\{e^\{\\beta\_\{t\}\}\+K\-1\},\\qquad\\delta\_\{t\}=\\frac\{K\-1\}\{e^\{\\beta\_\{t\}\}\+K\-1\}\.AfterbK=Θη​\(log⁡\(e​K\)\)b\_\{K\}=\\Theta\_\{\\eta\}\(\\log\(eK\)\)visits,δt≍η\[1\+\(t−bK\)\+\]−1\\delta\_\{t\}\\asymp\_\{\\eta\}\[1\+\(t\-b\_\{K\}\)\_\{\+\}\]^\{\-1\}\. ForSAS\_\{A\}independent calibration blocks and a held\-out row,

𝔯K​\(SA\)=𝔼⁡\[δN2\],N∼Binomial⁡\(SA,1/K\)\.\\mathfrak\{r\}\_\{K\}\(S\_\{A\}\)=\\mathbb\{E\}\[\\delta\_\{N\}^\{2\}\],\\quad N\\sim\\operatorname\{Binomial\}\(S\_\{A\},1/K\)\.For0<ε≤1/160<\\varepsilon\\leq 1/16, the time to𝔯K≤ε\\mathfrak\{r\}\_\{K\}\\leq\\varepsilonisΘη\(K\{ε−1/2\+log\(eK\)\}\)\\Theta\_\{\\eta\}\(K\\\{\\varepsilon^\{\-1/2\}\+\\log\(eK\)\\\}\); atK=1K=1, leakage and calibration cost vanish\.

#### Learning and storing the values\.

After calibration, we freeze the route and fit an output vector onAn=\{i:qi≥n−1\}A\_\{n\}=\\\{i:q\_\{i\}\\geq n^\{\-1\}\\\}\. This active set contains coordinates whose energy is large enough to resolve at sample sizenn\. The natural box is𝒲=∏i∈An\[−Ri,Ri\]\\mathcal\{W\}=\\prod\_\{i\\in A\_\{n\}\}\[\-R\_\{i\},R\_\{i\}\]; output\-vector entries outsideAnA\_\{n\}are fixed at zero\. The attended featurehth\_\{t\}contains the attention\-weighted coordinate indicators restricted toAnA\_\{n\}\. Letμsgd=916​min\|z\|≤R​σ′​\(z\)\>0\\mu\_\{\\rm sgd\}=\\frac\{9\}\{16\}\\min\_\{\|z\|\\leq R\}\\sigma^\{\\prime\}\(z\)\>0be the step\-size constant used in the formal theorem\. Starting fromw1=0w\_\{1\}=0, useηt=2/\[μsgd​\(t\+1\)\]\\eta\_\{t\}=2/\[\\mu\_\{\\rm sgd\}\(t\+1\)\]and returnw¯n=2​∑t=1nt​wt/\[n⁡\(n\+1\)\]\\bar\{w\}\_\{n\}=2\\sum\_\{t=1\}^\{n\}tw\_\{t\}/\[n\(n\+1\)\]\. IfAnA\_\{n\}is empty, return zero\. ProjectionΠ𝒲D\\Pi\_\{\\mathcal\{W\}\}^\{D\}uses the frequency\-weighted norm‖v‖D2=∑i∈Anpi​vi2\\\|v\\\|\_\{D\}^\{2\}=\\sum\_\{i\\in A\_\{n\}\}p\_\{i\}v\_\{i\}^\{2\}\. On the active coordinates,DDis restricted toAnA\_\{n\}, and the update is

wt\+1=Π𝒲D​\[wt−ηt​D−1​\{σ⁡\(wt⊤​ht\)−Yt\}​ht\],D=diag⁡\(p\)\.w\_\{t\+1\}=\\Pi\_\{\\mathcal\{W\}\}^\{D\}\\\!\\left\[w\_\{t\}\-\\eta\_\{t\}D^\{\-1\}\\\{\\sigma\(w\_\{t\}^\{\\top\}h\_\{t\}\)\-Y\_\{t\}\\\}h\_\{t\}\\right\],\\qquad D=\\operatorname\{diag\}\(p\)\.\(4\.2\)
The final vector is encoded in coordinatewise scalar formats usingBVB\_\{V\}value\-state bits\. The format\-range ratioκi\\kappa\_\{i\}compares its stored range with the logit radiusRiR\_\{i\}; unit ratios recover the representation problem of Section[3](https://arxiv.org/html/2609.13500#S3)\. We writeF^\\widehat\{F\}for the resulting causal predictor andℛθ​\(F^\)\\mathcal\{R\}\_\{\\theta\}\(\\widehat\{F\}\)for its population cross\-entropy on a complete prediction block\. The termℰarithmetic\\mathcal\{E\}\_\{\\rm arithmetic\}accounts for deployment arithmetic error under the formal theorem’s numerical assumptions\.

###### Theorem 4\.3\(Causal self\-attention realization; informal version of[TheoremC\.5](https://arxiv.org/html/2609.13500#A3.Thmtheorem5)\)\.

The weighted\-average SGD with scalar formats in[TheoremC\.5](https://arxiv.org/html/2609.13500#A3.Thmtheorem5)satisfies

𝔼\[ℛθ\(F^\)−ℒ∞\]≲R,K,ηΦμ\(n−1\)\+Dq,κfmt\(BV\)\+𝔯K\(SA\)\+ℰarithmetic,\\mathbb\{E\}\[\\mathcal\{R\}\_\{\\theta\}\(\\widehat\{F\}\)\-\\mathcal\{L\}\_\{\\infty\}\]\\lesssim\_\{R,K,\\eta\}\\Phi\_\{\\mu\}\(n^\{\-1\}\)\+D^\{\\rm fmt\}\_\{q,\\kappa\}\(B\_\{V\}\)\+\\mathfrak\{r\}\_\{K\}\(S\_\{A\}\)\+\\mathcal\{E\}\_\{\\rm arithmetic\},\(4\.3\)whereBVB\_\{V\}is the value\-state budget andκi≥1\\kappa\_\{i\}\\geq 1the public format\-range ratio, with

Dq,κfmt​\(BV\)=infbi≥0,∑ibi≤BV∑iqi​min⁡\{1,κi2​2−2​bi\}\.D^\{\\rm fmt\}\_\{q,\\kappa\}\(B\_\{V\}\)=\\inf\_\{b\_\{i\}\\geq 0,\\,\\sum\_\{i\}b\_\{i\}\\leq B\_\{V\}\}\\sum\_\{i\}q\_\{i\}\\min\\\{1,\\kappa\_\{i\}^\{2\}2^\{\-2b\_\{i\}\}\\\}\.Forκi=1\\kappa\_\{i\}=1, this isΦμ​\(τBV\)\\Phi\_\{\\mu\}\(\\tau\_\{B\_\{V\}\}\), and the core surface holds at total budgetBBfor power\-law spectra, fixedKK,o⁡\(B\)o\(B\)routing bits and lower\-order routing/arithmetic errors\.

Conditioned on the route, output risk is strongly convex on fitted coordinates in the frequency\-weighted norm; preconditioning balances their learning rates\. Appendix[B](https://arxiv.org/html/2609.13500#A2)gives gradient, state, and format details\.

#### The total resources\.

WriteBAB\_\{A\}for the bits encoding the route, so total learned state isB=BV\+BAB=B\_\{V\}\+B\_\{A\}, and total training blocks areN=n\+SAN=n\+S\_\{A\}\. Letq\(r\)q\_\{\(r\)\}list coordinate energies in decreasing order, with decay exponentξ\>0\\xi\>0\. Charging both stages preserves the leading resource exponents: forq\(r\)≍r−\(1\+ξ\)q\_\{\(r\)\}\\asymp r^\{\-\(1\+\\xi\)\}, fixedKK, unit midpoint formats and the deployment\-error condition in[PropositionS\.1](https://arxiv.org/html/2609.13500#A19.Thmtheorem1), riskε\\varepsilonis achieved withN=n\+SA=O\(ε−\(1\+ξ\)/ξ\)N=n\+S\_\{A\}=O\(\\varepsilon^\{\-\(1\+\\xi\)/\\xi\}\)total blocks andB=BV\+BA=O\(ε−1/ξ\)B=B\_\{V\}\+B\_\{A\}=O\(\\varepsilon^\{\-1/\\xi\}\)total learned bits\. Calibration blocks and route\-state bits are lower\-order terms\.

## 5Adaptive Representation and Finite\-Precision Realization

Two representation questions remain after the resource law\. How should bits be allocated when the energy decay is unknown? How accurately does a finite\-precision attention network evaluate the retained parameters? The first result constructs an estimator and a storage schedule; the second maps supplied parameters to prediction error after numerical evaluation\. Here scalar coordinates are indexed byjj, andΔj\\Delta\_\{j\}denotes the radius writtenRiR\_\{i\}in Section[3](https://arxiv.org/html/2609.13500#S3)\. The lossℒθ​\(f\)\\mathcal\{L\}\_\{\\theta\}\(f\)is the same population cross\-entropy\.

#### Adaptive representation\.

###### Theorem 5\.1\(Adaptation with a known ordering; informal version of[TheoremH\.1](https://arxiv.org/html/2609.13500#A8.Thmtheorem1)\)\.

Fix0<χmin≤χmax<∞0<\\chi\_\{\\min\}\\leq\\chi\_\{\\max\}<\\infty, and suppose the canonical mode order is known\. Assumeqj≍j−\(1\+χ\)q\_\{j\}\\asymp j^\{\-\(1\+\\chi\)\}with common comparison constants and unknownχ∈\[χmin,χmax\]\\chi\\in\[\\chi\_\{\\min\},\\chi\_\{\\max\}\]\. For everyn,B∈ℕn,B\\in\\mathbb\{N\}, one clipped likelihood estimator and one deterministic integer bit schedule, both independent ofχ\\chi, satisfy simultaneously throughout this interval

supθj∈\[−Δj,Δj\]𝔼θ\[ℒθ\(f^\)−ℒθ\(θ\)\]≲n−χ/\(1\+χ\)\+B−χ\.\\sup\_\{\\theta\_\{j\}\\in\[\-\\Delta\_\{j\},\\Delta\_\{j\}\]\}\\mathbb\{E\}\_\{\\theta\}\\\!\\left\[\\mathcal\{L\}\_\{\\theta\}\(\\widehat\{f\}\)\-\\mathcal\{L\}\_\{\\theta\}\(\\theta\)\\right\]\\lesssim n^\{\-\\chi/\(1\+\\chi\)\}\+B^\{\-\\chi\}\.\(5\.1\)

ChooseA\>χmax/2A\>\\chi\_\{\\max\}/2and the largest prefixmmsatisfying

bj=⌊A​log2⁡\(m/j\)⌋\+\(j≤m\),∑j≤mbj≤B\.b\_\{j\}=\\left\\lfloor A\\log\_\{2\}\(m/j\)\\right\\rfloor\_\{\+\}\\quad\(j\\leq m\),\\qquad\\sum\_\{j\\leq m\}b\_\{j\}\\leq B\.\(5\.2\)Thenm≍Bm\\asymp B, and both quantization and omitted\-tail errors areO⁡\(m−χ\)O\(m^\{\-\\chi\}\)uniformly over the exponent interval\. The logarithmic schedule spends most bits on the leading coordinates while matching the tail error for every allowed decay rate\.

#### Finite\-precision self\-attention\.

We construct a network that evaluates a supplied logit table for the firstMMcoordinates and uses zero for the rest\. HereMMis the retained prefix length,t⋆t\_\{\\star\}is the matching candidate position, andβ≥0\\beta\\geq 0controls the matching attention score\. The candidate’s mode labeljtj\_\{t\}identifies its scalar table entry\. Use public basis addressesat∈ℝKa\_\{t\}\\in\\mathbb\{R\}^\{K\}, mode vectorsej\(M\)=eje\_\{j\}^\{\(M\)\}=e\_\{j\}forj≤Mj\\leq Mand zero otherwise, andd=K\+M\+1d=K\+M\+1\. Candidate and query tokens arext=\(at,ejt\(M\),0\)x\_\{t\}=\(a\_\{t\},e\_\{j\_\{t\}\}^\{\(M\)\},0\)andxq=\(at⋆,0,1\)x\_\{q\}=\(a\_\{t\_\{\\star\}\},0,1\)\. The fixed\-key construction in[Proposition4\.1](https://arxiv.org/html/2609.13500#S4.Thmtheorem1)gives

⟨WQ​xq,WK​xt⟩dh=β⁡⟨at⋆,at⟩\.\\frac\{\\langle W\_\{Q\}x\_\{q\},W\_\{K\}x\_\{t\}\\rangle\}\{\\sqrt\{d\_\{h\}\}\}=\\beta\\langle a\_\{t\_\{\\star\}\},a\_\{t\}\\rangle\.\(5\.3\)With the value projection onto mode coordinates, a strict causal mask, and an output vectorwwsupported only on those coordinates,

hq=∑t=1Kαt​e¯jt\(M\),αt=exp\(β𝟏\{t=t⋆\}\)eβ\+K−1,z=w⊤​hq,h\_\{q\}=\\sum\_\{t=1\}^\{K\}\\alpha\_\{t\}\\bar\{e\}\_\{j\_\{t\}\}^\{\(M\)\},\\qquad\\alpha\_\{t\}=\\frac\{\\exp\(\\beta\\mathbf\{1\}\\\{t=t\_\{\\star\}\\\}\)\}\{e^\{\\beta\}\+K\-1\},\\qquad z=w^\{\\top\}h\_\{q\},\(5\.4\)wheree¯j\(M\)=\(0,ej\(M\),0\)\\bar\{e\}\_\{j\}^\{\(M\)\}=\(0,e\_\{j\}^\{\(M\)\},0\)\. Writewjmode=wK\+jw\_\{j\}^\{\\rm mode\}=w\_\{K\+j\}, and setϑj=wjmode\\vartheta\_\{j\}=w\_\{j\}^\{\\rm mode\}forj≤Mj\\leq Mandϑj=0\\vartheta\_\{j\}=0otherwise\.

The numerical calculation below uses unit roundoffuu, an upper bound on the relative error of the listed scalar operations\. Its four error terms respectively measure selecting a distractor, arithmetic rounding, storing the retained logits, and predicting omitted coordinates with zero\.

###### Theorem 5\.2\(Finite\-precision realization; informal version of[TheoremJ\.1](https://arxiv.org/html/2609.13500#A10.Thmtheorem1)\)\.

FixK,M≥1K,M\\geq 1,R<∞R<\\infty,β≥0\\beta\\geq 0, and the construction above\.

1. 1\.In exact arithmetic, if\|wjmode\|≤R\\lvert w\_\{j\}^\{\\rm mode\}\\rvert\\leq Rfor everyj∈\[M\]j\\in\[M\], then a query whose matching candidate has any modej≥1j\\geq 1satisfies \|z−ϑj\|≤2​R​\(K−1\)​e−β\.\\lvert z\-\\vartheta\_\{j\}\\rvert\\leq 2R\(K\-1\)e^\{\-\\beta\}\.\(5\.5\)
2. 2\.Setwjmode=θjw\_\{j\}^\{\\rm mode\}=\\theta\_\{j\}forj∈\[M\]j\\in\[M\]and store it withbj∈ℕ0b\_\{j\}\\in\\mathbb\{N\}\_\{0\}bits on\[−Δj,Δj\]\[\-\\Delta\_\{j\},\\Delta\_\{j\}\]\. Assume stable max\-shifted softmax with exactly representable shifted scores0,−β0,\-\\betaand an exponent range containing their exponentials\. Exponentials, multiplications, and divisions have relative error at mostuu; sums use balanced pairwise accumulation\. For the resulting predictorffpf\_\{\\rm fp\}, if\(⌈log2⁡K⌉\+4\)​u≤1/4\(\\lceil\\log\_\{2\}K\\rceil\+4\)u\\leq 1/4, then the held\-out excess cross\-entropy obeys ℒθ​\(ffp\)−ℒ∞≤CR​\{\(K−1\)2​e−2​β\+u2​\(1\+log⁡K\)2\+∑j=1Mqj​2−2​bj\+∑j\>Mqj\}\.\\mathcal\{L\}\_\{\\theta\}\(f\_\{\\rm fp\}\)\-\\mathcal\{L\}\_\{\\infty\}\\leq C\_\{R\}\\left\\\{\(K\-1\)^\{2\}e^\{\-2\\beta\}\+u^\{2\}\(1\+\\log K\)^\{2\}\+\\sum\_\{j=1\}^\{M\}q\_\{j\}2^\{\-2b\_\{j\}\}\+\\sum\_\{j\>M\}q\_\{j\}\\right\\\}\.\(5\.6\)
3. 3\.Within the separable scalar\-table representation in which coordinatejjhas at most2bj2^\{b\_\{j\}\}values, the bit\-dependent term in[Equation5\.6](https://arxiv.org/html/2609.13500#S5.E6)is tight up to constants\.

Here and below,CRC\_\{R\}depends only on the logit boundRR\.

The bound separates routing, arithmetic, quantization, and omitted energy\. Pairwise summation gives logarithmic dependence onKKin arithmetic error before squaring\. This result evaluates supplied logits;[Theorem4\.3](https://arxiv.org/html/2609.13500#S4.Thmtheorem3)controls their learning\. Dense storage accounting and proofs appear in Appendices[B](https://arxiv.org/html/2609.13500#A2),[H](https://arxiv.org/html/2609.13500#A8), and[J](https://arxiv.org/html/2609.13500#A10)\.

## 6Bit Complexity and Compute–Precision Scaling

Figure 1:Retained precision sets resource ceilings and error floors\.Left: analytic bit\-operation ratio with retained high\-precision work\. Right: risk–cost laws under the stated arithmetic model; fixed softmax precision creates a numerical floor\.Representation accuracy is useful only if the chosen arithmetic can preserve it\. This section separates the precision needed as the represented prefix grows, the work of a specified dense graph, and the savings from lowering precision at a fixed shape\.

#### Error and cost as model size grows\.

HereMMis the number of represented scalar modes,bbis arithmetic precision, andχ\>0\\chi\>0is the model\-error decay exponent\. The operation exponentf≥0f\\geq 0describes per\-block work;ν≥0\\nu\\geq 0describes how scalar\-operation cost grows with bit width\. The functionκ⁡\(M\)\\kappa\(M\)amplifies stored perturbations into prediction error\. Suppose representingMMmodes has errorΘ⁡\(M−χ\)\\Theta\(M^\{\-\\chi\}\), each processed block costsΘ⁡\(Mf\)\\Theta\(M^\{f\}\)scalar operations,Θ⁡\(M1\+χ\)\\Theta\(M^\{1\+\\chi\}\)blocks are necessary and sufficient, abb\-bit operation costsΘ⁡\(bν\)\\Theta\(b^\{\\nu\}\), and stored perturbation2−b2^\{\-b\}produces errorΘ⁡\(κ​\(M\)2​2−2​b\)\\Theta\(\\kappa\(M\)^\{2\}2^\{\-2b\}\)\. Under these two\-sided assumptions,ℛ⁡\(C\)\\mathcal\{R\}\(C\)is the best held\-out risk at arithmetic budgetCC\.

#### Counting a dense forward–backward pass\.

To relate these abstract costs to an implementation, fix a conventional Transformer graph\. Its matrix arithmetic and softmax arithmetic can use different precisions, denotedbmmb\_\{\\rm mm\}andbsmb\_\{\\rm sm\}\. For a dense Transformer with sequence lengthℓ\\ell, widthdd, feed\-forward widthmffm\_\{\\rm ff\},HHheads, andLLlayers, its conventional forward–backward arithmetic is, up to constants,

L⁡\[\(ℓ​d2\+ℓ​d​mff\+ℓ2​d\)​𝖬​\(bmm\)\+H​ℓ2​𝖲​\(bsm\)\]\.L\\\!\\left\[\(\\ell d^\{2\}\+\\ell dm\_\{\\rm ff\}\+\\ell^\{2\}d\)\\mathsf\{M\}\(b\_\{\\rm mm\}\)\+H\\ell^\{2\}\\mathsf\{S\}\(b\_\{\\rm sm\}\)\\right\]\.Here𝖬⁡\(bmm\)\\mathsf\{M\}\(b\_\{\\rm mm\}\)and𝖲⁡\(bsm\)\\mathsf\{S\}\(b\_\{\\rm sm\}\)are the bit\-operation costs of a matrix multiply–accumulate and a softmax scalar operation at their respective precisions\. This forward–backward count excludes optimizer updates\. Recomputation reduces saved attention state while preserving the dense arithmetic order\.

###### Theorem 6\.1\(Precision\-aware compute phases; informal version of[TheoremM\.1](https://arxiv.org/html/2609.13500#A13.Thmtheorem1)\)\.

For the two\-sided risk–cost family of[TheoremM\.1](https://arxiv.org/html/2609.13500#A13.Thmtheorem1), withχ\>0\\chi\>0,f,ν≥0f,\\nu\\geq 0, andn⁡\(M\)≍M1\+χn\(M\)\\asymp M^\{1\+\\chi\}:

1. 1\.If1≲κ⁡\(M\)≲Mρ1\\lesssim\\kappa\(M\)\\lesssim M^\{\\rho\}, then the least model\-error\-matching precision isb=Θ⁡\(log⁡M\)b=\\Theta\(\\log M\), and ℛ\(C\)−ℒ∞≍C−χ/\(f\+1\+χ\)\(logC\)χ​ν/\(f\+1\+χ\)\\mathcal\{R\}\(C\)\-\\mathcal\{L\}\_\{\\infty\}\\asymp C^\{\-\\chi/\(f\+1\+\\chi\)\}\(\\log C\)^\{\\chi\\nu/\(f\+1\+\\chi\)\}\(6\.1\)up to other slowly varying factors specified in the assumptions\.
2. 2\.Ifκ⁡\(M\)=exp⁡\(Θ⁡\(Mλ\)\)\\kappa\(M\)=\\exp\(\\Theta\(M^\{\\lambda\}\)\),λ\>0\\lambda\>0, thenb=Θ⁡\(Mλ\)b=\\Theta\(M^\{\\lambda\}\), and the loss exponent isχ/\(f\+1\+χ\+λ​ν\)\\chi/\(f\+1\+\\chi\+\\lambda\\nu\)\.
3. 3\.Ifqj≍j−\(1\+χ\)q\_\{j\}\\asymp j^\{\-\(1\+\\chi\)\}, adaptive per\-mode precision has errorΘ⁡\(B−χ\)\\Theta\(B^\{\-\\chi\}\), whereas a uniform\-width prefix has optimal error Θ⁡\(\(B/log⁡B\)−χ\)\.\\Theta\(\(B/\\log B\)^\{\-\\chi\}\)\.\(6\.2\)

Each converse uses the representation, operation set, storage layout, and bit\-operation cost specified above\.

The polynomial\-conditioning exponentρ\\rhoin this theorem describesκ⁡\(M\)\\kappa\(M\), independently of the spectral tail index used in Section[3](https://arxiv.org/html/2609.13500#S3)\. Matching numerical error toM−χM^\{\-\\chi\}requires

b≥log2⁡κ⁡\(M\)\+χ2​log2​M−O⁡\(1\)\.b\\geq\\log\_\{2\}\\kappa\(M\)\+\\frac\{\\chi\}\{2\}\\log\_\{2\}M\-O\(1\)\.\(6\.3\)

#### Savings at a fixed shape\.

We now hold model dimensions fixed and change only the numerical formats\. This compares two implementations of the same graph\. Retained high\-precision work produces an Amdahl\-type ceiling: reducing one part’s cost leaves the other part unchanged\. At fixed shape, letG,SG,Scount matrix and softmax operations,ν,νsm\\nu,\\nu\_\{\\rm sm\}their precision\-cost exponents,QQthe quantized scalar count, andRhpR\_\{\\rm hp\}the retained high\-precision bits\. Lowering the matrix and stored\-scalar precision from baselineb0b\_\{0\}tobbgives

BitOpsRatio=G​b0ν\+S​bsmνsmG​bν\+S​bsmνsm,Compressionbit=Q​b0\+RhpQ​b\+Rhp\.\\operatorname\{BitOpsRatio\}=\\frac\{Gb\_\{0\}^\{\\nu\}\+Sb\_\{\\rm sm\}^\{\\nu\_\{\\rm sm\}\}\}\{Gb^\{\\nu\}\+Sb\_\{\\rm sm\}^\{\\nu\_\{\\rm sm\}\}\},\\qquad\\operatorname\{Compression\}\_\{\\rm bit\}=\\frac\{Qb\_\{0\}\+R\_\{\\rm hp\}\}\{Qb\+R\_\{\\rm hp\}\}\.In this calculation,QQcounts quantized scalars rather than query rows, andSScounts softmax operations rather than learned states\. Ratios larger than one denote reduced bit operations or storage\. Thus fixed\-precision softmax, master weights, optimizer state, or saved high\-precision activations can dominate even when matrix precision falls\.

Appendix[B](https://arxiv.org/html/2609.13500#A2)derives the dense forward–backward count, storage requirements, and precision–risk trade\-off\.

## 7Experiments

#### Experimental design\.

We test spectral risk, coupling, and attention learning\. Training and held\-out blocks are independent, and comparisons are paired by seed or evaluation item\. The fixed\-source synthetic studies evaluate excess cross\-entropy \(CE\), the loss above the source’s Bayes predictor and use 95% bias\-corrected and accelerated \(BCa\) bootstrap intervals over paired seeds\. Complementary mini\-batch and terminal\-iterate studies test robustness to the learning implementation\. Appendices[Q](https://arxiv.org/html/2609.13500#A17)and[R](https://arxiv.org/html/2609.13500#A18)specify the algorithms, evaluation units, and uncertainty procedures\.

Figure 2:One spectrum organizes data and memory\.Four fixed sources, each with 256 energy groups, atK=32K=32: held\-out excess CE versusGμ​\(n,B\)=Φμ​\(n−1\)\+Φμ​\(τB\)G\_\{\\mu\}\(n,B\)=\\Phi\_\{\\mu\}\(n^\{\-1\}\)\+\\Phi\_\{\\mu\}\(\\tau\_\{B\}\)\. Blue circles: weighted\-average SGD; orange squares: the oracle\-coordinate clipped estimator\. Each curve averages 30 budget pairs into six bins chosen only fromGμG\_\{\\mu\}, within each seed\. Bands: 95% BCa over eight paired seeds\.
#### Data–memory surface and exponent agreement\.

Does the same source statistic organize both resources in the stated learner? We fix four sources, each with 256 energy groups rather than 256 training examples, and cross five sample budgets with six value\-state budgets\. Each sample budget restarts the theorem’s weighted\-average SGD in FP64; all state budgets encode that same fitted vector\. Full\-source excess CE follows the resource ordering ofGμG\_\{\\mu\}for both SGD and the oracle\-coordinate estimator \([Figure2](https://arxiv.org/html/2609.13500#S7.F2)\)\. Unbinned risk\-to\-GμG\_\{\\mu\}ratios reveal the remaining finite\-resource variation \([Figure11](https://arxiv.org/html/2609.13500#A18.F11)\)\. The six\-profile exponent grid and candidate\-count replication appear in Appendix[P](https://arxiv.org/html/2609.13500#A16), alongside the spectral slopes \([Table4\(b\)](https://arxiv.org/html/2609.13500#A16.T4.st2)\)\.

#### Same\-marginal coupling\.

Can identical block\-energy and block\-dimension marginals produce different data laws as well as different memory laws? On the same normalized dyadic source, the fitted easy/hard data exponents are0\.509/0\.3390\.509/0\.339forχ=1\\chi=1and0\.681/0\.4890\.681/0\.489forχ=2\\chi=2, against predictions1/2,1/31/2,1/3and2/3,1/22/3,1/2, respectively\. Fits use the same last four sample budgets for all 16 seeds\.[Figure3](https://arxiv.org/html/2609.13500#S7.F3)\(a,d\) shows every budget; panels \(b,e\) and[Table2\(b\)](https://arxiv.org/html/2609.13500#S7.T2.st2)retain the complementary storage separation\. Pairing changes both resource exponents, not just their constants\.

Figure 3:Pairing separates data and memory; routing enables learning\.\(a,d\) Fixed\-source data sweeps forχ=1,2\\chi=1,2; blue/orange denote easy/hard\. \(b,e\) Their complementary storage sweeps, evaluated by population excess CE\. \(c\) Squared leakage versus row visits,K=8/32K=8/32\(blue/green\)\. \(f\) Finite\-source excess CE with learned/known/frozen routes \(blue circles/green triangles/orange squares\), without an added asymptotic tail\. Intervals: 95% BCa over 16 data seeds \(a,d\), 16 source amplitudes \(b,e\), or eight paired seeds \(c,f\)\.
#### Learned routing and interventions\.

Routing experiments train routing and output parameters through the actual network forward map\. Its 256 combinations of source profile, represented prefix, candidate count, and seed separately restart each learner at three nominal sample budgets and reuse fitted states across bit budgets\. Routing leakage decreases with actual visits, and learned routing approaches the known\-route learner while a frozen uniform route does not \([Figure3](https://arxiv.org/html/2609.13500#S7.F3)\(c,f\)\)\.[Tables2\(c\)](https://arxiv.org/html/2609.13500#S7.T2.st3)and[2\(d\)](https://arxiv.org/html/2609.13500#S7.T2.st4)separates data, tail, routing, and format terms and changes routing, preconditioning, or precision one factor at a time\. At the unit\-bit\-ratio setting in Appendix[Table3](https://arxiv.org/html/2609.13500#A16.T3), learned and known routing yield0\.0191±0\.00020\.0191\\pm 0\.0002and0\.0192±0\.00020\.0192\\pm 0\.0002augmented loss, versus0\.0605±0\.00080\.0605\\pm 0\.0008with a frozen route; these augmented losses include the commonM−χM^\{\-\\chi\}proxy\. Removing preconditioning gives0\.0230±0\.00030\.0230\\pm 0\.0003\. The half\-bit\-ratio table reveals the uniform\-precision penalty at a tighter memory budget, complementing this unit\-bit\-ratio comparison\.

Figure 4:Data and deployed storage in Pythia\.\(a–f\) 70M, 160M, 410M, 1B, 1\.4B, 2\.8B test CE after2262^\{26\}extra tokens\. Blue: 2/3/4/6/8\-bit and BF16 \(left to right\); dashed orange: additive fits to first\-three\-budget 4/8\-bit/BF16 validation\. All points hold out data; 6\-bit also holds out format\. Shared log\-y; bands: 95% BCa, eight paired training seeds\.
#### Stored formats and softmax precision\.

We compare a shared quantization scale, scales per energy group or per coordinate, and adaptive bit widths\. Three conditioning regimes vary the format\-range amplification relative to logit radii\. A softmax sweep records numerical format, underflow, and matching mass atK∈\{8,32,128,512\}K\\in\\\{8,32,128,512\\\}\. At eight bits, adaptive widths reduce format distortion to7\.82×10−107\.82\\times 10^\{\-10\}, compared with8\.46×10−68\.46\\times 10^\{\-6\}for uniform block widths and1\.08×10−51\.08\\times 10^\{\-5\}for a shared scale\. Appendix[Figure8](https://arxiv.org/html/2609.13500#A16.F8)shows the format trade\-off and retained high\-precision work: bit allocation controls distortion, while the fraction quantized determines resource savings\.

#### Profile robustness\.

We test broken\-power\-law, logarithmically corrected, and perturbed\-order profiles using each source’s exact finite spectrum\. The empirical allocator uses fitted logits and counts to estimate energies, subtracting a sampling\-noise correction\. Known\-energy allocation instead minimizes representation distortion using the true energies; uniform widths provide a second reference\. Comparisons use eight paired seeds\. Appendix[R](https://arxiv.org/html/2609.13500#A18)gives both objectives, distinct from the deterministic unknown\-exponent schedule of[Theorem5\.1](https://arxiv.org/html/2609.13500#S5.Thmtheorem1)\. It retains the advantage over uniform allocation across the three displayed deviations \(Appendix[Figure6](https://arxiv.org/html/2609.13500#A16.F6)\)\.

Table 2:Deployment, pairing, and learning mechanisms\.\(a\) HQQ\-g32 minus BF16 CE: 96 WikiText\-103 windows, 512 LAMBADA items\. \(b\) Storage slopes, 16 seeds; E/H: easy/hard\. \(c,d\) SGD decomposition/interventions:10−310^\{\-3\}nats, eight seeds; \(d\) has\(χ,γ\)=\(1,\.5\)\(\\chi,\\gamma\)=\(1,\.5\),M=64M=64,K=32K=32, bit ratio \.5 \(Table[3](https://arxiv.org/html/2609.13500#A16.T3): 1\)\.±\\pm: rounded 95% BCa envelopes \(Unif\.: constant across seeds\)\. Bold/underline: lowest/second\-lowest in \(a,d\)\.\(a\)Pythia weight\-only PTQ; lower is better\.ModelWikiTextLAMBADA70M3\.009±0\.037\\pm 0\.0373\.012±0\.386\\pm 0\.386160M1\.325±0\.020\\pm 0\.0200\.589±0\.225\\pm 0\.225410M0\.156±0\.003\\pm 0\.0030\.319±0\.082\\pm 0\.0821B0\.054±0\.002\\pm 0\.0020\.021±0\.028\\pm 0\.0281\.4B0\.066±0\.003\\pm 0\.0030\.149±0\.045\\pm 0\.0452\.8B0\.048±0\.003\\pm 0\.0030\.007±0\.034\\pm 0\.034

\(b\)Same\-marginal coupling\.\(χ,E/H\)\(\\chi,\\mathrm\{E/H\}\)TheoryHeld\-out CE\(1,E\)\(1,E\)1\.0000\.972±0\.0016\\pm 0\.0016\(1,H\)\(1,H\)0\.5000\.503±0\.0003\\pm 0\.0003\(2,E\)\(2,E\)2\.0002\.016±0\.0011\\pm 0\.0011\(2,H\)\(2,H\)1\.0001\.002±0\.0002\\pm 0\.0002

\(c\)Signed full\-source excess\-CE terms\.DataTailRouteFormat3\.584±\.219\\pm\.2191\.218±\.029\\pm\.029\.0270±\.0031\\pm\.00311\.491±\.275\\pm\.275

\(d\)Interventions; excess CE, lower is better\.FullKnownNo pre\.FrozenUnif\.19\.01±\.27\\pm\.2719\.06±\.26\\pm\.2622\.47±\.21\\pm\.2160\.50±\.74\\pm\.7489\.65±\.00\\pm\.00

#### Pretrained Transformers\.

Weight\-only post\-training quantization \(PTQ\) tests the deployment effect across six Pythia checkpoints atstep143000\. We compare tensor/channel 8\-bit integer formats \(INT8\), 4\-bit integer \(INT4\) round\-to\-nearest \(RTN\), and half\-quadratic quantization \(HQQ\) with groups of 128 or 32 weights \(g128/g32\) against bfloat16 \(BF16\), using paired WikiText\-103 windows and LAMBADA items\. The HQQ\-g32 penalty generally decreases with scale, except at 1\.4B; the 2\.8B LAMBADA interval includes zero \([Table2\(a\)](https://arxiv.org/html/2609.13500#S7.T2.st1)\)\. Appendix[Figures9](https://arxiv.org/html/2609.13500#A16.F9)and[4\(a\)](https://arxiv.org/html/2609.13500#A16.T4.st1)retain all formats and BF16 baselines\. Softmax remains in 32\-bit floating point \(FP32\), so the comparison isolates weight representation\.

We additionally continue all six checkpoints along four nested WikiText\-103 token budgets, using eight training seeds per model, and materialize group\-128 2/3/4/6/8\-bit and BF16 deployment packages\. The complete scan separates model\-scale and format effects \([Figure4](https://arxiv.org/html/2609.13500#S7.F4)\(a–f\)\): 2/3\-bit quantization causes the largest loss increases at 70M/160M, while the four larger models have substantially smaller penalties\. Including 2\-bit calibration yields fits within the specified\[0,20\]\[0,20\]\-nat prediction range at 410M–2\.8B\. At 70M/160M, the 2\-bit losses exceed this range; the fitting procedure does not yield converged, in\-range solutions\. Each scale is fitted separately\.

The sensitivity analysis fits 4/8\-bit and BF16 validation configurations and holds out 6\-bit plus the largest token budget\. On untouched test windows, the additive response improves data\-extrapolation root\-mean\-square error \(RMSE\) relative to the data\-only fit at every scale\. The reductions are0\.910​\[0\.871,0\.944\]0\.910\\,\[0\.871,0\.944\]for 70M and0\.380​\[0\.352,0\.394\]0\.380\\,\[0\.352,0\.394\]for 160M; the four larger models show smaller absolute improvements\. For 6\-bit interpolation, the additive response has higher RMSE at 70M/160M and lower RMSE at 410M–2\.8B\. Thus model scale changes both quantization sensitivity and the predictive value of a smooth joint\-resource response\. Appendix[Figure7](https://arxiv.org/html/2609.13500#A16.F7)shows every format and budget with seed\-level uncertainty;[Tables5](https://arxiv.org/html/2609.13500#A16.T5)and[6](https://arxiv.org/html/2609.13500#A16.T6)report the corresponding prediction errors for each model\.

## 8Conclusion

Finite data and finite learned memory resolve the same predictive\-energy spectrum at different thresholds\. For the causal source studied here, this gives minimax risk, scaling exponents, and a data–memory crossover\. The complete curve identifies the positive spectrum, retaining information that a fitted exponent alone loses\.

Energy–dimension pairing determines the spectrum: identical block marginals can conceal different data and memory laws\. A masked query–key head learns the relevant route and values, connecting this statistical structure to attention\. Adaptive allocation, finite\-precision realization, and conditional compute laws quantify how the representation uses resources\.

Experiments trace these mechanisms through spectral collapse, coupling, routing, and numerical precision\. The organizing principle is the distribution of predictive information across independent coordinates: it explains both what data can reveal and what a finite predictor can retain\.

## References

- Alabdulmohsinet al\.\(2022\)I\. M\. Alabdulmohsin, B\. Neyshabur, and X\. ZhaiRevisiting neural scaling laws in language and vision\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Allen\-Zhu and Li \(2025\)Z\. Allen\-Zhu and Y\. LiPhysics of language models: part 3\.3, knowledge capacity scaling laws\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=FxNNiUgtfa),2404\.05405Cited by:[Appendix A](https://arxiv.org/html/2609.13500#A1.SS0.SSS0.Px2.p1.2),[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.13500#S2.T1.6.4.1.1.1)\.
- Aroraet al\.\(2018\)S\. Arora, R\. Ge, B\. Neyshabur, and Y\. ZhangStronger generalization bounds for deep nets via a compression approach\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 254–263\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p1.1)\.
- Ataee Tarzanaghet al\.\(2023\)D\. Ataee Tarzanagh, Y\. Li, X\. Zhang, and S\. OymakMax\-margin token selection in attention mechanism\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Bahriet al\.\(2024\)Y\. Bahri, E\. Dyer, J\. Kaplan, J\. Lee, and U\. SharmaExplaining neural scaling laws\.Proceedings of the National Academy of Sciences121\(27\)\.External Links:[Document](https://dx.doi.org/10.1073/pnas.2311878121)Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Bartlettet al\.\(2017\)P\. L\. Bartlett, D\. J\. Foster, and M\. J\. TelgarskySpectrally\-normalized margin bounds for neural networks\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p1.1)\.
- Bartlettet al\.\(2019\)P\. L\. Bartlett, N\. Harvey, C\. Liaw, and A\. MehrabianNearly\-tight VC\-dimension and pseudodimension bounds for piecewise linear neural networks\.Journal of Machine Learning Research20\(63\),pp\. 1–17\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p1.1)\.
- Berget al\.\(2021\)T\. Berg, O\. Ordentlich, and O\. ShayevitzDeterministic finite\-memory bias estimation\.InProceedings of the 34th Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.134,pp\. 566–585\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px2.p1.1)\.
- Blumenfeldet al\.\(2019\)Y\. Blumenfeld, D\. Gilboa, and D\. SoudryA mean field theory of quantized deep networks: the quantization–depth trade\-off\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Bordelonet al\.\(2024\)B\. Bordelon, A\. Atanasov, and C\. PehlevanA dynamical model of neural scaling laws\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 4345–4382\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Bordelonet al\.\(2020\)B\. Bordelon, A\. Canatar, and C\. PehlevanSpectrum dependent learning curves in kernel regression and wide neural networks\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 1024–1034\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Caballeroet al\.\(2023\)E\. Caballero, K\. Gupta, I\. Rish, and D\. KruegerBroken neural scaling laws\.InThe Eleventh International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Cabanneset al\.\(2024\)V\. Cabannes, B\. Simsek, and A\. BiettiLearning associative memories with gradient descent\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 5114–5134\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Cheeet al\.\(2025\)J\. Chee, A\. Backurs, R\. Heck, L\. Zhang, J\. Kulkarni, T\. Rothvoss, and S\. GopiDiscQuant: a quantization method for neural networks inspired by discrepancy theory\.InProceedings of Thirty Eighth Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.291,pp\. 924–951\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2020\)J\. Chen, Y\. Gai, Z\. Yao, M\. W\. Mahoney, and J\. E\. GonzalezA statistical framework for low\-bitwidth training of deep neural networks\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 883–894\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Chuet al\.\(2024\)T\. Chu, Z\. Song, and C\. YangHow to protect copyright data in optimization of large language models?\.Proceedings of the AAAI Conference on Artificial Intelligence38\(16\),pp\. 17871–17879\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i16.29741),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/29741)Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Clarket al\.\(2022\)A\. Clark, D\. De Las Casas, A\. Guy, A\. Mensch, M\. Paganini, J\. Hoffmann, B\. Damoc, B\. Hechtman, T\. Cai, S\. Borgeaud, G\. B\. Van Den Driessche, E\. Rutherford, T\. Hennigan, M\. J\. Johnson, A\. Cassirer, C\. Jones, E\. Buchatskaya, D\. Budden, L\. Sifre, S\. Osindero, O\. Vinyals, M\. Ranzato, J\. Rae, E\. Elsen, K\. Kavukcuoglu, and K\. SimonyanUnified scaling laws for routed language models\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 4057–4086\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Daliriet al\.\(2025\)M\. Daliri, Z\. Song, and C\. YangUnlock the theory behind scaling 1\-bit neural networks\.InConference on Parsimony and Learning,Proceedings of Machine Learning Research, Vol\.280,pp\. 545–598\.External Links:[Link](https://proceedings.mlr.press/v280/daliri25a.html)Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Daniely and Granot \(2019\)A\. Daniely and E\. GranotGeneralization bounds for neural networks via approximate description length\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p1.1)\.
- Daoet al\.\(2022\)T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. RéFlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.InAdvances in Neural Information Processing Systems,Vol\.35\.External Links:2205\.14135Cited by:[5th item](https://arxiv.org/html/2609.13500#A15.I1.i5.p1.1)\.
- Denget al\.\(2024\)Y\. Deng, Z\. Song, J\. Xiong, and C\. YangHow sparse attention approximates exact attention? your attention is naturallynCn^\{C\}\-sparse\.External Links:2404\.02690Cited by:[Appendix A](https://arxiv.org/html/2609.13500#A1.SS0.SSS0.Px2.p1.2),[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Dettmerset al\.\(2022\)T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. ZettlemoyerLLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Dettmers and Zettlemoyer \(2023\)T\. Dettmers and L\. ZettlemoyerThe case for 4\-bit precision: k\-bit inference scaling laws\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 7750–7774\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Donget al\.\(2020\)Z\. Dong, Z\. Yao, D\. Arfeen, A\. Gholami, M\. W\. Mahoney, and K\. KeutzerHAWQ\-V2: hessian aware trace\-weighted quantization of neural networks\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Feldmanet al\.\(2025\)V\. Feldman, G\. Kornowski, and X\. LyuTrade\-offs in data memorization via strong data processing inequalities\.InProceedings of the 38th Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.291,pp\. 1935–1973\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px2.p1.1)\.
- Frantaret al\.\(2023\)E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. AlistarhOPTQ: accurate quantization for generative pre\-trained transformers\.InThe Eleventh International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Goelet al\.\(2026\)G\. Goel, M\. Soltanolkotabi, and P\. L\. BartlettTraining dynamics of softmax self\-attention: fast global convergence via preconditioning\.External Links:2603\.01514Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1),[Table 1](https://arxiv.org/html/2609.13500#S2.T1.6.6.1.1.1)\.
- Gray and Neuhoff \(1998\)R\. M\. Gray and D\. L\. NeuhoffQuantization\.IEEE Transactions on Information Theory44\(6\),pp\. 2325–2383\.External Links:[Document](https://dx.doi.org/10.1109/18.720541)Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px2.p1.1)\.
- Hafez\-Kolahiet al\.\(2021\)H\. Hafez\-Kolahi, B\. Moniri, S\. Kasaei, and M\. S\. BaghshahRate\-distortion analysis of minimum excess risk in bayesian learning\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 3998–4007\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px2.p1.1)\.
- Henighanet al\.\(2020\)T\. Henighan, J\. Kaplan, M\. Katz, M\. Chen, C\. Hesse, J\. Jackson, H\. Jun, T\. B\. Brown, P\. Dhariwal, S\. Gray, C\. Hallacy, B\. Mann, A\. Radford, A\. Ramesh, N\. Ryder, D\. M\. Ziegler, J\. Schulman, D\. Amodei, and S\. McCandlishScaling laws for autoregressive generative modeling\.External Links:2010\.14701Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Hestnesset al\.\(2017\)J\. Hestness, S\. Narang, N\. Ardalani, G\. Diamos, H\. Jun, H\. Kianinejad, Md\. M\. A\. Patwary, Y\. Yang, and Y\. ZhouDeep learning scaling is predictable, empirically\.External Links:1712\.00409Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, O\. Vinyals, J\. Rae, and L\. SifreAn empirical analysis of compute\-optimal large language model training\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 30016–30030\.External Links:[Document](https://dx.doi.org/10.52202/068431-2176)Cited by:[§1](https://arxiv.org/html/2609.13500#S1.p1.1),[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.External Links:2001\.08361Cited by:[§1](https://arxiv.org/html/2609.13500#S1.p1.1),[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Keet al\.\(2025\)Y\. Ke, Y\. Liang, Z\. Shi, Z\. Song, and C\. YangCurse of attention: a kernel\-based perspective for why transformers fail to generalize on time series forecasting and beyond\.InConference on Parsimony and Learning,Proceedings of Machine Learning Research, Vol\.280,pp\. 675–738\.External Links:[Link](https://proceedings.mlr.press/v280/ke25a.html)Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Kumaret al\.\(2025\)T\. Kumar, Z\. Ankner, B\. F\. Spector, B\. Bordelon, N\. Muennighoff, M\. Paul, C\. Pehlevan, C\. Ré, and A\. RaghunathanScaling laws for precision\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Liet al\.\(2024\)Y\. Li, Y\. Huang, M\. E\. Ildiz, A\. S\. Rawat, and S\. OymakMechanics of next token prediction with self\-attention\.InProceedings of the 27th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.238,pp\. 685–693\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Liet al\.\(2025\)Z\. Li, X\. Jiang, L\. Liu, X\. Zhang, H\. Chen, and F\. ZhengOn the generalization ability of next\-token\-prediction pretraining\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 34943–34975\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Lianget al\.\(2025\)Y\. Liang, Z\. Shi, Z\. Song, and C\. YangTowards infinite\-long prefix in transformer\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 11127–11191\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.563),[Link](https://aclanthology.org/2025.emnlp-main.563/)Cited by:[Appendix A](https://arxiv.org/html/2609.13500#A1.SS0.SSS0.Px2.p1.2),[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Lotfiet al\.\(2022\)S\. Lotfi, M\. Finzi, S\. Kapoor, A\. Potapczynski, M\. Goldblum, and A\. G\. WilsonPAC\-bayes compression bounds so tight that they can explain generalization\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p1.1)\.
- Luschgyet al\.\(2010\)H\. Luschgy, G\. Pagès, and B\. WilbertzAsymptotically optimal quantization schemes for gaussian processes on hilbert spaces\.ESAIM: Probability and Statistics14,pp\. 93–116\.External Links:[Document](https://dx.doi.org/10.1051/ps%3A2008026)Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px2.p1.1)\.
- Luschgy and Pagès \(2004\)H\. Luschgy and G\. PagèsSharp asymptotics of the functional quantization problem for gaussian processes\.The Annals of Probability32\(2\),pp\. 1574–1599\.External Links:[Document](https://dx.doi.org/10.1214/009117904000000324)Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2609.13500#S2.T1.6.3.1.1.1)\.
- Maloneyet al\.\(2022\)A\. Maloney, D\. A\. Roberts, and J\. SullyA solvable model of neural scaling laws\.External Links:2210\.16859Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Michaudet al\.\(2023\)E\. Michaud, Z\. Liu, U\. Girit, and M\. TegmarkThe quantization model of neural scaling\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Micikeviciuset al\.\(2018\)P\. Micikevicius, S\. Narang, J\. Alben, G\. Diamos, E\. Elsen, D\. Garcia, B\. Ginsburg, M\. Houston, O\. Kuchaiev, G\. Venkatesh, and H\. WuMixed precision training\.InInternational Conference on Learning Representations,External Links:1710\.03740Cited by:[4th item](https://arxiv.org/html/2609.13500#A15.I1.i4.p1.1),[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Morriset al\.\(2025\)J\. X\. Morris, C\. Sitawarin, C\. Guo, N\. Kokhlikyan, G\. E\. Suh, A\. M\. Rush, K\. Chaudhuri, and S\. MahloujifarHow much do language models memorize?\.External Links:2505\.24832Cited by:[Appendix A](https://arxiv.org/html/2609.13500#A1.SS0.SSS0.Px2.p1.2),[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Nakada and Imaizumi \(2020\)R\. Nakada and M\. ImaizumiAdaptive approximation and generalization of deep neural network with intrinsic dimensionality\.Journal of Machine Learning Research21\(174\),pp\. 1–38\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p1.1)\.
- NVIDIA \(2026\)NVIDIATransformer engine 2\.18\.0: attention is all you need\!\.Note:Software documentation, accessed 2026\-08\-31Cited by:[1st item](https://arxiv.org/html/2609.13500#A15.I1.i1.p1.1),[5th item](https://arxiv.org/html/2609.13500#A15.I1.i5.p1.1)\.
- Panet al\.\(2025\)Z\. Pan, S\. Wang, L\. Pengfei, and J\. LiUnderstanding LLM behaviors via compression: data generation, knowledge acquisition and scaling laws\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 168404–168454\.External Links:[Document](https://dx.doi.org/10.52202/085713-5613),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/f61d7778e89b9221d1ea0ce8428b7014-Abstract-Conference.html)Cited by:[Appendix A](https://arxiv.org/html/2609.13500#A1.SS0.SSS0.Px2.p1.2),[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.13500#S2.T1.6.5.1.1.1)\.
- Paquetteet al\.\(2024\)E\. Paquette, C\. Paquette, L\. Xiao, and J\. Pennington4\+3 phases of compute\-optimal neural scaling laws\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Porianet al\.\(2024\)T\. Porian, M\. Wortsman, J\. Jitsev, L\. Schmidt, and Y\. CarmonResolving discrepancies in compute\-optimal scaling of language models\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Savareseet al\.\(2022\)P\. H\. P\. Savarese, X\. Yuan, Y\. Li, and M\. MaireNot all bits have equal value: heterogeneous precisions via trainable noise\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Schmidt\-Hieber \(2020\)J\. Schmidt\-HieberNonparametric regression using deep neural networks with ReLU activation function\.The Annals of Statistics48\(4\),pp\. 1875–1897\.External Links:[Document](https://dx.doi.org/10.1214/19-AOS1875)Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p1.1)\.
- Shannon \(1959\)C\. E\. ShannonCoding theorems for a discrete source with a fidelity criterion\.InIRE National Convention Record,Vol\.4,pp\. 142–163\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px2.p1.1)\.
- Sharma and Kaplan \(2020\)U\. Sharma and J\. KaplanA neural scaling law from the dimension of the data manifold\.External Links:2004\.10802Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Steinhardt and Duchi \(2015\)J\. Steinhardt and J\. DuchiMinimax rates for memory\-bounded sparse linear regression\.InProceedings of the 28th Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.40,pp\. 1564–1587\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px2.p1.1)\.
- Sunet al\.\(2025\)X\. Sun, S\. Li, R\. Xie, W\. Han, K\. Wu, Z\. Yang, Y\. Li, A\. Wang, S\. Li, J\. Xue, Y\. Cheng, Y\. Tao, Z\. Kang, C\. Xu, D\. Wang, and J\. JiangScaling laws for floating–point quantization training\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 57544–57570\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Suzuki \(2019\)T\. SuzukiAdaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p1.1)\.
- Trauger and Tewari \(2024\)J\. Trauger and A\. TewariSequence length independent norm\-based generalization bounds for transformers\.InProceedings of the 27th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.238,pp\. 1405–1413\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Xiaoet al\.\(2023\)G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. HanSmoothQuant: accurate and efficient post\-training quantization for large language models\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 38087–38099\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Xionget al\.\(2025\)J\. Xiong, J\. Shen, C\. Zheng, Z\. Wan, C\. Zhao, C\. Yang, F\. Ye, H\. Yang, L\. Kong, and N\. WongParallelComp: parallel long\-context compressor for length extrapolation\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 68998–69016\.External Links:[Link](https://proceedings.mlr.press/v267/xiong25b.html)Cited by:[Appendix A](https://arxiv.org/html/2609.13500#A1.SS0.SSS0.Px2.p1.2),[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Yanget al\.\(2026\)C\. Yang, N\. Srebro, and Z\. LiTight sample complexity of transformers\.InProceedings of Thirty Ninth Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.336,pp\. 6887–6923\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Yang and Li \(2026\)C\. Yang and X\. LiBeyond sparse weights: when is attention compressible?\.External Links:2608\.21541Cited by:[Appendix A](https://arxiv.org/html/2609.13500#A1.SS0.SSS0.Px2.p1.2),[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Yang \(2025\)C\. YangUnifying learning dynamics and generalization in transformers scaling law\.External Links:2512\.22088Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2021\)Z\. Yao, Z\. Dong, Z\. Zheng, A\. Gholami, J\. Yu, E\. Tan, L\. Wang, Q\. Huang, Y\. Wang, M\. W\. Mahoney, and K\. KeutzerHAWQ\-V3: dyadic neural network quantization\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 11875–11886\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Yüksel and Flammarion \(2025\)O\. K\. Yüksel and N\. FlammarionOn the sample complexity of next\-token prediction\.InProceedings of the 28th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.258,pp\. 694–702\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px4.p2.1)\.
- Zhanget al\.\(2026\)D\. Zhang, X\. Tang, Y\. Liang, and D\. ZouScaling laws for precision in high\-dimensional linear regression\.External Links:2602\.19241Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px3.p1.1)\.
- Zhu and Lafferty \(2014\)Y\. Zhu and J\. LaffertyQuantized estimation of gaussian sequence models in euclidean balls\.InAdvances in Neural Information Processing Systems,Vol\.27\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2609.13500#S2.T1.6.2.1.1.1)\.
- Zhu and Lafferty \(2018a\)Y\. Zhu and J\. LaffertyDistributed nonparametric regression under communication constraints\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 6009–6017\.Cited by:[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px2.p1.1)\.
- Zhu and Lafferty \(2018b\)Y\. Zhu and J\. LaffertyQuantized minimax estimation over sobolev ellipsoids\.Information and Inference: A Journal of the IMA7\(1\),pp\. 31–82\.External Links:[Document](https://dx.doi.org/10.1093/imaiai/iax007)Cited by:[§B\.1](https://arxiv.org/html/2609.13500#A2.SS1.p1.1),[§2](https://arxiv.org/html/2609.13500#S2.SS0.SSS0.Px2.p1.1)\.

APPENDIX

## Appendix ADiscussion, Scope, and Closest Precedents

#### One curve for two resources\.

The central object is the spectrum of predictive energy per independent coordinate\. Its integrated tail answers both what data cannot resolve and what a finite learned state cannot retain\. Data evaluate it at resolution1/n1/n; storage chooses a water level\. The same\-marginal construction shows why the energy–dimension pairing is indispensable\.

#### Knowledge capacity as an inverse resource curve\.

To express the information relation, place a fixed prior on the source parametersZZ, including the route when it is unknown\. LetSSbe the learned state formed from training blocks, and letXXcontain all candidates and the query of an independent held\-out block\. Conditional on\(X,Z\)\(X,Z\), the targetYYis independent of the training state\. Entropy and mutual information, denoted byHHandII, are measured in nats\. The conditional mutual\-information identity and the finite\-state constraint give

H\(Y∣X,S\)−H\(Y∣X,Z\)=I\(Y;Z∣X,S\),I\(Z;S\)≤H\(S\)≤Blog2\.H\(Y\\mid X,S\)\-H\(Y\\mid X,Z\)=I\(Y;Z\\mid X,S\),\\qquad I\(Z;S\)\\leq H\(S\)\\leq B\\log 2\.The first difference compares the Bayes predictive losses given the learned state and given the source parameters; a particular decoder can incur additional loss\. The prior is used for this information comparison, whereas the main minimax theorem takes a worst case over fixed parameters\. This identity places knowledge\-capacity and prediction\-risk laws on opposite directions of the same information path\. Allen\-Zhu and Li\([Allen\-Zhu and Li, 2025](https://arxiv.org/html/2609.13500#bib.bib59)\)turn factual generation loss into a lower bound on the information retained by model weights—a loss\-to\-bits question\. Our minimax law turnsnnobservations andBBretained bits into the smallest held\-out excess loss—a bits\-to\-risk question\. Deterministic factual recall and positive\-entropy prediction yield complementary geometries: the former prices discrete facts, while local KL curvature in the latter produces the coordinate distortionqi​2−2​biq\_\{i\}2^\{\-2b\_\{i\}\}, reverse water filling, and the common spectrum\. Frequency\-ranked knowledge also yields data and model scaling under a compression model\([Pan et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib60)\); here the paired predictive energies determine the complete finite\-\(n,B\)\(n,B\)curve, and the same\-marginal construction isolates the pairing information required for its exponents\. Empirical capacity estimates\([Allen\-Zhu and Li, 2025](https://arxiv.org/html/2609.13500#bib.bib59);[Morris et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib61)\)calibrate how efficiently a particular architecture stores information per parameter; the spectrum supplies the source\-dependent risk curve to which such an architectural calibration can be attached\. Sparse attention, prefix learning, and context compression\([Liang et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib64);[Deng et al\., 2024](https://arxiv.org/html/2609.13500#bib.bib66);[Xiong et al\., 2025](https://arxiv.org/html/2609.13500#bib.bib67);[Yang and Li, 2026](https://arxiv.org/html/2609.13500#bib.bib68)\)then connect the abstract state budget to the practical question of which predictive coordinates a Transformer should retain\.

#### Learning the routing and value parameters\.

Public addresses determine where comparisons occur, unknown routing is learned in the query projection, and unknown values are learned in the output coordinates\. Fixing the key projection preserves the route\-table scores and routing\-gradient trajectory\. This identifies the trainable parameters through which self\-attention acquires the source’s predictive structure\.

#### A hierarchy of resource laws\.

The arbitrary\-state lower bound applies to every learner with at most2B2^\{B\}final value states\. The routing lower bound belongs to the stated row\-symmetric online trajectory; the format lower bound applies to the required stored interface; the bit\-operation lower bound concerns the specified dense graph and arithmetic model\. Each result identifies the cost of a distinct bottleneck and makes the progression from statistical information to numerical computation explicit\.

#### From learning to finite\-precision prediction\.

The two\-stage learning theorem trains routing and output coordinates using public coordinate frequencies, supervised calibration, and exact\-arithmetic updates\. The finite\-precision theorem starts from supplied mode logits and quantifies their realization by the network; its converse concerns the separable scalar\-table representation\. The compute theorem studies precision allocation under a two\-sided numerical response assumption\. Together, these results form a three\-step account of acquiring parameters, representing them numerically, and allocating arithmetic resources\. A unified low\-precision training theorem would complete this chain at the level of the optimization trajectory\.

#### Precision in practical Transformer stacks\.

The benefit of a low\-bit matrix path depends on the fraction of storage and arithmetic it changes\. FP32 softmax, accumulation, master weights, optimizer states, and saved activations contribute retained high\-precision work\. Streaming attention reduces retained state while preserving the dense arithmetic order\. Device\-level speedups are governed by the corresponding throughput and memory\-traffic costs\.

#### Open directions\.

Three extensions would carry the spectrum principle into broader sequence models: jointly learning frequencies, ordering, queries, and keys; analyzing low\-precision stochastic updates along the routing and value trajectories; and estimating predictive\-energy spectra from unrestricted corpora\. These directions turn the present exact source model into a program for predicting resource tradeoffs in learned representations\.

## Appendix BExtended Representation and Compute Results

This appendix gives the complete statements and derivations summarized in Sections[5](https://arxiv.org/html/2609.13500#S5)and[6](https://arxiv.org/html/2609.13500#S6)\.

### B\.1Adaptive Schedule and Information Used

The schedule in[Equation5\.2](https://arxiv.org/html/2609.13500#S5.E2)knows the canonical index order and the interval\[χmin,χmax\]\[\\chi\_\{\\min\},\\chi\_\{\\max\}\], but notχ\\chi\. It adapts to the decay rate while treating energy ordering as side information\. Data\-driven allocation with an unknown ordering is related to adaptive quantized Sobolev estimation\([Zhu and Lafferty, 2018b](https://arxiv.org/html/2609.13500#bib.bib2)\); learning that ordering together with the bit schedule is a separate statistical problem\. The full proof and restatement are in Appendix[H](https://arxiv.org/html/2609.13500#A8)\.

### B\.2Finite\-Precision Forward Error and Storage

The matching attention weight in[Equation5\.4](https://arxiv.org/html/2609.13500#S5.E4)isα⋆=eβ/\(eβ\+K−1\)\\alpha\_\{\\star\}=e^\{\\beta\}/\(e^\{\\beta\}\+K\-1\)\. Since the output coordinates lie in\[−R,R\]\[\-R,R\],\|z−ϑj⋆\|≤2​R​\(1−α⋆\)≤2​R​\(K−1\)​e−β\\lvert z\-\\vartheta\_\{j\_\{\\star\}\}\\rvert\\leq 2R\(1\-\\alpha\_\{\\star\}\)\\leq 2R\(K\-1\)e^\{\-\\beta\}\. Uniformly quantizing\[−Δj,Δj\]\[\-\\Delta\_\{j\},\\Delta\_\{j\}\]with2bj2^\{b\_\{j\}\}values gives

\|w~jmode−wjmode\|≤2​Δj​2−bj\.\\lvert\\widetilde\{w\}\_\{j\}^\{\\rm mode\}\-w\_\{j\}^\{\\rm mode\}\\rvert\\leq 2\\Delta\_\{j\}2^\{\-b\_\{j\}\}\.\(B\.1\)For arithmetic error, letuube the unit roundoff of the scalar operations\. Max shifting produces unnormalized weights in\{1,e−β\}\\\{1,e^\{\-\\beta\}\\\}\. Balanced pairwise summation has depthh=⌈log2⁡K⌉h=\\lceil\\log\_\{2\}K\\rceil; putγh=h​u/\(1−h​u\)\\gamma\_\{h\}=hu/\(1\-hu\)\. Positive pairwise summation incurs relative error at mostγh\\gamma\_\{h\}\. Composing exponential, denominator, and division errors gives, when\(h\+4\)​u≤1/4\(h\+4\)u\\leq 1/4,

∥α^−α∥1≤8​\(h\+4\)​u\.\\lVert\\widehat\{\\alpha\}\-\\alpha\\rVert\_\{1\}\\leq 8\(h\+4\)u\.\(B\.2\)The final weighted sum can have mixed signs, so an absolute bound is needed:

\|fl⁡\(w~⊤​α^\)−w~⊤​α\|≤CR​u​\(1\+log⁡K\)\.\\lvert\\operatorname\{fl\}\(\\widetilde\{w\}^\{\\top\}\\widehat\{\\alpha\}\)\-\\widetilde\{w\}^\{\\top\}\\alpha\\rvert\\leq C\_\{R\}u\(1\+\\log K\)\.\(B\.3\)The absolute bound also handles cancellation\. Combining routing, quantization, and arithmetic errors, squaring and averaging over the mode, and applying bounded log\-loss curvature yields[Equation5\.6](https://arxiv.org/html/2609.13500#S5.E6), including the unrepresented tail\. The detailed proof and restatement of[Theorem5\.2](https://arxiv.org/html/2609.13500#S5.Thmtheorem2)are in Appendix[J](https://arxiv.org/html/2609.13500#A10)\.

#### Learned state versus physical storage\.

Hered=K\+M\+1d=K\+M\+1is the token width of the particular realization in[Section5](https://arxiv.org/html/2609.13500#S5.SS0.SSS0.Px2), withMMretained scalar output coordinates\. If the score matrix, value projection, and output head are dense arrays, their physical scalar\-slot count is

N=2​d2\+d\.N=2d^\{2\}\+d\.\(B\.4\)Only theMMoutput coordinates depend on the source, and their description length is∑j=1Mbj\\sum\_\{j=1\}^\{M\}b\_\{j\}\. ThusNN,MM, andBBare different resources even in this fixed\-depth, one\-head realization with public orthogonal addresses\. The finite\-precision theorem initializes those coordinates with supplied logits;[Theorem4\.3](https://arxiv.org/html/2609.13500#S4.Thmtheorem3)separately analyzes gradient training of routing and output values\.

### B\.3Forward–Backward Bit Complexity

Consider eager, non\-checkpointed reverse\-mode differentiation \(standard backpropagation\) through a dense Transformer with sequence lengthℓ\\ell, hidden widthdd, feed\-forward widthmffm\_\{\\rm ff\},HHattention heads whose widths sum todd, andLLlayers\. Matrix multiplications use a format indexed bybmmb\_\{\\rm mm\}, the stable max\-shifted softmax uses compute precisionbsmb\_\{\\rm sm\}, and a materialized attention probability usesbstb\_\{\\rm st\}storage bits\. The indexbmmb\_\{\\rm mm\}determines all operand, output, and accumulator widths in this format family\. Let𝖬⁡\(bmm\)\\mathsf\{M\}\(b\_\{\\rm mm\}\)be the bit\-operation cost of one such multiply–accumulate\. Let𝖲⁡\(bsm\)\\mathsf\{S\}\(b\_\{\\rm sm\}\)be the amortized per\-attention\-entry cost of stable softmax and its reverse\-mode Jacobian–vector product, including exponentials, reductions, division, and pointwise arithmetic\. We assume normalization, activation, and cast costs per entry do not asymptotically exceed these matrix and softmax costs\. We do not count the optimizer update\. This conventional query/key/value/output stack is a separate computational graph from the smaller realization in[Section5](https://arxiv.org/html/2609.13500#S5.SS0.SSS0.Px2)\. Letbwb\_\{w\}andbab\_\{a\}denote aggregate bits per stored weight and activation slot at peak memory, and letMemhp\\operatorname\{Mem\}\_\{\\rm hp\}denote retained high\-precision state\.

###### Lemma B\.1\(Transformer forward–backward bit complexity; informal version of[LemmaK\.1](https://arxiv.org/html/2609.13500#A11.Thmtheorem1)\)\.

For the graph above, with positive dimensions and conventional dense matrix multiplication,

BitOpsfb≍L⁡\[\(ℓ​d2\+ℓ​d​mff\+ℓ2​d\)​𝖬​\(bmm\)\+H​ℓ2​𝖲​\(bsm\)\]\.\\operatorname\{BitOps\}\_\{\\rm fb\}\\asymp L\\Bigl\[\\bigl\(\\ell d^\{2\}\+\\ell dm\_\{\\rm ff\}\+\\ell^\{2\}d\\bigr\)\\mathsf\{M\}\(b\_\{\\rm mm\}\)\+H\\ell^\{2\}\\mathsf\{S\}\(b\_\{\\rm sm\}\)\\Bigr\]\.\(B\.5\)Memfb≍bw​L​\(d2\+d​mff\)\+ba​L​ℓ​\(d\+mff\)\+bst​L​H​ℓ2\+Memhp\.\\operatorname\{Mem\}\_\{\\rm fb\}\\asymp b\_\{w\}L\(d^\{2\}\+dm\_\{\\rm ff\}\)\+b\_\{a\}L\\ell\(d\+m\_\{\\rm ff\}\)\+b\_\{\\rm st\}LH\\ell^\{2\}\+\\operatorname\{Mem\}\_\{\\rm hp\}\.\(B\.6\)The memoryMemfb\\operatorname\{Mem\}\_\{\\rm fb\}is measured in bits\. The valuesbwb\_\{w\},bab\_\{a\}, andbstb\_\{\\rm st\}count aggregate bits per logical weight, activation, and saved\-attention slot at peak memory; they include any same\-shaped gradient or backward\-workspace buffer materialized by this eager implementation\. The termMemhp\\operatorname\{Mem\}\_\{\\rm hp\}contains master weights, optimizer state, and all other explicitly retained high\-precision data\. A streaming, recomputing attention implementation replaces the quadratic attention state saved for backward by linear online\-softmax state plus temporary tile workspace, while leaving the dense attention arithmetic order unchanged\.

The three matrix terms in[EquationB\.5](https://arxiv.org/html/2609.13500#A2.E5)come respectively from the input/output projections, the two feed\-forward projections, and the score–value attention products\. Reverse mode performs two matrix products for each forward matrix product, so it changes constants rather than orders\. For a softmax rowppand incoming vectorvv, its derivative is applied as\(diag⁡\(p\)−p​p⊤\)​v=p⊙\(v−⟨p,v⟩​𝟏\)\(\\operatorname\{diag\}\(p\)\-pp^\{\\top\}\)v=p\\odot\(v\-\\langle p,v\\rangle\\mathbf\{1\}\)\. Thus the Jacobian–vector product costsΘ⁡\(H​ℓ2\)\\Theta\(H\\ell^\{2\}\), without forming anℓ×ℓ\\ell\\times\\ellJacobian for every row\. The proof and a shape\-by\-shape operation count are in[AppendixK](https://arxiv.org/html/2609.13500#A11)\.

### B\.4The Compute–Precision Pareto Law

LetMMdenote the number of represented modes\. Under a specified shape schedule\(ℓ,d,mff,H,L\)​\(M\)\(\\ell,d,m\_\{\\rm ff\},H,L\)\(M\),[LemmaB\.1](https://arxiv.org/html/2609.13500#A2.Thmtheorem1)givesG⁡\(M\)≍L⁡\(ℓ​d2\+ℓ​d​mff\+ℓ2​d\)G\(M\)\\asymp L\(\\ell d^\{2\}\+\\ell dm\_\{\\rm ff\}\+\\ell^\{2\}d\)andS⁡\(M\)≍L​H​ℓ2S\(M\)\\asymp LH\\ell^\{2\}\. We writeG⁡\(M\)≍MgG\(M\)\\asymp M^\{g\}for quantizable matrix work andS⁡\(M\)≍MsS\(M\)\\asymp M^\{s\}for softmax work, whereg,s≥0g,s\\geq 0\. Reaching model errorM−χM^\{\-\\chi\}requiresn⁡\(M\)≍M1\+χn\(M\)\\asymp M^\{1\+\\chi\}independent blocks\. Letℰ\\mathcal\{E\}denote held\-out excess risk, and letκmm\\kappa\_\{\\rm mm\}andκsm\\kappa\_\{\\rm sm\}measure how matrix and softmax rounding errors are amplified\. We assume the following two\-sided error and cost relations, which separate effective matrix precisionbbfrom effective softmax precisionbsmb\_\{\\rm sm\}:

ℰ⁡\(M,b,bsm\)≍M−χ\+κmm​\(M\)2​2−2​b\+κsm​\(M\)2​2−2​bsm\.\\mathcal\{E\}\(M,b,b\_\{\\rm sm\}\)\\asymp M^\{\-\\chi\}\+\\kappa\_\{\\rm mm\}\(M\)^\{2\}2^\{\-2b\}\+\\kappa\_\{\\rm sm\}\(M\)^\{2\}2^\{\-2b\_\{\\rm sm\}\}\.\(B\.7\)C≍M1\+χ​\[G⁡\(M\)​bν\+S⁡\(M\)​bsmνsm\]\.C\\asymp M^\{1\+\\chi\}\\left\[G\(M\)b^\{\\nu\}\+S\(M\)b\_\{\\rm sm\}^\{\\nu\_\{\\rm sm\}\}\\right\]\.\(B\.8\)The exponentsν,νsm≥0\\nu,\\nu\_\{\\rm sm\}\\geq 0belong to the stated arithmetic model\. The abstract format family has total width proportional to its effective precision, equivalently unit roundoffΘ⁡\(2−b\)\\Theta\(2^\{\-b\}\); concrete floating\-point formats are discussed separately below\.

###### Theorem B\.2\(Compute–precision Pareto law; informal version of[TheoremL\.1](https://arxiv.org/html/2609.13500#A12.Thmtheorem1)\)\.

Assume[EquationsB\.7](https://arxiv.org/html/2609.13500#A2.E7)and[B\.8](https://arxiv.org/html/2609.13500#A2.E8),κmm​\(M\)≍Mr\\kappa\_\{\\rm mm\}\(M\)\\asymp M^\{r\}, andκsm​\(M\)≍Mrsm\\kappa\_\{\\rm sm\}\(M\)\\asymp M^\{r\_\{\\rm sm\}\}, with all displayed relations two\-sided andr,rsm≥0r,r\_\{\\rm sm\}\\geq 0\. For the specified dense graph,H≤dH\\leq dimpliesG≳SG\\gtrsim Sand henceg≥sg\\geq s\. Item 3 concerns general operation counts satisfying the same two\-sided relations; items 1, 2, and 4 apply to both settings\.

1. 1\.The smallest matrix precision that keeps matrix rounding at the model error is bmatch​\(M\)=\(r\+χ2\)​log2​M\+O⁡\(1\)\.b\_\{\\rm match\}\(M\)=\\left\(r\+\\frac\{\\chi\}\{2\}\\right\)\\log\_\{2\}M\+O\(1\)\.\(B\.9\)
2. 2\.If the matrix term dominates the bracket in[EquationB\.8](https://arxiv.org/html/2609.13500#A2.E8)at matched precisions—in particular in the caseg≥sg\\geq swhen the remaining precision factors do not reverse that dominance—then infM,b,bsm:cost≤Cℰ≍C−χ/\(1\+χ\+g\)\(logC\)χ​ν/\(1\+χ\+g\),\\inf\_\{M,b,b\_\{\\rm sm\}:\\,\\mathrm\{cost\}\\leq C\}\\mathcal\{E\}\\asymp C^\{\-\\chi/\(1\+\\chi\+g\)\}\(\\log C\)^\{\\chi\\nu/\(1\+\\chi\+g\)\},\(B\.10\)up to other declared slowly varying factors\.
3. 3\.For general operation counts, ifs\>gs\>g,bsmb\_\{\\rm sm\}is fixed, and the softmax term dominates computation, then for budgets in the pre\-floor range C1/\(1\+χ\+s\)≲22​bsm/\(χ\+2​rsm\),C^\{1/\(1\+\\chi\+s\)\}\\lesssim 2^\{2b\_\{\\rm sm\}/\(\\chi\+2r\_\{\\rm sm\}\)\},infM,b:cost≤Cℰ≍C−χ/\(1\+χ\+s\)\.\\inf\_\{M,b:\\,\\mathrm\{cost\}\\leq C\}\\mathcal\{E\}\\asymp C^\{\-\\chi/\(1\+\\chi\+s\)\}\.\(B\.11\)Reducing matrix precision cannot improve this leading exponent\.
4. 4\.At fixed matrix precisionbb, with softmax precision optimized, the model–rounding crossover and the infimum error over model scales are Mb≍22​b/\(χ\+2​r\),ℰfloor\(b\)≍2−2χb/\(χ\+2r\)\.M\_\{b\}\\asymp 2^\{2b/\(\\chi\+2r\)\},\\qquad\\mathcal\{E\}\_\{\\rm floor\}\(b\)\\asymp 2^\{\-2\\chi b/\(\\chi\+2r\)\}\.\(B\.12\)A fixedbsmb\_\{\\rm sm\}\-bit softmax similarly supports the scaling phase only while M≲22​bsm/\(χ\+2​rsm\)\.M\\lesssim 2^\{2b\_\{\\rm sm\}/\(\\chi\+2r\_\{\\rm sm\}\)\}\.\(B\.13\)

The displayed two\-sided risk and arithmetic assumptions define the class over which the infima and matching bounds are taken\.

The theorem distinguishes increasing precision from fixed\-precision deployment\. Polynomial conditioning requires only logarithmically increasing precision, so adaptive precision changes a logarithmic factor, not the power exponent\. Conversely, permanently fixing the precision creates a numerical floor\. For a concrete 32\-bit floating\-point \(FP32\) softmax with unit roundoffusmu\_\{\\rm sm\}, provided the stated exponent\-range and relative\-error assumptions remain valid,[EquationB\.13](https://arxiv.org/html/2609.13500#A2.E13)readsM≲usm−2/\(χ\+2rsm\)M\\lesssim u\_\{\\rm sm\}^\{\-2/\(\\chi\+2r\_\{\\rm sm\}\)\}; the total format width is not substituted for effective accuracy bits\. Exponential conditioning, treated in[Theorem6\.1](https://arxiv.org/html/2609.13500#S6.Thmtheorem1), is the case in which precision itself creates a new power\-law phase\.

#### Limits imposed by high\-precision work\.

At a fixed model shape, lowering a matrix path from baseline precisionb0b\_\{0\}tobbgives the relative bit\-operation count

BitOpsRatio=G​b0ν\+S​bsmνsmG​bν\+S​bsmνsm\.\\operatorname\{BitOpsRatio\}=\\frac\{Gb\_\{0\}^\{\\nu\}\+Sb\_\{\\rm sm\}^\{\\nu\_\{\\rm sm\}\}\}\{Gb^\{\\nu\}\+Sb\_\{\\rm sm\}^\{\\nu\_\{\\rm sm\}\}\}\.\(B\.14\)IfQQstored scalars can be quantized andRhpR\_\{\\rm hp\}bits must remain at high precision, the corresponding storage ratio is

Compressionbit=Q​b0\+RhpQ​b\+Rhp\.\\operatorname\{Compression\}\_\{\\rm bit\}=\\frac\{Qb\_\{0\}\+R\_\{\\rm hp\}\}\{Qb\+R\_\{\\rm hp\}\}\.\(B\.15\)These formulas expose the ceiling imposed by high\-precision normalization, master weights, optimizer states, or saved activations\. Actual graphics processor speed also depends on memory traffic, kernel support, and hardware throughput\.

The two analytic consequences appear in[Figure1](https://arxiv.org/html/2609.13500#S6.F1): its left panel shows the retained\-high\-precision ceiling in[EquationB\.14](https://arxiv.org/html/2609.13500#A2.E14), and its right panel compares the three risk–cost regimes above\.

## Appendix CProofs for the Main Results

The proofs follow the paper’s mechanism order: the spectral identities and their exponents, the query–key realization and online route, the network upper bound, and the same\-marginal separation\.

Throughout this appendix,iiindexes the scalar coordinates defined in[Section3](https://arxiv.org/html/2609.13500#S3)\. Their energies areqi=pi​Ri2q\_\{i\}=p\_\{i\}R\_\{i\}^\{2\}, the spectrum isμ=∑iδqi\\mu=\\sum\_\{i\}\\delta\_\{q\_\{i\}\}, andΦμ​\(t\)=∑imin⁡\{qi,t\}\\Phi\_\{\\mu\}\(t\)=\\sum\_\{i\}\\min\\\{q\_\{i\},t\\\}\. The water levelτB\\tau\_\{B\}is selected by the storage budget in that setup\. Each restatement below uses the same objects as its corresponding main\-text result\.

###### Theorem C\.1\(One spectrum controls data and memory; formal version of[Theorem3\.1](https://arxiv.org/html/2609.13500#S3.Thmtheorem1)\)\.

Letqi\>0q\_\{i\}\>0,∑iqi<∞\\sum\_\{i\}q\_\{i\}<\\infty, and letμ\\mu,Φμ\\Phi\_\{\\mu\}, andτB\\tau\_\{B\}be defined above\. Then

𝒮q​\(n\)\\displaystyle\\mathcal\{S\}\_\{q\}\(n\):=∑imin⁡\{qi,n−1\}=Φμ​\(n−1\),\\displaystyle:=\\sum\_\{i\}\\min\\\{q\_\{i\},n^\{\-1\}\\\}=\\Phi\_\{\\mu\}\(n^\{\-1\}\),\(C\.1\)Dq​\(B\)\\displaystyle D\_\{q\}\(B\):=infbi≥0∑ibi≤B∑iqi​2−2​bi=Φμ​\(τB\)\.\\displaystyle:=\\inf\_\{\\begin\{subarray\}\{c\}b\_\{i\}\\geq 0\\\\ \\sum\_\{i\}b\_\{i\}\\leq B\\end\{subarray\}\}\\sum\_\{i\}q\_\{i\}2^\{\-2b\_\{i\}\}=\\Phi\_\{\\mu\}\(\\tau\_\{B\}\)\.\(C\.2\)For the bounded\-logit value experiment,

ℜ∗value\(n,B\)≍RΦμ\(n−1\)\+Φμ\(τB\)\.\\boxed\{\\mathfrak\{R\}^\{\*\}\_\{\\rm value\}\(n,B\)\\asymp\_\{R\}\\Phi\_\{\\mu\}\(n^\{\-1\}\)\+\\Phi\_\{\\mu\}\(\\tau\_\{B\}\)\.\}\(C\.3\)Moreover, for everyt\>0t\>0,

Φμ,\+′​\(t\)=μ⁡\(\(t,∞\)\),Φμ,−′​\(t\)=μ⁡\(\[t,∞\)\)\.\\Phi^\{\\prime\}\_\{\\mu,\+\}\(t\)=\\mu\(\(t,\\infty\)\),\\qquad\\Phi^\{\\prime\}\_\{\\mu,\-\}\(t\)=\\mu\(\[t,\\infty\)\)\.\(C\.4\)Hence the exact curveΦμ\\Phi\_\{\\mu\}recovers the positive spectrum, including atom multiplicities, up to coordinate relabeling\.

###### Proof\.

Summability gives𝒩μ​\(t\)<∞\\mathcal\{N\}\_\{\\mu\}\(t\)<\\inftyfor everyt\>0t\>0, and the largest positive energy is attained\. Tonelli’s theorem yields

Φμ\(t\)=∑i∫0t𝟏\{s<qi\}ds=∑imin\{qi,t\},\\Phi\_\{\\mu\}\(t\)=\\sum\_\{i\}\\int\_\{0\}^\{t\}\\mathbf\{1\}\\\{s<q\_\{i\}\\\}\\,\\mathrm\{d\}s=\\sum\_\{i\}\\min\\\{q\_\{i\},t\\\},which proves[Equation3\.1](https://arxiv.org/html/2609.13500#S3.E1)\. For a finite storage budget, reverse water filling has rates

bi∗=12​log2​\(qi/τB\)\+\.b\_\{i\}^\{\*\}=\\frac\{1\}\{2\}\\log\_\{2\}\(q\_\{i\}/\\tau\_\{B\}\)\_\{\+\}\.Here the positive part applies to the logarithm\. To justify the optimizer without interchanging a limit and an infimum, write

B⁡\(τ\)=12​ln⁡2​∑i\[ln⁡\(qi/τ\)\]\+\.B\(\\tau\)=\\frac\{1\}\{2\\ln 2\}\\sum\_\{i\}\[\\ln\(q\_\{i\}/\\tau\)\]\_\{\+\}\.On every compact subinterval of\(0,qmax\]\(0,q\_\{\\max\}\], only finitely many summands can be nonzero\. ThusB⁡\(τ\)B\(\\tau\)is continuous, strictly decreasing belowqmaxq\_\{\\max\}, equals zero atqmaxq\_\{\\max\}, and tends to infinity asτ↓0\\tau\\downarrow 0\. This proves existence and uniqueness ofτB\\tau\_\{B\}forB\>0B\>0; forB=0B=0takeτ0=qmax\\tau\_\{0\}=q\_\{\\max\}\. Forλ=2​\(ln⁡2\)​τB\\lambda=2\(\\ln 2\)\\tau\_\{B\}, scalar convexity gives, for everybi≥0b\_\{i\}\\geq 0,

qi​2−2​bi\+λ​bi≥qi​2−2​bi∗\+λ​bi∗\.q\_\{i\}2^\{\-2b\_\{i\}\}\+\\lambda b\_\{i\}\\geq q\_\{i\}2^\{\-2b\_\{i\}^\{\*\}\}\+\\lambda b\_\{i\}^\{\*\}\.All distortion sums are bounded by∑iqi\\sum\_\{i\}q\_\{i\}, and the two bit sums are finite\. Summing the inequality over all coordinates shows that every feasible allocation has distortion at least∑iqi​2−2​bi∗\+λ⁡\(B−∑ibi\)≥∑iqi​2−2​bi∗\\sum\_\{i\}q\_\{i\}2^\{\-2b\_\{i\}^\{\*\}\}\+\\lambda\(B\-\\sum\_\{i\}b\_\{i\}\)\\geq\\sum\_\{i\}q\_\{i\}2^\{\-2b\_\{i\}^\{\*\}\}\. This proves optimality directly in the countable problem\. Sinceqi​2−2​bi∗=min⁡\{qi,τB\}q\_\{i\}2^\{\-2b\_\{i\}^\{\*\}\}=\\min\\\{q\_\{i\},\\tau\_\{B\}\\\}, this proves[Equation3\.2](https://arxiv.org/html/2609.13500#S3.E2)\. The bounded\-logit curvature, Assouad, and arbitrary codebook bounds, proved in[TheoremG\.2](https://arxiv.org/html/2609.13500#A7.Thmtheorem2), giveℜvalue∗\(n,B\)≍R𝒮q\(n\)\+Dq\(B\)\\mathfrak\{R\}^\{\*\}\_\{\\rm value\}\(n,B\)\\asymp\_\{R\}\\mathcal\{S\}\_\{q\}\(n\)\+D\_\{q\}\(B\), proving[Equation3\.3](https://arxiv.org/html/2609.13500#S3.E3)\. Finally, for0<h<t/20<h<t/2, both one\-sided difference quotients ofmin⁡\{qi,t\}\\min\\\{q\_\{i\},t\\\}lie in\[0,1\]\[0,1\]and vanish wheneverqi<t/2q\_\{i\}<t/2\. There are only finitely many remaining coordinates\. Taking their limits gives respectively𝟏\{qi\>t\}\\mathbf\{1\}\\\{q\_\{i\}\>t\\\}and𝟏\{qi≥t\}\\mathbf\{1\}\\\{q\_\{i\}\\geq t\\\}, proving[Equation3\.4](https://arxiv.org/html/2609.13500#S3.E4)\. Their difference is exactly the multiplicity of the atom attt\. ∎

###### Corollary C\.2\(Spectral scaling exponents; formal version of[Corollary3\.2](https://arxiv.org/html/2609.13500#S3.Thmtheorem2)\)\.

If𝒩μ​\(t\)∼t−ρ​L​\(1/t\)\\mathcal\{N\}\_\{\\mu\}\(t\)\\sim t^\{\-\\rho\}L\(1/t\)for0<ρ<10<\\rho<1, whereLLis slowly varying, then

Φμ\(n−1\)∈RV−\(1−ρ\),Φμ\(τB\)∈RV−\(1−ρ\)/ρ\.\\Phi\_\{\\mu\}\(n^\{\-1\}\)\\in\\operatorname\{RV\}\_\{\-\(1\-\\rho\)\},\\qquad\\Phi\_\{\\mu\}\(\\tau\_\{B\}\)\\in\\operatorname\{RV\}\_\{\-\(1\-\\rho\)/\\rho\}\.Foraj≍j−\(1\+χ\)a\_\{j\}\\asymp j^\{\-\(1\+\\chi\)\}anddj≍jγd\_\{j\}\\asymp j^\{\\gamma\},ρ=\(1\+γ\)/\(1\+χ\+γ\)\\rho=\(1\+\\gamma\)/\(1\+\\chi\+\\gamma\), giving ratesn−χ/\(1\+χ\+γ\)n^\{\-\\chi/\(1\+\\chi\+\\gamma\)\}andB−χ/\(1\+γ\)B^\{\-\\chi/\(1\+\\gamma\)\}, up to slowly varying factors\.

###### Proof\.

We first prove the regular\-variation assertion under its stated hypothesis, then treat power comparisons separately\. AbbreviateN​\(t\)=𝒩μ​\(t\)N\(t\)=\\mathcal\{N\}\_\{\\mu\}\(t\)\. For fixedx\>0x\>0,N⁡\(t​x\)/N⁡\(t\)→x−ρN\(tx\)/N\(t\)\\to x^\{\-\\rho\}ast↓0t\\downarrow 0\. Choose0<ϵ<min⁡\{ρ,1−ρ\}0<\\epsilon<\\min\\\{\\rho,1\-\\rho\\\}\. The ratio atx=1/2x=1/2, iterated over dyadic intervals and combined with monotonicity, gives

N⁡\(t​x\)N⁡\(t\)≤Cϵ​x−ρ−ϵ\(0<x≤1\),N⁡\(t​x\)N⁡\(t\)≤Cϵ​x−ρ\+ϵ\(1≤x≤t0/t\)\\frac\{N\(tx\)\}\{N\(t\)\}\\leq C\_\{\\epsilon\}x^\{\-\\rho\-\\epsilon\}\\quad\(0<x\\leq 1\),\\qquad\\frac\{N\(tx\)\}\{N\(t\)\}\\leq C\_\{\\epsilon\}x^\{\-\\rho\+\\epsilon\}\\quad\(1\\leq x\\leq t\_\{0\}/t\)for a fixed sufficiently smallt0t\_\{0\}\. For example, chooset0t\_\{0\}so that the ratioN⁡\(s/2\)/N⁡\(s\)N\(s/2\)/N\(s\)lies between2ρ−ϵ2^\{\\rho\-\\epsilon\}and2ρ\+ϵ2^\{\\rho\+\\epsilon\}for alls≤t0s\\leq t\_\{0\}, and telescope; roundingxxto its neighboring powers of two changes onlyCϵC\_\{\\epsilon\}\. Consequently dominated convergence inΦμ​\(t\)/\(t​N​\(t\)\)=∫01N⁡\(t​x\)/N⁡\(t\)​𝑑x\\Phi\_\{\\mu\}\(t\)/\(tN\(t\)\)=\\int\_\{0\}^\{1\}N\(tx\)/N\(t\)\\,\\mathrm\{d\}xgives

Φμ​\(t\)∼t1−ρ​L​\(1/t\)1−ρ\.\\Phi\_\{\\mu\}\(t\)\\sim\\frac\{t^\{1\-\\rho\}L\(1/t\)\}\{1\-\\rho\}\.Layer integration of the storage budget gives

ℬμ​\(τ\)=12​ln⁡2​∫τqmax𝒩μ​\(t\)t​𝑑t∼τ−ρ​L​\(1/τ\)2​ρ​ln⁡2\.\\mathcal\{B\}\_\{\\mu\}\(\\tau\)=\\frac\{1\}\{2\\ln 2\}\\int\_\{\\tau\}^\{q\_\{\\max\}\}\\frac\{\\mathcal\{N\}\_\{\\mu\}\(t\)\}\{t\}\\,\\mathrm\{d\}t\\sim\\frac\{\\tau^\{\-\\rho\}L\(1/\\tau\)\}\{2\\rho\\ln 2\}\.Indeed, after substitutingt=τ​xt=\\tau x, the second domination bound makes the integral converge to∫1∞x−ρ−1​𝑑x=1/ρ\\int\_\{1\}^\{\\infty\}x^\{\-\\rho\-1\}\\,\\mathrm\{d\}x=1/\\rho\. The part abovet0t\_\{0\}is finite and negligible relative toN⁡\(τ\)N\(\\tau\)\. ThusB⁡\(c​τ\)/B⁡\(τ\)→c−ρB\(c\\tau\)/B\(\\tau\)\\to c^\{\-\\rho\}\. Fora\>0a\>0, sandwiching the inverse between\(a−1/ρ−ε\)τB\(a^\{\-1/\\rho\}\-\\varepsilon\)\\tau\_\{B\}and\(a−1/ρ\+ε\)τB\(a^\{\-1/\\rho\}\+\\varepsilon\)\\tau\_\{B\}, and then sendingε↓0\\varepsilon\\downarrow 0, provesτa​B/τB→a−1/ρ\\tau\_\{aB\}/\\tau\_\{B\}\\to a^\{\-1/\\rho\}\. The same monotone sandwich forΦμ\\Phi\_\{\\mu\}givesΦμ\(τa​B\)/Φμ\(τB\)→a−\(1−ρ\)/ρ\\Phi\_\{\\mu\}\(\\tau\_\{aB\}\)/\\Phi\_\{\\mu\}\(\\tau\_\{B\}\)\\to a^\{\-\(1\-\\rho\)/\\rho\}\. This proves both regular\-variation indices without assuming a differentiable slowly varying factor\.

For the block profile, dimensions are positive integers, so the stipulated comparisondj≍jγd\_\{j\}\\asymp j^\{\\gamma\}hasγ≥0\\gamma\\geq 0\. Coordinate energyaj/dj≍j−\(1\+χ\+γ\)a\_\{j\}/d\_\{j\}\\asymp j^\{\-\(1\+\\chi\+\\gamma\)\}occursdj≍jγd\_\{j\}\\asymp j^\{\\gamma\}times\. Bounds on the two comparison constants show that every index belowct−1/\(1\+χ\+γ\)ct^\{\-1/\(1\+\\chi\+\\gamma\)\}is active and no index aboveCt−1/\(1\+χ\+γ\)Ct^\{\-1/\(1\+\\chi\+\\gamma\)\}is active\. Summing the multiplicities therefore givesN⁡\(t\)≍t−ρN\(t\)\\asymp t^\{\-\\rho\}withρ=\(1\+γ\)/\(1\+χ\+γ\)\\rho=\(1\+\\gamma\)/\(1\+\\chi\+\\gamma\)\. Integrating these two\-sided bounds directly yields

Φμ\(t\)≍t1−ρ,B\(τ\)≍τ−ρ,Φμ\(τB\)≍B−\(1−ρ\)/ρ\.\\Phi\_\{\\mu\}\(t\)\\asymp t^\{1\-\\rho\},\\qquad B\(\\tau\)\\asymp\\tau^\{\-\\rho\},\\qquad\\Phi\_\{\\mu\}\(\\tau\_\{B\}\)\\asymp B^\{\-\(1\-\\rho\)/\\rho\}\.These power orders do not require regular variation of the individual profile or of an atomic counting tail\. ∎

###### Proposition C\.3\(Exact query–key realization; formal version of[Proposition4\.1](https://arxiv.org/html/2609.13500#S4.Thmtheorem1)\)\.

Under the strict causal mask exposing exactly theKKpreceding candidates,[Equation4\.1](https://arxiv.org/html/2609.13500#S4.E1)reproduces the route\-table scores and attention probabilities exactly\. If only the query\-address block ofWQW\_\{Q\}is trained andWKW\_\{K\}is fixed, Euclidean SGD onWQW\_\{Q\}induces the same update ofAA; the attended feature and output\-vector gradient are unchanged\.

###### Proof\.

BecausePQ​xq=eqP\_\{Q\}x\_\{q\}=e\_\{q\},PC​xs=esP\_\{C\}x\_\{s\}=e\_\{s\}, andJ⊤​J=IKJ^\{\\top\}J=I\_\{K\}, direct substitution gives[Equation4\.1](https://arxiv.org/html/2609.13500#S4.E1)\. IfG=∇AℓG=\\nabla\_\{A\}\\ell, the physical query\-projection gradient is∇WQℓ=J​G⊤​PQ\\nabla\_\{W\_\{Q\}\}\\ell=JG^\{\\top\}P\_\{Q\}\. Restricting training to this address block therefore maps one Euclidean step toWQ​\(A−η​G\)W\_\{Q\}\(A\-\\eta G\)\. The scores, candidate\-only masked softmax, and attended value feature coincide\. The output loss sees the same feature, so its gradient with respect towwalso coincides\. Orthogonality of the residual subspace towwmakes the residual logit zero\. ∎

###### Proposition C\.4\(Online routing dynamics; formal version of[Proposition4\.2](https://arxiv.org/html/2609.13500#S4.Thmtheorem2)\)\.

ForK≥2K\\geq 2, zero initialization and softmax cross\-entropy steps of size0<η≤10<\\eta\\leq 1give the margin and distractor mass afterttrow visits:

βt\+1=βt\+η​Keβt\+K−1,δt=K−1eβt\+K−1\.\\beta\_\{t\+1\}=\\beta\_\{t\}\+\\eta\\frac\{K\}\{e^\{\\beta\_\{t\}\}\+K\-1\},\\qquad\\delta\_\{t\}=\\frac\{K\-1\}\{e^\{\\beta\_\{t\}\}\+K\-1\}\.AfterbK=Θη​\(log⁡\(e​K\)\)b\_\{K\}=\\Theta\_\{\\eta\}\(\\log\(eK\)\)visits,δt≍η\[1\+\(t−bK\)\+\]−1\\delta\_\{t\}\\asymp\_\{\\eta\}\[1\+\(t\-b\_\{K\}\)\_\{\+\}\]^\{\-1\}\. ForSAS\_\{A\}independent calibration blocks and a held\-out row,

𝔯K​\(SA\)=𝔼⁡\[δN2\],N∼Binomial⁡\(SA,1/K\),\\mathfrak\{r\}\_\{K\}\(S\_\{A\}\)=\\mathbb\{E\}\[\\delta\_\{N\}^\{2\}\],\\quad N\\sim\\operatorname\{Binomial\}\(S\_\{A\},1/K\),and the hitting time to𝔯K​\(SA\)≤ε\\mathfrak\{r\}\_\{K\}\(S\_\{A\}\)\\leq\\varepsilon,0<ε≤1/160<\\varepsilon\\leq 1/16, isΘη\(K\{ε−1/2\+log\(eK\)\}\)\\Theta\_\{\\eta\}\(K\\\{\\varepsilon^\{\-1/2\}\+\\log\(eK\)\\\}\)\. ForK=1K=1, leakage and calibration cost are zero\.

###### Proof\.

Zero initialization preserves equality among distractor scores\. Ifutu\_\{t\}is the correct score andvtv\_\{t\}a distractor score, one softmax cross\-entropy update gives

ut\+1=ut\+η⁡\(1−p⋆\),vt\+1=vt−η​p0,u\_\{t\+1\}=u\_\{t\}\+\\eta\(1\-p\_\{\\star\}\),\\qquad v\_\{t\+1\}=v\_\{t\}\-\\eta p\_\{0\},which proves the margin recurrence\. Introduce

ΨK​\(β\)=eβ−1\+\(K−1\)​βK\.\\Psi\_\{K\}\(\\beta\)=\\frac\{e^\{\\beta\}\-1\+\(K\-1\)\\beta\}\{K\}\.Putdt=βt\+1−βtd\_\{t\}=\\beta\_\{t\+1\}\-\\beta\_\{t\}\. Sinceβt≥0\\beta\_\{t\}\\geq 0,0<dt≤η0<d\_\{t\}\\leq\\eta, andΨK′​\(βt\)​dt=η\\Psi\_\{K\}^\{\\prime\}\(\\beta\_\{t\}\)d\_\{t\}=\\eta\. Moreover1≤ΨK′​\(βt\+v\)/ΨK′​\(βt\)≤eη1\\leq\\Psi\_\{K\}^\{\\prime\}\(\\beta\_\{t\}\+v\)/\\Psi\_\{K\}^\{\\prime\}\(\\beta\_\{t\}\)\\leq e^\{\\eta\}for0≤v≤dt0\\leq v\\leq d\_\{t\}\. Integrating and summing givesη​t≤ΨK​\(βt\)≤eη​η​t\\eta t\\leq\\Psi\_\{K\}\(\\beta\_\{t\}\)\\leq e^\{\\eta\}\\eta t\. Define the positive burn\-in time

bK=min⁡\{t:βt≥ln⁡\(2​K\)\}\.b\_\{K\}=\\min\\\{t:\\beta\_\{t\}\\geq\\ln\(2K\)\\\}\.The preceding bounds, and the overshootdt≤ηd\_\{t\}\\leq\\eta, implybK≍ηln\(eK\)b\_\{K\}\\asymp\_\{\\eta\}\\ln\(eK\), includingK=2K=2\. Setxt=eβt/\(K−1\)x\_\{t\}=e^\{\\beta\_\{t\}\}/\(K\-1\)\. Its increment satisfies

xt​dt≤xt\+1−xt≤eη​xt​dt,xt​dt=η​KK−1​xt1\+xt\.x\_\{t\}d\_\{t\}\\leq x\_\{t\+1\}\-x\_\{t\}\\leq e^\{\\eta\}x\_\{t\}d\_\{t\},\\qquad x\_\{t\}d\_\{t\}=\\frac\{\\eta K\}\{K\-1\}\\frac\{x\_\{t\}\}\{1\+x\_\{t\}\}\.At burn\-in2≤xbK≤4​eη2\\leq x\_\{b\_\{K\}\}\\leq 4e^\{\\eta\}, and thereafter each increment is between two positive constants depending only onη\\eta\. HenceδbK\+k=1/\(1\+xbK\+k\)≍η\(1\+k\)−1\\delta\_\{b\_\{K\}\+k\}=1/\(1\+x\_\{b\_\{K\}\+k\}\)\\asymp\_\{\\eta\}\(1\+k\)^\{\-1\}\. Before burn\-in,xt<4​eηx\_\{t\}<4e^\{\\eta\}, soδt≍η1\\delta\_\{t\}\\asymp\_\{\\eta\}1\. This proves the shifted reciprocal bound at every integer time\.

For a held\-out row, independence and uniform query addresses giveN∼Binomial⁡\(SA,1/K\)N\\sim\\operatorname\{Binomial\}\(S\_\{A\},1/K\); writem=SA/Km=S\_\{A\}/K\. Ifm≥4​bKm\\geq 4b\_\{K\}, the binomial exponential\-moment bound givesPr\{N<m/2\}≤e−m/8\\Pr\\\{N<m/2\\\}\\leq e^\{\-m/8\}\. On the complementary event,δN2≤Cη​\(1\+m\)−2\\delta\_\{N\}^\{2\}\\leq C\_\{\\eta\}\(1\+m\)^\{\-2\}\. Thus

𝔼δN2≤Cη\(1\+m\)−2\+e−m/8,\\mathbb\{E\}\\delta\_\{N\}^\{2\}\\leq C\_\{\\eta\}\(1\+m\)^\{\-2\}\+e^\{\-m/8\},which is at mostε\\varepsilonwhenm≥Cη\{ln\(eK\)\+ε−1/2\}m\\geq C\_\{\\eta\}\\\{\\ln\(eK\)\+\\varepsilon^\{\-1/2\}\\\}\. For the reverse bound, the increment estimate holds globally fromx0≤1x\_\{0\}\\leq 1, givingxt≤1\+2​η​eη​tx\_\{t\}\\leq 1\+2\\eta e^\{\\eta\}t\. Convexity of\(2\+2​η​eη​t\)−2\(2\+2\\eta e^\{\\eta\}t\)^\{\-2\}and Jensen’s inequality imply𝔼​δN2≥\(2\+2​η​eη​m\)−2\\mathbb\{E\}\\delta\_\{N\}^\{2\}\\geq\(2\+2\\eta e^\{\\eta\}m\)^\{\-2\}\. Sinceε≤1/16\\varepsilon\\leq 1/16, this forcesm≥cηε−1/2m\\geq c\_\{\\eta\}\\varepsilon^\{\-1/2\}\. For the logarithmic term,βt≤2​η​t\\beta\_\{t\}\\leq 2\\eta t\. Ifm≤ln⁡\(K−1\)/\(8​η\)m\\leq\\ln\(K\-1\)/\(8\\eta\), Markov’s inequality gives probability at least1/21/2toN≤2​mN\\leq 2m, whereβN≤12​ln⁡\(K−1\)\\beta\_\{N\}\\leq\\frac\{1\}\{2\}\\ln\(K\-1\)andδN≥1/2\\delta\_\{N\}\\geq 1/2\. Then𝔼​δN2≥1/8\\mathbb\{E\}\\delta\_\{N\}^\{2\}\\geq 1/8, a contradiction\. This forcesm≳ηlnKm\\gtrsim\_\{\\eta\}\\ln K; boundedKK, includingK=2K=2, is already covered by the first lower bound\. Taking the maximum of the two lower bounds proves the claimed order of the hitting time\. With a single candidate routing is exact without calibration\. ∎

In the following restatement,Δi=Ri\\Delta\_\{i\}=R\_\{i\}denotes the same public logit radius as in the main\-text setup\.

###### Theorem C\.5\(Causal self\-attention realization; formal version of[Theorem4\.3](https://arxiv.org/html/2609.13500#S4.Thmtheorem3)\)\.

For the source in[Section3](https://arxiv.org/html/2609.13500#S3), let Stage A start from zero and use the routing update in[Proposition4\.2](https://arxiv.org/html/2609.13500#S4.Thmtheorem2)onSAS\_\{A\}independent calibration blocks\. Freeze the route before Stage V, which usesn≥1n\\geq 1independent labeled blocks\. With publicpi,Δip\_\{i\},\\Delta\_\{i\}, set

An=\{i:qi≥n−1\},𝒲=∏i∈An\[−Δi,Δi\],D=diag⁡\(pi\)i∈An,w1=0\.A\_\{n\}=\\\{i:q\_\{i\}\\geq n^\{\-1\}\\\},\\quad\\mathcal\{W\}=\\prod\_\{i\\in A\_\{n\}\}\[\-\\Delta\_\{i\},\\Delta\_\{i\}\],\\quad D=\\operatorname\{diag\}\(p\_\{i\}\)\_\{i\\in A\_\{n\}\},\\quad w\_\{1\}=0\.Apply[Equation4\.2](https://arxiv.org/html/2609.13500#S4.E2)to the attended feature restricted toAnA\_\{n\}, with zero output coordinates elsewhere, and use

μsgd=916​min\|z\|≤R​σ′​\(z\),ηt=2μsgd​\(t\+1\),w¯n=2n⁡\(n\+1\)​∑t=1nt​wt\.\\mu\_\{\\rm sgd\}=\\frac\{9\}\{16\}\\min\_\{\|z\|\\leq R\}\\sigma^\{\\prime\}\(z\),\\qquad\\eta\_\{t\}=\\frac\{2\}\{\\mu\_\{\\rm sgd\}\(t\+1\)\},\\qquad\\bar\{w\}\_\{n\}=\\frac\{2\}\{n\(n\+1\)\}\\sum\_\{t=1\}^\{n\}t\\,w\_\{t\}\.IfAnA\_\{n\}is empty, takew¯n=0\\bar\{w\}\_\{n\}=0\. For each public format\-range ratioκi≥1\\kappa\_\{i\}\\geq 1and integer widthk≥0k\\geq 0, the scalar decoder is defined as follows\. Ifκi​2−k≥1\\kappa\_\{i\}2^\{\-k\}\\geq 1, its output is zero\. Otherwise quantize uniformly with2k2^\{k\}midpoint values on\[−κi​Δi,κi​Δi\]\[\-\\kappa\_\{i\}\\Delta\_\{i\},\\kappa\_\{i\}\\Delta\_\{i\}\], then clip the decoded value to\[−Δi,Δi\]\[\-\\Delta\_\{i\},\\Delta\_\{i\}\]\. The allocation is public and deterministic; quantizew¯n\\bar\{w\}\_\{n\}with total width at mostBVB\_\{V\}, choosing an allocation within an arbitrarily small error of the infimum below and flooring its widths\. Letℰarithmetic\\mathcal\{E\}\_\{\\rm arithmetic\}bound the mean squared difference between the computed and exact logits of this stored network\. Then

𝔼\[ℛθ\(F^\)−ℒ∞\]≲R,K,ηΦμ\(n−1\)\+Dq,κfmt\(BV\)\+𝔯K\(SA\)\+ℰarithmetic,\\mathbb\{E\}\[\\mathcal\{R\}\_\{\\theta\}\(\\widehat\{F\}\)\-\\mathcal\{L\}\_\{\\infty\}\]\\lesssim\_\{R,K,\\eta\}\\Phi\_\{\\mu\}\(n^\{\-1\}\)\+D^\{\\rm fmt\}\_\{q,\\kappa\}\(B\_\{V\}\)\+\\mathfrak\{r\}\_\{K\}\(S\_\{A\}\)\+\\mathcal\{E\}\_\{\\rm arithmetic\},\(C\.5\)whereBVB\_\{V\}is the value\-state bit budget and

Dq,κfmt​\(BV\)=infbi≥0,∑ibi≤BV∑iqi​min⁡\{1,κi2​2−2​bi\}\.D^\{\\rm fmt\}\_\{q,\\kappa\}\(B\_\{V\}\)=\\inf\_\{b\_\{i\}\\geq 0,\\,\\sum\_\{i\}b\_\{i\}\\leq B\_\{V\}\}\\sum\_\{i\}q\_\{i\}\\min\\\{1,\\kappa\_\{i\}^\{2\}2^\{\-2b\_\{i\}\}\\\}\.Forκi=1\\kappa\_\{i\}=1, this equalsΦμ​\(τBV\)\\Phi\_\{\\mu\}\(\\tau\_\{B\_\{V\}\}\), and the network attains the core surface at total budgetBBfor power\-law spectra, fixedKK,o⁡\(B\)o\(B\)routing bits, and lower\-order routing/arithmetic errors\.

###### Proof\.

The finite active set has sizem=\|An\|≤n​∑iqim=\|A\_\{n\}\|\\leq n\\sum\_\{i\}q\_\{i\}\. We first analyze the exact output before storage, conditioning on Stage A\.

#### Feature covariance and curvature\.

ForK≥2K\\geq 2, writeIIfor the matching candidate’s mode andI1,…,IK−1I\_\{1\},\\ldots,I\_\{K\-1\}for the distractor modes\. They are independent with lawpp, also conditional on the query row and its calibration history\. Restricting the coordinate vectors toAnA\_\{n\}, the feature is

h=\(1−δq\)​eI\+δqK−1​∑s=1K−1eIs\.h=\(1\-\\delta\_\{q\}\)e\_\{I\}\+\\frac\{\\delta\_\{q\}\}\{K\-1\}\\sum\_\{s=1\}^\{K\-1\}e\_\{I\_\{s\}\}\.Expanding its second moment separates equal\-candidate and distinct\-candidate products:

𝔼⁡\[h​h⊤∣Q=q\]=cd,q​D\+\(1−cd,q\)​pA​pA⊤,cd,q=\(1−δq\)2\+δq2K−1,\\mathbb\{E\}\[hh^\{\\top\}\\mid Q=q\]=c\_\{d,q\}D\+\(1\-c\_\{d,q\}\)p\_\{A\}p\_\{A\}^\{\\top\},\\qquad c\_\{d,q\}=\(1\-\\delta\_\{q\}\)^\{2\}\+\\frac\{\\delta\_\{q\}^\{2\}\}\{K\-1\},wherepA=\(pi\)i∈Anp\_\{A\}=\(p\_\{i\}\)\_\{i\\in A\_\{n\}\}\. Withrcal=K−1​∑qδq2r\_\{\\rm cal\}=K^\{\-1\}\\sum\_\{q\}\\delta\_\{q\}^\{2\}andc¯d=K−1​∑qcd,q\\bar\{c\}\_\{d\}=K^\{\-1\}\\sum\_\{q\}c\_\{d,q\}, the event𝒢=\{rcal≤1/16\}\\mathcal\{G\}=\\\{r\_\{\\rm cal\}\\leq 1/16\\\}gives

c¯d≥\(1−1K​∑qδq\)2≥916\.\\bar\{c\}\_\{d\}\\geq\\left\(1\-\\frac\{1\}\{K\}\\sum\_\{q\}\\delta\_\{q\}\\right\)^\{2\}\\geq\\frac\{9\}\{16\}\.LetF⁡\(w\)=𝔼⁡\[log⁡\(1\+ew⊤​h\)−Y​w⊤​h∣Stage A\]F\(w\)=\\mathbb\{E\}\[\\log\(1\+e^\{w^\{\\top\}h\}\)\-Yw^\{\\top\}h\\mid\\text\{Stage A\}\]\. Everyw∈𝒲w\\in\\mathcal\{W\}has\|w⊤​h\|≤R\|w^\{\\top\}h\|\\leq R\. Hence, on𝒢\\mathcal\{G\}, its Hessian satisfies∇2F​\(w\)⪰μsgd​D\\nabla^\{2\}F\(w\)\\succeq\\mu\_\{\\rm sgd\}D, withμsgd\\mu\_\{\\rm sgd\}as in the statement\. The stochastic gradientgt=\(σ⁡\(wt⊤​ht\)−Yt\)​htg\_\{t\}=\(\\sigma\(w\_\{t\}^\{\\top\}h\_\{t\}\)\-Y\_\{t\}\)h\_\{t\}is unbiased conditional on the past, and, form≥1m\\geq 1,

𝔼\[∥gt∥D−12∣wt,Stage A\]≤tr\(D−1𝔼hh⊤\)=c¯dm\+\(1−c¯d\)∑i∈Anpi≤m\.\\mathbb\{E\}\[\\\|g\_\{t\}\\\|\_\{D^\{\-1\}\}^\{2\}\\mid w\_\{t\},\\text\{Stage A\}\]\\leq\\operatorname\{tr\}\(D^\{\-1\}\\mathbb\{E\}hh^\{\\top\}\)=\\bar\{c\}\_\{d\}m\+\(1\-\\bar\{c\}\_\{d\}\)\\sum\_\{i\\in A\_\{n\}\}p\_\{i\}\\leq m\.ForK=1K=1, useh=eIh=e\_\{I\},c¯d=1\\bar\{c\}\_\{d\}=1, andrcal=0r\_\{\\rm cal\}=0; all the same inequalities hold\.

#### Projected SGD and its output\.

The compact box has a minimizerw∗w^\{\*\}ofFF\. Nonexpansiveness of projection in theDD\-norm and strong convexity give, conditional on𝒢\\mathcal\{G\}and the calibration history,

2​ηt​𝔼​\[F⁡\(wt\)−F⁡\(w∗\)\]≤\(1−μsgd​ηt\)​Vt−Vt\+1\+ηt2​m,Vt=𝔼​‖wt−w∗‖D2\.2\\eta\_\{t\}\\mathbb\{E\}\[F\(w\_\{t\}\)\-F\(w^\{\*\}\)\]\\leq\(1\-\\mu\_\{\\rm sgd\}\\eta\_\{t\}\)V\_\{t\}\-V\_\{t\+1\}\+\\eta\_\{t\}^\{2\}m,\\qquad V\_\{t\}=\\mathbb\{E\}\\\|w\_\{t\}\-w^\{\*\}\\\|\_\{D\}^\{2\}\.Multiply byt/\(2​ηt\)t/\(2\\eta\_\{t\}\)and substitute the specified step size:

t​𝔼​\[F⁡\(wt\)−F⁡\(w∗\)\]≤μsgd4​\{t⁡\(t−1\)​Vt−t⁡\(t\+1\)​Vt\+1\}\+m​tμsgd​\(t\+1\)\.t\\mathbb\{E\}\[F\(w\_\{t\}\)\-F\(w^\{\*\}\)\]\\leq\\frac\{\\mu\_\{\\rm sgd\}\}\{4\}\\\{t\(t\-1\)V\_\{t\}\-t\(t\+1\)V\_\{t\+1\}\\\}\+\\frac\{mt\}\{\\mu\_\{\\rm sgd\}\(t\+1\)\}\.The distance terms telescope, with zero initial coefficient and a nonpositive final term\. Convexity ofFFtherefore yields

𝔼⁡\[F⁡\(w¯n\)−F⁡\(w∗\)\]≤2​mμsgd​\(n\+1\)\.\\mathbb\{E\}\[F\(\\bar\{w\}\_\{n\}\)\-F\(w^\{\*\}\)\]\\leq\\frac\{2m\}\{\\mu\_\{\\rm sgd\}\(n\+1\)\}\.

#### Comparison with the source\.

LetθiA=θi𝟏\{i∈An\}\\theta^\{A\}\_\{i\}=\\theta\_\{i\}\\mathbf\{1\}\\\{i\\in A\_\{n\}\\\}\. It belongs to𝒲\\mathcal\{W\}, and

\|θI−\(θA\)⊤h\|≤\|θI\|𝟏\{I∉An\}\+2Rδq\.\|\\theta\_\{I\}\-\(\\theta^\{A\}\)^\{\\top\}h\|\\leq\|\\theta\_\{I\}\|\\mathbf\{1\}\\\{I\\notin A\_\{n\}\\\}\+2R\\delta\_\{q\}\.Squaring, averaging, and applying upper logit curvature give

F⁡\(w∗\)−ℒ∞≤F⁡\(θA\)−ℒ∞≤CR​\(∑i∉Anqi\+rcal\)\.F\(w^\{\*\}\)\-\\mathcal\{L\}\_\{\\infty\}\\leq F\(\\theta^\{A\}\)\-\\mathcal\{L\}\_\{\\infty\}\\leq C\_\{R\}\\left\(\\sum\_\{i\\notin A\_\{n\}\}q\_\{i\}\+r\_\{\\rm cal\}\\right\)\.On𝒢c\\mathcal\{G\}^\{c\}, every exact output remains in\[−R,R\]\[\-R,R\], so its excess loss is bounded by a constant depending only onRR\. FurthermorePr⁡\(𝒢c\)≤16​𝔼​rcal\\Pr\(\\mathcal\{G\}^\{c\}\)\\leq 16\\mathbb\{E\}r\_\{\\rm cal\}, and independence of a held\-out query gives𝔼​rcal=𝔯K​\(SA\)\\mathbb\{E\}r\_\{\\rm cal\}=\\mathfrak\{r\}\_\{K\}\(S\_\{A\}\)\. Combining the events proves

𝔼⁡\[F⁡\(w¯n\)−ℒ∞\]≤CR​\{mn\+∑i∉Anqi\+𝔯K​\(SA\)\}=CR​\{Φμ​\(n−1\)\+𝔯K​\(SA\)\}\.\\mathbb\{E\}\[F\(\\bar\{w\}\_\{n\}\)\-\\mathcal\{L\}\_\{\\infty\}\]\\leq C\_\{R\}\\left\\\{\\frac\{m\}\{n\}\+\\sum\_\{i\\notin A\_\{n\}\}q\_\{i\}\+\\mathfrak\{r\}\_\{K\}\(S\_\{A\}\)\\right\\\}=C\_\{R\}\\\{\\Phi\_\{\\mu\}\(n^\{\-1\}\)\+\\mathfrak\{r\}\_\{K\}\(S\_\{A\}\)\\\}\.Whenm=0m=0, the zero output pays at mostCR​∑iqi=CR​Φμ​\(n−1\)C\_\{R\}\\sum\_\{i\}q\_\{i\}=C\_\{R\}\\Phi\_\{\\mu\}\(n^\{\-1\}\), so the same bound holds\.

#### Finite\-state representation\.

For anyx∈\[−Δi,Δi\]x\\in\[\-\\Delta\_\{i\},\\Delta\_\{i\}\], the statedkk\-bit decoder obeys

\|Qi,k​\(x\)−x\|2≤Δi2​min⁡\{1,κi2​2−2​k\}\.\|Q\_\{i,k\}\(x\)\-x\|^\{2\}\\leq\\Delta\_\{i\}^\{2\}\\min\\\{1,\\kappa\_\{i\}^\{2\}2^\{\-2k\}\\\}\.Indeed, the zero branch has error at mostΔi\\Delta\_\{i\}; in the grid branch the midpoint error is at mostκi​Δi​2−k\\kappa\_\{i\}\\Delta\_\{i\}2^\{\-k\}, and clipping decreases it\. Both the branch and the codebook are public\. Roundingbib\_\{i\}down tokik\_\{i\}multiplies this upper bound by at most four\. Only finitely manykik\_\{i\}are positive, and their product codebook has at most2∑iki≤2BV2^\{\\sum\_\{i\}k\_\{i\}\}\\leq 2^\{B\_\{V\}\}entries\. A zero\-width coordinate is decoded as zero becauseκi≥1\\kappa\_\{i\}\\geq 1\.

Extendw¯n\\bar\{w\}\_\{n\}by zero outsideAnA\_\{n\}, and apply the public scalar decoder on every allocated coordinate\. Letei=Qi,ki​\(w¯n,i\)−w¯n,ie\_\{i\}=Q\_\{i,k\_\{i\}\}\(\\bar\{w\}\_\{n,i\}\)\-\\bar\{w\}\_\{n,i\}and letH=∑saQ​s​eCsH=\\sum\_\{s\}a\_\{Qs\}e\_\{C\_\{s\}\}be the full attended coordinate feature\. Conditional on the fitted state, Jensen’s inequality and the iid candidate marginals give

𝔼⁡\[\(e⊤​H\)2∣fitted state\]≤𝔼⁡\[∑saQ​s​eCs2∣fitted state\]=∑ipi​ei2\.\\mathbb\{E\}\[\(e^\{\\top\}H\)^\{2\}\\mid\\text\{fitted state\}\]\\leq\\mathbb\{E\}\\\!\\left\[\\sum\_\{s\}a\_\{Qs\}e\_\{C\_\{s\}\}^\{2\}\\mid\\text\{fitted state\}\\right\]=\\sum\_\{i\}p\_\{i\}e\_\{i\}^\{2\}\.It follows that quantizing the average costs at most4​∑iqi​min⁡\{1,κi2​2−2​bi\}4\\sum\_\{i\}q\_\{i\}\\min\\\{1,\\kappa\_\{i\}^\{2\}2^\{\-2b\_\{i\}\}\\\}in mean squared logit error\. Lower curvature converts the preceding learning bound to squared logit error; the squared triangle inequality adds storage and arithmetic error, and global upper curvature converts back to cross\-entropy\. Taking the infimum over allocations proves[Equation4\.3](https://arxiv.org/html/2609.13500#S4.E3)\.

Finally, a calibrated row is determined by its visit count and its matching position, with one additional symbol for an unvisited row\. It therefore uses at most

BA=K⁡\{⌈log2⁡\(SA\+1\)⌉\+⌈log2⁡\(K\+1\)⌉\}B\_\{A\}=K\\\{\\lceil\\log\_\{2\}\(S\_\{A\}\+1\)\\rceil\+\\lceil\\log\_\{2\}\(K\+1\)\\rceil\\\}bits: the fixed decoder reconstructs its scores from the recurrence\. The retained value and routing state together use at mostBV\+BAB\_\{V\}\+B\_\{A\}bits; the stated arithmetic error accounts for numerical reconstruction\. ForK=1K=1the route uses no state\. Whenκi=1\\kappa\_\{i\}=1, the format functional is exactlyDq​\(BV\)=Φμ​\(τBV\)D\_\{q\}\(B\_\{V\}\)=\\Phi\_\{\\mu\}\(\\tau\_\{B\_\{V\}\}\), giving the final claim at value budgetBVB\_\{V\}\. For ranked coordinate energiesq\(r\)≍r−\(1\+ξ\)q\_\{\(r\)\}\\asymp r^\{\-\(1\+\\xi\)\},Dq​\(BV\)≍BV−ξD\_\{q\}\(B\_\{V\}\)\\asymp B\_\{V\}^\{\-\\xi\}; in the equal\-energy block profile,ξ=χ/\(1\+γ\)\\xi=\\chi/\(1\+\\gamma\)\. IfBA=o⁡\(B\)B\_\{A\}=o\(B\)andBV=B−BAB\_\{V\}=B\-B\_\{A\}, thenBV≍BB\_\{V\}\\asymp B, proving the total\-budget conclusion\. ∎

###### Theorem C\.6\(Same block marginals, different scaling laws; formal version of[Theorem3\.3](https://arxiv.org/html/2609.13500#S3.Thmtheorem3)\)\.

For everyχ,γ\>0\\chi,\\gamma\>0, there are two positive\-conditional\-entropy causal\-retrieval sources with exactly the same block\-energy multiset and the same block\-dimension multiset at every dyadic level\. They differ only in the pairing\. All displayed dimensions in this construction are rounded upward to integers\. With the common fixed\-KKroute revealed,

ℜeasy\(n,B\)≍n−χ/\(1\+χ\)\+B−χ,\\mathfrak\{R\}\_\{\\rm easy\}\(n,B\)\\asymp n^\{\-\\chi/\(1\+\\chi\)\}\+B^\{\-\\chi\},whereas

ℜhard\(n,B\)≍n−χ/\(1\+χ\+γ\)\+B−χ/\(1\+γ\)\.\\mathfrak\{R\}\_\{\\rm hard\}\(n,B\)\\asymp n^\{\-\\chi/\(1\+\\chi\+\\gamma\)\}\+B^\{\-\\chi/\(1\+\\gamma\)\}\.

###### Proof\.

Fixδ\>γ​χ\\delta\>\\gamma\\chi\. At each levelℓ≥1\\ell\\geq 1, take2ℓ2^\{\\ell\}blocks of energyc​2−\(1\+χ\)​ℓc2^\{\-\(1\+\\chi\)\\ell\}and2ℓ2^\{\\ell\}blocks of energyc​2−\(1\+χ\+δ\)​ℓc2^\{\-\(1\+\\chi\+\\delta\)\\ell\}\. The dimension multiset consists of2ℓ2^\{\\ell\}copies each of11anddℓ=⌈2γ​ℓ⌉d\_\{\\ell\}=\\lceil 2^\{\\gamma\\ell\}\\rceil\. In the easy pairing the higher energy has dimension one; in the hard pairing it has dimensiondℓd\_\{\\ell\}\. Divide a block’s energy equally among its coordinates\. Both multisets agree exactly at every level, including after the integer rounding of dimensions\.

To realize these spectra as sources, choose any fixed radiusR0\>0R\_\{0\}\>0within the logit bound, and choosec\>0c\>0so that total energy isR02R\_\{0\}^\{2\}\. Give a coordinate of energyqiq\_\{i\}probabilitypi=qi/R02p\_\{i\}=q\_\{i\}/R\_\{0\}^\{2\}and radiusΔi=R0\\Delta\_\{i\}=R\_\{0\}\. These probabilities sum to one\. Independent candidate draws and the same fixed\-KKretrieval map give the source in[Section3](https://arxiv.org/html/2609.13500#S3); bounded Bernoulli logits give strictly positive conditional entropy\.

A generic component with exponents\>0s\>0and dimension exponentg≥0g\\geq 0hasΘ⁡\(2\(1\+g\)​ℓ\)\\Theta\(2^\{\(1\+g\)\\ell\}\)coordinates of energyΘ⁡\(2−\(1\+s\+g\)​ℓ\)\\Theta\(2^\{\-\(1\+s\+g\)\\ell\}\)at levelℓ\\ell\. The active levels at thresholdttlie between two cutoffs differing by a constant number of levels, each of sizelog2⁡\(1/t\)/\(1\+s\+g\)\\log\_\{2\}\(1/t\)/\(1\+s\+g\)\. A geometric sum therefore gives

Ns,g​\(t\)≍t−ρ⁡\(s,g\),ρ⁡\(s,g\)=1\+g1\+s\+g\.N\_\{s,g\}\(t\)\\asymp t^\{\-\\rho\(s,g\)\},\\qquad\\rho\(s,g\)=\\frac\{1\+g\}\{1\+s\+g\}\.Use the two\-sided integral comparison at the end of the preceding spectral proof, rather than regular variation: dyadic spectra can have log\-periodic fluctuations\. This gives data and storage exponentss/\(1\+s\+g\)s/\(1\+s\+g\)ands/\(1\+g\)s/\(1\+g\), respectively\. The easy components have\(s,g\)=\(χ,0\),\(χ\+δ,γ\)\(s,g\)=\(\\chi,0\),\(\\chi\+\\delta,\\gamma\)\. The inequalityδ\>γ​χ\\delta\>\\gamma\\chimakes the second component’s two exponents strictly larger\. The hard components have\(s,g\)=\(χ,γ\),\(χ\+δ,0\)\(s,g\)=\(\\chi,\\gamma\),\(\\chi\+\\delta,0\), again with the second faster\. For data, component errors add exactly\. For storage, each full distortion is at least that of the slower component at budgetBB, while allocatingB/2B/2to each component gives the matching upper bound\. Applying[TheoremC\.1](https://arxiv.org/html/2609.13500#A3.Thmtheorem1)now proves all four displayed rates\. ∎

## Appendix DPreliminaries and Notation

We begin by specifying what is observed, what is predicted, and which resources are counted\.[SectionD\.1](https://arxiv.org/html/2609.13500#A4.SS1)introduces the regenerative autoregressive source\.[SectionD\.2](https://arxiv.org/html/2609.13500#A4.SS2)defines held\-out loss and learners with finitely many data\-dependent states\.[SectionD\.3](https://arxiv.org/html/2609.13500#A4.SS3)adds mode\-dependent representation costs, and[SectionD\.4](https://arxiv.org/html/2609.13500#A4.SS4)distinguishes learned bits from physical storage and arithmetic cost\.

### D\.1A regenerative autoregressive source

This section writes the revealed\-coordinate value problem with scalar indexjj:Δj\\Delta\_\{j\}is the logit radius denotedRiR\_\{i\}in the main text, and𝔈∗​\(n,B\)\\mathfrak\{E\}^\{\*\}\(n,B\)below isℜvalue∗​\(n,B\)\\mathfrak\{R\}^\{\*\}\_\{\\rm value\}\(n,B\)\. For a positive integermm, write\[m\]=\{1,…,m\}\[m\]=\\\{1,\\ldots,m\\\}, and letℕ0=\{0,1,2,…\}\\mathbb\{N\}\_\{0\}=\\\{0,1,2,\\ldots\\\}\. We writeBer⁡\(p\)\\Ber\(p\)for the Bernoulli distribution with success probabilitypp\.

One regeneration block containsKKcandidate tokens followed by a query and a binary target token\. Candidatet∈\[K\]t\\in\[K\]carries a public addressata\_\{t\}and an observed mode labeljtj\_\{t\}\. The query carries one addressat⋆a\_\{t\_\{\\star\}\}, so its causally relevant mode is

J≔jt⋆,ℙ⁡\(J=j\)=pj\.J\\coloneqq j\_\{t\_\{\\star\}\},\\qquad\\mathbb\{P\}\(J=j\)=p\_\{j\}\.Conditionally onJ=jJ=j, the next token obeys

Y∣J=j∼Ber\(σ\(θj\)\),σ\(z\)=11\+e−z\.Y\\mid J=j\\sim\\Ber\(\\sigma\(\\theta\_\{j\}\)\),\\qquad\\sigma\(z\)=\\frac\{1\}\{1\+e^\{\-z\}\}\.The parameter isθ=\(θj\)j≥1\\theta=\(\\theta\_\{j\}\)\_\{j\\geq 1\}withθj∈\[−Δj,Δj\]⊆\[−R,R\]\\theta\_\{j\}\\in\[\-\\Delta\_\{j\},\\Delta\_\{j\}\]\\subseteq\[\-R,R\], whereR<∞R<\\inftyis fixed\. We define the predictive energy

qj≔pj​Δj2q\_\{j\}\\coloneqq p\_\{j\}\\Delta\_\{j\}^\{2\}and assume∑jqj<∞\\sum\_\{j\}q\_\{j\}<\\infty\. We writeZ=\(\{\(at,jt\)\}t=1K,at⋆,Y\)∈𝒵Z=\(\\\{\(a\_\{t\},j\_\{t\}\)\\\}\_\{t=1\}^\{K\},a\_\{t\_\{\\star\}\},Y\)\\in\\mathcal\{Z\}for the observed training block; the source parameters\(θj\)j\(\\theta\_\{j\}\)\_\{j\}are unknown\. A regeneration delimiter resets the random variables, making distinct blocks independent\. Tokens inside a block are causally ordered, but the statistical sample sizenncounts independent blocks, not tokens\. Bounded logits imply0<σ⁡\(−R\)≤ℙ⁡\(Y=1∣J\)≤σ⁡\(R\)<10<\\sigma\(\-R\)\\leq\\mathbb\{P\}\(Y=1\\mid J\)\\leq\\sigma\(R\)<1, so the conditional entropy and Bayes cross\-entropy are strictly positive\.

### D\.2Held\-out risk and finite learned states

For a logit predictorf=\(fj\)jf=\(f\_\{j\}\)\_\{j\}, define held\-out cross\-entropy

ℒθ​\(f\)≔𝔼θ​\[−Y​log⁡σ⁡\(fJ\)−\(1−Y\)​log⁡\(1−σ⁡\(fJ\)\)\]\.\\mathcal\{L\}\_\{\\theta\}\(f\)\\coloneqq\\mathbb\{E\}\_\{\\theta\}\\\!\\left\[\-Y\\log\\sigma\(f\_\{J\}\)\-\(1\-Y\)\\log\(1\-\\sigma\(f\_\{J\}\)\)\\right\]\.The Bayes predictor isfj=θjf\_\{j\}=\\theta\_\{j\}; its loss is denotedℒ∞≔ℒθ​\(θ\)\>0\\mathcal\{L\}\_\{\\infty\}\\coloneqq\\mathcal\{L\}\_\{\\theta\}\(\\theta\)\>0\. Training and evaluation regeneration blocks are independent\.

For an integerB≥0B\\geq 0, aBB\-bit learner consists of an encoder

ϕ:𝒵n⟶\[2B\]\\phi:\\mathcal\{Z\}^\{n\}\\longrightarrow\[2^\{B\}\]and a fixed decoderg:\[2B\]×ℕ→\[−R,R\]g:\[2^\{B\}\]\\times\\mathbb\{N\}\\to\[\-R,R\]\. The decoder may share arbitrary computation across modes; only its data\-dependent state is limited to2B2^\{B\}possibilities\. Randomized learners are allowed, although conditioning on their random seed shows that they do not improve the Bayes lower bounds used below\. Because the Bayes loss depends onθ\\theta, we define minimax*excess*risk directly and use the notation

𝔈∗​\(n,B\)≔infϕ,gsupθj∈\[−Δj,Δj\]𝔼θ​\[ℒθ​\(g⁡\(ϕ⁡\(Zn\),⋅\)\)−ℒθ​\(θ\)\]\.\\mathfrak\{E\}^\{\*\}\(n,B\)\\coloneqq\\inf\_\{\\phi,g\}\\sup\_\{\\theta\_\{j\}\\in\[\-\\Delta\_\{j\},\\Delta\_\{j\}\]\}\\mathbb\{E\}\_\{\\theta\}\\\!\\left\[\\mathcal\{L\}\_\{\\theta\}\(g\(\\phi\(Z^\{n\}\),\\cdot\)\)\-\\mathcal\{L\}\_\{\\theta\}\(\\theta\)\\right\]\.We usex≍yx\\asymp ywhen positive constants bound their ratio in both directions\. These constants may depend onRRand on stated regular\-variation constants, but not onnn,BB, or the number of active modes\.

### D\.3Heterogeneous representation costs

The statistical theorem usesqjq\_\{j\}\. The structural theorem allows a more general representable energyaj\>0a\_\{j\}\>0, a per\-bit difficultycj\>0c\_\{j\}\>0, and a numerical amplificationκj\>0\\kappa\_\{j\}\>0\. Assigningbj≥0b\_\{j\}\\geq 0bits to modejjincurs distortionaj​2−2​bja\_\{j\}2^\{\-2b\_\{j\}\}and costcj​bjc\_\{j\}b\_\{j\}\. Its optimal distortion under budgetBBis

D⁡\(B\)≔infbj≥0∑jcj​bj≤B∑jaj​2−2​bj\.D\(B\)\\coloneqq\\inf\_\{\\begin\{subarray\}\{c\}b\_\{j\}\\geq 0\\\\ \\sum\_\{j\}c\_\{j\}b\_\{j\}\\leq B\\end\{subarray\}\}\\sum\_\{j\}a\_\{j\}2^\{\-2b\_\{j\}\}\.\(D\.1\)Letvj≥0v\_\{j\}\\geq 0record a separately specified statistical residual for modejj\. This auxiliary quantity is used in the complete tuple summary below, not in the spectral formula forD⁡\(B\)D\(B\)\. For a finite mode collection, the unlabeled joint measure

η≔∑jδ\(vj,aj,cj,κj\)\\eta\\coloneqq\\sum\_\{j\}\\delta\_\{\(v\_\{j\},a\_\{j\},c\_\{j\},\\kappa\_\{j\}\)\}\(D\.2\)records statistical residualvjv\_\{j\}, representable energy, difficulty, and amplification together\. The cost\-weighted difficulty\-normalized spectrum is

μ≔∑jcj​δaj/cj\.\\mu\\coloneqq\\sum\_\{j\}c\_\{j\}\\,\\delta\_\{a\_\{j\}/c\_\{j\}\}\.\(D\.3\)For unit costs andaj=qja\_\{j\}=q\_\{j\}, this reduces to the coordinate spectrum in[Section3](https://arxiv.org/html/2609.13500#S3); general costs weight each atom bycjc\_\{j\}\. We will compare unlabeled summaries of these mode\-dependent quantities\. Such a summary is sufficient for the complete budget–distortion curve if equal summary values imply equalD⁡\(B\)D\(B\)for everyB≥0B\\geq 0\.

### D\.4Neural and arithmetic resources

In[Section5](https://arxiv.org/html/2609.13500#S5.SS0.SSS0.Px2),X∈ℝ\(K\+1\)×dX\\in\\mathbb\{R\}^\{\(K\+1\)\\times d\}denotes the matrix of candidate and query token vectors, andU∈ℝd×dU\\in\\mathbb\{R\}^\{d\\times d\}denotes the bilinear attention score matrix\. The symbolNNcounts physically stored dense scalar parameter slots\. It is not identified withBB, the number of bits in the data\-dependent learned state\.

The symbolCCdenotes a non\-statistical cost budget: representation cost in[AppendixF](https://arxiv.org/html/2609.13500#A6)and arithmetic computation in[SectionsB\.3](https://arxiv.org/html/2609.13500#A2.SS3)and[6](https://arxiv.org/html/2609.13500#S6)\. These meanings occur in separate results and are never equated\. WhenCCdenotes computation, its arithmetic cost model is stated explicitly\. In[Section6](https://arxiv.org/html/2609.13500#S6), onebb\-bit scalar operation costsΘ⁡\(bν\)\\Theta\(b^\{\\nu\}\), whereν≥0\\nu\\geq 0is fixed\. In[SectionB\.3](https://arxiv.org/html/2609.13500#A2.SS3),ℓ\\ell,dd,mffm\_\{\\rm ff\},HH, andLLdenote sequence length, hidden width, feed\-forward width, attention\-head count, and layer count\. The symbolbmmb\_\{\\rm mm\}indexes a matrix\-arithmetic format family whose operand, output, and accumulator widths are prescribed functions ofbmmb\_\{\\rm mm\};bsmb\_\{\\rm sm\}specifies the effective softmax precision; andbstb\_\{\\rm st\}is the number of bits used to store an attention probability\. Unit roundoff is denoteduu\. These variables specify arithmetic work and storage;[AppendixA](https://arxiv.org/html/2609.13500#A1)discusses their relation to implementations\.

## Appendix ELearning after Data: A Finite\-Bit Autoregressive Minimax Law

Two irreducible losses govern finite\-bit prediction\. We first lower\-bound the distortion of any learned state with at most2B2^\{B\}possible values, even when its coordinates are decoded jointly\. We then combine this bound with statistical uncertainty across regeneration blocks and evaluate the resulting risk law for power\-law energies\. Full proofs appear in[AppendixG](https://arxiv.org/html/2609.13500#A7)\.

###### Lemma E\.1\(Arbitrary joint finite\-state representations; informal version of[LemmaG\.1](https://arxiv.org/html/2609.13500#A7.Thmtheorem1)\)\.

LetΘj\\Theta\_\{j\}be independent and uniform on\[−Δj,Δj\]\[\-\\Delta\_\{j\},\\Delta\_\{j\}\], letpj\>0p\_\{j\}\>0, and putqj=pj​Δj2q\_\{j\}=p\_\{j\}\\Delta\_\{j\}^\{2\}\. ForB∈ℕ0B\\in\\mathbb\{N\}\_\{0\}, every encoder–decoder pair with at most2B2^\{B\}reconstruction vectors satisfies

∑jpj​𝔼​\(Θj−Θ^j\)2≥2π​e​infbj≥0∑jbj≤B∑jqj​2−2​bj\.\\sum\_\{j\}p\_\{j\}\\mathbb\{E\}\(\\Theta\_\{j\}\-\\widehat\{\\Theta\}\_\{j\}\)^\{2\}\\geq\\frac\{2\}\{\\pi e\}\\inf\_\{\\begin\{subarray\}\{c\}b\_\{j\}\\geq 0\\\\ \\sum\_\{j\}b\_\{j\}\\leq B\\end\{subarray\}\}\\sum\_\{j\}q\_\{j\}2^\{\-2b\_\{j\}\}\.\(E\.1\)The reconstruction vectors may couple all coordinates\.

Here the*codebook*is simply the set of at most2B2^\{B\}reconstruction vectors available to the decoder\. The entropy proof controls total codebook cardinality, so it covers shared representations that compress several modes jointly as well as coordinatewise quantizers\.

###### Theorem E\.2\(Finite\-bit autoregressive minimax law; informal version of[TheoremG\.2](https://arxiv.org/html/2609.13500#A7.Thmtheorem2)\)\.

FixR<∞R<\\infty\. For everyn∈ℕn\\in\\mathbb\{N\},B∈ℕ0B\\in\\mathbb\{N\}\_\{0\}, and source family specified by\(pj,Δj\)j≥1\(p\_\{j\},\\Delta\_\{j\}\)\_\{j\\geq 1\}as in[SectionD\.1](https://arxiv.org/html/2609.13500#A4.SS1), there are constants0<cR≤CR<∞0<c\_\{R\}\\leq C\_\{R\}<\\inftysuch that

cR​\{Sq​\(n\)\+Dq​\(B\)\}≤𝔈∗​\(n,B\)≤CR​\{Sq​\(n\)\+Dq​\(B\)\},c\_\{R\}\\\{S\_\{q\}\(n\)\+D\_\{q\}\(B\)\\\}\\leq\\mathfrak\{E\}^\{\*\}\(n,B\)\\leq C\_\{R\}\\\{S\_\{q\}\(n\)\+D\_\{q\}\(B\)\\\},\(E\.2\)where

Sq​\(n\)≔∑jmin⁡\{qj,n−1\},Dq​\(B\)≔infbj≥0∑jbj≤B∑jqj​2−2​bj\.S\_\{q\}\(n\)\\coloneqq\\sum\_\{j\}\\min\\\{q\_\{j\},n^\{\-1\}\\\},\\qquad D\_\{q\}\(B\)\\coloneqq\\inf\_\{\\begin\{subarray\}\{c\}b\_\{j\}\\geq 0\\\\ \\sum\_\{j\}b\_\{j\}\\leq B\\end\{subarray\}\}\\sum\_\{j\}q\_\{j\}2^\{\-2b\_\{j\}\}\.\(E\.3\)The lower bound holds for arbitrary2B2^\{B\}\-state joint encoders and arbitrary fixed shared decoding computation\.

#### Why the two terms add\.

On bounded logits, excess Bernoulli cross\-entropy is equivalent to squared logit error\. A clipped modewise likelihood estimate contributesmin⁡\{qj,1/n\}\\min\\\{q\_\{j\},1/n\\\}: a rare or weak mode is no harder to ignore than to estimate\. Quantizing the resulting coordinates according to a feasible\(bj\)\(b\_\{j\}\)contributesqj​2−2​bjq\_\{j\}2^\{\-2b\_\{j\}\}\. For the converse, an Assouad hypercube gives the statistical term\. Giving the encoder the true parameter can only make representation easier;[LemmaE\.1](https://arxiv.org/html/2609.13500#A5.Thmtheorem1)then gives the bit term\. The actual risk exceeds the maximum of these two lower bounds, which is at least half their sum\. This proves the constant\-factor equivalence in[EquationE\.2](https://arxiv.org/html/2609.13500#A5.E2)\.

###### Corollary E\.3\(Power\-law energies; informal version of[CorollaryG\.3](https://arxiv.org/html/2609.13500#A7.Thmtheorem3)\)\.

Suppose thatn,B≥1n,B\\geq 1and, for someχ\>0\\chi\>0and constants0<c−≤c\+<∞0<c\_\{\-\}\\leq c\_\{\+\}<\\infty,

c−​j−\(1\+χ\)≤qj≤c\+​j−\(1\+χ\)\(j≥1\)\.c\_\{\-\}j^\{\-\(1\+\\chi\)\}\\leq q\_\{j\}\\leq c\_\{\+\}j^\{\-\(1\+\\chi\)\}\\qquad\(j\\geq 1\)\.Then

𝔈∗\(n,B\)≍n−χ/\(1\+χ\)\+B−χ\.\\mathfrak\{E\}^\{\*\}\(n,B\)\\asymp n^\{\-\\chi/\(1\+\\chi\)\}\+B^\{\-\\chi\}\.\(E\.4\)The data–bit crossover occurs atB≍n1/\(1\+χ\)B\\asymp n^\{1/\(1\+\\chi\)\}\.

To see the two exponents, splitSq​\(n\)S\_\{q\}\(n\)whereqjq\_\{j\}crosses1/n1/n\. ForDq​\(B\)D\_\{q\}\(B\), reverse water filling equalizesqj​2−2​bjq\_\{j\}2^\{\-2b\_\{j\}\}on active modes\. Ifmmmodes are active, Stirling’s formula givesB≍mB\\asymp m, while active distortion and the inactive tail are both≍m−χ\\asymp m^\{\-\\chi\}\.

#### What the law identifies\.

The theorem identifies the data and retained\-bit dependence for the stated positive\-entropy autoregressive source\. Its distinctive role is to couple held\-out log\-loss with arbitrary shared decoding; the next structural results show what additional information determines the resulting exponent\.

## Appendix FWhy Marginal Summaries Do Not Determine Scaling

This section asks which information about mode difficulty identifies a scaling law\. Here cost is charged directly to selecting or representing a coordinate\. The main\-text construction instead realizes difficulty through the number of independent coordinates within an energy group\. Two problems with identical energy and cost marginals first show what separate summaries lose\. The joint spectrum then recovers exactly the complete budget–distortion curve and gives the smallest exact summary for separable allocation\.

### F\.1Same marginals, different exponents

Consider first a binary selection rule\. Selecting modejjpayscjc\_\{j\}and removes energyqjq\_\{j\}; unselected energy remains as error\. Under total representation\-cost budgetCC, its optimum is

𝒜⁡\(C\)≔infS⊆ℕ∑j∈Scj≤C∑j∉Sqj\.\\mathcal\{A\}\(C\)\\coloneqq\\inf\_\{\\begin\{subarray\}\{c\}S\\subseteq\\mathbb\{N\}\\\\ \\sum\_\{j\\in S\}c\_\{j\}\\leq C\\end\{subarray\}\}\\sum\_\{j\\notin S\}q\_\{j\}\.\(F\.1\)
###### Theorem F\.1\(Marginals do not identify scaling; informal version of[TheoremI\.1](https://arxiv.org/html/2609.13500#A9.Thmtheorem1)\)\.

For everyχ,γ\>0\\chi,\\gamma\>0, there are two instances of[EquationF\.1](https://arxiv.org/html/2609.13500#A6.E1)with identical energy multisets and identical cost multisets at every dyadic level, but

𝒜easy\(C\)≍C−χ,𝒜hard\(C\)≍C−χ/\(1\+γ\)\.\\mathcal\{A\}\_\{\\rm easy\}\(C\)\\asymp C^\{\-\\chi\},\\qquad\\mathcal\{A\}\_\{\\rm hard\}\(C\)\\asymp C^\{\-\\chi/\(1\+\\gamma\)\}\.\(F\.2\)The same two exponents hold when selection is replaced by continuous bits with distortionqj​2−2​bjq\_\{j\}2^\{\-2b\_\{j\}\}and costcj​bjc\_\{j\}b\_\{j\}\. Hence no summary that retains only the two marginal multisets identifies the optimal exponent in either representation class\.

The construction makes the missing information visible\. At levelℓ\\ell, create2ℓ2^\{\\ell\}high\-energy modes of size2−\(1\+χ\)​ℓ2^\{\-\(1\+\\chi\)\\ell\}and2ℓ2^\{\\ell\}lower\-energy modes of size2−\(1\+χ\+δ\)​ℓ2^\{\-\(1\+\\chi\+\\delta\)\\ell\}, withδ\>γ​χ\\delta\>\\gamma\\chi\. The costs are2ℓ2^\{\\ell\}copies of11and2ℓ2^\{\\ell\}copies of2γ​ℓ2^\{\\gamma\\ell\}\. The easy instance pairs high energy with unit cost; the hard instance pairs it with the larger cost\. All marginals agree level by level\. A component with multiplicity2ℓ2^\{\\ell\}, energy2−\(1\+s\)​ℓ2^\{\-\(1\+s\)\\ell\}, and cost2g​ℓ2^\{g\\ell\}has exponents/\(1\+g\)s/\(1\+g\)\. This yields[EquationF\.2](https://arxiv.org/html/2609.13500#A6.E2); the full lower bound also permits non\-prefix selections through a fractional relaxation\.

The causal\-source coupling experiment is shown in[Figure3](https://arxiv.org/html/2609.13500#S7.F3)\(b,e\), together with analytic distortion;[Table2\(b\)](https://arxiv.org/html/2609.13500#S7.T2.st2)compares theoretical and held\-out slopes\.

### F\.2The joint spectrum

The full unlabeled joint measureη\\etain[EquationD\.2](https://arxiv.org/html/2609.13500#A4.E2)retains the pairing by construction\. For the separable exponential response[EquationD\.1](https://arxiv.org/html/2609.13500#A4.E1), it can be compressed further\.

###### Theorem F\.2\(Joint energy–difficulty characterization; informal version of[TheoremI\.2](https://arxiv.org/html/2609.13500#A9.Thmtheorem2)\)\.

For a finite collection of modes,η\\etais a maximal invariant under permutations of mode labels: it retains all information except the arbitrary names of the modes\. For a finite or countable collection, consider the separable problem[EquationD\.1](https://arxiv.org/html/2609.13500#A4.E1)and assume

∑jaj<∞,∑j:aj/cj\>tcj<∞for everyt\>0\.\\sum\_\{j\}a\_\{j\}<\\infty,\\qquad\\sum\_\{j:a\_\{j\}/c\_\{j\}\>t\}c\_\{j\}<\\infty\\quad\\text\{for every \}t\>0\.Then the spectrumμ=∑jcj​δaj/cj\\mu=\\sum\_\{j\}c\_\{j\}\\delta\_\{a\_\{j\}/c\_\{j\}\}determinesD⁡\(B\)D\(B\)for everyB≥0B\\geq 0\. Conversely, the complete curveB↦D⁡\(B\)B\\mapsto D\(B\)determinesμ\\muon\(0,∞\)\(0,\\infty\)\. Consequently,μ\\muis a minimal exact unlabeled summary of the complete budget–distortion curve for this representation class\.

Putrj=aj/cjr\_\{j\}=a\_\{j\}/c\_\{j\}\. The coordinatewise optimum has a water levelτ\>0\\tau\>0:

bj​\(τ\)=12​log2​\(rj/τ\)\+\.b\_\{j\}\(\\tau\)=\\frac\{1\}\{2\}\\log\_\{2\}\(r\_\{j\}/\\tau\)\_\{\+\}\.\(F\.3\)Writing the result in terms ofμ\\mugives two integral identities,

B⁡\(τ\)=12​log⁡2​∫log⁡\(r/τ\)\+​μ​\(𝑑r\),D⁡\(B⁡\(τ\)\)=∫min⁡\{r,τ\}​μ​\(𝑑r\)\.B\(\\tau\)=\\frac\{1\}\{2\\log 2\}\\int\\log\(r/\\tau\)\_\{\+\}\\,\\mu\(\\,\\mathrm\{d\}r\),\\qquad D\(B\(\\tau\)\)=\\int\\min\\\{r,\\tau\\\}\\,\\mu\(\\,\\mathrm\{d\}r\)\.\(F\.4\)These identities prove sufficiency\. They also make the inverse result possible\. At differentiability points the envelope theorem gives

D′​\(B\)=−2​log⁡2​τ​\(B\)\.D^\{\\prime\}\(B\)=\-2\\log 2\\,\\tau\(B\)\.\(F\.5\)Thus the curve recovers the water level, henceB⁡\(τ\)B\(\\tau\)\. Withs=log⁡τs=\\log\\tau,

dd​s​B​\(es\)=−12​log⁡2​μ​\(\(es,∞\)\)\\frac\{\\,\\mathrm\{d\}\}\{\\,\\mathrm\{d\}s\}B\(e^\{s\}\)=\-\\frac\{1\}\{2\\log 2\}\\mu\(\(e^\{s\},\\infty\)\)\(F\.6\)at every continuity point\. The right\-continuous tail, including its jumps, determinesμ\\mu\.

Minimality now has a precise meaning\. If another summaryTTdetermines the complete curve, thenT⁡\(x\)=T⁡\(y\)T\(x\)=T\(y\)impliesDx=DyD\_\{x\}=D\_\{y\}, which by[EquationF\.6](https://arxiv.org/html/2609.13500#A6.E6)impliesμx=μy\\mu\_\{x\}=\\mu\_\{y\}\. Thereforeμ\\muis a function of every sufficient summary\. It is also strictly coarser than the full unlabeled tableη\\eta: it retains only the cost\-weighted distribution of energy per unit difficulty\.

###### Corollary F\.3\(Joint\-tail exponent\)\.

Ifaj≍j−αa\_\{j\}\\asymp j^\{\-\\alpha\}withα\>1\\alpha\>1andcj≍jγc\_\{j\}\\asymp j^\{\\gamma\}withγ\>−1\\gamma\>\-1, then

D\(B\)≍B−\(α−1\)/\(1\+γ\)\.D\(B\)\\asymp B^\{\-\(\\alpha\-1\)/\(1\+\\gamma\)\}\.

Indeed, the active thresholdmmobeysB≍m1\+γB\\asymp m^\{1\+\\gamma\}, while the represented distortion and omitted tail are both≍m1−α\\asymp m^\{1\-\\alpha\}\.

#### When the ingredients can be measured\.

Controlled interventions provide a route to estimating the ingredients: counts estimate mode frequency, held\-out ablations estimate predictive energy, and deliberate rounding perturbations estimate numerical amplification\. The identification theorem specifies how these quantities determine the resource curve; extracting them from passive natural\-language observations is an open estimation problem\.

## Appendix GProofs for the Finite\-Bit Autoregressive Minimax Law

This appendix proves the arbitrary\-codebook entropy lemma, the statistical upper and lower bounds, and the power\-law corollary in that order\.

###### Lemma G\.1\(Arbitrary joint finite\-state representations; formal version of[LemmaE\.1](https://arxiv.org/html/2609.13500#A5.Thmtheorem1)\)\.

LetΘj\\Theta\_\{j\}be independent and uniform on\[−Δj,Δj\]\[\-\\Delta\_\{j\},\\Delta\_\{j\}\], letpj\>0p\_\{j\}\>0, and putqj=pj​Δj2q\_\{j\}=p\_\{j\}\\Delta\_\{j\}^\{2\}\. ForB∈ℕ0B\\in\\mathbb\{N\}\_\{0\}, every encoder–decoder pair with at most2B2^\{B\}reconstruction vectors satisfies

∑jpj​𝔼​\(Θj−Θ^j\)2≥2π​e​infbj≥0∑jbj≤B∑jqj​2−2​bj\.\\sum\_\{j\}p\_\{j\}\\mathbb\{E\}\(\\Theta\_\{j\}\-\\widehat\{\\Theta\}\_\{j\}\)^\{2\}\\geq\\frac\{2\}\{\\pi e\}\\inf\_\{\\begin\{subarray\}\{c\}b\_\{j\}\\geq 0\\\\ \\sum\_\{j\}b\_\{j\}\\leq B\\end\{subarray\}\}\\sum\_\{j\}q\_\{j\}2^\{\-2b\_\{j\}\}\.\(G\.1\)The reconstruction vectors may couple all coordinates\.

###### Proof\.

First restrict to a finite coordinate setAA\. Write

dj≔pj​𝔼​\(Θj−Θ^j\)2\.d\_\{j\}\\coloneqq p\_\{j\}\\mathbb\{E\}\(\\Theta\_\{j\}\-\\widehat\{\\Theta\}\_\{j\}\)^\{2\}\.For anyS⊆AS\\subseteq A, data processing and the codebook cardinality imply

B​log⁡2≥I⁡\(ΘS,Θ^\)\.B\\log 2\\geq I\(\\Theta\_\{S\};\\widehat\{\\Theta\}\)\.\(G\.2\)Independence givesh⁡\(ΘS\)=∑j∈Slog⁡\(2​Δj\)h\(\\Theta\_\{S\}\)=\\sum\_\{j\\in S\}\\log\(2\\Delta\_\{j\}\)\. Moreover,

h⁡\(ΘS∣Θ^\)\\displaystyle h\(\\Theta\_\{S\}\\mid\\widehat\{\\Theta\}\)=h⁡\(ΘS−Θ^S∣Θ^\)\\displaystyle=h\(\\Theta\_\{S\}\-\\widehat\{\\Theta\}\_\{S\}\\mid\\widehat\{\\Theta\}\)≤h⁡\(ΘS−Θ^S\)\\displaystyle\\leq h\(\\Theta\_\{S\}\-\\widehat\{\\Theta\}\_\{S\}\)≤∑j∈Sh⁡\(Θj−Θ^j\)\\displaystyle\\leq\\sum\_\{j\\in S\}h\(\\Theta\_\{j\}\-\\widehat\{\\Theta\}\_\{j\}\)≤12​∑j∈Slog⁡\(2​π​e​djpj\),\\displaystyle\\leq\\frac\{1\}\{2\}\\sum\_\{j\\in S\}\\log\\\!\\left\(2\\pi e\\,\\frac\{d\_\{j\}\}\{p\_\{j\}\}\\right\),where the last line is the Gaussian maximum\-entropy inequality at fixed second moment\. To see it directly, compare the error density to the centered Gaussian of variancedj/pjd\_\{j\}/p\_\{j\}: nonnegativity of relative entropy givesh⁡\(X\)≤12​log⁡\(2​π​dj/pj\)\+𝔼​X2/\(2​dj/pj\)h\(X\)\\leq\\frac\{1\}\{2\}\\log\(2\\pi d\_\{j\}/p\_\{j\}\)\+\\mathbb\{E\}X^\{2\}/\(2d\_\{j\}/p\_\{j\}\)\. If an entropy is−∞\-\\infty, the same bound follows by approximation\. Zero mean\-square error on a nondegenerate uniform coordinate would require infinite information, so it cannot occur for a finite codebook\. All information and entropy expressions here use natural logarithms\. Therefore

I⁡\(ΘS,Θ^\)≥12​∑j∈Slog⁡2​qjπ​e​dj\.I\(\\Theta\_\{S\};\\widehat\{\\Theta\}\)\\geq\\frac\{1\}\{2\}\\sum\_\{j\\in S\}\\log\\frac\{2q\_\{j\}\}\{\\pi e\\,d\_\{j\}\}\.\(G\.3\)
ChooseS=\{j:dj<2​qj/\(π​e\)\}S=\\\{j:d\_\{j\}<2q\_\{j\}/\(\\pi e\)\\\}and define

bj=12​log⁡2​\[log⁡2​qjπ​e​dj\]\+\.b\_\{j\}=\\frac\{1\}\{2\\log 2\}\\left\[\\log\\frac\{2q\_\{j\}\}\{\\pi e\\,d\_\{j\}\}\\right\]\_\{\+\}\.Equations \([G\.2](https://arxiv.org/html/2609.13500#A7.E2)\)–\([G\.3](https://arxiv.org/html/2609.13500#A7.E3)\) give∑jbj≤B\\sum\_\{j\}b\_\{j\}\\leq B\. OnSS,

dj=2π​e​qj​2−2​bj,d\_\{j\}=\\frac\{2\}\{\\pi e\}q\_\{j\}2^\{\-2b\_\{j\}\},while offSS,bj=0b\_\{j\}=0anddj≥\(2/\(π​e\)\)​qjd\_\{j\}\\geq\(2/\(\\pi e\)\)q\_\{j\}\. Hence

∑j∈Adj≥2π​e​infbj≥0∑j∈Abj≤B∑j∈Aqj​2−2​bj\.\\sum\_\{j\\in A\}d\_\{j\}\\geq\\frac\{2\}\{\\pi e\}\\inf\_\{\\begin\{subarray\}\{c\}b\_\{j\}\\geq 0\\\\ \\sum\_\{j\\in A\}b\_\{j\}\\leq B\\end\{subarray\}\}\\sum\_\{j\\in A\}q\_\{j\}2^\{\-2b\_\{j\}\}\.
For countably many coordinates, letAm↑ℕA\_\{m\}\\uparrow\\mathbb\{N\}and call the finite variational valueDAm​\(B\)D\_\{A\_\{m\}\}\(B\)\. ClearlyDAm​\(B\)≤Dq​\(B\)D\_\{A\_\{m\}\}\(B\)\\leq D\_\{q\}\(B\)\. Extending an optimizer onAmA\_\{m\}by zeros gives

Dq​\(B\)≤DAm​\(B\)\+∑j∉Amqj\.D\_\{q\}\(B\)\\leq D\_\{A\_\{m\}\}\(B\)\+\\sum\_\{j\\notin A\_\{m\}\}q\_\{j\}\.Since∑jqj<∞\\sum\_\{j\}q\_\{j\}<\\infty,DAm​\(B\)↑Dq​\(B\)D\_\{A\_\{m\}\}\(B\)\\uparrow D\_\{q\}\(B\)\. Monotone convergence of the distortion completes the proof\. Randomized encoders do not improve the Bayes risk: condition on the encoder seed and average the deterministic bound\. ∎

###### Theorem G\.2\(Finite\-bit autoregressive minimax law; formal version of[TheoremE\.2](https://arxiv.org/html/2609.13500#A5.Thmtheorem2)\)\.

FixR<∞R<\\infty\. For everyn∈ℕn\\in\\mathbb\{N\},B∈ℕ0B\\in\\mathbb\{N\}\_\{0\}, and source family specified by\(pj,Δj\)j≥1\(p\_\{j\},\\Delta\_\{j\}\)\_\{j\\geq 1\}as in[SectionD\.1](https://arxiv.org/html/2609.13500#A4.SS1), there are constants0<cR≤CR<∞0<c\_\{R\}\\leq C\_\{R\}<\\inftysuch that

cR​\{Sq​\(n\)\+Dq​\(B\)\}≤𝔈∗​\(n,B\)≤CR​\{Sq​\(n\)\+Dq​\(B\)\},c\_\{R\}\\\{S\_\{q\}\(n\)\+D\_\{q\}\(B\)\\\}\\leq\\mathfrak\{E\}^\{\*\}\(n,B\)\\leq C\_\{R\}\\\{S\_\{q\}\(n\)\+D\_\{q\}\(B\)\\\},\(G\.4\)where

Sq​\(n\)≔∑jmin⁡\{qj,n−1\},Dq​\(B\)≔infbj≥0∑jbj≤B∑jqj​2−2​bj\.S\_\{q\}\(n\)\\coloneqq\\sum\_\{j\}\\min\\\{q\_\{j\},n^\{\-1\}\\\},\\qquad D\_\{q\}\(B\)\\coloneqq\\inf\_\{\\begin\{subarray\}\{c\}b\_\{j\}\\geq 0\\\\ \\sum\_\{j\}b\_\{j\}\\leq B\\end\{subarray\}\}\\sum\_\{j\}q\_\{j\}2^\{\-2b\_\{j\}\}\.\(G\.5\)The lower bound holds for arbitrary2B2^\{B\}\-state joint encoders and arbitrary fixed shared decoding computation\.

###### Proof\.

We proceed in four steps\.

#### Step 1: loss curvature\.

The Bernoulli log\-partition functionA⁡\(t\)=log⁡\(1\+et\)A\(t\)=\\log\(1\+e^\{t\}\)satisfies

A′′​\(t\)=σ⁡\(t\)​\(1−σ⁡\(t\)\)\.A^\{\\prime\\prime\}\(t\)=\\sigma\(t\)\(1\-\\sigma\(t\)\)\.On\[−R,R\]\[\-R,R\], this derivative lies between two positive constants\. The excess cross\-entropy of logitθ^\\widehat\{\\theta\}is the Bregman divergence ofAA, or equivalently the Kullback–Leibler \(KL\) divergence, which we denote byKL\\KL\. Taylor’s theorem therefore yields

cR\(θ−θ^\)2≤KL\(Ber\(σ\(θ\)\)∥Ber\(σ\(θ^\)\)\)≤CR\(θ−θ^\)2\.c\_\{R\}\(\\theta\-\\widehat\{\\theta\}\)^\{2\}\\leq\\KL\(\\Ber\(\\sigma\(\\theta\)\)\\,\\\|\\,\\Ber\(\\sigma\(\\widehat\{\\theta\}\)\)\)\\leq C\_\{R\}\(\\theta\-\\widehat\{\\theta\}\)^\{2\}\.\(G\.6\)Clipping a decoded logit to\[−R,R\]\[\-R,R\]also cannot increase cross\-entropy: the derivative with respect to the decoded logitzzisσ⁡\(z\)−σ⁡\(θ\)\\sigma\(z\)\-\\sigma\(\\theta\), which has the sign ofz−θz\-\\theta\. Clipping preserves the decoder’s cardinality\. Thus both minimax bounds can be proved using bounded decoded logits and weighted squared error\.

#### Step 2: statistical upper bound\.

LetNj∼Binomial⁡\(n,pj\)N\_\{j\}\\sim\\operatorname\{Binomial\}\(n,p\_\{j\}\)be the training count of modejj\. IfNj≥1N\_\{j\}\\geq 1, project the empirical Bernoulli mean onto\[σ⁡\(−Δj\),σ⁡\(Δj\)\]\[\\sigma\(\-\\Delta\_\{j\}\),\\sigma\(\\Delta\_\{j\}\)\]and apply the logit map\. Projection cannot increase probability error, and the logit is Lipschitz on\[σ⁡\(−R\),σ⁡\(R\)\]\[\\sigma\(\-R\),\\sigma\(R\)\]\. Thus its conditional mean\-square error is at mostCR/NjC\_\{R\}/N\_\{j\}\. The zero estimator has error at mostΔj2\\Delta\_\{j\}^\{2\}\. Choose between them using the observed count, yielding

𝔼⁡\[\(θ^j−θj\)2∣Nj\]≤CR′​min⁡\{Δj2,\(Nj\+1\)−1\}\.\\mathbb\{E\}\[\(\\widehat\{\\theta\}\_\{j\}\-\\theta\_\{j\}\)^\{2\}\\mid N\_\{j\}\]\\leq C\_\{R\}^\{\\prime\}\\min\\\{\\Delta\_\{j\}^\{2\},\(N\_\{j\}\+1\)^\{\-1\}\\\}\.\(G\.7\)The identity

𝔼​1Nj\+1=1−\(1−pj\)n\+1\(n\+1\)​pj≤1\(n\+1\)​pj\\mathbb\{E\}\\frac\{1\}\{N\_\{j\}\+1\}=\\frac\{1\-\(1\-p\_\{j\}\)^\{n\+1\}\}\{\(n\+1\)p\_\{j\}\}\\leq\\frac\{1\}\{\(n\+1\)p\_\{j\}\}\(G\.8\)follows by integrating the binomial generating function\. Therefore

pj​𝔼​\(θ^j−θj\)2\\displaystyle p\_\{j\}\\mathbb\{E\}\(\\widehat\{\\theta\}\_\{j\}\-\\theta\_\{j\}\)^\{2\}≤CR′​min⁡\{pj​Δj2,pj​𝔼​1Nj\+1\}\\displaystyle\\leq C\_\{R\}^\{\\prime\}\\min\\left\\\{p\_\{j\}\\Delta\_\{j\}^\{2\},\\,p\_\{j\}\\mathbb\{E\}\\frac\{1\}\{N\_\{j\}\+1\}\\right\\\}≤CR′′​min⁡\{qj,n−1\}\.\\displaystyle\\leq C\_\{R\}^\{\\prime\\prime\}\\min\\\{q\_\{j\},n^\{\-1\}\\\}\.Summing proves theSq​\(n\)S\_\{q\}\(n\)upper bound\.

Now select an allocation within an arbitrarily small additive error ofDq​\(B\)D\_\{q\}\(B\), round each positivebjb\_\{j\}down to an integer, and uniformly quantize the clipped estimate on\[−Δj,Δj\]\[\-\\Delta\_\{j\},\\Delta\_\{j\}\]\. Flooring uses at mostBBbits and enlarges2−2​bj2^\{\-2b\_\{j\}\}by at most four\. By the squared triangle inequality and[EquationG\.6](https://arxiv.org/html/2609.13500#A7.E6),

𝔈∗​\(n,B\)≤CR​\{Sq​\(n\)\+Dq​\(B\)\}\.\\mathfrak\{E\}^\{\*\}\(n,B\)\\leq C\_\{R\}\\\{S\_\{q\}\(n\)\+D\_\{q\}\(B\)\\\}\.\(G\.9\)

#### Step 3: statistical lower bound\.

Fix a finite coordinate setAA\. On coordinatej∈Aj\\in A, use the two parameters±δj\\pm\\delta\_\{j\}, where

δj=cR′min\{Δj,\(npj\)−1/2\}\\delta\_\{j\}=c\_\{R\}^\{\\prime\}\\min\\\{\\Delta\_\{j\},\(np\_\{j\}\)^\{\-1/2\}\\\}andcR′\>0c\_\{R\}^\{\\prime\}\>0is sufficiently small\. Neighboring hypercube vertices differ only whenJ=jJ=j, and bounded Bernoulli curvature gives

KL\(Pθn∥Pθ\(j\)n\)≤CRnpjδj2≤CR\(cR′\)2\.\\KL\(P\_\{\\theta\}^\{\\,n\}\\,\\\|\\,P\_\{\\theta^\{\(j\)\}\}^\{\\,n\}\)\\leq C\_\{R\}np\_\{j\}\\delta\_\{j\}^\{2\}\\leq C\_\{R\}\(c\_\{R\}^\{\\prime\}\)^\{2\}\.Here the design distribution is independent of the hypercube signs\. The one\-block KL is therefore exactlypjp\_\{j\}times the changed coordinate’s Bernoulli KL, and independence of blocks multiplies it bynn\. ChoosecR′c\_\{R\}^\{\\prime\}so that this product KL is at most1/81/8\. For completeness, fix all signs exceptjjand test the remaining sign by the sign ofθ^j\\widehat\{\\theta\}\_\{j\}\. A wrong sign incurs squared error at leastδj2\\delta\_\{j\}^\{2\}\. The sum of the two testing errors is at least1−TV⁡\(P\+,P−\)≥3/41\-\\operatorname\{TV\}\(P\_\{\+\},P\_\{\-\}\)\\geq 3/4, sinceTV⁡\(P\+,P−\)≤KL\(P\+∥P−\)/2\\operatorname\{TV\}\(P\_\{\+\},P\_\{\-\}\)\\leq\\sqrt\{\\KL\(P\_\{\+\}\\\|P\_\{\-\}\)/2\}\. This inequality follows by partitioning at the positive part of the density difference, applying the log\-sum inequality, and usingkl⁡\(a,b\)≥2​\(a−b\)2\\mathrm\{kl\}\(a,b\)\\geq 2\(a\-b\)^\{2\}; the latter follows from its second derivative inaa,1/\[a⁡\(1−a\)\]≥41/\[a\(1\-a\)\]\\geq 4\. Averaging over the uniform signs and summing the coordinate losses yields

infθ^supθ𝔼​∑j∈Apj​\(θ^j−θj\)2≥cR​∑j∈Amin⁡\{qj,n−1\}\.\\inf\_\{\\widehat\{\\theta\}\}\\sup\_\{\\theta\}\\mathbb\{E\}\\sum\_\{j\\in A\}p\_\{j\}\(\\widehat\{\\theta\}\_\{j\}\-\\theta\_\{j\}\)^\{2\}\\geq c\_\{R\}\\sum\_\{j\\in A\}\\min\\\{q\_\{j\},n^\{\-1\}\\\}\.\(G\.10\)The resulting constant is independent of the size ofAA\. Since the full risk dominates each finite\-coordinate risk, taking the supremum over increasing finite sets gives theSq​\(n\)S\_\{q\}\(n\)lower bound; no interchange of an infinite supremum and a prior integral is needed\.

#### Step 4: representation lower bound\.

Give the encoderθ\\thetain addition to the data\. This enlarges the encoder class and can only reduce its minimax risk\. Under the product\-uniform prior on the parameter box,[LemmaE\.1](https://arxiv.org/html/2609.13500#A5.Thmtheorem1)gives

inf\|range⁡\(θ^\)\|≤2Bsupθ𝔼​∑jpj​\(θ^j−θj\)2≥c​Dq​\(B\)\.\\inf\_\{\\lvert\\operatorname\{range\}\(\\widehat\{\\theta\}\)\\rvert\\leq 2^\{B\}\}\\sup\_\{\\theta\}\\mathbb\{E\}\\sum\_\{j\}p\_\{j\}\(\\widehat\{\\theta\}\_\{j\}\-\\theta\_\{j\}\)^\{2\}\\geq cD\_\{q\}\(B\)\.\(G\.11\)The original constrained risk is at least both[EquationsG\.10](https://arxiv.org/html/2609.13500#A7.E10)and[G\.11](https://arxiv.org/html/2609.13500#A7.E11)\. Sincemax⁡\{x,y\}≥\(x\+y\)/2\\max\\\{x,y\\\}\\geq\(x\+y\)/2,[EquationG\.6](https://arxiv.org/html/2609.13500#A7.E6)proves the matching lower bound in[EquationE\.2](https://arxiv.org/html/2609.13500#A5.E2)\. ∎

###### Corollary G\.3\(Power\-law energies; formal version of[CorollaryE\.3](https://arxiv.org/html/2609.13500#A5.Thmtheorem3)\)\.

Suppose thatn,B≥1n,B\\geq 1and, for someχ\>0\\chi\>0and constants0<c−≤c\+<∞0<c\_\{\-\}\\leq c\_\{\+\}<\\infty,

c−​j−\(1\+χ\)≤qj≤c\+​j−\(1\+χ\)\(j≥1\)\.c\_\{\-\}j^\{\-\(1\+\\chi\)\}\\leq q\_\{j\}\\leq c\_\{\+\}j^\{\-\(1\+\\chi\)\}\\qquad\(j\\geq 1\)\.Then

𝔈∗\(n,B\)≍n−χ/\(1\+χ\)\+B−χ\.\\mathfrak\{E\}^\{\*\}\(n,B\)\\asymp n^\{\-\\chi/\(1\+\\chi\)\}\+B^\{\-\\chi\}\.\(G\.12\)The data–bit crossover occurs atB≍n1/\(1\+χ\)B\\asymp n^\{1/\(1\+\\chi\)\}\.

###### Proof\.

The comparison assumption does not require the actual sequenceqjq\_\{j\}to be monotone\. Putq¯j=j−\(1\+χ\)\\bar\{q\}\_\{j\}=j^\{\-\(1\+\\chi\)\}, so thatc−​q¯j≤qj≤c\+​q¯jc\_\{\-\}\\bar\{q\}\_\{j\}\\leq q\_\{j\}\\leq c\_\{\+\}\\bar\{q\}\_\{j\}\. Taking infima of the coordinatewise distortion inequalities gives

c−​Dq¯​\(B\)≤Dq​\(B\)≤c\+​Dq¯​\(B\)\.c\_\{\-\}D\_\{\\bar\{q\}\}\(B\)\\leq D\_\{q\}\(B\)\\leq c\_\{\+\}D\_\{\\bar\{q\}\}\(B\)\.Similarly,min⁡\{c​q,t\}\\min\\\{cq,t\\\}lies betweenmin⁡\{c,1\}​min​\{q,t\}\\min\\\{c,1\\\}\\min\\\{q,t\\\}andmax⁡\{c,1\}​min​\{q,t\}\\max\\\{c,1\\\}\\min\\\{q,t\\\}\. It suffices to prove both rates forq¯\\bar\{q\}\. Letmn=⌊n1/\(1\+χ\)⌋m\_\{n\}=\\lfloor n^\{1/\(1\+\\chi\)\}\\rfloor\. Then

Sq¯\(n\)≍mnn\+∑j\>mnj−\(1\+χ\)≍n−χ/\(1\+χ\)\.S\_\{\\bar\{q\}\}\(n\)\\asymp\\frac\{m\_\{n\}\}\{n\}\+\\sum\_\{j\>m\_\{n\}\}j^\{\-\(1\+\\chi\)\}\\asymp n^\{\-\\chi/\(1\+\\chi\)\}\.
ForDq¯​\(B\)D\_\{\\bar\{q\}\}\(B\), the water\-filling solution gives

bj=12​\[log2⁡\(q¯j/τ\)\]\+\.b\_\{j\}=\\frac\{1\}\{2\}\[\\log\_\{2\}\(\\bar\{q\}\_\{j\}/\\tau\)\]\_\{\+\}\.The comparison profile is monotone, so its active set is a prefix\[m\]\[m\]\. Form≥1m\\geq 1,\(m\+1\)−\(1\+χ\)≤τ≤m−\(1\+χ\)\(m\+1\)^\{\-\(1\+\\chi\)\}\\leq\\tau\\leq m^\{\-\(1\+\\chi\)\}\. Writingv=τ−1/\(1\+χ\)∈\[m,m\+1\]v=\\tau^\{\-1/\(1\+\\chi\)\}\\in\[m,m\+1\],

B=1\+χ2​ln⁡2​\{m​ln⁡v−ln⁡\(m\!\)\}=Θ⁡\(m\)B=\\frac\{1\+\\chi\}\{2\\ln 2\}\\\{m\\ln v\-\\ln\(m\!\)\\\}=\\Theta\(m\)asm→∞m\\to\\infty, by Stirling’s formula and0≤m​ln⁡\(v/m\)≤10\\leq m\\ln\(v/m\)\\leq 1\. Active distortion ism​τ≍m−χm\\tau\\asymp m^\{\-\\chi\}, and the inactive tail has the same order\. ThusDq​\(B\)≍B−χD\_\{q\}\(B\)\\asymp B^\{\-\\chi\}\. The two terms cross whenB−χ≍n−χ/\(1\+χ\)B^\{\-\\chi\}\\asymp n^\{\-\\chi/\(1\+\\chi\)\}, namelyB≍n1/\(1\+χ\)B\\asymp n^\{1/\(1\+\\chi\)\}\. ∎

## Appendix HProof of Adaptation with a Known Ordering

This appendix verifies that one rounded schedule works uniformly over the specified exponent interval\.

###### Theorem H\.1\(Adaptation with a known ordering; formal version of[Theorem5\.1](https://arxiv.org/html/2609.13500#S5.Thmtheorem1)\)\.

Fix0<χmin≤χmax<∞0<\\chi\_\{\\min\}\\leq\\chi\_\{\\max\}<\\infty, and suppose the canonical mode order is known\. Assumeqj≍j−\(1\+χ\)q\_\{j\}\\asymp j^\{\-\(1\+\\chi\)\}with common comparison constants and unknownχ∈\[χmin,χmax\]\\chi\\in\[\\chi\_\{\\min\},\\chi\_\{\\max\}\]\. For everyn,B∈ℕn,B\\in\\mathbb\{N\}, one clipped likelihood estimator and one deterministic integer bit schedule, both independent ofχ\\chi, satisfy simultaneously throughout this interval

supθj∈\[−Δj,Δj\]𝔼θ\[ℒθ\(f^\)−ℒθ\(θ\)\]≲n−χ/\(1\+χ\)\+B−χ\.\\sup\_\{\\theta\_\{j\}\\in\[\-\\Delta\_\{j\},\\Delta\_\{j\}\]\}\\mathbb\{E\}\_\{\\theta\}\\\!\\left\[\\mathcal\{L\}\_\{\\theta\}\(\\widehat\{f\}\)\-\\mathcal\{L\}\_\{\\theta\}\(\\theta\)\\right\]\\lesssim n^\{\-\\chi/\(1\+\\chi\)\}\+B^\{\-\\chi\}\.\(H\.1\)

###### Proof\.

The estimator in the statistical upper bound of[TheoremE\.2](https://arxiv.org/html/2609.13500#A5.Thmtheorem2)uses onlyNjN\_\{j\},Δj\\Delta\_\{j\}, and the observed labels, notχ\\chi\. It contributesSq\(n\)≍n−χ/\(1\+χ\)S\_\{q\}\(n\)\\asymp n^\{\-\\chi/\(1\+\\chi\)\}\.

ChooseA\>χmax/2A\>\\chi\_\{\\max\}/2\. Before rounding, the schedule in[Equation5\.2](https://arxiv.org/html/2609.13500#S5.E2)uses

A​∑j≤mlog2⁡\(m/j\)=A​log2⁡\(mm/m\!\)=Θ⁡\(m\)\.A\\sum\_\{j\\leq m\}\\log\_\{2\}\(m/j\)=A\\log\_\{2\}\(m^\{m\}/m\!\)=\\Theta\(m\)\.Flooring cannot increase the budget\. It also preserves a linear lower bound: everyj≤m2−2/Aj\\leq m2^\{\-2/A\}receives at least two unrounded bits and therefore at least one rounded bit\. Hence the rounded total isΘ⁡\(m\)\\Theta\(m\), and the maximal feasiblemmsatisfiesm≍Bm\\asymp B\.

Flooring enlarges each nonzero distortion by at most four\. Consequently, uniformly overχ≤χmax\\chi\\leq\\chi\_\{\\max\},

∑j≤mqj​2−2​bj\\displaystyle\\sum\_\{j\\leq m\}q\_\{j\}2^\{\-2b\_\{j\}\}≲m−2​A​∑j≤mj2​A−1−χ\\displaystyle\\lesssim m^\{\-2A\}\\sum\_\{j\\leq m\}j^\{2A\-1\-\\chi\}≲m−χ,\\displaystyle\\lesssim m^\{\-\\chi\},because2​A−χ≥2​A−χmax\>02A\-\\chi\\geq 2A\-\\chi\_\{\\max\}\>0: integral comparison bounds the power sum byC​m2​A−χ/\(2​A−χ\)Cm^\{2A\-\\chi\}/\(2A\-\\chi\), with one constant throughout the interval\. Likewise the tail constants are uniform becauseχ≥χmin\>0\\chi\\geq\\chi\_\{\\min\}\>0\. For the finitely many budgets before the linear bound on the rounded schedule applies, enlarge the same constant;B∈ℕB\\in\\mathbb\{N\}is positive\. The omitted tail is∑j\>mqj≍m−χ\\sum\_\{j\>m\}q\_\{j\}\\asymp m^\{\-\\chi\}\. Sincem≍Bm\\asymp B, the representation contribution isO⁡\(B−χ\)O\(B^\{\-\\chi\}\), proving[Equation5\.1](https://arxiv.org/html/2609.13500#S5.E1)\. ∎

## Appendix IProofs for Joint Difficulty

This appendix proves the same\-marginal separation and then the positive joint\-summary theorem\.

###### Theorem I\.1\(Marginals do not identify scaling; formal version of[TheoremF\.1](https://arxiv.org/html/2609.13500#A6.Thmtheorem1)\)\.

For everyχ,γ\>0\\chi,\\gamma\>0, there are two instances of[EquationF\.1](https://arxiv.org/html/2609.13500#A6.E1)with identical energy multisets and identical cost multisets at every dyadic level, but

𝒜easy\(C\)≍C−χ,𝒜hard\(C\)≍C−χ/\(1\+γ\)\.\\mathcal\{A\}\_\{\\rm easy\}\(C\)\\asymp C^\{\-\\chi\},\\qquad\\mathcal\{A\}\_\{\\rm hard\}\(C\)\\asymp C^\{\-\\chi/\(1\+\\gamma\)\}\.\(I\.1\)The same two exponents hold when selection is replaced by continuous bits with distortionqj​2−2​bjq\_\{j\}2^\{\-2b\_\{j\}\}and costcj​bjc\_\{j\}b\_\{j\}\. Hence no summary that retains only the two marginal multisets identifies the optimal exponent in either representation class\.

###### Proof\.

Fixδ\>γ​χ\\delta\>\\gamma\\chi\. At each levelℓ≥1\\ell\\geq 1, create two groups, each of multiplicity2ℓ2^\{\\ell\}\. Their energies are

qℓH=2−\(1\+χ\)​ℓ,qℓL=2−\(1\+χ\+δ\)​ℓ,q\_\{\\ell\}^\{\\rm H\}=2^\{\-\(1\+\\chi\)\\ell\},\\qquad q\_\{\\ell\}^\{\\rm L\}=2^\{\-\(1\+\\chi\+\\delta\)\\ell\},and the level’s cost multiset contains2ℓ2^\{\\ell\}copies of11and2ℓ2^\{\\ell\}copies of2γ​ℓ2^\{\\gamma\\ell\}\. The easy instance pairsqℓHq\_\{\\ell\}^\{\\rm H\}with unit cost; the hard instance pairs it with2γ​ℓ2^\{\\gamma\\ell\}\. The energy and cost multisets therefore agree at every level\.

Consider a generic component of multiplicity2ℓ2^\{\\ell\}, per\-item energy2−\(1\+s\)​ℓ2^\{\-\(1\+s\)\\ell\}, and per\-item cost2g​ℓ2^\{g\\ell\}\. Buying the component through levelLLcosts

∑ℓ≤L2\(1\+g\)​ℓ=Θ⁡\(2\(1\+g\)​L\),\\sum\_\{\\ell\\leq L\}2^\{\(1\+g\)\\ell\}=\\Theta\(2^\{\(1\+g\)L\}\),and leaves energy

∑ℓ\>L2−s​ℓ=Θ⁡\(2−s​L\)\.\\sum\_\{\\ell\>L\}2^\{\-s\\ell\}=\\Theta\(2^\{\-sL\}\)\.Thus its selection exponent iss/\(1\+g\)s/\(1\+g\)\. This prefix is also a matching lower bound: within the component, energy per cost decreases withℓ\\ell, so the fractional relaxation is a threshold policy, and every integral policy is no better\.

In the easy instance, the high\-energy component has exponentχ\\chi, while the low\-energy component has exponent\(χ\+δ\)/\(1\+γ\)\>χ\(\\chi\+\\delta\)/\(1\+\\gamma\)\>\\chi\. Splitting the budget by fixed fractions therefore gives, and the high\-energy relaxation lower\-bounds,𝒜easy​\(C\)≍C−χ\\mathcal\{A\}\_\{\\rm easy\}\(C\)\\asymp C^\{\-\\chi\}\. In the hard instance, the high\-energy component has exponentχ/\(1\+γ\)\\chi/\(1\+\\gamma\), while the low\-energy unit\-cost component has exponentχ\+δ\\chi\+\\delta\. Hence𝒜hard\(C\)≍C−χ/\(1\+γ\)\\mathcal\{A\}\_\{\\rm hard\}\(C\)\\asymp C^\{\-\\chi/\(1\+\\gamma\)\}\.

For continuous bits, the optimality condition on an active coordinate is

qj​2−2​bjcj=τ\.\\frac\{q\_\{j\}2^\{\-2b\_\{j\}\}\}\{c\_\{j\}\}=\\tau\.For the generic component above, the active levels form a prefix throughLL\. At levelℓ\\ell,bℓb\_\{\\ell\}is proportional toL−ℓL\-\\ell; therefore

∑ℓ≤L2\(1\+g\)​ℓ​bℓ=Θ⁡\(2\(1\+g\)​L\)\.\\sum\_\{\\ell\\leq L\}2^\{\(1\+g\)\\ell\}b\_\{\\ell\}=\\Theta\(2^\{\(1\+g\)L\}\)\.Both active distortion and the inactive tail areΘ⁡\(2−s​L\)\\Theta\(2^\{\-sL\}\)\. The same exponents/\(1\+g\)s/\(1\+g\)follows, and applying it to the four components completes the proof\. ∎

###### Theorem I\.2\(Joint energy–difficulty characterization; formal version of[TheoremF\.2](https://arxiv.org/html/2609.13500#A6.Thmtheorem2)\)\.

For a finite collection of modes,η\\etais a maximal invariant under permutations of mode labels: it retains all information except the arbitrary names of the modes\. For a finite or countable collection, consider the separable problem[EquationD\.1](https://arxiv.org/html/2609.13500#A4.E1)and assume

∑jaj<∞,∑j:aj/cj\>tcj<∞for everyt\>0\.\\sum\_\{j\}a\_\{j\}<\\infty,\\qquad\\sum\_\{j:a\_\{j\}/c\_\{j\}\>t\}c\_\{j\}<\\infty\\quad\\text\{for every \}t\>0\.Then the spectrumμ=∑jcj​δaj/cj\\mu=\\sum\_\{j\}c\_\{j\}\\delta\_\{a\_\{j\}/c\_\{j\}\}determinesD⁡\(B\)D\(B\)for everyB≥0B\\geq 0\. Conversely, the complete curveB↦D⁡\(B\)B\\mapsto D\(B\)determinesμ\\muon\(0,∞\)\(0,\\infty\)\. Consequently,μ\\muis a minimal exact unlabeled summary of the complete budget–distortion curve for this representation class\.

###### Proof\.

For a finite collection, equality ofη=∑jδ\(vj,aj,cj,κj\)\\eta=\\sum\_\{j\}\\delta\_\{\(v\_\{j\},a\_\{j\},c\_\{j\},\\kappa\_\{j\}\)\}is equality of the multisets of tuples, including multiplicities\. A bijection between equal multisets is a permutation of mode labels, and every permutation preservesη\\eta\. Thusη\\etais a maximal invariant\.

For the separable allocation, putrj=aj/cjr\_\{j\}=a\_\{j\}/c\_\{j\}\. The Lagrangian is

∑jcj​rj​2−2​bj\+λ⁡\(∑jcj​bj−B\)\.\\sum\_\{j\}c\_\{j\}r\_\{j\}2^\{\-2b\_\{j\}\}\+\\lambda\\left\(\\sum\_\{j\}c\_\{j\}b\_\{j\}\-B\\right\)\.The coordinatewise Karush–Kuhn–Tucker equations give

bj​\(τ\)=12​log2​\(rj/τ\)\+,τ=λ2​log⁡2\.b\_\{j\}\(\\tau\)=\\frac\{1\}\{2\}\\log\_\{2\}\(r\_\{j\}/\\tau\)\_\{\+\},\\qquad\\tau=\\frac\{\\lambda\}\{2\\log 2\}\.Substitution yields

B⁡\(τ\)=12​log⁡2​∑jcj​log⁡\(rj/τ\)\+,B\(\\tau\)=\\frac\{1\}\{2\\log 2\}\\sum\_\{j\}c\_\{j\}\\log\(r\_\{j\}/\\tau\)\_\{\+\},and

D⁡\(B⁡\(τ\)\)=∑jcj​min⁡\{rj,τ\}\.D\(B\(\\tau\)\)=\\sum\_\{j\}c\_\{j\}\\min\\\{r\_\{j\},\\tau\\\}\.Here is a direct countable\-dimensional justification\. SetA=∑jaj<∞A=\\sum\_\{j\}a\_\{j\}<\\infty\. For everyτ\>0\\tau\>0,

∑rj\>τcj≤A/τ,B⁡\(τ\)≤A2​\(ln⁡2\)​τ,\\sum\_\{r\_\{j\}\>\\tau\}c\_\{j\}\\leq A/\\tau,\\qquad B\(\\tau\)\\leq\\frac\{A\}\{2\(\\ln 2\)\\tau\},where the second inequality uses\[ln⁡x\]\+≤x\[\\ln x\]\_\{\+\}\\leq x\. On any compact interval of positive water levels, the summands definingB⁡\(τ\)B\(\\tau\)have a summable dominating bound proportional toaja\_\{j\}\. HenceBBis continuous\. It is strictly decreasing whenever it is positive, tends to zero at the upper end of its support \(possibly infinity\), and tends to infinity asτ↓0\\tau\\downarrow 0, since even one positive coordinate does\. Thus everyB\>0B\>0has a unique water level\. The scalar convex inequality

aj​2−2​bj\+λ​cj​bj≥aj​2−2​bj​\(τ\)\+λ​cj​bj​\(τ\)a\_\{j\}2^\{\-2b\_\{j\}\}\+\\lambda c\_\{j\}b\_\{j\}\\geq a\_\{j\}2^\{\-2b\_\{j\}\(\\tau\)\}\+\\lambda c\_\{j\}b\_\{j\}\(\\tau\)can be summed directly: all distortion and feasible cost sums are finite\. Withλ=2​\(ln⁡2\)​τ\\lambda=2\(\\ln 2\)\\tauandB⁡\(τ\)=BB\(\\tau\)=B, it proves optimality over every feasible countable allocation\. Layer integration now gives[EquationF\.4](https://arxiv.org/html/2609.13500#A6.E4), proving sufficiency ofμ\\mu\.

To recover the spectrum, let0<B1<B20<B\_\{1\}<B\_\{2\}and putλi=2​\(ln⁡2\)​τ​\(Bi\)\\lambda\_\{i\}=2\(\\ln 2\)\\tau\(B\_\{i\}\)\. Applying optimality with each of the two Lagrange multipliers to the other’s solution gives

−λ1≤D⁡\(B2\)−D⁡\(B1\)B2−B1≤−λ2\.\-\\lambda\_\{1\}\\leq\\frac\{D\(B\_\{2\}\)\-D\(B\_\{1\}\)\}\{B\_\{2\}\-B\_\{1\}\}\\leq\-\\lambda\_\{2\}\.The inverseτ⁡\(B\)\\tau\(B\)is continuous on\(0,∞\)\(0,\\infty\)\. Letting the two budgets coalesce therefore proves, in the countable system itself,

D′​\(B\)=−λ=−2​log⁡2​τ​\(B\)D^\{\\prime\}\(B\)=\-\\lambda=\-2\\log 2\\,\\tau\(B\)at everyB\>0B\>0\. Hence the complete curve determinesτ⁡\(B\)\\tau\(B\)and its inverseB⁡\(τ\)B\(\\tau\), with the zero branch above a finite support endpoint\. Fors=ln⁡τs=\\ln\\tau, difference quotients have a summable dominating bound: onlyrj\>es−hr\_\{j\}\>e^\{s\-h\}contribute, whose total cost is finite\. Thus the right derivative is

d\+d​sB\(es\)=−12​log⁡2∑j:rj\>escj=−12​log⁡2μ\(\(es,∞\)\)\\frac\{\\,\\mathrm\{d\}\_\{\+\}\}\{\\,\\mathrm\{d\}s\}B\(e^\{s\}\)=\-\\frac\{1\}\{2\\log 2\}\\sum\_\{j:r\_\{j\}\>e^\{s\}\}c\_\{j\}=\-\\frac\{1\}\{2\\log 2\}\\mu\(\(e^\{s\},\\infty\)\)and the left derivative replacesrj\>esr\_\{j\}\>e^\{s\}byrj≥esr\_\{j\}\\geq e^\{s\}\. Their difference recovers every atom’s weight; the tail recovers the measure on every positive interval\. The endpointD⁡\(0\)=AD\(0\)=Afollows by dominated convergence fromB↓0B\\downarrow 0\. This argument never differentiates a limit of finite\-dimensional value functions\.

Finally, suppose a summaryTTdetermines the complete curve\. IfT⁡\(x\)=T⁡\(y\)T\(x\)=T\(y\), thenDx​\(B\)=Dy​\(B\)D\_\{x\}\(B\)=D\_\{y\}\(B\)for everyBB, so the inversion above givesμx=μy\\mu\_\{x\}=\\mu\_\{y\}\. Thereforeμ\\muis a function ofTT, proving minimality in the stated sense\. ∎

###### Proof of[CorollaryF\.3](https://arxiv.org/html/2609.13500#A6.Thmtheorem3)\.

Puta¯j=j−α\\bar\{a\}\_\{j\}=j^\{\-\\alpha\}andc¯j=jγ\\bar\{c\}\_\{j\}=j^\{\\gamma\}, witha−​a¯j≤aj≤a\+​a¯ja\_\{\-\}\\bar\{a\}\_\{j\}\\leq a\_\{j\}\\leq a\_\{\+\}\\bar\{a\}\_\{j\}andc−​c¯j≤cj≤c\+​c¯jc\_\{\-\}\\bar\{c\}\_\{j\}\\leq c\_\{j\}\\leq c\_\{\+\}\\bar\{c\}\_\{j\}\. Feasible\-set inclusion gives

a−​Da¯,c¯​\(B/c−\)≤Da,c​\(B\)≤a\+​Da¯,c¯​\(B/c\+\)\.a\_\{\-\}D\_\{\\bar\{a\},\\bar\{c\}\}\(B/c\_\{\-\}\)\\leq D\_\{a,c\}\(B\)\\leq a\_\{\+\}D\_\{\\bar\{a\},\\bar\{c\}\}\(B/c\_\{\+\}\)\.It suffices to use the monotone ratioa¯j/c¯j=j−\(α\+γ\)\\bar\{a\}\_\{j\}/\\bar\{c\}\_\{j\}=j^\{\-\(\\alpha\+\\gamma\)\}\. Sinceα\>1\\alpha\>1andγ\>−1\\gamma\>\-1, its exponent is positive\. Ifmmmodes are active, putv=τ−1/\(α\+γ\)∈\[m,m\+1\]v=\\tau^\{\-1/\(\\alpha\+\\gamma\)\}\\in\[m,m\+1\]\. The budget is

B=α\+γ2​ln⁡2​∑j≤mjγ​ln⁡\(v/j\)≍m1\+γ\.B=\\frac\{\\alpha\+\\gamma\}\{2\\ln 2\}\\sum\_\{j\\leq m\}j^\{\\gamma\}\\ln\(v/j\)\\asymp m^\{1\+\\gamma\}\.For the lower bound restrict tom/4≤j≤m/2m/4\\leq j\\leq m/2; for the upper bound use integral comparison and∫01xγ​ln⁡\(1/x\)​𝑑x=\(1\+γ\)−2\\int\_\{0\}^\{1\}x^\{\\gamma\}\\ln\(1/x\)\\,\\mathrm\{d\}x=\(1\+\\gamma\)^\{\-2\}\. Replacingmmbyvvin the logarithm adds onlyO⁡\(mγ\)O\(m^\{\\gamma\}\)\. The inactive tail is≍m1−α\\asymp m^\{1\-\\alpha\}\. The active distortion isτ​∑j≤mcj\\tau\\sum\_\{j\\leq m\}c\_\{j\}, withτ≍m−\(α\+γ\)\\tau\\asymp m^\{\-\(\\alpha\+\\gamma\)\}, and has the same order\. Eliminatingmmgives

D\(B\)≍B−\(α−1\)/\(1\+γ\)\.D\(B\)\\asymp B^\{\-\(\\alpha\-1\)/\(1\+\\gamma\)\}\.∎

## Appendix JProof of the Finite\-Precision Realization

This appendix verifies the exact matrix map, the floating\-point perturbation, and the lower bound for the stated separable representation\.

We use the candidate/query encoding and projections of[Sections5](https://arxiv.org/html/2609.13500#S5.SS0.SSS0.Px2)and[5\.4](https://arxiv.org/html/2609.13500#S5.E4)\. The supplied logit table isϑj=wjmode\\vartheta\_\{j\}=w\_\{j\}^\{\\rm mode\}forj≤Mj\\leq M, extended by zero outside the represented prefix;θj\\theta\_\{j\}remains the source parameter\.

###### Theorem J\.1\(Finite\-precision realization; formal version of[Theorem5\.2](https://arxiv.org/html/2609.13500#S5.Thmtheorem2)\)\.

FixK,M≥1K,M\\geq 1,R<∞R<\\infty,β≥0\\beta\\geq 0, and the construction above\.

1. 1\.In exact arithmetic, if\|wjmode\|≤R\\lvert w\_\{j\}^\{\\rm mode\}\\rvert\\leq Rfor everyj∈\[M\]j\\in\[M\], then a query whose matching candidate has any modej≥1j\\geq 1satisfies \|z−ϑj\|≤2​R​\(K−1\)​e−β\.\\lvert z\-\\vartheta\_\{j\}\\rvert\\leq 2R\(K\-1\)e^\{\-\\beta\}\.\(J\.1\)
2. 2\.Setwjmode=θjw\_\{j\}^\{\\rm mode\}=\\theta\_\{j\}forj∈\[M\]j\\in\[M\]and store it withbj∈ℕ0b\_\{j\}\\in\\mathbb\{N\}\_\{0\}bits on\[−Δj,Δj\]\[\-\\Delta\_\{j\},\\Delta\_\{j\}\]\. Assume stable max\-shifted softmax with exactly representable shifted scores0,−β0,\-\\betaand an exponent range containing their exponentials\. Exponentials, multiplications, and divisions have relative error at mostuu; sums use balanced pairwise accumulation\. For the resulting predictorffpf\_\{\\rm fp\}, if\(⌈log2⁡K⌉\+4\)​u≤1/4\(\\lceil\\log\_\{2\}K\\rceil\+4\)u\\leq 1/4, then the held\-out excess cross\-entropy obeys ℒθ​\(ffp\)−ℒ∞≤CR​\{\(K−1\)2​e−2​β\+u2​\(1\+log⁡K\)2\+∑j=1Mqj​2−2​bj\+∑j\>Mqj\}\.\\mathcal\{L\}\_\{\\theta\}\(f\_\{\\rm fp\}\)\-\\mathcal\{L\}\_\{\\infty\}\\leq C\_\{R\}\\left\\\{\(K\-1\)^\{2\}e^\{\-2\\beta\}\+u^\{2\}\(1\+\\log K\)^\{2\}\+\\sum\_\{j=1\}^\{M\}q\_\{j\}2^\{\-2b\_\{j\}\}\+\\sum\_\{j\>M\}q\_\{j\}\\right\\\}\.\(J\.2\)
3. 3\.Within the separable scalar\-table representation in which coordinatejjhas at most2bj2^\{b\_\{j\}\}values, the bit\-dependent term in[Equation5\.6](https://arxiv.org/html/2609.13500#S5.E6)is tight up to constants\.

Here and below,CRC\_\{R\}depends only on the logit boundRR\.

###### Proof\.

For the score matrixU=WQ⊤​WK/dhU=W\_\{Q\}^\{\\top\}W\_\{K\}/\\sqrt\{d\_\{h\}\}of the stated construction,

xq⊤​U​xt=\{β,t=t⋆,0,t≠t⋆\.x\_\{q\}^\{\\top\}Ux\_\{t\}=\\begin\{cases\}\\beta,&t=t\_\{\\star\},\\\\ 0,&t\\neq t\_\{\\star\}\.\\end\{cases\}Softmax therefore gives[Equation5\.4](https://arxiv.org/html/2609.13500#S5.E4)\. Since∑t≠t⋆αt=1−α⋆\\sum\_\{t\\neq t\_\{\\star\}\}\\alpha\_\{t\}=1\-\\alpha\_\{\\star\}and\|ϑj\|≤R\\lvert\\vartheta\_\{j\}\\rvert\\leq R,

\|z−ϑj⋆\|\\displaystyle\\lvert z\-\\vartheta\_\{j\_\{\\star\}\}\\rvert=\|∑t≠t⋆αt​\(ϑjt−ϑj⋆\)\|\\displaystyle=\\lvert\\sum\_\{t\\neq t\_\{\\star\}\}\\alpha\_\{t\}\(\\vartheta\_\{j\_\{t\}\}\-\\vartheta\_\{j\_\{\\star\}\}\)\\rvert≤2​R​\(1−α⋆\)\\displaystyle\\leq 2R\(1\-\\alpha\_\{\\star\}\)=2​R​K−1eβ\+K−1\\displaystyle=2R\\frac\{K\-1\}\{e^\{\\beta\}\+K\-1\}≤2​R​\(K−1\)​e−β\.\\displaystyle\\leq 2R\(K\-1\)e^\{\-\\beta\}\.
Uniformly quantizing\[−Δj,Δj\]\[\-\\Delta\_\{j\},\\Delta\_\{j\}\]with2bj2^\{b\_\{j\}\}values gives\|w~jmode−wjmode\|≤2​Δj​2−bj\\lvert\\widetilde\{w\}\_\{j\}^\{\\rm mode\}\-w\_\{j\}^\{\\rm mode\}\\rvert\\leq 2\\Delta\_\{j\}2^\{\-b\_\{j\}\}forj∈\[M\]j\\in\[M\]\. Writeϑ~j=w~jmode\\widetilde\{\\vartheta\}\_\{j\}=\\widetilde\{w\}\_\{j\}^\{\\rm mode\}on\[M\]\[M\]andϑ~j=0\\widetilde\{\\vartheta\}\_\{j\}=0otherwise\. Distractor quantization contributes at most2​R​\(1−α⋆\)2R\(1\-\\alpha\_\{\\star\}\), so it is absorbed by the routing term\.

It remains to justify the arithmetic term\. Stable max shifting replaces the scoresst∈\{0,β\}s\_\{t\}\\in\\\{0,\\beta\\\}bys¯t=st−β∈\{−β,0\}\\bar\{s\}\_\{t\}=s\_\{t\}\-\\beta\\in\\\{\-\\beta,0\\\}without changing the exact softmax\. Letyt=es¯ty\_\{t\}=e^\{\\bar\{s\}\_\{t\}\}andh=⌈log2⁡K⌉h=\\lceil\\log\_\{2\}K\\rceil\. By assumption, the computed unnormalized weight is

y^t=yt​\(1\+δt\),\|δt\|≤u\.\\widehat\{y\}\_\{t\}=y\_\{t\}\(1\+\\delta\_\{t\}\),\\qquad\\lvert\\delta\_\{t\}\\rvert\\leq u\.Balanced positive summation gives

S^=\(∑ty^t\)​\(1\+δS\),\|δS\|≤γh≔h​u1−h​u\.\\widehat\{S\}=\\left\(\\sum\_\{t\}\\widehat\{y\}\_\{t\}\\right\)\(1\+\\delta\_\{S\}\),\\qquad\\lvert\\delta\_\{S\}\\rvert\\leq\\gamma\_\{h\}\\coloneqq\\frac\{hu\}\{1\-hu\}\.Including the final division, the relative\-error factors can be bounded explicitly\. Putδ¯=∑tαt​δt\\bar\{\\delta\}=\\sum\_\{t\}\\alpha\_\{t\}\\delta\_\{t\}, so\|δ¯\|≤u\|\\bar\{\\delta\}\|\\leq u, and let\|δD,t\|≤u\|\\delta\_\{D,t\}\|\\leq ube the division error\. Then

α^tαt=\(1\+δt\)​\(1\+δD,t\)\(1\+δ¯\)​\(1\+δS\)\.\\frac\{\\widehat\{\\alpha\}\_\{t\}\}\{\\alpha\_\{t\}\}=\\frac\{\(1\+\\delta\_\{t\}\)\(1\+\\delta\_\{D,t\}\)\}\{\(1\+\\bar\{\\delta\}\)\(1\+\\delta\_\{S\}\)\}\.When\(h\+4\)​u≤1/4\(h\+4\)u\\leq 1/4, we haveγh≤4​h​u/3\\gamma\_\{h\}\\leq 4hu/3, and both denominator factors are bounded away from zero\. Subtracting numerator and denominator and bounding the linear and quadratic terms gives\|α^t/αt−1\|≤8​\(h\+4\)​u\|\\widehat\{\\alpha\}\_\{t\}/\\alpha\_\{t\}\-1\|\\leq 8\(h\+4\)u\. Consequently

α^t=αt​\(1\+εt\),\|εt\|≤8​\(h\+4\)​u\.\\widehat\{\\alpha\}\_\{t\}=\\alpha\_\{t\}\(1\+\\varepsilon\_\{t\}\),\\qquad\\lvert\\varepsilon\_\{t\}\\rvert\\leq 8\(h\+4\)u\.Summing againstαt\\alpha\_\{t\}proves[EquationB\.2](https://arxiv.org/html/2609.13500#A2.E2)\.

For the final dot product, each multiplication has relative error at mostuu, and balanced signed summation has absolute error at most aγh\\gamma\_\{h\}multiple of the sum of magnitudes\. Since

∑t\|ϑ~jt\|​α^t≤R⁡\{1\+O⁡\(u⁡\(1\+log⁡K\)\)\},\\sum\_\{t\}\\lvert\\widetilde\{\\vartheta\}\_\{j\_\{t\}\}\\rvert\\widehat\{\\alpha\}\_\{t\}\\leq R\\\{1\+O\(u\(1\+\\log K\)\)\\\},we obtain[EquationB\.3](https://arxiv.org/html/2609.13500#A2.E3)\. For every matching mode, represented or not, combining the contributions gives

\|z^−θj⋆\|≤4R\(K−1\)e−β\+2Δj⋆2−bj⋆𝟏\{j⋆≤M\}\+\|θj⋆\|𝟏\{j⋆\>M\}\+CRu\(1\+logK\)\.\\lvert\\widehat\{z\}\-\\theta\_\{j\_\{\\star\}\}\\rvert\\leq 4R\(K\-1\)e^\{\-\\beta\}\+2\\Delta\_\{j\_\{\\star\}\}2^\{\-b\_\{j\_\{\\star\}\}\}\\mathbf\{1\}\\\{j\_\{\\star\}\\leq M\\\}\+\|\\theta\_\{j\_\{\\star\}\}\|\\mathbf\{1\}\\\{j\_\{\\star\}\>M\\\}\+C\_\{R\}u\(1\+\\log K\)\.The two mode\-specific terms are mutually exclusive\. The squared sum is therefore bounded by a constant times routing error, arithmetic error, and their squared mode\-specific sum\. Averaging over the selected coordinate adds∑j≤Mqj​2−2​bj\+∑j\>Mqj\\sum\_\{j\\leq M\}q\_\{j\}2^\{\-2b\_\{j\}\}\+\\sum\_\{j\>M\}q\_\{j\}\. An unrepresented matching mode can still attend to represented distractors; that contribution is already bounded by the routing term above, rather than being assumed zero\. FinallyA′′​\(z\)≤1/4A^\{\\prime\\prime\}\(z\)\\leq 1/4for every real logit, so the upper KL curvature bound applies even if floating\-point error movesz^\\widehat\{z\}slightly outside the original box\. This proves[Equation5\.6](https://arxiv.org/html/2609.13500#S5.E6)\.

For the converse, restrict to the separable scalar table\. Coordinatejjhas at most2bj2^\{b\_\{j\}\}reconstruction values\. Under the same product\-uniform prior on the coordinate logits, the one\-dimensional specialization of[LemmaE\.1](https://arxiv.org/html/2609.13500#A5.Thmtheorem1)gives Bayes weighted squared error at leastc​qj​2−2​bjcq\_\{j\}2^\{\-2b\_\{j\}\}on each coordinate\. Expectations add under this one common prior, so the sum lower\-bounds worst\-case scalar\-table distortion; it is not a pointwise assertion about every parameter\. Apply the lower curvature bound to the clipped table decoder to obtain the stated tightness within that representation\. The construction stores oned×dd\\times dscore matrix, oned×dd\\times dvalue matrix, and onedd\-coordinate head, henceN=2​d2\+dN=2d^\{2\}\+d\. ∎

## Appendix KProof of the Transformer Bit\-Complexity Lemma

This appendix counts the dense forward–backward graph stated in[SectionB\.3](https://arxiv.org/html/2609.13500#A2.SS3)\. Its sequence length, hidden width, feed\-forward width, head count, and depth areℓ,d,mff,H,L\\ell,d,m\_\{\\rm ff\},H,L, respectively\. The precision indices specify the operand, accumulator, and stored\-activation formats whose costs are counted\.

###### Lemma K\.1\(Transformer forward–backward bit complexity; formal version of[LemmaB\.1](https://arxiv.org/html/2609.13500#A2.Thmtheorem1)\)\.

For the graph above, with positive dimensions and conventional dense matrix multiplication,

BitOpsfb≍L⁡\[\(ℓ​d2\+ℓ​d​mff\+ℓ2​d\)​𝖬​\(bmm\)\+H​ℓ2​𝖲​\(bsm\)\]\.\\operatorname\{BitOps\}\_\{\\rm fb\}\\asymp L\\Bigl\[\\bigl\(\\ell d^\{2\}\+\\ell dm\_\{\\rm ff\}\+\\ell^\{2\}d\\bigr\)\\mathsf\{M\}\(b\_\{\\rm mm\}\)\+H\\ell^\{2\}\\mathsf\{S\}\(b\_\{\\rm sm\}\)\\Bigr\]\.\(K\.1\)Memfb≍bw​L​\(d2\+d​mff\)\+ba​L​ℓ​\(d\+mff\)\+bst​L​H​ℓ2\+Memhp\.\\operatorname\{Mem\}\_\{\\rm fb\}\\asymp b\_\{w\}L\(d^\{2\}\+dm\_\{\\rm ff\}\)\+b\_\{a\}L\\ell\(d\+m\_\{\\rm ff\}\)\+b\_\{\\rm st\}LH\\ell^\{2\}\+\\operatorname\{Mem\}\_\{\\rm hp\}\.\(K\.2\)The memoryMemfb\\operatorname\{Mem\}\_\{\\rm fb\}is measured in bits\. The valuesbwb\_\{w\},bab\_\{a\}, andbstb\_\{\\rm st\}count aggregate bits per logical weight, activation, and saved\-attention slot at peak memory; they include any same\-shaped gradient or backward\-workspace buffer materialized by this eager implementation\. The termMemhp\\operatorname\{Mem\}\_\{\\rm hp\}contains master weights, optimizer state, and all other explicitly retained high\-precision data\. A streaming, recomputing attention implementation replaces the quadratic attention state saved for backward by linear online\-softmax state plus temporary tile workspace, while leaving the dense attention arithmetic order unchanged\.

###### Proof\.

Suppressing constant head reshapes, the four query, key, value, and output projections perform4​ℓ​d24\\ell d^\{2\}multiply–accumulates per layer\. LetQmatQ\_\{\\rm mat\},KmatK\_\{\\rm mat\}, andVmatV\_\{\\rm mat\}denote the query, key, and value matrices, and letPPdenote the matrix of attention probabilities\. The query–key score productQmat​Kmat⊤Q\_\{\\rm mat\}K\_\{\\rm mat\}^\{\\top\}and probability–value productP​VmatPV\_\{\\rm mat\}each performℓ2​d\\ell^\{2\}d, and the two feed\-forward maps perform2​ℓ​d​mff2\\ell dm\_\{\\rm ff\}\. Thus the forward matrix count is

Fmat=4​ℓ​d2\+2​ℓ2​d\+2​ℓ​d​mff\.F\_\{\\rm mat\}=4\\ell d^\{2\}\+2\\ell^\{2\}d\+2\\ell dm\_\{\\rm ff\}\.\(K\.3\)For any productY=A​BY=AB, reverse mode forms∇A=∇YB⊤\\nabla\_\{A\}=\\nabla\_\{Y\}B^\{\\top\}and∇B=A⊤​∇Y\\nabla\_\{B\}=A^\{\\top\}\\nabla\_\{Y\}\. Each has the same scalar multiply–accumulate count as the forward product\. Consequently the matrix part of one forward–backward pass is exactly three times[EquationK\.3](https://arxiv.org/html/2609.13500#A11.E3), up to additions, transposes, and casts\.

There areH​ℓH\\ellsoftmax rows of lengthℓ\\ell\. Stable max shifting, exponentiation, reduction, division, and the reverse\-mode row calculation together costΘ⁡\(H​ℓ2​𝖲​\(bsm\)\)\\Theta\(H\\ell^\{2\}\\mathsf\{S\}\(b\_\{\\rm sm\}\)\)by the definition of𝖲\\mathsf\{S\}\. Given an incoming row gradientvv, the softmax Jacobian is applied by

p⊙\(v−⟨p,v⟩​𝟏\),p\\odot\\bigl\(v\-\\langle p,v\\rangle\\mathbf\{1\}\\bigr\),which uses a constant number of length\-ℓ\\ellpasses\. No explicitℓ×ℓ\\ell\\times\\ellJacobian is formed\. Multiplying byLLand absorbing fixed projection constants proves[EquationB\.5](https://arxiv.org/html/2609.13500#A2.E5)\. Normalization, pointwise activation, dropout, and casts have onlyO⁡\(L​ℓ​\(d\+mff\)\+L​H​ℓ2\)O\(L\\ell\(d\+m\_\{\\rm ff\}\)\+LH\\ell^\{2\}\)scalar entries and are the stated lower\-order terms for this computation\.

For memory, the dense projection and feed\-forward weights containΘ⁡\(L⁡\(d2\+d​mff\)\)\\Theta\(L\(d^\{2\}\+dm\_\{\\rm ff\}\)\)scalars\. Saved residual, projection, and feed\-forward activations containΘ⁡\(L​ℓ​\(d\+mff\)\)\\Theta\(L\\ell\(d\+m\_\{\\rm ff\}\)\)scalars\. Eager reverse mode also savesL​H​ℓ2LH\\ell^\{2\}attention probabilities\. Multiplying each collection by its aggregate bits per logical slot \(including same\-shaped peak gradient or workspace buffers\) and adding separately retained high\-precision state gives[EquationB\.6](https://arxiv.org/html/2609.13500#A2.E6)\. A tiled online softmax stores row normalization statistics and tiles rather than the complete probability matrix, so its persistent attention state is linear inH​ℓH\\ell; recomputing tiles during backward does not change the dense arithmetic order\. These counts also give the two\-sided bounds because each named tensor and matrix product is present in the specified graph\. ∎

## Appendix LProof of the Compute–Precision Pareto Law

The proof distinguishes the precision required to match a target error from the exact precision chosen by a finite\-budget optimizer\.

###### Theorem L\.1\(Compute–precision Pareto law; formal version of[TheoremB\.2](https://arxiv.org/html/2609.13500#A2.Thmtheorem2)\)\.

Assume[EquationsB\.7](https://arxiv.org/html/2609.13500#A2.E7)and[B\.8](https://arxiv.org/html/2609.13500#A2.E8),κmm​\(M\)≍Mr\\kappa\_\{\\rm mm\}\(M\)\\asymp M^\{r\}, andκsm​\(M\)≍Mrsm\\kappa\_\{\\rm sm\}\(M\)\\asymp M^\{r\_\{\\rm sm\}\}, with all displayed relations two\-sided andr,rsm≥0r,r\_\{\\rm sm\}\\geq 0\. For the specified dense graph,H≤dH\\leq dimpliesG≳SG\\gtrsim Sand henceg≥sg\\geq s\. Item 3 concerns general operation counts satisfying the same two\-sided relations; items 1, 2, and 4 apply to both settings\.

1. 1\.The smallest matrix precision that keeps matrix rounding at the model error is bmatch​\(M\)=\(r\+χ2\)​log2​M\+O⁡\(1\)\.b\_\{\\rm match\}\(M\)=\\left\(r\+\\frac\{\\chi\}\{2\}\\right\)\\log\_\{2\}M\+O\(1\)\.\(L\.1\)
2. 2\.If the matrix term dominates the bracket in[EquationB\.8](https://arxiv.org/html/2609.13500#A2.E8)at matched precisions—in particular in the caseg≥sg\\geq swhen the remaining precision factors do not reverse that dominance—then infM,b,bsm:cost≤Cℰ≍C−χ/\(1\+χ\+g\)\(logC\)χ​ν/\(1\+χ\+g\),\\inf\_\{M,b,b\_\{\\rm sm\}:\\,\\mathrm\{cost\}\\leq C\}\\mathcal\{E\}\\asymp C^\{\-\\chi/\(1\+\\chi\+g\)\}\(\\log C\)^\{\\chi\\nu/\(1\+\\chi\+g\)\},\(L\.2\)up to other declared slowly varying factors\.
3. 3\.For general operation counts, ifs\>gs\>g,bsmb\_\{\\rm sm\}is fixed, and the softmax term dominates computation, then for budgets in the pre\-floor range C1/\(1\+χ\+s\)≲22​bsm/\(χ\+2​rsm\),C^\{1/\(1\+\\chi\+s\)\}\\lesssim 2^\{2b\_\{\\rm sm\}/\(\\chi\+2r\_\{\\rm sm\}\)\},infM,b:cost≤Cℰ≍C−χ/\(1\+χ\+s\)\.\\inf\_\{M,b:\\,\\mathrm\{cost\}\\leq C\}\\mathcal\{E\}\\asymp C^\{\-\\chi/\(1\+\\chi\+s\)\}\.\(L\.3\)Reducing matrix precision cannot improve this leading exponent\.
4. 4\.At fixed matrix precisionbb, with softmax precision optimized, the model–rounding crossover and the infimum error over model scales are Mb≍22​b/\(χ\+2​r\),ℰfloor\(b\)≍2−2χb/\(χ\+2r\)\.M\_\{b\}\\asymp 2^\{2b/\(\\chi\+2r\)\},\\qquad\\mathcal\{E\}\_\{\\rm floor\}\(b\)\\asymp 2^\{\-2\\chi b/\(\\chi\+2r\)\}\.\(L\.4\)A fixedbsmb\_\{\\rm sm\}\-bit softmax similarly supports the scaling phase only while M≲22​bsm/\(χ\+2​rsm\)\.M\\lesssim 2^\{2b\_\{\\rm sm\}/\(\\chi\+2r\_\{\\rm sm\}\)\}\.\(L\.5\)

The displayed two\-sided risk and arithmetic assumptions define the class over which the infima and matching bounds are taken\.

###### Proof\.

The matrix rounding term is at most a constant multiple of the model term exactly when

M2​r​2−2​b≲M−χ\.M^\{2r\}2^\{\-2b\}\\lesssim M^\{\-\\chi\}\.Taking base\-two logarithms proves[EquationB\.9](https://arxiv.org/html/2609.13500#A2.E9); the two\-sided risk assumption gives necessity for matching the model\-error scale\. For the dense graph, positive head dimensions sum todd, so

S⁡\(M\)≍L​H​ℓ2≤L​d​ℓ2≲G⁡\(M\)\.S\(M\)\\asymp LH\\ell^\{2\}\\leq Ld\\ell^\{2\}\\lesssim G\(M\)\.Henceg≥sg\\geq s\. The general\-count branch uses[EquationsB\.7](https://arxiv.org/html/2609.13500#A2.E7)and[B\.8](https://arxiv.org/html/2609.13500#A2.E8)with independent nonnegative exponentsg,sg,s\.

Suppose first that the matrix term dominates the bracket in[EquationB\.8](https://arxiv.org/html/2609.13500#A2.E8)\. Choose both numerical errors below a constant multiple ofM−χM^\{\-\\chi\}\. By[EquationB\.9](https://arxiv.org/html/2609.13500#A2.E9),

C≍M1\+χ\+g​\(log⁡M\)ν\.C\\asymp M^\{1\+\\chi\+g\}\(\\log M\)^\{\\nu\}\.Letζ=1\+χ\+g\\zeta=1\+\\chi\+g\. Substitution intoMζ​\(log⁡M\)νM^\{\\zeta\}\(\\log M\)^\{\\nu\}gives cost within constant factors ofCCat

M≍C1/ζ\(logC\)−ν/ζ\.M\\asymp C^\{1/\\zeta\}\(\\log C\)^\{\-\\nu/\\zeta\}\.Substituting intoM−χM^\{\-\\chi\}proves the upper bound in[EquationB\.10](https://arxiv.org/html/2609.13500#A2.E10)\. For the converse, consider any feasible configuration with risk at mostε\\varepsilon\. The risk relation gives

M≳ε−1/χ,2−2​b≲ε,b≳log\(1/ε\),M\\gtrsim\\varepsilon^\{\-1/\\chi\},\\qquad 2^\{\-2b\}\\lesssim\\varepsilon,\\qquad b\\gtrsim\\log\(1/\\varepsilon\),usingr≥0r\\geq 0\. The matrix contribution alone forces

C≳ε−ζ/χ\[log\(1/ε\)\]ν\.C\\gtrsim\\varepsilon^\{\-\\zeta/\\chi\}\[\\log\(1/\\varepsilon\)\]^\{\\nu\}\.Inverting proves the lower rate for every configuration\. Forν=0\\nu=0, the logarithmic factor equals one and the same argument applies\.

Now use general operation counts withs\>gs\>g, fixedbsmb\_\{\\rm sm\}, and dominating softmax work before its numerical floor\. Matching the matrix error still costs onlyb=Θ⁡\(log⁡M\)b=\\Theta\(\\log M\), but the leading cost is

C≍M1\+χ\+s​bsmνsm≍M1\+χ\+s\.C\\asymp M^\{1\+\\chi\+s\}b\_\{\\rm sm\}^\{\\nu\_\{\\rm sm\}\}\\asymp M^\{1\+\\chi\+s\}\.Within the range whereM2​rsm​2−2​bsm≲M−χM^\{2r\_\{\\rm sm\}\}2^\{\-2b\_\{\\rm sm\}\}\\lesssim M^\{\-\\chi\}, substitution proves[EquationB\.11](https://arxiv.org/html/2609.13500#A2.E11)\. Conversely, risk at mostε\\varepsilonrequiresM≳ε−1/χM\\gtrsim\\varepsilon^\{\-1/\\chi\}, and the softmax cost is at least a constant timesM1\+χ\+sM^\{1\+\\chi\+s\}\. ThusC≳ε−\(1\+χ\+s\)/χC\\gtrsim\\varepsilon^\{\-\(1\+\\chi\+s\)/\\chi\}\. Since the leading term contains nobb, changing matrix precision cannot change this exponent\. Equivalently, substitutingM≍C1/\(1\+χ\+s\)M\\asymp C^\{1/\(1\+\\chi\+s\)\}gives exactly the pre\-floor budget condition in the theorem statement\.

Finally, at fixedbb, the model and matrix\-rounding terms cross when

M−χ≍M2​r​2−2​b,Mχ\+2​r≍22​b\.M^\{\-\\chi\}\\asymp M^\{2r\}2^\{\-2b\},\\qquad M^\{\\chi\+2r\}\\asymp 2^\{2b\}\.This givesMbM\_\{b\}in[EquationB\.12](https://arxiv.org/html/2609.13500#A2.E12)\. Choose softmax precision so its error is at most the other terms\. IfM≤MbM\\leq M\_\{b\}, the model term is at least a constant timesMb−χM\_\{b\}^\{\-\\chi\}; ifM≥MbM\\geq M\_\{b\}, the matrix\-rounding term is, sincer≥0r\\geq 0\. AtMbM\_\{b\}both have this order\. Whenr=0r=0, the comparison expressionM−χ\+2−2​bM^\{\-\\chi\}\+2^\{\-2b\}decreases to2−2​b2^\{\-2b\}; its infimum has the displayed order andMbM\_\{b\}is the crossover scale\. Replacingr,br,bbyrsm,bsmr\_\{\\rm sm\},b\_\{\\rm sm\}proves[EquationB\.13](https://arxiv.org/html/2609.13500#A2.E13)\. The fixed\-share ratios in[EquationsB\.14](https://arxiv.org/html/2609.13500#A2.E14)and[B\.15](https://arxiv.org/html/2609.13500#A2.E15)follow by dividing the baseline cost or storage by its quantized counterpart\. ∎

## Appendix MProof of the Compute Phases

This appendix derives each phase only from the two\-sided resource assumptions in[Section6](https://arxiv.org/html/2609.13500#S6)\.

###### Theorem M\.1\(Precision\-aware compute phases; formal version of[Theorem6\.1](https://arxiv.org/html/2609.13500#S6.Thmtheorem1)\)\.

Letχ\>0\\chi\>0andf,ν≥0f,\\nu\\geq 0\. Optimize over configurations usingn⁡\(M\)≍M1\+χn\(M\)\\asymp M^\{1\+\\chi\}blocks whose total excess risk and cost satisfy, uniformly overM,b≥1M,b\\geq 1,

ℰ⁡\(M,b\)≍M−χ\+κ​\(M\)2​2−2​b,cost⁡\(M,b\)≍Mf\+1\+χ​bν\.\\mathcal\{E\}\(M,b\)\\asymp M^\{\-\\chi\}\+\\kappa\(M\)^\{2\}2^\{\-2b\},\\qquad\\operatorname\{cost\}\(M,b\)\\asymp M^\{f\+1\+\\chi\}b^\{\\nu\}\.Letℛ⁡\(C\)−ℒ∞=infcost⁡\(M,b\)≤Cℰ⁡\(M,b\)\\mathcal\{R\}\(C\)\-\\mathcal\{L\}\_\{\\infty\}=\\inf\_\{\\operatorname\{cost\}\(M,b\)\\leq C\}\\mathcal\{E\}\(M,b\)\. Then:

1. 1\.If1≲κ⁡\(M\)≲Mρ1\\lesssim\\kappa\(M\)\\lesssim M^\{\\rho\}, then the least model\-error\-matching precision isb=Θ⁡\(log⁡M\)b=\\Theta\(\\log M\), and ℛ\(C\)−ℒ∞≍C−χ/\(f\+1\+χ\)\(logC\)χ​ν/\(f\+1\+χ\)\\mathcal\{R\}\(C\)\-\\mathcal\{L\}\_\{\\infty\}\\asymp C^\{\-\\chi/\(f\+1\+\\chi\)\}\(\\log C\)^\{\\chi\\nu/\(f\+1\+\\chi\)\}\(M\.1\)up to other slowly varying factors specified in the assumptions\.
2. 2\.Ifκ⁡\(M\)=exp⁡\(Θ⁡\(Mλ\)\)\\kappa\(M\)=\\exp\(\\Theta\(M^\{\\lambda\}\)\),λ\>0\\lambda\>0, thenb=Θ⁡\(Mλ\)b=\\Theta\(M^\{\\lambda\}\), and the loss exponent isχ/\(f\+1\+χ\+λ​ν\)\\chi/\(f\+1\+\\chi\+\\lambda\\nu\)\.
3. 3\.Ifqj≍j−\(1\+χ\)q\_\{j\}\\asymp j^\{\-\(1\+\\chi\)\}, adaptive per\-mode precision has errorΘ⁡\(B−χ\)\\Theta\(B^\{\-\\chi\}\), whereas a uniform\-width prefix has optimal error Θ⁡\(\(B/log⁡B\)−χ\)\.\\Theta\(\(B/\\log B\)^\{\-\\chi\}\)\.\(M\.2\)

Each converse uses the representation, operation set, storage layout, and bit\-operation cost specified above\.

###### Proof\.

Matching numerical error to approximation error requires

κ​\(M\)2​2−2​b≲M−χ\.\\kappa\(M\)^\{2\}2^\{\-2b\}\\lesssim M^\{\-\\chi\}\.Taking base\-two logarithms gives[Equation6\.3](https://arxiv.org/html/2609.13500#S6.E3)\. The two\-sided amplification assumption gives the matching necessity\.

If1≲κ⁡\(M\)≲Mρ1\\lesssim\\kappa\(M\)\\lesssim M^\{\\rho\}, thenb=Θ⁡\(log⁡M\)b=\\Theta\(\\log M\)\. Multiplying per\-block work, number of blocks, and bit\-operation cost yields

C≍Mf​M1\+χ​\(log⁡M\)ν=Mζ​\(log⁡M\)ν,ζ=f\+1\+χ\.C\\asymp M^\{f\}M^\{1\+\\chi\}\(\\log M\)^\{\\nu\}=M^\{\\zeta\}\(\\log M\)^\{\\nu\},\\qquad\\zeta=f\+1\+\\chi\.Substitution into the preceding cost relation gives the budget\-matched scale

M≍C1/ζ\(logC\)−ν/ζ\.M\\asymp C^\{1/\\zeta\}\(\\log C\)^\{\-\\nu/\\zeta\}\.Substitution intoM−χM^\{\-\\chi\}proves the upper bound in[Equation6\.1](https://arxiv.org/html/2609.13500#S6.E1)\. Conversely, risk at mostε\\varepsilonandκ⁡\(M\)≳1\\kappa\(M\)\\gtrsim 1requireM≳ε−1/χM\\gtrsim\\varepsilon^\{\-1/\\chi\}andb≳log⁡\(1/ε\)b\\gtrsim\\log\(1/\\varepsilon\)\. The necessary block count and per\-block work give

C≳ε−\(f\+1\+χ\)/χ\[log\(1/ε\)\]ν\.C\\gtrsim\\varepsilon^\{\-\(f\+1\+\\chi\)/\\chi\}\[\\log\(1/\\varepsilon\)\]^\{\\nu\}\.Inverting proves the lower bound, includingν=0\\nu=0\.

Ifκ⁡\(M\)=exp⁡\(Θ⁡\(Mλ\)\)\\kappa\(M\)=\\exp\(\\Theta\(M^\{\\lambda\}\)\)forλ\>0\\lambda\>0, then[Equation6\.3](https://arxiv.org/html/2609.13500#S6.E3)givesb=Θ⁡\(Mλ\)b=\\Theta\(M^\{\\lambda\}\), so

C≍Mf\+1\+χ\+λ​ν\.C\\asymp M^\{f\+1\+\\chi\+\\lambda\\nu\}\.This gives the claimed upper rate\. Conversely, risk at mostε\\varepsilonforcesM≳ε−1/χM\\gtrsim\\varepsilon^\{\-1/\\chi\}andb≳Mλ\+log⁡\(1/ε\)b\\gtrsim M^\{\\lambda\}\+\\log\(1/\\varepsilon\), whence

C≳Mf\+1\+χ\+λ​ν≳ε−\(f\+1\+χ\+λν\)/χ\.C\\gtrsim M^\{f\+1\+\\chi\+\\lambda\\nu\}\\gtrsim\\varepsilon^\{\-\(f\+1\+\\chi\+\\lambda\\nu\)/\\chi\}\.This proves the exponent in both directions\.

For adaptive precision, setaj=qja\_\{j\}=q\_\{j\}andcj=1c\_\{j\}=1in[CorollaryF\.3](https://arxiv.org/html/2609.13500#A6.Thmtheorem3); thenD⁡\(B\)≍B−χD\(B\)\\asymp B^\{\-\\chi\}\. A uniform\-width prefix of lengthMMand widthbbusesB=M​bB=Mbbits\. Its omitted tail isΘ⁡\(M−χ\)\\Theta\(M^\{\-\\chi\}\), and its represented\-coordinate quantization isΘ⁡\(2−2​b\)\\Theta\(2^\{\-2b\}\)because∑j≤Mqj=Θ⁡\(1\)\\sum\_\{j\\leq M\}q\_\{j\}=\\Theta\(1\)\. ChooseM≍B/log⁡BM\\asymp B/\\log Bwith a sufficiently small constant andb=⌊B/M⌋b=\\lfloor B/M\\rfloorfor the upper bound in[Equation6\.2](https://arxiv.org/html/2609.13500#S6.E2)\. For the converse, setTB=\(B/log2⁡B\)−χT\_\{B\}=\(B/\\log\_\{2\}B\)^\{\-\\chi\}and fixL\>2/χL\>2/\\chi\. IfM≤L​B/log2⁡BM\\leq LB/\\log\_\{2\}B, the tail is at leastc​L−χ​TBcL^\{\-\\chi\}T\_\{B\}\. Otherwiseb≤B/M<\(log2⁡B\)/Lb\\leq B/M<\(\\log\_\{2\}B\)/L, so2−2​b\>B−2/L≳TB2^\{\-2b\}\>B^\{\-2/L\}\\gtrsim T\_\{B\}for sufficiently largeBB\. Every uniform\-width prefix therefore pays at least a constant timesTBT\_\{B\}, including zero\-width prefixes\. ∎

## Appendix NExperimental Details

This appendix describes the 1,024\-mode synthetic study\. Its five\-seed percentile intervals apply to this study; the other experiments use the BCa procedures and seed or item units specified in their respective captions\.

#### Source families\.

Each run uses10241024modes and bounded logits\. For eachχ∈\{0\.5,1,2\}\\chi\\in\\\{0\.5,1,2\\\}, two different frequency–amplitude decompositions produce the same powerqj≍j−\(1\+χ\)q\_\{j\}\\asymp j^\{\-\(1\+\\chi\)\}\. Training blocks are independent regenerations\. Held\-out cross\-entropy is evaluated exactly from the population probabilities, so no test\-set Monte Carlo error is mixed into the reported training\-seed variability\.

#### Grid and methods\.

The grid contains four values ofnn, five values ofBB, three exponents, two decompositions, four allocation methods, and five seeds, for4⋅5⋅3⋅2⋅4⋅5=24004\\cdot 5\\cdot 3\\cdot 2\\cdot 4\\cdot 5=2400configuration–seed pairs\. The known\-energy method solves the finite energy sequence’s water\-filling problem\. The universal method uses[Equation5\.2](https://arxiv.org/html/2609.13500#S5.E2)\. Uniform width assigns a common precision to a prefix, and shuffled allocation breaks the energy–precision pairing\. Allocations never spend precision beyond the statistical resolution of a mode\.

#### Uncertainty\.

For each of the480480configurations, we report the mean over five seeds\. The plotted interval is the 2\.5 and 97\.5 percentile envelope from50005000bootstrap resamples of the seed mean, using bootstrap seed zero and expanding the interval when necessary to contain the observed mean\.

#### Reproducibility details\.

The raw rows, summaries, deterministic analytic curves, plotting sources, and experiment code accompany the paper\. The experiments use local CPUs, one common exponent specification across configurations, and analytic arithmetic cost as the theoretical resource\.

## Appendix OMixed Precision in Practice

The resource variables in[SectionB\.3](https://arxiv.org/html/2609.13500#A2.SS3)deliberately separate compute precision from storage precision\. In the asymptotic Pareto theorem,bbis an effective accuracy parameter with unit roundoffΘ⁡\(2−b\)\\Theta\(2^\{\-b\}\), and the modeled format width is proportional tobb\. For a concrete floating\-point format, total width and significand precision must instead be distinguished: the numerical horizon uses the actual unit roundoff, not the total number of stored bits\. This distinction is necessary in real mixed\-precision systems:

- •A softmax may compute its maximum, exponential, reduction, and division in 32\-bit floating point \(FP32\) even when the two surrounding attention matrix products use lower precision\. NVIDIA Transformer Engine, for example, documents an 8\-bit floating\-point \(FP8\) attention path with FP8 matrix multiplications and FP32 softmax\([NVIDIA, 2026](https://arxiv.org/html/2609.13500#bib.bib13)\)\.
- •A matrix kernel may accumulate products in a precision wider than its input and output formats\. Accumulation precision therefore belongs in the arithmetic model; it does not imply that the complete attention probability tensor is stored at that precision\.
- •The softmax output can be cast back to a lower\-precision format before the value product\. Conversely, an eager implementation may retain the probability tensor for backward\. These are distinct choices represented by softmax effective accuracy andbstb\_\{\\rm st\}, respectively\.
- •Mixed\-precision training may retain full\-precision master weights and optimizer state even when forward weights, activations, and gradients use a narrower format\([Micikevicius et al\., 2018](https://arxiv.org/html/2609.13500#bib.bib11)\)\. Such residual state is included inMemhp\\operatorname\{Mem\}\_\{\\rm hp\}, not hidden inside an effective average bit width\.
- •Tiled online softmax can avoid materializing the quadratic attention matrix by saving normalization statistics and recomputing attention tiles in backward\([Dao et al\., 2022](https://arxiv.org/html/2609.13500#bib.bib12);[NVIDIA, 2026](https://arxiv.org/html/2609.13500#bib.bib13)\)\. This changes the storage model and memory traffic, while dense attention still performs a quadratic number of score and value arithmetic operations\.

These observations explain why[EquationsB\.14](https://arxiv.org/html/2609.13500#A2.E14)and[B\.15](https://arxiv.org/html/2609.13500#A2.E15)are operation and storage ratios rather than throughput predictions\. Kernel fusion, memory traffic, device\-specific integer or floating\-point units, communication, and parallel scheduling can make the same bit\-operation count run at different wall\-clock speeds\. Implementations can therefore differ in speed even when these analytic counts agree\.

## Appendix PExtended Experimental Results

This appendix examines robustness to spectral shape and ordering, numerical formats, and pretrained\-model scale, and provides detailed slope estimates and reference losses\. Captions specify uncertainty for each stochastic quantity\.

### P\.1Candidate\-Count Replication and Exponent Grid

The pairedK=8K=8versusK=32K=32comparison in[Figure10](https://arxiv.org/html/2609.13500#A18.F10)tests how candidate count affects the finite\-sample construction and trained learner\. The profile\-by\-budget comparison in[Figure5](https://arxiv.org/html/2609.13500#A16.F5)tests exponent agreement\.[Table4\(b\)](https://arxiv.org/html/2609.13500#A16.T4.st2)gives unbinned numerical slope estimates\. The two\-stage study crosses four profiles and prefixes, three sample and state budgets,K=8,32K=8,32, and eight paired seeds\. Each sample budget restarts the learner, whereas state budgets reuse the fitted weights\. Repeated settings at the minimumn=64n=64are aggregated within seed before bootstrapping\. Display bins depend only onGμMG\_\{\\mu\_\{M\}\}\.

Figure 5:Six\-profile exponent agreement\.\(a\) Data\-limited and \(b\) memory\-limited fitted exponents for profile indices\(0\.5,0\),\(0\.5,0\.5\),\(1,0\),\(1,0\.5\),\(2,0\),\(2,0\.5\)\(0\.5,0\),\(0\.5,0\.5\),\(1,0\),\(1,0\.5\),\(2,0\),\(2,0\.5\)\. Green, blue, and orange denote low, middle, and high fixed resource ratios; gray diamonds give the theoretical exponent\. Bands are 95% BCa intervals over sixteen paired slopes: eight seeds for each of two frequency–amplitude decompositions \(constant and alternating logit radius\)\.
### P\.2Routing, Interventions, and Profile Robustness

The routing measurements and learned/known/frozen comparison in[Figure3](https://arxiv.org/html/2609.13500#S7.F3)\(c,f\) relate retrieval quality to prediction error\. The decomposition and interventions in[Tables2\(c\)](https://arxiv.org/html/2609.13500#S7.T2.st3)and[2\(d\)](https://arxiv.org/html/2609.13500#S7.T2.st4)isolate the error sources\.[Figure6](https://arxiv.org/html/2609.13500#A16.F6)tests allocation when the spectrum departs from an exact power law\.

Figure 6:Allocation beyond exact power laws\.Left, center, and right show a broken power law, logarithmic correction, and perturbed ordering, respectively, at\(χ,γ,M\)=\(1,0\.5,256\)\(\\chi,\\gamma,M\)=\(1,0\.5,256\)\. Each panel compares known\-energy, adaptive, and uniform allocation\. Known\-energy uses the trueqiq\_\{i\}; empirical allocation uses the count\-corrected scores in[AppendixR](https://arxiv.org/html/2609.13500#A18)\. Bands are 95% BCa intervals over eight paired seeds\.Table 3:Routing and precision contributions to prediction loss\.At\(χ,γ,M,K\)=\(1,0\.5,64,32\)\(\\chi,\\gamma,M,K\)=\(1,0\.5,64,32\), data ratio two and unit bit ratio: augmented loss isLlr,q\+M−χL\_\{\\mathrm\{lr\},q\}\+M^\{\-\\chi\}; Route\+ and Format\+ are the positive contrastsR\+R\_\{\+\}andF\+F\_\{\+\}defined in Appendix[R](https://arxiv.org/html/2609.13500#A18)\. Lower is better\. Construct\. denotes the finite\-sample construction\. Each±\\pmis the larger deviation to the 95% BCa endpoints\.MethodAug\. lossRoute\+Format\+Two\-stage0\.0191±0\.0002\\pm 0\.00020\.0000±0\.0000\\pm 0\.00000\.0000±0\.0000\\pm 0\.0000Construct\.0\.0170±0\.0002\\pm 0\.00020\.0002±0\.0000\\pm 0\.00000\.0002±0\.0001\\pm 0\.0001Known route0\.0192±0\.0002\\pm 0\.00020\.0000±0\.0000\\pm 0\.00000\.0000±0\.0000\\pm 0\.0000Frozen route0\.0605±0\.0008\\pm 0\.00080\.0339±0\.0017\\pm 0\.00170\.0000±0\.0000\\pm 0\.0000No precond\.0\.0230±0\.0003\\pm 0\.00030\.0003±0\.0000\\pm 0\.00000\.0000±0\.0000\\pm 0\.0000Uniform bits0\.0191±0\.0004\\pm 0\.00040\.0000±0\.0000\\pm 0\.00000\.0000±0\.0000\\pm 0\.0000

### P\.3Stored Precision and Pretrained Transformers

Figure 7:Complete additional\-data and deployment\-format scan\.\(a–f\) Pythia 70M, 160M, 410M, 1B, 1\.4B, and 2\.8B test CE after four nested additional\-token budgets\. Curves show complete group\-128 2/3/4/6/8\-bit deployments and BF16; the shared logarithmic CE axis compares absolute loss and keeps low\-bit collapse visible\. Bands are 95% BCa intervals over eight paired training seeds evaluated on the same 96 test windows\.At 70M and 160M, the 2\-bit calibration CE reaches 114\.67 and 97\.78 nats, respectively\. These losses exceed the pre\-specified\[0,20\]\[0,20\]\-nat prediction range, and the fitting procedure yields no converged, in\-range solutions when calibration includes 2\-bit weights\. At 410M–2\.8B, all four response functions yield such solutions\. A sensitivity analysis fits 4/8\-bit and BF16 separately at every scale and predicts 6\-bit loss and the largest data budget\.[Tables5](https://arxiv.org/html/2609.13500#A16.T5),[6](https://arxiv.org/html/2609.13500#A16.T6),[7](https://arxiv.org/html/2609.13500#A16.T7)and[8](https://arxiv.org/html/2609.13500#A16.T8)report both calibration choices;[AppendixR](https://arxiv.org/html/2609.13500#A18)defines the response functions and evaluation sets\.

Figure 8:Stored precision and retained high\-precision work\.Left: format distortion versus nominal bits for shared\-scale, block\-scale, and adaptive\-bit representations of the causal source\. Right: analytic bit\-operation ratios for 8\-bit and 4\-bit matrix paths with FP32 softmax retained\. The source\-format curves use paired 95% BCa intervals; the right\-hand curves evaluate the analytic operation\-count formula\.Figure 9:Finite precision across six Pythia scales\.Left and center: paired cross\-entropy changes on WikiText\-103 \(96 paired windows\) and LAMBADA \(512 paired examples\), respectively\. Right: parameter\-weighted relative Frobenius error for the two HQQ formats, averaging‖W^−W‖F/‖W‖F\\\|\\widehat\{W\}\-W\\\|\_\{F\}/\\\|W\\\|\_\{F\}over layers with weights proportional to their parameter counts\. Blue circles, green triangles, orange squares, gray down\-triangles, and magenta diamonds denote INT8\-channel, INT8\-tensor, INT4\-RTN\-g128, INT4\-HQQ\-g128, and INT4\-HQQ\-g32, respectively\.Table 4:Reference losses and spectral slopes\.Left: Pythia BF16 CE over 96 WikiText\-103 windows or 512 LAMBADA items; bold/underlined marks the lowest/second\-lowest value per dataset\. Right: slope against the spectral predictor \(reference one\), from eight unbinned seed fits\. Each±\\pmis the larger deviation to the 95% BCa endpoints\.\(a\)Pythia BF16 baseline; lower is better\.ModelWikiTextLAMBADA70M6\.201±0\.047\\pm 0\.0476\.781±0\.323\\pm 0\.323160M4\.388±0\.056\\pm 0\.0564\.965±0\.339\\pm 0\.339410M2\.971±0\.059\\pm 0\.0592\.283±0\.216\\pm 0\.2161B2\.580±0\.059\\pm 0\.0591\.504±0\.179\\pm 0\.1791\.4B2\.548±0\.061\\pm 0\.0611\.470±0\.186\\pm 0\.1862\.8B2\.390±0\.060\\pm 0\.0601\.346±0\.170\\pm 0\.170

\(b\)Construction and trained\-learner slopes\.ProfileConstruct\.Trained\(0\.5,0\)\(0\.5,0\)0\.971±0\.040\\pm 0\.0400\.955±0\.033\\pm 0\.033\(0\.5,0\.5\)\(0\.5,0\.5\)0\.933±0\.017\\pm 0\.0170\.855±0\.028\\pm 0\.028\(1,0\)\(1,0\)0\.969±0\.029\\pm 0\.0290\.859±0\.018\\pm 0\.018\(1,0\.5\)\(1,0\.5\)0\.952±0\.023\\pm 0\.0230\.879±0\.026\\pm 0\.026

#### Predicting unseen data and storage budgets\.

With 4/8\-bit and BF16 calibration, the additive model lowers prediction error at the largest data budget for every scale \([Table5](https://arxiv.org/html/2609.13500#A16.T5)\)\. For 6\-bit interpolation, it raises error at 70M/160M and lowers it at the four larger scales \([Table6](https://arxiv.org/html/2609.13500#A16.T6)\)\. Each paired 95% BCa interval for these additive\-minus\-data differences excludes zero\. An interaction further improves extrapolation at 160M, 410M and 1\.4B; the corresponding intervals at the other three scales include zero\. The benefit of modeling both resources therefore depends on scale and on which resource is extrapolated\.[Tables7](https://arxiv.org/html/2609.13500#A16.T7)and[8](https://arxiv.org/html/2609.13500#A16.T8)give the predictions when calibration also includes 2\-bit weights\.

Table 5:Predicting the largest data budget from 4/8\-bit and BF16 calibration\.Root\-mean\-square error \(RMSE, nats; lower is better\) over 4/6/8\-bit and BF16 test CE at2262^\{26\}additional tokens\. Each response function in[AppendixR](https://arxiv.org/html/2609.13500#A18)is fitted on validation CE at the first three data budgets\. Eight paired training seeds and 96 fixed test windows are used; 5,000 seed\-level BCa resamples refit each function\. Smaller±\\pmvalues are rounded\-up symmetric envelopes of the 95% intervals\. Bold/underline mark the lowest/second\-lowest displayed RMSE per row\.ModelData onlyStorage onlyAdditiveInteraction70M2\.060±0\.0612\.060\\,\{\\scriptstyle\\pm\\,0\.061\}1\.293±0\.0231\.293\\,\{\\scriptstyle\\pm\\,0\.023\}1\.151¯±0\.080\\underline\{1\.151\}\\,\{\\scriptstyle\\pm\\,0\.080\}1\.149±0\.087\\mathbf\{1\.149\}\\,\{\\scriptstyle\\pm\\,0\.087\}160M0\.805±0\.0160\.805\\,\{\\scriptstyle\\pm\\,0\.016\}0\.619±0\.0340\.619\\,\{\\scriptstyle\\pm\\,0\.034\}0\.425¯±0\.037\\underline\{0\.425\}\\,\{\\scriptstyle\\pm\\,0\.037\}0\.392±0\.036\\mathbf\{0\.392\}\\,\{\\scriptstyle\\pm\\,0\.036\}410M0\.086±0\.0040\.086\\,\{\\scriptstyle\\pm\\,0\.004\}0\.156±0\.0050\.156\\,\{\\scriptstyle\\pm\\,0\.005\}0\.058¯±0\.006\\underline\{0\.058\}\\,\{\\scriptstyle\\pm\\,0\.006\}0\.054±0\.006\\mathbf\{0\.054\}\\,\{\\scriptstyle\\pm\\,0\.006\}1B0\.061¯±0\.003\\underline\{0\.061\}\\,\{\\scriptstyle\\pm\\,0\.003\}0\.096±0\.0020\.096\\,\{\\scriptstyle\\pm\\,0\.002\}0\.041±0\.003\\mathbf\{0\.041\}\\,\{\\scriptstyle\\pm\\,0\.003\}0\.041±0\.003\\mathbf\{0\.041\}\\,\{\\scriptstyle\\pm\\,0\.003\}1\.4B0\.057¯±0\.003\\underline\{0\.057\}\\,\{\\scriptstyle\\pm\\,0\.003\}0\.085±0\.0010\.085\\,\{\\scriptstyle\\pm\\,0\.001\}0\.033±0\.004\\mathbf\{0\.033\}\\,\{\\scriptstyle\\pm\\,0\.004\}0\.033±0\.004\\mathbf\{0\.033\}\\,\{\\scriptstyle\\pm\\,0\.004\}2\.8B0\.055¯±0\.003\\underline\{0\.055\}\\,\{\\scriptstyle\\pm\\,0\.003\}0\.074±0\.0020\.074\\,\{\\scriptstyle\\pm\\,0\.002\}0\.034±0\.004\\mathbf\{0\.034\}\\,\{\\scriptstyle\\pm\\,0\.004\}0\.034±0\.004\\mathbf\{0\.034\}\\,\{\\scriptstyle\\pm\\,0\.004\}Table 6:Predicting 6\-bit loss from 4/8\-bit and BF16 calibration\.RMSE \(nats; lower is better\) over the first three data budgets, holding out 6\-bit weights from calibration\. Response functions, eight paired training seeds, 96 fixed test windows, refitted 95% BCa envelopes, and ranking conventions match[Table5](https://arxiv.org/html/2609.13500#A16.T5)\.ModelData onlyStorage onlyAdditiveInteraction70M1\.304±0\.015\\mathbf\{1\.304\}\\,\{\\scriptstyle\\pm\\,0\.015\}1\.845±0\.0211\.845\\,\{\\scriptstyle\\pm\\,0\.021\}1\.842¯±0\.023\\underline\{1\.842\}\\,\{\\scriptstyle\\pm\\,0\.023\}1\.842¯±0\.023\\underline\{1\.842\}\\,\{\\scriptstyle\\pm\\,0\.023\}160M0\.649±0\.016\\mathbf\{0\.649\}\\,\{\\scriptstyle\\pm\\,0\.016\}0\.732±0\.0170\.732\\,\{\\scriptstyle\\pm\\,0\.017\}0\.731¯±0\.017\\underline\{0\.731\}\\,\{\\scriptstyle\\pm\\,0\.017\}0\.731¯±0\.017\\underline\{0\.731\}\\,\{\\scriptstyle\\pm\\,0\.017\}410M0\.093¯±0\.003\\underline\{0\.093\}\\,\{\\scriptstyle\\pm\\,0\.003\}0\.094±0\.0030\.094\\,\{\\scriptstyle\\pm\\,0\.003\}0\.081±0\.003\\mathbf\{0\.081\}\\,\{\\scriptstyle\\pm\\,0\.003\}0\.081±0\.003\\mathbf\{0\.081\}\\,\{\\scriptstyle\\pm\\,0\.003\}1B0\.054¯±0\.002\\underline\{0\.054\}\\,\{\\scriptstyle\\pm\\,0\.002\}0\.055±0\.0020\.055\\,\{\\scriptstyle\\pm\\,0\.002\}0\.046±0\.002\\mathbf\{0\.046\}\\,\{\\scriptstyle\\pm\\,0\.002\}0\.046±0\.002\\mathbf\{0\.046\}\\,\{\\scriptstyle\\pm\\,0\.002\}1\.4B0\.053±0\.0020\.053\\,\{\\scriptstyle\\pm\\,0\.002\}0\.047¯±0\.001\\underline\{0\.047\}\\,\{\\scriptstyle\\pm\\,0\.001\}0\.041±0\.002\\mathbf\{0\.041\}\\,\{\\scriptstyle\\pm\\,0\.002\}0\.041±0\.002\\mathbf\{0\.041\}\\,\{\\scriptstyle\\pm\\,0\.002\}2\.8B0\.051±0\.0020\.051\\,\{\\scriptstyle\\pm\\,0\.002\}0\.043¯±0\.002\\underline\{0\.043\}\\,\{\\scriptstyle\\pm\\,0\.002\}0\.039±0\.002\\mathbf\{0\.039\}\\,\{\\scriptstyle\\pm\\,0\.002\}0\.039±0\.002\\mathbf\{0\.039\}\\,\{\\scriptstyle\\pm\\,0\.002\}Table 7:Predicting the largest data budget with 2\-bit weights included in calibration\.RMSE \(nats; lower is better\) over all six deployment formats at2262^\{26\}additional tokens, fitted using 2/4/8\-bit and BF16 validation CE at the first three budgets\. This bounded\-response comparison covers 410M–2\.8B\. The 70M/160M 2\-bit calibration losses exceed its\[0,20\]\[0,20\]\-nat range;[Figure7](https://arxiv.org/html/2609.13500#A16.F7)shows their measured quantization losses\. Eight paired seeds, 96 test windows, 95% BCa envelopes and ranking conventions match[Table5](https://arxiv.org/html/2609.13500#A16.T5)\.ModelData onlyStorage onlyAdditiveInteraction410M3\.776±0\.0443\.776\\,\{\\scriptstyle\\pm\\,0\.044\}1\.354±0\.0081\.354\\,\{\\scriptstyle\\pm\\,0\.008\}1\.328¯±0\.030\\underline\{1\.328\}\\,\{\\scriptstyle\\pm\\,0\.030\}1\.327±0\.036\\mathbf\{1\.327\}\\,\{\\scriptstyle\\pm\\,0\.036\}1B3\.426±0\.0323\.426\\,\{\\scriptstyle\\pm\\,0\.032\}1\.088±0\.0101\.088\\,\{\\scriptstyle\\pm\\,0\.010\}1\.050¯±0\.019\\underline\{1\.050\}\\,\{\\scriptstyle\\pm\\,0\.019\}1\.045±0\.022\\mathbf\{1\.045\}\\,\{\\scriptstyle\\pm\\,0\.022\}1\.4B3\.724±0\.0313\.724\\,\{\\scriptstyle\\pm\\,0\.031\}1\.094±0\.0111\.094\\,\{\\scriptstyle\\pm\\,0\.011\}1\.063¯±0\.020\\underline\{1\.063\}\\,\{\\scriptstyle\\pm\\,0\.020\}1\.060±0\.028\\mathbf\{1\.060\}\\,\{\\scriptstyle\\pm\\,0\.028\}2\.8B3\.255±0\.0243\.255\\,\{\\scriptstyle\\pm\\,0\.024\}0\.797±0\.0050\.797\\,\{\\scriptstyle\\pm\\,0\.005\}0\.772¯±0\.006\\underline\{0\.772\}\\,\{\\scriptstyle\\pm\\,0\.006\}0\.771±0\.007\\mathbf\{0\.771\}\\,\{\\scriptstyle\\pm\\,0\.007\}Table 8:Predicting 3/6\-bit loss with 2\-bit weights included in calibration\.RMSE \(nats; lower is better\) at the first three data budgets, holding out 3/6\-bit weights from 2/4/8\-bit/BF16 calibration\. This comparison covers 410M–2\.8B, with the calibration range and scale coverage described in[Table7](https://arxiv.org/html/2609.13500#A16.T7)\. Eight paired training seeds, 96 test windows, 95% BCa envelopes and ranking conventions match[Table5](https://arxiv.org/html/2609.13500#A16.T5)\.ModelData onlyStorage onlyAdditiveInteraction410M1\.976±0\.0161\.976\\,\{\\scriptstyle\\pm\\,0\.016\}1\.573±0\.0321\.573\\,\{\\scriptstyle\\pm\\,0\.032\}1\.568±0\.033\\mathbf\{1\.568\}\\,\{\\scriptstyle\\pm\\,0\.033\}1\.569¯±0\.033\\underline\{1\.569\}\\,\{\\scriptstyle\\pm\\,0\.033\}1B1\.870±0\.0081\.870\\,\{\\scriptstyle\\pm\\,0\.008\}1\.512¯±0\.014\\underline\{1\.512\}\\,\{\\scriptstyle\\pm\\,0\.014\}1\.511±0\.014\\mathbf\{1\.511\}\\,\{\\scriptstyle\\pm\\,0\.014\}1\.511±0\.014\\mathbf\{1\.511\}\\,\{\\scriptstyle\\pm\\,0\.014\}1\.4B2\.152±0\.0142\.152\\,\{\\scriptstyle\\pm\\,0\.014\}1\.585¯±0\.012\\underline\{1\.585\}\\,\{\\scriptstyle\\pm\\,0\.012\}1\.584±0\.012\\mathbf\{1\.584\}\\,\{\\scriptstyle\\pm\\,0\.012\}1\.584±0\.012\\mathbf\{1\.584\}\\,\{\\scriptstyle\\pm\\,0\.012\}2\.8B1\.874±0\.0031\.874\\,\{\\scriptstyle\\pm\\,0\.003\}1\.145¯±0\.003\\underline\{1\.145\}\\,\{\\scriptstyle\\pm\\,0\.003\}1\.144±0\.003\\mathbf\{1\.144\}\\,\{\\scriptstyle\\pm\\,0\.003\}1\.144±0\.003\\mathbf\{1\.144\}\\,\{\\scriptstyle\\pm\\,0\.003\}

## Appendix QExperiment and Reproducibility Details

We retain the scientific configuration and seed\- or item\-level measurements for every reported result\. This appendix specifies the sampling scheme, numerical formats, software environment, and aggregation procedure; the accompanying code instantiates the complete grids\.

#### Software and execution framework\.

The local central\-processing\-unit \(CPU\) runs used Python 3\.11\.15 with NumPy 2\.0\.2, SciPy 1\.13\.1, Pandas 2\.3\.3, PyArrow 21\.0\.0, and Matplotlib 3\.9\.4\. The graphics\-processing\-unit \(GPU\) runs used Python 3\.12\.3, PyTorch 2\.13\.0 with CUDA 13\.0, TorchAO 0\.18\.0, Transformers 5\.16\.1, Datasets 5\.0\.1, and Hugging Face Hub 1\.29\.0\. Thefinitebit\_exp0\.1\.0 package defines the scientific grids in Python\. PyTorch automatic differentiation supplies the Stage A and Stage V gradients in the mini\-batch implementation\. The weighted\-average FP64 reference implements the same explicit scalar gradient on its theorem\-specified active set\. Independent worker processes run the two\-stage training jobs\. Pythia models are loaded once per model–format bundle and evaluated in inference mode\. CPU experiments ran on an Apple M5 system; GPU experiments used an NVIDIA RTX PRO 6000 Blackwell Server Edition\. Raw seed\- and item\-level measurements are stored in Parquet files\.

#### Pythia items and operation count\.

Tokenization is performed once per model tokenizer on the raw WikiText\-103 test text, after which contiguous 2,049\-token windows provide 2,048 next\-token losses per item\. All numerical formats receive identical item indices\. The analytic operation count includes the query, key, value, and output projections; the two attention matrix products; and both feed\-forward products\.

#### Reproducible result generation\.

Empirical curves are generated from seed\- or item\-level records using the uncertainty procedure specified in each caption\.

The data–memory study crosses six energy–dimension profiles, seven model cutoffs from 16 to 1,024, and three normalized data and state budgets\. It evaluates population excess CE over each finite source, including coordinates predicted with zero logit\. The mini\-batch two\-stage study additionally uses the analytic tail proxy described in[AppendixR](https://arxiv.org/html/2609.13500#A18); the fixed\-source weighted\-SGD study measures its omitted\-coordinate loss directly\. The pretrained\-model comparisons use identical items for every format, including 96 paired WikiText\-103 windows per model–format configuration\.

### Q\.1Additional coupling, robustness, and pretrained\-model experiments

The additional experiments comprise a dyadic coupling test with 16 paired seeds, a two\-stage study with eight paired seeds, robustness tests for three departures from exact power laws, and six Pythia sizes under six deployment formats on two datasets\. The two\-stage learner differentiates the Bernoulli loss through the attention\-weighted network output and updates the stored value parameters; the direct finite\-sample parameter construction is included as a separate likelihood\-based comparator\.

#### Six\-scale weight\-only post\-training quantization\.

All Pythia models use the fixedstep143000revision\. WikiText\-103 uses 96 common non\-overlapping 2,048\-token windows, and LAMBADA uses 512 common examples with last\-token cross\-entropy\. The formats are bfloat16 \(BF16\), 8\-bit integer \(INT8\) with one tensor scale, INT8 with one scale per output channel, 4\-bit integer \(INT4\) round\-to\-nearest with groups of 128 weights, and half\-quadratic quantization \(HQQ\) for INT4 groups of 128 or 32 weights\. Every reported difference is paired with the BF16 result on the same item\. Eager attention receives BF16 scores and returns FP32 softmax probabilities in the measured software stack\. The evaluation microbatch is one for both datasets\.

The item\-level records retain model, format, dataset, item index, cross\-entropy, represented weight error, quantized\-parameter fraction, and output\-logit divergence\. Appendix[P](https://arxiv.org/html/2609.13500#A16)reports the detailed figures and tables\.

## Appendix RJoint Resource Evaluation

#### Fixed\-source comparisons\.

We vary data and value\-state budgets while holding the generating distribution fixed\. The same\-marginal experiment samples the infinite dyadic source atχ∈\{1,2\}\\chi\\in\\\{1,2\\\},γ=1\\gamma=1, with a common normalization for both pairings\. A sparse clipped Bernoulli estimator uses unbounded storage, isolating the data exponent\. Unobserved coordinates contribute their actual probability mass times the zero\-prediction excess cross\-entropy\. The seven sample budgets are28,210,…,2202^\{8\},2^\{10\},\\ldots,2^\{20\}; the primary slope uses the last four budgets for every seed and pairing\.

#### The theorem’s learning algorithm\.

For\(χ,γ\)=\(0\.5,0\),\(0\.5,0\.5\),\(1,0\),\(1,0\.5\)\(\\chi,\\gamma\)=\(0\.5,0\),\(0\.5,0\.5\),\(1,0\),\(1,0\.5\), the source contains 256 energy groups, each withdj=⌈jγ⌉d\_\{j\}=\\lceil j^\{\\gamma\}\\rceilscalar coordinates\. This source size is distinct from the number of sampled regeneration blocks\. We compareK=8,32K=8,32candidates with eight independent seeds\. At eachn∈\{28,210,…,216\}n\\in\\\{2^\{8\},2^\{10\},\\ldots,2^\{16\}\\\}, we restart the projected, preconditioned SGD algorithm of[TheoremC\.5](https://arxiv.org/html/2609.13500#A3.Thmtheorem5)from zero, using its active set, step sizes and weighted average\. We implement the numerical reference in FP64\. Calibration usesSA=⌈4​K​\[log⁡\(e​K\)\+n\]⌉S\_\{A\}=\\lceil 4K\[\\log\(eK\)\+\\sqrt\{n\}\]\\rceilindependent blocks\. Each fitted state is encoded atBV=8,16,32,64,128,256B\_\{V\}=8,16,32,64,128,256bits using the theorem’s midpoint scalar format\. Inactive fitted coordinates are zero before encoding; positive\-width midpoint decoding also acts on those zero entries\. An oracle\-coordinate clipped estimator uses the same labeled blocks with the selected coordinate revealed\. We report both value and routing bits, calibration and prediction samples, and the losses on 65,536 shared independent test contexts\. Oracle and learned routes, before and after quantization, define signed counterfactual differences on these identical contexts\.

#### Additional training and deployed storage\.

Starting from the Pythia 70M, 160M, 410M, 1B, 1\.4B, and 2\.8B checkpoints at step143000, each of eight seeds follows one continued\-training trajectory with checkpoints after220,222,224,2262^\{20\},2^\{22\},2^\{24\},2^\{26\}additional WikiText\-103 tokens\. Training uses 2048\-token contexts, an effective batch of 32,768 tokens, FP32 parameters and optimizer states, and BF16 computation\. AdamW uses learning rate10−510^\{\-5\},β=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), weight decay 0\.1, gradient clipping at 1, and an eight\-step warm\-up followed by a constant learning rate\. Deployment uses BF16 or group\-128 symmetric 2/3/4/6/8\-bit round\-to\-nearest linear weights\. Integer payloads, scales, padding, retained BF16 parameters and metadata are all counted\. Evaluation uses 96 fixed test windows and 96 fixed validation windows\.

#### Prediction on held\-out resource configurations\.

We fit data\-only, storage\-only, additive and interacting response models on the first three training budgets at 2/4/8\-bit and BF16\. The 3/6\-bit formats test storage interpolation, while the largest training budget tests data extrapolation\. Fits use validation losses; test\-window losses remain untouched until evaluation\. Comparisons use paired training\-seed uncertainty; the common initial pretrained checkpoint is evaluated once and its uncertainty uses windows\. Lett=T/220t=T/2^\{20\}denote additional tokens in units of the smallest budget, and letssbe deployment\-package bytes divided by the BF16 package size\. Writingx=t−αx=t^\{\-\\alpha\}andy=s−β−1y=s^\{\-\\beta\}\-1, the four fitted CE models are

c\+a​x⏟data only,c\+d​y⏟storage only,c\+a​x\+d​y⏟additive,c\+a​x\+d​y\+h​x​y⏟interaction\.\\underbrace\{c\+ax\}\_\{\\text\{data only\}\},\\qquad\\underbrace\{c\+dy\}\_\{\\text\{storage only\}\},\\qquad\\underbrace\{c\+ax\+dy\}\_\{\\text\{additive\}\},\\qquad\\underbrace\{c\+ax\+dy\+hxy\}\_\{\\text\{interaction\}\}\.We fit each model scale separately by least squares on validation CE, withc,a,d∈\[0,20\]c,a,d\\in\[0,20\],α,β∈\[0\.01,4\]\\alpha,\\beta\\in\[0\.01,4\], andh∈\[−20,20\]h\\in\[\-20,20\]\. The fitting procedure requires convergence and calibration predictions in the pre\-specified range\[0,20\]\[0,20\]nats; held\-out predictions are checked against the same range\. After observing the large 2\-bit losses at 70M and 160M, we also analyze 4/8\-bit and BF16 calibration, with 6\-bit interpolation and 4/6/8\-bit/BF16 extrapolation\. This sensitivity analysis uses the same response functions and parameter bounds\. Root\-mean\-square error \(RMSE\) compares predicted and measured test CE across the held\-out resource configurations, after averaging the 96 windows within each configuration\.

#### Route\-adjusted and complete\-source losses\.

LetLlr,q,Llr,u,Lor,uL\_\{\\mathrm\{lr\},q\},L\_\{\\mathrm\{lr\},u\},L\_\{\\mathrm\{or\},u\}denote finite\-source excess CE for learned\-route quantized, learned\-route unquantized, and known\-route unquantized predictions\. All subtract the same source Bayes loss\. The route\-adjusted analysis definesE=Llr,q\+M−χE=L\_\{\\mathrm\{lr\},q\}\+M^\{\-\\chi\},R\+=\(Llr,u−Lor,u\)\+R\_\{\+\}=\(L\_\{\\mathrm\{lr\},u\}\-L\_\{\\mathrm\{or\},u\}\)\_\{\+\}, andF\+=\(Llr,q−Llr,u\)\+F\_\{\+\}=\(L\_\{\\mathrm\{lr\},q\}\-L\_\{\\mathrm\{lr\},u\}\)\_\{\+\}\. Its spectrum comparison usesE−R\+E\-R\_\{\+\}; its component sum is an upper envelope ofEE\. HereMMis the energy\-group cutoff andM−χM^\{\-\\chi\}is an analytic tail proxy\. Our fixed\-source study instead evaluates the generating distribution without adding that proxy and retains all four counterfactual losses\. The oracle loss splits over actual held\-out targets inside and outsideAnA\_\{n\}; adding the signed differencesLlr,u−Lor,uL\_\{\\mathrm\{lr\},u\}\-L\_\{\\mathrm\{or\},u\}andLlr,q−Llr,uL\_\{\\mathrm\{lr\},q\}\-L\_\{\\mathrm\{lr\},u\}gives exactly the complete\-source excess lossLlr,qL\_\{\\mathrm\{lr\},q\}\. The tail term is the actual zero\-prediction KL on omitted targets\. This identity holds per test context, before any averaging, by[PropositionS\.2](https://arxiv.org/html/2609.13500#A19.Thmtheorem2)\.

#### Empirical and known\-energy allocation\.

The robustness study’s adaptive allocator uses the count\-corrected score

si=\[pi​θ^i2−pimax⁡\{1,Ni\}\]\+,s\_\{i\}=\\left\[p\_\{i\}\\widehat\{\\theta\}\_\{i\}^\{2\}\-\\frac\{p\_\{i\}\}\{\\max\\\{1,N\_\{i\}\\\}\}\\right\]\_\{\+\},whereNiN\_\{i\}is the observed count andθ^i\\widehat\{\\theta\}\_\{i\}the bounded Bernoulli maximum\-likelihood estimate\. Both this rule and the known\-energy reference water\-fill∑isi​4−bi\\sum\_\{i\}s\_\{i\}4^\{\-b\_\{i\}\}, floor widths, and assign remaining bits by marginal gains; the reference substitutessi=qis\_\{i\}=q\_\{i\}\. The empirical subtraction prioritizes resolved coordinates, whereas the known\-energy reference minimizes the true\-energy distortion objective\. These distinct objectives allow different finite\-sample CE even at the same state budget\. The unknown\-exponent schedule in[Theorem5\.1](https://arxiv.org/html/2609.13500#S5.Thmtheorem1)is a separate, deterministic construction\.

Figure 10:Candidate\-count replication under route\-adjusted loss\.\(a\) Direct finite\-sample construction; \(b\) mini\-batch two\-stage implementation\. Horizontal and vertical axes showE−R\+E\-R\_\{\+\}atK=32K=32andK=8K=8, respectively, using identical spectral bins\. Legends give\(χ,γ\)\(\\chi,\\gamma\); the dashed identity line is deterministic\. Intervals on both axes are 95% BCa intervals over eight paired seeds\.Figure 11:Unbinned risk\-to\-spectrum ratios on fixed sources\.Panels \(a\)–\(d\) use the four profiles of[Figure2](https://arxiv.org/html/2609.13500#S7.F2), atK=32K=32\. The horizontal axis isGμG\_\{\\mu\}; the vertical axis is measured excess CE divided byGμG\_\{\\mu\}\. Each point represents a distinct sample–state budget pair; lines connect budget pairs ordered byGμG\_\{\\mu\}, rather than a fitted response\. Blue/orange denote weighted SGD and the oracle\-coordinate estimator\. Bands retain the raw 95% BCa intervals over eight paired seeds\.

## Appendix STotal Resources and Fixed\-Source Evaluation

The following consequence accounts jointly for the value\-learning and routing stages\. Writeq\(r\)q\_\{\(r\)\}for the decreasing coordinate energies andξ\\xifor their decay exponent\.

###### Proposition S\.1\(Total sample and learned\-state requirements\)\.

Fix an integerK≥1K\\geq 1,R\>0R\>0, and0<η≤10<\\eta\\leq 1\. Use the iid\-candidate source, public frequencies and calibration supervision of[TheoremC\.5](https://arxiv.org/html/2609.13500#A3.Thmtheorem5), with unit midpoint formats\. Supposec​r−\(1\+ξ\)≤q\(r\)≤C​r−\(1\+ξ\)cr^\{\-\(1\+\\xi\)\}\\leq q\_\{\(r\)\}\\leq Cr^\{\-\(1\+\\xi\)\}, whereξ\>0\\xi\>0and0<c≤C<∞0<c\\leq C<\\inftyare fixed\. LetCL≥1C\_\{L\}\\geq 1be a constant in its four\-term risk upper bound\. Set

CD=2\+C/ξ,a=max⁡\{1,\(1\+ξ\)/\(2​ln⁡2\)\},CB=C⁡\(1\+1/ξ\)​\(2​a\)ξ\.C\_\{D\}=2\+C/\\xi,\\quad a=\\max\\\{1,\(1\+\\xi\)/\(2\\ln 2\)\\\},\\quad C\_\{B\}=C\(1\+1/\\xi\)\(2a\)^\{\\xi\}\.LetCA​\(η\)C\_\{A\}\(\\eta\)be an upper\-bound constant in the routing hitting\-time bound of[PropositionC\.4](https://arxiv.org/html/2609.13500#A3.Thmtheorem4)for thresholds at most1/161/16\. For0<ε≤10<\\varepsilon\\leq 1, choose

h\\displaystyle h=min⁡\{1/16,ε/\(4​CL\)\},\\displaystyle=\\min\\\{1/16,\\varepsilon/\(4C\_\{L\}\)\\\},n\\displaystyle n=max⁡\{1,⌈\(4​CL​CD/ε\)\(1\+ξ\)/ξ⌉\},\\displaystyle=\\max\\\{1,\\lceil\(4C\_\{L\}C\_\{D\}/\\varepsilon\)^\{\(1\+\\xi\)/\\xi\}\\rceil\\\},BV\\displaystyle B\_\{V\}=max⁡\{1,⌈\(4​CL​CB/ε\)1/ξ⌉\}\.\\displaystyle=\\max\\\{1,\\lceil\(4C\_\{L\}C\_\{B\}/\\varepsilon\)^\{1/\\xi\}\\rceil\\\}\.ForK≥2K\\geq 2, take

SA\\displaystyle S\_\{A\}=⌈CA\(η\)K\{h−1/2\+ln\(eK\)\}⌉,\\displaystyle=\\lceil C\_\{A\}\(\\eta\)K\\\{h^\{\-1/2\}\+\\ln\(eK\)\\\}\\rceil,BA\\displaystyle B\_\{A\}=K⁡\{⌈log2⁡\(SA\+1\)⌉\+⌈log2⁡\(K\+1\)⌉\};\\displaystyle=K\\\{\\lceil\\log\_\{2\}\(S\_\{A\}\+1\)\\rceil\+\\lceil\\log\_\{2\}\(K\+1\)\\rceil\\\};forK=1K=1, takeSA=BA=0S\_\{A\}=B\_\{A\}=0\. If deployment arithmetic has mean squared logit error at mostε/\(4​CL\)\\varepsilon/\(4C\_\{L\}\), the weighted\-SGD learner has expected excess risk at mostε\\varepsilon, usesN=n\+SAN=n\+S\_\{A\}blocks and retains at mostB=BV\+BAB=B\_\{V\}\+B\_\{A\}learned bits\. At fixed source and routing constants,

N=O\(ε−\(1\+ξ\)/ξ\),B=O\(ε−1/ξ\),SA/n→0,BA/BV→0\.N=O\(\\varepsilon^\{\-\(1\+\\xi\)/\\xi\}\),\\quad B=O\(\\varepsilon^\{\-1/\\xi\}\),\\qquad S\_\{A\}/n\\to 0,\\quad B\_\{A\}/B\_\{V\}\\to 0\.Forχ\>0\\chi\>0,γ≥0\\gamma\\geq 0, block energiesaj≍j−\(1\+χ\)a\_\{j\}\\asymp j^\{\-\(1\+\\chi\)\}, dimensionsdj≍jγd\_\{j\}\\asymp j^\{\\gamma\}and equal coordinate energiesqj,ℓ=aj/djq\_\{j,\\ell\}=a\_\{j\}/d\_\{j\}within each block, the coordinate exponent isξ=χ/\(1\+γ\)\\xi=\\chi/\(1\+\\gamma\)\.

###### Proof\.

For any integerm≥1m\\geq 1, the energy envelope and integral comparison give

Φμ​\(n−1\)≤m/n\+C​∑r\>mr−1−ξ≤m/n\+\(C/ξ\)​m−ξ\.\\Phi\_\{\\mu\}\(n^\{\-1\}\)\\leq m/n\+C\\sum\_\{r\>m\}r^\{\-1\-\\xi\}\\leq m/n\+\(C/\\xi\)m^\{\-\\xi\}\.Choosem=⌈n1/\(1\+ξ\)⌉m=\\lceil n^\{1/\(1\+\\xi\)\}\\rceil\. Sincen1/\(1\+ξ\)≤m≤2​n1/\(1\+ξ\)n^\{1/\(1\+\\xi\)\}\\leq m\\leq 2n^\{1/\(1\+\\xi\)\}, this is at mostCDn−ξ/\(1\+ξ\)C\_\{D\}n^\{\-\\xi/\(1\+\\xi\)\}, includingn=1n=1\.

For a storage budgetb≥2​ab\\geq 2a, putm=⌊b/a⌋m=\\lfloor b/a\\rfloorand allocatebr=\(1\+ξ\)​log2⁡\(m/r\)/2b\_\{r\}=\(1\+\\xi\)\\log\_\{2\}\(m/r\)/2for1≤r≤m1\\leq r\\leq m, with zero widths thereafter\. The inequality

∑r=1mln⁡\(m/r\)≤∫0mln⁡\(m/x\)​𝑑x=m\\sum\_\{r=1\}^\{m\}\\ln\(m/r\)\\leq\\int\_\{0\}^\{m\}\\ln\(m/x\)\\,dx=mshows that total width is at most\(1\+ξ\)​m/\(2​ln⁡2\)≤a​m≤b\(1\+\\xi\)m/\(2\\ln 2\)\\leq am\\leq b\. Each retained distortion term is at mostC​m−1−ξCm^\{\-1\-\\xi\}, and the tail is at most\(C/ξ\)​m−ξ\(C/\\xi\)m^\{\-\\xi\}\. Sincem≥b/\(2​a\)m\\geq b/\(2a\),

Dq​\(b\)≤C⁡\(1\+1/ξ\)​m−ξ≤CB​b−ξ\.D\_\{q\}\(b\)\\leq C\(1\+1/\\xi\)m^\{\-\\xi\}\\leq C\_\{B\}b^\{\-\\xi\}\.For1≤b<2​a1\\leq b<2a, zero allocation has distortion at most∑rq\(r\)≤C⁡\(1\+1/ξ\)≤CB​b−ξ\\sum\_\{r\}q\_\{\(r\)\}\\leq C\(1\+1/\\xi\)\\leq C\_\{B\}b^\{\-\\xi\}, proving the same bound for allb≥1b\\geq 1\. Summability controls every countable tail\. The theorem’s scalar implementation floors public continuous widths; this cannot exceed the bit budget and multiplies each distortion term by at most four\. Its constant is already included inCLC\_\{L\}\. A public allocation needs no additional learned index code\.

The ceilings inn,BVn,B\_\{V\}can only decrease their nonincreasing error functionals, so each is at mostε/\(4​CL\)\\varepsilon/\(4C\_\{L\}\)\. ForK≥2K\\geq 2, the positive thresholdh≤1/16h\\leq 1/16lies in the routing theorem’s range\. The prescribedSAS\_\{A\}exceeds its hitting\-time bound\. Margin increases with row visits, and counts couple monotonically as calibration blocks are added; leakage therefore remains belowh≤ε/\(4​CL\)h\\leq\\varepsilon/\(4C\_\{L\}\)\. ForK=1K=1, leakage is zero\. The arithmetic premise bounds the fourth term\. Substitution in[EquationC\.5](https://arxiv.org/html/2609.13500#A3.E5)now gives risk at mostCL⋅4​ε/\(4​CL\)=εC\_\{L\}\\cdot 4\\varepsilon/\(4C\_\{L\}\)=\\varepsilon\.

The independent stages consume distinct blocks, totalingn\+SAn\+S\_\{A\}\. Each routing row is reconstructed from its count in\{0,…,SA\}\\\{0,\\ldots,S\_\{A\}\\\}and its destination or unseen symbol\. The displayed integer\-width fields encode all these possibilities\. Concatenating theirBAB\_\{A\}\-bit code with the value code gives at most2BA\+BV2^\{B\_\{A\}\+B\_\{V\}\}retained states\. Numerical reconstruction is charged to the arithmetic term\.

For sufficiently smallε\\varepsilon,h=ε/\(4​CL\)h=\\varepsilon/\(4C\_\{L\}\)\. The explicit resource choices then yield

SA/n\\displaystyle S\_\{A\}/n=OK​\(ε1/2\+1/ξ\+ε1\+1/ξ\),\\displaystyle=O\_\{K\}\(\\varepsilon^\{1/2\+1/\\xi\}\+\\varepsilon^\{1\+1/\\xi\}\),BA/BV\\displaystyle B\_\{A\}/B\_\{V\}=OK​\(ε1/ξ​ln⁡\(1/ε\)\)\.\\displaystyle=O\_\{K\}\(\\varepsilon^\{1/\\xi\}\\ln\(1/\\varepsilon\)\)\.Both tend to zero sinceξ\>0\\xi\>0\. Adding the routing costs gives the two total\-resource orders\.

Finally, every coordinate in blockjjhas energy comparable toj−\(1\+χ\+γ\)j^\{\-\(1\+\\chi\+\\gamma\)\}, uniformly over itsdj≍jγd\_\{j\}\\asymp j^\{\\gamma\}coordinates\. Upper and lower threshold comparisons therefore give

\#\{qi≥t\}≍∑j≤c0t−1/\(1\+χ\+γ\)jγ≍t−\(1\+γ\)/\(1\+χ\+γ\),\\\#\\\{q\_\{i\}\\geq t\\\}\\asymp\\sum\_\{j\\leq c\_\{0\}t^\{\-1/\(1\+\\chi\+\\gamma\)\}\}j^\{\\gamma\}\\asymp t^\{\-\(1\+\\gamma\)/\(1\+\\chi\+\\gamma\)\},with separate positivec0c\_\{0\}for the upper and lower comparisons\. Inversion provesq\(r\)≍r−\(1\+χ\+γ\)/\(1\+γ\)q\_\{\(r\)\}\\asymp r^\{\-\(1\+\\chi\+\\gamma\)/\(1\+\\gamma\)\}and the assertedξ\\xi\. The argument allows bounded nonmonotonicity in the original block labels\. ∎

###### Proposition S\.2\(Fixed\-source loss and paired counterfactuals\)\.

Letppbe a fixed probability distribution on a finite or countable coordinate set,\|θi\|≤Δi≤R\|\\theta\_\{i\}\|\\leq\\Delta\_\{i\}\\leq R,qi=pi​Δi2q\_\{i\}=p\_\{i\}\\Delta\_\{i\}^\{2\}, andd\(θ,z\)=DKL\(Ber\(σ\(θ\)\)∥Ber\(σ\(z\)\)\)d\(\\theta,z\)=D\_\{\\rm KL\}\(\\operatorname\{Ber\}\(\\sigma\(\\theta\)\)\\\|\\operatorname\{Ber\}\(\\sigma\(z\)\)\)\. Conditional on a fitted table\|wi\|≤R\|w\_\{i\}\|\\leq Rsupported onSS, a fresh oracle\-routed example has excess risk

L⁡\(w,oracle\)\\displaystyle L\(w,\\mathrm\{oracle\}\)=∑i∈Spi​d​\(θi,wi\)\+∑i∉Spi​d​\(θi,0\),\\displaystyle=\\sum\_\{i\\in S\}p\_\{i\}d\(\\theta\_\{i\},w\_\{i\}\)\+\\sum\_\{i\\notin S\}p\_\{i\}d\(\\theta\_\{i\},0\),0≤∑i∉Spi​d​\(θi,0\)\\displaystyle 0\\leq\\sum\_\{i\\notin S\}p\_\{i\}d\(\\theta\_\{i\},0\)≤18​∑i∉Sqi\.\\displaystyle\\leq\\tfrac\{1\}\{8\}\\sum\_\{i\\notin S\}q\_\{i\}\.If all\|θi\|=a\|\\theta\_\{i\}\|=a, the omitted term equalsp⁡\(Sc\)​d​\(a,0\)p\(S^\{c\}\)d\(a,0\), also for a data\-dependentSS\.

On the same full candidate context, letLr​vL\_\{rv\}be excess loss with router∈\{0,1\}r\\in\\\{0,1\\\}\(oracle, learned\) and value statev∈\{0,1\}v\\in\\\{0,1\\\}\(unquantized, quantized\), with all four predicted logits in\[−R,R\]\[\-R,R\]\. Pointwise and under every common empirical or population average,

L11\\displaystyle L\_\{11\}=L00\+\(L10−L00\)\+\(L11−L10\)\\displaystyle=L\_\{00\}\+\(L\_\{10\}\-L\_\{00\}\)\+\(L\_\{11\}\-L\_\{10\}\)=L00\+\(L01−L00\)\+\(L11−L01\)\.\\displaystyle=L\_\{00\}\+\(L\_\{01\}\-L\_\{00\}\)\+\(L\_\{11\}\-L\_\{01\}\)\.The two format contrasts differ byL11−L10−L01\+L00L\_\{11\}\-L\_\{10\}\-L\_\{01\}\+L\_\{00\}; individual contrasts may be negative\. If two predictions in\[−R,R\]\[\-R,R\]agree outside an eventAA, their expected CE differs by at most2​R​Pr⁡\(A\)2R\\Pr\(A\)\. IfAAis that some candidate lies inTT, thenPr⁡\(A\)≤K​p​\(T\)\\Pr\(A\)\\leq Kp\(T\)\.

###### Proof\.

For target coordinateii, conditional expected CE minus Bayes CE is exactlyd⁡\(θi,z\)d\(\\theta\_\{i\},z\)\. Conditional on training, a fresh target retains lawpp, giving∑ipi​d​\(θi,wi\)\\sum\_\{i\}p\_\{i\}d\(\\theta\_\{i\},w\_\{i\}\)\. Nonnegative terms permit countable summation by monotone convergence\. Splitting atSS, where the fitted table is zero on its complement, proves the oracle identity\.

PutA⁡\(z\)=ln⁡\(1\+ez\)A\(z\)=\\ln\(1\+e^\{z\}\)\. Direct substitution givesd⁡\(θ,z\)=A⁡\(z\)−A⁡\(θ\)−A′​\(θ\)​\(z−θ\)d\(\\theta,z\)=A\(z\)\-A\(\\theta\)\-A^\{\\prime\}\(\\theta\)\(z\-\\theta\)\. Since0<A′′​\(z\)=σ⁡\(z\)​\(1−σ⁡\(z\)\)≤1/40<A^\{\\prime\\prime\}\(z\)=\\sigma\(z\)\(1\-\\sigma\(z\)\)\\leq 1/4, Taylor’s integral remainder implies0≤d⁡\(θ,0\)≤θ2/80\\leq d\(\\theta,0\)\\leq\\theta^\{2\}/8\. Multiply bypip\_\{i\}, useθi2≤Δi2\\theta\_\{i\}^\{2\}\\leq\\Delta\_\{i\}^\{2\}, and sum to obtain the tail bound\. Exchanging Bernoulli outcomes givesd⁡\(−a,0\)=d⁡\(a,0\)d\(\-a,0\)=d\(a,0\), proving the constant\-magnitude formula conditional on each fitted support\.

For learned routing,∑saQ​s​wCs\\sum\_\{s\}a\_\{Qs\}w\_\{C\_\{s\}\}can be nonzero even when the target is outsideSS, because a distractor may lie inSS\. The four losses therefore evaluate the full context with a common target law\. Cancelling intermediate terms proves both identities pointwise; linearity preserves them under averaging\. Subtracting the two format contrasts gives their stated interaction\. For a concrete sign example, the one\-bit midpoint format on\[−1,1\]\[\-1,1\]sendsw=0\.1w=0\.1to0\.50\.5\. Atθ=0\.5\\theta=0\.5, the contrast is−d⁡\(0\.5,0\.1\)<0\-d\(0\.5,0\.1\)<0; atθ=0\.1\\theta=0\.1, it isd⁡\(0\.1,0\.5\)\>0d\(0\.1,0\.5\)\>0\.

Finally, per\-label CEℓy​\(z\)=A⁡\(z\)−y​z\\ell\_\{y\}\(z\)=A\(z\)\-yzhas derivativeσ⁡\(z\)−y\\sigma\(z\)\-yof absolute value at most one\. Hence\|ℓy​\(z\)−ℓy​\(z′\)\|≤\|z−z′\|≤2​R\|\\ell\_\{y\}\(z\)\-\\ell\_\{y\}\(z^\{\\prime\}\)\|\\leq\|z\-z^\{\\prime\}\|\\leq 2R\. Multiplying by the indicator ofAAand averaging proves the approximation bound, also after subtracting the common Bayes risk\. The union bound givesPr⁡\(A\)≤∑s=1KPr⁡\(Cs∈T\)=K​p​\(T\)\\Pr\(A\)\\leq\\sum\_\{s=1\}^\{K\}\\Pr\(C\_\{s\}\\in T\)=Kp\(T\); it needs only the candidate marginals, not independence\. ∎