How Many Thoughts Can a Vector Hold? The Capacity of Reasoning by Superposition

# How Many Thoughts Can a Vector Hold?The Capacity of Reasoning by Superposition
Source: [https://arxiv.org/html/2609.13747](https://arxiv.org/html/2609.13747)
Chang Liu Zhongguancun AcademyBeijing, Chinaliuchang@bza\.edu\.cnJingwen Fu Zhongguancun AcademyBeijing, Chinajwfu99@gmail\.com††thanks:Corresponding author\.

###### Abstract

Large language models solve difficult problems by carrying intermediate computation across many reasoning steps\. Conventional chain\-of\-thought writes this computation as tokens; recent continuous and recurrent approaches instead move part of it into fixed\-dimensional latent states, where one thought can superpose several alternatives\. This shift raises a basic design question: as reasoning proceeds, what should a continuous thought preserve? The natural strategy is to discard earlier computation and keep only the current frontier: storing more objects appears to dilute the state and waste its limited capacity\. We show that this intuition can be exactly backwards\. When the same downstream computation is applied to both states, a cumulative superposition that retains the reached history can require less representation dimension than a frontier superposition that stores only the current alternatives\. At a fixed hidden width, this advantage lets a latent reasoner retain more usable evidence, distinguish more possible downstream outcomes, and delay the point at which its compressed state becomes unreliable\. The reversal occurs because useful historical components reinforce one another coherently, while unrelated alternatives remain random interference\. This perspective also resolves a practical design question: how should a model weight the memories accumulated inside a latent state when their future use is not yet known? Across the full family of reusable weighted superpositions, concentrating on a few recent or salient objects creates a weakly represented memory that bottlenecks later attention\. Uniform cumulative weighting avoids this failure, and we prove that it is minimax\-optimal for robust future reasoning\. Our results turn superposition from an observed latent phenomenon into a design principle: balanced cumulative memory can make a fixed representational budget support more reliable and reusable computation\.

## 1Introduction

What should a continuous thought preserve as reasoning unfolds? This question is becoming increasingly relevant as language\-model reasoning moves from explicit text into continuous computation\. Continuous chain\-of\-thought feeds hidden states directly back into the model\([Hao et al\., 2024](https://arxiv.org/html/2609.13747#bib.bib6)\); looped and recurrent Transformers repeatedly update a latent workspace\([Xu and Sato, 2024](https://arxiv.org/html/2609.13747#bib.bib12);[Fu et al\., 2026](https://arxiv.org/html/2609.13747#bib.bib13)\); and reasoning by superposition shows that one continuous thought can encode several search alternatives and expand them in parallel\([Zhu et al\., 2025b](https://arxiv.org/html/2609.13747#bib.bib7)\)\. These results establish that superposed latent reasoning is possible\. They leave open a consequential question for model design: among the many superpositions that can represent a computation, which one uses a limited hidden dimension most effectively?

We study three structures that isolate this choice\. A*frontier superposition*stores the alternatives active at the current step, a*cumulative superposition*stores the full reached history, and a general*weighted superposition*distributes representational energy arbitrarily across that history\. All three use the same graph, downstream query, candidate bank, and decoder\. The comparison therefore changes only what the continuous thought preserves, not the task it is asked to solve\.

Ordinary capacity intuition favors frontier superposition: storing more objects in one normalized vector dilutes their individual coefficients and should make them harder to recover\([Elhage et al\., 2022](https://arxiv.org/html/2609.13747#bib.bib11)\)\. We show that this intuition can be exactly backwards for reasoning\. Cumulative superposition can remain usable at a smaller hidden dimension despite storing strictly more objects, because useful historical components reinforce one another coherently while unrelated alternatives remain random interference\. Superposition capacity is therefore governed not by support size alone, but by how the stored information supports future computation\.

This reversal holds for a broad, state\-independent class of downstream queries, which may be continuous, correlated, asymmetric, and identity\-dependent\. The condition captures bounded\-influence computations over accumulated objects, including whole\-set aggregation, additive route objectives, algorithmic execution traces, and intermediate\-step evaluation\([Zaheer et al\., 2017](https://arxiv.org/html/2609.13747#bib.bib23);[Lee et al\., 2019](https://arxiv.org/html/2609.13747#bib.bib24);[Kool et al\., 2019](https://arxiv.org/html/2609.13747#bib.bib25);[Veličković et al\., 2022](https://arxiv.org/html/2609.13747#bib.bib26);[Lightman et al\., 2024](https://arxiv.org/html/2609.13747#bib.bib27)\)\.

Figure 1:Our theory in one picture\. Attention identifies active edge sources and copies their destinations into the next reasoning state\. Frontier superposition overwrites the state and discards earlier reached objects, whereas cumulative superposition preserves them through the residual path\. The retained history supplies coherent evidence to later computation, allowing cumulative superposition to remain usable at a smaller hidden dimension\. For general weighted superpositions, uneven coefficients create a weakest\-object routing bottleneck; uniform cumulative weighting removes this bottleneck\.The comparison also has a direct interpretation inside a Transformer: attention expands the current computation, while residual writing determines whether earlier objects are discarded or preserved\. For general weighted superpositions, concentrating weight on a few objects leaves other memories too weak for later attention to reuse\. When their future use is unknown, uniform cumulative weighting solves the resulting robust design problem:

𝒂cum∈arg⁡min𝒂⁡𝔇⁡\(𝒂,𝒰C\),\\boldsymbol\{a\}\_\{\\mathrm\{cum\}\}\\in\\arg\\min\_\{\\boldsymbol\{a\}\}\\mathfrak\{D\}\(\\boldsymbol\{a\};\\mathcal\{U\}\_\{C\}\),\(1\)where𝔇\\mathfrak\{D\}is the worst\-case end\-to\-end critical dimension\. This suggests a simple principle for latent reasoning: expand the current computation while preserving its history with balanced representational weight\.

#### Contributions\.

Our main contributions are:

1. 1\.Critical\-dimension theory\.We derive an exact finite\-dimensional score law and show that cumulative superposition requires fewer dimensions than frontier superposition for a broad class of nonlocalized queries\. This result shifts the relevant notion of capacity from how many objects are stored to how effectively they support future computation, explaining when retaining more history benefits a fixed\-width reasoner\.
2. 2\.End\-to\-end Transformer mechanism\.We realize both superpositions through the same attention–residual construction and identify the weakest retained coefficient as the routing bottleneck\. This connects the abstract dimension advantage to multi\-step Transformer computation and provides a concrete diagnostic for failures of reusable latent memory\.
3. 3\.A minimax design principle\.Across reusable weighted superpositions, uniform cumulative weighting is sharply near\-optimal for fixed nonlocalized query classes and exactly minimax when future queries are unknown\. This provides a robust rule for allocating latent memory, supported by width\-scaling and token\-level experiments in both lightweight and GPT\-2 models\.

## 2Related Work

#### Explicit and continuous reasoning\.

Chain\-of\-thought externalizes intermediate computation as tokens\([Wei et al\., 2022](https://arxiv.org/html/2609.13747#bib.bib2);[Merrill and Sabharwal, 2024](https://arxiv.org/html/2609.13747#bib.bib10)\), whereas COCONUT feeds continuous thoughts back into the model\([Hao et al\., 2024](https://arxiv.org/html/2609.13747#bib.bib6)\)\. Recent theory shows that continuous tokens can maintain and explore several reasoning traces in parallel\([Zhu et al\., 2025b](https://arxiv.org/html/2609.13747#bib.bib7);[Gozeten et al\., 2025](https://arxiv.org/html/2609.13747#bib.bib9)\); subsequent work studies how such superpositions emerge during training\([Zhu et al\., 2025a](https://arxiv.org/html/2609.13747#bib.bib8)\)\. Our question is different from existence or learnability: among frontier, cumulative, and weighted states, which representation requires the least embedding dimension for the same later computation?

#### Capacity of superposition\.

Feature\-superposition analyses explain how non\-orthogonal features share a low\-dimensional space and how interference depends on feature sparsity, importance, and polysemanticity\([Elhage et al\., 2022](https://arxiv.org/html/2609.13747#bib.bib11);[Foote, 2024](https://arxiv.org/html/2609.13747#bib.bib20);[Gupta and Kumar, 2025](https://arxiv.org/html/2609.13747#bib.bib21);[Bereska et al\., 2025](https://arxiv.org/html/2609.13747#bib.bib22)\)\. These analyses primarily concern recovery or interpretation of individual features\. Our result isolates a reasoning\-specific reversal: when later computation aggregates several stored objects, additional components can contribute coherent signal, making query\-aligned margin—rather than support size—the operative capacity\.

#### Recurrent computation and latent memory\.

Universal and looped Transformers reuse computation depth while maintaining a latent workspace\([Dehghani et al\., 2019](https://arxiv.org/html/2609.13747#bib.bib3);[Giannou et al\., 2023](https://arxiv.org/html/2609.13747#bib.bib4);[Xu and Sato, 2024](https://arxiv.org/html/2609.13747#bib.bib12);[Fu et al\., 2026](https://arxiv.org/html/2609.13747#bib.bib13)\), while recurrent\-memory and buffer\-based models carry information through dedicated latent tokens\([Bulatov et al\., 2022](https://arxiv.org/html/2609.13747#bib.bib5);[Yang et al\., 2024](https://arxiv.org/html/2609.13747#bib.bib16);[Lin et al\., 2026](https://arxiv.org/html/2609.13747#bib.bib17)\)\. Related work studies the difficulty of multi\-hop reasoning in long contexts and without explicit intermediate steps\([Li et al\., 2024](https://arxiv.org/html/2609.13747#bib.bib15);[Yao et al\., 2025](https://arxiv.org/html/2609.13747#bib.bib14)\), as well as limits and representations of hidden reasoning\([Brown\-Cohen et al\., 2026](https://arxiv.org/html/2609.13747#bib.bib18);[Hu et al\., 2024](https://arxiv.org/html/2609.13747#bib.bib19)\)\. Our dimension laws separate two roles of this latent budget: storing atomic states and keeping a compact state aligned with computations that must reuse them later\.

## 3Setup: Three Superposition Structures

LetG=\(V,E\)G=\(V,E\)be a directed graph withn=\|V\|n=\|V\|and rootrr\. Coordinates of vectors inℝn\\mathbb\{R\}^\{n\}are indexed by verticesv∈Vv\\in V\(also called objects or nodes\)\. At a fixed depthcc,

F:=\{v:dist⁡\(r,v\)=c\},R:=\{v:dist⁡\(r,v\)≤c\},F:=\\\{v:\\operatorname\{dist\}\(r,v\)=c\\\},\\qquad R:=\\\{v:\\operatorname\{dist\}\(r,v\)\\leq c\\\},\(2\)with\|F\|=k\|F\|=k,\|R\|=m\|R\|=m,F⊆RF\\subseteq R, and1≤k≤m1\\leq k\\leq m\. WriteDDfor the graph depth and, at layercc,

Rc:=\{v:dist⁡\(r,v\)≤c\},Fc:=\{v:dist⁡\(r,v\)=c\},Nc:=Rc\+1∖Rc,R\_\{c\}:=\\\{v:\\operatorname\{dist\}\(r,v\)\\leq c\\\},\\qquad F\_\{c\}:=\\\{v:\\operatorname\{dist\}\(r,v\)=c\\\},\\qquad N\_\{c\}:=R\_\{c\+1\}\\setminus R\_\{c\},\(3\)withrc:=\|Nc\|r\_\{c\}:=\|N\_\{c\}\|\. Write𝟏S\\mathbf\{1\}\_\{S\}for the00–11indicator ofS⊆VS\\subseteq Vandev:=𝟏\{v\}e\_\{v\}:=\\mathbf\{1\}\_\{\\\{v\\\}\}\.

Each vertex has an independent embeddinggv∼N⁡\(0,Id/d\)g\_\{v\}\\sim N\(0,I\_\{d\}/d\)\. Equivalently, the columns ofG∈ℝd×nG\\in\\mathbb\{R\}^\{d\\times n\}are i\.i\.d\.N⁡\(0,1/d\)N\(0,1/d\)\. Fora∈ℝna\\in\\mathbb\{R\}^\{n\},

Xa:=G​a=∑v∈Vav​gv,qa:=Xa‖Xa‖\.X\_\{a\}:=Ga=\\sum\_\{v\\in V\}a\_\{v\}g\_\{v\},\\qquad q\_\{a\}:=\\frac\{X\_\{a\}\}\{\\left\\\|X\_\{a\}\\right\\\|\}\.\(4\)Sinceqaq\_\{a\}is invariant to positive scaling we take‖a‖2=1\\left\\\|a\\right\\\|\_\{2\}=1\. Writeχd\\chi\_\{d\}\(alsoCdC\_\{d\}\) for a chi random variable withdddegrees of freedom\.

The states under comparison are

aF:=𝟏Fk,aR:=𝟏Rm,a∈𝒜S,a\_\{F\}:=\\frac\{\\mathbf\{1\}\_\{F\}\}\{\\sqrt\{k\}\},\\qquad a\_\{R\}:=\\frac\{\\mathbf\{1\}\_\{R\}\}\{\\sqrt\{m\}\},\\qquad a\\in\\mathcal\{A\}\_\{S\},\(5\)where𝒜S:=\{a∈ℝ\+n:supp\(a\)⊆S,‖a‖2=1\}\\mathcal\{A\}\_\{S\}:=\\\{a\\in\\mathbb\{R\}\_\{\+\}^\{n\}:\\operatorname\{supp\}\(a\)\\subseteq S,\\ \\left\\\|a\\right\\\|\_\{2\}=1\\\}forS⊆VS\\subseteq V, andqF:=qaFq\_\{F\}:=q\_\{a\_\{F\}\},qR:=qaRq\_\{R\}:=q\_\{a\_\{R\}\}\. A weighted sequence is𝒂=\(ac\)c=0D\\boldsymbol\{a\}=\(a\_\{c\}\)\_\{c=0\}^\{D\}withac∈𝒜Rca\_\{c\}\\in\\mathcal\{A\}\_\{R\_\{c\}\}, andamin,c:=minv∈Rc⁡ac,va\_\{\\min,c\}:=\\min\_\{v\\in R\_\{c\}\}a\_\{c,v\}\.

Fix, independently ofGG, unit directionsu0,u1,…,uM∈ℝnu\_\{0\},u\_\{1\},\\ldots,u\_\{M\}\\in\\mathbb\{R\}^\{n\}, withu0u\_\{0\}useful andu1,…,uMu\_\{1\},\\ldots,u\_\{M\}competing; writeXuj:=G​ujX\_\{u\_\{j\}\}:=Gu\_\{j\}\. Every state uses the same decoder

j^​\(qa\):=arg⁡max0≤j≤M​⟨qa,Xuj⟩,\\widehat\{j\}\(q\_\{a\}\):=\\arg\\max\_\{0\\leq j\\leq M\}\\langle q\_\{a\},X\_\{u\_\{j\}\}\\rangle,\(6\)and succeeds whenj^​\(qa\)=0\\widehat\{j\}\(q\_\{a\}\)=0\. When the bank is understood we writeuuforu0u\_\{0\}\. The coordinateuvu\_\{v\}is query weight on vertexvv, not a competing candidate; a node readout isu=evu=e\_\{v\}\. The main text assumes isotropic competition:

u1,…,uM​are orthonormal and orthogonal to​span⁡\{a,u0\}\.u\_\{1\},\\ldots,u\_\{M\}\\text\{ are orthonormal and orthogonal to \}\\operatorname\{span\}\\\{a,u\_\{0\}\\\}\.\(7\)When several states supported onRRare compared, one common bank is used:u1,…,uMu\_\{1\},\\ldots,u\_\{M\}are orthonormal in the orthogonal complement of theRR\-coordinates, which requiresM≤n−mM\\leq n\-m\.

Alignment and the two uniform specializations are

μ⁡\(a,u\)\\displaystyle\\mu\(a,u\):=⟨a,u⟩∈\[0,1\],\\displaystyle:=\\langle a,u\\rangle\\in\[0,1\],μF​\(u\)\\displaystyle\\mu\_\{F\}\(u\):=⟨aF,u⟩=∑v∈Fuvk,\\displaystyle:=\\langle a\_\{F\},u\\rangle=\\frac\{\\sum\_\{v\\in F\}u\_\{v\}\}\{\\sqrt\{k\}\},μR​\(u\)\\displaystyle\\mu\_\{R\}\(u\):=⟨aR,u⟩=∑v∈Ruvm,\\displaystyle:=\\langle a\_\{R\},u\\rangle=\\frac\{\\sum\_\{v\\in R\}u\_\{v\}\}\{\\sqrt\{m\}\},\(8\)for nonnegative unituusupported onRR\. Forδ∈\(0,1\)\\delta\\in\(0,1\),

d⋆​\(a,u,M,δ\)\\displaystyle d\_\{\\star\}\(a,u;M,\\delta\):=min⁡\{d:ℙ⁡\(j^​\(qa\)=0\)≥1−δ\},\\displaystyle:=\\min\\\{d:\\mathbb\{P\}\(\\widehat\{j\}\(q\_\{a\}\)=0\)\\geq 1\-\\delta\\\},\(9\)d⋆,F​\(u\)\\displaystyle d\_\{\\star,F\}\(u\):=d⋆​\(aF,u,M,δ\),\\displaystyle:=d\_\{\\star\}\(a\_\{F\},u;M,\\delta\),d⋆,R​\(u\)\\displaystyle d\_\{\\star,R\}\(u\):=d⋆​\(aR,u,M,δ\),\\displaystyle:=d\_\{\\star\}\(a\_\{R\},u;M,\\delta\),\(10\)under \([7](https://arxiv.org/html/2609.13747#S3.E7)\), with the value∞\\inftyifμ⁡\(a,u\)=0\\mu\(a,u\)=0and1−δ\>1/\(M\+1\)1\-\\delta\>1/\(M\+1\)\.

A query class is a nonempty compact𝒰⊆𝒜S\\mathcal\{U\}\\subseteq\\mathcal\{A\}\_\{S\}; at a fixed depth,S=RS=R\. Set

Γ⁡\(a,𝒰\)\\displaystyle\\Gamma\(a;\\mathcal\{U\}\):=infu∈𝒰⟨a,u⟩,\\displaystyle:=\\inf\_\{u\\in\\mathcal\{U\}\}\\langle a,u\\rangle,\(11\)D⋆​\(a,𝒰,M,δ\)\\displaystyle D\_\{\\star\}\(a;\\mathcal\{U\},M,\\delta\):=min⁡\{d:infu∈𝒰ℙ⁡\(j^​\(qa\)=0\)≥1−δ\}\.\\displaystyle:=\\min\\left\\\{d:\\inf\_\{u\\in\\mathcal\{U\}\}\\mathbb\{P\}\(\\widehat\{j\}\(q\_\{a\}\)=0\)\\geq 1\-\\delta\\right\\\}\.\(12\)d1−δ\\displaystyle d\_\{1\-\\delta\}:=min⁡\{d∈𝒟test:Acc^​\(d\)≥1−δ\}\.\\displaystyle:=\\min\\\{d\\in\\mathcal\{D\}\_\{\\mathrm\{test\}\}:\\widehat\{\\mathrm\{Acc\}\}\(d\)\\geq 1\-\\delta\\\}\.\(13\),which is the discrete version of the theoretical critical dimension\(d⋆​\(δ\)\)\(d\_\{\\star\}\(\\delta\)\)in experiments Call𝒰⊆𝒜R\\mathcal\{U\}\\subseteq\\mathcal\{A\}\_\{R\}CC\-nonlocalized, for a constantC≥1C\\geq 1, if

‖u‖∞≤Cm,u∈𝒰\.\\left\\\|u\\right\\\|\_\{\\infty\}\\leq\\frac\{C\}\{\\sqrt\{m\}\},\\qquad u\\in\\mathcal\{U\}\.\(14\)
Statements containingo⁡\(1\)o\(1\)are asM→∞M\\to\\inftyat fixedδ\\delta, withnnlarge enough thatM≤n−mM\\leq n\-m; remainders ind⋆d\_\{\\star\}are uniform overμ∈\(0,1\]\\mu\\in\(0,1\]\.

## 4The Dimension Limit

We first derive the dimension law for an arbitrary weighted state\. Frontier and cumulative comparisons will then follow by substitution, not by changing the readout\.

###### Lemma 4\.1\(Exact score competition\)\.

Fix unita,u0a,u\_\{0\}withμ:=⟨a,u0⟩∈\[0,1\]\\mu:=\\langle a,u\_\{0\}\\rangle\\in\[0,1\], and assume \([7](https://arxiv.org/html/2609.13747#S3.E7)\)\. There exist mutually independentZ0,Z1,…,ZM∼N⁡\(0,1\)Z\_\{0\},Z\_\{1\},\\ldots,Z\_\{M\}\\sim N\(0,1\)andCd∼χdC\_\{d\}\\sim\\chi\_\{d\}such that

d​⟨qa,Xu0⟩\\displaystyle\\sqrt\{d\}\\,\\langle q\_\{a\},X\_\{u\_\{0\}\}\\rangle=𝑑​μ​Cd\+1−μ2​Z0,\\displaystyle\\overset\{d\}\{=\}\\mu C\_\{d\}\+\\sqrt\{1\-\\mu^\{2\}\}\\,Z\_\{0\},\(15\)d​⟨qa,Xuj⟩\\displaystyle\\sqrt\{d\}\\,\\langle q\_\{a\},X\_\{u\_\{j\}\}\\rangle=𝑑​Zj,1≤j≤M\.\\displaystyle\\overset\{d\}\{=\}Z\_\{j\},\\qquad 1\\leq j\\leq M\.\(16\)Consequently,

ℙ⁡\(j^​\(qa\)=0\)=ℙ⁡\(μ​Cd\+1−μ2​Z0\>max1≤j≤M⁡Zj\)\.\\mathbb\{P\}\(\\widehat\{j\}\(q\_\{a\}\)=0\)=\\mathbb\{P\}\\left\(\\mu C\_\{d\}\+\\sqrt\{1\-\\mu^\{2\}\}\\,Z\_\{0\}\>\\max\_\{1\\leq j\\leq M\}Z\_\{j\}\\right\)\.\(17\)

Lemma[4\.1](https://arxiv.org/html/2609.13747#S4.Thmtheorem1)already exposes the mechanism\. Dimension increases only the aligned signalμ​Cd≈μ​d\\mu C\_\{d\}\\approx\\mu\\sqrt\{d\}\. TheMMunrelated candidates contribute a maximum of order2​log⁡M\\sqrt\{2\\log M\}\.

###### Theorem 4\.2\(Dimension phase transition\)\.

Consider any sequence withM→∞M\\to\\infty, unit statesa=aMa=a\_\{M\}, useful directionsu0=u0,Mu\_\{0\}=u\_\{0,M\}, alignmentsμM∈\(0,1\]\\mu\_\{M\}\\in\(0,1\], and dimensionsdMd\_\{M\}\. Under the fixed\-bank condition \([7](https://arxiv.org/html/2609.13747#S3.E7)\), if

dM​μM22​log⁡M⟶λ,\\frac\{d\_\{M\}\\mu\_\{M\}^\{2\}\}\{2\\log M\}\\longrightarrow\\lambda,\(18\)forλ∈\[0,∞\]\\lambda\\in\[0,\\infty\], then

ℙ⁡\(j^​\(qa\)=0\)⟶\{0,λ<1,1,λ\>1\.\\mathbb\{P\}\(\\widehat\{j\}\(q\_\{a\}\)=0\)\\longrightarrow\\begin\{cases\}0,&\\lambda<1,\\\\ 1,&\\lambda\>1\.\\end\{cases\}\(19\)At the boundaryλ=1\\lambda=1, the theorem makes no zero–one claim\.

###### Corollary 4\.4\(Matching critical dimension\)\.

Fixδ∈\(0,1\)\\delta\\in\(0,1\)\. For any sequenceM→∞M\\to\\inftyandμM∈\(0,1\]\\mu\_\{M\}\\in\(0,1\], the critical dimension at which superposition reasoning succeeds with high probability is

d⋆​\(a,u0,M,δ\)=\(2\+o⁡\(1\)\)​log⁡MμM2\.d\_\{\\star\}\(a,u\_\{0\};M,\\delta\)=\(2\+o\(1\)\)\\frac\{\\log M\}\{\\mu\_\{M\}^\{2\}\}\.\(20\)Moreover, letN:=n−mN:=n\-mand define

Mmax​\(d,μ,δ,N\):=max⁡\{1≤M≤N:pd,M​\(μ\)≥1−δ\},M\_\{\\max\}\(d,\\mu,\\delta,N\):=\\max\\\{1\\leq M\\leq N:p\_\{d,M\}\(\\mu\)\\geq 1\-\\delta\\\},\(21\)wherepd,M​\(μ\)p\_\{d,M\}\(\\mu\)denotes the probability in \([17](https://arxiv.org/html/2609.13747#S4.E17)\)\. Along any sequence for whichmin⁡\{d​μ2,log⁡N\}→∞\\min\\\{d\\mu^\{2\},\\log N\\\}\\to\\infty,

log⁡Mmax=\(1\+o⁡\(1\)\)​min​\{d​μ22,log⁡N\}\.\\log M\_\{\\max\}=\(1\+o\(1\)\)\\min\\left\\\{\\frac\{d\\mu^\{2\}\}\{2\},\\log N\\right\\\}\.\(22\)

## 5Why Cumulative Superposition Wins

We now answer three questions\. When cumulative needs fewer dimensions? How a Transformer implements its update, and whether another weighting can do better once the full history must remain usable?

### 5\.1Cumulative needs fewer dimensions

#### Task regime\.

Nonlocalization is a state\-independent bounded\-influence condition, it constrains only the downstream queryuuand never compares its alignments withaRa\_\{R\}andaFa\_\{F\}\. Since‖u‖2=1\\left\\\|u\\right\\\|\_\{2\}=1, a fixedCCmeans that every query draws on at leastm/C2m/C^\{2\}reached objects, Such readouts arise naturally in set and multiset aggregation, additive objectives over visited objects, and supervision over algorithmic or reasoning traces\([Zaheer et al\., 2017](https://arxiv.org/html/2609.13747#bib.bib23);[Lee et al\., 2019](https://arxiv.org/html/2609.13747#bib.bib24);[Kool et al\., 2019](https://arxiv.org/html/2609.13747#bib.bib25);[Veličković et al\., 2022](https://arxiv.org/html/2609.13747#bib.bib26);[Lightman et al\., 2024](https://arxiv.org/html/2609.13747#bib.bib27)\)\. Continuous and superposed reasoning provide the corresponding state geometry of multi\-step latent search\([Hao et al\., 2024](https://arxiv.org/html/2609.13747#bib.bib6);[Zhu et al\., 2025b](https://arxiv.org/html/2609.13747#bib.bib7)\)\. On the state side,k=o⁡\(m\)k=o\(m\)says that the states activated at the current step grow more slowly than the accumulated computation\. It includes chains and fixed\-width traces, wherek=O⁡\(1\)k=O\(1\)andm=Θ⁡\(c\)m=\\Theta\(c\)at depthcc, as well as grid\-like breadth\-first search indddimensions, wherek=Θ⁡\(cd−1\)k=\\Theta\(c^\{d\-1\}\)andm=Θ⁡\(cd\)m=\\Theta\(c^\{d\}\)\. FixedCCandk=o⁡\(m\)k=o\(m\)therefore capture a broad growing\-history regime in which queries reuse a constant fraction of an expanding trace while its current frontier occupies a vanishing fraction\.

###### Theorem 5\.1\(Sharp separation for nonlocalized query classes\)\.

FixF⊊RF\\subsetneq R, with\|F\|=k<m=\|R\|\|F\|=k<m=\|R\|, and use the same candidate bank for both states\. Let𝒰⊆𝒜R\\mathcal\{U\}\\subseteq\\mathcal\{A\}\_\{R\}be any nonempty compactCC\-nonlocalized query class\. Then

Γ⁡\(aR,𝒰\)≥1C,Γ⁡\(aF,𝒰\)≤C​km,\\Gamma\(a\_\{R\};\\mathcal\{U\}\)\\geq\\frac\{1\}\{C\},\\qquad\\Gamma\(a\_\{F\};\\mathcal\{U\}\)\\leq C\\sqrt\{\\frac\{k\}\{m\}\},\(23\)Consequently, wheneverΓ⁡\(aF,𝒰\)\>0\\Gamma\(a\_\{F\};\\mathcal\{U\}\)\>0,

D⋆​\(aR,𝒰,M,δ\)D⋆​\(aF,𝒰,M,δ\)≤C4​km​\(1\+o⁡\(1\)\)\.\\frac\{D\_\{\\star\}\(a\_\{R\};\\mathcal\{U\},M,\\delta\)\}\{D\_\{\\star\}\(a\_\{F\};\\mathcal\{U\},M,\\delta\)\}\\leq C^\{4\}\\frac\{k\}\{m\}\(1\+o\(1\)\)\.\(24\)For every fixedCC, this ratio tends to zero in the growing\-history regimek=o⁡\(m\)k=o\(m\); thus cumulative needs asymptotically fewer dimensions for the same query class and error\. IfΓ⁡\(aF,𝒰\)=0\\Gamma\(a\_\{F\};\\mathcal\{U\}\)=0, frontier instead has infinite critical dimension above chance while cumulative remains finite\.

The reason is coherent addition\. Readout succeeds when

⟨a,u⟩​d⏟useful signal≳2​log⁡M⏟unrelated competition\.\\underbrace\{\\langle a,u\\rangle\\sqrt\{d\}\}\_\{\\text\{useful signal\}\}\\gtrsim\\underbrace\{\\sqrt\{2\\log M\}\}\_\{\\text\{unrelated competition\}\}\.\(25\)For cumulative,⟨aR,u⟩=‖u‖1/m\\langle a\_\{R\},u\\rangle=\\left\\\|u\\right\\\|\_\{1\}/\\sqrt\{m\}: useful historical components add before normalization, while unrelated candidates remain centered fluctuations\. The extra stored objects therefore strengthen a history\-wide computation instead of acting as independent noise\. This is the mathematical source of the apparent reversal\.

### 5\.2Transformer realization and its dimension cost

At depthcc, letNc=Rc\+1∖RcN\_\{c\}=R\_\{c\+1\}\\setminus R\_\{c\}be the newly reached set andrc=\|Nc\|r\_\{c\}=\|N\_\{c\}\|\. An edge tokene=\(s,t\)e=\(s,t\)stores\[gs;gt\]\[g\_\{s\};g\_\{t\}\], and the latent token stores the current superposition\. A visited\-edge mask exposes the current expansion candidates; we assume one active parent per node inNcN\_\{c\}so that active destinations are averaged uniformly\.

###### Theorem 5\.3\(Transformer realization and routing dimension\)\.

A width\-O⁡\(d\)O\(d\)construction with two attention blocks, a standard residual, and RMSNorm implements the following update with vanishing error:

qc\+1=RMSNorm⁡\(λc​qc\+ηc​oc\),oc=1rc​∑t∈Ncgt\+o⁡\(1\)\.q\_\{c\+1\}=\\operatorname\{RMSNorm\}\\\!\\left\(\\lambda\_\{c\}q\_\{c\}\+\\eta\_\{c\}o\_\{c\}\\right\),\\qquad o\_\{c\}=\\frac\{1\}\{\\sqrt\{r\_\{c\}\}\}\\sum\_\{t\\in N\_\{c\}\}g\_\{t\}\+o\(1\)\.\(26\)The first attention block compares the source\-membership scoreβ⁡⟨qc,gs⟩\\beta\\langle q\_\{c\},g\_\{s\}\\ranglewith a threshold token and writes a gate on each edge; the second averages the destinations of the gated edges\. The coefficient choices

\(λc,ηc\)=\(0,1\)and\(λc,ηc\)=\(\|Rc\|,rc\)\(\\lambda\_\{c\},\\eta\_\{c\}\)=\(0,1\)\\quad\\text\{and\}\\quad\(\\lambda\_\{c\},\\eta\_\{c\}\)=\(\\sqrt\{\|R\_\{c\}\|\},\\sqrt\{r\_\{c\}\}\)\(27\)produce frontier and cumulative, respectively, while other nonnegative coefficients produce layer\-weighted recursive updates\. No FFN is required\.

More generally, the same membership head can route from any weighted state sequence𝐚=\(ac\)c=0D\\boldsymbol\{a\}=\(a\_\{c\}\)\_\{c=0\}^\{D\}\. If every layer containsΘ⁡\(n\)\\Theta\(n\)independent inactive source keys andamin,c:=minv∈Rc⁡ac,va\_\{\\min,c\}:=\\min\_\{v\\in R\_\{c\}\}a\_\{c,v\}, its routing critical dimension is

droute​\(𝒂,δ\)=Θ⁡\(maxc<D⁡log⁡\(D​n/δ\)amin,c2\),d\_\{\\mathrm\{route\}\}\(\\boldsymbol\{a\},\\delta\)=\\Theta\\\!\\left\(\\max\_\{c<D\}\\frac\{\\log\(Dn/\\delta\)\}\{a\_\{\\min,c\}^\{2\}\}\\right\),\(28\)with infinite dimension when a required coefficient is zero\. For a uniform state onssobjects this reduces toΘ⁡\(s​log⁡\(D​n/δ\)\)\\Theta\(s\\log\(Dn/\\delta\)\)\.

### 5\.3Cumulative is minimax under nonlocalized future use

Consider any weighted sequence readable by the Transformer membership head,𝒂=\(ac\)c=0D\\boldsymbol\{a\}=\(a\_\{c\}\)\_\{c=0\}^\{D\}withac∈𝒜Rca\_\{c\}\\in\\mathcal\{A\}\_\{R\_\{c\}\}\. If every reached object may later be queried as an edge source, its smallest coefficient controls the worst membership margin\. Theorem[5\.3](https://arxiv.org/html/2609.13747#S5.Thmtheorem3)and Corollary[4\.4](https://arxiv.org/html/2609.13747#S4.Thmtheorem4)show that its actual end\-to\-end critical dimension is, up to universal constants, the scale

𝔇⁡\(𝒂,𝒰\):=max⁡\{maxc<D⁡log⁡\(D​n/δ\)\(minv∈Rc⁡ac,v\)2,2​log⁡MΓ​\(aD,𝒰\)2\}\.\\mathfrak\{D\}\(\\boldsymbol\{a\};\\mathcal\{U\}\):=\\max\\left\\\{\\max\_\{c<D\}\\frac\{\\log\(Dn/\\delta\)\}\{\(\\min\_\{v\\in R\_\{c\}\}a\_\{c,v\}\)^\{2\}\},\\frac\{2\\log M\}\{\\Gamma\(a\_\{D\};\\mathcal\{U\}\)^\{2\}\}\\right\\\}\.\(29\)This optimizes over the full weighted family𝒜Rc\\mathcal\{A\}\_\{R\_\{c\}\}, which contains the recursive updates in Theorem[5\.3](https://arxiv.org/html/2609.13747#S5.Thmtheorem3)as well as their two canonical endpoints\.

###### Theorem 5\.5\(Sharp universal and robust optimality of cumulative\)\.

Suppose every reached object must remain eligible for later source\-membership attention\. Let𝒰\\mathcal\{U\}be any nonempty compact C\-query class, and let𝐚R=\(aRc\)c=0D\\boldsymbol\{a\}\_\{R\}=\(a\_\{R\_\{c\}\}\)\_\{c=0\}^\{D\}be the uniform cumulative sequence\. Then

𝔇⁡\(𝒂R,𝒰\)≤C2​infac∈𝒜Rc0≤c≤D𝔇⁡\(𝒂,𝒰\)\.\\mathfrak\{D\}\(\\boldsymbol\{a\}\_\{R\};\\mathcal\{U\}\)\\leq C^\{2\}\\inf\_\{\\begin\{subarray\}\{c\}a\_\{c\}\\in\\mathcal\{A\}\_\{R\_\{c\}\}\\\\ 0\\leq c\\leq D\\end\{subarray\}\}\\mathfrak\{D\}\(\\boldsymbol\{a\};\\mathcal\{U\}\)\.\(30\)The factorC2C^\{2\}is sharp in the readout\-dominated regime\. Moreover, cumulative is exactly robust\-minimax over the full nonlocalized family:

𝒂R∈arg⁡minac∈𝒜Rc0≤c≤D⁡𝔇⁡\(𝒂,𝒰C​\(RD\)\)\.\\boldsymbol\{a\}\_\{R\}\\in\\arg\\min\_\{\\begin\{subarray\}\{c\}a\_\{c\}\\in\\mathcal\{A\}\_\{R\_\{c\}\}\\\\ 0\\leq c\\leq D\\end\{subarray\}\}\\mathfrak\{D\}\(\\boldsymbol\{a\};\\mathcal\{U\}\_\{C\}\(R\_\{D\}\)\)\.\(31\)Thus, for fixedC=O⁡\(1\)C=O\(1\), cumulative is rate\-minimax\-optimal for every nonlocalized query class and exactly optimal when nonlocalization is the only prior on future use\.

## 6Experiment

### 6\.1Nonlocalized separation

We test Theorem[5\.1](https://arxiv.org/html/2609.13747#S5.Thmtheorem1)on a grid overm∈\{64,128,256,512\}m\\in\\\{64,128,256,512\\\},k/m∈\{1/32,…,1/2\}k/m\\in\\\{1/32,\\ldots,1/2\\\},C∈\{1,1\.25,1\.5,2\}C\\in\\\{1,1\.25,1\.5,2\\\}, andM∈\{210,214\}M\\in\\\{2^\{10\},2^\{14\}\\\}, estimating worst\-caseD⋆D\_\{\\star\}by binary search onddwith20002000Monte Carlo trials \(δ=0\.1\\delta=0\.1\)\. Each query is drawn independently ofGGand projected onto the cap‖u‖∞≤C/m\\left\\\|u\\right\\\|\_\{\\infty\}\\leq C/\\sqrt\{m\}\.*Dense*uses random positive weights on allmmcoordinates, modeling uneven aggregation over a visited set\([Zaheer et al\., 2017](https://arxiv.org/html/2609.13747#bib.bib23)\)\.*Temporal*centers an exponential bump along the trace, modeling localized process supervision over a CoT\([Lightman et al\., 2024](https://arxiv.org/html/2609.13747#bib.bib27)\)\.*Block*partitions the trace into segments with different scales, modeling staged algorithmic readouts\([Veličković et al\., 2022](https://arxiv.org/html/2609.13747#bib.bib26)\)\.*Union*mixes the three generators\. The*sharp witness*is the theorem’s example, uniform on⌈m/C2⌉\\lceil m/C^\{2\}\\rceilcoordinates that containFF\.*Zero\-overlap*is uniform onR∖FR\\setminus F, soΓ⁡\(aF,𝒰\)=0\\Gamma\(a\_\{F\};\\mathcal\{U\}\)=0\.

Figure 2:Cumulative superposition is both broadly more dimension\-efficient and structurally favored by growing history\. \(a\) Across diverse nonlocalized query families, its critical\-dimension ratio remains below the theoretical envelope, while the sharp construction approaches equality\. \(b\) As the reached history grows, cumulative preserves coherent query signal, whereas frontier discards reusable evidence and its signal eventually falls below the fixed competition scale\.#### Protocol\.

We use the ProsQA\([Zhong et al\., 2022](https://arxiv.org/html/2609.13747#bib.bib1)\)graph\-memory task and hold the graph trace, candidate bank, and Gaussian codebook fixed when comparing state constructions\. Queries are drawn before the Gaussian map\. The controlled attention experiment uses an identity\-regularized source\-membership router atd=192d=192and evaluates five independent Gaussian codebooks\.

### 6\.2Cumulative has a lower dimension threshold use

Across the paired five\-seed sweeps, cumulative crosses the same 90% held\-out success threshold at substantially lower width on both architectures\. Its interpolated critical dimension is40\.3±1\.140\.3\\pm 1\.1, compared with110\.6±1\.6110\.6\\pm 1\.6for frontier, in the lightweight two\-block model; with the two\-layer GPT\-2 backbone it is34\.4±3\.834\.4\\pm 3\.8, compared with66\.2±5\.966\.2\\pm 5\.9\. The corresponding cumulative/frontier ratios are0\.365±0\.0080\.365\\pm 0\.008and0\.521±0\.0670\.521\\pm 0\.067\. Thus the separation predicted by Theorem 5\.1 persists when the analytic router is replaced by a trained causal Transformer\. Figure[3](https://arxiv.org/html/2609.13747#S6.F3)shows both width sweeps and their paired critical\-dimension estimates under matched evaluation protocols\.

\(b\) Paired five\-seed results

Figure 3:One empirical view of dimension separation and weighted optimality\.The shared legend specifies six reusable weighting schemes\. \(a1\) The original regularized two\-block diagnostic; \(a2\) the GPT\-2 replication reports the mean with one\-standard\-deviation bands\. Within each panel the trained router is shared by every curve, and uniform cumulative crosses the90%90\\%worst\-query criterion first\. \(b\) Paired five\-seed results show that cumulative reaches matched held\-out accuracy at substantially lower critical width than frontier for both the lightweight two\-block model and GPT\-2\. Entries are means with two\-sided 95% Student\-ttintervals;d0\.9d\_\{0\.9\}is linearly interpolated, and “Ideal” directly supplies the analytic state\. Full configurations are deferred to the appendix\.
### 6\.3Uniform weighting prevents a token\-level routing bottleneck

We next test the mechanism behind Theorem 5\.3\. Figure[4](https://arxiv.org/html/2609.13747#S6.F4)\(a\) shows the predicted two\-term geometry: uniform cumulative is the joint lower\-left point after normalizing routing and worst\-query readout width by the uniform construction\. Panels \(b,c\) expose what this optimum means inside the router\. Uniform weights equalize the coefficient of every reusable memory token; recency and random weights produce low\-coefficient tokens, and those tokens receive lower membership margins\. Consequently, joint route success falls from0\.9060\.906to0\.5860\.586and0\.5370\.537, while the per\-inactive\-key false\-positive rate rises from0\.03%0\.03\\%to0\.57%0\.57\\%and1\.31%1\.31\\%, respectively\.

Figure 4:Why uniform cumulative is minimax among reusable weighted states\. \(a\) Pareto geometry of routing and worst\-query readout width terms, normalized by uniform cumulative; lower is better\. \(b\) A token\-level heatmap of normalized coefficients over reached\-history position, averaged over test traces and five Gaussian codebooks\. \(c\) The same router’s membership margins\. The summary below gives joint routing success and per\-inactive\-token false\-positive rate\.

## 7Conclusion

We characterized frontier, cumulative, and general weighted reasoning superpositions through three linked results\. First, a sharp Gaussian dimension law reduces reliable readout to state–query alignment; for any fixed nonlocalized query class, cumulative requires asymptotically fewer dimensions than frontier in the growing\-history regimek=o⁡\(m\)k=o\(m\)\. Second, two attention blocks, a residual path, and RMSNorm realize both states as endpoints of one weighted Transformer update, attention selects newly reached objects, whereas the residual determines whether previous computation is discarded or retained\. The same construction shows that the smallest reusable coefficient controls routing width\. Third, optimizing routing and readout jointly over the full weighted family makes uniform cumulative superposition exactly minimax\-optimal when reached history remains reusable and future use is exchangeable\.

These results explain the apparent paradox that storing more objects can make reasoning easier\. cumulative therefore converts reusable history into coherent signal rather than additional interference\. This gives “stronger” an operational meaning—fewer dimensions for the same query class, transition, and target error—and identifies cumulative as the capacity\-optimal target in this regime\. The analysis isolates capacity and representability under Gaussian embeddings and an isotropic candidate bank; Appendix[C](https://arxiv.org/html/2609.13747#A3)extends the reduction to correlated banks with positive population margins\. Extending the theory to learned anisotropic codebooks and explaining when optimization discovers the required routing and weighting remain important next steps\.

## References

- Bereskaet al\.\(2025\)L\. Bereska, Z\. Tzifa\-Kratira, R\. Samavi, and E\. GavvesSuperposition as lossy compression: measure with sparse autoencoders and connect to adversarial vulnerability\.arXiv preprint arXiv:2512\.13568\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px2.p1.1)\.
- Brown\-Cohenet al\.\(2026\)J\. Brown\-Cohen, D\. Lindner, and R\. ShahQuantifying the necessity of chain of thought through opaque serial depth\.arXiv preprint arXiv:2603\.09786\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px3.p1.1)\.
- Bulatovet al\.\(2022\)A\. Bulatov, Y\. Kuratov, and M\. BurtsevRecurrent memory transformer\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px3.p1.1)\.
- Dehghaniet al\.\(2019\)M\. Dehghani, S\. Gouws, O\. Vinyals, J\. Uszkoreit, and L\. KaiserUniversal transformers\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px3.p1.1)\.
- Elhageet al\.\(2022\)N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan, S\. Kravec, Z\. Hatfield\-Dodds, R\. Lasenby, D\. Drain, C\. Chen,et al\.Toy models of superposition\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2022/toy_model/index.html)Cited by:[§1](https://arxiv.org/html/2609.13747#S1.p3.1),[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px2.p1.1)\.
- Foote \(2024\)A\. FooteTackling polysemanticity with neuron embeddings\.arXiv preprint arXiv:2411\.08166\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px2.p1.1)\.
- Fuet al\.\(2026\)R\. Fu, Z\. Yang, J\. Zhang, J\. Ma, H\. Chen, Y\. Li, and Y\. ChangSimply stabilizing the loop via fully looped transformer\.arXiv preprint arXiv:2605\.18797\.Cited by:[§1](https://arxiv.org/html/2609.13747#S1.p1.1),[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px3.p1.1)\.
- Giannouet al\.\(2023\)A\. Giannou, S\. Rajput, J\. Sohn, K\. Lee, J\. D\. Lee, and D\. PapailiopoulosLooped transformers as programmable computers\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 11398–11442\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px3.p1.1)\.
- Gozetenet al\.\(2025\)H\. A\. Gozeten, M\. E\. Ildiz, X\. Zhang, H\. Harutyunyan, A\. S\. Rawat, and S\. OymakContinuous chain of thought enables parallel exploration and reasoning\.arXiv preprint arXiv:2505\.23648\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px1.p1.1)\.
- Gupta and Kumar \(2025\)M\. Gupta and D\. KumarDisentangling polysemantic neurons with a null\-calibrated polysemanticity index and causal patch interventions\.arXiv preprint arXiv:2508\.16950\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px2.p1.1)\.
- Haoet al\.\(2024\)S\. Hao, S\. Sukhbaatar, D\. Su, X\. Li, Z\. Hu, J\. Weston, and Y\. TianTraining large language models to reason in a continuous latent space\.arXiv preprint arXiv:2412\.06769\.Cited by:[§1](https://arxiv.org/html/2609.13747#S1.p1.1),[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2609.13747#S5.SS1.SSS0.Px1.p1.1)\.
- Huet al\.\(2024\)L\. Hu, L\. Liu, S\. Yang, X\. Chen, Z\. Tan, M\. A\. Ali, M\. Li, and D\. WangUnderstanding reasoning in chain\-of\-thought from the hopfieldian view\.arXiv preprint arXiv:2410\.03595\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px3.p1.1)\.
- Koolet al\.\(2019\)W\. Kool, H\. van Hoof, and M\. WellingAttention, learn to solve routing problems\!\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.13747#S1.p4.1),[§5\.1](https://arxiv.org/html/2609.13747#S5.SS1.SSS0.Px1.p1.1)\.
- Leeet al\.\(2019\)J\. Lee, Y\. Lee, J\. Kim, A\. Kosiorek, S\. Choi, and Y\. W\. TehSet transformer: a framework for attention\-based permutation\-invariant neural networks\.InProceedings of the 36th International Conference on Machine Learning,pp\. 3744–3753\.Cited by:[§1](https://arxiv.org/html/2609.13747#S1.p4.1),[§5\.1](https://arxiv.org/html/2609.13747#S5.SS1.SSS0.Px1.p1.1)\.
- Liet al\.\(2024\)Y\. Li, S\. Liang, M\. R\. Lyu, and L\. WangMaking long\-context language models better multi\-hop reasoners\.arXiv preprint arXiv:2408\.03246\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px3.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.13747#S1.p4.1),[§5\.1](https://arxiv.org/html/2609.13747#S5.SS1.SSS0.Px1.p1.1),[§6\.1](https://arxiv.org/html/2609.13747#S6.SS1.p1.1)\.
- Linet al\.\(2026\)H\. Lin, M\. Cai, J\. Wen, and H\. ZhangTyler: typed latent reasoning for language models – when to think, what to compute, and how much to allocate\.arXiv preprint arXiv:2606\.16360\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px3.p1.1)\.
- Merrill and Sabharwal \(2024\)W\. Merrill and A\. SabharwalThe expressive power of transformers with chain of thought\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px1.p1.1)\.
- Veličkovićet al\.\(2022\)P\. Veličković, A\. P\. Badia, D\. Budden, R\. Pascanu, A\. Banino, M\. Dashevskiy, R\. Hadsell, and C\. BlundellThe CLRS algorithmic reasoning benchmark\.InProceedings of the 39th International Conference on Machine Learning,pp\. 22084–22102\.Cited by:[§1](https://arxiv.org/html/2609.13747#S1.p4.1),[§5\.1](https://arxiv.org/html/2609.13747#S5.SS1.SSS0.Px1.p1.1),[§6\.1](https://arxiv.org/html/2609.13747#S6.SS1.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px1.p1.1)\.
- Xu and Sato \(2024\)K\. Xu and I\. SatoOn expressive power of looped transformers: theoretical analysis and enhancement via timestep encoding\.arXiv preprint arXiv:2410\.01405\.Cited by:[§1](https://arxiv.org/html/2609.13747#S1.p1.1),[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2024\)L\. Yang, Z\. Yu, T\. Zhang, S\. Cao, M\. Xu, W\. Zhang, J\. E\. Gonzalez, and B\. CuiBuffer of thoughts: thought\-augmented reasoning with large language models\.arXiv preprint arXiv:2406\.04271\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px3.p1.1)\.
- Yaoet al\.\(2025\)Y\. Yao, Y\. Du, D\. Zhu, M\. Hahn, and A\. KollerLanguage models can learn implicit multi\-hop reasoning, but only if they have lots of training data\.arXiv preprint arXiv:2505\.17923\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px3.p1.1)\.
- Zaheeret al\.\(2017\)M\. Zaheer, S\. Kottur, S\. Ravanbakhsh, B\. Poczos, R\. Salakhutdinov, and A\. J\. SmolaDeep sets\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§1](https://arxiv.org/html/2609.13747#S1.p4.1),[§5\.1](https://arxiv.org/html/2609.13747#S5.SS1.SSS0.Px1.p1.1),[§6\.1](https://arxiv.org/html/2609.13747#S6.SS1.p1.1)\.
- Zhonget al\.\(2022\)W\. Zhong, Y\. Gao, N\. Ding, Y\. Qin, Z\. Liu, M\. Zhou, J\. Wang, J\. Yin, and N\. DuanProQA: structural prompt\-based pre\-training for unified question answering\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 4230–4243\.External Links:[Link](https://aclanthology.org/2022.naacl-main.313/),[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.313)Cited by:[§6\.1](https://arxiv.org/html/2609.13747#S6.SS1.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2025a\)H\. Zhu, S\. Hao, Z\. Hu, J\. Jiao, S\. Russell, and Y\. TianEmergence of superposition: unveiling the training dynamics of chain of continuous thought\.arXiv preprint arXiv:2509\.23365\.Cited by:[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2025b\)H\. Zhu, S\. Hao, Z\. Hu, J\. Jiao, S\. Russell, and Y\. TianReasoning by superposition: a theoretical perspective on chain of continuous thought\.InAdvances in Neural Information Processing Systems,Note:arXiv:2505\.12514Cited by:[§D\.1](https://arxiv.org/html/2609.13747#A4.SS1.p1.1),[§1](https://arxiv.org/html/2609.13747#S1.p1.1),[§2](https://arxiv.org/html/2609.13747#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2609.13747#S5.SS1.SSS0.Px1.p1.1)\.

## Appendix AProofs for the Dimension Limit

###### Proof of Lemma[4\.1](https://arxiv.org/html/2609.13747#S4.Thmtheorem1)\.

LetY:=G​aY:=Ga\. Since‖a‖2=1\\left\\\|a\\right\\\|\_\{2\}=1,Y∼N⁡\(0,Id/d\)Y\\sim N\(0,I\_\{d\}/d\)andd​‖Y‖∼χd\\sqrt\{d\}\\left\\\|Y\\right\\\|\\sim\\chi\_\{d\}\.

Ifμ<1\\mu<1, define

c:=u0−μ​a1−μ2\.c:=\\frac\{u\_\{0\}\-\\mu a\}\{\\sqrt\{1\-\\mu^\{2\}\}\}\.\(32\)Thenccis unit norm,c⟂ac\\perp a, andu0=μ​a\+1−μ2​cu\_\{0\}=\\mu a\+\\sqrt\{1\-\\mu^\{2\}\}c\. Ifμ=1\\mu=1, the term involvingccbelow is absent\. Gaussianity andc⟂ac\\perp aimply thatG​cGcis independent ofYY\. Conditioned onYY, rotational invariance gives

d​⟨Y/‖Y‖,G​c⟩∼N⁡\(0,1\),\\sqrt\{d\}\\langle Y/\\left\\\|Y\\right\\\|,Gc\\rangle\\sim N\(0,1\),\(33\)and this random variable is independent of‖Y‖\\left\\\|Y\\right\\\|\. Therefore

d​⟨qa,G​u0⟩\\displaystyle\\sqrt\{d\}\\langle q\_\{a\},Gu\_\{0\}\\rangle=μ​d​‖Y‖\+1−μ2​d​⟨Y/‖Y‖,G​c⟩\\displaystyle=\\mu\\sqrt\{d\}\\left\\\|Y\\right\\\|\+\\sqrt\{1\-\\mu^\{2\}\}\\sqrt\{d\}\\langle Y/\\left\\\|Y\\right\\\|,Gc\\rangle=𝑑​μ​Cd\+1−μ2​Z0\.\\displaystyle\\overset\{d\}\{=\}\\mu C\_\{d\}\+\\sqrt\{1\-\\mu^\{2\}\}Z\_\{0\}\.\(34\)
For eachj≥1j\\geq 1,uj⟂au\_\{j\}\\perp a, soG​ujGu\_\{j\}is independent ofYYand

d​⟨Y/‖Y‖,G​uj⟩∼N⁡\(0,1\)\.\\sqrt\{d\}\\langle Y/\\left\\\|Y\\right\\\|,Gu\_\{j\}\\rangle\\sim N\(0,1\)\.\(35\)The orthogonality ofu1,…,uMu\_\{1\},\\ldots,u\_\{M\}and their orthogonality tou0u\_\{0\}imply joint independence of all the Gaussian terms\. Comparing the scores proves \([17](https://arxiv.org/html/2609.13747#S4.E17)\)\.

LetSμ:=μ​Cd\+1−μ2​ZS\_\{\\mu\}:=\\mu C\_\{d\}\+\\sqrt\{1\-\\mu^\{2\}\}ZandFμ​\(t\):=ℙ⁡\(Sμ≤t\)F\_\{\\mu\}\(t\):=\\mathbb\{P\}\(S\_\{\\mu\}\\leq t\)\. For0<μ<10<\\mu<1, putσ2:=1−μ2\\sigma^\{2\}:=1\-\\mu^\{2\}\. Differentiating under the integral against theχd\\chi\_\{d\}density gives

∂Fμ​\(t\)∂μ=−Kd,μ,t∫0∞\(c−μt\)cd−1exp\(−\(c−μ​t\)22​σ2\)dc,\\frac\{\\partial F\_\{\\mu\}\(t\)\}\{\\partial\\mu\}=\-K\_\{d,\\mu,t\}\\int\_\{0\}^\{\\infty\}\(c\-\\mu t\)c^\{d\-1\}\\exp\\left\(\-\\frac\{\(c\-\\mu t\)^\{2\}\}\{2\\sigma^\{2\}\}\\right\)dc,\(36\)whereKd,μ,t\>0K\_\{d,\\mu,t\}\>0\. Denote the integral byII\. Since\(c−μt\)e−\(c−μt\)2/\(2σ2\)\(c\-\\mu t\)e^\{\-\(c\-\\mu t\)^\{2\}/\(2\\sigma^\{2\}\)\}equals−σ2\-\\sigma^\{2\}times the derivative of the exponential, integration by parts yields

I=\{σ2​exp⁡\(−μ2​t22​σ2\),d=1,σ2​\(d−1\)​∫0∞cd−2​exp⁡\(−\(c−μ​t\)22​σ2\)​dc,d\>1\.I=\\begin\{cases\}\\sigma^\{2\}\\exp\\left\(\-\\dfrac\{\\mu^\{2\}t^\{2\}\}\{2\\sigma^\{2\}\}\\right\),&d=1,\\\\\[5\.0pt\] \\sigma^\{2\}\(d\-1\)\\displaystyle\\int\_\{0\}^\{\\infty\}c^\{d\-2\}\\exp\\left\(\-\\dfrac\{\(c\-\\mu t\)^\{2\}\}\{2\\sigma^\{2\}\}\\right\)dc,&d\>1\.\\end\{cases\}\(37\)ThusI\>0I\>0and∂μFμ​\(t\)<0\\partial\_\{\\mu\}F\_\{\\mu\}\(t\)<0for everytt\. Continuity handlesμ=0,1\\mu=0,1, soSμS\_\{\\mu\}is strictly increasing in first\-order stochastic order\. Conditional onSμ=sS\_\{\\mu\}=s, success againstMMindependent standard\-normal decoys has probabilityΦ​\(s\)M\\Phi\(s\)^\{M\}, a strictly increasing function ofss\. Therefore the unconditional success probability is strictly increasing inμ\\mu\. The critical\-dimension conclusion follows immediately\. ∎

###### Lemma A\.1\(Gaussian maximum and chi concentration\)\.

LetZ1,…,ZMZ\_\{1\},\\ldots,Z\_\{M\}be independentN⁡\(0,1\)N\(0,1\)variables and letCd∼χdC\_\{d\}\\sim\\chi\_\{d\}\. AsM,d→∞M,d\\to\\infty,

maxj≤M⁡Zj2​log⁡M→1,Cdd→1\\frac\{\\max\_\{j\\leq M\}Z\_\{j\}\}\{\\sqrt\{2\\log M\}\}\\to 1,\\qquad\\frac\{C\_\{d\}\}\{\\sqrt\{d\}\}\\to 1\(38\)in probability\.

###### Proof\.

For the upper Gaussian bound, for everyε\>0\\varepsilon\>0, the union bound andℙ\(Z≥t\)≤e−t2/2\\mathbb\{P\}\(Z\\geq t\)\\leq e^\{\-t^\{2\}/2\}give

ℙ⁡\(maxj≤M⁡Zj\>\(1\+ε\)​2​log⁡M\)\\displaystyle\\mathbb\{P\}\\left\(\\max\_\{j\\leq M\}Z\_\{j\}\>\(1\+\\varepsilon\)\\sqrt\{2\\log M\}\\right\)≤M​exp⁡\(−\(1\+ε\)2​log⁡M\)\\displaystyle\\leq M\\exp\(\-\(1\+\\varepsilon\)^\{2\}\\log M\)=M−2​ε−ε2→0\.\\displaystyle=M^\{\-2\\varepsilon\-\\varepsilon^\{2\}\}\\to 0\.\(39\)For the lower bound, lett=\(1−ε\)​2​log⁡Mt=\(1\-\\varepsilon\)\\sqrt\{2\\log M\}\. The Gaussian Mills lower bound gives, for a universalc\>0c\>0and all sufficiently largett,

ℙ⁡\(Z\>t\)≥c​e−t2/2t\.\\mathbb\{P\}\(Z\>t\)\\geq c\\frac\{e^\{\-t^\{2\}/2\}\}\{t\}\.\(40\)ThereforeM​ℙ​\(Z\>t\)→∞M\\mathbb\{P\}\(Z\>t\)\\to\\infty, and independence yields

ℙ⁡\(maxj≤M⁡Zj≤t\)=\(1−ℙ⁡\(Z\>t\)\)M≤e−M​ℙ​\(Z\>t\)→0\.\\mathbb\{P\}\\left\(\\max\_\{j\\leq M\}Z\_\{j\}\\leq t\\right\)=\(1\-\\mathbb\{P\}\(Z\>t\)\)^\{M\}\\leq e^\{\-M\\mathbb\{P\}\(Z\>t\)\}\\to 0\.\(41\)This proves the first convergence\. For the second,Cd2/dC\_\{d\}^\{2\}/dis the average ofddindependent squared standard Gaussians, so it converges to one in probability by the weak law of large numbers\. The square root is continuous on\(0,∞\)\(0,\\infty\)\. ∎

###### Proof of Theorem[4\.2](https://arxiv.org/html/2609.13747#S4.Thmtheorem2)\.

Divide the two sides of \([17](https://arxiv.org/html/2609.13747#S4.E17)\) by2​log⁡M\\sqrt\{2\\log M\}\. Lemma[A\.1](https://arxiv.org/html/2609.13747#A1.Thmtheorem1)gives

maxj≤M⁡Zj2​log⁡M→1\.\\frac\{\\max\_\{j\\leq M\}Z\_\{j\}\}\{\\sqrt\{2\\log M\}\}\\to 1\.\(42\)Ifλ=0\\lambda=0, then

𝔼⁡\[\(μM​CdM2​log⁡M\)2\]=dM​μM22​log⁡M→0,\\mathbb\{E\}\\left\[\\left\(\\frac\{\\mu\_\{M\}C\_\{d\_\{M\}\}\}\{\\sqrt\{2\\log M\}\}\\right\)^\{2\}\\right\]=\\frac\{d\_\{M\}\\mu\_\{M\}^\{2\}\}\{2\\log M\}\\to 0,\(43\)so the aligned term converges to zero in probability without requiringdM→∞d\_\{M\}\\to\\infty\.

Now supposeλ∈\(0,∞\]\\lambda\\in\(0,\\infty\]\. SinceμM≤1\\mu\_\{M\}\\leq 1, the assumed ratio impliesdM→∞d\_\{M\}\\to\\infty, and The aligned term satisfies

μM​CdM2​log⁡M\\displaystyle\\frac\{\\mu\_\{M\}C\_\{d\_\{M\}\}\}\{\\sqrt\{2\\log M\}\}=dM​μM22​log⁡M​CdMdM→λ,\\displaystyle=\\sqrt\{\\frac\{d\_\{M\}\\mu\_\{M\}^\{2\}\}\{2\\log M\}\}\\frac\{C\_\{d\_\{M\}\}\}\{\\sqrt\{d\_\{M\}\}\}\\to\\sqrt\{\\lambda\},\(44\)where convergence to\+∞\+\\inftyis understood whenλ=∞\\lambda=\\infty\. Because\|1−μM2\|≤1\|\\sqrt\{1\-\\mu\_\{M\}^\{2\}\}\|\\leq 1and a standard Gaussian is tight,

1−μM2​Z02​log⁡M→0\\frac\{\\sqrt\{1\-\\mu\_\{M\}^\{2\}\}Z\_\{0\}\}\{\\sqrt\{2\\log M\}\}\\to 0\(45\)in probability\. If1<λ<∞1<\\lambda<\\infty, choose a constant strictly between11andλ\\sqrt\{\\lambda\}; the true score lies above that constant and the maximum decoy below it with probability tending to one\. If0<λ<10<\\lambda<1, choose a constant strictly betweenλ\\sqrt\{\\lambda\}and11and reverse the argument\. The casesλ=0\\lambda=0andλ=∞\\lambda=\\inftyfollow from the two endpoint limits above\. This proves \([19](https://arxiv.org/html/2609.13747#S4.E19)\)\. ∎

###### Proof of Corollary[4\.4](https://arxiv.org/html/2609.13747#S4.Thmtheorem4)\.

For fixedMMandμ\>0\\mu\>0, the success probability in \([17](https://arxiv.org/html/2609.13747#S4.E17)\) is nondecreasing indd\. To see this, coupleCd\+1=Cd2\+Z2C\_\{d\+1\}=\\sqrt\{C\_\{d\}^\{2\}\+Z^\{2\}\}with an additional independent standard GaussianZZ; thenCd\+1≥CdC\_\{d\+1\}\\geq C\_\{d\}almost surely, while all other variables in \([17](https://arxiv.org/html/2609.13747#S4.E17)\) are unchanged\.

Fixε\>0\\varepsilon\>0\. Theorem[4\.2](https://arxiv.org/html/2609.13747#S4.Thmtheorem2)implies that at dimensions

d−=\(1−ε\)​2​log⁡MμM2,d\+=\(1\+ε\)​2​log⁡MμM2,d\_\{\-\}=\(1\-\\varepsilon\)\\frac\{2\\log M\}\{\\mu\_\{M\}^\{2\}\},\\qquad d\_\{\+\}=\(1\+\\varepsilon\)\\frac\{2\\log M\}\{\\mu\_\{M\}^\{2\}\},\(46\)the success probabilities tend to zero and one, respectively\. For all sufficiently largeMM, the critical dimension lies betweend−d\_\{\-\}andd\+d\_\{\+\}\. Lettingε↓0\\varepsilon\\downarrow 0proves \([20](https://arxiv.org/html/2609.13747#S4.E20)\)\. Solving the same transition relation forlog⁡M\\log Mgives \([22](https://arxiv.org/html/2609.13747#S4.E22)\)\. ∎

## Appendix BProofs for Frontier, Cumulative, and Weighted States

###### Proof of Theorem[5\.1](https://arxiv.org/html/2609.13747#S5.Thmtheorem1)\.

Fixu∈𝒰u\\in\\mathcal\{U\}\. Nonnegativity, unit norm, and \([14](https://arxiv.org/html/2609.13747#S3.E14)\) give

1=‖u‖22≤‖u‖∞​‖u‖1≤Cm​‖u‖1\.1=\\left\\\|u\\right\\\|\_\{2\}^\{2\}\\leq\\left\\\|u\\right\\\|\_\{\\infty\}\\left\\\|u\\right\\\|\_\{1\}\\leq\\frac\{C\}\{\\sqrt\{m\}\}\\left\\\|u\\right\\\|\_\{1\}\.\(47\)Therefore

⟨aR,u⟩=‖u‖1m≥1C\.\\langle a\_\{R\},u\\rangle=\\frac\{\\left\\\|u\\right\\\|\_\{1\}\}\{\\sqrt\{m\}\}\\geq\\frac\{1\}\{C\}\.\(48\)For frontier,

⟨aF,u⟩=1k​∑v∈Fuv≤k​‖u‖∞≤C​km\.\\langle a\_\{F\},u\\rangle=\\frac\{1\}\{\\sqrt\{k\}\}\\sum\_\{v\\in F\}u\_\{v\}\\leq\\sqrt\{k\}\\left\\\|u\\right\\\|\_\{\\infty\}\\leq C\\sqrt\{\\frac\{k\}\{m\}\}\.\(49\)Taking infima proves \([23](https://arxiv.org/html/2609.13747#S5.E23)\)\.

Compactness imply that the worst\-query success probability under stateaais exactlypd,M​\(Γ​\(a,𝒰\)\)p\_\{d,M\}\(\\Gamma\(a;\\mathcal\{U\}\)\)\. IfΓ⁡\(aF,𝒰\)\>0\\Gamma\(a\_\{F\};\\mathcal\{U\}\)\>0, the uniform critical\-dimension law gives

D⋆​\(aR,𝒰,M,δ\)D⋆​\(aF,𝒰,M,δ\)=Γ​\(aF,𝒰\)2Γ​\(aR,𝒰\)2​\(1\+o⁡\(1\)\)≤C4​km​\(1\+o⁡\(1\)\)\.\\frac\{D\_\{\\star\}\(a\_\{R\};\\mathcal\{U\},M,\\delta\)\}\{D\_\{\\star\}\(a\_\{F\};\\mathcal\{U\},M,\\delta\)\}=\\frac\{\\Gamma\(a\_\{F\};\\mathcal\{U\}\)^\{2\}\}\{\\Gamma\(a\_\{R\};\\mathcal\{U\}\)^\{2\}\}\(1\+o\(1\)\)\\leq C^\{4\}\\frac\{k\}\{m\}\(1\+o\(1\)\)\.\(50\)The remainder is uniform over positive alignments by the asymptotic convention\. IfΓ⁡\(aF,𝒰\)=0\\Gamma\(a\_\{F\};\\mathcal\{U\}\)=0, compactness supplies a query with zero frontier alignment; its correct answer is exchangeable with theMMdecoys, giving the stated infinite critical dimension\.

It remains to prove sharpness\. Suppose first thats=m/C2s=m/C^\{2\}is an integer andk≤sk\\leq s\. ChooseS⊆RS\\subseteq Rwith\|S\|=s\|S\|=sandF⊆SF\\subseteq S, and letuv=C/mu\_\{v\}=C/\\sqrt\{m\}onSSand zero elsewhere\. Thenuuis a nonnegative unit vector satisfying \([14](https://arxiv.org/html/2609.13747#S3.E14)\)\. For the compact singleton class𝒰=\{u\}\\mathcal\{U\}=\\\{u\\\},

Γ⁡\(aR,𝒰\)=1C,Γ⁡\(aF,𝒰\)=C​km,\\Gamma\(a\_\{R\};\\mathcal\{U\}\)=\\frac\{1\}\{C\},\\qquad\\Gamma\(a\_\{F\};\\mathcal\{U\}\)=C\\sqrt\{\\frac\{k\}\{m\}\},\(51\)so equality holds in \([24](https://arxiv.org/html/2609.13747#S5.E24)\)\. Ifm/C2m/C^\{2\}is not an integer, use⌊m/C2⌋\\lfloor m/C^\{2\}\\rfloorcoordinates at the cap and one residual coordinate\. The rounding changes the ratio by1\+o⁡\(1\)1\+o\(1\)wheneverm/C2→∞m/C^\{2\}\\to\\infty\. The conditionk≤m/C2k\\leq m/C^\{2\}holds eventually in every fixed\-CCgrowing\-history regime withk=o⁡\(m\)k=o\(m\)\. ∎

It follows by takingu=aRu=a\_\{R\}: then⟨aR,u⟩=1\\langle a\_\{R\},u\\rangle=1and⟨aF,u⟩=k/m\\langle a\_\{F\},u\\rangle=\\sqrt\{k/m\}, and substitution into Corollary[4\.4](https://arxiv.org/html/2609.13747#S4.Thmtheorem4)gives the two dimensions\.

###### Proof of Theorem[5\.3](https://arxiv.org/html/2609.13747#S5.Thmtheorem3)\.

We use the standard finite\-depth convention for this construction: fixDDandδ∈\(0,1\)\\delta\\in\(0,1\), letn→∞n\\to\\infty, omit empty transitions, and start from the exact intended stateq0q\_\{0\}\. For the reusable weighted sequence, letIc⊆V∖RcI\_\{c\}\\subseteq V\\setminus R\_\{c\}be theΘ⁡\(n\)\\Theta\(n\)independent inactive source keys exposed at layercc, and define

ℰroute\(d\):=⋂c<D\{minv∈Rc⟨qac,gv⟩\>τc\>maxs∈Ic⟨qac,gs⟩\}\.\\mathcal\{E\}\_\{\\rm route\}\(d\):=\\bigcap\_\{c<D\}\\left\\\{\\min\_\{v\\in R\_\{c\}\}\\langle q\_\{a\_\{c\}\},g\_\{v\}\\rangle\>\\tau\_\{c\}\>\\max\_\{s\\in I\_\{c\}\}\\langle q\_\{a\_\{c\}\},g\_\{s\}\\rangle\\right\\\}\.\(52\)Heredroute​\(𝒂,δ\)d\_\{\\rm route\}\(\\boldsymbol\{a\},\\delta\)is the leastddfor which some thresholds\(τc\)c<D\(\\tau\_\{c\}\)\_\{c<D\}makeℙ⁡\(ℰroute​\(d\)\)≥1−δ\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\rm route\}\(d\)\)\\geq 1\-\\delta\. Frontier uses the identical definition with its required sourcesFcF\_\{c\}in place ofRcR\_\{c\}; the full\-history version above is the one used by Theorem[5\.5](https://arxiv.org/html/2609.13747#S5.Thmtheorem5)\.

We first prove the routing bounds\. Fix a layer and abbreviatea=aca=a\_\{c\},α=amin,c\\alpha=a\_\{\\min,c\}, andY=G​aY=Ga\. For every required sourcev∈Rcv\\in R\_\{c\}, Gaussian regression withev=av​a\+\(ev−av​a\)e\_\{v\}=a\_\{v\}a\+\(e\_\{v\}\-a\_\{v\}a\)gives

d​⟨qa,gv⟩​=𝑑​av​Cd\+1−av2​Zv,\\sqrt\{d\}\\langle q\_\{a\},g\_\{v\}\\rangle\\overset\{d\}\{=\}a\_\{v\}C\_\{d\}\+\\sqrt\{1\-a\_\{v\}^\{2\}\}\\,Z\_\{v\},\(53\)whereCd=d​‖Y‖∼χdC\_\{d\}=\\sqrt\{d\}\\left\\\|Y\\right\\\|\\sim\\chi\_\{d\}and eachZvZ\_\{v\}is marginally standard normal\. The activeZvZ\_\{v\}need not be mutually independent, which is irrelevant for the union bound\. For every inactives∈Ics\\in I\_\{c\},es⟂ae\_\{s\}\\perp a, so

d​⟨qa,gs⟩∼N⁡\(0,1\),\\sqrt\{d\}\\langle q\_\{a\},g\_\{s\}\\rangle\\sim N\(0,1\),\(54\)and these variables are independent withinIcI\_\{c\}\.

Chooseτc=3​α/8\\tau\_\{c\}=3\\alpha/8\. On the event

Cdd≥34,minv∈Rc⁡Zv≥−α​d4,maxs∈Ic⁡Zs≤α​d4,\\frac\{C\_\{d\}\}\{\\sqrt\{d\}\}\\geq\\frac\{3\}\{4\},\\quad\\min\_\{v\\in R\_\{c\}\}Z\_\{v\}\\geq\-\\frac\{\\alpha\\sqrt\{d\}\}\{4\},\\quad\\max\_\{s\\in I\_\{c\}\}Z\_\{s\}\\leq\\frac\{\\alpha\\sqrt\{d\}\}\{4\},\(55\)every required score is at leastα/2\\alpha/2and every inactive score is at mostα/4\\alpha/4\. Standard chi\-square and Gaussian tails therefore give universalA,c\>0A,c\>0such that

ℙ⁡\(ℰroute​\(d\)c\)≤A​∑c<Dn​exp⁡\(−c​d​amin,c2\)\.\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\rm route\}\(d\)^\{c\}\)\\leq A\\sum\_\{c<D\}n\\exp\(\-cda\_\{\\min,c\}^\{2\}\)\.\(56\)Takingd≥C0​maxc<D​log⁡\(D​n/δ\)/amin,c2d\\geq C\_\{0\}\\max\_\{c<D\}\\log\(Dn/\\delta\)/a\_\{\\min,c\}^\{2\}in \([56](https://arxiv.org/html/2609.13747#A2.E56)\), withC0C\_\{0\}sufficiently large, proves the upper bound in \([28](https://arxiv.org/html/2609.13747#S5.E28)\)\.

For the lower bound, at each layer selectvcv\_\{c\}with coefficientac,vc=amin,ca\_\{c,v\_\{c\}\}=a\_\{\\min,c\}\. Successful common\-threshold routing necessarily implies

⟨qac,gvc⟩\>maxs∈Ic⁡⟨qac,gs⟩\.\\langle q\_\{a\_\{c\}\},g\_\{v\_\{c\}\}\\rangle\>\\max\_\{s\\in I\_\{c\}\}\\langle q\_\{a\_\{c\}\},g\_\{s\}\\rangle\.\(57\)This is exactly the readout event of Section[4](https://arxiv.org/html/2609.13747#S4)with useful directionevce\_\{v\_\{c\}\}, alignmentamin,ca\_\{\\min,c\}, and\|Ic\|=Θ⁡\(n\)\|I\_\{c\}\|=\\Theta\(n\)orthogonal competitors\. Corollary[4\.4](https://arxiv.org/html/2609.13747#S4.Thmtheorem4)at fixedδ\\deltayields

droute≥\(2−o⁡\(1\)\)​maxc<D​log⁡namin,c2,d\_\{\\rm route\}\\geq\(2\-o\(1\)\)\\max\_\{c<D\}\\frac\{\\log n\}\{a\_\{\\min,c\}^\{2\}\},\(58\)For fixedDDandδ\\delta,log⁡\(D​n/δ\)=Θ⁡\(log⁡n\)\\log\(Dn/\\delta\)=\\Theta\(\\log n\), so this lower bound and \([56](https://arxiv.org/html/2609.13747#A2.E56)\) prove \([28](https://arxiv.org/html/2609.13747#S5.E28)\)\. If a required coefficient is zero, its score is exchangeable with the inactive scores and its success is at most1/\(\|Ic\|\+1\)1/\(\|I\_\{c\}\|\+1\); hence the critical dimension is infinite whenever the target success exceeds this chance level, as it eventually does for fixedδ\\delta\. For a uniform state onssobjects,amin,c=s−1/2a\_\{\\min,c\}=s^\{\-1/2\}, giving the final assertion\.

We next realize the update\. The argument below uses the full\-history event \([55](https://arxiv.org/html/2609.13747#A2.E55)\)\. For the frontier transition, replaceRcR\_\{c\}byFcF\_\{c\}in that event and in the preceding tail calculation; because its state is uniform onFcF\_\{c\}, the same calculation applies withamin,c=\|Fc\|−1/2a\_\{\\min,c\}=\|F\_\{c\}\|^\{\-1/2\}\. Augment every token byO⁡\(1\)O\(1\)constant coordinates\. In the first block an edge query\[gs;1\]\[g\_\{s\};1\]attends to the latent key\[qc;0\]\[q\_\{c\};0\]and threshold key\[0;τc\]\[0;\\tau\_\{c\}\]; assigning scalar values one and zero writeshe=σ⁡\(β⁡\(⟨qc,gs⟩−τc\)\)h\_\{e\}=\\sigma\(\\beta\(\\langle q\_\{c\},g\_\{s\}\\rangle\-\\tau\_\{c\}\)\)\. On \([55](https://arxiv.org/html/2609.13747#A2.E55)\) the raw gap is at leastγc:=amin,c/8\\gamma\_\{c\}:=a\_\{\\min,c\}/8\. Thus active gates are1\+O⁡\(e−β​γc\)1\+O\(e^\{\-\\beta\\gamma\_\{c\}\}\)and inactive gates areO⁡\(e−β​γc\)O\(e^\{\-\\beta\\gamma\_\{c\}\}\)\.

The second block has a constant\-query destination head with scalar gate keys and valuesgtg\_\{t\}, together with a self\-copy head returningqcq\_\{c\}\. Denote its raw destination average byo¯c\\bar\{o\}\_\{c\}\. If its temperature isTT, the inactive attention mass is bounded by

∑e∈E−αe≤\|E−\|\|E\+\|​exp⁡\{−T⁡\(1−O⁡\(e−β​γc\)\)\}\.\\sum\_\{e\\in E\_\{\-\}\}\\alpha\_\{e\}\\leq\\frac\{\|E\_\{\-\}\|\}\{\|E\_\{\+\}\|\}\\exp\\\{\-T\(1\-O\(e^\{\-\\beta\\gamma\_\{c\}\}\)\)\\\}\.\(59\)Letα⋆\\alpha\_\{\\star\}denote the smallest required\-source coefficient across the fixed computation \(using\|Fc\|−1/2\|F\_\{c\}\|^\{\-1/2\}for the frontier replacement just described\)\. Chooseβ,T\\beta,Tso thatT​e−β​γc→0Te^\{\-\\beta\\gamma\_\{c\}\}\\to 0andT−log⁡\|E\|→∞T\-\\log\|E\|\\to\\infty\. Sinced≳log⁡nd\\gtrsim\\log nalso impliesmaxv⁡‖gv‖=Op​\(1\)\\max\_\{v\}\\left\\\|g\_\{v\}\\right\\\|=O\_\{p\}\(1\), the temperatures may be increased further so that, uniformly over the fixed number of layers,

‖rc​o¯c−1rc​XN‖=op​\(α⋆\),XN:=∑t∈Ncgt\.\\left\\\|\\sqrt\{r\_\{c\}\}\\,\\bar\{o\}\_\{c\}\-\\frac\{1\}\{\\sqrt\{r\_\{c\}\}\}X\_\{N\}\\right\\\|=o\_\{p\}\(\\alpha\_\{\\star\}\),\\qquad X\_\{N\}:=\\sum\_\{t\\in N\_\{c\}\}g\_\{t\}\.\(60\)Thus, after the output scaling,oc:=rc​o¯co\_\{c\}:=\\sqrt\{r\_\{c\}\}\\bar\{o\}\_\{c\}equals theoco\_\{c\}in \([26](https://arxiv.org/html/2609.13747#S5.E26)\), with a rate strong enough for both coefficient choices in \([27](https://arxiv.org/html/2609.13747#S5.E27)\)\.

The output projection combines the two heads as\(λc−1\)​qc\+ηc​oc\(\\lambda\_\{c\}\-1\)q\_\{c\}\+\\eta\_\{c\}o\_\{c\}; the residual adds backqcq\_\{c\}\. Directional RMSNorm means the usual positive scalar RMS normalization \(with zero stabilization for exact unit normalization\); its stabilization and gain do not change the represented direction\. Settingλc=0\\lambda\_\{c\}=0leaves the normalized new frontier\. For cumulative, letXR=∑v∈RcgvX\_\{R\}=\\sum\_\{v\\in R\_\{c\}\}g\_\{v\}and useXNX\_\{N\}from \([60](https://arxiv.org/html/2609.13747#A2.E60)\)\. Since

‖XR‖\|Rc\|=1\+Op\(d−1/2\),\\frac\{\\left\\\|X\_\{R\}\\right\\\|\}\{\\sqrt\{\|R\_\{c\}\|\}\}=1\+O\_\{p\}\(d^\{\-1/2\}\),\(61\)the pre\-normalization vector satisfies

‖\|Rc\|​qRc\+rc​oc−\(XR\+XN\)‖‖XR\+XN‖=Op\(d−1/2\)\+op\(α⋆\)\.\\frac\{\\left\\\|\\sqrt\{\|R\_\{c\}\|\}q\_\{R\_\{c\}\}\+\\sqrt\{r\_\{c\}\}o\_\{c\}\-\(X\_\{R\}\+X\_\{N\}\)\\right\\\|\}\{\\left\\\|X\_\{R\}\+X\_\{N\}\\right\\\|\}=O\_\{p\}\(d^\{\-1/2\}\)\+o\_\{p\}\(\\alpha\_\{\\star\}\)\.\(62\)The normalization map obeys‖x/‖x‖−y/‖y‖‖≤2​‖x−y‖/‖y‖\\left\\\|x/\\left\\\|x\\right\\\|\-y/\\left\\\|y\\right\\\|\\right\\\|\\leq 2\\left\\\|x\-y\\right\\\|/\\left\\\|y\\right\\\|whenever‖x−y‖≤‖y‖/2\\left\\\|x\-y\\right\\\|\\leq\\left\\\|y\\right\\\|/2; hence RMSNorm returnsqRc\+1\+op​\(α⋆\)q\_\{R\_\{c\+1\}\}\+o\_\{p\}\(\\alpha\_\{\\star\}\)\. Indeed,d​α⋆2≳log⁡nd\\alpha\_\{\\star\}^\{2\}\\gtrsim\\log nmakesd−1/2=o\(α⋆\)d^\{\-1/2\}=o\(\\alpha\_\{\\star\}\); hence the state error is smaller than every subsequent membership gap\. Induction over fixedDDtherefore preserves both routing and directional accuracy and proves simultaneous realization\. Other nonnegative output coefficients give the stated layer\-weighted recursion\. All operations are attention, a linear output projection, a residual, or RMSNorm, so no FFN is needed\. ∎

###### Proof of Theorem[5\.5](https://arxiv.org/html/2609.13747#S5.Thmtheorem5)\.

WriteL:=log⁡\(D​n/δ\)L:=\\log\(Dn/\\delta\)andB:=2​log⁡MB:=2\\log M, and interpret a zero alignment or reusable coefficient as infinite width\. For everyc<Dc<Dand everyac∈𝒜Rca\_\{c\}\\in\\mathcal\{A\}\_\{R\_\{c\}\},

\(minv∈Rc⁡ac,v\)2≤1\|Rc\|​∑v∈Rcac,v2=1\|Rc\|,\\left\(\\min\_\{v\\in R\_\{c\}\}a\_\{c,v\}\\right\)^\{2\}\\leq\\frac\{1\}\{\|R\_\{c\}\|\}\\sum\_\{v\\in R\_\{c\}\}a\_\{c,v\}^\{2\}=\\frac\{1\}\{\|R\_\{c\}\|\},\(63\)with equality only atac=aRca\_\{c\}=a\_\{R\_\{c\}\}\. Hence the routing part of𝔇⁡\(𝒂R,𝒰\)\\mathfrak\{D\}\(\\boldsymbol\{a\}\_\{R\};\\mathcal\{U\}\)is no larger than that of any𝔇⁡\(𝒂,𝒰\)\\mathfrak\{D\}\(\\boldsymbol\{a\};\\mathcal\{U\}\)\.

For the readout part, Theorem[5\.1](https://arxiv.org/html/2609.13747#S5.Thmtheorem1)givesΓ⁡\(aRD,𝒰\)≥1/C\\Gamma\(a\_\{R\_\{D\}\};\\mathcal\{U\}\)\\geq 1/C\. Cauchy–Schwarz givesΓ⁡\(aD,𝒰\)≤1\\Gamma\(a\_\{D\};\\mathcal\{U\}\)\\leq 1for every unitaDa\_\{D\}, so

BΓ​\(aRD,𝒰\)2≤C2​B≤C2​BΓ​\(aD,𝒰\)2\.\\frac\{B\}\{\\Gamma\(a\_\{R\_\{D\}\};\\mathcal\{U\}\)^\{2\}\}\\leq C^\{2\}B\\leq C^\{2\}\\frac\{B\}\{\\Gamma\(a\_\{D\};\\mathcal\{U\}\)^\{2\}\}\.\(64\)Combining the routing inequality and \([64](https://arxiv.org/html/2609.13747#A2.E64)\), and usingC≥1C\\geq 1, yields

𝔇⁡\(𝒂R,𝒰\)≤C2​𝔇​\(𝒂,𝒰\)\\mathfrak\{D\}\(\\boldsymbol\{a\}\_\{R\};\\mathcal\{U\}\)\\leq C^\{2\}\\mathfrak\{D\}\(\\boldsymbol\{a\};\\mathcal\{U\}\)\(65\)for every weighted sequence𝒂\\boldsymbol\{a\}\. Taking the infimum proves \([30](https://arxiv.org/html/2609.13747#S5.E30)\)\.

The factor is sharp\. Supposes:=\|RD\|/C2s:=\|R\_\{D\}\|/C^\{2\}is an integer, chooseS⊆RDS\\subseteq R\_\{D\}with\|S\|=s\|S\|=s, and letu=𝟏S/su=\\mathbf\{1\}\_\{S\}/\\sqrt\{s\}and𝒰=\{u\}\\mathcal\{U\}=\\\{u\\\}\. This class isCC\-nonlocalized\. Cumulative has terminal alignment1/C1/C, whereas the weighted terminal stateaD=ua\_\{D\}=uhas alignment one\. Keep all earlier states cumulative\. Whenever the readout scaleBBis at least the cumulative routing scaleL​maxc<D​\|Rc\|L\\max\_\{c<D\}\|R\_\{c\}\|, the two objectives are respectivelyC2​BC^\{2\}BandBB, so equality holds in \([30](https://arxiv.org/html/2609.13747#S5.E30)\)\.

It remains to prove exact robust optimality\. Putm:=\|RD\|m:=\|R\_\{D\}\|and letu⋆u\_\{\\star\}minimize‖u‖1\\left\\\|u\\right\\\|\_\{1\}over the nonempty compact set𝒰C​\(RD\)\\mathcal\{U\}\_\{C\}\(R\_\{D\}\)\. This full class is closed under every coordinate permutationPPofRDR\_\{D\}\. Hence, for anyaD∈𝒜RDa\_\{D\}\\in\\mathcal\{A\}\_\{R\_\{D\}\},

Γ⁡\(aD,𝒰C​\(RD\)\)\\displaystyle\\Gamma\(a\_\{D\};\\mathcal\{U\}\_\{C\}\(R\_\{D\}\)\)≤1m\!​∑P⟨aD,P​u⋆⟩=‖aD‖1​‖u⋆‖1m\\displaystyle\\leq\\frac\{1\}\{m\!\}\\sum\_\{P\}\\langle a\_\{D\},Pu\_\{\\star\}\\rangle=\\frac\{\\left\\\|a\_\{D\}\\right\\\|\_\{1\}\\left\\\|u\_\{\\star\}\\right\\\|\_\{1\}\}\{m\}≤‖u⋆‖1m=Γ⁡\(aRD,𝒰C​\(RD\)\)\.\\displaystyle\\leq\\frac\{\\left\\\|u\_\{\\star\}\\right\\\|\_\{1\}\}\{\\sqrt\{m\}\}=\\Gamma\(a\_\{R\_\{D\}\};\\mathcal\{U\}\_\{C\}\(R\_\{D\}\)\)\.\(66\)The second inequality uses‖aD‖1≤m​‖aD‖2=m\\left\\\|a\_\{D\}\\right\\\|\_\{1\}\\leq\\sqrt\{m\}\\left\\\|a\_\{D\}\\right\\\|\_\{2\}=\\sqrt\{m\}; the final equality follows because⟨aRD,u⟩=‖u‖1/m\\langle a\_\{R\_\{D\}\},u\\rangle=\\left\\\|u\\right\\\|\_\{1\}/\\sqrt\{m\}\. ThusaRDa\_\{R\_\{D\}\}minimizes the robust readout term\. Equation \([63](https://arxiv.org/html/2609.13747#A2.E63)\) shows that every earlieraRca\_\{R\_\{c\}\}minimizes its routing term\. The cumulative sequence therefore minimizes their maximum, proving \([31](https://arxiv.org/html/2609.13747#S5.E31)\)\. ∎

## Appendix CExact Reduction for an Arbitrary Fixed Bank

The orthogonal bank makes the main dimension law transparent\. The following identity shows that the same signal\-versus\-competition structure persists for an arbitrary fixed correlated bank\.

###### Theorem C\.1\(Arbitrary\-bank Gaussian reduction\)\.

Fix a unit stateaaand deterministic candidate directionsu0,u1,…,uMu\_\{0\},u\_\{1\},\\ldots,u\_\{M\}, independent ofGG\. For eachj≥1j\\geq 1, let

hj:=u0−uj,γj:=⟨a,hj⟩\>0,rj:=hj−γj​a∈a⟂\.h\_\{j\}:=u\_\{0\}\-u\_\{j\},\\qquad\\gamma\_\{j\}:=\\langle a,h\_\{j\}\\rangle\>0,\\qquad r\_\{j\}:=h\_\{j\}\-\\gamma\_\{j\}a\\in a^\{\\perp\}\.\(67\)Letg∼N⁡\(0,In\)g\\sim N\(0,I\_\{n\}\)andCd∼χdC\_\{d\}\\sim\\chi\_\{d\}be independent\. Then

ℙ\(j^\(qa\)=0\)=ℙ\(max1≤j≤M⟨g,−rj/γj⟩<Cd\)\.\\mathbb\{P\}\(\\widehat\{j\}\(q\_\{a\}\)=0\)=\\mathbb\{P\}\\left\(\\max\_\{1\\leq j\\leq M\}\\langle g,\-r\_\{j\}/\\gamma\_\{j\}\\rangle<C\_\{d\}\\right\)\.\(68\)

###### Proof\.

The correct candidate beats candidatejjexactly when

0\\displaystyle 0<⟨G​a,G⁡\(u0−uj\)⟩\\displaystyle<\\langle Ga,G\(u\_\{0\}\-u\_\{j\}\)\\rangle=γj​‖G​a‖2\+⟨G​a,G​rj⟩\.\\displaystyle=\\gamma\_\{j\}\\left\\\|Ga\\right\\\|^\{2\}\+\\langle Ga,Gr\_\{j\}\\rangle\.\(69\)Sinceγj\>0\\gamma\_\{j\}\>0, simultaneous success is equivalent to

maxj≤M⟨Ga,G\(−rj/γj\)⟩<‖Ga‖2\.\\max\_\{j\\leq M\}\\langle Ga,G\(\-r\_\{j\}/\\gamma\_\{j\}\)\\rangle<\\left\\\|Ga\\right\\\|^\{2\}\.\(70\)Everyrjr\_\{j\}is orthogonal toaa\. Joint Gaussianity therefore makesG​aGaindependent of the family\{G\(−rj/γj\)\}j≤M\\\{G\(\-r\_\{j\}/\\gamma\_\{j\}\)\\\}\_\{j\\leq M\}\. Conditional onY:=G​aY:=Ga, the process on the left of \([70](https://arxiv.org/html/2609.13747#A3.E70)\) has the same law as

‖Y‖d\{⟨g,−rj/γj⟩\}j≤M\.\\frac\{\\left\\\|Y\\right\\\|\}\{\\sqrt\{d\}\}\\left\\\{\\langle g,\-r\_\{j\}/\\gamma\_\{j\}\\rangle\\right\\\}\_\{j\\leq M\}\.\(71\)Canceling‖Y‖\>0\\left\\\|Y\\right\\\|\>0and usingd​‖Y‖∼χd\\sqrt\{d\}\\left\\\|Y\\right\\\|\\sim\\chi\_\{d\}proves \([68](https://arxiv.org/html/2609.13747#A3.E68)\)\. ∎

Theorem[C\.1](https://arxiv.org/html/2609.13747#A3.Thmtheorem1)requires no semantic choice of query and no orthogonality\. Correlation changes only the maximum of the fixed Gaussian family\. In the isotropic model, that maximum is\(1/μ\)​maxj≤M​Zj\(1/\\mu\)\\max\_\{j\\leq M\}Z\_\{j\}up to a common centered Gaussian term, recovering Theorem[4\.2](https://arxiv.org/html/2609.13747#S4.Thmtheorem2)\. More generally, the critical dimension is the squared typical scale of the Gaussian maximum in \([68](https://arxiv.org/html/2609.13747#A3.E68)\)\. The frontier–cumulative comparison remains a comparison under the same bank; the simplifiedlog⁡M/μ2\\log M/\\mu^\{2\}form is obtained when the competition geometry is common to both states\.

## Appendix DExperimental Details and Additional Results

### D\.1Dataset and query construction

We use the public graph\-4 ProsQA splits released with[Zhu et al\. \(2025b\)](https://arxiv.org/html/2609.13747#bib.bib7)\. Table[1](https://arxiv.org/html/2609.13747#A4.T1)records the full split statistics\. The paired width sweep uses every training and test trace and a fixed 120\-trace validation subset\.

Table 1:ProsQA graph\-4 statistics\. Graph sizes and solution lengths are averaged over traces\.Queries are sampled in coefficient space before the Gaussian embedding\. Each query contains the complete terminal frontier but remains strictly partial, uses between75%75\\%and85\.7%85\.7\\%of the reached history, and has within\-support coefficient ratio at mostρ=1\.2\\rho=1\.2\. These choices imply the certified capC=1\.386C=1\.386; the largest realized caps are smaller on every split\. The same query, Gaussian draw, useful candidate, and decoy bank are reused for frontier and cumulative\.

Table 2:Audit of the fixed nonlocalized query bank\. “Queries” counts coefficient\-space queries before repeated Gaussian test draws\.
### D\.2Paired width\-sweep configuration

The learned construction is the two\-attention router in Theorem[5\.3](https://arxiv.org/html/2609.13747#S5.Thmtheorem3)\. Its source query and key projections are initialized asI\+0\.08​Z/dI\+0\.08Z/\\sqrt\{d\}; the membership inverse temperature and destination temperature are initialized to88and1010and trained jointly\. Frontier and cumulative share all router parameters and optimizer updates\. The best epoch is selected by the sum of their validation state cosines\.

Table 3:Configuration of the formal paired experiment\.At test time, success is top\-1 retrieval of the fixed useful candidate against 4,096 decoys\. Each training seed is evaluated on 419 traces, eight queries per trace, and four Gaussian draws\. Statistical intervals are therefore computed across the five independent training seeds, rather than by treating the within\-seed evaluations as independent replications\. During training, each trace uses one query per epoch, cycling deterministically through the pre\-generated eight\-query bank; hence four epochs expose four query IDs per trace\.

### D\.3Complete fixed\-query results

Table[4](https://arxiv.org/html/2609.13747#A4.T4)gives the complete width sweep underlying Figure[3](https://arxiv.org/html/2609.13747#S6.F3)\. Intervals are two\-sided 95% Student\-ttintervals across five training seeds\. “Ideal” supplies the analytic state to the same decoder and is not an additional trained model\.

Table 4:Held\-out top\-1 accuracy at every tested width\.The threshold is stable at the individual\-seed level \(Table[5](https://arxiv.org/html/2609.13747#A4.T5)\)\. The learned cumulative/frontier ratio has mean0\.3910\.391and 95% interval±0\.067\\pm 0\.067; the fixed\-alignment prediction is\(μF/μR\)2=\(0\.5011/0\.8893\)2=0\.317\(\\mu\_\{F\}/\\mu\_\{R\}\)^\{2\}=\(0\.5011/0\.8893\)^\{2\}=0\.317\. Atd=40d=40, learned\-state cosines are0\.966±0\.0060\.966\\pm 0\.006for frontier and0\.933±0\.0060\.933\\pm 0\.006for cumulative\. Thus the cumulative accuracy advantage cannot be explained by a higher state cosine\.

Table 5:Empirical critical dimensiond0\.9d\_\{0\.9\}for every training seed\.
### D\.4Weighted routing protocol and raw codebook results

Atd=192d=192, the identity\-regularized router is trained for four epochs on 600 traces and six profiles with 64 Gaussian route decoys, AdamW \(lr=3×10−3\\mathrm\{lr\}=3\\times 10^\{\-3\}, weight decay10−410^\{\-4\}\), clip1\.01\.0, accumulation 16, initialization noise0\.020\.02, and identity penalty1010\. We select by mean route success minus0\.020\.02projection error on 80 validation traces, 256 Gaussian decoys, and one draw\. The frozen checkpoint is evaluated without retraining on five codebooks, 120 test traces, two draws, all unreached graph keys, and 512 additional Gaussian decoys per route step\.

Table 6:Raw reusable\-token routing results atd=192d=192\. Each cell reports mean joint route\-step success \(fraction\) / per\-inactive\-key false\-positive rate \(%\)\. A route step succeeds iff every currently reached node is accepted and every unreached graph node and all 512 Gaussian decoys are rejected\.The corresponding mean minimum coefficient, normalized by the uniform value, is1\.0001\.000,0\.8140\.814, and0\.7750\.775\. The monotone association between this weakest coefficient, joint success, and false positives is the token\-level pattern predicted by theamina\_\{\\min\}mechanism in Theorem[5\.5](https://arxiv.org/html/2609.13747#S5.Thmtheorem5)\.

### D\.5GPT\-2 replication: configuration and complete results

The GPT\-2 replication uses the same ProsQA graph\-4 traces and the same paired comparison, but replaces the lightweight router with a two\-layer causal GPT\-2 Transformer\. Each edge is represented by causal source and destination tokens followed by one continuous\-thought token\. The two states share the model, optimizer updates, coefficient\-space query, Gaussian codebook, useful candidate, and decoy bank; only the frontier versus cumulative residual update differs\. Queries are generated before the Gaussian map, use the full reached support \(α=1\\alpha=1\), and draw positive coefficients with imbalance at mostρ=1\.1\\rho=1\.1\.

Table 7:Complete configuration of the paired GPT\-2 width sweep\.Each seed–width model is evaluated on 120 held\-out traces, four fixed queries per trace, and two Gaussian draws, with the useful candidate ranked against 4,096 fixed decoys\. Table[8](https://arxiv.org/html/2609.13747#A4.T8)reports all tested widths; intervals are two\-sided 95% Student\-ttintervals over the five model seeds\.

Table 8:Complete GPT\-2 held\-out top\-1 accuracy sweep\.The seed\-level interpolants in Table[9](https://arxiv.org/html/2609.13747#A4.T9)are the raw values summarized in the main\-text figure\. Cumulative has a lower critical dimension for every paired seed; the mean ratio is0\.521±0\.0670\.521\\pm 0\.067\.

Table 9:GPT\-2 interpolated critical dimensiond0\.9d\_\{0\.9\}for every model seed\.
### D\.6Weighted routing protocol and raw codebook results

Atd=192d=192, the identity\-regularized router is trained for four epochs on 600 traces and six profiles with 64 Gaussian route decoys, AdamW \(lr=3×10−3\\mathrm\{lr\}=3\\times 10^\{\-3\}, weight decay10−410^\{\-4\}\), clip1\.01\.0, accumulation 16, initialization noise0\.020\.02, and identity penalty1010\. We select by mean route success minus0\.020\.02projection error on 80 validation traces, 256 Gaussian decoys, and one draw\. The frozen checkpoint is evaluated without retraining on five codebooks, 120 test traces, two draws, all unreached graph keys, and 512 additional Gaussian decoys per route step\.

Table 10:Raw reusable\-token routing results atd=192d=192\. Each cell reports mean joint route\-step success \(fraction\) / per\-inactive\-key false\-positive rate \(%\)\. A route step succeeds iff every currently reached node is accepted and every unreached graph node and all 512 Gaussian decoys are rejected\.The corresponding mean minimum coefficient, normalized by the uniform value, is1\.0001\.000,0\.8140\.814, and0\.7750\.775\. The monotone association between this weakest coefficient, joint success, and false positives is the token\-level pattern predicted by theamina\_\{\\min\}mechanism in Theorem[5\.5](https://arxiv.org/html/2609.13747#S5.Thmtheorem5)\.

### D\.7Software and computing resources

We use Python 3\.10, PyTorch 2\.5\.1, CUDA 12\.4, NumPy 2\.1\.3, and Transformers 4\.46\.2 on NVIDIA A100 GPUs\. The lightweight sweep contains 55 trained models and records 37\.6 summed GPU\-hours; the GPT\-2 replication contains 35 single\-GPU training runs\. The four\-width weighted run records 0\.17 GPU\-hours and its token analysis reuses thed=192d=192checkpoint\.