The Dynamics of Continuous Mixture Collapse in Language Models

# The Dynamics of Continuous Mixture Collapse in Language Models
Source: [https://arxiv.org/html/2609.02049](https://arxiv.org/html/2609.02049)
Ali BackourAffiliation:Massachusetts Institute of TechnologyAffiliation:Cambridge, MA 02139Email:[abackour@mit\.edu](mailto:)

###### Abstract

LLMs latent\-state reasoning methods replace discrete intermediate tokens with continuous states, such as weighted mixtures of token embeddings, to retain multiple possible reasoning directions rather than committing to one\. Yet pretrained language models often fail to preserve these mixtures\. We study*why*through a combination of theoretical analysis and controlled empirical investigations on a variety of models\. We identify three independent, distinct sources of failure\. First, transformer architectures already distort mixture geometry, and training substantially amplifies this effect\. Moreover, the failure can occur even if the model transports mixtures perfectly linearly: the softmax readout and autoregressive feedback form a dynamical system that either amplifies small differences until one component of the mixture dominates or contracts different mixtures until they become indistinguishable\. We verify this theoretical prediction empirically: the observed transition between contraction and amplification occurs near the theoretical threshold derived by our analysis, and pretrained\-model rollouts lie predominantly on the amplifying side\. Finally, we generalize to mixtures of many components and show that exact preservation generally requires context\-dependent correction, whose required dimensionality can grow with the number of components\.

## 1Introduction

Large language models have achieved remarkable results across a wide range of complex reasoning tasks\. A key technique behind that success is chain\-of\-thought \(CoT\) reasoning\[[9](https://arxiv.org/html/2609.02049#bib.bib1),[14](https://arxiv.org/html/2609.02049#bib.bib2),[8](https://arxiv.org/html/2609.02049#bib.bib3),[13](https://arxiv.org/html/2609.02049#bib.bib4)\], which has the model work through a problem step by step by generating intermediate reasoning steps in natural language\. At each step, the model computes a distribution over the vocabulary but commits to a single token, discarding the information carried by all other candidates and preventing it from revisiting them\. This commitment makes generation*readable*, but intermediate reasoning need not be written for human consumption\. Forcing every step through this discrete linguistic bottleneck may therefore limit the abstract concepts the model can represent and manipulate\.

Building on that, several recent works replace discrete intermediate reasoning tokens with continuous reasoning states\.*Soft Thinking*constructs each reasoning state as a probability\-weighted mixture of vocabulary embeddings,e⁡\(p\)=∑ipi​ei,e\(p\)=\\sum\_\{i\}p\_\{i\}e\_\{i\},and feeds this continuous representation back into the model at the next reasoning step\[[18](https://arxiv.org/html/2609.02049#bib.bib6)\]\. Other continuous\-reasoning methods use different representations:*Coconut*directly feeds the model’s last hidden state back as the next input embedding\[[7](https://arxiv.org/html/2609.02049#bib.bib5)\], while*CODI*learns a continuous chain of thought by distilling explicit chain\-of\-thought reasoning into continuous hidden states\[[12](https://arxiv.org/html/2609.02049#bib.bib7)\]\.*Mixture of Inputs*keeps the discarded distribution without any training, feeding back a Bayesian posterior over the sampled token and the distribution it came from\[[22](https://arxiv.org/html/2609.02049#bib.bib8)\]\.

Continuous reasoning is appealing for two main reasons\. First, it can be moreefficient: continuous states can compress reasoning that would otherwise require many discrete tokens so they can represent several reasoning traces within the same computation\[[7](https://arxiv.org/html/2609.02049#bib.bib5),[12](https://arxiv.org/html/2609.02049#bib.bib7),[6](https://arxiv.org/html/2609.02049#bib.bib12)\]\. Second, it enables broaderexploration: rather than committing to a single intermediate token, a continuous state can retain multiple possible reasoning directions and propagate them in parallel\[[18](https://arxiv.org/html/2609.02049#bib.bib6),[7](https://arxiv.org/html/2609.02049#bib.bib5),[6](https://arxiv.org/html/2609.02049#bib.bib12)\]\. This intuition also has theoretical support: continuous chain\-of\-thought allows a two\-layer transformer to solve graph reachability by maintaining a continuous state of reasoning traces, and later work shows that such continuous states can emerge through gradient\-based training\[[20](https://arxiv.org/html/2609.02049#bib.bib10),[21](https://arxiv.org/html/2609.02049#bib.bib11)\]\.

However, recent work finds that pretrained models do not reliably preserve these continuous states\. Wu et al\. find that Soft Thinking decoding is dominated by the highest\-probability component of the reasoning state, creating a greedy feedback loop that suppresses alternative reasoning paths\[[16](https://arxiv.org/html/2609.02049#bib.bib13)\]\. Rizvi\-Martel et al\. similarly find that, in training\-free and fine\-tuned latent\-reasoning settings, continuous state either collapses or is not used, while signs of continuous state appear only in models trained from scratch with latent thoughts\[[11](https://arxiv.org/html/2609.02049#bib.bib14)\]\.

In this work, we ask why\. We show that collapse has architectural, learned, and dynamical sources, each capable of destroying the mixture on its own\. Comparing pretrained models with matched randomly initialized controls across eight models separates distortion induced by the architecture from that added by training\. We then show that this problem persists even under perfect linear transport\. The softmax and feedback between successive generation steps can themselves destroy the mixture: when the two continuous reasoning components lead to sufficiently different next\-step preferences, small imbalances are amplified until one component dominates; when their preferences are similar, different mixtures instead converge and become indistinguishable\. Finally, extending the analysis toKKcomponents, we show that preserving a mixture generally requires a correction that adapts to the context, and that the amount of context\-dependent information required can grow with the number of components under plausible conditions\.

## 2Where Continuous Mixtures Lose Their Geometry

Letei∈ℝde\_\{i\}\\in\\mathbb\{R\}^\{d\}be the embedding of tokenii\. Forppon the simplex, the injected continuous state ise⁡\(p\)=∑ipi​eie\(p\)=\\sum\_\{i\}p\_\{i\}e\_\{i\}\. We place it at an interior slot and ask a question after that slot, so the model must use the injected state to answer\. Writeqiq\_\{i\}for the next\-token distribution when componentiioccupies the slot as an ordinary token andq⁡\(p\)q\(p\)for the distribution under the continuous state\.

![Refer to caption](https://arxiv.org/html/2609.02049v1/figures/p3_simplex.png)Figure 1:Three\-way mixtures collapse toward individual components\.Each point represents an injected convex combination of three token embeddings, positioned by its requested mixture weights and colored by the model’s probabilities forred,green, andblue\. Faithful preservation would reproduce the smooth reference simplex on the left\. Qwen3\.5\-4B \(center\) and Gemma\-4\-E4B\-it \(right\) instead map most interior mixtures to near\-pure colors, with narrow transition boundaries between components\.As a motivation, take three words whose colors are obvious and different \(e\.g\.,*blood*,*grass*,*sky*\) and build one vector that is half the first and a quarter each of the other two for instance\. Put that single vector in the slot and ask*What color is\[ \_ \]? Answer with a single word\.*If the model treated the blend as a blend, the answer should be50%50\\%red,25%25\\%green and25%25\\%blue\. Averaged over the six triples and ten rewordings of the question, however, the next token prediction was99\.5%99\.5\\%red\. Figure[1](https://arxiv.org/html/2609.02049#S2.F1)repeats this across a lattice of 325 three\-way mixtures\. In both models, almost the entire interior of the simplex is mapped to one near\-pure component, with abrupt transitions across narrow boundaries\.

To separate distortion caused by the transformer architecture itself from distortion induced by training, we compare each pretrained model with five randomly initialized networks of exactly the same architecture\. The untrained models therefore retain the same nonlinear blocks, normalization, attention structure, and depth, but remove the effect of the learned weights\. We run this experiment on a ladder of models for both Qwen3\.5\[[10](https://arxiv.org/html/2609.02049#bib.bib19)\]and Gemma 4\[[5](https://arxiv.org/html/2609.02049#bib.bib20)\]\.

To measure how well a mixture is preserved, leth⁡\(w\)∈ℝdh\(w\)\\in\\mathbb\{R\}^\{d\}be the hidden state produced by an injected mixture with weightww\. We define therecovered mixture weightas

α⁡\(w\)=⟨h⁡\(w\)−h⁡\(0\),h⁡\(1\)−h⁡\(0\)⟩∥h⁡\(1\)−h⁡\(0\)∥2\.\\alpha\(w\)=\\frac\{\\langle h\(w\)\-h\(0\),\\,h\(1\)\-h\(0\)\\rangle\}\{\\lVert h\(1\)\-h\(0\)\\rVert^\{2\}\}\.\(1\)
This measures whereh⁡\(w\)h\(w\)lies between the two components’ hidden states\. Exact mixture preservation givesα⁡\(w\)=w\\alpha\(w\)=w\.

We summarize preservation by averaging the absolute difference between the requested weightwwand the recovered weightα⁡\(w\)\\alpha\(w\):

CALIB=1−1Z​𝔼w​\[\|α⁡\(w\)−w\|\]\.\\mathrm\{CALIB\}=1\-\\frac\{1\}\{Z\}\\mathbb\{E\}\_\{w\}\\left\[\|\\alpha\(w\)\-w\|\\right\]\.\(2\)
Here,ZZscales the score so that exact preservation givesCALIB=1\\mathrm\{CALIB\}=1, while a hard threshold atw=0\.5w=0\.5givesCALIB=0\\mathrm\{CALIB\}=0\.

![Refer to caption](https://arxiv.org/html/2609.02049v1/figures/p5_layers.png)Figure 2:Training drives mixture collapse through depth\.Recovered mixture weightα⁡\(w\)\\alpha\(w\)as a function of the requested mixture weightwwat five normalized depths\. Rows show Qwen3\.5\-4B and Gemma\-4\-E4B\-it; columns show the trained model and the mean over five randomly initialized controls of the same architecture\. The diagonal corresponds to exact mixture preservation\. Trained models develop increasingly step\-like responses through depth, while the matched controls remain closer to the diagonal\. Panel titles report final\-layer CALIB, corresponding to Table[1](https://arxiv.org/html/2609.02049#S2.T1)\. Results use the same 1000\-item benchmark described in Appendix[A](https://arxiv.org/html/2609.02049#A1)\.Table 1:Mixture preservation in trained models and matched untrained controls\.CALIB \(Eq\.[2](https://arxiv.org/html/2609.02049#S2.E2)\) scores exact preservation of the injected mixture as11and a hard threshold atw=0\.5w=0\.5as00\. Left: CALIB across depth for Qwen3\.5\-4B and Gemma\-4\-E4B\-it, corresponding to Figure[2](https://arxiv.org/html/2609.02049#S2.F2)\. Right: final\-layer CALIB across a broader ladder of eight models\. Each untrained value is the mean over five random initializations of the same architecture, with standard deviation reported alongside it\.final CALIBModeltraineduntr\.Qwen3\.5\-0\.8B0\.6320\.818±0\.0020\.818\_\{\\pm 0\.002\}Qwen3\.5\-2B0\.5060\.780±0\.0040\.780\_\{\\pm 0\.004\}Qwen3\.5\-4B0\.3320\.732±0\.0030\.732\_\{\\pm 0\.003\}Qwen3\.5\-9B0\.2490\.730±0\.0050\.730\_\{\\pm 0\.005\}Qwen3\.5\-27B0\.1300\.611±0\.0040\.611\_\{\\pm 0\.004\}gemma\-4\-E4B\-it0\.4750\.765±0\.0030\.765\_\{\\pm 0\.003\}gemma\-4\-12B\-it0\.2270\.581±0\.0050\.581\_\{\\pm 0\.005\}gemma\-4\-31B\-it0\.1600\.569±0\.0030\.569\_\{\\pm 0\.003\}

Figure[2](https://arxiv.org/html/2609.02049#S2.F2)and Table[1](https://arxiv.org/html/2609.02049#S2.T1)show a clear separation between trained models and matched untrained controls\. With depth, trained networks increasingly distort the injected mixture, whereas the controls remain substantially closer to linear interpolation\. The two families exhibit different depthwise profiles: Qwen accumulates collapse toward the output, while Gemma shows stronger non\-monotonic distortion within the stack\. Across the broader model ladder, trained models consistently preserve mixtures less faithfully than their controls, and this gap generally widens with scale\. Thus, architectural effects alone induce some distortion, but training substantially amplifies it\.

## 3Mixture Collapse as a Dynamical System

The previous section identified both architectural and training\-induced distortion of the mixture\. A natural question is whether removing these effects would be enough to preserve the mixture\. We show that it would not: even if the transformer propagated the mixture*perfectly linearly*, the softmax readout and its autoregressive feedback introduce a separate source of collapse\.

Fix a context and let two componentsAAandBBproduce next\-token distributionsqAq\_\{A\}andqBq\_\{B\}\. Suppose, optimistically, that their mixed final hidden state is preserved exactly linearly\. Since the output head is linear, the corresponding logits satisfy

z⁡\(w\)=w​zA\+\(1−w\)​zB\.z\(w\)=wz\_\{A\}\+\(1\-w\)z\_\{B\}\.After applying the softmax,

qw​\(y\)∝qA​\(y\)w​qB​\(y\)1−w,q\_\{w\}\(y\)\\propto q\_\{A\}\(y\)^\{w\}q\_\{B\}\(y\)^\{1\-w\},\(3\)so linear interpolation in logit space becomes a weighted*geometric*mixture in probability space, rather than the arithmetic mixturew​qA\+\(1−w\)​qBwq\_\{A\}\+\(1\-w\)q\_\{B\}\. Autoregressive decoding then feeds this distorted mixture back into the model, turning the effect into a recursive dynamical system\.

### 3\.1Recursive Dynamics of Mixture Propagation

Here, for simplicity, we restrict the vocabulary to two tokensAAandBB, and track how probability mass moves between them over the rollout\. At steptt, letwt∈\[0,1\]w\_\{t\}\\in\[0,1\]be the mixture weight onAA, so that1−wt1\-w\_\{t\}is the weight onBB\. It is convenient to recenter this quantity as

ut=2​wt−1,u\_\{t\}=2w\_\{t\}\-1,\(4\)so thatut=−1u\_\{t\}=\-1corresponds to pureBB,ut=0u\_\{t\}=0to an equal mixture, andut=\+1u\_\{t\}=\+1to pureAA\. The sign ofutu\_\{t\}therefore indicates which token has the majority, while\|ut\|\|u\_\{t\}\|measures how strongly the mixture favors one token over the other\.

At the current contextctc\_\{t\}, we evaluate the modelMMseparately on the two pure components,

qtA=M\(ct∥A\),qtB=M\(ct∥B\)\.q^\{A\}\_\{t\}=M\(c\_\{t\}\\\|A\),\\qquad q^\{B\}\_\{t\}=M\(c\_\{t\}\\\|B\)\.Each branch’s log\-odds between that pair is

ΔA,t=log⁡qtA​\(A\)qtA​\(B\),ΔB,t=log⁡qtB​\(A\)qtB​\(B\),\\Delta\_\{A,t\}=\\log\\frac\{q^\{A\}\_\{t\}\(A\)\}\{q^\{A\}\_\{t\}\(B\)\},\\qquad\\Delta\_\{B,t\}=\\log\\frac\{q^\{B\}\_\{t\}\(A\)\}\{q^\{B\}\_\{t\}\(B\)\},\(5\)soΔA,t\\Delta\_\{A,t\}measures how strongly branchAAprefers its own successor toBB’s\. We write

Lt=ΔA,t−ΔB,t2,bt=ΔA,t\+ΔB,t2\.L\_\{t\}=\\frac\{\\Delta\_\{A,t\}\-\\Delta\_\{B,t\}\}\{2\},\\qquad b\_\{t\}=\\frac\{\\Delta\_\{A,t\}\+\\Delta\_\{B,t\}\}\{2\}\.\(6\)
LtL\_\{t\}is the*coupling*: it measures how differently the two branches rank the competing descendants\. Large\|Lt\|\|L\_\{t\}\|means their relative preferences differ strongly, while small\|Lt\|\|L\_\{t\}\|means they make similar relative predictions\. In contrast,btb\_\{t\}is the*field*: it measures the common offset in the two branches’ log\-odds\.

Under exact linear logit transport, the next log\-odds isℓt\+1=wt​ΔA,t\+\(1−wt\)​ΔB,t=bt\+Lt​ut\\ell\_\{t\+1\}=w\_\{t\}\\Delta\_\{A,t\}\+\(1\-w\_\{t\}\)\\Delta\_\{B,t\}=b\_\{t\}\+L\_\{t\}u\_\{t\}\. Using2​σ​\(x\)−1=tanh⁡\(x/2\)2\\sigma\(x\)\-1=\\tanh\(x/2\)gives

ut\+1=tanh⁡\(bt\+Lt​ut2\)\.u\_\{t\+1\}=\\tanh\\\!\\left\(\\frac\{b\_\{t\}\+L\_\{t\}u\_\{t\}\}\{2\}\\right\)\.\(7\)
For fixedbbandLL, Eq\.[7](https://arxiv.org/html/2609.02049#S3.E7)is the standard fixed\-point iteration associated with the mean\-field Curie–Weiss self\-consistency equation\[[15](https://arxiv.org/html/2609.02049#bib.bib22)\]\. The language\-model setting is non\-autonomous:LtL\_\{t\}andbtb\_\{t\}vary as the rollout evolves\.

### 3\.2The Critical Threshold: Amplification and Washout

We now study what happens to the mixture over time under these dynamics\. To isolate the effect of the coupling, we first setbt=0b\_\{t\}=0and assumeLt\>0L\_\{t\}\>0\. Equation[7](https://arxiv.org/html/2609.02049#S3.E7)then becomes

ut\+1=tanh⁡\(Lt​ut2\)\.u\_\{t\+1\}=\\tanh\\\!\\left\(\\frac\{L\_\{t\}u\_\{t\}\}\{2\}\\right\)\.\(8\)
Take first the constant\-coupling caseLt=LL\_\{t\}=L\. The balanced stateu=0u=0is always a fixed point, but its stability changes atL=2L=2\. Indeed,

∂ut\+1∂ut\|ut=0=L2\.\\left\.\\frac\{\\partial u\_\{t\+1\}\}\{\\partial u\_\{t\}\}\\right\|\_\{u\_\{t\}=0\}=\\frac\{L\}\{2\}\.
Henceu=0u=0is stable forL<2L<2and unstable forL\>2L\>2\. WhenL\>2L\>2, two additional stable fixed points±u∗​\(L\)\\pm u\_\{\\ast\}\(L\)appear, where

u∗​\(L\)=tanh⁡\(L​u∗​\(L\)2\)\.u\_\{\\ast\}\(L\)=\\tanh\\\!\\left\(\\frac\{Lu\_\{\\ast\}\(L\)\}\{2\}\\right\)\.\(9\)
Starting from anyu0≠0u\_\{0\}\\neq 0, the sign of the initial majority is preserved and the trajectory converges to the corresponding polarized fixed point\. This is the classical bifurcation of the mean\-field ferromagnet\[[3](https://arxiv.org/html/2609.02049#bib.bib21), Ch\. 2\]\. The polarization rapidly approaches a pure state asLLgrows: for example,L=4L=4givesu∗≃0\.96u\_\{\\ast\}\\simeq 0\.96, whileL=8L=8givesu∗≃0\.9993u\_\{\\ast\}\\simeq 0\.9993, andu∗​\(L\)→1u\_\{\\ast\}\(L\)\\to 1asL→∞L\\to\\infty\.

However, the LLM setting is harder than the classical Curie–Weiss case because the context evolves, soLtL\_\{t\}is not fixed\. The following two results characterize the corresponding time\-varying dynamics on either side of the critical value\.

###### Theorem 1\(Uniform coupling implies persistent polarization\)\.

Letu0≠0u\_\{0\}\\neq 0, letbt=0b\_\{t\}=0, and supposeLt≥Lmin\>2L\_\{t\}\\geq L\_\{\\min\}\>2for everytt\. Letu∗​\(Lmin\)∈\(0,1\)u\_\{\\ast\}\(L\_\{\\min\}\)\\in\(0,1\)be the positive solution of

u=tanh⁡\(Lmin​u2\)\.u=\\tanh\\\!\\left\(\\frac\{L\_\{\\min\}u\}\{2\}\\right\)\.
Thensign⁡\(ut\)=sign⁡\(u0\)\\operatorname\{sign\}\(u\_\{t\}\)=\\operatorname\{sign\}\(u\_\{0\}\)for alltt, and

lim inft→∞\|ut\|≥u∗​\(Lmin\)\.\\liminf\_\{t\\to\\infty\}\|u\_\{t\}\|\\geq u\_\{\\ast\}\(L\_\{\\min\}\)\.\(10\)

###### Theorem 2\(Subcritical coupling washes out the mixture\)\.

Suppose

0≤Lt≤Lmax<20\\leq L\_\{t\}\\leq L\_\{\\max\}<2
along every rollout\. Then

\|uT\|≤\(Lmax2\)T​\|u0\|\+12​∑t=0T−1\(Lmax2\)T−1−t​\|bt\|\.\|u\_\{T\}\|\\leq\\left\(\\frac\{L\_\{\\max\}\}\{2\}\\right\)^\{T\}\|u\_\{0\}\|\+\\frac\{1\}\{2\}\\sum\_\{t=0\}^\{T\-1\}\\left\(\\frac\{L\_\{\\max\}\}\{2\}\\right\)^\{T\-1\-t\}\|b\_\{t\}\|\.
Consequently, ifbt=0b\_\{t\}=0, thenut→0u\_\{t\}\\to 0independently of the initial mixture\.

The proofs are given in Appendix[B](https://arxiv.org/html/2609.02049#A2)\. Theorem[1](https://arxiv.org/html/2609.02049#Thmtheorem1)follows by a monotonicity comparison, while Theorem[2](https://arxiv.org/html/2609.02049#Thmtheorem2)follows from a mean\-value bound on the same recursion\.

Together, the two results describe complementary failure modes on either side of the critical value\. If the evolving context keeps the coupling uniformly above the critical value, the initial majority is magnified and the mixture polarizes to an almost pure state for largeLminL\_\{\\min\}\. Below the critical value, the dynamics instead become contractive and progressively erase distinctions between different mixtures\. In particular, avoiding a vertex is not enough to preserve the mixture: distinct states such as70/3070/30and30/7030/70become indistinguishable at a geometric rate, and the information carried by the mixture is lost\. Thus the two sides of the threshold destroy information in complementary ways\.

![Refer to caption](https://arxiv.org/html/2609.02049v1/figures/p7_branch.png)Figure 3:Recursive softmax reproduces polarization in LLM rollouts\.The two pure branches evolve under the model, while their mixture is formed only by linearly interpolating the endpoint logits before the softmax\. Rows are the two families, Qwen3\.5\-4B above and Gemma\-4\-E4B\-it below\. The left column tracks the median mixture weightwtw\_\{t\}from each initial mixturew0w\_\{0\}; the right column shows both measured coefficients of the recursion along the same rollouts, the couplingLtL\_\{t\}and the field magnitude\|bt\|\|b\_\{t\}\|\. Every curve is a median over the same10001000\-item benchmark used in Figure[2](https://arxiv.org/html/2609.02049#S2.F2)and described in Appendix[A](https://arxiv.org/html/2609.02049#A1), rolled out for5050steps per item\.![Refer to caption](https://arxiv.org/html/2609.02049v1/figures/p8_gain.png)Figure 4:Reducing the gain replaces one failure mode with the other, in both families\.A feedback temperatureτ\\taudivides the effective coupling,Lteff=Lt/τL^\{\\mathrm\{eff\}\}\_\{t\}=L\_\{t\}/\\tau, so sweepingτ\\tauwalks the same measured items from the polarizing regimeLteff\>2L^\{\\mathrm\{eff\}\}\_\{t\}\>2into the contractive oneLteff<2L^\{\\mathrm\{eff\}\}\_\{t\}<2\. Each panel shows the medianutu\_\{t\}from two symmetric startsu0=±0\.5u\_\{0\}=\\pm 0\.5, one panel per family, with color carrying the*median*effective coupling each curve achieves\. Well above threshold the two starts separate to opposite vertices; well below it they merge onto the balanced state, and the turn happens around the predictedLteff=2L^\{\\mathrm\{eff\}\}\_\{t\}=2\. The couplings are theLtL\_\{t\}sequences measured in Figure[3](https://arxiv.org/html/2609.02049#S3.F3), over the same10001000\-item benchmark of Appendix[A](https://arxiv.org/html/2609.02049#A1)\.

## 4Recursive Softmax in Pretrained Language Models

The theory above predicts two qualitatively different failure modes depending on the coupling\. We now measure these dynamics directly in pretrained language\-model trajectories\. To isolate the effect of the recursive softmax, we run the model only on the two pure branches and form their mixture directly in logit space, so that mixture transport is exactly linear by construction and the only nonlinearity acting on the mixture is the softmax\.

Following the notation of §[3\.1](https://arxiv.org/html/2609.02049#S3.SS1), the state at stepttis\(At,Bt,wt,ct\)\(A\_\{t\},B\_\{t\},w\_\{t\},c\_\{t\}\), and we evaluate the model on the two pure branches as before,qtA=M\(ct∥At\)q^\{A\}\_\{t\}=M\(c\_\{t\}\\\|A\_\{t\}\)andqtB=M\(ct∥Bt\)q^\{B\}\_\{t\}=M\(c\_\{t\}\\\|B\_\{t\}\), using their greedy predictions as the next pair of components,

At\+1=arg⁡maxy​qtA​\(y\),Bt\+1=arg⁡maxy​qtB​\(y\)\.A\_\{t\+1\}=\\arg\\max\_\{y\}q^\{A\}\_\{t\}\(y\),\\qquad B\_\{t\+1\}=\\arg\\max\_\{y\}q^\{B\}\_\{t\}\(y\)\.\(11\)
The two branches therefore evolve normally with the context\. We then interpolate only their endpoint logits,

ztmix=wt​ztA\+\(1−wt\)​ztB,z\_\{t\}^\{\\mathrm\{mix\}\}=w\_\{t\}z\_\{t\}^\{A\}\+\(1\-w\_\{t\}\)z\_\{t\}^\{B\},apply the softmax, and define the next mixture weight from the probability assigned by the model to the two new components under the mixed logits,

wt\+1=pt​\(At\+1\)pt​\(At\+1\)\+pt​\(Bt\+1\)\.w\_\{t\+1\}=\\frac\{p\_\{t\}\(A\_\{t\+1\}\)\}\{p\_\{t\}\(A\_\{t\+1\}\)\+p\_\{t\}\(B\_\{t\+1\}\)\}\.andct\+1c\_\{t\+1\}is the context updated with the new mixture\. Because the endpoints and context evolve at every step, the correspondingLtL\_\{t\}andbtb\_\{t\}are re\-measured throughout the rollout\.

Figure[3](https://arxiv.org/html/2609.02049#S3.F3)tests this prediction on pretrained model trajectories while removing nonlinear transformer processing from the mixture path\. The left panel shows the resulting evolution of the mixture weightwtw\_\{t\}: mixtures initialized just above and below1/21/2rapidly diverge toward opposite branches\. The right panel explains this behavior through the two measured coefficients of Eq\.[7](https://arxiv.org/html/2609.02049#S3.E7)\. Across both model families,LtL\_\{t\}remains predominantly above the critical valueL=2L=2, placing the dynamics in the majority\-amplifying regime of §[3\.2](https://arxiv.org/html/2609.02049#S3.SS2), whilebtb\_\{t\}is much smaller\.

### 4\.1Crossing the Critical Threshold

The two regimes on both sides of the critical threshold can be walked between on the measured dynamics\. A temperatureτ\\tauon the feedback logits rescales the coupling asLteff=Lt/τL\_\{t\}^\{\\mathrm\{eff\}\}=L\_\{t\}/\\tau, varying the gain without changing the model, the items or the branch trajectories, so we sweepτ\\tauover the couplings measured in Figure[3](https://arxiv.org/html/2609.02049#S3.F3)and start the recursion from two symmetric statesu0=±0\.5u\_\{0\}=\\pm 0\.5\.

Figure[4](https://arxiv.org/html/2609.02049#S3.F4)shows the predicted transition in both families: at large effective coupling the two trajectories separate to opposite branches; at small effective coupling they merge and lose their initial distinction, and the turn happens around the theoretical valueLeff=2L^\{\\mathrm\{eff\}\}=2which is exactly what the theory predicts\.

## 5The Cost of CorrectingKK\-Way Mixtures

The analysis above is the special case of mixtures of two tokens\. We now generalize toKKcomponents\. Letp∈int⁡ΔK−1p\\in\\operatorname\{int\}\\Delta^\{K\-1\}denote their mixture weights and use log\-ratio coordinates

a=ψ⁡\(p\)∈ℝK−1,ψi​\(p\)=log⁡pipK\.a=\\psi\(p\)\\in\\mathbb\{R\}^\{K\-1\},\\qquad\\psi\_\{i\}\(p\)=\\log\\frac\{p\_\{i\}\}\{p\_\{K\}\}\.\(12\)
Although the textual contextctc\_\{t\}is discrete, the model operates on a continuous representation of it\. Letxt=x⁡\(ct\)∈ℝnx\_\{t\}=x\(c\_\{t\}\)\\in\\mathbb\{R\}^\{n\}denote this contextual representation\. One decoding step then induces a map

at\+1=Φ⁡\(xt,at\),a\_\{t\+1\}=\\Phi\(x\_\{t\},a\_\{t\}\),\(13\)where the contextual representation and candidate identities may change during the rollout; the binary linear\-logit reduction recovers Eq\.[7](https://arxiv.org/html/2609.02049#S3.E7)\.

Ifata\_\{t\}represents the proportions of unresolved components, the ideal behavior is that one reasoning step leaves them unchanged,

at\+1=at\.a\_\{t\+1\}=a\_\{t\}\.\(14\)
The previous sections establish, theoretically and empirically, that in generalΦ⁡\(xt,at\)≠at\\Phi\(x\_\{t\},a\_\{t\}\)\\neq a\_\{t\}\. Moreover, the distortion depends on the context, so a fixed transformation ofata\_\{t\}cannot in general undo it\. Suppose instead that, before feeding the mixture into the model, we apply a corrector

a~t=f⁡\(at,r⁡\(xt\)\),\\widetilde\{a\}\_\{t\}=f\\\!\\left\(a\_\{t\},r\(x\_\{t\}\)\\right\),\(15\)wherer⁡\(xt\)∈ℝmr\(x\_\{t\}\)\\in\\mathbb\{R\}^\{m\}is context\-dependent information supplied to the corrector\. Faithful propagation requires

Φ⁡\(xt,f⁡\(at,r⁡\(xt\)\)\)=at\.\\Phi\\\!\\left\(x\_\{t\},f\(a\_\{t\},r\(x\_\{t\}\)\)\\right\)=a\_\{t\}\.\(16\)
We now ask how much context\-dependent information such a corrector may require\. Fix a single interior target statea¯\\bar\{a\}and require only that the corrector keep this one state fixed under nearby perturbations of the continuous contextual representation:

Φ⁡\(x,f⁡\(a¯,r⁡\(x\)\)\)=a¯for every​x​near​x0\.\\Phi\\\!\\left\(x,f\(\\bar\{a\},r\(x\)\)\\right\)=\\bar\{a\}\\qquad\\text\{for every \}x\\text\{ near \}x\_\{0\}\.\(17\)
Even this weak requirement can demand a number of context\-dependent quantities proportional to the number of components\.

###### Theorem 3\(Context\-dimension lower bound\)\.

Assume thatΦ\\Phi,ff, andrrare continuously differentiable and that Eq\.[17](https://arxiv.org/html/2609.02049#S5.E17)holds\. Then

m≥rank⁡Dx​Φ​\(x0,f⁡\(a¯,r⁡\(x0\)\)\)\.m\\geq\\operatorname\{rank\}D\_\{x\}\\Phi\\\!\\left\(x\_\{0\},f\(\\bar\{a\},r\(x\_\{0\}\)\)\\right\)\.\(18\)
In particular, if local variations of the continuous contextual representation move the output independently in allK−1K\-1mixture directions, then

The proof is a chain\-rule argument on the anchoring condition \(Appendix[B](https://arxiv.org/html/2609.02049#A2)\)\. The rank of this Jacobian need not be small: local changes in the contextual representation can affect the final hidden state along many independent directions\. As a result, faithfully correcting a mixture as it passes through a pretrained model generally requires a non\-trivial dependence on the context, and the amount of context\-dependent information needed can itself be large\.

## 6Discussion and Implications

Taken together, our experiments and theory give a unified account of why continuous mixtures are difficult to preserve\. For each failure mode, we identify a corresponding mechanism and test its consequences: the transformer architecture already distorts mixtures, training substantially amplifies this distortion, and even under idealized linear transport the softmax feedback dynamics can either amplify or erase differences between components\. The rollout experiments reproduce the behavior predicted by this dynamical analysis, suggesting that these are not merely artifacts of the representation\-level probes\. This also explains why a simple fix such as globally reweighting or softening the mixture is unlikely to be sufficient\. The required correction changes with the context, and ourKK\-way analysis shows that, under full\-rank context variation, exact local preservation can require at leastK−1K\-1context\-dependent degrees of freedom\. Thus, preserving mixtures is not just a matter of choosing better weights, but of actively compensating for context\-dependent dynamics throughout the rollout\.

#### Relation to prior work\.

Continuous\- and latent\-reasoning methods show that continuous computation can be useful and, trained appropriately, can carry multiple reasoning traces\[[7](https://arxiv.org/html/2609.02049#bib.bib5),[12](https://arxiv.org/html/2609.02049#bib.bib7),[18](https://arxiv.org/html/2609.02049#bib.bib6),[20](https://arxiv.org/html/2609.02049#bib.bib10),[21](https://arxiv.org/html/2609.02049#bib.bib11),[6](https://arxiv.org/html/2609.02049#bib.bib12)\], while recent empirical work finds pretrained models collapsing or ignoring such continuous states\[[16](https://arxiv.org/html/2609.02049#bib.bib13),[11](https://arxiv.org/html/2609.02049#bib.bib14)\]\. Our results connect the two: continuity alone does not preserve continuous states, and what decides the outcome is whether the representation, the readout and the closed\-loop dynamics together preserve the semantics assigned to the continuous state\. The positive and negative results describe different dynamical regimes of one problem\. Training\-free remedies already target the failure from this direction\.*SeLaR*gates soft embeddings on entropy and adds a contrastive term that pushes the blend away from the dominant token\[[4](https://arxiv.org/html/2609.02049#bib.bib9)\], and*Mixture of Inputs*reweights what is fed back\[[22](https://arxiv.org/html/2609.02049#bib.bib8)\], which in the language of §[3\.2](https://arxiv.org/html/2609.02049#S3.SS2)is an attempt to hold the effective coupling down without falling into the contractive regime, and Theorem[2](https://arxiv.org/html/2609.02049#Thmtheorem2)says that is the trade such a term has to manage\.

#### Future directions\.

Each cause suggests its own intervention\. Architectural and training\-induced distortion could be addressed by training explicitly on mixed or latent states, so that mixture geometry survives the network\[[7](https://arxiv.org/html/2609.02049#bib.bib5),[12](https://arxiv.org/html/2609.02049#bib.bib7),[1](https://arxiv.org/html/2609.02049#bib.bib17),[19](https://arxiv.org/html/2609.02049#bib.bib18)\]; supervising a latent directly as an average of the embeddings it is meant to stand for, or confining it to the vocabulary space, are two ways of doing that already in use\[[17](https://arxiv.org/html/2609.02049#bib.bib16),[2](https://arxiv.org/html/2609.02049#bib.bib15)\]\. The recursive softmax failure instead suggests bypassing the vocabulary readout altogether: rather than projecting a hidden state to logits, one can feed the continuous hidden state directly into the next step\. Coconut takes precisely this route, using the previous final hidden state as the next input embedding\[[7](https://arxiv.org/html/2609.02049#bib.bib5)\]; its gains are nevertheless task\-dependent, suggesting that removing the softmax feedback loop addresses one source of collapse but is not by itself sufficient for robust continuous reasoning\. And because the required correction moves with the context, a generalKK\-way solution likely needs a context\-conditioned controller whose capacity grows with the number of components\. Learning and testing such neutral dynamics is the natural next step toward systems that preserve, rather than merely encode, multiple possibilities\.

## 7Limitations and Broader Impact

Our experiments use controlled embedding mixtures as an operational notion of preserving multiple components; this may not capture every form of semantic continuous state in continuous\-reasoning methods\. The empirical analysis is primarily binary and covers two model families, while theKK\-way result is theoretical and its full\-rank condition is not measured directly\. Matched random controls isolate effects associated with learned weights but do not identify which aspects of training cause the distortion, and we study mixture preservation rather than downstream performance in a fully trained continuous\-reasoning system\.

This work is primarily diagnostic\. Better understanding and controlling mixture collapse may improve the reliability of latent\-reasoning systems, but stronger latent reasoning can also make intermediate computation less human\-readable\. This increases the importance of developing monitoring and interpretability methods alongside improvements in continuous reasoning\.

## 8Acknowledgments

We thank Maxwell Sun, Beshr Islam Bouli, Seb Losada and Nour Massri for helpful discussions and feedback\.

## References

- \[1\]\(2025\)Soft tokens, hard truths\.Note:arXiv:2509\.19170External Links:2509\.19170,[Link](https://arxiv.org/abs/2509.19170)Cited by:[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px2.p1.1)\.
- \[2\]J\. Deng, L\. Pang, Z\. Wei, S\. Xu, Z\. Duan, K\. Xu, Y\. Song, H\. Shen, and X\. Cheng\(2026\)LLM latent reasoning as chain of superposition\.Note:arXiv:2510\.15522External Links:2510\.15522,[Link](https://arxiv.org/abs/2510.15522)Cited by:[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px2.p1.1)\.
- \[3\]S\. Friedli and Y\. Velenik\(2017\)Statistical mechanics of lattice systems: a concrete mathematical introduction\.Cambridge University Press\.Note:[https://doi\.org/10\.1017/9781316882603](https://doi.org/10.1017/9781316882603)Cited by:[§3\.2](https://arxiv.org/html/2609.02049#S3.SS2.p7.1)\.
- \[4\]R\. Fu and G\. Luo\(2026\)SeLaR: selective latent reasoning in large language models\.Note:arXiv:2604\.08299External Links:2604\.08299,[Link](https://arxiv.org/abs/2604.08299)Cited by:[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px1.p1.1)\.
- \[5\]Gemma Teamet al\.\(2026\)Gemma 4 technical report\.Note:arXiv:2607\.02770External Links:2607\.02770,[Link](https://arxiv.org/abs/2607.02770)Cited by:[§2](https://arxiv.org/html/2609.02049#S2.p3.1)\.
- \[6\]H\. A\. Gozeten, M\. E\. Ildiz, X\. Zhang, H\. Harutyunyan, A\. S\. Rawat, and S\. Oymak\(2026\)Continuous chain of thought enables parallel exploration and reasoning\.Note:arXiv:2505\.23648External Links:2505\.23648,[Link](https://arxiv.org/abs/2505.23648)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p3.1),[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px1.p1.1)\.
- \[7\]S\. Hao, S\. Sukhbaatar, D\. Su, X\. Li, Z\. Hu, J\. Weston, and Y\. Tian\(2026\)Training large language models to reason in a continuous latent space\.Note:arXiv:2412\.06769External Links:2412\.06769,[Link](https://arxiv.org/abs/2412.06769)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p2.1),[§1](https://arxiv.org/html/2609.02049#S1.p3.1),[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px2.p1.1)\.
- \[8\]T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa\(2023\)Large language models are zero\-shot reasoners\.Note:arXiv:2205\.11916External Links:2205\.11916,[Link](https://arxiv.org/abs/2205.11916)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p1.1)\.
- \[9\]M\. Nye, A\. J\. Andreassen, G\. Gur\-Ari, H\. Michalewski, J\. Austin, D\. Bieber, D\. Dohan, A\. Lewkowycz, M\. Bosma, D\. Luan, C\. Sutton, and A\. Odena\(2021\)Show your work: scratchpads for intermediate computation with language models\.Note:arXiv:2112\.00114External Links:2112\.00114,[Link](https://arxiv.org/abs/2112.00114)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p1.1)\.
- \[10\]Qwen Team\(2026\)Qwen3\.5: towards native multimodal agents\.Note:[https://qwen\.ai/blog?id=qwen3\.5](https://qwen.ai/blog?id=qwen3.5)External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§2](https://arxiv.org/html/2609.02049#S2.p3.1)\.
- \[11\]M\. Rizvi\-Martel, G\. Rabusseau, and M\. Mosbach\(2026\)The illusion of superposition? A principled analysis of latent thinking in language models\.Note:arXiv:2604\.06374External Links:2604\.06374,[Link](https://arxiv.org/abs/2604.06374)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p4.1),[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px1.p1.1)\.
- \[12\]Z\. Shen, H\. Yan, L\. Zhang, Z\. Hu, Y\. Du, and Y\. He\(2025\)CODI: compressing chain\-of\-thought into continuous space via self\-distillation\.Note:arXiv:2502\.21074External Links:2502\.21074,[Link](https://arxiv.org/abs/2502.21074)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p2.1),[§1](https://arxiv.org/html/2609.02049#S1.p3.1),[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px2.p1.1)\.
- \[13\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2023\)Self\-consistency improves chain of thought reasoning in language models\.Note:arXiv:2203\.11171External Links:2203\.11171,[Link](https://arxiv.org/abs/2203.11171)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p1.1)\.
- \[14\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou\(2023\)Chain\-of\-thought prompting elicits reasoning in large language models\.Note:arXiv:2201\.11903External Links:2201\.11903,[Link](https://arxiv.org/abs/2201.11903)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p1.1)\.
- \[15\]P\. Weiss\(1907\)L’hypothèse du champ moléculaire et la propriété ferromagnétique\.J\. Phys\. Theor\. Appl\.6\(1\),pp\. 661–690\.Note:[https://hal\.science/jpa\-00241247](https://hal.science/jpa-00241247)External Links:[Link](https://hal.science/jpa-00241247),[Document](https://dx.doi.org/10.1051/jphystap%3A019070060066100)Cited by:[§3\.1](https://arxiv.org/html/2609.02049#S3.SS1.p5.1)\.
- \[16\]J\. Wu, J\. Lu, Z\. Ren, G\. Hu, Z\. Wu, D\. Dai, and H\. Wu\(2025\)LLMs are single\-threaded reasoners: demystifying the working mechanism of soft thinking\.Note:arXiv:2508\.03440External Links:2508\.03440,[Link](https://arxiv.org/abs/2508.03440)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p4.1),[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px1.p1.1)\.
- \[17\]V\. Yerram, H\. He, and E\. Choi\(2026\)Training continuous chain of thought models: a tale of two regimes\.Note:arXiv:2607\.16972External Links:2607\.16972,[Link](https://arxiv.org/abs/2607.16972)Cited by:[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px2.p1.1)\.
- \[18\]Z\. Zhang, X\. He, W\. Yan, A\. Shen, C\. Zhao, S\. Wang, Y\. Shen, and X\. E\. Wang\(2025\)Soft Thinking: unlocking the reasoning potential of LLMs in continuous concept space\.Note:arXiv:2505\.15778External Links:2505\.15778,[Link](https://arxiv.org/abs/2505.15778)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p2.1),[§1](https://arxiv.org/html/2609.02049#S1.p3.1),[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px1.p1.1)\.
- \[19\]Z\. Zheng, Y\. Gu, W\. Liu, Y\. W\. Teh, and W\. S\. Lee\(2026\)SofT\-GRPO: surpassing discrete\-token LLM reinforcement learning via Gumbel\-reparameterized soft\-thinking policy optimization\.Note:arXiv:2511\.06411External Links:2511\.06411,[Link](https://arxiv.org/abs/2511.06411)Cited by:[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px2.p1.1)\.
- \[20\]H\. Zhu, S\. Hao, Z\. Hu, J\. Jiao, S\. Russell, and Y\. Tian\(2025\)Reasoning by superposition: a theoretical perspective on chain of continuous thought\.Note:arXiv:2505\.12514External Links:2505\.12514,[Link](https://arxiv.org/abs/2505.12514)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p3.1),[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px1.p1.1)\.
- \[21\]H\. Zhu, S\. Hao, Z\. Hu, J\. Jiao, S\. Russell, and Y\. Tian\(2026\)Emergence of superposition: unveiling the training dynamics of chain of continuous thought\.Note:arXiv:2509\.23365External Links:2509\.23365,[Link](https://arxiv.org/abs/2509.23365)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p3.1),[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px1.p1.1)\.
- \[22\]Y\. Zhuang, L\. Liu, C\. Singh, J\. Shang, and J\. Gao\(2025\)Text generation beyond discrete token sampling\.Note:arXiv:2505\.14827External Links:2505\.14827,[Link](https://arxiv.org/abs/2505.14827)Cited by:[§1](https://arxiv.org/html/2609.02049#S1.p2.1),[§6](https://arxiv.org/html/2609.02049#S6.SS0.SSS0.Px1.p1.1)\.

## Appendix ABenchmark Construction

We construct a benchmark of 1000 binary mixture items spanning seven semantic categories: color, temperature, size, speed, hardness, weight, and brightness\. Each item consists of a prompt with one injection slot, two component words, and two corresponding answer words\. At mixture weightpp, the slot is filled with

e⁡\(p\)=p​e1\+\(1−p\)​e2,e\(p\)=p\\,e\_\{1\}\+\(1\-p\)\\,e\_\{2\},wheree1e\_\{1\}ande2e\_\{2\}are the component embeddings\. We then measure the model’s relative probability assigned to the two answer words at the next\-token position\. Thus,p=1p=1andp=0p=0recover the two ordinary\-token endpoints, while intermediateppvalues test whether the model preserves the injected mixture\.

#### Item selection\.

An item is included only if \(i\) both component words and both answer words are single tokens under both model families, and \(ii\) at each endpoint the trained model assigns at least0\.50\.5probability to the corresponding answer word\. The second condition ensures that the underlying question is well posed before testing mixtures\. The surviving items are intersected across model families, yielding one shared 1000\-item benchmark used for all trained models and their matched randomly initialized controls\.

#### Prompting\.

We use a chat template that closes the reasoning block before the answer position, so that the next\-token probability is concentrated on the requested one\-word answer rather than on intermediate reasoning text\.

Table[2](https://arxiv.org/html/2609.02049#A1.T2)shows representative benchmark items\.

Table 2:Ten of the 1000 benchmark items\.One per category, then a second pass in the same order; every category the benchmark contains appears before any repeats\.Promptis the template with the injection slot marked\[ \_ \]; the slot is filled with the convex combinationp​e​\(first\)\+\(1−p\)​e​\(second\)p\\,e\(\\text\{first\}\)\+\(1\-p\)\\,e\(\\text\{second\}\)of the twocomponents’ embeddings, never with a real token\.Answersare the two words whose probability ratio is the readout: withp=1p=1the model should say the first, withp=0p=0the second, and the question is what it does in between\. Every item is single\-token under both tokenizers and clears the0\.50\.5answer\-mass gate in both families\.

## Appendix BProofs

### Proof of Theorem[1](https://arxiv.org/html/2609.02049#Thmtheorem1)\(uniform coupling implies persistent polarization\)

###### Proof\.

WriteFL​\(u\)=tanh⁡\(L​u/2\)F\_\{L\}\(u\)=\\tanh\(Lu/2\)\. Foru\>0u\>0it is increasing inuuand inLL, andFL​\(−u\)=−FL​\(u\)F\_\{L\}\(\-u\)=\-F\_\{L\}\(u\), so without loss of generality takeu0\>0u\_\{0\}\>0\. Since everyFLtF\_\{L\_\{t\}\}maps\(0,1\)\(0,1\)into itself,ut\>0u\_\{t\}\>0for allttand the sign is preserved\.

Putv0=u0v\_\{0\}=u\_\{0\}andvt\+1=FLmin​\(vt\)v\_\{t\+1\}=F\_\{L\_\{\\min\}\}\(v\_\{t\}\), the constant\-coupling system at the smallest coupling encountered\. Ifut≥vtu\_\{t\}\\geq v\_\{t\}thenut\+1=FLt​\(ut\)≥FLmin​\(ut\)≥FLmin​\(vt\)=vt\+1u\_\{t\+1\}=F\_\{L\_\{t\}\}\(u\_\{t\}\)\\geq F\_\{L\_\{\\min\}\}\(u\_\{t\}\)\\geq F\_\{L\_\{\\min\}\}\(v\_\{t\}\)=v\_\{t\+1\}, using monotonicity inLLthen inuu; by inductionut≥vtu\_\{t\}\\geq v\_\{t\}for everytt\.

ForLmin\>2L\_\{\\min\}\>2the mapFLminF\_\{L\_\{\\min\}\}has exactly one positive fixed pointu∗​\(Lmin\)u\_\{\\ast\}\(L\_\{\\min\}\), and it is attracting from anyv0∈\(0,1\)v\_\{0\}\\in\(0,1\): on\(0,u∗\)\(0,u\_\{\\ast\}\)we haveFLmin​\(v\)\>vF\_\{L\_\{\\min\}\}\(v\)\>vand on\(u∗,1\)\(u\_\{\\ast\},1\)we haveFLmin​\(v\)<vF\_\{L\_\{\\min\}\}\(v\)<v, sovtv\_\{t\}is monotone and bounded, hence convergent, and its limit is a fixed point, which can only beu∗u\_\{\\ast\}\. Thereforelim inftut≥limtvt=u∗​\(Lmin\)\\liminf\_\{t\}u\_\{t\}\\geq\\lim\_\{t\}v\_\{t\}=u\_\{\\ast\}\(L\_\{\\min\}\)\. ∎

### Proof of Theorem[2](https://arxiv.org/html/2609.02049#Thmtheorem2)\(subcritical coupling washes out the mixture\)

###### Proof\.

We have

ut\+1=tanh⁡\(Lt​ut\+bt2\)\.u\_\{t\+1\}=\\tanh\\\!\\left\(\\frac\{L\_\{t\}u\_\{t\}\+b\_\{t\}\}\{2\}\\right\)\.
Using\|tanh⁡\(x\)\|≤\|x\|\|\\tanh\(x\)\|\\leq\|x\|and0≤Lt≤Lmax<20\\leq L\_\{t\}\\leq L\_\{\\max\}<2,

\|ut\+1\|≤Lt2​\|ut\|\+\|bt\|2≤Lmax2​\|ut\|\+bmax2\.\|u\_\{t\+1\}\|\\leq\\frac\{L\_\{t\}\}\{2\}\|u\_\{t\}\|\+\\frac\{\|b\_\{t\}\|\}\{2\}\\leq\\frac\{L\_\{\\max\}\}\{2\}\|u\_\{t\}\|\+\\frac\{b\_\{\\max\}\}\{2\}\.
Iterating this inequality gives

\|uT\|≤\(Lmax2\)T​\|u0\|\+12​∑t=0T−1\(Lmax2\)T−1−t​\|bt\|\|u\_\{T\}\|\\leq\\left\(\\frac\{L\_\{\\max\}\}\{2\}\\right\)^\{T\}\|u\_\{0\}\|\+\\frac\{1\}\{2\}\\sum\_\{t=0\}^\{T\-1\}\\left\(\\frac\{L\_\{\\max\}\}\{2\}\\right\)^\{T\-1\-t\}\|b\_\{t\}\|Whenbt=0b\_\{t\}=0, this reduces to

\|uT\|≤\(Lmax2\)T​\|u0\|\|u\_\{T\}\|\\leq\\left\(\\frac\{L\_\{\\max\}\}\{2\}\\right\)^\{T\}\|u\_\{0\}\|
Since

Lmax2<1,\\frac\{L\_\{\\max\}\}\{2\}<1,the right\-hand side converges to zero asT→∞T\\to\\infty\. Therefore

independently of the initial mixture\.

∎

### Proof of Theorem[3](https://arxiv.org/html/2609.02049#Thmtheorem3)\(context\-dimension lower bound\)

###### Proof\.

Writea~​\(x\)=f​\(a¯,r​\(x\)\)\\widetilde\{a\}\(x\)=f\\\!\\left\(\\bar\{a\},r\(x\)\\right\)for the corrected mixture at contextual representationxx, and define

F​\(x\)=Φ​\(x,a~​\(x\)\)\.F\(x\)=\\Phi\\\!\\left\(x,\\widetilde\{a\}\(x\)\\right\)\.Eq\.[17](https://arxiv.org/html/2609.02049#S5.E17)says exactly thatF⁡\(x\)=a¯F\(x\)=\\bar\{a\}for everyxxin a neighborhood ofx0x\_\{0\}\. A map that is constant on an open set has vanishing derivative there, soD​F​\(x0\)=0DF\(x\_\{0\}\)=0\.

Throughout,Dx​ΦD\_\{x\}\\PhiandDa​ΦD\_\{a\}\\Phidenote the partial derivatives ofΦ\\Phiwith respect to its first and second arguments, matching the notation of Eq\.[18](https://arxiv.org/html/2609.02049#S5.E18), and every derivative below is evaluated at the anchor point\(x0,a~​\(x0\)\)=\(x0,f⁡\(a¯,r⁡\(x0\)\)\)\\left\(x\_\{0\},\\widetilde\{a\}\(x\_\{0\}\)\\right\)=\\left\(x\_\{0\},f\(\\bar\{a\},r\(x\_\{0\}\)\)\\right\)\. Note thatxxentersFFthrough both arguments ofΦ\\Phi, so the two contributions must be summed\. SinceΦ\\Phi,ffandrrare continuously differentiable, the chain rule gives

0=D​F​\(x0\)=Dx​Φ\+Da​Φ​Dr​f​Dx​r,0=DF\(x\_\{0\}\)=D\_\{x\}\\Phi\+D\_\{a\}\\Phi\\,D\_\{r\}f\\,D\_\{x\}r,and therefore

Dx​Φ=−Da​Φ​Dr​f​Dx​r\.D\_\{x\}\\Phi=\-\\,D\_\{a\}\\Phi\\,D\_\{r\}f\\,D\_\{x\}r\.The right\-hand side factors throughDx​rD\_\{x\}r, andrrtakes values inℝm\\mathbb\{R\}^\{m\}, sorank⁡\(Dx​r\)≤m\\operatorname\{rank\}\(D\_\{x\}r\)\\leq m\. A product’s rank is at most the rank of any factor, hence

rank⁡\(Dx​Φ\)≤rank⁡\(Dx​r\)≤m,\\operatorname\{rank\}\\\!\\left\(D\_\{x\}\\Phi\\right\)\\leq\\operatorname\{rank\}\\\!\\left\(D\_\{x\}r\\right\)\\leq m,which is Eq\.[18](https://arxiv.org/html/2609.02049#S5.E18)\. If in addition local variations of the contextual representation move the output independently in allK−1K\-1mixture directions — that is, ifDx​ΦD\_\{x\}\\Phihas full rankK−1K\-1— thenm≥K−1m\\geq K\-1\. ∎