MUX: Continuous Reasoning via Multiplexed Tokens

# MUX: Continuous Reasoning via Multiplexed Tokens
Source: [https://arxiv.org/html/2607.18264](https://arxiv.org/html/2607.18264)
###### Abstract

Language models solve complex problems by articulating intermediate reasoning steps in natural language\. While effective, this process is computationally bottlenecked: each reasoning step conveys only a single subword, and many are spent expressing a thought instead of carrying out computation\. We proposeMUX, a simple method for high\-bandwidth and compact reasoning based on distillation of discrete reasoning into continuous multiplexed tokens in a latent space\. Here, each latent token is trained to represent a weighted linear superposition \(multiplexing\) of a span of discrete reasoning subwords, where this superposition is lossless by construction and the span can be fully recovered \(demultiplexing\)\. We prove that simple position\-dependent weightings, such as suitable geometric decay, support lossless multiplexing, which in turn prevents shortcut behaviors caused by latent collapse\. We further show that multiplexed reasoning can perform parallel exploration in problems that require search\. Across 32 evaluation settings spanning four language models,MUXoutperforms strong latent reasoning baselines\. Ablation and probing analyses further show that the learned latent tokens encode faithful and interpretable reasoning\. Our results suggest that lossless superposition as local learning targets constitutes a sufficient condition for achieving strong and efficient latent continuous reasoning\. Code:[https://github\.com/MisakiTaro0414/mux](https://github.com/MisakiTaro0414/mux)

## 1Introduction

Modern language models are capable of solving complex problems in domains such as mathematics, coding, and commonsense tasks through their*reasoning*mechanism\(Hurstet al\.,[2024](https://arxiv.org/html/2607.18264#bib.bib54); Anilet al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib55); Touvronet al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib56)\)\. In autoregressive language models, this mechanism typically involves verbalizing intermediate solution steps in natural language before producing the final answer\(Nyeet al\.,[2021](https://arxiv.org/html/2607.18264#bib.bib1); Weiet al\.,[2022](https://arxiv.org/html/2607.18264#bib.bib2); Kojimaet al\.,[2022](https://arxiv.org/html/2607.18264#bib.bib60)\)\. However, this mode of operation imposes a strict constraint on the computational bandwidth since each reasoning step transmits only a single subword\. Moreover, many of these steps are redundant\(Xiaet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib22); Liet al\.,[2026b](https://arxiv.org/html/2607.18264#bib.bib23)\), since the model mirrors problem\-solving patterns learned from human\-generated corpora, which are inherently more optimized for communication than computation\. These limitations motivate the development of approaches that enable higher\-bandwidth and more compact reasoning in language models\.

Reasoning in continuous latent spaces has emerged as an alternative paradigm, where a language model sequentially predicts continuous vectors instead of subwords before answering\(Haoet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib11); Xuet al\.,[2025b](https://arxiv.org/html/2607.18264#bib.bib12)\)\. These latent reasoning approaches have high bandwidths, since each step can convey multiple subwords simultaneously by encoding them*in superposition*\. This notably enables exploring different problem\-solving paths in parallel, offering potential improvements in planning and search tasks\(Zhuet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib17); Gozetenet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib18)\)\. Despite such potential, latent reasoning methods have not yet been widely adopted, in part because they are notoriously hard to learn\. One class of methods relies on temporal backpropagation of*trajectory\-level losses*\(Haoet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib11); Shenet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib13)\), which tend to produce shortcut or uninformative latent tokens\(Zhanget al\.,[2025c](https://arxiv.org/html/2607.18264#bib.bib70); Cuiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib74)\)\. Other approaches define*local distillation losses*for each latent token based on discrete reasoning traces\(Weiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib15); Kuzinaet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib16)\)\. These methods avoid shortcuts, but at the cost of additional technical complexity such as autoregressive decoders or cache compression, as well as potentially restricting the ability to maintain diverse hypotheses needed for search\(Cuiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib74)\)\. Such apparent tradeoff motivates our key question:*What should constitute the supervision target for continuous latent reasoning?*

![Refer to caption](https://arxiv.org/html/2607.18264v1/x1.png)Figure 1:Overview ofMUX\.Given a question, the language model predicts a sequence of continuous latent reasoning tokens\. Each token is linearly projected to the vocabulary space and is trained to represent a local span of discrete reasoning steps\. We construct the learning target from each discrete span as a weighted average of one\-hot encodings, and train each latent token with local KL divergence loss\. The answer is trained with standard cross\-entropy loss\.We address this question withMUX, a simple and novel training method for high\-bandwidth, compact latent reasoning in language models\. Here, we view the local distillation setting, where each latent token is supervised to represent a span of discrete reasoning steps, as learning a fixed\-dimensional, continuous signal that encodes a varying\-length categorical signal\. This naturally connects to*multiplexing*in communication systems, which allows multiple logical signals to share a common physical medium\. Drawing inspiration from code\-division multiplexing\(Fanet al\.,[2020](https://arxiv.org/html/2607.18264#bib.bib82)\), we propose to leverage a multiplexed encoding for variable\-length categorical signals based on*linear superpositions*of one\-hot encodings\. Our hypothesis is that \(i\) local distillation from such multiplexed targets induces faithful latent reasoning, provided that these targets are*lossless*encodings of discrete reasoning spans, while \(ii\) natively supporting joint encoding of multiple possibilities, thanks to their superposed construction\. This is also conceptually simpler than prior methods, as it does not require an auxiliary autoregressive decoder or a compressed cache target\. Our main contributions can be summarized as:

1. \(1\)Latent reasoning via multiplexed tokens\.We introduceMUX, a local distillation method for continuous latent reasoning based on multiplexed targets \([Figure˜1](https://arxiv.org/html/2607.18264#S1.F1)\)\. For each latent token, we define a vocabulary\-space target by taking a position\-weighted linear superposition of one\-hot encodings in its corresponding discrete reasoning span\. The model is trained to match this target through a linear\-softmax head with a KL loss\.
2. \(2\)Lossless multiplexing\.We identify simple classes of positional weightings that guarantee lossless multiplexing, such that each superposed target fully preserves the discrete reasoning span it represents\. These include geometric, sinusoidal, and rotary weightings, characterized by a subset\-sum separation condition\. We show that lossless multiplexing prevents shortcut behaviors found in prior methods caused by latent collapse\.
3. \(3\)Parallel search via multiplexing\.We show that, in problems requiring breadth\-first search \(BFS\), multiplexed tokens are expressive enough to represent and update multiple hypotheses simultaneously, owing to their natively superposed construction, thereby implementing each BFS step using a single latent token\. The result implies that parallel search can naturally emerge from serial supervision via multiplexing\.
4. \(4\)Empirical results\.MUXis the best latent reasoning method across 32 mathematical reasoning settings spanning two training corpora, four language models, and four test sets\. It also surpasses strong discrete and continuous reasoning baselines on two search benchmarks\. Through probing analysis, we show that the learned latent tokens encode interpretable reasoning content and contribute meaningfully to final prediction\.

## 2Related work

We provide an overview of related work\. An extended discussion can be found in[Section˜7\.5](https://arxiv.org/html/2607.18264#S7.SS5)\.

#### Reasoning in language models\.

Prior work has shown that language models benefit from making intermediate computations explicit\. Early scratchpad methods\(Nyeet al\.,[2021](https://arxiv.org/html/2607.18264#bib.bib1)\)showed that learning intermediate computation steps improves algorithmic problem solving\. Chain\-of\-thought \(CoT\) prompting\(Weiet al\.,[2022](https://arxiv.org/html/2607.18264#bib.bib2); Kojimaet al\.,[2022](https://arxiv.org/html/2607.18264#bib.bib60)\)established natural language reasoning as a general mechanism for arithmetic, logical, and commonsense problem solving\. Recent work has identified inefficiencies in language reasoning, showing that many reasoning tokens can be pruned\(Liet al\.,[2026b](https://arxiv.org/html/2607.18264#bib.bib23); Zhanget al\.,[2025a](https://arxiv.org/html/2607.18264#bib.bib46)\)or compressed\(Xiaet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib22); Liet al\.,[2026a](https://arxiv.org/html/2607.18264#bib.bib47)\)with small degradation in accuracy\. We share this motivation, but instead of shortening discrete reasoning at inference time, we distill their traces into compact continuous reasoning through training\.

#### Reasoning in continuous latent spaces\.

A growing line of work explores reasoning in continuous latent spaces, broadly categorized into global and local methods\. Global methods supervise the answer token or trajectory endpoint and learn latent reasoning via temporal backpropagation, as in Coconut\(Haoet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib11)\)and CODI\(Shenet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib13)\)\. Local methods instead supervise each latent token to represent a span of discrete steps by aligning them in some choice of representation space, typically via auxiliary modules: SIM\-CoT\(Weiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib15)\)aligns in autoregressively decoded text space, KaVa\(Kuzinaet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib16)\)in a key\-value cache space\. We question their complexity, and instead leverage linearly superposed representations in vocabulary space\. While some prior work autoregress vocabulary\-space vectors\(Zhanget al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib48); Denget al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib59); Tanget al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib49)\), we use vocabulary projection only for supervision and autoregress directly in latent space at inference\.

Theoretical works studied benefits of continuous latent reasoning, in particular superposition and parallel search\(Gozetenet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib18); Zhuet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib17); Wuet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib14)\)\. On the other hand, recent analyses caution that latent reasoning need not automatically encode faithful computations, sometimes acting as uninterpretable placeholders or exploiting shortcuts\(Zhanget al\.,[2025c](https://arxiv.org/html/2607.18264#bib.bib70)\)\.Cuiet al\.\([2026](https://arxiv.org/html/2607.18264#bib.bib74)\)find pervasive shortcut behavior in global methods, and report that existing local methods mitigate shortcuts but trade off the ability to maintain diverse hypotheses in latent tokens\.Dilgren and Wiegreffe \([2026](https://arxiv.org/html/2607.18264#bib.bib75)\)propose vocabulary projection as a tool for interpreting latent tokens, arguing interpretability itself is a signal of reasoning correctness\. Together with earlier probing studies\(Cywińskiet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib76); Liang and Pan,[2026](https://arxiv.org/html/2607.18264#bib.bib77)\), these motivate supervision methods whose targets are locally decodable, tied to explicit reasoning, and compatible with parallel search by design\.

## 3MUX: Continuous reasoning via multiplexed tokens

### 3\.1Problem setup

#### Language reasoning\.

Let𝒱\\mathcal\{V\}be a discrete vocabulary of subwords and let𝒱∗\\mathcal\{V\}^\{\*\}be its associated text space\. We denote text of lengthLLby𝐲=\(𝐲1,…,𝐲L\)\{\\bf y\}=\(\{\\bf y\}^\{1\},\.\.\.,\{\\bf y\}^\{L\}\)with each𝐲l∈𝒱\{\\bf y\}^\{l\}\\in\\mathcal\{V\}\. Language models generate continuations of a text by autoregressively predicting the next subword\. While a language model may directly answer a given question𝐪↦𝐚^\{\\bf q\}\\mapsto\\hat\{\\bf a\}by continuation, prompting an intermediate reasoning𝐫∈𝒱∗\{\\bf r\}\\in\\mathcal\{V\}^\{\*\}before answering\(𝐪,𝐫\)↦𝐚^\(\{\\bf q\},\{\\bf r\}\)\\mapsto\\hat\{\\bf a\}improves performance\. This is, however, computationally inefficient\.

#### Continuous latent reasoning\.

To overcome the efficiency limitations of discrete reasoning, we reason in a choice of continuous vector spaceX=ℝdX=\\mathbb\{R\}^\{d\}, where we denote byX∗X^\{\*\}the set of vector sequences\. For each question𝐪\{\\bf q\}, we would like to train a language model to articulate latent reasoning𝐱∈X∗\{\\bf x\}\\in X^\{\*\}by autoregressing on continuous tokens𝐱1,𝐱2,…∈X\{\\bf x\}\_\{1\},\{\\bf x\}\_\{2\},\.\.\.\\in Xbefore answering\(𝐪,𝐱\)↦𝐚^\(\{\\bf q\},\{\\bf x\}\)\\mapsto\\hat\{\\bf a\}\. In practice, we treatX∗X^\{\*\}asXK=ℝK×dX^\{K\}=\\mathbb\{R\}^\{K\\times d\}for a choice ofKK, which restricts each reasoning to a sequence ofKKvectors\. Following prior work, we assume availability of triples\(𝐪,𝐫,𝐚\)\(\{\\bf q\},\{\\bf r\},\{\\bf a\}\)containing discrete reasoning traces𝐫\{\\bf r\}, and use them to learn latent reasoning𝐱\{\\bf x\}via distillation\.

#### Local distillation\.

We focus on local distillation, where latent tokens are supervised with local spans of discrete reasoning steps\. We assume each discrete trace𝐫\{\\bf r\}is chunked into spans\(𝐫1,…,𝐫M\)\(\{\\bf r\}\_\{1\},\.\.\.,\{\\bf r\}\_\{M\}\)where𝐫i=\(ri1,…,riSi\)∈𝒱Si\{\\bf r\}\_\{i\}=\(r\_\{i\}^\{1\},\\ldots,r\_\{i\}^\{S\_\{i\}\}\)\\in\\mathcal\{V\}^\{S\_\{i\}\}\. For example, in algorithmic and mathematical tasks, each span can be a step of computation, and in natural language, each span can be a sentence\. If a trace has more spans than latent tokens,M\>KM\>K, some of the spans are merged heuristically \([Section˜11\.3](https://arxiv.org/html/2607.18264#S11.SS3)\); we thus assumeM≤KM\\leq Konward\. IfM<KM<K, some latent tokens have no aligned span\. We let𝒦\\mathcal\{K\}denote the subset of latent token positions with nonempty span, noting\|𝒦\|=M\|\\mathcal\{K\}\|=M\.

In local distillation, latent reasoning𝐱\{\\bf x\}is trained so that each token𝐱i∈ℝd\{\\bf x\}\_\{i\}\\in\\mathbb\{R\}^\{d\}matches a span𝐫i∈𝒱∗\{\\bf r\}\_\{i\}\\in\\mathcal\{V\}^\{\*\}in some representation space𝒵\\mathcal\{Z\}\. This goal can be formalized asf​\(𝐱i\)=g​\(𝐫i\)f\(\{\\bf x\}\_\{i\}\)=g\(\{\\bf r\}\_\{i\}\)for some choice of mapsf:ℝd→𝒵f:\\mathbb\{R\}^\{d\}\\to\\mathcal\{Z\}andg:𝒱∗→𝒵g:\\mathcal\{V\}^\{\*\}\\to\\mathcal\{Z\}\. The representation space and the maps constitute the core design decision of local distillation methods\. SIM\-CoT\(Weiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib15)\)uses𝒵=𝒱∗\\mathcal\{Z\}=\\mathcal\{V\}^\{\*\}with an autoregressivef:ℝd→𝒱∗f:\\mathbb\{R\}^\{d\}\\to\\mathcal\{V\}^\{\*\}andg=idg=\{\\rm id\}, and KaVa\(Kuzinaet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib16)\)takes as𝒵\\mathcal\{Z\}the space of key\-value cache and performs cache distillation\. In contrast, we simply choose𝒵\\mathcal\{Z\}as the vocabulary simplexΔ\|𝒱\|−1\\Delta^\{\|\\mathcal\{V\}\|\-1\}, or the space of\|𝒱\|\|\\mathcal\{V\}\|\-dimensional probability vectors\.

### 3\.2Local distillation by multiplexing

We now present our method for continuous latent reasoning via local distillation\. To motivate it, we consider the case where𝒵\\mathcal\{Z\}is a fixed\-dimensional vector space\. Then,g:𝒱∗→𝒵g:\\mathcal\{V\}^\{\*\}\\to\\mathcal\{Z\}can be viewed as an operator that combines a variable\-dimensional categorical signali↦𝐫ii\\mapsto\{\\bf r\}\_\{i\}into one, fixed\-dimensional continuous signali↦g​\(𝐫i\)i\\mapsto g\(\{\\bf r\}\_\{i\}\), which defines the learning targetf∗​\(𝐱i\)=g​\(𝐫i\)f^\{\*\}\(\{\\bf x\}\_\{i\}\)=g\(\{\\bf r\}\_\{i\}\)for each latent token𝐱i\{\\bf x\}\_\{i\}via an optimal decoderf∗f^\{\*\}\. Given this observation, it is natural to conceptualizeggas a type of*multiplexed*encoding of variable\-length categorical signal,g=𝗆𝗎𝗑g=\\mathsf\{mux\}\. We now identify the core requirement for local distillation based on multiplexing as follows\.

###### Definition 1\(Multiplexing\)\.

A map𝗆𝗎𝗑:𝒱∗→𝒵\\mathsf\{mux\}:\\mathcal\{V\}^\{\*\}\\to\\mathcal\{Z\}isspanwise injectiveif𝗆𝗎𝗑\|𝒱S\\mathsf\{mux\}\|\_\{\\mathcal\{V\}^\{S\}\}is injective for any span lengthS≥1S\\geq 1\. We say latent reasoning\(𝐱1,…,𝐱K\)\(\{\\bf x\}\_\{1\},\\ldots,\{\\bf x\}\_\{K\}\)under an optimal decoderf∗:ℝd→𝒵f^\{\*\}:\\mathbb\{R\}^\{d\}\\to\\mathcal\{Z\}multiplexesdiscrete reasoning spans\(𝐫1,…,𝐫M\)\(\{\\bf r\}\_\{1\},\\ldots,\{\\bf r\}\_\{M\}\)if there exists a spanwise injective𝗆𝗎𝗑\\mathsf\{mux\}satisfying

f∗​\(𝐱i\)=𝗆𝗎𝗑​\(𝐫i\),∀i∈𝒦\.f^\{\*\}\(\{\\bf x\}\_\{i\}\)=\\mathsf\{mux\}\(\{\\bf r\}\_\{i\}\),\\quad\\forall i\\in\\mathcal\{K\}\.\(1\)

Equation \([1](https://arxiv.org/html/2607.18264#S3.E1)\) requires that each latent token represents a local span of a discrete reasoning trace through multiplexing\. As a training objective, it is used to drivef​\(𝐱i\)f\(\{\\bf x\}\_\{i\}\)towardf∗​\(𝐱i\)=𝗆𝗎𝗑​\(𝐫i\)f^\{\*\}\(\{\\bf x\}\_\{i\}\)=\\mathsf\{mux\}\(\{\\bf r\}\_\{i\}\)by jointly learning the latent reasoning𝐱\{\\bf x\}and the decoderff\. Injectivity means that the representation𝗆𝗎𝗑\\mathsf\{mux\}is lossless, admitting an inverse \(demultiplexing\)\. Under lossless multiplexing, the local distillation target𝗆𝗎𝗑​\(𝐫i\)\\mathsf\{mux\}\(\{\\bf r\}\_\{i\}\)is fixed\-dimensional, allowing for scalable optimization, and encodes full information of each span𝐫i\{\\bf r\}\_\{i\}, enabling faithful reasoning\.

![Refer to caption](https://arxiv.org/html/2607.18264v1/x2.png)Figure 2:Lossless multiplexing of a span<<5\+3=8\>\>through position\-weighted linear superposition\.#### Multiplexing via linear superposition\.

Constructing a spanwise lossless multiplexer is nontrivial, as it must handle variable\-length categorical signals\. Here, inspired by code\-division schemes in communication systems, we propose a class of simple and training\-free multiplexers based on linear superposition of one\-hot encodings in the vocabulary space\. Concretely, for each discrete reasoning span𝐫i=\(ri1,…,riSi\)\{\\bf r\}\_\{i\}=\(r\_\{i\}^\{1\},\\ldots,r\_\{i\}^\{S\_\{i\}\}\), we define

𝗆𝗎𝗑​\(𝐫i\)≔∑j=1Siαj\(i\)​onehot​\(rij\),αj\(i\)≔wj∑ℓ=1Siwℓ,\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\\coloneqq\\sum\_\{j=1\}^\{S\_\{i\}\}\\alpha\_\{j\}^\{\(i\)\}\\,\\mathrm\{onehot\}\(r\_\{i\}^\{j\}\),\\qquad\\alpha\_\{j\}^\{\(i\)\}\\coloneqq\\frac\{w\_\{j\}\}\{\\sum\_\{\\ell=1\}^\{S\_\{i\}\}w\_\{\\ell\}\},\(2\)wherew\[⋅\]:ℕ→ℝ\+w\_\{\[\\cdot\]\}:\\mathbb\{N\}\\to\\mathbb\{R\}\_\{\+\}is a choice of positional weighting\. Because the coefficientsαj\(i\)\\alpha\_\{j\}^\{\(i\)\}are positive and normalized,𝗆𝗎𝗑​\(𝐫i\)\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)always lies in the vocabulary simplexΔ\|𝒱\|−1\\Delta^\{\|\\mathcal\{V\}\|\-1\}\. To match such targets following \([1](https://arxiv.org/html/2607.18264#S3.E1)\), we decode each latent token𝐱i\{\\bf x\}\_\{i\}through a linear\-softmax head

f​\(𝐱i\)≔softmax​\(W​𝐱i/τ\),W∈ℝ\|𝒱\|×d,f\(\\mathbf\{x\}\_\{i\}\)\\coloneqq\\mathrm\{softmax\}\(W\\mathbf\{x\}\_\{i\}/\\tau\),\\qquad W\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\},\(3\)whereWWis the pretrained language model’s unembedding layer andτ\>0\\tau\>0is a temperature variable\. At inference time, the model still autoregresses latent tokens𝐱i\{\\bf x\}\_\{i\}in hidden space, and the vocabulary projection is needed only for their supervision\.

#### Positional weighting\.

We propose three families of positional weightingsw\[⋅\]w\_\{\[\\cdot\]\}\([Figure˜2](https://arxiv.org/html/2607.18264#S3.F2)\):

1. \(1\)*Geometric\.*wj=ρj−1w\_\{j\}=\\rho^\{j\-1\}with decay rateρ∈\(0,1\)\\rho\\in\(0,1\)\. Earlier positions receive exponentially more weight, producing a monotonically decaying profile\.
2. \(2\)*Sinusoidal\.*wj=exp⁡\(λ​sj\)w\_\{j\}=\\exp\(\\lambda s\_\{j\}\)with scoressj=sin⁡\(π2⋅j−1max⁡\(S−1,1\)\)s\_\{j\}=\\sin\(\\tfrac\{\\pi\}\{2\}\\cdot\\tfrac\{j\-1\}\{\\max\(S\-1,1\)\}\)and scaleλ\>0\\lambda\>0\. This induces a monotonically increasing weighting that peaks near the end of a span\.
3. \(3\)*Rotary\.*wj=exp⁡\(λ​sj\)w\_\{j\}=\\exp\(\\lambda s\_\{j\}\)with scoressj=1P​∑p=1Pcos⁡\(θp​\(j−1\)\)s\_\{j\}=\\tfrac\{1\}\{P\}\\sum\_\{p=1\}^\{P\}\\cos\(\\theta\_\{p\}\(j\-1\)\), where\(θp\)p≤P\(\\theta\_\{p\}\)\_\{p\\leq P\}is a set of positive frequencies analogous to rotary position embeddings\(Suet al\.,[2024](https://arxiv.org/html/2607.18264#bib.bib52)\)\. Averaging cosine components across frequencies yields an expressive positional weighting\.

All of the above weightings yield spanwise lossless multiplexing when configured properly, as we show in[Section˜4\.1](https://arxiv.org/html/2607.18264#S4.SS1)\. This outcome is not trivial\. The sum𝗆𝗎𝗑​\(𝐫i\)\\mathsf\{mux\}\(\{\\bf r\}\_\{i\}\)records only the*total*mass of each unique subword in a span𝐫i\{\\bf r\}\_\{i\}, so if a subword appears more than once, its positions in the span can be ambiguous\. For example, with uniform weightingαj=1/S\\alpha\_\{j\}=1/S, the mass of a subword only counts how many times it appears, dropping the positions\.

Therefore, the key to lossless multiplexing is to choose positional weightsα1,…,αS\\alpha\_\{1\},\.\.\.,\\alpha\_\{S\}such that different sets of positions always produce different total masses\. Our theory in[Section˜4\.1](https://arxiv.org/html/2607.18264#S4.SS1)formalizes this as a subset\-sum separation condition on the weights, satisfied by all of our weightings for proper hyperparameters\. Then, even if a subword appears multiple times, its total mass uniquely determines which positions it occupies, so the original span can be recovered exactly from its multiplexing\. In[Section˜4\.2](https://arxiv.org/html/2607.18264#S4.SS2), we show that this property consequently prevents shortcut behaviors that are caused by the collapse of latent tokens\.

#### Training objective\.

In order to train a language model to perform continuous latent reasoning, we use a composite lossℒ=ℒanswer\+β​ℒlocal\+γ​ℒglobal\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{answer\}\}\+\\beta\\,\\mathcal\{L\}\_\{\\mathrm\{local\}\}\+\\gamma\\,\\mathcal\{L\}\_\{\\mathrm\{global\}\}with weightsβ,γ≥0\\beta,\\gamma\\geq 0, where each term corresponds to answer prediction loss, local distillation loss under multiplexed targets \([2](https://arxiv.org/html/2607.18264#S3.E2)\), and an optional trajectory\-level loss, detailed as follows\. The answer loss is standard cross entropyℒanswer=−log⁡pθ​\(𝐚∣𝐪,𝐱1,…,𝐱K\)\\mathcal\{L\}\_\{\\mathrm\{answer\}\}=\-\\log p\_\{\\theta\}\(\\mathbf\{a\}\\mid\\mathbf\{q\},\\mathbf\{x\}\_\{1\},\\ldots,\\mathbf\{x\}\_\{K\}\), wherepθp\_\{\\theta\}is the likelihood evaluated by the language model\. For the local distillation loss, recall that the decoderf:ℝd→Δ\|𝒱\|−1f:\\mathbb\{R\}^\{d\}\\to\\Delta^\{\|\\mathcal\{V\}\|\-1\}is a linear projection followed by tempered softmax \([3](https://arxiv.org/html/2607.18264#S3.E3)\)\. For each latent token𝐱i∈𝒦\{\\bf x\}\_\{i\\in\\mathcal\{K\}\}aligned with a nonempty span𝐫i\\mathbf\{r\}\_\{i\}, we minimize the KL divergence between the model predictionf​\(𝐱i\)f\(\{\\bf x\}\_\{i\}\)and the multiplexed target𝗆𝗎𝗑​\(𝐫i\)\\mathsf\{mux\}\(\{\\bf r\}\_\{i\}\):

ℒlocal=1\|𝒦\|​∑i∈𝒦KL​\(𝗆𝗎𝗑​\(𝐫i\)∥f​\(𝐱i\)\)\.\\mathcal\{L\}\_\{\\mathrm\{local\}\}=\\frac\{1\}\{\|\\mathcal\{K\}\|\}\\sum\_\{i\\in\\mathcal\{K\}\}\\mathrm\{KL\}\\\!\\left\(\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\\;\\big\\\|\\;f\(\\mathbf\{x\}\_\{i\}\)\\right\)\.\(4\)Lastly, followingShenet al\.\([2025](https://arxiv.org/html/2607.18264#bib.bib13)\), we use an optional trajectory\-level loss that aligns the hidden features at the answer token in the language model with continuous reasoning, with respect to those from a model with discrete reasoning, trained with standard next\-token prediction on discrete reasoning traces\. We employ parameter sharing between the two models, which offers efficiency\. The trajectory\-level loss provides learning signal for the “spare” tokens𝐱M\+1,…,𝐱K\\mathbf\{x\}\_\{M\+1\},\.\.\.,\\mathbf\{x\}\_\{K\}whenM<KM<K, which lack local targets\. For the respective hidden features𝐡answercont\\mathbf\{h\}\_\{\\mathrm\{answer\}\}^\{\\mathrm\{cont\}\}and𝐡answerdisc\\mathbf\{h\}\_\{\\mathrm\{answer\}\}^\{\\mathrm\{disc\}\}, we useℒglobal=‖𝐡answercont−sg⁡\(𝐡answerdisc\)‖22\\mathcal\{L\}\_\{\\mathrm\{global\}\}=\\\|\\mathbf\{h\}\_\{\\mathrm\{answer\}\}^\{\\mathrm\{cont\}\}\-\\operatorname\{sg\}\(\\mathbf\{h\}\_\{\\mathrm\{answer\}\}^\{\\mathrm\{disc\}\}\)\\\|\_\{2\}^\{2\}, wheresg⁡\(⋅\)\\operatorname\{sg\}\(\\cdot\)denotes the stop\-gradient operator\. Together, the composite loss provides direct learning signal for every latent token as well as answer prediction\.

## 4Theoretical analysis

We organize the theory around the utility of multiplexing for latent reasoning\. In[Section˜4\.1](https://arxiv.org/html/2607.18264#S4.SS1), we ask when multiplexing is lossless, and which positional weightings satisfy this criterion\. Losslessness guarantees that every latent token encodes faithful computation without degrading into uninformative placeholders\. In[Section˜4\.2](https://arxiv.org/html/2607.18264#S4.SS2), we make this precise and show that multiplexing prevents latent collapse\. In[Section˜4\.3](https://arxiv.org/html/2607.18264#S4.SS3), we prove that multiplexed tokens can implement parallel search by encoding an entire search frontier\. All proofs are in[Section˜9\.1](https://arxiv.org/html/2607.18264#S9.SS1)\.

### 4\.1Lossless multiplexing

Consider multiplexing a discrete reasoning span𝐫i=\(ri1,…,riS\)\{\\bf r\}\_\{i\}=\(r\_\{i\}^\{1\},\.\.\.,r\_\{i\}^\{S\}\)into a continuous token𝗆𝗎𝗑​\(𝐫i\)\\mathsf\{mux\}\(\{\\bf r\}\_\{i\}\)using normalized masses𝜶=\(α1,…,αS\)\\boldsymbol\{\\alpha\}\\\!=\\\!\(\\alpha\_\{1\},\.\.\.,\\alpha\_\{S\}\)\([2](https://arxiv.org/html/2607.18264#S3.E2)\)\. Each span𝐫i\{\\bf r\}\_\{i\}is a short sequence of reasoning tokens, and the weightsαj\\alpha\_\{j\}determine how much each position contributes to the resulting latent representation\. Our goal is to identify conditions that make𝗆𝗎𝗑\\mathsf\{mux\}spanwise lossless or injective \([Definition˜1](https://arxiv.org/html/2607.18264#Thmtheorem1)\)\. The following quantity will be central in our results:

###### Definition 2\(Subset\-sum separation\)\.

Let𝒞S\\mathcal\{C\}\_\{S\}be the set of all nonzero sequences𝐜=\(c1,…,cS\)\{\\bf c\}=\(c\_\{1\},\.\.\.,c\_\{S\}\)taking values in\{−1,0,1\}\\\{\-1,0,1\\\}and consider the following measure of subset\-sum collisions:

ℰ​\(𝜶\)≔min𝐜∈𝒞S⁡\|∑j=1Scj​αj\|\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\\coloneqq\\min\_\{\{\\bf c\}\\in\\mathcal\{C\}\_\{S\}\}\\left\|\\sum\_\{j=1\}^\{S\}c\_\{j\}\\alpha\_\{j\}\\right\|\(5\)

Intuitively,ℰ​\(𝜶\)\>0\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\>0if and only if there are no distinct subsets of\{1,…,S\}\\\{1,\.\.\.,S\\\}having an identical total mass\. We now characterize the exact criterion for lossless multiplexing as follows\.

###### Proposition 3\(Span\-level lossless multiplexing\)\.

Assume\|𝒱\|\>1\|\\mathcal\{V\}\|\>1and fix a span lengthSS\. Then the map𝗆𝗎𝗑:𝒱S→Δ\|𝒱\|−1\\mathsf\{mux\}:\\mathcal\{V\}^\{S\}\\to\\Delta^\{\|\\mathcal\{V\}\|\-1\}is injective if and only ifℰ​\(𝛂\)\>0\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\>0\.

The result shows that every finite span of discrete reasoning can be recovered exactly \(demultiplexed\) from its weighted linear superposition if the subset sums of the weights never collide\. Based on this single\-span case, we now consider an extension to full reasoning trace; the only additional ingredient is that a full reasoning trace is chunked into several spans, possibly of different lengths\.

###### Corollary 4\(Trace\-level lossless multiplexing\)\.

Let𝐫=\(𝐫1,…,𝐫M\)\\mathbf\{r\}=\(\\mathbf\{r\}\_\{1\},\\ldots,\\mathbf\{r\}\_\{M\}\)be a discrete reasoning trace where each𝐫i\{\\bf r\}\_\{i\}has lengthSiS\_\{i\}and normalized masses𝛂\(i\)\\boldsymbol\{\\alpha\}^\{\(i\)\}\. Ifℰ​\(𝛂\(i\)\)\>0\\mathcal\{E\}\\bigl\(\\boldsymbol\{\\alpha\}^\{\(i\)\}\\bigr\)\>0for everyii, then𝐫\\mathbf\{r\}is uniquely recoverable from the collection of multiplexed targets𝗆𝗎𝗑​\(𝐫i\)\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)together withSiS\_\{i\}and𝛂\(i\)\\boldsymbol\{\\alpha\}^\{\(i\)\}\.

We now identify the hyperparameter choices for our positional weightings \([Section˜3\.2](https://arxiv.org/html/2607.18264#S3.SS2)\) that support lossless multiplexing\. This can be characterized compactly: geometric weights admit an exact algebraic criterion, and the remaining exponential weights are injective if the scores are distinct\.

###### Proposition 5\(Weightings for lossless multiplexing\)\.

1. \(i\)*Geometric\.*Forwj=ρj−1w\_\{j\}=\\rho^\{j\-1\}, multiplexing is injective iffρ\\rhois not a root of any nonzero polynomial∑j=1Scj​xj−1\\sum\_\{j=1\}^\{S\}c\_\{j\}x^\{j\-1\}with coefficientscj∈\{−1,0,1\}c\_\{j\}\\in\\\{\-1,0,1\\\}\. Ifρ∈\(0,1\)\\rho\\in\(0,1\)is rational, this holds for all finiteSS\.
2. \(ii\)*Exponential\.*Forwj=exp⁡\(λ​sj\)w\_\{j\}=\\exp\(\\lambda s\_\{j\}\), if span lengthS≥2S\\geq 2ands1,…,sSs\_\{1\},\.\.\.,s\_\{S\}are pairwise distinct, then multiplexing is injective for all but finitely many values ofλ\\lambda\.

The sinusoidal and rotary weightings are special cases of the exponentialwj=exp⁡\(λ​sj\)w\_\{j\}=\\exp\(\{\\lambda s\_\{j\}\}\), and so the above result implies that sinusoidal weighting is generally lossless\. For rotary weightings, we prove a simple sufficient condition that all of its frequencies lie on the first decreasing branch of cosine\. Together, these results show that simple weightings can achieve spanwise lossless multiplexing\.

###### Corollary 6\(Sinusoidal and rotary weightings for lossless multiplexing\)\.

1. \(i\)For anyS≥2S\\geq 2, sinusoidal weighting yields an injective multiplexing for all but finitely manyλ\\lambda\.
2. \(ii\)If0<θp​\(S−1\)<π​∀p0\\\!<\\\!\\theta\_\{p\}\(S\\\!\-\\\!1\)\\\!<\\\!\\pi\\,\\forall p, then rotary weighting yields an injective multiplexing for all but finitely manyλ\\lambda\.

#### Finite precision\.

The results above are under exact arithmetic\. In practice, multiplexing is done in finite precision, so it is natural to ask whether our findings remain meaningful\. Letεfp\\varepsilon\_\{\\mathrm\{fp\}\}be the worst\-case error between multiplexed target and its finite\-precision rounding\. In[Section˜9\.2](https://arxiv.org/html/2607.18264#S9.SS2), we show that the separation marginℰ​\(𝜶\)\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)also governs numerical error: ifεfp<ℰ​\(𝜶\)/2\\varepsilon\_\{\\mathrm\{fp\}\}<\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)/2, then the original span remains exactly recoverable\. Under the standard unit\-roundoff model\(Goldberg,[1991](https://arxiv.org/html/2607.18264#bib.bib63); Higham,[2002](https://arxiv.org/html/2607.18264#bib.bib64)\),εfp\\varepsilon\_\{\\mathrm\{fp\}\}admits anO​\(S​u\)O\(Su\)bound for span lengthSSand arithmetic precisionuu\. For our default geometric weightingρ=0\.9\\rho=0\.9infloat32, multiplexing is lossless for all span lengths faced in experiments \(S≤11S\\leq 11\)\.

### 4\.2Latent diversity

Intuitively, lossless multiplexing is beneficial as it enforces latent tokens to hold meaningful computation\. We make this intuition precise and show thatMUXguarantees diversity of latent tokens, avoiding semantic homogenization of latent reasoning shown byWeiet al\.\([2026](https://arxiv.org/html/2607.18264#bib.bib15)\)for global methods\.

###### Definition 7\(Latent collapse\)\.

A continuous reasoning𝐱\{\\bf x\}exhibits*collapse at levelε≥0\\varepsilon\\geq 0*if

1\|𝒦\|2​∑i,j∈𝒦‖𝐱i−𝐱j‖22≤ε\.\\frac\{1\}\{\|\\mathcal\{K\}\|^\{2\}\}\\sum\_\{i,j\\in\\mathcal\{K\}\}\\\|\\mathbf\{x\}\_\{i\}\-\\mathbf\{x\}\_\{j\}\\\|\_\{2\}^\{2\}\\leq\\varepsilon\.

###### Definition 8\(Target diversity\)\.

The*target diversity*of discrete reasoning𝐫\{\\bf r\}under multiplexing is𝒟≔mini,j∈𝒦,i≠j⁡‖𝗆𝗎𝗑​\(𝐫i\)−𝗆𝗎𝗑​\(𝐫j\)‖1\\mathcal\{D\}\\coloneqq\\min\_\{\\begin\{subarray\}\{c\}i,j\\in\\mathcal\{K\},i\\neq j\\end\{subarray\}\}\\\|\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\-\\mathsf\{mux\}\(\\mathbf\{r\}\_\{j\}\)\\\|\_\{1\}\.

Whenever two discrete spans are distinct𝐫i≠𝐫j\{\\bf r\}\_\{i\}\\neq\{\\bf r\}\_\{j\}and the positional weighting is losslessℰ​\(𝜶\)\>0\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\>0,[Proposition˜3](https://arxiv.org/html/2607.18264#Thmtheorem3)guarantees𝗆𝗎𝗑​\(𝐫i\)≠𝗆𝗎𝗑​\(𝐫j\)\\mathsf\{mux\}\(\{\\bf r\}\_\{i\}\)\\neq\\mathsf\{mux\}\(\{\\bf r\}\_\{j\}\), so𝒟\>0\\mathcal\{D\}\>0\. We now show that local distillation on diverse targets forces diversity in latent tokens\. This is empirically supported in[Section˜8\.2](https://arxiv.org/html/2607.18264#S8.SS2)\.

###### Proposition 9\(Non\-collapsing guarantee for multiplexed distillation\)\.

LetW~:=W/τ\\widetilde\{W\}:=W/\\taube the scaled readout matrix\. Suppose𝒟\>0\\mathcal\{D\}\>0,‖W~‖op\>0\\\|\\widetilde\{W\}\\\|\_\{\\mathrm\{op\}\}\>0, andℒlocal≤δ<𝒟2/8\\mathcal\{L\}\_\{\\mathrm\{local\}\}\\leq\\delta<\\mathcal\{D\}^\{2\}/8\. Then

1\|𝒦\|2​∑i,j∈𝒦‖𝐱i−𝐱j‖22≥\|𝒦\|−1\|𝒦\|​\(𝒟−2​2​δ‖W~‖op​C\|𝒱\|\)2,\\frac\{1\}\{\|\\mathcal\{K\}\|^\{2\}\}\\sum\_\{i,j\\in\\mathcal\{K\}\}\\\|\{\\bf x\}\_\{i\}\-\{\\bf x\}\_\{j\}\\\|\_\{2\}^\{2\}\\;\\geq\\;\\frac\{\|\\mathcal\{K\}\|\-1\}\{\|\\mathcal\{K\}\|\}\\left\(\\frac\{\\mathcal\{D\}\-2\\sqrt\{2\\delta\}\}\{\\\|\\widetilde\{W\}\\\|\_\{\\mathrm\{op\}\}\\,C\_\{\|\\mathcal\{V\}\|\}\}\\right\)^\{2\},\(6\)whereC\|𝒱\|C\_\{\|\\mathcal\{V\}\|\}depends only on\|𝒱\|\|\\mathcal\{V\}\|\. Thus, latent tokens cannot collapse at any level below the right\-hand side\.

### 4\.3Parallel search with multiplexed reasoning

We now consider search problems where each latent token has to represent a*set*of hypotheses\. In a graph reachability problem, there may be several nodes that have been explored and are waiting to be expanded\. A continuous token can, in principle, carry such a set all at once in superposition, instead of forcing the model to commit to one possibility\. We show thatMUXpreserves this advantage\.

As a setup, consider the depth\-HHreachability problem on a finite directed graphG=\(𝒩,E\)G=\(\\mathcal\{N\},E\): given a source nodes∈𝒩s\\in\\mathcal\{N\}and a target nodet∈𝒩t\\in\\mathcal\{N\}, the task is to determine whether there is a directed paths→ts\\to tof length≤H\\leq H\. A standard breadth\-first search \(BFS\) maintains two sets at each stepkk: the frontier node setFkF\_\{k\}discovered for the first time, and the node setUkU\_\{k\}discovered so far\. Denoting byN\+​\(B\)N^\{\+\}\(B\)the out\-neighborhood of a node setBB, each BFS step updates, fromF0=\{s\},U0=\{s\}F\_\{0\}=\\\{s\\\},U\_\{0\}=\\\{s\\\}:

Fk\+1=N\+​\(Fk\)∖Uk,Uk\+1=Uk∪Fk\+1,k=0,…,H−1\.F\_\{k\+1\}=N^\{\+\}\(F\_\{k\}\)\\setminus U\_\{k\},\\qquad U\_\{k\+1\}=U\_\{k\}\\cup F\_\{k\+1\},\\qquad k=0,\\dots,H\-1\.Suppose the discrete reasoning at stepkkis𝐫k=\(rk1,…,rk\|Fk\|\)\\mathbf\{r\}\_\{k\}=\(r\_\{k\}^\{1\},\\dots,r\_\{k\}^\{\|F\_\{k\}\|\}\)that lists the elements ofFkF\_\{k\}in an arbitrary order\. In this setting, the object of interest is which nodes are inFkF\_\{k\}, which can be fully encoded with multiplexing𝗆𝗎𝗑​\(𝐫k\)=1\|Fk\|​∑j=1\|Fk\|𝗈𝗇𝖾𝗁𝗈𝗍​\(rkj\)\\mathsf\{mux\}\(\\mathbf\{r\}\_\{k\}\)=\\frac\{1\}\{\|F\_\{k\}\|\}\\sum\_\{j=1\}^\{\|F\_\{k\}\|\}\\mathsf\{onehot\}\(r\_\{k\}^\{j\}\)as a uniform distribution overFkF\_\{k\}\. We now prove that this target is expressive enough to carry and expand an entire frontierFkF\_\{k\}together with the discovered setUkU\_\{k\}, thus implementing BFS\.

###### Proposition 10\(Parallel BFS with multiplexing\)\.

There exists a sequence of continuous tokens\(𝐱0,…,𝐱H\)\(\{\\bf x\}\_\{0\},\\ldots,\{\\bf x\}\_\{H\}\)such that, for everyk≤Hk\\leq H:

1. \(i\)𝐱k\{\\bf x\}\_\{k\}is a deterministic function of𝐱k−1\{\\bf x\}\_\{k\-1\}andGG,
2. \(ii\)FkF\_\{k\}andUkU\_\{k\}can be recovered from𝐱k\{\\bf x\}\_\{k\}, and so reachability𝟏​\(t∈UH\)\\boldsymbol\{1\}\(t\\in U\_\{H\}\)can be recovered from𝐱H\{\\bf x\}\_\{H\},
3. \(iii\)wheneverFk≠∅F\_\{k\}\\neq\\varnothing,𝗆𝗎𝗑​\(𝐫k\)\\mathsf\{mux\}\(\\mathbf\{r\}\_\{k\}\)can be recovered from𝐱k\{\\bf x\}\_\{k\}, up to arbitrary precision with softmax\.

The result implies that parallel search can naturally emerge from serial supervision via multiplexing\.

## 5Experiments

We evaluateMUXon mathematical reasoning \([Section˜5\.1](https://arxiv.org/html/2607.18264#S5.SS1)\), verify its parallel search capabilities \([Section˜5\.2](https://arxiv.org/html/2607.18264#S5.SS2)\), and analyze the role of key design choices \([Section˜5\.3](https://arxiv.org/html/2607.18264#S5.SS3)\)\. Interpretability and attention analysis can be found in[Sections˜8\.2](https://arxiv.org/html/2607.18264#S8.SS2)and[8\.3](https://arxiv.org/html/2607.18264#S8.SS3), and training cost analysis can be found in[Section˜8\.4](https://arxiv.org/html/2607.18264#S8.SS4)\.

### 5\.1Mathematical reasoning

Table 1:Mathematical reasoning test accuracies \(%\)\.†and‡are fromShenet al\.\([2025](https://arxiv.org/html/2607.18264#bib.bib13)\)andKuzinaet al\.\([2026](https://arxiv.org/html/2607.18264#bib.bib16)\), respectively\. We underlineMUXwhen it outperforms SFT\-CoT\.MUXreports±\\pm1 std\. over 3 seeds\. We did not conduct iCoT/Coconut OOD tests on NL due to their low ID scores\.MethodGSM8K\-AUGGSM8K\-AUG\-NLIDSVAMPGSM\-HardMultiArithIDSVAMPGSM\-HardMultiArithGPT\-2SFT\-CoT44\.1†41\.8†9\.8†90\.7†34\.236\.97\.188\.7No\-CoT†19\.116\.44\.341\.119\.116\.44\.341\.1*Latent reasoning*iCoT30\.1†29\.4†5\.7†55\.5†3\.2–––Coconut34\.1†36\.4†7\.9†82\.2†24\.9–––CODI43\.742\.99\.992\.834\.130\.86\.858\.9SIM\-CoT42\.642\.69\.492\.830\.927\.56\.553\.9MUX48\.1±\\pm0\.345\.0±\\pm0\.710\.6±\\pm0\.593\.0±\\pm0\.837\.4±\\pm0\.236\.7±\\pm0\.78\.9±\\pm0\.472\.4±\\pm1\.6LLaMA 3\.2 1B\-InstructSFT\-CoT61\.6†66\.7†15\.6†99\.3†53\.262\.913\.398\.5No\-CoT†30\.944\.17\.170\.930\.944\.17\.170\.9*Latent reasoning*iCoT19\.0†40\.9†4\.4†39\.0†15\.215\.2–––Coconut45\.3†48\.8†9\.9†90\.1†24\.2–––CODI55\.661\.112\.896\.147\.955\.311\.396\.7SIM\-CoT56\.161\.512\.796\.228\.443\.06\.659\.4MUX56\.7±\\pm0\.563\.6±\\pm1\.013\.0±\\pm0\.298\.5±\\pm0\.950\.3±\\pm0\.357\.5±\\pm0\.611\.6±\\pm0\.296\.9±\\pm0\.6*Latent reasoning via Jacobi iterations*PCCoT53\.557\.612\.997\.250\.154\.612\.296\.8KaVa‡56\.558\.912\.7–55\.758\.612\.8–MUX58\.0±\\pm0\.561\.8±\\pm0\.512\.9±\\pm0\.498\.7±\\pm0\.757\.2±\\pm0\.660\.6±\\pm2\.013\.4±\\pm0\.599\.2±\\pm0\.3

Table 2:Scaling to larger backbones on GSM8K\-AUG \(%\)\.⋄results fromWeiet al\.\([2026](https://arxiv.org/html/2607.18264#bib.bib15)\)\. We underlineMUXwhen it outperforms SFT\-CoT\. Single runs due to resource limits\.MethodLLaMA 3\.2 3BLLaMA 3\.1 8BIDSVAMPGSM\-HardMultiArithIDSVAMPGSM\-HardMultiArithSFT\-CoT⋄71\.571\.017\.098\.371\.773\.116\.598\.3No\-CoT⋄38\.352\.99\.588\.739\.555\.39\.888\.0CODI⋄60\.873\.314\.398\.761\.178\.115\.599\.5SIM\-CoT62\.374\.914\.698\.864\.179\.416\.3100\.0MUX65\.077\.115\.2100\.068\.180\.117\.1100\.0#### Setup\.

We follow the protocol of prior work and, for training, use two reasoning\-augmented mathematical corpora built upon GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.18264#bib.bib32)\)\. GSM8K\-AUG\(Shenet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib13)\)includes structured reasoning from GPT\-4\(Achiamet al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib66)\), whereas GSM8K\-AUG\-NL\(Denget al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib7)\)includes informal linguistic reasoning\. We use four test sets: GSM8K test split which is in\-domain, and out\-of\-domain arithmetic datasets SVAMP\(Patelet al\.,[2021](https://arxiv.org/html/2607.18264#bib.bib33)\), GSM\-Hard\(Gaoet al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib34)\), and MultiArith\(Roy and Roth,[2015](https://arxiv.org/html/2607.18264#bib.bib35)\)to test for transferability under distribution shift\. We mainly use GPT\-2\(Radfordet al\.,[2019](https://arxiv.org/html/2607.18264#bib.bib36)\)and LLaMA 3\.2 1B\-Instruct\(Meta,[2024](https://arxiv.org/html/2607.18264#bib.bib37)\)as backbone language models forMUXand baselines, and post\-train them via LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.18264#bib.bib73)\)\. To assess scalability, we also test larger backbones LLaMA 3\.2 3B and 3\.1 8B on GSM8K\-AUG following the protocol ofWeiet al\.\([2026](https://arxiv.org/html/2607.18264#bib.bib15)\); we were unable to train them on GSM8K\-AUG\-NL due to resource limits, as reasoning traces therein are considerably longer\. ForMUXand baselines, we mainly follow the setup ofShenet al\.\([2025](https://arxiv.org/html/2607.18264#bib.bib13)\), generating six latent tokens sequentially\. For improved scalability, we also experiment with the setup ofWuet al\.\([2025](https://arxiv.org/html/2607.18264#bib.bib14)\)where 24 latent tokens are generated in parallel via three Jacobi iterations\(Ortega and Rheinboldt,[2000](https://arxiv.org/html/2607.18264#bib.bib78)\), using LLaMA 3\.2 1B\-Instruct as backbone\. CommonsenseQA\(Talmoret al\.,[2019](https://arxiv.org/html/2607.18264#bib.bib80)\)and StrategyQA\(Gevaet al\.,[2021](https://arxiv.org/html/2607.18264#bib.bib81)\), which are non\-mathematical, have been tested in prior work\(Shenet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib13); Wuet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib14); Weiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib15)\), but are known to produce high\-variance, unreliable results for latent reasoning methods\(Shenet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib13); Wuet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib14)\)\. We therefore omit them\.

We compare against non\-reasoning, discrete\-reasoning, and latent\-reasoning baselines\. SFT\-CoT is supervised on discrete reasoning; No\-CoT predicts only the answer; iCoT\(Denget al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib7)\)internalizes discrete reasoning into a forward pass; Coconut\(Haoet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib11)\)and CODI\(Shenet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib13)\)rely on trajectory\-level losses for latent reasoning; SIM\-CoT\(Weiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib15)\)adds local distillation via an autoregressive decoder\. In the parallel decoding setting, we test latent methods PCCoT\(Wuet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib14)\)and KaVa\(Kuzinaet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib16)\)which are developed in the setting\.

#### Results\.

[Tables˜1](https://arxiv.org/html/2607.18264#S5.T1)and[2](https://arxiv.org/html/2607.18264#S5.T2)show the results\.MUXachieves the best latent reasoning performance in all 32 settings, surpassing both global \(iCoT, Coconut, CODI, PCCoT\) and local \(SIM\-CoT, KaVa\) distillation methods for latent reasoning often by a large margin\. Strikingly,MUXeven outperforms discrete\-reasoning SFT\-CoT in 15 cases spanning all model scales and both in\-domain and out\-of\-domain evaluations\. This result is surprising since it shows thatMUXis able to outperform the target of distillation, in a computationally efficient manner since generating six latent reasoning tokens corresponds to roughly2\.4×2\.4\\timesand5\.9×5\.9\\timesfewer reasoning tokens than SFT\-CoT on GSM8K\-AUG and GSM8K\-AUG\-NL, respectively\. We conjecture that multiplexing for local distillation regularizes the language models to exhibit good generalization behaviors, while acquiring efficiency via compact superposed reasoning\. Overall, the results suggest thatMUXis a simple method that learns strong, generalizable, and efficient latent reasoning that scales with language model sizes\. We further disentangle the effect of local supervision from that of global distillation in[Section˜8\.1](https://arxiv.org/html/2607.18264#S8.SS1), where the multiplexed target withγ=0\\gamma\{=\}0consistently surpasses SIM\-CoT under the same regime\.

### 5\.2Parallel search

#### Setup\.

We evaluateMUXon tasks that require search, aiming to verify our theoretical results in[Section˜4\.3](https://arxiv.org/html/2607.18264#S4.SS3)\. We consider two benchmarks, each naturally cast as a depth\-HHreachability problem on a finite directed graph so that the BFS frontier and discovered set are well defined\. We set the number of latent tokens equal to graph depth\. In this setting, a sequential strategy tracking a single hypothesis per token cannot explore all states within this budget\. Therefore, accuracy gains reflect the model’s ability to represent and update multiple candidates in parallel\. Further details are in[Section˜10](https://arxiv.org/html/2607.18264#S10)\.

Table 3:Search accuracies \(%\)\.MethodMNNSGame24No\-CoT68\.474\.4±2\.174\.4\{\\scriptstyle\\pm 2\.1\}SFT\-CoT84\.6±2\.184\.6\{\\scriptstyle\\pm 2\.1\}84\.3±1\.584\.3\{\\scriptstyle\\pm 1\.5\}Coconut92\.8±0\.692\.8\{\\scriptstyle\\pm 0\.6\}78\.6±2\.078\.6\{\\scriptstyle\\pm 2\.0\}CoT2298\.9±0\.398\.9\{\\scriptstyle\\pm 0\.3\}85\.0±1\.585\.0\{\\scriptstyle\\pm 1\.5\}MUX99\.6±0\.3\\textbf\{99\.6\}\{\\scriptstyle\\pm 0\.3\}88\.7±1\.1\\textbf\{88\.7\}\{\\scriptstyle\\pm 1\.1\}
#### MNNS\.

The minimum nonnegative sum \(MNNS\) task\(Gozetenet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib18)\)asks, given a set of integersa1,…,aHa\_\{1\},\\ldots,a\_\{H\}, for the smallest nonnegative value ofσ1​a1\+…\+σH​aH\\sigma\_\{1\}a\_\{1\}\+\.\.\.\+\\sigma\_\{H\}a\_\{H\}over signsσk=±1\\sigma\_\{k\}=\\pm 1\. This can be viewed as a search problem over a directed graphG=\(𝒩,E\)G=\(\\mathcal\{N\},E\)whose node set is𝒩=\{\(k,z\):k∈\{0,…,H\},z​a reachable partial sum\}\\mathcal\{N\}=\\\{\(k,z\):k\\in\\\{0,\\ldots,H\\\},\\;z\\text\{ a reachable partial sum\}\\\}, with edges\(\(k,z\),\(k\+1,z±ak\+1\)\)∈E\(\(k,z\),\(k\{\+\}1,z\\pm a\_\{k\+1\}\)\)\\in E\. The source iss=\(0,0\)s=\(0,0\)and the answer is the minimum nonnegativezzsuch that\(H,z\)∈UH\(H,z\)\\in U\_\{H\}\. At each depthkk, the frontierFkF\_\{k\}contains all partial\-sum states discovered for the first time\.

#### Game of 24\.

We introduce a new arithmetic search benchmark based on the Game of 24\(Yaoet al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib6)\)\. GivenCCcards drawn from\{1,…,D\}\\\{1,\\ldots,D\\\}and an operator set𝒪⊆\{\+,−,×\}\\mathcal\{O\}\\subseteq\\\{\+,\-,\\times\\\}, the task is to determine whether the value 24 is reachable by folding the cards left to right, accumulating with an operator from𝒪\\mathcal\{O\}\. This defines a layered directed graphG=\(𝒩,E\)G=\(\\mathcal\{N\},E\)whose nodes at depthkkare all reachable values after accumulating thekk\-th card, and edges correspond to the available operations\. The task is binary reachability:y=𝟏​\(24∈UC−1\)y=\\boldsymbol\{1\}\(24\\in U\_\{C\-1\}\)\. We useC=5C=5cards, digits\{1,…,5\}\\\{1,\\ldots,5\\\}, and𝒪=\{\+,−,×\}\\mathcal\{O\}=\\\{\+,\-,\\times\\\}\.

#### Results\.

[Table˜3](https://arxiv.org/html/2607.18264#S5.T3)shows the results, averaged over 3 seeds\.MUXachieves the best search performance in both tasks, directly verifying our claims in[Section˜4\.3](https://arxiv.org/html/2607.18264#S4.SS3)that multiplexed supervision can give rise to latent reasoning that performs parallel search\. This supports thatMUXis capable of exploring multiple hypotheses in superposition, a core advantage of latent reasoning methods\.

### 5\.3Ablation studies

#### Contribution of local loss\.

MUXlearns from local and global distillation lossesβ​ℒlocal\+γ​ℒglobal\\beta\\,\\mathcal\{L\}\_\{\\mathrm\{local\}\}\+\\gamma\\,\\mathcal\{L\}\_\{\\mathrm\{global\}\}withβ,γ≥0\\beta,\\gamma\\geq 0\([Section˜3\.2](https://arxiv.org/html/2607.18264#S3.SS2)\)\. To isolate the role of local distillation via multiplexing, we setγ=0\\gamma=0, and compare against SIM\-CoT, which also uses lossβ​ℒlocal′\+γ​ℒglobal\\beta\\,\\mathcal\{L\}\_\{\\mathrm\{local\}\}^\{\\prime\}\+\\gamma\\mathcal\{L\}\_\{\\mathrm\{global\}\}whereℒlocal′\\mathcal\{L\}\_\{\\mathrm\{local\}\}^\{\\prime\}learns local distillation in autoregressively decoded text space, by setting itsγ=0\\gamma=0\. This allows comparing the quality of local loss in a controlled manner\.[Table˜4](https://arxiv.org/html/2607.18264#S5.T4)shows thatMUXindeed has a higher\-quality local loss, although being simpler and not requiring an auxiliary autoregressive decoder\. A more comprehensive comparison can be found in[Section˜8\.1](https://arxiv.org/html/2607.18264#S8.SS1)\.

As a complementary analysis, we takeMUXand vary how many latent tokens receive local loss\.[Figure˜3](https://arxiv.org/html/2607.18264#S5.F3)shows accuracy rises from 32\.7% with no distilled tokens to 48\.2% with six\. The model without distillation still performs latent reasoning, but no tokens are matched with discrete reasoning\. This result shows that the gains ofMUXcome from local distillation, not merely from effective depth of latent reasoning\.

#### Chunking strategy\.

WhenM\>KM\>K, theMMdiscrete spans must be merged intoKKspans before being distilled intoKKlatent tokens \([Section˜3\.1](https://arxiv.org/html/2607.18264#S3.SS1)\)\. We compare randomized chunking, fixed deterministic chunking, and no chunking \(truncation\)\.[Table˜4](https://arxiv.org/html/2607.18264#S5.T4)shows that randomized chunking performs best\. We attribute this to its resampled boundaries, which act as a form of structured data augmentation\.

Table 4:Ablations \(LLaMA 1B; GSM8K\-AUG\)\.MethodIDSVAMPGSM\-HardMultiArithLocal distillation only \(γ=0\\gamma=0\)SIM\-CoT31\.644\.07\.569\.5MUX48\.751\.210\.698\.9Chunking strategyNone54\.361\.312\.596\.5Deterministic55\.760\.612\.697\.3Random56\.661\.313\.098\.3Positional weightingUniform54\.262\.412\.896\.1Rotary55\.064\.412\.498\.3Sinusoidal55\.261\.112\.699\.4Geometric57\.263\.112\.998\.3![Refer to caption](https://arxiv.org/html/2607.18264v1/x3.png)Figure 3:Effect of local distillation loss\.![Refer to caption](https://arxiv.org/html/2607.18264v1/x4.png)Figure 4:Probe accuracy
#### Positional weighting\.

We test the role of positional weighting, which theoretically affects multiplexing losslessness \([Sections˜3\.2](https://arxiv.org/html/2607.18264#S3.SS2)and[4\.1](https://arxiv.org/html/2607.18264#S4.SS1)\)\. We compare our weightings, proven to be lossless, against \(lossy\) uniform weighting\.[Table˜4](https://arxiv.org/html/2607.18264#S5.T4)shows that while lossless weighting is better, the gap is modest, which could be attributed to the fact that many reasoning spans in GSM8K\-AUG are highly structured, e\.g\.,<<60/2 = 30\>\>, so ordering of subwords can often be inferred even from bags of subwords\. Nevertheless, the overall gains suggest that losslessness is beneficial in practice\. We further train a small MLP to demultiplex spans𝐫i\{\\bf r\}\_\{i\}from𝗆𝗎𝗑​\(𝐫i\)\\mathsf\{mux\}\(\{\\bf r\}\_\{i\}\)\.[Figure˜4](https://arxiv.org/html/2607.18264#S5.F4)shows that uniform weighting allows nontrivial accuracy, confirming that ordering of subwords is partially recoverable from occurrences\. Lossless weightings are still better, agreeing with latent reasoning performances\.

## 6Conclusion

We introducedMUX, a simple local distillation method for continuous latent reasoning based on position\-weighted superposition in vocabulary space\. Each latent token is trained to represent an aligned span of discrete reasoning via a multiplexed target that is easy to compute, theoretically grounded, and empirically effective\. We showed that suitable positional weightings support exact span recovery, and that multiplexed targets can express parallel search dynamics\. Across multiple models and benchmarks,MUXconsistently improved upon strong baselines\. These results suggest that simple, interpretable local targets can make latent reasoning stronger and easier to train\.

## Acknowledgments

The authors would like to thank Xingyue Huang, Louis Tichelman, and Angelo Gnazzo for valuable discussions\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1)\.
- R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.\(2023\)Gemini: a family of highly capable multimodal models\.arXiv preprint arXiv:2312\.11805\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p1.1)\.
- Compressed chain of thought: efficient reasoning through dense representations\.arXiv preprint arXiv:2412\.13171\.Cited by:[§7\.2](https://arxiv.org/html/2607.18264#S7.SS2.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1)\.
- Y\. Cui, Z\. Dai, B\. He, Z\. Shi, H\. Liu, R\. Sun, Z\. Liu, Y\. Xing, J\. Tang, and B\. Dumoulin \(2026\)How do latent reasoning methods perform under weak and strong supervision?\.arXiv preprint arXiv:2602\.22441\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p2.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p2.1),[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px1.p1.1),[§7\.5](https://arxiv.org/html/2607.18264#S7.SS5.p3.1),[§7\.5](https://arxiv.org/html/2607.18264#S7.SS5.p4.2)\.
- B\. Cywiński, E\. Ryd, S\. Rajamanoharan, and N\. Nanda \(2025\)Towards eliciting latent knowledge from llms with mechanistic interpretability\.arXiv preprint arXiv:2505\.14352\.Cited by:[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p2.1)\.
- J\. Deng, L\. Pang, Z\. Wei, S\. Xu, Z\. Duan, K\. Xu, Y\. Song, H\. Shen, and X\. Cheng \(2025\)Latent reasoning in llms as a vocabulary\-space superposition\.arXiv preprint arXiv:2510\.15522\.Cited by:[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Deng, Y\. Choi, and S\. Shieber \(2024\)From explicit cot to implicit cot: learning to internalize cot step by step\.arXiv preprint arXiv:2405\.14838\.Cited by:[§7\.2](https://arxiv.org/html/2607.18264#S7.SS2.p1.1)\.
- Y\. Deng, K\. Prasad, R\. Fernandez, P\. Smolensky, V\. Chaudhary, and S\. Shieber \(2023\)Implicit chain of thought reasoning via knowledge distillation\.arXiv preprint arXiv:2311\.01460\.Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p2.1),[§7\.2](https://arxiv.org/html/2607.18264#S7.SS2.p1.1)\.
- C\. Dilgren and S\. Wiegreffe \(2026\)Are latent reasoning models easily interpretable?\.arXiv preprint arXiv:2604\.04902\.Cited by:[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p2.1),[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px1.p1.1),[§8\.2](https://arxiv.org/html/2607.18264#S8.SS2.p1.1),[§8\.2](https://arxiv.org/html/2607.18264#S8.SS2.p3.4)\.
- J\. Fan, Y\. Gu, M\. Hachimori, and Y\. Miao \(2020\)Signature codes for weighted binary adder channel and multimedia fingerprinting\.IEEE Transactions on Information Theory67\(1\),pp\. 200–216\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p3.1)\.
- L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)Pal: program\-aided language models\.InInternational conference on machine learning,pp\. 10764–10799\.Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1),[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p1.1)\.
- M\. Geva, D\. Khashabi, E\. Segal, T\. Khot, D\. Roth, and J\. Berant \(2021\)Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies\.Transactions of the Association for Computational Linguistics9,pp\. 346–361\.Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1)\.
- D\. Goldberg \(1991\)What every computer scientist should know about floating\-point arithmetic\.ACM computing surveys \(CSUR\)23\(1\),pp\. 5–48\.Cited by:[§4\.1](https://arxiv.org/html/2607.18264#S4.SS1.SSS0.Px1.p1.9)\.
- S\. Goyal, Z\. Ji, A\. S\. Rawat, A\. K\. Menon, S\. Kumar, and V\. Nagarajan \(2024\)Think before you speak: training language models with pause tokens\.InThe Twelfth International Conference on Learning Representations,Cited by:[§7\.2](https://arxiv.org/html/2607.18264#S7.SS2.p1.1)\.
- H\. A\. Gozeten, M\. E\. Ildiz, X\. Zhang, H\. Harutyunyan, A\. S\. Rawat, and S\. Oymak \(2026\)Continuous chain of thought enables parallel exploration and reasoning\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p2.1),[§10\.1](https://arxiv.org/html/2607.18264#S10.SS1.p1.5),[§12](https://arxiv.org/html/2607.18264#S12.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p2.1),[§5\.2](https://arxiv.org/html/2607.18264#S5.SS2.SSS0.Px2.p1.11),[§7\.4](https://arxiv.org/html/2607.18264#S7.SS4.p1.4)\.
- S\. Hao, S\. Sukhbaatar, D\. Su, X\. Li, Z\. Hu, J\. E\. Weston, and Y\. Tian \(2025\)Training large language models to reason in a continuous latent space\.InSecond Conference on Language Modeling,Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p2.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p2.1),[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px1.p1.1),[§8\.2](https://arxiv.org/html/2607.18264#S8.SS2.p3.4)\.
- D\. Hendrycks and K\. Gimpel \(2016\)Gaussian error linear units \(gelus\)\.arXiv preprint arXiv:1606\.08415\.Cited by:[§11\.2](https://arxiv.org/html/2607.18264#S11.SS2.p2.7)\.
- D\. Herel and T\. Mikolov \(2024\)Thinking tokens for language modeling\.arXiv preprint arXiv:2405\.08644\.Cited by:[§7\.2](https://arxiv.org/html/2607.18264#S7.SS2.p1.1)\.
- N\. J\. Higham \(2002\)Accuracy and stability of numerical algorithms\.SIAM\.Cited by:[§4\.1](https://arxiv.org/html/2607.18264#S4.SS1.SSS0.Px1.p1.9)\.
- E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1)\.
- W\. Huang, Y\. Xiong, X\. Ye, Z\. Deng, H\. Chen, Z\. Lin, and G\. Ding \(2025\)Fast quiet\-STaR: thinking without thought tokens\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 18771–18781\.Cited by:[§7\.2](https://arxiv.org/html/2607.18264#S7.SS2.p1.1)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p1.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§8\.4](https://arxiv.org/html/2607.18264#S8.SS4.p1.9)\.
- R\. M\. Karp \(2009\)Reducibility among combinatorial problems\.In50 Years of Integer Programming 1958\-2008: from the Early Years to the State\-of\-the\-Art,pp\. 219–241\.Cited by:[§10\.1](https://arxiv.org/html/2607.18264#S10.SS1.p1.5)\.
- T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa \(2022\)Large language models are zero\-shot reasoners\.Advances in neural information processing systems35,pp\. 22199–22213\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p1.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px1.p1.1),[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p1.1)\.
- A\. Kuzina, M\. Pióro, and B\. E\. Bejnordi \(2026\)KaVa: latent reasoning via compressed KV\-cache distillation\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p2.1),[§12](https://arxiv.org/html/2607.18264#S12.SS0.SSS0.Px1.p1.1),[§12](https://arxiv.org/html/2607.18264#S12.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2607.18264#S3.SS1.SSS0.Px3.p2.14),[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p2.1),[Table 1](https://arxiv.org/html/2607.18264#S5.T1),[Table 1](https://arxiv.org/html/2607.18264#S5.T1.6.3),[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px2.p1.1),[§7\.5](https://arxiv.org/html/2607.18264#S7.SS5.p4.2),[§8\.2](https://arxiv.org/html/2607.18264#S8.SS2.p1.1),[§8\.4](https://arxiv.org/html/2607.18264#S8.SS4.p2.10)\.
- J\. Li, R\. Li, Y\. Zhou, B\. Ma, and J\. Z\. Pan \(2026a\)Chain of thought compression: a theoritical analysis\.arXiv preprint arXiv:2601\.21576\.Cited by:[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px1.p1.1),[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p2.1)\.
- Z\. Li, J\. Zhong, Z\. Zheng, X\. Wen, Z\. Xu, Y\. Cheng, F\. Zhang, and Q\. Xu \(2026b\)Making slow thinking faster: compressing LLM chain\-of\-thought via step entropy\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p1.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px1.p1.1),[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p2.1)\.
- J\. Liang and L\. Pan \(2026\)Do latent\-cot models think step\-by\-step? a mechanistic study on sequential reasoning tasks\.arXiv preprint arXiv:2602\.00449\.Cited by:[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p2.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[§10\.1](https://arxiv.org/html/2607.18264#S10.SS1.SSS0.Px3.p1.3)\.
- Meta \(2024\)Llama 3\.2: Open\-Source AI Models by Meta\.Note:[https://www\.llama\.com/docs/model\-cards\-and\-prompt\-formats/llama3\_2/](https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_2/)Accessed: 2026\-05\-13Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1)\.
- M\. Nye, A\. J\. Andreassen, G\. Gur\-Ari, H\. Michalewski, J\. Austin, D\. Bieber, D\. Dohan, A\. Lewkowycz, M\. Bosma, D\. Luan,et al\.\(2021\)Show your work: scratchpads for intermediate computation with language models\.arXiv preprint arXiv:2112\.00114\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p1.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px1.p1.1),[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p1.1)\.
- J\. M\. Ortega and W\. C\. Rheinboldt \(2000\)Iterative solution of nonlinear equations in several variables\.SIAM\.Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1)\.
- A\. Patel, S\. Bhattamishra, and N\. Goyal \(2021\)Are nlp models really able to solve simple math word problems?\.InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies,pp\. 2080–2094\.Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1)\.
- S\. Roy and D\. Roth \(2015\)Solving general arithmetic word problems\.InProceedings of the 2015 conference on empirical methods in natural language processing,pp\. 1743–1752\.Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1)\.
- Z\. Shen, H\. Yan, L\. Zhang, Z\. Hu, Y\. Du, and Y\. He \(2025\)Codi: compressing chain\-of\-thought into continuous space via self\-distillation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 677–693\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p2.1),[§11\.1](https://arxiv.org/html/2607.18264#S11.SS1.p1.1),[§12](https://arxiv.org/html/2607.18264#S12.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2607.18264#S3.SS2.SSS0.Px3.p1.15),[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p2.1),[Table 1](https://arxiv.org/html/2607.18264#S5.T1),[Table 1](https://arxiv.org/html/2607.18264#S5.T1.6.3),[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px1.p1.1),[§7\.5](https://arxiv.org/html/2607.18264#S7.SS5.p3.1),[§8\.2](https://arxiv.org/html/2607.18264#S8.SS2.p1.1)\.
- D\. Su, H\. Zhu, Y\. Xu, J\. Jiao, Y\. Tian, and Q\. Zheng \(2025\)Token assorted: mixing latent and text tokens for improved language model reasoning\.InForty\-second International Conference on Machine Learning,Cited by:[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px3.p1.1)\.
- J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu \(2024\)Roformer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[item \(3\)](https://arxiv.org/html/2607.18264#S3.I1.i3.p1.3)\.
- A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)Commonsenseqa: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 4149–4158\.Cited by:[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1)\.
- Y\. Tang, L\. Dong, Y\. Hao, Q\. Dong, F\. Wei, and J\. Gu \(2026\)Multiplex thinking: reasoning via token\-wise branch\-and\-merge\.arXiv preprint arXiv:2601\.08808\.Cited by:[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p1.1),[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px4.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p1.1)\.
- M\. Tutek, F\. H\. Chaleshtori, A\. Marasović, and Y\. Belinkov \(2025\)Measuring chain of thought faithfulness by unlearning reasoning steps\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 9946–9971\.Cited by:[§9\.3](https://arxiv.org/html/2607.18264#S9.SS3.p6.12)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InThe Eleventh International Conference on Learning Representations,Cited by:[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p1.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px1.p1.1),[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p1.1)\.
- X\. Wei, X\. Liu, Y\. Zang, X\. Dong, Y\. Cao, J\. Wang, X\. Qiu, and D\. Lin \(2026\)SIM\-cot: supervised implicit chain\-of\-thought\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p2.1),[§12](https://arxiv.org/html/2607.18264#S12.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2607.18264#S3.SS1.SSS0.Px3.p2.14),[§4\.2](https://arxiv.org/html/2607.18264#S4.SS2.p1.1),[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p2.1),[Table 2](https://arxiv.org/html/2607.18264#S5.T2),[Table 2](https://arxiv.org/html/2607.18264#S5.T2.2.1),[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px2.p1.1),[§7\.5](https://arxiv.org/html/2607.18264#S7.SS5.p4.2),[§8\.2](https://arxiv.org/html/2607.18264#S8.SS2.p1.1),[§8\.4](https://arxiv.org/html/2607.18264#S8.SS4.p2.10)\.
- H\. Wu, Z\. Teng, and K\. Tu \(2025\)Parallel continuous chain\-of\-thought with jacobi iteration\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 914–926\.Cited by:[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p2.1),[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.18264#S5.SS1.SSS0.Px1.p2.1),[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px3.p1.1)\.
- J\. Wu, J\. Lu, Z\. Ren, G\. Hu, Z\. Wu, D\. Dai, and H\. Wu \(2026\)LLMs are single\-threaded reasoners: demystifying the working mechanism of soft thinking\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px4.p1.1)\.
- H\. Xia, C\. T\. Leong, W\. Wang, Y\. Li, and W\. Li \(2025\)Tokenskip: controllable chain\-of\-thought compression in llms\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 3351–3363\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p1.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px1.p1.1),[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p2.1)\.
- J\. Xu, M\. Zhou, W\. Liu, H\. Liu, S\. Han, and D\. Zhang \(2025a\)TwT: thinking without tokens by habitual reasoning distillation with multi\-teachers’ guidance\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 16475–16489\.Cited by:[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px3.p1.1)\.
- Y\. Xu, X\. Guo, Z\. Zeng, and C\. Miao \(2025b\)Softcot: soft chain\-of\-thought for efficient reasoning with llms\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 23336–23351\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p2.1),[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px3.p1.1)\.
- Y\. Xu, X\. Guo, Z\. Zeng, and C\. Miao \(2025c\)Softcot\+\+: test\-time scaling with soft chain\-of\-thought reasoning\.arXiv preprint arXiv:2505\.11484\.Cited by:[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px3.p1.1)\.
- S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. Griffiths, Y\. Cao, and K\. Narasimhan \(2023\)Tree of thoughts: deliberate problem solving with large language models\.Advances in neural information processing systems36,pp\. 11809–11822\.Cited by:[§10\.2](https://arxiv.org/html/2607.18264#S10.SS2.p1.1),[§5\.2](https://arxiv.org/html/2607.18264#S5.SS2.SSS0.Px3.p1.11),[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p1.1)\.
- E\. Zelikman, G\. R\. Harik, Y\. Shao, V\. Jayasiri, N\. Haber, and N\. Goodman \(2024\)Quiet\-STar: language models can teach themselves to think before speaking\.InFirst Conference on Language Modeling,Cited by:[§7\.2](https://arxiv.org/html/2607.18264#S7.SS2.p1.1)\.
- E\. Zelikman, Y\. Wu, J\. Mu, and N\. Goodman \(2022\)Star: bootstrapping reasoning with reasoning\.Advances in Neural Information Processing Systems35,pp\. 15476–15488\.Cited by:[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p1.1)\.
- J\. Zhang, Y\. Zhu, M\. Sun, Y\. Luo, S\. Qiao, L\. Du, D\. Zheng, H\. Chen, and N\. Zhang \(2025a\)Lightthinker: thinking step\-by\-step compression\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 13318–13339\.Cited by:[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px1.p1.1),[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p2.1)\.
- J\. Zhang, Q\. Lin, S\. Rajmohan, and D\. Zhang \(2025b\)From reasoning to answer: empirical, attention\-based and mechanistic insights into distilled deepseek r1 models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 3985–4002\.Cited by:[§9\.3](https://arxiv.org/html/2607.18264#S9.SS3.p6.12)\.
- Y\. Zhang, B\. Tang, T\. Ju, S\. Duan, and G\. Liu \(2025c\)Do latent tokens think? a causal and adversarial analysis of chain\-of\-continuous\-thought\.arXiv preprint arXiv:2512\.21711\.Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p2.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p2.1),[§7\.5](https://arxiv.org/html/2607.18264#S7.SS5.p3.1)\.
- Z\. Zhang, X\. He, W\. Yan, A\. Shen, C\. Zhao, and X\. E\. Wang \(2026\)Soft thinking: unlocking the reasoning potential of LLMs in continuous concept space\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p1.1),[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px4.p1.1)\.
- Z\. Zheng, Y\. Gu, W\. Liu, Y\. W\. Teh, and W\. S\. Lee \(2025\)SofT\-grpo: surpassing discrete\-token llm reinforcement learning via gumbel\-reparameterized soft\-thinking policy optimization\.arXiv preprint arXiv:2511\.06411\.Cited by:[§7\.3](https://arxiv.org/html/2607.18264#S7.SS3.SSS0.Px4.p1.1)\.
- D\. Zhou, N\. Schärli, L\. Hou, J\. Wei, N\. Scales, X\. Wang, D\. Schuurmans, C\. Cui, O\. Bousquet, Q\. V\. Le, and E\. H\. Chi \(2023\)Least\-to\-most prompting enables complex reasoning in large language models\.InThe Eleventh International Conference on Learning Representations,Cited by:[§7\.1](https://arxiv.org/html/2607.18264#S7.SS1.p1.1)\.
- H\. Zhu, S\. Hao, Z\. Hu, J\. Jiao, S\. Russell, and Y\. Tian \(2026\)Reasoning by superposition: a theoretical perspective on chain of continuous thought\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.18264#S1.p2.1),[§2](https://arxiv.org/html/2607.18264#S2.SS0.SSS0.Px2.p2.1),[§7\.4](https://arxiv.org/html/2607.18264#S7.SS4.p1.4)\.

\\beginappendix

## 7Extended related work

We expand the related\-work discussion from the main text and then summarize the main distinctions in[Table˜5](https://arxiv.org/html/2607.18264#S7.T5)\.

### 7\.1Reasoning in language

Chain\-of\-thought \(CoT\) prompting\(Weiet al\.,[2022](https://arxiv.org/html/2607.18264#bib.bib2); Kojimaet al\.,[2022](https://arxiv.org/html/2607.18264#bib.bib60)\)showed that asking a language model to articulate intermediate reasoning steps dramatically improves performance on arithmetic, symbolic, and commonsense tasks\.Nyeet al\.\([2021](https://arxiv.org/html/2607.18264#bib.bib1)\)introduced scratchpads as a training\-time analogue, where intermediate tokens serve as an explicit computation buffer\. Subsequent methods refine how this buffer is generated, verified, or searched, including self\-consistency\(Wanget al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib4)\), STaR\(Zelikmanet al\.,[2022](https://arxiv.org/html/2607.18264#bib.bib3)\), least\-to\-most prompting\(Zhouet al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib5)\), Tree of Thoughts\(Yaoet al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib6)\), and PAL\(Gaoet al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib34)\)\.

At the same time, recent work has shown that such traces are often far more verbose than what the underlying computation requires\. TokenSkip\(Xiaet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib22)\), step\-entropy pruning\(Liet al\.,[2026b](https://arxiv.org/html/2607.18264#bib.bib23)\), LightThinker\(Zhanget al\.,[2025a](https://arxiv.org/html/2607.18264#bib.bib46)\), and ALiCoT\(Liet al\.,[2026a](https://arxiv.org/html/2607.18264#bib.bib47)\)all indicate that explicit CoTs contain redundant linguistic overhead\. These findings motivateMUX\. Our goal is not to compress a generated reasoning at inference time, but to use the redundancy of discrete traces to train a smaller number of continuous reasoning states\.

### 7\.2Implicit reasoning and internalization

A related line of work tries to keep the benefits of intermediate computation while removing the need to emit intermediate language at inference time\. iCoT\(Denget al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib7)\)and its stepwise extension\(Denget al\.,[2024](https://arxiv.org/html/2607.18264#bib.bib8)\)distill explicit reasoning into computations within a single forward pass\. Pause tokens\(Goyalet al\.,[2024](https://arxiv.org/html/2607.18264#bib.bib9)\), Quiet\-STaR\(Zelikmanet al\.,[2024](https://arxiv.org/html/2607.18264#bib.bib39)\), Fast Quiet\-STaR\(Huanget al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib40)\), and thinking tokens\(Herel and Mikolov,[2024](https://arxiv.org/html/2607.18264#bib.bib24)\)increase internal compute by inserting special positions that need not correspond to normal language\. Compressed CoT\(Cheng and Van Durme,[2024](https://arxiv.org/html/2607.18264#bib.bib10)\)similarly move toward denser reasoning representations\. These methods increase effective compute depth without proportionally increasing output length\.

### 7\.3Continuous latent reasoning

Continuous reasoning methods go further by operating in a latent vector space and feeding latent tokens back to the model\. We organize the literature by the type of supervision employed\.

#### Global supervision\.

Coconut\(Haoet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib11)\)is the foundational method, replacing discrete reasoning with latent recurrence, forming a “chain of continuous thought\.” Its training uses a curriculum that gradually transitions from discrete reasoning to fully latent reasoning\. Coconut demonstrated that continuous reasoning can support breadth\-first\-style exploration, but intermediate states are trained only from the final answer loss, which leaves the intermediate latent trajectory unsupervised\. CODI\(Shenet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib13)\)strengthened this with self\-distillation, aligning the continuous and discrete reasoning modes in terms of the hidden state used to predict the final answer\. Both methods supervise the reasoning process mainly through the final answer or trajectory endpoint\. In our terminology, they are*global*supervision methods that do not supervise what each latent token should represent\. Recent empirical analyses show that this lack of intermediate supervision typically leads to shortcut behavior, as globally supervised models can achieve high accuracy without meaningfully relying on the latent reasoning tokens\(Cuiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib74); Dilgren and Wiegreffe,[2026](https://arxiv.org/html/2607.18264#bib.bib75)\)\.

#### Local supervision via auxiliary components\.

MUXis closer to*local*supervision methods, which supervise each latent reasoning token with a choice of target\. SIM\-CoT\(Weiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib15)\)identifies a critical limitation of global supervision: as the number of latent tokens increases, they become homogeneous and training collapses\. To address this, SIM\-CoT uses an auxiliary autoregressive decoder during training that forces each latent token to encode its aligned discrete reasoning span, providing local supervision\. KaVa\(Kuzinaet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib16)\)takes a different approach by distilling the teacher’s compressed key\-value \(KV\) cache into the student model layer by layer\. The supervision target is the teacher’s cache dynamics, providing a rich but structurally complex signal\. Both show that local supervision is effective in mitigating the failure mode of global supervision methods, but each requires additional components, which are an auxiliary decoder or a KV compression module\.

#### Parallelization and efficiency\.

PCCoT\(Wuet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib14)\)improves the efficiency of continuous reasoning by parallelizing sequential predictions of latent tokens via Jacobi iterations, reducing inference latency while maintaining accuracy\. SoftCoT\(Xuet al\.,[2025b](https://arxiv.org/html/2607.18264#bib.bib12)\)generates soft reasoning tokens from a frozen model using a trained projection layer, and SoftCoT\+\+\(Xuet al\.,[2025c](https://arxiv.org/html/2607.18264#bib.bib41)\)extends this to test\-time compute scaling\. Token Assorted\(Suet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib25)\)mixes discrete and continuous tokens in a hybrid reasoning trace, allowing the model to choose when to reason in language and when to reason in latent space\. TWT\(Xuet al\.,[2025a](https://arxiv.org/html/2607.18264#bib.bib42)\)distills reasoning from multiple teacher models into habitual latent computation\.

#### Inference\-time continuous reasoning\.

Another related line of work considers reasoning in a continuous space only at*inference time*by modifying the decoding procedure of a pretrained language model\. Soft Thinking\(Zhanget al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib48)\)replaces discrete subword selection with probability\-weighted mixtures of vocabulary embeddings\. Subsequent work studies its limitations and variants\(Wuet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib51); Tanget al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib49); Zhenget al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib50)\)\. Multiplex Thinking\(Tanget al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib49)\)samples a set of subwords at each reasoning step and aggregates their embeddings into a single continuous*multiplex token*, maintaining vocabulary embedding priors while enabling on\-policy RL\. These methods are complementary to us\. They modify the decoding procedure of a pretrained model, whereasMUXis a training\-time method for latent reasoning distillation\. This separation lets us use vocabulary space for interpretable supervision without necessarily committing to it during inference time\.

### 7\.4Theoretical foundations of continuous reasoning

A growing theoretical literature formalizes the advantages of continuous over discrete reasoning\.Zhuet al\.\([2026](https://arxiv.org/html/2607.18264#bib.bib17)\)prove that a two\-layer transformer withdiameter​\(G\)\\mathrm\{diameter\}\(G\)steps of continuous reasoning can solve directed graph reachability on graphGG\. The key mechanism is*superposition*: each continuous thought vector encodes multiple search frontiers simultaneously, enabling parallel breadth\-first search\. In contrast, discrete reasoning requiresO​\(\|V​\(G\)\|2\)O\(\|V\(G\)\|^\{2\}\)steps with constant\-depth transformers\. CoT22\(Gozetenet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib18)\)provides complementary results for search problems, showing that supervision against latent token distributions induces parallel exploration\. Our work connects to this line in two ways\. We prove that our multiplexed targets are*lossless*under standard positional weightings, and that a latent recurrence over such targets can implement exact parallel breadth\-first exploration\.

### 7\.5Positioning ofMUX

Table 5:Qualitative comparison of reasoning methods\.✓= favorable,✗= unfavorable\.MethodSupervisionLosslessShortcut\-freeTrain eff\.Infer\. eff\.InterpretableSFT\-CoTDiscrete✓✓✓✗✓CODIGlobal✗✗✓✓✗SIM\-CoTLocal✓✓✗✓✓KaVaLocal✗✓✓✓✓MUXLocal✓✓✓✓✓[Table˜5](https://arxiv.org/html/2607.18264#S7.T5)summarizes the positioning ofMUXrelative to representative baselines along five desirable properties: whether the training signal preserves the full discrete reasoning trace \(*lossless*\), whether latent tokens avoid collapsing into uninformative placeholders \(*shortcut\-free*\), whether training and inference are efficient \(*train/infer\. eff\.*\), and whether intermediate latent states can be decoded into human\-readable content \(*interpretable*\)\.

SFT\-CoT directly supervises every discrete reasoning subword via cross\-entropy, so losslessness and shortcut avoidance hold by construction\. Training is efficient as it processes only the discrete reasoning sequence, but inference requires generating the full trace, which is2\.42\.4–5\.9×5\.9\\timesmore subwords than the compact budgets used by continuous methods \([Section˜5\.1](https://arxiv.org/html/2607.18264#S5.SS1)\)\. The output is natural language, making it inherently interpretable\.

CODI\(Shenet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib13)\)performs trajectory\-level global distillation, aligning the student’s hidden state to the teacher’s at the answer position\. Whether this preserves the full reasoning content depends on how much the teacher’s hidden state actually encodes about reasoning span\. Since there is no structural guarantee, we mark it as not lossless\. Supervision acts only at the trajectory endpoint, and recent analyses show that this leads to pervasive shortcut behavior\. Models trained with global losses can achieve high accuracy without meaningfully relying on intermediate latent tokens\(Zhanget al\.,[2025c](https://arxiv.org/html/2607.18264#bib.bib70); Cuiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib74)\)\. Training and inference are both efficient, but without per\-token supervision, individual latent tokens are hard to interpret \([Section˜8\.2](https://arxiv.org/html/2607.18264#S8.SS2)\)\.

SIM\-CoT\(Weiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib15)\)and KaVa\(Kuzinaet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib16)\)represent two flavors of local supervision\. SIM\-CoT attaches an auxiliary autoregressive decoder that reconstructs the full aligned reasoning span from each latent token, providing a lossless training signal\. KaVa instead distills compressed key\-value cache states from the teacher via an importance\-based eviction mechanism \(R\-KV\) that selectively discards KV pairs, making its supervision lossy\. Both methods mitigate shortcut behavior through step\-level local supervision\(Cuiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib74)\)\. KaVa’s auxiliary cost is negligible, whereas SIM\-CoT’s decoder adds1616–3232% training overhead \([Section˜8\.4](https://arxiv.org/html/2607.18264#S8.SS4)\)\. Both can produce interpretable latent tokens: SIM\-CoT through its decoder output, and KaVa through vocabulary projection of its distilled representations\.

MUXis the only method in this comparison that satisfies all five properties\. The multiplexed targets are provably lossless under suitable positional weightings \([Propositions˜3](https://arxiv.org/html/2607.18264#Thmtheorem3)and[5](https://arxiv.org/html/2607.18264#Thmtheorem5)\), and the non\-collapsing guarantee \([Proposition˜9](https://arxiv.org/html/2607.18264#Thmtheorem9)\) prevents shortcut behavior\. Training adds only a KL divergence over vocabulary distributions \(<<0\.01% of the base cost \([Section˜8\.4](https://arxiv.org/html/2607.18264#S8.SS4)\)\), and inference uses the same compact latent token budget as other continuous methods\. Each latent token can be read out through the pretrained unembedding layer, giving a vocabulary distribution that reflects the aligned reasoning content \([Section˜8\.2](https://arxiv.org/html/2607.18264#S8.SS2)\)\.

## 8Supplementary results

### 8\.1Contribution of local distillation

Table 6:Test accuracies \(%\) with local distillation only \(γ=0\\gamma\{=\}0\)\.MethodGSM8K\-AUGGSM8K\-AUG\-NLIDSVAMPGSM\-HardMultiArithIDSVAMPGSM\-HardMultiArithGPT\-2SIM\-CoT \(γ=0\\gamma\{=\}0\)29\.526\.56\.848\.921\.423\.44\.934\.4MUX\(γ=0\\gamma\{=\}0\)38\.634\.49\.275\.331\.928\.17\.048\.3LLaMA 3\.2 1B\-InstructSIM\-CoT \(γ=0\\gamma\{=\}0\)31\.644\.07\.569\.530\.144\.06\.762\.2MUX\(γ=0\\gamma\{=\}0\)48\.751\.210\.698\.938\.945\.49\.677\.0LLaMA 3\.2 3B\(GSM8K\-AUG\)LLaMA 3\.1 8B\(GSM8K\-AUG\)SIM\-CoT \(γ=0\\gamma\{=\}0\)49\.164\.612\.4100\.045\.669\.711\.995\.0MUX\(γ=0\\gamma\{=\}0\)51\.565\.412\.797\.250\.671\.313\.295\.6To isolate the contribution of the local supervision, we remove the global trajectory\-level distillation loss by settingγ=0\\gamma\{=\}0in bothMUXand SIM\-CoT\.[Table˜6](https://arxiv.org/html/2607.18264#S8.T6)reports the results across all model\-dataset combinations\.MUX\(γ=0\\gamma\{=\}0\) outperforms SIM\-CoT \(γ=0\\gamma\{=\}0\) in 23 of 24 settings\. These results imply that the multiplexed target is the source of performance gain ofMUX, as without any trajectory\-level supervision signal, multiplexed local supervision outperforms SIM\-CoT’s autoregressive decoder\-based local supervision\. Notably,MUXachieves this with a simpler architecture, since no auxiliary decoder is needed\.

### 8\.2Interpretability analysis

![Refer to caption](https://arxiv.org/html/2607.18264v1/x5.png)\(a\)Mathematical reasoning \(GSM8K\-AUG\)
![Refer to caption](https://arxiv.org/html/2607.18264v1/x6.png)\(b\)Parallel search \(MNNS\)

Figure 5:Top\-5 LM\-head decoded subwords per latent token\.Prior works interpret latent reasoning by projecting latent tokens back into vocabulary space\(Shenet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib13); Weiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib15); Kuzinaet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib16)\), arguing that interpretability itself signals quality of latent reasoning\(Dilgren and Wiegreffe,[2026](https://arxiv.org/html/2607.18264#bib.bib75)\)\.[Figure˜5](https://arxiv.org/html/2607.18264#S8.F5)shows representative examples of this analysis applied to our method\.MUXproduces interpretable latent tokens in both mathematical reasoning and parallel search settings\. For math reasoning, the top decoded subwords correspond to the operands, operators, and intermediate results of each step\. For parallel search, the decoded tokens recover the BFS frontier at each depth, and confirm that a single latent token maintains multiple hypotheses in superposition\.

In contrast, Coconut and CODI predict the correct answers, but their decoded tokens are uninformative and do not align with the reasoning spans\. The latent tokens fromMUXare not only useful for prediction, but also easier to read out\.

We complement this with quantitative metrics measured across all test examples\. Following the vocabulary\-projection probing approach used in prior work\(Haoet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib11); Dilgren and Wiegreffe,[2026](https://arxiv.org/html/2607.18264#bib.bib75)\), for each latent token𝐱i\\mathbf\{x\}\_\{i\}we project it through pretrained unembedding layer of the language model to obtain a distribution over the vocabulary and extract the top\-NNdecoded subwords\. We compare these against the reference discrete reasoning span𝐫i\\mathbf\{r\}\_\{i\}aligned to slotiiand report three metrics:

- •*Recall@NN*: the fraction of tokens in𝐫i\\mathbf\{r\}\_\{i\}that appear among the top\-NNdecoded subwords\.
- •*Step Alignment*: the fraction of slots for which the top\-NNdecoded set has its highest token overlap with the correct \(diagonally aligned\) reasoning step\.
- •*MRR*\(Mean Reciprocal Rank\): the average of1/rank1/\\text\{rank\}over all tokens in𝐫i\\mathbf\{r\}\_\{i\}, where rank is determined by the decoded vocabulary distribution\.

All metrics are micro\-averaged over slots and examples\. We useN=5N\{=\}5throughout\.

![Refer to caption](https://arxiv.org/html/2607.18264v1/x7.png)\(a\)Mathematical reasoning \(GSM8K\-AUG\)
![Refer to caption](https://arxiv.org/html/2607.18264v1/x8.png)\(b\)Parallel search \(MNNS\)

Figure 6:Quantitative interpretability results\.#### Mathematical reasoning\.

[Figure˜6\(a\)](https://arxiv.org/html/2607.18264#S8.F6.sf1)reports results on LLaMA 3\.2 1B\-Instruct trained on GSM8K\-AUG, evaluated over all test examples\.MUXrecovers68\.6%68\.6\\%of reference tokens \(Recall@5\) and achieves70\.2%70\.2\\%Step Alignment, confirming that latent tokens encode both the content and position of their aligned spans\. Removing the global loss \(MUX\(γ=0\\gamma\{=\}0\) \) yields nearly identical scores \(68\.1%68\.1\\%,69\.7%69\.7\\%\), pointing to local multiplexed supervision as the driver of interpretability\. CODI, trained with only global distillation, reaches12\.5%12\.5\\%Recall@5 and8\.6%8\.6\\%MRR, consistent with its latent tokens not preserving readable reasoning content\.

#### Parallel search\.

To evaluate whether latent tokens encode search structure, we apply the same metrics to the MNNS task\. We define two reference targets for each latent token at depthkk: the*trace*, which is the single partial sum along the optimal reasoning path at stepkk, and the*frontier*, which is the complete set of all partial sums reachable at depthkkover every possible sign assignment\. Trace metrics test whether the model recovers the particular solution path; frontier metrics test whether the latent state encodes the full distribution of reachable states at each depth\.[Figure˜6\(b\)](https://arxiv.org/html/2607.18264#S8.F6.sf2)shows the results\.MUXachieves90\.3%90\.3\\%trace Recall@5 and82\.3%82\.3\\%trace Step Alignment, compared to Coconut’s17\.4%17\.4\\%and23\.9%23\.9\\%\. The pattern is equally strong for frontier metrics \(90\.4%90\.4\\%vs\.13\.6%13\.6\\%Recall@5;91\.6%91\.6\\%vs\.22\.0%22\.0\\%Step Alignment\)\. These numbers show thatMUXlatent tokens encode both the solution path and the reachable states at each depth, consistent with the analysis in[Section˜4\.3](https://arxiv.org/html/2607.18264#S4.SS3)\.

### 8\.3Attention analysis

![Refer to caption](https://arxiv.org/html/2607.18264v1/x9.png)Figure 7:Attention analysis on GSM8K\-AUG\.We add an attention\-based diagnostic to test whether latent reasoning meaningfully contributes to final answer\. On LLaMA 3\.2 1B\-Instruct trained on GSM8K\-AUG, we measure how much the model attends to its latent tokens when producing the answer\. We extract last\-layer attention on all test examples at the answer\-interface tokens\{<EOT\>,The,answer,is,:\}\\\{\\texttt\{<EOT\>\},\\texttt\{The\},\\texttt\{answer\},\\texttt\{is\},\\texttt\{:\}\\\}and compute two quantities\. Letαlat\\alpha\_\{\\mathrm\{lat\}\}be the total attention weight on allKKlatent tokens, and letNpreN\_\{\\mathrm\{pre\}\}be the number of total preceding tokens\. We define*reasoning attention mass*=αlat=\\alpha\_\{\\mathrm\{lat\}\}, and*reasoning attention lift*=αlat/\(K/Npre\)=\\alpha\_\{\\mathrm\{lat\}\}\\,/\\,\(K/N\_\{\\mathrm\{pre\}\}\)\. A lift of1\.01\.0means the model distributes attention uniformly and values above1\.01\.0mean latent tokens receive more attention than their share within the context\.

[Figure˜7](https://arxiv.org/html/2607.18264#S8.F7)reports the results\. Panel \(a\) shows thatMUXassigns higher reasoning attention mass than CODI across almost every answer\-prediction step\. Panel \(b\) compares the per\-example distribution of reasoning attention lift at two scopes: the full answer bridge \(all five interface tokens\) and the final prediction token \(:\)\.MUXachieves higher lift in both cases \(0\.6330\.633vs\.0\.5420\.542at the answer bridge;0\.5440\.544vs\.0\.2950\.295at the final token\)\. In addition, on85\.1%85\.1\\%of examplesMUXroutes more attention through latent reasoning tokens at the answer bridge, rising to91\.8%91\.8\\%at the final prediction token\. Thus, relative to CODI,MUXmore effectively utilizes its learned latent reasoning when generating the answer\. We leave a theoretical explanation in[Section˜9\.3](https://arxiv.org/html/2607.18264#S9.SS3)\.[Figure˜8](https://arxiv.org/html/2607.18264#S8.F8)shows a representative last\-layer attention map\.MUXforms a clear autoregressive chain among its latent tokens before the answer bridge; CODI’s attention is diffuse and largely bypasses the latent tokens\.

### 8\.4Training cost analysis

Table 7:Training cost relative to CODI \(LLaMA\-1B\)Relative training cost\(vs\. CODI\)MethodGSM8K\-AUGGSM8K\-AUG\-NLSFT\-CoT0\.56×\\times0\.64×\\timesCoconut0\.44×\\times0\.36×\\timesCODI1\.00×\\times1\.00×\\timesSIM\-CoT1\.16×\\times1\.32×\\timesKaVa≈\\approx1\.00×\\times≈\\approx1\.00×\\timesMUX≈\\approx1\.00×\\times≈\\approx1\.00×\\timesWe compare the per\-step training FLOPs of each method using the standard approximation\(Kaplanet al\.,[2020](https://arxiv.org/html/2607.18264#bib.bib79)\): the forward pass costs≈2​P​L\\approx 2PLand the backward pass≈4​P​L\\approx 4PL, giving a total of≈6​P​L\\approx 6PLFLOPs per training step, wherePPis the number of model parameters andLLis the sequence length\. All self\-distillation methods \(CODI, SIM\-CoT, KaVa, andMUX\) process both a teacher sequence of lengthLt=Lq\+Lc\+LaL\_\{t\}=L\_\{q\}\+L\_\{c\}\+L\_\{a\}\(question, full chain\-of\-thought, answer\) and a student sequence of lengthLs=Lq\+K\+LaL\_\{s\}=L\_\{q\}\+K\+L\_\{a\}\(question,KKcontinuous tokens, answer\)\. Since the teacher cross\-entropy loss is included in the total training loss and gradients flow through both paths, each incurs the full6​P​L6PLtraining cost\. SFT\-CoT trains only on the teacher sequence, and Coconut trains only on the student sequence\.

The methods differ only in their auxiliary losses\. SIM\-CoT\(Weiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib15)\)trains a full auxiliary decoder withPdec=PP\_\{\\text\{dec\}\}=Pparameters on the chain\-of\-thought tokens, adding6​Pdec⋅Lc6P\_\{\\text\{dec\}\}\\cdot L\_\{c\}FLOPs per step\. KaVa\(Kuzinaet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib16)\)adds a KV\-cache matching loss \(Eq\. 7 in their paper\) with cost𝒪​\(M​H​L​d\)\\mathcal\{O\}\(MHLd\)whereMMis the number of retained KV pairs,HHthe number of KV heads,LLthe number of layers, andddthe head dimension\.MUXadds a multiplexed KL divergence overKKvocabulary distributions, costing𝒪​\(K​\|𝒱\|\)\\mathcal\{O\}\(K\|\\mathcal\{V\}\|\)\. Both KaVa’s andMUX’s auxiliary costs are negligible \(<<0\.01% of the base cost\)\. However, KaVa’s KV\-cache distillation requires an importance\-based eviction mechanism \(R\-KV\) that scores and selectively discards teacher KV pairs before matching, introducing additional architectural complexity and a lossy compression step that is absent inMUX\.

[Table˜7](https://arxiv.org/html/2607.18264#S8.T7)reports the total relative training cost for LLaMA\-1B on both GSM8K\-AUG \(Lq=55L\_\{q\}\{=\}55,Lc=25L\_\{c\}\{=\}25,La=8L\_\{a\}\{=\}8,K=6K\{=\}6\) and GSM8K\-AUG\-NL \(Lq=55L\_\{q\}\{=\}55,Lc=62L\_\{c\}\{=\}62,La=8L\_\{a\}\{=\}8,K=6K\{=\}6\)\. CODI, KaVa, andMUXhave effectively identical training cost on both datasets\. SIM\-CoT is 16% more expensive on AUG and 32% on AUG\-NL, as its decoder overhead scales with chain length\.

![Refer to caption](https://arxiv.org/html/2607.18264v1/x10.png)Figure 8:Attention routing through continuous reasoning tokens\.

## 9Proofs and theoretical details

### 9\.1Proofs of the main results

See[3](https://arxiv.org/html/2607.18264#Thmtheorem3)

###### Proof\.

We prove both directions\.

#### Sufficiency\.

Assumeℰ​\(𝜶\)\>0\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\>0\. We will show that𝗆𝗎𝗑\\mathsf\{mux\}is injective\.

Take two token sequences

\(r1,…,rS\),\(r′⁣1,…,r′⁣S\)\(r^\{1\},\\dots,r^\{S\}\),\\qquad\(r^\{\\prime 1\},\\dots,r^\{\\prime S\}\)such that

𝗆𝗎𝗑​\(r1,…,rS\)=𝗆𝗎𝗑​\(r′⁣1,…,r′⁣S\)\.\\mathsf\{mux\}\(r^\{1\},\\dots,r^\{S\}\)=\\mathsf\{mux\}\(r^\{\\prime 1\},\\dots,r^\{\\prime S\}\)\.This means that the two sequences induce exactly the same target distribution over the vocabulary\.

For each vocabulary tokenv∈𝒱v\\in\\mathcal\{V\}, define the set of positions at whichvvappears:

Av=\{j∈\{1,…,S\}:rj=v\},Bv=\{j∈\{1,…,S\}:r′⁣j=v\}\.A\_\{v\}=\\\{j\\in\\\{1,\\dots,S\\\}:r^\{j\}=v\\\},\\qquad B\_\{v\}=\\\{j\\in\\\{1,\\dots,S\\\}:r^\{\\prime j\}=v\\\}\.Because the two target distributions are equal, for everyv∈𝒱v\\in\\mathcal\{V\}we have

∑j∈Avαj=∑j∈Bvαj\.\\sum\_\{j\\in A\_\{v\}\}\\alpha\_\{j\}=\\sum\_\{j\\in B\_\{v\}\}\\alpha\_\{j\}\.Suppose, for contradiction, thatAv≠BvA\_\{v\}\\neq B\_\{v\}for somevv\. Then the coefficient vector defined by

cj=\{1,j∈Av∖Bv,−1,j∈Bv∖Av,0,otherwisec\_\{j\}=\\begin\{cases\}1,&j\\in A\_\{v\}\\setminus B\_\{v\},\\\\ \-1,&j\\in B\_\{v\}\\setminus A\_\{v\},\\\\ 0,&\\text\{otherwise\}\\end\{cases\}is nonzero and satisfies

∑j=1Scj​αj=∑j∈Avαj−∑j∈Bvαj=0\.\\sum\_\{j=1\}^\{S\}c\_\{j\}\\alpha\_\{j\}=\\sum\_\{j\\in A\_\{v\}\}\\alpha\_\{j\}\-\\sum\_\{j\\in B\_\{v\}\}\\alpha\_\{j\}=0\.This contradictsℰ​\(𝜶\)\>0\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\>0\. ThereforeAv=BvA\_\{v\}=B\_\{v\}for every tokenvv\.

Now fix any positionj∈\{1,…,S\}j\\in\\\{1,\\dots,S\\\}\. There is exactly one vocabulary tokenvvsuch thatj∈Avj\\in A\_\{v\}, namelyv=rjv=r^\{j\}\. SinceAv=BvA\_\{v\}=B\_\{v\}, we also havej∈Bvj\\in B\_\{v\}, sor′⁣j=v=rjr^\{\\prime j\}=v=r^\{j\}\. As this holds for every positionjj, the two sequences are identical\. Hence𝗆𝗎𝗑\\mathsf\{mux\}is injective\.

#### Necessity\.

Assumeℰ​\(𝜶\)=0\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)=0\. Then, by definition, there exists a nonzero coefficient vector

𝐜=\(c1,…,cS\)∈\{−1,0,1\}S\\mathbf\{c\}=\(c\_\{1\},\\dots,c\_\{S\}\)\\in\\\{\-1,0,1\\\}^\{S\}such that

∑j=1Scj​αj=0\.\\sum\_\{j=1\}^\{S\}c\_\{j\}\\alpha\_\{j\}=0\.Define two subsets

A=\{j:cj=1\},B=\{j:cj=−1\}\.A=\\\{j:c\_\{j\}=1\\\},\\qquad B=\\\{j:c\_\{j\}=\-1\\\}\.Since𝐜≠𝟎\\mathbf\{c\}\\neq\\mathbf\{0\}, at least one ofAAorBBis non\-empty\. Moreover,

∑j∈Aαj=∑j∈Bαj\.\\sum\_\{j\\in A\}\\alpha\_\{j\}=\\sum\_\{j\\in B\}\\alpha\_\{j\}\.Choose two distinct vocabulary tokensu,v∈𝒱u,v\\in\\mathcal\{V\}\. Construct two sequences by

rj=\{u,j∈A,v,j∉A,r′⁣j=\{u,j∈B,v,j∉B\.r^\{j\}=\\begin\{cases\}u,&j\\in A,\\\\ v,&j\\notin A,\\end\{cases\}\\qquad r^\{\\prime j\}=\\begin\{cases\}u,&j\\in B,\\\\ v,&j\\notin B\.\\end\{cases\}BecauseA≠BA\\neq B, the sequences are different\. However, the probability mass of tokenuuunder the first sequence is∑j∈Aαj\\sum\_\{j\\in A\}\\alpha\_\{j\}, while under the second sequence it is∑j∈Bαj\\sum\_\{j\\in B\}\\alpha\_\{j\}; these are equal\. The same is true for tokenvv, since both distributions sum to11, and all other tokens have probability0\. Therefore the two sequences induce exactly the same target distribution\. Hence𝗆𝗎𝗑\\mathsf\{mux\}is not injective\.

We have shown that𝗆𝗎𝗑\\mathsf\{mux\}is injective if and only ifℰ​\(𝜶\)\>0\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\>0\. ∎

See[4](https://arxiv.org/html/2607.18264#Thmtheorem4)

###### Proof\.

Fixi∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}\. By assumption,

ℰ​\(𝜶\(i\)\)\>0\.\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}^\{\(i\)\}\)\>0\.Therefore, by[Proposition˜3](https://arxiv.org/html/2607.18264#Thmtheorem3), the multiplexed target𝗆𝗎𝗑​\(𝐫i\)\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\), together with the span lengthSiS\_\{i\}and the corresponding masses𝜶\(i\)\\boldsymbol\{\\alpha\}^\{\(i\)\}, uniquely determines the full aligned span

𝐫i=\(ri1,…,riSi\)\.\\mathbf\{r\}\_\{i\}=\(r\_\{i\}^\{1\},\\dots,r\_\{i\}^\{S\_\{i\}\}\)\.This is true for every spani=1,…,Mi=1,\\ldots,M\.

Once all spans𝐫i\\mathbf\{r\}\_\{i\}have been recovered, the original reasoning trace is obtained by concatenating them in the same order\. Thus the ordered tuple

\(\(Si,𝜶\(i\),𝗆𝗎𝗑​\(𝐫i\)\)\)i=1M\\Bigl\(\(S\_\{i\},\\boldsymbol\{\\alpha\}^\{\(i\)\},\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\)\\Bigr\)\_\{i=1\}^\{M\}determines the full reasoning trace uniquely\. ∎

See[5](https://arxiv.org/html/2607.18264#Thmtheorem5)

###### Proof\.

For geometric weighting,

αj=ρj−1∑l=1Sρl−1\.\\alpha\_\{j\}=\\frac\{\\rho^\{j\-1\}\}\{\\sum\_\{l=1\}^\{S\}\\rho^\{l\-1\}\}\.Let

ZS=∑l=1Sρl−1\.Z\_\{S\}=\\sum\_\{l=1\}^\{S\}\\rho^\{l\-1\}\.Since0<ρ<10<\\rho<1, we haveZS\>0Z\_\{S\}\>0\.

Take any coefficient vector𝐜∈\{−1,0,1\}S\\mathbf\{c\}\\in\\\{\-1,0,1\\\}^\{S\}\. Then

∑j=1Scj​αj=∑j=1Scj​ρj−1ZS=1ZS​∑j=1Scj​ρj−1\.\\sum\_\{j=1\}^\{S\}c\_\{j\}\\alpha\_\{j\}=\\sum\_\{j=1\}^\{S\}c\_\{j\}\\frac\{\\rho^\{j\-1\}\}\{Z\_\{S\}\}=\\frac\{1\}\{Z\_\{S\}\}\\sum\_\{j=1\}^\{S\}c\_\{j\}\\rho^\{j\-1\}\.BecauseZS\>0Z\_\{S\}\>0, this quantity is zero if and only if

∑j=1Scj​ρj−1=0\.\\sum\_\{j=1\}^\{S\}c\_\{j\}\\rho^\{j\-1\}=0\.Therefore

ℰ​\(𝜶\)\>0⟺∑j=1Scj​ρj−1≠0for every​𝐜∈\{−1,0,1\}S∖\{𝟎\}\.\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\>0\\quad\\Longleftrightarrow\\quad\\sum\_\{j=1\}^\{S\}c\_\{j\}\\rho^\{j\-1\}\\neq 0\\quad\\text\{for every \}\\mathbf\{c\}\\in\\\{\-1,0,1\\\}^\{S\}\\setminus\\\{\\mathbf\{0\}\\\}\.[Proposition˜3](https://arxiv.org/html/2607.18264#Thmtheorem3)now gives the stated equivalence\.

To prove that ifρ∈\(0,1\)\\rho\\in\(0,1\)is rational, then geometric weighting is injective for every finite span lengthSS, supposeρ=p/q∈\(0,1\)\\rho=p/q\\in\(0,1\)is rational in lowest terms and that geometric weighting were not injective\. We proved that there would exist a nonzero polynomial

P​\(x\)=∑j=1Scj​xj−1,cj∈\{−1,0,1\},P\(x\)=\\sum\_\{j=1\}^\{S\}c\_\{j\}x^\{j\-1\},\\qquad c\_\{j\}\\in\\\{\-1,0,1\\\},such thatP​\(ρ\)=0P\(\\rho\)=0\. If necessary, divide out the largest power ofxxso that the constant term is nonzero\. The resulting polynomial still has integer coefficients, is nonzero, has constant term±1\\pm 1, and has leading coefficient±1\\pm 1\. By the rational root theorem, any rational root must be an integer divisor of the constant term divided by an integer divisor of the leading coefficient, hence must belong to\{±1\}\\\{\\pm 1\\\}\. This contradictsρ∈\(0,1\)\\rho\\in\(0,1\)\. Therefore no such polynomial exists, and the weighting is injective\.

To prove part \(ii\), let us introduce the following lemma on exponential polynomials first\.

###### Lemma 11\.

Letλ1,…,λn∈ℝ\\lambda\_\{1\},\\dots,\\lambda\_\{n\}\\in\\mathbb\{R\}be pairwise distinct, and let

f​\(x\)=∑m=1nbm​eλm​xf\(x\)=\\sum\_\{m=1\}^\{n\}b\_\{m\}e^\{\\lambda\_\{m\}x\}with real coefficientsbmb\_\{m\}, not all zero\. Thenffhas at mostn−1n\-1real zeros\.

###### Proof\.

We use induction onnn\.

Ifn=1n=1, then

f​\(x\)=b1​eλ1​xf\(x\)=b\_\{1\}e^\{\\lambda\_\{1\}x\}withb1≠0b\_\{1\}\\neq 0, sof​\(x\)≠0f\(x\)\\neq 0for allxx\. Thus the claim holds\.

Assume the statement holds forn−1n\-1, and consider

f​\(x\)=∑m=1nbm​eλm​xwithλ1<λ2<⋯<λn\.f\(x\)=\\sum\_\{m=1\}^\{n\}b\_\{m\}e^\{\\lambda\_\{m\}x\}\\quad\\text\{with\}\\quad\\lambda\_\{1\}<\\lambda\_\{2\}<\\cdots<\\lambda\_\{n\}\.Define

g​\(x\)=e−λ1​x​f​\(x\)=b1\+∑m=2nbm​e\(λm−λ1\)​x\.g\(x\)=e^\{\-\\lambda\_\{1\}x\}f\(x\)=b\_\{1\}\+\\sum\_\{m=2\}^\{n\}b\_\{m\}e^\{\(\\lambda\_\{m\}\-\\lambda\_\{1\}\)x\}\.The functionsffandgghave the same zeros becausee−λ1​xe^\{\-\\lambda\_\{1\}x\}is never zero\.

SupposegghasNNdistinct real zeros\. By Rolle’s theorem,g′g^\{\\prime\}has at leastN−1N\-1distinct real zeros\. But

g′​\(x\)=∑m=2nbm​\(λm−λ1\)​e\(λm−λ1\)​xg^\{\\prime\}\(x\)=\\sum\_\{m=2\}^\{n\}b\_\{m\}\(\\lambda\_\{m\}\-\\lambda\_\{1\}\)e^\{\(\\lambda\_\{m\}\-\\lambda\_\{1\}\)x\}is again an exponential polynomial, now withn−1n\-1pairwise distinct exponents\. By the induction hypothesis,g′g^\{\\prime\}has at mostn−2n\-2real zeros\. ThereforeN−1≤n−2N\-1\\leq n\-2, which impliesN≤n−1N\\leq n\-1\.

Hencegg, and therefore alsoff, has at mostn−1n\-1real zeros\. ∎

Now suppose the scoress1,…,sSs\_\{1\},\\dots,s\_\{S\}are pairwise distinct, and define

αj​\(λ\)=eλ​sj∑l=1Seλ​sl\.\\alpha\_\{j\}\(\\lambda\)=\\frac\{e^\{\\lambda s\_\{j\}\}\}\{\\sum\_\{l=1\}^\{S\}e^\{\\lambda s\_\{l\}\}\}\.By[Proposition˜3](https://arxiv.org/html/2607.18264#Thmtheorem3), injectivity fails if and only if there exists a nonzero coefficient vector

𝐜=\(c1,…,cS\)∈\{−1,0,1\}S\\mathbf\{c\}=\(c\_\{1\},\\dots,c\_\{S\}\)\\in\\\{\-1,0,1\\\}^\{S\}such that

∑j=1Scj​αj​\(λ\)=0\.\\sum\_\{j=1\}^\{S\}c\_\{j\}\\alpha\_\{j\}\(\\lambda\)=0\.Since the denominator∑leλ​sl\\sum\_\{l\}e^\{\\lambda s\_\{l\}\}is strictly positive, this is equivalent to

∑j=1Scj​eλ​sj=0\.\\sum\_\{j=1\}^\{S\}c\_\{j\}e^\{\\lambda s\_\{j\}\}=0\.For fixed nonzero𝐜\\mathbf\{c\}, the function

f𝐜​\(λ\)=∑j=1Scj​eλ​sjf\_\{\\mathbf\{c\}\}\(\\lambda\)=\\sum\_\{j=1\}^\{S\}c\_\{j\}e^\{\\lambda s\_\{j\}\}is a nonzero exponential polynomial with pairwise distinct exponentssjs\_\{j\}\. By[Lemma˜11](https://arxiv.org/html/2607.18264#Thmtheorem11),f𝐜f\_\{\\mathbf\{c\}\}has only finitely many real zeros\.

There are only finitely many nonzero coefficient vectors in\{−1,0,1\}S\\\{\-1,0,1\\\}^\{S\}\. Therefore the union of the zero sets of all such functionsf𝐜f\_\{\\mathbf\{c\}\}is finite\. Call this unionDD\. Ifλ∉D\\lambda\\notin D, then no nontrivial signed sum vanishes, soℰ​\(𝜶\)\>0\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\>0\. By[Proposition˜3](https://arxiv.org/html/2607.18264#Thmtheorem3), the encoding is injective\. ∎

See[6](https://arxiv.org/html/2607.18264#Thmtheorem6)

###### Proof\.

Part \(i\) is immediate because the function

u↦sin⁡\(π2​u\)u\\mapsto\\sin\\left\(\\frac\{\\pi\}\{2\}u\\right\)is strictly increasing on\[0,1\]\[0,1\], so the sinusoidal scores are pairwise distinct\. For part \(ii\), assume

0<θp​\(S−1\)<πfor every​p=1,…,P\.0<\\theta\_\{p\}\(S\-1\)<\\pi\\qquad\\text\{for every \}p=1,\\dots,P\.Fixp∈\{1,…,P\}p\\in\\\{1,\\dots,P\\\}\. For eachj=1,…,S−1j=1,\\dots,S\-1,

0≤\(j−1\)​θp<j​θp<π\.0\\leq\(j\-1\)\\theta\_\{p\}<j\\theta\_\{p\}<\\pi\.The cosine function is strictly decreasing on the interval\[0,π\]\[0,\\pi\]\. Hence

cos⁡\(θp​\(j−1\)\)\>cos⁡\(θp​j\)for​j=1,…,S−1\.\\cos\\bigl\(\\theta\_\{p\}\(j\-1\)\\bigr\)\>\\cos\(\\theta\_\{p\}j\)\\qquad\\text\{for \}j=1,\\dots,S\-1\.Averaging these inequalities overp∈\{1,…,P\}p\\in\\\{1,\\dots,P\\\}gives

1P​∑p=1Pcos⁡\(θp​\(j−1\)\)\>1P​∑p=1Pcos⁡\(θp​j\),\\frac\{1\}\{P\}\\sum\_\{p=1\}^\{P\}\\cos\\bigl\(\\theta\_\{p\}\(j\-1\)\\bigr\)\>\\frac\{1\}\{P\}\\sum\_\{p=1\}^\{P\}\\cos\(\\theta\_\{p\}j\),that is,

sj\>sj\+1for​j=1,…,S−1\.s\_\{j\}\>s\_\{j\+1\}\\qquad\\text\{for \}j=1,\\dots,S\-1\.Thus the rotary scalar scores are strictly decreasing and therefore pairwise distinct\. The injectivity claim then follows immediately from[Proposition˜5](https://arxiv.org/html/2607.18264#Thmtheorem5)\. ∎

See[9](https://arxiv.org/html/2607.18264#Thmtheorem9)

###### Proof\.

Let

n:=\|𝒦\|,W~:=W/τ,mi:=𝗆𝗎𝗑​\(𝐫i\),qi:=f​\(𝐱i\)=softmax​\(W~​𝐱i\),ei:=‖mi−qi‖1\.n:=\|\\mathcal\{K\}\|,\\qquad\\widetilde\{W\}:=W/\\tau,\\qquad m\_\{i\}:=\\mathsf\{mux\}\(\{\\bf r\}\_\{i\}\),\\qquad q\_\{i\}:=f\(\{\\bf x\}\_\{i\}\)=\\mathrm\{softmax\}\(\\widetilde\{W\}\{\\bf x\}\_\{i\}\),\\qquad e\_\{i\}:=\\\|m\_\{i\}\-q\_\{i\}\\\|\_\{1\}\.
By Pinsker’s inequality,

ei≤2​DKL​\(mi∥qi\)for every​i∈𝒦\.e\_\{i\}\\leq\\sqrt\{2\\,D\_\{\\mathrm\{KL\}\}\(m\_\{i\}\\,\\\|\\,q\_\{i\}\)\}\\qquad\\text\{for every \}i\\in\\mathcal\{K\}\.Hence, by Jensen’s inequality and the assumptionℒlocal≤δ\\mathcal\{L\}\_\{\\mathrm\{local\}\}\\leq\\delta,

1n​∑i∈𝒦ei≤1n​∑i∈𝒦2​DKL​\(mi∥qi\)≤2⋅1n​∑i∈𝒦DKL​\(mi∥qi\)≤2​δ\.\\frac\{1\}\{n\}\\sum\_\{i\\in\\mathcal\{K\}\}e\_\{i\}\\leq\\frac\{1\}\{n\}\\sum\_\{i\\in\\mathcal\{K\}\}\\sqrt\{2\\,D\_\{\\mathrm\{KL\}\}\(m\_\{i\}\\,\\\|\\,q\_\{i\}\)\}\\leq\\sqrt\{2\\cdot\\frac\{1\}\{n\}\\sum\_\{i\\in\\mathcal\{K\}\}D\_\{\\mathrm\{KL\}\}\(m\_\{i\}\\,\\\|\\,q\_\{i\}\)\}\\leq\\sqrt\{2\\delta\}\.
For any distincti,j∈𝒦i,j\\in\\mathcal\{K\},[Definition˜8](https://arxiv.org/html/2607.18264#Thmtheorem8)and the triangle inequality give

‖qi−qj‖1≥‖mi−mj‖1−‖mi−qi‖1−‖mj−qj‖1≥𝒟−ei−ej\.\\\|q\_\{i\}\-q\_\{j\}\\\|\_\{1\}\\geq\\\|m\_\{i\}\-m\_\{j\}\\\|\_\{1\}\-\\\|m\_\{i\}\-q\_\{i\}\\\|\_\{1\}\-\\\|m\_\{j\}\-q\_\{j\}\\\|\_\{1\}\\geq\\mathcal\{D\}\-e\_\{i\}\-e\_\{j\}\.Averaging over all ordered pairsi≠ji\\neq jyields

1n​\(n−1\)​∑i≠j‖qi−qj‖1≥𝒟−1n​\(n−1\)​∑i≠j\(ei\+ej\)\.\\frac\{1\}\{n\(n\-1\)\}\\sum\_\{i\\neq j\}\\\|q\_\{i\}\-q\_\{j\}\\\|\_\{1\}\\geq\\mathcal\{D\}\-\\frac\{1\}\{n\(n\-1\)\}\\sum\_\{i\\neq j\}\(e\_\{i\}\+e\_\{j\}\)\.Since

1n​\(n−1\)​∑i≠j\(ei\+ej\)=2n​∑i∈𝒦ei,\\frac\{1\}\{n\(n\-1\)\}\\sum\_\{i\\neq j\}\(e\_\{i\}\+e\_\{j\}\)=\\frac\{2\}\{n\}\\sum\_\{i\\in\\mathcal\{K\}\}e\_\{i\},we obtain

1n​\(n−1\)​∑i≠j‖qi−qj‖1≥𝒟−2n​∑i∈𝒦ei≥𝒟−2​2​δ\.\\frac\{1\}\{n\(n\-1\)\}\\sum\_\{i\\neq j\}\\\|q\_\{i\}\-q\_\{j\}\\\|\_\{1\}\\geq\\mathcal\{D\}\-\\frac\{2\}\{n\}\\sum\_\{i\\in\\mathcal\{K\}\}e\_\{i\}\\geq\\mathcal\{D\}\-2\\sqrt\{2\\delta\}\.Becauseδ<𝒟2/8\\delta<\\mathcal\{D\}^\{2\}/8, the right\-hand side is strictly positive\.

Letσ\\sigmadenote the softmax map\. For𝐳∈ℝ\|𝒱\|\\mathbf\{z\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}, write

Jσ​\(𝐳\)=Diag⁡\(σ​\(𝐳\)\)−σ​\(𝐳\)​σ​\(𝐳\)⊤J\_\{\\sigma\}\(\\mathbf\{z\}\)=\\operatorname\{Diag\}\(\\sigma\(\\mathbf\{z\}\)\)\-\\sigma\(\\mathbf\{z\}\)\\sigma\(\\mathbf\{z\}\)^\{\\\!\\top\}for its Jacobian matrix, and define

C\|𝒱\|:=sup𝐳∈ℝ\|𝒱\|‖Jσ​\(𝐳\)‖2→1,‖A‖2→1:=sup‖𝐯‖2=1‖A​𝐯‖1\.C\_\{\|\\mathcal\{V\}\|\}:=\\sup\_\{\\mathbf\{z\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}\}\\\|J\_\{\\sigma\}\(\\mathbf\{z\}\)\\\|\_\{2\\to 1\},\\qquad\\\|A\\\|\_\{2\\to 1\}:=\\sup\_\{\\\|\\mathbf\{v\}\\\|\_\{2\}=1\}\\\|A\\mathbf\{v\}\\\|\_\{1\}\.ThenC\|𝒱\|C\_\{\|\\mathcal\{V\}\|\}depends only on\|𝒱\|\|\\mathcal\{V\}\|\. By the mean value theorem, for everyi,ji,j,

‖qi−qj‖1=‖σ​\(W~​𝐱i\)−σ​\(W~​𝐱j\)‖1≤C\|𝒱\|​‖W~​\(𝐱i−𝐱j\)‖≤C\|𝒱\|​‖W~‖op​‖𝐱i−𝐱j‖\.\\\|q\_\{i\}\-q\_\{j\}\\\|\_\{1\}=\\\|\\sigma\(\\widetilde\{W\}\{\\bf x\}\_\{i\}\)\-\\sigma\(\\widetilde\{W\}\{\\bf x\}\_\{j\}\)\\\|\_\{1\}\\leq C\_\{\|\\mathcal\{V\}\|\}\\,\\\|\\widetilde\{W\}\(\{\\bf x\}\_\{i\}\-\{\\bf x\}\_\{j\}\)\\\|\\leq C\_\{\|\\mathcal\{V\}\|\}\\,\\\|\\widetilde\{W\}\\\|\_\{\\mathrm\{op\}\}\\,\\\|\{\\bf x\}\_\{i\}\-\{\\bf x\}\_\{j\}\\\|\.Therefore

1n​\(n−1\)​∑i≠j‖𝐱i−𝐱j‖≥𝒟−2​2​δ‖W~‖op​C\|𝒱\|\.\\frac\{1\}\{n\(n\-1\)\}\\sum\_\{i\\neq j\}\\\|\{\\bf x\}\_\{i\}\-\{\\bf x\}\_\{j\}\\\|\\geq\\frac\{\\mathcal\{D\}\-2\\sqrt\{2\\delta\}\}\{\\\|\\widetilde\{W\}\\\|\_\{\\mathrm\{op\}\}\\,C\_\{\|\\mathcal\{V\}\|\}\}\.Applying Jensen’s inequality,

1n​\(n−1\)​∑i≠j‖𝐱i−𝐱j‖2≥\(1n​\(n−1\)​∑i≠j‖𝐱i−𝐱j‖\)2≥\(𝒟−2​2​δ‖W~‖op​C\|𝒱\|\)2\.\\frac\{1\}\{n\(n\-1\)\}\\sum\_\{i\\neq j\}\\\|\{\\bf x\}\_\{i\}\-\{\\bf x\}\_\{j\}\\\|^\{2\}\\geq\\left\(\\frac\{1\}\{n\(n\-1\)\}\\sum\_\{i\\neq j\}\\\|\{\\bf x\}\_\{i\}\-\{\\bf x\}\_\{j\}\\\|\\right\)^\{2\}\\geq\\left\(\\frac\{\\mathcal\{D\}\-2\\sqrt\{2\\delta\}\}\{\\\|\\widetilde\{W\}\\\|\_\{\\mathrm\{op\}\}\\,C\_\{\|\\mathcal\{V\}\|\}\}\\right\)^\{2\}\.Since the diagonal terms vanish,

1n2​∑i,j∈𝒦‖𝐱i−𝐱j‖2=n−1n⋅1n​\(n−1\)​∑i≠j‖𝐱i−𝐱j‖2≥n−1n​\(𝒟−2​2​δ‖W~‖op​C\|𝒱\|\)2\.\\frac\{1\}\{n^\{2\}\}\\sum\_\{i,j\\in\\mathcal\{K\}\}\\\|\{\\bf x\}\_\{i\}\-\{\\bf x\}\_\{j\}\\\|^\{2\}=\\frac\{n\-1\}\{n\}\\cdot\\frac\{1\}\{n\(n\-1\)\}\\sum\_\{i\\neq j\}\\\|\{\\bf x\}\_\{i\}\-\{\\bf x\}\_\{j\}\\\|^\{2\}\\geq\\frac\{n\-1\}\{n\}\\left\(\\frac\{\\mathcal\{D\}\-2\\sqrt\{2\\delta\}\}\{\\\|\\widetilde\{W\}\\\|\_\{\\mathrm\{op\}\}\\,C\_\{\|\\mathcal\{V\}\|\}\}\\right\)^\{2\}\.This proves the claimed lower bound on the average pairwise squared distance\. Hence the continuous tokens cannot exhibit representation collapse at any levelε\\varepsilonbelow this quantity\. Since the average of these nonnegative squared distances is at least this quantity, there exists a distinct pairi,j∈𝒦i,j\\in\\mathcal\{K\}such that

‖𝐱i−𝐱j‖≥n−1n​𝒟−2​2​δ‖W~‖op​C\|𝒱\|\.\\\|\{\\bf x\}\_\{i\}\-\{\\bf x\}\_\{j\}\\\|\\geq\\sqrt\{\\frac\{n\-1\}\{n\}\}\\,\\frac\{\\mathcal\{D\}\-2\\sqrt\{2\\delta\}\}\{\\\|\\widetilde\{W\}\\\|\_\{\\mathrm\{op\}\}\\,C\_\{\|\\mathcal\{V\}\|\}\}\.∎

See[10](https://arxiv.org/html/2607.18264#Thmtheorem10)

###### Proof\.

We construct a recurrence over continuous tokens and verify that it exactly implements breadth\-first search\.

For a setB⊆𝒩B\\subseteq\\mathcal\{N\}, let1B∈\{0,1\}n1\_\{B\}\\in\\\{0,1\\\}^\{n\}denote its indicator vector, wheren=\|𝒩\|n=\|\\mathcal\{N\}\|\. At stepkk, let the continuous token be the pair

\(fk,uk\)∈\{0,1\}2​n,\(f\_\{k\},u\_\{k\}\)\\in\\\{0,1\\\}^\{2n\},where

fk=1Fk,uk=1Uk\.f\_\{k\}=1\_\{F\_\{k\}\},\\qquad u\_\{k\}=1\_\{U\_\{k\}\}\.Thus the token stores the current frontier and the set of visited nodes\.

Initialize

f0=1\{s\},u0=1\{s\}\.f\_\{0\}=1\_\{\\\{s\\\}\},\\qquad u\_\{0\}=1\_\{\\\{s\\\}\}\.
LetA∈\{0,1\}n×nA\\in\\\{0,1\\\}^\{n\\times n\}be the adjacency matrix of the graph, with

Au​v=1⇔\(u,v\)∈E\.A\_\{uv\}=1\\iff\(u,v\)\\in E\.Given the token\(fk,uk\)\(f\_\{k\},u\_\{k\}\), define the next token by

gk\+1=1​\[A⊤​fk\>0\],g\_\{k\+1\}=1\[A^\{\\top\}f\_\{k\}\>0\],fk\+1=gk\+1⊙\(1−uk\),f\_\{k\+1\}=g\_\{k\+1\}\\odot\(1\-u\_\{k\}\),uk\+1=uk\+fk\+1\.u\_\{k\+1\}=u\_\{k\}\+f\_\{k\+1\}\.
We claim that for everyk≤Hk\\leq H,

fk=1Fk,uk=1Uk\.f\_\{k\}=1\_\{F\_\{k\}\},\\qquad u\_\{k\}=1\_\{U\_\{k\}\}\.
The claim is immediate atk=0k=0\. Assume it holds at stepkk\. For any nodev∈𝒩v\\in\\mathcal\{N\},

\(gk\+1\)v=1⇔\(A⊤​fk\)v\>0⇔∃u∈Fk​such that​\(u,v\)∈E⇔v∈N\+​\(Fk\)\.\(g\_\{k\+1\}\)\_\{v\}=1\\iff\(A^\{\\top\}f\_\{k\}\)\_\{v\}\>0\\iff\\exists\\,u\\in F\_\{k\}\\text\{ such that \}\(u,v\)\\in E\\iff v\\in N^\{\+\}\(F\_\{k\}\)\.Therefore

gk\+1=1N\+​\(Fk\)\.g\_\{k\+1\}=1\_\{N^\{\+\}\(F\_\{k\}\)\}\.Hence

fk\+1=1N\+​\(Fk\)⊙\(1−1Uk\)=1N\+​\(Fk\)∖Uk=1Fk\+1\.f\_\{k\+1\}=1\_\{N^\{\+\}\(F\_\{k\}\)\}\\odot\(1\-1\_\{U\_\{k\}\}\)=1\_\{N^\{\+\}\(F\_\{k\}\)\\setminus U\_\{k\}\}=1\_\{F\_\{k\+1\}\}\.Also, by the BFS update,Fk\+1∩Uk=∅F\_\{k\+1\}\\cap U\_\{k\}=\\varnothing, so

uk\+1=uk\+fk\+1=1Uk\+1Fk\+1=1Uk∪Fk\+1=1Uk\+1\.u\_\{k\+1\}=u\_\{k\}\+f\_\{k\+1\}=1\_\{U\_\{k\}\}\+1\_\{F\_\{k\+1\}\}=1\_\{U\_\{k\}\\cup F\_\{k\+1\}\}=1\_\{U\_\{k\+1\}\}\.This proves the claim by induction\.

It follows that the recurrence exactly tracks the breadth\-first frontier and visited set at every step\. In particular, the final answer is exact:

y=1​\[t∈UH\]\.y=1\[t\\in U\_\{H\}\]\.Indeed, sinceuH=1UHu\_\{H\}=1\_\{U\_\{H\}\}is part of the final token, the answer head can read the coordinate corresponding tottand output the correct answer\.

It remains to recover the frontier distribution\. IfFk=∅F\_\{k\}=\\varnothing, one may use a designated null distribution\. Assume now thatFk≠∅F\_\{k\}\\neq\\varnothing\. Since

the frontier is explicitly encoded in the token, so define

pk​\(v\)=\(fk\)v‖fk‖1\.p\_\{k\}\(v\)=\\frac\{\(f\_\{k\}\)\_\{v\}\}\{\\\|f\_\{k\}\\\|\_\{1\}\}\.Then

pk​\(v\)=\{1/\|Fk\|,v∈Fk,0,v∉Fk\.p\_\{k\}\(v\)=\\begin\{cases\}1/\|F\_\{k\}\|,&v\\in F\_\{k\},\\\\ 0,&v\\notin F\_\{k\}\.\\end\{cases\}By the setup of[Section˜5\.2](https://arxiv.org/html/2607.18264#S5.SS2), this is exactlymux​\(rk\)\\mathrm\{mux\}\(r\_\{k\}\)\.

Finally, if one insists on a standard softmax readout with finite logits, exact zeros outsideFkF\_\{k\}are impossible, but arbitrarily good approximation is still possible\. ForB\>0B\>0, define

ℓk​\(v\)=B​\(\(fk\)v−1\)\.\\ell\_\{k\}\(v\)=B\\bigl\(\(f\_\{k\}\)\_\{v\}\-1\\bigr\)\.Then

ℓk​\(v\)=\{0,v∈Fk,−B,v∉Fk\.\\ell\_\{k\}\(v\)=\\begin\{cases\}0,&v\\in F\_\{k\},\\\\ \-B,&v\\notin F\_\{k\}\.\\end\{cases\}Letm=\|Fk\|m=\|F\_\{k\}\|\. The corresponding softmax distribution is

pk\(B\)​\(v\)=eℓk​\(v\)∑u∈𝒩eℓk​\(u\)\.p\_\{k\}^\{\(B\)\}\(v\)=\\frac\{e^\{\\ell\_\{k\}\(v\)\}\}\{\\sum\_\{u\\in\\mathcal\{N\}\}e^\{\\ell\_\{k\}\(u\)\}\}\.Since

∑u∈𝒩eℓk​\(u\)=m\+\(\|𝒩\|−m\)​e−B,\\sum\_\{u\\in\\mathcal\{N\}\}e^\{\\ell\_\{k\}\(u\)\}=m\+\(\|\\mathcal\{N\}\|\-m\)e^\{\-B\},we obtain

pk\(B\)​\(v\)=\{1m\+\(\|𝒩\|−m\)​e−B,v∈Fk,e−Bm\+\(\|𝒩\|−m\)​e−B,v∉Fk\.p\_\{k\}^\{\(B\)\}\(v\)=\\begin\{cases\}\\dfrac\{1\}\{m\+\(\|\\mathcal\{N\}\|\-m\)e^\{\-B\}\},&v\\in F\_\{k\},\\\\\[5\.38193pt\] \\dfrac\{e^\{\-B\}\}\{m\+\(\|\\mathcal\{N\}\|\-m\)e^\{\-B\}\},&v\\notin F\_\{k\}\.\\end\{cases\}Therefore

pk\(B\)→mux​\(rk\)as​B→∞\.p\_\{k\}^\{\(B\)\}\\to\\mathrm\{mux\}\(r\_\{k\}\)\\qquad\\text\{as \}B\\to\\infty\.So the frontier distribution is recoverable from the continuous token exactly, and realizable by a standard softmax readout up to arbitrarily small error\. ∎

### 9\.2Multiplexing under finite precision

[Proposition˜3](https://arxiv.org/html/2607.18264#Thmtheorem3)characterizes lossless multiplexing in exact arithmetic: for a fixed span lengthSS, injectivity of𝗆𝗎𝗑:𝒱S→Δ\|𝒱\|−1\\mathsf\{mux\}:\\mathcal\{V\}^\{S\}\\to\\Delta^\{\|\\mathcal\{V\}\|\-1\}is equivalent toℰ​\(𝜶\)\>0\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\>0\. We now make the finite\-precision version of this statement explicit\. The argument has two steps\. First, the same marginℰ​\(𝜶\)\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)is the minimum coordinatewise separation between distinct exact targets\. Second, under the standard unit\-roundoff model, target construction introduces anO​\(S​u\)O\(Su\)perturbation for span lengthSSand unit roundoffuu\. We state everything for one fixed span lengthSS; for full traces with varying span lengths, the argument applies spanwise exactly as in[Corollary˜4](https://arxiv.org/html/2607.18264#Thmtheorem4)\. We use theℓ∞\\ell\_\{\\infty\}norm because each coordinate of𝗆𝗎𝗑​\(𝐫\)\\mathsf\{mux\}\(\{\\bf r\}\)is a subset sum of the masses, and the separation margin in[Definition˜2](https://arxiv.org/html/2607.18264#Thmtheorem2)is coordinatewise\.

We first identifyℰ​\(𝜶\)\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)as the minimumℓ∞\\ell\_\{\\infty\}\-distance between two distinct exact multiplexed targets\.

###### Proposition 12\(Separation between distinct exact multiplexed targets\)\.

Assume\|𝒱\|\>1\|\\mathcal\{V\}\|\>1\. Then

min𝐫≠𝐫′∈𝒱S⁡‖𝗆𝗎𝗑​\(𝐫\)−𝗆𝗎𝗑​\(𝐫′\)‖∞=ℰ​\(𝜶\)\.\\min\_\{\{\\bf r\}\\neq\{\\bf r\}^\{\\prime\}\\in\\mathcal\{V\}^\{S\}\}\\\|\\mathsf\{mux\}\(\{\\bf r\}\)\-\\mathsf\{mux\}\(\{\\bf r\}^\{\\prime\}\)\\\|\_\{\\infty\}=\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\.

###### Proof\.

Take any distinct𝐫,𝐫′∈𝒱S\{\\bf r\},\{\\bf r\}^\{\\prime\}\\in\\mathcal\{V\}^\{S\}\. For each vocabulary symbolv∈𝒱v\\in\\mathcal\{V\},

\(𝗆𝗎𝗑​\(𝐫\)−𝗆𝗎𝗑​\(𝐫′\)\)v=∑j=1Scj\(v\)​αj,cj\(v\)=𝟏​\[rj=v\]−𝟏​\[\(r′\)j=v\]∈\{−1,0,1\}\.\\bigl\(\\mathsf\{mux\}\(\{\\bf r\}\)\-\\mathsf\{mux\}\(\{\\bf r\}^\{\\prime\}\)\\bigr\)\_\{v\}=\\sum\_\{j=1\}^\{S\}c\_\{j\}^\{\(v\)\}\\alpha\_\{j\},\\qquad c\_\{j\}^\{\(v\)\}=\\mathbf\{1\}\[r^\{j\}=v\]\-\\mathbf\{1\}\[\(r^\{\\prime\}\)^\{j\}=v\]\\in\\\{\-1,0,1\\\}\.If𝐫≠𝐫′\{\\bf r\}\\neq\{\\bf r\}^\{\\prime\}, then for at least onevvthe coefficient vector\(c1\(v\),…,cS\(v\)\)\(c\_\{1\}^\{\(v\)\},\\dots,c\_\{S\}^\{\(v\)\}\)is nonzero\. By[Definition˜2](https://arxiv.org/html/2607.18264#Thmtheorem2),

\|\(𝗆𝗎𝗑​\(𝐫\)−𝗆𝗎𝗑​\(𝐫′\)\)v\|≥ℰ​\(𝜶\),\\left\|\\bigl\(\\mathsf\{mux\}\(\{\\bf r\}\)\-\\mathsf\{mux\}\(\{\\bf r\}^\{\\prime\}\)\\bigr\)\_\{v\}\\right\|\\geq\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\),and hence

‖𝗆𝗎𝗑​\(𝐫\)−𝗆𝗎𝗑​\(𝐫′\)‖∞≥ℰ​\(𝜶\)\.\\\|\\mathsf\{mux\}\(\{\\bf r\}\)\-\\mathsf\{mux\}\(\{\\bf r\}^\{\\prime\}\)\\\|\_\{\\infty\}\\geq\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\.Taking the minimum over all distinct pairs yields

min𝐫≠𝐫′⁡‖𝗆𝗎𝗑​\(𝐫\)−𝗆𝗎𝗑​\(𝐫′\)‖∞≥ℰ​\(𝜶\)\.\\min\_\{\{\\bf r\}\\neq\{\\bf r\}^\{\\prime\}\}\\\|\\mathsf\{mux\}\(\{\\bf r\}\)\-\\mathsf\{mux\}\(\{\\bf r\}^\{\\prime\}\)\\\|\_\{\\infty\}\\geq\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\.
For the reverse inequality, choose a nonzero vector𝐜=\(c1,…,cS\)∈\{−1,0,1\}S\{\\bf c\}=\(c\_\{1\},\\dots,c\_\{S\}\)\\in\\\{\-1,0,1\\\}^\{S\}attaining the minimum in \([5](https://arxiv.org/html/2607.18264#S4.E5)\)\. Since\|𝒱\|\>1\|\\mathcal\{V\}\|\>1, pick distinct symbolsu,v∈𝒱u,v\\in\\mathcal\{V\}, and define𝐫,𝐫′∈𝒱S\{\\bf r\},\{\\bf r\}^\{\\prime\}\\in\\mathcal\{V\}^\{S\}by

rj=\{u,cj=1,v,cj∈\{−1,0\},\(r′\)j=\{u,cj=−1,v,cj∈\{1,0\}\.r^\{j\}=\\begin\{cases\}u,&c\_\{j\}=1,\\\\ v,&c\_\{j\}\\in\\\{\-1,0\\\},\\end\{cases\}\\qquad\(r^\{\\prime\}\)^\{j\}=\\begin\{cases\}u,&c\_\{j\}=\-1,\\\\ v,&c\_\{j\}\\in\\\{1,0\\\}\.\\end\{cases\}Then the only possibly nonzero coordinates of𝗆𝗎𝗑​\(𝐫\)−𝗆𝗎𝗑​\(𝐫′\)\\mathsf\{mux\}\(\{\\bf r\}\)\-\\mathsf\{mux\}\(\{\\bf r\}^\{\\prime\}\)are theuu\- andvv\-coordinates, equal to

∑j=1Scj​αjand−∑j=1Scj​αj,\\sum\_\{j=1\}^\{S\}c\_\{j\}\\alpha\_\{j\}\\qquad\\text\{and\}\\qquad\-\\sum\_\{j=1\}^\{S\}c\_\{j\}\\alpha\_\{j\},respectively\. Therefore

‖𝗆𝗎𝗑​\(𝐫\)−𝗆𝗎𝗑​\(𝐫′\)‖∞=\|∑j=1Scj​αj\|=ℰ​\(𝜶\),\\\|\\mathsf\{mux\}\(\{\\bf r\}\)\-\\mathsf\{mux\}\(\{\\bf r\}^\{\\prime\}\)\\\|\_\{\\infty\}=\\left\|\\sum\_\{j=1\}^\{S\}c\_\{j\}\\alpha\_\{j\}\\right\|=\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\),which proves the reverse inequality\. ∎

[Proposition˜12](https://arxiv.org/html/2607.18264#Thmtheorem12)is the exact\-arithmetic separation statement\. It shows that any perturbation smaller than half of this margin preserves unique demultiplexing\.

###### Corollary 13\(Stable demultiplexing under bounded perturbation\)\.

Assume\|𝒱\|\>1\|\\mathcal\{V\}\|\>1\. Let𝐫∈𝒱S\{\\bf r\}\\in\\mathcal\{V\}^\{S\}, and lety∈ℝ\|𝒱\|y\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}satisfy

‖y−𝗆𝗎𝗑​\(𝐫\)‖∞<ℰ​\(𝜶\)2\.\\\|y\-\\mathsf\{mux\}\(\{\\bf r\}\)\\\|\_\{\\infty\}<\\frac\{\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\}\{2\}\.Then𝐫\{\\bf r\}is the unique minimum\-ℓ∞\\ell\_\{\\infty\}demultiplexing ofyy, i\.e\.

arg⁡min𝐬∈𝒱S⁡‖y−𝗆𝗎𝗑​\(𝐬\)‖∞=\{𝐫\}\.\\arg\\min\_\{\{\\bf s\}\\in\\mathcal\{V\}^\{S\}\}\\\|y\-\\mathsf\{mux\}\(\{\\bf s\}\)\\\|\_\{\\infty\}=\\\{\{\\bf r\}\\\}\.

###### Proof\.

Take any competitor𝐬≠𝐫\{\\bf s\}\\neq\{\\bf r\}\. By[Proposition˜12](https://arxiv.org/html/2607.18264#Thmtheorem12),

‖𝗆𝗎𝗑​\(𝐫\)−𝗆𝗎𝗑​\(𝐬\)‖∞≥ℰ​\(𝜶\)\.\\\|\\mathsf\{mux\}\(\{\\bf r\}\)\-\\mathsf\{mux\}\(\{\\bf s\}\)\\\|\_\{\\infty\}\\geq\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\.Hence the triangle inequality gives

‖y−𝗆𝗎𝗑​\(𝐬\)‖∞≥‖𝗆𝗎𝗑​\(𝐫\)−𝗆𝗎𝗑​\(𝐬\)‖∞−‖y−𝗆𝗎𝗑​\(𝐫\)‖∞\>ℰ​\(𝜶\)−ℰ​\(𝜶\)2=ℰ​\(𝜶\)2\.\\\|y\-\\mathsf\{mux\}\(\{\\bf s\}\)\\\|\_\{\\infty\}\\geq\\\|\\mathsf\{mux\}\(\{\\bf r\}\)\-\\mathsf\{mux\}\(\{\\bf s\}\)\\\|\_\{\\infty\}\-\\\|y\-\\mathsf\{mux\}\(\{\\bf r\}\)\\\|\_\{\\infty\}\>\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\-\\frac\{\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\}\{2\}=\\frac\{\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\}\{2\}\.On the other hand,

‖y−𝗆𝗎𝗑​\(𝐫\)‖∞<ℰ​\(𝜶\)2\.\\\|y\-\\mathsf\{mux\}\(\{\\bf r\}\)\\\|\_\{\\infty\}<\\frac\{\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\}\{2\}\.Therefore

‖y−𝗆𝗎𝗑​\(𝐫\)‖∞<‖y−𝗆𝗎𝗑​\(𝐬\)‖∞for every​𝐬≠𝐫,\\\|y\-\\mathsf\{mux\}\(\{\\bf r\}\)\\\|\_\{\\infty\}<\\\|y\-\\mathsf\{mux\}\(\{\\bf s\}\)\\\|\_\{\\infty\}\\qquad\\text\{for every \}\{\\bf s\}\\neq\{\\bf r\},so𝐫\{\\bf r\}is the unique minimizer\. ∎

[Corollary˜13](https://arxiv.org/html/2607.18264#Thmtheorem13)applies to any perturbationyynear an exact multiplexed target, regardless of its source\. In this paper we use it for finite\-precision target construction\. Let𝗆𝗎𝗑~​\(𝐫\)\\widetilde\{\\mathsf\{mux\}\}\(\{\\bf r\}\)denote the target materialized by the implementation, and define the worst\-case target\-construction error

εfp:=sup𝐫∈𝒱S‖𝗆𝗎𝗑~​\(𝐫\)−𝗆𝗎𝗑​\(𝐫\)‖∞\.\\varepsilon\_\{\\mathrm\{fp\}\}:=\\sup\_\{\{\\bf r\}\\in\\mathcal\{V\}^\{S\}\}\\\|\\widetilde\{\\mathsf\{mux\}\}\(\{\\bf r\}\)\-\\mathsf\{mux\}\(\{\\bf r\}\)\\\|\_\{\\infty\}\.This is a target\-side quantity: it can include rounding of the masses, approximate normalization, and summation error\. We now makeεfp\\varepsilon\_\{\\mathrm\{fp\}\}explicit under the standard unit\-roundoff model\. Letuudenote the unit roundoff, and define

γn​\(u\):=n​u1−n​u,n​u<1\.\\gamma\_\{n\}\(u\):=\\frac\{nu\}\{1\-nu\},\\qquad nu<1\.Assume the exact normalized massesαj\\alpha\_\{j\}are fixed first, and that:

1. \(i\)eachαj\\alpha\_\{j\}is stored once in the working format asα^j\\widehat\{\\alpha\}\_\{j\}, with \|α^j−αj\|≤u​αj;\|\\widehat\{\\alpha\}\_\{j\}\-\\alpha\_\{j\}\|\\leq u\\alpha\_\{j\};
2. \(ii\)each coordinate of𝗆𝗎𝗑~​\(𝐫\)\\widetilde\{\\mathsf\{mux\}\}\(\{\\bf r\}\)is formed by naively summing the relevant stored massesα^j\\widehat\{\\alpha\}\_\{j\}in the same arithmetic\.

This isolates the floating\-point error after the exact masses are fixed\.

###### Corollary 14\(Floating\-point sufficient condition\)\.

Under the model above,

εfp≤ηS​\(u\):=u\+\(1\+u\)​γS−1​\(u\)=S​u1−\(S−1\)​u\.\\varepsilon\_\{\\mathrm\{fp\}\}\\leq\\eta\_\{S\}\(u\):=u\+\(1\+u\)\\gamma\_\{S\-1\}\(u\)=\\frac\{Su\}\{1\-\(S\-1\)u\}\.Consequently, exact recovery is guaranteed whenever

ηS​\(u\)<ℰ​\(𝜶\)2\.\\eta\_\{S\}\(u\)<\\frac\{\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\}\{2\}\.

###### Proof\.

Fix𝐫∈𝒱S\{\\bf r\}\\in\\mathcal\{V\}^\{S\}and a vocabulary symbolv∈𝒱v\\in\\mathcal\{V\}\. Let

Iv:=\{j:rj=v\},xv:=∑j∈Ivαj,x^v:=∑j∈Ivα^j\.I\_\{v\}:=\\\{j:r^\{j\}=v\\\},\\qquad x\_\{v\}:=\\sum\_\{j\\in I\_\{v\}\}\\alpha\_\{j\},\\qquad\\widehat\{x\}\_\{v\}:=\\sum\_\{j\\in I\_\{v\}\}\\widehat\{\\alpha\}\_\{j\}\.IfIv=∅I\_\{v\}=\\varnothing, thenxv=x^v=0x\_\{v\}=\\widehat\{x\}\_\{v\}=0, so the bound is trivial\. AssumeIv≠∅I\_\{v\}\\neq\\varnothing\. Since all terms are nonnegative,

\|x^v−xv\|≤∑j∈Iv\|α^j−αj\|≤u​∑j∈Ivαj=u​xv\.\|\\widehat\{x\}\_\{v\}\-x\_\{v\}\|\\leq\\sum\_\{j\\in I\_\{v\}\}\|\\widehat\{\\alpha\}\_\{j\}\-\\alpha\_\{j\}\|\\leq u\\sum\_\{j\\in I\_\{v\}\}\\alpha\_\{j\}=ux\_\{v\}\.Letx~v\\widetilde\{x\}\_\{v\}be the value obtained by naively summing the stored massesα^j\\widehat\{\\alpha\}\_\{j\}\. Standard floating\-point summation bounds give

x~v=x^v​\(1\+θ\|Iv\|−1\),\|θ\|Iv\|−1\|≤γ\|Iv\|−1​\(u\)≤γS−1​\(u\)\.\\widetilde\{x\}\_\{v\}=\\widehat\{x\}\_\{v\}\(1\+\\theta\_\{\|I\_\{v\}\|\-1\}\),\\qquad\|\\theta\_\{\|I\_\{v\}\|\-1\}\|\\leq\\gamma\_\{\|I\_\{v\}\|\-1\}\(u\)\\leq\\gamma\_\{S\-1\}\(u\)\.Therefore

\|x~v−x^v\|≤γS−1​\(u\)​x^v≤\(1\+u\)​γS−1​\(u\)​xv,\|\\widetilde\{x\}\_\{v\}\-\\widehat\{x\}\_\{v\}\|\\leq\\gamma\_\{S\-1\}\(u\)\\,\\widehat\{x\}\_\{v\}\\leq\(1\+u\)\\gamma\_\{S\-1\}\(u\)\\,x\_\{v\},where we usedx^v≤\(1\+u\)​xv\\widehat\{x\}\_\{v\}\\leq\(1\+u\)x\_\{v\}\. Combining the two bounds yields

\|x~v−xv\|≤\(u\+\(1\+u\)​γS−1​\(u\)\)​xv≤u\+\(1\+u\)​γS−1​\(u\)\.\|\\widetilde\{x\}\_\{v\}\-x\_\{v\}\|\\leq\\bigl\(u\+\(1\+u\)\\gamma\_\{S\-1\}\(u\)\\bigr\)x\_\{v\}\\leq u\+\(1\+u\)\\gamma\_\{S\-1\}\(u\)\.Taking the maximum overvvproves

εfp≤u\+\(1\+u\)​γS−1​\(u\)=S​u1−\(S−1\)​u\.\\varepsilon\_\{\\mathrm\{fp\}\}\\leq u\+\(1\+u\)\\gamma\_\{S\-1\}\(u\)=\\frac\{Su\}\{1\-\(S\-1\)u\}\.The recovery condition then follows from[Corollary˜13](https://arxiv.org/html/2607.18264#Thmtheorem13)\. ∎

For round\-to\-nearest arithmetic,

uFP32=2−24≈5\.96×10−8\.u\_\{\\mathrm\{FP32\}\}=2^\{\-24\}\\approx 5\.96\\times 10^\{\-8\}\.Hence, for2≤S≤322\\leq S\\leq 32,

ηS​\(uFP32\)≤1\.91×10−6,\\eta\_\{S\}\(u\_\{\\mathrm\{FP32\}\}\)\\leq 1\.91\\times 10^\{\-6\},
#### Geometric weights\.

The floating\-point bound above is independent of the weighting family\. The weighting enters only throughℰ​\(𝜶\)\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\. For rational geometric weights, this margin admits an exact integer\-arithmetic representation\.

###### Corollary 15\(Rational geometric weights\)\.

Supposeρ=p/q∈\(0,1\)\\rho=p/q\\in\(0,1\)is rational in lowest terms and

αj=ρj−1∑ℓ=0S−1ρℓ,j=1,…,S\.\\alpha\_\{j\}=\\frac\{\\rho^\{j\-1\}\}\{\\sum\_\{\\ell=0\}^\{S\-1\}\\rho^\{\\ell\}\},\\qquad j=1,\\dots,S\.Then

ℰ​\(𝜶\)=q−pqS−pS​mS,\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)=\\frac\{q\-p\}\{q^\{S\}\-p^\{S\}\}\\,m\_\{S\},where

mS:=min𝐜∈\{−1,0,1\}S∖\{0\}⁡\|∑j=1Scj​pj−1​qS−j\|\.m\_\{S\}:=\\min\_\{\{\\bf c\}\\in\\\{\-1,0,1\\\}^\{S\}\\setminus\\\{0\\\}\}\\left\|\\sum\_\{j=1\}^\{S\}c\_\{j\}p^\{j\-1\}q^\{S\-j\}\\right\|\.MoreovermS≥1m\_\{S\}\\geq 1, and therefore

ℰ​\(𝜶\)≥q−pqS−pS\.\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\\geq\\frac\{q\-p\}\{q^\{S\}\-p^\{S\}\}\.Consequently, a sufficient condition for exact recovery is

ηS​\(u\)<q−p2​\(qS−pS\)\.\\eta\_\{S\}\(u\)<\\frac\{q\-p\}\{2\(q^\{S\}\-p^\{S\}\)\}\.

###### Proof\.

Using

∑ℓ=0S−1\(pq\)ℓ=qS−pSqS−1​\(q−p\),\\sum\_\{\\ell=0\}^\{S\-1\}\\Bigl\(\\frac\{p\}\{q\}\\Bigr\)^\{\\ell\}=\\frac\{q^\{S\}\-p^\{S\}\}\{q^\{S\-1\}\(q\-p\)\},we can rewrite the normalized masses as

αj=\(q−p\)​pj−1​qS−jqS−pS\.\\alpha\_\{j\}=\\frac\{\(q\-p\)p^\{j\-1\}q^\{S\-j\}\}\{q^\{S\}\-p^\{S\}\}\.Hence, for any nonzero𝐜∈\{−1,0,1\}S\{\\bf c\}\\in\\\{\-1,0,1\\\}^\{S\},

∑j=1Scj​αj=q−pqS−pS​∑j=1Scj​pj−1​qS−j\.\\sum\_\{j=1\}^\{S\}c\_\{j\}\\alpha\_\{j\}=\\frac\{q\-p\}\{q^\{S\}\-p^\{S\}\}\\sum\_\{j=1\}^\{S\}c\_\{j\}p^\{j\-1\}q^\{S\-j\}\.Taking absolute values and then the minimum over all nonzero𝐜\{\\bf c\}gives the exact formula forℰ​\(𝜶\)\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\. The quantity inside the absolute value is an integer\. It is nonzero for every nonzero𝐜\{\\bf c\}, because otherwise∑j=1Scj​ρj−1=0\\sum\_\{j=1\}^\{S\}c\_\{j\}\\rho^\{j\-1\}=0, contradicting[Proposition˜5](https://arxiv.org/html/2607.18264#Thmtheorem5)\(a\)\. ThereforemS≥1m\_\{S\}\\geq 1, which yields the lower bound\. The final condition follows by combining this lower bound with[Corollary˜14](https://arxiv.org/html/2607.18264#Thmtheorem14)\. ∎

For our default choiceρ=9/10\\rho=9/10,

ℰ​\(𝜶\)=mS10S−9S,mS=min𝐜∈\{−1,0,1\}S∖\{0\}⁡\|∑j=1Scj​9j−1​10S−j\|\.\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)=\\frac\{m\_\{S\}\}\{10^\{S\}\-9^\{S\}\},\\qquad m\_\{S\}=\\min\_\{\{\\bf c\}\\in\\\{\-1,0,1\\\}^\{S\}\\setminus\\\{0\\\}\}\\left\|\\sum\_\{j=1\}^\{S\}c\_\{j\}\\,9^\{j\-1\}10^\{S\-j\}\\right\|\.This quantity can be evaluated exactly offline by integer arithmetic for each span lengthSSused in practice\. ForS≤30S\\leq 30, exact evaluation gives

ℰ​\(𝜶\)≈3\.98×10−6​at​S=11,ℰ​\(𝜶\)≈1\.20×10−6​at​S=12,\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\\approx 3\.98\\times 10^\{\-6\}\\ \\text\{at \}S=11,\\qquad\\mathcal\{E\}\(\\boldsymbol\{\\alpha\}\)\\approx 1\.20\\times 10^\{\-6\}\\ \\text\{at \}S=12,By contrast,

η11​\(uFP32\)≈6\.56×10−7,η12​\(uFP32\)≈7\.15×10−7,\\eta\_\{11\}\(u\_\{\\mathrm\{FP32\}\}\)\\approx 6\.56\\times 10^\{\-7\},\\qquad\\eta\_\{12\}\(u\_\{\\mathrm\{FP32\}\}\)\\approx 7\.15\\times 10^\{\-7\},Therefore, for the default geometric choiceρ=0\.9\\rho=0\.9, the conservative certificate from[Corollary˜14](https://arxiv.org/html/2607.18264#Thmtheorem14)holds in FP32 up toS=11S=11\.

### 9\.3Why local distillation preserves answer\-side use of latent reasoning

This section gives an objective\-level explanation for the attention pattern observed in[Section˜8\.2](https://arxiv.org/html/2607.18264#S8.SS2)\. The two auxiliary terms inℒ=ℒanswer\+β​ℒlocal\+γ​ℒglobal\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{answer\}\}\+\\beta\\,\\mathcal\{L\}\_\{\\mathrm\{local\}\}\+\\gamma\\,\\mathcal\{L\}\_\{\\mathrm\{global\}\}constrain different objects\. The local termℒlocal\\mathcal\{L\}\_\{\\mathrm\{local\}\}constrains each latent reasoning token𝐱i\\mathbf\{x\}\_\{i\}toward its own aligned target𝗆𝗎𝗑​\(𝐫i\)\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\), whereasℒglobal\\mathcal\{L\}\_\{\\mathrm\{global\}\}constrains only the aggregate hidden state used to produce the answer\. We show that only the former yields a tokenwise lower bound on answer\-side routing through previous latent reasoning tokens\. Our positive result is a*routing\-transfer*statement\. We do not claim that answer\-side routing through latent reasoning appears automatically\. Instead, we isolate the regime in which the aligned multiplexed targets already have an answer\-side advantage over non\-reasoning context, and ask whether local distillation preserves that advantage after those targets are replaced by actual latent tokens\.

Fix an answer\-interface tokentt\. In[Section˜8\.2](https://arxiv.org/html/2607.18264#S8.SS2)these are the tokens

\{<EOT\>,The,answer,is,:\}\.\\\{\\texttt\{<EOT\>\},\\ \\texttt\{The\},\\ \\texttt\{answer\},\\ \\texttt\{is\},\\ \\texttt\{:\}\\\}\.Letℬt\\mathcal\{B\}\_\{t\}denote the set of non\-reasoning positions visible tottthat are shared by the discrete and continuous reasoning modes, namely question tokens and answer\-bridge tokens\. Recall that𝒦⊆\{1,…,K\}\\mathcal\{K\}\\subseteq\\\{1,\\dots,K\\\}is the set of latent\-token positions whose aligned span is non\-empty\.

For eachi∈𝒦i\\in\\mathcal\{K\}, let

st,i:Δ\|𝒱\|−1→ℝs\_\{t,i\}:\\Delta^\{\|\\mathcal\{V\}\|\-1\}\\to\\mathbb\{R\}denote the attention logit assigned by tokenttin the*continuous reasoning mode*to positioniias a function of the represented contentf​\(𝐱i\)f\(\\mathbf\{x\}\_\{i\}\)\. For each background positionb∈ℬtb\\in\\mathcal\{B\}\_\{t\}, letξt​\(b\)∈ℝ\\xi\_\{t\}\(b\)\\in\\mathbb\{R\}denote its corresponding attention logit in the same mode\. We define the total attention mass assigned byttto previous latent reasoning tokens by

At​\(𝐱\)=∑i∈𝒦exp⁡\(st,i​\(f​\(𝐱i\)\)\)∑i∈𝒦exp⁡\(st,i​\(f​\(𝐱i\)\)\)\+∑b∈ℬtexp⁡\(ξt​\(b\)\)\.A\_\{t\}\(\\mathbf\{x\}\)=\\frac\{\\sum\_\{i\\in\\mathcal\{K\}\}\\exp\(s\_\{t,i\}\(f\(\\mathbf\{x\}\_\{i\}\)\)\)\}\{\\sum\_\{i\\in\\mathcal\{K\}\}\\exp\(s\_\{t,i\}\(f\(\\mathbf\{x\}\_\{i\}\)\)\)\+\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}\\exp\(\\xi\_\{t\}\(b\)\)\}\.\(7\)
To state the routing bound, it is enough to summarize the answer\-side geometry at tokenttby two intrinsic quantities\. First, define the aligned\-target margin

Δtref:=mini∈𝒦,b∈ℬt⁡\(st,i​\(𝗆𝗎𝗑​\(𝐫i\)\)−ξt​\(b\)\)\.\\Delta\_\{t\}^\{\\mathrm\{ref\}\}:=\\min\_\{i\\in\\mathcal\{K\},\\,b\\in\\mathcal\{B\}\_\{t\}\}\\Bigl\(s\_\{t,i\}\(\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\)\-\\xi\_\{t\}\(b\)\\Bigr\)\.This is the worst\-case logit margin, in the continuous reasoning mode, between an aligned multiplexed target and a background position\. Second, forδ\>0\\delta\>0, define the local score\-drift modulus

ωt​\(δ\):=maxi∈𝒦​supp∈Δ\|𝒱\|−1:DKL​\(𝗆𝗎𝗑​\(𝐫i\)∥p\)≤δ\(st,i​\(𝗆𝗎𝗑​\(𝐫i\)\)−st,i​\(p\)\)\.\\omega\_\{t\}\(\\delta\):=\\max\_\{i\\in\\mathcal\{K\}\}\\sup\_\{\\begin\{subarray\}\{c\}p\\in\\Delta^\{\|\\mathcal\{V\}\|\-1\}:\\\\ D\_\{\\mathrm\{KL\}\}\(\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\\,\\\|\\,p\)\\leq\\delta\\end\{subarray\}\}\\Bigl\(s\_\{t,i\}\(\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\)\-s\_\{t,i\}\(p\)\\Bigr\)\.This quantity measures the largest downward change in routing score caused by replacing the aligned target with any content inside a KL\-ball of radiusδ\\delta\.

###### Proposition 16\(Local distillation preserves answer\-side routing\)\.

Fix an answer\-interface tokenttandδ\>0\\delta\>0\. Define the set of well\-aligned latent\-token positions by

𝒦δ=\{i∈𝒦:DKL​\(𝗆𝗎𝗑​\(𝐫i\)∥f​\(𝐱i\)\)≤δ\}\.\\mathcal\{K\}\_\{\\delta\}=\\left\\\{i\\in\\mathcal\{K\}:D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\\,\\\|\\,f\(\\mathbf\{x\}\_\{i\}\)\\right\)\\leq\\delta\\right\\\}\.Then

\|𝒦δ\|≥\|𝒦\|​\(1−ℒlocalδ\),\|\\mathcal\{K\}\_\{\\delta\}\|\\geq\|\\mathcal\{K\}\|\\left\(1\-\\frac\{\\mathcal\{L\}\_\{\\mathrm\{local\}\}\}\{\\delta\}\\right\),\(8\)and

At​\(𝐱\)≥\|𝒦δ\|​exp⁡\(Δtref−ωt​\(δ\)\)\|𝒦δ\|​exp⁡\(Δtref−ωt​\(δ\)\)\+\|ℬt\|\.A\_\{t\}\(\\mathbf\{x\}\)\\geq\\frac\{\|\\mathcal\{K\}\_\{\\delta\}\|\\,\\exp\(\\Delta\_\{t\}^\{\\mathrm\{ref\}\}\-\\omega\_\{t\}\(\\delta\)\)\}\{\|\\mathcal\{K\}\_\{\\delta\}\|\\,\\exp\(\\Delta\_\{t\}^\{\\mathrm\{ref\}\}\-\\omega\_\{t\}\(\\delta\)\)\+\|\\mathcal\{B\}\_\{t\}\|\}\.\(9\)In particular, ifΔtref\>ωt​\(δ\)\\Delta\_\{t\}^\{\\mathrm\{ref\}\}\>\\omega\_\{t\}\(\\delta\), then everyi∈𝒦δi\\in\\mathcal\{K\}\_\{\\delta\}satisfies

st,i​\(f​\(𝐱i\)\)\>ξt​\(b\),∀b∈ℬt,s\_\{t,i\}\(f\(\\mathbf\{x\}\_\{i\}\)\)\>\\xi\_\{t\}\(b\),\\qquad\\forall b\\in\\mathcal\{B\}\_\{t\},so each well\-aligned latent reasoning token individually outranks every background position at tokentt\.

###### Proof\.

We first prove \([8](https://arxiv.org/html/2607.18264#S9.E8)\)\.

For eachi∈𝒦i\\in\\mathcal\{K\}, define

di:=DKL\(𝗆𝗎𝗑\(𝐫i\)∥f\(𝐱i\)\)\.d\_\{i\}:=D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\\,\\\|\\,f\(\\mathbf\{x\}\_\{i\}\)\\right\)\.By \([4](https://arxiv.org/html/2607.18264#S3.E4)\),

ℒlocal=1\|𝒦\|​∑i∈𝒦di\.\\mathcal\{L\}\_\{\\mathrm\{local\}\}=\\frac\{1\}\{\|\\mathcal\{K\}\|\}\\sum\_\{i\\in\\mathcal\{K\}\}d\_\{i\}\.Let

ℰδ:=\{i∈𝒦:di\>δ\}\.\\mathcal\{E\}\_\{\\delta\}:=\\\{i\\in\\mathcal\{K\}:d\_\{i\}\>\\delta\\\}\.Each index inℰδ\\mathcal\{E\}\_\{\\delta\}contributes more thanδ\\deltato the sum, hence

∑i∈𝒦di≥∑i∈ℰδdi\>\|ℰδ\|​δ\.\\sum\_\{i\\in\\mathcal\{K\}\}d\_\{i\}\\geq\\sum\_\{i\\in\\mathcal\{E\}\_\{\\delta\}\}d\_\{i\}\>\|\\mathcal\{E\}\_\{\\delta\}\|\\,\\delta\.Dividing by\|𝒦\|\|\\mathcal\{K\}\|gives

ℒlocal\>\|ℰδ\|\|𝒦\|​δ,\\mathcal\{L\}\_\{\\mathrm\{local\}\}\>\\frac\{\|\\mathcal\{E\}\_\{\\delta\}\|\}\{\|\\mathcal\{K\}\|\}\\,\\delta,so

\|ℰδ\|<\|𝒦\|​ℒlocalδ\.\|\\mathcal\{E\}\_\{\\delta\}\|<\|\\mathcal\{K\}\|\\,\\frac\{\\mathcal\{L\}\_\{\\mathrm\{local\}\}\}\{\\delta\}\.Since𝒦δ=𝒦∖ℰδ\\mathcal\{K\}\_\{\\delta\}=\\mathcal\{K\}\\setminus\\mathcal\{E\}\_\{\\delta\}, we obtain

\|𝒦δ\|=\|𝒦\|−\|ℰδ\|≥\|𝒦\|​\(1−ℒlocalδ\),\|\\mathcal\{K\}\_\{\\delta\}\|=\|\\mathcal\{K\}\|\-\|\\mathcal\{E\}\_\{\\delta\}\|\\geq\|\\mathcal\{K\}\|\\left\(1\-\\frac\{\\mathcal\{L\}\_\{\\mathrm\{local\}\}\}\{\\delta\}\\right\),which proves \([8](https://arxiv.org/html/2607.18264#S9.E8)\)\.

We now prove \([9](https://arxiv.org/html/2607.18264#S9.E9)\)\. Fix anyi∈𝒦δi\\in\\mathcal\{K\}\_\{\\delta\}\. By definition of𝒦δ\\mathcal\{K\}\_\{\\delta\},

DKL​\(𝗆𝗎𝗑​\(𝐫i\)∥f​\(𝐱i\)\)≤δ\.D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\\,\\\|\\,f\(\\mathbf\{x\}\_\{i\}\)\\right\)\\leq\\delta\.Therefore, by definition ofωt​\(δ\)\\omega\_\{t\}\(\\delta\),

st,i​\(f​\(𝐱i\)\)≥st,i​\(𝗆𝗎𝗑​\(𝐫i\)\)−ωt​\(δ\)\.s\_\{t,i\}\(f\(\\mathbf\{x\}\_\{i\}\)\)\\geq s\_\{t,i\}\(\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\)\-\\omega\_\{t\}\(\\delta\)\.By definition ofΔtref\\Delta\_\{t\}^\{\\mathrm\{ref\}\}, for everyb∈ℬtb\\in\\mathcal\{B\}\_\{t\},

st,i​\(𝗆𝗎𝗑​\(𝐫i\)\)≥ξt​\(b\)\+Δtref\.s\_\{t,i\}\(\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\)\\geq\\xi\_\{t\}\(b\)\+\\Delta\_\{t\}^\{\\mathrm\{ref\}\}\.Combining the two displays gives

st,i​\(f​\(𝐱i\)\)≥ξt​\(b\)\+Δtref−ωt​\(δ\),∀b∈ℬt\.s\_\{t,i\}\(f\(\\mathbf\{x\}\_\{i\}\)\)\\geq\\xi\_\{t\}\(b\)\+\\Delta\_\{t\}^\{\\mathrm\{ref\}\}\-\\omega\_\{t\}\(\\delta\),\\qquad\\forall b\\in\\mathcal\{B\}\_\{t\}\.\(10\)
Choosebt⋆∈ℬtb\_\{t\}^\{\\star\}\\in\\mathcal\{B\}\_\{t\}satisfying

ξt​\(bt⋆\)=maxb∈ℬt⁡ξt​\(b\)\.\\xi\_\{t\}\(b\_\{t\}^\{\\star\}\)=\\max\_\{b\\in\\mathcal\{B\}\_\{t\}\}\\xi\_\{t\}\(b\)\.Applying \([10](https://arxiv.org/html/2607.18264#S9.E10)\) withb=bt⋆b=b\_\{t\}^\{\\star\}gives

st,i​\(f​\(𝐱i\)\)≥ξt​\(bt⋆\)\+Δtref−ωt​\(δ\)\.s\_\{t,i\}\(f\(\\mathbf\{x\}\_\{i\}\)\)\\geq\\xi\_\{t\}\(b\_\{t\}^\{\\star\}\)\+\\Delta\_\{t\}^\{\\mathrm\{ref\}\}\-\\omega\_\{t\}\(\\delta\)\.Exponentiating both sides,

exp⁡\(st,i​\(f​\(𝐱i\)\)\)≥exp⁡\(Δtref−ωt​\(δ\)\)​exp⁡\(ξt​\(bt⋆\)\)\.\\exp\(s\_\{t,i\}\(f\(\\mathbf\{x\}\_\{i\}\)\)\)\\geq\\exp\(\\Delta\_\{t\}^\{\\mathrm\{ref\}\}\-\\omega\_\{t\}\(\\delta\)\)\\,\\exp\(\\xi\_\{t\}\(b\_\{t\}^\{\\star\}\)\)\.This holds for everyi∈𝒦δi\\in\\mathcal\{K\}\_\{\\delta\}\. Summing overi∈𝒦δi\\in\\mathcal\{K\}\_\{\\delta\},

∑i∈𝒦exp⁡\(st,i​\(f​\(𝐱i\)\)\)≥\|𝒦δ\|​exp⁡\(Δtref−ωt​\(δ\)\)​exp⁡\(ξt​\(bt⋆\)\)\.\\sum\_\{i\\in\\mathcal\{K\}\}\\exp\(s\_\{t,i\}\(f\(\\mathbf\{x\}\_\{i\}\)\)\)\\geq\|\\mathcal\{K\}\_\{\\delta\}\|\\,\\exp\(\\Delta\_\{t\}^\{\\mathrm\{ref\}\}\-\\omega\_\{t\}\(\\delta\)\)\\,\\exp\(\\xi\_\{t\}\(b\_\{t\}^\{\\star\}\)\)\.On the other hand, by maximality ofξt​\(bt⋆\)\\xi\_\{t\}\(b\_\{t\}^\{\\star\}\),

∑b∈ℬtexp⁡\(ξt​\(b\)\)≤\|ℬt\|​exp⁡\(ξt​\(bt⋆\)\)\.\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}\\exp\(\\xi\_\{t\}\(b\)\)\\leq\|\\mathcal\{B\}\_\{t\}\|\\,\\exp\(\\xi\_\{t\}\(b\_\{t\}^\{\\star\}\)\)\.Substituting these two bounds into \([7](https://arxiv.org/html/2607.18264#S9.E7)\) gives \([9](https://arxiv.org/html/2607.18264#S9.E9)\)\. The final claim follows directly from \([10](https://arxiv.org/html/2607.18264#S9.E10)\)\. ∎

[Proposition˜16](https://arxiv.org/html/2607.18264#Thmtheorem16)shows that the local objective controls how many positions stay inside a KL\-ball around their aligned targets, while the sign and magnitude ofΔtref−ωt​\(δ\)\\Delta\_\{t\}^\{\\mathrm\{ref\}\}\-\\omega\_\{t\}\(\\delta\)determine whether the answer\-side preference survives inside that ball\. The count bound becomes informative onceδ\>ℒlocal\\delta\>\\mathcal\{L\}\_\{\\mathrm\{local\}\}, but the proposition itself does not assume any positivity condition on the margin\.

To connect this statement back to the*discrete reasoning mode*, letℓ¯t​\(v\)\\bar\{\\ell\}\_\{t\}\(v\)denote the attention logit from tokenttto a discrete reasoning positionvv\. For each aligned span𝐫i=\(𝐫i1,…,𝐫i\|𝐫i\|\)\\mathbf\{r\}\_\{i\}=\(\\mathbf\{r\}\_\{i\}^\{1\},\\dots,\\mathbf\{r\}\_\{i\}^\{\|\\mathbf\{r\}\_\{i\}\|\}\), define the corresponding span\-level routing score by

s¯t,i:=log​∑u=1\|𝐫i\|exp⁡\(ℓ¯t​\(𝐫iu\)\)\.\\bar\{s\}\_\{t,i\}:=\\log\\sum\_\{u=1\}^\{\|\\mathbf\{r\}\_\{i\}\|\}\\exp\(\\bar\{\\ell\}\_\{t\}\(\\mathbf\{r\}\_\{i\}^\{u\}\)\)\.\(11\)Thus,s¯t,i\\bar\{s\}\_\{t,i\}is the log\-sum\-exp score assigned by tokenttto the entire aligned span𝐫i\\mathbf\{r\}\_\{i\}in the discrete reasoning mode\. For each background positionb∈ℬtb\\in\\mathcal\{B\}\_\{t\}, letξ¯t​\(b\)∈ℝ\\bar\{\\xi\}\_\{t\}\(b\)\\in\\mathbb\{R\}denote its attention logit in the discrete reasoning mode\. Now define the discrete routing margin

Δtdisc:=mini∈𝒦,b∈ℬt⁡\(s¯t,i−ξ¯t​\(b\)\)\.\\Delta\_\{t\}^\{\\mathrm\{disc\}\}:=\\min\_\{i\\in\\mathcal\{K\},\\,b\\in\\mathcal\{B\}\_\{t\}\}\\bigl\(\\bar\{s\}\_\{t,i\}\-\\bar\{\\xi\}\_\{t\}\(b\)\\bigr\)\.This is the formal version of the answer\-side preference for aligned reasoning spans studied in prior routing analyses\(Zhanget al\.,[2025b](https://arxiv.org/html/2607.18264#bib.bib58); Tuteket al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib57)\)\. Also define the calibration gap

Γt:=max⁡\{maxi∈𝒦⁡\|st,i​\(𝗆𝗎𝗑​\(𝐫i\)\)−s¯t,i\|,maxb∈ℬt⁡\|ξt​\(b\)−ξ¯t​\(b\)\|\}\.\\Gamma\_\{t\}:=\\max\\left\\\{\\max\_\{i\\in\\mathcal\{K\}\}\\left\|s\_\{t,i\}\(\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\)\-\\bar\{s\}\_\{t,i\}\\right\|,\\max\_\{b\\in\\mathcal\{B\}\_\{t\}\}\\left\|\\xi\_\{t\}\(b\)\-\\bar\{\\xi\}\_\{t\}\(b\)\\right\|\\right\\\}\.Then, for everyi∈𝒦i\\in\\mathcal\{K\}andb∈ℬtb\\in\\mathcal\{B\}\_\{t\},

st,i​\(𝗆𝗎𝗑​\(𝐫i\)\)−ξt​\(b\)≥s¯t,i−ξ¯t​\(b\)−2​Γt,s\_\{t,i\}\(\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)\)\-\\xi\_\{t\}\(b\)\\geq\\bar\{s\}\_\{t,i\}\-\\bar\{\\xi\}\_\{t\}\(b\)\-2\\Gamma\_\{t\},hence

Δtref≥Δtdisc−2​Γt\.\\Delta\_\{t\}^\{\\mathrm\{ref\}\}\\geq\\Delta\_\{t\}^\{\\mathrm\{disc\}\}\-2\\Gamma\_\{t\}\.Substituting this into[Proposition˜16](https://arxiv.org/html/2607.18264#Thmtheorem16)yields

At​\(𝐱\)≥\|𝒦δ\|​exp⁡\(Δtdisc−2​Γt−ωt​\(δ\)\)\|𝒦δ\|​exp⁡\(Δtdisc−2​Γt−ωt​\(δ\)\)\+\|ℬt\|\.A\_\{t\}\(\\mathbf\{x\}\)\\geq\\frac\{\|\\mathcal\{K\}\_\{\\delta\}\|\\,\\exp\(\\Delta\_\{t\}^\{\\mathrm\{disc\}\}\-2\\Gamma\_\{t\}\-\\omega\_\{t\}\(\\delta\)\)\}\{\|\\mathcal\{K\}\_\{\\delta\}\|\\,\\exp\(\\Delta\_\{t\}^\{\\mathrm\{disc\}\}\-2\\Gamma\_\{t\}\-\\omega\_\{t\}\(\\delta\)\)\+\|\\mathcal\{B\}\_\{t\}\|\}\.This is the sense in which local distillation preserves answer\-side routing: a routing preference present in the discrete trace transfers to the continuous mode provided the aligned targets retain a positive calibrated margin and the actual distilled tokens do not drift far enough to erase it\.

We now contrast this with trajectory\-level supervision alone\. Let𝐡t⋆∈ℝd\\mathbf\{h\}\_\{t\}^\{\\star\}\\in\\mathbb\{R\}^\{d\}denote the answer\-interface hidden state in the discrete reasoning mode at tokentt\. Write the hidden state in the continuous reasoning mode as

𝐡t​\(𝐱\)=∑i∈𝒦at,i​\(𝐱\)​𝐯t,i\+∑b∈ℬtat,b​\(𝐱\)​𝐯t,b,\\mathbf\{h\}\_\{t\}\(\\mathbf\{x\}\)=\\sum\_\{i\\in\\mathcal\{K\}\}a\_\{t,i\}\(\\mathbf\{x\}\)\\,\\mathbf\{v\}\_\{t,i\}\+\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}a\_\{t,b\}\(\\mathbf\{x\}\)\\,\\mathbf\{v\}\_\{t,b\},\(12\)whereat,i​\(𝐱\)a\_\{t,i\}\(\\mathbf\{x\}\)andat,b​\(𝐱\)a\_\{t,b\}\(\\mathbf\{x\}\)are the attention weights at tokentt, and𝐯t,i,𝐯t,b∈ℝd\\mathbf\{v\}\_\{t,i\},\\mathbf\{v\}\_\{t,b\}\\in\\mathbb\{R\}^\{d\}are the corresponding value vectors\. Within this abstraction,

At​\(𝐱\)=∑i∈𝒦at,i​\(𝐱\)\.A\_\{t\}\(\\mathbf\{x\}\)=\\sum\_\{i\\in\\mathcal\{K\}\}a\_\{t,i\}\(\\mathbf\{x\}\)\.
###### Proposition 17\(Global distillation alone does not identify routing\)\.

Fix an answer\-interface tokentt\. Assume that there exist coefficients\(λb\)b∈ℬt\(\\lambda\_\{b\}\)\_\{b\\in\\mathcal\{B\}\_\{t\}\}with

λb≥0,∑b∈ℬtλb=1,\\lambda\_\{b\}\\geq 0,\\qquad\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}\\lambda\_\{b\}=1,such that

‖∑b∈ℬtλb​𝐯t,b−𝐡t⋆‖2≤ε0\.\\left\\\|\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}\\lambda\_\{b\}\\mathbf\{v\}\_\{t,b\}\-\\mathbf\{h\}\_\{t\}^\{\\star\}\\right\\\|\_\{2\}\\leq\\varepsilon\_\{0\}\.\(13\)Then for everyη∈\(0,1\)\\eta\\in\(0,1\), there exists a choice of attention weights at tokenttsuch that

At​\(𝐱\)=ηA\_\{t\}\(\\mathbf\{x\}\)=\\etaand

‖𝐡t​\(𝐱\)−𝐡t⋆‖2≤ε0\+Ct​η,\\\|\\mathbf\{h\}\_\{t\}\(\\mathbf\{x\}\)\-\\mathbf\{h\}\_\{t\}^\{\\star\}\\\|\_\{2\}\\leq\\varepsilon\_\{0\}\+C\_\{t\}\\eta,where

Ct=‖∑b∈ℬtλb​𝐯t,b‖2\+maxi∈𝒦⁡‖𝐯t,i‖2\.C\_\{t\}=\\left\\\|\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}\\lambda\_\{b\}\\mathbf\{v\}\_\{t,b\}\\right\\\|\_\{2\}\+\\max\_\{i\\in\\mathcal\{K\}\}\\\|\\mathbf\{v\}\_\{t,i\}\\\|\_\{2\}\.Consequently,ℒglobal\\mathcal\{L\}\_\{\\mathrm\{global\}\}alone does not imply any strictly positive lower bound on answer\-side attention to previous latent reasoning tokens\.

###### Proof\.

Define

𝐡¯t:=∑b∈ℬtλb​𝐯t,b\.\\bar\{\\mathbf\{h\}\}\_\{t\}:=\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}\\lambda\_\{b\}\\mathbf\{v\}\_\{t,b\}\.By \([13](https://arxiv.org/html/2607.18264#S9.E13)\),

‖𝐡¯t−𝐡t⋆‖2≤ε0\.\\\|\\bar\{\\mathbf\{h\}\}\_\{t\}\-\\mathbf\{h\}\_\{t\}^\{\\star\}\\\|\_\{2\}\\leq\\varepsilon\_\{0\}\.
Fix anyη∈\(0,1\)\\eta\\in\(0,1\)\. Choose the attention weights at tokenttby

at,i\(η\)​\(𝐱\):=η\|𝒦\|,i∈𝒦,a\_\{t,i\}^\{\(\\eta\)\}\(\\mathbf\{x\}\):=\\frac\{\\eta\}\{\|\\mathcal\{K\}\|\},\\qquad i\\in\\mathcal\{K\},and

at,b\(η\)​\(𝐱\):=\(1−η\)​λb,b∈ℬt\.a\_\{t,b\}^\{\(\\eta\)\}\(\\mathbf\{x\}\):=\(1\-\\eta\)\\lambda\_\{b\},\\qquad b\\in\\mathcal\{B\}\_\{t\}\.These weights are nonnegative and satisfy

∑i∈𝒦at,i\(η\)​\(𝐱\)\+∑b∈ℬtat,b\(η\)​\(𝐱\)=η\+\(1−η\)​∑b∈ℬtλb=1\.\\sum\_\{i\\in\\mathcal\{K\}\}a\_\{t,i\}^\{\(\\eta\)\}\(\\mathbf\{x\}\)\+\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}a\_\{t,b\}^\{\(\\eta\)\}\(\\mathbf\{x\}\)=\\eta\+\(1\-\\eta\)\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}\\lambda\_\{b\}=1\.Hence they define a valid attention distribution in the attention\-mixture abstraction\.

By construction, the total attention mass on previous latent reasoning tokens is exactly

At​\(𝐱\)=∑i∈𝒦at,i\(η\)​\(𝐱\)=η\.A\_\{t\}\(\\mathbf\{x\}\)=\\sum\_\{i\\in\\mathcal\{K\}\}a\_\{t,i\}^\{\(\\eta\)\}\(\\mathbf\{x\}\)=\\eta\.
Substituting the chosen weights into \([12](https://arxiv.org/html/2607.18264#S9.E12)\) gives

𝐡t\(η\)​\(𝐱\)=∑i∈𝒦η\|𝒦\|​𝐯t,i\+∑b∈ℬt\(1−η\)​λb​𝐯t,b\.\\mathbf\{h\}\_\{t\}^\{\(\\eta\)\}\(\\mathbf\{x\}\)=\\sum\_\{i\\in\\mathcal\{K\}\}\\frac\{\\eta\}\{\|\\mathcal\{K\}\|\}\\,\\mathbf\{v\}\_\{t,i\}\+\\sum\_\{b\\in\\mathcal\{B\}\_\{t\}\}\(1\-\\eta\)\\lambda\_\{b\}\\,\\mathbf\{v\}\_\{t,b\}\.Using the definition of𝐡¯t\\bar\{\\mathbf\{h\}\}\_\{t\}, we may rewrite this as

𝐡t\(η\)​\(𝐱\)=\(1−η\)​𝐡¯t\+η\|𝒦\|​∑i∈𝒦𝐯t,i\.\\mathbf\{h\}\_\{t\}^\{\(\\eta\)\}\(\\mathbf\{x\}\)=\(1\-\\eta\)\\bar\{\\mathbf\{h\}\}\_\{t\}\+\\frac\{\\eta\}\{\|\\mathcal\{K\}\|\}\\sum\_\{i\\in\\mathcal\{K\}\}\\mathbf\{v\}\_\{t,i\}\.Subtracting𝐡t⋆\\mathbf\{h\}\_\{t\}^\{\\star\}yields

𝐡t\(η\)​\(𝐱\)−𝐡t⋆=\(𝐡¯t−𝐡t⋆\)−η​𝐡¯t\+η\|𝒦\|​∑i∈𝒦𝐯t,i\.\\mathbf\{h\}\_\{t\}^\{\(\\eta\)\}\(\\mathbf\{x\}\)\-\\mathbf\{h\}\_\{t\}^\{\\star\}=\(\\bar\{\\mathbf\{h\}\}\_\{t\}\-\\mathbf\{h\}\_\{t\}^\{\\star\}\)\-\\eta\\,\\bar\{\\mathbf\{h\}\}\_\{t\}\+\\frac\{\\eta\}\{\|\\mathcal\{K\}\|\}\\sum\_\{i\\in\\mathcal\{K\}\}\\mathbf\{v\}\_\{t,i\}\.Taking norms and applying the triangle inequality,

‖𝐡t\(η\)​\(𝐱\)−𝐡t⋆‖2≤‖𝐡¯t−𝐡t⋆‖2\+η​‖𝐡¯t‖2\+‖η\|𝒦\|​∑i∈𝒦𝐯t,i‖2\.\\\|\\mathbf\{h\}\_\{t\}^\{\(\\eta\)\}\(\\mathbf\{x\}\)\-\\mathbf\{h\}\_\{t\}^\{\\star\}\\\|\_\{2\}\\leq\\\|\\bar\{\\mathbf\{h\}\}\_\{t\}\-\\mathbf\{h\}\_\{t\}^\{\\star\}\\\|\_\{2\}\+\\eta\\\|\\bar\{\\mathbf\{h\}\}\_\{t\}\\\|\_\{2\}\+\\left\\\|\\frac\{\\eta\}\{\|\\mathcal\{K\}\|\}\\sum\_\{i\\in\\mathcal\{K\}\}\\mathbf\{v\}\_\{t,i\}\\right\\\|\_\{2\}\.For the last term,

‖η\|𝒦\|​∑i∈𝒦𝐯t,i‖2≤η\|𝒦\|​∑i∈𝒦‖𝐯t,i‖2≤η​maxi∈𝒦⁡‖𝐯t,i‖2\.\\left\\\|\\frac\{\\eta\}\{\|\\mathcal\{K\}\|\}\\sum\_\{i\\in\\mathcal\{K\}\}\\mathbf\{v\}\_\{t,i\}\\right\\\|\_\{2\}\\leq\\frac\{\\eta\}\{\|\\mathcal\{K\}\|\}\\sum\_\{i\\in\\mathcal\{K\}\}\\\|\\mathbf\{v\}\_\{t,i\}\\\|\_\{2\}\\leq\\eta\\max\_\{i\\in\\mathcal\{K\}\}\\\|\\mathbf\{v\}\_\{t,i\}\\\|\_\{2\}\.Combining the preceding two displays with‖𝐡¯t−𝐡t⋆‖2≤ε0\\\|\\bar\{\\mathbf\{h\}\}\_\{t\}\-\\mathbf\{h\}\_\{t\}^\{\\star\}\\\|\_\{2\}\\leq\\varepsilon\_\{0\}, we obtain

‖𝐡t\(η\)​\(𝐱\)−𝐡t⋆‖2≤ε0\+η​\(‖𝐡¯t‖2\+maxi∈𝒦⁡‖𝐯t,i‖2\)\.\\\|\\mathbf\{h\}\_\{t\}^\{\(\\eta\)\}\(\\mathbf\{x\}\)\-\\mathbf\{h\}\_\{t\}^\{\\star\}\\\|\_\{2\}\\leq\\varepsilon\_\{0\}\+\\eta\\left\(\\\|\\bar\{\\mathbf\{h\}\}\_\{t\}\\\|\_\{2\}\+\\max\_\{i\\in\\mathcal\{K\}\}\\\|\\mathbf\{v\}\_\{t,i\}\\\|\_\{2\}\\right\)\.By the definition ofCtC\_\{t\}, this is

‖𝐡t\(η\)​\(𝐱\)−𝐡t⋆‖2≤ε0\+Ct​η,\\\|\\mathbf\{h\}\_\{t\}^\{\(\\eta\)\}\(\\mathbf\{x\}\)\-\\mathbf\{h\}\_\{t\}^\{\\star\}\\\|\_\{2\}\\leq\\varepsilon\_\{0\}\+C\_\{t\}\\eta,which proves the claim\. ∎

[Proposition˜17](https://arxiv.org/html/2607.18264#Thmtheorem17)says that matching the answer\-interface hidden state does not identify the routing pattern\. The same observation applies toℒCE\\mathcal\{L\}\_\{\\mathrm\{CE\}\}: if two routing patterns induce the same answer\-interface hidden state, then they incur the same answer loss\.

[Propositions˜16](https://arxiv.org/html/2607.18264#Thmtheorem16)and[17](https://arxiv.org/html/2607.18264#Thmtheorem17)separate what the local and global objectives control\. Local distillation yields a tokenwise lower bound on answer\-side attention exactly when the aligned targets retain a positive answer\-side margin relative to their local score drift\. By contrast, global supervision alone does not provide any analogous guarantee\. The endpoint hidden state can be matched while the attention mass on previous latent reasoning tokens is made arbitrarily small\.

## 10Benchmark details

### 10\.1MNNS task

The Minimum Non\-Negative Sum \(MNNS\) task\(Gozetenet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib18)\)takes as inputHHpositive integersa1,…,aHa\_\{1\},\\ldots,a\_\{H\}and asks for the minimum value of∑k=1Hσk​ak≥0\\sum\_\{k=1\}^\{H\}\\sigma\_\{k\}a\_\{k\}\\geq 0over all sign assignmentsσk∈\{−1,\+1\}\\sigma\_\{k\}\\in\\\{\-1,\+1\\\}\. This is equivalent to finding the partition of\{a1,…,aH\}\\\{a\_\{1\},\\ldots,a\_\{H\}\\\}into two subsets with minimal non\-negative difference, a variant of the subset\-sum problem\(Karp,[2009](https://arxiv.org/html/2607.18264#bib.bib71)\)\.

#### Graph construction\.

We define a layered directed graphG=\(𝒩,E\)G=\(\\mathcal\{N\},E\)with

𝒩=\{\(k,z\):k∈\{0,…,H\},z​is a partial sum reachable at depth​k\},\\mathcal\{N\}=\\\{\(k,z\):k\\in\\\{0,\\ldots,H\\\},\\;z\\text\{ is a partial sum reachable at depth \}k\\\},and edges\(\(k,z\),\(k\+1,z\+ak\+1\)\),\(\(k,z\),\(k\+1,z−ak\+1\)\)∈E\(\(k,z\),\(k\{\+\}1,z\+a\_\{k\+1\}\)\),\\;\(\(k,z\),\(k\{\+\}1,z\-a\_\{k\+1\}\)\)\\in E\. The source iss=\(0,0\)s=\(0,0\)\. At depthkk, the BFS frontierFkF\_\{k\}contains all partial\-sum states discovered for the first time, and the discovered setUkU\_\{k\}accumulates all states seen up to depthkk\. The answer is the minimum non\-negativezzsuch that\(H,z\)∈UH\(H,z\)\\in U\_\{H\}\.

#### Data and vocabulary\.

ForH=4H\{=\}4digits drawn from\{1,…,9\}\\\{1,\\ldots,9\\\}, the vocabulary consists of integers in\[−S,S\]\[\-S,S\]\(withSSchosen so all reachable partial sums are covered\) plus special tokens\. The input is formatted as⟨bos⟩​a1​a2​…​aH→\\langle\\textsc\{bos\}\\rangle\\;a\_\{1\}\\;a\_\{2\}\\;\\ldots\\;a\_\{H\}\\;\\to, and the output is the optimal sum value followed by⟨eos⟩\\langle\\textsc\{eos\}\\rangle\. Permutations of the same integer multiset are assigned to the same data split \(80%/20% train/val\) to prevent data leakage\.

#### Architecture\.

We use a 2\-layer, 2\-head GPT\-2 model with embedding dimensiond=32d\{=\}32, trained from scratch with AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2607.18264#bib.bib72)\)\(learning rate10−410^\{\-4\}, no weight decay\)\. Each of theH−1H\{\-\}1intermediate steps corresponds to one latent token; the final discrete token produces the answer\.

### 10\.2Game of 24 task

The Game of 24\(Yaoet al\.,[2023](https://arxiv.org/html/2607.18264#bib.bib6)\)is a classic arithmetic puzzle: given a set of numbers, the goal is to combine them using arithmetic operations to reach the target value 24\. We formulate a sequential variant that naturally maps to a layered reachability problem\.

#### Task formulation\.

GivenCCcards with values drawn from\{1,…,D\}\\\{1,\\ldots,D\\\}and an operator set𝒪\\mathcal\{O\}, the model must determine whether the target value 24 is reachable by processing the cards strictly left to right\. Starting with the first card as accumulator, at each stepk∈\{1,…,C−1\}k\\in\\\{1,\\ldots,C\{\-\}1\\\}one applies an operation∘k∈𝒪\\circ\_\{k\}\\in\\mathcal\{O\}to produceAk=Ak−1∘kdk\+1A\_\{k\}=A\_\{k\-1\}\\circ\_\{k\}d\_\{k\+1\}, wheredk\+1d\_\{k\+1\}is the\(k\+1\)\(k\{\+\}1\)\-th card\. The answer isy=𝟏​\[24∈AC−1\]y=\\mathbf\{1\}\[24\\in A\_\{C\-1\}\], whereAC−1A\_\{C\-1\}is the set of accumulated values reachable after allCCcards over all operation sequences\.

#### Graph construction\.

This defines a layered directed graphG=\(𝒩,E\)G=\(\\mathcal\{N\},E\)\. The node set at depthkkconsists of all intermediate values reachable after incorporatingk\+1k\{\+\}1cards:

𝒩k=\{v:v​is an accumulated value reachable at step​k\}\.\\mathcal\{N\}\_\{k\}=\\\{v:v\\text\{ is an accumulated value reachable at step \}k\\\}\.Edges connect each nodev∈𝒩kv\\in\\mathcal\{N\}\_\{k\}to nodes\{v∘dk\+2:∘∈𝒪\}∩𝒩k\+1\\\{v\\circ d\_\{k\+2\}:\\circ\\in\\mathcal\{O\}\\\}\\cap\\mathcal\{N\}\_\{k\+1\}, and the source iss=d1s=d\_\{1\}\. At each step, the BFS frontierFkF\_\{k\}records the set of newly discovered accumulated values, so the uniform multiplexed target𝗆𝗎𝗑​\(𝐫k\)\\mathsf\{mux\}\(\\mathbf\{r\}\_\{k\}\)is the distribution over the current frontier\.

#### Configuration\.

We useC=5C\{=\}5cards, digit range\{1,…,5\}\\\{1,\\ldots,5\\\}, and operator set𝒪=\{\+,−,×\}\\mathcal\{O\}=\\\{\+,\-,\\times\\\}\. The dataset is balanced \(50% reachable, 50% unreachable\)\. We assign all permutations of the same card multiset to the same split\. Because order affects the left\-to\-right process, this split prevents memorization of card multisets while still evaluating order\-sensitive reasoning\. Training uses 3 random seeds\.

#### Architecture\.

We use the same 2\-layer, 2\-head GPT\-2 model withd=32d\{=\}32as for MNNS\. Each of theC−1=4C\{\-\}1=4fold steps corresponds to one latent token; the final discrete token produces the YES/NO answer\.

## 11Method and training details

### 11\.1Implementation details

[Tables˜8](https://arxiv.org/html/2607.18264#S11.T8)and[9](https://arxiv.org/html/2607.18264#S11.T9)summarize the hyperparameters used for allMUXexperiments\. We follow the same experimental protocol as CODI\(Shenet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib13)\)\. All models were trained using a single H100 GPU with 96 GB of VRAM\. Experiments with GPT\-2 and LLaMA 1B took around 24 hours, while the experiments with larger backbones \(LLaMA 3B/8B\) ran for 2–3 days to complete\.

Table 8:LoRA adapter configuration \(shared across all models\)\.HyperparameterValueLoRA rankrr128LoRA alpha32LoRA dropout0\.1Table 9:Training hyperparameters forMUX\. Method\-specific parameters are listed in the top block; standard optimization settings are in the bottom block\.GPT\-2LLaMA 3\.2 1BLLaMA 3\.2 3BLLaMA 3\.1 8BHyperparameterAugNLAugNLAugAugMethod\-specificContinuous tokensKK666666Weighting functionsin\.sin\.geo\.sin\.geo\.geo\.Decay rateρ\\rho\(geo\.\)——0\.9—0\.90\.9Positional scaleλ\\lambda\(sin\.\)1\.01\.0—1\.0——Temperatureτ\\tau1\.01\.01\.01\.01\.01\.0Chunking strategyrand\.rand\.rand\.rand\.rand\.rand\.Local loss weightβ\\beta1\.01\.01\.01\.01\.01\.0Global distill\. weightγ\\gamma1\.01\.020\.020\.020\.020\.0Answer loss weight1\.01\.01\.01\.01\.01\.0Ref\. answer loss weight1\.01\.01\.01\.01\.01\.0Projection dim7687682048204830724096Layer\-wise std norm✓✓✓✓✓✓OptimizationOptimizerAdamWLR schedulercosineWarmup ratio0\.03Effective batch size128Learning rate3e\-33e\-38e\-48e\-43e\-41e\-4Weight decay0\.010\.010\.10\.10\.10\.1Gradient clipping1\.01\.02\.02\.02\.02\.0Epochs4040101086#### MUX∗\(K=24K\{=\}24\) configuration\.

The parallel\-decoding variantMUX∗reported in[Table˜1](https://arxiv.org/html/2607.18264#S5.T1)usesK=24K\{=\}24latent tokens generated viaT=3T\{=\}3Jacobi iterations\. We use uniform token chunking\. Geometric positional weighting is applied with decay0\.90\.9\. The local loss weight isβ=1\.0\\beta\{=\}1\.0and the global distillation weight isγ=20\.0\\gamma\{=\}20\.0\. Optimization settings match the LLaMA 3\.2 1B column of[Table˜9](https://arxiv.org/html/2607.18264#S11.T9)\.

### 11\.2Details of probing for positional weighting

In[Section˜5\.3](https://arxiv.org/html/2607.18264#S5.SS3), to study the role of positional weighting, we trained an MLP probe on discrete reasoning spans𝐫i=\(ri1,…,riSi\)\\mathbf\{r\}\_\{i\}=\(r\_\{i\}^\{1\},\\dots,r\_\{i\}^\{S\_\{i\}\}\)\. For each non\-empty span𝐫i\\mathbf\{r\}\_\{i\}, we construct the same multiplexed target as in \([2](https://arxiv.org/html/2607.18264#S3.E2)\)\.

The probe takes𝗆𝗎𝗑​\(𝐫i\)\\mathsf\{mux\}\(\\mathbf\{r\}\_\{i\}\)as input and predicts the original span𝐫i\\mathbf\{r\}\_\{i\}\. We use a 5\-layer MLP with hidden sizes1024,512,256,512,10241024,512,256,512,1024and GELU\(Hendrycks and Gimpel,[2016](https://arxiv.org/html/2607.18264#bib.bib38)\)activations, without input normalization\. We set the maximum sequence length to 128, strip the delimiters<<and\>\>from each extracted step, and train for 20 epochs with batch size 128 and learning rate10−310^\{\-3\}\. The data are extracted from GSM8K\-AUG and split into 901,661 training spans and 100,185 evaluation spans using a 0\.1 test split\. For geometric weighting, we useρ=0\.9\\rho=0\.9\. For sinusoidal weighting, we useτ=1\\tau=1\. For rotary weighting, we usebase=1000\\mathrm\{base\}=1000\.

### 11\.3Details of span\-level alignments

The main text only assumes an order\-preserving alignment between theMMdiscrete reasoning spans and theKKlatent\-token slots\. Let\(𝐫~1,…,𝐫~M\)\(\\tilde\{\\mathbf\{r\}\}\_\{1\},\\ldots,\\tilde\{\\mathbf\{r\}\}\_\{M\}\)denote the aligned spans used for local supervision, where each non\-empty𝐫~i\\tilde\{\\mathbf\{r\}\}\_\{i\}is a contiguous block of the original reasoning spans and the original order is preserved\. WhenM≤KM\\leq K, all alignment variants reduce to the same prefix assignment:𝐫~i=𝐫i\\tilde\{\\mathbf\{r\}\}\_\{i\}=\\mathbf\{r\}\_\{i\}fori≤Mi\\leq M, and the remainingK−MK\-Mslots are left empty\. The differences arise only in the overfull regimeM\>KM\>K, which we summarize below\.

#### No chunking\.

Assign one span to each latent token until one of the two sequences ends\. Equivalently,𝐫~i=𝐫i\\tilde\{\\mathbf\{r\}\}\_\{i\}=\\mathbf\{r\}\_\{i\}fori≤min⁡\(M,K\)i\\leq\\min\(M,K\)\. IfM\>KM\>K, the remainingM−KM\-Kspans are discarded, so this is lossless only whenM≤KM\\leq K\.

#### Deterministic chunking\.

WhenM\>KM\>K, partition theMMreasoning spans intoKKcontiguous groups with roughly equal sizes\. WritingM=q​K\+rM=qK\+rwith0≤r<K0\\leq r<K, the firstK−rK\-rgroups have sizeqqand the lastrrgroups have sizeq\+1q\+1\. Equivalently, group sizes differ by at most one, with extra spans assigned to later groups\.

#### Random chunking\.

WhenM\>KM\>K, sampleK−1K\-1cut points uniformly without replacement from\{1,…,M−1\}\\\{1,\\dots,M\-1\\\}, sort them, and use the induced intervals to formKKpositive contiguous groups\. This yields a random monotone partition of theMMspans intoKKchunks\. This is the default variant used in our main experiments\. Randomness is resampled during training, so the model sees multiple valid local segmentations of the same reasoning trace without changing the answer target\.

## 12Limitations and broader impact

#### Limitations\.

Our losslessness guarantees are stated under exact arithmetic\. In finite precision, very long spans or large vocabularies may approach the separation boundary analyzed in[Section˜9\.2](https://arxiv.org/html/2607.18264#S9.SS2), though the analysis confirms that losslessness is preserved in practical regimes \(standard span lengths and float32 precision\)\. Our empirical evaluation focuses on mathematical reasoning and parallel search tasks\. This is the established evaluation setting adopted by prior latent reasoning methods\(Shenet al\.,[2025](https://arxiv.org/html/2607.18264#bib.bib13); Weiet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib15); Kuzinaet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib16); Gozetenet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib18)\)\. ExtendingMUXto broader reasoning domains such as multi\-hop question answering, code generation, and open\-ended planning is a natural next step\.

#### Broader impact\.

Compressing reasoning into fewer latent tokens can reduce inference cost\. A concern with latent reasoning is reduced interpretability\. Users cannot easily audit intermediate steps\(Kuzinaet al\.,[2026](https://arxiv.org/html/2607.18264#bib.bib16)\)\.MUXpartially addresses this, since each latent token can be decoded into human\-readable content through the LM head\. Still, decoded tokens are approximate, not verbatim reasoning, so users should not treat them as ground truth\. More broadly, more efficient reasoning inherits the risks of the underlying models, which can produce incorrect, biased, or overconfident outputs\.