INTACT: Inducing Latent Reasoning on a Frozen Backbone

# INTACT: Inducing Latent Reasoning on a Frozen Backbone
Source: [https://arxiv.org/html/2610.19568](https://arxiv.org/html/2610.19568)
Atula TejaswiHaris VikaloSujay SanghaviAffiliation:The University of Texas at Austin

###### Abstract

Latent reasoning replaces natural\-language chains of thought with compact continuous “soft\-token” representations, reducing the number of intermediate tokens that must be decoded autoregressively\. We use a small external encoder to induce latent reasoning in a frozen backbone model\. The encoder iteratively maps the backbone’s hidden states to new latent blocks, constructing a latent reasoning prefix, after which the frozen model writes its remaining reasoning and its answer as text\. Our method, INTACT, trains this encoder through distillation to reproduce the effect of written reasoning steps while keeping the model’s behavior close to that of the frozen backbone, and then reinforces it on answer correctness\. Across four frozen backbones from 1B to 8B, INTACT improves both GSM8K and average out\-of\-domain accuracy, including GSM8K gains from75\.8%75\.8\\%to79\.3%79\.3\\%onLlama\-3\.2\-3Band from77\.6%77\.6\\%to79\.7%79\.7\\%onQwen2\.5\-3B\. It also reduces end\-to\-end latency per problem by19%19\\%onLlama\-3\.2\-3Band outperforms the evaluated latent\-reasoning baselines at the 3B and 8B scales\. Beyond the mathematics setting, the same recipe trained on a multi\-domain corpus improves the frozen backbone on twelve evaluation benchmarks spanning mathematics, science, and commonsense, without task\-specific adaptation\.

## 1Introduction

Figure 1:Accuracy and total positions across three reasoning domains\.A single INTACT encoder is evaluated on mathematics, science and commonsense benchmarks\. INTACT and Nemo\-SFT are both trained on the Llama\-Nemotron mixture\. INTACT trains only the encoder and leavesLlama\-3\.2\-3Bfrozen, whereas Nemo\-SFT updates the language model via supervised fine\-tuning\.[Section4\.5](https://arxiv.org/html/2610.19568#S4.SS5)describes the datasets and evaluation setup\.Language models are increasingly applied to complex problems across domains such as mathematics, code, and commonsense reasoning, where the answer follows from a sequence of dependent steps rather than from recall\([Cobbe et al\., 2021](https://arxiv.org/html/2610.19568#bib.bib30);[Hendrycks et al\., 2021b](https://arxiv.org/html/2610.19568#bib.bib31);[Jain et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib33)\)\. In such settings, chain\-of\-thought prompting draws out the intermediate steps in natural language and substantially improves accuracy\([Wei et al\., 2022](https://arxiv.org/html/2610.19568#bib.bib2);[Kojima et al\., 2022](https://arxiv.org/html/2610.19568#bib.bib3);[Wang et al\., 2023](https://arxiv.org/html/2610.19568#bib.bib5)\)\. That accuracy is paid for in the output sequence, since every intermediate token must be decoded autoregressively before the answer appears, and the cost grows with reasoning depth\([Chen et al\., 2025b](https://arxiv.org/html/2610.19568#bib.bib9);[Snell et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib6)\)\.

*Latent reasoning*proposes to move part of this computation out of language space, replacing some or all of the written steps with continuous vectors that the model consumes directly\([Hao et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib14);[Shen et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib16)\)\. These vectors may provide compact representations of longer textual traces, and recent work has learned such representations by distilling from an explicit reasoner\([Shen et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib16);[Wei et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib36)\), by multiplexing several reasoning threads into one sequence\([Suleymanzade et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib35)\), or by looping a transformer over its own latent state\([Fan et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib1);[Saunshi et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib20)\)\.

Yet most latent\-reasoning methods update the backbone alongside the latent mechanism\([Hao et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib14);[Shen et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib16);[Fan et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib1);[Suleymanzade et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib35)\)\. On capable backbones, such updates risk eroding existing reasoning abilities, including those needed for tasks beyond the fine\-tuning data\([Xu et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib29)\)\. Freezing the backbone leaves the original model intact and recoverable, but places the burden on the latent interface to produce inputs the frozen model can use\. One response to this constraint is SoftCoT, which derives soft thoughts from the question with a small assistant\([Xu et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib29)\)\. We instead condition a small encoder on the backbone’s hidden states over the question and preceding latent blocks\. This leads to our central question:*Can we induce latent reasoning on a frozen backbone, improving both its accuracy and its efficiency?*

![Refer to caption](https://arxiv.org/html/2610.19568v1/fig_tacit_stage1.png)Figure 2:Stage 1, progressive substitution via distillation\.The language modelMMis frozen and only the encoderfθf\_\{\\theta\}is trained\. At depthii, the encoder produces𝐯i\\mathbf\{v\}\_\{i\}fromMM’s states over𝐱\\mathbf\{x\}and the preceding blocks𝐯1:i−1\\mathbf\{v\}\_\{1:i\-1\}, completing the student prefix\[𝐱,𝐯1:i\]\[\\mathbf\{x\},\\mathbf\{v\}\_\{1:i\}\]against the teacher’s\[𝐱,𝐳1:i\]\[\\mathbf\{x\},\\mathbf\{z\}\_\{1:i\}\]\. The substitution lossℒsub\\mathcal\{L\}\_\{\\mathrm\{sub\}\}distillspTp\_\{T\}intopSp\_\{S\}over the shared continuation𝐮i\\mathbf\{u\}\_\{i\}, without directly aligning𝐯j\\mathbf\{v\}\_\{j\}and𝐳j\\mathbf\{z\}\_\{j\}\.Stage 2, latent\-policy refinement, not shown, then refines the same encoder for answer correctness using group\-relative policy gradients, leavingMMfrozen\.We proposeINTACT, illustrated in[Figure2](https://arxiv.org/html/2610.19568#S1.F2)\. An encoder reads the frozen language model’s final\-layer states and producesKKlatent blocks ofccvectors each \(K=4K=4andc=4c=4in our experiments\)\. During training, the firstjjblocks learn to replace the firstjjreasoning steps\. Blockjjis generated from the question representation together with the frozen model’s representation of blocks1,…,j−11,\\ldots,j\-1\. At inference, the resultingK​cKcvectors are inserted after the question in place of the corresponding initial textual reasoning, after which the frozen model generates a solution in its own words\.

Training proceeds in two stages\. Stage 1 trains each block to reproduce the effect of the written reasoning step it replaces, while a preservation term keeps the latent\-conditioned model’s behavior close to the frozen model’s own\. Stage 2 then refines the encoder against answer correctness using group\-relative advantages\. Across four backbones from 1B to 8B, INTACT improves upon the frozen backbone while reducing the number of generated tokens, and it surpasses the evaluated latent\-reasoning methods at the 3B and 8B scales\. We summarize our contributions as follows:

1. 1\.Frozen\-backbone latent inference\.INTACT constructs a latent prefix sequentially inKKblocks\. Each block is produced from the frozen model’s hidden states over the question and preceding blocks, thereby conditioning the encoder on the model’s representation of the prefix constructed so far\. Once the prefix is complete, the frozen model generates its textual reasoning and answer without further encoder calls\.
2. 2\.Two\-stage encoder training\.Progressive substitution distills the effect of successive written reasoning steps into latent blocks, while on\-policy preservation constrains their effect on generated continuations\. A second stage refines the encoder on answer correctness using group\-relative policy gradients\.
3. 3\.Accuracy with fewer tokens\.Under*mathematics training on GSM8K\-Aug*, INTACT improves frozen\-model accuracy while generating fewer tokens across four Llama and Qwen backbones from 1B to 8B\. It also outperforms the evaluated latent\-reasoning baselines at the 3B and 8B scales \([Section4\.2](https://arxiv.org/html/2610.19568#S4.SS2)\)\.
4. 4\.Multi\-domain transfer\.Under*multi\-domain training on Llama\-Nemotron*, INTACT improves accuracy and generates fewer tokens across twelve evaluation benchmarks without benchmark\-specific adaptation, including commonsense, a domain absent from training \([Figure1](https://arxiv.org/html/2610.19568#S1.F1),[Section4\.5](https://arxiv.org/html/2610.19568#S4.SS5)\)\.

## 2Related Work

#### Chain\-of\-thought and test\-time scaling\.

Prompting or training models to produce intermediate steps before answering improves accuracy on hard problems\([Wei et al\., 2022](https://arxiv.org/html/2610.19568#bib.bib2);[Kojima et al\., 2022](https://arxiv.org/html/2610.19568#bib.bib3)\), and the gains grow with the language\-space computation spent, whether by aggregating sampled chains\([Wang et al\., 2023](https://arxiv.org/html/2610.19568#bib.bib5)\), scaling test\-time compute\([Snell et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib6);[Muennighoff et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib7)\), or reinforcement learning that elicits extended traces\([DeepSeek\-AI et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib8)\)\. The cost of these tokens has in turn motivated work on overthinking and reasoning efficiency\([Chen et al\., 2025b](https://arxiv.org/html/2610.19568#bib.bib9)\)\.

#### Latent and continuous reasoning\.

A complementary line moves reasoning into non\-linguistic computation\. Pause and filler tokens add computation without content\([Goyal et al\., 2024](https://arxiv.org/html/2610.19568#bib.bib10);[Pfau et al\., 2024](https://arxiv.org/html/2610.19568#bib.bib11)\), and implicit\-CoT methods distill the trace away entirely\([Deng et al\., 2023](https://arxiv.org/html/2610.19568#bib.bib12);[Deng et al\., 2024](https://arxiv.org/html/2610.19568#bib.bib13)\)\. Coconut feeds last hidden states back as input embeddings, unrolling a chain of continuous thoughts\([Hao et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib14)\), which may encode superpositions of parallel search paths\([Zhu et al\., 2025a](https://arxiv.org/html/2610.19568#bib.bib15)\)\. CODI compresses the chain by self\-distillation\([Shen et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib16)\)and SIM\-CoT adds step\-level supervision\([Wei et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib36)\), while Soft Thinking replaces sampled tokens with probability\-weighted mixtures at inference\([Zhang et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib17)\), although the benefits of this training\-free approach have been contested\([Wu et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib18)\)\. Others make continuous chains trainable with reinforcement learning\([Butt et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib19)\), move the reasoning loop inside the network\([Saunshi et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib20);[Geiping et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib21)\), or pretrain with continuous concepts\([Tack et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib22)\)\.[Zhu et al\. \(2025b\)](https://arxiv.org/html/2610.19568#bib.bib23)survey the area\. Many of these methods update the backbone, and most stop the model from writing its own reasoning\.

#### Compression and frozen\-backbone latents\.

Text\-space compression shortens prompts and traces while keeping the medium fixed\([Jiang et al\., 2023](https://arxiv.org/html/2610.19568#bib.bib26)\)\. Compressed chain\-of\-thought generates dense contemplation tokens as trace surrogates\([Cheng and Van Durme, 2024](https://arxiv.org/html/2610.19568#bib.bib27)\), Token Assorted mixes discrete latent codes with text\([Su et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib28)\), and latent diffusion methods denoise reasoning steps in a learned space that is co\-trained for noise tolerance\([Kang et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib25);[Li et al\., 2022](https://arxiv.org/html/2610.19568#bib.bib24)\)\. SoftCoT prepends soft tokens from a small assistant model to a frozen backbone and still generates language\-space reasoning afterwards\([Xu et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib29)\)\.[AppendixA](https://arxiv.org/html/2610.19568#A1)compares INTACT with individual prior methods\.

## 3Method

#### Notation\.

LetMMbe a language model with vocabulary𝒱\\mathcal\{V\}and input\-embedding dimensiondMd\_\{M\}, and let𝒱∗\\mathcal\{V\}^\{\\ast\}denote sequences of tokens in𝒱\\mathcal\{V\}\. For a token string𝐱∈𝒱∗\\mathbf\{x\}\\in\\mathcal\{V\}^\{\\ast\}, letE⁡\(𝐱\)E\(\\mathbf\{x\}\)denote the corresponding sequence of token embeddings\. We consider inputs toMMthat mix a token sequence𝐱\\mathbf\{x\}with a sequence𝐯\\mathbf\{v\}of non\-token vectors, and we write such inputs in embedding space, so that the concatenation of the two is\[E⁡\(𝐱\),𝐯\]\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\]\. We writeH⁡\(⋅\)H\(\\cdot\)for the final\-layer hidden statesMMproduces on a mixed input of this kind, so𝐡=H⁡\(\[E⁡\(𝐱\),𝐯\]\)\\mathbf\{h\}=H\(\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\]\)denotes its states over the whole sequence\. Where the embedding map is clear from context we abbreviate\[E⁡\(𝐱\),𝐯\]\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\]as\[𝐱,𝐯\]\[\\mathbf\{x\},\\mathbf\{v\}\], and we write conditioning on such an input aspM\(⋅∣𝐱,𝐯\)p\_\{M\}\(\\cdot\\mid\\mathbf\{x\},\\mathbf\{v\}\)\. A training example is a triple\(𝐱,𝐳1:S,𝐲\)\(\\mathbf\{x\},\\mathbf\{z\}\_\{1:S\},\\mathbf\{y\}\), where𝐱\\mathbf\{x\}is a question,𝐳1:S=\(𝐳1,…,𝐳S\)\\mathbf\{z\}\_\{1:S\}=\(\\mathbf\{z\}\_\{1\},\\ldots,\\mathbf\{z\}\_\{S\}\)is a reference reasoning trace ofSSsteps with𝐳i∈𝒱ni\\mathbf\{z\}\_\{i\}\\in\\mathcal\{V\}^\{n\_\{i\}\}, and𝐲\\mathbf\{y\}is the final answer\. Without a latent prefix, the unmodified backbone generates a continuation𝐲^∼pM\(⋅∣𝐱\)\\hat\{\\mathbf\{y\}\}\\sim p\_\{M\}\(\\cdot\\mid\\mathbf\{x\}\)containing its own reasoning and a final answerAns⁡\(𝐲^\)\\operatorname\{Ans\}\(\\hat\{\\mathbf\{y\}\}\), requiring an autoregressive decoding step for every token in it\.[Table9](https://arxiv.org/html/2610.19568#A2.T9)collects the remaining symbols\.

### 3\.1Latent Encoderfθf\_\{\\theta\}

The latent encoderfθf\_\{\\theta\}maps a sequence of hidden states𝐡\\mathbf\{h\}to a block ofcccontinuous vectors\. Vectors within a block are generated jointly in a single encoder pass, whereas blocks are constructed sequentially\. To produce blockjj, the encoder reads the frozen model’s final\-layer states over the question and the preceding latent blocks:

𝐯j=fθ\(j\)\(H\(\[E\(𝐱\),𝐯1:j−1\]\)\)∈ℝc×dM\.\\mathbf\{v\}\_\{j\}=f\_\{\\theta\}^\{\(j\)\}\\bigl\(H\(\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\_\{1:j\-1\}\]\)\\bigr\)\\in\\mathbb\{R\}^\{c\\times d\_\{M\}\}\.Forj=1j=1, the input consists only of the question states,H⁡\(E⁡\(𝐱\)\)H\(E\(\\mathbf\{x\}\)\)\. After each block is appended, the frozen model processes the extended prefix, providing the hidden states used to generate the next block\. This recurrence is mediated by the frozen model’s representations of the preceding blocks, rather than by feeding their raw vectors directly back to the encoder\. The encoder therefore builds the latent prefix from representations supplied byMM, without learning a separate question encoder\.

*Example\.*For a6060\-token question, the frozen model supplies the statesH⁡\(E⁡\(𝐱\)\)H\(E\(\\mathbf\{x\}\)\)to the encoder, which returns the first block𝐯1=fθ\(1\)​\(H⁡\(E⁡\(𝐱\)\)\)\\mathbf\{v\}\_\{1\}=f^\{\(1\)\}\_\{\\theta\}\(H\(E\(\\mathbf\{x\}\)\)\)ofc=4c=4vectors\. These are appended to the input, so the model reads\[E⁡\(𝐱\),𝐯1\]\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\_\{1\}\]and producesH⁡\(\[E⁡\(𝐱\),𝐯1\]\)H\(\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\_\{1\}\]\)\. The encoder reads the sequence of6464hidden states to produce𝐯2=fθ\(2\)​\(H⁡\(\[E⁡\(𝐱\),𝐯1\]\)\)\\mathbf\{v\}\_\{2\}=f^\{\(2\)\}\_\{\\theta\}\(H\(\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\_\{1\}\]\)\), and so on:Prefix positions:60→\+𝐯164→\+𝐯268→\+𝐯372→\+𝐯476\.\\text\{Prefix positions:\}\\quad 60\\xrightarrow\{\+\\mathbf\{v\}\_\{1\}\}64\\xrightarrow\{\+\\mathbf\{v\}\_\{2\}\}68\\xrightarrow\{\+\\mathbf\{v\}\_\{3\}\}72\\xrightarrow\{\+\\mathbf\{v\}\_\{4\}\}76\.AfterK=4K=4blocks the prefix holds7676positions,6060question tokens and1616latent vectors, and the model generates its reasoning and answer with no further encoder calls\.

#### Encoder architecture\.

The encoder projects the context fromdMd\_\{M\}to a hidden dimension ofd=1024d=1024\. It then applies six Transformer layers, each consisting of cross\-attention from theccqueries to the context, self\-attention among the queries, and a feed\-forward network, with pre\-normalization and residual connections throughout\. A final linear layer projects each output todMd\_\{M\}\. Each output vector is normalized and rescaled by a shared learned scalar\. The encoder has107107M parameters\. All parameters are shared across blocks except for the queriesqjq\_\{j\}\.

#### Inference\.

Given a question𝐱\\mathbf\{x\}, the encoder is invokedKKtimes, after whichMMgenerates its reasoning and answer autoregressively from\[E\(𝐱\),𝐯1:K\]\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\_\{1:K\}\]without further encoder calls\.[Algorithm1](https://arxiv.org/html/2610.19568#alg1)states the procedure\.

Algorithm 1INTACT inference\. The language modelMMis frozen and the encoderfθf\_\{\\theta\}is fixed\.1:question

𝐱\\mathbf\{x\}; frozen LM

MM; trained encoder

fθf\_\{\\theta\};

K=4K\{=\}4blocks of

c=4c\{=\}4vectors

2:

H⁡\(⋅\)H\(\\cdot\)denotes the final\-layer states

MMproduces on an embedding\-space input, as in[Section3](https://arxiv.org/html/2610.19568#S3)

3:for

j=1,…,Kj=1,\\dots,Kdo

4:

𝐯j←fθ\(j\)\(H\(\[E\(𝐱\),𝐯1:j−1\]\)\)\\mathbf\{v\}\_\{j\}\\leftarrow f\_\{\\theta\}^\{\(j\)\}\\bigl\(H\(\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\_\{1:j\-1\}\]\)\\bigr\)⊳\\trianglerightone block per call,ccvectors at a time

5:endfor

6:

𝐲^∼pM\(⋅∣\[E\(𝐱\),𝐯1:K\]\)\\hat\{\\mathbf\{y\}\}\\sim p\_\{M\}\\bigl\(\\cdot\\mid\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\_\{1:K\}\]\\bigr\)⊳\\trianglerightno further encoder calls

7:return

Ans⁡\(𝐲^\)\\operatorname\{Ans\}\(\\hat\{\\mathbf\{y\}\}\)

### 3\.2Training

Training proceeds in two stages, neither of which updates the language model\. Every parameter ofMMremains fixed, and although gradients pass through it, onlyθ\\thetais updated, which separates learning the latent interface from relearning the reasoning behavior of the backbone\. Stage 1 trains through distillation to replace written reasoning steps with latent blocks, and Stage 2 then refines the same encoder against answer correctness\.[Algorithm2](https://arxiv.org/html/2610.19568#alg2)summarizes both stages\.

#### Stage 1: progressive substitution\.

Stage 1 trains successive latent blocks to reproduce the effect of corresponding written reasoning steps\. At substitution depthii, the*teacher context*contains the firstiireference steps,CiT=\[𝐱,𝐳1:i\]C\_\{i\}^\{\\mathrm\{T\}\}=\[\\mathbf\{x\},\\mathbf\{z\}\_\{1:i\}\], while the*student context*replaces them with latent blocks,CiS=\[𝐱,𝐯1:i\]C\_\{i\}^\{\\mathrm\{S\}\}=\[\\mathbf\{x\},\\mathbf\{v\}\_\{1:i\}\]\. Both are followed by the same continuation𝐮i=\[𝐳i\+1:S,𝐲\]\\mathbf\{u\}\_\{i\}=\[\\mathbf\{z\}\_\{i\+1:S\},\\mathbf\{y\}\]\. For two contextsAAandBB, we define the continuation\-level behavioral divergence

D\(A∥B;𝐮\)=1\|𝐮\|∑tKL\(pM\(⋅∣A,𝐮<t\)∥pM\(⋅∣B,𝐮<t\)\)\.D\(A\\\|B;\\mathbf\{u\}\)=\\frac\{1\}\{\|\\mathbf\{u\}\|\}\\sum\_\{t\}\\operatorname\{KL\}\\\!\\left\(p\_\{M\}\(\\cdot\\mid A,\\mathbf\{u\}\_\{<t\}\)\\,\\\|\\,p\_\{M\}\(\\cdot\\mid B,\\mathbf\{u\}\_\{<t\}\)\\right\)\.The substitution loss at depthiiisℒsub\(i\)=D\(CiT∥CiS;𝐮i\)\\smash\{\\mathcal\{L\}\_\{\\mathrm\{sub\}\}^\{\(i\)\}\}=D\(\\smash\{C\_\{i\}^\{\\mathrm\{T\}\}\}\\\|\\smash\{C\_\{i\}^\{\\mathrm\{S\}\}\};\\mathbf\{u\}\_\{i\}\), self\-distillation through one frozen model, with the reference steps supplying the behavior to reproduce and both distributions evaluated under teacher forcing on the shared continuation\. The written steps occupy more positions than the latent blocks that replace them, so we shift the latent span to keep the continuation at the same positions in both\.[AppendixC\.1](https://arxiv.org/html/2610.19568#A3.SS1)gives the position assignment and the stages it applies to\. We optimize depthsi=1,…,Ki=1,\\ldots,Kin order, using the substitution loss only for examples withS\>iS\>i, and take one encoder update at each depth\. After every update,𝐯1:i\\mathbf\{v\}\_\{1:i\}is regenerated with the new parameters before constructing depthi\+1i\+1\. Gradients pass through the preceding blocks and the fixed language model, but only the shared encoder parametersθ\\thetaare updated\.

#### On\-policy preservation\.

Becauseℒsub\\mathcal\{L\}\_\{\\mathrm\{sub\}\}is teacher\-forced, it cannot see failures that appear only when the model generates freely\. We therefore sample a rollout under the complete prefix,𝐲^∼pM\(⋅∣𝐱,𝐯1:K\)\\hat\{\\mathbf\{y\}\}\\sim p\_\{M\}\(\\cdot\\mid\\mathbf\{x\},\\mathbf\{v\}\_\{1:K\}\), and penalize the intervention’s effect along it withℒpre=D\(𝐱∥\[𝐱,𝐯1:K\];𝐲^\)\\mathcal\{L\}\_\{\\mathrm\{pre\}\}=D\(\\mathbf\{x\}\\\|\[\\mathbf\{x\},\\mathbf\{v\}\_\{1:K\}\];\\hat\{\\mathbf\{y\}\}\)\. The sampled continuation is held fixed, and the unsteered model supplies the target distribution, regularizing the latent\-conditioned model on prefixes encountered during free generation\. Preservation applies only ati=Ki=K, so the Stage 1 loss of[Algorithm2](https://arxiv.org/html/2610.19568#alg2)isℒS1\(i\)=ℒsub\(i\)\+𝟏\[i=K\]βℒpre\\smash\{\\mathcal\{L\}\_\{\\mathrm\{S1\}\}^\{\(i\)\}\}=\\smash\{\\mathcal\{L\}\_\{\\mathrm\{sub\}\}^\{\(i\)\}\}\+\\mathbf\{1\}\[i=K\]\\,\\beta\\mathcal\{L\}\_\{\\mathrm\{pre\}\}\.

#### Stage 2: latent\-policy refinement\.

Stage 1 learns from the behavior induced by reference traces\. Stage 2 instead refines the encoder against answer correctness\. The encoder and frozen model together defineπθ\(𝐲^∣𝐱\)=pM\(𝐲^∣𝐱,𝐯1:K\)\\pi\_\{\\theta\}\(\\hat\{\\mathbf\{y\}\}\\mid\\mathbf\{x\}\)=p\_\{M\}\(\\hat\{\\mathbf\{y\}\}\\mid\\mathbf\{x\},\\mathbf\{v\}\_\{1:K\}\)with theKKblocks𝐯1:K\\mathbf\{v\}\_\{1:K\}built sequentially by the encoderfθf\_\{\\theta\}, which alone is updated\. For each question, we sampleGGcompletions𝐲^1,…,𝐲^G\\hat\{\\mathbf\{y\}\}^\{1\},\\ldots,\\hat\{\\mathbf\{y\}\}^\{G\}and assign the binary correctness rewardrg=R⁡\(𝐲^g,𝐲\)r^\{g\}=R\(\\hat\{\\mathbf\{y\}\}^\{g\},\\mathbf\{y\}\), whereRRindicates whether the extracted answer matches the reference under the answer\-comparison protocol of[AppendixC\.4](https://arxiv.org/html/2610.19568#A3.SS4)\. We standardize rewards within the group to obtainAg=\(rg−r¯\)/\(σr\+ϵ\)A^\{g\}=\(r^\{g\}\-\\bar\{r\}\)/\(\\sigma\_\{r\}\+\\epsilon\)and minimize the per\-sequence\-normalized policy loss

ℒRL=−1G∑g=1GAg1\|𝐲^g\|∑tlogpM\(y^tg∣𝐱,𝐯1:K,𝐲^<tg\)\.\\mathcal\{L\}\_\{\\mathrm\{RL\}\}=\-\\frac\{1\}\{G\}\\sum\_\{g=1\}^\{G\}A^\{g\}\\frac\{1\}\{\|\\hat\{\\mathbf\{y\}\}^\{g\}\|\}\\sum\_\{t\}\\log p\_\{M\}\(\\hat\{y\}\_\{t\}^\{g\}\\mid\\mathbf\{x\},\\mathbf\{v\}\_\{1:K\},\\hat\{\\mathbf\{y\}\}\_\{<t\}^\{g\}\)\.Groups in which every completion receives the same reward have zero advantage and make no update\. The Stage 2 block of[Algorithm2](https://arxiv.org/html/2610.19568#alg2)summarizes this update\.

Algorithm 2INTACT training\. The language modelMMis frozen throughout; onlyθ\\thetais learned\.1:example

\(𝐱,𝐳1:S,𝐲\)\(\\mathbf\{x\},\\mathbf\{z\}\_\{1:S\},\\mathbf\{y\}\); frozen LM

MM; encoder

fθf\_\{\\theta\};

K=4K\{=\}4blocks of

c=4c\{=\}4vectors

2:

H⁡\(⋅\)H\(\\cdot\)denotes the final\-layer states

MMproduces on an embedding\-space input, as in[Section3](https://arxiv.org/html/2610.19568#S3)

3:

D\(A∥B;𝐮\)=1\|𝐮\|∑tKL\(pM\(⋅\|A,𝐮<t\)∥pM\(⋅\|B,𝐮<t\)\)D\(A\\\|B;\\mathbf\{u\}\)\\\!=\\\!\\frac\{1\}\{\|\\mathbf\{u\}\|\}\\sum\_\{t\}\\mathrm\{KL\}\\bigl\(p\_\{M\}\(\\cdot\|A,\\mathbf\{u\}\_\{<t\}\)\\\|p\_\{M\}\(\\cdot\|B,\\mathbf\{u\}\_\{<t\}\)\\bigr\)
4:Stage 1: distill the chain into latents

5:forminibatch, then

i=1,…,Ki=1,\\dots,Kdo⊳\\trianglerightℒsub\\mathcal\{L\}\_\{\\mathrm\{sub\}\}uses examples withS\>iS\>i

6:

𝐯j←fθ\(j\)\(H\(\[E\(𝐱\),𝐯1:j−1\]\)\)\\mathbf\{v\}\_\{j\}\\leftarrow f\_\{\\theta\}^\{\(j\)\}\\bigl\(H\(\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\_\{1:j\-1\}\]\)\\bigr\),

j=1,…,ij=1,\\dots,i
7:

ℒ←D\(\[𝐱,𝐳1:i\]∥\[𝐱,𝐯1:i\];\[𝐳i\+1:S,𝐲\]\)\\mathcal\{L\}\\leftarrow D\\bigl\(\[\\mathbf\{x\},\\mathbf\{z\}\_\{1:i\}\]\\,\\\|\\,\[\\mathbf\{x\},\\mathbf\{v\}\_\{1:i\}\]\\,;\\,\[\\mathbf\{z\}\_\{i\+1:S\},\\mathbf\{y\}\]\\bigr\)⊳\\trianglerightℒsub\\mathcal\{L\}\_\{\\mathrm\{sub\}\}:iilatent blocks replaceiitext steps

8:if

i=Ki=Kthen

9:

𝐲^∼pM\(⋅∣𝐱,𝐯1:K\)\\hat\{\\mathbf\{y\}\}\\sim p\_\{M\}\(\\cdot\\mid\\mathbf\{x\},\\mathbf\{v\}\_\{1:K\}\);

ℒ←ℒ\+βD\(𝐱∥\[𝐱,𝐯1:K\];𝐲^\)\\mathcal\{L\}\\leftarrow\\mathcal\{L\}\+\\beta\\,D\\bigl\(\\mathbf\{x\}\\\|\[\\mathbf\{x\},\\mathbf\{v\}\_\{1:K\}\];\\hat\{\\mathbf\{y\}\}\\bigr\)⊳\\trianglerightℒpre\\mathcal\{L\}\_\{\\mathrm\{pre\}\}: match unsteeredMM

10:endif

11:

θ←θ−η​∇θℒ\\theta\\leftarrow\\theta\-\\eta\\nabla\_\{\\theta\}\\mathcal\{L\}⊳\\trianglerightbackprop through fixedMM; updateθ\\thetaonly

12:endfor

13:Stage 2: reinforce with answer reward

14:forminibatchdo

15:

𝐯j←fθ\(j\)\(H\(\[E\(𝐱\),𝐯1:j−1\]\)\)\\mathbf\{v\}\_\{j\}\\leftarrow f\_\{\\theta\}^\{\(j\)\}\\bigl\(H\(\[E\(\\mathbf\{x\}\),\\mathbf\{v\}\_\{1:j\-1\}\]\)\\bigr\),

j=1,…,Kj=1,\\dots,K; sample

𝐲^1,…,𝐲^G∼pM\(⋅∣𝐱,𝐯1:K\)\\hat\{\\mathbf\{y\}\}^\{1\},\\dots,\\hat\{\\mathbf\{y\}\}^\{G\}\\sim p\_\{M\}\(\\cdot\\mid\\mathbf\{x\},\\mathbf\{v\}\_\{1:K\}\)
16:

rg←R⁡\(𝐲^g,𝐲\)r^\{g\}\\leftarrow R\(\\hat\{\\mathbf\{y\}\}^\{g\},\\mathbf\{y\}\),

Ag←\(rg−r¯\)/\(σr\+ϵ\)A^\{g\}\\leftarrow\(r^\{g\}\-\\bar\{r\}\)/\(\\sigma\_\{r\}\+\\epsilon\)⊳\\trianglerightgroup\-relative advantage

17:

θ←θ\+η1G∑gAg\|𝐲^g\|∑t∇θlogpM\(y^tg∣𝐱,𝐯1:K,𝐲^<tg\)\\theta\\leftarrow\\theta\+\\eta\\tfrac\{1\}\{G\}\\sum\_\{g\}\\tfrac\{A^\{g\}\}\{\|\\hat\{\\mathbf\{y\}\}^\{g\}\|\}\\sum\_\{t\}\\nabla\_\{\\theta\}\\log p\_\{M\}\\bigl\(\\hat\{y\}^\{g\}\_\{t\}\\mid\\mathbf\{x\},\\mathbf\{v\}\_\{1:K\},\\hat\{\\mathbf\{y\}\}^\{g\}\_\{<t\}\\bigr\)⊳\\trianglerightper\-sequence mean

18:endfor

## 4Experiments

### 4\.1Setup

#### INTACT variants\.

We consider two training settings, on separate corpora and never mixed\. Each encoder reads one backbone’s hidden states and is trained for that backbone alone\.

- •Mathematics training on GSM8K\-Augfollows prior latent\-reasoning work and uses385385K examples\([Deng et al\., 2023](https://arxiv.org/html/2610.19568#bib.bib12)\)whose solutions are sequences of compact arithmetic steps\. It is trained separately for each of the four backbones and evaluated in domain on the GSM8K test set\([Cobbe et al\., 2021](https://arxiv.org/html/2610.19568#bib.bib30)\)and out of domain on SVAMP, GSM\-Hard and MultiArith, over which we report an unweighted macro\-average\.
- •Multi\-domain training on Llama\-Nemotronuses the Llama\-Nemotron post\-training corpus\([Bercovich et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib41)\), approximately64%64\\%mathematics,21%21\\%science and14%14\\%code\. It is trained once, onLlama\-3\.2\-3B\-Instruct, and evaluated on twelve held\-out mathematics, science and commonsense benchmarks\.

The four backbones areLlama\-3\.2\-1B\-Instruct,Llama\-3\.2\-3B\-Instruct,Llama\-3\.1\-8B\-Instruct\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.19568#bib.bib37)\), and the baseQwen2\.5\-3B\([Yang et al\., 2024](https://arxiv.org/html/2610.19568#bib.bib34)\)\. INTACT usesK=4K=4blocks ofc=4c=4vectors in both settings, with the same encoder architecture and objectives throughout and the language model frozen\.[AppendixC\.2](https://arxiv.org/html/2610.19568#A3.SS2)gives the diverse\-domain corpus selection and segmentation into steps, and[AppendixE](https://arxiv.org/html/2610.19568#A5)the twelve evaluation benchmarks\.

#### Baselines\.

We compare against the unchangedFrozen LM, and against three supervised baselines that update the language model and so measure what the same supervision achieves when changing the backbone is allowed\.No\-CoTtrains on the answer alone\.Aug\-SFTtrains on GSM8K\-Aug, whose steps are compact symbolic traces, primarily calculator expressions\.Nemo\-SFTinstead trains on the Llama\-Nemotron mixture, whose steps are complete natural\-language worked solutions\. We further compare withCODI\([Shen et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib16)\),LOTUS\([Fan et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib1)\), andMUX\([Suleymanzade et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib35)\), which replace written reasoning with continuous or recurrent computation but likewise update the language model\.[AppendixC\.3](https://arxiv.org/html/2610.19568#A3.SS3)describes how each baseline was trained or, where a result is quoted, which table it is taken from\.

#### Evaluation\.

We evaluate on full test splits with generation caps of768768tokens formathematicsand2,0482\{,\}048for themulti\-domainsetting, whose natural\-language solutions are longer\. We report answer accuracy and mean generated tokens\. Wherever total token use is shown, INTACT is also charged for its1616latent positions\. Accuracy for INTACT is averaged over three encoder seeds, with±\\pmgiving the sample standard deviation where it is shown\. Token counts are measured on the single seed that initializes Stage 2, so the two stages stay comparable\. Ablations and controls use one encoder seed\. Quoted baselines retain their source protocols\.[AppendixC\.4](https://arxiv.org/html/2610.19568#A3.SS4)specifies the prompts, decoding parameters, and answer extraction\.

### 4\.2Main Results

Table 1:Accuracy \(%\) at a 768\-token generation cap across four backbones\. Both INTACT stages are trained on GSM8K\-Aug\.*OOD*is the average on SVAMP, GSM\-Hard and MultiArith\.#### Against the frozen backbone\.

[Table1](https://arxiv.org/html/2610.19568#S4.T1)reports accuracy at the 768\-token cap, where both stages improve GSM8K and out\-of\-domain accuracy over the frozen backbone on all four models\. Stage 2 further improves six of the eight Stage 1 scores\. It is0\.30\.3points lower on 3B out\-of\-domain accuracy and0\.10\.1points lower on 8B GSM8K\. The largest Stage 1 gains occur at 8B, where the bare\-prompt frozen model often continues generating after producing its answer\. To separate this formatting issue from reasoning accuracy, we also evaluate the frozen model with its native chat template\. This restores reliable termination and raises its accuracy to78\.478\.4, but leaves it below both stages of INTACT\.[Table12](https://arxiv.org/html/2610.19568#A4.T12)reports the comparison\. The size of the gain varies widely across backbones\. Relative to the frozen backbone, Stage 2 adds11\.311\.3points on GSM8K at 8B and2\.12\.1onQwen2\.5\-3B, the strongest frozen baseline in the table\. At 1B the Stage 1 encoder is flat on GSM8K \(44\.244\.2against44\.044\.0\) while gaining3\.53\.5out of domain\.[Appendix Table11](https://arxiv.org/html/2610.19568#A4.T11)reports the complete budget sweep and seed variation\.

Table 2:Accuracy \(%\) on four mathematics benchmarks\. Our Frozen, SFT, and INTACT evaluations use a 768\-token cap\. CODI, MUX, and LOTUS use their source protocols, including reproductions\.†and‡mark results quoted from MUX\([Suleymanzade et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib35)\)and LOTUS\([Fan et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib1)\), respectively\.*OOD*is the mean of SVAMP, GSM\-Hard, and MultiArith\. Shading marks INTACT, with±\\pmdenoting standard deviation over three encoder seeds\. Best and second\-best latent methods areboldandunderlined\. A dash marks a configuration for which no comparable trained checkpoint is available at that scale\.
#### Comparison with latent methods\.

[Table2](https://arxiv.org/html/2610.19568#S4.T2)reports accuracy on the four backbones\. AtLlama\-3\.2\-1B, INTACT remains below Aug\-SFT and the latent baselines, reaching48\.748\.7on GSM8K and54\.554\.5out of domain, compared with61\.661\.6and60\.560\.5for Aug\-SFT\. This ordering reverses at 3B, where INTACT reaches79\.379\.3and68\.868\.8, outperforming the strongest published latent baseline by9\.39\.3and4\.74\.7points and Aug\-SFT by7\.87\.8and6\.76\.7points\. The advantage over Aug\-SFT persists at 8B \(83\.6/72\.183\.6/72\.1versus74\.2/67\.074\.2/67\.0\) and onQwen2\.5\-3B\(79\.7/75\.179\.7/75\.1versus74\.8/69\.574\.8/69\.5\), showing that it extends beyond the Llama family\. The margin over the latent baselines therefore widens with scale, from−8\.6\-8\.6on GSM8K at 1B to\+9\.3\+9\.3at 3B and\+18\.2\+18\.2at 8B\. Aug\-SFT is strongest at 1B, whereas Nemo\-SFT leads on GSM8K and the out\-of\-domain average at 3B and above, exceeding Stage 2 by2\.12\.1–3\.73\.7and1\.31\.3–1\.71\.7points, respectively\. Thus, INTACT does not uniformly outperform supervised fine\-tuning\. It improves the frozen model and achieves the strongest GSM8K and average results among the evaluated latent methods at 3B and above, without modifying the language model\.

### 4\.3Inference Efficiency

Figure 3:GSM8K accuracy versus total positions onLlama\-3\.2\-3B, with INTACT trained on GSM8K\-Aug\. Frozen, INTACT, and Nemo\-SFT use a 768\-token cap\. CODI, MUX, and LOTUS follow their source protocols\. Counts include latent positions where applicable\.#### Accuracy–generation cost\.

We measure cost in total positions, defined as generated tokens plus INTACT’s1616latent positions, excluding the question\.[Figure3](https://arxiv.org/html/2610.19568#S4.F3)compares operating points onLlama\-3\.2\-3B\. INTACT reaches79\.379\.3accuracy at211211positions, compared with75\.875\.8at261261for the frozen backbone\. Aug\-SFT, CODI, and MUX occupy more compressed operating points, but reach only71\.571\.5,60\.860\.8, and65\.065\.0accuracy, respectively\. Unlike CODI and MUX, INTACT retains a textual reasoning continuation, and unlike all three it improves the accuracy–efficiency trade\-off without updating the backbone\.

#### Effect of the two stages\.

[Table3](https://arxiv.org/html/2610.19568#S4.T3)reports generated tokens across backbones\. Both stages shorten generation in every case, and which contributes more depends on the backbone\. The effect is particularly pronounced at 8B, where generation falls from754754tokens for the frozen backbone to269269after Stage 1 and182182after Stage 2\. This shortening does not generally come at the expense of accuracy\. Stage 2 improves six of the eight in\-domain and out\-of\-domain measurements reported in[Table1](https://arxiv.org/html/2610.19568#S4.T1)\.

#### End\-to\-end latency\.

Token counts provide an implementation\-independent measure of generation cost, but do not directly determine wall\-clock time\. We therefore measure latency on the full GSM8K test split usingLlama\-3\.2\-3Bat batch size one on one H100 NVL\. As[Table4](https://arxiv.org/html/2610.19568#S4.T4)shows, INTACT reduces mean latency from4,1804\{,\}180to3,3863\{,\}386ms per problem, a19%19\\%reduction that includes latent\-prefix construction\. Constructing the prefix costs127\.5127\.5ms, while the longer context raises measured decoding cost from16\.0316\.03to16\.6716\.67ms per generated token\. Emitting6565fewer tokens more than offsets these costs, showing that the reduction in generation length translates into a measurable, though not proportional, reduction in latency\. Both systems generate under the same bare prompt\.[Table12](https://arxiv.org/html/2610.19568#A4.T12)reports how the frozen backbone’s token cost changes under its chat template\.

Table 3:Mean generated tokens at a768768\-token cap, for encoders trained on GSM8K\-Aug\. Both stages shorten generation on every backbone, in domain and out, and Stage 2 is shorter than Stage 1 in every row\.*OOD*is the average over SVAMP, GSM\-Hard and MultiArith\.Table 4:Mean end\-to\-end latency per GSM8K problem onLlama\-3\.2\-3B, measured over the full test split at batch 1 on one H100 NVL\. The saving comes from emitting fewer tokens, not from faster decoding\.

### 4\.4Ablations

Each ablation isolates one design decision and holds the rest of the recipe fixed\. The preservation and capacity ablations useQwen2\.5\-3B, while the substitution\-depth sweep and the architecture and prefix controls of[Table5](https://arxiv.org/html/2610.19568#S4.T5)useLlama\-3\.1\-8B\. All are scored under the protocol of[Section4\.1](https://arxiv.org/html/2610.19568#S4.SS1)with a768768\-token cap\.

GSM8K↑\\uparrowOOD↑\\uparrowTok\.↓\\downarrowFrozen backbone77\.674\.6273*Preservation anchoruu*None \(β=0\\beta=0\)42\.954\.580Gold chain74\.069\.1240Frozen rollout76\.276\.2309Student rollout78\.875\.5246*Block widthcc\(atK=4K\{=\}4\)*c=1c=1\(44pos\.\)73\.772\.7207c=2c=2\(88pos\.\)78\.076\.4285c=4c=4\(1616pos\.\)78\.875\.5246c=8c=8\(3232pos\.\)73\.772\.2316

Figure 4:Left:ablations onQwen2\.5\-3B\. Each block varies one design decision, and shaded rows indicate settings used by INTACT\.Right:substitution depth sweep onLlama\-3\.1\-8B\. At depthiionly the firstiiofKKblocks are supplied, soi=0i=0is the frozen backbone andi=Ki=Kthe full Stage 1 prefix\. Both panels use one encoder seed\.*OOD*is the macro\-average over the three held\-out benchmarks\.#### Preservation anchor\.

Becauseℒsub\\mathcal\{L\}\_\{\\mathrm\{sub\}\}is scored on a reference continuation, it does not constrain the model on its own free generations\. The variants below differ only in the continuation𝐮\\mathbf\{u\}on whichℒpre​\(𝐮\)\\mathcal\{L\}\_\{\\mathrm\{pre\}\}\(\\mathbf\{u\}\)is scored, and we compare omitting the term with three choices of𝐮\\mathbf\{u\}\.

- •Gold chain\.𝐮=\[𝐳1:S,𝐲\]\\mathbf\{u\}=\[\\mathbf\{z\}\_\{1:S\},\\mathbf\{y\}\], the written trace itself, so𝐯1:K\\mathbf\{v\}\_\{1:K\}is scored while the model is given the very steps it replaces\.
- •Frozen rollout\.𝐮∼pM\(⋅∣𝐱\)\\mathbf\{u\}\\sim p\_\{M\}\(\\cdot\\mid\\mathbf\{x\}\), free generation from the backbone without the latent prefix\.
- •Student rollout\.𝐮∼pM\(⋅∣𝐱,𝐯1:K\)\\mathbf\{u\}\\sim p\_\{M\}\(\\cdot\\mid\\mathbf\{x\},\\mathbf\{v\}\_\{1:K\}\), free generation from the latent\-conditioned model, the continuations it actually produces\.

In[Figure4](https://arxiv.org/html/2610.19568#S4.F4)\(left\), omitting preservation collapses the output to8080tokens, so its apparent efficiency is degraded generation rather than compression\. The gold chain stabilizes training but stays below the backbone on both accuracy measures\. A frozen rollout removes that teacher forcing and gives the best out\-of\-domain accuracy of the four anchors, but remains off\-policy for the latent\-conditioned model and lengthens generation to309309tokens\. Only the student rollout improves on the backbone in accuracy and in length at once\.

#### Substitution depth\.

Stage 1 supervises every available depth, so each block is learned in the context created by its predecessors\. The sweep holds the encoder fixed and varies only the number of blocks supplied at inference\. OnLlama\-3\.1\-8B\([Figure4](https://arxiv.org/html/2610.19568#S4.F4), right\), increasing the prefix fromi=0i=0toi=Ki=Kraises GSM8K accuracy by12\.112\.1points and the out\-of\-domain average by18\.218\.2, while reducing generation from754754to269269tokens on GSM8K and from745745to207207on the out\-of\-domain average\. Accuracy therefore depends on the depth supplied, not only on the presence of a latent prefix\.

#### Latent capacity\.

The available substitution depth ismin⁡\(K,S−1\)\\min\(K,S\-1\), and only8%8\\%of GSM8K\-Aug chains support five steps\. We therefore fixK=4K=4and vary the block widthcc, giving4​c4clatent positions\. Performance is non\-monotonic:c=1c=1provides insufficient capacity, whilec=8c=8reduces accuracy and lengthens generation\. Among the narrower variants,c=1c=1saves tokens at the cost of accuracy, whereasc=2c=2improves accuracy but generates more tokens than the backbone\. Onlyc=4c=4improves on the frozen backbone while also generating fewer tokens, providing the best accuracy–efficiency balance\.

Table 5:Architecture and prefix controls onLlama\-3\.1\-8B\.Architecture variants retrain the encoder with the latent blocks constructed differently\. Prefix interventions instead modify a trained Stage 1 prefix only at inference\.*OOD*averages SVAMP, GSM\-Hard and MultiArith\. Each trained arm uses one encoder seed\.GSM8KOODAcc\.↑\\uparrowTokens↓\\downarrowAcc\.↑\\uparrowTokens↓\\downarrow*Reference systems*Frozen backbone72\.375453\.9745INTACT Stage 184\.426972\.1207INTACT Stage 283\.718272\.8155*Architecture variants \(retrained\)*Independently built blocks82\.476769\.9761Jointly built blocks82\.052270\.1598*Prefix interventions \(Stage 1, inference only\)*Within\-question block shuffle77\.043867\.3401Cross\-question prefix70\.951364\.5480
#### Encoder construction\.

[Table5](https://arxiv.org/html/2610.19568#S4.T5)isolates two components of the encoder architecture\. Building the four blocks jointly, in a single encoder call, so that they can coordinate with each other but receive no feedback from the blocks already placed, reduces GSM8K accuracy from84\.484\.4to82\.082\.0and raises generation from269269to522522tokens, reaching the cap on52\.5%52\.5\\%of problems\. This variant keeps the four progressive substitution depths and differs from its reference only in building the blocks together\.[AppendixC\.1](https://arxiv.org/html/2610.19568#A3.SS1)records what is held fixed\. Building the blocks independently, so that every block uses only the backbone’s hidden states over the question and not those induced by preceding latent blocks, leaves accuracy at82\.482\.4but increases generation from269269to767767tokens, comparable to the frozen backbone’s754754\. Both variants retain most of the accuracy of the full construction\. The clearer difference is in generation length, where the full construction uses269269tokens against767767and522522, so it reaches higher accuracy while generating substantially less\.

#### Prefix order and specificity\.

[Table5](https://arxiv.org/html/2610.19568#S4.T5)also tests the trained Stage 1 prefix through inference\-only interventions\. Shuffling its blocks reduces GSM8K and OOD accuracy from84\.4/72\.184\.4/72\.1to77\.0/67\.377\.0/67\.3, while replacing it with a prefix generated for another question reduces them to70\.9/64\.570\.9/64\.5\. Both interventions also lengthen generation substantially\. The prefix is therefore sensitive to both block order and the source question, rather than acting as a generic learned prompt\.

### 4\.5One Encoder Across Diverse Reasoning Domains

Figure 5:One encoder across twelve benchmarks\.Accuracy onLlama\-3\.2\-3Bacross mathematics, science, and commonsense\. Each benchmark line connects the frozen backbone to Stage 2, with Stage 1 marking the intermediate result\. Both Nemo\-SFT and INTACT are trained on the Llama\-Nemotron mixture\. The same INTACT encoder is used on every benchmark\.#### Setup\.

We next test whether a single latent encoder can improve reasoning across domains beyond GSM8K\-Aug\. We train on the mathematics, science, and code solutions in Llama\-Nemotron and evaluate the same encoder, without benchmark\-specific adaptation, on twelve mathematics, science, and commonsense benchmarks\. Commonsense evaluation tests transfer beyond the domains explicitly represented in training\. Every system in this setting is scored at the2,0482\{,\}048\-token budget of[AppendixC\.4](https://arxiv.org/html/2610.19568#A3.SS4), which the long natural\-language solutions of this mixture require\. All systems in this setting see the same prompt\.

#### Results\.

Table 6:Conditioned encoder versus a learned constant\-prefix control in the diverse\-domain setting onLlama\-3\.2\-3B\. Results use the full test splits\.*Math*,*Science*and*Comm\.*are macro\-averages over four, five and three benchmarks respectively\.[Figure5](https://arxiv.org/html/2610.19568#S4.F5)shows that one encoder improves the frozen backbone on all twelve benchmarks\. Gains are largest in science and commonsense, reaching22\.522\.5points on ARC\-Challenge and23\.823\.8on HellaSwag, while mathematics gains range from0\.40\.4to3\.73\.7points\. The commonsense result is particularly notable because that domain is absent from training\. The improvements are largest where the frozen backbone is weakest, with the exception of GSM\-Hard, on which no system improves substantially\. Compared with Nemo\-SFT, INTACT is nearly tied on science \(1\.01\.0point lower\) and trails on mathematics \(2\.52\.5points\), but leads on commonsense \(6\.26\.2points\) in domain\-average accuracy, while leaving the backbone frozen\.

#### Constant\-prefix control\.

To test whether the gains arise from a generic learned prompt, we replace the encoder outputs withK​cKclearned vectors shared across all questions and train them with the same objective\. This prefix can learn common formatting and termination behavior but cannot encode question\-specific information\.[Table6](https://arxiv.org/html/2610.19568#S4.T6)reports it after each stage, with the checkpoint for each selected by the same procedure as INTACT’s\. The conditioned encoder leads it on science by9\.29\.2points after Stage 1 and9\.09\.0after Stage 2, and on commonsense by4\.44\.4and13\.113\.1, while the two score similarly on mathematics at both stages\. Conditioning therefore helps where the gain over the frozen backbone is largest, and its commonsense advantage widens with Stage 2 rather than narrowing\. The mathematics tie is not evidence that the prefix carries nothing question\-specific there\. Replacing a trained prefix with one generated for a different question reduces GSM8K accuracy to70\.970\.9onLlama\-3\.1\-8B, below the frozen backbone’s72\.372\.3\([Table5](https://arxiv.org/html/2610.19568#S4.T5)\)\.

#### Extraction and truncation\.

We test whether the gains reflect answer\-extraction failures or truncation of the frozen backbone\. Answer extraction succeeds for99\.299\.2–100%100\\%of frozen generations on all benchmarks except ECQA \(89\.3%89\.3\\%\)\. Some generations reach the token cap, but truncation rarely prevents scoring\. HellaSwag exhibits the most truncation, yet99\.2%99\.2\\%of its generations still yield an extractable answer\. These artifacts therefore do not explain the overall gains, although extraction failures may contribute on ECQA\.

## 5Conclusion

We studied whether latent reasoning can improve a capable language model without updating its backbone\. INTACT trains a small encoder that reads the frozen model’s own hidden states to reproduce the effect of early written reasoning steps, first through distillation and then through latent\-policy refinement, while the language model remains frozen\. Across four backbones, it improves accuracy and reduces generation relative to the frozen model, and it outperforms the evaluated latent\-reasoning baselines on larger backbones\. One encoder trained on mathematics, science, and code also improves the same frozen backbone on held\-out mathematics, science, and commonsense benchmarks without task\-specific adaptation\. Together, these results show that latent reasoning can complement a capable model’s existing reasoning, rather than depend on rewriting its weights or maximizing compression\.

#### Limitations\.

One limitation is that the latent blocks cannot be directly interpreted as the written steps they replace\. Another is that the backbone still decodes autoregressively, so shorter generations do not necessarily produce proportional wall\-clock savings \([Section4\.3](https://arxiv.org/html/2610.19568#S4.SS3)\)\. INTACT also still retains language\-space reasoning, so it remains more expensive than methods that compress reasoning into a handful of tokens\.

## References

- Aggarwalet al\.\(2021\)S\. Aggarwal, D\. Mandowara, V\. Agrawal, D\. Khandelwal, P\. Singla, and D\. GargExplanations for CommonsenseQA: new dataset and models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 3050–3065\.Cited by:[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.11.2)\.
- Bercovichet al\.\(2025\)A\. Bercovich, I\. Levy, I\. Golan,et al\.Llama\-nemotron: efficient reasoning models\.arXiv preprint arXiv:2505\.00949\.Cited by:[§E\.1](https://arxiv.org/html/2610.19568#A5.SS1.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2610.19568#S4.I1.i2.p1.1)\.
- Buttet al\.\(2025\)N\. Butt, A\. Kwiatkowski, I\. Labiad, J\. Kempe, and Y\. OllivierSoft tokens, hard truths\.arXiv preprint arXiv:2509\.19170\.Cited by:[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2025a\)X\. Chen, A\. Zhao, H\. Xia, X\. Lu, H\. Wang, Y\. Chen, W\. Zhang, J\. Wang, W\. Li, and X\. ShenReasoning beyond language: a comprehensive survey on latent chain\-of\-thought reasoning\.arXiv preprint arXiv:2505\.16782\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1)\.
- Chenet al\.\(2025b\)X\. Chen, J\. Xu, T\. Liang, Z\. He, J\. Pang, D\. Yu, L\. Song, Q\. Liu, M\. Zhou, Z\. Zhang, R\. Wang, Z\. Tu, H\. Mi, and D\. YuDo not think that much for 2\+3=? On the overthinking of long reasoning models\.InInternational Conference on Machine Learning,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.19568#S1.p1.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px1.p1.1)\.
- Cheng and Van Durme \(2024\)J\. Cheng and B\. Van DurmeCompressed chain of thought: efficient reasoning through dense representations\.arXiv preprint arXiv:2412\.13171\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p2.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px3.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.7.1),[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.8.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.2.2),[§1](https://arxiv.org/html/2610.19568#S1.p1.1),[1st item](https://arxiv.org/html/2610.19568#S4.I1.i1.p1.1)\.
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-AI, D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.Nature645,pp\. 633–638\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p1.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px1.p1.1)\.
- Denget al\.\(2024\)Y\. Deng, Y\. Choi, and S\. ShieberFrom explicit CoT to implicit CoT: learning to internalize CoT step by step\.arXiv preprint arXiv:2405\.14838\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p2.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Denget al\.\(2023\)Y\. Deng, K\. Prasad, R\. Fernandez, P\. Smolensky, V\. Chaudhary, and S\. ShieberImplicit chain of thought reasoning via knowledge distillation\.arXiv preprint arXiv:2311\.01460\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p2.1),[§E\.1](https://arxiv.org/html/2610.19568#A5.SS1.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1),[1st item](https://arxiv.org/html/2610.19568#S4.I1.i1.p1.1)\.
- Fanet al\.\(2026\)Y\. Fan, A\. Svete, and K\. LeeBridging the gap between latent and explicit reasoning with looped transformers\.arXiv preprint arXiv:2606\.31779\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1),[Table 7](https://arxiv.org/html/2610.19568#A1.T7.2.1.5.1),[§C\.3](https://arxiv.org/html/2610.19568#A3.SS3.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.19568#S1.p2.1),[§1](https://arxiv.org/html/2610.19568#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.19568#S4.SS1.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2610.19568#S4.T2)\.
- Gaoet al\.\(2023\)L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. NeubigPAL: program\-aided language models\.InInternational Conference on Machine Learning,PMLR, Vol\.202,pp\. 10764–10799\.Cited by:[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.3.1)\.
- Geipinget al\.\(2025\)J\. Geiping, S\. McLeish, N\. Jain, J\. Kirchenbauer, S\. Singh, B\. R\. Bartoldson, B\. Kailkhura, A\. Bhatele, and T\. GoldsteinScaling up test\-time compute with latent reasoning: a recurrent depth approach\.InAdvances in Neural Information Processing Systems,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Goyalet al\.\(2024\)S\. Goyal, Z\. Ji, A\. S\. Rawat, A\. K\. Menon, S\. Kumar, and V\. NagarajanThink before you speak: training language models with pause tokens\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p2.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri,et al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2610.19568#S4.SS1.SSS0.Px1.p1.2)\.
- Haoet al\.\(2025\)S\. Hao, S\. Sukhbaatar, D\. Su, X\. Li, Z\. Hu, J\. Weston, and Y\. TianTraining large language models to reason in a continuous latent space\.InConference on Language Modeling,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1),[Table 7](https://arxiv.org/html/2610.19568#A1.T7.2.1.2.1),[§1](https://arxiv.org/html/2610.19568#S1.p2.1),[§1](https://arxiv.org/html/2610.19568#S1.p3.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Hendryckset al\.\(2021a\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,Cited by:[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.10.1)\.
- Hendryckset al\.\(2021b\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§1](https://arxiv.org/html/2610.19568#S1.p1.1)\.
- Jainet al\.\(2025\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.19568#S1.p1.1)\.
- Jianget al\.\(2023\)H\. Jiang, Q\. Wu, C\. Lin, Y\. Yang, and L\. QiuLLMLingua: compressing prompts for accelerated inference of large language models\.InConference on Empirical Methods in Natural Language Processing,pp\. 13358–13376\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p1.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px3.p1.1)\.
- Kanget al\.\(2026\)H\. Kang, Y\. Zhang, N\. L\. Kuang, N\. Majamaki, N\. Jaitly, Y\. Ma, and L\. QinLaDiR: latent diffusion enhances LLMs for text reasoning\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px3.p1.1)\.
- Kojimaet al\.\(2022\)T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. IwasawaLarge language models are zero\-shot reasoners\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 22199–22213\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.19568#S1.p1.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px1.p1.1)\.
- Kuzinaet al\.\(2026\)A\. Kuzina, M\. Pióro, and B\. Ehteshami BejnordiKaVa: latent reasoning via compressed KV\-cache distillation\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1)\.
- Lesteret al\.\(2021\)B\. Lester, R\. Al\-Rfou, and N\. ConstantThe power of scale for parameter\-efficient prompt tuning\.InConference on Empirical Methods in Natural Language Processing,pp\. 3045–3059\.Cited by:[§A\.2](https://arxiv.org/html/2610.19568#A1.SS2.p1.1)\.
- Liet al\.\(2022\)X\. Li, J\. Thickstun, I\. Gulrajani, P\. S\. Liang, and T\. B\. HashimotoDiffusion\-LM improves controllable text generation\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 4328–4343\.Cited by:[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px3.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p1.1)\.
- Mihaylovet al\.\(2018\)T\. Mihaylov, P\. Clark, T\. Khot, and A\. SabharwalCan a suit of armor conduct electricity? a new dataset for open book question answering\.InConference on Empirical Methods in Natural Language Processing,pp\. 2381–2391\.Cited by:[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.6.2)\.
- Muennighoffet al\.\(2025\)N\. Muennighoff, Z\. Yang, W\. Shi, X\. L\. Li, L\. Fei\-Fei, H\. Hajishirzi, L\. Zettlemoyer, P\. Liang, E\. Candès, and T\. HashimotoS1: simple test\-time scaling\.InConference on Empirical Methods in Natural Language Processing,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p1.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px1.p1.1)\.
- Nyeet al\.\(2021\)M\. Nye, A\. J\. Andreassen, G\. Gur\-Ari, H\. Michalewski, J\. Austin, D\. Bieber, D\. Dohan, A\. Lewkowycz, M\. Bosma, D\. Luan, C\. Sutton, and A\. OdenaShow your work: scratchpads for intermediate computation with language models\.arXiv preprint arXiv:2112\.00114\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p1.1)\.
- Patelet al\.\(2021\)A\. Patel, S\. Bhattamishra, and N\. GoyalAre NLP models really able to solve simple math word problems?\.InConference of the North American Chapter of the Association for Computational Linguistics,pp\. 2080–2094\.Cited by:[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.4.1)\.
- Pfauet al\.\(2024\)J\. Pfau, W\. Merrill, and S\. R\. BowmanLet’s think dot by dot: hidden computation in transformer language models\.InFirst Conference on Language Modeling,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p2.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Roy and Roth \(2015\)S\. Roy and D\. RothSolving general arithmetic word problems\.InConference on Empirical Methods in Natural Language Processing,pp\. 1743–1752\.Cited by:[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.5.1)\.
- Sapet al\.\(2019\)M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. ChoiSocial IQa: commonsense reasoning about social interactions\.InConference on Empirical Methods in Natural Language Processing,Cited by:[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.12.1)\.
- Saunshiet al\.\(2025\)N\. Saunshi, N\. Dikkala, Z\. Li, S\. Kumar, and S\. J\. ReddiReasoning with latent thoughts: on the power of looped transformers\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1),[§1](https://arxiv.org/html/2610.19568#S1.p2.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Shenet al\.\(2025\)Z\. Shen, H\. Yan, L\. Zhang, Z\. Hu, Y\. Du, and Y\. HeCODI: compressing chain\-of\-thought into continuous space via self\-distillation\.InConference on Empirical Methods in Natural Language Processing,Suzhou, China\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1),[Table 7](https://arxiv.org/html/2610.19568#A1.T7.2.1.3.1),[§1](https://arxiv.org/html/2610.19568#S1.p2.1),[§1](https://arxiv.org/html/2610.19568#S1.p3.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2610.19568#S4.SS1.SSS0.Px2.p1.1)\.
- Snellet al\.\(2025\)C\. Snell, J\. Lee, K\. Xu, and A\. KumarScaling LLM test\-time compute optimally can be more effective than scaling parameters for reasoning\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.19568#S1.p1.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px1.p1.1)\.
- Suet al\.\(2025\)D\. Su, H\. Zhu, Y\. Xu, J\. Jiao, Y\. Tian, and Q\. ZhengToken assorted: mixing latent and text tokens for improved language model reasoning\.InInternational Conference on Machine Learning,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p2.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px3.p1.1)\.
- Suleymanzadeet al\.\(2026\)A\. Suleymanzade, H\. A\. Gozeten, M\. Bronstein, İ\. İ\. Ceylan, and J\. KimMUX: continuous reasoning via multiplexed tokens\.arXiv preprint arXiv:2607\.18264\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1),[Table 7](https://arxiv.org/html/2610.19568#A1.T7.2.1.6.1),[§C\.3](https://arxiv.org/html/2610.19568#A3.SS3.SSS0.Px1.p1.1),[§C\.3](https://arxiv.org/html/2610.19568#A3.SS3.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.19568#S1.p2.1),[§1](https://arxiv.org/html/2610.19568#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.19568#S4.SS1.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2610.19568#S4.T2)\.
- Tacket al\.\(2026\)J\. Tack, J\. Lanchantin, J\. Yu, A\. Cohen, I\. Kulikov, J\. Lan, S\. Hu, Y\. Tian, J\. Weston, and X\. LiLLM pretraining with continuous concepts\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2023\)X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. ZhouSelf\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.19568#S1.p1.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px1.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 24824–24837\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.19568#S1.p1.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px1.p1.1)\.
- Weiet al\.\(2026\)X\. Wei, X\. Liu, Y\. Zang, X\. Dong, Y\. Cao, J\. Wang, X\. Qiu, and D\. LinSIM\-CoT: supervised implicit chain\-of\-thought\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1),[Table 7](https://arxiv.org/html/2610.19568#A1.T7.2.1.4.1),[§1](https://arxiv.org/html/2610.19568#S1.p2.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Welblet al\.\(2017\)J\. Welbl, N\. F\. Liu, and M\. GardnerCrowdsourcing multiple choice science questions\.InWorkshop on Noisy User\-generated Text,Cited by:[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.9.1)\.
- Wuet al\.\(2025\)C\. Wu, J\. Lu, Z\. Ren, G\. Hu, Z\. Wu, D\. Dai, and H\. WuLLMs are single\-threaded reasoners: demystifying the working mechanism of soft thinking\.arXiv preprint arXiv:2508\.03440\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Xuet al\.\(2025\)Y\. Xu, X\. Guo, Z\. Zeng, and C\. MiaoSoftCoT: soft chain\-of\-thought for efficient reasoning with LLMs\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 23336–23351\.Cited by:[§A\.2](https://arxiv.org/html/2610.19568#A1.SS2.p1.1),[Table 7](https://arxiv.org/html/2610.19568#A1.T7.2.1.8.1),[§1](https://arxiv.org/html/2610.19568#S1.p3.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei,et al\.Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§4\.1](https://arxiv.org/html/2610.19568#S4.SS1.SSS0.Px1.p1.2)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InAnnual Meeting of the Association for Computational Linguistics,Cited by:[Table 15](https://arxiv.org/html/2610.19568#A5.T15.4.13.1)\.
- Zhanget al\.\(2025\)Z\. Zhang, X\. He, W\. Yan, A\. Shen, C\. Zhao, S\. Wang, Y\. Shen, and X\. E\. WangSoft thinking: unlocking the reasoning potential of LLMs in continuous concept space\.InAdvances in Neural Information Processing Systems,Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1),[Table 7](https://arxiv.org/html/2610.19568#A1.T7.2.1.7.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2025a\)H\. Zhu, S\. Hao, Z\. Hu, J\. Jiao, S\. Russell, and Y\. TianReasoning by superposition: a theoretical perspective on chain of continuous thought\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2025b\)R\. Zhu, T\. Peng, T\. Cheng, X\. Qu, J\. Huang, D\. Zhu, H\. Wang, K\. Xue, X\. Zhang, Y\. Shan,et al\.A survey on latent reasoning\.arXiv preprint arXiv:2507\.06203\.Cited by:[§A\.1](https://arxiv.org/html/2610.19568#A1.SS1.p3.1),[§2](https://arxiv.org/html/2610.19568#S2.SS0.SSS0.Px2.p1.1)\.

Appendix

## Appendix AExtended Related Work

[Section2](https://arxiv.org/html/2610.19568#S2)introduces the work most directly connected to INTACT\. Here we organize the broader literature by what happens to the written reasoning chain\. It may remain in language, be removed, or be represented by continuous state\. This distinction matters for INTACT, which replaces only an opening segment and leaves a frozen language model to produce the continuation\.

### A\.1From Written to Latent Reasoning

Scratchpads and chain\-of\-thought make intermediate computation explicit\([Nye et al\., 2021](https://arxiv.org/html/2610.19568#bib.bib4);[Wei et al\., 2022](https://arxiv.org/html/2610.19568#bib.bib2);[Kojima et al\., 2022](https://arxiv.org/html/2610.19568#bib.bib3)\)\. More language\-space computation can improve accuracy through sampled chains\([Wang et al\., 2023](https://arxiv.org/html/2610.19568#bib.bib5)\), test\-time compute allocation\([Snell et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib6);[Muennighoff et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib7)\), process supervision\([Lightman et al\., 2024](https://arxiv.org/html/2610.19568#bib.bib32)\), or reinforcement learning on verifiable answers\([DeepSeek\-AI et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib8)\)\. Its cost is autoregressive decoding, and long\-reasoning models can spend unnecessary computation on easy problems\([Chen et al\., 2025b](https://arxiv.org/html/2610.19568#bib.bib9)\)\. Prompt compression reduces this cost while retaining a discrete, readable input\([Jiang et al\., 2023](https://arxiv.org/html/2610.19568#bib.bib26)\)\. After its latent prefix, INTACT likewise returns to language\-space generation\.

Other methods remove or replace the written chain\. Implicit chain\-of\-thought distills a reasoning teacher into a student that answers directly\([Deng et al\., 2023](https://arxiv.org/html/2610.19568#bib.bib12)\), while stepwise internalization progressively deletes reasoning steps from the training target\([Deng et al\., 2024](https://arxiv.org/html/2610.19568#bib.bib13)\)\. Pause and filler tokens instead reserve computation without assigning those positions linguistic content\([Goyal et al\., 2024](https://arxiv.org/html/2610.19568#bib.bib10);[Pfau et al\., 2024](https://arxiv.org/html/2610.19568#bib.bib11)\)\. Compressed chain\-of\-thought predicts dense contemplation tokens as a surrogate for a trace\([Cheng and Van Durme, 2024](https://arxiv.org/html/2610.19568#bib.bib27)\), and Token Assorted combines discrete codes with text\([Su et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib28)\)\. These approaches reduce or eliminate the written intermediate sequence\. INTACT is trained to re\-encode its firstKKsteps and preserves a written continuation\.

Continuous reasoning admits several constructions\. Coconut feeds the model’s last hidden state back as its next input embedding\([Hao et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib14)\); CODI adds self\-distillation between explicit and latent reasoning modes\([Shen et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib16)\); and SIM\-CoT supervises individual latent steps with an auxiliary decoder\([Wei et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib36)\)\. Soft Thinking is training\-free and replaces a sampled token with a probability\-weighted mixture of embeddings\([Zhang et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib17)\), although subsequent work questions whether this mixture provides genuinely parallel computation\([Wu et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib18)\)\. Related architectures spend depth rather than generated tokens\([Saunshi et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib20);[Geiping et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib21)\)\. LOTUS jointly refines padded latent blocks through a looped backbone and supervises their token\-level readout\([Fan et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib1)\); MUX generates latent tokens sequentially but gives each a locally decodable target that multiplexes a span of the reference trace\([Suleymanzade et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib35)\); and KaVa uses a compressed teacher KV\-cache as its distillation target\([Kuzina et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib50)\)\. LaDiR instead denoises blocks in a variational latent space whose frozen decoder reconstructs readable reasoning steps\([Kang et al\., 2026](https://arxiv.org/html/2610.19568#bib.bib25)\)\. Surveys by[Zhu et al\. \(2025b\)](https://arxiv.org/html/2610.19568#bib.bib23)and[Chen et al\. \(2025a\)](https://arxiv.org/html/2610.19568#bib.bib51)provide broader taxonomies\.

### A\.2Positioning of INTACT

INTACT is defined by two choices\. The language model remains frozen, and the latents replace written steps rather than merely adding a task\-specific prompt\. Prompt tuning also keeps the model frozen, but learns one constant prefix per task\([Lester et al\., 2021](https://arxiv.org/html/2610.19568#bib.bib49)\)\. SoftCoT is the closest input\-dependent alternative: a small frozen assistant reads the question and a learned projection maps its hidden states into soft tokens for a frozen backbone\([Xu et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib29)\)\. It emits those tokens in one pass and uses them as additional context\. INTACT instead derives each block from the backbone’s own states, conditions it on the blocks already placed, and trains it to replace the effect of written steps\.

[Table7](https://arxiv.org/html/2610.19568#A1.T7)sets out the design choices that separate these methods\.*Frozen*means the language model’s parameters never change;*readsMM’s states*, that the latents are computed from its hidden states rather than from its output distribution or from the question alone;*recurrent substitution*, that they are produced in sequence, each conditioned on those already placed, and stand in for written steps rather than supplementing them;*interpretable*, that the paper recovers token\-level content from intermediate latents; and*multi\-domain artifact*, that one trained artifact is evaluated across reasoning domains without per\-task adaptation\. The last two columns record reported evidence, not an intrinsic limitation of methods without a checkmark\.

Table 7:Design properties of representative latent\-reasoning methods\. Checkmarks record reported design choices or evidence, not a ranking of the methods\.The methods that update the backbone answer a different experimental question\. Coconut, CODI, SIM\-CoT, LOTUS, and MUX jointly learn a latent mechanism and alter the language model, whereas INTACT isolates the interface by propagating gradients through fixed model weights\. LOTUS also differs structurally\. Its blocks are refined jointly rather than emitted as recurrent substitutes\. CODI, SIM\-CoT, LOTUS, and MUX report token\-level probes or readouts of their latents\. INTACT does not make an interpretability claim\.

Among the methods in[Table7](https://arxiv.org/html/2610.19568#A1.T7), INTACT alone combines a frozen backbone, backbone\-state feedback, and recurrent replacement of written steps\. Soft Thinking is frozen and sequential, but constructs each soft token from an output distribution over the vocabulary rather than from hidden states\. INTACT is also the only method in the table evaluated as one trained artifact across reasoning domains\. These distinctions describe the intervention being evaluated\. They do not imply that updating the backbone or making latents directly readable is undesirable\.

#### Against SoftCoT\.

SoftCoT is the closest prior learned latent interface that also keeps the language model frozen\. We use the authors’ training script to train SoftCoT onLlama\-3\.1\-8B\-Instruct\. For a like\-for\-like comparison, both methods are evaluated under the SoftCoT protocol\. The INTACT row is therefore a separate single\-checkpoint evaluation and is not directly comparable to the three\-seed results in Tables[1](https://arxiv.org/html/2610.19568#S4.T1)and[2](https://arxiv.org/html/2610.19568#S4.T2)\.[Table8](https://arxiv.org/html/2610.19568#A1.T8)reports both accuracy and token use\.

Table 8:Comparison with SoftCoT onLlama\-3\.1\-8B\-Instructunder the SoftCoT evaluation protocol\. Each entry reports accuracy \(%\)/mean total tokens, including latent and generated tokens\. OOD is the macro\-average over SVAMP, GSM\-Hard, and MultiArith\. Every entry is measured here\. We train SoftCoT with the authors’ script and evaluate both methods under their protocol\. INTACT uses mathematics training \(GSM8K\-Aug\)\.INTACT uses fewer total tokens on every benchmark and achieves a higher OOD average, although SoftCoT remains slightly more accurate on GSM\-Hard\. The separation is wider in cost than in accuracy\. INTACT leads by one point of out\-of\-domain accuracy and by a24%24\\%reduction in tokens,167167on average compared to SoftCoT’s220220\. That pattern is consistent with the two designs, since SoftCoT’s assistant reads the question alone and emits its tokens in a single pass, whereas each INTACT block is computed from the frozen model’s state after the blocks already placed, and is trained to reproduce the effect of the written step it replaces rather than to supply additional context\.

## Appendix BNotation

[Table9](https://arxiv.org/html/2610.19568#A2.T9)collects the symbols used in the paper\.

Table 9:Notation used in this paper\.
## Appendix CExperimental Details

### C\.1INTACT Training

#### Data and depth sampling\.

For comparison against the latent\-reasoning methods of[Table2](https://arxiv.org/html/2610.19568#S4.T2), we train on the corpus they use, GSM8K\-Aug, described in[AppendixE](https://arxiv.org/html/2610.19568#A5)\. We parse its calculator annotations as ordered reasoning steps and retain the322,712322\{,\}712examples containing at least two steps\. For a trace ofSSsteps, Stage 1 supervises depthsi≤min⁡\(K,S−1\)i\\leq\\min\(K,S\-1\), leaving at least one written step in the shared continuation\. Only55\.5%55\.5\\%,23\.2%23\.2\\%, and8\.3%8\.3\\%of the retained examples support depths two, three, and four, respectively\. We therefore divide the corpus into the four length bucketsS=2S=2,S=3S=3,S=4S=4, andS≥5S\\geq 5and sample them uniformly\.[Figure6](https://arxiv.org/html/2610.19568#A3.F6)illustrates a complete four\-block substitution\.

Figure 6:Progressive substitution on a mathematics example\.GSM8K\-Aug provides five ordered calculations\. The question is shortened here for space\. At full substitution, the student replaces the first four written stepsz1:4z\_\{1:4\}with latent blocksv1:4v\_\{1:4\}, while the final calculationz5z\_\{5\}and answer remain the shared continuation\. The frozen language model evaluates both prefixes, andℒsub\\mathcal\{L\}\_\{\\mathrm\{sub\}\}at depthi=4i=4matches the resulting continuation distributions\. Vertical alignment denotes which written step each latent block replaces, not a claim that the latent vectors decode to that step\.
#### INTACT Stage 1\.

Unless otherwise stated, experiments useK=4K=4blocks ofc=4c=4vectors\. Each minibatch produces one encoder update at every depth supported by its examples\. Preservation rollouts are sampled from the latent\-conditioned model with temperature0\.70\.7and top\-p=0\.95p=0\.95, then held fixed whileℒpre\\mathcal\{L\}\_\{\\mathrm\{pre\}\}is evaluated\. The preservation weights areβ=0\.28\\beta=0\.28,0\.300\.30,0\.340\.34, and0\.280\.28for Llama\-1B, Llama\-3B, Llama\-8B, and Qwen\-3B, respectively\.

Teacher and student prefixes can occupy different numbers of positions\. Letℓz=\|𝐳1:i\|\\ell\_\{z\}=\|\\mathbf\{z\}\_\{1:i\}\|andℓv=i​c\\ell\_\{v\}=ic\. We assign the latent span position IDs beginning at

max⁡\(\|𝐱\|,\|𝐱\|\+ℓz−ℓv\),\\max\\\!\\left\(\|\\mathbf\{x\}\|,\\,\|\\mathbf\{x\}\|\+\\ell\_\{z\}\-\\ell\_\{v\}\\right\),which aligns the shared continuation wheneverℓz≥ℓv\\ell\_\{z\}\\geq\\ell\_\{v\}and otherwise prevents the latent span from overlapping the question\. This shift applies only to Stage 1 substitution, where teacher and student are evaluated on the same continuation\. Preservation, Stage 2, and inference generate without a reference trace, so the question and latent prefix use contiguous position IDs\.

On GSM8K\-Aug the shift is systematic\. The shortest step is55tokens againstc=4c=4latent slots, soℓz\>ℓv\\ell\_\{z\}\>\\ell\_\{v\}at every depth and the clamp never applies\. The displacement grows from33positions ati=1i\{=\}1to1414ati=4i\{=\}4, roughly a quarter of the prefix, so Stage 1 trains on a layout that inference does not reproduce\.

Llama encoders are trained for1,5001\{,\}500minibatch iterations, with checkpoints saved every100100iterations\. Qwen encoders are trained for3,0003\{,\}000iterations and saved every200200\. We select checkpoints by generation accuracy on300300held\-out GSM8K\-Aug questions that do not overlap the GSM8K test set\.

#### INTACT Stage 2\.

Each Stage 2 run begins from its selected Stage 1 checkpoint\. We first evaluate four completions for each of the first3,0003\{,\}000eligible training questions and retain only questions for which the completions include both a correct and an incorrect answer\. This produces between1,4731\{,\}473and1,9061\{,\}906questions across the Stage 1 seed lineages\. During training, every visit draws eight fresh completions\. Questions whose rewards all agree contribute no update\.

An optimizer step contains1212questions and eight completions per question, sampled at temperature1\.01\.0\. We optimize the per\-sequence\-normalized objective for800800steps using AdamW with learning rate10−510^\{\-5\}, zero weight decay, and gradient clipping at1\.01\.0\. The reported checkpoint is selected by generation accuracy on the held\-out GSM8K\-Aug split\.

The frozen language model is kept in bfloat16 and the encoder in float32\. Gradients pass through the language model, but only the encoder is updated\. Each run uses one NVIDIA H100 NVL\.[Table10](https://arxiv.org/html/2610.19568#A3.T10)summarizes the optimization settings\. Reported uncertainty is the sample standard deviation over the corresponding three encoder seed lineages\.

Table 10:Hyperparameters for both stages\. The optimizer is AdamW throughout\.
#### Architecture controls\.

The two architecture variants of[Table5](https://arxiv.org/html/2610.19568#S4.T5)are retrained rather than modified at inference, and the jointly built variant is compared against a reference run that differs from it in one setting alone\. Both draw the same GSM8K\-Aug stream, take the same teacher continuations at the same four substitution depths, carry preservation weightβ=0\.34\\beta=0\.34against the student’s own rollout, train for1,5001\{,\}500steps atK=4K=4and so take the same number of optimizer updates, and select a checkpoint by generation accuracy on the held\-out validation split\. The jointly built encoder flattens allK​cKcqueries into one encoder pass, so its blocks attend to one another and receive the progressive losses at the same four depths\. What it does not see is the backbone’s state after the preceding blocks have been placed\.

### C\.2Diverse\-Domain Training and Evaluation

#### Training corpus\.

We construct a12,00012\{,\}000\-example training stream from Llama\-Nemotron\. Checkpoints are selected on200200held\-out GSM8K\-Aug validation questions that are disjoint from the GSM8K test set\. The resulting mixture is64\.3%64\.3\\%mathematics,21\.4%21\.4\\%science, and14\.3%14\.3\\%code\. For mathematics, we use the reasoning\-off collection of direct solutions fromLlama\-3\.3\-70B\-Instruct\. The reasoning\-on collection instead places the usable solution after a substantially longer exploratory trace\. The science and code examples are shared by both collections\. We convert responses to ordered steps using equation\-bearing spans for mathematics, sentences for science, and lines of fenced programs for code\.

#### Training\.

We train the same107107M encoder architecture onLlama\-3\.2\-3B\-Instruct, withK=4K=4blocks ofc=4c=4vectors\. Stage 1 runs for1,5001\{,\}500steps with learning rate10−410^\{\-4\}, batch size88, a200200\-step warmup, and preservation weightβ=0\.30\\beta=0\.30, using all three domains\. Stage 2 starts from that checkpoint and runs for800800group\-relative policy\-gradient steps with learning rate10−510^\{\-5\}, batch size1212, and groups of eight rollouts sampled at temperature1\.01\.0\. Its rollout pool is75%75\\%Nemotron mathematics and25%25\\%Nemotron science, because code is not used in Stage 2\. Mathematics receives a numeric\-correctness reward, and science receives an option\-letter correctness reward\. Rollouts are capped at256256tokens, and no KL penalty to a reference policy is used\. All other settings follow[AppendixC\.1](https://arxiv.org/html/2610.19568#A3.SS1)and the language model remains frozen throughout\.

#### Evaluation\.

We evaluate under the sampling protocol of[AppendixC\.4](https://arxiv.org/html/2610.19568#A3.SS4)with a2,0482\{,\}048\-token budget\. The loose budget limits the censoring caused by unfinished frozen\-model generations, which still run to the cap on a larger fraction of the mathematics examples than Stage 2 does\. Reported INTACT costs include both the generated tokens and the1616latent positions\.

### C\.3Baselines

#### Result provenance\.

AtLlama\-3\.2\-1BandLlama\-3\.2\-3B, we quote No\-CoT, Aug\-SFT, MUX, and CODI from[Suleymanzade et al\. \(2026\)](https://arxiv.org/html/2610.19568#bib.bib35), and LOTUS from[Fan et al\. \(2026\)](https://arxiv.org/html/2610.19568#bib.bib1)\. Measured entries atLlama\-3\.1\-8BandQwen2\.5\-3Bare ours\. No directly comparable latent\-reasoning checkpoint has been released for these backbones\. Published 8B MUX and CODI results use base models, whereas our 8B backbone is instruction\-tuned\.

#### No\-CoT and Aug\-SFT\.

Our No\-CoT and Aug\-SFT runs at 8B and on Qwen use the same script and differ only in the supervision target\. We apply rank\-128128LoRA to theq,k,v,o,gate,up,downprojections of every layer, withα=2​r\\alpha=2r, dropout0\.050\.05, and no bias\. Training uses AdamW with learning rate10−410^\{\-4\}, zero weight decay,100100warmup steps followed by cosine decay, gradient clipping at1\.01\.0, sequence length512512, and effective batch size1616in bfloat16\. Both train on all385,530385\{,\}530GSM8K\-Aug examples, where INTACT uses the322,712322\{,\}712of them that carry at least two steps \([AppendixC\.1](https://arxiv.org/html/2610.19568#A3.SS1)\)\. The adapters follow the same rank\-128128configuration at every scale\.

#### Nemo\-SFT\.

We train this baseline on all four backbones using the same script and LoRA configuration as Aug\-SFT, but replace GSM8K\-Aug with the Llama\-Nemotron mixture from[AppendixC\.2](https://arxiv.org/html/2610.19568#A3.SS2)\. Thus the two baselines differ only in their supervision\. Aug\-SFT uses terse calculator expressions averaging roughly3131tokens, and Nemo\-SFT uses natural\-language solutions averaging roughly750750tokens\. We select the step\-200200checkpoint on every backbone\.

#### Latent\-reasoning baselines\.

For LOTUS onQwen2\.5\-3B, we run the authors’ code withcthought=25c\_\{\\mathrm\{thought\}\}=25, six latent loops, constant AdamW learning rate5×10−55\\times 10^\{\-5\}, effective batch size128128, and seven epochs\. This covers the curriculum throughmax\_latent\_stage=6\. We report the final epoch using the authors’ evaluation script\. The quoted MUX results use the shared LoRA configuration reported in its Table 8: rank128128,α=32\\alpha=32, and dropout0\.10\.1\. The quoted CODI results come from the same codebase and configuration, with CODI’s additional hidden\-size projector included in its trainable\-parameter count\. We also train MUX onLlama\-3\.1\-8B\-Instruct, where no checkpoint is released\. MUX’s published learning rate of8×10−48\\times 10^\{\-4\}collapses on this backbone, so we use10−410^\{\-4\}with gradient clipping at1\.01\.0and warmup ratio0\.100\.10, and otherwise follow theirtrain\_llama8b\_gsm8k\_aug\.sh\.[Figure14](https://arxiv.org/html/2610.19568#A4.F14)additionally includes SIM\-CoT at 3B\. We take its GSM8K and out\-of\-domain accuracies from Table 2 of[Suleymanzade et al\. \(2026\)](https://arxiv.org/html/2610.19568#bib.bib35)and estimate its trainable parameters as the shared rank\-128128LoRA adapter plus a full auxiliary decoder withPdec=PP\_\{\\mathrm\{dec\}\}=P, following their Appendix B\.4\.

### C\.4Evaluation Protocol

#### Prompting and decoding\.

For numerical benchmarks, the model receives the bare question followed by a newline\. Multiple\-choice prompts instead useQuestion:followed by the question and lettered options\. Neither format includes few\-shot demonstrations or a model\-specific chat template\. Following the CODI/MUX protocol, we sample with temperature0\.10\.1, top\-kk4040, and top\-pp0\.950\.95\. Quoted results retain the protocol of their source\.

#### Generation budget\.

The GSM8K\-Aug mathematics experiments use a768768\-token cap for every system, and the Llama\-Nemotron diverse\-domain experiment uses2,0482\{,\}048\. The larger budget reflects that mixture’s longer targets\. Its steps are complete natural\-language solutions rather than compact arithmetic, and at a512512\-token cap the frozen backbone is truncated on up to13\.9%13\.9\\%of mathematics problems, suppressing the reference point the gains are measured against\.[AppendixD\.1](https://arxiv.org/html/2610.19568#A4.SS1)sweeps the cap from128128to768768tokens\. From512512to768768tokens, most systems move by tenths of a point,Qwen2\.5\-3Bstill gains1\.31\.3, and the frozenLlama\-3\.1\-8Bloses accuracy because it rarely terminates\.[Table8](https://arxiv.org/html/2610.19568#A1.T8)instead adopts the1,0241\{,\}024\-token cap of the SoftCoT protocol for both of its rows\. Quoted results retain the budget of their source\.

#### Answer extraction\.

We discard text following a newly generatedQuestion:, which base models sometimes emit after completing a solution\. For numerical benchmarks, we extract, in order, a value following\#\#\#\#or\#\#\#, an explicit “answer is” phrase, a boxed value, or the final number in the remaining completion\. A prediction is correct when the extracted value matches the gold answer within10−410^\{\-4\}\. For multiple\-choice benchmarks, we extract an explicit answer letter first, then the last matching option text, and finally the last standalone letter\. Correctness is exact match with the gold option letter\.

## Appendix DExtended Results

### D\.1INTACT and the Frozen Backbone Across Budgets

Table 11:Accuracy \(%\) as a function of the generation budget, with the encoder trained on GSM8K\-Aug\.*GSM8K*is the matching test set and*OOD*averages SVAMP, GSM\-Hard, and MultiArith\. Stage 1 is progressive substitution alone and Stage 2 adds latent\-policy refinement\. Columns with±\\pmshow mean±\\pmsample standard deviation over three encoder seeds\.[Table11](https://arxiv.org/html/2610.19568#A4.T11)compares the frozen backbone with both INTACT stages at every backbone and every generation budget, of which[Table1](https://arxiv.org/html/2610.19568#S4.T1)shows one cap\. All budget values are scored from prefixes of the same generation pass\. The frozenLlama\-3\.1\-8Brow falls as the budget grows because that model rarely terminates, reaching the cap on97\.1%97\.1\\%of GSM8K generations against8\.6%8\.6\\%for Stage 1 and none for Stage 2\.

#### Prompt format\.

To isolate the effect of the prompt format, we re\-evaluate the three instruction\-tuned Llama backbones with their own chat templates, a sample of which is shown in[Figure7](https://arxiv.org/html/2610.19568#A4.F7)\. The template helps at 8B, where it largely removes the termination failure and raises GSM8K accuracy from72\.372\.3to78\.478\.4, but the effect is inconsistent across scales\. Accuracy rises at 1B and falls at 3B\. INTACT Stage 2, still scored under the bare prompt, remains more accurate than the chat\-prompted frozen model on GSM8K and out of domain at all three scales, while its token cost is closer, lower at 8B and higher at 1B and 3B \([Table12](https://arxiv.org/html/2610.19568#A4.T12)\)\.

Bare question John runs 60 miles in 3 days\. How many miles per day?\\n

Llama\-3\.1\-8B chat template <\|begin\_of\_text\|\> <\|start\_header\_id\|\>system<\|end\_header\_id\|\> Cutting Knowledge Date: December 2023 Today Date: 26 Jul 2024 <\|eot\_id\|\> <\|start\_header\_id\|\>user<\|end\_header\_id\|\> John runs 60 miles in 3 days\. How many miles per day?<\|eot\_id\|\> <\|start\_header\_id\|\>assistant<\|end\_header\_id\|\>

Figure 7:Inputs compared for the frozenLlama\-3\.1\-8B\-Instruct\. The question is illustrative\. The chat template adds its default system metadata and turn markers\.Table 12:Prompt sensitivity of the frozen Llama backbones at a768768\-token cap\. The two frozen rows differ only by prompt format\. The INTACT rows are Stage 2, evaluated with the bare prompt for reference\.

### D\.2Accuracy–Efficiency Operating Points

Figures[8](https://arxiv.org/html/2610.19568#A4.F8)–[10](https://arxiv.org/html/2610.19568#A4.F10)compare accuracy with total positions\. Upper left is better, and INTACT counts include all1616latent positions\. Figures[11](https://arxiv.org/html/2610.19568#A4.F11)–[13](https://arxiv.org/html/2610.19568#A4.F13)instead sweep the generation budget, with dashed curves updating the language model and solid curves keeping it frozen\.Llama\-3\.2\-1Bis omitted because it was not evaluated on the finer budget grid these panels require\. At 8B, no comparable CODI or LOTUS checkpoint is available and the MUX point is our reproduction \([AppendixC\.3](https://arxiv.org/html/2610.19568#A3.SS3)\)\.

Figure 8:Llama\-3\.2\-3B: accuracy against total positions on GSM8K \(left\) and the out\-of\-domain macro\-average \(right\)\. Upper left is better\. Frozen and Nemo\-SFT are evaluated at the768768\-token cap, and every other system at the operating point it reports\.Figure 9:Llama\-3\.1\-8B: accuracy against total positions on GSM8K \(left\) and the out\-of\-domain macro\-average \(right\)\. Upper left is better\. Frozen and Nemo\-SFT are evaluated at the768768\-token cap, and every other system at the operating point it reports\.Figure 10:Qwen2\.5\-3B: accuracy against total positions on GSM8K \(left\) and the out\-of\-domain macro\-average \(right\)\. Upper left is better\. Frozen and Nemo\-SFT are evaluated at the768768\-token cap, and every other system at the operating point it reports\.Figure 11:Llama\-3\.2\-3B: accuracy against the generation budget on GSM8K \(left\) and the out\-of\-domain macro\-average \(right\)\. Solid curves keep the language model frozen and dashed curves update it\. Every budget is scored from prefixes of one generation pass, so a curve shows what that same run would have returned under a tighter cap\.Figure 12:Llama\-3\.1\-8B: accuracy against the generation budget on GSM8K \(left\) and the out\-of\-domain macro\-average \(right\)\. Solid curves keep the language model frozen and dashed curves update it\. Every budget is scored from prefixes of one generation pass, so a curve shows what that same run would have returned under a tighter cap\.Figure 13:Qwen2\.5\-3B: accuracy against the generation budget on GSM8K \(left\) and the out\-of\-domain macro\-average \(right\)\. Solid curves keep the language model frozen and dashed curves update it\. Every budget is scored from prefixes of one generation pass, so a curve shows what that same run would have returned under a tighter cap\.
### D\.3Efficiency

Three quantities are reported, and they are not interchangeable\.

- •Generated tokens\.The text the backbone emits, excluding the latent prefix\.[Table3](https://arxiv.org/html/2610.19568#S4.T3)and the ablations report this\.
- •Total positions\.Generated tokens plus INTACT’sK​c=16Kc=16latent slots, excluding the question\. The operating\-point figures and[Table14](https://arxiv.org/html/2610.19568#A4.T14)use this\.
- •End\-to\-end latency\.Wall\-clock milliseconds per problem, including both latent\-prefix construction and text generation\.[Table4](https://arxiv.org/html/2610.19568#S4.T4)reports this\.

#### Trainable parameters and training cost\.

[Figure14](https://arxiv.org/html/2610.19568#A4.F14)places theLlama\-3\.2\-3Bsystems of[Table2](https://arxiv.org/html/2610.19568#S4.T2)against the number of parameters each trains, together with SIM\-CoT\. INTACT trains a107107M encoder and never updates the backbone, so it is the leftmost point in both panels and the most accurate of the methods that leave the backbone frozen\. The baselines run from194\.5194\.5M for the LoRA systems to3\.403\.40B for SIM\-CoT, which trains an adapter alongside a full auxiliary decoder and so updates more than a full fine\-tune does\. Nemo\-SFT is more accurate, but reaches that accuracy by updating194\.5194\.5M backbone parameters\. No paper in this comparison reports its own trainable count, so each is derived from the method’s description\.[AppendixC\.3](https://arxiv.org/html/2610.19568#A3.SS3)gives the sources and derivations\.

Figure 14:Accuracy against trainable parameters onLlama\-3\.2\-3B, where up and to the left is better\.*OOD*averages SVAMP, GSM\-Hard and MultiArith\. INTACT trains a107107M\-parameter encoder and never updates the backbone, while the baselines train between195195M and3\.403\.40B\. No method here reports a trainable count, so each is derived \([AppendixC\.3](https://arxiv.org/html/2610.19568#A3.SS3)\)\. Mathematics training \(GSM8K\-Aug\), except Nemo\-SFT \(Llama\-Nemotron\)\.Freezing the backbone also removes its parameter gradients and optimizer state, although its activations remain necessary for gradients to reach the encoder\. On one H100 NVL andLlama\-3\.2\-3B, Stage 1 takes6\.26\.2hours and Stage 2 a further10\.310\.3hours, placing the complete two\-stage recipe below1717GPU\-hours\.

### D\.4Diverse\-Domain Generalization

#### Accuracy by stage\.

Table 13:Accuracy \(%\) of oneLlama\-3\.2\-3Bencoder across twelve benchmarks at a2,0482\{,\}048\-token budget, trained on the Llama\-Nemotron mixture rather than GSM8K\-Aug\. Each system is a single checkpoint used in every domain, not a per\-domain best\.Δ\\Deltais Stage 2 against the frozen backbone\. Every split is scored in full\. Bold is the best of the three trained systems\.DomainBenchmark𝐧\\mathbf\{n\}FrozenStage 1Stage 2Nemo\-SFT𝚫\\mathbf\{\\Delta\}ScienceOpenBookQA50048\.463\.867\.671\.2\+19\.2\+19\.2ARC\-Challenge117250\.165\.472\.673\.0\+22\.5\+22\.5ARC\-Easy237667\.580\.886\.285\.5\+18\.7\+18\.7SciQ100077\.386\.087\.386\.5\+10\.0\+10\.0MMLU\-science139743\.849\.354\.356\.8\+10\.5\+10\.5CommonsenseECQA216252\.658\.564\.267\.0\+11\.6\+11\.6SocialIQA195442\.950\.364\.346\.7\+21\.4\+21\.4HellaSwag1004232\.439\.056\.252\.5\+23\.8\+23\.8MathematicsGSM8K131975\.477\.678\.783\.4\+3\.3\+3\.3SVAMP100077\.980\.881\.683\.3\+3\.7\+3\.7MultiArith60095\.597\.097\.598\.7\+2\.0\+2\.0GSM\-Hard131926\.226\.226\.628\.9\+0\.4\+0\.4[Table13](https://arxiv.org/html/2610.19568#A4.T13)gives the accuracies behind[Figure5](https://arxiv.org/html/2610.19568#S4.F5)\. The two stages divide the work differently by domain\. In science, Stage 1 contributes15\.415\.4of the final19\.219\.2\-point gain on OpenBookQA and15\.315\.3of22\.522\.5points on ARC\-Challenge\. Stage 2 has its largest effect in commonsense\. It adds14\.014\.0points on SocialIQA beyond the7\.47\.4\-point Stage 1 gain\. Its contribution on mathematics is smaller but consistently positive, ranging from0\.40\.4to1\.11\.1points\. Across the three commonsense benchmarks, Stage 2 reaches61\.6%61\.6\\%compared with55\.4%55\.4\\%for Nemo\-SFT\.

#### Token cost and termination\.

Table 14:Mean total positions per problem for the encoder trained on the Llama\-Nemotron mixture, at a2,0482\{,\}048\-token budget\. INTACT columns include the1616latent prefix positions as well as the tokens generated, so a number here is comparable across rows and to[Figure1](https://arxiv.org/html/2610.19568#S1.F1)\. Lower is better, and bold is the cheapest system on a row\.[Table14](https://arxiv.org/html/2610.19568#A4.T14)gives what each system spends to reach the accuracies of[Table13](https://arxiv.org/html/2610.19568#A4.T13)\. Stage 2 uses fewer total positions than the frozen backbone on all twelve benchmarks while improving accuracy on each\. Stage 1 is cheaper on ten of twelve but is more expensive on ARC\-Challenge and SocialIQA\. The shortening is therefore not imposed mechanically by the latent prefix\. It emerges most consistently after latent\-policy refinement\.

The2,0482\{,\}048\-token budget reduces but does not eliminate truncation of frozen\-model generations\. Their reported means consequently remain more sensitive to the evaluation cap than those of INTACT and Nemo\-SFT, which terminate more consistently\.

## Appendix EDatasets and Evaluation Splits

### E\.1Training Corpora

The experiments use two training corpora, each for a separate encoder\.

#### GSM8K\-Aug\.

The main mathematics experiments of[Section4\.1](https://arxiv.org/html/2610.19568#S4.SS1)useGSM8K\-Aug\([Deng et al\., 2023](https://arxiv.org/html/2610.19568#bib.bib12)\), a385385K\-example augmentation of GSM8K whose solutions consist of ordered, compact arithmetic steps\. This structure directly supports progressive substitution\. Latent blockiireplaces written stepiiwithout additional segmentation\.

#### Llama\-Nemotron\.

The diverse\-domain experiment of[Section4\.5](https://arxiv.org/html/2610.19568#S4.SS5)usesLlama\-Nemotron post\-training data\([Bercovich et al\., 2025](https://arxiv.org/html/2610.19568#bib.bib41)\), which combines mathematics, science, and code under one collection process\. We use its reasoning\-off subset, whose responses are direct written solutions rather than search traces\.[AppendixC\.2](https://arxiv.org/html/2610.19568#A3.SS2)describes the sampling and step extraction\.

### E\.2Evaluation Benchmarks and Splits

Table 15:Evaluation benchmarks and scored split sizes, grouped by domain\. Every row uses the complete evaluation split after ECQA de\-duplication\.*Form*is either a numeric answer or anmm\-way letter choice\.DomainBenchmark𝐧\\mathbf\{n\}FormMathematicsGSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.19568#bib.bib30)\)1319numberGSM\-Hard\([Gao et al\., 2023](https://arxiv.org/html/2610.19568#bib.bib48)\)1319numberSVAMP\([Patel et al\., 2021](https://arxiv.org/html/2610.19568#bib.bib46)\)1000numberMultiArith\([Roy and Roth, 2015](https://arxiv.org/html/2610.19568#bib.bib47)\)600numberScienceOpenBookQA\([Mihaylov et al\., 2018](https://arxiv.org/html/2610.19568#bib.bib39)\)5004\-wayARC\-Easy\([Clark et al\., 2018](https://arxiv.org/html/2610.19568#bib.bib40)\)23764\-wayARC\-Challenge\([Clark et al\., 2018](https://arxiv.org/html/2610.19568#bib.bib40)\)11724\-waySciQ\([Welbl et al\., 2017](https://arxiv.org/html/2610.19568#bib.bib44)\)10004\-wayMMLU\-science\([Hendrycks et al\., 2021a](https://arxiv.org/html/2610.19568#bib.bib45)\)13974\-wayCommonsenseECQA\([Aggarwal et al\., 2021](https://arxiv.org/html/2610.19568#bib.bib38)\)21625\-waySocialIQA\([Sap et al\., 2019](https://arxiv.org/html/2610.19568#bib.bib43)\)19543\-wayHellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2610.19568#bib.bib42)\)10,04210\{,\}0424\-way#### Benchmark coverage\.

[Table15](https://arxiv.org/html/2610.19568#A5.T15)lists every benchmark scored in the paper\. Each benchmark uses its complete evaluation split, subject only to the ECQA de\-duplication described below\. GSM8K is the matching test set for GSM8K\-Aug\. For the diverse\-domain experiment, we check for overlap after case\-folding the question text and removing punctuation and multiple\-choice options\. We find no exact question\-text match between the examples used in either training stage and any of the twelve evaluation splits\. No evaluation question is used for checkpoint selection\. Selection uses the disjoint 200\-question GSM8K\-Aug validation split described in[AppendixC\.2](https://arxiv.org/html/2610.19568#A3.SS2)\. Multiple\-choice benchmarks share the prompt format specified in[AppendixC\.4](https://arxiv.org/html/2610.19568#A3.SS4)\.

#### Dataset\-specific conventions\.

*MMLU\-science*is not a released split\. It aggregates the eight MMLU subjects covered by Llama\-Nemotron’s science data: high\-school and college physics, chemistry, and biology, together with conceptual physics and astronomy\. ECQA contains3232test questions that also occur in its training split, inherited from CommonsenseQA\. We remove these questions before scoring, leaving2,1622\{,\}162examples\.