Training Continuous Chain of Thought Models: A Tale of Two Regimes

# Training Continuous Chain of Thought Models: A Tale of Two Regimes
Source: [https://arxiv.org/html/2607.16972](https://arxiv.org/html/2607.16972)
###### Abstract

Continuous Chain\-of\-Thought methods replace verbose reasoning traces with a short sequence of dense latent representations\. Earlier continuous CoT methods indirectly supervise the latent representations such that its final state match that of verbose reasoning traces, requiring autoregressive, slow generation during training\. We introduce C\-MTP, a simpler, faster direct supervision approach that models each latent as an average of the embeddings in the CoT traces to be compressed\. Our approach outperforms a prior direct supervision method that approximates the distribution of compressed tokens, and performs competitively to slower indirect supervision approaches in existing evaluation setup with simplified CoT traces \(less than 100 tokens\)\. Lastly, we extend the evaluation of Continuous CoT methods to complex tasks with longer reasoning traces \(≥\\geqfew hundreds reasoning tokens\)\. We find both direct and indirect supervision training methods perform poorly \(roughly 65% performance drop\) in this setting, revealing the limitations of current continuous CoT methods\. The code and checkpoints are released at[https://github\.com/Varun221/cmtp\_research](https://github.com/Varun221/cmtp_research)\.

Machine Learning, ICML

![Refer to caption](https://arxiv.org/html/2607.16972v1/x1.png)Figure 1:Head\-to\-Head comparison of standard CoT finetuning with two ContinuousCoT training methods:Direct Supervision\(C\-MTP, our proposed method\) andIndirect Supervision\(CODI\) on three datasets of different CoT traces\.*Structured*consists of compact mathematical expressions of∼\\sim25 tokens,*Semi\-Natural*contains sentence per\-step explanations of∼\\sim62 tokens while*Realistic*captures traces generated from off\-the\-shelf LLM \(∼\\sim350 tokens on average\)\. Both methods remain competitive with CoT\-SFT on structured/semi\-natural traces but collapse to∼\\sim35% of its accuracy on realistic traces\.## 1Introduction

Recent LLMs achieve remarkable performance on complex reasoning tasks via generating lengthy sequence of tokens that describe their reasoning process, often called chain\-of\-thought \(CoT\) traces\(Weiet al\.,[2022b](https://arxiv.org/html/2607.16972#bib.bib28); Kojimaet al\.,[2022](https://arxiv.org/html/2607.16972#bib.bib29)\), expanding the expressive power of LLMs\(Merrill and Sabharwal,[2024](https://arxiv.org/html/2607.16972#bib.bib23)\)\. However, autoregressively generating long CoT traces increases the computational cost significantly\.

Continuous Chain of Thought \(ContinuousCoT\), pioneered byHaoet al\.\([2025](https://arxiv.org/html/2607.16972#bib.bib14)\), allows an LLM to reason entirely in a short sequence of dense latent representations without discrete “thinking” tokens, significantly reducing reasoning length and, therefore, inference compute\. Current approaches exceed the performance of discrete CoT on structured graph search tasks like ProsQA\(Haoet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib14)\)and ProntoQA\(Saparov and He,[2023](https://arxiv.org/html/2607.16972#bib.bib13)\)and show competitive performance \(about 10% performance degradation\) on math reasoning tasks\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.16972#bib.bib18); Denget al\.,[2024](https://arxiv.org/html/2607.16972#bib.bib9)\)These methods\(Shenet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib12); Weiet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib15)\)train a model to autoregressively predict intermediate latents and use a CoT teacher to supervise the final output\. This allows the model to compress varying length CoT trajectories into a fixed number of latents\. The sequential autoregressive passes required during training make it computationally expensive\. As these methods utilize chain\-of\-thought traces*indirectly*, we refer to them as usingindirect supervision\. These methods start from a pretrained discrete token LLM and fine\-tune it to predict latents that are fed back into the model for a fixed number of steps\.

Recent work, CoLaR\(Tanet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib16)\), mitigates this by modeling latents as parameterized Gaussian distributions conditioned on token embeddings aggregated from a CoT teacher\. This enables a parallelized training objective and brings the training distribution closer to pretraining\. However, the model must predict a distribution over the latent space, which poses optimization challenges\. Since this method utilizes CoT traces directly for training and prediction we term such methods as usingdirect supervision\. Unlike indirect supervision method, direct supervision methods enable parallel training\.

In this work, we introduce a new direct supervision approach:C\-MTP, that models Multi\-Token Prediction \(MTP\) to simulate latent reasoning\. We formalize intermediate latents as aggregated token embeddings within a chain\-of\-thought, training the model to predict the tokens composing each latent rather than a continuous vector\. This aligns with the pretraining objective while enabling fully parallel training via teacher forcing over CoT traces, eliminating the need for sequential autoregressive passes\. Our evaluations show that C\-MTP significantly outperforms existing direct supervision methods due to its simplicity and alignment with the pretraining objective\. With small structured reasoning traces, C\-MTP is faster, sample\-efficient and generalizes better to out\-of\-distribution problems \([Sections4\.2\.1](https://arxiv.org/html/2607.16972#S4.SS2.SSS1)and[4\.2](https://arxiv.org/html/2607.16972#S4.SS2)\) than indirect supervision methods\.

On verbose semi\-natural CoTs, we find that indirect supervision outperforms direct supervision, primarily because its recurrent training compresses long reasoning traces into few latents, while direct supervision is bounded by the original trace length and suffers from error accumulation under teacher\-forcing at inference\. Finally, on moving to realistic CoT traces from a strong teacher, both regimes fall well short of CoT\-SFT\. This suggests that structured CoT evaluations, which dominate current benchmarks, overstate the practical utility of ContinuousCoT methods\.

We state our contributions as follows:

- •We introduceC\-MTP, a novel direct supervision framework that uses Multi\-Token Prediction to simulate continuous latent reasoning\.
- •On structured reasoning tasks, C\-MTP outperforms existing direct supervision methods and matches or exceeds indirect supervision methods while being faster, more sample\-efficient, and generalizing better to out\-of\-distribution problems\.
- •We conduct a systematic empirical comparison across structured, semi\-natural and realistic CoT trace regimes, showing that indirect supervision overtakes direct on verbose semi\-natural traces through its compression objective, and that both regimes fall short of CoT\-SFT on realistic traces, exposing the limitations of ContinuousCoT approaches\.

## 2Related Work

Chain\-of\-Thought \(CoT\)\(Weiet al\.,[2022b](https://arxiv.org/html/2607.16972#bib.bib28); Kojimaet al\.,[2022](https://arxiv.org/html/2607.16972#bib.bib29)\)has found a wide range of applications\. It improves self\-consistency\(Wanget al\.,[2023b](https://arxiv.org/html/2607.16972#bib.bib30)\), simplifies complex problem solving through decomposition\(Zhouet al\.,[2023](https://arxiv.org/html/2607.16972#bib.bib31)\), facilitates planning\(Wanget al\.,[2023a](https://arxiv.org/html/2607.16972#bib.bib32)\), automates demonstration generation\(Zhanget al\.,[2023](https://arxiv.org/html/2607.16972#bib.bib33)\), and enhances supervised fine\-tuning\(Weiet al\.,[2022a](https://arxiv.org/html/2607.16972#bib.bib34); Chunget al\.,[2024](https://arxiv.org/html/2607.16972#bib.bib35); Zelikmanet al\.,[2022](https://arxiv.org/html/2607.16972#bib.bib36); Hsiehet al\.,[2023](https://arxiv.org/html/2607.16972#bib.bib37)\)\. These advancements have culminated in recent “thinking” models\(Guoet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib39); OpenAI,[2026](https://arxiv.org/html/2607.16972#bib.bib38)\)\.

ContinuousCoT, or latent reasoning, encompasses diverse approaches\. COCONUT\(Haoet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib14)\)pioneered autoregressive generation with latents, inspiring subsequent research\. Formal analyses\(Zhuet al\.,[2025a](https://arxiv.org/html/2607.16972#bib.bib40); Zouet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib41); Xu and Sato,[2025](https://arxiv.org/html/2607.16972#bib.bib42)\)demonstrate its theoretical advantages over explicit CoT in specific scenarios\. In this context,*latents*can encode multiple solution trajectories\(Gozetenet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib43); Zhanget al\.,[2025b](https://arxiv.org/html/2607.16972#bib.bib44); Wuet al\.,[2025b](https://arxiv.org/html/2607.16972#bib.bib45); Tanget al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib46); Buttet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib53)\), provide additional compute via filler tokens\(Pfauet al\.,[2024](https://arxiv.org/html/2607.16972#bib.bib47); Goyalet al\.,[2024](https://arxiv.org/html/2607.16972#bib.bib48)\), or use recurrent looping\(Saunshiet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib49); Fuet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib50); Zenget al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib51); Zhuet al\.,[2025b](https://arxiv.org/html/2607.16972#bib.bib52); Geipinget al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib57)\)\.

In this work, we utilize latents to encode intermediate reasoning steps through both indirect\(Shenet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib12); Weiet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib15)\)and direct supervision\(Tanet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib16); Cheng and Durme,[2024](https://arxiv.org/html/2607.16972#bib.bib66)\)\. Concurrently, the field is advancing novel training architectures\(Wuet al\.,[2025a](https://arxiv.org/html/2607.16972#bib.bib54); Shaoet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib55)\)and hybrid models that adaptively switch between latent reasoning and token\-based generation\(Shiet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib56); Yueet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib58); Xuet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib59); Zhuet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib61); Zhanget al\.,[2025a](https://arxiv.org/html/2607.16972#bib.bib60)\)\. Our work underscores the need for this adaptivity\.

Finally, while Multi\-Token Prediction \(MTP\) improves pretraining\(DeepSeek\-AI and team,[2025](https://arxiv.org/html/2607.16972#bib.bib63); Gloeckleet al\.,[2024](https://arxiv.org/html/2607.16972#bib.bib17)\), accelerates speculative decoding\(Caiet al\.,[2024](https://arxiv.org/html/2607.16972#bib.bib64)\), and can be learned via self\-distillation\(Zhaoet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib65)\), it remains largely unexplored in the context of latent reasoning\.

## 3Methodology

### 3\.1Notation and Problem Setup

Consider an LLMMθM\_\{\\theta\}\. Let𝒱\\mathcal\{V\}be its Vocabulary andE∈ℝ\|𝒱\|×dE\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}be the Embedding matrix\. Given an input promptQ=\(q1,q2,…,qm\)∈𝒱mQ=\(q\_\{1\},q\_\{2\},\\ldots,q\_\{m\}\)\\in\\mathcal\{V\}^\{m\}we denote the embedded prompt asE​\(Q\)=\(e​\(q1\),e​\(q2\),…,e​\(qm\)\)E\(Q\)=\(e\(q\_\{1\}\),e\(q\_\{2\}\),\\ldots,e\(q\_\{m\}\)\), wheree​\(qi\)∈ℝde\(q\_\{i\}\)\\in\\mathbb\{R\}^\{d\}\. Given a prompt, we can obtain the final layer hidden state at the final position from a student modelMθM\_\{\\theta\}with a function denoted byHθ\(\.\)∈ℝdH\_\{\\theta\}\(\.\)\\in\\mathbb\{R\}^\{d\}\.

Let’s consider that we have access toKKreasoning steps for the given prompt\. Denote them asy:=\(y1,y2,…,yK\)y:=\(y\_\{1\},y\_\{2\},\\ldots,y\_\{K\}\)\. Eachyi=\(yi​1,yi​2,…,yi​Li\)∈𝒱Liy\_\{i\}=\(y\_\{i1\},y\_\{i2\},\\ldots,y\_\{iL\_\{i\}\}\)\\in\\mathcal\{V\}^\{L\_\{i\}\}andLiL\_\{i\}is the token length of theii\-th textual reasoning step\. Finally, we denote an answer asa=\(a1,a2,…,an\)∈𝒱na=\(a\_\{1\},a\_\{2\},\\ldots,a\_\{n\}\)\\in\\mathcal\{V\}^\{n\}\.

Standard CoT supervised finetuning optimizes the following objective:

LC​o​T=−log⁡Pθ​\(a∣Q,y1,y2,…,yK\)\\displaystyle L\_\{CoT\}=\-\\log P\_\{\\theta\}\(a\\mid Q,y\_\{1\},y\_\{2\},\\ldots,y\_\{K\}\)\(1\)WherePθ\(\.\)P\_\{\\theta\}\(\.\)is the probability distribution defined by the student modelMθM\_\{\\theta\}\. \(ThroughHθ\(\.\)H\_\{\\theta\}\(\.\)and a softmax over the vocabulary\)\. Continuous CoT aims to replace textual reasoning vectors with a sequence of dense latentsz:=\(z1,z2,…,zK\)z:=\(z\_\{1\},z\_\{2\},\\ldots,z\_\{K\}\)\*\*\*Note that number of latents need not equal reasoning steps of a problem\. Here we consider equal case for simplicity\.obtained by

zk=Hθ​\(I\(k−1\)\)I\(k\)=I\(k−1\)⊕zk\\displaystyle z\_\{k\}=H\_\{\\theta\}\(I^\{\(k\-1\)\}\)\\;\\;\\;\\;I^\{\(k\)\}=I^\{\(k\-1\)\}\\oplus z\_\{k\}\(2\)Where,I\(0\)=E​\(Q\)I^\{\(0\)\}=E\(Q\)is the embedded input prompt and eachzi∈ℝdz\_\{i\}\\in\\mathbb\{R\}^\{d\}\.⊕\\oplusdenotes concatenation along the sequence axis\. The Continuous Chain of Thought is therefore defined as the sequence of continuous vectorsz:=z1,z2,…,zKz:=z\_\{1\},z\_\{2\},\\ldots,z\_\{K\}\. The primary goal of all ContinuousCoT methods is to design an effective training recipe to learnzzsuch that the final answeraacan be generated conditioned only onzzandQQ\.

Table 1:Comparison of indirect and direct supervision regimes for ContinuousCoT training\.Indirect methods reuse the model’s own final\-layer activation via recurrence, producing continuous latents\. Direct methods instead map predictions through the embedding table, constraining outputs to the token space\. For indirect methods, the reasoning length \(number of recurrent latent steps\) is a fixed hyperparameter set before training, whereas direct methods operate over variable\-length CoT traces via teacher forcing\.
### 3\.2Prior Work: Indirect Supervision Methods

Coconut\(Haoet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib14)\)uses a curriculum\-based strategy where at each iteration, one of the textual reasoning stepsyiy\_\{i\}is replaced with the corresponding latentziz\_\{i\}starting fromi=0i=0\. The following objective is optimized at each stepii\.

Li=−log⁡Pθ​\(a∣Q,z1,…,zi,yi\+1,…,yK\)\\displaystyle L\_\{i\}=\-\\log P\_\{\\theta\}\(a\\mid Q,z\_\{1\},\\ldots,z\_\{i\},y\_\{i\+1\},\\ldots,y\_\{K\}\)\(3\)This progressively teachesMθM\_\{\\theta\}to generate semantically useful latents and eventually answer based on these latents alone\. CODI\(Shenet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib12)\)improves training efficiency by replacing curriculum with distillation\. The student,MθM\_\{\\theta\}autoregressively generatesKKlatents during training\. Using these latents, it is trained to generate the answer\.

LS​t​u​d​e​n​t=−log⁡Pθ​\(a∣Q,z1,z2,…,zK\)\\displaystyle L\_\{Student\}=\-\\log P\_\{\\theta\}\(a\\mid Q,z\_\{1\},z\_\{2\},\\ldots,z\_\{K\}\)\(4\)The teacherMϕM\_\{\\phi\}\(warm started withθ\\theta\) is trained using standard CoT traces using[Equation1](https://arxiv.org/html/2607.16972#S3.E1)†††Shenet al\.\([2025](https://arxiv.org/html/2607.16972#bib.bib12)\)uses multi\-task training where student and teacher share same weights\. We see the effect of this in more detail in Appendix[F\.1](https://arxiv.org/html/2607.16972#A6.SS1)\. The intermediate layer activations ofMϕM\_\{\\phi\}are distilled toMθM\_\{\\theta\}to guide student training substantially improving training efficiency and downstream performance\.

LK​D=1M​∑l=1D\|s​g​\[hϕl\]−hθl\|\\displaystyle L\_\{KD\}=\\frac\{1\}\{M\}\\sum\_\{l=1\}^\{D\}\|sg\[h\_\{\\phi\}^\{l\}\]\-h\_\{\\theta\}^\{l\}\|\(5\)WhereDDis the number of layers in the model,s​gsgdenotes stop\-gradient, andhlh^\{l\}denotes thell\-th layer activation of the token before the start of answer generation, indexed by the corresponding model \(ϕ\\phifor teacher,θ\\thetafor student\)\. Recently,Weiet al\.\([2026](https://arxiv.org/html/2607.16972#bib.bib15)\)argued that whenKKis increased to larger values, the latents degenerate to similar representations\. To stabilize this effect they introduce step\-level supervision using intermediate reasoning stepsyy\. A Decoder LLMNψN\_\{\\psi\}is trained to autoregressively generate thekk\-th reasoning step conditioned only on the latentzkz\_\{k\}\. Authors show that adding this step\-level supervision to CODI or Coconut improves performance significantly on downstream tasks\.

These works share three characteristics: First, the number of latent steps is fixed prior to training, resulting in fixed\-length reasoning steps\. Second, the sequential, autoregressive generation of latents during training imposes significant overhead\. Finally, these methods leverage textual reasoning stepsindirectlyfor supervision; either through teacher distillation or by using latents to decode them back\.

### 3\.3Prior Work: Direct Supervision Methods

CoLaR\(Tanet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib16)\)proposed a method to utilize reasoning steps directly by compressing embeddings of everyssconsecutive tokens in reasoning traceyyinto a latentziz\_\{i\}\. During training, one token is sampled from eachss\-token group to form a subsampled reasoning trace\. Model,MθM\_\{\\theta\}is then trained via teacher forcing to predict this sampled trace and the final answer\. This approach trains the model’s predictions to approximate a multi\-modal distribution representing all potential tokens within each compressed group\. The aggregated embeddings are modeled as parameterized Gaussians, and a dedicated latent head is trained to predict the mean and standard deviation for each step\. During inference, they use reparameterization trick to sample from this distribution, passing the resulting latent vector to the next autoregressive step\.

### 3\.4C\-MTP: Continuous Chain of Thought via Multi\-Token Prediction

Similar to CoLaR, we view intermediate latent as aggregated token embeddings\. However instead of modeling them as gaussians we provide a simpler recipe using Multi\-Token Prediction\(Gloeckleet al\.,[2024](https://arxiv.org/html/2607.16972#bib.bib17)\)\.‡‡‡We describe our exact architecture in[AppendixA](https://arxiv.org/html/2607.16972#A1)\(Appendix\)

During training, we set a desired compression factorss, the number of tokens per span\. Given the reasoning tracey=\(c1,…,cL\)y=\(c\_\{1\},\\ldots,c\_\{L\}\), we partition it intor=⌈L/s⌉r=\\lceil L/s\\rceilnon\-overlapping spans of lengthss:

Ci=\(c\(i−1\)​s\+1,…,ci​s\),i=1,…,r\\displaystyle C\_\{i\}=\(c\_\{\(i\-1\)s\+1\},\\ldots,c\_\{is\}\),\\quad i=1,\\ldots,r\(6\)Each span is compressed into a single latent by averaging its token embeddings§§§Tanet al\.\([2026](https://arxiv.org/html/2607.16972#bib.bib16)\)divides bys\\sqrt\{s\}instead ofss\. We did not find any difference in our experiments and hence use simple average for simplicity\.:

zi=f​\(Ci\)=1s​∑j=1sE​\(c\(i−1\)​s\+j\)\\displaystyle z\_\{i\}=f\(C\_\{i\}\)=\\frac\{1\}\{s\}\\sum\_\{j=1\}^\{s\}E\\\!\\left\(c\_\{\(i\-1\)s\+j\}\\right\)\(7\)This yields a sequence ofrrlatentsz=\(z1,…,zr\)z=\(z\_\{1\},\\ldots,z\_\{r\}\)that replaces the full reasoning traceyyduring the forward pass\. At each token position, we want to modelPθ​\(zi∣Q,z1​…​zi−1\)P\_\{\\theta\}\(z\_\{i\}\\mid Q,z\_\{1\}\\ldots z\_\{i\-1\}\)\. Using[Equation7](https://arxiv.org/html/2607.16972#S3.E7), we cast the problem of next latent prediction as predicting the tokens composing the latent\. We train the student,MθM\_\{\\theta\}to minimize

Lm​u​l​t​i=−∑i=1r∑t∈Cilog⁡Pθ​\(t∣E​\(Q\),z1,…,zi−1\)\\displaystyle L\_\{multi\}=\-\\sum\_\{i=1\}^\{r\}\\sum\_\{t\\in C\_\{i\}\}\\log P\_\{\\theta\}\(t\\mid E\(Q\),z\_\{1\},\\ldots,z\_\{i\-1\}\)\(8\)The answer supervision stays unchanged

La​n​s=−∑j=1nlog⁡Pθ​\(aj∣E​\(Q\),z1:r\)\\displaystyle L\_\{ans\}=\-\\sum\_\{j=1\}^\{n\}\\log P\_\{\\theta\}\(a\_\{j\}\\mid E\(Q\),z\_\{1:r\}\)\(9\)LM​T​P=Lm​u​l​t​i\+La​n​s\\displaystyle L\_\{MTP\}=L\_\{multi\}\+L\_\{ans\}\(10\)To guide training, we also employ a CoT Teacher for distillation usingLK​DL\_\{KD\}\(Defined in[Equation5](https://arxiv.org/html/2607.16972#S3.E5)\)\. Overall, we employ two training configurations in this work:Warmstart Initwhere we first train a CoT teacher usingLC​o​TL\_\{CoT\}, use it to initialize the student, and then train the student usingLM​T​P\+LK​DL\_\{MTP\}\+L\_\{KD\}; andMultitask Trainingwhere we train the same backbone to optimize bothLC​o​TL\_\{CoT\}andLM​T​P\+LK​DL\_\{MTP\}\+L\_\{KD\}simultaneously\. We found that Warmstart Init performs better in out\-of\-domain tasks and keep it for main experiments, we use multitask training for fair comparison with CODI in relevant experiments\. We present ablations for Warmstart Init & MultiTask Training and the effect ofLK​DL\_\{KD\}in[Section4\.2\.2](https://arxiv.org/html/2607.16972#S4.SS2.SSS2), showing that C\-MTP is effective even without distillation from a teacher; we retainLK​DL\_\{KD\}in the main experiments to provide the strongest possible performance for C\-MTP and to keep the training setup consistent with CODI which also uses distillation\.

During inference, the trainedMθM\_\{\\theta\}first goes through a CoT phase and then answer phase\. In the CoT phase, at each stepiithe model samplessstokens fromPθ\(⋅∣E\(Q\),z1,…,zi−1\)P\_\{\\theta\}\(\\cdot\\mid E\(Q\),z\_\{1\},\\ldots,z\_\{i\-1\}\), aggregates their embeddings to form the next latentziz\_\{i\}via[Equation7](https://arxiv.org/html/2607.16972#S3.E7), and appends it to the latent sequence\. This repeats until the end\-of\-thinking token is among the sampled tokens\. In the answer phase, the model switches to standard autoregressive next\-token prediction conditioned onE​\(Q\)E\(Q\)and the accumulated latentsz1:rz\_\{1:r\}, continuing until the end\-of\-sequence token is produced\.

Table 2:GSM8k\-Aug results on LLaMA\-3\.2\-1B\-Instruct\.Accuracy is averaged over 5 runs \(run\-to\-run deviation<<1%\)\. \#L denotes the mean chain\-of\-thought length \(in steps\) with standard deviation, computed over the full evaluation set\.Training MethodGSM8k\-TestGSM\-HardSVAMPMultiArithAcc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)StandardCoT61\.9427\.17±11\.2427\.17\_\{\\pm 11\.24\}14\.2234\.43±15\.1134\.43\_\{\\pm 15\.11\}64\.9613\.15±5\.3013\.15\_\{\\pm 5\.30\}98\.8915\.46±1\.9615\.46\_\{\\pm 1\.96\}IndirectCoconut¶¶¶Values taken fromShenet al\.\([2025](https://arxiv.org/html/2607.16972#bib.bib12)\)45\.306\.009\.96\.0048\.806\.0090\.106\.00SupervisionCODI53\.036\.0012\.406\.0060\.266\.0099\.116\.00Sim\-CoT53\.956\.0012\.456\.0058\.626\.0097\.786\.00DirectCoLaR\-241\.2312\.99±5\.5312\.99\_\{\\pm 5\.53\}8\.8716\.30±8\.3016\.30\_\{\\pm 8\.30\}52\.466\.31±2\.666\.31\_\{\\pm 2\.66\}90\.567\.39±1\.437\.39\_\{\\pm 1\.43\}SupervisionCoLaR\-332\.659\.01±3\.759\.01\_\{\\pm 3\.75\}6\.7910\.96±6\.3010\.96\_\{\\pm 6\.30\}48\.364\.49±1\.914\.49\_\{\\pm 1\.91\}87\.335\.33±1\.935\.33\_\{\\pm 1\.93\}CoLaR\-429\.606\.80±2\.716\.80\_\{\\pm 2\.71\}6\.607\.66±2\.977\.66\_\{\\pm 2\.97\}46\.823\.47±1\.483\.47\_\{\\pm 1\.48\}88\.114\.13±0\.674\.13\_\{\\pm 0\.67\}C\-MTP\-253\.3012\.07±5\.5312\.07\_\{\\pm 5\.53\}12\.5914\.15±6\.4414\.15\_\{\\pm 6\.44\}62\.105\.14±2\.585\.14\_\{\\pm 2\.58\}95\.336\.29±1\.206\.29\_\{\\pm 1\.20\}C\-MTP\-346\.507\.94±3\.697\.94\_\{\\pm 3\.69\}10\.589\.02±4\.549\.02\_\{\\pm 4\.54\}56\.083\.43±1\.663\.43\_\{\\pm 1\.66\}97\.334\.16±0\.674\.16\_\{\\pm 0\.67\}C\-MTP\-440\.705\.84±2\.715\.84\_\{\\pm 2\.71\}9\.296\.85±4\.346\.85\_\{\\pm 4\.34\}50\.982\.39±1\.472\.39\_\{\\pm 1\.47\}94\.673\.12±0\.563\.12\_\{\\pm 0\.56\}

## 4Experimental Results and Analysis

In this section we discuss the setup, training details, experiments and detailed analysis of the proposed architecture\.

### 4\.1Setup

Datasets and Tasks\.We primarily useGSM8k\-Aug\(Denget al\.,[2024](https://arxiv.org/html/2607.16972#bib.bib9)\), an augmented version of Grade\-School level Math reasoning dataset GSM8k\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.16972#bib.bib18)\)\. It comprises∼\\sim385k training samples of which we reserve 1% for validation and use rest for training\. It is proven effective for training ContinuousCoT methods as it does not contain any natural language but only structured mathematical expressions leading to the final answer such as≪18\.5−8\.5=10≫≪20\+15\+30=65≫\\ll 18\.5\-8\.5=10\\gg\\ll 20\+15\+30=65\\gg\. We also useGSM8k\-Aug\-NL, a version that preserves natural language explanations\. As each sentence perfectly corresponds to one reasoning step in structured trace, we term this dataset as Semi\-Natural\. For experiments with realistic CoT traces, we use challengingMATH\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.16972#bib.bib19)\)problems and generate CoT traces from the target model being trained\. For evaluation, we use the test splits of GSM8k to evaluate in\-domain performance, andGSM\-Hard\(Gaoet al\.,[2023](https://arxiv.org/html/2607.16972#bib.bib21)\),SVAMP\(Patelet al\.,[2021](https://arxiv.org/html/2607.16972#bib.bib20)\)andMultiArith\(Roy and Roth,[2015](https://arxiv.org/html/2607.16972#bib.bib22)\)to evaluate out\-of\-domain generalization on math tasks\. We also useMATHtest set for more complex natural language reasoning evaluation\. See[AppendixB](https://arxiv.org/html/2607.16972#A2)in the appendix for example traces\.

Baselines\.We consider the following baselines: \(1\)CoT\(Chain\-of\-Thought\) finetuning, where the model is trained to generate cot tokens followed by the final answer\. We compare with two indirect supervision methods: \(2\)CODI\(Shenet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib12)\)and \(3\)Sim\-CoT\(Weiet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib15)\)\. For direct supervision we compare with \(4\)CoLaR\(Tanet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib16)\)\. These methods are discussed extensively in[Section3\.2](https://arxiv.org/html/2607.16972#S3.SS2)\. We use their official implementations and training scripts for our experiments\. For CoLaR, we compare with three compression factorss∈\{2,3,4\}s\\in\\\{2,3,4\\\}which approximately correspond to ratio of original chain of thought and generation length of the trained model\.

Training and Implementation Details\.We experiment with two backbones:Llama\-3\.2\-1B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.16972#bib.bib24)\)andQwen2\.5\-1\.5B\-Instruct\(Qwenet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib25)\)\. Following prior works, all experiments use a trainable LoRA module\(Huet al\.,[2021](https://arxiv.org/html/2607.16972#bib.bib26)\)and are initialized with a trained CoT checkpoint\. Full training and implementation details are provided in[AppendixG](https://arxiv.org/html/2607.16972#A7)\(Appendix\)\. For C\-MTP, we use Warmstart Init for the main experiments, and Multitask Training for analysis in[Section4\.2\.1](https://arxiv.org/html/2607.16972#S4.SS2.SSS1)to ensure a fair comparison with CODI\. We present a detailed comparison of the two training configurations along with analysis of effect ofLK​DL\_\{KD\}in[Section4\.2\.2](https://arxiv.org/html/2607.16972#S4.SS2.SSS2)\.

### 4\.2Training on Structured CoT: GSM8k\-Aug

[Tables2](https://arxiv.org/html/2607.16972#S3.T2)and[11](https://arxiv.org/html/2607.16972#A7.T11)compares C\-MTP against the baselines from[Sections3\.2](https://arxiv.org/html/2607.16972#S3.SS2)and[3\.3](https://arxiv.org/html/2607.16972#S3.SS3)on GSM8k\-Aug, reporting downstream accuracy and chain\-of\-thought length\.C\-MTP outperforms CoLaRacross all compression ratios: on LLaMA\-3\.2\-1B\-Instruct, C\-MTP at 4×\\timescompression matches CoLaR at 2×\\times, while converging in∼\\sim5 epochs versus CoLaR’s∼\\sim50, highlighting C\-MTP’s stronger alignment with the pretraining objective\. Across models, neither approach consistently dominates\. On Llama, indirect supervision outperforms direct on in\-distribution data but underperforms on out\-of\-distribution; on Qwen \(See[SectionG\.2](https://arxiv.org/html/2607.16972#A7.SS2)\), this pattern reverses\. The lack of a consistent winner across both models and distribution settings suggests the two approaches are broadly comparable, with relative performance being model\-dependent\. Following sections explore potential reasons why direct supervision can be preferable\.

#### 4\.2\.1Simplicity, Efficiency, and Generalization

The simplicity of the proposed direct supervision approach, C\-MTP, yields practical advantages such as faster training, stronger sample efficiency, and better generalization to unseen problem structures\. We analyze each of these dimensions below, comparing C\-MTP against CODI under controlled settings\.

Fixed Wall\-Clock Time Comparison\.Our analysis in[SectionG\.4](https://arxiv.org/html/2607.16972#A7.SS4)in the appendix reveals that a single C\-MTP training step is approximately2×2\\timesfaster than CODI on LLaMA\-3\.2\-1B, allowing C\-MTP to process roughly twice as many samples within the same wall\-clock budget\. We note that this2×2\\timesfactor represents a lower bound: for larger models like Qwen2\.5\-1\.5B, the sequential memory access overhead of CODI’s latent passes grows more pronounced, widening the gap further\. To maintain wall\-clock parity, we allocate C\-MTP’s speedup toward doubling the training samples while holding epochs constant \(details in[SectionG\.4](https://arxiv.org/html/2607.16972#A7.SS4)\)\. Despite this conservative estimate, C\-MTP\-4 consistently outperforms CODI across all budget levels in[Table12](https://arxiv.org/html/2607.16972#A7.T12)\(Appendix\), with gains on GSM8k\-Test particularly pronounced\.

Sample Efficiency\.To understand how the two training paradigms compare under varying amounts of supervision, we train both methods on randomly sampled subsets of GSM8k\-Aug \(12k, 25k, 50k, 100k and 200k samples\) and evaluate on downstream tasks\. For each subset size, we sweep over a predefined set of hyperparameters and select the best\-performing configuration by validation loss\.[Figure5](https://arxiv.org/html/2607.16972#A7.F5)\(Appendix\) shows accuracy as a function of training set size\. In the low\-data regime \(up to∼\\sim100k samples\), C\-MTP consistently outperforms CODI, with the gap most pronounced at 12k samples\. On in\-distribution tasks \(GSM8k\-Test, GSM\-Hard\), C\-MTP’s advantage is particularly clear, whereas on out\-of\-distribution tasks \(SVAMP, MultiArith\) the gap is smaller and more variable\.

Generalization to Problem Structures\.GSM8k\-Aug\(Denget al\.,[2024](https://arxiv.org/html/2607.16972#bib.bib9)\)was constructed by prompting GPT\-4 to generate problems similar to GSM8k, thereby expanding the dataset∼50×\\sim 50\\timesvia prompting and heuristic filtering\. This process results in substantial structural redundancy: our analysis shows that over 50% of all training samples reduce to only 100 distinct problem structures\. We present more analysis in[AppendixD](https://arxiv.org/html/2607.16972#A4)\(Appendix\)\. To evaluate out\-of\-domain generalization to problem structures not seen during training, we partition the data by held\-out problem structures for each reasoning lengthn∈\{2,3,4\}n\\in\\\{2,3,4\\\}\.

[Table13](https://arxiv.org/html/2607.16972#A7.T13)\(Appendix\) reports in\-domain and out\-of\-domain accuracy for C\-MTP and CODI across reasoning lengthsn∈\{2,3,4\}n\\in\\\{2,3,4\\\}\. C\-MTP generalizes better to held\-out problem structures in all settings\. The gap is particularly clear atn=3n=3andn=4n=4, where CODI’s out\-of\-domain accuracy drops substantially \(e\.g\., 27\.10 and 13\.83 onn=3n=3andn=4n=4respectively\) while C\-MTP retains stronger performance \(42\.86 and 24\.04\)\. In\-domain, the two methods are more competitive, with CODI performing slightly better atn=2n=2while C\-MTP leads at longer reasoning chains\. Overall, these results suggest that the sequential training of indirect supervision methods leads to overfitting to the specific problem structures seen during training\. Full details on partitioning and training provided in[SectionG\.5](https://arxiv.org/html/2607.16972#A7.SS5)\(Appendix\)\.

#### 4\.2\.2Ablation Studies

We ablate two design choices of C\-MTP on structured chain of thought traces \(GSM8k\-Aug\) with LLaMA\-3\.2\-1B\-Instruct \([Table3](https://arxiv.org/html/2607.16972#S4.T3)\)\. We present full results with reasoning lengths in[SectionsF\.1](https://arxiv.org/html/2607.16972#A6.SS1)and[F\.2](https://arxiv.org/html/2607.16972#A6.SS2)\(Appendix\)\.

Effect of distillation \(LK​DL\_\{KD\}\)\.Removing the teacher distillation loss leaves performance largely unchanged ats∈\{2,3\}s\\in\\\{2,3\\\}\(e\.g\.,53\.30→52\.1253\.30\\to 52\.12on GSM8k\-Test for C\-MTP\-2\), with a more noticeable drop only at the most aggressive compressions=4s=4\. C\-MTP thus learns effectively on its own: the warmstart initialization and the simplicity of the training objective, rather than distillation, drive its performance\. We retainLK​DL\_\{KD\}in the main experiments for strongest performance and to keep the setup consistent with CODI\.

Warmstart vs\. Multitask training\.Warmstart Init and Multitask Training yield comparable accuracy \(C\-MTP\-3: 46\.50 vs\. 48\.29 on GSM8k\-Test\)\. We use Warmstart Init for the main experiments as the more principled choice, and Multitask Training only where a fair comparison with CODI requires it \([Section4\.2\.1](https://arxiv.org/html/2607.16972#S4.SS2.SSS1)\)\.

Table 3:C\-MTP ablations on GSM8k\-Aug \(LLaMA\-3\.2\-1B\-Instruct\)\.Accuracy \(↑\\uparrow\), averaged over 5 runs\. Default C\-MTP rows use Warmstart Init withLK​DL\_\{KD\}; sub\-rows ablate the distillation loss \(w/oLK​DL\_\{KD\}\) and the training configuration \(Multitask\)\. Full tables with reasoning lengths in[SectionsF\.1](https://arxiv.org/html/2607.16972#A6.SS1)and[F\.2](https://arxiv.org/html/2607.16972#A6.SS2)\.Figure 2:Fraction of self\-consistent expressions by absolute position in C\-MTP\-X and CoT traces\. We only plot for samples for which CoT arrives at the right answer\. Trained on GSM8k\-Aug, Eval: GSM8k\-Test\.![Refer to caption](https://arxiv.org/html/2607.16972v1/assets/experroraccum.png)Table 4:Training on Semi\-Natural CoT: GSM8k\-Aug\-NL\. Accuracy is averaged over 5 runs \(run\-to\-run deviation<<1%\)\. \#L denotes the mean chain\-of\-thought length \(in steps\) with standard deviation, computed over the full evaluation set\.Training MethodGSM8k\-TestGSM\-HardSVAMPMultiArithAcc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)StandardCoT53\.6870\.04±30\.0270\.04\_\{\\pm 30\.02\}13\.0478\.38±33\.0178\.38\_\{\\pm 33\.01\}60\.8836\.77±18\.5236\.77\_\{\\pm 18\.52\}96\.3341\.64±10\.8241\.64\_\{\\pm 10\.82\}IndirectCODI47\.826\.0010\.996\.0056\.826\.0095\.566\.00SupervisionSim\-CoT44\.356\.0010\.116\.0055\.166\.0097\.896\.00DirectCoLaR\-223\.4635\.63±14\.4635\.63\_\{\\pm 14\.46\}5\.4338\.99±15\.0238\.99\_\{\\pm 15\.02\}37\.7019\.15±9\.4819\.15\_\{\\pm 9\.48\}72\.2220\.69±7\.5920\.69\_\{\\pm 7\.59\}SupervisionCoLaR\-319\.8823\.31±10\.0123\.31\_\{\\pm 10\.01\}4\.6125\.55±10\.6525\.55\_\{\\pm 10\.65\}33\.8612\.14±5\.5612\.14\_\{\\pm 5\.56\}67\.6713\.86±3\.8213\.86\_\{\\pm 3\.82\}CoLaR\-418\.0618\.15±8\.2818\.15\_\{\\pm 8\.28\}3\.7520\.36±9\.3320\.36\_\{\\pm 9\.33\}33\.169\.43±4\.519\.43\_\{\\pm 4\.51\}63\.8910\.96±4\.6810\.96\_\{\\pm 4\.68\}C\-MTP\-243\.4934\.00±15\.0434\.00\_\{\\pm 15\.04\}10\.1337\.32±15\.8737\.32\_\{\\pm 15\.87\}56\.6016\.53±8\.8316\.53\_\{\\pm 8\.83\}91\.2218\.74±5\.0418\.74\_\{\\pm 5\.04\}C\-MTP\-334\.0422\.74±10\.1822\.74\_\{\\pm 10\.18\}7\.7324\.64±10\.7824\.64\_\{\\pm 10\.78\}48\.2410\.57±5\.5710\.57\_\{\\pm 5\.57\}80\.3312\.42±3\.4812\.42\_\{\\pm 3\.48\}C\-MTP\-425\.6017\.13±7\.7917\.13\_\{\\pm 7\.79\}6\.2019\.00±9\.2519\.00\_\{\\pm 9\.25\}38\.787\.98±4\.167\.98\_\{\\pm 4\.16\}67\.119\.26±2\.899\.26\_\{\\pm 2\.89\}

### 4\.3Training on Semi\-Natural CoT: GSM8k\-Aug\-NL

[Table4](https://arxiv.org/html/2607.16972#S4.T4)compares C\-MTP and baselines on GSM8k\-Aug\-NL, which contains verbose natural language reasoning traces averaging∼\\sim70 steps — roughly2\.5×2\.5\\timeslonger than GSM8k\-Aug\.C\-MTP outperforms CoLaRacross all compression ratios, consistent with[Section4\.2](https://arxiv.org/html/2607.16972#S4.SS2)\. More strikingly, the gap between indirect and direct Supervision widens considerably relative to the structured setting: CODI achieves 47\.82% on GSM8k\-Test with just 6 latent steps, while CoLaR\-2 collapses to 23\.46%, and even C\-MTP\-2, despite using∼\\sim34 steps, reaches only 43\.49%\.

Two factors explain why verbose data disproportionately hurts parallel methods\. First, indirect supervision is inherently suited to compression: a fixed latent budget forces the model to discard filler tokens and irrelevant content, whereas parallel methods must predict the full verbose trace without any such pressure\. Second, teacher forcing causes a train\-test mismatch: at inference, the model conditions on its own prior predictions rather than ground\-truth CoT traces, with errors cascading particularly severely through math expressions\. As we see in the realistic setting, this effect is especially pronounced on math expressions where a single erroneous token can effect the accuracy of the entire CoT trace\.

### 4\.4Analysis of C\-MTP Generations

Training on GSM8k\-Aug teaches models to reason using only expressions relevant to computing the final answer\. We compute the mathematical validity of each generated expression for each sample and plot the average fraction of valid expressions at each position in the generated CoT trace for both C\-MTP and CoT in[Figure2](https://arxiv.org/html/2607.16972#S4.F2)\. We see that C\-MTP starts with a high fraction at the beginning of the trace and falls off towards the end, especially fors=4s=4where the model is forced to compress more\. We further analyze generations upon training on GSM8k\-Aug\-NL, where the traces contain both math expressions and natural language\. We find that the model gets things wrong primarily in math expressions \(See examples in[AppendixE](https://arxiv.org/html/2607.16972#A5)\) especially when a span boundary occurs in between\. Upon an incorrect generation the model is forced to condition on its own erroneous output \(error accumulation\), which leads to an incorrect answer\. We detail few selected examples in[SectionG\.6](https://arxiv.org/html/2607.16972#A7.SS6)\(Appendix\)\.

We further experiment with CommonsenseQA\(Talmoret al\.,[2019](https://arxiv.org/html/2607.16972#bib.bib67)\), a commonsense multiple choice question answering dataset paired with natural language CoTs generated by GPT4omini\(Shenet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib12)\)in[SectionG\.7](https://arxiv.org/html/2607.16972#A7.SS7)\(Appendix\)\. Interestingly, we do not see a severe degradation of C\-MTP with increasing span lengths, validating our observation that errors cascade mainly through precise tokens such as math expressions\.

Table 5:Results on MATH across two backbones\. Models that perform best using greedy decoding report SD of 0\.0\.![Refer to caption](https://arxiv.org/html/2607.16972v1/x2.png)Figure 3:LLaMA\-3\.2\-1B\-Instruct Results at a glance\.A comparative analysis of weighted average accuracy \(%\) versus the average number of reasoning steps \(log scale\) for StandardCoT and two regimes \(direct and indirect\) of ContinuousCoT across three levels of trace difficulty\. On simpler tasks \(structured and semi\-natural\), both regimes perform competitively with Standard CoT\. However, on complex, realistic tasks, performance across both regimes collapses to roughly 30–35% of CoT accuracy\. This demonstrates that simplified benchmarks overstate the practical effectiveness of continuous CoT models in realistic settings\.
### 4\.5Training on Realistic CoT: MATH

To evaluate how direct and indirect supervision scale to realistic CoT traces, we compare the strongest representative from each regime: CODI for indirect supervision and C\-MTP for direct supervision\. Since our earlier experiments already identify the best method in each regime, and realistic traces are expensive to train on, we use a single representative per regime\. We generate answers to all problems in the train set, filter out wrong answers and use the correct generations for training\. We finetune CoT\-SFT, C\-MTP\-2, CODI\-6/10/20 to compare discrete CoT, direct and indirect supervision methods\.

In[Table14](https://arxiv.org/html/2607.16972#A7.T14), we see that both methods drastically fail achieving only∼\\sim12\-15% accuracy compared to∼\\sim53% for CoT\-SFT for Qwen2\.5\-1\.5B\-Instruct\. Experiments with Llama show similar trends\. Although CODI’s reasoning traces are not interpretable, analysis of traces generated by C\-MTP underline similar issues of error accumulation seen in semi\-natural tasks\. The model generates the right natural language tokens but makes arithmetic errors, drops signs or numbers or produces malformed expressions, which leads to an irrecoverable error in the trace and an incorrect final answer, suggesting that future ability to improve such methods may require a combination of the two\. Direct supervision may be essential for its efficiency and alignment with the token\-based training objective and indirect supervision may be essential for its ability to compress long traces into fixed latent budgets and avoid train\-test mismatch\.

## 5Limitations and Future Work

Our work centers on two popular LLMs in the 1\-1\.5B size range evaluated on mathematical reasoning tasks, which may limit the generality of our findings\. We focus on math because prior work has shown that latent reasoning methods struggle to match CoT in this domain, making it a natural testbed\. Extending the evaluation of C\-MTP to tasks such as code, and multi\-hop question answering with larger scale models is a natural next step\. More fundamentally, our realistic\-trace results suggest that neither direct nor indirect supervision alone suffices: a promising direction is hybrid approaches that adaptively switch between token\-based and continuous reasoning\(Shiet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib56); Xuet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib59); Zhuet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib61)\), which may be essential for scaling ContinuousCoT to realistic settings\.

## 6Conclusion

We propose C\-MTP, a direct supervision method using multi\-token prediction to simulate ContinuousCoT, which substantially outperforms comparable approaches\. Through systematic analysis of direct and indirect supervision across Structured, Semi\-Natural, and Realistic CoT traces, we find: direct supervision \(C\-MTP\) is preferable on structured tasks for its efficiency; indirect supervision dominates on verbose reasoning tasks, better compressing long traces into fixed latent budgets where direct methods fail and face train\-test mismatch; and on realistic traces, both regimes fall far short of CoT\-SFT \([Figure1](https://arxiv.org/html/2607.16972#S0.F1)\)\. Our results suggest that structured and semi\-natural training significantly overstates the effectiveness of Continuous CoT methods\.

## Acknowledgements

This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise\. The work is partially funded by NSF CAREER award 2443271 and NSF award RI\-2521091\.

## References

- N\. Butt, A\. Kwiatkowski, I\. Labiad, J\. Kempe, and Y\. Ollivier \(2025\)Soft tokens, hard truths\.External Links:2509\.19170,[Link](https://arxiv.org/abs/2509.19170)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- T\. Cai, Y\. Li, Z\. Geng, H\. Peng, J\. D\. Lee, D\. Chen, and T\. Dao \(2024\)Medusa: simple LLM inference acceleration framework with multiple decoding heads\.pp\. 5209–5235\.External Links:[Link](https://proceedings.mlr.press/v235/cai24b.html)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p4.1)\.
- J\. Cheng and B\. V\. Durme \(2024\)Compressed chain of thought: efficient reasoning through dense representations\.CoRRabs/2412\.13171\.External Links:[Link](https://doi.org/10.48550/arXiv.2412.13171),[Document](https://dx.doi.org/10.48550/ARXIV.2412.13171),2412\.13171Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p3.1)\.
- H\. W\. Chung, L\. Hou, S\. Longpre, B\. Zoph, Y\. Tay, W\. Fedus, Y\. Li, X\. Wang, M\. Dehghani, S\. Brahma, A\. Webson, S\. S\. Gu, Z\. Dai, M\. Suzgun, X\. Chen, A\. Chowdhery, A\. Castro\-Ros, M\. Pellat, K\. Robinson, D\. Valter, S\. Narang, G\. Mishra, A\. Yu, V\. Y\. Zhao, Y\. Huang, A\. M\. Dai, H\. Yu, S\. Petrov, E\. H\. Chi, J\. Dean, J\. Devlin, A\. Roberts, D\. Zhou, Q\. V\. Le, and J\. Wei \(2024\)Scaling instruction\-finetuned language models\.J\. Mach\. Learn\. Res\.25,pp\. 70:1–70:53\.External Links:[Link](https://jmlr.org/papers/v25/23-0870.html)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.CoRRabs/2110\.14168\.External Links:[Link](https://arxiv.org/abs/2110.14168),2110\.14168Cited by:[§1](https://arxiv.org/html/2607.16972#S1.p2.1),[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p1.2)\.
- DeepSeek\-AI and team \(2025\)DeepSeek\-v3 technical report\.External Links:2412\.19437,[Link](https://arxiv.org/abs/2412.19437)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p4.1)\.
- Y\. Deng, Y\. Choi, and S\. M\. Shieber \(2024\)From explicit cot to implicit cot: learning to internalize cot step by step\.CoRRabs/2405\.14838\.External Links:[Link](https://doi.org/10.48550/arXiv.2405.14838),[Document](https://dx.doi.org/10.48550/ARXIV.2405.14838),2405\.14838Cited by:[§G\.5](https://arxiv.org/html/2607.16972#A7.SS5.p1.1),[§1](https://arxiv.org/html/2607.16972#S1.p2.1),[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p1.2),[§4\.2\.1](https://arxiv.org/html/2607.16972#S4.SS2.SSS1.p4.2)\.
- T\. Fu, Y\. You, Z\. Chen, G\. Dai, H\. Yang, and Y\. Wang \(2025\)Think\-at\-hard: selective latent iterations to improve reasoning language models\.CoRRabs/2511\.08577\.External Links:[Link](https://doi.org/10.48550/arXiv.2511.08577),[Document](https://dx.doi.org/10.48550/ARXIV.2511.08577),2511\.08577Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)PAL: program\-aided language models\.InProceedings of the 40th International Conference on Machine LearningProceedings of the 2015 Conference on Empirical Methods in Natural Language ProcessingThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9\-14, 2023The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025Proceedings of the 2025 Conference on Empirical Methods in Natural Language ProcessingForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, J\. Scarlett, L\. Màrquez, C\. Callison\-Burch, J\. Su, S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, A\. Oh, S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, A\. Oh, A\. Rogers, J\. Boyd\-Graber, N\. Okazaki, S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, A\. Oh, A\. Rogers, J\. L\. Boyd\-Graber, N\. Okazaki, C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, V\. Peng, R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, F\. Berkenkamp, J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Proceedings of Machine Learning ResearchFindings of ACLProceedings of Machine Learning Research, Vol\.202,pp\. 10764–10799\.External Links:[Link](https://proceedings.mlr.press/v202/gao23f.html)Cited by:[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p1.2)\.
- J\. Geiping, S\. McLeish, N\. Jain, J\. Kirchenbauer, S\. Singh, B\. R\. Bartoldson, B\. Kailkhura, A\. Bhatele, and T\. Goldstein \(2025\)Scaling up test\-time compute with latent reasoning: A recurrent depth approach\.CoRRabs/2502\.05171\.External Links:[Link](https://doi.org/10.48550/arXiv.2502.05171),[Document](https://dx.doi.org/10.48550/ARXIV.2502.05171),2502\.05171Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- F\. Gloeckle, B\. Y\. Idrissi, B\. Rozière, D\. Lopez\-Paz, and G\. Synnaeve \(2024\)Better & faster large language models via multi\-token prediction\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research,pp\. 15706–15734\.External Links:[Link](https://proceedings.mlr.press/v235/gloeckle24a.html)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p4.1),[§3\.4](https://arxiv.org/html/2607.16972#S3.SS4.p1.1)\.
- S\. Goyal, Z\. Ji, A\. S\. Rawat, A\. K\. Menon, S\. Kumar, and V\. Nagarajan \(2024\)Think before you speak: training language models with pause tokens\.External Links:[Link](https://openreview.net/forum?id=ph04CRkPdC)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- H\. A\. Gozeten, M\. E\. Ildiz, X\. Zhang, H\. Harutyunyan, A\. S\. Rawat, and S\. Oymak \(2025\)Continuous chain of thought enables parallel exploration and reasoning\.CoRRabs/2505\.23648\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.23648),[Document](https://dx.doi.org/10.48550/ARXIV.2505.23648),2505\.23648Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, and A\. R\. et al \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p3.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv e\-prints,pp\. arXiv–2501\.External Links:[Link](https://arxiv.org/abs/2501.12948)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- S\. Hao, S\. Sukhbaatar, D\. Su, X\. Li, Z\. Hu, J\. E\. Weston, and Y\. Tian \(2025\)Training large language models to reason in a continuous latent space\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Itxz7S4Ip3)Cited by:[§1](https://arxiv.org/html/2607.16972#S1.p2.1),[§2](https://arxiv.org/html/2607.16972#S2.p2.1),[§3\.2](https://arxiv.org/html/2607.16972#S3.SS2.p1.4)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.NeurIPS\.Cited by:[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p1.2)\.
- C\. Hsieh, C\. Li, C\. Yeh, H\. Nakhost, Y\. Fujii, A\. Ratner, R\. Krishna, C\. Lee, and T\. Pfister \(2023\)Distilling step\-by\-step\! outperforming larger language models with less training data and smaller model sizes\.pp\. 8003–8017\.External Links:[Link](https://doi.org/10.18653/v1/2023.findings-acl.507),[Document](https://dx.doi.org/10.18653/V1/2023.FINDINGS-ACL.507)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2021\)LoRA: low\-rank adaptation of large language models\.External Links:2106\.09685,[Link](https://arxiv.org/abs/2106.09685)Cited by:[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p3.1)\.
- T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa \(2022\)Large language models are zero\-shot reasoners\.External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.16972#S1.p1.1),[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- W\. Merrill and A\. Sabharwal \(2024\)The expressive power of transformers with chain of thought\.External Links:[Link](https://openreview.net/forum?id=NjNGlPh8Wh)Cited by:[§1](https://arxiv.org/html/2607.16972#S1.p1.1)\.
- OpenAI \(2026\)OpenAI o1 system card\.External Links:2412\.16720,[Link](https://arxiv.org/abs/2412.16720)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- A\. Patel, S\. Bhattamishra, and N\. Goyal \(2021\)Are NLP models really able to solve simple math word problems?\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 2080–2094\.External Links:[Link](https://aclanthology.org/2021.naacl-main.168/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.168)Cited by:[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p1.2)\.
- J\. Pfau, W\. Merrill, and S\. R\. Bowman \(2024\)Let’s think dot by dot: hidden computation in transformer language models\.CoRRabs/2404\.15758\.External Links:[Link](https://doi.org/10.48550/arXiv.2404.15758),[Document](https://dx.doi.org/10.48550/ARXIV.2404.15758),2404\.15758Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p3.1)\.
- S\. Roy and D\. Roth \(2015\)Solving general arithmetic word problems\.Lisbon, Portugal,pp\. 1743–1752\.External Links:[Link](https://aclanthology.org/D15-1202/),[Document](https://dx.doi.org/10.18653/v1/D15-1202)Cited by:[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p1.2)\.
- A\. Saparov and H\. He \(2023\)Language models are greedy reasoners: A systematic formal analysis of chain\-of\-thought\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/forum?id=qFVVBzXxR2V)Cited by:[§1](https://arxiv.org/html/2607.16972#S1.p2.1)\.
- N\. Saunshi, N\. Dikkala, Z\. Li, S\. Kumar, and S\. J\. Reddi \(2025\)Reasoning with latent thoughts: on the power of looped transformers\.External Links:[Link](https://openreview.net/forum?id=din0lGfZFd)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- C\. Shao, D\. Li, F\. Meng, and J\. Zhou \(2025\)Continuous autoregressive language models\.External Links:2510\.27688,[Link](https://arxiv.org/abs/2510.27688)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p3.1)\.
- Z\. Shen, H\. Yan, L\. Zhang, Z\. Hu, Y\. Du, and Y\. He \(2025\)CODI: compressing chain\-of\-thought into continuous space via self\-distillation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 677–693\.External Links:[Link](https://doi.org/10.18653/v1/2025.emnlp-main.36),[Document](https://dx.doi.org/10.18653/V1/2025.EMNLP-MAIN.36)Cited by:[§G\.7](https://arxiv.org/html/2607.16972#A7.SS7.p1.1),[Appendix H](https://arxiv.org/html/2607.16972#A8.p1.1),[§1](https://arxiv.org/html/2607.16972#S1.p2.1),[§2](https://arxiv.org/html/2607.16972#S2.p3.1),[§3\.2](https://arxiv.org/html/2607.16972#S3.SS2.p1.7),[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p2.1),[§4\.4](https://arxiv.org/html/2607.16972#S4.SS4.p2.1),[footnote †](https://arxiv.org/html/2607.16972#footnote2),[footnote ¶](https://arxiv.org/html/2607.16972#footnote5)\.
- D\. Shi, A\. Asi, K\. Li, X\. Yuan, L\. Pan, W\. Lee, and W\. Xiao \(2026\)SwiReasoning: switch\-thinking in latent and explicit for pareto\-superior reasoning llms\.External Links:2510\.05069,[Link](https://arxiv.org/abs/2510.05069)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p3.1),[§5](https://arxiv.org/html/2607.16972#S5.p1.1)\.
- A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)CommonsenseQA: a question answering challenge targeting commonsense knowledge\.Minneapolis, Minnesota,pp\. 4149–4158\.External Links:[Link](https://aclanthology.org/N19-1421/),[Document](https://dx.doi.org/10.18653/v1/N19-1421)Cited by:[§G\.7](https://arxiv.org/html/2607.16972#A7.SS7.p1.1),[§4\.4](https://arxiv.org/html/2607.16972#S4.SS4.p2.1)\.
- W\. Tan, J\. Li, J\. Ju, Z\. Luo, R\. Song, and J\. Luan \(2026\)Think silently, think fast: dynamic latent compression of LLM reasoning chains\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=AQsko3PPUe)Cited by:[Appendix H](https://arxiv.org/html/2607.16972#A8.p1.1),[§1](https://arxiv.org/html/2607.16972#S1.p3.1),[§2](https://arxiv.org/html/2607.16972#S2.p3.1),[§3\.3](https://arxiv.org/html/2607.16972#S3.SS3.p1.5),[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p2.1),[footnote §](https://arxiv.org/html/2607.16972#footnote4)\.
- Y\. Tang, L\. Dong, Y\. Hao, Q\. Dong, F\. Wei, and J\. Gu \(2026\)Multiplex thinking: reasoning via token\-wise branch\-and\-merge\.CoRRabs/2601\.08808\.External Links:[Link](https://doi.org/10.48550/arXiv.2601.08808),[Document](https://dx.doi.org/10.48550/ARXIV.2601.08808),2601\.08808Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- L\. Wang, W\. Xu, Y\. Lan, Z\. Hu, Y\. Lan, R\. K\. Lee, and E\. Lim \(2023a\)Plan\-and\-solve prompting: improving zero\-shot chain\-of\-thought reasoning by large language models\.Toronto, Canada,pp\. 2609–2634\.External Links:[Link](https://aclanthology.org/2023.acl-long.147/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.147)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023b\)Self\-consistency improves chain of thought reasoning in language models\.External Links:[Link](https://openreview.net/forum?id=1PL1NIMMrw)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- J\. Wei, M\. Bosma, V\. Y\. Zhao, K\. Guu, A\. W\. Yu, B\. Lester, N\. Du, A\. M\. Dai, and Q\. V\. Le \(2022a\)Finetuned language models are zero\-shot learners\.External Links:[Link](https://openreview.net/forum?id=gEZrGCozdqR)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022b\)Chain\-of\-thought prompting elicits reasoning in large language models\.External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.16972#S1.p1.1),[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- X\. Wei, X\. Liu, Y\. Zang, X\. Dong, Y\. Cao, J\. Wang, X\. Qiu, and D\. Lin \(2026\)SIM\-cot: supervised implicit chain\-of\-thought\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=6YRJ4jmVQl)Cited by:[§1](https://arxiv.org/html/2607.16972#S1.p2.1),[§2](https://arxiv.org/html/2607.16972#S2.p3.1),[§3\.2](https://arxiv.org/html/2607.16972#S3.SS2.p1.22),[§4\.1](https://arxiv.org/html/2607.16972#S4.SS1.p2.1)\.
- H\. Wu, Z\. Teng, and K\. Tu \(2025a\)Parallel continuous chain\-of\-thought with jacobi iteration\.Suzhou, China,pp\. 914–926\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.47/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.47),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p3.1)\.
- J\. Wu, J\. Lu, Z\. Ren, G\. Hu, Z\. Wu, D\. Dai, and H\. Wu \(2025b\)LLMs are single\-threaded reasoners: demystifying the working mechanism of soft thinking\.External Links:2508\.03440,[Link](https://arxiv.org/abs/2508.03440)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- K\. Xu and I\. Sato \(2025\)A formal comparison between chain\-of\-thought and latent thought\.CoRRabs/2509\.25239\.External Links:[Link](https://doi.org/10.48550/arXiv.2509.25239),[Document](https://dx.doi.org/10.48550/ARXIV.2509.25239),2509\.25239Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- X\. Xu, T\. Yu, X\. Chen, H\. Wang, J\. McAuley, and S\. Mitra \(2026\)ThinkRouter: efficient reasoning via routing thinking between latent and discrete spaces\.External Links:2602\.11683,[Link](https://arxiv.org/abs/2602.11683)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p3.1),[§5](https://arxiv.org/html/2607.16972#S5.p1.1)\.
- Z\. Yue, B\. Jin, H\. Zeng, H\. Zhuang, Z\. Qin, J\. Yoon, L\. Shang, J\. Han, and D\. Wang \(2025\)Hybrid latent reasoning via reinforcement learning\.External Links:2505\.18454,[Link](https://arxiv.org/abs/2505.18454)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p3.1)\.
- E\. Zelikman, Y\. Wu, J\. Mu, and N\. D\. Goodman \(2022\)STaR: bootstrapping reasoning with reasoning\.External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/639a9a172c044fbb64175b5fad42e9a5-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- B\. Zeng, S\. Song, S\. Huang, Y\. Wang, H\. Li, Z\. He, X\. Wang, Z\. Li, and Z\. Lin \(2026\)PonderLM: pretraining language models to ponder in continuous space\.External Links:2505\.20674,[Link](https://arxiv.org/abs/2505.20674)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- J\. Zhang, Y\. Zhu, M\. Sun, Y\. Luo, S\. Qiao, L\. Du, D\. Zheng, H\. Chen, and N\. Zhang \(2025a\)LightThinker: thinking step\-by\-step compression\.External Links:2502\.15589,[Link](https://arxiv.org/abs/2502.15589)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p3.1)\.
- Z\. Zhang, X\. He, W\. Yan, A\. Shen, C\. Zhao, S\. Wang, Y\. Shen, and X\. E\. Wang \(2025b\)Soft thinking: unlocking the reasoning potential of llms in continuous concept space\.CoRRabs/2505\.15778\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.15778),[Document](https://dx.doi.org/10.48550/ARXIV.2505.15778),2505\.15778Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- Z\. Zhang, A\. Zhang, M\. Li, and A\. Smola \(2023\)Automatic chain of thought prompting in large language models\.External Links:[Link](https://openreview.net/forum?id=5NTt8GFjUHkr)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- G\. Zhao, R\. Xie, A\. Wang, S\. Li, H\. Xie, and X\. Sun \(2026\)Self\-distillation for multi\-token prediction\.CoRRabs/2603\.23911\.External Links:[Link](https://doi.org/10.48550/arXiv.2603.23911),[Document](https://dx.doi.org/10.48550/ARXIV.2603.23911),2603\.23911Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p4.1)\.
- D\. Zhou, N\. Schärli, L\. Hou, J\. Wei, N\. Scales, X\. Wang, D\. Schuurmans, C\. Cui, O\. Bousquet, Q\. V\. Le, and E\. H\. Chi \(2023\)Least\-to\-most prompting enables complex reasoning in large language models\.External Links:[Link](https://openreview.net/forum?id=WZH7099tgfM)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p1.1)\.
- H\. Zhu, S\. Hao, Z\. Hu, J\. Jiao, S\. Russell, and Y\. Tian \(2025a\)Reasoning by superposition: A theoretical perspective on chain of continuous thought\.CoRRabs/2505\.12514\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.12514),[Document](https://dx.doi.org/10.48550/ARXIV.2505.12514),2505\.12514Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- R\. Zhu, Z\. Wang, K\. Hua, T\. Zhang, Z\. Li, H\. Que, B\. Wei, Z\. Wen, F\. Yin, H\. Xing, L\. Li, J\. Shi, K\. Ma, S\. Li, T\. Kergan, A\. Smith, X\. Qu, M\. Hui, B\. Wu, Q\. Min, H\. Huang, X\. Zhou, W\. Ye, J\. Liu, J\. Yang, Y\. Shi, C\. Lin, E\. Zhao, T\. Cai, G\. Zhang, W\. Huang, Y\. Bengio, and J\. Eshraghian \(2025b\)Scaling latent reasoning via looped language models\.CoRRabs/2510\.25741\.External Links:[Link](https://doi.org/10.48550/arXiv.2510.25741),[Document](https://dx.doi.org/10.48550/ARXIV.2510.25741),2510\.25741Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.
- Y\. Zhu, J\. Zhang, Z\. Wan, Y\. Luo, S\. Qiao, Z\. Gui, D\. Zheng, L\. Liang, H\. Chen, and N\. Zhang \(2026\)LightThinker\+\+: from reasoning compression to memory management\.External Links:2604\.03679,[Link](https://arxiv.org/abs/2604.03679)Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p3.1),[§5](https://arxiv.org/html/2607.16972#S5.p1.1)\.
- J\. Zou, Y\. Xiong, and Y\. Liu \(2026\)Capabilities and fundamental limits of latent chain\-of\-thought\.CoRRabs/2602\.01148\.External Links:[Link](https://doi.org/10.48550/arXiv.2602.01148),[Document](https://dx.doi.org/10.48550/ARXIV.2602.01148),2602\.01148Cited by:[§2](https://arxiv.org/html/2607.16972#S2.p2.1)\.

## Table of Contents

- A\.MTP Architecture
- B\.Example of CoT Traces for Three Difficulties
- C\.Timing Analysis for Fixed Wallclock Time Experiments
- D\.Analysis of Problem Structures in GSM8k\-Aug
- E\.Examples of Error Accumulation during MTP Inference
- F\.C\-MTP Ablation Studies - F\.1Multi\-Task Training vs Warmstart Initialization - F\.2Ablation on C\-MTP: Training withoutLK​DL\_\{KD\}
- G\.Extended Experimental Results - G\.1Setup Details - G\.2Training on Structured CoT: GSM8k\-Aug - G\.3Sample Efficiency - G\.4Fixed WallClock Time Comparison - G\.5Generalization to Problem Structures - G\.6Full Example of MTP Error Accumulation - G\.7Experiments on CommonsenseQA
- H\.CoLaR \- CODI Implementation Discrepancy

## Appendix AMTP Architecture

Standard LLMs generate a single probability distribution over𝒱\\mathcal\{V\}\. To generate multiple distributions from the same backbone, we train a small MLP to project the final layer activation into activations corresponding to multiple tokens\. We then use the same output embedding matrix to generate logits for all tokens\.

At each thinking stepii, the hidden state is obtained by passing the prompt and all predicted latents through the model:

hi=Hθ​\(E​\(Q\),z1,…,zi−1\)∈ℝd\\displaystyle h\_\{i\}=H\_\{\\theta\}\(E\(Q\),\\,z\_\{1\},\\ldots,z\_\{i\-1\}\)\\in\\mathbb\{R\}^\{d\}\(11\)A 2\-layer MLP:g∈ℝd×s​dg\\in\\mathbb\{R\}^\{d\\times sd\}maps this tosstoken representations simultaneously:

Oi=reshape​\(g​\(hi\)\)∈ℝs×d\\displaystyle O\_\{i\}=\\mathrm\{reshape\}\(g\(h\_\{i\}\)\)\\in\\mathbb\{R\}^\{s\\times d\}\(12\)RowOi​\[j\]O\_\{i\}\[j\]is the predicted representation for thejj\-th token of the next spanCi\+1C\_\{i\+1\}\. Logits are obtained by projecting through the output embedding matrixE∈ℝ\|𝒱\|×dE\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}:

Pθ​\(Ci\+1​\[j\]∣E​\(Q\),z1:i\)=softmax​\(Oi​\[j\]⋅E′\)\\displaystyle P\_\{\\theta\}\(C\_\{i\+1\}\[j\]\\mid E\(Q\),z\_\{1:i\}\)=\\mathrm\{softmax\}\\\!\\left\(O\_\{i\}\[j\]\\cdot E^\{\\prime\}\\;\\right\)\(13\)A single forward pass thus produces allsstoken distributions in parallel, with no sequential decoding within a span\. We then sample one token from each distribution to form the next spanCi\+1C\_\{i\+1\}, and aggregate their embeddings to obtain the next latentzi\+1z\_\{i\+1\}via[Equation7](https://arxiv.org/html/2607.16972#S3.E7)\.

## Appendix BExample of CoT Traces for Three Difficulties

Bella bought 18 packs of chocolates each containing 12 chocolates\. If Bella gives away 6 packs, how many dozens of chocolates will she have left?

StructuredSemi\-NaturalRealistic≪\\ll18\*12=216≫\\gg
≪\\ll216\-6\*12=144≫\\gg
≪\\ll144/12=12≫\\ggBella initially had 18 packs \* 12 chocolates/pack = 216 chocolates\.
After giving away chocolates, Bella has 216 chocolates \- 6 packs \* 12 chocolates/pack = 144 chocolates left\.
Thus, Bella is left with 144 chocolates / 12 = 12 dozens of chocolates\.To determine how many dozens of chocolates Bella has left after giving some away, we can follow these steps:
1\. Calculate the total number of chocolates Bella initially had:
\- Each pack contains 12 chocolates\.
\- Bella bought 18 packs\.
\- Therefore, the total number of chocolates is:
18​packs×12​chocolates per pack=216​chocolates18\\text\{ packs\}\\times 12\\text\{ chocolates per pack\}=216\\text\{ chocolates\}
2\. Determine the number of chocolates in the packs Bella gave away:
\- Bella gave away 6 packs\.
\- Each pack contains 12 chocolates\.
\- Therefore, the number of chocolates given away is:
6​packs×12​chocolates per pack=72​chocolates6\\text\{ packs\}\\times 12\\text\{ chocolates per pack\}=72\\text\{ chocolates\}
3\. Calculate the number of chocolates remaining:
\- Subtract the number of chocolates given away from the initial total:
216​chocolates−72​chocolates=144​chocolates216\\text\{ chocolates\}\-72\\text\{ chocolates\}=144\\text\{ chocolates\}
4\. Convert the remaining chocolates into dozens:
\- There are 12 chocolates in a dozen\.
\- To find out how many dozens of chocolates remain, divide the number of chocolates by 12:
\\frac\{144 \\text\{ chocolates\}\}\{12 \\text\{ chocolates per dozen\}\} = 12 \\text\{ dozens\}
Thus, the number of dozens of chocolates Bella has left is \\\(\\boxed\{12\}\\\)Table 6:CoT Trace ExamplesRealistic Trace generated from Qwen2\.5\-1\.5B\-Instruct\.

## Appendix CTiming Analysis for Fixed Wallclock Time Experiments

To ensure a fair comparison for our fixed wallclock time experiments, we measure the latency of a single training step \(one batch forward and backward\) for both architectures\. We benchmark our direct supervision method \(MTP\) using a span length of 4 against indirect supervision \(CODI\) using 6 latent steps\. These configurations were selected as they result in similar compression ratios on the GSM8k\-Aug dataset\. All timing evaluations are conducted on an H200 GPU, with each sequence partitioned into a 60\-30\-10 ratio token split for the question, CoT, and answer, respectively\. Model configurations are detailed in[Table7](https://arxiv.org/html/2607.16972#A3.T7)\.

Table 7:Latency of a single training pass \(forward \+ backward\) on NVIDIA H200At this scale, CODI requires approximately2×2\\timesthe training time of MTP for a batch size of 16\. While one might reasonably expect a6×6\\timesoverhead for CODI due to the requirement of 6 sequential forward passes for the latent steps, KV caching during training significantly accelerates this computation\. As a result, the observed2×2\\timesoverhead is primarily bottlenecked by the initial forward pass for the question sequence and the final forward pass for the answer sequence\. In contrast, our MTP approach computes the forward pass for all positions simultaneously, as the outputs at all positions are known during training\.

MTP training demonstrates compute\-bound characteristics \(similar to standard NTP training\), as the step latency scales almost linearly \(nearly doubling from 343\.10 ms to 630\.26 ms\) when the batch size is doubled from 16 to 32\. Conversely, CODI exhibits memory\-bound behavior, showing a distinctly sub\-linear increase in latency across the same batch sizes\. This indicates that a substantial portion of CODI’s training time is consumed by memory overhead and the sequential memory accesses required for latent computations\. Consequently, this leads to an under\-utilization of GPU compute resources during training\. The memory overhead is dominated by the need to update the KVCache states, consequently this is larger for Qwen2\.5\-1\.5B due to larger hidden dimensions\.

## Appendix DAnalysis of Problem Structures in GSM8k\-Aug

As the dataset consists of expressions in the CoT traces, we parse the expressions into templates to analyze the diversity of problem structures\. Some examples of templates are given below:

Count in Train: 24353Template: a/b=r0 Q: A school has 3 classrooms, each with the same number of students\. If there are 36 students in total, how many students are there in each classroom? CoT:≪\\ll36/3=12≫\\gg

Count in Train: 22451Template: a\*b=r0 Q: Jacob is building a dog house\. He uses 8 nails for each plank and needs 6 planks\. How many nails will he need in total? CoT:≪\\ll8\*6=48≫\\gg

Count in Train: 14989Template: a\*b=r0, r0\*c=r1 Q: A basketball team consists of 12 players\. Each player needs two basketball shoes\. The cost of each shoe is $75\. How much will it cost for the basketball team to buy shoes? CoT:≪\\ll12\*2=24≫\\gg≪\\ll24\*75=1800≫\\gg

Cumulative coverage[Figure4](https://arxiv.org/html/2607.16972#A4.F4)shows that 80% of the dataset consists of 3k structures which is less than 1% the size of the dataset\.

Figure 4:Cumulative coverage of top\-n problem structures in GSM8k\-Aug![Refer to caption](https://arxiv.org/html/2607.16972v1/assets/structures.png)We find that top 10, 50, 100, 500 structures cover 28\.3%, 46\.5%, 53\.3% and 66\.5% of the 380k samples of GSM8k\-Aug which is a considerable portion\. We hence argue that this dataset, does not reflect real world diversity of typical SFT finetuning\.

## Appendix EExamples of Error Accumulation during MTP Inference

A Collection of all wrong arithmetic expressions on Semi\-Natural CoT

500000\*\.150=150 ;3\*3=99; 75\-15=15 ; 20\-1=10 ; 1010\*1=10 ; 16/3=3 ; 20\+2=60 ; 4\*4=160 ; 10\.2\*10=24 ; 24\*45=108108 ; 30/100=54 ; 180\+324=324 ; 180\-0\.3=120 ; 180\+120=360 ; 2\*8080=160 ; 6\*55=330330 ; 90/4=9 ; 7\*7=1 ; 90/31=2 ; 3\*3=90 ; 25/100=4 ; 16\-4=8 ; 20\*20\.2=4 ; 2020\-4=16 ; 8000\*1\.2=960 ;21/12=1; 1\.1\*75=4 ; 2\+4=10 ; 6/44=1 ; 2/3=6 ; 3/5=9 ; \.67\+9=15 ;3\*2=6\.67; 3\*10=6 ; 5\*15=9 ; \.67\+9=21 ; 23\-25=8 ; 23\-37=36 ; 2\*2=8 ; 4\+3=9 ; 55\*2=10 ; 50/0\.75=25 ; 75\.50\+126=292\.5 ; 18\*162=111 ; 11\-18=5 ; 5\*7=106 ; 18\*162=108 ; 162\-108=90 ; 2\+20=40 ; 40\+65=155 ; 155/3=51 ; 25\*100=25 ; 33/3=23 ; 3/220=20\.33 ; 40\+85=225 ; 195\-160=40 ; 2\*8=26 ; 26/60=1 ; 3\*3=1 ; 3\+6=15 ; 3\*6=2 ; 3\*3=2 ; 3\*6=4 ; 1/3=2 ; 2/3=3\.3 ; \.25\+1\.25=2\.75 ; 0\-100=1700 ; 400\*3=120 ; 1200\-1000=2000 ; 40\-200=40 ; 400\-10=290\.00 ; 7\+14=28 ; 5\+7=19 ; 80\+23=203 ; 203\-23=80 ; 40/2=100 ; 200\*40=80 ; 12/4=24 ; 15\*15=120 ; 4\*120=90 ;4\-1=33; 6\*6=48 ; 18\+12=25 ; 5000\-3755=125 ; 3750\-5000=3750 ; 25\*40=10 ; 100\*25=5 ; 4\+4=9 ; 2525\-9=16 ; 500\+800=130 ; 1300\+13=139 ; 1300\+130=123 ; 000\+25000=1 ; 126\*60=756 ; 100\*840=84 ; 840\+756=149 ; 140\+280=605 ; 140\+340=625 ; 5\*88=40 ; 2/100=15 ; 13\.35\+34\.2=84 ; 128\+256=512 ; 16/2=1 ; 1/2=8 ; 8\+4=33 ; 4/3=8 ; 4\+2=30 ; 740/110=6 ; 180/110=16 ; 4/4=100100 ; 2\*100=50 ; 4\+4=4 ; 24/66=4 ; 9\+28=77 ; 125\-2=510 ; 4\-2=125 ; 23\*23=92 ; 10/100=60 ; 10/100=66 ; 10/100=600 ; 600600\+600=6600 ; 10/100=1320 ; 15400\+6000=22000 ; 8800/88=1080 ; 75\*4=3 ; 0\*\.110=22 ;

A lot of these mistakes, although seem far from correct can be easily explained in the context of MTP\. 3\*2=6\.67has an extra prediction of ”\.67“ due to 2\-token prediction\. The following CoT trace conditioned on this wrong output\. 12/21=1When 2\-tokens are aggregated, both the numbers turned out to be same, hence the result was 1\. 3\*3=99has an extra prediction of ”9“ due to 2\-token prediction\. and so on\.\.

## Appendix FC\-MTP Ablation Studies

In this section we present two ablation studies of C\-MTP\. The first study compares multi\-task training with warmstart initialization, and the second ablates the necessity ofLK​DL\_\{KD\}by training without distillation loss\.

### F\.1Multi\-Task Training vs Warmstart Initialization

We experiment with Multi\-Task Training and Warmstart Initialization on full Structured CoT trace dataset\. Since C\-MTP is a small addition on top of CoT, we view Warmstart Initialization as the more principled way but fair comparison with CODI compelled us to experiment with Multi\-task training setup\. Our results show similar performance with both settings, hence we use Warmstart Init for our main table\.

Table 8:Comparison of Warmstart Initialization \(WS\-Init\) and Multitask Training \(MultiTask\-Train\)\. with LLama3\.2\-1B\-Instruct on GSM8k\-Aug \(Structured\) CoT traces\.
### F\.2Ablation on C\-MTP: Training withoutLK​DL\_\{KD\}

In this section, we ablate the effect of using the knowledge distillation lossLK​DL\_\{KD\}in our training setup\. We compare the performance of our model with and without theLK​DL\_\{KD\}term, while keeping all other training configurations constant\. The results are summarized in[Table9](https://arxiv.org/html/2607.16972#A6.T9)\. We observe that the performance difference is minimal, suggesting that the C\-MTP training is able to learn effectively even without distilling from a frozen teacher\. The warmstart initialization from the CoT checkpoint provides a strong starting point and the simplicity of the C\-MTP training objective prevents catastrophic forgetting\.

Table 9:GSM8k\-Aug results on LLaMA\-3\.2\-1B\-Instruct with noLK​DL\_\{KD\}on C\-MTP\.Accuracy is averaged over 5 runs \(run\-to\-run deviation<<1%\)\. \#L denotes the mean chain\-of\-thought length \(in steps\) with standard deviation, computed over the full evaluation set\.Training MethodGSM8k\-TestGSM\-HardSVAMPMultiArithAcc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)StandardCoT61\.9427\.17±11\.2427\.17\_\{\\pm 11\.24\}14\.2234\.43±15\.1134\.43\_\{\\pm 15\.11\}64\.9613\.15±5\.3013\.15\_\{\\pm 5\.30\}98\.8915\.46±1\.9615\.46\_\{\\pm 1\.96\}SupervisionCODI53\.036\.0012\.406\.0060\.266\.0099\.116\.00Sim\-CoT53\.956\.0012\.456\.0058\.626\.0097\.786\.00DirectCoLaR\-241\.2312\.99±5\.5312\.99\_\{\\pm 5\.53\}8\.8716\.30±8\.3016\.30\_\{\\pm 8\.30\}52\.466\.31±2\.666\.31\_\{\\pm 2\.66\}90\.567\.39±1\.437\.39\_\{\\pm 1\.43\}SupervisionCoLaR\-332\.659\.01±3\.759\.01\_\{\\pm 3\.75\}6\.7910\.96±6\.3010\.96\_\{\\pm 6\.30\}48\.364\.49±1\.914\.49\_\{\\pm 1\.91\}87\.335\.33±1\.935\.33\_\{\\pm 1\.93\}CoLaR\-429\.606\.80±2\.716\.80\_\{\\pm 2\.71\}6\.607\.66±2\.977\.66\_\{\\pm 2\.97\}46\.823\.47±1\.483\.47\_\{\\pm 1\.48\}88\.114\.13±0\.674\.13\_\{\\pm 0\.67\}C\-MTP\-253\.3012\.07±5\.5312\.07\_\{\\pm 5\.53\}12\.5914\.15±6\.4414\.15\_\{\\pm 6\.44\}62\.105\.14±2\.585\.14\_\{\\pm 2\.58\}95\.336\.29±1\.206\.29\_\{\\pm 1\.20\}\-LK​DL\_\{KD\}52\.1212\.10±5\.8812\.10\_\{\\pm 5\.88\}12\.1914\.39±6\.9414\.39\_\{\\pm 6\.94\}59\.505\.27±2\.655\.27\_\{\\pm 2\.65\}94\.676\.23±0\.986\.23\_\{\\pm 0\.98\}C\-MTP\-346\.507\.94±3\.697\.94\_\{\\pm 3\.69\}10\.589\.02±4\.549\.02\_\{\\pm 4\.54\}56\.083\.43±1\.663\.43\_\{\\pm 1\.66\}97\.334\.16±0\.674\.16\_\{\\pm 0\.67\}\-LK​DL\_\{KD\}46\.007\.92±3\.657\.92\_\{\\pm 3\.65\}10\.608\.99±4\.348\.99\_\{\\pm 4\.34\}56\.103\.43±1\.693\.43\_\{\\pm 1\.69\}96\.674\.17±0\.634\.17\_\{\\pm 0\.63\}C\-MTP\-440\.705\.84±2\.715\.84\_\{\\pm 2\.71\}9\.296\.85±4\.346\.85\_\{\\pm 4\.34\}50\.982\.39±1\.472\.39\_\{\\pm 1\.47\}94\.673\.12±0\.563\.12\_\{\\pm 0\.56\}\-LK​DL\_\{KD\}36\.605\.84±2\.665\.84\_\{\\pm 2\.66\}8\.796\.82±3\.226\.82\_\{\\pm 3\.22\}48\.702\.42±1\.512\.42\_\{\\pm 1\.51\}94\.783\.13±0\.513\.13\_\{\\pm 0\.51\}

## Appendix GExtended Experimental Results

In this section we present additional details and results for the experiments described in the main paper\.

### G\.1Setup Details

[Table10](https://arxiv.org/html/2607.16972#A7.T10)reports the training configuration for every run, following these conventions\.*Data*denotes the training corpus:*Structured*\(GSM8k\-Aug,icotin code\),*Semi\-Nat\.*\(semi\-natural language CoT,icot\-nlin code\), and*Realistic*\(mathllama/mathqwenin code\)\. The effective batch size is reported as \(per\-device batch×\\timesgradient\-accumulation steps\)\. All runs use a warmup ratio of0\.050\.05, except the CODI runs, which use0\.030\.03following original implementation\. All models are trained with LoRA adapters unless marked*Full*\(full fine\-tuning\)\. The last column reports the C\-MTP span length or the number of CODI latent tokens, where applicable; it is left blank otherwise\.

Table 10:Final hyperparameters for all experiments\.Learning rates use the shorthand5​e\-​4=5×10−45\\text\{e\-\}4=5\\times 10^\{\-4\}\. The effective batch size is reported as \(per\-device batch×\\timesgradient\-accumulation steps\)\.BackboneDataSettingBatch \(p×\\timesg\)Ep\.LRWDMax Tok\.Span/Lat\.Explicit CoTLlamaStructuredLoRA128 \(64×\\times2\)45e\-40\.01200–LlamaStructuredFull128 \(64×\\times2\)32e\-50\.01200–LlamaSemi\-Nat\.LoRA128 \(32×\\times4\)55e\-40\.01300–LlamaSemi\-Nat\.Full128 \(32×\\times4\)42e\-50\.01300–QwenStructuredLoRA128 \(32×\\times4\)55e\-40\.1200–QwenStructuredFull128 \(32×\\times4\)32e\-40\.1200–LlamaRealisticLoRA64 \(32×\\times2\)55e\-40\.11024–QwenRealisticLoRA64 \(16×\\times4\)55e\-40\.11024–C\-MTPLlamaStructuredLoRA128 \(32×\\times4\)55e\-40\.012002 / 3 / 4LlamaStructuredLoRA \(multitask\)128 \(32×\\times4\)85e\-40\.012003LlamaSemi\-Nat\.LoRA128 \(32×\\times4\)55e\-40\.013002 / 3 / 4QwenStructuredLoRA128 \(32×\\times4\)55e\-40\.12002 / 3 / 4LlamaRealisticLoRA†64 \(8×\\times8\)85e\-40\.110242QwenRealisticLoRA†64 \(8×\\times8\)85e\-40\.110242CODILlamaStructuredLoRA128 \(64×\\times2\)87e\-40\.12006LlamaSemi\-Nat\.LoRA128 \(64×\\times2\)87e\-40\.13006LlamaRealisticLoRA128 \(16×\\times8\)105e\-40\.110246 / 10 / 20QwenRealisticLoRA128 \(16×\\times8\)105e\-40\.110246 / 10 / 20
†The C\-MTP student and teacher are initialized from the merged LoRA checkpoint of the corresponding*Realistic*CoT run \(rows above\)\.

All experiments were conducted on a single NVIDIA H200 GPU\.

### G\.2Training on Structured CoT: GSM8k\-Aug

We present results on Qwen2\.5\-1\.5B\-Instruct in[Table11](https://arxiv.org/html/2607.16972#A7.T11)\. Our experiments on training Sim\-CoT with this backbone were rife with training instabilities leading to suboptimal downstream results\. Therefore, we did not add them in the table\.

Table 11:GSM8k\-Aug results on Qwen2\.5\-1\.5B\-Instruct\.Accuracy is averaged over 5 runs \(run\-to\-run deviation<<1%\)\. \#L denotes the mean chain\-of\-thought length \(in steps\) with standard deviation, computed over the full evaluation set\.Training MethodGSM8k\-TestGSM\-HardSVAMPMultiArithAcc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)Acc\. \(↑\\uparrow\)\# L \(↓\\downarrow\)StandardCoT68\.2336\.46±16\.5636\.46\_\{\\pm 16\.56\}26\.2357\.37±29\.6057\.37\_\{\\pm 29\.60\}68\.3618\.30±9\.2418\.30\_\{\\pm 9\.24\}98\.0019\.18±2\.4519\.18\_\{\\pm 2\.45\}SequentialCODI44\.506\.00\\bm\{6\.00\}10\.466\.00\\bm\{6\.00\}63\.106\.006\.0092\.226\.006\.00ParallelCoLaR\-237\.4817\.86±8\.4017\.86\_\{\\pm 8\.40\}9\.0127\.55±14\.2227\.55\_\{\\pm 14\.22\}49\.908\.36±4\.388\.36\_\{\\pm 4\.38\}84\.339\.30±2\.019\.30\_\{\\pm 2\.01\}CoLaR\-330\.6912\.04±5\.4212\.04\_\{\\pm 5\.42\}7\.1918\.36±10\.0518\.36\_\{\\pm 10\.05\}43\.645\.80±2\.895\.80\_\{\\pm 2\.89\}68\.446\.35±1\.036\.35\_\{\\pm 1\.03\}CoLaR\-425\.289\.29±4\.139\.29\_\{\\pm 4\.13\}6\.0214\.14±8\.4614\.14\_\{\\pm 8\.46\}40\.224\.57±2\.174\.57\_\{\\pm 2\.17\}73\.334\.88±0\.874\.88\_\{\\pm 0\.87\}C\-MTP\-250\.5716\.60±8\.2016\.60\_\{\\pm 8\.20\}11\.7825\.08±12\.5625\.08\_\{\\pm 12\.56\}54\.207\.51±4\.427\.51\_\{\\pm 4\.42\}90\.448\.27±1\.478\.27\_\{\\pm 1\.47\}C\-MTP\-342\.2910\.84±5\.5210\.84\_\{\\pm 5\.52\}10\.6016\.01±8\.2916\.01\_\{\\pm 8\.29\}45\.964\.79±2\.884\.79\_\{\\pm 2\.88\}79\.225\.53±1\.315\.53\_\{\\pm 1\.31\}C\-MTP\-434\.427\.99±4\.037\.99\_\{\\pm 4\.03\}8\.0211\.77±6\.2911\.77\_\{\\pm 6\.29\}44\.243\.40±2\.08\\bm\{3\.40\_\{\\pm 2\.08\}\}72\.113\.94±0\.95\\bm\{3\.94\_\{\\pm 0\.95\}\}
### G\.3Sample Efficiency

To understand how the two training paradigms compare under varying amounts of supervision, we train both methods on randomly sampled subsets of GSM8k\-Aug \(12k, 25k, 50k, 100k and 200k samples\) and evaluate on downstream tasks\. For fair comparison with CODI, we use Multitask\-Train configuration where we train the same backbone for CoT and C\-MTP and use the CoT inference output as the teacher\. For each random subset size we run a grid search overl​r∈lr\\in\{5e\-4, 7e\-4, 8e\-4\} ande​p​o​c​h​s∈epochs\\in\{6, 7, 10\} and select the configuration with final best validation loss\.[Figure5](https://arxiv.org/html/2607.16972#A7.F5)shows accuracy as a function of training set size\. In the low\-data regime \(up to∼\\sim100k samples\), C\-MTP consistently outperforms CODI, with the gap most pronounced at 12k samples\. On in\-distribution tasks \(GSM8k\-Test, GSM\-Hard\), C\-MTP’s advantage is particularly clear, whereas on out\-of\-distribution tasks \(SVAMP, MultiArith\) the gap is smaller and more variable\.

Figure 5:Sample efficiency comparison on GSM8k\-Aug\.![Refer to caption](https://arxiv.org/html/2607.16972v1/assets/sampleffall.png)
### G\.4Fixed WallClock Time Comparison

Our analysis \([Table7](https://arxiv.org/html/2607.16972#A3.T7)\) reveals that a single C\-MTP training step is approximately2×2\\timesfaster than CODI on LLaMA\-3\.2\-1B, allowing C\-MTP to process roughly twice as many samples within the same wall\-clock budget\. This2×2\\timesfactor represents a lower bound: for larger models like Qwen2\.5\-1\.5B\-Instruct, the sequential memory access overhead of CODI’s latent forward passes grows more pronounced, widening the gap further\.

In a finetuning setup, this increased compute can be allocated in several ways, such as additional samples, more epochs, a larger base model, or increased sequence length\. Here, we isolate the effect of increased sample count under direct supervision \(C\-MTP\) versus indirect supervision \(CODI\)\. To maintain2×2\\timeswall\-clock parity, we hold the number of epochs constant and set learning rates based on sample\-efficiency grid sweeps\. We note that this configuration is not necessarily optimal and serves primarily to illustrate the magnitude of gains achievable under a fixed time budget\. Overall,[Table12](https://arxiv.org/html/2607.16972#A7.T12)shows that C\-MTP\-4 consistently outperforms CODI across all budget levels, with particularly pronounced gains on GSM8k\-Test\.

Table 12:Fixed Wall clock time Comparison on GSM8k\-aug\. We report downstream accuracy on each \(\# Samples, Epochs\) configuration on CODI and C\-MTP\-4\. C\-MTP\-4 consistently outperforms CODI across all budget levels, with gains on GSM8k\-Test and MultiArith particularly pronounced\.
### G\.5Generalization to Problem Structures

GSM8k\-Aug\(Denget al\.,[2024](https://arxiv.org/html/2607.16972#bib.bib9)\)was constructed by prompting GPT\-4 to generate problems similar to GSM8k, thereby expanding the dataset∼50×\\sim 50\\timesvia prompting and heuristic filtering\. This process results in substantial structural redundancy: our analysis shows that over 50% of all training samples reduce to only 100 distinct problem structures\. To evaluate out\-of\-domain generalization to problem structures not seen during training, we partition the data by held\-out problem structures\. More analysis in[AppendixD](https://arxiv.org/html/2607.16972#A4)\(Appendix\)

Concretely, let𝒟n⊂𝒟\\mathcal\{D\}^\{n\}\\subset\\mathcal\{D\}denote the subset of samples containingnnreasoning steps, forn∈\{2,3,4\}n\\in\\\{2,3,4\\\}\. We partition each𝒟n\\mathcal\{D\}^\{n\}by problem structure into two disjoint, roughly equal\-sized splits𝒟An\\mathcal\{D\}^\{n\}\_\{A\}and𝒟Bn\\mathcal\{D\}^\{n\}\_\{B\}, ensuring no structure appears in both\. Within each split, we hold out 2% of problems as a test set\. For each\(n,X\)∈\{2,3,4\}×\{A,B\}\(n,X\)\\in\\\{2,3,4\\\}\\times\\\{A,B\\\}, we train on𝒟Xn\\mathcal\{D\}^\{n\}\_\{X\}and evaluate on the test sets of*both*𝒟An\\mathcal\{D\}^\{n\}\_\{A\}and𝒟Bn\\mathcal\{D\}^\{n\}\_\{B\}, yielding in\-domain accuracy \(same split\) and out\-of\-domain accuracy \(held\-out structures\)\.

To ensure a fair comparison, we first train C\-MTP\-4 on all splits and measure the average generation length on each test set:𝒟2=3\.23±0\.55\\mathcal\{D\}^\{2\}\{=\}3\.23\\pm 0\.55,𝒟3=5\.38±0\.75\\mathcal\{D\}^\{3\}\{=\}5\.38\\pm 0\.75,𝒟4=7\.04±1\.13\\mathcal\{D\}^\{4\}\{=\}7\.04\\pm 1\.13tokens\. Since CODI requires the number of latents to be specified before training, we use these lengths to guide the choice, setting the latent count to 4, 6, and 8 forn∈\{2,3,4\}n\\in\\\{2,3,4\\\}respectively\.

[Table13](https://arxiv.org/html/2607.16972#A7.T13)\(Appendix\) reports in\-domain and out\-of\-domain accuracy for C\-MTP and CODI across reasoning lengthsn∈\{2,3,4\}n\\in\\\{2,3,4\\\}\. C\-MTP generalizes better to held\-out problem structures in all settings\. The gap is particularly clear atn=3n=3andn=4n=4, where CODI’s out\-of\-domain accuracy drops substantially \(e\.g\., 27\.10 and 13\.83 onn=3n=3andn=4n=4respectively\) while C\-MTP retains stronger performance \(42\.86 and 24\.04\)\. In\-domain, the two methods are more competitive, with CODI performing slightly better atn=2n=2while C\-MTP leads at longer reasoning chains\. Overall, these results suggest that the sequential training of indirect supervision methods leads to overfitting to the specific problem structures seen during training\.

Table 13:in\-domainandout\-of\-domainevaluation results for C\-MTP\-4 and CODI across reasoning lengthsn∈\{2,3,4\}n\\in\\\{2,3,4\\\}\. Each matrix entry reports accuracy when trained on the row set and evaluated on the column set\.C\-MTP

CODI

𝒟A2\\mathcal\{D\}^\{2\}\_\{A\}𝒟B2\\mathcal\{D\}^\{2\}\_\{B\}𝒟A2\\mathcal\{D\}^\{2\}\_\{A\}73\.4173\.4153\.5453\.54𝒟B2\\mathcal\{D\}^\{2\}\_\{B\}60\.4060\.4072\.4272\.42
𝒟A2\\mathcal\{D\}^\{2\}\_\{A\}𝒟B2\\mathcal\{D\}^\{2\}\_\{B\}𝒟A2\\mathcal\{D\}^\{2\}\_\{A\}75\.1175\.1151\.5651\.56𝒟B2\\mathcal\{D\}^\{2\}\_\{B\}63\.7963\.7976\.5276\.52
𝒟A3\\mathcal\{D\}^\{3\}\_\{A\}𝒟B3\\mathcal\{D\}^\{3\}\_\{B\}𝒟A3\\mathcal\{D\}^\{3\}\_\{A\}56\.4656\.4642\.8642\.86𝒟B3\\mathcal\{D\}^\{3\}\_\{B\}46\.0946\.0955\.3855\.38
𝒟A3\\mathcal\{D\}^\{3\}\_\{A\}𝒟B3\\mathcal\{D\}^\{3\}\_\{B\}𝒟A3\\mathcal\{D\}^\{3\}\_\{A\}41\.8841\.8827\.1027\.10𝒟B3\\mathcal\{D\}^\{3\}\_\{B\}26\.4226\.4240\.4140\.41
𝒟A4\\mathcal\{D\}^\{4\}\_\{A\}𝒟B4\\mathcal\{D\}^\{4\}\_\{B\}𝒟A4\\mathcal\{D\}^\{4\}\_\{A\}32\.5532\.5524\.0424\.04𝒟B4\\mathcal\{D\}^\{4\}\_\{B\}24\.6824\.6833\.4033\.40
𝒟A4\\mathcal\{D\}^\{4\}\_\{A\}𝒟B4\\mathcal\{D\}^\{4\}\_\{B\}𝒟A4\\mathcal\{D\}^\{4\}\_\{A\}17\.8717\.8713\.8313\.83𝒟B4\\mathcal\{D\}^\{4\}\_\{B\}17\.2317\.2318\.5118\.51

### G\.6Full Example of MTP Error Accumulation

We present an example of how an error at the initial part of the sequence can cascade to the rest of the CoT\. / represents span prediction boundary\.

Question:Janet has 16 eggs\. She uses 3 eggs to make breakfast and 4 eggs to bake muffins\. If she sells the remaining eggs at the farmers’ market for 2 each, how much money does she make at the farmers’ market each day?CoT:She has 16 eggs \- 3 eggs \- 4 eggs = 9 eggs left for the farmers’ market\. Thus, Janet makes 9 \*2=2=18 in the farmers’ market each day\.MTP Prediction:Janet/ has /16 \-/ 3/ = /13 eggs/ left after/ her breakfast/\. She/ bakes/ muffins/ with /13\*/ 4/ = /52eggs/\. She/ sells /52\-/ 16/ = /36 eggs/ at the/ farmers’/ market daily/\. So/, she/ makes /36 \*/ 2/ = /72 daily/ at the/ farmers’/ market\.

Question:Emma can make and upload 72 vlogs per month\. But she was only able to make 18 vlogs for the first week, 21 vlogs for the second week, and 15 vlogs for the third week\. How many vlogs should she do to complete the 72 vlogs per month?CoT:Emma was able to make 18 \+ 21 \+ 15 = 54 vlogs for the first three weeks\. So, she needs to make 72 \- 54 = 18 more vlogs to meet the 72 vlogs per month requirement\.MTP Prediction:Emma has first/ to /72 \- /18 \- /21 = /45vib/logs to meet/ her quota\./ Finally her third/ week, she/ has /45\- /15 = /30 moreib/ toib to/ meet\. quota/ Thus, of/ she should /72 \+ /30 = /102 moreks/ to meet the/ monthly72\.

### G\.7Experiments on CommonsenseQA

We experiment with CommonsenseQA\(Talmoret al\.,[2019](https://arxiv.org/html/2607.16972#bib.bib67)\), a commonsense multiple\-choice question\-answering dataset paired with natural language CoTs generated by GPT\-4o\-mini\(Shenet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib12)\)\. We see that Indirect Supervision \(CODI\) remains preferable even exceeding CoT\-SFT in this case; their compression objective allows them to discard the filler words and focus on the key associations required to get the final answer\. For direct supervision \(C\-MTP\), we find that although the traces are nearly as long as semi\-natural math traces, we do not see similar degradation with increasing span lengths\. Natural language, devoid of mathematically precise tokens, makes the final answer robust to small errors in multi\-token prediction\.

Table 14:Results on CommonsenseQA with Qwen2\.5\-1\.5B\-Instruct\.Question:Question: What is it called when you are talking to someone?\\nChoices:\\nA: communication\\nB: quiet\\nC: boredom\\nD: persuaded\\nE: learnMTP Thinking:After finishing a bottle,/ the most logical place would/ be be a sink it/ in, it, it/ is it for the for/ cleaning\.\. The options/ other either do not,/ logically the to the\./ other, is / the water,,/ dishwasher, is the the/ most appropriate for for/ cleaning\.MTP Final Answer:The answer is: DGround truth:D

## Appendix HCoLaR \- CODI Implementation Discrepancy

We acknowledge that our reproduced results for CODI\(Shenet al\.,[2025](https://arxiv.org/html/2607.16972#bib.bib12)\)differ from the baseline results reported by CoLaR\(Tanet al\.,[2026](https://arxiv.org/html/2607.16972#bib.bib16)\)\. This discrepancy likely stems from implementation differences, as the official CODI codebase may not have been accessible during CoLaR’s development\. Our experiments use the official CODI repository, yielding results consistent with those reported in the original paper\.