Think in Latent, Explain in Language: Self-Explainable Latent Reasoning

# Think in Latent, Explain in Language: Self-Explainable Latent Reasoning
Source: [https://arxiv.org/html/2608.13570](https://arxiv.org/html/2608.13570)
###### Abstract

Latent reasoning has emerged as a powerful alternative to text\-based Chain\-of\-Thought \(CoT\), offering significant gains in computational efficiency by compressing verbose reasoning into compact embeddings\. However, compressing reasoning into the latent space renders the thinking opaque, hindering its interpretability\. Current methods present a stark trade\-off: they either function as unexplainable “black boxes” \(e\.g\., Coconut\), where the latent reasoning is not human\-readable, or rely on separate post\-hoc decoders for explainability \(e\.g\., Heima\), introducing architectural overhead and decoupling the explanation from the actual reasoning process\. In this work, we present a unified framework for Self\-Explainable Latent Reasoning \(SELR\) that trains a single model to perform efficient and inherently explainable latent reasoning\. Our core contribution is a novel multi\-task training objective that optimizes for two goals simultaneously: \(1\) an Answer Loss that optimizes the latent reasoning trajectory to produce accurate final answers, and \(2\) a CoT Loss that explicitly trains the same model to decode its own latent representations back into human\-understandable reasoning steps\. This design ensures that generated latent representations are both task\-effective and semantically interpretable, eliminating the need for external decoders\. We validate the effectiveness of SELR on both Large Language Models \(LLMs\) and Vision\-Language Models \(VLMs\), demonstrating that SELR achieves superior token efficiency and accuracy compared to baselines, while uniquely providing self\-contained explainability without auxiliary models\. Project page is available at[https://jasondayuan\.github\.io/SELR/](https://jasondayuan.github.io/SELR/)\.

Machine Learning, ICML

![Refer to caption](https://arxiv.org/html/2608.13570v1/x1.png)Figure 1:Demonstration of the SELR framework\.Left \(Reasoning\):The image and question are provided as input to the VLM, which performs latent space reasoning to generate latent thoughts and a final answer\.Right \(Decoding\):The generated latent thoughts are then provided as input back to the very same VLM, along with a decoding prompt\. The VLM then decodes its own latent thoughts, presenting the information encoded within them as human\-readable text\.## 1Introduction

Large Language Models \(LLMs\) have traditionally been trained to reason in the*language space*, where they elaborate their thinking process with textual Chain\-of\-Thought \(CoT\)\(Weiet al\.,[2022](https://arxiv.org/html/2608.13570#bib.bib2)\)\. However, this discrete language space may not be the optimal medium for reasoning: Text\-based CoT can be excessively verbose, with many tokens serving textual coherence rather than actual reasoning, and is fundamentally constrained by a fixed, discrete vocabulary\. To overcome these limitations, a new paradigm of reasoning in a*continuous latent space*\(Haoet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib1)\)has emerged: Models can reason more effectively by feeding the last hidden state \(a “continuous thought”\) directly back into the model as the next input embedding, bypassing the discrete text generation step\. This allows models to produce their thinking procedure with continuous tokens, which can encode more dense information as they are no longer constrained by the discrete embedding space\. This paradigm enables unique synergy in Vision\-Language Models \(VLMs\), as it bridges the modality gap between continuous visual embedding and discrete textual tokens, allowing the model to process visual information in its native continuous format\. Consequently, latent space reasoning models have the potential to be significantly more*token\-efficient*than their text\-based counterparts\.

Despite this promise, current latent space reasoning models face two significant challenges\. The first is thelack of supervision\. While humans can write textual CoT steps to serve as a supervision source for text\-based reasoning models, we cannot supervise latent space models with the same approach, as there is no human\-interpretable “ground truth” for such continuous thoughts\. Prior efforts have explored methods like multi\-stage curriculum learning\(Haoet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib1)\)to gradually shift the thinking procedure from the language space to the latent space, but there is no generally agreed\-upon approach for learning latent space reasoning effectively\.

The second problem is thelack of interpretability\. When a latent space reasoning model produces its thinking procedure in the latent space, there is no direct approach for us to know*what the model is “thinking” about*\. Although we can probe the continuous thought tokens to find their most similar text tokens in the model’s language space\(Haoet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib1)\), this only explains one token at a time and is poorly human\-readable\. Another explored approach\(Shenet al\.,[2025a](https://arxiv.org/html/2608.13570#bib.bib3)\)achieves explainability by training a separate, post\-hoc decoder model, but this disjoint architecture introduces significant parameter overhead and risks decoupling the explanation from the actual reasoning logic, as the explainer is distinct from the reasoner\. While token\-efficient thinking is desired, we lose the crucial ability to conveniently understand the model’s reasoning path when necessary\.

To address these challenges, we propose a unified framework, Self\-Explainable Latent Reasoning \(SELR\), that targets both supervision and interpretability simultaneously\. Our central design is amulti\-task learning objectivethat trains a single model to be both an efficient latent space*reasoner*and its own latent\-to\-language*translator*\. Specifically, the model is trained to optimize two goals concurrently: \(1\) anAnswer Lossthat guides the latent reasoning to produce the correct final answer, ensuring high reasoning performance, and \(2\) aCoT Lossthat explicitly trains the*same model*to decode its own latent representations back into human\-understandable reasoning steps\. A demonstration of this process is provided in Figure[1](https://arxiv.org/html/2608.13570#S0.F1)\.

This multi\-task approach provides a synergistic solution to both aforementioned challenges\. The CoT Loss provides a rich, text\-based supervisory signal that guides the formation of the latent thoughts, addressing the supervision challenge\. Simultaneously, it forces the model to learn latent representations that are inherently aligned with human logic, thus solving the interpretability problem\. The model learns to generate*token\-efficient*thoughts that are not only*effective*for solving the task but also inherently*explainable*\.

We conduct comprehensive experiments to explore the best practices for training such models\. We first run extensive ablations and comparative experiments on lightweight LLMs using small\-scale, text\-only datasets to identify the most effective strategies\. We then demonstrate that these strategies generalize successfully to the complex VLM domain\. In particular, we show that a state\-of\-the\-art VLM \(Qwen2\.5\-VL\(Baiet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib8)\)\) fine\-tuned with SELR achieves simultaneous gains in both efficiency and accuracy by generating significantly fewer tokens and attaining higher scores across multiple VLM benchmarks, while demonstrating strong interpretability by decoding its own continuous thoughts\.

Our key contributions are threefold:

1. 1\.We propose a novel framework SELR that,*for the first time*, trains a single model to reason efficiently in latent space while also being able to translate its latent thoughts into human\-readable text, solving the concurrent problems of supervision and interpretability\.
2. 2\.We are*the first*to successfully apply this explainable latent reasoning paradigm to VLMs, demonstrating that a state\-of\-the\-art model can achieve simultaneous improvements in both reasoning accuracy and token efficiency\.
3. 3\.We provide a comprehensive analysis of training strategies, establishing a set of generalizable best practices for developing efficient and explainable latent space reasoning models\.

## 2Related Work

Chain\-of\-Thought Reasoning\.The Chain\-of\-Thought \(CoT\) reasoning paradigm begins with prompting techniques that elicit step\-by\-step reasoning in LLMs, either through few\-shot examples\(Weiet al\.,[2022](https://arxiv.org/html/2608.13570#bib.bib2)\)or simple prompts\(Kojimaet al\.,[2022](https://arxiv.org/html/2608.13570#bib.bib13)\)\. This approach is enhanced by decoding strategies such as self\-consistency\(Wanget al\.,[2023](https://arxiv.org/html/2608.13570#bib.bib14)\)and generalized by search\-based structures like Tree of Thoughts\(Yaoet al\.,[2023](https://arxiv.org/html/2608.13570#bib.bib15)\)\. The CoT framework is also adapted for vision\-language tasks\(Zhanget al\.,[2024](https://arxiv.org/html/2608.13570#bib.bib16)\)\. CoT has shifted from prompting to internalization, using process supervision to train models like OpenAI o1\(OpenAI,[2024b](https://arxiv.org/html/2608.13570#bib.bib17)\)or based on reinforcement learning like DeepSeek R1\(DeepSeek\-AI,[2025](https://arxiv.org/html/2608.13570#bib.bib18)\)\.

Latent Space Reasoning\.To overcome the computational inefficiency of explicit CoT, the seminal Coconut framework\(Haoet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib1)\)moves reasoning to a continuous latent space by training a model to “internalize” text\-based CoT traces, feeding the last hidden state back as a “continuous thought\.” Latent space reasoning allows the model to perform advanced, non\-deterministic reasoning like a latent Breadth\-First Search\(Zhuet al\.,[2025a](https://arxiv.org/html/2608.13570#bib.bib19)\)and has been extended in various directions, such as iterative refinement\(Geipinget al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib20)\), energy minimization\(Gladstoneet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib21)\), or pre\-training integrating\(Zhuet al\.,[2025b](https://arxiv.org/html/2608.13570#bib.bib22)\)\. Follow\-up methods of Coconut, such as CODI\(Shenet al\.,[2025b](https://arxiv.org/html/2608.13570#bib.bib25)\), CoLaR\(Tanet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib23)\), and HRPO\(Yueet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib24)\)improve efficiency by compressing CoT using strategies like curriculum learning or reinforcement learning\. However, the “black box” interpretability problem persists\. While the Heima framework\(Shenet al\.,[2025a](https://arxiv.org/html/2608.13570#bib.bib3)\)addresses this, it requires training separate models: an encoder to compress the thoughts and a decoder to interpret them\. Our SELR method solves this by training a single, unified model to be both an efficient reasoner and its own interpreter\.

## 3Approach

![Refer to caption](https://arxiv.org/html/2608.13570v1/x2.png)Figure 2:Visualization of our SELR framework, which consists of two stages\.\(I\) Latent Space Reasoning:The model first generates a sequence of latent thoughts and the final answer, given the image and question\. Heima utilizes special “Thinking of Summary/Caption/Reasoning” tokens for latent reasoning, whereas SELR generates latent thoughts sequentially, similar to Coconut\.\(II\) CoT Decoding:The generated latent thoughts are then decoded\. Heima requires training multiple, separate LLM decoders for each reasoning step, whileSELRtrains the*single, original model*to be its own translator, using the very VLM that generates the latent space reasoning to decode its own latent thoughts\. For SELR methods trained with Single Step Loss, we decode the reasoning steps one\-by\-one from each latent thought; for SELR methods trained with Full CoT Loss, we decode the entire CoT with all latent thoughts at once\.In this section, we begin by establishing the preliminaries of the VLM generation process\. Building on this foundation, we introduce the core optimization objectives of SELR, specifically the Answer Loss and CoT Loss\. Finally, we detail the complete training framework, presenting both our single\-stage and multi\-stage curriculum strategies\.

### 3\.1Preliminary: VLM Generation

Given an imageIIand a text token sequenceXtX\_\{t\}, the generation process of a VLM can be described as follows:

Ev\\displaystyle E\_\{v\}=𝒫​\(𝒱​\(I\)\),Et=ℰ​\(Xt\),\\displaystyle=\\mathcal\{P\}\(\\mathcal\{V\}\(I\)\),E\_\{t\}=\\mathcal\{E\}\(X\_\{t\}\),H\\displaystyle H=LLM​\(\[Ev;Et\]\),\\displaystyle=\\text\{LLM\}\(\[E\_\{v\};E\_\{t\}\]\),p​\(xl\+1\|I,Xt\)\\displaystyle p\(x\_\{l\+1\}\|I,X\_\{t\}\)=Softmax​\(W​hl\),\\displaystyle=\\text\{Softmax\}\(Wh\_\{l\}\),where𝒱\\mathcal\{V\}is the vision encoder \(e\.g\., a ViT\(Dosovitskiyet al\.,[2021](https://arxiv.org/html/2608.13570#bib.bib4)\)\);𝒫\\mathcal\{P\}is the projection layer;ℰ\\mathcal\{E\}is the embedding matrix;Ev∈ℝlv×dE\_\{v\}\\in\\mathbb\{R\}^\{l\_\{v\}\\times d\}is the sequence of image embeddings;Et∈ℝlt×dE\_\{t\}\\in\\mathbb\{R\}^\{l\_\{t\}\\times d\}is the sequence of text token embeddings;H∈ℝl×dH\\in\\mathbb\{R\}^\{l\\times d\}is the sequence of final hidden states, wherel=lv\+ltl=l\_\{v\}\+l\_\{t\};hlh\_\{l\}is the final hidden state corresponding to positionll, which is the last position of the input sequence;WWis the parameter of the language model head\.

### 3\.2Learning Objective

![Refer to caption](https://arxiv.org/html/2608.13570v1/x3.png)Figure 3:Illustration of the SELR multi\-task loss objectives\. TheAnswer Lossis computed on the answer tokens, conditioned on the image, question, and latent thoughts\. TheFull CoT LossandSingle Step Losstrains the model to decode its latent thoughts back into human\-readable text, calculated on the entire CoT sequence and the sampled CoT step respectively\.The core design of SELR is that a model’s internal reasoning state should be both capable of solving the task and*self\-explainable*\. To achieve this, we employ a multi\-task learning objective on a dataset of image\-question pairs annotated with ground\-truth CoT and final answers\. Each instance can be defined as:

\(I,Xq,\{XCoTk\}k=1K,Xa\),\\displaystyle\(I,X\_\{q\},\\\{X\_\{\\text\{CoT\}\_\{k\}\}\\\}\_\{k=1\}^\{K\},X\_\{a\}\),whereIIis the image,XqX\_\{q\}is the question,XCoTkX\_\{\\text\{CoT\}\_\{k\}\}is thekk\-th reasoning step, andXaX\_\{a\}is the answer\.

Our optimization objective is the weighted sum of two loss components:

1. 1\.Answer Loss:Cross\-entropy loss of answer tokensXaX\_\{a\}given imageII, questionXqX\_\{q\}, and latent reasoning\.
2. 2\.CoT Loss:Cross\-entropy loss of the entire CoT sequenceXCoT=\[XCoT1,…,XCoTK\]X\_\{\\text\{CoT\}\}=\[X\_\{\\text\{CoT\}\_\{1\}\},\\dots,X\_\{\\text\{CoT\}\_\{K\}\}\]or a single CoT stepXCoTiX\_\{\\text\{CoT\}\_\{i\}\}given the latent reasoning\.

As an additional note, although this formulation is presented for VLMs, it is adaptable to LLM training by simply removing the image\-related components \(e\.g\.,IIandEvE\_\{v\}\)\.

#### 3\.2\.1Latent Space Reasoning Generation

To define the Answer Loss and the CoT Loss, we first specify the generation process for latent space reasoning, as both losses are conditioned upon it\. Unlike standard generation, which projects final hidden states into a distribution over the vocabulary to sample discrete tokens, latent space reasoning feeds the final hidden state directly back as the next input embedding, effectively bypassing the language model head, as shown in Figure[2](https://arxiv.org/html/2608.13570#S3.F2)\.

SELR uses two special tokens,<bot\>and<eot\>, to mark the beginning and end of the latent reasoning process\. The full generation process is detailed in Algorithm[1](https://arxiv.org/html/2608.13570#alg1)\.

The process begins by initializing an embedding sequenceEEwith the image features𝒫​\(𝒱​\(I\)\)\\mathcal\{P\}\(\\mathcal\{V\}\(I\)\), the question embeddingsℰ​\(Xq\)\\mathcal\{E\}\(X\_\{q\}\), and the<bot\>embedding in order\. The model then performsllatentl\_\{\\text\{latent\}\}autoregressive steps: in each step, the model takes the current sequenceEEas input, and the final hidden state from the last positionH​\[−1,:\]H\[\-1,:\]is appended back toEEto serve as the input for the next step\. The final output is a sequence of latent thoughtsElatent∈ℝllatent×dE\_\{\\text\{latent\}\}\\in\\mathbb\{R\}^\{l\_\{\\text\{latent\}\}\\times d\}, which used to compute losses afterwards\.

Algorithm 1Latent Space Reasoning Generation0:Image

II, Question

XqX\_\{q\}, Latent length

llatentl\_\{\\text\{latent\}\}
1:Initialize:

E←\[𝒫​\(𝒱​\(I\)\);ℰ​\(Xq\);ℰ​\(<bot\>\)\]E\\leftarrow\[\\mathcal\{P\}\(\\mathcal\{V\}\(I\)\);\\mathcal\{E\}\(X\_\{q\}\);\\mathcal\{E\}\(\\texttt\{<bot\>\}\)\]
2:Initialize:

Elatent←\[\]E\_\{\\text\{latent\}\}\\leftarrow\[\]
3:for

k=1k=1to

llatentl\_\{\\text\{latent\}\}do

4:

H←LLM​\(E\)H\\leftarrow\\text\{LLM\}\(E\)⊳\\trianglerightForward pass

5:

hlast←H​\[−1,:\]h\_\{\\text\{last\}\}\\leftarrow H\[\-1,:\]⊳\\trianglerightFinal hidden state of last position

6:

E←\[E;hlast\]E\\leftarrow\[E;h\_\{\\text\{last\}\}\]⊳\\trianglerightAppend state as next input

7:

Elatent←\[Elatent;hlast\]E\_\{\\text\{latent\}\}\\leftarrow\[E\_\{\\text\{latent\}\};h\_\{\\text\{last\}\}\]⊳\\trianglerightStore latent thought

8:endfor

9:Return

ElatentE\_\{\\text\{latent\}\}

Thinking Budget\.A key design choice in our framework is the length of the latent reasoning sequencellatentl\_\{\\text\{latent\}\}\. We explore two strategies:

1. 1\.Fixed Length:llatentl\_\{\\text\{latent\}\}is a constant hyperparameter which remains the same for all data instances\.
2. 2\.Variable Length:llatentl\_\{\\text\{latent\}\}is set to match the number of ground\-truth reasoning stepsKKin the data instance\. Therefore, its value varies from one instance to another\.

For models trained with a fixed latent length, we simply terminate the latent generation loop afterllatentl\_\{\\text\{latent\}\}steps at inference time\. For models trained with a variable latent length, a fixed budget is still required at inference as the ground\-truthKKis unavailable\. We have explored training the model to autonomously terminate its latent reasoning, but it was found that this objective is significantly hard to train\. Therefore, we apply a predefined fixed budget for models trained with variable latent length during inference as well\.

#### 3\.2\.2Answer Loss

The Answer Loss is a standard cross\-entropy loss over the ground\-truth answer tokensXaX\_\{a\}, given the imageII, questionXqX\_\{q\}, and latent thoughtsElatentE\_\{\\text\{latent\}\}as context, as depicted in Figure[3](https://arxiv.org/html/2608.13570#S3.F3)\. This loss supervises the model to be capable of arriving at the correct answer with latent space reasoning\.

To compute this loss, we first construct the embedding sequenceEanspromptE\_\{\\text\{ans\}\}^\{\\text\{prompt\}\}, which contains the original inputs, the full latent thought sequence, and the<eot\>token:

Eansprompt=\[𝒫​\(𝒱​\(I\)\);ℰ​\(Xq\);ℰ​\(<bot\>\);Elatent;ℰ​\(<eot\>\)\]\\displaystyle E\_\{\\text\{ans\}\}^\{\\text\{prompt\}\}=\[\\mathcal\{P\}\(\\mathcal\{V\}\(I\)\);\\mathcal\{E\}\(X\_\{q\}\);\\mathcal\{E\}\(\\texttt\{<bot\>\}\);E\_\{\\text\{latent\}\};\\mathcal\{E\}\(\\texttt\{<eot\>\}\)\]The Answer LossLansL\_\{\\text\{ans\}\}is defined as the average negative log\-likelihood over the answer tokens:

Lans\\displaystyle L\_\{\\text\{ans\}\}=−1\|Xa\|​log⁡p​\(Xa\|Eansprompt\)\\displaystyle=\-\\frac\{1\}\{\|X\_\{a\}\|\}\\log p\(X\_\{a\}\|E^\{\\text\{prompt\}\}\_\{\\text\{ans\}\}\)wherelog⁡p​\(Xa\|Eansprompt\)\\log p\(X\_\{a\}\|E^\{\\text\{prompt\}\}\_\{\\text\{ans\}\}\)is the sum of the log\-likelihoods of the tokens inXaX\_\{a\}, conditioned on bothEanspromptE^\{\\text\{prompt\}\}\_\{\\text\{ans\}\}and other preceding tokens in the answer sequence\.

#### 3\.2\.3CoT Loss

The CoT Loss is designed to provide supervision and ensure the interpretability of the latent thoughts\. As introduced earlier, CoT Loss is the loss of the entire CoT sequenceXCoTX\_\{\\text\{CoT\}\}or a single CoT stepXCoTiX\_\{\\text\{CoT\}\_\{i\}\}given the model’s latent space reasoning\. The former type is namedFull CoT Loss, and the latter is namedSingle Step Loss\.

1. 1\.Full CoT Loss \(Global Alignment\)\.Provided with a decoding promptXdecX\_\{\\text\{dec\}\}\(e\.g\., “Provide the reasoning process encoded in the embeddings provided between the<bot\>and<eot\>tokens by the user\.”\) and the entire latent thought sequenceElatentE\_\{\\text\{latent\}\}, the Full CoT Loss is the loss of the entire textual CoTXCoTX\_\{\\text\{CoT\}\}, as shown in Figure[3](https://arxiv.org/html/2608.13570#S3.F3)\. Edec,fullprompt\\displaystyle E\_\{\\text\{dec,full\}\}^\{\\text\{prompt\}\}=\[ℰ​\(Xdec\);ℰ​\(<bot\>\);Elatent;ℰ​\(<eot\>\)\]\\displaystyle=\[\\mathcal\{E\}\(X\_\{\\text\{dec\}\}\);\\mathcal\{E\}\(\\texttt\{<bot\>\}\);E\_\{\\text\{latent\}\};\\mathcal\{E\}\(\\texttt\{<eot\>\}\)\]Ldec,full\\displaystyle L\_\{\\text\{dec,full\}\}=−1\|XCoT\|​log⁡p​\(XCoT\|Edec,fullprompt\)\\displaystyle=\-\\frac\{1\}\{\|X\_\{\\text\{CoT\}\}\|\}\\log p\(X\_\{\\text\{CoT\}\}\|E\_\{\\text\{dec,full\}\}^\{\\text\{prompt\}\}\)
2. 2\.Single Step Loss \(Fine\-Grained Alignment\)\.Provided with the decoding promptXdecX\_\{\\text\{dec\}\}and a single latent thought from the entire latent thought sequenceElatentiE\_\{\\text\{latent\}\_\{i\}\}, the Single Step Loss is the loss of the corresponding textual reasoning stepXCoTiX\_\{\\text\{CoT\}\_\{i\}\}, as shown in Figure[3](https://arxiv.org/html/2608.13570#S3.F3)\. Note that this loss can only be used with variable latent reasoning length, due to the one\-to\-one correspondence between textual reasoning steps and latent thoughts\. Eans,stepprompt\\displaystyle E\_\{\\text\{ans,step\}\}^\{\\text\{prompt\}\}=\[ℰ​\(Xdec\);ℰ​\(Xb​o​t\);Elatenti;ℰ​\(Xe​o​t\)\]\\displaystyle=\[\\mathcal\{E\}\(X\_\{\\text\{dec\}\}\);\\mathcal\{E\}\(X\_\{bot\}\);E\_\{\\text\{latent\}\_\{i\}\};\\mathcal\{E\}\(X\_\{eot\}\)\]Ldec,step\\displaystyle L\_\{\\text\{dec,step\}\}=−1\|XCoTi\|​log⁡p​\(XCoTi\|Eans,stepprompt\)\\displaystyle=\-\\frac\{1\}\{\|X\_\{\\text\{CoT\}\_\{i\}\}\|\}\\log p\(X\_\{\\text\{CoT\}\_\{i\}\}\|E\_\{\\text\{ans,step\}\}^\{\\text\{prompt\}\}\)

It is worth noting that the reasoning step used in each training step is sampled from a distribution,i\.e\., for a data instance withKKreasoning steps, we sample the step indexiifrom a probability distribution over\{1,2,…,K\}\\\{1,2,\\dots,K\\\}\. We consider two types of distributions: \(1\)Uniform, where each step has equal probabilityp​\(i\)=1/Kp\(i\)=1/K; and \(2\)Exponential, where the probability doubles at each step, defined asp​\(i\)=2i/\(2K\+1−1\)p\(i\)=2^\{i\}/\(2^\{K\+1\}\-1\)fori∈\{1,…,K\}i\\in\\\{1,\\dots,K\\\}\.

Crucially, during CoT decoding, the model is*not*shown the original image or question—only the latent thoughts\. This self\-regularization forces the latent thoughts to become a self\-contained information bottleneck, carrying all necessary context for the solution\.

### 3\.3Learning Stages

According to the number of stages involved during training, we categorize our training frameworks into two types: \(1\)Single\-Stage Methods, which optimize a weighted sum of the aforementioned losses throughout training, and \(2\)Multi\-Stage Methods, which employ a curriculum of sequential stages with distinct loss configurations\.

#### 3\.3\.1Single\-Stage Methods

Based on previous discussion, we can see that there are two design options for single\-stage methods: \(1\) the latent reasoning lengthllatentl\_\{\\text\{latent\}\},i\.e\., using fixed or variable latent length, and \(2\) the specific combination of losses to optimize\.

For both LLM and VLM, we define the single\-stage SELR method as using fixed latent length and the weighted sum of Answer Loss and Full CoT Loss as the objective\. Both losses have a weight of 1\.0, thus the objective is effectively the sum of both losses\.

#### 3\.3\.2Multi\-Stage Methods

We observed that the performance of the single\-stage method was limited, particularly for LLMs\. To improve the model’s ability to learn effective latent representations, we introduce a multi\-stage training strategy, inspired by the curriculum learning approach of Coconut\(Haoet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib1)\)\. This method consists of multiple stages: an initial alignment stage, followed by a refinement stage\. The specific implementation differs between LLMs and VLMs\.

For LLM, the first stage trains the model with variable latent length and an objective that sums the Answer Loss and the Single Step Loss\. This aligns the latent thoughts with the ground\-truth reasoning steps\. The second stage then trains the model with fixed latent length, and switches the objective to the sum of Answer Loss and Full CoT Loss\.

For VLM, the first stage is identical to the LLM method\. In the second stage, the model is also trained with fixed latent length\. However, unlike the LLM method, we continue to use the sum of the Answer Loss and the Single Step Loss instead of switching to the Full CoT Loss\. This design is intentional\. The ground\-truth reasoning steps in the VLM dataset, LLaVA\-CoT\-100k\(Xuet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib5)\), are long and information\-dense, often spanning multiple sentences, whereas the LLM dataset’s steps are single math equations\. In support of this design choice, we find that the decoding quality degrades if we use Full CoT Loss for the second stage in Section[4\.2\.3](https://arxiv.org/html/2608.13570#S4.SS2.SSS3)\. Crucially, every instance in LLaVA\-CoT\-100k provides exactly three fixed stages \(summary, caption, reasoning\), so the VLM latent length is effectively a constant, i\.e\.,llatent=K=3l\_\{\\text\{latent\}\}=K=3\. This preserves the one\-to\-one correspondence between latent thoughts and reasoning steps required by Single Step Loss even in the second stage, and the fixed ordering \(summary→\\rightarrowcaption→\\rightarrowreasoning\) allows each latent position to specialize in encoding a specific type of information\. See Appendix[K](https://arxiv.org/html/2608.13570#A11)for a detailed discussion on step ordering\.

## 4Experiments

Table 1:Main VLM benchmark results\. All presented methods are based on Qwen2\.5\-VL\-3B\-Instruct\. The value enclosed by parentheses in the Average column reflects the change from the base model\. SELR not only improves base model accuracy \(in contrast, Heima degrades accuracy\), but also drastically reduces token count\. Best results on each benchmark is shown inbold\.Table 2:LLM results on GSM8k, SVAMP, GSM\-Hard, and MultiArith\. All models are based on LLaMA\-3\.2\-1B\-Instruct\. SELR \(Multi\) consistently improves upon the single\-stage variant and the Coconut baseline, and remains highly competitive with CoLaR\.In this section, we first introduce the experiment setup, including the datasets, implementation details, baselines, and evaluation protocols for our VLM and LLM experiments\. We then present the main results, where we compare SELR against baselines on a suite of benchmarks and datasets\. Finally, we conduct a comprehensive ablation study to validate our design choices and provide insights into the effectiveness and interpretability of our latent reasoning method\.

### 4\.1Experiment Setup

#### 4\.1\.1Dataset

VLM Training\.We use the LLaVA\-CoT\-100k dataset\(Xuet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib5)\)for VLM training\. This is a specialized reasoning dataset for VLMs, comprising 100k image\-text pairs\. It integrates samples from several widely used VQA datasets and is notable for providing three stages of CoT reasoning for each sample: summary, caption, and reasoning\.

LLM Training\.For LLM experiments, we use the GSM8k\-Aug dataset proposed byDenget al\.\([2023](https://arxiv.org/html/2608.13570#bib.bib6)\)\. It is an augmented version of the GSM8k\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.13570#bib.bib7)\)training set, containing 385k grade\-school math problems generated by prompting GPT\-4\.

#### 4\.1\.2Model Training

We use Qwen2\.5\-VL\-3B\-Instruct\(Baiet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib8)\)for VLM experiments and LLaMA\-3\.2\-1B\-Instruct\(Meta,[2024b](https://arxiv.org/html/2608.13570#bib.bib9)\)for LLM experiments\. All models are fine\-tuned using LoRA\(Huet al\.,[2022](https://arxiv.org/html/2608.13570#bib.bib10)\)and trained with the AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2608.13570#bib.bib11)\)optimizer using the Hugging Face Accelerate library\(Guggeret al\.,[2022](https://arxiv.org/html/2608.13570#bib.bib12)\)\. For VLM training, all SELR methods are trained for a total of 3 epochs\. For LLM training, the single\-stage method is trained for 10 epochs, while the multi\-stage method is trained for 10 epochs in the first phase and 5 epochs in the second\. Detailed hyperparameters, including LoRA configurations and scheduler details, are provided in Appendix[A](https://arxiv.org/html/2608.13570#A1)\.

#### 4\.1\.3Baselines

For our LLM evaluation, we compare against three methods that share the same LLaMA\-3\.2\-1B\-Instruct backbone: \(1\)CoT\-SFT:Model finetuned directly on the ground\-truth CoTs and answers in the GSM8k\-Aug dataset for 15 epochs\. \(2\)Coconut:A latent space reasoning method trained by gradually replacing textual CoT steps with latent thoughts\(Haoet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib1)\)\. To ensure fair comparison, we retrain the method for 15 epochs as well with its original code\. Specific training configurations are detailed in Appendix[A](https://arxiv.org/html/2608.13570#A1)\. \(3\)CoLaR:A latent reasoning framework that dynamically compresses multiple consecutive reasoning tokens into single latent embeddings via an auxiliary prediction objective\(Tanet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib23)\)\. We present its best reported results, which use a test\-time compression factor of 2, for comparison\.

For the VLM experiments, we utilize Qwen2\.5\-VL\-3B\-Instruct as the backbone\. In addition to the base model, we compare against a LLaVA\-CoT\-100k fine\-tuned version and a reimplementation of Heima\(Shenet al\.,[2025a](https://arxiv.org/html/2608.13570#bib.bib3)\)\. Further implementation details are provided in Appendix[A](https://arxiv.org/html/2608.13570#A1)\.

Here we also clarify why two other recent latent reasoning methods are not included as direct baselines\. HRPO\(Yueet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib24)\)mixes latent embeddings with discrete text tokens via a learnable gating mechanism rather than fully replacing textual CoT with latent representations, and therefore does not achieve the same reasoning length compression as purely latent methods\. CODI\(Shenet al\.,[2025b](https://arxiv.org/html/2608.13570#bib.bib25)\)compresses CoT into continuous space via self\-distillation but does not provide any mechanism for decoding or explaining its latent representations\. Since SELR’s primary contribution is self\-explainability, comparing against a method without this capability would not meaningfully evaluate our core claim\. Both methods are complementary to SELR rather than direct competitors\.

#### 4\.1\.4Evaluation

To comprehensively evaluate our methods, we test our models on a wide array of benchmarks and datasets\.

VLM Evaluation\.In accordance with Heima, we perform zero\-shot evaluation on MMStar\(Chenet al\.,[2024](https://arxiv.org/html/2608.13570#bib.bib26)\), MMBenchV1\.1\(Liuet al\.,[2024](https://arxiv.org/html/2608.13570#bib.bib27)\), MMVet\(Yuet al\.,[2024](https://arxiv.org/html/2608.13570#bib.bib28)\), MathVista\(Luet al\.,[2024](https://arxiv.org/html/2608.13570#bib.bib30)\), AI2D\(Hiippalaet al\.,[2020](https://arxiv.org/html/2608.13570#bib.bib31)\), and HallusionBench\(Guanet al\.,[2024](https://arxiv.org/html/2608.13570#bib.bib29)\)\. MMStar, MMBench, and MMVet are used to evaluate core visual question\-answering and reasoning abilities\. MathVista and AI2D test complex mathematical and scientific diagram understanding\. Hallusion is used to measure the model’s tendency to produce factual inaccuracies or hallucinations\. The evaluations are conducted with VLMEvalKit\(Duanet al\.,[2024](https://arxiv.org/html/2608.13570#bib.bib32)\)for standardized evaluation, using GPT\-4o\(OpenAI,[2024a](https://arxiv.org/html/2608.13570#bib.bib33)\)to score the responses on MMVet and MathVista, and exact matching for all other benchmarks\. For decoding quality evaluation, we split the LLaVA\-CoT\-100k dataset into a 95% train split and a 5% test split, and retrain the SELR methods on the train split\. We also utilize BLEU\-4\(Papineniet al\.,[2002](https://arxiv.org/html/2608.13570#bib.bib34)\), METEOR\(Banerjee and Lavie,[2005](https://arxiv.org/html/2608.13570#bib.bib36)\), ROUGE\(Lin,[2004](https://arxiv.org/html/2608.13570#bib.bib35)\), BERTScore\(Zhanget al\.,[2020](https://arxiv.org/html/2608.13570#bib.bib37)\), and GPT\-4o to evaluate the similarity of the decoded results on the test set\. The prompt used for similarity evaluation with GPT\-4o is provided in Appendix[F](https://arxiv.org/html/2608.13570#A6)\.

LLM Evaluation\.We perform in\-domain evaluation on the GSM8k test set and out\-of\-distribution \(OOD\) evaluation on SVAMP\(Patelet al\.,[2021](https://arxiv.org/html/2608.13570#bib.bib38)\), MultiArith\(Roy and Roth,[2015](https://arxiv.org/html/2608.13570#bib.bib39)\), and GSM\-Hard\(Gaoet al\.,[2023](https://arxiv.org/html/2608.13570#bib.bib40)\)\. We report the final answer accuracy for all benchmarks\.

### 4\.2Main Results

#### 4\.2\.1VLM Results

In Table[1](https://arxiv.org/html/2608.13570#S4.T1), Qwen2\.5\-VL\-3B\-Instruct\-SFT is the base model finetuned on LLaVA\-CoT\-100k for 3 epochs; SELR \(Answer Loss Only\) refers to the method of training the model with only the Answer Loss; SELR \(Single\) refers to the single\-stage method for VLM training \(Section[3\.3\.1](https://arxiv.org/html/2608.13570#S3.SS3.SSS1)\); SELR \(Multi, Uniform\) and SELR \(Multi, Exponential\) refer to the multi\-stage method for VLM training \(Section[3\.3\.2](https://arxiv.org/html/2608.13570#S3.SS3.SSS2)\), using the uniform distribution and exponential distribution for the Single Step Loss, respectively\.

In terms of performance, Table[1](https://arxiv.org/html/2608.13570#S4.T1)shows that SELR has a consistent positive gain over the average performance of the original model, improving by 0\.86% at best\. As comparison, Heima’s performance is worse than the base model, showing a 1\.92% drop\. This shows that our method, despite introducing great changes in the model’s generation paradigm, still preserves and even improves the model’s general and reasoning capabilities\.

SELR also shows great improvements on efficiency, reducing the response length by a large margin, while improving performance at the same time\. Our multi\-stage variants achieve an average response length of approximately 13 tokens and a reduction of over 70% from our base model\. Our model responses are also more concise than the Heima baseline, which averages 22\.84 tokens per response\. We further validate generalization at the 7B scale in Appendix[H](https://arxiv.org/html/2608.13570#A8)and on a different training dataset without fixed\-format structure in Appendix[J](https://arxiv.org/html/2608.13570#A10)\. We also report wall\-clock inference latency in Appendix[G](https://arxiv.org/html/2608.13570#A7), where SELR achieves an 8\.7×\\timesspeedup over SFT\.

MMVet Performance\.The MMVet score for SELR multi\-stage variants is lower than the base model\. This is not a general degradation but is localized to specific capability splits: the decline is driven by compromised multi\-step reasoning \(e\.g\., math sub\-problems\) and long\-form generation capabilities \(e\.g\., free\-form writing\), both of which rely on extended token generation that latent compression naturally limits\. This is a shared challenge for latent\-space thinking models: Heima also degrades on MMVet, and more severely than SELR \(34\.26 vs\. 36\.10\)\. SELR outperforms baselines on pure visual perception tasks within MMVet \(e\.g\., direct OCR, spatial localization\), partially offsetting the overall decline\.

SFT Underperformance\.Qwen2\.5\-VL\-3B\-Instruct\-SFT performs worse on average than the original model despite being fine\-tuned on LLaVA\-CoT\-100k\. This is not a bug but an expected consequence of training on the specially structured CoT data: the dataset requires responses in a rigid three\-step format, introducing a format shift from the base model’s default behavior\. This forces the model to produce multi\-stage reasoning even for simple questions where the base model would answer concisely, and the structured format may conflict with pre\-training reasoning patterns\. We observe the same degradation at the 7B scale in Table[16](https://arxiv.org/html/2608.13570#A8.T16)\.

Token Count\.The token counts in Table[1](https://arxiv.org/html/2608.13570#S4.T1)are the total generated output tokens, including latent special tokens<bot\>/<eot\>and the final answer,*not*decoded reasoning tokens\.

#### 4\.2\.2LLM Results

The LLM results are presented in Table[2](https://arxiv.org/html/2608.13570#S4.T2)\. In the table, SELR \(Single\) refers to the model trained with the single\-stage method \(Section[3\.3\.1](https://arxiv.org/html/2608.13570#S3.SS3.SSS1)\); SELR \(Multi\) refers to the model trained with the multi\-stage method \(Section[3\.3\.2](https://arxiv.org/html/2608.13570#S3.SS3.SSS2)\)\.

As we can see from Table[2](https://arxiv.org/html/2608.13570#S4.T2), SELR \(Multi\) outperforms the Coconut baseline and our single\-stage variant on both in\-domain \(e\.g\.,, GSM8k\) and out\-of\-domain \(OOD\) datasets \(e\.g\.,, SVAMP\), demonstrating the effectiveness of the multi\-stage training curriculum\. Furthermore, it remains highly competitive with CoLaR, notably surpassing it on the in\-domain GSM8k dataset and the challenging GSM\-Hard benchmark\. It is also notable that the reasoning length of CoLaR is longer than SELR, being at most more than twice as long \(14\.0 vs\. 6\.0\)\.

We note that the remaining gap between SELR and CoT\-SFT reflects a difference in reasoning budget rather than reasoning quality: CoT\-SFT generates over 150 text tokens per question, while SELR uses only 6 latent tokens\. When CoT\-SFT is constrained to the same 6\-token budget, its performance collapses to 10\.31% on GSM8k, compared to SELR’s 42\.46% \. See Appendix[I](https://arxiv.org/html/2608.13570#A9)for the full controlled\-budget comparison\.

#### 4\.2\.3Decoding Quality

As mentioned in Section[4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4), we also evaluate the decoding results with several evaluation metrics and GPT\-4o\. The results are presented in Table[3](https://arxiv.org/html/2608.13570#S4.T3)\.

As we can see from Table[3](https://arxiv.org/html/2608.13570#S4.T3), the summaries, captions, and reasoning decoded from the latent thoughts are more similar to the ground truth than Heima\. The VLM methods that use the Single Step Loss can decode the corresponding latent thoughts back to textual summaries, captions, and reasoning consistently better\. Notably, in addition to the Heima encoder, Heima trains a decoder for each of the latent thoughts \(summary, caption, reasoning\), therefore 4 models are involved to decode the latent thoughts\. However, in SELR, only one model is needed\. Additionally, the Heima decoders are based on LLaMA3\.1\-8B\-Instruct\(Meta,[2024b](https://arxiv.org/html/2608.13570#bib.bib9)\), which is more than twice the size of our backbone\. Despite this significant architectural advantage favoring the Heima baseline, our method achieves superior decoding quality\. This observation demonstrates the synergy enabled by unifying reasoning and self\-explanation in a shared model\.

When evaluating CoT similarity of the full decoded sequence with the original CoT sequence in Table[4](https://arxiv.org/html/2608.13570#S4.T4), it is evident that the VLM methods that use the Single Step Loss achieve better similarity, which confirms our intuition that step\-wise decoding is better than decoding the full sequence at the same time for VLMs\.

Table 3:Decoding quality evaluation\. We compare the similarity of the decoded text to the ground\-truth for theSummary,Caption, andReasoningsteps from the LLaVA\-CoT\-100k test split\. Our SELR methods consistently outperform the Heima baseline across all steps\.Table 4:Full CoT Loss vs\. Single Step Loss for decoding\. We compare the full\-sequence decoding quality of the single\-stage method against the multi\-stage methods\. For the multi\-stage methods, the full CoT is acquired by concatenating the decoded summary, caption, and reasoning\.
#### 4\.2\.4Faithfulness Analysis

A natural question is whether the decoded CoT faithfully reflects the model’s actual reasoning process\. We address this with a*consistency metric*: GPT\-4o scores the alignment between the decoded thoughts and the model’s final answer on a 0–1 scale, evaluated on a held\-out 5% test split of LLaVA\-CoT\-100k\. Results are shown in Table[5](https://arxiv.org/html/2608.13570#S4.T5)\.

Table 5:Faithfulness analysis via consistency metric\. We report GPT\-4o alignment scores between decoded CoT and the final answer, stratified by whether the model answered correctly\.The substantial gap between correct \(∼0\.5\{\\sim\}0\.5\) and incorrect \(∼0\.26\{\\sim\}0\.26\) predictions demonstrates that the decoded CoT meaningfully correlates with the model’s reasoning success, suggesting it reflects genuine reasoning content rather than generic post\-hoc outputs\. Among variants, SELR \(Multi, Exponential\) yields the highest faithfulness, validating our design choices\.

To directly compare against Heima, we evaluate both methods under the same consistency metric on MathVista\. As shown in Table[6](https://arxiv.org/html/2608.13570#S4.T6), SELR \(Single\) achieves a consistency score of 0\.2097, substantially higher than Heima’s 0\.1671 \(\+25\.5% relative improvement\)\. This advantage stems from SELR’s unified architecture: the same model parameters responsible for reasoning are also responsible for decoding, inherently aligning the explanation with the actual computation\. In contrast, Heima’s separate\-decoder design creates a structural gap between the model that reasons and the model that explains\. Combined with the decoding\-quality comparison in Table[3](https://arxiv.org/html/2608.13570#S4.T3), this provides end\-to\-end evidence that SELR is more faithful than Heima on both proxies for faithfulness\.

Table 6:Consistency comparison with Heima on MathVista\. SELR achieves substantially higher consistency scores, indicating that its decoded thoughts are more aligned with the model’s final answer\.

### 4\.3Ablation Study

#### 4\.3\.1VLM Ablations

CoT Loss\.Comparing SELR \(Answer Loss Only\) with the other three SELR methods in Table[1](https://arxiv.org/html/2608.13570#S4.T1), we can see that training the model only on the Answer Loss is detrimental to the model’s performance\. In stark contrast, introducing any type of CoT Loss immediately reverses this degradation, yielding gains over the base model\. This strongly indicates that by forcing the latent thoughts to be explainable, it guides the model to learn more effective and structured representations, which in turn improves the quality of the answer generation\. This conclusion is further corroborated with LLM ablations, detailed in the next section\.

SELR vs\. Supervised Fine\-Tuning\.We conduct an ablation to isolate the source of our performance gains\. A key question is whether the improvement comes from our SELR framework or simply from exposing the model to the LLaVA\-CoT\-100k dataset\. To test this, we trained a Qwen2\.5\-VL\-3B\-Instruct\-SFT baseline, which fine\-tunes the base model on the same dataset using a standard textual CoT objective\. As shown in Table[1](https://arxiv.org/html/2608.13570#S4.T1), this SFT approach is less performant\. This demonstrates that the performance gain is not an artifact of the dataset, but is a direct result of our methods\. The SELR methods successfully teaches the model to internalize the reasoning process, leading to higher accuracy and reduction in response length\.

#### 4\.3\.2LLM Ablations

Table 7:Ablation study of single\-stage SELR on LLM benchmarks\. Results show that the CoT Loss and the fixed latent length are critical for performance\.Table 8:Ablation study of multi\-stage SELR on LLM benchmarks\. We validate the effectiveness of the components in the multi\-stage method\. The results confirm the benefits of using an exponential sampling distribution, including the Single Step Loss in Stage 1, and using a fixed latent length\.Single\-Stage Methods\.Similar to the VLM ablations, Table[7](https://arxiv.org/html/2608.13570#S4.T7)shows that the CoT loss is beneficial to the answer generation\. Also, we can see that training with a fixed latent length is better than training with variable length, which coincides with Coconut’s observation that training an auxiliary model to predict the ending of the latent space reasoning achieves similar results to fixing the latent space reasoning length\.

Multi\-Stage Methods\.As in Table[8](https://arxiv.org/html/2608.13570#S4.T8), each option is integral to the final performance of the multi\-stage method: the exponential distribution focuses on the more difficult ending steps; the Answer Loss during the first phase helps with regularizing the latent thoughts; and again, using a fixed latent length helps with performance\.

## 5Conclusion

In this work, we propose Self\-Explainable Latent Reasoning \(SELR\), a unified framework that trains a single, unified model to be both an efficient reasoner and its own translator\. Our core contribution is a multi\-task learning objective that combines a standard answer loss with a CoT loss\. Experimental results demonstrate that our method is effective for both VLM and LLM tasks, yielding performance gains while significantly reducing response length\. By successfully bridging the gap between efficiency and interpretability, SELR provides a practical path toward developing efficient and explainable latent space reasoning models\. A promising avenue for future work is the development of a stable, learned stopping mechanism, which we have identified as a significant challenge\.

## Acknowledgements

This work was supported in part by NSF under Grants 2106825 and 2519216, the DARPA Young Faculty Award, the ONR Grant N00014\-26\-1\-2099, the NIFA Award 2020\-67021\-32799, the Amazon\-Illinois Center on AI for Interactive Conversational Experiences, the Capital One Illinois Center for Generative AI Safety, Knowledge Systems, and Cybersecurity, the IBM\-Illinois Discovery Accelerator Institute, and Apple AIML Academic Research Program\. This work used computational resources, including the NCSA Delta and DeltaAI supercomputers through allocations CIS230012, CIS230013, CIS240133, and CIS240387 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support \(ACCESS\) program, as well as the TACC Frontera supercomputer, Amazon Web Services \(AWS\), and OpenAI API through the National Artificial Intelligence Research Resource \(NAIRR\) Pilot\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here\.

## References

- S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin \(2025\)Qwen2\.5\-VL technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[§A\.1](https://arxiv.org/html/2608.13570#A1.SS1.p4.1),[§A\.2](https://arxiv.org/html/2608.13570#A1.SS2.p1.1),[Appendix H](https://arxiv.org/html/2608.13570#A8.p1.1),[§1](https://arxiv.org/html/2608.13570#S1.p6.1),[§4\.1\.2](https://arxiv.org/html/2608.13570#S4.SS1.SSS2.p1.1)\.
- B\. Baker, J\. Huizinga, L\. Gao, Z\. Dou, M\. Y\. Guan, A\. Madry, W\. Zaremba, J\. Pachocki, and D\. Farhi \(2025\)Monitoring reasoning models for misbehavior and the risks of promoting obfuscation\.arXiv preprint arXiv:2503\.11926\.Cited by:[Appendix L](https://arxiv.org/html/2608.13570#A12.p2.1)\.
- S\. Banerjee and A\. Lavie \(2005\)METEOR: an automatic metric for MT evaluation with improved correlation with human judgments\.InProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization,pp\. 65–72\.Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- L\. Chen, J\. Li, X\. Dong, P\. Zhang, Y\. Zang, Z\. Chen, H\. Duan, J\. Wang, Y\. Qiao, D\. Lin, and F\. Zhao \(2024\)Are we on the right way for evaluating large vision\-language models?\.InNeurIPS,Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.1\.1](https://arxiv.org/html/2608.13570#S4.SS1.SSS1.p2.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p1.1)\.
- Y\. Deng, K\. Prasad, R\. Fernandez, P\. Smolensky, V\. Chaudhary, and S\. Shieber \(2023\)Implicit chain of thought reasoning via knowledge distillation\.arXiv preprint arXiv:2311\.01460\.Cited by:[§4\.1\.1](https://arxiv.org/html/2608.13570#S4.SS1.SSS1.p2.1)\.
- A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby \(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.InICLR,Cited by:[§3\.1](https://arxiv.org/html/2608.13570#S3.SS1.p1.12)\.
- H\. Duan, X\. Fang, J\. Yang, X\. Zhao, Y\. Qiao, M\. Li, A\. Agarwal, Z\. Chen, L\. Chen, Y\. Liu, Y\. Ma, H\. Sun, Y\. Zhang, S\. Lu, T\. H\. Wong, W\. Wang, P\. Zhou, X\. Li, C\. Fu, J\. Cui, J\. Chen, E\. Song, S\. Mao, S\. Ding, T\. Liang, Z\. Zhang, X\. Dong, Y\. Zang, P\. Zhang, J\. Wang, D\. Lin, and K\. Chen \(2024\)VLMEvalKit: an open\-source toolkit for evaluating large multi\-modality models\.InACM Multimedia,Cited by:[§A\.2](https://arxiv.org/html/2608.13570#A1.SS2.p1.1),[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)PAL: program\-aided language models\.InICML,Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p3.1)\.
- J\. Geiping, S\. McLeish, N\. Jain, J\. Kirchenbauer, S\. Singh, B\. R\. Bartoldson, B\. Kailkhura, A\. Bhatele, and T\. Goldstein \(2025\)Scaling up test\-time compute with latent reasoning: a recurrent depth approach\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p2.1)\.
- A\. Gladstone, G\. Nanduru, M\. M\. Islam, P\. Han, H\. Ha, A\. Chadha, Y\. Du, H\. Ji, J\. Li, and T\. Iqbal \(2025\)Energy\-based transformers are scalable learners and thinkers\.arXiv preprint arXiv:2507\.02092\.Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p2.1)\.
- T\. Guan, F\. Liu, X\. Wu, R\. Xian, Z\. Li, X\. Liu, X\. Wang, L\. Chen, F\. Huang, Y\. Yacoob, D\. Manocha, and T\. Zhou \(2024\)HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision\-language models\.InCVPR,Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- S\. Gugger, L\. Debut, T\. Wolf, P\. Schmid, Z\. Mueller, S\. Mangrulkar, M\. Sun, and B\. Bossan \(2022\)Accelerate: training and inference at scale made simple, efficient and adaptable\.\.Note:[https://github\.com/huggingface/accelerate](https://github.com/huggingface/accelerate)Cited by:[§4\.1\.2](https://arxiv.org/html/2608.13570#S4.SS1.SSS2.p1.1)\.
- S\. Hao, S\. Sukhbaatar, D\. Su, X\. Li, Z\. Hu, J\. Weston, and Y\. Tian \(2025\)Training large language models to reason in a continuous latent space\.InCOLM,Cited by:[§A\.1](https://arxiv.org/html/2608.13570#A1.SS1.p3.1),[Table 17](https://arxiv.org/html/2608.13570#A9.T17.5.1.5.4.1),[§1](https://arxiv.org/html/2608.13570#S1.p1.1),[§1](https://arxiv.org/html/2608.13570#S1.p2.1),[§1](https://arxiv.org/html/2608.13570#S1.p3.1),[§2](https://arxiv.org/html/2608.13570#S2.p2.1),[§3\.3\.2](https://arxiv.org/html/2608.13570#S3.SS3.SSS2.p1.1),[§4\.1\.3](https://arxiv.org/html/2608.13570#S4.SS1.SSS3.p1.1),[Table 2](https://arxiv.org/html/2608.13570#S4.T2.4.1.2.2.1)\.
- T\. Hiippala, M\. Alikhani, J\. Haverinen, T\. Kalliokoski, E\. Logacheva, S\. Orekhova, A\. Tuomainen, M\. Stone, and J\. A\. Bateman \(2020\)AI2D\-RST: a multimodal corpus of 1000 primary school science diagrams\.Language Resources and Evaluation55\(3\),pp\. 661–688\.Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InICLR,Cited by:[§4\.1\.2](https://arxiv.org/html/2608.13570#S4.SS1.SSS2.p1.1)\.
- Y\. Jia, J\. Li, X\. Yue, B\. Li, P\. Nie, K\. Zou, and W\. Chen \(2025\)VisualWebInstruct: scaling up multimodal instruction data through web search\.InEMNLP,Cited by:[Appendix J](https://arxiv.org/html/2608.13570#A10.p1.1)\.
- T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa \(2022\)Large language models are zero\-shot reasoners\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p1.1)\.
- T\. Lanham, A\. Chen, A\. Radhakrishnan, B\. Steiner, C\. Denison, D\. Hernandez, D\. Li, E\. Durmus, E\. Hubinger, J\. Kernion, K\. Lukošiūtė, K\. Nguyen, N\. Cheng, N\. Joseph, N\. Schiefer, O\. Rausch, R\. Larson, S\. McCandlish, S\. Kundu, S\. Kadavath, S\. Yang, T\. Henighan, T\. Maxwell, T\. Telleen\-Lawton, T\. Hume, Z\. Hatfield\-Dodds, J\. Kaplan, J\. Brauner, S\. R\. Bowman, and E\. Perez \(2023\)Measuring faithfulness in chain\-of\-thought reasoning\.arXiv preprint arXiv:2307\.13702\.Cited by:[Appendix L](https://arxiv.org/html/2608.13570#A12.p2.1)\.
- C\. Lin \(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,pp\. 74–81\.Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu, K\. Chen, and D\. Lin \(2024\)MMBench: is your multi\-modal model an all\-around player?\.InECCV,Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.InICLR,Cited by:[§4\.1\.2](https://arxiv.org/html/2608.13570#S4.SS1.SSS2.p1.1)\.
- P\. Lu, H\. Bansal, T\. Xia, J\. Liu, C\. Li, H\. Hajishirzi, H\. Cheng, K\. Chang, M\. Galley, and J\. Gao \(2024\)MathVista: evaluating mathematical reasoning of foundation models in visual contexts\.InICLR,Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- Meta \(2024a\)Llama 3\.2: revolutionizing edge ai and vision with open, customizable models\.Note:[https://ai\.meta\.com/blog/llama\-3\-2\-connect\-2024\-vision\-edge\-mobile\-devices/](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)Cited by:[§A\.1](https://arxiv.org/html/2608.13570#A1.SS1.p5.1)\.
- Meta \(2024b\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1\.2](https://arxiv.org/html/2608.13570#S4.SS1.SSS2.p1.1),[§4\.2\.3](https://arxiv.org/html/2608.13570#S4.SS2.SSS3.p2.1)\.
- OpenAI \(2024a\)GPT\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- OpenAI \(2024b\)OpenAI o1 system card\.arXiv preprint arXiv:2412\.16720\.Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p1.1)\.
- K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu \(2002\)Bleu: a method for automatic evaluation of machine translation\.InACL,Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- A\. Patel, S\. Bhattamishra, and N\. Goyal \(2021\)Are NLP models really able to solve simple math word problems?\.InNAACL,Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p3.1)\.
- S\. Roy and D\. Roth \(2015\)Solving general arithmetic word problems\.InEMNLP,Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p3.1)\.
- X\. Shen, Y\. Wang, X\. Shi, Y\. Wang, P\. Zhao, and J\. Gu \(2025a\)Efficient reasoning with hidden thinking\.arXiv preprint arXiv:2501\.19201\.Cited by:[§A\.1](https://arxiv.org/html/2608.13570#A1.SS1.p5.1),[Appendix K](https://arxiv.org/html/2608.13570#A11.p1.2),[Appendix F](https://arxiv.org/html/2608.13570#A6.p1.1),[§1](https://arxiv.org/html/2608.13570#S1.p3.1),[§2](https://arxiv.org/html/2608.13570#S2.p2.1),[§4\.1\.3](https://arxiv.org/html/2608.13570#S4.SS1.SSS3.p2.1),[Algorithm 2](https://arxiv.org/html/2608.13570#alg2)\.
- Z\. Shen, H\. Yan, L\. Zhang, Z\. Hu, Y\. Du, and Y\. He \(2025b\)CODI: compressing chain\-of\-thought into continuous space via self\-distillation\.InEMNLP,Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p2.1),[§4\.1\.3](https://arxiv.org/html/2608.13570#S4.SS1.SSS3.p3.1)\.
- W\. Tan, J\. Li, J\. Ju, Z\. Luo, J\. Luan, and R\. Song \(2025\)Think silently, think fast: dynamic latent compression of LLM reasoning chains\.InNeurIPS,Cited by:[Table 17](https://arxiv.org/html/2608.13570#A9.T17.5.1.6.5.1),[§2](https://arxiv.org/html/2608.13570#S2.p2.1),[§4\.1\.3](https://arxiv.org/html/2608.13570#S4.SS1.SSS3.p1.1),[Table 2](https://arxiv.org/html/2608.13570#S4.T2.4.1.3.3.1)\.
- M\. Turpin, J\. Michael, E\. Perez, and S\. R\. Bowman \(2023\)Language models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.InNeurIPS,Cited by:[Appendix L](https://arxiv.org/html/2608.13570#A12.p2.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InICLR,Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2608.13570#S1.p1.1),[§2](https://arxiv.org/html/2608.13570#S2.p1.1)\.
- G\. Xu, P\. Jin, Z\. Wu, H\. Li, Y\. Song, L\. Sun, and L\. Yuan \(2025\)LLaVA\-CoT: let vision language models reason step\-by\-step\.InICCV,Cited by:[§A\.1](https://arxiv.org/html/2608.13570#A1.SS1.p4.1),[Appendix K](https://arxiv.org/html/2608.13570#A11.p1.2),[§3\.3\.2](https://arxiv.org/html/2608.13570#S3.SS3.SSS2.p3.3),[§4\.1\.1](https://arxiv.org/html/2608.13570#S4.SS1.SSS1.p1.1)\.
- S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. Griffiths, Y\. Cao, and K\. Narasimhan \(2023\)Tree of thoughts: deliberate problem solving with large language models\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p1.1)\.
- W\. Yu, Z\. Yang, L\. Li, J\. Wang, K\. Lin, Z\. Liu, X\. Wang, and L\. Wang \(2024\)MM\-Vet: evaluating large multimodal models for integrated capabilities\.InICML,Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- Z\. Yue, B\. Jin, H\. Zeng, H\. Zhuang, Z\. Qin, J\. Yoon, L\. Shang, J\. Han, and D\. Wang \(2025\)Hybrid latent reasoning via reinforcement learning\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p2.1),[§4\.1\.3](https://arxiv.org/html/2608.13570#S4.SS1.SSS3.p3.1)\.
- T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi \(2020\)BERTScore: evaluating text generation with BERT\.InICLR,Cited by:[§4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4.p2.1)\.
- Z\. Zhang, A\. Zhang, M\. Li, hai zhao, G\. Karypis, and A\. Smola \(2024\)Multimodal chain\-of\-thought reasoning in language models\.TMLR\.Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p1.1)\.
- H\. Zhu, S\. Hao, Z\. Hu, J\. Jiao, S\. Russell, and Y\. Tian \(2025a\)Reasoning by superposition: a theoretical perspective on chain of continuous thought\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p2.1)\.
- R\. Zhu, Z\. Wang, K\. Hua, T\. Zhang, Z\. Li, H\. Que, B\. Wei, Z\. Wen, F\. Yin, H\. Xing, L\. Li, J\. Shi, K\. Ma, S\. Li, T\. Kergan, A\. Smith, X\. Qu, M\. Hui, B\. Wu, Q\. Min, H\. Huang, X\. Zhou, W\. Ye, J\. Liu, J\. Yang, Y\. Shi, C\. Lin, E\. Zhao, T\. Cai, G\. Zhang, W\. Huang, Y\. Bengio, and J\. Eshraghian \(2025b\)Scaling latent reasoning via looped language models\.arXiv preprint arXiv:2510\.25741\.Cited by:[§2](https://arxiv.org/html/2608.13570#S2.p2.1)\.

## Appendix

This appendix provides comprehensive supplementary material to support the main paper\. Specifically, Appendix[A](https://arxiv.org/html/2608.13570#A1)details our experimental setups\. Appendix[B](https://arxiv.org/html/2608.13570#A2)presents additional qualitative examples and visualizations of decoded latent thoughts\. Appendix[C](https://arxiv.org/html/2608.13570#A3)evaluates our Self\-Explainable Latent Reasoning \(SELR\) framework on alternative SFT base configurations\. Appendix[D](https://arxiv.org/html/2608.13570#A4)and Appendix[E](https://arxiv.org/html/2608.13570#A5)present ablation studies regarding the choice of fixed latent lengths and the evaluation protocol for variable\-length architectures, respectively\. Appendix[F](https://arxiv.org/html/2608.13570#A6)provides the exact prompt template utilized for GPT\-4o automated evaluation\. Appendix[G](https://arxiv.org/html/2608.13570#A7)reports empirical wall\-clock inference latencies\. Appendix[H](https://arxiv.org/html/2608.13570#A8)demonstrates performance scaling results on a larger 7B parameter foundation model\. Appendix[I](https://arxiv.org/html/2608.13570#A9)provides a controlled\-budget analysis contrasting SELR against Chain\-of\-Thought \(CoT\) fine\-tuning\. Appendix[J](https://arxiv.org/html/2608.13570#A10)evaluates out\-of\-distribution dataset generalization on unstructured text annotations\. Appendix[K](https://arxiv.org/html/2608.13570#A11)discusses the ordering design choices for visual reasoning steps\. Finally, Appendix[L](https://arxiv.org/html/2608.13570#A12)outlines the limitations of this work along with avenues for future research\.

## Appendix AImplementation Details

In this section, we introduce the details of our model training and evaluation\.

### A\.1Training Configuration

We first detail the hyperparameters used for training our SELR methods, as well as those for the reimplemented baselines\. The specific hyperparameters for experiments on LLMs are listed in Table[9](https://arxiv.org/html/2608.13570#A1.T9), while the hyperparameters for VLM experiments are provided in Table[10](https://arxiv.org/html/2608.13570#A1.T10)\.

As outlined in Section[3\.3](https://arxiv.org/html/2608.13570#S3.SS3), the multi\-stage SELR method on LLMs is divided into two distinct stages\. The first stage consists of 10 epochs with a learning rate of 5e\-4, followed by a second stage of 5 epochs with a reduced learning rate of 1e\-4\. This results in a total training duration of 15 epochs\.

For Coconut\(Haoet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib1)\), to ensure a rigorous comparison, we reproduce it directly using the official repository code111[https://github\.com/facebookresearch/coconut](https://github.com/facebookresearch/coconut)\. We set the total training budget to 15 epochs to match the setting of our method: following the original paper’s curriculum setup, the model is trained for 3 epochs in the initial stage, followed by 3 epochs for each of the remaining stages \(comprising 3 intermediate stages and 1 final stage\)\. It is important to note that the original Coconut implementation employs full fine\-tuning with full precision\. Consequently, the computational cost for the original training setup for Coconut is significantly higher, requiring over 50 hours on four NVIDIA A100 GPUs, while our method requires about 30 hours\.

The Qwen2\.5\-VL\-3B\-Instruct\-SFT model is Qwen2\.5\-VL\-3B\-Instruct\(Baiet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib8)\)finetuned on LLaVA\-CoT\-100k\(Xuet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib5)\)\. More specifically, it is finetuned on LLaVA\-CoT\-100k with the structured XML tags \(e\.g\., <SUMMARY\> and </SUMMARY\>\) included\. In the original paper, it is mentioned that training without the structured XML tags can cause degradation in performance, and it is recommended to finetune with the tags included\. We also provide another version, Qwen2\.5\-VL\-3B\-Instruct\-SFT \(w/o Structured Tags\), in Table[11](https://arxiv.org/html/2608.13570#A2.T11), which is finetuned on LLaVA\-CoT\-100k without the tags\. The results agree with the LLaVA\-CoT paper, showing a 5\.14% drop in average performance if the base model is finetuned without tags\.

For Heima\(Shenet al\.,[2025a](https://arxiv.org/html/2608.13570#bib.bib3)\), we adhere to the training configurations and curriculum provided in the original paper\. Since the original model is based on LLaVA\-CoT, which is essentially Llama\-3\.2\-11B\-Vision\-Instruct\(Meta,[2024a](https://arxiv.org/html/2608.13570#bib.bib42)\)fine\-tuned on LLaVA\-CoT\-100k, our reimplementation of Heima is trained on Qwen2\.5\-VL\-3B\-Instruct\-SFT\. In the progressive decoding phase, we gradually transform the reasoning steps into three corresponding thinking tokens, dedicating one epoch to the distillation of each\. Subsequently, we train for an additional epoch in the recovering phase using a slightly lower learning rate, which effectively repeats the final stage of the progressive encoding phase\.

### A\.2Inference Settings

We use greedy decoding for LLM inference and sampling\-based decoding for VLM inference\. For sampling, we set the temperature to 1e\-2, top\-ppto 1e\-3, and set a repetition penalty of 1\.0, which are essentially the default values provided for Qwen2\.5\-VL\-3B\-Instruct\(Baiet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib8)\)in VLMEvalKit\(Duanet al\.,[2024](https://arxiv.org/html/2608.13570#bib.bib32)\), so that we can ensure a fair comparison with baselines\.

Table 9:Training Hyperparameters for LLM Experiments\.N/A means that the corresponding hyperparameter is not applicable to the experiment\. For example, all LoRA hyperparameters are N/A for Coconut, because it performs full fine\-tuning without LoRA\.Table 10:Training Hyperparameters for VLM Experiments\.The SFT baseline and SELR variants are fine\-tuned on Qwen2\.5\-VL\-3B\-Instruct for 3 epochs with identical optimization settings to ensure a fair comparison\. Heima is trained in accordance with its proposed curriculum\.

## Appendix BAdditional Qualitative Results

In this section, we present additional qualitative examples to further demonstrate the effectiveness of our SELR method compared to baselines\. As shown in Figure[5](https://arxiv.org/html/2608.13570#A12.F5), our method gives concise responses, improving token efficiency significantly\. Furthermore, our method shows better performance over the base model in these examples\.

We also provide additional examples in Figure[6](https://arxiv.org/html/2608.13570#A12.F6), demonstrating that our method produces human\-readable and coherent reasoning that leads to the correct answers\.

![Refer to caption](https://arxiv.org/html/2608.13570v1/figures/unfaithful_decoding.png)Figure 4:An example of unfaithful CoT decoding\.The decoded CoT is correct, but the direct answer is wrong, indicating that the decoded CoT is not the true reasoning path for the model’s answer\.Table 11:Performance of SELR variants trained on Qwen2\.5\-VL\-3B\-Instruct\-SFT\.We report the accuracy and average token count per response across six benchmarks\. Qwen2\.5\-VL\-3B\-Instruct\-SFT and Qwen2\.5\-VL\-3B\-Instruct\-SFT \(w/o Structured Tags\) are Qwen2\.5\-VL\-3B\-Instruct finetuned on LLaVA\-CoT\-100k with and without structured tags, respectively\. The value enclosed by the parentheses in the Average column reflects the change from the Qwen2\.5\-VL\-3B\-Instruct\-SFT model\. SELR consistently outperforms both SFT baselines in average accuracy while reducing token usage by over 90%\.Table 12:Ablation study on latent length for LLM benchmarks\.We train the SELR \(Single\) method for 2 epochs across varying latent lengths\. The results confirm thatllatent=6l\_\{\\text\{latent\}\}=6achieves the best performance across the majority of datasets\.
## Appendix CSELR on Qwen2\.5\-VL\-3B\-Instruct\-SFT

For a fair comparison, it is necessary to isolate the performance gains attributed to our latent reasoning framework from those obtained simply by fine\-tuning on the target dataset\. In Table[11](https://arxiv.org/html/2608.13570#A2.T11), we present the results of SELR \(Single\), SELR \(Multi, Uniform\), and SELR \(Multi, Exponential\) when they are trained based on Qwen2\.5\-VL\-3B\-Instruct\-SFT \(w/o Structured Tags\)\. The training hyperparameters are identical to those listed in Table[10](https://arxiv.org/html/2608.13570#A1.T10)\. As we can see from Table[11](https://arxiv.org/html/2608.13570#A2.T11), our method still shows consistent gains in terms of performance and efficiency, with an approximately 6% gain over Qwen2\.5\-VL\-3B\-Instruct\-SFT \(w/o Structured Tags\) in performance, and over 90% gain in token efficiency\.

Although Qwen2\.5\-VL\-3B\-Instruct\-SFT \(w/o Structured Tags\) has a poorer performance relative to Qwen2\.5\-VL\-3B\-Instruct\-SFT, we train on this version nevertheless to avoid the shifting of generation pattern \(i\.e\., with or without the tags\)\. Despite this less performant base model, the models trained with SELR still surpass Qwen2\.5\-VL\-3B\-Instruct\-SFT, underscoring the efficacy of our method\.

## Appendix DLatent Length Ablation

For the LLM experiments, we use a latent length of 6 to maintain consistency with the Coconut baseline\. To validate this hyperparameter choice, we conduct an ablation study on the latent sequence length\. Note that this is not viable for the SELR method on VLMs, because Single Step Loss requires one\-to\-one correspondence between the latent thoughts and the reasoning steps\.

The results are shown in Table[12](https://arxiv.org/html/2608.13570#A2.T12)\. For these experiments, we stick to the training hyperparameters listed in the SELR \(Single\) configuration of Table[9](https://arxiv.org/html/2608.13570#A1.T9), except for the training duration, which is reduced to 2 epochs for efficiency\. The results indicate that a latent length of 6 provides the optimal balance; reducing the length to 2 or 4 limits the model’s reasoning capacity, while increasing it to 8 does not yield significant improvements\.

## Appendix EEvaluating Variable Latent Length Methods

We now clarify the evaluation protocol for our SELR method trained with variable latent length in Tables[7](https://arxiv.org/html/2608.13570#S4.T7)and[8](https://arxiv.org/html/2608.13570#S4.T8)\. As discussed in Section[3\.2\.1](https://arxiv.org/html/2608.13570#S3.SS2.SSS1), it is hard to train the model to predict the<eot\>token, so we have to cut off the latent reasoning generation with a fixed budget during evaluation\. Specifically, an inference budget ofbbimplies that the model generates exactlybblatent thought tokens following the<bot\>token\. We then terminate the latent reasoning generation process, append the<eot\>token, and proceed to generate the final answer\.

In Tables[7](https://arxiv.org/html/2608.13570#S4.T7)and[8](https://arxiv.org/html/2608.13570#S4.T8), we report the optimal accuracy achieved across all tested budgets for each benchmark\. This provides an upper\-bound performance estimate for the variable length method\. Notably, even when granted this “oracle” budget selection, the variable length approach generally underperforms compared to our proposed fixed latent length strategy\. For completeness, we provide the full evaluation results with different budgets in Tables[13](https://arxiv.org/html/2608.13570#A5.T13)and[14](https://arxiv.org/html/2608.13570#A5.T14)\.

Table 13:Evaluation results for the single\-stage SELR method trained with variable latent length\.We evaluate the model with fixed budgets ranging from 1 to 5 latent tokens\. The optimal result for each dataset is inbold\.Table 14:Evaluation results for the multi\-stage SELR method trained with variable latent length\.We evaluate the model with fixed budgets ranging from 1 to 5 latent tokens\. The optimal result for each dataset is inbold\.
## Appendix FPrompt for GPT\-4o Evaluation

As mentioned in Section[4\.1\.4](https://arxiv.org/html/2608.13570#S4.SS1.SSS4), we utilize GPT\-4o to evaluate the similarity between the decoded CoTs and the ground truth CoTs\. To ensure a fair comparison with Heima\(Shenet al\.,[2025a](https://arxiv.org/html/2608.13570#bib.bib3)\), we use the identical prompt that Heima uses, listed in the original paper’s appendix\. Here, we provide a copy of it for clarity in Algorithm[2](https://arxiv.org/html/2608.13570#alg2)\.

## Appendix GInference Latency

To confirm that SELR’s token reduction translates into real wall\-clock efficiency gains, we report per\-sample inference latency on MathVista in Table[15](https://arxiv.org/html/2608.13570#A7.T15)\. SELR \(Single\) achieves the highest accuracy \(65\.10%\) and the lowest latency \(0\.81s per sample\), yielding an8\.7×\\timesspeedupover the SFT baseline\.

Table 15:Per\-sample inference latency on MathVista\. SELR achieves the best accuracy with the lowest latency, demonstrating that token reduction directly translates into wall\-clock speedup\.
## Appendix HQwen2\.5\-VL\-7B\-Instruct Experiments

To validate generalization across model scales, we trained SELR on Qwen2\.5\-VL\-7B\-Instruct\(Baiet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib8)\)\. Table[16](https://arxiv.org/html/2608.13570#A8.T16)presents the results\. The same pattern holds at the 7B scale: SELR improves average accuracy over the original model \(\+0\.11\), SFT degrades accuracy despite generating far more tokens, and SELR achieves over 80% token reduction\. This confirms that SELR’s benefits generalize across model scales\.

Table 16:Results on Qwen2\.5\-VL\-7B\-Instruct\.Each cell shows Accuracy \(Avg\. \#Tokens\)\. SELR improves accuracy over the original model while achieving over 80% token reduction, confirming generalization across model scales\.
## Appendix IControlled\-Budget LLM Comparison

A key question is whether the performance gap between SELR and CoT\-SFT on LLM benchmarks reflects a difference in reasoning mechanism or simply in reasoning budget\. CoT\-SFT generates over 150 reasoning tokens per question on GSM8k, while SELR uses only 6 latent tokens\. To disentangle this, we constrain CoT\-SFT to a comparable budget by truncating its generation to 6 tokens plus the answer, and also report its direct\-answer \(zero reasoning token\) performance\. Results are shown in Table[17](https://arxiv.org/html/2608.13570#A9.T17)\.

Table 17:Controlled\-budget LLM comparison\.When CoT\-SFT is constrained to the same 6\-token budget as SELR, its performance collapses\. SELR achieves 42\.46% on GSM8k vs\. CoT\-SFT’s 10\.31% under the same budget, demonstrating far more efficient reasoning encoding\.The results clearly demonstrate that CoT\-SFT’s advantage comes from its much longer reasoning trace, not from a superior reasoning mechanism\. When constrained to the same 6\-token budget, CoT\-SFT achieves only 10\.31% on GSM8k, compared to SELR’s 42\.46%—a 4×\\timesadvantage for SELR\. This confirms that SELR encodes reasoning information far more efficiently than textual CoT on a per\-token basis\.

## Appendix JGeneralization to VisualWebInstruct\-Verified

To verify that SELR’s effectiveness is not limited to the LLaVA\-CoT\-100k dataset or its specific structured format, we train both SELR and CoT\-SFT on VisualWebInstruct\-Verified\(Jiaet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib45)\), a more recent VLM reasoning dataset with a completely different source and format\. Results on Qwen2\.5\-VL\-3B\-Instruct are shown in Table[18](https://arxiv.org/html/2608.13570#A10.T18)\.

Table 18:Results on VisualWebInstruct\-Verified\.SELR outperforms CoT\-SFT on 5 of 6 benchmarks, confirming that SELR’s mechanism is CoT\-format agnostic and generalizes to unstructured reasoning datasets\.This demonstrates that SELR works with both structured and unstructured reasoning annotations, and the improvements are not an artifact of a particular dataset structure\.

## Appendix KDiscussion on Reasoning Step Ordering

For the VLM setting, SELR uses a fixed ordering of reasoning steps: summary→\\rightarrowcaption→\\rightarrowreasoning\. This ordering is inherited from the LLaVA\-CoT\-100k dataset\(Xuet al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib5)\)and is also adopted by LLaVA\-CoT and Heima\(Shenet al\.,[2025a](https://arxiv.org/html/2608.13570#bib.bib3)\)\.

This fixed ordering offers several advantages:

1. 1\.Natural reasoning flow\.The ordering reflects a natural reasoning process for visual question\-answering: first understand the question \(summary\), then describe relevant visual content \(caption\), and finally reason toward the answer\. Each step logically builds upon the previous one\.
2. 2\.Positional specialization\.A fixed order allows each latent position to specialize \(e\.g\., the first latent always encodes summaries\), simplifying learning\. Randomizing the order would require the model to additionally learn to identify the type of information in each position, increasing the learning difficulty\.
3. 3\.Consistency with prior work\.The same ordering is used by all prior methods trained on LLaVA\-CoT\-100k, ensuring fair comparison\.

We note that this ordering question applies*only*to the VLM setting; in the LLM setting, SELR uses unstructured CoT steps that follow the natural reasoning sequence without imposing any predefined structure\.

## Appendix LLimitations

Despite its positive results, our method still admittedly has some limitations\.

First, faithful reasoning remains a recognized open challenge even in the text\-CoT literature\. Prior work has shown that text\-based CoT models exhibit systematic unfaithfulness: prompt\-bias injection reveals that models virtually never verbalize the biases that flip their answers\(Turpinet al\.,[2023](https://arxiv.org/html/2608.13570#bib.bib41)\); behavioral perturbations of the CoT show wide variation across tasks\(Lanhamet al\.,[2023](https://arxiv.org/html/2608.13570#bib.bib43)\); and even state\-of\-the\-art reasoning models acknowledge influential hints less than 40% of the time\(Bakeret al\.,[2025](https://arxiv.org/html/2608.13570#bib.bib44)\)\. These measurement techniques all rely on prompt\-level interventions or token\-level perturbations of discrete CoT, with no direct counterpart in continuous latent space\. SELR’s contribution is to make latent reasoning*decodable*in the first place—a prerequisite for any faithfulness analysis on latents—while prior latent methods \(e\.g\., Coconut, CoLaR, CODI\) cannot decode their representations at all\.

We note that the term “Self\-Explainable” should be interpreted as self\-decodable latents with meaningful faithfulness signals, rather than a guarantee of full reasoning transparency\. As shown in Figure[4](https://arxiv.org/html/2608.13570#A2.F4), the decoded CoT can occasionally produce correct reasoning while the model fails to give the correct answer directly, indicating potential misalignment between the decoded text and the model’s true internal process\. Nevertheless, SELR opens the door to adapting text\-CoT faithfulness techniques to latent reasoning for the first time, and our consistency metric represents a concrete first step toward this goal\.

Second, our methods ultimately relies on a fixed latent reasoning length, which could be suboptimal, and it is more intuitive for the model to be able to decide when to stop the latent reasoning on its own\. However, the<eot\>token is hard to predict with the LM head, as mentioned in Section[3\.2\.1](https://arxiv.org/html/2608.13570#S3.SS2.SSS1)\.

We expect that developing a more advanced and learnable stopping mechanism can help the model determine the end of the latent space thinking mode, but this is beyond our current research scope\.

For future work, we hope to refine our method to provide a latent space reasoning framework that provides better explainability and flexibility, while retaining efficiency and efficacy\.

![Refer to caption](https://arxiv.org/html/2608.13570v1/x4.png)Figure 5:More qualitative examples of LLaVA\-CoT, Heima, Qwen2\.5\-VL\-3B\-Instruct, and our SELR on multimodal tasks\.![Refer to caption](https://arxiv.org/html/2608.13570v1/figures/decoding_examples.png)Figure 6:More qualitative examples demonstrating SELR’s capability to decode latent thoughts into human\-readable summaries, captions, and reasoning steps\.Algorithm 2GPT\-4o Prompt for CoT Decoding Evaluation \(from Heima\(Shenet al\.,[2025a](https://arxiv.org/html/2608.13570#bib.bib3)\)\)Input:ImageI, questionQ, decoded CoTCoT^\\hat\{\\textbf\{CoT\}\}, ground truthCoT, type of CoT stageT∈\[caption,summary,reasoning\]\\textbf\{T\}\\in\[\\textbf\{caption\},\\textbf\{summary\},\\textbf\{reasoning\}\]\.

Output:An integer representing the rank of similarity betweenCoT^\\hat\{\\text\{CoT\}\}andCoTin \[1, 5\]\.

User:When responding to questions about an image, a deep analysis is crucial for providing accurate answers\. The analysis of an image\-question pair could be one of the following components:

Summary– A brief restatement or paraphrasing of the question\.

Caption– A description or summary of the content of the image\.

Reasoning– A logical explanation of how the answer is derived from the image and the question\.

You will be provided with one of them along with the ground truth\. Your task is to evaluate whether the analysis closely aligns with the ground truth according to the given image and question pair\.

User:In this conversation, you will be given a generatedTand its ground truth\. TheTis:CoT^\\hat\{\\textbf\{CoT\}\}\. The ground truth is:CoT\.

User:Following is the given image:I The corresponding question is:Q\.

User:Please rank the similarity with an integer between 1 and 5, where the larger number means the generatedTis closer to the ground truth\. Please rate the similarity on a scale from 1 to 5, where:

1: Completely unrelated\.The generatedTand ground truth discuss entirely different themes, and there is no overlap in content, or subject matter\. Example: Ground Truth: …; GeneratedT: …

2: Minimally related\.The generatedTand ground truth are tangentially connected\. Only a minimum fraction of the theme or content in the ground truth is mentioned in the generatedT\. Example: Ground Truth: …; GeneratedT: …

3: Somewhat related but with notable discrepancies\.The generatedTand ground truth share key elements in theme or content but exhibit clear differences in focus, description, or details\. While the overall themes or settings may overlap \(e\.g\., animals, fences, grassy area\), the generatedTintroduces significant factual errors or omits important details\. Example: Ground Truth: …; GeneratedT: …

4: Closely related with small differences\.The generatedTand ground truth align on the main theme and share most of the key details\. However, there are minor differences in phrasing, specific details, or focus\. Example: Ground Truth: …; GeneratedT: …

5: Nearly identical\.The generatedTand ground truth are highly similar, sharing nearly all content, details, and key descriptions, with only minor or negligible phrasing differences\. Example: Ground Truth: …; GeneratedT: …

The output should be in a JSON format: \{“T”:\(Rank\),“reason”:…\}\\\{\\text\{\`\`\{T\}''\}:\(\\text\{Rank\}\),\\text\{\`\`reason''\}:\.\.\.\\\} \(Rank\) is the integer of the similarity rank\. “reason” stores the reason for ranking a givenTand ground truth\.