Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

# Latent Time Modelling in Diffusion Language Models
Source: [https://arxiv.org/html/2607.01774](https://arxiv.org/html/2607.01774)
## Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

Maximo Rulli1Thomas Fontanari\*,1Simone Petruzzi\*,1Federico Alvetreti1 Giorgio Strano1Donato Crisostomi1Giorgos Nikolaou2Tommaso Mencattini2 Andrea Santilli3Emanuele Rodolà1Simone Scardapane1Alessio Devoto3

###### Abstract

Diffusion Language Models \(DLMs\) have recently emerged as a promising alternative to autoregressive models\. Unlike standard diffusion\-based approaches, DLMs are not explicitly conditioned on a timestep, raising a natural question: do these models internally represent denoising progress, and how is such information used downstream? In this work, we show that DLMs do in fact encode a latent representation related to the diffusion timestep within their residual streams\. We find that this signal can be reliably extracted using probes across layers, indicating that denoising progress is decodable from internal activations\. We further demonstrate that steering the model along a low\-dimensional subspace associated with the inferred timestep allows us to systematically modulate its notion of denoising progress, leading to predictable changes in model confidence and entropy\. Finally, we analyse the geometry of the identified representation, showing that it exhibits structured and interpretable properties in activation space, and shedding light on how such a signal is processed by these models\.

Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

††\*Equal contribution\.1Sapienza University of Rome2EPFL3Independent researcher
Correspondence:rulli\.2154435@studenti\.uniroma1\.it\.## 1Introduction

Large Language Models \(LLMs\)\(Anthropic,[2025](https://arxiv.org/html/2607.01774#bib.bib33); Meta AI,[2025](https://arxiv.org/html/2607.01774#bib.bib29); Achiamet al\.,[2023](https://arxiv.org/html/2607.01774#bib.bib30); DeepSeek\-AI,[2024](https://arxiv.org/html/2607.01774#bib.bib34); Gemma Team,[2025](https://arxiv.org/html/2607.01774#bib.bib32)\)are driving a paradigm shift across various scientific and social domains\. Understanding the capabilities of these models remains an active area of research\. Many works aim to characterise the manifolds and subspaces spanned by the features of these models’ activations\(Saglamet al\.,[2026](https://arxiv.org/html/2607.01774#bib.bib13); Tibliaset al\.,[2026](https://arxiv.org/html/2607.01774#bib.bib8); Joshiet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib7); Shaiet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib9); Bhallaet al\.,[2026](https://arxiv.org/html/2607.01774#bib.bib14)\), helping explain how they represent and transform concepts across their computations\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x1.png)Figure 1:3D projection of latent denoising step modelling for LLaDA\. LLaDA represents itsτ\\tausubspace as a low\-dimensional manifold\-like curve, progressively modelling the denoising progress from all \[mask\] \(red\) to no remaining \[mask\] \(purple\)\.Recently, large\-scale Masked Diffusion Language Models\(Nieet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib38); Zhuet al\.,[2026](https://arxiv.org/html/2607.01774#bib.bib40); Yeet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib39)\)have emerged as an alternative paradigm for text generation\. Rather than generating strictly left\-to\-right, they progressively denoise \[mask\] tokens in a BERT\-style manner\(Devlinet al\.,[2019](https://arxiv.org/html/2607.01774#bib.bib36)\); throughout this work, we call this masked\-token family DLMs\. Despite the growing interest in DLMs, most existing research has primarily focused on improving their efficiency and generation quality\(Wuet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib54); Arriolaet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib49); Duet al\.,[2026](https://arxiv.org/html/2607.01774#bib.bib41); Zekriet al\.,[2026](https://arxiv.org/html/2607.01774#bib.bib42); Yuet al\.,[2026](https://arxiv.org/html/2607.01774#bib.bib52)\)\. In contrast, their internal mechanisms and representations remain largely unexplored\. More recently, a small but growing body of work has begun investigating the interpretability of these models\. For instance,Wanget al\.\([2026](https://arxiv.org/html/2607.01774#bib.bib4)\)train Sparse Autoencoders\(Cunninghamet al\.,[2023](https://arxiv.org/html/2607.01774#bib.bib2)\)on LLaDA and Dream to enable concept steering at inference time,Rulliet al\.\([2025](https://arxiv.org/html/2607.01774#bib.bib19)\)analyse how attention sinks move across denoising steps and how robust DLMs are to pruning them,Wenet al\.\([2026](https://arxiv.org/html/2607.01774#bib.bib5)\)demonstrate how \[mask\] tokens can be manipulated to induce harmful or deceptive generations, andPiskorzet al\.\([2025](https://arxiv.org/html/2607.01774#bib.bib6)\)show that DLM performance is strongly influenced by the number of \[mask\] tokens present at each denoising step\. While these works provide valuable insights into the behaviour of DLMs, the internal dynamics underlying the denoising process itself remain poorly understood\. In particular, it remains unclear whether DLMs internally represent denoising progress during generation\. Previous theoretical work suggests that the strong performance of absorbing\-state DLMs may be largely attributed to the special role of the \[mask\] token: the reverse process only needs to denoise masked positions, and the corresponding score can be expressed as a clean\-data conditional distribution up to an analytic time\-dependent factor\(Ouet al\.,[2024](https://arxiv.org/html/2607.01774#bib.bib1)\)\. Complementarily, masked diffusion objectives can be reformulated in terms of the number of masked tokens, making continuous denoising time closely related to a relaxation of the mask ratio and connecting these models to time\-agnostic masked models\(Zhenget al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib47)\)\. This makes the fraction of masked tokens a natural observable proxy for denoising progress, but leaves open whether and how such a signal is internally represented by the model\.

In this work, we investigate this through the following three research questions:

RQ1Do Diffusion Language Models internally represent a signal related to the denoising step?

RQ2Is the identified signal important to the generation process? How does modifying it affect the models’ downstream computations?

RQ3What characteristics does the signal exhibit? Is there a unified model\-level representation?

We address these questions in three stages: first by probing for denoising\-time information, then by causally steering the recovered signal, and finally by characterising its geometry across layers\. We conduct our analysis on LLaDA\-1\.5\(Zhuet al\.,[2026](https://arxiv.org/html/2607.01774#bib.bib40)\)\(which we refer to as LLaDA\) and Dream\(Yeet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib39)\), two representative large\-scale masked diffusion language models\.

## 2Background

DLMs are trained to recover a clean sequencex0x\_\{0\}from a corrupted version of it\(Liet al\.,[2022](https://arxiv.org/html/2607.01774#bib.bib62); Sahooet al\.,[2024](https://arxiv.org/html/2607.01774#bib.bib43); Ouet al\.,[2024](https://arxiv.org/html/2607.01774#bib.bib1); Shiet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib44)\)\. In absorbing\-state models, the corrupted sequencexsx\_\{s\}is obtained fromx0x\_\{0\}by replacing a subset of its positions with a special\[mask\]\[\\textsc\{mask\}\]token\. We define:

ℳ​\(xs\)=\{j:xsj=\[mask\]\}\\mathcal\{M\}\(x\_\{s\}\)=\\\{j:x\_\{s\}^\{j\}=\[\\textsc\{mask\}\]\\,\\\}to be the set of masked positions\. The scalars∈\[0,1\]s\\in\[0,1\]controls the corruption level:s=0s=0recovers the clean sequence, whiles=1s=1masks every position\. During training, a noise levels∼𝒰​\(0,1\]s\\sim\\mathcal\{U\}\(0,1\]is sampled and each position is corrupted independently according to the forward process

xsj∼Cat​\(⋅;p=\(1−s\)​x0j\+s​\[mask\]\)x\_\{s\}^\{j\}\\sim\\mathrm\{Cat\}\\big\(\\cdot;p=\(1\-s\)\\,x\_\{0\}^\{j\}\+s\\,\[\\textsc\{mask\}\]\\big\)so that positionjjkeeps its original token with probability1−s1\-sand is replaced by\[mask\]\[\\textsc\{mask\}\]with probabilityss\. The DLMpθp\_\{\\theta\}takes the corrupted sequencexsx\_\{s\}as input and predicts, at every positionjj, a distributionpθ​\(x0j∣xs\)p\_\{\\theta\}\(x\_\{0\}^\{j\}\\mid x\_\{s\}\)over the original vocabulary𝒱\\mathcal\{V\}\. The training minimises a cross\-entropy loss applied only at the masked positions, reweighted by1/s1/s:

−𝔼x0,s,xs​\[1s​∑j∈ℳ​\(xs\)log⁡pθ​\(x0j∣xs\)\]\.\-\\mathbb\{E\}\_\{x\_\{0\},\\,s,\\,x\_\{s\}\}\\left\[\\frac\{1\}\{s\}\\sum\_\{j\\in\\mathcal\{M\}\(x\_\{s\}\)\}\\log p\_\{\\theta\}\(x^\{j\}\_\{0\}\\mid x\_\{s\}\)\\right\]\.\(1\)Inference proceeds through a sequence ofTTdenoising steps\. Generation is initialised at stept=0t=0withLL\[mask\] tokens appended to the prompt, and ends att=Tt=Twith all tokens unmasked\. At each steptt, the model operates on the current partially\-masked sequencextx\_\{t\}to obtainpθ\(⋅∣xt\)p\_\{\\theta\}\(\\cdot\\mid x\_\{t\}\)at every masked position, and a subset ofℳ​\(xt\)\\mathcal\{M\}\(x\_\{t\}\)is selected to be unmasked by sampling according to an unmasking policy\. This policy determines how many tokens are revealed per step\. At inference time we measure the fraction of unmasked tokens as

τt:=1−\|ℳ​\(xt\)\|L\.\\tau\_\{t\}:=1\-\\frac\{\|\\mathcal\{M\}\(x\_\{t\}\)\|\}\{L\}\.\(2\)In[Appendix˜A](https://arxiv.org/html/2607.01774#A1), we provide more details of howτt\\tau\_\{t\}is equivalent, in expectation, to the complement of the continuous diffusion\-time variabless\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x2.png)Figure 2:MLP probes recover theτ\\tausignal\. TheR2\\rm\{R\}^\{2\}coefficient \(higher is better\) degrades as we probe deeper into the model, to the point where the MSE almost reduces to half the variance ofτ\\tau\. Moreover, both \[mask\] and non\-\[mask\] tokens seem to carry this information, attaining similarly high coefficients\.We therefore interpretτt\\tau\_\{t\}as an empirical measure of denoising progress and use it as a proxy for the notion of denoising time in masked diffusion language models\.

## 3Recoveringτ\\tau

In this section we address RQ[1](https://arxiv.org/html/2607.01774#S1)by attempting to recover theτ\\tausignal from the model’s hidden states\. To test whetherτ\\taucan be recovered from the residual stream of the model, we train MLP probes to predict the currentτt\\tau\_\{t\}of a given sequence by using a residual\-stream hidden state as input\. Specifically, we fit an MLP \(ϕl​\(⋅\):ℝd→\(0,1\)\\phi\_\{l\}\(\\cdot\):\\mathbb\{R\}^\{d\}\\rightarrow\(0,1\)\) for each layerllin the model, whereddis the dimensionality of the hidden state \(see[Appendix˜B](https://arxiv.org/html/2607.01774#A2)for architectural and training details\)\. Each MLP is trained to minimise the regression loss

ℒMLP=𝔼t,n,j​\[\(τt−ϕl​\(ht,l,nj\)\)2\]\\mathcal\{L\}\_\{\\text\{MLP\}\}=\\mathbb\{E\}\_\{t,n,j\}\\left\[\\left\(\\tau\_\{t\}\-\\phi\_\{l\}\(h^\{j\}\_\{t,l,n\}\)\\right\)^\{2\}\\right\]wherennis the example index,jjis the token index in the example sequence, andht,l,njh^\{j\}\_\{t,l,n\}denotes the hidden state\. Throughout this work, we use𝔼r\\mathbb\{E\}\_\{r\}to denote the empirical expectation over the corresponding quantityrr\. We fit separate probes for every layer in both LLaDA and Dream\. Additionally, we vary the subset of tokens used for training, considering only \[mask\] tokens, non\-\[mask\] tokens, or all tokens\. This allows us to investigate whether information related toτ\\tauis preferentially encoded in specific types of tokens\.[Figure˜2](https://arxiv.org/html/2607.01774#S2.F2)shows theR2\\rm\{R\}^\{2\}obtained for each layer of LLaDA\. The probes maintain anR2\>0\.5\\mathrm\{R\}^\{2\}\>0\.5across all layers of the model, indicating that information aboutτ\\tauis consistently represented throughout the network depth\. Performance slightly degrades in the earliest and latest layers, suggesting that the representation becomes less accessible at the boundaries of the computation\. We observe similar trends for Dream \(see[Appendix˜C](https://arxiv.org/html/2607.01774#A3)\)\.

We also find that activations corresponding to \[mask\] tokens yield marginally better predictions than non\-\[mask\] tokens, though both token types support accurate recovery\. This suggests thatτ\\tauinformation is not exclusively localised to masked positions, but is instead distributed across the residual stream\.

Importantly, each MLP probe operates on a single hidden state independently, without access to any sequence\-level context or neighbouring activations, hinting that individual token representations themselves encode substantial information aboutτ\\tau\. Thus, the high performance of the probes indicates that individual token representationscarry a latent representationof the sequence\-level statisticτ\\tau\.

As a result, we answer affirmatively to RQ[1](https://arxiv.org/html/2607.01774#S1)and conclude that DLMs do internally model a signal related to the current denoising step\. In the next section, we show a natural construction of a signal related toτ\\tauand use it to steer the model at inference time\.

✓ Takeaway for RQ1*Diffusion Language Models do internally encode a denoising\-step\-related signal, and we can accurately recover it\.*

## 4Assessing the Signal’s Importance

Having established that DLMs encode information about the denoising process in their residual stream, we now investigate whether this signal can be explicitly extracted and used to steer the model\. In particular, we aim to identify directions in activation space associated with the denoising\-time statisticτ\\tau, and study whether steering along these directions changes the model’s behaviour during inference\.

### 4\.1Approximatingτ\\tau

To characterise and approximate the internal representation ofτ\\tau, we turn to mean activation vectors\(Gurneeet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib3); Wanget al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib25); Panicksseryet al\.,[2024](https://arxiv.org/html/2607.01774#bib.bib24)\)\. The key intuition behind this approach is that averaging activations over many samples suppresses instance\-specific information while preserving systematic signals shared across examples, such as the one associated withτ\\tau\.

FollowingWanget al\.\([2025](https://arxiv.org/html/2607.01774#bib.bib25)\), we compute mean activation vectors over all hidden states in the response window, grouping them according to their denoising step\. More formally, for each denoising stepttand layerll, we define:

μt,l:=𝔼n,j​\[ht,l,nj\],\\mu\_\{t,l\}:=\\mathbb\{E\}\_\{n,j\}\\left\[h^\{j\}\_\{t,l,n\}\\right\],\(3\)
We discretise the denoising process into100100bins, yielding100100mean vectors per layer\. This results in32003200mean vectors for LLaDA and28002800for Dream\.

To verify whether the computed mean vectors capture the same signal identified by the probes from[Section˜3](https://arxiv.org/html/2607.01774#S3), we evaluate the correlation between the probe predictionsϕl​\(μt,l\)\\phi\_\{l\}\(\\mu\_\{t,l\}\)and the normalised timestep indext/100t/100\. If both methods capture a similar latent direction, then the probe should behave monotonically as the mean vector’s index increases\. For LLaDA, the correlations reach0\.9760\.976Pearson and0\.9800\.980Spearman, while for Dream they reach0\.9620\.962and0\.9740\.974, respectively\. Moreover, as shown in[Figure˜3](https://arxiv.org/html/2607.01774#S4.F3), the probe predictions align almost perfectly with the denoising bins encoded by each mean vector\. These results suggest that theτ\\tauinformation identified by the probes is not merely recoverable at the token level, but also emerges as a robust global structure in the residual stream after averaging across examples\. In other words, theτ\\tausignal appears to correspond to a coherent latent direction that is consistently represented across tokens and sequences, making it a natural target for the steering interventions we introduce next\. Having verified that the mean vectors track the same signal recovered by the probes, we next test whether this structure is causally relevant by intervening on it during inference\.

### 4\.2Steering

We use the computed activations to steer the model by swapping the mean vectors for the current step with those of the targett^\\hat\{t\}\(wheret^\\hat\{t\}is the discretised target bin whose denoising progress we steer the model towards\):

h~t,lj:=ht,lj−μt,l\+μt^,l\.\\tilde\{h\}^\{j\}\_\{t,l\}:=h^\{j\}\_\{t,l\}\-\\mu\_\{t,l\}\+\\mu\_\{\\hat\{t\},l\}\.\(4\)We note that, since the perturbation is defined as a differenceΔlt→t^:=μt^,l−μt,l\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}:=\\mu\_\{\\hat\{t\},l\}\-\\mu\_\{t,l\}, the shared componentμ¯l:=𝔼t​\[μt,l\]\\bar\{\\mu\}\_\{l\}:=\\mathbb\{E\}\_\{t\}\[\\mu\_\{t,l\}\]cancels out\. This removes the part of the representation that is independent of denoising progress and intrinsic to the hidden states of layerll\. Based on the results shown in[Figure˜2](https://arxiv.org/html/2607.01774#S2.F2), we apply steering across all tokens to ensure that noτ\\tau\-related information is inadvertently preserved when perturbing the hidden states\. As a control condition, we additionally introduce a perturbation obtained by randomly sampling directions from the empirical covariance matrix\. Specifically, we construct the empirical covariance matrixCt,lC\_\{t,l\}\(formed from the activations at stepttand layerll\), and we sample a random vector that is applied across all hidden states at that step:

h~t,lj:=ht,lj\+at→t^,a∼𝒩​\(𝟎,Ct,l\)\\tilde\{h\}^\{j\}\_\{t,l\}:=h^\{j\}\_\{t,l\}\+a^\{t\\to\\hat\{t\}\},\\quad a\\sim\\mathcal\{N\}\(\\mathbf\{0\},C\_\{t,l\}\)\(5\)We denote byat→t^a^\{t\\to\\hat\{t\}\}the normalised version of the sampled vectoraa, rescaled such that‖at→t^‖=‖μt^,l−μt,l‖\\left\\\|a^\{t\\to\\hat\{t\}\}\\right\\\|=\\left\\\|\\mu\_\{\\hat\{t\},l\}\-\\mu\_\{t,l\}\\right\\\|\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x3.png)
![Refer to caption](https://arxiv.org/html/2607.01774v1/x4.png)

Figure 3:ϕl​\(μt,l\)\\phi\_\{l\}\(\\mu\_\{t,l\}\)andμt,l\\mu\_\{t,l\}are highly correlated\. For each of the foundμt,l\\mu\_\{t,l\}we computeϕl​\(μt,l\)\\phi\_\{l\}\(\\mu\_\{t,l\}\)and plot it against the correspondingτ=t/100\\tau=t/100\. We observe high correlations for both LLaDA \(left\) and Dream \(right\)\.
### 4\.3Experimental results

We now investigate whether steering the internal representation ofτ\\tauproduces consistent downstream changes in the model predictions\. Intuitively, if the characterised signal truly relates to a model’s mechanism for representingτ\\tau, then shifting this representation towards larger or smallerτ\\tauvalues should alter the model’s confidence, entropy, and token distributions accordingly\. To quantify these effects, we compare the clean token distributions

pt,nj:=pθ​\(x0,nj∣xt,n\)p^\{j\}\_\{t,n\}:=p\_\{\\theta\}\(x^\{j\}\_\{0,n\}\\mid x\_\{t,n\}\)against the steered ones

p~t,nj:=pθ​\(x0,nj∣x~t,n\),\\tilde\{p\}^\{j\}\_\{t,n\}:=p\_\{\\theta\}\(x^\{j\}\_\{0,n\}\\mid\\tilde\{x\}\_\{t,n\}\),wheren∈\[N\]n\\in\[N\]denotes the sequence index\.

We evaluate three complementary quantities: the variation in entropy, the variation in confidence, and the KL\-divergence between clean and steered distributions:

Δ​S¯t\\displaystyle\\Delta\\bar\{S\}\_\{t\}:=𝔼n,j​\[H​\(p~t,nj\)−H​\(pt,nj\)\],\\displaystyle=\\mathbb\{E\}\_\{n,j\}\\left\[\\mathrm\{H\}\(\\tilde\{p\}^\{j\}\_\{t,n\}\)\-\\mathrm\{H\}\(p^\{j\}\_\{t,n\}\)\\right\],\(6\)Δ​c¯t\\displaystyle\\Delta\\bar\{c\}\_\{t\}:=𝔼n,j​\[max⁡\(p~t,nj\)−max⁡\(pt,nj\)\],\\displaystyle=\\mathbb\{E\}\_\{n,j\}\\left\[\\max\(\\tilde\{p\}^\{j\}\_\{t,n\}\)\-\\max\(p^\{j\}\_\{t,n\}\)\\right\],KL¯t\\displaystyle\\overline\{\\mathrm\{KL\}\}\_\{t\}:=𝔼n,j​\[DKL​\(pt,nj∥p~t,nj\)\]\.\\displaystyle=\\mathbb\{E\}\_\{n,j\}\\left\[D\_\{\\mathrm\{KL\}\}\\left\(p^\{j\}\_\{t,n\}\\,\\\|\\,\\tilde\{p\}^\{j\}\_\{t,n\}\\right\)\\right\]\.
We apply the steering vectors at different transformer layers and observe that the strongest downstream effects consistently emerge in the final layers of both LLaDA and Dream\.[Figure˜4](https://arxiv.org/html/2607.01774#S4.F4)presents the results obtained when steering layer 29 of LLaDA\. Similar trends are observed in Dream when steering layer 25\. In[Appendix˜C](https://arxiv.org/html/2607.01774#A3), we show results across various layers for both LLaDA and Dream\.

Remarkably, steering with[Equation˜4](https://arxiv.org/html/2607.01774#S4.E4)produces a behaviour that closely matches what would be expected from directly modifying the model’s internal notion ofτ\\tau\. When steering towards larger values oft^\\hat\{t\}relative to the current steptt, i\.e\.t^−t\>0\\hat\{t\}\-t\>0, the model becomes more confident and its entropy decreases\. Conversely, whent^−t<0\\hat\{t\}\-t<0, confidence decreases and entropy increases\. At the same time, the KL divergence grows approximately proportionally to the distance\|t^−t\|\|\\hat\{t\}\-t\|, indicating progressively larger deviations from the original distribution as the steering target moves further away from the current denoising step\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x5.png)Figure 4:Steering on layer 29 downstream effects\. We steer the activations of layer 29 using[Equation˜4](https://arxiv.org/html/2607.01774#S4.E4)targeting differentt^\\hat\{t\}values \(blue\), and compare the resulting entropy, confidence, and KL\-divergence variations against norm\-matched[Equation˜5](https://arxiv.org/html/2607.01774#S4.E5)perturbations \(red\)\. Steering along the foundτ\\taudirections produces systematic and interpretable effects\.To verify that these effects are specific to the identified signal and are not a phenomenon of the denoising mechanics of these models, we compare them against[Equation˜5](https://arxiv.org/html/2607.01774#S4.E5)with matched norm\. Unlike theτ\\tau\-based perturbations, random perturbations do not induce coherent trends in entropy or confidence across denoising steps\. Moreover, their downstream impact is substantially weaker\. For a comparable perturbation norm, the KL divergence induced by random perturbations is approximately half of that produced byτ\\tausteering, suggesting that the learned directions correspond to a particularly sensitive and semantically meaningful subspace in the residual stream\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x6.png)Figure 5:Average probe drift across layers and steps after steering in LLaDA\.The model progressively compensates for the perturbation, reducing the discrepancy between clean and steered representations ofτ\\tau\.Contrary to the behaviour observed in the final layers, steering in shallow layers produces effects that are largely indistinguishable from the random baseline\. Representative results for shallow\- and mid\-layer steering experiments, are reported in[Appendix˜C](https://arxiv.org/html/2607.01774#A3)\. Combined with the probe analysis in[Figure˜2](https://arxiv.org/html/2607.01774#S2.F2), this suggests thatτ\\tauinformation is not merely propagated unchanged across layers: early perturbations can be attenuated by later computation as the model progressively compensates for them\. We test this by injecting a steering perturbation at layer 6 and tracking the probe\-predictedτ\\tauacross the remaining layers\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x7.png)\(a\)The PCA distribution of the mean vectors\.
![Refer to caption](https://arxiv.org/html/2607.01774v1/x8.png)\(b\)Dream’s layer 25 2D projection

Figure 6:PCA distribution and sampled projection on the top\-2 principal components of the mean vectors\.Left: PCA distribution of the mean vectors for LLaDA; notably, across layers, most of the variance of the identified set of mean vectors can be explained by fewer than three dimensions\. Right: sampled projections for Dream\. We use the mean vectors obtained from layer 25 of Dream\. We observe a parabola\-like geometry\.
### What happens to the signal at early layers?

To explain the null result obtained when steering early layers, we investigate this phenomenon more deeply and measure how the probe predictions evolve after steering\. If the model does indeed correct an early perturbation as computation proceeds, then theτ\\tauvalue read by the probes should drift back towards its clean value at deeper layers; conversely, a perturbation that survives unchanged would keep the steered and clean predictions far apart throughout\. We therefore track the average deviation between the probe outputs of clean and steered activations across layers

δ¯l:=𝔼t​\[\|𝔼j​\[ϕl​\(h~t,lj\)−ϕl​\(ht,lj\)\]\|\]\.\\bar\{\\delta\}\_\{l\}:=\\mathbb\{E\}\_\{t\}\\left\[\\left\|\\mathbb\{E\}\_\{j\}\\left\[\\phi\_\{l\}\(\\tilde\{h\}^\{j\}\_\{t,l\}\)\-\\phi\_\{l\}\(h^\{j\}\_\{t,l\}\)\\right\]\\right\|\\right\]\.Starting from a perturbation injected at layer 6, we evaluate how different steering targetst^\\hat\{t\}propagate through the networks’ depth\.

As shown in[Figure˜5](https://arxiv.org/html/2607.01774#S4.F5), the model is able to correct most of the perturbation introduced in early layers, such that the probe predictions under the perturbation become nearly indistinguishable from the clean ones in the final blocks\.

However, steering toward extreme values such ast^=100\\hat\{t\}=100remains harder to compensate for, leaving a persistent discrepancy even in deeper layers\. This observation also helps explain the sharp peaks previously observed for shallow\-layer steering \(see[Appendix˜C](https://arxiv.org/html/2607.01774#A3)\): at those denoising steps, the model is temporarily unable to fully recover from the injected perturbation before producing its prediction \(additional analysis is provided in[Appendix˜E](https://arxiv.org/html/2607.01774#A5)\)\. Having established that this signal affects the models’ output distributions in a predictable way, we now move on to characterising its properties and look for the emergence of model\-level patterns in how the signal is organised\.

✓ Takeaway for RQ2*The found signal has direct causal implications for modelling dynamics, predictably affecting downstream entropy, confidence and KL divergence\. The models also internally recompute it several times across their depth, allowing for correction\.*

## 5Characterising the Signal

Having established how the models react to changes in the signal, we now examine how this signal is represented internally and whether consistent structural patterns emerge\.

### 5\.1μ\\muvectors’ low\-dimensional structure

We begin by analysing the structural properties of the discovered mean vectors, focusing on the subspace spanned by these representations\. Specifically, we examine its effective dimensionality\. The resulting distribution of explained variance across principal components is shown in[Figure˜6\(a\)](https://arxiv.org/html/2607.01774#S4.F6.sf1)\. Surprisingly, a large fraction of the intra\-layer variance can be captured by a single principal component, with the distribution heavily concentrated above90%90\\%explained variance\. Motivated by this observation, we project the mean vectors from selected layers into two\- and three\-dimensional PCA spaces to better visualise their structure\. We show a sample 2D and 3D projection in[Figure˜6\(b\)](https://arxiv.org/html/2607.01774#S4.F6.sf2)and[Figure˜1](https://arxiv.org/html/2607.01774#S1.F1), respectively\. Interestingly, we find the 3D shape to be closely related to structures that models have been shown to exhibit when operating on counting or time\-dependent and sequentially ordered tasks\(Gurneeet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib3); Engelset al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib12); Karkadaet al\.,[2026](https://arxiv.org/html/2607.01774#bib.bib11); Modellet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib10)\), characterised by a continuous and ordered representation of the time\-axis being analysed \(in our caseτ\\tau\)\.

To test whether the functional effect ofτ\\tausteering is concentrated in this low\-dimensional subspace, we repeat the intervention of[Equation˜4](https://arxiv.org/html/2607.01774#S4.E4), but restrict the steering vectorΔlt→t^:=μt^,l−μt,l\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}:=\\mu\_\{\\hat\{t\},l\}\-\\mu\_\{t,l\}to a chosen set of principal directions\. Concretely, we isolate the part ofΔlt→t^\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}lying in the subspace spanned by the firstkkprincipal components and the part orthogonal to it, rescaling each to match the norm of the original steering vector:

Δ∥,lt→t^\\displaystyle\\Delta\_\{\\parallel,l\}^\{t\\to\\hat\{t\}\}:=P∥,l​Δlt→t^​‖Δlt→t^‖‖P∥,l​Δlt→t^‖\\displaystyle=P\_\{\\parallel,l\}\\,\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}\\,\\frac\{\\\|\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}\\\|\}\{\\\|P\_\{\\parallel,l\}\\,\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}\\\|\}\(7\)Δ⟂,lt→t^\\displaystyle\\Delta\_\{\\perp,l\}^\{t\\to\\hat\{t\}\}:=P⟂,l​Δlt→t^​‖Δlt→t^‖‖P⟂,l​Δlt→t^‖\\displaystyle=P\_\{\\perp,l\}\\,\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}\\,\\frac\{\\\|\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}\\\|\}\{\\\|P\_\{\\perp,l\}\\,\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}\\\|\}HereP∥,lP\_\{\\parallel,l\}denotes the orthogonal projection onto the subspace spanned by the firstkkprincipal components of layerll, andP⟂,l=I−P∥,lP\_\{\\perp,l\}=I\-P\_\{\\parallel,l\}the projection onto its complement\. Because the top components capture most of the variance \([Figure˜6\(a\)](https://arxiv.org/html/2607.01774#S4.F6.sf1)\), we have‖P∥,l​Δlt→t^‖≫‖P⟂,l​Δlt→t^‖\\\|P\_\{\\parallel,l\}\\,\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}\\\|\\gg\\\|P\_\{\\perp,l\}\\,\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}\\\|; rescaling both perturbations to‖Δlt→t^‖\\\|\\Delta\_\{l\}^\{t\\to\\hat\{t\}\}\\\|therefore isolates the effect of the steering*direction*from that of its*magnitude*\. As shown in[Figure˜7](https://arxiv.org/html/2607.01774#S5.F7), projecting onto just the first two principal components recovers the same behaviour as the full, unprojected perturbation\. Interestingly, the orthogonal perturbationΔ⟂,lt→t^\\Delta\_\{\\perp,l\}^\{t\\to\\hat\{t\}\}produces incoherent effects at the same norm, implying that those directions carry little of theτ\\taurepresentation\. We present more results on[Appendix˜D](https://arxiv.org/html/2607.01774#A4)\. These results motivate a closer examination of whether this geometry can be consistently described across layers and aggregated into a unified, model\-level representation\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x9.png)Figure 7:Low\-dimensional subspace steering in LLaDA\.Using[Equation˜7](https://arxiv.org/html/2607.01774#S5.E7), we steer the model within the two\-dimensional subspace \(k=2k=2\) spanned by the top principal components of the layer\-29 mean vectors\. Steering within the subspace closely resembles the unrestricted one, while the orthogonal perturbation produces incoherent effects\.
### 5\.2A shared representation across layers

From the observation that many layers exhibit a similar parabolic shape, we investigate whether a general model\-level 2D representation of the mean vector components exists\. In particular, we aim to model a 2D trajectoryf​\(t\):\[100\]→ℝ2f\(t\):\[100\]\\to\\mathbb\{R\}^\{2\}that describes the geometry of the top two principal components across the entire model\.

To this end, we collect the 2D projections from all layers and standardise them to have zero mean and unit variance\. We then compute, for eacht∈\[100\]t\\in\[100\]\(corresponding to the 100 mean vectors projected per layer\), the empirical mean of the 2D points associated with the mean vectors binned bytt\. This yields a mean\-standardised projection across layers\. From this procedure, we obtain a non\-parametric 2D trajectory that represents the standardised expectation of the projections for eachτ\\tauindex\. We report the resulting plot in[Figure˜8](https://arxiv.org/html/2607.01774#S5.F8)for LLaDA \(see[Appendix˜F](https://arxiv.org/html/2607.01774#A6)for similar results on Dream\)\. Remarkably, all points closely follow the proposed shared trajectory, with error bars remaining within approximately0\.10\.1, indicating that the proposed approximation faithfully captures the general model\-level geometry of the subspace\. This further implies that the principal components of the mean vector distributions follow the same parabolic shape up to scaling and offset factors\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x10.png)Figure 8:The shared 2\-dimensional geometry oft^\\hat\{t\}in LLaDA\.We compute an average trajectory by taking the standardised 2D PC projections of the mean vectors across the model\. The parabolic geometry is shared across layers\. PC1 and PC2 are standardised to unit variance\.While this construction reveals a stable and coherent 2D structure shared across layers, it remains unclear whether the underlying representation is consistently encoded throughout the depth of the model or whether it emerges from layer\-specific implementations\. We next address this question by studying the cross\-layer alignment of the discovered mean vectors\.

### 5\.3Cross\-layerτ\\taurepresentations

Based on these results, we further investigate how the discovered mean vectors are connected across layers\. In particular, we ask whether the model maintains a consistent direction\-wise representation of a givenμt\\mu\_\{t\}vector across depth, or instead develops layer\-local representations that do not generalise\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x11.png)Figure 9:Average cosine similarity of same\-indexed vectors across layers in LLaDA\.We observe that most layers maintain a highly correlated representation of the same indextt, and that this relationship degrades with distance\. Layer 32 instead maintains a representation that is largely independent of the other layers\.To address this question, we compute the average pairwise cosine similarity between all centred vectors encoding a given indexttacross different layers\. We report the results in[Figure˜9](https://arxiv.org/html/2607.01774#S5.F9)for LLaDA and in[Appendix˜F](https://arxiv.org/html/2607.01774#A6)for Dream\.

Notably, we observe a clear structural difference between the two models in how they preserve representations of the same mean vector across depth\. In LLaDA, most layers exhibit strong alignment in their representations, with the exception of layer 32, which appears nearly orthogonal to the others\. In contrast, Dream shows a more heterogeneous organisation: representations are only consistently aligned within specific blocks of layers, while layers outside these blocks are largely uncorrelated with those inside them\. This suggests that there may be a model\-level basis for encodingτ\\tauacross layers, as having high correlation implies that the downstream effects would be similar if interchanging the mean vectors from those layers\. Although the previous sections establish the existence of structured and consistentτ\\taurepresentations across layers, they do not reveal how these representations are formed within the computations of each layer, which we investigate in the next section\.

### How isτ\\taurepresented within a layer?

To query the models’ internal representation of the mean vectors we employ the same approach as proposed in[Equation˜3](https://arxiv.org/html/2607.01774#S4.E3), but now capture activations inside the models’ intermediate layer computations\. With these mean vectors, we measure the cosine similarity of all same\-dimensional representations inside the models’ layers \(as the MLP has a different hidden dimension compared to the rest of the model\)\. We present the results for LLaDA at[Figure˜10](https://arxiv.org/html/2607.01774#S5.F10)\(and[Appendix˜F](https://arxiv.org/html/2607.01774#A6)for Dream\)\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x12.png)Figure 10:Average cosine similarity across layer components in LLaDA\.Whereh′:=hl\+Self\-Attention​\(⋅\)h^\{\\prime\}:=h\_\{l\}\+\\text\{Self\-Attention\}\(\\cdot\), andh^:=RMSNorm​\(h′\)\\hat\{h\}:=\\text\{RMSNorm\}\(h^\{\\prime\}\)\. We clearly see how across most of the internal representations there is high\-correlation or almost complete orthogonality\. We note how self\-attention and the outputs of the out matrix seem to be anticorrelated\.As the results suggest, most operations before and after the MLP remain highly correlated; nonetheless, inside the MLP, the mean vectors produce almost completely independent representations ofτ\\tau, as for both models they are orthogonal in expectation \(with the exception of the post up\-projection matrix and post activation function, which remain highly correlated\)\. We note a particular pattern in both models between the post\-MLP and post\-attention similarities, as both are highly anti\-correlated, sharing a common direction but pointing in opposite ways\. This phenomenon can be better appreciated in[Figure˜11](https://arxiv.org/html/2607.01774#S5.F11)\. Interestingly, the two stages agree on the distance from the centre of aτ\\tauvalue but not on its sign\. We hypothesise that this seeming contradiction in the meanτ\\taurepresentation may help the model correct the signal as shown in[Section˜4](https://arxiv.org/html/2607.01774#S4.SSx1)\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x13.png)Figure 11:Average cosine similarity of self\-attention and post\-WoutW\_\{\\text\{out\}\}vectors\.Layer\-29 vectors from LLaDA reveal a clear geometric organization overtt: representations for distant indices become nearly anti\-aligned, and a transition region emerges aroundt=50t=50, where vectors are approximately perpendicular\.✓ Takeaway for RQ3*The models represent the mean vectors subspace in an ordered and manifold\-like manner\. Most layers in LLaDA share common semantics for this space, while Dream is organised into representation blocks\. Finally, self\-attention and the MLP have opposite representations\.*

## 6Related Work

LLM Interpretability\. Prior work studies activation subspaces and representation geometry to explain LLM behaviour\. Closest to us,Gurneeet al\.\([2025](https://arxiv.org/html/2607.01774#bib.bib3)\)use mean activation vectors to analyse and steer a character counting task\. We similarly use mean vectors, but study a distributed denoising\-time signal and its propagation across layers\.Engelset al\.\([2025](https://arxiv.org/html/2607.01774#bib.bib12)\),Modellet al\.\([2025](https://arxiv.org/html/2607.01774#bib.bib10)\), andKarkadaet al\.\([2026](https://arxiv.org/html/2607.01774#bib.bib11)\)analyse concept and token\-level representations in language models, describing low\-dimensional manifold geometries that, in the case of ordered concepts such as historical years, resemble the one we find forτ\\tau\. In contrast to their work, we study a sequence\-level statistic that the model reconstructs as "meta\-level" information to track denoising progress, rather than a human\-like concept \(such as years or months\) whose geometry has been tied to statistical co\-occurrences\.

DLM Denoising Dynamics and Interpretability\. Recent DLM work has examined both steering and internal behaviour\.Zhouet al\.\([2026](https://arxiv.org/html/2607.01774#bib.bib50)\)find that concepts are steerable only at particular denoising stages; in contrast, ourτ\\tau\-related signal persists throughout diffusion and is repeatedly recomputed\.Rossiet al\.\([2026](https://arxiv.org/html/2607.01774#bib.bib51)\)study DLM length\-awareness through downstream performance, whereas we characterise an internal representation tied to denoising progress\.

## 7Conclusion and Discussion

We characterise a denoising\-progress\-related signal in the residual streams of DLMs\. The signal is probe\-decodable across layers and token types, causally steerable through mean vector directions, repeatedly corrected across depth, and represented in a structured low\-dimensional subspace\. Steering this representation produces predictable changes in entropy, confidence, and KL divergence, indicating that the recovered directions are not merely descriptive but functionally relevant to the model’s computation\.

A natural next question is how this signal is computed\. Our results show that DLMs internally track information closely related to the fraction of unmasked tokens, but they do not identify the circuit that constructs or updates this representation\. Future work could therefore investigate whether the signal is derived from explicit mask\-ratio estimation, from distributed sequence\-level statistics, or from interactions between attention and MLP components\. Understanding this computation would better clarify how DLMs operate in this subspace and enable better architecture designs for these models\.

## 8Limitations

Although our work aims to characterise and explain the representation ofτ\\tauinside DLMs, several open questions remain\. First, our analysis is restricted to LLaDA and Dream, two DLMs trained with the same cross\-entropy loss over \[mask\] tokens; it therefore remains unexplored how theτ\\tauphenomenon emerges in other DLMs that similarly lack explicit timestep conditioning, such as block\-diffusion models\(Chenget al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib48); Arriolaet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib49); Wuet al\.,[2025](https://arxiv.org/html/2607.01774#bib.bib54)\)\. Moreover, it is unclear whether the identifiedτ\\tausignal can be exploited at inference time to enable more efficient decoding—or even remasking—strategies, which we leave as a promising direction for performance improvements\. Finally, the token\-level effects of steering remain unexplored: our analyses focus on sequence\-level statistics rather than examining which tokens change, and how, as the signal is steered\.

## 9Acknowledgements

This work was supported by a grant from Coefficient Giving, administered by the Berkeley Existential Risk Initiative \(BERI\), and by Sapienza grant RG123188B3EF6A80 \(CENTS\)\. We acknowledge ISCRA for awarding this project access to the LEONARDO supercomputer, owned by the EuroHPC Joint Undertaking and hosted by CINECA \(Italy\)\. We thank Fastweb S\.p\.A\. for providing the computational resources used in this paper\. We also thank Donatella Genovese and Francesco Piccolo for helpful discussions and valuable feedback\.

We used LLMs only for minor grammar, wording, and readability checks\. We reviewed and edited the final manuscript and take full responsibility for its content\.

## 10Ethical considerations

This work analyses the internal representations of publicly released diffusion language models \(LLaDA and Dream\) and does not involve human subjects, new data collection, or the release of new models\. We foresee no ethical concerns arising from this work\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p1.1)\.
- System card: Claude Opus 4 & Claude Sonnet 4\.Note:[https://www\.anthropic\.com/claude\-4\-system\-card](https://www.anthropic.com/claude-4-system-card)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p1.1)\.
- M\. Arriola, A\. Gokaslan, J\. Chiu, Z\. Yang, Z\. Qi, J\. Han, S\. Sahoo, and V\. Kuleshov \(2025\)Block diffusion: interpolating between autoregressive and diffusion language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 50726–50753\.Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1),[§8](https://arxiv.org/html/2607.01774#S8.p1.3)\.
- U\. Bhalla, T\. Fel, C\. Rager, S\. Feucht, T\. Haklay, D\. Wurgaft, S\. Boppana, M\. Kowal, V\. Shyam, J\. Merullo, A\. Geiger, and E\. S\. Lubana \(2026\)Do sparse autoencoders capture concept manifolds?\.External Links:2604\.28119,[Link](https://arxiv.org/abs/2604.28119)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p1.1)\.
- S\. Cheng, Y\. Bian, D\. Liu, L\. Zhang, Q\. Yao, Z\. Tian, W\. Wang, Q\. Guo, K\. Chen, B\. Qi,et al\.\(2025\)Sdar: a synergistic diffusion\-autoregression paradigm for scalable sequence generation\.arXiv preprint arXiv:2510\.06303\.Cited by:[§8](https://arxiv.org/html/2607.01774#S8.p1.3)\.
- H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. Sharkey \(2023\)Sparse autoencoders find highly interpretable features in language models\.External Links:2309\.08600,[Link](https://arxiv.org/abs/2309.08600)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-V3 technical report\.External Links:2412\.19437Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p1.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.External Links:1810\.04805,[Link](https://arxiv.org/abs/1810.04805)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1)\.
- Z\. Du, K\. Xia, X\. Zhong, Y\. Fu, N\. Oswald, B\. Ji, B\. Khailany, P\. Molchanov, and Y\. Lin \(2026\)R2\-dllm: accelerating diffusion large language models via spatio\-temporal redundancy reduction\.arXiv preprint arXiv:2604\.18995\.Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1)\.
- J\. Engels, E\. Michaud, I\. Liao, W\. Gurnee, and M\. Tegmark \(2025\)Not all language model features are one\-dimensionally linear\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 84591–84622\.Cited by:[§5\.1](https://arxiv.org/html/2607.01774#S5.SS1.p1.2),[§6](https://arxiv.org/html/2607.01774#S6.p1.1)\.
- Gemma Team \(2025\)Gemma 3 technical report\.External Links:2503\.19786Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p1.1)\.
- W\. Gurnee, E\. Ameisen, I\. Kauvar, T\. ,Julius, A\. Pearce, C\. Olah, and J\. Batson \(2025\)When models manipulate manifolds: the geometry of a counting task\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2025/linebreaks/index.html)Cited by:[§4\.1](https://arxiv.org/html/2607.01774#S4.SS1.p1.2),[§5\.1](https://arxiv.org/html/2607.01774#S5.SS1.p1.2),[§6](https://arxiv.org/html/2607.01774#S6.p1.1)\.
- P\. Huang, S\. Liu, Z\. Liu, Y\. Yan, S\. Wang, Z\. Chen, and T\. Xiao \(2025\)Pc\-sampler: position\-aware calibration of decoding bias in masked diffusion models\.arXiv preprint arXiv:2508\.13021\.Cited by:[Appendix C](https://arxiv.org/html/2607.01774#A3.SS0.SSS0.Px6.p1.4)\.
- A\. Joshi, D\. Bhatt, and A\. Modi \(2025\)Geometry of decision making in language models\.External Links:2511\.20315,[Link](https://arxiv.org/abs/2511.20315)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p1.1)\.
- D\. Karkada, D\. J\. Korchinski, A\. Nava, M\. Wyart, and Y\. Bahri \(2026\)Symmetry in language statistics shapes the geometry of model representations\.arXiv preprint arXiv:2602\.15029\.Cited by:[§5\.1](https://arxiv.org/html/2607.01774#S5.SS1.p1.2),[§6](https://arxiv.org/html/2607.01774#S6.p1.1)\.
- X\. Li, J\. Thickstun, I\. Gulrajani, P\. S\. Liang, and T\. B\. Hashimoto \(2022\)Diffusion\-lm improves controllable text generation\.Advances in neural information processing systems35,pp\. 4328–4343\.Cited by:[§2](https://arxiv.org/html/2607.01774#S2.p1.4)\.
- I\. Loshchilov and F\. Hutter \(2017\)Decoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[Appendix B](https://arxiv.org/html/2607.01774#A2.p1.4)\.
- Meta AI \(2025\)Introducing Llama 4: advancing multimodal intelligence\.Note:[https://ai\.meta\.com/blog/llama\-4\-multimodal\-intelligence/](https://ai.meta.com/blog/llama-4-multimodal-intelligence/)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p1.1)\.
- A\. Modell, P\. Rubin\-Delanchy, and N\. Whiteley \(2025\)The origins of representation manifolds in large language models\.arXiv preprint arXiv:2505\.18235\.Cited by:[§5\.1](https://arxiv.org/html/2607.01774#S5.SS1.p1.2),[§6](https://arxiv.org/html/2607.01774#S6.p1.1)\.
- S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\. Wen, and C\. Li \(2025\)Large language diffusion models\.External Links:2502\.09992,[Link](https://arxiv.org/abs/2502.09992)Cited by:[Appendix A](https://arxiv.org/html/2607.01774#A1.p1.2),[§1](https://arxiv.org/html/2607.01774#S1.p2.1)\.
- J\. Ou, S\. Nie, K\. Xue, F\. Zhu, J\. Sun, Z\. Li, and C\. Li \(2024\)Your absorbing discrete diffusion secretly models the conditional distributions of clean data\.arXiv preprint arXiv:2406\.03736\.Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1),[§2](https://arxiv.org/html/2607.01774#S2.p1.4)\.
- N\. Panickssery, N\. Gabrieli, J\. Schulz, M\. Tong, J\. Hubinger, and A\. Turner \(2024\)Steering llama 2 via contrastive activation addition\.arXiv preprint arXiv:2312\.06681\.Cited by:[§4\.1](https://arxiv.org/html/2607.01774#S4.SS1.p1.2)\.
- J\. Piskorz, C\. Pinneri, A\. Correia, M\. Alfarra, R\. Garrepalli, and C\. Louizos \(2025\)Masks can be distracting: on context comprehension in diffusion language models\.External Links:2511\.21338,[Link](https://arxiv.org/abs/2511.21338)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1)\.
- V\. Rossi, G\. Cirò, D\. Beltrame, L\. Gandolfi, P\. Röttger, and D\. Hovy \(2026\)Diffusion language models are natively length\-aware\.arXiv preprint arXiv:2603\.06123\.Cited by:[§6](https://arxiv.org/html/2607.01774#S6.p2.1)\.
- M\. E\. Rulli, S\. Petruzzi, E\. Michielon, F\. Silvestri, S\. Scardapane, and A\. Devoto \(2025\)Attention sinks in diffusion language models\.arXiv preprint arXiv:2510\.15731\.Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1)\.
- B\. Saglam, P\. Kassianik, B\. Nelson, S\. Weerawardhena, Y\. Singer, and A\. Karbasi \(2026\)Large language models encode semantics and alignment in linearly separable representations\.External Links:2507\.09709,[Link](https://arxiv.org/abs/2507.09709)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p1.1)\.
- S\. S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. Marroquin, J\. T\. Chiu, A\. Rush, and V\. Kuleshov \(2024\)Simple and effective masked diffusion language models\.External Links:2406\.07524,[Link](https://arxiv.org/abs/2406.07524)Cited by:[§2](https://arxiv.org/html/2607.01774#S2.p1.4)\.
- A\. S\. Shai, S\. E\. Marzen, L\. Teixeira, A\. G\. Oldenziel, and P\. M\. Riechers \(2025\)Transformers represent belief state geometry in their residual stream\.External Links:2405\.15943,[Link](https://arxiv.org/abs/2405.15943)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p1.1)\.
- J\. Shi, K\. Han, Z\. Wang, A\. Doucet, and M\. K\. Titsias \(2025\)Simplified and generalized masked diffusion for discrete data\.External Links:2406\.04329,[Link](https://arxiv.org/abs/2406.04329)Cited by:[§2](https://arxiv.org/html/2607.01774#S2.p1.4)\.
- F\. Tiblias, I\. Bigoulaeva, J\. Niu, S\. Balloccu, and I\. Gurevych \(2026\)Hypothesis\-driven feature manifold analysis in llms via supervised multi\-dimensional scaling\.External Links:2510\.01025,[Link](https://arxiv.org/abs/2510.01025)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p1.1)\.
- W\. Wang, J\. Yang, and W\. Peng \(2025\)Semantics\-adaptive activation intervention for llms via dynamic steering vectors\.arXiv preprint arXiv:2410\.12299\.Cited by:[§4\.1](https://arxiv.org/html/2607.01774#S4.SS1.p1.2),[§4\.1](https://arxiv.org/html/2607.01774#S4.SS1.p2.2)\.
- X\. Wang, B\. Jiang, Y\. Wan, B\. Yang, L\. Kong, and D\. Zou \(2026\)DLM\-scope: mechanistic interpretability of diffusion language models via sparse autoencoders\.External Links:2602\.05859,[Link](https://arxiv.org/abs/2602.05859)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1)\.
- Z\. Wen, J\. Qu, Z\. Chen, X\. Lu, D\. Liu, Z\. Liu, R\. Wu, Y\. Yang, X\. Jin, H\. Xu, X\. Liu, W\. Li, C\. Lu, J\. Shao, C\. He, and L\. Zhang \(2026\)The devil behind the mask: an emergent safety vulnerability of diffusion llms\.External Links:2507\.11097,[Link](https://arxiv.org/abs/2507.11097)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1)\.
- C\. Wu, H\. Zhang, S\. Xue, S\. Diao, Y\. Fu, Z\. Liu, P\. Molchanov, P\. Luo, S\. Han, and E\. Xie \(2025\)Fast\-dllm v2: efficient block\-diffusion llm\.External Links:2509\.26328,[Link](https://arxiv.org/abs/2509.26328)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1),[§8](https://arxiv.org/html/2607.01774#S8.p1.3)\.
- J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. Kong \(2025\)Dream 7b: diffusion large language models\.External Links:2508\.15487,[Link](https://arxiv.org/abs/2508.15487)Cited by:[Appendix A](https://arxiv.org/html/2607.01774#A1.p1.2),[§1](https://arxiv.org/html/2607.01774#S1.p2.1),[§1](https://arxiv.org/html/2607.01774#S1.p7.1)\.
- Y\. Yu, Y\. Jian, J\. Wang, Z\. Zhou, D\. Zhuang, X\. Fang, S\. Yanamandra, X\. Wu, Q\. Wu, S\. L\. Song,et al\.\(2026\)Introspective diffusion language models\.arXiv preprint arXiv:2604\.11035\.Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1)\.
- O\. Zekri, T\. Uscidda, N\. Boullé, and A\. Korba \(2026\)Generalized discrete diffusion from snapshots\.arXiv preprint arXiv:2603\.21342\.Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1)\.
- K\. Zheng, Y\. Chen, H\. Mao, M\. Liu, J\. Zhu, and Q\. Zhang \(2025\)Masked diffusion models are secretly time\-agnostic masked models and exploit inaccurate categorical sampling\.External Links:2409\.02908,[Link](https://arxiv.org/abs/2409.02908)Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1)\.
- H\. Zhou, S\. Roy, and R\. Gangadharaiah \(2026\)Steering without breaking: mechanistically informed interventions for discrete diffusion language models\.arXiv preprint arXiv:2605\.10971\.Cited by:[§6](https://arxiv.org/html/2607.01774#S6.p2.1)\.
- F\. Zhu, R\. Wang, S\. Nie, X\. Zhang, C\. Wu, J\. Zhou, Y\. Lin, J\. Wen, and C\. Li \(2026\)Llada 1\.5: variance\-reduced preference optimization for large language diffusion models\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11425–11460\.Cited by:[§1](https://arxiv.org/html/2607.01774#S1.p2.1),[§1](https://arxiv.org/html/2607.01774#S1.p7.1)\.

## Appendix ADenoising Progress and the Training Noising Level

Here we provide more details about DLMs’ training objectives and clarify the relationship between the training noising levelssand the empirical denoising\-progress variableτt\\tau\_\{t\}used throughout the paper\. For more details about these models’ formulations we refer the reader toNieet al\.\([2025](https://arxiv.org/html/2607.01774#bib.bib38)\)andYeet al\.\([2025](https://arxiv.org/html/2607.01774#bib.bib39)\)\. In absorbing\-mask diffusion language models, the fraction of response tokens that remain masked is an empirical analogue of the training noising level, while the fraction of response tokens that have been unmasked measures denoising progress\.

#### Absorbing\-mask corruption\.

Letx0jx\_\{0\}^\{j\}denote the clean token at positionjj, represented as a one\-hot vector\. We write \[mask\] both for the absorbing mask token and, when used inside a categorical parameter, for its corresponding one\-hot vector\. At noising levels∈\[0,1\]s\\in\[0,1\], each token is independently replaced by \[mask\] with probabilityss\. For a single positionjj,

qs∣0j​\(z∣x0j\)=\{1−s,z=x0j,s,z=\[mask\],0,otherwise\.q^\{j\}\_\{s\\mid 0\}\(z\\mid x\_\{0\}^\{j\}\)=\\begin\{cases\}1\-s,&z=x\_\{0\}^\{j\},\\\\ s,&z=\[\\textsc\{mask\}\],\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}
Equivalently,

qs∣0j\(⋅∣x0j\)=Cat\(⋅;\(1−s\)x0j\+s\[mask\]\)\.q^\{j\}\_\{s\\mid 0\}\(\\cdot\\mid x\_\{0\}^\{j\}\)=\\operatorname\{Cat\}\\bigl\(\\cdot;\(1\-s\)x\_\{0\}^\{j\}\+s\[\\textsc\{mask\}\]\\bigr\)\.
Assuming independent corruption across positions, the full corrupted sequence distribution factorises as

qs∣0\(⋅∣x0\)=∏jqs∣0j\(⋅∣x0j\)\.q\_\{s\\mid 0\}\(\\cdot\\mid x\_\{0\}\)=\\prod\_\{j\}q^\{j\}\_\{s\\mid 0\}\(\\cdot\\mid x\_\{0\}^\{j\}\)\.
Thus,ssis the probability that a token is masked, and1−s1\-sis the probability that it remains visible\.

#### Training objectives\.

For a corrupted sequencexsx\_\{s\}, define the masked\-position set

ℳ​\(xs\):=\{j:xsj=\[mask\]\}\.\\mathcal\{M\}\(x\_\{s\}\):=\\\{j:x\_\{s\}^\{j\}=\[\\textsc\{mask\}\]\\\}\.
The pretraining objective can then be written compactly as

ℒpre=−𝔼x0,s,xs​\[1s​∑j∈ℳ​\(xs\)log⁡pθ​\(x0j∣xs\)\]\.\\mathcal\{L\}\_\{\\mathrm\{pre\}\}=\-\\mathbb\{E\}\_\{x\_\{0\},s,x\_\{s\}\}\\left\[\\frac\{1\}\{s\}\\sum\_\{j\\in\\mathcal\{M\}\(x\_\{s\}\)\}\\log p\_\{\\theta\}\(x\_\{0\}^\{j\}\\mid x\_\{s\}\)\\right\]\.\(8\)
During supervised fine\-tuning, the prompt is kept fixed and only response tokens are corrupted\. Letp0p\_\{0\}denote the prompt,r0r\_\{0\}the clean response, andrsr\_\{s\}the corrupted response\. Define

ℳ​\(rs\):=\{j:rsj=\[mask\]\}\.\\mathcal\{M\}\(r\_\{s\}\):=\\\{j:r\_\{s\}^\{j\}=\[\\textsc\{mask\}\]\\\}\.
The SFT objective is

ℒSFT=−𝔼p0,r0,s,rs​\[1s​∑j∈ℳ​\(rs\)log⁡pθ​\(r0j∣p0,rs\)\]\.\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}=\-\\mathbb\{E\}\_\{p\_\{0\},r\_\{0\},s,r\_\{s\}\}\\left\[\\frac\{1\}\{s\}\\sum\_\{j\\in\\mathcal\{M\}\(r\_\{s\}\)\}\\log p\_\{\\theta\}\(r\_\{0\}^\{j\}\\mid p\_\{0\},r\_\{s\}\)\\right\]\.\(9\)
Both objectives therefore train the model on partially masked sequences, where the expected fraction of masked tokens is controlled byss\.

#### Inference\-time denoising progress\.

At inference time, the model does not sample a corrupted sequence directly fromqs∣0q\_\{s\\mid 0\}\. Instead, it starts from a fully masked response and progressively unmasks tokens over discrete denoising stepst∈\{0,…,T\}t\\in\\\{0,\\ldots,T\\\}\.

LetRRbe the set of response positions, withL:=\|R\|L:=\|R\|\. For a realised inference statextx\_\{t\}, define the response\-local masked and unmasked sets as

ℳR​\(xt\)\\displaystyle\\mathcal\{M\}\_\{R\}\(x\_\{t\}\):=\{j∈R:xtj=\[mask\]\},\\displaystyle=\\\{j\\in R:x\_\{t\}^\{j\}=\[\\textsc\{mask\}\]\\\},\(10\)𝒰R​\(xt\)\\displaystyle\\mathcal\{U\}\_\{R\}\(x\_\{t\}\):=\{j∈R:xtj≠\[mask\]\}\.\\displaystyle=\\\{j\\in R:x\_\{t\}^\{j\}\\neq\[\\textsc\{mask\}\]\\\}\.
These sets partition the response positions, so

\|ℳR​\(xt\)\|\+\|𝒰R​\(xt\)\|=L\.\|\\mathcal\{M\}\_\{R\}\(x\_\{t\}\)\|\+\|\\mathcal\{U\}\_\{R\}\(x\_\{t\}\)\|=L\.
We define the empirical denoising progress at stepttas

τt:=\|𝒰R​\(xt\)\|L\.\\tau\_\{t\}:=\\frac\{\|\\mathcal\{U\}\_\{R\}\(x\_\{t\}\)\|\}\{L\}\.\(11\)
Its complement is the remaining response\-local mask ratio:

1−τt=\|ℳR​\(xt\)\|L\.1\-\\tau\_\{t\}=\\frac\{\|\\mathcal\{M\}\_\{R\}\(x\_\{t\}\)\|\}\{L\}\.\(12\)
Thus,τt=0\\tau\_\{t\}=0corresponds to a fully masked response, whileτt=1\\tau\_\{t\}=1corresponds to a fully unmasked response\.

#### Connection to the training noising level\.

Under the SFT corruption process, each response token is masked independently with probabilityss\. Therefore, forrs∼qs∣0\(⋅∣r0\)r\_\{s\}\\sim q\_\{s\\mid 0\}\(\\cdot\\mid r\_\{0\}\),

𝔼rs​\[\|ℳ​\(rs\)\|L\]=s\.\\mathbb\{E\}\_\{r\_\{s\}\}\\left\[\\frac\{\|\\mathcal\{M\}\(r\_\{s\}\)\|\}\{L\}\\right\]=s\.\(13\)
Similarly, if𝒰​\(rs\):=\{j:rsj≠\[mask\]\}\\mathcal\{U\}\(r\_\{s\}\):=\\\{j:r\_\{s\}^\{j\}\\neq\[\\textsc\{mask\}\]\\\}, then

𝔼rs​\[\|𝒰​\(rs\)\|L\]=1−s\.\\mathbb\{E\}\_\{r\_\{s\}\}\\left\[\\frac\{\|\\mathcal\{U\}\(r\_\{s\}\)\|\}\{L\}\\right\]=1\-s\.\(14\)
Consequently, the realised inference\-time mask ratio\(1−τt\)\(1\-\\tau\_\{t\}\)plays the role of an effective noising level, whileτt\\tau\_\{t\}measures effective denoising progress\.

#### Interpretation\.

The equivalence above should be understood at the level of mask ratios, not as an equality between the training and inference distributions\. During training,ssindexes an independently sampled corruption process\. During inference, the sequence of states is generated by the model and its unmasking policy\. Nevertheless, both procedures produce partially masked responses, and each such response has a well\-defined response\-local mask ratio\.

We therefore useτt\\tau\_\{t\}as an empirical measure of denoising progress throughout the paper\. Its complement,1−τt1\-\\tau\_\{t\}, is the realised fraction of response tokens that remain masked and is directly analogous to the training noising levelss\. Recoveringτt\\tau\_\{t\}from the residual stream can thus be interpreted as recovering the model’s internal representation of its position along the denoising trajectory\.

## Appendix BProbe Architectural and Training Details

We use a 5\-layer residual neural network with LayerNorm\+GELU blocks, shown in[Figure˜12](https://arxiv.org/html/2607.01774#A2.F12), and bound the outputs to the interval\(0,1\)\(0,1\)by applying a sigmoid function to the MLP logits, since the targetτ\\tauis itself bounded\. We train the probes for 20 epochs using 300 training examples and 100 validation examples\. For each example and at each epoch, we dynamically vary the generation length and number of steps to improve the probe’s coverage\. We implement batch gradient descent by updating the parameters after each full example, i\.e\., by batching the gradients over the entire denoising stage\. We employ the AdamWLoshchilov and Hutter \([2017](https://arxiv.org/html/2607.01774#bib.bib57)\)weight optimiser, setting the learning rate toα=10−3\\alpha=10^\{\-3\}and the weight decay coefficient to6×10−66\\times 10^\{\-6\}\. All experiments introduced earlier were conducted on NVIDIA H100 and A100 GPUs\.

Figure 12:MLP probe architecture\.Eachblockis aLinear→LayerNorm→GELU\\mathrm\{Linear\}\\\!\\to\\\!\\mathrm\{LayerNorm\}\\\!\\to\\\!\\mathrm\{GELU\}unit; the probe stacks five such blocks around a single residual connection, and a sigmoid head bounds the output to\(0,1\)\(0,1\)to match the range ofτ\\tau\. The hidden widthwwis capped at10241024\.
## Appendix CAdditional Experiments on Probing and Steeringτ\\tau

In this section, we present additional results concerning the probes’ capabilities to correctly identify the value ofτ\\tau, followed by further analyses of the mean activation vectors’ ability to steer the model\. The first part extends the probing analysis to Dream and to linear probes; the second broadens the steering analysis to additional layers, generation lengths, and downstream tasks\.

#### MLP probes on Dream\.

[Figure˜13](https://arxiv.org/html/2607.01774#A3.F13)shows theR2\\rm\{R\}^\{2\}obtained using the MLP probe over Dream\. Similarly to what was observed with LLaDA, the probes’ accuracy is high across the whole model but drops in the deeper layers\.

#### Linear probes\.

We also train linear probes with the same sigmoid output and regression objective as the MLPs \([Figure˜12](https://arxiv.org/html/2607.01774#A2.F12)\), reporting the results in[Figure˜15](https://arxiv.org/html/2607.01774#A3.F15)\. Both models’ linear probes trackτ\\tauwell through the early and middle layers, but their accuracy then falls off more steeply than the MLP probes as we move deeper, and in the final layers it collapses essentially to the baseline of predicting the meanτ\\tau\. Across the broad mid\-to\-deep range the MLP probes retain markedly higherR2\\rm\{R\}^\{2\}\([Figure˜2](https://arxiv.org/html/2607.01774#S2.F2),[Figure˜13](https://arxiv.org/html/2607.01774#A3.F13)\), so for those layers we read the gap not as theτ\\tausignal disappearing at the token level, but as it becoming less linearly accessible: the information is still present, yet increasingly encoded in a form a single linear readout cannot capture\. Only at the very last layers, where the MLP performance also degrades, does the token\-level signal become genuinely hard to recover by either probe\. The choice of training tokens also matters more here than for the MLPs: for LLaDA, restricting training to non\-\[mask\] tokens opens a noticeable gap relative to \[mask\] or all tokens—consistent in direction with the smaller \[mask\] advantage seen for the MLP probes, but more pronounced—whereas Dream is mostly insensitive to the token subset\.

We stress that this does not contradict the low\-dimensionality ofτ\\tauestablished earlier: that structure is a property of the mean vectors, which average over many tokens and examples and thereby expose sequence\-level regularities to which an individual token’s activation—and hence a token\-level linear probe—has no comparable access\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x14.png)Figure 13:𝐑2\\mathbf\{R\}^\{2\}coefficients of MLPs in Dream\. TheR2\\rm\{R\}^\{2\}coefficient slightly degrades as we probe deeper into the model, which remains consistent with the observations of[Figure˜2](https://arxiv.org/html/2607.01774#S2.F2)\.
#### Mean\-vector steering on Dream\.

Having shown thatτ\\tauis decodable across both models, we now turn from reading the signal to perturbing it, extending the steering analysis beyond the layers reported in the main text\. We begin with Dream:[Figure˜14](https://arxiv.org/html/2607.01774#A3.F14)shows the downstream effects of steering at layer 25\. We observe a similar trend as in LLaDA, where entropy decreases and confidence increases when we steer towards later denoising steps\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x15.png)Figure 14:Mean\-steering downstream effects on layer 25 of Dream\. We steered the activations in layer 25 using the mean\-ratio vectors \(blue\) targeting differentτ\\tauvalues, and measured the variation in entropy, confidence and the KL divergence\. We compared it against random perturbations \(red\)\. Steering with the mean vectors has an effect that is consistent with what would be expected\.![Refer to caption](https://arxiv.org/html/2607.01774v1/x16.png)
![Refer to caption](https://arxiv.org/html/2607.01774v1/x17.png)

Figure 15:Linear probe performance\. The linear probes degrade markedly as we probe deeper into the model—more so than the MLP probes shown in[Figure˜2](https://arxiv.org/html/2607.01774#S2.F2)and[Figure˜13](https://arxiv.org/html/2607.01774#A3.F13)—and fall to the mean\-τ\\taubaseline in the final layers for both LLaDA \(top\) and Dream \(bottom\)\. This indicates that, at the token level,τ\\tauinformation becomes increasingly hard to decode linearly with depth, while an MLP still recovers much of it until the last few layers\.
#### Steering across intervention layers\.

[Figure˜20](https://arxiv.org/html/2607.01774#A6.F20)and[Figure˜21](https://arxiv.org/html/2607.01774#A6.F21)show the downstream results for LLaDA and Dream across a range of intervention layers and target binst^\\hat\{t\}\.

#### Robustness to generation length\.

In[Figure˜19](https://arxiv.org/html/2607.01774#A6.F19)we ablate the effect of generation length, repeating the mean\-steering analysis atL=64L=64andL=128L=128\(with the number of denoising steps matched toLL\)\. The downstream effects follow the same trajectory as in the main results across all steering targetst^\\hat\{t\}, indicating that the previously\-observed phenomena are not artifacts of a particular generation length\.

#### Downstream task performance\.

In[Table˜1](https://arxiv.org/html/2607.01774#A3.T1)we show the effect of mean vector steering on downstream task performance\. We observe that steering does not lead to a collapse in performance across the range oft^\\hat\{t\}values, with LLaDA remaining within a few points of its base score on all three benchmarks\. Dream is more sensitive, particularly on GSM8K, where performance degrades substantially att^=100\\hat\{t\}=100\. A qualitative inspection of Dream’s incorrect GSM8K generations reveals that this degradation is largely driven by the premature emission of end\-of\-sequence \(EOS\) tokens\. Steering towardst^=100\\hat\{t\}=100leads the model to unmask EOS tokens both early in the denoising trajectory and at early positions within the response window\. Since these models are trained to keep predicting EOS once the first one has been emitted, a single EOS placed near the start of the response causes the generation to collapse into a degenerate, EOS\-filled sequence\. This failure mode is consistent with the decoding biases reported byHuanget al\.\([2025](https://arxiv.org/html/2607.01774#bib.bib55)\), who show that uncertainty\-based samplers in DLMs over\-select trivial, high\-confidence tokens—including EOS, newlines, and punctuation—during the first denoising steps, and tend to commit the two ends of the sequence before its centre \(a “U\-shaped” decoding trajectory\)\. Our intervention can be read as amplifying this bias from the representation side: by driving the model’s internal estimate of denoising progress to its maximum, we further inflate the probability of completion tokens such as EOS, which in turn triggers the collapse described above\. Taken together, and while we leave a quantitative analysis to future work, this suggests that theτ\\tausignal is not a content\-agnostic progress counter, but is instead entangled with the token distribution itself—carrying latent information about which tokens become more or less probable as denoising advances, with sequence\-ending tokens such as EOS being a clear example\.

Table 1:Effect of mean\-steering on LLaDA \(layer 29\) and Dream \(layer 25\) on downstream performance\.

## Appendix DLow\-dimensional Steering

In this section we provide further insights on the causal importance of the found low\-dimensional subspaces when steering LLaDA and Dream\. To do so, we employ the subspace steering method introduced in[Equation˜7](https://arxiv.org/html/2607.01774#S5.E7)\. As[Figure˜22](https://arxiv.org/html/2607.01774#A6.F22)and[Figure˜23](https://arxiv.org/html/2607.01774#A6.F23)suggest, with just the 1D projection of the mean vectors, we can recover a steering downstream impact similar to the one we attain when performing the steering with the untouched mean vectors \([Equation˜4](https://arxiv.org/html/2607.01774#S4.E4)\)\. Furthermore, as we increase the subspaces’ dimensions from one to two, the results resemble with high precision the phenomena induced by the untouched mean vectors, implying that the 2D parabolas described at[Figure˜8](https://arxiv.org/html/2607.01774#S5.F8), do encode the majority of theτ\\tau\-relevant geometry\. Finally, we observe how the effects caused byΔl,⟂t→t^\\Delta^\{t\\to\\hat\{t\}\}\_\{l,\\perp\}remain of low\-impact and do not follow any clear trend, implying that those dimensions have little relevance to modellingτ\\tau\.

## Appendix EDepth Correction

Here we provide a finer\-grained view of the depth\-correction phenomenon discussed in[Section˜4](https://arxiv.org/html/2607.01774#S4.SSx1)\.[Figure˜16](https://arxiv.org/html/2607.01774#A5.F16)shows the per\-step unrolled version of[Figure˜5](https://arxiv.org/html/2607.01774#S4.F5)for differentt^\\hat\{t\}: instead of averaging the probe driftδ¯l\\bar\{\\delta\}\_\{l\}over denoising steps, we plot one column per step, so that averaging the columns recovers the[Figure˜5](https://arxiv.org/html/2607.01774#S4.F5)curve\. Early denoising steps are markedly more sensitive to the injected perturbation: the absolute probe drift remains substantial across nearly the entire depth of the network, propagating up to layer3131, while at later steps it is suppressed within the first few layers above the injection site\. This is consistent with the sharp peaks observed for shallow\-layer steering \(see[Appendix˜C](https://arxiv.org/html/2607.01774#A3)\), and indicates that the model’s ability to correct an injectedt^\\hat\{t\}depends strongly on the denoising step at which the intervention occurs\. Moreover, the residual drift decreases monotonically as the steered step becomes more compatible witht^=100\\hat\{t\}=100, i\.e\. asttapproaches the end of the denoising trajectory\.

\(a\)t^=1\\hat\{t\}=1![Refer to caption](https://arxiv.org/html/2607.01774v1/x18.png)
\(b\)t^=20\\hat\{t\}=20![Refer to caption](https://arxiv.org/html/2607.01774v1/x19.png)
\(c\)t^=40\\hat\{t\}=40![Refer to caption](https://arxiv.org/html/2607.01774v1/x20.png)
\(d\)t^=100\\hat\{t\}=100![Refer to caption](https://arxiv.org/html/2607.01774v1/x21.png)

Figure 16:Probe drift across layers and denoising steps when steering at layer 6\.Each column shows the absolute driftδ¯l\\bar\{\\delta\}\_\{l\}between clean and steered probe predictions at a given denoising steptt, resolved across probe layers\. We can observe how only fort^=100\\hat\{t\}=100the perturbation at some steps penetrates all across the model and achieves a distortion at the last layers\.
## Appendix FDream’s Mean Activation Analysis

In this section, we present additional results regarding Dream’s mean activations, mirroring our LLaDA analysis from the main text\. Similarly to the behaviour observed in LLaDA, we find that the representations associated withτ\\tauconcentrate in a structured low\-dimensional subspace, exhibiting a smooth and ordered trajectory across denoising time\.

#### Shared 2D geometry\.

[Figure˜17\(a\)](https://arxiv.org/html/2607.01774#A6.F17.sf1)shows the shared 2D geometry obtained by averaging the standardised PCA projections across layers for Dream\. As in LLaDA, the resulting geometry follows a clear parabola\-like trajectory, supporting the idea of a shared geometry ofτ\\tauacross the whole model\. Nonetheless, differently from LLaDA, Dream’s parabola has a bigger spread at the two endpointst=1t=1andt=100t=100, implying that, at the boundaries, the geometrical organisation significantly changes across layers\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x22.png)\(a\)The shared 2\-dimensional geometry oft^\\hat\{t\}in Dream\.We compute an average trajectory by taking the standardised 2D PC projections of the mean vectors across the model\. The parabolic geometry is shared across layers, similarly to LLaDA\. PC1 and PC2 are standardised to unit variance\.
![Refer to caption](https://arxiv.org/html/2607.01774v1/x23.png)\(b\)3D visualisation of the PCA projection of the mean vectors for layer 25 of Dream\.As in LLaDA, we again observe a low\-dimensional ordered structure\. Notably, two points, corresponding tot=1t=1andt=100t=100, lie far from the rest of the geometry, suggesting a special representation\.

Figure 17:Low\-dimensional geometry ofτ\\tau’s mean\-vectors in Dream\.Left: shared 2D mean\-vector trajectory across layers\. Right: 3D PCA projection for layer 25\.
#### 3D geometry\.

[Figure˜17\(b\)](https://arxiv.org/html/2607.01774#A6.F17.sf2)presents a 3D PCA projection of the mean vectors from layer 25 of Dream\. Again, the resulting trajectory reveals a low\-dimensional, ordered structure\. Notably, the representations associated with the extreme denoising steps, namely the initial and final steps, appear as outliers, suggesting that the model may behave qualitatively differently at these extremes, which helps explain the big boundary deviations of[Figure˜17\(a\)](https://arxiv.org/html/2607.01774#A6.F17.sf1)\.

#### Cross\-layer alignment\.

Mirroring our LLaDA analysis in[Figure˜9](https://arxiv.org/html/2607.01774#S5.F9), we investigate how theτ\\taurepresentations align across layers for Dream\. In contrast to LLaDA, which exhibits strong global alignment across most layers,[Figure˜18\(a\)](https://arxiv.org/html/2607.01774#A6.F18.sf1)shows that Dream presents a substantially more heterogeneous organisation\. Interestingly, the model appears to partition its layers into three main computational stages, with high cosine similarity within each block and weaker alignment across blocks\.

#### Within\-layer components\.

We present the cosine similarity across layer components for Dream \(as for LLaDA in[Figure˜10](https://arxiv.org/html/2607.01774#S5.F10)\)\. In[Figure˜18\(b\)](https://arxiv.org/html/2607.01774#A6.F18.sf2)we can appreciate how Dream follows an internal geometric organisation of the mean vectors similar to LLaDA\.

![Refer to caption](https://arxiv.org/html/2607.01774v1/x24.png)\(a\)Average mean vector cosine similarity for Dream\.Dream presents a more orthogonal relationship than LLaDA, indicating that layers are more independent in their representation ofτ\\tau\. We also observe three structured high\-similarity blocks aligned with the model’s computation stages: layers 1–6, 7–20, and 21–27\. Note again the last layer’s orthogonality\.
![Refer to caption](https://arxiv.org/html/2607.01774v1/x25.png)\(b\)Average cosine similarity across layer components in Dream\.The pre\-MLP components remain highly correlated, while the MLP representations split into two groups: the gate path, and the up\-projection together with the Hadamard product\. As in LLaDA, self\-attention and MLP vectors are anticorrelated\.

Figure 18:Cosine\-similarity structure of mean vectors in Dream\.Left: average similarity across layers\. Right: average similarity across intermediate layer components\.![Refer to caption](https://arxiv.org/html/2607.01774v1/x26.png)\(a\)L=64L=64,t^∈\{1,16,32,48,64\}\\hat\{t\}\\in\\\{1,16,32,48,64\\\}\.
![Refer to caption](https://arxiv.org/html/2607.01774v1/x27.png)\(b\)L=128L=128,t^∈\{1,32,64,96,128\}\\hat\{t\}\\in\\\{1,32,64,96,128\\\}\.

Figure 19:Mean\-steering downstream effects across generation lengths for LLaDA\. Ablation over generation lengthL∈\{64,128\}L\\in\\\{64,128\\\}, with the number of denoising steps matched toLL\. Each row shows, left to right,Δ​S¯t\\Delta\\bar\{S\}\_\{t\},Δ​c¯t\\Delta\\bar\{c\}\_\{t\}andKL¯t\\overline\{\\mathrm\{KL\}\}\_\{t\}versus the denoising steptt, for five steering targetst^\\hat\{t\}\. Vertical axes use a symmetric\-log scale to resolve the convergence near 0\.\(a\)Layer=3=3![Refer to caption](https://arxiv.org/html/2607.01774v1/x28.png)
\(b\)Layer=6=6![Refer to caption](https://arxiv.org/html/2607.01774v1/x29.png)
\(c\)Layer=12=12![Refer to caption](https://arxiv.org/html/2607.01774v1/x30.png)
\(d\)Layer=15=15![Refer to caption](https://arxiv.org/html/2607.01774v1/x31.png)
\(e\)Layer=21=21![Refer to caption](https://arxiv.org/html/2607.01774v1/x32.png)
\(f\)Layer=27=27![Refer to caption](https://arxiv.org/html/2607.01774v1/x33.png)

Figure 20:Layer\-wise mean\-steering effects in LLaDA\.We apply mean vector steering at different intervention layers and target denoising\-progress binst^∈\{1,25,50,75,100\}\\hat\{t\}\\in\\\{1,25,50,75,100\\\}\. Each panel reports the downstream effect on entropy drift, confidence drift and KL divergence across denoising steps\.\(a\)Layer=3=3![Refer to caption](https://arxiv.org/html/2607.01774v1/x34.png)
\(b\)Layer=6=6![Refer to caption](https://arxiv.org/html/2607.01774v1/x35.png)
\(c\)Layer=12=12![Refer to caption](https://arxiv.org/html/2607.01774v1/x36.png)
\(d\)Layer=15=15![Refer to caption](https://arxiv.org/html/2607.01774v1/x37.png)
\(e\)Layer=18=18![Refer to caption](https://arxiv.org/html/2607.01774v1/x38.png)
\(f\)Layer=21=21![Refer to caption](https://arxiv.org/html/2607.01774v1/x39.png)

Figure 21:Layer\-wise mean\-steering effects in Dream\.We apply mean vector steering at different intervention layers and target denoising\-progress binst^∈\{1,25,50,75,100\}\\hat\{t\}\\in\\\{1,25,50,75,100\\\}\. Each panel reports the downstream effect on entropy drift, confidence drift and KL divergence across denoising steps\.\(a\)Layer=6=6,k=1k=1![Refer to caption](https://arxiv.org/html/2607.01774v1/x40.png)
\(b\)Layer=6=6,k=10k=10![Refer to caption](https://arxiv.org/html/2607.01774v1/x41.png)
\(c\)Layer=15=15,k=1k=1![Refer to caption](https://arxiv.org/html/2607.01774v1/x42.png)
\(d\)Layer=15=15,k=2k=2![Refer to caption](https://arxiv.org/html/2607.01774v1/x43.png)
\(e\)Layer=15=15,k=10k=10![Refer to caption](https://arxiv.org/html/2607.01774v1/x44.png)
\(f\)Layer=29=29,k=1k=1![Refer to caption](https://arxiv.org/html/2607.01774v1/x45.png)
\(g\)Layer=29=29,k=2k=2![Refer to caption](https://arxiv.org/html/2607.01774v1/x46.png)
\(h\)Layer=29=29,k=10k=10![Refer to caption](https://arxiv.org/html/2607.01774v1/x47.png)

Figure 22:Low\-dimensional steering on LLaDA\.LLaDA’s mean vectors concentrate around a low\-dimensional subspace\. Steering across the top\-1, top\-2, and top\-10 principal components yields results similar to using the unprojected mean vector\.\(a\)Layer=6=6,k=1k=1![Refer to caption](https://arxiv.org/html/2607.01774v1/x48.png)
\(b\)Layer=6=6,k=10k=10![Refer to caption](https://arxiv.org/html/2607.01774v1/x49.png)
\(c\)Layer=15=15,k=1k=1![Refer to caption](https://arxiv.org/html/2607.01774v1/x50.png)
\(d\)Layer=15=15,k=2k=2![Refer to caption](https://arxiv.org/html/2607.01774v1/x51.png)
\(e\)Layer=25=25,k=1k=1![Refer to caption](https://arxiv.org/html/2607.01774v1/x52.png)
\(f\)Layer=25=25,k=2k=2![Refer to caption](https://arxiv.org/html/2607.01774v1/x53.png)
\(g\)Layer=25=25,k=4k=4![Refer to caption](https://arxiv.org/html/2607.01774v1/x54.png)
\(h\)Layer=25=25,k=10k=10![Refer to caption](https://arxiv.org/html/2607.01774v1/x55.png)

Figure 23:Low\-dimensional steering on Dream\.Dream’s mean vectors concentrate around a low\-dimensional subspace as in LLaDA\. Similar to[Figure˜22](https://arxiv.org/html/2607.01774#A6.F22), steering across the top\-1, top\-2, top\-4, and top\-10 principal components yields results similar to using the unprojected mean vector\.