Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Models

# Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Models
Source: [https://arxiv.org/html/2605.10971](https://arxiv.org/html/2605.10971)
Hanhan Zhou∗ AWS AI Labs Santa Clara, CA 95054 hanhanz@amazon\.com &Shamik Roy∗ AWS AI Labs Santa Clara, CA 95054 royshami@amazon\.com Rashmi Gangadharaiah AWS AI Labs Santa Clara, CA 95054 rgangad@amazon\.com

###### Abstract

Discrete diffusion language models \(DLMs\) generate text by iteratively denoising all positions in parallel, offering an alternative to autoregressive models\. Controlled generation methods for DLMs, imported from autoregressive models, apply uniform intervention at every denoising steps\. We show this uniform schedule degrades quality, and the damage compounds when multiple attributes are steered jointly\. To diagnose the failure, we train sparse autoencoders on four DLMs \(124M\-8B parameters\) and find that different attributes commit on distinct schedules, varying in timing, sharpness, and magnitude\. For instance, topic commits within the first 2% of denoising, whereas sentiment emerges gradually over 20% of the process\. Consequently, uniform intervention wastes steering capacity on steps where the target attribute has already solidified or has yet to emerge\. We propose a novel adaptive scheduler that concentrates interventions on the steps where an attribute is actively forming and leaves the rest of generation untouched\. The cost\-control trade\-off admits a closed\-form characterization: the advantage of adaptive over uniform scheduling is governed by a single dispersion statistic of the commitment distribution\. Across four DLMs and seven steering tasks, our method achieves precise control without the degradation typical of uniform interventions\. Especially on challenging simultaneous three\-attribute control, it reaches up to 93% steering strength, beating the strongest baseline by up to 15% points while preserving generation quality\.

\*\*footnotetext:Equal contribution\.## 1Introduction

Discrete diffusion language models \(DLMs\) generate text by iteratively denoising corrupted token sequences and have recently become competitive with autoregressive models\[[2](https://arxiv.org/html/2605.10971#bib.bib7),[32](https://arxiv.org/html/2605.10971#bib.bib2),[20](https://arxiv.org/html/2605.10971#bib.bib3),[42](https://arxiv.org/html/2605.10971#bib.bib4),[25](https://arxiv.org/html/2605.10971#bib.bib6)\]\. Unlike autoregressive generation, where each token is produced once in fixed left\-to\-right order, DLMs refine all positions in parallel across hundreds of steps, creating a temporal trajectory where semantic content is progressively committed\. This raises a fundamental question: how do DLMs organize semantic attributes across denoising, and can that temporal structure be exploited for controlled generation?

Sparse autoencoders \(SAEs\) decompose dense activations into sparse, interpretable features\[[12](https://arxiv.org/html/2605.10971#bib.bib10),[9](https://arxiv.org/html/2605.10971#bib.bib8),[36](https://arxiv.org/html/2605.10971#bib.bib9)\]and have been used to steer autoregressive models\[[18](https://arxiv.org/html/2605.10971#bib.bib15),[43](https://arxiv.org/html/2605.10971#bib.bib20),[10](https://arxiv.org/html/2605.10971#bib.bib16),[1](https://arxiv.org/html/2605.10971#bib.bib14)\]\. Concurrent work explores applying SAEs to DLMs\[[39](https://arxiv.org/html/2605.10971#bib.bib1)\]and steering DLMs via reference\-sequence alignment\[[3](https://arxiv.org/html/2605.10971#bib.bib11)\]\. Other approaches from autoregressive models, including contrastive vectors\[[46](https://arxiv.org/html/2605.10971#bib.bib24),[30](https://arxiv.org/html/2605.10971#bib.bib41)\]and probes\[[6](https://arxiv.org/html/2605.10971#bib.bib42)\], can also be adapted to DLMs\. All existing methods, however, apply the same intervention at every denoising step, treating the trajectory as temporally uniform\. This degrades generation quality in both single\- and multi\-attribute control, inflating perplexity and collapsing diversity, while increasing cross\-attribute interference, the unintended shifting of non\-target attributes\[[26](https://arxiv.org/html/2605.10971#bib.bib25),[34](https://arxiv.org/html/2605.10971#bib.bib26)\], as we verify in §[5](https://arxiv.org/html/2605.10971#S5)\. Yet how attributes form and commit across denoising steps remains underexplored, leaving open whether temporally informed intervention can avoid this quality cost\.

![Refer to caption](https://arxiv.org/html/2605.10971v1/x1.png)

Figure 1:Method overview\.\(1\) SAEs trained per\-layer decompose the residual stream into interpretable features\. Contrastive selection identifies attribute\-relevant feature setsFaℓF\_\{a\}^\{\\ell\}, whose temporal commitment across denoising steps defines an adaptive schedulewdyn​\(t\)w\_\{\\mathrm\{dyn\}\}\(t\)\. \(2\) At each steptt, a sparse contrastive shiftαeff​\(a,t,ℓ\)​δ\(a\)\\alpha\_\{\\mathrm\{eff\}\}\(a,t,\\ell\)\\,\\delta^\{\(a\)\}is applied to selected features and decoded with residual correction, enabling multi\-attribute composition\.We take an interpretability\-first approach to address this gap \(Figure[1](https://arxiv.org/html/2605.10971#S1.F1)\)\. We train SAEs on four DLMs spanning two training objectives, three architectures, and a 60×\\timesscale range \(MDLM\[[32](https://arxiv.org/html/2605.10971#bib.bib2)\], SEDD\[[20](https://arxiv.org/html/2605.10971#bib.bib3)\], DREAM\[[42](https://arxiv.org/html/2605.10971#bib.bib4)\], and LLaDA\[[25](https://arxiv.org/html/2605.10971#bib.bib6)\]\) and use the resulting features to characterize how attributes evolve across the denoising trajectory\. Our analysis \(§[3](https://arxiv.org/html/2605.10971#S3)\) identifies three properties that directly inform steering:*commitment timing*, where attributes commit at different denoising steps;*commitment sharpness*, where emergence may be concentrated in a narrow window or spread across the trajectory; and*effect\-size asymmetry*, where attributes carry different per\-feature discriminative strength\. These findings explain the quality cost of uniform steering: a constant intervention spends budget on steps where the target attribute has already committed or has not yet begun to emerge\. From these empirical profiles, we derive*adaptive steering*\(§[4](https://arxiv.org/html/2605.10971#S4)\), a schedule that concentrates intervention where each attribute is actively forming and leaves the rest of the trajectory undisturbed\. We pose this as a budgeted optimization and show that the optimal schedule allocates intervention proportionally to each step’s gain\-to\-cost ratio \(Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1)\)\. For multi\-attribute composition, we steer through disjoint SAE feature sets and bound cross\-attribute interference via the decoder Gram structure \(Proposition[1](https://arxiv.org/html/2605.10971#Thmproposition1)\)\. Effect\-size asymmetry motivates an effectiveness\-ratio calibration that rebalances per\-attribute strengths so weaker attributes are not drowned out\.

Across four DLMs spanning two training objectives, three architectures, and a 60×\\timesscale range, seven single\- and multi\-attribute conditions, and four baselines \(§[5](https://arxiv.org/html/2605.10971#S5)\), adaptive steering matches or exceeds baseline control while keeping perplexity and diversity close to the unsteered model\. The regimes follow Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1): on models with sharp temporal profiles, Adaptive matches Uniform’s control at substantially lower perplexity \(e\.g\., 50–57 vs\. 59–93 on MDLM\); on models with flat profiles such as LLaDA, the two converge\. On simultaneous steering of 3 attributes: sentiment, topic, and formality, the method reaches 93% geometric\-mean classifier confidence at usable text quality, exceeding the strongest baseline by up to 15 percentage points\.

In summary, we contribute: \(i\) a mechanistic characterization of how semantic attributes emerge during DLM denoising, identifying commitment timing, sharpness, and effect\-size asymmetry as primary axes of variation across attribute, model, and training objective \(§[3](https://arxiv.org/html/2605.10971#S3)\); \(ii\) an adaptive steering framework with closed\-form characterizations of scheduling efficiency \(Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1)\) and multi\-attribute interference \(Proposition[1](https://arxiv.org/html/2605.10971#Thmproposition1)\), where the efficiency result yields a falsifiable prediction about which models benefit \(§[4](https://arxiv.org/html/2605.10971#S4)\); and \(iii\) a comprehensive evaluation across four DLMs demonstrating that adaptive steering achieves strong single\- and multi\-attribute control with lower quality cost and interference than four baselines \(§[5](https://arxiv.org/html/2605.10971#S5)\)\.

## 2Related Work

SAEs have become a standard tool forfeature interpretabilityin autoregressive LMs, linking features to concepts via vocabulary projections\[[12](https://arxiv.org/html/2605.10971#bib.bib10),[9](https://arxiv.org/html/2605.10971#bib.bib8),[36](https://arxiv.org/html/2605.10971#bib.bib9)\], with applications tocausal steeringsuch as detoxification and safety control\[[18](https://arxiv.org/html/2605.10971#bib.bib15),[43](https://arxiv.org/html/2605.10971#bib.bib20),[10](https://arxiv.org/html/2605.10971#bib.bib16),[1](https://arxiv.org/html/2605.10971#bib.bib14)\], though effectiveness can be sensitive to feature selection and layer choice\[[31](https://arxiv.org/html/2605.10971#bib.bib21),[5](https://arxiv.org/html/2605.10971#bib.bib22)\]\. For DLMs, concurrent work has begun exploring interpretability and control\. DLM\-Scope\[[39](https://arxiv.org/html/2605.10971#bib.bib1)\]trains SAEs on DREAM and LLaDA and studiestemporal dynamicsof decoding order \(how representations evolve across denoising\), but does not perform across\-objectivecomparison \(analysis across training losses on identical architecture/data\) or demonstratecompositional control\(steering multiple attributes simultaneously\)\. ILRR\[[3](https://arxiv.org/html/2605.10971#bib.bib11)\]steers MDLM and LLaDA via reference\-sequence alignment without feature\-level interpretability or composability\. Other efforts study AR vs\. DLM representations\[[16](https://arxiv.org/html/2605.10971#bib.bib12)\]and temporal attention for hallucination detection\[[19](https://arxiv.org/html/2605.10971#bib.bib13)\]\. Multi\-attribute control in AR models requires specialized methods\[[26](https://arxiv.org/html/2605.10971#bib.bib25),[34](https://arxiv.org/html/2605.10971#bib.bib26)\]; we show that SAE feature disjointness in DLMs enables additive composition\.Multi\-scaleuniversality studies have compared feature overlap across sizes\[[35](https://arxiv.org/html/2605.10971#bib.bib28)\]and architectures\[[37](https://arxiv.org/html/2605.10971#bib.bib27)\], but not the training objective as a causal variable\. We train SAEs on four DLMs spanning two objectives and a 60×\\timesscale range, uncover temporal dynamics that explain when attributes become steerable, and leverage these findings for adaptive compositional control evaluated against multiplebaselines\(Table[1](https://arxiv.org/html/2605.10971#S2.T1)\)\.

Table 1:Comparison of our work with concurrent efforts\. ✓ = fully,∼\\sim= partially, ✗ = not addressed\.WorkModelsApproachFeatureInterpretabilityCausalSteeringTemporalDynamicsCross\-ObjectiveCompositionalControlMulti\-ScaleBaselinesDLM\-Scope\[[39](https://arxiv.org/html/2605.10971#bib.bib1)\]Dream, LLaDASAE \(Top\-K, 16K\)∼\\sim✓✓✗✗✗∼\\simILRR\[[3](https://arxiv.org/html/2605.10971#bib.bib11)\]MDLM, LLaDAReference\-based✗✓∼\\sim✗✗∼\\sim✓Skip\-to\-Good\-Part\[[16](https://arxiv.org/html/2605.10971#bib.bib12)\]LLaDA, DreamProbing / RSA✗✗∼\\sim∼\\sim✗✗✗TDGNet\[[19](https://arxiv.org/html/2605.10971#bib.bib13)\]LLaDA, DreamTemporal graphs✗✗✓✗✗✗✗OursMDLM, SEDD,Dream, LLaDASAE \(Top\-K, 12–16K\)✓✓✓✓✓✓✓

## 3Interpreting Discrete Diffusion LMs via Sparse Autoencoders

Sparse autoencoders \(SAEs\) decompose dense activations into overcomplete dictionaries of interpretable features\[[12](https://arxiv.org/html/2605.10971#bib.bib10),[9](https://arxiv.org/html/2605.10971#bib.bib8),[36](https://arxiv.org/html/2605.10971#bib.bib9)\]and have been used to steer autoregressive LMs\[[18](https://arxiv.org/html/2605.10971#bib.bib15),[43](https://arxiv.org/html/2605.10971#bib.bib20),[10](https://arxiv.org/html/2605.10971#bib.bib16),[1](https://arxiv.org/html/2605.10971#bib.bib14)\]\. Whether SAEs can meaningfully decompose DLM representations remains understudied\. DLMs offer a structural advantage that AR models lack: because all positions are refined across a temporal trajectory, semantic features*emerge*progressively and different attributes may commit at different denoising steps, creating natural intervention points\. For multi\-attribute control, attributes may interact through temporal overlap, effect\-size imbalance, or shared feature subspaces, making it essential to characterize these dynamics\. In this section, we train SAEs on four DLMs spanning different objectives, architectures, and scales, and study how features encoding sentiment, topic, and style evolve and interact\.

### 3\.1Training SAEs on Discrete Diffusion Models

We train TopK sparse autoencoders\[[12](https://arxiv.org/html/2605.10971#bib.bib10),[45](https://arxiv.org/html/2605.10971#bib.bib30)\]on residual\-stream activations from MDLM\[[32](https://arxiv.org/html/2605.10971#bib.bib2)\], SEDD\[[20](https://arxiv.org/html/2605.10971#bib.bib3)\], LLaDA\[[25](https://arxiv.org/html/2605.10971#bib.bib6)\], and DREAM\[[42](https://arxiv.org/html/2605.10971#bib.bib4)\]\. The training layers are selected via a diffusion logits lens \(§[C\.2](https://arxiv.org/html/2605.10971#A3.SS2)\) and a layer probing study \(§[C\.3\.2](https://arxiv.org/html/2605.10971#A3.SS3.SSS2)\), which together reveal when and where token identities crystallize during generation and identify the layers most suitable for SAE training in DLMs\. Given a hidden state𝐱∈ℝdmodel\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}, the SAE encodes it into an overcomplete latent space \(dSAE≫dmodeld\_\{\\text\{SAE\}\}\\gg d\_\{\\text\{model\}\}\), applies aTopK\\mathrm\{TopK\}activation to retain only thekklargest entries as a sparse code𝐡\\mathbf\{h\}, and reconstructs via a learned decoder:𝐡=TopK​\(\(𝐱−𝐛dec\)​𝐖enc\+𝐛enc,k\),𝐱^=𝐡​𝐖dec\+𝐛dec\.\\mathbf\{h\}=\\mathrm\{TopK\}\\\!\\bigl\(\(\\mathbf\{x\}\-\\mathbf\{b\}\_\{\\text\{dec\}\}\)\\,\\mathbf\{W\}\_\{\\text\{enc\}\}\+\\mathbf\{b\}\_\{\\text\{enc\}\},\\;k\\bigr\),\\;\\hat\{\\mathbf\{x\}\}=\\mathbf\{h\}\\,\\mathbf\{W\}\_\{\\text\{dec\}\}\+\\mathbf\{b\}\_\{\\text\{dec\}\}\.Each nonzero entryhjh\_\{j\}corresponds to a*feature*whose decoder direction𝐰jdec\\mathbf\{w\}\_\{j\}^\{\\text\{dec\}\}\(thejj\-th column of𝐖dec\\mathbf\{W\}\_\{\\text\{dec\}\}\) is the direction it adds to the residual stream\. Training minimizes MSE reconstruction loss with an auxiliary dead\-neuron loss \(§[C](https://arxiv.org/html/2605.10971#A3)\)\. Unlike AR models, DLM activations vary with masking rate, so we train on activations sampled across uniformly distributed masking rates to ensure generalization across the full denoising trajectory\. Table[2](https://arxiv.org/html/2605.10971#S3.T2)summarizes all models and SAE configurations\.

Table 2:Model and SAE summary\. All SAEs use TopK activation withk=32k\{=\}32\.ModelParamsArch\.Diffusion LossSAE Layersdmodeld\_\{\\text\{model\}\}dSAEd\_\{\\text\{SAE\}\}dSAE/dmodeld\_\{\\text\{SAE\}\}/d\_\{\\text\{model\}\}MDLM\[[32](https://arxiv.org/html/2605.10971#bib.bib2)\]124MGPT\-2\[[28](https://arxiv.org/html/2605.10971#bib.bib56)\]Absorbing5, 6, 776812,28816×\\timesSEDD\[[20](https://arxiv.org/html/2605.10971#bib.bib3)\]124MGPT\-2Score\-entropy5, 6, 776812,28816×\\timesLLaDA\[[25](https://arxiv.org/html/2605.10971#bib.bib6)\]8BLLaMA\[[38](https://arxiv.org/html/2605.10971#bib.bib58)\]Absorbing8, 14, 20, 264,09616,3844×\\timesDREAM\[[42](https://arxiv.org/html/2605.10971#bib.bib4)\]7BQwen2\[[41](https://arxiv.org/html/2605.10971#bib.bib57)\]Absorbing8, 13, 17, 233,58414,3364×\\times

### 3\.2Extracting Attribute Features from SAEs

We study three attributes:sentiment,topic, andstyle, common controllability targets and a natural combination for studying multi\-attribute interplay, as they capture orthogonal aspects of text: subject matter, opinion polarity, and writing register\. Following work on contrastive steering directions\[[46](https://arxiv.org/html/2605.10971#bib.bib24),[30](https://arxiv.org/html/2605.10971#bib.bib41),[6](https://arxiv.org/html/2605.10971#bib.bib42)\], we construct balanced binary corpora: Sports vs\. Business from AG News\[[44](https://arxiv.org/html/2605.10971#bib.bib40)\]for topic, positive vs\. negative IMDB reviews\[[22](https://arxiv.org/html/2605.10971#bib.bib39)\]for sentiment, and formal vs\. informal text for style, using the Pavlick corpus\[[27](https://arxiv.org/html/2605.10971#bib.bib43)\]for MDLM/SEDD and classifier\-labeled IMDB reviews\[[4](https://arxiv.org/html/2605.10971#bib.bib47)\]for LLaDA/DREAM to match their generation domain \(§[C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1)\)\. For style, we track formality features on MDLM/SEDD, whose baseline formality is near chance \(∼\{\\sim\}53–58%\), and informality features on LLaDA/DREAM, whose generations are already highly formal \(∼\{\\sim\}74–78%; §[5](https://arxiv.org/html/2605.10971#S5)\), so that the features we study correspond to the direction with greater room for control in later steering studies \(§[4](https://arxiv.org/html/2605.10971#S4)\)\. To identify attribute\-encoding features, we noise each text at multiple mask ratios, pass the noised inputs through the model, and ReLU\-encode the hidden states through the SAE at each layer \(using ReLU rather than TopK to avoid biasing effect\-size estimates; §[C\.5\.2](https://arxiv.org/html/2605.10971#A3.SS5.SSS2)\)\. For each featurejj, we compute the Cohen’sddeffect size\[[11](https://arxiv.org/html/2605.10971#bib.bib49)\]:dj=\(h¯j\+−h¯j−\)/\(Var​\(hj\+\)\+Var​\(hj−\)\)/2d\_\{j\}=\(\\bar\{h\}\_\{j\}^\{\+\}\-\\bar\{h\}\_\{j\}^\{\-\}\)/\\sqrt\{\(\\text\{Var\}\(h\_\{j\}^\{\+\}\)\+\\text\{Var\}\(h\_\{j\}^\{\-\}\)\)/2\}, whereh¯j\+\\bar\{h\}\_\{j\}^\{\+\}andh¯j−\\bar\{h\}\_\{j\}^\{\-\}are the mean activations for the two contrastive classes\. We retain the top 50 features per direction per layer that pass a Mann–WhitneyUUtest\[[24](https://arxiv.org/html/2605.10971#bib.bib50)\]\(p<0\.01p<0\.01, Bonferroni\-corrected\[[8](https://arxiv.org/html/2605.10971#bib.bib51)\]\) for the interpretability analysis below\. Vocabulary grounding \(projecting decoder columns through𝐖unembed\\mathbf\{W\}\_\{\\text\{unembed\}\}\) confirms that retained features are semantically coherent \(§[B](https://arxiv.org/html/2605.10971#A2)\)\.

### 3\.3Interpreting Temporal Dynamics of Features

To analyze feature dynamics, we generateNNsamples per model with SAE hooks active at every step, record per\-step feature activations, filter to classifier\-confident samples using off\-the\-shelf attribute classifiers \(since not all generated texts express the target attribute; §[C\.6](https://arxiv.org/html/2605.10971#A3.SS6)\), and partition the trajectory intoBBtemporal blocks\. We quantify emergence timing via*block fractions*, the share of total non\-negative activation change in each block:fb=max⁡\(0,h¯eb−h¯sb\)/∑b′max⁡\(0,h¯eb′−h¯sb′\)f\_\{b\}=\{\\max\(0,\\bar\{h\}\_\{e\_\{b\}\}\{\-\}\\bar\{h\}\_\{s\_\{b\}\}\)\}\\big/\{\\sum\_\{b^\{\\prime\}\}\\max\(0,\\bar\{h\}\_\{e\_\{b^\{\\prime\}\}\}\{\-\}\\bar\{h\}\_\{s\_\{b^\{\\prime\}\}\}\)\}, wheresbs\_\{b\}andebe\_\{b\}are the start and end steps of blockbb\. We summarize our key observations below\.

Commitment timing: attributes commit at different denoising steps\.Attribute emergence timing varies by attribute, model, and layer\. For topic \(Figure[2\(a\)](https://arxiv.org/html/2605.10971#S3.F2.sf1)\), MDLM and SEDD commit within the first few percent of denoising, with deeper layers faster; LLaDA shows gradual trajectories; DREAM defers emergence to the final quarter, likely due to its AR\-based initialization which inherits late\-trajectory token\-finalization behavior\. Figure[2\(c\)](https://arxiv.org/html/2605.10971#S3.F2.sf3)reportstc​\(p\)t\_\{c\}\(p\)\(trajectory percentage at whichpp% of emergence has occurred\) at the deepest layer per model\. The patterns differ strikingly: on MDLM, topic reaches 50% emergence attc=22t\_\{c\}\{=\}22\(2\.1% of trajectory\) while sentiment reaches it attc=216t\_\{c\}\{=\}216\(21%\), a∼10×\{\\sim\}10\\timesgap\. Emergence trajectories for sentiment and style are in §[D\.1](https://arxiv.org/html/2605.10971#A4.SS1)\.

Commitment sharpness: models differ in how concentrated emergence is\.Figure[2\(b\)](https://arxiv.org/html/2605.10971#S3.F2.sf2)quantifies emergence concentration via block fractions at the deepest layer\. On MDLM, block 0 captures 100% of topic emergence but only 31% of sentiment, which is distributed across later blocks\. SEDD mirrors MDLM’s early topic commitment \(same architecture\)\. LLaDA shows front\-loading in block 0 ranging from∼\{\\sim\}20% \(topic, style\) to∼\{\\sim\}80% \(sentiment\)\. DREAM concentrates*all*emergence in the final two blocks, with blocks 0–5 contributing zero\. Block fraction analysis for other layers is in §[D\.2](https://arxiv.org/html/2605.10971#A4.SS2)\. Commitment timing and sharpness together define each model’s temporal profile; the diversity of these profiles across models, layers, and attributes means that*no single fixed schedule can be optimal for controllability*, motivating adaptive steering \(§[4\.1](https://arxiv.org/html/2605.10971#S4.SS1)\)\.

![Refer to caption](https://arxiv.org/html/2605.10971v1/x2.png)\(a\)Emergence trajectories![Refer to caption](https://arxiv.org/html/2605.10971v1/x3.png)\(b\)Block fractions![Refer to caption](https://arxiv.org/html/2605.10971v1/x4.png)\(c\)Commitment times
Figure 2:Temporal hierarchy of feature emergence\.\(a\)Topic emergence \(normalized\|a​\(t\)−a​\(0\)\|\|a\(t\)\-a\(0\)\|\) across layers\.\(b\)Block fractions \(K=8K\{=\}8\) at deepest layer\.\(c\)Commitment timetc​\(p\)t\_\{c\}\(p\): % of trajectory at whichpp% of emergence has occurred \(deepest layer, all attributes\)\. Green = early; red = late\.![Refer to caption](https://arxiv.org/html/2605.10971v1/x5.png)\(a\)Cumulative effect size![Refer to caption](https://arxiv.org/html/2605.10971v1/x6.png)\(b\)Total effect size![Refer to caption](https://arxiv.org/html/2605.10971v1/x7.png)\(c\)Effect spread![Refer to caption](https://arxiv.org/html/2605.10971v1/x8.png)\(d\)Cross\-model alignment
Figure 3:Training objective shapes feature geometry\.\(a–c\)MDLM vs\. SEDD \(layer 7, Cohen’sd\>0d\>0features\):\(a\)cumulative Cohen’sddvs\. rank;\(b\)total∑d\\sum d;\(c\)effective dimensionality\.\(d\)Cross\-model alignment: sharing a loss family yields higher similarity than sharing architecture\.Effect\-size asymmetry: attributes differ in per\-feature discriminative strength\.Topic features carry2−3×2\{\-\}3\\timeslarger per\-feature Cohen’s\|d\|\|d\|than sentiment across all models \(Figure[3\(a\)](https://arxiv.org/html/2605.10971#S3.F3.sf1)\), meaning equal\-strength interventions over\-steer topic and under\-steer sentiment, motivating effectiveness\-ratio calibration \(§[4\.1](https://arxiv.org/html/2605.10971#S4.SS1)\)\. Comparing MDLM and SEDD \(same architecture and data, different loss\) at layer 7 reveals further structure \(Figure[3](https://arxiv.org/html/2605.10971#S3.F3)\): SEDD carries2\.1×2\.1\\times\(sentiment\) and3\.4×3\.4\\times\(formality\) more total signal \(∑d\\sum d, Figure[3\(b\)](https://arxiv.org/html/2605.10971#S3.F3.sf2)\), while topic is comparable \(0\.8×0\.8\\times\)\. The effective dimensionality \(\(∑d\)2/∑d2\(\\sum d\)^\{2\}/\\sum d^\{2\}; Figure[3\(c\)](https://arxiv.org/html/2605.10971#S3.F3.sf3)\) shows SEDD spreads sentiment across2\.6×2\.6\\timesand formality across2\.2×2\.2\\timesmore features; topic is more concentrated\. We examine the effect on steering in §[5](https://arxiv.org/html/2605.10971#S5)\.

Training objective shapes feature geometry\.Cross\-model alignment in vocabulary space \(details in §[D\.5](https://arxiv.org/html/2605.10971#A4.SS5)\) confirms the training objective as the dominant factor \(Figure[3\(d\)](https://arxiv.org/html/2605.10971#S3.F3.sf4)\): MDLM and SEDD yield near\-orthogonal features \(cosine sim\. 0\.08\), while MDLM and DREAM \(different architecture,56×56\\timesscale gap, same loss family\) achieve 0\.44 mean and 0\.88 max similarity\.

We additionally analyze whether features encode attributes*anticipatorily*\(firing on still\-masked positions\) or*reactively*\(responding to visible tokens\): topic is anticipatory on MDLM with depth\-dependent strength; SEDD also shows compressed but qualitatively similar patterns; DREAM shows minimal differentiation until late denoising \(§[D\.4](https://arxiv.org/html/2605.10971#A4.SS4)\)\. Together, these findings motivate a steering framework that adapts to each model’s temporal structure, as developed in §[4](https://arxiv.org/html/2605.10971#S4)\.

## 4Interpretability Informed Steering

The analysis in §[3](https://arxiv.org/html/2605.10971#S3)reveals that attributes commit at different steps with different sharpness, carry different per\-feature discriminative strengths, and encode anticipatorily\. We exploit this by steering in SAE feature space, modulating intervention strength across denoising steps and attributes\.

Setup\.At denoising stepttand layerℓ\\ell, we encodeht=Enc​\(xt\)h\_\{t\}=\\mathrm\{Enc\}\(x\_\{t\}\), reconstructx^t=Dec​\(ht\)\\hat\{x\}\_\{t\}=\\mathrm\{Dec\}\(h\_\{t\}\), and modify features on the attribute\-specific setFaℓF\_\{a\}^\{\\ell\}:

h~t,j=ht,j\+αeff​\(a,t,ℓ\)​δj\(a\)\(j∈Faℓ\),\\tilde\{h\}\_\{t,j\}\\;=\\;h\_\{t,j\}\+\\alpha\_\{\\mathrm\{eff\}\}\(a,t,\\ell\)\\,\\delta\_\{j\}^\{\(a\)\}\\qquad\(j\\in F\_\{a\}^\{\\ell\}\),\(1\)whereδj\(a\)=h¯j,\+\(a\)−h¯j,−\(a\)\\delta\_\{j\}^\{\(a\)\}=\\bar\{h\}\_\{j,\+\}^\{\(a\)\}\-\\bar\{h\}\_\{j,\-\}^\{\(a\)\}is the contrastive shift, signed so that positiveαeff\\alpha\_\{\\mathrm\{eff\}\}steers toward the target pole\. The forwarded residual isxt′=xt−x^t\+x~tx^\{\\prime\}\_\{t\}=x\_\{t\}\-\\hat\{x\}\_\{t\}\+\\tilde\{x\}\_\{t\}withx~t=Dec​\(h~t\)\\tilde\{x\}\_\{t\}=\\mathrm\{Dec\}\(\\tilde\{h\}\_\{t\}\); the off\-basis termxt−x^tx\_\{t\}\-\\hat\{x\}\_\{t\}preserves SAE\-residual information, and the affine decoder givesxt′−xt=WFaℓ​\(αeff​δ\(a\)\)x^\{\\prime\}\_\{t\}\-x\_\{t\}=W\_\{F\_\{a\}^\{\\ell\}\}\(\\alpha\_\{\\mathrm\{eff\}\}\\,\\delta^\{\(a\)\}\), so the intervention acts only through the selected coordinates\.

### 4\.1Steering Configurations

We decompose the effective steering strength into three components:

αeff​\(a,t,ℓ\)=α⏟overall scale⋅r​\(a\)⏟attribute weight⋅wdyn​\(t;a,ℓ\)⏟temporal weight\.\\alpha\_\{\\mathrm\{eff\}\}\(a,t,\\ell\)=\\underbrace\{\\alpha\}\_\{\\text\{overall scale\}\}\\cdot\\underbrace\{r\(a\)\}\_\{\\text\{attribute weight\}\}\\cdot\\underbrace\{w\_\{\\mathrm\{dyn\}\}\(t;a,\\ell\)\}\_\{\\text\{temporal weight\}\}\.\(2\)The scalarα\\alphacontrols the overall steering magnitude \(swept during evaluation\);r​\(a\)∈\(0,1\]r\(a\)\\in\(0,1\]rebalances strength across attributes; andwdyn​\(t;a,ℓ\)∈\[0,1\]w\_\{\\text\{dyn\}\}\(t;a,\\ell\)\\in\[0,1\]modulates strength across denoising steps according to each attribute’s empirical commitment profile\.

For multiple attributes𝒜=\{a1,…,am\}\\mathcal\{A\}=\\\{a\_\{1\},\\dots,a\_\{m\}\\\}, we compose interventions additively in feature space:

h~t,j=ReLU​\(ht,j\+∑i:j∈Faiℓαeff​\(ai,t,ℓ\)​δj\(ai\)\),\\tilde\{h\}\_\{t,j\}\\;=\\;\\mathrm\{ReLU\}\\\!\\left\(h\_\{t,j\}\+\\sum\_\{i:\\,j\\in F\_\{a\_\{i\}\}^\{\\ell\}\}\\alpha\_\{\\mathrm\{eff\}\}\(a\_\{i\},t,\\ell\)\\,\\delta\_\{j\}^\{\(a\_\{i\}\)\}\\right\),\(3\)with steered residualxt′=xt−x^t\+x~tx^\{\\prime\}\_\{t\}=x\_\{t\}\-\\hat\{x\}\_\{t\}\+\\tilde\{x\}\_\{t\}\. The ReLU clamp enforces the non\-negative activation domain of the TopK SAE \(a singleδ\(a\)\\delta^\{\(a\)\}at moderateαeff\\alpha\_\{\\mathrm\{eff\}\}does not drive coordinates negative, but a sum of updates can\)\. This formulation is effective because the contrastive selection procedure \(§[3\.2](https://arxiv.org/html/2605.10971#S3.SS2)\) yields near\-disjoint feature sets across attributes\. Different choices ofwdynw\_\{\\mathrm\{dyn\}\}andrryield four steering configurations of increasing sophistication, each incorporating a successive interpretability finding\.

Uniform \(r=1r\{=\}1,wdyn=1w\_\{\\mathrm\{dyn\}\}\{=\}1\)\.Constantα\\alphaat every step\. This ignores the temporal structure uncovered in §[3\.3](https://arxiv.org/html/2605.10971#S3.SS3): different attributes commit at different denoising steps \(e\.g\., topic within 2% while sentiment emerges over 20%\+; Figure[2\(c\)](https://arxiv.org/html/2605.10971#S3.F2.sf3)\), so a uniform schedule over\-steers already\-committed attributes and under\-steers those still emerging\.

Adaptive \(r=1r\{=\}1\)\.Directly operationalizes commitment timing and sharpness \(Figures[2\(a\)](https://arxiv.org/html/2605.10971#S3.F2.sf1)–[2\(b\)](https://arxiv.org/html/2605.10971#S3.F2.sf2)\): intervention is concentrated during active emergence and relaxed elsewhere\. We partition theTTdenoising steps intoBBblocks and measure the*commitment fraction*, the share of total feature activation growth in each block:fb\(a,ℓ\)=∑j∈Faℓmax⁡\(0,h¯j\(sbend\)−h¯j\(sbstart\)\)∑b′∑j∈Faℓmax⁡\(0,h¯j\(sb′end\)−h¯j\(sb′start\)\),f\_\{b\}^\{\(a,\\ell\)\}\\;=\\;\\frac\{\\sum\_\{j\\in F\_\{a\}^\{\\ell\}\}\\max\\\!\\big\(0,\\,\\bar\{h\}\_\{j\}^\{\(s\_\{b\}^\{\\mathrm\{end\}\}\)\}\-\\bar\{h\}\_\{j\}^\{\(s\_\{b\}^\{\\mathrm\{start\}\}\)\}\\big\)\}\{\\sum\_\{b^\{\\prime\}\}\\sum\_\{j\\in F\_\{a\}^\{\\ell\}\}\\max\\\!\\big\(0,\\,\\bar\{h\}\_\{j\}^\{\(s\_\{b^\{\\prime\}\}^\{\\mathrm\{end\}\}\)\}\-\\bar\{h\}\_\{j\}^\{\(s\_\{b^\{\\prime\}\}^\{\\mathrm\{start\}\}\)\}\\big\)\},whereh¯j\(t\)\\bar\{h\}\_\{j\}^\{\(t\)\}is the mean activation of featurejjat steptt\(averaged over the contrastive corpus\) and negative differences are clamped to zero, capturing net emergence only\. The adaptive weight for all stepsttwithin blockbbis

wdyn​\(t;a,ℓ\)=fb\(a,ℓ\)maxb′⁡fb′\(a,ℓ\),w\_\{\\mathrm\{dyn\}\}\(t;a,\\ell\)\\;=\\;\\frac\{f\_\{b\}^\{\(a,\\ell\)\}\}\{\\max\_\{b^\{\\prime\}\}f\_\{b^\{\\prime\}\}^\{\(a,\\ell\)\}\},\(4\)with the peak\-emergence block receiving weight 1\. When emergence is sharply concentrated \(e\.g\., topic on MDLM, where block 0 captures 100%; §[3\.3](https://arxiv.org/html/2605.10971#S3.SS3)\),wdyn≈1w\_\{\\mathrm\{dyn\}\}\\approx 1early and≈0\{\\approx\}\\,0thereafter; when emergence is spread evenly \(e\.g\., LLaDA\),wdyn≈1w\_\{\\mathrm\{dyn\}\}\\approx 1everywhere and Adaptive degenerates to Uniform, yielding a testable prediction confirmed in §[5](https://arxiv.org/html/2605.10971#S5)\.

Uniform\+E and Adaptive\+E\.Addresses effect\-size asymmetry \(Figure[3\(a\)](https://arxiv.org/html/2605.10971#S3.F3.sf1)\): attributes carry different per\-feature Cohen’sddmagnitudes, so the sameα\\alphaproduces disproportionate perturbations \(e\.g\., topic features carry2−3×2\{\-\}3\\timeslarger\|d\|\|d\|than sentiment; §[3\.3](https://arxiv.org/html/2605.10971#S3.SS3)\), causing stronger attributes to saturate before weaker ones reach target\. We calibrate from single\-attribute dose\-response sweeps: for each attributeaa, letαa∗\\alpha^\{\*\}\_\{a\}be the minimum strength reaching a reference control threshold\. The effectiveness ratio isr​\(a\)=αa∗/maxa′⁡αa′∗r\(a\)=\\alpha^\{\*\}\_\{a\}\\big/\\max\_\{a^\{\\prime\}\}\\alpha^\{\*\}\_\{a^\{\\prime\}\}, so the hardest\-to\-steer attribute receivesr=1r\{=\}1and easier attributes are scaled down\. We calibrate empirically rather than derivingr​\(a\)r\(a\)directly from Cohen’sddratios because the relationship between feature\-level effect size and classifier response is nonlinear and model\-dependent\. This normalizes the effect ofα\\alphaacross attributes, allowing a single global budget without attribute\-specific imbalances\. Uniform\+E setswdyn=1w\_\{\\mathrm\{dyn\}\}\{=\}1; Adaptive\+E uses both components\.

### 4\.2Theoretical Analysis

We derive the optimal temporal allocation in closed form and characterize the adaptive\-vs\-uniform gain, then bound cross\-attribute interference from the additive composition\.

Trajectory\-level cost decomposition\.The steered and unsteered reverse processes are Markov chains with common initial distribution, so the chain rule givesKL\(p~θ∥pθ\)=∑t𝔼xt\[KL\(p~θ\(xt−1\|xt\)∥pθ\(xt−1\|xt\)\)\]\\mathrm\{KL\}\(\\tilde\{p\}\_\{\\theta\}\\\|p\_\{\\theta\}\)=\\sum\_\{t\}\\mathbb\{E\}\_\{x\_\{t\}\}\[\\mathrm\{KL\}\(\\tilde\{p\}\_\{\\theta\}\(x\_\{t\-1\}\|x\_\{t\}\)\\\|p\_\{\\theta\}\(x\_\{t\-1\}\|x\_\{t\}\)\)\]\. Writing the per\-step gain asΔ​gt=αt​st\+o​\(αt\)\\Delta g\_\{t\}=\\alpha\_\{t\}s\_\{t\}\+o\(\\alpha\_\{t\}\)\(st≥0s\_\{t\}\\geq 0\) withΔ​qt​\(αt\)=12​ct​αt2\+o​\(αt2\)\\Delta q\_\{t\}\(\\alpha\_\{t\}\)=\\tfrac\{1\}\{2\}c\_\{t\}\\alpha\_\{t\}^\{2\}\+o\(\\alpha\_\{t\}^\{2\}\), the allocation problem is then

minαt≥0⁡12​∑t=0T−1ct​αt2s\.t\.∑t=0T−1αt​st≥Etarget\.\\min\_\{\\alpha\_\{t\}\\geq 0\}\\;\\tfrac\{1\}\{2\}\\sum\_\{t=0\}^\{T\-1\}c\_\{t\}\\alpha\_\{t\}^\{2\}\\quad\\text\{s\.t\.\}\\quad\\sum\_\{t=0\}^\{T\-1\}\\alpha\_\{t\}s\_\{t\}\\;\\geq\\;E\_\{\\mathrm\{target\}\}\.\(5\)
Adaptive vs\. uniform scheduling\.The KKT conditions for \([5](https://arxiv.org/html/2605.10971#S4.E5)\) yieldαt⋆∝st/ct\\alpha\_\{t\}^\{\\star\}\\propto s\_\{t\}/c\_\{t\}\(§[A\.1](https://arxiv.org/html/2605.10971#A1.SS1)\)\. We compare against the uniform allocationαt≡α\\alpha\_\{t\}\\equiv\\alpha\.

###### Theorem 1\(Adaptive\-vs\-uniform efficiency ratio\)\.

Assumest≥0s\_\{t\}\\geq 0,ct\>0c\_\{t\}\>0, andEtarget\>0E\_\{\\mathrm\{target\}\}\>0\. Among schedules with12​∑tct​αt2=B\\tfrac\{1\}\{2\}\\sum\_\{t\}c\_\{t\}\\alpha\_\{t\}^\{2\}=B, letE⋆E^\{\\star\}andEunifE\_\{\\mathrm\{unif\}\}denote the maximum attribute shifts under the optimal and uniform schedules\. Then

ρ2:=\(E⋆Eunif\)2=\(∑tst2/ct\)​\(∑tct\)\(∑tst\)2=1\+CVc2​\(s/c\),\\rho^\{2\}\\;:=\\;\\left\(\\frac\{E^\{\\star\}\}\{E\_\{\\mathrm\{unif\}\}\}\\right\)^\{2\}\\;=\\;\\frac\{\\big\(\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\\big\)\\big\(\\sum\_\{t\}c\_\{t\}\\big\)\}\{\\big\(\\sum\_\{t\}s\_\{t\}\\big\)^\{2\}\}\\;=\\;1\\;\+\\;\\mathrm\{CV\}\_\{c\}^\{2\}\(s/c\),\(6\)whereCVc\\mathrm\{CV\}\_\{c\}is the coefficient of variation ofst/cts\_\{t\}/c\_\{t\}under the cost\-weighted measurect/∑t′ct′c\_\{t\}/\\sum\_\{t^\{\\prime\}\}c\_\{t^\{\\prime\}\}, with equalityρ=1\\rho=1iffst/cts\_\{t\}/c\_\{t\}is constant intt\. \(See §[A\.1](https://arxiv.org/html/2605.10971#A1.SS1)for assumptions and full proof\.\)

Interpretation\.The adaptive gain is governed by cost\-weighted dispersion ofst/cts\_\{t\}/c\_\{t\}: large when the attribute commits in a few steps, near one when emergence is spread evenly\. For any heuristic schedule,E/E⋆=cos⁡θE/E^\{\\star\}=\\cos\\thetawhereθ\\thetais the angle between\(αt​ct\)t\(\\alpha\_\{t\}\\sqrt\{c\_\{t\}\}\)\_\{t\}and\(st/ct\)t\(s\_\{t\}/\\sqrt\{c\_\{t\}\}\)\_\{t\}\(Corollary[2](https://arxiv.org/html/2605.10971#Thmcorollary2), §[A\.1\.3](https://arxiv.org/html/2605.10971#A1.SS1.SSS3)\); equivalently,B/B⋆=1/cos2⁡θB/B^\{\\star\}=1/\\cos^\{2\}\\thetaat matched shift\.

Proxy fidelity\.Neithersts\_\{t\}norctc\_\{t\}is directly observable\. The proxywdynw\_\{\\mathrm\{dyn\}\}approximatesαt⋆∝st/ct\\alpha\_\{t\}^\{\\star\}\\propto s\_\{t\}/c\_\{t\}under two conditions \(§[A\.1\.6](https://arxiv.org/html/2605.10971#A1.SS1.SSS6)\): \(i\) active\-set growth tracks the marginal sensitivitysts\_\{t\}, and \(ii\)ctc\_\{t\}varies slowly relative tosts\_\{t\}\. The resulting efficiency isEproxy/E⋆=cos⁡θproxyE\_\{\\mathrm\{proxy\}\}/E^\{\\star\}=\\cos\\theta\_\{\\mathrm\{proxy\}\}\(Corollary[3](https://arxiv.org/html/2605.10971#Thmcorollary3), §[A\.1\.3](https://arxiv.org/html/2605.10971#A1.SS1.SSS3)\)\. Moreover,sts\_\{t\}decomposes over positions with unmasked contributions vanishing \(§[A\.1\.5](https://arxiv.org/html/2605.10971#A1.SS1.SSS5)\), so anticipatory features \(§[D\.4](https://arxiv.org/html/2605.10971#A4.SS4)\) drive the effective gain; since they contribute disproportionately to activation growth in early blocks, the temporal proxy implicitly concentrates intervention where it has the most effect\. §[5](https://arxiv.org/html/2605.10971#S5)confirms the predicted regimes: a substantial gap on MDLM and none on LLaDA\.

Decoder\-space interference bound\.To analyze cross\-attribute interference from \([3](https://arxiv.org/html/2605.10971#S4.E3)\), letΔ​xt\(i\)\\Delta x\_\{t\}^\{\(i\)\}denote the residual\-space contribution of attributeii\.

###### Proposition 1\(Decoder\-space interference bound\)\.

LetG=Wdec⊤​WdecG=W\_\{\\mathrm\{dec\}\}^\{\\top\}W\_\{\\mathrm\{dec\}\}andG\(i​j\)G^\{\(ij\)\}its submatrix onFaiℓ×FajℓF\_\{a\_\{i\}\}^\{\\ell\}\\times F\_\{a\_\{j\}\}^\{\\ell\}\. IfFaiℓ∩Fajℓ=∅F\_\{a\_\{i\}\}^\{\\ell\}\\cap F\_\{a\_\{j\}\}^\{\\ell\}=\\emptyset, then

\|cos⁡\(Δ​xt\(i\),Δ​xt\(j\)\)\|≤σmax​\(G\(i​j\)\)/σmin​\(G\(i​i\)\)​σmin​\(G\(j​j\)\)\.\\big\|\\cos\(\\Delta x\_\{t\}^\{\(i\)\},\\Delta x\_\{t\}^\{\(j\)\}\)\\big\|\\;\\leq\\;\\sigma\_\{\\max\}\(G^\{\(ij\)\}\)\\big/\\sqrt\{\\sigma\_\{\\min\}\(G^\{\(ii\)\}\)\\,\\sigma\_\{\\min\}\(G^\{\(jj\)\}\)\}\.\(7\)

The bound depends only on decoder weights \(detailed and proof in §[A\.2](https://arxiv.org/html/2605.10971#A1.SS2); realized values in §[A\.3](https://arxiv.org/html/2605.10971#A1.SS3)\)\.

## 5Experiments

### 5\.1Experimental Setup

Table 3:Steering results\. Conf: target classifier confidence \(%, geometric mean for multi\-attribute\); PPL: perplexity; d2: distinct bigram ratio \(%\)\. Each row is the best operating point from anα\\alphasweep under quality gates \(§[5\.2](https://arxiv.org/html/2605.10971#S5.SS2)\)\. Selectedα\\alphavalues and standard deviations are in §[E](https://arxiv.org/html/2605.10971#A5)\.Single\-AttributeMulti\-AttributeSentiment \(S\)Topic \(T\)Style \(St\)S\+TS\+StSt\+TS\+T\+StConf↑\\uparrowPPL↓\\downarrowd2↑\\uparrowConf↑\\uparrowPPL↓\\downarrowd2↑\\uparrowConf↑\\uparrowPPL↓\\downarrowd2↑\\uparrowConf↑\\uparrowPPL↓\\downarrowd2↑\\uparrowConf↑\\uparrowPPL↓\\downarrowd2↑\\uparrowConf↑\\uparrowPPL↓\\downarrowd2↑\\uparrowConf↑\\uparrowPPL↓\\downarrowd2↑\\uparrowMDLM\-124MNo Steering25\.525\.5434392\.692\.618\.618\.6434392\.692\.653\.353\.3434392\.692\.622\.122\.1434392\.692\.639\.439\.4434392\.692\.636\.036\.0434392\.692\.632\.532\.5434392\.692\.6Uniform99\.999\.9939393\.793\.799\.999\.9595993\.093\.070\.070\.0828293\.893\.896\.596\.5919193\.193\.148\.948\.9888893\.993\.974\.574\.5737392\.992\.978\.878\.8828293\.193\.1Adaptive93\.793\.7575793\.693\.699\.999\.9505092\.592\.571\.771\.7575793\.393\.397\.597\.5717192\.592\.563\.763\.7737393\.593\.577\.677\.6595991\.991\.981\.381\.3848492\.892\.8Uniform\+E—————————99\.299\.2969693\.393\.365\.565\.5888894\.194\.175\.275\.2838392\.992\.980\.380\.3848493\.393\.3Adaptive\+E—————————95\.695\.6585892\.992\.969\.069\.0636394\.194\.172\.772\.7505092\.792\.780\.080\.0626293\.193\.1Contrastive Vec\.95\.095\.042142192\.592\.568\.968\.935435493\.293\.243\.343\.337937995\.695\.674\.274\.237637691\.891\.879\.679\.649449495\.395\.363\.863\.838938995\.195\.174\.574\.546646694\.594\.5Probe74\.374\.358158196\.496\.499\.299\.223023092\.492\.466\.466\.41285128598\.498\.486\.386\.332432494\.194\.167\.167\.185685697\.097\.076\.076\.087387397\.897\.871\.871\.879879897\.497\.4PCA20\.720\.749049096\.996\.915\.715\.733733794\.394\.342\.842\.838338395\.895\.815\.715\.731031093\.593\.530\.330\.340640695\.995\.928\.528\.534634694\.794\.724\.524\.539639695\.995\.9SEDD\-124MNo Steering29\.029\.0444492\.592\.520\.320\.3444492\.592\.557\.757\.7444492\.592\.524\.624\.6444492\.592\.543\.443\.4444492\.592\.539\.039\.0444492\.592\.535\.735\.7444492\.592\.5Uniform98\.898\.8949491\.191\.1100\.0100\.0575791\.591\.593\.793\.7979784\.184\.196\.496\.4989889\.389\.386\.186\.1989885\.185\.184\.584\.5939388\.188\.187\.087\.0888888\.588\.5Adaptive76\.876\.8555590\.390\.3100\.0100\.0494990\.790\.791\.291\.2484887\.287\.296\.496\.4676787\.287\.280\.680\.6515184\.484\.489\.589\.5575784\.084\.092\.592\.5636383\.783\.7Uniform\+E—————————96\.896\.8959591\.191\.195\.895\.8969687\.787\.785\.085\.0989888\.388\.379\.379\.3999989\.789\.7Adaptive\+E—————————80\.180\.1595991\.591\.579\.279\.2585889\.989\.989\.589\.5575784\.084\.071\.171\.1616191\.091\.0Contrastive Vec\.99\.899\.8888891\.391\.399\.999\.9535391\.591\.591\.791\.7818191\.591\.599\.599\.5919192\.192\.198\.098\.0797990\.090\.083\.983\.9868692\.592\.587\.587\.5878791\.791\.7Probe95\.795\.7969691\.491\.487\.887\.8767692\.892\.887\.287\.2919191\.991\.985\.185\.1989892\.392\.385\.685\.6888891\.991\.974\.074\.0989892\.992\.979\.979\.9949492\.592\.5PCA33\.333\.3515192\.692\.632\.932\.9404092\.492\.456\.756\.7404091\.891\.825\.125\.1484892\.892\.870\.070\.0727292\.992\.940\.940\.9373791\.791\.745\.545\.5434392\.792\.7LLaDA\-8BNo Steering69\.169\.1161685\.185\.113\.913\.9161685\.185\.122\.222\.2161685\.185\.141\.541\.5161685\.185\.145\.645\.6161685\.185\.118\.118\.1161685\.185\.135\.135\.1161685\.185\.1Uniform89\.089\.0121276\.576\.599\.999\.96644\.244\.269\.069\.07743\.743\.797\.397\.36643\.843\.865\.065\.07743\.643\.668\.068\.06645\.245\.271\.171\.17745\.245\.2Adaptive79\.679\.6141480\.680\.699\.799\.76650\.950\.966\.166\.18846\.846\.894\.794\.78856\.956\.967\.267\.27745\.445\.469\.869\.87746\.746\.772\.372\.37746\.146\.1Uniform\+E—————————92\.492\.47754\.454\.461\.361\.37743\.143\.167\.067\.06643\.543\.568\.768\.76643\.443\.4Adaptive\+E—————————89\.689\.69965\.065\.058\.058\.0111163\.163\.156\.456\.4111158\.758\.766\.066\.08852\.352\.3Contrastive Vec\.97\.197\.1101072\.772\.729\.629\.68874\.474\.453\.753\.77771\.471\.459\.959\.98881\.981\.967\.767\.78866\.266\.245\.945\.98875\.975\.957\.157\.1101084\.184\.1Probe89\.089\.07769\.369\.332\.332\.38870\.370\.348\.648\.68866\.966\.955\.255\.28874\.374\.367\.267\.28872\.672\.645\.045\.09979\.679\.654\.354\.37767\.167\.1PCA94\.294\.28873\.073\.024\.124\.18873\.273\.249\.749\.79973\.473\.457\.857\.87761\.561\.573\.273\.28869\.969\.939\.639\.67766\.066\.049\.549\.58873\.273\.2Prompt80\.980\.9161684\.784\.764\.964\.9101070\.070\.045\.145\.1202065\.365\.369\.669\.6171782\.782\.748\.448\.4222280\.480\.455\.955\.9121274\.974\.953\.953\.9191986\.286\.2DREAM\-7BNo Steering57\.757\.7141487\.987\.910\.510\.5141487\.987\.926\.526\.5141487\.987\.934\.134\.1141487\.987\.942\.142\.1141487\.987\.918\.518\.5141487\.987\.931\.631\.6141487\.987\.9Uniform99\.599\.5141484\.584\.5100\.0100\.0535383\.683\.697\.597\.5757560\.460\.499\.899\.8262678\.678\.697\.097\.0616167\.267\.282\.882\.8727286\.286\.286\.386\.3676783\.783\.7Adaptive97\.697\.6121285\.185\.1100\.0100\.0373747\.647\.696\.396\.3606054\.454\.499\.399\.3505056\.856\.896\.296\.2343454\.354\.390\.890\.8949470\.770\.789\.289\.2747477\.677\.6Uniform\+E—————————100\.0100\.0212177\.477\.495\.395\.3525266\.366\.388\.288\.2707085\.385\.391\.191\.1787882\.882\.8Adaptive\+E—————————99\.899\.8171776\.776\.794\.594\.5313156\.156\.186\.386\.3838379\.979\.993\.093\.0727263\.963\.9Contrastive Vec\.95\.395\.3161683\.683\.688\.788\.7151589\.489\.492\.892\.8414179\.079\.085\.685\.6151579\.979\.990\.990\.9414170\.370\.373\.373\.3505078\.178\.179\.179\.1515176\.676\.6Probe86\.686\.6151587\.287\.276\.776\.7141483\.283\.298\.598\.5484894\.494\.465\.765\.7151588\.188\.181\.681\.6515189\.289\.281\.881\.8565688\.488\.473\.173\.1363688\.788\.7PCA65\.365\.3151588\.788\.718\.318\.3161682\.682\.627\.827\.8151589\.489\.448\.748\.7131379\.479\.440\.640\.6151591\.191\.124\.324\.3141485\.285\.232\.332\.3141489\.789\.7Prompt80\.380\.3121281\.881\.861\.561\.5131384\.184\.120\.620\.6151578\.778\.771\.871\.8121280\.280\.241\.141\.1121273\.873\.843\.143\.1151586\.786\.757\.857\.8141478\.678\.6

Models and attributes:We evaluate the four SAE steering modes from §[4\.1](https://arxiv.org/html/2605.10971#S4.SS1)\(Uniform, Adaptive, Uniform\+E, and Adaptive\+E\) on all four DLMs \(Table[2](https://arxiv.org/html/2605.10971#S3.T2)\) across 7 attribute combinations: 3 single\-attribute, sentiment \(S\), topic \(T\), and style \(St\), and 4 multi\-attribute \(S\+T, S\+St, St\+T, S\+T\+St\)\. Sentiment is steered toward positive, topic toward sports, and style toward formality \(MDLM/SEDD\) or informality \(LLaDA/DREAM\)\. Each attribute is steered using the top 20 features per direction; a feature count ablation \(§[F\.2](https://arxiv.org/html/2605.10971#A6.SS2)\) shows that lower\-ranked features increasingly encode correlated attributes \(e\.g\., sentiment features that also shift formality\), increasing cross\-attribute interference\. MDLM and SEDD generate unconditionally over 1024 denoising steps; LLaDA and DREAM use 10 domain\-neutral prompts \(e\.g\., “I think this is”\) with temperature 0\.8/0\.7 over 64/256 steps \(§[C\.7](https://arxiv.org/html/2605.10971#A3.SS7)\)\.

Baselines:We compare against three residual\-stream baselines without SAE decomposition:Contrastive Vectors\(mean class difference\[[46](https://arxiv.org/html/2605.10971#bib.bib24),[30](https://arxiv.org/html/2605.10971#bib.bib41)\]\),Probe\(logistic regression weight\[[6](https://arxiv.org/html/2605.10971#bib.bib42)\]\), andPCA\(top principal component of contrastive activations\)\. For LLaDA and DREAM, we additionally evaluatePrompt Steering, prepending an attribute instruction to the prompt \(§[C\.8](https://arxiv.org/html/2605.10971#A3.SS8)\)\. We note that other baselines such as ILRR\[[3](https://arxiv.org/html/2605.10971#bib.bib11)\]require a reference sequence \(a different problem setup\), and classifier\-based guidance methods\[[14](https://arxiv.org/html/2605.10971#bib.bib59)\]require differentiable classifiers integrated into the denoising loop and are not directly applicable to masked discrete diffusion\.

Metrics:Steering performance is measured by off\-the\-shelf classifier confidence: DistilBERT\-SST2\[[33](https://arxiv.org/html/2605.10971#bib.bib55)\]for sentiment, BERT\-AG News\[[13](https://arxiv.org/html/2605.10971#bib.bib54)\]for topic, and a RoBERTa formality ranker\[[4](https://arxiv.org/html/2605.10971#bib.bib47)\]for style \(§[C\.9](https://arxiv.org/html/2605.10971#A3.SS9)\)\. Quality is measured by GPT\-2 perplexity \(PPL\) and distinct bigram ratio \(dist\-2\)\. On LLaDA, aggressive steering produces repetitive text that achieves low PPL by exploiting n\-gram patterns; dist\-2 is therefore the primary quality indicator for this model\. For each method, we generate 200 samples acrossα=1\\alpha=1–1515\(11–2020for SEDD, due to slower saturation\) and select theα\\alphamaximizing target confidence \(single\-attribute\) or geometric mean confidence \(multi\-attribute\), subject todist​\-​2≥50%\\mathrm\{dist\\text\{\-\}2\}\\geq 50\\%of baseline andPPL<100\\mathrm\{PPL\}<100\(MDLM/SEDD baselines exempt, as they produce high PPL regardless ofα\\alpha\)\.

### 5\.2Results

Single\-attribute steering:Table[3](https://arxiv.org/html/2605.10971#S5.T3)shows results across all conditions \(selectedα\\alphavalues and standard deviations in §[E](https://arxiv.org/html/2605.10971#A5)\)\. SAE methods achieve strong control: 70–100% on MDLM, 77–100% on SEDD, 66–100% on LLaDA, and 96–100% on DREAM\. The gap is largest on LLaDA topic, where SAE methods reach 99–100% while baselines remain at 24–32% \(near the 14% unsteered rate\) and prompt steering reaches only 65%\. On MDLM, Adaptive matches Uniform’s control at lower PPL \(50–57 vs\. 59–93\) by targeting high\-emergence blocks \(§[3\.3](https://arxiv.org/html/2605.10971#S3.SS3)\); on LLaDA, whose flat temporal profile reduces Adaptive to approximately Uniform \(§[4\.1](https://arxiv.org/html/2605.10971#S4.SS1)\), both perform comparably\. SEDD is a notable exception: Contrastive Vectors matches SAE performance \(100% S, 100% T, 92% St\), as SEDD’s distributed feature geometry \(Figure[3](https://arxiv.org/html/2605.10971#S3.F3)\) lets the mean\-difference vector aggregate many aligned contributions; on MDLM, the same vector dilutes strong features, widening the SAE advantage\.

Multi\-attribute steering:SAE methods enable simultaneous control, with triple\-condition confidence reaching 81% \(MDLM\), 93% \(SEDD\), 72% \(LLaDA\), and 93% \(DREAM\)\. On LLaDA, the best SAE method exceeds the best baseline by \+15pp \(72% vs\. 57%\), driven by topic conditions\. On MDLM, Adaptive resolves inter\-attribute interference: for S\+St, Uniform achieves only 49% while Adaptive reaches 64% by targeting sentiment’s later emergence blocks \(§[3\.3](https://arxiv.org/html/2605.10971#S3.SS3)\), avoiding disruption of formality’s earlier commitment\. Adaptive\+E combines both mechanisms, matching Uniform’s triple control \(80% vs\. 79%\) at substantially lower PPL \(50–63 vs\. 73–91\); notably, the quality gap between Adaptive and Uniform*widens*in multi\-attribute steering as uniform perturbation compounds across attributes\. On LLaDA, E\-ratio calibration yields the largest diversity gains: Adaptive\+E preservesd2d\_\{2\}at 52–65% where Uniform collapses to 43–45%, trading minimal control for richer output \(§[4](https://arxiv.org/html/2605.10971#S4)\)\.

![Refer to caption](https://arxiv.org/html/2605.10971v1/x9.png)Figure 4:MDLM\-124M trade\-off curves\. Top: target confidence vs\.α\\alpha\. Bottom: perplexity vs\.α\\alpha\. Adaptive matches or exceeds Uniform in control without exploding PPL\. Full curves in §[F\.1](https://arxiv.org/html/2605.10971#A6.SS1)\.Quality–control trade\-off:Figure[4](https://arxiv.org/html/2605.10971#S5.F4)shows theα\\alphasweep on MDLM for all four SAE methods; full curves including baselines and all models are in §[F\.1](https://arxiv.org/html/2605.10971#A6.SS1)\. The pattern generalizes: on MDLM, Uniform drives PPL above 500 for topic while Adaptive matches rates at far lower PPL by targeting high\-emergence blocks; on LLaDA, Uniform drives dist\-2 from 85% to 44% while Adaptive maintains 47–81%\. Baselines on LLaDA maintain near non\-steering quality but fail to shift target attributes\. MDLM shows the clearest Adaptive–Uniform separation; LLaDA is hardest to steer \(particularly informality, 66–69% from 22%\)\. On SEDD, baselines maintain higher diversity than SAE methods \(dist\-2 91–93% vs\. 84–91%\), making them competitive given SEDD’s distributed representations\. We report latency analysis and qualitative examples for all DLMs in §[F\.4](https://arxiv.org/html/2605.10971#A6.SS4)and §[F\.5](https://arxiv.org/html/2605.10971#A6.SS5), respectively\.

Cross\-attribute interference:We quantify interference as the mean absolute pp shift of non\-target attributes from their unsteered baselines \(Figure[14](https://arxiv.org/html/2605.10971#A6.F14), §[F\.3](https://arxiv.org/html/2605.10971#A6.SS3)\)\. Zero interference is intuitively impossible because some attributes are naturally correlated \(e\.g\., positive sentiment and formality\); nonetheless, Adaptive consistently produces less interference than Uniform, with the largest gap on MDLM \(up to 52pp for Uniform vs\. 31pp for Adaptive\), because targeting high\-emergence steps avoids disrupting other attributes during their critical commitment\.

Special case: DREAM:For all combinations, Adaptive matches Uniform’s high control but with mixed quality trade\-offs \(PPL improves on some conditions, regresses on others;d2d\_\{2\}systematically lower\)\. We attribute this to DREAM’s AR\-based initialization, which inherits late\-trajectory token\-finalization behavior from autoregressive pretraining \(§[F\.1](https://arxiv.org/html/2605.10971#A6.SS1)\)\.

## 6Conclusion

We use sparse autoencoders to reveal how semantic features emerge during discrete diffusion denoising, discovering that attributes commit at different steps, with different sharpness, and at different magnitudes\. These findings directly inform an adaptive steering framework that concentrates intervention where each attribute is actively forming, achieving strong multi\-attribute control at substantially lower perplexity and interference than uniform or non\-SAE baselines\. More broadly, our results show that the iterative structure of DLMs provides a natural foundation for temporally aligned, composable interventions, turning interpretability into a practical lever for controlled generation\.

Limitations:Our evaluation relies on classifier\-based metrics that may not capture nuanced attribute expression\. Generated sequences are short \(64–1024 tokens\); steering at longer lengths remains unexplored\. Diversity decreases at high steering strengths in SAE\-based methods\. Extending to finer\-grained attributes and learning steering schedules from feature dynamics are open directions\.

## References

- \[1\]\(2025\)SAEs are good for steering – if you select the right features\.arXiv preprint arXiv:2505\.20063\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§2](https://arxiv.org/html/2605.10971#S2.p1.1),[§3](https://arxiv.org/html/2605.10971#S3.p1.1)\.
- \[2\]J\. Austin, D\. D\. Johnson, J\. Ho, D\. Tarlow, and R\. Van Den Berg\(2021\)Structured denoising diffusion models in discrete state\-spaces\.Advances in neural information processing systems34,pp\. 17981–17993\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p1.1)\.
- \[3\]E\. Avrahami and E\. Nachmani\(2026\)ILRR: inference\-time steering method for masked diffusion language models\.arXiv preprint arXiv:2601\.21647\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[Table 1](https://arxiv.org/html/2605.10971#S2.T1.6.4.4.4.4.4.4.4.3),[§2](https://arxiv.org/html/2605.10971#S2.p1.1),[§5\.1](https://arxiv.org/html/2605.10971#S5.SS1.p2.1)\.
- \[4\]N\. Babakov, D\. Dale, I\. Gusev, I\. Krotova, and A\. Panchenko\(2023\)Don’t lose the message while paraphrasing: a study on content preserving style transfer\.InInternational Conference on Applications of Natural Language to Information Systems,pp\. 47–61\.Cited by:[§C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1.p4.20),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.7.4),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.7.5),[Table 16](https://arxiv.org/html/2605.10971#A3.T16.4.1.4.2),[§3\.2](https://arxiv.org/html/2605.10971#S3.SS2.p1.10),[§5\.1](https://arxiv.org/html/2605.10971#S5.SS1.p3.8)\.
- \[5\]S\. Basu, S\. Y\. Patel, P\. Sheth, B\. Muralidharan, N\. Elamaran, A\. Kinra, J\. Morgan, and R\. Batniji\(2026\)Interpretability without actionability: mechanistic methods cannot correct language model errors despite near\-perfect internal representations\.arXiv preprint arXiv:2603\.18353\.Cited by:[§2](https://arxiv.org/html/2605.10971#S2.p1.1)\.
- \[6\]Y\. Belinkov\(2022\-03\)Probing classifiers: promises, shortcomings, and advances\.Computational Linguistics48\(1\),pp\. 207–219\.External Links:[Link](https://aclanthology.org/2022.cl-1.7/),[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00422)Cited by:[§C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1.p3.6),[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§3\.2](https://arxiv.org/html/2605.10971#S3.SS2.p1.10),[§5\.1](https://arxiv.org/html/2605.10971#S5.SS1.p2.1)\.
- \[7\]N\. Belrose, I\. Ostrovsky, L\. McKinney, Z\. Furman, L\. Smith, D\. Halawi, S\. Biderman, and J\. Steinhardt\(2023\)Eliciting latent predictions from transformers with the tuned lens\.arXiv preprint arXiv:2303\.08112\.Cited by:[§C\.2](https://arxiv.org/html/2605.10971#A3.SS2.p1.1)\.
- \[8\]C\. Bonferroni\(1936\)Teoria statistica delle classi e calcolo delle probabilita\.Pubblicazioni del R istituto superiore di scienze economiche e commericiali di firenze8,pp\. 3–62\.Cited by:[item 7](https://arxiv.org/html/2605.10971#A3.I5.i7.p1.5),[§3\.2](https://arxiv.org/html/2605.10971#S3.SS2.p1.10)\.
- \[9\]T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly,et al\.\(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§2](https://arxiv.org/html/2605.10971#S2.p1.1),[§3](https://arxiv.org/html/2605.10971#S3.p1.1)\.
- \[10\]S\. Cho, Z\. Wu, and A\. Koshiyama\(2025\)CorrSteer: generation\-time llm steering via correlated sparse autoencoder features\.arXiv preprint arXiv:2508\.12535\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§2](https://arxiv.org/html/2605.10971#S2.p1.1),[§3](https://arxiv.org/html/2605.10971#S3.p1.1)\.
- \[11\]J\. Cohen\(2013\)Statistical power analysis for the behavioral sciences\.routledge\.Cited by:[item 6](https://arxiv.org/html/2605.10971#A3.I5.i6.p1.1.1),[§3\.2](https://arxiv.org/html/2605.10971#S3.SS2.p1.10)\.
- \[12\]H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. Sharkey\(2023\)Sparse autoencoders find highly interpretable features in language models\.arXiv preprint arXiv:2309\.08600\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§2](https://arxiv.org/html/2605.10971#S2.p1.1),[§3\.1](https://arxiv.org/html/2605.10971#S3.SS1.p1.10),[§3](https://arxiv.org/html/2605.10971#S3.p1.1)\.
- \[13\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[Table 16](https://arxiv.org/html/2605.10971#A3.T16.4.1.3.2),[§5\.1](https://arxiv.org/html/2605.10971#S5.SS1.p3.8)\.
- \[14\]P\. Dhariwal and A\. Nichol\(2021\)Diffusion models beat gans on image synthesis\.Advances in neural information processing systems34,pp\. 8780–8794\.Cited by:[§5\.1](https://arxiv.org/html/2605.10971#S5.SS1.p2.1)\.
- \[15\]D\. L\. Donoho and M\. Elad\(2003\)Optimally sparse representation in general \(nonorthogonal\) dictionaries via l1 minimization\.Proceedings of the National Academy of Sciences100\(5\),pp\. 2197–2202\.Cited by:[§A\.2\.3](https://arxiv.org/html/2605.10971#A1.SS2.SSS3.Px2.p1.7)\.
- \[16\]R\. Goel, R\. Garrepalli, S\. Agrawal, C\. Lott, M\. Lee, and F\. Porikli\(2026\)Skip to the good part: representation structure & inference\-time layer skipping in diffusion vs\. autoregressive llms\.arXiv preprint arXiv:2603\.07475\.Cited by:[Table 1](https://arxiv.org/html/2605.10971#S2.T1.8.6.6.6.6.6.6.6.3),[§2](https://arxiv.org/html/2605.10971#S2.p1.1)\.
- \[17\]A\. Gokaslan and V\. Cohen\(2019\)OpenWebText corpus\.Cited by:[§C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1.p4.20)\.
- \[18\]A\. Goyal, V\. Rathi, W\. Yeh, Y\. Wang, Y\. Chen, and H\. Sundaram\(2025\)Breaking bad tokens: detoxification of llms using sparse autoencoders\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 12702–12720\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§2](https://arxiv.org/html/2605.10971#S2.p1.1),[§3](https://arxiv.org/html/2605.10971#S3.p1.1)\.
- \[19\]A\. Hemmat, P\. Torr, Y\. Chen, and J\. Yu\(2026\)TDGNet: hallucination detection in diffusion language models via temporal dynamic graphs\.arXiv preprint arXiv:2602\.08048\.Cited by:[Table 1](https://arxiv.org/html/2605.10971#S2.T1.8.6.6.6.6.6.6.8.1),[§2](https://arxiv.org/html/2605.10971#S2.p1.1)\.
- \[20\]A\. Lou, C\. Meng, and S\. Ermon\(2024\)Discrete diffusion modeling by estimating the ratios of the data distribution\.InInternational Conference on Machine Learning,pp\. 32819–32848\.Cited by:[§C\.1](https://arxiv.org/html/2605.10971#A3.SS1.p1.1),[§1](https://arxiv.org/html/2605.10971#S1.p1.1),[§1](https://arxiv.org/html/2605.10971#S1.p3.1),[§3\.1](https://arxiv.org/html/2605.10971#S3.SS1.p1.10),[Table 2](https://arxiv.org/html/2605.10971#S3.T2.7.5.5.2)\.
- \[21\]J\. J\. Louviere, T\. N\. Flynn, and A\. A\. J\. Marley\(2015\)Best\-worst scaling: theory, methods and applications\.Cambridge University Press\.Cited by:[§C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1.p4.20)\.
- \[22\]A\. L\. Maas, R\. E\. Daly, P\. T\. Pham, D\. Huang, A\. Y\. Ng, and C\. Potts\(2011\-06\)Learning word vectors for sentiment analysis\.InProceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies,D\. Lin, Y\. Matsumoto, and R\. Mihalcea \(Eds\.\),Portland, Oregon, USA,pp\. 142–150\.External Links:[Link](https://aclanthology.org/P11-1015/)Cited by:[§C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1.p2.5),[§C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1.p4.20),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.5.2),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.5.3),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.5.4),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.5.5),[§3\.2](https://arxiv.org/html/2605.10971#S3.SS2.p1.10)\.
- \[23\]A\. Makhzani and B\. Frey\(2013\)K\-sparse autoencoders\.arXiv preprint arXiv:1312\.5663\.Cited by:[§C\.1](https://arxiv.org/html/2605.10971#A3.SS1.p2.3)\.
- \[24\]H\. B\. Mann and D\. R\. Whitney\(1947\)On a test of whether one of two random variables is stochastically larger than the other\.The annals of mathematical statistics,pp\. 50–60\.Cited by:[item 7](https://arxiv.org/html/2605.10971#A3.I5.i7.p1.5),[§3\.2](https://arxiv.org/html/2605.10971#S3.SS2.p1.10)\.
- \[25\]S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. ZHOU, Y\. Lin, J\. Wen, and C\. LiLarge language diffusion models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§C\.1](https://arxiv.org/html/2605.10971#A3.SS1.p1.1),[§1](https://arxiv.org/html/2605.10971#S1.p1.1),[§1](https://arxiv.org/html/2605.10971#S1.p3.1),[§3\.1](https://arxiv.org/html/2605.10971#S3.SS1.p1.10),[Table 2](https://arxiv.org/html/2605.10971#S3.T2.8.6.6.2)\.
- \[26\]N\. Oozeer, L\. Marks, F\. Barez, and A\. Abdullah\(2025\)Beyond linear steering: unified multi\-attribute control for language models\.Findings of the Association for Computational Linguistics: EMNLP,pp\. 23513–23557\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§2](https://arxiv.org/html/2605.10971#S2.p1.1)\.
- \[27\]E\. Pavlick and J\. Tetreault\(2016\)An empirical analysis of formality in online communication\.Transactions of the association for computational linguistics4,pp\. 61–74\.Cited by:[§C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1.p4.20),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.7.2),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.7.3),[§3\.2](https://arxiv.org/html/2605.10971#S3.SS2.p1.10)\.
- \[28\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[Table 16](https://arxiv.org/html/2605.10971#A3.T16.4.1.5.2),[Table 2](https://arxiv.org/html/2605.10971#S3.T2.6.4.4.4)\.
- \[29\]S\. Rao and J\. Tetreault\(2018\)Dear sir or madam, may i introduce the gyafc dataset: corpus, benchmarks and metrics for formality style transfer\.InProceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 \(long papers\),pp\. 129–140\.Cited by:[§C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1.p4.20)\.
- \[30\]N\. Rimsky, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. Turner\(2024\-08\)Steering llama 2 via contrastive activation addition\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15504–15522\.External Links:[Link](https://aclanthology.org/2024.acl-long.828/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.828)Cited by:[§C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1.p3.6),[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§3\.2](https://arxiv.org/html/2605.10971#S3.SS2.p1.10),[§5\.1](https://arxiv.org/html/2605.10971#S5.SS1.p2.1)\.
- \[31\]R\. Ronge, M\. Maier, and F\. Eberhardt\(2026\)When the coffee feature activates on coffins: an analysis of feature extraction and steering for mechanistic interpretability\.arXiv preprint arXiv:2601\.03047\.Cited by:[§2](https://arxiv.org/html/2605.10971#S2.p1.1)\.
- \[32\]S\. S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. Marroquin, J\. T\. Chiu, A\. Rush, and V\. Kuleshov\(2024\)Simple and effective masked diffusion language models\.Advances in Neural Information Processing Systems37,pp\. 130136–130184\.Cited by:[§C\.1](https://arxiv.org/html/2605.10971#A3.SS1.p1.1),[§1](https://arxiv.org/html/2605.10971#S1.p1.1),[§1](https://arxiv.org/html/2605.10971#S1.p3.1),[§3\.1](https://arxiv.org/html/2605.10971#S3.SS1.p1.10),[Table 2](https://arxiv.org/html/2605.10971#S3.T2.6.4.4.2)\.
- \[33\]V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf\(2019\)DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.arXiv preprint arXiv:1910\.01108\.Cited by:[Table 16](https://arxiv.org/html/2605.10971#A3.T16.4.1.2.2),[§5\.1](https://arxiv.org/html/2605.10971#S5.SS1.p3.8)\.
- \[34\]O\. Shafran, S\. Ronen, O\. Fahn, S\. Ravfogel, A\. Geiger, and M\. Geva\(2026\)From directions to regions: decomposing activations in language models via local geometry\.arXiv preprint arXiv:2602\.02464\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§2](https://arxiv.org/html/2605.10971#S2.p1.1)\.
- \[35\]D\. Son, S\. Rathore, A\. Rufail, A\. Simon, D\. Zhang, S\. Dave, C\. Blondin, K\. Zhu, and S\. O’Brien\(2025\)Semantic convergence: investigating shared representations across scaled llms\.arXiv preprint arXiv:2507\.22918\.Cited by:[§2](https://arxiv.org/html/2605.10971#S2.p1.1)\.
- \[36\]A\. Templeton, T\. Conerly, J\. Marcus,et al\.\(2024\)Scaling monosemanticity: extracting interpretable features from Claude 3 Sonnet\.Transformer Circuits Thread\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§2](https://arxiv.org/html/2605.10971#S2.p1.1),[§3](https://arxiv.org/html/2605.10971#S3.p1.1)\.
- \[37\]H\. Thasarathan, J\. Forsyth, T\. Fel, M\. Kowal, and K\. G\. Derpanis\(2025\)Universal sparse autoencoders: interpretable cross\-model concept alignment\.InForty\-second International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2605.10971#S2.p1.1)\.
- \[38\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[Table 2](https://arxiv.org/html/2605.10971#S3.T2.8.6.6.4)\.
- \[39\]X\. Wang, B\. Jiang, Y\. Wan, B\. Yang, L\. Kong, and D\. Zou\(2026\)DLM\-scope: mechanistic interpretability of diffusion language models via sparse autoencoders\.arXiv preprint arXiv:2602\.05859\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[Table 1](https://arxiv.org/html/2605.10971#S2.T1.4.2.2.2.2.2.2.2.3),[§2](https://arxiv.org/html/2605.10971#S2.p1.1)\.
- \[40\]B\. P\. Welford\(1962\)Note on a method for calculating corrected sums of squares and products\.Technometrics4\(3\),pp\. 419–420\.Cited by:[item 5](https://arxiv.org/html/2605.10971#A3.I5.i5.p1.1)\.
- \[41\]A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou,et al\.\(2024\)Qwen2 technical report\.arXiv preprint arXiv:2407\.10671\.Cited by:[Table 2](https://arxiv.org/html/2605.10971#S3.T2.9.7.7.4)\.
- \[42\]J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. Kong\(2025\)Dream 7b: diffusion large language models\.arXiv preprint arXiv:2508\.15487\.Cited by:[§C\.1](https://arxiv.org/html/2605.10971#A3.SS1.p1.1),[§1](https://arxiv.org/html/2605.10971#S1.p1.1),[§1](https://arxiv.org/html/2605.10971#S1.p3.1),[§3\.1](https://arxiv.org/html/2605.10971#S3.SS1.p1.10),[Table 2](https://arxiv.org/html/2605.10971#S3.T2.9.7.7.2)\.
- \[43\]J\. Yeon, F\. Cinus, Y\. Wu, and L\. Luceri\(2025\)GSAE: graph\-regularized sparse autoencoders for robust llm safety steering\.arXiv preprint arXiv:2512\.06655\.Cited by:[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§2](https://arxiv.org/html/2605.10971#S2.p1.1),[§3](https://arxiv.org/html/2605.10971#S3.p1.1)\.
- \[44\]X\. Zhang, J\. Zhao, and Y\. LeCun\(2015\)Character\-level convolutional networks for text classification\.Advances in neural information processing systems28\.Cited by:[§C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1.p3.6),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.6.2),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.6.3),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.6.4),[Table 15](https://arxiv.org/html/2605.10971#A3.T15.2.2.6.5),[§3\.2](https://arxiv.org/html/2605.10971#S3.SS2.p1.10)\.
- \[45\]X\. Zhu, M\. M\. Khalili, and Z\. Zhu\(2025\)AbsTopK: rethinking sparse autoencoders for bidirectional features\.arXiv preprint arXiv:2510\.00404\.Cited by:[§3\.1](https://arxiv.org/html/2605.10971#S3.SS1.p1.10)\.
- \[46\]A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski,et al\.\(2023\)Representation engineering: a top\-down approach to ai transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[§C\.3\.3](https://arxiv.org/html/2605.10971#A3.SS3.SSS3.p2.1),[§C\.3](https://arxiv.org/html/2605.10971#A3.SS3.p1.1),[§C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1.p3.6),[§1](https://arxiv.org/html/2605.10971#S1.p2.1),[§3\.2](https://arxiv.org/html/2605.10971#S3.SS2.p1.10),[§5\.1](https://arxiv.org/html/2605.10971#S5.SS1.p2.1)\.

## Appendix AProofs and Extended Discussion for Steering Methods

We restate each result for convenience and provide full proofs\.

### A\.1Proof of Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1)\(Adaptive\-vs\-uniform efficiency ratio\)

#### A\.1\.1Trajectory\-level KL: exact additive decomposition

Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1)optimizes against the budget12​∑tct​αt2≤B\\tfrac\{1\}\{2\}\\sum\_\{t\}c\_\{t\}\\alpha\_\{t\}^\{2\}\\leq B, where eachctc\_\{t\}is the second\-order cost of perturbation at steptt\. This budget arises as a tractable additive decomposition of the trajectory\-level KL between steered and unsteered reverse processes; we derive the decomposition here\.

Letpθ,p~θp\_\{\\theta\},\\tilde\{p\}\_\{\\theta\}denote the joint distributions of the unsteered and steered reverse trajectories\(xT,xT−1,…,x0\)\(x\_\{T\},x\_\{T\-1\},\\ldots,x\_\{0\}\)\. Both are Markov: the reverse transition at stepttispθ​\(xt−1∣xt\)p\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\}\)for the unsteered model andp~θ​\(xt−1∣xt\)\\tilde\{p\}\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\}\)for the steered model \(which differs only in the residual stream perturbation at steptt\)\. Both processes share the same initial distribution atxTx\_\{T\}\(e\.g\., the fully\-masked prior in absorbing\-state diffusion\), soKL​\(p~θ​\(xT\)∥pθ​\(xT\)\)=0\\mathrm\{KL\}\(\\tilde\{p\}\_\{\\theta\}\(x\_\{T\}\)\\,\\\|\\,p\_\{\\theta\}\(x\_\{T\}\)\)=0\. The chain rule for KL on Markov chains applied to the reverse joint, then it gives

KL\(p~θ∥pθ\)=∑t=1T𝔼xt∼p~θ\[KL\(p~θ\(xt−1∣xt\)∥pθ\(xt−1∣xt\)\)\]\.\\mathrm\{KL\}\\\!\\big\(\\tilde\{p\}\_\{\\theta\}\\,\\big\\\|\\,p\_\{\\theta\}\\big\)\\;=\\;\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\_\{x\_\{t\}\\sim\\tilde\{p\}\_\{\\theta\}\}\\\!\\Big\[\\mathrm\{KL\}\\big\(\\tilde\{p\}\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\}\)\\,\\big\\\|\\,p\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\}\)\\big\)\\Big\]\.\(8\)
##### First\-order change\-of\-measure approximation\.

The outer expectation in \([8](https://arxiv.org/html/2605.10971#A1.E8)\) is under the steered measurep~θ\\tilde\{p\}\_\{\\theta\}, whilextx\_\{t\}\-marginal samples are easier to obtain underpθp\_\{\\theta\}\. Replacingp~θ\\tilde\{p\}\_\{\\theta\}bypθp\_\{\\theta\}in the outer expectation incurs an error that is sub\-leading inα\\alpha: the Radon–Nikodym derivatived​p~θ/d​pθd\\tilde\{p\}\_\{\\theta\}/dp\_\{\\theta\}equals11atα=0\\alpha=0and varies smoothly, so the difference𝔼p~θ​\[Φ​\(xt\)\]−𝔼pθ​\[Φ​\(xt\)\]\\mathbb\{E\}\_\{\\tilde\{p\}\_\{\\theta\}\}\[\\Phi\(x\_\{t\}\)\]\-\\mathbb\{E\}\_\{p\_\{\\theta\}\}\[\\Phi\(x\_\{t\}\)\]isO​\(α\)O\(\\alpha\)for any boundedΦ\\Phi\. Since the integrandΦt\(xt\):=KL\(p~θ\(xt−1∣xt\)∥pθ\(xt−1∣xt\)\)\\Phi\_\{t\}\(x\_\{t\}\):=\\mathrm\{KL\}\(\\tilde\{p\}\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\}\)\\,\\\|\\,p\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\}\)\)is itselfO​\(α2\)O\(\\alpha^\{2\}\)in leading order \(Lemma[1](https://arxiv.org/html/2605.10971#Thmlemma1)below\), the change\-of\-measure correction contributesO​\(α\)⋅O​\(α2\)=O​\(α3\)O\(\\alpha\)\\cdot O\(\\alpha^\{2\}\)=O\(\\alpha^\{3\}\), which is sub\-leading relative to theO​\(α2\)O\(\\alpha^\{2\}\)leading cost\. The replacement therefore preserves the leading\-order behavior of the total cost\. We use this approximation throughout\.

##### Per\-step quadratic expansion\.

For each summand, a second\-order Taylor expansion of the KL aroundαt=0\\alpha\_\{t\}=0gives

Δqt\(αt\)=𝔼xt∼pθ\[KL\(p~θ\(xt−1∣xt\)∥pθ\(xt−1∣xt\)\)\]=12ctαt2\+o\(αt2\),\\Delta q\_\{t\}\(\\alpha\_\{t\}\)\\;=\\;\\mathbb\{E\}\_\{x\_\{t\}\\sim p\_\{\\theta\}\}\\\!\\Big\[\\mathrm\{KL\}\\big\(\\tilde\{p\}\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\}\)\\,\\big\\\|\\,p\_\{\\theta\}\(x\_\{t\-1\}\\mid x\_\{t\}\)\\big\)\\Big\]\\;=\\;\\tfrac\{1\}\{2\}c\_\{t\}\\alpha\_\{t\}^\{2\}\+o\(\\alpha\_\{t\}^\{2\}\),where the linear term vanishes becauseαt=0\\alpha\_\{t\}=0globally minimizes each KL summand \(KL is non\-negative and zero atp~θ=pθ\\tilde\{p\}\_\{\\theta\}=p\_\{\\theta\}, so the gradient at zero perturbation vanishes\)\. The coefficientctc\_\{t\}is given by the Jacobian\-pullback expression of Lemma[1](https://arxiv.org/html/2605.10971#Thmlemma1)\. Summing acrossttand combining with the first\-order attribute gainΔ​gt​\(αt\)=αt​st\+o​\(αt\)\\Delta g\_\{t\}\(\\alpha\_\{t\}\)=\\alpha\_\{t\}s\_\{t\}\+o\(\\alpha\_\{t\}\)gives the budgeted allocation problem stated in the main text:

minαt≥0⁡12​∑t=0T−1ct​αt2s\.t\.∑t=0T−1αt​st≥Etarget\.\\min\_\{\\alpha\_\{t\}\\geq 0\}\\;\\tfrac\{1\}\{2\}\\sum\_\{t=0\}^\{T\-1\}c\_\{t\}\\alpha\_\{t\}^\{2\}\\quad\\text\{s\.t\.\}\\quad\\sum\_\{t=0\}^\{T\-1\}\\alpha\_\{t\}s\_\{t\}\\;\\geq\\;E\_\{\\mathrm\{target\}\}\.\(9\)

#### A\.1\.2Proof of the efficiency ratio theorem

##### Assumption 1 \(Finite\-horizon Markov reverse process\)\.

The steered and unsteered reverse processes, denoted byp~θ\\tilde\{p\}\_\{\\theta\}andpθp\_\{\\theta\}, respectively, are finite\-horizon Markov chains with horizonT<∞T<\\infty, sharing the same fully noised initial distribution atxTx\_\{T\}\.

##### Assumption 2 \(Smoothness of the attribute score\)\.

The attribute score functionggis twice continuously differentiable with respect to the steering strength atαt=0\\alpha\_\{t\}=0\. Specifically,ggis instantiated as the softmax probability of a classifier evaluated on the realized output sequence, which is smooth with respect to the residual\-stream activation at the intervention layer\.

##### Assumption 3 \(Local validity of Taylor expansions\)\.

There exists a neighborhood aroundαt=0\\alpha\_\{t\}=0in which

Δ​gt​\(αt\)=αt​st\+o​\(αt\),Δ​qt​\(αt\)=12​ct​αt2\+o​\(αt2\),\\Delta g\_\{t\}\(\\alpha\_\{t\}\)=\\alpha\_\{t\}s\_\{t\}\+o\(\\alpha\_\{t\}\),\\qquad\\Delta q\_\{t\}\(\\alpha\_\{t\}\)=\\frac\{1\}\{2\}c\_\{t\}\\alpha\_\{t\}^\{2\}\+o\(\\alpha\_\{t\}^\{2\}\),and the operating regime satisfiesαt=O​\(λ\)\\alpha\_\{t\}=O\(\\lambda\)such that higher\-order terms are negligible\.

##### Assumption 4 \(Nondegeneracy and feasibility\)\.

For alltt, the coefficients satisfyst≥0s\_\{t\}\\geq 0andct\>0c\_\{t\}\>0, with∑tst\>0\\sum\_\{t\}s\_\{t\}\>0andEtarget\>0E\_\{\\mathrm\{target\}\}\>0\. These conditions ensure feasibility of the constrained optimization problem in Eq\. \([5](https://arxiv.org/html/2605.10971#S4.E5)\)\.

Assumption A1 justifies the additive cost decomposition \(App\. §[A\.1\.1](https://arxiv.org/html/2605.10971#A1.SS1.SSS1)\)\. Assumptions A2 and A3 ensure thatctc\_\{t\}is well defined as the second derivative of the output KL \. Finally, Assumption A4 guarantees that the associated KKT system admits a unique nontrivial solution\.

##### Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1)\(restated\)\.

Adaptive\-vs\-uniform efficiency ratio\.Consider \([9](https://arxiv.org/html/2605.10971#A1.E9)\) withst≥0s\_\{t\}\\geq 0,ct\>0c\_\{t\}\>0, andEtarget\>0E\_\{\\mathrm\{target\}\}\>0\. Fix a quality budgetB=12​∑tct​αt2B=\\tfrac\{1\}\{2\}\\sum\_\{t\}c\_\{t\}\\alpha\_\{t\}^\{2\}and letE⋆E^\{\\star\},EunifE\_\{\\mathrm\{unif\}\}denote the maximum attribute shifts achievable at this budget under the optimal and uniform schedules respectively\. Then

ρ2:=\(E⋆Eunif\)2=\(∑tst2/ct\)​\(∑tct\)\(∑tst\)2=1\+CVc2​\(s/c\),\\rho^\{2\}\\;:=\\;\\left\(\\frac\{E^\{\\star\}\}\{E\_\{\\mathrm\{unif\}\}\}\\right\)^\{2\}\\;=\\;\\frac\{\\big\(\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\\big\)\\big\(\\sum\_\{t\}c\_\{t\}\\big\)\}\{\\big\(\\sum\_\{t\}s\_\{t\}\\big\)^\{2\}\}\\;=\\;1\\;\+\\;\\mathrm\{CV\}\_\{c\}^\{2\}\(s/c\),\(10\)whereCVc\\mathrm\{CV\}\_\{c\}denotes the coefficient of variation ofst/cts\_\{t\}/c\_\{t\}under the cost\-weighted measurect/∑t′ct′c\_\{t\}/\\sum\_\{t^\{\\prime\}\}c\_\{t^\{\\prime\}\}\. The optimum is achieved byαt⋆=λ​st/ct\\alpha\_\{t\}^\{\\star\}=\\lambda s\_\{t\}/c\_\{t\}withλ=Etarget/∑tst2/ct\\lambda=E\_\{\\mathrm\{target\}\}/\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\. Equalityρ=1\\rho=1holds iffst/cts\_\{t\}/c\_\{t\}is constant intt\.

###### Proof\.

The proof has three parts: \(i\) derive the optimum location via KKT, \(ii\) computeEunifE\_\{\\mathrm\{unif\}\}for the constant\-α\\alphaschedule, and \(iii\) show the resulting ratio equals1\+CVc2​\(s/c\)1\+\\mathrm\{CV\}\_\{c\}^\{2\}\(s/c\)\.

Part 1: optimum location\.The Lagrangian for \([9](https://arxiv.org/html/2605.10971#A1.E9)\) is

ℒ=12​∑tct​αt2−μ​\(∑tαt​st−Etarget\)−∑tνt​αt,\\mathcal\{L\}\\;=\\;\\tfrac\{1\}\{2\}\\sum\_\{t\}c\_\{t\}\\alpha\_\{t\}^\{2\}\\;\-\\;\\mu\\\!\\left\(\\sum\_\{t\}\\alpha\_\{t\}s\_\{t\}\-E\_\{\\mathrm\{target\}\}\\right\)\\;\-\\;\\sum\_\{t\}\\nu\_\{t\}\\alpha\_\{t\},withμ≥0\\mu\\geq 0andνt≥0\\nu\_\{t\}\\geq 0\. Stationarity givesct​αt=μ​st\+νtc\_\{t\}\\alpha\_\{t\}=\\mu s\_\{t\}\+\\nu\_\{t\}, i\.e\.,αt=\(μ​st\+νt\)/ct\\alpha\_\{t\}=\(\\mu s\_\{t\}\+\\nu\_\{t\}\)/c\_\{t\}\.

Forttwithst\>0s\_\{t\}\>0: ifαt=0\\alpha\_\{t\}=0thenνt=−μ​st≤0\\nu\_\{t\}=\-\\mu s\_\{t\}\\leq 0, which combined withνt≥0\\nu\_\{t\}\\geq 0forcesμ​st=0\\mu s\_\{t\}=0\. Sincest\>0s\_\{t\}\>0this requiresμ=0\\mu=0, but then allαt=0\\alpha\_\{t\}=0, violating∑tαt​st≥Etarget\>0\\sum\_\{t\}\\alpha\_\{t\}s\_\{t\}\\geq E\_\{\\mathrm\{target\}\}\>0\. Thereforeαt\>0\\alpha\_\{t\}\>0on\{t:st\>0\}\\\{t:s\_\{t\}\>0\\\}, complementary slackness givesνt=0\\nu\_\{t\}=0, andαt⋆=μ​st/ct\\alpha\_\{t\}^\{\\star\}=\\mu s\_\{t\}/c\_\{t\}\. Forttwithst=0s\_\{t\}=0,αt⋆=0\\alpha\_\{t\}^\{\\star\}=0is consistent with all KKT conditions and contributes nothing to either objective or constraint\.

The control constraint binds at the optimum \(else scaling allαt\\alpha\_\{t\}down by1−ϵ1\-\\epsilonwould strictly decrease the objective while remaining feasible\), so

∑tμ​st2ct=Etarget⟹μ⋆=Etarget∑tst2/ct\.\\sum\_\{t\}\\mu\\frac\{s\_\{t\}^\{2\}\}\{c\_\{t\}\}\\;=\\;E\_\{\\mathrm\{target\}\}\\quad\\Longrightarrow\\quad\\mu^\{\\star\}\\;=\\;\\frac\{E\_\{\\mathrm\{target\}\}\}\{\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\}\.Settingλ:=μ⋆\\lambda:=\\mu^\{\\star\}recovers the statedαt⋆=λ​st/ct\\alpha\_\{t\}^\{\\star\}=\\lambda s\_\{t\}/c\_\{t\}\. The optimal cost is

B⋆=12​∑tct​\(αt⋆\)2=12​λ2​∑tst2ct=Etarget22​∑tst2/ct,B^\{\\star\}\\;=\\;\\tfrac\{1\}\{2\}\\sum\_\{t\}c\_\{t\}\(\\alpha\_\{t\}^\{\\star\}\)^\{2\}\\;=\\;\\tfrac\{1\}\{2\}\\lambda^\{2\}\\sum\_\{t\}\\frac\{s\_\{t\}^\{2\}\}\{c\_\{t\}\}\\;=\\;\\frac\{E\_\{\\mathrm\{target\}\}^\{2\}\}\{2\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\},and the maximum shift at any budgetBBisE⋆=2​B⋅∑tst2/ctE^\{\\star\}=\\sqrt\{2B\\cdot\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\}\. Strict convexity of the objective \(sincect\>0c\_\{t\}\>0\) gives uniqueness\.

Part 2: uniform schedule shift\.Fix the uniform scheduleαt=α\\alpha\_\{t\}=\\alphafor alltt\. The budget constraint becomes

12​∑tct​α2=B⟹α=2​B∑tct\.\\tfrac\{1\}\{2\}\\sum\_\{t\}c\_\{t\}\\alpha^\{2\}\\;=\\;B\\quad\\Longrightarrow\\quad\\alpha\\;=\\;\\sqrt\{\\frac\{2B\}\{\\sum\_\{t\}c\_\{t\}\}\}\.The corresponding attribute shift is

Eunif=∑tα​st=α​∑tst=2​B∑tct⋅∑tst\.E\_\{\\mathrm\{unif\}\}\\;=\\;\\sum\_\{t\}\\alpha s\_\{t\}\\;=\\;\\alpha\\sum\_\{t\}s\_\{t\}\\;=\\;\\sqrt\{\\frac\{2B\}\{\\sum\_\{t\}c\_\{t\}\}\}\\cdot\\sum\_\{t\}s\_\{t\}\.
Part 3: Adaptive\-vs\-uniform efficiency ratio and CV identity\.Taking the ratio ofE⋆E^\{\\star\}andEunifE\_\{\\mathrm\{unif\}\}at matched budgetBB,

ρ2=\(E⋆\)2\(Eunif\)2=2​B⋅∑tst2/ct\(2​B/∑tct\)⋅\(∑tst\)2=\(∑tst2/ct\)​\(∑tct\)\(∑tst\)2\.\\rho^\{2\}\\;=\\;\\frac\{\(E^\{\\star\}\)^\{2\}\}\{\(E\_\{\\mathrm\{unif\}\}\)^\{2\}\}\\;=\\;\\frac\{2B\\cdot\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\}\{\(2B/\\sum\_\{t\}c\_\{t\}\)\\cdot\(\\sum\_\{t\}s\_\{t\}\)^\{2\}\}\\;=\\;\\frac\{\(\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\)\(\\sum\_\{t\}c\_\{t\}\)\}\{\(\\sum\_\{t\}s\_\{t\}\)^\{2\}\}\.This proves the first equality in \([10](https://arxiv.org/html/2605.10971#A1.E10)\)\.

For the CV form, define the cost\-weighted probability measureπt=ct/C\\pi\_\{t\}=c\_\{t\}/CwhereC=∑t′ct′C=\\sum\_\{t^\{\\prime\}\}c\_\{t^\{\\prime\}\}, and writeXt=st/ctX\_\{t\}=s\_\{t\}/c\_\{t\}\. Underπ\\pi,

𝔼π​\[X\]=∑tπt​Xt=1C​∑tst,𝔼π​\[X2\]=∑tπt​Xt2=1C​∑tst2ct\.\\mathbb\{E\}\_\{\\pi\}\[X\]\\;=\\;\\sum\_\{t\}\\pi\_\{t\}X\_\{t\}\\;=\\;\\frac\{1\}\{C\}\\sum\_\{t\}s\_\{t\},\\qquad\\mathbb\{E\}\_\{\\pi\}\[X^\{2\}\]\\;=\\;\\sum\_\{t\}\\pi\_\{t\}X\_\{t\}^\{2\}\\;=\\;\\frac\{1\}\{C\}\\sum\_\{t\}\\frac\{s\_\{t\}^\{2\}\}\{c\_\{t\}\}\.Substituting these into the ratio derived above,

ρ2=\(∑tst2/ct\)​\(∑tct\)\(∑tst\)2=C⋅𝔼π​\[X2\]⋅C\(C⋅𝔼π​\[X\]\)2=𝔼π​\[X2\]\(𝔼π​\[X\]\)2\.\\rho^\{2\}\\;=\\;\\frac\{\(\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\)\(\\sum\_\{t\}c\_\{t\}\)\}\{\(\\sum\_\{t\}s\_\{t\}\)^\{2\}\}\\;=\\;\\frac\{C\\cdot\\mathbb\{E\}\_\{\\pi\}\[X^\{2\}\]\\cdot C\}\{\(C\\cdot\\mathbb\{E\}\_\{\\pi\}\[X\]\)^\{2\}\}\\;=\\;\\frac\{\\mathbb\{E\}\_\{\\pi\}\[X^\{2\}\]\}\{\(\\mathbb\{E\}\_\{\\pi\}\[X\]\)^\{2\}\}\.By the elementary identity𝔼​\[X2\]/𝔼​\[X\]2=1\+Var​\(X\)/𝔼​\[X\]2=1\+CV2​\(X\)\\mathbb\{E\}\[X^\{2\}\]/\\mathbb\{E\}\[X\]^\{2\}=1\+\\mathrm\{Var\}\(X\)/\\mathbb\{E\}\[X\]^\{2\}=1\+\\mathrm\{CV\}^\{2\}\(X\),

ρ2=1\+CVc2​\(s/c\)\.\\rho^\{2\}\\;=\\;1\+\\mathrm\{CV\}\_\{c\}^\{2\}\(s/c\)\.
Equality condition\.ρ=1\\rho=1iffCVc​\(s/c\)=0\\mathrm\{CV\}\_\{c\}\(s/c\)=0iffst/cts\_\{t\}/c\_\{t\}is constant on\{t:ct\>0\}\\\{t:c\_\{t\}\>0\\\}\. Equivalently, by Cauchy–Schwarz applied toct\\sqrt\{c\_\{t\}\}andst/cts\_\{t\}/\\sqrt\{c\_\{t\}\},

\(∑tst\)2=\(∑tct⋅stct\)2≤\(∑tct\)​\(∑tst2ct\),\\Big\(\\sum\_\{t\}s\_\{t\}\\Big\)^\{2\}\\;=\\;\\Big\(\\sum\_\{t\}\\sqrt\{c\_\{t\}\}\\cdot\\frac\{s\_\{t\}\}\{\\sqrt\{c\_\{t\}\}\}\\Big\)^\{2\}\\;\\leq\\;\\Big\(\\sum\_\{t\}c\_\{t\}\\Big\)\\Big\(\\sum\_\{t\}\\frac\{s\_\{t\}^\{2\}\}\{c\_\{t\}\}\\Big\),with equality iffct∝st/ct\\sqrt\{c\_\{t\}\}\\propto s\_\{t\}/\\sqrt\{c\_\{t\}\}, i\.e\.,st∝cts\_\{t\}\\propto c\_\{t\}\. This recovers the same condition\. ∎

###### Corollary 1\(Specialization under slowly\-varying cost\)\.

Whenct≈cc\_\{t\}\\approx cfor alltt, the efficiency ratio simplifies to

ρ2≈T​∑tst2\(∑tst\)2=1\+CV2​\(s\),\\rho^\{2\}\\;\\approx\\;\\frac\{T\\sum\_\{t\}s\_\{t\}^\{2\}\}\{\(\\sum\_\{t\}s\_\{t\}\)^\{2\}\}\\;=\\;1\+\\mathrm\{CV\}^\{2\}\(s\),the squared coefficient of variation ofsts\_\{t\}under the uniform measure\. The optimum isαt⋆∝st\\alpha\_\{t\}^\{\\star\}\\propto s\_\{t\}and the achievable shift at fixed budget is governed by the dispersion ofsts\_\{t\}alone\. This is the regime in which the proxy of Appendix[A\.1\.6](https://arxiv.org/html/2605.10971#A1.SS1.SSS6)approximates the optimum most directly: whenctc\_\{t\}is approximately constant, the proxy’s tightness reduces to the alignment between activation growth andsts\_\{t\}\.

#### A\.1\.3Suboptimality of any feasible schedule

Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1)compares optimal to uniform; the same Cauchy–Schwarz machinery characterizes the suboptimality of any feasible schedule, including the proxy used in our experiments\.

###### Corollary 2\(Schedule efficiency\)\.

For any schedule\{αt≥0\}\\\{\\alpha\_\{t\}\\geq 0\\\}that exhausts the budget \(12​∑tct​αt2=B\\tfrac\{1\}\{2\}\\sum\_\{t\}c\_\{t\}\\alpha\_\{t\}^\{2\}=B\), the achieved attribute shift satisfies

EE⋆=cos⁡θ,θ=∠​\(\(αt​ct\)t,\(st/ct\)t\),\\frac\{E\}\{E^\{\\star\}\}\\;=\\;\\cos\\theta,\\qquad\\theta\\;=\\;\\angle\\\!\\Big\(\\big\(\\alpha\_\{t\}\\sqrt\{c\_\{t\}\}\\big\)\_\{t\},\\;\\big\(s\_\{t\}/\\sqrt\{c\_\{t\}\}\\big\)\_\{t\}\\Big\),with equalityE=E⋆E=E^\{\\star\}iffαt∝st/ct\\alpha\_\{t\}\\propto s\_\{t\}/c\_\{t\}\. Equivalently,B/B⋆=1/cos2⁡θB/B^\{\\star\}=1/\\cos^\{2\}\\thetaat matched control\.

###### Proof\.

Setut=αt​ctu\_\{t\}=\\alpha\_\{t\}\\sqrt\{c\_\{t\}\}andvt=st/ctv\_\{t\}=s\_\{t\}/\\sqrt\{c\_\{t\}\}\. The budget constraint is‖u‖2=2​B\\\|u\\\|^\{2\}=2Band the achieved shift isE=⟨u,v⟩E=\\langle u,v\\rangle\. From Part 1 of the proof of Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1),‖v‖2=∑tst2/ct\\\|v\\\|^\{2\}=\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}, and the maximum shift at budgetBBisE⋆=2​B⋅∑tst2/ct=‖u‖⋅‖v‖E^\{\\star\}=\\sqrt\{2B\\cdot\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\}=\\\|u\\\|\\cdot\\\|v\\\|\. By Cauchy–Schwarz,

E=⟨u,v⟩≤‖u‖​‖v‖=E⋆,EE⋆=⟨u,v⟩‖u‖​‖v‖=cos⁡θ,E\\;=\\;\\langle u,v\\rangle\\;\\leq\\;\\\|u\\\|\\,\\\|v\\\|\\;=\\;E^\{\\star\},\\qquad\\frac\{E\}\{E^\{\\star\}\}\\;=\\;\\frac\{\\langle u,v\\rangle\}\{\\\|u\\\|\\,\\\|v\\\|\}\\;=\\;\\cos\\theta,with equality iffu∝vu\\propto v, i\.e\.,αt∝st/ct\\alpha\_\{t\}\\propto s\_\{t\}/c\_\{t\}\.

For the cost\-ratio form: at matched control levelE¯\\bar\{E\}, suppose the heuristic schedule achievesE=E¯E=\\bar\{E\}at costBB, while the optimal schedule achievesE¯\\bar\{E\}at costB⋆B^\{\\star\}\. From theE⋆E^\{\\star\}formula,E⋆​\(B\)=2​B​‖v‖E^\{\\star\}\(B\)=\\sqrt\{2B\}\\,\\\|v\\\|andE⋆​\(B⋆\)=2​B⋆​‖v‖=E¯E^\{\\star\}\(B^\{\\star\}\)=\\sqrt\{2B^\{\\star\}\}\\,\\\|v\\\|=\\bar\{E\}, soB⋆=E¯2/\(2​‖v‖2\)B^\{\\star\}=\\bar\{E\}^\{2\}/\(2\\\|v\\\|^\{2\}\)\. From the cosine identity,E¯=E⋆​\(B\)​cos⁡θ=2​B​‖v‖​cos⁡θ\\bar\{E\}=E^\{\\star\}\(B\)\\cos\\theta=\\sqrt\{2B\}\\,\\\|v\\\|\\cos\\theta, soB=E¯2/\(2​‖v‖2​cos2⁡θ\)B=\\bar\{E\}^\{2\}/\(2\\\|v\\\|^\{2\}\\cos^\{2\}\\theta\)\. Dividing,

BB⋆=1cos2⁡θ\.\\frac\{B\}\{B^\{\\star\}\}\\;=\\;\\frac\{1\}\{\\cos^\{2\}\\theta\}\.∎

###### Corollary 3\(Cost of using a heuristic schedule shape\)\.

For any scheduleαt=κ​wt\\alpha\_\{t\}=\\kappa w\_\{t\}withwt≥0w\_\{t\}\\geq 0andκ\\kappachosen to exhaust the budget,

EE⋆=∑twt​st∑twt2​ct⋅∑tst2/ct\.\\frac\{E\}\{E^\{\\star\}\}\\;=\\;\\frac\{\\sum\_\{t\}w\_\{t\}s\_\{t\}\}\{\\sqrt\{\\sum\_\{t\}w\_\{t\}^\{2\}c\_\{t\}\}\\cdot\\sqrt\{\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\}\}\.The scalarκ\\kappacancels: efficiency depends only on the*shape*ofwtw\_\{t\}, not its magnitude\.

###### Proof\.

The budget constraint12​∑tct​\(κ​wt\)2=B\\tfrac\{1\}\{2\}\\sum\_\{t\}c\_\{t\}\(\\kappa w\_\{t\}\)^\{2\}=Bgivesκ=2​B/∑twt2​ct\\kappa=\\sqrt\{2B/\\sum\_\{t\}w\_\{t\}^\{2\}c\_\{t\}\}\. Substituting into Corollary[2](https://arxiv.org/html/2605.10971#Thmcorollary2), the angle factor in the numerator becomes⟨κ​w​c,s/c⟩=κ​∑twt​st\\langle\\kappa w\\sqrt\{c\},s/\\sqrt\{c\}\\rangle=\\kappa\\sum\_\{t\}w\_\{t\}s\_\{t\}, and‖κ​w​c‖=κ​∑twt2​ct=2​B\\\|\\kappa w\\sqrt\{c\}\\\|=\\kappa\\sqrt\{\\sum\_\{t\}w\_\{t\}^\{2\}c\_\{t\}\}=\\sqrt\{2B\}\. The ratioE/E⋆=cos⁡θE/E^\{\\star\}=\\cos\\thetafrom Corollary[2](https://arxiv.org/html/2605.10971#Thmcorollary2)expands as stated, andκ\\kappacancels between numerator and denominator\. ∎

This corollary licenses the practice in the experimental section of fixing the shape viawdynw\_\{\\mathrm\{dyn\}\}and sweepingα\\alphaseparately: only the shape determines the proxy’s distance from optimal in the Fisher\-weighted geometry\.

#### A\.1\.4Operational form ofctc\_\{t\}

The main text definesctc\_\{t\}as the second derivative of the per\-step output\-distribution KL atαt=0\\alpha\_\{t\}=0\. We give an explicit chain\-rule expression and verify positivity\.

Letut∈ℝdu\_\{t\}\\in\\mathbb\{R\}^\{d\}denote the steering direction in residual\-stream space at steptt: under composition rule \([1](https://arxiv.org/html/2605.10971#S4.E1)\),ut=WFaℓ​δ\(a\)u\_\{t\}=W\_\{F\_\{a\}^\{\\ell\}\}\\delta^\{\(a\)\}\. Letft:ℝd→ℝVf\_\{t\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{V\}be the map from the residual stream at the intervention layer to the output logits at steptt, with JacobianJt=∂ft/∂xtJ\_\{t\}=\\partial f\_\{t\}/\\partial x\_\{t\}\. LetFtout∈ℝV×VF\_\{t\}^\{\\mathrm\{out\}\}\\in\\mathbb\{R\}^\{V\\times V\}denote the Fisher information of the output categorical distribution with respect to its logits, evaluated at the unsteered logitsft​\(xt\)f\_\{t\}\(x\_\{t\}\)\.

###### Lemma 1\.

ct=ut⊤​Jt⊤​Ftout​Jt​utc\_\{t\}=u\_\{t\}^\{\\top\}J\_\{t\}^\{\\top\}F\_\{t\}^\{\\mathrm\{out\}\}J\_\{t\}u\_\{t\}, withct≥0c\_\{t\}\\geq 0\. Strict positivity holds wheneverJt​utJ\_\{t\}u\_\{t\}is not in the null space ofFtoutF\_\{t\}^\{\\mathrm\{out\}\}\.

###### Proof\.

Writing the perturbed logits asθ~​\(αt\)=ft​\(xt\+αt​ut\)\\tilde\{\\theta\}\(\\alpha\_\{t\}\)=f\_\{t\}\(x\_\{t\}\+\\alpha\_\{t\}u\_\{t\}\)and Taylor\-expanding the per\-step KL,

KL​\(pft​\(xt\)∥pθ~​\(αt\)\)=12​αt2⋅\(Jt​ut\)⊤​Ftout​\(Jt​ut\)\+O​\(αt3\)\.\\mathrm\{KL\}\\\!\\big\(p\_\{f\_\{t\}\(x\_\{t\}\)\}\\,\\big\\\|\\,p\_\{\\tilde\{\\theta\}\(\\alpha\_\{t\}\)\}\\big\)\\;=\\;\\tfrac\{1\}\{2\}\\alpha\_\{t\}^\{2\}\\cdot\(J\_\{t\}u\_\{t\}\)^\{\\top\}F\_\{t\}^\{\\mathrm\{out\}\}\(J\_\{t\}u\_\{t\}\)\+O\(\\alpha\_\{t\}^\{3\}\)\.The linear term vanishes becauseαt=0\\alpha\_\{t\}=0globally minimizes the KL \(non\-negative everywhere, zero at the origin\), forcing the gradient to vanish there\. Matching againstΔ​qt​\(αt\)=12​ct​αt2\+o​\(αt2\)\\Delta q\_\{t\}\(\\alpha\_\{t\}\)=\\tfrac\{1\}\{2\}c\_\{t\}\\alpha\_\{t\}^\{2\}\+o\(\\alpha\_\{t\}^\{2\}\)gives the stated form\. Positivity follows fromFtout⪰0F\_\{t\}^\{\\mathrm\{out\}\}\\succeq 0, with strict inequality whenJt​utJ\_\{t\}u\_\{t\}has a non\-uniform component \(the null space ofFtoutF\_\{t\}^\{\\mathrm\{out\}\}for a categorical distribution is the all\-ones direction in logit space, corresponding to logit shifts that leave the softmax invariant\)\. ∎

##### Remark\.

ctc\_\{t\}is the pullback of the output Fisher through the layers above the intervention site\. It is not the Fisher information of a parametric family in the standard sense; calling it “Fisher information” without specifying the family would be a misuse of the term\. The operational characterization above is what enters the optimization; the chain\-rule expression is included for completeness and is not used directly in our schedule\. The global hypothesisct\>0c\_\{t\}\>0in Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1)holds operationally becauseJt​utJ\_\{t\}u\_\{t\}is generically non\-uniform \(the steering directionut=WFaℓ​δ\(a\)u\_\{t\}=W\_\{F\_\{a\}^\{\\ell\}\}\\delta^\{\(a\)\}has support across many output dimensions, and the layers above the intervention site mix coordinates\), placingJt​utJ\_\{t\}u\_\{t\}outside the all\-ones null space ofFtoutF\_\{t\}^\{\\mathrm\{out\}\}\.

#### A\.1\.5Position\-wise decomposition of the local gain

The local gainst=∂g/∂αts\_\{t\}=\\partial g/\\partial\\alpha\_\{t\}admits a position\-wise decomposition into direct and indirect pathways\. The direct pathway provably localizes to masked positions; the indirect pathway is non\-zero in general but is sub\-leading in the regime supported by our measurements\.

##### Setup\.

Fix a denoising trajectory and a steptt\. The steering operation applies a residual\-stream perturbationαt​ut\\alpha\_\{t\}u\_\{t\}at steptt, whereut=WFaℓ​δ\(a\)∈ℝdu\_\{t\}=W\_\{F\_\{a\}^\{\\ell\}\}\\delta^\{\(a\)\}\\in\\mathbb\{R\}^\{d\}is the same direction at every sequence position \(the SAE injection is position\-wise but uses the sameδ\(a\)\\delta^\{\(a\)\}across positions\)\. Letℳt⊆\{1,…,L\}\\mathcal\{M\}\_\{t\}\\subseteq\\\{1,\\ldots,L\\\}denote the set of masked positions at steptt; we condition on the masking schedule \(which positions are unmasked when\), so that the only stochasticity is in token content\. For unmasked positionsp∉ℳtp\\notin\\mathcal\{M\}\_\{t\}, the realized tokentokenp\\mathrm\{token\}\_\{p\}was determined at the step at whichppwas unmasked, prior to steptt, and is fixed conditional on the trajectory up to that step\.

###### Lemma 2\(Direct\-effect localization\)\.

Under the setup above, the local gain decomposes as

st=∑q∂g∂tokenq⋅∂tokenq∂xt,q⋅∂xt,q∂αt⏟direct effect​stdir\+∑q∂g∂tokenq​∑p≠q∂tokenq∂xt,p⋅∂xt,p∂αt⏟indirect effect​stind,s\_\{t\}\\;=\\;\\underbrace\{\\sum\_\{q\}\\frac\{\\partial g\}\{\\partial\\,\\mathrm\{token\}\_\{q\}\}\\cdot\\frac\{\\partial\\,\\mathrm\{token\}\_\{q\}\}\{\\partial x\_\{t,q\}\}\\cdot\\frac\{\\partial x\_\{t,q\}\}\{\\partial\\alpha\_\{t\}\}\}\_\{\\text\{direct effect \}s\_\{t\}^\{\\mathrm\{dir\}\}\}\\;\+\\;\\underbrace\{\\sum\_\{q\}\\frac\{\\partial g\}\{\\partial\\,\\mathrm\{token\}\_\{q\}\}\\,\\sum\_\{p\\neq q\}\\frac\{\\partial\\,\\mathrm\{token\}\_\{q\}\}\{\\partial x\_\{t,p\}\}\\cdot\\frac\{\\partial x\_\{t,p\}\}\{\\partial\\alpha\_\{t\}\}\}\_\{\\text\{indirect effect \}s\_\{t\}^\{\\mathrm\{ind\}\}\},where the direct sum collects thep=qp=qdiagonal of the position\-by\-position chain rule and the indirect sum collects the off\-diagonal terms\. The direct effect reduces to a sum over masked positions:

stdir=∑q∈ℳt∂g∂tokenq⋅∂tokenq∂xt,q⋅∂xt,q∂αt\.s\_\{t\}^\{\\mathrm\{dir\}\}\\;=\\;\\sum\_\{q\\in\\mathcal\{M\}\_\{t\}\}\\frac\{\\partial g\}\{\\partial\\,\\mathrm\{token\}\_\{q\}\}\\cdot\\frac\{\\partial\\,\\mathrm\{token\}\_\{q\}\}\{\\partial x\_\{t,q\}\}\\cdot\\frac\{\\partial x\_\{t,q\}\}\{\\partial\\alpha\_\{t\}\}\.

###### Proof\.

Forq∉ℳtq\\notin\\mathcal\{M\}\_\{t\}, the realizedtokenq\\mathrm\{token\}\_\{q\}was determined at the step at whichqqwas unmasked \(prior to steptt\) and does not depend onxt,qx\_\{t,q\}\. By the chain rule,∂tokenq/∂xt,q=0\\partial\\mathrm\{token\}\_\{q\}/\\partial x\_\{t,q\}=0, so the diagonalp=qp=qterm vanishes\. The direct\-effect sum therefore restricts toq∈ℳtq\\in\\mathcal\{M\}\_\{t\}\. ∎

##### Indirect pathway\.

The lemma controls only the direct effect\. The indirect effect at unmasked positionppis non\-zero in general: althoughtokenp\\mathrm\{token\}\_\{p\}is fixed, the residual\-stream activationxt,px\_\{t,p\}enters the key/value projections used by self\-attention at masked positionsq∈ℳt′q\\in\\mathcal\{M\}\_\{t^\{\\prime\}\}fort′≥tt^\{\\prime\}\\geq t, and so can influencetokenq\\mathrm\{token\}\_\{q\}at those positions\. Whether the indirect path is sub\-leading relative to the direct path is a regime\-dependent question that we cannot settle in closed form\.

We treat direct\-path dominance as a modeling assumption supported by the masked\-fraction analysis of Appendix[D\.4](https://arxiv.org/html/2605.10971#A4.SS4): across our four models, attribute\-relevant features fire preferentially on masked positions throughout the early\-to\-middle trajectory, with masked\-fraction values ranging from 0\.55 to 0\.80 for anticipatory attributes\. This is consistent with — though does not prove — direct\-path dominance, since features that fire on masked positions are positioned to exert direct influence on the tokens decoded there\. The framework’s predictive accuracy in Section[5](https://arxiv.org/html/2605.10971#S5)\(especially the LLaDA predictionρ≈1\\rho\\approx 1confirmed by Adaptive≈\\approxUniform\) provides post\-hoc validation that the assumption is operationally adequate\.

#### A\.1\.6Connecting the optimal schedule to the active\-set proxy

Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1)prescribesαt⋆∝st/ct\\alpha\_\{t\}^\{\\star\}\\propto s\_\{t\}/c\_\{t\}, but neithersts\_\{t\}norctc\_\{t\}is directly observable:sts\_\{t\}requires gradient information about a target attribute classifier on the unrolled denoising trajectory, andctc\_\{t\}requires the Jacobian\-pullback of Lemma[1](https://arxiv.org/html/2605.10971#Thmlemma1)\. Our schedulewdynw\_\{\\mathrm\{dyn\}\}uses positive activation growth as a proxy\. We justify the proxy through representational commitment under the TopK gate and characterize when it is tight\.

##### Active\-set transitions as a measure of representational commitment\.

The TopK gate produces, at each stepttand each input, a sparse active set𝒜t⊆\{1,…,dSAE\}\\mathcal\{A\}\_\{t\}\\subseteq\\\{1,\\ldots,d\_\{\\mathrm\{SAE\}\}\\\}of features whose pre\-gate scores exceed the top\-kkthreshold\. The steering operation itself does*not*depend on𝒜t\\mathcal\{A\}\_\{t\}: the residual\-stream perturbationα​WFaℓ​δ\(a\)\\alpha\\,W\_\{F\_\{a\}^\{\\ell\}\}\\delta^\{\(a\)\}is added directly to the post\-encode hidden state regardless of which features are currently active\. The proxy is therefore not a statement about gradient localization through the gate, but about the model’s*use*of attribute\-relevant directions\. When attribute featuresj∈Faℓj\\in F\_\{a\}^\{\\ell\}enter𝒜t\\mathcal\{A\}\_\{t\}on average across the contrastive corpus, the model is, on average, beginning to compute on those directions; subsequent layers read out from the residual stream, and a perturbation along directions the model is starting to use propagates through more of the downstream computation than a perturbation along directions still effectively unused\.

This rationale is intuitive rather than rigorous: we cannot prove that residual\-space perturbations along high\-growth directions produce larger∂g/∂αt\\partial g/\\partial\\alpha\_\{t\}than perturbations along low\-growth directions, only that this is plausible given how the model’s forward pass distributes information\. The framework’s predictive accuracy in Section[5](https://arxiv.org/html/2605.10971#S5)provides post\-hoc validation\.

##### Why activation growth, not activation level\.

A natural alternative would be to weight the schedule by raw activation levelAt\(a,ℓ\)=∑j∈Faℓh¯j\(t\)A\_\{t\}^\{\(a,\\ell\)\}=\\sum\_\{j\\in F\_\{a\}^\{\\ell\}\}\\bar\{h\}\_\{j\}^\{\(t\)\}\. This over\-weights steps where attribute features are already stably active across the corpus: the model has already committed to those features, and perturbing further along directions that are already saturated may have diminishing marginal effect\. Activation growth\[Δ​At\(a,ℓ\)\]\+\[\\Delta A\_\{t\}^\{\(a,\\ell\)\}\]\_\{\+\}targets steps where features are crossing into the active set on the corpus, where the model’s representational state for that attribute is most malleable\. The asymmetric clamping at zero is necessary: steps with negative growth correspond to features dropping out of the active set, which we do not want to amplify steering on\.

##### Proxy fidelity and matched\-shape efficiency\.

By Corollary[3](https://arxiv.org/html/2605.10971#Thmcorollary3), the efficiency of the proxy schedule relative to optimal is

EproxyE⋆=∑twdyn​\(t\)​st∑twdyn​\(t\)2​ct⋅∑tst2/ct=cos⁡θproxy,\\frac\{E\_\{\\mathrm\{proxy\}\}\}\{E^\{\\star\}\}\\;=\\;\\frac\{\\sum\_\{t\}w\_\{\\mathrm\{dyn\}\}\(t\)s\_\{t\}\}\{\\sqrt\{\\sum\_\{t\}w\_\{\\mathrm\{dyn\}\}\(t\)^\{2\}c\_\{t\}\}\\cdot\\sqrt\{\\sum\_\{t\}s\_\{t\}^\{2\}/c\_\{t\}\}\}\\;=\\;\\cos\\theta\_\{\\mathrm\{proxy\}\},whereθproxy\\theta\_\{\\mathrm\{proxy\}\}is the angle betweenwdyn​\(t\)​ctw\_\{\\mathrm\{dyn\}\}\(t\)\\sqrt\{c\_\{t\}\}andst/cts\_\{t\}/\\sqrt\{c\_\{t\}\}\. The proxy is exactly optimal when \(i\) active\-set growth is proportional tost/cts\_\{t\}/c\_\{t\}and \(ii\)ctc\_\{t\}varies slowly relative tosts\_\{t\}, so the geometry collapses to the constant\-cost case of Corollary[1](https://arxiv.org/html/2605.10971#Thmcorollary1)\.

##### When the proxy is loose, and what we observe\.

Two regimes follow from Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1)and Corollary[3](https://arxiv.org/html/2605.10971#Thmcorollary3):

- •*Sharp regime \(CVc​\(s/c\)\\mathrm\{CV\}\_\{c\}\(s/c\)large\)\.*The optimal schedule concentrates budget on a few steps, and any schedule whose shape matches that concentration recovers most ofE⋆E^\{\\star\}\. On MDLM, where block 0 captures essentially all of topic emergence, the activation\-growth profile is itself sharp, and Adaptive substantially outperforms Uniform at matched control — consistent with the proxy aligning with the sensitivity profile\.
- •*Flat regime \(CVc​\(s/c\)≈0\\mathrm\{CV\}\_\{c\}\(s/c\)\\approx 0\)\.*By Theorem[1](https://arxiv.org/html/2605.10971#Thmtheorem1),ρ≈1\\rho\\approx 1and no schedule can improve over uniform regardless of the algorithm\. On LLaDA, the activation\-growth profile is approximately uniform across blocks, and Adaptive performs comparably to Uniform — a confirmation of the framework rather than of the proxy’s tightness\.

The proxy can therefore be loose without being harmful: in the flat regime, any schedule does about equally well; in the sharp regime, the proxy tracks the optimal shape closely enough to recover most of the available efficiency ratio\. The regime where the proxy would actively underperform Uniform is one where activation growth and sensitivity are anti\-correlated\. We do not have direct evidence that this regime is avoided in our four models — our argument that Adaptive does not underperform Uniform is empirical \(Table[3](https://arxiv.org/html/2605.10971#S5.T3)\) rather than structural\.

### A\.2Decoder\-space interference: bound and analysis

The main text states the decoder\-Gram bound \([7](https://arxiv.org/html/2605.10971#S4.E7)\) as a worst\-case guarantee on residual\-space coupling under disjoint feature sets\. The empirical coupling we measure \(Appendix[A\.3](https://arxiv.org/html/2605.10971#A1.SS3)\) is roughly an order of magnitude tighter than the bound; we therefore present the bound as a lemma, with the empirical observation as the operative claim, and analyze the gap between them\.

##### Assumption 1 \(Disjoint feature sets\)\.

For alli≠ji\\neq j, the feature sets satisfy

Faiℓ∩Fajℓ=∅\.F\_\{a\_\{i\}\}^\{\\ell\}\\cap F\_\{a\_\{j\}\}^\{\\ell\}=\\emptyset\.The small\-overlap correction for the empirically near\-disjoint regime, where\|Faiℓ∩Fajℓ\|≤2\|F\_\{a\_\{i\}\}^\{\\ell\}\\cap F\_\{a\_\{j\}\}^\{\\ell\}\|\\leq 2, is discussed in App\. §[A\.2\.4](https://arxiv.org/html/2605.10971#A1.SS2.SSS4)\.

##### Assumption 2 \(Positive\-definite intra\-attribute Gram matrices\)\.

For each attributeaia\_\{i\}, the intra\-attribute Gram matrix

G\(i​i\)=WFaiℓ⊤​WFaiℓG^\{\(ii\)\}=W\_\{F\_\{a\_\{i\}\}^\{\\ell\}\}^\{\\top\}W\_\{F\_\{a\_\{i\}\}^\{\\ell\}\}is positive definite on its support\. Equivalently, the decoder columns indexed byFaiℓF\_\{a\_\{i\}\}^\{\\ell\}are linearly independent\. This condition generically holds when\|Faiℓ\|≪d\|F\_\{a\_\{i\}\}^\{\\ell\}\|\\ll dand decoder columns are unit normalized\.

##### Assumption 3 \(Nonzero perturbations\)\.

The perturbations satisfy

Δ​xt\(i\)≠0,Δ​xt\(j\)≠0,\\Delta x\_\{t\}^\{\(i\)\}\\neq 0,\\qquad\\Delta x\_\{t\}^\{\(j\)\}\\neq 0,ensuring that the cosine term in Eq\. \([7](https://arxiv.org/html/2605.10971#S4.E7)\) is well defined\. This condition holds wheneverαeff​\(ai,t,ℓ\)\>0\\alpha\_\{\\mathrm\{eff\}\}\(a\_\{i\},t,\\ell\)\>0,δ\(ai\)\\delta^\{\(a\_\{i\}\)\}has a nonzero component, and the ReLU clamp does not zero the entire perturbation overFaiℓF\_\{a\_\{i\}\}^\{\\ell\}\.

Assumption 1 provides the structural precondition of the proposition\. Assumption 2 guarantees that the denominator of Eq\. \([7](https://arxiv.org/html/2605.10971#S4.E7)\) is strictly positive\. Finally, Assumption 3 ensures that the cosine similarity is well defined\.

#### A\.2\.1Subspace containment and the role of ReLU

The bound is stated for post\-ReLU residual perturbations\. We first establish that disjointness places each attribute’s perturbation in its own decoder subspace and that ReLU preserves this structure\.

###### Lemma 3\(Subspace containment under disjointness\)\.

LetΔ​xt\(i\)=x~t−x~t\[∖i\]\\Delta x\_\{t\}^\{\(i\)\}=\\tilde\{x\}\_\{t\}\-\\tilde\{x\}\_\{t\}^\{\[\\setminus i\]\}denote the residual\-space contribution of attributeiiunder composition rule \([3](https://arxiv.org/html/2605.10971#S4.E3)\), wherex~t\[∖i\]\\tilde\{x\}\_\{t\}^\{\[\\setminus i\]\}is the steered reconstruction with attributeiiomitted\. IfFaiℓ∩Fajℓ=∅F^\{\\ell\}\_\{a\_\{i\}\}\\cap F^\{\\ell\}\_\{a\_\{j\}\}=\\emptysetfor allj≠ij\\neq i, thenΔ​xt\(i\)∈span​\(WFaiℓ\)\\Delta x\_\{t\}^\{\(i\)\}\\in\\mathrm\{span\}\(W\_\{F^\{\\ell\}\_\{a\_\{i\}\}\}\), and the ReLU clamp in \([3](https://arxiv.org/html/2605.10971#S4.E3)\) alters the magnitudes withinΔ​ht\(i\):=h~t−h~t\[∖i\]\\Delta h\_\{t\}^\{\(i\)\}:=\\tilde\{h\}\_\{t\}\-\\tilde\{h\}\_\{t\}^\{\[\\setminus i\]\}but preserves its support onFaiℓF^\{\\ell\}\_\{a\_\{i\}\}\.

###### Proof\.

Write the pre\-ReLU composed feature update ash^t,j=ht,j\+∑i:j∈Faiℓαeff​\(ai,t,ℓ\)​δj\(ai\)\\hat\{h\}\_\{t,j\}=h\_\{t,j\}\+\\sum\_\{i:\\,j\\in F^\{\\ell\}\_\{a\_\{i\}\}\}\\alpha\_\{\\mathrm\{eff\}\}\(a\_\{i\},t,\\ell\)\\delta\_\{j\}^\{\(a\_\{i\}\)\}\. Under disjointness, the inner sum has at most one term: for anyjj, eitherj∈Faiℓj\\in F^\{\\ell\}\_\{a\_\{i\}\}for exactly oneii\(call iti​\(j\)i\(j\)\), orjjlies in no attribute set\. Consequentlyh^t,j−ht,j\\hat\{h\}\_\{t,j\}\-h\_\{t,j\}depends on attributei​\(j\)i\(j\)alone\.

The ReLU clamph~t,j=ReLU​\(h^t,j\)\\tilde\{h\}\_\{t,j\}=\\mathrm\{ReLU\}\(\\hat\{h\}\_\{t,j\}\)acts coordinate\-wise; its clipping decision atjjdepends only onh^t,j\\hat\{h\}\_\{t,j\}, which depends on only one attribute\. Thereforeh~t,j−h~t,j\[∖i\]\\tilde\{h\}\_\{t,j\}\-\\tilde\{h\}\_\{t,j\}^\{\[\\setminus i\]\}is non\-zero only forj∈Faiℓj\\in F^\{\\ell\}\_\{a\_\{i\}\}, regardless of whether ReLU clipped the value\. ApplyingWdecW\_\{\\mathrm\{dec\}\},

Δ​xt\(i\)=Wdec​Δ​ht\(i\)=WFaiℓ​\(Δ​ht\(i\)\)\|Faiℓ∈span​\(WFaiℓ\)\.\\Delta x\_\{t\}^\{\(i\)\}\\;=\\;W\_\{\\mathrm\{dec\}\}\\Delta h\_\{t\}^\{\(i\)\}\\;=\\;W\_\{F^\{\\ell\}\_\{a\_\{i\}\}\}\\big\(\\Delta h\_\{t\}^\{\(i\)\}\\big\)\\big\|\_\{F^\{\\ell\}\_\{a\_\{i\}\}\}\\;\\in\\;\\mathrm\{span\}\\big\(W\_\{F^\{\\ell\}\_\{a\_\{i\}\}\}\\big\)\.ReLU may shrink magnitudes onFaiℓF^\{\\ell\}\_\{a\_\{i\}\}but cannot move mass off the support\. ∎

##### What disjointness buys\.

A weaker version of subspace containment — that disjoint feature*indices*produce non\-overlapping coordinate updates — follows trivially from the structure of \([3](https://arxiv.org/html/2605.10971#S4.E3)\)\. Lemma[3](https://arxiv.org/html/2605.10971#Thmlemma3)is stronger: it controls the post\-ReLU residual perturbation, not just the feature update, by exploiting that ReLU is coordinate\-wise and that disjointness localizes its decision per attribute\. This is what makes the Gram bound apply to the actual perturbations rather than only to a first\-order linearization\.

#### A\.2\.2Decoder\-Gram bound

###### Lemma 4\(Decoder\-Gram interference bound\)\.

LetG=Wdec⊤​WdecG=W\_\{\\mathrm\{dec\}\}^\{\\top\}W\_\{\\mathrm\{dec\}\}and letG\(i​j\)G^\{\(ij\)\}denote its submatrix indexed byFaiℓ×FajℓF^\{\\ell\}\_\{a\_\{i\}\}\\times F^\{\\ell\}\_\{a\_\{j\}\}\. IfFaiℓ∩Fajℓ=∅F^\{\\ell\}\_\{a\_\{i\}\}\\cap F^\{\\ell\}\_\{a\_\{j\}\}=\\emptyset, then for any post\-ReLU residual perturbationsΔ​xt\(i\),Δ​xt\(j\)\\Delta x\_\{t\}^\{\(i\)\},\\Delta x\_\{t\}^\{\(j\)\}produced by \([3](https://arxiv.org/html/2605.10971#S4.E3)\),

\|cos⁡\(Δ​xt\(i\),Δ​xt\(j\)\)\|≤σmax​\(G\(i​j\)\)σmin​\(G\(i​i\)\)​σmin​\(G\(j​j\)\)\.\\big\|\\cos\\\!\\big\(\\Delta x\_\{t\}^\{\(i\)\},\\,\\Delta x\_\{t\}^\{\(j\)\}\\big\)\\big\|\\;\\leq\\;\\frac\{\\sigma\_\{\\max\}\(G^\{\(ij\)\}\)\}\{\\sqrt\{\\sigma\_\{\\min\}\(G^\{\(ii\)\}\)\\,\\sigma\_\{\\min\}\(G^\{\(jj\)\}\)\}\}\.

###### Proof\.

By Lemma[3](https://arxiv.org/html/2605.10971#Thmlemma3),Δ​xt\(i\)=WFaiℓ​Δ​ht\(i\)\\Delta x\_\{t\}^\{\(i\)\}=W\_\{F^\{\\ell\}\_\{a\_\{i\}\}\}\\Delta h\_\{t\}^\{\(i\)\}withΔ​ht\(i\)\\Delta h\_\{t\}^\{\(i\)\}supported onFaiℓF^\{\\ell\}\_\{a\_\{i\}\}, similarly for attributejj\. Writeδ\(i\)=\(Δ​ht\(i\)\)\|Faiℓ\\delta^\{\(i\)\}=\(\\Delta h\_\{t\}^\{\(i\)\}\)\|\_\{F^\{\\ell\}\_\{a\_\{i\}\}\}\. Then

⟨Δ​xt\(i\),Δ​xt\(j\)⟩=⟨WFaiℓ​δ\(i\),WFajℓ​δ\(j\)⟩=δ\(i\)⊤​G\(i​j\)​δ\(j\)\.\\big\\langle\\Delta x\_\{t\}^\{\(i\)\},\\,\\Delta x\_\{t\}^\{\(j\)\}\\big\\rangle\\;=\\;\\big\\langle W\_\{F^\{\\ell\}\_\{a\_\{i\}\}\}\\delta^\{\(i\)\},\\,W\_\{F^\{\\ell\}\_\{a\_\{j\}\}\}\\delta^\{\(j\)\}\\big\\rangle\\;=\\;\\delta^\{\(i\)\\top\}G^\{\(ij\)\}\\delta^\{\(j\)\}\.By the operator\-norm definition ofσmax\\sigma\_\{\\max\},

\|δ\(i\)⊤​G\(i​j\)​δ\(j\)\|≤‖δ\(i\)‖⋅‖G\(i​j\)​δ\(j\)‖≤σmax​\(G\(i​j\)\)​‖δ\(i\)‖​‖δ\(j\)‖\.\\big\|\\delta^\{\(i\)\\top\}G^\{\(ij\)\}\\delta^\{\(j\)\}\\big\|\\;\\leq\\;\\\|\\delta^\{\(i\)\}\\\|\\cdot\\\|G^\{\(ij\)\}\\delta^\{\(j\)\}\\\|\\;\\leq\\;\\sigma\_\{\\max\}\(G^\{\(ij\)\}\)\\,\\\|\\delta^\{\(i\)\}\\\|\\,\\\|\\delta^\{\(j\)\}\\\|\.For each attribute,

‖Δ​xt\(i\)‖2=δ\(i\)⊤​G\(i​i\)​δ\(i\)≥σmin​\(G\(i​i\)\)​‖δ\(i\)‖2,\\\|\\Delta x\_\{t\}^\{\(i\)\}\\\|^\{2\}\\;=\\;\\delta^\{\(i\)\\top\}G^\{\(ii\)\}\\delta^\{\(i\)\}\\;\\geq\\;\\sigma\_\{\\min\}\(G^\{\(ii\)\}\)\\,\\\|\\delta^\{\(i\)\}\\\|^\{2\},where positive definiteness ofG\(i​i\)G^\{\(ii\)\}on the support ofδ\(i\)\\delta^\{\(i\)\}ensuresσmin​\(G\(i​i\)\)\>0\\sigma\_\{\\min\}\(G^\{\(ii\)\}\)\>0\. Combining,

\|cos⁡\(Δ​xt\(i\),Δ​xt\(j\)\)\|=\|δ\(i\)⊤​G\(i​j\)​δ\(j\)\|‖Δ​xt\(i\)‖​‖Δ​xt\(j\)‖≤σmax​\(G\(i​j\)\)σmin​\(G\(i​i\)\)​σmin​\(G\(j​j\)\)\.\\big\|\\cos\\\!\\big\(\\Delta x\_\{t\}^\{\(i\)\},\\,\\Delta x\_\{t\}^\{\(j\)\}\\big\)\\big\|\\;=\\;\\frac\{\|\\delta^\{\(i\)\\top\}G^\{\(ij\)\}\\delta^\{\(j\)\}\|\}\{\\\|\\Delta x\_\{t\}^\{\(i\)\}\\\|\\,\\\|\\Delta x\_\{t\}^\{\(j\)\}\\\|\}\\;\\leq\\;\\frac\{\\sigma\_\{\\max\}\(G^\{\(ij\)\}\)\}\{\\sqrt\{\\sigma\_\{\\min\}\(G^\{\(ii\)\}\)\\,\\sigma\_\{\\min\}\(G^\{\(jj\)\}\)\}\}\.The bound is independent ofδ\(i\),δ\(j\)\\delta^\{\(i\)\},\\delta^\{\(j\)\}, so it holds uniformly over post\-ReLU perturbations and is computable a priori from the SAE decoder\. ∎

#### A\.2\.3Why the bound is small in practice

Three structural properties of SAE\-based steering favor a small bound\.

##### Contrastive selection produces near\-orthogonal feature sets\.

Features are selected by attribute\-specific Cohen’sdd\(\|dj\|≥0\.2\|d\_\{j\}\|\\geq 0\.2,p<0\.01p<0\.01via Mann–Whitney\)\. A feature with high\|d\|\|d\|for sentiment typically has low\|d\|\|d\|for topic and vice versa, because sentiment and topic separate text along weakly correlated semantic axes\. Empirically,\|Faiℓ∩Fajℓ\|≤2\|F^\{\\ell\}\_\{a\_\{i\}\}\\cap F^\{\\ell\}\_\{a\_\{j\}\}\|\\leq 2for all model–attribute pairs in our experiments, so off\-diagonal blocksG\(i​j\)G^\{\(ij\)\}are populated almost entirely by inner products between decoder columns selected for different attributes\.

##### Overcompleteness and unit\-norm decoder columns suppress off\-diagonal correlation\.

Our SAEs use expansion factors4×4\\timesto16×16\\times\(K∈\{12,288,14,336,16,384\}K\\in\\\{12\{,\}288,14\{,\}336,16\{,\}384\\\}vs\.d∈\{768,3,584,4,096\}d\\in\\\{768,3\{,\}584,4\{,\}096\\\}\)\. Decoder columns are normalized to unit norm after each training step\. Trained overcomplete dictionaries withK≫dK\\gg dare observed to exhibit approximate incoherence, with\|⟨wj,wk⟩\|\|\\langle w\_\{j\},w\_\{k\}\\rangle\|small for typicalj≠kj\\neq k, in line with whatDonoho and Elad \[[15](https://arxiv.org/html/2605.10971#bib.bib5)\]establish for optimal incoherent dictionaries \(their result is for random/optimal dictionaries; for trained SAEs the same scaling is empirical, not guaranteed\)\. The Frobenius bound

σmax​\(G\(i​j\)\)≤‖G\(i​j\)‖F≤\|Faiℓ\|⋅\|Fajℓ\|⋅O​\(1/d\)\\sigma\_\{\\max\}\(G^\{\(ij\)\}\)\\;\\leq\\;\\\|G^\{\(ij\)\}\\\|\_\{F\}\\;\\leq\\;\\sqrt\{\|F^\{\\ell\}\_\{a\_\{i\}\}\|\\cdot\|F^\{\\ell\}\_\{a\_\{j\}\}\|\}\\cdot O\(1/\\sqrt\{d\}\)is loose in the rank dimension but captures the qualitative scaling: the bound shrinks asddgrows\. The diagonal blocksG\(i​i\)G^\{\(ii\)\}have unit diagonal and small off\-diagonal entries, so by Gershgorin’s theoremσmin​\(G\(i​i\)\)≥1−\(\|Faiℓ\|−1\)⋅O​\(1/d\)\\sigma\_\{\\min\}\(G^\{\(ii\)\}\)\\geq 1\-\(\|F^\{\\ell\}\_\{a\_\{i\}\}\|\-1\)\\cdot O\(1/\\sqrt\{d\}\), bounded away from zero whenever\|Faiℓ\|≪d\|F^\{\\ell\}\_\{a\_\{i\}\}\|\\ll\\sqrt\{d\}\.

##### Realized Gram values\.

Appendix[A\.3](https://arxiv.org/html/2605.10971#A1.SS3)reports the realized bound on Lemma[4](https://arxiv.org/html/2605.10971#Thmlemma4)for our trained SAEs at the layers used for steering, with\|Faℓ\|=50\|F^\{\\ell\}\_\{a\}\|=50\. The realized cosine coupling is at most 0\.09 across disjoint feature sets \(mean 0\.04\), an order of magnitude tighter than the worst\-case bound\. This is consistent with the empirical cross\-attribute interference reported in Figure[14](https://arxiv.org/html/2605.10971#A6.F14)\.

##### Contrast with dense steering directions\.

Dense methods \(PCA, linear probes, contrastive vectors\) steer by adding a single directionva∈ℝdv\_\{a\}\\in\\mathbb\{R\}^\{d\}\. For two attributes the cross\-cosine is justcos⁡\(va1,va2\)\\cos\(v\_\{a\_\{1\}\},v\_\{a\_\{2\}\}\)— a single number, with no structural mechanism to keep it small\. We measure values in\[0\.1,0\.4\]\[0\.1,0\.4\]for sentiment vs\. topic across our models, large enough to produce the cross\-attribute interference visible in Figure[14](https://arxiv.org/html/2605.10971#A6.F14)for the dense baselines\. SAE steering replaces this single coupling with the sparserσmax​\(G\(i​j\)\)\\sigma\_\{\\max\}\(G^\{\(ij\)\}\)regime above\.

#### A\.2\.4Small\-overlap correction

Strict disjointnessFaiℓ∩Fajℓ=∅F^\{\\ell\}\_\{a\_\{i\}\}\\cap F^\{\\ell\}\_\{a\_\{j\}\}=\\emptysetis required for Lemma[3](https://arxiv.org/html/2605.10971#Thmlemma3)\. Contrastive selection produces near\-disjointness, with\|Faiℓ∩Fajℓ\|≤2\|F^\{\\ell\}\_\{a\_\{i\}\}\\cap F^\{\\ell\}\_\{a\_\{j\}\}\|\\leq 2in our experiments\. We characterize the resulting first\-order correction\.

LetSi​j=Faiℓ∩FajℓS\_\{ij\}=F^\{\\ell\}\_\{a\_\{i\}\}\\cap F^\{\\ell\}\_\{a\_\{j\}\}\. OnSi​jS\_\{ij\}, the pre\-ReLU update at coordinatej∈Si​jj\\in S\_\{ij\}depends on both attributes:h^t,j=ht,j\+αeff​\(ai,t,ℓ\)​δj\(ai\)\+αeff​\(aj,t,ℓ\)​δj\(aj\)\\hat\{h\}\_\{t,j\}=h\_\{t,j\}\+\\alpha\_\{\\mathrm\{eff\}\}\(a\_\{i\},t,\\ell\)\\delta\_\{j\}^\{\(a\_\{i\}\)\}\+\\alpha\_\{\\mathrm\{eff\}\}\(a\_\{j\},t,\\ell\)\\delta\_\{j\}^\{\(a\_\{j\}\)\}\. ReLU’s clipping decision atjjdepends on the sum, introducing genuine cross\-attribute coupling\. Decompose

Δ​xt\(i\)=Δ​xt\(i,disjoint\)\+Δ​xt\(i,overlap\),\\Delta x\_\{t\}^\{\(i\)\}\\;=\\;\\Delta x\_\{t\}^\{\(i,\\mathrm\{disjoint\}\)\}\+\\Delta x\_\{t\}^\{\(i,\\mathrm\{overlap\}\)\},where the first term is the contribution fromFaiℓ∖Si​jF^\{\\ell\}\_\{a\_\{i\}\}\\setminus S\_\{ij\}\(covered by Lemma[3](https://arxiv.org/html/2605.10971#Thmlemma3)\) and the second is fromSi​jS\_\{ij\}\. The Gram bound applies to the disjoint part\. The overlap part contributes

∥Δxt\(i,overlap\)∥≤σmax\(WSi​j\)⋅∥Δht\(i\)\|Si​j∥,\\big\\\|\\Delta x\_\{t\}^\{\(i,\\mathrm\{overlap\}\)\}\\big\\\|\\;\\leq\\;\\sigma\_\{\\max\}\(W\_\{S\_\{ij\}\}\)\\cdot\\big\\\|\\Delta h\_\{t\}^\{\(i\)\}\\big\|\_\{S\_\{ij\}\}\\big\\\|,which is bounded by the sameO​\(1/d\)O\(1/\\sqrt\{d\}\)scaling as the off\-diagonal Gram entries times the small factor\|Si​j\|/\|Faiℓ\|\|S\_\{ij\}\|/\|F^\{\\ell\}\_\{a\_\{i\}\}\|\. In our experiments this fraction is at most2/50=0\.042/50=0\.04, and we verify in Figure[14](https://arxiv.org/html/2605.10971#A6.F14)that the realized cross\-attribute cosine on multi\-attribute steering is consistent with the disjoint\-case bound to within this correction\. The overlap can therefore be absorbed without invalidating the qualitative conclusion\.

### A\.3Decoder Gram structure and attribute coupling

We evaluate the interference bound of Lemma[4](https://arxiv.org/html/2605.10971#Thmlemma4)on our trained SAEs across all four models and all steering layers, using the top\-50 features \(\|Faℓ\|=50\|F^\{\\ell\}\_\{a\}\|=50\) for both sentiment and topic attributes\.

##### Setup\.

For each model–layer pair we extract the decoder submatricesWFiW\_\{F\_\{i\}\}andWFjW\_\{F\_\{j\}\}corresponding to the sentiment and topic feature sets, compute the inter\-attribute Gram submatrixG\(i​j\)=WFi⊤​WFjG^\{\(ij\)\}=W\_\{F\_\{i\}\}^\{\\top\}W\_\{F\_\{j\}\}and intra\-attribute matricesG\(i​i\),G\(j​j\)G^\{\(ii\)\},G^\{\(jj\)\}, and report both:

1. \(i\)the*theoretical bound*σmax​\(G\(i​j\)\)/σmin​\(G\(i​i\)\)​σmin​\(G\(j​j\)\)\\sigma\_\{\\max\}\(G^\{\(ij\)\}\)/\\sqrt\{\\sigma\_\{\\min\}\(G^\{\(ii\)\}\)\\,\\sigma\_\{\\min\}\(G^\{\(jj\)\}\)\}from Lemma[4](https://arxiv.org/html/2605.10971#Thmlemma4), and
2. \(ii\)the*realized cosine coupling*\|cos⁡\(Δ​x\(i\),Δ​x\(j\)\)\|\|\\cos\(\\Delta x^\{\(i\)\},\\Delta x^\{\(j\)\}\)\|computed with the actual contrastive shiftsδ\(a\)\\delta^\{\(a\)\}used during steering\.

We also report the disjoint variant\|cos\|disj\|\\cos\|\_\{\\mathrm\{disj\}\}, which removes overlapping features \(Fai∩FajF\_\{a\_\{i\}\}\\cap F\_\{a\_\{j\}\}\) so that the preconditionFai∩Faj=∅F\_\{a\_\{i\}\}\\cap F\_\{a\_\{j\}\}=\\emptysetis satisfied exactly\.

##### Empirical coupling is near\-zero\.

The realized cosine coupling is small across all models and layers:

- •With overlap:max⁡\|cos\|=0\.253\\max\|\\cos\|=0\.253\(MDLM L6, 6 shared features\), mean=0\.058=0\.058across all 38 pairs\.
- •Disjoint feature sets:max⁡\|cos\|disj=0\.092\\max\|\\cos\|\_\{\\mathrm\{disj\}\}=0\.092\(SEDD L8\), mean=0\.028=0\.028across all 38 pairs\.

Removing overlapping features reduces the maximum coupling by2\.7×2\.7\\times\(from 0\.253 to 0\.092\), confirming that shared features are the dominant source of residual\-space alignment\. Even with overlap included, coupling remains well below thresholds that would produce observable interference in downstream classifiers\.

##### Feature set overlap\.

The top\-50 sentiment and topic feature sets share 0–16 features depending on model and layer\. LLaDA L8 has the largest overlap \(16/50\), likely because early layers encode less attribute\-specific information\. Later layers consistently show lower overlap \(0–3 features\), consistent with increasing specialization\. Lemma[4](https://arxiv.org/html/2605.10971#Thmlemma4)formally requires disjoint sets; in practice, composability experiments use the full \(potentially overlapping\) feature sets, and the empirical coupling remains small regardless\.

##### Cross\-model and cross\-topic patterns\.

We additionally computed coupling for all four topic categories \(Sports, Business, World, Sci/Tech\) at every Dream and LLaDA layer \(24 extra pairs\)\. Results are consistent:max⁡\|cos\|disj=0\.082\\max\|\\cos\|\_\{\\mathrm\{disj\}\}=0\.082\(LLaDA L14, Sent×\\timesBusiness\), mean=0\.037=0\.037\. No model, layer, or topic category exhibits systematically higher coupling\.

##### Interpretation of the gap between bound and reality\.

Lemma[4](https://arxiv.org/html/2605.10971#Thmlemma4)provides a valid but conservative bound: it guarantees that coupling cannot exceed a computable threshold under any post\-ReLU perturbations consistent with disjoint feature sets\. The empirical coupling is roughly an order of magnitude tighter than the bound\. The gap suggests that the contrastive shiftsδ\(a\)\\delta^\{\(a\)\}used in our steering avoid the worst\-case directions in the off\-diagonal Gram structure — equivalently, contrastive selection finds attribute\-aligned directions that are also geometrically near\-orthogonal in residual space, beyond what disjointness alone guarantees\. Whether this is a generic property of SAE training under contrastive selection or specific to our four models is an open question; tightening the bound via data\-dependent analysis ofδ\(a\)\\delta^\{\(a\)\}is a natural direction for future work\.

## Appendix BVocabulary Grounding of Top Contrastive Features

To qualitatively verify that contrastive SAE features encode semantically coherent concepts, we project each feature’s decoder column through the model’s unembedding matrix and inspect the highest\-scoring vocabulary tokens\. Concretely, for a featurejjwith learned decoder vector𝐰jdec∈ℝd\\mathbf\{w\}\_\{j\}^\{\\text\{dec\}\}\\in\\mathbb\{R\}^\{d\}, we compute the logit\-space projection𝐰jdec⋅𝐖unembed⊤∈ℝ\|𝒱\|\\mathbf\{w\}\_\{j\}^\{\\text\{dec\}\}\\cdot\\mathbf\{W\}\_\{\\text\{unembed\}\}^\{\\top\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}and report the tokens with the largest scores\. Intuitively, these are the vocabulary items that the feature most strongly promotes in the model’s output distribution when active\.

For each attribute \(topic, formality/informality, sentiment\), we select the top 3 features in each direction by Cohen’sdd: the 5 positive\-direction features with the largestd\>0d\>0\(more active for the target class\) and the 5 negative\-direction features with the most negatived<0d<0\(more active for the contrast class\)\. In Table[4](https://arxiv.org/html/2605.10971#A2.T4), we report results at the deepest SAE layer for each model: layer 7 for MDLM\-124M and SEDD\-124M, layer 26 for LLaDA\-8B, and layer 23 for DREAM\-7B\. MDLM and SEDD use the GPT\-2 tokenizer \(50K English tokens\); LLaDA and DREAM use multilingual vocabularies \(126K and 152K tokens, respectively\), so some features project onto non\-English tokens \(Chinese glosses are provided where applicable\)\.

Table 4:Vocabulary grounding of top contrastive SAE features \(via𝐖dec⋅𝐖unembed⊤\\mathbf\{W\}\_\{\\text\{dec\}\}\\cdot\\mathbf\{W\}\_\{\\text\{unembed\}\}^\{\\top\}\)\. For each model and attribute, the top\-3 positive\-ddand top\-3 negative\-ddfeatures at the deepest SAE layer are shown\.ModelAttributePositive\-dd\(target class\)Negative\-dd\(contrast class\)Feature \(dd\)Top TokensFeature \(dd\)Top TokensMDLM \(L7\)TopicF6047 \(\+\+0\.60\)midfield, Isles, winger, TraffordF8014 \(−\-0\.67\)Securities, industrial, regulatory, commercialF7923 \(\+\+0\.58\)mph, Guinness, Citation, maximumF533 \(−\-0\.67\)2030, 2020, average, 2025, 2050F474 \(\+\+0\.56\)quarterbacks, quarterback, tackles, linebackersF6218 \(−\-0\.61\)Avg, \[\+, /$, DAQ, avgSentimentF6624 \(\+\+0\.17\)solid, ocally, composure, ACTIONF11410 \(−\-0\.30\)\!\!\!\!\!, ???, \!\!\!\!, ……, \!\!\!F5123 \(\+\+0\.16\)DJs, eaturing, magic, creatF10932 \(−\-0\.27\)my, idiot, NS, replies, CPF6921 \(\+\+0\.15\)explan, overview, Learn, depthF2643 \(−\-0\.27\)textures, texture, weird, Illusion, gutsFormalityF1866 \(\+\+0\.45\)reportedly, spokeswoman, watchdog, commissionerF5013 \(−\-0\.56\)…, I, \!, myself, ?\!F8014 \(\+\+0\.37\)Securities, industrial, industry, regulatoryF751 \(−\-0\.53\)…, …, \[…\], …, \[…\]F7544 \(\+\+0\.33\)quished, Instit, Aether, orgF7258 \(−\-0\.53\)GOODMAN, —, —SEDD \(L7\)TopicF606 \(\+\+0\.84\)matchups, rematch, matchup, challenger, clinchF7260 \(−\-0\.73\)payable, cash, dollars, money, feesF5606 \(\+\+0\.63\)championships, jerseys, playoffs, playoffF1914 \(−\-0\.66\)GDP, repairs, ppm, gallon, gallonsF3948 \(\+\+0\.45\)Featuring, teamed, eaturing, featuringF9697 \(−\-0\.57\)liberal, government, neoliberal, privatizationSentimentF10956 \(\+\+0\.27\)morrow, thence, tion, splendF3169 \(−\-0\.21\)abusers, porn, sexist, moms, KidsF3948 \(\+\+0\.25\)Featuring, teamed, eaturing, featuringF8644 \(−\-0\.20\)nobody, nothing, nothing, Nothing, THINGF2014 \(\+\+0\.24\)dynam, affordable, unbeat, appreciatedF4439 \(−\-0\.20\)disgusting, vile, despicable, crap, disgraceFormalityF12001 \(\+\+0\.63\)\[CJK\], Participant, ortium, ilaterallyF9490 \(−\-0\.67\)pie, hell, —, damn, …F4051 \(\+\+0\.58\)Treaty, NATO, treaty, milit, YugoslaviaF11298 \(−\-0\.61\)\]\], ín, tar, eq, ›F9697 \(\+\+0\.56\)liberal, government, neoliberal, privatizationF7306 \(−\-0\.55\)§§, Crossref, fmt, ÍÍ, nosLLaDA \(L26\)TopicF9143 \(\+\+1\.13\)Para, 增量 \(increment\), 晗, \_eventF5553 \(−\-0\.80\)Oro, leness, experien, \-war, LahF1834 \(\+\+1\.05\)Historic, atch, 奥斯 \(Aus\-\), stackedF6163 \(−\-0\.68\)INCLUDING, IllegalStateException, SubscriptionsF2897 \(\+\+0\.63\)won, awarded, year, 提名, winF1518 \(−\-0\.66\)\-producing, 资产负债 \(balance sheet\), produceSentimentF12180 \(\+\+0\.43\)\(always, commended, throughout, PERFF5461 \(−\-0\.44\)Avoid, Avoid, avoided, avoid, AVF2833 \(\+\+0\.38\)recommended, recommend, deserves, recommendingF10262 \(−\-0\.42\)throughout, whatsoever, anders, OEMF7427 \(\+\+0\.29\)aras, bec, 小小, apical, 笃F12851 \(−\-0\.42\)失败 \(failure\), 失败的 \(failed\), amateurFormalityF8446 \(\+\+0\.47\)monies, thru, angst, lite, youngsterF7446 \(−\-1\.07\)however, alot, barley, ,D, ectF13972 \(\+\+0\.44\)idency, ”Don, cretion, ener, narcF12552 \(−\-0\.66\)sublicense, fucking, empire, fuckedF12180 \(\+\+0\.41\)\(always, commended, throughout, PERFF9058 \(−\-0\.61\)\}else, ”,&, ,const, ,免费, \.第DREAM \(L23\)TopicF6961 \(\+\+1\.67\)zed, games, \-sided, ided, 1F14304 \(−\-1\.12\)company, angled, business, undergrad, \(\);F4856 \(\+\+1\.28\)team, pall, Players, players, gameF13053 \(−\-0\.89\)computer, software, computers, Computer, dataF7524 \(\+\+0\.91\)team, Team, training, fre, trialsF6220 \(−\-0\.83\)union, union, unions, wright, SHASentimentF10880 \(\+\+0\.51\)simply, potency, consum, bullish, zenF8145 \(−\-0\.51\)worse, S, literally, oa, LiterF5417 \(\+\+0\.36\)and, all, each, is, evenF13133 \(−\-0\.34\)0, 1, prohibits, V, ementF3924 \(\+\+0\.32\)in, also, that, really, isF6642 \(−\-0\.33\)worse, worst, \-bottom, Worst, buffaloFormalityF4302 \(\+\+0\.59\), ; … …F10886 \(−\-1\.14\)people, like, all, and, outF4221 \(\+\+0\.58\)Parliamentary, anyhow, programmes, forbF11172 \(−\-0\.77\)abyte, oppos, imagin, Homeland, igyF5033 \(\+\+0\.47\)cer, bloodstream, vintage, DNA, sapF1918 \(−\-0\.70\)1, 2, 0, 4, ’

## Appendix CDetailed Experimental Setup

### C\.1Models and SAE Training

We study four DLMs that span two training objectives, two scales, and two provenance strategies: MDLM\[[32](https://arxiv.org/html/2605.10971#bib.bib2)\]and SEDD\[[20](https://arxiv.org/html/2605.10971#bib.bib3)\]are 124M\-parameter GPT\-2 architectures trained on OpenWebText with absorbing\-state and score\-entropy objectives, respectively—sharing architecture and data but differing only in loss function, forming a natural experiment for isolating the effect of training objective on learned representations\. Dream\[[42](https://arxiv.org/html/2605.10971#bib.bib4)\]\(7B\) is fine\-tuned from Qwen2 with masked diffusion, and LLaDA\[[25](https://arxiv.org/html/2605.10971#bib.bib6)\]\(8B\) is trained from scratch with a masked absorbing objective, providing a contrast in model provenance at the 7–8B scale\.

For each model, we train TopK sparse autoencoders\[[23](https://arxiv.org/html/2605.10971#bib.bib29)\]on residual\-stream activations extracted from middle\-to\-late transformer layers via forward hooks\. The SAE encodes an input activation𝐱∈ℝd\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\}into a sparse latent representation𝐡∈ℝD\\mathbf\{h\}\\in\\mathbb\{R\}^\{D\}\(D≫dD\\gg d\) and reconstructs:

𝐡\\displaystyle\\mathbf\{h\}=TopK​\(\(𝐱−𝐛dec\)​𝐖enc\+𝐛enc,k\),\\displaystyle=\\mathrm\{TopK\}\\bigl\(\(\\mathbf\{x\}\-\\mathbf\{b\}\_\{\\text\{dec\}\}\)\\mathbf\{W\}\_\{\\text\{enc\}\}\+\\mathbf\{b\}\_\{\\text\{enc\}\},\\;k\\bigr\),\(11\)𝐱^\\displaystyle\\hat\{\\mathbf\{x\}\}=𝐡​𝐖dec\+𝐛dec,\\displaystyle=\\mathbf\{h\}\\,\\mathbf\{W\}\_\{\\text\{dec\}\}\+\\mathbf\{b\}\_\{\\text\{dec\}\},\(12\)whereTopK\\mathrm\{TopK\}retains only thekklargest activations and zeros the rest, providing an exact sparsity guarantee\. We usek=32k\{=\}32for all models\. Decoder columns are normalized to unit norm after each training step\. We train with MSE reconstruction loss plus an auxiliary loss \(λ=1/32\\lambda\{=\}1/32\) that encourages dead neurons \(inactive for\>10,000\{\>\}10\{,\}000steps\) to reconstruct the current residual error\. All SAEs are trained for 80K steps with AdamW \(lr10−410^\{\-4\}, cosine schedule to10−510^\{\-5\}\) on batches of 4,096 activations\. All hyperparameters used for training SAE can be found in Table[10](https://arxiv.org/html/2605.10971#A3.T10)\.

##### DLM\-specific design choice: noise\-level diversity\.

Unlike autoregressive models where each token has a single context, DLM activations depend on the noise levelσ\\sigma\(or equivalently, the fraction of masked tokens\)\. We extract activations across uniformly sampled noise levels during training, ensuring the SAE learns features that are robust across the full denoising trajectory rather than specialized to a single noise regime\. This is validated by our sigma\-stratified analysis \(§[C\.4](https://arxiv.org/html/2605.10971#A3.SS4)\), which shows feature activations vary by less than 15% across noise levels\.

### C\.2Prediction Formation via Diffusion Logit Lens

We introduce the*Diffusion Logit Lens*, adapting the logit lens technique\[[7](https://arxiv.org/html/2605.10971#bib.bib60)\]to discrete diffusion language models\. By projecting intermediate\-layer hidden states through the unembedding matrix at each denoising step, we construct a two\-dimensional*prediction formation surface*\(layer×\\timestimestep\) that reveals when and where token identities crystallize during generation\.

![Refer to caption](https://arxiv.org/html/2605.10971v1/x10.png)Figure 5:Prediction formation via logit lens across four diffusion language models\. Each panel shows token agreement between intermediate\-layer representations and the final output, as a function of layer depth and denoising progress\. Small models \(MDLM, SEDD\) form predictions gradually across layers and time, while large models \(DREAM, LLaDA\) exhibit sharp layer boundaries where predictions crystallize abruptly\.#### C\.2\.1Setup

For each model, we generate samples using the standard denoising schedule, hooking into every transformer block at each step\. At layerℓ\\elland steptt, we extract the hidden state𝐡tℓ∈ℝdmodel\\mathbf\{h\}\_\{t\}^\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\}and project it into vocabulary space:

y^tℓ=arg⁡max⁡𝐖unembed​𝐡tℓ,\\hat\{y\}\_\{t\}^\{\\ell\}=\\arg\\max\\;\\mathbf\{W\}\_\{\\text\{unembed\}\}\\,\\mathbf\{h\}\_\{t\}^\{\\ell\},\(13\)comparingy^tℓ\\hat\{y\}\_\{t\}^\{\\ell\}against the final generated token at each position\. We report*agreement*: the fraction of positions where the intermediate prediction matches the final output\. For models with explicit noise conditioning \(MDLM, SEDD\), we additionally evaluate a*conditioned*lens that applies the full output head including adaptive layer normalization \(adaLN\) with noise\-level conditioning\. Table[5](https://arxiv.org/html/2605.10971#A3.T5)summarizes the experimental configuration\.

Table 5:Logit lens experimental configuration per model\.ModelSamplesStepsLayersLens TypeNotesMDLM\-124M20012813Raw \+ ConditionedadaLN sigma conditioningSEDD\-124M20025613Raw \+ ConditionedScore\-entropy output headDREAM\-7B1006415RawAR\-style logit shiftLLaDA\-8B646417RawNo noise conditioning
#### C\.2\.2Finding 1: Small Models Show Gradual Diagonal Crystallization

MDLM\-124M’s conditioned logit lens reveals a smooth diagonal gradient in the \(layer, timestep\) space\. Agreement with the final output increases gradually across both dimensions simultaneously, reaching 86\.9% at the final layer and step\.

Table 6:MDLM\-124M conditioned lens agreement \(%\) at selected layers and steps\.LayerStep 0Step 64Step 127Block 00\.0211\.320\.9Block 53\.837\.954\.9Block 83\.853\.377\.4Block 113\.660\.186\.9Every layer contributes incrementally to prediction formation—there is no single “decision layer\.” The prediction surface is truly two\-dimensional: both more denoising steps*and*deeper layers are jointly required for accurate predictions\. This distributed processing is consistent with MDLM’s relatively shallow architecture \(12 blocks\), which forces all layers to contribute\.

##### Sigma conditioning is essential\.

The raw logit lens \(projecting hidden states directly through𝐖unembed\\mathbf\{W\}\_\{\\text\{unembed\}\}without noise\-level conditioning\) achieves a maximum of only∼\{\\sim\}8% agreement on MDLM\. The conditioned lens \(applying the full output layer with adaLN sigma modulation\) reaches 87%\. This 10×\\timesgap proves that sigma conditioning is not merely a scaling factor—it fundamentally transforms the representation space\. Without knowing the noise level, intermediate hidden states are essentially unreadable\. The adaLN mechanism acts as a “decoder ring” that makes the residual stream interpretable at each noise level\.

#### C\.2\.3Finding 2: Masked vs\. Unmasked Position Dynamics

Separating agreement by position type reveals distinct dynamics:

##### MDLM\.

Unmasked \(already\-revealed\) positions are correctly predicted by block 5 onward \(\>\>80% agreement\)—these are trivially predictable since the token is visible\. Masked positions carry the interesting signal: their agreement rises from 3\.6% \(step 0\) to 53\.1% \(final step\) at block 11, indicating that the model progressively “plans” future tokens at still\-masked positions\. This anticipatory encoding is the mechanistic basis for why SAE features fire on masked positions before content commitment \(§[3\.3](https://arxiv.org/html/2605.10971#S3.SS3)\)\.

##### DREAM\.

Unmasked agreement peaks at 97\.6% \(step 32\) but*degrades*to 66\.4% \(step 63\), mirroring the V\-shaped entropy\. Even the model’s ability to predict already\-visible tokens deteriorates at low mask ratios, confirming the out\-of\-distribution hypothesis\.

#### C\.2\.4Cross\-Model Summary

Table 7:Cross\-model comparison of prediction formation properties\.PropertyMDLMSEDDDREAMLLaDACrystallizationGradual diagonalGradual diagonalSharp \(L22\)Sharp \(L26\)Layer contributionAll layersAll layersL22\+ onlyL26\+ onlyRaw lens max8%72%93%100%Entropy trajectoryMonotonic↓\\downarrowMonotonic↓\\downarrowV\-shapedMonotonic↓\\downarrowSigma conditioning neededYes \(essential\)Yes \(but raw works\)N/AN/AMax agreement87%86%93%100%The logit lens reveals a fundamental dichotomy in how DLMs form predictions:small models crystallize graduallyacross a 2D surface \(requiring both depth and denoising progress\), whilelarge models exhibit sharp phase transitionsat specific layers\. The training objective further modulates readability—score\-entropy \(SEDD\) produces representations that are9×9\\timesmore readable without conditioning than absorbing\-state \(MDLM\)\. These macroscopic patterns have direct implications for SAE\-based steering: features in pre\-crystallization layers \(e\.g\., DREAM L8–L20\) encode abstract semantic information without token commitment, making them natural intervention points for high\-level attribute control without disrupting low\-level token selection\.

### C\.3SAE Layer Selection

Selecting which transformer layers to train SAEs on is non\-trivial: in autoregressive models, middle layers are known to balance representational richness with causal malleability\[[46](https://arxiv.org/html/2605.10971#bib.bib24)\], but this principle has never been validated for discrete diffusion transformers\. We describe the systematic methodology used and summarize the results for each model\.

#### C\.3\.1Methodology

We apply a four\-stage protocol uniformly across all models:

1. 1\.Linear probing\.Logistic regression with 5\-fold stratified cross\-validation on mean\-pooled activations from every transformer layer\. We probe two attributes: sentiment \(IMDB, 2\-class\) and topic \(AG News, 4\-class\)\. To account for the stochastic nature of diffusion representations, each text is passed through the model with 3 independent noise samples \(mask ratio 0\.5\), yielding3×3\\timesthe effective dataset size \(6,000 data points per layer from 2,000 source texts\)\.
2. 2\.SAE training\.TopK SAEs are trained on candidate layers with identical hyperparameters per model \(Table[10](https://arxiv.org/html/2605.10971#A3.T10)\), ensuring fair comparison\. We monitor reconstruction loss and dead neuron fraction as quality indicators\.
3. 3\.Contrastive feature identification\.Per\-feature Cohen’sddbetween attribute groups, retaining features with\|d\|\>0\.05\|d\|\>0\.05andp<0\.01p<0\.01\. This reveals which layers produce the strongest and most numerous attribute\-discriminative features\.
4. 4\.Steering ablation\.The decisive test: we compare steering effectiveness \(positive\-rate swing across anα\\alphasweep\) for different layer windows, isolating the causal contribution of each layer group\.

#### C\.3\.2Layer Probing Results

Table[8](https://arxiv.org/html/2605.10971#A3.T8)reports sentiment probing accuracy at selected layers for all four models\.

Table 8:Linear probing accuracy \(sentiment, 5\-fold CV\) at selected layers\. Layers inboldare those selected for SAE training\. MDLM and SEDD share the GPT\-2 architecture \(12 layers\); Dream has 28 layers \(Qwen2\); LLaDA has 32 layers \(LLaMA\)\.LayerMDLMSEDDDreamLLaDA\(12L, 124M\)\(12L, 124M\)\(28L, 7B\)\(32L, 8B\)080\.981\.588\.586\.4482\.277\.890\.788\.1582\.778\.6——683\.179\.9——784\.081\.3——884\.7†82\.195\.091\.4984\.481\.8——1084\.082\.4——1184\.483\.2†——13——97\.6—14———95\.517——98\.2†—20——98\.097\.323——98\.098\.1†26———98\.027——98\.0—31———97\.9†Peak probing accuracy for that model\.Three patterns emerge:

- •Monotonic rise then plateau\.All four models show increasing probing accuracy with depth, saturating in the upper layers\. MDLM plateaus at L8–11 \(∼\\sim84%\); Dream and LLaDA plateau earlier in relative terms \(∼\\simL13/L17 and∼\\simL20/L23 respectively\), with much higher absolute accuracy \(\>\>97%\) reflecting the larger model capacity\.
- •SEDD dips before rising\.Unlike MDLM, SEDD shows a characteristic dip at L1–5 \(77–79%\) below the L0 baseline \(81\.5%\), before rising through L6–11\. This U\-shaped profile suggests that score\-entropy diffusion initially disrupts the pretrained embeddings’ linear separability before rebuilding it in later layers\.
- •Scale raises the floor\.The 7–8B models \(Dream, LLaDA\) start at 86–89% even at L0, compared to 81% for the 124M models\. Richer input embeddings \(Qwen2\.5 / LLaMA\) provide a higher baseline that subsequent layers refine\.

Topic probing follows a similar profile but peaks later: MDLM at L11 \(79\.1%\), Dream at L23 \(95\.3%\), LLaDA at L28 \(95\.7%\)\. This offset is consistent with topic being a more global, multi\-token property that requires deeper processing\.

#### C\.3\.3Steering Ablation: MDLM

The decisive experiment compares four 3\-layer windows for sentiment steering with identical settings \(50 features per layer,α∈\[−3,\+3\]\\alpha\\in\[\-3,\+3\], 512 samples perα\\alpha\):

Table 9:MDLM sentiment steering ablation across layer windows\. Swing = positive rate atα=−3\\alpha\{=\}\{\-\}3minusα=\+3\\alpha\{=\}\{\+\}3\.Layer windowSwing \(pp\)Max pos\. rateMin pos\. ratePPL range\[5, 6, 7\]74\.276\.0%1\.8%42–51\[\[7, 8, 9\]\]70\.370\.5%0\.2%40–49\[\[8, 9, 10\]\]62\.163\.3%1\.2%40–46\[\[9, 10, 11\]\]59\.862\.7%2\.9%40–43Steering swing decreases monotonically with depth \(−\-3\.6 pp per layer shift on average\), revealing a starkprobing–steering dissociation: the layers with the highest probing accuracy \(L8–11\) are the least causally effective for steering\. This mirrors findings from autoregressive representation engineering\[[46](https://arxiv.org/html/2605.10971#bib.bib24)\]and extends the principle to discrete diffusion transformers\.

The mechanism is*causal distance to output*: perturbations at L5–7 traverse 5–7 subsequent transformer blocks, amplifying the steering signal through residual\-stream propagation\. Later\-layer perturbations have fewer blocks to propagate through and may be partially undone by the final layer norm and unembedding\.

#### C\.3\.4Layer Selection per Model

##### MDLM: layers \[5, 6, 7\]\.

Selected by the steering ablation above\. Despite probing peaking at L8 \(84\.7%\), L5–7 achieve the strongest steering \(74\.2pp swing\)\. Topic steering on the same layers yields 100pp swing \(100% Sports atα=−3\\alpha\{=\}\{\-\}3\), independently validating the choice\.

##### SEDD: layers \[5, 6, 7\]\.

SEDD shows*no*probing–steering dissociation: at L8–10 the swing reaches 90\.8pp, far exceeding L5–7 \(38\.8pp\)\. Score\-entropy training appears to co\-locate probing accuracy and causal malleability in the same layers which is the opposite of MDLM’s pattern under absorbing\-state training, suggesting the dissociation stems from the loss objective rather than the architecture\. To preserve a clean MDLM–SEDD contrast \(same architecture, different loss\), we train SAEs at the same layers \(L5–7\) on both models throughout; per\-model layer optimization would conflate the loss comparison with layer choice and obscure the controlled comparison in §[3\.3](https://arxiv.org/html/2605.10971#S3.SS3)\.

##### Dream\-7B: layers \[8, 13, 17, 23\]\.

Selected to span the transition zone \(L8, pre\-plateau\) through the probing plateau \(L13–23\)\. The plateau makes layer discrimination via probing alone impossible; direct steering comparisons confirm that L17 \(sentiment\) and L23 \(topic\) produce the strongest features\. Even spacing across the 28\-layer model ensures coverage of early\-, mid\-, and late\-stage representations\.

##### LLaDA\-8B: layers \[8, 14, 20, 26\]\.

Follows the same even\-spacing strategy across 32 layers, covering transition \(L8\), mid\-rise \(L14\), near\-peak \(L20\), and plateau \(L26\)\. Feature identification reveals the strongest contrastive effects at L14 \(max\|d\|=0\.63\|d\|\{=\}0\.63\), while the dissociation experiment shows L26 dominates steering \(67pp alone\)\. This suggests a mild dissociation similar to MDLM: the layer with the strongest individual features is not the layer with the greatest causal impact\.

#### C\.3\.5SAE Training Hyperparameters

Table 10:SAE training configuration per model\. All SAEs use TopK activation with auxiliary dead\-neuron loss\.ParameterMDLMSEDDDreamLLaDAdmodeld\_\{\\text\{model\}\}7687683,5844,096dsaed\_\{\\text\{sae\}\}12,28812,28814,33616,384Expansion ratio16×\\times16×\\times4×\\times4×\\timeskk\(TopK\)32323232Training steps50K80K80K40KBatch size4,0964,0964,0964,096Learning rate10−410^\{\-4\}10−410^\{\-4\}10−410^\{\-4\}10−410^\{\-4\}LR scheduleCosineCosineCosineCosineAux loss coef1/321/321/321/32EMA decay0\.9990\.9990\.9990\.999Training dataOWTOWTOWTOWT
#### C\.3\.6Summary

The layer selection study yields two findings of independent interest:

1. 1\.Probing–steering dissociation is loss\-dependent\.MDLM \(absorbing loss\) shows clear dissociation — middle layers steer best despite later layers probing best\. SEDD \(score\-entropy loss\) shows no dissociation — later layers both probe and steer best\. LLaDA shows suggestive dissociation similar to MDLM\. This implicates the training objective, not the architecture, as the key determinant of where causal malleability resides\.
2. 2\.The causal malleability principle extends to diffusion transformers\.For absorbing\-loss DLMs, middle layers provide the best balance of representational richness and propagation distance, consistent with findings from autoregressive representation engineering\. This principle appears robust across model scales \(124M to 8B\) and architectures \(GPT\-2, Qwen2, LLaMA\)\.

### C\.4Sigma\-Stratified Feature Analysis

Unlike autoregressive models where each token is conditioned on a fixed left context, DLM activations depend on the noise levelσ\\sigma\(equivalently, the fraction of masked tokens\)\. During SAE training, we extract activations across uniformly sampled noise levels, ensuring the SAE learns features that span the full denoising trajectory\. A natural question is whether the learned features are*robust*across noise levels or*specialized*to particular noise regimes\. We answer this with a sigma\-stratified analysis of feature discriminability\.

#### C\.4\.1Methodology

We partition the noise schedule into five equal\-frequency bins based on the percentiles ofσ\\sigmaobserved during activation extraction \(Table[11](https://arxiv.org/html/2605.10971#A3.T11)\)\. For each bin, we recompute the Cohen’sddeffect size for the top\-20 most discriminative features per layer\. We report the*coefficient of variation*\(CV\) of\|d\|\|d\|across bins as the primary measure of noise\-level sensitivity:

CV=std​\(\|d1\|,…,\|d5\|\)mean​\(\|d1\|,…,\|d5\|\)×100%\\text\{CV\}=\\frac\{\\text\{std\}\(\|d\_\{1\}\|,\\dots,\|d\_\{5\}\|\)\}\{\\text\{mean\}\(\|d\_\{1\}\|,\\dots,\|d\_\{5\}\|\)\}\\times 100\\%\(14\)wheredid\_\{i\}is the Cohen’sddof a given feature computed within sigma binii\.

Table 11:Sigma bin boundaries \(equal\-frequency, MDLM\)\. Lowσ\\sigmacorresponds to near\-clean text; highσ\\sigmacorresponds to heavily masked text\. Each bin contains∼1\.79\{\\sim\}1\.79M token positions \(sentiment\) or∼334\{\\sim\}334K \(topic\)\.Binσlow\\sigma\_\{\\text\{low\}\}σhigh\\sigma\_\{\\text\{high\}\}Interpretation13\.1×10−53\.1\\times 10^\{\-5\}0\.225Near\-clean20\.2250\.507Low noise30\.5070\.911Medium noise40\.9111\.604High noise51\.6046\.906Near\-fully\-masked
#### C\.4\.2Results

Table[12](https://arxiv.org/html/2605.10971#A3.T12)summarizes the distribution of CV values across the top\-20 features for each layer and attribute\. We observe moderate but heterogeneous variation across noise levels, with median CV values ranging from 13% to 22% depending on the layer and attribute\.

Table 12:Coefficient of variation \(CV\) of\|d\|\|d\|across sigma bins for the top\-20 features per layer\. Lower CV indicates more noise\-robust discriminability\.AttributeLayerMedian CVMean CV% with CV<15%\{<\}15\\%% with CV<25%\{<\}25\\%SentimentL516\.9%20\.1%45%80%L620\.6%24\.3%20%65%L713\.1%20\.1%55%65%TopicL516\.3%17\.2%30%95%L622\.1%21\.8%30%70%L715\.5%17\.4%40%90%Several patterns emerge\. First, a substantial fraction of features \(30–55%, depending on the layer\) exhibit CV<15%\{<\}\\,15\\%, indicating that their discriminability is largely preserved across the full noise schedule\. Second, the majority of features \(65–95%\) have CV<25%\{<\}\\,25\\%, confirming that SAE features trained on mixed\-noise activations successfully generalize across denoising stages rather than collapsing to noise\-regime\-specific detectors\. Third, the remaining features with higher CV reveal a structured dependence onσ\\sigma: some features strengthen with increasing noise \(capturing distributional or structural cues available even in heavily corrupted text\) while others weaken \(encoding fine\-grained lexical or semantic cues that emerge only as text nears its final form\)\.

#### C\.4\.3Per\-Feature Sigma Profiles

Table[13](https://arxiv.org/html/2605.10971#A3.T13)shows representative per\-bin Cohen’sddprofiles for selected sentiment features at Layer 7, illustrating the diversity of sigma\-dependence patterns\.

Table 13:Per\-bin\|d\|\|d\|for selected sentiment features \(Layer 7\)\. Features are categorized by their sigma\-dependence pattern\.FeatureOverallddBin 1Bin 2Bin 3Bin 4Bin 5PatternNoise\-robust\(CV<\{<\}15%\)F2846−\-0\.1600\.1510\.1520\.1600\.1620\.170StableF26600\.1310\.1230\.1290\.1310\.1370\.137StableF75900\.1840\.1700\.1790\.1900\.2010\.210Gradual increaseNoise\-sensitive\(CV\>\{\>\}25%\)F1758−\-0\.2350\.1600\.1900\.2180\.2870\.308Increases withσ\\sigmaF6680−\-0\.1660\.0670\.0960\.1600\.2150\.227Strong increaseF11495−\-0\.1700\.0540\.0710\.1420\.2330\.250Strong increase
#### C\.4\.4Directional Analysis

We additionally examine whether features tend to become more or less discriminative at higher noise levels\. Across layers and attributes, the direction of sigma\-dependence is approximately evenly split: roughly half of features show increasing\|d\|\|d\|withσ\\sigma\(strengthening under noise\) and half show decreasing\|d\|\|d\|\(weakening under noise\)\. This balanced split holds for both sentiment \(50/50 at L5, 55/45 at L6, 40/60 at L7\) and topic classification \(35/65 at L5, 45/55 at L6, 35/65 at L7\), with a slight tendency for topic features to weaken at higher noise levels\.

This bidirectional structure suggests that the SAE decomposes DLM representations into complementary feature populations:

- •Low\-σ\\sigmafeaturesencode fine\-grained semantic content \(e\.g\., sentiment\-bearing lexical items\) that is most discriminative when text is nearly resolved\.
- •High\-σ\\sigmafeaturescapture coarser distributional properties \(e\.g\., topic or register\) that the model infers even from heavily corrupted input\.

This functional decomposition is unique to the diffusion setting and has no direct analog in autoregressive models, where all tokens share a single \(fully observed\) conditioning context\.

#### C\.4\.5Implications for Steering

The sigma\-stratified analysis has practical implications for feature\-based steering of DLMs\. Since feature discriminability varies across the denoising trajectory, a sigma\-adaptive steering strategy—applying different feature weights at different noise levels—could yield more targeted interventions than uniform steering across all timesteps\. We leave exploration of sigma\-adaptive steering to future work, noting that the infrastructure for step\-conditional steering is already demonstrated in our denoising dynamics experiments \(§[3\.3](https://arxiv.org/html/2605.10971#S3.SS3)\)\.

### C\.5Contrastive Feature Extraction

#### C\.5\.1Datasets

We study three attributes in this paper–Sentiment, Topic, and Style\. The datasets used to model these attributes are explained below\.

Sentiment:We use the IMDB movie review dataset proposed byMaaset al\.\[[22](https://arxiv.org/html/2605.10971#bib.bib39)\]for sentiment\. We sample5,0005,000positive reviews \(label=1\{\}=1\) and5,0005,000negative reviews \(label=0\{\}=0\) from this dataset and use them for learning sentiment features\. All texts are truncated to512512tokens\.

Topic:We use the AG News datasetZhanget al\.\[[44](https://arxiv.org/html/2605.10971#bib.bib40)\], which contains news articles across four topics: World, Sports, Business, and Sci/Tech\. We study the topic Sports in this paper\. Two potential contrastive approaches exist for feature extraction: \(1\) contrasting Sports against all other topics combined \(one\-vs\-all\), and \(2\) contrasting Sports against a single other topic \(one\-vs\-one\)\. Prior work on contrastive representation learning suggests that narrower, more targeted contrasts yield cleaner feature directions:Zouet al\.\[[46](https://arxiv.org/html/2605.10971#bib.bib24)\]showed that contrastive pair specificity affects steering direction quality,Rimskyet al\.\[[30](https://arxiv.org/html/2605.10971#bib.bib41)\]demonstrated that carefully curated contrastive pairs outperform generic ones for activation steering, andBelinkov \[[6](https://arxiv.org/html/2605.10971#bib.bib42)\]established that the choice of control task determines what linguistic properties a probe captures\. Consistent with these findings, we empirically observe that one\-vs\-one contrasts produce features that are more attribute\-specific and less likely to interfere with other attributes during multi\-attribute steering\. Hence, we sample5,0005\{,\}000Sports articles \(label=1\{\}=1\) and5,0005\{,\}000Business articles \(label=0\{\}=0\), totaling10,00010\{,\}000samples, and use them for learning sports topic related features\. Texts are truncated to512512tokens\.

Style:We study formality as our style attribute\. For MDLM and SEDD, which are unconditional models trained on OpenWebText\[[17](https://arxiv.org/html/2605.10971#bib.bib52)\], we use the formality corpus ofPavlick and Tetreault \[[27](https://arxiv.org/html/2605.10971#bib.bib43)\]\. This dataset contains9,2749\{,\}274sentences from news articles and reader comments, each annotated with a continuous formality score on a\[−3,\+3\]\[\-3,\+3\]scale by crowdworkers using Best\-Worst Scaling\[[21](https://arxiv.org/html/2605.10971#bib.bib44)\]\. We threshold these scores to create a binary contrast: sentences with average score≥1\.0\\geq 1\.0are labeled formal \(class A,2,5972\{,\}597sentences\) and sentences with average score≤−1\.0\\leq\-1\.0are labeled informal \(class B,2,6002\{,\}600sentences\)\. Sentences with scores between−1\.0\-1\.0and1\.01\.0are excluded as ambiguous\. We balance to2,5002\{,\}500samples per class, totaling5,0005\{,\}000samples\. Positive\-ddfeatures correspond to formal features; negative\-ddto informal\. LLaDA and DREAM are conditional models that we prompt with domain\-neutral opinion starters \(e\.g\., “I think this is”, “The experience was”\) to generate review\-style text\. The corpus byPavlick and Tetreault \[[27](https://arxiv.org/html/2605.10971#bib.bib43)\]is domain\-mismatched for these models: its formality features are learned from news articles and government proceedings, encoding vocabulary patterns \(e\.g\., legislative terminology, formal news register\) that do not appear in review\-style generations\. When applied to LLaDA, these features caused text degradation–steering toward formality produced repetitive “president of the Senate” loops rather than genuinely formal reviews, because the features activated domain\-specific vocabulary instead of domain\-general register shifts\. Hence, we instead label20,00020\{,\}000IMDB reviews from\[[22](https://arxiv.org/html/2605.10971#bib.bib39)\]with an off\-the\-shelf formality classifier111[https://huggingface\.co/s\-nlp/roberta\-base\-formality\-ranker](https://huggingface.co/s-nlp/roberta-base-formality-ranker)\[[4](https://arxiv.org/html/2605.10971#bib.bib47)\]that was trained on the GYAFC corpus\[[29](https://arxiv.org/html/2605.10971#bib.bib45)\]and the online formality corpus ofPavlick and Tetreault \[[27](https://arxiv.org/html/2605.10971#bib.bib43)\]\. This ensures the formality features are learned from the same domain \(movie reviews\) that the prompted models generate\. Reviews with formality score\>0\.9\>0\.9form the formal class \(6,5876\{,\}587reviews\) and reviews with score<0\.1<0\.1form the informal class \(263263reviews\)\. We balance to263263samples per class, limited by the informal count\. As with MDLM and SEDD, positive\-ddfeatures correspond to formal features and negative\-ddto informal\.

#### C\.5\.2Feature Extraction Procedure

The feature extraction procedure works as follows\. For each dataset pair \(class A vs\. class B\):

1. 1\.Tokenization:We tokenize each text with the model’s tokenizer, truncating or padding to the model’s sequence length\.
2. 2\.Apply noise:Then for each text, we create 3 noised versions at mask ratios\{0\.3,0\.5,0\.7\}\\\{0\.3,0\.5,0\.7\\\}by randomly replacing tokens with\[MASK\]\. This ensures features are identified across noise levels rather than at a single denoising stage\.
3. 3\.Forward pass:Next, we run the noised input through the model to obtain hidden states at each SAE layer\.
4. 4\.ReLU SAE encoding \(per\-token\):for each token position at each SAE layer, we compute: activations=ReLU​\(\(𝐡−𝐛dec\)​𝐖enc\+𝐛enc\),\\displaystyle\\text\{activations\}=\\mathrm\{ReLU\}\\\!\\bigl\(\(\\mathbf\{h\}\-\\mathbf\{b\}\_\{\\text\{dec\}\}\)\\,\\mathbf\{W\}\_\{\\text\{enc\}\}\+\\mathbf\{b\}\_\{\\text\{enc\}\}\\bigr\),\(15\)This produces a\(seq\_len×dSAE\)\(\\text\{seq\\\_len\}\\times d\_\{\\text\{SAE\}\}\)activation matrix per sample per layer\. Although the SAEs use TopK activation during training and normal inference, we use full ReLU encoding for feature extraction\. TopK retains only the K most active features per token, so a contrastive feature outside the top K would appear inactive, biasing its Cohen’sddtoward zero\. ReLU returns the true activation for alldSAEd\_\{\\text\{SAE\}\}features, enabling accurate effect\-size estimation\.
5. 5\.Streaming statistics:We accumulate per\-feature mean and variance for class A and class B using Welford’s online algorithm\[[40](https://arxiv.org/html/2605.10971#bib.bib48)\], avoiding the need to store all activations in memory\.
6. 6\.Cohen’sdd\[[11](https://arxiv.org/html/2605.10971#bib.bib49)\]:for each of thedSAEd\_\{\\text\{SAE\}\}features at each layer, we compute: dj=h¯jA−h¯jB\(Var​\(hjA\)\+Var​\(hjB\)\)/2\.d\_\{j\}=\\frac\{\\bar\{h\}\_\{j\}^\{A\}\-\\bar\{h\}\_\{j\}^\{B\}\}\{\\sqrt\{\(\\mathrm\{Var\}\(h\_\{j\}^\{A\}\)\+\\mathrm\{Var\}\(h\_\{j\}^\{B\}\)\)/2\}\}\.\(16\)Positiveddindicates a feature more active for class A; negativeddfor class B\.
7. 7\.Statistical validation:Next, we rank features by\|d\|\|d\|, take the top 300 candidates, and run Mann\-WhitneyUUtests\[[24](https://arxiv.org/html/2605.10971#bib.bib50)\]withp<0\.01p<0\.01\(Bonferroni\-corrected\[[8](https://arxiv.org/html/2605.10971#bib.bib51)\]across 300 tests\)\. We retain the top 50 positive\-ddand top 50 negative\-ddfeatures per layer \(indices and statistics\) for our study\. The top 20 per direction are used for steering\.
8. 8\.Per\-feature mean shifts:For each retained featurejj, we store the calibrated shift Δj=h¯jA−h¯jB,\\Delta\_\{j\}=\\bar\{h\}\_\{j\}^\{A\}\-\\bar\{h\}\_\{j\}^\{B\},\(17\)Here,AAandBBare the two contrastive classes\. At steering time, the hidden state is modified ashj←hj\+α⋅Δjh\_\{j\}\\leftarrow h\_\{j\}\+\\alpha\\cdot\\Delta\_\{j\}\. This is self\-calibrating: features with large class differences receive large shifts, while features with small differences receive proportionally small shifts\.

Table[14](https://arxiv.org/html/2605.10971#A3.T14)shows the feature extraction hyperparameters for all models, and Table[15](https://arxiv.org/html/2605.10971#A3.T15)shows per\-model configurations\.

Table 14:Feature extraction hyperparameters for all models\.ParameterValueBatch size48Noise samples per text3Mask ratios\{0\.3,0\.5,0\.7\}\\\{0\.3,0\.5,0\.7\\\}Top candidates forUU\-test300Mann\-Whitneypp\-value0\.01 \(Bonferroni\)Features saved per direction50Features used for steering20Random seed42Table 15:Per\-model extraction configurations\.MDLMSEDDLLaDADREAMSAE layers5, 6, 75, 6, 78, 14, 20, 268, 13, 17, 23dSAEd\_\{\\text\{SAE\}\}12,28812,28816,38414,336dmodeld\_\{\\text\{model\}\}7687684,0963,584Sentiment sourceIMDB\[[22](https://arxiv.org/html/2605.10971#bib.bib39)\]IMDB\[[22](https://arxiv.org/html/2605.10971#bib.bib39)\]IMDB\[[22](https://arxiv.org/html/2605.10971#bib.bib39)\]IMDB\[[22](https://arxiv.org/html/2605.10971#bib.bib39)\]Topic sourceAG News\[[44](https://arxiv.org/html/2605.10971#bib.bib40)\]AG News\[[44](https://arxiv.org/html/2605.10971#bib.bib40)\]AG News\[[44](https://arxiv.org/html/2605.10971#bib.bib40)\]AG News\[[44](https://arxiv.org/html/2605.10971#bib.bib40)\]Formality sourcePavlick and Tetreault \[[27](https://arxiv.org/html/2605.10971#bib.bib43)\]Pavlick and Tetreault \[[27](https://arxiv.org/html/2605.10971#bib.bib43)\]IMDB\+RoBERTa\[[4](https://arxiv.org/html/2605.10971#bib.bib47)\]IMDB\+RoBERTa\[[4](https://arxiv.org/html/2605.10971#bib.bib47)\]

### C\.6Temporal Dynamics Tracking

To construct the dynamics\-weighted \(adaptive\) steering schedule, we track how contrastive feature activations evolve across the denoising trajectory\. Table[18](https://arxiv.org/html/2605.10971#A3.T18)lists the per\-model configurations\.

1. 1\.We generateNNsamples with SAE hooks active at all denoising steps\.
2. 2\.At each step, we ReLU\-encode the hidden states through the SAE and record activations for the top\-20 contrastive features per attribute\.
3. 3\.We store per\-sample, per\-step activations as an\(N×T×F\)\(N\\times T\\times F\)tensor per layer, whereTTis the number of denoising steps andFFis the number of tracked features\.
4. 4\.Since not all generated samples express the target attribute, we classify allNNtexts with attribute\-specific classifiers \(Table[16](https://arxiv.org/html/2605.10971#A3.T16)\) and retain only confident samples \(softmax probability above a threshold; see Table[17](https://arxiv.org/html/2605.10971#A3.T17)\) to avoid diluting the emergence signal\. Informality uses a lower threshold \(0\.6\) because baseline formality is high \(81–85% for LLaDA/DREAM\), so very few generated texts are confidently informal at 0\.8\.
5. 5\.Then we average dynamics over the filtered samples to obtain per\-step mean activation curves\.
6. 6\.We compute block fractions by dividing the trajectory intoBBtemporal blocks, computing the activation change in each block, clamping negatives to zero, and normalizing to sum to one\.

Table 16:Evaluation classifiers\. These classifiers are used for evaluation and for filtering dynamics samples\. They are not used during contrastive feature extraction for sentiment or topic, which rely on human\-labeled datasets\. For formality, the RoBERTa ranker is additionally used to label IMDB reviews for LLaDA and DREAM extraction \(§[C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1)\); however, the labeled texts \(IMDB training set\) and the evaluated texts \(model\-generated\) are disjoint populations, so there is no direct circularity\.AttributeModelHuggingFace IDSentimentDistilBERT\[[33](https://arxiv.org/html/2605.10971#bib.bib55)\]fine\-tuned on SST\-2distilbert/distilbert\-base\-uncased\-finetuned\-sst\-2\-englishTopicBERT\[[13](https://arxiv.org/html/2605.10971#bib.bib54)\]fine\-tuned on AG Newsfabriceyhc/bert\-base\-uncased\-ag\_newsFormalityRoBERTa\[[4](https://arxiv.org/html/2605.10971#bib.bib47)\]formality rankers\-nlp/roberta\-base\-formality\-rankerPerplexityGPT\-2\[[28](https://arxiv.org/html/2605.10971#bib.bib56)\]openai\-community/gpt2

Table 17:Classifier confidence thresholds for dynamics filtering\.AttributeThreshold% samples retainedSentiment0\.889–97%Topic0\.886–97%Formality0\.87–69%Informality0\.65–69%Table 18:Per\-model dynamics tracking configurations\.MDLMSEDDLLaDADREAMSamples \(NN\)1,000–5,0005,0001,0001,000Denoising steps \(TT\)1,0241,02464256Temporal blocks \(BB\)161688Steps per block6464832Batch size16488
### C\.7SAE Steering Experiments

#### C\.7\.1Steering Mechanism

At each denoising step, for each SAE layer, a forward hook performs the following:

1. 1\.Encode:𝐡enc=ReLU​\(𝐡​𝐖enc\+𝐛enc\)\\mathbf\{h\}\_\{\\text\{enc\}\}=\\mathrm\{ReLU\}\(\\mathbf\{h\}\\,\\mathbf\{W\}\_\{\\text\{enc\}\}\+\\mathbf\{b\}\_\{\\text\{enc\}\}\)
2. 2\.Modify:𝐡enc​\[j\]←𝐡enc​\[j\]\+α⋅Δj\\mathbf\{h\}\_\{\\text\{enc\}\}\[j\]\\leftarrow\\mathbf\{h\}\_\{\\text\{enc\}\}\[j\]\+\\alpha\\cdot\\Delta\_\{j\}for each target featurejj
3. 3\.Decode:𝐡modified=𝐡enc​𝐖dec\+𝐛dec\\mathbf\{h\}\_\{\\text\{modified\}\}=\\mathbf\{h\}\_\{\\text\{enc\}\}\\,\\mathbf\{W\}\_\{\\text\{dec\}\}\+\\mathbf\{b\}\_\{\\text\{dec\}\}
4. 4\.Replace: substitute the original hidden state with𝐡modified\\mathbf\{h\}\_\{\\text\{modified\}\}

The shiftΔj\\Delta\_\{j\}for each feature is the calibrated mean difference computed during extraction \(§[C\.5\.2](https://arxiv.org/html/2605.10971#A3.SS5.SSS2)\)\.

#### C\.7\.2Steering Modes

We evaluate two schedule types with two alpha\-scaling strategies, yielding four modes\.

##### Uniform schedule\.

A constantα\\alphais applied at every layer and every denoising step\.

##### Adaptive schedule\.

The denoising trajectory is divided intoBBtemporal blocks \(Table[18](https://arxiv.org/html/2605.10971#A3.T18)\)\. Each block has a*block fraction*computed from the dynamics tracking \(§[C\.6](https://arxiv.org/html/2605.10971#A3.SS6)\), reflecting what fraction of total feature emergence occurs in that block\. At each step, the steering strength is scaled by the block fraction of the step’s temporal block\. Steps in high\-activity blocks receive proportionally more steering; steps in low\-activity blocks receive less\. This concentrates steering effort where the model is actively making semantic decisions\.

##### E\-ratio scaling \(for multi\-attribute steering\)\.

Different attributes require different steering strengths to achieve comparable classifier rates\. We normalize this asymmetry with E ratios: \(1\) run single\-attribute steering acrossα=1\\alpha=1–1515; \(2\) identifyα∗\\alpha^\{\*\}at which each attribute reaches approximately 80% classifier rate; \(3\) computeEattr=αattr∗/maxattrs⁡\(α∗\)E\_\{\\text\{attr\}\}=\\alpha^\{\*\}\_\{\\text\{attr\}\}/\\max\_\{\\text\{attrs\}\}\(\\alpha^\{\*\}\)\. The hardest attribute receivesE=1\.0E\{=\}1\.0; easier attributes receiveE<1\.0E\{<\}1\.0\. During multi\-attribute steering,αeff=αglobal×Eattr\\alpha\_\{\\text\{eff\}\}=\\alpha\_\{\\text\{global\}\}\\times E\_\{\\text\{attr\}\}\.

Crossing the two schedules \(uniform, adaptive\) with two scaling strategies \(rawα\\alpha, E\-scaledα\\alpha\) gives four modes:Uniform,Adaptive,Uniform\+E, andAdaptive\+E\. Table[19](https://arxiv.org/html/2605.10971#A3.T19)reports the E ratios\.

Table 19:E ratios for multi\-attribute steering\. The hardest attribute \(ratio=1\.0\{\}=1\.0\) is in bold\.ModelModeSentimentTopicStyleMDLMUniform1\.0000\.4000\.400Adaptive1\.0000\.2310\.308SEDDUniform1\.0000\.1670\.333Adaptive1\.0000\.1180\.118LLaDAUniform0\.4550\.5451\.000Adaptive1\.0000\.5710\.500DREAMUniform1\.0000\.4000\.800Adaptive1\.0000\.4000\.800

#### C\.7\.3Attribute Combinations

We evaluate 7 attribute combinations per model: 3 single\-attribute \(sentiment, topic, style\) and 4 multi\-attribute \(sentiment\+topic, sentiment\+style, style\+topic, sentiment\+topic\+style\)\. Sentiment is steered toward positive and topic toward Sports in all combinations\. For the style attribute, MDLM and SEDD steer toward*formality*\(positive\-ddfeatures\), since their baseline formality is near chance \(54% and 53%, respectively\)\. LLaDA and DREAM steer toward*informality*\(negative\-ddfeatures, with inverted shift direction\), since their baselines are already highly formal \(85% and 81%\)\.

#### C\.7\.4Steering Hyperparameters

Table[20](https://arxiv.org/html/2605.10971#A3.T20)shows the per\-model steering parameters\. We generate with MDLM and SEDD unconditionally \(no prompt\)\. For LLaDA and DREAM, we use 10 domain\-neutral prompts \(e\.g\., “I think this is”, “In my opinion”\) with 20 samples per prompt, yielding 200 total per condition\. The 10 prompts are listed in Table[21](https://arxiv.org/html/2605.10971#A3.T21)\. SEDD’s alpha range extends to 20 because sentiment required higherα\\alphato reach saturation on this model\. Note that, MDLM and SEDD are unconditional diffusion models that do not use temperature\-based sampling\. Diversity arises from the stochastic denoising process itself: MDLM samples a random initial sequence from the prior and applies stochastic DDPM updates at each step, while SEDD uses an analytic predictor with Gumbel\-max sampling\. Each generation starts from a different random noise realization, producing diverse outputs without an explicit temperature parameter\.

Table 20:Per\-model steering parameters\.MDLMSEDDLLaDADREAMSamples per condition200200200200Denoising steps1,0241,02464256Sequence length1,0241,0246464Batch size1012–2048TemperatureN/AN/A0\.80\.7Remasking strategyN/AN/Alow\-confidenceentropyPromptsNoneNone10 neutral10 neutralα\\alpharange1–151–201–151–15Seed42424242Table 21:Prompts used for LLaDA and DREAM generation\.\#Prompt1“I think this is”2“In my opinion, this was”3“I would say this is”4“Overall, I found this to be”5“To be honest, this was”6“Many people found this to be”7“It turns out that this was”8“The experience was”9“This was definitely”10“I have to say, this is”

### C\.8Baseline Methods

We compare SAE steering against three activation\-space baselines and one prompt\-based baseline\.

#### C\.8\.1Activation\-Space Baselines

These baselines addα⋅𝐯\\alpha\\cdot\\mathbf\{v\}to hidden states at each SAE layer at each denoising step, using the same forward\-hook mechanism as SAE steering but without the encode–modify–decode cycle\.

Table 22:Activation\-space baseline methods\.MethodUses SAE?Direction computationContrastive vectorNo𝐯=𝐡¯A−𝐡¯B\\mathbf\{v\}=\\overline\{\\mathbf\{h\}\}^\{A\}\-\\overline\{\\mathbf\{h\}\}^\{B\}, unit\-normalized\. Mean difference in raw activation space\.ProbeNoWeight vector from logistic regression on class labels\.PCANoFirst principal component of contrastive activations, oriented positive→\\tonegative\.These baselines use the same attribute combinations and alpha range as SAE steering, with 200 samples per condition\. Activations for direction computation are loaded from the Phase 1 extraction cache\.

#### C\.8\.2Prompt Instruction Baseline

For LLaDA and DREAM, we additionally evaluate a prompt\-based baseline that prepends an explicit attribute instruction to each generation prompt\. No activation\-space intervention is applied\. The instruction prefix is concatenated with each of the 10 standard prompts \(Table[21](https://arxiv.org/html/2605.10971#A3.T21)\), and 20 samples are generated per prefixed prompt \(200 total\), matching the SAE steering setup\. Table[23](https://arxiv.org/html/2605.10971#A3.T23)lists the prefixes for each attribute combination\.

This baseline is not applicable to MDLM and SEDD, which are unconditional models that generate by denoising from pure noise with no prompt interface\.

Table 23:Instruction prefixes for the prompt baseline \(LLaDA and DREAM\)\. Each prefix is prepended to the 10 standard prompts from Table[21](https://arxiv.org/html/2605.10971#A3.T21)\.CombinationPrefixSentiment“Write in positive sentiment: ”Topic“Write about sports: ”Style“Write informally: ”Sentiment \+ Topic“Write in positive sentiment about sports: ”Sentiment \+ Style“Write in positive sentiment informally: ”Style \+ Topic“Write about sports informally: ”Sentiment \+ Topic \+ Style“Write in positive sentiment about sports informally: ”This baseline tests whether the base model can follow explicit attribute instructions through prompting alone, providing a reference for how much control is achievable without any activation\-level intervention\.

### C\.9Evaluation Metrics

For each steering condition, we report the*mean classifier confidence*\(mean softmax probability for the target class across 200 samples\) using the attribute classifiers listed in Table[16](https://arxiv.org/html/2605.10971#A3.T16)\. We measure text quality with three metrics: GPT\-2 perplexity \(PPL; closer to baseline is better\), distinct bigram ratio \(dist\-2; higher is better\), and within\-sample repetition rate \(rep\_rate; lower is better\)\. For each steering method, when reporting results, we select the best operating\-pointα\\alphathat maximizes the mean confidence for the target attribute \(single\-attribute\) or geometric mean of confidences across steered attributes \(multi\-attribute\), subject to two quality gates:dist​\-​2≥50%\\mathrm\{dist\\text\{\-\}2\}\\geq 50\\%of baseline andPPL<100\\mathrm\{PPL\}<100\.

## Appendix DExtended Interpretability Analysis

This section provides the full set of interpretability analyses summarized in Section[3](https://arxiv.org/html/2605.10971#S3)of the main paper\. For each analysis type, we describe the generation methodology, present the complete figures spanning all models and layers, and discuss patterns not covered in the main text\. Throughout, the three attributes studied are positive sentiment \(IMDB\), sports topic \(AG News\), and style \(formality for MDLM/SEDD; informality for LLaDA/DREAM\), as detailed in §[C\.5\.1](https://arxiv.org/html/2605.10971#A3.SS5.SSS1)\.

### D\.1Emergence Trajectories for All Attributes

##### Methodology\.

The main paper \(Figure[2\(a\)](https://arxiv.org/html/2605.10971#S3.F2.sf1)\) shows emergence trajectories for*topic*features only\. Here we present the full4×34\\times 3grid covering all three attributes across all four models\. For each model, attribute, and layer, the emergence trajectory is computed as follows:

1. 1\.We generateNNsamples \(Table[18](https://arxiv.org/html/2605.10971#A3.T18)\) with SAE hooks active at every denoising step, recording per\-step activations for the top\-20 contrastive features per attribute per layer\.
2. 2\.We filter to classifier\-confident samples \(Table[17](https://arxiv.org/html/2605.10971#A3.T17)\) and average across retained samples to obtain a per\-step mean activation curveh¯​\(t\)\\bar\{h\}\(t\)\.
3. 3\.We compute absolute deviation from baseline:\|h¯​\(t\)−h¯​\(0\)\|\|\\bar\{h\}\(t\)\-\\bar\{h\}\(0\)\|, which captures emergence regardless of whether the mean activation increases or decreases during denoising\.
4. 4\.We apply moving\-average smoothing with model\-specific window sizes \(MDLM/SEDD: no smoothing for 1024\-step trajectories since they are sufficiently smooth; LLaDA: window=5\{\}=5for 64\-step trajectories; DREAM: window=9\{\}=9for 256\-step trajectories\) to reduce per\-step noise while preserving emergence timing\.
5. 5\.We normalize each curve to\[0,1\]\[0,1\]via min\-max scaling, so that0corresponds to baseline \(fully masked\) and11to full emergence\.

Each panel in Figure[6](https://arxiv.org/html/2605.10971#A4.F6)shows a single model–attribute combination, with one line per SAE layer\. This reveals both the*timing*of emergence \(how early or late features activate\) and the*depth gradient*\(whether shallow or deep layers commit first\)\.

![Refer to caption](https://arxiv.org/html/2605.10971v1/x11.png)Figure 6:Emergence trajectories for all four models and all three attributes \(positive sentiment, sports topic, formal/informal style\)\. Each panel shows normalized absolute deviation\|h¯​\(t\)−h¯​\(0\)\|\|\\bar\{h\}\(t\)\-\\bar\{h\}\(0\)\|from baseline across the full denoising trajectory, with one line per SAE layer\. MDLM and SEDD \(top two rows\) use 1024 denoising steps; LLaDA uses 64 steps; DREAM uses 256 steps\. The main paper Figure[2\(a\)](https://arxiv.org/html/2605.10971#S3.F2.sf1)shows the topic column only\.

### D\.2Block Fractions Across All Layers

##### Methodology\.

The main paper \(Figure[2\(b\)](https://arxiv.org/html/2605.10971#S3.F2.sf2)\) shows block fractions at the deepest layer per model only\. Here we present block fractions at*every*SAE layer, revealing how the temporal distribution of emergence shifts with depth\. Block fractions are computed as follows:

1. 1\.The denoising trajectory is divided intoK=8K\{=\}8equal temporal blocks\. For MDLM/SEDD \(1024 steps\), each block spans 128 steps; for LLaDA \(64 steps\), 8 steps; for DREAM \(256 steps\), 32 steps\.
2. 2\.For each blockbb, we compute the activation increase from the block’s start to its end:δb=h¯​\(eb\)−h¯​\(sb\)\\delta\_\{b\}=\\bar\{h\}\(e\_\{b\}\)\-\\bar\{h\}\(s\_\{b\}\), wheresbs\_\{b\}andebe\_\{b\}are the first and last steps of blockbb\.
3. 3\.Negative deltas are clamped to zero:δb\+=max⁡\(0,δb\)\\delta\_\{b\}^\{\+\}=\\max\(0,\\delta\_\{b\}\)\. This prevents blocks where activation temporarily decreases \(e\.g\., due to non\-monotonic trajectories\) from receiving negative weight\.
4. 4\.Block fractions are computed by normalizing:fb=δb\+/∑b′δb′\+f\_\{b\}=\\delta\_\{b\}^\{\+\}/\\sum\_\{b^\{\\prime\}\}\\delta\_\{b^\{\\prime\}\}^\{\+\}, so they sum to one\.

Figure[7](https://arxiv.org/html/2605.10971#A4.F7)presents the full results, organized as a 4\-row grid \(one row per model, one column per layer\)\. The deepest\-layer panels correspond to the main paper’s Figure[2\(b\)](https://arxiv.org/html/2605.10971#S3.F2.sf2)\.

![Refer to caption](https://arxiv.org/html/2605.10971v1/x12.png)Figure 7:Block fractions across all SAE layers for all four models\. Each row corresponds to a model; each column to a layer\. Comparing columns within a row reveals how the temporal distribution of emergence shifts with layer depth\. MDLM and SEDD \(rows 1–2, 3 layers each\) show clear depth gradients; LLaDA \(row 3, 4 layers\) shows mild variation; DREAM \(row 4, 4 layers\) is uniformly late across all layers\.

### D\.3Cumulative Effect Size Across Models and Layers

##### Methodology\.

The main paper \(Figure[3](https://arxiv.org/html/2605.10971#S3.F3)\) compares cumulative effect size between MDLM and SEDD at layer 7 only\. Here we present cumulative\|d\|\|d\|curves across all layers and all four models\. For each model, attribute, and layer, the procedure is:

1. 1\.We load the full set ofdSAEd\_\{\\text\{SAE\}\}Cohen’sddvalues computed during contrastive feature extraction \(§[C\.5\.2](https://arxiv.org/html/2605.10971#A3.SS5.SSS2)\)\.
2. 2\.We take the absolute value\|dj\|\|d\_\{j\}\|and sort features in descending order\.
3. 3\.We plot the cumulative sum∑i=1r\|d\(i\)\|\\sum\_\{i=1\}^\{r\}\|d\_\{\(i\)\}\|versus feature rankrrfor the top 50 features\.

The cumulative\|d\|\|d\|curve captures how discriminative signal is distributed across features: a steep initial rise indicates that a few features carry most of the signal \(concentrated representation\), while a gradual rise indicates distributed encoding across many features\. Figure[8](https://arxiv.org/html/2605.10971#A4.F8)presents the full4×\(3​–​4\)4\\times\(3\\text\{\-\-\}4\)grid\.

![Refer to caption](https://arxiv.org/html/2605.10971v1/x13.png)Figure 8:Cumulative\|\|Cohen’sd\|d\|vs\. feature rank for all four models and all SAE layers\. Each row is a model \(MDLM\-124M, SEDD\-124M, LLaDA\-8B, DREAM\-7B\); each column is a layer\. Features are sorted by\|d\|\|d\|in descending order and the top 50 are shown\. Topic features \(blue\) consistently dominate, accumulating the most signal per feature rank\. Attributes: positive sentiment \(red\), sports topic \(blue\), and formal/informal style \(green\)\.

### D\.4Anticipatory vs\. Reactive Encoding

##### Methodology\.

To understand whether contrastive SAE features encode attribute information*before*a token is revealed \(anticipatory\) or*after*\(reactive\), we partition feature activations at each denoising step by mask status\. For each steptt, each token position is either masked \(not yet revealed\) or unmasked \(already decoded\)\. We compute separate mean activations over the top\-20 contrastive features for each pool:

- •h¯mask​\(t\)\\bar\{h\}^\{\\text\{mask\}\}\(t\): mean activation across masked positions\.
- •h¯unmask​\(t\)\\bar\{h\}^\{\\text\{unmask\}\}\(t\): mean activation across unmasked positions\.

The*masked fraction*is thenh¯mask​\(t\)/\(h¯mask​\(t\)\+h¯unmask​\(t\)\)\\bar\{h\}^\{\\text\{mask\}\}\(t\)/\(\\bar\{h\}^\{\\text\{mask\}\}\(t\)\+\\bar\{h\}^\{\\text\{unmask\}\}\(t\)\)\. Values above 0\.5 indicate*anticipatory*encoding: features fire preferentially on positions where no token has been revealed yet, suggesting the model is “planning ahead” for what will appear there\. Values below 0\.5 indicate*reactive*encoding: features respond to tokens that have already been placed\. Smoothing is applied with the same model\-specific window sizes as for emergence curves \(MDLM/SEDD: window=31\{\}=31; LLaDA: window=5\{\}=5; DREAM: window=9\{\}=9\)\.

Figure[9](https://arxiv.org/html/2605.10971#A4.F9)shows the masked fraction across denoising progress \(0–100%\) for each attribute and layer on all four models\.

##### Discussion\.

- •MDLM\.Topic is consistently anticipatory across all layers \(∼\{\\sim\}0\.55–0\.80 early on\), with the strongest signal at layer 7\. Sentiment is strongly reactive at layer 5 \(starting at∼\{\\sim\}0\.10\) and progressively less so at deeper layers; layer 7 shows anticipatory sentiment early \(∼\{\\sim\}0\.65\)\. Formality exhibits a similar depth gradient: reactive at layer 5, mixed at deeper layers\.
- •SEDD\.Qualitatively similar to MDLM—topic is anticipatory, sentiment is reactive at shallow layers—but with noisier trajectories and compressed dynamic range\. This reflects SEDD’s score\-entropy training objective, which produces smaller absolute SAE activations: the masked fraction ratio remains meaningful \(topic is still above 0\.5, sentiment below\), but the signal\-to\-noise ratio is lower because both masked and unmasked activations are closer to the ReLU threshold\.
- •LLaDA\.Topic is anticipatory at layers 8–14, then transitions to reactive at deeper layers\. Sentiment and informality are mildly reactive across most layers\. Layer 26 shows anticipatory sentiment\.
- •DREAM\.All attributes remain near 0\.5 for the first∼\{\\sim\}75% of denoising, consistent with the absence of feature emergence during this phase \(§[3\.3](https://arxiv.org/html/2605.10971#S3.SS3)\)\. Late\-stage differentiation is minimal, with slight reactive shifts for topic at layers 8 and 23\.
- •Convergence toward equilibrium\.All anticipatory signals decay toward 0\.5 as denoising progresses and the mask ratio drops\. This is expected: as most positions become unmasked, the distinction between masked and unmasked pools diminishes\.

The anticipatory–reactive distinction has implications for steering: anticipatory features are active early and on positions that will be filled later, suggesting that intervening on these features during early denoising steps can influence the content of tokens not yet decoded\. Reactive features, by contrast, are most active on already\-visible tokens, making them suitable for reinforcing or modifying existing content in later denoising stages\.

![Refer to caption](https://arxiv.org/html/2605.10971v1/x14.png)Figure 9:Masked fraction of SAE feature activation vs\. denoising progress for all four models and layers\. Values above the dashed line \(0\.5\) indicate*anticipatory*encoding; values below indicate*reactive*encoding\. Each row is a model; each column is a layer\. Attributes studied: positive sentiment \(red\), sports topic \(blue\), and formal/informal style \(green\)\. SEDD shows qualitatively similar patterns to MDLM but with compressed dynamic range due to smaller absolute SAE activations from score\-entropy training \(see text\)\.

### D\.5Cross\-Model Feature Cosine Similarity

This section details the methodology and full results behind the cross\-model feature alignment analysis\. The goal is to test whether models that share a training objective learn geometrically aligned SAE features, even when they differ in architecture and scale\.

##### Comparison pairs\.

We compare three model pairs that isolate the effect of training objective from architecture and scale:

1. \(a\)MDLM↔\\leftrightarrowSEDD\(124M↔\\leftrightarrow124M\): same GPT\-2 architecture, same OpenWebText training data,*different*training objectives \(absorbing diffusion vs\. score\-entropy diffusion\)\.
2. \(b\)MDLM↔\\leftrightarrowDream\(124M↔\\leftrightarrow7B\):*same*absorbing\-diffusion loss family, different architectures \(GPT\-2 vs\. Transformer\-XL variant\),56×56\\timesscale difference\.
3. \(c\)MDLM↔\\leftrightarrowLLaDA\(124M↔\\leftrightarrow8B\):*same*absorbing\-diffusion loss family, different architectures \(GPT\-2 vs\. LLaMA\-based\),64×64\\timesscale difference\.

For each pair, we select the top\-50 features by\|Cohen’s​d\|\|\\text\{Cohen's \}d\|for sentiment and topic attributes from each model’s SAE\.

##### Method 1: Direct decoder\-space comparison \(same hidden dimension\)\.

When both models share the same hidden dimensiondd\(MDLM and SEDD, bothd=768d\{=\}768\), we compare decoder directions directly:

sim​\(fiA,fjB\)=cos⁡\(𝐰dec,iA,𝐰dec,jB\),\\mathrm\{sim\}\(f\_\{i\}^\{A\},f\_\{j\}^\{B\}\)=\\cos\\\!\\big\(\\mathbf\{w\}^\{A\}\_\{\\mathrm\{dec\},i\},\\;\\mathbf\{w\}^\{B\}\_\{\\mathrm\{dec\},j\}\\big\),\(18\)where𝐰dec,i∈ℝd\\mathbf\{w\}\_\{\\mathrm\{dec\},i\}\\in\\mathbb\{R\}^\{d\}is theii\-th row of𝐖dec\\mathbf\{W\}\_\{\\mathrm\{dec\}\},ℓ2\\ell\_\{2\}\-normalized\. We compute the full50×5050\\times 50similarity matrix across each model’s top features and extract, for each feature in modelAA, its*best\-match*similaritymaxj⁡sim​\(fiA,fjB\)\\max\_\{j\}\\mathrm\{sim\}\(f\_\{i\}^\{A\},f\_\{j\}^\{B\}\)\. The “mean best\-match similarity” averages these 50 best\-match values\.

This comparison is valid because both SAEs operate in the same residual\-stream vector space: both models share the GPT\-2 architecture and the SAEs are trained on activations at corresponding layers \(L5, L6, L7\)\.

##### Method 2: Vocabulary\-projected comparison \(different hidden dimensions\)\.

When hidden dimensions differ \(MDLMd=768d\{=\}768vs\. Dreamd=3584d\{=\}3584or LLaDAd=4096d\{=\}4096\), direct decoder comparison is not possible\. Instead, we project each decoder direction through the model’s unembedding matrix𝐖unembed\\mathbf\{W\}\_\{\\mathrm\{unembed\}\}into a shared vocabulary space:

𝐯iA=𝐰^dec,iA⋅𝐖unembedA⊤‖𝐰^dec,iA⋅𝐖unembedA⊤‖,sim​\(fiA,fjB\)=𝐯iA⋅𝐯jB,\\mathbf\{v\}\_\{i\}^\{A\}=\\frac\{\\hat\{\\mathbf\{w\}\}^\{A\}\_\{\\mathrm\{dec\},i\}\\cdot\{\\mathbf\{W\}^\{A\}\_\{\\mathrm\{unembed\}\}\}^\{\\top\}\}\{\\big\\\|\\hat\{\\mathbf\{w\}\}^\{A\}\_\{\\mathrm\{dec\},i\}\\cdot\{\\mathbf\{W\}^\{A\}\_\{\\mathrm\{unembed\}\}\}^\{\\top\}\\big\\\|\},\\qquad\\mathrm\{sim\}\(f\_\{i\}^\{A\},f\_\{j\}^\{B\}\)=\\mathbf\{v\}\_\{i\}^\{A\}\\cdot\\mathbf\{v\}\_\{j\}^\{B\},\(19\)where𝐰^dec,i\\hat\{\\mathbf\{w\}\}\_\{\\mathrm\{dec\},i\}denotes theℓ2\\ell\_\{2\}\-normalized decoder row and𝐯iA∈ℝ\|Vshared\|\\mathbf\{v\}\_\{i\}^\{A\}\\in\\mathbb\{R\}^\{\|V\_\{\\mathrm\{shared\}\}\|\}is the projected, renormalized vector restricted to shared vocabulary tokens\. The shared vocabulary consists of all tokens present in both models’ tokenizers \(\|Vshared\|=42,284\|V\_\{\\mathrm\{shared\}\}\|=42\{,\}284for MDLM–Dream;42,45042\{,\}450for MDLM–LLaDA\)\.

This comparison is inherently*asymmetric*: the best match for MDLM featurefif\_\{i\}among Dream’s features \(A→BA\\to Bdirection\) need not coincide with the best match in the reverse direction \(B→AB\\to A\)\. We report both directions separately and average them for the summary statistics\.

##### Layer alignment\.

For the cross\-architecture comparisons, we match layers by functional role based on prior analysis:

- •Sentiment: MDLM L5↔\\leftrightarrowDream L17↔\\leftrightarrowLLaDA L26 \(the layer with strongest sentiment features in each model\)\.
- •Topic: MDLM L6↔\\leftrightarrowDream L23↔\\leftrightarrowLLaDA L26 \(strongest topic features\)\.

For MDLM↔\\leftrightarrowSEDD, both models share the same architecture, so we compare at matched layers \(L5, L6, L7\)\.

##### Full results\.

Table 24:Cross\-model feature alignment\.*Best\-match sim\.*is the mean cosine similarity between each feature in modelAAand its nearest neighbour in modelBB\. For cross\-architecture pairs, both search directions \(A→BA\{\\to\}BandB→AB\{\\to\}A\) are shown\.*Overall mean*is the mean of the full50×5050\\times 50similarity matrix \(a random\-alignment baseline; expected≈0\\approx 0for unrelated feature sets\)\.PairAttributeBest\-match \(A→BA\{\\to\}B\)Best\-match \(B→AB\{\\to\}A\)Max sim\.Overall meanMDLM↔\\leftrightarrowSEDD\(diff\. loss, same arch\.\)Sentiment \(L5\)0\.0790\.133—Topic \(L6\)0\.0810\.116—Mixed \(L7\)0\.0810\.123—MDLM↔\\leftrightarrowDream\(same loss, diff\. arch\.\)Sentiment \(L5↔\\leftrightarrowL17\)0\.3280\.5200\.881−\-0\.0001Topic \(L6↔\\leftrightarrowL23\)0\.3640\.5670\.7960\.015MDLM↔\\leftrightarrowLLaDA\(same loss, diff\. arch\.\)Sentiment \(L5↔\\leftrightarrowL26\)0\.2330\.2110\.543−\-0\.0004Topic \(L6↔\\leftrightarrowL26\)0\.3290\.2450\.7050\.007
##### Key observations\.

- •Different loss⇒\\Rightarroworthogonal features\.MDLM and SEDD features show mean best\-match cosine of0\.080\.08across all three layers—indistinguishable from random alignment in 768\-dimensional space \(expected random cosine≈1/d≈0\.036\\approx 1/\\sqrt\{d\}\\approx 0\.036, with the slightly higher observed values attributable to both models learning*some*shared structure from the same training data\)\. The maximum pairwise similarity is only 0\.133, confirming that no individual feature pair is aligned\.
- •Same loss⇒\\Rightarrowsubstantial alignment despite scale gap\.Same\-loss\-family pairs show3−6×3\{\-\}6\\timeshigher alignment\. MDLM–Dream mean best\-match similarity averages0\.440\.44\(across both directions and both attributes\), with individual feature pairs reaching0\.880\.88cosine\. MDLM–LLaDA averages0\.250\.25mean /0\.710\.71max—lower than Dream, likely reflecting the greater architectural distance \(LLaMA\-based vs\. GPT\-2\)\.
- •Directional asymmetry\.Dream→\\toMDLM best\-match similarity \(0\.52, 0\.57\) consistently exceeds MDLM→\\toDream \(0\.33, 0\.36\)\. This indicates that Dream’s larger SAE \(K=14,336K\{=\}14\{,\}336,d=3,584d\{=\}3\{,\}584\) learns more specific features—each Dream feature maps cleanly to a single MDLM counterpart—while MDLM’s smaller SAE \(K=12,288K\{=\}12\{,\}288,d=768d\{=\}768\) produces more diffuse features that partially match multiple Dream features\. For MDLM–LLaDA, the asymmetry is smaller \(0\.23 vs\. 0\.21 for sentiment; 0\.33 vs\. 0\.24 for topic\), suggesting that LLaDA’s features, while in a higher\-dimensional space, are not as sharply specialized as Dream’s\.
- •Overall mean≈0\\approx 0confirms no systematic bias\.The overall mean similarity \(averaging all 2,500 entries in the50×5050\\times 50matrix\) is effectively zero for all cross\-architecture pairs \(\|\|mean\|<0\.02\|<0\.02\), confirming that the high best\-match values reflect genuine feature\-level correspondence rather than a global bias in the projection space\.
- •Topic features align more than sentiment\.Across both cross\-architecture pairs, topic features show higher best\-match similarity \(MDLM–Dream: 0\.47 vs\. 0\.42; MDLM–LLaDA: 0\.29 vs\. 0\.22\) and higher max similarity\. This is consistent with topic being a more lexically grounded attribute: topic features project onto tight vocabulary clusters \(e\.g\., “touchdowns”, “quarterback” for Sports\), which are more likely to be shared across models than the subtler evaluative vocabulary associated with sentiment\.

## Appendix ESteering Results with Standard Deviations

The steering results with standard deviations for single attributes are shown in Table[25](https://arxiv.org/html/2605.10971#A5.T25)and results with standard deviations for multi\-attributes are shown in Table[26](https://arxiv.org/html/2605.10971#A5.T26)\. The best operating\-pointα\\alphafor each method is indicated as a superscript\.

Table 25:Single\-attribute steering results with standard deviations\. Format: mean±\\pmstd\. Superscripts on Conf denote the selectedα\\alpha\. Same quality gate and selection criteria as Table[3](https://arxiv.org/html/2605.10971#S5.T3)\.Sentiment \(S\)Topic \(T\)Style \(St\)Conf↑\\uparrowPPL↓\\downarrowd2↑\\uparrowConf↑\\uparrowPPL↓\\downarrowd2↑\\uparrowConf↑\\uparrowPPL↓\\downarrowd2↑\\uparrowMDLM\-124MNo Steering25\.5±38\.625\.5\{\\pm\}38\.643±1243\{\\pm\}1292\.6±2\.392\.6\{\\pm\}2\.318\.6±38\.618\.6\{\\pm\}38\.643±1243\{\\pm\}1292\.6±2\.392\.6\{\\pm\}2\.353\.3±16\.053\.3\{\\pm\}16\.043±1243\{\\pm\}1292\.6±2\.392\.6\{\\pm\}2\.3Uniform99\.9±0\.31299\.9\{\\pm\}0\.3^\{12\}93±2593\{\\pm\}2593\.7±1\.793\.7\{\\pm\}1\.799\.9±0\.0299\.9\{\\pm\}0\.0^\{2\}59±1359\{\\pm\}1393\.0±1\.693\.0\{\\pm\}1\.670\.0±7\.6570\.0\{\\pm\}7\.6^\{5\}82±1282\{\\pm\}1293\.8±1\.293\.8\{\\pm\}1\.2Adaptive93\.7±20\.01593\.7\{\\pm\}20\.0^\{15\}57±1257\{\\pm\}1293\.6±1\.693\.6\{\\pm\}1\.699\.9±0\.1699\.9\{\\pm\}0\.1^\{6\}50±1150\{\\pm\}1192\.5±2\.092\.5\{\\pm\}2\.071\.7±9\.21571\.7\{\\pm\}9\.2^\{15\}57±857\{\\pm\}893\.3±1\.593\.3\{\\pm\}1\.5Contrastive Vec\.95\.0±16\.31595\.0\{\\pm\}16\.3^\{15\}421±123421\{\\pm\}12392\.5±5\.192\.5\{\\pm\}5\.168\.9±45\.61168\.9\{\\pm\}45\.6^\{11\}354±117354\{\\pm\}11793\.2±2\.993\.2\{\\pm\}2\.943\.3±16\.2443\.3\{\\pm\}16\.2^\{4\}379±99379\{\\pm\}9995\.6±2\.495\.6\{\\pm\}2\.4Probe74\.3±36\.61474\.3\{\\pm\}36\.6^\{14\}581±140581\{\\pm\}14096\.4±1\.696\.4\{\\pm\}1\.699\.2±6\.31399\.2\{\\pm\}6\.3^\{13\}230±54230\{\\pm\}5492\.4±3\.292\.4\{\\pm\}3\.266\.4±11\.31366\.4\{\\pm\}11\.3^\{13\}1285±2261285\{\\pm\}22698\.4±0\.798\.4\{\\pm\}0\.7PCA20\.7±36\.21420\.7\{\\pm\}36\.2^\{14\}490±120490\{\\pm\}12096\.9±3\.496\.9\{\\pm\}3\.415\.7±36\.0415\.7\{\\pm\}36\.0^\{4\}337±99337\{\\pm\}9994\.3±2\.994\.3\{\\pm\}2\.942\.8±16\.5542\.8\{\\pm\}16\.5^\{5\}383±94383\{\\pm\}9495\.8±2\.595\.8\{\\pm\}2\.5SEDD\-124MNo Steering29\.0±41\.029\.0\{\\pm\}41\.044±1744\{\\pm\}1792\.5±2\.592\.5\{\\pm\}2\.520\.3±39\.220\.3\{\\pm\}39\.244±1744\{\\pm\}1792\.5±2\.592\.5\{\\pm\}2\.557\.7±19\.157\.7\{\\pm\}19\.144±1744\{\\pm\}1792\.5±2\.592\.5\{\\pm\}2\.5Uniform98\.8±5\.81298\.8\{\\pm\}5\.8^\{12\}94±1894\{\\pm\}1891\.1±2\.191\.1\{\\pm\}2\.1100\.0±0\.02100\.0\{\\pm\}0\.0^\{2\}57±2157\{\\pm\}2191\.5±2\.691\.5\{\\pm\}2\.693\.7±4\.71393\.7\{\\pm\}4\.7^\{13\}97±1697\{\\pm\}1684\.1±2\.484\.1\{\\pm\}2\.4Adaptive76\.8±37\.12076\.8\{\\pm\}37\.1^\{20\}55±1255\{\\pm\}1290\.3±2\.990\.3\{\\pm\}2\.9100\.0±0\.04100\.0\{\\pm\}0\.0^\{4\}49±1249\{\\pm\}1290\.7±3\.090\.7\{\\pm\}3\.091\.2±6\.32091\.2\{\\pm\}6\.3^\{20\}48±748\{\\pm\}787\.2±2\.887\.2\{\\pm\}2\.8Contrastive Vec\.99\.8±0\.61899\.8\{\\pm\}0\.6^\{18\}88±2388\{\\pm\}2391\.3±2\.391\.3\{\\pm\}2\.399\.9±0\.01599\.9\{\\pm\}0\.0^\{15\}53±1853\{\\pm\}1891\.5±2\.291\.5\{\\pm\}2\.291\.7±8\.42091\.7\{\\pm\}8\.4^\{20\}81±1481\{\\pm\}1491\.5±2\.491\.5\{\\pm\}2\.4Probe95\.7±18\.0695\.7\{\\pm\}18\.0^\{6\}96±3396\{\\pm\}3391\.4±2\.391\.4\{\\pm\}2\.387\.8±31\.8787\.8\{\\pm\}31\.8^\{7\}76±2976\{\\pm\}2992\.8±2\.392\.8\{\\pm\}2\.387\.2±12\.7987\.2\{\\pm\}12\.7^\{9\}91±2191\{\\pm\}2191\.9±2\.291\.9\{\\pm\}2\.2PCA33\.3±43\.0533\.3\{\\pm\}43\.0^\{5\}51±1451\{\\pm\}1492\.6±2\.292\.6\{\\pm\}2\.232\.9±46\.42032\.9\{\\pm\}46\.4^\{20\}40±1140\{\\pm\}1192\.4±2\.892\.4\{\\pm\}2\.856\.7±18\.4156\.7\{\\pm\}18\.4^\{1\}40±1140\{\\pm\}1191\.8±2\.591\.8\{\\pm\}2\.5LLaDA\-8BNo Steering69\.1±44\.869\.1\{\\pm\}44\.816±1816\{\\pm\}1885\.1±20\.985\.1\{\\pm\}20\.913\.9±31\.313\.9\{\\pm\}31\.316±1816\{\\pm\}1885\.1±20\.985\.1\{\\pm\}20\.922\.2±23\.022\.2\{\\pm\}23\.016±1816\{\\pm\}1885\.1±20\.985\.1\{\\pm\}20\.9Uniform89\.0±29\.81589\.0\{\\pm\}29\.8^\{15\}12±912\{\\pm\}976\.5±22\.976\.5\{\\pm\}22\.999\.9±0\.61299\.9\{\\pm\}0\.6^\{12\}6±26\{\\pm\}244\.2±13\.344\.2\{\\pm\}13\.369\.0±24\.51269\.0\{\\pm\}24\.5^\{12\}7±67\{\\pm\}643\.7±20\.543\.7\{\\pm\}20\.5Adaptive79\.6±37\.91479\.6\{\\pm\}37\.9^\{14\}14±1014\{\\pm\}1080\.6±23\.280\.6\{\\pm\}23\.299\.7±2\.51599\.7\{\\pm\}2\.5^\{15\}6±46\{\\pm\}450\.9±18\.450\.9\{\\pm\}18\.466\.1±27\.61566\.1\{\\pm\}27\.6^\{15\}8±98\{\\pm\}946\.8±25\.946\.8\{\\pm\}25\.9Contrastive Vec\.97\.1±7\.1497\.1\{\\pm\}7\.1^\{4\}10±1110\{\\pm\}1172\.7±18\.772\.7\{\\pm\}18\.729\.6±41\.81429\.6\{\\pm\}41\.8^\{14\}8±48\{\\pm\}474\.4±25\.374\.4\{\\pm\}25\.353\.7±25\.41153\.7\{\\pm\}25\.4^\{11\}7±47\{\\pm\}471\.4±25\.671\.4\{\\pm\}25\.6Probe89\.0±29\.5789\.0\{\\pm\}29\.5^\{7\}7±37\{\\pm\}369\.3±26\.169\.3\{\\pm\}26\.132\.3±40\.11432\.3\{\\pm\}40\.1^\{14\}8±58\{\\pm\}570\.3±24\.770\.3\{\\pm\}24\.748\.6±23\.21448\.6\{\\pm\}23\.2^\{14\}8±58\{\\pm\}566\.9±25\.766\.9\{\\pm\}25\.7PCA94\.2±21\.5294\.2\{\\pm\}21\.5^\{2\}8±68\{\\pm\}673\.0±25\.473\.0\{\\pm\}25\.424\.1±38\.3924\.1\{\\pm\}38\.3^\{9\}8±48\{\\pm\}473\.2±26\.173\.2\{\\pm\}26\.149\.7±21\.2949\.7\{\\pm\}21\.2^\{9\}9±59\{\\pm\}573\.4±24\.473\.4\{\\pm\}24\.4Prompt80\.9±36\.480\.9\{\\pm\}36\.416±1416\{\\pm\}1484\.7±20\.984\.7\{\\pm\}20\.964\.9±45\.564\.9\{\\pm\}45\.510±910\{\\pm\}970\.0±25\.270\.0\{\\pm\}25\.245\.1±24\.945\.1\{\\pm\}24\.920±2620\{\\pm\}2665\.3±28\.065\.3\{\\pm\}28\.0DREAM\-7BNo Steering57\.7±46\.957\.7\{\\pm\}46\.914±914\{\\pm\}987\.9±13\.987\.9\{\\pm\}13\.910\.5±25\.710\.5\{\\pm\}25\.714±914\{\\pm\}987\.9±13\.987\.9\{\\pm\}13\.926\.5±22\.826\.5\{\\pm\}22\.814±914\{\\pm\}987\.9±13\.987\.9\{\\pm\}13\.9Uniform99\.5±5\.51499\.5\{\\pm\}5\.5^\{14\}14±614\{\\pm\}684\.5±13\.184\.5\{\\pm\}13\.1100\.0±0\.06100\.0\{\\pm\}0\.0^\{6\}53±2653\{\\pm\}2683\.6±10\.783\.6\{\\pm\}10\.797\.5±2\.51597\.5\{\\pm\}2\.5^\{15\}75±9275\{\\pm\}9260\.4±13\.860\.4\{\\pm\}13\.8Adaptive97\.6±13\.91497\.6\{\\pm\}13\.9^\{14\}12±512\{\\pm\}585\.1±14\.485\.1\{\\pm\}14\.4100\.0±0\.012100\.0\{\\pm\}0\.0^\{12\}37±2637\{\\pm\}2647\.6±12\.447\.6\{\\pm\}12\.496\.3±3\.41596\.3\{\\pm\}3\.4^\{15\}60±13560\{\\pm\}13554\.4±17\.354\.4\{\\pm\}17\.3Contrastive Vec\.95\.3±19\.21595\.3\{\\pm\}19\.2^\{15\}16±1016\{\\pm\}1083\.6±15\.783\.6\{\\pm\}15\.788\.7±26\.61588\.7\{\\pm\}26\.6^\{15\}15±715\{\\pm\}789\.4±9\.089\.4\{\\pm\}9\.092\.8±7\.1992\.8\{\\pm\}7\.1^\{9\}41±4341\{\\pm\}4379\.0±22\.979\.0\{\\pm\}22\.9Probe86\.6±32\.41586\.6\{\\pm\}32\.4^\{15\}15±1115\{\\pm\}1187\.2±12\.987\.2\{\\pm\}12\.976\.7±38\.21576\.7\{\\pm\}38\.2^\{15\}14±814\{\\pm\}883\.2±16\.783\.2\{\\pm\}16\.798\.5±3\.11498\.5\{\\pm\}3\.1^\{14\}48±5348\{\\pm\}5394\.4±16\.994\.4\{\\pm\}16\.9PCA65\.3±44\.8965\.3\{\\pm\}44\.8^\{9\}15±1015\{\\pm\}1088\.7±13\.688\.7\{\\pm\}13\.618\.3±32\.41518\.3\{\\pm\}32\.4^\{15\}16±1916\{\\pm\}1982\.6±15\.982\.6\{\\pm\}15\.927\.8±23\.21527\.8\{\\pm\}23\.2^\{15\}15±1015\{\\pm\}1089\.4±12\.289\.4\{\\pm\}12\.2Prompt80\.3±36\.580\.3\{\\pm\}36\.512±812\{\\pm\}881\.8±18\.681\.8\{\\pm\}18\.661\.5±47\.061\.5\{\\pm\}47\.013±513\{\\pm\}584\.1±14\.084\.1\{\\pm\}14\.020\.6±21\.420\.6\{\\pm\}21\.415±915\{\\pm\}978\.7±20\.578\.7\{\\pm\}20\.5

Table 26:Multi\-attribute steering results with per\-attribute confidence and standard deviations\. S: sentiment confidence; T: topic confidence; St: style confidence \(formal for MDLM/SEDD, informal for LLaDA/DREAM\)\. Superscripts on the first attribute denote selectedα\\alpha\. Same quality gate and selection criteria as Table[3](https://arxiv.org/html/2605.10971#S5.T3)\.S\+TS\+StSt\+TS\+T\+StS↑\\uparrowT↑\\uparrowPPL↓\\downarrowd2↑\\uparrowS↑\\uparrowSt↑\\uparrowPPL↓\\downarrowd2↑\\uparrowSt↑\\uparrowT↑\\uparrowPPL↓\\downarrowd2↑\\uparrowS↑\\uparrowT↑\\uparrowSt↑\\uparrowPPL↓\\downarrowd2↑\\uparrowMDLM\-124MNo Steering25\.5±38\.625\.5\{\\pm\}38\.618\.6±38\.618\.6\{\\pm\}38\.643±1243\{\\pm\}1292\.6±2\.392\.6\{\\pm\}2\.325\.5±38\.625\.5\{\\pm\}38\.653\.3±16\.053\.3\{\\pm\}16\.043±1243\{\\pm\}1292\.6±2\.392\.6\{\\pm\}2\.353\.3±16\.053\.3\{\\pm\}16\.018\.6±38\.618\.6\{\\pm\}38\.643±1243\{\\pm\}1292\.6±2\.392\.6\{\\pm\}2\.325\.5±38\.625\.5\{\\pm\}38\.618\.6±38\.618\.6\{\\pm\}38\.653\.3±16\.053\.3\{\\pm\}16\.043±1243\{\\pm\}1292\.6±2\.392\.6\{\\pm\}2\.3Uniform93\.2±18\.4393\.2\{\\pm\}18\.4^\{3\}99\.9±0\.099\.9\{\\pm\}0\.091±1791\{\\pm\}1793\.1±1\.593\.1\{\\pm\}1\.528\.5±38\.2428\.5\{\\pm\}38\.2^\{4\}69\.3±8\.669\.3\{\\pm\}8\.688±1388\{\\pm\}1393\.9±1\.393\.9\{\\pm\}1\.349\.1±6\.2249\.1\{\\pm\}6\.2^\{2\}99\.9±0\.099\.9\{\\pm\}0\.073±1373\{\\pm\}1392\.9±1\.592\.9\{\\pm\}1\.586\.3±26\.9286\.3\{\\pm\}26\.9^\{2\}99\.9±0\.099\.9\{\\pm\}0\.050\.2±7\.150\.2\{\\pm\}7\.182±1482\{\\pm\}1493\.1±1\.393\.1\{\\pm\}1\.3Adaptive95\.2±18\.81595\.2\{\\pm\}18\.8^\{15\}99\.9±0\.099\.9\{\\pm\}0\.071±1671\{\\pm\}1692\.5±1\.692\.5\{\\pm\}1\.655\.0±44\.61555\.0\{\\pm\}44\.6^\{15\}72\.4±10\.272\.4\{\\pm\}10\.273±1073\{\\pm\}1093\.5±1\.293\.5\{\\pm\}1\.258\.0±10\.11458\.0\{\\pm\}10\.1^\{14\}97\.2±15\.797\.2\{\\pm\}15\.759±1059\{\\pm\}1091\.9±1\.891\.9\{\\pm\}1\.890\.1±23\.81590\.1\{\\pm\}23\.8^\{15\}95\.8±19\.795\.8\{\\pm\}19\.758\.1±7\.658\.1\{\\pm\}7\.684±1484\{\\pm\}1492\.8±1\.492\.8\{\\pm\}1\.4Uniform\+E98\.5±6\.4698\.5\{\\pm\}6\.4^\{6\}99\.9±0\.099\.9\{\\pm\}0\.096±2096\{\\pm\}2093\.3±1\.393\.3\{\\pm\}1\.364\.8±41\.9764\.8\{\\pm\}41\.9^\{7\}66\.3±11\.966\.3\{\\pm\}11\.988±1488\{\\pm\}1494\.1±1\.394\.1\{\\pm\}1\.350\.6±6\.2650\.6\{\\pm\}6\.2^\{6\}99\.9±0\.099\.9\{\\pm\}0\.083±1483\{\\pm\}1492\.9±1\.492\.9\{\\pm\}1\.491\.4±22\.2491\.4\{\\pm\}22\.2^\{4\}99\.9±0\.099\.9\{\\pm\}0\.049\.7±6\.749\.7\{\\pm\}6\.784±1684\{\\pm\}1693\.3±1\.493\.3\{\\pm\}1\.4Adaptive\+E93\.9±19\.61493\.9\{\\pm\}19\.6^\{14\}97\.4±15\.697\.4\{\\pm\}15\.658±1458\{\\pm\}1492\.9±1\.692\.9\{\\pm\}1\.673\.7±38\.91573\.7\{\\pm\}38\.9^\{15\}64\.4±12\.564\.4\{\\pm\}12\.563±1163\{\\pm\}1194\.1±1\.494\.1\{\\pm\}1\.452\.5±9\.71452\.5\{\\pm\}9\.7^\{14\}92\.9±25\.492\.9\{\\pm\}25\.450±1050\{\\pm\}1092\.7±1\.992\.7\{\\pm\}1\.992\.9±21\.21492\.9\{\\pm\}21\.2^\{14\}92\.9±25\.592\.9\{\\pm\}25\.554\.1±8\.554\.1\{\\pm\}8\.562±1262\{\\pm\}1293\.1±1\.593\.1\{\\pm\}1\.5Contrastive Vec\.84\.4±31\.51584\.4\{\\pm\}31\.5^\{15\}64\.0±47\.764\.0\{\\pm\}47\.7376±128376\{\\pm\}12891\.8±3\.891\.8\{\\pm\}3\.891\.7±22\.51591\.7\{\\pm\}22\.5^\{15\}67\.5±16\.767\.5\{\\pm\}16\.7494±128494\{\\pm\}12895\.3±3\.595\.3\{\\pm\}3\.528\.7±7\.81528\.7\{\\pm\}7\.8^\{15\}98\.9±9\.998\.9\{\\pm\}9\.9389±96389\{\\pm\}9695\.1±2\.695\.1\{\\pm\}2\.693\.9±18\.81593\.9\{\\pm\}18\.8^\{15\}77\.3±41\.377\.3\{\\pm\}41\.352\.3±13\.552\.3\{\\pm\}13\.5466±113466\{\\pm\}11394\.5±3\.194\.5\{\\pm\}3\.1Probe72\.8±37\.11572\.8\{\\pm\}37\.1^\{15\}99\.8±1\.399\.8\{\\pm\}1\.3324±65324\{\\pm\}6594\.1±2\.094\.1\{\\pm\}2\.063\.1±42\.51363\.1\{\\pm\}42\.5^\{13\}71\.1±14\.871\.1\{\\pm\}14\.8856±176856\{\\pm\}17697\.0±2\.197\.0\{\\pm\}2\.152\.5±11\.21452\.5\{\\pm\}11\.2^\{14\}99\.4±7\.199\.4\{\\pm\}7\.1873±180873\{\\pm\}18097\.8±0\.997\.8\{\\pm\}0\.964\.8±39\.61564\.8\{\\pm\}39\.6^\{15\}98\.4±12\.198\.4\{\\pm\}12\.152\.1±12\.252\.1\{\\pm\}12\.2798±164798\{\\pm\}16497\.4±1\.197\.4\{\\pm\}1\.1PCA20\.1±34\.21220\.1\{\\pm\}34\.2^\{12\}11\.2±30\.711\.2\{\\pm\}30\.7310±94310\{\\pm\}9493\.5±4\.093\.5\{\\pm\}4\.019\.9±34\.2719\.9\{\\pm\}34\.2^\{7\}40\.7±14\.740\.7\{\\pm\}14\.7406±102406\{\\pm\}10295\.9±2\.895\.9\{\\pm\}2\.844\.3±16\.4144\.3\{\\pm\}16\.4^\{1\}12\.8±33\.112\.8\{\\pm\}33\.1346±95346\{\\pm\}9594\.7±3\.394\.7\{\\pm\}3\.318\.6±34\.6618\.6\{\\pm\}34\.6^\{6\}15\.4±35\.315\.4\{\\pm\}35\.339\.6±14\.539\.6\{\\pm\}14\.5396±106396\{\\pm\}10695\.9±2\.795\.9\{\\pm\}2\.7SEDD\-124MNo Steering29\.0±41\.029\.0\{\\pm\}41\.020\.3±39\.220\.3\{\\pm\}39\.244±1744\{\\pm\}1792\.5±2\.592\.5\{\\pm\}2\.529\.0±41\.029\.0\{\\pm\}41\.057\.7±19\.157\.7\{\\pm\}19\.144±1744\{\\pm\}1792\.5±2\.592\.5\{\\pm\}2\.557\.7±19\.157\.7\{\\pm\}19\.120\.3±39\.220\.3\{\\pm\}39\.244±1744\{\\pm\}1792\.5±2\.592\.5\{\\pm\}2\.529\.0±41\.029\.0\{\\pm\}41\.020\.3±39\.220\.3\{\\pm\}39\.257\.7±19\.157\.7\{\\pm\}19\.144±1744\{\\pm\}1792\.5±2\.592\.5\{\\pm\}2\.5Uniform92\.7±19\.8492\.7\{\\pm\}19\.8^\{4\}100\.0±0\.0100\.0\{\\pm\}0\.098±2498\{\\pm\}2489\.3±2\.389\.3\{\\pm\}2\.376\.3±34\.3876\.3\{\\pm\}34\.3^\{8\}95\.9±4\.795\.9\{\\pm\}4\.798±1698\{\\pm\}1685\.1±2\.585\.1\{\\pm\}2\.569\.0±8\.9469\.0\{\\pm\}8\.9^\{4\}100\.0±0\.0100\.0\{\\pm\}0\.093±2193\{\\pm\}2188\.1±2\.888\.1\{\\pm\}2\.892\.2±20\.7392\.2\{\\pm\}20\.7^\{3\}100\.0±0\.0100\.0\{\\pm\}0\.068\.9±9\.868\.9\{\\pm\}9\.888±2088\{\\pm\}2088\.5±2\.788\.5\{\\pm\}2\.7Adaptive92\.8±18\.51892\.8\{\\pm\}18\.5^\{18\}100\.0±0\.0100\.0\{\\pm\}0\.067±1367\{\\pm\}1387\.2±2\.487\.2\{\\pm\}2\.465\.9±38\.31965\.9\{\\pm\}38\.3^\{19\}95\.3±4\.795\.3\{\\pm\}4\.751±851\{\\pm\}884\.4±3\.384\.4\{\\pm\}3\.378\.9±6\.92078\.9\{\\pm\}6\.9^\{20\}100\.0±0\.0100\.0\{\\pm\}0\.057±1257\{\\pm\}1284\.0±2\.484\.0\{\\pm\}2\.495\.0±16\.32095\.0\{\\pm\}16\.3^\{20\}100\.0±0\.0100\.0\{\\pm\}0\.082\.5±7\.082\.5\{\\pm\}7\.063±1363\{\\pm\}1383\.7±2\.783\.7\{\\pm\}2\.7Uniform\+E95\.3±17\.9995\.3\{\\pm\}17\.9^\{9\}98\.4±12\.198\.4\{\\pm\}12\.195±2395\{\\pm\}2391\.1±2\.091\.1\{\\pm\}2\.095\.8±16\.11195\.8\{\\pm\}16\.1^\{11\}95\.8±5\.795\.8\{\\pm\}5\.796±1696\{\\pm\}1687\.7±2\.887\.7\{\\pm\}2\.880\.2±10\.1680\.2\{\\pm\}10\.1^\{6\}89\.7±29\.789\.7\{\\pm\}29\.798±2098\{\\pm\}2088\.3±2\.888\.3\{\\pm\}2\.894\.7±17\.6894\.7\{\\pm\}17\.6^\{8\}60\.4±48\.460\.4\{\\pm\}48\.482\.7±12\.382\.7\{\\pm\}12\.399±2299\{\\pm\}2289\.7±2\.689\.7\{\\pm\}2\.6Adaptive\+E82\.8±32\.91982\.8\{\\pm\}32\.9^\{19\}77\.4±41\.277\.4\{\\pm\}41\.259±1659\{\\pm\}1691\.5±2\.091\.5\{\\pm\}2\.070\.2±39\.42070\.2\{\\pm\}39\.4^\{20\}88\.3±12\.788\.3\{\\pm\}12\.758±1158\{\\pm\}1189\.9±2\.889\.9\{\\pm\}2\.878\.9±6\.92078\.9\{\\pm\}6\.9^\{20\}100\.0±0\.0100\.0\{\\pm\}0\.057±1257\{\\pm\}1284\.0±2\.484\.0\{\\pm\}2\.477\.1±37\.32077\.1\{\\pm\}37\.3^\{20\}62\.9±47\.362\.9\{\\pm\}47\.373\.4±15\.173\.4\{\\pm\}15\.161±1461\{\\pm\}1491\.0±2\.391\.0\{\\pm\}2\.3Contrastive Vec\.99\.0±4\.02099\.0\{\\pm\}4\.0^\{20\}100\.0±0\.0100\.0\{\\pm\}0\.091±2591\{\\pm\}2592\.1±1\.992\.1\{\\pm\}1\.999\.7±0\.62099\.7\{\\pm\}0\.6^\{20\}96\.4±5\.196\.4\{\\pm\}5\.179±1479\{\\pm\}1490\.0±2\.490\.0\{\\pm\}2\.468\.2±13\.32068\.2\{\\pm\}13\.3^\{20\}99\.5±6\.899\.5\{\\pm\}6\.886±2086\{\\pm\}2092\.5±1\.792\.5\{\\pm\}1\.797\.2±12\.82097\.2\{\\pm\}12\.8^\{20\}85\.5±34\.885\.5\{\\pm\}34\.879\.8±11\.879\.8\{\\pm\}11\.887±1787\{\\pm\}1791\.7±1\.891\.7\{\\pm\}1\.8Probe91\.6±25\.8791\.6\{\\pm\}25\.8^\{7\}78\.6±39\.778\.6\{\\pm\}39\.798±3198\{\\pm\}3192\.3±2\.692\.3\{\\pm\}2\.689\.5±27\.0689\.5\{\\pm\}27\.0^\{6\}81\.7±16\.681\.7\{\\pm\}16\.688±2788\{\\pm\}2791\.9±2\.491\.9\{\\pm\}2\.459\.0±19\.7959\.0\{\\pm\}19\.7^\{9\}89\.1±30\.289\.1\{\\pm\}30\.298±3798\{\\pm\}3792\.9±2\.392\.9\{\\pm\}2\.394\.8±19\.6794\.8\{\\pm\}19\.6^\{7\}74\.3±43\.274\.3\{\\pm\}43\.270\.7±18\.370\.7\{\\pm\}18\.394±2994\{\\pm\}2992\.5±2\.292\.5\{\\pm\}2\.2PCA28\.3±40\.4528\.3\{\\pm\}40\.4^\{5\}21\.9±40\.621\.9\{\\pm\}40\.648±1248\{\\pm\}1292\.8±2\.292\.8\{\\pm\}2\.259\.7±43\.11959\.7\{\\pm\}43\.1^\{19\}80\.4±14\.580\.4\{\\pm\}14\.572±1972\{\\pm\}1992\.9±2\.092\.9\{\\pm\}2\.054\.5±18\.5654\.5\{\\pm\}18\.5^\{6\}27\.2±43\.827\.2\{\\pm\}43\.837±1037\{\\pm\}1091\.7±2\.791\.7\{\\pm\}2\.741\.4±44\.01741\.4\{\\pm\}44\.0^\{17\}33\.3±46\.533\.3\{\\pm\}46\.561\.7±16\.361\.7\{\\pm\}16\.343±1443\{\\pm\}1492\.7±2\.392\.7\{\\pm\}2\.3LLaDA\-8BNo Steering69\.1±44\.869\.1\{\\pm\}44\.813\.9±31\.313\.9\{\\pm\}31\.316±1816\{\\pm\}1885\.1±20\.985\.1\{\\pm\}20\.969\.1±44\.869\.1\{\\pm\}44\.822\.2±23\.022\.2\{\\pm\}23\.016±1816\{\\pm\}1885\.1±20\.985\.1\{\\pm\}20\.913\.9±31\.313\.9\{\\pm\}31\.322\.2±23\.022\.2\{\\pm\}23\.016±1816\{\\pm\}1885\.1±20\.985\.1\{\\pm\}20\.969\.1±44\.869\.1\{\\pm\}44\.813\.9±31\.313\.9\{\\pm\}31\.322\.2±23\.022\.2\{\\pm\}23\.016±1816\{\\pm\}1885\.1±20\.985\.1\{\\pm\}20\.9Uniform94\.7±20\.81294\.7\{\\pm\}20\.8^\{12\}99\.9±0\.399\.9\{\\pm\}0\.36±36\{\\pm\}343\.8±15\.543\.8\{\\pm\}15\.576\.0±41\.81176\.0\{\\pm\}41\.8^\{11\}54\.0±28\.154\.0\{\\pm\}28\.17±67\{\\pm\}643\.6±20\.243\.6\{\\pm\}20\.292\.0±26\.5992\.0\{\\pm\}26\.5^\{9\}44\.0±28\.444\.0\{\\pm\}28\.46±46\{\\pm\}445\.2±17\.045\.2\{\\pm\}17\.086\.2±33\.0886\.2\{\\pm\}33\.0^\{8\}80\.2±38\.980\.2\{\\pm\}38\.946\.8±28\.546\.8\{\\pm\}28\.57±137\{\\pm\}1345\.2±18\.145\.2\{\\pm\}18\.1Adaptive91\.1±26\.71291\.1\{\\pm\}26\.7^\{12\}98\.2±12\.298\.2\{\\pm\}12\.28±58\{\\pm\}556\.9±19\.656\.9\{\\pm\}19\.673\.2±42\.81573\.2\{\\pm\}42\.8^\{15\}61\.1±28\.561\.1\{\\pm\}28\.57±67\{\\pm\}645\.4±24\.345\.4\{\\pm\}24\.394\.9±20\.41394\.9\{\\pm\}20\.4^\{13\}44\.6±27\.544\.6\{\\pm\}27\.57±57\{\\pm\}546\.7±19\.746\.7\{\\pm\}19\.785\.0±33\.71285\.0\{\\pm\}33\.7^\{12\}86\.8±32\.286\.8\{\\pm\}32\.245\.2±28\.145\.2\{\\pm\}28\.17±67\{\\pm\}646\.1±19\.546\.1\{\\pm\}19\.5Uniform\+E89\.4±28\.81589\.4\{\\pm\}28\.8^\{15\}95\.4±19\.895\.4\{\\pm\}19\.87±67\{\\pm\}654\.4±20\.154\.4\{\\pm\}20\.160\.6±47\.71260\.6\{\\pm\}47\.7^\{12\}61\.9±26\.061\.9\{\\pm\}26\.07±57\{\\pm\}543\.1±20\.143\.1\{\\pm\}20\.187\.5±31\.41087\.5\{\\pm\}31\.4^\{10\}46\.4±28\.846\.4\{\\pm\}28\.86±46\{\\pm\}443\.5±18\.643\.5\{\\pm\}18\.681\.3±37\.1981\.3\{\\pm\}37\.1^\{9\}77\.5±40\.577\.5\{\\pm\}40\.547\.4±29\.147\.4\{\\pm\}29\.16±66\{\\pm\}643\.4±18\.343\.4\{\\pm\}18\.3Adaptive\+E91\.7±25\.91591\.7\{\\pm\}25\.9^\{15\}87\.5±31\.987\.5\{\\pm\}31\.99±89\{\\pm\}865\.0±24\.265\.0\{\\pm\}24\.276\.6±41\.41576\.6\{\\pm\}41\.4^\{15\}39\.3±30\.239\.3\{\\pm\}30\.211±1011\{\\pm\}1063\.1±27\.563\.1\{\\pm\}27\.570\.9±43\.61570\.9\{\\pm\}43\.6^\{15\}41\.9±30\.141\.9\{\\pm\}30\.111±1011\{\\pm\}1058\.7±24\.158\.7\{\\pm\}24\.184\.7±34\.81584\.7\{\\pm\}34\.8^\{15\}71\.4±44\.171\.4\{\\pm\}44\.141\.9±30\.441\.9\{\\pm\}30\.48±108\{\\pm\}1052\.3±22\.152\.3\{\\pm\}22\.1Contrastive Vec\.93\.4±21\.81593\.4\{\\pm\}21\.8^\{15\}26\.4±40\.426\.4\{\\pm\}40\.48±58\{\\pm\}581\.9±18\.381\.9\{\\pm\}18\.395\.3±18\.91095\.3\{\\pm\}18\.9^\{10\}40\.1±25\.840\.1\{\\pm\}25\.88±58\{\\pm\}566\.2±22\.066\.2\{\\pm\}22\.036\.8±45\.91336\.8\{\\pm\}45\.9^\{13\}55\.1±23\.655\.1\{\\pm\}23\.68±48\{\\pm\}475\.9±26\.375\.9\{\\pm\}26\.399\.0±2\.81199\.0\{\\pm\}2\.8^\{11\}30\.3±41\.030\.3\{\\pm\}41\.042\.0±26\.442\.0\{\\pm\}26\.410±410\{\\pm\}484\.1±20\.184\.1\{\\pm\}20\.1Probe78\.1±37\.7378\.1\{\\pm\}37\.7^\{3\}32\.3±41\.632\.3\{\\pm\}41\.68±38\{\\pm\}374\.3±22\.474\.3\{\\pm\}22\.484\.5±31\.4484\.5\{\\pm\}31\.4^\{4\}49\.8±19\.049\.8\{\\pm\}19\.08±38\{\\pm\}372\.6±20\.572\.6\{\\pm\}20\.535\.5±41\.71035\.5\{\\pm\}41\.7^\{10\}54\.5±18\.754\.5\{\\pm\}18\.79±49\{\\pm\}479\.6±20\.879\.6\{\\pm\}20\.888\.7±29\.61588\.7\{\\pm\}29\.6^\{15\}28\.9±40\.028\.9\{\\pm\}40\.045\.4±25\.345\.4\{\\pm\}25\.37±47\{\\pm\}467\.1±25\.067\.1\{\\pm\}25\.0PCA92\.5±22\.8692\.5\{\\pm\}22\.8^\{6\}23\.0±39\.123\.0\{\\pm\}39\.17±47\{\\pm\}461\.5±27\.861\.5\{\\pm\}27\.895\.7±14\.21095\.7\{\\pm\}14\.2^\{10\}50\.6±20\.650\.6\{\\pm\}20\.68±78\{\\pm\}769\.9±22\.269\.9\{\\pm\}22\.236\.1±46\.81136\.1\{\\pm\}46\.8^\{11\}43\.1±28\.943\.1\{\\pm\}28\.97±47\{\\pm\}466\.0±24\.066\.0\{\\pm\}24\.082\.7±35\.7182\.7\{\\pm\}35\.7^\{1\}30\.3±41\.030\.3\{\\pm\}41\.035\.5±26\.835\.5\{\\pm\}26\.88±48\{\\pm\}473\.2±24\.173\.2\{\\pm\}24\.1Prompt88\.7±29\.288\.7\{\\pm\}29\.250\.5±48\.450\.5\{\\pm\}48\.417±1717\{\\pm\}1782\.7±21\.982\.7\{\\pm\}21\.977\.6±40\.277\.6\{\\pm\}40\.219\.3±23\.819\.3\{\\pm\}23\.822±2222\{\\pm\}2280\.4±25\.180\.4\{\\pm\}25\.174\.5±41\.974\.5\{\\pm\}41\.937\.3±23\.937\.3\{\\pm\}23\.912±912\{\\pm\}974\.9±24\.874\.9\{\\pm\}24\.892\.4±25\.292\.4\{\\pm\}25\.248\.2±49\.148\.2\{\\pm\}49\.121\.2±22\.821\.2\{\\pm\}22\.819±1819\{\\pm\}1886\.2±21\.186\.2\{\\pm\}21\.1DREAM\-7BNo Steering57\.7±46\.957\.7\{\\pm\}46\.910\.5±25\.710\.5\{\\pm\}25\.714±914\{\\pm\}987\.9±13\.987\.9\{\\pm\}13\.957\.7±46\.957\.7\{\\pm\}46\.926\.5±22\.826\.5\{\\pm\}22\.814±914\{\\pm\}987\.9±13\.987\.9\{\\pm\}13\.910\.5±25\.710\.5\{\\pm\}25\.726\.5±22\.826\.5\{\\pm\}22\.814±914\{\\pm\}987\.9±13\.987\.9\{\\pm\}13\.957\.7±46\.957\.7\{\\pm\}46\.910\.5±25\.710\.5\{\\pm\}25\.726\.5±22\.826\.5\{\\pm\}22\.814±914\{\\pm\}987\.9±13\.987\.9\{\\pm\}13\.9Uniform99\.8±1\.1599\.8\{\\pm\}1\.1^\{5\}99\.9±0\.099\.9\{\\pm\}0\.026±1126\{\\pm\}1178\.6±12\.978\.6\{\\pm\}12\.9100\.0±0\.015100\.0\{\\pm\}0\.0^\{15\}93\.9±3\.593\.9\{\\pm\}3\.561±2161\{\\pm\}2167\.2±9\.967\.2\{\\pm\}9\.998\.1±12\.5498\.1\{\\pm\}12\.5^\{4\}67\.6±20\.067\.6\{\\pm\}20\.072±6472\{\\pm\}6486\.2±10\.986\.2\{\\pm\}10\.997\.2±14\.7497\.2\{\\pm\}14\.7^\{4\}98\.8±10\.298\.8\{\\pm\}10\.262\.9±21\.662\.9\{\\pm\}21\.667±6467\{\\pm\}6483\.7±11\.483\.7\{\\pm\}11\.4Adaptive98\.6±7\.31398\.6\{\\pm\}7\.3^\{13\}100\.0±0\.0100\.0\{\\pm\}0\.050±2550\{\\pm\}2556\.8±13\.256\.8\{\\pm\}13\.2100\.0±0\.015100\.0\{\\pm\}0\.0^\{15\}92\.4±4\.192\.4\{\\pm\}4\.134±1434\{\\pm\}1454\.3±11\.754\.3\{\\pm\}11\.799\.0±9\.8899\.0\{\\pm\}9\.8^\{8\}82\.5±7\.782\.5\{\\pm\}7\.794±5394\{\\pm\}5370\.7±13\.170\.7\{\\pm\}13\.195\.6±18\.4695\.6\{\\pm\}18\.4^\{6\}98\.3±12\.398\.3\{\\pm\}12\.373\.7±11\.073\.7\{\\pm\}11\.074±6674\{\\pm\}6677\.6±13\.577\.6\{\\pm\}13\.5Uniform\+E100\.0±0\.011100\.0\{\\pm\}0\.0^\{11\}100\.0±0\.0100\.0\{\\pm\}0\.021±821\{\\pm\}877\.4±10\.677\.4\{\\pm\}10\.6100\.0±0\.015100\.0\{\\pm\}0\.0^\{15\}90\.6±4\.290\.6\{\\pm\}4\.252±2552\{\\pm\}2566\.3±10\.566\.3\{\\pm\}10\.596\.4±17\.1596\.4\{\\pm\}17\.1^\{5\}80\.1±13\.280\.1\{\\pm\}13\.270±7970\{\\pm\}7985\.3±13\.185\.3\{\\pm\}13\.199\.5±7\.0799\.5\{\\pm\}7\.0^\{7\}95\.8±18\.895\.8\{\\pm\}18\.877\.9±13\.777\.9\{\\pm\}13\.778±6278\{\\pm\}6282\.8±10\.282\.8\{\\pm\}10\.2Adaptive\+E99\.6±3\.51399\.6\{\\pm\}3\.5^\{13\}99\.9±0\.099\.9\{\\pm\}0\.017±717\{\\pm\}776\.7±13\.176\.7\{\\pm\}13\.1100\.0±0\.015100\.0\{\\pm\}0\.0^\{15\}88\.9±5\.788\.9\{\\pm\}5\.731±1831\{\\pm\}1856\.1±11\.956\.1\{\\pm\}11\.988\.2±31\.5788\.2\{\\pm\}31\.5^\{7\}84\.5±11\.184\.5\{\\pm\}11\.183±8283\{\\pm\}8279\.9±12\.979\.9\{\\pm\}12\.999\.9±0\.91599\.9\{\\pm\}0\.9^\{15\}98\.4±12\.298\.4\{\\pm\}12\.280\.6±11\.180\.6\{\\pm\}11\.172±3772\{\\pm\}3763\.9±11\.563\.9\{\\pm\}11\.5Contrastive Vec\.99\.5±3\.71599\.5\{\\pm\}3\.7^\{15\}71\.7±43\.871\.7\{\\pm\}43\.815±815\{\\pm\}879\.9±17\.579\.9\{\\pm\}17\.592\.7±23\.11392\.7\{\\pm\}23\.1^\{13\}89\.1±9\.789\.1\{\\pm\}9\.741±6141\{\\pm\}6170\.3±24\.770\.3\{\\pm\}24\.753\.6±44\.81453\.6\{\\pm\}44\.8^\{14\}93\.0±6\.593\.0\{\\pm\}6\.550±5950\{\\pm\}5978\.1±22\.578\.1\{\\pm\}22\.594\.9±17\.91594\.9\{\\pm\}17\.9^\{15\}52\.5±44\.652\.5\{\\pm\}44\.689\.8±8\.989\.8\{\\pm\}8\.951±13451\{\\pm\}13476\.6±21\.076\.6\{\\pm\}21\.0Probe81\.9±35\.51481\.9\{\\pm\}35\.5^\{14\}49\.5±47\.049\.5\{\\pm\}47\.015±915\{\\pm\}988\.1±12\.888\.1\{\\pm\}12\.869\.7±42\.41269\.7\{\\pm\}42\.4^\{12\}93\.5±9\.593\.5\{\\pm\}9\.551±5851\{\\pm\}5889\.2±17\.589\.2\{\\pm\}17\.569\.6±41\.51469\.6\{\\pm\}41\.5^\{14\}94\.1±7\.494\.1\{\\pm\}7\.456±4656\{\\pm\}4688\.4±17\.988\.4\{\\pm\}17\.978\.7±38\.01578\.7\{\\pm\}38\.0^\{15\}48\.2±43\.048\.2\{\\pm\}43\.092\.3±12\.992\.3\{\\pm\}12\.936±2336\{\\pm\}2388\.7±15\.988\.7\{\\pm\}15\.9PCA67\.9±44\.91367\.9\{\\pm\}44\.9^\{13\}29\.4±42\.429\.4\{\\pm\}42\.413±913\{\\pm\}979\.4±17\.679\.4\{\\pm\}17\.654\.0±47\.8454\.0\{\\pm\}47\.8^\{4\}27\.1±21\.227\.1\{\\pm\}21\.215±1015\{\\pm\}1091\.1±10\.091\.1\{\\pm\}10\.019\.0±34\.41219\.0\{\\pm\}34\.4^\{12\}29\.5±23\.329\.5\{\\pm\}23\.314±1014\{\\pm\}1085\.2±15\.285\.2\{\\pm\}15\.254\.9±47\.8454\.9\{\\pm\}47\.8^\{4\}13\.2±28\.013\.2\{\\pm\}28\.028\.9±23\.028\.9\{\\pm\}23\.014±814\{\\pm\}889\.7±10\.889\.7\{\\pm\}10\.8Prompt80\.3±37\.080\.3\{\\pm\}37\.063\.2±47\.063\.2\{\\pm\}47\.012±612\{\\pm\}680\.2±19\.480\.2\{\\pm\}19\.459\.6±46\.159\.6\{\\pm\}46\.122\.7±23\.222\.7\{\\pm\}23\.212±712\{\\pm\}773\.8±23\.073\.8\{\\pm\}23\.062\.2±46\.562\.2\{\\pm\}46\.524\.0±23\.224\.0\{\\pm\}23\.215±615\{\\pm\}686\.7±14\.686\.7\{\\pm\}14\.676\.2±40\.276\.2\{\\pm\}40\.270\.6±44\.370\.6\{\\pm\}44\.326\.7±25\.026\.7\{\\pm\}25\.014±914\{\\pm\}978\.6±20\.878\.6\{\\pm\}20\.8

## Appendix FSteering Analysis and Ablations

### F\.1Steering Trade\-off Curves

Figures[10](https://arxiv.org/html/2605.10971#A6.F10)–[13](https://arxiv.org/html/2605.10971#A6.F13)present the full steering trade\-off curves for all four models with all methods\. Each figure shows target confidence \(top\) and quality metric \(bottom\) across theα\\alphasweep for all 7 attribute conditions\. The main paper \(Figure[4](https://arxiv.org/html/2605.10971#S5.F4)\) shows SAE methods only on MDLM\.

![Refer to caption](https://arxiv.org/html/2605.10971v1/x15.png)Figure 10:MDLM\-124M steering trade\-off curves \(all methods\)\. Top: target confidence vs\.α\\alpha\. Bottom: GPT\-2 perplexity vs\.α\\alpha\. MDLM shows the clearest Adaptive–Uniform separation: Adaptive \(orange\) matches or exceeds Uniform \(blue\) in steering while maintaining substantially lower PPL\.![Refer to caption](https://arxiv.org/html/2605.10971v1/x16.png)Figure 11:SEDD\-124M steering trade\-off curves \(all methods\)\. Top: target confidence vs\.α\\alpha\. Bottom: GPT\-2 perplexity vs\.α\\alpha\. Contrastive Vectors \(green\) matches SAE methods on sentiment and topic, consistent with SEDD’s distributed feature geometry \(§[3\.3](https://arxiv.org/html/2605.10971#S3.SS3)\)\.![Refer to caption](https://arxiv.org/html/2605.10971v1/x17.png)Figure 12:LLaDA\-8B steering trade\-off curves \(all methods\)\. Top: target confidence vs\.α\\alpha\. Bottom: distinct bigram ratio \(dist\-2\) vs\.α\\alpha; we report dist\-2 rather than PPL because LLaDA tends to generate repetitive text, making low PPL misleading\. SAE methods consistently outperform baselines, with the gap largest for topic and multi\-attribute conditions\.![Refer to caption](https://arxiv.org/html/2605.10971v1/x18.png)Figure 13:DREAM\-7B steering trade\-off curves \(all methods\)\. Top: target confidence vs\.α\\alpha\. Bottom: GPT\-2 perplexity vs\.α\\alpha\. DREAM achieves near\-perfect steering at moderateα\\alphabut exhibits the sharpest quality degradation, consistent with its concentrated late\-block emergence\.
### F\.2Feature Count Ablation

We steer sentiment toward positive usingN∈\{1,2,5,10,20,50\}N\\in\\\{1,2,5,10,20,50\\\}features per direction with Uniform steering atα=5\\alpha\{=\}5and report the positive\-direction classifier confidences for all three attributes\. Table[27](https://arxiv.org/html/2605.10971#A6.T27)shows that increasingNNraises target sentiment confidence but also increases interference on non\-target attributes \(topic and style\)\. AtN=20N\{=\}20, all models achieve strong sentiment control \(71–84%\) with modest interference; atN=50N\{=\}50, SEDD formality shifts from 57\.7% baseline to 94\.5% and MDLM formality rises to 67%\. We therefore useN=20N\{=\}20features throughout\.

Table 27:Feature count ablation: positive\-direction sentiment steering atα=5\\alpha\{=\}5, varying the number of featuresNN\. Pos = positive sentiment confidence \(%\), Spt = sports topic confidence \(%\), Frm = formality confidence \(%\)\.MDLM\-124MSEDD\-124MDREAM\-7BLLaDA\-8BNNPosSptFrmPosSptFrmPosSptFrmPosSptFrmNo Steering25\.518\.653\.329\.020\.357\.757\.710\.573\.569\.113\.977\.8141\.520\.051\.029\.023\.061\.569\.57\.582\.573\.57\.081\.5250\.523\.550\.037\.526\.066\.570\.08\.082\.072\.59\.080\.0565\.022\.046\.053\.524\.577\.578\.06\.585\.075\.08\.083\.51075\.026\.551\.565\.020\.580\.080\.55\.090\.579\.07\.083\.52084\.025\.055\.571\.520\.090\.082\.54\.092\.580\.03\.582\.05092\.018\.067\.088\.517\.594\.589\.57\.591\.578\.06\.084\.5

### F\.3Cross\-Attribute Interference

For each method at its Table[3](https://arxiv.org/html/2605.10971#S5.T3)operating point, we measure cross\-attribute interference as the mean absolute percentage\-point deviation of non\-target attribute classifier confidences from their unsteered baselines\. Givenmmattributes steered simultaneously, the interference for methodMMis1m​∑a∉target\|ca\(M\)−ca\(unsteered\)\|\\frac\{1\}\{m\}\\sum\_\{a\\notin\\text\{target\}\}\|c\_\{a\}^\{\(M\)\}\-c\_\{a\}^\{\(\\text\{unsteered\}\)\}\|, wherecac\_\{a\}is the classifier confidence for attributeaa\. Figure[14](https://arxiv.org/html/2605.10971#A6.F14)reports this metric across all models and steering conditions\.

![Refer to caption](https://arxiv.org/html/2605.10971v1/x19.png)Figure 14:Cross\-attribute interference at the Table[3](https://arxiv.org/html/2605.10971#S5.T3)operating points: mean absolute pp deviation of non\-target attributes from unsteered baselines\. Adaptive \(orange\) consistently produces less interference than Uniform \(blue\), with the largest gap on MDLM\.
### F\.4Latency Overhead

Table[28](https://arxiv.org/html/2605.10971#A6.T28)reports per\-sample wall\-clock times for unsteered generation, SAE steering \(uniform schedule,α=5\\alpha\{=\}5\), and contrastive\-vector steering, measured over 20 single\-sample generations per method after a warm\-up pass\.

Table 28:Generation latency \(seconds per sample, batch size 1\)\. SAE overhead is the ratio of SAE steering time to baseline\. CV = contrastive vector\.ModelStepsBaseline \(s\)SAE \(s\)SAE OverheadCV \(s\)CV OverheadMDLM \(124M\)102425\.6630\.071\.17×\\times25\.731\.00×\\timesSEDD \(124M\)102421\.4922\.551\.05×\\times22\.151\.03×\\timesDREAM \(7B\)2567\.528\.411\.12×\\times7\.531\.00×\\timesLLaDA \(8B\)640\.831\.251\.51×\\times0\.810\.97×\\timesSAE steering adds 5–51% overhead depending on model size and step count\. The overhead is lowest on SEDD \(1\.05×\\times\), where the 1024\-step analytic predictor dominates runtime and the SAE encode–decode \(768→12,288→768768\\to 12\{,\}288\\to 768\) is negligible by comparison\. MDLM \(1\.17×\\times\) has a simpler per\-step update, so the SAE cost is proportionally larger\. For the 7B\-scale models, DREAM \(1\.12×\\times, 256 steps\) amortizes the SAE cost across expensive transformer forward passes, while LLaDA \(1\.51×\\times, 64 steps\) has the highest overhead because each of its few steps is fast \(∼\{\\sim\}13ms\), making the per\-step SAE cost \(4096→16,384→40964096\\to 16\{,\}384\\to 4096across 4 layers\) a larger fraction\. Contrastive\-vector steering adds negligible overhead \(∼\{\\sim\}1\.00×\\times\) since it requires only a single vector addition per layer\. MDLM was benchmarked on an A10G GPU; all other models on A100\.

### F\.5Qualitative Steering Examples

Tables[29](https://arxiv.org/html/2605.10971#A6.T29)–[32](https://arxiv.org/html/2605.10971#A6.T32)show representative generations under triple\-attribute steering \(S\+T\+St\) for each model, all four SAE steering modes, and baselines \(Contrastive Vectors and Prompt where applicable\)\. MDLM and SEDD are steered toward positive sentiment, sports topic, and formality; LLaDA and DREAM toward positive sentiment, sports topic, and informality\. Texts are truncated; classifier confidences are from the off\-the\-shelf classifiers described in §[C\.9](https://arxiv.org/html/2605.10971#A3.SS9)\.

Table 29:MDLM qualitative examples \(triple\-attribute: positive, sports, formal\)\. S/T = sentiment/topic confidence \(%\); Frm = formality confidence \(%\)\.MethodSTFrmPPLGenerated Text \(truncated\)Unsteered106243Its immigration reform bill, making reference again to immigration law Tuesday night… The disagreements declared on cases on which judges are working are “a case of man’s versus a judge, the issues that arise that have to be heard by one court\.”Uniform10010047259with also the former rivals side and the stars for the replacement and they was make a breakthrough at the end of 2011, moving to the series win which capped a five\-match Premier career…Adaptive1001005054“I’m really happy, I’m thinking and looking forward to Southampton,” Thomas this week told Sky Sport\. “I am always hungry to go home\.” The Shearsborough stadium won’t have any “New England” colours against Coventry…Uniform\+E10010051101went back into form, making the Crystal Palace bench on Tuesday night and his hopes of winning the game on Sunday and defeat Manchester United\. The former first\-time winner and two\-time ace culminated on the pre\-match session…Adaptive\+E991004051went deeper into future, making it again to take news Tuesday night when his team will win the AFC championship Sunday at Dick Bowl in Spokane\. “I’m going there at 1 p\.m\. tomorrow to try to see what happens,” Bowl said\.Contrastive Vec\.100037329with also the fan t; and feed through for the fans and you with the fan letters to some of her characters, and then they started doing me and fronted and played us those episodes…

Table 30:SEDD qualitative examples \(triple\-attribute: positive, sports, formal\)\. S/T = sentiment/topic confidence \(%\); Frm = formality confidence \(%\)\.MethodSTFrmPPLGenerated Text \(truncated\)Unsteered8003029everybody, that’s what happens if you know people are really getting the pressure there is you look at no one else they’re busy with that\. I’m like things keep on happening every day is that it is thrusting into my life…Uniform9910067312complete the third and 2nd all Mexico and scored a one goal that had the European League victories final of ten and scored the final that turned for victory of World title against Brazil USA’s Ners Joao…Adaptive1001006149Two games hard\-won, Wade again in the fourth quarter and scored again with one tip of Jimmy Wingard’s fade\-away drive to draw a one minute tie with the Pacers\. He scored twice…Uniform\+E10010074135great nation is case of it all, although there is one of the most sophisticated players of the sport, Diego Brasil, who was restored for a new level to have with a strong analytical background…Adaptive\+E9910077102of Jordi has emerged as the next president in each club and country, the extraordinary result is that Spain has re\-established itself… In an elegant football football held to the centre of the Catalan club victory…Contrastive Vec\.1001008536in the bottom of the 10th inning was the night when Babe Ruth tied the National League Division scoreless with a three\-run solo third, the first time his career led, but fell foul\. The Pirates were 2 ahead of the San Francisco Giants…

Table 31:LLaDA qualitative examples \(triple\-attribute: positive, sports, informal\)\. S/T = sentiment/topic confidence \(%\); Inf = informality confidence \(%\)\.MethodSTInfPPLGenerated Text \(truncated\)Unsteered250126a great question\. I have been thinking about it for a while and I am trying to come up with a solution that would work for me\. I have a lot of money and I want to invest it in real estate\.Uniform1100936what you mean by “after the game” lol but not at the end of the game but not at the end of the game but not at the end of the game…Adaptive10083705a great idea for the game\. I think it would be a great way to make the game more fun and competitive by giving the players a big advantage at the start of the game\.Uniform\+E941007211a great way to get a kick out of the game\. you can get the ball at the start of the game and then after the game you can get it back and throw it in the end zone\. i was a hater in the past\.Adaptive\+E10099753a great way to get to know the game\. I think this is a great way to make friends\. I think this is a great way to get to know the game\.Contrastive Vec\.97100208a good question\. It’s always difficult to predict the outcome of a match, but it’s also difficult to predict the outcome of a game\. I think the important thing is to be prepared for the match…Prompt100209a good book for people who are interested in sports\. It has a lot of information about different sports and how to play them\. I like the way it is written because it is easy to understand…

Table 32:DREAM qualitative examples \(triple\-attribute: positive, sports, informal\)\. S/T = sentiment/topic confidence \(%\); Inf = informality confidence \(%\)\.MethodSTInfPPLGenerated Text \(truncated\)Unsteered00114a good way for beginners to understand that if you have an array that is large enough, you can treat it as a queue\. However, when you use an array as a queue, it is easy to reach the capacity of the queue…Uniform10010077144a match in 3 final the all match good there time time and they was great team up all good lost and win a match of the game and go won a win of the game they 2 game…Adaptive1001007637a great match\. Its a great match i love it and the match was 20\-20 and when the match started was 2\-3 they played alot of people thought they could be in the back of the match go 4\-5…Uniform\+E1001008722a good game, but it needs to be more fun and not so boring it is more of a best ball game\. 1st game ever\. this game was really fun\. playa 11\. this game was so great…Adaptive\+E1001008421a good game, but it needs to be more fun and not so boring it is more of a basket ball game\. 1st game ever\. this game was good but i liked the first game better\. This game was awesome i liked it it was fun and i like it even though it was hard\. Great gameContrastive Vec\.10016728a great way to get kids to think about how many days are in a week\! I love the whole concept of how many days in a week and then i can ask them how many days in a week…Prompt100996511a way to do that\. It’s kind of a good example of how to do it\. It’s not a good example of how to do it, but I think it’s a great example of how to do it…

On MDLM and SEDD, Uniform steering achieves high target confidence but produces garbled text \(PPL\>\>250\), while Adaptive matches confidence at 5–6×\\timeslower perplexity with coherent output\. E\-ratio calibration \(Uniform\+E, Adaptive\+E\) provides intermediate quality\. On LLaDA and DREAM, all SAE modes successfully shift attributes from near\-chance baselines while maintaining low perplexity; E\-ratio variants achieve the highest simultaneous control across all three attributes\. The DREAM unsteered example \(formal, technical text with St==1%\) contrasts sharply with steered outputs that exhibit clear informality markers \(lowercase, casual phrasing, slang\)\.

## Broader Impacts

Mechanistic interpretability is a foundational tool for AI safety: models whose internal computations can be traced and steered are easier to audit, correct, and align\. Our work extends this agenda to discrete diffusion language models, a class that has received less interpretability attention than autoregressive LLMs but is becoming competitive in capability\. The framework supports legitimate downstream uses including content moderation, domain adaptation without retraining, and accessibility\-oriented control of register or sentiment\. The interpretability tools themselves let practitioners diagnose failures and understand where in the denoising trajectory particular attributes commit—information not visible from external behavior alone\.

However, the same steering capability that enables beneficial control also enables manipulation: an adversary with white\-box access could shift outputs toward biased or deceptive content while preserving fluency, in ways simple prompting cannot\. We release no models, datasets, or steering pipelines targeting harmful attributes; trained SAEs and feature sets are intended for research on benign attributes and would need retraining for deployment