一个通道都不落下：通过残差门控理解扩散加速

# No Channels Left Behind: Understanding Diffusion Acceleration through Residual Gates
Source: [https://arxiv.org/html/2610.40914](https://arxiv.org/html/2610.40914)
Jaemyung YuAffiliation:NAVER AI LabJaehui HwangAffiliation:NAVER AI LabJong\-Seok LeeAffiliation:Yonsei UniversityAffiliation:Corresponding authors\.Dongyoon HanAffiliation:NAVER AI LabAffiliation:Corresponding authors\.

###### Abstract

Training acceleration methods for diffusion transformers, such as REPA and dispersive regularization, are commonly explained by additional supervision, spatial\-structure priors, or improved feature geometry\. We revisit this view and uncover a striking internal change: the AdaLN\-Zero residual gateα\\alpha, which directly controls how conditioned updates enter the residual stream, is substantially reorganized by these methods\. This observation motivates our hypothesis that acceleration arises, at least in part, from reshaping the residual gating pathway rather than solely from the feature structures explicitly imposed by auxiliary objectives\. Supporting this view, accelerated models are markedly more sensitive to perturbations ofα\\alpha, and their gate geometry reflects semantic relationships among classes\. We further introduce the normalized participation ratio \(nPR\) to characterize the channel\-wise organization of the gates and find that, across REPA teachers, projector architectures, and radial\-objective targets, broader gate participation accompanies better generation quality\. Guided by this insight, we propose a simple teacher\-free radial objective that promotes broader gate participation through feature\-norm regularization, without pairwise feature comparisons\. On ImageNet256×256256\\times 256with SiT\-B/2, our method improves FID over dispersive loss while increasing mean gate participation over timesteps and classes, further supporting the link between gate organization and training acceleration\. Our results identify residual\-gate reorganization as a measurable axis of diffusion training acceleration and suggest a simple direction for further acceleration\. Our code is available at[https://github\.com/naver\-ai/radialobj](https://github.com/naver-ai/radialobj)

## 1Introduction

Recent advances have substantially accelerated diffusion transformer training by injecting additional auxiliary objectives through representation alignment \(REPA\) with frozen\-teacher supervision\([Yu et al\., 2025](https://arxiv.org/html/2610.40914#bib.bib25)\), spatially structured representation alignment\([Singh et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib22)\), and feature\-geometry regularization\([Wang and He, 2025](https://arxiv.org/html/2610.40914#bib.bib23)\)\. Despite their differences, these approaches are largely understood through the representational structures explicitly imposed by their objectives\. Yet,are these structures themselves responsible for faster training, or do they reshape a more fundamental mechanism within the transformer?

We revisit this question through the conditioning pathway of AdaLN\-Zero\([Peebles and Xie, 2023](https://arxiv.org/html/2610.40914#bib.bib16)\)\. In diffusion transformers such as DiT\([Peebles and Xie, 2023](https://arxiv.org/html/2610.40914#bib.bib16)\)and SiT\([Ma et al\., 2024](https://arxiv.org/html/2610.40914#bib.bib13)\), timestep and class embeddings generate modulation vectors for every transformer block\. Among them, the residual gate𝜶\\bm\{\\alpha\}111We refer to𝜶\\bm\{\\alpha\}as agate, as it modulates the embeddings through channel\-wise multiplication\.plays a particularly direct role: it scales each channel of a branch output before residual addition, thus controlling how strongly each feature channel contributes to the residual stream across all token positions\. This suggests a complementary view of diffusion acceleration: auxiliary objectives may not only reshape the features they supervise, but also reorganize*how those features are injected into the residual stream*\. We therefore conjecture that residual gating may provide a common internal signature of accelerated diffusion training \([Fig\.1](https://arxiv.org/html/2610.40914#S1.F1)\)\.

To this end, we begin with two simple interventions: removing the AdaLN\-Zero modulation pathway during training and perturbing the residual gate itself at test time\. Gating matters even for the baseline \(without acceleration\), but strikingly, accelerated models are far more vulnerable to its disruption\. This contrast suggests that acceleration does more than improve the features being learned: it fundamentally changes how the model uses them through the gating pathway\. Motivated by this observation, we first ask whether residual gates carry interpretable semantic structure\. Using WordNet\([Fellbaum, 1998](https://arxiv.org/html/2610.40914#bib.bib7)\), we find that distances between class\-conditioned gates increase with semantic distance between classes, indicating that the gating pathway reflects the semantic organization of the generated categories\. Next, we quantify gate participation using thenormalized participation ratio\(nPR\), a scale\-invariant measure of how broadly gate magnitude is distributed across channels that also guides our subsequent method design\. Higher gate participation, averaged over timesteps and classes, tracks better generation quality across REPA teachers and, for a fixed teacher, across projector designs, identifying gate participation as a correlate of generation quality within each sweep\.

Based on gate participation, we introduce the Radial objective, a simple teacher\-free objective that acts only on hidden\-feature norms\. Unlike representation alignment, it requires no pretrained teacher guidance; unlike dispersive regularization, it requires no pairwise feature comparisons\. Instead, the Radial objective penalizes deviations of each sample’s flattened feature norm from a target\. Despite acting only on feature norms, this objective increases residual\-gate participation and improves generation quality\. As a result, our method improves both Fréchet inception distance\([Heusel et al\., 2017](https://arxiv.org/html/2610.40914#bib.bib8), FID;\)and mean gate participation, supporting residual\-gate participation as a candidate mechanism underlying diffusion training acceleration\. Quantitatively, on ImageNet256×256256\\times 256\([Deng et al\., 2009](https://arxiv.org/html/2610.40914#bib.bib3)\)with SiT\-B/2 trained for 400K steps, the Radial objective achieves an FID of 11\.83, improving over Vanilla222We refer to Vanilla as a baseline SiT\([Ma et al\., 2024](https://arxiv.org/html/2610.40914#bib.bib13)\)model\.\(16\.24\) and Dispersive \(13\.19\), while increasing the mean final\-block MLP gate participation from 0\.434 in Vanilla to 0\.642\.

Overall, our results point beyond the representations explicitly targeted by existing acceleration methods\. These objectives also reorganize the residual gating pathway, whose channel participation tracks generation quality in controlled sweeps\. We argue that this perspective identifies residual\-gate organization as a previously overlooked axis of diffusion acceleration and provides a simple route to promote it without either a representation teacher or pairwise feature comparisons\.

![Refer to caption](https://arxiv.org/html/2610.40914v1/teaser.png)Figure 1:Overview: from residual gate insights to improved diffusion training\.\(Left\)AdaLN\-Zero gates scale residual\-branch outputs channel\-wise, with each gate shared across token positions\.\(Center\)Our proposednormalized participation ratio \(nPR\)measures how evenly squared gate magnitudes are distributed across channels\. Across REPA teachers, higher mean gate participation is associated with lower FID\.\(Right\)Our proposedRadial objectivepenalizes per\-sample feature norms without a pretrained teacher or pairwise feature comparisons\.
## 2Background

##### Diffusion transformers

process latent image tokens through transformer blocks containing a multihead self\-attention \(MHSA\) branch and an MLP branch\([Peebles and Xie, 2023](https://arxiv.org/html/2610.40914#bib.bib16)\)\. Each branch adds an update to the residual stream\. We study the AdaLN\-Zero conditioning used in these blocks\.

Let𝑯∈ℝN×D\\bm\{H\}\\in\\mathbb\{R\}^\{N\\times D\}denoteNNtokens withDDchannels\. The conditioning pathway produces the modulation vectors𝜶,𝜷,𝜸∈ℝD\\bm\{\\alpha\},\\bm\{\\beta\},\\bm\{\\gamma\}\\in\\mathbb\{R\}^\{D\}and the output embedding𝑯\+\\bm\{H\}^\{\+\}is sequentially computed as𝑯\+=𝑯\+𝜶⊙𝑼\\bm\{H\}^\{\+\}=\\bm\{H\}\+\\bm\{\\alpha\}\\odot\\bm\{U\}, where𝑼\\bm\{U\}can be eitherMHSA​\(𝑯~\)\\texttt\{MHSA\}\(\\widetilde\{\\bm\{H\}\}\)orMLP​\(𝑯~\)\\texttt\{MLP\}\(\\widetilde\{\\bm\{H\}\}\), and𝑯~=\(𝟏\+𝜸\)⊙LayerNorm⁡\(𝑯\)\+𝜷\\widetilde\{\\bm\{H\}\}=\(\\bm\{1\}\+\\bm\{\\gamma\}\)\\odot\\operatorname\{\\texttt\{LayerNorm\}\}\(\\bm\{H\}\)\+\\bm\{\\beta\}\. Here,𝜸\\bm\{\\gamma\}and𝜷\\bm\{\\beta\}specify the scale and shift of the normalized branch input;𝜶\\bm\{\\alpha\}scales the branch output before residual addition\. Accordingly,𝜶\\bm\{\\alpha\}may be interpreted as a residual gating vector\. The Hadamard product⊙\\odotacts channel\-wise, with each vector broadcast across tokens; MHSA and MLP branches have separate modulation parameters in each block\. Note that𝜶\\bm\{\\alpha\}is structurally distinct: Whereas𝜷\\bm\{\\beta\}and𝜸\\bm\{\\gamma\}modulate the branch input and are mediated by the subsequent projection layers in MHSA or MLP,𝜶\\bm\{\\alpha\}directly gates the branch output before residual addition\. We thus conjecture that training acceleration is governed primarily by𝜶\\bm\{\\alpha\}\.

We refer to𝜶\\bm\{\\alpha\}as thechannel\-wise residual gate:αd\\alpha\_\{d\}multiplies channelddat every token position\. Its entries are signed scaling coefficients\. The gate is obtained directly from the conditioning pathway, without an image forward pass\. Because𝜶\\bm\{\\alpha\}directly scales the residual branch output, it lets us examine how conditioning controls each channel’s contribution to the residual stream\. We first test how generation quality responds to changes in the gate’s channel\-wise magnitude distribution, then examine the structure of gates learned under accelerated training\. Prior work has examined AdaLN initialization and parameter updates\([Zhu et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib26)\)and the semantic structure of conditioning embeddings\([Pham et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib17)\)\. How the residual gate values organize channel contributions under accelerated training remains underexplored\.

Diffusion transformer acceleration and its mechanics\.Recent auxiliary losses to accelerate diffusion transformer training fall into two broad categories: teacher\-guided \(REPA\([Yu et al\., 2025](https://arxiv.org/html/2610.40914#bib.bib25)\)and iREPA\([Singh et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib22)\)\) and teacher\-free \(dispersive loss\([Wang and He, 2025](https://arxiv.org/html/2610.40914#bib.bib23)\)\)\. REPA and iREPA accelerate training by aligning diffusion features with representations from a frozen vision foundation model through a learnable projector\. Dispersive loss is a regularizer that encourages larger pairwiseℓ2\\ell\_\{2\}distances between the hidden features of samples within a batch\.

## 3Understanding Acceleration in Diffusion Transformers

### 3\.1Motivation

What lies beyond existing representation\-level supervision?Existing explanations for diffusion acceleration largely focus on the representations explicitly targeted by the auxiliary objective: REPA guides hidden features toward pretrained visual representations\([Yu et al\., 2025](https://arxiv.org/html/2610.40914#bib.bib25)\), while dispersive objectives reshape their geometry\([Wang and He, 2025](https://arxiv.org/html/2610.40914#bib.bib23)\)\. We conjecture that these feature\-level effects are not the full story\. Instead, acceleration may critically depend on how such supervision reorganizes the AdaLN\-Zero gating pathway \(beyond prior initialization analyses\([Zhu et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib26)\)\), which directly controls the contribution of conditioned residual updates throughout the network\.

Pilot study 1: How does gating affect generation quality during training?We perform a pilot study by training models with in\-context conditioning\([Peebles and Xie, 2023](https://arxiv.org/html/2610.40914#bib.bib16)\)in place of AdaLN\-Zero, removing its conditional modulation pathway, including the residual gate\. The FID improvements from both Dispersive and REPA become substantially smaller under in\-context conditioning \([Table1](https://arxiv.org/html/2610.40914#S3.T1)\)\. Relative to Vanilla, REPA’s FID change shifts from−18\.24\-18\.24to−13\.48\-13\.48, while Dispersive’s shifts from−4\.98\-4\.98to\+1\.15\+1\.15\. This comparison supports a role for the AdaLN\-Zero modulation pathway in the gains from auxiliary objectives; Pilot 2 isolates the residual gate\.

Table 1:FID changes under different conditioning architectures\. Replacing AdaLN\-Zero with in\-context conditioning \(i\.e\., removing the AdaLN\-Zero modulation, including residual gates\) substantially weakens the acceleration gains\. All models are SiT\-B/2 trained for 100K steps; FID \(↓\\downarrow\) follows[SectionA\.3](https://arxiv.org/html/2610.40914#A1.SS3), andΔ\\Deltais the change from Vanilla with the same conditioning\.Pilot study 2: Effect of gating at test time\.After training, we perturb the gate values of frozen Vanilla, Dispersive, and REPA models during sampling to measure their sensitivity to changes in the learned gates\. This intervention evaluates sensitivity without allowing the model weights to adapt to the modified gates\. We use SiT\-XL/2 checkpoints trained for 400K steps\. At each sampling step, we redistribute the squared magnitudes of the MLP gate at the eighth layer \(L8\), where the REPA objective is applied, making the distribution either more uniform or more concentrated while preserving the gate’s signs andℓ2\\ell\_\{2\}norm\. We match the relative perturbation magnitudeδ=‖𝜶new−𝜶‖2/‖𝜶‖2\\delta=\\\|\\bm\{\\alpha\}^\{\\textrm\{new\}\}\-\\bm\{\\alpha\}\\\|\_\{2\}/\\\|\\bm\{\\alpha\}\\\|\_\{2\}across these directions and models\.

Figure 2:Does gate redistribution in Vanilla, Dispersive, and REPA affect FID?The center \(i\.e\.,x=0x=0\) is the unmodified baseline; moving left makes gate magnitudes more uniform, and moving right makes their distribution more concentrated\. Tick labels give relative gate perturbation magnitudeδ\\delta\. All three panels show FID change from each model’s own baseline\.At a matched perturbation magnitude ofδ=0\.30\\delta=0\.30, Dispersive shows larger absolute FID changes than Vanilla, while REPA responds much more strongly \([Fig\.2](https://arxiv.org/html/2610.40914#S3.F2)\)\. Both studies show that generation quality is sensitive to𝜶\\bm\{\\alpha\}, most strongly in REPA \(Δ​FID\\Delta\\text\{FID\}up to\+5\.356\+5\.356atδ=0\.30\\delta=0\.30\)\. With frozen weights, this test measures sensitivity to departures from the learned gates, not a preference for uniform gates;[Section3\.2\.2](https://arxiv.org/html/2610.40914#S3.SS2.SSS2)compares gates learned under different training\. Since REPA/iREPA is primarily understood through representation alignment, this raises a question:Could the gating pathway be an underlying piece of the acceleration?

Figure 3:Illustration of a WordNet hierarchy and path distance\.The highlighted shortest path from dog to tiger spans five hypernym edges\.Figure 4:Change in residual gates versus WordNet distance\. We compare how the gates change in Vanilla, REPA, and Dispersive models att=0\.5t=0\.5\.\(Left\)Euclidean distance between gate magnitudes𝒂i=\|𝜶i\|\\bm\{a\}\_\{i\}=\|\\bm\{\\alpha\}\_\{i\}\|, divided by its bin mean at WordNet distance 2\.\(Right\)Cosine distance after centering𝒂i\\bm\{a\}\_\{i\}across classes, shifted by its bin mean at WordNet distance 2 to start at zero\.

### 3\.2Empirical Insights with Residual Gating

Here, we investigate the role of the residual gate in accelerated diffusion\-transformer training\. Our pilot study revealed that generation quality in existing training acceleration methods is unexpectedly sensitive to the residual gate𝜶\\bm\{\\alpha\}, which suggests that the gating pathway may play a more central role than previously appreciated\. This leads us to revisit REPA’s acceleration from a different perspective\. We study the structure of the learned gates to explore this possibility\.

#### 3\.2\.1How Do Residual Gates Behave in Training Acceleration Methods?

Residual gates reflect semantic distance\.We examine how class\-conditioned gate distances respond to semantic distance, and whether this response differs across training objectives\. The semantic distance is measured as WordNet distance\([Fellbaum, 1998](https://arxiv.org/html/2610.40914#bib.bib7)\), which is the number of edges on a shortest path between class synsets\. The partial WordNet noun hierarchy in[Fig\.4](https://arxiv.org/html/2610.40914#S3.F4)highlights the five\-edge path from dog to tiger\. We compare the class\-conditioned MLP gates of Vanilla, REPA, and Dispersive; experimental settings are provided in[SectionB\.3](https://arxiv.org/html/2610.40914#A2.SS3)\. We uset=0\.5t=0\.5as a representative midpoint of the diffusion trajectory;[SectionB\.4](https://arxiv.org/html/2610.40914#A2.SS4)reports the same analysis att=0\.25t=0\.25and0\.750\.75\. We compute distances for all 499,500 unordered class pairs and average them at each WordNet distance\. Every curve attains its minimum bin mean at WordNet distance 2\. Because overall gate scale differs across models \([Fig\.5](https://arxiv.org/html/2610.40914#S3.F5)\), we divide each Euclidean curve by this minimum; for the scale\-invariant centered cosine distance, we subtract it\.

Figure 5:Histograms of signed learned MLP gate entriesαd\\alpha\_\{d\}for Vanilla, REPA, and Dispersive, using the same models as[Fig\.4](https://arxiv.org/html/2610.40914#S3.F4)\. Att=0\.5t=0\.5, we pool all 1,000 classes and 1,152 channels without normalization, using common width\-0\.025 bins, lightly smoothed with a Gaussian kernel \(σ=1\\sigma=1bin\)\. All three panels show\[−2,2\]\[\-2,2\]with independently scaled linear y\-axes\. Bin percentages use all entries, including those outside the displayed range\.Gate distances generally increase with semantic distance in all three models, with the largest increase under REPA \([Fig\.4](https://arxiv.org/html/2610.40914#S3.F4)\)\. At WordNet distance 22, Euclidean distances reach1\.678×1\.678\\timestheir starting value for REPA, compared with1\.217×1\.217\\timesfor Vanilla and1\.428×1\.428\\timesfor Dispersive; centered cosine distances increase by0\.5740\.574,0\.2250\.225, and0\.4410\.441, respectively\.

The hierarchy analysis shows that class\-conditioned gates reflect semantic hierarchy, as the distance between gate vectors increases with semantic distance between classes\. Since a larger distance between two gate vectors indicates greater separation in gate space, this trend suggests that REPA induces the strongest semantic separation among class\-conditioned gates\. Together, these observations motivate quantifying the channel\-wise magnitude distributions of these gates and examining their relationship with generation quality across training configurations\.

#### 3\.2\.2Defining Gate Participation and Its Impact

Defining gate participation metric\.We quantify gate participation with the participation ratio \(PR\), which measures how evenly the squared gate magnitudes are distributed across channels\. For a nonzero gate vector with squared magnitudessd=αd2s\_\{d\}=\\alpha\_\{d\}^\{2\}, the PR and its normalized form \(nPR\)333[Pham et al\. \(2026\)](https://arxiv.org/html/2610.40914#bib.bib17)also employ nPR to analyze the semantic structure of conditioning embeddings\. We instead measure it on the residual gates and compare it across training objectives and against generation quality\.are defined as follows:

PR⁡\(𝜶\)=\(∑d=1Dsd\)2∑d=1Dsd2,nPR⁡\(𝜶\)=PR⁡\(𝜶\)D,𝜶∈ℝD\.\\mathrm\{PR\}\(\\bm\{\\alpha\}\)=\\frac\{\\left\(\\sum\_\{d=1\}^\{D\}s\_\{d\}\\right\)^\{2\}\}\{\\sum\_\{d=1\}^\{D\}s\_\{d\}^\{2\}\},\\quad\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)=\\frac\{\\mathrm\{PR\}\(\\bm\{\\alpha\}\)\}\{D\},\\quad\\bm\{\\alpha\}\\in\\mathbb\{R\}^\{D\}\.\(1\)We refer tonPR⁡\(𝜶\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)as gate participation\. HighernPR⁡\(𝜶\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)indicates that squared gate magnitude is spread more evenly across channels; lowernPR⁡\(𝜶\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)indicates concentration in fewer channels\. Equal nonzero magnitudes onmmof theDDchannels givenPR⁡\(𝜶\)=m/D\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)=m/D, with a range of\[1/D,1\]\[1/D,1\]for nonzero gates\. The statistic is invariant to uniform nonzero rescaling\. It measures the allocation of gate magnitude across channels, rather than the overall gate scale of the residual update\.

We then inspect the learned MLP gate values in Vanilla, REPA, and Dispersive\.[Fig\.5](https://arxiv.org/html/2610.40914#S3.F5)pools the signed gate entries across all 1,000 classes att=0\.5t=0\.5, without rescaling them\. Their overall scales differ, with standard deviations of 0\.049 for Vanilla, 0\.285 for Dispersive, and 0\.793 for REPA\.

Gate participation correlates with generation quality\.We next examine whether mean gate participation over timesteps and classes is actually correlated with generation quality\. We vary the representation\-alignment target to examine the relationship across training configurations\. A trainable projector maps the diffusion features to the teacher representation, with alignment applied at corresponding tokens\. We conduct class\-conditional ImageNet256×256256\\times 256generation and report FID\. Throughout, we measure gate participation at the block where the auxiliary objective is applied: L8 for SiT\-XL/2 REPA, L4 for SiT\-B/2 REPA, and L12 for Dispersive and the Radial objective, with Vanilla references measured at the same block\. We measure MLP gates at L8 in the released REPA checkpoints\([Yu et al\., 2025](https://arxiv.org/html/2610.40914#bib.bib25)\)corresponding to eight teacher encoders\. For each checkpoint, we computenPR⁡\(𝜶⁡\(t,y\)\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\(t,y\)\)separately for 1,000 classes and 101 uniformly spaced timesteps, then average the 101,000 values\. Across eight REPA teachers, mean MLP gate participation at L8 is strongly negatively correlated with FID \(r=−0\.962r=\-0\.962;[Fig\.6\(a\)](https://arxiv.org/html/2610.40914#S3.F6.sf1)\)\. The teacher sweep reveals an association between gate participation and generation quality across alignment targets\.

\(a\)Teacher sweep\(b\)Projector sweep\(c\)Radial objective
Figure 6:Gate participation and generation quality across teacher, projector, and radial\-target sweeps\.\(a\)REPA with eight teachers\.\(b\)Projector architectures under a fixed teacher\.\(c\)Radial\-target sweep at 100K steps, with Vanilla and Dispersive as reference points\. All three horizontal axes shownPR⁡\(𝜶\)¯\\overline\{\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)\}over 101 uniformly spaced model timesteps in\[0,1\]\[0,1\]and 1,000 classes\. Fits and correlations use the sweep points only, excluding the reference points in\(c\)\.Gate participation can track quality with a fixed teacher\.We now verify that the association between mean gate participation and generation quality also holds when the teacher is fixed and the projector changes; fixing the teacher removes teacher quality as a common cause of both gate participation and FID\. iREPA\([Singh et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib22)\)replaces REPA’s three\-layer MLP projector with a3×33\\times 3convolution, providing a precedent for varying the projection head\.[Table2](https://arxiv.org/html/2610.40914#S3.T2)compares four projector architectures under fixed teacher, target preprocessing, alignment weight, and training budget\.

Table 2:Projector components and results\.We use SiT\-B/2 trained for 50K steps\. Check marks and crosses indicate present and absent components;𝑾i\\bm\{W\}\_\{i\}denotes affine slots\.nPR⁡\(𝜶\)¯\\overline\{\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)\}is computed for MLP gates at the fourth layer \(L4\) over 101 uniformly spaced model timesteps in\[0,1\]\[0,1\]and 1,000 classes\. All four use the same teacher targets and alignment settings \([SectionB\.6](https://arxiv.org/html/2610.40914#A2.SS6)\)\.Reducing the MLP from three affine layers to two and then to one increases mean gate participation from0\.3730\.373to0\.4610\.461and0\.5920\.592, while lowering FID from58\.5358\.53to56\.4456\.44and51\.3351\.33\. Removing the nonlinearities while retaining three affine layers gives FID52\.8752\.87\. Across the four variants, L4 MLP gate participation averaged over timesteps and classes correlates with FID atr=−0\.991r=\-0\.991\([Fig\.6\(b\)](https://arxiv.org/html/2610.40914#S3.F6.sf2)\)\. Together with the teacher sweep, these results show that the participation\-quality association appears both when the alignment target changes and when the projector changes under a fixed teacher\. This raises the question of whether encouraging higher gate participation improves generation quality\.

## 4Increasing Gate Participation for Better Generation Quality

This section studies whether encouraging higher gate participation during training improves generation quality\. We introduce the Radial objective, which acts on hidden\-feature norms, and examine when its local gradient contribution increases gate participation\.

The radial objective\.Based on our empirical studies above, we propose a simple training acceleration method, coined Radial objective, which adds a per\-sample feature\-norm penalty to the velocity prediction loss\([Ma et al\., 2024](https://arxiv.org/html/2610.40914#bib.bib13)\)\. Our insight is that the embedding right before the residual addition directly interacts with the residual gate𝜶\\bm\{\\alpha\}, which would likely increase gate participation \(i\.e\.,nPR⁡\(𝜶\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)\) and ultimately accelerate training\. Given the output𝒁i∈ℝN×D\\bm\{Z\}\_\{i\}\\in\\mathbb\{R\}^\{N\\times D\}of the final block, define𝒛i=flatten⁡\(𝒁i\)\\bm\{z\}\_\{i\}=\\operatorname\{flatten\}\(\\bm\{Z\}\_\{i\}\)and optimize the following objective:

ℒ=ℒvelocity\+λ​𝔼i​\[\(‖𝒛i‖2−c\)2\],\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{velocity\}\}\+\\lambda\\,\\mathbb\{E\}\_\{i\}\\big\[\(\\\|\\bm\{z\}\_\{i\}\\\|\_\{2\}\-c\)^\{2\}\\big\],\(2\)where the norm is over all tokens and channels of one sample, andccdenotes a flattened feature\-norm target;λ\>0\\lambda\>0weights the auxiliary penalty\. We sweep the radial target overc≈k​N​Dc\\approx k\\sqrt\{ND\}withk∈\{2,4,8,10,14\}k\\in\\\{2,4,8,10,14\\\}and report the results for each target;[SectionC\.1](https://arxiv.org/html/2610.40914#A3.SS1)gives the exact values\. We use the target with the lowest observed FID \(k=10k=10\) for the performance comparison\. This objective requires neither a teacher nor distances between different samples\.

Promoting gate participation with our radial objective\.At the current model state, fix a condition\(t,y\)\(t,y\)and analyze its gate𝜶=𝜶θ​\(t,y\)\\bm\{\\alpha\}=\\bm\{\\alpha\}\_\{\\theta\}\(t,y\)\. For an input at this condition, the final\-block output is𝒁=𝑯\+𝜶⊙𝑼=𝑯\+𝑼​diag⁡\(𝜶\)\\bm\{Z\}=\\bm\{H\}\+\\bm\{\\alpha\}\\odot\\bm\{U\}=\\bm\{H\}\+\\bm\{U\}\\operatorname\{diag\}\(\\bm\{\\alpha\}\)\. Below,𝔼\[⋅∣t,y\]\\mathbb\{E\}\[\\,\\cdot\\mid t,y\]averages over images and noises at the same\(t,y\)\(t,y\); it does not require an actual minibatch to repeat\(t,y\)\(t,y\)\. To isolate gate\-weighted channel norms, we drop the skip–residual cross term from‖𝑯\+𝑼​diag⁡\(𝜶\)‖F2\\\|\\bm\{H\}\+\\bm\{U\}\\operatorname\{diag\}\(\\bm\{\\alpha\}\)\\\|\_\{\\mathrm\{F\}\}^\{2\}and analyze the norm

r~𝜶=\(∥𝑯∥F2\+∑d=1Dαd2∥𝑼:,d∥22\)1/2,\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}=\\left\(\\\|\\bm\{H\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\sum\_\{d=1\}^\{D\}\\alpha\_\{d\}^\{2\}\\\|\\bm\{U\}\_\{:,d\}\\\|\_\{2\}^\{2\}\\right\)^\{1/2\},\(3\)where∥𝑼:,d∥22\\\|\\bm\{U\}\_\{:,d\}\\\|\_\{2\}^\{2\}is the squared pre\-gating norm of channeldd, summed over tokens\. We denote the radial objective evaluated with this norm,𝔼\[\(r~𝜶−c\)2∣t,y\]\\mathbb\{E\}\[\(\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}\-c\)^\{2\}\\mid t,y\], byℒ~rad\\widetilde\{\\mathcal\{L\}\}\_\{\\mathrm\{rad\}\}\. We assume*channel\-symmetric*pre\-gating norm patterns within this conditional analysis, assigning no a priori preference to any residual channel\. This assumption puts channels on equal footing before gating \([AppendixD](https://arxiv.org/html/2610.40914#A4)\)\.

###### Proposition 1\(Gate participation under channel symmetry\)\. Under channel symmetry and the regularity conditions in[AppendixD](https://arxiv.org/html/2610.40914#A4), a sufficiently small conditional gate update𝜶\+=𝜶−η​∇𝜶ℒ~rad​\(𝜶\)\\bm\{\\alpha\}^\{\+\}=\\bm\{\\alpha\}\-\\eta\\nabla\_\{\\bm\{\\alpha\}\}\\widetilde\{\\mathcal\{L\}\}\_\{\\mathrm\{rad\}\}\(\\bm\{\\alpha\}\), withη\>0\\eta\>0, satisfiesnPR⁡\(𝜶\+\)≥nPR⁡\(𝜶\)\.\\mathrm\{nPR\}\(\\bm\{\\alpha\}^\{\+\}\)\\geq\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)\.\(4\)

The mechanism follows from the relative gate gradient,

−1αd∂ℒ~rad∂αd=2𝔼\[\(cr~𝜶−1\)∥𝑼:,d∥22\|t,y\],\-\\frac\{1\}\{\\alpha\_\{d\}\}\\frac\{\\partial\\widetilde\{\\mathcal\{L\}\}\_\{\\mathrm\{rad\}\}\}\{\\partial\\alpha\_\{d\}\}=2\\,\\mathbb\{E\}\\\!\\left\[\\left\(\\frac\{c\}\{\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}\}\-1\\right\)\\\|\\bm\{U\}\_\{:,d\}\\\|\_\{2\}^\{2\}\\,\\middle\|\\,t,y\\right\],\(5\)which equalsΔ​αd/\(η​αd\)\\Delta\\alpha\_\{d\}/\(\\eta\\alpha\_\{d\}\)withΔ​αd=αd\+−αd\\Delta\\alpha\_\{d\}=\\alpha\_\{d\}^\{\+\}\-\\alpha\_\{d\}\.c/r~𝜶−1c/\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}\-1is specific to each input embedding and shared by all channels of that input; channels differ only through∥𝑼:,d∥22\\\|\\bm\{U\}\_\{:,d\}\\\|\_\{2\}^\{2\}andαd\\alpha\_\{d\}\. Consider two otherwise matched input norm patterns at the same\(t,y\)\(t,y\), differing only by exchanging two pre\-gating channel norms\. If the larger channel norm is paired with the larger gate, the whole\-input norm becomes larger, reducing the shared radial coefficient; swapping the channel norms gives the opposite effect\. Under channel symmetry, the two patterns are weighted equally\. Averaging over such realizations gives weaker gates no smaller fractional magnitude updates than stronger ones:

\|αi\|\>\|αj\|⟹Δ​αiαi≤Δ​αjαj\.\|\\alpha\_\{i\}\|\>\|\\alpha\_\{j\}\|\\quad\\Longrightarrow\\quad\\frac\{\\Delta\\alpha\_\{i\}\}\{\\alpha\_\{i\}\}\\leq\\frac\{\\Delta\\alpha\_\{j\}\}\{\\alpha\_\{j\}\}\.\(6\)For sufficiently small, sign\-preserving steps, weaker gates can thus grow faster or shrink more slowly, reducing gate concentration\. Under the same analysis, positive and negative quadratic norm objectives yield uniform relative gate rescaling, whereas the radial objective introduces input\-dependent inverse\-norm weighting\. This result concerns the conditional gate\-space update in the cross\-term\-excluded analysis, rather than the full parameter update during training\.

Higher participation and lower FID under radial training\.We search over target norms using SiT\-B/2 trained for 100K steps\. We calibrateλ\\lambdaover the first five training iterations so that the weighted auxiliary gradient norm is approximately 10% of the velocity\-loss gradient norm\. We measure MLP gates at the twelfth layer \(L12\), on which the dispersive loss and the radial objective act\. The target is selected by the lowest observed FID;[SectionC\.1](https://arxiv.org/html/2610.40914#A3.SS1)reports the search values and sensitivity\. Across the five settings in[Fig\.6\(c\)](https://arxiv.org/html/2610.40914#S3.F6.sf3), L12 MLP gate participation averaged over timesteps and classes is negatively correlated with FID \(r=−0\.916r=\-0\.916\)\. These measurements test whether radial training yields higher participation together with improved generation quality\. FID decreases from 45\.72 at the lowest target to 39\.43 at the selected target \(k=10k=10\), while mean gate participation increases from 0\.381 to 0\.577\.

Generation quality with and without a teacher\.[Table3](https://arxiv.org/html/2610.40914#S4.T3)compares SiT\-B/2 models at 400K training steps\. We train Radial objective with the selected targetc=10​N​Dc=10\\sqrt\{ND\}\. Its FID of 11\.83 improves on Vanilla \(16\.24\) and Dispersive \(13\.19\) while requiring no teacher\. Its mean L12 MLP gate participation of 0\.642 is likewise the highest, compared with 0\.434 for Vanilla and 0\.525 for Dispersive\.

Table 3:Generation quality at 400K training steps, SiT\-B/2 at256×256256\\times 256\. We report the results of Vanilla, Dispersive, and the proposed Radial objective, with REPA as a teacher\-guided reference\. The best teacher\-free results are in bold\.Gate participation during training\.We track gate participation at 50K, 100K, 200K, and 400K training steps for Vanilla, Dispersive, and Radial\. At each checkpoint, we evaluate L12 MLP gate participation averaged over timesteps and classes using EMA weights;[Fig\.7](https://arxiv.org/html/2610.40914#S4.F7)\(right\) connects these measurements without smoothing\. Radial maintains higher gate participation than Vanilla at all four checkpoints\. Dispersive also exceeds Vanilla, with participation peaking at 200K\.

Figure 7:Training dynamics over 400K steps\.\(Left\)Training MSE\.\(Center\)Normalized auxiliary losses\.\(Right\)MeannPR⁡\(𝜶\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)over 101 uniformly spaced timesteps in\[0,1\]\[0,1\]and 1,000 classes\.Training\-loss visualization\.We also report the recorded velocity\-prediction MSE and auxiliary training losses\. Here,ℒaux\\mathcal\{L\}\_\{\\mathrm\{aux\}\}denotes the additional regularization term: the dispersive loss for Dispersive and the feature\-norm penalty in[Eq\.2](https://arxiv.org/html/2610.40914#S4.E2)for Radial, each measured before weighting by its coefficient\. Vanilla has no auxiliary term\. For the auxiliary curves in[Fig\.7](https://arxiv.org/html/2610.40914#S4.F7)\(center\), we plot the normalized gapℒ~aux=\(Lt−Lmin\)/\(L0−Lmin\)\\widetilde\{\\mathcal\{L\}\}\_\{\\mathrm\{aux\}\}=\(L\_\{t\}\-L\_\{\\min\}\)/\(L\_\{0\}\-L\_\{\\min\}\), whereLtL\_\{t\}is the raw auxiliary loss at training steptt,L0L\_\{0\}is its step\-zero value, andLmin=−ln⁡32L\_\{\\min\}=\-\\ln 32for Dispersive and00for Radial\.

## 5Discussion and Limitations

Interpreting iREPA and spatial structure\.iREPA motivates its convolutional projector and spatial normalization as ways to improve the transfer of spatial structure from teacher representations\([Singh et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib22)\)\. Its spatial diagnostics, including LDS, summarize how similarity between patch tokens varies with their spatial separation\. Our fixed\-teacher projector sweep in[Section3\.2\.2](https://arxiv.org/html/2610.40914#S3.SS2.SSS2)helps distinguish properties that change together when replacing an MLP with a convolution: nonlinear depth, channel transformation, and spatial mixing\. At 50K steps, the1×11\\times 1projector reaches FID 51\.33, compared with 51\.68 for an additional3×33\\times 3convolutional projector trained under the same teacher normalization and alignment settings\. The comparable quality of these two heads shows that neighboring\-token mixing inside the projector is not required to attain this performance in the tested configuration\. The lower FID of the deep linear head relative to the nonlinear MLP also motivates examining the effect of projector nonlinearities\.

Association and intervention\.The teacher and projector sweeps show a recurring relationship between gate participation and FID within their respective experimental settings\. The post\-hoc pilot establishes sensitivity to gate modification, but its uniformizing interventions do not consistently improve generation quality\. The WordNet analysis shows class\-related structure in the gates; it does not establish a lower bound on the participation needed to represent a semantic hierarchy\. The Radial objective provides a simple teacher\-free training objective that yields higher participation and lower FID in the tested target sweep\. It acts on feature norms and affects the gates indirectly, alongside other model parameters\. The local gradient proposition specifies conditions for a positive radial contribution \([SectionD\.1](https://arxiv.org/html/2610.40914#A4.SS1)\); it does not guarantee participation growth under the full optimizer or establish that such growth causes the FID improvement\.

## 6Related Work

Representation alignment in diffusion transformers\.REPA supervises intermediate diffusion features with pretrained visual representations, motivated by the difficulty of learning useful representations from the generative objective alone\([Yu et al\., 2025](https://arxiv.org/html/2610.40914#bib.bib25)\)\. Subsequent work examines the alignment target and projector in iREPA\([Singh et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib22)\), and the timing and scope of alignment in HASTE, which combines feature and attention\-map supervision before terminating alignment during training\([Wang et al\., 2025](https://arxiv.org/html/2610.40914#bib.bib24)\)\. Our analysis concerns the internal modulation induced by these training choices, focusing on residual\-gate participation and geometry\.

Spatial structure in diffusion representations\.iREPA compares global semantic performance with spatial self\-similarity in teacher representations and motivates changes intended to improve the transfer of spatial information\([Singh et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib22)\)\. HASTE also incorporates spatial relationships through attention\-map supervision\([Wang et al\., 2025](https://arxiv.org/html/2610.40914#bib.bib24)\)\. Our projector experiments instead examine the role of projection architecture within the tested alignment configurations, without making a broader claim about the necessity of spatial information in diffusion models\.

Adaptive modulation and gating in DiTs\.DiT uses adaptive normalization and residual scaling through AdaLN\-Zero\([Peebles and Xie, 2023](https://arxiv.org/html/2610.40914#bib.bib16)\); SiT adopts this transformer backbone for interpolant\-based generative modeling\([Ma et al\., 2024](https://arxiv.org/html/2610.40914#bib.bib13)\)\. Related analyses examine modulation initialization\([Zhu et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib26)\)and the concentration of semantic information in conditional embeddings\([Pham et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib17)\)\. We focus specifically on the learned branch\-specific residual gate𝜶\\bm\{\\alpha\}, its allocation across channels, and its geometry across training objectives\.

Representation geometry and effective dimensionality\.Sparsity measures characterize the concentration of coefficients\([Hurley and Rickard, 2008](https://arxiv.org/html/2610.40914#bib.bib10)\), while intrinsic\-dimension estimates have been used to assess self\-supervised representations\([Mordacq et al\., 2026](https://arxiv.org/html/2610.40914#bib.bib14)\)\. Dispersive training modifies representation geometry through an auxiliary separation objective\([Wang and He, 2025](https://arxiv.org/html/2610.40914#bib.bib23)\)\. We apply participation ratio to squared gate coordinates in the model’s channel basis\. This measures the allocation of gate magnitude across channels, rather than the spectral rank or intrinsic dimension of hidden representations, and serves here as a diagnostic associated with generation quality\.

## 7Conclusion

We examined diffusion transformer acceleration through class\-conditioned residual gates\. Accelerated models depend more strongly on these gates: removing the AdaLN\-Zero modulation shrinks their gains, and perturbing𝜶\\bm\{\\alpha\}degrades REPA the most\. The gates also reflect the semantic hierarchy among classes, most strongly under REPA, and their gate participation tracks FID across REPA teachers and, for a fixed teacher, across projectors\. Building on this, the teacher\-free Radial objective raised gate participation and improved FID over Dispersive without pairwise feature comparisons\. Because gate participation is computed from the conditioning pathway alone, without an image forward pass, it offers a lightweight quantity to monitor in accelerated diffusion training\.

### AI use statement

In this work, we used generative AI tools to write small pieces of code for implementing our methods, to assist in writing proofs, and to formulate mathematical claims\. We have not used generative AI tools to develop theoretical models or conceptual frameworks, to provide critical ingredients for proofs, to propose or refine hypotheses, to design or give feedback on the research methodology or experiments, to interpret results, or to translate text\. Synthetic data generation, dataset cleaning and reformatting, and qualitative or thematic data analysis are not applicable to this work\. Additionally, we used generative AI tools to search for related work, to edit the paper for readability, and to check its formatting\. We emphasize that generative AI did not originate the core methodology or generate the full experimental codebase\. We have reviewed all AI\-assisted work: the authors derived and verified all mathematical claims and proofs themselves, AI\-assisted code was validated through sanity checks and by reproducing published baseline results, and every work found through AI\-assisted search was checked against the original paper\. We take responsibility for the final content of this work, including text, claims, code, analyses, and results produced with the aid of generative AI\.

### Reproducibility statement

We provide all implementation details and hyperparameters in[AppendicesA](https://arxiv.org/html/2610.40914#A1),[B](https://arxiv.org/html/2610.40914#A2)and[C](https://arxiv.org/html/2610.40914#A3)\.

## References

- Bolyaet al\.\(2025\)D\. Bolya, P\. Huang, P\. Sun, J\. H\. Cho, A\. Madotto, C\. Wei, T\. Ma, J\. Zhi, J\. Rajasegaran, H\. Rasheed, J\. Wang, M\. Monteiro, H\. Xu, S\. Dong, N\. Ravi, D\. Li, P\. Dollár, and C\. FeichtenhoferPerception Encoder: The best visual embeddings are not at the output of the network\.InConference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§B\.5](https://arxiv.org/html/2610.40914#A2.SS5.p1.1)\.
- Chenet al\.\(2021\)X\. Chen, S\. Xie, and K\. HeAn Empirical Study of Training Self\-Supervised Vision Transformers\.InIEEE International Conference on Computer Vision \(ICCV\),Cited by:[§B\.5](https://arxiv.org/html/2610.40914#A2.SS5.p1.1)\.
- Denget al\.\(2009\)J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and Li Fei\-FeiImageNet: A large\-scale hierarchical image database\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§A\.1](https://arxiv.org/html/2610.40914#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.40914#S1.p4.1)\.
- Dhariwal and Nichol \(2021\)P\. Dhariwal and A\. NicholDiffusion Models Beat GANs on Image Synthesis\.InConference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§A\.3](https://arxiv.org/html/2610.40914#A1.SS3.p1.1)\.
- Dormand and Prince \(1980\)J\.R\. Dormand and P\.J\. PrinceA family of embedded Runge\-Kutta formulae\.Journal of Computational and Applied Mathematics\.Cited by:[§A\.3](https://arxiv.org/html/2610.40914#A1.SS3.p1.1)\.
- Fanet al\.\(2025\)D\. Fan, S\. Tong, J\. Zhu, K\. Sinha, Z\. Liu, X\. Chen, M\. Rabbat, N\. Ballas, Y\. LeCun, A\. Bar, and S\. XieScaling Language\-Free Visual Representation Learning\.InIEEE International Conference on Computer Vision \(ICCV\),Cited by:[§B\.5](https://arxiv.org/html/2610.40914#A2.SS5.p1.1)\.
- C\. Fellbaum \(Ed\.\) \(1998\)C\. Fellbaum \(Ed\.\)WordNet: An Electronic Lexical Database\.MIT Press\.External Links:ISBN 978\-0\-262\-27255\-1Cited by:[§1](https://arxiv.org/html/2610.40914#S1.p3.1),[§3\.2\.1](https://arxiv.org/html/2610.40914#S3.SS2.SSS1.p1.1)\.
- Heuselet al\.\(2017\)M\. Heusel, H\. Ramsauer, T\. Unterthiner, B\. Nessler, and S\. HochreiterGANs Trained by a Two Time\-Scale Update Rule Converge to a Local Nash Equilibrium\.InConference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§A\.3](https://arxiv.org/html/2610.40914#A1.SS3.p1.1),[§1](https://arxiv.org/html/2610.40914#S1.p4.1)\.
- Ho and Salimans \(2022\)J\. Ho and T\. SalimansClassifier\-Free Diffusion Guidance\.arXiv preprint arXiv:2207\.12598\.Cited by:[§B\.2](https://arxiv.org/html/2610.40914#A2.SS2.p2.1)\.
- Hurley and Rickard \(2008\)N\. P\. Hurley and S\. T\. RickardComparing Measures of Sparsity\.IEEE Transactions on Information Theory\.Cited by:[§6](https://arxiv.org/html/2610.40914#S6.p4.1)\.
- Kynkäänniemiet al\.\(2019\)T\. Kynkäänniemi, T\. Karras, S\. Laine, J\. Lehtinen, and T\. AilaImproved Precision and Recall Metric for Assessing Generative Models\.InConference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§A\.3](https://arxiv.org/html/2610.40914#A1.SS3.p1.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled Weight Decay Regularization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§A\.1](https://arxiv.org/html/2610.40914#A1.SS1.SSS0.Px1.p1.1)\.
- Maet al\.\(2024\)N\. Ma, M\. Goldstein, M\. S\. Albergo, N\. M\. Boffi, E\. Vanden\-Eijnden, and S\. XieSiT: Exploring Flow and Diffusion\-based Generative Models with Scalable Interpolant Transformers\.InEuropean Conference on Computer Vision \(ECCV\),Cited by:[§A\.1](https://arxiv.org/html/2610.40914#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.40914#S1.p2.1),[Table 3](https://arxiv.org/html/2610.40914#S4.T3.4.3.1),[§4](https://arxiv.org/html/2610.40914#S4.p2.1),[§6](https://arxiv.org/html/2610.40914#S6.p3.1),[footnote 2](https://arxiv.org/html/2610.40914#footnote2)\.
- Mordacqet al\.\(2026\)J\. Mordacq, V\. Kalogeiton, and S\. OudotIdEst: Assessing Self\-Supervised Learning Representations via Intrinsic Dimension\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§6](https://arxiv.org/html/2610.40914#S6.p4.1)\.
- Oquabet al\.\(2024\)M\. Oquab, T\. Darcet, T\. Moutakanni, H\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby, M\. Assran, N\. Ballas, W\. Galuba, R\. Howes, P\. Huang, S\. Li, I\. Misra, M\. Rabbat, V\. Sharma, G\. Synnaeve, H\. Xu, H\. Jegou, J\. Mairal, P\. Labatut, A\. Joulin, and P\. BojanowskiDINOv2: Learning Robust Visual Features without Supervision\.Transactions on Machine Learning Research \(TMLR\)\.Cited by:[§B\.2](https://arxiv.org/html/2610.40914#A2.SS2.p1.1)\.
- Peebles and Xie \(2023\)W\. Peebles and S\. XieScalable Diffusion Models with Transformers\.InIEEE International Conference on Computer Vision \(ICCV\),Cited by:[§B\.1](https://arxiv.org/html/2610.40914#A2.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.40914#S1.p2.1),[§2](https://arxiv.org/html/2610.40914#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.40914#S3.SS1.p2.1),[§6](https://arxiv.org/html/2610.40914#S6.p3.1)\.
- Phamet al\.\(2026\)T\. X\. Pham, K\. Zhang, J\. W\. Hong, and C\. D\. YooA Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.40914#S2.SS0.SSS0.Px1.p3.1),[§6](https://arxiv.org/html/2610.40914#S6.p3.1),[footnote 3](https://arxiv.org/html/2610.40914#footnote3)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. SutskeverLearning Transferable Visual Models From Natural Language Supervision\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§B\.5](https://arxiv.org/html/2610.40914#A2.SS5.p1.1)\.
- Rombachet al\.\(2022\)R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. OmmerHigh\-Resolution Image Synthesis with Latent Diffusion Models\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§B\.2](https://arxiv.org/html/2610.40914#A2.SS2.p2.1)\.
- Salimanset al\.\(2016\)T\. Salimans, I\. Goodfellow, W\. Zaremba, V\. Cheung, A\. Radford, and X\. ChenImproved Techniques for Training GANs\.InConference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§A\.3](https://arxiv.org/html/2610.40914#A1.SS3.p1.1)\.
- Siméoniet al\.\(2025\)O\. Siméoni, H\. V\. Vo, M\. Seitzer, F\. Baldassarre, M\. Oquab, C\. Jose, V\. Khalidov, M\. Szafraniec, S\. Yi, M\. Ramamonjisoa, F\. Massa, D\. Haziza, L\. Wehrstedt, J\. Wang, T\. Darcet, T\. Moutakanni, L\. Sentana, C\. Roberts, A\. Vedaldi, J\. Tolan, J\. Brandt, C\. Couprie, J\. Mairal, H\. Jégou, P\. Labatut, and P\. BojanowskiDINOv3\.arXiv preprint arXiv:2508\.10104\.Cited by:[§B\.5](https://arxiv.org/html/2610.40914#A2.SS5.p1.1)\.
- Singhet al\.\(2026\)J\. Singh, X\. Leng, Z\. Wu, L\. Zheng, R\. Zhang, E\. Shechtman, and S\. XieWhat matters for Representation Alignment: Global Information or Spatial Structure?\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.5](https://arxiv.org/html/2610.40914#A2.SS5.p1.1),[§1](https://arxiv.org/html/2610.40914#S1.p1.1),[§2](https://arxiv.org/html/2610.40914#S2.SS0.SSS0.Px1.p4.1),[§3\.2\.2](https://arxiv.org/html/2610.40914#S3.SS2.SSS2.p4.1),[§5](https://arxiv.org/html/2610.40914#S5.p1.1),[§6](https://arxiv.org/html/2610.40914#S6.p1.1),[§6](https://arxiv.org/html/2610.40914#S6.p2.1)\.
- Wang and He \(2025\)R\. Wang and K\. HeDiffuse and Disperse: Image Generation with Representation Regularization\.arXiv preprint arXiv:2506\.09027\.Cited by:[§A\.1](https://arxiv.org/html/2610.40914#A1.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.40914#S1.p1.1),[§2](https://arxiv.org/html/2610.40914#S2.SS0.SSS0.Px1.p4.1),[§3\.1](https://arxiv.org/html/2610.40914#S3.SS1.p1.1),[Table 3](https://arxiv.org/html/2610.40914#S4.T3.4.4.1),[§6](https://arxiv.org/html/2610.40914#S6.p4.1)\.
- Wanget al\.\(2025\)Z\. Wang, W\. Zhao, Y\. Zhou, Z\. Li, Z\. Liang, M\. Shi, X\. Zhao, P\. Zhou, K\. Zhang, Z\. Wang, K\. Wang, and Y\. YouREPA Works Until It Doesn’t: Early\-Stopped, Holistic Alignment Supercharges Diffusion Training\.InConference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§6](https://arxiv.org/html/2610.40914#S6.p1.1),[§6](https://arxiv.org/html/2610.40914#S6.p2.1)\.
- Yuet al\.\(2025\)S\. Yu, S\. Kwak, H\. Jang, J\. Jeong, J\. Huang, J\. Shin, and S\. XieRepresentation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.40914#S1.p1.1),[§2](https://arxiv.org/html/2610.40914#S2.SS0.SSS0.Px1.p4.1),[§3\.1](https://arxiv.org/html/2610.40914#S3.SS1.p1.1),[§3\.2\.2](https://arxiv.org/html/2610.40914#S3.SS2.SSS2.p3.1),[Table 3](https://arxiv.org/html/2610.40914#S4.T3.4.7.1.1),[§6](https://arxiv.org/html/2610.40914#S6.p1.1)\.
- Zhuet al\.\(2026\)J\. Zhu, M\. Ding, B\. Duan, L\. Wang, and J\. WangUnveiling the Secret of AdaLN\-Zero in Diffusion Transformer\.IEEE Transactions on Pattern Analysis and Machine Intelligence \(TPAMI\)\.Cited by:[§2](https://arxiv.org/html/2610.40914#S2.SS0.SSS0.Px1.p3.1),[§3\.1](https://arxiv.org/html/2610.40914#S3.SS1.p1.1),[§6](https://arxiv.org/html/2610.40914#S6.p3.1)\.

## Appendix

## Appendix ACommon Setup

### A\.1Models and Training

All experiments use class\-conditional ImageNet\([Deng et al\., 2009](https://arxiv.org/html/2610.40914#bib.bib3)\)generation at256×256256\\times 256resolution with SiT\([Ma et al\., 2024](https://arxiv.org/html/2610.40914#bib.bib13)\)and AdaLN\-Zero, except the in\-context conditioning ablation of Pilot study 1 \([SectionB\.1](https://arxiv.org/html/2610.40914#A2.SS1)\)\. The teacher sweep, gate intervention, hierarchy analysis, and gate histograms use SiT\-XL/2, with 28 blocks, hidden width 1,152, and 16 attention heads\. The projector and Radial objective experiments use SiT\-B/2, with 12 blocks, width 768, and 12 heads\. Both architectures use2×22\\times 2latent patches and an MLP expansion ratio of four\. Layer numbers are one\-based: L1 denotes the first transformer block\.

##### Local SiT\-B/2 training\.

Unless stated otherwise, our SiT\-B/2 runs use seed 42, global batch size 256, bfloat16 mixed precision, and AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2610.40914#bib.bib12)\)with a constant learning rate of10−410^\{\-4\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999, and zero weight decay\. The label\-drop probability is 0\.1 and the EMA decay is 0\.9999\. Our SiT\-B/2 Dispersive runs use the best\-performing PyTorch configuration of the official implementation\([Wang and He, 2025](https://arxiv.org/html/2610.40914#bib.bib23)\): the InfoNCE\-L2 dispersive loss on the final\-block output \(L12\) with coefficientλ=0\.25\\lambda=0\.25and temperatureτ=1\\tau=1, computed within each per\-GPU batch of 32 on eight GPUs\. The velocity objective uses a linear interpolation path\. Training uses cached ImageNet latents from the ft\-MSE VAE, sampled from the stored posterior\.

### A\.2Gate Participation Measurement

Unless a fixed timestep is stated, we evaluate the conditioning pathway on 101 evenly spaced timesteps in\[0,1\]\[0,1\], including both endpoints, and all 1,000 ImageNet class labels\. For each MLP gate vector𝜶⁡\(t,y\)∈ℝD\\bm\{\\alpha\}\(t,y\)\\in\\mathbb\{R\}^\{D\}, we computenPR⁡\(𝜶\)=\(∑dαd2\)2/\(D​∑dαd4\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)=\(\\sum\_\{d\}\\alpha\_\{d\}^\{2\}\)^\{2\}/\(D\\sum\_\{d\}\\alpha\_\{d\}^\{4\}\)and take the arithmetic mean over the 101,000 conditions, which givesnPR⁡\(𝜶\)¯\\overline\{\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)\}\. We average participation values rather than gate vectors, and do not concatenate MHSA and MLP gates\. Gates are computed in FP32 and statistics in FP64\. We use the convention𝒙t=\(1−t\)​𝒙data\+t​𝒙noise\\bm\{x\}\_\{t\}=\(1\-t\)\\bm\{x\}\_\{\\mathrm\{data\}\}\+t\\bm\{x\}\_\{\\mathrm\{noise\}\}and convert it to each checkpoint’s input convention\. In the scatter plots, regression lines are ordinary least\-squares fits of FID tonPR⁡\(𝜶\)¯\\overline\{\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)\}, andrris the Pearson correlation across the plotted runs\.

### A\.3FID Evaluation

Local FIDs use the ADM evaluator\([Dhariwal and Nichol, 2021](https://arxiv.org/html/2610.40914#bib.bib4)\)and the ImageNet\-256 reference statistics, with 50,000 generated images, EMA model weights, thesd\-vae\-ft\-emadecoder, CFG 1\.5, and sampling seed 42\. The sampler is adaptive Dormand–Prince \(dopri5\)\([Dormand and Prince, 1980](https://arxiv.org/html/2610.40914#bib.bib5)\), with absolute tolerance10−610^\{\-6\}and relative tolerance10−310^\{\-3\}\. Besides FID\([Heusel et al\., 2017](https://arxiv.org/html/2610.40914#bib.bib8)\), the evaluator reports sFID\([Dhariwal and Nichol, 2021](https://arxiv.org/html/2610.40914#bib.bib4)\), inception score \(IS\)\([Salimans et al\., 2016](https://arxiv.org/html/2610.40914#bib.bib20)\), and precision and recall\([Kynkäänniemi et al\., 2019](https://arxiv.org/html/2610.40914#bib.bib11)\)\. The gate intervention pilot and the teacher sweep use different protocols, described in[SectionsB\.2](https://arxiv.org/html/2610.40914#A2.SS2)and[B\.5](https://arxiv.org/html/2610.40914#A2.SS5)\.

## Appendix BDetails of the Gate Analysis

### B\.1In\-Context Conditioning Pilot

[Table1](https://arxiv.org/html/2610.40914#S3.T1)compares six SiT\-B/2 runs trained for 100K steps with the setup of[SectionA\.1](https://arxiv.org/html/2610.40914#A1.SS1)and evaluated at their 100K EMA checkpoints with the local FID protocol of[SectionA\.3](https://arxiv.org/html/2610.40914#A1.SS3)\. Within each conditioning architecture, the three runs differ only in the auxiliary objective\. REPA aligns L4 with DINOv2\-B through a three\-layer MLP projector with coefficient 0\.5, and Dispersive uses coefficient 0\.25 at the final block \(L12\)\.

##### In\-context conditioning\.

Following DiT\([Peebles and Xie, 2023](https://arxiv.org/html/2610.40914#bib.bib16)\), we replace AdaLN\-Zero with two conditioning tokens, a timestep embedding and a class embedding, prepended to the patch tokens\. The blocks are standard pre\-LN transformer blocks with affine LayerNorm and no scale, shift, or gate\. The final layer is an affine LayerNorm followed by a linear projection without modulation, and the two conditioning tokens are discarded before unpatchifying\. Patch tokens use fixed 2D sine\-cosine positional embeddings, and the conditioning tokens receive zero positional embeddings\. Auxiliary losses are computed on the patch tokens only\.

### B\.2Gate Intervention Pilot

The pilot uses SiT\-XL/2 EMA checkpoints at 400K steps for Vanilla, Dispersive, and REPA with DINOv2\-B\([Oquab et al\., 2024](https://arxiv.org/html/2610.40914#bib.bib15)\); the REPA checkpoint is the DINOv2\-B checkpoint of the teacher sweep \([SectionB\.5](https://arxiv.org/html/2610.40914#A2.SS5)\)\. To compare Dispersive and REPA on the same block, the Dispersive checkpoint applies the dispersive loss at L8, where these REPA checkpoints apply alignment, with coefficient 0\.5 andτ=1\\tau=1\. At every sampling step, we modify only the realized L8 MLP gate\. Writingpd=αd2/‖𝜶‖22p\_\{d\}=\\alpha\_\{d\}^\{2\}/\\\|\\bm\{\\alpha\}\\\|\_\{2\}^\{2\}, the uniform direction usespd′=\(1−ϵ\)​pd\+ϵ/Dp^\{\\prime\}\_\{d\}=\(1\-\\epsilon\)p\_\{d\}\+\\epsilon/D, while the concentrated direction usespd′=pdκ/∑jpjκp^\{\\prime\}\_\{d\}=p\_\{d\}^\{\\kappa\}/\\sum\_\{j\}p\_\{j\}^\{\\kappa\}withκ\>1\\kappa\>1\. We reconstructαd′=sign⁡\(αd\)​‖𝜶‖2​pd′\\alpha^\{\\prime\}\_\{d\}=\\operatorname\{sign\}\(\\alpha\_\{d\}\)\\\|\\bm\{\\alpha\}\\\|\_\{2\}\\sqrt\{p^\{\\prime\}\_\{d\}\}and solve forϵ\\epsilonorκ\\kappaseparately for each condition to match the target relative perturbationδ\\delta\. This preserves signs, gate norm, and channel ordering\. The dose grid containsδ=0\.10\\delta=0\.10,0\.300\.30, and a model\-specific near\-maximal feasible dose \(0\.6270\.627for Vanilla,0\.5550\.555for Dispersive, and0\.5900\.590for REPA\)\.

Each evaluation uses 50,000 generated images, CFG 1\.0\([Ho and Salimans, 2022](https://arxiv.org/html/2610.40914#bib.bib9)\), and 50\-step Euler sampling, with sampling seed 42 shared between interventions and their unmodified baselines\. Vanilla and Dispersive use the SiT sampling convention and ft\-EMA decoder\([Rombach et al\., 2022](https://arxiv.org/html/2610.40914#bib.bib19)\), while the REPA checkpoint uses the REPA convention and ft\-MSE decoder\. Because the pipelines differ, we report FID changes relative to each model’s own baseline\.

### B\.3Gate Histograms and WordNet Distances

The channel histograms and hierarchy analysis use the L8 MLP gates of the SiT\-XL/2 EMA checkpoints at 400K steps for Vanilla, Dispersive, and REPA \(DINOv2\-B\), evaluated att=0\.5t=0\.5for all 1,000 classes\.

##### Gate histograms\.

We pool the signed gate entries over all1,000⋅1,1521\{,\}000\\cdot 1\{,\}152coordinates without centering, squaring, or normalization\. We use width\-0\.025 bins spanning\[−8,8\]\[\-8,8\], which cover every entry, and display\[−2,2\]\[\-2,2\]\. Heights are percentages of all coordinates per bin, lightly smoothed with a Gaussian kernel of standard deviation one bin\.

##### WordNet distances\.

For every class, we take the absolute value of the gate and center these magnitudes across the 1,000 classes separately for each channel\. We use WordNet 3\.0 shortest hypernym\-path lengths over all\(10002\)=499,500\\binom\{1000\}\{2\}=499\{,\}500unordered class pairs\. The Euclidean distance is computed between gate magnitudes, which class centering does not change, and the cosine distance between centered magnitudes\. We average distances over pairs within each WordNet distance bin, retaining bins with at least 500 pairs \(distances 2–22\)\. The minimum bin mean of every curve occurs at WordNet distance 2\. We divide each Euclidean curve by this minimum, which removes the difference in overall gate scale between models, and subtract it from each centered cosine curve, which is already scale\-invariant\.

### B\.4Gate Structure at Additional Timesteps

The WordNet\-distance comparison and the gate histograms in the main text uset=0\.5t=0\.5, the mean of the uniform timestep distribution used in training\. For the same checkpoints and L8 MLP gates, we additionally evaluatet=0\.25t=0\.25and0\.750\.75\.[Fig\.A](https://arxiv.org/html/2610.40914#A2.F1)shows the same qualitative relationship at both timesteps: gate distances increase with WordNet distance, with REPA showing the largest separation across semantic distances\.[Fig\.B](https://arxiv.org/html/2610.40914#A2.F2)shows that the differences in gate scale between the three models persist at both timesteps\.

Figure A:WordNet\-distance analysis at additional timesteps, using the same checkpoints and distance definitions as[Fig\.4](https://arxiv.org/html/2610.40914#S3.F4)\. Top row:t=0\.25t=0\.25; bottom row:t=0\.75t=0\.75\. Left: Euclidean distance of gate magnitudes\. Right: cosine distance of class\-centered gate magnitudes\. Euclidean curves are divided by, and cosine curves shifted by, their own bin mean at WordNet distance 2, with the same retained WordNet bins as in the main text\.Figure B:Signed gate histograms at additional timesteps, using the same checkpoints, bins, smoothing, and display range as[Fig\.5](https://arxiv.org/html/2610.40914#S3.F5)\. Top row:t=0\.25t=0\.25; bottom row:t=0\.75t=0\.75\. Columns: Vanilla, Dispersive, and REPA, with per\-method y\-axis limits shared across the two timesteps\.
### B\.5Teacher Sweep

We use eight REPA SiT\-XL/2 EMA checkpoints at 400K steps released by[Singh et al\. \(2026\)](https://arxiv.org/html/2610.40914#bib.bib22)on Hugging Face \(REPA\-E/iREPA\-collections\), with CLIP\-L\([Radford et al\., 2021](https://arxiv.org/html/2610.40914#bib.bib18)\), DINOv2\-B, DINOv3\-B and DINOv3\-H\+\([Siméoni et al\., 2025](https://arxiv.org/html/2610.40914#bib.bib21)\), LangPE\-G and PE\-G\([Bolya et al\., 2025](https://arxiv.org/html/2610.40914#bib.bib1)\), MoCo\-v3\-B\([Chen et al\., 2021](https://arxiv.org/html/2610.40914#bib.bib2)\), and WebSSL\-1B\([Fan et al\., 2025](https://arxiv.org/html/2610.40914#bib.bib6)\)teachers\. The FIDs are taken from the evaluations released in the same repository at step 400K, which use the ADM evaluator, CFG 1\.0, and SDE sampling with 250 steps\. We measurenPR⁡\(𝜶\)¯\\overline\{\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)\}for the L8 MLP gate of each checkpoint\.

### B\.6Projector Sweep

All projector variants use SiT\-B/2 with DINOv2\-B alignment after the fourth block \(L4\) and alignment coefficient 0\.5\. Each variant is trained for 50K steps with the setup of[SectionA\.1](https://arxiv.org/html/2610.40914#A1.SS1)and evaluated with the local FID protocol\.

##### Projection heads\.

The three\-layer MLP has widths768→1024→1024→768768\\to 1024\\to 1024\\to 768, with SiLU between affine layers; the two\-layer MLP has widths768→1024→768768\\to 1024\\to 768, with one SiLU\. The deep linear head uses the same widths as the three\-layer MLP without intermediate nonlinearities, and the single linear head is768→768768\\to 768\. Their parameter counts, including biases, are 2,624,256, 1,574,656, 2,624,256, and 590,592, respectively\. All variants normalize teacher targets within each image and channel as\(x−0\.7​μN\)/\(σN\+10−6\)\(x\-0\.7\\mu\_\{N\}\)/\(\\sigma\_\{N\}\+10^\{\-6\}\), whereμN\\mu\_\{N\}andσN\\sigma\_\{N\}are the mean and standard deviation over tokens\.

## Appendix CDetails of the Radial Objective

### C\.1Target Sweep and Target Choice

The radial penalty acts on the output of the final SiT\-B/2 block \(L12\), flattened to a256⋅768256\\cdot 768dimensional vector per sample\. All sweep points use 100K\-step EMA checkpoints trained with the setup of[SectionA\.1](https://arxiv.org/html/2610.40914#A1.SS1)and evaluated with the local FID protocol\.

##### Targets and coefficients\.

We search overc∈\{875\.885,1751\.77,3503\.54,4500,6000\}c\\in\\\{875\.885,1751\.77,3503\.54,4500,6000\\\}\. ForN=256N=256andD=768D=768,N​D≈443\.4\\sqrt\{ND\}\\approx 443\.4, and these targets correspond tok≈\{2,4,8,10,14\}k\\approx\\\{2,4,8,10,14\\\}inc=k​N​Dc=k\\sqrt\{ND\}\. All five runs useλ=6\.032×10−8\\lambda=6\.032\\times 10^\{\-8\}, calibrated over the first five training iterations to make the weighted auxiliary gradient norm approximately 10% of the velocity\-loss gradient norm\. We select the target with the lowest FID\.

##### Reference points\.

The Vanilla and Dispersive reference points are 100K EMA checkpoints evaluated with the same protocol; Dispersive uses coefficient 0\.25 at the final block\.

##### Sensitivity\.

[Fig\.C](https://arxiv.org/html/2610.40914#A3.F3)plots FID and gate participation against the target\. The lowest target yields FID 45\.72, close to Vanilla’s 45\.41; the target 3503\.54 reaches 40\.02, the selected target 4500 reaches 39\.43, and the largest target 6000 reaches 39\.44\. L12 MLP gate participation rises from 0\.381 at the lowest target to 0\.577 at the selected target and 0\.564 atc=6000c=6000\.

Figure C:Sensitivity to the flattened feature\-norm target for SiT\-B/2 at 100K steps\. Left: ADM FID\. Right:nPR⁡\(𝜶\)¯\\overline\{\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)\}for L12 MLP gates over 101 uniformly spaced model timesteps in\[0,1\]\[0,1\]and 1,000 classes\.

### C\.2Performance Comparison

[Table3](https://arxiv.org/html/2610.40914#S4.T3)uses SiT\-B/2 at256×256256\\times 256, 400K training steps, the setup of[SectionA\.1](https://arxiv.org/html/2610.40914#A1.SS1), and the local FID protocol of[SectionA\.3](https://arxiv.org/html/2610.40914#A1.SS3)\. The three teacher\-free rows are the same runs as the training trajectories in[Fig\.7](https://arxiv.org/html/2610.40914#S4.F7)\. Dispersive uses coefficient 0\.25 at the final block, and the Radial objective uses the targetc=10​N​D≈4434c=10\\sqrt\{ND\}\\approx 4434with coefficientλ=5×10−8\\lambda=5\\times 10^\{\-8\}\. REPA aligns L4 with DINOv2\-B through a three\-layer MLP projector with coefficient 0\.5\. Mean gate participation is measured at L12 on the same EMA checkpoints\.

### C\.3Training Dynamics

[Fig\.7](https://arxiv.org/html/2610.40914#S4.F7)uses the three 400K SiT\-B/2 runs of[SectionC\.2](https://arxiv.org/html/2610.40914#A3.SS2)\. The loss curves are smoothed with a trailing 5,000\-step moving average over every logged iteration\. Each raw, unweighted auxiliary loss is normalized as\(Lt−Lmin\)/\(L0−Lmin\)\(L\_\{t\}\-L\_\{\\min\}\)/\(L\_\{0\}\-L\_\{\\min\}\), withLmin=−ln⁡32L\_\{\\min\}=\-\\ln 32for Dispersive andLmin=0L\_\{\\min\}=0for Radial\. The gate participation panel evaluates the L12 MLP gate of the EMA checkpoints at 50K, 100K, 200K, and 400K steps with the protocol of[SectionA\.2](https://arxiv.org/html/2610.40914#A1.SS2)\.

## Appendix DConditions and Proof of Proposition[1](https://arxiv.org/html/2610.40914#Thmproposition1)

##### Analytical setting\.

Fix\(t,y\)\(t,y\)at the current model state and let𝜶=𝜶θ​\(t,y\)∈ℝD\\bm\{\\alpha\}=\\bm\{\\alpha\}\_\{\\theta\}\(t,y\)\\in\\mathbb\{R\}^\{D\}\. We represent the input realizations at this condition byBBfeature pairs𝑯\(b\),𝑼\(b\)∈ℝN×D\\bm\{H\}^\{\(b\)\},\\bm\{U\}^\{\(b\)\}\\in\\mathbb\{R\}^\{N\\times D\}, held fixed with respect to the local gate variation, so that𝔼\[⋅∣t,y\]\\mathbb\{E\}\[\\,\\cdot\\mid t,y\]in the main text is the average overbb\. These realizations share\(t,y\)\(t,y\)and hence the gate; they need not appear together in a training minibatch\. Herer~𝜶\(b\)\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b\)\}denotes[Eq\.3](https://arxiv.org/html/2610.40914#S4.E3)evaluated at\(𝑯\(b\),𝑼\(b\)\)\(\\bm\{H\}^\{\(b\)\},\\bm\{U\}^\{\(b\)\}\)\. The analyzed loss is

ℒ~rad​\(𝜶\)=1B​∑b=1B\(r~𝜶\(b\)−c\)2\.\\widetilde\{\\mathcal\{L\}\}\_\{\\mathrm\{rad\}\}\(\\bm\{\\alpha\}\)=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\(\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b\)\}\-c\)^\{2\}\.\(7\)Herec\>0c\>0\. At the update point, allαd\\alpha\_\{d\}are nonzero and allr~𝜶\(b\)\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b\)\}are positive\. These conditions ensure that the finite\-average loss,nPR⁡\(𝜶\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\), and the fractional gate updates are well\-defined and smooth locally\. A positive auxiliary coefficient merely rescales the step size\.

##### Precise channel\-symmetry assumption\.

For every permutationπ\\piof theDDchannel indices, the two multisets

\{\(∥𝑯\(b\)∥F2,∥𝑼:,1\(b\)∥22,…,∥𝑼:,D\(b\)∥22\)\}b=1B,\\displaystyle\\left\\\{\\left\(\\\|\\bm\{H\}^\{\(b\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\},\\\|\\bm\{U\}\_\{:,1\}^\{\(b\)\}\\\|\_\{2\}^\{2\},\\ldots,\\\|\\bm\{U\}\_\{:,D\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\\right\)\\right\\\}\_\{b=1\}^\{B\},\(8\)\{\(∥𝑯\(b\)∥F2,∥𝑼:,π\(1\)\(b\)∥22,…,∥𝑼:,π\(D\)\(b\)∥22\)\}b=1B\\displaystyle\\left\\\{\\left\(\\\|\\bm\{H\}^\{\(b\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\},\\\|\\bm\{U\}\_\{:,\\pi\(1\)\}^\{\(b\)\}\\\|\_\{2\}^\{2\},\\ldots,\\\|\\bm\{U\}\_\{:,\\pi\(D\)\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\\right\)\\right\\\}\_\{b=1\}^\{B\}\(9\)coincide, including multiplicities\. The skip norm remains attached to its sample pattern, and the gate is not permuted\. This condition concerns squared\-norm patterns, not individual feature entries or their signs\. It implies identical averaged squared channel norms, but is stronger than equality of these averages alone\. Neither Gaussianity nor independence is assumed\. Exact symmetry is a property of the analytical model, not a consequence of an architecture lacking hard\-coded channel preferences and not a claim about observed mini\-batches\.

##### Relative gate update\.

Differentiating[Eq\.7](https://arxiv.org/html/2610.40914#A4.E7)gives

−∂ℒ~rad∂αd=2​αdB∑b\(cr~𝜶\(b\)−1\)∥𝑼:,d\(b\)∥22,\-\\frac\{\\partial\\widetilde\{\\mathcal\{L\}\}\_\{\\mathrm\{rad\}\}\}\{\\partial\\alpha\_\{d\}\}=\\frac\{2\\alpha\_\{d\}\}\{B\}\\sum\_\{b\}\\left\(\\frac\{c\}\{\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b\)\}\}\-1\\right\)\\\|\\bm\{U\}\_\{:,d\}^\{\(b\)\}\\\|\_\{2\}^\{2\},\(10\)which gives[Eq\.5](https://arxiv.org/html/2610.40914#S4.E5)\. Symmetry makesB−1∑b∥𝑼:,d\(b\)∥22B^\{\-1\}\\sum\_\{b\}\\\|\\bm\{U\}\_\{:,d\}^\{\(b\)\}\\\|\_\{2\}^\{2\}independent ofdd\.

##### Ordering of the inverse\-norm\-weighted responses\.

Fori≠ji\\neq j, define the norm with the two channel norms interchanged:

r~𝜶\(b;i↔j\)=\(∥𝑯\(b\)∥F2\+αi2∥𝑼:,j\(b\)∥22\+αj2∥𝑼:,i\(b\)∥22\+∑k≠i,jαk2∥𝑼:,k\(b\)∥22\)1/2\.\\begin\{split\}\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b;i\\leftrightarrow j\)\}=\\Big\(&\\\|\\bm\{H\}^\{\(b\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\alpha\_\{i\}^\{2\}\\\|\\bm\{U\}\_\{:,j\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\+\\alpha\_\{j\}^\{2\}\\\|\\bm\{U\}\_\{:,i\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\\\\ &\+\\sum\_\{k\\neq i,j\}\\alpha\_\{k\}^\{2\}\\\|\\bm\{U\}\_\{:,k\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\\Big\)^\{1/2\}\.\\end\{split\}\(11\)The channel\-symmetry assumption gives

1B∑b∥𝑼:,i\(b\)∥22r~𝜶\(b\)−1B∑b∥𝑼:,j\(b\)∥22r~𝜶\(b\)\\displaystyle\\frac\{1\}\{B\}\\sum\_\{b\}\\frac\{\\\|\\bm\{U\}\_\{:,i\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\}\{\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b\)\}\}\-\\frac\{1\}\{B\}\\sum\_\{b\}\\frac\{\\\|\\bm\{U\}\_\{:,j\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\}\{\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b\)\}\}=12​B∑b\(∥𝑼:,i\(b\)∥22−∥𝑼:,j\(b\)∥22\)\(1r~𝜶\(b\)−1r~𝜶\(b;i↔j\)\)\\displaystyle\\quad=\\frac\{1\}\{2B\}\\sum\_\{b\}\(\\\|\\bm\{U\}\_\{:,i\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\-\\\|\\bm\{U\}\_\{:,j\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\)\\left\(\\frac\{1\}\{\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b\)\}\}\-\\frac\{1\}\{\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b;i\\leftrightarrow j\)\}\}\\right\)=−αi2−αj22​B∑b\(∥𝑼:,i\(b\)∥22−∥𝑼:,j\(b\)∥22\)2r~𝜶\(b\)r~𝜶\(b;i↔j\)\(r~𝜶\(b\)\+r~𝜶\(b;i↔j\)\)\.\\displaystyle\\quad=\-\\frac\{\\alpha\_\{i\}^\{2\}\-\\alpha\_\{j\}^\{2\}\}\{2B\}\\sum\_\{b\}\\frac\{\(\\\|\\bm\{U\}\_\{:,i\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\-\\\|\\bm\{U\}\_\{:,j\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\)^\{2\}\}\{\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b\)\}\\,\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b;i\\leftrightarrow j\)\}\(\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b\)\}\+\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b;i\\leftrightarrow j\)\}\)\}\.\(12\)The summands in the final line are nonnegative\. Hence larger squared gate magnitudes have no larger inverse\-norm\-weighted responses, proving[Eq\.6](https://arxiv.org/html/2610.40914#S4.E6)\. Equality of two squared gate magnitudes also implies equality of their weighted responses\. For unequal squared gate magnitudes, the ordering is strict if their squared channel norms differ in at least one sample\.

##### One\-step non\-decrease ofnPR⁡\(𝜶\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)\.

For the proof only, define

𝒗=−∇𝜶ℒ~rad​\(𝜶\),ρd=vdαd,pd=αd2∑jαj2\.\\bm\{v\}=\-\\nabla\_\{\\bm\{\\alpha\}\}\\widetilde\{\\mathcal\{L\}\}\_\{\\mathrm\{rad\}\}\(\\bm\{\\alpha\}\),\\qquad\\rho\_\{d\}=\\frac\{v\_\{d\}\}\{\\alpha\_\{d\}\},\\qquad p\_\{d\}=\\frac\{\\alpha\_\{d\}^\{2\}\}\{\\sum\_\{j\}\\alpha\_\{j\}^\{2\}\}\.\(13\)Allpd\>0p\_\{d\}\>0, and the prior nPR definition givesnPR⁡\(𝜶\)=1/\(D​∑dpd2\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)=1/\(D\\sum\_\{d\}p\_\{d\}^\{2\}\)\. Differentiation in direction𝒗\\bm\{v\}yields

⟨∇𝜶​log​nPR​\(𝜶\),𝒗⟩\\displaystyle\\left\\langle\\nabla\_\{\\bm\{\\alpha\}\}\\log\\mathrm\{nPR\}\(\\bm\{\\alpha\}\),\\bm\{v\}\\right\\rangle=4​\(∑dpd​ρd−∑dpd2​ρd∑dpd2\)\\displaystyle=4\\left\(\\sum\_\{d\}p\_\{d\}\\rho\_\{d\}\-\\frac\{\\sum\_\{d\}p\_\{d\}^\{2\}\\rho\_\{d\}\}\{\\sum\_\{d\}p\_\{d\}^\{2\}\}\\right\)=−4∑dpd2∑i<jpipj\(pi−pj\)\(ρi−ρj\)≥0\.\\displaystyle=\-\\frac\{4\}\{\\sum\_\{d\}p\_\{d\}^\{2\}\}\\sum\_\{i<j\}p\_\{i\}p\_\{j\}\(p\_\{i\}\-p\_\{j\}\)\(\\rho\_\{i\}\-\\rho\_\{j\}\)\\geq 0\.\(14\)Each summand has the stated sign by[Eq\.6](https://arxiv.org/html/2610.40914#S4.E6)\. If the directional derivative is strictly positive, Taylor expansion gives

nPR⁡\(𝜶\+η​𝒗\)−nPR⁡\(𝜶\)=η⁡⟨∇𝜶nPR​\(𝜶\),𝒗⟩\+O⁡\(η2\)\>0\\begin\{split\}&\\mathrm\{nPR\}\(\\bm\{\\alpha\}\+\\eta\\bm\{v\}\)\-\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)\\\\ &\\qquad=\\eta\\left\\langle\\nabla\_\{\\bm\{\\alpha\}\}\\mathrm\{nPR\}\(\\bm\{\\alpha\}\),\\bm\{v\}\\right\\rangle\+O\(\\eta^\{2\}\)\>0\\end\{split\}\(15\)for every sufficiently small positiveη\\eta\.

It remains to justify finite\-step non\-decrease when the derivative is zero; a nonnegative first derivative alone would not suffice\. In that case, every summand in[Eq\.14](https://arxiv.org/html/2610.40914#A4.E14)is zero\. Thus,pi≠pjp\_\{i\}\\neq p\_\{j\}impliesρi=ρj\\rho\_\{i\}=\\rho\_\{j\}\. Ifpi=pjp\_\{i\}=p\_\{j\}, the equal\-gate case of[Eq\.12](https://arxiv.org/html/2610.40914#A4.E12)also impliesρi=ρj\\rho\_\{i\}=\\rho\_\{j\}\. Consequently all fractional rates have a common valueρ\\rho; hence𝒗=ρ​𝜶\\bm\{v\}=\\rho\\bm\{\\alpha\}and the step is exactly𝜶\+=\(1\+η​ρ\)​𝜶\\bm\{\\alpha\}^\{\+\}=\(1\+\\eta\\rho\)\\bm\{\\alpha\}\. This preservesnPR⁡\(𝜶\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)whenever the scale is nonzero\. Together the two cases establish the finite\-step inequality in Proposition[1](https://arxiv.org/html/2610.40914#Thmproposition1)\.

The increase is strict if there existi,j,bi,j,bsuch thatαi2≠αj2\\alpha\_\{i\}^\{2\}\\neq\\alpha\_\{j\}^\{2\}and∥𝑼:,i\(b\)∥22≠∥𝑼:,j\(b\)∥22\\\|\\bm\{U\}\_\{:,i\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\\neq\\\|\\bm\{U\}\_\{:,j\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\. The argument concerns sufficiently small steps, not arbitrary learning rates\. Choosing the step small enough also preserves gate signs\.

##### Quadratic comparison\.

For the same cross\-term\-excluded norm,

𝒬±​\(𝜶\)=±1B∑b\(r~𝜶\(b\)\)2=±\[1B∑b∥𝑯\(b\)∥F2\+∑dαd21B∑b∥𝑼:,d\(b\)∥22\]\.\\begin\{split\}\\mathcal\{Q\}\_\{\\pm\}\(\\bm\{\\alpha\}\)&=\\pm\\frac\{1\}\{B\}\\sum\_\{b\}\(\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b\)\}\)^\{2\}\\\\ &=\\pm\\left\[\\frac\{1\}\{B\}\\sum\_\{b\}\\\|\\bm\{H\}^\{\(b\)\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\sum\_\{d\}\\alpha\_\{d\}^\{2\}\\frac\{1\}\{B\}\\sum\_\{b\}\\\|\\bm\{U\}\_\{:,d\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\\right\]\.\\end\{split\}\(16\)For this calculation only, letq¯\\bar\{q\}denote the common averageB−1∑b∥𝑼:,d\(b\)∥22B^\{\-1\}\\sum\_\{b\}\\\|\\bm\{U\}\_\{:,d\}^\{\(b\)\}\\\|\_\{2\}^\{2\}\. A gradient step gives𝜶\+=\(1∓2​η​q¯\)​𝜶\\bm\{\\alpha\}^\{\+\}=\(1\\mp 2\\eta\\bar\{q\}\)\\bm\{\\alpha\}and therefore preservesnPR⁡\(𝜶\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)whenever the scale factor is nonzero\. ∎

### D\.1Scope of the Analysis

The actual residual norm satisfies

‖𝑯\(b\)\+𝑼\(b\)​diag⁡\(𝜶\)‖F2=\(r~𝜶\(b\)\)2\+2∑dαd⟨𝑯:,d\(b\),𝑼:,d\(b\)⟩\.\\begin\{split\}&\\\|\\bm\{H\}^\{\(b\)\}\+\\bm\{U\}^\{\(b\)\}\\operatorname\{diag\}\(\\bm\{\\alpha\}\)\\\|\_\{\\mathrm\{F\}\}^\{2\}\\\\ &\\qquad=\(\\widetilde\{r\}\_\{\\bm\{\\alpha\}\}^\{\(b\)\}\)^\{2\}\+2\\sum\_\{d\}\\alpha\_\{d\}\\langle\\bm\{H\}\_\{:,d\}^\{\(b\)\},\\bm\{U\}\_\{:,d\}^\{\(b\)\}\\rangle\.\\end\{split\}\(17\)The analysis intentionally omits the last term\. Orthogonality such as𝒉⊤​𝒖=0\\bm\{h\}^\{\\top\}\\bm\{u\}=0, or⟨𝑯,𝑼⟩F=0\\langle\\bm\{H\},\\bm\{U\}\\rangle\_\{\\mathrm\{F\}\}=0, does not imply that this gate\-weighted cross term vanishes\. Nor does a small cross\-term value at one gate imply a small derivative with respect to that gate\. The proposition applies to the cross\-term\-excluded norm, not to the full residual norm without additional assumptions\.

In actual training the gate is parameterized; other features, task gradients, and optimizer state can also change\. The proposition concerns one local Euclidean gate update and does not imply monotonicnPR⁡\(𝜶\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)growth under those full dynamics or a causal relationship betweennPR⁡\(𝜶\)\\mathrm\{nPR\}\(\\bm\{\\alpha\}\)and FID\. The quadratic comparison is likewise restricted to the same analysis\.