Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers

Hugging Face Daily Papers Papers
huggingface curated

Summary

# Paper page - Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers Source: [https://huggingface.co/papers/2605.06169](https://huggingface.co/papers/2605.06169) **Mean Mode Screaming \(MMS\)**— the abrupt entry event into a silent, mean\-dominated collapse state in ultra\-deep Diffusion Transformers\. Optimization can remain stable for thousands of steps and then diverge within a few updates, with the loss returning to near its initialization level\. We trace

Scaling Diffusion Transformers (DiTs) to hundreds of layers introduces a structural vulnerability: networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation. Through mechanistic auditing, we isolate the trigger event of this collapse as Mean Mode Screaming (MMS). MMS can occur even when training appears stable, with a mean-coherent backward shock on residual writers that opens deep residual branches and drives the network into a mean-dominated state. We show this behavior is driven by an exact decomposition of these gradients into mean-coherent and centered components, compounded by the structural suppression of attention-logit gradients through the null space of the Softmax Jacobian once values homogenize. To address this, we propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline; it tracks close to the baseline's pre-crash trajectory while remaining substantially better than token-isotropic gating methods such as LayerScale across the full schedule. Finally, we present a 1000-layer DiT as a scale-validation run at boundary scales, establishing that the architecture remains stably trainable at extreme depth.
Original Article
View Cached Full Text

Cached at: 05/11/26, 10:44 AM

Paper page - Mean Mode Screaming: Mean–Variance Split Residuals for 1000-Layer Diffusion Transformers

Source: https://huggingface.co/papers/2605.06169 Mean Mode Screaming (MMS)— the abrupt entry event into a silent, mean-dominated collapse state in ultra-deep Diffusion Transformers. Optimization can remain stable for thousands of steps and then diverge within a few updates, with the loss returning to near its initialization level. We trace this to a mean-coherent backward shock onresidual writersthat opens deep residual branches and drives the network into a state where token representations homogenize and centered variation is suppressed.

Mechanistically, the failure exploits a geometric asymmetry between the token-mean and centered subspaces. Row-stochastic attention strictly preserves pure-mean states, while gradients admit an exact decomposition into mean-coherent and centered components — as token alignment increases, the mean-coherent component accumulates in an O(T) coherent regime and dominates the residual update. Once values homogenize, attention-logit gradients are suppressed through the null space of the Softmax Jacobian, locking the network into the collapsed state. Existing depth stabilizers (ReZero, LayerScale) shrink the two modes by the same factor — this stabilizes training but also damps the signal-bearing centered mode, slowing convergence.

Mean–Variance Split (MV-Split) Residualscombine a separately gained centered residual update with a leaky trunk-mean replacement, regulating the mean path without scaling down the centered path by the same factor. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline and converges faster than LayerScale across the full schedule. A 1000-layer DiT serves as a scale-validation run at boundary scales.

🌐Interactive gradient-diagnosis viewerreplays the actual W&B run that crashed — per-layer 3D token-flow driven by W&B metrics. Scrub the timeline, switch the driving metric, hover for live values:https://erwold.github.io/mv-split/

🤗 1000-layer DiT weights:https://huggingface.co/StableKirito/mvsplit-dit-1000l

Similar Articles

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

Hugging Face Daily Papers

SMELT is a method that loops middle layers in Mixture-of-Experts Transformers to improve training efficiency and downstream performance while matching compute, parameter, and cache budgets, leading to faster loss reduction and practical gains.

Looped Diffusion Transformer: the same blocks run several times inside each denoising step, and loop depth can be changed at sampling time [R]

Reddit r/MachineLearning

介绍 Looped Diffusion Transformer 论文：通过在每个去噪步骤内重复运行共享的 transformer 块，并结合深度监督和自调制注意力，使 260M 模型以 4.9 倍更少的推理计算超越 6.5 倍大的模型，且循环深度可在采样时调节；作者还用 MIT 开源权重在 M5 Pro 上实测了不同深度的效果。

A Mesoscopic View of Transformer Weights Through Row and Column Scale Fields

arXiv cs.LG

This paper introduces 'row and column scale fields'—median-centred log-RMS profiles of Transformer weight matrices over their channels—to mesoscopically analyze how weight magnitude is distributed across functional channels, showing how these fields evolve through training, align across projections, relate to AdamW's second-moment structure, and how edits to them affect model loss.

Generalization Dynamics of LM Pre-training (4 minute read)

TLDR AI

该论文提出"模式跳变"(mode-hopping)概念，指出LM预训练中平滑下降的损失与稳定基准分数会掩盖模型答题方式的突变（如OLMo3-32B在算术提示上81%→0%→81.7%的反复），并通过六项廉价行为测试排除了评估指标非线性与局部优化波动等简单解释。