Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers

Hugging Face Daily Papers 论文
huggingface curated

摘要

# Paper page - Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers Source: [https://huggingface.co/papers/2605.06169](https://huggingface.co/papers/2605.06169) **Mean Mode Screaming \(MMS\)**— the abrupt entry event into a silent, mean\-dominated collapse state in ultra\-deep Diffusion Transformers\. Optimization can remain stable for thousands of steps and then diverge within a few updates, with the loss returning to near its initialization level\. We trace

Scaling Diffusion Transformers (DiTs) to hundreds of layers introduces a structural vulnerability: networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation. Through mechanistic auditing, we isolate the trigger event of this collapse as Mean Mode Screaming (MMS). MMS can occur even when training appears stable, with a mean-coherent backward shock on residual writers that opens deep residual branches and drives the network into a mean-dominated state. We show this behavior is driven by an exact decomposition of these gradients into mean-coherent and centered components, compounded by the structural suppression of attention-logit gradients through the null space of the Softmax Jacobian once values homogenize. To address this, we propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline; it tracks close to the baseline's pre-crash trajectory while remaining substantially better than token-isotropic gating methods such as LayerScale across the full schedule. Finally, we present a 1000-layer DiT as a scale-validation run at boundary scales, establishing that the architecture remains stably trainable at extreme depth.
查看原文
查看缓存全文

缓存时间: 2026/05/11 10:44

Paper page - Mean Mode Screaming: Mean–Variance Split Residuals for 1000-Layer Diffusion Transformers

Source: https://huggingface.co/papers/2605.06169 Mean Mode Screaming (MMS)— the abrupt entry event into a silent, mean-dominated collapse state in ultra-deep Diffusion Transformers. Optimization can remain stable for thousands of steps and then diverge within a few updates, with the loss returning to near its initialization level. We trace this to a mean-coherent backward shock onresidual writersthat opens deep residual branches and drives the network into a state where token representations homogenize and centered variation is suppressed.

Mechanistically, the failure exploits a geometric asymmetry between the token-mean and centered subspaces. Row-stochastic attention strictly preserves pure-mean states, while gradients admit an exact decomposition into mean-coherent and centered components — as token alignment increases, the mean-coherent component accumulates in an O(T) coherent regime and dominates the residual update. Once values homogenize, attention-logit gradients are suppressed through the null space of the Softmax Jacobian, locking the network into the collapsed state. Existing depth stabilizers (ReZero, LayerScale) shrink the two modes by the same factor — this stabilizes training but also damps the signal-bearing centered mode, slowing convergence.

Mean–Variance Split (MV-Split) Residualscombine a separately gained centered residual update with a leaky trunk-mean replacement, regulating the mean path without scaling down the centered path by the same factor. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline and converges faster than LayerScale across the full schedule. A 1000-layer DiT serves as a scale-validation run at boundary scales.

🌐Interactive gradient-diagnosis viewerreplays the actual W&B run that crashed — per-layer 3D token-flow driven by W&B metrics. Scrub the timeline, switch the driving metric, hover for live values:https://erwold.github.io/mv-split/

🤗 1000-layer DiT weights:https://huggingface.co/StableKirito/mvsplit-dit-1000l

相似文章

SMELT: 计算匹配的MoE循环Transformer扩展定律

Hugging Face Daily Papers

SMELT是一种方法，它在Mixture-of-Experts Transformer中循环中间层，以在匹配计算、参数和缓存预算的同时，提高训练效率和下游性能，从而实现更快的损失减少和实际收益。

循环扩散 Transformer：在每个去噪步骤中重复运行相同的模块，并且可以在采样时调整循环深度 [R]

Reddit r/MachineLearning

介绍 Looped Diffusion Transformer 论文：通过在每个去噪步骤内重复运行共享的 transformer 块，并结合深度监督和自调制注意力，使 260M 模型以 4.9 倍更少的推理计算超越 6.5 倍大的模型，且循环深度可在采样时调节；作者还用 MIT 开源权重在 M5 Pro 上实测了不同深度的效果。

Transformer 残差流的动力学：谱几何与网络拓扑的耦合

arXiv cs.LG

本文对生产规模的大型语言模型进行了完整的 Jacobian 特征分解，揭示了从旋转主导的早期层到对称后期层的习得谱梯度，以及一个压缩扰动的低秩瓶颈。结果将扰动传播与压缩与网络功能拓扑联系起来。

# 通过行与列缩放场看 Transformer 权重：一个介观视角

arXiv cs.LG

本文将「行与列尺度场」（row and column scale fields）引入研究：即对 Transformer 权重矩阵沿其通道维度计算的、经过中位数中心化的 log-RMS 剖面，用于在介观尺度上分析权重幅值在各功能通道间如何分布。文章展示了这些尺度场在训练过程中的演化规律、在不同投影之间的对齐关系、与 AdamW 二阶矩结构的联系，以及对这些尺度场进行编辑会如何影响模型损失。

LM 预训练的泛化动力学（4 分钟阅读）

TLDR AI

该论文提出"模式跳变"(mode-hopping)概念，指出LM预训练中平滑下降的损失与稳定基准分数会掩盖模型答题方式的突变（如OLMo3-32B在算术提示上81%→0%→81.7%的反复），并通过六项廉价行为测试排除了评估指标非线性与局部优化波动等简单解释。