LayerRoPE：动态深度感知的幅度与角度叠加
摘要
提出 LayerRoPE，将 Transformer 中隐式的深度位置编码显式化：用单一共享 γ 向量与深度条件标量替代逐层 γ，在参数减少、FLOPs 几乎不变的情况下，在最高 512 层的深度上实现更稳定的训练与显著更高的计算效率（达到 Pre-Norm 1.3B 损失所需算力减少 3.4 倍）。
arXiv:2610.09179v1 Announce Type: new
Abstract: As data propagates through a Transformer, the norm of its hidden states grows by orders of magnitude with depth, a phenomenon framed as 'curse of depth' and nearly universally treated as a pathology to be suppressed. We take the opposite view. Across 16 pre-trained LLMs from 9 families, spanning dense, mixture-of-experts and hybrid architectures and Pre-, Peri- and Post-Norm designs, we find that this growth reflects an emergent depth-positional encoding, carried by the only learned per-layer gain on the residual stream, the normalization weight $\gamma$: with depth, $\gamma$ grows in magnitude and rotates in direction, jointly encoding the layer index. We make this depth-conditioned encoding explicit with LayerRoPE, an implicit analog of RoPE along the depth axis, which replaces all layerwise $\gamma$ vectors with a single shared vector and depth-conditioned scalars, at a net reduction in parameters and $<0.02\%$ change in FLOPs. Across a model ladder scaled up to $100$B+ tokens, LayerRoPE consistently outperforms Pre-, Post- and Peri-Norm and Layer-Norm Scaling, reaching Pre-Norm's 1.3B loss with $3.4\times$ less compute; LayerRoPE is the only approach that shows strong convergence and improves near monotonically as depth scales to 512 layers. It improves learning-rate sensitivity by $3$-$10\times$, and transfers naively to and consistently improves looped latent models and Vision Transformers. Inspecting its learned schedule inverts the prevailing premise: LayerRoPE does not shrink the residual stream but widens it, damping what each block reads while amplifying what it writes. Depth stability, our results suggest, calls not for suppressing the residual stream, but for depth-conditioned regulation of the computational blocks it feeds.
查看缓存全文
缓存时间: 2026/10/08 12:55
# LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition Source: [https://arxiv.org/abs/2610.09179](https://arxiv.org/abs/2610.09179) [View PDF](https://arxiv.org/pdf/2610.09179)[HTML \(experimental\)](https://arxiv.org/html/2610.09179v1) > Abstract:As data propagates through a Transformer, the norm of its hidden states grows by orders of magnitude with depth, a phenomenon framed as 'curse of depth' and nearly universally treated as a pathology to be suppressed\. We take the opposite view\. Across 16 pre\-trained LLMs from 9 families, spanning dense, mixture\-of\-experts and hybrid architectures and Pre\-, Peri\- and Post\-Norm designs, we find that this growth reflects an emergent depth\-positional encoding, carried by the only learned per\-layer gain on the residual stream, the normalization weight $\\gamma$: with depth, $\\gamma$ grows in magnitude and rotates in direction, jointly encoding the layer index\. We make this depth\-conditioned encoding explicit with LayerRoPE, an implicit analog of RoPE along the depth axis, which replaces all layerwise $\\gamma$ vectors with a single shared vector and depth\-conditioned scalars, at a net reduction in parameters and $<0\.02\\%$ change in FLOPs\. Across a model ladder scaled up to $100$B\+ tokens, LayerRoPE consistently outperforms Pre\-, Post\- and Peri\-Norm and Layer\-Norm Scaling, reaching Pre\-Norm's 1\.3B loss with $3\.4\\times$ less compute; LayerRoPE is the only approach that shows strong convergence and improves near monotonically as depth scales to 512 layers\. It improves learning\-rate sensitivity by $3$\-$10\\times$, and transfers naively to and consistently improves looped latent models and Vision Transformers\. Inspecting its learned schedule inverts the prevailing premise: LayerRoPE does not shrink the residual stream but widens it, damping what each block reads while amplifying what it writes\. Depth stability, our results suggest, calls not for suppressing the residual stream, but for depth\-conditioned regulation of the computational blocks it feeds\. ## Submission history From: Shikhar Srivastava \[[view email](https://arxiv.org/show-email/e1cea716/2610.09179)\] **\[v1\]**Tue, 6 Oct 2026 22:30:58 UTC \(4,739 KB\)
相似文章
单块多深度：通过深度编程专家实现递归视觉 Transformer
该论文提出 reViT，通过在每个递归深度用一个共享专家库的凸组合来表示 FFN，使单个 Transformer 块以递归方式应用即可在相近推理 FLOPs 下匹配全深度视觉编码器的精度，并支持弹性深度部署与权重量级压缩。
DeepLoop：循环Transformer的深度缩放
DeepLoop为循环Transformer引入了一种残差缩放方法，该方法根据参数访问进行调整，从而在物理块跨多轮重用时提高稳定性和性能。
循环扩散 Transformer：在每个去噪步骤中重复运行相同的模块，并且可以在采样时调整循环深度 [R]
介绍 Looped Diffusion Transformer 论文：通过在每个去噪步骤内重复运行共享的 transformer 块，并结合深度监督和自调制注意力，使 260M 模型以 4.9 倍更少的推理计算超越 6.5 倍大的模型，且循环深度可在采样时调节；作者还用 MIT 开源权重在 M5 Pro 上实测了不同深度的效果。
通过层特定位置嵌入缩放缓解Transformer中的位置偏差
介绍LPES，一种层特定位置嵌入缩放方法，通过使用贝塞尔曲线的遗传算法为每层分配不同的缩放因子，缓解LLM中的“中间丢失”问题，无需微调或增加延迟即可实现高达11.2%的准确率提升。
ShellElastic：可扩展弹性预训练，同时实现最大化更新与超参数迁移
提出ShellElastic，一种将隐藏维度组织为稠密核心与递增外壳的弹性Transformer架构，使单次全宽前向即可同时包含所有子网络的精确隐藏状态；并引入μE扩展最大化更新参数化（μP），保证子网络在超网络加宽或增加外壳时仍被最大化更新且超参数可迁移，性能优于MatFormer等弹性基线并逼近分别训练的模型。