residual-connections

Multi-Stream Attention Residuals (MSAR) 将注意力残差（AttnRes）从单流扩展为多头深度注意力，通过多尺度因果卷积引入流多样性，在 18B/28B MoE 规模的 12 项下游基准上一致优于标准残差、mHC 和 AttnRes，scaling law 实验显示相对标准残差基线有 1.58 倍的 iso-loss 算力加速。

This paper introduces DepthBench, a controlled benchmark for studying computational depth in Transformers by varying width-depth ratios. It identifies that residual connection designs like HC and FullAttnRes are key to translating architectural depth into effective computation.

This paper proposes Orthogonal Hyper-Connections (oHC), which restricts the residual mixing matrix to the rotation group SO(n) to maintain training stability and prevent stream homogenization. Evaluated on downstream tasks, oHC outperforms baselines like single-stream residuals and manifold-constrained hyper-connections.

A reproducibility study reveals asymmetric effects when removing residual connections in Pre-LN transformers: attention-skip removal leads to collapse, while FFN-skip removal allows partial recovery at smaller scales.

This arXiv paper introduces TEMPER, a tensorized parameterization for hyper-connection residual routing that reduces parameter growth while preserving expressive, manifold-constrained routing. Experiments show it matches or outperforms existing methods with substantially fewer additional parameters.

This paper unifies Euclidean residual connections and geodesic normalization on the hypersphere, introducing a one-parameter family of angular retractions called p-SpheretNorm. The proposed norm-preserving algebraic methods outperform existing deep connection schemes on nanoGPT, showing the exponential map is just one end of a spectrum.

Frac-Connections is a new method that partitions hidden states instead of expanding them, extending Hyper-Connections to fractional expansion rates. It achieves stronger downstream performance than residual connections with virtually no extra compute, as validated on up to 7B MoE language models.

This paper introduces Multi-Resolution Residual Routing (WAV v1), an extension of Block Attention Residuals that augments block representations with directional detail bases, improving deep decoder-only Transformer training.

This paper proposes Diffusion-Adaptive Routing (DAR), a learnable, timestep-adaptive residual replacement that improves cross-layer information flow in Diffusion Transformers, leading to significant training acceleration and quality improvements.

DeepSeek has published a paper introducing mHC (Manifold-Constrained Hyper-Connections), a fundamental rewrite of the Transformer architecture that stabilizes large models by replacing standard residual connections with mathematically constrained multi-stream pathways.