Tag
This paper unifies Euclidean residual connections and geodesic normalization on the hypersphere, introducing a one-parameter family of angular retractions called p-SpheretNorm. The proposed norm-preserving algebraic methods outperform existing deep connection schemes on nanoGPT, showing the exponential map is just one end of a spectrum.
Frac-Connections is a new method that partitions hidden states instead of expanding them, extending Hyper-Connections to fractional expansion rates. It achieves stronger downstream performance than residual connections with virtually no extra compute, as validated on up to 7B MoE language models.
This paper introduces Multi-Resolution Residual Routing (WAV v1), an extension of Block Attention Residuals that augments block representations with directional detail bases, improving deep decoder-only Transformer training.
This paper proposes Diffusion-Adaptive Routing (DAR), a learnable, timestep-adaptive residual replacement that improves cross-layer information flow in Diffusion Transformers, leading to significant training acceleration and quality improvements.
DeepSeek has published a paper introducing mHC (Manifold-Constrained Hyper-Connections), a fundamental rewrite of the Transformer architecture that stabilizes large models by replacing standard residual connections with mathematically constrained multi-stream pathways.