Tag
This paper proposes Orthogonal Hyper-Connections (oHC), which restricts the residual mixing matrix to the rotation group SO(n) to maintain training stability and prevent stream homogenization. Evaluated on downstream tasks, oHC outperforms baselines like single-stream residuals and manifold-constrained hyper-connections.
This arXiv paper introduces TEMPER, a tensorized parameterization for hyper-connection residual routing that reduces parameter growth while preserving expressive, manifold-constrained routing. Experiments show it matches or outperforms existing methods with substantially fewer additional parameters.
This paper introduces xHC (Expanded Hyper-Connections), a method that enables meaningful expansion of residual streams in Transformers beyond the previous N=4 limit, achieving consistent improvements on 18B and 28B MoE models while adding only modest training FLOPs. It also presents xHC-Flash to reduce memory traffic, making large-N residual-stream expansion practical for LLM pre-training.
TBP-mHC introduces a novel parameterization for manifold-constrained hyper connections in residual networks, achieving full expressivity of the Birkhoff polytope with O(n^2) degrees of freedom and improved stability and scalability.