xHC: Expanded Hyper-Connections

Hugging Face Daily Papers Papers
hyper-connections transformer residual-stream scaling efficiency language-model pre-training

Summary

This paper introduces xHC (Expanded Hyper-Connections), a method that enables meaningful expansion of residual streams in Transformers beyond the previous N=4 limit, achieving consistent improvements on 18B and 28B MoE models while adding only modest training FLOPs. It also presents xHC-Flash to reduce memory traffic, making large-N residual-stream expansion practical for LLM pre-training.

Hyper-Connections (HC) expand the residual stream of Transformers into N parallel streams, providing a form of memory scaling beyond model width and depth. Manifold-Constrained HC (mHC) stabilizes this formulation at scale. The large gains from N{=}1 to N{=}4 suggest residual-stream expansion as a promising scaling axis. However, existing HC-family methods typically stop at N{=}4. Our experiments reveal why: scaling mHC beyond this point yields diminishing performance gains and rapidly increasing training cost. We attribute this limitation to two bottlenecks: insufficient write-back information for an expanding number of streams and residual-mixing generation whose cost scales cubically with N. To address both bottlenecks, we propose xHC (Expanded Hyper-Connections), the first HC-family method to achieve meaningful expansion beyond N{=}4. xHC combines temporal feature augmentation for richer write-back with a sparse residual-stream architecture that updates only k=4 of the N=16 streams while retaining dense access to the full residual state. Across 18B and 28B MoE models, xHC delivers strong and consistent downstream improvements. On an 18B MoE model, xHC improves the average downstream score by 4.0 points over mHC, while adding only modest training FLOPs over the vanilla baseline. Scaling-law experiments show that the vanilla and mHC require 1.50times and 1.19times the compute of xHC, respectively, to reach the same loss. Practical large-N training also requires controlling memory traffic from the expanded residual state. We therefore introduce xHC-Flash, which reduces the per-sublayer memory traffic from 73.5C to 40C, comparable to the 34C required by mHC at N{=}4, while retaining the gains of full xHC. Together, xHC and xHC-Flash make large-N residual-stream expansion effective and practical for LLM pre-training.
Original Article
View Cached Full Text

Cached at: 07/20/26, 09:40 AM

Paper page - xHC: Expanded Hyper-Connections

Source: https://huggingface.co/papers/2607.14530 Published on Jul 16

#1 Paper of the day Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Hyper-Connections(HC)expandtheresidualstreamofTransformersintoNparallelstreams,providingaformofmemoryscalingbeyondmodelwidthanddepth.Manifold-ConstrainedHC(mHC)stabilizesthisformulationatscale.ThelargegainsfromN{=}1toN{=}4suggestresidual-streamexpansionasapromisingscalingaxis.However,existingHC-familymethodstypicallystopatN{=}4.Ourexperimentsrevealwhy:scalingmHCbeyondthispointyieldsdiminishingperformancegainsandrapidlyincreasingtrainingcost.Weattributethislimitationtotwobottlenecks:insufficientwrite-backinformationforanexpandingnumberofstreamsandresidual-mixinggenerationwhosecostscalescubicallywithN.Toaddressbothbottlenecks,weproposexHC(ExpandedHyper-Connections),thefirstHC-familymethodtoachievemeaningfulexpansionbeyondN{=}4.xHCcombinestemporalfeatureaugmentationforricherwrite-backwithasparseresidual-streamarchitecturethatupdatesonlyk=4oftheN=16streamswhileretainingdenseaccesstothefullresidualstate.Across18Band28BMoEmodels,xHCdeliversstrongandconsistentdownstreamimprovements.Onan18BMoEmodel,xHCimprovestheaveragedownstreamscoreby4.0pointsovermHC,whileaddingonlymodesttrainingFLOPsoverthevanillabaseline.Scaling-lawexperimentsshowthatthevanillaandmHCrequire1.50timesand1.19timesthecomputeofxHC,respectively,toreachthesameloss.Practicallarge-Ntrainingalsorequirescontrollingmemorytrafficfromtheexpandedresidualstate.WethereforeintroducexHC-Flash,whichreducestheper-sublayermemorytrafficfrom73.5Cto40C,comparabletothe34CrequiredbymHCatN{=}4,whileretainingthegainsoffullxHC.Together,xHCandxHC-Flashmakelarge-Nresidual-streamexpansioneffectiveandpracticalforLLMpre-training.

View arXiv pageView PDFGitHub21Add to collection

Get this paper in your agent:

hf papers read 2607\.14530

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.14530 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.14530 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.14530 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Multi-Head Attention Residuals

Hugging Face Daily Papers

Introduces Multi-Head Attention Residuals (MHAR), which reshapes the routing query into per-subspace heads so each feature subspace reads depth history via its own softmax. Trained from scratch on a Nemotron-based corpus, MHAR consistently improves validation loss over standard Transformers from 100M to 1B scale and also boosts mid-training downstream accuracy.

Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory

arXiv cs.CL

This paper introduces CoMem, a method that exploits the depth-wise division of labor in LLMs to cache intermediate residual tensors and recompute only upper layers for retrieval, enabling bounded read compute and memory independent of stored-context length. Evaluated on Qwen3-8B, CoMem achieves strong long-context performance with significant memory savings and prefill speedups.