Low-Rank Attention Residuals

# Low-Rank Attention Residuals
Source: [https://arxiv.org/html/2607.09694](https://arxiv.org/html/2607.09694)
###### Abstract

Attention Residuals \(AttnRes\) replace the fixed residual sum with depth\-wise attention over previous sub\-layer outputs in Large Language Models \(LLMs\), but use each output as both a full\-dimensional key and value\. This couples routing with representation and makes depth\-routing scores scale with hidden widthdd\. We propose*Low\-Rank Attention Residuals*\(LR\-AttnRes\), which keep full\-dimensional residual values while usingrr\-dimensional keys, withr≪dr\\ll d, for routing\.*Projected LR\-AttnRes*\(P\-LR\-AttnRes\) emits learned low\-rank keys from existing output projections, decoupling routing from residual content and achieving the best validation loss among the variants tested\.*Sliced LR\-AttnRes*\(S\-LR\-AttnRes\) uses the lastrrdimensions of each value as the routing key, removing the auxiliary key\-projection path and reducing total residual\-side FLOPs while still improving performance\. Comprehensive sweeps show that depth\-wise routing can be effective with far fewer dimensions than the model width\. We release[code](https://github.com/jon123boss/LR-AttnRes)and models to facilitate future research\.

![Refer to caption](https://arxiv.org/html/2607.09694v1/x1.png)Figure 1:Validation loss versus percentage of added FLOPs relative to the non\-embedding Transformer core for the baseline,AttnRes, and low\-rankAttnResvariants near0\.5​B0\.5\\text\{B\}parameters trained on10​B10\\text\{B\}tokens\. For projected low\-rank variants, the plotted FLOPs include the auxiliary key\-projection path; for standardAttnResand sliced low\-rank variants, they are the depth\-wise residual kernel FLOPs\.## 1Introduction

Residual connections are central to the optimization of deep networks\(Heet al\.,[2016](https://arxiv.org/html/2607.09694#bib.bib1)\)\. This is especially true in Transformer\-based large language models\(Vaswaniet al\.,[2017](https://arxiv.org/html/2607.09694#bib.bib2)\), where residual streams allow information to accumulate across depth and also serve as gradient highways for optimization\. In a PreNorm decoder\(Xionget al\.,[2020](https://arxiv.org/html/2607.09694#bib.bib3)\), each attention or feed\-forward sub\-layer reads a normalized version of the current stream and writes an output back into it\. This fixed additive rule is simple and stable, but it gives every previous output a fixed unit coefficient\. The model can learn what each sub\-layer writes, yet it cannot directly choose which previous writes should be emphasized when constructing the next hidden state\.

Attention Residuals \(AttnRes\) address this limitation by replacing fixed accumulation with attention over depth\(Teamet al\.,[2026](https://arxiv.org/html/2607.09694#bib.bib4)\)\. At each residual site, the hidden state is formed as a learned weighted combination of earlier sub\-layer outputs\. This is analogous to token attention, but the “tokens” are previous sub\-layer outputs\. The full version attends over all previous sub\-layer outputs, whileBlock AttnResgroups several outputs into block summaries to reduce storage and communication\.

The standardAttnResdesign makes the sub\-layer output𝒐i∈ℝd\{\\bm\{o\}\}\_\{i\}\\in\\mathbb\{R\}^\{d\}serve two roles at once\. It is the value mixed into the residual stream, and after normalization it is also the key used to decide how strongly that source should be selected\. These roles need not require the same representation\. A residual output should remain a full\-capacity carrier of information for downstream layers\. A routing key only needs to distinguish among depth sources\. Using𝒐i\{\\bm\{o\}\}\_\{i\}as both value and key therefore creates a representational tension, and it also makes the score computation scale with the hidden dimensiondddespite depth attention selecting among only tens of sub\-layer sources rather than the vastly larger token axis for which such high\-dimensional comparisons are more naturally justified\.

We introduceLow\-Rank Attention Residuals\(LR\-AttnRes\), where the value mixed into the residual stream remains full\-dimensional, but the depth\-routing score is computed in a much smallerrr\-dimensional space\. We distinguish two ways to obtain this low\-rank key\. The first isProjected Low\-Rank Attention Residuals\(P\-LR\-AttnRes\)\.P\-LR\-AttnResextends the sub\-layer output projection from dimensionddto dimensiond\+rd\+rand splits the result into a full output and a learned routing key\. This decouples routing from representation and gives the strongest validation performance among the variants tested\.

The second isSliced Low\-Rank Attention Residuals\(S\-LR\-AttnRes\)\. Instead of adding a learned key projection,S\-LR\-AttnRessimply uses the lastrrdimensions of the full residual value as the routing key\. This partially reuses the value representation, so it is less expressive thanP\-LR\-AttnRes, but it removes the auxiliary key\-projection path entirely\. As a result, the low\-rank score computation becomes a true net reduction in added FLOPs rather than only a reduction in the depth\-wise attention kernel\. Empirically,S\-LR\-AttnResimproves validation loss while preserving a projection\-free routing path\.

The central empirical result is that low\-rank routing improves both validation loss and routing behavior\. BlockP\-LR\-AttnReswith 8 blocks andr=32r=32obtains the best loss, 2\.9477\. BlockS\-LR\-AttnReswith 8 blocks andr=64r=64obtains 2\.9480 while adding only 0\.093% FLOPs relative to the non\-embedding Transformer core\. Compared with the strongest standard AttnRes result in this setting, this sliced variant improves loss by 0\.0193 and uses 3\.25×\\timesless added compute\. Compared with the best projected model, it is within 0\.0003 validation loss while using 11\.3×\\timesless added compute\.

Our contributions are:

- •We introduceLR\-AttnRes, a low\-rank depth\-routing family that keeps residual values full\-dimensional while reducing the routing key width fromddtorr\.
- •We proposeP\-LR\-AttnRes, which emits a learned low\-rank routing key from existing output projections and decouples source selection from residual content\.
- •We proposeS\-LR\-AttnRes, which uses the lastrrvalue dimensions as the routing key, adds no key\-projection parameters, and reduces total residual\-side FLOPs\.

## 2Related Work

#### Cross\-layer connections\.

A broad line of work modifies the residual pathway\. DenseNet concatenates earlier outputs, while DenseFormer combines layer outputs with learned static depth\-wise weights\(Huanget al\.,[2017](https://arxiv.org/html/2607.09694#bib.bib13); Pagliardiniet al\.,[2024](https://arxiv.org/html/2607.09694#bib.bib14)\)\. Other methods introduce more expressive cross\-layer access, including retrospective layer attention, learned augmented residual layers, multi\-stream or hyper\-connection mechanisms, and input\-dependent depth mixing\(Li and Huang,[2025](https://arxiv.org/html/2607.09694#bib.bib15); Menghaniet al\.,[2025](https://arxiv.org/html/2607.09694#bib.bib16); Zhuet al\.,[2025](https://arxiv.org/html/2607.09694#bib.bib17); Xieet al\.,[2025](https://arxiv.org/html/2607.09694#bib.bib18); Xiaoet al\.,[2025](https://arxiv.org/html/2607.09694#bib.bib19)\)\.

AttnResis the most direct predecessor of our work\(Teamet al\.,[2026](https://arxiv.org/html/2607.09694#bib.bib4)\)\. In standardAttnRes, however, the same full\-dimensional tensor is used both as the residual value and as the routing key\.LR\-AttnReskeeps the cross\-layer routing framework but isolates a more specific design axis: the representation and dimensionality used to score depth sources\. Instead of treating full\-width value\-as\-key routing as a default, we show that low\-dimensional routing keys can improve the validation–FLOPs tradeoff\.

#### Source representations in attention residuals\.

Recent work suggests that the choice of routed source representation strongly affects depth\-wise attention\. Delta Attention Residuals argue that cumulative hidden states can be redundant and lead to low\-contrast depth routing\. They therefore route over per\-sub\-layer or block\-level deltas rather than cumulative states\(Luoet al\.,[2026a](https://arxiv.org/html/2607.09694#bib.bib12)\)\. OASIS studies a different failure mode ofAttnRes\-style models, focusing on attention sinks, activation outliers, and the effect of dual token/depth normalization\. It introduces null\-aware routing to reduce sink\-dominated behavior and improve robustness\(Luoet al\.,[2026b](https://arxiv.org/html/2607.09694#bib.bib20)\)\.LR\-AttnResis complementary: it keeps the residual value path full\-dimensional and instead changes the descriptor used for source selection\.

## 3Method

### 3\.1Depth residual notation

Consider a decoder\-only Transformer withLLblocks andM=2​LM=2Lresidual\-writing sub\-layers: one attention sub\-layer and one feed\-forward sub\-layer per block\. For clarity, equations omit batch and sequence dimensions\. Let𝒐i∈ℝd\{\\bm\{o\}\}\_\{i\}\\in\\mathbb\{R\}^\{d\}denote the output written by sub\-layerii, and let𝒐0∈ℝd\{\\bm\{o\}\}\_\{0\}\\in\\mathbb\{R\}^\{d\}denote the token embedding\. We index residual read sites byt∈\{1,…,M\+1\}t\\in\\\{1,\\ldots,M\+1\\\}, where sitettreads from sources written before that site\. A standard residual stream supplies the fixed sum

𝒉t=𝒐0\+∑i=1t−1𝒐i\{\\bm\{h\}\}\_\{t\}=\{\\bm\{o\}\}\_\{0\}\+\\sum\_\{i=1\}^\{t\-1\}\{\\bm\{o\}\}\_\{i\}\(1\)to residual sitett\.

AttnResreplaces this fixed accumulation with learned attention over depth\. At residual sitett, the hidden state is formed as a weighted mixture over a source set𝒮t⊆\{0,…,t−1\}\\mathcal\{S\}\_\{t\}\\subseteq\\\{0,\\ldots,t\-1\\\}:

αt,i\\displaystyle\\alpha\_\{t,i\}=softmaxi∈𝒮t​\(𝒒t⊤​RMSNorm⁡\(𝒐i\)\),\\displaystyle=\\mathrm\{softmax\}\_\{i\\in\\mathcal\{S\}\_\{t\}\}\\left\(\{\\bm\{q\}\}\_\{t\}^\{\\top\}\\operatorname\{RMSNorm\}\(\{\\bm\{o\}\}\_\{i\}\)\\right\),\(2\)𝒉t\\displaystyle\{\\bm\{h\}\}\_\{t\}=∑i∈𝒮tαt,i​𝒐i\.\\displaystyle=\\sum\_\{i\\in\\mathcal\{S\}\_\{t\}\}\\alpha\_\{t,i\}\{\\bm\{o\}\}\_\{i\}\.\(3\)Here𝒒t∈ℝd\{\\bm\{q\}\}\_\{t\}\\in\\mathbb\{R\}^\{d\}is a learned pseudo\-query associated with residual read sitett\. Thus, unlike a standard residual stream, the model can learn which previous sub\-layer outputs to read from rather than always summing all previous outputs equally\. In this formulation, the same tensor𝒐i\{\\bm\{o\}\}\_\{i\}is used as the value in Eq\.[3](https://arxiv.org/html/2607.09694#S3.E3)and, after normalization, as the key in Eq\.[2](https://arxiv.org/html/2607.09694#S3.E2)\.

#### FullAttnRes\.

The full version attends to every previous residual\-writing source:

𝒮tfull=\{0,1,…,t−1\}\.\\mathcal\{S\}\_\{t\}^\{\\mathrm\{full\}\}=\\\{0,1,\\ldots,t\-1\\\}\.\(4\)FullAttnResgives the maximum depth\-routing flexibility, because every residual site can directly select from the embedding and all earlier attention and feed\-forward outputs\. The cost is that all previous full\-dimensional outputs must be retained as attention sources\.

#### Block AttnRes\.

Block AttnResreduces the number of stored sources by grouping contiguous sub\-layer outputs into block summaries\. At a residual sitett, the model attends to the embedding, all completed block summaries, and the current partial block if one exists\. The partial block contains the outputs that have been written since the most recent completed block boundary but have not yet accumulated enough outputs to form a completed block\.

LetBBbe a completed block containingc=\|B\|c=\|B\|sub\-layer outputs\. The block summary is the sum

𝒃Bsum=∑i∈B𝒐i\.\{\\bm\{b\}\}\_\{B\}^\{\\mathrm\{sum\}\}=\\sum\_\{i\\in B\}\{\\bm\{o\}\}\_\{i\}\.\(5\)Now consider a residual sitettthat lies inside an unfinished block\. LetPt⊆\{1,…,t−1\}P\_\{t\}\\subseteq\\\{1,\\ldots,t\-1\\\}denote the current partial block: the set of residual\-writing sub\-layers after the last completed block boundary and before sitett\. IfPtP\_\{t\}is nonempty, its partial summary is

𝒃Ptsum=∑i∈Pt𝒐i\.\{\\bm\{b\}\}\_\{P\_\{t\}\}^\{\\mathrm\{sum\}\}=\\sum\_\{i\\in P\_\{t\}\}\{\\bm\{o\}\}\_\{i\}\.\(6\)

### 3\.2Low\-rank routing with full\-dimensional values

LR\-AttnReskeeps the same full\-dimensional value path asAttnResbut computes depth\-routing scores in a smaller key space\. Each source is represented as

sourcei:\(𝒐i,𝒌i\),𝒐i∈ℝd,𝒌i∈ℝr,r≪d\.\\text\{source \}i:\\qquad\(\{\\bm\{o\}\}\_\{i\},\{\\bm\{k\}\}\_\{i\}\),\\qquad\{\\bm\{o\}\}\_\{i\}\\in\\mathbb\{R\}^\{d\},\\quad\{\\bm\{k\}\}\_\{i\}\\in\\mathbb\{R\}^\{r\},\\quad r\\ll d\.\(7\)The value𝒐i\{\\bm\{o\}\}\_\{i\}remains the ordinary full\-dimensional sub\-layer output\. The key𝒌i\{\\bm\{k\}\}\_\{i\}is anrr\-dimensional routing descriptor used only to select among depth sources\. At residual sitett, a learned static query𝒒t∈ℝr\{\\bm\{q\}\}\_\{t\}\\in\\mathbb\{R\}^\{r\}scores normalized keys:

𝒌¯i\\displaystyle\\bar\{\{\\bm\{k\}\}\}\_\{i\}=RMSNorm⁡\(𝒌i\),\\displaystyle=\\operatorname\{RMSNorm\}\(\{\\bm\{k\}\}\_\{i\}\),\(8\)αt,i\\displaystyle\\alpha\_\{t,i\}=softmaxi∈𝒮t​\(𝒒t⊤​𝒌¯i\),\\displaystyle=\\mathrm\{softmax\}\_\{i\\in\\mathcal\{S\}\_\{t\}\}\\left\(\{\\bm\{q\}\}\_\{t\}^\{\\top\}\\bar\{\{\\bm\{k\}\}\}\_\{i\}\\right\),\(9\)𝒉t\\displaystyle\{\\bm\{h\}\}\_\{t\}=∑i∈𝒮tαt,i​𝒐i\.\\displaystyle=\\sum\_\{i\\in\\mathcal\{S\}\_\{t\}\}\\alpha\_\{t,i\}\{\\bm\{o\}\}\_\{i\}\.\(10\)Key normalization in Eq\.[8](https://arxiv.org/html/2607.09694#S3.E8)is the default\.

This changes only the routing descriptor\. The output mixed into the residual stream is still the fulldd\-dimensional𝒐i\{\\bm\{o\}\}\_\{i\}, soLR\-AttnResdoes not impose a low\-rank bottleneck on residual values\. The two variants below differ only in how𝒌i\{\\bm\{k\}\}\_\{i\}is obtained\.

### 3\.3Projected low\-rank keys

P\-LR\-AttnResuses a learned low\-rank key emitted from the same output projection that produces the residual value\. Let𝒂i∈ℝmi\{\\bm\{a\}\}\_\{i\}\\in\\mathbb\{R\}^\{m\_\{i\}\}be the activation passed to the output projection of residual\-writing sub\-layerii\. For an attention sub\-layer, this is the activation passed to the attention output projection, so typicallymi=dm\_\{i\}=d\. For a SwiGLU feed\-forward sub\-layer, this is the activation passed to the second/output projection of the FFN, somim\_\{i\}is the SwiGLU hidden width\. If the ordinary output projection isWiO∈ℝd×miW\_\{i\}^\{O\}\\in\\mathbb\{R\}^\{d\\times m\_\{i\}\}, the standard residual contribution is

𝒐i=WiO​𝒂i\.\{\\bm\{o\}\}\_\{i\}=W\_\{i\}^\{O\}\{\\bm\{a\}\}\_\{i\}\.\(11\)P\-LR\-AttnResadds a low\-rank key projection and computes the output and key together:

\[𝒐i𝒌iP\]=\[WiOWiK\]​𝒂i,WiO∈ℝd×mi,WiK∈ℝr×mi,𝒐i∈ℝd,𝒌iP∈ℝr,\\begin\{bmatrix\}\{\\bm\{o\}\}\_\{i\}\\\\ \{\\bm\{k\}\}\_\{i\}^\{\\mathrm\{P\}\}\\end\{bmatrix\}=\\begin\{bmatrix\}W\_\{i\}^\{O\}\\\\ W\_\{i\}^\{K\}\\end\{bmatrix\}\{\\bm\{a\}\}\_\{i\},\\qquad W\_\{i\}^\{O\}\\in\\mathbb\{R\}^\{d\\times m\_\{i\}\},\\qquad W\_\{i\}^\{K\}\\in\\mathbb\{R\}^\{r\\times m\_\{i\}\},\\qquad\{\\bm\{o\}\}\_\{i\}\\in\\mathbb\{R\}^\{d\},\\qquad\{\\bm\{k\}\}\_\{i\}^\{\\mathrm\{P\}\}\\in\\mathbb\{R\}^\{r\},\(12\)In the attention sub\-layer, the fused matrix augments the attention output projection\. In the SwiGLU feed\-forward sub\-layer, it augments the second/output projection that maps the gated hidden activation back to the model dimension\.

The embedding source has no sub\-layer output projection\. The defaultP\-LR\-AttnResembedding key is therefore produced by a small low\-rank projection:

𝒐0=Embed⁡\(x\),𝒌0P=W0K​𝒐0,𝒌¯0P=RMSNorm⁡\(𝒌0P\),W0K∈ℝr×d\.\{\\bm\{o\}\}\_\{0\}=\\operatorname\{Embed\}\(x\),\\qquad\{\\bm\{k\}\}\_\{0\}^\{\\mathrm\{P\}\}=W\_\{0\}^\{K\}\{\\bm\{o\}\}\_\{0\},\\qquad\\bar\{\{\\bm\{k\}\}\}\_\{0\}^\{\\mathrm\{P\}\}=\\operatorname\{RMSNorm\}\(\{\\bm\{k\}\}\_\{0\}^\{\\mathrm\{P\}\}\),\\qquad W\_\{0\}^\{K\}\\in\\mathbb\{R\}^\{r\\times d\}\.\(13\)
P\-LR\-AttnResis the accuracy\-oriented variant\. Because the key has its own learned projection, the model can use the full value vector𝒐i\{\\bm\{o\}\}\_\{i\}for residual\-stream content while using a separate representation for routing\. Empirically, this gives better validation loss than the sliced variant\. Its drawback is that the auxiliary key projections add projection FLOPs and parameters\. The detailed accounting is given in Appendix[A](https://arxiv.org/html/2607.09694#A1)\.

### 3\.4Sliced low\-rank keys

S\-LR\-AttnResremoves the learned key\-projection path\. Instead, it uses the lastrrdimensions of the residual value as the key:

𝒌iS=tailr⁡\(𝒐i\)=\(𝒐i\)d−r\+1:d∈ℝr,𝒌¯iS=RMSNorm⁡\(𝒌iS\)\.\{\\bm\{k\}\}\_\{i\}^\{\\mathrm\{S\}\}=\\operatorname\{tail\}\_\{r\}\(\{\\bm\{o\}\}\_\{i\}\)=\(\{\\bm\{o\}\}\_\{i\}\)\_\{d\-r\+1:d\}\\in\\mathbb\{R\}^\{r\},\\qquad\\bar\{\{\\bm\{k\}\}\}\_\{i\}^\{\\mathrm\{S\}\}=\\operatorname\{RMSNorm\}\(\{\\bm\{k\}\}\_\{i\}^\{\\mathrm\{S\}\}\)\.\(14\)The embedding source uses the same rule:

𝒐0=Embed⁡\(x\),𝒌0S=tailr⁡\(𝒐0\),𝒌¯0S=RMSNorm⁡\(𝒌0S\)\.\{\\bm\{o\}\}\_\{0\}=\\operatorname\{Embed\}\(x\),\\qquad\{\\bm\{k\}\}\_\{0\}^\{\\mathrm\{S\}\}=\\operatorname\{tail\}\_\{r\}\(\{\\bm\{o\}\}\_\{0\}\),\\qquad\\bar\{\{\\bm\{k\}\}\}\_\{0\}^\{\\mathrm\{S\}\}=\\operatorname\{RMSNorm\}\(\{\\bm\{k\}\}\_\{0\}^\{\\mathrm\{S\}\}\)\.\(15\)
This variant deliberately trades routing expressivity for lower cost\. The use of the lastrrdimensions is only a convention\. Since hidden coordinates have no intrinsic order, any fixed subset ofrrcoordinates is equivalent up to a global permutation of the model’s hidden dimensions\. UnlikeP\-LR\-AttnRes, it does not allocate separate key\-projection rows and does not require a separate key tensor to be produced by the sub\-layer\. The routing key can be implemented as a view into the cached residual value\. Whenr=dr=d,S\-LR\-AttnResreduces toAttnRes\.

### 3\.5Efficient inference compatibility

BothP\-LR\-AttnResandS\-LR\-AttnRessupport the efficient inference strategy used by baseAttnRes\. The two\-phase strategy forBlock AttnResrelies on two properties: first, residual\-site queries are input\-independent; second, the inter\-block and intra\-block attention results can be merged with an online\-softmax update\.LR\-AttnRespreserves both properties\. It changes only the key representation used for routing, while keeping the residual values inℝd\\mathbb\{R\}^\{d\}\.

## 4Experiments

### 4\.1Experimental setup

#### Architecture\.

Unless otherwise stated, all experiments use a 24\-layer decoder\-only Transformer with hidden dimensiond=1024d=1024, 16 attention heads, SwiGLU hidden widthm=2816m=2816, RoPE\(Suet al\.,[2024](https://arxiv.org/html/2607.09694#bib.bib6)\)baseθ=500,000\\theta=500\{,\}000, QK normalization\(Henryet al\.,[2020](https://arxiv.org/html/2607.09694#bib.bib7)\), PreNorm, FlashAttention\(Daoet al\.,[2022](https://arxiv.org/html/2607.09694#bib.bib8)\), untied input and output embeddings, and a vocabulary size of100,277100\{,\}277\. The context length is 2048 tokens\. Each Transformer block has one attention sub\-layer and one MLP sub\-layer, so the model has2​L=482L=48residual\-writing sub\-layers\. Depth\-wise pseudo\-queries are initialized to zero and key normalization is enabled\.

#### Training\.

All models are trained from scratch on Ultra\-FineWeb\(Wanget al\.,[2025](https://arxiv.org/html/2607.09694#bib.bib11)\)for up to 10B tokens\. We use a global batch size of 262,144 tokens\. We optimize with Muon\(Jordanet al\.,[2024](https://arxiv.org/html/2607.09694#bib.bib9)\)and Adam\(Kingma and Ba,[2015](https://arxiv.org/html/2607.09694#bib.bib10)\)\. Muon uses learning rate10−310^\{\-3\}, momentum0\.950\.95, and weight decay0\.10\.1\. Adam uses learning rate3×10−43\\times 10^\{\-4\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95, and weight decay0\. Gradients are clipped to norm 1\.0\. We use a linear learning\-rate schedule with 2000 warmup steps and20%20\\%warmdown steps\. We use document masking and z\-loss with weight10−510^\{\-5\}\.

Table 1:Added FLOPs are reported as a percentage of the non\-embedding Transformer core\. ForP\-LR\-AttnRes, this column includes the auxiliary learned key\-projection path; forAttnResandS\-LR\-AttnRes, it is the depth\-wise residual kernel cost because no projected\-key path is present\. Validation loss is computed on200​M200\\mathrm\{M\}held\-out tokens while all diagnostics use1​M1\\mathrm\{M\}held\-out tokens\.MethodBlock countNNRankrrAdded FLOPs \(%\)Val\. lossBaseline Transformer––0\.00003\.0009FullAttnResFull–0\.81312\.9752BlockAttnRes4–0\.11162\.9797BlockAttnRes8–0\.17542\.9778BlockAttnRes16–0\.30292\.9673FullP\-LR\-AttnResFull160\.89662\.9606FullP\-LR\-AttnResFull321\.38652\.9538FullP\-LR\-AttnResFull642\.36652\.9501BlockP\-LR\-AttnRes4321\.02482\.9488BlockP\-LR\-AttnRes8321\.05772\.9477BlockP\-LR\-AttnRes16321\.12352\.9543FullS\-LR\-AttnResFull160\.41292\.9762FullS\-LR\-AttnResFull320\.41932\.9682FullS\-LR\-AttnResFull640\.43202\.9638FullS\-LR\-AttnResFull1280\.45742\.9630FullS\-LR\-AttnResFull2560\.50822\.9626FullS\-LR\-AttnResFull5120\.60992\.9617BlockS\-LR\-AttnRes4640\.05932\.9533BlockS\-LR\-AttnRes8640\.09322\.9480BlockS\-LR\-AttnRes16640\.16092\.9494

### 4\.2Validation–FLOPs tradeoff

Table[1](https://arxiv.org/html/2607.09694#S4.T1)shows that standardAttnResimproves over the baseline at all tested granularities, with the best standard result coming from the finer block settingN=16N=16, which reaches validation loss2\.96732\.9673at0\.3029%0\.3029\\%added FLOPs\. Full standardAttnResis more expensive at0\.8131%0\.8131\\%added FLOPs and reaches2\.97522\.9752validation loss, suggesting that simply exposing every previous sub\-layer output does not always result in the best validation loss and may instead introduce source\-count noise\. We analyze this effect further in Section[4\.7](https://arxiv.org/html/2607.09694#S4.SS7)\.

Projected low\-rank routing gives the strongest validation losses among the reported variants\. The best row is BlockP\-LR\-AttnReswithN=8N=8andr=32r=32, which reaches2\.94772\.9477validation loss at1\.0577%1\.0577\\%added FLOPs\. The nearbyN=4,r=32N=4,r=32projected block model reaches2\.94882\.9488, while the full projected model improves as rank increases fromr=16r=16tor=64r=64\. This supports the main design hypothesis: a learned low\-dimensional routing descriptor can be more useful than reusing the full residual value as the key\.

The sliced models show that learned projected keys are not strictly necessary for a strong result\. FullS\-LR\-AttnResimproves steadily asrrgrows, from2\.97622\.9762atr=16r=16to2\.96172\.9617atr=512r=512, while still trailing fullP\-LR\-AttnRes\. AlthoughS\-LR\-AttnResbecomes exactly equivalent toAttnReswhenr=dr=d, the results show no evidence of degradation up tor=d/2r=d/2\. Performance continues to improve, albeit incrementally, across the tested ranks, suggesting that any negative effect of value\-as\-key routing appears only near the full\-width limit\. More strikingly, block sliced routing is highly competitive: BlockS\-LR\-AttnReswithN=8,r=64N=8,r=64reaches2\.94802\.9480validation loss at only0\.0932%0\.0932\\%added FLOPs, nearly matching the best projected result while avoiding the projected\-key path\. This makes sliced block routing the best Pareto point in terms of validation loss and added FLOPs\.

### 4\.3Block granularity

Block mode trades off the flexibility of full depth\-wise access against the memory and communication savings of compressed block representations\. In our setting,N=4N=4is a coarse block representation,N=8N=8is a medium\-grained representation, andN=16N=16is a finer\-grained representation closer to fullAttnRes\.

The results suggest that block granularity is not merely an approximation to full depth attention\. For standardAttnRes,N=16N=16outperforms both full and coarser block modes\. For low\-rank routing,N=8N=8is best, whileN=4N=4andN=16N=16remain very close\. One possible explanation is that block summaries act as a useful inductive bias and regularizer: they reduce the number of depth sources while preserving coarse stage information, which can make routing easier and less noisy\.

### 4\.4Why depth routing requires fewer dimensions than token routing

The rank sweep suggests that depth\-wise residual routing is a much lower\-dimensional discrimination problem than token attention\. In the main model, there areL=24L=24Transformer blocks and2​L=482L=48residual\-writing sub\-layers, so full depth routing chooses among at most4949sources including the embedding\. Block mode reduces this number further\. By contrast, token attention at context length20482048chooses among up to20482048token positions per layer and head, and its keys must represent rich contextual content, position, syntax, semantics, and long\-range dependencies\. A depth\-routing key only needs to help a static residual\-site query select among a small ordered set of previous writes\.

The empirical results are consistent with this view\. FullS\-LR\-AttnReswith onlyr=16r=16already nearly matches full standardAttnRes, andr=32r=32surpasses it\. FullP\-LR\-AttnResimproves substantially byr=32r=32, and blockP\-LR\-AttnResreaches the best validation loss with onlyr=32r=32\. The strongest sliced block result usesr=64r=64and nearly matches the best projected result\. These outcomes suggest that depth\-routing keys do not need to preserve the full token\-level representation\.

### 4\.5Effective number of sources

We next analyze how many residual sources are effectively used by each depth\-wise attention distribution\. For a residual read sitettwith source set𝒮t\\mathcal\{S\}\_\{t\}, let𝜶t​\(x\)\\bm\{\\alpha\}\_\{t\}\(x\)denote the source\-attention distribution for token positionxx, wherexxranges over the batch and sequence positions used in the probe\. We define the effective number of sources as the entropy perplexity of this distribution:

Neff​\(t,x\)=exp⁡\(H​\(𝜶t​\(x\)\)\),H​\(𝜶t​\(x\)\)=−∑i∈𝒮tαt,i​\(x\)​log⁡αt,i​\(x\)\.N\_\{\\mathrm\{eff\}\}\(t,x\)=\\exp\\left\(H\(\\bm\{\\alpha\}\_\{t\}\(x\)\)\\right\),\\qquad H\(\\bm\{\\alpha\}\_\{t\}\(x\)\)=\-\\sum\_\{i\\in\\mathcal\{S\}\_\{t\}\}\\alpha\_\{t,i\}\(x\)\\log\\alpha\_\{t,i\}\(x\)\.\(16\)This quantity equals11when all attention mass is placed on one source and equalskkwhen attention is uniform overkksources\. Thus, it can be interpreted as the number of sources actively used by a residual read\.

![Refer to caption](https://arxiv.org/html/2607.09694v1/x2.png)

![Refer to caption](https://arxiv.org/html/2607.09694v1/x3.png)

Figure 2:Effective number of residual sources across residual read sites\. Left: comparison between standardAttnRes, full low\-rank routing, and block low\-rank routing\. Right: rank sweep for fullS\-LR\-AttnResand fullP\-LR\-AttnRes\. Higher values indicate broader residual aggregation; lower values indicate sharper source selection\.The rank sweep in Figure[2](https://arxiv.org/html/2607.09694#S4.F2)shows that higher\-dimensional routing keys can support sharper source selection\. Asrrincreases, the model has more degrees of freedom to separate previous residual writes in the routing space\. This generally lowers the effective number of sources at later read sites, meaning that probability mass is concentrated on fewer sources\. In this sense, larger routing dimensions provide greater discriminative power for depth\-wise selection\.

However, greater discriminative power does not automatically imply better validation performance\. Standard fullAttnResuses the entiredd\-dimensional residual value as its routing key, so it gives the depth softmax the highest\-dimensional descriptor among the tested variants\. Yet fullAttnResis not the best\-performing method in Table[1](https://arxiv.org/html/2607.09694#S4.T1)\. Moreover, its effective\-source curve is more unsteady across residual read sites than the smoother low\-rank curves\. This suggests that full\-dimensional value\-as\-key routing can be overly sensitive to content variation due to the output being used as both the value and the key\.

These results support the low\-rank routing interpretation\. Depth routing needs enough dimensions to distinguish useful residual sources, sub\-layer types, and coarse depth stages, but it does not necessarily benefit from using the entire residual width as the key\. A very high\-dimensional key can make source selection sharper, but it can also make the routing distribution noisier and more coupled to incidental value\-space features\. Low\-rank keys act as a useful bottleneck: they preserve enough discriminative power for source selection while filtering out unstable high\-dimensional variation\. This helps explain whyP\-LR\-AttnResandS\-LR\-AttnRescan outperform standardAttnResdespite using far fewer routing dimensions\.

### 4\.6Mean attention heatmaps

To complement the entropy\-based effective\-source analysis, we also visualize the mean depth\-wise attention weights directly\. For each residual read site, we average the source\-attention distribution over validation tokens and plot the resulting source\-by\-depth pattern\. Figure[3](https://arxiv.org/html/2607.09694#S4.F3)shows fullS\-LR\-AttnResat several ranks in the first row and block variants withN=8N=8in the second row\.

![Refer to caption](https://arxiv.org/html/2607.09694v1/x4.png)Figure 3:Mean depth\-wise attention heatmaps\. Top row: fullS\-LR\-AttnReswithr=16r=16,r=128r=128, andr=512r=512\. Bottom row: standardAttnRes,P\-LR\-AttnRes, andS\-LR\-AttnResin block mode withN=8N=8\. Values are averaged over validation tokens and clipped at the99\.599\.5th percentile for readability\. The two rows use separate color scales, so the comparison should focus on the structure and smoothness of each attention pattern rather than absolute color intensity across rows\.The fullS\-LR\-AttnResheatmaps show a clear rank\-dependent smoothness pattern\. At very small rank, especiallyr=16r=16, the attention map is comparatively smooth: probability mass changes gradually across residual read sites, and the model tends to form broad depth preferences rather than highly source\-specific peaks\. With only a small routing key, the model can distinguish coarse source groups and broad depth stages, but it has fewer degrees of freedom to form irregular token\-dependent source preferences\.

As the rank increases tor=128r=128andr=512r=512, the heatmaps become visibly less smooth\. Checkerboard\-like patterns, localized spikes, and sharper structures appear\. This indicates that larger routing keys provide more discriminative power over depth sources\. The same trend also matches the effective\-source curves in Figure[2](https://arxiv.org/html/2607.09694#S4.F2)\.

The block heatmaps show a complementary effect\. Standard blockAttnReswithN=8N=8has sharp block\-boundary structure and several abrupt transitions, indicating strong source selection but also relatively coarse and sometimes saturated routing\. The projected low\-rank block model withN=8,r=32N=8,r=32is smoother and more balanced across source slots, which suggests that the learned low\-rank key space filters out some of the noise present in full\-dimensional value\-as\-key routing\. The sliced block model withN=8,r=64N=8,r=64lies between these two extremes: it retains strong early\-source and block\-stage preferences while avoiding the most irregular full\-dimensional patterns\.

### 4\.7Source count sensitivity in full depth routing

The number of depth sources is itself an important part of the routing problem\. In our main setting, the model hasL=24L=24Transformer blocks and two residual\-writing sub\-layers per block, so full depth routing attends over at most2​L\+1=492L\+1=49sources including the embedding\. This is already large enough that full standardAttnResis worse than the best blockAttnResresult, and it is also worse than the best low\-rank block variants\. Thus, even when full routing improves over the baseline, exposing every sub\-layer output is not always optimal: many sources can be weakly useful, redundant, or noisy, and block summaries can act as a regularizer by reducing the candidate set\.

This source\-count view also helps explain the negative full\-AttnResresult reported by Delta Attention Residuals\(Luoet al\.,[2026a](https://arxiv.org/html/2607.09694#bib.bib12)\)\. Their deeper settings test models withL=36L=36Transformer blocks\. Under the same per\-sub\-layer counting used here, full routing then exposes2​L\+1=732L\+1=73sources including the embedding\. This is a much noisier selection problem than the4949\-source setting tested in this paper\. With so many candidate residual sources, full attention over depth can diffuse probability mass across redundant outputs or overfit incidental source\-level variation, making the full variant worse than the baseline in those experiments\.

By contrast, the originalAttnResablations used a smaller source regime\. A 16\-block Transformer corresponds to roughly2⋅16\+1=332\\cdot 16\+1=33full depth sources including the embedding, so full routing has fewer candidates and is less exposed to source\-count noise\(Teamet al\.,[2026](https://arxiv.org/html/2607.09694#bib.bib4)\)\. The three regimes are therefore consistent: around3333sources, fullAttnRescan remain clean and effective; around4949sources, it still improves over the baseline but becomes worse than block routing; around7373sources, the full source set can become noisy enough that fullAttnResfalls below the baseline\.

### 4\.8Gradient norm stability

Table 2:Average gradient norm for the baseline, fullAttnRes, and full low\-rank variants\. Lower values indicate smaller average gradient scale under the same training recipe\.We also compare average gradient norms to test whether low\-rank routing changes the optimization dynamics of depth\-wise residual attention\. The baseline Transformer has average gradient norm0\.97750\.9775, while fullAttnResincreases this slightly to1\.00301\.0030\. This is consistent with the value\-as\-key tension: standardAttnResuses the samedd\-dimensional tensor for residual content and routing, so the routing path can introduce additional competition on the same representation used by downstream layers\.

The projected variants show the clearest stability benefit\. Relative to full\-dimensionalAttnRes, moving to projected low\-rank keys sharply reduces the average gradient norm, and all testedP\-LR\-AttnResranks remain well below the baseline\. Within the projected sweep, increasing the dedicated key rank fromr=16r=16tor=64r=64further lowers the norm from0\.81660\.8166to0\.76240\.7624\. We interpret this trend as reduced routing difficulty\. InP\-LR\-AttnRes, the routing key is a separate learned representation, so increasingrrgives the depth\-routing mechanism more freedom with which to distinguish sources\. The model therefore does not need to compress the source\-selection problem into an overly small key space, nor does it need to repurpose the residual\-value coordinates for routing\. As the dedicated routing space grows, source selection becomes easier and the gradient scale decreases\.

S\-LR\-AttnResshows the opposite trend because increasingrrdoes not add a separate routing representation\. Instead, it makes the routing key use a larger slice of the same residual value vector\. Apart from the nearly tiedr=16r=16andr=32r=32rows, the average gradient norm rises as rank increases:0\.81230\.8123atr=64r=64,0\.85770\.8577atr=128r=128,0\.87570\.8757atr=256r=256, and0\.90840\.9084atr=512r=512\. Thus, increasing the sliced rank gives the softmax more value coordinates to use for source selection, but those coordinates are still part of the residual representation itself\. Larger sliced keys therefore reintroduce more value/key coupling: more of the same tensor must simultaneously carry downstream content and participate in routing competition\.

Together, these results suggest that the effect of rank depends on whether the added dimensions are dedicated routing dimensions or reused value dimensions\. InP\-LR\-AttnRes, largerrrreduces routing difficulty because the model receives more separate key capacity\. InS\-LR\-AttnRes, largerrrincreases representational tension because routing consumes a larger fraction of the residual value stream\.

## 5Conclusion

We introducedLR\-AttnRes, a family of attention\-residual mechanisms that keeps residual values full\-dimensional while routing over low\-dimensional keys\. This separates the representation used to carry information from the descriptor used to select among depth sources, making routing\-key dimensionality an explicit design choice rather than an implicit consequence of the hidden width\. Across both projected and sliced variants, low\-rank routing improves the validation–FLOPs tradeoff over standardAttnRes:P\-LR\-AttnResachieves the best validation loss by learning dedicated routing keys, whileS\-LR\-AttnResremains highly competitive without adding a key\-projection path\. The routing analyses further suggest that cross\-layer source selection is a structured, low\-source\-count problem that does not require full\-width value\-as\-key comparisons\.

## References

- T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)FlashAttention: fast and memory\-efficient exact attention with io\-awareness\.InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 \- December 9, 2022,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html)Cited by:[§4\.1](https://arxiv.org/html/2607.09694#S4.SS1.SSS0.Px1.p1.5)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.In2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27\-30, 2016,pp\. 770–778\.External Links:[Link](https://doi.org/10.1109/CVPR.2016.90),[Document](https://dx.doi.org/10.1109/CVPR.2016.90)Cited by:[§1](https://arxiv.org/html/2607.09694#S1.p1.1)\.
- A\. Henry, P\. R\. Dachapally, S\. S\. Pawar, and Y\. Chen \(2020\)Query\-key normalization for transformers\.InFindings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16\-20 November 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Findings of ACL,pp\. 4246–4253\.External Links:[Link](https://doi.org/10.18653/v1/2020.findings-emnlp.379),[Document](https://dx.doi.org/10.18653/V1/2020.FINDINGS-EMNLP.379)Cited by:[§4\.1](https://arxiv.org/html/2607.09694#S4.SS1.SSS0.Px1.p1.5)\.
- G\. Huang, Z\. Liu, L\. van der Maaten, and K\. Q\. Weinberger \(2017\)Densely connected convolutional networks\.In2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21\-26, 2017,pp\. 2261–2269\.External Links:[Link](https://doi.org/10.1109/CVPR.2017.243),[Document](https://dx.doi.org/10.1109/CVPR.2017.243)Cited by:[§2](https://arxiv.org/html/2607.09694#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. Bernstein \(2024\)Muon: an optimizer for hidden layers in neural networks\.External Links:[Link](https://kellerjordan.github.io/posts/muon/)Cited by:[§4\.1](https://arxiv.org/html/2607.09694#S4.SS1.SSS0.Px2.p1.9)\.
- D\. P\. Kingma and J\. Ba \(2015\)Adam: A method for stochastic optimization\.In3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7\-9, 2015, Conference Track Proceedings,Y\. Bengio and Y\. LeCun \(Eds\.\),External Links:[Link](http://arxiv.org/abs/1412.6980)Cited by:[§4\.1](https://arxiv.org/html/2607.09694#S4.SS1.SSS0.Px2.p1.9)\.
- H\. Li and X\. Huang \(2025\)Enhancing layer attention efficiency through pruning redundant retrievals\.CoRRabs/2503\.06473\.External Links:[Link](https://doi.org/10.48550/arXiv.2503.06473),[Document](https://dx.doi.org/10.48550/ARXIV.2503.06473),2503\.06473Cited by:[§2](https://arxiv.org/html/2607.09694#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Luo, Z\. Cai, and J\. Hu \(2026a\)Delta attention residuals\.CoRRabs/2605\.18855\.External Links:[Link](https://doi.org/10.48550/arXiv.2605.18855),[Document](https://dx.doi.org/10.48550/ARXIV.2605.18855),2605\.18855Cited by:[§2](https://arxiv.org/html/2607.09694#S2.SS0.SSS0.Px2.p1.1),[§4\.7](https://arxiv.org/html/2607.09694#S4.SS7.p2.3)\.
- H\. Luo, H\. Dai, S\. Zhang, X\. Chen, E\. H\. Jiang, Y\. Li, J\. Huang, C\. Qiu, C\. Xu, Z\. Pan, H\. Zhang, B\. Wang, and Y\. Chen \(2026b\)Attention sinks and outliers in attention residuals\.CoRRabs/2605\.17887\.External Links:[Link](https://doi.org/10.48550/arXiv.2605.17887),[Document](https://dx.doi.org/10.48550/ARXIV.2605.17887),2605\.17887Cited by:[§2](https://arxiv.org/html/2607.09694#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Menghani, R\. Kumar, and S\. Kumar \(2025\)LAuReL: learned augmented residual layer\.InForty\-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13\-19, 2025,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267\.External Links:[Link](https://proceedings.mlr.press/v267/menghani25a.html)Cited by:[§2](https://arxiv.org/html/2607.09694#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Pagliardini, A\. Mohtashami, F\. Fleuret, and M\. Jaggi \(2024\)DenseFormer: enhancing information flow in transformers via depth weighted averaging\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/f67449c7ab72f441d3a713b046c6818c-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2607.09694#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Su, M\. H\. M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu \(2024\)RoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.External Links:[Link](https://doi.org/10.1016/j.neucom.2023.127063),[Document](https://dx.doi.org/10.1016/J.NEUCOM.2023.127063)Cited by:[§4\.1](https://arxiv.org/html/2607.09694#S4.SS1.SSS0.Px1.p1.5)\.
- K\. Team, G\. Chen, Y\. Zhang, J\. Su, W\. Xu, S\. Pan, Y\. Wang, Y\. Wang, G\. Chen, B\. Yin, Y\. Chen, J\. Yan, M\. Wei, Y\. Zhang, F\. Meng, C\. Hong, X\. Xie, S\. Liu, E\. Lu, Y\. Tai, Y\. Chen, X\. Men, H\. Guo, Y\. Charles, H\. Lu, L\. Sui, J\. Zhu, Z\. Zhou, W\. He, W\. Huang, X\. Xu, Y\. Wang, G\. Lai, Y\. Du, Y\. Wu, Z\. Yang, and X\. Zhou \(2026\)Attention residuals\.External Links:2603\.15031,[Link](https://arxiv.org/abs/2603.15031)Cited by:[§1](https://arxiv.org/html/2607.09694#S1.p2.1),[§2](https://arxiv.org/html/2607.09694#S2.SS0.SSS0.Px1.p2.1),[§4\.7](https://arxiv.org/html/2607.09694#S4.SS7.p3.4)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4\-9, 2017, Long Beach, CA, USA,I\. Guyon, U\. von Luxburg, S\. Bengio, H\. M\. Wallach, R\. Fergus, S\. V\. N\. Vishwanathan, and R\. Garnett \(Eds\.\),pp\. 5998–6008\.External Links:[Link](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)Cited by:[§1](https://arxiv.org/html/2607.09694#S1.p1.1)\.
- Y\. Wang, Z\. Fu, J\. Cai, P\. Tang, H\. Lyu, Y\. Fang, Z\. Zheng, J\. Zhou, G\. Zeng, C\. Xiao, X\. Han, and Z\. Liu \(2025\)Ultra\-fineweb: efficient data filtering and verification for high\-quality LLM training data\.CoRRabs/2505\.05427\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.05427),[Document](https://dx.doi.org/10.48550/ARXIV.2505.05427),2505\.05427Cited by:[§4\.1](https://arxiv.org/html/2607.09694#S4.SS1.SSS0.Px2.p1.9)\.
- D\. Xiao, Q\. Meng, S\. Li, and X\. Yuan \(2025\)MUDDFormer: breaking residual bottlenecks in transformers via multiway dynamic dense connections\.InForty\-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13\-19, 2025,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267\.External Links:[Link](https://proceedings.mlr.press/v267/xiao25d.html)Cited by:[§2](https://arxiv.org/html/2607.09694#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Xie, Y\. Wei, H\. Cao, C\. Zhao, C\. Deng, J\. Li, D\. Dai, H\. Gao, J\. Chang, L\. Zhao, S\. Zhou, Z\. Xu, Z\. Zhang, W\. Zeng, S\. Hu, Y\. Wang, J\. Yuan, L\. Wang, and W\. Liang \(2025\)MHC: manifold\-constrained hyper\-connections\.CoRRabs/2512\.24880\.External Links:[Link](https://doi.org/10.48550/arXiv.2512.24880),[Document](https://dx.doi.org/10.48550/ARXIV.2512.24880),2512\.24880Cited by:[§2](https://arxiv.org/html/2607.09694#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Xiong, Y\. Yang, D\. He, K\. Zheng, S\. Zheng, C\. Xing, H\. Zhang, Y\. Lan, L\. Wang, and T\. Liu \(2020\)On layer normalization in the transformer architecture\.InProceedings of the 37th International Conference on Machine Learning,ICML’20\.Cited by:[§1](https://arxiv.org/html/2607.09694#S1.p1.1)\.
- D\. Zhu, H\. Huang, Z\. Huang, Y\. Zeng, Y\. Mao, B\. Wu, Q\. Min, and X\. Zhou \(2025\)Hyper\-connections\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=9FqARW7dwB)Cited by:[§2](https://arxiv.org/html/2607.09694#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AFLOPs, activation memory, and parameter accounting

This appendix gives the detailed accounting for standardAttnRes, projected low\-rank keys, and sliced low\-rank keys\. The main distinction is between two types of cost\. The*depth\-wise residual kernel*is the attention\-over\-depth computation itself: it scores retained sources and mixes their values\. The*projected\-key path*is the extra work needed only byP\-LR\-AttnResto produce learned low\-rank keys\.

Let𝒯\\mathcal\{T\}be the set of residual read sites and let

Ssrc=∑t∈𝒯nt,nt=\|𝒮t\|\.S\_\{\\mathrm\{src\}\}=\\sum\_\{t\\in\\mathcal\{T\}\}n\_\{t\},\\qquad n\_\{t\}=\|\\mathcal\{S\}\_\{t\}\|\.\(17\)The source\-count termSsrcS\_\{\\mathrm\{src\}\}depends on whether the method uses full or block sources\.

### A\.1Depth\-wise residual kernel FLOPs

OrdinaryAttnResuses the samedd\-dimensional tensor as both key and value\. At residual sitett, the score computation costsnt​dn\_\{t\}dmultiply\-adds and the weighted value mixture costs anothernt​dn\_\{t\}dmultiply\-adds:

CAttnRes​\(t\)=nt​d\+nt​d=2​nt​d\.C\_\{\\mathrm\{AttnRes\}\}\(t\)=n\_\{t\}d\+n\_\{t\}d=2n\_\{t\}d\.\(18\)Summing over residual read sites gives

CAttnRes=2​d​Ssrc\.C\_\{\\mathrm\{AttnRes\}\}=2dS\_\{\\mathrm\{src\}\}\.\(19\)
BothP\-LR\-AttnResandS\-LR\-AttnResuse anrr\-dimensional key and a fulldd\-dimensional value\. Therefore the depth\-wise residual kernel cost is

CLR,kernel​\(t\)=nt​r\+nt​d=nt​\(d\+r\),C\_\{\\mathrm\{LR,kernel\}\}\(t\)=n\_\{t\}r\+n\_\{t\}d=n\_\{t\}\(d\+r\),\(20\)and

CLR,kernel=\(d\+r\)​Ssrc\.C\_\{\\mathrm\{LR,kernel\}\}=\(d\+r\)S\_\{\\mathrm\{src\}\}\.\(21\)The percentage reduction in the depth\-wise residual kernel relative to ordinaryAttnResis

%ΔCkernel\\displaystyle\\%\\,\\Delta C\_\{\\mathrm\{kernel\}\}=100​\(1−CLR,kernelCAttnRes\)\\displaystyle=100\\left\(1\-\\frac\{C\_\{\\mathrm\{LR,kernel\}\}\}\{C\_\{\\mathrm\{AttnRes\}\}\}\\right\)\(22\)=100​\(1−\(d\+r\)​Ssrc2​d​Ssrc\)\\displaystyle=100\\left\(1\-\\frac\{\(d\+r\)S\_\{\\mathrm\{src\}\}\}\{2dS\_\{\\mathrm\{src\}\}\}\\right\)\(23\)=100​\(d−r2​d\)\.\\displaystyle=100\\left\(\\frac\{d\-r\}\{2d\}\\right\)\.\(24\)The source\-count term cancels, so the same kernel percentage applies to full and block source sets\. Softmax and RMSNorm terms are lower\-order for this comparison\. Including key normalization strengthens the low\-rank advantage because the normalized key width changes fromddtorr\.

### A\.2Projected\-key FLOPs

P\-LR\-AttnResproduces a learned low\-rank key by addingrrrows to output projections that already exist\. Letmim\_\{i\}be the input width of the output projection for residual\-writing sub\-layerii\. The extra projected\-key cost is

CP,key​\-​proj=r​∑i=12​Lmi\+r​d,C\_\{\\mathrm\{P,key\\text\{\-\}proj\}\}=r\\sum\_\{i=1\}^\{2L\}m\_\{i\}\+rd,\(25\)where the finalr​drdterm is the embedding key projection\. For a Transformer block with one attention output projection of input widthddand one SwiGLU output projection of input widthmm,

∑i=12​Lmi=L​\(d\+m\),\\sum\_\{i=1\}^\{2L\}m\_\{i\}=L\(d\+m\),\(26\)so

CP,key​\-​proj=r​\[L​\(d\+m\)\+d\]\.C\_\{\\mathrm\{P,key\\text\{\-\}proj\}\}=r\\bigl\[L\(d\+m\)\+d\\bigr\]\.\(27\)
Thus the total residual\-side cost ofP\-LR\-AttnResis

CP,total=\(d\+r\)​Ssrc\+r​\[L​\(d\+m\)\+d\]\.C\_\{\\mathrm\{P,total\}\}=\(d\+r\)S\_\{\\mathrm\{src\}\}\+r\\bigl\[L\(d\+m\)\+d\\bigr\]\.\(28\)Relative to ordinaryAttnRes, the net percentage change is

%ΔCP,total=100\(1−\(d\+r\)​Ssrc\+r​\[L​\(d\+m\)\+d\]2​d​Ssrc\)\.\\%\\,\\Delta C\_\{\\mathrm\{P,total\}\}=100\\left\(1\-\\frac\{\(d\+r\)S\_\{\\mathrm\{src\}\}\+r\\bigl\[L\(d\+m\)\+d\\bigr\]\}\{2dS\_\{\\mathrm\{src\}\}\}\\right\)\.\(29\)Therefore,P\-LR\-AttnResreduces the depth\-wise residual kernel cost, but it reduces*total*residual\-side FLOPs only when

r​\[L​\(d\+m\)\+d\]<\(d−r\)​Ssrc\.r\\bigl\[L\(d\+m\)\+d\\bigr\]<\(d\-r\)S\_\{\\mathrm\{src\}\}\.\(30\)This condition can fail, especially in block mode whereSsrcS\_\{\\mathrm\{src\}\}is small\. In that case,P\-LR\-AttnResimproves validation loss but does not reduce total added FLOPs once projected\-key computation is included\.

### A\.3Sliced\-key FLOPs

S\-LR\-AttnResobtains its key by slicing the lastrrdimensions of the already\-computed residual value:

𝒌iS=tailr⁡\(𝒐i\)\.\{\\bm\{k\}\}\_\{i\}^\{\\mathrm\{S\}\}=\\operatorname\{tail\}\_\{r\}\(\{\\bm\{o\}\}\_\{i\}\)\.\(31\)It therefore has no auxiliary key projection:

CS,key​\-​proj=0\.C\_\{\\mathrm\{S,key\\text\{\-\}proj\}\}=0\.\(32\)Its total residual\-side cost is just the low\-rank depth\-wise residual kernel:

CS,total=\(d\+r\)​Ssrc\.C\_\{\\mathrm\{S,total\}\}=\(d\+r\)S\_\{\\mathrm\{src\}\}\.\(33\)ThusS\-LR\-AttnResreduces total residual\-side FLOPs relative to ordinaryAttnReswheneverr<dr<d:

%ΔCS,total=100\(1−\(d\+r\)​Ssrc2​d​Ssrc\)=100\(d−r2​d\)\.\\%\\,\\Delta C\_\{\\mathrm\{S,total\}\}=100\\left\(1\-\\frac\{\(d\+r\)S\_\{\\mathrm\{src\}\}\}\{2dS\_\{\\mathrm\{src\}\}\}\\right\)=100\\left\(\\frac\{d\-r\}\{2d\}\\right\)\.\(34\)This is the main efficiency advantage of slicing\.P\-LR\-AttnResandS\-LR\-AttnReshave the same low\-rank depth\-wise kernel, but onlyS\-LR\-AttnResremoves the projected\-key path entirely\.

### A\.4Activation memory

LetKsrcK\_\{\\mathrm\{src\}\}be the number of retained depth sources\. For fullAttnRes, this is the number of previous residual\-writing sources\. ForBlock AttnRes, this is the number of retained block sources\. Assume a fused depth\-wise attention kernel, analogous to FlashAttention, so the attention\-weight vector is not materialized for backward\. If each activation element usesbbbytes, the dominant source\-cache activation memory for ordinaryAttnResis

AAttnRes=b​Ksrc​d\.A\_\{\\mathrm\{AttnRes\}\}=bK\_\{\\mathrm\{src\}\}d\.\(35\)
P\-LR\-AttnRescaches the full\-dimensional values and the projected low\-rank keys:

AP=b​Ksrc​\(d\+r\)\.A\_\{\\mathrm\{P\}\}=bK\_\{\\mathrm\{src\}\}\(d\+r\)\.\(36\)Therefore the retained source\-cache activation memory increases by

%ΔAP=100\(APAAttnRes−1\)=100rd\.\\%\\,\\Delta A\_\{\\mathrm\{P\}\}=100\\left\(\\frac\{A\_\{\\mathrm\{P\}\}\}\{A\_\{\\mathrm\{AttnRes\}\}\}\-1\\right\)=100\\frac\{r\}\{d\}\.\(37\)
S\-LR\-AttnRescan implement the key as a view into the lastrrcoordinates of the cached value\. In that implementation, it does not require a separate key cache:

AS=b​Ksrc​d=AAttnRes\.A\_\{\\mathrm\{S\}\}=bK\_\{\\mathrm\{src\}\}d=A\_\{\\mathrm\{AttnRes\}\}\.\(38\)Thus the dominant retained source\-cache activation increase is

%ΔAS=0\.\\%\\,\\Delta A\_\{\\mathrm\{S\}\}=0\.\(39\)An implementation may optionally cache normalized sliced keys, but this is not required by the method\.

### A\.5Parameters

The parameter count forP\-LR\-AttnResconsists of projected key rows and residual\-site queries\. For residual\-writing sub\-layerii, adding anrr\-dimensional key to an output projection with input widthmim\_\{i\}addsr​mirm\_\{i\}parameters\. The embedding key projection addsr​drdparameters\. The2​L2Lresidual read sites each have onerr\-dimensional static query, adding2​L​r2Lrparameters\. Thus

Δ​PP,key\\displaystyle\\Delta P\_\{\\mathrm\{P,key\}\}=r​∑i=12​Lmi\+r​d,\\displaystyle=r\\sum\_\{i=1\}^\{2L\}m\_\{i\}\+rd,\(40\)Δ​PP,query\\displaystyle\\Delta P\_\{\\mathrm\{P,query\}\}=2​L​r,\\displaystyle=2Lr,\(41\)Δ​PP\\displaystyle\\Delta P\_\{\\mathrm\{P\}\}=r​∑i=12​Lmi\+r​d\+2​L​r\.\\displaystyle=r\\sum\_\{i=1\}^\{2L\}m\_\{i\}\+rd\+2Lr\.\(42\)With one attention output projection of widthddand one SwiGLU output projection of widthmmper Transformer block,

Δ​PP=r​\[L​\(d\+m\)\+d\+2​L\]\.\\Delta P\_\{\\mathrm\{P\}\}=r\\bigl\[L\(d\+m\)\+d\+2L\\bigr\]\.\(43\)
S\-LR\-AttnReshas no key\-projection parameters\. It only needs the low\-rank residual\-site queries:

Δ​PS=2​L​r\.\\Delta P\_\{\\mathrm\{S\}\}=2Lr\.\(44\)
Relative to a base model withPbaseP\_\{\\mathrm\{base\}\}parameters, the parameter increases are

%ΔPP\\displaystyle\\%\\,\\Delta P\_\{\\mathrm\{P\}\}=100​r​\[L​\(d\+m\)\+d\+2​L\]Pbase,\\displaystyle=100\\frac\{r\\bigl\[L\(d\+m\)\+d\+2L\\bigr\]\}\{P\_\{\\mathrm\{base\}\}\},\(45\)%ΔPS\\displaystyle\\%\\,\\Delta P\_\{\\mathrm\{S\}\}=100​2​L​rPbase\.\\displaystyle=100\\frac\{2Lr\}\{P\_\{\\mathrm\{base\}\}\}\.\(46\)
If ordinaryAttnResis the reference instead of the no\-AttnResbaseline, then ordinaryAttnResalready contains2​L​d2Ldquery parameters under the same parameter\-free RMSNorm convention\. The parameter differences become

Δ​PP−vs−AttnRes\\displaystyle\\Delta P\_\{\\mathrm\{P\-vs\-AttnRes\}\}=r​\[L​\(d\+m\)\+d\+2​L\]−2​L​d,\\displaystyle=r\\bigl\[L\(d\+m\)\+d\+2L\\bigr\]\-2Ld,\(47\)Δ​PS−vs−AttnRes\\displaystyle\\Delta P\_\{\\mathrm\{S\-vs\-AttnRes\}\}=2​L​r−2​L​d=−2​L​\(d−r\)\.\\displaystyle=2Lr\-2Ld=\-2L\(d\-r\)\.\(48\)Thus sliced low\-rank routing uses fewer query parameters than full\-dimensionalAttnResand adds no key\-projection parameters\.

### A\.6Numerical example for the experimental model

For the main experimental model,d=1024d=1024,r=32r=32,L=24L=24, andm=2816m=2816\. The low\-rank depth\-wise residual kernel reduction is

100​\(1024−322⋅1024\)=48\.44%\.100\\left\(\\frac\{1024\-32\}\{2\\cdot 1024\}\\right\)=48\.44\\%\.\(49\)BothP\-LR\-AttnResandS\-LR\-AttnResobtain this kernel reduction\.

The projected\-key path inP\-LR\-AttnResadds

CP,key​\-​proj=32​\[24​\(1024\+2816\)\+1024\]=2,981,888C\_\{\\mathrm\{P,key\\text\{\-\}proj\}\}=32\\left\[24\(1024\+2816\)\+1024\\right\]=2\{,\}981\{,\}888\(50\)multiply\-adds per token\.S\-LR\-AttnResremoves this term\.

The projected\-key parameter increase is

Δ​PP=32​\[24​\(1024\+2816\)\+1024\+48\]=2,983,424,\\Delta P\_\{\\mathrm\{P\}\}=32\\left\[24\(1024\+2816\)\+1024\+48\\right\]=2\{,\}983\{,\}424,\(51\)while the sliced\-key parameter increase is only

Δ​PS=2⋅24⋅32=1,536\.\\Delta P\_\{\\mathrm\{S\}\}=2\\cdot 24\\cdot 32=1\{,\}536\.\(52\)
For retained source\-cache activations,P\-LR\-AttnResincreases the value\-plus\-key cache by

100⁤321024=3\.125%,100\\frac\{32\}\{1024\}=3\.125\\%,\(53\)whereasS\-LR\-AttnResadds no separate source\-key cache when implemented as a slice of the cached value\.

For a dense SwiGLU Transformer core with

Cbase=L​\(4​d2\+3​d​m\),C\_\{\\mathrm\{base\}\}=L\(4d^\{2\}\+3dm\),\(54\)the projected\-key parameter overhead, ignoring embeddings and output heads, is approximately

100⁤321024​1\+2816/10244\+3​\(2816/1024\)=0\.957%\.100\\frac\{32\}\{1024\}\\frac\{1\+2816/1024\}\{4\+3\(2816/1024\)\}=0\.957\\%\.\(55\)The exact projected\-key numerator above gives roughly the same scale, while the sliced\-key query\-only overhead is negligible\.

Under the added\-FLOP accounting used in the main results table, the projected\-key path contributes approximately0\.9673%0\.9673\\%of the non\-embedding Transformer core forr=32r=32\. Thus the full projectedr=32r=32row has total added FLOPs0\.4193%\+0\.9673%=1\.3866%0\.4193\\%\+0\.9673\\%=1\.3866\\%, while block projectedN=8,r=32N=8,r=32has total added FLOPs0\.0904%\+0\.9673%=1\.0577%0\.0904\\%\+0\.9673\\%=1\.0577\\%\.S\-LR\-AttnReshas no projected\-key path, so its total added FLOPs equal its depth\-kernel compute\.

Table 3:Residual\-side cost summary\.Ssrc=∑t∈𝒯\|𝒮t\|S\_\{\\mathrm\{src\}\}=\\sum\_\{t\\in\\mathcal\{T\}\}\|\\mathcal\{S\}\_\{t\}\|is the total number of source reads across residual sites, andKsrcK\_\{\\mathrm\{src\}\}is the number of retained source activations\.