CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry

# Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry
Source: [https://arxiv.org/html/2606.26538](https://arxiv.org/html/2606.26538)
###### Abstract

Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value\. We present two efficiency methods that exploit this asymmetry\. CascadeFormer tapers width with depth to match the uneven information flow across layers, achieving comparable perplexity to a uniform baseline at the same training budget while reducing latency by 8\.6% and increasing throughput by 9\.4%\. CascadeFlow Pruning removes layers using accumulated training gradients, with no post hoc analysis\. It outperforms standard heuristics on perplexity and rank\-stability and stays competitive on downstream accuracy\. To motivate these methods, we proposeGradient Fan\-in Asymmetry\(GFA\) as a structural account of why deeper layers contribute less\. In Pre\-LayerNorm residual stacks, the gradient at a layer is the sum of an identity path and all downstream functional paths, producing a gradient fan\-in that decays linearly with depth \(and quadratically under deep supervision\), yielding richer gradients for early layers and sparser ones for later layers\. We provide correlational and interventional evidence for GFA on models trained from scratch up to 1\.2B parameters\. Across Transformers and ResNets, accumulated training gradients follow the theoretical fan\-in and are associated with post hoc layer importance\. Two interventions point to structure rather than magnitude as the bottleneck: equalizing per\-layer gradient norms does not restore late\-layer value, while increasing downstream path counts via parameter\-shared repetition restores and elevates it\. Whether gradient magnitude proxies fan\-in beyond high\-rank regimes, and how these dynamics behave at the 100B\+ scale, remain open questions\.

## Introduction

The uniform scaling of transformer blocksVaswani et al\. \([2017](https://arxiv.org/html/2606.26538#bib.bib36)\), simply repeating identical layers to create deeper models, has been the driving principle behind the success of Large Language Models\([Radford et al\.,](https://arxiv.org/html/2606.26538#bib.bib26); Brown et al\.,[2020](https://arxiv.org/html/2606.26538#bib.bib1); Touvron et al\.,[2023](https://arxiv.org/html/2606.26538#bib.bib35)\)\. However, uniform stacking masks a functional asymmetry\. For instance, when we evaluate a pretrained LLaMA model on WikiText, its deeper layers exhibit high representational similarity, a key indicator of redundancy\(Gromov et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib8)\)\(Figure[1](https://arxiv.org/html/2606.26538#S1.F1)a\)\. This asymmetry is even more pronounced in architectures like LayerSkip\(Elhoushi et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib5)\), which skips later layers by exiting the network early, revealing their sharply declining functional contribution \(Figure[1](https://arxiv.org/html/2606.26538#S1.F1)b\)\.

The conventional explanation for this phenomenon points to attenuated gradients in deeper layersLi et al\. \([2025](https://arxiv.org/html/2606.26538#bib.bib19)\)\. The explanation is correct, but it may treat a symptom rather than the underlying factor\. We propose that the relevant quantity is not gradient magnitude but the compositional diversity of the gradient, a structural bottleneck we termGradient Fan\-in Asymmetry \(GFA\)\. Residual connections\(He et al\.,[2016a](https://arxiv.org/html/2606.26538#bib.bib12)\)turn the network into an implicit ensemble of paths of varying lengths\(Veit et al\.,[2016](https://arxiv.org/html/2606.26538#bib.bib37)\), so shallow layers receive gradient from all downstream blocks while the deepest layers, aggregating from few, receive a structurally simple, information\-poor gradient\.

We support the GFA hypothesis and demonstrate its utility through a sequence of empirical arguments, all on models trained from scratch up to 1\.2B parameters\. First, we observe a positive correlation between per\-layer gradient normsg¯i\\bar\{g\}\_\{i\}and eventual functional importanceΔ​ℳi\\Delta\\mathcal\{M\}\_\{i\}\. We then move beyond correlation with two interventional tests: one ablative, showing that artificially amplifying late\-layer gradient magnitude fails to restore late\-layer importance, and one constructive, showing that structurally increasing their path counts via layer repetition does restore it\. These interventions are consistent with a structural \(not magnitude\) bottleneck\. Finally, we translate the GFA account into two practical applications:CascadeFlow Pruning \(CFP\), an efficient method leveraging accumulated training gradients to outperform standard pruning heuristics, and theCascadeFormer, an architecture that tapers width with depth to align model capacity with the natural flow of compositional gradient diversity, improving inference efficiency at fixed training FLOPs\.

We reframe gradient magnitude as a proxy for a structural information imbalance, not as a cause to be fixed\. Our contributions are:

- •We design the CascadeFormer, an architecture that internalizes the GFA principle\. By tapering network width with depth to match the natural flow of compositional gradient diversity, it reduces latency and increases throughput over a uniform baseline with equal training FLOPs and comparable perplexity\.
- •We introduce CascadeFlow Pruning \(CFP\), an efficient method that leverages accumulated training gradients as a proxy for structural importance to prune layers, outperforming standard heuristics without requiring expensive post hoc analysis\.
- •We propose Gradient Fan\-in Asymmetry as a structural accountof layer redundancy and provide correlational and interventional evidence for it on models trained from scratch up to 1\.2B parameters\. Two complementary interventions, an ablative test \(artificially equalizing gradient norms fails to restore importance\) and a constructive one \(structurally increasing path counts via layer repetition succeeds\), are consistent with a bottleneck rooted in the gradient’s compositional complexity rather than its raw magnitude\. Whether magnitude proxies fan\-in beyond high\-rank regimes, and how these dynamics behave at the 100B\+ scale, remain open questions\.

![Refer to caption](https://arxiv.org/html/2606.26538v1/x1.png)Figure 1:Deeper Transformer layers show diminishing contributions\.Illustrative motivation from pretrained public checkpoints \(distinct from the from\-scratch models studied in the rest of the paper\)\. \(a\) In a pretrained LLaMA 13B model, representational similarity across layers increases with depth, signalling growing redundancy\. \(b\) A pretrained LayerSkip LLaMA 8B \(an early\-exit/deep\-supervised model, where the effect is pronounced by design\) makes the consequence explicit: layer importance, measured by Functional Importance \(Δ​ℳ\\Delta\\mathcal\{M\}\) upon its removal, is concentrated in the initial layers while the functional value of later layers collapses\.
## Related Work

##### Layer Redundancy for Model Efficiency\.

Deep networks exhibit layer redundancy that can be exploited for compression and faster inference\(Gromov et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib8); Sun et al\.,[2025](https://arxiv.org/html/2606.26538#bib.bib33); Chen et al\.,[2025](https://arxiv.org/html/2606.26538#bib.bib3)\)\. Structured pruning removes entire blocks with minor loss\(Chen et al\.,[2023](https://arxiv.org/html/2606.26538#bib.bib2); Frantar and Alistarh,[2023](https://arxiv.org/html/2606.26538#bib.bib7); Ma et al\.,[2023](https://arxiv.org/html/2606.26538#bib.bib22); Xia et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib39); Kim et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib16); He et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib14); Sun et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib32)\), while training\-time methods like LayerDrop\(Fan et al\.,[2020](https://arxiv.org/html/2606.26538#bib.bib6)\)and early\-exit/skip mechanisms\(Elhoushi et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib5); Xin et al\.,[2020](https://arxiv.org/html/2606.26538#bib.bib40); Liu et al\.,[2020](https://arxiv.org/html/2606.26538#bib.bib20); Zhao et al\.,[2025](https://arxiv.org/html/2606.26538#bib.bib45); Men et al\.,[2025](https://arxiv.org/html/2606.26538#bib.bib23)\)allow dynamic redundancy management\. These works document*that*redundancy exists and how to exploit it; the mechanistic accounts that do exist attribute it largely to gradient*magnitude*attenuation, whereas we propose a*compositional*\(fan\-in\) account\.

##### The Architectural Roots of Redundancy\.

Residual networks\(He et al\.,[2016a](https://arxiv.org/html/2606.26538#bib.bib12),[b](https://arxiv.org/html/2606.26538#bib.bib13)\)can be viewed as implicit ensembles of shorter pathsVeit et al\. \([2016](https://arxiv.org/html/2606.26538#bib.bib37)\), a property inherited by Transformers with Pre\-LayerNorm architectures\(Xiong et al\.,[2020](https://arxiv.org/html/2606.26538#bib.bib41); Touvron et al\.,[2023](https://arxiv.org/html/2606.26538#bib.bib35)\)\. This structure often causes deeper layers to contribute minimally, leading to redundancy\(Takase et al\.,[2023](https://arxiv.org/html/2606.26538#bib.bib34); Sun et al\.,[2025](https://arxiv.org/html/2606.26538#bib.bib33); Li et al\.,[2025](https://arxiv.org/html/2606.26538#bib.bib19)\)\. A complementary line of work shows how architectural and normalization choices regulate gradient propagation across depth\(Wang et al\.,[2022](https://arxiv.org/html/2606.26538#bib.bib38); Shleifer et al\.,[2021](https://arxiv.org/html/2606.26538#bib.bib28); Li et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib18)\)\. Our methods target the Pre\-LN setting\. Building on these results, we hypothesize that the depth\-wise allocation of gradient signal is structurally induced rather than driven by weight magnitude, and we provide correlational and interventional evidence for this\. We then exploit the effect to design more efficient architectures along both depth and width\. Observations that attention\-layer redundancy is inherent and stable across training stages\(He et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib14)\)point to a structural rather than incidental origin\.

##### Quantifying Layer Importance\.

Identifying redundant layers requires a reliable importance metric\. Magnitude and first/second\-order criteria provide strong baselines\(Han et al\.,[2015](https://arxiv.org/html/2606.26538#bib.bib11),[2016](https://arxiv.org/html/2606.26538#bib.bib10); Molchanov et al\.,[2017](https://arxiv.org/html/2606.26538#bib.bib24); Lee et al\.,[2019](https://arxiv.org/html/2606.26538#bib.bib17); Frantar and Alistarh,[2023](https://arxiv.org/html/2606.26538#bib.bib7)\)\. Other methods rely on measuring the effects of redundancy: output\-similarity \(e\.g\., cosine between adjacent layers\), correlates high similarity with low importance\(Gromov et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib8); Yang et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib43); Jiang et al\.,[2025](https://arxiv.org/html/2606.26538#bib.bib15); Chen et al\.,[2025](https://arxiv.org/html/2606.26538#bib.bib3);[Song et al\.,](https://arxiv.org/html/2606.26538#bib.bib31)\), while perturbation\-based metrics such as \(Δ\\DeltaPPL\) upon single layer removal directly quantify its functional contribution\(Kim et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib16)\)\. Gradient\-based signals have been used as local surrogates for importance, but typically in heuristic form \(e\.g\., saliencySmilkov et al\. \([2017](https://arxiv.org/html/2606.26538#bib.bib29)\); Selvaraju et al\. \([2020](https://arxiv.org/html/2606.26538#bib.bib27)\), Taylor criteria\(Yang et al\.,[2023](https://arxiv.org/html/2606.26538#bib.bib44); Ma et al\.,[2023](https://arxiv.org/html/2606.26538#bib.bib22)\)\)\. We propose that gradient dynamics do more than proxy importance: through a compositional fan\-in asymmetry, they may help account for the final functional hierarchy, and they predict it on the from\-scratch models \(up to 1\.2B parameters\) we study\. We provide correlational and interventional evidence for this\.

## Gradient Fan\-in Asymmetry

##### The Phenomenon\.

We propose Gradient Fan\-in Asymmetry \(GFA\), a structural imbalance in the composition of gradient signals within deep residual architectures, as an account of layer redundancy\. This asymmetry arises because shallow layers receive gradients aggregated across numerous downstream computational paths, while deep layers receive them from a progressively smaller set\. We hypothesize that this disparity in fan\-in shapes thecompositional diversityof the resulting gradient\. Under this account, deep\-layer gradients are compositionally simpler and thus information\-poorer, leading to less effective weight updates\. The limitation is structural, not one of magnitude\. An optimizer that only rescales gradients cannot supply the missing compositional information\.

##### GFA in Residual Networks\.

We define a Pre\-LN Transformer asxl\+1=xl\+Fl​\(xl\)x\_\{l\+1\}=x\_\{l\}\+F\_\{l\}\(x\_\{l\}\), whereFlF\_\{l\}represents the block’s complete transformation, including LayerNorm and sublayers\. The gradient at its input,gl≡∂ℒ/∂xlg\_\{l\}\\equiv\\partial\\mathcal\{L\}/\\partial x\_\{l\}, unrolls into a cumulative sum over all downstream blocks:

gl=gN\+∑k=lN−1JkT​gk\+1,g\_\{l\}\\;=\\;g\_\{N\}\\;\+\\;\\sum\_\{k=l\}^\{N\-1\}\{J\_\{k\}^\{T\}\}g\_\{k\+1\},\(1\)whereJkJ\_\{k\}is the Jacobian of thekk\-th block’s transformation\. This structure means the gradient at layerllaggregates signals from an identity path and all subsequent functional paths\. We term the number of these aggregated signals the gradient fan\-in,ϕl\\phi\_\{l\}, which decreases linearly with depth \(visualized by the solid paths in Fig\.[2](https://arxiv.org/html/2606.26538#S3.F2)\)\. A formal counting rule is provided in Appendix[B\.1](https://arxiv.org/html/2606.26538#A2.SS1)\.

##### Amplification via Deep Supervision\.

Architectures employing deep supervision, such as LayerSkip, amplify GFA\. By introducing auxiliary loss heads at intermediate layers, they create new gradient hierarchies that backpropagate to shallower layers\. This transforms the linear fan\-in disparity into a quadratic one \(illustrated by the additional dotted paths in Fig\.[2](https://arxiv.org/html/2606.26538#S3.F2)\), concentrating gradient information in the shallowest layers\. Appendix[B\.1](https://arxiv.org/html/2606.26538#A2.SS1)gives the full derivation\. Deep supervision therefore exacerbates the structural imbalance rather than solving it\.

##### Analysis and Prediction\.

Our fan\-in proxy is not the2N−l2^\{N\-l\}combinatorial paths of a full expansion\. Our proxy measures the number of distinct signal channels aggregated at a layer, not their information quality \(e\.g\., orthogonality\), which remains an empirical question\. The distinction motivates our central hypothesis:gradient norm may be a symptom rather than the underlying factor\. Under this account, a small gradient norm in a deep layer reflects its limited access to compositional information\. An optimizer like AdamWLoshchilov and Hutter \([2018](https://arxiv.org/html/2606.26538#bib.bib21)\)can rescale this gradient, but it cannot invent compositional information that is structurally absent\. From this account, we derive a testable prediction: an architectural change that structurally increases a layer’s downstream gradient fan\-in should increase its functional importance\. We test this prediction directly with virtual depth and use the account to motivate our methods\.

![Refer to caption](https://arxiv.org/html/2606.26538v1/x2.png)Figure 2:Gradient Fan\-in Asymmetry arises from a structural imbalance in gradient paths\.The unwrapped view shows that the gradient at any layerllis a sum over signals from an identity path and all subsequent functional paths \(Eq\.[1](https://arxiv.org/html/2606.26538#S3.E1)\)\. The number of contributing functional paths \(solid lines\) decreases linearly with depth, creating a compositional asymmetry\. Deep supervision \(dotted lines\) exacerbates this imbalance by introducing new gradient hierarchies from auxiliary losses, which transforms the fan\-in disparity from linear to quadratic\.

## Empirical Evidence and Applications

We test Gradient Fan\-in Asymmetry in three stages\. We first show that gradient flow is structurally skewed towards early layers and that this training signal predicts the final functional hierarchy\. We then perform two interventions that separate magnitude from information content and are consistent with structure as the bottleneck\. Finally, we translate the GFA account into two applications, CascadeFlow Pruning and CascadeFormer, that improve efficiency at fixed training cost\.

### 4\.1Setup

##### Models and architectures

We evaluate three residual families\. For language modelling we train a sixteen\-layer, approximately 1\.2B\-parameter, Llama base Transformer\(Touvron et al\.,[2023](https://arxiv.org/html/2606.26538#bib.bib35)\)referred to as Vanilla and a LayerSkip variant\(Elhoushi et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib5)\)\. For vision we train a ResNet\-50\(He et al\.,[2016a](https://arxiv.org/html/2606.26538#bib.bib12)\)\. To encode the GFA prior we modify the Vanilla architecture to create CascadeFormer which tapers width with depth to align capacity with the decay in compositional gradient diversity\. All models are trained from scratch\.

##### Datasets and tasks

Language models are trained on a seven billion token subset of Dolma\(Soldaini et al\.,[2024](https://arxiv.org/html/2606.26538#bib.bib30)\)for next token prediction\. ResNet\-50 is trained on ImageNet\-1K\(Deng et al\.,[2009](https://arxiv.org/html/2606.26538#bib.bib4)\)\. Training hyperparameters and optimizer settings are in Appendix[A](https://arxiv.org/html/2606.26538#A1)\. For our primary architectural comparison, we train the proposed CascadeFormerA2, the full baseline, and a 15\-layer Baseline, using three different random seeds\.

### 4\.2Quantifying Gradient Flow and Functional Importance

To empirically test our hypothesis, we require metrics that can directly link training dynamics with the final functional hierarchy of the model\.

##### Accumulated Gradient Share \(g¯i\\bar\{g\}\_\{i\}\)\.

To capture a layer’s overall contribution during training, we accumulate the L2 norm of the gradients with respect to its parameters,θi\\theta\_\{i\}, overTTtraining steps\. This accumulated value is then normalized by the total sum from allNNlayers to yield the relative gradient share,g¯i\\bar\{g\}\_\{i\}:

g¯i=∑t=1T∥∇θiℒt∥2∑j=1N∑t=1T∥∇θjℒt∥2\.\\bar\{g\}\_\{i\}=\\frac\{\\sum\_\{t=1\}^\{T\}\\lVert\\nabla\_\{\\theta\_\{i\}\}\\mathcal\{L\}\_\{t\}\\rVert\_\{2\}\}\{\\sum\_\{j=1\}^\{N\}\\sum\_\{t=1\}^\{T\}\\lVert\\nabla\_\{\\theta\_\{j\}\}\\mathcal\{L\}\_\{t\}\\rVert\_\{2\}\}\.\(2\)This metric serves as a direct, data\-driven proxy for the structural information flow predicted by GFA\.

##### Functional Importance \(Δ​ℳi\\Delta\\mathcal\{M\}\_\{i\}\)\.

We quantify a layer’s functional importance by measuring the performance degradation when its contribution is removed\. This is achieved by ablating layerii, bypassing its computational block while preserving the residual path to the subsequent layer\. The functional importance,Δ​ℳi\\Delta\\mathcal\{M\}\_\{i\}is the absolute degradation in the task metricMMresulting from this ablation\. For language models, this is the increase in perplexity \(Δ\\DeltaPPL\), and for vision models, the change in top\-1 accuracy \(Δ\\DeltaAcc\)\. A largerΔ​ℳi\\Delta\\mathcal\{M\}\_\{i\}signifies greater functional importance\.

![Refer to caption](https://arxiv.org/html/2606.26538v1/x3.png)Figure 3:Gradient flow is inherently front\-loaded in deep residual architectures\.The average L2 gradient norm per layer in a Vanilla Transformer follows a linear decay, closely tracking its theoretical fan\-in \(black, dashed\)\. In contrast, LayerSkip’s deep supervision mechanism induces a quadratic decay, a behaviour accurately modelled by its own theoretical curve \(grey, dotted\)\. ResNet\-50 shows the same front\-loaded decay, tracking the structural fan\-in predicted by GFA\.

### 4\.3Correlational evidence across architectures

##### Structural fan\-in aligns with gradient flow

GFA predicts that the downstream gradient fan\-in decreases with depth, inducing a front\-loaded gradient distribution\. Figure[3](https://arxiv.org/html/2606.26538#S4.F3)illustrates this empirical pattern across three architectures alongside their theoretical fan\-in decay\. For the vanilla network, this decay is linear, while for LayerSkip it is quadratic\. We observe that the theoretical fan\-in and the empirical gradient normg¯i\\bar\{g\}\_\{i\}share the same monotonic ordering\. Complete derivations and path counts are provided in Appendix[B\.1](https://arxiv.org/html/2606.26538#A2.SS1)\.

![Refer to caption](https://arxiv.org/html/2606.26538v1/x4.png)Figure 4:Training gradient flow predicts final layer importance\.The accumulated gradient shareg¯i\\bar\{g\}\_\{i\}during training shows a strong Spearman correlation with post hoc functional importanceΔ​ℳi\\Delta\\mathcal\{M\}\_\{i\}\. The correlation is significant in the Vanilla Transformer withρ=0\.62\\rho=0\.62and in ResNet\-50 withρ=0\.83\\rho=0\.83, and is near perfect in LayerSkip withρ=0\.99\\rho=0\.99\.
##### Gradient flow is associated with the functional hierarchy

We correlate each layer’sg¯i\\bar\{g\}\_\{i\}with its ablation\-based importanceΔ​ℳi\\Delta\\mathcal\{M\}\_\{i\}\. Figure[4](https://arxiv.org/html/2606.26538#S4.F4)shows a strong positive Spearman correlation for the Vanilla Transformer withρ=0\.62\\rho=0\.62andp=0\.02p=0\.02and for ResNet\-50 withρ=0\.83\\rho=0\.83andp<0\.01p<0\.01\. The relationship tightens in LayerSkip withρ=0\.99\\rho=0\.99andp<0\.01p<0\.01where supervision accentuates early\-layer fan\-in\. Because LayerSkip’s auxiliary losses attach directly to early layers, this near\-perfect correlation partly reflects that design and is best read as a consistency check rather than independent confirmation; the Vanilla \(ρ=0\.62\\rho\{=\}0\.62\) and ResNet\-50 \(ρ=0\.83\\rho\{=\}0\.83\) results, which lack this mechanism, are the load\-bearing correlational evidence\. This links the structural gradient skew during training to the final functional hierarchy, and suggests the accumulated gradient share as a predictive proxy for functional importance on the models studied here\.

### 4\.4Interventions separate magnitude from structure

Correlation does not imply causation\. To disentangle the roles of gradient magnitude and structure, we conduct two interventional tests designed to probe the GFA hypothesis\.

##### Equalizing magnitude does not restore importance

We first test the alternative hypothesis: that small gradient magnitude is the direct cause of redundancy\. During training, we insert a hook that rescales per\-layer gradients to have an equal L2 norm\. To accommodate this artificial amplification, we proportionally scaled the gradient clip norm, deriving the factor from the ratio of maximum observed norms between the hooked and standard models, details shown in Appendix[A\.2\.2](https://arxiv.org/html/2606.26538#A1.SS2.SSS2)\. If magnitude were the causal factor, this should rescue the importance of later layers\. The result, shown in Figure[5](https://arxiv.org/html/2606.26538#S4.F5)\(top\), is the opposite\. The figure shows a logarithmic comparison against a vanilla model, with an inset detailing the validation gradient distribution; equalizing the norms not only fails to restore importance but actively harms the contribution of deep layers\. Amplifying an information\-poor signal does not add compositional content; it only increases the norm and can destabilize learning\.

##### Increasing path counts restores importance

Next, we directly test the structural component of GFA against a vanilla 8\-layer reference\. We engineer an increase in the downstream gradient paths for deep layers by repeating the final four layers of an 8\-layer model with shared parameters, which increases a layer’s virtual depth and its gradient fan\-in without adding parameters\. Specifically, we repeat the last four layers with a pattern of two, three, three, and five repeats respectively, a pattern designed to increase the fan\-in with depth\. This modification dramatically alters the structural fan\-in for the deepest layers \(detailed in Appendix[B\.1](https://arxiv.org/html/2606.26538#A2.SS1)\), for instance changing the counts for layers 5 through 8 from \[5, 4, 3, 2\] to \[27, 33, 24, 20\]\. GFA predicts these layers should become more functionally important\.

Figure[5](https://arxiv.org/html/2606.26538#S4.F5)\(bottom\) supports this prediction\. The inset shows that gradients in the repeated deep layers increase substantially, even exceeding those of early layers\. Consequently, their functional importance rises, diminishing the relative contribution of the initial four layers\. These modified deep layers, now recipients of a more compositionally complex gradient, become more critical than the untouched shallow layers\. Together, these interventions provide interventional evidence consistent with functional hierarchy being shaped by the structural flow of gradient information, for which magnitude is a correlated proxy\.

![Refer to caption](https://arxiv.org/html/2606.26538v1/x5.png)Figure 5:Interventional tests are consistent with gradient structure shaping importance\.Top equalizing L2 gradient norms across layers does not rescue and reduces deep layer importance\. Bottom increasing downstream path counts by parameter shared repetition restores and elevates late layer importance\. A logarithmic y\-axis spans the wide dynamic range\. What matters is how each layer’s importance changes relative to its baseline\.

### 4\.5Applications informed by GFA

We now apply GFA to two efficiency methods: a pruning rule and a tapered architecture\.

##### CascadeFlow Pruning uses training dynamics

We prune layers using the accumulated gradient share,g¯i\\bar\{g\}\_\{i\}, gathered directly during training\. Our method, CascadeFlow Pruning \(CFP\), uses the L2 norm of these gradients \(Eq\.[2](https://arxiv.org/html/2606.26538#S4.E2)\) as a proxy for a layer’s functional importance, ranking them accordingly\. This approach eliminates the need for expensive post hoc computation required by alternative heuristics like hidden state similarityGromov et al\. \([2025](https://arxiv.org/html/2606.26538#bib.bib9)\), Taylor based methodsMa et al\. \([2023](https://arxiv.org/html/2606.26538#bib.bib22)\), or parameter magnitude pruningHan et al\. \([2015](https://arxiv.org/html/2606.26538#bib.bib11)\)\. CFP then removes the lowest ranked layers, irrespective of their original consecutive block structure\.

###### Evaluation

We evaluated pruning strategies on the Dolma 2\.6M token holdout set and evaluation set of HellaSwag benchmark\. We report perplexity \(PPL\) on this Dolma 2\.6M holdout, on which the unpruned baseline scores 17\.94\. On HellaSwag, we adopt the zero\-shot protocol from OLMoOLMo et al\. \([2024](https://arxiv.org/html/2606.26538#bib.bib25)\), ranking candidates by their relative conditional log\-probability\. While this zero\-shot approach does not measure post\-fine\-tuning adaptability, it provides a direct and efficient benchmark for quantifying performance degradation as a function of pruning\.

Table[1](https://arxiv.org/html/2606.26538#S4.T1)summarizes the results\. The similarity method becomes competitive only under the most aggressive pruning, but typically at a higher perplexity cost\. In contrast, Taylor and Magnitude methods are unstable\. Their reliance on a per\-layer forward pass yields importance rankings that fluctuate across seeds, leading to misrankings and sharp performance degradation\. Our CFP avoids this sensitivity, producing stable rankings and a more graceful degradation profile\.

###### Implementation via Layer Passthrough

Pruning a Transformer layer in our framework requires no architectural modification: we implement it by a simple*passthrough*in the forward pass since skipped layers are simply treated as identity functions\. Specifically, during inference the model iterates over the stack of decoder layers\. For each indexii, ifiibelongs to the pruned setℐprune\\mathcal\{I\}\_\{\\text\{prune\}\}, we skip the forward call to the corresponding block and directly forward the input hidden state to the next layer:

hi\+1=hi,if​i∈ℐprune,h\_\{i\+1\}\\;=\\;h\_\{i\},\\qquad\\text\{if \}i\\in\\mathcal\{I\}\_\{\\text\{prune\}\},This design choice allows CFP to be integrated into existing Transformer codebases \(e\.g\.,LLaMA\-style decoders\) with only a few lines of modification\.

##### CascadeFormer\.

Our second application internalizes the GFA principle directly into the model’s architecture\. Because compositional gradient diversity decays with depth, uniform capacity allocation is inefficient\. We therefore designed theCascadeFormer, an architecture that tapers model width to align its capacity with this information flow\. For a model withNNlayers, indexedl∈\{0,…,N−1\}l\\in\\\{0,\\dots,N\-1\\\}, we apply tapering rules to either the attention, FFN sublayers or both\.

###### Attention Tapering\.

We reduce the number of attention heads, and thus the attention dimensiondattn​\(l\)d\_\{\\text\{attn\}\}\(l\), in discrete steps governed by,dattn​\(l\)=dattn,0−Sd⋅⌊l/Fd⌋,d\_\{\\text\{attn\}\}\(l\)=d\_\{\\text\{attn\},0\}\-S\_\{d\}\\cdot\\lfloor l/F\_\{d\}\\rfloor,wheredattn,0d\_\{\\text\{attn\},0\}is the initial dimension,SdS\_\{d\}is the dimensional reduction per step, andFdF\_\{d\}controls the frequency of the reduction\.

###### FFN Tapering\.

We reduce the FFN’s inner dimensiondffn​\(l\)d\_\{\\text\{ffn\}\}\(l\)linearly with depth according to the rule,dffn​\(l\)=dffn,0−Sf⋅l,d\_\{\\text\{ffn\}\}\(l\)=d\_\{\\text\{ffn\},0\}\-S\_\{f\}\\cdot l,wheredffn,0d\_\{\\text\{ffn\},0\}is the initial dimension andSfS\_\{f\}controls the steepness of the linear taper\.

kkMetricCFP \(Ours\)SimTaylorMagnitudeBaseline \(k=0\)PPL=17\.94±\\pm0\.00, Acc\.=0\.39±\\pm0\.001PPL↓\\downarrow19\.84819\.848p m0\.08221\.94521\.945p m 0\.46224\.70024\.700p m 0\.55924\.36124\.361p m 2\.357Acc\.↑\\uparrow0\.3810\.381p m 0\.0020\.3810\.381p m 0\.0020\.3840\.384p m0\.0010\.3620\.362p m 0\.0132PPL↓\\downarrow23\.22623\.226p m0\.10628\.48028\.480p m 0\.331127\.744127\.744p m 15\.14841\.87541\.875p m 4\.822Acc\.↑\\uparrow0\.3720\.372p m0\.0010\.3660\.366p m 0\.0010\.3690\.369p m 0\.0030\.3250\.325p m 0\.0064PPL↓\\downarrow59\.79059\.790p m1\.84059\.79059\.790p m1\.8404715\.0724715\.072p m 4166\.454332\.174332\.174p m 216\.582Acc\.↑\\uparrow0\.3340\.334p m 0\.0040\.3340\.334p m 0\.0040\.3360\.336p m0\.0020\.2850\.285p m 0\.0106PPL↓\\downarrow167\.006167\.006p m9\.205180\.862180\.862p m 27\.3041193\.1591193\.159p m 547\.7093099\.5303099\.530p m 1708\.037Acc\.↑\\uparrow0\.2990\.299p m 0\.0040\.3040\.304p m 0\.0030\.3050\.305p m0\.0020\.2690\.269p m 0\.0028PPL↓\\downarrow911\.748911\.748p m55\.001911\.748911\.748p m55\.0011403\.4341403\.434p m 434\.8941237\.2121237\.212p m 132\.241Acc\.↑\\uparrow0\.2850\.285p m0\.0010\.2850\.285p m0\.0010\.2780\.278p m 0\.0060\.2640\.264p m 0\.001Table 1:CFP achieves the lowest perplexity and the most stable rankings under aggressive layer pruning\.We evaluate CFP \(Ours\) against standard pruning heuristics by removing an increasing number of layers \(kk\)\. CFP consistently achieves the lowest perplexity \(PPL\) and maintains competitive downstream accuracy \(Acc\), particularly at higher sparsities\.
###### Variant Configurations\.

We define six CascadeFormer variants based on these rules, categorized by low \(subscript 1\) and high \(subscript 2\) tapering intensity\. The specific hyperparameters for each variant are detailed in Table[4](https://arxiv.org/html/2606.26538#A1.T4)\. The combined variants \(C\) apply both attention and FFN tapering schemes simultaneously\. To ensure a fair comparison, we also trained six baseline models whose computational cost was scaled linearly by reducing their layer count\.

Table[2](https://arxiv.org/html/2606.26538#S4.T2)quantifies the performance and efficiency of our GFA\-informed architectures\. Our CascadeFormer\-A2A\_\{2\}was designed with a training FLOP budget equivalent to the Vanilla\-15L baseline\. It achieves comparable perplexity to the baseline \(17\.84±0\.0217\.84\\pm 0\.02vs17\.84±0\.0317\.84\\pm 0\.03\) while reducing inference latency by 8\.6% and increasing throughput by 9\.4%\. These efficiency figures are specific to a single A100 withtorch\.compile\-selected kernels and to the Vanilla\-15L FLOP\-matching; absolute gains may vary with hardware, batch size, and baseline choice\.

To further understand the design space, we explored additional variants applying the tapering principle to FFN layers \(F1, F2\) and in combination \(C1, C2\)\. While all GFA\-informed models are competitive, the attention\-tapered variants \(A1, A2\) perform best, which is consistent with the self\-attention mechanism being the most effective target for tapering; we note, however, that head\-count reduction also yields the largest kernel\-efficiency gains, so our experiments do not isolate the two effects\.

##### Latency Measurement Protocol\.

All inference metrics were measured on a single A100 GPU\. To ensure fair comparison, we used a consistent batch size, context length, and generation length for all models\. We leveragedtorch\.compilewith themode=’max\-autotune’andfullgraph=Trueoptions to minimize implementation\-specific overhead and accurately reflect the inherent hardware\-friendliness of each architecture\. We report the median of 100 timed runs after a warmup phase of 10 steps to measure accurate execution time\.

Core MetricsHardware EfficiencyModelPPL↓\\downarrowParams\(B\)Util\.\(TFLOP/s\)↑\\uparrowCost\(TFLOPs\)Latency\(ms\)↓\\downarrowThroughput\(tok/s\)↑\\uparrowUniform BaselinesVanilla\-16L17\.691\.2853\.87±\\pm0\.304\.8289\.47±\\pm0\.5022,890±\\pm129Vanilla\-15L17\.841\.2154\.23±\\pm0\.344\.5483\.81±\\pm0\.5324,439±\\pm153Vanilla\-14L18\.001\.1554\.68±\\pm0\.384\.2778\.09±\\pm0\.5426,227±\\pm181Vanilla\-13L18\.171\.0855\.18±\\pm0\.394\.0072\.40±\\pm0\.5228,287±\\pm201Vanilla\-12L18\.821\.0155\.74±\\pm0\.343\.7266\.75±\\pm0\.4030,683±\\pm186CascadeFormer \(Ours\)CascadeFormer\-A117\.791\.2555\.39±\\pm0\.374\.7285\.17±\\pm0\.5724,048±\\pm161CascadeFormer\-A217\.841\.2259\.77±\\pm0\.384\.5876\.62±\\pm0\.4826,731±\\pm169CascadeFormer\-F117\.881\.1951\.16±\\pm0\.274\.4386\.65±\\pm0\.4723,635±\\pm127CascadeFormer\-F218\.101\.0949\.92±\\pm0\.244\.0581\.06±\\pm0\.3925,266±\\pm123CascadeFormer\-C117\.941\.1653\.00±\\pm0\.354\.3381\.71±\\pm0\.5425,066±\\pm164CascadeFormer\-C218\.301\.0353\.88±\\pm0\.493\.8170\.65±\\pm0\.6628,990±\\pm266Table 2:CascadeFormer matches the baseline’s perplexity at lower latency and higher throughput\.Our CascadeFormer\-A2A\_\{2\}model, which tapers its attention mechanism according to GFA principles, matches the perplexity of its Vanilla\-15L baseline that was trained with an equivalent computational budget while being substantially faster, reducing inference latency by 8\.6% while increasing throughput by 9\.4%\.

## Discussion and Conclusion

This work reframes layer redundancy in residual networks not as a failure of optimization, but as a phenomenon that may follow from their structure\. We presented evidence that Gradient Fan\-in Asymmetry offers a structural account of layer redundancy, supported by correlational and interventional evidence on from\-scratch models up to 1\.2B parameters\. The lasting contribution is a pair of efficiency methods: CascadeFlow Pruning, a more effective pruning method, and CascadeFormer, an efficient architecture that aligns its capacity with the asymmetric flow of information\. The GFA account motivated both designs; a reader who is unconvinced by the account can still adopt the methods on their measured merits\.

##### Design Tension and Future Directions\.

Our findings expose a design tension\. One path is to embrace the asymmetry, as CascadeFormer does, leading to intentionally heterogeneous architectures that allocate resources where learning dynamics can best use them\. A second path is to counteract GFA, aiming to force uniform functional contribution\. This second path is complicated by evidence from architectures like LayerSkip\. By imposing deep supervision, these models create an extreme gradient hierarchy that forces shallow layers to become functionally self\-sufficient\. Shallow layers may hold latent capacity that standard end\-to\-end training, with its gentler GFA decay, does not fully use\. The limitation may then lie in the training dynamic rather than in layer capacity\. Such a pursuit would require new architectural components or training schemes that can inject compositional diversity into deep layer gradients, perhaps through novel long range information pathways or regularization techniques\. Whether uniform contribution is achievable, or even desirable, remains open\.

## Limitations

Our analysis frames Gradient Fan\-in Asymmetry in terms of path quantity, using fan\-in as a proxy for compositional diversity\. This leaves two questions explicitly open\. First, magnitude proxies fan\-in only when the gradient is high\-rank: a layer with small but high\-quality \(high effective\-rank, near\-orthogonal\) gradients need not be redundant, so measuring gradient effective rank, orthogonality, and information content is the natural next step\. Second, all of our evidence is on from\-scratch models up to 1\.2B parameters; while the GFA account is stated in architectural terms, whether it holds at the 100B\+ parameter scale is an open empirical question and we do not claim universality\. Our quality evidence for CascadeFormer is limited to language\-model perplexity and zero\-shot HellaSwag\. Finally, our proposed pruning method, CFP, requires access to training\-time gradients, making it inapplicable for post hoc pruning of pre\-trained, closed\-source models\.

## References

- Brown et al\. \(2020\)Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert\-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and 12 others\. 2020\.[Language Models are Few\-Shot Learners](https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)\.In*Advances in Neural Information Processing Systems*, volume 33, pages 1877–1901\. Curran Associates, Inc\.
- Chen et al\. \(2023\)Tianyi Chen, Tianyu Ding, Badal Yadav, Ilya Zharkov, and Luming Liang\. 2023\.[LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery](https://doi.org/10.48550/arXiv.2310.18356)\.*arXiv preprint*\.ArXiv:2310\.18356 \[cs\]\.
- Chen et al\. \(2025\)Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, and Hong Chen\. 2025\.[Streamlining Redundant Layers to Compress Large Language Models](https://doi.org/10.48550/arXiv.2403.19135)\.*arXiv preprint*\.ArXiv:2403\.19135 \[cs\]\.
- Deng et al\. \(2009\)Jia Deng, Wei Dong, Richard Socher, Li\-Jia Li, Kai Li, and Li Fei\-Fei\. 2009\.[ImageNet: A large\-scale hierarchical image database](https://doi.org/10.1109/CVPR.2009.5206848)\.In*2009 IEEE Conference on Computer Vision and Pattern Recognition*, pages 248–255\.ISSN: 1063\-6919\.
- Elhoushi et al\. \(2024\)Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole\-Jean Wu\. 2024\.[LayerSkip: Enabling Early Exit Inference and Self\-Speculative Decoding](https://doi.org/10.18653/v1/2024.acl-long.681)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 12622–12642, Bangkok, Thailand\. Association for Computational Linguistics\.
- Fan et al\. \(2020\)Angela Fan, Edouard Grave, and Armand Joulin\. 2020\.[Reducing transformer depth on demand with structured dropout](https://openreview.net/forum?id=SylO2yStDr)\.In*International Conference on Learning Representations*\.
- Frantar and Alistarh \(2023\)Elias Frantar and Dan Alistarh\. 2023\.[SparseGPT: Massive Language Models Can Be Accurately Pruned in One\-Shot](https://doi.org/10.48550/arXiv.2301.00774)\.*arXiv preprint*\.ArXiv:2301\.00774 \[cs\]\.
- Gromov et al\. \(2024\)Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Dan Roberts\. 2024\.[The Unreasonable Ineffectiveness of the Deeper Layers](https://openreview.net/forum)\.
- Gromov et al\. \(2025\)Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A\. Roberts\. 2025\.[The Unreasonable Ineffectiveness of the Deeper Layers](https://doi.org/10.48550/arXiv.2403.17887)\.*arXiv preprint*\.ArXiv:2403\.17887 \[cs\]\.
- Han et al\. \(2016\)Song Han, Huizi Mao, and William J\. Dally\. 2016\.[Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://doi.org/10.48550/arXiv.1510.00149)\.*arXiv preprint*\.ArXiv:1510\.00149 \[cs\]\.
- Han et al\. \(2015\)Song Han, Jeff Pool, John Tran, and William J\. Dally\. 2015\.[Learning both Weights and Connections for Efficient Neural Networks](https://doi.org/10.48550/arXiv.1506.02626)\.*arXiv preprint*\.ArXiv:1506\.02626 \[cs\]\.
- He et al\. \(2016a\)Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun\. 2016a\.[Deep Residual Learning for Image Recognition](https://doi.org/10.1109/CVPR.2016.90)\.In*2016 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 770–778, Las Vegas, NV, USA\. IEEE\.
- He et al\. \(2016b\)Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun\. 2016b\.[Identity mappings in deep residual networks](https://arxiv.org/abs/1603.05027)\.*Preprint*, arXiv:1603\.05027\.
- He et al\. \(2024\)Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li\. 2024\.[What Matters in Transformers? Not All Attention is Needed](https://doi.org/10.48550/arXiv.2406.15786)\.*arXiv preprint*\.ArXiv:2406\.15786 \[cs\]\.
- Jiang et al\. \(2025\)Jiachen Jiang, Jinxin Zhou, and Zhihui Zhu\. 2025\.[Tracing representation progression: Analyzing and enhancing layer\-wise similarity](https://openreview.net/forum?id=vVxeFSR4fU)\.In*The Thirteenth International Conference on Learning Representations*\.
- Kim et al\. \(2024\)Bo\-Kyeong Kim, Geonmin Kim, Tae\-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung\-Kyu Song\. 2024\.[Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods](https://doi.org/10.48550/arXiv.2402.02834)\.*arXiv preprint*\.ArXiv:2402\.02834 \[cs\]\.
- Lee et al\. \(2019\)Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H\. S\. Torr\. 2019\.[SNIP: Single\-shot Network Pruning based on Connection Sensitivity](https://doi.org/10.48550/arXiv.1810.02340)\.*arXiv preprint*\.ArXiv:1810\.02340 \[cs\]\.
- Li et al\. \(2024\)Pengxiang Li, Lu Yin, and Shiwei Liu\. 2024\.[Mix\-LN: Unleashing the Power of Deeper Layers by Combining Pre\-LN and Post\-LN](https://openreview.net/forum?id=BChpQU64RG)\.
- Li et al\. \(2025\)Pengxiang Li, Lu Yin, and Shiwei Liu\. 2025\.[Mix\-LN: Unleashing the Power of Deeper Layers by Combining Pre\-LN and Post\-LN](https://doi.org/10.48550/arXiv.2412.13795)\.*arXiv preprint*\.ArXiv:2412\.13795 \[cs\]\.
- Liu et al\. \(2020\)Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju\. 2020\.[Fastbert: a self\-distilling bert with adaptive inference time](https://arxiv.org/abs/2004.02178)\.*Preprint*, arXiv:2004\.02178\.
- Loshchilov and Hutter \(2018\)Ilya Loshchilov and Frank Hutter\. 2018\.[Decoupled Weight Decay Regularization](https://openreview.net/forum?id=Bkg6RiCqY7)\.
- Ma et al\. \(2023\)Xinyin Ma, Gongfan Fang, and Xinchao Wang\. 2023\.[LLM\-Pruner: On the Structural Pruning of Large Language Models](https://doi.org/10.48550/arXiv.2305.11627)\.*arXiv preprint*\.ArXiv:2305\.11627 \[cs\]\.
- Men et al\. \(2025\)Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and weipeng chen\. 2025\.[ShortGPT: Layers in large language models are more redundant than you expect](https://openreview.net/forum?id=JMNht3SmcG)\.
- Molchanov et al\. \(2017\)Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz\. 2017\.[Pruning Convolutional Neural Networks for Resource Efficient Inference](https://doi.org/10.48550/arXiv.1611.06440)\.*arXiv preprint*\.ArXiv:1611\.06440 \[cs\]\.
- OLMo et al\. \(2024\)Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 21 others\. 2024\.[2 olmo 2 furious](https://arxiv.org/abs/2501.00656)\.*Preprint*, arXiv:2501\.00656\.
- \(26\)Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever\.Language Models are Unsupervised Multitask Learners\.
- Selvaraju et al\. \(2020\)Ramprasaath R\. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra\. 2020\.[Grad\-CAM: Visual Explanations from Deep Networks via Gradient\-based Localization](https://doi.org/10.1007/s11263-019-01228-7)\.*International Journal of Computer Vision*, 128\(2\):336–359\.ArXiv:1610\.02391 \[cs\]\.
- Shleifer et al\. \(2021\)Sam Shleifer, Jason Weston, and Myle Ott\. 2021\.[NormFormer: Improved Transformer Pretraining with Extra Normalization](https://doi.org/10.48550/arXiv.2110.09456)\.*arXiv preprint*\.ArXiv:2110\.09456 \[cs\]\.
- Smilkov et al\. \(2017\)Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg\. 2017\.[SmoothGrad: removing noise by adding noise](https://doi.org/10.48550/arXiv.1706.03825)\.*arXiv preprint*\.ArXiv:1706\.03825 \[cs\]\.
- Soldaini et al\. \(2024\)Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, and 17 others\. 2024\.[Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://doi.org/10.18653/v1/2024.acl-long.840)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 15725–15788, Bangkok, Thailand\. Association for Computational Linguistics\.
- \(31\)Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae\-Joon Kim\.SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks\.
- Sun et al\. \(2024\)Mingjie Sun, Zhuang Liu, Anna Bair, and J\. Zico Kolter\. 2024\.[A Simple and Effective Pruning Approach for Large Language Models](https://doi.org/10.48550/arXiv.2306.11695)\.*arXiv preprint*\.ArXiv:2306\.11695 \[cs\]\.
- Sun et al\. \(2025\)Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu\. 2025\.[The Curse of Depth in Large Language Models](https://doi.org/10.48550/arXiv.2502.05795)\.*arXiv preprint*\.ArXiv:2502\.05795 \[cs\]\.
- Takase et al\. \(2023\)Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki\. 2023\.[B2T Connection: Serving Stability and Performance in Deep Transformers](https://doi.org/10.48550/arXiv.2206.00330)\.*arXiv preprint*\.ArXiv:2206\.00330 \[cs\]\.
- Touvron et al\. \(2023\)Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie\-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample\. 2023\.[LLaMA: Open and Efficient Foundation Language Models](https://doi.org/10.48550/arXiv.2302.13971)\.*arXiv preprint*\.ArXiv:2302\.13971 \[cs\]\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin\. 2017\.[Attention is All you Need](https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)\.In*Advances in Neural Information Processing Systems*, volume 30\. Curran Associates, Inc\.
- Veit et al\. \(2016\)Andreas Veit, Michael Wilber, and Serge Belongie\. 2016\.[Residual networks behave like ensembles of relatively shallow networks](https://arxiv.org/abs/1605.06431)\.*Preprint*, arXiv:1605\.06431\.
- Wang et al\. \(2022\)Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei\. 2022\.[DeepNet: Scaling Transformers to 1,000 Layers](https://doi.org/10.48550/arXiv.2203.00555)\.*arXiv preprint*\.ArXiv:2203\.00555 \[cs\]\.
- Xia et al\. \(2024\)Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen\. 2024\.[Sheared LLaMA: Accelerating Language Model Pre\-training via Structured Pruning](https://doi.org/10.48550/arXiv.2310.06694)\.*arXiv preprint*\.ArXiv:2310\.06694 \[cs\]\.
- Xin et al\. \(2020\)Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin\. 2020\.[Deebert: Dynamic early exiting for accelerating bert inference](https://arxiv.org/abs/2004.12993)\.*Preprint*, arXiv:2004\.12993\.
- Xiong et al\. \(2020\)Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu\. 2020\.[On Layer Normalization in the Transformer Architecture](https://proceedings.mlr.press/v119/xiong20b.html)\.In*Proceedings of the 37th International Conference on Machine Learning*, pages 10524–10533\. PMLR\.ISSN: 2640\-3498\.
- Xu et al\. \(2021\)Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen\. 2021\.[GSPMD: General and Scalable Parallelization for ML Computation Graphs](https://doi.org/10.48550/arXiv.2105.04663)\.*arXiv preprint*\.ArXiv:2105\.04663 \[cs\]\.
- Yang et al\. \(2024\)Yifei Yang, Zouying Cao, and Hai Zhao\. 2024\.[LaCo: Large Language Model Pruning via Layer Collapse](https://doi.org/10.48550/arXiv.2402.11187)\.*arXiv preprint*\.ArXiv:2402\.11187 \[cs\]\.
- Yang et al\. \(2023\)Ziqing Yang, Yiming Cui, Xin Yao, and Shijin Wang\. 2023\.[Gradient\-based intra\-attention pruning on pre\-trained language models](https://doi.org/10.18653/v1/2023.acl-long.156)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 2775–2790, Toronto, Canada\. Association for Computational Linguistics\.
- Zhao et al\. \(2025\)Anhao Zhao, Fanghua Ye, Yingqi Fan, Junlong Tong, Zhiwei Fei, Hui Su, and Xiaoyu Shen\. 2025\.[SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling](https://doi.org/10.48550/arXiv.2506.04179)\.*arXiv preprint*\.ArXiv:2506\.04179 \[cs\]\.

## Use of Large Language Models

We disclose our use of Large Language Models \(LLMs\) in preparing this manuscript\. Our use was focused on two areas: manuscript preparation and literature discovery\.

##### Manuscript Preparation

We utilized an LLM for copy\-editing tasks, including correcting grammar, refining prose for clarity, and ensuring stylistic consistency\. The research questions, methodology, analysis, and conclusions are the authors’ own\.

##### Literature Discovery

An LLM was employed as a preliminary tool to broaden our literature search\. Its function was to identify potentially relevant papers and suggest alternative search keywords based on our initial research scope\. Every paper cited in this work was subsequently read, critically evaluated, and integrated into our analysis by the authors to confirm its relevance and correctness\.

## Reproducibility Statement

To ensure the reproducibility of our findings, we document our methodology, data, and computational environment below\.Code:The complete codebase to reproduce all experiments, including model training, evaluation, and figure generation, will be made publicly available on GitHub upon publication\.Environment:The hardware and software stacks for training \(TPU v4\) and evaluation \(NVIDIA A100/A6000\), including all library versions, are documented in Appendix[A\.1](https://arxiv.org/html/2606.26538#A1.SS1)\.Data:All experiments utilize public datasets\. Language models were trained on the publicly available pre\-tokenized Dolma dataset from Hugging Face\. Vision models were trained on the standard ILSVRC 2012 ImageNet\-1k dataset\.Methodology:Full architectural specifications for all models, including our novel CascadeFormer, are in Appendix[A\.2](https://arxiv.org/html/2606.26538#A1.SS2)and[A\.2\.1](https://arxiv.org/html/2606.26538#A1.SS2.SSS1)\. The precise methodology for our gradient\-equalization intervention, in which amplifying late\-layer gradient magnitude does not restore late\-layer importance, is detailed in Appendix[A\.2\.2](https://arxiv.org/html/2606.26538#A1.SS2.SSS2)\.Hyperparameters:All training and evaluation hyperparameters are enumerated in Appendix[A\.3](https://arxiv.org/html/2606.26538#A1.SS3)\. All experiments can be reproduced with fixed random seed of324709for model initialization and data loading\.

## Appendix AExperiment Details

We detail the experimental setup below\. We detail the model architectures, training hyperparameters, and the specific implementation of our proposed CascadeFormer variants as well as layer\-wise pruning heuristic details\.

### A\.1Computational Resources and Software

#### A\.1\.1Training Environment

All model training was conducted on Google Cloud Platform \(GCP\) using a 128\-core TPU v4 Pod slice\. Our training stack utilized PyTorch2\.7\.0with the correspondingtorch\_xla==2\.7\.0library\.

To distribute training, we employed the GSPMD \(Xu et al\. \([2021](https://arxiv.org/html/2606.26538#bib.bib42)\)\) partitioner\. We utilized a 1D sharding strategy where model parameters, gradients, and optimizer states were fully sharded across the data\-parallel dimension\. This approach efficiently managed memory while scaling computation across all 128 TPU cores\. All training was performed using afloat32data type\.

#### A\.1\.2Evaluation Environment

All evaluations were performed on a single GPU, using either an NVIDIA A100\-80GB or an NVIDIA A6000\-48GB\. The evaluation framework was PyTorch2\.7\.0\. All experiments were conducted infloat32precision to mitigate any possibility of precision error\.

#### A\.1\.3Implementation Details

Our model architecture is a modification of the Llama implementation from the Hugging Facetransformers==4\.51\.3library\. This same codebase served as the basis for our layer\-dropping ablation studies\. All experimental results were logged using the Weights & Biases \(wandb\) platform, and figures included in this paper were generated withmatplotlib\.

### A\.2Model Configurations

The core of our investigation involves three distinct architectures to test the generalizability of our claims\. Their primary configurations are summarized in Table[3](https://arxiv.org/html/2606.26538#A1.T3)\.

ParameterVanilla TransformerLayerSkip TransformerResNet\-50Base ArchitectureLlama\-3\.2\-1BLlama\-3\.2\-1BResNet familyNumber of Layers/Blocks161616 \(Blocks\)Hidden Dimension \(dm​o​d​e​ld\_\{model\}\)20482048–FFN Inner Dimension \(df​f​nd\_\{ffn\}\)81928192–Number of Attention Heads3232–Vocabulary Size50,28050,280–Block Size20482048–Classes––1000Table 3:Architectural details of the primary models used in our experiments\.Configurations for the three architectures, spanning language and vision \(Transformer, LayerSkip, ResNet\-50\)\.#### A\.2\.1CascadeFormer Architecture Details

The CascadeFormer is motivated by the GFA hypothesis\. While standard Transformers allocate uniform capacity to all layers, our architecture is designed under the hypothesis that compositional information flows in a cascade, reducing with depth\. The CascadeFormer is designed in accordance with this hypothesized flow, tapering its width to match computational capacity to the signal’s richness\. This is achieved by progressively reducing the dimensions of the attention and feed\-forward network \(FFN\) sub\-layers\. Based on a 16\-layer Transformer baseline \(l∈\{0,1,…,15\}l\\in\\\{0,1,\\dots,15\\\}\), we define a set of variants with modulated tapering intensity to explore this new design space\.

Variant NameTapering Target\(s\)IntensityGoverning ParametersCascadeFormer\-A1Attention DimensionLowFd=4F\_\{d\}=4CascadeFormer\-A2Attention DimensionHighFd=2F\_\{d\}=2CascadeFormer\-F1FFN DimensionLowSf=128S\_\{f\}=128CascadeFormer\-F2FFN DimensionHighSf=256S\_\{f\}=256CascadeFormer\-C1Attention & FFNLowFd=4,Sf=128F\_\{d\}=4,\\ S\_\{f\}=128CascadeFormer\-C2Attention & FFNHighFd=2,Sf=256F\_\{d\}=2,\\ S\_\{f\}=256Table 4:Hyperparameter configurations for the CascadeFormer variants\.The parametersFdF\_\{d\}andSfS\_\{f\}control the tapering intensity for the attention and FFN dimensions, respectively\. LowerFdF\_\{d\}and higherSfS\_\{f\}values correspond to more aggressive capacity reduction\.
#### A\.2\.2Dynamic Gradient Scaling Intervention

Our intervention mechanism operates by dynamically rescaling gradients on a layer\-wise basis during training\. This ensures that the gradient magnitudes are harmonized across all layers before the optimizer step\. The procedure is executed as follows:

1. 1\.Layer\-wise Norm Computation:For each layeriiin the network, we aggregate all parameter gradients associated with it\. We then compute the Euclidean \(L2L\_\{2\}\) norm of these concatenated gradients, denoted asnin\_\{i\}\.
2. 2\.Target Norm Identification:We identify the maximum gradient norm across all layers,ntarget=maxi⁡\(ni\)n\_\{\\text\{target\}\}=\\max\_\{i\}\(n\_\{i\}\)\. This value serves as the reference magnitude for scaling\.
3. 3\.Scaling Factor Derivation:A scaling factorλi\\lambda\_\{i\}is calculated for each layer by dividing the target norm by the layer’s individual norm:λi=ntarget/ni\\lambda\_\{i\}=n\_\{\\text\{target\}\}/n\_\{i\}\. To handle layers with zero gradients and prevent division\-by\-zero,λi\\lambda\_\{i\}is set to1\.01\.0ifni=0n\_\{i\}=0\.
4. 4\.In\-place Gradient Application:Finally, every parameter gradient within a given layeriiis multiplied in\-place by its corresponding scaling factorλi\\lambda\_\{i\}\. This operation normalizes the influence of each layer’s gradient relative to the layer with the strongest signal\.

### A\.3Training Hyperparameters

All models were trained using the hyperparameters detailed in Table[5](https://arxiv.org/html/2606.26538#A1.T5), set per domain for competitive baselines\.

HyperparameterTransformer modelsResNet\-50Datasetpretokenized\-dolmaImageNet\-1kAdamWβ1,β2\\beta\_\{1\},\\beta\_\{2\}\(0\.9, 0\.95\)\(0\.9, 0\.999\)Weight Decay0\.10\.05Learning Rate ScheduleCosine w/ WarmupCosine w/ WarmupPeak Learning Rate2e\-45e\-4Warmup Steps20005 EpochsBatch Size64512Epochs130Gradient Clipping Norm1\.0\-Table 5:Training hyperparameters for all experiments\.This table provides the complete set of hyperparameters used for training the language and vision models, ensuring full reproducibility of our results\.
### A\.4Pruning Heuristic Details

To provide full transparency for the pruning comparison presented in Table[1](https://arxiv.org/html/2606.26538#S4.T1), we detail each pruning heuristic below\.

##### Magnitude\-based HeuristicHan et al\. \([2015](https://arxiv.org/html/2606.26538#bib.bib11)\)

Classical magnitude pruning removes small\-magnitude weights and is a strong baseline for sparsification\. In the block\-level variant used here, a block’s score is the aggregate of absolute parameter values across its trainable weights; for projection matrices, values are first reduced along the input dimension to obtain per\-output scores, then aggregated within the block\. Token embeddings and the LM head are excluded, and the final normalization is protected\. Blocks are ranked by increasing score, and thekklowest are pruned\.

##### Taylor\-based HeuristicMa et al\. \([2023](https://arxiv.org/html/2606.26538#bib.bib22)\)

The first\-order criterion estimates loss increase from removing a block by accumulating, on a small calibration set, the absolute elementwise product of each parameter and its gradient\. Scores are summed across batches; for projection matrices, per\-output reduction precedes block\-level aggregation\. Token embeddings are excluded and the final normalization is protected\. Blocks are ranked by increasing score, and thekklowest are pruned\.

##### Similarity\-based Heuristic\.Gromov et al\. \([2025](https://arxiv.org/html/2606.26538#bib.bib9)\)

This heuristic is based on representational similarity analysis, computing the angular distance between adjacent layer representations\. Blocks ofkklayers whose removal yields the smallest change are considered less critical and are pruned\.

##### CFP \(CascadeFlow Pruning\) Heuristic\.

In contrast, our proposed CFP method prunes layers with the lowest accumulated L2 gradient norms from training\.

## Appendix BAdditional Results and Analyses

We give additional layer\-importance profiles, layer\-wise trend plots, and a representational\-similarity analysis that support the main findings\.

### B\.1Gradient Fan\-in Derivations

##### Definition and Counting Rule\.

We define gradient fan\-in as the number of downstream transformation edges aggregated at a layer’s input,xlx\_\{l\}\. This first\-order proxy, counting the identity path, each downstream block’s Jacobian branch, and the final head, captures the structural asymmetry driving GFA\. It is a count of contributing channels, not a combinatorial path enumeration\. For a standardNN\-block stack with one head, the fan\-in at layerllis:

ϕl=\(N−l\)\+2,\(functional blocks \+ identity path \+ head\)\\phi\_\{l\}=\(N\-l\)\+2,\\quad\\text\{\(functional blocks \+ identity path \+ head\)\}\(3\)

##### Fan\-in Under Deep Supervision\.

Architectures with deep supervision introduce an auxiliary loss headℒk\\mathcal\{L\}\_\{k\}at the output of each blockkk, with the total loss beingℒtotal=ℒN\+∑k=0N−1αk​ℒk\\mathcal\{L\}\_\{\\text\{total\}\}=\\mathcal\{L\}\_\{N\}\+\\sum\_\{k=0\}^\{N\-1\}\\alpha\_\{k\}\\mathcal\{L\}\_\{k\}\. The gradient at the input to blockll,gl′g^\{\\prime\}\_\{l\}, now aggregates signals from all downstream loss functions:

gl′=∂ℒtotal∂xl=∂ℒN∂xl⏟Original Hierarchy\+∑k=lN−1αk​∂ℒk∂xl⏟Auxiliary Hierarchies\.g^\{\\prime\}\_\{l\}=\\frac\{\\partial\\mathcal\{L\}\_\{\\text\{total\}\}\}\{\\partial x\_\{l\}\}=\\underbrace\{\\frac\{\\partial\\mathcal\{L\}\_\{N\}\}\{\\partial x\_\{l\}\}\}\_\{\\text\{Original Hierarchy\}\}\+\\sum\_\{k=l\}^\{N\-1\}\\alpha\_\{k\}\\underbrace\{\\frac\{\\partial\\mathcal\{L\}\_\{k\}\}\{\\partial x\_\{l\}\}\}\_\{\\text\{Auxiliary Hierarchies\}\}\.\(4\)Each auxiliary lossℒk\\mathcal\{L\}\_\{k\}\(sourced atxk\+1x\_\{k\+1\}\) effectively creates a new residual sub\-network for its gradient to traverse back toxlx\_\{l\}\. The fan\-in from a single auxiliary loss at layerkkto layerllis analogous to a standard network of length\(k\+1−l\)\(k\+1\-l\)and is thus\(k−l\)\+1\+1=\(k−l\+2\)\(k\-l\)\+1\+1=\(k\-l\+2\)\. Assuming for simplicity that it is attached atxkx\_\{k\}, the fan\-in we use below is\(k−l\)\+1\(k\-l\)\+1\. The total fan\-inϕl′\\phi^\{\\prime\}\_\{l\}is the sum of the fan\-in from the original path and all new auxiliary paths originating from layersllthroughN−1N\-1:

ϕl′=ϕl\+∑k=lN−1\(\(k−l\)\+1\),\\phi^\{\\prime\}\_\{l\}=\\phi\_\{l\}\+\\sum\_\{k=l\}^\{N\-1\}\(\(k\-l\)\+1\),\(5\)whereϕl=\(N−l\)\+2\\phi\_\{l\}=\(N\-l\)\+2is the standard single\-head fan\-in from Eq\.[3](https://arxiv.org/html/2606.26538#A2.E3)\. Substitutingϕl\\phi\_\{l\}and lettingj=k−lj=k\-l, the summation becomes:

ϕl′=\(\(N−l\)\+2\)\+∑j=0N−1−l\(j\+1\)\.\\phi^\{\\prime\}\_\{l\}=\(\(N\-l\)\+2\)\+\\sum\_\{j=0\}^\{N\-1\-l\}\(j\+1\)\.\(6\)
Solving this arithmetic series and combining terms yields:

ϕl′=\(\(N−l\)\+2\)\+\(N−l\)​\(N−l\+1\)2\.\\phi^\{\\prime\}\_\{l\}=\(\(N\-l\)\+2\)\+\\frac\{\(N\-l\)\(N\-l\+1\)\}\{2\}\.\(7\)The leading12​\(N−l\)2\\tfrac\{1\}\{2\}\(N\-l\)^\{2\}term shows that deep supervision transforms the linear fan\-in disparity into a quadratic one, amplifying GFA\.

##### Fan\-in with Virtual Depth\.

Consider 8 physical blocks unrolled intoNvirt=17N\_\{\\text\{virt\}\}\{=\}17virtual positions with shared parameters: L1–L4 \(1×\\times\), L5 \(2×\\times\), L6 \(3×\\times\), L7 \(3×\\times\), L8 \(5×\\times\)\. LetViV\_\{i\}be the block at virtual positionii\(i=1,…,17i\{=\}1,\\ldots,17\)\. The fan\-in for a physical layerLpL\_\{p\}sums over its virtual instances:

FanIn​\(Lp\)=∑i:Vi=Lp\(Nvirt−i\+2\)\.\\text\{FanIn\}\(L\_\{p\}\)\\;=\\;\\sum\_\{i:\\,V\_\{i\}=L\_\{p\}\}\\big\(N\_\{\\text\{virt\}\}\-i\+2\\big\)\.\(8\)
This yields the following Standard→\\toVirtual counts: L1: 9→\\to18, L2: 8→\\to17, L3: 7→\\to16, L4: 6→\\to15, L5: 5→\\to27, L6: 4→\\to33, L7: 3→\\to24, L8: 2→\\to20\.

### B\.2Layer\-wise Gradient and Importance Trends

The following figures provide a detailed visualization of the data from Figure[4](https://arxiv.org/html/2606.26538#S4.F4)\.

![Refer to caption](https://arxiv.org/html/2606.26538v1/x6.png)Figure 6:Layer\-wise comparison for the Vanilla Transformer\.This figure details the relationship between relative L2 gradient norm \(red, dashed\) and functional importance \(blue, solid\)\. The correspondence visually reinforces the correlation \(ρ=0\.62\\rho=0\.62\) from Figure[4](https://arxiv.org/html/2606.26538#S4.F4)a\.![Refer to caption](https://arxiv.org/html/2606.26538v1/x7.png)Figure 7:Layer\-wise comparison for the LayerSkip Transformer\.This figure details the relationship between relative L2 gradient norm \(red, dashed\) and functional importance \(blue, solid\)\. The curves track closely, matching the near\-perfect correlation \(ρ=0\.99\\rho=0\.99\) from Figure[4](https://arxiv.org/html/2606.26538#S4.F4)b\.![Refer to caption](https://arxiv.org/html/2606.26538v1/x8.png)Figure 8:Layer\-wise comparison for ResNet\-50\.This figure details the relationship between relative L2 gradient norm \(red, dashed\) and functional importance \(blue, solid\)\. A clear positive relationship is evident, corroborating the correlation \(ρ=0\.83\\rho=0\.83\) from Figure[4](https://arxiv.org/html/2606.26538#S4.F4)c\.