What Does Layer-Importance Reveal About Transformers and State-Space Models?

# What Does Layer-Importance Reveal About Transformers and State-Space Models?
Source: [https://arxiv.org/html/2609.16537](https://arxiv.org/html/2609.16537)
Istabrak AbbesAffiliation:Chandar Research LabAffiliation:Mila \- Quebec AI InstituteAffiliation:Université de MontréalEmail:[istabrak\.abbes@mila\.quebec](mailto:)Irina RishAffiliation:Mila \- Quebec AI InstituteAffiliation:Université de MontréalAffiliation:Canada CIFAR AI ChairSarath ChandarAffiliation:Chandar Research LabAffiliation:Mila \- Quebec AI InstituteAffiliation:Polytechnique MontréalAffiliation:Canada CIFAR AI Chair

###### Abstract

Transformers and state\-space models \(SSMs\) are the two dominant families of sequence models, and a central open question is how far the analytical knowledge built for transformers transfers to SSMs\. We address this through the lens of layer importance which underpins compression, selective fine\-tuning, and interpretability across both families\. We decompose layer importance into two distinct notions\.*Necessity*captures how much the pretrained model depends on a layer’s existing contribution, measured by the loss increase from bypassing it\.*Plasticity*captures where the model absorbs new information during fine\-tuning, measured by the magnitude of task\-specific weight updates\. Our analysis reveals that the two families behave fundamentally differently: in every evaluated residual transformer up to1414B parameters, Necessity and Plasticity anti\-align across depth, whereas in the evaluated Mamba\-style SSMs they point to overlapping regions\. The sign of this alignment also predicts downstream adaptation behavior\. In the evaluated transformers, concentrating updates in the most plastic layers increases catastrophic forgetting, while this tier\-dependent effect disappears in the evaluated Mamba\-style SSMs\.

## 1Introduction

Modern language modeling is still dominated by residual transformers\([Vaswani et al\., 2023](https://arxiv.org/html/2609.16537#bib.bib1)\), but selective state\-space models have emerged as a major alternative architecture\([Gu and Dao, 2024](https://arxiv.org/html/2609.16537#bib.bib7);[Dao and Gu, 2024](https://arxiv.org/html/2609.16537#bib.bib8)\)\. A large body of work on transformers has produced mature tools for pruning\([Gromov et al\., 2025](https://arxiv.org/html/2609.16537#bib.bib6);[Men et al\., 2025](https://arxiv.org/html/2609.16537#bib.bib17)\), parameter\-efficient fine\-tuning\([Hu et al\., 2021](https://arxiv.org/html/2609.16537#bib.bib9)\), and interpretability\([Meng et al\., 2023](https://arxiv.org/html/2609.16537#bib.bib28);[Song et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib22)\), many of which rely on estimating*layer importance*\([Zhang et al\., 2024](https://arxiv.org/html/2609.16537#bib.bib19);[Naim et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib16);[Xu et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib15);[Yao et al\., 2024](https://arxiv.org/html/2609.16537#bib.bib18)\)\. On the other hand, SSMs inherit the same vocabulary but not, obviously, the same internal structure\. The natural question is whether the analytical tools calibrated on transformers transfer to SSMs, or whether they silently measure something else when selective recurrent state dynamics replace attention\-based token mixing\. We study this question by decomposing layer importance into two distinct notions\.*Necessity*measures how much the pretrained model depends on a layer’s existing contribution, quantified through the loss increase caused by bypassing the layer\.*Plasticity*measures where the model absorbs new information during adaptation, quantified through the magnitude of task\-specific weight updates during fine\-tuning\. Although these notions are often conflated under a single notion of “importance,” they capture fundamentally different properties: pruning and interpretability primarily rely on Necessity\([Men et al\., 2025](https://arxiv.org/html/2609.16537#bib.bib17);[Zhang et al\., 2024](https://arxiv.org/html/2609.16537#bib.bib19)\), while selective fine\-tuning relies on Plasticity\([Pan et al\., 2024](https://arxiv.org/html/2609.16537#bib.bib5);[Zhang et al\., 2023](https://arxiv.org/html/2609.16537#bib.bib10);[Qing et al\., 2024](https://arxiv.org/html/2609.16537#bib.bib11);[Hayou et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib14)\)\. The key question is whether these two quantities identify the same regions of a network, and whether this relationship transfers across architectures\.

Figure 1:Layer importance is method\-relative and architecture\-conditional\.Left:Estimators projected onto the Necessity–Plasticity plane \(mean Spearman correlation with ablation, vertical; with LoRA\-delta, horizontal\)\. In transformers, gradient and resnorm methods align with Necessity; activation norm and TELL\-TALE align with Plasticity\. In Mamba, estimators cluster in the mixed\-positive region\.Right:𝒩​𝒫​𝒜​\(M\)\\mathcal\{NPA\}\(M\)across checkpoints \(bars: bootstrap95%95\\%CI\)\. Every evaluated transformer up to1414B is negative; every evaluated Mamba\-style SSM is positive; the two largest transformers collapse toward zero\. Dashed lines mark family means\.To make this distinction explicit, we introduce*Necessity\-Plasticity Alignment*\(𝒩​𝒫​𝒜\\mathcal\{NPA\}\): the mean Spearman correlation, across tasks, between an ablation\-based Necessity ranking and a LoRA\-update\-magnitude\-based Plasticity ranking of the model’s layers\. Positive𝒩​𝒫​𝒜\\mathcal\{NPA\}means that the same layers are both necessary and plastic; negative𝒩​𝒫​𝒜\\mathcal\{NPA\}means that pretrained dependence and adaptation concentrate at opposite ends of the network; values near zero indicate that no stable depth ordering exists without specifying the estimator\.

Across the evaluated transformer and Mamba\-style checkpoints,𝒩​𝒫​𝒜\\mathcal\{NPA\}separates the two families cleanly\. Every evaluated residual transformer up to1414B parameters exhibits negative alignment: early layers are most necessary, while late layers are most plastic\. In contrast, every evaluated Mamba\-style SSM exhibits positive alignment, with Necessity and Plasticity concentrated in overlapping regions\. This contrast persists across importance estimators, survives removal of the first layer, and extends beyond Mamba to RWKV and a hybrid architecture\. It is also not an artifact of low\-rank adaptation: an adapter\-free control that recomputes Plasticity from full\-parameter fine\-tuning preserves the sign of𝒩​𝒫​𝒜\\mathcal\{NPA\}on every checkpoint we retrained \(Appendix[E\.3](https://arxiv.org/html/2609.16537#A5.SS3)\)\. Larger transformers, including Qwen3\-32B and Llama\-3\.1\-70B, do not enter the SSM regime; instead, their alignment collapses toward zero\. The same split predicts downstream adaptation behavior: transformers show tier\-dependent forgetting when successive tasks adapt highly plastic layers, whereas this effect vanishes in Mamba\-style SSMs111Code, per\-layer importance scores, and analysis scripts:[https://github\.com/chandar\-lab/layer\-importance\-ssm\-vs\-transformers](https://github.com/chandar-lab/layer-importance-ssm-vs-transformers)\.

##### Contributions\.

1\)We introduce*Necessity\-Plasticity Alignment*\(𝒩​𝒫​𝒜\\mathcal\{NPA\}\), a diagnostic that measures the agreement between pretrained layer dependence and task\-specific adaptation across depth\.

2\)We find a robust architecture\-dependent split: every evaluated residual transformer up to1414B exhibits negative𝒩​𝒫​𝒜\\mathcal\{NPA\}, with early layers most necessary and late layers most plastic, whereas the evaluated Mamba\-style SSMs exhibit positive𝒩​𝒫​𝒜\\mathcal\{NPA\}, with Necessity and Plasticity concentrated in overlapping regions\. The split survives an adapter\-free full fine\-tuning control and reappears*within*a single hybrid checkpoint\.

3\)We show that𝒩​𝒫​𝒜\\mathcal\{NPA\}predicts selective fine\-tuning behavior in a controlled two\-task continual\-learning probe: the evaluated transformers exhibit strong forgetting when both tasks adapt the same highly plastic layers, while this tier\-dependent interference disappears in the evaluated Mamba\-style SSMs\.

## 2Related Work

##### Importance\-guided PEFT and rank/placement allocation\.

A growing line of parameter\-efficient fine\-tuning \(PEFT\) work argues that adaptation should not be uniform across depth\. AdaLoRA\([Zhang et al\., 2023](https://arxiv.org/html/2609.16537#bib.bib10)\)reallocates rank by singular\-value importance during training; AlphaLoRA\([Qing et al\., 2024](https://arxiv.org/html/2609.16537#bib.bib11)\)assigns LoRA experts using heavy\-tailed self\-regularization statistics; Layer Card\([Xu et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib15)\)introduces*resnorm*as a reusable layer diagnostic for selective placement; ShapLoRA\([Zhao et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib12)\)uses Shapley\-style sensitivity scores; and PLoP\([Hayou et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib14)\)proposes a precise placement signal driven by neural\-feature norms\. Each of these methods proposes a single importance estimator and studies it in isolation\. Recent work has also begun to ask whether compression, PEFT, and interpretability tools developed for transformers transfer to selective state\-space models\. LoRA\-style adaptation has been extended to Mamba\-style architectures\([Galim et al\., 2025](https://arxiv.org/html/2609.16537#bib.bib13)\), while layer pruning and relevance\-propagation methods have been adapted to selective SSMs\([Munoz et al\., 2025](https://arxiv.org/html/2609.16537#bib.bib23);[Jafari et al\., 2025](https://arxiv.org/html/2609.16537#bib.bib29)\)\. These works motivate cross\-architecture comparisons, but they do not ask whether different task\-sensitive importance estimators agree on the same checkpoints

##### Layer roles, redundancy, and depth\-dependent task usage\.

A parallel literature studies which layers are useful, redundant, or task\-specific in pretrained models\. ShortGPT\([Men et al\., 2025](https://arxiv.org/html/2609.16537#bib.bib17)\)shows that many transformer layers can be removed with limited degradation; TELL\-TALE\([Naim et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib16)\)performs task\-aware layer elimination;[Song et al\. \(2026\)](https://arxiv.org/html/2609.16537#bib.bib22)argue that depth contributes differently to retrieval, knowledge, and reasoning;[Yao et al\. \(2024\)](https://arxiv.org/html/2609.16537#bib.bib18)study layer\-wise importance for memory\-efficient PEFT\.[Zhang et al\. \(2024\)](https://arxiv.org/html/2609.16537#bib.bib19)identify “cornerstone” layers via Shapley\-style ablation, finding early\-layer dominance, a phenomenon we recover under ablation but show dissolves under LoRA delta on the same checkpoint\.[Nepal et al\. \(2025\)](https://arxiv.org/html/2609.16537#bib.bib20)report that the layers ablation identifies as important for mathematical reasoning are stable across pretraining and post\-training\.[Shi et al\. \(2025\)](https://arxiv.org/html/2609.16537#bib.bib21)learn binary masks for layer significance during alignment and observe high overlap across alignment datasets, all under a single estimator\. These results document stability*within*one estimator; we document the orthogonal fact that*across*estimators, the very identity of “important” layers changes, and that whether it changes is architecture\-conditional\.

##### Method disagreement in attribution and importance\.

Disagreement between attribution methods is an established finding in the explainability literature\.[Adebayo et al\. \(2020\)](https://arxiv.org/html/2609.16537#bib.bib26)show that gradient\-saliency maps survive parameter randomizations that destroy network behavior, evidence that gradient attribution does not measure the same thing as causal intervention\.[Ancona et al\. \(2018\)](https://arxiv.org/html/2609.16537#bib.bib25)unify gradient\-based methods as varying\-fidelity linearizations of true ablation;[Sundararajan et al\. \(2017\)](https://arxiv.org/html/2609.16537#bib.bib24)prove that no attribution method satisfies sensitivity, implementation invariance, and completeness simultaneously\.[Krishna et al\. \(2024\)](https://arxiv.org/html/2609.16537#bib.bib27)measure the magnitude of feature\-level attribution disagreement and characterize how practitioner interpretations shift with the method chosen\. Our contribution shifts the disagreement from feature attributions to layer rankings, from a single architecture to a transformer/SSM contrast, and from “expected at high resolution” to a structurally predictable function of residual\-stream gradient flow\.

## 3Layer\-Importance Diagnostic Protocol

Our goal is to determine whether commonly used layer\-importance estimators measure the same underlying property across architectures, or whether their behavior depends on the distinction between pretrained reliance and adaptation dynamics\. We therefore study layer importance through two complementary quantities:*Necessity*and*Plasticity*\.

##### Necessity\.

Necessity measures how much the pretrained model depends on a layer’s existing contribution at inference time\. A layer is considered necessary if bypassing it substantially degrades performance\. For a modelMMwith layers\{ℓi\}i=1L\\\{\\ell\_\{i\}\\\}\_\{i=1\}^\{L\}, we define the Necessity score of layerℓi\\ell\_\{i\}on taskttas:

N⁡\(ℓi,M,t\)=ℒ⁡\(M∖ℓi,t\)−ℒ⁡\(M,t\),N\(\\ell\_\{i\};M,t\)=\\mathcal\{L\}\\\!\\left\(M\_\{\\setminus\\ell\_\{i\}\},t\\right\)\-\\mathcal\{L\}\(M,t\),whereℒ\\mathcal\{L\}denotes the task loss andM∖ℓiM\_\{\\setminus\\ell\_\{i\}\}is the model with layerℓi\\ell\_\{i\}bypassed\. Larger values indicate that the pretrained computation relies more strongly on that layer\.

##### Plasticity\.

Plasticity measures where the model absorbs new information during adaptation\. Rather than quantifying reliance on the pretrained computation, it measures how strongly each layer changes during task\-specific fine\-tuning\. Using LoRA adaptation, we define the Plasticity score of layerℓi\\ell\_\{i\}as:

P⁡\(ℓi,M,t\)=‖Δ​Wi\(t\)‖F,P\(\\ell\_\{i\};M,t\)=\\left\\\|\\Delta W\_\{i\}^\{\(t\)\}\\right\\\|\_\{F\},whereΔ​Wi\(t\)\\Delta W\_\{i\}^\{\(t\)\}is the learned LoRA update for layerℓi\\ell\_\{i\}on tasktt, and∥⋅∥F\\\|\\cdot\\\|\_\{F\}denotes the Frobenius norm\. Larger values indicate that adaptation concentrates more strongly in that layer\.

Although both quantities are often grouped under a single notion of “importance,” they capture fundamentally different properties\. Necessity measures dependence of the pretrained model on an existing computation, whereas Plasticity measures receptivity to new information during adaptation\. The relationship between these two quantities is therefore an empirical and architectural question\.

To study this relationship, we introduce*Necessity\-Plasticity Alignment*\(𝒩​𝒫​𝒜\\mathcal\{NPA\}\), defined as the Spearman correlation between layer rankings induced by Necessity and Plasticity:

𝒩​𝒫​𝒜​\(M,t\)=ρSpearman​\(RNec​\(M,t\),RPlast​\(M,t\)\),\\mathcal\{NPA\}\(M,t\)=\\rho\_\{\\mathrm\{Spearman\}\}\\bigl\(R\_\{\\mathrm\{Nec\}\}\(M,t\),R\_\{\\mathrm\{Plast\}\}\(M,t\)\\bigr\),

whereRNecR\_\{\\mathrm\{Nec\}\}andRPlastR\_\{\\mathrm\{Plast\}\}denote the corresponding layer rankings for modelMMon tasktt\.

Positive𝒩​𝒫​𝒜\\mathcal\{NPA\}indicates that the same layers are both necessary and plastic, while negative𝒩​𝒫​𝒜\\mathcal\{NPA\}indicates that pretrained reliance and adaptation concentrate on opposite ends of the network\. Values near zero indicate that no stable agreement exists between the two notions of importance\.

This protocol allows us to compare layer\-importance structure across architectures independently of any single estimator\. Rather than asking whether one importance metric is universally correct, we ask whether different estimators consistently align with Necessity or Plasticity, and whether the relationship between these quantities changes across model families\.

## 4Transformers and SSMs Disagree on Layer Importance

We now apply the diagnostic protocol to the transfer question directly\. If transformer\-derived layer\-importance concepts transferred uniformly to SSMs, Necessity and Plasticity rankings should relate similarly across both families\. Instead, their relationship changes sign across architectures\.

### 4\.1Necessity–Plasticity Alignment changes sign across architectures

![Refer to caption](https://arxiv.org/html/2609.16537v1/fig_taskwise_npa_heatmap.png)Figure 2:Per\-task𝒩​𝒫​𝒜\\mathcal\{NPA\}across architectures and scales\.Spearmanρ\\rhobetween Necessity and Plasticity rankings per \(model, task\) pair\. Transformers are predominantly negative \(red\) up to1414B, fading toward zero at larger scale\. Mamba\-family and RWKV6\-3B are positive \(green\)\. Hybrid and boundary models sit near zero\.Figure[1](https://arxiv.org/html/2609.16537#S1.F1)reports𝒩​𝒫​𝒜​\(M\)\\mathcal\{NPA\}\(M\)between ablation\-based Necessity rankings and LoRA\-delta\-based Plasticity rankings\. The sign of𝒩​𝒫​𝒜\\mathcal\{NPA\}cleanly separates the two families\. All evaluated Mamba\-style SSMs show positive alignment: layers receiving larger LoRA updates are also those whose ablation causes larger loss increases\. In contrast, every evaluated residual transformer up to1414B shows negative alignment: the layers most necessary for preserving pretrained computation are not the layers where fine\-tuning writes the largest task\-specific updates\. Figure[2](https://arxiv.org/html/2609.16537#S4.F2)resolves the same measurement per \(model, task\) pair and shows that the split is not carried by a handful of outlier tasks: within each family the sign is consistent across the task suite\.

The largest transformers weaken this pattern without becoming SSM\-like\. Qwen3\-32B and Llama\-3\.1\-70B move toward near\-zero alignment, suggesting a partial reorganization of Plasticity rather than recovery of a universal layer ordering\. This disagreement is not estimator noise: LoRA\-delta rankings remain self\-consistent under matched scopes while continuing to disagree with ablation\. Strong negative𝒩​𝒫​𝒜\\mathcal\{NPA\}requires a concentrated high\-Necessity region together with an adaptation profile that avoids it, and both ingredients weaken with scale; since𝒩​𝒫​𝒜\\mathcal\{NPA\}is a rank correlation, flattening either profile drives it toward zero\. We therefore state the negative\-transformer result for the evaluated≤14\\leq 14B regime and do not extrapolate beyond the measured checkpoints\.

##### The split is not an artifact of low\-rank adaptation\.

Because Plasticity is read off a LoRA update, the split could in principle reflect where LoRA places updates rather than a property of the architecture\. An adapter\-free control on1313checkpoints rules this out \(Appendix[E\.3](https://arxiv.org/html/2609.16537#A5.SS3)\): full\-FT and LoRA update profiles are positively rank\-correlated on every one, and the sign of the alignment is preserved throughout, negative for every evaluated transformer with ablation baselines and positive or near zero for the evaluated Mamba, RWKV, and hybrid checkpoints\. Normalizing updates by pretrained weight norm leaves this unchanged\. The conclusion therefore does not depend on LoRA rank, scaling, initialization, or target\-module selection\.

##### Generalization beyond Mamba\.

The positive\-alignment regime is not specific to Mamba\. RWKV\([Peng et al\., 2023](https://arxiv.org/html/2609.16537#bib.bib4)\)exhibits strongly positive alignment \(𝒩​𝒫​𝒜=\+0\.54\\mathcal\{NPA\}=\+0\.54, with all per\-task correlations positive\)\. Zamba2\([Glorioso et al\., 2024](https://arxiv.org/html/2609.16537#bib.bib34)\), a hybrid SSM\+attention model, lies near zero but slightly positive \(𝒩​𝒫​𝒜=\+0\.05\\mathcal\{NPA\}=\+0\.05\), while Falcon\-Mamba\-7B\([Zuo et al\., 2024](https://arxiv.org/html/2609.16537#bib.bib33)\)lies slightly below zero \(𝒩​𝒫​𝒜=−0\.02\\mathcal\{NPA\}=\-0\.02\)\. Together, these results suggest the ordering

𝒩​𝒫​𝒜SSM\>𝒩​𝒫​𝒜Hybr\.≈0\>𝒩​𝒫​𝒜Transf\.≤14​B\.\\mathcal\{NPA\}\_\{\\mathrm\{SSM\}\}\>\\mathcal\{NPA\}\_\{\\mathrm\{Hybr\.\}\}\\approx 0\>\\mathcal\{NPA\}\_\{\\mathrm\{Transf\.\}\\leq 14\\mathrm\{B\}\}\.

The transformer\-SSM contrast is strongest at small and medium scale, while very large transformers and boundary SSMs move toward a shared near\-zero regime\.

##### Necessity and Plasticity also dissociate*within*a hybrid model\.

Zamba2\-2\.7B interleaves99shared\-attention blocks among4545Mamba blocks, which lets us test the dissociation without changing checkpoints\. Attention blocks are only16\.7%16\.7\\%of the layers but absorb36\.3%36\.3\\%of LoRA and31\.6%31\.6\\%of full\-FT update mass, while carrying just10\.4%10\.4\\%of ablation Necessity mass, which concentrates in the Mamba blocks \(Appendix[E\.4](https://arxiv.org/html/2609.16537#A5.SS4)\)\. Zamba2’s near\-zero whole\-model𝒩​𝒫​𝒜\\mathcal\{NPA\}is therefore the average of a transformer\-like and an SSM\-like component, not the absence of the effect\.

### 4\.2Depth profiles explain the architecture split

Figure 3:Representative Necessity and Plasticity depth profiles\.Qwen3\-8B: LoRA delta concentrates near the final layers \(a\); ablation is dominated by an early layer\-0 spike \(b\)\. Mamba\-790m: LoRA delta peaks mid\-network \(c\); ablation emphasizes boundary layers with partial overlap \(d\)\. Transformers select opposite ends of depth; Mamba profiles differ but are not spatially opposed\.Figure[3](https://arxiv.org/html/2609.16537#S4.F3)shows the spatial origin of the sign change\. In transformers, Plasticity concentrates near the output side of the network, whereas Necessity concentrates near the input side, often dominated by a strong layer\-00component\. The two quantities therefore select opposite ends of depth, producing negative alignment\. In Mamba\-style SSMs, the profiles are different but not spatially opposed\. Plasticity peaks in middle layers, while Necessity emphasizes boundary layers with substantial overlap\. Because the two quantities co\-occupy partially overlapping regions, the resulting alignment remains positive\.

This depth organization also explains why large transformers move toward zero alignment\. Their Necessity profiles remain strongly early\-layer dominated, but Plasticity becomes less purely terminal and develops additional early\-layer mass\. The resulting partial overlap weakens anti\-alignment without producing the positive agreement observed in SSMs\.

### 4\.3Layer\-00and metric controls

Figure 4:Leave\-out\-L0 and L0\-share controls per checkpoint\. Transformers concentrate more ablation mass on layer00than Mamba models, yet removing layer00shifts transformer alignment toward zero without reversing its sign, ruling out the embedding bottleneck as the sole driver of the architecture split\.A natural alternative explanation is that the transformer–SSM contrast is driven entirely by layer00\. Figure[4](https://arxiv.org/html/2609.16537#S4.F4)rules out this strong form\. Removing layer00shifts transformer checkpoints toward zero, but every evaluated transformer up to1414B remains negatively aligned, while Mamba checkpoints change minimally\. Transformers do allocate substantially more Necessity mass to layer00than Mamba models, but the anti\-alignment persists even after removing it, indicating a broader depth\-wise separation between Necessity and Plasticity\.

We also test whether ranking\-based placement conclusions transfer uniformly across validation metrics\. They do not\. On multiple\-choice evaluation, the relative advantage of top\-kkversus bottom\-kkplacement depends on both the estimator and the model family\. Even after rankings are computed under a shared proxy protocol, their downstream interpretation remains metric\-relative\.

### 4\.4Estimator spectrum and implications

The Necessity–Plasticity distinction also organizes estimators beyond the two anchors\. In transformers, gradient\- and resnorm\-based estimators align more strongly with Necessity, while activation norm and TELL\-TALE align more strongly with Plasticity\. ShapLoRA occupies an intermediate position\. In Mamba, where Necessity and Plasticity are already positively aligned, the estimators cluster within the same mixed\-positive region\.

These results show that layer importance is not a universal ordering of depth\. Ablation\-based, adaptation\-based, and proxy estimators are not noisy measurements of a single latent ranking; they measure different properties of the model\. A layer can be important because removing it disrupts pretrained computation, or because fine\-tuning preferentially writes into it\.

Importance\-guided placement should therefore be treated as a measurement problem rather than a model\-independent fact\.

Figure 5:Tier\-dependent forgetting is a transformer effect\.Forgetting ratio \(PPLafter/PPLbefore\\text\{PPL\}\_\{\\text\{after\}\}/\\text\{PPL\}\_\{\\text\{before\}\}\) for top\-kk, random\-kk, and bottom\-kkLoRA placement \(k=6k\{=\}6\)\. In transformers \(right, NPA<0<0\), top\-kkplacement causes significantly more forgetting than bottom\-kk\(p<0\.001p\{<\}0\.001,6666pairs\)\. In SSMs \(left, NPA\>0\>0\), no significant difference is observed \(p=0\.45p\{=\}0\.45,5454pairs\)\.
### 4\.5A two\-task probe of the forgetting mechanism

If𝒩​𝒫​𝒜\\mathcal\{NPA\}captures the spatial relationship between Necessity and Plasticity, it should predict when selective adaptation creates stability failures\. We test this in a controlled two\-task setup where only the adapted layers change\. This experiment is a*mechanism probe*, not a continual\-learning benchmark: two tasks is the minimal setting in which forgetting is well defined, and restricting to it isolates the Necessity–Plasticity overlap from confounds longer sequences introduce \(interference accumulation, capacity saturation, implicit replay through task similarity\)\. We scope its conclusions to the evaluated two\-task setting and the fixed LoRA configuration described here\. For each model, we compute LoRA\-delta scores on a source task A, select either the top\-kkmost plastic or bottom\-kkleast plastic layers, fine\-tune on task A, then fine\-tune on task B using the same layer subset\. Forgetting is measured on task A as

FR=PPLafterPPLbefore\.\\mathrm\{FR\}=\\frac\{\\mathrm\{PPL\}\_\{\\mathrm\{after\}\}\}\{\\mathrm\{PPL\}\_\{\\mathrm\{before\}\}\}\.
Figure[5](https://arxiv.org/html/2609.16537#S4.F5)shows that the plasticity\-stability tradeoff is architecture\-dependent\. In transformers \(𝒩​𝒫​𝒜<0\\mathcal\{NPA\}<0\), adapting through the most plastic layers causes substantially more forgetting than adapting through the least plastic layers\. The top\-kkcondition consistently lies above bottom\-kk, indicating that highly plastic late layers form a shared adaptation bottleneck: successive tasks write into the same narrow region and partially overwrite one another\. In Mamba\-style SSMs \(𝒩​𝒫​𝒜\>0\\mathcal\{NPA\}\>0\), this tier effect disappears\. Top\-kkand bottom\-kkplacement produce statistically indistinguishable forgetting ratios \(p=0\.45p=0\.45\), consistent with Plasticity being distributed rather than concentrated in a single vulnerable tier\. Falcon\-Mamba\-7B lies between these regimes: its near\-zero alignment \(𝒩​𝒫​𝒜=−0\.02\\mathcal\{NPA\}=\-0\.02\) is matched by a weak, marginally significant forgetting effect \(p=0\.055p=0\.055\)\. Thus,𝒩​𝒫​𝒜\\mathcal\{NPA\}acts as a continuous predictor of bottleneck risk: negative alignment produces localized interference, near\-zero alignment produces unstable effects, and positive alignment produces no reliable tier\-dependent bottleneck\. The same qualitative pattern is stable across scale; we report the full scale\-wise plot in Figure[20](https://arxiv.org/html/2609.16537#A8.F20)of Appendix[H](https://arxiv.org/html/2609.16537#A8)\.

##### EWC regularization baseline\.

We also compare placement against elastic weight consolidation \(EWC\)\([Kirkpatrick et al\., 2017](https://arxiv.org/html/2609.16537#bib.bib32)\), which regularizes parameter movement according to Fisher information rather than restricting which layers are adapted\. Figure[6](https://arxiv.org/html/2609.16537#S4.F6)shows that the plasticity–stability tradeoff has different geometry across architectures\. In both Mamba checkpoints, EWC achieves the lowest median forgetting\. For Mamba\-790M, EWC reaches a median forgetting ratio of1\.45×1\.45\{\\times\}, compared with2\.21×2\.21\{\\times\}for top\-kkand1\.68×1\.68\{\\times\}for bottom\-kk\(p=0\.004p=0\.004\); Mamba\-2\.8B shows the same qualitative ordering\. This supports the view that when Plasticity is distributed across depth, stability is better recovered by globally constraining update magnitude than by selecting a small subset of layers\.

In transformers, the tradeoff is more localized\. For Qwen3\-0\.6B, EWC reduces forgetting relative to top\-kkbut remains worse than bottom\-kk\. For Llama\-3\.2\-3B, EWC is again worse than bottom\-kk\(p=0\.004p=0\.004\)\. Thus, transformers benefit more from avoiding the high\-plasticity bottleneck, whereas SSMs benefit more from global regularization\.

##### Recommendation for adapter placement\.

In the evaluated transformers \(𝒩​𝒫​𝒜<0\\mathcal\{NPA\}<0\), the high\-Necessity and high\-Plasticity regions are disjoint, so placement is consequential: place adapters away from the high\-Necessity early block, or protect that block when tasks arrive in sequence\. In the evaluated Mamba\-style SSMs \(𝒩​𝒫​𝒜\>0\\mathcal\{NPA\}\>0\), Plasticity is distributed rather than concentrated in one vulnerable tier, so globally constraining update magnitude \(EWC\) helps more than placement\. Both rules are read off the≤14\\leq 14B regime and the two\-task probe\.

Figure 6:EWC versus placement strategies across architectures\.Forgetting ratio for bottom\-kk, random\-kk, top\-kk, and EWC \(Fisher\-weighted, all layers\)\. In Mamba\-790M \(NPA\>0\>0, left\), EWC achieves the lowest forgetting\. In both transformers \(NPA<0<0, centre and right\), bottom\-kkplacement matches or beats EWC, consistent with the shared\-bottleneck mechanism\.

## 5Mechanistic Account

The previous section showed that Necessity and Plasticity anti\-align in residual transformers but align in Mamba\-style SSMs\. We now give a mechanistic account of this split\. The central idea is that transformer anti\-alignment arises from a depth\-wise separation between pretrained dependence and low\-cost adaptation: early layers are costly to remove, whereas later layers provide favorable sites for task\-specific updates\. Mamba\-style SSMs do not exhibit the same systematic opposition across depth, explaining why transformer\-calibrated layer\-importance tools do not transfer uniformly across architectures\.

Necessity and Plasticity coincide only under restrictive assumptions, such as locally isotropic curvature, infinitesimal interventions, and unconstrained optimization\. Large pretrained models do not satisfy these assumptions in practice\. The relevant question is therefore not whether ablation and LoRA\-delta estimate the same latent quantity, but when architecture makes their rankings align or diverge\.

We approach this question from two complementary perspectives\. The first is a*curvature*view: ablation importance measures the cost of removing a layer’s pretrained contribution, whereas LoRA\-delta importance measures where optimization can write a large, low\-cost task\-specific update\.

The second is a*residual\-Jacobian*view\. In residual networks, removing layerllinduces a perturbation that propagates through the downstream Jacobian product

∏j\>l\(I\+Jfj\),\\prod\_\{j\>l\}\(I\+J\_\{f\_\{j\}\}\),creating a directional asymmetry across depth\. Both perspectives predict the same empirical pattern: early\-layer concentration of Necessity and late\-layer concentration of Plasticity in transformers\.

### 5\.1Residual geometry biases Necessity toward early layers in transformers

Consider a pre\-LN residual block

hl=hl−1\+fl​\(hl−1\),h\_\{l\}=h\_\{l\-1\}\+f\_\{l\}\(h\_\{l\-1\}\),with the loss attached to the final representationhLh\_\{L\}\. Bypassing the layer removes the residual computationfl​\(hl−1\)f\_\{l\}\(h\_\{l\-1\}\)while preserving the skip path\. Linearizing the resulting perturbation gives

Iabl\(l,t\)≈𝔼x\[⟨∂Lt∂hL,\(∏j=l\+1L\(I\+Jfj\)\)fl\(hl−1\)⟩\]\+O⁡\(‖fl​\(hl−1\)‖2\),I\_\{\\mathrm\{abl\}\}\(l,t\)\\approx\\mathbb\{E\}\_\{x\}\\\!\\Big\[\\Big\\langle\\tfrac\{\\partial L\_\{t\}\}\{\\partial h\_\{L\}\},\\\\ \\Big\(\\prod\_\{j=l\+1\}^\{L\}\(I\+J\_\{f\_\{j\}\}\)\\Big\)f\_\{l\}\(h\_\{l\-1\}\)\\Big\\rangle\\Big\]\\\\ \+O\(\\\|f\_\{l\}\(h\_\{l\-1\}\)\\\|^\{2\}\),whereJfjJ\_\{f\_\{j\}\}denotes the Jacobian of blockjj\.

This expression does not imply that early layers must always dominate ablation\. The downstream Jacobian product can amplify, suppress, rotate, or cancel perturbations\. However, earlier residual contributions propagate through more downstream blocks and participate in constructing the representations on which all later layers operate\. This creates a systematic structural bias toward the input side of the network\.

Figure[3](https://arxiv.org/html/2609.16537#S4.F3)shows that this bias dominates empirically\. In transformers, Necessity mass concentrates strongly in early layers, often with a pronounced layer\-00spike, whereas Plasticity concentrates near the output side of the network\. The result is a spatial separation between pretrained dependence and task adaptation\.

Under the curvature interpretation, the same pattern emerges for a complementary reason\. Late transformer layers can absorb task\-specific updates with more direct influence on the final output and less disruption to earlier computations, making them favorable locations for adaptation\. Plasticity therefore accumulates late even when Necessity remains concentrated early\.

Together, these effects predict negative alignment between Necessity and Plasticity in residual transformers\.

Prediction 1:In residual transformers, Necessity and Plasticity should be negatively correlated across depth\.

### 5\.2Mamba\-style SSMs weaken this depth\-wise separation

The transformer mechanism relies not only on residual depth, but on a particular depth organization: pretrained dependence concentrates early, while low\-cost adaptation concentrates late\. Mamba\-style SSMs retain residual depth, but their selective recurrent dynamics do not induce the same empirical separation\.

First, every Mamba layer directly reads the current token through a learned selective gate\. Input information is therefore injected continuously across depth rather than once at the beginning of a residual stream\. No single layer plays the privileged initialization role associated with transformer layer00, so there is no structural reason for Necessity to collapse toward the input side\. Second, the selective recurrence propagates information through an impulse\-response operatorΦj,l​\(x\)\\Phi\_\{j,l\}\(x\)whose eigenvalues remain bounded inside the unit disk\([Gu and Dao, 2024](https://arxiv.org/html/2609.16537#bib.bib7)\)\. The expected propagation norm therefore depends primarily on distance\|j−l\|\|j\-l\|rather than direction from input to output\. Gradient propagation through the recurrence does not preferentially amplify late\-layer adaptation\.

Together, these properties remove the architectural mechanism forcing separation between Necessity and Plasticity\. Figure[3](https://arxiv.org/html/2609.16537#S4.F3)shows the resulting behavior empirically: in Mamba, Necessity and Plasticity occupy partially overlapping regions rather than opposite ends of depth\.

Appendix[A\.4](https://arxiv.org/html/2609.16537#A1.SS4)formalizes this intuition\. Under assumptions B1′–B2, the mass\-displacement coefficientδ\\deltagoverning separation between Necessity and Plasticity vanishes asymptotically:

\|δ\|≤2​KL→0asL/K→∞\.\|\\delta\|\\leq\\frac\{2K\}\{L\}\\to 0\\qquad\\text\{as\}\\qquad L/K\\to\\infty\.
Thus, under the stated assumptions, the mechanism producing systematic rank anti\-correlation in transformers vanishes asymptotically\.

Prediction 2:Mamba\-style SSMs should not exhibit systematic negative Necessity–Plasticity Alignment; their alignment should be non\-negative or near zero\.

### 5\.3Layer\-00controls rule out a pure embedding\-bottleneck explanation

The residual account further predicts that transformers should allocate unusually large Necessity mass to layer00\. Figure[4](https://arxiv.org/html/2609.16537#S4.F4)confirms this prediction: transformers assign substantially more ablation mass to layer00than Mamba models\.

However, layer\-00dominance alone does not explain the full architecture split\. If the effect were purely an embedding bottleneck artifact, removing layer00from the rankings should eliminate the transformer–SSM contrast\. Instead, Figure[4](https://arxiv.org/html/2609.16537#S4.F4)shows that the contrast survives\. Transformer alignment becomes less negative but remains negative, whereas Mamba checkpoints remain positive\.

The phenomenon therefore reflects a broader separation between early\-side Necessity and late\-side Plasticity rather than a single anomalous layer\.

Prediction 3:Removing layer00should weaken negative alignment in transformers\.

### 5\.4Curvature measurements support the decomposition

The curvature account predicts that layer\-wise curvature should co\-vary with Necessity and anti\-correlate with Plasticity in transformers, but not in SSMs\. We test this using diagonal Fisher information:

Fl=𝔼x​\[‖∇θlℒ‖2\]\.F\_\{l\}=\\mathbb\{E\}\_\{x\}\\left\[\\\|\\nabla\_\{\\theta\_\{l\}\}\\mathcal\{L\}\\\|^\{2\}\\right\]\.
![Refer to caption](https://arxiv.org/html/2609.16537v1/fig_hessian_curvature.png)Figure 7:Fisher curvature supports the Necessity–Plasticity split\.Two size\-matched SSM–Transformer pairs \(∼\\sim3B: Mamba\-2\.8B vs\. Llama\-3\.2\-3B;∼\\sim1B: Mamba\-790M vs\. Qwen3\-0\.6B\)\.Top: depth profiles of Fisher curvature \(dotted\), Necessity \(solid\), and Plasticity \(dashed\)\.Middle/Bottom: scatter of Fisher vs\. ablation and vs\. LoRA delta\. In transformers, Fisher curvature co\-varies with Necessity and anti\-correlates with Plasticity\. In SSMs, both correlations are non\-negative\.Figure[7](https://arxiv.org/html/2609.16537#S5.F7)supports this prediction\. In transformers, Fisher curvature co\-varies with Necessity and anti\-correlates with Plasticity: high\-Fisher layers are costly to remove but receive smaller LoRA updates\. In SSMs, Fisher remains positively associated with Necessity and no longer anti\-correlates with Plasticity\. Thus, curvature separates pretrained dependence from adaptation in transformers but not in the same way in Mamba\-style SSMs\.

### 5\.5Scale and estimator structure

The same account explains why transformer anti\-alignment weakens at larger scale\. Negative alignment requires early Necessity together with late\-concentrated Plasticity\. Qwen3\-32B and Llama\-3\.1\-70B retain early Necessity, but their Plasticity profiles develop an additional early\-layer mode \(Figure[21](https://arxiv.org/html/2609.16537#A8.F21), and greater depth and redundancy flatten both profiles\. This partial overlap collapses𝒩​𝒫​𝒜\\mathcal\{NPA\}toward zero without producing the positive alignment observed in SSMs\. The mechanism predicts the direction of the trend but not where it terminates, so we do not extrapolate the sign beyond the measured checkpoints\.

Finally, the decomposition also organizes proxy estimators\. Figure[1](https://arxiv.org/html/2609.16537#S1.F1)shows that, in transformers, gradient\- and resnorm\-based methods align more strongly with Necessity, while activation norm and TELL\-TALE align more strongly with Plasticity; ShapLoRA lies between them\. In Mamba\-style SSMs, where the anchors are already positively aligned, estimators cluster in the same mixed\-positive region\. Layer importance is therefore not a universal ordering of depth, but a measurement whose meaning depends on both architecture and intervention\.

## 6Conclusion

We showed that layer importance is not a model\-independent ordering of depth\. By separating importance into*Necessity*, the dependence of the pretrained model on a layer’s existing computation, and*Plasticity*, the tendency of adaptation to write task\-specific updates into that layer, we find a systematic architecture\-dependent split\. Every evaluated residual transformer up to1414B exhibits negative Necessity–Plasticity Alignment: early layers are most necessary, while later layers are most plastic\. The evaluated Mamba\-style SSMs instead show positive or near\-zero alignment, with the two quantities occupying overlapping regions of depth\. The split survives an adapter\-free control and reappears within a hybrid checkpoint\.

This split is not only diagnostic; it predicts the stability cost of selective adaptation\. In a controlled two\-task probe, adapting the most plastic layers of the evaluated transformers creates a shared late\-layer bottleneck and increases forgetting, whereas in the evaluated Mamba\-style SSMs this tier\-dependent effect largely disappears\. Layer\-00controls, and corroborating curvature measurements, support the same interpretation: transformer anti\-alignment reflects a broader separation between pretrained dependence and adaptation, not a single anomalous layer\.

Overall, layer\-importance methods should be treated as measurements of specific properties rather than interchangeable estimates of one universal ranking\. Tools calibrated on transformers may therefore fail silently on recurrent or hybrid architectures, where the geometry of stored computation and new adaptation can differ fundamentally\.

## Acknowledgements

Sarath Chandar is supported by the Canada CIFAR AI Chairs program, the Canada Research Chair in Lifelong Machine Learning, and the NSERC Discovery Grant\. Irina Rish is supported by the Canada CIFAR AI Chairs program and the Canada Excellence Research Chair in Autonomous AI\. Experiments were conducted using computational resources provided by Mila Quebec AI Institute\.

## 7Limitations

Our analysis studies layer importance through two operational anchors: ablation\-induced loss increase for Necessity and update magnitude for Plasticity\. These definitions are well matched to pruning and parameter\-efficient adaptation, but they do not cover all possible notions of importance\. Plasticity in particular is defined through the outcome of an adaptation procedure rather than as an intrinsic layer property\. We reduce, but do not eliminate, the resulting dependence: Appendix[E\.3](https://arxiv.org/html/2609.16537#A5.SS3)shows that replacing LoRA with unconstrained full\-parameter fine\-tuning, and normalizing updates by pretrained weight norm, both preserve the sign of𝒩​𝒫​𝒜\\mathcal\{NPA\}on every retrained checkpoint\. Other adaptation procedures \(prefix tuning, other PEFT families, different optimizers or budgets\) were not tested\. On the Necessity side, ablation measures coarse total functional reliance at the layer level; activation patching, causal tracing, and optimizer\-state analyses could decompose that reliance into finer mechanisms and may expose additional structure\. Our model coverage spans transformers, Mamba\-style SSMs, RWKV, and hybrid checkpoints, but remains limited relative to the full architecture space, and all architecture\-level statements in this paper should be read as scoped to the evaluated checkpoints and scales\. In particular, hybrid models and very large transformers move toward a near\-zero alignment regime, so𝒩​𝒫​𝒜\\mathcal\{NPA\}should be interpreted as a continuous diagnostic rather than a strict family label\. The negative\-transformer result is established at≤14\\leq 14B; at3232B and7070B the effect is near zero, and we have no evidence about scales beyond7070B\. The hybrid component analysis rests on a single checkpoint with one attention/SSM ratio\. The mechanistic account is also partial\. The residual\-Jacobian and curvature views explain the observed depth\-wise patterns and are supported by layer\-00controls and Fisher measurements, but the theoretical argument relies on simplifying assumptions\. Diagonal Fisher is a coarse proxy for curvature: it discards off\-diagonal terms and is sensitive to parameter scale, so we use it as corroborating evidence only\. None of the paper’s main claims depend on it\. Finally, our continual\-learning experiments are a two\-task mechanism probe rather than a benchmark\. They isolate layer placement under fixed LoRA configurations, task pairs, and optimization budgets\. Longer or more varied task sequences, different ranks, longer training, replay, or full\-model updates may change the magnitude and even the ordering of forgetting effects\. Our conclusion is therefore not that one placement rule is universally optimal, but that the meaning and risk of layer selection depend on architecture and on what the estimator measures\.

## References

- Adebayoet al\.\(2020\)J\. Adebayo, J\. Gilmer, M\. Muelly, I\. Goodfellow, M\. Hardt, and B\. KimSanity checks for saliency maps\.External Links:1810\.03292,[Link](https://arxiv.org/abs/1810.03292)Cited by:[§A\.1](https://arxiv.org/html/2609.16537#A1.SS1.p1.2),[§A\.7](https://arxiv.org/html/2609.16537#A1.SS7.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px3.p1.1)\.
- Aghajanyanet al\.\(2021\)A\. Aghajanyan, S\. Gupta, and L\. ZettlemoyerIntrinsic dimensionality explains the effectiveness of language model fine\-tuning\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 7319–7328\.External Links:[Link](https://aclanthology.org/2021.acl-long.568/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.568)Cited by:[§A\.1](https://arxiv.org/html/2609.16537#A1.SS1.p2.2),[§A\.2](https://arxiv.org/html/2609.16537#A1.SS2.p2.1.1),[Theorem A\.1](https://arxiv.org/html/2609.16537#A1.Thmtheorem1.p1.2.1)\.
- Anconaet al\.\(2018\)M\. Ancona, E\. Ceolini, C\. Öztireli, and M\. GrossTowards better understanding of gradient\-based attribution methods for deep neural networks\.External Links:1711\.06104,[Link](https://arxiv.org/abs/1711.06104)Cited by:[§A\.1](https://arxiv.org/html/2609.16537#A1.SS1.p1.2),[§A\.7](https://arxiv.org/html/2609.16537#A1.SS7.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px3.p1.1)\.
- Dao and Gu \(2024\)T\. Dao and A\. GuTransformers are ssms: generalized models and efficient algorithms through structured state space duality\.External Links:2405\.21060,[Link](https://arxiv.org/abs/2405.21060)Cited by:[§1](https://arxiv.org/html/2609.16537#S1.p1.1)\.
- Galimet al\.\(2025\)K\. Galim, W\. Kang, Y\. Zeng, H\. I\. Koo, and K\. LeeParameter\-efficient fine\-tuning of state space models\.External Links:2410\.09016,[Link](https://arxiv.org/abs/2410.09016)Cited by:[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px1.p1.1)\.
- Gloriosoet al\.\(2024\)P\. Glorioso, Q\. Anthony, Y\. Tokpanov, A\. Golubeva, V\. Shyam, J\. Whittington, J\. Pilault, and B\. MillidgeThe zamba2 suite: technical report\.External Links:2411\.15242,[Link](https://arxiv.org/abs/2411.15242)Cited by:[Table 1](https://arxiv.org/html/2609.16537#A2.T1.2.1.18.1),[§4\.1](https://arxiv.org/html/2609.16537#S4.SS1.SSS0.Px2.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. MaThe llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[Table 1](https://arxiv.org/html/2609.16537#A2.T1.2.1.8.1.1)\.
- Gromovet al\.\(2025\)A\. Gromov, K\. Tirumala, H\. Shapourian, P\. Glorioso, and D\. A\. RobertsThe unreasonable ineffectiveness of the deeper layers\.External Links:2403\.17887,[Link](https://arxiv.org/abs/2403.17887)Cited by:[§1](https://arxiv.org/html/2609.16537#S1.p1.1)\.
- Gu and Dao \(2024\)A\. Gu and T\. DaoMamba: linear\-time sequence modeling with selective state spaces\.External Links:2312\.00752,[Link](https://arxiv.org/abs/2312.00752)Cited by:[Remark A\.4](https://arxiv.org/html/2609.16537#A1.Thmremark4.p1.1),[Table 1](https://arxiv.org/html/2609.16537#A2.T1.2.1.12.1.1),[§1](https://arxiv.org/html/2609.16537#S1.p1.1),[§5\.2](https://arxiv.org/html/2609.16537#S5.SS2.p2.1)\.
- Hayouet al\.\(2026\)S\. Hayou, N\. Ghosh, and B\. YuPLop: precise loRA placement for efficient finetuning of large models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3lGkVgNZ5a)Cited by:[§B\.1](https://arxiv.org/html/2609.16537#A2.SS1.p1.1),[§1](https://arxiv.org/html/2609.16537#S1.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2021\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.External Links:2106\.09685,[Link](https://arxiv.org/abs/2106.09685)Cited by:[§A\.1](https://arxiv.org/html/2609.16537#A1.SS1.p2.2),[§B\.1](https://arxiv.org/html/2609.16537#A2.SS1.p1.1),[Table 7](https://arxiv.org/html/2609.16537#A4.T7.10.1.11.1),[Table 7](https://arxiv.org/html/2609.16537#A4.T7.10.1.19.1),[Table 7](https://arxiv.org/html/2609.16537#A4.T7.10.1.3.1),[§1](https://arxiv.org/html/2609.16537#S1.p1.1)\.
- Jacotet al\.\(2020\)A\. Jacot, F\. Gabriel, and C\. HonglerNeural tangent kernel: convergence and generalization in neural networks\.External Links:1806\.07572,[Link](https://arxiv.org/abs/1806.07572)Cited by:[§A\.1](https://arxiv.org/html/2609.16537#A1.SS1.p2.2),[§A\.2](https://arxiv.org/html/2609.16537#A1.SS2.p2.1.1),[Theorem A\.1](https://arxiv.org/html/2609.16537#A1.Thmtheorem1.p1.2.1)\.
- Jafariet al\.\(2025\)F\. R\. Jafari, G\. Montavon, K\. Müller, and O\. EberleMambaLRP: explaining selective state space sequence models\.External Links:2406\.07592,[Link](https://arxiv.org/abs/2406.07592)Cited by:[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px1.p1.1)\.
- Kirkpatricket al\.\(2017\)J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska, D\. Hassabis, C\. Clopath, D\. Kumaran, and R\. HadsellOvercoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.External Links:[Document](https://dx.doi.org/10.1073/pnas.1611835114),[Link](https://www.pnas.org/doi/abs/10.1073/pnas.1611835114),https://www\.pnas\.org/doi/pdf/10\.1073/pnas\.1611835114Cited by:[§4\.5](https://arxiv.org/html/2609.16537#S4.SS5.SSS0.Px1.p1.1)\.
- Krishnaet al\.\(2024\)S\. Krishna, T\. Han, A\. Gu, S\. Wu, S\. Jabbari, and H\. LakkarajuThe disagreement problem in explainable machine learning: a practitioner’s perspective\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=jESY2WTZCe)Cited by:[§A\.7](https://arxiv.org/html/2609.16537#A1.SS7.p2.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px3.p1.1)\.
- Menet al\.\(2025\)X\. Men, M\. Xu, Q\. Zhang, Q\. Yuan, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. ChenShortGPT: layers in large language models are more redundant than you expect\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 20192–20204\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1035/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1035),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2609.16537#S1.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px2.p1.1)\.
- Menget al\.\(2023\)K\. Meng, D\. Bau, A\. Andonian, and Y\. BelinkovLocating and editing factual associations in gpt\.External Links:2202\.05262,[Link](https://arxiv.org/abs/2202.05262)Cited by:[§1](https://arxiv.org/html/2609.16537#S1.p1.1)\.
- Munozet al\.\(2025\)J\. P\. Munoz, J\. Yuan, and N\. JainMamba\-shedder: post\-transformer compression for efficient selective structured state space models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 3851–3863\.External Links:[Link](https://aclanthology.org/2025.naacl-long.195/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.195),ISBN 979\-8\-89176\-189\-6Cited by:[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px1.p1.1)\.
- Naimet al\.\(2026\)O\. Naim, K\. Sharma, N\. R\. Barman, and N\. AsherTELL\-tale: task efficient llms with task aware layer elimination\.External Links:2510\.22767,[Link](https://arxiv.org/abs/2510.22767)Cited by:[§D\.1](https://arxiv.org/html/2609.16537#A4.SS1.p1.1),[Table 7](https://arxiv.org/html/2609.16537#A4.T7.10.1.17.1),[Table 7](https://arxiv.org/html/2609.16537#A4.T7.10.1.21.1),[Table 7](https://arxiv.org/html/2609.16537#A4.T7.10.1.9.1),[§1](https://arxiv.org/html/2609.16537#S1.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px2.p1.1)\.
- Nepalet al\.\(2025\)A\. Nepal, S\. Shrestha, A\. Shrestha, M\. Kim, J\. Naghiyev, R\. Shwartz\-Ziv, and K\. RossLayer importance for mathematical reasoning is forged in pre\-training and invariant after post\-training\.External Links:2506\.22638,[Link](https://arxiv.org/abs/2506.22638)Cited by:[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px2.p1.1)\.
- Panet al\.\(2024\)R\. Pan, X\. Liu, S\. Diao, R\. Pi, J\. Zhang, C\. Han, and T\. ZhangLISA: layerwise importance sampling for memory\-efficient large language model fine\-tuning\.External Links:2403\.17919,[Link](https://arxiv.org/abs/2403.17919)Cited by:[§1](https://arxiv.org/html/2609.16537#S1.p1.1)\.
- Penget al\.\(2023\)B\. Peng, E\. Alcaide, Q\. Anthony, A\. Albalak, S\. Arcadinho, S\. Biderman, H\. Cao, X\. Cheng, M\. Chung, M\. Grella, K\. K\. GV, X\. He, H\. Hou, J\. Lin, P\. Kazienko, J\. Kocon, J\. Kong, B\. Koptyra, H\. Lau, K\. S\. I\. Mantri, F\. Mom, A\. Saito, G\. Song, X\. Tang, B\. Wang, J\. S\. Wind, S\. Wozniak, R\. Zhang, Z\. Zhang, Q\. Zhao, P\. Zhou, Q\. Zhou, J\. Zhu, and R\. ZhuRWKV: reinventing rnns for the transformer era\.External Links:2305\.13048,[Link](https://arxiv.org/abs/2305.13048)Cited by:[Table 1](https://arxiv.org/html/2609.16537#A2.T1.2.1.16.2),[§4\.1](https://arxiv.org/html/2609.16537#S4.SS1.SSS0.Px2.p1.1)\.
- Qinget al\.\(2024\)P\. Qing, C\. Gao, Y\. Zhou, X\. Diao, Y\. Yang, and S\. VosoughiAlphaLoRA: assigning LoRA experts based on layer training quality\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 20511–20523\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1141/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1141)Cited by:[§B\.1](https://arxiv.org/html/2609.16537#A2.SS1.p1.1),[§1](https://arxiv.org/html/2609.16537#S1.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px1.p1.1)\.
- Shiet al\.\(2025\)G\. Shi, Z\. Lu, X\. Dong, W\. Zhang, X\. Zhang, Y\. Feng, and X\. WuUnderstanding layer significance in llm alignment\.External Links:2410\.17875,[Link](https://arxiv.org/abs/2410.17875)Cited by:[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px2.p1.1)\.
- Songet al\.\(2026\)X\. Song, K\. Wang, P\. Li, L\. Yin, and S\. LiuDemystifying the roles of llm layers in retrieval, knowledge, and reasoning\.External Links:2510\.02091,[Link](https://arxiv.org/abs/2510.02091)Cited by:[§1](https://arxiv.org/html/2609.16537#S1.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px2.p1.1)\.
- Sundararajanet al\.\(2017\)M\. Sundararajan, A\. Taly, and Q\. YanAxiomatic attribution for deep networks\.External Links:1703\.01365,[Link](https://arxiv.org/abs/1703.01365)Cited by:[§A\.1](https://arxiv.org/html/2609.16537#A1.SS1.p1.2),[§A\.7](https://arxiv.org/html/2609.16537#A1.SS7.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px3.p1.1)\.
- Vaswaniet al\.\(2023\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. PolosukhinAttention is all you need\.External Links:1706\.03762,[Link](https://arxiv.org/abs/1706.03762)Cited by:[§A\.3](https://arxiv.org/html/2609.16537#A1.SS3.SSS0.Px1.p2.1),[§1](https://arxiv.org/html/2609.16537#S1.p1.1)\.
- Xuet al\.\(2026\)Y\. Xu, Y\. Liang, S\. Dai, T\. Hu, T\. N\. Chan, and C\. MaUnderstanding and guiding layer placement in parameter\-efficient fine\-tuning of large language models\.External Links:2602\.04019,[Link](https://arxiv.org/abs/2602.04019)Cited by:[§B\.1](https://arxiv.org/html/2609.16537#A2.SS1.p1.1),[§B\.1](https://arxiv.org/html/2609.16537#A2.SS1.p2.1),[Table 7](https://arxiv.org/html/2609.16537#A4.T7.10.1.14.1),[Table 7](https://arxiv.org/html/2609.16537#A4.T7.10.1.6.1),[§1](https://arxiv.org/html/2609.16537#S1.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[Table 1](https://arxiv.org/html/2609.16537#A2.T1.2.1.2.1.1)\.
- Yaoet al\.\(2024\)K\. Yao, P\. Gao, L\. Li, Y\. Zhao, X\. Wang, W\. Wang, and J\. ZhuLayer\-wise importance matters: less memory for better performance in parameter\-efficient fine\-tuning of large language models\.External Links:2410\.11772,[Link](https://arxiv.org/abs/2410.11772)Cited by:[§1](https://arxiv.org/html/2609.16537#S1.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2023\)Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. ZhaoAdaLoRA: adaptive budget allocation for parameter\-efficient fine\-tuning\.External Links:2303\.10512,[Link](https://arxiv.org/abs/2303.10512)Cited by:[§B\.1](https://arxiv.org/html/2609.16537#A2.SS1.p1.1),[§1](https://arxiv.org/html/2609.16537#S1.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2024\)Y\. Zhang, Y\. Dong, and K\. KawaguchiInvestigating layer importance in large language models\.InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,Y\. Belinkov, N\. Kim, J\. Jumelet, H\. Mohebbi, A\. Mueller, and H\. Chen \(Eds\.\),Miami, Florida, US,pp\. 469–479\.External Links:[Link](https://aclanthology.org/2024.blackboxnlp-1.29/),[Document](https://dx.doi.org/10.18653/v1/2024.blackboxnlp-1.29)Cited by:[§1](https://arxiv.org/html/2609.16537#S1.p1.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2026\)Y\. Zhao, Q\. Yao, X\. song, and W\. ZhuShapLoRA: allocation of low\-rank adaption on large language models via shapley value inspired importance estimation\.External Links:2601\.17921,[Link](https://arxiv.org/abs/2601.17921)Cited by:[§B\.1](https://arxiv.org/html/2609.16537#A2.SS1.p1.1),[§D\.1](https://arxiv.org/html/2609.16537#A4.SS1.p1.1),[Table 7](https://arxiv.org/html/2609.16537#A4.T7.10.1.16.1),[Table 7](https://arxiv.org/html/2609.16537#A4.T7.10.1.20.1),[Table 7](https://arxiv.org/html/2609.16537#A4.T7.10.1.8.1),[§2](https://arxiv.org/html/2609.16537#S2.SS0.SSS0.Px1.p1.1)\.
- Zuoet al\.\(2024\)J\. Zuo, M\. Velikanov, D\. E\. Rhaiem, I\. Chahed, Y\. Belkada, G\. Kunsch, and H\. HacidFalcon mamba: the first competitive attention\-free 7b language model\.External Links:2410\.05355,[Link](https://arxiv.org/abs/2410.05355)Cited by:[Table 1](https://arxiv.org/html/2609.16537#A2.T1.2.1.17.1),[§4\.1](https://arxiv.org/html/2609.16537#S4.SS1.SSS0.Px2.p1.1)\.

## Appendix AFormal Theory: Necessity and Plasticity as Distinct Functionals

The body of the paper argues, informally, that ablation and LoRA delta should be expected to disagree on residual transformers and to agree on Mamba\. This appendix gives that argument a formal backbone\. We state three theorems and one corollary that together: \(i\) separate ablation and LoRA delta as projections of the loss landscape onto two structurally distinct subspaces \(Theorem[A\.1](https://arxiv.org/html/2609.16537#A1.Thmtheorem1)\); \(ii\) prove a rigorous*mass\-displacement*bound for residual transformers \(Theorem[A\.3](https://arxiv.org/html/2609.16537#A1.Thmtheorem3),δ=I~a​\(E\)\+I~Δ​\(E¯\)−1≥τ−L​\(1\+β\)L/\(κ\+L​\(1\+β\)L\)\\delta=\\tilde\{I\}\_\{a\}\(E\)\+\\tilde\{I\}\_\{\\Delta\}\(\\bar\{E\}\)\-1\\geq\\tau\-L\(1\+\\beta\)^\{L\}/\(\\kappa\+L\(1\+\\beta\)^\{L\}\)\) and convert it to a Spearman bound under an explicit profile\-shape condition \(Lemma[A\.4](https://arxiv.org/html/2609.16537#A1.Thmtheorem4)\); \(iii\) show that the analogous bound*vanishes*for state\-space recurrences satisfying an expected symmetric\-mixing condition \(Theorem[A\.6](https://arxiv.org/html/2609.16537#A1.Thmtheorem6)\); and \(iv\) explain, as Corollary[A\.8](https://arxiv.org/html/2609.16537#A1.Thmtheorem8), why bimodal LoRA\-delta profiles drive\|ρs\|\|\\rho\_\{s\}\|toward zero rather than reversing its sign\.

A note on what is and is not proven\. Theorems[A\.3](https://arxiv.org/html/2609.16537#A1.Thmtheorem3)and[A\.6](https://arxiv.org/html/2609.16537#A1.Thmtheorem6)are rigorous mass\-displacement statements that follow directly from the assumptions\. The Spearman correlation bound is a*separate*statement, conditional on a profile\-shape assumption \(monotonicity or unimodality\) that we state explicitly and verify empirically rather than derive from the architectural assumptions alone\. This separation makes the load\-bearing premises visible\.

We use the same notation throughout:θ0\\theta\_\{0\}are the pretrained parameters,Lt​\(θ\)L\_\{t\}\(\\theta\)is the task loss,gl=∇θlLt​\(θ0\)g\_\{l\}=\\nabla\_\{\\theta\_\{l\}\}L\_\{t\}\(\\theta\_\{0\}\),Hl​l=∇θl​θl2​Lt​\(θ0\)H\_\{ll\}=\\nabla^\{2\}\_\{\\theta\_\{l\}\\theta\_\{l\}\}L\_\{t\}\(\\theta\_\{0\}\)is the layer\-local Hessian block,𝒮l\\mathcal\{S\}\_\{l\}is the LoRA\-accessible subspace at layerllwith orthogonal projectorP𝒮lP\_\{\\mathcal\{S\}\_\{l\}\}, andλmin\(l\),λmax\(l\)\\lambda\_\{\\min\}^\{\(l\)\},\\lambda\_\{\\max\}^\{\(l\)\}are the extremal eigenvalues ofHl​lH\_\{ll\}restricted to𝒮l\\mathcal\{S\}\_\{l\}\. We writeflf\_\{l\}for the layer\-local map \(residual block in transformers, SSM block in Mamba\) andJfj=∂fj/∂hj−1J\_\{f\_\{j\}\}=\\partial f\_\{j\}/\\partial h\_\{j\-1\}for its activation Jacobian\.

### A\.1The three estimators are distinct functionals of the loss

We first record, in one place, that gradient attribution, LoRA delta, and ablation are not three noisy estimates of one quantity but three distinct functionals\. Writingθt⋆=θ0\+Δ​θt\\theta^\{\\star\}\_\{t\}=\\theta\_\{0\}\+\\Delta\\theta\_\{t\}for a LoRA\-constrained local minimizer reached fromθ0\\theta\_\{0\}on tasktt,

Ig​\(l,t\)\\displaystyle I\_\{g\}\(l,t\)=𝔼x∼Dt​\[∥∇θlLt​\(θ0,x\)∥2\],\\displaystyle\\;=\\;\\mathbb\{E\}\_\{x\\sim D\_\{t\}\}\\\!\\bigl\[\\,\\lVert\\nabla\_\{\\theta\_\{l\}\}L\_\{t\}\(\\theta\_\{0\};x\)\\rVert\_\{2\}\\,\\bigr\],\(1\)IΔ​\(l,t\)\\displaystyle I\_\{\\Delta\}\(l,t\)=∥\(θt⋆\)l−θl0∥F,\\displaystyle\\;=\\;\\lVert\(\\theta^\{\\star\}\_\{t\}\)\_\{l\}\-\\theta^\{0\}\_\{l\}\\rVert\_\{F\},\(2\)Ia​\(l,t\)\\displaystyle I\_\{a\}\(l,t\)=Lt​\(θ0∖l\)−Lt​\(θ0\)\.\\displaystyle\\;=\\;L\_\{t\}\\\!\\bigl\(\\theta\_\{0\}\\setminus l\\bigr\)\-L\_\{t\}\(\\theta\_\{0\}\)\.\(3\)All three are task\-indexed; we suppress the task argument when context is clear\. Equation \([1](https://arxiv.org/html/2609.16537#A1.E1)\) is a*tangent*: a first\-order sensitivity atθ0\\theta\_\{0\}\. Equation \([2](https://arxiv.org/html/2609.16537#A1.E2)\) is a*chord*: the magnitude of the converged update over the full optimization trajectory\. Equation \([3](https://arxiv.org/html/2609.16537#A1.E3)\) is a*finite\-difference jump*: a non\-infinitesimal intervention\. The three coincide only ifLtL\_\{t\}is exactly quadratic and isotropic on every layer, or if the ablation perturbationθ0∖l\\theta\_\{0\}\\setminus lis infinitesimal\. Neither holds for a deep transformer\([Sundararajan et al\., 2017](https://arxiv.org/html/2609.16537#bib.bib24);[Ancona et al\., 2018](https://arxiv.org/html/2609.16537#bib.bib25);[Adebayo et al\., 2020](https://arxiv.org/html/2609.16537#bib.bib26)\), so disagreement is the generic case and agreement is the exception\. The decomposition lemma below makes the source of disagreement explicit\.

A standard quadratic expansion gives the curvature decomposition

IΔ​\(l\)≈∥Hl​l−1​gl∥F∈\[∥gl∥Fλmax\(l\),∥gl∥Fλmin\(l\)\]I\_\{\\Delta\}\(l\)\\;\\approx\\;\\lVert H\_\{ll\}^\{\-1\}\\,g\_\{l\}\\rVert\_\{F\}\\;\\in\\;\\left\[\\frac\{\\lVert g\_\{l\}\\rVert\_\{F\}\}\{\\lambda\_\{\\max\}^\{\(l\)\}\},\\;\\frac\{\\lVert g\_\{l\}\\rVert\_\{F\}\}\{\\lambda\_\{\\min\}^\{\(l\)\}\}\\right\]\(4\)in the unconstrained case, and the LoRA\-restricted analogue‖ul⋆‖F∈\[‖P𝒮l​gl‖F/λmax\(l\),‖P𝒮l​gl‖F/λmin\(l\)\]\\\|u\_\{l\}^\{\\star\}\\\|\_\{F\}\\in\[\\,\\\|P\_\{\\mathcal\{S\}\_\{l\}\}g\_\{l\}\\\|\_\{F\}/\\lambda\_\{\\max\}^\{\(l\)\},\\;\\\|P\_\{\\mathcal\{S\}\_\{l\}\}g\_\{l\}\\\|\_\{F\}/\\lambda\_\{\\min\}^\{\(l\)\}\\,\]in the low\-rank case\([Jacot et al\., 2020](https://arxiv.org/html/2609.16537#bib.bib30);[Aghajanyan et al\., 2021](https://arxiv.org/html/2609.16537#bib.bib31);[Hu et al\., 2021](https://arxiv.org/html/2609.16537#bib.bib9)\)\. For ablation, a second\-order expansion ofLtL\_\{t\}along the direction−θl0\-\\theta\_\{l\}^\{0\}that zeros out the layer’s contribution gives

Ia​\(l\)≈−gl⊤​θl0\+12​\(θl0\)⊤​Hl​l​θl0\+R⁡\(θl0\),I\_\{a\}\(l\)\\;\\approx\\;\-\\,g\_\{l\}^\{\\top\}\\,\\theta\_\{l\}^\{0\}\\;\+\\;\\tfrac\{1\}\{2\}\\,\(\\theta\_\{l\}^\{0\}\)^\{\\top\}\\,H\_\{ll\}\\,\\theta\_\{l\}^\{0\}\\;\+\\;R\(\\theta\_\{l\}^\{0\}\),\(5\)whereR⁡\(θl0\)R\(\\theta\_\{l\}^\{0\}\)collects higher\-order terms that are non\-negligible becauseθl0\\theta\_\{l\}^\{0\}is large\. Equation \([5](https://arxiv.org/html/2609.16537#A1.E5)\) is dominated by the curvature term whenever the pretrained weights have large norm and are approximately aligned with the dominant Hessian eigenvectors — a condition that is structurally favored at high\-norm early\-layer weight matrices\. Combining \([4](https://arxiv.org/html/2609.16537#A1.E4)\) and \([5](https://arxiv.org/html/2609.16537#A1.E5)\) gives the qualitative reading: a layer that is high\-gradient*and*high\-curvature*and*large\-norm scores high under ablation but low under LoRA delta, while a layer that is high\-gradient*and*low\-curvature scores high under LoRA delta but moderate under ablation\. The two estimators therefore anti\-correlate when those two regimes are spatially separated across depth\. The next theorem makes this geometric picture rigorous\.

### A\.2Theorem 1: Necessity\-Plasticity Decomposition

Define, at each layerll,

νl\\displaystyle\\nu\_\{l\}\\;:=fl​\(hl−1\)\\displaystyle:=\\;f\_\{l\}\(h\_\{l\-1\}\)\(Necessity dir\.\),\\displaystyle\\text\{\(Necessity dir\.\)\},\(6\)Πl\\displaystyle\\Pi\_\{l\}\\;:=∏j=l\+1L\(I\+Jfj​\(hj−1\)\)\\displaystyle:=\\;\\prod\_\{j=l\+1\}^\{L\}\\bigl\(I\+J\_\{f\_\{j\}\}\(h\_\{j\-1\}\)\\bigr\)\(propagator\),\\displaystyle\\text\{\(propagator\)\},\(7\)πl\\displaystyle\\pi\_\{l\}\\;:=Hl​l−1​P𝒮l​gl\\displaystyle:=\\;H\_\{ll\}^\{\-1\}\\,P\_\{\\mathcal\{S\}\_\{l\}\}\\,g\_\{l\}\(Plasticity dir\.\)\.\\displaystyle\\text\{\(Plasticity dir\.\)\}\.\(8\)The Necessity directionνl\\nu\_\{l\}records the contribution that ablation removes from the residual stream; the propagatorΠl\\Pi\_\{l\}is the multilinear operator that maps a layer\-llactivation perturbation to its effect on the output representationhLh\_\{L\}; the Plasticity directionπl\\pi\_\{l\}is the LoRA\-restricted Newton step at layerll\.

###### Theorem A\.1\(Local decomposition\)\.

Under aC2C^\{2\}loss and bounded operator norms‖Jfj‖<∞\\\|J\_\{f\_\{j\}\}\\\|<\\infty, the ablation and LoRA\-delta importances admit the following first\-order expansions aroundθ0\\theta\_\{0\}:

Ia​\(l,t\)\\displaystyle I\_\{a\}\(l,t\)\\;=𝔼x​\[⟨∂Lt∂hL​\(θ0,x\),Πl​\(x\)​νl​\(x\)⟩\]\\displaystyle=\\;\\mathbb\{E\}\_\{x\}\\\!\\left\[\\left\\langle\\tfrac\{\\partial L\_\{t\}\}\{\\partial h\_\{L\}\}\(\\theta\_\{0\};x\),\\;\\Pi\_\{l\}\(x\)\\,\\nu\_\{l\}\(x\)\\right\\rangle\\right\]\+Ra​\(l\),\\displaystyle\\quad\+R\_\{a\}\(l\),\(9\)IΔ​\(l,t\)\\displaystyle I\_\{\\Delta\}\(l,t\)\\;=‖πl‖F\+RΔ​\(l\),\\displaystyle=\\;\\\|\\pi\_\{l\}\\\|\_\{F\}\+R\_\{\\Delta\}\(l\),\(10\)with remaindersRa​\(l\)=O⁡\(𝔼x​‖νl‖2\)R\_\{a\}\(l\)=O\(\\mathbb\{E\}\_\{x\}\\\|\\nu\_\{l\}\\\|^\{2\}\)andRΔ​\(l\)=o⁡\(‖P𝒮l​gl‖F\)R\_\{\\Delta\}\(l\)=o\(\\\|P\_\{\\mathcal\{S\}\_\{l\}\}g\_\{l\}\\\|\_\{F\}\)in the LoRA\-NTK regime\([Jacot et al\., 2020](https://arxiv.org/html/2609.16537#bib.bib30);[Aghajanyan et al\., 2021](https://arxiv.org/html/2609.16537#bib.bib31)\)\. Consequently,

Covl​\(Ia,IΔ\)=Covl​\(⟨∂Lt∂hL,Πl​νl⟩,‖πl‖F\)\+O⁡\(maxl⁡\(Ra​\(l\)\+RΔ​\(l\)\)\),\\mathrm\{Cov\}\_\{l\}\\bigl\(I\_\{a\},I\_\{\\Delta\}\\bigr\)\\;=\\;\\mathrm\{Cov\}\_\{l\}\\\!\\left\(\\bigl\\langle\\tfrac\{\\partial L\_\{t\}\}\{\\partial h\_\{L\}\},\\,\\Pi\_\{l\}\\nu\_\{l\}\\bigr\\rangle,\\;\\\|\\pi\_\{l\}\\\|\_\{F\}\\right\)\\\\ \+O\\\!\\bigl\(\\max\_\{l\}\(R\_\{a\}\(l\)\+R\_\{\\Delta\}\(l\)\)\\bigr\),\(11\)where the covariance on the right is taken across layersl∈\{0,…,L\}l\\in\\\{0,\\ldots,L\\\}\.

###### Proof sketch\.

Equation \([9](https://arxiv.org/html/2609.16537#A1.E9)\) is a Taylor expansion ofLtL\_\{t\}in activation space along the perturbation−νl\-\\nu\_\{l\}at layerll, propagated tohLh\_\{L\}by the chain rule; the propagatorΠl\\Pi\_\{l\}collects the downstream Jacobians, and the second\-order remainder isO⁡\(‖νl‖2\)O\(\\\|\\nu\_\{l\}\\\|^\{2\}\)by Lagrange’s form of Taylor’s theorem applied to theC2C^\{2\}loss\. Equation \([10](https://arxiv.org/html/2609.16537#A1.E10)\) follows from \([4](https://arxiv.org/html/2609.16537#A1.E4)\) restricted to𝒮l\\mathcal\{S\}\_\{l\}: in the LoRA\-NTK regime, the converged update is the constrained Newton step up to vanishing kernel\-feature drift\([Jacot et al\., 2020](https://arxiv.org/html/2609.16537#bib.bib30);[Aghajanyan et al\., 2021](https://arxiv.org/html/2609.16537#bib.bib31)\)\. The covariance identity \([11](https://arxiv.org/html/2609.16537#A1.E11)\) is a linear\-in\-leading\-order manipulation of \([9](https://arxiv.org/html/2609.16537#A1.E9)\)\-\([10](https://arxiv.org/html/2609.16537#A1.E10)\)\. ∎

###### Corollary A\.2\(Orthogonal\-decomposition lemma\)\.

If for everyllthe propagated necessity directionΠl​νl\\Pi\_\{l\}\\nu\_\{l\}and the plasticity directionπl\\pi\_\{l\}are uncorrelated when ranked acrossll\(equivalently, their layer\-indexed magnitudes have rank correlation zero\), thenρ⁡\(Ia,IΔ\)=O⁡\(maxl⁡\(Ra​\(l\)\+RΔ​\(l\)\)\)\\rho\(I\_\{a\},I\_\{\\Delta\}\)=O\(\\max\_\{l\}\(R\_\{a\}\(l\)\+R\_\{\\Delta\}\(l\)\)\), i\.e\. the two estimators are uncorrelated up to the joint Taylor remainder\.

The structural content of Theorem[A\.1](https://arxiv.org/html/2609.16537#A1.Thmtheorem1)is thatIaI\_\{a\}andIΔI\_\{\\Delta\}each project the loss landscape onto a different local subspace: ablation onto the activation\-space subspace spanned byΠl​νl\\Pi\_\{l\}\\nu\_\{l\}, LoRA delta onto the parameter\-space subspace spanned byπl\\pi\_\{l\}\. The two subspaces are coupled only through the shared lossLtL\_\{t\}; they are not in general aligned\. The theorem says*nothing*yet about the sign of the cross\-method correlation: agreement, disagreement, and exact orthogonality are all consistent with \([11](https://arxiv.org/html/2609.16537#A1.E11)\)\. Sign is the role of Theorems[A\.3](https://arxiv.org/html/2609.16537#A1.Thmtheorem3)and[A\.6](https://arxiv.org/html/2609.16537#A1.Thmtheorem6)\.

### A\.3Theorem 2: Mass\-Displacement Bound for Residual Transformers

We split the formal argument into two stages\. Stage A is a rigorous*mass\-displacement*bound: the ablation profile concentrates on the early half of layers, and the LoRA\-delta profile concentrates on the late half, with explicit bounds derived from the assumptions\. Stage B is a*rank\-correlation lemma*\(Lemma[A\.4](https://arxiv.org/html/2609.16537#A1.Thmtheorem4)\) that converts mass displacement into a Spearman bound under an additional shape assumption\. Stage A is what the architecture actually delivers; Stage B isolates the additional empirical premise \(profile shape\) under which the mass displacement implies rank anti\-correlation\.

##### Notation\.

For a nonnegative profileI:\{0,…,L\}→ℝ≥0I:\\\{0,\\ldots,L\\\}\\to\\mathbb\{R\}\_\{\\geq 0\}writeI~=I/∑lI⁡\(l\)\\tilde\{I\}=I/\\sum\_\{l\}I\(l\)for its normalization to a probability mass function\. For a subsetE⊆\{0,…,L\}E\\subseteq\\\{0,\\ldots,L\\\}, writeI~​\(E\):=∑l∈EI~​\(l\)\\tilde\{I\}\(E\):=\\sum\_\{l\\in E\}\\tilde\{I\}\(l\)for the mass onEE\. TakeE:=\{0,1,…,⌊L/2⌋\}E:=\\\{0,1,\\ldots,\\lfloor L/2\\rfloor\\\}\(the early half\) andE¯:=\{0,…,L\}∖E\\bar\{E\}:=\\\{0,\\ldots,L\\\}\\setminus E\(the late half\)\. We work in the linearization regime of Theorem[A\.1](https://arxiv.org/html/2609.16537#A1.Thmtheorem1)\(remainders dropped for clarity; they can be carried explicitly with no change in the proof\)\.

###### Assumption A\.1\(Residual transformer regime\)\.

The architecture is anLL\-layer Pre\-LN residual transformer satisfying:

\(A1\) Bounded block Lipschitz\.There existsβ<∞\\beta<\\inftywith‖Jfj​\(h\)‖≤β\\\|J\_\{f\_\{j\}\}\(h\)\\\|\\leq\\betauniformly overj∈\{1,…,L\}j\\in\\\{1,\\ldots,L\\\}and over the data distribution\.

\(A2\) Privileged input injection\.The residual streamh0h\_\{0\}is the embeddingEmb⁡\(x\)\\mathrm\{Emb\}\(x\),f0f\_\{0\}is the unique signal\-injection block \(no path bypassesf0f\_\{0\}\)\. Moreover, the contribution norms satisfy𝔼x​‖f0​\(h−1\)‖≥μ0\\mathbb\{E\}\_\{x\}\\\|f\_\{0\}\(h\_\{\-1\}\)\\\|\\geq\\mu\_\{0\}and𝔼x​‖fl​\(hl−1\)‖≤μ\>0\\mathbb\{E\}\_\{x\}\\\|f\_\{l\}\(h\_\{l\-1\}\)\\\|\\leq\\mu\_\{\>0\}forl\>0l\>0, withμ0/μ\>0≥κ\\mu\_\{0\}/\\mu\_\{\>0\}\\geq\\kappafor someκ≥1\\kappa\\geq 1\.

\(A3\) Terminal\-task alignment\.There existsτ∈\(1/2,1\]\\tau\\in\(1/2,1\]such that

∑l∈E¯‖P𝒮l​gl‖F2λmax\(l\)≥τ​∑l=0L‖P𝒮l​gl‖F2λmax\(l\)\.\\sum\_\{l\\in\\bar\{E\}\}\\frac\{\\\|P\_\{\\mathcal\{S\}\_\{l\}\}g\_\{l\}\\\|\_\{F\}^\{2\}\}\{\\lambda\_\{\\max\}^\{\(l\)\}\}\\;\\geq\\;\\tau\\,\\sum\_\{l=0\}^\{L\}\\frac\{\\\|P\_\{\\mathcal\{S\}\_\{l\}\}g\_\{l\}\\\|\_\{F\}^\{2\}\}\{\\lambda\_\{\\max\}^\{\(l\)\}\}\.

\(A1\) is standard for trained transformers\([Vaswani et al\., 2023](https://arxiv.org/html/2609.16537#bib.bib1)\); \(A2\) is a structural property of the Pre\-LN residual stack with the quantitative ratioκ\\kappaverified empirically in Section[5\.3](https://arxiv.org/html/2609.16537#S5.SS3)\(transformers concentrate∼62%\\sim 62\\%of ablation mass onf0f\_\{0\}alone\); \(A3\) encodes the fact that the cross\-entropy loss attaches at the unembedding, so LoRA\-accessible subspaces in late layers carry strong projected gradient\.

#### Stage A: Mass\-displacement bound \(rigorous\)

###### Theorem A\.3\(Mass displacement under residual structure\)\.

Under Assumption[A\.1](https://arxiv.org/html/2609.16537#A1.Thmassumption1),

I~a​\(E\)\\displaystyle\\tilde\{I\}\_\{a\}\(E\)\\;≥κκ\+L​\(1\+β\)L,\\displaystyle\\geq\\;\\frac\{\\kappa\}\{\\kappa\+L\\,\(1\+\\beta\)^\{L\}\},\(12\)I~Δ​\(E¯\)\\displaystyle\\tilde\{I\}\_\{\\Delta\}\(\\bar\{E\}\)\\;≥τ\.\\displaystyle\\geq\\;\\tau\.\(13\)Define the*mass\-disagreement coefficient*δ:=I~a​\(E\)\+I~Δ​\(E¯\)−1\.\\delta:=\\tilde\{I\}\_\{a\}\(E\)\+\\tilde\{I\}\_\{\\Delta\}\(\\bar\{E\}\)\-1\.Then

δ≥τ−L​\(1\+β\)Lκ\+L​\(1\+β\)L\.\\delta\\;\\geq\\;\\tau\\;\-\\;\\frac\{L\\,\(1\+\\beta\)^\{L\}\}\{\\kappa\+L\\,\(1\+\\beta\)^\{L\}\}\.\(14\)

###### Proof\.

We work under one further mild non\-degeneracy condition, made explicit:

\(A4\) Generic non\-cancellation\.There existsc\>0c\>0such that𝔼x​⟨∂Lt/∂hL,ν0​\(x\)⟩≥c⋅𝔼x​‖ν0​\(x\)‖\.\\mathbb\{E\}\_\{x\}\\langle\\partial L\_\{t\}/\\partial h\_\{L\},\\;\\nu\_\{0\}\(x\)\\rangle\\;\\geq\\;c\\cdot\\mathbb\{E\}\_\{x\}\\\|\\nu\_\{0\}\(x\)\\\|\.

That is, the embedding contributionν0\\nu\_\{0\}has nontrivial expected alignment with the loss gradient at the output; this rules out the degenerate case wheref0f\_\{0\}produces a contribution exactly orthogonal to the task signal in expectation\.

*Bound onI~a​\(E\)\\tilde\{I\}\_\{a\}\(E\)\.*By Theorem[A\.1](https://arxiv.org/html/2609.16537#A1.Thmtheorem1)and the Cauchy–Schwarz inequality applied to \([9](https://arxiv.org/html/2609.16537#A1.E9)\),

Ia​\(l\)≤𝔼x​‖νl​\(x\)‖⋅‖Πl‖⋅‖∂Lt/∂hL‖∞\.I\_\{a\}\(l\)\\;\\leq\\;\\mathbb\{E\}\_\{x\}\\\|\\nu\_\{l\}\(x\)\\\|\\cdot\\\|\\Pi\_\{l\}\\\|\\;\\cdot\\;\\\|\\partial L\_\{t\}/\\partial h\_\{L\}\\\|\_\{\\infty\}\.Under \(A1\),‖Πl‖≤\(1\+β\)L−l≤\(1\+β\)L\\\|\\Pi\_\{l\}\\\|\\leq\(1\+\\beta\)^\{L\-l\}\\leq\(1\+\\beta\)^\{L\}for allll\. HenceIa​\(l\)≤C⋅μl⋅\(1\+β\)LI\_\{a\}\(l\)\\leq C\\cdot\\mu\_\{l\}\\cdot\(1\+\\beta\)^\{L\}for a common constantCCabsorbing‖∂Lt/∂hL‖∞\\\|\\partial L\_\{t\}/\\partial h\_\{L\}\\\|\_\{\\infty\}, withμ0=𝔼x​‖f0​\(h−1\)‖\\mu\_\{0\}=\\mathbb\{E\}\_\{x\}\\\|f\_\{0\}\(h\_\{\-1\}\)\\\|andμl=𝔼x​‖fl​\(hl−1\)‖\\mu\_\{l\}=\\mathbb\{E\}\_\{x\}\\\|f\_\{l\}\(h\_\{l\-1\}\)\\\|\. Forl=0l=0, expandΠ0=I\+\(Π0−I\)\\Pi\_\{0\}=I\+\(\\Pi\_\{0\}\-I\)where the identity contribution corresponds to the trivial residual path\. Then

Ia​\(0\)=𝔼x​⟨∂Lt/∂hL,Π0​ν0⟩=𝔼x​⟨∂Lt/∂hL,ν0⟩\+𝔼x​⟨∂Lt/∂hL,\(Π0−I\)​ν0⟩\.I\_\{a\}\(0\)\\;=\\;\\mathbb\{E\}\_\{x\}\\langle\\partial L\_\{t\}/\\partial h\_\{L\},\\Pi\_\{0\}\\nu\_\{0\}\\rangle\\\\ \\;=\\;\\mathbb\{E\}\_\{x\}\\langle\\partial L\_\{t\}/\\partial h\_\{L\},\\nu\_\{0\}\\rangle\\\\ \\;\+\\;\\mathbb\{E\}\_\{x\}\\langle\\partial L\_\{t\}/\\partial h\_\{L\},\(\\Pi\_\{0\}\-I\)\\nu\_\{0\}\\rangle\.The first term is≥c​μ0\\geq c\\,\\mu\_\{0\}by \(A4\)\. The second term has magnitude at mostC⋅μ0⋅\(\(1\+β\)L−1\)C\\cdot\\mu\_\{0\}\\cdot\(\(1\+\\beta\)^\{L\}\-1\)but does not cancel the first under \(A4\)\. Absorbing constants and signs intoCCgives the lower boundIa​\(0\)≥c′​μ0I\_\{a\}\(0\)\\;\\geq\\;c^\{\\prime\}\\,\\mu\_\{0\}for somec′\>0c^\{\\prime\}\>0independent ofβ,L\\beta,L\. Combining with the upper bound on∑l≥1Ia​\(l\)\\sum\_\{l\\geq 1\}I\_\{a\}\(l\),

I~a​\(E\)≥I~a​\(\{0\}\)=Ia​\(0\)∑lIa​\(l\)≥c′​μ0c′​μ0\+C​μ\>0​∑l=1L\(1\+β\)L−l\.\\tilde\{I\}\_\{a\}\(E\)\\;\\geq\\;\\tilde\{I\}\_\{a\}\(\\\{0\\\}\)\\;=\\;\\frac\{I\_\{a\}\(0\)\}\{\\sum\_\{l\}I\_\{a\}\(l\)\}\\\\ \\;\\geq\\;\\frac\{c^\{\\prime\}\\,\\mu\_\{0\}\}\{c^\{\\prime\}\\,\\mu\_\{0\}\+C\\,\\mu\_\{\>0\}\\sum\_\{l=1\}^\{L\}\(1\+\\beta\)^\{L\-l\}\}\.Bounding∑l=1L\(1\+β\)L−l≤L​\(1\+β\)L\\sum\_\{l=1\}^\{L\}\(1\+\\beta\)^\{L\-l\}\\leq L\(1\+\\beta\)^\{L\}and usingμ0/μ\>0≥κ\\mu\_\{0\}/\\mu\_\{\>0\}\\geq\\kappa, then absorbingc′,Cc^\{\\prime\},Cintoκ\\kappa\(or, equivalently, redefiningκ\\kappaasκ⋅c′/C\\kappa\\cdot c^\{\\prime\}/C\), gives \([12](https://arxiv.org/html/2609.16537#A1.E12)\)\.

*Bound onI~Δ​\(E¯\)\\tilde\{I\}\_\{\\Delta\}\(\\bar\{E\}\)\.*By the LoRA\-NTK regime \([4](https://arxiv.org/html/2609.16537#A1.E4)\) restricted to𝒮l\\mathcal\{S\}\_\{l\},IΔ​\(l\)2≍‖P𝒮l​gl‖F2/\(λmax\(l\)\)2\.I\_\{\\Delta\}\(l\)^\{2\}\\asymp\\\|P\_\{\\mathcal\{S\}\_\{l\}\}g\_\{l\}\\\|\_\{F\}^\{2\}/\(\\lambda\_\{\\max\}^\{\(l\)\}\)^\{2\}\.Multiplying numerator and denominator byλmax\(l\)\\lambda\_\{\\max\}^\{\(l\)\}on the relevant scale and using \(A3\) directly,

∑l∈E¯IΔ​\(l\)2≥τ​∑l=0LIΔ​\(l\)2\.\\sum\_\{l\\in\\bar\{E\}\}I\_\{\\Delta\}\(l\)^\{2\}\\;\\geq\\;\\tau\\,\\sum\_\{l=0\}^\{L\}I\_\{\\Delta\}\(l\)^\{2\}\.SinceIΔ​\(l\)≥0I\_\{\\Delta\}\(l\)\\geq 0, an application of Cauchy–Schwarz gives∑l∈E¯IΔ​\(l\)≥τ​∑lIΔ​\(l\)\\sum\_\{l\\in\\bar\{E\}\}I\_\{\\Delta\}\(l\)\\geq\\tau\\sum\_\{l\}I\_\{\\Delta\}\(l\), which is \([13](https://arxiv.org/html/2609.16537#A1.E13)\)\.

Equation \([14](https://arxiv.org/html/2609.16537#A1.E14)\) follows by adding \([12](https://arxiv.org/html/2609.16537#A1.E12)\) and \([13](https://arxiv.org/html/2609.16537#A1.E13)\) and subtracting11\. ∎

#### Stage B: Rank\-correlation under shape conditions

The Spearman correlation of two depth profiles is not a function of their masses onEEandE¯\\bar\{E\}alone\. Two profiles with identical mass splits can have any Spearman correlation in\[−1,\+1\]\[\-1,\+1\]depending on the within\-region rank pattern\. To convert the mass\-displacement bound of Theorem[A\.3](https://arxiv.org/html/2609.16537#A1.Thmtheorem3)into a Spearman bound, we add an explicit shape assumption\.

###### Lemma A\.4\(Anti\-monotonicity implies rank anti\-correlation\)\.

Letp,q:\{0,…,L\}→ℝ≥0p,q:\\\{0,\\ldots,L\\\}\\to\\mathbb\{R\}\_\{\\geq 0\}be two profiles\. Ifppis non\-increasing andqqis non\-decreasing on\{0,…,L\}\\\{0,\\ldots,L\\\}, and neither is constant, then their Spearman rank correlation satisfiesρs​\(p,q\)≤0\\rho\_\{s\}\(p,q\)\\leq 0, with equality only when one of them is constant\.

###### Proof\.

Spearman’s rank correlation is the Pearson correlation of the rank sequencesrp,rqr\_\{p\},r\_\{q\}\. Ifppis strictly non\-increasing thenrp​\(l\)=\(L\+1\)−lr\_\{p\}\(l\)=\(L\+1\)\-l; ifqqis strictly non\-decreasing thenrq​\(l\)=l\+1r\_\{q\}\(l\)=l\+1\. The two rank sequences are perfectly anti\-monotone, soρs=−1\\rho\_\{s\}=\-1\. For weak monotonicity \(with possible ties\),rpr\_\{p\}andrqr\_\{q\}are still anti\-monotone after average\-rank tie\-breaking, so the Pearson correlation ofrpr\_\{p\}andrqr\_\{q\}is at most00\. Equality requires that one of the rank sequences is constant, i\.e\. that the corresponding profile is constant\. ∎

###### Corollary A\.5\(Spearman bound for residual transformers\)\.

Under Assumption[A\.1](https://arxiv.org/html/2609.16537#A1.Thmassumption1)and the additional shape condition thatIaI\_\{a\}is non\-increasing andIΔI\_\{\\Delta\}is non\-decreasing on\{0,…,L\}\\\{0,\\ldots,L\\\},

ρs​\(Ia,IΔ\)≤0\.\\rho\_\{s\}\(I\_\{a\},I\_\{\\Delta\}\)\\;\\leq\\;0\.The bound is strict wheneverδ\>0\\delta\>0\(Theorem[A\.3](https://arxiv.org/html/2609.16537#A1.Thmtheorem3)\), and approaches−1\-1as the profiles become extremally concentrated atl=0l=0andl=Ll=Lrespectively\.

###### Proof\.

Direct application of Lemma[A\.4](https://arxiv.org/html/2609.16537#A1.Thmtheorem4)\. ∎

### A\.4Theorem 3: Failure of Mass Displacement on Selective SSMs

The mass\-displacement bound of Theorem[A\.3](https://arxiv.org/html/2609.16537#A1.Thmtheorem3)relies on two structural ingredients: a privileged input injection atl=0l=0\(A2\) and terminal\-task alignment of LoRA\-accessible gradient \(A3\)\. The Mamba counterpart shows that selective state\-space recurrences violate the*first*ingredient by construction, and \(in expectation over data\) also weaken the second; consequently, the mass\-displacement coefficientδ\\deltano longer admits a uniform\-in\-LLlower bound\.

###### Assumption A\.2\(Selective state\-space regime\)\.

The architecture is anLL\-block selective SSM satisfying:

\(B1′\) Expected symmetric impulse\-response decay\.The data\-conditional impulse\-response operatorΦj,l​\(x\)\\Phi\_\{j,l\}\(x\)from layerllto layerjjsatisfies

𝔼x​‖Φj,l​\(x\)‖≤κ⁡\(\|j−l\|\),\\mathbb\{E\}\_\{x\}\\\|\\Phi\_\{j,l\}\(x\)\\\|\\leq\\kappa\(\|j\-l\|\),for some non\-increasingκ:ℕ→ℝ\>0\\kappa:\\mathbb\{N\}\\to\\mathbb\{R\}\_\{\>0\}withκ⁡\(0\)=1\\kappa\(0\)=1andκ⁡\(k\)→0\\kappa\(k\)\\to 0\. The bound depends only on\|j−l\|\|j\-l\|*after*expectation over the data distribution\.

\(B2\) No privileged input injection\.𝔼x​‖fl​\(hl−1\)‖≤μ\\mathbb\{E\}\_\{x\}\\\|f\_\{l\}\(h\_\{l\-1\}\)\\\|\\leq\\muuniformly acrossl∈\{0,…,L\}l\\in\\\{0,\\ldots,L\\\}includingl=0l=0\. Input information enters via per\-token selective gates at every layer; no single layer is the unique initialization of the residual stream\.

###### Theorem A\.6\(Failure of mass displacement on selective SSMs\)\.

Under Assumption[A\.2](https://arxiv.org/html/2609.16537#A1.Thmassumption2), withE=\{0,…,⌊L/2⌋\}E=\\\{0,\\ldots,\\lfloor L/2\\rfloor\\\},

\|I~a​\(E\)−12\|≤KL,\|I~Δ​\(E¯\)−12\|≤KL,\\bigl\|\\,\\tilde\{I\}\_\{a\}\(E\)\-\\tfrac\{1\}\{2\}\\,\\bigr\|\\;\\leq\\;\\frac\{K\}\{L\},\\qquad\\bigl\|\\,\\tilde\{I\}\_\{\\Delta\}\(\\bar\{E\}\)\-\\tfrac\{1\}\{2\}\\,\\bigr\|\\;\\leq\\;\\frac\{K\}\{L\},\(15\)whereK:=∑k=0Lκ⁡\(k\)K:=\\sum\_\{k=0\}^\{L\}\\kappa\(k\)is the integrated impulse\-response constant\. Consequently the mass\-disagreement coefficientδ:=I~a​\(E\)\+I~Δ​\(E¯\)−1\\delta:=\\tilde\{I\}\_\{a\}\(E\)\+\\tilde\{I\}\_\{\\Delta\}\(\\bar\{E\}\)\-1satisfies

\|δ\|≤2​KL,\|\\,\\delta\\,\|\\;\\leq\\;\\frac\{2K\}\{L\},\(16\)which vanishes asL/K→∞L/K\\to\\infty\.

###### Proof\.

*Bound onI~a​\(E\)\\tilde\{I\}\_\{a\}\(E\)\.*By Theorem[A\.1](https://arxiv.org/html/2609.16537#A1.Thmtheorem1)and \(B1′\),Ia​\(l\)≤𝔼x​‖νl​\(x\)‖⋅𝔼x​‖ΦL,l​\(x\)‖≤μ⋅κ⁡\(L−l\)I\_\{a\}\(l\)\\leq\\mathbb\{E\}\_\{x\}\\\|\\nu\_\{l\}\(x\)\\\|\\cdot\\mathbb\{E\}\_\{x\}\\\|\\Phi\_\{L,l\}\(x\)\\\|\\leq\\mu\\cdot\\kappa\(L\-l\)to leading order\. The total ablation mass is bounded by∑lμ​κ​\(L−l\)=μ​K\\sum\_\{l\}\\mu\\kappa\(L\-l\)=\\mu K\. The early\-half mass is∑l∈Eμ​κ​\(L−l\)\\sum\_\{l\\in E\}\\mu\\kappa\(L\-l\)and the late\-half mass is∑l∈E¯μ​κ​\(L−l\)\\sum\_\{l\\in\\bar\{E\}\}\\mu\\kappa\(L\-l\); the difference between the two partial sums is bounded byμ⋅maxk⁡\|κ⁡\(k\)−κ⁡\(L−k\)\|≤μ⋅κ⁡\(0\)=μ\\mu\\cdot\\max\_\{k\}\|\\kappa\(k\)\-\\kappa\(L\-k\)\|\\leq\\mu\\cdot\\kappa\(0\)=\\mu\. Normalizing byμ​K\\mu Kand noting\|I~a​\(E\)−1/2\|≤μ/\(μ​K\)⋅1/2⋅μ=K/L\\bigl\|\\tilde\{I\}\_\{a\}\(E\)\-1/2\\bigr\|\\leq\\mu/\(\\mu K\)\\cdot 1/2\\cdot\\mu=K/Lunder the additional regularity thatκ\\kappais bounded \(with explicit constants suppressed\), gives the first inequality in \([15](https://arxiv.org/html/2609.16537#A1.E15)\)\.

*Bound onI~Δ​\(E¯\)\\tilde\{I\}\_\{\\Delta\}\(\\bar\{E\}\)\.*Without \(A3\), the LoRA\-delta mass is not forced to be terminal\. Under \(B1′\), the backward propagation of the unembedding gradient inherits the same symmetric expected decay, so‖P𝒮l​gl‖F\\\|P\_\{\\mathcal\{S\}\_\{l\}\}g\_\{l\}\\\|\_\{F\}admits a bound of the formC​κ​\(L−l\)C\\kappa\(L\-l\)\. The same telescoping argument as forIaI\_\{a\}yields\|I~Δ​\(E¯\)−1/2\|≤K/L\|\\tilde\{I\}\_\{\\Delta\}\(\\bar\{E\}\)\-1/2\|\\leq K/L\.

Combining the two displays gives \([16](https://arxiv.org/html/2609.16537#A1.E16)\)\. ∎

###### Corollary A\.7\(Spearman bound is uninformative on selective SSMs\)\.

Under Assumption[A\.2](https://arxiv.org/html/2609.16537#A1.Thmassumption2),δ→0\\delta\\to 0asL/K→∞L/K\\to\\infty\. Lemma[A\.4](https://arxiv.org/html/2609.16537#A1.Thmtheorem4)therefore does not forceρs​\(Ia,IΔ\)≤0\\rho\_\{s\}\(I\_\{a\},I\_\{\\Delta\}\)\\leq 0: the architectural mechanism producing rank anti\-correlation in residual transformers is absent\. Whether the empiricalρs\\rho\_\{s\}is positive, zero, or weakly negative depends on*additional*structure \(such as shared mid\-network unimodality ofIaI\_\{a\}andIΔI\_\{\\Delta\}, observed in Figure[3](https://arxiv.org/html/2609.16537#S4.F3)\(c, d\) for Mamba\) that is not provided by \(B1′\)–\(B2\) alone\.

### A\.5Corollary: Scale\-Induced Collapse, Not Reversal

The two largest transformers in our experiments deviate from the Theorem[A\.3](https://arxiv.org/html/2609.16537#A1.Thmtheorem3)regime: their placement ranking alignment collapses toward zero rather than remaining strongly negative \(Section[5\.4](https://arxiv.org/html/2609.16537#S5.SS4)\)\. The corollary below shows that this behavior is consistent with the formal theory: it does not require Mamba\-style mixing\.

###### Corollary A\.8\(Scale\-induced collapse\)\.

Suppose Assumption[A\.1](https://arxiv.org/html/2609.16537#A1.Thmassumption1)\(A1\)\-\(A2\) hold but \(A3\) is replaced by a*bimodal*mass split forIΔI\_\{\\Delta\}:I~Δ​\(E¯\)=τlate\\tilde\{I\}\_\{\\Delta\}\(\\bar\{E\}\)=\\tau\_\{\\mathrm\{late\}\}andI~Δ​\(E\)=τearly\\tilde\{I\}\_\{\\Delta\}\(E\)=\\tau\_\{\\mathrm\{early\}\}withτlate\+τearly≤1\\tau\_\{\\mathrm\{late\}\}\+\\tau\_\{\\mathrm\{early\}\}\\leq 1\. The ablation mass\-displacement bound \([12](https://arxiv.org/html/2609.16537#A1.E12)\) is unchanged\. The mass\-disagreement coefficient becomes

δ=I~a​\(E\)\+τlate−1≥κκ\+L​\(1\+β\)L\+τlate−1\.\\delta\\;=\\;\\tilde\{I\}\_\{a\}\(E\)\+\\tau\_\{\\mathrm\{late\}\}\-1\\\\ \\;\\geq\\;\\frac\{\\kappa\}\{\\kappa\+L\\,\(1\+\\beta\)^\{L\}\}\+\\tau\_\{\\mathrm\{late\}\}\-1\.\(17\)In particular,δ\\deltavanishes \(and the Lemma[A\.4](https://arxiv.org/html/2609.16537#A1.Thmtheorem4)\-induced bound onρs\\rho\_\{s\}becomes vacuous\) at the threshold

τlate⋆=1−κκ\+L​\(1\+β\)L\.\\tau\_\{\\mathrm\{late\}\}^\{\\,\\star\}\\;=\\;1\-\\frac\{\\kappa\}\{\\kappa\+L\\,\(1\+\\beta\)^\{L\}\}\.\(18\)Below this threshold, the architectural mechanism no longer forcesρs<0\\rho\_\{s\}<0; the empirical\|ρs\|\|\\rho\_\{s\}\|is then expected to scale linearly withτlate−τlate⋆\\tau\_\{\\mathrm\{late\}\}\-\\tau\_\{\\mathrm\{late\}\}^\{\\,\\star\}\.

###### Proof\.

The bound onI~a​\(E\)\\tilde\{I\}\_\{a\}\(E\)is unchanged because \(A1\)\-\(A2\) are unchanged\. By assumptionI~Δ​\(E¯\)=τlate\\tilde\{I\}\_\{\\Delta\}\(\\bar\{E\}\)=\\tau\_\{\\mathrm\{late\}\}\. Substituting into the definitionδ=I~a​\(E\)\+I~Δ​\(E¯\)−1\\delta=\\tilde\{I\}\_\{a\}\(E\)\+\\tilde\{I\}\_\{\\Delta\}\(\\bar\{E\}\)\-1gives \([17](https://arxiv.org/html/2609.16537#A1.E17)\)\. The threshold \([18](https://arxiv.org/html/2609.16537#A1.E18)\) is the value ofτlate\\tau\_\{\\mathrm\{late\}\}at which the right\-hand side of \([17](https://arxiv.org/html/2609.16537#A1.E17)\) equals zero\. ∎

### A\.6Falsifiable predictions

The formal theory generates five concrete, falsifiable predictions\. \(P1\)\-\(P3\) are partially confirmed in the body of the paper; \(P4\)\-\(P5\) are open\.

\(P1\) Leave\-out\-L0 in transformers\.Removing layer00from bothIaI\_\{a\}andIΔI\_\{\\Delta\}in a residual transformer should reduce\|ρs\|\|\\rho\_\{s\}\|\(by reducingI~a​\(E\)\\tilde\{I\}\_\{a\}\(E\)in the mass\-displacement bound \([12](https://arxiv.org/html/2609.16537#A1.E12)\)\) but should not flip its sign, becauseI~a​\(E\)\\tilde\{I\}\_\{a\}\(E\)remains\>1/2\>1/2even after removing the layer\-00contribution as long as input\-side mass is concentrated more broadly near the bottleneck\. Confirmed in Section[5\.3](https://arxiv.org/html/2609.16537#S5.SS3)and Appendix[E](https://arxiv.org/html/2609.16537#A5)\.

\(P2\) Architecture transfer\.Any architecture satisfying Assumption[A\.2](https://arxiv.org/html/2609.16537#A1.Thmassumption2)\(B1′\)\-\(B2\) — full SSMs, RWKV, linear attention with symmetric kernels in expectation — should exhibitδ→0\\delta\\to 0asL/K→∞L/K\\to\\inftyand therefore lack the architectural mechanism forcingρs<0\\rho\_\{s\}<0\. Confirmed for Mamba on every paired checkpoint\.

\(P3\) LoRA\-target alignment\.Choosing LoRA target modules whose subspace𝒮l\\mathcal\{S\}\_\{l\}is aligned with the dominant Hessian eigenbasis ofθl\\theta\_\{l\}should reduce the curvature factor in Equation \([4](https://arxiv.org/html/2609.16537#A1.E4)\), decreasingτ\\tauin Assumption[A\.1](https://arxiv.org/html/2609.16537#A1.Thmassumption1)\(A3\) and weakening the late\-half mass concentration ofIΔI\_\{\\Delta\}\. The submodule\-matched control \(Appendix[E\.2](https://arxiv.org/html/2609.16537#A5.SS2)\) is a partial realization; a full test requires varying the LoRA\-target choice while holding everything else fixed\.

\(P4\) Threshold predicts magnitude\.Under Corollary[A\.8](https://arxiv.org/html/2609.16537#A1.Thmtheorem8),\|ρs\|\|\\rho\_\{s\}\|should vanish atτlate⋆\\tau\_\{\\mathrm\{late\}\}^\{\\,\\star\}\(Eq\. \([18](https://arxiv.org/html/2609.16537#A1.E18)\)\) and grow linearly with the gapτlate−τlate⋆\\tau\_\{\\mathrm\{late\}\}\-\\tau\_\{\\mathrm\{late\}\}^\{\\,\\star\}above it\. This is testable by computing the late\-half LoRA\-delta mass on each transformer checkpoint and regressing\|ρs\|\|\\rho\_\{s\}\|against the predicted threshold\.

\(P5\) Per\-layer\-input transformer\.If a residual transformer is modified to inject the embedding at every layer \(so \(A2\) fails becausef0f\_\{0\}is no longer the unique initialization of the residual stream\), the mass\-displacement bound onI~a​\(E\)\\tilde\{I\}\_\{a\}\(E\)degrades to the Theorem[A\.6](https://arxiv.org/html/2609.16537#A1.Thmtheorem6)regime,δ→0\\delta\\to 0, andρs\\rho\_\{s\}should no longer be forced negative\. This is the strongest test of the formal theory: a parameterization change predicted to remove the anti\-correlation mechanism without changing the architecture family\.

P5 is a falsification target\. If a per\-layer\-input transformer still showsρs<0\\rho\_\{s\}<0, then Assumption \(A2\) is not the load\-bearing structural property the theory claims it is, and the mechanism would have to be located elsewhere \(e\.g\. in the attention/MLP factorization rather than the embedding bottleneck\)\.

### A\.7Connection to prior theoretical work

The general claim that gradient, ablation, and update\-magnitude attributions probe distinct functionals is established in the attribution literature:[Sundararajan et al\. \(2017\)](https://arxiv.org/html/2609.16537#bib.bib24)prove that no method satisfies the*Sensitivity*,*Implementation Invariance*, and*Completeness*axioms simultaneously;[Ancona et al\. \(2018\)](https://arxiv.org/html/2609.16537#bib.bib25)unify gradient\-based methods as varying\-fidelity linearizations of true ablation;[Adebayo et al\. \(2020\)](https://arxiv.org/html/2609.16537#bib.bib26)show empirically that gradient\-saliency maps survive parameter randomizations that destroy the network’s behavior\. Our contribution is to make the corresponding statement precise at the*layer level*, derive the specific transformer pattern from residual\-stream gradient flow under explicit assumptions, and quantify the practical consequence as a mass\-displacement bound \(rigorous from Assumption[A\.1](https://arxiv.org/html/2609.16537#A1.Thmassumption1)alone\) coupled with a separate rank\-correlation lemma \(Lemma[A\.4](https://arxiv.org/html/2609.16537#A1.Thmtheorem4), conditional on profile\-shape monotonicity\)\.

The closest prior decomposition is the disagreement\-attribution framework of[Krishna et al\. \(2024\)](https://arxiv.org/html/2609.16537#bib.bib27): observe disagreement, neither derives a quantitative bound from architectural structure\. Theorem[A\.3](https://arxiv.org/html/2609.16537#A1.Thmtheorem3)provides such a bound, and Theorem[A\.6](https://arxiv.org/html/2609.16537#A1.Thmtheorem6)establishes its architectural specificity\. To our knowledge, no prior work has stated the sign of cross\-method correlation as a function of input\-injection asymmetry and impulse\-response symmetry\.

## Appendix BExperimental Details

### B\.1Background

Low\-Rank Adaptation \(LoRA\)\([Hu et al\., 2021](https://arxiv.org/html/2609.16537#bib.bib9)\)fine\-tunes a pretrained model by inserting trainable low\-rank updatesΔ​W=B​A\\Delta W=BAinto selected weight matrices \(B∈ℝd×rB\\in\\mathbb\{R\}^\{d\\times r\},A∈ℝr×kA\\in\\mathbb\{R\}^\{r\\times k\},r≪min⁡\(d,k\)r\\ll\\min\(d,k\)\), keeping the base parameters frozen\. Under a fixed budget, this forces a placement decision: which layers get adapters\.*Importance\-guided*placement\([Zhang et al\., 2023](https://arxiv.org/html/2609.16537#bib.bib10);[Qing et al\., 2024](https://arxiv.org/html/2609.16537#bib.bib11);[Xu et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib15);[Zhao et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib12);[Hayou et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib14)\)replaces uniform allocation by estimating which layers matter most for a target task\. The premise is that layer importance is a recoverable property of depth\.

We write the importance of layerllfor taskttunder methodmmasIm​\(l,t\)∈ℝ≥0I\_\{m\}\(l,t\)\\in\\mathbb\{R\}\_\{\\geq 0\}\. The key observation is that “matters” is not uniquely defined\. The five estimators we study fall into three categories:*ablation*measures functional necessity \(which computations the model already depends on, scored by the loss increase from disabling a layer\-local computation\);*LoRA delta attribution*measures adaptation pressure \(where the optimizer writes task\-specific updates during fine\-tuning, scored by the merged update magnitude\); and three sensitivity proxies \(gradient attribution, resnorm\([Xu et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib15)\), activation norm\) are training\-free signals serving as diagnostic baselines\. Necessity and adaptation pressure need not coincide: a layer can be functionally critical yet undergo little adaptation, or absorb a large task\-specific update without being uniquely indispensable\. Whether ablation and LoRA delta produce compatible rankings, and whether that compatibility is architecture\-dependent, is the empirical question of this paper\.

### B\.2Models

Table 1:Models used in this study\.LLdenotes the number of layers \(transformer blocks or SSM blocks\)\. The Qwen3, Llama, and Mamba checkpoints form the primary paired set; RWKV6\-3B, Falcon\-Mamba\-7B, and Zamba2\-2\.7B are validation architectures\. Zamba2\-2\.7B’s5454layers comprise4545Mamba blocks and99shared\-attention hybrid blocks at positions6,12,…,516,12,\\ldots,51\.Table 2:Task\-native MCQ accuracy audit for causal\-validation rows\. Each entry averages top\-kkand bottom\-kkaccuracy over the available MCQ tasks: MMLU, ARC\-Challenge, GPQA, PIQA, and WinoGrande\. When multiple runs are available for the same model and method, we average them\.Δ\\Deltais top\-kkminus bottom\-kk\. Generation\-style tasks are excluded because their main diagnostic validation uses perplexity rather than task\-native pass@1/F1/exact\-match metrics\.The near\-flat profiles arise because gradient norms in a pretrained model reflect the training\-data activation mixture rather than task\-specific sensitivity\.

### B\.3Discriminability across models and methods

Table 3:Minimum pairwise cosine similarity \(dmind\_\{\\min\}\) across models and methods\. Lower values indicate stronger task discrimination\. Bold indicates the most discriminative method per model\. “—” indicates the method was not run for that model\.
### B\.4Necessity–Plasticity projection of additional estimators

We test whether the Necessity–Plasticity decomposition extends beyond the two task\-sensitive estimators by projecting each available layer\-importance estimator onto a two\-axis plane\. The Plasticity loadingρP\\rho\_\{P\}is the mean Spearman correlation between the estimator’s per\-layer scores and LoRA\-delta scores\. The Necessity loadingρN\\rho\_\{N\}is the mean Spearman correlation between the estimator’s per\-layer scores and ablation scores\.

On transformers, four of five additional estimators load clearly on one side: Gradient \(ρP=−0\.66\\rho\_\{P\}=\-0\.66,ρN=\+0\.47\\rho\_\{N\}=\+0\.47\) and Resnorm \(ρP=−0\.89\\rho\_\{P\}=\-0\.89,ρN=\+0\.46\\rho\_\{N\}=\+0\.46\) project onto Necessity; Activation norm \(ρP=\+0\.84\\rho\_\{P\}=\+0\.84,ρN=−0\.35\\rho\_\{N\}=\-0\.35\) and TELL\-TALE \(ρP=\+0\.65\\rho\_\{P\}=\+0\.65,ρN=−0\.32\\rho\_\{N\}=\-0\.32\) project onto Plasticity; ShapLoRA lies near the boundary \(ρP=\+0\.22\\rho\_\{P\}=\+0\.22,ρN=−0\.08\\rho\_\{N\}=\-0\.08\)\. The classification has a mechanistic reading: estimators evaluated at the pretrained checkpoint load more on Necessity, while estimators that incorporate fine\-tuning trajectories or post\-adaptation activations load more on Plasticity\. On Mamba, where Necessity and Plasticity are already positively aligned at the family level, all tested estimators land in the mixed\-positive region\. No estimator falls in the lower\-left quadrant, so ablation and LoRA delta span the relevant measurement regimes\.

![Refer to caption](https://arxiv.org/html/2609.16537v1/fig7_resnorm_heatmap.png)Figure 8:Resnorm profiles for Qwen3\-8B and Llama\-3\.1\-8B\. The projected\-residual diagnostic concentrates mass in the earliest layers across tasks, missing the terminal cluster identified by LoRA delta\.

## Appendix CMulti\-Seed and Budget Robustness

### C\.1Multi\-seed causal validation

We repeat all causal\-validation runs for Qwen3\-8B and Llama\-3\.1\-8B under seeds 42, 43, and 44 for six methods: LoRA delta, ablation, gradient, resnorm, activation norm, and random placement\. This section reports aggregate and per\-task robustness results\.

##### Metrics\.

For generation tasks \(HumanEval, MBPP, GSM8K, SQuAD, LongBench\), we report the PPL gap, defined as bottomkPPL minus topkPPL; positive values support the ranking\. For MCQ tasks \(GPQA, MMLU, ARC, PIQA, WinoGrande\), we report the accuracy gap, defined as topkaccuracy minus bottomkaccuracy; positive values indicate that top\-kklayers yield higher accuracy\. The binaryhypothesis\_supportedflag is derived from the appropriate metric for each task type\.

##### Summary\.

The method\-flipping pattern is stable across all three seeds at the level of the directional support indicator: LongBench is supported by LoRA delta in 3/3 seeds; SQuAD is supported by ablation in 3/3 seeds; PIQA fails under LoRA delta in 3/3 seeds\. Because support is binary and some generation tasks are validated with perplexity rather than task\-native scores, these seeded summaries should be read as robustness of the proxy signal, not as proof of uniformly strong practical gains\. Random selection stabilizes near 57%/63%, and resnorm falls at or below chance on both models across all seeds\.

Figure 9:Mean directional causal\-validation support rate±\\pmstd across three seeds for Qwen3\-8B and Llama\-3\.1\-8B\. The dashed line marks 50%\. Error bars show std across seeds\. LoRA delta is the most consistently interpretable high\-performing method across the two checkpoints, while gradient can achieve competitive aggregate support despite its near\-flat rankings\. Resnorm falls at or below random on both models\.![Refer to caption](https://arxiv.org/html/2609.16537v1/fig_multiseed_per_task_heatmap.png)Figure 10:Fraction of seeds where the causal hypothesis is directionally supported, per task and method\. Green indicates supported in all three seeds; red indicates unsupported in all three seeds; yellow indicates mixed\. Code tasks are robustly supported by LoRA delta under the common validation loop, while WinoGrande and ARC are robustly unsupported under LoRA delta\.Figure 11:Metric gaps across seeds\. Left column: PPL gap \(bottomkminus topk\) for generation tasks\. Right column: accuracy gap \(topkminus bottomk\) for MCQ tasks\. Positive values support the hypothesis\. Error bars show std across three seeds\. These magnitudes should be inspected directly rather than inferred from the binary support label alone\.Figure 12:Cross\-seed stability of support decisions\. Green indicates supported in all three seeds; orange indicates mixed support; red indicates unsupported in all three seeds\. LoRA delta shows the highest fraction of fully stable supported tasks for code and long\-context\. Commonsense tasks are stably unsupported under LoRA delta across seeds\.

### C\.2Seed coverage

Table 4:Per\-checkpoint multi\-seed coverage of the causal\-validation runs\. Brace\-enclosed lists are the seed indices with completed result files\. “LoRA\-δ\\deltasupport per seed” shows the number of directionally supported tasks at each completed seed\.For multi\-seed runs, entries report the number of supported tasks for seeds 42/43/44

### C\.3Causal\-validation margin magnitudes

The main text uses top\-kkversus bottom\-kksupport labels to test whether an importance ranking has directional causal consequences under selective LoRA placement\. These labels record the sign of the comparison, not the magnitude of the downstream change\. We therefore report the corresponding metric gaps here\. For generation\-style tasks evaluated with perplexity, the gap isPPLbottom−PPLtop\\mathrm\{PPL\}\_\{\\mathrm\{bottom\}\}\-\\mathrm\{PPL\}\_\{\\mathrm\{top\}\}, so positive values favor the top\-kklayers\. For multiple\-choice tasks evaluated with accuracy, the gap isAcctop−Accbottom\\mathrm\{Acc\}\_\{\\mathrm\{top\}\}\-\\mathrm\{Acc\}\_\{\\mathrm\{bottom\}\}\.

Figure[11](https://arxiv.org/html/2609.16537#A3.F11)reports per\-task gaps with seed error bars for Qwen3\-8B and Llama\-3\.1\-8B\. Tables[7](https://arxiv.org/html/2609.16537#A4.T7),[8](https://arxiv.org/html/2609.16537#A4.T8), and[5](https://arxiv.org/html/2609.16537#A3.T5)report primary\-metric gaps for competitor placement methods, the Qwen3\-14B and Mamba\-790m extension runs, and thekk\-sweep respectively\.

The margin results show three regimes\. First, a small number of task–model–method triples have substantively large gaps, including Qwen3\-1\.7B on SQuAD under ablation, Qwen3\-1\.7B on WinoGrande where bottom\-kkablation layers win by a large margin, and Mamba\-790m on SQuAD under LoRA delta\. Second, most supported pairs are small in absolute magnitude, with typical primary\-metric gaps in the0\.010\.01–0\.050\.05range\. Third, random placement, competitor methods, and several larger\-scale extension runs remain close to zero on average\. Thus, the causal\-validation results should not be read as evidence that any single estimator provides a large universal downstream gain\. They instead show that the direction of selective\-placement validation is method\-relative, even when the absolute margins are modest\.

Figure 13:Performance change under uniform LoRA relative to the frozen model\. Uniform LoRA substantially improves generation perplexity, but does not uniformly improve multiple\-choice accuracy\.Figure 14:Recovery ratioℛ\\mathcal\{R\}for top\-kkplacement under LoRA delta and resnorm\. LoRA delta recovers most of the full\-LoRA improvement on code generation, while resnorm is inconsistent across tasks\.![Refer to caption](https://arxiv.org/html/2609.16537v1/fig6_per_task_support_heatmap.png)Figure 15:Per\-task causal\-validation support across methods, shown as the fraction of seeds in which each method supports the hypothesis\. Code tasks are the most consistently recoverable; commonsense tasks are the least\.
### C\.4Sensitivity of causal validation to the budgetkk

The main protocol fixesk=⌈L/6⌉k=\\lceil L/6\\rceil\. To check how much of the mixed\-to\-negative LoRA\-delta verdict depends on that particular budget, we run an additionalkk\-sweep for six transformer checkpoints across both families\. For each model, we evaluate four values ofkkspanning roughlyk/L∈\[0\.08,0\.31\]k/L\\in\[0\.08,0\.31\]and report directional support together with the mean topk\-minus\-bottomkprimary\-metric gap, using the same task\-native metrics and the same LoRA rank and training budget as the main experiments\.

Table 5:kk\-sweep causal validation on transformer checkpoints with LoRA\-delta top\-kk\. Each row is one\(m​o​d​e​l,k\)\(model,k\)configuration; all runs use the ten\-task suite\. Support is the fraction of tasks on which top\-kkoutperforms bottom\-kkon the task\-native primary metric\. Mean primary gap is averaged across tasks; positive means top\-kkis better\.ModelFamilykkk/Lk/LSupportMean primary gapLlama\-3\.2\-1BLlama20\.1255/105/10\+0\.002\+0\.002Llama\-3\.2\-1BLlama30\.1883/103/10\+0\.005\+0\.005Llama\-3\.2\-1BLlama40\.2503/103/10−0\.006\-0\.006Llama\-3\.2\-1BLlama50\.3125/105/10\+0\.002\+0\.002Llama\-3\.2\-3BLlama30\.1072/102/10−0\.031\-0\.031Llama\-3\.2\-3BLlama40\.1431/101/10−0\.033\-0\.033Llama\-3\.2\-3BLlama50\.1791/101/10−0\.040\-0\.040Llama\-3\.2\-3BLlama70\.2502/102/10−0\.026\-0\.026Llama\-3\.1\-8BLlama30\.0942/102/10−0\.040\-0\.040Llama\-3\.1\-8BLlama50\.1562/102/10−0\.049\-0\.049Llama\-3\.1\-8BLlama60\.1881/101/10−0\.055\-0\.055Llama\-3\.1\-8BLlama80\.2501/101/10−0\.027\-0\.027Qwen3\-1\.7BQwen30\.1074/104/10\+0\.010\+0\.010Qwen3\-1\.7BQwen40\.1435/105/10\+0\.012\+0\.012Qwen3\-1\.7BQwen50\.1793/103/10\+0\.002\+0\.002Qwen3\-1\.7BQwen70\.2504/104/10\+0\.008\+0\.008Qwen3\-4BQwen30\.0832/102/10\+0\.021\+0\.021Qwen3\-4BQwen50\.1393/103/10\+0\.025\+0\.025Qwen3\-4BQwen60\.1673/103/10\+0\.021\+0\.021Qwen3\-4BQwen80\.2223/103/10\+0\.015\+0\.015Qwen3\-8BQwen30\.0831/101/10−0\.019\-0\.019Qwen3\-8BQwen50\.1391/101/10−0\.011\-0\.011Qwen3\-8BQwen60\.1674/104/10−0\.005\-0\.005Qwen3\-8BQwen80\.2222/102/10−0\.015\-0\.015Llama family—all—s¯=0\.23\\bar\{s\}=0\.23g¯=−0\.025\\bar\{g\}=\-0\.025Qwen family—all—s¯=0\.29\\bar\{s\}=0\.29g¯=\+0\.005\\bar\{g\}=\+0\.005

The qualitative verdict is insensitive to the budget choice: thek=⌈L/6⌉k=\\lceil L/6\\rceilrow sits inside the envelope swept byk∈\[2,8\]k\\in\[2,8\]at each checkpoint, and neither family benefits consistently from a differentkk\. Across all twelve Llama\(m​o​d​e​l,k\)\(model,k\)points, mean support is0\.230\.23and the mean primary gap is−0\.025\-0\.025; across the twelve Qwen points, mean support is0\.290\.29and the mean primary gap is\+0\.005\+0\.005\. The method\-relative disagreement reported in the main text is therefore not an artifact of the fixed budget\. The caveat is that this sweep was not run on Mamba checkpoints or on the largest transformers, and it sweeps only LoRA delta rather than all estimators\.

### C\.5Task\-A adaptation quality: top\-kkversus bottom\-kk

A potential confound in the continual\-learning protocol is that bottom\-kklayers \(least plastic\) may simply learn Task A less thoroughly than top\-kklayers, mechanically producing a lower forgetting ratio without any genuine bottleneck effect\. Table[6](https://arxiv.org/html/2609.16537#A3.T6)addresses this directly by reporting the post\-Task\-A perplexity \(PPLbefore\\text\{PPL\}\_\{\\text\{before\}\}, measured after Task\-A fine\-tuning and before Task\-B training\) for both placement strategies across all models, together with the resulting forgetting ratios\.

Two patterns emerge\. First, intransformers, top\-kkdoes achieve modestly better Task\-A adaptation: averaged across all transformer checkpoints and pairs \(n=75n\{=\}75\), PPLbefore\{\}\_\{\\text\{before\}\}is1\.711\.71for top\-kkversus2\.002\.00for bottom\-kk— a17%17\\%gap, meaning the least plastic layers absorb Task\-A somewhat less\. However, the forgetting ratio gap is substantially larger:3\.243\.24for top\-kkversus2\.332\.33for bottom\-kk— a39%39\\%difference in the direction opposite to what a pure “learned less, forgets less” account would predict\. A confounder that accounts for a17%17\\%advantage in starting perplexity cannot explain a39%39\\%disadvantage in forgetting ratio; the bottleneck effect is not an artifact of differential Task\-A learning\.

Second, inSSMs, top\-kkand bottom\-kkPPLbefore\{\}\_\{\\text\{before\}\}are virtually identical \(6\.536\.53vs\.6\.736\.73, a3%3\\%gap,n=54n\{=\}54\), and the forgetting ratios are likewise statistically indistinguishable \(2\.112\.11vs\.2\.002\.00\)\. This confirms that the null forgetting result in SSMs is not driven by top\-kkand bottom\-kklearning at different rates\.

Table 6:Task\-A post\-training perplexity \(PPLbefore\{\}\_\{\\text\{before\}\}, lower is better\) and forgetting ratio \(PPLafter\{\}\_\{\\text\{after\}\}/PPLbefore\{\}\_\{\\text\{before\}\}, lower is better\) for top\-kkvs\. bottom\-kkplacement across all CL models\. Values are mean±\\pmstd across all task pairs and seeds\. The “ratio” column shows \(bottom\-kkPPLbefore\{\}\_\{\\text\{before\}\}\) / \(top\-kkPPLbefore\{\}\_\{\\text\{before\}\}\); values above1\.01\.0indicate that top\-kkachieves better Task\-A adaptation\. Family summary rows pool all pairs within the family\. RWKV6\-3B omitted \(numerical divergence in top\-kkruns\)\.

## Appendix DEstimator and Architecture Extensions

### D\.1Head\-to\-head comparison with published placement methods

We re\-implement two competitor placement methods—ShapLoRA\([Zhao et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib12)\)and TELL\-TALE\([Naim et al\., 2026](https://arxiv.org/html/2609.16537#bib.bib16)\)—and run them through the same top\-kkversus bottom\-kkcausal\-validation protocol on Qwen3\-8B, Llama\-3\.1\-8B, and Mamba\-790m at seed 42\. This comparison is not intended as a refutation of either method in its original setting; the goal is to test whether the same matched diagnostic protocol produces a single estimator that dominates across architectures\.

Table[7](https://arxiv.org/html/2609.16537#A4.T7)reports directional support rates restricted to the task list common to every method on each model: a seven\-task subset for the two transformer checkpoints, and a nine\-task subset for Mamba\-790m\. On both transformer checkpoints, neither competitor beats random under the matched protocol\. The same pattern holds on Mamba\-790m, where both competitors land below LoRA delta\. Mean primary\-metric gaps are near zero or negative for both competitors on every tested model, indicating that the under\-performance is directional rather than a thresholding artifact of the binary support metric\.

Table 7:Causal\-validation comparison against published placement methods at seed 42\. Each block restricts to the task list common to every method listed for that model\. Support is the fraction of tasks on which top\-kkoutperforms bottom\-kkunder the loss\-aligned validation metric\. Mean PPL gap is bottomkminus topkon perplexity\-validated tasks; mean primary gap is topkminus bottomkon task\-native primary metrics\. Empty primary\-gap entries indicate runs whose logs predate the task\-native evaluation path\. LoRA\-delta primary\-gap entries are taken from the task\-native audit and may use a different task subset\.
### D\.2Scale and architecture extension: Qwen3\-14B and Mamba\-790m

The Qwen3\-8B and Llama\-3\.1\-8B seeded study leaves two natural questions open: whether the scale\-dependent weakening of LoRA delta persists at larger transformer sizes, and whether the transformer–SSM contrast survives a seeded causal\-validation setup\. To address both, we rerun the causal\-validation pipeline on Qwen3\-14B and Mamba\-790m at seeds 43 and 44, with the samek=6k=6, LoRA rank 8, and 300 training steps as the 8B study\. Each model is run under three placement regimes: LoRA\-delta top\-kk, random top\-kk, and uniform LoRA over all layers\. For Mamba\-790m, we report aggregates on the nine non\-LongBench tasks because the A1 task list and the Mamba\-790m importance scores use different LongBench variants\.

Table[8](https://arxiv.org/html/2609.16537#A4.T8)summarizes directional support rates and mean primary\-metric gaps\. On Mamba\-790m, LoRA delta yields a positive mean topk\-minus\-bottomkprimary\-metric gap at both seeds, while random top\-kkis near zero or negative\. On Qwen3\-14B, the picture is inverted: random top\-kkmatches or exceeds LoRA delta both on support rate and on mean top\-minus\-bottom gap\. This is consistent with the Qwen3\-8B to Qwen3\-32B bimodality reported in Figure[21](https://arxiv.org/html/2609.16537#A8.F21): in the transformer family, LoRA delta’s prescriptive content weakens as scale grows, while in the SSM family it remains informative under the same validation recipe\.

Table 8:Scale/architecture extension: per\-seed directional support rate and mean primary\-metric gap for LoRA delta and random top\-kk, plus per\-seed uniform\-LoRA win rate against the no\-adapter baseline\. Both checkpoints run withk=6k=6, LoRA rank 8, 300 training steps, at seeds 43 and 44\.†Mamba\-790m LongBench was not evaluated in the A1 extension: the updated A1 task list useslongbench\_multifieldqa\_enwhereas the Mamba\-790m LoRA\-delta importance scores coverlongbench\_single\_doc; regenerating Mamba importance against the new variant was out of scope\. Rows therefore report the nine tasks common to both checkpoints\.

Figure 16:Per\-task topkminus bottomkprimary\-metric gap on Qwen3\-14B and Mamba\-790m for LoRA delta and random top\-kkplacement\. Bars are seed\-averaged means; error bars show the seed range over seeds 43 and 44\. LongBench is excluded to keep the task set comparable across checkpoints\.
### D\.3Mamba scale sweep

Tables[9](https://arxiv.org/html/2609.16537#A4.T9)and[10](https://arxiv.org/html/2609.16537#A4.T10)report causal validation for all four Mamba checkpoints\. Under ablation, top\-kklayers follow the boundary pattern at all scales\. Under LoRA delta, top\-kklayers shift to mid\-network positions and support rates remain high, including tasks such as WinoGrande and PIQA that fail systematically in transformers under LoRA delta\.

Table 9:Mamba causal validation under ablation across four model sizes\. Top\-kklayers follow the boundary pattern at all scales\.Table 10:Mamba causal validation under LoRA delta across four model sizes\. Mid\-network layers are selected at all scales; support rates are 9–10/10\.

## Appendix EAlternative\-Mechanism Controls

### E\.1The sign flip is not a layer\-0 artifact

Table 11:Necessity\-Plasticity Alignment𝒩​𝒫​𝒜​\(M\)\\mathcal\{NPA\}\(M\)for the 14 primary paired pretrained checkpoints \(Qwen3, Llama, Mamba\)\.𝒩​𝒫​𝒜​\(M\)\\mathcal\{NPA\}\(M\)is the mean Spearman rank correlation between LoRA\-delta and ablation importance vectors acrossT=10T=10tasks\.𝒩​𝒫​𝒜∖L0​\(M\)\\mathcal\{NPA\}^\{\\setminus L\_\{0\}\}\(M\)recomputes the same correlation with layer 0 excluded from both vectors\. L0 share is the fraction of total ablation mass concentrated in the first layer\. Family means in bold are arithmetic averages over the listed checkpoints\. RWKV6\-3B, Zamba2\-2\.7B, and Falcon\-Mamba\-7B are reported in Section[4\.1](https://arxiv.org/html/2609.16537#S4.SS1)as validation architectures\.ModelFamilyLL𝑪⁡\(𝑴\)C\(M\)𝑪⁡\(𝑴\)∖𝑳𝟎C\(M\)\\setminus\\\!L\_\{0\}L0 shareQwen3\-0\.6BTransformer28−0\.25\-0\.25−0\.16\-0\.160\.89Qwen3\-1\.7BTransformer28−0\.47\-0\.47−0\.40\-0\.400\.94Qwen3\-4BTransformer36−0\.23\-0\.23−0\.16\-0\.160\.61Qwen3\-8BTransformer36−0\.30\-0\.30−0\.24\-0\.240\.60Qwen3\-14BTransformer40−0\.06\-0\.06−0\.10\-0\.100\.60Qwen3\-32BTransformer64\+0\.07\+0\.07\+0\.10\+0\.100\.60Llama\-3\.2\-1BTransformer16−0\.74\-0\.74−0\.69\-0\.690\.32Llama\-3\.2\-3BTransformer28−0\.48\-0\.48−0\.43\-0\.430\.49Llama\-3\.1\-8BTransformer32−0\.62\-0\.62−0\.59\-0\.590\.54Llama\-3\.1\-70BTransformer80\+0\.01\+0\.01\+0\.05\+0\.050\.61Mamba\-130mSSM24\+0\.22\+0\.22\+0\.22\+0\.220\.30Mamba\-370mSSM48\+0\.22\+0\.22\+0\.20\+0\.200\.29Mamba\-790mSSM48\+0\.25\+0\.25\+0\.26\+0\.260\.09Mamba\-2\.8bSSM64\+0\.28\+0\.28\+0\.29\+0\.290\.07Transformer mean \(n=10\)−0\.31\\mathbf\{\-0\.31\}−0\.26\\mathbf\{\-0\.26\}0\.62Transformer mean \(≤\\leq14B, n=8\)−0\.39\\mathbf\{\-0\.39\}−0\.35\\mathbf\{\-0\.35\}0\.62Mamba mean \(n=4\)\+0\.24\\mathbf\{\+0\.24\}\+0\.24\\mathbf\{\+0\.24\}0\.19

Figure 17:Per\-checkpoint distribution of the structural asymmetry summarized by leave\-out\-L0 and L0\-share controls\. Dashed lines mark family means; two\-sided Mann–Whitney Upp\-values compare family membership\.The L0 control rules out the simplest explanation for the transformer–Mamba split\. Transformers place much more ablation mass on layer 0 than Mambas, so the control is necessary\. However, removing layer 0 shifts transformer correlations toward zero without changing the qualitative sign for transformers up to 14B\. The observed anti\-correlation is therefore not reducible to a single embedding\-bottleneck layer\.

### E\.2The sign flip is not a submodule\-scope artifact

In the main experiments, LoRA delta is computed on trainable LoRA target modules, whereas ablation removes a layer\-local computation\. This raises a natural concern: perhaps the observed disagreement is caused by comparing different parts of the block rather than by comparing different notions of importance\. To address this, we recompute placement ranking alignment under matched submodule scopes\. For each matched setting, both estimators are restricted to the same part of the transformer block\.

The cross\-method anti\-correlation is negative for every available submodule in Llama\-3\.1\-8B:ρ=−0\.581\\rho=\-0\.581for attention,−0\.266\-0\.266for MLP, and−0\.298\-0\.298for full\-block comparisons\. The corresponding top\-6 layer overlaps are 15%, 17%, and 27%\. For two random 6\-of\-32 selections, the expected overlap is6×6/32=1\.1256\\times 6/32=1\.125layers, or 18\.75% of a top\-6 set\. Thus the attention and MLP overlaps are at or below random, whereas the full\-block overlap remains far below within\-LoRA overlaps\. This rules out the possibility that the full\-block anti\-correlation is an artifact of mixing attention and MLP signals\.

Within LoRA delta, submodules converge on the same layers\. The mean Spearman correlation between LD\-attn and LD\-MLP is\+0\.839\+0\.839in Llama and\+0\.659\+0\.659in Qwen, rising to\+0\.883\+0\.883for LD\-attn versus LD\-full in Llama\. Top\-6 overlap ranges from 55–60%\. This indicates that the terminal cluster is not a property of which projections are updated: adapting attention alone, MLP alone, or both simultaneously identifies the same final depth region\.

Within ablation, attention and MLP criticality are more model\-dependent\. In Llama, Abl\-attn versus Abl\-MLP yieldsρ=\+0\.427\\rho=\+0\.427and 67% top\-6 overlap, suggesting moderate co\-localization\. In Qwen3\-8B, the same comparison yieldsρ=−0\.027\\rho=\-0\.027and 27% overlap, indicating structurally distinct attention\-critical and MLP\-critical layer sets\. Thus, if using ablation to guide submodule\-specific placement, the submodule choice matters; if using LoRA delta, the choice is much less important\.

Table 12:Submodule\-matched placement ranking alignment\. The Llama\-3\.1\-8B full\-block row is the headline control: both estimators are applied at the block level, yet the correlation remains negative\. Within\-LoRA\-delta correlations between attention\- and MLP\-matched scorings are strongly positive, ruling out within\-method noise\.Table 13:Mean Spearmanρ\\rhobetween condition pairs for Llama\-3\.1\-8B and Qwen3\-8B\. Cross\-method comparisons are negative in Llama across all submodules; within\-LoRA\-delta comparisons are positive in both models; within\-ablation comparisons reveal model\-dependent divergence\.![Refer to caption](https://arxiv.org/html/2609.16537v1/fig_sub_correlation_matrix.png)Figure 18:Full Spearmanρ\\rhomatrix across matched\-control conditions for Llama\-3\.1\-8B and Qwen3\-8B\. The top\-left block contains within\-ablation correlations; the bottom\-right block contains within\-LoRA\-delta correlations\. The off\-diagonal quadrants show the cross\-method signal\.![Refer to caption](https://arxiv.org/html/2609.16537v1/fig_sub_rho_summary.png)Figure 19:Summary Spearmanρ\\rhoheatmap by comparison type and submodule\. The cross\-method row is negative for Llama across all submodules\. The within\-LoRA\-delta row is positive for both models\. The within\-ablation row reveals that Qwen3\-8B has near\-zero attention–MLP correlation, indicating structurally separate attention\-critical and MLP\-critical layer sets\.> ### E\.3The sign flip is not a LoRA artifact: adapter\-free full fine\-tuning control Plasticity is defined in the main text through the magnitude of a LoRA update, which raises the possibility that𝒩​𝒫​𝒜\\mathcal\{NPA\}measures where LoRA places updates rather than where the architecture absorbs new information\. We therefore rerun the Plasticity measurement without adapters: for each checkpoint we fine\-tune*all*parameters on the same tasks, under the same data and budget, and recompute the per\-layer update magnitude‖Δ​Wl‖F\\\|\\Delta W\_\{l\}\\\|\_\{F\}from the difference between the fine\-tuned and pretrained weights\. This removes every LoRA\-specific degree of freedom at once: rank, scaling factorα\\alpha, initialization, and the choice of target modules\. Table[14](https://arxiv.org/html/2609.16537#A5.T14)reports the result on the1313checkpoints we were able to fully retrain\. Two quantities matter\. First,ρ⁡\(Δ​Wfull,Δ​WLoRA\)\\rho\(\\Delta W^\{\\mathrm\{full\}\},\\Delta W^\{\\mathrm\{LoRA\}\}\)is positive on*every*checkpoint, so unconstrained fine\-tuning and LoRA select the same layers; LoRA is not redirecting updates to an unrepresentative subset of depth\. Second,ρ⁡\(Δ​Wfull,ablation\)\\rho\(\\Delta W^\{\\mathrm\{full\}\},\\text\{ablation\}\)— the adapter\-free analogue of𝒩​𝒫​𝒜\\mathcal\{NPA\}— is negative for every evaluated transformer with an ablation baseline, and positive or near zero for every evaluated Mamba, RWKV, and hybrid checkpoint\. The sign therefore matches the LoRA\-based𝒩​𝒫​𝒜\\mathcal\{NPA\}of Table[11](https://arxiv.org/html/2609.16537#A5.T11)in all1313cases\. Correlation between the two adaptation procedures declines with model size \(from0\.960\.96on Qwen3\-0\.6B to0\.390\.39on Qwen3\-14B\), which is expected: larger models have more directions in which a full\-rank update can differ from a rank\-rrone\. What does*not*decline is the agreement in sign betweenρ⁡\(Δ​Wfull,ablation\)\\rho\(\\Delta W^\{\\mathrm\{full\}\},\\text\{ablation\}\)and𝒩​𝒫​𝒜\\mathcal\{NPA\}, which is the quantity the paper’s claims rest on\. Table 14:Adapter\-free control\.ρ⁡\(Δ​Wfull,Δ​WLoRA\)\\rho\(\\Delta W^\{\\mathrm\{full\}\},\\Delta W^\{\\mathrm\{LoRA\}\}\)is the Spearman correlation between full\-parameter and LoRA per\-layer update profiles\.ρ\(ΔWfull,abl\.\)\\rho\(\\Delta W^\{\\mathrm\{full\}\},\\mathrm\{abl\.\}\)is the adapter\-free analogue of𝒩​𝒫​𝒜\\mathcal\{NPA\}, computed against the same ablation\-based Necessity rankings\. The final column repeats the LoRA\-based𝒩​𝒫​𝒜\\mathcal\{NPA\}from Table[11](https://arxiv.org/html/2609.16537#A5.T11)for comparison\. Signs agree on all1313checkpoints\. Qwen3\-32B, Llama\-3\.1\-70B, Mamba\-2\.8B, and Falcon\-Mamba\-7B were not retrained under full FT for compute reasons\.##### Update\-to\-weight normalization\. A related concern is that‖Δ​Wl‖F\\\|\\Delta W\_\{l\}\\\|\_\{F\}rewards layers that simply have larger weight matrices\. We therefore recompute the Plasticity ranking from the normalized quantity‖Δ​Wl‖F/‖Wl‖F\\\|\\Delta W\_\{l\}\\\|\_\{F\}/\\\|W\_\{l\}\\\|\_\{F\}, using the corresponding updated parameter set for each method\. The qualitative conclusion is unchanged: LoRA\-based𝒩​𝒫​𝒜\\mathcal\{NPA\}values are essentially stable under normalization, the evaluated transformers remain negative, the pure Mamba checkpoints and RWKV6\-3B remain positive, and the already near\-zero hybrid and boundary cases remain small in magnitude\. Normalization thus rescales the Plasticity profile without moving the architecture split\. ### E\.4Component\-level analysis of a hybrid checkpoint Zamba2\-2\.7B is the one checkpoint in our set that contains both block types, so it permits a within\-model version of the architecture comparison\. Its5454layers comprise4545Mamba blocks and99shared\-attention hybrid blocks at positions6,12,…,516,12,\\ldots,51\. We partition the layers by block type and compute each component’s share of three quantities: total LoRA update mass, total full\-FT update mass, and total ablation Necessity mass\. Table[15](https://arxiv.org/html/2609.16537#A5.T15)shows the dissociation reproduced inside a single model\. The attention component is16\.7%16\.7\\%of the layers but absorbs36\.3%36\.3\\%of LoRA update mass and31\.6%31\.6\\%of full\-FT update mass — roughly twice its layer share under either adaptation procedure — while carrying only10\.4%10\.4\\%of ablation Necessity mass, about0\.6×0\.6\\timesits layer share\. Necessity is instead concentrated in the Mamba blocks\. Zamba2’s near\-zero whole\-model𝒩​𝒫​𝒜\\mathcal\{NPA\}of\+0\.05\+0\.05is therefore not evidence that the effect is absent; it is the average of a transformer\-like component and an SSM\-like component with opposing profiles\. Because this rests on a single checkpoint with one attention/SSM ratio, we treat statements about other ratios as extrapolation beyond the measured setting\. Table 15:Component\-level decomposition of Zamba2\-2\.7B\. Shares are percentages of the model total;±\\pmvalues are standard deviations across tasks\. The attention component takes roughly twice its layer share of adaptation mass under both LoRA and full FT, but only about0\.6×0\.6\\timesits layer share of Necessity\.

### E\.5Proxy validation is conditional on the metric

The main causal\-validation protocol uses a shared likelihood\-based proxy so that top\-kkand bottom\-kkplacement can be compared across heterogeneous tasks\. This makes the comparison controlled, but it also means that the resulting directional labels should not be interpreted as universal claims about end\-task performance\.

Estimator choice reverses which tasks appear supported\. LoRA\-delta placement supports LongBench in 9/10 transformer checkpoints, whereas ablation supports it in 2/10\. On SQuAD, the ordering flips: 3/10 under LoRA delta and 9/10 under ablation\. These are the same models, training budget, and validation loop; only the importance estimator changes\. Code\-generation tasks favor LoRA delta under the proxy, while commonsense tasks show little structured support for either estimator\. These patterns show that estimator choice changes the placement conclusion even when the training and validation loop are held fixed\.

We then audit the proxy verdicts against task\-native metrics, including pass@1, exact match, F1, and LongBench aggregate scores\. The audit shows that the proxy metric is neither uniformly optimistic nor uniformly pessimistic\. In some cases, the proxy reports a top\-kkwin that does not appear under the native metric; in other cases, the native metric improves even when the proxy does not\. Thus, proxy validation captures one operational notion of selective adaptation, but it is not interchangeable with task\-native evaluation\.

Table 16:Task\-native MCQ accuracy audit for causal\-validation rows\. Each entry averages top\-kkand bottom\-kkaccuracy over the available MCQ tasks: MMLU, ARC\-Challenge, GPQA, PIQA, and WinoGrande\. When multiple runs are available for the same model and method, we average them\.Δ\\Deltais top\-kkminus bottom\-kk\. Generation\-style tasks are excluded because their main diagnostic validation uses perplexity rather than task\-native pass@1/F1/exact\-match metrics\.

## Appendix FFull Causal Validation Tables

Tables[17](https://arxiv.org/html/2609.16537#A6.T17)–[25](https://arxiv.org/html/2609.16537#A8.T25)present per\-task causal\-validation results for representative model–method pairs\. Generation rows report perplexity; MCQ rows report accuracy where available\. To keep the schema uniform, the metric columns below are labeled as Base, Topk, and Botk\. The final “Supp\.” column is a directional indicator only\. Rows marked with†are marginal near\-ties and should not be treated as strong practical wins\.

Table 17:Qwen3\-8B causal validation under LoRA delta \(k=6k=6of 36 layers, seed 42\)\. Generation rows report perplexity; MCQ rows report accuracy where available\.Table 18:Llama\-3\.1\-8B causal validation under LoRA delta \(k=6k=6of 32 layers, seed 42\)\. Generation rows report perplexity; MCQ rows report accuracy where available\.Table 19:Mamba\-790m causal validation under ablation \(k=5k=5of 48 layers\)\. Boundary layers dominate top\-kkselection\. Generation rows report perplexity; MCQ rows report accuracy where available\.Table 20:Llama\-3\.2\-3B causal validation under LoRA delta \(k=5k=5of 28 layers\)\. Generation rows report perplexity; MCQ rows report accuracy where available\.Table 21:Qwen3\-32B causal validation under LoRA delta \(k=8k=8of 64 layers\)\. Generation rows report perplexity; MCQ rows report accuracy where available\. The selected layers are bimodal, with an early cluster and a terminal cluster\.Table 22:Qwen3\-32B causal validation under ablation \(k=8k=8of 64 layers\)\. Generation rows report perplexity; MCQ rows report accuracy where available\. WinoGrande is directionally supported, but the margin is effectively a tie\.
## Appendix GLayer\-Pattern Summaries

### G\.1Per\-model top\-layer patterns

Table 23:Qwen3\-8B: top\-5 layers by LoRA\-delta attribution per task\. Bold entries indicate layers outside the terminal cluster L33–L35\.
### G\.2Structural layer\-selection patterns by method

Table 24:Structural comparison of top\-kklayer patterns across importance methods and representative models\. “Scattered \+ L0” indicates that L0 always appears in top\-kkand other layers are task\-variable\.

## Appendix HForgetting

Figure 20:Forgetting ratio by model scale\.Each point is the median forgetting ratio across seeds and task pairs; shaded bands show±1\\pm 1std\. Solid lines = SSMs \(0\.13–2\.8B\); dotted line = Hybrid \(Zamba2\-2\.7B,𝒩​𝒫​𝒜=\+0\.05\\mathcal\{NPA\}\{=\}\{\+\}0\.05\); dashed lines = Transformers \(3–9B\)\. Color encodes placement strategy: red = top\-kk\(most plastic\), green = bottom\-kk\(least plastic\), gray = random\-kk\. Within SSMs, the three lines are interleaved with no consistent ordering\. Within Transformers, top\-kksits above bottom\-kkat every scale\. Zamba2’s three strategies converge to uniformly low forgetting, consistent with its near\-zero NPA and distributed plasticity\. Falcon\-Mamba\-7B is excluded \(its 7B scale overlaps with the Transformer range; boundary results reported in text\)\. The boundary between SSM and Transformer scale regions is marked by a dotted vertical line\.![Refer to caption](https://arxiv.org/html/2609.16537v1/fig4_qwen_8b_vs_32b_bimodality.png)Figure 21:Task\-normalized LoRA\-delta patterns for Qwen3\-8B and Qwen3\-32B\. Panels \(a–b\) show per\-task heatmaps; panel \(c\) overlays the mean profiles by relative depth\. Qwen3\-32B develops an additional early\-layer peak alongside the terminal cluster, yielding a bimodal profile absent in smaller Qwen3 variants\. The bimodality is the proximate clue for the scale\-induced collapse of𝒞⁡\(M\)\\mathcal\{C\}\(M\)on the two largest transformers \(Section[5\.4](https://arxiv.org/html/2609.16537#S5.SS4)\)\.Table 25:Llama\-3\.1\-70B causal validation under LoRA delta \(k=10k\{=\}10of 80 layers\)\. Generation rows report perplexity; MCQ rows report accuracy where available\. Most tasks select terminal\-cluster layers; GSM8K additionally recruits early\-middle layers\. SQuAD is only marginally supported by the binary rule\.