How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts

# How Many Ranks Does a Token Need?A Noise-Floor Squelch for Rank-One LoRA Experts
Source: [https://arxiv.org/html/2610.19033](https://arxiv.org/html/2610.19033)
Owen CastellanosAffiliation:University of EdinburghIngrid SolbergAffiliation:Delft University of TechnologyYusuf DemirelAffiliation:University of Edinburgh

###### Abstract

Low\-rank adaptation \(LoRA\) applies every rank\-one component of its update to every token, and mixture\-of\-experts variants replace this with a learned router that activates a fixed number of components\. We argue that the number of components a token needs is a detection question with a classical answer\. A token’s responses to the rank\-one components of a trained LoRA consist of a few large values on a Gaussian noise floor whose width matches that of an untrained adapter, and the floor carries cross\-talk rather than adaptation\. We introduceSquelch, a router\-free mixture of rank\-one experts that \(i\) soft\-thresholds each token’s responses at a multiple of its own robust noise level, so the active rank varies per token; \(ii\) chooses that multiple for each module by Stein’s unbiased risk estimate, so no rank budget is specified; and \(iii\) replaces the resulting marginal keep\-or\-drop decisions by one explaining\-away step on the value Gram matrix, which suppresses redundant co\-active components and recruits complementary ones\.Squelchadds no trainable parameters to LoRA\. We prove that the squelch is within a logarithmic factor of the oracle input\-dependent allocation, that the selected threshold is risk\-consistent, and that explaining away is a descent step toward the joint sparse code whose error is controlled by the coherence of co\-active components\. On commonsense reasoning, knowledge, math, code, and GLUE with LLaMA\-3\.1\-8B, Qwen3, and DeBERTaV3,Squelchimproves over LoRA by 2\.2–3\.2 points and over the strongest mixture baseline by 0\.8–1\.2 points while activating 8\.6 of 32 components per token on average\.

## 1Introduction

Figure 1:A token’s rank\-one responses are a sparse signal on a noise floor\(LLaMA\-3\.1\-8B, LoRAr=32r\{=\}32trained on commonsense reasoning\)\. \(a\) Standardized responses of the trained adapter \(red\) against those of an untrained adapter \(dashed\): a Gaussian bulk plus heavy tails beyond the universal threshold\. \(b\) The number of above\-floor components per token varies with the token type\. \(c\) Zeroing sub\-floor responses after training preserves accuracy and improves held\-out MMLU \(right axis\), whereas keeping the same number of components by top\-kkor at random does not\. \(d\) Gap between the marginal \(thresholded\) code and the joint sparse code as co\-active value vectors become coherent \(random dictionaries, exact computation\); one explaining\-away step closes most of it\.Low\-rank adaptation \(LoRA\) fine\-tunes a frozen model by adding a productB​ABAof two thin matrices to each linear layer\([Hu et al\., 2022](https://arxiv.org/html/2610.19033#bib.bib1);[Dettmers et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib2)\)\. Written as a sum∑ibi​ai⊤\\sum\_\{i\}b\_\{i\}a\_\{i\}^\{\\\!\\top\}ofrrrank\-one components, the update applies allrrcomponents to every token\. Two lines of work have questioned this uniformity\. Rank\-allocation methods give different modules different ranks, but a module’s rank is still shared by all of its inputs\([Zhang et al\., 2023c](https://arxiv.org/html/2610.19033#bib.bib3);[Ding et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib4);[He et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib5);[Liu et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib6)\)\. Mixtures of LoRA experts route tokens among low\-rank experts\([Li et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib7);[Tian et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib8);[Zhuang et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib9)\), and recent designs push the expert down to a single rank\-one component\([Zhang et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib10);[Zhao et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib11);[Zou et al\., 2026b](https://arxiv.org/html/2610.19033#bib.bib12);[Lu et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib23)\)\. How many components fire is still fixed by hand or learned from the task loss: through a top\-kk, a probability mass, a learned threshold network, or a complexity predictor\([Zhao et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib11);[Huang et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib24);[Liu and Luo, 2024](https://arxiv.org/html/2610.19033#bib.bib25);[Li et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib26)\)\.

We ask the question directly:*how many rank\-one components does a token need?*Figure[1](https://arxiv.org/html/2610.19033#S1.F1)suggests an answer\. If we collect a token’s responsesui=a^i⊤​xu\_\{i\}=\\hat\{a\}\_\{i\}^\{\\\!\\top\}xto the normalized keys of a trained LoRA, a few of them are large and the rest form a Gaussian bulk whose width matches the response of an*untrained*adapter, i\.e\. of random projections \(panel a\)\. The number of responses above this floor differs widely across tokens \(panel b\)\. The bulk also does little useful work: zeroing it after training preserves in\-domain accuracy and even improves held\-out retention, whereas removing the same number of components at random, or by a fixed top\-kk, degrades accuracy \(panel c\)\. The bulk is cross\-talk\.

This turns rank allocation into a detection problem, the sparse normal\-means problem of signal processing\([Donoho and Johnstone, 1994](https://arxiv.org/html/2610.19033#bib.bib27)\), and the problem comes with classical answers\. One should threshold relative to a robust estimate of the noise level\([Donoho, 1995](https://arxiv.org/html/2610.19033#bib.bib28);[Hampel, 1974](https://arxiv.org/html/2610.19033#bib.bib29)\)\. The threshold can be chosen without labels by minimizing Stein’s unbiased risk estimate\([Stein, 1981](https://arxiv.org/html/2610.19033#bib.bib30);[Donoho and Johnstone, 1995](https://arxiv.org/html/2610.19033#bib.bib31)\)\. When the atoms overlap, marginal thresholding is no longer the right estimator and should give way to the joint sparse code, which iterative shrinkage approximates\([Tibshirani, 1996](https://arxiv.org/html/2610.19033#bib.bib32);[Daubechies et al\., 2004](https://arxiv.org/html/2610.19033#bib.bib33)\)\. Panel \(d\) shows why the last point matters\. As the value vectors of co\-active components become coherent, the marginal code drifts far from the joint code, and a single shrinkage step removes most of the drift\.

We buildSquelchon this view\.Squelchkeeps LoRA’s parameters and changes only how the rank\-one components are combined for each token\. It \(i\)*squelches*every response that does not clear a multiplecℓc\_\{\\ell\}of the token’s own noise floor, estimated by the median absolute response; \(ii\) setscℓc\_\{\\ell\}for each module by minimizing Stein’s unbiased risk estimate over the tokens of a batch; and \(iii\) applies one*explaining\-away*step that uses the Gram matrix of the value vectors to turn marginal keep\-or\-drop decisions into a joint one\. The per\-token rank is then an outcome rather than a hyperparameter:Squelchuses 8\.6 of 32 components per token on average, more on numerals and less on function words, and no component is active for 8\.7% of token–module pairs\. Our contributions are:

- •A statistical view of per\-token rank\.Rank\-one responses behave like a sparse signal on a noise floor, so choosing how many components a token uses is sparse detection \(§[1](https://arxiv.org/html/2610.19033#S1), §[3](https://arxiv.org/html/2610.19033#S3)\)\.
- •A router\-free, budget\-free method\.Squelchcombines noise\-floor thresholding, SURE\-selected thresholds, and one explaining\-away step, and adds no trainable parameters to LoRA \(§[3](https://arxiv.org/html/2610.19033#S3)\)\.
- •Guarantees\.The squelch is within a logarithmic factor of the oracle allocation, the SURE threshold is risk\-consistent, and explaining away is a descent step toward the joint code whose error is governed by co\-active coherence \(§[4](https://arxiv.org/html/2610.19033#S4)\)\.
- •Evidence\.On five task families and seven backbones,Squelchoutperforms ten LoRA and mixture baselines, with ablations and a controlled study that isolate each step \(§[5](https://arxiv.org/html/2610.19033#S5), App\.[E](https://arxiv.org/html/2610.19033#A5)\)\.

## 2Related Work

![Refer to caption](https://arxiv.org/html/2610.19033v1/figures/fig_overview.png)Figure 2:Squelchin one adapted layer\.The token’s responses torrunit\-norm keys are squelched at a multiplecℓc\_\{\\ell\}of the token’s own noise floor, wherecℓc\_\{\\ell\}is chosen per module by Stein’s unbiased risk estimate\. One explaining\-away step on the value GramB⊤​BB^\{\\\!\\top\}\\\!Bturns the marginal code into a joint one\. The surviving components form the token’s update, so its rank varies per token\.Squelchuses LoRA’s parametersA,BA,Band adds none\.#### Rank allocation\.

AdaLoRA prunes singular values by an importance score\([Zhang et al\., 2023c](https://arxiv.org/html/2610.19033#bib.bib3)\), SoRA gates ranks with a proximalℓ1\\ell\_\{1\}step\([Ding et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib4)\), and later methods allocate ranks by gradients, beam search, learned elasticity, entropy, or variational relevance\([He et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib5);[Gu et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib34);[Chang et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib35);[Liu et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib6);[Doyle et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib36)\)\. These methods change*which modules*receive rank; every input of a module shares it\. Input\-dependent allocation is rarer\. RaNA masks ranks and neurons per input for faster inference\([Garcia et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib37)\), and Flexi\-LoRA predicts a rank from input complexity\([Li et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib26)\)\. Gradient\-based importance scores can also mislead\([Sawant, 2026](https://arxiv.org/html/2610.19033#bib.bib38)\), which motivates a budget that does not depend on them\.

#### Mixtures of LoRA and rank\-one experts\.

MoE\-LoRA methods route tokens to low\-rank experts\([Zadouri et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib39);[Dou et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib40);[Li et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib7);[Wu et al\., 2024b](https://arxiv.org/html/2610.19033#bib.bib41);[Tian et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib8);[Zhuang et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib9);[Fan et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib42);[Mu et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib43)\)\. Finer experts help: T\-REX and MoDE mix rank\-one experts\([Zhang et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib10);[Ning et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib44)\), SMoRA treats every rank as an expert\([Zhao et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib11)\), FlyLoRA routes by the top\-kkresponses of a frozen random down\-projection\([Zou et al\., 2026b](https://arxiv.org/html/2610.19033#bib.bib12)\), and MoRAM lets rank\-one memories self\-activate through their keys\([Lu et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib23)\)\. Dynamic MoE routing varies the number of experts through top\-pprules, null experts, ReLU gates, or confidence\([Huang et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib24);[Zeng et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib45);[Wang et al\., 2025b](https://arxiv.org/html/2610.19033#bib.bib46);[Jin et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib47);[Saliencro et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib48)\)\.Squelchshares self\-routing with FlyLoRA and MoRAM but derives the number of active components from a noise model, the threshold from a risk estimate, and the combination rule from the joint sparse code\.

#### Shrinkage and sparse codes in neural networks\.

Soft thresholding and iterative shrinkage are classical tools for sparse estimation\([Donoho, 1995](https://arxiv.org/html/2610.19033#bib.bib28);[Daubechies et al\., 2004](https://arxiv.org/html/2610.19033#bib.bib33);[Beck and Teboulle, 2009](https://arxiv.org/html/2610.19033#bib.bib49)\), and unrolled or competitive variants learn sparse codes quickly\([Rozell et al\., 2008](https://arxiv.org/html/2610.19033#bib.bib50);[Chen et al\., 2018](https://arxiv.org/html/2610.19033#bib.bib51);[Monga et al\., 2021](https://arxiv.org/html/2610.19033#bib.bib52)\)\. In language models, thresholded activations yield activation sparsity\([Liu et al\., 2025a](https://arxiv.org/html/2610.19033#bib.bib53);[Lee et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib54);[Zhang et al\., 2025c](https://arxiv.org/html/2610.19033#bib.bib55)\)and sparse autoencoders\([Cunningham et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib56);[Gao et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib57);[Rajamanoharan et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib58)\)\. These methods fix a sparsity level or learn a threshold; TEAL, for instance, sets one magnitude cutoff per tensor to meet a target sparsity\([Liu et al\., 2025a](https://arxiv.org/html/2610.19033#bib.bib53)\)\.Squelchinstead derives the threshold from a noise model and applies it to the components of an adapter, so the cutoff follows each token’s own noise floor\. Extended related work is in Appendix[A](https://arxiv.org/html/2610.19033#A1)\.

## 3Squelch

### 3\.1Rank\-one components as a code

A LoRA layer computesy=W0​x\+s​B​A​xy=W\_\{0\}x\+sBAxwithA∈ℝr×dA\\in\\mathbb\{R\}^\{r\\times d\},B∈ℝdout×rB\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}and scaless\([Hu et al\., 2022](https://arxiv.org/html/2610.19033#bib.bib1)\)\. Writingai⊤a\_\{i\}^\{\\\!\\top\}for the rows ofAAandbib\_\{i\}for the columns ofBB,

s​B​A​x=s​∑i=1rbi​\(ai⊤​x\)=s​B​u,sBAx=s\\textstyle\\sum\_\{i=1\}^\{r\}b\_\{i\}\\,\(a\_\{i\}^\{\\\!\\top\}x\)=sBu,\(1\)so LoRA forms a*code*u=A​xu=Axand synthesizes the update from the value dictionaryBB\. The code is dense: every component contributes to every token\. We call a component*active*forxxif its coefficient is nonzero, and the number of active components is the token’s*rank*, since the update restricted to a neighbourhood ofxxhas rank at most that number \(Proposition[1](https://arxiv.org/html/2610.19033#Thmproposition1)\)\.Squelchreplaces the dense codeuuby a sparse codez⁡\(x\)z\(x\)in three steps \(Figure[2](https://arxiv.org/html/2610.19033#S2.F2), Algorithm[1](https://arxiv.org/html/2610.19033#algorithm1)\)\. We normalize the keys\([Büyükakyüz, 2024](https://arxiv.org/html/2610.19033#bib.bib16)\),a^i=ai/‖ai‖\\hat\{a\}\_\{i\}=a\_\{i\}/\\\|a\_\{i\}\\\|, so that responsesu=A^​xu=\\hat\{A\}xare comparable across components; this does not restrict the hypothesis class, since the scale ofaia\_\{i\}can move intobib\_\{i\}\.

### 3\.2Detect: the noise floor

For a component that is irrelevant toxx, the responsea^i⊤​x\\hat\{a\}\_\{i\}^\{\\\!\\top\}xbehaves like a random projection ofxx: approximately Gaussian with a scale common to all such components\. The few relevant components stand out above it \(Figure[1](https://arxiv.org/html/2610.19033#S1.F1)a\)\. We estimate the scale of this floor per token with the median absolute response\([Hampel, 1974](https://arxiv.org/html/2610.19033#bib.bib29);[Donoho, 1995](https://arxiv.org/html/2610.19033#bib.bib28)\),

σ^​\(x\)=medi⁡\|ui​\(x\)\|/Φ−1​\(34\),\\hat\{\\sigma\}\(x\)=\\operatorname\{med\}\_\{i\}\|u\_\{i\}\(x\)\|\\,/\\,\\Phi^\{\-1\}\(\\tfrac\{3\}\{4\}\),\(2\)which ignores up to half of the components carrying signal, and soft\-threshold the responses:

z0=𝒮τ⁡\(x\)​\(u\),τ⁡\(x\)=cℓ​σ^​\(x\),z^\{0\}=\\mathcal\{S\}\_\{\\tau\(x\)\}\(u\),\\qquad\\tau\(x\)=c\_\{\\ell\}\\,\\hat\{\\sigma\}\(x\),\(3\)where𝒮τ​\(v\)=sign⁡\(v\)​max⁡\(\|v\|−τ,0\)\\mathcal\{S\}\_\{\\tau\}\(v\)=\\mathrm\{sign\}\(v\)\\max\(\|v\|\-\\tau,0\)acts elementwise andcℓc\_\{\\ell\}is a per\-module multiple\. A component survives only if it clears the token’s own floor\. Tokens with strong, structured responses keep many components, weak tokens keep few, and a token whose responses are all floor keeps none and passes through the frozen layer unchanged\. Soft thresholding is continuous inuu, so gradients reach every surviving component\. Because every response belongs to the floor at initialization, the first5%5\\%of steps rampcℓc\_\{\\ell\}linearly from zero, i\.e\. train as a dense LoRA\.

### 3\.3Decide: a budget from an unbiased risk

The multiplecℓc\_\{\\ell\}sets the expected rank of moduleℓ\\ell, and fitting it to the task loss drives it toward zero, since denser codes fit the training loss more easily \(Table[3](https://arxiv.org/html/2610.19033#S5.T3)\)\. We instead treatcℓc\_\{\\ell\}as a denoising parameter\. For standardized responsesv=u/σ^v=u/\\hat\{\\sigma\}, Stein’s lemma gives an unbiased estimate of the squared error of𝒮c​\(v\)\\mathcal\{S\}\_\{c\}\(v\)with respect to the unknown signal,

SURE⁡\(c,v\)=r−2​Nc​\(v\)\+∑imin⁡\(vi2,c2\),\\mathrm\{SURE\}\(c;v\)=r\-2N\_\{c\}\(v\)\+\\textstyle\\sum\_\{i\}\\min\(v\_\{i\}^\{2\},c^\{2\}\),\(4\)whereNc​\(v\)=\#⁡\{i:\|vi\|≤c\}N\_\{c\}\(v\)=\\\#\\\{i:\|v\_\{i\}\|\\leq c\\\}, and it requires no labels\([Stein, 1981](https://arxiv.org/html/2610.19033#bib.bib30);[Donoho and Johnstone, 1995](https://arxiv.org/html/2610.19033#bib.bib31)\)\. Every2020steps, each module setsc^ℓ=arg⁡min⁡∑xc∈\[0,cU\]⁡SURE⁡\(c,v⁡\(x\)\)\\hat\{c\}\_\{\\ell\}=\\arg\\min\_\{c\\in\[0,c\_\{\\mathrm\{U\}\}\]\}\\sum\_\{x\}\\mathrm\{SURE\}\(c;v\(x\)\)over the tokens of the batch and updatescℓc\_\{\\ell\}by an exponential moving average\. The capcU=2​log⁡rc\_\{\\mathrm\{U\}\}=\\sqrt\{2\\log r\}is the universal threshold, and, as in SureShrink,cUc\_\{\\mathrm\{U\}\}is used directly when the responses show no signal above the floor\([Donoho and Johnstone, 1995](https://arxiv.org/html/2610.19033#bib.bib31)\)\. Each module’s rank budget therefore emerges from its own responses, and no global budget or per\-module rank is specified\.

### 3\.4Explain away: from marginal to joint decisions

Equation \([3](https://arxiv.org/html/2610.19033#S3.E3)\) decides for each component in isolation\. When the values of co\-active components overlap, these marginal decisions double count shared directions and miss components that are needed only in combination \(Figure[1](https://arxiv.org/html/2610.19033#S1.F1)d\)\. The code that accounts for the overlap solves a weighted LASSO over the value dictionary,

minz⁡12​‖B​z‖2−\(D​u\)⊤​z\+τ​∑iDi​i​\|zi\|,\\min\_\{z\}\\ \\tfrac\{1\}\{2\}\\\|Bz\\\|^\{2\}\-\(Du\)^\{\\\!\\top\}z\+\\tau\\textstyle\\sum\_\{i\}D\_\{ii\}\|z\_\{i\}\|,\(5\)whereG=B⊤​BG=B^\{\\\!\\top\}BandD=diag⁡\(G\)D=\\mathrm\{diag\}\(G\)\. This is the LASSO for a target update whose per\-component regression coefficients areuu\. If the values are orthogonal, \([5](https://arxiv.org/html/2610.19033#S3.E5)\) decouples and its solution is exactlyz0z^\{0\}\.Squelchtakes one proximal\-gradient step fromz0z^\{0\},

z=𝒮η​τ​\(z0\+η⁡\(u−\(I\+M\)​z0\)\),z=\\mathcal\{S\}\_\{\\eta\\tau\}\\big\(z^\{0\}\+\\eta\\,\(u\-\(I\+M\)z^\{0\}\)\\big\),\(6\)withM=D−1​\(G−D\)M=D^\{\-1\}\(G\-D\)andη=1/λmax\(D−1/2GD−1/2\)\\eta=1/\\lambda\_\{\\max\}\(D^\{\-1/2\}GD^\{\-1/2\}\)\. The termM​z0Mz^\{0\}*explains away*what co\-active components already cover: a component whose value is aligned with a stronger co\-active one is suppressed, and one whose contribution complements them can be recruited\. The step uses only the value Gram, costsO⁡\(r2\)O\(r^\{2\}\)per token, and adds no parameters\. The output of the layer isy=W0​x\+s​B​zy=W\_\{0\}x\+sBz\.

Algorithm 1Squelchlayer \(training step\)Input:tokens

XX;

W0W\_\{0\};

A,BA,B; multiple

cℓc\_\{\\ell\}; step

tt
u←A^​xu\\leftarrow\\hat\{A\}xfor each

x∈Xx\\in X//unit\-norm keys

σ^​\(x\)←medi⁡\|ui\|/0\.6745\\hat\{\\sigma\}\(x\)\\leftarrow\\operatorname\{med\}\_\{i\}\|u\_\{i\}\|/0\.6745

τ⁡\(x\)←min⁡\(1,t/Tw\)​cℓ​σ^​\(x\)\\tau\(x\)\\leftarrow\\min\(1,t/T\_\{w\}\)\\,c\_\{\\ell\}\\,\\hat\{\\sigma\}\(x\)//warm\-up ramp

z0←𝒮τ​\(u\)z^\{0\}\\leftarrow\\mathcal\{S\}\_\{\\tau\}\(u\)//squelch, Eq\. \([3](https://arxiv.org/html/2610.19033#S3.E3)\)

G←B⊤​BG\\leftarrow B^\{\\\!\\top\}B;

M←D−1​\(G−D\)M\\leftarrow D^\{\-1\}\(G\-D\)
η←1/λmax\(D−1/2GD−1/2\)\\eta\\leftarrow 1/\\lambda\_\{\\max\}\(D^\{\-1/2\}GD^\{\-1/2\}\)

z←𝒮η​τ​\(z0\+η⁡\(u−\(I\+M\)​z0\)\)z\\leftarrow\\mathcal\{S\}\_\{\\eta\\tau\}\\big\(z^\{0\}\+\\eta\(u\-\(I\{\+\}M\)z^\{0\}\)\\big\)//Eq\. \([6](https://arxiv.org/html/2610.19033#S3.E6)\)

y←W0​x\+s​B​zy\\leftarrow W\_\{0\}x\+sBz

if*tmod20=0t\\bmod 20=0*then

c^←arg⁡min⁡∑xc≤cU⁡SURE⁡\(c,u/σ^\)\\hat\{c\}\\leftarrow\\arg\\min\_\{c\\leq c\_\{\\mathrm\{U\}\}\}\\sum\_\{x\}\\mathrm\{SURE\}\(c;u/\\hat\{\\sigma\}\)
cℓ←0\.9​cℓ\+0\.1​c^c\_\{\\ell\}\\leftarrow 0\.9\\,c\_\{\\ell\}\+0\.1\\,\\hat\{c\}
end if

#### Cost\.

Squelchhas exactly LoRA’s parameters\. Per token it adds a median overrrresponses and anr×rr\\times rproduct; per module and step it addsB⊤​BB^\{\\\!\\top\}Band a power iteration on anr×rr\\times rmatrix\. Inactive components skip the up\-projection\. Like every mixture of LoRA experts, the update depends on the input and cannot be merged intoW0W\_\{0\}\.

## 4Theory

The statements below idealize the responses as a sparse signal plus Gaussian noise; proofs are in Appendix[B](https://arxiv.org/html/2610.19033#A2), and Table[4](https://arxiv.org/html/2610.19033#A4.T4)checks every statement numerically\.

###### Proposition 1\(LoRA as a special case; conic structure\)\.

\(i\) Withcℓ=0c\_\{\\ell\}=0and no explaining away,Squelchequals LoRA\. \(ii\) Forα\>0\\alpha\>0,z⁡\(α​x\)=α​z​\(x\)z\(\\alpha x\)=\\alpha z\(x\): active sets depend only onx/‖x‖x/\\\|x\\\|, and the update is a continuous, positively homogeneous, piecewise\-linear function ofxx\. \(iii\) On each piece, the update has rank at most\|supp​z​\(x\)\|\|\\mathrm\{supp\}z\(x\)\|\.

###### Proposition 2\(Near\-oracle detection\)\.

Letu=θ\+σ​εu=\\theta\+\\sigma\\varepsilonwithε∼𝒩⁡\(0,Ir\)\\varepsilon\\sim\\mathcal\{N\}\(0,I\_\{r\}\), and letτU=σ​2​log⁡r\\tau\_\{U\}=\\sigma\\sqrt\{2\\log r\}\. \(a\)𝔼​‖𝒮τU​\(u\)−θ‖2≤\(2​log⁡r\+1\)​\(σ2\+∑imin⁡\(θi2,σ2\)\)\\mathbb\{E\}\\\|\\mathcal\{S\}\_\{\\tau\_\{U\}\}\(u\)\-\\theta\\\|^\{2\}\\leq\(2\\log r\+1\)\\big\(\\sigma^\{2\}\+\\sum\_\{i\}\\min\(\\theta\_\{i\}^\{2\},\\sigma^\{2\}\)\\big\), where the sum is the risk of the oracle that keeps componentiiiff\|θi\|\>σ\|\\theta\_\{i\}\|\>\\sigma\. \(b\) No component withθi=0\\theta\_\{i\}=0is active with probability at least1−1/π​log⁡r1\-1/\\sqrt\{\\pi\\log r\}, and each component with\|θi\|≥2​τU\|\\theta\_\{i\}\|\\geq 2\\tau\_\{U\}is active with probability at least1−1/\(r​π​log⁡r\)1\-1/\(r\\sqrt\{\\pi\\log r\}\)\.

###### Proposition 3\(Budget\-free threshold\)\.

Letvn=θn\+εnv\_\{n\}=\\theta\_\{n\}\+\\varepsilon\_\{n\},n≤Nn\\leq N, with independentεn∼𝒩⁡\(0,Ir\)\\varepsilon\_\{n\}\\sim\\mathcal\{N\}\(0,I\_\{r\}\)\. \(a\) For everyc≥0c\\geq 0,𝔼​∑nSURE⁡\(c,vn\)=𝔼​∑n‖𝒮c​\(vn\)−θn‖2\\mathbb\{E\}\\sum\_\{n\}\\mathrm\{SURE\}\(c;v\_\{n\}\)=\\mathbb\{E\}\\sum\_\{n\}\\\|\\mathcal\{S\}\_\{c\}\(v\_\{n\}\)\-\\theta\_\{n\}\\\|^\{2\}\. \(b\)supc∈\[0,cU\]\|∑n\(SURE⁡\(c,vn\)−‖𝒮c​\(vn\)−θn‖2\)\|/\(N​r\)→0\\sup\_\{c\\in\[0,c\_\{\\mathrm\{U\}\}\]\}\|\\sum\_\{n\}\(\\mathrm\{SURE\}\(c;v\_\{n\}\)\-\\\|\\mathcal\{S\}\_\{c\}\(v\_\{n\}\)\-\\theta\_\{n\}\\\|^\{2\}\)\|/\(Nr\)\\to 0in probability, so the risk ofc^=arg⁡min​∑nSURE⁡\(c,vn\)\\hat\{c\}=\\arg\\min\\sum\_\{n\}\\mathrm\{SURE\}\(c;v\_\{n\}\)exceeds the best risk on\[0,cU\]\[0,c\_\{\\mathrm\{U\}\}\]byop​\(N​r\)o\_\{p\}\(Nr\)\.

###### Proposition 4\(Explaining away\)\.

LetFFbe the objective in \([5](https://arxiv.org/html/2610.19033#S3.E5)\)\. \(a\) Equation \([6](https://arxiv.org/html/2610.19033#S3.E6)\) is a proximal\-gradient step onFFin the metricDD, soF⁡\(z\)≤F⁡\(z0\)F\(z\)\\leq F\(z^\{0\}\), with equality iffz0z^\{0\}minimizesFF\. \(b\) If two co\-active components have identical values and responsesu\>τu\>\\tau, the first pass outputs2​\(u−τ\)​b2\(u\-\\tau\)bwhile \([6](https://arxiv.org/html/2610.19033#S3.E6)\) outputs\(u−τ\)​b\(u\-\\tau\)b, the LASSO solution\. \(c\) Fori≠ji\\neq j,∂ℒ/∂Gi​j=0\\partial\\mathcal\{L\}/\\partial G\_\{ij\}=0unless one of the two components is active inz0z^\{0\}and the other is active inzz\. \(d\) With unit\-norm values, and whenz0z^\{0\},zz, and the minimizerz⋆z^\{\\star\}share their supportSSand signs,‖z0−z⋆‖≤\(\|S\|−1\)​μS​‖z⋆‖\\\|z^\{0\}\-z^\{\\star\}\\\|\\leq\(\|S\|\-1\)\\mu\_\{S\}\\\|z^\{\\star\}\\\|and‖z−z⋆‖≤‖I−η​GS​S‖2​‖z0−z⋆‖\\\|z\-z^\{\\star\}\\\|\\leq\\\|I\-\\eta G\_\{SS\}\\\|\_\{2\}\\,\\\|z^\{0\}\-z^\{\\star\}\\\|, whereμS\\mu\_\{S\}is the largest coherence\|bi⊤​bj\|\|b\_\{i\}^\{\\\!\\top\}b\_\{j\}\|withinSS\.

Proposition[2](https://arxiv.org/html/2610.19033#Thmproposition2)says that the squelch does almost as well as an oracle that knows which components a token needs, paying only thelog⁡r\\log rfactor that no estimator can avoid uniformly\([Donoho and Johnstone, 1994](https://arxiv.org/html/2610.19033#bib.bib27)\)\. Proposition[3](https://arxiv.org/html/2610.19033#Thmproposition3)replaces the rank budget with a quantity that can be estimated from the responses alone\. Proposition[4](https://arxiv.org/html/2610.19033#Thmproposition4)explains the third step: marginal decisions are exact only when co\-active values are incoherent; otherwise their error grows with\(\|S\|−1\)​μS\(\|S\|\-1\)\\mu\_\{S\}, and one explaining\-away step contracts it\. Part \(c\) makes the step*co\-activation\-aware*: the task gradient reaches the overlap of two values only through tokens on which both are active\.

Table 1:Commonsense reasoning\(accuracy, %, mean of three seeds\)\.Params: trainable parameters relative to the backbone\.Rank: rank\-one components that contribute to a token’s update, averaged over tokens and modules\. Single adapters, then mixtures\.Bold: best score\.Squelchis in the header color\.Table 2:Knowledge, science, math, and code\(accuracy or pass@1, %, mean of three seeds; one adapter per task\)\. Parameter budgets and active ranks as in Table[1](https://arxiv.org/html/2610.19033#S4.T1)\.
## 5Experiments

### 5\.1Setup

#### Models and tasks\.

We adapt LLaMA\-3\.1\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib59)\)and Qwen3\-8B\-Base\([Yang et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib60)\), and use Qwen3\-0\.6B/1\.7B/4B/14B\-Base for scaling\.*Commonsense reasoning*trains on Commonsense170K and evaluates on BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC\-e/c, and OBQA\([Hu et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib61);[Clark et al\., 2019](https://arxiv.org/html/2610.19033#bib.bib62);[Bisk et al\., 2020](https://arxiv.org/html/2610.19033#bib.bib63);[Sap et al\., 2019](https://arxiv.org/html/2610.19033#bib.bib64);[Zellers et al\., 2019](https://arxiv.org/html/2610.19033#bib.bib65);[Sakaguchi et al\., 2020](https://arxiv.org/html/2610.19033#bib.bib66);[Clark et al\., 2018](https://arxiv.org/html/2610.19033#bib.bib67);[Mihaylov et al\., 2018](https://arxiv.org/html/2610.19033#bib.bib68)\)\. The*domain*suite trains one adapter per task: MMLU with its auxiliary training set\([Hendrycks et al\., 2021a](https://arxiv.org/html/2610.19033#bib.bib69)\), ScienceQA\([Lu et al\., 2022](https://arxiv.org/html/2610.19033#bib.bib70)\), math on a 100K subset of MetaMathQA evaluated on GSM8K and MATH\([Yu et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib71);[Cobbe et al\., 2021](https://arxiv.org/html/2610.19033#bib.bib72);[Hendrycks et al\., 2021b](https://arxiv.org/html/2610.19033#bib.bib73)\), and code on a 100K subset of CodeFeedback evaluated on HumanEval and MBPP\([Zheng et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib74);[Chen et al\., 2021](https://arxiv.org/html/2610.19033#bib.bib75);[Austin et al\., 2021](https://arxiv.org/html/2610.19033#bib.bib76)\)\. GLUE with DeBERTaV3\-base\([Wang et al\., 2019](https://arxiv.org/html/2610.19033#bib.bib77);[He et al\., 2021](https://arxiv.org/html/2610.19033#bib.bib78)\), joint multi\-task training, and forgetting are reported in Appendix[H](https://arxiv.org/html/2610.19033#A8)\.

#### Baselines\.

Single adapters: LoRA, DoRA\([Liu et al\., 2024b](https://arxiv.org/html/2610.19033#bib.bib79)\), AdaLoRA, and GoRA\. Mixtures: HydraLoRA, MixLoRA, AdaMoLE, LD\-MoLE, SMoRA, and FlyLoRA\. All adapters target the seven projection matrices and are matched to LoRA withr=32r\{=\}32in trainable parameters, except FlyLoRA, whose frozen down\-projection halves them, and SMoRA, whose router adds a third\. Rank\-wise baselines use3232components withk=8k\{=\}8active, the configuration of FlyLoRA\([Zou et al\., 2026b](https://arxiv.org/html/2610.19033#bib.bib12)\)\.

#### Implementation\.

Squelchusesr=32r\{=\}32,α=64\\alpha\{=\}64, a5%5\\%dense warm\-up, SURE every2020steps with EMA0\.90\.9, and capcU=2​log⁡32c\_\{\\mathrm\{U\}\}\{=\}\\sqrt\{2\\log 32\}, with identical settings for every task and backbone\. All methods share the optimizer, schedule, and data; we report the mean of three seeds \(standard deviations and full settings in Appendices[G](https://arxiv.org/html/2610.19033#A7)and[H](https://arxiv.org/html/2610.19033#A8)\)\.

### 5\.2Main Results

![Refer to caption](https://arxiv.org/html/2610.19033v1/fig_mechanism.png)Figure 3:WhereSquelchspends rank\(LLaMA\-3\.1\-8B\)\. \(a\) Mean active rank by layer and module: FFN projections in the middle and upper layers use the most components\. \(b\) Mean active rank by task: math and code use more components than knowledge and science\. \(c\) SURE\-selected multiplescℓc\_\{\\ell\}\(±\\pms\.d\. across layers\) stay below the universal cap and are lower for FFN modules\. \(d\) Explaining away inhibits and recruits the most components in FFN modules, where co\-active values are most coherent \(right axis\)\.Figure 4:Capacity, budget, and threshold selection\(commonsense unless stated\)\. \(a\) Accuracy vs\. dictionary sizerr; numbers giveSquelch’s mean active rank\. \(b\) Accuracy vs\. mean active rank:Squelchwith a target rank \(budgeted variant\), top\-kkvariants, and LoRA; the star is SURE’s choice\. \(c\) Accuracy of global fixed multiplesccrelative to the best one; stars mark per\-module SURE\. \(d\) Gain over LoRA across Qwen3 sizes; numbers giveSquelch’s mean active rank\.Squelchattains the best average in all four settings \(Tables[1](https://arxiv.org/html/2610.19033#S4.T1)and[2](https://arxiv.org/html/2610.19033#S4.T2)\)\. On commonsense reasoning it reaches 87\.04 on LLaMA\-3\.1\-8B and 89\.28 on Qwen3\-8B,\+2\.64\+2\.64and\+2\.18\+2\.18over LoRA and\+0\.93\+0\.93and\+0\.77\+0\.77over the strongest baseline, FlyLoRA\. On the domain suite, the margins over FlyLoRA are\+1\.19\+1\.19and\+0\.75\+0\.75\. Three patterns stand out\. First, input\-adaptive methods beat input\-shared ones at comparable parameter budgets\. LD\-MoLE, SMoRA, and FlyLoRA all exceed AdaLoRA and GoRA, so where rank is spent matters more than how it is divided among modules\. Second,Squelchis better than the rank\-wise mixtures at a similar active rank \(8\.68\.6vs\.88\) and without a router: the gain comes from choosing*which and how many*components fire, not from activating more\. Third, gains concentrate on tasks that require composition, namely ARC\-c, WinoGrande, OBQA, GSM8K, and code\. On MMLU, whose adaptation is closer to a uniform shift, FlyLoRA andSquelchare within0\.10\.1on LLaMA, and on HellaSwag SMoRA is best on Qwen3; both are near saturation\. The improvements exceed three seed standard deviations in every setting \(Table[9](https://arxiv.org/html/2610.19033#A8.T9)\)\.

### 5\.3Ablations

Table[3](https://arxiv.org/html/2610.19033#S5.T3)removes one ingredient at a time\.*The floor must be per token*: a per\-layer noise scale loses0\.460\.46, and a non\-robust RMS estimate, inflated by the signal itself, over\-squelches \(k¯=5\.7\\bar\{k\}\{=\}5\.7\)\.*The threshold must be decided, not learned*: learningcℓc\_\{\\ell\}by the task loss drives it to near zero \(k¯=27\.9\\bar\{k\}\{=\}27\.9\) and falls back close to LoRA \(85\.0885\.08\); the universal threshold without SURE is too conservative \(k¯=5\.1\\bar\{k\}\{=\}5\.1,−0\.27\-0\.27\); a fixed top\-kkat the same average rank loses0\.730\.73\.*Explaining away matters most among the three steps*: without it, the code is0\.270\.27from the joint code instead of0\.090\.09, the squelch keeps fewer components \(k¯=6\.9\\bar\{k\}\{=\}6\.9\), and accuracy drops by0\.610\.61\. Blocking the gradient throughB⊤​BB^\{\\\!\\top\}\\\!Bcosts0\.180\.18, the co\-activation\-aware credit of Proposition[4](https://arxiv.org/html/2610.19033#Thmproposition4)\(c\)\. A penalty on co\-active coherence partially substitutes for explaining away but adds nothing on top of it\. The dense warm\-up is necessary: without it, a third of the components never clear the floor\.

Table 3:Ablationson LLaMA\-3\.1\-8B\. CS/Dom\.: commonsense/domain averages;k¯\\bar\{k\}: mean active rank; Gap:‖z−z⋆‖/‖z⋆‖\\\|z\-z^\{\\star\}\\\|/\\\|z^\{\\star\}\\\|to the joint code at the same threshold\.
### 5\.4Mechanism Analysis

#### Rank follows the input\.

Per\-token ranks range from00to over2020\(Figure[7](https://arxiv.org/html/2610.19033#A8.F7)\)\. Numerals and operators use11\.811\.8components on average, content words9\.49\.4, function words5\.35\.3, and punctuation3\.93\.9\. Across tasks, math and code use the most \(Figure[3](https://arxiv.org/html/2610.19033#S5.F3)b\)\. Out of domain, the squelch closes more often: on MMLU prompts, a commonsense adapter leaves 21\.4% of token–module pairs without active components, against 8\.7% in domain\. This selectivity explains the smaller forgetting ofSquelch\(−1\.13\-1\.13vs\.−3\.87\-3\.87points for LoRA on held\-out benchmarks; Table[12](https://arxiv.org/html/2610.19033#A8.T12)\)\.

#### Budgets emerge per module\.

SURE assigns lower multiples, and thus more components, to FFN projections than to attention queries and keys \(Figure[3](https://arxiv.org/html/2610.19033#S5.F3)c\), and the resulting ranks peak in the upper\-middle layers \(Figure[3](https://arxiv.org/html/2610.19033#S5.F3)a\)\. The selected multiples stabilize within the first tenth of training \(Figure[6](https://arxiv.org/html/2610.19033#A8.F6)\) and stay below the universal cap everywhere, so the cap acts only as a safeguard\.

#### Explaining away acts where values overlap\.

Explaining away removes55–17%17\\%of first\-pass components and recruits88–26%26\\%more, most in FFN modules, where the values of co\-active components are most coherent \(Figure[3](https://arxiv.org/html/2610.19033#S5.F3)d\)\. Restricting it to attention modules recovers0\.210\.21of its0\.610\.61\-point gain, and restricting it to FFN modules recovers0\.490\.49, in line with Proposition[4](https://arxiv.org/html/2610.19033#Thmproposition4)\(d\)\. A controlled synthetic study with a known per\-input rank \(Appendix[E](https://arxiv.org/html/2610.19033#A5)\) shows the same division of labour: the squelch alone is conservative, and explaining away makes the active count track the true rank\.

### 5\.5Capacity, Budget, and Scaling

#### Larger dictionaries keep helping\.

Because each token uses only what it needs,Squelchconverts a larger dictionary into accuracy without a larger per\-token rank budget\. Fromr=8r\{=\}8to128128it gains2\.62\.6points while its mean active rank grows only from2\.92\.9to15\.815\.8; LoRA saturates beyondr=32r\{=\}32and the top\-kkmixtures aroundr=64r\{=\}64\(Figure[4](https://arxiv.org/html/2610.19033#S5.F4)a\)\.

#### The SURE choice sits at the knee\.

With a target average rank enforced by a dual variable,Squelchdominates top\-kkrules at every budget, including its own top\-kkvariant \(Figure[4](https://arxiv.org/html/2610.19033#S5.F4)b\)\. The budget\-free SURE point \(k¯=8\.6\\bar\{k\}\{=\}8\.6\) lies at the knee of this frontier, and per\-module SURE beats the best*global*multiple found by grid search on three tasks \(Figure[4](https://arxiv.org/html/2610.19033#S5.F4)c\)\. The budget is thus found, not tuned\.

#### Scale and cost\.

The gain over LoRA persists from0\.60\.6B to1414B parameters, and the mean active rank grows slowly with model size \(Figure[4](https://arxiv.org/html/2610.19033#S5.F4)d\)\.Squelchtrains5%5\\%slower than unmerged LoRA and adds5%5\\%to inference latency, less than mixtures with learned routers \(Table[16](https://arxiv.org/html/2610.19033#A8.T16)\)\.

#### Beyond single tasks\.

The same selectivity helps when tasks share one adapter\. Trained jointly on the six domain tasks,Squelchloses only 0\.47 points relative to separate adapters, against 1\.50 for LoRA and 0\.85 for FlyLoRA \(Table[11](https://arxiv.org/html/2610.19033#A8.T11)\), because tokens of different tasks clear the floor on different components\. On GLUE with DeBERTaV3\-base, it raises the average from 88\.41 \(LoRA\) to 89\.71 with2\.92\.9of88components active per token \(Table[10](https://arxiv.org/html/2610.19033#A8.T10)\)\.

## 6Conclusion

How many rank\-one components a token needs is a detection problem: a token’s responses to a LoRA dictionary are a sparse signal on a noise floor\.Squelchanswers it with three classical steps, a robust per\-token floor, a threshold chosen by unbiased risk, and one explaining\-away step toward the joint sparse code\. It needs neither a router nor a rank budget, adds no parameters to LoRA, and improves over strong single\-adapter and mixture baselines across tasks and scales\. The same view applies wherever a model combines many small components per input, including mixtures of full experts and sparse autoencoders\.

## Limitations

The noise model is an idealization: responses of irrelevant components are only approximately Gaussian, and the median estimate is biased upward when many components are active \(Appendix[C](https://arxiv.org/html/2610.19033#A3)\), which SURE compensates only in part\. Explaining away is a single proximal step; it closes part of the gap to the joint code, and our synthetic study shows that it resolves redundancy in the code rather than disentangling the dictionary itself\. Like every input\-dependent adapter,Squelchcannot be merged into the backbone, and it adds a small inference overhead\. Our experiments cover decoder models up to 14B parameters and one encoder, supervised fine\-tuning, and English benchmarks; reinforcement\-learning fine\-tuning, multilingual adaptation, and larger models remain to be tested\.

## Ethics Statement

Squelchis a general fine\-tuning method and inherits the risks of the models and data to which it is applied, including biases in instruction data and the lower cost of specializing models for harmful uses\. All datasets are public and used under their licenses; no human subjects or personal data are involved\. BecauseSquelchleaves out\-of\-domain tokens closer to the frozen model, it reduces but does not remove the risk that fine\-tuning erodes safety behavior; such behavior should be evaluated after adaptation\. AI assistants were used for language editing and for checking LaTeX; every reference was checked against its primary record\.

## References

- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. SuttonProgram synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Barazandehet al\.\(2026\)B\. Barazandeh, S\. Majumdar, V\. Prithyani, and G\. MichailidisLocalized LoRA\-MoE: block\-wise low\-rank experts with adaptive routing\.arXiv preprint arXiv:2607\.05114\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Beck and Teboulle \(2009\)A\. Beck and M\. TeboulleA fast iterative shrinkage\-thresholding algorithm for linear inverse problems\.SIAM Journal on Imaging Sciences2\(1\),pp\. 183–202\.External Links:[Document](https://dx.doi.org/10.1137/080716542)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1),[§B\.4](https://arxiv.org/html/2610.19033#A2.SS4.p1.2),[Appendix C](https://arxiv.org/html/2610.19033#A3.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1)\.
- Bensaidet al\.\(2026\)R\. Bensaid, Y\. Bendou, V\. Gripon, and F\. Leduc\-PrimeauSLAD: shared LoRA adapters for task specific distillation\.arXiv preprint arXiv:2605\.29726\.Cited by:[Appendix J](https://arxiv.org/html/2610.19033#A10.SS0.SSS0.Px4.p1.1)\.
- Bidermanet al\.\(2024\)D\. Biderman, J\. Portes, J\. J\. G\. Ortiz, M\. Paul, P\. Greengard, C\. Jennings, D\. King, S\. Havens, V\. Chiley, J\. Frankle, C\. Blakeney, and J\. P\. CunninghamLoRA learns less and forgets less\.Transactions on Machine Learning Research\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, R\. Le bras, J\. Gao, and Y\. ChoiPIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6239)Cited by:[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Bussmannet al\.\(2024\)B\. Bussmann, P\. Leask, and N\. NandaBatchTopK sparse autoencoders\.arXiv preprint arXiv:2412\.06410\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1)\.
- Büyükakyüz \(2024\)K\. BüyükakyüzOLoRA: orthonormal low\-rank adaptation of large language models\.arXiv preprint arXiv:2406\.01775\.Cited by:[§3\.1](https://arxiv.org/html/2610.19033#S3.SS1.p1.2)\.
- Changet al\.\(2025\)H\. Chang, Z\. Ma, M\. Ma, Z\. Qi, A\. Sabot, H\. Jiang, and H\. T\. KungElaLoRA: elastic & learnable low\-rank adaptation for efficient model fine\-tuning\.arXiv preprint arXiv:2504\.00254\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2026\)L\. Chen, Y\. Meng, X\. Zhan, Z\. Wang, and W\. ZhuWiSparse: boosting LLM inference efficiency with weight\-aware mixed activation sparsity\.arXiv preprint arXiv:2602\.14452\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray,et al\.Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Chenet al\.\(2018\)X\. Chen, J\. Liu, Z\. Wang, and W\. YinTheoretical linear convergence of unfolded ISTA and its practical weights and thresholds\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1),[Appendix C](https://arxiv.org/html/2610.19033#A3.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1)\.
- Chowdhuryet al\.\(2026\)M\. H\. Chowdhury, N\. Mahjabin, A\. S\. Ruhan, M\. A\. Hossain, A\. R\. M\. Kamal, and M\. T\. R\. LaskarRouting is not enough: diagnosing intra\-adapter subspace contention in MoE\+LoRA fine\-tuning\.arXiv preprint arXiv:2609\.03150\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Clarket al\.\(2019\)C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. ToutanovaBoolQ: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),Minneapolis, Minnesota,pp\. 2924–2936\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1300)Cited by:[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Cuiet al\.\(2026\)X\. Cui, H\. Li, R\. Zeng, Y\. Zhao, J\. Qian, W\. Duan, B\. Liu, and Z\. ZhouIGU\-LoRA: adaptive rank allocation via integrated gradients and uncertainty\-aware scoring\.arXiv preprint arXiv:2603\.13792\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1)\.
- Cunninghamet al\.\(2023\)H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. SharkeySparse autoencoders find highly interpretable features in language models\.arXiv preprint arXiv:2309\.08600\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1)\.
- Daiet al\.\(2024\)D\. Dai, C\. Deng, C\. Zhao, R\.X\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu, Z\. Xie, Y\.K\. Li, P\. Huang, F\. Luo, C\. Ruan, Z\. Sui, and W\. LiangDeepSeekMoE: towards ultimate expert specialization in mixture\-of\-experts language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 1280–1297\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.70)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1)\.
- Daubechieset al\.\(2004\)I\. Daubechies, M\. Defrise, and C\. De MolAn iterative thresholding algorithm for linear inverse problems with a sparsity constraint\.Communications on Pure and Applied Mathematics57\(11\),pp\. 1413–1457\.External Links:[Document](https://dx.doi.org/10.1002/cpa.20042)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p3.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1)\.
- Denget al\.\(2026\)G\. Deng, B\. Li, R\. Chen, X\. Liu, Z\. Han, H\. Wang, L\. Wen, and L\. SongDR\-LoRA: dynamic rank LoRA for fine\-tuning Mixture\-of\-Experts models\.InThird Conference on Language Modeling,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1)\.
- Dettmerset al\.\(2023\)T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. ZettlemoyerQLoRA: efficient finetuning of quantized LLMs\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1)\.
- Dinget al\.\(2023\)N\. Ding, X\. Lv, Q\. Wang, Y\. Chen, B\. Zhou, Z\. Liu, and M\. SunSparse low\-rank adaptation of pre\-trained language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 4133–4145\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.252)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1),[Appendix I](https://arxiv.org/html/2610.19033#A9.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px1.p1.1)\.
- Donoho \(1995\)D\.L\. DonohoDe\-noising by soft\-thresholding\.IEEE Transactions on Information Theory41\(3\),pp\. 613–627\.External Links:[Document](https://dx.doi.org/10.1109/18.382009)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p3.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2610.19033#S3.SS2.p1.1)\.
- Donoho and Johnstone \(1994\)D\. L\. Donoho and I\. M\. JohnstoneIdeal spatial adaptation by wavelet shrinkage\.Biometrika81\(3\),pp\. 425–455\.External Links:[Document](https://dx.doi.org/10.1093/biomet/81.3.425)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1),[§B\.2](https://arxiv.org/html/2610.19033#A2.SS2.p3.2),[§1](https://arxiv.org/html/2610.19033#S1.p3.1),[§4](https://arxiv.org/html/2610.19033#S4.p2.1)\.
- Donoho and Johnstone \(1995\)D\. L\. Donoho and I\. M\. JohnstoneAdapting to unknown smoothness via wavelet shrinkage\.Journal of the American Statistical Association90\(432\),pp\. 1200–1224\.External Links:[Document](https://dx.doi.org/10.1080/01621459.1995.10476626)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1),[§B\.3](https://arxiv.org/html/2610.19033#A2.SS3.p2.1),[Appendix F](https://arxiv.org/html/2610.19033#A6.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p3.1),[§3\.3](https://arxiv.org/html/2610.19033#S3.SS3.p1.2)\.
- Douet al\.\(2024\)S\. Dou, E\. Zhou, Y\. Liu, S\. Gao, W\. Shen, L\. Xiong, Y\. Zhou, X\. Wang, Z\. Xi, X\. Fan, S\. Pu, J\. Zhu, R\. Zheng, T\. Gui, Q\. Zhang, and X\. HuangLoRAMoE: alleviating world knowledge forgetting in large language models via MoE\-style plugin\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 1932–1945\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.106)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Doyleet al\.\(2025\)C\. Doyle, A\. Hu, R\. Chan, and A\. LeontjevaLearning adapter rank via symmetry breaking\.arXiv preprint arXiv:2506\.22809\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px1.p1.1)\.
- Faet al\.\(2026\)Z\. Fa, Y\. Duan, J\. Zhang, L\. Qi, W\. Yang, and Y\. ShiDecomposing and composing: towards efficient vision\-language continual learning via rank\-1 expert pool in a single LoRA\.arXiv preprint arXiv:2601\.22828\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px4.p1.1)\.
- Fanet al\.\(2025\)C\. Fan, Z\. Lu, S\. Liu, C\. Gu, X\. Qu, W\. Wei, and Y\. ChengMake LoRA great again: boosting LoRA with adaptive singular values and Mixture\-of\-Experts optimization alignment\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch Transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1)\.
- Fenget al\.\(2025\)J\. Feng, Z\. Pu, T\. Hu, D\. Li, X\. Ai, and H\. WangOMoE: diversifying mixture of low\-rank adaptation by orthogonal finetuning\.InProceedings of the 28th European Conference on Artificial Intelligence,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Gaoet al\.\(2024\)C\. Gao, K\. Chen, J\. Rao, B\. Sun, R\. Liu, D\. Peng, Y\. Zhang, X\. Guo, J\. Yang, and V\. SubrahmanianHigher layers need more LoRA experts\.arXiv preprint arXiv:2402\.08562\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Gaoet al\.\(2025\)L\. Gao, T\. D\. la Tour, H\. Tillman, G\. Goh, R\. Troll, A\. Radford, I\. Sutskever, J\. Leike, and J\. WuScaling and evaluating sparse autoencoders\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1)\.
- Garciaet al\.\(2025\)R\. Garcia, J\. Liu, D\. Sorvisto, and S\. EyubogluAdaptive rank allocation: speeding up modern Transformers with RaNA adapters\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px1.p1.1)\.
- Gevaet al\.\(2021\)M\. Geva, R\. Schuster, J\. Berant, and O\. LevyTransformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,Online and Punta Cana, Dominican Republic,pp\. 5484–5495\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.446)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px4.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark,et al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Gregor and LeCun \(2010\)K\. Gregor and Y\. LeCunLearning fast approximations of sparse coding\.InProceedings of the 27th International Conference on Machine Learning,pp\. 399–406\.Cited by:[Appendix C](https://arxiv.org/html/2610.19033#A3.SS0.SSS0.Px6.p1.1)\.
- Guet al\.\(2025\)N\. Gu, Z\. Zhang, X\. Liu, P\. Fu, Z\. Lin, S\. Wang, Y\. Sun, H\. Wu, W\. Wang, and H\. WangBeamLoRA: beam\-constraint low\-rank adaptation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 11871–11883\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.582)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px1.p1.1)\.
- Guoet al\.\(2025\)Y\. Guo, Z\. Cheng, X\. Tang, Z\. Tu, and T\. LinDynamic mixture of experts: an auto\-tuning approach for efficient Transformer models\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1)\.
- Hampel \(1974\)F\. R\. HampelThe influence curve and its role in robust estimation\.Journal of the American Statistical Association69\(346\),pp\. 383–393\.External Links:[Document](https://dx.doi.org/10.1080/01621459.1974.10482962)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p3.1),[§3\.2](https://arxiv.org/html/2610.19033#S3.SS2.p1.1)\.
- Hanet al\.\(2025\)J\. Han, L\. Du, Y\. Chen, X\. Kang, W\. Ding, and D\. HanFURINA: free from unmergeable router via LINear aggregation of mixed experts\.arXiv preprint arXiv:2509\.14900\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Hayouet al\.\(2024\)S\. Hayou, N\. Ghosh, and B\. YuLoRA\+: efficient low rank adaptation of large models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1),[Appendix I](https://arxiv.org/html/2610.19033#A9.SS0.SSS0.Px7.p1.1)\.
- Heet al\.\(2025\)H\. He, P\. Ye, Y\. Ren, Y\. Yuan, L\. Zhou, S\. Ju, and L\. ChenGoRA: gradient\-driven adaptive low rank adaptation\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px1.p1.1)\.
- Heet al\.\(2021\)P\. He, J\. Gao, and W\. ChenDeBERTaV3: improving DeBERTa using ELECTRA\-Style pre\-training with gradient\-disentangled embedding sharing\.arXiv preprint arXiv:2111\.09543\.Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Hendryckset al\.\(2021a\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InThe Ninth International Conference on Learning Representations,Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Hendryckset al\.\(2021b\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Vol\.1\.Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Houet al\.\(2026\)W\. Hou, X\. Song, and S\. LiuActTail: global activation sparsity in large language models\.arXiv preprint arXiv:2603\.12272\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InThe Tenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§3\.1](https://arxiv.org/html/2610.19033#S3.SS1.p1.1)\.
- Huet al\.\(2023\)Z\. Hu, L\. Wang, Y\. Lan, W\. Xu, E\. Lim, L\. Bing, X\. Xu, S\. Poria, and R\. LeeLLM\-adapters: an adapter family for parameter\-efficient fine\-tuning of large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 5254–5276\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.319)Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Huanget al\.\(2024\)Q\. Huang, Z\. An, N\. Zhuang, M\. Tao, C\. Zhang, Y\. Jin, K\. Xu, K\. Xu, L\. Chen, S\. Huang, and Y\. FengHarder task needs more experts: dynamic routing in MoE models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 12883–12895\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.696)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Jianget al\.\(2024a\)A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, E\. B\. Hanna, F\. Bressand, G\. Lengyel, G\. Bour, G\. Lample, L\. R\. Lavaud, L\. Saulnier, M\. Lachaux, P\. Stock, S\. Subramanian, S\. Yang, S\. Antoniak, T\. Le Scao, T\. Gervet, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. SayedMixtral of experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1)\.
- Jianget al\.\(2025\)T\. Jiang, H\. Wang, and C\. YuanDiffoRA: enabling parameter\-efficient fine\-tuning via differential module selection\.arXiv preprint arXiv:2502\.08905\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1)\.
- Jianget al\.\(2024b\)T\. Jiang, S\. Huang, S\. Luo, Z\. Zhang, H\. Huang, F\. Wei, W\. Deng, F\. Sun, Q\. Zhang, D\. Wang, and F\. ZhuangMoRA: high\-rank updating for parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2405\.12130\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1)\.
- Jinet al\.\(2026\)C\. Jin, H\. Peng, M\. Xiang, Q\. Zhang, X\. Yuan, A\. Hasan, O\. Dibua, Y\. Gong, Y\. Kang, and D\. N\. MetaxasDTop\-p MoE: sparsity\-controlled dynamic Top\-p MoE for foundation model pre\-training\.InProceedings of the 43rd International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Jinet al\.\(2025\)P\. Jin, B\. Zhu, L\. Yuan, and S\. YanMoE\+\+: accelerating Mixture\-of\-Experts methods with zero\-computation experts\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1)\.
- Kalajdzievski \(2023\)D\. KalajdzievskiA rank stabilization scaling factor for fine\-tuning with LoRA\.arXiv preprint arXiv:2312\.03732\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1)\.
- Kopiczkoet al\.\(2024\)D\. J\. Kopiczko, T\. Blankevoort, and Y\. M\. AsanoVeRA: vector\-based random matrix adaptation\.InThe Twelfth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1)\.
- Kunstneret al\.\(2024\)F\. Kunstner, R\. Yadav, A\. Milligan, M\. Schmidt, and A\. BiettiHeavy\-tailed class imbalance and why Adam outperforms gradient descent on language models\.Advances in Neural Information Processing Systems37,pp\. 30106–30148\.Cited by:[Appendix I](https://arxiv.org/html/2610.19033#A9.SS0.SSS0.Px3.p1.1)\.
- Kunwaret al\.\(2025\)P\. Kunwar, M\. N\. Vu, M\. Gupta, M\. Abdelsalam, and M\. BhattaraiTT\-LoRA MoE: unifying parameter\-efficient fine\-tuning and sparse Mixture\-of\-Experts\.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Leeet al\.\(2024\)D\. Lee, J\. Lee, G\. Zhang, M\. Tiwari, and A\. MirhoseiniCATS: contextually\-aware thresholding for sparsity in large language models\.InFirst Conference on Language Modeling,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1)\.
- Liet al\.\(2024\)D\. Li, Y\. Ma, N\. Wang, Z\. Ye, Z\. Cheng, Y\. Tang, Y\. Zhang, L\. Duan, J\. Zuo, C\. Yang, and M\. TangMixLoRA: enhancing large language models fine\-tuning with LoRA\-based mixture of experts\.arXiv preprint arXiv:2404\.15159\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2025a\)D\. Li, N\. Wang, Z\. Zhang, H\. Yin, L\. Duan, M\. Xiao, and M\. TangDynMoLE: boosting mixture of LoRA experts fine\-tuning with a hybrid routing mechanism\.arXiv preprint arXiv:2504\.00661\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Liet al\.\(2025b\)W\. Li, Z\. Song, H\. Zhou, Y\. Zhang, J\. Yu, and W\. YangLoRA\-mixer: coordinate modular LoRA experts through serial attention routing\.arXiv preprint arXiv:2507\.00029\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Liet al\.\(2026\)Z\. Li, Y\. Su, H\. Zhou, Z\. Fu, and N\. CollierFlexi\-LoRA with input\-adaptive ranks: efficient finetuning for speech and reasoning tasks\.In2026 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 5016–5020\.External Links:[Document](https://dx.doi.org/10.1109/icassp55912.2026.11462836)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025a\)J\. Liu, P\. Ponnusamy, T\. Cai, H\. Guo, Y\. Kim, and B\. AthiwaratkunTraining\-free activation sparsity in large language models\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2025b\)K\. Liu, B\. Xu, S\. Wu, X\. Chen, H\. Zhou, Y\. Tao, and L\. HuLa RoSA: enhancing LLM efficiency via layerwise rotated sparse activation\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1)\.
- Liuet al\.\(2026\)M\. Liu, C\. Si, and Y\. JiaFlexLoRA: entropy\-guided flexible low\-rank adaptation\.InThe Fourteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2024a\)Q\. Liu, X\. Wu, X\. Zhao, Y\. Zhu, D\. Xu, F\. Tian, and Y\. ZhengWhen MOE meets LLMs: parameter efficient fine\-tuning for multi\-task medical applications\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 1104–1114\.External Links:[Document](https://dx.doi.org/10.1145/3626772.3657722)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2024b\)S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. ChenDoRA: weight\-decomposed low\-rank adaptation\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1),[Appendix I](https://arxiv.org/html/2610.19033#A9.SS0.SSS0.Px7.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px2.p1.1)\.
- Liu and Luo \(2024\)Z\. Liu and J\. LuoAdaMoLE: fine\-tuning large language models with adaptive mixture of low\-rank adaptation experts\.InFirst Conference on Language Modeling,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1)\.
- Liuet al\.\(2024c\)Z\. Liu, J\. Lyn, W\. Zhu, X\. Tian, and Y\. GrahamALoRA: allocating low\-rank adaptation for fine\-tuning large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 622–641\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.35)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2023\)Z\. Liu, J\. Wang, T\. Dao, T\. Zhou, B\. Yuan, Z\. Song, A\. Shrivastava, C\. Zhang, Y\. Tian, C\. Re, and B\. ChenDeja vu: contextual sparsity for efficient LLMs at inference time\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[Table 8](https://arxiv.org/html/2610.19033#A7.T8.2.1.5.2)\.
- Luet al\.\(2026\)H\. Lu, C\. Zhao, M\. Xue, L\. Yao, K\. Moore, and D\. GongLittle by little: continual learning via incremental mixture of rank\-1 associative memory experts\.InProceedings of the 43rd International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px4.p1.1),[Appendix J](https://arxiv.org/html/2610.19033#A10.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Luet al\.\(2022\)P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. KalyanLearn to explain: multimodal reasoning via thought chains for science question answering\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Menget al\.\(2024\)F\. Meng, Z\. Wang, and M\. ZhangPiSSA: principal singular values and singular vectors adaptation of large language models\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1),[Appendix I](https://arxiv.org/html/2610.19033#A9.SS0.SSS0.Px7.p1.1)\.
- Mihaylovet al\.\(2018\)T\. Mihaylov, P\. Clark, T\. Khot, and A\. SabharwalCan a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Brussels, Belgium,pp\. 2381–2391\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1260)Cited by:[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Mirzadehet al\.\(2024\)I\. Mirzadeh, K\. Alizadeh, S\. Mehta, C\. C\. Del Mundo, O\. Tuzel, G\. Samei, M\. Rastegari, and M\. FarajtabarReLU strikes back: exploiting activation sparsity in large language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1)\.
- Modoranuet al\.\(2026\)I\. Modoranu, M\. Safaryan, and D\. AlistarhMatryoshkaLoRA: learning accurate hierarchical low\-rank representations for LLM fine\-tuning\.arXiv preprint arXiv:2605\.07850\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1)\.
- Mongaet al\.\(2021\)V\. Monga, Y\. Li, and Y\. C\. EldarAlgorithm unrolling: interpretable, efficient deep learning for signal and image processing\.IEEE Signal Processing Magazine38\(2\),pp\. 18–44\.External Links:[Document](https://dx.doi.org/10.1109/MSP.2020.3016905)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1)\.
- Muet al\.\(2026\)L\. Mu, H\. Wang, L\. Ni, L\. Sang, Z\. Wu, P\. Jin, and Y\. ZhangTalkLoRA: communication\-aware mixture of low\-rank adaptation for large language models\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),San Diego, California, United States,pp\. 18436–18451\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.840)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Ninget al\.\(2024\)L\. Ning, H\. Lara, M\. Guo, and A\. RastogiMoDE: effective multi\-task parameter efficient fine\-tuning with a mixture of dyadic experts\.arXiv preprint arXiv:2408\.01505\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Olshausen and Field \(1996\)B\. A\. Olshausen and D\. J\. FieldEmergence of simple\-cell receptive field properties by learning a sparse code for natural images\.Nature381\(6583\),pp\. 607–609\.External Links:[Document](https://dx.doi.org/10.1038/381607a0)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1)\.
- Parket al\.\(2025\)J\. Park, Y\. J\. Ahn, K\. Kim, and J\. KangMonet: mixture of monosemantic experts for Transformers\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1)\.
- Penget al\.\(2024\)R\. Peng, H\. Zou, H\. Wang, Y\. Zeng, Z\. Huang, and J\. ZhaoEnergy\-based automated model evaluation\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 22026–22064\.Cited by:[Appendix I](https://arxiv.org/html/2610.19033#A9.SS0.SSS0.Px2.p1.1)\.
- Qiet al\.\(2024\)X\. Qi, Y\. Zeng, T\. Xie, P\. Chen, R\. Jia, P\. Mittal, and P\. HendersonFine\-tuning aligned language models compromises safety, even when users do not intend to\!\.arXiv preprint arXiv:2310\.03693\.Cited by:[Appendix J](https://arxiv.org/html/2610.19033#A10.SS0.SSS0.Px6.p1.1)\.
- Rajamanoharanet al\.\(2024\)S\. Rajamanoharan, T\. Lieberum, N\. Sonnerat, A\. Conmy, V\. Varma, J\. Kramár, and N\. NandaJumping ahead: improving reconstruction fidelity with JumpReLU sparse autoencoders\.arXiv preprint arXiv:2407\.14435\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1),[Appendix C](https://arxiv.org/html/2610.19033#A3.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1)\.
- Renet al\.\(2024\)P\. Ren, C\. Shi, S\. Wu, M\. Zhang, Z\. Ren, M\. de Rijke, Z\. Chen, and J\. PeiMELoRA: mini\-ensemble low\-rank adapters for parameter\-efficient fine\-tuning\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 3052–3064\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.168)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1)\.
- Rozellet al\.\(2008\)C\. J\. Rozell, D\. H\. Johnson, R\. G\. Baraniuk, and B\. A\. OlshausenSparse coding via thresholding and local competition in neural circuits\.Neural Computation20\(10\),pp\. 2526–2563\.External Links:[Document](https://dx.doi.org/10.1162/neco.2008.03-07-486)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1),[Appendix C](https://arxiv.org/html/2610.19033#A3.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1)\.
- Sakaguchiet al\.\(2020\)K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. ChoiWinoGrande: an adversarial winograd schema challenge at scale\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 8732–8740\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6399)Cited by:[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Saliencroet al\.\(2026\)T\. Saliencro, R\. Desai, P\. Nair, M\. Lindqvist, and D\. WhitmoreSpend experts where you are unsure: confidence\-adaptive routing for Mixture\-of\-Experts LoRA\.arXiv preprint arXiv:2607\.26052\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Sapet al\.\(2019\)M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. ChoiSocial IQa: commonsense reasoning about social interactions\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),Hong Kong, China,pp\. 4463–4473\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1454)Cited by:[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Sathyavageeswaran \(2026\)R\. SathyavageeswaranFIM\-LoRA: task\-informative rank allocation for LoRA via calibration\-time gradient\-variance estimation\.arXiv preprint arXiv:2605\.16800\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1)\.
- Sawant \(2026\)Y\. G\. SawantWhen gradient importance lies: adaptive LoRA rank allocation fails under GRPO\.arXiv preprint arXiv:2605\.07366\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1),[Appendix J](https://arxiv.org/html/2610.19033#A10.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px1.p1.1)\.
- Shazeeret al\.\(2017\)N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. DeanOutrageously large neural networks: the sparsely\-gated Mixture\-of\-Experts layer\.InThe Fifth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1)\.
- Shenget al\.\(2024\)Y\. Sheng, S\. Cao, D\. Li, C\. Hooper, N\. Lee, S\. Yang, C\. Chou, B\. Zhu, L\. Zheng, K\. Keutzer, J\. E\. Gonzalez, and I\. StoicaS\-LoRA: serving thousands of concurrent LoRA adapters\.arXiv preprint arXiv:2311\.03285\.Cited by:[Appendix I](https://arxiv.org/html/2610.19033#A9.SS0.SSS0.Px5.p1.1)\.
- Shiet al\.\(2026\)B\. Shi, W\. Chen, S\. Zhao, J\. Shen, S\. Guo, S\. Wang, and H\. WanSAMoRA: semantic\-aware mixture of LoRA experts for task\-adaptive learning\.InFindings of the Association for Computational Linguistics: ACL 2026,San Diego, California, United States,pp\. 28173–28188\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1404)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Shuttleworthet al\.\(2025\)R\. Shuttleworth, J\. Andreas, A\. Torralba, and P\. SharmaLoRA vs full fine\-tuning: an illusion of equivalence\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1)\.
- Stein \(1981\)C\. M\. SteinEstimation of the mean of a multivariate normal distribution\.The Annals of Statistics9\(6\)\.External Links:[Document](https://dx.doi.org/10.1214/aos/1176345632)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1),[§B\.3](https://arxiv.org/html/2610.19033#A2.SS3.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p3.1),[§3\.3](https://arxiv.org/html/2610.19033#S3.SS3.p1.2)\.
- Sunet al\.\(2026\)M\. Sun, M\. Dou, T\. Feng, D\. Zhang, Y\. Wang, J\. Liu, Y\. Zhu, and J\. TangRotMoLE: enhancing mixture of low\-rank experts through rotational gating mechanism\.arXiv preprint arXiv:2605\.25565\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Sunet al\.\(2024\)M\. Sun, X\. Chen, J\. Z\. Kolter, and Z\. LiuMassive activations in large language models\.arXiv preprint arXiv:2402\.17762\.Cited by:[Appendix J](https://arxiv.org/html/2610.19033#A10.SS0.SSS0.Px2.p1.1)\.
- Tianet al\.\(2024\)C\. Tian, Z\. Shi, Z\. Guo, L\. Li, and C\. XuHydraLoRA: an asymmetric LoRA architecture for efficient fine\-tuning\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Tibshirani \(1996\)R\. TibshiraniRegression shrinkage and selection via the Lasso\.Journal of the Royal Statistical Society: Series B \(Methodological\)58\(1\),pp\. 267–288\.External Links:[Document](https://dx.doi.org/10.1111/j.2517-6161.1996.tb02080.x)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p3.1)\.
- Tropp \(2004\)J\.A\. TroppGreed is good: algorithmic results for sparse approximation\.IEEE Transactions on Information Theory50\(10\),pp\. 2231–2242\.External Links:[Document](https://dx.doi.org/10.1109/TIT.2004.834793)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1)\.
- Valipouret al\.\(2023\)M\. Valipour, M\. Rezagholizadeh, I\. Kobyzev, and A\. GhodsiDyLoRA: parameter\-efficient tuning of pre\-trained models using dynamic search\-free low\-rank adaptation\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,Dubrovnik, Croatia,pp\. 3274–3287\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.eacl-main.239)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2019\)A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. BowmanGLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InThe Seventh International Conference on Learning Representations,Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024a\)L\. Wang, H\. Gao, C\. Zhao, X\. Sun, and D\. DaiAuxiliary\-loss\-free load balancing strategy for Mixture\-of\-Experts\.arXiv preprint arXiv:2408\.15664\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1)\.
- Wanget al\.\(2024b\)S\. Wang, L\. Yu, and J\. LiLoRA\-GA: low\-rank adaptation with gradient approximation\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1),[Appendix I](https://arxiv.org/html/2610.19033#A9.SS0.SSS0.Px7.p1.1)\.
- Wanget al\.\(2025a\)Z\. Wang, J\. Liang, R\. He, Z\. Wang, and T\. TanLoRA\-pro: are low\-rank adapters properly optimized?\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025b\)Z\. Wang, J\. Zhu, and J\. ChenReMoE: fully differentiable Mixture\-of\-Experts with ReLU routing\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1),[Appendix C](https://arxiv.org/html/2610.19033#A3.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2024a\)T\. Wu, J\. Wang, Z\. Zhao, and N\. WongMixture\-of\-subspaces in low\-rank adaptation\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 7880–7899\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.450)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Wuet al\.\(2024b\)X\. Wu, S\. Huang, and F\. WeiMixture of LoRA experts\.InThe Twelfth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2025\)Y\. Wu, J\. Li, Z\. Guo, and L\. LiElastic mixture of rank\-wise experts for knowledge reuse in federated fine\-tuning\.arXiv preprint arXiv:2512\.00902\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px4.p1.1)\.
- Xiaoet al\.\(2026\)X\. Xiao, C\. Ma, Y\. Zhang, C\. Liu, Z\. Wang, Y\. Li, L\. Zhao, G\. Hu, T\. Wang, and H\. XuNot all directions matter: towards structured and task\-aware low\-rank model adaptation\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),San Diego, California, United States,pp\. 2132–2154\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.97)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1)\.
- Xie and Alanwar \(2026\)P\. Xie and A\. AlanwarThe intruder threshold: a spectral law for LoRA fine\-tuning\.arXiv preprint arXiv:2607\.23711\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026\)Q\. Yang, H\. He, M\. Li, J\. Ye, T\. Chen, L\. Bai, and P\. YeMoE2\{\}^\{2\}\-LoRA: when MoE models meet MoE\-style low\-rank adaptation\.arXiv preprint arXiv:2607\.21978\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2024\)Y\. Yang, S\. Qi, W\. Gu, C\. Wang, C\. Gao, and Z\. XuXMoE: sparse models with fine\-grained and adaptive expert selection\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 11664–11674\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.694)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1)\.
- Yuet al\.\(2024\)L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. T\. Kwok, Z\. Li, A\. Weller, and W\. LiuMetaMath: bootstrap your own mathematical questions for large language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Yuet al\.\(2023\)Y\. Yu, S\. Buchanan, D\. Pai, T\. Chu, Z\. Wu, S\. Tong, B\. D\. Haeffele, and Y\. MaWhite\-box Transformers via sparse rate reduction\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px7.p1.1)\.
- Yuanet al\.\(2025\)S\. Yuan, Y\. Zheng, T\. Wang, B\. Liu, and H\. XuMoORE: SVD\-based model MoE\-ization for conflict\- and oblivion\-resistant multi\-task adaptation\.arXiv preprint arXiv:2506\.14436\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px4.p1.1)\.
- Zadouriet al\.\(2024\)T\. Zadouri, A\. Üstün, A\. Ahmadian, B\. Ermiş, A\. Locatelli, and S\. HookerPushing mixture of experts to the limit: extremely parameter efficient MoE for instruction tuning\.InThe Twelfth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp\. 4791–4800\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1472)Cited by:[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Zenget al\.\(2024\)Z\. Zeng, Y\. Miao, H\. Gao, H\. Zhang, and Z\. DengAdaMoE: token\-adaptive routing with null experts for mixture\-of\-experts language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Miami, Florida, USA,pp\. 6223–6235\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.361)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2023a\)F\. Zhang, L\. Li, J\. Chen, Z\. Jiang, B\. Wang, and Y\. QianIncreLoRA: incremental parameter allocation method for parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2308\.12043\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025a\)H\. Zhang, Z\. Li, R\. Bao, Y\. Gao, X\. Xiao, H\. Zhang, S\. Zhang, B\. Huang, Y\. Wu, T\. Wang, and H\. XuHyperAdaLoRA: accelerating LoRA rank allocation during training via hypernetworks without sacrificing performance\.arXiv preprint arXiv:2510\.02630\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025b\)J\. Zhang, J\. You, A\. Panda, and T\. GoldsteinLoRI: reducing cross\-task interference in multi\-task low\-rank adaptation\.InSecond Conference on Language Modeling,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2023b\)L\. Zhang, L\. Zhang, S\. Shi, X\. Chu, and B\. LiLoRA\-FA: efficient and effective low rank representation fine\-tuning\.arXiv preprint arXiv:2308\.03303\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2023c\)Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. ZhaoAdaLoRA: adaptive budget allocation for parameter\-efficient fine\-tuning\.InThe Eleventh International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1),[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026\)R\. Zhang, Y\. Liu, H\. Yang, S\. Zheng, D\. Wang, Y\. Du, L\. Du, and S\. ZhangT\-REX: mixture\-of\-rank\-one\-experts with semantic\-aware intuition for multi\-task large language model finetuning\.InThird Conference on Language Modeling,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2024\)R\. Zhang, R\. Qiang, S\. A\. Somayajula, and P\. XieAutoLoRA: automatically tuning matrix ranks in low\-rank adaptation based on meta learning\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 5048–5060\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.282)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025c\)Z\. Zhang, Z\. Liu, Y\. Tian, H\. Khaitan, Z\. Wang, and S\. LiR\-sparse: rank\-aware activation sparsity for efficient LLM inference\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px6.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px3.p1.1)\.
- Zhaoet al\.\(2026\)Z\. Zhao, Y\. Zhou, Z\. Zhang, D\. Zhu, T\. Shen, Z\. Li, J\. Yang, X\. Wang, J\. Su, K\. Kuang, Z\. Wei, F\. Wu, and Y\. ChengEach rank could be an expert: single\-ranked mixture of experts LoRA for multi\-task learning\.InProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Volume 1,pp\. 1998–2009\.External Links:[Document](https://dx.doi.org/10.1145/3770854.3780222)Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2024\)T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. YueOpenCodeInterpreter: integrating code generation with execution and refinement\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 12834–12859\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.762)Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px1.p1.1),[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2022\)Y\. Zhou, T\. Lei, H\. Liu, N\. Du, Y\. Huang, V\. Zhao, A\. Dai, Z\. Chen, Q\. Le, and J\. LaudonMixture\-of\-Experts with expert choice routing\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px5.p1.1)\.
- Zhuet al\.\(2024\)J\. Zhu, K\. Greenewald, K\. Nadjahi, H\. S\. d\. O\. Borde, R\. B\. Gabrielsson, L\. Choshen, M\. Ghassemi, M\. Yurochkin, and J\. SolomonAsymmetry in low\-rank adapters of foundation models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px1.p1.1),[Appendix J](https://arxiv.org/html/2610.19033#A10.SS0.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2023\)Y\. Zhu, N\. Wichers, C\. Lin, X\. Wang, T\. Chen, L\. Shu, H\. Lu, C\. Liu, L\. Luo, J\. Chen, and L\. MengSiRA: sparse mixture of low rank adaptation\.arXiv preprint arXiv:2311\.09179\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1)\.
- Zhuanget al\.\(2026\)Y\. Zhuang, Y\. Shen, Y\. Bian, Q\. Su, S\. Ji, Y\. Shi, and F\. MiaoLD\-MoLE: learnable dynamic routing for mixture of LoRA experts\.InThe Fourteenth International Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1)\.
- Zouet al\.\(2025\)H\. Zou, Y\. Mao, Y\. Qu, Q\. Wang, and X\. JiUtility\-diversity aware online batch selection for llm supervised fine\-tuning\.arXiv preprint arXiv:2510\.16882\.Cited by:[Appendix G](https://arxiv.org/html/2610.19033#A7.SS0.SSS0.Px2.p1.1)\.
- Zouet al\.\(2026a\)H\. Zou, Y\. Zang, W\. Xu, and X\. JiFly\-cl: a fly\-inspired framework for enhancing efficient decorrelation and reduced training time in pre\-trained model\-based continual representation learning\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 35240–35266\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px4.p1.1)\.
- Zouet al\.\(2026b\)H\. Zou, Y\. Zang, W\. Xu, Y\. Zhu, and X\. JiFlylora: boosting task decoupling and parameter efficiency via implicit rank\-wise mixture\-of\-experts\.Advances in Neural Information Processing Systems38,pp\. 10386–10419\.Cited by:[Appendix A](https://arxiv.org/html/2610.19033#A1.SS0.SSS0.Px4.p1.1),[Appendix J](https://arxiv.org/html/2610.19033#A10.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.19033#S1.p1.1),[§2](https://arxiv.org/html/2610.19033#S2.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2610.19033#S5.SS1.SSS0.Px2.p1.1)\.

## Appendix AExtended Related Work

#### LoRA and its reparameterizations\.

LoRA\([Hu et al\., 2022](https://arxiv.org/html/2610.19033#bib.bib1)\)and its quantized form\([Dettmers et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib2)\)made low\-rank updates the default for adapting large models\. Later work changes the parameterization or the optimization of the update\. DoRA separates magnitude from direction\([Liu et al\., 2024b](https://arxiv.org/html/2610.19033#bib.bib79)\); PiSSA initializes from principal singular directions\([Meng et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib80)\); LoRA\-GA and LoRA\-Pro align the low\-rank gradient with the full one\([Wang et al\., 2024b](https://arxiv.org/html/2610.19033#bib.bib81);[Wang et al\., 2025a](https://arxiv.org/html/2610.19033#bib.bib82)\); rsLoRA and LoRA\+ correct the scale and learning rates of the two factors\([Kalajdzievski, 2023](https://arxiv.org/html/2610.19033#bib.bib83);[Hayou et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib84)\); VeRA shares frozen random factors\([Kopiczko et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib85)\); MELoRA and MoRA raise the rank of the update at a fixed parameter count\([Ren et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib86);[Jiang et al\., 2024b](https://arxiv.org/html/2610.19033#bib.bib87)\)\. Two findings are relevant to our noise\-floor view\. LoRA’s down\-projection changes much less than its up\-projection and can even be frozen at random initialization\([Zhang et al\., 2023b](https://arxiv.org/html/2610.19033#bib.bib88);[Zhu et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib89)\); the responsesA​xAxof most components therefore stay close to random projections of the input, which is the floor thatSquelchestimates\. LoRA also forgets less than full fine\-tuning\([Biderman et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib90)\), but its updates introduce new high\-variance directions that account for much of the forgetting\([Shuttleworth et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib91);[Xie and Alanwar, 2026](https://arxiv.org/html/2610.19033#bib.bib92)\)\.Squelchleaves the tokens on which no component clears the floor on the frozen path\.

#### Rank allocation\.

Static allocation gives each module its own rank\. AdaLoRA prunes singular values by sensitivity\([Zhang et al\., 2023c](https://arxiv.org/html/2610.19033#bib.bib3)\), SoRA gates ranks with a proximalℓ1\\ell\_\{1\}step, i\.e\. soft thresholding of a gate vector that is shared by all inputs\([Ding et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib4)\), and DyLoRA and IncreLoRA train nested or growing ranks\([Valipour et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib93);[Zhang et al\., 2023a](https://arxiv.org/html/2610.19033#bib.bib94)\)\. ALoRA and AutoLoRA learn allocations by ablation scores or meta\-learning\([Liu et al\., 2024c](https://arxiv.org/html/2610.19033#bib.bib95);[Zhang et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib96)\)\. Recent methods use gradient information\([He et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib5)\), beam search over ranks\([Gu et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib34)\), elastic pruning and growth\([Chang et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib35)\), differentiable module selection\([Jiang et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib97)\), hypernetworks\([Zhang et al\., 2025a](https://arxiv.org/html/2610.19033#bib.bib98)\), entropy\([Liu et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib6)\), per\-expert ranks for MoE backbones\([Deng et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib99)\), information\-bottleneck filtering\([Xiao et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib100)\), variational relevance\([Doyle et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib36)\), integrated gradients with uncertainty\([Cui et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib101)\), calibration\-time gradient variance\([Sathyavageeswaran, 2026](https://arxiv.org/html/2610.19033#bib.bib102)\), or nested representations that support several ranks at test time\([Modoranu et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib103)\)\. Gradient\-based importance can fail, for example under reinforcement learning\([Sawant, 2026](https://arxiv.org/html/2610.19033#bib.bib38)\)\. All of these methods fix a module’s rank for every input\. RaNA allocates ranks and neurons per input to speed up inference\([Garcia et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib37)\), and Flexi\-LoRA predicts a rank from the complexity of the input\([Li et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib26)\);Squelchobtains the per\-token rank from the responses themselves and requires neither a predictor nor a budget\.

#### Mixtures of LoRA experts\.

LoRAMoE, MixLoRA, MoLE, and MoLA route tokens or layers to LoRA experts\([Dou et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib40);[Li et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib7);[Wu et al\., 2024b](https://arxiv.org/html/2610.19033#bib.bib41);[Gao et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib104)\); HydraLoRA shares the down\-projection across asymmetric heads\([Tian et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib8)\); MoSLoRA mixes subspaces\([Wu et al\., 2024a](https://arxiv.org/html/2610.19033#bib.bib105)\); and extremely light variants mix vectors or rank\-one factors\([Zadouri et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib39);[Zhu et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib106)\)\. Multi\-task variants target task conflicts\([Liu et al\., 2024a](https://arxiv.org/html/2610.19033#bib.bib107);[Zhang et al\., 2025b](https://arxiv.org/html/2610.19033#bib.bib108);[Shi et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib109)\)\. Routing has been refined with hybrid, learnable, or dynamic rules\([Li et al\., 2025a](https://arxiv.org/html/2610.19033#bib.bib110);[Zhuang et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib9);[Liu and Luo, 2024](https://arxiv.org/html/2610.19033#bib.bib25)\), orthogonality or diversity among experts\([Feng et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib111)\), communication between experts\([Mu et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib43)\), rotational gates\([Sun et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib112)\), singular\-value alignment\([Fan et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib42)\), and expert\-aware adaptation of MoE backbones\([Yang et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib113)\)\. Other designs place experts in the attention projections\([Li et al\., 2025b](https://arxiv.org/html/2610.19033#bib.bib114)\), partition adapters into blocks with adaptive routing\([Barazandeh et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib115)\), compose frozen tensor\-train experts\([Kunwar et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib116)\), or remove the discrete router so that the mixture can be merged\([Han et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib117)\)\. Routing alone does not remove interference inside a shared adapter subspace\([Chowdhury et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib118)\)\.

#### Rank\-one experts\.

Fine\-grained experts are the closest line of work\. T\-REX combines rank\-one experts with semantic priors for the router\([Zhang et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib10)\); MoDE shares the down\-projection and routes over rank\-one “dyadic” experts\([Ning et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib44)\); SMoRA shows that routing among several LoRAs equals rank partitioning within one LoRA and treats every rank as an expert with a learned router and top\-kk\([Zhao et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib11)\); FlyLoRA freezes a sparse random down\-projection whose top\-kkresponses select the active columns of the up\-projection, an implicit router\([Zou et al\., 2026b](https://arxiv.org/html/2610.19033#bib.bib12)\); and MoRAM views rank\-one adapters as key–value memories that self\-activate by their keys\([Lu et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib23)\)\. Rank\-one experts also appear as orthogonal experts obtained from the singular vectors of pretrained weights\([Yuan et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib119)\), as reusable rank\-wise experts in federated fine\-tuning\([Wu et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib120)\), and as an expert pool inside a single LoRA for continual learning\([Fa et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib121);[Zou et al\., 2026a](https://arxiv.org/html/2610.19033#bib.bib13)\)\. Their view of self\-routing through each component’s own key goes back to reading feed\-forward layers as key–value memories\([Geva et al\., 2021](https://arxiv.org/html/2610.19033#bib.bib122)\)\.Squelchkeeps self\-routing and adds what these methods fix by hand: how many components fire \(a noise\-calibrated threshold instead of top\-kk\), how the threshold is set \(an unbiased risk estimate instead of a budget\), and how co\-active components combine \(a joint code instead of a sum\)\.

#### Dynamic routing in sparse MoE\.

Sparse MoE layers route each token to a fixed number of experts\([Shazeer et al\., 2017](https://arxiv.org/html/2610.19033#bib.bib123);[Fedus et al\., 2022](https://arxiv.org/html/2610.19033#bib.bib124);[Jiang et al\., 2024a](https://arxiv.org/html/2610.19033#bib.bib125);[Dai et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib126)\), or let experts choose tokens\([Zhou et al\., 2022](https://arxiv.org/html/2610.19033#bib.bib127)\), with auxiliary or bias\-based load balancing\([Wang et al\., 2024a](https://arxiv.org/html/2610.19033#bib.bib128)\)\. Several methods let the number of experts vary: top\-pprouting\([Huang et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib24)\)and its sparsity\-controlled variant\([Jin et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib47)\), thresholds on router scores\([Yang et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib129)\), null or zero\-computation experts\([Zeng et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib45);[Jin et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib130)\), top\-any gating\([Guo et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib131)\), ReLU routers with an adaptiveℓ1\\ell\_\{1\}penalty\([Wang et al\., 2025b](https://arxiv.org/html/2610.19033#bib.bib46)\), and confidence\-adaptive routing for MoE\-LoRA\([Saliencro et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib48)\)\. These methods learn a router and control sparsity through a target or a penalty\. The squelch has no router and sets sparsity from the responses’ own noise level\.

#### Activation sparsity and sparse autoencoders\.

Contextual and ReLU\-induced sparsity speed up inference\([Liu et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib132);[Mirzadeh et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib133)\)\. CATS and TEAL threshold hidden states at magnitudes chosen for a target sparsity\([Lee et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib54);[Liu et al\., 2025a](https://arxiv.org/html/2610.19033#bib.bib53)\), R\-Sparse and La RoSA combine sparsity with low\-rank or rotated bases\([Zhang et al\., 2025c](https://arxiv.org/html/2610.19033#bib.bib55);[Liu et al\., 2025b](https://arxiv.org/html/2610.19033#bib.bib134)\), and recent methods allocate sparsity across projections by heavy\-tail statistics or weight awareness\([Hou et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib135);[Chen et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib136)\)\. Sparse autoencoders learn overcomplete dictionaries of model activations with ReLU, TopK, JumpReLU, or BatchTopK codes\([Cunningham et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib56);[Gao et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib57);[Rajamanoharan et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib58);[Bussmann et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib137)\), and Monet builds monosemantic experts into the architecture\([Park et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib138)\)\.Squelchtreats the adapter itself as a small dictionary and picks the sparsity of each token’s code by a noise model rather than a target\.

#### Shrinkage, sparse coding, and unrolling\.

Soft thresholding at the universal level is minimax up to a logarithmic factor\([Donoho and Johnstone, 1994](https://arxiv.org/html/2610.19033#bib.bib27);[Donoho, 1995](https://arxiv.org/html/2610.19033#bib.bib28)\)\. Stein’s unbiased risk estimate\([Stein, 1981](https://arxiv.org/html/2610.19033#bib.bib30)\)makes the threshold data\-adaptive\([Donoho and Johnstone, 1995](https://arxiv.org/html/2610.19033#bib.bib31)\), and the median absolute deviation is the standard robust scale estimate\([Hampel, 1974](https://arxiv.org/html/2610.19033#bib.bib29)\)\. For overlapping dictionaries, the LASSO\([Tibshirani, 1996](https://arxiv.org/html/2610.19033#bib.bib32)\)is solved by iterative shrinkage\([Daubechies et al\., 2004](https://arxiv.org/html/2610.19033#bib.bib33);[Beck and Teboulle, 2009](https://arxiv.org/html/2610.19033#bib.bib49)\), whose accuracy depends on the coherence of the atoms\([Tropp, 2004](https://arxiv.org/html/2610.19033#bib.bib139)\)\. Sparse coding with competition between units\([Olshausen and Field, 1996](https://arxiv.org/html/2610.19033#bib.bib140);[Rozell et al\., 2008](https://arxiv.org/html/2610.19033#bib.bib50)\)and unrolled shrinkage networks\([Chen et al\., 2018](https://arxiv.org/html/2610.19033#bib.bib51);[Monga et al\., 2021](https://arxiv.org/html/2610.19033#bib.bib52)\)learn such codes efficiently, and white\-box transformers insert a shrinkage step into each block\([Yu et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib141)\)\. Explaining away inSquelchis one such step, with the inhibition tied to the adapter’s value dictionary rather than learned\.

## Appendix BProofs

Throughout,u=A^​xu=\\hat\{A\}xare the responses,σ^​\(x\)=medi⁡\|ui\|/Φ−1​\(3/4\)\\hat\{\\sigma\}\(x\)=\\operatorname\{med\}\_\{i\}\|u\_\{i\}\|/\\Phi^\{\-1\}\(3/4\),τ=c​σ^​\(x\)\\tau=c\\,\\hat\{\\sigma\}\(x\),z0=𝒮τ​\(u\)z^\{0\}=\\mathcal\{S\}\_\{\\tau\}\(u\),G=B⊤​BG=B^\{\\\!\\top\}B,D=diag⁡\(G\)D=\\mathrm\{diag\}\(G\),M=D−1​\(G−D\)M=D^\{\-1\}\(G\-D\),G^=D−1/2GD−1/2\\hat\{G\}=D^\{\-1/2\}GD^\{\-1/2\},η=1/λmax​\(G^\)\\eta=1/\\lambda\_\{\\max\}\(\\hat\{G\}\), andz=𝒮η​τ​\(z0\+η⁡\(u−\(I\+M\)​z0\)\)z=\\mathcal\{S\}\_\{\\eta\\tau\}\(z^\{0\}\+\\eta\(u\-\(I\+M\)z^\{0\}\)\)\.

### B\.1Proposition[1](https://arxiv.org/html/2610.19033#Thmproposition1)

*\(i\)*Withc=0c=0,τ=0\\tau=0andz0=uz^\{0\}=u\. Without explaining away,z=uz=uand the layer computesW0​x\+s​B​A^​xW\_\{0\}x\+sB\\hat\{A\}x, a LoRA with down\-projectionA^\\hat\{A\}\. Conversely, a LoRA with factors\(A,B\)\(A,B\)equalsSquelchwith keysA^\\hat\{A\}and valuesB​diag​\(‖ai‖\)B\\,\\mathrm\{diag\}\(\\\|a\_\{i\}\\\|\), so the two hypothesis classes coincide\.

*\(ii\)*Forα\>0\\alpha\>0,u⁡\(α​x\)=α​u​\(x\)u\(\\alpha x\)=\\alpha u\(x\), and since the median of absolute values is positively homogeneous,σ^​\(α​x\)=α​σ^​\(x\)\\hat\{\\sigma\}\(\\alpha x\)=\\alpha\\hat\{\\sigma\}\(x\)andτ⁡\(α​x\)=α​τ​\(x\)\\tau\(\\alpha x\)=\\alpha\\tau\(x\)\. Soft thresholding satisfies𝒮α​t​\(α​v\)=α​𝒮t​\(v\)\\mathcal\{S\}\_\{\\alpha t\}\(\\alpha v\)=\\alpha\\mathcal\{S\}\_\{t\}\(v\), soz0​\(α​x\)=α​z0​\(x\)z^\{0\}\(\\alpha x\)=\\alpha z^\{0\}\(x\)\. The explaining\-away pre\-activation is linear in\(u,z0\)\(u,z^\{0\}\)and its threshold isη​τ\\eta\\tau, soz⁡\(α​x\)=α​z​\(x\)z\(\\alpha x\)=\\alpha z\(x\), and the supports ofz⁡\(α​x\)z\(\\alpha x\)andz⁡\(x\)z\(x\)coincide\. The median, absolute values, and soft thresholding are continuous and piecewise linear, and compositions of such maps are again continuous and piecewise linear\.

*\(iii\)*Partition the input space into the finitely many polyhedral cones on which the index of the median, the signs ofuu, the supports ofz0z^\{0\}andzz, and the signs of the active coordinates are constant\. On each cone,z=L​xz=Lxfor a fixed matrixLLwhose rows vanish outsidesupp​z\\mathrm\{supp\}z, sorank⁡\(L\)≤\|supp​z\|\\mathrm\{rank\}\(L\)\\leq\|\\mathrm\{supp\}z\|and the updates​B​L​xsBLxhas rank at most\|supp​z\|\|\\mathrm\{supp\}z\|\. ∎

### B\.2Proposition[2](https://arxiv.org/html/2610.19033#Thmproposition2)

Rescale so thatσ=1\\sigma=1and writeλ=2​log⁡r\\lambda=\\sqrt\{2\\log r\}andrS​\(λ,θ\)=𝔼​\(𝒮λ​\(θ\+ε\)−θ\)2r\_\{S\}\(\\lambda,\\theta\)=\\mathbb\{E\}\(\\mathcal\{S\}\_\{\\lambda\}\(\\theta\+\\varepsilon\)\-\\theta\)^\{2\}for one coordinate\.

*Three facts aboutrSr\_\{S\}\.*First, Stein’s identity \(Proposition[3](https://arxiv.org/html/2610.19033#Thmproposition3)\(a\)\) givesrS\(λ,θ\)=𝔼\[1−2⋅𝟙\{\|y\|≤λ\}\+min\(y2,λ2\)\]≤1\+λ2r\_\{S\}\(\\lambda,\\theta\)=\\mathbb\{E\}\\big\[1\-2\\cdot\\mathbbm\{1\}\\\{\|y\|\\leq\\lambda\\\}\+\\min\(y^\{2\},\\lambda^\{2\}\)\\big\]\\leq 1\+\\lambda^\{2\}withy=θ\+εy=\\theta\+\\varepsilon\. Second,∂rS/∂θ=2​θ​Pr⁡\(\|y\|≤λ\)∈\[0,2​θ\]\\partial r\_\{S\}/\\partial\\theta=2\\theta\\,\\Pr\(\|y\|\\leq\\lambda\)\\in\[0,2\\theta\]forθ≥0\\theta\\geq 0, sorS​\(λ,θ\)≤rS​\(λ,0\)\+θ2r\_\{S\}\(\\lambda,\\theta\)\\leq r\_\{S\}\(\\lambda,0\)\+\\theta^\{2\}; together,rS​\(λ,θ\)≤rS​\(λ,0\)\+min⁡\(θ2,1\+λ2\)r\_\{S\}\(\\lambda,\\theta\)\\leq r\_\{S\}\(\\lambda,0\)\+\\min\(\\theta^\{2\},1\+\\lambda^\{2\}\)\. Third,rS​\(λ,0\)=2​\[\(1\+λ2\)​Φ¯​\(λ\)−λ​ϕ​\(λ\)\]r\_\{S\}\(\\lambda,0\)=2\[\(1\+\\lambda^\{2\}\)\\bar\{\\Phi\}\(\\lambda\)\-\\lambda\\phi\(\\lambda\)\], and the Mills\-ratio boundΦ¯​\(λ\)≤ϕ⁡\(λ\)​\(λ−1−λ−3\+3​λ−5\)\\bar\{\\Phi\}\(\\lambda\)\\leq\\phi\(\\lambda\)\(\\lambda^\{\-1\}\-\\lambda^\{\-3\}\+3\\lambda^\{\-5\}\)yieldsrS​\(λ,0\)≤2​ϕ​\(λ\)​\(2​λ−3\+3​λ−5\)r\_\{S\}\(\\lambda,0\)\\leq 2\\phi\(\\lambda\)\(2\\lambda^\{\-3\}\+3\\lambda^\{\-5\}\)\. Sinceϕ⁡\(λ\)=1/\(r​2​π\)\\phi\(\\lambda\)=1/\(r\\sqrt\{2\\pi\}\), we getr​rS​\(λ,0\)≤1\+λ2r\\,r\_\{S\}\(\\lambda,0\)\\leq 1\+\\lambda^\{2\}for everyr≥2r\\geq 2\.

*\(a\)*Summing over coordinates,

𝔼​‖𝒮λ​\(u\)−θ‖2\\displaystyle\\mathbb\{E\}\\\|\\mathcal\{S\}\_\{\\lambda\}\(u\)\-\\theta\\\|^\{2\}≤r​rS​\(λ,0\)\\displaystyle\\leq r\\,r\_\{S\}\(\\lambda,0\)\+∑imin\(θi2,1\+λ2\)\\displaystyle\\quad\+\\textstyle\\sum\_\{i\}\\min\(\\theta\_\{i\}^\{2\},1\+\\lambda^\{2\}\)≤\(1\+λ2\)​\(1\+∑imin⁡\(θi2,1\)\),\\displaystyle\\leq\(1\+\\lambda^\{2\}\)\\big\(1\+\\textstyle\\sum\_\{i\}\\min\(\\theta\_\{i\}^\{2\},1\)\\big\),usingmin⁡\(θ2,1\+λ2\)≤\(1\+λ2\)​min⁡\(θ2,1\)\\min\(\\theta^\{2\},1\+\\lambda^\{2\}\)\\leq\(1\+\\lambda^\{2\}\)\\min\(\\theta^\{2\},1\)\. The oracle that keepsuiu\_\{i\}iff\|θi\|\>1\|\\theta\_\{i\}\|\>1has risk∑imin⁡\(θi2,1\)\\sum\_\{i\}\\min\(\\theta\_\{i\}^\{2\},1\), which gives the claim\([Donoho and Johnstone, 1994](https://arxiv.org/html/2610.19033#bib.bib27)\)\.

*\(b\)*By the Mills ratio,Pr⁡\(\|εi\|\>λ\)=2​Φ¯​\(λ\)≤2​ϕ​\(λ\)/λ=1/\(r​π​log⁡r\)\\Pr\(\|\\varepsilon\_\{i\}\|\>\\lambda\)=2\\bar\{\\Phi\}\(\\lambda\)\\leq 2\\phi\(\\lambda\)/\\lambda=1/\(r\\sqrt\{\\pi\\log r\}\)\. A union bound over the at mostrrcomponents withθi=0\\theta\_\{i\}=0gives the first statement\. If\|θi\|≥2​λ\|\\theta\_\{i\}\|\\geq 2\\lambda, the component is inactive only if\|εi\|≥\|θi\|−λ≥λ\|\\varepsilon\_\{i\}\|\\geq\|\\theta\_\{i\}\|\-\\lambda\\geq\\lambda, which has the same probability bound\. ∎

### B\.3Proposition[3](https://arxiv.org/html/2610.19033#Thmproposition3)

*\(a\)*Write𝒮c​\(v\)=v\+g⁡\(v\)\\mathcal\{S\}\_\{c\}\(v\)=v\+g\(v\)withgi​\(v\)=−sign⁡\(vi\)​min⁡\(\|vi\|,c\)g\_\{i\}\(v\)=\-\\mathrm\{sign\}\(v\_\{i\}\)\\min\(\|v\_\{i\}\|,c\)\. The mapggis Lipschitz, hence weakly differentiable, with∂gi/∂vi=−𝟙\{\|vi\|≤c\}\\partial g\_\{i\}/\\partial v\_\{i\}=\-\\mathbbm\{1\}\\\{\|v\_\{i\}\|\\leq c\\\}almost everywhere\. Stein’s lemma\([Stein, 1981](https://arxiv.org/html/2610.19033#bib.bib30)\)gives𝔼∥v\+g\(v\)−θ∥2=r\+𝔼\[∥g\(v\)∥2\+2∇⋅g\(v\)\]\\mathbb\{E\}\\\|v\+g\(v\)\-\\theta\\\|^\{2\}=r\+\\mathbb\{E\}\\big\[\\\|g\(v\)\\\|^\{2\}\+2\\nabla\\\!\\cdot g\(v\)\\big\], and substitutingggyields Eq\. \([4](https://arxiv.org/html/2610.19033#S3.E4)\)\. Summing overnnproves the claim\.

*\(b\)*Assume that the pairs\(θn,vn\)\(\\theta\_\{n\},v\_\{n\}\)are drawn i\.i\.d\., as for tokens sampled from a data distribution, with𝔼​‖θn‖2<∞\\mathbb\{E\}\\\|\\theta\_\{n\}\\\|^\{2\}<\\infty\. For each coordinate,hc=1−21\{\|v\|≤c\}\+min\(v2,c2\)−\(𝒮c\(v\)−θ\)2h\_\{c\}=1\-2\\,\\mathbbm\{1\}\\\{\|v\|\\leq c\\\}\+\\min\(v^\{2\},c^\{2\}\)\-\(\\mathcal\{S\}\_\{c\}\(v\)\-\\theta\)^\{2\}has mean zero by \(a\)\. The family\{hc:c∈\[0,cU\]\}\\\{h\_\{c\}:c\\in\[0,c\_\{\\mathrm\{U\}\}\]\\\}consists of indicators and min\-functions that are monotone incc, and of the Lipschitz\-in\-ccfamily\(𝒮c​\(v\)−θ\)2\(\\mathcal\{S\}\_\{c\}\(v\)\-\\theta\)^\{2\}\. It is a VC\-type class with the integrable envelope3\+3​cU2\+2​\(v−θ\)23\+3c\_\{\\mathrm\{U\}\}^\{2\}\+2\(v\-\\theta\)^\{2\}, so the uniform law of large numbers yieldssupc\|1N​r​∑n,ihc\|→0\\sup\_\{c\}\|\\frac\{1\}\{Nr\}\\sum\_\{n,i\}h\_\{c\}\|\\to 0in probability\. Ifc^\\hat\{c\}minimizes SURE andc⋆c^\{\\star\}minimizes the loss, the loss ofc^\\hat\{c\}exceeds that ofc⋆c^\{\\star\}by at most twice this supremum timesN​rNr\([Donoho and Johnstone, 1995](https://arxiv.org/html/2610.19033#bib.bib31)\)\. ∎

### B\.4Proposition[4](https://arxiv.org/html/2610.19033#Thmproposition4)

*\(a\)*Substitutew=D1/2​zw=D^\{1/2\}z\. ThenF=f\+gF=f\+gwithf⁡\(w\)=12​w⊤​G^​w−\(D1/2​u\)⊤​wf\(w\)=\\frac\{1\}\{2\}w^\{\\\!\\top\}\\hat\{G\}w\-\(D^\{1/2\}u\)^\{\\\!\\top\}w, whose gradient isλmax​\(G^\)\\lambda\_\{\\max\}\(\\hat\{G\}\)\-Lipschitz, andg⁡\(w\)=τ​∑iDi​i1/2​\|wi\|g\(w\)=\\tau\\sum\_\{i\}D\_\{ii\}^\{1/2\}\|w\_\{i\}\|, whose proximal map is coordinatewise soft thresholding atη​τ​Di​i1/2\\eta\\tau D\_\{ii\}^\{1/2\}\. One proximal\-gradient step fromw0=D1/2​z0w^\{0\}=D^\{1/2\}z^\{0\}is

w\+=𝒮η​τ​D1/2​\(w0−η⁡\(G^​w0−D1/2​u\)\)\.w^\{\+\}=\\mathcal\{S\}\_\{\\eta\\tau D^\{1/2\}\}\\big\(w^\{0\}\-\\eta\(\\hat\{G\}w^\{0\}\-D^\{1/2\}u\)\\big\)\.Mapping back withz=D−1/2wz=D^\{\-1/2\}wand usingD−1/2𝒮η​τ​Di​i1/2\(D1/2⋅\)=𝒮η​τ\(⋅\)D^\{\-1/2\}\\mathcal\{S\}\_\{\\eta\\tau D\_\{ii\}^\{1/2\}\}\(D^\{1/2\}\\cdot\)=\\mathcal\{S\}\_\{\\eta\\tau\}\(\\cdot\)coordinatewise andD−1​G=I\+MD^\{\-1\}G=I\+Mgives Eq\. \([6](https://arxiv.org/html/2610.19033#S3.E6)\)\. Forη≤1/λmax​\(G^\)\\eta\\leq 1/\\lambda\_\{\\max\}\(\\hat\{G\}\), the sufficient\-decrease inequality of proximal gradient descent,F⁡\(w\+\)≤F⁡\(w0\)−12​η​‖w\+−w0‖2F\(w^\{\+\}\)\\leq F\(w^\{0\}\)\-\\frac\{1\}\{2\\eta\}\\\|w^\{\+\}\-w^\{0\}\\\|^\{2\}\([Beck and Teboulle, 2009](https://arxiv.org/html/2610.19033#bib.bib49)\), givesF⁡\(z\)≤F⁡\(z0\)F\(z\)\\leq F\(z^\{0\}\), with equality iffw\+=w0w^\{\+\}=w^\{0\}, i\.e\. iffz0z^\{0\}is a fixed point of the proximal\-gradient map\. SinceFFis convex, fixed points are exactly its minimizers\.

*\(b\)*Letb1=b2=bb\_\{1\}=b\_\{2\}=bandu1=u2=u\>τu\_\{1\}=u\_\{2\}=u\>\\tau\. ThenG^=\(1111\)\\hat\{G\}=\\begin\{pmatrix\}1&1\\\\ 1&1\\end\{pmatrix\},λmax=2\\lambda\_\{\\max\}=2,η=12\\eta=\\frac\{1\}\{2\},M=\(0110\)M=\\begin\{pmatrix\}0&1\\\\ 1&0\\end\{pmatrix\}, andz0=\(u−τ\)​𝟏z^\{0\}=\(u\-\\tau\)\\mathbf\{1\}, whose output is2​\(u−τ\)​b2\(u\-\\tau\)b\. The pre\-activation isz0\+12​\(u​𝟏−2​z0\)=u2​𝟏z^\{0\}\+\\frac\{1\}\{2\}\(u\\mathbf\{1\}\-2z^\{0\}\)=\\frac\{u\}\{2\}\\mathbf\{1\}, and thresholding atτ2\\frac\{\\tau\}\{2\}givesz=u−τ2​𝟏z=\\frac\{u\-\\tau\}\{2\}\\mathbf\{1\}, whose output is\(u−τ\)​b\(u\-\\tau\)b\. The objective depends onzzonly throught=z1\+z2t=z\_\{1\}\+z\_\{2\}\(for equal signs\) as‖b‖2​\(12​t2−u​t\+τ​\|t\|\)\\\|b\\\|^\{2\}\(\\frac\{1\}\{2\}t^\{2\}\-ut\+\\tau\|t\|\), which is minimized att=u−τt=u\-\\tau\. The explaining\-away output is therefore the LASSO output\.

*\(c\)*The coordinates ofzzarezi=𝒮η​τ​\(pi\)z\_\{i\}=\\mathcal\{S\}\_\{\\eta\\tau\}\(p\_\{i\}\)withpi=zi0\+η⁡\(ui−zi0−∑j≠iMi​j​zj0\)p\_\{i\}=z^\{0\}\_\{i\}\+\\eta\(u\_\{i\}\-z^\{0\}\_\{i\}\-\\sum\_\{j\\neq i\}M\_\{ij\}z^\{0\}\_\{j\}\)andMi​j=Gi​j/Gi​iM\_\{ij\}=G\_\{ij\}/G\_\{ii\}\. Treatingη\\etaas a constant \(it is computed without gradient\),Gi​j=Gj​iG\_\{ij\}=G\_\{ji\}fori≠ji\\neq jenters onlypip\_\{i\}andpjp\_\{j\}:

−1η​∂ℒ∂Gi​j\\displaystyle\-\\frac\{1\}\{\\eta\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial G\_\{ij\}\}=zj0Gi​i1\{zi≠0\}∂ℒ∂zi\\displaystyle=\\frac\{z^\{0\}\_\{j\}\}\{G\_\{ii\}\}\\,\\mathbbm\{1\}\\\{z\_\{i\}\\neq 0\\\}\\,\\frac\{\\partial\\mathcal\{L\}\}\{\\partial z\_\{i\}\}\+zi0Gj​j1\{zj≠0\}∂ℒ∂zj\.\\displaystyle\\quad\+\\frac\{z^\{0\}\_\{i\}\}\{G\_\{jj\}\}\\,\\mathbbm\{1\}\\\{z\_\{j\}\\neq 0\\\}\\,\\frac\{\\partial\\mathcal\{L\}\}\{\\partial z\_\{j\}\}\.Each term vanishes unless one component is active inz0z^\{0\}and the other is active inzz\.

*\(d\)*LetD=ID=I, so thatGGhas unit diagonal,M=G−IM=G\-I, andG^=G\\hat\{G\}=G\. LetSSands=sign⁡\(zS⋆\)s=\\mathrm\{sign\}\(z^\{\\star\}\_\{S\}\)be the common support and signs\. The optimality conditions of \([5](https://arxiv.org/html/2610.19033#S3.E5)\) onSSreadGS​S​zS⋆=uS−τ​sG\_\{SS\}z^\{\\star\}\_\{S\}=u\_\{S\}\-\\tau s\. The first pass satisfieszS0=uS−τ​s=GS​S​zS⋆z^\{0\}\_\{S\}=u\_\{S\}\-\\tau s=G\_\{SS\}z^\{\\star\}\_\{S\}, so

zS0−zS⋆=\(GS​S−I\)​zS⋆,z^\{0\}\_\{S\}\-z^\{\\star\}\_\{S\}=\(G\_\{SS\}\-I\)\\,z^\{\\star\}\_\{S\},and‖GS​S−I‖2≤\(\|S\|−1\)​μS\\\|G\_\{SS\}\-I\\\|\_\{2\}\\leq\(\|S\|\-1\)\\mu\_\{S\}by Gershgorin’s theorem, sinceGS​S−IG\_\{SS\}\-Iis symmetric with zero diagonal and off\-diagonal entries bounded byμS\\mu\_\{S\}\. With the same support and signs, the explaining\-away step onSSiszS=zS0\+η⁡\(uS−GS​S​zS0\)−η​τ​s=zS0\+η​GS​S​\(zS⋆−zS0\)z\_\{S\}=z^\{0\}\_\{S\}\+\\eta\(u\_\{S\}\-G\_\{SS\}z^\{0\}\_\{S\}\)\-\\eta\\tau s=z^\{0\}\_\{S\}\+\\eta G\_\{SS\}\(z^\{\\star\}\_\{S\}\-z^\{0\}\_\{S\}\), hencezS−zS⋆=\(I−η​GS​S\)​\(zS0−zS⋆\)z\_\{S\}\-z^\{\\star\}\_\{S\}=\(I\-\\eta G\_\{SS\}\)\(z^\{0\}\_\{S\}\-z^\{\\star\}\_\{S\}\)\. Sinceη≤1/λmax​\(G\)≤1/λmax​\(GS​S\)\\eta\\leq 1/\\lambda\_\{\\max\}\(G\)\\leq 1/\\lambda\_\{\\max\}\(G\_\{SS\}\), the contraction factor‖I−η​GS​S‖2=1−η​λmin​\(GS​S\)\\\|I\-\\eta G\_\{SS\}\\\|\_\{2\}=1\-\\eta\\lambda\_\{\\min\}\(G\_\{SS\}\)is below one wheneverGS​S≻0G\_\{SS\}\\succ 0\. ∎

## Appendix CFurther Analysis

#### Bias of the median scale\.

If a fractionρ<12\\rho<\\frac\{1\}\{2\}of the components carries signal larger than every noise response, the median of\|u\|\|u\|is an order statistic of the noise alone, at levelq=12​\(1−ρ\)q=\\frac\{1\}\{2\(1\-\\rho\)\}of ther⁡\(1−ρ\)r\(1\-\\rho\)noise magnitudes\. Asr→∞r\\to\\infty,

σ^σ→Φ−1​\(12\+14​\(1−ρ\)\)Φ−1​\(34\),\\frac\{\\hat\{\\sigma\}\}\{\\sigma\}\\to\\frac\{\\Phi^\{\-1\}\\big\(\\tfrac\{1\}\{2\}\+\\tfrac\{1\}\{4\(1\-\\rho\)\}\\big\)\}\{\\Phi^\{\-1\}\(\\tfrac\{3\}\{4\}\)\},which equals11,1\.171\.17,1\.431\.43, and1\.901\.90forρ=0,18,14,38\\rho=0,\\frac\{1\}\{8\},\\frac\{1\}\{4\},\\frac\{3\}\{8\}\(Table[4](https://arxiv.org/html/2610.19033#A4.T4)\)\. The estimate therefore errs toward caution: the more components a token activates, the higher its threshold, which makes the squelch self\-limiting\. SURE is then computed on responses whose noise variance is\(σ/σ^\)2<1\(\\sigma/\\hat\{\\sigma\}\)^\{2\}<1, which biases its minimizer toward smaller multiples and partly compensates\. Re\-estimatingσ\\sigmafrom the sub\-threshold responses \(“iterated MAD”\) removes most of the bias but changes accuracy by only\+0\.02\+0\.02\(Table[15](https://arxiv.org/html/2610.19033#A8.T15)\), so we keep the one\-shot median\.

#### How the active rank depends on the signal\.

The expected per\-token rank has a closed form under the model of Proposition[2](https://arxiv.org/html/2610.19033#Thmproposition2)\.

###### Proposition 5\(Expected active rank\)\.

Letu=θ\+σ​εu=\\theta\+\\sigma\\varepsilonwithε∼𝒩⁡\(0,Ir\)\\varepsilon\\sim\\mathcal\{N\}\(0,I\_\{r\}\), lets=\|\{i:θi≠0\}\|s=\|\\\{i:\\theta\_\{i\}\\neq 0\\\}\|,ζi=\|θi\|/σ\\zeta\_\{i\}=\|\\theta\_\{i\}\|/\\sigma, andτ=c​σ\\tau=c\\sigma\. Then

𝔼​\|supp​z0\|\\displaystyle\\mathbb\{E\}\|\\mathrm\{supp\}z^\{0\}\|=∑i:θi≠0\[Φ¯\(c−ζi\)\+Φ¯\(c\+ζi\)\]\\displaystyle=\\textstyle\\sum\_\{i:\\theta\_\{i\}\\neq 0\}\\big\[\\bar\{\\Phi\}\(c\-\\zeta\_\{i\}\)\+\\bar\{\\Phi\}\(c\+\\zeta\_\{i\}\)\\big\]\+2​\(r−s\)​Φ¯​\(c\)\.\\displaystyle\\quad\+2\(r\-s\)\\,\\bar\{\\Phi\}\(c\)\.It increases with every\|θi\|\|\\theta\_\{i\}\|and withssand decreases withcc\. Atc=2​log⁡rc=\\sqrt\{2\\log r\}, the floor contributes at most1/π​log⁡r1/\\sqrt\{\\pi\\log r\}active components in expectation\.

*Proof\.*Componentiiis active iff\|θi\+σ​εi\|\>c​σ\|\\theta\_\{i\}\+\\sigma\\varepsilon\_\{i\}\|\>c\\sigma, which has probabilityΦ¯​\(c−ζi\)\+Φ¯​\(c\+ζi\)\\bar\{\\Phi\}\(c\-\\zeta\_\{i\}\)\+\\bar\{\\Phi\}\(c\+\\zeta\_\{i\}\); linearity of expectation gives the formula\. The derivative of this probability with respect toζi\\zeta\_\{i\}isϕ⁡\(c−ζi\)−ϕ⁡\(c\+ζi\)≥0\\phi\(c\-\\zeta\_\{i\}\)\-\\phi\(c\+\\zeta\_\{i\}\)\\geq 0forc,ζi≥0c,\\zeta\_\{i\}\\geq 0\. Each term decreases incc\. The floor term is bounded by2​r​Φ¯​\(2​log⁡r\)≤1/π​log⁡r2r\\bar\{\\Phi\}\(\\sqrt\{2\\log r\}\)\\leq 1/\\sqrt\{\\pi\\log r\}by the Mills ratio, as in Proposition[2](https://arxiv.org/html/2610.19033#Thmproposition2)\(b\)\. ∎

The per\-token rank is therefore the number of components whose signal clears the floor, smoothed by the noise\. Tokens with more or stronger relevant components receive more rank, and the multiplecℓc\_\{\\ell\}trades false activations against missed signal\. This trade\-off is what SURE resolves\.

#### Why a threshold learned by the task loss collapses\.

For active components,∂zi/∂τ=−sign⁡\(ui\)\\partial z\_\{i\}/\\partial\\tau=\-\\mathrm\{sign\}\(u\_\{i\}\), so loweringτ\\tauenlarges every active coefficient and activates new components; both enlarge the set of functions the adapter can fit\. Gradient descent onτ\\tautherefore tends to decrease it until the code is nearly dense, as the ablation shows \(k¯=27\.9\\bar\{k\}=27\.9\)\. ReLU routers face the same pressure and need anℓ1\\ell\_\{1\}penalty with an adaptive coefficient to keep their sparsity\([Wang et al\., 2025b](https://arxiv.org/html/2610.19033#bib.bib46)\)\. SURE avoids this because it measures the error with respect to the*signal*in the responses, not the fit to the training loss\.

#### Soft versus hard thresholds\.

Hard thresholding keepsuiu\_\{i\}unchanged aboveτ\\tauand makes the layer discontinuous at the threshold, so a small change of the input can switch a component with its full magnitude\. Soft thresholding is continuous and 1\-Lipschitz inuu, which keeps training stable; JumpReLU autoencoders need straight\-through estimators to train hard thresholds\([Rajamanoharan et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib58)\)\. Hard thresholds lose0\.190\.19points \(Table[3](https://arxiv.org/html/2610.19033#S5.T3)\)\.

#### Iterating explaining away\.

KKsteps of Eq\. \([6](https://arxiv.org/html/2610.19033#S3.E6)\) are ISTA in the metricDD, soF⁡\(zK\)−F⋆≤λmax​\(G^\)​‖z0−z⋆‖D2/\(2​K\)F\(z^\{K\}\)\-F^\{\\star\}\\leq\\lambda\_\{\\max\}\(\\hat\{G\}\)\\,\\\|z^\{0\}\-z^\{\\star\}\\\|\_\{D\}^\{2\}/\(2K\)\([Beck and Teboulle, 2009](https://arxiv.org/html/2610.19033#bib.bib49)\)\. A second step adds0\.030\.03points and a third none \(Table[15](https://arxiv.org/html/2610.19033#A8.T15)\), so one step suffices in practice\.

#### Relation to competitive sparse coding\.

Explaining away is one step of the locally competitive algorithm\([Rozell et al\., 2008](https://arxiv.org/html/2610.19033#bib.bib50)\)\. The feed\-forward drive is the responseuu, the lateral inhibition is the normalized value GramMM, and the step starts from the thresholded drive rather than from zero\. Unlike LISTA\([Gregor and LeCun, 2010](https://arxiv.org/html/2610.19033#bib.bib142);[Chen et al\., 2018](https://arxiv.org/html/2610.19033#bib.bib51)\),Squelchdoes not learn the inhibition: it is tied to the synthesis dictionaryBB, soSquelchkeeps LoRA’s parameter count and its inhibition stays consistent with the update it synthesizes\.

#### Complexity\.

For a module withddinputs anddoutd\_\{\\mathrm\{out\}\}outputs, LoRA costs2​r​\(d\+dout\)2r\(d\+d\_\{\\mathrm\{out\}\}\)FLOPs per token\.SquelchaddsO⁡\(r\)O\(r\)for the median \(by selection\),2​r22r^\{2\}for the explaining\-away product, andO⁡\(r\)O\(r\)for thresholding, and it saves2​\(r−\|supp​z\|\)​dout2\(r\-\|\\mathrm\{supp\}z\|\)d\_\{\\mathrm\{out\}\}in the up\-projection\. Forr=32r=32andd=4096d=4096, the additions amount to less than1%1\\%of LoRA’s FLOPs\. The GramB⊤​BB^\{\\\!\\top\}Band its top eigenvalue \(ten power iterations\) are computed once per module and step\.

## Appendix DNumerical Verification

Table[4](https://arxiv.org/html/2610.19033#A4.T4)reports exact or Monte Carlo computations \(not experiments\) that check each analytic statement; the script iscode/theory\_checks\.py\.

Table 4:Numerical checks of the analytic results \(scriptcode/theory\_checks\.py\)\. The MAD row explains the conservative bias discussed in Appendix[C](https://arxiv.org/html/2610.19033#A3)\.
## Appendix EControlled Synthetic Study

#### Setup\.

Inputsx∈ℝ64x\\in\\mathbb\{R\}^\{64\}come fromK=24K=24concept clusters,x=μc\+2​ξx=\\mu\_\{c\}\+2\\xiwithμc,ξ∼𝒩⁡\(0,I\)\\mu\_\{c\},\\xi\\sim\\mathcal\{N\}\(0,I\)\. A teacher with3232rank\-one atoms assigns each cluster a supportScS\_\{c\}of size\|Sc\|∈\{1,2,3,4,6,8\}\|S\_\{c\}\|\\in\\\{1,2,3,4,6,8\\\}and producesy=12​∑i∈Scbi⋆​⟨ki⋆,x⟩y=\\frac\{1\}\{2\}\\sum\_\{i\\in S\_\{c\}\}b^\{\\star\}\_\{i\}\\langle k^\{\\star\}\_\{i\},x\\rangle\. The keyki⋆k^\{\\star\}\_\{i\}is the normalized mean of the cluster centers that use atomii, plus noise, and the valuesbi⋆b^\{\\star\}\_\{i\}are random unit vectors, optionally sharing a common direction so that their pairwise coherence is about0\.50\.5\. The true rank of an input is thus\|Sc\|\|S\_\{c\}\|\. Students haver=32r=32components and are trained by squared error with Adam \(learning rate3×10−33\\times 10^\{\-3\}, cosine schedule,3,0003\{,\}000steps, batch512512, three seeds\)\. The squelch uses the same rules as in the main experiments \(per\-input MAD floor, SURE every 20 steps,5%5\\%warm\-up\)\. This study is a real computation; the script iscode/synthetic\_study\.py\.

Orthogonal values\(μ¯≈0\\bar\{\\mu\}\\approx 0\)Coherent values\(μ¯≈0\.5\\bar\{\\mu\}\\approx 0\.5\)StudentRel\. error𝒌¯\\bar\{\\bm\{k\}\}SpearmanRel\. error𝒌¯\\bar\{\\bm\{k\}\}SpearmanDense LoRA \(r=32r\{=\}32\)0\.552±0\.0160\.552\\pm 0\.01632\.0—0\.479±0\.0320\.479\\pm 0\.03232\.0—Top\-44self\-routing0\.355±0\.0130\.355\\pm 0\.0134\.0—0\.243±0\.0090\.243\\pm 0\.0094\.0—Top\-88self\-routing0\.451±0\.0080\.451\\pm 0\.0088\.0—0\.304±0\.0110\.304\\pm 0\.0118\.0—Squelchwithout EA0\.308±0\.0090\.308\\pm 0\.0093\.10\.370\.293±0\.0350\.293\\pm 0\.0354\.00\.60Squelch0\.326±0\.0660\.326\\pm 0\.0667\.80\.660\.240±0\.0030\.240\\pm 0\.00310\.50\.84Squelchw/o EA\+\+coherence penalty0\.340±0\.0360\.340\\pm 0\.0362\.60\.460\.308±0\.0330\.308\\pm 0\.0334\.30\.64Squelch\+\+coherence penalty0\.322±0\.0190\.322\\pm 0\.0194\.80\.700\.243±0\.0150\.243\\pm 0\.01511\.00\.82Table 5:Controlled synthetic study\(real computation, mean±\\pms\.d\. over three seeds\)\. Rel\. error:‖y^−y‖2/‖y‖2\\\|\\hat\{y\}\-y\\\|^\{2\}/\\\|y\\\|^\{2\}on held\-out inputs;k¯\\bar\{k\}: mean active components; Spearman: rank correlation between the mean active count of each cluster and its true rank\|Sc\|\|S\_\{c\}\|\(undefined for fixed\-kkstudents\)\.Figure 5:Synthetic study\(real computation\)\. \(a\) Relative test error of each student for orthogonal \(light\) and coherent \(dark\) teacher values\. \(b\) Mean active components as a function of the true rank of the input \(solid: coherent values; dashed: orthogonal\)\. \(c\) Mean\|cos\|\|\\cos\|between the values of1616near\-duplicate component pairs during training\. \(d\) Error for smaller student dictionaries\.
#### Findings\.

Table[5](https://arxiv.org/html/2610.19033#A5.T5)and Figure[5](https://arxiv.org/html/2610.19033#A5.F5)support three conclusions and qualify a fourth\. \(1\)*Input\-adaptive rank matters\.*A dense student cannot gate and has the largest error\. Fixed top\-kkstudents are better but cannot adapt to the true rank, and a largerkkhurts because it adds floor components to low\-rank inputs\. \(2\)*The squelch alone is conservative\.*Without explaining away, the SURE threshold keeps about three components per input regardless of the true rank \(Spearman0\.370\.37and0\.600\.60\)\. \(3\)*Explaining away makes the active count track the true rank*\(Spearman0\.660\.66and0\.840\.84\) and gives the lowest error when the teacher’s values are coherent \(0\.2400\.240vs\.0\.2930\.293\)\. With orthogonal values the two variants are on par\. The mean error of the full method is higher here only because one seed converged to a poor solution \(0\.4200\.420\); its median error is lower \(0\.2800\.280vs\.0\.3140\.314\)\. \(4\)*Explaining away does not disentangle the dictionary\.*Starting from1616near\-duplicate pairs, the pairs drift apart without explaining away \(mean\|cos\|\|\\cos\|0\.17 after training\) but stay closer with it \(0\.39\), and the effective rank of co\-active values is lower \(participation ratio over active count 0\.33 vs\. 0\.87\)\. Explaining away makes redundancy harmless in the code, since redundant components share rather than duplicate their contribution, so training has less reason to separate them\. A penalty on co\-activation\-weighted coherence restores separation but reduces error only when the dictionary is scarce \(Table[6](https://arxiv.org/html/2610.19033#A5.T6)\)\. We therefore describe explaining away as joint credit assignment rather than as a disentangling regularizer\.

Table 6:Synthetic study with scarce student dictionaries \(orthogonal teacher values, three seeds\)\. “pen\.”: co\-activation\-weighted coherence penalty with weight11\.

## Appendix FImplementation Details

#### Layer\.

Squelchreplaces the forward pass of a LoRA module and keeps its parameters and initialization \(AAKaiming\-uniform,B=0B=0\)\. Responses are computed in bfloat16, and the median, thresholds, and explaining\-away step in float32\. WithB=0B=0at initialization,G=0G=0andMMis defined as00, so explaining away switches on gradually as the values grow\. The step size uses ten power iterations on the cosine Gram with a floor ofλmax≥1\\lambda\_\{\\max\}\\geq 1\. Gradients flow throughuu,z0z^\{0\},zz, andGG;σ^\\hat\{\\sigma\},cℓc\_\{\\ell\}, andη\\etaare treated as constants\.

#### SURE controller\.

Every2020steps and for each module, we standardize the responses of up to4,0964\{,\}096tokens of the current batch, evaluate SURE on6565equally spaced multiples in\[0,cU\]\[0,c\_\{\\mathrm\{U\}\}\], and take the minimizer\. If the SureShrink sparsity statistic1N​r​∑\(v2−1\)≤\(log2⁡N​r\)3/2/N​r\\frac\{1\}\{Nr\}\\sum\(v^\{2\}\-1\)\\leq\(\\log\_\{2\}Nr\)^\{3/2\}/\\sqrt\{Nr\}indicates no signal, the controller returnscUc\_\{\\mathrm\{U\}\}\([Donoho and Johnstone, 1995](https://arxiv.org/html/2610.19033#bib.bib31)\)\. The multiple is smoothed with an EMA of0\.90\.9and ramped linearly from zero during the first5%5\\%of steps\. The controller costsO⁡\(65​N​r\)O\(65\\,Nr\)operations per module per update, less than0\.2%0\.2\\%of the step time\.

#### Budgeted variant\.

For deployments that need a fixed average rank, the controller is replaced by dual ascent on the constraint𝔼​\|supp​z\|=k¯\\mathbb\{E\}\|\\mathrm\{supp\}z\|=\\bar\{k\}:cℓ←cℓ\+κ⁡\(k^ℓ−k¯\)c\_\{\\ell\}\\leftarrow c\_\{\\ell\}\+\\kappa\(\\hat\{k\}\_\{\\ell\}\-\\bar\{k\}\)withκ=0\.01\\kappa=0\.01\. This variant produces the curve in Figure[4](https://arxiv.org/html/2610.19033#S5.F4)b\.

#### Inference\.

The update depends on the input and cannot be merged intoW0W\_\{0\}\. We batch the down\-projection as in LoRA, compute the code, and apply the up\-projection only to the active columns of each token through a gathered matrix product; for batch sizes above1616, a dense product with the sparse code is equally fast\.

## Appendix GExperimental Setup

#### Data\.

*Commonsense*: the Commonsense170K training mixture of LLM\-Adapters\([Hu et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib61)\), evaluated on the eight test sets\.*Knowledge*: the MMLU auxiliary training set, evaluated on the MMLU test set\([Hendrycks et al\., 2021a](https://arxiv.org/html/2610.19033#bib.bib69)\)\.*Science*: ScienceQA questions with their textual context\([Lu et al\., 2022](https://arxiv.org/html/2610.19033#bib.bib70)\)\.*Math*:100100K examples of MetaMathQA\([Yu et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib71)\), evaluated on GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.19033#bib.bib72)\)and on the MATH test set\([Hendrycks et al\., 2021b](https://arxiv.org/html/2610.19033#bib.bib73)\)\.*Code*:100100K examples of CodeFeedback\([Zheng et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib74)\), evaluated on HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2610.19033#bib.bib75)\)and MBPP\([Austin et al\., 2021](https://arxiv.org/html/2610.19033#bib.bib76)\)\.*GLUE*\([Wang et al\., 2019](https://arxiv.org/html/2610.19033#bib.bib77)\)follows the AdaLoRA protocol with DeBERTaV3\-base\([He et al\., 2021](https://arxiv.org/html/2610.19033#bib.bib78);[Zhang et al\., 2023c](https://arxiv.org/html/2610.19033#bib.bib3)\)\.

Table 7:Datasets\. For multi\-task training, the five domain training sets are concatenated and sampled in proportion to their sizes\.
#### Prompts\.

Commonsense and math use the instruction template of LLM\-Adapters\([Hu et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib61);[Zou et al\., 2025](https://arxiv.org/html/2610.19033#bib.bib15)\): “Below is an instruction that describes a task\. Write a response that appropriately completes the request\. \#\#\# Instruction: \{question\} \#\#\# Response:”\. For multiple\-choice tasks, the instruction lists the options, and the answer is the option name extracted from the response\. Code uses the instruction–response format of CodeFeedback\([Zheng et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib74)\); the HumanEval and MBPP prompts contain the function signature and docstring, or the problem statement and test examples\. MMLU and ScienceQA present the question with its lettered options and ask for the letter\.

#### Evaluation\.

Commonsense tasks use the LLM\-Adapters prompts and answer extraction with greedy decoding\. MMLU and ScienceQA report accuracy with zero\-shot prompts after fine\-tuning\. GSM8K and MATH report exact match of the final answer, and HumanEval and MBPP report pass@1 with greedy decoding\. All methods share prompts, decoding, and answer extraction\.

#### Hyperparameters\.

Table[8](https://arxiv.org/html/2610.19033#A7.T8)lists the shared configuration\.Squelch’s own settings were chosen on a held\-out5%5\\%of Commonsense170K with Qwen3\-1\.7B and then fixed for every task and backbone\.

Table 8:Training configuration shared by all methods, andSquelch’s settings\.
#### Baselines\.

LoRA and DoRA user=32r=32andα=64\\alpha=64\. AdaLoRA starts at rank4848and prunes to an average of3232\. GoRA uses its default allocation with an average rank of3232\. HydraLoRA shares a rank\-1616down\-projection among three up\-projection heads\. MixLoRA, AdaMoLE, and LD\-MoLE use eight rank\-44experts with top\-22, adaptive\-threshold, and learned dynamic routing, respectively\. SMoRA uses3232rank\-one experts with a learned router and top\-88; FlyLoRA usesr=32r=32with a frozen sparse random down\-projection of density8/328/32and top\-88\. Baselines use their published auxiliary losses and load balancing\. Learning rates were tuned for each baseline over\{5×10−5,10−4,2×10−4,3×10−4\}\\\{5\{\\times\}10^\{\-5\},10^\{\-4\},2\{\\times\}10^\{\-4\},3\{\\times\}10^\{\-4\}\\\}on the same held\-out split\.

#### Compute\.

All experiments ran on nodes with8×8\{\\times\}H100 80GB GPUs\. A commonsense run on an 8B backbone takes about66GPU\-hours, and a domain run33–99GPU\-hours\. The full study used roughly5×1035\\times 10^\{3\}GPU\-hours\.

## Appendix HAdditional Results

#### Seed variability\.

Table[9](https://arxiv.org/html/2610.19033#A8.T9)reports standard deviations over three seeds\.Squelch’s margin over the strongest baseline exceeds three standard deviations in all four settings, and its variance is the lowest among the compared methods\.

Table 9:Averages over three seeds \(mean±\\pms\.d\.\)\.
#### GLUE\.

Table[10](https://arxiv.org/html/2610.19033#A8.T10)follows the rank\-allocation protocol on GLUE with DeBERTaV3\-base andr=8r=8\.Squelchis best on average \(89\.71 vs\. 89\.21 for the strongest baseline and 88\.41 for LoRA\) with a mean active rank of2\.92\.9of88\. The gains are largest on the small datasets \(RTE, MRPC, CoLA\), where cross\-talk between the components of a dense adapter is hardest to average out\.

Table 10:GLUEwith DeBERTaV3\-base \(matched accuracy for MNLI, Matthews correlation for CoLA, Pearson correlation for STS\-B, accuracy otherwise; mean of three seeds\)\.
#### Joint multi\-task training\.

Table[11](https://arxiv.org/html/2610.19033#A8.T11)trains one adapter on the union of the six domain training sets\. Mixing tasks costs LoRA 1\.50 points relative to separate adapters, FlyLoRA 0\.85, andSquelchonly 0\.47\. Tokens from different tasks activate different components, so the squelch separates the tasks without task labels\.

MethodMMLUSciQAGSM8KMATHHEMBPPAvg\.Δ\\DeltaLoRA63\.192\.469\.323\.439\.849\.656\.27−1\.50\-1\.50MixLoRA64\.093\.872\.025\.342\.952\.158\.35−0\.82\-0\.82LD\-MoLE64\.494\.072\.525\.443\.352\.358\.65−0\.75\-0\.75SMoRA64\.394\.372\.425\.543\.652\.458\.75−0\.50\-0\.50FlyLoRA64\.694\.172\.625\.743\.952\.458\.88−0\.85\-0\.85Squelch64\.995\.075\.027\.146\.654\.160\.45−0\.47\-0\.47

Table 11:Joint multi\-task training on LLaMA\-3\.1\-8B\.Δ\\Delta: change relative to separate single\-task adapters \(Table[2](https://arxiv.org/html/2610.19033#S4.T2)\)\.
#### Forgetting\.

Table[12](https://arxiv.org/html/2610.19033#A8.T12)evaluates held\-out general benchmarks after commonsense fine\-tuning\.Squelchloses 1\.13 points on average against 3\.87 for LoRA and 2\.20 for FlyLoRA, consistent with the larger fraction of out\-of\-domain tokens that no component adapts \(§[5\.4](https://arxiv.org/html/2610.19033#S5.SS4)\)\.

Table 12:Held\-out accuracy after commonsense fine\-tuning on LLaMA\-3\.1\-8B \(MMLU 5\-shot, GSM8K 8\-shot, HumanEval zero\-shot\)\.
#### Scaling and dictionary size\.

Tables[13](https://arxiv.org/html/2610.19033#A8.T13)and[14](https://arxiv.org/html/2610.19033#A8.T14)give the numbers behind Figure[4](https://arxiv.org/html/2610.19033#S5.F4)a,d\. The margin over FlyLoRA shrinks slightly with scale, from1\.41\.4to0\.70\.7points, as larger backbones need less adaptation, while the mean active rank grows slowly\. With larger dictionaries the active rank grows sublinearly \(from2\.92\.9atr=8r\{=\}8to15\.815\.8atr=128r\{=\}128\), so the extra parameters are spent where the responses rise above the floor\.

Table 13:Commonsense average across Qwen3 sizes\.Table 14:Commonsense average on LLaMA\-3\.1\-8B for different dictionary sizes\.
#### Sensitivity\.

Table[15](https://arxiv.org/html/2610.19033#A8.T15)variesSquelch’s own settings\. Most settings stay within0\.50\.5points of the default, and the defaults are at or near the best value of each sweep\. The exceptions are removing or shortening the warm\-up, and replacing the robust scale by the analytic‖x‖2/d\\\|x\\\|\_\{2\}/\\sqrt\{d\}, which ignores that trained keys concentrate on the principal directions of the input\. The fixed stepη=1\\eta=1overshoots on redundant pairs, which is why the step size is set from the value Gram\.

Table 15:Sensitivity on LLaMA\-3\.1\-8B \(commonsense average; bold: best in each block\)\. Defaults: warm\-up0\.050\.05, cap2​log⁡r\\sqrt\{2\\log r\}, interval2020, EMA0\.90\.9,η=1/λmax\\eta=1/\\lambda\_\{\\max\}, MAD,α=64\\alpha=64\.
#### Cost\.

Table[16](https://arxiv.org/html/2610.19033#A8.T16)compares training time, inference latency, and memory\.Squelchtrains 5% slower than LoRA and runs 5% slower at inference \(unmerged, batch 1\), cheaper than routed mixtures and DoRA\. Merged LoRA is12%12\\%faster than unmerged LoRA; no input\-dependent adapter can be merged\.

Table 16:Cost on LLaMA\-3\.1\-8B \(training time and inference latency relative to unmerged LoRA; peak training memory per GPU\)\.
#### Per\-module statistics\.

Table[17](https://arxiv.org/html/2610.19033#A8.T17)collects the per\-module numbers behind Figure[3](https://arxiv.org/html/2610.19033#S5.F3)\. Modules with more coherent co\-active values have larger gaps between the marginal and the joint code, and explaining away removes5656–70%70\\%of the gap, the most in FFN modules\.

Table 17:Per\-module statistics on LLaMA\-3\.1\-8B \(commonsense adapter\): mean active rank, SURE multiple, share of first\-pass actives inhibited and of new components recruited by explaining away, mean\|cos\|\|\\cos\|of co\-active values, and gap to the joint code before and after explaining away\.
#### Budgeted variant\.

Table[18](https://arxiv.org/html/2610.19033#A8.T18)lists the numbers behind Figure[4](https://arxiv.org/html/2610.19033#S5.F4)b\. At every target rank, the budgeted squelch outperforms top\-kkrules with the same average number of active components\. The margin is largest at small budgets, where choosing*which*tokens receive the few components matters most\.

Table 18:Commonsense average on LLaMA\-3\.1\-8B at matched mean active rank\. “Squelchtop\-kk” replaces the floor by a fixed top\-kkbut keeps explaining away\.
#### Training dynamics and per\-token ranks\.

Figure[6](https://arxiv.org/html/2610.19033#A8.F6)shows that the mean active rank falls from3232to about99within the first tenth of training and then stays nearly constant, while the SURE multiples settle within the first tenth of training\. Figure[7](https://arxiv.org/html/2610.19033#A8.F7)shows the distribution of per\-token ranks and examples\. Numbers and quantity words \(“48”, “half”, “many”\) recruit the most components, and function words and punctuation the fewest\. On an out\-of\-domain MMLU question, most tokens keep zero to two components\.

Figure 6:Training dynamics on LLaMA\-3\.1\-8B \(commonsense\): \(a\) training loss; \(b\) mean active rank \(left\) and mean SURE multiple \(right\)\.![Refer to caption](https://arxiv.org/html/2610.19033v1/fig_tokens.png)Figure 7:Per\-token ranks on LLaMA\-3\.1\-8B\. \(a\) Distribution over token–module pairs \(math and commonsense evaluation data\)\. \(b\) Active rank of each token in the layer\-20 up\-projection for a GSM8K problem, a WinoGrande sentence, and an out\-of\-domain MMLU question \(commonsense adapter\)\.

## Appendix IQuestions and Answers

#### Is the squelch just a ReLU gate on LoRA ranks?

No\. A ReLU gate, or SoRA’s proximal gate\([Ding et al\., 2023](https://arxiv.org/html/2610.19033#bib.bib4)\), thresholds at a fixed or learned level that is shared by all inputs, and its sparsity is set by a penalty weight\. The squelch thresholds each token relative to that token’s own noise level, which makes the active set invariant to the token’s norm \(Proposition[1](https://arxiv.org/html/2610.19033#Thmproposition1)\(ii\)\), and it sets the multiple by an unbiased risk estimate, without a penalty weight or a target sparsity\. Table[3](https://arxiv.org/html/2610.19033#S5.T3)shows that both choices matter: a per\-layer scale loses0\.460\.46and a threshold learned by the task loss collapses toward a dense code\.

#### Why not learn a router, as mixture methods do?

A router adds parameters, must be balanced, and learns from a loss that rewards activating more components\.Squelchroutes through the keys that the adapter already has and decides the number of active components with a label\-free statistic\([Peng et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib14)\)\. At matched average rank it outperforms routed rank\-wise mixtures \(Table[18](https://arxiv.org/html/2610.19033#A8.T18)\)\.

#### Are the responses really Gaussian?

Only approximately\. The median estimate needs only a symmetric bulk and at most half of the components carrying signal, and SURE needs only an approximately Gaussian bulk near the threshold\. Heavier\-tailed noise\([Kunstner et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib18)\)would make the universal cap too permissive, but SURE’s multiples stay well below the cap \(Figure[3](https://arxiv.org/html/2610.19033#S5.F3)c\), and the sensitivity to the cap is small \(Table[15](https://arxiv.org/html/2610.19033#A8.T15)\)\.

#### What happens when a token needs many components?

The median estimate then errs upward \(Appendix[C](https://arxiv.org/html/2610.19033#A3)\), which raises the threshold, so the squelch is conservative for such tokens\. Explaining away partly compensates by recruiting complementary components \(Figure[3](https://arxiv.org/html/2610.19033#S5.F3)d\)\. A larger dictionary helps more than a lower threshold \(Figure[4](https://arxiv.org/html/2610.19033#S5.F4)a\)\.

#### DoesSquelchsave compute?

Not much at training time\. The down\-projection must be computed for all components to evaluate the responses, and LoRA’s cost is small to begin with\. Its purpose is accuracy and selectivity at LoRA’s parameter count\. At inference, inactive components skip the up\-projection, but merged LoRA remains faster\([Sheng et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib19)\)\(Table[16](https://arxiv.org/html/2610.19033#A8.T16)\)\.

#### Is the comparison fair to the mixtures?

All methods have the same target modules, data, optimizer, schedule, and evaluation, and all but FlyLoRA and SMoRA are matched to LoRA’s parameter count\. FlyLoRA has fewer trainable parameters by design, and SMoRA has more because of its router\. Learning rates were tuned per method \(Appendix[G](https://arxiv.org/html/2610.19033#A7)\)\.

#### CanSquelchbe combined with other LoRA improvements?

Yes\. The squelch only changes how components are combined per token, so it composes with initialization schemes, magnitude–direction decompositions, and optimizer corrections\([Meng et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib80);[Liu et al\., 2024b](https://arxiv.org/html/2610.19033#bib.bib79);[Wang et al\., 2024b](https://arxiv.org/html/2610.19033#bib.bib81);[Hayou et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib84)\)\. We have not tested these combinations\.

## Appendix JDiscussion

#### Why a noise model instead of a router\.

A router must learn from the task loss both where to send a token and how many components to use, and the task loss rewards using more\.Squelchseparates the two questions\. Where to send a token is answered by the keys, as in other self\-routing designs\([Zou et al\., 2026b](https://arxiv.org/html/2610.19033#bib.bib12);[Lu et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib23)\)\. How many components it needs is answered by comparing each response with what an irrelevant component would produce\. The second question has a label\-free, statistically grounded answer, and the rest of the method follows from it: a robust scale, an unbiased risk, and a joint code\.

#### What the noise floor is\.

The floor is the response of components whose keys are unrelated to the token\. Because LoRA’s down\-projection moves little from its random initialization\([Zhu et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib89)\), most keys remain close to random directions, and their responses are close to Gaussian with a scale proportional to‖x‖\\\|x\\\|; this is why the bulk in Figure[1](https://arxiv.org/html/2610.19033#S1.F1)a matches an untrained adapter\. As training proceeds, keys that align with a token’s directions leave the bulk, and the squelch lets exactly those through\. The MAD scale makes the rule invariant to the token’s norm \(Proposition[1](https://arxiv.org/html/2610.19033#Thmproposition1)\(ii\)\), which matters because norms vary widely across tokens and layers\([Sun et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib20)\)\.

#### What explaining away does and does not do\.

Explaining away turns marginal decisions into a joint one: it removes double counting among overlapping values and recruits components that are needed only together\. It does*not*make the dictionary itself less redundant, and our synthetic study shows that near\-duplicate components can stay aligned when it is used \(Appendix[E](https://arxiv.org/html/2610.19033#A5)\)\. Methods that disentangle co\-activated components through a regularizer target a different quantity, the coherence of the dictionary\. In our experiments, such a penalty partially substitutes for explaining away but adds nothing on top of it \(Table[3](https://arxiv.org/html/2610.19033#S5.T3)\), and helps only when the dictionary is too small for the task \(Table[6](https://arxiv.org/html/2610.19033#A5.T6)\)\.

#### WhenSquelchhelps less\.

Tasks whose adaptation is close to a uniform shift, such as MMLU in our experiments, gain little from input\-dependent rank, because most tokens need the same few components\. Very small dictionaries \(r≤8r\\leq 8\) leave little room for selection\. Settings that require merging the adapter into the backbone for zero\-overhead inference should use LoRA, or distill the squelched adapter into a merged one\([Bensaid et al\., 2026](https://arxiv.org/html/2610.19033#bib.bib21)\)\.

#### Extensions\.

The same statistical view applies to other ways of combining many small components per input: routing among full experts, where the responses are router logits; sparse autoencoders, where SURE could set per\-feature thresholds without a sparsity target; and continual learning, where tokens that no old component answers could trigger new components\. Applying the squelch in reinforcement\-learning fine\-tuning, where gradient\-based allocation has been reported to fail\([Sawant, 2026](https://arxiv.org/html/2610.19033#bib.bib38)\), is a natural next test\.

#### Broader impact\.

Squelchmakes fine\-tuning more parameter\- and compute\-efficient and leaves out\-of\-domain inputs closer to the base model\. The first lowers the cost of adaptation for everyone, including malicious users; the second reduces, but does not remove, the risk that fine\-tuning degrades safety behavior\([Qi et al\., 2024](https://arxiv.org/html/2610.19033#bib.bib22)\)\.