Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

# Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
Source: [https://arxiv.org/html/2607.09287](https://arxiv.org/html/2607.09287)
Ivan IlinPhilip Zmushko222Work done while Philip Zmushko was an intern at KAUST; he is now affiliated with ISTA and was affiliated with Yandex Research before coming to KAUST\.Peter RichtárikKAUSTKAUSTKAUSTivan\.ilin@kaust\.edu\.sazmushko\.ph\.a@gmail\.competer\.richtarik@kaust\.edu\.sa

###### Abstract

Large language models \(LLMs\) remain expensive to fine\-tune because full\-parameter updates require substantial memory, compute, and per\-task storage\. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt\. We proposeSuper, a sparse parameter\-efficient fine\-tuning \(PEFT\) method that fixes a small trainable support using aWanda\-style activation\-weighted magnitude score\(Sunet al\.,[2023](https://arxiv.org/html/2607.09287#bib.bib16)\)computed from a calibration pass\. We then introduceSupra, a hybrid adapter that combines this sparse update withLoRAwhile preserving a matched trainable\-parameter budget through a simple budget\-splitting rule\. In single\-seed Math17K arithmetic experiments onLlama\-3\.2\-1BandMeta\-Llama\-3\-8B, the bestSuper/Supravariants achieve the highest average accuracy among the tested schedule\-selected adapter configurations\. We also include a PaFi\-style magnitude\-only support as a closest training\-free sparse baseline and find that low\-score supports under both magnitude andWanda\-style orderings can be effective\. These results suggest that simple pruning\-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low\-rank adapters\.

11footnotetext:Code repository:[https://github\.com/vectozavr/SuperTuning](https://github.com/vectozavr/SuperTuning)\.## 1Introduction

Large Language Models \(LLMs\) have become central to modern natural language processing, achieving strong results in a wide array of tasks including question answering, summarization, code generation, and reasoning\. However, their impressive capabilities come with a substantial cost: full fine\-tuning of such models requires considerable computational resources, high memory consumption, and significant storage for each downstream task\. This inefficiency becomes particularly problematic when deploying models in real\-world settings with resource constraints or personalization requirements\.

To address these limitations,*parameter\-efficient fine\-tuning*\(PEFT\) methods have been proposed\. These approaches update only a small fraction of the model’s parameters, leaving the majority of weights frozen\. Among the most successful PEFT techniques isLoRA\(Huet al\.,[2021](https://arxiv.org/html/2607.09287#bib.bib56)\), which injects low\-rank trainable matrices into existing layers\.LoRAenables tuning with minimal parameter overhead and has been widely adopted for its balance of efficiency and performance\.

Beyond low\-rank adaptation, recent work has explored the use of*sparse adaptation*, where only a small, selected subset of existing weights are updated\. Methods such asSIFT\(Songet al\.,[2023](https://arxiv.org/html/2607.09287#bib.bib75)\)andRoSA\(Nikdanet al\.,[2024](https://arxiv.org/html/2607.09287#bib.bib76)\)select salient weights for fine\-tuning based on importance scores or gradient signals\.PaFi\(Liaoet al\.,[2023](https://arxiv.org/html/2607.09287#bib.bib85)\)instead shows that a task\-agnostic sparse mask can be generated without data or gradients by using only pretrained weight magnitudes\.NeFT\(Xuet al\.,[2024](https://arxiv.org/html/2607.09287#bib.bib78)\)takes this further by identifying and training only the most critical neurons\.SpIEL\(Ansellet al\.,[2024](https://arxiv.org/html/2607.09287#bib.bib79)\)introduces scalable sparse tuning with structured expert layers, andSAT\(Maet al\.,[2024](https://arxiv.org/html/2607.09287#bib.bib80)\)proposes sparsity\-accelerated training with carefully selected updates\.S2​FT\\text\{S\}^\{2\}\\text\{FT\}\(Yanget al\.,[2024](https://arxiv.org/html/2607.09287#bib.bib81)\)combines sparsity with structured decomposition for efficient and generalizable tuning, whileGIFT\-SW\(Zhelninet al\.,[2024](https://arxiv.org/html/2607.09287#bib.bib82)\)uses noise\-injected fine\-tuning on salient weights to improve robustness\. Recent salience\-aware sparse PEFT work further studies static and dynamic masks and compares several salience metrics for sparse fine\-tuning\(Liuet al\.,[2025](https://arxiv.org/html/2607.09287#bib.bib84)\)\.

While sparse tuning methods are promising, they often require multiple training phases, gradient computations, or architectural modifications to identify important weights\. A natural way to improve parameter\-efficient tuning is to combine sparse and low\-rank updates\. This direction is also explored bySLTrain\(Hanet al\.,[2024](https://arxiv.org/html/2607.09287#bib.bib83)\)in a pretraining setting and, more directly for PEFT, byRoSA\(Nikdanet al\.,[2024](https://arxiv.org/html/2607.09287#bib.bib76)\), which integrates sparse and low\-rank adapters\. Low\-rank variants such asAdaLoRA\(Zhanget al\.,[2023](https://arxiv.org/html/2607.09287#bib.bib88)\)andDoRA\(Liuet al\.,[2024](https://arxiv.org/html/2607.09287#bib.bib89)\)also show that the basicLoRAparameterization can be improved by changing rank allocation or weight decomposition\. We study whether a pruning\-derived, activation\-aware, training\-free salience score can provide a useful fixed sparse support for PEFT, and whether this support can complementLoRAunder a transparent parameter\-count budget\.

In this paper, we introduceSuper, a sparse PEFT method based onWanda\-ordered weights\. Inspired by pruning techniques such asWanda\(Sunet al\.,[2023](https://arxiv.org/html/2607.09287#bib.bib16)\),Superidentifies and fine\-tunes a small set of weights under an activation\-weighted magnitude metric\. Unlike sparse PEFT approaches that require additional gradient computations or training stages for weight selection,Superrelies on a simple, interpretable, training\-free metric computed from calibration activations\. We study both TopK and BottomK regimes, selecting respectively the largest and smallestWanda\-style scores\. In our schedule\-selected arithmetic\-reasoning comparisons, low\-score supports often give the strongest sparse and sparse–low\-rank variants, although the best direction can depend on model size, training schedule, and whether the sparse support is used alone or combined withLoRA\. We also test a PaFi\-style magnitude\-only selection rule, replacing theWandascore by\|Wi​j\|\|W\_\{ij\}\|, to separate the effect of activation weighting from the simpler pretrained\-magnitude prior and to provide a closest training\-free sparse baseline\.

We further proposeSupra, a hybrid method that combinesSuperwithLoRAto jointly leverageWanda\-selected sparse updates and low\-rank adaptation\. This design belongs to the same broad family as hybrid sparse–low\-rank methods such asRoSA\(Nikdanet al\.,[2024](https://arxiv.org/html/2607.09287#bib.bib76)\); the main distinction is that our sparse support is selected once using a training\-free score, and we explicitly compareWanda\-style and magnitude\-only BottomK supports under a matched scalar\-parameter budget\. Our experiments fine\-tune two Llama\-family models on Math17K and evaluate arithmetic reasoning on AddSub, MultiArith, SingleEq, GSM8K, AQuA, and SVAMP under matched trainable\-parameter budgets\.

We summarize our key contributions as follows:

- •We proposeSuper, aWanda\-ordered sparse fine\-tuning method that updates only a small set of weights selected without training or gradient statistics\.
- •We introduceSupra, a hybrid PEFT strategy that combinesWanda\-selected sparse updates withLoRAunder an explicit parameter\-count budget\.
- •We use a simple budget\-matched rank conversion rule that maps a fixed parameter\-count budget into a layerwiseLoRArank; unlike adaptive rank\-allocation methods, this rule is a transparent budget conversion rather than a learned or importance\-weighted allocation\.
- •We evaluateLoRA,SIFT,RoSA, random sparse supports, PaFi\-style magnitude supports,Super, andSupraon Math17K arithmetic fine\-tuning under matched trainable\-parameter budgets\. In the reported single\-seed schedule\-selected comparisons, the best sparse and sparse–low\-rank variants achieve the highest average accuracy among the tested adapter configurations\.

[Tables˜1](https://arxiv.org/html/2607.09287#S1.T1)and[2](https://arxiv.org/html/2607.09287#S1.T2)summarize the primary matched\-budget arithmetic comparisons, and[Table˜3](https://arxiv.org/html/2607.09287#S1.T3)collects the corresponding sparse\-mask rules, average accuracies, and average perplexities\.Supra\-Magdenotes the same sparse–low\-rank budget split asSupra, but with the sparse support selected by bottom\|Wi​j\|\|W\_\{ij\}\|rather than by theWandascore\. The complete per\-benchmark accuracy and perplexity tables for the one\- and three\-epoch Math17K runs are reported in[Sections˜C\.2\.2](https://arxiv.org/html/2607.09287#A3.SS2.SSS2),[C\.2\.3](https://arxiv.org/html/2607.09287#A3.SS2.SSS3),[C\.3\.2](https://arxiv.org/html/2607.09287#A3.SS3.SSS2)and[C\.3\.3](https://arxiv.org/html/2607.09287#A3.SS3.SSS3)\.

The compact tables below are schedule\-selected summaries of the full one\- and three\-epoch grids reported in the appendix\. They are intended to highlight the best observed configuration for each method family under the matched\-budget protocol\. The same reading convention applies to both tables: for each method, we report the one\- or three\-epoch run with the higher average exact\-answer accuracy, and theEpochscolumn gives the selected schedule\. Bold marks the best observed matched\-budget adapter average, shaded rows are proposed methods, all entries are exact\-answer accuracy \(%\) across AddSub, MultiArith, SingleEq, GSM8K, AQuA, and SVAMP\.

### 1\.1Notation

All key notation used in this paper is summarized in a tabular form in[Appendix˜A](https://arxiv.org/html/2607.09287#A1); see Table[6](https://arxiv.org/html/2607.09287#A1.T6)\.

Table 1:Schedule\-selected Math17K arithmetic comparison forLlama\-3\.2\-1Bunder rank\-equivalent budgetr0=8r\_\{0\}=8\.MethodEpochsSelected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––13\.674\.6721\.462\.8121\.6511\.6012\.64Full fine\-tuning35⋅10−55\\cdot 10^\{\-5\}80\.0091\.6785\.6344\.0528\.7464\.1065\.70LoRA35⋅10−45\\cdot 10^\{\-4\}70\.6390\.8385\.2435\.3326\.7757\.6061\.07RoSA15⋅10−45\\cdot 10^\{\-4\}71\.6589\.1778\.3527\.9824\.8053\.7057\.61SIFT\(TopK\)110−410^\{\-4\}60\.5183\.1763\.1923\.433\.9440\.3045\.75SIFT\(RandK\)110−310^\{\-3\}62\.2890\.8373\.6231\.4617\.7248\.7054\.10Magnitude\(BottomK\)110−310^\{\-3\}73\.9290\.5081\.6930\.2524\.8054\.7059\.31Super\(BottomK\)35⋅10−45\\cdot 10^\{\-4\}69\.1185\.5070\.4734\.5025\.5947\.6055\.46Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)15⋅10−45\\cdot 10^\{\-4\}76\.7191\.6781\.1022\.9727\.5655\.1059\.18Supra\(BottomK,λ=0\.8\\lambda=0\.8\)35⋅10−45\\cdot 10^\{\-4\}82\.5385\.5084\.4536\.1624\.4160\.3062\.23

Table 2:Schedule\-selected Math17K arithmetic comparison forMeta\-Llama\-3\-8Bunder rank\-equivalent budgetr0=8r\_\{0\}=8\.MethodEpochsSelected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––22\.5321\.0036\.4210\.2424\.0224\.1023\.05Full fine\-tuning35⋅10−55\\cdot 10^\{\-5\}90\.6396\.1793\.7054\.3638\.1976\.2074\.87LoRA15⋅10−45\\cdot 10^\{\-4\}86\.3397\.6793\.7066\.3424\.4170\.6073\.17RoSA110−410^\{\-4\}86\.8497\.5092\.7258\.4535\.8379\.6075\.16SIFT\(TopK\)110−410^\{\-4\}86\.5898\.5094\.6963\.7635\.0479\.1076\.28SIFT\(RandK\)110−310^\{\-3\}84\.3098\.3393\.5063\.9945\.2881\.6077\.83Magnitude\(BottomK\)110−310^\{\-3\}90\.8997\.3395\.0868\.0141\.7381\.1079\.02Super\(BottomK\)310−410^\{\-4\}72\.1595\.6788\.5866\.3442\.9170\.7072\.73Supra\(BottomK,λ=0\.3\\lambda=0\.3\)15⋅10−45\\cdot 10^\{\-4\}89\.1195\.3396\.0667\.6341\.3482\.5078\.66Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)15⋅10−45\\cdot 10^\{\-4\}92\.9198\.5096\.8569\.8335\.4381\.2079\.12

Table 3:Summary of the schedule\-selected main\-text rows\. Fine\-tuning dataset: Math17K; rank\-equivalent budget:r0=8r\_\{0\}=8\. The score column gives the per\-weight scalar used to rank candidate sparse coordinates;Xj:X\_\{j:\}denotes the calibration activation vector feeding columnjjof a layer\. Average accuracy is exact\-answer accuracy \(%\) across AddSub, MultiArith, SingleEq, GSM8K, AQuA, and SVAMP; average PPL is test perplexity over the same benchmark set, where lower is better\.MethodPer\-weight scoreLlama\-3\.2\-1BMeta\-Llama\-3\-8BAcc\.PPLAcc\.PPLBase\(frozen\)–12\.642\.7323\.052\.22Full fine\-tuningall parameters65\.701\.1174\.871\.07LoRA–61\.071\.1773\.171\.14RoSA\|∂ℒ/∂Wi​j\|\\left\|\\nicefrac\{\{\\partial\\mathcal\{L\}\}\}\{\{\\partial W\_\{ij\}\}\}\\right\|57\.611\.2775\.161\.18SIFT\(TopK\)\|∂ℒ/∂Wi​j\|\\left\|\\nicefrac\{\{\\partial\\mathcal\{L\}\}\}\{\{\\partial W\_\{ij\}\}\}\\right\|45\.751\.3376\.281\.18SIFT\(RandK\)Unif​\(0,1\)\\mathrm\{Unif\}\(0,1\)54\.101\.2577\.831\.13Magnitude\(BottomK\)\|Wi​j\|\\left\|W\_\{ij\}\\right\|59\.311\.2479\.021\.13Super\(BottomK\)\|Wi​j\|​‖Xj:‖2\\left\|W\_\{ij\}\\right\|\\,\\left\\\|X\_\{j:\}\\right\\\|\_\{2\}55\.461\.2172\.731\.31Supra\(BottomK,λ=0\.3\\lambda=0\.3\)\|Wi​j\|​‖Xj:‖2\\left\|W\_\{ij\}\\right\|\\,\\left\\\|X\_\{j:\}\\right\\\|\_\{2\}55\.031\.2678\.661\.14Supra\(BottomK,λ=0\.8\\lambda=0\.8\)\|Wi​j\|​‖Xj:‖2\\left\|W\_\{ij\}\\right\|\\,\\left\\\|X\_\{j:\}\\right\\\|\_\{2\}62\.231\.1678\.281\.14Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)\|Wi​j\|\\left\|W\_\{ij\}\\right\|59\.181\.2579\.121\.14

## 2Related Work

### 2\.1Parameter–Efficient Fine‑Tuning \(PEFT\)

The prohibitive compute and memory cost of updating all parameters of a large language model \(LLM\) has motivated a rich literature on*parameter–efficient fine‑tuning*\(PEFT\)\. Classical adapter\-style methods insert small*dense*modules into each Transformer block \(e\.g\., adapters, prefix\-tuning, andLoRA\), greatly reducing trainable parameters, optimizer state, and per\-task storage\. Complementary work studies*sparse*or otherwise*structured*updates that explicitly select a subset of parameters to train, concentrating adaptation in a smaller set of task\-specific degrees of freedom\.

### 2\.2Sparse fine‑tuning

Several papers report that high accuracy can be recovered even on reasoning‑heavy tasks – when a model is updated on only a small fraction of its weights\.Songet al\.\([2023](https://arxiv.org/html/2607.09287#bib.bib75)\)analyze the PAC‑Bayesian generalization bound of PEFT and proposeSIFT, a gradient‑based algorithm that activates at mostkkparameters per layer during training\.Ansellet al\.\([2024](https://arxiv.org/html/2607.09287#bib.bib79)\)show that dynamically grown–pruned sparse deltas scale to 13‑B‑parameter LLaMA‑2 while retaining memory proportional to the sparsity pattern rather than to model size\. Orthogonally,Maet al\.\([2024](https://arxiv.org/html/2607.09287#bib.bib80)\)exploit the observation that only a minority of neurons fire on any given example and skip the forward/backward pass of inactive neurons to accelerate both continual pre‑training and supervised fine‑tuning by up to45%45\\%\. Building on structured sparsity,Yanget al\.\([2024](https://arxiv.org/html/2607.09287#bib.bib81)\)introduceS2FT, which selects a small set of attention heads and MLP channels, then co‑permutes weight matrices so the selected components form dense sub‑matrices that can be trained efficiently with ordinary GEMM kernels\.

For unstructured weight\-level sparse masks, different methods mainly differ in the scalar score used to rank candidate coordinates\. Given a fine\-tuning lossℒ\\mathcal\{L\}, we write the gradient\-magnitude score for an entryWi​jW\_\{ij\}as

Gi​j=\|∂ℒ∂Wi​j\|\.G\_\{ij\}=\\left\|\\frac\{\\partial\\mathcal\{L\}\}\{\\partial W\_\{ij\}\}\\right\|\.This is the score notation used in our method\-summary table for gradient\-based sparse selection methods such asSIFTand the sparse component ofRoSA\. In contrast,PaFiuses a data\-free pretrained\-magnitude score,

Bi​j=\|Wi​j\|,B\_\{ij\}=\|W\_\{ij\}\|,selecting sparse fine\-tuning coordinates from the magnitude ordering of the original weights\(Liaoet al\.,[2023](https://arxiv.org/html/2607.09287#bib.bib85)\)\. OurMagnitudeandSupra\-Magrows use this PaFi\-style score, whileSuperand the defaultSuprarows use the activation\-weightedWandascore introduced in[Section˜3](https://arxiv.org/html/2607.09287#S3)\.

PaFiis especially relevant because it shows that task\-agnostic sparse masks can be built from pretrained weight magnitudes alone\. We therefore treat magnitude\-only BottomK selection as a PaFi\-style sparse baseline\. Recent work such asGaLLoPalso highlights the usefulness of low\-magnitude parameters for sparse fine\-tuning, combining small pretrained magnitudes with large downstream gradient magnitudes\(Choudharyet al\.,[2025](https://arxiv.org/html/2607.09287#bib.bib77)\)\. In contrast, our defaultSupermasks are selected without downstream gradients, using a training\-freeWanda\-style activation\-weighted score computed from a calibration pass\. We also include a magnitude\-only support as a PaFi\-style baseline and study its combination withLoRA\.

### 2\.3Hybrid low‑rank & sparse methods

A complementary line of work seeks to combine the representational benefits of low‑rank adaptation with the compactness of sparsity\.RoSAis closely related because it combines sparse and low\-rank adaptation for PEFT:Nikdanet al\.\([2024](https://arxiv.org/html/2607.09287#bib.bib76)\)decompose the update into a low‑rank adapter plus a very sparse residual and optimize both jointly\. Our novelty is not that sparse and low\-rank updates can be combined, but that the sparse support is fixed by a simple training\-free ordering and that we systematically compareWanda\-style activation\-weighted supports, magnitude\-only supports, TopK/BottomK directions, and their sparse–low\-rank hybrids under matched scalar\-parameter budgets\. For the pre‑training regime,Hanet al\.\([2024](https://arxiv.org/html/2607.09287#bib.bib83)\)factorize each linear layer into a low‑rank term and a*fixed‑support*sparse term, achieving up to73%73\\%memory savings while matching full‑rank performance\.

### 2\.4Fine‑tuning at the neuron or weight level

Moving to finer granularity,Xuet al\.\([2024](https://arxiv.org/html/2607.09287#bib.bib78)\)supervise the updates at the level of individual neurons \(NeFT\), explicitly identifying task‑relevant neurons whose small subset updates can achieve higher accuracy than full‑parameter fine‑tuning\.Zhelninet al\.\([2024](https://arxiv.org/html/2607.09287#bib.bib82)\)extend the idea of salient‑weight selection withGIFT‑SW, which injects Gaussian noise into*non‑salient*columns while learning only the salient ones, closing the gap to full fine‑tuning under the same compute budget\.

### 2\.5Connections to pruning and outlier‑aware updates

Pruning and sparse fine\-tuning can be viewed as complementary mask\-selection problems\. Pruning asks which parameters can be removed or frozen while preserving the pretrained function; sparse fine\-tuning asks where a limited adaptation budget should be allocated\. This connection has appeared in several forms:FISH Maskuses Fisher information to precompute fixed sparse update masks\(Sunget al\.,[2021](https://arxiv.org/html/2607.09287#bib.bib90)\),PaFiselects low\-magnitude pretrained parameters for sparse fine\-tuning and explicitly frames the procedure as pruning\-and\-fine\-tuning\(Liaoet al\.,[2023](https://arxiv.org/html/2607.09287#bib.bib85)\), row\-based sparse fine\-tuning uses neural\-pruning feature\-importance metrics to select structured update supports\(Li and Bhaskara,[2025](https://arxiv.org/html/2607.09287#bib.bib86)\), andRoCoFTstudies row/column updates withWanda\-style row and column scoring among its selection strategies\(Kowsheret al\.,[2025](https://arxiv.org/html/2607.09287#bib.bib87)\)\. In this work, we turn this pruning\-to\-fine\-tuning connection into a simple weight\-level PEFT method: a calibration pass computes activation\-awareWanda\-style scores, and the resulting ordering defines a fixed sparse set of original model weights to train under a matched parameter budget\. We additionally include PaFi\-style magnitude masks, which remove the activation term and rank coordinates only by\|Wi​j\|\|W\_\{ij\}\|, as a direct comparison to the activation\-awareWandaordering\.

Our method is inspired by pruning research that measures weight saliencySk​qS\_\{kq\}for every weightWk​qW\_\{kq\}from a linear layerW∈ℝc×bW\\in\\mathbb\{R\}^\{c\\times b\}before deleting parameters\. Using∥⋅∥2\\\|\\cdot\\\|\_\{2\}for the Euclidean norm, theWandametric multiplies a weight’s magnitude by the norm of its input activation:

Sk​q=\(\|Wk​q\|​‖Xq:‖2\)2,S\_\{kq\}=\\left\(\|W\_\{kq\}\|\\\|X\_\{q:\}\\\|\_\{2\}\\right\)^\{2\},\(1\)whereXq:∈ℝ1×aX\_\{q:\}\\in\\mathbb\{R\}^\{1\\times a\}is theqthq^\{\\text\{th\}\}row of the input matrixX∈ℝb×aX\\in\\mathbb\{R\}^\{b\\times a\}, andaais the number of calibration token positions\. The squared form in[Equation˜1](https://arxiv.org/html/2607.09287#S2.E1)has an exact single\-weight reconstruction interpretation: as shown later in[Proposition˜3\.1](https://arxiv.org/html/2607.09287#S3.Thmtheorem1), zeroingWk​qW\_\{kq\}changes the layer output by squared Frobenius normWk​q2​‖Xq:‖22W\_\{kq\}^\{2\}\\\|X\_\{q:\}\\\|\_\{2\}^\{2\}\.

The metricSk​qS\_\{kq\}is used to approximate output\-level importance and enables one\-shot pruning without weight updates\(Sunet al\.,[2023](https://arxiv.org/html/2607.09287#bib.bib16)\)\. In pruning, low\-Wanda\-score weights are natural candidates for removal because they have small estimated contribution to the layer\-output reconstruction proxy\. In sparse fine\-tuning, we instead keep the model intact and ask whether this same activation\-aware ordering can identify a small set of weights that can be safely and effectively trained\. We therefore study both TopK and BottomK masks, selecting respectively the largest and smallestWanda\-style scores under the same sparse budget\. A natural intermediate possibility is to split the sparse budget between both ends of the score ordering; we evaluate this mixed TopK–BottomK support in[Section˜C\.1](https://arxiv.org/html/2607.09287#A3.SS1)\.

Our experiments suggest that BottomK entries often form useful sparse fine\-tuning supports in this setting\. This strengthens the connection to pruning: the weights that appear least critical for preserving the pretrained layer output can also provide useful degrees of freedom for task adaptation\. One possible explanation is that high\-score weights help preserve the pretrained computation, so perturbing them can be risky, while low\-score weights can be changed more freely\. Another is optimization\-related: because BottomK entries are less sensitive under the reconstruction proxy, they may tolerate larger learning rates and adapt faster under a limited training budget\. Thus, theWanda\-style score should be viewed as an activation\-aware ordering signal for sparse fine\-tuning, not only as a pruning metric\.

## 3Adaptation of Large Language Models

### 3\.1Notation

Let𝒩\\mathcal\{N\}denote a pre\-trained Large Language Model \(LLM\), and let𝒲=\{W\(1\),W\(2\),…,W\(m\)\}\\mathcal\{W\}=\\\{W^\{\(1\)\},W^\{\(2\)\},\\ldots,W^\{\(m\)\}\\\}be the collection of fully connected weight matrices in𝒩\\mathcal\{N\}considered for adaptation, including those within attention sublayers, with eachW\(i\)∈ℝci×biW^\{\(i\)\}\\in\\mathbb\{R\}^\{c\_\{i\}\\times b\_\{i\}\}for1≤i≤m1\\leq i\\leq m\. We use𝒜⊆\{1,…,m\}\\mathcal\{A\}\\subseteq\\\{1,\\ldots,m\\\}for the index set of matrices actually adapted by a given method; when all candidate matrices are adapted,𝒜=\{1,…,m\}\\mathcal\{A\}=\\\{1,\\ldots,m\\\}\. Let the vectorw¯∈ℝd¯\\bar\{w\}\\in\\mathbb\{R\}^\{\\bar\{d\}\}represent the remaining parameters of𝒩\\mathcal\{N\}, such as biases and normalization parameters, concatenated into a single vector\. Given a dataset𝒟\\mathcal\{D\}and a loss functionℒ​\(𝒟;𝒲,w¯\)\\mathcal\{L\}\(\\mathcal\{D\};\\mathcal\{W\},\\bar\{w\}\), the full fine\-tuning \(FFT\) of𝒩\\mathcal\{N\}can be formulated as the following optimization problem:

min𝒲,w¯⁡ℒ​\(𝒟;𝒲,w¯\)\.\\min\_\{\\mathcal\{W\},\\bar\{w\}\}\\mathcal\{L\}\(\\mathcal\{D\};\\mathcal\{W\},\\bar\{w\}\)\.\(2\)Due to the large scale of modern LLMs – often comprising billions of parameters – performing FFT is computationally expensive and memory\-intensive, making it impractical on standard GPUs\. A practical alternative is to apply lightweight modifications known as*adapters*, which we formalize next\.

LetΔ=\{Δ\(1\),Δ\(2\),…,Δ\(m\)\}\\Delta=\\\{\\Delta^\{\(1\)\},\\Delta^\{\(2\)\},\\ldots,\\Delta^\{\(m\)\}\\\}denote perturbations applied to the fully connected weights, withΔ\(i\)∈ℝci×bi\\Delta^\{\(i\)\}\\in\\mathbb\{R\}^\{c\_\{i\}\\times b\_\{i\}\}for allii\. Define the adapted weights as𝒲\+Δ=\{W\(1\)\+Δ\(1\),W\(2\)\+Δ\(2\),…,W\(m\)\+Δ\(m\)\}\\mathcal\{W\}\+\\Delta=\\\{W^\{\(1\)\}\+\\Delta^\{\(1\)\},W^\{\(2\)\}\+\\Delta^\{\(2\)\},\\ldots,W^\{\(m\)\}\+\\Delta^\{\(m\)\}\\\}, and letδ¯∈ℝd¯\\bar\{\\delta\}\\in\\mathbb\{R\}^\{\\bar\{d\}\}represent perturbations tow¯\\bar\{w\}\. The adapted model is then obtained by solving:

minΔ,δ¯⁡ℒ​\(𝒟;𝒲\+Δ,w¯\+δ¯\)s\.t\.𝒞​\(Δ,δ¯\),\\min\_\{\\Delta,\\bar\{\\delta\}\}\\mathcal\{L\}\(\\mathcal\{D\};\\mathcal\{W\}\+\\Delta,\\bar\{w\}\+\\bar\{\\delta\}\)\\quad\\text\{s\.t\.\}\\quad\\mathcal\{C\}\(\\Delta,\\bar\{\\delta\}\),\(3\)where𝒞​\(Δ,δ¯\)\\mathcal\{C\}\(\\Delta,\\bar\{\\delta\}\)encodes constraints on the perturbations \(e\.g\., low\-rank or sparse structure\) to reduce the number of optimized and stored task\-specific parameters\. Such constraints may also reduce compute when paired with suitable structure or kernels, but unstructured sparse or low\-rank adapters still require forward and backward propagation through the frozen pretrained model\. Notably, if no constraints are imposed, this setting reduces to standard FFT\.

In this work, we focus on the common scenario whereδ¯=0\\bar\{\\delta\}=0, though in principlew¯\\bar\{w\}can also be fine\-tuned, especially given its relatively small size compared to𝒲\\mathcal\{W\}\. Our reported experiments adapt the full set of standard Llama linear projections listed in[Section˜4](https://arxiv.org/html/2607.09287#S4), while the notation also allows partial adaptation through the index set𝒜\\mathcal\{A\}\.

##### LoRA: Low\-Rank Adaptation\.

Low\-rank adaptation \(LoRA\)\(Huet al\.,[2021](https://arxiv.org/html/2607.09287#bib.bib56)\)constrains each perturbationΔ\(i\)\\Delta^\{\(i\)\}to be low\-rank, specifically:

minΔ⁡ℒ​\(𝒟;𝒲\+Δ,w¯\)s\.t\.rank​\(Δ\(i\)\)≤r∀i\.\\min\_\{\\Delta\}\\mathcal\{L\}\(\\mathcal\{D\};\\mathcal\{W\}\+\\Delta,\\bar\{w\}\)\\quad\\text\{s\.t\.\}\\quad\\text\{rank\}\(\\Delta^\{\(i\)\}\)\\leq r\\quad\\forall\\,i\.\(4\)In the usual implementation, the update is parameterized as

Δ\(i\)=γi​L\(i\)​R\(i\),L\(i\)∈ℝci×ri,R\(i\)∈ℝri×bi,\\Delta^\{\(i\)\}=\\gamma\_\{i\}L^\{\(i\)\}R^\{\(i\)\},\\qquad L^\{\(i\)\}\\in\\mathbb\{R\}^\{c\_\{i\}\\times r\_\{i\}\},\\quad R^\{\(i\)\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times b\_\{i\}\},\(5\)whereγi\\gamma\_\{i\}is theLoRAscaling factor\. This usesri​\(ci\+bi\)r\_\{i\}\(c\_\{i\}\+b\_\{i\}\)trainable scalar parameters for layerii\. We use this implementation parameter count for budget matching because these are the scalar adapter weights optimized and stored by standardLoRA\.

##### SpA: Sparse Adaptation\.

Using∥⋅∥0\\\|\\cdot\\\|\_\{0\}to denote the number of nonzero entries, sparse adaptation \(SpA\)Sunget al\.\([2021](https://arxiv.org/html/2607.09287#bib.bib90)\)enforces sparsity in the perturbations:

minΔ⁡ℒ​\(𝒟;𝒲\+Δ,w¯\)s\.t\.‖Δ\(i\)‖0≤p​ci​bi∀i,\\min\_\{\\Delta\}\\mathcal\{L\}\(\\mathcal\{D\};\\mathcal\{W\}\+\\Delta,\\bar\{w\}\)\\quad\\text\{s\.t\.\}\\quad\\\|\\Delta^\{\(i\)\}\\\|\_\{0\}\\leq pc\_\{i\}b\_\{i\}\\quad\\forall\\,i,\(6\)wherep∈\(0,1\]p\\in\(0,1\]is the sparsity density\. It is common to fix the sparse support during training, thereby reducing the number of trainable parameters by a factor ofpp\.

### 3\.2Super: Mask Generation Algorithm for SpA\.

Let𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}denote a tokenized calibration set\. For a layer with input dimensionbb, running the frozen model on𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}yields the layer input activation matrixX∈ℝb×aX\\in\\mathbb\{R\}^\{b\\times a\}, whereaais the total number of calibration token positions\. For fixed calibration activationsXX, we introduce the mask\-generation mapping

ψX:ℝc×b×\{0,…,c​b\}×\{TopK,BottomK\}→\{0,1\}c×b\.\\psi\_\{X\}:\\mathbb\{R\}^\{c\\times b\}\\times\\\{0,\\ldots,cb\\\}\\times\\\{\\mathrm\{TopK\},\\mathrm\{BottomK\}\\\}\\to\\\{0,1\\\}^\{c\\times b\}\.It takes a weight matrixW∈ℝc×bW\\in\\mathbb\{R\}^\{c\\times b\}, a target number of sparse entriess∈\{0,…,c​b\}s\\in\\\{0,\\ldots,cb\\\}, and a selection directionρ∈\{TopK,BottomK\}\\rho\\in\\\{\\mathrm\{TopK\},\\mathrm\{BottomK\}\\\}, then generates the corresponding fine\-tuning maskψX​\(W,s,ρ\)\\psi\_\{X\}\(W,s,\\rho\)\. For each weightWi​jW\_\{ij\}, it evaluates the score

Ai​j=\|Wi​j\|​‖Xj:‖2\.A\_\{ij\}=\|W\_\{ij\}\|\\\|X\_\{j:\}\\\|\_\{2\}\.We use this activation\-weighted magnitude score as a selection rule for fine\-tuning: the scores are flattened within each layer, and the mask contains either the global topssentries or the global bottomssentries\. Because the square is monotone on nonnegative values, this ordering is the same as the ordering induced by the squaredWandascore in[Equation˜1](https://arxiv.org/html/2607.09287#S2.E1)\.

Our use of theWandascore differs from the original pruning rule in selection granularity: we flatten scores within each adapted layer and select a global layerwise support, whereas pruning implementations often impose row\-wise budgets\. We use the termWanda\-style to emphasize that we reuse the activation\-weighted magnitude score, not the full pruning procedure\. Row\-wise support selection is a natural follow\-up but is outside the scope of the present experiments\.

The output is a binary mask with ones at these selected positions:

ψX​\(W,s,ρ\)i​j≔\{1,\(i,j\)​is among the​s​largest entries of​A,ρ=TopK,1,\(i,j\)​is among the​s​smallest entries of​A,ρ=BottomK,0,otherwise\.\\psi\_\{X\}\(W,s,\\rho\)\_\{ij\}\\coloneqq\\begin\{cases\}1,&\(i,j\)\\text\{ is among the \}s\\text\{ largest entries of \}A,\\quad\\rho=\\mathrm\{TopK\},\\\\ 1,&\(i,j\)\\text\{ is among the \}s\\text\{ smallest entries of \}A,\\quad\\rho=\\mathrm\{BottomK\},\\\\ 0,&\\text\{otherwise\.\}\\end\{cases\}\(7\)Refer to[Appendix˜B](https://arxiv.org/html/2607.09287#A2)for additional examples\.

Alongside theWanda\-style mask, we also evaluate a PaFi\-style magnitude\-only rule\(Liaoet al\.,[2023](https://arxiv.org/html/2607.09287#bib.bib85)\)\. This variant uses

Bi​j=\|Wi​j\|B\_\{ij\}=\|W\_\{ij\}\|in place ofAi​jA\_\{ij\}and applies the same TopK or BottomK selection direction and the same sparse budget\. Thus the magnitude\-only sparse baseline andSupra\-Magdiffer fromSuperand the defaultSupraonly in the ranking score used to construct the fixed sparse support:\|Wi​j\|\|W\_\{ij\}\|rather than\|Wi​j\|​‖Xj:‖2\|W\_\{ij\}\|\\\|X\_\{j:\}\\\|\_\{2\}\. This lets us test whether the strong sparse supports come from the activation\-weightedWandaproxy itself or from the simpler pretrained\-magnitude prior used by PaFi\.

One can also combine both ends of the score ordering by assigning part of the sparse budget to TopK entries and the remainder to BottomK entries\. We report this mixed TopK–BottomK ablation in[Section˜C\.1](https://arxiv.org/html/2607.09287#A3.SS1); it preserves the same sparse parameter budget, but in our experiments it does not significantly improve over the pure BottomK support, which remains a strong choice in the schedule\-selected comparisons\.

The resulting mask\-generation procedure is summarized in[Algorithm˜1](https://arxiv.org/html/2607.09287#alg1)\.

Algorithm 1SpA mask generation byWanda\-style scores1:Input:Adapted\-layer index set

𝒜⊆\{1,…,m\}\\mathcal\{A\}\\subseteq\\\{1,\\ldots,m\\\}in a frozen pretrained model, weights

\{W\(i\)∈ℝci×bi\}i∈𝒜\\\{W^\{\(i\)\}\\in\\mathbb\{R\}^\{c\_\{i\}\\times b\_\{i\}\}\\\}\_\{i\\in\\mathcal\{A\}\}, tokenized calibration set

𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}, sparse budgets

si∈\{0,…,ci​bi\}s\_\{i\}\\in\\\{0,\\ldots,c\_\{i\}b\_\{i\}\\\}, selection direction

ρ∈\{TopK,BottomK\}\\rho\\in\\\{\\mathrm\{TopK\},\\mathrm\{BottomK\}\\\}\.

2:Run the frozen model on

𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}and collect input activations

X\(i\)∈ℝbi×aX^\{\(i\)\}\\in\\mathbb\{R\}^\{b\_\{i\}\\times a\}for each

i∈𝒜i\\in\\mathcal\{A\}\.

3:foreach adapted layer

i∈𝒜i\\in\\mathcal\{A\}do

4:Compute

M\(i\)=ψX\(i\)​\(W\(i\),si,ρ\)M^\{\(i\)\}=\\psi\_\{X^\{\(i\)\}\}\(W^\{\(i\)\},s\_\{i\},\\rho\)using[Equation˜7](https://arxiv.org/html/2607.09287#S3.E7)\.

5:endfor

6:Return:Fixed sparse masks

\{M\(i\)∈\{0,1\}ci×bi\}i∈𝒜\\\{M^\{\(i\)\}\\in\\\{0,1\\\}^\{c\_\{i\}\\times b\_\{i\}\}\\\}\_\{i\\in\\mathcal\{A\}\}\.

Unlike many existing methods, our approach does not require a complex or computationally intensive procedure for selecting the trainable parameters of the sparse adapter\. Instead, we adopt a simple strategy based on one calibration pass through the model\. We refer to this method asSuper–short for*Selective Update of Parameters via Extreme Ranking*\.

For layerii, letW\(i\)W^\{\(i\)\}be the frozen pretrained matrix and letX\(i\)X^\{\(i\)\}be the calibration activations collected at that layer\. Given a sparse budgetsis\_\{i\}and selection directionρ\\rho,Superconstructs

M\(i\)=ψX\(i\)​\(W\(i\),si,ρ\)M^\{\(i\)\}=\\psi\_\{X^\{\(i\)\}\}\(W^\{\(i\)\},s\_\{i\},\\rho\)\(8\)and trains a sparse delta on this fixed support\. We writeA⊙BA\\odot Bfor the element\-wise \(Hadamard\) product of matrices with the same shape:

W^\(i\)=W\(i\)\+M\(i\)⊙U\(i\),\\widehat\{W\}^\{\(i\)\}=W^\{\(i\)\}\+M^\{\(i\)\}\\odot U^\{\(i\)\},\(9\)whereU\(i\)∈ℝci×biU^\{\(i\)\}\\in\\mathbb\{R\}^\{c\_\{i\}\\times b\_\{i\}\}is trainable only on the entries selected byM\(i\)M^\{\(i\)\}\. Equivalently, one may view the selected entries ofW\(i\)W^\{\(i\)\}as being updated in place while all unselected entries remain frozen\.

The sparse delta is initialized at zero, so the initial adapted layer equals the pretrained layer\. For adapted\-layer index set𝒜\\mathcal\{A\}, theSuperobjective can be written as

min\{U\(i\)\}i∈𝒜⁡ℒ​\(𝒟;\{W^\(i\)\}i∈𝒜,\{W\(j\)\}j∉𝒜,w¯\),\\min\_\{\\\{U^\{\(i\)\}\\\}\_\{i\\in\\mathcal\{A\}\}\}\\mathcal\{L\}\\\!\\left\(\\mathcal\{D\};\\\{\\widehat\{W\}^\{\(i\)\}\\\}\_\{i\\in\\mathcal\{A\}\},\\\{W^\{\(j\)\}\\\}\_\{j\\notin\\mathcal\{A\}\},\\bar\{w\}\\right\),\(10\)with all pretrained weights outside the selected sparse entries frozen\.

The core idea behindSuperis to leverage theWandametric, which estimates the single\-weight contribution to layer\-wiseℓ2\\ell\_\{2\}reconstruction error upon removal\. The following identity gives the exact single\-weight form used by the score\.

###### Proposition 3\.1\(Single\-weight reconstruction identity\)\.

LetW∈ℝc×bW\\in\\mathbb\{R\}^\{c\\times b\},X∈ℝb×aX\\in\\mathbb\{R\}^\{b\\times a\},Y=W​XY=WX, and let∥⋅∥F\\\|\\cdot\\\|\_\{F\}denote the Frobenius norm\. LetW\(−k​q\)W^\{\(\-kq\)\}equalWWexcept that entryWk​qW\_\{kq\}is set to zero\. Then

‖\(W−W\(−k​q\)\)​X‖F2=Wk​q2​‖Xq:‖22\.\\\|\(W\-W^\{\(\-kq\)\}\)X\\\|\_\{F\}^\{2\}=W\_\{kq\}^\{2\}\\\|X\_\{q:\}\\\|\_\{2\}^\{2\}\.\(11\)

###### Proof\.

The differenceW−W\(−k​q\)W\-W^\{\(\-kq\)\}has a single nonzero entry, so\(W−W\(−k​q\)\)​X=Wk​q​ek​Xq:\(W\-W^\{\(\-kq\)\}\)X=W\_\{kq\}e\_\{k\}X\_\{q:\}, whereeke\_\{k\}is thekk\-th standard basis vector\. Taking the Frobenius norm givesWk​q2​‖Xq:‖22W\_\{kq\}^\{2\}\\\|X\_\{q:\}\\\|\_\{2\}^\{2\}\. ∎

Importantly, this identity concerns the reconstruction effect of deleting a pretrained weight, whereasSuperuses the same score only as a support\-selection heuristic for additive sparse updates\. Once a coordinate is selected, the learned delta is not constrained to equal the negative pretrained weight\. Therefore, theWanda\-style score should not be interpreted as an optimality criterion for sparse fine\-tuning\. Rather, it provides a training\-free ordering of coordinates by their estimated sensitivity in the frozen pretrained model\.

For a set of selected weights, the exact reconstruction effect also contains cross\-terms between activation rows\. Writing⟨⋅,⋅⟩\\langle\\cdot,\\cdot\\ranglefor the Euclidean inner product, for any binary maskM∈\{0,1\}c×bM\\in\\\{0,1\\\}^\{c\\times b\},

‖\(M⊙W\)​X‖F2=∑k=1c∑q=1b∑r=1bMk​q​Mk​r​Wk​q​Wk​r​⟨Xq:,Xr:⟩\.\\\|\(M\\odot W\)X\\\|\_\{F\}^\{2\}=\\sum\_\{k=1\}^\{c\}\\sum\_\{q=1\}^\{b\}\\sum\_\{r=1\}^\{b\}M\_\{kq\}M\_\{kr\}W\_\{kq\}W\_\{kr\}\\langle X\_\{q:\},X\_\{r:\}\\rangle\.\(12\)The independent bottom\-sis\_\{i\}selection used bySupercorresponds to the diagonal approximation

∑k=1c∑q=1bMk​q​Wk​q2​‖Xq:‖22,\\sum\_\{k=1\}^\{c\}\\sum\_\{q=1\}^\{b\}M\_\{kq\}W\_\{kq\}^\{2\}\\\|X\_\{q:\}\\\|\_\{2\}^\{2\},\(13\)which drops allq≠rq\\neq rcross\-terms\. Thus,Wanda\-style selection should be understood as a diagonal reconstruction heuristic rather than an exact global optimization criterion\.

Our central hypothesis is that training can be concentrated on parameters ordered by this reconstruction proxy to obtain useful adaptation under small parameter\-count budgets\. By selecting a fixed support according to theWanda\-style metric,Superprovides a simple method for sparse fine\-tuning that aligns with this reconstruction\-based heuristic\.

### 3\.3Supra: Budget\-Matched Low\-Rank and Sparse Adaptation\.

![Refer to caption](https://arxiv.org/html/2607.09287v1/x1.png)Figure 1:Supracombines a sparse adapter based onSuperweights with a low\-rank adapter\. To select the parameters for training in the sparse adapter, we employ theWandametric \(Equation[1](https://arxiv.org/html/2607.09287#S2.E1)\)\. The rankrrof the low\-rank adapter is determined from the matched parameter\-count budget using Equation[19](https://arxiv.org/html/2607.09287#S3.E19)\.Suprais a hybrid fine\-tuning strategy that combines two complementary approaches: a sparse adapter selected via theSuperstrategy and a low\-rank adapter based onLoRA\. This design belongs to the same broad family as hybrid sparse–low\-rank methods such asRoSA\(Nikdanet al\.,[2024](https://arxiv.org/html/2607.09287#bib.bib76)\); the main distinction is that our sparse support is selected once using a training\-free score, and we explicitly compareWanda\-style and magnitude\-only BottomK supports under a matched scalar\-parameter budget\.

For layerii,Suprauses the adapted weight

W^\(i\)=W\(i\)\+M\(i\)⊙U\(i\)\+γi​L\(i\)​R\(i\),\\widehat\{W\}^\{\(i\)\}=W^\{\(i\)\}\+M^\{\(i\)\}\\odot U^\{\(i\)\}\+\\gamma\_\{i\}L^\{\(i\)\}R^\{\(i\)\},\(14\)whereM\(i\)=ψX\(i\)​\(W\(i\),si,ρ\)M^\{\(i\)\}=\\psi\_\{X^\{\(i\)\}\}\(W^\{\(i\)\},s\_\{i\},\\rho\),U\(i\)U^\{\(i\)\}is trainable only on the support ofM\(i\)M^\{\(i\)\},L\(i\)∈ℝci×riL^\{\(i\)\}\\in\\mathbb\{R\}^\{c\_\{i\}\\times r\_\{i\}\},R\(i\)∈ℝri×biR^\{\(i\)\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times b\_\{i\}\}, andγi\\gamma\_\{i\}is the low\-rank scaling factor\. The selection directionρ\\rhogives the TopK or BottomK sparse\-support variant, and the pretrained matrixW\(i\)W^\{\(i\)\}is otherwise frozen\.

TheSupratraining objective optimizes only the adapter variables:

min\{U\(i\),L\(i\),R\(i\)\}i∈𝒜⁡ℒ​\(𝒟;\{W^\(i\)\}i∈𝒜,\{W\(j\)\}j∉𝒜,w¯\),\\min\_\{\\\{U^\{\(i\)\},L^\{\(i\)\},R^\{\(i\)\}\\\}\_\{i\\in\\mathcal\{A\}\}\}\\mathcal\{L\}\\\!\\left\(\\mathcal\{D\};\\\{\\widehat\{W\}^\{\(i\)\}\\\}\_\{i\\in\\mathcal\{A\}\},\\\{W^\{\(j\)\}\\\}\_\{j\\notin\\mathcal\{A\}\},\\bar\{w\}\\right\),\(15\)where𝒜\\mathcal\{A\}is the adapted\-layer index set and the pretrained model parameters are frozen\.

The intuition behind this design stems from the observation thatLoRA, by construction, learns updates in a low\-rank parameterization\. Although theLoRAadapter can be expanded into a full matrix, its implementation uses onlyri​\(ci\+bi\)r\_\{i\}\(c\_\{i\}\+b\_\{i\}\)trainable scalar parameters\. This evolving low\-rank component lets the model adjust task\-specific directions during fine\-tuning\.

In contrast, sparse adaptation bySuperselects a fixed subset of parameters to train\. These parameters are chosen based on their estimated layer\-wise influence under theWanda\-style reconstruction proxy\. Once selected, this subset remains constant during training, resulting in updates restricted to a fixed support\.

By combining these two adapters,Suprabalances two update types: the sparse adapter directly optimizes a fixedWanda\-ordered subset of pretrained weights, while theLoRAcomponent adapts to the target task by learning task\-specific directions in the parameter space\. In essence:

- •The sparse adapter focuses training on a fixed subset of parameters ordered by theWanda\-style reconstruction proxy\.
- •The low\-rank adapter enablestask\-specific adaptationby learning new subspaces of importance in a structured and efficient manner\.

This combination letsSuprainterpolate between sparse and low\-rank adaptation under explicit parameter\-count budgets\. The explicit control over the parameter budget and its division between sparse and low\-rank components enables practitioners to tailor the method to different resource constraints or adaptation scenarios\.

### 3\.4Budget\-Matched Rank Conversion inSupra

Supradetermines the low\-rank adapter rankrir\_\{i\}from a per\-layer parameter\-count budgetTiT\_\{i\}\. Using⌊⋅⌋\\lfloor\\cdot\\rfloorfor the floor function, this paper uses two budget conventions:

Tirank​\(r0\)\\displaystyle T\_\{i\}^\{\\mathrm\{rank\}\}\(r\_\{0\}\)=r0​\(ci\+bi\),\\displaystyle=r\_\{0\}\(c\_\{i\}\+b\_\{i\}\),\(16\)Tidensity​\(p\)\\displaystyle T\_\{i\}^\{\\mathrm\{density\}\}\(p\)=⌊p​ci​bi⌋\.\\displaystyle=\\lfloor pc\_\{i\}b\_\{i\}\\rfloor\.\(17\)The first is the*LoRA\-rank\-equivalent*budget: it givesSuperorSuprathe same number of trainable scalar parameters that a rank\-r0r\_\{0\}LoRAadapter would use in that layer\. This is the budget used in all reported experiments so thatSuperandSupramatch the correspondingLoRAbaseline by number of trainable scalar parameters\. The second is a density\-equivalent sparse budget; it is useful conceptually but is not the convention used for the main comparisons\. A fixed densityppand a fixed rank\-equivalent budget are generally different unless the implied density is layer\-dependent:

pi​\(r0\)=r0​\(ci\+bi\)ci​bi\.p\_\{i\}\(r\_\{0\}\)=\\frac\{r\_\{0\}\(c\_\{i\}\+b\_\{i\}\)\}\{c\_\{i\}b\_\{i\}\}\.\(18\)
To divide a chosen budgetTiT\_\{i\}between the low\-rank and sparse adapters, we introduceλ∈\[0,1\]\\lambda\\in\[0,1\], thelow\-rank budget fraction\. We useλ\\lambdarather thanα\\alphato avoid confusion with the standardLoRAscaling parameterαLoRA\\alpha\_\{\\mathrm\{LoRA\}\}\.

- •Ifλ=1\\lambda=1, the target budget is assigned to the low\-rank component\. UnderTi=Tirank​\(r0\)T\_\{i\}=T\_\{i\}^\{\\mathrm\{rank\}\}\(r\_\{0\}\), this recovers rank\-r0r\_\{0\}LoRA\.
- •Ifλ=0\\lambda=0, the full budget is allocated to the sparse adapter, resulting in the sparse\-onlySupervariant\.
- •Ifλ=0\.5\\lambda=0\.5, the target budget is approximately split between low\-rank and sparse adapters\.

Under this scheme, the low\-rank adapter receives approximatelyλ​Ti\\lambda T\_\{i\}trainable scalar parameters, while the sparse adapter is assigned the remaining parameter\-count budget\.

Unlike sparse adaptation, which directly operates on a fixed number of parameters, low\-rank adaptation requires specifying a rankrir\_\{i\}\. To bridge this gap, we define the following budget\-matched rule for selectingrir\_\{i\}based onλ\\lambdaandTiT\_\{i\}:

ri=⌊λ​Tici\+bi⌋\.r\_\{i\}=\\left\\lfloor\\frac\{\\lambda T\_\{i\}\}\{c\_\{i\}\+b\_\{i\}\}\\right\\rfloor\.\(19\)
This formulation ensures that the low\-rank adapter uses

li=ri​\(ci\+bi\)l\_\{i\}=r\_\{i\}\(c\_\{i\}\+b\_\{i\}\)\(20\)parameters, and the sparse adapter is assigned the remaining

si=Ti−lis\_\{i\}=T\_\{i\}\-l\_\{i\}\(21\)parameters\. Thereforesi≥0s\_\{i\}\\geq 0, and the total number of trainable parameters remains fixed:si\+li=Tis\_\{i\}\+l\_\{i\}=T\_\{i\}\.

Under the rank\-equivalent budget used in the main experiments,Ti=Tirank​\(r0\)=r0​\(ci\+bi\)T\_\{i\}=T\_\{i\}^\{\\mathrm\{rank\}\}\(r\_\{0\}\)=r\_\{0\}\(c\_\{i\}\+b\_\{i\}\), so the conversion simplifies to

ri=⌊λ​r0⌋,si=\(r0−⌊λ​r0⌋\)​\(ci\+bi\)\.r\_\{i\}=\\lfloor\\lambda r\_\{0\}\\rfloor,\\qquad s\_\{i\}=\\bigl\(r\_\{0\}\-\\lfloor\\lambda r\_\{0\}\\rfloor\\bigr\)\(c\_\{i\}\+b\_\{i\}\)\.\(22\)
Due to integer rank constraints in \([19](https://arxiv.org/html/2607.09287#S3.E19)\), the realized low\-rank ratio may deviate slightly from the target valueλ\\lambda\. In particular,li/\(li\+si\)l\_\{i\}/\(l\_\{i\}\+s\_\{i\}\)need not equalλ\\lambda\. This discrepancy arises from the floor operation, which ensures thatrir\_\{i\}remains an integer and that the low\-rank component never exceeds the assigned budget\. Under a density budgetTidensity​\(p\)T\_\{i\}^\{\\mathrm\{density\}\}\(p\),λ=1\\lambda=1may leave a sparse remainder unless that remainder is explicitly discarded; under the rank\-equivalent budgetTirank​\(r0\)T\_\{i\}^\{\\mathrm\{rank\}\}\(r\_\{0\}\),λ=1\\lambda=1givessi=0s\_\{i\}=0and recovers rank\-r0r\_\{0\}LoRA\.

## 4Experiments

### 4\.1Experimental Setup

##### Datasets

The main experiments use theMath17Kinstruction\-tuning dataset\. We split the fine\-tuning data into training and validation subsets and use the held\-out validation split only for learning\-rate and checkpoint selection\.

To the best of our knowledge and according to the dataset construction used in our experiments, we use Math17K rather than the earlier Math10K setup because Math10K contained examples overlapping with some of our evaluation benchmarks\. The Math17K split used in this paper excludes examples from AddSub, MultiArith, SingleEq, GSM8K, AQuA, and SVAMP, so the evaluation benchmarks are not part of the fine\-tuning data\.

We evaluate on six arithmetic\-reasoning benchmarks: AddSub\(Hosseiniet al\.,[2014](https://arxiv.org/html/2607.09287#bib.bib97)\), MultiArith\(Roy and Roth,[2016](https://arxiv.org/html/2607.09287#bib.bib96)\), SingleEq\(Koncel\-Kedziorskiet al\.,[2015](https://arxiv.org/html/2607.09287#bib.bib98)\), GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.09287#bib.bib92)\), AQuA\(Linget al\.,[2017](https://arxiv.org/html/2607.09287#bib.bib93)\), and SVAMP\(Patelet al\.,[2021](https://arxiv.org/html/2607.09287#bib.bib95)\)\. Thus the main protocol is task\-specific arithmetic fine\-tuning followed by arithmetic evaluation\.

##### Training Details

Except for the full\-parameter reference rows, all reported trainable methods use adapter\-based fine\-tuning, where the pretrained model remains frozen and only adapter parameters are optimized\. The reported experiments use two Llama\-family models:Llama\-3\.2\-1BandMeta\-Llama\-3\-8B\. For adapter methods, we adapt all standard linear projections in each Llama block:

q\_proj,k\_proj,v\_proj,o\_proj,gate\_proj,up\_proj,down\_proj\.\\texttt\{q\\\_proj\},\\ \\texttt\{k\\\_proj\},\\ \\texttt\{v\\\_proj\},\\ \\texttt\{o\\\_proj\},\\ \\texttt\{gate\\\_proj\},\\ \\texttt\{up\\\_proj\},\\ \\texttt\{down\\\_proj\}\.This is the full set of linear modules adapted in all reported adapter experiments\.

Compared methods\.We compareLoRA,SIFT\(TopK\),SIFT\(RandK\),RoSA,Super\(TopK\),Super\(BottomK\),Super\(RandK\),Magnitude\(TopK/BottomK\), andSuprawithλ∈\{0\.3,0\.5,0\.8\}\\lambda\\in\\\{0\.3,0\.5,0\.8\\\}under bothWanda\-selected and magnitude\-selected sparse supports\. ForSupra,λ\\lambdais the low\-rank budget fraction defined in[Section˜3\.3](https://arxiv.org/html/2607.09287#S3.SS3); in method names, TopK denotes selecting the largest scores and BottomK denotes selecting the smallest scores under the same budget\. TheMagnituderows are PaFi\-style magnitude\-mask sparse fine\-tuning baselines using the score\|Wi​j\|\|W\_\{ij\}\|, whileSuperand the defaultSuprarows use the activation\-weightedWandascore\. The Math17K one\- and three\-epoch variant grids for these methods, including accuracy and perplexity, are reported in the appendix; the hybrid TopK–BottomK ablation is collected separately in[Section˜C\.1](https://arxiv.org/html/2607.09287#A3.SS1)\. Each table row also lists the validation\-selected learning rate\.

Parameter budgets\.The main run uses the rank\-equivalent budgetr0=8r\_\{0\}=8\. For a baselineLoRArankr0r\_\{0\}, the target per\-layer parameter\-count budget isTirank​\(r0\)=r0​\(ci\+bi\)T\_\{i\}^\{\\mathrm\{rank\}\}\(r\_\{0\}\)=r\_\{0\}\(c\_\{i\}\+b\_\{i\}\), matching the number of trainable scalar parameters in rank\-r0r\_\{0\}LoRA\. Sparse methods train the same number of scalar entries in the corresponding layer, andSuprasplits the same numerical budget between sparse and low\-rank components\. Due to integer ranks and method\-specific implementation details, the realized trainable\-parameter counts may differ slightly, but all adapter methods use approximately the same model\-specific rank\-8 budget:5\.65\.6M trainable parameters forLlama\-3\.2\-1Band21\.021\.0M forMeta\-Llama\-3\-8B\. Our comparisons match trainable scalar counts across adapter methods\. Implementation\-level memory, checkpoint\-size, calibration\-time, and throughput measurements are reported separately in[Section˜4\.2](https://arxiv.org/html/2607.09287#S4.SS2); matching trainable scalar counts alone does not imply identical runtime behavior\.

Learning\-rate selection\.Because different PEFT parameterizations can have different optimization scales, we tune the learning rate independently for each method using the same candidate grid:

5⋅10−5,10−4,5⋅10−4,10−3,5⋅10−3,10−2,5⋅10−2,10−1\.5\\cdot 10^\{\-5\},\\ 10^\{\-4\},\\ 5\\cdot 10^\{\-4\},\\ 10^\{\-3\},\\ 5\\cdot 10^\{\-3\},\\ 10^\{\-2\},\\ 5\\cdot 10^\{\-2\},\\ 10^\{\-1\}\.Each method is trained at all candidate learning rates, and a single learning rate is selected for that method using the held\-out Math17K validation negative log\-likelihood\. The reported test accuracies and perplexities for a method all come from this one validation\-selected learning rate; we do not select learning rates using test benchmark accuracy\.

Optimization and calibration\.In the Math17K comparisons, we fine\-tune each candidate run for either one or three epochs, with batch size 16, maximum sequence length 256, and 100 warmup steps\. Training uses Adam with weight decay 0 under the standard causal language\-modeling objective\. Unless stated otherwise,LoRAusesαLoRA=16\\alpha\_\{\\mathrm\{LoRA\}\}=16, dropout 0\.05, and no bias training\. The compact main\-text tables report, for each method, the better of its one\- and three\-epoch runs by average benchmark accuracy; the appendix keeps the schedules separate\. ForSuperand the defaultSupravariants, theWanda\-style sparse masks are computed from 128 C4 calibration samples\. The magnitude\-only variants do not use a calibration pass; they rank candidate sparse coordinates by\|Wi​j\|\|W\_\{ij\}\|\. When a sparse support is used, it is fixed before fine\-tuning and only the selected sparse entries are optimized during Math17K fine\-tuning\.[Figure˜2](https://arxiv.org/html/2607.09287#S4.F2)shows the training and held\-out validation loss curves for theMeta\-Llama\-3\-8Bthree\-epoch Math17K runs, illustrating the overall optimization behavior under this protocol and the validation\-loss signal used for learning\-rate selection\. To check whether in\-domain input statistics materially change the selected sparse support, we also ran a targeted math\-calibrated ablation forLlama\-3\.2\-1Bon Math7K: the fine\-tuning setup is fixed, while the sparse\-mask calibration source is changed from C4 to Math7K\. The per\-benchmark results in[Table˜4](https://arxiv.org/html/2607.09287#S4.T4)show that this calibration change has only a small effect in that one\-epoch setting\.

Table 4:Calibration\-source ablation for sparse\-onlySuper\. Model:Llama\-3\.2\-1B; fine\-tuning dataset: Math7K; training budget: one epoch; rank\-equivalent budget:r0=8r\_\{0\}=8\. The Base row is frozen\. C4 rows use the default C4 calibration samples for theWanda\-style score; Math7K rows compute the same score on Math7K calibration samples\. Trainable rows use approximately5\.65\.6M trainable parameters and the validation\-selected learning rate\. Results are exact\-answer accuracy \(%\)\.MethodCalibrationSelected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––13\.674\.6721\.462\.8121\.6511\.6012\.64Super\(TopK\)C45⋅10−45\\cdot 10^\{\-4\}35\.9563\.8348\.6221\.4621\.6529\.0036\.75Super\(TopK\)Math7K5⋅10−45\\cdot 10^\{\-4\}38\.2361\.8348\.6222\.0620\.0829\.6036\.74Super\(BottomK\)C410−310^\{\-3\}40\.7659\.3352\.5621\.7617\.3232\.3037\.34Super\(BottomK\)Math7K10−310^\{\-3\}41\.0157\.8351\.3820\.1720\.0832\.2037\.11

![Refer to caption](https://arxiv.org/html/2607.09287v1/x2.png)

![Refer to caption](https://arxiv.org/html/2607.09287v1/x3.png)

Figure 2:Optimization curves forMeta\-Llama\-3\-8BMath17K fine\-tuning over three epochs\. Left: training loss, plotted with a log\-scaledyy\-axis\. Right: held\-out Math17K validation loss\. Each curve uses the validation\-selected learning rate for the corresponding method under the rank\-equivalentr0=8r\_\{0\}=8budget\.Evaluation metric\.We report exact\-answer accuracy on each math benchmark and the arithmetic mean over benchmarks\. For generation\-based accuracy, we use the arithmetic evaluation pipeline to extract either the numeric answer or the answer option letter for AQuA\. Perplexity is reported only as an auxiliary language\-modeling diagnostic and is not used for the main arithmetic conclusion\. Exact\-answer accuracy is the primary metric\. Lower perplexity does not necessarily imply higher benchmark accuracy because the language\-modeling loss can be dominated by non\-answer tokens, while the benchmark metric depends only on the extracted final numeric answer or multiple\-choice option\. In our evaluation pipeline, perplexity is computed over the gold target text appended after the prompt, with prompt tokens masked from the loss; the target may include rationale or answer text depending on the dataset record, plus the end\-of\-sequence token\. Per\-dataset perplexity is the exponentiated token\-weighted mean negative log\-likelihood over up to 120 examples per benchmark, and the reported average is an aggregate diagnostic over the same benchmark set\.

Hardware and reproducibility\.The main experiment is run on the KAUST ORIX cluster using 4 NVIDIA H200 GPUs with 140GB of memory per GPU\. The implementation is available at[https://github\.com/vectozavr/SuperTuning](https://github.com/vectozavr/SuperTuning)\. The launch pipeline exposes the random seed, logs all adapter runs to Weights & Biases, records the selected learning rate, and writes machine\-readable result files for table construction\. Repeating the full benchmark suite across multiple random seeds would be computationally expensive, so the reported tables use a single controlled seed \(seed 0\); estimating multi\-seed uncertainty is left for future work\.

### 4\.2Efficiency Measurements

Although all adapter methods are matched by trainable scalar count, this does not automatically imply identical memory or runtime behavior\. We therefore report implementation\-level efficiency measurements for representative Math17K profiling runs in[Table˜5](https://arxiv.org/html/2607.09287#S4.T5)\. The profiling setup uses the same models, adapted modules, rank\-equivalent budgetr0=8r\_\{0\}=8, batch size 16, micro\-batch size 16, maximum sequence length 256, C4 calibration source forWanda\-style masks, and NVIDIA H200 hardware as the main experiments\. To keep the measurement inexpensive, each row runs 20 optimizer steps with learning rate5⋅10−45\\cdot 10^\{\-4\}; these runs are used only for efficiency profiling and do not add accuracy baselines\.

These measurements should be interpreted as costs of our current implementation rather than theoretical sparse\-kernel speedups\. The optimizer\-state column reports measured Adam tensor storage, not a normalized two\-FP32\-state estimate: for full fine\-tuning the model parameters are bf16 in these runs, so the two Adam moment tensors occupy approximately the same memory as one FP32 tensor per trainable parameter\. TheSuper,Magnitude,Supra, andSupra\-Magimplementations store sparse trainable values and integer index buffers, so their measured optimizer\-state storage is sparse\. However, the current sparse linear layer forms a dense effective weight for the matrix multiply and computes a dense gradient matrix before gathering the selected sparse coordinates\. Thus, sparse trainable\-parameter counts reduce adapter checkpoint size and measured optimizer\-state storage, but do not by themselves guarantee wall\-clock speedups\.SIFTalso stores sparse trainable values, but its selection/update path requires dense gradients; this is reflected in its peak\-memory measurements\. Calibration time is reported only for methods with a separate C4 mask\-construction pass; methods whose sparse support is magnitude\-only, low\-rank\-only, full fine\-tuning, or selected inside training are marked “–”\.

Table 5:Implementation\-level efficiency measurements for Math17K profiling runs\. Both models use rank\-equivalent budgetr0=8r\_\{0\}=8, batch size 16, micro\-batch size 16, and 20 optimizer steps on a single NVIDIA H200 GPU\. Checkpoint size is the saved full\-model checkpoint for full fine\-tuning and the saved adapter state otherwise\. Peak memory is peak CUDA allocated memory during the measured training region\. Adam state is measured directly from tensor storage present in the instantiated optimizer after the profiling steps; it reflects the actual parameter dtypes and optimizer\-state representation used by each method rather than a uniform two\-FP32\-state estimate\. In particular, full fine\-tuning stores two bf16 Adam moment tensors in these runs, so its state occupies approximately the same memory as one FP32 tensor per trainable parameter\.ModelMethodTrainable ParamsCkpt\. SizeCalib\. TimePeak Mem\.Adam StateSteps/sTokens/sWall TimeLlama\-3\.2\-1BFull FT1\.24B2\.32 GiB–20\.88 GiB4\.60 GiB5\.688178193\.5 sRoSA5\.64M5\.4 MiB–17\.98 GiB10\.8 MiB4\.368136854\.6 sSIFT\(TopK\)5\.64M107\.6 MiB–21\.28 GiB43\.0 MiB7\.073221602\.8 sLoRA5\.64M21\.6 MiB–18\.04 GiB43\.0 MiB5\.808181973\.4 sSuper\(BottomK\)5\.64M43\.1 MiB2\.8 s15\.81 GiB43\.0 MiB7\.933248552\.5 sMagnitude\(BottomK\)5\.64M43\.1 MiB–15\.81 GiB43\.0 MiB8\.287259622\.4 sSupra\(BottomK,λ=0\.8\\lambda=0\.8\)5\.64M43\.1 MiB2\.8 s19\.55 GiB43\.0 MiB5\.831182693\.4 sSupra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)5\.64M43\.1 MiB–19\.54 GiB43\.0 MiB5\.743179933\.5 sMeta\-Llama\-3\-8BFull FT8\.03B14\.97 GiB–77\.91 GiB29\.92 GiB1\.706534411\.7 sRoSA20\.97M20\.1 MiB–54\.72 GiB40\.0 MiB1\.850579510\.8 sSIFT\(TopK\)20\.97M400\.1 MiB–78\.50 GiB160\.0 MiB1\.874587010\.7 sLoRA20\.97M80\.2 MiB–54\.90 GiB160\.0 MiB2\.04464039\.8 sSuper\(BottomK\)20\.97M160\.1 MiB11\.3 s46\.22 GiB160\.0 MiB2\.25270558\.9 sMagnitude\(BottomK\)20\.97M160\.1 MiB–46\.22 GiB160\.0 MiB2\.24970468\.9 sSupra\(BottomK,λ=0\.8\\lambda=0\.8\)20\.97M160\.3 MiB11\.2 s60\.46 GiB160\.0 MiB1\.716537711\.7 sSupra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)20\.97M160\.3 MiB–60\.44 GiB160\.0 MiB1\.689529111\.8 s

### 4\.3Comparison with other fine\-tuning methods

This subsection discusses the rank\-equivalentr0=8r\_\{0\}=8arithmetic results shown earlier in[Tables˜1](https://arxiv.org/html/2607.09287#S1.T1)and[2](https://arxiv.org/html/2607.09287#S1.T2)\. Each adapter method uses approximately the same rank\-equivalent trainable\-parameter budget, and each reported row uses the learning rate selected by held\-out Math17K validation loss for the corresponding epoch budget\. The compact tables select the better one\- or three\-epoch run for each method by average benchmark accuracy; forSuperandSupra, they also select the best observed variant from the corresponding support and budget\-split grid\. Because the schedule\-selected compact rows for our proposed methods use BottomK supports, we use BottomK as the default support forSuperandSupraunless stated otherwise\. The complete per\-benchmark accuracy and perplexity tables are reported in[Sections˜C\.2\.2](https://arxiv.org/html/2607.09287#A3.SS2.SSS2),[C\.2\.3](https://arxiv.org/html/2607.09287#A3.SS2.SSS3),[C\.3\.2](https://arxiv.org/html/2607.09287#A3.SS3.SSS2)and[C\.3\.3](https://arxiv.org/html/2607.09287#A3.SS3.SSS3)\.

For the 1B model,Supra\(BottomK,λ=0\.8\\lambda=0\.8\) reaches 62\.23% average exact\-answer accuracy, the strongest observed adapter result in the comparison\. The strongest external baseline isLoRAat 61\.07%, soSupraachieves a higher observed average by 1\.16 percentage points\. The sparse\-onlySuper\(BottomK\) row reaches 55\.46%, belowLoRAbut above the corresponding TopK sparse row in the full appendix table\.

For the 8B model, the strongest schedule\-selected result is obtained bySupra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\), which reaches 79\.12% average accuracy\. The magnitude\-only sparse baseline is essentially tied at 79\.02%, while the strongestWanda\-selectedSuprarow reaches 78\.66%\. Thus, the 8B results should be interpreted less as evidence that activation weighting is always superior and more as evidence that low\-score pretrained\-weight supports, especially low\-magnitude supports, are highly effective in this setting\. Combining such supports withLoRAcan preserve or slightly improve performance under the same scalar\-parameter budget\. Relative to external baselines,Supra\-Magachieves a higher observed average thanSIFT\(RandK\) by 1\.29 points and thanLoRAby 5\.95 points\.

Full fine\-tuning is included as an unbudgeted full\-parameter reference rather than as a PEFT baseline, since it updates all pretrained parameters\. We do not interpret these full fine\-tuning rows as fully optimized upper bounds; they are protocol\-matched full\-parameter reference runs using the same validation\-based learning\-rate selection framework\. OnLlama\-3\.2\-1B, full fine\-tuning reaches 59\.77% average accuracy after one epoch and 65\.70% after three epochs, while the best rank\-8\-equivalent adapter reaches 59\.69% and 62\.23% under the corresponding one\- and three\-epoch budgets\. On the 8B model, several adapter configurations achieve higher average accuracy than the full\-parameter reference\. We therefore use full fine\-tuning only as a reference point for the cost and behavior of updating all parameters under the same training setup\.

## 5Discussion

The main empirical conclusion is that fixed sparse adaptation supports can be highly effective for parameter\-efficient fine\-tuning\. Under the same rank\-equivalent trainable\-parameter budget and the same validation\-based learning\-rate selection protocol, the strongest sparse and sparse–low\-rank variants achieve the best observed average arithmetic accuracy in the single\-seed, schedule\-selected compact comparisons on both tested model sizes\. On the 1B model,Supra\(BottomK,λ=0\.8\\lambda=0\.8\) reaches 62\.23% average accuracy, a higher observed average than the strongest external baseline,LoRA, by 1\.16 percentage points\. On the 8B model,Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\) reaches 79\.12%, nearly tied with the magnitude\-only sparse baseline at 79\.02% and 1\.29 points above the strongest external baseline,SIFT\(RandK\)\. These gains are obtained with simple fixed mask construction and without gradient or curvature\-based mask search\.

The strongest schedule\-selected sparse supports usually come from the low\-score end of the ordering\. This pattern appears for both theWanda\-style activation\-weighted score and the PaFi\-style magnitude\-only score\. However, the best direction is not universal: in some fixed\-schedule sparse\-only runs, TopK or mixed TopK–BottomK supports can outperform pure BottomK\. We therefore view BottomK as a strong empirical default in our setting rather than as an optimal selection rule\.

The hybridSupraresults further suggest that sparse and low\-rank adaptation can be complementary under a fixed parameter budget, but the best allocation is not universal\.Suprais strongest among the proposed methods on the 1B model, whileSupra\-Magis strongest on the 8B model and is essentially tied with the magnitude\-only sparse baseline\. This suggests that the best allocation between sparse updates and low\-rank updates may depend on model scale, optimization budget, and task distribution, but the overall pattern is encouraging: sparse updates are competitive with standard PEFT baselines in these single\-seed comparisons, and combining them with low\-rank adapters can preserve or improve performance at the same trainable\-parameter count\.

Overall, our results support a practical view ofSuperandSupra: simple low\-score orderings can identify useful sparse adaptation coordinates\. A cheap activation\-based calibration pass provides one such ordering through theWanda\-style score, while magnitude\-only ordering is a strong PaFi\-style alternative\. More broadly, the experiments suggest that pruning\-inspired signals can be repurposed for fine\-tuning\.

## 6Limitations

Our study has several limitations that we hope future work will address\.

##### Single\-seed uncertainty\.

All main tables use a single controlled seed\. This is common in expensive LLM fine\-tuning studies but means that small differences between methods should be interpreted as observed single\-seed gaps rather than statistically established improvements\. Multi\-seed uncertainty estimates are left for future work\.

##### Schedule\-selected summaries\.

The compact main\-text tables select the better one\- or three\-epoch schedule for each method family and, forSuper/Supra, the best reported variant\. The appendix provides the fixed\-schedule grids\. The compact tables should therefore be interpreted as best\-observed schedule\-selected summaries rather than universal dominance claims\.

##### Scope of evaluation\.

The quantitative comparisons in this version focus on arithmetic\-reasoning evaluation withLlama\-3\.2\-1BandMeta\-Llama\-3\-8Bafter Math17K fine\-tuning\. This is a limited slice of the broad application space of LLMs\. In particular, 70B\-scale models, other architecture families, and language\-generation tasks that hinge on open\-ended semantics \(e\.g\., summarization, dialogue, or code synthesis\) may stress different model components than symbolic math reasoning\. The 8B results in[Section˜C\.3\.3](https://arxiv.org/html/2607.09287#A3.SS3.SSS3)are useful as a larger\-checkpoint check, but should not be read as a scaling law\.

##### Dependence on theWandasaliency proxy\.

Our new algorithmSuper\(and hence the defaultSupra\) selects trainable weights using an activation\-weighted magnitude heuristic inherited from theWandapruning metric\. Although this choice is fast and training\-free, it is still an*approximation*of true importance; weights that appear non\-salient under the proxy may in fact become critical once the task distribution shifts or once other parameters are updated\. The magnitude\-only baselines show that removing activation weighting can also work well in some settings\. We therefore do not claim that the current mask\-selection rule is optimal, or even that a pruning\-saliency score should always be used in the same direction for fine\-tuning\. Further ablations against activation\-only masks, row\-wise variants, alternative calibration sets, and more expressive but costlier criteria \(e\.g\. curvature\-aware saliency or gradient\-flow statistics\) remain important\.

##### Fixed sparse support\.

The sparse component ofSuprais*static*throughout fine\-tuning: once a weight is deemed non\-trainable, it can never be activated\. Dynamic sparsity—allowing the mask to grow, prune, or re\-allocate budget during training—has shown promise in other contexts and may close the gap to full fine\-tuning in edge cases where the initial selection is sub\-optimal\.

##### System efficiency\.

The profiling results in[Section˜4\.2](https://arxiv.org/html/2607.09287#S4.SS2)measure our current implementation, not an optimized sparse\-kernel system\. They use short fixed\-step runs, so they are useful for comparing implementation\-level memory, checkpoint size, calibration cost, and local throughput under the same setup, but they should not be interpreted as asymptotic runtime claims\. Additional sparse\-kernel engineering, sparse optimizer implementations, or different batching regimes could change the wall\-clock profile\.

##### Layer\-local parameter budgets\.

Our budget\-matched rank rule \([Equation˜19](https://arxiv.org/html/2607.09287#S3.E19)\) applies the same low\-rank budget fractionλ\\lambdaindependently to every adapted linear layer\. In practice, different layers contribute unequally to downstream task performance; early attention blocks, for example, often admit more aggressive compression than middle MLP layers\. A global, data\-driven re\-allocation of the parameter budget—akin to neural architecture search—might improve efficiency further\.

## Acknowledgments and Disclosure of Funding

The research reported in this publication was supported by funding from King Abdullah University of Science and Technology \(KAUST\): i\) KAUST Baseline Research Scheme, ii\) CRG Grant ORFS\-CRG12\-2024\-6460, and iii\) Center of Excellence for Generative AI, under award number 5940\.

## References

- Scaling sparse fine\-tuning to large language models\.arXiv preprint arXiv:2401\.16405\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.09287#S2.SS2.p1.3)\.
- A\. Choudhary, Y\. Sulaıman, L\. Mauch, G\. Boukli Hacene, F\. Cardinaux, and A\. Bosselut \(2025\)GaLLoP: gradient\-based sparse learning on low\-magnitude parameters\.arXiv preprint arXiv:2510\.19778\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2510.19778)Cited by:[§2\.2](https://arxiv.org/html/2607.09287#S2.SS2.p3.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.1](https://arxiv.org/html/2607.09287#S4.SS1.SSS0.Px1.p3.1)\.
- A\. Han, J\. Li, W\. Huang, M\. Hong, A\. Takeda, P\. Jawanpuria, and B\. Mishra \(2024\)SLTrain: a sparse plus low\-rank approach for parameter and memory efficient pretraining\.arXiv preprint arXiv:2406\.02214\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.09287#S2.SS3.p1.1)\.
- M\. J\. Hosseini, H\. Hajishirzi, O\. Etzioni, and N\. Kushman \(2014\)Learning to solve arithmetic word problems with verb categorization\.InProceedings of the 2014 conference on empirical methods in natural language processing \(EMNLP\),pp\. 523–533\.Cited by:[§4\.1](https://arxiv.org/html/2607.09287#S4.SS1.SSS0.Px1.p3.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2021\)Lora: low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p2.1),[§3\.1](https://arxiv.org/html/2607.09287#S3.SS1.SSS0.Px1.p1.1)\.
- R\. Koncel\-Kedziorski, H\. Hajishirzi, A\. Sabharwal, O\. Etzioni, and S\. D\. Ang \(2015\)Parsing algebraic word problems into equations\.Transactions of the Association for Computational Linguistics3,pp\. 585–597\.Cited by:[§4\.1](https://arxiv.org/html/2607.09287#S4.SS1.SSS0.Px1.p3.1)\.
- M\. Kowsher, T\. Esmaeilbeig, C\. Yu, C\. Chen, M\. Soltanalian, and N\. Yousefi \(2025\)Rocoft: efficient finetuning of large language models with row\-column updates\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 26659–26678\.Cited by:[§2\.5](https://arxiv.org/html/2607.09287#S2.SS5.p1.1)\.
- C\. Li and A\. Bhaskara \(2025\)An efficient sparse fine\-tuning with low quantization error via neural network pruning\.arXiv preprint arXiv:2502\.11439\.Cited by:[§2\.5](https://arxiv.org/html/2607.09287#S2.SS5.p1.1)\.
- B\. Liao, Y\. Meng, and C\. Monz \(2023\)Parameter\-efficient fine\-tuning without introducing new latency\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4242–4260\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p3.1.5),[§2\.2](https://arxiv.org/html/2607.09287#S2.SS2.p2.4),[§2\.5](https://arxiv.org/html/2607.09287#S2.SS5.p1.1),[§3\.2](https://arxiv.org/html/2607.09287#S3.SS2.p4.4)\.
- W\. Ling, D\. Yogatama, C\. Dyer, and P\. Blunsom \(2017\)Program induction by rationale generation: learning to solve and explain algebraic word problems\.arXiv preprint arXiv:1705\.04146\.Cited by:[§4\.1](https://arxiv.org/html/2607.09287#S4.SS1.SSS0.Px1.p3.1)\.
- S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen \(2024\)DoRA: weight\-decomposed low\-rank adaptation\.arXiv preprint arXiv:2402\.09353\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p4.1)\.
- X\. Liu, A\. Thomas, C\. Zhang, J\. Cheng, Y\. Zhao, and X\. Gao \(2025\)Refining salience\-aware sparse fine\-tuning strategies for language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 31932–31945\.External Links:[Link](https://aclanthology.org/2025.acl-long.1541/)Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p3.1)\.
- D\. Ma, L\. Chen, P\. Wang, H\. Xu, H\. Li, L\. Sun, S\. Zhu, S\. Fan, and K\. Yu \(2024\)Sparsity\-accelerated training for large language models\.arXiv preprint arXiv:2406\.01392\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.09287#S2.SS2.p1.3)\.
- M\. Nikdan, S\. Tabesh, E\. Crnčević, and D\. Alistarh \(2024\)Rosa: accurate parameter\-efficient fine\-tuning via robust adaptation\.arXiv preprint arXiv:2401\.04679\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p3.1),[§1](https://arxiv.org/html/2607.09287#S1.p4.1),[§1](https://arxiv.org/html/2607.09287#S1.p6.1),[§2\.3](https://arxiv.org/html/2607.09287#S2.SS3.p1.1),[§3\.3](https://arxiv.org/html/2607.09287#S3.SS3.p1.1)\.
- A\. Patel, S\. Bhattamishra, and N\. Goyal \(2021\)Are nlp models really able to solve simple math word problems?\.arXiv preprint arXiv:2103\.07191\.Cited by:[§4\.1](https://arxiv.org/html/2607.09287#S4.SS1.SSS0.Px1.p3.1)\.
- S\. Roy and D\. Roth \(2016\)Solving general arithmetic word problems\.arXiv preprint arXiv:1608\.01413\.Cited by:[§4\.1](https://arxiv.org/html/2607.09287#S4.SS1.SSS0.Px1.p3.1)\.
- W\. Song, Z\. Li, L\. Zhang, H\. Zhao, and B\. Du \(2023\)Sparse is enough in fine\-tuning pre\-trained large language models\.arXiv preprint arXiv:2312\.11875\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.09287#S2.SS2.p1.3)\.
- M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. Kolter \(2023\)A simple and effective pruning approach for large language models\.arXiv preprint arXiv:2306\.11695\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p5.1),[§2\.5](https://arxiv.org/html/2607.09287#S2.SS5.p3.1)\.
- Y\. Sung, V\. Nair, and C\. A\. Raffel \(2021\)Training neural networks with fixed sparse masks\.Advances in Neural Information Processing Systems34,pp\. 24193–24205\.Cited by:[§2\.5](https://arxiv.org/html/2607.09287#S2.SS5.p1.1),[§3\.1](https://arxiv.org/html/2607.09287#S3.SS1.SSS0.Px2.p1.1)\.
- H\. Xu, R\. Zhan, D\. F\. Wong, and L\. S\. Chao \(2024\)Let’s focus on neuron: neuron\-level supervised fine\-tuning for large language model\.arXiv preprint arXiv:2403\.11621\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p3.1),[§2\.4](https://arxiv.org/html/2607.09287#S2.SS4.p1.1)\.
- X\. Yang, J\. Leng, G\. Guo, J\. Zhao, R\. Nakada, L\. Zhang, H\. Yao, and B\. Chen \(2024\)S2FT: efficient, scalable and generalizable llm fine\-tuning by structured sparsity\.arXiv preprint arXiv:2412\.06289\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.09287#S2.SS2.p1.3)\.
- Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao \(2023\)Adaptive budget allocation for parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2303\.10512\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p4.1)\.
- M\. Zhelnin, V\. Moskvoretskii, E\. Shvetsov, E\. Venediktov, M\. Krylova, A\. Zuev, and E\. Burnaev \(2024\)GIFT\-sw: gaussian noise injected fine\-tuning of salient weights for llms\.arXiv preprint arXiv:2408\.15300\.Cited by:[§1](https://arxiv.org/html/2607.09287#S1.p3.1),[§2\.4](https://arxiv.org/html/2607.09287#S2.SS4.p1.1)\.

## Appendix ATable of Frequently Used Notation

Table 6:Notation table𝒩\\mathcal\{N\}–Pretrained language model\.mm–Number of candidate fully connected matrices considered for adaptation\.𝒲=\{W\(i\)\}i=1m\\mathcal\{W\}=\\\{W^\{\(i\)\}\\\}\_\{i=1\}^\{m\}–Candidate fully connected weight matrices considered for adaptation\.W\(i\)W^\{\(i\)\}–Weight matrixii,W\(i\)∈ℝci×biW^\{\(i\)\}\\in\\mathbb\{R\}^\{c\_\{i\}\\times b\_\{i\}\}, wherecic\_\{i\}andbib\_\{i\}are output and input dimensions\.w¯\\bar\{w\},δ¯\\bar\{\\delta\},d¯\\bar\{d\}–Remaining model parametersw¯∈ℝd¯\\bar\{w\}\\in\\mathbb\{R\}^\{\\bar\{d\}\}, their optional perturbationδ¯∈ℝd¯\\bar\{\\delta\}\\in\\mathbb\{R\}^\{\\bar\{d\}\}, and their dimension\.𝒟\\mathcal\{D\},ℒ\\mathcal\{L\}–Fine\-tuning dataset and training loss\.𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}–Tokenized calibration set used to collect activations for sparse\-mask construction\.𝒜\\mathcal\{A\}–Adapted\-layer index set,𝒜⊆\{1,…,m\}\\mathcal\{A\}\\subseteq\\\{1,\\ldots,m\\\}\.Δ\\Delta,Δ\(i\)\\Delta^\{\(i\)\}–Collection of additive perturbations and the perturbation applied toW\(i\)W^\{\(i\)\}\.𝒞​\(Δ,δ¯\)\\mathcal\{C\}\(\\Delta,\\bar\{\\delta\}\)–Constraint set defining the adapter structure, such as low\-rank or sparse updates\.W^\(i\)\\widehat\{W\}^\{\(i\)\}–Adapted weight matrix after adding sparse and/or low\-rank updates\.XX,X\(i\)X^\{\(i\)\}–Calibration activations for a generic linear layer or layerii;X∈ℝb×aX\\in\\mathbb\{R\}^\{b\\times a\}, whereaais the number of calibration token positions\.Xj:X\_\{j:\},Xq:X\_\{q:\},Xr:X\_\{r:\}–Rows ofXX;j,q,rj,q,rdenote input\-coordinate indices\.Wi​jW\_\{ij\},Wk​qW\_\{kq\},Wk​q\(i\)W^\{\(i\)\}\_\{kq\}–Entries of a generic weight matrix or of layerii;k,qk,qdenote row and column indices in theWandascore\.Gi​jG\_\{ij\}–Gradient\-magnitude score for sparse selection,Gi​j=\|∂ℒ/∂Wi​j\|G\_\{ij\}=\|\\partial\\mathcal\{L\}/\\partial W\_\{ij\}\|\.Bi​jB\_\{ij\}–PaFi\-style pretrained\-magnitude score,Bi​j=\|Wi​j\|B\_\{ij\}=\|W\_\{ij\}\|\.Ai​jA\_\{ij\}–Activation\-weighted magnitude score used for mask selection,Ai​j=\|Wi​j\|​‖Xj:‖2A\_\{ij\}=\|W\_\{ij\}\|\\\|X\_\{j:\}\\\|\_\{2\}\.Sk​qS\_\{kq\}–Wandasaliency score for entryWk​qW\_\{kq\}\.ρ\\rho–Sparse\-support selection direction,ρ∈\{TopK,BottomK\}\\rho\\in\\\{\\mathrm\{TopK\},\\mathrm\{BottomK\}\\\}\.ψX​\(W,s,ρ\)\\psi\_\{X\}\(W,s,\\rho\)≔\\coloneqqBinary mask in\{0,1\}c×b\\\{0,1\\\}^\{c\\times b\}selecting either the global topssor global bottomssentries ofWW, according toρ\\rho, ordered byAi​jA\_\{ij\}\.M\(i\)M^\{\(i\)\}–Binary sparse mask for layerii,M\(i\)=ψX\(i\)​\(W\(i\),si,ρ\)M^\{\(i\)\}=\\psi\_\{X^\{\(i\)\}\}\(W^\{\(i\)\},s\_\{i\},\\rho\)\.U\(i\)U^\{\(i\)\}–Sparse trainable delta matrix used throughM\(i\)⊙U\(i\)M^\{\(i\)\}\\odot U^\{\(i\)\}\.L\(i\),R\(i\)L^\{\(i\)\},R^\{\(i\)\}–LoRAfactors,L\(i\)∈ℝci×riL^\{\(i\)\}\\in\\mathbb\{R\}^\{c\_\{i\}\\times r\_\{i\}\}andR\(i\)∈ℝri×biR^\{\(i\)\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times b\_\{i\}\}\.γi\\gamma\_\{i\}–LoRAscaling factor for layerii\.pp–Sparse density,p∈\(0,1\]p\\in\(0,1\]\.TiT\_\{i\}–Parameter\-count budget assigned to layerii\.Tirank​\(r0\)T\_\{i\}^\{\\mathrm\{rank\}\}\(r\_\{0\}\)–LoRA\-rank\-equivalent budget,r0​\(ci\+bi\)r\_\{0\}\(c\_\{i\}\+b\_\{i\}\)\.Tidensity​\(p\)T\_\{i\}^\{\\mathrm\{density\}\}\(p\)–Density\-equivalent budget,⌊p​ci​bi⌋\\lfloor pc\_\{i\}b\_\{i\}\\rfloor\.r0r\_\{0\}–ReferenceLoRArank used to define the rank\-equivalent budget\.pi​\(r0\)p\_\{i\}\(r\_\{0\}\)–Layerwise density induced by the rank\-equivalent budget,r0​\(ci\+bi\)/\(ci​bi\)r\_\{0\}\(c\_\{i\}\+b\_\{i\}\)/\(c\_\{i\}b\_\{i\}\)\.λ\\lambda–Low\-rank budget fraction forSupra,λ∈\[0,1\]\\lambda\\in\[0,1\]\.rir\_\{i\}–Low\-rank adapter rank for layerii\.lil\_\{i\}–Number of low\-rank adapter parameters in layerii,li=ri​\(ci\+bi\)l\_\{i\}=r\_\{i\}\(c\_\{i\}\+b\_\{i\}\)\.sis\_\{i\}–Number of sparse\-adapter parameters in layerii,si=Ti−lis\_\{i\}=T\_\{i\}\-l\_\{i\}\.eke\_\{k\}–kthk^\{\\text\{th\}\}standard basis vector\.∥⋅∥2\\\|\\cdot\\\|\_\{2\}–Euclidean norm\.∥⋅∥0\\\|\\cdot\\\|\_\{0\}–Number of nonzero entries\.∥⋅∥F\\\|\\cdot\\\|\_\{F\}–Frobenius norm\.⟨x,y⟩\\langle x,y\\rangle–Euclidean inner product of vectorsxxandyy\.⊙\\odot–Element\-wise product of matrices with the same shape \(Hadamard product\)\.⌊x⌋\\lfloor x\\rfloor–Floor function, the greatest integer less than or equal toxx\.
## Appendix BWandametric for generating fine\-tuning mask

For fixed calibration activationsX∈ℝb×aX\\in\\mathbb\{R\}^\{b\\times a\}, whereaais the number of calibration token positions, we define the mask\-generation mapping

ψX:ℝc×b×\{0,…,c​b\}×\{TopK,BottomK\}→\{0,1\}c×b\.\\psi\_\{X\}:\\mathbb\{R\}^\{c\\times b\}\\times\\\{0,\\ldots,cb\\\}\\times\\\{\\mathrm\{TopK\},\\mathrm\{BottomK\}\\\}\\to\\\{0,1\\\}^\{c\\times b\}\.It takes a weight matrixW∈ℝc×bW\\in\\mathbb\{R\}^\{c\\times b\}, a target number of sparse entriess∈\{0,…,c​b\}s\\in\\\{0,\\ldots,cb\\\}, and a selection directionρ∈\{TopK,BottomK\}\\rho\\in\\\{\\mathrm\{TopK\},\\mathrm\{BottomK\}\\\}\. For each weightWi​jW\_\{ij\}, it evaluates the metricAi​j=\|Wi​j\|​‖Xj:‖2A\_\{ij\}=\|W\_\{ij\}\|\\\|X\_\{j:\}\\\|\_\{2\}\. The TopK variant selects the global topssentries in the layer, while the BottomK variant selects the global bottomssentries\. Since squaring is monotone on nonnegative scores, the same ordering is obtained from the squaredWandascore in[Equation˜1](https://arxiv.org/html/2607.09287#S2.E1)\. The output is a binary mask with ones at these selected positions:

ψX​\(W,s,ρ\)i​j≔\{1,\(i,j\)​is among the​s​largest entries of​A,ρ=TopK,1,\(i,j\)​is among the​s​smallest entries of​A,ρ=BottomK,0,otherwise\.\\psi\_\{X\}\(W,s,\\rho\)\_\{ij\}\\coloneqq\\begin\{cases\}1,&\(i,j\)\\text\{ is among the \}s\\text\{ largest entries of \}A,\\quad\\rho=\\mathrm\{TopK\},\\\\ 1,&\(i,j\)\\text\{ is among the \}s\\text\{ smallest entries of \}A,\\quad\\rho=\\mathrm\{BottomK\},\\\\ 0,&\\text\{otherwise\.\}\\end\{cases\}\(23\)
For example, let us have a weight matrix

W=\(3−2−241−6\),W=\\begin\{pmatrix\}3&\-2\\\\ \-2&4\\\\ 1&\-6\\\\ \\end\{pmatrix\},and an input matrix

X=\(4301\),X=\\begin\{pmatrix\}4&3\\\\ 0&1\\\\ \\end\{pmatrix\},now we compute the metric\|Wi​j\|​‖Xj:‖2\|W\_\{ij\}\|\\\|X\_\{j:\}\\\|\_\{2\}for every entry ofWW:

\(\|W11\|​‖X1:‖2\|W12\|​‖X2:‖2\|W21\|​‖X1:‖2\|W22\|​‖X2:‖2\|W31\|​‖X1:‖2\|W32\|​‖X2:‖2\)=\(3⋅52⋅12⋅54⋅11⋅56⋅1\)=\(15210456\),\\begin\{pmatrix\}\|W\_\{11\}\|\\\|X\_\{1:\}\\\|\_\{2\}&\|W\_\{12\}\|\\\|X\_\{2:\}\\\|\_\{2\}\\\\ \|W\_\{21\}\|\\\|X\_\{1:\}\\\|\_\{2\}&\|W\_\{22\}\|\\\|X\_\{2:\}\\\|\_\{2\}\\\\ \|W\_\{31\}\|\\\|X\_\{1:\}\\\|\_\{2\}&\|W\_\{32\}\|\\\|X\_\{2:\}\\\|\_\{2\}\\\\ \\end\{pmatrix\}=\\begin\{pmatrix\}3\\cdot 5&2\\cdot 1\\\\ 2\\cdot 5&4\\cdot 1\\\\ 1\\cdot 5&6\\cdot 1\\\\ \\end\{pmatrix\}=\\begin\{pmatrix\}15&2\\\\ 10&4\\\\ 5&6\\\\ \\end\{pmatrix\},then the TopK masks are

ψX​\(W,1,TopK\)=\(100000\),ψX​\(W,2,TopK\)=\(101000\),ψX​\(W,4,TopK\)=\(101011\)\.\\psi\_\{X\}\(W,1,\\mathrm\{TopK\}\)=\\begin\{pmatrix\}1&0\\\\ 0&0\\\\ 0&0\\\\ \\end\{pmatrix\},\\quad\\psi\_\{X\}\(W,2,\\mathrm\{TopK\}\)=\\begin\{pmatrix\}1&0\\\\ 1&0\\\\ 0&0\\\\ \\end\{pmatrix\},\\quad\\psi\_\{X\}\(W,4,\\mathrm\{TopK\}\)=\\begin\{pmatrix\}1&0\\\\ 1&0\\\\ 1&1\\\\ \\end\{pmatrix\}\.For the same scores, the BottomK variant instead selects the lowest\-scoring entries; for example,

ψX​\(W,2,BottomK\)=\(010100\)\.\\psi\_\{X\}\(W,2,\\mathrm\{BottomK\}\)=\\begin\{pmatrix\}0&1\\\\ 0&1\\\\ 0&0\\\\ \\end\{pmatrix\}\.

## Appendix CAdditional experiments

### C\.1Hybrid TopK–BottomKSuper

The hybridSupervariant tests whether a sparse support should be selected entirely from one end of theWanda\-score ordering, or whether mixing both ends can help\. For a linear layerll, letNlN\_\{l\}be the number of weights in the layer,sls\_\{l\}be the same rank\-equivalent sparse budget used bySuper, and letAi​j\(l\)=\|Wi​j\(l\)\|​‖Xj:\(l\)‖2A^\{\(l\)\}\_\{ij\}=\|W^\{\(l\)\}\_\{ij\}\|\\\|X^\{\(l\)\}\_\{j:\}\\\|\_\{2\}be the C4\-calibratedWanda\-style score\. The hybrid mask is

Mhyb\(l\)​\(β\)=TopK⁡\(A\(l\),ktop\)∪BottomK⁡\(A\(l\),kbottom\),M^\{\(l\)\}\_\{\\mathrm\{hyb\}\}\(\\beta\)=\\operatorname\{TopK\}\\\!\\left\(A^\{\(l\)\},k\_\{\\mathrm\{top\}\}\\right\)\\cup\\operatorname\{BottomK\}\\\!\\left\(A^\{\(l\)\},k\_\{\\mathrm\{bottom\}\}\\right\),where

ktop=round⁡\(β​sl\),kbottom=sl−ktop,β∈\[0,1\]\.k\_\{\\mathrm\{top\}\}=\\operatorname\{round\}\(\\beta s\_\{l\}\),\\qquad k\_\{\\mathrm\{bottom\}\}=s\_\{l\}\-k\_\{\\mathrm\{top\}\},\\qquad\\beta\\in\[0,1\]\.Thusβ\\betais the fraction of the sparse budget assigned to the largestWandascores\. The endpoints recover the original sparse\-onlySupervariants:β=1\\beta=1isSuper\(TopK\), whileβ=0\\beta=0isSuper\(BottomK\)\. Intermediate values mix high\-score and low\-score trainable weights while preserving the same number of trainable sparse parameters\. Equivalently, ifpl=sl/Nlp\_\{l\}=s\_\{l\}/N\_\{l\}is the realized layerwise sparse rate, then the split isktop≈Nl​pl​βk\_\{\\mathrm\{top\}\}\\approx N\_\{l\}p\_\{l\}\\betaandkbottom≈Nl​pl​\(1−β\)k\_\{\\mathrm\{bottom\}\}\\approx N\_\{l\}p\_\{l\}\(1\-\\beta\)\. The results below show that this mixed support can sometimes improve over pure TopK, but it does not improve the schedule\-selected sparse\-only results; pure BottomK remains the strongest mask in the main comparisons\.

Table 7:Hybrid TopK–BottomK sparse\-support ablation\. Model:Llama\-3\.2\-1B; fine\-tuning dataset: Math17K; training budget: one or three epochs as indicated; rank\-equivalent budget:r0=8r\_\{0\}=8\. The Base row is frozen\. All trainable masks use C4 calibration and approximately5\.65\.6M trainable sparse parameters\. Best average within each epoch block is bold\. Results are exact\-answer accuracy \(%\)\.MethodEpochs𝜷\\bm\{\\beta\}Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)–––13\.674\.6721\.462\.8121\.6511\.6012\.64Super\(TopK\)11\.05⋅10−45\\cdot 10^\{\-4\}47\.5985\.5057\.8724\.2616\.9337\.1044\.88Super\(Hybrid\)10\.85⋅10−45\\cdot 10^\{\-4\}42\.7885\.5059\.8417\.130\.3937\.3040\.49Super\(Hybrid\)10\.55⋅10−45\\cdot 10^\{\-4\}40\.5186\.8358\.2728\.2010\.2437\.9043\.66Super\(Hybrid\)10\.35⋅10−45\\cdot 10^\{\-4\}60\.7687\.1758\.2727\.2222\.4446\.2050\.34Super\(BottomK\)10\.010−310^\{\-3\}39\.7590\.1734\.6516\.3015\.3525\.8037\.00Super\(TopK\)31\.010−410^\{\-4\}31\.9087\.0022\.447\.4322\.447\.3029\.75Super\(Hybrid\)30\.810−410^\{\-4\}36\.9684\.8327\.367\.8119\.698\.6030\.88Super\(Hybrid\)30\.510−410^\{\-4\}58\.7385\.8357\.4820\.9220\.4733\.1046\.09Super\(Hybrid\)30\.310−410^\{\-4\}71\.1481\.3371\.8522\.2925\.2045\.1052\.82Super\(BottomK\)30\.05⋅10−45\\cdot 10^\{\-4\}69\.1185\.5070\.4734\.5025\.5947\.6055\.46

Table 8:Hybrid TopK–BottomK sparse\-support ablation\. Model:Meta\-Llama\-3\-8B; fine\-tuning dataset: Math17K; training budget: three epochs; rank\-equivalent budget:r0=8r\_\{0\}=8\. The Base row is frozen\. All trainable masks use C4 calibration and approximately21\.021\.0M trainable sparse parameters\. Results are exact\-answer accuracy \(%\)\.MethodEpochs𝜷\\bm\{\\beta\}Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)–––22\.5321\.0036\.4210\.2424\.0224\.1023\.05Super\(TopK\)31\.05⋅10−55\\cdot 10^\{\-5\}4\.8188\.1713\.199\.4835\.045\.9026\.10Super\(Hybrid\)30\.85⋅10−55\\cdot 10^\{\-5\}4\.0597\.0021\.6515\.1640\.1611\.5031\.59Super\(Hybrid\)30\.55⋅10−55\\cdot 10^\{\-5\}0\.5198\.1721\.8536\.2436\.6113\.9034\.55Super\(Hybrid\)30\.35⋅10−55\\cdot 10^\{\-5\}15\.1998\.6760\.4351\.2542\.5243\.2051\.88Super\(BottomK\)30\.010−410^\{\-4\}72\.1595\.6788\.5866\.3442\.9170\.7072\.73

### C\.2Llama\-3\.2\-1Bresults

#### C\.2\.1Math7K, one\-epoch comparison

[Tables˜9](https://arxiv.org/html/2607.09287#A3.T9)and[10](https://arxiv.org/html/2607.09287#A3.T10)report theLlama\-3\.2\-1BMath7K one\-epoch comparison\. This run is kept in the appendix and is not used for the main\-text claims\. TheCalib\.column gives the calibration source used to construct fixedWanda\-style sparse supports; “–” denotes methods that do not use a calibration pass\. Full fine\-tuning is included as an unbudgeted reference row separated from the matched\-budget adapter rows\.

Table 9:Arithmetic benchmark accuracy\. Model:Llama\-3\.2\-1B; fine\-tuning dataset: Math7K; training budget: one epoch; rank\-equivalent budget:r0=8r\_\{0\}=8\. Trainable adapter rows use approximately5\.65\.6M trainable parameters; the Base row is frozen\. Full fine\-tuning is an unbudgeted reference row separated by rules\. Bold marks the best observed matched\-budget adapter average\. Each row uses the learning rate selected by the held\-out validation split of the fine\-tuning set\. Results are exact\-answer accuracy \(%\)\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––13\.674\.6721\.462\.8121\.6511\.6012\.64Full fine\-tuning–5⋅10−55\\cdot 10^\{\-5\}36\.7167\.5049\.8025\.7821\.6535\.2039\.44LoRA–10−310^\{\-3\}36\.9664\.6750\.0021\.2321\.2634\.2038\.05RoSA–5⋅10−45\\cdot 10^\{\-4\}39\.7565\.5051\.5722\.9022\.4431\.8038\.99SIFT\(TopK\)–10−410^\{\-4\}29\.8755\.0038\.3916\.2220\.4722\.7030\.44SIFT\(RandK\)–10−310^\{\-3\}36\.2067\.3350\.5920\.8519\.6929\.7037\.39Super\(RandK\)–10−310^\{\-3\}38\.7362\.8348\.0321\.6820\.4730\.6037\.06Super\(TopK\)C45⋅10−45\\cdot 10^\{\-4\}35\.9563\.8348\.6221\.4621\.6529\.0036\.75Super\(BottomK\)C410−310^\{\-3\}40\.7659\.3352\.5621\.7617\.3232\.3037\.34Supra\(TopK,λ=0\.3\\lambda=0\.3\)C45⋅10−45\\cdot 10^\{\-4\}38\.7362\.5053\.3521\.3023\.2332\.4038\.59Supra\(TopK,λ=0\.5\\lambda=0\.5\)C45⋅10−45\\cdot 10^\{\-4\}38\.4864\.6751\.7723\.1221\.2631\.8038\.52Supra\(TopK,λ=0\.8\\lambda=0\.8\)C45⋅10−45\\cdot 10^\{\-4\}41\.0165\.8352\.7622\.4421\.6534\.3039\.67Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C410−310^\{\-3\}34\.1862\.5046\.4623\.9621\.2633\.0036\.89Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C410−310^\{\-3\}38\.2362\.6749\.2123\.0524\.0233\.2038\.40Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C410−310^\{\-3\}33\.1664\.8351\.3822\.5222\.8334\.0038\.12Super\(TopK\)Math7K5⋅10−45\\cdot 10^\{\-4\}38\.2361\.8348\.6222\.0620\.0829\.6036\.74Super\(BottomK\)Math7K10−310^\{\-3\}41\.0157\.8351\.3820\.1720\.0832\.2037\.11

Table 10:Test perplexity\. Model:Llama\-3\.2\-1B; fine\-tuning dataset: Math7K; training budget: one epoch; rank\-equivalent budget:r0=8r\_\{0\}=8\. These are the same validation\-selected checkpoints as[Table˜9](https://arxiv.org/html/2607.09287#A3.T9)\. Full fine\-tuning is an unbudgeted reference row separated by rules\. Bold marks the lowest matched\-budget adapter average PPL\. TheCalib\.column gives the calibration source used to construct fixedWanda\-style sparse supports; “–” denotes methods that do not use a calibration pass\. Lower is better\. Perplexities are computed on up to 120 examples per benchmark\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––2\.712\.812\.742\.453\.212\.942\.73Full fine\-tuning–5⋅10−55\\cdot 10^\{\-5\}2\.562\.702\.722\.393\.632\.932\.68LoRA–10−310^\{\-3\}2\.762\.922\.952\.533\.713\.182\.88RoSA–5⋅10−45\\cdot 10^\{\-4\}2\.562\.772\.782\.463\.562\.992\.74SIFT\(TopK\)–10−410^\{\-4\}2\.492\.622\.652\.373\.452\.872\.63SIFT\(RandK\)–10−310^\{\-3\}2\.482\.602\.642\.343\.342\.832\.60Super\(RandK\)–10−310^\{\-3\}2\.532\.612\.652\.353\.462\.852\.62Super\(TopK\)C45⋅10−45\\cdot 10^\{\-4\}2\.522\.642\.672\.343\.452\.882\.63Super\(BottomK\)C410−310^\{\-3\}2\.342\.492\.472\.253\.232\.662\.47Supra\(TopK,λ=0\.3\\lambda=0\.3\)C45⋅10−45\\cdot 10^\{\-4\}2\.672\.812\.842\.463\.653\.052\.78Supra\(TopK,λ=0\.5\\lambda=0\.5\)C45⋅10−45\\cdot 10^\{\-4\}2\.682\.772\.872\.463\.653\.072\.78Supra\(TopK,λ=0\.8\\lambda=0\.8\)C45⋅10−45\\cdot 10^\{\-4\}2\.722\.862\.902\.483\.643\.102\.82Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C410−310^\{\-3\}2\.812\.953\.012\.563\.813\.202\.92Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C410−310^\{\-3\}2\.772\.883\.002\.553\.703\.212\.89Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C410−310^\{\-3\}2\.812\.893\.052\.563\.673\.252\.91Super\(TopK\)Math7K5⋅10−45\\cdot 10^\{\-4\}2\.492\.632\.622\.343\.492\.812\.60Super\(BottomK\)Math7K10−310^\{\-3\}2\.322\.482\.442\.233\.222\.642\.45

#### C\.2\.2Math17K, one\-epoch comparison

[Tables˜11](https://arxiv.org/html/2607.09287#A3.T11)and[12](https://arxiv.org/html/2607.09287#A3.T12)report theLlama\-3\.2\-1BMath17K comparison for one epoch\. These rows are used together with the three\-epoch rows to form the schedule\-selected compact table in the main text\. Full fine\-tuning is included as an unbudgeted reference row separated from the matched\-budget adapter rows\. The separate hybrid TopK–BottomK sparse\-support ablation is reported in[Section˜C\.1](https://arxiv.org/html/2607.09287#A3.SS1)\.

Table 11:Arithmetic benchmark accuracy\. Model:Llama\-3\.2\-1B; fine\-tuning dataset: Math17K; training budget: one epoch; rank\-equivalent budget:r0=8r\_\{0\}=8\. The Base row is frozen\. Adapter rows use approximately5\.65\.6M trainable parameters\. Full fine\-tuning is an unbudgeted reference row separated by rules\. Bold marks the best observed matched\-budget adapter average\. Each row uses the learning rate selected by the held\-out validation split of the fine\-tuning set\. Results are exact\-answer accuracy \(%\)\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––13\.674\.6721\.462\.8121\.6511\.6012\.64Full fine\-tuning–5⋅10−55\\cdot 10^\{\-5\}70\.3891\.8377\.7636\.5424\.0258\.1059\.77LoRA–10−310^\{\-3\}73\.6785\.8377\.5630\.1024\.4153\.8057\.56RoSA–5⋅10−45\\cdot 10^\{\-4\}71\.6589\.1778\.3527\.9824\.8053\.7057\.61SIFT\(TopK\)–10−410^\{\-4\}60\.5183\.1763\.1923\.433\.9440\.3045\.75SIFT\(RandK\)–10−310^\{\-3\}62\.2890\.8373\.6231\.4617\.7248\.7054\.10Super\(TopK\)C45⋅10−45\\cdot 10^\{\-4\}47\.5985\.5057\.8724\.2616\.9337\.1044\.88Super\(BottomK\)C410−310^\{\-3\}39\.7590\.1734\.6516\.3015\.3525\.8037\.00Magnitude\(TopK\)–10−310^\{\-3\}77\.471\.5080\.7127\.7520\.8752\.1043\.40Magnitude\(BottomK\)–10−310^\{\-3\}73\.9290\.5081\.6930\.2524\.8054\.7059\.31Supra\(TopK,λ=0\.3\\lambda=0\.3\)C45⋅10−45\\cdot 10^\{\-4\}69\.8790\.3378\.7426\.6924\.8052\.0057\.07Supra\(TopK,λ=0\.5\\lambda=0\.5\)C45⋅10−45\\cdot 10^\{\-4\}61\.7786\.3354\.5324\.7225\.9835\.3048\.11Supra\(TopK,λ=0\.8\\lambda=0\.8\)C45⋅10−45\\cdot 10^\{\-4\}78\.2392\.1780\.7123\.6525\.9857\.4059\.69Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C410−310^\{\-3\}59\.2487\.3357\.4823\.122\.3631\.5043\.51Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C410−310^\{\-3\}76\.2079\.6779\.9228\.3524\.0259\.9058\.01Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C410−310^\{\-3\}74\.1889\.3379\.5329\.9526\.7753\.0058\.79Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)–5⋅10−45\\cdot 10^\{\-4\}76\.7191\.6781\.1022\.9727\.5655\.1059\.18Supra\-Mag\(BottomK,λ=0\.5\\lambda=0\.5\)–10−310^\{\-3\}66\.8488\.1778\.1530\.0222\.0549\.5055\.79Supra\-Mag\(BottomK,λ=0\.8\\lambda=0\.8\)–10−310^\{\-3\}73\.1688\.8375\.7933\.1324\.0252\.0057\.82

Table 12:Test perplexity\. Model:Llama\-3\.2\-1B; fine\-tuning dataset: Math17K; training budget: one epoch; rank\-equivalent budget:r0=8r\_\{0\}=8\. These are the same validation\-selected checkpoints as[Table˜11](https://arxiv.org/html/2607.09287#A3.T11)\. The Base row is frozen\. Full fine\-tuning is an unbudgeted reference row separated by rules\. Bold marks the lowest matched\-budget adapter average PPL\. TheCalib\.column gives the calibration source used to construct fixedWanda\-style sparse supports; “–” denotes methods that do not use a calibration pass\. Lower is better\. Perplexities are computed on up to 120 examples per benchmark\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––2\.712\.812\.742\.453\.212\.942\.73Full fine\-tuning–5⋅10−55\\cdot 10^\{\-5\}1\.101\.121\.141\.281\.591\.191\.21LoRA–10−310^\{\-3\}1\.121\.141\.161\.321\.661\.221\.24RoSA–5⋅10−45\\cdot 10^\{\-4\}1\.131\.161\.181\.351\.731\.241\.27SIFT\(TopK\)–10−410^\{\-4\}1\.181\.231\.231\.401\.831\.311\.33SIFT\(RandK\)–10−310^\{\-3\}1\.121\.151\.171\.321\.681\.231\.25Super\(TopK\)C45⋅10−45\\cdot 10^\{\-4\}1\.131\.171\.181\.341\.731\.251\.27Super\(BottomK\)C410−310^\{\-3\}1\.121\.151\.171\.321\.801\.231\.26Magnitude\(TopK\)–10−310^\{\-3\}1\.121\.161\.171\.331\.731\.231\.26Magnitude\(BottomK\)–10−310^\{\-3\}1\.121\.141\.161\.311\.641\.221\.24Supra\(TopK,λ=0\.3\\lambda=0\.3\)C45⋅10−45\\cdot 10^\{\-4\}1\.131\.151\.171\.331\.711\.231\.26Supra\(TopK,λ=0\.5\\lambda=0\.5\)C45⋅10−45\\cdot 10^\{\-4\}1\.121\.151\.171\.331\.691\.231\.25Supra\(TopK,λ=0\.8\\lambda=0\.8\)C45⋅10−45\\cdot 10^\{\-4\}1\.131\.161\.171\.341\.701\.241\.26Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C410−310^\{\-3\}1\.121\.141\.161\.321\.651\.221\.24Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C410−310^\{\-3\}1\.131\.131\.161\.311\.641\.221\.24Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C410−310^\{\-3\}1\.111\.131\.161\.311\.651\.211\.24Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)–5⋅10−45\\cdot 10^\{\-4\}1\.121\.151\.171\.321\.671\.231\.25Supra\-Mag\(BottomK,λ=0\.5\\lambda=0\.5\)–10−310^\{\-3\}1\.121\.131\.161\.321\.651\.221\.24Supra\-Mag\(BottomK,λ=0\.8\\lambda=0\.8\)–10−310^\{\-3\}1\.121\.131\.151\.311\.651\.211\.24

#### C\.2\.3Math17K, three\-epoch comparison

[Tables˜13](https://arxiv.org/html/2607.09287#A3.T13)and[14](https://arxiv.org/html/2607.09287#A3.T14)report theLlama\-3\.2\-1BMath17K comparison for three epochs\. These runs use the same rank\-equivalent adapter budget and evaluation suite as the one\-epoch comparison, with full fine\-tuning included as an unbudgeted reference row\.

![Refer to caption](https://arxiv.org/html/2607.09287v1/x4.png)

![Refer to caption](https://arxiv.org/html/2607.09287v1/x5.png)

Figure 3:Optimization curves forLlama\-3\.2\-1BMath17K fine\-tuning over three epochs\. Left: training loss, plotted with a log\-scaledyy\-axis\. Right: held\-out Math17K validation loss\. Each curve uses the validation\-selected learning rate for the corresponding method under the rank\-equivalentr0=8r\_\{0\}=8budget\.Table 13:Arithmetic benchmark accuracy\. Model:Llama\-3\.2\-1B; fine\-tuning dataset: Math17K; training budget: three epochs; rank\-equivalent budget:r0=8r\_\{0\}=8\. The Base row is frozen\. Adapter rows use approximately5\.65\.6M trainable parameters\. Full fine\-tuning is an unbudgeted reference row separated by rules\. Bold marks the best observed matched\-budget adapter average\. Each row uses the learning rate selected by the held\-out validation split of the fine\-tuning set\. Results are exact\-answer accuracy \(%\)\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––13\.674\.6721\.462\.8121\.6511\.6012\.64Full fine\-tuning–5⋅10−55\\cdot 10^\{\-5\}80\.0091\.6785\.6344\.0528\.7464\.1065\.70LoRA–5⋅10−45\\cdot 10^\{\-4\}70\.6390\.8385\.2435\.3326\.7757\.6061\.07RoSA–10−410^\{\-4\}59\.7589\.0071\.2627\.8221\.2646\.6052\.62SIFT\(TopK\)–10−410^\{\-4\}30\.8964\.1716\.737\.5124\.807\.4025\.25SIFT\(RandK\)–5⋅10−45\\cdot 10^\{\-4\}52\.1581\.6772\.8329\.7225\.9853\.7052\.68Super\(TopK\)C410−410^\{\-4\}31\.9087\.0022\.447\.4322\.447\.3029\.75Super\(BottomK\)C45⋅10−45\\cdot 10^\{\-4\}69\.1185\.5070\.4734\.5025\.5947\.6055\.46Magnitude\(TopK\)–5⋅10−45\\cdot 10^\{\-4\}72\.9184\.8373\.0333\.4323\.2345\.6055\.51Magnitude\(BottomK\)–5⋅10−45\\cdot 10^\{\-4\}61\.7725\.0069\.0933\.8125\.2046\.7043\.60Supra\(TopK,λ=0\.3\\lambda=0\.3\)C410−410^\{\-4\}45\.0690\.8372\.0532\.6024\.8047\.7052\.17Supra\(TopK,λ=0\.5\\lambda=0\.5\)C410−410^\{\-4\}63\.8081\.1775\.5931\.6923\.6252\.2054\.68Supra\(TopK,λ=0\.8\\lambda=0\.8\)C410−410^\{\-4\}64\.5676\.0072\.0530\.1024\.0247\.0052\.29Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C410−410^\{\-4\}60\.2589\.8372\.0531\.7725\.5950\.7055\.03Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C410−410^\{\-4\}61\.2792\.1774\.0230\.2520\.4754\.2055\.40Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C45⋅10−45\\cdot 10^\{\-4\}82\.5385\.5084\.4536\.1624\.4160\.3062\.23Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)–10−410^\{\-4\}60\.0089\.3368\.7033\.7425\.9844\.5053\.71Supra\-Mag\(BottomK,λ=0\.5\\lambda=0\.5\)–10−410^\{\-4\}60\.2590\.5075\.0031\.2425\.9854\.0056\.16Supra\-Mag\(BottomK,λ=0\.8\\lambda=0\.8\)–5⋅10−45\\cdot 10^\{\-4\}76\.4628\.1780\.5138\.2925\.5952\.6050\.27

Table 14:Test perplexity\. Model:Llama\-3\.2\-1B; fine\-tuning dataset: Math17K; training budget: three epochs; rank\-equivalent budget:r0=8r\_\{0\}=8\. These are the same validation\-selected checkpoints as[Table˜13](https://arxiv.org/html/2607.09287#A3.T13)\. The Base row is frozen\. Full fine\-tuning is an unbudgeted reference row separated by rules\. Bold marks the lowest matched\-budget adapter average PPL\. TheCalib\.column gives the calibration source used to construct fixedWanda\-style sparse supports; “–” denotes methods that do not use a calibration pass\. Lower is better\. Perplexities are computed on up to 120 examples per benchmark\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––2\.712\.812\.742\.453\.212\.942\.73Full fine\-tuning–5⋅10−55\\cdot 10^\{\-5\}1\.051\.051\.071\.161\.291\.101\.11LoRA–5⋅10−45\\cdot 10^\{\-4\}1\.071\.071\.111\.231\.451\.151\.17RoSA–10−410^\{\-4\}1\.131\.161\.181\.341\.701\.241\.27SIFT\(TopK\)–10−410^\{\-4\}1\.121\.161\.171\.331\.681\.241\.26SIFT\(RandK\)–5⋅10−45\\cdot 10^\{\-4\}1\.081\.101\.121\.261\.561\.181\.19Super\(TopK\)C410−410^\{\-4\}1\.161\.181\.211\.391\.971\.291\.31Super\(BottomK\)C45⋅10−45\\cdot 10^\{\-4\}1\.091\.101\.131\.271\.721\.181\.21Magnitude\(TopK\)–5⋅10−45\\cdot 10^\{\-4\}1\.091\.101\.131\.271\.721\.181\.21Magnitude\(BottomK\)–5⋅10−45\\cdot 10^\{\-4\}1\.081\.091\.121\.251\.591\.171\.19Supra\(TopK,λ=0\.3\\lambda=0\.3\)C410−410^\{\-4\}1\.111\.131\.151\.311\.761\.211\.24Supra\(TopK,λ=0\.5\\lambda=0\.5\)C410−410^\{\-4\}1\.111\.131\.151\.311\.671\.211\.24Supra\(TopK,λ=0\.8\\lambda=0\.8\)C410−410^\{\-4\}1\.111\.131\.161\.321\.711\.221\.24Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C410−410^\{\-4\}1\.131\.141\.181\.341\.711\.241\.26Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C410−410^\{\-4\}1\.121\.151\.171\.341\.811\.241\.26Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C45⋅10−45\\cdot 10^\{\-4\}1\.081\.081\.101\.231\.441\.141\.16Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)–10−410^\{\-4\}1\.141\.151\.181\.341\.791\.251\.27Supra\-Mag\(BottomK,λ=0\.5\\lambda=0\.5\)–10−410^\{\-4\}1\.131\.151\.181\.341\.791\.241\.27Supra\-Mag\(BottomK,λ=0\.8\\lambda=0\.8\)–5⋅10−45\\cdot 10^\{\-4\}1\.071\.081\.101\.221\.441\.141\.16

### C\.3Meta\-Llama\-3\-8Bresults

#### C\.3\.1Math7K, one\-epoch comparison

[Tables˜15](https://arxiv.org/html/2607.09287#A3.T15)and[16](https://arxiv.org/html/2607.09287#A3.T16)report the Math7K one\-epoch comparison forMeta\-Llama\-3\-8B\. We only report the completed 8B Math7K adapter suite here; the larger Math17K 8B comparisons are reported separately in[Sections˜C\.3\.2](https://arxiv.org/html/2607.09287#A3.SS3.SSS2)and[C\.3\.3](https://arxiv.org/html/2607.09287#A3.SS3.SSS3)\.

Table 15:Arithmetic benchmark accuracy\. Model:Meta\-Llama\-3\-8B; fine\-tuning dataset: Math7K; training budget: one epoch; rank\-equivalent budget:r0=8r\_\{0\}=8\. Adapter rows use approximately the rank\-8LoRAtrainable\-parameter budget for this model\. Each row uses the learning rate selected by the held\-out validation split of the fine\-tuning set\. Results are exact\-answer accuracy \(%\)\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––22\.5321\.0036\.4210\.2424\.0224\.1023\.05LoRA–5⋅10−45\\cdot 10^\{\-4\}83\.2991\.1792\.7265\.3525\.9875\.8072\.39RoSA–10−410^\{\-4\}82\.7887\.5091\.3464\.3728\.7474\.7071\.57SIFT\(TopK\)–10−410^\{\-4\}82\.2889\.1790\.9462\.4025\.2072\.6070\.43SIFT\(RandK\)–10−310^\{\-3\}78\.7392\.5091\.1465\.2025\.5974\.7071\.31Super\(RandK\)–10−310^\{\-3\}81\.7790\.8392\.3265\.8827\.1774\.8072\.13Super\(TopK\)C410−410^\{\-4\}84\.8190\.0088\.7862\.1726\.3875\.8071\.32Super\(BottomK\)C410−310^\{\-3\}81\.7789\.6792\.1365\.5027\.5674\.0071\.77Supra\(TopK,λ=0\.3\\lambda=0\.3\)C410−410^\{\-4\}84\.3089\.0091\.1463\.2327\.1774\.5071\.56Supra\(TopK,λ=0\.5\\lambda=0\.5\)C410−410^\{\-4\}83\.8089\.8390\.7562\.7030\.3173\.4071\.80Supra\(TopK,λ=0\.8\\lambda=0\.8\)C410−410^\{\-4\}85\.3288\.8392\.3263\.6827\.5674\.9072\.10Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C45⋅10−45\\cdot 10^\{\-4\}81\.7791\.1791\.5466\.9428\.7475\.5072\.61Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C45⋅10−45\\cdot 10^\{\-4\}81\.0189\.6790\.9466\.1926\.7777\.6072\.03Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C45⋅10−45\\cdot 10^\{\-4\}83\.0491\.8391\.3466\.0326\.7777\.2072\.70

Table 16:Test perplexity\. Model:Meta\-Llama\-3\-8B; fine\-tuning dataset: Math7K; training budget: one epoch; rank\-equivalent budget:r0=8r\_\{0\}=8\. These are the same validation\-selected checkpoints as[Table˜15](https://arxiv.org/html/2607.09287#A3.T15)\. TheCalib\.column gives the calibration source used to construct fixedWanda\-style sparse supports; “–” denotes methods that do not use a calibration pass\. Lower is better\. Perplexities are computed on up to 120 examples per benchmark\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––2\.152\.282\.301\.982\.642\.412\.22LoRA–5⋅10−45\\cdot 10^\{\-4\}2\.342\.502\.532\.152\.972\.642\.43RoSA–10−410^\{\-4\}2\.372\.542\.612\.203\.052\.732\.49SIFT\(TopK\)–10−410^\{\-4\}2\.432\.582\.742\.252\.952\.842\.55SIFT\(RandK\)–10−310^\{\-3\}2\.202\.372\.382\.092\.862\.492\.32Super\(RandK\)–10−310^\{\-3\}2\.162\.302\.322\.052\.862\.442\.27Super\(TopK\)C410−410^\{\-4\}2\.212\.382\.442\.082\.862\.552\.33Super\(BottomK\)C410−310^\{\-3\}2\.382\.512\.582\.193\.002\.692\.47Supra\(TopK,λ=0\.3\\lambda=0\.3\)C410−410^\{\-4\}2\.262\.432\.452\.112\.932\.572\.37Supra\(TopK,λ=0\.5\\lambda=0\.5\)C410−410^\{\-4\}2\.282\.452\.492\.122\.912\.602\.39Supra\(TopK,λ=0\.8\\lambda=0\.8\)C410−410^\{\-4\}2\.362\.552\.572\.172\.922\.692\.46Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C45⋅10−45\\cdot 10^\{\-4\}2\.182\.372\.352\.072\.912\.492\.31Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C45⋅10−45\\cdot 10^\{\-4\}2\.182\.372\.372\.082\.892\.502\.31Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C45⋅10−45\\cdot 10^\{\-4\}2\.262\.482\.432\.112\.892\.552\.37

#### C\.3\.2Math17K, one\-epoch comparison

[Tables˜17](https://arxiv.org/html/2607.09287#A3.T17)and[18](https://arxiv.org/html/2607.09287#A3.T18)report theMeta\-Llama\-3\-8BMath17K comparison for one epoch\. TheRoSArow uses the corrected one\-epochRoSArun; the remaining adapter rows come from the matching one\-epoch sweep without hybrid TopK–BottomK sparse supports\. Full fine\-tuning is included as an unbudgeted reference row\. The separate fixed\-learning\-rate sanity check forSuper\(BottomK\) is exported with the result artifacts, but is not included in the table because it was not selected by the validation\-loss protocol\.

Table 17:Arithmetic benchmark accuracy\. Model:Meta\-Llama\-3\-8B; fine\-tuning dataset: Math17K; training budget: one epoch; rank\-equivalent budget:r0=8r\_\{0\}=8\. Adapter rows use approximately the rank\-8LoRAtrainable\-parameter budget for this model\. Full fine\-tuning is an unbudgeted reference row separated by rules\. Bold marks the best observed matched\-budget adapter average\. TheCalib\.column gives the calibration source used to construct fixedWanda\-style sparse supports; “–” denotes methods that do not use a calibration pass\. Each row uses the learning rate selected by the held\-out validation split of the fine\-tuning set\. Results are exact\-answer accuracy \(%\)\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––22\.5321\.0036\.4210\.2424\.0224\.1023\.05Full fine\-tuning–5⋅10−55\\cdot 10^\{\-5\}86\.0847\.5090\.3550\.6428\.3576\.0063\.15LoRA–5⋅10−45\\cdot 10^\{\-4\}86\.3397\.6793\.7066\.3424\.4170\.6073\.17RoSA–10−410^\{\-4\}86\.8497\.5092\.7258\.4535\.8379\.6075\.16SIFT\(TopK\)–10−410^\{\-4\}86\.5898\.5094\.6963\.7635\.0479\.1076\.28SIFT\(RandK\)–10−310^\{\-3\}84\.3098\.3393\.5063\.9945\.2881\.6077\.83Super\(RandK\)–10−310^\{\-3\}6\.5899\.1716\.9334\.9535\.839\.0033\.74Super\(TopK\)C45⋅10−45\\cdot 10^\{\-4\}13\.6790\.6724\.4145\.1916\.5416\.0034\.41Super\(BottomK\)C410−310^\{\-3\}1\.2798\.173\.942\.5827\.170\.6022\.29Magnitude\(TopK\)–10−310^\{\-3\}54\.6898\.0061\.4257\.8537\.4035\.1057\.41Magnitude\(BottomK\)–10−310^\{\-3\}90\.8997\.3395\.0868\.0141\.7381\.1079\.02Supra\(TopK,λ=0\.3\\lambda=0\.3\)C410−410^\{\-4\}86\.8486\.1787\.4071\.4935\.8382\.0074\.95Supra\(TopK,λ=0\.5\\lambda=0\.5\)C410−410^\{\-4\}83\.0498\.0086\.0270\.2037\.8080\.2075\.88Supra\(TopK,λ=0\.8\\lambda=0\.8\)C410−410^\{\-4\}88\.1099\.0096\.6569\.9030\.7181\.7077\.68Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C45⋅10−45\\cdot 10^\{\-4\}89\.1195\.3396\.0667\.6341\.3482\.5078\.66Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C45⋅10−45\\cdot 10^\{\-4\}91\.3997\.5094\.0967\.8533\.8683\.4078\.02Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C45⋅10−45\\cdot 10^\{\-4\}90\.8996\.6796\.2667\.1736\.2282\.5078\.28Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)–5⋅10−45\\cdot 10^\{\-4\}92\.9198\.5096\.8569\.8335\.4381\.2079\.12Supra\-Mag\(BottomK,λ=0\.5\\lambda=0\.5\)–5⋅10−45\\cdot 10^\{\-4\}91\.3996\.8393\.9062\.4017\.7279\.3073\.59Supra\-Mag\(BottomK,λ=0\.8\\lambda=0\.8\)–5⋅10−45\\cdot 10^\{\-4\}87\.3499\.3387\.6064\.0629\.5377\.8074\.28

Table 18:Test perplexity\. Model:Meta\-Llama\-3\-8B; fine\-tuning dataset: Math17K; training budget: one epoch; rank\-equivalent budget:r0=8r\_\{0\}=8\. These are the same validation\-selected checkpoints as[Table˜17](https://arxiv.org/html/2607.09287#A3.T17)\. Full fine\-tuning is an unbudgeted reference row separated by rules\. Bold marks the lowest matched\-budget adapter average PPL\. TheCalib\.column gives the calibration source used to construct fixedWanda\-style sparse supports; “–” denotes methods that do not use a calibration pass\. Lower is better\. Perplexities are computed on up to 120 examples per benchmark\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––2\.152\.282\.301\.982\.642\.412\.22Full fine\-tuning–5⋅10−55\\cdot 10^\{\-5\}1\.071\.081\.091\.171\.351\.131\.14LoRA–5⋅10−45\\cdot 10^\{\-4\}1\.071\.091\.101\.171\.331\.131\.14RoSA–10−410^\{\-4\}1\.101\.121\.131\.211\.421\.171\.18SIFT\(TopK\)–10−410^\{\-4\}1\.101\.121\.131\.211\.421\.171\.18SIFT\(RandK\)–10−310^\{\-3\}1\.071\.081\.091\.171\.311\.131\.13Super\(RandK\)–10−310^\{\-3\}1\.071\.081\.091\.171\.311\.121\.13Super\(TopK\)C45⋅10−45\\cdot 10^\{\-4\}1\.081\.091\.101\.181\.351\.141\.15Super\(BottomK\)C410−310^\{\-3\}1\.071\.081\.091\.171\.311\.131\.13Magnitude\(TopK\)–10−310^\{\-3\}1\.071\.081\.091\.171\.321\.131\.13Magnitude\(BottomK\)–10−310^\{\-3\}1\.061\.081\.091\.161\.301\.121\.13Supra\(TopK,λ=0\.3\\lambda=0\.3\)C410−410^\{\-4\}1\.091\.111\.121\.201\.391\.151\.16Supra\(TopK,λ=0\.5\\lambda=0\.5\)C410−410^\{\-4\}1\.091\.111\.121\.201\.401\.161\.17Supra\(TopK,λ=0\.8\\lambda=0\.8\)C410−410^\{\-4\}1\.091\.121\.121\.201\.411\.161\.17Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C45⋅10−45\\cdot 10^\{\-4\}1\.071\.081\.101\.171\.321\.131\.14Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C45⋅10−45\\cdot 10^\{\-4\}1\.071\.081\.101\.171\.321\.131\.14Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C45⋅10−45\\cdot 10^\{\-4\}1\.071\.081\.101\.171\.321\.131\.14Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)–5⋅10−45\\cdot 10^\{\-4\}1\.071\.081\.101\.171\.321\.131\.14Supra\-Mag\(BottomK,λ=0\.5\\lambda=0\.5\)–5⋅10−45\\cdot 10^\{\-4\}1\.071\.081\.101\.171\.311\.121\.13Supra\-Mag\(BottomK,λ=0\.8\\lambda=0\.8\)–5⋅10−45\\cdot 10^\{\-4\}1\.071\.081\.101\.171\.321\.131\.14

#### C\.3\.3Math17K, three\-epoch comparison

[Tables˜19](https://arxiv.org/html/2607.09287#A3.T19)and[20](https://arxiv.org/html/2607.09287#A3.T20)report theMeta\-Llama\-3\-8BMath17K comparison summarized in the main text\. All rows in these tables use the three\-epoch schedule\. Full fine\-tuning is included as an unbudgeted reference row\. The corresponding one\-epoch comparison is reported separately in[Tables˜17](https://arxiv.org/html/2607.09287#A3.T17)and[18](https://arxiv.org/html/2607.09287#A3.T18), and the hybrid TopK–BottomKSupervariants are reported separately in[Table˜8](https://arxiv.org/html/2607.09287#A3.T8)\.

Table 19:Arithmetic benchmark accuracy\. Model:Meta\-Llama\-3\-8B; fine\-tuning dataset: Math17K; training budget: three epochs; rank\-equivalent budget:r0=8r\_\{0\}=8\. Adapter rows use approximately the rank\-8LoRAtrainable\-parameter budget for this model\. Full fine\-tuning is an unbudgeted reference row separated by rules\. Bold marks the best observed matched\-budget adapter average\. TheCalib\.column gives the calibration source used to construct fixedWanda\-style sparse supports; “–” denotes methods that do not use a calibration pass\. Each row uses the learning rate selected by the held\-out validation split of the fine\-tuning set\. Results are exact\-answer accuracy \(%\)\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––22\.5321\.0036\.4210\.2424\.0224\.1023\.05Full fine\-tuning–5⋅10−55\\cdot 10^\{\-5\}90\.6396\.1793\.7054\.3638\.1976\.2074\.87LoRA–10−410^\{\-4\}76\.7183\.0087\.9967\.1034\.2578\.6071\.27RoSA–5⋅10−55\\cdot 10^\{\-5\}34\.4397\.8334\.0640\.7140\.5530\.6046\.36SIFT\(TopK\)–5⋅10−55\\cdot 10^\{\-5\}45\.8295\.5082\.8768\.9942\.5278\.8069\.08SIFT\(RandK\)–10−410^\{\-4\}26\.0893\.3366\.3461\.1844\.0949\.3056\.72Super\(RandK\)–10−410^\{\-4\}0\.2597\.3348\.4359\.9737\.8031\.0045\.80Super\(TopK\)C45⋅10−55\\cdot 10^\{\-5\}4\.8188\.1713\.199\.4835\.045\.9026\.10Super\(BottomK\)C410−410^\{\-4\}72\.1595\.6788\.5866\.3442\.9170\.7072\.73Magnitude\(TopK\)–10−410^\{\-4\}13\.9297\.3373\.8264\.5244\.8861\.7059\.36Magnitude\(BottomK\)–10−410^\{\-4\}9\.1194\.6745\.2844\.0541\.3423\.3042\.96Supra\(TopK,λ=0\.3\\lambda=0\.3\)C45⋅10−55\\cdot 10^\{\-5\}35\.9590\.0052\.3660\.8041\.3432\.7052\.19Supra\(TopK,λ=0\.5\\lambda=0\.5\)C45⋅10−55\\cdot 10^\{\-5\}17\.2296\.6751\.1866\.7938\.9840\.7051\.92Supra\(TopK,λ=0\.8\\lambda=0\.8\)C45⋅10−55\\cdot 10^\{\-5\}0\.0097\.670\.002\.3532\.680\.0022\.12Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C410−410^\{\-4\}35\.7049\.8338\.9850\.7240\.1614\.3038\.28Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C410−310^\{\-3\}85\.3285\.0075\.7924\.0333\.4669\.6062\.20Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C410−410^\{\-4\}60\.7694\.1766\.3467\.4038\.9854\.5063\.69Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)–5⋅10−55\\cdot 10^\{\-5\}6\.0872\.1721\.0649\.5837\.017\.9032\.30Supra\-Mag\(BottomK,λ=0\.5\\lambda=0\.5\)–10−410^\{\-4\}60\.5150\.6754\.5348\.8236\.2220\.6045\.22Supra\-Mag\(BottomK,λ=0\.8\\lambda=0\.8\)–10−410^\{\-4\}64\.3097\.5068\.1167\.3236\.2262\.1065\.93

Table 20:Test perplexity\. Model:Meta\-Llama\-3\-8B; fine\-tuning dataset: Math17K; training budget: three epochs; rank\-equivalent budget:r0=8r\_\{0\}=8\. These are the same validation\-selected checkpoints as[Table˜19](https://arxiv.org/html/2607.09287#A3.T19)\. Full fine\-tuning is an unbudgeted reference row separated by rules\. Bold marks the lowest matched\-budget adapter average PPL\. TheCalib\.column gives the calibration source used to construct fixedWanda\-style sparse supports; “–” denotes methods that do not use a calibration pass\. Lower is better\. Perplexities are computed on up to 120 examples per benchmark\.MethodCalib\.Selected LRAddSubMultiArithSingleEqGSM8KAQuASVAMPAverageBase\(frozen\)––2\.152\.282\.301\.982\.642\.412\.22Full fine\-tuning–5⋅10−55\\cdot 10^\{\-5\}1\.031\.041\.041\.091\.171\.061\.07LoRA–10−410^\{\-4\}1\.061\.071\.081\.151\.341\.111\.12RoSA–5⋅10−55\\cdot 10^\{\-5\}1\.081\.101\.111\.191\.371\.151\.16SIFT\(TopK\)–5⋅10−55\\cdot 10^\{\-5\}1\.081\.101\.101\.191\.341\.141\.15SIFT\(RandK\)–10−410^\{\-4\}1\.101\.131\.151\.231\.441\.191\.19Super\(RandK\)–10−410^\{\-4\}1\.291\.161\.351\.351\.431\.401\.33Super\(TopK\)C45⋅10−55\\cdot 10^\{\-5\}1\.091\.121\.181\.231\.361\.211\.19Super\(BottomK\)C410−410^\{\-4\}1\.201\.201\.291\.331\.481\.361\.31Magnitude\(TopK\)–10−410^\{\-4\}1\.161\.131\.201\.261\.521\.241\.23Magnitude\(BottomK\)–10−410^\{\-4\}1\.091\.111\.121\.201\.361\.161\.17Supra\(TopK,λ=0\.3\\lambda=0\.3\)C45⋅10−55\\cdot 10^\{\-5\}1\.071\.081\.091\.171\.321\.121\.13Supra\(TopK,λ=0\.5\\lambda=0\.5\)C45⋅10−55\\cdot 10^\{\-5\}1\.071\.081\.091\.171\.331\.121\.13Supra\(TopK,λ=0\.8\\lambda=0\.8\)C45⋅10−55\\cdot 10^\{\-5\}1\.131\.141\.121\.181\.341\.171\.17Supra\(BottomK,λ=0\.3\\lambda=0\.3\)C410−410^\{\-4\}1\.061\.061\.081\.151\.271\.111\.12Supra\(BottomK,λ=0\.5\\lambda=0\.5\)C410−310^\{\-3\}1\.041\.051\.051\.111\.191\.081\.08Supra\(BottomK,λ=0\.8\\lambda=0\.8\)C410−410^\{\-4\}1\.061\.061\.081\.151\.291\.111\.12Supra\-Mag\(BottomK,λ=0\.3\\lambda=0\.3\)–5⋅10−55\\cdot 10^\{\-5\}1\.071\.081\.101\.181\.351\.131\.14Supra\-Mag\(BottomK,λ=0\.5\\lambda=0\.5\)–10−410^\{\-4\}1\.061\.061\.081\.151\.281\.111\.12Supra\-Mag\(BottomK,λ=0\.8\\lambda=0\.8\)–10−410^\{\-4\}1\.061\.061\.081\.151\.271\.111\.11