Building a Nested LADDER for LLM Unstructured Post-Training Pruning

# 1 Overview of LADDER. At each iteration, each eligible layer proposes a candidate pruning update using a fixed pruning step and a chosen local pruning backend. Each candidate is evaluated on calibration data, and the candidate with the least degradation per removed parameter is committed, producing an exactly nested mask trajectory. In our main experiments, the local backend is SparseGPT.
Source: [https://arxiv.org/html/2610.28773](https://arxiv.org/html/2610.28773)
Building a Nested LADDER for LLM Unstructured Post\-Training Pruning

Nicolas M\. Cuadrado ÁvilaIvo Gollini Navarrete

Martin TakáčSamuel HorváthMohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

![Refer to caption](https://arxiv.org/html/2610.28773v1/ladder_visual_abstract.png)Figure 1:Overview of LADDER\. At each iteration, each eligible layer proposes a candidate pruning update using a fixed pruning step and a chosen local pruning backend\. Each candidate is evaluated on calibration data, and the candidate with the least degradation per removed parameter is committed, producing an exactly nested mask trajectory\. In our main experiments, the local backend is SparseGPT\.###### Abstract

Non\-uniform allocation can improve post\-training pruning of large language models, yet quality can still degrade sharply at high sparsity\. To control degradation as sparsity increases, we introduce LADDER \(Layer\-WiseAllocation viaDamage\-DrivenEvaluation andRefinement\), which allocates sparsity incrementally\. At each step, it evaluates pruning an additional fraction of weights from each eligible layer and commits the layer update that causes the least end\-to\-end degradation per removed parameter\. This process produces non\-uniform sparsity profiles and exactly nested masks across multiple budgets\. We evaluate LADDER on five OPT, Llama, and Qwen models across1010–90%90\\%sparsity, using SparseGPT as its candidate\-generation backend and comparing against uniform and adaptive allocation baselines\. Results reveal a clear operating\-regime trade\-off: LADDER is comparable or slightly weaker at benign sparsity but becomes substantially more robust from70%70\\%onward\. This high\-sparsity advantage largely persists after posterior distillation across four of five models and translates into improved retention of aggregate zero\-shot accuracy\.

## 1Introduction

The strong performance of large language models \(LLMs\) has generally been attributed to parameter scaling, data volume, and increasing compute\([Kaplan et al\., 2020](https://arxiv.org/html/2610.28773#bib.bib32)\)\. However, as the costs of training and deploying large\-scale systems rise rapidly, some innovations become accessible only to well\-funded organizations\([Cottier et al\., 2025](https://arxiv.org/html/2610.28773#bib.bib40)\)\. For small\-scale systems, inference faces increasingly stringent constraints on memory, bandwidth, and latency, particularly on edge devices\([Qu et al\., 2025](https://arxiv.org/html/2610.28773#bib.bib41)\)\. At the same time, we observe a growing availability of open pre\-trained models, including the Qwen series\([Yang et al\., 2025a](https://arxiv.org/html/2610.28773#bib.bib15)\), Kimi[Moonshot\-AI \(2026\)](https://arxiv.org/html/2610.28773#bib.bib23), and DeepSeek[DeepSeek\-AI \(2026\)](https://arxiv.org/html/2610.28773#bib.bib22)\. As a result, post\-training compression becomes more attractive as training a model from scratch becomes more restricted\.

The “train\-large\-then\-compress” approach offers benefits beyond deployment convenience\.[Chang et al\. \(2021\)](https://arxiv.org/html/2610.28773#bib.bib24)shows that in high\-dimensional settings, training an overparameterized model and then pruning it can outperform training a smaller model from scratch, even when the relevant features are known\. This supports overparameterization and pre\-trained LLMs as a basis for creating efficient subnetworks\.

Numerous techniques have been developed for post\-training compression, including quantization\([Micikevicius et al\., 2022](https://arxiv.org/html/2610.28773#bib.bib13)\), knowledge distillation\([Xu et al\., 2024b](https://arxiv.org/html/2610.28773#bib.bib2)\), low\-rank approximation\([Denton et al\., 2014](https://arxiv.org/html/2610.28773#bib.bib14)\), and pruning\([Cheng et al\., 2024](https://arxiv.org/html/2610.28773#bib.bib39)\)\. Within the latter approach, unstructured pruning is a well\-studied approach that removes individual parameters without modifying the model architecture\.

Recent advancements in sparse representations and kernels have translated unstructured pruning into practical efficiency\. Notably, the MACKO\-SpMV kernel demonstrates that runtime and storage benefits improve as sparsity increases, reporting both memory reduction and enhanced inference speed for a 50% sparse LLM\([Macko and Boža, 2026](https://arxiv.org/html/2610.28773#bib.bib30)\)\. Alongside broader developments in GPU kernels\([Spector et al\., 2024](https://arxiv.org/html/2610.28773#bib.bib21)\), the findings suggest the real\-life feasibility of aggressive unstructured pruning\.

Early influential approaches for LLMs, such as SparseGPT\([Frantar and Alistarh, 2023](https://arxiv.org/html/2610.28773#bib.bib37)\)and Wanda\([Sun et al\., 2023](https://arxiv.org/html/2610.28773#bib.bib34)\), proposed effective methods for one\-shot pruning via local decisions about which weights to remove within each layer, under a uniform layer\-wise sparsity target\. Nevertheless, recent research has shown that a non\-uniform allocation of pruning across layers can lead to better performance, as redundancy varies with depth\([Tang et al\., 2026](https://arxiv.org/html/2610.28773#bib.bib11);[Gao et al\., 2025](https://arxiv.org/html/2610.28773#bib.bib36);[Le et al\., 2025](https://arxiv.org/html/2610.28773#bib.bib43)\)\. For instance,[Sun et al\. \(2026\)](https://arxiv.org/html/2610.28773#bib.bib42)demonstrated that some deep layers can be completely removed with minimal negative impact, a phenomenon they refer to as theCurse of Depth\. Despite these advances, existing non\-uniform methods deteriorate sharply under aggressive sparsity, motivating the search for methods that can mitigate the detrimental impact in the high\-sparsity regime and push the Pareto frontier\.

We introduce LADDER, a post\-training wrapper that controls non\-uniform sparsity allocation by repeatedly selecting the layer whose next pruning step causes the least end\-to\-end degradation\. This process preserves quality as sparsity increases and, as a structural consequence, produces nested mask supports across sparsity budgets\. We provide an overview in Figure[1](https://arxiv.org/html/2610.28773#S0.F1)\.

Our main contributions are: \(1\) We introduce a conservative approach to non\-uniform allocation that assigns sparsity incrementally according to the marginal end\-to\-end degradation measured from the current model state\. \(2\) We instantiate this approach as LADDER, a greedy deterministic backend\-agnostic protocol whose monotone pruning decisions additionally produce exactly nested mask supports across sparsity budgets\. \(3\) Across five OPT, Llama, and Qwen models, we identify an operating\-regime trade\-off: LADDER becomes substantially more robust under aggressive pruning, with this advantage largely surviving posterior distillation and translating into improved aggregate accuracy retention across seven zero\-shot downstream tasks\.

## 2Background

For each prunable layerℓ∈\{1,…,L\}\\ell\\in\\\{1,\\dots,L\\\}in a pre\-trained model, let𝑾ℓ∈ℝdout,ℓ×din,ℓ\\bm\{W\}\_\{\\ell\}\\in\{\\mathbb\{R\}\}^\{d\_\{\\mathrm\{out\},\\ell\}\\times d\_\{\\mathrm\{in\},\\ell\}\}denote the dense parameter matrix,𝑿ℓ∈ℝdin,ℓ×n\\bm\{X\}\_\{\\ell\}\\in\{\\mathbb\{R\}\}^\{d\_\{\\mathrm\{in\},\\ell\}\\times n\}the matrix of calibration activations,𝑴ℓ∈\{0,1\}dout,ℓ×din,ℓ\\bm\{M\}\_\{\\ell\}\\in\\\{0,1\\\}^\{d\_\{\\mathrm\{out\},\\ell\}\\times d\_\{\\mathrm\{in\},\\ell\}\}a binary pruning mask, and𝑾~ℓ\\widetilde\{\\bm\{W\}\}\_\{\\ell\}the post\-pruning parameters after any optional local compensation or reconstruction step\. We define the layer\-local degradation functional

ℰℓ​\(𝑴ℓ,𝑾~ℓ,𝑿ℓ\)=‖𝑾ℓ​𝑿ℓ−\(𝑴ℓ⊙𝑾~ℓ\)​𝑿ℓ‖F2,\\displaystyle\\textstyle\\mathcal\{E\}\_\{\\ell\}\(\\bm\{M\}\_\{\\ell\},\\widetilde\{\\bm\{W\}\}\_\{\\ell\};\\bm\{X\}\_\{\\ell\}\)=\\left\\lVert\\bm\{W\}\_\{\\ell\}\\bm\{X\}\_\{\\ell\}\-\(\\bm\{M\}\_\{\\ell\}\\odot\\widetilde\{\\bm\{W\}\}\_\{\\ell\}\)\\bm\{X\}\_\{\\ell\}\\right\\rVert\_\{\\mathrm\{F\}\}^\{2\},\(1\)as a standardized notation\. Although not every method optimizesℰℓ\\mathcal\{E\}\_\{\\ell\}directly, it provides a common notation to compare pruning methods through the choice of mask selection \(𝑴ℓ\\bm\{M\}\_\{\\ell\}\), weight reconstruction \(𝑾~ℓ\\widetilde\{\\bm\{W\}\}\_\{\\ell\}\), and sparsity allocation\.

We distinguish between*global*pruning, which distributes a total sparsity budget across layers, and*local*pruning, which focuses on pruning a specific layer after assigning it a sparsity level\.

Global Pruning\.This approach formulates pruning as a model\-level optimization problem\. Given a target sparsity, the goal is to identify the subset of parameters whose removal minimizes the expected degradation in model performance, regardless of the layers to which those parameters belong\. A commonly used tractable surrogate approximates the end\-to\-end objective with the sum of layer\-local reconstruction errors:

min\{𝑴ℓ,𝑾~ℓ\}ℓ=1L\\displaystyle\\textstyle\\min\_\{\\\{\\bm\{M\}\_\{\\ell\},\\widetilde\{\\bm\{W\}\}\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}\}\\quad∑ℓ=1Lℰℓ​\(𝑴ℓ,𝑾~ℓ,𝑿ℓ\)\\displaystyle\\sum\_\{\\ell=1\}^\{L\}\\mathcal\{E\}\_\{\\ell\}\(\\bm\{M\}\_\{\\ell\},\\widetilde\{\\bm\{W\}\}\_\{\\ell\};\\bm\{X\}\_\{\\ell\}\)subject to∑ℓ=1L‖𝑴ℓ‖0≤\(1−S\)​N,\\displaystyle\\sum\_\{\\ell=1\}^\{L\}\\left\\lVert\\bm\{M\}\_\{\\ell\}\\right\\rVert\_\{0\}\\leq\(1\-S\)N,wherenℓ=dout,ℓ×din,ℓn\_\{\\ell\}=d\_\{\\mathrm\{out\},\\ell\}\\times d\_\{\\mathrm\{in\},\\ell\}is the number of parameters in a given layer,N=∑ℓ=1LnℓN=\\sum\_\{\\ell=1\}^\{L\}n\_\{\\ell\}the total parameters in a model, andS∈\[0,1\)S\\in\[0,1\)the target global sparsity\. The global problem becomes one of minimizing the degradation by choosing the optimal vector\{sℓ\}ℓ=1L\\\{s\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}, wheresℓs\_\{\\ell\}is the realized local sparsity\.

Local Pruning\.On the other hand, this approach starts with a fixed layer\-wise sparsitysℓs\_\{\\ell\}and asks how to achieve that local budget with minimal damage\. We can express this problem as a constrained approximation problem:

min𝑴ℓ,𝑾~ℓ\\displaystyle\\textstyle\\min\_\{\\bm\{M\}\_\{\\ell\},\\widetilde\{\\bm\{W\}\}\_\{\\ell\}\}\\quadℰℓ​\(𝑴ℓ,𝑾~ℓ,𝑿ℓ\)\\displaystyle\\mathcal\{E\}\_\{\\ell\}\(\\bm\{M\}\_\{\\ell\},\\widetilde\{\\bm\{W\}\}\_\{\\ell\};\\bm\{X\}\_\{\\ell\}\)subject to‖𝑴ℓ‖0≤\(1−sℓ\)​nℓ\.\\displaystyle\\left\\lVert\\bm\{M\}\_\{\\ell\}\\right\\rVert\_\{0\}\\leq\(1\-s\_\{\\ell\}\)n\_\{\\ell\}\.Because jointly optimizing the pruning mask and the surviving weight values is intractable, existing approaches typically address only one of these variables\.

Score\-based methods focus on finding the optimal mask𝑴ℓ⋆\\bm\{M\}\_\{\\ell\}^\{\\star\}by solving

arg​min𝐌ℓ:‖𝐌ℓ‖0≤\(1−sℓ\)​nℓmin𝐖~ℓℰℓ\(𝐌ℓ,𝐖~ℓ;𝐗ℓ\),\\displaystyle\\textstyle\\argmin\_\{\\bm\{M\}\_\{\\ell\}:\\,\\left\\lVert\\bm\{M\}\_\{\\ell\}\\right\\rVert\_\{0\}\\leq\(1\-s\_\{\\ell\}\)n\_\{\\ell\}\}\\;\\min\_\{\\widetilde\{\\bm\{W\}\}\_\{\\ell\}\}\\mathcal\{E\}\_\{\\ell\}\(\\bm\{M\}\_\{\\ell\},\\widetilde\{\\bm\{W\}\}\_\{\\ell\};\\bm\{X\}\_\{\\ell\}\),using a tractable saliency ranking, without caring about the surviving weight values\. A good example are the inflentuial methods OBD\([LeCun et al\., 1989](https://arxiv.org/html/2610.28773#bib.bib45)\)and OBS\([Hassibi et al\., 1993](https://arxiv.org/html/2610.28773#bib.bib46)\)which use scalable second\-order information, an idea that has carried into modern settings\([Navarrete et al\., 2026](https://arxiv.org/html/2610.28773#bib.bib12)\)\. In contrast, reconstruction\-based methods focus on finding the optimal surviving weight values𝑾~ℓ⋆\\widetilde\{\\bm\{W\}\}\_\{\\ell\}^\{\\star\}conditional on a selected or progressively constructed mask:

arg​min𝐖~ℓ\\displaystyle\\textstyle\\argmin\_\{\\widetilde\{\\bm\{W\}\}\_\{\\ell\}\}\\quadℰℓ​\(𝑴ℓ⋆,𝑾~ℓ,𝑿ℓ\)\\displaystyle\\mathcal\{E\}\_\{\\ell\}\(\\bm\{M\}\_\{\\ell\}^\{\\star\},\\widetilde\{\\bm\{W\}\}\_\{\\ell\};\\bm\{X\}\_\{\\ell\}\)subject tosupp⁡\(𝑾~ℓ\)⊆supp⁡\(𝑴ℓ⋆\)\.\\displaystyle\\mathrm\{supp\}\(\\widetilde\{\\bm\{W\}\}\_\{\\ell\}\)\\subseteq\\mathrm\{supp\}\(\\bm\{M\}\_\{\\ell\}^\{\\star\}\)\.Rather than focusing on mask quality, this approach aims to reconstruct the surviving parameters to better maintain the activation dynamics before pruning\.

### 2\.1Pioneering Pruning Methods

Given a prescribed layer\-wise sparsity profile, magnitude pruning removes weights with the smallest absolute values\([Hagiwara, 1994](https://arxiv.org/html/2610.28773#bib.bib35);[Han et al\., 2015](https://arxiv.org/html/2610.28773#bib.bib31)\), while Wanda incorporates input\-activation norms into this ranking\([Sun et al\., 2023](https://arxiv.org/html/2610.28773#bib.bib34)\)\. Both are score\-based methods that leave surviving values unchanged\. SparseGPT instead uses approximate second\-order information to select removals while reconstructing surviving weights to preserve layer outputs\([Frantar and Alistarh, 2023](https://arxiv.org/html/2610.28773#bib.bib37)\)\. Together, these methods serve as standard reference points for evaluating post\-training unstructured pruning under prescribed layer\-wise sparsity targets\.

### 2\.2Layer\-Wise Pruning Allocation Methods

Various studies emphasize that compression should be strategically allocated rather than uniformly across layers\. OPQ determines layer\-wise pruning ratios and quantization levels from pretrained parameter distributions\([Hu et al\., 2021](https://arxiv.org/html/2610.28773#bib.bib26)\)\. DPFPS progressively adapts pruning ratios based on layer sensitivity during CNN training\([Ruan et al\., 2021](https://arxiv.org/html/2610.28773#bib.bib25)\)\. For LLMs, OWL applies non\-uniform layer sparsities based on activation\-outlier ratios\([Yin et al\., 2024](https://arxiv.org/html/2610.28773#bib.bib10)\), while DSA optimizes sparsity allocations based on layer statistics\([Li et al\., 2024](https://arxiv.org/html/2610.28773#bib.bib3)\)\. AlphaPruning leverages heavy\-tailed spectral metrics\([Lu et al\., 2024](https://arxiv.org/html/2610.28773#bib.bib7)\), and Lua\-LLM focuses on unstructured sparsity allocation\([Lu et al\., 2025](https://arxiv.org/html/2610.28773#bib.bib4)\)\. LSA employs minimal linear reconstruction error for sparsity assignment\([Yang et al\., 2025b](https://arxiv.org/html/2610.28773#bib.bib1)\), whereas BESA learns blockwise allocations with a differentiable reconstruction objective\([Xu et al\., 2024a](https://arxiv.org/html/2610.28773#bib.bib6)\)\. EvoPress utilizes evolutionary search to optimize dynamic compression profiles, accounting for interdependencies among decisions\([Sieberling et al\., 2025](https://arxiv.org/html/2610.28773#bib.bib5)\)\. SparseLLM coordinates pruning to address error propagation across transformer layers, albeit with homogeneous sparsity assumptions\([Bai et al\., 2024](https://arxiv.org/html/2610.28773#bib.bib38)\)\. Lastly, OATS combines sparse and low\-rank approximations using a specified compression profile\([Zhang and Papyan, 2024](https://arxiv.org/html/2610.28773#bib.bib29)\)\.

### 2\.3Elastic Model Families

A complementary line of work studies elastic neural networks, which construct nested subnetworks with shared parameters, enabling a single model to support multiple deployment budgets\. Slimmable Networks introduced runtime\-adjustable widths with independent normalization layers\([Yu et al\., 2018](https://arxiv.org/html/2610.28773#bib.bib20)\), while Universally Slimmable Networks extended this idea to support arbitrary widths\([Yu and Huang, 2019](https://arxiv.org/html/2610.28773#bib.bib17)\)\. More recently, these principles have been applied to adaptive\-width and depth language models\([Hou et al\., 2020](https://arxiv.org/html/2610.28773#bib.bib19)\), heterogeneous federated learning\([Horváth et al\., 2021](https://arxiv.org/html/2610.28773#bib.bib44)\), and nested Transformers such as MatFormer\([Devvrit et al\., 2024](https://arxiv.org/html/2610.28773#bib.bib33)\)and Flextron\([Cai et al\., 2024](https://arxiv.org/html/2610.28773#bib.bib18)\)\. Moreover, FlexRank demonstrated that nested models can be extracted from pre\-trained models using low\-rank decompositions\([Zaccone et al\., 2026](https://arxiv.org/html/2610.28773#bib.bib16)\)\.

### 2\.4Sparse\-Model Refinement and Knowledge Transfer

Knowledge distillation has become a standard approach for recovering performance after pruning\. CAP transfers knowledge from dense models and intermediate pruning states, showing that progressively sparser models can provide effective supervision during iterative pruning\([Xu et al\., 2022](https://arxiv.org/html/2610.28773#bib.bib27)\)\. AST similarly combines post\-pruning retraining with dense\-teacher supervision to improve the quality of semi\-structured sparse LLMs under limited data and computation budgets\([Huang et al\., 2025](https://arxiv.org/html/2610.28773#bib.bib28)\)\. All of these methods apply distillation for a single sparse endpoint\.

Algorithm 1LADDER single\-target allocation1:committed masks

\{𝑴ℓ\}\\\{\\bm\{M\}\_\{\\ell\}\\\}and weights

\{𝑾ℓ\}\\\{\\bm\{W\}\_\{\\ell\}\\\}
2:calibration data

3:target

SS, local backend,

γ\\gamma,

zminz\_\{\\min\}
4:Set round counter

r←0r\\leftarrow 0
5:whileglobal sparsity

<S<Sdo

6:Evaluate the committed model to obtain

mbase\(r\)m\_\{\\mathrm\{base\}\}^\{\(r\)\}
7:for allprunable layers

ℓ\\elldo

8:Compute

zℓ\(r\)z\_\{\\ell\}^\{\(r\)\}
9:if

zℓ\(r\)\>0z\_\{\\ell\}^\{\(r\)\}\>0then

10:Build candidate

\(𝑴ℓ~\(r\),𝑾~ℓ\(r\)\)\(\\widetilde\{\\bm\{M\}\_\{\\ell\}\}^\{\(r\)\},\\widetilde\{\\bm\{W\}\}\_\{\\ell\}^\{\(r\)\}\)
11:Evaluate the candidate and get

qℓ\(r\)q\_\{\\ell\}^\{\(r\)\}
12:endif

13:endfor

14:Choose

ℓ⋆←arg​minℓ:zℓ\(r\)\>0qℓ\(r\)\\ell^\{\\star\}\\leftarrow\\argmin\_\{\\ell:z\_\{\\ell\}^\{\(r\)\}\>0\}q\_\{\\ell\}^\{\(r\)\}
15:Commit

\(𝑴ℓ⋆,𝑾ℓ⋆\)←\(𝑴ℓ⋆~\(r\),𝑾~ℓ⋆\(r\)\)\(\\bm\{M\}\_\{\\ell^\{\\star\}\},\\bm\{W\}\_\{\\ell^\{\\star\}\}\)\\leftarrow\(\\widetilde\{\\bm\{M\}\_\{\\ell^\{\\star\}\}\}^\{\(r\)\},\\widetilde\{\\bm\{W\}\}\_\{\\ell^\{\\star\}\}^\{\(r\)\}\)
16:Increment

r←r\+1r\\leftarrow r\+1
17:endwhile

18:return

\{𝑴ℓ,𝑾ℓ\}ℓ=1L\\\{\\bm\{M\}\_\{\\ell\},\\bm\{W\}\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}

Figure 2:Pre\-distillation relative loss advantage of LADDER over each baseline\. The bars are the arithmetic mean of the relative advantages on WikiText\-2, PTB, and C4\. Positive values favor LADDER\.

## 3Methodology

### 3\.1Problem Setup

We consider the layer\-wise sparsity profile as the variable to be optimized\. Let

𝒔=\(s1,…,sL\)∈\[0,1\]L\\displaystyle\\textstyle\\bm\{s\}=\(s\_\{1\},\\dots,s\_\{L\}\)\\in\[0,1\]^\{L\}represent the vector of local sparsities, andSSthe desired global sparsity\. We can formulate the allocation problem as follows:

min𝒔∈\[0,1\]L⁡Φ⁡\(𝒔\)s\.t\.∑ℓ=1Lnℓ​sℓ=S​N,\\displaystyle\\textstyle\\min\_\{\\bm\{s\}\\in\[0,1\]^\{L\}\}\\Phi\(\\bm\{s\}\)\\quad\\text\{s\.t\.\}\\quad\\sum\_\{\\ell=1\}^\{L\}n\_\{\\ell\}s\_\{\\ell\}=SN,\(2\)whereΦ⁡\(𝒔\)\\Phi\(\\bm\{s\}\)is the end\-to\-end calibration degradation induced by a layer\-wise sparsity profile\. Here,NNcounts only the parameters in the prunable linear layers, soSSis global sparsity over that declared pruning scope rather than over every model parameter\. Under a differentiable relaxation, any interior optimum, wheresℓ⋆∈\(0,1\)s\_\{\\ell\}^\{\\star\}\\in\(0,1\)for every layer, must satisfy the following first\-order condition:

1nℓ​∂Φ⁡\(𝒔⋆\)∂sℓ=μfor all prunable layers​ℓ,\\displaystyle\\textstyle\\frac\{1\}\{n\_\{\\ell\}\}\\frac\{\\partial\\Phi\(\\bm\{s\}^\{\\star\}\)\}\{\\partial s\_\{\\ell\}\}=\\mu\\qquad\\text\{for all prunable layers \}\\ell,for some shared multiplierμ\\mu\. Thus, the marginal end\-to\-end damage per additional removed parameter must be equal across eligible layers\.

Uniform sparsity satisfies this condition only when those per\-parameter marginal damages happen to agree at the same local sparsity, which is generally not the case\. In practice, the sparsity profile is realized through binary keep\-or\-remove decisions, and pruning effects can interact across layers rather than vary smoothly or independently\. Consequently, the appropriate non\-uniform profile generally cannot be determined from fixed, independently estimated layer costs\.

### 3\.2Layer\-WiseAllocation

We introduce LADDER, which approximates the state\-dependent allocation through controlled discrete updates, re\-evaluating the available layer candidates after each committed step\. As shown in Figure[1](https://arxiv.org/html/2610.28773#S0.F1), the process begins with the dense model and a per\-layer score mask profile initialized to no pruning, represented as\{𝑴ℓ\(0\)=𝟏\}ℓ=1L\\\{\\bm\{M\}\_\{\\ell\}^\{\(0\)\}=\\mathbf\{1\}\\\}\_\{\\ell=1\}^\{L\}\.

At roundrr, every prunable layer proposes a candidate mask𝑴ℓ~\(r\)\\widetilde\{\\bm\{M\}\_\{\\ell\}\}^\{\(r\)\}that removes thezℓ\(r\)z\_\{\\ell\}^\{\(r\)\}least important parameters that survived from the current committed mask𝑴ℓ\(r\)\\bm\{M\}\_\{\\ell\}^\{\(r\)\}\. Layers for whichzℓ\(r\)=0z\_\{\\ell\}^\{\(r\)\}=0will no longer generate candidates, allowing the candidate pool to diminish in subsequent rounds\. The candidate set for roundrris defined as follows:

𝒞\(r\)=\{𝑴ℓ~\(r\)∣1≤ℓ≤L,zℓ\(r\)\>0\}\.\\displaystyle\\textstyle\\mathcal\{C\}^\{\(r\)\}=\\left\\\{\\widetilde\{\\bm\{M\}\_\{\\ell\}\}^\{\(r\)\}\\mid 1\\leq\\ell\\leq L,\\ z\_\{\\ell\}^\{\(r\)\}\>0\\right\\\}\.We evaluate all candidates using the same committed model state and a degradation criterion\. The committed mask profile is updated only with the one that shows the least degradation during each round\. This repeats until reaching the global target sparsitySS\.

### 3\.3Damage\-DrivenEvaluation

Importance Metric\.Our method uses existing local pruning methods as a backend to generate candidate masks\. Magnitude pruning and Wanda provide parameter scores, while SparseGPT additionally reconstructs the retained weights\.

For state\-dependent backends, the required statistics are refreshed from the current committed model: Wanda recomputes its activation\-dependent scores, and SparseGPT updates its curvature information\. The candidate for layerℓ\\ellat roundrris the pair\(𝑴ℓ~\(r\),𝑾~ℓ\(r\)\)\(\\widetilde\{\\bm\{M\}\_\{\\ell\}\}^\{\(r\)\},\\widetilde\{\\bm\{W\}\}\_\{\\ell\}^\{\(r\)\}\)\. For score\-only backends,𝑾~ℓ\(r\)=𝑾ℓ\(r\)⊙𝑴ℓ~\(r\)\\widetilde\{\\bm\{W\}\}\_\{\\ell\}^\{\(r\)\}=\\bm\{W\}\_\{\\ell\}^\{\(r\)\}\\odot\\widetilde\{\\bm\{M\}\_\{\\ell\}\}^\{\(r\)\}\.

The candidate’s support must always be a subset of the committed support\. Reconstruction\-based backends may alter surviving values, while masked coordinates stay zero\. Only the selected candidate remains\.

Defining the number of surviving parameters in layerℓ\\ellat roundrrasKℓ\(r\)=∥𝑴ℓ\(r\)∥0K\_\{\\ell\}^\{\(r\)\}=\\lVert\\bm\{M\}\_\{\\ell\}^\{\(r\)\}\\rVert\_\{0\}, the remaining pruning budget is

R\(r\)=⌊S​N⌋−∑ℓ=1L\(nℓ−Kℓ\(r\)\),\\displaystyle\\textstyle R^\{\(r\)\}=\\left\\lfloor SN\\right\\rfloor\-\\sum\_\{\\ell=1\}^\{L\}\\left\(n\_\{\\ell\}\-K\_\{\\ell\}^\{\(r\)\}\\right\),allowing us to define the candidate step for that layer:

zℓ\(r\)=min⁡\(R\(r\),Kℓ\(r\),max⁡\(zmin,⌈γ​nℓ⌉\)\)\.\\displaystyle\\textstyle z\_\{\\ell\}^\{\(r\)\}=\\min\\\!\\left\(R^\{\(r\)\},\\;K\_\{\\ell\}^\{\(r\)\},\\;\\max\\\!\\left\(z\_\{\\min\},\\left\\lceil\\gamma n\_\{\\ell\}\\right\\rceil\\right\)\\right\)\.Here,γ\\gammasets the nominal step as a fraction of the layer size, whilezminz\_\{\\min\}prevents excessively small updates\.

Degradation Criterion\.We defineΔℓ\(r\)\\Delta\_\{\\ell\}^\{\(r\)\}as the loss in model quality induced by a candidate:

Δℓ\(r\)=\{mcand\(r,ℓ\)−mbase\(r\),for lower\-is\-better metrics,mbase\(r\)−mcand\(r,ℓ\),for higher\-is\-better metrics\.\\displaystyle\\textstyle\\Delta\_\{\\ell\}^\{\(r\)\}=\\begin\{cases\}m\_\{\\mathrm\{cand\}\}^\{\(r,\\ell\)\}\-m\_\{\\mathrm\{base\}\}^\{\(r\)\},&\\text\{for lower\-is\-better metrics\},\\\\ m\_\{\\mathrm\{base\}\}^\{\(r\)\}\-m\_\{\\mathrm\{cand\}\}^\{\(r,\\ell\)\},&\\text\{for higher\-is\-better metrics\}\.\\end\{cases\}Here,mbase\(r\)m\_\{\\mathrm\{base\}\}^\{\(r\)\}is the primary metric of the committed model andmcand\(r,ℓ\)m\_\{\\mathrm\{cand\}\}^\{\(r,\\ell\)\}its value after temporarily activating the candidate for layerℓ\\ell\.

Candidates are ranked by relative degradation per added zero,

qℓ\(r\)=Δℓ\(r\)zℓ\(r\)\.\\displaystyle\\textstyle q\_\{\\ell\}^\{\(r\)\}=\\frac\{\\Delta\_\{\\ell\}^\{\(r\)\}\}\{z\_\{\\ell\}^\{\(r\)\}\}\.Ties are broken by raw degradation, then by the larger candidate step, and finally by stable layer order\. For each round, the selected layer solves

ℓ⋆=arg​minℓ:zℓ\(r\)\>0qℓ\(r\)\.\\displaystyle\\textstyle\\ell^\{\\star\}=\\argmin\_\{\\ell:\\,z\_\{\\ell\}^\{\(r\)\}\>0\}q\_\{\\ell\}^\{\(r\)\}\.Algorithm[1](https://arxiv.org/html/2610.28773#alg1)provides a detailed overview of our method\. The step fractionγ\\gammadictates a straightforward trade\-off: smaller steps lead to more precise allocation decisions but require more rounds to conclude\. Since candidates are assessed based on the current committed state, each decision incorporates the interactions evaluated up to that point\. However, the greedy trajectory may overlook options that could become more favorable after certain removals; thus, while exact nesting provides structural assurance, it does not guarantee optimality\.

Figure 3:Language\-modeling test loss before \(solid\) and after \(dashed\) posterior distillation on C4 and WikiText\-2\. Dataset panels are paired by model; the shared right axis gives the corresponding perplexity\.
### 3\.4Refinement

Nestedness\.We process requested sparsities in increasing order, initializing each run from the preceding checkpoint and restricting candidates to its support\. Consequently, forsa<sbs\_\{a\}<s\_\{b\}, the resulting masks satisfyMℓ\(b\)⊆Mℓ\(a\)M\_\{\\ell\}^\{\(b\)\}\\subseteq M\_\{\\ell\}^\{\(a\)\}\. Reconstruction\-based backends may still change surviving weight values between checkpoints\. We empirically verify zero support violations\. The complete analysis is provided in Appendix[A\.6](https://arxiv.org/html/2610.28773#A1.SS6)\.

Posterior Distillation\.Following prior work\([Xu et al\., 2022](https://arxiv.org/html/2610.28773#bib.bib27);[Huang et al\., 2025](https://arxiv.org/html/2610.28773#bib.bib28)\), we initialize one shared student from the pretrained dense model\. For each batch, all fixed sparsity masks are evaluated against the same dense\-teacher outputs, their temperature\-scaled KL losses are averaged, and one optimizer step updates the shared parameters\. We apply the same procedure to every baseline\.

Figure 4:Distilled zero\-shot accuracy across seven downstream tasks\. Rows share task accuracy limits across models\. Shaded regions show the task standard errors reported by the evaluation harness\. Horizontal references mark dense and random performance\.

## 4Experiments & Results

Our experiments test whether continuously reassessing the least harmful pruning step is more beneficial as sparsification becomes more aggressive\. We first identify when this approach outperforms state\-of\-the\-art non\-uniform methods\. Then, we investigate if this advantage carries through the distillation process to downstream applications\. Lastly, we verify that LADDER produces nested masks and use ablations to study how our allocation interacts with different local pruning backends, and whether continuing from masks produced by other non\-uniform methods can control their subsequent loss growth\.

### 4\.1Experimental Setup

Protocol\.We evaluate OPT\-125M, OPT\-350M, OPT\-1\.3B\([Zhang et al\., 2022](https://arxiv.org/html/2610.28773#bib.bib9)\), Qwen3\-0\.6B, and Llama\-3\.2\-1B\. We use the standard language\-modeling benchmarks WikiText\-2, PTB, and C4[Frantar and Alistarh \(2023\)](https://arxiv.org/html/2610.28773#bib.bib37);[Sun et al\. \(2023\)](https://arxiv.org/html/2610.28773#bib.bib34), to allow direct comparison with prior work\.

Furthermore, we evaluate seven downstream tasks to test diverse zero\-shot capabilities\. Following the pruning protocol from[Frantar and Alistarh \(2023\)](https://arxiv.org/html/2610.28773#bib.bib37), we prune all linear layers in the transformer blocks, excluding biases\. We evaluate requested global sparsities from 10% to 90% and focus the main comparisons on 50–90%, where sparse kernels have shown practical gains\([Macko and Boža, 2026](https://arxiv.org/html/2610.28773#bib.bib30)\)\.

Methods\.We compare LADDER against the non\-uniform allocation methods OWL, DSA, and LSA\. All methods use SparseGPT as a common local pruning backend, keeping local mask selection and weight reconstruction fixed to isolate the effect of the global allocation strategy\. All methods use the same 128 sequences of 2,048 tokens sampled from C4 for calibration; for LADDER, perplexity on this calibration sample additionally serves as the candidate\-selection criterion\. Method\-specific configurations and hyperparameters are provided in Appendix[A\.1](https://arxiv.org/html/2610.28773#A1.SS1)\.

Evaluation\.We report corpus\-level mean next\-token cross\-entropy and perplexity on WikiText\-2, PTB, and C4, together with zero\-shot accuracy on seven downstream tasks\. We additionally evaluate posterior distillation of the resulting sparsity\-specific masks using a shared student initialized from the pretrained dense weights\. Complete language\-modeling and distillation settings are provided in Appendix[A\.1](https://arxiv.org/html/2610.28773#A1.SS1), while downstream evaluation details are given in Appendix[A\.2](https://arxiv.org/html/2610.28773#A1.SS2)\.

### 4\.2Language\-Modeling Across Regimes

Figure[2](https://arxiv.org/html/2610.28773#S2.F2)shows an operating\-regime effect: the non\-uniform baselines are often competitive at lower sparsity, results are mixed around 50–60%, and LADDER separates most consistently from 70% onward before distillation\. The bars preserve the magnitude of the model\-level advantages across all five models, while Table[1](https://arxiv.org/html/2610.28773#S4.T1)reports the corresponding 70–90% loss and perplexity values for the three models highlighted in the main figures\. The complete five\-model table is deferred to the Appendix[A\.4](https://arxiv.org/html/2610.28773#A1.SS4)\.

Table 1:Pre\-distillation language\-modeling quality\. Each entry reports test loss followed by perplexity in parentheses for each dataset\. Lower is better, and bold indicates the best method for each model, dataset, and sparsity level\.Figure[3](https://arxiv.org/html/2610.28773#S3.F3)presents the pre\- and post\-distillation trajectories on the same scale\. The distillation process enhances all methods and significantly reduces their differences, particularly at moderate levels of sparsity, while still preserving the general trend observed at higher levels of sparsity\. However, the post\-distillation ordering remains architecture\-dependent: on Llama\-3\.2\-1B, OWL and LSA are competitive in some high\-sparsity settings\. Overall, distillation narrows the differences without eliminating architecture\-dependent behavior at high sparsity\.

### 4\.3Downstream Transfer

Figure[4](https://arxiv.org/html/2610.28773#S3.F4)presents the complete matched downstream comparison for Qwen3\-0\.6B, Llama\-3\.2\-1B, and OPT\-1\.3B\. The shaded regions show the task\-level bootstrap standard errors reported bylm\-eval\.

Across models and tasks, LADDER generally retains accuracy better as sparsity increases, although the strength and ordering of the methods vary across tasks\. When accuracy is macro\-averaged across the seven tasks, LADDER generally shows lower degradation from dense performance in the high\-sparsity regime\. Appendix[A\.2](https://arxiv.org/html/2610.28773#A1.SS2)reports the complete task\-averaged comparison\.

Several curves converge near task\-specific performance floors, so large language\-modeling improvements do not always translate proportionally to downstream accuracy\. We therefore interpret the results as evidence of more consistent downstream retention across the evaluated models and tasks, rather than dominance in every configuration\.

### 4\.4Ablations

Figure 5:Backend ablation on OPT\-125M\. For each local pruning rule, we compare the standalone method with LADDER allocation\.Figure[5](https://arxiv.org/html/2610.28773#S4.F5)shows how LADDER enhances local pruning metrics through its allocation process and identifies the most effective backend rule\. Withγ=0\.20\\gamma=0\.20, LADDER lowers test loss in 41 of 45 backend–dataset–sparsity comparisons\. It outperforms the corresponding standalone backend across all comparisons at 70%, 80%, and 90% sparsity\. Notably, SparseGPT significantly outperforms other backends, being the reason it was selected for the main experiments\.

Figure 6:Initializer ablation on OPT\-125M, averaged over WikiText\-2, PTB, and C4\. “Continued” applies LADDER from each baseline’s 70% mask\. Dashed lines denote standard LADDER\.Figure[6](https://arxiv.org/html/2610.28773#S4.F6)shows that LADDER enhances existing non\-uniform allocation when loss sharply increases\. Starting from the baselines’ 70% masks, our method achieves lower\-loss checkpoints at 80% and 90% compared to baseline masks directly generated at those sparsities\. This suggests that a cost\-effective allocation can initiate moderate sparsity, with LADDER reserved for challenging high\-sparsity adjustments\. Among the tested allocations, OWL offers the strongest initialization, closely aligning with the dense\-to\-sparse LADDER trajectory at 80% sparsity\. However, the standard dense\-to\-sparse LADDER outperforms at 90%, highlighting how initial allocation decisions can restrict later performance recovery\.

We further evaluate the role of the proposed nested trajectory by initializing LADDER for each target sparsity level from the dense checkpoint instead of from the preceding support\. We observed that the performance of both the independent and progressive variants is comparable at moderate sparsity\. However, at the highest sparsity, the nested trajectory reduces perplexity by 10\.6%, 8\.6%, and 15\.6% on WikiText\-2, PTB, and C4, respectively\. Importantly, the complete nested trajectory from 10% to 90% requires 333 search rounds, approximately the same as the 327 rounds required for a single independently optimized 90% mask, while simultaneously producing all intermediate sparsity checkpoints\. These results indicate that nesting not only reduces costs relative to an independent search but also provides a better starting point for a high\-sparsity search\.

### 4\.5Scalability and Deployment

LADDER is a one\-time post\-training procedure that uses a small calibration set without requiring retraining\. Evaluating a single candidate per round makes allocation decisions primarily influenced by depth and tensor granularity\. A naive implementation makes the process costlier with wider layers\. Nevertheless, efficient use of hardware allows parallel candidate evaluation\. This post\-training search is relatively small compared to pre\-training and can be amortized across future deployments, although larger model evaluations are still necessary\. We prioritize pruning quality over direct measurement of inference throughput\. At the moment, we rely on existing sparse\-kernel results that translate unstructured pruning into accelerated performance\([Macko and Boža, 2026](https://arxiv.org/html/2610.28773#bib.bib30)\)\.

## 5Conclusion

We introduced LADDER, a post\-training allocation process that repeatedly selects the least damaging pruning update as sparsity increases\. This controlled trajectory complements different local pruning rules, with SparseGPT as the strongest backend, while producing nested mask supports across budgets\. Across five models and three language\-modeling datasets, our method limits quality degradation in the aggressive 70–90% sparsity regime compared with OWL, DSA, and LSA\. Distillation narrows these differences, while downstream evaluation shows more consistent accuracy retention across models and tasks\. LADDER can also refine masks initialized by existing allocation methods, although its full dense\-to\-sparse trajectory remains strongest at 90%, suggesting the importance of earlier allocation decisions\. These results position our method as a flexible strategy for constructing robust high\-sparsity model families\.

### AI Use Statement

Generative AI tools were used to assist with software implementation and debugging, edit manuscript prose, and review the manuscript for clarity, internal consistency, and submission compliance\. They were not used to generate experimental measurements or replace the authors’ scientific judgment\. The authors reviewed and verified all AI\-assisted code and text and take full responsibility for the methods, experiments, claims, and conclusions\.

## Reproducibility Statement

To support reproducibility, an anonymized implementation of LADDER, together with the scripts used to produce all reported results, is available athttps://anonymous\.4open\.science/r/ladder/\. The repository includes the LADDER allocation procedure \(Algorithm 1\), the OWL, DSA, and LSA baselines, the posterior\-distillation pipeline, and the evaluation code\.

## References

- G\. Bai, Y\. Li, C\. Ling, K\. Kim, and L\. ZhaoSparseLLM: Towards Global Pruning of Pre\-trained Language Models\.Advances in Neural Information Processing Systems37,pp\. 46203–46225\(en\)\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/522134ee1c52c7a2b929bc87cfe1781c-Abstract-Conference.html),[Document](https://dx.doi.org/10.52202/079017-1468)Cited by:[§2\.2](https://arxiv.org/html/2610.28773#S2.SS2.p1.1)\.
- Caiet al\.\(2024\)R\. Cai, S\. Muralidharan, G\. Heinrich, H\. Yin, Z\. Wang, J\. Kautz, and P\. MolchanovFLEXTRON: many\-in\-one flexible large language model\.InProceedings of the 41st International Conference on Machine Learning,ICML’24, Vol\.235,Vienna, Austria,pp\. 5298–5311\.Cited by:[§2\.3](https://arxiv.org/html/2610.28773#S2.SS3.p1.1)\.
- Changet al\.\(2021\)X\. Chang, Y\. Li, S\. Oymak, and C\. ThrampoulidisProvable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks\.Proceedings of the AAAI Conference on Artificial Intelligence35\(8\),pp\. 6974–6983\(en\)\.External Links:ISSN 2374\-3468,[Link](https://ojs.aaai.org/index.php/AAAI/article/view/16859),[Document](https://dx.doi.org/10.1609/aaai.v35i8.16859)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p2.1)\.
- Chenget al\.\(2024\)H\. Cheng, M\. Zhang, and J\. Q\. ShiA Survey on Deep Neural Network Pruning\-Taxonomy, Comparison, Analysis, and Recommendations\.arXiv\.Note:arXiv:2308\.06767 \[cs\]External Links:[Link](http://arxiv.org/abs/2308.06767),[Document](https://dx.doi.org/10.48550/arXiv.2308.06767)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p3.1)\.
- Cottieret al\.\(2025\)B\. Cottier, R\. Rahman, L\. Fattorini, N\. Maslej, T\. Besiroglu, and D\. OwenThe rising costs of training frontier AI models\.arXiv\.Note:arXiv:2405\.21015 \[cs\]External Links:[Link](http://arxiv.org/abs/2405.21015),[Document](https://dx.doi.org/10.48550/arXiv.2405.21015)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-V4: Towards Highly Efficient Million\-Token Context Intelligence\.arXiv\.Note:arXiv:2606\.19348 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2606.19348),[Document](https://dx.doi.org/10.48550/arXiv.2606.19348)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p1.1)\.
- Dentonet al\.\(2014\)E\. Denton, W\. Zaremba, J\. Bruna, Y\. LeCun, and R\. FergusExploiting linear structure within convolutional networks for efficient evaluation\.InProceedings of the 28th International Conference on Neural Information Processing Systems \- Volume 1,NIPS’14, Vol\.1,Cambridge, MA, USA,pp\. 1269–1277\.Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p3.1)\.
- Devvritet al\.\(2024\)F\. Devvrit, S\. Kudugunta, A\. Kusupati, T\. Dettmers, K\. Chen, I\. S\. Dhillon, Y\. Tsvetkov, H\. Hajishirzi, S\. M\. Kakade, A\. Farhadi, and P\. JainMatFormer: Nested Transformer for Elastic Inference\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,\(en\)\.External Links:[Link](https://openreview.net/forum?id=fYa6ezMxD5)Cited by:[§2\.3](https://arxiv.org/html/2610.28773#S2.SS3.p1.1)\.
- Frantar and Alistarh \(2023\)E\. Frantar and D\. AlistarhSparseGPT: Massive Language Models Can be Accurately Pruned in One\-Shot\.InProceedings of the 40th International Conference on Machine Learning,pp\. 10323–10337\(en\)\.External Links:ISSN 2640\-3498,[Link](https://proceedings.mlr.press/v202/frantar23a.html)Cited by:[§A\.1](https://arxiv.org/html/2610.28773#A1.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.28773#S1.p5.1),[§2\.1](https://arxiv.org/html/2610.28773#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.28773#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.28773#S4.SS1.p2.1)\.
- Gaoet al\.\(2025\)C\. Gao, J\. Wang, and L\. JingMaximum redundancy pruning for network compression\.Computer Vision and Image Understanding259,pp\. 104404\.External Links:ISSN 1077\-3142,[Link](https://www.sciencedirect.com/science/article/pii/S1077314225001274),[Document](https://dx.doi.org/10.1016/j.cviu.2025.104404)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p5.1)\.
- Hagiwara \(1994\)M\. HagiwaraA simple and effective method for removal of hidden units and weights\.Neurocomputing6\(2\),pp\. 207–218\.External Links:ISSN 0925\-2312,[Link](https://www.sciencedirect.com/science/article/pii/0925231294900558),[Document](https://dx.doi.org/10.1016/0925-2312%2894%2990055-8)Cited by:[§2\.1](https://arxiv.org/html/2610.28773#S2.SS1.p1.1)\.
- Hanet al\.\(2015\)S\. Han, J\. Pool, J\. Tran, and W\. J\. DallyLearning both weights and connections for efficient neural networks\.InProceedings of the 29th International Conference on Neural Information Processing Systems \- Volume 1,NIPS’15, Vol\.1,Cambridge, MA, USA,pp\. 1135–1143\.Cited by:[§2\.1](https://arxiv.org/html/2610.28773#S2.SS1.p1.1)\.
- Hassibiet al\.\(1993\)B\. Hassibi, D\.G\. Stork, and G\.J\. WolffOptimal Brain Surgeon and general network pruning\.InIEEE International Conference on Neural Networks,pp\. 293–299 vol\.1\.External Links:[Link](https://ieeexplore.ieee.org/document/298572),[Document](https://dx.doi.org/10.1109/ICNN.1993.298572)Cited by:[§2](https://arxiv.org/html/2610.28773#S2.p5.2)\.
- Horváthet al\.\(2021\)S\. Horváth, S\. Laskaridis, M\. Almeida, I\. Leontiadis, S\. Venieris, and N\. LaneFjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 12876–12889\.External Links:[Link](https://proceedings.neurips.cc/paper/2021/hash/6aed000af86a084f9cb0264161e29dd3-Abstract.html)Cited by:[§2\.3](https://arxiv.org/html/2610.28773#S2.SS3.p1.1)\.
- Houet al\.\(2020\)L\. Hou, Z\. Huang, L\. Shang, X\. Jiang, X\. Chen, and Q\. LiuDynaBERT: Dynamic BERT with Adaptive Width and Depth\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 9782–9793\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/6f5216f8d89b086c18298e043bfe48ed-Abstract.html)Cited by:[§2\.3](https://arxiv.org/html/2610.28773#S2.SS3.p1.1)\.
- Huet al\.\(2021\)P\. Hu, X\. Peng, H\. Zhu, M\. M\. S\. Aly, and J\. LinOPQ: Compressing Deep Neural Networks with One\-shot Pruning\-Quantization\.Proceedings of the AAAI Conference on Artificial Intelligence35\(9\),pp\. 7780–7788\(en\)\.External Links:ISSN 2374\-3468,[Link](https://ojs.aaai.org/index.php/AAAI/article/view/16950),[Document](https://dx.doi.org/10.1609/aaai.v35i9.16950)Cited by:[§2\.2](https://arxiv.org/html/2610.28773#S2.SS2.p1.1)\.
- Huanget al\.\(2025\)W\. Huang, Y\. Hu, G\. Jian, J\. Zhu, and J\. ChenPruning Large Language Models with Semi\-Structural Adaptive Sparse Training\.Proceedings of the AAAI Conference on Artificial Intelligence39\(23\),pp\. 24167–24175\(en\)\.External Links:ISSN 2374\-3468,[Link](https://ojs.aaai.org/index.php/AAAI/article/view/34592),[Document](https://dx.doi.org/10.1609/aaai.v39i23.34592)Cited by:[§2\.4](https://arxiv.org/html/2610.28773#S2.SS4.p1.1),[§3\.4](https://arxiv.org/html/2610.28773#S3.SS4.p2.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling Laws for Neural Language Models\.arXiv\.Note:arXiv:2001\.08361 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2001.08361),[Document](https://dx.doi.org/10.48550/arXiv.2001.08361)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p1.1)\.
- Leet al\.\(2025\)K\. N\. Le, R\. Sato, D\. Nakashima, T\. Suzuki, and M\. L\. NguyenOptiPrune: Effective Pruning Approach for Every Target Sparsity\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 3600–3612\.External Links:[Link](https://aclanthology.org/2025.coling-main.243/)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p5.1)\.
- LeCunet al\.\(1989\)Y\. LeCun, J\. Denker, and S\. SollaOptimal Brain Damage\.InAdvances in Neural Information Processing Systems,Vol\.2\.External Links:[Link](https://proceedings.neurips.cc/paper/1989/hash/6c9882bbac1c7093bd25041881277658-Abstract.html)Cited by:[§2](https://arxiv.org/html/2610.28773#S2.p5.2)\.
- Liet al\.\(2024\)L\. Li, P\. Dong, Z\. Tang, X\. Liu, Q\. Wang, W\. Luo, W\. Xue, Q\. Liu, X\. Chu, and Y\. GuoDiscovering Sparsity Allocation for Layer\-wise Pruning of Large Language Models\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 141292–141317\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/ff997469ac66cf893c4183efeb22212a-Abstract-Conference.html),[Document](https://dx.doi.org/10.52202/079017-4487)Cited by:[§2\.2](https://arxiv.org/html/2610.28773#S2.SS2.p1.1)\.
- Luet al\.\(2024\)H\. Lu, Y\. Zhou, S\. Liu, Z\. Wang, M\. W\. Mahoney, and Y\. YangAlphaPruning: Using Heavy\-Tailed Self Regularization Theory for Improved Layer\-wise Pruning of Large Language Models\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 9117–9152\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/10fc83943b4540a9524af6fc67a23fef-Abstract-Conference.html),[Document](https://dx.doi.org/10.52202/079017-0289)Cited by:[§2\.2](https://arxiv.org/html/2610.28773#S2.SS2.p1.1)\.
- Luet al\.\(2025\)M\. Lu, J\. Sun, J\. Lin, Z\. Zhou, and G\. SunLua\-LLM: Learning Unstructured\-Sparsity Allocation for Large Language Models\.InAdvances in Neural Information Processing Systems,Vol\.38, Main Conference,pp\. 123455–123481\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/b2c39fe6ce838440faf03a0f780e7a63-Abstract-Conference.html),[Document](https://dx.doi.org/10.52202/085713-4115)Cited by:[§2\.2](https://arxiv.org/html/2610.28773#S2.SS2.p1.1)\.
- Macko and Boža \(2026\)V\. Macko and V\. BožaMACKO: Sparse matrix\-vector multiplication for low sparsity\.InForty\-third International Conference on Machine Learning,\(en\)\.External Links:[Link](https://openreview.net/forum?id=ah9xkFXCV6&referrer=%5Bthe%20profile%20of%20Vladim%C3%ADr%20Macko%5D(%2Fprofile%3Fid%3D~Vladim%C3%ADr_Macko1))Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.28773#S4.SS1.p2.1),[§4\.5](https://arxiv.org/html/2610.28773#S4.SS5.p1.1)\.
- Micikeviciuset al\.\(2022\)P\. Micikevicius, D\. Stosic, N\. Burgess, M\. Cornea, P\. Dubey, R\. Grisenthwaite, S\. Ha, A\. Heinecke, P\. Judd, J\. Kamalu, N\. Mellempudi, S\. Oberman, M\. Shoeybi, M\. Siu, and H\. WuFP8 Formats for Deep Learning\.\(en\)\.External Links:[Link](https://arxiv.org/abs/2209.05433v2)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p3.1)\.
- Moonshot\-AI \(2026\)Moonshot\-AIMoonshotai/Kimi\-K2\.6 · Hugging Face\.External Links:[Link](https://huggingface.co/moonshotai/Kimi-K2.6)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p1.1)\.
- Navarreteet al\.\(2026\)I\. G\. Navarrete, N\. M\. Cuadrado, M\. Takáč, and S\. HorváthWhat Scalable Second\-Order Information Knows for Pruning at Initialization\.InThe Third Conference on Parsimony and Learning \(Proceedings Track\),\(en\)\.External Links:[Link](https://openreview.net/forum?id=1EWyO9KJWq#discussion)Cited by:[§2](https://arxiv.org/html/2610.28773#S2.p5.2)\.
- Quet al\.\(2025\)G\. Qu, Q\. Chen, W\. Wei, Z\. Lin, X\. Chen, and K\. HuangMobile Edge Intelligence for Large Language Models: A Contemporary Survey\.IEEE Communications Surveys & Tutorials27\(6\),pp\. 3820–3860\.External Links:ISSN 1553\-877X,[Link](https://ieeexplore.ieee.org/abstract/document/10835069),[Document](https://dx.doi.org/10.1109/COMST.2025.3527641)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p1.1)\.
- Raffelet al\.\(2023\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. LiuExploring the Limits of Transfer Learning with a Unified Text\-to\-Text Transformer\.arXiv\.Note:arXiv:1910\.10683 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/1910.10683),[Document](https://dx.doi.org/10.48550/arXiv.1910.10683)Cited by:[§A\.1](https://arxiv.org/html/2610.28773#A1.SS1.SSS0.Px2.p1.1)\.
- Ruanet al\.\(2021\)X\. Ruan, Y\. Liu, B\. Li, C\. Yuan, and W\. HuDPFPS: Dynamic and Progressive Filter Pruning for Compressing Convolutional Neural Networks from Scratch\.Proceedings of the AAAI Conference on Artificial Intelligence35\(3\),pp\. 2495–2503\(en\)\.External Links:ISSN 2374\-3468,[Link](https://ojs.aaai.org/index.php/AAAI/article/view/16351),[Document](https://dx.doi.org/10.1609/aaai.v35i3.16351)Cited by:[§2\.2](https://arxiv.org/html/2610.28773#S2.SS2.p1.1)\.
- Sieberlinget al\.\(2025\)O\. Sieberling, D\. Kuznedelev, E\. Kurtic, and D\. AlistarhEvoPress: Accurate Dynamic Model Compression via Evolutionary Search\.arXiv\.Note:arXiv:2410\.14649 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2410.14649),[Document](https://dx.doi.org/10.48550/arXiv.2410.14649)Cited by:[§2\.2](https://arxiv.org/html/2610.28773#S2.SS2.p1.1)\.
- Spectoret al\.\(2024\)B\. F\. Spector, S\. Arora, A\. Singhal, A\. Parthasarathy, D\. Y\. Fu, and C\. ReThunderKittens: Simple, Fast, and Adorable Kernels\.InThe Thirteenth International Conference on Learning Representations,\(en\)\.External Links:[Link](https://openreview.net/forum?id=0fJfVOSUra)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p4.1)\.
- Sunet al\.\(2023\)M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. KolterA Simple and Effective Pruning Approach for Large Language Models\.InThe Twelfth International Conference on Learning Representations,\(en\)\.External Links:[Link](https://openreview.net/forum?id=PxoFut3dWW)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p5.1),[§2\.1](https://arxiv.org/html/2610.28773#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.28773#S4.SS1.p1.1)\.
- Sunet al\.\(2026\)W\. Sun, X\. Song, P\. Li, L\. Yin, Y\. Zheng, and S\. LiuThe Curse of Depth in Large Language Models\.arXiv\.Note:arXiv:2502\.05795 \[cs\]External Links:[Link](http://arxiv.org/abs/2502.05795),[Document](https://dx.doi.org/10.48550/arXiv.2502.05795)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p5.1)\.
- Tanget al\.\(2026\)S\. Tang, O\. Sieberling, E\. Kurtic, Z\. Shen, and D\. AlistarhDarwinLM: Evolutionary Structured Pruning of Large Language Models\.arXiv\.Note:arXiv:2502\.07780 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2502.07780),[Document](https://dx.doi.org/10.48550/arXiv.2502.07780)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p5.1)\.
- Xuet al\.\(2024a\)P\. Xu, W\. Shao, M\. Chen, S\. Tang, K\. Zhang, P\. Gao, F\. An, Y\. Qiao, and P\. LuoBESA: Pruning Large Language Models with Blockwise Parameter\-Efficient Sparsity Allocation\.arXiv\.Note:arXiv:2402\.16880 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/2402.16880),[Document](https://dx.doi.org/10.48550/arXiv.2402.16880)Cited by:[§2\.2](https://arxiv.org/html/2610.28773#S2.SS2.p1.1)\.
- Xuet al\.\(2022\)R\. Xu, F\. Luo, C\. Wang, B\. Chang, J\. Huang, S\. Huang, and F\. HuangFrom Dense to Sparse: Contrastive Pruning for Better Pre\-trained Language Model Compression\.Proceedings of the AAAI Conference on Artificial Intelligence36\(10\),pp\. 11547–11555\(en\)\.External Links:ISSN 2374\-3468,[Link](https://ojs.aaai.org/index.php/AAAI/article/view/21408),[Document](https://dx.doi.org/10.1609/aaai.v36i10.21408)Cited by:[§2\.4](https://arxiv.org/html/2610.28773#S2.SS4.p1.1),[§3\.4](https://arxiv.org/html/2610.28773#S3.SS4.p2.1)\.
- Xuet al\.\(2024b\)X\. Xu, M\. Li, C\. Tao, T\. Shen, R\. Cheng, J\. Li, C\. Xu, D\. Tao, and T\. ZhouA Survey on Knowledge Distillation of Large Language Models\.arXiv\.Note:arXiv:2402\.13116 \[cs\]External Links:[Link](http://arxiv.org/abs/2402.13116),[Document](https://dx.doi.org/10.48550/arXiv.2402.13116)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p3.1)\.
- Yanget al\.\(2025a\)A\. Yang, A\. Li, B\. Yang, C\. Huang,et al\.Qwen3 Technical Report\.arXiv\.Note:arXiv:2505\.09388 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2505.09388),[Document](https://dx.doi.org/10.48550/arXiv.2505.09388)Cited by:[§1](https://arxiv.org/html/2610.28773#S1.p1.1)\.
- Yanget al\.\(2025b\)Z\. Yang, C\. Deng, Q\. Chen, Z\. Zhou, and J\. ChengLSA: Layer\-wise Sparsity Allocation for Large Language Model Pruning Based on Minimal Linear Reconstruction Error\.InThe Fourteenth International Conference on Learning Representations,\(en\)\.External Links:[Link](https://openreview.net/forum?id=xq3lza5IjN)Cited by:[§2\.2](https://arxiv.org/html/2610.28773#S2.SS2.p1.1)\.
- Yinet al\.\(2024\)L\. Yin, Y\. Wu, Z\. Zhang, C\. Hsieh, Y\. Wang, Y\. Jia, G\. Li, A\. Jaiswal, M\. Pechenizkiy, Y\. Liang, M\. Bendersky, Z\. Wang, and S\. LiuOutlier weighed layerwise sparsity \(OWL\): a missing secret sauce for pruning LLMs to high sparsity\.InProceedings of the 41st International Conference on Machine Learning,ICML’24, Vol\.235,Vienna, Austria,pp\. 57101–57115\.Cited by:[§2\.2](https://arxiv.org/html/2610.28773#S2.SS2.p1.1)\.
- Yu and Huang \(2019\)J\. Yu and T\. HuangUniversally Slimmable Networks and Improved Training Techniques\.In2019 IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 1803–1811\(English\)\.External Links:ISBN 978\-1\-7281\-4803\-8,[Link](https://www.computer.org/csdl/proceedings-article/iccv/2019/480300b803/1hVlpaqRL7q),[Document](https://dx.doi.org/10.1109/ICCV.2019.00189)Cited by:[§2\.3](https://arxiv.org/html/2610.28773#S2.SS3.p1.1)\.
- Yuet al\.\(2018\)J\. Yu, L\. Yang, N\. Xu, J\. Yang, and T\. HuangSlimmable Neural Networks\.InInternational Conference on Learning Representations,\(en\)\.External Links:[Link](https://openreview.net/forum?id=H1gMCsAqY7)Cited by:[§2\.3](https://arxiv.org/html/2610.28773#S2.SS3.p1.1)\.
- Zacconeet al\.\(2026\)R\. Zaccone, S\. Laskaridis, M\. Ciccone, and S\. HorváthFlexRank: Nested Low\-Rank Knowledge Decomposition for Adaptive Model Deployment\.InForty\-third International Conference on Machine Learning,\(en\)\.External Links:[Link](https://openreview.net/forum?id=DK0kvnNelx)Cited by:[§2\.3](https://arxiv.org/html/2610.28773#S2.SS3.p1.1)\.
- Zhang and Papyan \(2024\)S\. Zhang and V\. PapyanOATS: Outlier\-Aware Pruning Through Sparse and Low Rank Decomposition\.InThe Thirteenth International Conference on Learning Representations,\(en\)\.External Links:[Link](https://openreview.net/forum?id=DLDuVbxORA)Cited by:[§2\.2](https://arxiv.org/html/2610.28773#S2.SS2.p1.1)\.
- Zhanget al\.\(2022\)S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin, T\. Mihaylov, M\. Ott, S\. Shleifer, K\. Shuster, D\. Simig, P\. S\. Koura, A\. Sridhar, T\. Wang, and L\. ZettlemoyerOPT: Open Pre\-trained Transformer Language Models\.arXiv\.Note:arXiv:2205\.01068 \[cs\.CL\]External Links:[Link](http://arxiv.org/abs/2205.01068),[Document](https://dx.doi.org/10.48550/arXiv.2205.01068)Cited by:[§4\.1](https://arxiv.org/html/2610.28773#S4.SS1.p1.1)\.

## Appendix ASupplementary Material

This appendix provides the experimental settings and the results omitted from the main paper\. It first describes pruning, evaluation, and distillation, then reports the complete language\-modeling and downstream results, followed by the ablations and mask analyses\.

### A\.1Reproducibility and Exact Configurations

#### Pruning and allocation configurations\.

Following[Frantar and Alistarh \(2023\)](https://arxiv.org/html/2610.28773#bib.bib37), we prune all linear layers within the transformer blocks while excluding biases\. OWL, DSA, LSA, and LADDER use SparseGPT as the common local pruning backend, with 128\-column blocks and mean\-diagonal Hessian damping of0\.010\.01\. Local mask selection and weight reconstruction are therefore fixed across these methods, isolating the effect of the global sparsity allocation\.

For LADDER, we set the minimum threshold tozmin=1z\_\{\\min\}=1and use a step fraction ofγ=0\.20\\gamma=0\.20for OPT and Qwen3\-0\.6B andγ=0\.10\\gamma=0\.10for Llama\-3\.2\-1B\. For OWL, we setM=10M=10, such that its layer\-wise outlier distribution counts scores exceeding ten times the corresponding block mean, and use an allocation spread ofλ=0\.08\\lambda=0\.08\. For DSA, we use the allocation expressionW:\(ABSLOG\)\-\(VAR\)\-\(ATAN, ASIN\)\-\(7\), obtained through the original evolutionary search and transferred unchanged to the evaluated models, withλ=0\.08\\lambda=0\.08\. For LSA, we use the LSAC variant withλ=0\.08\\lambda=0\.08and construct projection\-level reconstruction\-error profiles at 50% probe sparsity\.

Table[2](https://arxiv.org/html/2610.28773#A1.T2)summarizes the settings shared across the main comparisons, the method\-specific configurations, and the configurations used in the ablations\.

Table 2:Pruning, evaluation, and ablation configurations\.
#### Calibration dataset\.

All allocation methods use the same 128 sequences of 2,048 tokens sampled from the C4 training set\([Raffel et al\., 2023](https://arxiv.org/html/2610.28773#bib.bib8)\)\. These samples are used only to estimate the activation\- or curvature\-dependent quantities required by the pruning rules\. For LADDER, corpus perplexity on this sample additionally serves as the candidate\-selection criterion\. The calibration samples are never used to compute the reported language\-modeling loss\. Magnitude and Wanda appear only in the local\-pruning\-rule ablation\.

We pin the first English C4 training shard and load its first 2,048 nonempty documents\. Each document is tokenized with the tokenizer of the evaluated model, followed by the configured space separator, to form candidate 2,048\-token windows\. Windows that cross document boundaries are ineligible for calibration\. With seed 0, the sampler repeatedly selects an eligible document and then one of its eligible windows, both with replacement, until it collects 128 examples\. We use batch size 1 and record the sampled document indices and window offsets\. This reproduces the random\-document, random\-window structure of the reference Wanda and SparseGPT loaders while fixing the exact calibration sample\.

#### Language\-modeling evaluation\.

We evaluate corpus\-level mean next\-token cross\-entropyℒ\\mathcal\{L\}and the corresponding perplexityexp⁡\(ℒ\)\\exp\(\\mathcal\{L\}\)on WikiText\-2, PTB, and C4\. Evaluation measures next\-token loss on deterministic held\-out token streams\. For each split, raw documents are joined using the dataset\-specific separator, tokenized once, and divided into consecutive, nonoverlapping 2,048\-token windows; the incomplete tail is discarded\. Thus, every pruning method is scored on the same target tokens:

- •WikiText\-2 \(WT2\):theSalesforce/wikitextdataset with configurationwikitext\-2\-raw\-v1; we score the official test split\.
- •Penn Treebank \(PTB\):the public Zaremba text files; we score the official test split\.
- •C4:the Englishallenai/c4corpus\. We pin the first validation shard, join its first 1,100 documents, and retain the first 256 complete windows\. Calibration uses the separate training shard described above\.

#### Posterior\-distillation objective\.

Posterior distillation fixes the nine masks corresponding to sparsities from 10% to 90% and jointly optimizes a single shared student state\. The student is initialized from the pretrained dense weights; method\-specific SparseGPT reconstructed weights are not used for initialization\.

Let𝒮=\{s1,…,sJ\}\\mathcal\{S\}=\\\{s\_\{1\},\\ldots,s\_\{J\}\\\}denote the participating sparsity budgets and letpt\(T\)p\_\{t\}^\{\(T\)\}andps\(T\)p\_\{s\}^\{\(T\)\}be the dense\-teacher and masked\-student distributions at temperatureTT\. The shared student minimizes

ℒKD=T2\|𝒮\|∑s∈𝒮DKL\(pt\(T\)\(⋅∣x\)∥ps\(T\)\(⋅∣x\)\)\.\\displaystyle\\textstyle\\mathcal\{L\}\_\{\\mathrm\{KD\}\}=\\frac\{T^\{2\}\}\{\|\\mathcal\{S\}\|\}\\sum\_\{s\\in\\mathcal\{S\}\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{t\}^\{\(T\)\}\(\\cdot\\mid x\)\\,\\\|\\,p\_\{s\}^\{\(T\)\}\(\\cdot\\mid x\)\\right\)\.Every participating mask is evaluated against the same teacher outputs, the mask\-specific losses are averaged, and one optimizer step updates the shared parameters\. We train for five epochs using AdamW with a learning rate of10−410^\{\-4\}and temperatureT=2T=2\.

#### Implementation details\.

All pruning and distillation runs use seed 0 with deterministic algorithms, so each reported model–method pair represents one run rather than an average over independent runs\. Distillation reuses the 128 calibration sequences; for each batch, equally weighted gradients are accumulated over the nine fixed masks before one AdamW update, giving 128 updates per epoch and 640 updates in total\. We use gradient checkpointing\.

#### Computing infrastructure\.

Experiments were run on compute nodes equipped with four NVIDIA A100\-SXM4 GPUs with 40 GB of memory each, using NVIDIA driver 570\.195\.03 and CUDA 12\.8\.

### A\.2Downstream Summary

We uselm\-eval0\.4\.12 with zero demonstrations and batch size 1 on seven tasks:

- •ARC\-Easy and ARC\-Challenge:grade\-school multiple\-choice science questions in easier and reasoning\-intensive subsets\.
- •BoolQ:binary reading comprehension that asks whether a passage supports a question\.
- •HellaSwag:commonsense completion by selecting the most plausible continuation\.
- •OpenBookQA:multiple\-choice elementary science questions based on a small collection of core facts\.
- •PIQA:physical commonsense reasoning that selects the more plausible solution to a practical goal\.
- •WinoGrande:commonsense coreference resolution in ambiguity\-controlled sentence pairs\.

BoolQ uses task version 1; the other tasks use version 0\. We report normalized accuracy for the multiple\-choice tasks and ordinary accuracy for BoolQ and WinoGrande\. For aggregate reporting, we take the unweighted mean of the seven task accuracies and combine their task\-level bootstrap standard errors\. Repeated evaluation seeds for the same model state produced identical accuracies\. The uncertainty shown in the figures is therefore the task\-level bootstrap standard error returned bylm\-eval, not variation across independently pruned models\.

Figure 7:Mean post\-distillation zero\-shot accuracy over seven tasks for Qwen3\-0\.6B, Llama\-3\.2\-1B, and OPT\-1\.3B\. Curves compare methods from 50% to 90% sparsity; solid horizontal lines mark dense accuracy, while the dotted line marks the seven\-task mean random accuracy \(35\.7%\)\. Shading denotes propagatedlm\-evaluncertainty\.In Figure[7](https://arxiv.org/html/2610.28773#A1.F7), LADDER has the highest mean accuracy in seven of the nine model–sparsity comparisons at 70–90%\. OWL is highest for Llama\-3\.2\-1B at 80%, and LSA is highest for that model at 90%\.

### A\.3Ablations

#### Local pruning rule and external initialization\.

Table[3](https://arxiv.org/html/2610.28773#A1.T3)compares each local pruning rule used alone and within LADDER\. At 70–90% sparsity, LADDER lowers test loss in all 27 matched rule–dataset comparisons, and SparseGPT gives the lowest LADDER loss in all nine dataset–sparsity comparisons\.

Table[4](https://arxiv.org/html/2610.28773#A1.T4)starts LADDER from the 70% masks produced by OWL, DSA, or LSA\. Continuing with LADDER lowers loss relative to extending the original method in all 18 comparisons\. OWL gives the strongest external starting point, while the dense\-to\-sparse LADDER trajectory gives the lowest loss on every dataset at 90%\.

Table 3:Each local pruning rule used alone \(Base\) and within LADDER\. Values are test loss; bold marks the better matched result\.Table 4:Dataset\-level test loss for direct baselines, LADDER continuation from their 70% masks, and the standard dense\-to\-sparse LADDER trajectory\.

### A\.4Complete Language\-Modeling Results

\(a\) Qwen3\-0\.6B

\(b\) Llama\-3\.2\-1B

\(c\) OPT\-1\.3B

\(d\) OPT\-350M

\(e\) OPT\-125M

Figure 8:Fraction of the pruning\-induced test\-loss gap removed by posterior distillation,100​\(Lpre−Lpost\)/\(Lpre−Ldense\)100\(L\_\{\\mathrm\{pre\}\}\-L\_\{\\mathrm\{post\}\}\)/\(L\_\{\\mathrm\{pre\}\}\-L\_\{\\mathrm\{dense\}\}\)\. Higher is better; zero indicates no recovery\.Across the 180 model, dataset, method, and sparsity combinations at 70–90%, posterior distillation reduces test loss in 178 cases\. The amount recovered varies substantially, and the two exceptions are both 90% sparse OPT\-350M LADDER models\.

Table[5](https://arxiv.org/html/2610.28773#A1.T5)reports the pre\-distillation results for all five evaluated models\.

Table 5:Complete pre\-distillation language\-modeling results at 70–90% sparsity\. Entries report test loss followed by perplexity in parentheses; lower is better, and bold marks the best method for each configuration\.Table[6](https://arxiv.org/html/2610.28773#A1.T6)reports the corresponding results after posterior distillation\.

Table 6:Complete post\-distillation language\-modeling results at 70–90% sparsity\. Entries report test loss followed by perplexity in parentheses; lower is better, and bold marks the best method for each configuration\.
### A\.5Task\-Level Downstream Results

Figures[9](https://arxiv.org/html/2610.28773#A1.F9)–[13](https://arxiv.org/html/2610.28773#A1.F13)show the task\-level degradation from each model’s dense checkpoint before and after distillation\. Smaller values indicate better accuracy retention\.

These paired views preserve variation across tasks that is hidden by the macro average\. The leading method changes across tasks and sparsity levels, with ties and occasional reversals even when the language\-modeling losses are clearly separated\. The downstream evidence therefore supports a claim of stronger aggregate accuracy retention at high sparsity, rather than superiority on every task or configuration\.

![Refer to caption](https://arxiv.org/html/2610.28773v1/downstream_pruning_degradation_figure.png)

\(a\) Before posterior distillation ![Refer to caption](https://arxiv.org/html/2610.28773v1/downstream_distillation_degradation_figure.png) \(b\) After posterior distillation

Figure 9:Qwen3\-0\.6B zero\-shot accuracy degradation before and after posterior distillation\. Lower is better\.![Refer to caption](https://arxiv.org/html/2610.28773v1/x5.png)

\(a\) Before posterior distillation ![Refer to caption](https://arxiv.org/html/2610.28773v1/x6.png) \(b\) After posterior distillation

Figure 10:Llama\-3\.2\-1B zero\-shot accuracy degradation before and after posterior distillation\. Lower is better\.![Refer to caption](https://arxiv.org/html/2610.28773v1/x7.png)

\(a\) Before posterior distillation ![Refer to caption](https://arxiv.org/html/2610.28773v1/x8.png) \(b\) After posterior distillation

Figure 11:OPT\-1\.3B zero\-shot accuracy degradation before and after posterior distillation\. Lower is better\.![Refer to caption](https://arxiv.org/html/2610.28773v1/x9.png)

\(a\) Before posterior distillation ![Refer to caption](https://arxiv.org/html/2610.28773v1/x10.png) \(b\) After posterior distillation

Figure 12:OPT\-350M zero\-shot accuracy degradation before and after posterior distillation\. Lower is better\.![Refer to caption](https://arxiv.org/html/2610.28773v1/x11.png)

\(a\) Before posterior distillation ![Refer to caption](https://arxiv.org/html/2610.28773v1/x12.png) \(b\) After posterior distillation

Figure 13:OPT\-125M zero\-shot accuracy degradation before and after posterior distillation\. Lower is better\.
### A\.6Allocation Patterns and Mask Nestedness

#### Support nestedness\.

LetMℓ\(a\)M\_\{\\ell\}^\{\(a\)\}andMℓ\(b\)M\_\{\\ell\}^\{\(b\)\}denote the boolean keep masks for the same tensor at sparsity levelssa<sbs\_\{a\}<s\_\{b\}\. The higher\-sparsity mask is nested inside the lower\-sparsity mask whenMℓ\(b\)⊆Mℓ\(a\)⟺Mℓ\(b\)∧¬Mℓ\(a\)=0M\_\{\\ell\}^\{\(b\)\}\\subseteq M\_\{\\ell\}^\{\(a\)\}\\Longleftrightarrow M\_\{\\ell\}^\{\(b\)\}\\wedge\\neg M\_\{\\ell\}^\{\(a\)\}=0\. The corresponding model\-level violation count isV⁡\(a,b\)=∑ℓ=1L‖Mℓ\(b\)∧¬Mℓ\(a\)‖0V\(a,b\)=\\sum\_\{\\ell=1\}^\{L\}\\left\\lVert M\_\{\\ell\}^\{\(b\)\}\\wedge\\neg M\_\{\\ell\}^\{\(a\)\}\\right\\rVert\_\{0\}, with perfect nestedness if and only ifV⁡\(a,b\)=0V\(a,b\)=0\.

\(a\) Qwen3\-0\.6B

\(b\) Llama\-3\.2\-1B

\(c\) OPT\-1\.3B

\(d\) OPT\-350M

\(e\) OPT\-125M

Figure 14:Support across adjacent sparsity levels for the five evaluated models\. Each bar partitions the weights kept at the higher sparsity into those retained from the next lower sparsity and those reintroduced after having been pruned\.Across the four measured transitions from 50% to 90%, LADDER reintroduces zero weights for every model in Figure[14](https://arxiv.org/html/2610.28773#A1.F14)\. OWL, DSA, and LSA generate each target independently and reintroduce some weights in every measured transition\. This comparison verifies mask nestedness; it does not isolate its effect on language\-modeling performance\. SparseGPT can still update the values of surviving weights after an accepted step, although a pruned weight is never restored\. Figure[15](https://arxiv.org/html/2610.28773#A1.F15)shows that the selected layer sparsities change with both the model and the global sparsity target\. We therefore do not interpret the plots as a fixed ranking of transformer layers\.

\(a\) Qwen3\-0\.6B

\(b\) Llama\-3\.2\-1B

\(c\) OPT\-1\.3B

\(d\) OPT\-350M

\(e\) OPT\-125M

Figure 15:Layer allocations selected by LADDER with SparseGPT\. Each bar gives the percentage of prunable weights removed from a transformer block; overlaid bars show the allocations from 50% to 90% global sparsity\.