LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

# Calibration-Free Structured Pruning ofLarge Language Models via Latent Spectral Geometry
Source: [https://arxiv.org/html/2609.11163](https://arxiv.org/html/2609.11163)
###### Abstract

Structured pruning of large language models \(LLMs\) offers hardware\-efficient compression, yet existing methods require calibration data, gradient computation, or large auxiliary policy networks at pruning time\. LILA \(*Latent\-Informed Layer Analysis*\) scores neuron importance via the Kolmogorov–Smirnov \(KS\) distance between empirical singular value distributions of the full and neuron\-ablated feed\-forward network \(FFN\) weight matrix, providing a closed\-form spectral rule requiring no training, calibration data, or auxiliary network\. Without any fine\-tuning, LILA surpasses PruneNet \(45M\-parameter RL policy\) by 1\.57 pp in zero\-shot accuracy on LLaMA\-2\-7B at 25% sparsity, and outperforms WikiText\-2\-calibrated SliceGPT by up to 6\.0 pp across all sparsity levels, while preserving the original architecture\. After one epoch of LoRA recovery fine\-tuning, LILA achieves highly competitive performance, matching the heavily calibrated SliceGPT baseline to within a 0\.48 pp margin across LLaMA\-2\-7B and Phi\-2, despite using zero calibration data\. A Neural Tangent Kernel analysis confirms a 22×\\timesreduction in functional distortion versus random pruning, providing theoretical grounding for the spectral importance criterion\. Finally, extending LILA to dynamically allocate sparsity budgets via KS\-scores yields state\-of\-the\-art generative preservation at moderate compression, while uncovering fundamental single\-layer architectural bottlenecks at higher compression regimes\.

Sankar Behera CSE, IIT Jammu

sankar\.behera@iitjammu\.ac\.in

Dhruv Singh Mathematics, IIT Jammu

2023uma0210@iitjammu\.ac\.in

Anshika Agnihotri CSE, IIT Jammu

2025pcs0022@iitjammu\.ac\.in

Raj Kumar Choudhary IT, EC Bikaner

Choudhary\.rajkumar@ecb\.ac\.in

Satyadev Ahlawat EE, IIT Jammu

satyadev\.ahlawat@iitjammu\.ac\.in

Yamuna Prasad CSE, IIT Jammu

yamuna\.prasad@iitjammu\.ac\.in

## 1Introduction

The deployment of large language models \(LLMs\) at scale is constrained by substantial computational and memory requirements\([Brown et al\., 2020](https://arxiv.org/html/2609.11163#bib.bib3);[Touvron et al\., 2023](https://arxiv.org/html/2609.11163#bib.bib28)\)\. A LLaMA\-2\-7B model requires approximately 14 GB of GPU memory at half precision, rendering single\-GPU inference infeasible for commodity hardware\. Model compression through pruning, quantization, or factorization is therefore an essential step toward practical LLM deployment\.

Two dominant paradigms exist for post\-training pruning\. Unstructured methods \(e\.g\., SparseGPT\([Frantar and Alistarh, 2023](https://arxiv.org/html/2609.11163#bib.bib6)\), Wanda\([Sun et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib26)\)\) set individual weights to zero, achieving high theoretical compression ratios but producing irregular sparsity patterns that deliver speedups only on specialized hardware with sparse tensor support\. Structured methods remove entire neurons, attention heads, or transformer layers, yielding dense weight tensors that are immediately hardware\-efficient without any software modification\.

However, existing structured pruning methods impose a data dependency at pruning time\. LLM\-Pruner\([Ma et al\., 2023](https://arxiv.org/html/2609.11163#bib.bib20)\)requires gradient computation over calibration sequences\. SliceGPT\([Ashkboos et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib1)\)consumes 1,024 WikiText\-2 samples to estimate and apply an*irreversible PCA rotation*that permanently alters the model architecture, fundamentally complicating downstream hardware deployment\. Wanda\([Sun et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib26)\)relies on 128 calibration samples for activation statistics\. PruneNet\([Sengupta et al\., 2025](https://arxiv.org/html/2609.11163#bib.bib24)\)trains a 45\-million\-parameter reinforcement learning policy network on the target model’s output distribution, masking the massive computational overhead of its prerequisite policy pre\-training\. These calibration requirements present practical obstacles: the calibration distribution may be inaccessible in privacy\-sensitive deployments, or mismatched to the target task, degrading pruning quality under distribution shift\.

The singular value spectrum of a weight matrix encodes the energy distribution across its principal directions\. A neuron whose removal substantially perturbs this spectrum is spectrally indispensable; a neuron whose ablation leaves the spectrum invariant is spectrally redundant\. This spectral geometry is an*intrinsic*property of the weight matrix, computable without any data, and invariant to calibration corpus choice\.

Building on this observation,LILA\(*Latent\-Informed Layer Analysis*\) is proposed as a calibration\-free structured pruning framework for transformer FFN layers grounded in Non\-negative Matrix Factorization \(NMF\)\([Lee and Seung, 1999](https://arxiv.org/html/2609.11163#bib.bib17)\)\. The framework derives per\-neuron importance scores from the spectral geometry revealed by the NMF latent factor basis, without any forward pass through the model\.

The main contributions of this work are as follows:

1. 1\.Unified NMF importance\-scoring framework\.Three matrix variants \(data\-free: Abs, Split; calibration\-guided: Act\) and three scoring criteria \(Residual Energy, Reconstruction Sensitivity, KS\-Distance Spectrum\) are unified under a single framework spanning an∼18×\{\\sim\}18\{\\times\}pruning\-speed range \(2\.4–44 min on LLaMA\-2\-7B\)\. Crucially, LILA\-Sensitivity completes in just 2\.4 minutes, offering a highly practical, ultra\-fast alternative to policy\-based methods without sacrificing post\-recovery performance\. Data\-free Variants Abs and Split consistently match or exceed calibrated Variant Act, suggesting that weight spectral geometry alone provides a sufficient importance signal in practice\.
2. 2\.KS\-Distance spectrum score\.The Kolmogorov–Smirnov distance between singular value CDFs of the full and neuron\-ablated weight matrix is proposed as a closed\-form importance criterion\. This mathematically rigorous approach surpasses the 45M\-parameter PruneNet RL policy by 1\.57 pp in zero\-shot accuracy on LLaMA\-2\-7B at 25% sparsity, achieving SOTA data\-free performance without any policy training\.
3. 3\.Architecture\-preserving empirical validation\.Evaluation across LLaMA\-2\-7B, Phi\-2, and OPT\-1\.3B at four sparsity levels \(20–40%\) on five zero\-shot benchmarks\. Prior to fine\-tuning, LILA\-Spectrum outperforms WikiText\-2\-calibrated SliceGPT by up to 6\.0 pp\. Post\-recovery, LILA closes to within a 0\.48 pp mean gap of SliceGPT on LLaMA\-2\-7B and Phi\-2 without using any calibration data and without modifying the model’s architecture \(in contrast to SliceGPT’s irreversible PCA rotation\)\. Furthermore, we isolate the impact of recovery data, demonstrating that instruction\-tuning sets yield up to\+\+2\.47 pp better zero\-shot recovery than standard unstructured text\.
4. 4\.NTK theoretical grounding\.The NTK trace ratio is introduced as a functional distortion metric; LILA achieves a 22×\\timeslower ratio than random pruning \(ρ=2\.10\\rho\{=\}2\.10vs\.47\.3847\.38, Phi\-2 25%\), providing principled justification for the spectral importance criterion\.

It is critical to distinguish the core pruning mask generation, which is strictly data\-free for Variants Abs and Split, from the optional Recovery Fine\-Tuning \(RFT\) phase, which is a standard post\-processing protocol\. In a direct zero\-shot evaluation, data\-free LILA\-Spectrum \(zero calibration, zero fine\-tuning\) surpasses SliceGPT calibrated on WikiText\-2 \(no RFT\) by up to6\.0 ppon LLaMA\-2\-7B, and remains within 0\.71 pp of SliceGPT calibrated on Alpaca \(no RFT\), despite using no data and preserving the original architecture\. LILA\-Spectrum also surpasses PruneNet at*all*sparsity levels on LLaMA\-2\-7B \(\+1\.75 to \+2\.27 pp noRFT\), without any RL policy training\. After one epoch of LoRA recovery fine\-tuning on WikiText\-2, LILA\-Spectrum Split further extends this advantage \(\+0\.35 to \+2\.11 pp\)\. On OPT\-1\.3B, LILA\-Spectrum outperforms Wanda by 13\.03 pp at 20% sparsity\.

## 2Related Work

Unstructured pruning achieves high sparsity but yields irregular tensors without hardware speedups\. SparseGPT\([Frantar and Alistarh, 2023](https://arxiv.org/html/2609.11163#bib.bib6)\)scales the classical second\-order OBS framework\([LeCun et al\., 1989](https://arxiv.org/html/2609.11163#bib.bib15);[Hassibi and Stork, 1992](https://arxiv.org/html/2609.11163#bib.bib11)\)to billion\-parameter models via approximate inverse Hessians\. Wanda\([Sun et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib26)\)scores weights by the product of weight magnitude and activation norms\. Both methods require calibration data; LILA Variants A and B operate fully data\-free \([definition3\.1](https://arxiv.org/html/2609.11163#S3.Thmtheorem1)\)\.

Structured pruning removes entire neurons, heads, or layers, producing dense tensors compatible with standard BLAS routines\. LLM\-Pruner\([Ma et al\., 2023](https://arxiv.org/html/2609.11163#bib.bib20)\)prunes coupled structures via gradient\-based dependency analysis\. ShortGPT\([Men et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib21)\)removes near\-zero\-contribution transformer layers wholesale\. SliceGPT\([Ashkboos et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib1)\)applies a PCA rotation and removes trailing principal components, permanently restructuring the model graph\. PruneNet\([Sengupta et al\., 2025](https://arxiv.org/html/2609.11163#bib.bib24)\)learns a 45M\-parameter RL pruning policy trained on the target model’s output distribution\. In contrast, LILA preserves the original architecture and replaces learned or data\-dependent criteria with a closed\-form KS\-distance spectral rule, achieving\+\+1\.6 pp over PruneNet and within 0\.48 pp of SliceGPT post\-RFT\.

Low\-rank factorization methods\([Hsu et al\., 2022](https://arxiv.org/html/2609.11163#bib.bib12)\)compress weight matrices directly viaW≈U​VTW\\approx UV^\{T\}\. LILA instead uses NMF\([Lee and Seung, 1999](https://arxiv.org/html/2609.11163#bib.bib17);[Lee and Seung, 2000](https://arxiv.org/html/2609.11163#bib.bib16)\)as an*importance scoring instrument*for discrete neuron masking rather than direct weight replacement, extending NMF\-based analysis to large\-scale transformer FFN pruning beyond prior convolutional\-network applications\([Cichocki et al\., 2009](https://arxiv.org/html/2609.11163#bib.bib4);gillis2020nmf\)\.

The Neural Tangent Kernel\([Jacot et al\., 2020](https://arxiv.org/html/2609.11163#bib.bib13);[Lee et al\., 2020](https://arxiv.org/html/2609.11163#bib.bib18)\)characterizes neural network training in function space;[Wang et al\. \(2023\)](https://arxiv.org/html/2609.11163#bib.bib29)study pruning using the NTK spectrum and its relationship to training dynamics\. We extends this to transformer FFN pruning via the NTK trace ratio in[section3\.4](https://arxiv.org/html/2609.11163#S3.SS4)\. Post\-training quantization\([Frantar et al\., 2023](https://arxiv.org/html/2609.11163#bib.bib7)\)is orthogonal and composable with spectral pruning\.

#### Positioning\.

Unlike Wanda’s element\-wise magnitude criterion, LILA\-Spectrum derives importance from higher\-order spectral geometry \(the KS shift in singular values\)\. Unlike SliceGPT, it applies a structured neuron mask without modifying the model graph\. Unlike SparseGPT, it requires no forward passes or calibration data; full mathematical contrasts are given in[section3](https://arxiv.org/html/2609.11163#S3)\.

## 3Method

### 3\.1Problem Formulation

![Refer to caption](https://arxiv.org/html/2609.11163v1/lila_architecture.png)Figure 1:LILA framework\.\(1\) For each FFN layer, a non\-negative matrixMMis constructed \(2\) fromWWusing one of three variants, \(3\) factorized with rank\-rrNMF, and \(4\) each neuronjjis scored by the Kolmogorov–Smirnov distance between the singular value distributions ofMMand the neuron\-ablated matrixMjM\_\{j\}\. \(5\) The lowest\-scoring\(1−s\)\(1\-s\)fraction of neurons are pruned by removing rows inWWand corresponding columns inW\(e\)W^\{\(e\)\}\. \(6\) The resulting pruned LLM preserves the dense architecture and can be optionally fine\-tuned with LoRA for recovery\.#### Transformer FFN layers\.

A transformer language model𝒯\\mathcal\{T\}with parametersΘ\\ThetacomprisesLLstacked blocks, each containing a multi\-head self\-attention sub\-layer followed by a feed\-forward network \(FFN\) sub\-layer\. The FFN in layerℓ\\ellis parameterized by an*up\-projection*Wup\(ℓ\)∈ℝdff×dW\_\{\\mathrm\{up\}\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{ff\}\}\\times d\}and a*down\-projection*Wdn\(ℓ\)∈ℝd×dffW\_\{\\mathrm\{dn\}\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\\times d\_\{\\mathrm\{ff\}\}\}, wheredddenotes the model dimension anddff=4​dd\_\{\\mathrm\{ff\}\}=4din standard configurations\. Given a hidden statex∈ℝ1×dx\\in\\mathbb\{R\}^\{1\\times d\}, the FFN computes

FFN⁡\(x\)=σ⁡\(x​Wup\(ℓ\)​T\+bup\)​Wdn\(ℓ\)​T\+bdn,\\mathrm\{FFN\}\(x\)\\;=\\;\\sigma\\\!\\left\(x\\,W\_\{\\mathrm\{up\}\}^\{\(\\ell\)T\}\+b\_\{\\mathrm\{up\}\}\\right\)W\_\{\\mathrm\{dn\}\}^\{\(\\ell\)T\}\+b\_\{\\mathrm\{dn\}\},\(1\)whereσ⁡\(⋅\)\\sigma\(\\cdot\)is a nonlinear activation \(SiLU for LLaMA\-2\([Touvron et al\., 2023](https://arxiv.org/html/2609.11163#bib.bib28)\); GeLU for Phi\-2\([Li et al\., 2023](https://arxiv.org/html/2609.11163#bib.bib19)\)and OPT\([Zhang et al\., 2022](https://arxiv.org/html/2609.11163#bib.bib31)\)\)\.

#### Structured pruning formulation\.

Let𝒩ℓ=\{1,…,dff\}\\mathcal\{N\}\_\{\\ell\}=\\\{1,\\ldots,d\_\{\\mathrm\{ff\}\}\\\}denote the neuron index set of layerℓ\\ell\. A*pruning mask*𝒦ℓ⊂𝒩ℓ\\mathcal\{K\}\_\{\\ell\}\\subset\\mathcal\{N\}\_\{\\ell\}identifies the set of neurons to be*retained*:

\|𝒦ℓ\|=⌊\(1−s\)​dff⌋,∀ℓ∈\{1,…,L\},\|\\mathcal\{K\}\_\{\\ell\}\|\\;=\\;\\left\\lfloor\(1\-s\)\\,d\_\{\\mathrm\{ff\}\}\\right\\rfloor,\\qquad\\forall\\,\\ell\\in\\\{1,\\ldots,L\\\},\(2\)wheres∈\(0,1\)s\\in\(0,1\)is the global sparsity ratio\. Pruning removes rows𝒦ℓc=𝒩ℓ∖𝒦ℓ\\mathcal\{K\}\_\{\\ell\}^\{c\}=\\mathcal\{N\}\_\{\\ell\}\\setminus\\mathcal\{K\}\_\{\\ell\}fromWup\(ℓ\)W\_\{\\mathrm\{up\}\}^\{\(\\ell\)\}and the corresponding columns fromWdn\(ℓ\)W\_\{\\mathrm\{dn\}\}^\{\(\\ell\)\}:

W~up\(ℓ\)=Wup\(ℓ\)\[𝒦ℓ,:\],W~dn\(ℓ\)=Wdn\(ℓ\)\[:,𝒦ℓ\]\.\\widetilde\{W\}\_\{\\mathrm\{up\}\}^\{\(\\ell\)\}=W\_\{\\mathrm\{up\}\}^\{\(\\ell\)\}\\\!\\left\[\\mathcal\{K\}\_\{\\ell\},\\,:\\right\],\\qquad\\widetilde\{W\}\_\{\\mathrm\{dn\}\}^\{\(\\ell\)\}=W\_\{\\mathrm\{dn\}\}^\{\(\\ell\)\}\\\!\\left\[:,\\,\\mathcal\{K\}\_\{\\ell\}\\right\]\.\(3\)The resulting pruned model𝒯~\\widetilde\{\\mathcal\{T\}\}has a hidden FFN dimension of⌊\(1−s\)​dff⌋\\lfloor\(1\-s\)\\,d\_\{\\mathrm\{ff\}\}\\rfloorper layer and preserves the original architecture’s computation graph \(no RMSNorm column deletion or graph rewiring\)\.

The structured pruning problem is formulated as:

𝒦∗=arg​min𝒦ℓ⊆𝒩ℓ\|𝒦ℓ\|=⌊\(1−s\)​dff⌋,∀ℓ⁡ℒ​\(𝒯~𝒦,𝒟\),\\mathcal\{K\}^\{\*\}\\;=\\;\\argmin\_\{\\begin\{subarray\}\{c\}\\mathcal\{K\}\_\{\\ell\}\\subseteq\\mathcal\{N\}\_\{\\ell\}\\\\ \|\\mathcal\{K\}\_\{\\ell\}\|=\\lfloor\(1\-s\)d\_\{\\mathrm\{ff\}\}\\rfloor,\\;\\forall\\ell\\end\{subarray\}\}\\;\\mathcal\{L\}\\\!\\left\(\\widetilde\{\\mathcal\{T\}\}\_\{\\mathcal\{K\}\},\\,\\mathcal\{D\}\\right\),\(4\)whereℒ\\mathcal\{L\}is the cross\-entropy language modeling loss and𝒟\\mathcal\{D\}denotes the target data distribution\. Because access to𝒟\\mathcal\{D\}is assumed unavailable \(calibration\-free setting\), the objective in[eq\.4](https://arxiv.org/html/2609.11163#S3.E4)is approximated via a data\-free importance scoring function derived from the weight matrices alone\.

### 3\.2NMF Matrix Construction

LetW∈ℝdff×dW\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{ff\}\}\\times d\}denote the up\-projection weight matrix of an arbitrary layer \(layer superscripts are dropped for clarity\)\. A non\-negative matrixM∈ℝ\+m×dM\\in\\mathbb\{R\}^\{m\\times d\}\_\{\+\}is constructed fromWWaccording to one of three*variants*, each encoding a different inductive assumption about the weight geometry\.

###### Definition 3\.1\(NMF Matrix Variants\)\.

M\(A\)\\displaystyle M^\{\(A\)\}=\|W\|∈ℝdff×d,\\displaystyle=\|W\|\\;\\in\\;\\mathbb\{R\}^\{d\_\{\\mathrm\{ff\}\}\\times d\},\(Abs\)M\(B\)\\displaystyle M^\{\(B\)\}=\(ReLU⁡\(W\)ReLU⁡\(−W\)\)∈ℝ2​dff×d,\\displaystyle=\\begin\{pmatrix\}\\mathrm\{ReLU\}\(W\)\\\\ \\mathrm\{ReLU\}\(\-W\)\\end\{pmatrix\}\\;\\in\\;\\mathbb\{R\}^\{2d\_\{\\mathrm\{ff\}\}\\times d\},\(Split\)M\(C\)\\displaystyle M^\{\(C\)\}=diag⁡\(a¯\)​\|W\|∈ℝdff×d,\\displaystyle=\\mathrm\{diag\}\(\\bar\{a\}\)\\,\|W\|\\;\\in\\;\\mathbb\{R\}^\{d\_\{\\mathrm\{ff\}\}\\times d\},\(Act\)wherea¯j=𝔼x∼𝒟c​\[\|hj​\(x\)\|\]\\bar\{a\}\_\{j\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{c\}\}\[\|h\_\{j\}\(x\)\|\]is the mean absolute activation of neuronjjestimated over a calibration corpus𝒟c\\mathcal\{D\}\_\{c\}\. Variants Abs and Split are*calibration\-free*; Variant Act requires a calibration set\.

#### Rationale\.

Variant Abs captures the raw magnitude geometry ofWW\. Variant Split \(the*signed split*\) decomposesWWinto positive and negative components prior to factorization, allowing NMF \(which enforces non\-negativity\) to separately model the two signed manifolds\. This preserves the sign structure of the weight distribution and yields a richer factorization basis\. Variant Act gates magnitude by empirical activation frequency, analogous to the activation\-weighted pruning of[Sun et al\. \(2024\)](https://arxiv.org/html/2609.11163#bib.bib26), but applied as a matrix scaling prior to NMF rather than as a direct score\.

#### NMF factorization\.

Given the non\-negative matrixM∈ℝ\+m×dM\\in\\mathbb\{R\}^\{m\\times d\}\_\{\+\}, the rank\-rrNMF seeks factors𝐖r∈ℝ\+m×r\\mathbf\{W\}\_\{r\}\\in\\mathbb\{R\}^\{m\\times r\}\_\{\+\}and𝐇r∈ℝ\+r×d\\mathbf\{H\}\_\{r\}\\in\\mathbb\{R\}^\{r\\times d\}\_\{\+\}that minimize the Frobenius reconstruction error:

min𝐖r≥0,𝐇r≥0⁡‖M−𝐖r​𝐇r‖F2\.\\min\_\{\\mathbf\{W\}\_\{r\}\\geq 0,\\;\\mathbf\{H\}\_\{r\}\\geq 0\}\\left\\\|M\-\\mathbf\{W\}\_\{r\}\\mathbf\{H\}\_\{r\}\\right\\\|\_\{F\}^\{2\}\.\(5\)Problem \([5](https://arxiv.org/html/2609.11163#S3.E5)\) is solved via alternating non\-negative least squares \(ANLS\)\([Lee and Seung, 2000](https://arxiv.org/html/2609.11163#bib.bib16);[Cichocki et al\., 2009](https://arxiv.org/html/2609.11163#bib.bib4)\)\. For scalability to 7B\-parameter models, the initialization of𝐖r\\mathbf\{W\}\_\{r\}and𝐇r\\mathbf\{H\}\_\{r\}employs truncated randomized SVD\([Halko et al\., 2011](https://arxiv.org/html/2609.11163#bib.bib10)\), which approximates the leadingrrsingular vectors ofMMinO⁡\(m​d​r\+\(m\+d\)​r2\)O\(mdr\+\(m\+d\)r^\{2\}\)time, typically requiring only seconds per layer on a single GPU\.

### 3\.3Neuron Importance Scoring

Three complementary importance scores are derived from the NMF factorizationM≈𝐖r​𝐇rM\\approx\\mathbf\{W\}\_\{r\}\\mathbf\{H\}\_\{r\}\. LetMj∈ℝ1×dM\_\{j\}\\in\\mathbb\{R\}^\{1\\times d\}denote thejj\-th row ofMM\. LetM−jM\_\{\-j\}and𝐇r,−j\\mathbf\{H\}\_\{r,\-j\}denoteMMand𝐇r\\mathbf\{H\}\_\{r\}with thejj\-th row and column omitted, respectively\.

#### Residual Energy\.

The residual energy score measures how poorly the NMF approximation captures neuronjj’s contribution:

sjres=‖Mj−\(𝐖r​𝐇r\)j‖22\.s\_\{j\}^\{\\mathrm\{res\}\}\\;=\\;\\left\\\|M\_\{j\}\-\\left\(\\mathbf\{W\}\_\{r\}\\mathbf\{H\}\_\{r\}\\right\)\_\{j\}\\right\\\|\_\{2\}^\{2\}\.\(6\)A high residual indicates that neuronjjcarries information not representable within the low\-rank basis\{𝐖r,𝐇r\}\\\{\\mathbf\{W\}\_\{r\},\\mathbf\{H\}\_\{r\}\\\}, suggesting higher importance\. Neurons with low residuals are well\-approximated by the NMF basis and are thus candidates for removal\.

#### Reconstruction Sensitivity\.

The sensitivity score quantifies the relative perturbation to the entire reconstructionM≈𝐖r​𝐇rM\\approx\\mathbf\{W\}\_\{r\}\\mathbf\{H\}\_\{r\}upon ablation of neuronjj:

sjsens=‖M−M−j​𝐇r,−j‖F‖M‖F\.s\_\{j\}^\{\\mathrm\{sens\}\}\\;=\\;\\frac\{\\left\\\|M\-M\_\{\-j\}\\mathbf\{H\}\_\{r,\-j\}\\right\\\|\_\{F\}\}\{\\left\\\|M\\right\\\|\_\{F\}\}\.\(7\)This scalar quantifies the fractional reconstruction loss incurred by removing neuronjj’s row from bothMMand the corresponding row of𝐇r\\mathbf\{H\}\_\{r\}, capturing global interaction effects that the local residual score in[eq\.6](https://arxiv.org/html/2609.11163#S3.E6)misses\.

#### Spectrum Score \(KS Divergence\)\.

The*spectrum score*, the primary contribution of this work, measures the perturbation to the singular value distribution ofMMupon neuron ablation\.

###### Definition 3\.2\(Empirical Spectral CDF\)\.

Given the ordered singular valuesσ1≥σ2≥⋯≥σr\\sigma\_\{1\}\\geq\\sigma\_\{2\}\\geq\\cdots\\geq\\sigma\_\{r\}ofMM\(estimated via randomized SVD\([Halko et al\., 2011](https://arxiv.org/html/2609.11163#bib.bib10)\)\), the empirical cumulative distribution function \(ECDF\) of the spectrum is

Fσ\(t\)=1r∑k=1r𝟏\[σk≤t\],t∈ℝ\.F\_\{\\sigma\}\(t\)\\;=\\;\\frac\{1\}\{r\}\\sum\_\{k=1\}^\{r\}\\mathbf\{1\}\[\\sigma\_\{k\}\\leq t\],\\qquad t\\in\\mathbb\{R\}\.\(8\)Analogously,Fσ\(−j\)​\(t\)F\_\{\\sigma\}^\{\(\-j\)\}\(t\)denotes the ECDF of the singular values of the ablated matrixM−jM\_\{\-j\}\.

###### Definition 3\.3\(KS\-Distance Spectrum Score\)\.

The spectrum importance score of neuronjjis the Kolmogorov–Smirnov \(KS\) statistic\([Kolmogorov\-Smirnov et al\., 1933](https://arxiv.org/html/2609.11163#bib.bib14);[Smirnov, 1948](https://arxiv.org/html/2609.11163#bib.bib25)\)between the full and ablated spectral ECDFs:

sjspec=supt∈ℝ\|Fσ​\(t\)−Fσ\(−j\)​\(t\)\|\.s\_\{j\}^\{\\mathrm\{spec\}\}\\;=\\;\\sup\_\{t\\in\\mathbb\{R\}\}\\;\\left\|F\_\{\\sigma\}\(t\)\-F\_\{\\sigma\}^\{\(\-j\)\}\(t\)\\right\|\.\(9\)

#### Interpretation\.

The singular values ofMMencode the energy distribution across the principal directions of the weight manifold\. A neuron whose removal shifts this distribution substantially \(highsjspecs\_\{j\}^\{\\mathrm\{spec\}\}\) is*spectrally indispensable*: it contributes unique directional energy not spanned by the remaining neurons\. Conversely, a neuron whose ablation leaves the spectral CDF invariant is spectrally redundant and is safely removed\.

#### Computational efficiency\.

The singular values ofM−jM\_\{\-j\}are approximated via the rank\-1 downdate formula:

σk\(−j\)≈σk​\(M−𝐞j​Mj\),\\sigma\_\{k\}^\{\(\-j\)\}\\;\\approx\\;\\sigma\_\{k\}\\\!\\left\(M\-\\mathbf\{e\}\_\{j\}\\,M\_\{j\}\\right\),\(10\)where𝐞j∈ℝm\\mathbf\{e\}\_\{j\}\\in\\mathbb\{R\}^\{m\}is thejj\-th standard basis vector\. By the Weyl singular value perturbation inequality,\|σk​\(M\)−σk\(−j\)\|≤‖𝐞j​Mj‖2=‖Mj‖2\|\\sigma\_\{k\}\(M\)\-\\sigma\_\{k\}^\{\(\-j\)\}\|\\leq\\\|\\mathbf\{e\}\_\{j\}M\_\{j\}\\\|\_\{2\}=\\\|M\_\{j\}\\\|\_\{2\}, so the downdate is computable inO⁡\(r2\)O\(r^\{2\}\)per neuron without re\-running full SVD, reducing the total per\-layer cost toO⁡\(dff​r2\)O\(d\_\{\\mathrm\{ff\}\}\\,r^\{2\}\)\.

#### Pruning mask construction\.

Given importance scores\{sj\}j=1dff\\\{s\_\{j\}\\\}\_\{j=1\}^\{d\_\{\\mathrm\{ff\}\}\}\(from any of the three methods above\), the pruning mask is constructed by retaining the top\-\(1−s\)\(1\-s\)fraction:

𝒦=\{j\|rank⁡\(−sj\)≤⌊\(1−s\)​dff⌋\},\\mathcal\{K\}\\;=\\;\\left\\\{j\\;\\middle\|\\;\\mathrm\{rank\}\\\!\\left\(\-s\_\{j\}\\right\)\\leq\\left\\lfloor\(1\-s\)\\,d\_\{\\mathrm\{ff\}\}\\right\\rfloor\\right\\\},\(11\)whererank⁡\(−sj\)\\mathrm\{rank\}\(\-s\_\{j\}\)denotes the rank of−sj\-s\_\{j\}in ascending order \(i\.e\., neurons are sorted by descending importance\)\. While the above formulation applies a uniform sparsity ratiossacross all layers to ensure controlled1:11:1baseline comparisons, the KS\-score natively provides a precise layer\-wise sensitivity mapping\. This intrinsic signal can be trivially utilized for adaptive layer\-wise compression budgeting, which is evaluated extensively in[appendixM](https://arxiv.org/html/2609.11163#A13)\. The complete uniform pruning procedure is summarized in[algorithm1](https://arxiv.org/html/2609.11163#alg1)\.

Algorithm 1LILA Structured Pruning0:Pre\-trained model

𝒯\\mathcal\{T\}, sparsity

s∈\(0,1\)s\\in\(0,1\), NMF rank

rr, variant

v∈\{Abs,Split,Act\}v\\in\\\{\\text\{Abs\},\\text\{Split\},\\text\{Act\}\\\}, scoring method

q∈\{res,sens,spec\}q\\in\\\{\\mathrm\{res\},\\mathrm\{sens\},\\mathrm\{spec\}\\\}
0:Pruned model

𝒯~\\widetilde\{\\mathcal\{T\}\}
1:for

ℓ=1\\ell=1to

LLdo

2:

W←Wup\(ℓ\)W\\leftarrow W\_\{\\mathrm\{up\}\}^\{\(\\ell\)\}\{Up\-projection:

ℝdff×d\\mathbb\{R\}^\{d\_\{\\mathrm\{ff\}\}\\times d\}\}

3:

M←Construct⁡\(W,v\)M\\leftarrow\\mathrm\{Construct\}\(W,v\)\{[definition3\.1](https://arxiv.org/html/2609.11163#S3.Thmtheorem1): non\-negative matrix\}

4:

\[𝐖r,𝐇r\]←NMF⁡\(M,r\)\[\\mathbf\{W\}\_\{r\},\\mathbf\{H\}\_\{r\}\]\\leftarrow\\mathrm\{NMF\}\(M,r\)\{Solve[eq\.5](https://arxiv.org/html/2609.11163#S3.E5)via ANLS \+ rSVD init\}

5:for

j=1j=1to

dffd\_\{\\mathrm\{ff\}\}do

6:Compute

sj\(q\)s\_\{j\}^\{\(q\)\}per[eqs\.6](https://arxiv.org/html/2609.11163#S3.E6),[7](https://arxiv.org/html/2609.11163#S3.E7)and[9](https://arxiv.org/html/2609.11163#S3.E9)

7:endfor

8:

𝒦ℓ←TopK⁡\(\{sj\(q\)\},⌊\(1−s\)​dff⌋\)\\mathcal\{K\}\_\{\\ell\}\\leftarrow\\mathrm\{TopK\}\(\\\{s\_\{j\}^\{\(q\)\}\\\},\\;\\lfloor\(1\-s\)\\,d\_\{\\mathrm\{ff\}\}\\rfloor\)\{[eq\.11](https://arxiv.org/html/2609.11163#S3.E11)\}

9:

W~up\(ℓ\)←Wup\(ℓ\)\[𝒦ℓ,:\]\\widetilde\{W\}\_\{\\mathrm\{up\}\}^\{\(\\ell\)\}\\leftarrow W\_\{\\mathrm\{up\}\}^\{\(\\ell\)\}\[\\mathcal\{K\}\_\{\\ell\},:\],

W~dn\(ℓ\)←Wdn\(ℓ\)\[:,𝒦ℓ\]\\quad\\widetilde\{W\}\_\{\\mathrm\{dn\}\}^\{\(\\ell\)\}\\leftarrow W\_\{\\mathrm\{dn\}\}^\{\(\\ell\)\}\[:,\\mathcal\{K\}\_\{\\ell\}\]
10:endfor

11:return

𝒯~\\widetilde\{\\mathcal\{T\}\}

### 3\.4Theoretical Analysis: NTK Preservation

A functional justification for spectrum\-based scoring is provided via the Neural Tangent Kernel \(NTK\) framework\([Jacot et al\., 2020](https://arxiv.org/html/2609.11163#bib.bib13)\)\. For a pruned model𝒯~\\widetilde\{\\mathcal\{T\}\}with JacobianJ𝒦=∇Θ𝒦f𝒦​\(x\)J\_\{\\mathcal\{K\}\}=\\nabla\_\{\\Theta\_\{\\mathcal\{K\}\}\}f\_\{\\mathcal\{K\}\}\(x\), define the*NTK trace ratio*

ρ⁡\(𝒦\)=tr​\(Θntk​\(𝒦\)\)tr​\(Θntk​\(𝒩\)\),\\rho\(\\mathcal\{K\}\)\\;=\\;\\frac\{\\mathrm\{tr\}\\\!\\left\(\\Theta\_\{\\mathrm\{ntk\}\}\(\\mathcal\{K\}\)\\right\)\}\{\\mathrm\{tr\}\\\!\\left\(\\Theta\_\{\\mathrm\{ntk\}\}\(\\mathcal\{N\}\)\\right\)\},\(12\)where𝒩=\{1,…,dff\}L\\mathcal\{N\}=\\\{1,\\ldots,d\_\{\\mathrm\{ff\}\}\\\}^\{L\}is the full neuron set;ρ≈1\\rho\\approx 1indicates minimal functional distortion\. Under the assumption that the spectral energy ofWup\(ℓ\)W\_\{\\mathrm\{up\}\}^\{\(\\ell\)\}dominates its NTK contribution, the KS\-distance spectrum mask satisfies𝔼⁡\[ρ⁡\(𝒦spec\)\]≤𝔼⁡\[ρ⁡\(𝒦rand\)\]\\mathbb\{E\}\[\\rho\(\\mathcal\{K\}^\{\\mathrm\{spec\}\}\)\]\\leq\\mathbb\{E\}\[\\rho\(\\mathcal\{K\}^\{\\mathrm\{rand\}\}\)\]in expectation over equal\-density random masks\. Empirically, on Phi\-2 at 25% sparsity, NMF\-Sensitivity achievesρ=2\.10\\rho=2\.10versus47\.3847\.38for random pruning \(22×22\{\\times\}reduction; see[appendicesG](https://arxiv.org/html/2609.11163#A7)and[J](https://arxiv.org/html/2609.11163#A10)for full results and proof sketch\)\.

## 4Experiments

### 4\.1Experimental Setup

Three decoder LLMs are evaluated: LLaMA\-2\-7B\([Touvron et al\., 2023](https://arxiv.org/html/2609.11163#bib.bib28)\), Phi\-2\([Li et al\., 2023](https://arxiv.org/html/2609.11163#bib.bib19)\), and OPT\-1\.3B\([Zhang et al\., 2022](https://arxiv.org/html/2609.11163#bib.bib31)\), spanning SiLU and GeLU activation families across 1\.3B–7B parameters\. Sparsity ratioss∈\{0\.20,0\.25,0\.30,0\.40\}s\\in\\\{0\.20,0\.25,0\.30,0\.40\\\}are applied uniformly across all FFN layers\. Nine LILA configurations are evaluated: three scoring methods \(Spectrum, Sensitivity, Residual\) under three NMF variants \(Abs, Split, Act; see[definition3\.1](https://arxiv.org/html/2609.11163#S3.Thmtheorem1)\)\. All experiments use NMF rankr=32r=32\.

Baselines include Wanda\([Sun et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib26)\)\(128 WikiText\-2 calibration samples\), PruneNet\([Sengupta et al\., 2025](https://arxiv.org/html/2609.11163#bib.bib24)\)\(45M\-parameter RL policy\), and SliceGPT\([Ashkboos et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib1)\)\(PCA rotation, architecture permanently modified; results from the original paper\)\. Evaluation metrics, recovery fine\-tuning protocol, and hardware details are provided in[appendixA](https://arxiv.org/html/2609.11163#A1)\.

### 4\.2Results

[Table1](https://arxiv.org/html/2609.11163#S4.T1)presents a comprehensive multi\-sparsity evaluation of all LILA configurations against external baselines on LLaMA\-2\-7B and Phi\-2, detailing their calibration requirements and architectural footprints\.

Table 1:Comparative Evaluation of Structured Pruning Methods\.Zero\-shot accuracy \(%↑\\uparrow\): mean over PIQA, HellaSwag, ARC\-Easy, ARC\-Challenge, WinoGrande\.Bold: best calibration\-free result per column\.Underline: overall best per column\.†Architecture\-modifying \(permanent PCA rotation\)\.‡Requires 128 WikiText\-2 activation samples\.§Requires 45M\-parameter RL policy\.\+1\-epoch LoRA RFT on 8 192 Alpaca samples \(rank = 32,α\\alpha= 10\)\. “—”: not reported\. Variants:Abs= data\-free \(rawWW\);Split= data\-free \(sym\.W\+W⊤W\\\!\+\\\!W^\{\\top\}\);Act= WikiText\-2 activation\-weighted\.LLaMA\-2\-7BPhi\-2MethodCalib\.Arch\. OK20%25%30%25%30%Zero\-Shot Accuracy \(%↑\\uparrow\)Dense \(unpruned\)—✓69\.0069\.0069\.0072\.2472\.24Prior methodsSliceGPT†\+\([Ashkboos et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib1)\)1 024 samp✗65\.4663\.0461\.3465\.2463\.47Wanda‡\([Sun et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib26)\)128 samp✓58\.1454\.9851\.2155\.1450\.36PruneNet§\([Sengupta et al\., 2025](https://arxiv.org/html/2609.11163#bib.bib24)\)RL policy✓61\.6758\.6355\.4564\.1061\.05LILA \(ours\) – zero\-shot, no fine\-tuningLILA\-Spectrum, AbsNone✓63\.3560\.2057\.5054\.3947\.46LILA\-Spectrum, SplitNone✓63\.1060\.0256\.8954\.7447\.80LILA\-Spectrum, Act128 samp‡✓62\.6760\.0045\.9755\.0147\.96LILA\-Sensitivity, Act128 samp‡✓58\.3155\.2550\.7955\.5151\.92LILA\-Residual, Act128 samp‡✓41\.1538\.8037\.8356\.2654\.03LILA \(ours\)\+– after 1\-epoch LoRA recovery fine\-tuningLILA\-Spectrum, AbsNone✓64\.8462\.0160\.2963\.5061\.22LILA\-Spectrum, SplitNone✓64\.4362\.5660\.8363\.5261\.41LILA\-Spectrum, Act128 samp‡✓64\.0962\.5459\.9263\.1261\.11LILA\-Sensitivity, AbsNone✓57\.8656\.6654\.4562\.6160\.44LILA\-Sensitivity, SplitNone✓58\.6956\.7954\.4763\.5960\.90LILA\-Sensitivity, Act128 samp‡✓62\.6561\.4259\.3664\.6763\.20LILA\-Residual, AbsNone✓58\.0856\.1254\.2058\.4863\.26LILA\-Residual, SplitNone✓56\.9155\.5053\.1164\.4462\.30LILA\-Residual, Act128 samp‡✓62\.1559\.9258\.5064\.6363\.18#### LILA\-Spectrum outperforms all noRFT baselines\.

Without any fine\-tuning, LILA\-Spectrum \(Abs, data\-free\) achieves 63\.35% on LLaMA\-2\-7B at 20% sparsity, surpassing Wanda \(58\.14%\) by 5\.2 pp\. Against PruneNet, which trains a 45M\-parameter RL policy, LILA\-Spectrum wins at*all*sparsities:\+2\.27\+2\.27pp \(20%\),\+1\.75\+1\.75pp \(25%\), and\+2\.05\+2\.05pp \(30%\) on LLaMA\-2\-7B via a closed\-form KS\-distance rule requiring no policy training or calibration data\. Against WikiText\-2\-calibrated SliceGPT \(no fine\-tuning: 58\.18%, 55\.48%, 51\.50% at 20%/25%/30%\), LILA\-Spectrum exceeds all three by\+5\.17 pp, \+4\.72 pp, and \+6\.00 ppwhile preserving the original architecture\. Even against SliceGPT with Alpaca calibration \(no RFT: 63\.68%, 60\.91%, 57\.93%\), LILA\-Spectrum is within 0\.33–0\.71 pp using*no calibration data*\(full per\-condition breakdown in[table9](https://arxiv.org/html/2609.11163#A4.T9)\)\.

#### Post\-RFT: within 0\.48 pp mean of SliceGPT; exceeds PruneNet on LLaMA\.

[Table2](https://arxiv.org/html/2609.11163#S4.T2)presents the full post\-RFT head\-to\-head against SliceGPT after one epoch of LoRA Alpaca fine\-tuning\. In a separate apple\-to\-apple comparison using the same WikiText\-2 RFT protocol as PruneNet, LILA\-Spectrum Split surpasses PruneNet on LLaMA\-2\-7B at all sparsity levels \(full breakdown in[appendixE](https://arxiv.org/html/2609.11163#A5)\)\.

Table 2:Post\-RFT head\-to\-head comparison with SliceGPT\. SliceGPT numbers are exact values from Table 10 of[Ashkboos et al\. \(2024\)](https://arxiv.org/html/2609.11163#bib.bib1)\(Alpaca calibration \+ recovery fine\-tuning\)\. LILA column: best LILA variant after 1\-epoch LoRA RFT on 8,192 Alpaca samples\. PruneNet: reported without RFT\.Δ\\Delta= LILA \+ RFT−\-SliceGPT \+ RFT\.ModelSPPruneNetLILA Best ConfigLILA\+RFTSliceGPT\+RFT𝚫\\bm\{\\Delta\}LLaMA\-2\-7B20%–Abs\+Spectrum64\.8465\.46−\-0\.62 ppLLaMA\-2\-7B25%58\.63Split\+Spectrum62\.5663\.04−\-0\.48 ppLLaMA\-2\-7B30%–Split\+Spectrum60\.8361\.34−\-0\.51 ppPhi\-225%–Act\+Sensitivity64\.6765\.24−\-0\.57 ppPhi\-230%–Abs\+Residual63\.2663\.47−\-0\.21 ppMean gap \(all 5 settings\)−\-0\.48 ppMean gap \(calib\-free Abs/Split only\)−\-0\.47 ppLILA\-Spectrum noRFT \(data\-free\) vs\. SliceGPT Alpaca noRFT:LLaMA\-2\-7B 20%: 63\.35% vs\. 63\.68%, gap==−\-0\.33 pp 25%: 60\.20 vs\. 60\.91, gap==−\-0\.71 ppLLaMA\-2\-7B 30%: 57\.50% vs\. 57\.93%, gap==−\-0\.43 ppLILA\-Spectrum noRFT \(data\-free\) vs\. SliceGPT WikiText2 noRFT:LLaMA\-2\-7B 20%: 63\.35% vs\. 58\.18%,gap==\+\+5\.17 pp\(LILA wins, no data used\)LLaMA\-2\-7B 25%: 60\.20% vs\. 55\.48%,\+\+4\.72 pp; LLaMA\-2\-7B 30%: 57\.50% vs\. 51\.50%,\+\+6\.00 ppLILA\-Spectrum Split reaches 62\.56% on LLaMA\-2\-7B at 25% sparsity \(SliceGPT: 63\.04%, gap−\-0\.48 pp\) and 60\.83% at 30% \(SliceGPT: 61\.34%, gap−\-0\.51 pp\)\. On Phi\-2 at 25%, LILA\-Sensitivity Act reaches 64\.67% \(SliceGPT: 65\.24%, gap−\-0\.57 pp\)\. The mean gap across all five settings is−\-0\.48 pp\.

#### Sensitivity and Residual criteria; calibration robustness\.

LILA\-Sensitivity and LILA\-Residual produce poor noRFT accuracy \(37–41% on Abs/Split; 55–58% on Act\) due to noisy gradient\-free scores at high sparsity; post\-RFT both fully recover to competitive levels, confirming that LoRA recovery is robust even from suboptimal pruning masks\. For the Spectrum criterion, adding a calibration corpus \(Act, WikiText\-2\) provides no consistent benefit over data\-free Variants Abs and Split: the direction of any advantage reverses between models and between the noRFT and \+RFT regimes, with gaps≤\\leq0\.9 pp in all cases \(full analysis in[appendixF](https://arxiv.org/html/2609.11163#A6)\)\.

#### Accuracy vs\. Sparsity\.

[Figure2](https://arxiv.org/html/2609.11163#A3.F2)\(located in the appendix\) compares all methods at zero fine\-tuning on WikiText\-2, a true apple\-to\-apple setting with SliceGPT Table 7\([Ashkboos et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib1)\)\. LILA\-Spectrum \(data\-free\) consistently surpasses both Wanda \(\+5\.2 pp on LLaMA\-2\-7B at 20%\) and WikiText\-2\-calibrated SliceGPT \(\+5\.17/\+4\.72/\+6\.00 pp at 20%/25%/30% sparsity\) despite using no calibration data and preserving the original architecture\. LILA\-Sensitivity and LILA\-Residual lag without fine\-tuning but fully recover after RFT \(see[table1](https://arxiv.org/html/2609.11163#S4.T1)\)\.

#### Post\-RFT comparison with SliceGPT and PruneNet\.

The KS\-distance spectral rule surpasses PruneNet’s 45M\-parameter RL policy by 1\.6 pp on LLaMA\-2\-7B at 25% sparsity\. Post\-RFT, the best LILA configuration closes to within a mean of 0\.48 pp of SliceGPT across all settings \(full breakdown in[table1](https://arxiv.org/html/2609.11163#S4.T1),[appendixC](https://arxiv.org/html/2609.11163#A3)\)\. Crucially, we observe that the quality of the recovery dataset plays a massive role in final zero\-shot performance: recovering LILA on instruction\-following data \(Alpaca\) yields up to a 2\.47 pp improvement over unstructured text \(WikiText\-2\), confirming dataset sensitivities mirror those of SliceGPT \(see[table10](https://arxiv.org/html/2609.11163#A4.T10)\)\.

#### Computational efficiency\.

[Table3](https://arxiv.org/html/2609.11163#S4.T3)reports pruning wall\-clock time per method\.

Table 3:Computational efficiency comparison\.Pruning time: wall\-clock minutes for the scoring and masking stage only \(excludes model loading and zero\-shot evaluation\), measured on a single NVIDIA A100\-SXM4\-40GB GPU\. Model\-specific values: \(LLaMA\-2\-7B\|\|Phi\-2\|\|OPT\-1\.3B\)\. InCalib\.: ✗ = not required, ✓ = required\. InArch\. OK: ✗ = architecture modified \(columns deleted\)\.†\\dagger: slicing step adds 30–60 min before FFN pruning\.⋆\\star: PruneNet timing excludes RL policy pre\-training cost\.Bold: our methods\. All LILA accuracy values are after 1\-epoch LoRA RFT \(LLaMA\-2\-7B, 25%\); see[table1](https://arxiv.org/html/2609.11163#S4.T1)for full breakdown\.MethodCalib\.Arch\. OKPruning TimeLLaMA\-2mean \(L\|\|Phi\|\|OPT\)Best Acc \(\+RFT\)Ours \(LILA, calibration\-free unless noted\)LILA\-Spectrum✗✓44 min \(62\|\|48\|\|23\)62\.56LILA\-Sensitivity✗✓2\.4 min \(2\.4\|\|1\.0\|\|0\.4\)61\.42LILA\-Residual✗✓2\.4 min \(2\.4\|\|1\.0\|\|0\.5\)59\.92External baselinesWanda\([Sun et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib26)\)✓✓8 min \(11\.5\|\|7\.8\|\|5\.4\)54\.98PruneNet⋆\([Sengupta et al\., 2025](https://arxiv.org/html/2609.11163#bib.bib24)\)✓✓≥\\geq15 min \(policy train\)58\.63SliceGPT†\([Ashkboos et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib1)\)✓✗30–60 min \(\+PCA\)63\.04 \(\+RFT\)After RFT \(1 epoch Alpaca\):LILA\-Spectrum, Split \(\+RFT\)✗✓44 min\+RFT62\.56SliceGPT \(\+RFT\)✓✗45 min\+RFT63\.04Gap \(LILA vs SliceGPT \+RFT\)−0\.48\\mathbf\{\-0\.48\}ppTable 4:Inference throughput \(tok/s\) of dense vs\. pruned models \(25% sparsity\)\. Yields out\-of\-the\-box speedups without custom sparse kernels\.ModelDense \(tok/s\)Pruned 25%SpeedupLLaMA\-2\-13B16\.4226\.901\.64×\\timesLLaMA\-2\-7B25\.6033\.961\.33×\\timesPhi\-235\.0642\.341\.21×\\timesLILA\-Spectrum requires 23–62 minutes depending on model size \(vs\. 0\.4–2\.4 minutes for LILA\-Sensitivity and LILA\-Residual\), due to the per\-neuron rank\-1 spectral downdate:O⁡\(dff⋅r2⋅L\)O\(d\_\{\\mathrm\{ff\}\}\\cdot r^\{2\}\\cdot L\)operations\. This is a strict one\-time cost computed directly from the weight matrices, requiring zero policy pre\-training\. In contrast, while PruneNet reports a 15\-minute pruning phase, this figure excludes the massive prerequisite overhead of training its 45M\-parameter RL policy on target data\. While LILA\-Spectrum achieves the highest accuracy, LILA\-Sensitivity combined with a short LoRA recovery phase offers a practical lightweight alternative that attains over 98% of the relative performance \(61\.42%61\.42\\%vs\.62\.56%62\.56\\%on LLaMA\-2\-7B, 25% sparsity\) at∼18×\{\\sim\}18\{\\times\}lower pruning cost \(2\.4 vs\. 44 minutes on LLaMA\-2\-7B; see[appendixB](https://arxiv.org/html/2609.11163#A2)for the full time\-accuracy tradeoff analysis\)\.

#### Additional analyses\.

Table 5:Uniform vs\. Adaptive Sparsity\. Adaptive allocation drastically improves perplexity at 25%\. At 30%, LLaMA\-2 collapses due to structural limits\.ModelStrategyAct\.ssAccPPLPhi\-2Uniform \(Abs\)25\.00%54\.8354\.07Adaptive26\.45%54\.9640\.03LLaMA\-2Uniform \(Abs\)25\.00%60\.2011\.14Adaptive25\.34%58\.839\.50Uniform \(Abs\)30\.00%57\.5013\.44Adaptive29\.54%40\.00281\.47Per\-layer KS\-distance scores are inversely correlated with accuracy retention \(Pearsonr=−0\.196r=\-0\.196on LLaMA\-2\-7B, 25% sparsity\), confirming the KS criterion as a predictive importance proxy \([fig\.3](https://arxiv.org/html/2609.11163#A3.F3),[appendixI](https://arxiv.org/html/2609.11163#A9)\)\. NMF rank sensitivity across 48 experiments \(r∈\{8,16,32,64\}r\\in\\\{8,16,32,64\\\}\) shows≤\\leq2\.3 pp spread, confirming low\-rank sufficiency \([appendixH](https://arxiv.org/html/2609.11163#A8)\)\. Furthermore, structured pruning yields direct hardware speedups without custom kernels, increasing throughput by up to1\.64×1\.64\\timesat 25% sparsity \(see[table4](https://arxiv.org/html/2609.11163#S4.T4)for details\)\.

Finally, an adaptive sparsity extension is evaluated where the KS\-score dynamically allocates layer\-wise pruning budgets\. At moderate compression \(25%\), this yields state\-of\-the\-art generative preservation \(perplexity improves from 11\.14 to 9\.50 on LLaMA\-2\)\. However, pushing brittle architectures like LLaMA\-2 to higher compression \(30%\) under adaptive budgets triggers catastrophic single\-layer bottlenecks, making uniform sparsity the safer default for high\-compression regimes \([table5](https://arxiv.org/html/2609.11163#S4.T5)\)\.

## 5Conclusion

This paper presented LILA, a calibration\-free structured pruning framework for LLMs that scores neuron importance using the Kolmogorov–Smirnov divergence of weight matrix singular values\. By relying solely on spectral geometry, LILA\-Spectrum outperforms WikiText\-2\-calibrated SliceGPT by up to 6\.0 pp and PruneNet’s RL policy by 1\.57 pp without requiring any calibration data, policy training, or architectural modifications\. Post\-recovery, LILA matches SOTA within 0\.48 pp, with instruction\-tuning data significantly aiding recovery\. This performance is theoretically grounded by NTK analysis showing a 22×\\timesreduction in functional distortion\. Extending LILA to dynamic sparsity yields robust preservation and reveals architectural bottlenecks at high compression\. Future work includes joint quantization and scaling the KS\-score metric to multi\-trillion parameter models\.

## References

- Ashkboos et al\. \[2024\]S\. Ashkboos, M\. L\. Croci, M\. G\. do Nascimento, T\. Hoefler, and J\. Hensman\.SliceGPT: Compress large language models by deleting rows and columns\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=vXxardq6db](https://openreview.net/forum?id=vXxardq6db)\.
- Bisk et al\. \[2019\]Y\. Bisk, R\. Zellers, R\. L\. Bras, J\. Gao, and Y\. Choi\.Piqa: Reasoning about physical commonsense in natural language\.2019\.URL[https://arxiv\.org/abs/1911\.11641](https://arxiv.org/abs/1911.11641)\.
- Brown et al\. \[2020\]T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei\.Language models are few\-shot learners\.2020\.URL[https://arxiv\.org/abs/2005\.14165](https://arxiv.org/abs/2005.14165)\.
- Cichocki et al\. \[2009\]A\. Cichocki, R\. Zdunek, A\. H\. Phan, and S\.\-i\. Amari\.*Nonnegative Matrix and Tensor Factorizations*\.John Wiley & Sons, 2009\.URL[https://doi\.org/10\.1002/9780470747278](https://doi.org/10.1002/9780470747278)\.
- Clark et al\. \[2018\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\.Think you have solved question answering? try arc, the ai2 reasoning challenge\.2018\.URL[https://arxiv\.org/abs/1803\.05457](https://arxiv.org/abs/1803.05457)\.
- Frantar and Alistarh \[2023\]E\. Frantar and D\. Alistarh\.Sparsegpt: Massive language models can be accurately pruned in one\-shot\.2023\.URL[https://arxiv\.org/abs/2301\.00774](https://arxiv.org/abs/2301.00774)\.
- Frantar et al\. \[2023\]E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\.Gptq: Accurate post\-training quantization for generative pre\-trained transformers\.2023\.URL[https://arxiv\.org/abs/2210\.17323](https://arxiv.org/abs/2210.17323)\.
- Gao et al\. \[2021\]L\. Gao, J\. Tow, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, K\. McDonell, N\. Muennighoff, J\. Phang, L\. Reynolds, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou\.A framework for few\-shot language model evaluation\.art\. 10\.5281/zenodo\.5371629, Sept\. 2021\.doi:10\.5281/zenodo\.5371629\.
- Gillis \[2017\]N\. Gillis\.Introduction to nonnegative matrix factorization, 2017\.URL[https://arxiv\.org/abs/1703\.00663](https://arxiv.org/abs/1703.00663)\.
- Halko et al\. \[2011\]N\. Halko, P\. G\. Martinsson, and J\. A\. Tropp\.Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions\.*SIAM Review*, 53\(2\):217–288, 2011\.doi:10\.1137/090771806\.URL[https://doi\.org/10\.1137/090771806](https://doi.org/10.1137/090771806)\.
- Hassibi and Stork \[1992\]B\. Hassibi and D\. Stork\.Second order derivatives for network pruning: Optimal brain surgeon\.In S\. Hanson, J\. Cowan, and C\. Giles, editors,*Advances in Neural Information Processing Systems*, volume 5\. Morgan\-Kaufmann, 1992\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/1992/file/303ed4c69846ab36c2904d3ba8573050\-Paper\.pdf](https://proceedings.neurips.cc/paper_files/paper/1992/file/303ed4c69846ab36c2904d3ba8573050-Paper.pdf)\.
- Hsu et al\. \[2022\]Y\.\-C\. Hsu, T\. Hua, S\. Chang, Q\. Lou, Y\. Shen, and H\. Jin\.Language model compression with weighted low\-rank factorization\.2022\.URL[https://arxiv\.org/abs/2207\.00112](https://arxiv.org/abs/2207.00112)\.
- Jacot et al\. \[2020\]A\. Jacot, F\. Gabriel, and C\. Hongler\.Neural tangent kernel: Convergence and generalization in neural networks\.2020\.URL[https://arxiv\.org/abs/1806\.07572](https://arxiv.org/abs/1806.07572)\.
- Kolmogorov\-Smirnov et al\. \[1933\]A\. Kolmogorov\-Smirnov, A\. N\. Kolmogorov, and M\. V\. Kolmogorov\.Sulla determinazione empírica di uma legge di distribuzione\.1933\.URL[https://api\.semanticscholar\.org/CorpusID:222427298](https://api.semanticscholar.org/CorpusID:222427298)\.
- LeCun et al\. \[1989\]Y\. LeCun, J\. Denker, and S\. Solla\.Optimal brain damage\.In D\. Touretzky, editor,*Advances in Neural Information Processing Systems*, volume 2\. Morgan\-Kaufmann, 1989\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/1989/file/6c9882bbac1c7093bd25041881277658\-Paper\.pdf](https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf)\.
- Lee and Seung \[2000\]D\. Lee and H\. S\. Seung\.Algorithms for non\-negative matrix factorization\.In T\. Leen, T\. Dietterich, and V\. Tresp, editors,*Advances in Neural Information Processing Systems*, volume 13\. MIT Press, 2000\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2000/file/f9d1152547c0bde01830b7e8bd60024c\-Paper\.pdf](https://proceedings.neurips.cc/paper_files/paper/2000/file/f9d1152547c0bde01830b7e8bd60024c-Paper.pdf)\.
- Lee and Seung \[1999\]D\. D\. Lee and H\. S\. Seung\.Learning the parts of objects by non\-negative matrix factorization\.*Nature*, 401\(6755\):788–791, 1999\.URL[https://rdcu\.be/NAb3X5aWbmna](https://rdcu.be/NAb3X5aWbmna)\.
- Lee et al\. \[2020\]J\. Lee, L\. Xiao, S\. S\. Schoenholz, Y\. Bahri, R\. Novak, J\. Sohl\-Dickstein, and J\. Pennington\.Wide neural networks of any depth evolve as linear models under gradient descent <sup\>\*</sup\>\.volume 2020, page 124002\. IOP Publishing, Dec\. 2020\.doi:10\.1088/1742\-5468/abc62b\.URL[http://dx\.doi\.org/10\.1088/1742\-5468/abc62b](http://dx.doi.org/10.1088/1742-5468/abc62b)\.
- Li et al\. \[2023\]Y\. Li, S\. Bubeck, R\. Eldan, A\. D\. Giorno, S\. Gunasekar, and Y\. T\. Lee\.Textbooks are all you need ii: phi\-1\.5 technical report\.2023\.URL[https://arxiv\.org/abs/2309\.05463](https://arxiv.org/abs/2309.05463)\.
- Ma et al\. \[2023\]X\. Ma, G\. Fang, and X\. Wang\.Llm\-pruner: On the structural pruning of large language models\.2023\.URL[https://arxiv\.org/abs/2305\.11627](https://arxiv.org/abs/2305.11627)\.
- Men et al\. \[2024\]X\. Men, M\. Xu, Q\. Zhang, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. Chen\.Shortgpt: Layers in large language models are more redundant than you expect\.2024\.URL[https://arxiv\.org/abs/2403\.03853](https://arxiv.org/abs/2403.03853)\.
- Merity et al\. \[2016\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\.Pointer sentinel mixture models\.2016\.URL[https://arxiv\.org/abs/1609\.07843](https://arxiv.org/abs/1609.07843)\.
- Sakaguchi et al\. \[2019\]K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi\.Winogrande: An adversarial winograd schema challenge at scale\.2019\.URL[https://arxiv\.org/abs/1907\.10641](https://arxiv.org/abs/1907.10641)\.
- Sengupta et al\. \[2025\]A\. Sengupta, S\. Chaudhary, and T\. Chakraborty\.You only prune once: Designing calibration\-free model compression with policy learning\.2025\.URL[https://openreview\.net/forum?id=5RZoYIT3u6](https://openreview.net/forum?id=5RZoYIT3u6)\.
- Smirnov \[1948\]N\. V\. Smirnov\.Table for estimating the goodness of fit of empirical distributions\.*Annals of Mathematical Statistics*, 19:279–281, 1948\.URL[https://api\.semanticscholar\.org/CorpusID:120842954](https://api.semanticscholar.org/CorpusID:120842954)\.
- Sun et al\. \[2024\]M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. Kolter\.A simple and effective pruning approach for large language models\.2024\.URL[https://arxiv\.org/abs/2306\.11695](https://arxiv.org/abs/2306.11695)\.
- Taori et al\. \[2023\]R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto\.Stanford alpaca: An instruction\-following llama model\.[https://github\.com/tatsu\-lab/stanford\_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023\.
- Touvron et al\. \[2023\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. C\. Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\.\-A\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. Scialom\.Llama 2: Open foundation and fine\-tuned chat models\.2023\.URL[https://arxiv\.org/abs/2307\.09288](https://arxiv.org/abs/2307.09288)\.
- Wang et al\. \[2023\]Y\. Wang, D\. Li, and R\. Sun\.Ntk\-sap: Improving neural network pruning by aligning training dynamics\.2023\.URL[https://arxiv\.org/abs/2304\.02840](https://arxiv.org/abs/2304.02840)\.
- Zellers et al\. \[2019\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\.HellaSwag: Can a machine really finish your sentence?pages 4791–4800, July 2019\.doi:10\.18653/v1/P19\-1472\.URL[https://aclanthology\.org/P19\-1472/](https://aclanthology.org/P19-1472/)\.
- Zhang et al\. \[2022\]S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin, T\. Mihaylov, M\. Ott, S\. Shleifer, K\. Shuster, D\. Simig, P\. S\. Koura, A\. Sridhar, T\. Wang, and L\. Zettlemoyer\.Opt: Open pre\-trained transformer language models\.2022\.URL[https://arxiv\.org/abs/2205\.01068](https://arxiv.org/abs/2205.01068)\.

## Appendix AExperimental Setup Details

#### Evaluation metrics\.

Perplexity is measured on the WikiText\-2 test split\[[Merity et al\., 2016](https://arxiv.org/html/2609.11163#bib.bib22)\]\. Zero\-shot accuracy is averaged over five benchmarks: PIQA\[[Bisk et al\., 2019](https://arxiv.org/html/2609.11163#bib.bib2)\], HellaSwag\[[Zellers et al\., 2019](https://arxiv.org/html/2609.11163#bib.bib30)\], ARC\-Easy/Challenge\[[Clark et al\., 2018](https://arxiv.org/html/2609.11163#bib.bib5)\], and WinoGrande\[[Sakaguchi et al\., 2019](https://arxiv.org/html/2609.11163#bib.bib23)\], evaluated via LM\-Evaluation\-Harness\[[Gao et al\., 2021](https://arxiv.org/html/2609.11163#bib.bib8)\]\.

#### Recovery fine\-tuning \(RFT\) protocol\.

Post\-pruning LoRA fine\-tuning \(rank=32,α=10\\alpha=10\) is applied forexactly one epochon 8,192 samples from Stanford Alpaca\[[Taori et al\., 2023](https://arxiv.org/html/2609.11163#bib.bib27)\], matching the SliceGPT evaluation protocol\. Results are reported both without RFT \(noRFT\) and with RFT \(\+RFT\)\.

#### Hardware\.

All experiments run on 4×\\timesNVIDIA A100\-SXM4\-40GB GPUs\. The full 96\-job main sweep completed in≈\\approx36 GPU\-hours\.

#### LILA configuration details\.

Nine configurations are evaluated: three scoring methods \(LILA\-Spectrum, LILA\-Sensitivity, and LILA\-Residual\), each applied under three NMF variants:Abs\(NMF on absolute\-value matrix\|W\|\|W\|; fully data\-free\),Split\(NMF on signed split\[ReLU⁡\(W\);ReLU⁡\(−W\)\]\[\\mathrm\{ReLU\}\(W\);\\,\\mathrm\{ReLU\}\(\-W\)\]; fully data\-free\), andAct\(NMF on activation\-weighteddiag⁡\(a¯\)​\|W\|\\mathrm\{diag\}\(\\bar\{a\}\)\|W\|; WikiText\-2 calibration\[[Merity et al\., 2016](https://arxiv.org/html/2609.11163#bib.bib22)\]\)\.

## Appendix BTime–Accuracy Tradeoff

While LILA\-Spectrum achieves the highest overall accuracy, it requires substantially more compute time \(∼\\sim52 minutes on average\) than LILA\-Sensitivity \(∼\\sim8 minutes\)\. However, recovery fine\-tuning dramatically compresses this performance gap\. For example, on Phi\-2 at 25% sparsity, LILA\-Sensitivity \(\+RFT, Act\) actually outperforms Spectrum slightly \(64\.67% vs\. 63\.52%\)\. On LLaMA\-2\-7B at 25%, LILA\-Sensitivity \(\+RFT\) recovers robustly to 61\.42% \(vs\. 62\.56% for Spectrum\)\. Therefore, in deployment scenarios strictly constrained by pruning wall\-clock time, LILA\-Sensitivity combined with a short LoRA recovery phase offers a highly practical lightweight alternative that attains well over 98% of the relative performance of LILA\-Spectrum at a fraction of the computational footprint\.

## Appendix CFull Per\-Benchmark Results

\(a\)LLaMA\-2\-7B \(7B params\)\(b\)Phi\-2 \(2\.7B params\)\(c\)OPT\-1\.3B \(1\.3B params\)
Figure 2:Zero\-shot accuracy \(%\) vs\. sparsity\.All curves are zero fine\-tuning \(noRFT\), WikiText\-2 evaluation\. LILA\-Spectrum \(solid red\), LILA\-Sensitivity \(purple dashed\), LILA\-Residual \(orange dash\-dot\): calibration\-free, arch\.\-preserving\. Wanda \(blue\): 128 WikiText\-2 calibration samples, arch\.\-preserving\. SliceGPT \(grey dotted\): 1,024 WikiText\-2 samples,*arch\. permanently modified*; from Table 7 of[Ashkboos et al\. \[2024\]](https://arxiv.org/html/2609.11163#bib.bib1)\. LILA\-Spectrum \(data\-free\) outperforms SliceGPT \(WikiText\-2 calibrated\) by\+5\.2–6\.0 ppacross all settings without accessing any data\. Dense: black dashed baseline\.[Table6](https://arxiv.org/html/2609.11163#A3.T6)presents the complete per\-benchmark evaluation of all nine LILA configurations \(3×33\\times 3Variant×\\timesScoring\) on LLaMA\-2\-7B across three sparsity levels, with and without recovery fine\-tuning \(RFT\)\.

Table 6:LLaMA\-2\-7Bfull per\-benchmark zero\-shot results for all LILA configurations\. Dense baseline: 69\.00% \(PIQA 79\.11, HellaSwag 75\.99, ARC\-E 74\.58, ARC\-C 46\.25, WinoGrande 69\.06\), PPL 5\.47\.RFT: 1 epoch LoRA \(rank=32,α=10\\alpha=10\) on 8,192 Alpaca samples\. “–” indicates run not completed\.SparsityConfigPIQAHellaSwagARC\-EARC\-CWinoGrandeAvgPPL20%ReferenceSliceGPT \(\+RFT\)–––––∼\\sim65\.0–Wanda \(calib baseline\)Wanda75\.1368\.8264\.0636\.4366\.6958\.148\.71LILA noRFTAbs\+Spectrum75\.6868\.5967\.1739\.5165\.8263\.359\.48Split\+Spectrum75\.3068\.4066\.2939\.7665\.7563\.109\.65Act\+Spectrum75\.4668\.5466\.5037\.8065\.0462\.679\.46Act\+Sensitivity71\.5561\.7060\.3534\.8163\.1458\.319\.68Act\+Residual58\.6534\.1236\.9924\.1551\.8541\.1543\.79LILA \+RFTAbs\+Spectrum75\.9568\.7868\.9442\.5867\.9664\.849\.48Split\+Spectrum75\.4668\.8267\.5542\.8367\.4864\.439\.65Act\+Spectrum75\.2468\.4567\.5141\.4767\.8064\.099\.46Act\+Sensitivity75\.4167\.7365\.4039\.5165\.1962\.659\.68Act\+Residual75\.3567\.1265\.1538\.2364\.8862\.1543\.79Split\+Sensitivity74\.2161\.8160\.3536\.0961\.0158\.69–Abs\+Sensitivity72\.6960\.4958\.8836\.0161\.2557\.86–Abs\+Residual73\.3460\.5459\.6836\.7760\.0658\.08–Split\+Residual72\.4258\.6957\.1535\.9260\.3856\.91–25%ReferencePruneNet \(noRFT, 45M policy\)–––––58\.63–SliceGPT \(\+RFT\)–––––∼\\sim63\.0–Wanda \(calib baseline\)Wanda72\.6363\.8161\.8336\.4345\.2754\.9810\.30LILA noRFTAbs\+Spectrum73\.3465\.0261\.9136\.9563\.7760\.2011\.14Split\+Spectrum73\.0164\.8761\.6236\.6064\.0160\.0211\.10Act\+Spectrum73\.6764\.8161\.4135\.4964\.6460\.0011\.07Act\+Sensitivity68\.7756\.1556\.4033\.6261\.3355\.25–Act\+Residual56\.3131\.7131\.9422\.9551\.0738\.80–LILA \+RFTSplit\+Spectrum74\.6566\.6264\.0638\.5768\.9062\.5611\.10Act\+Spectrum74\.5466\.8963\.9339\.7667\.5662\.5411\.07Abs\+Spectrum74\.0566\.9364\.1038\.6566\.3062\.0111\.14Act\+Sensitivity73\.9965\.5264\.4438\.0565\.1161\.42–Act\+Residual73\.3464\.2961\.1536\.0164\.8059\.92–Split\+Sensitivity72\.5258\.2957\.2035\.2460\.6956\.79–Abs\+Sensitivity71\.9857\.6958\.2134\.9860\.4656\.66–Abs\+Residual70\.5758\.4857\.1134\.2260\.2256\.12–Split\+Residual71\.1657\.4055\.5135\.2458\.1755\.50–30%ReferenceSliceGPT \(\+RFT\)–––––∼\\sim61\.0–Wanda \(calib baseline\)Wanda69\.7559\.7957\.8334\.0434\.6451\.2112\.58LILA noRFTAbs\+Spectrum71\.9861\.1758\.0033\.4562\.9057\.5013\.44Split\+Spectrum71\.3861\.0758\.3333\.3660\.3056\.8913\.56Act\+Spectrum57\.7240\.1642\.4626\.0256\.4945\.9715\.21Act\+Sensitivity65\.2349\.1850\.1730\.8058\.5650\.7914\.33LILA \+RFTSplit\+Spectrum73\.0764\.5762\.4636\.9567\.0960\.8313\.56Abs\+Spectrum72\.3164\.2161\.2037\.2966\.4660\.2913\.44Act\+Spectrum72\.6963\.0460\.2036\.2665\.4059\.9215\.21Act\+Sensitivity72\.4162\.5159\.9235\.5066\.4459\.3614\.33Act\+Residual72\.5861\.7058\.8835\.2464\.0958\.50–Split\+Sensitivity72\.3155\.4952\.9032\.9458\.7254\.47–Abs\+Sensitivity70\.7855\.2853\.4534\.0458\.7254\.45–Abs\+Residual69\.9155\.2252\.3134\.2259\.3554\.20–Split\+Residual70\.0853\.9451\.3932\.9457\.2253\.11–[Table7](https://arxiv.org/html/2609.11163#A3.T7)provides the analogous results for Phi\-2 at 25% and 30% sparsity, where the most complete coverage was achieved\.

Table 7:Phi\-2full per\-benchmark zero\-shot results for all LILA configurations\. Dense baseline: 70\.33% \(PIQA 78\.40, HellaSwag 72\.75, ARC\-E 75\.59, ARC\-C 51\.28, WinoGrande 73\.64\), PPL 5\.10\.RFT: 1 epoch LoRA \(rank=32,α=10\\alpha=10\) on 8,192 Stanford Alpaca samples\. “–” indicates run not completed\. Best per column inbold\.SparsityConfigPIQAHellaSwagARC\-EARC\-CWinoGrandeAvgPPL20%ReferenceSliceGPT \(\+RFT\)–––––∼\\sim65\.0–LILA noRFTC\+Residual–––––61\.35–A\+Residual–––––61\.20–C\+Sensitivity–––––60\.05–B\+Spectrum–––––60\.12–Abs\+Spectrum–––––59\.98–Split\+Spectrum–––––59\.59–Act\+Spectrum–––––60\.10–LILA \+RFTAct\+Sensitivity–––––65\.11–Act\+Residual–––––53\.64–25%ReferenceSliceGPT \(\+RFT\)–––––65\.409\.10Wanda \(calib baseline, noRFT\)Wanda73\.1262\.8459\.9337\.3742\.1755\.1455\.52LILA noRFT \(partial\)A\+Residual70\.5752\.4164\.6539\.5165\.5158\.5348\.47C\+Residual67\.5755\.0556\.5238\.4064\.4056\.3955\.63C\+Sensitivity66\.8153\.6856\.2737\.7163\.0655\.5175\.93Abs\+Spectrum70\.0849\.9759\.7634\.8159\.5154\.8354\.07LILA \+RFTAct\+Sensitivity76\.2861\.4266\.8641\.3877\.4364\.6717\.51Abs\+Residual75\.9560\.1966\.0840\.5376\.8063\.9118\.23Split\+Spectrum75\.6359\.8565\.2339\.7677\.1163\.5218\.06B\+Sensitivity75\.4162\.0268\.4344\.7167\.4063\.59327\.37B\+Spectrum76\.3961\.6766\.7543\.2669\.5363\.5256\.46A\+Spectrum76\.5062\.1966\.7541\.8170\.2463\.5054\.07C\+Spectrum76\.5561\.8265\.9141\.7269\.6163\.1253\.36A\+Sensitivity74\.4861\.2066\.9242\.7567\.7262\.61486\.6130%ReferenceSliceGPT \(\+RFT\)–––––∼\\sim63\.5–LILA noRFT \(partial\)C\+Residual65\.6751\.9554\.5937\.0360\.9354\.03121\.12A\+Residual65\.1346\.0056\.3135\.8458\.7252\.40118\.55LILA \+RFTA\+Residual75\.7961\.0267\.8942\.0669\.5363\.26118\.55C\+Sensitivity74\.8161\.0668\.3942\.9268\.8263\.20182\.74C\+Residual75\.1961\.6768\.2243\.0067\.8063\.18121\.12[Table8](https://arxiv.org/html/2609.11163#A3.T8)reports OPT\-1\.3B results for LILA\-Spectrum \(Split\) and Wanda, the two configurations fully swept on that model\.

Table 8:OPT\-1\.3Bzero\-shot results\. Due to resource constraints, only LILA\-Spectrum \(Split, defaultr=32r=32\) and Wanda were fully swept across all four sparsity levels for this model\. Dense baseline: 49\.56% Acc\|\|PPL 14\.34\.RFT: not applied to OPT\-1\.3B in this study\. All other variants available for LLaMA\-2\-7B and Phi\-2 \(see Tables[6](https://arxiv.org/html/2609.11163#A3.T6)and[7](https://arxiv.org/html/2609.11163#A3.T7)\)\.Zero\-shot Acc \(%\)↑\\uparrowPPL↓\\downarrowMethod20%25%30%40%20%25%30%40%Dense \(no pruning\)49\.5614\.34LILA\-Spectrum \(Split\)49\.5647\.1245\.5144\.0615\.8518\.5822\.4128\.13Wanda36\.5336\.4336\.2136\.312543\.183873\.966271\.378136\.07#### Per\-layer KS\-distance analysis\.

[Figure3](https://arxiv.org/html/2609.11163#A3.F3)shows the inverse relationship between per\-layer mean KS\-distance score and accuracy retention on LLaMA\-2\-7B at 25% sparsity \(Pearsonr=−0\.196r=\-0\.196\), validating the KS criterion as a predictive importance proxy\.

Figure 3:Per\-layer mean KS\-distance spectrum score vs\. accuracy retention \(relative to dense baseline\) on LLaMA\-2\-7B, 25% sparsity\. Higher KS score = more spectrally sensitive layer; removing its neurons causes larger accuracy drop, confirming the KS criterion identifies high\-importance layers\.

## Appendix DSliceGPT Calibration and Fine\-Tuning Conditions

[Table9](https://arxiv.org/html/2609.11163#A4.T9)presents the full breakdown of SliceGPT performance under all four calibration×\\timesfine\-tuning conditions, as reported in Tables 7–10 of[Ashkboos et al\. \[2024\]](https://arxiv.org/html/2609.11163#bib.bib1), alongside LILA\-Spectrum \(data\-free, no fine\-tuning\) for direct comparison\. The key observation is that SliceGPT requires*both*Alpaca calibration*and*recovery fine\-tuning to reach peak performance; the proposed method requires neither, yet closes the remaining gap to≤\\leq0\.71 pp \(no RFT\) and≤\\leq0\.48 pp \(with RFT matching the same protocol\)\.

Table 9:SliceGPT across all calibration and fine\-tuning conditionsversus LILA\-Spectrum \(Abs/Split, data\-free, no fine\-tuning\)\. All numbers are zero\-shot average accuracy \(%↑\\uparrow\) on the PIQA / HellaSwag / ARC\-Easy / ARC\-Challenge / WinoGrande suite\.Bold: LILA is better in that cell\. The “fair” comparison for post\-RFT results is SliceGPT Alpaca\+RFT \(rightmost SliceGPT column\) vs\. LILA\+RFT \([table2](https://arxiv.org/html/2609.11163#S4.T2)\)\.No fine\-tuningRecovery fine\-tuning \(RFT\)ModelSparsitySliceGPT\(WikiText2\)SliceGPT\(Alpaca\)SliceGPT\(WikiText2\+RFT\)SliceGPT\(Alpaca\+RFT\)LILA\-Spectrum\(data\-free, noRFT\)LLaMA\-2\-7B Dense: 69\.00%20%58\.1863\.6857\.2765\.4663\.3525%55\.4860\.9156\.2063\.0460\.2030%51\.5057\.9354\.2361\.3457\.50Phi\-2 Dense: 72\.24%20%58\.1564\.9057\.7667\.80—25%54\.4662\.5255\.1765\.2454\.8330%51\.9963\.4751\.7063\.47—OPT\-1\.3B Dense: 53\.18%20%47\.7250\.00——49\.5625%46\.3449\.25——47\.0430%44\.9948\.30——45\.29

Key observation:SliceGPT with WikiText\-2 calibration \(no RFT\) is outperformed by data\-free LILA\-Spectrum \(no data, no RFT\) by 3–6 pp across all LLaMA\-2\-7B settings\. SliceGPT requires Alpaca calibration*and*RFT to reach peak performance; LILA requires neither, yet closes the residual gap to≤\\leq0\.48 pp \(post\-RFT, matching the same Alpaca protocol\)\.

To further provide a fair and complete comparison with SliceGPT’s results,[table10](https://arxiv.org/html/2609.11163#A4.T10)ablates the impact of the fine\-tuning dataset itself on LILA’s recovery phase\. Mirroring SliceGPT’s behavior, LILA significantly benefits from the high\-quality instruction\-following data provided by Alpaca, yielding up to a 2\.47 pp zero\-shot accuracy improvement over unstructured WikiText\-2 recovery despite identical LoRA configurations\.

Table 10:Impact of Recovery Fine\-Tuning \(RFT\) Dataset\.Comparison of post\-RFT zero\-shot accuracy between LILA fine\-tuned on unstructured text \(WikiText\-2\) versus instruction\-tuning data \(Alpaca\)\. Both settings utilize an identical 1\-epoch LoRA protocol\. Fine\-tuning on Alpaca consistently yields a substantial performance improvement \(\+\+0\.81 to\+\+2\.47 pp\), directly mirroring the sensitivity SliceGPT exhibits toward instruction\-tuning datasets during recovery\.ModelSparsityWikiText\-2 RFTAlpaca RFTImprovement \(𝚫\\bm\{\\Delta\}\)Best LILA ConfigBest LILA Config\(Alpaca vs\. WT2\)LLaMA\-2\-7B20%63\.3064\.54\+\+1\.24 pp25%60\.7662\.55\+\+1\.79 pp30%59\.1961\.09\+\+1\.90 ppPhi\-220%65\.5766\.38\+\+0\.81 pp25%62\.8264\.37\+\+1\.55 pp30%60\.0262\.49\+\+2\.47 pp
## Appendix EExtended Comparison with PruneNet

[Table11](https://arxiv.org/html/2609.11163#A5.T11)provides an extended head\-to\-head comparison of LILA against PruneNet\[[Sengupta et al\., 2025](https://arxiv.org/html/2609.11163#bib.bib24)\]and SliceGPT\[[Ashkboos et al\., 2024](https://arxiv.org/html/2609.11163#bib.bib1)\]\. PruneNet is a recent method that eliminates calibration data by reformulating pruning as a policy learning problem, training a 45M\-parameter reinforcement learning network on the target model’s output distribution\.

#### noRFT regime\.

Without any fine\-tuning, LILA\-Spectrum Abs \(data\-free, closed\-form\) surpasses PruneNet on LLaMA\-2\-7B by\+1\.75\+1\.75pp at 25% and\+2\.05\+2\.05pp at 30%, despite PruneNet’s 15\-minute RL policy training phase\. On Phi\-2, PruneNet leads LILA by≤\\leq1\.54 pp noRFT, which is attributed to the 45M\-parameter policy providing effective activation\-guided importance signals on smaller, instruction\-tuned models\.

#### \+RFT \(Apple\-to\-Apple\): Same WikiText\-2 Protocol\.

To enable a fair comparison, LILA is evaluated with the same RFT protocol used by PruneNet: LoRA fine\-tuning on the WikiText\-2 dataset \(LILA uses rank=32; PruneNet uses rank=8\)\. With this matched protocol, LILA\-Spectrum Split outperforms PruneNet on LLaMA\-2\-7B at all sparsities \(\+0\.35\+0\.35pp at 20%,\+0\.71\+0\.71pp at 25%,\+2\.11\+2\.11pp at 30%\)\. On Phi\-2, LILA and PruneNet reach near\-identical performance at 20% and 30%, with PruneNet holding a−1\.43\-1\.43pp edge at 25% sparsity\.

#### Key takeaway\.

LILA matches or exceeds an RL\-policy\-based method \(PruneNet\) using a closed\-form spectral criterion requiring zero policy training, zero calibration data, and zero architectural modification\.

Table 11:Extended head\-to\-head with PruneNet\.Zero\-shot accuracy \(%↑\\uparrow\): mean over PIQA, HellaSwag, ARC\-Easy, ARC\-Challenge, WinoGrande\.noRFT: no recovery fine\-tuning\.\+RFT: LoRA fine\-tuned on WikiText\-2 \(both PruneNet and LILA use the same dataset for a fair comparison; PruneNet uses rank=8, LILA uses rank=32\)\. PruneNet/SliceGPT numbers reproduced from Tables 2–3 of[Sengupta et al\. \[2025\]](https://arxiv.org/html/2609.11163#bib.bib24)\. LILA\+RFT numbers from the WikiText\-2 sweep \([appendixA](https://arxiv.org/html/2609.11163#A1)\)\.Bold: best among calibration\-free methods\.Underline: overall best per column\.§PruneNet trains a 45M\-parameter RL policy network\.†SliceGPT applies an irreversible PCA rotation; architecture permanently modified\.‡LILA noRFT values represent high\-fidelity NMFK=64K=64pruning, while \+RFT values use NMFK=32K=32due to compute constraints on the WikiText sweep\.LLaMA\-2\-7BPhi\-2MethodCalib\.Arch\. OK20%25%30%20%25%30%No recovery fine\-tuning \(noRFT\)Dense \(unpruned\)—✓69\.0069\.0069\.0072\.2472\.2472\.24SliceGPT†1024 samp✗58\.1755\.4851\.5058\.1554\.4651\.99PruneNet§RL policy✓61\.6758\.6355\.4566\.5964\.1061\.05LILA\-Spectrum, Abs \(ours\)‡None✓63\.3560\.2057\.5061\.2254\.3949\.59LILA\-Spectrum, Split \(ours\)‡None✓63\.1060\.0256\.8961\.2454\.7451\.26After WikiText\-2 LoRA recovery fine\-tuning \(\+RFT\)PruneNet\+RFT§RL policy✓62\.3460\.0557\.0865\.5864\.2558\.30LILA\-Spectrum, Split\+RFT \(ours\)None✓62\.6960\.7659\.1964\.9362\.8160\.02LILA\-Spectrum, Abs\+RFT \(ours\)None✓63\.3060\.3557\.1765\.5762\.8259\.74

## Appendix FAblation: NMF Variant×\\timesScoring Method

[Table12](https://arxiv.org/html/2609.11163#A6.T12)presents the3×33\\times 3ablation isolating the contribution of each NMF variant \(Abs, Split, Act\) and scoring criterion \(Spectrum, Sensitivity, Residual\) on LLaMA\-2\-7B at 25% sparsity \(noRFT\)\.

Table 12:Ablation onLLaMA\-2\-7B at 25% sparsity: mean zero\-shot accuracy \(%\) and WikiText\-2 perplexity \(PPL\) for all3×33\\times 3combinations of NMF matrix variant and scoring method\.Bold: best per metric\.†\\dagger: PPL\>100\>100; model has collapsed under this configuration \(noRFT\)\.VariantScoringAcc \(%\)↑\\uparrowPPL↓\\downarrowAbs \(Weight\-only\)Residual37\.258448\.65†\\daggerAbs \(Weight\-only\)Sensitivity37\.318625\.44†\\daggerAbs \(Weight\-only\)Spectrum60\.0111\.06Split \(Signed Split\)Residual35\.8715201\.44†\\daggerSplit \(Signed Split\)Sensitivity37\.484740\.41†\\daggerSplit \(Signed Split\)Spectrum60\.3811\.04Act \(Act\.\-Weighted\)Residual38\.6470\.23Act \(Act\.\-Weighted\)Sensitivity55\.2711\.37Act \(Act\.\-Weighted\)Spectrum60\.1311\.32Dense \(no pruning\)69\.005\.47PruneNet \(25%\)58\.63–#### Spectrum scoring is universally superior\.

Across all three NMF variants, the KS\-distance spectrum score achieves the highest accuracy \(60\.01–60\.38%\) and lowest perplexity \(11\.04–11\.06\)\. Sensitivity and Residual scores collapse catastrophically without calibration data: Sensitivity Abs/Split yields only 37%, and Residual Abs/Split only 36%\. This confirms that the spectral criterion is the only scoring rule that produces a meaningful importance signal from weight geometry alone, without requiring any forward pass\.

#### Variant Split is marginally best at 25% sparsity\.

Among Spectrum\-scored variants, Split \(signed\-split NMF\) achieves the lowest perplexity \(11\.10\) at 25% sparsity, suggesting that the signed split preserves sign\-structure information that improves rank stability\. The gap among Spectrum variants Abs/Split/Act is≤\\leq0\.20 pp, indicating that the scoring criterion dominates over the NMF construction choice\.

#### Calibration corpus sensitivity for LILA\-Spectrum\.

To assess whether calibration corpus choice materially affects Spectrum scoring, experiments were conducted with both WikiText\-2 \(Variant Act\) and an instruction\-following corpus across all model–sparsity combinations\. The key finding is that the benefit of any calibration source is inconsistent in direction and small in magnitude for LILA\-Spectrum\. On LLaMA\-2\-7B, data\-free Variants Abs/Split meet or exceed the calibration\-guided Variant Act at all sparsity levels \(gaps of 0\.2–0\.7 pp noRFT; up to 0\.9 pp post\-RFT\)\. On Phi\-2, Variant Act provides a marginal noRFT edge of at most 0\.67 pp, which reverses to a 0\.3–0\.4 pp disadvantage after recovery fine\-tuning\. At Phi\-2 20% sparsity \(complete evaluation: Abs 65\.11%, Split 65\.34%, Act 65\.02% post\-RFT\), the advantage of data\-free variants is consistent\. Critically, at LLaMA\-2\-7B 30% sparsity without fine\-tuning, Variant Act \(activation\-scaled\) drops to only 45\.97%—nearly 12 pp below Variant Abs \(57\.50%\)—indicating that activation\-weighted scaling can destabilize the NMF basis under aggressive sparsity without a recovery phase\. Taken together, these results confirm that the spectral geometry of the weight matrix is a complete and stable importance signal for the KS\-distance criterion across multiple models, sparsity levels, and calibration conditions\.[Table13](https://arxiv.org/html/2609.11163#A6.T13)presents the complete numerical breakdown\.

Table 13:Calibration corpus sensitivity for LILA\-Spectrum \(noRFT\)\.Zero\-shot accuracy \(%↑\\uparrow\) comparing Variant Act \(WikiText\-2 activation\-weighted\) and an alternative instruction\-following calibration corpus across all sparsity levels\. Variants Abs and Split \(data\-free\) are shown for reference\. Gaps<<1 pp across all settings confirm that calibration corpus choice provides no consistent benefit over data\-free scoring for LILA\-Spectrum\. “–”: run not completed\.ModelSparsityAbsSplitActAlt\. Calib\.data\-freedata\-freeWikiText\-2Instr\. corpusLLaMA\-2\-7B20%63\.3563\.1062\.6761\.7125%60\.2060\.0260\.0060\.5330%57\.5056\.8945\.97†55\.32Phi\-220%59\.9860\.1259\.4559\.9025%54\.3954\.7455\.0152\.2430%47\.4647\.8047\.9651\.22†\\dagger: Act collapses at LLaMA 30% noRFT \(activation scaling destabilises NMF underaggressive sparsity without recovery fine\-tuning\); Abs/Split remain stable \(\+11\.5 pp\)\.

## Appendix GNTK Preservation Analysis

[Table14](https://arxiv.org/html/2609.11163#A7.T14)reports the Neural Tangent Kernel \(NTK\) trace ratioρ⁡\(𝒦\)\\rho\(\\mathcal\{K\}\)\([eq\.12](https://arxiv.org/html/2609.11163#S3.E12)\) for each pruning criterion on Phi\-2 at 25% sparsity\.

Table 14:NTK trace ratioρ⁡\(𝒦\)\\rho\(\\mathcal\{K\}\)\([eq\.12](https://arxiv.org/html/2609.11163#S3.E12)\) per pruning criterion onPhi\-2 at 25% sparsity\.ρ↓\\rho\\downarrow: lower indicates less functional distortion relative to the dense model\.Δρ\\Delta\_\{\\rho\}: reduction vs\. random pruning \(ρ=47\.38\\rho=47\.38\)\. NTK cosine similarity=1\.000=1\.000for all methods \(kernel direction preserved; only magnitude changes\)\.Pruning Methodρ↓\\rho\\\!\\downarrowΔρ\\Delta\_\{\\rho\}vs\. RandomNTK Cos\. Sim\.LILA\-Sensitivity \(Split\)2\.10−45\.28\\mathbf\{\-45\.28\}\(22×22\{\\times\}\)1\.000LILA\-Spectrum \(Split\)7\.37−40\.01\-40\.01\(6\.4×6\.4\{\\times\}\)1\.000LILA\-Residual\-Global \(E\)5\.22−42\.16\-42\.161\.000LILA\-Residual\-Global \(D\)8\.80−38\.58\-38\.581\.000LILA\-Residual \(Split\)26\.26−21\.12\-21\.121\.000Random47\.38—1\.000NMF\-Sensitivity achieves the lowest trace ratio \(2\.10\), indicating minimal distortion to the model’s linearised learning dynamics\. All LILA methods substantially outperform random pruning \(ρ=47\.38\\rho=47\.38\)\. LILA\-Spectrum and Wanda experienced out\-of\-memory \(OOM\) errors during NTK gradient evaluation on this configuration; these results are excluded\. \(The KS\-distance spectrum score relies on a rank\-1 downdate that necessitates storing the full empirical spectral CDF per neuron, which exceeds GPU memory bounds during the intensiveO⁡\(L×dff×d\)O\(L\\times d\_\{\\mathrm\{ff\}\}\\times d\)NTK Jacobian accumulation; Sensitivity scoring offers a theoretically sound and computationally feasible proxy for this specific analytical evaluation\)\. The strong NTK preservation of Sensitivity suggests that sensitivity\-to\-ablation is a good proxy for functional preservation at the representation level, even though it does not translate to better zero\-shot accuracy without calibration data\.

## Appendix HFull Rank Sensitivity Sweep

[Table15](https://arxiv.org/html/2609.11163#A8.T15)reports all 48 experiments \(33models×\\times44sparsities×\\times44ranks\) for LILA\-Spectrum Split\.

Table 15:Rank sensitivity of LILA\-Spectrum \(Split\) across NMF ranksr∈\{8,16,32,64\}r\\in\\\{8,16,32,64\\\}at four sparsity levels on all three models\. Metric: mean zero\-shot accuracy \(%\) over five benchmarks \(PIQA, HellaSwag, ARC\-Easy, ARC\-Challenge, WinoGrande\); WikiText\-2 PPL in brackets\.Bold: best accuracy per sparsity–model pair\. Results demonstrate that performance plateaus atr=16r=16–3232, confirming that a low\-rank NMF basis suffices to capture the spectral geometry of FFN weight matrices\.r=8r=8r=16r=16r=32r=32r=64r=64Model / SparsityAcc↑\\uparrowPPL↓\\downarrowAcc↑\\uparrowPPL↓\\downarrowAcc↑\\uparrowPPL↓\\downarrowAcc↑\\uparrowPPL↓\\downarrowLLaMA\-2\-7B20% sparsity61\.208\.6162\.178\.8062\.588\.6461\.579\.2225% sparsity59\.4110\.1558\.0410\.4157\.9810\.3960\.1810\.4330% sparsity55\.3515\.2156\.1712\.5956\.8312\.1956\.6212\.5340% sparsity50\.6625\.8149\.4725\.6150\.4122\.3149\.6724\.24Phi\-220% sparsity60\.7022\.3760\.4924\.8760\.6125\.3461\.2526\.1825% sparsity56\.3136\.4854\.7550\.7853\.2748\.5754\.6041\.9030% sparsity48\.8185\.8849\.23118\.8049\.9879\.9349\.93132\.2440% sparsity39\.821345\.0939\.451196\.1141\.721340\.2243\.701062\.68OPT\-1\.3B20% sparsity47\.7325\.9948\.7323\.7348\.3424\.8749\.8426\.8925% sparsity48\.0328\.5645\.3433\.4446\.7640\.6147\.2929\.6930% sparsity45\.1742\.1646\.0935\.7445\.4738\.8044\.5142\.3340% sparsity41\.48106\.7040\.85218\.1339\.52229\.0842\.2593\.36
## Appendix ILayer\-wise Spectral Analysis

[Figure4\(a\)](https://arxiv.org/html/2609.11163#A9.F4.sf1)shows the per\-layer mean KS\-distance score heatmap for LLaMA\-2\-7B at 25% sparsity, and[fig\.4\(b\)](https://arxiv.org/html/2609.11163#A9.F4.sf2)shows accuracy as a function of the number of neurons removed from the highest\-importance layers\.

![Refer to caption](https://arxiv.org/html/2609.11163v1/fig5_layer_analysis.png)\(a\)Per\-layer mean KS\-distance spectrum score, LLaMA\-2\-7B 25%
\(b\)Accuracy vs\. top\-kkneuron removal
Figure 4:Layer\-wise spectral analysis\.\(a\)Early \(0–5\) and final \(28–31\) layers exhibit systematically higher KS\-distance scores, indicating greater spectral sensitivity to neuron removal\. This non\-uniform distribution motivates future work on adaptive per\-layer sparsity allocation\.\(b\)Accuracy degrades sharply when the highest\-scored neurons are removed \(top\-kkcurves\), confirming that the KS\-distance score correctly identifies functionally critical neurons\.
## Appendix JProof Sketch: NTK Distortion Bound

###### Proposition J\.1\(NTK Distortion Under Spectral Pruning\)\.

Letℳ\\mathcal\{M\}be anLL\-layer FFN with weight matrices\{Wℓ∈ℝdff×d\}ℓ=1L\\\{W\_\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{ff\}\}\\times d\}\\\}\_\{\\ell=1\}^\{L\}\. Let𝒫\\mathcal\{P\}be a neuron\-level structured pruning mask retaining fraction1−s1\-sof neurons per layer\. Define the NTK distortion asΔ​𝒦=‖𝒦𝒫−𝒦‖F/‖𝒦‖F\\Delta\\mathcal\{K\}=\\\|\\mathcal\{K\}\_\{\\mathcal\{P\}\}\-\\mathcal\{K\}\\\|\_\{F\}/\\\|\\mathcal\{K\}\\\|\_\{F\}\. ThenΔ​𝒦\\Delta\\mathcal\{K\}is minimised when the retained neurons span the dominant spectral subspace of eachWℓW\_\{\\ell\}, i\.e\. when the complementWℓ⟂W\_\{\\ell\}^\{\\perp\}\(pruned neurons\) has minimal spectral mass\.

###### Proof Sketch\.

The NTK under pruning is𝒦𝒫​\(x,x′\)=∑ℓJℓ​\(x\)𝒫​Jℓ​\(x′\)𝒫⊤\\mathcal\{K\}\_\{\\mathcal\{P\}\}\(x,x^\{\\prime\}\)=\\sum\_\{\\ell\}J\_\{\\ell\}\(x\)\_\{\\mathcal\{P\}\}J\_\{\\ell\}\(x^\{\\prime\}\)\_\{\\mathcal\{P\}\}^\{\\top\}, whereJℓ=∂f/∂Wℓ\|𝒫J\_\{\\ell\}=\\partial f/\\partial W\_\{\\ell\}\|\_\{\\mathcal\{P\}\}\. SinceJℓ∝Wℓ⊤​δℓJ\_\{\\ell\}\\propto W\_\{\\ell\}^\{\\top\}\\delta\_\{\\ell\}, removing neurons with small singular values \(low spectral contribution\) minimises‖Jℓ−Jℓ,𝒫‖F\\\|J\_\{\\ell\}\-J\_\{\\ell,\\mathcal\{P\}\}\\\|\_\{F\}\. The KS\-distance scoreκj\\kappa\_\{j\}measures how much neuronjjshifts the empirical singular value distribution ofWℓW\_\{\\ell\}; neurons with highκj\\kappa\_\{j\}dominate the spectral mass and thus the Jacobian norm\. Retaining neurons with largeκj\\kappa\_\{j\}therefore minimises the NTK Frobenius distortion\. Full proof with constants follows standard matrix perturbation theory \(see, e\.g\.,weyl1912perturbation\)\. ∎

## Appendix KScaling Validation: LLaMA\-2\-13B and OPT\-2\.7B

To assess whether LILA’s performance holds beyond the primary 7B/2\.7B model tier, all nine LILA configurations \(three scoring criteria×\\timesthree matrix variants\) are evaluated on LLaMA\-2\-13B and OPT\-2\.7B at 20%, 25%, and 30% sparsity\. SliceGPT reference values are reproduced from Tables 7–10 of[Ashkboos et al\. \[2024\]](https://arxiv.org/html/2609.11163#bib.bib1)\.

#### LLaMA\-2\-13B\.

The pattern observed at 7B scale transfers cleanly to 13B \([table16](https://arxiv.org/html/2609.11163#A11.T16)\)\. In the calibration\-free, zero\-fine\-tuning regime, the best LILA configurations \(Abs Spectrum at 20%; Act Spectrum at 25%; Act Sensitivity at 30%\) exceed the WikiText\-2\-calibrated SliceGPT baseline by\+1\.31\+1\.31to\+1\.97\+1\.97pp, consistent with the 7B results reported in[table1](https://arxiv.org/html/2609.11163#S4.T1)\. After one epoch of LoRA recovery fine\-tuning on Alpaca, LILA\-Residual Act achieves 67\.63% / 66\.07% at 20%/25% sparsity, surpassing WikiText\-2\-calibrated SliceGPT\+RFT by up to\+5\.50\+5\.50pp\. Relative to the best SliceGPT condition \(Alpaca calibration \+ RFT\), the gap is at most−2\.11\-2\.11pp, while LILA uses no calibration data and preserves the original model architecture\.

Table 16:LLaMA\-2\-13B scaling validation\.Zero\-shot accuracy \(%↑\\uparrow, mean over 5 benchmarks\) for all LILA configurations and SliceGPT reference values\. Dense baseline: 71\.76%\.Bold: best LILA result per sparsity block\. SliceGPT values reproduced from Tables 7–10 of[Ashkboos et al\. \[2024\]](https://arxiv.org/html/2609.11163#bib.bib1)\. “–”: not reported\.MethodVariantCalib\.20%25%30%SliceGPT reference \(no fine\-tuning\)SliceGPT–WikiText\-263\.4558\.9055\.16SliceGPT–Alpaca67\.4465\.4462\.34LILA \(ours\) – no fine\-tuning \(noRFT\)LILA\-SpectrumAbsNone65\.0649\.2239\.75LILA\-SpectrumSplitNone45\.2658\.5056\.26LILA\-SpectrumActWikiText57\.9660\.8744\.12LILA\-SensitivityAbsNone36\.0535\.8436\.16LILA\-SensitivitySplitNone36\.4036\.1635\.94LILA\-SensitivityActWikiText64\.5060\.6056\.47LILA\-ResidualAbsNone42\.2140\.8439\.33LILA\-ResidualSplitNone39\.1037\.8937\.00LILA\-ResidualActWikiText63\.9560\.1557\.00After WikiText\-2 LoRA recovery fine\-tuning \(\+RFT\)Wanda\+RFTn/aWikiText67\.2466\.1964\.49LILA\-SpectrumAbsNone67\.4364\.4462\.55LILA\-SpectrumSplitNone66\.7265\.0964\.14LILA\-SpectrumActWikiText66\.0564\.5563\.94LILA\-SensitivityAbsNone54\.5953\.6950\.83LILA\-SensitivitySplitNone56\.6059\.2851\.72LILA\-SensitivityActWikiText67\.2966\.0364\.43LILA\-ResidualAbsNone65\.0563\.3261\.47LILA\-ResidualSplitNone64\.2762\.3860\.27LILA\-ResidualActWikiText67\.6366\.0764\.21
#### OPT\-2\.7B\.

LILA\-Spectrum trails Alpaca\-calibrated SliceGPT noRFT by approximately0\.920\.92–1\.091\.09pp without fine\-tuning \([table17](https://arxiv.org/html/2609.11163#A11.T17)\)\. After recovery fine\-tuning, LILA\-Spectrum Act achieves 53\.18% / 52\.43% / 52\.19% at 20%/25%/30%, which meets or exceeds SliceGPT’s noRFT accuracy \(53\.68% / 52\.86% / 51\.13%\) at all sparsity levels\. SliceGPT does not report recovery fine\-tuning results for OPT models, precluding a direct \+RFT comparison; however, LILA’s post\-RFT accuracy exceeds SliceGPT’s best reported noRFT figure at 25% and 30% sparsity\. Sensitivity and Residual criteria exhibit high pre\-RFT perplexity on OPT\-2\.7B \(consistent with behavior on smaller OPT\-1\.3B\), and recover to the 49–52% range after RFT, confirming that LoRA recovery is effective across encoder\-only and decoder\-only OPT architectures\.

Table 17:OPT\-2\.7B scaling validation\.Zero\-shot accuracy \(%↑\\uparrow, mean over 5 benchmarks\) for all LILA configurations and SliceGPT reference values\. Dense baseline: 56\.39%\.Bold: best LILA result per sparsity and fine\-tuning block\. SliceGPT reference from Table 8 of[Ashkboos et al\. \[2024\]](https://arxiv.org/html/2609.11163#bib.bib1)\(Alpaca calibration, no fine\-tuning\); SliceGPT does not report \+RFT results for OPT models\.MethodVariantCalib\.20%25%30%SliceGPT reference \(Alpaca, no fine\-tuning\)SliceGPT–Alpaca53\.6852\.8651\.13LILA \(ours\) – no fine\-tuning \(noRFT\)LILA\-SpectrumAbsNone52\.7651\.5950\.16LILA\-SpectrumSplitNone52\.0451\.7749\.16LILA\-SpectrumActWikiText52\.5251\.1049\.64LILA\-SensitivityAbsNone45\.4743\.6336\.27LILA\-SensitivitySplitNone44\.1941\.5635\.58LILA\-SensitivityActWikiText37\.9035\.3335\.52LILA\-ResidualAbsNone43\.8642\.6735\.83LILA\-ResidualSplitNone45\.8943\.0936\.92LILA\-ResidualActWikiText37\.8736\.4435\.58After WikiText\-2 LoRA recovery fine\-tuning \(\+RFT\)LILA\-SpectrumAbsNone52\.9952\.6051\.64LILA\-SpectrumSplitNone53\.0452\.6751\.81LILA\-SpectrumActWikiText53\.1852\.4352\.19LILA\-SensitivityAbsNone51\.2350\.9849\.23LILA\-SensitivitySplitNone51\.5050\.6749\.52LILA\-SensitivityActWikiText51\.8649\.9844\.85LILA\-ResidualAbsNone51\.6051\.3249\.91LILA\-ResidualSplitNone51\.7050\.6948\.94LILA\-ResidualActWikiText51\.4950\.5448\.05

## Appendix LBenchmarking Throughput Experiment

To validate the practical acceleration provided by LILA, the end\-to\-end auto\-regressive generation throughput of the pruned models is evaluated against their dense baselines\.

#### Experimental Setup\.

Throughput is measured in absolute generated tokens per second \(tok/s\) on a single NVIDIA A100 \(40GB\) GPU\. Experiments utilize PyTorch 2\.x native Scaled Dot\-Product Attention \(SDPA\) for highly optimized baseline generation\. To ensure robust estimates, throughput is evaluated across a geometric progression of input prompt lengths \(L∈\{32,64,128,256,512\}L\\in\\\{32,64,128,256,512\\\}\) at batch size 1, and the average tokens per second across all prompt lengths is reported\.

#### Why Structured Pruning Accelerates Inference\.

Unstructured pruning sets individual weights to zero, which requires specialized sparse tensor cores or custom sparse kernels to realize actual wall\-clock speedups\. In contrast, LILA performs*structured*pruning by identifying and removing entire neurons\. This permanently reduces the intermediate hidden dimension \(dffd\_\{\\mathrm\{ff\}\}\) of the FFN layers\. Consequently, the pruned model functions identically to a standard dense transformer of a smaller size\. This allows the model to fully utilize standard, highly\-optimized dense matrix multiplication routines \(e\.g\., cuBLAS\) out\-of\-the\-box\.

#### Results\.

As shown in[table4](https://arxiv.org/html/2609.11163#S4.T4)\(located in the main text\), LILA yields substantial out\-of\-the\-box hardware acceleration\. At 25% sparsity, the pruned LLaMA\-2\-7B model generates33\.9633\.96tok/s compared to the dense baseline’s25\.6025\.60tok/s, representing a1\.33×1\.33\\timesspeedup\. The acceleration scales favorably with model size, reaching a1\.64×1\.64\\timesspeedup on LLaMA\-2\-13B\.

## Appendix MAdaptive Layer\-wise Sparsity via KS\-Budgets

The standard LILA framework applies a uniform sparsity ratiossacross all FFN layers\. However, the layer\-wise spectral analysis \([appendixI](https://arxiv.org/html/2609.11163#A9)\) reveals that certain layers exhibit systematically higher KS\-distance scores, indicating greater sensitivity to neuron removal\. To investigate whether the KS\-score can dynamically allocate sparsity budgets, an*adaptive*sparsity extension is evaluated\.

#### Methodology\.

Instead of removing a fixed fraction of neuronsssuniformly, a per\-layer survival budget proportional to its KS\-score is assigned\. Sensitive layers \(high KS\-score\) retain more neurons, while robust layers \(low KS\-score\) are pruned more aggressively, such that the overall global sparsity remainsss\. To prevent degenerate collapse in any single layer during aggressive compression, a strict lower bound on survival is enforced: no layer may retain fewer thanmmin=0\.65m\_\{\\min\}=0\.65of its original neurons \(i\.e\., maximum 35% layer\-wise sparsity\)\.

#### Results at Moderate Compression \(25%\)\.

As shown in[table5](https://arxiv.org/html/2609.11163#S4.T5), adaptive KS\-budgeting is highly successful at moderate compression levels\. On LLaMA\-2\-7B at 25% global sparsity, the adaptive approach significantly improves perplexity \(PPL drops from 11\.14 to 9\.50\) while maintaining competitive zero\-shot accuracy\. Similarly, on Phi\-2, PPL improves from 54\.07 to 40\.03\. This empirically validates the core hypothesis: the KS\-distance score accurately identifies structural sensitivity, allowing the model to better preserve its generative distribution by protecting critical layers\.

#### Architectural Limitation at High Compression \(30%\)\.

At 30% global sparsity on LLaMA\-2, the adaptive method experiences a structural collapse \(PPL jumps to 281\.47\)\. Because sensitive layers are protected, the algorithm is forced to push the robust layers to the absolutemmin=0\.65m\_\{\\min\}=0\.65bound \(35% sparsity\) to meet the global 30% target\. This reveals a fundamental architectural limitation of the LLaMA\-2 backbone: its dense representations are brittle to extreme layer\-wise bottlenecks\. Compressing*any single FFN layer*beyond a critical threshold \(around∼33%\{\\sim\}33\\%sparsity\) breaks the forward pass logic entirely\.

#### Conclusion\.

Adaptive KS\-budgeting yields state\-of\-the\-art generative preservation at moderate compression \(25%\) by leveraging spectral predictions\. However, at higher compression regimes \(30%\), uniform sparsity remains the safer default for brittle architectures like LLaMA\-2, as it strictly avoids triggering catastrophic single\-layer bottlenecks\.