PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning

# PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning
Source: [https://arxiv.org/html/2607.07557](https://arxiv.org/html/2607.07557)
Yazdan Jamshidi Palo Alto Networks yjamshidi@paloaltonetworks\.com &Alexey Shvets Palo Alto Networks ashvets@paloaltonetworks\.com

###### Abstract

One\-shot pruning methods like Wanda and SparseGPT apply the same sparsity ratio to every layer of a transformer, ignoring known variation in layer importance\. We propose PALS \(Percentile\-Aware Layerwise Sparsity\), which adjusts per\-layer sparsity based on the 99th percentile of activation magnitudes, bounded to±5%\\pm 5\\%around the target ratio\. On LLaMA\-2\-7B at 50% sparsity, PALS achieves 10\.96 WikiText\-2 perplexity versus 12\.92 for uniform Wanda \(mean over 9 runs,p<0\.001p<0\.001\)\. The benefit is architecture\-dependent: LLaMA\-3\-8B shows marginal gains and Mistral\-7B shows none\. We also find that gradient\-based allocation—the seemingly more principled approach—produces results worse than random, suggesting that gradient magnitude does not predict the impact of discrete weight removal\. PALS adds negligible cost to the pruning pipeline and requires no fine\-tuning\.

## 1Introduction

Large language models routinely reach billions of parameters\. A 7B\-parameter model takes roughly 13 GB in half precision and processes billions of multiply\-accumulate operations per token, which puts real\-time inference out of reach on modest hardware\. Post\-training pruning offers a direct way to cut both memory and compute: remove a fraction of the weights and run inference on the sparser model, without retraining from scratch\.

Recent one\-shot pruning methods have made this practical for LLMs\. WandaSunet al\.\([2024](https://arxiv.org/html/2607.07557#bib.bib8)\)scores each weight by the product of its magnitude and the mean absolute activation flowing through it \(\|wi​j\|⋅mean​\(\|ai\|\)\|w\_\{ij\}\|\\cdot\\text\{mean\}\(\|a\_\{i\}\|\)\), then drops the lowest\-scoring fraction\. SparseGPTFrantar and Alistarh \([2023](https://arxiv.org/html/2607.07557#bib.bib7)\)solves a heavier problem—layer\-wise sparse regression with approximate Hessian information—but in practice lands at similar perplexity on most benchmarks\. Both methods have become standard baselines\.

One thing both share is that they enforce*uniform*sparsity: if the target is 50%, every one of the 32 layers in LLaMA\-2 is pruned to exactly 50%\. This sits poorly with what we know about transformer layers\. Attention heads vary widely in importanceVoitaet al\.\([2019](https://arxiv.org/html/2607.07557#bib.bib9)\); Michelet al\.\([2019](https://arxiv.org/html/2607.07557#bib.bib10)\)\. Early layers tend to build generic representations; middle layers are often more redundant; late layers specialize for the output vocabulary\. Pruning all of them equally seems like it should leave performance on the table—and it does\.

We propose PALS \(Percentile\-Aware Layerwise Sparsity\), which adjusts the sparsity ratio for each layer based on activation statistics gathered from a small calibration set\. The idea is direct: compute the 99th percentile of activation magnitudes for each layer, normalize these scores, and use them to shift each layer’s sparsity target up or down\. Layers with large activation outliers—a sign of important information flow—get pruned less; layers with uniformly small activations get pruned more\. A±5%\\pm 5\\%band around the target prevents any single layer from being starved or overstuffed\. The procedure slots into Wanda’s pipeline with almost no extra cost, since Wanda already collects the activations\.

Before settling on activations, we tried the seemingly more natural approach of using gradient magnitudes to estimate layer importance\. This failed decisively\. Gradient\-based allocation produced 47 perplexity on LLaMA\-2\-7B, compared to 24 for*random*allocation and 11 for our activation\-based scheme\. We believe gradients reflect optimization dynamics—how much the loss would change under an infinitesimal perturbation—rather than the actual impact of removing a large fraction of weights\. But the scale of the failure was not something we anticipated\.

On LLaMA\-2\-7B at 50% sparsity, PALS brings WikiText\-2 perplexity from 12\.92 down to 10\.96, averaged over nine independent runs with different random seeds \(Welch’st​\(14\)=8\.1t\(14\)=8\.1,p<0\.001p<0\.001\)\. The picture is less clear on other architectures\. LLaMA\-3\-8B benefits marginally \(10\.45 vs\. 10\.48\), and Mistral\-7B does not benefit at all \(6\.31 either way\)\. We think this reflects how uniformly these newer models distribute computation across layers—when all layers are similarly important, non\-uniform allocation has nothing to exploit\. We present ablation studies covering the percentile choice, adaptation strength, and bound width, along with downstream evaluation on four benchmarks and practical speedup measurements\.

## 2Related Work

### 2\.1One\-Shot Pruning for LLMs

Magnitude pruningHanet al\.\([2015](https://arxiv.org/html/2607.07557#bib.bib6)\)—removing weights with the smallest absolute values—is the oldest and simplest approach but ignores activation patterns, which limits it on modern LLMs\. SparseGPTFrantar and Alistarh \([2023](https://arxiv.org/html/2607.07557#bib.bib7)\)improved on this substantially by formulating each layer’s pruning as a sparse reconstruction problem, using an approximate inverse Hessian to choose which weights to remove and how to adjust the survivors\. WandaSunet al\.\([2024](https://arxiv.org/html/2607.07557#bib.bib8)\)showed that a much simpler scoring rule—\|w\|⋅mean​\(\|a\|\)\|w\|\\cdot\\text\{mean\}\(\|a\|\), with no weight update—matches SparseGPT in most settings\. Both methods use the same sparsity ratio for every layer\. PALS keeps Wanda’s scoring but varies the fraction of weights removed per layer\.

### 2\.2Structured Pruning

An orthogonal line of work removes entire structural units for hardware\-friendly sparsity\. LLM\-PrunerMaet al\.\([2023](https://arxiv.org/html/2607.07557#bib.bib16)\)uses first\-order Taylor expansion to score and remove attention heads and MLP neurons\. SliceGPTAshkbooset al\.\([2024](https://arxiv.org/html/2607.07557#bib.bib17)\)projects weight matrices into a lower\-rank space and deletes entire rows and columns\. Sheared LLaMAZhanget al\.\([2024](https://arxiv.org/html/2607.07557#bib.bib15)\)combines structured pruning with continued pre\-training\. These approaches deliver better hardware utilization than unstructured pruning but typically require some form of fine\-tuning\. PALS stays within the unstructured, one\-shot setting\.

### 2\.3Non\-Uniform Layer Treatment

The idea that different layers should be treated differently is not new\. NISPYuet al\.\([2018](https://arxiv.org/html/2607.07557#bib.bib12)\)propagates importance scores between layers for vision models, but was designed for networks orders of magnitude smaller than current LLMs\. For large language models, most work on non\-uniform treatment has focused on depth pruning—removing entire transformer blocksKurticet al\.\([2022](https://arxiv.org/html/2607.07557#bib.bib14)\)—rather than adjusting within\-layer sparsity ratios\. To our knowledge, PALS is the first to report clear gains from varying per\-layer sparsity in one\-shot LLM pruning, and the first to document the failure of gradient\-based allocation in this setting\.

### 2\.4Activation Outliers in Transformers

Dettmers et al\.Dettmerset al\.\([2022](https://arxiv.org/html/2607.07557#bib.bib26)\)showed that large language models develop activation outliers—individual features with magnitudes 10–100×\\timeslarger than the rest—concentrated in specific dimensions and layers\. These outliers are essential for model quality: zeroing them out causes sharp accuracy drops\. Our use of the 99th percentile as a layer importance signal connects directly to this observation\. Layers with more extreme outliers are presumably processing more critical information, and PALS assigns them lower sparsity\. This also explains why the 99th percentile outperforms the mean as an importance metric: the mean averages out exactly the outliers that matter\.

## 3Method

Given a pretrained model withLLtransformer layers and a target sparsitystargets\_\{\\text\{target\}\}\(e\.g\., 0\.50 for 50% weight removal\), PALS adjusts the per\-layer sparsity based on activation statistics, then prunes each layer independently using Wanda’s scoring\.

### 3\.1Layer Importance from Activation Percentiles

For each layerll, we collect activationsA\(l\)∈ℝn×dA^\{\(l\)\}\\in\\mathbb\{R\}^\{n\\times d\}onnncalibration tokens acrossddhidden dimensions\. The layer importance score is the 99th percentile of the absolute activations:

I\(l\)=Percentile​\(\|A\(l\)\|,99\)I^\{\(l\)\}=\\text\{Percentile\}\\bigl\(\|A^\{\(l\)\}\|,\\;99\\bigr\)\(1\)The 99th percentile targets the outlier activations that Dettmers et al\.Dettmerset al\.\([2022](https://arxiv.org/html/2607.07557#bib.bib26)\)identified as critical, while being less noisy than the maximum\. To make scores comparable across layers, we standardize:

I^\(l\)=I\(l\)−I¯σI\\hat\{I\}^\{\(l\)\}=\\frac\{I^\{\(l\)\}\-\\bar\{I\}\}\{\\sigma\_\{I\}\}\(2\)whereI¯=1L​∑l=1LI\(l\)\\bar\{I\}=\\frac\{1\}\{L\}\\sum\_\{l=1\}^\{L\}I^\{\(l\)\}andσI\\sigma\_\{I\}is the standard deviation of the raw scores\.

### 3\.2Sparsity Allocation with Conservative Bounds

The normalized importanceI^\(l\)\\hat\{I\}^\{\(l\)\}determines how much each layer deviates from the global target:

s\(l\)=clip​\(starget\+α⋅I^\(l\),smin,smax\)s^\{\(l\)\}=\\text\{clip\}\\\!\\left\(s\_\{\\text\{target\}\}\+\\alpha\\cdot\\hat\{I\}^\{\(l\)\},\\;\\;s\_\{\\text\{min\}\},\\;s\_\{\\text\{max\}\}\\right\)\(3\)whereα\\alphacontrols adaptation strength\. The clip enforces:

smin=starget−0\.05,smax=starget\+0\.05s\_\{\\text\{min\}\}=s\_\{\\text\{target\}\}\-0\.05,\\qquad s\_\{\\text\{max\}\}=s\_\{\\text\{target\}\}\+0\.05\(4\)The±5%\\pm 5\\%bounds are important\. In preliminary experiments, wider bounds \(±20%\\pm 20\\%\) caused sharp degradation because heavily pruned layers created information bottlenecks—the rest of the network could not compensate for the lost capacity\. Conservative bounds trade off some potential gain for robustness\.

### 3\.3Weight Scoring and Pruning

After determinings\(l\)s^\{\(l\)\}, we score and prune weights using Wanda’s criterion:

score​\(wi​j\(l\)\)=\|wi​j\(l\)\|⋅meann​\(\|an​i\(l\)\|\)\\text\{score\}\(w\_\{ij\}^\{\(l\)\}\)=\|w\_\{ij\}^\{\(l\)\}\|\\cdot\\text\{mean\}\_\{n\}\\\!\\left\(\|a\_\{ni\}^\{\(l\)\}\|\\right\)\(5\)For each layerll, the bottoms\(l\)×100%s^\{\(l\)\}\\times 100\\%of weights by score are removed\. No weight updates, no fine\-tuning\.

### 3\.4Complete Procedure

Algorithm[1](https://arxiv.org/html/2607.07557#alg1)summarizes the pipeline\. Steps 1–5 take negligible time compared to Wanda’s scoring in steps 6–9, and the activation collection is shared with Wanda—so the only overhead is the percentile computation and sparsity reallocation\.

Algorithm 1PALS: Activation\-Based Layer\-Adaptive Pruning0:Pretrained model

ffwith

LLlayers, calibration data

𝒟\\mathcal\{D\}, target sparsity

stargets\_\{\\text\{target\}\}, strength

α=0\.05\\alpha\{=\}0\.05
1:Collect activations

A\(l\)A^\{\(l\)\}for each layer

llon

𝒟\\mathcal\{D\}
2:for

l=1l=1to

LLdo

3:

I\(l\)←Percentile​\(\|A\(l\)\|,99\)I^\{\(l\)\}\\leftarrow\\text\{Percentile\}\(\|A^\{\(l\)\}\|,99\)
4:endfor

5:Standardize:

I^\(l\)←\(I\(l\)−I¯\)/σI\\hat\{I\}^\{\(l\)\}\\leftarrow\(I^\{\(l\)\}\-\\bar\{I\}\)/\\sigma\_\{I\}for all

ll
6:for

l=1l=1to

LLdo

7:

s\(l\)←clip​\(starget\+α⋅I^\(l\),smin,smax\)s^\{\(l\)\}\\leftarrow\\text\{clip\}\(s\_\{\\text\{target\}\}\+\\alpha\\cdot\\hat\{I\}^\{\(l\)\},\\;s\_\{\\text\{min\}\},\\;s\_\{\\text\{max\}\}\)
8:Score weights:

scorei​j=\|wi​j\(l\)\|⋅meann​\(\|an​i\(l\)\|\)\\text\{score\}\_\{ij\}=\|w\_\{ij\}^\{\(l\)\}\|\\cdot\\text\{mean\}\_\{n\}\(\|a\_\{ni\}^\{\(l\)\}\|\)
9:Remove the bottom

s\(l\)×100%s^\{\(l\)\}\\times 100\\%of weights by score

10:endfor

11:returnPruned model

## 4Experimental Setup

#### Models\.

We evaluate on three architectures: LLaMA\-2\-7B \(6\.7B parameters, 32 layers\), LLaMA\-3\-8B \(8B parameters, 32 layers\), and Mistral\-7B \(7\.3B parameters, 32 layers\)\. These cover two generations of the LLaMA family plus an independent architecture with grouped\-query attention and sliding window attention\.

#### Evaluation\.

Perplexity is measured on the WikiText\-2Merityet al\.\([2016](https://arxiv.org/html/2607.07557#bib.bib21)\)test set using 2048\-token sequences\. For downstream evaluation, we report zero\-shot accuracy on BoolQClarket al\.\([2019](https://arxiv.org/html/2607.07557#bib.bib24)\), PIQABisket al\.\([2020](https://arxiv.org/html/2607.07557#bib.bib22)\), HellaSwagZellerset al\.\([2019](https://arxiv.org/html/2607.07557#bib.bib23)\), and MMLU, using thelm\-evaluation\-harnessframeworkGaoet al\.\([2021](https://arxiv.org/html/2607.07557#bib.bib27)\)\.

#### Calibration\.

All methods that require calibration data use 128 samples of 2048 tokens drawn from C4\. Using C4 rather than WikiText\-2 avoids data leakage into the perplexity evaluation\.

#### Baselines\.

We compare against: \(1\)Magnitude—remove weights with smallest\|w\|\|w\|, no calibration needed; \(2\)WandaSunet al\.\([2024](https://arxiv.org/html/2607.07557#bib.bib8)\)—activation\-aware scoring with uniform per\-layer sparsity; \(3\)SparseGPTFrantar and Alistarh \([2023](https://arxiv.org/html/2607.07557#bib.bib7)\)—Hessian\-based layer\-wise optimization; and \(4\)PALS\-Gradient—our allocation formula but substituting gradient norms‖∇W\(l\)ℒ‖F\\\|\\nabla\_\{W^\{\(l\)\}\}\\mathcal\{L\}\\\|\_\{F\}for activation percentiles\.

#### Runs and seeds\.

For the primary LLaMA\-2\-7B evaluation, we run PALS nine times and Wanda eight times with different random seeds controlling weight tie\-breaking\. This provides variance estimates and allows a two\-sample test\. Other configurations use single runs\.

#### Implementation\.

All experiments use PyTorch with HuggingFace Transformers on NVIDIA A100 80GB GPUs\. PALS usesα=0\.05\\alpha=0\.05and the 99th percentile, with±5%\\pm 5\\%bounds\. All pruning is one\-shot with no fine\-tuning or weight updates\.

## 5Results

### 5\.1Main Comparison on LLaMA\-2\-7B

Table[1](https://arxiv.org/html/2607.07557#S5.T1)presents the primary result on LLaMA\-2\-7B at 50% unstructured sparsity\.

Table 1:WikiText\-2 perplexity on LLaMA\-2\-7B at 50% sparsity\. Mean±\\pmstd overNNruns\.PALS achieves 10\.96 perplexity, down from Wanda’s 12\.92—a reduction of 1\.96 points\. The standard deviation across nine runs is 0\.59 and Wanda’s across eight runs is 0\.40, so the distributions barely overlap\. A Welch’s t\-test givest​\(14\)=8\.1t\(14\)=8\.1,p<0\.001p<0\.001\.111Degrees of freedom computed via the Welch\-Satterthwaite approximation:df=\(0\.592/9\+0\.402/8\)2/\(0\.594/92⋅8−1\+0\.404/82⋅7−1\)≈14\\text\{df\}=\(0\.59^\{2\}/9\+0\.40^\{2\}/8\)^\{2\}/\(0\.59^\{4\}/9^\{2\}\\cdot 8^\{\-1\}\+0\.40^\{4\}/8^\{2\}\\cdot 7^\{\-1\}\)\\approx 14\.

SparseGPT lands at 13\.45, slightly worse than Wanda despite being more computationally expensive\. Both gradient\-based methods—magnitude pruning \(43\.05\) and PALS\-Gradient \(45\.23\)—fail badly, which we return to in Section[6\.5](https://arxiv.org/html/2607.07557#S6.SS5)\.

Figure[1](https://arxiv.org/html/2607.07557#S5.F1)shows the comparison visually\.

![Refer to caption](https://arxiv.org/html/2607.07557v1/x1.png)Figure 1:WikiText\-2 perplexity at 50% sparsity on LLaMA\-2\-7B\. Error bars:±1\\pm 1std for PALS \(N=9N\{=\}9\)\.
### 5\.2Cross\-Model Results

Table[2](https://arxiv.org/html/2607.07557#S5.T2)extends the evaluation to LLaMA\-3\-8B and Mistral\-7B\.

Table 2:Cross\-model results at 50% sparsity \(WikiText\-2 perplexity\)\.The pattern is clear: PALS helps LLaMA\-2 substantially but does little for the other two\. Mistral\-7B is the most pruning\-friendly model in absolute terms \(6\.31 at 50% sparsity, only 33% above its dense baseline of 4\.74\), but PALS offers no improvement over uniform allocation\. LLaMA\-3\-8B falls between the two, with a negligible 0\.03 perplexity reduction\.

We think this reflects architectural differences\. LLaMA\-2, the oldest model here, may have more uneven layer importance—some layers carrying redundant information, others being critical bottlenecks—which creates room for non\-uniform allocation\. Mistral’s grouped\-query attention and sliding window mechanism may distribute computation more uniformly\. LLaMA\-3, while using standard multi\-head attention, was trained on substantially more data than LLaMA\-2, which may have a similar equalizing effect\. We discuss this further in Section[7\.2](https://arxiv.org/html/2607.07557#S7.SS2)\.

### 5\.3Downstream Task Performance

Perplexity measures language modeling quality, but deployment applications care about task accuracy\. Table[3](https://arxiv.org/html/2607.07557#S5.T3)reports zero\-shot accuracy on four benchmarks\.

Table 3:Zero\-shot accuracy \(%\) at 50% sparsity on LLaMA\-2\-7B\.PALS retains 93\.3% of the dense model’s average accuracy, compared to 91\.7% for Wanda\. The 1\.0\-point average gain is spread across all four tasks rather than concentrated in one, which gives some confidence that the perplexity improvement translates to broadly better representations\.

That said, the downstream gaps are modest\. On MMLU, PALS gets 42\.2 versus Wanda’s 41\.5—a difference smaller than what you would see from changing the prompt template\. Downstream benchmarks at this scale may lack the sensitivity to resolve the kind of gains PALS provides\.

### 5\.4Efficiency

Table[4](https://arxiv.org/html/2607.07557#S5.T4)reports deployment metrics on an A100 GPU with batch size 1 and sequence length 512\.

Table 4:Inference efficiency on LLaMA\-2\-7B at 50% sparsity \(A100, batch 1, seq\. len\. 512\)\.The 50% sparsity halves the stored model size from 13\.2 GB to 6\.6 GB\. Throughput improves by 1\.41×\\timesusing sparse matrix kernels, while peak memory drops modestly \(1\.10×\\times\) because activation memory is unchanged\. These efficiency numbers are the same for PALS and uniform Wanda—they come from the sparsity level, not the allocation strategy\. PALS gets better perplexity at the same sparsity budget\.

### 5\.5Performance Across Sparsity Levels

Figure[2](https://arxiv.org/html/2607.07557#S5.F2)shows WikiText\-2 perplexity across sparsity levels from 30% to 70%\.

![Refer to caption](https://arxiv.org/html/2607.07557v1/x2.png)Figure 2:WikiText\-2 perplexity across sparsity levels on LLaMA\-2\-7B\.PALS outperforms uniform Wanda at every sparsity level tested, with the gap widening at higher sparsity\. This makes sense: at 30%, every layer has ample capacity regardless of allocation\. At 70%, capacity is scarce, and putting it where it counts matters more\.

## 6Analysis

### 6\.1Learned Sparsity Patterns

Figure[3](https://arxiv.org/html/2607.07557#S6.F3)shows the per\-layer sparsity budgets PALS assigns across our three models\.

![Refer to caption](https://arxiv.org/html/2607.07557v1/x3.png)Figure 3:Per\-layer sparsity assigned by PALS at 50% target\. Layer 0 is closest to the input embedding\.All three models show a roughly U\-shaped pattern: early layers \(0–8\) and late layers \(21–31\) get sparsity below 50%, while middle layers \(9–20\) get sparsity above 50%\. The range stays within the±5%\\pm 5\\%bounds by construction, so every layer falls between 45% and 55%\.

This aligns with what is known about transformer layer function\. Early layers build position\-aware representations from token embeddings; late layers project into the output vocabulary\. Both are hard to replace with fewer weights\. Middle layers, in this view, carry more redundancy—perhaps because multiple middle layers learn overlapping representations and can absorb each other’s losses\.

One caveat: the pattern similarity across models is partly mechanical\. Because PALS normalizes importance scores and clips to±5%\\pm 5\\%, the output will look roughly U\-shaped for any model whose early and late layers have larger activation outliers than the middle\. Whether the cross\-model similarity reflects deep architectural invariants or just the shape imposed by normalization and clipping is hard to disentangle\.

### 6\.2Effect of Adaptation Strengthα\\alpha

Table[5](https://arxiv.org/html/2607.07557#S6.T5)shows howα\\alphaaffects perplexity on LLaMA\-2\-7B\. These are single runs \(exceptα=0\.05\\alpha=0\.05, which is the 9\-run mean\), so differences within the run\-to\-run noise band \(∼\\sim0\.6 PPL\) should not be over\-interpreted\.

Table 5:Ablation onα\\alpha\(LLaMA\-2\-7B, 50% sparsity, single runs exceptα=0\.05\\alpha\{=\}0\.05\)\.There is an interesting detail here\. Forα≥0\.07\\alpha\\geq 0\.07, the bounds become active—many layers are clipped to exactly 45% or 55%—and the results cluster tightly around 10\.1–10\.2\. These single\-run numbers are slightly better than our chosenα=0\.05\\alpha=0\.05\(10\.96\), but the difference of∼\\sim0\.8 PPL is within the run\-to\-run variance \(std = 0\.59 over the 9\-run evaluation atα=0\.05\\alpha=0\.05\)\. A single lucky seed atα=0\.07\\alpha=0\.07could easily account for the gap\.

We useα=0\.05\\alpha=0\.05because it represents the transition point: below it, the allocation is too timid; above it, the bounds do most of the work and the specific value ofα\\alphastops mattering\. The data is consistent with the interpretation that a coarser binary allocation \(every layer at either 45% or 55%\) works roughly as well as the smooth allocation PALS provides, though confirming this would require multi\-run evaluation at eachα\\alphavalue\.

Theα=0\.01\\alpha=0\.01entry \(14\.23 PPL\) is worse than uniform Wanda \(12\.92\)—a very small amount of adaptation apparently hurting\. We suspect this is seed noise in a single run, not a real effect\. Without multi\-run data at eachα\\alpha, we cannot say whether this non\-monotonicity is genuine\.

### 6\.3Effect of Bound Width

Table 6:Ablation on sparsity bounds \(LLaMA\-2\-7B,α=0\.05\\alpha\{=\}0\.05, single runs\)\.Table[6](https://arxiv.org/html/2607.07557#S6.T6)is the cleanest result in our ablations\. Performance degrades sharply once the bounds exceed±10%\\pm 10\\%, and unbounded allocation \(12–88% range\) is catastrophic\. At±20%\\pm 20\\%, the most aggressively pruned layers hit 70% sparsity, and the information loss there apparently cannot be compensated by gently pruned layers elsewhere\. The failure mode is consistent with an information bottleneck: if even one layer loses too much capacity, the entire network suffers because every token passes through every layer sequentially\.

The±5%\\pm 5\\%setting sits in a good spot\. Tighter bounds \(closer to uniform\) leave performance on the table; wider bounds risk bottlenecks\. Whether±5%\\pm 5\\%is universally best or happens to suit these three models is an open question\.

### 6\.4Choice of Activation Percentile

Table 7:Ablation on percentile choice \(LLaMA\-2\-7B, single runs\)\.The mean is a poor importance signal \(13\.78 PPL, worse than uniform Wanda\), confirming that outlier activations, not average activation level, drive layer importance for pruning\. Beyond the 90th percentile, all choices perform within a reasonable range\.

A careful reader will notice that the 99th percentile \(10\.96\) is not the best in this table: both the 95th \(10\.67\) and 99\.5th \(10\.11\) give lower perplexity\. Since these are single runs and the run\-to\-run variance atα=0\.05\\alpha=0\.05is 0\.59, the differences are not statistically distinguishable\. We chose the 99th percentile before running this ablation, based on its connection to the outlier phenomenonDettmerset al\.\([2022](https://arxiv.org/html/2607.07557#bib.bib26)\), and kept it rather than cherry\-picking after the fact\. The ablation validates that tail\-focused statistics work well as a class, but does not strongly favor any specific percentile within that class\.

### 6\.5Gradient\-Based Allocation Fails

Table 8:Layer importance metrics for sparsity allocation \(LLaMA\-2\-7B, 50% sparsity\)\.Table[8](https://arxiv.org/html/2607.07557#S6.T8)is, in our view, the most striking result in this paper\. Gradient\-based importance metrics—which have a long history of success in pruning smaller networksLeCunet al\.\([1989](https://arxiv.org/html/2607.07557#bib.bib5)\)—produce allocations that are worse than*random*\. Gradient norm gives 47\.32 PPL; first\-order Taylor expansion \(∑\|gi​j​wi​j\|\\sum\|g\_\{ij\}w\_\{ij\}\|\) does better at 38\.61 but is still far worse than random at 24\.37\.

To be clear about what “random allocation” means here: each layer gets a sparsity ratio drawn uniformly from\[starget−0\.05,starget\+0\.05\]\[s\_\{\\text\{target\}\}\-0\.05,\\;s\_\{\\text\{target\}\}\+0\.05\], with no information about layer importance\. It beats both gradient methods by a wide margin\.

Why do gradients fail? Two possibilities, both speculative:

Convergence confound\.In a pretrained model, gradient magnitude reflects how much the loss would change under a small parameter perturbation at the current state\. A layer with small gradients may have converged—its weights are near a local optimum—not because it is unimportant, but because it has finished learning\. Gradient\-based allocation would over\-prune such layers\.

Discrete–continuous mismatch\.Pruning removes a large, discrete fraction of weights\. Gradients measure infinitesimal, continuous sensitivity\. At 50% sparsity, the “perturbation” is far from infinitesimal\. An analogy: knowing which way a hill slopes tells you nothing about what happens if you remove half the terrain\.

Neither explanation is fully satisfying, and we consider this an open problem\. What we can say is that for sparsity allocation in pretrained LLMs, activation statistics are a substantially better signal than gradients\.

### 6\.6Calibration Stability

PALS relies on activation statistics from 128 calibration samples\. A natural concern is whether the importance scores are stable across different calibration draws\.

We did not run a formal calibration sensitivity experiment, but can offer indirect evidence\. The 99th percentile is computed over roughly128×2048≈262128\\times 2048\\approx 262K tokens per layer\. Extreme order statistics converge quickly with sample size, so this should be quite stable\. Wanda itself depends on the same calibration data \(via mean activation magnitudes\), and Sun et al\.Sunet al\.\([2024](https://arxiv.org/html/2607.07557#bib.bib8)\)reported that Wanda’s results are stable across calibration subsets\. Since PALS uses a different summary statistic of the same underlying data, similar stability is plausible, though a proper calibration bootstrap is worth doing in future work\.

Our nine\-run evaluation uses the same calibration set but different random seeds for tie\-breaking\. The resulting std of 0\.59 bounds the contribution of tie\-breaking noise\. Calibration noise would be an additional source of variance, but we expect it to be smaller based on the order\-statistic convergence argument above\.

## 7Discussion

### 7\.1Why Activations Work

The success of activation\-based importance connects to the growing literature on activation outliersDettmerset al\.\([2022](https://arxiv.org/html/2607.07557#bib.bib26)\)\. Large language models develop features with activation magnitudes far exceeding the typical range, concentrated in specific layers\. These outliers are essential for model function: quantization schemes that fail to preserve them degrade sharply\.

PALS identifies layers with large outliers \(via the 99th percentile\) and assigns them lower sparsity\. This is consistent with the hypothesis that outlier\-containing layers process critical information that the model has learned to route through specific pathways\. The connection also predicts that any importance metric sensitive to the tail of the activation distribution should work—and indeed, the 90th through max percentile all outperform the mean \(Table[7](https://arxiv.org/html/2607.07557#S6.T7)\)\.

### 7\.2Architecture Dependence

The large improvement on LLaMA\-2 \(15%\) versus zero on Mistral is worth understanding\. Three hypotheses:

Training data volume\.LLaMA\-2\-7B was trained on 2T tokens; LLaMA\-3\-8B on roughly 15T tokens\. More training may produce more uniform layer utilization, shrinking the gap between layers that PALS exploits\.

Architecture\.Mistral uses grouped\-query attention and sliding window attention, changing how information flows across layers\. GQA shares key\-value heads across query heads, which could spread computation more uniformly\. Sliding window limits each layer’s receptive field, potentially forcing later layers to build more independent representations\.

Model maturity\.Newer models may simply be better trained, with fewer “wasted” layers\. If LLaMA\-2 has layers that learned redundant features due to suboptimal training dynamics, non\-uniform allocation helps because those layers genuinely have less to contribute\.

We cannot distinguish between these with our current experiments\. The practical upshot: if you run PALS and the sparsity pattern looks nearly flat \(all layers close to 50%\), the model probably will not benefit, and uniform allocation is fine\.

### 7\.3Limitations

Unstructured sparsity\.The throughput gains in Table[4](https://arxiv.org/html/2607.07557#S5.T4)require sparse matrix kernels or hardware support \(e\.g\., NVIDIA’s 2:4 structured sparsity\)\. Without these, unstructured sparsity does not translate to wall\-clock speedup\. Extending PALS to N:M or block sparsity patterns is a natural next step\.

Single perplexity dataset\.Our primary metric is WikiText\-2 perplexity\. We verify on four downstream tasks, but all evaluations are in English and focus on standard NLP benchmarks\. Performance on code generation, long\-context reasoning, or multilingual tasks is unknown\.

No fine\-tuning\.We evaluate one\-shot pruning only\. Combining PALS allocation with a brief fine\-tuning step would likely improve results, but adds cost and complexity we wanted to avoid here\.

Model scale\.All experiments are on 7–8B parameter models\. Whether the gains hold at 13B, 70B, or larger is untested\. Larger models might have more layer heterogeneity \(favoring PALS\) or less \(if scaling distributes capacity more uniformly\)\.

## 8Conclusion

PALS adjusts per\-layer sparsity in one\-shot LLM pruning using activation percentiles, reducing LLaMA\-2\-7B perplexity from 12\.92 to 10\.96 at 50% sparsity with negligible extra cost\. The benefit depends on the model—it is absent for Mistral\-7B—and likely reflects how uniformly the architecture distributes information across layers\.

The gradient failure finding is, to us, the more interesting result\. Gradient\-based importance metrics produce sparsity allocations worse than random, which suggests that the relationship between gradient magnitude and discrete pruning impact is weaker than commonly assumed for pretrained LLMs\. Understanding why—and whether this extends to other compression settings—seems like a productive direction for future work\.

## Acknowledgments

We thank the authors of Wanda and SparseGPT for open\-sourcing their implementations\. Experiments were conducted on NVIDIA A100 GPUs\.

## References

- SliceGPT: compress large language models by deleting rows and columns\.arXiv preprint arXiv:2401\.15024\.Cited by:[§2\.2](https://arxiv.org/html/2607.07557#S2.SS2.p1.1)\.
- Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)PIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4](https://arxiv.org/html/2607.07557#S4.SS0.SSS0.Px2.p1.1)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.arXiv preprint arXiv:1905\.10044\.Cited by:[§4](https://arxiv.org/html/2607.07557#S4.SS0.SSS0.Px2.p1.1)\.
- T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer \(2022\)GPT3\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.Advances in Neural Information Processing Systems35,pp\. 30318–30332\.Cited by:[§2\.4](https://arxiv.org/html/2607.07557#S2.SS4.p1.1),[§3\.1](https://arxiv.org/html/2607.07557#S3.SS1.p1.7),[§6\.4](https://arxiv.org/html/2607.07557#S6.SS4.p2.1),[§7\.1](https://arxiv.org/html/2607.07557#S7.SS1.p1.1)\.
- E\. Frantar and D\. Alistarh \(2023\)SparseGPT: massive language models can be accurately pruned in one\-shot\.arXiv preprint arXiv:2301\.00774\.Cited by:[§1](https://arxiv.org/html/2607.07557#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.07557#S2.SS1.p1.1),[§4](https://arxiv.org/html/2607.07557#S4.SS0.SSS0.Px4.p1.2)\.
- L\. Gao, J\. Tow, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, K\. McDonell, N\. Muennighoff,et al\.\(2021\)A framework for few\-shot language model evaluation\.Cited by:[§4](https://arxiv.org/html/2607.07557#S4.SS0.SSS0.Px2.p1.1)\.
- S\. Han, J\. Pool, J\. Tran, and W\. Dally \(2015\)Learning both weights and connections for efficient neural network\.InAdvances in neural information processing systems,pp\. 1135–1143\.Cited by:[§2\.1](https://arxiv.org/html/2607.07557#S2.SS1.p1.1)\.
- E\. Kurtic, D\. Campos, T\. Nguyen, E\. Frantar, M\. Kurtz, B\. Fineran, M\. Goin, and D\. Alistarh \(2022\)Optimal brain compression: a framework for accurate post\-training quantization and pruning\.Advances in Neural Information Processing Systems35,pp\. 4475–4488\.Cited by:[§2\.3](https://arxiv.org/html/2607.07557#S2.SS3.p1.1)\.
- Y\. LeCun, J\. Denker, and S\. Solla \(1989\)Optimal brain damage\.InAdvances in neural information processing systems,pp\. 598–605\.Cited by:[§6\.5](https://arxiv.org/html/2607.07557#S6.SS5.p1.1)\.
- X\. Ma, G\. Fang, and X\. Wang \(2023\)LLM\-pruner: on the structural pruning of large language models\.Advances in Neural Information Processing Systems36\.Cited by:[§2\.2](https://arxiv.org/html/2607.07557#S2.SS2.p1.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.arXiv preprint arXiv:1609\.07843\.Cited by:[§4](https://arxiv.org/html/2607.07557#S4.SS0.SSS0.Px2.p1.1)\.
- P\. Michel, O\. Levy, and G\. Neubig \(2019\)Are sixteen heads really better than one?\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2607.07557#S1.p3.1)\.
- M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. Kolter \(2024\)A simple and effective pruning approach for large language models\.arXiv preprint arXiv:2306\.11695\.Cited by:[§1](https://arxiv.org/html/2607.07557#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.07557#S2.SS1.p1.1),[§4](https://arxiv.org/html/2607.07557#S4.SS0.SSS0.Px4.p1.2),[§6\.6](https://arxiv.org/html/2607.07557#S6.SS6.p2.1)\.
- E\. Voita, D\. Talbot, F\. Moiseev, R\. Sennrich, and I\. Titov \(2019\)Analyzing multi\-head self\-attention: specialized heads do the heavy lifting, the rest can be pruned\.arXiv preprint arXiv:1905\.09418\.Cited by:[§1](https://arxiv.org/html/2607.07557#S1.p3.1)\.
- R\. Yu, A\. Li, C\. Chen, J\. Lai, V\. I\. Morariu, X\. Han, M\. Gao, C\. Lin, and L\. S\. Davis \(2018\)NISP: pruning networks using neuron importance score propagation\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 9194–9203\.Cited by:[§2\.3](https://arxiv.org/html/2607.07557#S2.SS3.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.Cited by:[§4](https://arxiv.org/html/2607.07557#S4.SS0.SSS0.Px2.p1.1)\.
- M\. Zhang, T\. Gao, A\. Chen, J\. Herzig, and L\. Zettlemoyer \(2024\)Sheared llama: accelerating language model pre\-training via structured pruning\.arXiv preprint arXiv:2310\.06694\.Cited by:[§2\.2](https://arxiv.org/html/2607.07557#S2.SS2.p1.1)\.