Magnitude Profile Pruning：无需校准的Transformer压缩结构化注意力头剪枝

# Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression
Source: [https://arxiv.org/html/2609.26177](https://arxiv.org/html/2609.26177)
Kasun Dewage1, Marianna Pensky1, Heranga K\. Rathnasekara2, and Suranadi De Silva1

###### Abstract

Structured pruning of attention heads provides a hardware\-friendly way to compress Transformer language models\. However, existing methods for measuring head\-level importance require calibration data, gradient computation, or Hessian estimation\. These requirements add extra overhead and make the methods depend on the data\. Our work presentsMagnitude Profile \(MP\)scoring, a training\-free criterion for head importance that identifies dispensable heads through statistical outlier detection on weight row norms\. Heads whose projection weights fall within the population bulk are pruned, while heads exhibiting outlier norms, which carry disproportionate representational capacity, are preserved\. Our work further givesMP\-G, a variant that handles Grouped Query Attention \(GQA\) by distributing shared key\-value group scores across associated query heads\. Across five models evaluated on WikiText\-2 perplexity at 12\.5%–50% head sparsity, MP\-G achieves the best perplexity on OPT\-6\.7B at all sparsity levels \(18\.46 at 12\.5%, 27\.87 at 25%, 152\.0 at 50%\)\.MP\-Galso gives the best results on RoBERTa\-large at 12\.5% and 25% sparsity, with perplexity values of 7\.27 and 10\.28, outperforming calibration\-dependent baselines including Wanda\-Head, SparseGPT\-Head, and Gradient\-Head\. It requireszeroforward passes, calibration samples, or gradient computation\. At 50% sparsity, head pruning yields up to 16% parameter reduction with 50% attention FLOP savings\. Our results show that weight\-only statistical scoring can match or outperform data\-dependent methods for structured head pruning, providing a practical, zero\-cost criterion for Transformer compression\.

\(1\) Attention weights \(layerℓ\\ell\)WQW\_\{Q\}WKW\_\{K\}WVW\_\{V\}WOW\_\{O\}row normsri=‖WR,i,:‖2r\_\{i\}=\\\|W\_\{R,i,:\}\\\|\_\{2\},R∈\{Q,K,V\}R\\in\\\{Q,K,V\\\}column norms forWOW\_\{O\}*computed from weights only — no data*\(2\) Statistical outlier detectionrow indexiirir\_\{i\}μR\+z​σR\\mu\_\{R\}\+z\\sigma\_\{R\}excessexc​\(ri\)=max⁡\(0,ri−\(μR\+z​σR\)\)\\mathrm\{exc\}\(r\_\{i\}\)=\\max\\bigl\(0,\\;r\_\{i\}\-\(\\mu\_\{R\}\+z\\sigma\_\{R\}\)\\bigr\),z=2\.0z=2\.0\(3\) GQA\-aware head score \(MP\-G\)KV groupk=⌊h/g⌋k=\\lfloor h/g\\rfloorh1h\_\{1\}h2h\_\{2\}h3h\_\{3\}h4h\_\{4\}excK​V​\(k\)/g\\mathrm\{exc\}\_\{KV\}\(k\)/gsharedwith each ofggheads\+excQ​\(h\)\+excO​\(h\)\+\\;\\mathrm\{exc\}\_\{Q\}\(h\)\+\\mathrm\{exc\}\_\{O\}\(h\)per query headϕ​\(h\)=αq​\(excQ​\(h\)\+excO​\(h\)\)\+αk​v​excK​V​\(⌊h/g⌋\)/g\\phi\(h\)=\\alpha\_\{q\}\\bigl\(\\mathrm\{exc\}\_\{Q\}\(h\)\+\\mathrm\{exc\}\_\{O\}\(h\)\\bigr\)\+\\alpha\_\{kv\}\\,\\mathrm\{exc\}\_\{KV\}\(\\lfloor h/g\\rfloor\)/gg=H/Hk​vg=H/H\_\{kv\}; MHA \(Hk​v=HH\_\{kv\}\{=\}H\)⇒\\Rightarrowreduces to MP\(4\) Global ranking and pruningϕ\\phiprune⌊s⋅L⋅H⌋\\lfloor s\\cdot L\\cdot H\\rfloorlowestkeep outlier headsallL⋅HL\\cdot Hheads ranked globally by ascendingϕ\\phi;zeroWQ,WOW\_\{Q\},W\_\{O\}slices of pruned heads; zeroWK,WVW\_\{K\},W\_\{V\}once all query heads of a KV group are prunedCalibration\-free:no calibration data⋅\\cdotno forward or backward passes⋅\\cdotno Hessian estimation⋅\\cdotO​\(p\)O\(p\)weight\-only cost⋅\\cdotno fine\-tuning or weight reconstruction

Figure 1:Overview of Magnitude Profile Pruning \(MP\-G\)\. \(1\) For each layer, L2 row norms are computed for theWQ,WK,WVW\_\{Q\},W\_\{K\},W\_\{V\}projections and column norms forWOW\_\{O\}, using model weights only\. \(2\) Per projection, the excess of each norm beyond the statistical thresholdμR\+z​σR\\mu\_\{R\}\+z\\sigma\_\{R\}\(Eq\.[1](https://arxiv.org/html/2609.26177#S3.E1),z=2\.0z\{=\}2\.0by default\) identifies outlier rows; norms within the population bulk receive zero excess\. \(3\) Head scores aggregate query\-side excesses \(excQ\+excO\\mathrm\{exc\}\_\{Q\}\+\\mathrm\{exc\}\_\{O\}\) and, for GQA models, distribute the shared KV\-group excess equally across theg=H/Hk​vg=H/H\_\{kv\}query heads in the group \(Eq\.[3](https://arxiv.org/html/2609.26177#S3.E3)\); for MHA models MP\-G reduces to standard MP\. \(4\) Heads are ranked globally by ascending score and the⌊s⋅L⋅H⌋\\lfloor s\\cdot L\\cdot H\\rfloorlowest\-scoring heads are pruned by zeroing theirWQW\_\{Q\}andWOW\_\{O\}slices, withWK,WVW\_\{K\},W\_\{V\}zeroed once all query heads in a KV group are pruned \(Algorithm[1](https://arxiv.org/html/2609.26177#alg1)\)\. The entire procedure requires no calibration samples, forward passes, gradients, or Hessian estimation\.## IIntroduction

Large Transformer\-based language models\[[1](https://arxiv.org/html/2609.26177#bib.bib1)\]deliver state\-of\-the\-art performance across natural language tasks, but their computational and memory footprints challenge deployment in resource\-constrained settings\. Pruning is a well\-established strategy for reducing model size\[[2](https://arxiv.org/html/2609.26177#bib.bib2)\], yet most work has targetedunstructuredweight pruning\[[3](https://arxiv.org/html/2609.26177#bib.bib3),[4](https://arxiv.org/html/2609.26177#bib.bib4)\]\. This approach achieves fine\-grained sparsity but requires specialized hardware or software to convert sparsity into wall\-clock speedup\.

Structuredpruning at the attention head level is more hardware\-friendly: entire head weight slices can be zeroed, directly reducing the projection operations and attention computations\. However, existing head\-level importance criteria carry non\-trivial overhead\. Learned\-gate approaches\[[5](https://arxiv.org/html/2609.26177#bib.bib5)\]require training withL0L\_\{0\}regularization\. Gradient\-based criteria\[[6](https://arxiv.org/html/2609.26177#bib.bib6)\]and Taylor\-based methods\[[7](https://arxiv.org/html/2609.26177#bib.bib7)\]require backward passes over calibration data\. Even our adaptations of Wanda\[[4](https://arxiv.org/html/2609.26177#bib.bib4)\]and SparseGPT\[[3](https://arxiv.org/html/2609.26177#bib.bib3)\]to head granularity require forward passes and activation/Hessian collection over calibration samples\.

We argue that the weight matrices themselves contain sufficient signal to identify dispensable heads, without any data\-dependent computation\. Our key observation is that, in trained Transformers, a small subset of attention heads develops projection weights with significantly larger norms than the population average\. Theseoutlier headscarry more information, while heads with typical, near\-average norms are mostly redundant and can be safely removed\.

Building on this insight, we proposeMagnitude Profile \(MP\)scoring, which quantifies head importance via the excess of weight row norms beyond a statistical threshold\(μ\+z​σ\)\(\\mu\+z\\sigma\)\. Heads with no excess, i\.e\. those whose weights are statistically typical, receive a score of zero and are pruned first\. We further introduceMP\-G, a variant designed for modern Grouped Query Attention \(GQA\) architectures by computing shared key\-value group scores and distributing them proportionally across associated query heads\.

Our contributions are: 1\. We introduceMagnitude Profile \(MP\)scoring, a calibration\-free, gradient\-free head importance criterion based on statistical outlier detection in weight norms\. The method requires only model weight access, withO​\(p\)O\(p\)computation whereppis the number of attention parameters\. 2\. We proposeMP\-G, a GQA\-aware extension that accounts for key\-value head sharing in modern architectures such as LLaMA\-3 and Mistral\. 3\. We demonstrate through experiments onfive modelsacross encoder and decoder architectures that MP\-G achieves the best perplexity in several settings and top\-two performance in most primary model\-sparsity settings at 12\.5%–50% head sparsity, while requiring zero calibration overhead\. 4\. We provide efficiency analysis showing 50% head pruning yields approximately 7–16% total parameter reduction with 50% attention FLOP savings\.

## IIRelated Work

Attention Head Pruning\.Michel et al\.\[[6](https://arxiv.org/html/2609.26177#bib.bib6)\]showed that many attention heads can be removed with minimal accuracy loss and proposed gradient\-based importance scoring via Taylor expansion\. Voita et al\.\[[5](https://arxiv.org/html/2609.26177#bib.bib5)\]employed differentiable gates withL0L\_\{0\}regularization to identify and prune redundant heads during training\. Both methods require backward passes over task\-specific data, limiting their applicability to settings where calibration data and training infrastructure are available\.

Unstructured Pruning Criteria\.Magnitude pruning, which removes weights with smallest absolute values, remains a strong baseline\[[8](https://arxiv.org/html/2609.26177#bib.bib8)\]\. SparseGPT\[[3](https://arxiv.org/html/2609.26177#bib.bib3)\]uses approximate Hessian information to prune and reconstruct weights row\-by\-row with a single forward pass\. Wanda\[[4](https://arxiv.org/html/2609.26177#bib.bib4)\]combines weight magnitude with input activation norms, achieving competitive unstructured sparsity without weight updates\. Both were designed for element\-level sparsity, and their adaptation to structured head\-level pruning has not been systematically studied\.

Structured Pruning of LLMs\.LLM\-Pruner\[[9](https://arxiv.org/html/2609.26177#bib.bib9)\]uses gradient information to identify coupled structures for removal\. Sheared LLaMA\[[10](https://arxiv.org/html/2609.26177#bib.bib10)\]learns pruning masks jointly with continued pre\-training\. SliceGPT\[[11](https://arxiv.org/html/2609.26177#bib.bib11)\]removes entire rows and columns via PCA\-based projections\. CoFi\[[12](https://arxiv.org/html/2609.26177#bib.bib12)\]performs coarse\-grained and fine\-grained pruning jointly, targeting both attention heads and feed\-forward layers with learned masks\. These approaches target broader structural units \(layers, hidden dimensions\) and typically require substantial computation\. Our work focuses on head\-level pruning using a lightweight, training\-free criterion\.

## IIIMethod

### III\-AProblem Formulation

Consider a Transformer\[[1](https://arxiv.org/html/2609.26177#bib.bib1)\]withLLlayers, each containingHHattention heads of dimensiondh=dmodel/Hd\_\{h\}=d\_\{\\text\{model\}\}/H\. Each headhhin layerℓ\\ellis parameterized by projection matricesWQ\(ℓ,h\),WK\(ℓ,h\),WV\(ℓ,h\)∈ℝdh×dmodelW\_\{Q\}^\{\(\\ell,h\)\},W\_\{K\}^\{\(\\ell,h\)\},W\_\{V\}^\{\(\\ell,h\)\}\\in\\mathbb\{R\}^\{d\_\{h\}\\times d\_\{\\text\{model\}\}\}and output projection columnsWO,h⋅dh:\(h\+1\)⋅dh\(ℓ\)∈ℝdmodel×dhW\_\{O,\\,h\\cdot d\_\{h\}:\(h\+1\)\\cdot d\_\{h\}\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\\times d\_\{h\}\}\.

Given a target head sparsity ratios∈\(0,1\)s\\in\(0,1\), the goal is to identify and zero the Q, K, V, and O weight slices of⌊s⋅L⋅H⌋\\lfloor s\\cdot L\\cdot H\\rfloorheads globally\. Physical removal of pruned head dimensions \(model surgery\) is left for deployment; our experiments zero head slices in place\. This process requires a scoring functionϕ​\(ℓ,h\)∈ℝ≥0\\phi\(\\ell,h\)\\in\\mathbb\{R\}\_\{\\geq 0\}that assigns importance to each head; heads with the lowest scores are pruned\.

### III\-BMagnitude Profile Scoring

The central idea behind MP scoring is that trained Transformers exhibit heterogeneous weight norm distributions across heads\. Rather than treating raw weight magnitude as the importance signal, which conflates overall weight scale with head\-specific significance, MP identifies heads that arestatistically indistinguishablefrom the population as redundant, and preserves only those withoutlierweight patterns\.

For each projection matrixR∈\{Q,K,V\}R\\in\\\{Q,K,V\\\}of a given layer, we compute the L2 row normsri=‖WR,i,:‖2r\_\{i\}=\\\|W\_\{R,i,:\}\\\|\_\{2\}for all rowsii\. We then define the excess beyond a threshold controlled by a parameterzz:

exc​\(ri\)=max⁡\(0,ri−\(μr\+z⋅σr\)\)\\text\{exc\}\(r\_\{i\}\)=\\max\\\!\\big\(0,\\;r\_\{i\}\-\(\\mu\_\{r\}\+z\\cdot\\sigma\_\{r\}\)\\big\)\(1\)whereμr\\mu\_\{r\}andσr\\sigma\_\{r\}are the mean and standard deviation of all row norms in that projection matrix\. For the output projectionWOW\_\{O\}, column norms are used analogously, sinceWOW\_\{O\}is transposed relative to Q/K/V in terms of head indexing\.

The head\-level importance score aggregates the excesses across all four projection roles:

ϕMP​\(ℓ,h\)=∑R∈\{Q,K,V,O\}∑i∈rows​\(h,R\)exc​\(riR\)\\phi\_\{\\text\{MP\}\}\(\\ell,h\)=\\sum\_\{R\\in\\\{Q,K,V,O\\\}\}\\sum\_\{i\\in\\text\{rows\}\(h,R\)\}\\text\{exc\}\(r\_\{i\}^\{R\}\)\(2\)
The thresholdzzcontrols the sensitivity of outlier detection\. A highzz\(e\.g\., 2\.5\) preserves only extreme outliers and aggressively prunes “normal” heads\. A lowerzz\(e\.g\., 1\.5\) is more conservative and retains heads with moderately elevated norms\. We usez=2\.0z=2\.0as the default, following the common two\-standard\-deviation rule\.

Intuition\.If all heads in a layer have similar weight norms, they contribute roughly equally and some can be removed with limited damage\. But if a few heads have much larger norms, those heads likely carry specialized functions \(e\.g\., positional attention, syntactic parsing\) that would be destructive to remove\[[5](https://arxiv.org/html/2609.26177#bib.bib5),[13](https://arxiv.org/html/2609.26177#bib.bib13)\]\. MP formalizes this idea by assigning zero importance to heads within the bulk distribution and positive importance only to outliers\.

Computational cost\.MP requires a single pass over all attention weight matrices to compute norms and statistics\. For a model withppattention parameters, the computational cost isO​\(p\)O\(p\)with no forward passes, no backward passes, no calibration data, and no Hessian estimation\.

### III\-CMP\-G: GQA\-Aware Extension

Modern decoder models such as LLaMA\-3\[[21](https://arxiv.org/html/2609.26177#bib.bib21)\]and Mistral\[[22](https://arxiv.org/html/2609.26177#bib.bib22)\]use Grouped Query Attention \(GQA\)\[[16](https://arxiv.org/html/2609.26177#bib.bib16)\], whereHk​v<HH\_\{kv\}<Hkey\-value heads are shared across groups ofg=H/Hk​vg=H/H\_\{kv\}query heads\. Standard MP applies the same K/V excess to all query heads in a group, failing to account for the structural coupling\.

MP\-G addresses this by computing a per\-KV\-group score and distributing it equally across associated query heads:

ϕMP\-G​\(h\)=αq⋅\(excQ​\(h\)\+excO​\(h\)\)\+αk​v⋅excK​V​\(⌊h/g⌋\)g\\phi\_\{\\text\{MP\-G\}\}\(h\)=\\alpha\_\{q\}\\cdot\\big\(\\text\{exc\}\_\{Q\}\(h\)\+\\text\{exc\}\_\{O\}\(h\)\\big\)\+\\alpha\_\{kv\}\\cdot\\frac\{\\text\{exc\}\_\{KV\}\(\\lfloor h/g\\rfloor\)\}\{g\}\(3\)whereexcK​V​\(k\)=∑i∈rows​\(k\)\(exc​\(riK\)\+exc​\(riV\)\)\\text\{exc\}\_\{KV\}\(k\)=\\sum\_\{i\\in\\text\{rows\}\(k\)\}\\big\(\\text\{exc\}\(r\_\{i\}^\{K\}\)\+\\text\{exc\}\(r\_\{i\}^\{V\}\)\\big\)is the total excess for KV groupkk, andαq,αk​v\\alpha\_\{q\},\\alpha\_\{kv\}control the relative weighting of query\-side vs\. KV\-side contributions\. We useαq=αk​v=1\.0\\alpha\_\{q\}=\\alpha\_\{kv\}=1\.0by default\.

The division byggensures that the shared KV signal is not over\-counted across the group\. When all query heads in a KV group are pruned, the corresponding K/V weights are also zeroed, since no query head references them\.

For Multi\-Head Attention \(MHA\) models whereHk​v=HH\_\{kv\}=H, MP\-G reduces to standard MP\.

### III\-DPruning Procedure

Algorithm[1](https://arxiv.org/html/2609.26177#alg1)summarizes the complete pruning procedure\. Given a model and target sparsityss, we \(1\) compute MP\-G scores for all heads across all layers, \(2\) globally rank heads by ascending score, and \(3\) zero the weight slices of the lowest\-scoring heads until the target sparsity is reached\. No fine\-tuning or weight reconstruction is performed\.

Algorithm 1MP\-G Head Pruning0:Model

MMwith

LLlayers,

HHheads/layer,

Hk​vH\_\{kv\}KV heads/layer, target sparsity

ss, threshold

zz
1:

g←H/Hk​vg\\leftarrow H/H\_\{kv\}\{GQA group size\}

2:

𝑠𝑐𝑜𝑟𝑒𝑠←\[\]\\mathit\{scores\}\\leftarrow\[\]
3:foreach layer

ℓ=1,…,L\\ell=1,\\ldots,Ldo

4:foreach projection

R∈\{Q,K,V,O\}R\\in\\\{Q,K,V,O\\\}do

5:Compute row norms for

Q,K,VQ,K,Vand column norms for

OO; compute

μR,σR\\mu^\{R\},\\sigma^\{R\}
6:

exciR←max⁡\(0,riR−μR−z⋅σR\)​∀i\\text\{exc\}^\{R\}\_\{i\}\\leftarrow\\max\(0,r^\{R\}\_\{i\}\-\\mu^\{R\}\-z\\cdot\\sigma^\{R\}\)\\;\\;\\forall i
7:endfor

8:foreach KV group

k=1,…,Hk​vk=1,\\ldots,H\_\{kv\}do

9:

ek​v​\(k\)←∑i∈group​\(k\)\(exciK\+exciV\)e\_\{kv\}\(k\)\\leftarrow\\sum\_\{i\\in\\text\{group\}\(k\)\}\(\\text\{exc\}^\{K\}\_\{i\}\+\\text\{exc\}^\{V\}\_\{i\}\)
10:endfor

11:foreach query head

h=1,…,Hh=1,\\ldots,Hdo

12:

eq​\(h\)←∑i∈head​\(h\)\(exciQ\+exciO\)e\_\{q\}\(h\)\\leftarrow\\sum\_\{i\\in\\text\{head\}\(h\)\}\(\\text\{exc\}^\{Q\}\_\{i\}\+\\text\{exc\}^\{O\}\_\{i\}\)
13:

ϕ​\(ℓ,h\)←αq⋅eq​\(h\)\+αk​v⋅ek​v​\(⌊h/g⌋\)/g\\phi\(\\ell,h\)\\leftarrow\\alpha\_\{q\}\\cdot e\_\{q\}\(h\)\+\\alpha\_\{kv\}\\cdot e\_\{kv\}\(\\lfloor h/g\\rfloor\)/g
14:Append

\(ℓ,h,ϕ\)\(\\ell,h,\\phi\)to

𝑠𝑐𝑜𝑟𝑒𝑠\\mathit\{scores\}
15:endfor

16:endfor

17:Sort

𝑠𝑐𝑜𝑟𝑒𝑠\\mathit\{scores\}by

ϕ\\phiascending

18:

nprune←⌊s⋅L⋅H⌋n\_\{\\text\{prune\}\}\\leftarrow\\lfloor s\\cdot L\\cdot H\\rfloor
19:for

i=1,…,nprunei=1,\\ldots,n\_\{\\text\{prune\}\}do

20:Zero

WQ\(ℓ,h\),WO\(ℓ,h\)W\_\{Q\}^\{\(\\ell,h\)\},W\_\{O\}^\{\(\\ell,h\)\}slices for head

\(ℓ,h\)\(\\ell,h\)
21:ifall query heads in KV group

⌊h/g⌋\\lfloor h/g\\rfloorare prunedthen

22:Zero

WK\(ℓ,k\),WV\(ℓ,k\)W\_\{K\}^\{\(\\ell,k\)\},W\_\{V\}^\{\(\\ell,k\)\}slices

23:endif

24:endfor

25:returnPruned model

M′M^\{\\prime\}

### III\-EBaseline Scoring Methods

We compare MP\-G against four head\-level scoring criteria adapted from the pruning literature\.

Wanda\-Head\.Adapts the Wanda criterion\[[4](https://arxiv.org/html/2609.26177#bib.bib4)\]to head granularity by aggregating the product of weight magnitudes and input activation norms across Q/K/V rows and O columns of each head:

ϕWanda​\(h\)=\\displaystyle\\phi\_\{\\text\{Wanda\}\}\(h\)=\{\}∑R∈\{Q,K,V\}∑i∈rowsR​\(h\)∑j\|WR,i​j\|⋅‖Xjin‖2\\displaystyle\\sum\_\{R\\in\\\{Q,K,V\\\}\}\\sum\_\{i\\in\\text\{rows\}\_\{R\}\(h\)\}\\sum\_\{j\}\|W\_\{R,ij\}\|\\cdot\\\|X^\{\\mathrm\{in\}\}\_\{j\}\\\|\_\{2\}\(4\)\+∑j∈colsO​\(h\)∑i\|WO,i​j\|⋅‖XjO‖2\.\\displaystyle\+\\sum\_\{j\\in\\text\{cols\}\_\{O\}\(h\)\}\\sum\_\{i\}\|W\_\{O,ij\}\|\\cdot\\\|X^\{O\}\_\{j\}\\\|\_\{2\}\.This requires forward passes over calibration data to collect activation statistics\.

SparseGPT\-Head\.Aggregates a SparseGPT\-inspired Hessian\-diagonal importance scorewi​j2⋅Hj​jw\_\{ij\}^\{2\}\\cdot H\_\{jj\}\[[3](https://arxiv.org/html/2609.26177#bib.bib3)\]atthe head level, whereHj​jH\_\{jj\}is the diagonal of the empirical Hessian of the layer input\. This requires calibration data and Hessian estimation\.

Gradient\-Head\.Uses the first\-order criterion\|wi​j⋅gi​j\|\|w\_\{ij\}\\cdot g\_\{ij\}\|\[[6](https://arxiv.org/html/2609.26177#bib.bib6),[7](https://arxiv.org/html/2609.26177#bib.bib7)\], wheregi​jg\_\{ij\}is the gradient of the language modeling loss with respect to each weight, aggregated per head\. This requires a backward pass over calibration data\.

Hybrid\.Normalizes Magnitude, MP, and Wanda scores to\[0,1\]\[0,1\]via min\-max scaling, then combines:ϕHyb=λmag​s^mag\+λmp​s^mp\+λwanda​s^wanda\\phi\_\{\\text\{Hyb\}\}=\\lambda\_\{\\text\{mag\}\}\\hat\{s\}\_\{\\text\{mag\}\}\+\\lambda\_\{\\text\{mp\}\}\\hat\{s\}\_\{\\text\{mp\}\}\+\\lambda\_\{\\text\{wanda\}\}\\hat\{s\}\_\{\\text\{wanda\}\}withλmag=λmp=λwanda=1\.0\\lambda\_\{\\text\{mag\}\}=\\lambda\_\{\\text\{mp\}\}=\\lambda\_\{\\text\{wanda\}\}=1\.0\. This tests whether fusing weight\-only and calibration signals improves over either alone\.

## IVExperimental Setup

Models\.We evaluate five pre\-trained models spanning encoder and decoder architectures: RoBERTa\-base and RoBERTa\-large\[[17](https://arxiv.org/html/2609.26177#bib.bib17)\]\(125M, 12 heads×\\times12 layers; 355M, 16×\\times24\), OPT\-6\.7B\[[18](https://arxiv.org/html/2609.26177#bib.bib18)\]\(32×\\times32, MHA\), LLaMA\-3\-8B\[[21](https://arxiv.org/html/2609.26177#bib.bib21)\]\(32×\\times32, GQA with 8 KV heads\), and Mistral\-7B\-v0\.1\[[22](https://arxiv.org/html/2609.26177#bib.bib22)\]\(32×\\times32, GQA with 8 KV heads\)\.

Sparsity Levels\.We evaluate at 12\.5%, 25%, and 50% attention head sparsity, applied uniformly via global ranking across layers\.

Perplexity Evaluation\.For decoder models, we report causal language modeling perplexity on the WikiText\-2\[[20](https://arxiv.org/html/2609.26177#bib.bib20)\]validation split \(FP16\)\. For encoder models \(RoBERTa\), we report masked language modeling pseudo\-perplexity \(FP32\)\.

GLUE Evaluation\.For all reported GLUE runs, we fine\-tune RoBERTa\-large on SST\-2 \(accuracy\), MRPC \(F1\), and RTE \(accuracy\)\[[19](https://arxiv.org/html/2609.26177#bib.bib19)\]and then apply head pruning before evaluation \(fine\-tune\-then\-prune protocol\), with learning rate10−510^\{\-5\}and batch size 16 for 3 epochs \(SST\-2\), 5 epochs \(MRPC\), and 10 epochs \(RTE\)\.

Calibration\.Methods requiring calibration data use WikiText\-2 training samples with a sequence length of 512\. Wanda\-Head, SparseGPT\-Head, and Hybrid use 64 samples, while Gradient\-Head uses 32 samples for gradient collection\.MP and MP\-G require no calibration data\.

Hyperparameters\.Default thresholdz=2\.0z\{=\}2\.0for MP/MP\-G\. Hybrid uses equal combination weightsλ=1\.0\\lambda\{=\}1\.0\. All experiments use seed 0\.

## VResults

### V\-APerplexity Across Models

Table[I](https://arxiv.org/html/2609.26177#S5.T1)presents perplexity results across all five models at three sparsity levels\. For encoder models \(RoBERTa\), we report pseudo\-perplexity and for decoder models, we report causal language modeling perplexity on WikiText\-2\.

TABLE I:Perplexity \(↓\\downarrow\) on WikiText\-2 at head sparsitysswithz=2\.0z\{=\}2\.0\. RoBERTa models report pseudo\-perplexity \(FP32\); decoder models report causal LM perplexity \(FP16\)\. “–” indicates unavailable result\. Best inbold, secondunderlined\.On RoBERTa\-large, MP\-G achieves the lowest perplexity at 12\.5% sparsity \(7\.27\) and 25% sparsity \(10\.28\), outperforming all calibration\-dependent methods\. At 50%, MP\-G \(45\.76\) is close to the best method, SparseGPT\-Head \(42\.23\), while requiring no Hessian computation\. The gap of only 3\.5 perplexity points at this aggressive sparsity level is notable given MP\-G’s zero calibration cost\.

On RoBERTa\-base, Gradient\-Head performs best at all sparsity levels\. This suggests that gradient information is particularly valuable for smaller models where weight\-norm heterogeneity may be less pronounced\. MP\-G ranks second at 50% sparsity \(71\.25 vs\. 64\.74\), and outperforms Wanda\-Head \(103\.8\) and SparseGPT\-Head \(4730\) by a wide margin\.

MP\-G performs best on OPT\-6\.7B, obtaining the lowest perplexity at all three sparsity levels: 18\.46 \(12\.5%\), 27\.87 \(25%\), and 152\.0 \(50%\)\. At 12\.5%, MP\-G improves upon the second\-best method \(Hybrid, 25\.64\) by 28%\. At 50%, MP\-G reduces perplexity by 63% relative to Wanda\-Head \(406\.3\) and by 93% relative to SparseGPT\-Head \(2034\)\. These results indicate that the weight\-norm heterogeneity in OPT\-6\.7B is highly informative for head selection\. On OPT\-6\.7B, Gradient\-Head results at 25% and 50% sparsity are unavailable in our aggregate results, preventing direct comparison at higher pruning ratios\.

On LLaMA\-3\-8B, Gradient\-Head achieves the best results across all sparsity levels \(11\.12 at 12\.5%, 17\.69 at 25%, 47\.94 at 50%\), while MP\-G ranks second at 12\.5% \(20\.39\) and 25% \(41\.41\)\. On Mistral\-7B, MP\-G performs competitively at low sparsity \(22\.48 at 12\.5%\), while Gradient\-Head leads at 25% \(13\.30\) and 50% \(54\.11\)\. The Hybrid method achieves 14\.27 at 12\.5% on Mistral, its best result across all models\. At 50% sparsity on Mistral, among the non\-gradient baselines, Wanda\-Head \(552\.8\) outperforms both MP\-G \(1072\) and SparseGPT\-Head \(871\.8\), suggesting that activation\-based signals can be important in certain GQA architectures at high sparsity\. Gradient\-Head remains best in this setting\.

### V\-BEffect of the Thresholdzz

The MP thresholdzzcontrols outlier sensitivity: higherzzpreserves fewer, more extreme outlier heads\. Table[II](https://arxiv.org/html/2609.26177#S5.T2)shows its effect on selected model\-sparsity pairs\.

TABLE II:Effect of thresholdzzon MP\-G perplexity at 25% and 50% sparsity\.Two patterns are observed\. First, the optimalzzis model\-dependent: MHA models \(RoBERTa\-base\) often prefer lowerzz\(1\.5\), while GQA models \(LLaMA\-3\) and larger encoders \(RoBERTa\-large\) benefit from higherzz\(2\.0–2\.5\)\. This suggests that GQA architectures, where fewer KV heads create stronger outlier structure, benefit from more aggressive outlier\-based filtering\. Second,z=2\.5z\{=\}2\.5produces a large improvement on LLaMA\-3\-8B at 50% sparsity \(from 7227 to 282\.8\), indicating that at extreme pruning ratios, preserving only the most extreme outliers is critical\. These results suggest that a model\-adaptivezzselection strategy could further improve MP\-G, which we leave to future work\.

### V\-CDownstream Task Performance

Table[III](https://arxiv.org/html/2609.26177#S5.T3)reports GLUE\[[19](https://arxiv.org/html/2609.26177#bib.bib19)\]results for RoBERTa\-large with the fine\-tune\-then\-prune protocol\. At 25% sparsity, all reported methods retain substantial task performance on SST\-2 \(92\.8–95\.2% accuracy\)\. SparseGPT\-Head shows the strongest downstream resilience overall, maintaining 89\.6% SST\-2 accuracy and 81\.4 MRPC F1 even at 50% sparsity, where MP\-G, Wanda\-Head, and Hybrid collapse on MRPC\. The Hybrid method achieves the highest SST\-2 accuracy at 25% sparsity \(95\.2%\)\.

The gap between perplexity and downstream performance at 50% sparsity is worth examining: MP\-G achieves competitive perplexity \(45\.76 vs\. 42\.23 for SparseGPT\-Head\), yet SparseGPT\-Head shows better downstream robustness after post\-fine\-tuning pruning\. This suggests that Hessian\-guided scoring better preserves the task\-specific representations learned during fine\-tuning, whereas MP\-G preserves heads important for general language modeling\. At 50% sparsity on RTE, all reported methods converge to near\-chance accuracy \(47\.3%\), indicating this challenging task cannot tolerate removal of half its attention capacity\.

TABLE III:GLUE results for RoBERTa\-large \(fine\-tune\-then\-prune\)\. SST\-2 and RTE: accuracy; MRPC: F1\.
### V\-DEfficiency Analysis

At 50% head sparsity, all scoring methods remove the same number of attention heads, so the theoretical sparsity\-dependent efficiency effects are identical across methods\.

Parameter reduction ranges from 8% for GQA models \(where KV parameters are shared and MLP parameters dominate\) to 16% for MHA models\. All configurations achieve the theoretical 50% reduction in attention FLOPs\. However, measured wall\-clock change is modest, ranging from small slowdowns to about 3\.5% speedup, because standard PyTorch attention kernels do not dynamically skip zeroed heads\. Realizing the theoretical FLOP savings requires either model surgery to physically remove pruned head dimensions or integration with sparse inference engines, an important engineering step for deployment\.

## VIAnalysis

### VI\-AWhy Does MP\-G Work?

MP\-G’s effectiveness rests on the empirical observation that trained Transformer attention weights are not uniformly distributed across heads\. Instead, a minority of heads develop disproportionately large weight norms, an outlier structure that MP exploits\. This observation is consistent with prior findings that large\-magnitude features emerge in trained Transformers\[[14](https://arxiv.org/html/2609.26177#bib.bib14),[15](https://arxiv.org/html/2609.26177#bib.bib15)\]\. On OPT\-6\.7B and RoBERTa\-large, where this heterogeneity is most pronounced, MP\-G outperforms all baselines\. On LLaMA\-3\-8B, where Gradient\-Head performs best, the weight norm distribution may be more uniform, reducing MP’s discriminative power\.

This heterogeneity has been independently observed in other contexts: Voita et al\.\[[5](https://arxiv.org/html/2609.26177#bib.bib5)\]found that only a few “specialized” heads perform critical functions \(positional, syntactic, rare\-token attention\), while the majority are redundant\. Clark et al\.\[[13](https://arxiv.org/html/2609.26177#bib.bib13)\]showed that BERT attention heads exhibit distinct patterns, with some heads attending broadly while others focus on specific linguistic relations\. MP\-G can be understood as detecting these specialized heads via their weight\-norm signatures, without needing to observe their attention patterns on data\.

### VI\-BCalibration\-Free Advantage

A practical strength of MP\-G is its complete independence from data\. The method:

- •Requiresno calibration dataset, eliminating distributional mismatch concerns, which is particularly relevant when the pruning\-time domain differs from the deployment domain\.
- •Requiresno forward or backward passes, making it applicable even when GPU memory is insufficient to run inference on the full model\.
- •Executes inseconds\(weight norm computation\) rather than minutes \(forward passes over 64 calibration sequences of length 512\), making it suitable for rapid prototyping and hyperparameter sweeps\.

In contrast, Wanda\-Head, SparseGPT\-Head, and Gradient\-Head each require 32–64 calibration sequences, forward passes to collect activations or Hessians, and \(for Gradient\-Head\) backward passes\. For a 7B\-parameter model, this overhead is non\-negligible and scales with calibration set size\.

### VI\-CMethod Robustness Across Architectures

Table[IV](https://arxiv.org/html/2609.26177#S6.T4)summarizes competitive performance by counting the number of model\-sparsity combinations where each method achieves the best or second\-best perplexity\. We only consider results atz=2\.0z\{=\}2\.0where at least two methods produce finite perplexity\. All 15 model\-sparsity settings satisfy this condition\.

TABLE IV:Number of model\-sparsity pairs where each method achieves best or second\-best perplexity \(out of 15 comparable pairs atz=2\.0z\{=\}2\.0with≥2\{\\geq\}2finite results\)\.Gradient\-Head achieves the most first\-place finishes, with eight\. However, its OPT\-6\.7B results at 25% and 50% sparsity are unavailable in our aggregate results\. MP\-G achieves top\-two performance in most primary model\-sparsity settings and produces finite perplexity on every tested configuration\. This combination of competitive performance and broad robustness across architectures is a useful property when calibration data is unavailable\.

## VIIConclusion

We have introduced Magnitude Profile \(MP\-G\) scoring, a calibration\-free structured attention head pruning criterion based on weight\-norm outlier detection\. MP\-G requires access only to model weights\. It does not require calibration data, forward passes, gradients, or Hessian estimation, while still achieving competitive or superior perplexity in several settings and top\-two performance in most primary model\-sparsity settings\. On OPT\-6\.7B, MP\-G leads at all sparsity levels; on RoBERTa\-large, it leads at 12\.5% and 25% and closely matches the best Hessian\-based method at 50%\.

For practitioners, MP\-G offers a fast, zero\-cost initial pruning strategy: compute head scores in seconds, prune, and optionally fine\-tune if downstream task performance is critical\. The method is particularly suited to scenarios where calibration data is unavailable, domain\-mismatched, or where rapid pruning iteration is needed\. Future work should explore automatic selection ofzz, integration with MLP pruning for deeper compression, and model surgery or sparse kernel support to translate the theoretical FLOP savings into measured speedup\.

## References

- \[1\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin, “Attention is all you need,” inProc\. NeurIPS, 2017\.
- \[2\]Y\. LeCun, J\. S\. Denker, and S\. A\. Solla, “Optimal brain damage,” inAdvances in Neural Information Processing Systems 2, 1989\.
- \[3\]E\. Frantar and D\. Alistarh, “SparseGPT: Massive language models can be accurately pruned in one\-shot,” inProc\. ICML, 2023\.
- \[4\]M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. Kolter, “A simple and effective pruning approach for large language models,” inProc\. ICLR, 2024\.
- \[5\]E\. Voita, D\. Talbot, F\. Moiseev, R\. Sennrich, and I\. Titov, “Analyzing multi\-head self\-attention: Specialized heads do the heavy lifting, the rest can be pruned,” inProc\. ACL, 2019\.
- \[6\]P\. Michel, O\. Levy, and G\. Neubig, “Are sixteen heads really better than one?” inProc\. NeurIPS, 2019\.
- \[7\]P\. Molchanov, S\. Tyree, T\. Karras, T\. Aila, and J\. Kautz, “Pruning convolutional neural networks for resource efficient inference,” inProc\. ICLR, 2017\.
- \[8\]S\. Han, J\. Pool, J\. Tran, and W\. J\. Dally, “Learning both weights and connections for efficient neural networks,” inProc\. NeurIPS, 2015\.
- \[9\]X\. Ma, G\. Fang, and X\. Wang, “LLM\-Pruner: On the structural pruning of large language models,” inProc\. NeurIPS, 2023\.
- \[10\]M\. Xia, T\. Gao, Z\. Zeng, and D\. Chen, “Sheared LLaMA: Accelerating language model pre\-training via structured pruning,” inProc\. ICLR, 2024\.
- \[11\]S\. Ashkboos, M\. L\. Croci, M\. Gennari do Nascimento, T\. Hoefler, and J\. Hensman, “SliceGPT: Compress large language models by deleting rows and columns,” inProc\. ICLR, 2024\.
- \[12\]M\. Xia, Z\. Zhong, and D\. Chen, “Structured pruning learns compact and accurate models,” inProc\. ACL, 2022\.
- \[13\]K\. Clark, U\. Khandelwal, O\. Levy, and C\. D\. Manning, “What does BERT look at? An analysis of BERT’s attention,” inProc\. ACL Workshop BlackboxNLP, 2019\.
- \[14\]T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer, “LLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale,” inProc\. NeurIPS, 2022\.
- \[15\]O\. Kovaleva, A\. Romanov, A\. Rogers, and A\. Rumshisky, “Revealing the dark secrets of BERT,” inProc\. EMNLP\-IJCNLP, 2019\.
- \[16\]J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebrón, and S\. Sanghai, “GQA: Training generalized multi\-query transformer models from multi\-head checkpoints,” inProc\. EMNLP, 2023\.
- \[17\]Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,”arXiv preprint arXiv:1907\.11692, 2019\.
- \[18\]S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin, T\. Mihaylov, M\. Ott, S\. Shleifer, K\. Shuster, D\. Simig, P\. S\. Koura, A\. Sridhar, T\. Wang, and L\. Zettlemoyer, “OPT: Open pre\-trained transformer language models,”arXiv preprint arXiv:2205\.01068, 2022\.
- \[19\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman, “GLUE: A multi\-task benchmark and analysis platform for natural language understanding,” inProc\. ICLR, 2019\.
- \[20\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher, “Pointer sentinel mixture models,” inProc\. ICLR, 2017\.
- \[21\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, et al\., “The Llama 3 herd of models,”arXiv preprint arXiv:2407\.21783, 2024\.
- \[22\]A\. Q\. Jiang et al\., “Mistral 7B,”arXiv preprint arXiv:2310\.06825, 2023\.