Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention

# Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention
Source: [https://arxiv.org/html/2607.08027](https://arxiv.org/html/2607.08027)
###### Abstract

This paper proposes an improved structured pruning method for large language models \(LLMs\) that addresses key challenges in adapting Adaptive Feature Retention \(AFR\), an unstructured pruning technique, to structured pruning\. When applying AFR to structured pruning, three major problems arise: distribution mismatch between heterogeneous pruning scores, loss of sign information indicating optimization direction consistency, and influence of outliers\. To address these issues, we propose a unified approach combining power transformation for nonlinear distribution alignment, sign\-preserving score aggregation, and percentile\-based outlier removal\. Experiments on Llama\-3\-8B, Vicuna\-v1\.5\-13B, and LLaVA\-v1\.5\-13B demonstrate that our method maintains accuracy comparable to unstructured pruning while achieving practical inference speedup through structured pruning\.

## 1Introduction

Large language models \(LLMs\) demonstrate remarkable performance across diverse tasks, yet their billions of parameters impose substantial computational costs and memory requirements that hinder practical deployment\. Pruning, which removes redundant weights, offers a promising approach to model compression\.

Pruning methods are categorized into unstructured and structured pruning\. Unstructured pruning removes individual weights independently, achieving high accuracy but providing limited practical speedup due to irregular sparsity patterns\. Structured pruning removes entire neurons or channels, enabling efficient acceleration on standard hardware but often suffering from larger performance degradation due to reduced pruning granularity\.

Adaptive Feature Retention \(AFR\)\(Nittaet al\.,[2025](https://arxiv.org/html/2607.08027#bib.bib1)\)is an unstructured pruning method that combines feature\-based ReFer\(Nittaet al\.,[2024](https://arxiv.org/html/2607.08027#bib.bib2)\)and gradient\-based SNIP\(Leeet al\.,[2019](https://arxiv.org/html/2607.08027#bib.bib3)\)scores through standardization and summation, balancing preservation of pre\-trained feature representations with adaptation to downstream tasks\. When adapting AFR to structured pruning by aggregating weight\-level scores to neuron\-level scores, several challenges arise: distribution mismatch between heterogeneous scores \(ReFer exhibits wide value ranges while SNIP concentrates in \[0, 1\]\), loss of sign information indicating optimization direction consistency, and influence of outliers that dominate simple averaging\.

To address these challenges, we propose an integrated approach combining power transformation for nonlinear distribution alignment, sign\-preserving aggregation, and percentile\-based outlier removal\. Experiments on Llama\-3\-8B, Vicuna\-v1\.5\-13B\(Zhenget al\.,[2023](https://arxiv.org/html/2607.08027#bib.bib5)\), and LLaVA\-v1\.5\-13B\(Liuet al\.,[2024](https://arxiv.org/html/2607.08027#bib.bib6)\)demonstrate that our method significantly outperforms naive structured AFR and achieves comparable or superior performance to existing structured pruning methods while enabling 1\.56\-1\.57×\\timesinference speedup at 50% pruning rate\.

Our main contributions are summarized as follows:

- •We identify three fundamental challenges in adapting AFR to structured pruning: distribution mismatch between heterogeneous pruning scores, loss of sign information indicating optimization direction consistency, and influence of outliers during score aggregation\.
- •We propose an integrated approach that addresses these challenges through \(1\) power transformation for nonlinear distribution alignment, \(2\) sign\-preserving score aggregation to evaluate optimization direction consistency, and \(3\) percentile\-based outlier removal for robust aggregation\.
- •We conduct comprehensive experiments on three models \(Llama\-3\-8B, Vicuna\-v1\.5\-13B, and LLaVA\-v1\.5\-13B\) demonstrating that our method achieves substantial improvements over naive averaging \(up to 21\.27 points\) and outperforms existing structured pruning methods while enabling practical inference speedup \(1\.56\-1\.57×\\timesat 50% pruning\)\.

## 2Background: Adaptive Feature Retention

Adaptive Feature Retention \(AFR\)\(Nittaet al\.,[2025](https://arxiv.org/html/2607.08027#bib.bib1)\)is an unstructured pruning method for pre\-trained models that combines ReFer\(Nittaet al\.,[2024](https://arxiv.org/html/2607.08027#bib.bib2)\)and SNIP\(Leeet al\.,[2019](https://arxiv.org/html/2607.08027#bib.bib3)\)to balance preservation of learned feature spaces with adaptation to downstream tasks\. The AFR pruning scoreSA​F​R​\(θn\)S\_\{AFR\}\(\\theta\_\{n\}\)for weightθn\\theta\_\{n\}is defined as:

SA​F​R​\(θn\)=𝒵​\(\|∂∑l=1LFs​v​dl∂θn​θn\|\)\+𝒵​\(\|∂ℒ∂θn​θn\|\)S\_\{AFR\}\(\\theta\_\{n\}\)=\\mathcal\{Z\}\\left\(\\left\|\\frac\{\\partial\\sum\_\{l=1\}^\{L\}F^\{l\}\_\{svd\}\}\{\\partial\\theta\_\{n\}\}\\theta\_\{n\}\\right\|\\right\)\+\\mathcal\{Z\}\\left\(\\left\|\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\theta\_\{n\}\}\\theta\_\{n\}\\right\|\\right\)\(1\)where𝒵​\(⋅\)\\mathcal\{Z\}\(\\cdot\)denotes standardization \(z\-score normalization\), the first term represents ReFer, and the second term represents SNIP\. Here,Fs​v​dlF^\{l\}\_\{svd\}is the mean singular value from singular value decomposition of layerll’s output features, andℒ\\mathcal\{L\}is the task\-specific loss function\.

ReFer focuses on maintaining feature representations by measuring how each weight affects the feature space geometry captured during pre\-training\. It computes the sensitivity of feature space characteristics with respect to each weight, identifying parameters that are crucial for preserving the learned representations\. However, while ReFer excels at maintaining pre\-trained knowledge, it may inadvertently remove weights that are necessary for adapting to downstream tasks, as it does not directly consider task\-specific objectives\.

SNIP, on the other hand, uses the product of loss gradients and weights as the pruning score, directly measuring each weight’s contribution to the task\-specific loss function\. This gradient\-based approach naturally identifies parameters important for the current task\. However, for pre\-trained models, many weights have already converged to near\-optimal values, resulting in small gradients \(∂ℒ∂θn≈0\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\theta\_\{n\}\}\\approx 0\) that do not accurately reflect their importance\. This makes SNIP alone insufficient for proper importance evaluation in the context of pre\-trained models\.

By combining both scores through standardization and summation, AFR achieves effective pruning that preserves pre\-trained knowledge while enabling task adaptation\. The standardization ensures that both scores contribute equally despite their different scales, allowing the combined score to capture both feature preservation and task relevance\.

## 3Challenges in adapting AFR to structured pruning

To enable inference acceleration with AFR, we apply its weight\-level pruning scores to structured pruning\. This section identifies the problems that arise in this adaptation\.

### 3\.1Naive aggregation for structured pruning

To adapt the unstructured AFR method to structured pruning, we consider naive averaging for score aggregation as illustrated in Figure[1](https://arxiv.org/html/2607.08027#S3.F1)\. First, we compute pruning scores for each element of the weight matrix\. Then, we aggregate scores within each neuron and compute the average scoreS¯j\\bar\{S\}\_\{j\}at the neuron level as:

S¯j=1m​∑i=1m\|Si​j\|\\bar\{S\}\_\{j\}=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\|S\_\{ij\}\|\(2\)whereSi​jS\_\{ij\}is the pruning score of theii\-th weight in neuronjj, andmmis the number of weights per neuron\. Finally, neurons with the lowest average scores are removed according to the target pruning rate\.

![Refer to caption](https://arxiv.org/html/2607.08027v1/x1.png)Figure 1:Pruning score aggregation process for structured pruning\.Table 1:Preliminary experimental results \(accuracy %\)\.
### 3\.2Problems with naive aggregation

Table[1](https://arxiv.org/html/2607.08027#S3.T1)shows that applying naive averaging to AFR on LLMs results in severe performance degradation on natural language benchmarks\. We identify three underlying problems:

Problem 1: Distribution mismatch between heterogeneous scores\.ReFer and SNIP scores have fundamentally different distribution characteristics\. ReFer scores span a wide value range with relatively uniform distribution, while SNIP scores concentrate in the narrow \[0, 1\] range due to small gradients in pre\-trained models\. Simple standardization in Eq\. equation[1](https://arxiv.org/html/2607.08027#S2.E1)linearly amplifies the minute differences among SNIP’s densely clustered low\-importance parameters, introducing noise into the AFR scores\. This noise disrupts ReFer’s importance judgments and degrades pruning accuracy\.

Problem 2: Loss of optimization direction consistency\.ReFer and SNIP scores are computed as products of gradients and weights, where the sign encodes weight behavior during optimization\. In the weight update ruleθ←θ−η​∂L∂θ\\theta\\leftarrow\\theta\-\\eta\\frac\{\\partial L\}\{\\partial\\theta\}, positive scores indicate weights decreasing in magnitude, while negative scores indicate increasing magnitude\. For structured pruning, the consistency of optimization directions within a neuron is critical\. When all weights in a neuron optimize in the same direction, the neuron has a structurally coherent role\. Conversely, mixed directions indicate internal cancellation effects and limited overall contribution\. However, Eq\. equation[2](https://arxiv.org/html/2607.08027#S3.E2)takes absolute values before averaging, discarding this directional consistency information and preventing proper neuron\-level importance evaluation\.

Problem 3: Influence of outliers\.Figure[2](https://arxiv.org/html/2607.08027#S3.F2)shows the weight\-level score distribution for ReFer in one layer\. Most scores concentrate in a relatively narrow range, but extreme outliers exist at both tails\. The full score range spans−2,884\-2\{,\}884to9,8039\{,\}803\(width:12,68712\{,\}687\), but removing the extreme 2% reduces the range width to48\.848\.8\. With naive averaging, the presence or absence of outliers can reverse the relative ordering of neuron\-level scores, preventing proper importance evaluation\.

![Refer to caption](https://arxiv.org/html/2607.08027v1/x2.png)Figure 2:Example distribution of AFR pruning scores showing concentration in the center with extreme outliers at the tails\.

## 4Proposed method

We propose an integrated approach combining three techniques to address the problems identified above\.

### 4\.1Nonlinear distribution alignment via power transformation

To address Problem 1, we apply power transformation to SNIP scores before standardization\. For SNIP scoresnSNIPs\_\{n\}^\{\\text\{SNIP\}\}, we apply:

s^nSNIP=\(snSNIP\)b\\hat\{s\}\_\{n\}^\{\\text\{SNIP\}\}=\\left\(s\_\{n\}^\{\\text\{SNIP\}\}\\right\)^\{b\}\(3\)whereb\>1b\>1\. For0≤snSNIP<10\\leq s\_\{n\}^\{\\text\{SNIP\}\}<1, this transformation has two key properties that address the distribution mismatch problem\.

First, it provides noise suppression in the low\-importance region\. Sincexb≪xx^\{b\}\\ll xfor values near 0, minute differences in low\-importance parameters are nonlinearly compressed, effectively filtering out noise from the densely clustered low\-importance region of SNIP scores\.

Second, it enables signal preservation in the high\-importance region\. For values near 1,xb≈xx^\{b\}\\approx xholds approximately, preserving information for high\-importance parameters\. This asymmetric behavior creates a soft thresholding effect, where noise suppression strength adjusts continuously according to importance magnitude\.

After power transformation, we apply standardization to the transformed scores\. To quantify the improvement in distribution quality, we analyze the signal\-to\-noise ratio \(SNR\)\. Letμsignal\\mu\_\{\\text\{signal\}\}denote the mean absolute value of high\-importance scores \(top 10% by magnitude\) andσnoise\\sigma\_\{\\text\{noise\}\}denote the standard deviation of low\-importance scores \(bottom 50%\)\. The SNR before and after power transformation can be expressed as:

SNRoriginal=μsignalσnoise,SNRtransformed=μsignalbσnoiseb\\text\{SNR\}\_\{\\text\{original\}\}=\\frac\{\\mu\_\{\\text\{signal\}\}\}\{\\sigma\_\{\\text\{noise\}\}\},\\quad\\text\{SNR\}\_\{\\text\{transformed\}\}=\\frac\{\\mu\_\{\\text\{signal\}\}^\{b\}\}\{\\sigma\_\{\\text\{noise\}\}^\{b\}\}\(4\)Since SNIP scores concentrate in \[0, 1\] with most values near 0, we haveσnoise<μsignal<1\\sigma\_\{\\text\{noise\}\}<\\mu\_\{\\text\{signal\}\}<1\. Forb\>1b\>1, the power function satisfiesxb≪xx^\{b\}\\ll xforx≪1x\\ll 1andxb≲xx^\{b\}\\lesssim xforx≈1x\\approx 1, which meansσnoiseb≪μsignalb\\sigma\_\{\\text\{noise\}\}^\{b\}\\ll\\mu\_\{\\text\{signal\}\}^\{b\}\. Therefore, the ratioSNRtransformed/SNRoriginal=\(μsignal/σnoise\)b−1\\text\{SNR\}\_\{\\text\{transformed\}\}/\\text\{SNR\}\_\{\\text\{original\}\}=\(\\mu\_\{\\text\{signal\}\}/\\sigma\_\{\\text\{noise\}\}\)^\{b\-1\}increases exponentially withbb, demonstrating that power transformation amplifies the relative distinction between signal and noise\. Figure[3](https://arxiv.org/html/2607.08027#S4.F3)illustrates this effect: the left histogram shows how power transformation compresses the low\-value region before standardization, while the right histogram demonstrates how this compression leads to improved concentration after standardization\.

The power transformation reduces the varianceσS^2\\sigma\_\{\\hat\{S\}\}^\{2\}compared to the original scores, causing the standardization division to have a stronger stretching effect on the remaining distribution\. However, critically, only the signal component remaining after transformation is stretched, while noise has already been suppressed in the preceding step\. This improved SNR during scale adjustment enables more effective integration with ReFer scores\.

![Refer to caption](https://arxiv.org/html/2607.08027v1/x3.png)Figure 3:Distribution comparison before and after power transformation \(b=2\.7b=2\.7\)\. Left: SNIP score distributions before standardization show how power transformation compresses the densely clustered low\-value region\. Right: Distributions after standardization demonstrate improved concentration, with the transformed distribution \(orange\) exhibiting reduced spread in the low\-importance region while maintaining separation of high\-importance values\.
### 4\.2Sign\-preserving score aggregation

To address Problem 2, we propose aggregation with post\-averaging absolute values\. Instead of taking absolute values before averaging as in Eq\. equation[2](https://arxiv.org/html/2607.08027#S3.E2), we average the signed pruning scores within each neuron, then take the absolute value after averaging\.

This approach leverages sign information to evaluate optimization direction consistency\. If all weights have consistent signs, the average retains a large magnitude\. Conversely, if weights have mixed signs, the average approaches zero due to cancellation\. This naturally quantifies the degree of directional consistency\.

Neurons with consistent optimization directions indicate structurally coherent roles where all weights cooperate toward the same objective, making them valuable for preservation\. In contrast, neurons with mixed directions exhibit internal cancellation effects, suggesting limited overall contribution\. By preserving sign information during aggregation, our method enables proper evaluation of this structural property for structured pruning\.

### 4\.3Percentile\-based outlier removal

To mitigate Problem 3’s outlier influence, we introduce percentile\-based outlier removal\. To determine the optimal removal rate, we analyzed pruning score distribution statistics \(Table[2](https://arxiv.org/html/2607.08027#S4.T2)\)\.

At 2% removal, the range decreases to 48\.8, and kurtosis drops from 16,488,324 to 3,505 \(a 99\.98% reduction\)\. Kurtosis quantifies distribution tail heaviness; the original distribution’s extremely high kurtosis indicates a long\-tailed distribution\. From 1% to 2% removal, kurtosis decreases by 94\.2%, but from 2% to 3%, the decrease is only 54\.9%, suggesting outlier removal effects saturate beyond 2%\.

The interquartile range \(IQR\) represents the spread of the central 50% of the distribution\. Substantial IQR changes indicate removal of not just outliers but also central distribution information, suggesting information loss\. The original IQR is 91\.72, and at 2% removal it remains 86\.71, preserving the structure, while at 10% removal it becomes 68\.20\.

These results confirm that 2% removal optimally eliminates outlier influence \(from range and kurtosis perspectives\) while preserving original distribution information \(from IQR perspective\)\.

Table 2:Statistical changes by outlier removal rate\.Therefore, we compute the 2nd percentileP2P\_\{2\}and 98th percentileP98P\_\{98\}for each neuron’s score set and remove scores outside these thresholds\.

### 4\.4Integrated approach

We combine the techniques from Sections[4\.1](https://arxiv.org/html/2607.08027#S4.SS1)–[4\.3](https://arxiv.org/html/2607.08027#S4.SS3)into a unified method:

1. 1\.Outlier removal\.Remove scores outside the 2nd and 98th percentiles for each neuron’s ReFer and SNIP score sets\.
2. 2\.Sign\-preserving aggregation\.Compute the mean of cleaned scores while maintaining signs, then take the absolute value to obtain neuron\-level scores\.
3. 3\.Power transformation\.Apply power transformation \(Eq\. equation[3](https://arxiv.org/html/2607.08027#S4.E3)\) to aggregated neuron\-level SNIP scores\.
4. 4\.Score combination\.Sum transformed neuron\-level SNIP scores with neuron\-level ReFer scores to obtain final pruning scores\.

Concretely, for each neuron, we first remove weight\-level scores outside the 2nd and 98th percentiles for both ReFer and SNIP\. We then compute the average of the remaining scores while preserving signs, followed by taking absolute values to obtain neuron\-level scores for ReFer and SNIP separately\. The power transformation with exponentbbis applied to the neuron\-level SNIP score\. Finally, we sum the transformed SNIP score with the ReFer score to obtain the final neuron\-level pruning score\.

This integrated approach sequentially applies outlier suppression, sign consistency evaluation, and heterogeneous score distribution alignment\. Each component addresses a specific problem while building upon the outputs of previous steps, enabling stable and accurate pruning score evaluation for structured pruning\.

## 5Experiments

We evaluate our proposed method on Llama\-3\-8B, Vicuna\-v1\.5\-13B, and LLaVA\-v1\.5\-13B\.

### 5\.1Experimental setup

We apply structured pruning to the feed\-forward networks \(FFNs\) in each block of the models\. FFNs constitute a substantial portion of model parameters and are amenable to structured pruning due to their layer\-wise independence\. For Llama\-3\-8B and Vicuna\-v1\.5\-13B, which are language\-only models, we apply our method to all FFN layers across the transformer blocks\. Each FFN consists of multiple linear layers that can be pruned at the neuron level\.

LLaVA\-v1\.5\-13B presents a more complex scenario as it consists of a language part \(Vicuna\) and vision part \(CLIP\) connected through projection layers\. For the language part where calibration data has ground\-truth labels, we apply the full AFR method \(combining ReFer and SNIP\) to prune FFNs\. For the vision part, CLIP functions as a feature extractor without direct downstream task supervision, making SNIP’s loss gradients undefined since there is no task\-specific loss to differentiate through the vision encoder\. Therefore, we prune CLIP’s fully connected layers using only the ReFer component, which relies solely on feature space geometry and does not require task\-specific gradients\.

We evaluate at pruning rates of 20% and 50%, widely adopted in prior work as representative of moderate and aggressive pruning scenarios\. The power transformation exponent is set tob=2\.7b=2\.7based on systematic ablation experiments described in Appendix[A](https://arxiv.org/html/2607.08027#A1), which demonstrates that this value achieves optimal performance across multiple benchmarks\. For Llama\-3\-8B and Vicuna\-v1\.5\-13B, we use five benchmark datasets covering diverse language understanding capabilities: WinoGrande\(Sakaguchiet al\.,[2020](https://arxiv.org/html/2607.08027#bib.bib7)\)for commonsense reasoning, HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2607.08027#bib.bib8)\)for commonsense natural language inference, ARC\-easy and ARC\-challenge\(Clarket al\.,[2018](https://arxiv.org/html/2607.08027#bib.bib9)\)for question answering, and MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.08027#bib.bib10)\)for multitask language understanding\. For LLaVA\-v1\.5\-13B, we use three vision\-language benchmarks: GQA\(Hudson and Manning,[2019](https://arxiv.org/html/2607.08027#bib.bib11)\)for visual reasoning, VizWiz\(Gurariet al\.,[2018](https://arxiv.org/html/2607.08027#bib.bib12)\)for visual question answering, and ScienceQA\(Luet al\.,[2022](https://arxiv.org/html/2607.08027#bib.bib13)\)for multimodal science questions\. Accuracy serves as the evaluation metric across all benchmarks\. We compare against unstructured AFR, structured AFR with naive averaging, and existing structured pruning methods including LLM\-Pruner\(Maet al\.,[2023](https://arxiv.org/html/2607.08027#bib.bib14)\), LoRAP\(Liet al\.,[2024](https://arxiv.org/html/2607.08027#bib.bib15)\), and CFSP\(Wanget al\.,[2025](https://arxiv.org/html/2607.08027#bib.bib16)\)\.

Table 3:Comparison with existing methods on Llama\-3\-8B \(accuracy %\)\.
### 5\.2Comparison with existing methods

Tables[3](https://arxiv.org/html/2607.08027#S5.T3),[4](https://arxiv.org/html/2607.08027#S5.T4), and[5](https://arxiv.org/html/2607.08027#S5.T5)present results for Llama\-3\-8B, Vicuna\-v1\.5\-13B, and LLaVA\-v1\.5\-13B, respectively\.

For Llama\-3\-8B, our method achieves 64\.46 at 20% pruning and 47\.24 at 50% pruning, representing improvements of 21\.27 and 15\.13 points over naive averaging\. Compared to unstructured AFR, our method shows only 1\.45 point difference at 20% and exceeds it by 0\.32 points at 50%, maintaining performance comparable to unstructured pruning while enabling structured acceleration\. Compared to existing structured methods, our method outperforms the average by 3\.60 points at 20% and 9\.04 points at 50%\. At 50% pruning, the improvements are substantial: 12\.37 points over LLM\-Pruner, 8\.49 points over LoRAP, and 6\.47 points over CFSP\.

Similar trends appear for Vicuna\-v1\.5\-13B\. Our method achieves 63\.70% at 20% and 52\.91% at 50%, exceeding existing method averages by 1\.76 and 6\.40 points respectively\. At 50% pruning, our method maintains a clear advantage, with CFSP being the closest competitor at 51\.20% average accuracy\.

For the vision\-language model LLaVA\-v1\.5\-13B, our method achieves 62\.83% at 20% and 47\.08% at 50%, demonstrating effectiveness across different model architectures including VLMs\. At 20% pruning, performance slightly exceeds the original model by 0\.06 points, indicating effective identification and removal of redundant parameters\.

The performance gains result from the synergistic combination of distribution alignment, sign consistency evaluation, and outlier suppression\. The power transformation addresses heterogeneous score distributions, sign\-preserving aggregation captures structural properties of neuron\-level optimization, and outlier removal ensures stable aggregation resistant to extreme values\.

Table 4:Comparison with existing methods on Vicuna\-v1\.5\-13B \(accuracy %\)\.Table 5:Evaluation results on LLaVA\-v1\.5\-13B \(accuracy %\)\.
### 5\.3Resource reduction evaluation

Tables[6](https://arxiv.org/html/2607.08027#S5.T6)and[7](https://arxiv.org/html/2607.08027#S5.T7)show resource reduction effects for each pruning rate\. For Llama\-3\-8B, 20% pruning achieves 14\.0% parameter reduction \(from 8\.03B to 6\.90B\), 14\.0% VRAM reduction \(from 16\.06GB to 13\.08GB\), and 1\.10×\\timesinference speedup\. At 50% pruning, the reductions become more substantial: 35\.1% parameter reduction \(to 5\.21B\), 35\.1% VRAM reduction \(to 10\.42GB\), and 1\.57×\\timesinference speedup\.

For LLaVA\-v1\.5\-13B, 20% pruning achieves 13\.0% parameter reduction \(from 13\.3B to 11\.4B\), 11\.7% VRAM reduction, and 1\.20×\\timesspeedup\. At 50% pruning, LLaVA\-v1\.5\-13B achieves 32\.6% parameter reduction \(to 8\.98B\), 24\.9% VRAM reduction \(to 25\.55GB\), and 1\.56×\\timesspeedup\.

The speedup is less than the theoretical maximum \(2×\\timesfor 50% pruning\) because \(1\) only FFN layers are pruned while attention layers remain intact, and \(2\) memory access costs and kernel launch overhead become more significant proportionally\. Nevertheless, the achieved speedups represent practical improvements for deployment scenarios\. Structured pruning enables this acceleration without specialized sparse computation libraries, allowing the pruned models to run efficiently on standard hardware using dense matrix operations\.

Table 6:Resource reduction evaluation for Llama\-3\-8B\.Table 7:Resource reduction evaluation for LLaVA\-v1\.5\-13B\.

## 6Conclusion

This paper proposes an improved structured pruning method that maintains accuracy while enabling inference acceleration\. We identified three key challenges in adapting AFR to structured pruning: distribution mismatch between heterogeneous scores, loss of sign information, and outlier influence\. To address these, we propose an integrated approach combining power transformation for nonlinear distribution alignment, sign\-preserving aggregation, and percentile\-based outlier removal\.

Experiments on Llama\-3\-8B, Vicuna\-v1\.5\-13B, and LLaVA\-v1\.5\-13B demonstrated the effectiveness of our approach\. Our method achieves substantial improvements over naive averaging \(21\.27 and 15\.13 points at 20% and 50% pruning\) and outperforms existing structured pruning methods\. At 50% pruning on Llama\-3\-8B, our method achieves 35\.1% parameter reduction and 1\.57×\\timesinference speedup\. For LLaVA\-v1\.5\-13B, we achieve 32\.6% parameter reduction and 1\.56×\\timesspeedup\. Ablation studies confirm synergistic effects of all proposed components\.

Future work includes extending to attention modules, optimizing layer\-wise adaptive pruning rates\.

## References

- Think you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§5\.1](https://arxiv.org/html/2607.08027#S5.SS1.p3.1)\.
- D\. Gurari, Q\. Li, A\. J\. Stangl, A\. Guo, C\. Lin, K\. Grauman, J\. Luo, and J\. P\. Bigham \(2018\)VizWiz grand challenge: answering visual questions from blind people\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§5\.1](https://arxiv.org/html/2607.08027#S5.SS1.p3.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§5\.1](https://arxiv.org/html/2607.08027#S5.SS1.p3.1)\.
- D\. A\. Hudson and C\. D\. Manning \(2019\)GQA: a new dataset for real\-world visual reasoning and compositional question answering\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§5\.1](https://arxiv.org/html/2607.08027#S5.SS1.p3.1)\.
- N\. Lee, T\. Ajanthan, and P\. H\. S\. Torr \(2019\)SNIP: single\-shot network pruning based on connection sensitivity\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.08027#S1.p3.1),[§2](https://arxiv.org/html/2607.08027#S2.p1.2)\.
- G\. Li, Y\. Zeng, J\. Huang, Z\. Zhang, Z\. Wang, X\. Zhang, and Z\. Zhao \(2024\)LoRAP: transformer sub\-layers deserve differentiated structured compression for large language models\.arXiv preprint arXiv:2404\.09695\.Cited by:[§5\.1](https://arxiv.org/html/2607.08027#S5.SS1.p3.1)\.
- H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee \(2024\)Improved baselines with visual instruction tuning\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 26286–26296\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52733.2024.02484)Cited by:[§1](https://arxiv.org/html/2607.08027#S1.p4.1)\.
- P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. Kalyan \(2022\)Learn to explain: multimodal reasoning via thought chains for science question answering\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§5\.1](https://arxiv.org/html/2607.08027#S5.SS1.p3.1)\.
- X\. Ma, G\. Fang, and X\. Wang \(2023\)LLM\-Pruner: on the structural pruning of large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§5\.1](https://arxiv.org/html/2607.08027#S5.SS1.p3.1)\.
- T\. Nitta, T\. Hirakawa, T\. Yamashita, and H\. Fujiyoshi \(2025\)Single\-shot foresight pruning balancing knowledge retention and downstream task adaptation for pre\-trained models\.InMeeting on Image Recognition and Understanding \(MIRU\),Cited by:[§1](https://arxiv.org/html/2607.08027#S1.p3.1),[§2](https://arxiv.org/html/2607.08027#S2.p1.2)\.
- T\. Nitta, H\. Kohama, T\. Hirakawa, T\. Yamashita, and H\. Fujiyoshi \(2024\)Single\-shot foresight pruning for maintaining feature representations of pre\-trained models\.InMeeting on Image Recognition and Understanding \(MIRU\),Cited by:[§1](https://arxiv.org/html/2607.08027#S1.p3.1),[§2](https://arxiv.org/html/2607.08027#S2.p1.2)\.
- K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. Choi \(2020\)WinoGrande: an adversarial Winograd schema challenge at scale\.InAAAI Conference on Artificial Intelligence \(AAAI\),Cited by:[§5\.1](https://arxiv.org/html/2607.08027#S5.SS1.p3.1)\.
- Y\. Wang, M\. Ma, Z\. Wang, J\. Chen, L\. Shan, Q\. Yang, D\. Xu, M\. Liu, and B\. Qin \(2025\)CFSP: an efficient structured pruning framework for LLMs with coarse\-to\-fine activation information\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§5\.1](https://arxiv.org/html/2607.08027#S5.SS1.p3.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§5\.1](https://arxiv.org/html/2607.08027#S5.SS1.p3.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2607.08027#S1.p4.1)\.

## Appendix ADetailed analysis of power transformation exponent

This appendix provides comprehensive ablation experiments on the power transformation exponentbbto support our choice ofb=2\.7b=2\.7in the main paper\.

### A\.1Experimental setup

We systematically evaluate different values ofbbranging from 1\.0 to 3\.0 on Llama\-3\-8B at 20% pruning rate\. For each value ofbb, we apply the full proposed method \(power transformation \+ sign\-preserving aggregation \+ outlier removal\) and measure average accuracy across all five benchmark datasets \(WinoGrande, HellaSwag, ARC\-easy, ARC\-challenge, MMLU\)\.

### A\.2Results

Figure[4](https://arxiv.org/html/2607.08027#A1.F4)shows the relationship between the power transformation exponentbband pruning performance\. Table[8](https://arxiv.org/html/2607.08027#A1.T8)provides the detailed numerical results\. The results demonstrate several key findings:

Low exponent region \(b<1\.5b<1\.5\):Whenbbis too small, the transformation provides insufficient noise suppression\. Atb=1\.0b=1\.0\(no transformation\), the method achieves 61\.29% average accuracy\. Performance gradually improves asbbincreases: 61\.39% atb=1\.1b=1\.1, 62\.21% atb=1\.2b=1\.2, reaching 63\.82% atb=1\.4b=1\.4\. In this region, the SNIP scores retain too much of their original densely\-clustered distribution, and standardization amplifies noise from the low\-importance region\.

Transition region \(b=1\.5b=1\.5–2\.52\.5\):Performance continues to improve steadily, reaching 63\.78% atb=1\.5b=1\.5and peaking at 64\.31% atb=1\.7b=1\.7\. The curve shows some minor fluctuations but maintains an upward trend, with 64\.15% atb=2\.0b=2\.0and 64\.31% atb=2\.5b=2\.5\.

Optimal region \(b=2\.5b=2\.5–3\.03\.0\):Performance stabilizes at its highest level in this range\. Starting from 64\.31% atb=2\.5b=2\.5, accuracy reaches 64\.35% atb=2\.6b=2\.6, peaks at 64\.46% atb=2\.7b=2\.7, and maintains 64\.43% atb=2\.8b=2\.8, 64\.35% atb=2\.9b=2\.9, and 64\.44% atb=3\.0b=3\.0\. This plateau demonstrates remarkable robustness, with all values in the range of 64\.31–64\.46% \(a variation of only 0\.15 percentage points\)\. Within this range, the transformation optimally balances noise suppression and signal preservation\.

Performance gain:Comparing the optimal value \(b=2\.7b=2\.7, 64\.46%\) with no transformation \(b=1\.0b=1\.0, 61\.29%\), the power transformation contributes a 3\.17 percentage point improvement, demonstrating its significant impact on pruning quality\.

Table 8:Detailed ablation results for power transformation exponentbbon Llama\-3\-8B at 20% pruning \(accuracy %\)\.![Refer to caption](https://arxiv.org/html/2607.08027v1/x4.png)Figure 4:Ablation study of power transformation exponentbbon Llama\-3\-8B at 20% pruning rate\. The curve shows average accuracy across five benchmarks \(WinoGrande, HellaSwag, ARC\-easy, ARC\-challenge, MMLU\)\. Performance improves from 61\.29% atb=1\.0b=1\.0to a peak of 64\.46% atb=2\.7b=2\.7, then plateaus in the rangeb=2\.5b=2\.5–3\.03\.0with minimal variation \(0\.15 percentage points\)\.
### A\.3Analysis of the performance curve

The performance curve exhibits three distinct phases\. In the initial phase \(b<1\.5b<1\.5\), the transformation is too weak, and performance improves monotonically asbbincreases\. The transition phase \(b=1\.5b=1\.5–2\.52\.5\) shows continued improvement with minor fluctuations, suggesting the transformation is approaching its optimal compression level\. Finally, the plateau phase \(b≥2\.5b\\geq 2\.5\) demonstrates that the method has reached its performance ceiling, where further increasingbbneither helps nor hurts significantly\.

Interestingly, even atb=3\.0b=3\.0, performance remains at 64\.44%, only 0\.02 percentage points below the peak\. This robustness is practically valuable, as it means practitioners do not need precise tuning ofbb\. Any value in the range\[2\.5,3\.0\]\[2\.5,3\.0\]will yield near\-optimal results\.

### A\.4Implications

The existence of a clear optimal range with a relatively flat plateau has important practical implications\. Users can applyb=2\.7b=2\.7without extensive hyperparameter tuning, and small deviations \(±0\.3\\pm 0\.3\) have minimal impact on performance\. This robustness makes the method practical for deployment across different models and datasets\.