TriSP: Tri-Signal Structured Pruning for Large Language Models

# TriSP: Tri-Signal Structured Pruning for Large Language Models
Source: [https://arxiv.org/html/2607.22587](https://arxiv.org/html/2607.22587)
Manel Kara laouar National School of Artificial Intelligence \(ENSIA\) Sidi Abdellah Campus, Algiers, Algeria manel\.karalaouar@ensia\.edu\.dz &Soumia Bouyahiaoui National School of Artificial Intelligence \(ENSIA\) Sidi Abdellah Campus, Algiers, Algeria soumia\.bouyahiaoui@ensia\.edu\.dz &Aicha Boutorh National School of Artificial Intelligence \(ENSIA\) Sidi Abdellah Campus, Algiers, Algeria aicha\.boutorh@ensia\.edu\.dz

###### Abstract

Large language models \(LLMs\) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters\. Structured pruning addresses this by removing entire structures such as attention heads and Multi\-Layer Perceptron \(MLP\) neurons to produce smaller dense models that run efficiently on standard hardware\. However, existing methods rely on either gradient\-based importance estimation, which is memory\-prohibitive, or activation\-based statistical proxies, which do not directly measure the effect of removal on the loss\. Furthermore, the interaction between the importance criterion and the post\-pruning recovery strategy has not been systematically studied\. We propose TriSP \(Tri\-Signal Structured Pruning\), an importance metric that combines weight magnitude scaled by activation norm with first\-order gradient sensitivity via a geometric mean, producing a channel\-level score that captures both structural and loss\-sensitivity signals\. Combined with adaptive per\-layer budget allocation and low\-rank adaptation \(LoRA\) recovery, TriSP achieves the lowest perplexity and highest zero\-shot accuracy across all tested configurations, reaching 6\.80 WikiText\-2 perplexity at 20% pruning on LLaMA\-7B\. Inference throughput improves by 82% at 50% pruning, while still maintaining competitive performance\.

## 1Introduction

Large language models \(LLMs\) based on the transformer architecture\[[3](https://arxiv.org/html/2607.22587#bib.bib1)\]have become the dominant paradigm for natural language processing\. Models such as GPT\-4\[[1](https://arxiv.org/html/2607.22587#bib.bib2)\], LLaMA\[[26](https://arxiv.org/html/2607.22587#bib.bib3)\], Mistral\[[15](https://arxiv.org/html/2607.22587#bib.bib32)\], and DeepSeek\[[5](https://arxiv.org/html/2607.22587#bib.bib5)\]demonstrate strong performance across several tasks\. However, their practical deployment is constrained by the computational and memory resources they require: a 7\-billion parameter model in half\-precision occupies approximately 13 GB of GPU memory and requires hundreds of billions of multiply\-accumulate operations per forward pass\[[31](https://arxiv.org/html/2607.22587#bib.bib6)\]\.

*Structured pruning*reduces this cost by removing entire architectural components: attention heads, Multi\-Layer Perceptron \(MLP\) neurons, or channels, producing smaller dense models that benefit from standard GPU matrix operations\. Unlike unstructured pruning, which zeros individual weights and requires specialized sparse kernels for speedup, structured pruning yields direct latency and memory reductions on commodity hardware\. The central challenge is identifying which structures to remove with minimal quality degradation, which requires both an accurate importance criterion and an allocation strategy that distributes the pruning budget across layers in proportion to their redundancy\.

Existing approaches address these requirements with complementary but incomplete solutions\. FLAP\[[2](https://arxiv.org/html/2607.22587#bib.bib12)\]uses activation variance scaled by squared weight magnitude, a statistical proxy that is fast to compute but does not directly measure the effect of removal on the model’s loss\. LLM\-Pruner\[[20](https://arxiv.org/html/2607.22587#bib.bib11)\]uses Taylor\-based gradient importance with dependency graph analysis, which more directly estimates pruning impact but requires storing gradient tensors for all pretrained weights, making it memory\-prohibitive for large models\. Both methods apply post\-pruning recovery \(bias compensation for FLAP, LoRA fine\-tuning for LLM\-Pruner\) to restore quality, but no prior work has evaluated how the choice of recovery strategy affects the final quality\.

Two research gaps motivate this work\. First, gradient augmentation has consistently improved unstructured pruning: GBLM\-Pruner\[[9](https://arxiv.org/html/2607.22587#bib.bib9)\]adds a gradient term to Wanda’s\[[24](https://arxiv.org/html/2607.22587#bib.bib8)\]activation\-norm criterion, and Wanda\+\+\[[27](https://arxiv.org/html/2607.22587#bib.bib10)\]extends this with regional gradients\. However, this approach has not been transferred to structured pruning, where gradient\-based and activation\-based methods remain separate\. Second, FLAP’s bias compensation is presented as sufficient to eliminate fine\-tuning, yet quality degrades substantially at aggressive ratios, and no prior work has compared this static correction against LoRA recovery\[[14](https://arxiv.org/html/2607.22587#bib.bib17)\]under the same pruning criterion\. This paper makes two contributions:

1. 1\.TriSP \(Tri\-Signal Structured Pruning\), an importance metric that combines activation\-weighted magnitude with gradient sensitivity via a geometric mean, transferring gradient augmentation from unstructured to structured pruning\.
2. 2\.Asystematic comparisonof bias compensation vs\. LoRA recovery under the same pruning criterion, and comprehensive evaluation across five models\.

The remainder of this paper is organized as follows: Section[2](https://arxiv.org/html/2607.22587#S2)reviews related work, Section[3](https://arxiv.org/html/2607.22587#S3)describes the proposed methodology, Section[4](https://arxiv.org/html/2607.22587#S4)presents the experimental setup and results, and Section[5](https://arxiv.org/html/2607.22587#S5)concludes the paper\.

## 2Related Work

Structured pruning for LLMs removes entire structural units \(heads, neurons, channels\) from a pretrained model to produce a smaller dense architecture that directly benefits from standard GPU kernels\[[13](https://arxiv.org/html/2607.22587#bib.bib27)\]\. Methods differ along two axes: the importance criterion used to rank structures and whether post\-pruning recovery is applied\.

### 2\.1Gradient\-Based Methods

First\-order Taylor expansion provides a principled estimate of how much removing a structural unit affects the loss\. LLM\-Pruner\[[20](https://arxiv.org/html/2607.22587#bib.bib11)\]applies this criterion with approximate Hessian information, coupling parameters into structural groups \(heads, MLP neurons\) via a dependency graph and recovering quality with a lightweight LoRA adapter\. However, it requires storing full gradient tensors for all pretrained weights, making it memory\-intensive at scale\. Two subsequent methods address this bottleneck by computing Taylor importance from the small LoRA adapter matrices instead: LoRAPrune\[[29](https://arxiv.org/html/2607.22587#bib.bib16)\]enables pruning of LLaMA\-65B on a single A100 GPU and applies iterative progressive pruning, while DyLoRA\-Prune\[[18](https://arxiv.org/html/2607.22587#bib.bib28)\]replaces fixed\-rank LoRA with Dynamic Low\-Rank Adaptation, learning representations across a range of ranks simultaneously so that lower ranks capture the most critical information\.

An alternative line of work uses second\-order information\. SlimGPT\[[19](https://arxiv.org/html/2607.22587#bib.bib15)\]extends the Optimal Brain Surgeon \(OBS\) framework\[[12](https://arxiv.org/html/2607.22587#bib.bib20)\], previously applied to unstructured pruning \(SparseGPT\[[10](https://arxiv.org/html/2607.22587#bib.bib7)\]\) and quantization \(GPTQ\[[11](https://arxiv.org/html/2607.22587#bib.bib19)\]\), introducing Batched Greedy Pruning with grouped Cholesky decomposition for heads and dynamic group sizing for MLP channels\. Even without fine\-tuning, it reaches 52\.23% zero\-shot average at 50% on LLaMA\-7B, outperforming LLM\-Pruner with LoRA, while requiring only 7 GB of GPU memory at 20%\.

### 2\.2Gradient\-Free Methods

Activation\-based criteria avoid gradient computation entirely\. FLAP\[[2](https://arxiv.org/html/2607.22587#bib.bib12)\]scores channels by the sample variance of input activations weighted by squared weight\-column norm, replacing low\-fluctuation channels with a fixed bias correction that requires no gradients\. Similarly, Bonsai\[[16](https://arxiv.org/html/2607.22587#bib.bib29)\]frames importance as a regression problem, fitting a linear model over randomly sampled sub\-models to estimate each module’s contribution globally, enabling pruning on a single 48 GB A6000 where gradient\-based methods require 80–640 GB\. Probe Pruning\[[17](https://arxiv.org/html/2607.22587#bib.bib30)\]extends the activation\-based paradigm by making decisions online: a lightweight probe subset \(5% of samples\) runs ahead through a few layers, fusing resulting states with historical calibration states so that current batch characteristics dominate when they deviate from calibration, surpassing FLAP \(61\.0% vs\. 60\.6% commonsense accuracy at 40% on LLaMA\-2\-13B\) with only 1\.5% overhead in FLOPs\.

A separate family of gradient\-free methods operates at coarser granularity\. SliceGPT\[[4](https://arxiv.org/html/2607.22587#bib.bib13)\]exploits the computational invariance of RMSNorm transformers under orthogonal rotations, concentrating signal into leading PCA components and slicing off small\-eigenvalue dimensions uniformly across the model\. At the layer level, ShortGPT\[[21](https://arxiv.org/html/2607.22587#bib.bib14)\]removes entire transformer layers based on Block Influence, observing that middle layers carry the most redundancy, while BlockPruner\[[30](https://arxiv.org/html/2607.22587#bib.bib31)\]refines this by treating MHA and MLP blocks as separate pruning units and finding asymmetric redundancy: below 17% pruning MHA blocks are more dispensable, but beyond that threshold MHA becomes critical while MLP remains tolerant\.

### 2\.3Gradient Augmentation in Unstructured Pruning

In unstructured pruning, Wanda\[[24](https://arxiv.org/html/2607.22587#bib.bib8)\]demonstrates that a simple activation\-aware criterionSi​j=\|Wi​j\|⋅‖Xj‖2S\_\{ij\}=\|W\_\{ij\}\|\\cdot\\\|X\_\{j\}\\\|\_\{2\}can match SparseGPT’s accuracy without weight reconstruction\. GBLM\-Pruner\[[9](https://arxiv.org/html/2607.22587#bib.bib9)\]extends this score with a gradient term, consistently outperforming both Wanda and SparseGPT on LLaMA\-2\-7B at 50% sparsity\. Wanda\+\+\[[27](https://arxiv.org/html/2607.22587#bib.bib10)\]further reduces memory cost by restricting gradient computation to single decoder blocks and applies regional weight updates to minimize pruning\-induced output discrepancy\. However, this gradient augmentation has not been transferred to structured channel scoring\.

In this work, we propose TriSP, which bridges this gap by combining activation\-weighted magnitude with gradient sensitivity for structured pruning\.

## 3Methodology

The proposed method operates as a three\-stage pipeline \(Figure[1](https://arxiv.org/html/2607.22587#S3.F1)\): importance estimation, structural pruning via adaptive layer\-module \(AL\-AM\) allocation, and an optional low\-rank adaptation \(LoRA\) recovery\.

============================================================

![Refer to caption](https://arxiv.org/html/2607.22587v1/pipline_structered_pruning.jpg)Figure 1:Overview of the proposed TriSP structured pruning pipeline\.\(1\)Compute tri\-signal importance scoresSl,jS\_\{l,j\}, the geometric mean of WIFNl,jand GSl,j, combining weight magnitude, gradient sensitivity, and activation norm per channel from calibration data;\(2\)Standardize scores across layers via z\-score normalization and apply a unified threshold;\(3\)Physically remove pruned heads and neurons with adaptive per\-layer retention ratios, then inject LoRA adapters into the frozen pruned weights and merge after fine\-tuning\.### 3\.1Calibration

Importance estimation requires collecting activation statistics and gradients from a small calibration set\. For each sample, we perform a full forward pass through the model to obtain logits, compute the per\-sample cross\-entropy lossℒ\\mathcal\{L\}on next\-token prediction:

ℒ=−1T−1​∑t=1T−1log⁡P​\(xt\+1∣x1,…,xt\)\\mathcal\{L\}=\-\\frac\{1\}\{T\-1\}\\sum\_\{t=1\}^\{T\-1\}\\log P\(x\_\{t\+1\}\\mid x\_\{1\},\\dots,x\_\{t\}\)\(1\)whereTTis the sequence length andxtx\_\{t\}is the token at positiontt\. We then execute a backward pass to produce weight gradients\. The absolute gradients are accumulated across allNNcalibration samples to form a per\-weight average gradient magnitude:

G¯l,i​j=1N​∑t=1N\|∂ℒ\(t\)∂Wl,i​j\|\\bar\{G\}\_\{l,ij\}=\\frac\{1\}\{N\}\\sum\_\{t=1\}^\{N\}\\left\|\\frac\{\\partial\\mathcal\{L\}^\{\(t\)\}\}\{\\partial W\_\{l,ij\}\}\\right\|\(2\)whereWl,i​jW\_\{l,ij\}denotes the weight at rowii, columnjjof a linear layer in transformer layerll\.

During the forward pass, input activation statistics are collected at the output projection and down projection of each transformer layer as a running squaredℓ2\\ell\_\{2\}norm:

al,j\(t\)=nn\+b​al,j\(t−1\)\+‖Xl,j\(t\)‖22n\+ba\_\{l,j\}^\{\(t\)\}=\\frac\{n\}\{n\+b\}\\,a\_\{l,j\}^\{\(t\-1\)\}\+\\frac\{\\\|X\_\{l,j\}^\{\(t\)\}\\\|\_\{2\}^\{2\}\}\{n\+b\}\(3\)
whereal,j\(t\)a\_\{l,j\}^\{\(t\)\}is the accumulated squared activation norm for input channeljjin layerllat steptt,Xl,j\(t\)X\_\{l,j\}^\{\(t\)\}is the corresponding activation vector,nnis the number of samples seen so far, andbbis the current batch size\. After all calibration batches,al,ja\_\{l,j\}converges to the mean squared activation norm at channeljj\.

Although TriSP requires an additional backward pass compared to FLAP, this roughly doubles the per\-sample compute only during calibration, which is a one\-time cost that completes in minutes on a single GPU, and the improved pruning quality justifies the overhead\.

### 3\.2Tri\-Signal Structured Metric

We propose a tri\-signal importance metric that combines a structural signal from FLAP\[[2](https://arxiv.org/html/2607.22587#bib.bib12)\]with a gradient\-based sensitivity signal\. For a linear layer with weight matrixW∈ℝdout×dinW\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}\}, we define two per\-channel components\. The first is the*Weighted Input Feature Norm*\(WIFN\), which captures weight magnitude scaled by activation norm:

WIFNl,j=1dout​∑i=1dout\|Wl,i​j\|⋅al,j\+ϵ\\text\{WIFN\}\_\{l,j\}=\\frac\{1\}\{d\_\{\\text\{out\}\}\}\\sum\_\{i=1\}^\{d\_\{\\text\{out\}\}\}\|W\_\{l,ij\}\|\\cdot\\sqrt\{a\_\{l,j\}\+\\epsilon\}\(4\)whereWl,i​jW\_\{l,ij\}is the weight connecting input channeljjto output channeliiin layerll,doutd\_\{\\text\{out\}\}is the number of output channels,al,ja\_\{l,j\}is the accumulated squared activation norm from Equation[3](https://arxiv.org/html/2607.22587#S3.E3), andϵ=10−6\\epsilon=10^\{\-6\}\.

The second is the*Gradient Sensitivity*\(GS\) term, which captures how much the loss depends on each channel, again scaled by activation norm:

GSl,j=\(1dout​∑i=1dout\|Wl,i​j\|⋅G¯l,i​j\)⋅al,j\+ϵ\\text\{GS\}\_\{l,j\}=\\left\(\\frac\{1\}\{d\_\{\\text\{out\}\}\}\\sum\_\{i=1\}^\{d\_\{\\text\{out\}\}\}\|W\_\{l,ij\}\|\\cdot\\bar\{G\}\_\{l,ij\}\\right\)\\cdot\\sqrt\{a\_\{l,j\}\+\\epsilon\}\(5\)whereG¯l,i​j\\bar\{G\}\_\{l,ij\}is the averaged absolute gradient from Equation[2](https://arxiv.org/html/2607.22587#S3.E2)\. The final importance score is the geometric mean of these two components:

Sl,j=WIFNl,j⋅GSl,jS\_\{l,j\}=\\sqrt\{\\text\{WIFN\}\_\{l,j\}\\cdot\\text\{GS\}\_\{l,j\}\}\(6\)
This formulation captures three complementary signals:

- •Weight magnitude\|Wl,i​j\|\|W\_\{l,ij\}\|: channels connected through large weights carry more information\.
- •Gradient magnitudeG¯l,i​j\\bar\{G\}\_\{l,ij\}: channels with high gradient magnitude indicate higher importance for model performance\.
- •Input activation normal,j\\sqrt\{a\_\{l,j\}\}: channels that receive large activations during calibration contribute more to the layer output, consistent with the observation bySunet al\.\[[24](https://arxiv.org/html/2607.22587#bib.bib8)\]that outlier features in LLMs carry disproportionate importance\.

The geometric mean prevents the gradient term from dominating the score when gradients are noisy or have high variance across layers\.

### 3\.3Adaptive Layer\-Module Allocation

Following FLAP\[[2](https://arxiv.org/html/2607.22587#bib.bib12)\], we adopt an adaptive allocation strategy that distributes pruning budgets across layers and module types based on measured redundancy\.

##### Cross\-layer standardization\.

The raw TriSP scores differ in magnitude across layers because deeper layers tend to have different activation scales and gradient norms than shallow ones\. Applying a single global threshold to raw scores would prune too aggressively in layers with small scores and too conservatively in layers with large scores\. To make scores comparable across layers, we apply per\-layer z\-score standardization:

S^l,j=Sl,j−μlσl\\hat\{S\}\_\{l,j\}=\\frac\{S\_\{l,j\}\-\\mu\_\{l\}\}\{\\sigma\_\{l\}\}\(7\)whereSl,jS\_\{l,j\}is the TriSP importance score of channeljjin layerll, andμl\\mu\_\{l\}andσl\\sigma\_\{l\}are the mean and standard deviation of all channel scores within layerll\.

##### Attention head scoring\.

For multi\-head attention, the per\-channel TriSP scores of the output projection are grouped by head\. Each head spansdhd\_\{h\}contiguous channels, and the head\-level importance is the mean of its channel scores:

Hl,k=1dh​∑j∈headkS^l,jH\_\{l,k\}=\\frac\{1\}\{d\_\{h\}\}\\sum\_\{j\\in\\text\{head\}\_\{k\}\}\\hat\{S\}\_\{l,j\}\(8\)whereHl,kH\_\{l,k\}is the importance score of headkkin layerll,dhd\_\{h\}is the head dimension, andS^l,j\\hat\{S\}\_\{l,j\}are the z\-score normalized channel scores from Equation[7](https://arxiv.org/html/2607.22587#S3.E7)\.

##### Multi\-Layer Perceptron neuron scoring\.

Each MLP neuron corresponds to one intermediate dimension, shared across the gate, up, and down projections\. The standardized TriSP scores of the down projection’s input channels are used directly as per\-neuron importance scores\.

##### Global threshold with compression weighting\.

All head scores and neuron scores across all layers are concatenated into a single vector and sorted in descending order\. Because removing one attention head eliminates4×dh×dhidden4\\times d\_\{h\}\\times d\_\{\\text\{hidden\}\}parameters \(across four projections:WQW\_\{Q\},WKW\_\{K\},WVW\_\{V\}, andWOW\_\{O\}\), while removing one MLP neuron eliminates only3×dhidden3\\times d\_\{\\text\{hidden\}\}parameters \(across three projections:WgateW\_\{\\text\{gate\}\},WupW\_\{\\text\{up\}\}, andWdownW\_\{\\text\{down\}\}\), pruning a head removes far more parameters than pruning a neuron\. To account for this, each component is assigned acompression weightcic\_\{i\}that reflects its relative parameter cost: each attention head receivesci=4​dh/3c\_\{i\}=4d\_\{h\}/3, while each MLP neuron receivesci=1c\_\{i\}=1\.

The global threshold is determined in two steps\. First, the optimal cutoff positionk∗k^\{\*\}is found by scanning the sorted list and selecting the position where the cumulative compression weight is closest to the total compression weight that should be retained:

k∗=arg⁡min𝑘​\|∑m=1kcπ​\(m\)−\(1−p\)​∑ici\|k^\{\*\}=\\underset\{k\}\{\\arg\\min\}\\left\|\\sum\_\{m=1\}^\{k\}c\_\{\\pi\(m\)\}\-\(1\-p\)\\sum\_\{i\}c\_\{i\}\\right\|\(9\)whereπ\\piis the permutation that sorts all component scores in descending order,cπ​\(m\)c\_\{\\pi\(m\)\}is the compression weight of the component at sorted positionmm,ppis the target pruning ratio, and\(1−p\)​∑ici\(1\-p\)\\sum\_\{i\}c\_\{i\}is the total compression weight\. Then, the thresholdτ\\tauis set to the score at that position:

τ=Sπ​\(k∗\)\\tau=S\_\{\\pi\(k^\{\*\}\)\}\(10\)All components withSi\>τS\_\{i\}\>\\tauare retained; the rest are pruned\.

### 3\.4Structural Pruning

Once the binary masks are determined, we compress the model by physically removing the pruned dimensions from the weight matrices, producing a smaller dense model\.

The rows of the query projectionWQW\_\{Q\}, key projectionWKW\_\{K\}, and value projectionWVW\_\{V\}corresponding to pruned heads are removed, and the corresponding columns of the output projectionWOW\_\{O\}are removed\. For the MLP, we remove the rows of the up projectionWupW\_\{\\text\{up\}\}and gate projectionWgateW\_\{\\text\{gate\}\}corresponding to pruned neurons, and the corresponding columns of the down projectionWdownW\_\{\\text\{down\}\}\. The number of retained MLP neurons per layer is rounded down to the nearest multiple of 64 to ensure efficient GPU memory access and matrix computation\.

In the original FLAP formulation, a fixed bias vector compensates for the removed channels by adding the mean activation of pruned inputs to the layer output:bcomp=Wout⊤⋅x¯prunedb\_\{\\text\{comp\}\}=W\_\{\\text\{out\}\}^\{\\top\}\\cdot\\bar\{x\}\_\{\\text\{pruned\}\}, whereWoutW\_\{\\text\{out\}\}is the weight matrix of the output projection andx¯pruned\\bar\{x\}\_\{\\text\{pruned\}\}is the mean activation vector of the pruned channels computed over the calibration set\. This is a rank\-zero approximation that captures only the mean and ignores per\-sample variation\.

We deliberately omit this bias term, as the LoRA adapters applied in the recovery stage learn a richer, input\-dependent correction that subsumes what the fixed bias provides \(see Figure[2](https://arxiv.org/html/2607.22587#S4.F2)\)\.

### 3\.5Low\-Rank Adaptation Recovery

Structural pruning introduces a quality\-accuracy trade\-off\. To recover performance efficiently, we employ Low\-Rank Adaptation \(LoRA\)\[[14](https://arxiv.org/html/2607.22587#bib.bib17)\]as a post\-pruning recovery step\. LoRA injects low\-rank adapter matrices into the pruned model’s attention projections \(WQW\_\{Q\},WKW\_\{K\},WVW\_\{V\}, andWOW\_\{O\}\) of every transformer layer\. Only the adapter parameters are trained while all pretrained weights remain frozen, making the recovery lightweight\. After training, adapters are merged back into the pruned weights\.

Algorithm 1TriSP: Tri\-Signal Structured Pruning with LoRA Recovery1:Model

ℳ\\mathcal\{M\}with

LLtransformer layers, calibration set

𝒟cal\\mathcal\{D\}\_\{\\text\{cal\}\}\(

N=512N\{=\}512samples\), pruning ratio

pp, LoRA rank

rr, recovery dataset

𝒟rec\\mathcal\{D\}\_\{\\text\{rec\}\}
2:Pruned and recovered model

ℳ′\\mathcal\{M\}^\{\\prime\}
3:Stage 1: Importance Estimation

4:foreach layer

l=1,…,Ll=1,\\dots,Ldo

5:Initialize

al,j←0a\_\{l,j\}\\leftarrow 0,

G¯l,i​j←0\\bar\{G\}\_\{l,ij\}\\leftarrow 0; collect input activations at

WO\(l\)W\_\{O\}^\{\(l\)\}and

Wdown\(l\)W\_\{\\text\{down\}\}^\{\(l\)\}
6:endfor

7:foreach sample

\(x,y\)∈𝒟cal\(x,y\)\\in\\mathcal\{D\}\_\{\\text\{cal\}\}do

8:Forward pass:

y^←ℳ​\(x\)\\hat\{y\}\\leftarrow\\mathcal\{M\}\(x\); compute

ℒ\\mathcal\{L\}\(Eq\.[1](https://arxiv.org/html/2607.22587#S3.E1)\); backward pass

9:foreach layer

lldo

10:Accumulate

G¯l,i​j\\bar\{G\}\_\{l,ij\}\(Eq\.[2](https://arxiv.org/html/2607.22587#S3.E2)\); update

al,ja\_\{l,j\}\(Eq\.[3](https://arxiv.org/html/2607.22587#S3.E3)\)

11:endfor

12:endfor

13:foreach layer

lldo

14:Compute WIFNl,j\(Eq\.[4](https://arxiv.org/html/2607.22587#S3.E4)\), GSl,j\(Eq\.[5](https://arxiv.org/html/2607.22587#S3.E5)\), TriSP score

Sl,jS\_\{l,j\}\(Eq\.[6](https://arxiv.org/html/2607.22587#S3.E6)\)

15:endfor

16:Stage 2: Adaptive Layer\-Module \(AL\-AM\)

17:foreach layer

lldo

18:Standardize scores

S^l,j\\hat\{S\}\_\{l,j\}via z\-score normalization \(Eq\.[7](https://arxiv.org/html/2607.22587#S3.E7)\)

19:endfor

20:Aggregate per\-head importance

Hl,kH\_\{l,k\}\(Eq\.[8](https://arxiv.org/html/2607.22587#S3.E8)\);

21:Sort the scores in descending order

22:Assign compression weights:

ci←4​dh/3c\_\{i\}\\leftarrow 4d\_\{h\}/3for heads,

ci←1c\_\{i\}\\leftarrow 1for neurons

23:Find global threshold

τ\\taufor target ratio

pp\(Eq\.[9](https://arxiv.org/html/2607.22587#S3.E9)\); generate masks

Mi←𝟏​\[scorei\>τ\]M\_\{i\}\\leftarrow\\mathbf\{1\}\[\\text\{score\}\_\{i\}\>\\tau\]
24:Stage 3: Structural Pruning and Recovery

25:foreach layer

lldo

26:Remove pruned head rows from

WQ,WK,WVW\_\{Q\},W\_\{K\},W\_\{V\}and columns from

WOW\_\{O\}; remove pruned neuron rows from

Wup,WgateW\_\{\\text\{up\}\},W\_\{\\text\{gate\}\}and columns from

WdownW\_\{\\text\{down\}\};

27:endfor

28:Freeze all weights; inject LoRA adapters into

WQ,WK,WV,WOW\_\{Q\},W\_\{K\},W\_\{V\},W\_\{O\}
29:Fine\-tune adapters on

𝒟rec\\mathcal\{D\}\_\{\\text\{rec\}\}for 2 epochs; merge adapters into pruned weights

30:return

ℳ′\\mathcal\{M\}^\{\\prime\}

## 4Experiments and Results

### 4\.1Experimental Setup

##### Models\.

The primary pruning experiments are conducted on LLaMA\-2\-7B\[[26](https://arxiv.org/html/2607.22587#bib.bib3)\]and Vicuna\-7B\[[7](https://arxiv.org/html/2607.22587#bib.bib33)\]at pruning ratios of 20%, 30%, and 50%\. To assess generalization across architectures, the same pipeline is additionally evaluated on DeepSeek\-7B\[[5](https://arxiv.org/html/2607.22587#bib.bib5)\]and Mistral\-7B\[[15](https://arxiv.org/html/2607.22587#bib.bib32)\]\. Table[1](https://arxiv.org/html/2607.22587#S4.T1)summarizes the architectural configurations of all models\.

Table 1:Architectural configurations of all models used in this work\.ModelLLddHQH\_\{Q\}HK​VH\_\{KV\}dffd\_\{\\text\{ff\}\}\|V\|\|V\|ContextLLaMA\-2\-7B324096323211 00832 0004 096Vicuna\-7B324096323211 00832 0004 096DeepSeek\-7B304096323211 008102 4004 096Mistral\-7B32409632814 33632 00032 768
LL: layers,dd: hidden dim\.,HQH\_\{Q\}: query heads,HK​VH\_\{KV\}: KV heads,dffd\_\{\\text\{ff\}\}: intermediate dim\.,\|V\|\|V\|: vocab size\.

##### Baselines\.

The proposed TriSP pruning pipeline is compared against FLAP\[[2](https://arxiv.org/html/2607.22587#bib.bib12)\]\(with and without bias compensation\) and LLM\-Pruner\[[20](https://arxiv.org/html/2607.22587#bib.bib11)\]\(with and without LoRA recovery\)\. All methods use the same calibration set \(512 WikiText\-2 samples\) for fair comparison\.

##### Metrics\.

We evaluate four dimensions: \(i\) language modeling quality via WikiText\-2 test and Penn Treebank perplexity \(PTB\); \(ii\) downstream task accuracy via zero\-shot evaluation on six commonsense benchmarks: PIQA\[[6](https://arxiv.org/html/2607.22587#bib.bib22)\], HellaSwag\[[28](https://arxiv.org/html/2607.22587#bib.bib23)\], WinoGrande\[[23](https://arxiv.org/html/2607.22587#bib.bib24)\], ARC\-Easy and ARC\-Challenge\[[8](https://arxiv.org/html/2607.22587#bib.bib25)\], and OBQA\[[22](https://arxiv.org/html/2607.22587#bib.bib26)\]; \(iii\) computational cost via MACs; and \(iv\) inference efficiency via forward\-pass latency and throughput, measured over 100 timed runs at batch size 32 with sequence length 64 on a single NVIDIA H100 80GB GPU\.

##### LoRA recovery\.

LoRA adapters with rankr=8r=8and scaling factorα=16\\alpha=16are applied to the four attention projections of every transformer layer\. A dropout rate of 0\.05 is applied to the LoRA matrices\. Recovery fine\-tuning is performed on 25,000 samples of the Alpaca instruction\-tuning dataset\[[25](https://arxiv.org/html/2607.22587#bib.bib18)\]for 2 epochs, with a learning rate of10−410^\{\-4\}, cosine schedule\. Extending LoRA to MLP projections is left for future work, as attention\-only recovery was sufficient to achieve the best results at all tested ratios\.

### 4\.2Language Modeling Perplexity

Table 2:WikiText\-2 perplexity of pruning methods across two models and three pruning ratios\. Best results inbold\.RatioMethodRecoveryLLaMA\-7BVicuna\-7B0%Dense–5\.476\.7820%FLAPw/o tune7\.529\.50bias7\.119\.04w/o tune7\.068\.81TriSPLoRA6\.807\.7730%FLAPw/o tune9\.8712\.72bias8\.6111\.38w/o tune9\.0310\.95TriSPLoRA8\.059\.1050%FLAPw/o tune29\.4336\.86bias16\.0322\.45w/o tune27\.6331\.67TriSPLoRA13\.4414\.82Table[2](https://arxiv.org/html/2607.22587#S4.T2)reports WikiText\-2 perplexity across two models and three pruning ratios\. At all ratios and across both models, TriSP with LoRA recovery achieves the lowest perplexity\. On LLaMA\-7B, TriSP \+ LoRA reaches 6\.80 at 20%, 8\.05 at 30%, and 13\.44 at 50%, consistently outperforming both FLAP without recovery and FLAP with bias compensation\. On Vicuna\-7B at 50%, TriSP \+ LoRA achieves 14\.82 compared to 22\.45 for FLAP \+ bias, a 34% relative improvement\.

### 4\.3Zero\-Shot Downstream Tasks

Table 3:Zero\-shot commonsense reasoning accuracy \(%\) on LLaMA\-2\-7B at 20% and 50% pruning\. Best results inbold\.RatioMethodPIQAHellaSwagWinoGrandeARC\-eARC\-cOBQAAvg\.0%LLaMA\-7B78\.1374\.0669\.3875\.4643\.0942\.6063\.7920%LLM\-Pruner w/o tune74\.4361\.4659\.7568\.4334\.6439\.4056\.35LLM\-Pruner w/ LoRA75\.6364\.5761\.1772\.5238\.5740\.2058\.78FLAP w/o bias72\.8061\.5164\.0160\.3129\.6138\.8054\.51FLAP w/ bias73\.7263\.1464\.5664\.4432\.1740\.0056\.34TriSP w/o tune75\.9067\.9765\.6770\.5436\.4339\.6059\.35TriSP w/ LoRA76\.7768\.2066\.8573\.0240\.0241\.0060\.9850%LLM\-Pruner w/o tune53\.4827\.0348\.5430\.1820\.6524\.8034\.11LLM\-Pruner w/ LoRA62\.4037\.5651\.2244\.4422\.1028\.4041\.02FLAP w/o bias60\.0142\.5650\.7536\.7822\.9532\.0040\.84FLAP w/ bias60\.9440\.6253\.0443\.9822\.7033\.6042\.48TriSP w/o tune61\.4842\.4251\.6245\.9225\.0932\.2043\.12TriSP w/ LoRA67\.4648\.1753\.0456\.0229\.7834\.6048\.18

Table[3](https://arxiv.org/html/2607.22587#S4.T3)evaluates zero\-shot accuracy on six commonsense reasoning benchmarks for LLaMA\-2\-7B\. At 20% pruning, TriSP \+ LoRA achieves the highest average accuracy \(60\.98%\), outperforming LLM\-Pruner \+ LoRA \(58\.78%\), FLAP \+ bias \(56\.34%\), and FLAP without bias \(54\.51%\)\. TriSP \+ LoRA leads on all six benchmarks, with particularly strong margins on HellaSwag \(68\.20 vs\. 64\.57 for LLM\-Pruner\) and WinoGrande \(66\.85 vs\. 64\.56 for FLAP\)\.

At 50% pruning, all methods degrade substantially, but TriSP \+ LoRA retains the best average performance \(48\.18%\), with notable margins over all baselines, including LLM\-Pruner \+ LoRA on PIQA \(\+5\.06\), ARC\-e \(\+11\.58\), and HellaSwag \(\+10\.61\)\.

### 4\.4Generalization to Other Architectures

To validate that TriSP generalizes beyond the LLaMA\-2 family, we evaluate on DeepSeek\-7B and Mistral\-7B, which differ in several architectural dimensions detailed in Table[1](https://arxiv.org/html/2607.22587#S4.T1)\.

Table 4:Perplexity and zero\-shot accuracy \(%\) of pruning methods for DeepSeek\-7B\.RatioMethodWikiText\-2↓\\downarrowPTB↓\\downarrowPIQAHellaSwagWinoGrandeARC\-eARC\-cOBQAAvg\.↑\\uparrow0%Dense6\.849\.2778\.5173\.8570\.4874\.5842\.3245\.2064\.1620%LLM\-Pruner w/ LoRA9\.7914\.9677\.2066\.1263\.1470\.4136\.2641\.0059\.02FLAP w/ bias8\.3011\.5876\.3966\.5865\.5167\.8935\.8443\.8059\.34TriSP w/o tune8\.5312\.1977\.3170\.3264\.4068\.9837\.6343\.6060\.37TriSP w/ LoRA8\.2711\.9277\.8069\.8765\.0472\.6041\.8144\.6061\.9530%LLM\-Pruner w/ LoRA15\.8938\.2774\.3258\.1056\.9964\.0231\.7438\.4053\.93FLAP w/ bias10\.0013\.9873\.3959\.8563\.0662\.0433\.4540\.4055\.37TriSP w/o tune10\.6815\.5475\.1965\.5860\.5464\.8634\.9842\.6057\.29TriSP w/ LoRA9\.8214\.3475\.6865\.4362\.0468\.8636\.5243\.2058\.62

Table 5:Perplexity and zero\-shot accuracy \(%\) of pruning methods for Mistral\-7B\.RatioMethodWikiText\-2↓\\downarrowPTB↓\\downarrowPIQAHellaSwagWinoGrandeARC\-eARC\-cOBQAAvg\.↑\\uparrow0%Dense5\.2526\.9680\.2578\.7574\.6680\.2249\.8346\.6068\.3920%LLM\-Pruner w/ LoRA8\.1430\.8175\.6866\.9763\.3871\.1741\.4741\.4060\.01FLAP w/ bias6\.4642\.1469\.1052\.2266\.5466\.7133\.7940\.8054\.86TriSP w/o tune6\.1534\.3974\.3261\.9469\.0670\.8339\.5141\.4059\.51TriSP w/ LoRA6\.3532\.8376\.8867\.4967\.5674\.8745\.8243\.2062\.6430%LLM\-Pruner w/ LoRA11\.7252\.6272\.5859\.2058\.0963\.5133\.8738\.2054\.24FLAP w/ bias7\.7857\.2265\.5644\.7861\.6459\.8528\.8437\.6049\.71TriSP w/o tune7\.2643\.3167\.7951\.9764\.2565\.7832\.9439\.2053\.66TriSP w/ LoRA7\.0438\.2772\.4761\.2965\.6769\.5338\.1441\.0058\.02

##### DeepSeek\-7B\.

Table[4](https://arxiv.org/html/2607.22587#S4.T4)reports perplexity and zero\-shot accuracy at 20% and 30% pruning\. TriSP \+ LoRA achieves the best WikiText\-2 perplexity at both ratios \(8\.27 at 20%, 9\.82 at 30%\) and the highest average zero\-shot accuracy \(61\.95% at 20%, 58\.62% at 30%\)\. At 20%, TriSP \+ LoRA improves over FLAP \+ bias by 2\.62 points in average accuracy and over LLM\-Pruner \+ LoRA by 2\.93 points\. The gap widens at 30%, where TriSP leads FLAP \+ bias by 3\.26 points\.

##### Mistral\-7B\.

Table[5](https://arxiv.org/html/2607.22587#S4.T5)shows results on Mistral\-7B\. TriSP \+ LoRA achieves the best average accuracy at both ratios \(62\.64% at 20%, 58\.02% at 30%\)\. At 30%, TriSP outperforms FLAP \+ bias by 8\.31 points \(58\.02 vs\. 49\.71\) and LLM\-Pruner \+ LoRA by 3\.78 points\. Notably, TriSP without LoRA achieves the lowest WikiText\-2 perplexity at 20% \(6\.15 vs\. 6\.35 for TriSP \+ LoRA\), however, LoRA still improves average downstream accuracy \(62\.64 vs\. 59\.51\)\.

### 4\.5Inference Efficiency

Table 6:Inference statistics of compressed LLaMA\-2\-7B\. Measured with sequence length 64, batch size 32 on a single NVIDIA H100\.MethodRatioParamsMACsMemoryLatency \(ms\)Tokens/sLLaMA\-2\-7B0%6\.74B422\.85G12885 MiB49\.8341\.1kLLM\-Pruner20%5\.47B341\.67G10433 MiB50\.47 \(≈\\approx0%\)40\.6kFLAP20%5\.44B340\.06G10424 MiB49\.40 \(≈\\approx0%\)41\.5kTriSP \(Ours\)20%5\.43B339\.35G10462 MiB40\.94 \(↓\\downarrow18%\)50\.0kLLM\-Pruner50%3\.50B215\.62G6677 MiB26\.11\(↓\\downarrow48%\)78\.4kFLAP50%3\.50B215\.67G6726 MiB34\.02 \(↓\\downarrow32%\)60\.2kTriSP \(Ours\)50%3\.49B214\.82G6751 MiB27\.46 \(↓\\downarrow45%\)74\.6kTable[6](https://arxiv.org/html/2607.22587#S4.T6)reports inference statistics for LLaMA\-2\-7B models\. At 20% pruning, all three methods achieve comparable parameter counts \(∼\\sim5\.4B\) and MACs \(∼\\sim340G\)\. The key differentiator is latency: TriSP achieves an 18% latency reduction \(40\.94 ms vs\. 49\.83 ms for dense\) and a 22% throughput improvement over the dense baseline \(50\.0k tokens/s vs\. 41\.1k\), while both LLM\-Pruner \(50\.47 ms\) and FLAP \(49\.40 ms\) show weak speedup\. This difference is likely because TriSP rounds MLP dimensions to multiples of 64, ensuring efficient GPU memory access and matrix computation, while LLM\-Pruner and FLAP produce dimensions that do not align with the hardware execution granularity\.

At 50%, LLM\-Pruner achieves the lowest latency \(26\.11 ms,↓\\downarrow48%\), with TriSP close behind \(27\.46 ms,↓\\downarrow45%\), while FLAP lags at 34\.02 ms \(↓\\downarrow32%\)\. TriSP delivers 74\.6k tokens/s, an 82% throughput improvement over the dense baseline, compared to 78\.4k \(↑\\uparrow91%\) for LLM\-Pruner and 60\.2k \(↑\\uparrow46%\) for FLAP\. Memory savings scale proportionally with parameter reduction, dropping from 12,885 MiB to∼\\sim6,700 MiB at 50%\.

### 4\.6Ablation Studies

We systematically examine three components of the TriSP pipeline: the recovery strategy, the importance metric, and the allocation strategy\.

#### 4\.6\.1Effect of Recovery Strategy

20%30%40%50%0101020203030Pruning RatioPerplexity \(↓\\downarrow\)TriSP \(no recovery\)TriSP \+ biasTriSP \+ LoRAFigure 2:Vicuna\-7B: WikiText\-2 perplexity under TriSP pruning with different recovery strategies\.Figure[2](https://arxiv.org/html/2607.22587#S4.F2)compares three recovery strategies for TriSP pruning on Vicuna\-7B: no recovery, bias compensation, and LoRA fine\-tuning\. LoRA recovery performs best across all pruning ratios, reducing perplexity from 8\.81 to 7\.77 at 20%, from 10\.95 to 9\.10 at 30%, from 15\.83 to 10\.96 at 40%, and from 31\.67 to 14\.82 at 50%\. Bias compensation provides intermediate improvements at moderate ratios \(e\.g\., 9\.65 at 30% and 13\.02 at 40%\), but its advantage over no recovery diminishes relative to LoRA as compression increases\. At 50%, the gap between LoRA and bias widens sharply \(14\.82 vs\. 20\.79\)\. At 40%, LoRA reduces perplexity by 30\.7% relative to no recovery, while bias compensation achieves only 17\.8%\. This confirms that LoRA’s input\-dependent, rank\-rrcorrection subsumes and exceeds what a rank\-zero fixed bias term can provide\.

#### 4\.6\.2Importance Metric Ablation

Table[7](https://arxiv.org/html/2607.22587#S4.T7)isolates the contribution of each component in the TriSP metric by evaluating WIFN alone, GS alone, and the combined TriSP score on Vicuna\-7B\. At all pruning ratios, the combined TriSP metric outperforms either component in isolation: atp=0\.4p=0\.4, TriSP achieves 15\.83 compared to 16\.12 for WIFN and 17\.14 for GS\. The gap for the GS component grows from 0\.10 atp=0\.2p=0\.2to 1\.31 atp=0\.4p=0\.4, suggesting that gradient sensitivity alone overprunes certain channels that WIFN’s activation\-weighted magnitude signal protects\.

Table 7:Metric ablation on Vicuna\-7B: WikiText\-2 perplexity using individual importance signals vs\. the full TriSP metric\.Metricp=0\.2p=0\.2p=0\.3p=0\.3p=0\.4p=0\.4WIFN only8\.9211\.2616\.12GS only8\.9112\.1717\.14TriSP \(ours\)8\.8110\.9515\.83
#### 4\.6\.3Allocation Strategy Ablation

0\.20\.30\.40\.50202040406060Pruning RatioWikiText\-2 Perplexity \(↓\\downarrow\)UL\-UM \(uniform\)AL\-AM \(adaptive\)Δ\\Delta\(Gap\)Figure 3:WikiText\-2 perplexity: uniform \(UL\-UM\) vs\. adaptive \(AL\-AM\) allocation on LLaMA\-2\-7B\.Figure[3](https://arxiv.org/html/2607.22587#S4.F3)compares uniform layer\-module allocation \(UL\-UM\), which assigns the same pruning ratio to every layer and module type, against adaptive layer\-module allocation \(AL\-AM\) on LLaMA\-2\-7B\. At mild pruning \(p≤0\.3p\\leq 0\.3\), the gap is negligible \(Δ≤0\.34\\Delta\\leq 0\.34perplexity points\)\. Beyond 30%, the benefit grows sharply: atp=0\.4p=0\.4, AL\-AM reduces perplexity by 8\.27 points \(13\.84 vs\. 22\.11\), and atp=0\.5p=0\.5, the gap reaches 25\.83 points \(27\.63 vs\. 53\.46\)\. This demonstrates that adaptive budget redistribution becomes critical at aggressive pruning, where uniform allocation causes catastrophic over\-pruning in sensitive layers\.

### 4\.7Discussion

##### Why TriSP outperforms FLAP\.

FLAP uses activation variance as a statistical proxy for importance; TriSP includes\|Wi​j\|⋅G¯i​j\|W\_\{ij\}\|\\cdot\\bar\{G\}\_\{ij\}from the Taylor expansion, which directly measures first\-order loss sensitivity\. This explains why TriSP without recovery already outperforms FLAP without recovery at all ratios: the metric itself makes better pruning decisions, not just the recovery strategy\.

##### Why TriSP outperforms LLM\-Pruner\.

LLM\-Pruner models structural dependencies through dependency graphs, while TriSP treats channels independently\. Despite this, TriSP without recovery achieves higher average zero\-shot accuracy than LLM\-Pruner without recovery \(59\.35 vs\. 56\.35 at 20%\) and even surpasses LLM\-Pruner with LoRA \(58\.78\), indicating that the TriSP importance criterion itself makes better pruning decisions before any recovery is applied\. LLM\-Pruner’s dependency graph constrains which structures can be removed together, which may force the removal of important channels that happen to be coupled with unimportant ones\.

##### Recovery adequacy\.

At mild pruning \(20%\), bias compensation and LoRA yield similar improvements, because few channels are removed and the rank\-zero mean approximation is sufficient\. At aggressive pruning \(40–50%\), LoRA’s rank\-rrinput\-dependent correction vastly outperforms the fixed bias, because a fixed offset \(bias\) cannot capture the input\-dependent variation lost when many channels are removed\.

##### Practical deployment recommendations\.

At 20–30% pruning, perplexity increases by 1–3 points and zero\-shot accuracy drops by 2–6 points, with 14–24% latency reduction\. This operating range is suitable for most applications where modest quality loss is acceptable for cost reduction\. At 50% pruning, perplexity roughly doubles and zero\-shot accuracy drops by∼\\sim15 points, with 42–45% latency reduction\. This aggressive regime is appropriate only for latency\-critical applications with high quality tolerance\.

## 5Conclusion

We presented TriSP \(Tri\-Signal Structured Pruning\), a method that combines activation\-weighted magnitude with gradient sensitivity via a geometric mean to produce channel\-level importance scores\. Paired with adaptive layer\-module \(AL\-AM\) allocation and LoRA recovery, TriSP achieves the lowest perplexity and highest downstream accuracy across four models \(LLaMA\-2\-7B, Vicuna\-7B, DeepSeek\-7B, Mistral\-7B\) at all tested pruning ratios, while delivering 22–82% throughput improvements through hardware\-aligned pruning\.

The three ablation studies confirm that each component contributes independently: the TriSP metric outperforms either WIFN or GS alone, AL\-AM allocation becomes critical at aggressive pruning \(reducing perplexity by 25\.83 points at 50%\), and LoRA recovery outperforms bias compensation by an increasing margin as pruning intensifies \(30\.7% vs\. 17\.8% perplexity reduction at 40%\)\. The systematic comparison between bias compensation and LoRA recovery confirms that input\-dependent correction is essential at aggressive compression ratios\.

Current limitations include the restriction to decoder\-only transformer architectures at 7B scale, and evaluation limited to English\-language benchmarks\. Future work includes extending LoRA to MLP projections for better recovery, scaling to larger models \(30B\+\), evaluating on broader multilingual benchmarks, and applying TriSP to vision\-language models \(VLMs\)\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p1.1)\.
- \[2\]Y\. An, X\. Zhao, T\. Yu, M\. Tang, and J\. Wang\(2024\)Fluctuation\-based adaptive structured pruning for large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 10865–10873\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.22587#S2.SS2.p1.1),[§3\.2](https://arxiv.org/html/2607.22587#S3.SS2.p1.1),[§3\.3](https://arxiv.org/html/2607.22587#S3.SS3.p1.1),[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px2.p1.1)\.
- \[3\]V\. Ashish\(2017\)Attention is all you need\.Advances in neural information processing systems30,pp\. I\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p1.1)\.
- \[4\]S\. Ashkboos, M\. L\. Croci, M\. G\. d\. Nascimento, T\. Hoefler, and J\. Hensman\(2024\)SliceGPT: compress large language models by deleting rows and columns\.arXiv preprint arXiv:2401\.15024\.Cited by:[§2\.2](https://arxiv.org/html/2607.22587#S2.SS2.p2.1)\.
- \[5\]X\. Bi, D\. Chen, G\. Chen, S\. Chen, D\. Dai, C\. Deng, H\. Ding, K\. Dong, Q\. Du, Z\. Fu,et al\.\(2024\)Deepseek llm: scaling open\-source language models with longtermism\.arXiv preprint arXiv:2401\.02954\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px1.p1.1)\.
- \[6\]Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px3.p1.1)\.
- \[7\]W\. Chiang, Z\. Li, Z\. Lin, Y\. Sheng, Z\. Wu, H\. Zhang, L\. Zheng, S\. Zhuang, Y\. Zhuang, J\. E\. Gonzalez, I\. Stoica, and E\. P\. Xing\(2023\)Vicuna: an open\-source chatbot impressing gpt\-4 with 90%\* chatgpt quality\.LMSYS blog\.Cited by:[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px1.p1.1)\.
- \[8\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px3.p1.1)\.
- \[9\]R\. J\. Das, M\. Sun, L\. Ma, and Z\. Shen\(2023\)Beyond size: how gradients shape pruning decisions in large language models\.arXiv preprint arXiv:2311\.04902\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.22587#S2.SS3.p1.1)\.
- \[10\]E\. Frantar and D\. Alistarh\(2023\)SparseGPT: massive language models can be accurately pruned in one\-shot\.InInternational Conference on Machine Learning,pp\. 10323–10337\.Cited by:[§2\.1](https://arxiv.org/html/2607.22587#S2.SS1.p2.1)\.
- \[11\]E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\(2022\)Gptq: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§2\.1](https://arxiv.org/html/2607.22587#S2.SS1.p2.1)\.
- \[12\]B\. Hassibi and D\. Stork\(1992\)Second order derivatives for network pruning: optimal brain surgeon\.Advances in neural information processing systems5\.Cited by:[§2\.1](https://arxiv.org/html/2607.22587#S2.SS1.p2.1)\.
- \[13\]Y\. He and L\. Xiao\(2023\)Structured pruning for deep convolutional neural networks: a survey\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(5\),pp\. 2900–2919\.Cited by:[§2](https://arxiv.org/html/2607.22587#S2.p1.1)\.
- \[14\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.ICLR1\(2\),pp\. 3\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p4.1),[§3\.5](https://arxiv.org/html/2607.22587#S3.SS5.p1.4)\.
- \[15\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier,et al\.\(2023\)Mistral 7b\.arXiv preprint arXiv:2310\.06825\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px1.p1.1)\.
- \[16\]S\. Kolawole, L\. Dery, J\. Kagy, V\. Smith, G\. Neubig, and A\. Talwalkar\(2024\)Everybody prune now: structured pruning of llms with only forward passes\.arXiv preprint arXiv:2402\.05406\.Cited by:[§2\.2](https://arxiv.org/html/2607.22587#S2.SS2.p1.1)\.
- \[17\]Q\. Le, E\. Diao, Z\. Wang, X\. Wang, J\. Ding, L\. Yang, and A\. Anwar\(2025\)Probe pruning: accelerating llms through dynamic pruning via model probing\.arXiv preprint arXiv:2502\.15618\.Cited by:[§2\.2](https://arxiv.org/html/2607.22587#S2.SS2.p1.1)\.
- \[18\]L\. Li and L\. Guo\(2024\)Dynamic low\-rank adaptation based pruning algorithm for large language models\.In2024 7th International Conference on Pattern Recognition and Artificial Intelligence \(PRAI\),pp\. 1094–1099\.Cited by:[§2\.1](https://arxiv.org/html/2607.22587#S2.SS1.p1.1)\.
- \[19\]G\. Ling, Z\. Wang, Y\. Yan, and Q\. Liu\(2024\)SlimGPT: layer\-wise structured pruning for large language models\.Advances in Neural Information Processing Systems37,pp\. 107112–107137\.Cited by:[§2\.1](https://arxiv.org/html/2607.22587#S2.SS1.p2.1)\.
- \[20\]X\. Ma, G\. Fang, and X\. Wang\(2023\)LLM\-pruner: on the structural pruning of large language models\.Advances in Neural Information Processing Systems36,pp\. 21702–21720\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.22587#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px2.p1.1)\.
- \[21\]X\. Men, M\. Xu, Q\. Zhang, Q\. Yuan, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. Chen\(2025\)Shortgpt: layers in large language models are more redundant than you expect\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 20192–20204\.Cited by:[§2\.2](https://arxiv.org/html/2607.22587#S2.SS2.p2.1)\.
- \[22\]T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal\(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2381–2391\.Cited by:[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px3.p1.1)\.
- \[23\]K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi\(2021\)Winogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px3.p1.1)\.
- \[24\]M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. Kolter\(2023\)A simple and effective pruning approach for large language models\.arXiv preprint arXiv:2306\.11695\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.22587#S2.SS3.p1.1),[3rd item](https://arxiv.org/html/2607.22587#S3.I1.i3.p1.1)\.
- \[25\]R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto\(2023\)Alpaca: a strong, replicable instruction\-following model\.Stanford Center for Research on Foundation Models3\(6\),pp\. 7\.Cited by:[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px4.p1.3)\.
- \[26\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px1.p1.1)\.
- \[27\]Y\. Yang, K\. Zhen, B\. Ganesh, A\. Galstyan, G\. Huybrechts, M\. M"uller, J\. M\. K"ubler, R\. V\. Swaminathan, A\. Mouchtaris, S\. B\. Bodapati,et al\.\(2025\)Wanda\+\+: pruning large language models via regional gradients\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 4321–4333\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p4.1),[§2\.3](https://arxiv.org/html/2607.22587#S2.SS3.p1.1)\.
- \[28\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\(2019\)Hellaswag: can a machine really finish your sentence?\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 4791–4800\.Cited by:[§4\.1](https://arxiv.org/html/2607.22587#S4.SS1.SSS0.Px3.p1.1)\.
- \[29\]M\. Zhang, H\. Chen, C\. Shen, Z\. Yang, L\. Ou, X\. Yu, and B\. Zhuang\(2024\)LoRAPrune: structured pruning meets low\-rank parameter\-efficient fine\-tuning\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 3013–3026\.Cited by:[§2\.1](https://arxiv.org/html/2607.22587#S2.SS1.p1.1)\.
- \[30\]L\. Zhong, F\. Wan, R\. Chen, X\. Quan, and L\. Li\(2025\)BlockPruner: fine\-grained pruning for large language models\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 5065–5080\.Cited by:[§2\.2](https://arxiv.org/html/2607.22587#S2.SS2.p2.1)\.
- \[31\]X\. Zhu, J\. Li, Y\. Liu, C\. Ma, and W\. Wang\(2024\)A survey on model compression for large language models\.Transactions of the Association for Computational Linguistics12,pp\. 1556–1577\.Cited by:[§1](https://arxiv.org/html/2607.22587#S1.p1.1)\.

## Appendix APerplexity Scaling

Figure[4](https://arxiv.org/html/2607.22587#A1.F4)visualizes the perplexity scaling behavior on LLaMA\-2\-7B across pruning ratios\. Both FLAP and TriSP degrade gracefully at moderate ratios, but diverge sharply at 50%: TriSP \+ LoRA maintains 13\.44 while FLAP \+ bias rises to 16\.03\. TriSP without any recovery \(27\.63\) already outperforms FLAP without recovery \(29\.43\)\.

02020303050505510101515202025253030Pruning Ratio \(%\)WikiText\-2 Perplexity \(↓\\downarrow\)FLAP w/o biasFLAP w/ biasTriSP w/o tuneTriSP \+ LoRAFigure 4:WikiText\-2 perplexity scaling with pruning ratio on LLaMA\-2\-7B\. TriSP \+ LoRA maintains the lowest perplexity at all ratios, with the gap widening significantly at 50% pruning \(13\.44 vs\. 16\.03 for FLAP \+ bias\)\.
## Appendix BInference Throughput

Figure[5](https://arxiv.org/html/2607.22587#A2.F5)reports inference throughput for compressed LLaMA\-2\-7B models\. At 20% pruning, TriSP achieves 50\.0k tokens/s \(↑\\uparrow22%\), while LLM\-Pruner and FLAP show no meaningful speedup\. At 50%, TriSP delivers 74\.6k tokens/s with 45% latency reduction\.

DenseLLM\-P 20%FLAP 20%TriSP 20%LLM\-P 50%FLAP 50%TriSP 50%0202040406060808041\.141\.140\.640\.641\.541\.5505078\.478\.460\.260\.274\.674\.6Throughput \(kTokens/s\)Figure 5:Inference throughput of LLaMA\-2\-7B \(seq=64, batch=32, H100\)\. At 20% pruning, TriSP achieves 50\.0k tokens/s \(↑\\uparrow22%\) while LLM\-Pruner and FLAP show no meaningful speedup\. At 50%, TriSP delivers 74\.6k tokens/s \(↑\\uparrow82%\) with 45% latency reduction\.
## Appendix CDeepSeek\-7B Inference Analysis

Table[8](https://arxiv.org/html/2607.22587#A3.T8)extends the inference analysis to DeepSeek\-7B\. TriSP delivers the lowest latency at 20% and 30%\. Figure[6](https://arxiv.org/html/2607.22587#A3.F6)shows that the throughput gap widens with batch size, and Table[9](https://arxiv.org/html/2607.22587#A3.T9)quantifies the speedup at sequence length 512\.

Table 8:Statistics of the compressed DeepSeek\-7B model\. Inference is conducted with a sequence of 64 tokens at batch size 32 on a single NVIDIA H100\.MethodRatioParamsMACsMemoryLatency \(ms\)Tokens/sDeepSeek\-7B0%6\.91B415\.40G13181 MiB47\.9042\.8kLLM\-Pruner20%5\.72B339\.29G10912 MiB46\.26 \(↓\\downarrow3%\)44\.3kFLAP20%5\.70B337\.75G10914 MiB45\.13 \(↓\\downarrow6%\)45\.4kTriSP \(Ours\)20%5\.69B337\.04G10917 MiB41\.00 \(↓\\downarrow14%\)50\.0kLLM\-Pruner30%5\.13B301\.24G9778 MiB43\.30 \(↓\\downarrow10%\)47\.3kFLAP30%5\.09B298\.93G9753 MiB44\.15 \(↓\\downarrow8%\)46\.4kTriSP \(Ours\)30%5\.08B298\.16G9797 MiB36\.47 \(↓\\downarrow24%\)56\.2kLLM\-Pruner50%3\.87B221\.12G7391 MiB26\.45\(↓\\downarrow45%\)77\.4kFLAP50%3\.88B221\.20G7442 MiB35\.30 \(↓\\downarrow26%\)58\.0kTriSP \(Ours\)50%3\.87B220\.50G7470 MiB27\.97 \(↓\\downarrow42%\)73\.2k

202^\{0\}232^\{3\}242^\{4\}252^\{5\}262^\{6\}404060608080Batch SizeThroughput \(kTokens/s\)DenseTriSP 30%TriSP 50%Figure 6:Throughput vs\. batch size for TriSP\-pruned DeepSeek\-7B at sequence length 512\. Structured pruning yields consistent throughput gains that scale with batch size in the compute\-bound regime\.Table 9:Speedup of TriSP\-pruned DeepSeek\-7B relative to the dense baseline at sequence length 512\.Batch SizeTriSP 30%TriSP 50%11\.26×\\times1\.53×\\times81\.35×\\times1\.79×\\times161\.35×\\times1\.77×\\times321\.35×\\times1\.79×\\times641\.36×\\times1\.81×\\times
## Appendix DPer\-Layer Retention under AL\-AM

Figure[7](https://arxiv.org/html/2607.22587#A4.F7)visualizes the per\-layer retention ratio under AL\-AM allocation for LLaMA\-2\-7B\. At mild pruning \(p=0\.1p=0\.1\), the allocation is nearly uniform\. As the ratio increases, a U\-shaped pattern emerges: the first and last layers are increasingly preserved while middle layers are pruned most aggressively\.

02244668810101212141416161818202022222424262628283030111\.51\.5222\.52\.533Layer indexRetention ratiop=0\.1p=0\.1p=0\.2p=0\.2p=0\.3p=0\.3p=0\.4p=0\.4Figure 7:Per\-layer retention ratio under AL\-AM allocation for LLaMA\-2\-7B at four pruning ratios\. Values above the dashed line \(1\.01\.0\) indicate layers where more parameters are retained relative to a uniform baseline\. At mild pruning \(p=0\.1p=0\.1\), the allocation is nearly uniform; as the ratio increases, the U\-shaped pattern intensifies, with first and last layers increasingly preserved while middle layers \(12–22\) are pruned most aggressively\.
## Appendix EQualitative Generation Examples

TableLABEL:tab:qualitativepresents generated text from the dense LLaMA\-2\-7B baseline and TriSP\-pruned models \(20% and 30% with LoRA recovery\) using greedy decoding with a repetition penalty\. Our experiments demonstrate that the pruned LLaMA\-2\-7B models with 5\.43B and 4\.79B parameters, obtained through our TriSP pruning approach, are highly effective in retaining general knowledge and producing fluent, coherent text\.