Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization

# Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
Source: [https://arxiv.org/html/2607.22583](https://arxiv.org/html/2607.22583)
###### Abstract

Large Language Models \(LLMs\) have achieved widespread adoption because of their strong reasoning and query\-response capabilities\. However, deploying them in embedded and edge computing environments remains challenging because of strict latency, memory, and energy constraints\. Their large parameter counts and computational demands hinder efficient execution on resource\-constrained platforms\. Although model pruning has emerged as a viable solution for reducing scale while preserving performance, jointly optimizing layers, attention heads, and Multi\-Layer Perceptron \(MLP\) dimensions remains highly complex\. Exhaustively exploring this combined design space is computationally expensive and often leads to local optima or unstable configurations\. To address these limitations, we propose a hardware\-aware, multi\-objective structured pruning framework\. The proposed two\-stage method explicitly targets latency and model size for efficient deployment on edge devices\. In the coarse\-grained stage, multi\-objective depth pruning removes entire attention and MLP blocks to reduce computational load and memory usage\. In the subsequent fine\-grained stage, Parallel Bayesian Optimization \(PBO\) searches for the optimal layer\-wise pruning ratios for pruning under latency constraints, while importance\-based strategies rank the specific components to be pruned within each layer’s allocated budget\. Experimental results show that our approach reduces model complexity with minimal impact on commonsense reasoning tasks and zero\-shot performance\. Our method achieves a favorable trade\-off among accuracy, latency, and model size, making it suitable for edge deployment\. Across multiple LLMs at 37\.5%\\%and 50%\\%pruning ratios, the proposed approach achieves better performance on commonsense reasoning tasks than existing methods while reducing A100 latency to 73\.8512 s and Jetson Nano latency to 843 s for 100\-token inference\. These results highlight the potential of structured, multi\-objective optimization techniques to bridge the gap between large\-scale LLMs and resource\-constrained embedded platforms\.

## IIntroduction

Language models are not new, but the introduction of transformer architectures has accelerated the development of numerous Large Language Models \(LLMs\)\. LLMs are AI systems designed to understand, generate, and predict human language\. They are widely used for tasks such as speech recognition, machine translation, natural language generation, and optical character recognition\. One well\-known family of LLMs, Generative Pre\-trained Transformers \(GPTs\), powers popular chatbots such as ChatGPT, Claude, and Gemini\. However, a major challenge with these models is their large size, which makes them difficult to deploy on small edge devices or run locally under memory and latency constraints\.

Two primary strategies exist for obtaining small language models\. The first is to train a compact architecture from scratch\. However, this approach requires massive datasets and substantial computational resources\. Such requirements consume considerable energy and are often impractical in mobile and edge systems\[[2](https://arxiv.org/html/2607.22583#bib.bib2)\]\. Alternatively, researchers can compress existing large models to preserve performance at a smaller scale\.

Existing studies have demonstrated that LLMs contain substantial redundancy at different structural levels\. In transformer\-based LLMs, each block is mainly composed of a Multi\-Head Attention \(MHA\) module and a feed\-forward Multi\-Layer Perceptron \(MLP\) module\. MHA captures token\-to\-token interactions through multiple attention heads, whereas the MLP applies position\-wise nonlinear transformations that expand and project hidden representations\. Because these modules contribute differently to accuracy, parameter count, and latency, pruning can be performed at several granularities, ranging from entire transformer blocks to individual attention heads or MLP neurons\.

Pruning methods are generally categorized as structured, semi\-structured, or unstructured\. Unstructured pruning removes individual weights and can achieve high sparsity, but the resulting irregular sparse matrices often require specialized hardware or sparse kernels to deliver practical speedups\. In contrast, structured pruning removes complete architectural units, such as layers, attention heads, channels, or MLP neurons, producing dense sub\-networks that are easier to accelerate on standard edge devices\.

Structured pruning approaches can be further grouped into depth\-wise and width\-wise methods\. Depth\-wise methods remove complete transformer blocks or sub\-blocks, reducing model depth and memory footprint\. Methods such as\[[14](https://arxiv.org/html/2607.22583#bib.bib14)\]and\[[9](https://arxiv.org/html/2607.22583#bib.bib9)\]follow this direction by pruning transformer blocks at a coarse level\. However, coarse pruning alone may ignore the heterogeneous contribution of MHA and MLP components inside retained layers\. Width\-wise methods instead prune intra\-block structures, such as attention heads, neurons, or channels\. Approaches such as\[[5](https://arxiv.org/html/2607.22583#bib.bib5),[1](https://arxiv.org/html/2607.22583#bib.bib1),[18](https://arxiv.org/html/2607.22583#bib.bib18)\]use gradient\-based, activation\-based, or sensitivity\-based scores to identify less important components\. These methods provide finer control, but they often treat candidate components uniformly and do not always account for deployment constraints such as latency and active parameter count\.

Recent methods combine depth\-wise and width\-wise pruning either jointly or sequentially\. Approaches such as\[[8](https://arxiv.org/html/2607.22583#bib.bib8)\]and\[[7](https://arxiv.org/html/2607.22583#bib.bib7)\]prune both block\-level and component\-level structures, but they typically do not explicitly optimize the allocation of layer\-wise pruning ratios to identify configurations with lower latency\. This is important because the same sparsity budget can lead to very different inference behavior depending on how pruning is distributed across layers, MHA heads, and MLP neurons\.

The deployment of LLMs on edge devices with limited compute resources, such as the NVIDIA Jetson Nano or Raspberry Pi, demands pruning strategies that remain compatible with dense linear algebra units\. Structured pruning, which removes entire structural components, produces architecturally coherent models that can realize latency reductions on such hardware\. Unstructured pruning, by contrast, generates irregular sparsity that requires specialized support, e\.g\., Sparse Tensor Cores, to deliver performance gains, which limits its usefulness on mainstream edge accelerators\.

Despite the effectiveness of existing structured pruning methods, two challenges remain\. First, depth\-wise pruning must balance accuracy preservation with deployment\-oriented objectives such as latency and model size\. Second, after a coarse architecture is selected, the remaining pruning budget must be allocated across layers and components to identify configurations that reduce latency\. In this work, we address these issues through a two\-stage framework:

1. 1\.A multi\-objective depth\-pruning stage searches for Pareto\-optimal, layer\-pruned architectures by jointly evaluating output distribution loss and efficiency objectives\.
2. 2\.A parallel Bayesian optimization \(BO\) stage to determine layer\-wise pruning ratios that yield lower hardware latency\.

The contributions of this paper are as follows:

- •We formulate LLM structured pruning as a multi\-objective, two\-stage problem that jointly considers performance and hardware metrics \(latency and throughput\) while benchmarking the model on constrained edge devices\.
- •We introduce a parallel Bayesian optimization approach for allocating layer\-wise pruning ratios to find configurations with lower latency, enabling efficient exploration of sparsity distributions following the multi\-objective stage\.
- •We benchmark multiple Stage 2 importance strategies for subcomponent pruning under the same allocation framework, providing an empirical analysis of their accuracy, perplexity, and latency trade\-offs\.

## IIRelated Work

Recent advances in deep learning have led to the widespread adoption of LLMs like GPT, LLaMA\[[19](https://arxiv.org/html/2607.22583#bib.bib19)\], Qwen\[[24](https://arxiv.org/html/2607.22583#bib.bib24)\], Phi\[[25](https://arxiv.org/html/2607.22583#bib.bib25)\]and Mistral\[[20](https://arxiv.org/html/2607.22583#bib.bib20)\]in various applications\. However, these models’ massive size and computational requirements raise significant concerns regarding memory and latency\. To enable real\-time performance on resource\-constrained edge devices, model compression techniques such as quantization, knowledge distillation, and pruning have been explored\[[3](https://arxiv.org/html/2607.22583#bib.bib3)\]\. Among these, pruning is a critical method for reducing model size and improving energy efficiency while preserving predictive performance

The concept of pruning is well established, with one seminal work being Optimal Brain Damage \(OBD\) by LeCun et al\.\[[4](https://arxiv.org/html/2607.22583#bib.bib4)\], which used second\-order derivatives to identify and remove weights with minimal impact on the objective function\. We compare our approach with three main types of pruning: structured, semi\-structured, and unstructured\.

Early structured pruning approaches include LLM Pruner\[[5](https://arxiv.org/html/2607.22583#bib.bib5)\], a task\-agnostic method that removes non\-essential coupled structures \(dependency groups\) based on gradient information, recovering performance through Low\-Rank Adaptation \(LoRA\) fine\-tuning\. Similarly, Block Pruner performs fine\-grained structured pruning within transformer blocks by pruning both MLP and MHA sub\-components based on component\-level importance\[[9](https://arxiv.org/html/2607.22583#bib.bib9)\]\. Other methods focus on pruning model width and individual columns\[[5](https://arxiv.org/html/2607.22583#bib.bib5),[7](https://arxiv.org/html/2607.22583#bib.bib7)\]\.

SliceGPT utilizes Principal Component Analysis \(PCA\) to remove rows and columns across matrices\[[8](https://arxiv.org/html/2607.22583#bib.bib8)\]\. However, such approaches often require additional parameters or rigid structural adjustments to maintain compatibility\. The 2SSP framework\[[7](https://arxiv.org/html/2607.22583#bib.bib7)\]proposes a two\-stage approach: first, removing neurons from Feed\-Forward Networks \(FFNs\) based on output magnitude; and second, iteratively removing attention sub\-modules\. However, this sequential process risks missing the global optimum between the two stages\. In a comparable approach, Blockpruner\[[9](https://arxiv.org/html/2607.22583#bib.bib9)\]targets both attention heads and MLPs using component\-level importance scoring\. Unlike 2SSP, Blockpruner applies block\-level importance to prune both MLP and MHA components but fails to account for the importance of pruning at the sub\-component level\[[9](https://arxiv.org/html/2607.22583#bib.bib9)\]\. Recently, Coarse\-to\-Fine Structured Pruning \(CFSP\)\[[23](https://arxiv.org/html/2607.22583#bib.bib23)\]introduced a coarse\-to\-fine structured pruning framework that leverages activation information at both inter\-block and intra\-block levels to guide the pruning process\. In contrast, our approach explicitly formulates pruning as a multi\-objective optimization problem centered on deployment\-oriented criteria, such as latency and model size\. Rather than relying solely on activation\-based importance, our method identifies and refines multiple Pareto\-optimal coarse configurations\. This allows for the exploration of a more diverse solution space along the Pareto front\. These candidate solutions are then further optimized through a second\-stage importance calculation, ensuring a superior accuracy\-efficiency trade\-off tailored for edge deployment\.

Unstructured pruning methods prune individual weights using activation\-aware criteria, maintaining accuracy at high sparsity\. In practice, however, the irregular sparsity they induce is not efficiently supported by standard GPU kernels, so inference speedups on general\-purpose hardware are frequently limited\[[6](https://arxiv.org/html/2607.22583#bib.bib6),[16](https://arxiv.org/html/2607.22583#bib.bib16)\]\.

![Refer to caption](https://arxiv.org/html/2607.22583v1/pruning_framework.jpg)Figure 1:Overview of the proposed multi\-objective structured pruning framework, combining the full workflow for LLM compression with the two\-stage coarse\-to\-fine pruning process used for latency\- and model\-size\-aware optimization\.Although the aforementioned approaches are effective,they do not explicitly consider performance–efficiency trade\-offsand typically address the pruning problem in a single stage rather than in a hierarchical manner\. As a result, these methods may overlook globally optimal configurations when jointly optimizing model size, latency, and accuracy\. To overcome these limitations, we propose a multi\-objective optimization strategy that identifies Pareto\-optimal solutions and subsequently applies importance\-based pruning to refine sub\-components\. This explicit treatment of performance–efficiency trade\-offs guides Stage 2 toward deployment\-oriented models selected for favorable parameter count and lower latency\. By separating global architecture search from local component refinement, the two\-stage design achieves a more balanced optimization process and a more hardware\-efficient compression pipeline than single\-stage pruning\.

## IIIProposed Methodology

Our proposed methodology uses a global\-to\-local pruning framework that separates macro\-level architectural search from finer\-grained structural refinement, thereby simplifying the optimization process\. Figure[1](https://arxiv.org/html/2607.22583#S2.F1)provides an overview of the methodology with a numerical example of a 4\-block LLM at 50%\\%pruning\. In this example, Stage 1 prunes up to 50%\\%of the model parameters and generates multiple Pareto\-optimal solutions with varying parameter counts\. Stage 2 then refines these solutions by utilizing parallel Bayesian optimization to allocate the remaining layer\-wise pruning ratios and find the configuration with lower latency\. Next, it performs targeted pruning of sub\-components\. For ease of explanation, we consider the example of MHA heads and MLP neurons within those layers, both of which are evaluated using a defined importance metric i\.e\., 50%\\%minus the pruning already performed in Stage 1, to further reduce model complexity while preserving performance through latency\-aware configuration selection\. Algorithm[1](https://arxiv.org/html/2607.22583#alg1)summarizes the overall procedure\. The details of the two stages are as follows:

In Stage 1, candidate architectures are generated by pruning complete MHA and MLP sub\-blocks at the layer level\. This stage captures global architecture\-level trade\-offs and returns a set of Pareto\-optimal candidates rather than a single solution\. In Stage 2, the remaining sparsity budget is distributed across layers using Bayesian optimization\. Each Bayesian\-optimization candidate specifies layer\-wise pruning ratios for specific layer, and candidates are evaluated in parallel to reduce search time\. The selected allocation is then applied alongside an importance scoring metric to remove the lowest\-ranked sub\-components\.

As the two main and important components in LLM are Multi\-Head Attention \(MHA\) and Multi\-Layer Perceptron \(MLP\), we formalize the notation for these modules\. This notation provides the mathematical basis for the layer\-wise budget allocation and structural pruning strategies detailed in the subsequent sections\.

Algorithm 1Two\-Stage Pruning with Bayesian Layer\-Wise Allocation1:Input:Model

ℳ\\mathcal\{M\}, calibration data

𝒟\\mathcal\{D\}, validation data

𝒟val\\mathcal\{D\}\_\{\\text\{val\}\}, sparsity budget

SS
2:Output:Best pruned model

MbestM\_\{\\text\{best\}\}
3:Stage 1: Multi\-Objective Depth\-Wise Pruning

4:

𝒞←InitPopulation​\(\)\\mathcal\{C\}\\leftarrow\\textsc\{InitPopulation\}\(\)
5:whilestopping criterion is not metdo

6:foreach candidate

c∈𝒞c\\in\\mathcal\{C\}do

7:

Mc←PruneSubBlocks​\(ℳ,c\)M\_\{c\}\\leftarrow\\textsc\{PruneSubBlocks\}\(\\mathcal\{M\},c\)
8:Evaluate

McM\_\{c\}using quality loss and efficiency objectives

9:endfor

10:

𝒫←ParetoFront​\(𝒞\)\\mathcal\{P\}\\leftarrow\\textsc\{ParetoFront\}\(\\mathcal\{C\}\)
11:

𝒞←Evolve​\(𝒫\)\\mathcal\{C\}\\leftarrow\\textsc\{Evolve\}\(\\mathcal\{P\}\)
12:endwhile

13:

𝒮←SelectParetoCandidates​\(𝒫\)\\mathcal\{S\}\\leftarrow\\textsc\{SelectParetoCandidates\}\(\\mathcal\{P\}\)
14:Stage 2: Parallel Bayesian Allocation and Importance Benchmarking

15:

𝒜←InitBayesianOptimizer​\(S\)\\mathcal\{A\}\\leftarrow\\textsc\{InitBayesianOptimizer\}\(S\)
16:foreach coarse candidate

Mc∈𝒮M\_\{c\}\\in\\mathcal\{S\}do

17:whileBayesian optimization budget is not exhausteddo

18:Propose a batch of layer\-wise allocations

\{a1,…,am\}\\\{a\_\{1\},\\ldots,a\_\{m\}\\\}
19:Evaluate allocations in parallel to identify lower\-latency configurations

20:foreach allocation

aja\_\{j\}and each importance method

ggdo

21:Rank MHA heads and MLP neurons using

ggon

𝒟\\mathcal\{D\}
22:

Mc,j,g←PruneByAllocation​\(Mc,aj,g\)M\_\{c,j,g\}\\leftarrow\\textsc\{PruneByAllocation\}\(M\_\{c\},a\_\{j\},g\)
23:Score

Mc,j,gM\_\{c,j,g\}on

𝒟val\\mathcal\{D\}\_\{\\text\{val\}\}using quality and latency

24:endfor

25:Update Bayesian optimizer with observed scores

26:endwhile

27:endfor

28:

Mbest←SelectBestTradeoff​\(\)M\_\{\\text\{best\}\}\\leftarrow\\textsc\{SelectBestTradeoff\}\(\)
29:return

MbestM\_\{\\text\{best\}\}

### III\-AMulti\-Head Attention

Multi\-Head Attention \(MHA\) is the sequence\-context modeling module of a transformer block\. It projects the input sequence into query, key, and value representations and allows each token representation to aggregate information from other token positions through multiple attention heads\. LetX∈ℝS×dmodelX\\in\\mathbb\{R\}^\{S\\times d\_\{\\mathrm\{model\}\}\}denote the input hidden states, whereSSis the sequence length anddmodeld\_\{\\mathrm\{model\}\}is the hidden dimension\. For headii, the projections are

Qi=X​WiQ,Ki=X​WiK,Vi=X​WiV,Q\_\{i\}=XW\_\{i\}^\{Q\},\\quad K\_\{i\}=XW\_\{i\}^\{K\},\\quad V\_\{i\}=XW\_\{i\}^\{V\},\(1\)whereWiQW\_\{i\}^\{Q\},WiKW\_\{i\}^\{K\}, andWiVW\_\{i\}^\{V\}are head\-specific projection matrices\. The output of each head is computed as

headi=softmax​\(Qi​KiTdk\)​Vi\.\\mathrm\{head\}\_\{i\}=\\mathrm\{softmax\}\\\!\\left\(\\frac\{Q\_\{i\}K\_\{i\}^\{T\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)V\_\{i\}\.\(2\)The outputs of all heads are concatenated and passed through the output projectionWOW^\{O\}:

MHA​\(X\)=Concat​\(head1,…,headH\)​WO\.\\mathrm\{MHA\}\(X\)=\\mathrm\{Concat\}\(\\mathrm\{head\}\_\{1\},\\ldots,\\mathrm\{head\}\_\{H\}\)W^\{O\}\.\(3\)In Stage 2, all importance estimators are applied under the same structured pruning unit\. For MHA, the unit is an attention\-head group\. For layerlland headhh, we define this group as:

Gl,hMHA=\{Wl,hQ,Wl,hK,Wl,hV,Wl,hO\},G^\{\\mathrm\{MHA\}\}\_\{l,h\}=\\\{W^\{Q\}\_\{l,h\},W^\{K\}\_\{l,h\},W^\{V\}\_\{l,h\},W^\{O\}\_\{l,h\}\\\},\(4\)whereWl,hOW^\{O\}\_\{l,h\}denotes the output\-projection slice associated with headhh\. The importance estimators used in Stage 2 assign a score to each groupGl,hMHAG^\{\\mathrm\{MHA\}\}\_\{l,h\}, and the lowest\-scoring heads are removed according to the layer\-wise pruning ratios proposed by Bayesian optimization\. This group\-wise definition allows weight\-level estimators, such as Wanda\-SP or gradient\-based criteria, to be used consistently for structured head pruning by aggregating their scores over all parameters belonging to the same head\.

### III\-BFeed\-Forward Network

The Feed\-Forward Network \(FFN\), also referred to as the MLP module, applies token\-wise nonlinear transformations after attention\. Modern LLMs commonly use a gated MLP, such as SwiGLU, with up, gate, and down projections\. For inputX∈ℝS×dmodelX\\in\\mathbb\{R\}^\{S\\times d\_\{\\mathrm\{model\}\}\}, the MLP can be written as

MLP​\(X\)=\(SiLU​\(X​WGate\)⊙X​WUp\)​WDown,\\mathrm\{MLP\}\(X\)=\\left\(\\mathrm\{SiLU\}\(XW\_\{\\mathrm\{Gate\}\}\)\\odot XW\_\{\\mathrm\{Up\}\}\\right\)W\_\{\\mathrm\{Down\}\},\(5\)whereWUpW\_\{\\mathrm\{Up\}\}andWGateW\_\{\\mathrm\{Gate\}\}expand the hidden representation to the intermediate dimension,WDownW\_\{\\mathrm\{Down\}\}projects it back todmodeld\_\{\\mathrm\{model\}\}, and⊙\\odotdenotes element\-wise multiplication\. In Stage 2, MLP pruning removes structured intermediate\-neuron groups\. For layerlland intermediate neuronuu, the pruning group is defined as

Gl,uMLP=\{Wl,:,uGate,Wl,:,uUp,Wl,u,:Down\}\.G^\{\\mathrm\{MLP\}\}\_\{l,u\}=\\\{W^\{\\mathrm\{Gate\}\}\_\{l,:,u\},W^\{\\mathrm\{Up\}\}\_\{l,:,u\},W^\{\\mathrm\{Down\}\}\_\{l,u,:\}\\\}\.\(6\)Thus, pruning neuronuuconsistently removes the coupled gate, up, and down projection components associated with that neuron\. All Stage 2 importance estimators used in the ablation study score the same groupsGl,hMHAG^\{\\mathrm\{MHA\}\}\_\{l,h\}andGl,uMLPG^\{\\mathrm\{MLP\}\}\_\{l,u\}; only the scoring function changes across Importance, Wanda\-SP, Taylor, Gradient, and WActiGrad\.

The mathematical formulation of both stages is as follows:

### III\-CStage 1: Multi\-Objective Layer\-Wise Pruning

Given an LLMℳ\\mathcal\{M\}, which consists ofℒ\\mathcal\{L\}layers, the aim is to find the components that need to be pruned, including MLP and MHA\. Letl∈\{1,2,3,…,ℒ\}l\\in\\\{1,2,3,\\dots,\\mathcal\{L\}\\\}represent the current layer, and letxl,MHAx\_\{l,\\text\{MHA\}\}andxl,MLPx\_\{l,\\text\{MLP\}\}be binary decision variables indicating whether the respective MLP and MHA are retained \(0\) or pruned \(1\) in layerll\. The decision spaceθ\\thetarepresents all possible configurations of pruning decisions for the entire model\. For each layerll, the decision space consists of two binary variablesxl,MHAx\_\{l,\\text\{MHA\}\}andxl,MLPx\_\{l,\\text\{MLP\}\}, resulting in the following search space\.

Θ=∏l=1ℒ\{0,1\}2=\{0,1\}2​ℒ\\Theta=\\prod\_\{l=1\}^\{\\mathcal\{L\}\}\\\{0,1\\\}^\{2\}=\\\{0,1\\\}^\{2\\mathcal\{L\}\}\(7\)Where each configuration inθ\\thetacorresponds to the pruning configuration across allℒ\\mathcal\{L\}layers, consisting of all possible combinations of pruning or retaining the MLP and MHA\.

The pruning process is formulated as a multi\-objective process, where the aim is to find the decision vectorx=\{x1,MHA,x1,MLP,…,xℒ,MHA,xℒ,MLP\}∈Θx=\\\{x\_\{1,\\text\{MHA\}\},x\_\{1,\\text\{MLP\}\},\\ldots,x\_\{\\mathcal\{L\},\\text\{MHA\}\},x\_\{\\mathcal\{L\},\\text\{MLP\}\}\\\}\\in\\Thetathat minimizes both objectives, as represented by:

minx∈Θ⁡\(𝒪1​\(x\),𝒪2​\(x\)\)\\min\_\{x\\in\\Theta\}\\left\(\\mathcal\{O\}\_\{1\}\(x\),\\mathcal\{O\}\_\{2\}\(x\)\\right\)where𝒪1\\mathcal\{O\}\_\{1\}and𝒪2\\mathcal\{O\}\_\{2\}correspond, respectively, to \(i\) the performance loss between the original and pruned models and \(ii\) the number of parameters\. To compute𝒪1\\mathcal\{O\}\_\{1\}, we use the performance KL divergence\. KL divergence quantifies how much the output distribution of the pruned model diverges from the original model\. The KL divergence is computed as:

DKL​\(P∥Q\)=∑iP​\(i\)​log⁡\(P​\(i\)Q​\(i\)\)D\_\{\\mathrm\{KL\}\}\(P\\parallel Q\)=\\sum\_\{i\}P\(i\)\\,\\log\\\!\\left\(\\frac\{P\(i\)\}\{Q\(i\)\}\\right\)\(8\)HereP​\(i\)P\(i\)represents the probability distribution of the original modelℳO​r​i​g​i​n​a​l\\mathcal\{M\}\_\{Original\}andQ​\(i\)Q\(i\)represents the probability distribution of the pruned modelℳP​r​u​n​e​d\\mathcal\{M\}\_\{Pruned\}\.

Layer pruning in stage 1 is implemented by replacing the forward pass of targeted sub\-blocks with an identity mapping\. By utilizing the identity mapping pathways within the residual blocks, the pruned layers are effectively bypassed, allowing the hidden states to propagate through the network via the skip connections without further computation, as illustrated in Figure[2](https://arxiv.org/html/2607.22583#S3.F2)\.

![Refer to caption](https://arxiv.org/html/2607.22583v1/identity.jpg)Figure 2:Schematic representation of block\-level pruning in LLMs\. Figure \(a\) illustrates the standard flow with MHA and MLP sub\-blocks; Figure \(b\) demonstrates the ’pruned’ state where computation is skipped, effectively reducing the model depth while maintaining connectivity through identity mappings\.
### III\-DStage 2: Parallel Bayesian Layer\-Wise Allocation

Stage 2 searches for the optimal layer\-wise configuration on all Pareto fronts returned by Stage 1\. It applies parallel Bayesian optimization to search for layer\-wise pruning\-ratio allocations that provide lower latency for these candidates, ultimately selecting the best overall architecture\.

Letaadenote a candidate allocation vector, wherealMHAa\_\{l\}^\{\\mathrm\{MHA\}\}andalMLPa\_\{l\}^\{\\mathrm\{MLP\}\}are the pruning ratios assigned to MHA heads and MLP neurons in layerll\. The allocation search is formulated to find the configuration that jointly reduces validation perplexity and latency:

a∗=argmina∈𝒜\(\\displaystyle a^\{\*\}=\\arg\\min\_\{a\\in\\mathcal\{A\}\}\\;\(PPLval​\(Prune​\(M\(1\),a,g\)\),\\displaystyle\\mathrm\{PPL\}\_\{\\mathrm\{val\}\}\\\!\\left\(\\mathrm\{Prune\}\(M^\{\(1\)\},a,g\)\\right\),\(9\)T\(Prune\(M\(1\),a,g\)\)\)\\displaystyle T\\\!\\left\(\\mathrm\{Prune\}\(M^\{\(1\)\},a,g\)\\right\)\)subject to the remaining pruning budget

ℬ​\(a\)≤Srem,\\mathcal\{B\}\(a\)\\leq S\_\{\\mathrm\{rem\}\},\(10\)whereM\(1\)M^\{\(1\)\}is a Stage 1 Pareto candidate,ggis an existing importance strategy,PPLval​\(⋅\)\\mathrm\{PPL\}\_\{\\mathrm\{val\}\}\(\\cdot\)is the perplexity measured on the validation set,T​\(⋅\)T\(\\cdot\)is measured latency, andSremS\_\{\\mathrm\{rem\}\}is the remaining pruning budget\. The budget functionℬ​\(a\)\\mathcal\{B\}\(a\)converts the layer\-wise ratios into the corresponding removed\-parameter fraction, so that allocations with different sub\-component counts are compared consistently\. Bayesian optimization proposes batches of allocations in parallel, evaluates their perplexity and latency, and updates the surrogate model after each batch to identify configurations with lower latency\. For each allocation, we benchmark different existing importance strategies to rank removable sub\-components\.

## IVExperimental Settings

All experiments were conducted on three NVIDIA A100 80GB GPUs using PyTorch 2\.1 and the Hugging Face Transformers library\. We used the lm\-evaluation\-harness library for downstream task evaluation\. Evaluations were performed in a zero\-shot setting on Massive Multitask Language Understanding \(MMLU\), HellaSwag \(HS\), AI2 Reasoning Challenge \(ARC\)\-Easy/ARC\-Challenge, Physical Interaction Question Answering \(PIQA\), and WinoGrande \(WG\)\. We report normalized accuracy for all tasks except WinoGrande, for which we report raw accuracy\. For benchmarking, we used an NVIDIA Jetson Nano with an Ampere GPU, 1024 CUDA cores, 32 Tensor Cores, and 64 GB of LPDDR5 memory\.

In the first stage, we implemented the Non\-dominated Sorting Genetic Algorithm II \(NSGA\-II\) via Pymoo with simulated binary crossover and mask mutation \(population size 40, 40 generations\), and used 256 random FineWeb\-Edu samples to compute Kullback–Leibler \(KL\) divergence\. The choice of NSGA\-II is motivated by its effectiveness in solving multi\-objective Neural Architecture Search \(NAS\) problems\[[21](https://arxiv.org/html/2607.22583#bib.bib21),[22](https://arxiv.org/html/2607.22583#bib.bib22)\]\. For the implementation of BO, GPyTorch and BoTorch are used\.

![Refer to caption](https://arxiv.org/html/2607.22583v1/pareto.png)Figure 3:Stage\-1 Pareto solutions\. x\-axis: latency; y\-axis: fitness \(KL loss\)\.TABLE I:Zero\-shot benchmark results for four LLMs at 37\.5%\\%and 50%\\%sparsity levels\. WG denotes WinoGrande and HS denotes HellaSwag\. Gold and silver shading denote the best and second\-best performing pruned results, respectively\. For the proposed approach, we report the mean and standard deviation across five independent runs using FineWeb\-Edu and mixed calibration subsets\.37\.5% Sparsity50% SparsityMethodMMLUWGPIQAHSARC\-eARC\-cAvg\.MMLUWGPIQAHSARC\-eARC\-cAvg\.\\cellcolormodelrowMistral\-v0\.3 7B\\rowcolordenserowDense59\.0873\.7280\.3060\.9179\.6748\.8167\.0859\.0873\.7280\.3060\.9179\.6748\.8167\.08ShortGPT22\.6758\.5656\.9627\.7333\.5929\.2738\.13\\cellcolorgold\!6026\.5053\.9955\.4426\.9734\.05\\cellcolorsilver\!5024\.0636\.83Sliding Window25\.5457\.2259\.1929\.4134\.5126\.8838\.79\\cellcolorsilver\!5024\.2849\.2552\.4526\.6027\.8222\.3533\.79Block Pruner23\.5954\.6266\.1637\.9246\.2524\.3242\.1423\.0252\.4958\.1131\.7435\.8621\.7637\.16EvoPress25\.0057\.1468\.8239\.7950\.2125\.9444\.4823\.2949\.8860\.3432\.1336\.3222\.9537\.48SliceGPT23\.1561\.8865\.3436\.9542\.6821\.4241\.9023\.13\\cellcolorgold\!6055\.8058\.1130\.7331\.9018\.3436\.342SSP24\.4963\.14\\cellcolorgold\!6070\.29\\cellcolorsilver\!5041\.9949\.9624\.4945\.7322\.9555\.41\\cellcolorsilver\!5064\.74\\cellcolorsilver\!5035\.0737\.9219\.5439\.27\\rowcolorproprowProposed\\cellcolorsilver\!5036\.18±\\pm0\.19\\cellcolorsilver\!5065\.51±\\pm0\.2566\.05±\\pm0\.4938\.31±\\pm0\.21\\cellcolorsilver\!5056\.19±\\pm0\.31\\cellcolorsilver\!5032\.68±\\pm0\.21\\cellcolorsilver\!5049\.15±\\pm0\.2423\.17±\\pm0\.3753\.43±\\pm0\.3561\.75±\\pm0\.2932\.80±\\pm0\.38\\cellcolorsilver\!5045\.16±\\pm0\.3821\.84±\\pm0\.29\\cellcolorsilver\!5039\.69±\\pm0\.54\\rowcolorproprowProposed\+Mix\\cellcolorgold\!6037\.85±\\pm0\.87\\cellcolorgold\!6066\.25±\\pm0\.41\\cellcolorsilver\!5068\.92±\\pm0\.62\\cellcolorgold\!6042\.95±\\pm0\.86\\cellcolorgold\!6062\.54±\\pm0\.57\\cellcolorgold\!6033\.02±\\pm0\.67\\cellcolorgold\!6051\.92±\\pm0\.7323\.49±\\pm0\.85\\cellcolorsilver\!5055\.59±\\pm0\.91\\cellcolorgold\!6067\.79±\\pm0\.51\\cellcolorgold\!6036\.44±\\pm0\.74\\cellcolorgold\!6060\.02±\\pm0\.27\\cellcolorgold\!6032\.85±\\pm0\.26\\cellcolorgold\!6046\.03±\\pm0\.69\\cellcolormodelrowLLaMA\-2 7B\\rowcolordenserowDense40\.6768\.9078\.0757\.0976\.2243\.3460\.7140\.6768\.9078\.0757\.0976\.2243\.3460\.71ShortGPT32\.2560\.5459\.6333\.5441\.3328\.5042\.63\\cellcolorsilver\!5026\.33\\cellcolorsilver\!5055\.9654\.7928\.6332\.3723\.2936\.89Sliding Window33\.3858\.6460\.0733\.4736\.1528\.4141\.6924\.4652\.5754\.7926\.7628\.3222\.8734\.96Block Pruner23\.5955\.0966\.8736\.9250\.8024\.4942\.9623\.0849\.7260\.0130\.5538\.5921\.1637\.19EvoPress25\.6652\.0168\.6137\.1553\.2025\.9443\.7622\.9050\.2060\.0131\.6042\.59\\cellcolorsilver\!5023\.8938\.53SliceGPT23\.07\\cellcolorsilver\!5063\.8567\.9040\.4047\.5626\.1944\.8323\.08\\cellcolorgold\!6056\.04\\cellcolorsilver\!5060\.66\\cellcolorsilver\!5032\.6733\.5920\.3137\.722SSP27\.9161\.33\\cellcolorsilver\!5070\.2942\.7855\.9327\.3947\.6022\.9851\.0460\.1229\.8041\.5621\.0537\.76CFSP\[[23](https://arxiv.org/html/2607.22583#bib.bib23)\]27\.4263\.1369\.6441\.7857\.74\\cellcolorgold\!6037\.2949\.6223\.0551\.9659\.1626\.1840\.9720\.2936\.94\\rowcolorproprowProposed\\cellcolorsilver\!5033\.42±\\pm0\.8163\.30±\\pm0\.2370\.02±\\pm0\.26\\cellcolorsilver\!5045\.16±\\pm0\.04\\cellcolorsilver\!5061\.07±\\pm0\.1236\.09±\\pm0\.92\\cellcolorsilver\!5049\.79±\\pm0\.6523\.05±\\pm0\.7052\.33±\\pm0\.5560\.01±\\pm0\.6529\.77±\\pm0\.64\\cellcolorsilver\!5043\.61±\\pm0\.2123\.14±\\pm0\.68\\cellcolorsilver\!5038\.65±\\pm0\.63\\rowcolorproprowProposed\+Mix\\cellcolorgold\!6033\.89±\\pm0\.14\\cellcolorgold\!6064\.38±\\pm0\.11\\cellcolorgold\!6073\.39±\\pm0\.29\\cellcolorgold\!6045\.49±\\pm0\.72\\cellcolorgold\!6064\.60±\\pm1\.83\\cellcolorsilver\!5036\.18±\\pm0\.76\\cellcolorgold\!6052\.99±\\pm0\.17\\cellcolorgold\!6026\.95±\\pm0\.1851\.30±\\pm0\.47\\cellcolorgold\!6061\.57±\\pm0\.32\\cellcolorgold\!6032\.84±\\pm0\.38\\cellcolorgold\!6044\.85±\\pm0\.87\\cellcolorgold\!6024\.82±\\pm0\.15\\cellcolorgold\!6040\.39±\\pm0\.17\\cellcolormodelrowQwen\-2\.5 7B\\rowcolordenserowDense71\.8873\.2478\.5659\.9780\.3548\.2968\.7271\.8873\.2478\.5659\.9780\.3548\.2968\.72ShortGPT23\.0251\.4663\.9833\.9050\.5524\.7441\.2822\.8445\.1252\.3025\.4426\.1522\.1032\.33Sliding Window23\.7652\.4965\.0234\.3654\.4623\.1242\.2022\.9044\.8050\.1525\.1024\.3020\.9031\.36Block Pruner25\.1553\.3567\.4136\.9758\.5927\.2244\.7823\.0148\.2556\.7028\.3533\.1021\.4535\.14EvoPress24\.2155\.1767\.3036\.9559\.7627\.1345\.0923\.1549\.0258\.4030\.1234\.5522\.0836\.22SliceGPT22\.9157\.7065\.9434\.3248\.0220\.4841\.5622\.9550\.1159\.2029\.8831\.4019\.5035\.512SSP23\.34\\cellcolorsilver\!5061\.40\\cellcolorsilver\!5070\.29\\cellcolorsilver\!5043\.7552\.6526\.7946\.3723\.05\\cellcolorgold\!6058\.88\\cellcolorsilver\!5066\.00\\cellcolorsilver\!5037\.2143\.0120\.6541\.47\\rowcolorproprowProposed\\cellcolorsilver\!5034\.61±\\pm0\.2757\.22±\\pm0\.1767\.08±\\pm0\.1937\.78±\\pm0\.81\\cellcolorgold\!6061\.20±\\pm0\.28\\cellcolorsilver\!5031\.40±\\pm0\.51\\cellcolorsilver\!5048\.91±\\pm0\.17\\cellcolorsilver\!5023\.29±\\pm0\.2053\.38±\\pm0\.3965\.83±\\pm0\.4334\.94±\\pm0\.19\\cellcolorsilver\!5055\.05±\\pm0\.09\\cellcolorsilver\!5027\.82±\\pm0\.40\\cellcolorsilver\!5043\.39±\\pm0\.19\\rowcolorproprowProposed\+Mix\\cellcolorgold\!6038\.76±\\pm0\.41\\cellcolorgold\!6062\.85±\\pm0\.62\\cellcolorgold\!6071\.41±\\pm0\.86\\cellcolorgold\!6044\.59±\\pm0\.35\\cellcolorsilver\!5060\.45±\\pm0\.84\\cellcolorgold\!6032\.91±\\pm0\.82\\cellcolorgold\!6051\.84±\\pm0\.29\\cellcolorgold\!6027\.68±\\pm0\.59\\cellcolorsilver\!5058\.78±\\pm0\.07\\cellcolorgold\!6067\.83±\\pm0\.59\\cellcolorgold\!6037\.37±\\pm0\.18\\cellcolorgold\!6057\.45±\\pm0\.60\\cellcolorgold\!6032\.73±\\pm0\.18\\cellcolorgold\!6045\.34±\\pm0\.41\\cellcolormodelrowPhi\-3 14B\\rowcolordenserowDense67\.6175\.7781\.0164\.0484\.0560\.6772\.1967\.6175\.7781\.0164\.0484\.0560\.6772\.19ShortGPT27\.0352\.0956\.0427\.1534\.3428\.5048\.3424\.5850\.2853\.6526\.1329\.6724\.6634\.83Sliding Window25\.1750\.0452\.7725\.6526\.2223\.0447\.8023\.5349\.8852\.3425\.4725\.8820\.7332\.97Block Pruner27\.9061\.4068\.1242\.0062\.7537\.3748\.1623\.6652\.3360\.8832\.5646\.5527\.6540\.61EvoPress34\.6360\.1467\.8541\.4661\.7435\.5848\.4823\.4551\.6060\.1031\.0538\.4026\.1538\.46SliceGPT27\.2166\.6171\.1645\.4554\.3429\.7848\.5724\.1055\.3062\.4533\.1235\.6024\.8039\.232SSP\\cellcolorsilver\!5051\.8568\.82\\cellcolorgold\!6074\.97\\cellcolorsilver\!5051\.6067\.2638\.9949\.1025\.8856\.1264\.0535\.4048\.9228\.1543\.09\\rowcolorproprowProposed47\.55±\\pm0\.37\\cellcolorsilver\!5069\.51±\\pm0\.8573\.33±\\pm0\.4148\.69±\\pm0\.53\\cellcolorsilver\!5068\.35±\\pm0\.81\\cellcolorsilver\!5046\.22±\\pm0\.51\\cellcolorsilver\!5049\.45±\\pm0\.35\\cellcolorsilver\!5026\.24±\\pm0\.66\\cellcolorsilver\!5057\.38±\\pm1\.62\\cellcolorsilver\!5065\.18±\\pm1\.62\\cellcolorsilver\!5035\.88±\\pm1\.01\\cellcolorsilver\!5050\.72±\\pm0\.72\\cellcolorsilver\!5030\.46±\\pm0\.32\\cellcolorsilver\!5044\.64±\\pm0\.47\\rowcolorproprowProposed\+Mix\\cellcolorgold\!6052\.98±\\pm0\.46\\cellcolorgold\!6069\.61±\\pm0\.91\\cellcolorsilver\!5074\.81±\\pm0\.75\\cellcolorgold\!6052\.59±\\pm0\.39\\cellcolorgold\!6069\.74±\\pm0\.64\\cellcolorgold\!6046\.41±\\pm0\.27\\cellcolorgold\!6050\.09±\\pm0\.83\\cellcolorgold\!6026\.97±\\pm0\.71\\cellcolorgold\!6057\.83±\\pm0\.42\\cellcolorgold\!6065\.40±\\pm0\.53\\cellcolorgold\!6036\.79±\\pm0\.94\\cellcolorgold\!6055\.93±\\pm0\.17\\cellcolorgold\!6032\.85±\\pm0\.47\\cellcolorgold\!6045\.96±\\pm0\.85

We define the pruning ratioSSas the target sparsity level, aiming to pruneS%S\\%of the components from the given LLM, following standard practices in existing studies\[[11](https://arxiv.org/html/2607.22583#bib.bib11),[7](https://arxiv.org/html/2607.22583#bib.bib7)\]\. The distribution of this pruning ratio between Stage 1 and Stage 2 is determined by the results of Stage 1, which returns a set of Pareto\-optimal solutions with varying numbers of parameters\.

In Stage 1, pruning is applied up to the limitS%S\\%, generating multiple solutions\. Each solutioniiis characterized by its specific sparsity ratepi%p\_\{i\}\\%, defined as:

pi=Pruned parametersiTotal parametersp\_\{i\}=\\frac\{\\text\{Pruned parameters\}\_\{i\}\}\{\\text\{Total parameters\}\}\(11\)Stage 2 then allocates the remaining pruning budget,S−piS\-p\_\{i\}, for each respective solution using parallel Bayesian optimization\. The optimized allocation specifies how much pruning should be applied to sub\-components in each retained layer to obtain a configuration with lower latency\.

We conducted experiments at 37\.5% and 50% pruning rates on four different LLM families to evaluate performance across varying sparsity levels and demonstrate the generalization of the proposed approach\. We compared our method against existing studies without applying LoRA fine\-tuning to ensure a fair comparison and to demonstrate the effectiveness of our approach in the absence of fine\-tuning\.

In Stage 2, we search over the remaining layer\-wise pruning budgets of all Pareto\-optimal solutions\. We evaluate the resulting candidates using a multi\-task validation set composed of 1,024 samples drawn from several benchmarks, including HellaSwag, MMLU, ARC\-Easy, and ARC\-Challenge\. Additionally, we evaluate the candidates on the Wikitext\-2 dataset and report the results for both validation sets\. For each allocation, existing importance strategies are used to rank sub\-components, allowing us to benchmark their behavior under the same Bayesian allocation framework\. The final model is selected by lexicographic ranking: candidates are first ordered by downstream accuracy \(higher is better\), and ties are broken by lower computational cost, measured either by active parameter count or inference latency\.

## VResults and Discussion

In this section, we evaluate the proposed approach on multiple language\-modeling tasks across multiple LLMs, comparing its performance with existing baseline methods\. Given that our proposed approach utilizes both depth\-wise and component\-wise pruning, we benchmark against the structural pruning baselines reported in Table[I](https://arxiv.org/html/2607.22583#S4.T1), including ShortGPT\[[11](https://arxiv.org/html/2607.22583#bib.bib11)\], Sliding Window\[[10](https://arxiv.org/html/2607.22583#bib.bib10)\], Block Pruner\[[9](https://arxiv.org/html/2607.22583#bib.bib9)\], EvoPress\[[15](https://arxiv.org/html/2607.22583#bib.bib15)\], SliceGPT\[[8](https://arxiv.org/html/2607.22583#bib.bib8)\], 2SSP\[[7](https://arxiv.org/html/2607.22583#bib.bib7)\]and CFSP\[[23](https://arxiv.org/html/2607.22583#bib.bib23)\], a coarse\-to\-fine pruning approach\. We compare against CFSP only on LLaMA\-2\-7B, as their method supports only Llama\-based architectures\.

Figure[3](https://arxiv.org/html/2607.22583#S4.F3)shows the visualization of solutions obtained after the multi\-objective search on 37\.5%\\%pruning ratio, including individual solutions and Pareto front solutions\. The accuracy on downstream tasks is reported for four popular models: LLaMA 2\-7B\[[19](https://arxiv.org/html/2607.22583#bib.bib19)\], Qwen 2\.5 7B\[[24](https://arxiv.org/html/2607.22583#bib.bib24)\], Phi\-3 14B\[[25](https://arxiv.org/html/2607.22583#bib.bib25)\]and Mistral\-v0\.3\-7B\[[20](https://arxiv.org/html/2607.22583#bib.bib20)\]\. The detailed results are provided in Table[I](https://arxiv.org/html/2607.22583#S4.T1)\.

We evaluated our approach using both the WikiText\-2 and Mixed Calibration datasets as validation sets\. Due to time and resource constraints, we conducted five independent runs and reported their average and standard deviation\. The results at 37\.5%\\%and 50%\\%sparsity show that the proposed approach achieves the strongest overall average across all four model families\. In particular, it consistently preserves performance on ARC\-Easy and ARC\-Challenge, indicating that reasoning\-oriented capabilities remain robust under structured pruning\. At 50%\\%sparsity, competing approaches degrade more sharply, whereas the proposed approach still retains the highest average accuracy across all models\.

At 50%\\%sparsity, the method remains especially strong on ARC\-Easy for all four models and on ARC\-Challenge for Qwen\-2\.5 and Phi\-3\. Although 2SSP and SliceGPT obtain higher scores on selected tasks such as WinoGrande, PIQA, and HellaSwag, the proposed approach provides a more robust global accuracy–sparsity trade\-off\. The results of mixed calibration sets are reported asProposed\+Mix\. The resulting layer\-wise pruning patterns are visualized in Figure[4](https://arxiv.org/html/2607.22583#S5.F4)\. The pruning distribution is non\-uniform across layers and components, demonstrating that BO does not apply a fixed pruning ratio universally\. Instead, it removes more MHA or MLP capacity from layers that are less critical to the selected objectives, while preserving components that are highly sensitive to accuracy drops\. This behavior explains why the proposed method can reduce latency and active model size without causing the sharp downstream performance degradation typically observed in uniform or single\-stage pruning strategies\.

![Refer to caption](https://arxiv.org/html/2607.22583v1/pruned.png)Figure 4:Layer\-wise pruning patterns of MHA and MLP components under the two\-stage strategy at 37\.5%\\%sparsity\. The figure highlights how Stage\-1 and Stage\-2 distribute the pruning budget across layers and sub\-components\.In addition to zero\-shot accuracy, Table[II](https://arxiv.org/html/2607.22583#S5.T2)reports latency on 100\-token sequences averaged over 10 runs on both an NVIDIA A100 and an NVIDIA Jetson Nano\. The proposed approach achieves the lowest latency on both platforms \(73\.8512 s on A100 and 843 s on Jetson Nano\), confirming that the multi\-objective search and parallel BO search improve deployment efficiency together with accuracy\. These results show that the proposed method not only preserves downstream performance but also reduces end\-to\-end inference latency\.

TABLE II:Comparison of latency between the proposed approach and existing studies\. Results are shown in seconds, computed on 100\-token text sequences over 10 iterations on an NVIDIA A100 GPU and NVIDIA Jetson Nano\.MethodNVIDIA A100NVIDIA Jetson NanoSliding Window\[[10](https://arxiv.org/html/2607.22583#bib.bib10)\]82\.241629562SSP\[[7](https://arxiv.org/html/2607.22583#bib.bib7)\]92\.614731150Block Pruner\[[9](https://arxiv.org/html/2607.22583#bib.bib9)\]85\.467691074EvoPress\[[15](https://arxiv.org/html/2607.22583#bib.bib15)\]85\.578851187ShortGPT\[[11](https://arxiv.org/html/2607.22583#bib.bib11)\]81\.936381283\\rowcolorbaselinegray Proposed\\cellcolorbestgray73\.8512\\cellcolorbestgray843Although the proposed approach maintains an overall balance across all evaluated metrics, certain performance gaps arise because Stage\-1 prioritizes global latency and efficiency, while Stage\-2 relies on task\-agnostic importance scores from a general calibration set rather than optimizing for individual benchmarks\. Consequently, it selectively prunes components that offer minimal performance contributions while incurring high computational costs\. Despite these localized trade\-offs, the proposed approach achieves the highest average performance alongside the lowest latency across the tested hardware\.

In addition to benchmarking downstream tasks, we computed perplexity \(PPL\) scores on three benchmarks: WikiText\-2, FineWeb\-Edu, and C4\. The bar chart in Figure[5](https://arxiv.org/html/2607.22583#S5.F5)compares these scores at two sparsity rates \(37\.5%\\%and 50%\\%\) for two LLMs, LLaMA\-2\-7B and Mistral\-v3\. The sub\-figures show that the proposed approach achieves lower perplexity across all benchmarks and models, confirming its effectiveness beyond downstream\-task accuracy\.

![Refer to caption](https://arxiv.org/html/2607.22583v1/perplexity.png)Figure 5:Perplexity \(PPL\) comparison of various pruning methods across Mistral and LLaMA2 models\. Evaluation is conducted on Wikitext\-2, C4, and Fineweb\-E datasetsTo study the impact of the sparsity rate \(SS\) on performance, we calculate the perplexity scores on WikiText\-2 at different sparsity rates after applying Stage 1 and Stage 2 and compare them with baseline studies, as shown in Figure[6](https://arxiv.org/html/2607.22583#S5.F6)\. The results show that varyingSSsignificantly impacts performance; while smaller sparsity rates result in only a minor increase in perplexity, a substantial increase is observed beyond 40%\\%sparsity, with the rise becoming more pronounced between 50%\\%and 70%\\%\.

This is due to the removal of essential weights that encode the model’s fundamental linguistic patterns, which cannot be recovered once the redundancy threshold is surpassed\. The baseline methods did not show increased perplexity until 40%\\%, but after 40%\\%they exhibit a significant increase; however, the proposed approach maintains a lower increase in perplexity compared to existing studies even after 40%\\%\. This demonstrates that the proposed approach remains robust even as the sparsity rate increases\.

![Refer to caption](https://arxiv.org/html/2607.22583v1/perplexity_vs_sparsity.png)Figure 6:Comparison of the perplexity values in log scale on different sparsity rates of the proposed approach with existing studies### V\-AAblation studies

We conducted ablation studies to isolate the main design choices of the proposed framework\. We performed ablation studies on LLaMA\-2\-7B\. The analysis focuses on five primary questions: \(i\) whether the proxy metrics used during the search correlate with downstream accuracy; \(ii\) how the calibration set affects downstream performance and latency; \(iii\) whether the two\-stage design is more effective than using either stage alone; \(iv\) which objective combination provides the best accuracy–efficiency trade\-off; and \(v\) how different stage\-2 sparsity allocation and importance estimation strategies affect the pruned model’s performance\.

#### V\-A1Proxy Metrics and Downstream Accuracy

To validate our Stage 1 multi\-objective search, we analyze the correlation between the resulting Pareto fronts \(optimized for KL divergence and perplexity\) and final downstream accuracy\. We compute these proxy metrics for all candidate pruned models and compare them against their zero\-shot performance\. The motivation behind this analysis is to validate these proxies, as our search procedure relies on computationally inexpensive evaluations to filter candidates before selecting them for expensive downstream validation\.

![Refer to caption](https://arxiv.org/html/2607.22583v1/03_proxy_correlation_panel.png)Figure 7:Correlation between evaluation proxies and downstream accuracy across sparsity levels\.As shown in Figure[7](https://arxiv.org/html/2607.22583#S5.F7)and Table[III](https://arxiv.org/html/2607.22583#S5.T3), KL divergence provides a stronger signal at moderate sparsity, whereas perplexity becomes more predictive as pruning becomes more aggressive\. Specifically, at 37\.5% sparsity, KL divergence is highly informative \(r=−0\.967r=\-0\.967\), indicating that distributional similarity is an effective search objective when moderate pruning retains most of the model’s capacity\. However, when sparsity increases to 50%, perplexity becomes more predictive\. This suggests that while KL divergence is useful when pruning is conservative, perplexity becomes the more reliable proxy as aggressive pruning directly impacts core language modeling performance\.

TABLE III:Correlation and regression analysis with downstream average accuracy\.GroupPred\.𝒓\\boldsymbol\{r\}𝒓𝟐\\boldsymbol\{r^\{2\}\}𝝆\\boldsymbol\{\\rho\}𝝆𝟐\\boldsymbol\{\\rho^\{2\}\}Sp\. 0\.375KL div\.\-0\.9670\.935\-0\.7940\.630Sp\. 0\.375PPL\-0\.3580\.128\-0\.0060\.000Sp\. 0\.5KL div\.0\.4170\.1740\.3810\.145Sp\. 0\.5PPL\-0\.8280\.685\-0\.7860\.617
#### V\-A2Calibration\-Set Size

The calibration size controls the quality of the activation statistics used for pruning\. Figure[8](https://arxiv.org/html/2607.22583#S5.F8)summarizes the relationship between calibration size, downstream performance, latency, and throughput\. It shows that increasing the calibration set from 128 to 2048 samples consistently improves deployment metrics: mean latency decreases from approximately 4\.32 s to 4\.03 s, while throughput increases from about 118\.5 to 126\.2 tokens/s\. Downstream accuracy follows the same trend, rising from 39\.7%\\%at 128 samples to 49\.8%\\%at 2048 samples\.

![Refer to caption](https://arxiv.org/html/2607.22583v1/calibration_samples_tradeoff.png)Figure 8:Effect of calibration sample size on latency, throughput, and downstream accuracy\.The largest accuracy gain occurs between 512 and 1024 samples, suggesting that smaller calibration sets are insufficient to capture task\-relevant behavior\. However, the latency curve begins to saturate after 512 samples, showing that efficiency gains plateau earlier than accuracy gains\. This result supports using sufficiently large calibration sets while highlighting the practical trade\-off between calibration cost and marginal performance improvement\. Given the diminishing returns between 1024 and 2048 samples, we selected a calibration size of 1024 for our experiments to avoid unnecessary computational run\-time\.

#### V\-A3Stage\-Wise Combination Ablation

We evaluate whether a two\-stage pruning approach is essential, or if single\-stage pruning is equally effective\. To demonstrate this, Figure[9](https://arxiv.org/html/2607.22583#S5.F9)compares alternative pipeline decompositions: depth\-only, width\-only, sequential, reversed, and combined configurations\.

![Refer to caption](https://arxiv.org/html/2607.22583v1/h1h2_sparsity_downstream_latency_combined.png)Figure 9:Comparative Analysis of Downstream Accuracy and Final Latency for Various H1\-H2 Model Configurations\.These ablations reveal that depth pruning \(Stage 1\) alone removes large computational blocks at the cost of architectural balance, whereas width pruning alone offers fine control but misses block\-level redundancy\. Combining the two allows the method to first identify globally efficient candidates and then refine them through component\-level pruning with balance between latency and accuracy\. This confirms that the combined design is a core strategy rather than a mere sequential implementation detail\.

### V\-BStage\-2 Importance Estimator Ablation

We perform a comparative study of different importance estimation strategies in Stage 2 while keeping Stage 1 fixed\. Table[IV](https://arxiv.org/html/2607.22583#S5.T4)compares Stage 2 pruning methods in terms of average downstream accuracy, average perplexity over WikiText\-2, C4, and FineWeb\-Edu, and latency\.

TABLE IV:Comparison of Stage 2 pruning metrics across importance estimators\. Higher downstream average is better, while lower average PPL over WikiText\-2, C4, and FineWeb\-Edu and lower latency are better\.MethodDownstream Avg\.↑\\uparrowAvg\. PPL↓\\downarrowLatency↓\\downarrowImportance\[[12](https://arxiv.org/html/2607.22583#bib.bib12)\]0\.463429\.96883\.7887Wanda\-SP\[[17](https://arxiv.org/html/2607.22583#bib.bib17)\]\\cellcolorbestgray0\.492738\.62503\.9623Taylor\[[5](https://arxiv.org/html/2607.22583#bib.bib5)\]0\.482123\.62503\.7659Gradient\[[26](https://arxiv.org/html/2607.22583#bib.bib26)\]0\.4811\\cellcolorbestgray23\.53123\.7610WActiGrad\[[13](https://arxiv.org/html/2607.22583#bib.bib13)\]0\.441326\.3438\\cellcolorbestgray3\.7604Table[IV](https://arxiv.org/html/2607.22583#S5.T4)shows that Wanda\-SP achieves the highest downstream average accuracy \(0\.49270\.4927\), but it also has the highest average PPL \(38\.625038\.6250\) and the highest latency \(3\.96233\.9623\) among the tested estimators\. This suggests that the activation\-aware salience metric of Wanda\-SP\[[17](https://arxiv.org/html/2607.22583#bib.bib17)\]preserves structures that are useful for downstream prediction, but it may retain components that are less favorable for language\-modeling perplexity and inference speed\. In contrast, gradient\-based pruning achieves the lowest average PPL \(23\.531223\.5312\), while WActiGrad\[[13](https://arxiv.org/html/2607.22583#bib.bib13)\]provides the lowest latency \(3\.76043\.7604\) at the cost of the lowest downstream accuracy\. Taylor pruning offers a balanced trade\-off, with downstream accuracy close to Gradient and substantially lower PPL than Wanda\-SP\. Overall, these results show that the Stage 2 importance estimator should be selected according to the deployment objective: Wanda\-SP is preferable when downstream accuracy is prioritized, whereas Gradient or Taylor are more suitable when perplexity and latency are more important\.

### V\-CStage\-2 Sparsity Allocation Methods Ablation

TABLE V:Stage 2 allocation\-method comparison\. Lower average PPL over WikiText\-2, C4, and FineWeb\-Edu and lower latency are better, while higher throughput and downstream average are better\.MethodAvg\. PPL↓\\downarrowLatency↓\\downarrowThroughput↑\\uparrowDownstream Avg\.↑\\uparrowLSA\[[27](https://arxiv.org/html/2607.22583#bib.bib27)\]45\.86943\.8216133\.980\.3572DSA\[[28](https://arxiv.org/html/2607.22583#bib.bib28)\]35\.82963\.8948131\.600\.3850DLP\[[29](https://arxiv.org/html/2607.22583#bib.bib29)\]28\.02803\.8965131\.470\.4142ER\[[31](https://arxiv.org/html/2607.22583#bib.bib31)\]10\.36323\.8641132\.530\.4898OWL\[[30](https://arxiv.org/html/2607.22583#bib.bib30)\]10\.42513\.9467129\.870\.4929ER\+\[[31](https://arxiv.org/html/2607.22583#bib.bib31)\]10\.37453\.8689132\.350\.4898Uniform10\.36323\.8885131\.670\.4898Proposed\(Beta\)9\.63093\.7871135\.210\.5142Table[V](https://arxiv.org/html/2607.22583#S5.T5)compares the Stage 2 sparsity\-allocation strategies against the proposed allocation strategy, denoted as Proposed \(Beta\)\. The experimental results demonstrate that LSA\[[27](https://arxiv.org/html/2607.22583#bib.bib27)\]and DSA\[[28](https://arxiv.org/html/2607.22583#bib.bib28)\]yield lower downstream accuracy and significantly higher PPL compared to the alternative approaches\. In contrast, DLP\[[29](https://arxiv.org/html/2607.22583#bib.bib29)\], ER\[[31](https://arxiv.org/html/2607.22583#bib.bib31)\], and OWL\[[30](https://arxiv.org/html/2607.22583#bib.bib30)\]achieve a substantially better downstream accuracy and lower PPL score on average than LSA\[[27](https://arxiv.org/html/2607.22583#bib.bib27)\]and DSA\[[28](https://arxiv.org/html/2607.22583#bib.bib28)\]\. Amid these baselines, the proposed approach successfully maintains an optimal balance within the efficiency\-accuracy trade\-off, securing low latency alongside superior downstream accuracy\. Ultimately, evaluating the solution pool in parallel assists in exploring a large allocation search space both rapidly and efficiently\.

### V\-DObjective\-Combination Analysis

Our multi\-objective formulation consists of two main objectives: one for model performance and one for efficiency\. We evaluate two performance criteria \(KL\-divergence and perplexity\) and two efficiency criteria \(latency and number of parameters\)\. The results for the different objective combinations are given in Table[VI](https://arxiv.org/html/2607.22583#S5.T6)\.

TABLE VI:Objective\-combination comparison \(lower is better for Inference and PPL; higher is better for Accuracy\)\. Best per column is highlighted\.1st Objective2nd ObjectiveInference↓\\downarrowAccuracy↑\\uparrowPPL↓\\downarrowPerplexityParams7\.950\.58218\.2943KL\-DivergenceParams5\.86\\cellcolorbestgray0\.598\.1393PerplexityLatency6\.600\.57329\.2695KL\-DivergenceLatency\\cellcolorbestgray4\.860\.5647\\cellcolorbestgray7\.48With respect to the first objective, KL\-Divergence consistently outperforms perplexity in both efficiency and language modeling performance across all evaluated settings\. When the number of parameters is used as the second objective, KL\-Divergence yields lower inference time \(5\.86 s vs\. 7\.95 s\), higher mean downstream accuracy \(0\.593 vs\. 0\.582, averaged over tasks\), and lower average perplexity \(8\.14 vs\. 8\.29\), computed over WikiText\-2, C4, and FineWeb\-Edu\.

When latency is used as the second objective, the optimization prioritizes runtime efficiency over performance while still maintaining a balanced trade\-off between the two\. In this setting, KL/Latency achieves the lowest inference time \(4\.86 s\) and the lowest perplexity \(7\.49\), at the cost of a reduced mean downstream accuracy of 0\.565\. Overall, these results reveal a clear accuracy–latency trade\-off: KL/Params is preferable when accuracy preservation is prioritized, whereas KL/Latency is preferable under stringent latency constraints, offering an approximately 17%\\%reduction in inference time at the cost of a 0\.028 absolute \(2\.8 percentage\-point\) decrease in mean downstream accuracy\.

The selected candidate also exhibits non\-uniform layer\-wise pruning ratios, as shown in Figure[10](https://arxiv.org/html/2607.22583#S5.F10)\. This distribution confirms that the proposed optimizer does not impose a fixed sparsity pattern across the network\. It assigns pruning ratios according to layer sensitivity and lower\-latency deployment behavior\.

![Refer to caption](https://arxiv.org/html/2607.22583v1/llama2_7b_candidate1_layerwise_pruning_ratios.png)Figure 10:Layer\-wise pruning ratios for the selected LLaMA\-2\-7B candidate, showing the distribution of pruning across transformer layers under the proposed structured pruning strategy\.

## VIConclusion

In this paper, we introduced a two\-stage LLM pruning framework for deployment\-oriented compression\. In the coarse stage, a multi\-objective algorithm performs depth\-wise pruning by searching for Pareto\-optimal architectures under deployment constraints\. In the second stage, parallel Bayesian optimization allocates layer\-wise pruning ratios to find configurations with lower latency, while existing importance strategies are benchmarked for ranking the components to remove\.

We evaluated our approach on four different LLM families, including Mistral, Qwen, Phi, and Llama at 37\.5%\\%and 50%\\%sparsity rates\. The results show significant improvements across multiple downstream benchmarks, with consistent gains in reasoning tasks, reduced inference latency, and competitive perplexity under fixed sparsity budgets\. We also validated inference latency on the NVIDIA Jetson Nano to benchmark the performance of pruned LLMs on edge devices\. Additionally, we performed ablation studies to demonstrate the effects of using different objectives and stages\.

In future work, we plan to extend the Bayesian allocation framework by using search space partitioning approaches\.

## VIIAcknowledgement

This work was partially supported by the CPER project Cornella and the EXAMA \(Methods and Algorithms at Exascale\) project under grant ANR\-22\-EXNU\-0002\.

## References

- \[1\]Xia, Mengzhou, et al\. ”Sheared LLaMA: Accelerating Language Model Pre\-training via Structured Pruning\.” The Twelfth International Conference on Learning Representations \(2023\)\.
- \[2\]Van Nguyen, Chien, et al\. ”A Survey on Small Language Models\.” Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing\-Natural Language Processing in the Generative AI Era\. \(2025\)\.
- \[3\]Cheng, H\., Zhang, M\. & Shi, J\. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations\.IEEE Transactions On Pattern Analysis And Machine Intelligence\. \(2024\)
- \[4\]LeCun, Y\., Denker, J\. & Solla, S\. Optimal brain damage\.Advances In Neural Information Processing Systems\.2\(1989\)
- \[5\]Ma, X\., Fang, G\. & Wang, X\. Llm\-pruner: On the structural pruning of large language models\.Advances In Neural Information Processing Systems\.36pp\. 21702\-21720 \(2023\)
- \[6\]Gromov, Andrey, et al\. ”The Unreasonable Ineffectiveness of the Deeper Layers\.” The Thirteenth International Conference on Learning Representations\. \(2024\)
- \[7\]Sandri, F\., et al\. 2ssp: A two\-stage framework for structured pruning of llms\.ArXiv Preprint ArXiv:2501\.17771\. \(2025\)
- \[8\]Ashkboos, Saleh, et al\. ”SliceGPT: Compress Large Language Models by Deleting Rows and Columns\.” The Twelfth International Conference on Learning Representations\. \(2024\)
- \[9\]Zhong, L\., et al\. Blockpruner: Fine\-grained pruning for large language models\.Findings Of The Association For Computational Linguistics: ACL 2025\. pp\. 5065\-5080 \(2025\)
- \[10\]Ding, X\., et al\. A Sliding Layer Merging Method for Efficient Depth\-Wise Pruning in LLMs\.ArXiv Preprint ArXiv:2502\.19159\. \(2025\)
- \[11\]Men, X\., et al\. Shortgpt: Layers in large language models are more redundant than you expect\.Findings Of The Association For Computational Linguistics: ACL 2025\. pp\. 20192\-20204 \(2025\)
- \[12\]Sreenivas, Sharath Turuvekere, et al\. ”LLM Pruning and Distillation in Practice: The Minitron Approach\.”arXiv preprint arXiv:2408\.11796\. \(2024\)
- \[13\]Chitty\-Venkata, Krishna Teja, et al\. ”WActiGrad: structured pruning for efficient finetuning and inference of large language models on AI accelerators\.” European Conference on Parallel Processing\. Cham: Springer Nature Switzerland, 2024\.
- \[14\]Kim, Bo\-Kyeong, et al\. ”Shortened llama: A simple depth pruning for large language models\.” arXiv preprint arXiv:2402\.02834 11 \(2024\): 1\.
- \[15\]Sieberling, O\. et al\. EvoPress: Accurate Dynamic Model Compression via Evolutionary Search\.Forty\-second International Conference On Machine Learning\. \(2025\)\.
- \[16\]Frantar, Elias et al\. ”Sparsegpt: Massive language models can be accurately pruned in one\-shot\.” International conference on machine learning\. PMLR, \(2023\)\.
- \[17\]Sun, M\., Liu, Z\., Bair, A\. & Kolter, J\. Z\. A Simple and Effective Pruning Approach for Large Language Models\.International Conference on Learning Representations\. \(2024\)\.
- \[18\]Mugnaini, Leandro Giusti, et al\. ”Efficient LLMs with AMP: Attention Heads and MLP Pruning\.” arXiv preprint arXiv:2504\.21174 \(2025\)\.
- \[19\]Touvron, Hugo, et al\. ”Llama: Open and efficient foundation language models\.” arXiv preprint arXiv:2302\.13971 \(2023\)\.
- \[20\]Jiang, A\. Q\., Sablayrolles, A\., Mensch, A\., Bamford, C\., Chaplot, D\. S\.,et al\.Mistral 7B\.ArXiv preprint arXiv:2310\.06825\(2023\)\.
- \[21\]Deb, Kalyanmoy, et al\. ”A fast and elitist multiobjective genetic algorithm: NSGA\-II\.” IEEE transactions on evolutionary computation 6\.2 \(2002\)
- \[22\]Lu, Zhichao, et al\. ”Neural architecture search as multiobjective optimization benchmarks: Problem formulation and performance assessment\.” IEEE transactions on evolutionary computation 28\.2, 323\-337, \(2023\)\.
- \[23\]Wang, Y\., Ma, M\., Wang, Z\., Chen, J\., Shan, L\., Yang, Q\., Xu, D\., Liu, M\. & Qin, B\. CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse\-to\-Fine Activation Information\.Proceedings Of The 31st International Conference On Computational Linguistics\. pp\. 9311–9328 \(2025\)\.
- \[24\]Ahmed, Imtiaz, et al\. ”Qwen 2\.5: A comprehensive review of the leading resource\-efficient LLM with potential to surpass all competitors\.” Authorea Preprints \(2025\)\.
- \[25\]Abdin, Marah, et al\.*Phi\-4 Technical Report*\. arXiv preprint arXiv:2412\.08905 \(2024\)\.
- \[26\]Huang, Wei, Anda Cheng, and Yinggui Wang\. ”GradPruner: Gradient\-Guided Layer Pruning Enabling Efficient Fine\-Tuning and Inference for LLMs\.” arXiv preprint arXiv:2601\.19503 \(2026\)\.
- \[27\]Yang, Zhiguo, Changjian Deng, Qinke Chen, Zijing Zhou, and Jian Cheng\. ”LSA: Layer\-wise Sparsity Allocation for Large Language Model Pruning Based on Minimal Linear Reconstruction Error\.” International Conference on Learning Representations \(ICLR\), 2026\.
- \[28\]Li, Lujun, Peijie Dong, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo\. ”Discovering Sparsity Allocation for Layer\-wise Pruning of Large Language Models\.” Advances in Neural Information Processing Systems 37 \(NeurIPS\), 2024\.
- \[29\]Chen, Yuli, Bo Cheng, Jiale Han, Yingying Zhang, Yingting Li, and Shuhao Zhang\. ”DLP: Dynamic Layerwise Pruning in Large Language Models\.” Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:7934–7956, 2025\.
- \[30\]Yin, Lu, You Wu, Zhenyu Zhang, Cheng\-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Kumar Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu\. ”Outlier Weighed Layerwise Sparsity \(OWL\): A Missing Secret Sauce for Pruning LLMs to High Sparsity\.” Proceedings of the 41st International Conference on Machine Learning, PMLR 235:57101–57115, 2024\.
- \[31\]Huang, Weizhong, Yuxin Zhang, Xiawu Zheng, Fei Chao, and Rongrong Ji\. ”Determining Layer\-wise Sparsity for Large Language Models Through a Theoretical Perspective\.” Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:26175–26201, 2025\.