通过保留注意力的 MLP 裁剪解锁推理语言模型的结构化剪枝

# Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming
Source: [https://arxiv.org/html/2610.19121](https://arxiv.org/html/2610.19121)
Woosang Lim1,\*, Haneul Yoo2,\*, Vincent\-Daniel Yun3,\*, Sungjoo Yoo1

1Seoul National University, Republic of Korea 2New York University, United States 3University of Southern California, United States

[GitHub](https://github.com/daniel-eai/LayerTrim)

††thanks:Corresponding author: sungjoo\.yoo@gmail\.com\.\*These authors contributed equally\.###### Abstract

Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models\. However, existing methods can cause reasoning performance to collapse on long\-form reasoning tasks such as AIME\. We proposeLayerTrim, a training\-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth\. LayerTrim is the first training\-free structured pruning method designed to preserve long\-form reasoning without gradients or recovery training\. Across three reasoning language models and five benchmarks, LayerTrim outperforms existing structured pruning methods in most evaluated settings, improving average accuracy by up to 21\.4 percentage points over RESP\. OnQwen3\-14B, LayerTrim achieves 1\.24×\\timesfaster prefill and 19\.7% lower peak GPU memory than the dense model, while better preserving reasoning ability\. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy\. These results demonstrate that LayerTrim improves reasoning efficiency through both structural compression and reduced token generation\.

## 1Introduction

Reasoning language models \(RLMs\) achieve strong performance on complex tasks through chain\-of\-thought \(CoT\) reasoning\([DeepSeek\-AI et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib11);[Yang et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib12)\)\. However, generating long reasoning sequences incurs high inference costs, motivating more efficient inference\. Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models\([Ma et al\., 2023](https://arxiv.org/html/2610.19121#bib.bib6);[An et al\., 2024](https://arxiv.org/html/2610.19121#bib.bib7)\)\. However, applying pruning methods developed for non\-reasoning LLMs to RLMs can cause performance to collapse on tasks requiring long chains of thought, such as AIME\([Ding et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib17)\)\.

Figure 1:Accuracy versus generated tokens per response onQwen3\-14Bwith a pruning budget equivalent to 9 of its 40 decoder layers\. Both metrics are averaged across five benchmarks, with a maximum generation length of 32K tokens\.Figure[1](https://arxiv.org/html/2610.19121#S1.F1)illustrates this failure, where existing structured pruning methods lead to longer reasoning traces and lower accuracy\. To investigate this degradation, we analyze pruning strategies onQwen3\-14B\([Yang et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib12)\)\. Our analysis reveals three observations \(Figure[2](https://arxiv.org/html/2610.19121#S1.F2)\)\. First, MLP trimming causes less representation error than attention pruning or entire module removal\. Second, MLP feature importance is concentrated in a subset of features\. Third, concentrating MLP trimming in redundant layers produces less error than distributing the same budget across all layers\. These findings motivate selective MLP compression while preserving attention and decoder depth\.

Figure 2:Observations motivating LayerTrim onQwen3\-14B\. \(a\) LayerTrim reduces memory usage and inference latency with a pruning budget equivalent to 9 of 40 decoder layers\. \(b\) MLP trimming causes less representation error than attention pruning, module removal, or joint pruning\. \(c\) MLP feature importance is concentrated in a subset of features\. \(d\) Block Influence \(BI\)\([Men et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib9)\)\-based layer selection reduces error compared with random or all\-layer pruning\. Diagnostic settings are given in Appendix[D](https://arxiv.org/html/2610.19121#A4)\.Based on these findings, we proposeLayerTrim, a training\-free structured pruning method that identifies redundant decoderlayersand selectivelytrimstheir MLP features while preserving attention and decoder depth\. LayerTrim is the first training\-free structured pruning method designed to preserve long\-form reasoning and code generation performance without recovery training\. It uses dense\-model reasoning traces\([Lucas et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib13);[Wang et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib14)\)to select layers using Block Influence\([Men et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib9)\)and retain MLP features with the largest expected squared output contributions\.

Across three RLMs and five benchmarks, LayerTrim outperforms existing structured pruning methods in most evaluated settings, improving average accuracy by up to 21\.4 percentage points over RESP\([Wang et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib14)\)under matched parameter budgets\. OnQwen3\-14B, LayerTrim achieves higher accuracy while generating fewer tokens on average than the other evaluated pruning methods \(Figure[1](https://arxiv.org/html/2610.19121#S1.F1)\)\. It also reduces peak GPU memory, prefill latency, and decode latency by approximately 20%, 19%, and 17%, respectively, relative to the dense model at a pruning budget equivalent to 9 of 40 decoder layers \(Figure[2](https://arxiv.org/html/2610.19121#S1.F2)\(a\)\)\.

Our contributions are as follows:

- •We identify attention preservation and selective MLP trimming as strategies for reducing representation error compared with attention pruning or entire module removal\.
- •We introduceLayerTrim, a training\-free structured pruning method for long\-form reasoning without gradients or recovery training\.
- •We demonstrate improved reasoning efficiency, achieving higher accuracy with fewer generated tokens and better preserving reasoning completion\.

## 2Related Work

#### Structured pruning of LLMs\.

Unstructured pruning methods such as SparseGPT\([Frantar and Alistarh, 2023](https://arxiv.org/html/2610.19121#bib.bib1)\)and Wanda\([Sun et al\., 2024](https://arxiv.org/html/2610.19121#bib.bib2)\)remove individual weights, often requiring sparse kernels for practical acceleration\. In contrast, structured pruning removes computational units to produce smaller dense models\([Kim et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib4)\)\. LLM\-Pruner\([Ma et al\., 2023](https://arxiv.org/html/2610.19121#bib.bib6)\)and HFPrune\([Zhu et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib18)\)use gradient\- or Taylor\-based importance criteria, while FLAP\([An et al\., 2024](https://arxiv.org/html/2610.19121#bib.bib7)\)compensates for pruning errors\. Sheared LLaMA\([Xia et al\., 2024](https://arxiv.org/html/2610.19121#bib.bib8)\)incorporates additional training, and CoCurve\([Gong et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib19)\)considers interactions between structured units\. Depth pruning methods such as ShortGPT\([Men et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib9)\)identify redundant decoder layers using Block Influence, while LaCo\([Yang et al\., 2024](https://arxiv.org/html/2610.19121#bib.bib10)\)and GRASP\([Liu et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib16)\)reduce model depth through layer merging or replacement\. LoRP\([Yun et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib5)\)exploits inter\-layer redundancy and architectural locality to identify layers for training\-free depth pruning\. However, these methods do not explicitly consider long\-form CoT reasoning, and removing entire layers can degrade reasoning performance\.

#### Pruning RLMs\.

Pruning methods developed for non\-reasoning LLMs can degrade reasoning performance\([Ding et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib17)\)\. RAC\([Lucas et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib13)\)uses on\-policy reasoning traces for activation reconstruction, while OBC\-Prune\([Nguyen et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib3)\)extends reasoning\-aware calibration through outcome\-based token weighting\. Both rely on unstructured pruning, requiring sparse kernel support for practical acceleration\. RESP\([Wang et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib14)\)uses self\-generated reasoning traces, gradient\-based importance, and progressive recalibration for structured pruning, but retains lower reasoning accuracy than dense models across the evaluated benchmarks\.

Preserving long\-form reasoning remains a challenge for existing structured pruning approaches\. LayerTrim addresses this challenge by selectively trimming MLP features while preserving attention and decoder depth\.

## 3Method

### 3\.1Motivation

We first investigate three structural pruning decisions for RLMs: \(1\) which components to prune, \(2\) which MLP features to retain, and \(3\) how to allocate pruning across layers\. UsingQwen3\-14Band AIME reasoning traces generated by the dense model, we characterize MLP feature contributions and compare representation errors across pruning settings\.

#### Pruning targets\.

Methods such as RESP prune attention and MLPs jointly\. We separately compare attention and MLP removal, attention and MLP trimming, and joint trimming of both\. Removal deletes modules selected by low Block Influence, whereas trimming reduces internal width across all 40 decoder layers\.

Across all tested budgets, MLP\-only trimming produces lower final\-hidden representation error than module removal, attention trimming, or joint trimming of attention and MLPs \(Figure[2](https://arxiv.org/html/2610.19121#S1.F2)\(b\)\)\. These results motivate preserving attention and decoder depth while reducing MLP width\.

#### Importance concentration within MLPs\.

To select features for width reduction, we express the MLP output as a sum of intermediate feature terms\. For a SwiGLU MLP\([Shazeer, 2020](https://arxiv.org/html/2610.19121#bib.bib15)\)with intermediate widthII, letϕ\(ℓ\)​\(x\)=SiLU⁡\(Wgate\(ℓ\)​x\)⊙\(Wup\(ℓ\)​x\)∈ℝI\\phi^\{\(\\ell\)\}\(x\)=\\operatorname\{SiLU\}\(W\_\{\\mathrm\{gate\}\}^\{\(\\ell\)\}x\)\\odot\(W\_\{\\mathrm\{up\}\}^\{\(\\ell\)\}x\)\\in\\mathbb\{R\}^\{I\}denote its intermediate activation vector\. Thejjth feature contributescj\(ℓ\)\(x\)=ϕj\(ℓ\)\(x\)Wdown\(ℓ\)\[:,j\]c\_\{j\}^\{\(\\ell\)\}\(x\)=\\phi\_\{j\}^\{\(\\ell\)\}\(x\)W\_\{\\mathrm\{down\}\}^\{\(\\ell\)\}\[:,j\]to the MLP output, yieldingMLP\(ℓ\)​\(x\)=∑j=1Icj\(ℓ\)​\(x\)\\mathrm\{MLP\}^\{\(\\ell\)\}\(x\)=\\sum\_\{j=1\}^\{I\}c\_\{j\}^\{\(\\ell\)\}\(x\)\.

We define the feature importance score as the expected squared norm ofcj\(ℓ\)​\(x\)c\_\{j\}^\{\(\\ell\)\}\(x\):

sj\(ℓ\)\\displaystyle s\_\{j\}^\{\(\\ell\)\}=𝔼𝒟​\[‖cj\(ℓ\)​\(x\)‖22\]\\displaystyle=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\!\\left\[\\\|c\_\{j\}^\{\(\\ell\)\}\(x\)\\\|\_\{2\}^\{2\}\\right\]\(1\)=𝔼𝒟\[\(ϕj\(ℓ\)\(x\)\)2\]∥Wdown\(ℓ\)\[:,j\]∥22\\displaystyle=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\!\\left\[\(\\phi\_\{j\}^\{\(\\ell\)\}\(x\)\)^\{2\}\\right\]\\\|W\_\{\\mathrm\{down\}\}^\{\(\\ell\)\}\[:,j\]\\\|\_\{2\}^\{2\}Here,𝒟\\mathcal\{D\}denotes calibration token positions, andxxis the dense\-model MLP input\. Figure[2](https://arxiv.org/html/2610.19121#S1.F2)\(c\) shows that the top 38\.2% of features account for 89\.4% of the total feature importance, supporting their retention during MLP width reduction\.

#### Redundancy\-aware budget allocation\.

Feature importance determines which dimensions to retain within an MLP, while pruning budget allocation determines which layers undergo width reduction\. Figure[2](https://arxiv.org/html/2610.19121#S1.F2)\(d\) compares trimming all 40 MLPs, 14 randomly selected MLPs, and 14 MLPs selected by low Block Influence under matched parameter budgets\. Trimming the MLPs selected by low Block Influence produces lower final\-hidden representation error than either alternative\. The comparison with random selection indicates that layer selection matters beyond the number of trimmed MLPs\.

### 3\.2LayerTrim

Figure 3:Overview of LayerTrim\. LayerTrim collects layer\-level and MLP feature\-level statistics from reasoning traces generated by the dense model\. It selects redundant decoder layers and retains MLP features with larger output contributions\. The selected MLPs are reduced to smaller dense layers while preserving attention and decoder depth\.Guided by these observations, LayerTrim reduces MLP width while preserving attention and decoder depth\. It retains high\-importance features and concentrates pruning in less influential layers\. Figure[3](https://arxiv.org/html/2610.19121#S3.F3)summarizes five stages: reasoning\-trace calibration, decoder\-layer selection, MLP feature ranking, structured MLP trimming, and the resulting compressed model\. Algorithm[1](https://arxiv.org/html/2610.19121#alg1)in Appendix[A](https://arxiv.org/html/2610.19121#A1)gives the complete procedure\.

#### Reasoning\-trace calibration\.

Following prior work on pruning RLMs\([Lucas et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib13);[Wang et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib14)\), we collect statistics through teacher\-forced forward passes over reasoning traces generated by the dense model\. For each layer, we record decoder\-block inputs and outputs and MLP intermediate activations\. These statistics support layer selection and feature ranking without gradients\.

#### Decoder\-layer selection\.

To concentrate pruning in less influential layers, we useBlock Influence\(BI\)\([Men et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib9)\):

BIℓ=1−𝔼𝒟​\[cos⁡\(hin\(ℓ\),hout\(ℓ\)\)\]\\mathrm\{BI\}\_\{\\ell\}=1\-\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\!\\left\[\\cos\\\!\\left\(h\_\{\\mathrm\{in\}\}^\{\(\\ell\)\},h\_\{\\mathrm\{out\}\}^\{\(\\ell\)\}\\right\)\\right\]\(2\)wherehin\(ℓ\)h\_\{\\mathrm\{in\}\}^\{\(\\ell\)\}andhout\(ℓ\)h\_\{\\mathrm\{out\}\}^\{\(\\ell\)\}are decoder block inputs and outputs\. Low BI indicates a small directional change across a block\. For a budget ofNNlayer\-equivalents, we select theK=2​NK=2Nlowest\-BI layers, denotedℒK\\mathcal\{L\}\_\{K\}, and modify only their MLPs\. This choice spreads the pruning budget across more MLPs rather than applying stronger width reduction to fewer layers\.

#### MLP feature ranking\.

Letmmdenote the number of features retained in each selected MLP\. LetCCandIIbe the hidden and MLP intermediate widths, andPlayerP\_\{\\mathrm\{layer\}\}the prunable parameter count of one decoder layer\. Each removed feature eliminatesCCweights from each of the gate, up, and down projections, totaling3​C3Cparameters per feature and3​C​K​\(I−m\)3CK\(I\-m\)across theKKselected MLPs\. Setting this count equal to the budget,3​C​K​\(I−m\)=N​Player3CK\(I\-m\)=NP\_\{\\mathrm\{layer\}\}, and usingK=2​NK=2Ngives

m=round⁡\(I−Player6​C\)m=\\operatorname\{round\}\\\!\\left\(I\-\\frac\{P\_\{\\mathrm\{layer\}\}\}\{6C\}\\right\)\(3\)
For each selected MLP, we compute feature importancesj\(ℓ\)s\_\{j\}^\{\(\\ell\)\}using Equation[1](https://arxiv.org/html/2610.19121#S3.E1)\. We rank features by this score and retain the topmm,Sℓ⋆=TopM⁡\(s\(ℓ\),m\)S\_\{\\ell\}^\{\\star\}=\\operatorname\{TopM\}\(s^\{\(\\ell\)\},m\)\. To reduce calibration overhead, we approximate the squared MLP output change using the summed scores of removed features\.

#### Structured MLP trimming\.

We physically remove the remainingI−mI\-mfeatures from each selected MLP\. For eachℓ∈ℒK\\ell\\in\\mathcal\{L\}\_\{K\}, we retain rowsSℓ⋆S\_\{\\ell\}^\{\\star\}of the gate and up projections and columnsSℓ⋆S\_\{\\ell\}^\{\\star\}of the down projection\. The reduced projections have shapesm×Cm\\times CandC×mC\\times m\. The retained weights form a compressed checkpoint with smaller dense MLPs\. Attention modules, unselected MLPs, and decoder depth remain unchanged\. No inference\-time masks, fine\-tuning, distillation, or recovery training are required\.

## 4Experimental Results

Lp/Lt=7/40L\_\{p\}/L\_\{t\}=7/40Lp/Lt=9/40L\_\{p\}/L\_\{t\}=9/40MethodTypeAIME25AIME26GPQA\-DHumanEvalMBPP\+Avg\.AIME25AIME26GPQA\-DHumanEvalMBPP\+Avg\.Qwen3\-14B\(Dense\)72\.2±\\pm5\.175\.6±\\pm10\.262\.3±\\pm0\.693\.5±\\pm0\.980\.0±\\pm1\.376\.772\.2±\\pm5\.175\.6±\\pm10\.262\.3±\\pm0\.693\.5±\\pm0\.980\.0±\\pm1\.376\.7ShortGPTDepth0\.00\.05\.6±\\pm0\.519\.5±\\pm1\.26\.0±\\pm0\.66\.20\.00\.00\.3±\\pm0\.61\.4±\\pm0\.40\.2±\\pm0\.20\.4LaCoDepth0\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.0GRASPDepth0\.00\.00\.5±\\pm0\.00\.00\.00\.10\.00\.00\.00\.00\.00\.0FLAPWidth44\.4±\\pm3\.851\.1±\\pm8\.444\.1±\\pm0\.876\.8±\\pm3\.251\.6±\\pm0\.853\.638\.9±\\pm6\.941\.1±\\pm6\.935\.4±\\pm3\.959\.8±\\pm2\.849\.3±\\pm0\.644\.9RESPWidth57\.8±\\pm8\.462\.2±\\pm3\.848\.0±\\pm2\.284\.3±\\pm2\.772\.8±\\pm1\.965\.052\.2±\\pm5\.167\.8±\\pm11\.746\.0±\\pm2\.262\.0±\\pm2\.163\.7±\\pm1\.158\.3CoCurveWidth1\.1±\\pm1\.95\.6±\\pm1\.924\.6±\\pm1\.553\.9±\\pm2\.150\.4±\\pm0\.727\.10\.00\.07\.7±\\pm0\.625\.0±\\pm0\.634\.0±\\pm1\.413\.3LayerTrim \(Ours\)Width64\.4±\\pm5\.160\.0±\\pm5\.849\.7±\\pm2\.990\.2±\\pm1\.677\.2±\\pm1\.868\.356\.7±\\pm3\.363\.3±\\pm3\.342\.8±\\pm3\.588\.0±\\pm2\.774\.8±\\pm0\.365\.1Phi\-4\-reasoning\-plus\(Dense\)72\.2±\\pm5\.167\.8±\\pm5\.167\.7±\\pm1\.778\.7±\\pm2\.468\.8±\\pm2\.271\.072\.2±\\pm5\.167\.8±\\pm5\.167\.7±\\pm1\.778\.7±\\pm2\.468\.8±\\pm2\.271\.0ShortGPTDepth0\.00\.03\.0±\\pm0\.52\.8±\\pm1\.45\.4±\\pm1\.12\.20\.00\.09\.1±\\pm1\.30\.00\.01\.8LaCoDepth0\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.0GRASPDepth0\.00\.00\.5±\\pm0\.50\.00\.00\.10\.00\.00\.00\.00\.00\.0FLAPWidth22\.2±\\pm1\.927\.8±\\pm5\.141\.4±\\pm1\.368\.9±\\pm5\.851\.5±\\pm0\.742\.47\.8±\\pm3\.87\.8±\\pm1\.924\.1±\\pm0\.83\.7±\\pm0\.02\.5±\\pm0\.99\.2RESPWidth62\.2±\\pm1\.950\.0±\\pm5\.833\.5±\\pm0\.650\.6±\\pm2\.153\.2±\\pm0\.949\.940\.0±\\pm5\.845\.6±\\pm5\.115\.7±\\pm0\.534\.1±\\pm1\.237\.6±\\pm1\.134\.6CoCurveWidth8\.9±\\pm3\.83\.3±\\pm3\.317\.0±\\pm0\.857\.3±\\pm2\.245\.9±\\pm1\.326\.53\.3±\\pm0\.03\.3±\\pm0\.06\.2±\\pm0\.326\.6±\\pm3\.024\.9±\\pm1\.612\.9LayerTrim \(Ours\)Width60\.0±\\pm3\.360\.0±\\pm3\.353\.0±\\pm0\.574\.6±\\pm3\.063\.3±\\pm1\.062\.243\.3±\\pm3\.346\.7±\\pm0\.043\.3±\\pm0\.656\.9±\\pm2\.054\.3±\\pm1\.448\.9Magistral\-Small\-2506\(Dense\)56\.7±\\pm3\.361\.1±\\pm8\.462\.3±\\pm3\.094\.3±\\pm0\.977\.2±\\pm1\.170\.356\.7±\\pm3\.361\.1±\\pm8\.462\.3±\\pm3\.094\.3±\\pm0\.977\.2±\\pm1\.170\.3ShortGPTDepth0\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.0LaCoDepth0\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.0GRASPDepth0\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.00\.0FLAPWidth3\.3±\\pm0\.02\.2±\\pm1\.921\.9±\\pm0\.840\.0±\\pm2\.536\.5±\\pm2\.320\.81\.1±\\pm1\.90\.012\.6±\\pm2\.721\.5±\\pm0\.717\.8±\\pm0\.910\.6RESPWidth40\.0±\\pm3\.337\.8±\\pm5\.138\.4±\\pm1\.353\.7±\\pm1\.851\.3±\\pm1\.644\.218\.9±\\pm6\.915\.6±\\pm6\.924\.1±\\pm2\.627\.6±\\pm0\.935\.5±\\pm0\.924\.3CoCurveWidth1\.1±\\pm1\.91\.1±\\pm1\.915\.8±\\pm2\.056\.3±\\pm0\.950\.7±\\pm2\.125\.00\.00\.03\.7±\\pm0\.819\.5±\\pm1\.825\.0±\\pm1\.29\.6LayerTrim \(Ours\)Width30\.0±\\pm8\.842\.2±\\pm1\.952\.2±\\pm1\.590\.7±\\pm3\.773\.8±\\pm1\.257\.818\.9±\\pm8\.418\.9±\\pm1\.944\.3±\\pm1\.676\.0±\\pm3\.170\.5±\\pm2\.545\.7

Table 1:Main results across three RLMs under matched parameter budgets\. The two column groups correspond toLp/Lt=7/40L\_\{p\}/L\_\{t\}=7/40and9/409/40, whereLpL\_\{p\}is the layer\-equivalent pruning budget andLtL\_\{t\}is the original decoder depth\. Type indicates depth or width pruning\. Benchmark entries are mean±\\pmsample std\. over three seeds, and Avg\. is the unweighted mean accuracy \(%\) across the five benchmarks\.Boldmarks the best pruned\-model accuracy within each budget for each benchmark and the average\.### 4\.1Experimental Settings

#### Models and benchmarks\.

We use three RLMs and five benchmarks:

- •Models:Qwen3\-14B\([Yang et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib12)\),Phi\-4\-reasoning\-plus\([Abdin et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib20)\),Magistral\-Small\-2506\([Mistral\-AI et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib21)\);
- •Benchmarks: AIME2025[Zhang and Math\-AI \(2025\)](https://arxiv.org/html/2610.19121#bib.bib22), AIME2026[Zhang and Math\-AI \(2026\)](https://arxiv.org/html/2610.19121#bib.bib23), GPQA\-Diamond[Rein et al\. \(2024\)](https://arxiv.org/html/2610.19121#bib.bib24), HumanEval[Chen et al\. \(2021\)](https://arxiv.org/html/2610.19121#bib.bib26), and MBPP\+\([Liu et al\., 2023](https://arxiv.org/html/2610.19121#bib.bib25)\)\.

We report exact numerical\-answer accuracy on AIME, multiple\-choice accuracy on GPQA\-Diamond, and pass@1 on HumanEval and MBPP\+ using their original and extended test suites, respectively\. For each model, we use its original dense checkpoint as the reference to measure performance retention after pruning\.

#### Baselines and pruning budgets\.

We compare LayerTrim with three depth pruning methods and three width pruning methods:

- •Depth: ShortGPT\([Men et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib9)\), LaCo\([Yang et al\., 2024](https://arxiv.org/html/2610.19121#bib.bib10)\), GRASP\([Liu et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib16)\);
- •Width: FLAP\([An et al\., 2024](https://arxiv.org/html/2610.19121#bib.bib7)\), RESP\([Wang et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib14)\), CoCurve\([Gong et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib19)\)\.

We evaluate budgets ofN∈\{7,9\}N\\in\\\{7,9\\\}layer\-equivalents, each corresponding to the parameter count ofNNdecoder layers\. We match the pruning parameter budget across methods\. Budget allocation and method\-specific implementation details are provided in Appendix[B](https://arxiv.org/html/2610.19121#A2)\.

#### Calibration\.

For each model, we use the same calibration set across all pruning methods, consisting of 24 AIME 2024 reasoning traces generated by its dense checkpoint; following its original procedure, RESP additionally regenerates these traces during pruning\. Traces are generated with temperature0\.60\.6, seed00, and a maximum generation length of 16K tokens\. LayerTrim collects layer redundancy and MLP feature importance statistics from each complete prompt and up to its first 2K generated tokens, without updating its weights\.

#### Evaluation settings\.

We evaluate dense and pruned models using their native chat templates with reasoning enabled\. All models are evaluated with temperature0\.60\.6and top\-pp0\.950\.95\. The effective maximum output length is 32K tokens for all three models\. We evaluate each configuration with generation seeds00,11, and22, generating one response per problem for each seed\. We report the mean and sample standard deviation over the three seeds\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2610.19121#S4.T1)compares LayerTrim with existing pruning methods under matched parameter budgets\. LayerTrim achieves the highest average benchmark score among the pruning methods for all three models at both budgets\. Its lead over the strongest baseline grows as the budget increases fromN=7N=7toN=9N=9\. AtN=9N=9, LayerTrim exceeds the strongest baseline average by 6\.8, 14\.3, and 21\.4 percentage points onQwen3\-14B,Phi\-4\-reasoning\-plus, andMagistral\-Small\-2506, respectively\.

Although calibration uses only AIME 2024 reasoning traces, LayerTrim’s advantages extend to science and coding\. Compared with the strongest baseline on each benchmark, its gains reach up to 20\.2 percentage points on GPQA\-Diamond, 48\.4 on HumanEval, and 35\.0 on MBPP\+\.

### 4\.3Ablation Studies

We examine four design choices in LayerTrim: the pruning target, feature\-importance score, layer selection, and number of selected MLPs\. Table[2](https://arxiv.org/html/2610.19121#S4.T2)reports results onQwen3\-14Busing AIME25 and GPQA\-Diamond\. Detailed experimental settings are provided in Appendix[C](https://arxiv.org/html/2610.19121#A3)\. Additional calibration\-length, calibration\-source, and cross\-feature selection ablations appear in Appendix[C\.1](https://arxiv.org/html/2610.19121#A3.SS1)\.

VariantAIME25GPQA\-DAvg\.Pruning targetsAttention removal0\.0±\\pm0\.00\.7±\\pm0\.30\.4Attention trimming0\.0±\\pm0\.01\.3±\\pm0\.30\.7MLP removal0\.0±\\pm0\.017\.5±\\pm1\.18\.8Joint trimming \(25% attn\.\)43\.3±\\pm3\.353\.2±\\pm4\.248\.3MLP trimming62\.2±\\pm8\.455\.4±\\pm2\.458\.8Feature\-importance score𝔼⁡\[ϕj2\]\\mathbb\{E\}\[\\phi\_\{j\}^\{2\}\]61\.1±\\pm5\.148\.1±\\pm3\.354\.6‖wj‖22\\\|w\_\{j\}\\\|\_\{2\}^\{2\}22\.2±\\pm1\.941\.1±\\pm3\.031\.7Var⁡\(ϕj\)​‖wj‖22\\operatorname\{Var\}\(\\phi\_\{j\}\)\\\|w\_\{j\}\\\|\_\{2\}^\{2\}\(FLAP\)60\.0±\\pm8\.849\.5±\\pm4\.554\.8𝔼⁡\[ϕj2\]​‖wj‖22\\mathbb\{E\}\[\\phi\_\{j\}^\{2\}\]\\\|w\_\{j\}\\\|\_\{2\}^\{2\}\(Ours\)64\.4±\\pm5\.149\.7±\\pm2\.957\.1Layer selection and distributionMLP\-only BI56\.7±\\pm5\.851\.2±\\pm1\.554\.0Random50\.0±\\pm8\.846\.3±\\pm0\.848\.2All\-layer \(K=40K=40\)50\.0±\\pm8\.853\.5±\\pm0\.951\.8Block BI \(Ours\)64\.4±\\pm5\.149\.7±\\pm2\.957\.1Number of selected MLPsK=1\.5​NK=1\.5N66\.7±\\pm3\.346\.6±\\pm5\.456\.7K=3​NK=3N57\.8±\\pm3\.854\.9±\\pm1\.556\.4K=2​NK=2N\(Ours\)64\.4±\\pm5\.149\.7±\\pm2\.957\.1Table 2:Ablations onQwen3\-14B\. Scores \(%\) are mean±\\pmsample std over three seeds\. Avg\. averages AIME25 and GPQA\-Diamond; bold marks the best average in each group\. We remove approximately 1\.5B parameters in the first group and 2\.312B \(N=7N=7\) in the others\. See Appendix[C](https://arxiv.org/html/2610.19121#A3)for experimental settings\.#### Pruning targets\.

Figure[2](https://arxiv.org/html/2610.19121#S1.F2)\(b\) shows that MLP trimming produces the lowest final\-hidden NMSE\. The accuracy results in Table[2](https://arxiv.org/html/2610.19121#S4.T2)show the same pattern: MLP\-only trimming preserves substantially higher average accuracy than whole\-module removal or attention pruning\. It also outperforms joint attention–MLP trimming, which allocates 25% of the pruning budget to attention\. These results show that MLP\-only trimming minimizes pruning damage among the tested choices in both representation error and task accuracy\.

#### Feature\-Importance score\.

We compare alternative criteria for selecting MLP features during width reduction\. Scores based on activations achieve higher average accuracy than weight\-only ranking\. Our feature\-importance score achieves the highest average accuracy among the evaluated criteria, outperforming activation\-only ranking and FLAP’s variance\-based score\.

#### Layer selection and distribution\.

The accuracy results in Table[2](https://arxiv.org/html/2610.19121#S4.T2)support the BI\-based allocation in Figure[2](https://arxiv.org/html/2610.19121#S1.F2)\(d\): selecting low\-BI layers outperforms random selection and all\-layer trimming in average accuracy\. Block BI also achieves higher average accuracy than MLP\-only BI, which measures changes in the residual stream before and after the MLP update\. This supports measuring redundancy across the full decoder block rather than the MLP update alone\.

#### Number of selected MLPs\.

Under a fixed budget,K=1\.5​NK=1\.5Nachieves the highest AIME25 accuracy, whileK=3​NK=3Nachieves the highest GPQA\-Diamond accuracy\. The defaultK=2​NK=2Ngives the highest average accuracy by a small margin, providing a practical balance between the two tasks\.

Lp/Lt=7/40L\_\{p\}/L\_\{t\}=7/40Lp/Lt=9/40L\_\{p\}/L\_\{t\}=9/40MethodTypePrefill \(ms\)Decode \(ms/tok\)Peak Mem\. \(GiB\)Acc\. \(%\)Prefill \(ms\)Decode \(ms/tok\)Peak Mem\. \(GiB\)Acc\. \(%\)Qwen3\-14B\(Dense\)588 \(1\.00×\\times\)38\.0 \(1\.00×\\times\)28\.1 \(100%\)76\.7 \(100\.0%\)588 \(1\.00×\\times\)38\.0 \(1\.00×\\times\)28\.1 \(100%\)76\.7 \(100\.0%\)ShortGPTDepth502 \(1\.17×\\times\)31\.8 \(1\.19×\\times\)23\.8 \(84%\)6\.2 \(8\.1%\)470 \(1\.25×\\times\)30\.1 \(1\.26×\\times\)22\.5 \(80%\)0\.4 \(0\.5%\)LaCoDepth497 \(1\.18×\\times\)31\.8 \(1\.19×\\times\)23\.8 \(84%\)0\.0 \(0\.0%\)466 \(1\.26×\\times\)30\.1 \(1\.26×\\times\)22\.5 \(80%\)0\.0 \(0\.0%\)GRASPDepth498 \(1\.18×\\times\)34\.0 \(1\.12×\\times\)23\.8 \(85%\)0\.1 \(0\.1%\)466 \(1\.26×\\times\)32\.8 \(1\.16×\\times\)22\.6 \(80%\)0\.0 \(0\.0%\)FLAPWidth513 \(1\.15×\\times\)32\.6 \(1\.16×\\times\)23\.6 \(84%\)53\.6 \(69\.9%\)482 \(1\.22×\\times\)31\.2 \(1\.22×\\times\)22\.4 \(80%\)44\.9 \(58\.5%\)RESPWidth523 \(1\.12×\\times\)32\.9 \(1\.15×\\times\)23\.8 \(85%\)65\.0 \(84\.7%\)493 \(1\.19×\\times\)31\.4 \(1\.21×\\times\)22\.6 \(80%\)58\.3 \(76\.0%\)CoCurveWidth509 \(1\.16×\\times\)32\.8 \(1\.16×\\times\)23\.8 \(85%\)27\.1 \(35\.3%\)508 \(1\.16×\\times\)31\.4 \(1\.21×\\times\)22\.5 \(80%\)13\.3 \(17\.3%\)LayerTrim \(Ours\)Width510 \(1\.15×\\times\)32\.8 \(1\.16×\\times\)23\.8 \(85%\)68\.3 \(89\.0%\)474 \(1\.24×\\times\)31\.3 \(1\.21×\\times\)22\.6 \(80%\)65\.1 \(84\.9%\)Phi\-4\-reasoning\-plus\(Dense\)635 \(1\.00×\\times\)37\.7 \(1\.00×\\times\)28\.2 \(100%\)71\.0 \(100\.0%\)635 \(1\.00×\\times\)37\.7 \(1\.00×\\times\)28\.2 \(100%\)71\.0 \(100\.0%\)ShortGPTDepth539 \(1\.18×\\times\)31\.5 \(1\.20×\\times\)23\.6 \(84%\)2\.2 \(3\.1%\)497 \(1\.28×\\times\)29\.8 \(1\.27×\\times\)22\.3 \(79%\)1\.8 \(2\.5%\)LaCoDepth540 \(1\.17×\\times\)31\.5 \(1\.20×\\times\)23\.6 \(84%\)0\.0 \(0\.0%\)505 \(1\.26×\\times\)29\.8 \(1\.27×\\times\)22\.3 \(79%\)0\.0 \(0\.0%\)GRASPDepth548 \(1\.16×\\times\)32\.6 \(1\.16×\\times\)23\.7 \(84%\)0\.1 \(0\.1%\)514 \(1\.23×\\times\)31\.1 \(1\.21×\\times\)22\.4 \(80%\)0\.0 \(0\.0%\)FLAPWidth521 \(1\.22×\\times\)32\.4 \(1\.16×\\times\)23\.5 \(83%\)42\.4 \(59\.7%\)498 \(1\.27×\\times\)30\.8 \(1\.22×\\times\)22\.2 \(79%\)9\.2 \(13\.0%\)RESPWidth544 \(1\.17×\\times\)32\.4 \(1\.16×\\times\)23\.7 \(84%\)49\.9 \(70\.3%\)514 \(1\.23×\\times\)30\.9 \(1\.22×\\times\)22\.4 \(80%\)34\.6 \(48\.7%\)CoCurveWidth533 \(1\.19×\\times\)32\.6 \(1\.15×\\times\)23\.6 \(84%\)26\.5 \(37\.3%\)515 \(1\.23×\\times\)30\.9 \(1\.22×\\times\)22\.3 \(79%\)12\.9 \(18\.2%\)LayerTrim \(Ours\)Width557 \(1\.14×\\times\)32\.3 \(1\.17×\\times\)23\.7 \(84%\)62\.2 \(87\.6%\)519 \(1\.22×\\times\)30\.8 \(1\.22×\\times\)22\.4 \(80%\)48\.9 \(68\.9%\)Magistral\-Small\-2506\(Dense\)996 \(1\.00×\\times\)57\.4 \(1\.00×\\times\)44\.7 \(100%\)70\.3 \(100\.0%\)996 \(1\.00×\\times\)57\.4 \(1\.00×\\times\)44\.7 \(100%\)70\.3 \(100\.0%\)ShortGPTDepth836 \(1\.19×\\times\)47\.9 \(1\.20×\\times\)37\.4 \(84%\)0\.0 \(0\.0%\)779 \(1\.28×\\times\)45\.2 \(1\.27×\\times\)35\.3 \(79%\)0\.0 \(0\.0%\)LaCoDepth836 \(1\.19×\\times\)47\.9 \(1\.20×\\times\)37\.4 \(84%\)0\.0 \(0\.0%\)781 \(1\.28×\\times\)45\.2 \(1\.27×\\times\)35\.3 \(79%\)0\.0 \(0\.0%\)GRASPDepth797 \(1\.25×\\times\)49\.2 \(1\.17×\\times\)37\.5 \(84%\)0\.0 \(0\.0%\)763 \(1\.31×\\times\)46\.9 \(1\.23×\\times\)35\.4 \(79%\)0\.0 \(0\.0%\)FLAPWidth783 \(1\.27×\\times\)48\.8 \(1\.18×\\times\)37\.4 \(84%\)20\.8 \(29\.6%\)789 \(1\.26×\\times\)46\.4 \(1\.24×\\times\)35\.3 \(79%\)10\.6 \(15\.1%\)RESPWidth810 \(1\.23×\\times\)48\.8 \(1\.18×\\times\)37\.5 \(84%\)44\.2 \(62\.9%\)806 \(1\.24×\\times\)46\.3 \(1\.24×\\times\)35\.4 \(79%\)24\.3 \(34\.6%\)CoCurveWidth778 \(1\.28×\\times\)48\.8 \(1\.18×\\times\)37\.4 \(84%\)25\.0 \(35\.6%\)789 \(1\.26×\\times\)46\.4 \(1\.24×\\times\)35\.3 \(79%\)9\.6 \(13\.7%\)LayerTrim \(Ours\)Width834 \(1\.19×\\times\)48\.8 \(1\.18×\\times\)37\.5 \(84%\)57\.8 \(82\.2%\)781 \(1\.27×\\times\)46\.3 \(1\.24×\\times\)35\.4 \(79%\)45\.7 \(65\.0%\)

Table 3:Inference efficiency under matched parameter budgets across three RLMs\. We report prefill latency, decode latency per token, peak GPU memory, and average task accuracy\. Parentheses indicate speedup over the dense model for latency, relative memory usage for peak GPU memory, and accuracy retention relative to the dense model\. All methods are evaluated using their deployed pruned structures\.

### 4\.4Efficiency Measurement

#### Measurement settings\.

We measure prefill latency, decode latency, and peak GPU memory in bfloat16 at batch size 1 on a single NVIDIA RTX 6000 Ada \(48 GB\)\. Prefill latency is measured on a 2,048\-token prompt, and decode latency is averaged over 128 greedy decoding steps with a static KV cache\. Each configuration runs two warm\-up iterations followed by ten timed runs, synchronized usingtorch\.cuda\.synchronize\(\)\. We report median latency and peak GPU memory during inference, excluding model loading and pruning overhead\. All methods are evaluated under matched parameter budgets\.

#### Efficiency results\.

Table[3](https://arxiv.org/html/2610.19121#S4.T3)compares inference efficiency across three RLMs at two pruning budgets\. At the higher pruning budget, LayerTrim achieves prefill speedups of 1\.24×\\times, 1\.22×\\times, and 1\.27×\\times, and decode speedups of 1\.21×\\times, 1\.22×\\times, and 1\.24×\\timesonQwen3\-14B,Phi\-4\-reasoning\-plus, andMagistral\-Small\-2506, respectively\. Peak GPU memory is reduced by 19\.7%, 20\.3%, and 20\.8% on the corresponding models compared with their dense counterparts\. While achieving inference costs comparable to existing structured pruning methods, LayerTrim retains the highest average accuracy across all three models at both pruning budgets\. LayerTrim also requires only 27\.9 s of pruning preparation on Qwen3\-14B, 94× and 140× faster than RESP at N = 7 and N = 9, respectively \(Appendix[B\.1](https://arxiv.org/html/2610.19121#A2.SS1)\)\.

## 5Discussions

### 5\.1Analysis of LayerTrim

#### Attention preservation\.

We analyze how well MLP trimming preserves attention patterns on reasoning traces\. Panel[4\(a\)](https://arxiv.org/html/2610.19121#S5.F4.sf1)shows that MLP trimming produces the lowest attention KL divergence across the tested budgets\. At approximately 1\.5B removed parameters, panel[4\(b\)](https://arxiv.org/html/2610.19121#S5.F4.sf2)also shows the highest overlap with the dense model’s top\-8 attended tokens across all query–key distance bins, including 16K–32K\. These results suggest that MLP\-only trimming better preserves attention to earlier tokens during long reasoning, providing an attention\-level explanation for why MLP trimming better preserves reasoning performance\.

#### Feature\-Importance Surrogate\.

To reduce calibration overhead, we approximate the squared MLP output change by summing feature importance scores without computing cross\-feature terms\. For a setRRof removed features, we compare this approximation with the squared output change measured on the same dense\-model inputs:

Δℓ​\(R\)\\displaystyle\\Delta\_\{\\ell\}\(R\)=𝔼x​\[‖∑j∈Rcj\(ℓ\)​\(x\)‖22\]\\displaystyle=\\mathbb\{E\}\_\{x\}\\\!\\left\[\\left\\\|\\sum\_\{j\\in R\}c\_\{j\}^\{\(\\ell\)\}\(x\)\\right\\\|\_\{2\}^\{2\}\\right\]≈∑j∈Rsj\(ℓ\)\\displaystyle\\approx\\sum\_\{j\\in R\}s\_\{j\}^\{\(\\ell\)\}\(4\)
Panel[4\(c\)](https://arxiv.org/html/2610.19121#S5.F4.sf3)shows that the scores rank the measured changes across held\-out layer–width conditions, with Spearmanρ=0\.999\\rho=0\.999\. Table[7](https://arxiv.org/html/2610.19121#A3.T7)in Appendix[C\.1](https://arxiv.org/html/2610.19121#A3.SS1)reports comparable average accuracy to full\-error greedy selection with 13\.2×\\timesfaster calibration and selection\. These results support using the low\-overhead approximation to guide feature selection\.

### 5\.2Quantitative Analysis of Reasoning Traces

\(a\)Attention distribution\(b\)Attention overlap\(c\)Feature importance
Figure 4:Attention preservation and feature importance onQwen3\-14B\. \(a\) Attention KL divergence across pruning budgets\. \(b\) Overlap with the dense model’s top\-8 attended tokens by query–key distance, with approximately 1\.5B parameters removed\. \(c\) Summed feature importance scores versus squared MLP output change\. Experimental details are provided in Appendix[D](https://arxiv.org/html/2610.19121#A4)\.\(a\)Repetition, GPQA\-D\(b\)Coding, HumanEval\(c\)Arithmetic, AIME26
Figure 5:Quantitative analysis results comparing LayerTrim to existing pruning methodsFigure[5](https://arxiv.org/html/2610.19121#S5.F5)complements the experimental results in Table[1](https://arxiv.org/html/2610.19121#S4.T1)with diagnostics of the generated outputs and reasoning paths\. Panel[5\(a\)](https://arxiv.org/html/2610.19121#S5.F5.sf1)examines GPQA\-Diamond responses fromPhi\-4\-reasoning\-plusatLp/Lt=9/40L\_\{p\}/L\_\{t\}=9/40, measuring the ratio of responses reaching the max token limit and exhibiting strong repetition\. Here, we flag strong repetition when two consecutive, non\-overlapping 512\-token windows each satisfy1−U8/N8≥0\.81\-U\_\{8\}/N\_\{8\}\\geq 0\.8, whereU8U\_\{8\}andN8N\_\{8\}denote the numbers of unique and total token 8\-grams\. LaCo exhibits 100% token\-limit hits despite 0% detected strong repetition: its outputs continue as incoherent token fragments without forming the sustained exact\-token repetition captured by this criterion \(see Appendix[E](https://arxiv.org/html/2610.19121#A5)\)\. LayerTrim achieves token\-limit hits and strong repetition rates of 24\.4% and 21\.9%, respectively, compared with 70\.2% and 49\.2% for FLAP\. Together with its higher accuracy, these results show that LayerTrim combines stronger task performance with substantially fewer repetitive and budget\-exhausting outputs\.

Panel[5\(b\)](https://arxiv.org/html/2610.19121#S5.F5.sf2)evaluates code formulation on HumanEval usingQwen3\-14BatLp/Lt=9/40L\_\{p\}/L\_\{t\}=9/40\. The compilable\-code rate measures the ratio of responses containing at least one extracted Python code that compiles and defines the requested function\. LayerTrim yields 99\.6%, close to Dense’s 99\.4%, while the second\-best RESP yields 85\.0%\. Its advantage also extends to functional problem solving: among responses containing a qualifying candidate, benchmark correctness is 88\.4% for LayerTrim vs\. 72\.7% for RESP\. Thus, the overall HumanEval accuracy gap in Table[1](https://arxiv.org/html/2610.19121#S4.T1)\(88\.0% versus 62\.0%\) is accompanied by advantages in both forming compilable code and producing correct solutions when such code is present\.

Panel[5\(c\)](https://arxiv.org/html/2610.19121#S5.F5.sf3)examines arithmetic consistency on AIME26 usingPhi\-4\-reasoning\-plusatLp/Lt=7/40L\_\{p\}/L\_\{t\}=7/40\. We broadly extract numeric equalities, including those embedded in prose or chained expressions, deduplicate them within each response, and report the fraction that hold under exact arithmetic evaluation\. N/A in LaCo denotes that no eligible equality was extracted\. LayerTrim attains 94\.8% consistency, within 1\.6pp of Dense’s 96\.4%, compared with 83\.8% for FLAP and 73\.6% for CoCurve\. While RESP exhibits similar arithmetic consistency of 94\.1%, LayerTrim achieves substantially higher accuracy \(60\.0% vs\. 50\.0%\)\. This suggests that LayerTrim’s advantage extends beyond preserving individual arithmetic operations to better preserving the problem\-level reasoning needed to integrate locally correct calculations into a correct final solution\.

## 6Conclusion

LayerTrim reduces the inference cost of reasoning language models by selectively trimming MLP features while preserving attention and decoder depth\. It uses dense\-model reasoning traces to identify redundant layers and retain high\-importance features, without gradients or recovery training\. Across three models and five benchmarks, LayerTrim achieves higher average accuracy than the evaluated structured pruning baselines at both pruning budgets, while achieving comparable inference speedups\. LayerTrim offers a step toward more efficient deployment of reasoning language models through training\-free structured pruning\.

## Limitations

Our evaluation focuses on three reasoning language models and five benchmarks covering mathematical reasoning, scientific reasoning, and code generation\. Although LayerTrim outperforms the evaluated pruning baselines in most settings, experiments on additional model architectures, scales, and reasoning tasks would help assess its generalizability\. We also evaluate fixed pruning budgets and layer allocation settings\. Adapting these choices to individual models and tasks may further improve the trade\-off between model compression and reasoning accuracy\.

## Ethical considerations

Structured pruning modifies the internal architecture of language models and may affect their reasoning behavior\. Although LayerTrim aims to preserve reasoning accuracy, pruned models may still produce incorrect answers, incomplete reasoning traces, or unexpected outputs\. These risks should be considered when deploying compressed models in applications where reasoning accuracy is critical\. Our experiments use publicly available models and benchmarks and do not involve human participants or the collection of personal information\.

## References

- M\. Abdin, S\. Agarwal, A\. Awadallah, V\. Balachandran, H\. Behl, L\. Chen, G\. de Rosa, S\. Gunasekar, M\. Javaheripi, N\. Joshi, P\. Kauffmann, Y\. Lara, C\. C\. T\. Mendes, A\. Mitra, B\. Nushi, D\. Papailiopoulos, O\. Saarikivi, S\. Shah, V\. Shrivastava, V\. Vineet, Y\. Wu, S\. Yousefi, and G\. ZhengPhi\-4\-reasoning technical report\.arXiv preprint arXiv:2504\.21318\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.21318),[Link](https://arxiv.org/abs/2504.21318)Cited by:[Appendix F](https://arxiv.org/html/2610.19121#A6.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2610.19121#S4.I1.i1.p1.1)\.
- Anet al\.\(2024\)Y\. An, X\. Zhao, T\. Yu, M\. Tang, and J\. WangFluctuation\-based adaptive structured pruning for large language models\.InProceedings of the Thirty\-Eighth AAAI Conference on Artificial Intelligence and Thirty\-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence,External Links:ISBN 978\-1\-57735\-887\-9,[Link](https://doi.org/10.1609/aaai.v38i10.28960),[Document](https://dx.doi.org/10.1609/aaai.v38i10.28960)Cited by:[§1](https://arxiv.org/html/2610.19121#S1.p1.1),[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1),[2nd item](https://arxiv.org/html/2610.19121#S4.I2.i2.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2107.03374),[Link](https://arxiv.org/abs/2107.03374)Cited by:[Appendix F](https://arxiv.org/html/2610.19121#A6.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2610.19121#S4.I1.i2.p1.1)\.
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-AI, D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Ding, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Chen, J\. Yuan, J\. Tu, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. You, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. ZhangDeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-025-09422-z),[Link](https://doi.org/10.1038/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2610.19121#S1.p1.1)\.
- Dinget al\.\(2026\)L\. Ding, A\. Zhao, F\. Ye, Z\. Chen, and X\. ShenFrom LLMs to LRMs: rethinking pruning for reasoning\-centric models\.arXiv preprint arXiv:2601\.18091\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.18091),[Link](https://arxiv.org/abs/2601.18091)Cited by:[§1](https://arxiv.org/html/2610.19121#S1.p1.1),[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px2.p1.1)\.
- Frantar and Alistarh \(2023\)E\. Frantar and D\. AlistarhSparseGPT: massive language models can be accurately pruned in one\-shot\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 10323–10337\.External Links:[Link](https://proceedings.mlr.press/v202/frantar23a.html)Cited by:[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1)\.
- Gonget al\.\(2026\)Z\. Gong, Z\. Zeng, T\. Wang, Y\. Wang, H\. Anada, Z\. Wang, M\. Xiao, C\. Yuen, and W\. Y\. B\. LimCoCurve: cross\-module co\-pruning curvature for structured LLM pruning\.arXiv preprint arXiv:2607\.17568\.External Links:[Link](https://arxiv.org/abs/2607.17568)Cited by:[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1),[2nd item](https://arxiv.org/html/2610.19121#S4.I2.i2.p1.1)\.
- Kimet al\.\(2026\)M\. Kim, V\. Yun, Y\. Kim, S\. Cho, W\. Lim, and S\. LeeRethinking layer redundancy: calibration matters more than search in LLM depth pruning\.arXiv preprint arXiv:2604\.24938\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2604.24938),[Link](https://arxiv.org/abs/2604.24938)Cited by:[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 21558–21572\.External Links:[Document](https://dx.doi.org/10.52202/075280-0943),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/43e9d647ccd3e4b7b5baab53f0368686-Paper-Conference.pdf)Cited by:[Appendix F](https://arxiv.org/html/2610.19121#A6.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2610.19121#S4.I1.i2.p1.1)\.
- Liuet al\.\(2025\)K\. Liu, Y\. Zhang, N\. Cheng, Z\. Li, S\. Wang, and J\. XiaoGRASP: replace redundant layers with adaptive singular parameters for efficient model compression\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 26333–26348\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1338/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1338),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2610.19121#S4.I2.i1.p1.1)\.
- Lucaset al\.\(2026\)R\. Lucas, K\. Behdin, Z\. Wang, Q\. Song, S\. Tang, and R\. MazumderReasoning models can be accurately pruned via chain\-of\-thought reconstruction\.InInternational Conference on Learning Representations,C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(Eds\.\),Vol\.2026,pp\. 142840–142852\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/e70ffb3f05096e62b5077d8e1b62e668-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.19121#S1.p3.1),[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.19121#S3.SS2.SSS0.Px1.p1.1)\.
- Maet al\.\(2023\)X\. Ma, G\. Fang, and X\. WangLLM\-Pruner: on the structural pruning of large language models\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 21702–21720\.External Links:[Document](https://dx.doi.org/10.52202/075280-0950),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/44956951349095f74492a5471128a7e0-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.19121#S1.p1.1),[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1)\.
- Menet al\.\(2025\)X\. Men, M\. Xu, Q\. Zhang, Q\. Yuan, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. ChenShortGPT: layers in large language models are more redundant than you expect\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 20192–20204\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1035/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1035),ISBN 979\-8\-89176\-256\-5Cited by:[Figure 2](https://arxiv.org/html/2610.19121#S1.F2),[§1](https://arxiv.org/html/2610.19121#S1.p3.1),[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2610.19121#S3.SS2.SSS0.Px2.p1.1),[1st item](https://arxiv.org/html/2610.19121#S4.I2.i1.p1.1)\.
- Mistral\-AIet al\.\(2025\)Mistral\-AI, A\. Rastogi, A\. Q\. Jiang, A\. Lo, G\. Berrada, G\. Lample, J\. Rute, J\. Barmentlo, K\. Yadav, K\. Khandelwal, K\. R\. Chandu, L\. Blier, L\. Saulnier, M\. Dinot, M\. Darrin, N\. Gupta, R\. Soletskyi, S\. Vaze, T\. L\. Scao, Y\. Wang, A\. Yang, A\. H\. Liu, A\. Sablayrolles, A\. Héliou, A\. Martin, A\. Ehrenberg, A\. Agarwal, A\. Roux, A\. Darcet, A\. Mensch, B\. Bout, B\. Rozière, B\. D\. Monicault, C\. Bamford, C\. Wallenwein, C\. Renaudin, C\. Lanfranchi, D\. Dabert, D\. Mizelle, D\. de las Casas, E\. Chane\-Sane, E\. Fugier, E\. B\. Hanna, G\. Delerce, G\. Guinet, G\. Novikov, G\. Martin, H\. Jaju, J\. Ludziejewski, J\. Chabran, J\. Delignon, J\. Studnia, J\. Amar, J\. S\. Roberts, J\. Denize, K\. Saxena, K\. Jain, L\. Zhao, L\. Martin, L\. Gao, L\. R\. Lavaud, M\. Pellat, M\. Guillaumin, M\. Felardos, M\. Augustin, M\. Seznec, N\. Raghuraman, O\. Duchenne, P\. Wang, P\. von Platen, P\. Saffer, P\. Jacob, P\. Wambergue, P\. Kurylowicz, P\. R\. Muddireddy, P\. Chagniot, P\. Stock, P\. Agrawal, R\. Sauvestre, R\. Delacourt, S\. Gandhi, S\. Subramanian, S\. Dalal, S\. Gandhi, S\. Ghosh, S\. Mishra, S\. Aithal, S\. Antoniak, T\. Schueller, T\. Lavril, T\. Robert, T\. Wang, T\. Lacroix, V\. Nemychnikova, V\. Paltz, V\. Richard, W\. Li, W\. Marshall, X\. Zhang, and Y\. TangMagistral\.arXiv preprint arXiv:2506\.10910\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2506.10910),[Link](https://arxiv.org/abs/2506.10910)Cited by:[Appendix F](https://arxiv.org/html/2610.19121#A6.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2610.19121#S4.I1.i1.p1.1)\.
- Nguyenet al\.\(2026\)H\. L\. Nguyen, H\. H\. Tran, T\. Tran, and D\. D\. LeOBC\-Prune: outcome\-based calibration for large reasoning model pruning\.arXiv preprint arXiv:2609\.17890\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2609.17890),[Link](https://arxiv.org/abs/2609.17890)Cited by:[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px2.p1.1)\.
- Reinet al\.\(2024\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: a graduate\-level Google\-Proof Q&A benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[Appendix F](https://arxiv.org/html/2610.19121#A6.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2610.19121#S4.I1.i2.p1.1)\.
- Shazeer \(2020\)N\. ShazeerGLU variants improve transformer\.arXiv preprint arXiv:2002\.05202\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2002.05202),[Link](https://arxiv.org/abs/2002.05202)Cited by:[§3\.1](https://arxiv.org/html/2610.19121#S3.SS1.SSS0.Px2.p1.1)\.
- Sunet al\.\(2024\)M\. Sun, Z\. Liu, A\. Bair, and Z\. KolterA simple and effective pruning approach for large language models\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 4942–4964\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/14c856c7a41297804de4c4890e846b25-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2026\)Z\. Wang, E\. Diao, Q\. Le, P\. Wang, M\. Lee, S\. Yeh, E\. Stupachenko, H\. Feng, and L\. YangFrom local to global: revisiting structured pruning paradigms for large language models\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 35720–35739\.External Links:[Link](https://aclanthology.org/2026.acl-long.1653/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1653),ISBN 979\-8\-89176\-390\-6Cited by:[Appendix B](https://arxiv.org/html/2610.19121#A2.SS0.SSS0.Px4.p1.1)\.
- Wanget al\.\(2025\)Z\. Wang, E\. Diao, Q\. Le, P\. Wang, G\. Wang, M\. Lee, S\. Yeh, and L\. YangThink before you prune: self\-reflective structured pruning for reasoning language models\.arXiv preprint arXiv:2512\.02185\.Note:PreprintExternal Links:[Document](https://dx.doi.org/10.48550/arXiv.2512.02185),[Link](https://arxiv.org/abs/2512.02185)Cited by:[Appendix B](https://arxiv.org/html/2610.19121#A2.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.19121#S1.p3.1),[§1](https://arxiv.org/html/2610.19121#S1.p4.1),[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.19121#S3.SS2.SSS0.Px1.p1.1),[2nd item](https://arxiv.org/html/2610.19121#S4.I2.i2.p1.1)\.
- Xiaet al\.\(2024\)M\. Xia, T\. Gao, Z\. Zeng, and D\. ChenSheared LLaMA: accelerating language model pre\-training via structured pruning\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 5385–5409\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/160adf2dc118a920e7858484b92a37d8-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[Appendix F](https://arxiv.org/html/2610.19121#A6.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.19121#S1.p1.1),[§1](https://arxiv.org/html/2610.19121#S1.p2.1),[1st item](https://arxiv.org/html/2610.19121#S4.I1.i1.p1.1)\.
- Yanget al\.\(2024\)Y\. Yang, Z\. Cao, and H\. ZhaoLaCo: large language model pruning via layer collapse\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 6401–6417\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.372/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.372)Cited by:[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2610.19121#S4.I2.i1.p1.1)\.
- Yunet al\.\(2026\)V\. Yun, Y\. Kim, W\. Lim, Y\. Heo, M\. Kim, and S\. LeeLocality\-aware redundancy pruning for LLM depth compression\.arXiv preprint arXiv:2605\.27786\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.27786),[Link](https://arxiv.org/abs/2605.27786)Cited by:[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1)\.
- Zhang and Math\-AI \(2025\)Y\. Zhang and T\. Math\-AIAmerican invitational mathematics examination \(AIME\) 2025\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime25)Cited by:[Appendix F](https://arxiv.org/html/2610.19121#A6.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2610.19121#S4.I1.i2.p1.1)\.
- Zhang and Math\-AI \(2026\)Y\. Zhang and T\. Math\-AIAmerican invitational mathematics examination \(AIME\) 2026\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime26)Cited by:[Appendix F](https://arxiv.org/html/2610.19121#A6.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2610.19121#S4.I1.i2.p1.1)\.
- Zhuet al\.\(2026\)Y\. Zhu, J\. Wang, and C\. ShenHigh\-fidelity pruning for large language models\.arXiv preprint arXiv:2603\.08083\.External Links:[Link](https://arxiv.org/abs/2603.08083)Cited by:[§2](https://arxiv.org/html/2610.19121#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix

## Appendix ALayerTrim Procedure

Algorithm[1](https://arxiv.org/html/2610.19121#alg1)summarizes the procedure in Section[3\.2](https://arxiv.org/html/2610.19121#S3.SS2)for uniform gated MLPs\. Here,LLis the decoder depth,CCandIIare the hidden and intermediate widths, andPlayerP\_\{\\mathrm\{layer\}\}counts the attention and MLP projection parameters in one original block\. Calibration averages are taken over token positions\.

Algorithm 1LayerTrim: reasoning\-calibrated structured MLP pruning\.1:Dense model

ℳ\\mathcal\{M\}; calibration prompts

𝒬\\mathcal\{Q\}; layer\-equivalent budget

NN
2:Compressed model

ℳ~\\widetilde\{\\mathcal\{M\}\}with unchanged attention and depth

3:Generate reasoning traces

𝒯←Generate​\(ℳ,𝒬\)\\mathcal\{T\}\\leftarrow\\textsc\{Generate\}\(\\mathcal\{M\},\\mathcal\{Q\}\)\.

4:Form

𝒟\\mathcal\{D\}from each complete prompt and up to 2,048 generated tokens of its trace in

𝒯\\mathcal\{T\}\.

5:With gradients disabled, teacher\-force

ℳ\\mathcal\{M\}on

𝒟\\mathcal\{D\}and accumulate block cosine similarities and feature second moments

a^j\(ℓ\)=𝔼𝒟​\[\(ϕj\(ℓ\)\)2\]\\widehat\{a\}\_\{j\}^\{\(\\ell\)\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[\(\\phi\_\{j\}^\{\(\\ell\)\}\)^\{2\}\]\.

6:for

ℓ=1,…,L\\ell=1,\\ldots,Ldo

7:

BIℓ←1−𝔼x∼𝒟​\[cos⁡\(hin\(ℓ\)​\(x\),hout\(ℓ\)​\(x\)\)\]\\mathrm\{BI\}\_\{\\ell\}\\leftarrow 1\-\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\\!\\left\[\\cos\\\!\\left\(h\_\{\\mathrm\{in\}\}^\{\(\\ell\)\}\(x\),h\_\{\\mathrm\{out\}\}^\{\(\\ell\)\}\(x\)\\right\)\\right\]\.

8:endfor

9:

K←2​NK\\leftarrow 2N; require

K≤LK\\leq L\.

10:

ℒK←BottomK​\(\{BIℓ\}ℓ=1L,K\)\\mathcal\{L\}\_\{K\}\\leftarrow\\textsc\{BottomK\}\(\\\{\\mathrm\{BI\}\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\},K\)\.

11:

m←round⁡\(I−Player/\(6​C\)\)m\\leftarrow\\operatorname\{round\}\(I\-P\_\{\\mathrm\{layer\}\}/\(6C\)\)\.

12:Require

0<m≤I0<m\\leq I\.

13:

ℳ~←Copy​\(ℳ\)\\widetilde\{\\mathcal\{M\}\}\\leftarrow\\textsc\{Copy\}\(\\mathcal\{M\}\)\.

14:for all

ℓ∈ℒK\\ell\\in\\mathcal\{L\}\_\{K\}do

15:for

j=1,…,Ij=1,\\ldots,Ido

16:

s^j\(ℓ\)←a^j\(ℓ\)∥Wdown\(ℓ\)\[:,j\]∥22\\widehat\{s\}\_\{j\}^\{\(\\ell\)\}\\leftarrow\\widehat\{a\}\_\{j\}^\{\(\\ell\)\}\\\|W\_\{\\mathrm\{down\}\}^\{\(\\ell\)\}\[:,j\]\\\|\_\{2\}^\{2\}\.

17:endfor

18:

Sℓ⋆←TopM​\(\{s^j\(ℓ\)\}j=1I,m\)S\_\{\\ell\}^\{\\star\}\\leftarrow\\textsc\{TopM\}\(\\\{\\widehat\{s\}\_\{j\}^\{\(\\ell\)\}\\\}\_\{j=1\}^\{I\},m\)\.

19:

W~gate\(ℓ\)←Wgate\(ℓ\)\[Sℓ⋆,:\]\\widetilde\{W\}\_\{\\mathrm\{gate\}\}^\{\(\\ell\)\}\\leftarrow W\_\{\\mathrm\{gate\}\}^\{\(\\ell\)\}\[S\_\{\\ell\}^\{\\star\},:\];

W~up\(ℓ\)←Wup\(ℓ\)\[Sℓ⋆,:\]\\widetilde\{W\}\_\{\\mathrm\{up\}\}^\{\(\\ell\)\}\\leftarrow W\_\{\\mathrm\{up\}\}^\{\(\\ell\)\}\[S\_\{\\ell\}^\{\\star\},:\]\.

20:

W~down\(ℓ\)←Wdown\(ℓ\)\[:,Sℓ⋆\]\\widetilde\{W\}\_\{\\mathrm\{down\}\}^\{\(\\ell\)\}\\leftarrow W\_\{\\mathrm\{down\}\}^\{\(\\ell\)\}\[:,S\_\{\\ell\}^\{\\star\}\]\.

21:Install these projections in

ℳ~\\widetilde\{\\mathcal\{M\}\}and set this MLP’s width to

mm\.

22:endfor

23:return

ℳ~\\widetilde\{\\mathcal\{M\}\}

TopMreturns the indices of themmhighest\-scoring features\. The same indices select gate and up rows and down\-projection columns\.

## Appendix BMain Evaluation Implementation Details

#### Pruning budgets\.

For each model, we set the pruning budget toBN=N​PlayerB\_\{N\}=NP\_\{\\mathrm\{layer\}\}, wherePlayerP\_\{\\mathrm\{layer\}\}is the number of attention and MLP projection parameters in one original decoder layer andN∈\{7,9\}N\\in\\\{7,9\\\}\. All methods use the same target budget, with small differences in actual removal counts because heads and channels are removed in whole units\.

#### LayerTrim allocation\.

LayerTrim selectsK=2​NK=2NMLPs using Block Influence and retains the same widthmmin each selected MLP\. Removing one gated\-MLP feature affects a gate row, an up row, and a down\-projection column, totaling3​C3Cparameters\. The realized budget is3​C​K​\(I−m\)3CK\(I\-m\), withmmset by Equation[3](https://arxiv.org/html/2610.19121#S3.E3)\.

#### Calibration settings\.

All methods use the same 24 AIME 2024 reasoning traces generated by the corresponding dense model at temperature0\.60\.6, top\-pp0\.950\.95, and seed 0\. LayerTrim uses the complete prompt and up to the first 2,048 generated tokens from each trace for statistics collection\.

#### Baseline implementations\.

For all baselines except RESP, we use the authors’ official implementations, adapted to our models and matched\-budget\. We reimplement RESP using the public GISP\([Wang et al\., 2026](https://arxiv.org/html/2610.19121#bib.bib27)\)repository, following the GISP\-based structural pruning procedure described in the RESP paper\([Wang et al\., 2025](https://arxiv.org/html/2610.19121#bib.bib14)\), with decode\-only scoring and calibration regeneration\. All baselines use the shared calibration data described above\.

#### Generation settings\.

Dense and pruned models use the corresponding native reasoning\-enabled chat template\. Magistral uses its native Tekken tokenizer and official system prompt\. Sampling uses temperature0\.60\.6, top\-pp0\.950\.95, and seeds 0, 1, and 2\. Each seed produces one response per problem\. We set the maximum output length to 32K tokens\. Inference uses BF16 in vLLM\.

#### Answer and code scoring\.

We evaluate AIME and GPQA\-Diamond by exact matching of the extracted final answers\. For coding benchmarks, we report pass@1 using HumanEval’s original tests and MBPP\+’s extended tests\.

### B\.1Calibration Overhead

We compare LayerTrim and RESP onQwen3\-14Bin BF16 using a single NVIDIA RTX 6000 Ada \(48 GB\)\. Both use the same 24 AIME 2024 calibration traces, with up to 2,048 generated tokens per trace\. We measure calibration time from model loading until the pruning masks are computed\. RESP’s trace regeneration and required intermediate checkpoint preparation are included\. Initial trace generation and final weight export are excluded\.

MethodN=7N=7N=9N=9RESP2,626\.1±\\pm18\.23,898\.1±\\pm18\.0LayerTrim \(Ours\)27\.9±\\pm0\.627\.9±\\pm0\.6

Table 4:Pruning preparation time \(seconds\) on Qwen3\-14B\. Values are mean±\\pmsample std over three runs\.LayerTrim requires substantially less preparation time than RESP at both pruning budgets \(Table[4](https://arxiv.org/html/2610.19121#A2.T4)\)\.

## Appendix CAblation Experimental Details

#### Evaluation and reporting\.

Table[2](https://arxiv.org/html/2610.19121#S4.T2)evaluatesQwen3\-14Bon AIME25 and GPQA\-Diamond using the main evaluation protocol\. The repeated default entries reuse the same evaluated configuration\.

#### Pruning targets\.

The first group uses the pruning configurations of Figure[2](https://arxiv.org/html/2610.19121#S1.F2)\(b\)\. All three trimming conditions remove approximately 1\.494B parameters; attention and MLP removal delete approximately 1\.510B and 1\.604B, respectively, because whole\-module removal has discrete budgets\. Joint trimming assigns 25% of its budget to attention and 75% to MLPs\. Attention trimming removes selected dimensions within each attention head\.

#### Design\-ablation budget and calibration\.

The remaining three groups fix the budget atN=7N=7\(approximately 2\.312B removed parameters\) and use the default reasoning\-trace calibration in Section[4\.1](https://arxiv.org/html/2610.19121#S4.SS1)\.

#### Feature\-importance scores\.

The feature\-score group fixes the 14 Block\-BI\-selected MLPs and their retained widths\. Only the within\-MLP ranking score changes\. Here,ϕj\\phi\_\{j\}denotes an intermediate activation andwj=Wdown\[:,j\]w\_\{j\}=W\_\{\\mathrm\{down\}\}\[:,j\]denotes its down\-projection column\. The default score is𝔼⁡\[ϕj2\]​‖wj‖22\\mathbb\{E\}\[\\phi\_\{j\}^\{2\}\]\\\|w\_\{j\}\\\|\_\{2\}^\{2\}\. We compare it with activation\-only ranking using𝔼⁡\[ϕj2\]\\mathbb\{E\}\[\\phi\_\{j\}^\{2\}\], weight\-only ranking using‖wj‖22\\\|w\_\{j\}\\\|\_\{2\}^\{2\}, and variance\-based ranking usingVar⁡\(ϕj\)​‖wj‖22\\operatorname\{Var\}\(\\phi\_\{j\}\)\\\|w\_\{j\}\\\|\_\{2\}^\{2\}\.

#### Layer selection and distribution\.

The layer\-selection group fixes the feature score and total removal budget\. Block BI, MLP\-only BI, and random selection each chooseK=14K=14MLPs\. MLP\-only BI measures cosine distance between the residual stream before and after the MLP update\. Random selection uses one random fixed mask\. All\-layer allocation distributes the same removal budget across all 40 MLPs\.

#### Number of selected MLPs\.

The allocation group fixes the Block BI ranking and default feature score while varyingK∈\{11,14,21\}K\\in\\\{11,14,21\\\}\. Retained widths are adjusted to preserve the same total removal budget;K=1\.5​NK=1\.5Nis rounded to 11\. SmallerKKconcentrates pruning in fewer MLPs\.

### C\.1Additional Accuracy Ablations

These experiments follow the main evaluation protocol and reuse previously evaluated controls\. Avg\. averages the displayed benchmark means before rounding\.

#### Calibration length\.

We vary the number of generated reasoning tokens used to collect statistics on the same 24 AIME 2024 traces, retaining each complete prompt\. ForQwen3\-14BatN=7N=7, we recompute Block BI and feature importance at each length while keepingK=14K=14and the parameter budget fixed\. Accuracy is stable beyond 1,024 tokens \(Table[5](https://arxiv.org/html/2610.19121#A3.T5)\); we use 2,048 as the default\.

PrefixAIME25GPQA\-DHumanEvalAvg\.51252\.2±\\pm3\.852\.9±\\pm1\.387\.8±\\pm2\.764\.31,02462\.2±\\pm6\.953\.5±\\pm1\.889\.0±\\pm0\.668\.32,04864\.4±\\pm5\.149\.7±\\pm2\.990\.2±\\pm1\.668\.14,09668\.9±\\pm6\.951\.5±\\pm5\.091\.9±\\pm0\.470\.8

Table 5:Calibration\-length ablation onQwen3\-14BatN=7N=7\. Prefix is the maximum number of generated reasoning tokens per trace; each complete prompt is also included\. HumanEval reports pass@1 \(%\); the other benchmarks report accuracy \(%\)\. The pruning budget andK=14K=14are fixed\.Calibration sourceAIME25GPQA\-DAvg\.WikiText\-20\.0±\\pm0\.020\.0±\\pm2\.310\.0AIME prompts only22\.2±\\pm6\.938\.2±\\pm4\.730\.2AIME reasoning \(Ours\)64\.4±\\pm5\.149\.7±\\pm2\.957\.1

Table 6:Calibration\-source ablation on Qwen3\-14B atN=7N=7\. Entries are accuracy \(%\); Avg\. averages the two benchmarks\. WikiText\-2 and reasoning traces are token\-matched; prompt\-only calibration uses fewer tokens\.
#### Calibration source\.

We compare AIME 2024 reasoning traces, token\-matched WikiText\-2 text, and the same 24 AIME prompts without generated continuations onQwen3\-14BatN=7N=7\. We recompute Block BI and feature importance for each source, keeping the pruning budget andK=14K=14fixed\. Reasoning traces yield higher average accuracy than WikiText\-2 or prompts alone \(Table[6](https://arxiv.org/html/2610.19121#A3.T6)\)\.

#### Cross\-feature terms in feature selection\.

We compare feature\-importance ranking with greedy selection based on the squared MLP output change \(Equation[4](https://arxiv.org/html/2610.19121#S5.E4)\)\. Both useQwen3\-14BatN=7N=7with the same calibration traces, BI\-selected layers, and retained widths\. The squared output change includes cross\-feature terms𝔼⁡\[ϕi​ϕj\]​wi⊤​wj\\mathbb\{E\}\[\\phi\_\{i\}\\phi\_\{j\}\]w\_\{i\}^\{\\top\}w\_\{j\}\(i≠ji\\neq j\), which feature importance omits\. Both methods achieve 68\.1% average accuracy \(Table[7](https://arxiv.org/html/2610.19121#A3.T7)\), supporting feature importance as a lower\-cost selection criterion with comparable average accuracy\.

Selection criterionAIME25GPQA\-DHumanEvalAvg\.Squared MLP output change60\.0±\\pm6\.752\.5±\\pm1\.591\.9±\\pm1\.568\.1Feature importance \(Ours\)64\.4±\\pm5\.149\.7±\\pm2\.990\.2±\\pm1\.668\.1

Table 7:Comparison of feature\-selection criteria on Qwen3\-14B atN=7N=7\. HumanEval reports pass@1 \(%\); the other benchmarks report accuracy \(%\)\. Both methods use the same removal budget\.Full\-error calibration and selection take 13\.2×\\timesas long as diagonal ranking on the same hardware\.

## Appendix DRepresentation error and Attention Diagnostics Experimental Details

#### Representation error\.

Figure[2](https://arxiv.org/html/2610.19121#S1.F2)\(b\),\(d\) compares dense and pruned models on the same four held\-out AIME25 traces\. Each trace is truncated to 2,048 total tokens, with 256 continuation positions sampled for measurement\. Final\-hidden NMSE is

∑t‖htpruned−htdense‖22∑t‖htdense‖22\\frac\{\\sum\_\{t\}\\\|h\_\{t\}^\{\\mathrm\{pruned\}\}\-h\_\{t\}^\{\\mathrm\{dense\}\}\\\|\_\{2\}^\{2\}\}\{\\sum\_\{t\}\\\|h\_\{t\}^\{\\mathrm\{dense\}\}\\\|\_\{2\}^\{2\}\}wherettranges over measured positions across the traces\. In panel \(b\), we apply trimming to all 40 layers\.

#### Attention measurements\.

Panel[4](https://arxiv.org/html/2610.19121#S5.F4)\(a\)–\(b\) measures attention in the final two decoder layers \(38 and 39\) using three dense\-model AIME25 traces truncated to 32K total tokens\. Panel[4](https://arxiv.org/html/2610.19121#S5.F4)\(a\) reports KL divergence from dense to pruned attention distributions across pruning budgets, measured by the number of removed parameters\. Both panels exclude attention to the first token and rescale the remaining probabilities to sum to one\.

#### Attention overlap\.

For each query token and attention head, we select the eight earlier tokens that receive the highest attention weights from that query\. We select these tokens separately in the dense and pruned models\. Panel[4](https://arxiv.org/html/2610.19121#S5.F4)\(b\) shows the fraction of the dense model’s top\-eight tokens that remain in the pruned model’s top eight, grouped by how far each token precedes the query\. Higher overlap means that pruning better preserves which earlier tokens the model attends to\.

#### Pruning budgets\.

Panel[4](https://arxiv.org/html/2610.19121#S5.F4)\(b\) uses approximately 1\.5B removed parameters\. The component configurations and discrete\-budget differences are the same as the first group of Table[2](https://arxiv.org/html/2610.19121#S4.T2), described in Appendix[C](https://arxiv.org/html/2610.19121#A3)\.

#### Feature importance and output change\.

Panel[4](https://arxiv.org/html/2610.19121#S5.F4)\(c\) compares summed feature importance with the measured MLP output change across 70 combinations of layers and retained widths on four held\-out AIME25 traces\. Each point corresponds to a feature setRRremoved using the calibration ranking\. For activationϕj\\phi\_\{j\}and output weight vectorwjw\_\{j\}, the horizontal axis sums the importance of the removed features:

∑j∈R𝔼⁡\[ϕj2\]​‖wj‖22\\sum\_\{j\\in R\}\\mathbb\{E\}\[\\phi\_\{j\}^\{2\}\]\\\|w\_\{j\}\\\|\_\{2\}^\{2\}The vertical axis measures the squared MLP output change when these features are removed together, using the same inputs:

𝔼​‖∑j∈Rwj​ϕj‖22\\mathbb\{E\}\\left\\\|\\sum\_\{j\\in R\}w\_\{j\}\\phi\_\{j\}\\right\\\|\_\{2\}^\{2\}Both quantities are averaged over the held\-out inputs\. Feature importance ignores cross\-feature terms, while the measured output change includes them\. Their strong rank agreement shows that feature importance reliably identifies which pruning choices cause larger output changes\.

## Appendix EQualitative Analysis

The following examples supplement Figure[5](https://arxiv.org/html/2610.19121#S5.F5)with excerpts from the evaluated reasoning and coding outputs\. They illustrate repetition, incoherent token fragments, code\-format failures, and arithmetic errors; they are not prevalence estimates\. In particular, lexical collapse need not satisfy the exact\-token repetition criterion in the main text, explaining LaCo’s low detected repetition despite frequent token\-limit hits\.

Table 8:Qualitative analysis of example reasoning traces on GPQA\-Diamond usingPhi\-4\-reasoning\-plusatLp/Lt=9/40L\_\{p\}/L\_\{t\}=9/40and seed00\. \[…\] denotes omitted text\.Question:An electron is in the spin state\(3​i,4\)\(3i,4\)\. Find⟨Sy⟩\\langle S\_\{y\}\\rangle, givenσy=\(0−i​i0\)\\sigma\_\{y\}=\\left\(\\begin\{smallmatrix\}0&\-i\\\\ i&0\\end\{smallmatrix\}\\right\)\. Options: A\.−12ℏ/25\-12\\hbar/25; B\.12​ℏ/2512\\hbar/25; C\.−25ℏ/2\-25\\hbar/2; D\.25​ℏ/225\\hbar/2\.MethodOriginal trace excerpts and qualitative observationsDense2,880 tokensTerminated\[…\] Let psi = \(3i/5, 4/5\)\. Then sigma\_y = \[\[0, \-i\],\[i, 0\]\]\.We compute sigma\_y \|psi\> = \[\[0, \-i\],\[i, 0\]\] \* \(3i/5, 4/5\)^T = \(0\*\(3i/5\) \+ \-i\*\(4/5\), i\*\(3i/5\) \+ 0\*\(4/5\)\) = \(\-4i/5, i\*\(3i\)/5\)\. But note: i\*\(3i\) = 3\*\(i^2\) = \-3\. So second component becomes \-3/5\. So sigma\_y \|psi\> = \(\-4i/5, \-3/5\)\. \[…\] Then expectation value of S\_y is \(hbar/2\)\*\(\-24/25\) = \-12 hbar/25\. \[…\]Observation:Normalization and the matrix product remain consistent through the answer\.ShortGPT32,256 tokensLimit hit\[…\] The correct answer is letter letter option letter letter letter letter answer letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter letter le \[…\] negative polarity positive polarity negative polarity positive polarity negative polarity positive polarity negative polarity positive polarity negative polarity positive polarity \[…\]Observation:Answer\-format and polarity loops replace the scientific calculation\.LaCo32,256 tokensLimit hititness\_duperge hungerCreatorsgenerated Witness\_waycup pred Counter brushingonga cagescreatorBush\.IM?\\boxed\{?\}回Pot collectivelyWitnessm \[…\] vaeonusdfa832\.readlineLossUGCidalDigest HOLDERacdmumu recModifiedDate falsevale Peiadotent Circuit escapeurf Vitalgist Authority CHANNELembr\.qualCas447 multiline Highestline tet RingsanaxIF \[…\]Observation:Fragmented tokens continue to the cap without an interpretable calculation\.GRASP32,256 tokensLimit hit\[…\] I should answer answer\. Now the letter letter is answer\. It is " answer letter\. The letter letter\. So letter is " letter\. But answer\. So letter letter: letter\. answer letter\. So letter answer letter\. Actually letter letter: answer\. letter\. \[…\] letter letter letter letter\. letter letter letter\. So letter\. So letter\. So letter\. So letter\. So letter\. So l \[…\]Observation:Answer\-letter language repeats without advancing the derivation\.FLAP32,256 tokensLimit hit\[…\] \(\-3i\)\*\(\-4i\) \+ \(4\)\*\(\-3\) = 12i^2 \- 12 = 12\*\(\-1\) \-12 = \-24\. \[…\] lambda^2 \- \(i \* \(\-i\)\) = lambda^2 \- \(\-1\) = lambda^2 \+ 1 = 0, so eigenvalues are±\\pmi\. \[…\] So the answer is \-12hbar/25\. That is option A\.Therefore, answer A is the correct answer\.I’ll now provide answer A\.Thus answer A: \-12hbar/25\.Thus answer: Option A\.I’ll now answer in the requested format: \\boxed\{A\} is the answer\.But wait, but the computed result is \-12/25\. But the options are in terms of hbar\. So the answer is \-12hbar/25\.I must check the reasoning: \[…\]Observation:The correct option appears, but reconsideration continues to the cap; the saved scorer marks this response correct\.RESP32,256 tokensLimit hit\[…\] We are given an electron spin state \(3i, 4\)\. It is provided as spin state vector in spin space\. The spin state is \(3i, 4\)\. But let’s check: It is given as \(3i, 4\)\. Possibly the spin eigenstates are normalized? But not normalized\. But we can treat as spin spin vector\. \[…\] \_spin spin spin spin spin spin spin spin spin\_spin spin spin spin spin spin spin spin spin spin spin spin spin spin spin spin spin spin spin spin spin spin spin\_spin spin spin spin spin spin spin spin spin spinObservation:Relevant domain vocabulary gives way to a repeated token stream\.CoCurve32,256 tokensLimit hit\[…\] I suspect that the question is: "An electron is in the spin state \(3/i, 4\)"? Actually "3i, 4" means maybe a spin state written as a combination of components? Possibly the spin state is represented as a state vector with coefficients? \[…\]⟨\\langlepsi\| sigma\_y \|psi⟩\\rangle=⟨\\langlepsi\| sigma\_y \|psi⟩\\rangle=⟨\\langlepsi\| sigma\_y \|psi⟩\\rangle=⟨\\langlepsi\| sigma\_y \|psi⟩\\rangle\[…\]Observation:The expectation\-value notation is repeated instead of evaluated\.LayerTrim \(Ours\)1,253 tokensTerminated\[…\] But v^†\\daggerv = \|3i\|^2 \+ \|4\|^2 = \(9\) \+ \(16\) = 25\. \[…\] sigma\_y v = \[0 \* \(3i\) \+ \(\-i\)\* \(4\), \(i\)\*\(3i\) \+ 0\*\(4\)\] = \[\-i\*4, i\*\(3i\)\]\. But i\*\(3i\) = 3i^2 = 3\*\(\-1\) = \-3\. So sigma\_y v = \(\-4i, \-3\)\. \[…\] Then v^†\\daggersigma\_y v = \(\-3i\)\*\(\-4i\) \+ \(4\)\*\(\-3\)\. Let’s compute: \(\-3i\)\*\(\-4i\) = 12i^2 = 12\*\(\-1\) = \-12\. And \(4\)\*\(\-3\) = \-12\. So total v^†\\daggersigma\_y v = \-12 \-12 = \-24\. Then expectation value sigma\_y = \-24/25\.Then expectation value of spin S\_y = \(1/2\)\*\(\-24/25\)\*hbar = \-12\*hbar/25\. So the answer is option A: \-12\*hbar/25\. \[…\]Observation:The input, normalization, and intermediate products remain linked to a completed answer\.Table 9:Qualitative analysis of example reasoning traces on HumanEval usingQwen3\-14BatLp/Lt=9/40L\_\{p\}/L\_\{t\}=9/40and seed00\. \[…\] denotes omitted text\.Question:Completehas\_close\_elements\(numbers, threshold\): return whether any two input elements differ by less than the threshold\. The prompt example\(\[1\.0, 2\.0, 3\.0\], 0\.5\)requiresFalse\.MethodOriginal trace excerpts and qualitative observationsDense1,348 tokensTerminatedReasoning: \[…\] So for each i in range\(len\(numbers\)\), and for each j in range\(i\+1, len\(numbers\)\), compare numbers\[i\] and numbers\[j\]\.Final response: \[…\]def has\_close\_elements\(numbers: List\[float\], threshold: float\) \-\> bool:for i in range\(len\(numbers\)\):for j in range\(i \+ 1, len\(numbers\)\):if abs\(numbers\[i\] \- numbers\[j\]\) < threshold:return Truereturn False\[…\]Observation:Distinct\-index comparisons are preserved in the final function\.ShortGPT337 tokensTerminatedReasoning: \[…\] The function should check if any two numbers in the list are within the given threshold\. The function should be written in a Python block with the proper indentation\.Final response: \[…\] The function uses the ‘\*‘ operator to check if any two numbers in the list are within the given threshold\. The function returns ‘True‘ if any two numbers are within the given threshold, and ‘False‘ otherwise\.The function is as described, and the code is correct\.Observation:The response claims to provide an implementation, but supplies no comparison logic\.LaCo32,256 tokensLimit hit\[…\]imer Spo aggregateTimelineaslasca Jenner官 TC户攀筹progressbar共创的高度GD?\\boxed\{?\}栊cket clam命运随手oidantom迎接系列产品 movement踩春节路上功课NPCasl人性化刹档埠勿官\- \[…\]Observation:These mixed\-script fragments do not form an interpretable plan or function\.GRASP32,256 tokensLimit hitfrom typing import List, float, 2\.0, 3\.0, 4\.0, 0\.0, 2\.0, 0\.0, 0\.0, 0\.0, 0\.0, 0\.0, 0\.0, 0\.0, 0\.0, 0\.0, 0\.0, 0\.0, 0\.0, 0\.0, 0\.0, 0 \[…\] 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0: 0\.0:Observation:A malformed import leads into numeric repetition, rather than a function body\.FLAP4,122 tokensTerminatedReasoning: \[…\] So, I can sort the list first, then check each consecutive pair\. If any pair has a difference less than the threshold, then the answer is True\. Otherwise, it’s False\.Final response: \[…\]def has\_close\_elements\(numbers: List\[float\], threshold: float\) \-\> bool:sorted\_numbers = sorted\(numbers\)for i in range\(len\(sorted\_numbers\) \- 1\):if sorted\_numbers\[i\+1\] \- sorted\_numbers\[i\] < threshold:return Truereturn False\[…\]Observation:The intended algorithm survives, but the final if and return statements are incorrectly indented\.RESP3,176 tokensTerminatedReasoning: \[…\] Therefore, if the list is sorted, then the minimum difference between any two elements is the minimum of the adjacent differences\. \[…\]if len\(numbers\) < 2:return Falsenumbers\_sorted = sorted\(numbers\)Final response: \[…\]if len\(numbers\) < 2:return Falsenumbers\_sorted = sorted\(numbers\)\[…\]Observation:Correct relative indentation is lost when the full function is converted to a body\-only completion\.CoCurve115 tokensTerminatedReasoning \(empty\): <think\> </think\>Final response: \[…\]numbers = set\(numbers\) \# Remove duplicates to consider each number only oncefor num in numbers:if any\(abs\(num \- x\) < threshold for x in numbers\): \# Check if any number is within the threshold of the given numberreturn Truereturn False\[…\]Observation:The code compiles, but compares each value with itself; it incorrectly accepts the prompt’s first example\.LayerTrim \(Ours\)2,699 tokensTerminatedReasoning: \[…\] But the problem says "closer to each other than given threshold"\. So, if the difference is exactly equal to the threshold, then it’s not considered\. For example, if threshold is 0\.5, and two elements are 0\.5 apart, then it’s not considered\. So the condition is strictly less than\.Final response: \[…\]if len\(numbers\) < 2:return Falsesorted\_numbers = sorted\(numbers\)for i in range\(len\(sorted\_numbers\) \- 1\):if sorted\_numbers\[i\+1\] \- sorted\_numbers\[i\] < threshold:return Truereturn False\[…\]Observation:The final body preserves strict comparison and consistent indentation in the original prompt context\.Table 10:Qualitative analysis of example reasoning traces on AIME 2026 usingPhi\-4\-reasoning\-plusatLp/Lt=7/40L\_\{p\}/L\_\{t\}=7/40and seed00\. \[…\] denotes omitted text\.Question:LetNNbe the number of positive integer divisors of170171717017^\{17\}that leave a remainder of55when divided by1212\. Find the remainder whenNNis divided by10001000\.MethodOriginal trace excerpts and qualitative observationsDense3,907 tokensTerminated\[…\] 7 \* 11 = 77, 77 \* 13 = 1001, 1001 \* 17 = 17017\. \[…\] So indeed, the valid cases are: \(a even, d odd, b even\) and \(a odd, d even, b odd\)\. And each gives 9 possibilities for a, 9 for b, 9 for d\. So total 9^3 \+ 9^3 = 2\*729 = 1458\. And then times 18 choices for c yields 1458\*18 = 26244\.Then remainder mod 1000 is 26244 mod 1000 = 244\. \[…\]Observation:The two parity cases and independent exponent choices produce the correct count\.ShortGPT32,256 tokensLimit hit\[…\] But wait, is that all? Let’s verify product: 7 \* 11 = 77, 77 \*13= 100, 77\*13= 77\*10 \+ 77\*3? Let me compute: 77 \* 10=770, 77\*3=231, so product= 101, so no\. Let’s compute properly: 7\*11=77; 77\*13= 77\*10 \+77\*3 = 770 \+231 = 1, 770\+231= 1,101 indeed\. \[…\] And then 1,101 \*17 equals? 1,101\*17 = 1,101\*10 = 11,101? Let me compute: 1,101\*17 = 1,101\*7= 7,207, oh sorry, let me compute properly: 1,101\*17 = 1,101\*10= 11,101, but then 1,101\*17 = 1,101\*\(10\+7\) = 11,101 \+ 7,207 = 18,308, oh sorry, not exactly\. \[…\] divided by prime 17017 divided by prime 17017 divided by prime 17017 divided by prime 17017 divided by prime 17017 divided byObservation:An attempted verification introduces new arithmetic errors and then stalls\.LaCo32,256 tokensLimit hitreads acquaintance567eyJremball congressentyAAAIDEOanne chargerbatismars\-PackINS sachDDminer drift compensation Painterpressu \[…\] s Tape validenger Observable playEDGE downhill dashojufenpagedpaged acceptedplaetty tun Missspotify yields Delta Lower sweep monitor CommissionPARprotarkin monitored \[…\]Observation:No interpretable factorization or count is visible in these fragments\.GRASP32,256 tokensLimit hit\[…\] 17017 is exactly a product of 17, 19, and 13, but let’s check: 17 times 19 and 13: 17: 19 and then 13? So the factors of 17017 is 13, 17, 19\. But check: we see that 17 factor? Actually: 17017 is: product of 17, 19, 13? Let’s check: 17017 is a product of 13, 17, 19\. So the digits? Let’s check: 13\*17\*19? Let’s check: 17\*19? Actually, 17 times 19 is 60 maybe\. \[…\] yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes,Observation:The proposed factors multiply to 4,199; verification collapses into repetition\.FLAP11,066 tokensTerminated\[…\] I want to check one more possibility: Possibly the factorization of 17017 is different\. Let’s re\-check: 17017 = 17 \* 7 \* 11 \* 13, I should check: 17\*7 = 119, 119\*11 = 1309, 1309\*13 = 17017\. So that’s correct\. \[…\] For any exponent from 0 to 17, there are 9 odd choices and 10 even choices\.For the pattern \(odd, even, even\): \- a odd: 9 choices, b even: 10 choices, c even: 10 choices\. Total = 9×\\times10×\\times10 = 900\.For the pattern \(even, odd, odd\): \- a even: 10 choices, b odd: 9 choices, c odd: 9 choices\. Total = 10×\\times9×\\times9 = 810\. \[…\] Thus the total number N is 1710×\\times18 = 30780\. \[…\]Observation:Factorization is recovered, but a wrong parity count survives into the answer\.RESP4,054 tokensTerminated\[…\] For exponent range 0 to 17, there are 18 possibilities for each exponent\. Among these, the number of even numbers from 0 to 17: Even numbers: 0,2,4,…,16\. That’s 9 numbers\. The number of odd numbers: 1,3,…,17\. That’s also 9\. So for each exponent, there are 9 even and 9 odd possibilities\. \[…\] Now, remember that the exponent c for prime 13 can be any number from 0 to 17, which is 18 possibilities\. So overall, the number of divisors of 17017^\{17\} that are≡\\equiv5 mod 12 is: 1458 \* 18 = 1458 \* 18\. Let’s compute that: 1458 \* 18 = 1458 \* \(10 \+ 8\) = 1458\*10 \+ 1458\*8 = 14580 \+ 11664 = 26244\. Wait check: 1458 \* 18 = 1458 \* 2 \* 9 = 2916 \* 9 = 26244\. Yes\. \[…\]Observation:RESP also succeeds here, preserving both parity counts and the final multiplication\.CoCurve10,382 tokensTerminated\[…\] 17^1≡\\equiv17≡\\equiv17−\-12 = 4 \(mod 12\) For any exponent≥\\geq1, note that 17^n always gives 4 \(mod 12\) since 4^k is congruent to 4 for k≥\\geq1\. Thus: if d = 0→\\rightarrow17^0≡\\equiv1 \(mod 12\) if d≥\\geq1→\\rightarrow17^d≡\\equiv4 \(mod 12\) \[…\] Actually, 1458−\-1000 = 1458? No, more clearly: 1458−\-1000 = 1458? I’ll do simple subtraction: 1458−\-1000 = 558\. \[…\]Observation:A wrong subtraction changes the residue cases; a second survives into the reported remainder\.LayerTrim \(Ours\)5,234 tokensTerminated\[…\] So the number of ways for a, b, d is: \(9 ways for a if even, 9 ways for b if even, 9 ways for d if odd\) = 9\*9\*9 = 729, and \(9 ways for a if odd, 9 ways for b if odd, 9 ways for d if even\) = 729\. So total is 729\+729 = 1458\. And then c \(the exponent for 13\) is free, 18 possibilities\. So overall count is 18 \* 1458\. Wait, did I already incorporate c? Let’s check: \[…\] But earlier, I computed 729\+729 = 1458, but that 1458 is only for a, b, d\. Then multiply by 18 for c\. So total count N = 18 \* 1458 = 26244\. Let’s check: 1458 \* 18 = 1458\*10 \+ 1458\*8 = 14580 \+ 11664 = 26244\. \[…\] So N = 26244\. But then the problem asks: "Find the remainder when N is divided by 1000\." So 26244 mod 1000 = 244\. But wait, check: 26244 mod 1000 = 26244 \- 26\*1000 = 26244 \- 26000 = 244\. \[…\]Observation:The main count preserves both parity patterns and explicitly checks the independent factor of 18\.
## Appendix FLicenses and Terms of Use

All models and datasets used in this study are publicly available for research use\. We follow their respective licenses and usage conditions and use these resources solely for non\-commercial research\.

#### Models\.

Qwen3\-14B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.19121#bib.bib12)and Magistral\-Small\-2506[Mistral\-AI et al\. \(2025\)](https://arxiv.org/html/2610.19121#bib.bib21)are released under the Apache 2\.0 License\. Phi\-4\-reasoning\-plus[Abdin et al\. \(2025\)](https://arxiv.org/html/2610.19121#bib.bib20)is released under the MIT License\.

#### Evaluation datasets\.

AIME2025[Zhang and Math\-AI \(2025\)](https://arxiv.org/html/2610.19121#bib.bib22)and AIME2026[Zhang and Math\-AI \(2026\)](https://arxiv.org/html/2610.19121#bib.bib23)are distributed through the Math\-AI benchmark releases under the Apache 2\.0 License\. The original AIME problems are subject to the copyright of the Mathematical Association of America\. GPQA\-Diamond[Rein et al\. \(2024\)](https://arxiv.org/html/2610.19121#bib.bib24)is released under the CC BY 4\.0 License, with additional dataset access and redistribution conditions\. HumanEval[Chen et al\. \(2021\)](https://arxiv.org/html/2610.19121#bib.bib26)is distributed under the MIT License, while MBPP\+[Liu et al\. \(2023\)](https://arxiv.org/html/2610.19121#bib.bib25)is distributed through EvalPlus under the Apache 2\.0 License\.

#### Calibration data\.

We use AIME2024 problems for calibration, with reasoning traces generated by the original dense models\. The original examination problems are subject to the copyright and usage conditions of the Mathematical Association of America\. The generated reasoning traces are used for calibration and diagnostic analysis\. The calibration\-source ablation also uses WikiText\-2, whose dataset card lists CC BY\-SA 3\.0 and GFDL licenses\.222[https://huggingface\.co/datasets/Salesforce/wikitext](https://huggingface.co/datasets/Salesforce/wikitext)

#### Software\.

Hugging Face Transformers is released under the Apache 2\.0 License, while PyTorch is distributed under a BSD\-style license\. All software is used in accordance with its respective licensing terms\.