Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models

# Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models
Source: [https://arxiv.org/html/2607.18280](https://arxiv.org/html/2607.18280)
Chao Han,Haozhe Hu,Xiaoyu Shen

Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo Correspondence: Xiaoyu Shen[xyshen@eitech\.edu\.cn](https://arxiv.org/html/2607.18280v1/mailto:email@domain)

###### Abstract

Large language models \(LLMs\) are often compressed through static parameter pruning or dynamic token\-level computation, yet aggressive sparsification can trigger rapid performance degradation beyond an essential sparsity boundary\. This work asks*whether combining these two mechanisms can delay such degradation by distributing the compression burden*\. We study a minimalist compound sparsity framework that first applies low\-rank approximation and channel pruning to obtain a statically compressed backbone, and then introduces lightweight routers for per\-token dynamic layer skipping\. This design enables independent control of parameter sparsity and token\-level computation sparsity\. Experiments across language understanding and modeling benchmarks show that compound sparsity consistently outperforms single\-mechanism compression under the same total sparsity, delaying the decay point on understanding tasks and preserving stronger modeling performance\. Further analysis reveals cross\-dimensional interference between parameter pruning and token skipping, and shows that near\-balanced allocation is most effective under a fixed sparsity budget\. These results demonstrate that compound compression provides a practical way to improve LLM compression, while revealing a broader cross\-dimensional sparsity boundary that ultimately limits further compression\. Code will be available at https://github\.com/EIT\-NLP/LLM\-Pruning\.

Beyond Single\-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models

Chao Han, Haozhe Hu, Xiaoyu Shen††thanks:Corresponding AuthorNingbo Institute of Digital Twin, Eastern Institute of Technology, NingboCorrespondence: Xiaoyu Shen[xyshen@eitech\.edu\.cn](https://arxiv.org/html/2607.18280v1/mailto:email@domain)

## 1Introduction

Scaling large language models \(LLMs\) has led to remarkable progress in language understanding and generationBrownet al\.\([2020](https://arxiv.org/html/2607.18280#bib.bib3)\); Suet al\.\([2022](https://arxiv.org/html/2607.18280#bib.bib6),[2024](https://arxiv.org/html/2607.18280#bib.bib5)\); Ahnet al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib45)\); Linet al\.\([2025](https://arxiv.org/html/2607.18280#bib.bib1)\); Zhanget al\.\([2025](https://arxiv.org/html/2607.18280#bib.bib44)\)\. However, their high inference costs make deployment difficult in resource\-constrained environments\. Model compression has therefore become a central direction for reducing memory usage and computation while preserving model capabilityHanet al\.\([2016](https://arxiv.org/html/2607.18280#bib.bib13)\); Fanet al\.\([2025](https://arxiv.org/html/2607.18280#bib.bib4)\)\.

A key challenge in LLM compression is the existence of*essential sparsity*\. Compression can usually remove part of the model or computation with limited degradation, but after a critical point, further sparsification causes rapid performance decayJAISWALet al\.\([2023](https://arxiv.org/html/2607.18280#bib.bib60)\); Jaiswalet al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib61)\); Dinget al\.\([2026](https://arxiv.org/html/2607.18280#bib.bib11)\); Wuet al\.\([2026](https://arxiv.org/html/2607.18280#bib.bib7)\)\.

This sparsity boundary is usually studied through individual compression methods\. In LLM compression, two representative directions are static parameter pruning and dynamic token\-level computation\. Static parameter pruning reduces the structural cost of the model by removing or factorizing parameters before inferenceFrantar and Alistarh \([2023](https://arxiv.org/html/2607.18280#bib.bib19)\); Ashkbooset al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib21)\); Huanget al\.\([2025](https://arxiv.org/html/2607.18280#bib.bib20)\)\. Dynamic token\-level computation methods instead reduce inference cost during inference by allocating computation adaptively to different tokens or inputsRaposoet al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib29)\); Jianget al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib30)\); Hanet al\.\([2025](https://arxiv.org/html/2607.18280#bib.bib10)\); Heet al\.\([2026](https://arxiv.org/html/2607.18280#bib.bib8)\)\. Although these methods exploit different sources of redundancy, they are typically pushed and evaluated separately, so the resulting sparsity limit is often treated as a property of a single compression mechanism\.

This single\-mechanism view leaves an important question open:*if one compression mechanism eventually reaches its sparsity boundary, can combining it with another mechanism delay the onset of degradation?*The intuition is simple\. Redundancy in LLMs appears across multiple levels, including parameters, layers, and token\-level computationFuet al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib62)\); Dumitruet al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib63)\); Liang \([2025](https://arxiv.org/html/2607.18280#bib.bib64)\); Hanet al\.\([2026](https://arxiv.org/html/2607.18280#bib.bib9)\), so a target compression budget does not have to be imposed entirely on one source of redundancy\. Instead, part of the budget can be assigned to static parameter pruning to reduce the cost of the backbone, while another part can be assigned to dynamic token skipping to reduce input\-dependent computation\. By distributing compression pressure, compound compression may achieve the same total sparsity while keeping each individual mechanism in a less aggressive regime\.

In this work, we use the combination of static parameter pruning and dynamic token skipping as a representative and controllable case study for this question\. Concretely, we first apply LoRAPLiet al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib16)\)and channel pruning to obtain a statically compressed backbone, and then add lightweight routers for per\-token dynamic layer skippingZhaoet al\.\([2025](https://arxiv.org/html/2607.18280#bib.bib26)\)\. This design allows us to independently control parameter sparsity and token\-level computation sparsity, making it possible to analyze how different compression allocations affect the sparsity\-performance trade\-off\.

![Refer to caption](https://arxiv.org/html/2607.18280v1/x1.png)Figure 1:Overview of LLMs compression paradigms, compound sparse distributes sparsity burdens into orthogonal parameter and token space\.Our experiments show that compound sparsity consistently achieves better compression\-performance trade\-offs than single\-mechanism compression under the same total sparsity\. On understanding tasks, it delays the sharp decay point from around 20% to around 30% sparsity\. On language modeling, it also retains higher performance, e\.g\., about 70% relative performance at 30% sparsity, compared with about 60% for token skipping and 56% for parameter pruning\. Further analysis reveals cross\-dimensional interference: a mildly pruned backbone becomes more sensitive to token sparsity\. Under a fixed 50% total sparsity budget, near\-balanced allocation achieves the best performance\. These findings support the*Two Dimensions, One Wall*principle: compound compression delays performance decay, but a broader cross\-dimensional sparsity boundary remains\.

Our contributions are threefold\. \(1\) We study essential sparsity from a compound compression perspective, asking whether combining static parameter pruning and dynamic token skipping can delay performance decay\. \(2\) We introduce a minimalist and controllable framework based on LoRAP, channel pruning, and per\-token dynamic layer skipping, enabling independent control of parameter and token sparsity\. \(3\) We show that compound sparsity improves the compression\-performance trade\-off and delays decay compared with single\-mechanism compression, while still converging to a cross\-dimensional sparsity boundary\.

## 2Methodology

Our goal is to build a standardized, controllable paradigm for probing compound sparsity, to explore the interplay and fundamental limits between parameter sparsity and token\-level dynamic computation sparsity\. We adopt two established methods: LoRAPLiet al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib16)\)for static parameter pruning and SkipGPTZhaoet al\.\([2025](https://arxiv.org/html/2607.18280#bib.bib26)\)for token\-wise dynamic layer skipping\. Using off\-the\-shelf techniques rules out biases from bespoke designs, ensuring our observations purely reflect the effects of cross\-dimensional sparsity allocation\.

### 2\.1Compound Compression Pipeline

We first formulate the forward computation paradigm within a standard transformer layer of pre\-trained LLMs\. Denote the original forward mapping of an uncompressed transformer layer \(as Figure[1](https://arxiv.org/html/2607.18280#S1.F1)\(a\) shown\) as:

𝒚=ℱ​\(𝒙\)\\boldsymbol\{y\}=\\mathcal\{F\}\(\\boldsymbol\{x\}\)\(1\)where𝒙∈ℝN×d\\boldsymbol\{x\}\\in\\mathbb\{R\}^\{N\\times d\}denotes the input token embedding sequence with sequence lengthNNand hidden dimensiondd, andℱ​\(⋅\)\\mathcal\{F\}\(\\cdot\)represents the full computational operator of the intact transformer layer\.

#### Step 1: Static Parameter Space Compression\.

We first conduct structured parameter pruning upon the original layer operatorℱ​\(⋅\)\\mathcal\{F\}\(\\cdot\)\. Letsp∈\[0,1\]s\_\{p\}\\in\[0,1\]denote theparameter sparsity ratio, which quantifies the proportion of pruned redundant parameters within the layer\. After parameter pruning, the original full\-layer operator is converted into a sparse parameter backbone operatorℱp​\(⋅\)\\mathcal\{F\}\_\{p\}\(\\cdot\)\. Correspondingly, the overall computational overhead of this layer is reduced to\(1−sp\)\(1\-s\_\{p\}\)of the original magnitude\. The forward propagation after parameter compression \(see Figure[1](https://arxiv.org/html/2607.18280#S1.F1)\(b\)\) is formulated as:

𝒚p=ℱp​\(𝒙\)\\boldsymbol\{y\}\_\{p\}=\\mathcal\{F\}\_\{p\}\(\\boldsymbol\{x\}\)\(2\)
In our implementation, we adopt LoRAPLiet al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib16)\)for parameter pruning, which applies low\-rank decomposition to attention modules and channel pruning to MLP modules\.

#### Step 2: Dynamic Token\-level Computational Compression\.

Built upon the pruned sparse backbone, we further integrate lightweight inference\-adaptive routing mechanism to perform token\-wise computational skipping\. We definest∈\[0,1\]s\_\{t\}\\in\[0,1\]as thetoken computational sparsity ratio, indicating the expected proportion of tokens that are exempted from full\-layer forward computation\.

Following SkipGPTZhaoet al\.\([2025](https://arxiv.org/html/2607.18280#bib.bib26)\), a lightweight post\-trained routing module is deployed within each transformer layer to produce binary routing indicators for every individual token\. The entire input token sequence𝒙\\boldsymbol\{x\}is divided into two disjoint subsets \(corresponding to blue and gray cubes in Figure[1](https://arxiv.org/html/2607.18280#S1.F1)\(c\)\) according to routing decisions: Active token subset𝒙𝒜\\boldsymbol\{x\}\_\{\\mathcal\{A\}\}which execute forward computation and Inactive token subset𝒙ℐ\\boldsymbol\{x\}\_\{\\mathcal\{I\}\}which direct delivered to subsequent layer\.

The final hybrid forward computation of the compound sparse layer can be formally written in piece\-wise form:

𝒚t=\{ℱp​\(𝒙\),𝒙∈𝒙𝒜𝒙,𝒙∈𝒙ℐ\\boldsymbol\{y\}\_\{\\text\{t\}\}=\\begin\{cases\}\\mathcal\{F\}\_\{p\}\(\\boldsymbol\{x\}\),&\\boldsymbol\{x\}\\in\\boldsymbol\{x\}\_\{\\mathcal\{A\}\}\\\\ \\boldsymbol\{x\},&\\boldsymbol\{x\}\\in\\boldsymbol\{x\}\_\{\\mathcal\{I\}\}\\end\{cases\}\(3\)This hierarchical two\-stage compression paradigm strictly separates parameter structural sparsity and token adaptive computational sparsity, and realizes decoupled control over sparsity intensity in two orthogonal optimization dimensions\.

### 2\.2Definition of Compound Sparsity

In this work, we uniformly define sparsity from the perspective of*effective inference computation reduction ratio*, which maintains consistent measurement criteria across different compression dimensionsRaposoet al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib29)\); Jianget al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib30)\)and facilitates fair comparison\.

After cascaded parameter pruning and dynamic token skipping, the retained computational volume of the target layer is quantified as the product of reserved computation ratio in two dimensions:

Retained Computation Ratio=\(1−sp\)⋅\(1−st\)\\text\{Retained Computation Ratio\}=\(1\-s\_\{p\}\)\\cdot\(1\-s\_\{t\}\)Accordingly, we define the*global compound sparsity degree*ScompS\_\{\\text\{comp\}\}as the total proportion of eliminated computation resources:

Scomp=1−\(1−sp\)​\(1−st\)S\_\{\\text\{comp\}\}=1\-\(1\-s\_\{p\}\)\(1\-s\_\{t\}\)\(4\)
Under fixed target sparsityStargetS\_\{\\text\{target\}\}, we can adjust the combination of\(sp,st\)\(s\_\{p\},s\_\{t\}\)to realize diversified sparsity allocation strategies: allocating more compression budget to parameter dimension, placing more sparsity burden on token dynamic dimension, or adopting balanced dual\-dimensional allocation\. Such flexible allocation strategy constitutes the core experimental basis for us to explore whether shifting compression pressure can evade single\-dimensional sparsity collapse limitation\.

### 2\.3Balanced Sparsity Allocation

We adopt uniform sparsity assignment for balanced compression budget distribution\. Given parameter sparsitysps\_\{p\}and token sparsitysts\_\{t\}, the overall compound sparsity is defined asScomp=1−\(1−sp\)​\(1−st\)S\_\{\\text\{comp\}\}=1\-\(1\-s\_\{p\}\)\(1\-s\_\{t\}\)\. For a targeted global sparsity level, we evenly split compression intensity across two dimensions by settingsp=st=Xs\_\{p\}=s\_\{t\}=X\. Solving\(1−X\)2=1−Scomp\(1\-X\)^\{2\}=1\-S\_\{\\text\{comp\}\}yields the unified sparsity ratio\. Mathematically, equal allocation minimizes the maximum sparsity imposed on either dimension, which serves as our fundamental allocation principle in experiments\. Section[3\.3](https://arxiv.org/html/2607.18280#S3.SS3.SSS0.Px2)shows such paradigm also works well in practical\.

## 3Experiments

### 3\.1Setup

We use Llama3\.1\-8b as the base model and Redpajama as training data\. We evaluate model performance on both understanding and modeling tasks\.Understanding:Accuracy on BoolQClarket al\.\([2019](https://arxiv.org/html/2607.18280#bib.bib32)\), PIQABisket al\.\([2020](https://arxiv.org/html/2607.18280#bib.bib33)\), HellaSwagZellerset al\.\([2019](https://arxiv.org/html/2607.18280#bib.bib34)\), WinograndeSakaguchiet al\.\([2021](https://arxiv.org/html/2607.18280#bib.bib35)\), ARC\-E/ARC\-CClarket al\.\([2018](https://arxiv.org/html/2607.18280#bib.bib36)\), and OBQAMihaylovet al\.\([2018](https://arxiv.org/html/2607.18280#bib.bib37)\)\.Modeling: Perplexity \(PPL\) on WikiText\-2Merityet al\.\([2017](https://arxiv.org/html/2607.18280#bib.bib39)\)\. All tasks are evaluated by lm\-eval 0\.4\.9Gaoet al\.\([2024](https://arxiv.org/html/2607.18280#bib.bib40)\)\. We report relative performance \(e\.g\. compressed accuracy/original accuracy\) to intuitively quantify the preservation of model capabilities\.

### 3\.2Main Results

![Refer to caption](https://arxiv.org/html/2607.18280v1/x2.png)\(a\)Performance degradation curve of understanding tasks\.
![Refer to caption](https://arxiv.org/html/2607.18280v1/x3.png)\(b\)Performance degradation curve of modeling tasks\.

Figure 2:Sparsity\-performance landscape of different sparse paradigms\.#### Modeling is more sensitive\.

First, language modeling suffers severer performance degradation than language understanding\. The difference lies in task nature: multiple\-choice understanding tasks are robust to compression noise due to simple linguistic patterns, while modeling tasks demand precise context prediction and are highly sensitive to sparsity\-induced knowledge loss\.

#### Decay is delayed\.

Second, the essential sparsity phenomenon is consistently observed in compound compression, with distinct degradation patterns on understanding tasks\. Taking 90% relative performance as the critical boundary, understanding tasks only lose 10% performance within 0\-30% sparsity but suffer a sharp 25% performance drop at 30–60% sparsity\. Notably, the compound sparsity paradigm delays the essential sparsity collapse threshold from 20% \(single\-dimensional compression\) to a higher sparsity range\. Differently, language modeling only exhibits early\-stage essential sparsity within 0–10% sparsity, followed by approximate linear degradation, forming an inverse scaling curve\.

#### Compound sparsity performs best\.

Third, dual\-dimensional sparsity allocation yields persistent performance gains under identical total compression sparsity\. Compared with single\-dimensional compression, compound sparsity strategy distributes compression budgets across parameter and token dimensions, fully excavating and utilizing the heterogeneous redundant information of LLMs parameter and token space\. This orthogonal redundancy utilization avoids over\-compression on a single dimension, mitigates premature performance collapse, and thus achieves better trade\-offs between compression ratio and model performance\.

### 3\.3Further Studies

#### Cross\-dimensional interference\.

Table 1:Performance of Different Methods under Varying Sparsity RatiosWe further explore the interplay between parameter and token sparsity by comparing performance degradation curves of dynamic token skipping on the dense baseline and the model with 10% parameter sparsity\. While parameter pruning and token skipping independently reduce computational overhead, we observe evident cross\-dimensional interference in practical performance\. Even the 10% parameter\-sparse model, which retains 95% performance of the original model, becomes more vulnerable to token compression\. When token sparsity reaches 60%, the compound sparse model only maintains 83% baseline performance\. Such cross\-dimensional fragility reveals why a unified inherent limit exists for compound sparsity\.

#### Allocation matters\.

Table 2:Sensitivity Analysis of Performance w\.r\.t\. Parameter SparsityWe next study how to allocate a fixed compression budget across the two dimensions\. We analyze performance across cross\-dimensional sparsity settings under 50% total compound sparsity\. As parameter sparsity increases from 0% to 50%, performance follows a convex curve: it improves with mild parameter sparsity but degrades drastically at high levels\. Optimal results appear at a near\-uniform allocation \(sp≈30%,st≈28\.6%s\_\{p\}\\approx 30\\%,s\_\{t\}\\approx 28\.6\\%\)\. This verifies our proposition that balanced cross\-dimensional sparsity lowers single\-dimensional compression intensity and achieves the best trade\-off\.

## 4Conclusion

This work explores the fundamental limit of compound sparsity in LLMs compression using a concise, controllable framework that decouples static parameter pruning and dynamic token skipping\. Our empirical evidence proves that sharing compression pressure across parameter and token dimensions cannot bypass the inherent sparsity limit, but merely defers performance degradation\. A universal cross\-dimensional sparsity wall remains the ultimate constraint\. We also quantify sparsity\-performance trends and identify distinct degradation behaviors across language tasks\. Beyond revealing core compression principles, our findings provide principled references for cross\-dimensional budget scheduling\.

## Limitations

Our empirical analysis is conducted based on a limited set of LLM architectures and two representative compression paradigms, i\.e\., static parameter pruning and dynamic token skipping\. Although we adopt classic and widely validated compression schemes to ensure credible observations, the current conclusions are not fully universal across arbitrary model architectures or diverse compression pipelines\. The generalization of our discovered compound sparsity principles to other model families, scaling sizes, and advanced compression methods remains to be further verified\. Future work can extend our probing framework to broader model settings and more compression variants to establish a more generalizable understanding of cross\-dimensional sparsity limits\.

## References

- J\. Ahn, R\. Verma, R\. Lou, D\. Liu, R\. Zhang, and W\. Yin \(2024\)Large language models for mathematical reasoning: progresses and challenges\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop,pp\. 225–237\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p1.1)\.
- S\. Ashkboos, M\. L\. Croci, M\. G\. do Nascimento, T\. Hoefler, and J\. Hensman \(2024\)SliceGPT: compress large language models by deleting rows and columns\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p3.1)\.
- Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§3\.1](https://arxiv.org/html/2607.18280#S3.SS1.p1.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p1.1)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)Boolq: exploring the surprising difficulty of natural yes/no questions\.arXiv preprint arXiv:1905\.10044\.Cited by:[§3\.1](https://arxiv.org/html/2607.18280#S3.SS1.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§3\.1](https://arxiv.org/html/2607.18280#S3.SS1.p1.1)\.
- L\. Ding, A\. Zhao, F\. Ye, Z\. Chen, and X\. Shen \(2026\)From llms to lrms: rethinking pruning for reasoning\-centric models\.arXiv preprint arXiv:2601\.18091\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p2.1)\.
- R\. Dumitru, P\. I\. Clotan, V\. Yadav, D\. Peteleaza, and M\. Surdeanu \(2024\)Change is the only constant: dynamic llm slicing based on layer redundancy\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 9912–9920\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p4.1)\.
- Y\. Fan, A\. Zhao, J\. Fu, J\. Tong, H\. Su, Y\. Pan, W\. Zhang, and X\. Shen \(2025\)VisiPruner: decoding discontinuous cross\-modal dynamics for efficient multimodal llms\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 18896–18913\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p1.1)\.
- E\. Frantar and D\. Alistarh \(2023\)SparseGPT: massive language models can be accurately pruned in one\-shot\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p3.1)\.
- Q\. Fu, M\. Cho, T\. Merth, S\. Mehta, M\. Rastegari, and M\. Najibi \(2024\)LazyLLM: dynamic token pruning for efficient long context llm inference\.InWorkshop on Efficient Systems for Foundation Models II@ ICML2024,Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p4.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602)Cited by:[§3\.1](https://arxiv.org/html/2607.18280#S3.SS1.p1.1)\.
- C\. Han, H\. Hu, F\. Ma, W\. Zhang, and X\. Shen \(2026\)UniRank: unified rank allocation for low\-rank llm compression\.External Links:2606\.21847,[Link](https://arxiv.org/abs/2606.21847)Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p4.1)\.
- C\. Han, Y\. Liang, Z\. Xuan, D\. Wu, W\. Zhang, and X\. Shen \(2025\)Informed routing in llms: smarter token\-level computation for faster inference\.arXiv preprint arXiv:2510\.13831\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p3.1)\.
- S\. Han, H\. Mao, and W\. J\. Dally \(2016\)Deep compression: compressing deep neural network with pruning, trained quantization and huffman coding\.InICLR,Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p1.1)\.
- Z\. He, A\. Zhao, X\. Shen, C\. Wu, and L\. He \(2026\)SkipOPU: an fpga\-based overlay processor for large language models with dynamically allocated computation\.arXiv preprint arXiv:2603\.14785\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p3.1)\.
- W\. Huang, Y\. Zhang, X\. Zheng, F\. Chao, and R\. Ji \(2025\)Determining layer\-wise sparsity for large language models through a theoretical perspective\.InForty\-second International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p3.1)\.
- A\. Jaiswal, Z\. Gan, X\. Du, B\. Zhang, Z\. Wang, and Y\. Yang \(2024\)Compressing llms: the truth is rarely pure and never simple\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 36973–36993\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p2.1)\.
- A\. JAISWAL, S\. Liu, T\. Chen, and Z\. "\. Wang \(2023\)The emergence of essential sparsity in large pre\-trained models: the weights that matter\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 38887–38901\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p2.1)\.
- Y\. Jiang, H\. Wang, L\. Xie, H\. Zhao, H\. Qian, J\. Lui,et al\.\(2024\)D\-llm: a token adaptive computing resource allocation strategy for large language models\.Advances in Neural Information Processing Systems37,pp\. 1725–1749\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.18280#S2.SS2.p1.1)\.
- G\. Li, Y\. Tang, and W\. Zhang \(2024\)LoRAP: transformer sub\-layers deserve differentiated structured compression for large language models\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.18280#S2.SS1.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2607.18280#S2.p1.1)\.
- J\. E\. Liang \(2025\)From entropy rate to redundancy: information dynamics in large language models\.InNeurIPS 2025 Workshop on Structured Probabilistic Inference\{\\\{\\\\backslash&\}\\\}Generative Modeling,Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p4.1)\.
- X\. Lin, Z\. Huang, Z\. Zhang, J\. Zhou, and E\. Chen \(2025\)Explore what llm does not know in complex question answering\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 24585–24594\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p1.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2017\)Pointer sentinel mixture models\.InICLR,Vol\.abs/1609\.07843\.Cited by:[§3\.1](https://arxiv.org/html/2607.18280#S3.SS1.p1.1)\.
- T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal \(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2381–2391\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1260)Cited by:[§3\.1](https://arxiv.org/html/2607.18280#S3.SS1.p1.1)\.
- D\. Raposo, S\. Ritter, B\. Richards, T\. Lillicrap, P\. C\. Humphreys, and A\. Santoro \(2024\)Mixture\-of\-depths: dynamically allocating compute in transformer\-based language models\.arXiv preprint arXiv:2404\.02258\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.18280#S2.SS2.p1.1)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2021\)Winogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§3\.1](https://arxiv.org/html/2607.18280#S3.SS1.p1.1)\.
- H\. Su, Z\. Tian, X\. Shen, and X\. Cai \(2024\)Unraveling the mystery of scaling laws: part i\.arXiv preprint arXiv:2403\.06563\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p1.1)\.
- H\. Su, X\. Zhou, H\. Yu, X\. Shen, Y\. Chen, Z\. Zhu, Y\. Yu, and J\. Zhou \(2022\)Welm: a well\-read pre\-trained language model for chinese\.arXiv preprint arXiv:2209\.10372\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p1.1)\.
- H\. Wu, Y\. Fan, J\. Dai, J\. Tong, Y\. Ma, and X\. Shen \(2026\)Hidrop: hierarchical vision token reduction in mllms via late injection, concave pyramid pruning, and early exit\.arXiv preprint arXiv:2602\.23699\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p2.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)Hellaswag: can a machine really finish your sentence?\.arXiv preprint arXiv:1905\.07830\.Cited by:[§3\.1](https://arxiv.org/html/2607.18280#S3.SS1.p1.1)\.
- Z\. Zhang, C\. Wang, Y\. Wang, E\. Shi, Y\. Ma, W\. Zhong, J\. Chen, M\. Mao, and Z\. Zheng \(2025\)Llm hallucinations in practical code generation: phenomena, mechanism, and mitigation\.Proceedings of the ACM on Software Engineering2\(ISSTA\),pp\. 481–503\.Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p1.1)\.
- A\. Zhao, F\. Ye, Y\. Fan, J\. Tong, J\. Xiong, Z\. Fei, H\. Su, and X\. Shen \(2025\)SkipGPT: each token is one of a kind\.InForty\-second International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.18280#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.18280#S2.SS1.SSS0.Px2.p2.3),[§2](https://arxiv.org/html/2607.18280#S2.p1.1)\.