Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression

arXiv cs.AI Papers
llm-compression sparsity quantization low-rank-approximation efficiency scalability optimization

Summary

The paper introduces the 'Compression Trinity' framework, jointly applying sparsity, quantization, and low-rank approximations to compress Large Language Models for improved efficiency and performance.

arXiv:2608.24070v1 Announce Type: new Abstract: Prohibitive computational and environmental costs impede the scalable deployment of Large Language Models (LLMs). Traditional compression techniques (sparsity, quantization, low-rank approximations) are typically applied in isolation, and each hits an accuracy-efficiency wall. This thesis proposes the "Compression Trinity," a unified framework that applies the three pillars jointly: sparsity to reduce computation, quantization to minimize memory bandwidth, and low-rank approximations to recover accuracy. To accelerate pretraining, we apply the Trinity to the optimizer and model architecture. MKOR approximates curvature via block-diagonal sparsity and low-rank inversion, maintaining numerical stability for quantized states; it reduces curvature update complexity from $O(d^3)$ to $O(d^2)$ and accelerates convergence by up to 1.85x over KFAC. SLoPe accelerates training by up to 1.25x via a double-pruned backward pass for N:M sparsity, using low-rank "lazy" adapters in the final 1% of training to recover accuracy. For post-training compression, OPTIMA stabilizes static masks in a zero-training regime by formulating weight reconstruction as globally optimal column-wise quadratic programs, improving zero-shot accuracy by up to 3.97%. Given a fine-tuning budget, PATCH breaks the ceiling of static masks by learning a dynamic hybrid sparsity ratio between 0% and 50%, yielding up to 1.38x speedups. Finally, SLiM realizes the full Compression Trinity in one shot, using mathematically derived low-rank adapters to recover information lost to quantization and sparsity, improving accuracy by up to 5.66% over state-of-the-art methods and outperforming uncompressed dense models at equal parameter budgets by 0.6%. Together, these results show that jointly applying the Compression Trinity is essential for efficient, scalable, high-performance LLMs.
Original Article
View Cached Full Text

Cached at: 08/26/26, 09:21 AM 

# Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression
Source: [https://arxiv.org/abs/2608.24070](https://arxiv.org/abs/2608.24070)
[View PDF](https://arxiv.org/pdf/2608.24070)

> Abstract:Prohibitive computational and environmental costs impede the scalable deployment of Large Language Models \(LLMs\)\. Traditional compression techniques \(sparsity, quantization, low\-rank approximations\) are typically applied in isolation, and each hits an accuracy\-efficiency wall\. This thesis proposes the "Compression Trinity," a unified framework that applies the three pillars jointly: sparsity to reduce computation, quantization to minimize memory bandwidth, and low\-rank approximations to recover accuracy\. To accelerate pretraining, we apply the Trinity to the optimizer and model architecture\. MKOR approximates curvature via block\-diagonal sparsity and low\-rank inversion, maintaining numerical stability for quantized states; it reduces curvature update complexity from $O\(d^3\)$ to $O\(d^2\)$ and accelerates convergence by up to 1\.85x over KFAC\. SLoPe accelerates training by up to 1\.25x via a double\-pruned backward pass for N:M sparsity, using low\-rank "lazy" adapters in the final 1% of training to recover accuracy\. For post\-training compression, OPTIMA stabilizes static masks in a zero\-training regime by formulating weight reconstruction as globally optimal column\-wise quadratic programs, improving zero\-shot accuracy by up to 3\.97%\. Given a fine\-tuning budget, PATCH breaks the ceiling of static masks by learning a dynamic hybrid sparsity ratio between 0% and 50%, yielding up to 1\.38x speedups\. Finally, SLiM realizes the full Compression Trinity in one shot, using mathematically derived low\-rank adapters to recover information lost to quantization and sparsity, improving accuracy by up to 5\.66% over state\-of\-the\-art methods and outperforming uncompressed dense models at equal parameter budgets by 0\.6%\. Together, these results show that jointly applying the Compression Trinity is essential for efficient, scalable, high\-performance LLMs\.

## Submission history

From: Mohammad Mozaffari \[[view email](https://arxiv.org/show-email/be0f1db8/2608.24070)\] **\[v1\]**Tue, 25 Aug 2026 05:09:07 UTC \(6,983 KB\)

Similar Articles

Break Through the Compression Bottleneck: From Theory to Practice

arXiv cs.CL

This paper provides the first mathematical proof that low-rank decomposition and quantization are non-orthogonal when combined for LLM compression, leading to performance degradation, and proposes a novel Diagonal Adhesive Method (DAM) to mitigate this loss.