Break Through the Compression Bottleneck: From Theory to Practice

# Break Through the Compression Bottleneck: From Theory to Practice
Source: [https://arxiv.org/html/2607.20434](https://arxiv.org/html/2607.20434)
Xiusheng Huang1,2,3, Lu Wang4, Yequan Wang3†\\dagger, Jun Zhao1,2, Kang Liu1,2†\\dagger 1The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China 2School of Artificial Intelligence, University of Chinese Academy of Sciences 3Beijing Academy of Artificial Intelligence4Ritzz\-AI huangxiusheng2020@ia\.ac\.cn,\{wangluloveslezhi,tshwangyequan\}@gmail\.com, \{jzhao,kliu\}@nlpr\.ia\.ac\.cn

###### Abstract

As the parameter size of language models continues to grow, effective model compression is required to reduce their computational and memory overhead\. Existing compression methods suffer from bottleneck issues: when the compression ratio is increased, performance degrades significantly\. Low\-rank decomposition and quantization are two prominent compression methods that have been proven to significantly reduce the computational and memory requirements of Large Language Models \(LLMs\) while maintaining model accuracy\. Evidently, combining these two methods will break through the existing compression bottleneck\. However, how these two methods interact when combined remains a critical question for developers, as many assume they are orthogonal, meaning their combination would not introduce additional errors beyond those independently introduced by each method\. This paper provides the first mathematical proof that low\-rank decomposition and quantization are non\-orthogonal\. We validate these findings through a series of experiments on large language models\. Our results demonstrate that these methods are non\-orthogonal, and their combination leads to significant performance degradation\. Importantly, we propose a novel approach Diagonal Adhesive Method \(DAM\), which can effectively combine the two methods and mitigate the performance loss\. Our research provides deep insights into model compression and lays a solid theoretical and experimental foundation for future related studies\.

Break Through the Compression Bottleneck: From Theory to Practice

Xiusheng Huang1,2,3, Lu Wang4, Yequan Wang3†\\dagger, Jun Zhao1,2, Kang Liu1,2†\\dagger1The Key Laboratory of Cognition and Decision Intelligence for Complex Systems,Institute of Automation, Chinese Academy of Sciences, Beijing, China2School of Artificial Intelligence, University of Chinese Academy of Sciences3Beijing Academy of Artificial Intelligence4Ritzz\-AIhuangxiusheng2020@ia\.ac\.cn,\{wangluloveslezhi,tshwangyequan\}@gmail\.com,\{jzhao,kliu\}@nlpr\.ia\.ac\.cn

††footnotetext:†\\daggerCorresponding authors## 1Introduction

In recent years, Large Language Models \(LLMs\) have demonstrated excellent performance in numerous natural language processing tasks, thanks to their increasing parameter counts\(Huanget al\.,[2021](https://arxiv.org/html/2607.20434#bib.bib53),[2022](https://arxiv.org/html/2607.20434#bib.bib54)\)\. However, this growth in parameter size comes at the cost of significantly higher computational and storage demands\(Menget al\.,[2022](https://arxiv.org/html/2607.20434#bib.bib26)\)\. Consequently, efficient and low\-cost deployment of LLMs has become a critical area of research\(Guptaet al\.,[2023](https://arxiv.org/html/2607.20434#bib.bib25)\)\. Prior work in this field can be broadly categorized into two main approaches: Architecture\-modifying techniques and Architecture\-agnostic techniques\(Dinget al\.,[2023](https://arxiv.org/html/2607.20434#bib.bib50)\)\.

Architecture\-modifying techniques include distillation and pruning\(Hwanget al\.,[2021](https://arxiv.org/html/2607.20434#bib.bib29)\)\. Distillation explicitly extracts knowledge from a large model and utilizes it to train a smaller one\(Poradaet al\.,[2021](https://arxiv.org/html/2607.20434#bib.bib37)\)\. Pruning, on the other hand, removes less important parameters to reduce computational and storage overhead\. However, both distillation and pruning are often impractical for LLMs due to their substantial requirements for training data and compute resources\. Architecture\-agnostic techniques include quantization and low\-rank decomposition\(Levyet al\.,[2017](https://arxiv.org/html/2607.20434#bib.bib38)\)\. Quantization reduces the precision of model weights or activations, typically from 32\-bit floating\-point to lower bit representations like 8\-bit or 4\-bit integers, or even binary\(Ashkbooset al\.,[2025](https://arxiv.org/html/2607.20434#bib.bib23)\)\. Low\-rank decomposition approximates weight matrices with lower\-rank matrices, reducing parameter count while keeping the weights in floating\-point format\(Yuanet al\.,[2023](https://arxiv.org/html/2607.20434#bib.bib13)\)\. Quantization and low\-rank decomposition are popular choices for deploying LLMs practically because they are low\-cost, architecture\-agnostic, and generally perform well\.

Existing quantization and low\-rank decomposition methods both face performance bottlenecks\(Sunet al\.,[2025](https://arxiv.org/html/2607.20434#bib.bib105); Yuanet al\.,[2023](https://arxiv.org/html/2607.20434#bib.bib13)\)\. For example, quantization can maintain good performance at precisions above W4A4KV4, but when the model is further compressed to lower bit\-widths, there is a significant and unacceptable drop in accuracy\(Huet al\.,[2025](https://arxiv.org/html/2607.20434#bib.bib108)\)\. Similarly, low\-rank decomposition suffers a notable decline in performance when the compression ratio exceeds 50%\(Wanget al\.,[2024](https://arxiv.org/html/2607.20434#bib.bib14)\)\. Extensive experiments indicate that W4A4KV4 and a 50% compression ratio represent the current bottlenecks for model compression techniques, limiting the potential for further high\-fidelity compression\. An inspiring idea is to effectively combine these two architecture\-agnostic techniques\.

It is natural to expect that directly combining quantization and low\-rank decomposition can yield significant benefits in terms of computation and storage costs\. However, the potential drawbacks remain unclear\. Previous studies have assumed that these two methods are orthogonal, meaning their combination would not introduce additional errors beyond those of each individual method\(Wanget al\.,[2024](https://arxiv.org/html/2607.20434#bib.bib14)\)\. However, these studies have primarily focused on quantizing weights only, making extra error from combining quantization and low\-rank decomposition less noticeable, and overlooking the widespread presence of outliers when quantifying activation values\. These studies have failed to properly investigate the overall impact of utilizing both methods together or to provide effective strategies for their integration\. This poses a major challenge for further low\-cost deployment of LLMs\.

To address these challenges, we first conduct a theoretical analysis from both the tensor and dot\-product perspectives, demonstrating that quantization and low\-rank decomposition are non\-orthogonal and thus introduce additional error\. In addition, we find that the order in which these two methods are applied significantly affects model performance, and we derive the theoretically optimal sequence—applying low\-rank decomposition before quantization\. Finally, we identify outliers of activation as a key factor impacting compressed model performance, and propose a Diagonal Adhesive Method \(DAM\) that can significantly reduce the extra losses caused by the combination of quantization and low\-rank decomposition\.

To the best of our knowledge, we are the first to theoretically demonstrate that quantization and low\-rank decomposition are non\-orthogonal and provide the correct compression order\. Based on the outliers issue inherent in low\-rank decomposition, we propose the DAM method\. Extensive experiments have shown that the DAM method significantly improves the performance of compressed models\. Our research provides deep insights into model compression, and lays a solid theoretical and experimental foundation for future related studies\. Our contributions are summarized below:

- •We mathematically prove that quantization and low\-rank decomposition are non\-orthogonal operations\. Based on compression error analysis, their combination introduces compound errors and leads to performance degradation\. Our findings provide a theoretical foundation and challenge the conventional belief that combining quantization and low\-rank decomposition does not significantly impact performance\.
- •To improve the performance of combining quantization and low\-rank decomposition, we are the first to derive the optimal order—applying low\-rank decomposition before quantization\. This finding is further supported by extensive experimental results\.
- •We propose a Diagonal Adhesive Method \(DAM\) to address the performance degradation caused by the combination of quantization and low\-rank decomposition\. Extensive experiments demonstrate that while maintaining low cost and high speed, DAM significantly improves performance\.

## 2Related Work

##### Quantization\.

Post\-training quantization \(PTQ\) has emerged as a prominent technique for large language models \(LLMs\) due to its efficiency\. Current PTQ methods can generally be categorized into weight\-only and weight\-activation quantization\. To minimize memory usage, some strategies concentrate on weight\-only quantization\. GPTQ employs Hessian\-based error compensation to achieve significant compression rates by reducing quantization errors\(Frantaret al\.,[2022](https://arxiv.org/html/2607.20434#bib.bib15)\)\. AWQ\(Linet al\.,[2024](https://arxiv.org/html/2607.20434#bib.bib16)\)enhances performance by tackling the effects of activation outliers on weight quantization\(Leeet al\.,[2023](https://arxiv.org/html/2607.20434#bib.bib17)\)\. QuIP\(Cheeet al\.,[2023](https://arxiv.org/html/2607.20434#bib.bib18)\)and QuIP\#\(Tsenget al\.,[2024](https://arxiv.org/html/2607.20434#bib.bib19)\)utilize random Hadamard matrices for incoherent processing and apply vector quantization to weights, resulting in improved performance compared to reduced precision quantization\. SmoothQuant\(Xiaoet al\.,[2023](https://arxiv.org/html/2607.20434#bib.bib20)\)shifts the difficulty of quantization from activations to weights through a mathematical transformation\. OmniQuant\(Shaoet al\.,[2023](https://arxiv.org/html/2607.20434#bib.bib21)\)further boosts performance by training quantization parameters and transformation coefficients\. Additionally, I\-LLM\(Huet al\.,[2024](https://arxiv.org/html/2607.20434#bib.bib22)\)proposes a strategy for integer\-only quantization and inference through fully\-smooth block reconstruction and fully integer operators\. Recently, QuaRot\(Ashkbooset al\.,[2025](https://arxiv.org/html/2607.20434#bib.bib23)\)employs random rotation matrices to facilitate 4\-bit quantization of weights and activations, while SpinQuant\(Liuet al\.,[2024](https://arxiv.org/html/2607.20434#bib.bib24)\)learns these matrices to refine the 4\-bit quantization process\.

##### Low\-Rank Decomposition\.

Singular Value Decomposition \(SVD\) is a common technique for reducing matrix size by approximating a matrix with two smaller low\-rank matrices\(Golubet al\.,[1987](https://arxiv.org/html/2607.20434#bib.bib11)\)\. In the realm of LLM compression, only a limited number of SVD\-based methods have been suggested\. Specifically, standard SVD focuses solely on compressing the original weight matrix without accounting for the significance of the parameters, which may result in a higher compression error\. To tackle this issue, FWSVD method was introduced that incorporates Fisher information to assess parameter importance\([Hsuet al\.,](https://arxiv.org/html/2607.20434#bib.bib12)\)\. However, this approach necessitates a complex gradient calculation, requiring significant resources for LLM compression\. Another challenge with standard SVD is the distribution of activation, which can influence compression accuracy\. To address this, ASVD method was proposed that scales the weight matrix utilizing a diagonal matrix to reflect the impact of input channels on the weights\(Yuanet al\.,[2023](https://arxiv.org/html/2607.20434#bib.bib13)\)\. Nonetheless, both methods do not establish a clear relationship between singular values and compression loss\.

##### Combining Quantization and Low\-rank Decomposition\.

Previous studies have explored the combination of quantization and low\-rank decomposition for model compression\. SVD\-LLM only compresses the model’s weights without addressing the activation values and neglects their existence\(Wanget al\.,[2024](https://arxiv.org/html/2607.20434#bib.bib14)\)\. Prior research has not provided clear conclusions regarding the optimal order of these two methods, nor has it offered solutions to the problem of outliers\. These issues pose significant challenges for further high\-quality model compression\.

## 3Non\-orthogonality of Quantization and Low\-rank Decomposition

##### Definition 3\.1 \(Quantization Method\)\.

The existing quantization method is a block based quantization method that divides the weight matrix into multiple blocks and quantizes each block independently\. For each block, utilize the maximum absolute value within that block as the scaling factor\.

Q​\(W\)=Round​\(Wmax⁡\(\|W\|\)⋅2b−1\)Q\(W\)=\\text\{Round\}\(\\frac\{W\}\{\\max\(\|W\|\)\}\\cdot 2^\{b\-1\}\)\(1\)
among them,Q​\(W\)Q\(W\)represents the quantized block,WWrepresents the original weight block,max⁡\(\|W\|\)\\max\(\|W\|\)represents the maximum absolute value of the elements in blockWW,bbrepresents the quantization bit width, Round\(⋅\\cdot\) represents rounding operation\. The inverse quantification formula is

D​\(Q​\(W\)\)=Q​\(W\)∗max⁡\(\|W\|\)/\(2b−1\)D\(Q\(W\)\)=Q\(W\)\*\\max\(\|W\|\)/\(2^\{b\}\-1\)\(2\)

##### Definition 3\.2 \(Quantization Error\)\.

We formalize the quantification error and theoretical error boundary as follows

E​\(W​x\)=\\displaystyle E\(Wx\)=\|W​x−D​\(Q​\(W\)\)​x\|,\|E​\(W​x\)\|\\displaystyle\|Wx\-D\(Q\(W\)\)x\|,\\ \\ \|E\(Wx\)\|\(3\)≤max⁡\(\|W\|\)/\(2∗\(2b−1\)\)​x\\displaystyle\\leq\\max\(\|W\|\)/\(2\*\(2^\{b\}\-1\)\)x
whereE​\(W​x\)E\(Wx\)is the quantization error and\|E​\(W​x\)\|\|E\(Wx\)\|is the maximum error boundary,xxis the input for weightWW\. The detailed proof process is provided in Appendix[A](https://arxiv.org/html/2607.20434#A1)\.

##### Definition 3\.3 \(Low\-Rank Decomposition Method\)\.

For any matrixW∈ℝm​×​nW\\in\\mathbb\{R\}^\{m×n\}, its Singular Value Decomposition \(SVD\) can be expressed as

W=U​Σ​VT\\displaystyle W=U\\Sigma V^\{T\}\(4\)
whereU∈ℝm×mU\\in\\mathbb\{R\}^\{m\\times m\}is a left singular vector matrix \(orthogonal matrix\),Σ∈ℝm×n\\Sigma\\in\\mathbb\{R\}^\{m\\times n\}is a singular value diagonal matrix,V∈ℝn×nV\\in\\mathbb\{R\}^\{n\\times n\}is the transpose \(orthogonal matrix\) of the right singular vector matrix\. Appendix[B](https://arxiv.org/html/2607.20434#A2)shows the Low\-Rank Decomposition Error\.

### 3\.1Tensor\-level Analysis

##### Definition 3\.5 \(Compression Error\)\.

Previous studies did not consider the optimal application order of quantization and low\-rank decomposition, and we provided compression errors for different orders\. For the compression error ofl∘ql\\circ q, we have:

El\\displaystyle E\_\{l\}=‖W−Ur∗Σr∗VrT‖F,\\displaystyle=\|\|W\-U\_\{r\}\*\\Sigma\_\{r\}\*V\_\{r\}^\{T\}\|\|\_\{F\},\(5\)Eq\\displaystyle E\_\{q\}=‖Q​\(Ur\)∗Q​\(Σr\)∗Q​\(VrT\)−Ur∗Σr∗VrT‖F,\\displaystyle=\|\|Q\(U\_\{r\}\)\*Q\(\\Sigma\_\{r\}\)\*Q\(V\_\{r\}^\{T\}\)\-U\_\{r\}\*\\Sigma\_\{r\}\*V\_\{r\}^\{T\}\|\|\_\{F\},El∘q\\displaystyle E\_\{l\\circ q\}=‖Q​\(Ur\)∗Q​\(Σr\)∗Q​\(VrT\)−W‖F\\displaystyle=\|\|Q\(U\_\{r\}\)\*Q\(\\Sigma\_\{r\}\)\*Q\(V\_\{r\}^\{T\}\)\-W\|\|\_\{F\}
wherel∘ql\\circ qrepresents utilizing low\-rank decomposition first, and then utilizing quantization methods,rrrepresents the compression ratio of low\-rank decomposition\.Q​\(⋅\)Q\(\\cdot\)represents the utilize of quantitative methods\. For the compression error ofq∘lq\\circ l, we have:

Eq′\\displaystyle E\_\{q^\{\\prime\}\}=‖Q​\(W\)−W‖F,\\displaystyle=\|\|Q\(W\)\-W\|\|\_\{F\},\(6\)El′\\displaystyle E\_\{l^\{\\prime\}\}=‖S​V​Dr​\(Q​\(W\)\)−Q​\(W\)‖F\\displaystyle=\|\|SVD\_\{r\}\(Q\(W\)\)\-Q\(W\)\|\|\_\{F\}Eq∘l\\displaystyle E\_\{q\\circ l\}=‖S​V​Dr​\(Q​\(W\)\)−W‖F\\displaystyle=\|\|SVD\_\{r\}\(Q\(W\)\)\-W\|\|\_\{F\}

##### Definition 3\.6 \(Tensor\-level Orthogonality\)\.

If any combination order ofqqandlldoes not introduce additional errors, we define the two compression methods as orthogonal, satisfying the following inequality:

∀W∈ℝn,‖El∘q​\(W\)‖\\displaystyle\\forall W\\in\\mathbb\{R\}^\{n\},\\\|E\_\{l\\circ q\}\(W\)\\\|≤El​\(W\)\+Eq​\(W\)\\displaystyle\\leq E\_\{l\}\(W\)\+E\_\{q\}\(W\)\(7\)and​‖Eq∘l​\(W\)‖\\displaystyle\\text\{ and \}\\\|E\_\{q\\circ l\}\(W\)\\\|≤Eq′​\(W\)\+El′​\(W\)\\displaystyle\\leq E\_\{q^\{\\prime\}\}\(W\)\+E\_\{l^\{\\prime\}\}\(W\)

##### Theorem 3\.7\.

According to the Appendix[C](https://arxiv.org/html/2607.20434#A3), we prove that low\-rank decomposition followed by quantization does not introduce additional compression errors, satisfying the following inequality:

∀W∈ℝn,‖El∘q​\(W\)‖≤El​\(W\)\+Eq​\(W\)\\displaystyle\\forall W\\in\\mathbb\{R\}^\{n\},\\\|E\_\{l\\circ q\}\(W\)\\\|\\leq E\_\{l\}\(W\)\+E\_\{q\}\(W\)\(8\)
Eq\.[8](https://arxiv.org/html/2607.20434#S3.E8)states that after low\-rank decomposition, the maximum error in the matrix has a specific limit and is strongly correlated with the singular valuesσi\\sigma\_\{i\}, ensuring the determinacy of the scale quantization parameter\. Therefore, the quantization error of non\-zero vectors before and after low\-rank decomposition remains unchanged\.

##### Theorem 3\.8\.

According to the Appendix[D](https://arxiv.org/html/2607.20434#A4), we demonstrate that quantization before low\-rank decomposition introduces additional errors:

∃W∈ℝn,‖Eq∘l​\(W\)‖\>Eq′​\(W\)\+El′​\(W\)\\displaystyle\\exists W\\in\\mathbb\{R\}^\{n\},\\\|E\_\{q\\circ l\}\(W\)\\\|\>E\_\{q^\{\\prime\}\}\(W\)\+E\_\{l^\{\\prime\}\}\(W\)\(9\)
Eq\.[9](https://arxiv.org/html/2607.20434#S3.E9)states that the quantization operationQ​\(⋅\)Q\(\\cdot\)introduces rounding errors, which can alter the singular value distribution of the matrix, affecting the accuracy ofS​V​DSVDdecomposition and ultimately leading to a cumulative effect of errors greater than simple superposition\. This indicates that the compression strategy of quantization first and thenS​V​DSVDwill lead to larger errors, and it is better to adopt the strategy ofS​V​DSVDfirst and then quantization\.

### 3\.2Dot\-product Level Analysis

In this section, we delve into the combined effects of quantification and low\-rank decomposition at the dot product level\. Our analysis focuses on the application of quantization and low\-rank decomposition to weight, while activation only applies quantization operations\. We first extend the definition of compression error to the dot\-product level\.

##### Definition 3\.9 \(Compression Error on Dot\-product Level\)\.

Letx,w∈ℝnx,w\\in\\mathbb\{R\}^\{n\}denote the inputs of a compressionl/q:ℝn⇒ℝnl/q:\\mathbb\{R\}^\{n\}\\Rightarrow\\mathbb\{R\}^\{n\}and the dot product operation<\.,\.\>:ℝn×ℝn⇒ℝ<\.,\.\>:\\mathbb\{R\}^\{n\}\\times\\mathbb\{R\}^\{n\}\\Rightarrow\\mathbb\{R\}\. We defineEq,lD​\(𝐱,𝐰\)=⟨𝐱,𝐰⟩−⟨q​\(𝐱\),q∘l​\(𝐰\)⟩E\_\{q,l\}^\{D\}\(\\mathbf\{x\},\\mathbf\{w\}\)=\\langle\\mathbf\{x\},\\mathbf\{w\}\\rangle\-\\langle q\(\\mathbf\{x\}\),q\\circ l\(\\mathbf\{w\}\)\\rangleas the compression error on dot\-product level\. Meanwhile, we defineEl,qD​\(𝐱,𝐰\)=⟨𝐱,𝐰⟩−⟨q​\(𝐱\),l∘q​\(𝐰\)⟩E\_\{l,q\}^\{D\}\(\\mathbf\{x\},\\mathbf\{w\}\)=\\langle\\mathbf\{x\},\\mathbf\{w\}\\rangle\-\\langle q\(\\mathbf\{x\}\),l\\circ q\(\\mathbf\{w\}\)\\rangleas the compression error when utilizing another order to compress weight\.

##### Definition 3\.10 \(Dot\-product Level Orthogonality\)\.

Letffdenote a composition ofqqandllin any order,f:=q∘lf:=q\\circ lorf:=l∘qf:=l\\circ q\. We assume that if the combination of quantization and low\-rank decomposition does not introduce additional errors, then they are orthogonal, defined as follows:

∀𝐱,𝐰∈ℝn,\|El,qD​\(𝐱,𝐰\)\|=\|Eq,lD​\(𝐱,𝐰\)\|\\displaystyle\\forall\\mathbf\{x\},\\mathbf\{w\}\\in\\mathbb\{R\}^\{n\},\|E\_\{l,q\}^\{D\}\(\\mathbf\{x\},\\mathbf\{w\}\)\|=\|E\_\{q,l\}^\{D\}\(\\mathbf\{x\},\\mathbf\{w\}\)\|\(10\)

##### Theorem 3\.11\.

Letqqbe the quantization, s be low\-rank decomposition\. We demonstrate that any order of quantization and low\-rank decomposition will result in additional compression errors:

∃𝐱,𝐰∈ℝn,\|El,qD​\(𝐱,𝐰\)\|<\|Eq,lD​\(𝐱,𝐰\)\|\\displaystyle\\exists\\mathbf\{x\},\\mathbf\{w\}\\in\\mathbb\{R\}^\{n\},\|E\_\{l,q\}^\{D\}\(\\mathbf\{x\},\\mathbf\{w\}\)\|<\|E\_\{q,l\}^\{D\}\(\\mathbf\{x\},\\mathbf\{w\}\)\|\(11\)
The Appendix[E](https://arxiv.org/html/2607.20434#A5)provides a detailed proof process\. We have demonstrated that quantization and low\-rank decomposition are non orthogonal at the dot\-product level\.

![Refer to caption](https://arxiv.org/html/2607.20434v1/before.jpeg)

![Refer to caption](https://arxiv.org/html/2607.20434v1/after.jpg)

Fig\. 1:Visualization of activation values<Σr​VrT\><\\Sigma\_\{r\}V\_\{r\}^\{T\}\>, the left and right images respectively show the 3D visualization of activation values before and after utilizing DAM\. The X and Y axes represent the dimensions of the activation matrix, while the Z axis indicates the magnitude of the values\. The left figure shows that before applying the DAM method, the activation values are steeply distributed with clear outliers, resulting in a Z\-axis maximum of 32, which significantly increases compression error\. The right figure demonstrates that the DAM method effectively eliminates outliers, reducing the Z\-axis maximum to 3\.0\.

## 4Diagonal Adhesive Method

The theory in Section[3](https://arxiv.org/html/2607.20434#S3)suggests that utilizing low\-rank decomposition before quantization is the optimal compression strategy\. However, these two compression methods have been utilized independently in the past, and existing research has not yet explored the problems and solutions when combined\.

After undergoing low\-rank decomposition, the weight is decomposed into the form ofUr​Σr​VrTU\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}, whereUrU\_\{r\}andVrTV\_\{r\}^\{T\}are relatively smooth states\. In the quantization process, we choose to quantizeQ​\(Ur\)Q\(U\_\{r\}\)andQ​\(Σr​VrT\)Q\(\\Sigma\_\{r\}V\_\{r\}^\{T\}\)\. However, due to the presence ofΣr\\Sigma\_\{r\}, there is a significant quantization error when quantizingQ​\(Σr​VrT\)Q\(\\Sigma\_\{r\}V\_\{r\}^\{T\}\)\. This is because<Σr​VrT\><\\Sigma\_\{r\}V\_\{r\}^\{T\}\>contains outliers \(As shown in the left image of Figure[1](https://arxiv.org/html/2607.20434#S3.F1)\), which poses a huge challenge for quantifying LLM\. Outliers refer to a small number of elements that are significantly larger than the mean of the entire matrix\. They reduce the effective utilization of the quantization space and increase quantization error\.

### 4\.1Problem Modeling

To address this issue, we propose a Diagonal Adhesive Method \(DAM\)\. Introduce diagonal matrixaaand rewrite the decomposition as:

Ur​Σr​VrT=\(Ur​a\)⋅\(a−1​Σr​VrT\)\\displaystyle U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}=\(U\_\{r\}a\)\\cdot\(a^\{\-1\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\)\(12\)
where the diagonal elementsai\>0a\_\{i\}\>0of a are used to scale the columns ofUrU\_\{r\}and rows ofΣr​VrT\\Sigma\_\{r\}V\_\{r\}^\{T\}\. DAM shifts the burden of outlier removal from<Σr​VrT\><\\Sigma\_\{r\}V\_\{r\}^\{T\}\>to<Ur\><U\_\{r\}\>\. While eliminating outliers, DAM has no impact on the model’s inference speed or storage cost\. The quantized product is:

Q​\(Ur​a\)⋅Q​\(a−1​Σr​VrT\)\\displaystyle Q\(U\_\{r\}a\)\\cdot Q\(a^\{\-1\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\)\(13\)
The objective is to choose a to minimize the quantization error, that is to minimize:

‖Q​\(Ur​a\)⋅Q​\(a−1​Σr​VrT\)−Ur​Σr​VrT‖F2\\displaystyle\|\|Q\(U\_\{r\}a\)\\cdot Q\(a^\{\-1\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\)\-U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\|\|\_\{F\}^\{2\}\(14\)

### 4\.2Quantization Error Analysis

Assume the quantization error is additive noise, that is:

Q​\(W\)=W\+E\\displaystyle Q\(W\)=W\+E\(15\)
where each element ofEEis independent with zero mean, and the variance is related to the quantization step size\. For the scaled matricesUr​aU\_\{r\}aanda−1​Σr​VrTa^\{\-1\}\\Sigma\_\{r\}V\_\{r\}^\{T\}, their quantization error variances are proportional toai2a\_\{i\}^\{2\}andσi2ai2\\frac\{\\sigma\_\{i\}^\{2\}\}\{a\_\{i\}^\{2\}\}respectively, whereσi\\sigma\_\{i\}is theii\-th diagonal element ofΣr\\Sigma\_\{r\}\. The quantized product error can be approximated as:

Q​\(Ur​a\)⋅Q​\(1a​Σr​VrT\)−Ur​Σr​VrT\\displaystyle Q\(U\_\{r\}a\)\\cdot Q\\left\(\\frac\{1\}\{a\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\\right\)\-U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\(16\)≈Ur​a​E2\+E1​1a​Σr​VrT\\displaystyle\\approx U\_\{r\}aE\_\{2\}\+E\_\{1\}\\frac\{1\}\{a\}\\Sigma\_\{r\}V\_\{r\}^\{T\}
whereE1E\_\{1\}andE2E\_\{2\}are quantization error matrices\.

### 4\.3Error Decomposition and Optimization

For each rankii, analyze independently whereuiu\_\{i\}andviv\_\{i\}are theii\-th columns ofUrU\_\{r\}andVrV\_\{r\}respectively, andσi\\sigma\_\{i\}is theii\-th diagonal element ofΣr\\Sigma\_\{r\}\. The squared Frobenius norm of the quantization error is:

‖ai​ui​e2,i\+σiai​e1,i​viT‖F2\\displaystyle\\left\\\|a\_\{i\}u\_\{i\}e\_\{2,i\}\+\\frac\{\\sigma\_\{i\}\}\{a\_\{i\}\}e\_\{1,i\}v\_\{i\}^\{T\}\\right\\\|\_\{F\}^\{2\}⇒𝔼​\[‖ai​ui​e2,i‖F2\]\\displaystyle\\Rightarrow\\mathbb\{E\}\\left\[\\left\\\|a\_\{i\}u\_\{i\}e\_\{2,i\}\\right\\\|\_\{F\}^\{2\}\\right\]\(17\)\+𝔼​\[‖σiai​e1,i​viT‖F2\]\\displaystyle\+\\mathbb\{E\}\\left\[\\left\\\|\\frac\{\\sigma\_\{i\}\}\{a\_\{i\}\}e\_\{1,i\}v\_\{i\}^\{T\}\\right\\\|\_\{F\}^\{2\}\\right\]
wheree1,ie\_\{1,i\}ande2,ie\_\{2,i\}are quantization error vectors\. Assuming quantization error variances areVar​\(e1,i\)=c1​ai2\\text\{Var\}\(e\_\{1,i\}\)=c\_\{1\}a\_\{i\}^\{2\}andVar​\(e2,i\)=c2​σi2ai2\\text\{Var\}\(e\_\{2,i\}\)=c\_\{2\}\\frac\{\\sigma\_\{i\}^\{2\}\}\{a\_\{i\}^\{2\}\}, the total error is:

c2​σi2​n\+c1​σi2​m\\displaystyle c\_\{2\}\\sigma\_\{i\}^\{2\}n\+c\_\{1\}\\sigma\_\{i\}^\{2\}m\(18\)
To balance both terms, chooseaia\_\{i\}such that:

c2​σi2​n=c1​σi2​m⟹c2c1=mn\\displaystyle c\_\{2\}\\sigma\_\{i\}^\{2\}n=c\_\{1\}\\sigma\_\{i\}^\{2\}m\\Longrightarrow\\frac\{c\_\{2\}\}\{c\_\{1\}\}=\\frac\{m\}\{n\}\(19\)
This shows there exists anaia\_\{i\}that balances both terms, thus minimizing the total error\.

### 4\.4Existence of Optimal Diagonal Matrix

For each rankii, chooseaia\_\{i\}to balance the variance terms of quantization error, that is:

ai2∝c2​σi2​nc1​m\\displaystyle a\_\{i\}^\{2\}\\propto\\sqrt\{\\frac\{c\_\{2\}\\sigma\_\{i\}^\{2\}n\}\{c\_\{1\}m\}\}\(20\)
Then, the diagonal elements of diagonal matrixaaare:

ai=\(c2​σi2​nc1​m\)1/4\\displaystyle a\_\{i\}=\\left\(\\frac\{c\_\{2\}\\sigma\_\{i\}^\{2\}n\}\{c\_\{1\}m\}\\right\)^\{1/4\}\(21\)
Since the objective function is continuous and has a lower bound, according to the extreme value theorem, there exists such a diagonal matrixaathat minimizes the quantization error\. We constructed an error reconstruction loss to optimize the diagonal matrixaa:

ℒr​e​c​o​n=‖W​x−Q​\(Ur​a\)​Q​\(1a​Σr​VrT\)​x‖F2\\displaystyle\\mathcal\{L\}\_\{recon\}=\\\|Wx\-Q\(U\_\{r\}a\)Q\(\\frac\{1\}\{a\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\)x\\\|\_\{F\}^\{2\}\(22\)

## 5Experiments

##### Models and Datasets\.

We apply our method to the entire LLaMA family, including LLaMA\-1 \(7B\-30B\)\(Touvronet al\.,[2023a](https://arxiv.org/html/2607.20434#bib.bib51)\), LLaMA\-2 \(7B\-13B\)\(Touvronet al\.,[2023b](https://arxiv.org/html/2607.20434#bib.bib33)\), and LLaMA\-3\-8B\. We report perplexity \(PPL\) scores on the WikiText2\(Merityet al\.,[2016](https://arxiv.org/html/2607.20434#bib.bib56)\)test set\. In addition, we also evaluate the models on up to nine zero\-shot tasks utilizing thelm\-evaluation\-harness\(Gaoet al\.,[2024](https://arxiv.org/html/2607.20434#bib.bib57)\), including BoolQ\(Clarket al\.,[2019](https://arxiv.org/html/2607.20434#bib.bib58)\), HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2607.20434#bib.bib59)\), LAMBADA \(OpenAI\)\(Radfordet al\.,[2019](https://arxiv.org/html/2607.20434#bib.bib60)\), OpenBookQA \(OBQA\)\(Mihaylovet al\.,[2018](https://arxiv.org/html/2607.20434#bib.bib61)\), PIQA\(Bisket al\.,[2020](https://arxiv.org/html/2607.20434#bib.bib62)\), SIQA\(Sapet al\.,[2019](https://arxiv.org/html/2607.20434#bib.bib63)\), WinoGrande\(Sakaguchiet al\.,[2021](https://arxiv.org/html/2607.20434#bib.bib64)\), ARC\-Easy, and ARC\-Challenge\(Boratkoet al\.,[2018](https://arxiv.org/html/2607.20434#bib.bib65)\)\.

##### Experiments Setup\.

In all experiments, quantization and low\-rank decomposition will be applied to weights, while activation will only utilize quantization\. For low\-rank decomposition, we adopt the state\-of\-the\-art SVD\-LLM\(Wanget al\.,[2024](https://arxiv.org/html/2607.20434#bib.bib14)\)approach\. For quantization, we leverage the commonly utilized GPTQ\(Frantaret al\.,[2022](https://arxiv.org/html/2607.20434#bib.bib15)\)technique\. The experiment mainly includes three parts: \(1\) verification of compression order, \(2\) verification of non orthogonality of quantization and low\-rank decomposition, and \(3\) verification of diagonal adhesive method\. Appendix[F](https://arxiv.org/html/2607.20434#A6)offers more experimental results\.

### 5\.1Verification of Compression Order

To verify the optimal compression order, we performed order validation experiments\. This section offers empirical evidence that applying low\-rank decomposition before quantization achieves superior model performance compared to the opposite sequence\. These findings align with the conclusions presented in Section[3\.1](https://arxiv.org/html/2607.20434#S3.SS1)\. Table[1](https://arxiv.org/html/2607.20434#S5.T1)details the performance across different quantization bit\-widths and low\-rank decomposition ratios, examining both compression orders\. For instance, “4\-16\-16” signifies that the weights in the quantized model are 4\-bit, while activations and the KV cache maintain full precision\. “40%” represents a 40% compression ratio for low\-rank decomposition\. Optimal results are displayed in bold\.

We experimentally validated two different orderings: \(Q⇒\\RightarrowL\)—Quantization then Low\-rank decomposition, and \(L⇒\\RightarrowQ\)—Low\-rank decomposition then Quantization\. As shown in Table[1](https://arxiv.org/html/2607.20434#S5.T1), the L⇒\\RightarrowQ configuration demonstrates significantly better model performance than Q⇒\\RightarrowL\. Furthermore, the performance gap favoring L⇒\\RightarrowQ widens with increasing compression ratios\. Consistent with the discussion in Section[3\.1](https://arxiv.org/html/2607.20434#S3.SS1), SVD Low\-rank decomposition bounds the maximum error in the matrix, ensuring effective control over the accumulated error in the L⇒\\RightarrowQ approach\.

Table 1:The results of validating the compression order, we highlight the superior metrics, with the quantization set to W4A4KV4\.The compression ratio of the low\-rank decomposition is 50%\.Table 2:The results of the orthogonality verification experiment: 0\-shot refers to the average performance across the nine downstream tasks in Table[1](https://arxiv.org/html/2607.20434#S5.T1), and THo represents the orthogonality threshold\. When 0\-shot is lower than THo, it demonstrates that quantization and low\-rank decomposition are non\-orthogonal\.
### 5\.2Non\-orthogonality Verification of Quantization and Low\-Rank Decomposition

Orthogonality Threshold: Following Equation[11](https://arxiv.org/html/2607.20434#S3.E11), we introduce an orthogonality threshold,T​H​oTHo\. LetPoP\_\{o\}represent the performance of the original model\. The performance of the quantized model \(e\.g\., accuracy\) is denoted asPo​qP\_\{oq\}, and the resulting quantization loss isLo​q=Po−Po​qL\_\{oq\}=P\_\{o\}\-P\_\{oq\}\. Similarly, the low\-rank decomposition loss isLo​l=Po−Po​lL\_\{ol\}=P\_\{o\}\-P\_\{ol\}\. The orthogonality threshold,T​H​oTHo, is then defined as:

T​H​o=Po−Lo​q−Lo​l\\displaystyle THo=P\_\{o\}\-L\_\{oq\}\-L\_\{ol\}\(23\)
For accuracy, where higher values indicate better performance,T​H​oTHowill be less thanPoP\_\{o\}\. Conversely, for perplexity, where lower values are better,T​H​oTHowill be greater thanPoP\_\{o\}\.

This section aims to validate our conclusion from Section[3\.2](https://arxiv.org/html/2607.20434#S3.SS2): combining quantization and sparsity introduces additional error, indicating non\-orthogonality\. We present results for the L\-Q \(Low\-rank decomposition followed by Quantization\) order only\. As shown in Table[2](https://arxiv.org/html/2607.20434#S5.T2), when utilizing accuracy as the metric, the combined model’s performance significantly below the orthogonality thresholdT​H​oTHo\. These empirical findings support the validity of Equation[11](https://arxiv.org/html/2607.20434#S3.E11), confirming the non\-orthogonal nature of quantization and low\-rank decomposition\.

Intriguingly, despite the substantial difference in parameter layer count between LLaMA\-2\-7B and LLaMA\-2\-70B, the performance deviation fromT​H​oTHoremains comparable\. This contradicts our expectation, as we would typically anticipate a significant increase in accumulated compression error with more layers\. We posit that this is due to the smaller outlier magnitudes across layers in LLaMA\-2\-70B\. Consequently, quantizing after low\-rank decomposition does not introduce a proportionally larger additional error\. To explore this, we performed ablation studies\.

\#BitsMethod\(Ratio\)LLaMA\-3\-8BLLaMA\-3\-70BLLaMA\-2\-7BLLaMA\-2\-13BLLaMA\-2\-70BLLaMA\-7BW\-A\-KV0\-shot\(↑\\color\[rgb\]\{0,1,0\}\\uparrow\)Wiki\(↓\\color\[rgb\]\{0,1,0\}\\downarrow\)0\-shot\(↑\\color\[rgb\]\{0,1,0\}\\uparrow\)Wiki\(↓\\color\[rgb\]\{0,1,0\}\\downarrow\)0\-shot\(↑\\color\[rgb\]\{0,1,0\}\\uparrow\)Wiki\(↓\\color\[rgb\]\{0,1,0\}\\downarrow\)0\-shot\(↑\\color\[rgb\]\{0,1,0\}\\uparrow\)Wiki\(↓\\color\[rgb\]\{0,1,0\}\\downarrow\)0\-shot\(↑\\color\[rgb\]\{0,1,0\}\\uparrow\)Wiki\(↓\\color\[rgb\]\{0,1,0\}\\downarrow\)0\-shot\(↑\\color\[rgb\]\{0,1,0\}\\uparrow\)Wiki\(↓\\color\[rgb\]\{0,1,0\}\\downarrow\)16\-16\-16FP16 \(0%\)68\.096\.1473\.812\.8665\.215\.4767\.614\.8871\.593\.3264\.455\.684\-16\-16Q⇒\\RightarrowL \(40%\)42\.343\.847\.727\.936\.437\.441\.239\.745\.329\.935\.437\.8L⇒\\RightarrowQ \(40%\)52\.828\.458\.322\.847\.827\.651\.726\.456\.224\.547\.027\.8\\cellcolorgreen\!10Ours\(40%\)\\cellcolorgreen\!1057\.7\\cellcolorgreen\!1021\.4\\cellcolorgreen\!1064\.7\\cellcolorgreen\!1015\.3\\cellcolorgreen\!1054\.4\\cellcolorgreen\!1020\.5\\cellcolorgreen\!1057\.3\\cellcolorgreen\!1020\.5\\cellcolorgreen\!1062\.3\\cellcolorgreen\!1017\.8\\cellcolorgreen\!1053\.2\\cellcolorgreen\!1021\.3Q⇒\\RightarrowL \(20%\)47\.336\.952\.424\.745\.435\.747\.234\.349\.330\.544\.535\.4L⇒\\RightarrowQ \(20%\)57\.923\.460\.413\.655\.621\.256\.320\.558\.422\.655\.425\.1\\cellcolorgreen\!10Ours\(20%\)\\cellcolorgreen\!1063\.6\\cellcolorgreen\!1016\.5\\cellcolorgreen\!1066\.8\\cellcolorgreen\!107\.1\\cellcolorgreen\!1060\.4\\cellcolorgreen\!1015\.4\\cellcolorgreen\!1062\.4\\cellcolorgreen\!1013\.4\\cellcolorgreen\!1064\.4\\cellcolorgreen\!1011\.3\\cellcolorgreen\!1059\.4\\cellcolorgreen\!1016\.44\-4\-16Q⇒\\RightarrowL \(40%\)40\.445\.145\.728\.934\.238\.939\.141\.043\.131\.033\.239\.3L⇒\\RightarrowQ \(40%\)50\.430\.256\.525\.045\.929\.149\.628\.354\.926\.844\.929\.7\\cellcolorgreen\!10Ours\(40%\)\\cellcolorgreen\!1055\.5\\cellcolorgreen\!1023\.3\\cellcolorgreen\!1062\.8\\cellcolorgreen\!1017\.5\\cellcolorgreen\!1052\.1\\cellcolorgreen\!1022\.5\\cellcolorgreen\!1055\.1\\cellcolorgreen\!1023\.1\\cellcolorgreen\!1060\.1\\cellcolorgreen\!1019\.8\\cellcolorgreen\!1051\.0\\cellcolorgreen\!1024\.3Q⇒\\RightarrowL \(20%\)45\.038\.749\.926\.843\.737\.645\.136\.447\.833\.041\.937\.5L⇒\\RightarrowQ \(20%\)55\.825\.558\.415\.253\.323\.554\.422\.356\.224\.853\.127\.7\\cellcolorgreen\!10Ours\(20%\)\\cellcolorgreen\!1061\.7\\cellcolorgreen\!1018\.8\\cellcolorgreen\!1064\.5\\cellcolorgreen\!1010\.4\\cellcolorgreen\!1058\.9\\cellcolorgreen\!1017\.8\\cellcolorgreen\!1060\.1\\cellcolorgreen\!1015\.5\\cellcolorgreen\!1062\.3\\cellcolorgreen\!1013\.8\\cellcolorgreen\!1057\.2\\cellcolorgreen\!1018\.54\-4\-4Q⇒\\RightarrowL \(40%\)39\.246\.244\.829\.833\.339\.737\.941\.942\.232\.132\.340\.1L⇒\\RightarrowQ \(40%\)49\.531\.154\.626\.144\.729\.948\.529\.453\.827\.743\.830\.5\\cellcolorgreen\!10Ours\(40%\)\\cellcolorgreen\!1054\.6\\cellcolorgreen\!1024\.5\\cellcolorgreen\!1061\.7\\cellcolorgreen\!1018\.8\\cellcolorgreen\!1051\.2\\cellcolorgreen\!1023\.3\\cellcolorgreen\!1054\.2\\cellcolorgreen\!1024\.3\\cellcolorgreen\!1059\.3\\cellcolorgreen\!1020\.7\\cellcolorgreen\!1050\.1\\cellcolorgreen\!1025\.4Q⇒\\RightarrowL \(20%\)43\.839\.748\.827\.742\.538\.544\.037\.849\.035\.040\.838\.6L⇒\\RightarrowQ \(20%\)54\.526\.657\.417\.352\.124\.652\.823\.555\.125\.752\.129\.0\\cellcolorgreen\!10Ours\(20%\)\\cellcolorgreen\!1060\.3\\cellcolorgreen\!1019\.6\\cellcolorgreen\!1063\.4\\cellcolorgreen\!1011\.5\\cellcolorgreen\!1057\.6\\cellcolorgreen\!1018\.8\\cellcolorgreen\!1059\.0\\cellcolorgreen\!1016\.8\\cellcolorgreen\!1061\.4\\cellcolorgreen\!1015\.0\\cellcolorgreen\!1056\.1\\cellcolorgreen\!1019\.8

Table 3:Results of the diagonal adhesive method\. The compression ratios of the low\-rank decomposition are 20% and 40%, respectively\.![Refer to caption](https://arxiv.org/html/2607.20434v1/fig/neus32_beautified.jpg)\(a\)The result on LLaMA\-2\-7B model\.
![Refer to caption](https://arxiv.org/html/2607.20434v1/fig/neus80_beautified.jpg)\(b\)The result on LLaMA\-2\-70B model\.

Fig\. 2:Compression errors \(L2\) and outliers \(Anomaly Values\) across different layers\. The models utilized include LLaMA\-2\-7B \(32 layers\) and LLaMA\-2\-70B \(80 layers\)\.##### Ablation Experiment\.

In Figure[2](https://arxiv.org/html/2607.20434#S5.F2), the experimental results indicate that the L2 error on LLaMA\-2\-7B and LLaMA\-2\-70B is similar\. Compared to Q⇒\\RightarrowL, L⇒\\RightarrowQ exhibits lower L2 error\. In addition, LLaMA\-2\-70B exhibits smallerOutlieron per\-layer and reduced accumulated compression error compared to LLaMA\-2\-7B\. This also supports the rationale behind our DAM method, which addresses outlier mitigation\.

### 5\.3Verification of the Diagonal Adhesive Method

This section serves to verify the effectiveness of the Diagonal Adhesive Method\. As presented in the Table[3](https://arxiv.org/html/2607.20434#S5.T3), our method demonstrably enhances the performance of the L⇒\\RightarrowQ compression approach\. Compared to L⇒\\RightarrowQ, DAM narrows the performance gap of the LLaMA3\-8B model by 42\.6% under the 4\-4\-4 \(20%\) setting\. In addition, compared to Q⇒\\RightarrowL, DAM improves the performance of the LLaMA3\-8B model by 39\.28% in the 4\-4\-4 \(40%\) setting\. Appendix[H](https://arxiv.org/html/2607.20434#A8)offers more discussion on different compression orders\.

## 6Conclusion

In this paper, we provide theoretical and practical guidance for future model compression methods\. Firstly, we conduct theoretical analysis from the perspectives of tensors and dot products, demonstrating that quantization and low\-rank decomposition are non\-orthogonal and will introduce additional errors\. Additionally, we find that the order in which these two methods are applied significantly affects model performance, and we derive the theoretically optimal sequence—applying low\-rank decomposition before quantization\. Finally, we propose a learnable Diagonal Adhesive Method \(DAM\), which will significantly reduce the additional losses caused by quantization and low\-rank decomposition\. Extensive experiments demonstrate that while maintaining low cost and high speed, DAM significantly improves performance and breaks through the existing compression bottleneck\.

## Limitations

Firstly, due to limitations in computing resources, we did not conduct relevant experiments on larger language models\. Secondly, due to limited experimental resources, there is a lack of experiments conducted on different types of GPUs to verify the widespread practicality of the verification method\.

## Acknowledgments

This work was supported by Beijing Natural Science Foundation \(L243006\), the National Natural Science Foundation of China \(No\. U24A20335\), the independent research project of the Key Laboratory of Cognition and Decision Intelligence for Complex Systems\.

## References

- S\. Ashkboos, A\. Mohtashami, M\. Croci, B\. Li, P\. Cameron, M\. Jaggi, D\. Alistarh, T\. Hoefler, and J\. Hensman \(2025\)Quarot: outlier\-free 4\-bit inference in rotated llms\.Advances in Neural Information Processing Systems37,pp\. 100213–100240\.Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p2.1),[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px1.p1.1)\.
- Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.
- M\. Boratko, H\. Padigela, D\. Mikkilineni, P\. Yuvraj, R\. Das, A\. McCallum, M\. Chang, A\. Fokoue\-Nkoutche, P\. Kapanipathi, N\. Mattei,et al\.\(2018\)A systematic classification of knowledge, reasoning, and context within the arc dataset\.arXiv preprint arXiv:1806\.00358\.Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.
- J\. Chee, Y\. Cai, V\. Kuleshov, and C\. M\. De Sa \(2023\)Quip: 2\-bit quantization of large language models with guarantees\.Advances in Neural Information Processing Systems36,pp\. 4396–4429\.Cited by:[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)Boolq: exploring the surprising difficulty of natural yes/no questions\.arXiv preprint arXiv:1905\.10044\.Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.
- N\. Ding, Y\. Qin, G\. Yang, F\. Wei, Z\. Yang, Y\. Su, S\. Hu, Y\. Chen, C\. Chan, W\. Chen,et al\.\(2023\)Parameter\-efficient fine\-tuning of large\-scale pre\-trained language models\.Nature Machine Intelligence5\(3\),pp\. 220–235\.Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p1.1)\.
- E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh \(2022\)Gptq: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px2.p1.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)A framework for few\-shot language model evaluation\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.
- G\. H\. Golub, A\. Hoffman, and G\. W\. Stewart \(1987\)A generalization of the eckart\-young\-mirsky matrix approximation theorem\.Linear Algebra and its applications88,pp\. 317–327\.Cited by:[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Gupta, D\. Mondal, A\. Sheshadri, W\. Zhao, X\. Li, S\. Wiegreffe, and N\. Tandon \(2023\)Editing common sense in transformers\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 8214–8232\.Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p1.1)\.
- \[11\]Y\. Hsu, T\. Hua, S\. Chang, Q\. Lou, Y\. Shen, and H\. JinLanguage model compression with weighted low\-rank factorization\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Hu, Y\. Cheng, D\. Yang, Z\. Xu, Z\. Yuan, J\. Yu, C\. Xu, Z\. Jiang, and S\. Zhou \(2025\)OstQuant: refining large language model quantization with orthogonal and scaling transformations for better distribution fitting\.arXiv preprint arXiv:2501\.13987\.Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p3.1)\.
- X\. Hu, Y\. Cheng, D\. Yang, Z\. Yuan, J\. Yu, C\. Xu, and S\. Zhou \(2024\)I\-llm: efficient integer\-only inference for fully\-quantized low\-bit large language models\.arXiv preprint arXiv:2405\.17849\.Cited by:[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Huang, Y\. Chen, S\. Wu, J\. Zhao, Y\. Xie, and W\. Sun \(2021\)Named entity recognition via noise aware training mechanism with data filter\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,pp\. 4791–4803\.Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p1.1)\.
- X\. Huang, H\. Yang, Y\. Chen, J\. Zhao, K\. Liu, W\. Sun, and Z\. Zhao \(2022\)Document\-level relation extraction via pair\-aware and entity\-enhanced representation learning\.InProceedings of the 29th International Conference on Computational Linguistics,pp\. 2418–2428\.Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p1.1)\.
- J\. D\. Hwang, C\. Bhagavatula, R\. Le Bras, J\. Da, K\. Sakaguchi, A\. Bosselut, and Y\. Choi \(2021\)On symbolic and neural commonsense knowledge graphs\.Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p2.1)\.
- C\. Lee, J\. Jin, T\. Kim, H\. Kim, and E\. Park \(2023\)Owq: lessons learned from activation outliers for weight quantization in large language models\.arXiv preprint arXiv:2306\.022722\.Cited by:[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px1.p1.1)\.
- O\. Levy, M\. Seo, E\. Choi, and L\. Zettlemoyer \(2017\)Zero\-shot relation extraction via reading comprehension\.arXiv preprint arXiv:1706\.04115\.Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p2.1)\.
- J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han \(2024\)Awq: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of Machine Learning and Systems6,pp\. 87–100\.Cited by:[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Liu, C\. Zhao, I\. Fedorov, B\. Soran, D\. Choudhary, R\. Krishnamoorthi, V\. Chandra, Y\. Tian, and T\. Blankevoort \(2024\)Spinquant: llm quantization with learned rotations\.arXiv preprint arXiv:2405\.16406\.Cited by:[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in gpt\.Advances in Neural Information Processing Systems35,pp\. 17359–17372\.Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p1.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.arXiv preprint arXiv:1609\.07843\.Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.
- T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal \(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.arXiv preprint arXiv:1809\.02789\.Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.
- I\. Porada, K\. Suleman, A\. Trischler, and J\. C\. K\. Cheung \(2021\)Modeling event plausibility with consistent conceptual abstraction\.arXiv preprint arXiv:2104\.10247\.Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p2.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2021\)Winogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.
- M\. Sap, H\. Rashkin, D\. Chen, R\. LeBras, and Y\. Choi \(2019\)Socialiqa: commonsense reasoning about social interactions\.arXiv preprint arXiv:1904\.09728\.Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.
- W\. Shao, M\. Chen, Z\. Zhang, P\. Xu, L\. Zhao, Z\. Li, K\. Zhang, P\. Gao, Y\. Qiao, and P\. Luo \(2023\)Omniquant: omnidirectionally calibrated quantization for large language models\.arXiv preprint arXiv:2308\.13137\.Cited by:[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Sun, R\. Liu, H\. Bai, H\. Bao, K\. Zhao, Y\. Li, J\. Hu, X\. Yu, L\. Hou, C\. Yuan, X\. Jiang, W\. Liu, and J\. Yao \(2025\)FlatQuant: flatness matters for llm quantization\.External Links:2410\.09426,[Link](https://arxiv.org/abs/2410.09426)Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p3.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023a\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023b\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.
- A\. Tseng, J\. Chee, Q\. Sun, V\. Kuleshov, and C\. De Sa \(2024\)Quip\#: even better llm quantization with hadamard incoherence and lattice codebooks\.arXiv preprint arXiv:2402\.04396\.Cited by:[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Wang, Y\. Zheng, Z\. Wan, and M\. Zhang \(2024\)Svd\-llm: truncation\-aware singular value decomposition for large language model compression\.arXiv preprint arXiv:2403\.07378\.Cited by:[Appendix B](https://arxiv.org/html/2607.20434#A2.p1.7),[§1](https://arxiv.org/html/2607.20434#S1.p3.1),[§1](https://arxiv.org/html/2607.20434#S1.p4.1),[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px2.p1.1)\.
- G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han \(2023\)Smoothquant: accurate and efficient post\-training quantization for large language models\.InInternational Conference on Machine Learning,pp\. 38087–38099\.Cited by:[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Yuan, Y\. Shang, Y\. Song, Q\. Wu, Y\. Yan, and G\. Sun \(2023\)Asvd: activation\-aware singular value decomposition for compressing large language models\.arXiv preprint arXiv:2312\.05821\.Cited by:[§1](https://arxiv.org/html/2607.20434#S1.p2.1),[§1](https://arxiv.org/html/2607.20434#S1.p3.1),[§2](https://arxiv.org/html/2607.20434#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)Hellaswag: can a machine really finish your sentence?\.arXiv preprint arXiv:1905\.07830\.Cited by:[§5](https://arxiv.org/html/2607.20434#S5.SS0.SSS0.Px1.p1.1)\.

## Appendix AProof Process of Theorem 3\.2

To better understand the maximum error bound of Eq\.[3](https://arxiv.org/html/2607.20434#S3.E3), it is necessary to combine the definition of quantization and dequantization, the mathematical expression of error, and the extreme value analysis\. The detailed derivation process is as follows:

### Step 1: Definition of Quantization Error

The quantization errorE​\(W​x\)E\(Wx\)represents the output difference when the original weightWWand the dequantized weightD​\(Q​\(W\)\)D\(Q\(W\)\)are multiplied by the inputxx, that is:

E​\(W​x\)=W​x−D​\(Q​\(W\)\)​x\\displaystyle E\(Wx\)=Wx\-D\(Q\(W\)\)x\(24\)
Among them,D​\(Q​\(W\)\)D\(Q\(W\)\)is the dequantized weight\.

### Step 2: Mathematical Expression of Dequantization

According to Eq\.[3](https://arxiv.org/html/2607.20434#S3.E3), the expression of the dequantization operationD​\(Q​\(W\)\)D\(Q\(W\)\)is:

D​\(Q​\(W\)\)=Q​\(W\)⋅max⁡\(\|W\|\)2b−1\\displaystyle D\(Q\(W\)\)=Q\(W\)\\cdot\\frac\{\\max\(\|W\|\)\}\{2^\{b\}\-1\}\(25\)
Among them:

- •Q​\(W\)Q\(W\)is the quantized weight,
- •max⁡\(\|W\|\)\\max\(\|W\|\)is the maximum absolute value of the elements in the weight block,
- •bbis the quantization bit width\.

### Step 3: Expansion of Output Error

Substitute the dequantization expression into the error definition, and expand to get:

E​\(W​x\)=W​x−\(Q​\(W\)⋅max⁡\(\|W\|\)2b−1\)​x\\displaystyle E\(Wx\)=Wx\-\\left\(Q\(W\)\\cdot\\frac\{\\max\(\|W\|\)\}\{2^\{b\}\-1\}\\right\)x\(26\)
Extract the common factorxx, which can be simplified as:

E​\(W​x\)=\(W−Q​\(W\)⋅max⁡\(\|W\|\)2b−1\)​x\\displaystyle E\(Wx\)=\\left\(W\-Q\(W\)\\cdot\\frac\{\\max\(\|W\|\)\}\{2^\{b\}\-1\}\\right\)x\(27\)

### Step 4: Upper Bound Analysis of Quantization Error

During the quantization process, the quantization error of each parameter satisfies:

W−Q​\(W\)⋅max⁡\(\|W\|\)2b−1≤max⁡\(\|W\|\)2⋅\(2b−1\)\\displaystyle W\-Q\(W\)\\cdot\\frac\{\\max\(\|W\|\)\}\{2^\{b\}\-1\}\\leq\\frac\{\\max\(\|W\|\)\}\{2\\cdot\(2^\{b\}\-1\)\}\(28\)
The derivation of this inequality is based on the property of the quantization step size: the quantization step size ismax⁡\(\|W\|\)2b−1\\frac\{\\max\(\|W\|\)\}\{2^\{b\}\-1\}, and the absolute value of the rounding error does not exceed half of the step size \(i\.e\.,max⁡\(\|W\|\)2⋅\(2b−1\)\\frac\{\\max\(\|W\|\)\}\{2\\cdot\(2^\{b\}\-1\)\}\)\.

### Step 5: Boundary Derivation of Output Error

Substitute the upper bound of the quantization error into the output error expression, and get:

\|E​\(W​x\)\|≤max⁡\(\|W\|\)2⋅\(2b−1\)⋅x\\displaystyle\|E\(Wx\)\|\\leq\\frac\{\\max\(\|W\|\)\}\{2\\cdot\(2^\{b\}\-1\)\}\\cdot x\(29\)

### Final Conclusion

Through the above steps, the theoretical upper bound of the quantization error can be derived as:

\|E​\(W​x\)\|≤max⁡\(\|W\|\)2⋅\(2b−1\)⋅x\\displaystyle\|E\(Wx\)\|\\leq\\frac\{\\max\(\|W\|\)\}\{2\\cdot\(2^\{b\}\-1\)\}\\cdot x\(30\)

## Appendix BLow\-Rank Decomposition Error

In SVD decomposition,U=\[u1,u2,u3,…,ur\]U=\[u\_\{1\},u\_\{2\},u\_\{3\},\.\.\.,u\_\{r\}\],Σ=d​i​a​g​\(σ1,σ2,σ3,…,σr\)\\Sigma=diag\(\\sigma\_\{1\},\\sigma\_\{2\},\\sigma\_\{3\},\.\.\.,\\sigma\_\{r\}\), andV=\[v1,v2,v3,…,vr\]V=\[v\_\{1\},v\_\{2\},v\_\{3\},\.\.\.,v\_\{r\}\]\. Then, the smallest singular values inΣ\\Sigmaare truncated to obtain the compressed weight matrixW′=U×T​r​u​n​c\.\(Σ\)×VT×S−1W^\{\\prime\}=U\\times Trunc\.\(\\Sigma\)\\times V^\{T\}\\times S^\{\-1\}\. Inspired by SVD\-LLM\(Wanget al\.,[2024](https://arxiv.org/html/2607.20434#bib.bib14)\), The Frobenius norm of matrixWWwith dimensionm×nm\\times ncan be deduced into the square root of the trace of its gram matrix, which is:

‖W‖F≜\(∑j=1n∑i=1m\|wi​j\|2\)12=\[trace​\(WT​W\)\]12\\displaystyle\\\|W\\\|\_\{F\}\\triangleq\\left\(\\sum\_\{j=1\}^\{n\}\\sum\_\{i=1\}^\{m\}\|w\_\{ij\}\|^\{2\}\\right\)^\{\\frac\{1\}\{2\}\}=\[\\text\{trace\}\(W^\{T\}W\)\]^\{\\frac\{1\}\{2\}\}\(31\)
Given an inputXX, we obtain the compression lossLiL\_\{i\}when truncating theit​hi^\{th\}singular value ofS−1​XS^\{\-1\}Xto reduce its rank for compression:

Li\\displaystyle L\_\{i\}=‖\(W−W′\)​X‖F=‖σi​ui​viT​S−1​X‖F\\displaystyle=\\\|\(W\-W^\{\\prime\}\)X\\\|\_\{F\}=\\\|\\sigma\_\{i\}u\_\{i\}v\_\{i\}^\{T\}S^\{\-1\}X\\\|\_\{F\}\(32\)=σi​trace​\(ui​viT​S−1​X​XT​\(S−1\)T​vi​uiT\)12\\displaystyle=\\sigma\_\{i\}\\text\{trace\}\(u\_\{i\}v\_\{i\}^\{T\}S^\{\-1\}XX^\{T\}\(S^\{\-1\}\)^\{T\}v\_\{i\}u\_\{i\}^\{T\}\)^\{\\frac\{1\}\{2\}\}
Since U and V are orthogonal matrices, we have

viT​vi=uiT​ui=I;viT​vj=uiT​uj=0,∀i≠j;\\displaystyle v\_\{i\}^\{T\}v\_\{i\}=u\_\{i\}^\{T\}u\_\{i\}=I;v\_\{i\}^\{T\}v\_\{j\}=u\_\{i\}^\{T\}u\_\{j\}=0,\\forall i\\neq j;\(33\)trace​\(vi​viT\)=trace​\(ui​uiT\)=1\\displaystyle\\text\{trace\}\(v\_\{i\}v\_\{i\}^\{T\}\)=\\text\{trace\}\(u\_\{i\}u\_\{i\}^\{T\}\)=1
we set the whitening matrixSSis the Cholesky decomposition ofX​XTXX^\{T\}, andS​ST=X​XTSS^\{T\}=XX^\{T\}\. We can obtain:

Li\\displaystyle L\_\{i\}=‖σi​ui​viT​S−1​X‖F\\displaystyle=\\\|\\sigma\_\{i\}u\_\{i\}v\_\{i\}^\{T\}S^\{\-1\}X\\\|\_\{F\}\(34\)=σi​trace​\(ui​viT​S−1​X​XT​\(S−1\)T​vi​uiT\)12\\displaystyle=\\sigma\_\{i\}\\text\{trace\}\(u\_\{i\}v\_\{i\}^\{T\}S^\{\-1\}XX^\{T\}\(S^\{\-1\}\)^\{T\}v\_\{i\}u\_\{i\}^\{T\}\)^\{\\frac\{1\}\{2\}\}=σi​trace​\(ui​viT​vi​uiT\)12=σi\\displaystyle=\\sigma\_\{i\}\\text\{trace\}\(u\_\{i\}v\_\{i\}^\{T\}v\_\{i\}u\_\{i\}^\{T\}\)^\{\\frac\{1\}\{2\}\}=\\sigma\_\{i\}
Therefore,LiL\_\{i\}of truncatingσi\\sigma\_\{i\}equals to the singular valueσi\\sigma\_\{i\}itself\.

## Appendix CProof Process of Theorem 3\.7

To simplify the proof process, all formulas omit the inputxx\. We rewriteEl∘qE\_\{l\\circ q\}as

El∘q=‖Q​\(Ur\)​Q​\(Σr\)​Q​\(VrT\)−W‖F\\displaystyle E\_\{l\\circ q\}=\\\|Q\(U\_\{r\}\)Q\(\\Sigma\_\{r\}\)Q\(V\_\{r\}^\{T\}\)\-W\\\|\_\{F\}\(35\)
Insert the middle termUr​Σr​VrTU\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}:

El∘q=\\displaystyle E\_\{l\\circ q\}=∥∥\[Q\(Ur\)Q\(Σr\)Q\(VrT\)−UrΣrVrT\]\+\\displaystyle\\\|\\\|\[Q\(U\_\{r\}\)Q\(\\Sigma\_\{r\}\)Q\(V\_\{r\}^\{T\}\)\-U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\]\+\(36\)\[UrΣrVrT−W\]∥∥F\\displaystyle\[U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\-W\]\\\|\\\|\_\{F\}
For the Frobenius norm, the triangle inequality holds:

‖A\+B‖F≤‖A‖F\+‖B‖F\\displaystyle\|\|A\+B\|\|\_\{F\}\\leq\|\|A\|\|\_\{F\}\+\|\|B\|\|\_\{F\}\(37\)
We obtain:

El∘q≤\\displaystyle E\_\{l\\circ q\}\\leq‖‖Q​\(Ur\)​Q​\(Σr\)​Q​\(VrT\)−Ur​Σr​VrT‖‖F\\displaystyle\\\|\\\|Q\(U\_\{r\}\)Q\(\\Sigma\_\{r\}\)Q\(V\_\{r\}^\{T\}\)\-U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\\\|\\\|\_\{F\}\(38\)\+‖‖Ur​Σr​VrT−W‖‖F\\displaystyle\+\\\|\\\|U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\-W\\\|\\\|\_\{F\}
Then, we have

El∘q≤Eq\+El\\displaystyle E\_\{l\\circ q\}\\leq E\_\{q\}\+E\_\{l\}\(39\)
##### Strict proof\.

To prove the strictness of this inequality, we need to consider the properties of the Frobenius norm:

Definition of Frobenius norm:

‖A‖​F=∑i,j​\|ai​j\|2\\displaystyle\|\|A\|\|F=\\sqrt\{\\sum\{i,j\}\|a\_\{ij\}\|^\{2\}\}\(40\)
Meanwhile, for matrices A and B:

‖A\+B‖F2=‖A‖F2\+‖B‖F2\+2​t​r​\(AT​B\)\\displaystyle\|\|A\+B\|\|\_\{F\}^\{2\}=\|\|A\|\|\_\{F\}^\{2\}\+\|\|B\|\|\_\{F\}^\{2\}\+2tr\(A^\{T\}B\)\(41\)
In our case:

A\\displaystyle A=Q​\(Ur\)​Q​\(Σr\)​Q​\(VrT\)−Ur​Σr​VrT\\displaystyle=Q\(U\_\{r\}\)Q\(\\Sigma\_\{r\}\)Q\(V\_\{r\}^\{T\}\)\-U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\(42\)B\\displaystyle B=Ur​Σr​VrT−W\\displaystyle=U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\-W
Then

El∘q2\\displaystyle E\_\{l\\circ q\}^\{2\}=‖A\+B‖F2\\displaystyle=\\\|A\+B\\\|\_\{F\}^\{2\}\(43\)=‖A‖F2\+‖B‖F2\+2​t​r​\(AT​B\)\\displaystyle=\\\|A\\\|\_\{F\}^\{2\}\+\\\|B\\\|\_\{F\}^\{2\}\+2tr\(A^\{T\}B\)=Eq2\+El2\+2​t​r​\(AT​B\)\\displaystyle=E\_\{q\}^\{2\}\+E\_\{l\}^\{2\}\+2tr\(A^\{T\}B\)
Sincet​r​\(AT​B\)tr\(A^\{T\}B\)may be negative, this ensures the validity of the inequality\.

Considering the upper bound of quantization error:

\|E​\(W\)\|≤m​a​x​\(\|W\|\)/\(2​\(2b−1\)\)\\displaystyle\|E\(W\)\|\\leq max\(\|W\|\)/\(2\(2^\{b\}\-1\)\)\(44\)
and the SVD decomposition error Es is determined by the truncated singular values:

Li=σi\\displaystyle L\_\{i\}=\\sigma\_\{i\}\(45\)
This ensures that\|E​\(W\)\|\|E\(W\)\|andLiL\_\{i\}is bounded, thereby ensuring an upper bound on the overall errorEl∘qE\_\{l\\circ q\}\.

## Appendix DProof Process of Theorem 3\.8

To simplify the proof process, all formulas omit the inputxx\. The upper bound of quantization error:

\|E​\(W\)\|≤m​a​x​\(\|W\|\)/\(2​\(2b−1\)\)\\displaystyle\|E\(W\)\|\\leq max\(\|W\|\)/\(2\(2^\{b\}\-1\)\)\(46\)
The quantization operation Q \(W\) introduces nonlinear errors, which can: \(1\)Changing the Singular Value Distribution of a Matrix\. \(2\)The low\-rank structure of the influence matrix\. Perform SVD decomposition on the quantized matrix Q \(W\):

Li=S​V​Dr​\(Q​\(W\)\)=σi\\displaystyle L\_\{i\}=SVD\_\{r\}\(Q\(W\)\)=\\sigma\_\{i\}\(47\)
### D\.1Proof of Error Amplification Effect

##### Error propagation analysis\.

Firstly, quantization introduces errors:

Q​\(W\)=W\+Eq\\displaystyle Q\(W\)=W\+E\_\{q\}\(48\)
Among them,EqE\_\{q\}is the quantization error matrix\. Then, perform SVD decomposition onQ​\(W\)Q\(W\):

S​V​Dr​\(Q​\(W\)\)=S​V​Dr​\(W\+Eq\)\\displaystyle SVD\_\{r\}\(Q\(W\)\)=SVD\_\{r\}\(W\+E\_\{q\}\)\(49\)
Due to SVD’s sensitivity to disturbances, the presence ofEqE\_\{q\}can lead to:

S​V​Dr​\(W\+Eq\)≠S​V​Dr​\(W\)\+S​V​Dr​\(Eq\)\\displaystyle SVD\_\{r\}\(W\+E\_\{q\}\)\\neq SVD\_\{r\}\(W\)\+SVD\_\{r\}\(E\_\{q\}\)\(50\)

##### Nonlinear amplification effect\.

Considering matrix perturbation theory, for small perturbationsEqE\_\{q\}:

σi​\(W\+Eq\)=σi​\(W\)\+δ​σi\\displaystyle\\sigma\_\{i\}\(W\+E\_\{q\}\)=\\sigma\_\{i\}\(W\)\+\\delta\\sigma\_\{i\}\(51\)
Among them,δ​σi\\delta\\sigma\_\{i\}not only depends on the size ofEqE\_\{q\}, but also on the singular value distribution ofWW\. This leads to:

‖S​V​Dr​\(Q​\(W\)\)−W‖F\>\\displaystyle\\\|SVD\_\{r\}\(Q\(W\)\)\-W\\\|\_\{F\}\>‖S​V​Dr​\(W\)−W‖F\+\\displaystyle\\\|SVD\_\{r\}\(W\)\-W\\\|\_\{F\}\+\(52\)‖Q​\(W\)−W‖F\\displaystyle\\\|Q\(W\)\-W\\\|\_\{F\}

##### Strict proof\.

We assume:

Eq∘l=‖S​V​Dr​\(Q​\(W\)\)−W‖F\\displaystyle E\_\{q\\circ l\}=\\\|SVD\_\{r\}\(Q\(W\)\)\-W\\\|\_\{F\}\(53\)
Insert middle itemQ​\(W\)Q\(W\):

Eq∘l=\\displaystyle E\_\{q\\circ l\}=∥\[SVDr\(Q\(W\)\)−Q\(W\)\]\+\\displaystyle\\\|\[SVD\_\{r\}\(Q\(W\)\)\-Q\(W\)\]\+\(54\)\[Q​\(W\)−W\]∥F\\displaystyle\[Q\(W\)\-W\]\\\|\_\{F\}
The properties of Frobenius norm:

‖A\+B‖F2=‖A‖F2\+‖B‖F2\+2​t​r​\(AT​B\)\\displaystyle\\\|A\+B\\\|\_\{F\}^\{2\}=\\\|A\\\|\_\{F\}^\{2\}\+\\\|B\\\|\_\{F\}^\{2\}\+2tr\(A^\{T\}B\)\(55\)
Applied to our situation:

Eq∘l2=El′2\+Eq′2\+\\displaystyle E\_\{q\\circ l\}^\{2\}=E\_\{l^\{\\prime\}\}^\{2\}\+E\_\{q^\{\\prime\}\}^\{2\}\+\(56\)2​t​r​\(\[S​V​Dr​\(Q​\(W\)\)−Q​\(W\)\]T​\[Q​\(W\)−W\]\)\\displaystyle 2tr\(\[SVD\_\{r\}\(Q\(W\)\)\-Q\(W\)\]^\{T\}\[Q\(W\)\-W\]\)
The key point lies in the cross item:

2​t​r​\(\[S​V​Dr​\(Q​\(W\)\)−Q​\(W\)\]T​\[Q​\(W\)−W\]\)\\displaystyle 2tr\(\[SVD\_\{r\}\(Q\(W\)\)\-Q\(W\)\]^\{T\}\[Q\(W\)\-W\]\)\(57\)
This item is usually positive because: \(1\) The quantization error changes the singular value structure of the matrix\. \(2\) SVD decomposition is performed on the quantized matrix, preserving the main structure distorted by quantization\. \(3\) This structural distortion is related to the direction of the original quantization error\. So we obtain:

Eq∘l2\>\(El′\+Eq′\)2\\displaystyle E\_\{q\\circ l\}^\{2\}\>\(E\_\{l^\{\\prime\}\}\+E\_\{q^\{\\prime\}\}\)^\{2\}\(58\)
This means:

Eq∘l\>El′\+Eq′\\displaystyle E\_\{q\\circ l\}\>E\_\{l^\{\\prime\}\}\+E\_\{q^\{\\prime\}\}\(59\)

##### Numerical stability analysis\.

The error amplification of SVD after quantization can also be analyzed from the perspective of numerical stability: \(1\) The quantization operation Q \(·\) will introduce rounding errors\. \(2\) These rounding errors will affect the condition numbers of the matrix\. \(3\) The variation of the condition number will affect the accuracy of SVD decomposition\. \(4\) The cumulative effect that ultimately leads to errors is greater than a simple superposition\.

## Appendix EProof Process of Theorem 3\.11

To prove the optimal quantization order at the dot product level, we need to analyze the error sources, structures, and propagation mechanisms of both orders, and compare their error magnitudes through rigorous mathematical derivation\.

### E\.1Core Definitions and Notations

Let the original weight matrix bew∈ℝn×mw\\in\\mathbb\{R\}^\{n\\times m\}and the activation value bex∈ℝnx\\in\\mathbb\{R\}^\{n\}\. The original Dot\-product is:

⟨x,w⟩=xT​w\\displaystyle\\langle x,w\\rangle=x^\{T\}w\(60\)
#### E\.1\.1Low\-Rank Decomposition:

Any matrixwwcan be decomposed into a product of low\-rank matricesw≈A​Bw\\approx AB, whereA∈ℝn×kA\\in\\mathbb\{R\}^\{n\\times k\}andB∈ℝk×mB\\in\\mathbb\{R\}^\{k\\times m\}withk≪min⁡\(n,m\)k\\ll\\min\(n,m\)\(low\-rank dimension\)\. The decomposition error isr=w−A​Br=w\-AB\(usually small, as low\-rank decomposition is an optimal approximation\)\.

#### E\.1\.2Quantization:

The quantization functionQ​\(⋅\)Q\(\\cdot\)converts a high\-precision matrix into a low\-precision one, introducing quantization error:

- •For a quantized matrixMM,Q​\(M\)=M\+eMQ\(M\)=M\+e\_\{M\}, whereeMe\_\{M\}is the quantization error \(satisfying\|eM\|≪\|M\|\|e\_\{M\}\|\\ll\|M\|, since quantization error is usually much smaller than the original value\)\.

### E\.2Error Derivation for Both Orders

We need to calculate the error between the Dot\-product under each order and the original value⟨x,w⟩\\langle x,w\\rangle, and compare their magnitudes\.

#### E\.2\.1Low\-Rank Decomposition Followed by Quantization \(Order 1\)

Steps:

1. 1\.Perform low\-rank decomposition onww:w=A​B\+rw=AB\+r\(whererris the decomposition error, negligible, approximatelyw≈A​Bw\\approx AB\)\.
2. 2\.QuantizeAAandBBrespectively:Aq=Q​\(A\)=A\+eAA\_\{q\}=Q\(A\)=A\+e\_\{A\},Bq=Q​\(B\)=B\+eBB\_\{q\}=Q\(B\)=B\+e\_\{B\}\(whereeAe\_\{A\}andeBe\_\{B\}are quantization errors ofAAandBB\)\.
3. 3\.The quantized weight matrix is: wq​1=Aq​Bq=\(A\+eA\)​\(B\+eB\)\\displaystyle w\_\{q1\}=A\_\{q\}B\_\{q\}=\(A\+e\_\{A\}\)\(B\+e\_\{B\}\)\(61\)

Expanding and ignoring second\-order small errors \(eA​eBe\_\{A\}e\_\{B\}, as the product of quantization errors is even smaller\):

wq​1≈A​B\+A​eB\+eA​B\\displaystyle w\_\{q1\}\\approx AB\+Ae\_\{B\}\+e\_\{A\}B\(62\)
The Dot\-product in this case is:

⟨x,wq​1⟩≈⟨x,A​B⟩\+⟨x,A​eB⟩\+⟨x,eA​B⟩\\displaystyle\\langle x,w\_\{q1\}\\rangle\\approx\\langle x,AB\\rangle\+\\langle x,Ae\_\{B\}\\rangle\+\\langle x,e\_\{A\}B\\rangle\(63\)
Sincew≈A​Bw\\approx AB, the original Dot\-product⟨x,w⟩≈⟨x,A​B⟩\\langle x,w\\rangle\\approx\\langle x,AB\\rangle\. Thus, the errorE1E\_\{1\}for Order 1 is:

E1≈⟨x,A​eB⟩\+⟨x,eA​B⟩\\displaystyle E\_\{1\}\\approx\\langle x,Ae\_\{B\}\\rangle\+\\langle x,e\_\{A\}B\\rangle\(64\)

#### E\.2\.2Quantization Followed by Low\-Rank Decomposition \(Order 2\)

Steps:

1. 1\.Directly quantizeww:wq=Q​\(w\)=w\+eww\_\{q\}=Q\(w\)=w\+e\_\{w\}\(whereewe\_\{w\}is the quantization error ofww\)\.
2. 2\.Perform low\-rank decomposition onwqw\_\{q\}:wq=A′​B′\+r′w\_\{q\}=A^\{\\prime\}B^\{\\prime\}\+r^\{\\prime\}\(whereA′∈ℝn×kA^\{\\prime\}\\in\\mathbb\{R\}^\{n\\times k\},B′∈ℝk×mB^\{\\prime\}\\in\\mathbb\{R\}^\{k\\times m\}, andr′r^\{\\prime\}is the decomposition error\)\.

The low\-rank approximation of the quantized weight isA′​B′=wq−r′=w\+ew−r′A^\{\\prime\}B^\{\\prime\}=w\_\{q\}\-r^\{\\prime\}=w\+e\_\{w\}\-r^\{\\prime\}\. The Dot\-product is:

⟨x,A′​B′⟩=⟨x,w⟩\+⟨x,ew⟩−⟨x,r′⟩\\displaystyle\\langle x,A^\{\\prime\}B^\{\\prime\}\\rangle=\\langle x,w\\rangle\+\\langle x,e\_\{w\}\\rangle\-\\langle x,r^\{\\prime\}\\rangle\(65\)
Thus, the errorE2E\_\{2\}for Order 2 is:

E2=⟨x,ew⟩−⟨x,r′⟩\\displaystyle E\_\{2\}=\\langle x,e\_\{w\}\\rangle\-\\langle x,r^\{\\prime\}\\rangle\(66\)

### E\.3Error Comparison and Proof

We need to prove\|E1\|<\|E2\|\|E\_\{1\}\|<\|E\_\{2\}\|\(smaller error norm\), focusing on the scale of error sources and the ability of low\-rank decomposition to suppress errors\.

#### E\.3\.1Difference in Quantization Error Scale

The total energy \(squared norm\) of quantization error is positively correlated with the number of elements in the quantized matrix \(assuming the variance of quantization error for each element is the same\):

- •In Order 1, the quantized objects areAAandBB, with a total number of elementsk​\(n\+m\)k\(n\+m\)\(sinceA∈n×kA\\in n\\times kandB∈k×mB\\in k\\times m\)\.
- •In Order 2, the quantized object isww, with a total number of elementsn​mnm\(sincew∈n×mw\\in n\\times m\)\.

Sincek≪min⁡\(n,m\)k\\ll\\min\(n,m\), it is obvious that:

k​\(n\+m\)≪n​m\\displaystyle k\(n\+m\)\\ll nm\(67\)
Therefore, the total energy of quantization error in Order 1 is much smaller than that in Order 2:

\|eA\|2\+\|eB\|2≪\|ew\|2\\displaystyle\|e\_\{A\}\|^\{2\}\+\|e\_\{B\}\|^\{2\}\\ll\|e\_\{w\}\|^\{2\}\(68\)

#### E\.3\.2Structural Difference in Error Propagation

- •ErrorE1E\_\{1\}in Order 1: The error terms⟨x,A​eB⟩\\langle x,Ae\_\{B\}\\rangleand⟨x,eA​B⟩\\langle x,e\_\{A\}B\\rangleare products of low\-rank matrices and quantization errors, constrained by the low\-rank dimensionkk\. For example: \|A​eB\|≤\|A\|⋅\|eB\|,\|eA​B\|≤\|eA\|⋅\|B\|\\displaystyle\|Ae\_\{B\}\|\\leq\|A\|\\cdot\|e\_\{B\}\|,\|e\_\{A\}B\|\\leq\|e\_\{A\}\|\\cdot\|B\|\(69\) SinceAAandBBare results of low\-rank decomposition, their norms\|A\|\|A\|and\|B\|\|B\|are not excessively large, soE1E\_\{1\}is “constrained" by the low\-rank structure\.
- •ErrorE2E\_\{2\}in Order 2: The core of the error is⟨x,ew⟩\\langle x,e\_\{w\}\\rangle, whereewe\_\{w\}is a high\-rank matrix \(sincewwitself is high\-rank, and the quantization error retains the high\-rank property\)\. The low\-rank decomposition errorr′r^\{\\prime\}cannot offset the high\-rank components ofewe\_\{w\}\(low\-rank matrices cannot approximate high\-rank errors\)\. Thus,⟨x,ew⟩\\langle x,e\_\{w\}\\rangledominates the error, and due to the large\|ew\|\|e\_\{w\}\|\(see Eq\.[68](https://arxiv.org/html/2607.20434#A5.E68)\),E2E\_\{2\}is significantly larger\.

#### E\.3\.3Rigorous Inequality Derivation

Combining the above analysis and using the Cauchy\-Schwarz inequality:

- •ForE1E\_\{1\}: \|E1\|\\displaystyle\|E\_\{1\}\|≤\|x\|⋅\(\|A​eB\|\+\|eA​B\|\)\\displaystyle\\leq\|x\|\\cdot\(\|Ae\_\{B\}\|\+\|e\_\{A\}B\|\)\(70\)≤\|x\|⋅\(\|A\|​\|eB\|\+\|eA\|​\|B\|\)\\displaystyle\\leq\|x\|\\cdot\(\|A\|\|e\_\{B\}\|\+\|e\_\{A\}\|\|B\|\)Since\|eA\|\|e\_\{A\}\|and\|eB\|\|e\_\{B\}\|are small \(Eq\.[68](https://arxiv.org/html/2607.20434#A5.E68)\),\|E1\|\|E\_\{1\}\|is small\.
- •ForE2E\_\{2\}: \|E2\|≥\|\|⟨x,ew⟩\|−\|⟨x,r′⟩\|\|≥\|x\|⋅\(\|ew\|−\|r′\|\)\\displaystyle\|E\_\{2\}\|\\geq\\left\|\|\\langle x,e\_\{w\}\\rangle\|\-\|\\langle x,r^\{\\prime\}\\rangle\|\\right\|\\geq\|x\|\\cdot\(\|e\_\{w\}\|\-\|r^\{\\prime\}\|\)\(71\) Since\|ew\|≫\|r′\|\|e\_\{w\}\|\\gg\|r^\{\\prime\}\|\(high\-rank errors cannot be eliminated by low\-rank decomposition\),\|E2\|≈\|x\|⋅\|ew\|\|E\_\{2\}\|\\approx\|x\|\\cdot\|e\_\{w\}\|, which is much larger than\|E1\|\|E\_\{1\}\|\.

At the Dot\-product level, the errorE1E\_\{1\}of low\-rank decomposition followed by quantization is significantly smaller than the errorE2E\_\{2\}of quantization followed by low\-rank decomposition because the quantized objects are smaller in scale \(low\-rank matrices\) and the error is constrained by the low\-rank structure\. That is:

\|E1\|<\|E2\|\\displaystyle\\boxed\{\|E\_\{1\}\|<\|E\_\{2\}\|\}\(72\)

## Appendix FSpeedup Experiments

![Refer to caption](https://arxiv.org/html/2607.20434v1/x1.png)

![Refer to caption](https://arxiv.org/html/2607.20434v1/x2.png)

Fig\. 3:Prefill and decoding speedup across different models\. We decode 256 tokens after the prefill on a sequence length of 2048\. The quantization setting is w4a4kv4\. The left figure represents the Prefilling stage, and the right figure represents the Decoding stage\.We conducted prefilling and decoding tests across several models\. Prefilling acceleration improved by 4\.8x while ratio is 50%, and decoding acceleration improved by 3\.2x while ratio is 50%\.

## Appendix GSupplementary experimental results

Our work mainly theoretically proves the non\-orthogonality between quantization and low\-rank decomposition and provides the optimal compression order\. Our compression process consists of two steps: first quantization, then compression\. To verify our theoretical analysis, the traditional GPTQ method was used for quantization in the paper, but this quantization method has large quantization errors\. Since our method is compatible with existing quantization methods, we replaced the quantization method with the popular QuaRot method, and the experimental results are as follows:

### G\.1Experimental results of other models

We conducted experiments on the Qwen2\-7B and Mistral\-7B models with the same setup as those in the paper, and the results in W4\-A4\-KV4 are as follows:

Table 4:Experimental results on Qwen2\-7B and Mistral\-7B \(W4\-A4\-KV4 setup\)The experimental results show that our method achieves significant compression performance on the Qwen model\.

Furthermore, we replaced the traditional quantization method with QuaRot and conducted similar experiments, with the results as follows:

Table 5:Experimental results with QuaRot quantizationThe experimental results indicate that our method is not only compatible with other PTQ methods but also achieves superior performance\.

## Appendix HDiscussion on different compression orders

On the premise of achieving a compressed model with a small compression error, in this paper, we prove the superiority of performing low\-rank decomposition first and then quantization\. However, the compression order of performing quantization first and then low\-rank decomposition is still meaningful\.

The following is a detailed analysis of its advantages from both technical principles and practical effects:

### I\. Quantization Provides a “Simplified Input" for Low\-Rank Decomposition, Reducing Decomposition Difficulty and Retaining Redundancy

The core of quantization is to compress the model by reducing the numerical precision of parameters \(e\.g\., from 32\-bit floating\-point→\\rightarrow16\-bit floating\-point→\\rightarrow8\-bit integer\)\. Essentially, iteliminates redundant precision informationin parameters \(i\.e\., subtle numerical differences that have minimal impact on model performance\)\. This preprocessing significantly benefits subsequent low\-rank decomposition:

##### 1\. Reducing “Noise Interference" in Parameters and Enhancing Core Information Capture in Low\-Rank Decomposition

Low\-rank decomposition \(e\.g\., SVD singular value decomposition, matrix factorization\) aims to decompose high\-rank matrices \(such as convolution kernels or fully connected layer weights\) into products of low\-rank matrices \(e\.g\.,W=A×BW=A\\times B, where the ranks ofAAandBBare much smaller than that ofWW\), preserving the “principal components" critical to model performance\.

However, original high\-precision parameters may contain numerous subtle numerical fluctuations \(which can be regarded as “noise"\)\. These fluctuations are not core to the model’s decision\-making but interfere with the identification of principal components in low\-rank decomposition \(e\.g\., singular value decomposition may misclassify noise as components needing retention\)\.

Quantization first filters out this “noise" from redundant precision, making the numerical distribution of parameters more regular \(e\.g\., quantized parameters concentrate on limited discrete values\)\. This allows low\-rank decomposition to focus more on the core numerical patterns that truly affect model outputs, thereby retaining key information more efficiently and reducing information loss during decomposition\.

##### 2\. Reducing the Dynamic Range of Parameters and Improving the Stability of Low\-Rank Decomposition

The dynamic range of original high\-precision parameters can be large \(e\.g\., floating\-point parameters may span multiple orders of magnitude\)\. When processing data with a large dynamic range, the numerical stability of low\-rank decomposition may decline \(e\.g\., when singular values differ significantly in magnitude, components corresponding to small singular values are easily ignored, leading to loss of useful information\)\.

Quantization maps parameters to a smaller dynamic range \(e\.g\., the range of 8\-bit integers is typically\[−128,127\]\[\-128,127\]\),narrowing the numerical span of parameters\. This enhances the numerical stability of low\-rank decomposition during calculations \(such as singular value sorting and low\-rank matrix reconstruction\) and reduces decomposition errors caused by an excessively large dynamic range\.

### II\. Reducing Computational Costs of Low\-Rank Decomposition and Improving Compression Efficiency

Low\-rank decomposition has high computational complexity \(e\.g\., the time complexity of SVD decomposition for anM×NM\\times Nmatrix isO​\(M2​N\+M​N2\)O\(M^\{2\}N\+MN^\{2\}\)\), but quantization can significantly reduce resource consumption in this process:

##### Reducing Parameter Storage and Computation to Accelerate Decomposition

After quantization, the bit\-width of parameters decreases \(e\.g\., from 32\-bit to 8\-bit, reducing storage by 75%\)\. During low\-rank decomposition,memory usage and computation time for matrix operations\(such as matrix multiplication and singular value solving\) decrease significantly\.

For example, decomposing a quantized 8\-bit integer weight matrix is several times faster on the same hardware compared to the original 32\-bit floating\-point matrix, with lower memory usage \(especially for large\-scale models like Transformers and ResNets, the effect is more pronounced\)\.

To validate this, we conducted experiments: we used INT8 SpinQuant quantization \+ 20% Low\-rank decomposition \(Q⇒L\\text\{Q\}\\Rightarrow\\text\{L\}\) compression, comparing it against standalone 20% Low\-rank decomposition \(Only\-L\)\.

Qwen2\-7B \(20%\)MMLUHumanEvalGSM8KMATHTimeFP16 \(0%\)70\.351\.279\.944\.2\- \- minOnly\-L48\.029\.747\.919\.2132 minQ⇒L\\text\{Q\}\\Rightarrow\\text\{L\}48\.329\.347\.819\.663minTable 6:Results for Qwen2\-7B \(20% compression\)Mistral\-7B \(20%\)MMLUHumanEvalGSM8KMATHTimeFP16 \(0%\)64\.229\.352\.213\.1\- \- minOnly\-L46\.318\.729\.94\.7136 minQ⇒L\\text\{Q\}\\Rightarrow\\text\{L\}46\.518\.329\.34\.862minTable 7:Results for Mistral\-7B \(20% compression\)The Table[6](https://arxiv.org/html/2607.20434#A8.T6)and Table[7](https://arxiv.org/html/2607.20434#A8.T7)show that compared with standalone low\-rank decomposition, the “quantization first, then low\-rank decomposition” approach achieves comparable performance and outperforms the standalone low\-rank decomposition method in some metrics\. Moreover, it improves the compression speed by2\.2x, which is a significant advantage of this approach\. It is worth noting that previous works often regarded quantization and low\-rank decomposition as orthogonal\.

The above indicates that quantization followed by low\-rank decomposition is also a valuable compression method\. In particular, existing low\-bit quantization methods can retain approximately 98% of model performance, making their gains for low\-rank decomposition increasingly valuable\. Previous work did not explore reasonable compression orders, assuming the two orders are orthogonal\. We theoretically propose the optimal compression order, providing a theoretical foundation for future research in the field of compression\.