LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

# LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
Source: [https://arxiv.org/html/2607.03057](https://arxiv.org/html/2607.03057)
###### Abstract

The rapid growth in the parameter scale of large language models \(LLMs\) has created a strong demand for efficient compression techniques\. As a hardware\-agnostic and highly compatible approach, low\-rank compression has been widely adopted to reduce both memory footprint and computational cost\. However, existing SVD\-based methods are still largely driven by local reconstruction objectives, overlooking two critical limitations: rank budgets are often allocated without explicitly considering layer\-wise loss sensitivity, and local approximation errors can propagate and accumulate through the residual stream, leading to amplified global deviations from the original model\. To address these issues, we proposeLACE\-SVD, aLoss\-Aware SVD framework withCumulativeError correction for LLM compression\. LACE\-SVD first estimates the calibration negative\-log\-likelihood increase induced by candidate layer\-wise compression ratios and solves a budget\-constrained allocation problem to assign rank budgets\. It then refines the compressed model with closed\-form local updates and introduces a propagation\-aware correction for residual\-stream output modules, reducing layer\-output discrepancy as a proxy for cumulative error propagation\. Experimental results demonstrate that at a high compression ratio \(0\.6\), the WikiText\-2 PPL of our method on LLaMA\-7B \(32\.57\) is significantly better than that of Dobi\-SVD \(46\.18\)\.

LACE\-SVD: Loss\-Aware SVD with Cumulative Error Correction for LLM Compression

Zhuowen Liu1Longkun Hao2Shiyu Feng3Xiaowen Chang4Ruiqun Li5Changqun Li6†

## 1Introduction

Large language models \(LLMs\), including GPT FamilyAchiamet al\.\([2023](https://arxiv.org/html/2607.03057#bib.bib17)\); Dettmerset al\.\([2022](https://arxiv.org/html/2607.03057#bib.bib18)\), LLaMA FamilyTouvronet al\.\([2023a](https://arxiv.org/html/2607.03057#bib.bib19)\); Grattafioriet al\.\([2024](https://arxiv.org/html/2607.03057#bib.bib20)\), and Qwen FamilyBaiet al\.\([2023](https://arxiv.org/html/2607.03057#bib.bib22)\); Yanget al\.\([2025](https://arxiv.org/html/2607.03057#bib.bib23)\)have achieved remarkable performance across commonsense reasoning, question answering, document understanding, and few\-shot learning\. However, their strong capabilities come with substantial memory and computational costs, making billion\-parameter models difficult to deploy under practical latency, memory, and energy constraints\. To address this challenge, post\-training compression has become an important research direction, including quantizationFrantaret al\.\([2022](https://arxiv.org/html/2607.03057#bib.bib24)\); Huet al\.\([2025](https://arxiv.org/html/2607.03057#bib.bib30)\), pruningAshkbooset al\.\([2024](https://arxiv.org/html/2607.03057#bib.bib9)\), knowledge distillationHintonet al\.\([2015](https://arxiv.org/html/2607.03057#bib.bib26)\), and low\-rank decompositionGolubet al\.\([1987](https://arxiv.org/html/2607.03057#bib.bib28)\); Yuanet al\.\([2023](https://arxiv.org/html/2607.03057#bib.bib4)\)\. Among these approaches, singular value decomposition \(SVD\) based compression is particularly attractive because it directly factorizes dense weight matrices into low\-rank components, reducing both model size and matrix multiplication cost without requiring large\-scale retraining\.

Recent SVD\-based compression methodsYuanet al\.\([2023](https://arxiv.org/html/2607.03057#bib.bib4)\); Liet al\.\([2025](https://arxiv.org/html/2607.03057#bib.bib34)\);[Wanget al\.](https://arxiv.org/html/2607.03057#bib.bib1)improve over vanilla truncated SVD by using activation statistics or calibration data\. These methods rescale or whiten weight matrices before decomposition so that the low\-rank approximation better matches the activation distribution of each layer\. Although effective, they are still mainly driven by local reconstruction objectives\. That is, they aim to preserve the output of each compressed module or layer under calibration activations, while the global language modeling behavior is only indirectly considered\.

This local view leads to two limitations\. First, different layers exhibit different sensitivity to compression\. A uniform rank ratio may over\-compress loss\-critical layers and under\-compress robust layers, resulting in a suboptimal use of the global parameter budget\. Second, reconstruction errors are not isolated across layers\. Once an early layer is compressed, its output perturbation enters the residual stream and changes the input distribution of subsequent layers\. These perturbations can accumulate through attention and MLP blocks, causing the compressed model to deviate from the original model even when each individual layer has a small local reconstruction error\.

Motivated by these observations, we proposeLACE\-SVD, aLoss\-Aware SVD framework withCumulativeError correction for LLM compression\. LACE\-SVD addresses the above two problems from complementary perspectives\. First, it performs loss\-aware layer\-wise rank allocation by evaluating how candidate layer\-wise compression ratios affect calibration negative log\-likelihood, and then selecting one ratio per layer under a global parameter budget\. This allows the method to allocate more capacity to layers that are more important for language modeling quality\. Second, LACE\-SVD introduces cumulative error correction after SVD decomposition\. We refine the low\-rank factors with closed\-form local updates using full\-precision calibration activations\. Furthermore, for modules that directly write to the residual stream, such as attention output projections and MLP down projections, we introduce a propagation\-aware correction mechanism\. This correction does not directly optimize the language modeling loss; instead, it reduces the discrepancy between the compressed layer output and the original layer output, serving as a practical proxy for mitigating cumulative error propagation\. An acceptance gate keeps the correction only when it improves layer\-level output fidelity on calibration data\. We empirically demonstrate that our method significantly outperforms Dobi\-SVD at high compression ratios, achieving 32\.57 WikiText\-2 PPL on LLaMA\-7B \(0\.6 ratio\) compared to Dobi\-SVD’s 46\.18\. In summary, LACE\-SVD shifts post\-training SVD compression from purely local matrix approximation toward loss\-aware budget allocation and cumulative error correction\. Our contributions are as follows:

- •We identify that existing SVD\-based LLM compression methods are limited by heuristic rank allocation and cumulative error propagation across layers, showing that minimizing local reconstruction error alone is insufficient for preserving end\-to\-end model fidelity\.
- •We propose a loss\-aware layer\-wise rank allocation strategy that uses calibration negative log\-likelihood to select compression ratios under a global parameter budget, allocating more capacity to layers that are more critical to language modeling performance\.
- •We introduce cumulative error correction through propagation\-aware local updates for residual\-stream output modules, reducing layer\-output discrepancy between the compressed and original models without requiring full end\-to\-end fine\-tuning\.

## 2Related Works

Large Language Model Compression\.LLM compression broadly falls into pruning, quantization, knowledge distillation, and low\-rank approximation\. Unstructured and structured pruning\(Ashkbooset al\.,[2024](https://arxiv.org/html/2607.03057#bib.bib9); Zhanget al\.,[2023](https://arxiv.org/html/2607.03057#bib.bib31); Maet al\.,[2023](https://arxiv.org/html/2607.03057#bib.bib32); Dettmerset al\.,[2024](https://arxiv.org/html/2607.03057#bib.bib33)\)often struggle with a trade\-off between hardware efficiency and severe accuracy degradation\. Quantization\(Nagelet al\.,[2021](https://arxiv.org/html/2607.03057#bib.bib29); Frantaret al\.,[2022](https://arxiv.org/html/2607.03057#bib.bib24); Huet al\.,[2025](https://arxiv.org/html/2607.03057#bib.bib30); Linet al\.,[2024](https://arxiv.org/html/2607.03057#bib.bib16); Huet al\.,[2024](https://arxiv.org/html/2607.03057#bib.bib25)\)delivers strong memory savings but loses flexibility and performance under aggressive low\-bit settings, while knowledge distillation\(Hintonet al\.,[2015](https://arxiv.org/html/2607.03057#bib.bib26)\)requires costly retraining\. In contrast, low\-rank approximation via Singular Value Decomposition \(SVD\)\(Yuanet al\.,[2023](https://arxiv.org/html/2607.03057#bib.bib4);[Wanget al\.,](https://arxiv.org/html/2607.03057#bib.bib1)\)offers a post\-training, hardware\-agnostic solution that is naturally orthogonal to other compression paradigms\.

SVD\-based Compression for LLMs\.SVD\(Golubet al\.,[1987](https://arxiv.org/html/2607.03057#bib.bib28)\)reduces matrix sizes by truncating the smallest singular values\. Low\-rank approximation using SVD has been widely studied as an efficient approach for compressing LLMs\. Early works mainly focused on minimizing per\-layer truncation loss\. To incorporate parameter importance, FWSVD\([Hsuet al\.,](https://arxiv.org/html/2607.03057#bib.bib3)\)introduces Fisher information to weigh the importance of parameters, but its complex gradient calculation demands substantial computing and memory resources\. To address the impact of activation distributions, ASVD\(Yuanet al\.,[2023](https://arxiv.org/html/2607.03057#bib.bib4)\)scales the weight matrix with a diagonal matrix to normalize the impact of input channels\. SVD\-LLM\([Wanget al\.,](https://arxiv.org/html/2607.03057#bib.bib1)\)makes further advancements by proposing a data\-whitening strategy on the input matrix to mitigate its impact, achieving a theoretical minimal truncation loss\. Building upon this, SVD\-LLM v2Wanget al\.\([2025b](https://arxiv.org/html/2607.03057#bib.bib2)\)further optimizes the truncation process by dynamically assigning unique, layer\-wise compression ratios based on theoretical loss\. Furthermore, Dobi\-SVDWanget al\.\([2025a](https://arxiv.org/html/2607.03057#bib.bib5)\)tackles the challenges of layer\-wise rank allocation and information loss by introducing a differentiable truncation mechanism for adaptive rank search\. To address broader architectural issues, SAES\-SVDHuet al\.\([2026](https://arxiv.org/html/2607.03057#bib.bib7)\)tackles the problem of cross\-layer error accumulation by introducing a cumulative error\-aware compression objective alongside an adaptive weighting mechanism to dynamically suppress both local and propagated errors\.

![Refer to caption](https://arxiv.org/html/2607.03057v1/x1.png)Figure 1:Overall architecture of LACE\-SVD\.Original LLM layers are first decomposed into initial low\-rank weights using Activation\-Whitened SVD\. A loss\-aware allocation strategy then dynamically selects the optimal rank ratio \(rgr\_\{g\}\) for each layer\. Finally, an error correction pipeline, incorporating Local Updates and Cumulative Error Correction \(CEC\), is applied to compensate for the cumulative errors before assembling the Compressed LLM\.![Refer to caption](https://arxiv.org/html/2607.03057v1/x2.png)Figure 2:Layer\-wise compression ratio allocation for LLaMA\-7B under different targets\.The loss\-aware allocation strategy dynamically assigns compression budgets across 32 layers\. Compared to the uniform baseline \(red dashed line\), the algorithm consistently preserves more parameters in the early layers \(L00–L06, green region\) to maintain foundational lexical and syntactic features, as well as in the final layers \(L27–L31\), which act as a highly sensitive bottleneck crucial for vocabulary alignment\. Conversely, the intermediate layers \(L08–L26, red region\) are aggressively pruned due to greater semantic and rank redundancy\. This distinct U\-shaped “preserve\-ends, prune\-middle” pattern remains robustly consistent across all target compression constraints \(20% to 80%\)\.
## 3Method

### 3\.1Overview

Given a pretrained LLMfθf\_\{\\theta\}with Transformer layers\{Fℓ\}ℓ=1L\\\{F\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}, our goal is to obtain a compressed modelfθ^f\_\{\\hat\{\\theta\}\}under a target global compression ratioρ\\rho, while preserving the behavior of the full\-precision model\. LACE\-SVD follows a post\-training compression setting and does not require end\-to\-end fine\-tuning\.

The method consists of three main stages\. First, we compute activation\-whitened low\-rank decompositions for linear projection matrices using calibration activations\. Second, we perform loss\-aware layer\-wise rank allocation by estimating the calibration loss increase caused by different candidate compression ratios for each layer\. Third, after decomposing the selected layers, we refine the low\-rank factors with closed\-form local updates and further apply cumulative error correction to residual\-stream output modules\. The algorithm description is provided in Appendix[A](https://arxiv.org/html/2607.03057#A1)\.

### 3\.2Activation\-Whitened SVD

For a linear projection with weight matrixW∈ℝdout×dinW\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}and calibration inputsX∈ℝn×dinX\\in\\mathbb\{R\}^\{n\\times d\_\{\\mathrm\{in\}\}\}, vanilla truncated SVD minimizes the reconstruction error ofWWitself\. However, in an LLM, the effect of approximatingWWdepends on the distribution of its input activations\. We therefore use an activation\-whitened decomposition\.

We first estimate the input covariance:

LetCCbe the Cholesky factor ofGG, i\.e\.,

G≈C​C⊤\.G\\approx CC^\{\\top\}\.\(2\)We then decompose the activation\-scaled weight matrix:

W​C=U​Σ​V⊤\.WC=U\\Sigma V^\{\\top\}\.\(3\)Keeping the top\-kksingular components gives the compressed approximation:

W^=Uk​Σk​Vk⊤​C−1\.\\widehat\{W\}=U\_\{k\}\\Sigma\_\{k\}V\_\{k\}^\{\\top\}C^\{\-1\}\.\(4\)
This corresponds to the profiling matrix path in the implementation: calibration activations are accumulated to build the whitening matrix, and the SVD is performed on the scaled weight matrix before mapping the approximation back to the original weight space\.

### 3\.3Loss\-Aware Layer\-Wise Rank Allocation

Uniformly assigning the same compression ratio to all layers ignores the fact that layers have different sensitivity to compression\. LACE\-SVD therefore selects layer\-wise compression ratios using calibration negative log\-likelihood\.

Letℛρ\\mathcal\{R\}\_\{\\rho\}denote the candidate keep\-ratio set associated with the target budgetρ\\rho\. For each layerℓ\\elland candidate ratior∈ℛρr\\in\\mathcal\{R\}\_\{\\rho\}, we temporarily compress only layerℓ\\ellwhile keeping the other layers unchanged\. We then evaluate the calibration loss increase:

Δℓ,r=ℒcalib​\(fθ\(ℓ,r\)\)−ℒcalib​\(fθ\),\\Delta\_\{\\ell,r\}=\\mathcal\{L\}\_\{\\mathrm\{calib\}\}\\left\(f\_\{\\theta\}^\{\(\\ell,r\)\}\\right\)\-\\mathcal\{L\}\_\{\\mathrm\{calib\}\}\\left\(f\_\{\\theta\}\\right\),\(5\)wherefθf\_\{\\theta\}is the full\-precision model andfθ\(ℓ,r\)f\_\{\\theta\}^\{\(\\ell,r\)\}denotes the model whoseℓ\\ell\-th layer is compressed with ratiorr\.

The implementation evaluates candidate ratios by temporarily replacing the weights in a layer with their low\-rank reconstructed dense weights, measuring the calibration loss, and then restoring the original weights\. The resulting per\-layer table contains the tuple

\(r,Cℓ,r,Δℓ,r\),\(r,C\_\{\\ell,r\},\\Delta\_\{\\ell,r\}\),\(6\)whereCℓ,rC\_\{\\ell,r\}is the parameter cost of compressing layerℓ\\ellwith ratiorr\.

Given these tables, LACE\-SVD solves a multiple\-choice budget allocation problem:

min\{rℓ\}ℓ=1L​∑ℓ=1LΔℓ,rℓ,\\min\_\{\\\{r\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}\}\\sum\_\{\\ell=1\}^\{L\}\\Delta\_\{\\ell,r\_\{\\ell\}\},\(7\)subject to

∑ℓ=1LCℓ,rℓ≤ρ​Cfull,rℓ∈ℛρ\.\\sum\_\{\\ell=1\}^\{L\}C\_\{\\ell,r\_\{\\ell\}\}\\leq\\rho C\_\{\\mathrm\{full\}\},\\quad r\_\{\\ell\}\\in\\mathcal\{R\}\_\{\\rho\}\.\(8\)HereCfullC\_\{\\mathrm\{full\}\}is the total number of parameters before compression\. This objective explicitly connects rank allocation with language\-modeling degradation, allowing the method to allocate more rank budget to loss\-sensitive layers\.

In the implementation, this optimization is solved by a discretized dynamic programming procedure\. For efficiency, candidate evaluation can be performed in two stages: a coarse screening stage selects promising ratios for each layer, and a full calibration evaluation stage computes the final loss table\. This is an engineering acceleration and does not change the allocation objective\.

### 3\.4Simultaneous Local Closed\-Form Update

After selecting layer\-wise compression ratios, we replace each target projection with a low\-rank factorization\. For a compressed projection, write

W^=U​V,\\widehat\{W\}=UV,\(9\)whereVVis the input\-side factor andUUis the output\-side factor\. Given calibration inputXXand the corresponding full\-precision output

we define the low\-rank hidden feature

The output\-side factor is then updated by solving a regularized least\-squares problem:

U⋆=arg⁡minU⁡‖Z​U⊤−Y‖F2\+λU​‖U−U0‖F2,U^\{\\star\}=\\arg\\min\_\{U\}\\left\\\|ZU^\{\\top\}\-Y\\right\\\|\_\{F\}^\{2\}\+\\lambda\_\{U\}\\left\\\|U\-U\_\{0\}\\right\\\|\_\{F\}^\{2\},\(12\)whereU0U\_\{0\}is the initial factor obtained from SVD andλU\\lambda\_\{U\}is a ridge regularization coefficient\.

The implementation accumulates the normal equations over calibration batches and solves this update in closed form\. It also supports a weighted variant:

U⋆=arg⁡minU⁡‖D1/2​\(Z​U⊤−Y\)‖F2\+λU​‖U−U0‖F2,\\displaystyle U^\{\\star\}=\\arg\\min\_\{U\}\\left\\\|D^\{1/2\}\\left\(ZU^\{\\top\}\-Y\\right\)\\right\\\|\_\{F\}^\{2\}\+\\lambda\_\{U\}\\left\\\|U\-U\_\{0\}\\right\\\|\_\{F\}^\{2\},

\(13\)
whereDDis a diagonal sample\-weight matrix computed from the selected weighting mode\. In the main configuration, the residual\-based weighting mode is used\.

LACE\-SVD uses a simultaneous local update mode\. Specifically, the calibration statistics for modules in a layer are collected using full\-precision layer activations, and the low\-rank factors are solved after the statistics are accumulated\. This avoids immediately feeding partially compressed intermediate states into later modules during calibration, which can otherwise make the local targets unstable\.

The implementation also optionally evaluates a bi\-side closed\-form correction, where both the output\-side and input\-side factors are considered\. However, the candidate correction is accepted only if it improves the held\-out reconstruction objective by at least a minimum relative gain\. Thus, the method does not assume that the bi\-side update is always beneficial; it selects the safer update according to calibration reconstruction error\.

### 3\.5Cumulative Error Correction

While local reconstruction objectives effectively reduce module\-level errors, they do not fully address cumulative error propagation\. In Transformer architectures, specific projection modules write directly back to the residual stream\. Errors within these modules inevitably alter the hidden states consumed by all subsequent layers, leading to compounded degradation across the network\.

To mitigate this, we define a subset of target modules, denoted as𝒮\\mathcal\{S\}, within each Transformer layer\. LACE\-SVD introduces a propagation\-aware cumulative error correction mechanism applied to the modules in𝒮\\mathcal\{S\}\. For a given layerℓ\\elland a target modules∈𝒮s\\in\\mathcal\{S\}, letMℓ,sfullM\_\{\\ell,s\}^\{\\mathrm\{full\}\}represent the output of the full\-precision module, andMℓ,scmpM\_\{\\ell,s\}^\{\\mathrm\{cmp\}\}denote the output of the compressed module under the same calibration input\. We construct a propagation\-aware target as follows:

Tℓ,spa=Mℓ,scmp\+α​\(Mℓ,sfull−Mℓ,scmp\),T\_\{\\ell,s\}^\{\\mathrm\{pa\}\}=M\_\{\\ell,s\}^\{\\mathrm\{cmp\}\}\+\\alpha\\left\(M\_\{\\ell,s\}^\{\\mathrm\{full\}\}\-M\_\{\\ell,s\}^\{\\mathrm\{cmp\}\}\\right\),\(14\)whereα∈\[0,1\]\\alpha\\in\[0,1\]is a hyperparameter that controls the correction strength\. An ablation study onα\\alphais provided in Appendix[C](https://arxiv.org/html/2607.03057#A3)\.

This target moves the compressed residual\-stream contribution toward the full\-precision contribution\. Importantly, this step does not directly optimize the language modeling loss\. Instead, it uses module and layer output discrepancy as a practical proxy for cumulative error propagation\. After constructingTℓ,spaT\_\{\\ell,s\}^\{\\mathrm\{pa\}\}, we re\-solve the output\-side factor of the selected module using the same closed\-form local update objective, replacingYYwithTℓ,spaT\_\{\\ell,s\}^\{\\mathrm\{pa\}\}\.

To prevent over\-correction, LACE\-SVD applies a layer\-level acceptance gate\. LetHℓH\_\{\\ell\}be the calibration input to layerℓ\\ell,FℓfullF\_\{\\ell\}^\{\\mathrm\{full\}\}be the full\-precision layer,FℓcmpF\_\{\\ell\}^\{\\mathrm\{cmp\}\}be the compressed layer before propagation\-aware correction, andFℓpaF\_\{\\ell\}^\{\\mathrm\{pa\}\}be the corrected layer\. The correction is accepted only if

‖Fℓpa​\(Hℓ\)−Fℓfull​\(Hℓ\)‖F2<‖Fℓcmp​\(Hℓ\)−Fℓfull​\(Hℓ\)‖F2\.\\left\\\|F\_\{\\ell\}^\{\\mathrm\{pa\}\}\(H\_\{\\ell\}\)\-F\_\{\\ell\}^\{\\mathrm\{full\}\}\(H\_\{\\ell\}\)\\right\\\|\_\{F\}^\{2\}<\\left\\\|F\_\{\\ell\}^\{\\mathrm\{cmp\}\}\(H\_\{\\ell\}\)\-F\_\{\\ell\}^\{\\mathrm\{full\}\}\(H\_\{\\ell\}\)\\right\\\|\_\{F\}^\{2\}\.

\(15\)
If this criterion is not satisfied, the propagation\-aware update is rejected and the previous compressed factors are restored\. This gate ensures that cumulative error correction is retained only when it improves layer\-level fidelity\.

## 4Experiments

Models and Datasets\.To comprehensively evaluate the performance of LACE\-SVD across different models and tasks, we conducted systematic experiments on mainstream open\-source LLMs using standard language understanding and generation benchmarks, covering a wide range of parameter budgets\. By default, all results are obtained by applying a single\-pass SVD layer by layer, without any extra task\-specific fine\-tuning after compression\. For the model selection, we chose well\-known architectures, including three different LLM families at different scales \(LLaMA\-7B, 13B, LLaMA2\-7BTouvronet al\.\([2023b](https://arxiv.org/html/2607.03057#bib.bib21)\), OPT\-6\.7BZhanget al\.\([2022](https://arxiv.org/html/2607.03057#bib.bib45)\), Vicuna\-7BChianget al\.\([2023](https://arxiv.org/html/2607.03057#bib.bib46)\)and Mistral\-7BJianget al\.\([2023](https://arxiv.org/html/2607.03057#bib.bib47)\)\. Regarding the evaluation, we mainly focused on natural language reasoning and understanding\. Specifically, to assess the language coherence and modeling capacity of the compressed models, we calculated the perplexity on the WikiText2Merityet al\.\([2016](https://arxiv.org/html/2607.03057#bib.bib36)\)and C4Raffelet al\.\([2020](https://arxiv.org/html/2607.03057#bib.bib37)\)datasets with a sequence length of 2048\. We also measured zero\-shot accuracy on several datasets—including ARC\-ChallengeClarket al\.\([2018](https://arxiv.org/html/2607.03057#bib.bib38)\), ARC\-EasyClarket al\.\([2018](https://arxiv.org/html/2607.03057#bib.bib38)\), HellaSwagZellerset al\.\([2019](https://arxiv.org/html/2607.03057#bib.bib39)\), MathQAAminiet al\.\([2019](https://arxiv.org/html/2607.03057#bib.bib40)\), PIQABisket al\.\([2020](https://arxiv.org/html/2607.03057#bib.bib41)\), and WinoGrandeSakaguchiet al\.\([2019](https://arxiv.org/html/2607.03057#bib.bib42)\)—to examine how well the models generalize across a variety of tasks\.

Baselines\.Under a unified calibration and evaluation protocol, we compared our method against several well\-established SVD\-based low\-rank compression approaches, such as ASVDYuanet al\.\([2023](https://arxiv.org/html/2607.03057#bib.bib4)\), SVD\-LLM[Wanget al\.](https://arxiv.org/html/2607.03057#bib.bib1), FWSVD[Hsuet al\.](https://arxiv.org/html/2607.03057#bib.bib3), Dobi\-SVDWanget al\.\([2025a](https://arxiv.org/html/2607.03057#bib.bib5)\)\. In addition to SVD methods, we compared our work with other types of LLM compression approaches\. Specifically, these include three state\-of\-the\-art pruning\-based methods: LLM\-PrunerMaet al\.\([2023](https://arxiv.org/html/2607.03057#bib.bib32)\), SliceGPTAshkbooset al\.\([2024](https://arxiv.org/html/2607.03057#bib.bib9)\), and BlockPrunerZhonget al\.\([2025](https://arxiv.org/html/2607.03057#bib.bib44)\)\. Implementation Details is provided in Appendix[B](https://arxiv.org/html/2607.03057#A2)\.

Main Results\.We conducted a comprehensive evaluation of our proposed method \(LACE\-SVD\) for low\-rank compression of LLMs and compared it with several strong SVD\-based baselines across different compression ratios\. Table 1 summarizes the perplexity and zero\-shot evaluation results on the representative LLaMA\-7B\. Overall, LACE\-SVD consistently achieves superior performance\. At the highly challenging 0\.6 compression ratio, LACE\-SVD exhibits remarkable robustness\. While existing advanced methods suffer severe degradation—with SVD\-LLM and Dobi\-SVD yielding WikiText\-2 perplexities of 53\.74 and 46\.18, respectively—our method maintains a significantly lower perplexity of 32\.57\. Beyond perplexity, LACE\-SVD effectively minimizes the degradation in downstream capabilities across varying ratios\. For instance, at the 0\.4 compression ratio, LACE\-SVD restricts the average zero\-shot accuracy drop to 23\.1%, which is substantially lower than those of Dobi\-SVD \(26\.9%\) and SVD\-LLM \(28\.9%\)\.

Table 1:Perplexity and zero\-shot evaluation of LLaMA\-7B across seven benchmark datasets under varying compression ratios\. The table compares LACE\-SVD with competing SVD\-based methods \(ASVD∗, FWSVD, SVD\-LLM , Dobi\-SVD\)\. Methods with fine\-tuning are marked by†\.RatioMethodWiki2↓\\downarrowC4↓\\downarrowOpenb\.↑\\uparrowARC\_e↑\\uparrowARC\_c↑\\uparrowWinoG\.↑\\uparrowHellaS\.↑\\uparrowPIQA↑\\uparrowMathQA↑\\uparrowAvg\.↑\\uparrowDrop↓\\downarrow0\.0Baseline5\.687\.340\.280\.670\.380\.670\.560\.780\.270\.520\.000\.2FWSVD2e52e30\.090\.110\.060\.050\.080\.100\.050\.0884\.6%ASVD†11\.1415\.930\.250\.530\.270\.640\.410\.680\.240\.4317\.3%SVD\-LLM7\.9415\.840\.220\.580\.290\.630\.430\.690\.240\.4415\.4%Dobi\-SVD8\.5410\.010\.260\.590\.310\.660\.440\.700\.230\.4611\.5%LACE\-SVD7\.3910\.990\.260\.620\.330\.660\.450\.710\.240\.479\.6%0\.4FWSVD2e41e40\.060\.050\.020\.020\.000\.050\.030\.0394\.2%ASVD†1e31e30\.130\.280\.220\.480\.260\.550\.190\.3042\.3%SVD\-LLM13\.1149\.830\.190\.420\.250\.580\.330\.600\.210\.3728\.9%Dobi\-SVD13\.5423\.540\.220\.410\.270\.580\.340\.610\.230\.3826\.9%LACE\-SVD12\.0021\.300\.230\.500\.260\.600\.350\.620\.230\.4023\.1%0\.6FWSVD3e42e40\.060\.010\.000\.000\.010\.010\.000\.0198\.1%ASVD†6e44e50\.120\.260\.210\.490\.260\.530\.180\.2944\.2%SVD\-LLM53\.74345\.490\.140\.280\.220\.500\.270\.550\.210\.3140\.4%Dobi\-SVD46\.18190\.620\.150\.310\.200\.520\.280\.540\.220\.3238\.4%LACE\-SVD32\.5772\.150\.160\.320\.230\.520\.290\.550\.220\.3336\.5%0\.8SVD\-LLM134962240\.070\.030\.010\.040\.020\.070\.010\.0492\.3%LACE\-SVD238\.05406\.500\.140\.270\.200\.510\.260\.530\.210\.3042\.3%

Performance on Different LLM Architectures\.To evaluate the generalizability of our proposed LACE\-SVD across diverse model architectures, we benchmark it against baseline methods on four distinct models from different LLM families: OPT\-6\.7B, LLaMA\-2\-7B, Mistral\-7B, and Vicuna\-7B\. All models are evaluated under a 20% compression ratio on WikiText\-2 \(perplexity\) and six common sense reasoning datasets \(average accuracy\)\. As demonstrated in Table[2](https://arxiv.org/html/2607.03057#S4.T2), vanilla SVD and FWSVD suffer from catastrophic collapse across all four models, yielding practically unusable perplexities and near\-zero zero\-shot accuracies\. Notably, compared to the strongest baseline SVD\-LLM, our LACE\-SVD achieves lower perplexity on every model\. For instance, on Mistral\-7B and Vicuna\-7B, LACE\-SVD reduces the perplexity to 8\.46 and 7\.91, respectively, significantly outperforming SVD\-LLM\. Furthermore, LACE\-SVD consistently achieves the highest average accuracy among all compressed models, further validating its superior ability to preserve the downstream reasoning capabilities of various LLMs\.

Table 2:Perplexity \(↓\\downarrow\) of LACE\-SVD and baselines on WikiText\-2 and the average accuracy \(↑\\uparrow\) of the six common sense reasoning datasets of four different LLMs\-OPT\-6\.7B, LLaMA\-2\-7B, Mistral\-7B, and Vicuna\-7B–under 20% compression ratio\.OPT\-6\.7BLLaMA 2\-7BMistral\-7BVicuna\-7BMethodPerplexity↓\\downarrowAccuracy↑\\uparrowPerplexity↓\\downarrowAccuracy↑\\uparrowPerplexity↓\\downarrowAccuracy↑\\uparrowPerplexity↓\\downarrowAccuracy↑\\uparrowOriginal10\.860\.525\.470\.575\.250\.616\.780\.56SVD662750\.03181920\.091596270\.03186440\.05FWSVD145590\.0623600\.1263570\.0827580\.09ASVD82\.000\.3210\.100\.3613\.720\.3216\.230\.33SVD\-LLM16\.040\.418\.500\.5310\.210\.428\.410\.51LACE\-SVD15\.390\.437\.540\.558\.460\.507\.910\.54

![Refer to caption](https://arxiv.org/html/2607.03057v1/x3.png)Figure 3:Perplexity \(↓\\downarrow\) and average accuracy \(↑\\uparrow\) of LLaMA\-13B under 20% compression\.
![Refer to caption](https://arxiv.org/html/2607.03057v1/x4.png)Figure 4:Memory usage and inference speedup of LLaMA\-7B on varying compression ratios\.

Performance on LLMs with Larger Scales\.To examine the generalizability and scalability of our proposed method, we evaluate its performance on larger language models, specifically LLaMA\-13B, under a 20% compression ratio\. As shown in Fig[4](https://arxiv.org/html/2607.03057#S4.F4), LACE\-SVD consistently outperforms all baseline methods\.

Memory Usage and Inference Speedup\.Low\-rank approximation inherently decreases both computational complexity and parameter storage, leading to a dual reduction in memory footprint and inference latency\. We evaluated the inference speedup and memory consumption of our proposed LACE\-SVD on LLaMA\-7B under varying compression ratios on a single NVIDIA H200 GPU\. As illustrated in Figure[4](https://arxiv.org/html/2607.03057#S4.F4), LACE\-SVD achieves consistent and significant inference acceleration over the uncompressed baseline\. At a 20% compression ratio, the model achieves a 1\.26×\\timesspeedup while reducing the memory requirement from the baseline’s 12\.90 GB to 10\.55 GB\. As the compression ratio increases, the efficiency gains become even more pronounced\. At a 40% compression ratio, the speedup reaches 1\.51×\\timeswith an 8\.15 GB memory footprint\. Most notably, at an aggressive 60% compression ratio, LACE\-SVD delivers nearly a factor of two acceleration \(1\.97×\\timesspeedup\) while slashing the memory demand by more than half, down to just 5\.60 GB\. These strictly aligned results strongly demonstrate the practicality of LACE\-SVD in facilitating highly efficient LLM deployment in resource\-constrained environments\.

Comparison with Structured Pruning\.We also evaluate our proposed LACE\-SVD against three state\-of\-the\-art structured pruning\-based LLM compression methods: LLM\-PrunerMaet al\.\([2023](https://arxiv.org/html/2607.03057#bib.bib32)\), SliceGPTAshkbooset al\.\([2024](https://arxiv.org/html/2607.03057#bib.bib9)\), and BlockPrunerZhonget al\.\([2025](https://arxiv.org/html/2607.03057#bib.bib44)\)\. The comparison is conducted on LLaMA\-7B using the WikiText\-2 dataset under strictly identical memory budgets, ranging from 10 GB down to 7 GB\. As shown in Table[3](https://arxiv.org/html/2607.03057#S4.T3), LACE\-SVD consistently and significantly outperforms all three structured pruning baselines across all memory constraints\. More importantly, while the pruning\-based methods suffer from severe performance degradation as the memory budget becomes tighter, our method exhibits remarkable robustness\. For instance, when the memory budget drops to 8 GB and 7 GB, the perplexities of the pruning methods soar to the range of 16\.39 to 43\.05\. In stark contrast, LACE\-SVD maintains remarkably low and stable perplexities of 11\.27 and 18\.81, respectively\. In particular, under the highly restrictive 7 GB memory budget, LACE\-SVD achieves a 13% reduction in perplexity even when compared to the best\-performing pruning baseline, LLM\-Pruner \(18\.81 vs\. 21\.68\), demonstrating the superior memory efficiency and preservation ability of our low\-rank approach\.

Table 3:WikiText\-2 perplexity of LLaMA\-7B against structured pruning across different memory budgets\.Under Various Memory BudgetsMethod10 GB9 GB8 GB7 GBLLM\-Pruner9\.8812\.2118\.9421\.68SliceGPT8\.7812\.7316\.3927\.41BlockPruner9\.4012\.7619\.7843\.05LACE\-SVD7\.608\.8511\.2718\.81### 4\.1Ablation Study on Core Components

We conduct a systematic ablation study under a fixed0\.60\.6compression ratio to isolate the contributions of LACE\-SVD’s core components over the foundational Activation\-Whitened SVD baseline, denoted as SVD\-LLM \(Table[4](https://arxiv.org/html/2607.03057#S4.T4)\)\. At this aggressive compression level, the baseline suffers severe performance degradation, making these structural enhancements strictly necessary\.Loss\-Aware Rank Allocation:Replacing uniform rank assignment with loss\-aware allocation improves WikiText\-2 PPL from 53\.74 to 45\.18\. By explicitly tying allocation to language\-modeling degradation, it efficiently directs the parameter budget to loss\-sensitive layers, preventing catastrophic information bottlenecking in critical transformer blocks\.Simultaneous Local Update:Applying the closed\-form local update to the baseline yields a clear gain, reducing PPL from 53\.74 to 43\.56\. Solving the regularized least\-squares problem minimizes module\-level errors, while our simultaneous mode avoids calibration instabilities caused by partially compressed intermediate states\. This ensures the reconstructed activations faithfully match the full\-precision outputs before progressing deeper into the network\.Cumulative Error Correction \(CEC\):Adding CEC over the local update further reduces PPL to 38\.25\. This confirms that adjusting residual\-stream projections, guarded by a layer\-level acceptance gate, successfully mitigates the compounded degradation that local objectives alone cannot resolve\.Full LACE\-SVD Synergy:Combining all components achieves the best performance \(PPL: 32\.57\)\. The allocation module provides an optimal parameter foundation, while the combined correction pipeline systematically compensates for truncation gaps at both module and network levels\. Ultimately, our unified framework reduces the PPL by over 21 points compared to the baseline, demonstrating that structural allocation and multi\-granularity error correction are highly synergistic\.

Table 4:Ablation study on the core components of LACE\-SVD\.We report the Perplexity \(PPL, lower is better\) on WikiText\-2 under 0\.6 compression ratio\.SVD\-LLMdenotes the foundational Activation\-Whitened SVD\.Method VariantWikitext\-2 PPL↓\\downarrowSVD\-LLM \(Baseline\)53\.74SVD\-LLM \+ Allocation45\.18SVD\-LLM \+ Local Update43\.56SVD\-LLM \+ Local Update \+ CEC38\.25Full LACE\-SVD32\.57

## 5Conclusion

In this paper, we introduced LACE\-SVD to address the suboptimal parameter distribution and error accumulation issues in existing SVD\-based LLM compression\. It bridges the gap between local matrix factorization and global model fidelity through two mechanisms: loss\-aware layer\-wise rank allocation and propagation\-aware cumulative error correction\. Experiments show our approach maintains strong language modeling performance and significantly outperforms strong baselines under high compression ratios, proving its effectiveness for resource\-constrained LLM deployment\.

## Limitations

While LACE\-SVD is highly effective for post\-training LLM compression, it involves two practical trade\-offs: First, our rank allocation evaluates layer sensitivities independently\. Although this ignores non\-linear cross\-layer coupling, this necessary approximation ensures computational tractability and enables an efficient dynamic programming solution\. Second, the cumulative error correction uses layer\-output discrepancy \(L2L\_\{2\}distance\) as a proxy instead of directly optimizing the global loss\. This design deliberately avoids the prohibitive costs of end\-to\-end backpropagation, striking a practical balance between compression efficiency and model fidelity\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1)\.
- A\. Amini, S\. Gabriel, S\. Lin, R\. Koncel\-Kedziorski, Y\. Choi, and H\. Hajishirzi \(2019\)MathQA: towards interpretable math word problem solving with operation\-based formalisms\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 2357–2367\.External Links:[Link](https://aclanthology.org/N19-1245/),[Document](https://dx.doi.org/10.18653/v1/N19-1245)Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p1.1)\.
- S\. Ashkboos, M\. L\. Croci, M\. G\. do Nascimento, T\. Hoefler, and J\. Hensman \(2024\)SliceGPT: compress large language models by deleting rows and columns\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=vXxardq6db)Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1),[§2](https://arxiv.org/html/2607.03057#S2.p1.1),[§4](https://arxiv.org/html/2607.03057#S4.p2.1),[§4](https://arxiv.org/html/2607.03057#S4.p7.1)\.
- J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.\(2023\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1)\.
- Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p1.1)\.
- W\. Chiang, Z\. Li, Z\. Lin, Y\. Sheng, Z\. Wu, H\. Zhang, L\. Zheng, S\. Zhuang, Y\. Zhuang, J\. E\. Gonzalez,et al\.\(2023\)Vicuna: an open\-source chatbot impressing gpt\-4 with 90%\* chatgpt quality\.See https://vicuna\. lmsys\. org \(accessed 14 April 2023\)2\(3\),pp\. 6\.Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the AI2 reasoning challenge\.CoRRabs/1803\.05457\.External Links:[Link](http://arxiv.org/abs/1803.05457),1803\.05457Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p1.1)\.
- T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer \(2022\)Gpt3\. int8 \(\): 8\-bit matrix multiplication for transformers at scale\.Advances in neural information processing systems35,pp\. 30318–30332\.Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1)\.
- T\. Dettmers, R\. A\. Svirschevski, V\. Egiazarian, D\. Kuznedelev, E\. Frantar, S\. Ashkboos, A\. Borzunov, T\. Hoefler, and D\. Alistarh \(2024\)SpQR: a sparse\-quantized representation for near\-lossless llm weight compression\.In12th International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.03057#S2.p1.1)\.
- E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh \(2022\)Gptq: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1),[§2](https://arxiv.org/html/2607.03057#S2.p1.1)\.
- G\. Golub, A\. Hoffman, and G\. Stewart \(1987\)A generalization of the eckart\-young\-mirsky matrix approximation theorem\.Linear Algebra and its Applications88,pp\. 317–327\.Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1),[§2](https://arxiv.org/html/2607.03057#S2.p2.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1),[§2](https://arxiv.org/html/2607.03057#S2.p1.1)\.
- \[14\]Y\. Hsu, T\. Hua, S\. Chang, Q\. Lou, Y\. Shen, and H\. JinLanguage model compression with weighted low\-rank factorization\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.03057#S2.p2.1),[§4](https://arxiv.org/html/2607.03057#S4.p2.1)\.
- X\. Hu, Y\. Cheng, D\. Yang, Z\. Chen, Z\. Xu, Z\. Yuan, S\. Zhou,et al\.\(2025\)OSTQuant: refining large language model quantization with orthogonal and scaling transformations for better distribution fitting\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1),[§2](https://arxiv.org/html/2607.03057#S2.p1.1)\.
- X\. Hu, Y\. Cheng, D\. Yang, Z\. Yuan, J\. Yu, C\. Xu, and S\. Zhou \(2024\)I\-llm: efficient integer\-only inference for fully\-quantized low\-bit large language models\.arXiv preprint arXiv:2405\.17849\.Cited by:[§2](https://arxiv.org/html/2607.03057#S2.p1.1)\.
- X\. Hu, D\. Yang, Y\. Cheng, Z\. Chen, and Z\. Xu \(2026\)SAES\-svd: self\-adaptive suppression of accumulated and local errors for svd\-based llm compression\.arXiv preprint arXiv:2602\.03051\.Cited by:[§2](https://arxiv.org/html/2607.03057#S2.p2.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p1.1)\.
- Z\. Li, M\. Xia, J\. Zhang, Z\. Hui, H\. Qin, L\. Kong, Y\. Zhang, and X\. Yang \(2025\)Adasvd: adaptive singular value decomposition for large language models\.arXiv preprint arXiv:2502\.01403\.Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p2.1)\.
- J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han \(2024\)Awq: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of machine learning and systems6,pp\. 87–100\.Cited by:[§2](https://arxiv.org/html/2607.03057#S2.p1.1)\.
- X\. Ma, G\. Fang, and X\. Wang \(2023\)Llm\-pruner: on the structural pruning of large language models\.Advances in neural information processing systems36,pp\. 21702–21720\.Cited by:[§2](https://arxiv.org/html/2607.03057#S2.p1.1),[§4](https://arxiv.org/html/2607.03057#S4.p2.1),[§4](https://arxiv.org/html/2607.03057#S4.p7.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.arXiv e\-prints,pp\. arXiv–1609\.Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p1.1)\.
- M\. Nagel, M\. Fournarakis, R\. A\. Amjad, Y\. Bondarenko, M\. Van Baalen, and T\. Blankevoort \(2021\)A white paper on neural network quantization\.arXiv preprint arXiv:2106\.08295\.Cited by:[§2](https://arxiv.org/html/2607.03057#S2.p1.1)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p1.1)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2019\)WinoGrande: an adversarial winograd schema challenge at scale\.arXiv preprint arXiv:1907\.10641\.Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023a\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023b\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p1.1)\.
- Q\. Wang, J\. Ke, M\. Tomizuka, Y\. Chen, K\. Keutzer, and C\. Xu \(2025a\)Dobi\-svd: differentiable svd for llm compression and some new perspectives\.arXiv preprint arXiv:2502\.02723\.Cited by:[§2](https://arxiv.org/html/2607.03057#S2.p2.1),[§4](https://arxiv.org/html/2607.03057#S4.p2.1)\.
- X\. Wang, S\. Alam, Z\. Wan, H\. Shen, and M\. Zhang \(2025b\)SVD\-LLM v2: optimizing singular value truncation for large language model compression\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4287–4296\.External Links:[Link](https://aclanthology.org/2025.naacl-long.217/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.217),ISBN 979\-8\-89176\-189\-6Cited by:[§2](https://arxiv.org/html/2607.03057#S2.p2.1)\.
- \[30\]X\. Wang, Y\. Zheng, Z\. Wan, and M\. ZhangSVD\-llm: truncation\-aware singular value decomposition for large language model compression\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p2.1),[§2](https://arxiv.org/html/2607.03057#S2.p1.1),[§2](https://arxiv.org/html/2607.03057#S2.p2.1),[§4](https://arxiv.org/html/2607.03057#S4.p2.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1)\.
- Z\. Yuan, Y\. Shang, Y\. Song, D\. Yang, Q\. Wu, Y\. Yan, and G\. Sun \(2023\)Asvd: activation\-aware singular value decomposition for compressing large language models\.arXiv preprint arXiv:2312\.05821\.Cited by:[§1](https://arxiv.org/html/2607.03057#S1.p1.1),[§1](https://arxiv.org/html/2607.03057#S1.p2.1),[§2](https://arxiv.org/html/2607.03057#S2.p1.1),[§2](https://arxiv.org/html/2607.03057#S2.p2.1),[§4](https://arxiv.org/html/2607.03057#S4.p2.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 4791–4800\.External Links:[Link](https://aclanthology.org/P19-1472/),[Document](https://dx.doi.org/10.18653/v1/P19-1472)Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p1.1)\.
- M\. Zhang, H\. Chen, C\. Shen, Z\. Yang, L\. Ou, X\. Yu, and B\. Zhuang \(2023\)Loraprune: pruning meets low\-rank parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2106\.08295\.Cited by:[§2](https://arxiv.org/html/2607.03057#S2.p1.1)\.
- S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin,et al\.\(2022\)Opt: open pre\-trained transformer language models\.arXiv preprint arXiv:2205\.01068\.Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p1.1)\.
- L\. Zhong, F\. Wan, R\. Chen, X\. Quan, and L\. Li \(2025\)Blockpruner: fine\-grained pruning for large language models\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 5065–5080\.Cited by:[§4](https://arxiv.org/html/2607.03057#S4.p2.1),[§4](https://arxiv.org/html/2607.03057#S4.p7.1)\.

## Appendix AAlgorithm

Algorithm 1LACE\-SVD: Loss\-Aware and Cumulative Error Corrected SVD0:Pretrained LLM

fθf\_\{\\theta\}with layers

\{Fℓ\}1L\\\{F\_\{\\ell\}\\\}\_\{1\}^\{L\}, calibration data

XX, target budget

ρ\\rho, candidate ratios

ℛρ\\mathcal\{R\}\_\{\\rho\}, correction strength

α\\alpha, regularization

λU\\lambda\_\{U\}\.

0:Compressed LLM

fθ^f\_\{\\hat\{\\theta\}\}
1:Compute Activation\-Whitened SVD for all weights to obtain initial factors\.

2:Contribution 1: Loss\-Aware Layer\-Wise Rank Allocation

3:foreach layer

ℓ∈\{1,…,L\}\\ell\\in\\\{1,\\dots,L\\\}and candidate ratio

r∈ℛρr\\in\\mathcal\{R\}\_\{\\rho\}do

4:Evaluate loss gap:

Δℓ,r←ℒcalib​\(fθ\(ℓ,r\)\)−ℒcalib​\(fθ\)\\Delta\_\{\\ell,r\}\\leftarrow\\mathcal\{L\}\_\{\\mathrm\{calib\}\}\\big\(f\_\{\\theta\}^\{\(\\ell,r\)\}\\big\)\-\\mathcal\{L\}\_\{\\mathrm\{calib\}\}\(f\_\{\\theta\}\)
5:endfor

6:Solve DP:

\{rℓ⋆\}1L←arg⁡min​∑Δℓ,rℓs\.t\.∑Cℓ,rℓ≤ρ​Cfull\\\{r\_\{\\ell\}^\{\\star\}\\\}\_\{1\}^\{L\}\\leftarrow\\arg\\min\\sum\\Delta\_\{\\ell,r\_\{\\ell\}\}\\quad\\text\{s\.t\.\}\\quad\\sum C\_\{\\ell,r\_\{\\ell\}\}\\leq\\rho C\_\{\\mathrm\{full\}\}
7:Contribution 2: Local Update & Cumulative Error Correction

8:foreach layer

ℓ=1​…​L\\ell=1\\dots Ldo

9:foreach target projection

WWin

FℓF\_\{\\ell\}do

10:Truncate factors to rank

rℓ⋆r\_\{\\ell\}^\{\\star\}to get

U0U\_\{0\}and

VV\.

11:Hidden feature

Z←X​V⊤Z\\leftarrow XV^\{\\top\}, full\-precision target

Y←X​W⊤Y\\leftarrow XW^\{\\top\}\.

12:*Simultaneous Local Closed\-Form Update*

13:

U⋆←arg⁡minU⁡‖Z​U⊤−Y‖F2\+λU​‖U−U0‖F2U^\{\\star\}\\leftarrow\\arg\\min\_\{U\}\\\|ZU^\{\\top\}\-Y\\\|\_\{F\}^\{2\}\+\\lambda\_\{U\}\\\|U\-U\_\{0\}\\\|\_\{F\}^\{2\}
14:*Cumulative Error Correction \(CEC\)*

15:ifmodule

s∈𝒮=\{o\_proj,down\_proj\}s\\in\\mathcal\{S\}=\\\{\\texttt\{o\\\_proj\},\\texttt\{down\\\_proj\}\\\}then

16:Target

Tℓ,spa←Mℓ,scmp\+α​\(Mℓ,sfull−Mℓ,scmp\)T\_\{\\ell,s\}^\{\\mathrm\{pa\}\}\\leftarrow M\_\{\\ell,s\}^\{\\mathrm\{cmp\}\}\+\\alpha\(M\_\{\\ell,s\}^\{\\mathrm\{full\}\}\-M\_\{\\ell,s\}^\{\\mathrm\{cmp\}\}\)
17:Re\-solve for

Upa⋆U\_\{\\mathrm\{pa\}\}^\{\\star\}replacing

YYwith

Tℓ,spaT\_\{\\ell,s\}^\{\\mathrm\{pa\}\}\.

18:Let

FℓpaF\_\{\\ell\}^\{\\mathrm\{pa\}\}and

FℓcmpF\_\{\\ell\}^\{\\mathrm\{cmp\}\}be layers using

Upa⋆U\_\{\\mathrm\{pa\}\}^\{\\star\}and

U⋆U^\{\\star\}respectively\.

19:if

‖Fℓpa​\(Hℓ\)−Fℓfull​\(Hℓ\)‖F2<‖Fℓcmp​\(Hℓ\)−Fℓfull​\(Hℓ\)‖F2\\\|F\_\{\\ell\}^\{\\mathrm\{pa\}\}\(H\_\{\\ell\}\)\-F\_\{\\ell\}^\{\\mathrm\{full\}\}\(H\_\{\\ell\}\)\\\|\_\{F\}^\{2\}<\\\|F\_\{\\ell\}^\{\\mathrm\{cmp\}\}\(H\_\{\\ell\}\)\-F\_\{\\ell\}^\{\\mathrm\{full\}\}\(H\_\{\\ell\}\)\\\|\_\{F\}^\{2\}then

20:Accept correction:

U⋆←Upa⋆U^\{\\star\}\\leftarrow U\_\{\\mathrm\{pa\}\}^\{\\star\}
21:endif

22:endif

23:Update compressed projection:

W^←U⋆​V\\widehat\{W\}\\leftarrow U^\{\\star\}V
24:endfor

25:endfor

26:return

fθ^f\_\{\\hat\{\\theta\}\}defined by updated

\{W^\}\\\{\\widehat\{W\}\\\}

## Appendix BImplementation Details and Main Hyperparameters\.

To ensure a fair comparison, we followed SVD\-LLM and randomly selected 256 samples from WikiText\-2 as calibration data\. All experiments were executed on multiple NVIDIA H200 GPUs\.

ItemValueBackboneLLaMA\-7BTarget compression ratioρ=0\.4\\rho=0\.4Calibration corpusWikitext\-2Whitening samples256Whitening sequence length2048Loss\-aware batches64Loss\-aware sequence length1024Loss\-aware batch size16Candidate evaluationtwo\-stageStage\-1 batches16Stage\-1 top\-kk4DP budget bins4000Local update modesimultaneousLocal update samples64Local update micro\-batch size8Minimum held\-out gain2×10−42\\times 10^\{\-4\}Propagation\-aware moduleso\_proj,down\_projPropagation strengthα=0\.7\\alpha=0\.7Propagation gate batches64Table 5:Main hyperparameters for the LLaMA\-7Bρ=0\.4\\rho=0\.4setting\.
## Appendix CAblation Study on Correction Strength \(α\\alpha\)

In the LACE\-SVD local update stage, we employ a propagation\-aware cumulative error correction mechanism for modules in𝒮\\mathcal\{S\}\. As formulated in Equation[14](https://arxiv.org/html/2607.03057#S3.E14), the target activationTℓ,spaT\_\{\\ell,s\}^\{\\mathrm\{pa\}\}is constructed by interpolating between the full\-precision module outputMℓ,sfullM\_\{\\ell,s\}^\{\\mathrm\{full\}\}and the compressed module outputMℓ,scmpM\_\{\\ell,s\}^\{\\mathrm\{cmp\}\}using the correction strengthα∈\[0,1\]\\alpha\\in\[0,1\]\. To investigate the impact ofα\\alphaon the final language modeling performance, we conduct an ablation study on the WikiText\-2 dataset across various values\. The perplexity \(PPL\) trends are illustrated in Figure[5](https://arxiv.org/html/2607.03057#A3.F5)\.

As observed in Figure[5](https://arxiv.org/html/2607.03057#A3.F5), the performance exhibits a highly non\-linear relationship withα\\alpha\. Specifically, the framework achieves the best perplexity \(32\.57\) whenα=0\.7\\alpha=\\textbf\{0\.7\}\. In this regime, the target blends the full\-precision output with a moderate proportion of the currentMℓ,scmpM\_\{\\ell,s\}^\{\\mathrm\{cmp\}\}, acting as a strong implicit regularizer that restricts the magnitude of the matrix update\. This prevents aggressive overfitting and preserves the baseline orthogonality of the truncated SVD\. Interestingly, a counter\-intuitive performance spike \(severe degradation\) occurs whenα\\alphais slightly increased to the range of\[0\.8,0\.9\]\[\\textbf\{0\.8\},\\textbf\{0\.9\}\], peaking at over38forα=0\.8\\alpha=\\textbf\{0\.8\}\. We attribute this degradation to the structural noise introduced into the closed\-form solver: blending this specific, unbalanced ratio of the inherently biased compressed output into the target forces the solver to fit a distorted objective, leading to catastrophic error accumulation in deeper layers\. Conversely, asα\\alphareaches1\.0, the objective degenerates toTℓ,spa=Mℓ,sfullT\_\{\\ell,s\}^\{\\mathrm\{pa\}\}=M\_\{\\ell,s\}^\{\\mathrm\{full\}\}, functioning as pure teacher forcing\. While supplying the exact, uncompressed target activations allows the performance to recover significantly \(yielding a PPL of approximately33\.6\), it remains sub\-optimal compared to the interpolated target atα=0\.7\\alpha=\\textbf\{0\.7\}\. Based on these empirical findings, we fixα=0\.7\\alpha=\\textbf\{0\.7\}for all our main experiments to ensure optimal and stable layer\-wise reconstruction\.

![Refer to caption](https://arxiv.org/html/2607.03057v1/x5.png)Figure 5:Ablation study of the correction strengthα\\alphaon WikiText\-2 perplexity using the LLaMA\-7B model\. The compression ratio \(0\.6\) is kept identical across all settings\.
## Appendix DCalibration Setup

We use Wikitext\-2 as the calibration corpus\. Whitening statistics are computed from 256 calibration samples with sequence length 2048\. For loss\-aware layerwise allocation, candidate ratios are evaluated using 64 calibration batches with sequence length 1024 and batch size 16\.

### D\.1Candidate Ratio Search

For each target compression ratioρ\\rho, we construct a finite candidate setℛρ\\mathcal\{R\}\_\{\\rho\}aroundρ\\rho\. The exact candidate set is treated as an experimental hyperparameter and varies with the target compression ratio\. The final layerwise ratios are selected by solving the multiple\-choice knapsack problem\.

### D\.2Two\-Stage Candidate Evaluation

Evaluating all candidates for all layers on the full calibration set is expensive\. We therefore use a two\-stage candidate evaluation strategy\. In the first stage, all candidates are evaluated using 16 calibration batches\. For each layer, we retain the top candidates according to calibration loss increase and cost\-effectiveness\. In the second stage, the retained candidates are evaluated using the full calibration set\.

This procedure reduces computational cost but does not change the final budgeted optimization objective\.

### D\.3Local Update Details

After layerwise allocation, we perform simultaneous local update\. All linear modules in a layer are fitted using the original full\-precision activations\. We use residual\-based token weighting and a held\-out selection rule\. The minimum relative held\-out improvement is2×10−42\\times 10^\{\-4\}\.

The ridge coefficients are:

λU=10−5,λV=10−4\.\\lambda\_\{U\}=10^\{\-5\},\\qquad\\lambda\_\{V\}=10^\{\-4\}\.\(16\)The singular value floor used for numerical stability is10−510^\{\-5\}\.

### D\.4Propagation\-Aware Refinement Details

Propagation\-aware residual local update is applied too\_projanddown\_proj\. We use correction strengthα=0\.7\\alpha=0\.7and evaluate the layer\-output acceptance gate on at most 64 calibration batches per layer\. Candidate updates that fail the gate are discarded and the previous factors are restored\.

### D\.5Efficiency Engineering

We cache loss\-aware candidate tables to avoid repeated candidate evaluation\. The cache stores layerwise candidate losses, parameter costs, and metadata\. We also optionally preload calibration batches onto the GPU to reduce host\-device transfer overhead\. These components are used only for efficiency and do not alter the compression objective\.