Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression

# Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression
Source: [https://arxiv.org/html/2607.18284](https://arxiv.org/html/2607.18284)
\\tfootnote

This research was supported by the EU Project VOXReality \(Voice\-driven Interaction in XR Spaces\) under Grant 101070521\.

\\corresp

Corresponding author: Athanasios Ntovas \(e\-mail: atdovas@iti\.gr\)\.

ALEXANDROS DOUMANOGLOU 1PETROS DRAKOULIS1 and DIMITRIS ZARPALAS1Information Technologies Institute \(ITI\), Centre for Research and Technology HELLAS \(CERTH\), Thessaloniki, Greece\. \(email: \{atdovas,aldoum,petros\.drakoulis,zarpalas\}@iti\.gr\)

###### Abstract

To excel at their domain, large language models are comprised of billions of parameters\. Yet, this comes at the cost of huge memory requirements, restricting their applicability in resource\-constrained environments\. To address the problem of neural network \(NN\) compression, Singular Value Decomposition \(SVD\) has played a key role as a fundamental component for matrix compression through decomposition\. To minimize compression error and to maximize the efficacy of the compressed model on the downstream tasks, previous works focused on low\-rank approximation of the NN’s weight matrices either from the perspective of parameter importance or per\-layer functional equivalence\. While previous works studied the aforementioned perspectives in isolation, in this work, we are investigating the effectiveness of an approach that combines ideas from these two perspectives in a single objective\. In parallel to this, an important aspect that affects the compression quality is the distribution of the compression rate across layers and NN parameters\. Earlier works mostly considered distributing the compression rate uniformly across layers and network weights or relied on computationally expensive heuristic search\. Contrary to them, in this work, we propose an enhanced and computationally efficient algorithm for dynamic compression rate allocation\. Experimental results support the efficacy of the proposed approach, which performs on par or substantially better than the previous state\-of\-the\-art, especially under high compression ratios\.

###### Index Terms:

language models, neural network compression, neuron importance, SVD

## IIntroduction

The advances in Artificial Intelligence \(AI\) over the last decade have enabled a multitude of applications that were previously considered unreachable using traditional programming techniques\. Nowadays, AI powers computer vision applications that can understand our world and chatbots that can communicate with humans in natural language, among others\. Currently, AI is mainly realized by deep neural networks \(DNNs\), which are computational architectures that are trained to solve tasks from a particular domain at human\-level performance\. Large Language Models \(LLMs\)\[gpt,bert,gemma2,gemma3,llama,claude\]constitute DNNs that excel in Natural Language Understanding \(NLU\) and Processing \(NLP\) and enable fluent human\-machine interfaces based on natural language\. However, these DNNs come with a ton of parameters and a large memory and computational footprint\. With AI becoming ubiquitous, their efficient deployment in resource\-constrained environments such as edge devices, smartphones, and mini\-PCs becomes a hard requirement\. Thus, compressing LLMs for efficient deployment has naturally emerged as a field of its own\[comp\-survey1,comp\-survey2\]\.

Since 2017, the transformer architecture\[transformer\]has been the main workhorse behind LLMs\. In their simplest form, transformers process natural language by splitting sentences and words into tokens, which are subsequently processed sequentially by a series of transformer blocks, organized in layers\. Each transformer block consists of a self\-attention module and a feedforward layer, both of which are parameterized by matrices of weights, with each weight corresponding to a parameter that is tuned during network training\. Compressing a transformer network can be accomplished in a variety of ways, but the end goal is common: to reduce the number of network parameters that need to be stored in memory with a minimal compromise in network performance\.

![Refer to caption](https://arxiv.org/html/2607.18284v1/sections/images/architecture-diagram.png)Figure 1:Architectural overview of the hybrid compression pipeline \(NIDA\-SVD\)\. Our approach synthesizes the weight importance with data\-aware low\-rank approximation, guided by our dynamic rank allocation scheme\. In the architectural representation, layer components such as𝑸i\{\\bm\{Q\}\}\_\{i\}\(Query\) and𝑲i\{\\bm\{K\}\}\_\{i\}\(Key\) are used to denote the corresponding compressible weight matrices𝑾Qi\{\\bm\{W\}\}\_\{Q\_\{i\}\}and𝑾Ki\{\\bm\{W\}\}\_\{K\_\{i\}\}respectively\.One line of research for LLM compression attempts to reduce the LLM’s parameters by approximating the weight matrices of the transformer blocks with the product of two low\-rank matrices, a technique also known aslow\-rank approximation\. Since low\-rank, these matrices are crafted to have a sum of parameters that is less than the parameters found in the original matrix\. Singular Value Decomposition \(SVD\)\[svd\]is a matrix decomposition method\[comp\-survey3\]that plays a key role in the low\-rank approximation problem, as, given a target rank, it is proven to produce the best possible approximation with a minimum matrix reconstruction error\.

Beyond its naive use, SVD has been employed in more sophisticated approaches for compressing DNN matrices\. First, using weighted low\-rank factorization, FWSVD\[fwsvd\]takes into account the importance of each parameter to the DNN’s output, improving model performance under equivalent compression ratios\. And second, the data\-aware approach of SVD\-LLMv2\[svd\-llmv2\]aims for a functionally equivalent approximation of the DNN’s weight matrix by taking into account layer inputs to be multiplied with the transformation matrix through a small calibration dataset\. Each one of the approaches has demonstrated significant improvements compared to baselines, yet an approach combining ideas from both is currently missing from the literature\.

In this work, our first contribution is to address this gap\. Since data\-aware low rank approximation has been the previous state\-of\-the\-art in LLM compression, we opt for keeping it as a fundamental component of our method while seeking to improve it\. Our investigation and analysis show that naively combining data\-aware low rank approximation with theparameterimportance estimation method that was suggested by FWSVD degrades the LLM’s performance on downstream tasks compared to using data\-aware low rank approximation alone\. To overcome this, we propose to useneuronimportance estimation, an alternative approach which considers parameters in functional groups, that often leads to substantial improvements in downstream model performance compared to the previous state\-of\-the\-art under similar compression ratios\.

Besides the contributions that we make in the compression method itself, we also consider the problem of parameter count allocation across weight matrices\. Although in previous works uniform allocation has been the most common approach to distribute parameters across matrices, in the recent approach of SVD\-LLMv2, a novel algorithm has been proposed as an improvement of the previous naive uniform strategy\. This previous approach relies on grouping weight matrices based on their functionality in the transformer blocks\. In this work, we make a second contribution by providing experimental evidence that a different grouping strategy, based on layer index, is more effective\. Our analysis, ablation studies, and experimental findings support that when compressing BERT\[bert\], DistilBERT\[distilbert\], MobileBERT\[mobilebert\], and TinyBERT\[tinybert\], foundational models for natural language understanding, in the vast majority of cases, the proposed approach achieves state\-of\-the\-art performance compared to other SVD\-based algorithms\. Furthermore, we include a computational analysis across these compressed architectures \(DistilBERT, MobileBERT, and TinyBERT\) detailing the significant reduction in both MFLOPS/token and total parameter count\.

## IIRelated Work

Several approaches have been proposed to compress transformer architectures\[transformer\]in pre\-trained LLMs\[bert,roberta,deberta,debertav3,llama,gemma2,gemma3,gpt\]\. Broadly, these methods fall into four categories\.

Pruningmethods\[prunning\-1,prune\-slicegpt,llm\-surgeon\-prune,sparse\-gpt\-prune\]aim to eliminate as many parameters as possible, effectively zeroing out the corresponding weights in the model’s transformation matrices\. On the one hand,unstructuredpruning removes individual parameters without constraints, but typically requires specialized hardware for efficient deployment\. On the other hand, to improve hardware compatibility,structuredpruning removes entire rows or columns from weight matrices, which aligns well with standard hardware architectures optimized for fast matrix multiplication\.

Distillation\-based methods\[distilbert,tinybert,dynabert\]leverage knowledge distillation\[distillation\-survey\]to reduce the number of neural network \(NN\) parameters by training a smaller model to mimic the behavior of the original\. While often effective, this approach can be computationally expensive, as it requires retraining a model from scratch\.

Third, and most relevant to our work,low\-rank approximationmethods\[comp\-survey4\]approximate transformer weight matrices using singular value decomposition \(SVD\), factorizing them into the product of two low\-rank matrices\. Standard SVD treats all entries as equally important for minimizing reconstruction error\. FWSVD\[fwsvd\]improves on this by weighting parameters according to their importance to the model’s output\. Parameter importance is determined via differentiating the task loss with respect to the parameter and taking the magnitude of the gradient, a process that requires additional access to a calibration dataset\. A more recent study\[features\-low\-rank\-weights\-not\]revealed that most transformer matrices are not low rank, and thus, when trying to compress them via low\-rank approximation, the resulting compressed networks exhibit significant performance drops\. However, it was found that the opposite holds for intermediate token representations, which appear to have a low rank structure\. Since the rank of a matrix product is less than or equal to the minimum among the ranks of individual matrices participating in the multiplication, DRONE\[drone\], ASVD\[asvd\], and SVD\-LLM\[svd\-llm\]approached compression from the perspective of functional equivalence, minimizing errors in the result of matrix multiplications\. However, DRONE required caching intermediate feature activations, resulting in large memory requirements\. SVD\-LLM alleviated this restriction by relying only on their covariance matrix\. At the same time, SVD\-LLM ensured faithful estimation of the matrix multiplication error based on the truncated singular values, a limitation of ASVD\. Yet, in its initial version, SVD\-LLM relied on Cholesky decomposition, which \(a\) requires positive\-definite matrices, a requirement that is not always fulfilled, and \(b\) can suffer from numerical instabilities during iterative optimization\. Both issues were addressed by SVD\-LLMv2\[svd\-llmv2\], using a two\-step SVD algorithm\. Low\-rank approximation has also been combined with knowledge distillation in\[matrix\-decomp\]\. Our approach combines SVD\-LLMv2, the best performing method among the previously discussed state\-of\-the\-art, with ideas inspired from FWSVD\.

In low\-rank approximation approaches, a fundamental hyperparameter that controls the compression rate \- performance tradeoff is the rank of the compressed matrix\. Since the network itself has varying sensitivity with respect to the different matrices across layers, distributing the compression rate across matrices for optimal performance typically requires an exhaustive sensitivity analysis, which is practically infeasible\. Some of the approaches\[fwsvd,svd\-llm\]simply distribute the compression rate evenly across layers, while other approaches\[drone,asvd,svd\-llmv2\]heuristically simplify the exhaustive search in more manageable terms\. Our approach is computationally efficient and builds on top of the rank selection algorithm of SVD\-LLMv2\.

Fourth, and orthogonal to the previous approaches for LLM compression, is weightquantization\[spin\-quantization,quantization\-1,actaware\-quantization,bert\-quantization\], which typically reduces the memory requirements to perform model inference by quantizing the parameters of the LLM under a fixed budget of binary digits\. Recent advances in post\-training quantization \(PTQ\)\[ptq\], demonstrate that high\-accuracy LLM quantization can be achieved without retraining\. In parallel, quantization\-aware training \(QAT\) techniques\[qat\]tailored for LLMs further improve robustness by modeling quantization noise during fine\-tuning\. Similar to other low\-rank approximation techniques, our approach can additionally benefit from such schemes\.

## IIIBackground

In this section, we provide foundational components to understand our approach\. We begin by revisiting core elements of the Transformer architecture\[transformer\], followed by a description of BERT\[bert\], the Language Model that will become the basic subject of our experiments and an overview of the architectures of DistilBERT, MobileBERT, and TinyBERT, given their role as additional models for our compression method\. We then review SVD\[svd\]and its application as a post\-training compression technique\. Finally, we provide an overview of two prominent SVD\-based methods, FWSVD\[fwsvd\]and SVD\-LLMv2\[svd\-llmv2\], which serve both as comparison baselines and inspiration for the development of the proposed method\.

### III\-AThe Transformer Architecture

In transformer architectures, a token is the smallest unit of input \(such as a word, subword, or character\) that the model processes after being mapped to an embedding vector\. Let𝑿∈ℝD×N\{\\bm\{X\}\}\\in\\mathbb\{R\}^\{D\\times N\}denote a matrix ofNNtokens, each one represented by a D\-dimensional vector in the embedding space\.

The core of a Transformer is a stack of layers\. Eachtransformer layerorblockcontains two main components: a self\-attention mechanism and a feed\-forward network \(FFN\)\. The self\-attention mechanism uses two matrices, namely the query𝑸=𝑾QT​𝑿\{\\bm\{Q\}\}=\{\\bm\{W\}\}\_\{Q\}^\{T\}\{\\bm\{X\}\}and key𝑲=𝑾KT​𝑿\{\\bm\{K\}\}=\{\\bm\{W\}\}\_\{K\}^\{T\}\{\\bm\{X\}\}to weigh the importance of each token to the others and create a contextualized representation by linearly combining a matrix of values𝑽=𝑾VT​𝑿\{\\bm\{V\}\}=\{\\bm\{W\}\}\_\{V\}^\{T\}\{\\bm\{X\}\}\. The output of the self\-attention mechanism is subsequently transformed by a linear layer with weight matrix𝑾a\{\\bm\{W\}\}\_\{a\}\. The feed\-forward network, also referred to as the multi\-layer perceptron \(MLP\) block, then processes the previous output to learn higher\-level features\. In the self\-attention mechanism, four weight matrices are involved, which are amenable to compression, namely𝑾Q,𝑾K\{\\bm\{W\}\}\_\{Q\},\{\\bm\{W\}\}\_\{K\},𝑾V\{\\bm\{W\}\}\_\{V\}, and𝑾a\{\\bm\{W\}\}\_\{a\}, while the feed\-forward block introduces two additional matrices, with the first performing up\-projection and the second down\-projection\. We refer to those matrices as𝑾u\{\\bm\{W\}\}\_\{u\}and𝑾d\{\\bm\{W\}\}\_\{d\}, respectively\.

These two components of the transformer block are wrapped in a residual connection, and each one is followed by layer normalization, which both are essential for stable training and scaling to the deep architectures that characterize modern LLMs\.

### III\-BNeurons and Pre\-Activations

A neuron in a NN can be understood as a simple function that computes the inner product between an input vector and a weight vector\. If the input is an embedding vector𝒙∈ℝD\{\\bm\{x\}\}\\in\\mathbb\{R\}^\{D\}corresponding to a token representation, and the neuron’s weights are𝒘∈ℝD\{\\bm\{w\}\}\\in\\mathbb\{R\}^\{D\}, the output is given byy=𝒘T​𝒙y=\{\\bm\{w\}\}^\{T\}\{\\bm\{x\}\}, which is a scalar representing the neuron’s output, also sometimes referred to as the neuron’spre\-activationdue to often becoming the input to a non\-linear activation function such as the rectified linear unit \(ReLU\)\[DLBook\]\.

At a layer level, many neurons are organized together, and their weights form a weight matrix𝑾∈ℝD×D′\{\\bm\{W\}\}\\in\\mathbb\{R\}^\{D\\times D^\{\\prime\}\}, whereD′D^\{\\prime\}corresponds to the number of neurons in the layer\. Additionally, a collection ofNNinput token representations may be arranged in a matrix𝑿∈ℝD×N\{\\bm\{X\}\}\\in\\mathbb\{R\}^\{D\\times N\}\. Then, the layer’s output computation can be expressed compactly as𝒀=𝑾T​𝑿\{\\bm\{Y\}\}=\{\\bm\{W\}\}^\{T\}\{\\bm\{X\}\}, where the output𝒀∈ℝD′×N\{\\bm\{Y\}\}\\in\\mathbb\{R\}^\{D^\{\\prime\}\\times N\}is a matrix, and each element of𝒀\{\\bm\{Y\}\}corresponds to the output of a single neuron for a single token\. Typically, atransformer layeris comprised of a set of weight matrices, each one being responsible for supporting a different mechanism, such as the self\-attention mechanism or the transformation of the feed\-forward network\. Still, we will be referring to these individual weight matrices aslayersdue to stacking a collection of neurons\. In most cases, we will be mentioning transformer layers explicitly, but occasionally whether the term layer refers to them can be inferred from context\.

### III\-CBERT\-based Architectures

The standard Bidirectional Encoder Representations from Transformers \(BERT\) model is an encoder\-only architecture composed of 12 identical Transformer blocks, all following the description given in Section[III\-A](https://arxiv.org/html/2607.18284#S3.SS1)\. BERT pre\-training involves bi\-directional mask\-language modeling, i\.e\. training the network to predict masked tokens by conditioning both on the left and on the right context of the masked token, and additionally performing next sentence prediction, i\.e\. to predict whether a second sentence is a natural continuation of the first one\. This pre\-training scheme allows BERT to excel in NLU tasks with subsequent fine\-tuning and a minimal computation budget\.

DistilBERT is essentially a compact version of BERT, successfully distilled down to 6 layers half the size of the original resulting in40%40\\%fewer parameters\. The trick is knowledge distillation: the small student model is trained not just on text, but also on the outputs of the larger BERT teacher, which allows it to retain a remarkable≈97%\\approx 97\\%of the original model’s language understanding capability\.

MobileBERT was designed specifically for speed and efficiency on mobile devices\. Its unique structure uses a bottleneck design by introducing an intermediate projection layer that significantly narrows the Transformer’s hidden dimensions, coupled with factorized self\-attention mechanisms\. This innovative layer decomposition reduces the parameter count and latency dramatically, allowing it to maintain strong performance while being highly efficient\.

TinyBERT adopts a more aggressive strategy for model compression, aiming to substantially reduce model size while preserving downstream performance\. Its training relies on a comprehensive two\-stage knowledge distillation framework\. In the first stage, the student model learns from the teacher’s internal representations by distilling information from intermediate Transformer layer outputs\. In the second stage, it captures the teacher’s behavior by distilling its attention distributions\. By jointly aligning both hidden states and attention patterns with those of the teacher, TinyBERT achieves significant compression often up to7\.5×7\.5\\timessmaller than BERT base while maintaining strong task performance\.

### III\-DLow\-rank approximation for model compression

A common approach to reduce the size of large neural networks is via SVD and low\-rank approximation\. A weight matrix𝑾∈ℝm×n\{\\bm\{W\}\}\\in\\mathbb\{R\}^\{m\\times n\}can be approximated as𝑾′=𝑼r​𝚺r​𝑽rT\{\\bm\{W\}\}^\{\\prime\}=\{\\bm\{U\}\}\_\{r\}\{\\bm\{\\Sigma\}\}\_\{r\}\{\\bm\{V\}\}\_\{r\}^\{T\}, where𝑼r∈ℝm×r\{\\bm\{U\}\}\_\{r\}\\in\\mathbb\{R\}^\{m\\times r\}and𝑽r∈ℝn×r\{\\bm\{V\}\}\_\{r\}\\in\\mathbb\{R\}^\{n\\times r\}are truncated orthogonal matrices,𝚺r∈ℝr×r\{\\bm\{\\Sigma\}\}\_\{r\}\\in\\mathbb\{R\}^\{r\\times r\}is a diagonal matrix of the toprrsingular values, andr≤rank​\(W\)r\\leq\\text\{rank\}\(W\)\. The singular values in𝚺r\{\\bm\{\\Sigma\}\}\_\{r\}, ordered from largest to smallest, represent the contribution of the corresponding components of𝑼r\{\\bm\{U\}\}\_\{r\}and𝑽r\{\\bm\{V\}\}\_\{r\}in the original matrix𝑾\{\\bm\{W\}\}\.

This technique is used to compress the large, weight matrices found in models like BERT\. Instead of storing the full original matrix𝑾∈ℝm×n\{\\bm\{W\}\}\\in\\mathbb\{R\}^\{m\\times n\}, we can store two smaller matrices,𝑨∈ℝm×r\{\\bm\{A\}\}\\in\\mathbb\{R\}^\{m\\times r\}and𝑩∈ℝr×n\{\\bm\{B\}\}\\in\\mathbb\{R\}^\{r\\times n\}, which, when multiplied, approximate the original matrix \(𝑾≈𝑨​𝑩\{\\bm\{W\}\}\\approx\{\\bm\{A\}\}\{\\bm\{B\}\}\)\. This can be done by simply setting𝑨=𝑼r​𝚺r\{\\bm\{A\}\}=\{\\bm\{U\}\}\_\{r\}\\sqrt\{\{\\bm\{\\Sigma\}\}\_\{r\}\}and𝑩=𝚺r​𝑽rT\{\\bm\{B\}\}=\\sqrt\{\{\\bm\{\\Sigma\}\}\_\{r\}\}\{\\bm\{V\}\}\_\{r\}^\{T\}, with𝚺r\\sqrt\{\{\\bm\{\\Sigma\}\}\_\{r\}\}denoting the matrix with all elements equal to the square root of𝚺r\{\\bm\{\\Sigma\}\}\_\{r\}\.

According to our definition, the original model’s matrix𝑾\{\\bm\{W\}\}hasm​nmnparameters\. The total number of parameters in𝑨\{\\bm\{A\}\}and𝑩\{\\bm\{B\}\}ism​r\+r​n=r​\(m\+n\)mr\+rn=r\(m\+n\), where r is the chosen rank for the approximation\. For effective compression, the new parameter count must be less than the original\. This condition is met when:m​r\+r​n<m​nmr\+rn<mn\. This concludes thatrrmust be chosen such thatr<\(m​n\)/\(m\+n\)r<\(mn\)/\(m\+n\)\. A smallerrrdecreases parameter count and speeds up computations, but increases approximation error, which can affect performance\.

The foundational method for low\-rank approximation within the Transformer architecture is truncated SVD \(tSVD\), which is deterministic\. tSVD works by isolating and keeping only the most important singular components \(vectors and values\) while discarding the rest\. The complexity of tSVD is managed by utilizing specialized Krylov\-based algorithms\[tsvd\]that focus computation exclusively on these dominant components, avoiding the cost of calculating the full matrix spectrum\. An alternative approach is randimized SVD \(rSVD\)\[rSVD\], which uses random projections to create a smaller, sampled version of the matrix, significantly reducing the computation time\. While rSVD offers a notable speed advantage, our methodology relies on the deterministic tSVD because it provides the essential guarantee of optimality \(Eckart\-Young theorem\) and numerical stability\. This stability is crucial for zero\-shot compression tasks\.

### III\-EFWSVD

Fisher\-Weighted Singular Value Decomposition \(FWSVD\) extends low\-rank approximation by integrating the concept of parameter importance\. While the SVD method assumes that all parameters of a model’s weight matrix𝑾\{\\bm\{W\}\}have the same importance to the matrix reconstruction error, FWSVD leverages the Fisher information matrix to quantify the importance of each parameter in𝑾\{\\bm\{W\}\}to the task loss, ensuring that the most critical components for model performance are better preserved during the compression process\.

For a single parameterwi​jw\_\{ij\}at the location\(i,j\)\(i,j\)in a model’s weight matrix𝑾\{\\bm\{W\}\}, the Fisher Information, denoted asI^wi​j\\hat\{I\}\_\{w\_\{ij\}\}, measures the amount of information that a datasetDDprovides about the parameter\. This is computed as the sample average of the squared partial derivative of the task’s loss functionL​\(di;wi​j\)L\(d\_\{i\};w\_\{ij\}\)with respect towi​jw\_\{ij\}, given by:

I^wi​j=1\|D\|​∑i=1\|D\|\(∂∂wi​j​L​\(di;wi​j\)\)2\\hat\{I\}\_\{w\_\{ij\}\}=\\frac\{1\}\{\|D\|\}\\sum\_\{i=1\}^\{\|D\|\}\\left\(\\frac\{\\partial\}\{\\partial w\_\{ij\}\}L\(d\_\{i\};w\_\{ij\}\)\\right\)^\{2\}\(1\)withdid\_\{i\}denoting theii\-th sample of the dataset and\|D\|\|D\|its total number of samples\.

The Fisher\-Weighted approach changes the optimization objective from a generic mathematical one to a task\-specific one\. While the standard SVD objective is to find a low\-rank approximation𝑾′\{\\bm\{W\}\}^\{\\prime\}that minimizes the reconstruction error, expressed asmin𝑾′​‖𝑾−𝑾′‖2\\text\{min\}\_\{\{\\bm\{W\}\}^\{\\prime\}\}\|\|\{\\bm\{W\}\}\-\{\\bm\{W\}\}^\{\\prime\}\|\|\_\{2\}, the FWSVD objective is given bymin𝑾′​‖𝑰^w∘\(𝑾−𝑾′\)‖F\\text\{min\}\_\{\{\\bm\{W\}\}^\{\\prime\}\}\|\|\\hat\{\{\\bm\{I\}\}\}\_\{w\}\\circ\(\{\\bm\{W\}\}\-\{\\bm\{W\}\}^\{\\prime\}\)\|\|\_\{F\}with𝑰^w\\hat\{\{\\bm\{I\}\}\}\_\{w\}a matrix with elementsI^wi​j\\hat\{I\}\_\{w\_\{ij\}\}and∘\\circdenoting the Hadamard product\. However, this optimization problem does not have a closed\-form solution\. For this reason, the authors of\[fwsvd\]proposed an approximation:min𝑾′​‖𝑰^​𝑾−𝑰^​𝑾′‖F\\text\{min\}\_\{\{\\bm\{W\}\}^\{\\prime\}\}\|\|\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\-\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}^\{\\prime\}\|\|\_\{F\}, with𝑰^\\hat\{\{\\bm\{I\}\}\}a diagonal matrix whose diagonal elementI^i\\hat\{I\}\_\{i\}in rowiiis equal toI^i=∑jI^wi​j\\hat\{I\}\_\{i\}=\\sqrt\{\\sum\_\{j\}\\hat\{I\}\_\{w\_\{ij\}\}\}\. In this way, each neuroniiis assigned an importance weightI^i\\hat\{I\}\_\{i\}based on the Fisher information that the dataset provides regarding its parameters\. The approximation holds due to the fact that whenever all elements in a row of𝑾\{\\bm\{W\}\}share the same importance, the Hadamard product can be written as a standard matrix product with a diagonal matrix\.

### III\-FSVD\-LLMv2

The most prominent SVD\-based work is SVD\-LLMv2, which combines the concepts of data\-aware low\-rank approximation with a computationally efficient and effective rank allocation algorithm, an algorithm that assigns ranks to each one of the approximated matrices under a predefined parameter budget\. Instead of minimizing the error on the weight matrix \(as SVD and FWSVD do\), the optimization objective of SVD\-LLMv2 is defined on the layer’s output pre\-activations, given bymin𝑾′​‖𝑾​𝑿−𝑾′​𝑿‖2\\text\{min\}\_\{\{\\bm\{W\}\}^\{\\prime\}\}\|\|\{\\bm\{W\}\}\{\\bm\{X\}\}\-\{\\bm\{W\}\}^\{\\prime\}\{\\bm\{X\}\}\|\|\_\{2\}, with𝑿\{\\bm\{X\}\}denoting a matrix of token representations to be transformed by the layer’s matrix𝑾\{\\bm\{W\}\}\. The SVD\-LLMv2 approach, by focusing on the layer’s functional behavior, has been shown to better preserve task performance\.

Additionally, the algorithm for matrix approximation that was introduced by SVD\-LLMv2 ensures that the error in the layer’s output pre\-activations can be directly predicted from the truncated singular values\. This is a significant improvement over previous approaches\[asvd,drone\], which suffer from a sharp drop in performance when truncating the smallest singular values due to the fact that they are not directly related to the truncation error of the layer’s pre\-activations\.

Rather than applying a uniform compression ratio to every layer, which can be inefficient given that the sensitivity of the network with respect to different layers may vary, SVD\-LLMv2 considers a heterogeneous allocation\. It first groups the weights by their type, regardless of transformer layer index \(e\.g𝑾K,𝑾Q,𝑾V,𝑾a,𝑾u,𝑾d\{\\bm\{W\}\}\_\{K\},\{\\bm\{W\}\}\_\{Q\},\{\\bm\{W\}\}\_\{V\},\{\\bm\{W\}\}\_\{a\},\{\\bm\{W\}\}\_\{u\},\{\\bm\{W\}\}\_\{d\}, etc\.\)\. Then it assigns a target rank to each of the weight matrices in the group by using the error in the layer’s pre\-activations under uniform rank allocation as a proxy for the layer’s sensitivity with respect to rank reduction\. This approach considers the matrix sensitivity to rank reduction from a functional perspective, resulting in a better trade\-off between model size and performance preservation\. What makes this grouping strategy efficient is that matrices within the same group share the same target parameter sub\-budget, i\.e\. a parameter sub\-budget is defined for each group based on the overall target compression ratio, and subsequently this sub\-budget is distributed across matrices within the group\.

## IVMotivation

### IV\-ACombining Neuron Importance with Data\-Aware Low Rank Approximation

Summarizing the discussion of the previous Section, to compute the low\-rank approximation of a weight matrix, on the one hand, FWSVD considers the effect of each parameter on the task loss, which we refer to asparameter importance\(PI\)\. On the other hand, SVD\-LLMv2 considers the effect of the parameters on the layer’s output pre\-activations, aiming for a functional equivalence of the layer before and after the compression\. In this work, we are inspired by both methods in order to explore an approach that combines ideas from both perspectives\. In particular, instead of minimizingmin𝑾′​‖𝑾​𝑿−𝑾′​𝑿‖F\\min\_\{\{\\bm\{W\}\}^\{\\prime\}\}\|\|\{\\bm\{W\}\}\{\\bm\{X\}\}\-\{\\bm\{W\}\}^\{\\prime\}\{\\bm\{X\}\}\|\|\_\{F\}aiming for low reconstruction error on the layer’s output, i\.e\. the functional equivalence objective of SVD\-LLMv2, we also consider the effect of the layer’s output pre\-activation on the task loss\. This leads to the following optimization objectivemin𝑾′​‖𝑰^y∘\(𝑾​𝑿−𝑾′​𝑿\)‖F\\min\_\{\{\\bm\{W\}\}^\{\\prime\}\}\|\|\\hat\{\{\\bm\{I\}\}\}\_\{y\}\\circ\(\{\\bm\{W\}\}\{\\bm\{X\}\}\-\{\\bm\{W\}\}^\{\\prime\}\{\\bm\{X\}\}\)\|\|\_\{F\}, with𝑰^y\\hat\{\{\\bm\{I\}\}\}\_\{y\}denoting a matrix containing the importance of each neuron’s output to the task loss\.

Solving this optimization problem faces the same limitation as the original formulation of FWSVD, i\.e\. a lack of closed\-form solution\. For this reason, we also approximate the Hadamard product by considering standard multiplication with a diagonal matrix𝑰^\\hat\{\{\\bm\{I\}\}\}whose diagonal element in rowiiis equal to the overall importance of theii\-th neuron to the task loss\. The latter is computed by aggregating the neuron’s importance over several samples in a calibration dataset\. Thus, in our approach, we find𝑾′\{\\bm\{W\}\}^\{\\prime\}that minimizes‖𝑰^​𝑾​𝑿−𝑰^​𝑾′​𝑿‖F\|\|\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{X\}\}\-\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}^\{\\prime\}\{\\bm\{X\}\}\|\|\_\{F\}\. A first intuitive choice for computing the diagonal elements of𝑰^\\hat\{\{\\bm\{I\}\}\}is to aggregateparameter importancesfor each neuron, as it is done in FWSVD\. However, in the experiments, we provide statistically significant evidence that this is suboptimal, leading to compressed networks that perform worse than when using data\-aware low\-rank approximation alone\. As we discuss in Section[V\-A](https://arxiv.org/html/2607.18284#S5.SS1), we were able to improve on that by considering a different strategy\. This strategy is based on adirect wayto measureneuron importance\(NI\) that does not rely on theparameter importancepreviously considered by FWSVD\.

### IV\-BDynamic Rank Allocation Across Matrices

Each target compression ratio \(C​RCR\) can be directly translated to a total parameter budget under which the compressed LLM should fit\. In low\-rank approximation, the number of parameters in a layer is directly controlled by the rank of the approximation\. Thus, distributing the total available parameters to individual layers can be accomplished by deciding on the rank of the approximation for each weight matrix\. Performingrank allocation, that is, distributing target matrix ranks under a fixed parameter budget to individual layers, is a combinatorial and practically intractable problem, as optimal allocation relies on exhaustive search\.

Previous approaches either resorted to proportionally equivalent reduction of parameters across layers\[fwsvd,svd\-llm\]\(which we call uniform allocation\), or based their decision on algorithms that were inspired by studies in a reduced search space\. For instance,\[asvd\]found that optimal rank allocation varies both with layer depth and the layer’s functional type \(i\.e\. if it corresponds to𝑾Q,𝑾V,𝑾a,𝑾u\{\\bm\{W\}\}\_\{Q\},\{\\bm\{W\}\}\_\{V\},\{\\bm\{W\}\}\_\{a\},\{\\bm\{W\}\}\_\{u\}, etc\.\)\. Meanwhile,\[drone\]found that compression distortions from lower layers may result in progressively accumulative errors towards the latter layers, implying that rank allocation should vary with depth and compressing lower layers should be more conservative compared to higher ones\. Additionally,\[svd\-llmv2\]focused on allocating ranks for layers in the same functional group, exploiting the fact that the matrices in the same group share the same size\. Despite the reduction in the search space, the rank allocation algorithm of\[drone\]is still computationally expensive and is driven by model performance instead of parameter budget\. Even though the rank allocation algorithm of\[svd\-llmv2\]is computationally efficient, adapting it to a different grouping strategy requires addressing the challenge of matrix size variability among matrices in the same group\. Our approach addresses these challenges by building on the findings of\[drone\], while adapting the computationally efficient algorithm of\[svd\-llmv2\]to group layers by their depth index instead of their functional role\.

## VProposed Method

Our proposed hybrid compression methodology is a post\-training, SVD\-based pipeline that fuses key concepts from recent advancements to achieve more efficient and effective compression\. As shown in Figure 1, our approach synthesizes data\-aware low\-rank approximation with the concept of Fisher\-weighted neuron importance\. We also introduce a heterogeneous rank allocation that improves upon previous methods by adapting compression on a per\-layer basis\. The following subsections detail the core components of our pipeline, outlining our complete process from the computation of neuron importance to the final data\-aware low\-rank approximation and rank allocation algorithm\.

### V\-ANeuron Importance Estimation

We propose to estimate neuron importance by computing the amount of Fisher Information that a dataset provides about the neuron’soutput pre\-activation,instead of the neuron’s parameters\. Following the notation of Section[IV\-A](https://arxiv.org/html/2607.18284#S4.SS1), we calculate the element of𝑰^y\\hat\{\{\\bm\{I\}\}\}\_\{y\}at the position\(i,j\)\(i,j\)for thekk\-th sample of the datasetdkd\_\{k\}as:

I^yi​j=\(∂∂yi​j​L​\(yi​j​\(dk\)\)\)2\\hat\{I\}\_\{y\_\{ij\}\}=\\left\(\\frac\{\\partial\}\{\\partial y\_\{ij\}\}L\(y\_\{ij\}\(d\_\{k\}\)\)\\right\)^\{2\}\(2\)withLLthe task loss function andyi​jy\_\{ij\}a spatial element in the matrix of layer outputs:𝒀=𝑾T​𝑿\{\\bm\{Y\}\}=\{\\bm\{W\}\}^\{T\}\{\\bm\{X\}\}\. We subsequently form the neuron importance matrix𝑰^\\hat\{\{\\bm\{I\}\}\}, a diagonal matrix with its element in rowiibeing equal to the importance of neuronii:

I^i=1\|D\|​∑k=1\|D\|𝔼j​\[I^yi​j\|dk\]\\hat\{I\}\_\{i\}=\\sqrt\{\\frac\{1\}\{\|D\|\}\\sum\_\{k=1\}^\{\|D\|\}\\mathbb\{E\}\_\{j\}\\big\[\\hat\{I\}\_\{y\_\{ij\}\}\|d\_\{k\}\\big\]\}\(3\)

### V\-BData\-Aware Low Rank Approximation driven by Neuron Importance

We adapt the weight truncation algorithm of SVD\-LLMv2 by integrating the neuron importance matrix𝑰^\\hat\{\{\\bm\{I\}\}\}of the previous section into the process of low\-rank approximation\. Algorithm[1](https://arxiv.org/html/2607.18284#alg1)summarizes the process of decomposing a matrix𝑾\{\\bm\{W\}\}to its low\-rank approximation𝑾′=𝑨​𝑩\{\\bm\{W\}\}^\{\\prime\}=\{\\bm\{A\}\}\{\\bm\{B\}\}and is a generalization over the algorithm proposed by SVD\-LLMv2, with the two algorithms being identical whenever the neuron importance matrix is equal to the identity matrix\.

Algorithm 1Weight Truncation Algorithm1:Input:

𝑾\{\\bm\{W\}\}: Original weight matrix

2:

𝑿\{\\bm\{X\}\}: Matrix of layer input activations

3:

𝑰^\\hat\{\{\\bm\{I\}\}\}: Diagonal matrix of neuron importances

4:

RR: Target rank for the low\-rank approximation

5:Output:

𝑨,𝑩,𝑾′\{\\bm\{A\}\},\{\\bm\{B\}\},\{\\bm\{W\}\}^\{\\prime\}: Low rank approximation factors

𝑨,𝑩\{\\bm\{A\}\},\{\\bm\{B\}\}and compressed weight matrix

𝑾′\{\\bm\{W\}\}^\{\\prime\}\.

6:procedureWeight\_Truncation\(

𝑾,𝑿,𝑰,R\{\\bm\{W\}\},\{\\bm\{X\}\},\{\\bm\{I\}\},R\)

7:

𝑺←𝑿​𝑿T\{\\bm\{S\}\}\\leftarrow\{\\bm\{X\}\}\{\\bm\{X\}\}^\{T\}⊳\\trianglerightConstruct matrixSSfromXX

8:

𝑼s,𝑺s,𝑽s←SVD​\(𝑺\)\{\\bm\{U\}\}\_\{s\},\{\\bm\{S\}\}\_\{s\},\{\\bm\{V\}\}\_\{s\}\\leftarrow\\bm\{\\textrm\{SVD\}\}\(\{\\bm\{S\}\}\)⊳\\trianglerightPerform SVD on matrixSS

9:

𝑫←𝑰^​𝑾​𝑼s​𝑺s\{\\bm\{D\}\}\\leftarrow\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{U\}\}\_\{s\}\\sqrt\{\{\\bm\{S\}\}\_\{s\}\}⊳\\trianglerightConstruct matrixDD

10:

𝑼w​s,𝑺w​s,𝑽w​s←SVD​\(𝑫\)\{\\bm\{U\}\}\_\{ws\},\{\\bm\{S\}\}\_\{ws\},\{\\bm\{V\}\}\_\{ws\}\\leftarrow\\bm\{\\textrm\{SVD\}\}\(\{\\bm\{D\}\}\)⊳\\trianglerightPerform SVD on matrix𝑫\{\\bm\{D\}\}

11:

𝑼w​st,𝑺w​st,𝑽w​st←Truncate​\(𝑼w​s,𝑺w​s,𝑽w​s,R\)\{\\bm\{U\}\}\_\{ws\}^\{t\},\{\\bm\{S\}\}\_\{ws\}^\{t\},\{\\bm\{V\}\}\_\{ws\}^\{t\}\\leftarrow\\bm\{\\textrm\{Truncate\}\}\(\{\\bm\{U\}\}\_\{ws\},\{\\bm\{S\}\}\_\{ws\},\{\\bm\{V\}\}\_\{ws\},R\)⊳\\trianglerightPerform SVD truncation based on target rank approximationRR

12:

𝑨←𝑰^−1​𝑼w​st​𝑺w​st\{\\bm\{A\}\}\\leftarrow\\hat\{\{\\bm\{I\}\}\}^\{\-1\}\{\\bm\{U\}\}\_\{ws\}^\{t\}\\sqrt\{\{\\bm\{S\}\}\_\{ws\}^\{t\}\}⊳\\trianglerightThe first matrix of low\-rank approximation

13:

𝑩←𝑺w​st​𝑽w​st​𝑺s−1​𝑼s−1\{\\bm\{B\}\}\\leftarrow\\sqrt\{\{\\bm\{S\}\}\_\{ws\}^\{t\}\}\{\\bm\{V\}\}\_\{ws\}^\{t\}\\sqrt\{\{\\bm\{S\}\}\_\{s\}\}^\{\-1\}\{\\bm\{U\}\}\_\{s\}^\{\-1\}⊳\\trianglerightThe second matrix of low\-rank approximation

14:

𝑾′←𝑨​𝑩\{\\bm\{W\}\}^\{\\prime\}\\leftarrow\{\\bm\{A\}\}\{\\bm\{B\}\}⊳\\trianglerightReconstructed low\-rank approximation𝑾′\{\\bm\{W\}\}^\{\\prime\}

15:return

𝑨,𝑩,𝑾′\{\\bm\{A\}\},\{\\bm\{B\}\},\{\\bm\{W\}\}^\{\\prime\}
16:endprocedure

LetL=‖𝑰^​𝑾​𝑿−𝑰^​𝑾′​𝑿‖FL=\|\|\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{X\}\}\-\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}^\{\\prime\}\{\\bm\{X\}\}\|\|\_\{F\}denote the weighted compression loss when approximating𝑾\{\\bm\{W\}\}with a low rank matrix𝑾′\{\\bm\{W\}\}^\{\\prime\}\. For any given rank of the approximation, the theoretical minimum loss is given by the error induced by the truncated SVD decomposition of𝑰^​𝑾​𝑿\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{X\}\}, denoted as‖𝐒𝐕𝐃​\(𝑰^​𝑾​𝑿\)‖F\|\|\\mathbf\{SVD\}\(\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{X\}\}\)\|\|\_\{F\}\.

###### Theorem 1\.

If𝐔s,𝐒s,𝐕s\{\\bm\{U\}\}\_\{s\},\{\\bm\{S\}\}\_\{s\},\{\\bm\{V\}\}\_\{s\}are obtained by the SVD decomposition of𝐗​𝐗T\{\\bm\{X\}\}\{\\bm\{X\}\}^\{T\}and𝐔w​st,𝐒w​st,𝐕w​st\{\\bm\{U\}\}\_\{ws\}^\{t\},\{\\bm\{S\}\}\_\{ws\}^\{t\},\{\\bm\{V\}\}\_\{ws\}^\{t\}are obtained by the truncated SVD decomposition of𝐈^​𝐖​𝐔s​𝐒s\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{U\}\}\_\{s\}\\sqrt\{\\bm\{S\}\}\_\{s\}, the compressed weight matrix𝐖′=𝐈^−1​𝐔w​st​𝐒w​st​𝐕w​st​𝐒s−1​𝐔s−1\{\\bm\{W\}\}^\{\\prime\}=\\hat\{\{\\bm\{I\}\}\}^\{\-1\}\{\\bm\{U\}\}\_\{ws\}^\{t\}\{\\bm\{S\}\}\_\{ws\}^\{t\}\{\\bm\{V\}\}\_\{ws\}^\{t\}\\sqrt\{\{\\bm\{S\}\}\_\{s\}\}^\{\-1\}\{\\bm\{U\}\}\_\{s\}^\{\-1\}minimizes the weighted compression lossLL\.

###### Proof\.

Let𝑼x,𝑺x,𝑽x\{\\bm\{U\}\}\_\{x\},\{\\bm\{S\}\}\_\{x\},\{\\bm\{V\}\}\_\{x\}denote the matrices from the SVD decomposition of𝑿\{\\bm\{X\}\}\. Since𝑿​𝑿T\{\\bm\{X\}\}\{\\bm\{X\}\}^\{T\}is symmetric,𝑼s=𝑽s\{\\bm\{U\}\}\_\{s\}=\{\\bm\{V\}\}\_\{s\}\. Moroever,𝑼x=𝑼s\{\\bm\{U\}\}\_\{x\}=\{\\bm\{U\}\}\_\{s\}and𝑺x=𝑺s\{\\bm\{S\}\}\_\{x\}=\\sqrt\{\{\\bm\{S\}\}\_\{s\}\}\. Let also𝑪=𝑼s​𝑺s\{\\bm\{C\}\}=\{\\bm\{U\}\}\_\{s\}\\sqrt\{\{\\bm\{S\}\}\_\{s\}\}which implies that𝑪−1=𝑺s−1​𝑼s−1\{\\bm\{C\}\}^\{\-1\}=\\sqrt\{\{\\bm\{S\}\}\_\{s\}\}^\{\-1\}\{\\bm\{U\}\}\_\{s\}^\{\-1\}\. It is easy to show that𝑪−1​𝑿=𝑽x\{\\bm\{C\}\}^\{\-1\}\{\\bm\{X\}\}=\{\\bm\{V\}\}\_\{x\}which is orthogonal and thus does not affect the Frobenius norm under matrix multiplication\. For the proposed choice of𝑾′\{\\bm\{W\}\}^\{\\prime\}the weighted compression lossLLbecomes:

L\\displaystyle L=‖𝑰^​𝑾​𝑿−𝑰^​𝑾′​𝑿‖F\\displaystyle=\|\|\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{X\}\}\-\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}^\{\\prime\}\{\\bm\{X\}\}\|\|\_\{F\}=‖𝑰^​𝑾​𝑿−𝑰^​𝑰^−1​𝑼w​st​𝑺w​st​𝑽w​st​𝑺s−1​𝑼s−1​𝑿‖F\\displaystyle=\|\|\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{X\}\}\-\\hat\{\{\\bm\{I\}\}\}\\hat\{\{\\bm\{I\}\}\}^\{\-1\}\{\\bm\{U\}\}\_\{ws\}^\{t\}\{\\bm\{S\}\}\_\{ws\}^\{t\}\{\\bm\{V\}\}\_\{ws\}^\{t\}\\sqrt\{\{\\bm\{S\}\}\_\{s\}\}^\{\-1\}\{\\bm\{U\}\}\_\{s\}^\{\-1\}\{\\bm\{X\}\}\|\|\_\{F\}=‖𝑰^​𝑾​𝑪​𝑪−1​𝑿−𝑼w​st​𝑺w​st​𝑽w​st​𝑪−1​𝑿‖F\\displaystyle=\|\|\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{C\}\}\{\\bm\{C\}\}^\{\-1\}\{\\bm\{X\}\}\-\{\\bm\{U\}\}\_\{ws\}^\{t\}\{\\bm\{S\}\}\_\{ws\}^\{t\}\{\\bm\{V\}\}\_\{ws\}^\{t\}\{\\bm\{C\}\}^\{\-1\}\{\\bm\{X\}\}\|\|\_\{F\}=‖\(𝑰^​𝑾​𝑪−𝑼w​st​𝑺w​st​𝑽w​st\)​𝑪−1​𝑿‖F\\displaystyle=\|\|\(\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{C\}\}\-\{\\bm\{U\}\}\_\{ws\}^\{t\}\{\\bm\{S\}\}\_\{ws\}^\{t\}\{\\bm\{V\}\}\_\{ws\}^\{t\}\)\{\\bm\{C\}\}^\{\-1\}\{\\bm\{X\}\}\|\|\_\{F\}=‖\(𝑰^​𝑾​𝑪−𝑼w​st​𝑺w​st​𝑽w​st\)‖F\\displaystyle=\|\|\(\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{C\}\}\-\{\\bm\{U\}\}\_\{ws\}^\{t\}\{\\bm\{S\}\}\_\{ws\}^\{t\}\{\\bm\{V\}\}\_\{ws\}^\{t\}\)\|\|\_\{F\}=‖𝐒𝐕𝐃​\(𝑰^​𝑾​𝑪\)‖F\\displaystyle=\|\|\\mathbf\{SVD\}\(\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{C\}\}\)\|\|\_\{F\}=‖𝐒𝐕𝐃​\(𝑰^​𝑾​𝑼s​𝑺s\)‖F\\displaystyle=\|\|\\mathbf\{SVD\}\(\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{U\}\}\_\{s\}\\sqrt\{\{\\bm\{S\}\}\_\{s\}\}\)\|\|\_\{F\}=‖𝐒𝐕𝐃​\(𝑰^​𝑾​𝑼x​𝑺x\)‖F\\displaystyle=\|\|\\mathbf\{SVD\}\(\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{U\}\}\_\{x\}\{\\bm\{S\}\}\_\{x\}\)\|\|\_\{F\}=‖𝐒𝐕𝐃​\(𝑰^​𝑾​𝑼x​𝑺x​𝑽x\)‖F\\displaystyle=\|\|\\mathbf\{SVD\}\(\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{U\}\}\_\{x\}\{\\bm\{S\}\}\_\{x\}\{\\bm\{V\}\}\_\{x\}\)\|\|\_\{F\}=‖𝐒𝐕𝐃​\(𝑰^​𝑾​𝑿\)‖F\\displaystyle=\|\|\\mathbf\{SVD\}\(\\hat\{\{\\bm\{I\}\}\}\{\\bm\{W\}\}\{\\bm\{X\}\}\)\|\|\_\{F\}which is equal to the theoretical minimum\. ∎

Algorithm[1](https://arxiv.org/html/2607.18284#alg1)computes𝑾′\{\\bm\{W\}\}^\{\\prime\}based on Theorem[1](https://arxiv.org/html/2607.18284#Thmtheorem1)and suggests a low rank matrix approximation that minimizes the weighted compression loss\.

### V\-CDynamic Rank Allocation

Following Section[III\-F](https://arxiv.org/html/2607.18284#S3.SS6), and based on the findings of\[drone\]we propose an adaptation of the rank allocation algorithm of\[svd\-llmv2\]by changing the grouping strategy of the weight matrices to be based on transformer layer index \(i\.e\. layer’s depth in the transformer’s sequence of layers\)\. This change in the grouping implies that a target parameter sub\-budget is defined for all matrices within a transformer layer, instead of matrices with the same functional role\. The algorithm still distributes this sub\-budget to each matrix in the group based on matrix sensitivity to rank reduction, as it was proposed in\[svd\-llmv2\]\.

Our layer index\-based rank allocation method groups all weight types \(e\.g\.𝑾Q\{\\bm\{W\}\}\_\{Q\},𝑾K\{\\bm\{W\}\}\_\{K\},𝑾V\{\\bm\{W\}\}\_\{V\},𝑾a\{\\bm\{W\}\}\_\{a\},𝑾u\{\\bm\{W\}\}\_\{u\}and𝑾d\{\\bm\{W\}\}\_\{d\}\) with the same transformer layer indexll, by treating each one of the transformer layers in the BERT\-family models as a distinct group\. This layer\-wise approach modifies the algorithm to derive a compression ratiorrfor each weight matrix and ensures iterative refinement for optimal balance\.

Algorithm 2Layer Index\-based Rank Allocation Algorithm1:Input:

MM: Original LLM

2:

𝑿\{\\bm\{X\}\}: Input activations

3:

RR: Effective target compression ratio

4:Output:

RdR\_\{d\}: Compression rank allocation per layer \(list of lists\)

5:procedureRank\_Allocation\(

M,𝑿,RM,\{\\bm\{X\}\},R\)

6:

G←Group​\(M\)G\\leftarrow\\bm\{\\textrm\{Group\}\}\(M\)⊳\\trianglerightGroup weights by layer index

7:

Rd←∅R\_\{d\}\\leftarrow\\emptyset⊳\\trianglerightInitialize the compression ratio list

8:for

ggin

GGdo

9:

LG←∅L\_\{G\}\\leftarrow\\emptyset⊳\\trianglerightInitialize the loss list in the group

10:for

𝑾\{\\bm\{W\}\}in

ggdo

11:

Lm​i​n←Theoretical\_Loss​\(𝑾,𝑿,R\)L\_\{min\}\\leftarrow\\bm\{\\textrm\{Theoretical\\\_Loss\}\}\(\{\\bm\{W\}\},\{\\bm\{X\}\},R\)⊳\\trianglerightCompute per weight theoretical loss

12:

LG←LG∪Lm​i​nL\_\{G\}\\leftarrow L\_\{G\}\\cup L\_\{min\}⊳\\trianglerightAppend loss to list

13:endfor

14:

LG←1/Log​\(LG\)L\_\{G\}\\leftarrow 1/\\bm\{\\textrm\{Log\}\}\(L\_\{G\}\)⊳\\trianglerightNormalizeLGL\_\{G\}

15:

l​a​y​e​r​\_​r​a​t​i​o​s←∅layer\\\_ratios\\leftarrow\\emptyset⊳\\trianglerightInitialize layer compression ratios list

16:for

Lm​i​nL\_\{min\}in

LGL\_\{G\}do

17:

r←Len​\(LG\)×R×Lm​i​n/Sum​\(LG\)r\\leftarrow\\bm\{\\textrm\{Len\}\}\(L\_\{G\}\)\\times R\\times L\_\{min\}/\\bm\{\\textrm\{Sum\}\}\(L\_\{G\}\)⊳\\trianglerightAllocate ratios proportionally to normalized loss

18:

l​a​y​e​r​\_​r​a​t​i​o​s←l​a​y​e​r​\_​r​a​t​i​o​s∪rlayer\\\_ratios\\leftarrow layer\\\_ratios\\cup r⊳\\trianglerightAppend ratio to list

19:endfor

20:

Rd←Rd∪l​a​y​e​r​\_​r​a​t​i​o​sR\_\{d\}\\leftarrow R\_\{d\}\\cup layer\\\_ratios⊳\\trianglerightAppendl​a​y​e​r​\_​r​a​t​i​o​slayer\\\_ratioslist toR​dRdlist

21:endfor

22:

Rd←Convert\_Ratios\_to\_Ranks​\(M,Rd\)R\_\{d\}\\leftarrow\\bm\{\\textrm\{Convert\\\_Ratios\\\_to\\\_Ranks\}\}\(M,R\_\{d\}\)⊳\\trianglerightConvert ratios to ranks for each weight

23:

t​o​t​a​l​\_​p​a​r​a​m​s←Count\_Total\_Parameters​\(M\)total\\\_params\\leftarrow\\bm\{\\textrm\{Count\\\_Total\\\_Parameters\}\}\(M\)⊳\\trianglerightCount all the parameters of the model

24:

c​o​m​p​\_​p​a​r​a​m​s←Count\_Compressed\_Parameters​\(M,Rd\)comp\\\_params\\leftarrow\\bm\{\\textrm\{Count\\\_Compressed\\\_Parameters\}\}\(M,R\_\{d\}\)⊳\\trianglerightCount all the parameters of the compressed model

25:

achieved\_ratio←1−comp\_params/total\_params\)achieved\\\_ratio\\leftarrow 1\-comp\\\_params/total\\\_params\)⊳\\trianglerightCompute achieved ratio after rank allocation

26:

i←0i\\leftarrow 0
27:while

\|a​c​h​i​e​v​e​d​\_​r​a​t​i​o−R\|\>ε\|achieved\\\_ratio\-R\|\>\\varepsilondo

28:

s​i​g​n←a​c​h​i​e​v​e​d​\_​r​a​t​i​o−Rsign\\leftarrow achieved\\\_ratio\-R
29:

s​c​a​l​e←\(1−R\)/\(1−a​c​h​i​e​v​e​d​\_​r​a​t​i​o\)scale\\leftarrow\(1\-R\)/\(1\-achieved\\\_ratio\)⊳\\trianglerightCompute scaling factor

30:for

l​a​y​e​r​\_​r​a​n​k​slayer\\\_ranksin

RdR\_\{d\}do

31:for

rrin

l​a​y​e​r​\_​r​a​n​k​slayer\\\_ranksdo

32:

r←r\+s​i​g​n⋅ir\\leftarrow r\+sign\\cdot i⊳\\trianglerightIteratively adjust rank based on error direction

33:

Rd←Update​\(Rd,r\)R\_\{d\}\\leftarrow\\bm\{\\textrm\{Update\}\}\(R\_\{d\},r\)⊳\\trianglerightUpdate rank allocation list

34:endfor

35:endfor

36:

i←i\+1i\\leftarrow i\+1
37:endwhile

38:return

RdR\_\{d\}
39:endprocedure

Algorithm[2](https://arxiv.org/html/2607.18284#alg2)summarizes our layer index\-based rank allocation strategy\. The algorithm requires as an input the original LLM, a representative set of input activations resulting from the calibration dataset, and the effective compression ratio\. The effective compression ratio is calculated to account for parameters non amenable to compression, possibly due to the high sensitivity of the network with respect to these parameters, while still aiming for a specific overall target compression ratio\. The algorithm first groups the model’s weights by their layer \(line 2\) and then computes a normalized theoretical loss \(lines 6\-10\), which serves as a proxy for that layer’s sensitivity to rank reduction\. For a given matrix𝑾\{\\bm\{W\}\}, the theoretical loss function first translates the target compression ratio \(RR\) into the corresponding retained SVD rank \(kk\) to compute its low\-rank approximation𝑾′\{\\bm\{W\}\}^\{\\prime\}\. It then uses𝑾′\{\\bm\{W\}\}^\{\\prime\}to calculate the error on the layer’s output pre\-activationsLmin=‖𝑾​𝑿−𝑾′​𝑿‖FL\_\{\\min\}=\\\|\{\\bm\{W\}\}\{\\bm\{X\}\}\-\{\\bm\{W\}\}^\{\\prime\}\{\\bm\{X\}\}\\\|\_\{F\}, indicated astheoretical loss\. Compression ratios are then distributed proportionally to these normalized errors \(lines 12\-15\), ensuring that layers deemed more sensitive to rank reduction receive higher ranks\. Line 18 performs the necessary transformation from the calculated proportional ratios \(RdR\_\{d\}\) into the specific, non\-zero integer SVD ranks \(kk\) required for each weight matrix𝑾\{\\bm\{W\}\}in the model \(MM\)\. To further meet the consistency with the global compression target, the iterative refinement loop \(lines 23\-34\) adjusts matrix ranks to ensure that the desired ratio is achieved\. The update rule \(lines 28\-29\) ensures convergence by increasing/decreasing matrix ranks whenever the compressed model is under/over the target parameter budget\. This dynamic adjustment makes the allocation procedure adaptive and computationally efficient, avoiding exhaustive search strategies\.

Overall, our layer index\-based rank allocation algorithm balances two objectives: \(i\) preserving the most critical layers through proportional loss\-aware rank allocation, and \(ii\) meeting the global compression constraint through iterative refinement\.

![Refer to caption](https://arxiv.org/html/2607.18284v1/sections/images/ranks-comparison/query_ranks_comparison.png)

\(a\)𝑾Q\{\\bm\{W\}\}\_\{Q\}

![Refer to caption](https://arxiv.org/html/2607.18284v1/sections/images/ranks-comparison/key_ranks_comparison.png)

\(b\)𝑾K\{\\bm\{W\}\}\_\{K\}

![Refer to caption](https://arxiv.org/html/2607.18284v1/sections/images/ranks-comparison/value_ranks_comparison.png)

\(c\)𝑾V\{\\bm\{W\}\}\_\{V\}

![Refer to caption](https://arxiv.org/html/2607.18284v1/sections/images/ranks-comparison/attention_ranks_comparison.png)

\(d\)𝑾a\{\\bm\{W\}\}\_\{a\}

![Refer to caption](https://arxiv.org/html/2607.18284v1/sections/images/ranks-comparison/up_ranks_comparison.png)

\(e\)𝑾u\{\\bm\{W\}\}\_\{u\}

![Refer to caption](https://arxiv.org/html/2607.18284v1/sections/images/ranks-comparison/down_ranks_comparison.png)

\(f\)𝑾d\{\\bm\{W\}\}\_\{d\}

Figure 2:Visualization of BERT model ranks under 30% compression on QNLI dataset, comparing SVD\-LLMv2 \(role\-based\), Uniform, and NIDA\-SVD strategies: \(a\) Query, \(b\) Key, \(c\) Value, \(d\) Attention, \(e\) Up, \(f\) Down\.![Refer to caption](https://arxiv.org/html/2607.18284v1/sections/images/ranks-comparison/down_ranks_comparison_all_ratios.png)

\(a\)𝑾d\{\\bm\{W\}\}\_\{d\}

![Refer to caption](https://arxiv.org/html/2607.18284v1/sections/images/ranks-comparison/value_ranks_comparison_all_ratios.png)

\(b\)𝑾V\{\\bm\{W\}\}\_\{V\}

Figure 3:Visualization of BERT model weight ranks across different compression ratios \(layer\-based strategy\) on the MRPC dataset: \(a\) Down, \(b\) Value\.

## VIExperimental Results

### VI\-AExperimental Setup

We evaluate the effectiveness of our method by compressing the standard 12\-layer BERT transformer architecture\. We consider BERT, fine\-tuned on 8 datasets of the GLUE\-Benchmark\[glue\-benchmark\], namelycola, mnli\-m, mnli\-mm, mrpc, qnli, qqp, sst\-2andsts\-b\. Each dataset corresponds to a different task, and the performance of the \(compressed\) models is evaluated under a task\-specific metric\. Single sentence tasks,colaandsst\-2are measured by Matthew’s correlation and classification accuracy, respectively\. The sentence similarity tasksmrpcandqqpare assessed by F\-1 score, whilests\-b, using Pearson\-Spearman correlation\. Finally, natural language inference tasksmnli\-m,mnli\-mmandqnliare measured by accuracy\.

For each one of the layers in the BERT architecture, the matrices that we consider for compression are the ones discussed in Section[III\-A](https://arxiv.org/html/2607.18284#S3.SS1), namely𝑾K,𝑾Q,𝑾V,𝑾a,𝑾u,𝑾d\{\\bm\{W\}\}\_\{K\},\{\\bm\{W\}\}\_\{Q\},\{\\bm\{W\}\}\_\{V\},\{\\bm\{W\}\}\_\{a\},\{\\bm\{W\}\}\_\{u\},\{\\bm\{W\}\}\_\{d\}\. We do not consider compressing the token embedding matrix, nor the matrices involved in the layer\-normalization blocks\. To compute the uncentered covariance matrix𝑺\{\\bm\{S\}\}in Algorithm[1](https://arxiv.org/html/2607.18284#alg1), we randomly sample a \(balanced\) dataset of 256 text samples from the dataset’s training split\. We also compute neuron importances \(NIs\) using the full training split of the dataset, as it is done in FWSVD for parameter importances \(PIs\)\. In contrast to FWSVD or SVD\-LLMv2 which additionally consider fine\-tuning the compressed models to recover performance, we emphasize results and comparisons without a fine\-tuning step\. Even though fine\-tuning may boost the results of any previous SVD\-based method, including ours, we want to highlight the benefits of the proposed approach, namely Neuron Importance driven Data\-Aware SVD \(NIDA\-SVD\), under a low computation budget which does not involve finetuning\. Nevertheless, we report results with fine\-tuning when compressing MobileBERT and TinyBERT\.

The experimental setup is extended to include the structurally similar DistilBERT, as well as the specialized MobileBERT and TinyBERT architectures\. Since these models are direct descendants of BERT, they are also fine\-tuned and evaluated on specific subsets of the GLUE Benchmark using the corresponding task\-specific metrics described above\. Specifically, the models are evaluated on the following subsets: DistilBERT onmnli\-mm,mrpc,qnli,qqp, andsst2, MobileBERT onmrpc,qnli,qqp,sst2, andstsb, and TinyBERT onmnli\-mm,mrpc,qnli,qqp,sst2, andstsb\. For all models, compression is applied to their Transformer encoder blocks and dense layers\. The methodology for computing the matrix𝑺\{\\bm\{S\}\}and the neuron importance𝑰\{\\bm\{I\}\}remains identical for all models\.

In the following subsections, we present experimental results comparing our method with the recent state\-of\-the\-art in low\-rank approximation: SVD, FWSVD, and SVD\-LLMv2, and conduct ablation studies under different strategies regarding parameter importance, neuron importance, and rank allocation\. Since the algorithm of SVD\-LLMv2 performs rank allocation based on the layer’s functional role in the transformer block, we call itrole\-based allocation\. Contrariwise, our algorithm approaches rank allocation from the perspective of the layer’s depth index, and thus we refer to it asindex\-based allocation\.

### VI\-BComparison against the state\-of\-the\-art

Table[I](https://arxiv.org/html/2607.18284#S6.T1)summarizes performance metrics for the BERT model compressed under different Compression Ratios \(CR\) with a uniform rank allocation strategy\. Overall, in 35 out of 40 cases \(87\.5%\), our method is ranked first; the compressed model retains most of its task performance when compressed with our method\. In the rest of the cases, our method is ranked second, remaining competent to the top\-performing method, which is always SVD\-LLMv2\. We emphasize the improvement that our method offers compared to the previous state\-of\-the\-art, especially in the highest compression rate \(0\.5\): \+3\.15%absolute improvementincola, \+4\.16% inmnli\-m, \+4\.82% inmnli\-mm, \+1\.54% inmrpc, \+2\.48% inqnli, \+2\.35% inqqp, and \+2\.33% instsb\.

TABLE I:Compressed model performance metrics for different compression ratios \(CR\) on the BERT base model under uniform rank allocation\. In 35 out of 40 cases \(87\.5%\), our method \(NIDA\-SVD\) achieves state\-of\-the\-art performance\. The best results are inbold, while the second\-best areunderlined\.Similarly, Table[II](https://arxiv.org/html/2607.18284#S6.T2)summarizes performance metrics for the BERT model compressed either with our method or the previous state\-of\-the\-art, SVD\-LLMv2\. In this comparison, we consider each method to use its own rank allocation algorithm:role\-basedallocation for SVD\-LLMv2 andindex\-basedallocation for the proposed method\. For the extended experiments presented below, Table 3 \(DistilBERT\), Table 4 \(MobileBERT\), and Table 5 \(TinyBERT\), each utilized its own rank allocation strategy \(role\-based for SVD\-LLMv2 and index\-based for NIDA\-SVD\)\. For BERT, In 35 out of 40 cases \(87\.5%\), our method outperforms SVD\-LLMv2 while remaining comparable to SVD\-LLMv2 in the remaining 5\. In this case, under the highest compression ratio \(0\.5\), our method makes anabsolute improvementof \+4\.41% incola, \+4\.94% inmnli\-m, \+5\.01% inmnli\-mm, \+1\.82% inmrpc, \+4\.14% inqnli, \+1\.92% inqqpand \+6\.41% instsb\.

TABLE II:Comparison of our method \(NIDA\-SVD\) against the previous state\-of\-the\-art \(SVD\-LLMv2\) for BERT base model under different compression ratios \(CR\)\. Each method is complemented with its own rank allocation algorithm\. In 34 out of 40 cases \(85%\), our method outperforms SVD\-LLMv2, especially in high compression ratios\. The best results are inbold\.Table[III](https://arxiv.org/html/2607.18284#S6.T3)presents the DistilBERT results, comparing our NIDA\-SVD against the SVD\-LLMv2\. Our method shows a clear performance advantage, achieving the best result in 23 out of 25 comparisons \(92%\)\. This advantage is especially pronounced at the compression ratio of 0\.3, where NIDA\-SVD delivers significant absolute improvements, including \+0\.90% inmnli\-mmand \+2\.43% inqnli\.

TABLE III:Comparison of our method \(NIDA\-SVD\) against the previous state\-of\-the\-art \(SVD\-LLMv2\) for DistilBERT base model under different compression ratios \(CR\)\. Each method is complemented with its own rank allocation algorithm\. In 23 out of 25 cases \(92%\), our method outperforms SVD\-LLMv2, especially in high compression ratios\. The best results are inbold\.The MobileBERT results, detailed in Table[IV](https://arxiv.org/html/2607.18284#S6.T4), compare our NIDA\-SVD approach against the SVD\-LLMv2\. Our methodology demonstrates a clear performance advantage in 21 of the 25 total comparisons \(84%\)\. This general advantage is mostly observed across 0\.3 compression ratio, includingqqp\(\+3\.68%\),stsb\(\+2\.03%\), andsst2\(\+1\.03%\)\.

TABLE IV:Comparison of our method \(NIDA\-SVD\) against the previous state\-of\-the\-art \(SVD\-LLMv2\) for MobileBERT base model under different compression ratios \(CR\)\. Each method is complemented with its own rank allocation algorithm\. In 22 out of 25 cases \(88%\), our method outperforms SVD\-LLMv2, especially in high compression ratios\. The best results are inbold\.The compression performance on the highly reduced TinyBERT architecture is documented in Table[V](https://arxiv.org/html/2607.18284#S6.T5)\. Comparing NIDA\-SVD against the SVD\-LLMv2, NIDA\-SVD exhibits a robust performance advantage, securing the superior metric in 17 out of 18 total comparisons \(94\.4%\)\. The most substantial performance gains are concentrated at the compression ratio of 0\.2\. At this level, NIDA\-SVD registers notable absolute increases, specifically \+1\.43% inqnli, \+0\.98% inmnli\-mm, and \+0\.54% inqqp, confirming the efficacy of the data\-aware approach even on extremely compact models\. Note that TinyBERT, being a highly distilled and compact model \(14\.3M parameters\), was only compressed up to a compression ratio of 0\.3 \(70% parameter reduction\), as pushing for a higher compression ratio would likely lead to catastrophic performance collapse due to its limited architectural redundancy\.

TABLE V:Comparison of our method \(NIDA\-SVD\) against the previous state\-of\-the\-art \(SVD\-LLMv2\) for TinyBERT model under different compression ratios \(CR\)\. Each method is complemented with its own rank allocation algorithm\. In 17 out of 18 cases \(94%\), our method outperforms SVD\-LLMv2, especially in high compression ratios\. The best results are inbold\.While the primary focus of this work is post\-training compression, it is informative to examine how well the performance of highly compressed models is recovered by subsequent fine\-tuning\. Table[VI](https://arxiv.org/html/2607.18284#S6.T6)presents the results after a single fine\-tuning epoch on both MobileBERT and TinyBERT, compressed at 0\.5 and 0\.3 compression ratio, respectively\. Even after introducing the fine\-tuning step, NIDA\-SVD generally maintains better results than SVD\-LLMv2\. For MobileBERT, NIDA\-SVD registers meaningful gains inqnliandsst2\(both achieving an approximate \+1\.26% improvement over SVD\-LLMv2\)\. Similarly, on TinyBERT, NIDA\-SVD captures an advantage in the inference tasks \(mrpcandqnli\), with theqnlimetric recovering \+0\.92% better than the baseline\. Overall, across the six task\-model pairs evaluated, the NIDA\-SVD compressed models yield better accuracy in four instances, underscoring the robustness of our data\-aware, importance\-driven initialization\.

TABLE VI:Single\-epoch fine\-tuning comparison of NIDA\-SVD and SVD\-LLMv2 on MobileBERT at 0\.5 compression ration and TinyBERT at 0\.3 compression ratio\. The results \(F1/Accuracy on MRPC, QNLI, SST\-2\) show that fine\-tuning after NIDA\-SVD compression achieves better accuracy in 4 out of 6 cases \(66\.6%\), demonstrating robust performance despite the substantial compression level\.
### VI\-CAblation Studies

TABLE VII:Ablation Study under different compression ratios \(CR\): NIDA\-SVD \(ours\) using Parameter Importance \(PI\) as suggested in FWSVD, and our proposed Neuron Importance \(NI\)\. In 38 out of 40 cases \(95%\), NI is more effective than PI and in 35 \(87\.5%\) it improves upon SVD\-LLMv2\. In 26 out of 40 cases \(65%\), using PI makes the results worse than not using it at all \(NIDA\-SVD \(PI\) vs SVD\-LLMv2\)\. The best results are inbold, while the second\-best areunderlined\.TABLE VIII:Ablation Study: Our index\-based rank allocation algorithm improves vanilla SVD\-LLMv2 compared to other allocation strategies in 27 out of 40 cases \(67\.5%\)\. The best results are inbold, while the second\-best areunderlined\.TABLE IX:Ablation Study: Our index\-based rank allocation algorithm improves NIDA\-SVD compared to other allocation strategies in 28 out of 40 cases \(70%\)\. The best results are inbold, while the second\-best areunderlined\.In Table[VII](https://arxiv.org/html/2607.18284#S6.T7)we provide experimental evidence that under uniform rank allocation, naively combining FWSVD with SVD\-LLMv2 \(NIDA\-SVD \(PI\)\) in 26 out of the 40 cases \(more than half of them\), the compressed model’s task performance is inferior compared to when not using PI at all \(in which case the algorithm is equivalent to vanilla SVD\-LLMv2\)\. We note that NIDA\-SVD \(PI\) is equivalent to Algorithm[1](https://arxiv.org/html/2607.18284#alg1)but with𝑰^\\hat\{\{\\bm\{I\}\}\}computed as discussed in Section[III\-E](https://arxiv.org/html/2607.18284#S3.SS5), whereas in NIDA\-SVD \(NI\),𝑰^\\hat\{\{\\bm\{I\}\}\}is computed as discussed in Section[V\-A](https://arxiv.org/html/2607.18284#S5.SS1)\. Overall, in 38 out of 40 cases \(95%\), NI is more effective than PI, and in 35 it improves upon SVD\-LLMv2, while remaining competent in the rest of them\.

Tables[VIII](https://arxiv.org/html/2607.18284#S6.T8)and[IX](https://arxiv.org/html/2607.18284#S6.T9)summarize experimental results regarding the efficacy of our proposed index\-based rank allocation, compared to other strategies that were previously considered\. The experiments show that regardless of the compression algorithm being SVD\-LLMv2 or NIDA\-SVD, index\-based allocation is a better choice for the majority of the cases \(more than 67\.5% of them\)\. Furthermore, a row\-wise comparison of Tables[VIII](https://arxiv.org/html/2607.18284#S6.T8)and[IX](https://arxiv.org/html/2607.18284#S6.T9)reveals that, in most cases, NIDA\-SVD outperforms SVD\-LLMv2 under any rank allocation algorithm\. While for uniform rank allocation this was discussed in the previous section, for role\-based allocation, NIDA\-SVD outperforms SVD\-LLMv2 in 34 cases \(85%\) while for index\-based allocation in 33 \(82\.5%\)\. In many cases the improvement is substantial, especially at high compression ratios: by up to \+4\.54%absolute improvementin role\-based rank allocation and \+7\.97% for index\-based rank allocation\.

### VI\-DRank Allocation Analysis

The rank allocation analysis indicates that the index\-based strategy \(NIDA\-SVD\) offers advantages over the role\-based approach \(SVD\-LLMv2\), mainly because the role\-based method handles matrix ranks in a more uniform manner across all transformer layers \(especially in Figure[2](https://arxiv.org/html/2607.18284#S5.F2)\(a\),\(b\) and\(e\), resulting in a flatter rank profile that fails to capture the intrinsic heterogeneity in network’s sensitivity among layers\. This results in role\-based ranks being consistently closer to uniform values, reflecting a more uniform compression approach across the network\. In contrast, the index\-based strategy, as detailed in Section[V\-C](https://arxiv.org/html/2607.18284#S5.SS3)and Algorithm[2](https://arxiv.org/html/2607.18284#alg2), groups all compressible weight types by individual layer index, allowing for a more targeted distribution of ranks that adaptively varies according to the specific redundancy and importance of each layer\. This approach enables the index\-based method to exploit layer\-specific redundancies more effectively, as demonstrated in the plots of Figure[2](https://arxiv.org/html/2607.18284#S5.F2)\(a\)\-\(f\)\. Particularly in Figure[2](https://arxiv.org/html/2607.18284#S5.F2)\(e\), the role\-based ranks align much more closely with the uniform values when compared to the index\-based ranks\.

In Figure[2](https://arxiv.org/html/2607.18284#S5.F2)\(a\) and \(b\), the two images illustrate the rank allocations for the down projection and value weight matrices across the 12 layers of the BERT model under varying compression ratios \(0\.1 to 0\.5\) using the index\-based strategy\. For all compression ratios, the behavior of ranks within the same weight type follows a consistent pattern\. Specifically, for the𝑾d\{\\bm\{W\}\}\_\{d\}\(Figure[3](https://arxiv.org/html/2607.18284#S5.F3)\(a\)\), the earlier layers \(e\.g\. 0\-5\) are more sensitive to compression, requiring more parameters to maintain performance, in contrast to the later layers \(e\.g\. 6\-11\)\. Also, for the𝑾V\{\\bm\{W\}\}\_\{V\}\(Figure[3](https://arxiv.org/html/2607.18284#S5.F3)\(b\)\), the earlier layers are less sensitive to compression than the later layers\. As we can see, there are different patterns for different weight types across the layers\.

Generally, we cannot assume that a weight will exhibit a specific behavior in rank compression across layers, as the sensitivity to compression can vary depending on the weight type and its index within the model\. Our rank allocation algorithm identifies these distinct patterns by analyzing the intrinsic redundancy and importance of each layer, dynamically adjusting the ranks to optimize performance\.

### VI\-EMemory and Computational Analysis

TABLE X:Computational and Memory Analysis\. Comparison of MLFLOPS/token and total parameters \(in Millions\) for base and compressed DistilBERT, MobileBERT, and TinyBERT modelsThe primary motivation for model compression is to minimize memory and inference time, especially for deployment on resource\-constrained devices\. To quantify the advances achieved by our low\-rank approximation methods, we conducted a complexity analysis measuring both the memory requirements, represented by total parameters \(in millions\), and the computational cost, measured in MFLOPS/token\. This analysis conducted in DistilBERT and MobileBERT for comrpession ratio 0\.5, and TinyBERT for compression ratio 0\.3, comparing the uncompressed base models against the two SVD\-based \(SVD\-LLMv2 and NIDA\-SVD\) compression approaches\.

The results presented in Table[X](https://arxiv.org/html/2607.18284#S6.T10)clearly demonstrate the reduction in model size\. For DistilBERT, compression reduces the total parameters from 66\.9 million to approximately 33\.4 million, achieving the targeted 50 reduction\. Similarly the memory size of the MobileBERT reduced from 24\.5 million to about 12\.1 million\. This parameter reduction directly translated to a massive decrease in computational cost, as the MFLOPS/token for DistilBERT drops from 86 MFLOPS to just 19 MFLOPS, corresponding to a speedup factor of approximately 4\.5\. The most dramatic efficiency improvement is seen in TinyBERT, where a 30% parameter reduction \(from 14\.3M to 10\.0M\) results in a spectacular 90% reduction in computational cost, dropping from≈\\approx9 MFLOPS/token down to less than 1 MFLOP/token\.

Finaly, the comparison between our NIDA\-SVD method and the SVD\-LLMv2 method reveals that our importance\-aware rank allocation strategy does not impose a measurable computational overhead\. Both methods achieve nearly identical parameter counts and, consequently, highly similar MFLOPS/token values across all architectures\. This is an expected outcome, as the fundamental change to the network’s architecture \(replacing a dense matrix𝑾\{\\bm\{W\}\}with two smaller matrices𝑨\{\\bm\{A\}\}and𝑩\{\\bm\{B\}\}\) is the same for both approaches, regardless of how the compression rate is distributed, since the total computational budget for a fixed compression ratio remains constant\. This confirms that in the rank allocation proccess, the performance efficacy of NIDA\-SVD in contrast to other SVD\-based methods is achieved solely through a smarter distribution of ranks \(resources\) across layers, not by increasing the computational expense\.

## VIIConclusion

In summary, this work introduced NIDA\-SVD, a novel framework to compress LLM matrices that integrates neuron importance with data\-aware low\-rank approximation, alongside a computationally efficient algorithm for dynamic rank allocation\. Experimental results demonstrate that NIDA\-SVD consistently outperforms prior state\-of\-the\-art methods under diverse allocation strategies, while our proposed rank allocation algorithm also independently strengthens both NIDA\-SVD and previous approaches\. Together, these contributions establish a robust and versatile compression solution, achieving substantial performance gains, especially at high compression ratios, paving the way for a more effective deployment of large\-scale language models in resource\-constrained setups\.

## References

![[Uncaptioned image]](https://arxiv.org/html/2607.18284v1/bios/atdovas/image.jpg)Athanasios Ntovasreceived his diploma from the Computers and Communication Engineering Department of the University of Thessaly \(UTH\) in 2019\.Since April 2020, he has been working as a research assistant in the Information Technologies Institute \(ITI\) of the Centre for Research and Technology Hellas \(CERTH\)\. His research interests include artificial intelligence, computer vision, and digital signal processing\.![[Uncaptioned image]](https://arxiv.org/html/2607.18284v1/bios/aldoum/alex.jpg)Alexandros Doumanoglouhas received the diploma of Electrical and Computer Engineer from the Aristotle University of Thessaloniki \(A\.U\.Th\) and is currently doing his PhD in Explainable and Interpretable Artificial Intelligence at the department of Advanced Computing Sciences at Maastricht University, under the supervision of Prof\. Kurt Driessens\.He joined the Information Technologies Institute in 2012, and since then he has been working as a research assistant in the fields of computer vision, 3D graphics, and machine learning\. His current research interests include computer vision, unsupervised learning, representation learning, mechanistic interpretability, and explainable and interpretable methods for deep learning models\.![[Uncaptioned image]](https://arxiv.org/html/2607.18284v1/bios/drak/image.jpg)Petros Drakoulisreceived his BSc in Informatics from International Hellenic University and his MSc in Digital Media and Computational Intelligence from Aristotle University of Thessaloniki\.In 2018, he joined the Visual Computing Laboratory of CERTH\-ITI where he works as a Research Associate ever since\. His main areas of interest include Software Engineering, Visual Computing, Machine Learning and Graphics\.![[Uncaptioned image]](https://arxiv.org/html/2607.18284v1/bios/zarpalas/dimitris.jpg)Dimitris Zarpalasholds the diploma of Electrical and Computer Engineer from Aristotle University of Thessaloniki, A\.U\.Th, an MSc in Electrical Engineering \(focusing on computer vision\) from The Pennsylvania State University, and a PhD in medical informatics \(Health Science School, department of Medicine, A\.U\.Th\)\.He has joined the Information Technologies Institute in 2007, and is currently a researcher, grade C\. His research interests include real time tele\-immersion applications \(3D reconstruction of moving humans and their compression\), 3D computer vision, 3D medical image processing, shape analysis of anatomical structures, 3D object recognition, motion capturing and evaluation, while in the past has also worked in indexing, search and retrieval, classification of 3D objects and 3D model watermarking\.\\EOD