LittleBit：通过潜在分解实现超低比特量化

# Ultra Low-Bit Quantizationvia Latent Factorization
Source: [https://arxiv.org/html/2506.13771](https://arxiv.org/html/2506.13771)
\*\*footnotetext:Equal contribution††footnotetext:Corresponding author## LittleBit: Ultra Low\-Bit Quantization via Latent Factorization

###### Abstract

The deployment of large language models \(LLMs\) is frequently hindered by prohibitive memory and computational requirements\. While quantization mitigates these bottlenecks, maintaining model fidelity in the sub\-1\-bit regime remains a persistent challenge\. In this paper, we introduceLittleBit, a novel framework for extreme LLM compression\. We target quantization rates as low as0\.10\.1bits per weight \(BPW\), achieving a memory reduction of approximately31×31\\times, which effectively compresses Llama2\-13B to under0\.90\.9GB\. We represent weights via low\-rank latent matrix factorization and subsequently binarize the resulting factors\. To counteract the information loss inherent to such drastic precision reduction, we integrate a multi\-scale compensation mechanism that learns importance parameters across row, column, and latent dimensions\. Two primary contributions enable effective training: Dual Sign\-Value\-Independent Decomposition \(Dual\-SVID\) for quantization\-aware training \(QAT\) initialization, and Residual Compensation to minimize approximation errors\. Extensive experiments confirm the superiority ofLittleBitin the sub\-1\-bit domain; for instance, our method at0\.10\.1BPW surpasses the performance of leading techniques operating at0\.70\.7BPW on Llama2\-7B\. We establish a new size\-performance trade\-off—unlocking a potential11\.6×11\.6\\timesinference speedup relative to FP16—and render powerful LLMs practical for resource\-constrained environments\. Our code is available at[https://github\.com/SamsungLabs/LittleBit](https://github.com/SamsungLabs/LittleBit)\.

## 1Introduction

Large language models \(LLMs\) based on the Transformer architecture\[[1](https://arxiv.org/html/2506.13771#bib.bib1)\]have fundamentally transformed natural language processing\. However, the scale of these models, often reaching hundreds of billions or trillions of parameters\[[2](https://arxiv.org/html/2506.13771#bib.bib44),[3](https://arxiv.org/html/2506.13771#bib.bib43)\], results in prohibitive computational and memory costs\. In particular, the high demand for GPU VRAM hinders widespread deployment on consumer or edge devices\[[4](https://arxiv.org/html/2506.13771#bib.bib45),[5](https://arxiv.org/html/2506.13771#bib.bib46),[6](https://arxiv.org/html/2506.13771#bib.bib47)\]\.

Model quantization, which reduces numerical precision, serves as a primary technique to address these bottlenecks\[[7](https://arxiv.org/html/2506.13771#bib.bib29)\]\. While post\-training quantization \(PTQ\) methods such as GPTQ\[[8](https://arxiv.org/html/2506.13771#bib.bib9)\]and AWQ\[[9](https://arxiv.org/html/2506.13771#bib.bib13)\]effectively compress models to approximately 4 bits, achieving further compression \(e\.g\., to 1\-bit\) typically requires quantization\-aware training \(QAT\)\[[10](https://arxiv.org/html/2506.13771#bib.bib48),[11](https://arxiv.org/html/2506.13771#bib.bib49)\]to maintain performance\. QAT has demonstrated the capability to enable 1\-bit compression while preserving model fidelity\[[12](https://arxiv.org/html/2506.13771#bib.bib12),[13](https://arxiv.org/html/2506.13771#bib.bib24),[14](https://arxiv.org/html/2506.13771#bib.bib33)\]\.

Nevertheless, even 1\-bit models \(e\.g\., approximately 15\.4 GB for 70B parameters\[[7](https://arxiv.org/html/2506.13771#bib.bib29)\]\) may exceed the capacity of highly resource\-constrained devices\[[15](https://arxiv.org/html/2506.13771#bib.bib50)\]\. This limitation motivates the exploration of sub\-1\-bit quantization\. Although prior work\[[16](https://arxiv.org/html/2506.13771#bib.bib30)\]has investigated this area, maintaining performance at extremely low effective bits \(e\.g\., 0\.55 bits per weight \(BPW\)\) while ensuring computational efficiency without restrictive hardware dependencies remains challenging\[[16](https://arxiv.org/html/2506.13771#bib.bib30)\]\. Consequently, it is necessary to develop new techniques that achieve such extreme model compression, aiming for bit rates around 0\.1 BPW, while preserving model performance\.

Figure 1:Low\-bit quantization perplexity on Llama2\-13B \(WikiText\-2\)\.LittleBitsurpasses the state\-of\-the\-art sub\-1\-bit quantization technique\. Below 0\.5 BPW, where the leading prior method degrades sharply, ours remains robust down to 0\.1 BPW\.We derive our approach from two observations\. First, LLM weight matrices often exhibit pronounced low\-rank structure\[[17](https://arxiv.org/html/2506.13771#bib.bib51),[18](https://arxiv.org/html/2506.13771#bib.bib52)\]\. This observation suggests that factorization methods such as Singular Value Decomposition \(SVD\)\[[19](https://arxiv.org/html/2506.13771#bib.bib53)\]may offer a more stable compression pathway than pruning\[[20](https://arxiv.org/html/2506.13771#bib.bib54)\], particularly under extreme compression regimes\. For instance, at compression ratios exceeding 50%, SVD\-based approaches\[[21](https://arxiv.org/html/2506.13771#bib.bib32)\]consistently outperform pruning methods such as Wanda\[[22](https://arxiv.org/html/2506.13771#bib.bib14)\]\(see[AppendixE](https://arxiv.org/html/2506.13771#A5)for details\)\. Second, binarization inherently causes information loss\[[23](https://arxiv.org/html/2506.13771#bib.bib34)\]\. Recent high\-performing 1\-bit methods\[[13](https://arxiv.org/html/2506.13771#bib.bib24),[24](https://arxiv.org/html/2506.13771#bib.bib25),[25](https://arxiv.org/html/2506.13771#bib.bib35)\]demonstrate that comprehensive scaling over multiple dimensions \(e\.g\., row and column\) is crucial for mitigating this loss and stabilizing performance\[[25](https://arxiv.org/html/2506.13771#bib.bib35)\]\.

Building on these insights, we introduceLittleBit, a novel method for extreme sub\-1\-bit quantization \(e\.g\., 0\.1 BPW\)\. InLittleBit, we first represent weights via low\-rank factorization \(𝐖≈𝐔𝐕⊤\\mathbf\{W\}\\approx\\mathbf\{U\}\\mathbf\{V\}^\{\\top\}\) and subsequently binarize these factors\. To counteract errors from binarization, we leverage a multi\-scale compensation mechanism that applies learnable scales across rows, columns, and anadditional latentdimension\. We complement this architecture with Residual Compensation\.

Extensive experiments demonstrate the superiority ofLittleBitover STBLLM\[[16](https://arxiv.org/html/2506.13771#bib.bib30)\], the leading sub\-1\-bit technique, on LLMs ranging from 1\.3B to 32B parameters\. Notably,LittleBitachieves competitive performance on Llama2\-13B at 0\.1 BPW \([Fig\.1](https://arxiv.org/html/2506.13771#S1.F1)\), surpassing STBLLM at 0\.55 BPW\. Furthermore, the 32BLittleBitmodel at just 0\.3 BPW maintains strong performance, substantially outperforming STBLLM and establishing a new state\-of\-the\-art for sub\-1\-bit quantization\.

In summary, our contributions are as follows:

- •We proposeLittleBit, a novel framework unifying latent matrix factorization with multi\-scale compensation to enable extreme sub\-1\-bit quantization\. This method achieves effective bits down to 0\.1 BPW while preserving model performance, establishing a viable size\-performance trade\-off for deploying LLMs in resource\-constrained environments\.
- •We introduce Dual\-SVID for the effective initialization of factorized structures and integrate Residual Compensation to mitigate approximation errors\. These techniques collectively stabilize quantization\-aware training \(QAT\) and counteract the information loss inherent in drastic precision reduction, ensuring robust learning dynamics\.
- •We provide extensive empirical validation demonstrating thatLittleBitconsistently outperforms STBLLM\[[16](https://arxiv.org/html/2506.13771#bib.bib30)\], the prior leading sub\-1\-bit technique, across various model scales\. Our results confirm thatLittleBitmaintains superior fidelity in the deep compression regime, where baseline methods typically suffer severe degradation\.

## 2Related Works

### 2\.1Binarization and Quantization

Network binarization aims for extreme compression and acceleration by constraining weights or activations to±1\\pm 1\. This enables efficient bitwise operations\[[26](https://arxiv.org/html/2506.13771#bib.bib36),[27](https://arxiv.org/html/2506.13771#bib.bib37)\]\. While early studies such as BinaryConnect\[[28](https://arxiv.org/html/2506.13771#bib.bib38)\]and BNNs\[[26](https://arxiv.org/html/2506.13771#bib.bib36)\]demonstrated feasibility, they encountered substantial accuracy degradation\[[29](https://arxiv.org/html/2506.13771#bib.bib40)\]\. Researchers partially mitigated this degradation by introducing scaling factors\[[30](https://arxiv.org/html/2506.13771#bib.bib39)\]and employing the Straight\-Through Estimator \(STE\)\[[31](https://arxiv.org/html/2506.13771#bib.bib28)\]to handle the non\-differentiable quantization function during training\[[32](https://arxiv.org/html/2506.13771#bib.bib41)\]\.

The direct application of these early techniques to large language models \(LLMs\) typically results in severe performance loss\[[33](https://arxiv.org/html/2506.13771#bib.bib42)\]\. Consequently, LLM\-specific quantization strategies have emerged\. These are primarily categorized into post\-training quantization \(PTQ\) and quantization\-aware training \(QAT\)\[[34](https://arxiv.org/html/2506.13771#bib.bib56)\]\. PTQ adapts pre\-trained models and achieves competitive results at approximately 4\-bit precision with methods such as GPTQ\[[8](https://arxiv.org/html/2506.13771#bib.bib9)\]and AWQ\[[9](https://arxiv.org/html/2506.13771#bib.bib13)\]\. However, these methods generally struggle to maintain performance below 2\-bit precision\[[33](https://arxiv.org/html/2506.13771#bib.bib42),[7](https://arxiv.org/html/2506.13771#bib.bib29)\]\. The sub\-1\-bit regime presents particular challenges for PTQ methods due to severe information loss\[[35](https://arxiv.org/html/2506.13771#bib.bib55)\]\. Recent work such as STBLLM\[[16](https://arxiv.org/html/2506.13771#bib.bib30)\]demonstrates that incorporating structured binarized compression can extend PTQ into the sub\-1\-bit regime, but a non\-negligible accuracy gap persists\.

In contrast, QAT integrates the quantization process directly into the training loop\. This enables the model to learn and adapt to the low\-precision format\[[36](https://arxiv.org/html/2506.13771#bib.bib10)\]\. This approach generally yields superior performance compared to PTQ at very low bit\-widths\. Successful QAT approaches for low\-bit scenarios often employ sophisticated scaling mechanisms alongside adaptive training, as evidenced by recent methods\[[13](https://arxiv.org/html/2506.13771#bib.bib24),[24](https://arxiv.org/html/2506.13771#bib.bib25)\]\. Despite these advancements, consistently achieving high model fidelity when quantizing to less than1\.01\.0BPW remains a considerable challenge\. This persisting difficulty underscores the need for novel architectures and training strategies specifically designed for such extreme compression regimes\.

### 2\.2Low\-Rank Approximation in Quantization

Beyond quantization, the inherent parameter redundancy within deep neural networks\[[35](https://arxiv.org/html/2506.13771#bib.bib55)\]can be effectively exploited using low\-rank approximation methods\. This redundancy is particularly evident in LLMs\[[37](https://arxiv.org/html/2506.13771#bib.bib62),[38](https://arxiv.org/html/2506.13771#bib.bib63)\]\. Techniques such as Singular Value Decomposition \(SVD\)\[[19](https://arxiv.org/html/2506.13771#bib.bib53)\]allow large weight matrices to be represented as products of smaller factors\. This offers an alternative compression strategy that may be more resilient than pruning, especially under high compression regimes\[[20](https://arxiv.org/html/2506.13771#bib.bib54)\]\. Recognizing the potential for complementary benefits, researchers have explored combining low\-rank factorization to reduce parameter count with quantization to lower parameter precision\[[39](https://arxiv.org/html/2506.13771#bib.bib65)\]\. Initial studies applied SVD in various roles within the LLM context\. Some approaches used activation statistics to guide decomposition\[[21](https://arxiv.org/html/2506.13771#bib.bib32)\], while others integrated SVD principles into initialization or parameter\-efficient updates\[[40](https://arxiv.org/html/2506.13771#bib.bib60),[41](https://arxiv.org/html/2506.13771#bib.bib59),[13](https://arxiv.org/html/2506.13771#bib.bib24)\]\. Further works applied transformations prior to factorization\[[42](https://arxiv.org/html/2506.13771#bib.bib27)\]or employed low\-rank structures to model quantization errors\[[43](https://arxiv.org/html/2506.13771#bib.bib57)\]\.

![Refer to caption](https://arxiv.org/html/2506.13771v5/littlebit_overview.png)Figure 2:Comparison of a standard Transformer layer \(left\) and theLittleBitarchitecture \(right\)\.LittleBitperforms linear transformation using parallel Primary and Residual pathways\. The Primary path employs binarized factors \(𝐔sign,𝐕sign\\mathbf\{U\}\_\{\\mathrm\{sign\}\},\\mathbf\{V\}\_\{\\mathrm\{sign\}\}\) and FP16 scales \(𝐡,𝐠,ℓ\\mathbf\{h\},\\mathbf\{g\},\\bm\{\\ell\}\) on input𝐗\\mathbf\{X\}, initialized from𝐖\\mathbf\{W\}via Dual\-SVID\. Simultaneously, the Residual path computes a correction with its own parameters \(𝐔res,sign,𝐕res,sign,𝐡res,𝐠res,ℓres\\mathbf\{U\}\_\{\\mathrm\{res,sign\}\},\\mathbf\{V\}\_\{\\mathrm\{res,sign\}\},\\mathbf\{h\}\_\{\\mathrm\{res\}\},\\mathbf\{g\}\_\{\\mathrm\{res\}\},\\bm\{\\ell\}\_\{\\mathrm\{res\}\}\) from the approximation residual\. Their outputs sum to form𝐘\\mathbf\{Y\}, eliminating storage of the effective weight matrices𝐖^pri\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\}\}and𝐖^res\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{res\}\}\.A more integrated approach involves incorporating low\-rank concepts directly within QAT methods\. For example, DL\-QAT\[[44](https://arxiv.org/html/2506.13771#bib.bib58)\]jointly learns a representation amenable to both low\-rank approximation and low\-precision quantization\. However, this method is currently limited to 3\-bit quantization\. Optimizing structural properties such as rank and numerical precision during training represents a promising path toward effective LLM compression\. There is potential for further exploration of even lower precision quantization\.

## 3Methodology

This section detailsLittleBit\. It describes the factorized architecture with multi\-scale compensation \([Section3\.1](https://arxiv.org/html/2506.13771#S3.SS1)\), the Dual\-SVID initialization for stable QAT \([Section3\.2](https://arxiv.org/html/2506.13771#S3.SS2)\), and Residual Compensation for enhanced fidelity \([Section3\.3](https://arxiv.org/html/2506.13771#S3.SS3)\)\. These components are optimized via QAT, as detailed in[Section4](https://arxiv.org/html/2506.13771#S4)\.

### 3\.1LittleBitArchitecture

TheLittleBitarchitecture, depicted in[Fig\.2](https://arxiv.org/html/2506.13771#S2.F2), redesigns linear layers for extreme compression \(e\.g\., 0\.1 BPW\) by synergistically combining low\-rank factorization and binarization\. The method leverages the observed low\-rank structure in LLM weights\[[45](https://arxiv.org/html/2506.13771#bib.bib31)\]to approximate the weight matrix𝐖∈ℝdout×din\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}as:

𝐖≈𝐔𝐕⊤\(𝐔∈ℝdout×r,𝐕∈ℝdin×r,r≪min⁡\(din,dout\)\)\\mathbf\{W\}\\approx\\mathbf\{U\}\\mathbf\{V\}^\{\\top\}\\quad\(\\mathbf\{U\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\},\\mathbf\{V\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\\times r\},r\\ll\\min\(d\_\{\\mathrm\{in\}\},d\_\{\\mathrm\{out\}\}\)\)\(1\)We prioritize this factorization\-based approach over other structural compression methods, such as pruning, due to its superior robustness in extreme compression regimes\. As the comparative analysis against methods like Wanda demonstrates \([AppendixE](https://arxiv.org/html/2506.13771#A5)\), SVD\-based compression maintains model integrity far more effectively at high compression ratios\. This provides a more stable foundation for ultra low\-bit quantization\. The resulting factors𝐔\\mathbf\{U\}and𝐕\\mathbf\{V\}are then binarized:

𝐔sign=sign⁡\(𝐔\),𝐕sign=sign⁡\(𝐕\)∈\{−1,\+1\}\\mathbf\{U\}\_\{\\mathrm\{sign\}\}=\\mathrm\{sign\}\(\\mathbf\{U\}\),\\quad\\mathbf\{V\}\_\{\\mathrm\{sign\}\}=\\mathrm\{sign\}\(\\mathbf\{V\}\)\\in\\\{\-1,\+1\\\}\(2\)To compensate for the significant information loss incurred by binarization\[[23](https://arxiv.org/html/2506.13771#bib.bib34)\],LittleBitincorporates learnable FP16 scales\. Beyond standard row \(𝐡\\mathbf\{h\}\) and column \(𝐠\\mathbf\{g\}\) scaling\[[33](https://arxiv.org/html/2506.13771#bib.bib42),[13](https://arxiv.org/html/2506.13771#bib.bib24),[25](https://arxiv.org/html/2506.13771#bib.bib35)\], the architecture introduces anadditional latentscale \(ℓ∈ℝr\\bm\{\\ell\}\\in\\mathbb\{R\}^\{r\}\)\. This latent scale addresses the factorization structure by learning the relative importance of each of therrlatent dimensions, which correspond to the columns of the binarized factors𝐔sign\\mathbf\{U\}\_\{\\mathrm\{sign\}\}and𝐕sign\\mathbf\{V\}\_\{\\mathrm\{sign\}\}:

𝐡∈ℝdout,𝐠∈ℝdin,ℓ∈ℝr\\mathbf\{h\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\},\\quad\\mathbf\{g\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\},\\quad\\bm\{\\ell\}\\in\\mathbb\{R\}^\{r\}\(3\)The primary effective weight,𝐖^pri\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\}\}, is implicitly constructed from these components:

𝐖^pri=diag⁡\(𝐡\)​𝐔sign​diag​\(ℓ\)​𝐕sign⊤​diag​\(𝐠\)\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\}\}=\\mathrm\{diag\}\(\\mathbf\{h\}\)\\mathbf\{U\}\_\{\\mathrm\{sign\}\}\\mathrm\{diag\}\(\\bm\{\\ell\}\)\\mathbf\{V\}\_\{\\mathrm\{sign\}\}^\{\\top\}\\mathrm\{diag\}\(\\mathbf\{g\}\)\(4\)Instead of storing or optimizing𝐖^pri\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\}\}directly, the model learns latent full\-precision factors𝐔\\mathbf\{U\}and𝐕\\mathbf\{V\}\. These are binarized during the forward pass using[Eq\.2](https://arxiv.org/html/2506.13771#S3.E2), along with the FP16 scales𝐡,𝐠,ℓ\\mathbf\{h\},\\mathbf\{g\},\\bm\{\\ell\}\. This factorized representation enables efficient computation during the forward pass\.

#### Proposition 1

Let𝐗∈ℝseq×din\\mathbf\{X\}\\in\\mathbb\{R\}^\{\\mathrm\{seq\}\\times d\_\{\\mathrm\{in\}\}\}be the input matrix\. The forward computation𝐘=𝐗​𝐖^pri⊤\\mathbf\{Y\}=\\mathbf\{X\}\{\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\}\}\}^\{\\top\}using the primary approximation \([Eq\.4](https://arxiv.org/html/2506.13771#S3.E4)\) is efficiently computed as:

𝐘=\(\(\(\(𝐗⊙𝐠\)​𝐕sign\)⊙ℓ\)​𝐔sign⊤\)⊙𝐡\\mathbf\{Y\}=\(\(\(\(\\mathbf\{X\}\\odot\\mathbf\{g\}\)\\mathbf\{V\}\_\{\\mathrm\{sign\}\}\)\\odot\\bm\{\\ell\}\)\\mathbf\{U\}\_\{\\mathrm\{sign\}\}^\{\\top\}\)\\odot\\mathbf\{h\}\(5\)where⊙\\odotdenotes element\-wise multiplication with broadcasting\.

This decomposition \([Eq\.5](https://arxiv.org/html/2506.13771#S3.E5)\) replaces a large high\-precision General Matrix Multiply \(GEMM\) operation with two smaller binary matrix multiplications and element\-wise scaling operations\. This offers computational advantages and reduces memory requirements\. The effective bits per weight \(BPW\) is determined by the storage cost of these learnable parameters relative to the original matrix size \(see[AppendixD](https://arxiv.org/html/2506.13771#A4)for details\)\.

### 3\.2Dual\-SVID Initialization

Naively initializing the highly constrainedLittleBitstructure can lead to unstable QAT\. To circumvent this, we proposeDual\-SVID, an SVD\-based initialization method designed to provide a starting point where the initial effective weight𝐖^pri,0\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\},0\}closely approximates𝐖\\mathbf\{W\}\. Dual\-SVID aims to preserve essential sign and magnitude information from the optimal low\-rank SVD factors \(obtained from𝐖≈𝐔′​𝐕′⊤\\mathbf\{W\}\\approx\\mathbf\{U\}^\{\\prime\}\{\\mathbf\{V\}^\{\\prime\}\}^\{\\top\}, then truncated\) and map this information to the learnableLittleBitparameters \(𝐔sign,𝐕sign,𝐡,𝐠,ℓ\\mathbf\{U\}\_\{\\mathrm\{sign\}\},\\mathbf\{V\}\_\{\\mathrm\{sign\}\},\\mathbf\{h\},\\mathbf\{g\},\\bm\{\\ell\}\)\. The process involves three steps: \(1\) Obtaining𝐔′,𝐕′\\mathbf\{U\}^\{\\prime\},\\mathbf\{V\}^\{\\prime\}via a truncated SVD\. \(2\) Setting initial binary factors based on signs \([Eq\.6](https://arxiv.org/html/2506.13771#S3.E6)\)\. \(3\) Decomposing the magnitudes\|𝐔′\|∈ℝdout×r\|\\mathbf\{U\}^\{\\prime\}\|\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}and\|𝐕′\|∈ℝdin×r\|\\mathbf\{V\}^\{\\prime\}\|\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\\times r\}\. To initialize the scales, a rank\-1 SVD \(or equivalent rank\-1 approximation\) is performed on each of these magnitude matrices separately\. For\|𝐔′\|\|\\mathbf\{U\}^\{\\prime\}\|, its best rank\-1 approximation can be written as𝐬U,0​\(ℓu,0\)⊤\\mathbf\{s\}\_\{U,0\}\(\\bm\{\\ell\}\_\{u,0\}\)^\{\\top\}, where𝐬U,0∈ℝdout\\mathbf\{s\}\_\{U,0\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\}becomes the initial row scale𝐡0\\mathbf\{h\}\_\{0\}, andℓu,0∈ℝr\\bm\{\\ell\}\_\{u,0\}\\in\\mathbb\{R\}^\{r\}is a component contributing to the latent scale\. Similarly, for\|𝐕′\|\|\\mathbf\{V\}^\{\\prime\}\|, its rank\-1 approximation𝐬V,0​\(ℓv,0\)⊤\\mathbf\{s\}\_\{V,0\}\(\\bm\{\\ell\}\_\{v,0\}\)^\{\\top\}provides the initial column scale𝐠0=𝐬V,0∈ℝdin\\mathbf\{g\}\_\{0\}=\\mathbf\{s\}\_\{V,0\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\}and another latent scale componentℓv,0∈ℝr\\bm\{\\ell\}\_\{v,0\}\\in\\mathbb\{R\}^\{r\}\. These steps effectively disentangle multi\-dimensional magnitude variations and allow for the estimation of initial row \(𝐡0\\mathbf\{h\}\_\{0\}\), column \(𝐠0\\mathbf\{g\}\_\{0\}\), and the final latent scale \(ℓ0=ℓu,0⊙ℓv,0\\bm\{\\ell\}\_\{0\}=\\bm\{\\ell\}\_\{u,0\}\\odot\\bm\{\\ell\}\_\{v,0\}by element\-wise product\), as shown in[Eqs\.7](https://arxiv.org/html/2506.13771#S3.E7)and[8](https://arxiv.org/html/2506.13771#S3.E8):

𝐔sign,0=sign⁡\(𝐔′\),𝐕sign,0=sign⁡\(𝐕′\)\\displaystyle\\mathbf\{U\}\_\{\\mathrm\{sign\},0\}=\\mathrm\{sign\}\(\\mathbf\{U\}^\{\\prime\}\),\\quad\\mathbf\{V\}\_\{\\mathrm\{sign\},0\}=\\mathrm\{sign\}\(\\mathbf\{V\}^\{\\prime\}\)\(6\)\|𝐔′\|≈𝐡0​\(ℓu,0\)⊤,\|𝐕′\|≈𝐠0​\(ℓv,0\)⊤\\displaystyle\|\\mathbf\{U\}^\{\\prime\}\|\\approx\\mathbf\{h\}\_\{0\}\(\\bm\{\\ell\}\_\{u,0\}\)^\{\\top\},\\quad\|\\mathbf\{V\}^\{\\prime\}\|\\approx\\mathbf\{g\}\_\{0\}\(\\bm\{\\ell\}\_\{v,0\}\)^\{\\top\}\(7\)ℓ0=ℓu,0⊙ℓv,0\\displaystyle\\bm\{\\ell\}\_\{0\}=\\bm\{\\ell\}\_\{u,0\}\\odot\\bm\{\\ell\}\_\{v,0\}\(8\)
The resulting initial effective weight𝐖^pri,0=diag⁡\(𝐡0\)​𝐔sign,0​diag​\(ℓ0\)​𝐕sign,0⊤​diag​\(𝐠0\)\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\},0\}=\\mathrm\{diag\}\(\\mathbf\{h\}\_\{0\}\)\\mathbf\{U\}\_\{\\mathrm\{sign\},0\}\\mathrm\{diag\}\(\\bm\{\\ell\}\_\{0\}\)\\mathbf\{V\}\_\{\\mathrm\{sign\},0\}^\{\\top\}\\mathrm\{diag\}\(\\mathbf\{g\}\_\{0\}\)preserves key structural information\. The learnable parameters are initialized as follows: the latent factors𝐔\\mathbf\{U\}and𝐕\\mathbf\{V\}are initialized with the factors𝐔′\\mathbf\{U\}^\{\\prime\}and𝐕′\\mathbf\{V\}^\{\\prime\}obtained from SVD, respectively, and the scales𝐡,𝐠,ℓ\\mathbf\{h\},\\mathbf\{g\},\\bm\{\\ell\}are initialized to𝐡0,𝐠0,ℓ0\\mathbf\{h\}\_\{0\},\\mathbf\{g\}\_\{0\},\\bm\{\\ell\}\_\{0\}\.

### 3\.3Residual Compensation

To further improve fidelity without increasing the primary path’s rank, we introduceResidual Compensation\. This technique is motivated by the theoretical insight that a two\-stage error correction can be more effective than a single, larger approximation \([AppendixA](https://arxiv.org/html/2506.13771#A1)\)\. Crucially, this method does not increase the model’s total parameter budget; instead, the fixed bit budget of a single, higher\-rank approximation is strategically reallocated into two lower\-rank paths: a primary and a residual path\. The term ‘residual’ primarily describes its role during initialization, where the auxiliary path is configured to model the error of the primary approximation\. During QAT, both paths are optimized jointly to collectively represent the original weight\. The advantage of this dual\-path approach over a single path is empirically validated in our ablation studies \([SectionB\.1](https://arxiv.org/html/2506.13771#A2.SS1)\)\.

![Refer to caption](https://arxiv.org/html/2506.13771v5/weight_analysis.png)Figure 3:Visualization of Dual\-SVID initialized weight components for a selected layer in Llama2\-7B \(Query Weight, Layer 0\)\. Columns, from left to right, represent effective bits of 0\.1, 0\.3, 0\.55, 0\.7, 0\.8, and 1\.0 BPW\. Rows display the primary approximation \(𝐖^pri,0\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\},0\}\), the residual approximation \(𝐖^res,0\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{res\},0\}\), and their sum \(𝐖^0=𝐖^pri,0\+𝐖^res,0\\widehat\{\\mathbf\{W\}\}\_\{0\}=\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\},0\}\+\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{res\},0\}\)\. The rightmost image shows the corresponding crop of the original weight matrix \(𝐖\\mathbf\{W\}\) for reference\.To implement this, Residual Compensation employs a parallel auxiliary path that learns an approximation𝐖^res\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{res\}\}of the residual error\. This auxiliary path mirrors the structure of theLittleBitpath:

𝐖^res=diag⁡\(𝐡res\)​𝐔res,sign​diag​\(ℓres\)​𝐕res,sign⊤​diag​\(𝐠res\)\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{res\}\}=\\mathrm\{diag\}\(\\mathbf\{h\}\_\{\\mathrm\{res\}\}\)\\mathbf\{U\}\_\{\\mathrm\{res,sign\}\}\\mathrm\{diag\}\(\\bm\{\\ell\}\_\{\\mathrm\{res\}\}\)\\mathbf\{V\}\_\{\\mathrm\{res,sign\}\}^\{\\top\}\\mathrm\{diag\}\(\\mathbf\{g\}\_\{\\mathrm\{res\}\}\)\(9\)
The parameters of this auxiliary path \(𝐔res,sign,𝐕res,sign,𝐡res,𝐠res,ℓres\\mathbf\{U\}\_\{\\mathrm\{res,sign\}\},\\mathbf\{V\}\_\{\\mathrm\{res,sign\}\},\\mathbf\{h\}\_\{\\mathrm\{res\}\},\\mathbf\{g\}\_\{\\mathrm\{res\}\},\\bm\{\\ell\}\_\{\\mathrm\{res\}\}\) are also learnable during QAT\. They are initialized using the Dual\-SVID strategy applied to the initial residual error𝐖res,0=𝐖−𝐖^pri,0\\mathbf\{W\}\_\{\\mathrm\{res\},0\}=\\mathbf\{W\}\-\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\},0\}\. Employing a separate path allows the model to specifically learn and compensate for the potentially distinct characteristics of this residual error\. The final effective weight is the sum of both approximations:

𝐖^=𝐖^pri\+𝐖^res\\widehat\{\\mathbf\{W\}\}=\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\}\}\+\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{res\}\}\(10\)
This targeted error correction is often advantageous for maintaining performance at extreme compression levels, as demonstrated by the ablation studies presented in[SectionB\.1](https://arxiv.org/html/2506.13771#A2.SS1)\. It is important to note that all learnable parameters from both the primary and residual paths are included when calculating the final BPW \(see[AppendixD](https://arxiv.org/html/2506.13771#A4)for calculation details\)\.

[Figure3](https://arxiv.org/html/2506.13771#S3.F3)visualizes the initial state of weight components \(𝐖^pri,0\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\},0\},𝐖^res,0\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{res\},0\}, and their sum𝐖^0\\widehat\{\\mathbf\{W\}\}\_\{0\}\) for a Llama2\-7B query weight, derived via Dual\-SVID across effective bits from 0\.1 to 1\.0, compared against the original weight𝐖\\mathbf\{W\}\(far right\)\. Even at a low 0\.1 BPW, the primary approximation𝐖^pri,0\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\},0\}\(top row, leftmost\) captures the dominant low\-rank structure of𝐖\\mathbf\{W\}, albeit with considerable simplification\. As effective bits increase,𝐖^pri,0\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\},0\}progressively refines details and more effectively suppresses disruptive approximation noise, allowing underlying weight patterns to emerge with greater clarity\. A key observation is that𝐖^0\\widehat\{\\mathbf\{W\}\}\_\{0\}\(bottom row\) at a modest 0\.3 BPW—benefiting from the initialized residual component𝐖^res,0\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{res\},0\}\(middle row\)—often presents a more faithful initial representation of𝐖\\mathbf\{W\}than𝐖^pri,0\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\},0\}alone even at a higher 1\.0 BPW\. This underscores that Residual Compensation, even at the initialization stage, is vital for capturing complexities missed by the primary low\-rank approximation, thereby providing a more faithful starting point for QAT\.

## 4Experiments

### 4\.1Settings

#### Evaluation Setup

We evaluatedLittleBitacross diverse LLM families, including Llama\[[46](https://arxiv.org/html/2506.13771#bib.bib2)\], Llama2\[[47](https://arxiv.org/html/2506.13771#bib.bib3)\], Llama3\[[48](https://arxiv.org/html/2506.13771#bib.bib4)\], OPT\[[49](https://arxiv.org/html/2506.13771#bib.bib5)\], Phi\-4\[[50](https://arxiv.org/html/2506.13771#bib.bib6)\], and QwQ\[[51](https://arxiv.org/html/2506.13771#bib.bib7)\]\. These models span parameter scales from 1\.3B to 32B\. Perplexity \(PPL\) on the WikiText\-2\[[52](https://arxiv.org/html/2506.13771#bib.bib16)\]validation dataset served as the primary performance metric\.[AppendixC](https://arxiv.org/html/2506.13771#A3)provides additional results on the C4\[[53](https://arxiv.org/html/2506.13771#bib.bib17)\]and PTB\[[54](https://arxiv.org/html/2506.13771#bib.bib18)\]datasets\. Furthermore, we assessed zero\-shot accuracy on common reasoning benchmarks: WinoGrande\[[55](https://arxiv.org/html/2506.13771#bib.bib22)\], OpenBookQA \(OBQA\)\[[56](https://arxiv.org/html/2506.13771#bib.bib23)\], HellaSwag\[[57](https://arxiv.org/html/2506.13771#bib.bib21)\], BoolQ\[[58](https://arxiv.org/html/2506.13771#bib.bib20)\], ARC\-Easy \(ARC\-e\), ARC\-Challenge \(ARC\-c\)\[[59](https://arxiv.org/html/2506.13771#bib.bib19)\], and PIQA\[[60](https://arxiv.org/html/2506.13771#bib.bib8)\]\.

#### Training Details

We optimized theLittleBitmodel parameters, initialized via the Dual\-SVID method \([Section3\.2](https://arxiv.org/html/2506.13771#S3.SS2)\), using QAT with knowledge distillation \(KD\)\[[61](https://arxiv.org/html/2506.13771#bib.bib64),[36](https://arxiv.org/html/2506.13771#bib.bib10),[62](https://arxiv.org/html/2506.13771#bib.bib61)\]\. The original pre\-trained full\-precision model functioned as the teacher \(𝒯\\mathcal\{T\}\) for theLittleBitstudent model \(𝒮\\mathcal\{S\}\)\. The QAT objective combines the standard output Kullback\-Leibler \(KL\) divergence loss,ℒout\\mathcal\{L\}\_\{\\mathrm\{out\}\}, and an intermediate layer mean squared error \(MSE\) loss,ℒinter\\mathcal\{L\}\_\{\\mathrm\{inter\}\}, to match hidden representations\. We weighted these terms using an empirically determined coefficientλ=10\\lambda=10:

ℒQAT=ℒout\+λ​ℒinter\.\\mathcal\{L\}\_\{\\mathrm\{QAT\}\}=\\mathcal\{L\}\_\{\\mathrm\{out\}\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{inter\}\}\.\(11\)Adhering to the protocol in\[[24](https://arxiv.org/html/2506.13771#bib.bib25)\], the training data combined WikiText\-2 with selected partitions from C4\. The configuration included a sequence length of 2048 tokens, 5 epochs, the Adam optimizer \(β1=0\.9,β2=0\.999\\beta\_\{1\}=0\.9,\\beta\_\{2\}=0\.999\)\[[63](https://arxiv.org/html/2506.13771#bib.bib15)\], and a cosine learning rate decay with 2% warm\-up \(seefor details\)\. For models employing Grouped\-Query Attention \(GQA\)\[[64](https://arxiv.org/html/2506.13771#bib.bib66)\], such as Llama3, Phi\-4, and QwQ, we specifically adjusted the latent ranksrrfor key \(K\) and value \(V\) projection layers \(see[SectionB\.3](https://arxiv.org/html/2506.13771#A2.SS3)\) to maintain performance under extreme quantization\. We employedSmoothSign\(forward:sign⁡\(x\)\\mathrm\{sign\}\(x\); backward gradient:tanh⁡\(k​x\)\\tanh\(kx\),k=100k=100\) for backpropagation, as it demonstrated superior performance compared to the Straight\-Through Estimator \(STE\)\[[31](https://arxiv.org/html/2506.13771#bib.bib28)\]\(see[SectionB\.2](https://arxiv.org/html/2506.13771#A2.SS2)\)\. Although proxy gradients are established in the literature, this work validates the specific application of thetanh⁡\(100​x\)\\tanh\(100x\)derivative as a stable gradient for QAT in the ultra\-low\-bit regime\. Our ablation study confirms the effectiveness of this approach \([SectionB\.2](https://arxiv.org/html/2506.13771#A2.SS2)\)\.

#### Baselines

We benchmarkedLittleBitacross effective bit rates ranging from approximately 1\.0 down to 0\.1 bits per weight \(BPW\)\. In the challenging sub\-1\-bit regime, we compared our method against STBLLM\[[16](https://arxiv.org/html/2506.13771#bib.bib30)\], a post\-training quantization \(PTQ\) method that employs N:M sparsity\. For settings approximating 1\.0 BPW, we compared against the 1\-bit PTQ method BiLLM\[[7](https://arxiv.org/html/2506.13771#bib.bib29)\]and quantization\-aware training \(QAT\) approaches such as OneBit\[[13](https://arxiv.org/html/2506.13771#bib.bib24)\]and BinaryMoS\[[24](https://arxiv.org/html/2506.13771#bib.bib25)\]\. We included results from standard low\-bit methods, such as GPTQ\[[8](https://arxiv.org/html/2506.13771#bib.bib9)\]and OmniQuant\[[65](https://arxiv.org/html/2506.13771#bib.bib11)\], to provide broader context\.

### 4\.2Main Results

#### Superior Perplexity in Sub\-1\-bit Regime

[Table1](https://arxiv.org/html/2506.13771#S4.T1)presents the perplexity \(PPL\) results ofLittleBitcompared to baseline methods\. The proposed method demonstrates robust performance, particularly within the sub\-1\-bit regime\. Relative to STBLLM\[[16](https://arxiv.org/html/2506.13771#bib.bib30)\],LittleBitachieves markedly superior PPL scores at0\.80\.8,0\.70\.7, and0\.550\.55BPW\. For instance, on Llama2\-7B,LittleBitat0\.550\.55BPW achieves a PPL of10\.4710\.47, which represents a substantial improvement over the30\.6730\.67score reported for STBLLM\. We observe comparable gains across other models and bit\-widths\. For Llama2\-13B at0\.80\.8BPW,LittleBitrecords a PPL of8\.528\.52, whereas STBLLM records11\.9011\.90\.

Table 1:Perplexity \(PPL\) comparison on WikiText\-2 across various LLMs and quantization methods\. Lower PPL indicates better performance\. BPW denotes effective bits per weight\. Methods marked with†use quantization\-aware training \(QAT\)\. STBLLM\[[16](https://arxiv.org/html/2506.13771#bib.bib30)\]utilizes N:M sparsity \(ratio in parentheses\)\.LittleBitdemonstrates strong performance, particularly in the sub\-1\-bit regime\.Table 2:Zero\-shot accuracy \(%\) comparison on common sense reasoning benchmarks\. Compares Full Precision \(FP16\) models against STBLLM\[[16](https://arxiv.org/html/2506.13771#bib.bib30)\]andLittleBitat 0\.55 and 0\.3 BPW\.\(a\)Phi\-4 14B\(b\)QwQ\-32B
Figure 4:Zero\-shot accuracy \(%\) on 7 common sense reasoning tasks\. The comparison highlights the robustness ofLittleBit\(solid lines\) against STBLLM \(dashed lines\) in the sub\-0\.5 BPW regime\. While STBLLM degrades sharply,LittleBitpreserves viable reasoning capabilities for both \(a\) Phi\-4 and \(b\) QwQ models\.
#### Extreme Low\-BPW Stability

A primary advantage ofLittleBitlies in its exceptional stability and performance in the extreme sub\-0\.5 BPW range, specifically at0\.30\.3BPW and0\.10\.1BPW\. In this regime, methods such as STBLLM exhibit severe performance degradation\. For example, Llama2\-7B yields a PPL of1\.8×1031\.8\\times 10^\{3\}at0\.30\.3BPW\. Conversely,LittleBitmaintains strong PPL scores, such as12\.0012\.00for Llama2\-7B at0\.30\.3BPW and15\.9215\.92at an unprecedented rate of0\.10\.1BPW\. When configured for effective bits approximating1\.01\.0BPW by adjusting the latent rankrr,LittleBityields perplexity scores comparable to specialized 1\-bit QAT methods\. For example, on Llama2\-7B,LittleBitachieves9\.089\.08compared to8\.368\.36for OneBit\[[13](https://arxiv.org/html/2506.13771#bib.bib24)\]\. It also remains competitive with PTQ methods such as BiLLM\[[7](https://arxiv.org/html/2506.13771#bib.bib29)\]\. Although BinaryMoS\[[24](https://arxiv.org/html/2506.13771#bib.bib25)\]may exhibit slightly lower PPL in certain 1\-bit scenarios \(e\.g\., Llama2\-7B: BinaryMoS7\.747\.74\), this performance difference is attributable to its use of more complex mechanisms, such as dynamic scaling, which represent a distinct set of architectural and computational trade\-offs\. Our analysis of the trade\-off involved in adjusting the latent rank to tune effective bits from1\.01\.0down to0\.10\.1BPW indicates that performance degradation is minimal down to approximately0\.550\.55BPW\. However, a quantization cliff appears between0\.30\.3and0\.10\.1BPW\. This observation positions the0\.30\.3–0\.550\.55BPW range as an optimal sweet spot for balancing compression and accuracy\.

#### Model Generalization & Zero\-Shot

We consistently observe the advantages ofLittleBitacross a diverse set of model architectures and sizes\. This includes the Llama3 family, which is typically considered challenging for quantization, as well as other contemporary models such as Phi\-4\. For example, Llama3\-8B quantized byLittleBitto0\.550\.55BPW yields a PPL of18\.4718\.47\. This result substantially outperforms the STBLLM score of241\.95241\.95at0\.550\.55BPW with 4:8 sparsity\. The zero\-shot evaluation results presented in[Table2](https://arxiv.org/html/2506.13771#S4.T2)corroborate these perplexity\-based findings\. For instance, Llama2\-7B compressed byLittleBitto0\.550\.55BPW achieves a mean accuracy of47\.26%47\.26\\%, and the model maintains45\.20%45\.20\\%mean accuracy even at an extreme0\.30\.3BPW\. These results compare favorably with those of STBLLM at similar or even higher effective bits\. STBLLM on Llama2\-7B at0\.550\.55BPW scores44\.29%44\.29\\%\. This outcome suggests superior preservation of the intrinsic reasoning capabilities of the models even under extreme compression\. It highlights the efficacy ofLittleBitin creating highly compact yet powerful LLMs\.

### 4\.3Reasoning Performance of Extremely Compressed LLMs

The capability ofLittleBitto achieve sub\-1\-bit compression while maintaining robust model fidelity prompts an evaluation of its reasoning abilities, particularly for large\-scale models under extreme compression\. We investigated the Phi\-4 14B model and extended our analysis to the QwQ\-32B model, comparing their performance against STBLLM across 7 zero\-shot reasoning tasks\.

When compressed withLittleBitto0\.550\.55BPW, Phi\-4 achieved an average accuracy of approximately52\.3%52\.3\\%\. Although this performance is marginally lower than STBLLM at a similar compression level \(54\.2%54\.2\\%at0\.550\.55BPW\),LittleBitdemonstrates substantially more graceful degradation\. At the more aggressive0\.30\.3BPW setting, theLittleBit\-compressed Phi\-4 maintained an average accuracy of48\.7%48\.7\\%, which markedly outperforms STBLLM at0\.30\.3BPW \(40\.3%40\.3\\%\)\. Even at an extreme0\.10\.1BPW, Phi\-4 retained a notable accuracy of43\.6%43\.6\\%\. We observed a consistent trend with the QwQ\-32B model, as illustrated in[Figure4](https://arxiv.org/html/2506.13771#S4.F4)\. While STBLLM exhibited severe performance collapse in the sub\-0\.5 BPW regime,LittleBitmaintained a stable trajectory down to 0\.1 BPW\. These results highlight the ability ofLittleBitto preserve robust reasoning capabilities across varying model scales as compression becomes increasingly extreme\.

Table 3:Memory footprint comparison \(GB\) for Llama2\[[47](https://arxiv.org/html/2506.13771#bib.bib3)\]models under different quantization methods\.LittleBitis evaluated at various BPWs, achieved by adjusting the latent rankrr\. Compression factors relative to FP16 are shown in parentheses\.The gap between the effective BPW and the compression factor arises because quantization is applied only to the Transformer blocks, while components like the embedding layer and the lm\_head remain in FP16\.Figure 5:Conceptual view of KV Cache storage: the standard method \(left\) stores the full hidden dimension \(dmodeld\_\{\\mathrm\{model\}\}\), whereasLittleBit\(right\) caches a reduced latent dimension \(rr\)BPWKV Latent RankKV Cache\(rr\)Reduction Factor0\.801,624≈\\approx2\.5×\\times0\.551,112≈\\approx3\.7×\\times0\.30600≈\\approx6\.8×\\times0\.10192≈\\approx21\.3×\\times

Table 4:Estimated KV cache memory reduction for Llama2\-7B usingLittleBit, with a reduction factor of approximatelydmodel/rd\_\{\\mathrm\{model\}\}/r\.

## 5Analysis

#### Memory Footprint Reduction

LittleBitis designed for resource\-constrained environments, such as on\-device deployment, and achieves substantial reductions in model memory footprint\. As summarized in[Table3](https://arxiv.org/html/2506.13771#S4.T3), quantizing a Llama2\-7B model \(originally13\.4913\.49GB in FP16\) to an effective bit rate of0\.30\.3BPW usingLittleBitreduces its required storage to0\.790\.79GB\. This corresponds to a compression factor exceeding17×17\\times\. At the extreme setting of0\.10\.1BPW, the footprint is further reduced to0\.630\.63GB \(over21×21\\timescompression\)\. We observe comparable reductions for larger models, such as Llama2\-70B, where the memory footprint decreases from138\.04138\.04GB to under22GB at0\.10\.1BPW, achieving nearly70×70\\timescompression\. These substantial memory reductions significantly expand the range of devices capable of hosting large\-scale language models, thereby enabling complex language tasks on hardware with limited VRAM or storage capacity\.

#### KV Cache Compression

The proposed factorization \([Eq\.4](https://arxiv.org/html/2506.13771#S3.E4)\) inherently compresses the KV cache when applied to key and value projection matrices\. Because the forward computation \([Eq\.5](https://arxiv.org/html/2506.13771#S3.E5)\) operates on latent states, intermediate rankrrvectors are cached rather than fulldmodel\{d\}\_\{\\mathrm\{model\}\}\-dimensional vectors, as illustrated in[Fig\.5](https://arxiv.org/html/2506.13771#S4.F5)\. Consequently, KV cache memory requirements are reduced by a factor of approximatelydmodel/r\{d\}\_\{\\mathrm\{model\}\}/\{r\}\. For example, a reduction of up to 21\.3×\\timesis observed for Llama2\-7B at 0\.1 BPW \([Section4\.3](https://arxiv.org/html/2506.13771#S4.SS3)\)\. While this effect aligns with explicit compression methods such as MLA\[[66](https://arxiv.org/html/2506.13771#bib.bib26)\]and ASVD\[[42](https://arxiv.org/html/2506.13771#bib.bib27)\], the approach described herein achieves this via unified factorization within attention layers, thereby reducing both weight and activation memory simultaneously\.

Figure 6:Kernel\-level latency \(bars, left axis\) and speedup relative to FP16 \(dashed line, right axis\) on an NVIDIA A100 GPU for a Llama2\-70B MLP layer \(8,192×28,6728,192\\times 28,672\)\. The benchmark compares the FP16 GEMM baseline with OneBit and the proposedLittleBit\(LB\) kernel across various BPW configurations ranging from 1\.0 to 0\.1\.
#### Latency Considerations

In addition to memory optimization, inference latency presents a critical performance metric\. The factorized architecture \([Eq\.5](https://arxiv.org/html/2506.13771#S3.E5)\) facilitates computational acceleration through low\-rank binary operations, distinct from methods solely prioritizing memory or requiring specific hardware\. To evaluate this capability, a custom 1\-bit GEMV CUDA kernel was developed to accelerate the primary computational path\. As illustrated in[Fig\.6](https://arxiv.org/html/2506.13771#S5.F6)for a Llama2\-70B MLP layer, the kernel demonstrates speedups that inversely correlate with bit\-width, peaking at11\.6×\\timesover an optimized FP16 baseline at 0\.1 BPW\.

These findings suggest the proposed method enables efficient deployment with reduced latency\. The discrepancy between theoretical computational gains and practical latency is likely attributable to memory access dominance during small\-batch inference\. The factorized approach involves multiple memory\-bound operations, and the custom kernel has not yet reached the optimization level of industry\-standard libraries\.provides a comprehensive analysis covering theoretical FLOPs, practical latency, and end\-to\-end decoding throughput\.

## 6Conclusion

This study proposedLittleBit, a method designed to advance LLM compression into the sub\-0\.5 BPW regime while maintaining viable performance at 0\.1 BPW\. Performance improvements were realized through SVD\-inspired latent matrix factorization, factor binarization, and a multi\-scale compensation mechanism encompassing row, column, and latent dimensions\. Furthermore, Dual\-SVID initialization was introduced to stabilize quantization\-aware training \(QAT\), alongside Residual Compensation for error mitigation\. Empirical results demonstrate that the proposed method markedly outperforms prior sub\-1\-bit techniques and preserves fidelity across LLMs scaling up to 32B parameters\. Consequently,LittleBitsuggests a favorable size–performance trade\-off, thereby facilitating LLM deployment in resource\-constrained environments\. Future research may address practical deployment through hardware co\-design for edge devices, such as neural processing units \(NPUs\)\. Additional improvements in model fidelity could be examined by integrating advanced techniques such as non\-uniform bit allocation, applied either across model layers or within single layers via dynamic bit budget distribution between primary and residual paths\.

## Acknowledgements

We are grateful to Dr\. Heonjae Ha, Dr\. SangJeong Lee, Minseop Choi, Changdong Kim, and Hyochan Chong for their insightful discussions and generous support in this work\.

## References

- \[1\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is All You Need\.Advances in Neural Information Processing Systems30\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p1.1)\.
- \[2\]D\. Narayanan, M\. Shoeybi, J\. Casper, P\. LeGresley, M\. Patwary, V\. Korthikanti, D\. Vainbrand, P\. Kashinkunti, J\. Bernauer, B\. Catanzaro,et al\.\(2021\)Efficient Large\-Scale Language Model Training on GPU Clusters Using Megatron\-LM\.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,pp\. 1–15\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p1.1)\.
- \[3\]M\. Isaev, N\. McDonald, and R\. Vuduc\(2023\)Scaling Infrastructure to Support Multi\-Trillion Parameter LLM Training\.InArchitecture and System Support for Transformer Models \(ASSYST@ ISCA 2023\),Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p1.1)\.
- \[4\]Y\. Zheng, Y\. Chen, B\. Qian, X\. Shi, Y\. Shu, and J\. Chen\(2025\)A Review on Edge Large Language Models: Design, Execution, and Applications\.ACM Computing Surveys57\(8\),pp\. 1–35\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p1.1)\.
- \[5\]X\. Pan, E\. Li, Q\. Li, S\. Liang, Y\. Shan, K\. Zhou, Y\. Luo, X\. Wang, and J\. Zhang\(2024\)InstInfer: In\-Storage Attention Offloading for Cost\-Effective Long\-Context LLM Inference\.arXiv preprint arXiv:2409\.04992\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p1.1)\.
- \[6\]L\. Zhang, Z\. Zhang, R\. Li, Z\. Tian, S\. Mei, D\. Li,et al\.\(2025\)Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 17393–17406\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p1.1)\.
- \[7\]W\. Huang, Y\. Liu, H\. Qin, Y\. Li, S\. Zhang, X\. Liu, M\. Magno, and X\. Qi\(2024\)BiLLM: Pushing the Limit of Post\-Training Quantization for LLMs\.arXiv preprint arXiv:2402\.04291\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p2.1),[§1](https://arxiv.org/html/2506.13771#S1.p3.1),[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2506.13771#S4.SS2.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2506.13771#S4.T3.12.1.1.3)\.
- \[8\]E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\(2022\)GPTQ: Accurate Post\-Training Quantization for Generative Pre\-Trained Transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p2.1),[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px3.p1.1)\.
- \[9\]J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han\(2024\)AWQ: Activation\-aware Weight Quantization for On\-Device LLM Compression and Acceleration\.Proceedings of Machine Learning and Systems6,pp\. 87–100\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p2.1),[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p2.1)\.
- \[10\]B\. Jacob, S\. Kligys, B\. Chen, M\. Zhu, M\. Tang, A\. Howard, H\. Adam, and D\. Kalenichenko\(2018\)Quantization and Training of Neural Networks for Efficient Integer\-Arithmetic\-Only Inference\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 2704–2713\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p2.1)\.
- \[11\]J\. Kim, Y\. Bhalgat, J\. Lee, C\. Patel, and N\. Kwak\(2019\)QKD: Quantization\-aware Knowledge Distillation\.arXiv preprint arXiv:1911\.12491\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p2.1)\.
- \[12\]H\. Wang, S\. Ma, L\. Dong, S\. Huang, H\. Wang, L\. Ma, F\. Yang, R\. Wang, Y\. Wu, and F\. Wei\(2023\)BitNet: Scaling 1\-bit Transformers for Large Language Models\.arXiv preprint arXiv:2310\.11453\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p2.1)\.
- \[13\]Y\. Xu, X\. Han, Z\. Yang, S\. Wang, Q\. Zhu, Z\. Liu, W\. Liu, and W\. Che\(2024\)OneBit: Towards Extremely Low\-bit Large Language Models\.Advances in Neural Information Processing Systems37,pp\. 66357–66382\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p2.1),[§1](https://arxiv.org/html/2506.13771#S1.p4.1),[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p3.1),[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2506.13771#S3.SS1.p1.3),[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2506.13771#S4.SS2.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2506.13771#S4.T3.12.1.1.4)\.
- \[14\]L\. Ma, M\. Sun, and Z\. Shen\(2024\)FBI\-LLM: Scaling up Fully Binarized LLMs from Scratch via Autoregressive Distillation\.arXiv preprint arXiv:2407\.07093\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p2.1)\.
- \[15\]Y\. Zheng, Y\. Chen, B\. Qian, X\. Shi, Y\. Shu, and J\. Chen\(2025\)A Review on Edge Large Language Models: Design, Execution, and Applications\.ACM Computing Surveys57\(8\),pp\. 1–35\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p3.1)\.
- \[16\]P\. Dong, L\. Li, Y\. Zhong, D\. Du, R\. Fan, Y\. Chen, Z\. Tang, Q\. Wang, W\. Xue, Y\. Guo,et al\.\(2024\)STBLLM: Breaking the 1\-bit Barrier with Structured Binary LLMs\.arXiv preprint arXiv:2408\.01803\.Cited by:[Table 8](https://arxiv.org/html/2506.13771#A3.T8),[3rd item](https://arxiv.org/html/2506.13771#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2506.13771#S1.p3.1),[§1](https://arxiv.org/html/2506.13771#S1.p6.1),[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2506.13771#S4.SS2.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2506.13771#S4.T1),[Table 2](https://arxiv.org/html/2506.13771#S4.T2),[Table 2](https://arxiv.org/html/2506.13771#S4.T2.5)\.
- \[17\]A\. Jaiswal, L\. Yin, Z\. Zhang, S\. Liu, J\. Zhao, Y\. Tian, and Z\. Wang\(2024\)From Low Rank Gradient Subspace Stabilization to Low\-Rank Weights: Observations, Theories, and Applications\.arXiv preprint arXiv:2407\.11239\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p4.1)\.
- \[18\]Q\. Wang, J\. Ke, M\. Tomizuka, Y\. Chen, K\. Keutzer, and C\. Xu\(2025\)Dobi\-SVD: Differentiable SVD for LLM Compression and Some New Perspectives\.arXiv preprint arXiv:2502\.02723\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p4.1)\.
- \[19\]G\. H\. Golub and C\. Reinsch\(1971\)Singular Value Decomposition and Least Squares Solutions\.InLinear algebra,pp\. 134–151\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p4.1),[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1)\.
- \[20\]S\. Han, H\. Mao, and W\. J\. Dally\(2015\)Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding\.arXiv preprint arXiv:1510\.00149\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p4.1),[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1)\.
- \[21\]X\. Wang, S\. Alam, Z\. Wan, H\. Shen, and M\. Zhang\(2025\)SVD\-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression\.arXiv preprint arXiv:2503\.12340\.Cited by:[Table 9](https://arxiv.org/html/2506.13771#A5.T9.5.1.1.3),[Appendix E](https://arxiv.org/html/2506.13771#A5.p1.1),[§1](https://arxiv.org/html/2506.13771#S1.p4.1),[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1)\.
- \[22\]M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. Kolter\(2023\)A Simple and Effective Pruning Approach for Large Language Models\.arXiv preprint arXiv:2306\.11695\.Cited by:[Table 9](https://arxiv.org/html/2506.13771#A5.T9.5.1.1.2),[Appendix E](https://arxiv.org/html/2506.13771#A5.p1.1),[§1](https://arxiv.org/html/2506.13771#S1.p4.1)\.
- \[23\]R\. Gong, Y\. Ding, Z\. Wang, C\. Lv, X\. Zheng, J\. Du, Y\. Yong, S\. Gu, H\. Qin, J\. Guo,et al\.\(2025\)A Survey of Low\-bit Large Language Models: Basics, Systems, and Algorithms\.Neural Networks,pp\. 107856\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p4.1),[§3\.1](https://arxiv.org/html/2506.13771#S3.SS1.p1.3)\.
- \[24\]D\. Jo, T\. Kim, Y\. Kim, and J\. Kim\(2024\)Mixture of Scales: Memory\-Efficient Token\-Adaptive Binarization for Large Language Models\.Advances in Neural Information Processing System37,pp\. 137474–137494\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p4.1),[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p3.1),[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px2.p1.2),[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2506.13771#S4.SS2.SSS0.Px2.p1.1)\.
- \[25\]Z\. Li, X\. Yan, T\. Zhang, H\. Qin, D\. Xie, J\. Tian, L\. Kong, Y\. Zhang, X\. Yang,et al\.\(2024\)ARB\-LLM: Alternating Refined Binarizations for Large Language Models\.arXiv preprint arXiv:2410\.03129\.Cited by:[§1](https://arxiv.org/html/2506.13771#S1.p4.1),[§3\.1](https://arxiv.org/html/2506.13771#S3.SS1.p1.3)\.
- \[26\]I\. Hubara, M\. Courbariaux, D\. Soudry, R\. El\-Yaniv, and Y\. Bengio\(2016\)Binarized Neural Networks\.Advances in Neural Information Processing Systems29\.Cited by:[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p1.1)\.
- \[27\]C\. Fu, S\. Zhu, H\. Su, C\. Lee, and J\. Zhao\(2018\)Towards Fast and Energy\-Efficient Binarized Neural Network Inference on FPGA\.arXiv preprint arXiv:1810\.02068\.Cited by:[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p1.1)\.
- \[28\]M\. Courbariaux, Y\. Bengio, and J\. David\(2015\)BinaryConnect: Training Deep Neural Networks with Binary Weights during Propagations\.Vol\.28\.Cited by:[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p1.1)\.
- \[29\]H\. Qin, R\. Gong, X\. Liu, X\. Bai, J\. Song, and N\. Sebe\(2020\)Binary Neural Networks: A Survey\.Pattern Recognition105,pp\. 107281\.Cited by:[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p1.1)\.
- \[30\]M\. Rastegari, V\. Ordonez, J\. Redmon, and A\. Farhadi\(2016\)XNOR\-Net: ImageNet Classification Using Binary Convolutional Neural Networks\.InEuropean Conference on Computer Vision,pp\. 525–542\.Cited by:[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p1.1)\.
- \[31\]Y\. Bengio, N\. Léonard, and A\. Courville\(2013\)Estimating or Propagating Gradients through Stochastic Neurons for Conditional Computation\.arXiv preprint arXiv:1308\.3432\.Cited by:[§B\.2](https://arxiv.org/html/2506.13771#A2.SS2.p1.1),[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px2.p1.2)\.
- \[32\]M\. Schoenbauer, D\. Moro, L\. Lew, and A\. Howard\(2024\)Custom Gradient Estimators are Straight\-Through Estimators in Disguise\.arXiv preprint arXiv:2405\.05171\.Cited by:[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p1.1)\.
- \[33\]Y\. Shang, Z\. Yuan, Q\. Wu, and Z\. Dong\(2023\)PB\-LLM: Partially Binarized Large Language Models\.arXiv preprint arXiv:2310\.00034\.Cited by:[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p2.1),[§3\.1](https://arxiv.org/html/2506.13771#S3.SS1.p1.3)\.
- \[34\]A\. Gholami, S\. Kim, Z\. Dong, Z\. Yao, M\. W\. Mahoney, and K\. Keutzer\(2022\)A Survey of Quantization Methods for Efficient Neural Network Inference\.InLow\-power computer vision,pp\. 291–326\.Cited by:[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p2.1)\.
- \[35\]M\. Denil, B\. Shakibi, L\. Dinh, M\. Ranzato, and N\. De Freitas\(2013\)Predicting Parameters in Deep Learning\.Advances in Neural Information Processing Systems26\.Cited by:[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1)\.
- \[36\]Z\. Liu, B\. Oguz, C\. Zhao, E\. Chang, P\. Stock, Y\. Mehdad, Y\. Shi, R\. Krishnamoorthi, and V\. Chandra\(2024\)LLM\-QAT: Data\-Free Quantization Aware Training for Large Language Models\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 467–484\.Cited by:[§2\.1](https://arxiv.org/html/2506.13771#S2.SS1.p3.1),[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px2.p1.1)\.
- \[37\]X\. Men, M\. Xu, Q\. Zhang, Q\. Yuan, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. Chen\(2025\)ShortGPT: Layers in Large Language Models are More Redundant Than You Expect\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 20192–20204\.Cited by:[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1)\.
- \[38\]S\. He, G\. Sun, Z\. Shen, and A\. Li\(2024\)What Matters in Transformers? Not All Attention is Needed\.arXiv preprint arXiv:2406\.15786\.Cited by:[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1)\.
- \[39\]R\. Saha, N\. Sagan, V\. Srivastava, A\. Goldsmith, and M\. Pilanci\(2024\)Compressing Large Language Models using Low Rank and Low Precision Decomposition\.Advances in Neural Information Processing Systems37,pp\. 88981–89018\.Cited by:[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1)\.
- \[40\]Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao\(2023\)AdaLoRA: Adaptive Budget Allocation for Parameter\-Efficient Fine\-Tuning\.arXiv preprint arXiv:2303\.10512\.Cited by:[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1)\.
- \[41\]F\. Meng, Z\. Wang, and M\. Zhang\(2024\)PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models\.Advances in Neural Information Processing Systems37,pp\. 121038–121072\.Cited by:[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1)\.
- \[42\]Z\. Yuan, Y\. Shang, Y\. Song, D\. Yang, Q\. Wu, Y\. Yan, and G\. Sun\(2023\)ASVD: Activation\-aware Singular Value Decomposition for Compressing Large Language Models\.arXiv preprint arXiv:2312\.05821\.Cited by:[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1),[§5](https://arxiv.org/html/2506.13771#S5.SS0.SSS0.Px2.p1.1)\.
- \[43\]C\. Zhang, J\. Cheng, G\. A\. Constantinides, and Y\. Zhao\(2024\)LQER: Low\-Rank Quantization Error Reconstruction for LLMs\.arXiv preprint arXiv:2402\.02446\.Cited by:[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p1.1)\.
- \[44\]W\. Ke, Z\. Li, D\. Li, L\. Tian, and E\. Barsoum\(2024\)DL\-QAT: Weight\-Decomposed Low\-Rank Quantization\-Aware Training for Large Language Models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,pp\. 113–119\.Cited by:[§2\.2](https://arxiv.org/html/2506.13771#S2.SS2.p2.1)\.
- \[45\]X\. Wang, Y\. Zheng, Z\. Wan, and M\. Zhang\(2024\)SVD\-LLM: Truncation\-aware Singular Value Decomposition for Large Language Model Compression\.arXiv preprint arXiv:2403\.07378\.Cited by:[§3\.1](https://arxiv.org/html/2506.13771#S3.SS1.p1.1)\.
- \[46\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)LLaMA: Open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[47\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: Open Foundation and Fine\-tuned Chat Models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2506.13771#S4.T3)\.
- \[48\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The Llama 3 Herd of Models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[49\]S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin,et al\.\(2022\)OPT: Open Pre\-trained Transformer Language Models\.arXiv preprint arXiv:2205\.01068\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[50\]M\. Abdin, J\. Aneja, H\. Behl, S\. Bubeck, R\. Eldan, S\. Gunasekar, M\. Harrison, R\. J\. Hewett, M\. Javaheripi, P\. Kauffmann,et al\.\(2024\)Phi\-4 Technical Report\.arXiv preprint arXiv:2412\.08905\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[51\]Q\. Team\(2025\)QwQ\-32B: Embracing the Power of Reinforcement Learning\.External Links:[Link](https://qwenlm.github.io/blog/qwq-32b/)Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[52\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\(2016\)Pointer Sentinel Mixture Models\.arXiv preprint arXiv:1609\.07843\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[53\]C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu\(2020\)Exploring the Limits of Transfer Learning with a Unified Text\-to\-Text Transformer\.Journal of Machine Learning Research21\(140\),pp\. 1–67\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[54\]M\. Marcus, G\. Kim, M\. A\. Marcinkiewicz, R\. MacIntyre, A\. Bies, M\. Ferguson, K\. Katz, and B\. Schasberger\(1994\)The Penn Treebank: Annotating Predicate Argument Structure\.InProceedings of the Workshop on Human Language Technology,HLT ’94,pp\. 114–119\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[55\]K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi\(2021\)WinoGrande: An Adversarial Winograd Schema Challenge at Scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[56\]T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal\(2018\)Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering\.arXiv preprint arXiv:1809\.02789\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[57\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\(2019\)HellaSwag: Can a Machine Really Finish Your Sentence?\.arXiv preprint arXiv:1905\.07830\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[58\]C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova\(2019\)BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions\.arXiv preprint arXiv:1905\.10044\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[59\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[60\]Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)PIQA: Reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px1.p1.1)\.
- \[61\]G\. Hinton, O\. Vinyals, and J\. Dean\(2015\)Distilling the Knowledge in a Neural Network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px2.p1.1)\.
- \[62\]W\. Zhang, L\. Hou, Y\. Yin, L\. Shang, X\. Chen, X\. Jiang, and Q\. Liu\(2020\)TernaryBERT: Distillation\-aware Ultra\-low Bit BERT\.arXiv preprint arXiv:2009\.12812\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px2.p1.1)\.
- \[63\]D\. P\. Kingma and J\. Ba\(2015\)Adam: A Method for Stochastic Optimization\.InProceedings of the Third International Conference on Learning Representations \(ICLR\),Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px2.p1.2)\.
- \[64\]J\. Ainslie, J\. Lee\-Thorp, M\. De Jong, Y\. Zemlyanskiy, F\. Lebrón, and S\. Sanghai\(2023\)GQA: Training Generalized Multi\-Query Transformer Models from Multi\-Head Checkpoints\.arXiv preprint arXiv:2305\.13245\.Cited by:[§B\.3](https://arxiv.org/html/2506.13771#A2.SS3.p1.1),[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px2.p1.2)\.
- \[65\]W\. Shao, M\. Chen, Z\. Zhang, P\. Xu, L\. Zhao, Z\. Li, K\. Zhang, P\. Gao, Y\. Qiao, and P\. Luo\(2023\)OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models\.arXiv preprint arXiv:2308\.13137\.Cited by:[§4\.1](https://arxiv.org/html/2506.13771#S4.SS1.SSS0.Px3.p1.1)\.
- \[66\]DeepSeek\-AI\(2024\)DeepSeek\-V2: A Strong Mixture\-of\-Experts Language Model with Expert Attention\.arXiv preprint arXiv:2405\.04434\.Cited by:[§5](https://arxiv.org/html/2506.13771#S5.SS0.SSS0.Px2.p1.1)\.
- \[67\]N\. Shazeer\(2019\)Fast Transformer Decoding: One Write\-Head is All You Need\.arXiv preprint arXiv:1911\.02150\.Cited by:[§B\.3](https://arxiv.org/html/2506.13771#A2.SS3.p1.1)\.

## Appendix AMathematical Analysis and Proofs

This section provides mathematical details for theLittleBitcomputation and explores theoretical aspects related to low\-rank quantization errors, offering motivation for certain design choices\.

###### Proposition 1\(LittleBitForward Pass Computation \(Proof Detail\)\)\.

As stated in[Section3\.1](https://arxiv.org/html/2506.13771#S3.SS1)\(Proposition 1\), for input𝐗∈ℝseq×din\\mathbf\{X\}\\in\\mathbb\{R\}^\{\\mathrm\{seq\}\\times d\_\{\\mathrm\{in\}\}\}, the primary quantized weight matrix is𝐖^pri=diag⁡\(𝐡\)​𝐔sign​diag​\(ℓ\)​𝐕sign⊤​diag​\(𝐠\)\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\}\}=\\mathrm\{diag\}\(\\mathbf\{h\}\)\\mathbf\{U\}\_\{\\mathrm\{sign\}\}\\mathrm\{diag\}\(\\bm\{\\ell\}\)\\mathbf\{V\}\_\{\\mathrm\{sign\}\}^\{\\top\}\\mathrm\{diag\}\(\\mathbf\{g\}\)\. The forward pass𝐘=𝐗​𝐖^pri⊤\\mathbf\{Y\}=\\mathbf\{X\}\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\}\}^\{\\top\}can be computed as shown in[Eq\.12](https://arxiv.org/html/2506.13771#A1.E12)\.

𝐘=\(\(\(\(𝐗⊙𝐠\)​𝐕sign\)⊙ℓ\)​𝐔sign⊤\)⊙𝐡\.\\mathbf\{Y\}=\(\(\(\(\\mathbf\{X\}\\odot\\mathbf\{g\}\)\\mathbf\{V\}\_\{\\mathrm\{sign\}\}\)\\odot\\bm\{\\ell\}\)\\mathbf\{U\}\_\{\\mathrm\{sign\}\}^\{\\top\}\)\\odot\\mathbf\{h\}\.\(12\)Terms are defined in[Section3\.1](https://arxiv.org/html/2506.13771#S3.SS1)\.

###### Proof\.

Starting with𝐘=𝐗​𝐖^pri⊤\\mathbf\{Y\}=\\mathbf\{X\}\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\}\}^\{\\top\}:

𝐘\\displaystyle\\mathbf\{Y\}=𝐗​\(diag⁡\(𝐡\)​𝐔sign​diag​\(ℓ\)​𝐕sign⊤​diag​\(𝐠\)\)⊤\\displaystyle=\\mathbf\{X\}\\left\(\\mathrm\{diag\}\(\\mathbf\{h\}\)\\mathbf\{U\}\_\{\\mathrm\{sign\}\}\\mathrm\{diag\}\(\\bm\{\\ell\}\)\\mathbf\{V\}\_\{\\mathrm\{sign\}\}^\{\\top\}\\mathrm\{diag\}\(\\mathbf\{g\}\)\\right\)^\{\\top\}=𝐗​diag​\(𝐠\)​𝐕sign​diag​\(ℓ\)​𝐔sign⊤​diag​\(𝐡\)\\displaystyle=\\mathbf\{X\}\\,\\mathrm\{diag\}\(\\mathbf\{g\}\)\\mathbf\{V\}\_\{\\mathrm\{sign\}\}\\mathrm\{diag\}\(\\bm\{\\ell\}\)\\mathbf\{U\}\_\{\\mathrm\{sign\}\}^\{\\top\}\\mathrm\{diag\}\(\\mathbf\{h\}\)=\(\(\(\(𝐗⊙𝐠\)​𝐕sign\)⊙ℓ\)​𝐔sign⊤\)⊙𝐡\.\\displaystyle=\(\(\(\(\\mathbf\{X\}\\odot\\mathbf\{g\}\)\\mathbf\{V\}\_\{\\mathrm\{sign\}\}\)\\odot\\bm\{\\ell\}\)\\mathbf\{U\}\_\{\\mathrm\{sign\}\}^\{\\top\}\)\\odot\\mathbf\{h\}\.\(13\)This yields[Eq\.12](https://arxiv.org/html/2506.13771#A1.E12), where⊙\\odotimplies element\-wise multiplication with broadcasting\. ∎

###### Claim 1\(Quantization Error vs\. Factor Rank\)\.

Consider a simplified model where a rank\-rrmatrix𝐖\(r\)=𝐔\(r\)​\(𝐕\(r\)\)⊤\\mathbf\{W\}^\{\(r\)\}=\\mathbf\{U\}^\{\(r\)\}\(\\mathbf\{V\}^\{\(r\)\}\)^\{\\top\}is approximated\. Let𝐔sign\(r\)=sign⁡\(𝐔\(r\)\)\\mathbf\{U\}\_\{\\mathrm\{sign\}\}^\{\(r\)\}=\\mathrm\{sign\}\(\\mathbf\{U\}^\{\(r\)\}\),𝐕sign\(r\)=sign⁡\(𝐕\(r\)\)\\mathbf\{V\}\_\{\\mathrm\{sign\}\}^\{\(r\)\}=\\mathrm\{sign\}\(\\mathbf\{V\}^\{\(r\)\}\)\. If magnitudes are crudely approximated using fixed rank\-1 SVDs of\|𝐔\(r\)\|\|\\mathbf\{U\}^\{\(r\)\}\|and\|𝐕\(r\)\|\|\\mathbf\{V\}^\{\(r\)\}\|to form scales𝐬U\(r\),𝐬V\(r\)\\mathbf\{s\}\_\{U\}^\{\(r\)\},\\mathbf\{s\}\_\{V\}^\{\(r\)\}:

𝐖^r:=\(𝐔sign\(r\)⊙\(𝐬U\(r\)​𝟏r⊤\)\)​\(𝐕sign\(r\)⊙\(𝐬V\(r\)​𝟏r⊤\)\)⊤\.\\widehat\{\\mathbf\{W\}\}\_\{r\}:=\\left\(\\mathbf\{U\}\_\{\\mathrm\{sign\}\}^\{\(r\)\}\\odot\(\\mathbf\{s\}\_\{U\}^\{\(r\)\}\\mathbf\{1\}^\{\\top\}\_\{r\}\)\\right\)\\left\(\\mathbf\{V\}\_\{\\mathrm\{sign\}\}^\{\(r\)\}\\odot\(\\mathbf\{s\}\_\{V\}^\{\(r\)\}\\mathbf\{1\}^\{\\top\}\_\{r\}\)\\right\)^\{\\top\}\.\(14\)The errorℰ⁡\(r\):=‖𝐖\(r\)−𝐖^r‖F\\mathcal\{E\}\(r\):=\\left\\\|\\mathbf\{W\}^\{\(r\)\}\-\\widehat\{\\mathbf\{W\}\}\_\{r\}\\right\\\|\_\{F\}might be non\-decreasing withrr\.

###### Proof Sketch / Heuristic Argument\.

As rankrr\(and thus complexity of𝐖\(r\)\\mathbf\{W\}^\{\(r\)\},𝐔\(r\)\\mathbf\{U\}^\{\(r\)\},𝐕\(r\)\\mathbf\{V\}^\{\(r\)\}\) increases, the fixed\-structure rank\-1 magnitude approximation \(via𝐬U\(r\),𝐬V\(r\)\\mathbf\{s\}\_\{U\}^\{\(r\)\},\\mathbf\{s\}\_\{V\}^\{\(r\)\}\) may become a relatively poorer fit for the increasingly complex actual magnitudes of𝐔\(r\),𝐕\(r\)\\mathbf\{U\}^\{\(r\)\},\\mathbf\{V\}^\{\(r\)\}\. This degradation in scaling accuracy could lead toℰ⁡\(r\)\\mathcal\{E\}\(r\)not decreasing, or even increasing, motivating more sophisticated scaling as inLittleBit\. ∎

###### Claim 2\(Quantization Bias vs\. SVD Component Structure\)\.

Let𝐖\\mathbf\{W\}’s rank\-rrSVD approximation be𝐖~r=𝐖~r1\+𝐑~r2\\widetilde\{\\mathbf\{W\}\}\_\{r\}=\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\}\+\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}\(primary𝐖~r1\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\}, residual𝐑~r2\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}\)\. Consider a quantization operator𝒬\\mathcal\{Q\}\. The relative quantization error‖𝒬⁡\(𝐀\)−𝐀‖F/‖𝐀‖F\\\|\\mathcal\{Q\}\(\\mathbf\{A\}\)\-\\mathbf\{A\}\\\|\_\{F\}/\\\|\\mathbf\{A\}\\\|\_\{F\}might be larger for matrices𝐀\\mathbf\{A\}with "flatter" singular value spectra or less energy concentration \(e\.g\.,𝐑~r2\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}vs\.𝐖~r1\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\}\)\. This suggests that𝒬\\mathcal\{Q\}applied to𝐑~r2\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}might be less effective \(i\.e\., higher relative quantization error\) than when applied to𝐖~r1\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\}\.

###### Proof Sketch\.

𝐖~r1\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\}has dominant singular values, while𝐑~r2\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}has smaller ones, often resulting in lower energy concentration for𝐑~r2\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}\. If𝒬\\mathcal\{Q\}’s relative error is sensitive to this \(e\.g\., higher for lower concentration\), quantizing𝐖~r1\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\}and𝐑~r2\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}separately may be better than quantizing𝐖~r\\widetilde\{\\mathbf\{W\}\}\_\{r\}jointly, as it allows adapting𝒬\\mathcal\{Q\}to their differing structures\. ∎

###### Proposition 2\(Potential Advantage of Two\-Stage Quantization of SVD Components\)\.

Using the setup of[Claim2](https://arxiv.org/html/2506.13771#Thmclaim2), let𝐖^rsingle=𝒬⁡\(𝐖~r\)\\widehat\{\\mathbf\{W\}\}\_\{r\}^\{\\mathrm\{single\}\}=\\mathcal\{Q\}\(\\widetilde\{\\mathbf\{W\}\}\_\{r\}\)and𝐖^two−stage=𝒬⁡\(𝐖~r1\)\+𝒬⁡\(𝐑~r2\)\\widehat\{\\mathbf\{W\}\}^\{\\mathrm\{two\-stage\}\}=\\mathcal\{Q\}\(\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\}\)\+\\mathcal\{Q\}\(\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}\)\. The two\-stage errorℰtwo−stage=‖𝐖−𝐖^two−stage‖F\\mathcal\{E\}\_\{\\mathrm\{two\-stage\}\}=\\left\\\|\\mathbf\{W\}\-\\widehat\{\\mathbf\{W\}\}^\{\\mathrm\{two\-stage\}\}\\right\\\|\_\{F\}can be less than the single\-stage errorℰsingle−stage=‖𝐖−𝐖^rsingle‖F\\mathcal\{E\}\_\{\\mathrm\{single\-stage\}\}=\\left\\\|\\mathbf\{W\}\-\\widehat\{\\mathbf\{W\}\}\_\{r\}^\{\\mathrm\{single\}\}\\right\\\|\_\{F\}\. This occurs if the quantization error of the sum is greater than the sum of \(vectorial\) quantization errors of parts:

‖\(𝒬⁡\(𝐖~r1\)−𝐖~r1\)\+\(𝒬⁡\(𝐑~r2\)−𝐑~r2\)‖F<‖𝒬⁡\(𝐖~r\)−𝐖~r‖F\.\\\|\(\\mathcal\{Q\}\(\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\}\)\-\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\}\)\+\(\\mathcal\{Q\}\(\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}\)\-\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}\)\\\|\_\{F\}<\\\|\\mathcal\{Q\}\(\\widetilde\{\\mathbf\{W\}\}\_\{r\}\)\-\\widetilde\{\\mathbf\{W\}\}\_\{r\}\\\|\_\{F\}\.\(15\)

###### Proof Outline and Discussion\.

Let𝐄trunc=𝐖−𝐖~r\\mathbf\{E\}\_\{\\mathrm\{trunc\}\}=\\mathbf\{W\}\-\\widetilde\{\\mathbf\{W\}\}\_\{r\}\. Errors areΔ1=𝒬⁡\(𝐖~r1\)−𝐖~r1\\Delta\_\{1\}=\\mathcal\{Q\}\(\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\}\)\-\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\},Δ2=𝒬⁡\(𝐑~r2\)−𝐑~r2\\Delta\_\{2\}=\\mathcal\{Q\}\(\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}\)\-\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\},Δsum=𝒬⁡\(𝐖~r\)−𝐖~r\\Delta\_\{\\mathrm\{sum\}\}=\\mathcal\{Q\}\(\\widetilde\{\\mathbf\{W\}\}\_\{r\}\)\-\\widetilde\{\\mathbf\{W\}\}\_\{r\}\. Thenℰtwo−stage=‖𝐄trunc−\(Δ1\+Δ2\)‖F\\mathcal\{E\}\_\{\\mathrm\{two\-stage\}\}=\\\|\\mathbf\{E\}\_\{\\mathrm\{trunc\}\}\-\(\\Delta\_\{1\}\+\\Delta\_\{2\}\)\\\|\_\{F\}andℰsingle−stage=‖𝐄trunc−Δsum‖F\\mathcal\{E\}\_\{\\mathrm\{single\-stage\}\}=\\\|\\mathbf\{E\}\_\{\\mathrm\{trunc\}\}\-\\Delta\_\{\\mathrm\{sum\}\}\\\|\_\{F\}\. The condition[Eq\.15](https://arxiv.org/html/2506.13771#A1.E15)makesℰtwo−stage<ℰsingle−stage\\mathcal\{E\}\_\{\\mathrm\{two\-stage\}\}<\\mathcal\{E\}\_\{\\mathrm\{single\-stage\}\}likely\. Non\-linear𝒬\\mathcal\{Q\}\(e\.g\., with adaptive scaling\) can satisfy this by better handling the distinct characteristics of𝐖~r1\\widetilde\{\\mathbf\{W\}\}\_\{r\_\{1\}\}and𝐑~r2\\widetilde\{\\mathbf\{R\}\}\_\{r\_\{2\}\}\. This motivatesLittleBit’s separate residual handling, though its residual is learned differently\. ∎

## Appendix BAblation Study

### B\.1Residual Compensation

To provide further intuition behind the benefits of Residual Compensation, we visualize the output activations of selected Transformer layers under different quantization schemes\. Specifically, we extract the final outputs from Transformer layers 0, 5, 10, and 15 of an OPT\-1\.3B model, evaluated on two randomly sampled input sequences\. For each layer and input, we display a set of three activation maps corresponding to the full\-precision baseline, as well as two quantized variants—with and without residual compensation\. These groupings allow us to directly observe the structural differences induced by the presence or absence of the residual compensation\.

As shown in[Fig\.7](https://arxiv.org/html/2506.13771#A2.F7), residual compensation preserves a high\-fidelity resemblance to the full\-precision baseline across all inspected layers\. In the no\-residual setting, the output feature distributions exhibit noticeable deformation relative to the full\-precision baseline\. In contrast, applying residual compensation consistently restores structural similarity to the original distribution, preserving both directional patterns and magnitude diversity\.

[Table5](https://arxiv.org/html/2506.13771#A2.T5)presents the PPL results for the OPT\-1\.3B model, comparing configurations with and without residual compensation\. As indicated in the table, at the extremely low bit of 0\.1 BPW, the version with residual compensation \(PPL 60\.011\) performed worse than the version without it \(PPL 48\.512\)\. This suggests that for relatively small models \(1\.3B\), when pushing effective bits to such extreme lows, the additional complexity introduced by the residual compensation mechanism might outweigh its benefits and instead hinder performance\. However, for the same OPT\-1\.3B model at other effective bits evaluated \(ranging from 0\.3 BPW to 1\.0 BPW\), the inclusion of residual compensation consistently resulted in better \(lower\) PPL\. Moreover, experiments conducted with other, larger models generally showed that residual compensation tended to provide superior performance\. In light of these overall advantages observed across various settings and particularly for larger models, residual compensation was adopted as a standard component in all experiments presented in this study\.

Figure 7:Visualization of Transformer layer outputs for layers 0, 5, 10, and 15 under theLittleBitquantization scheme on OPT\-1\.3B\. For each of two randomly sampled input sequences \(annotated at the top\), we show horizontally grouped triplets of activation maps per layer: full\-precision baseline \(left\), quantized with residual compensation \(center\), and quantized without residual compensation \(right\)\.Table 5:Perplexity \(WikiText\-2\) comparison between Residual and Non\-Residual for OPT\-1\.3B at various BPWs\. Learning rate was 8e\-5\. Lower PPL is better\.
### B\.2SmoothSign versus Straight\-Through Estimator

For backpropagating through the non\-differentiablesign⁡\(x\)\\mathrm\{sign\}\(x\)function, we compared the Straight\-Through Estimator \(STE\)\[[31](https://arxiv.org/html/2506.13771#bib.bib28)\]with our proposedSmoothSigntechnique\. SmoothSign employs thesign⁡\(x\)\\mathrm\{sign\}\(x\)function for the forward pass\. For the backward pass, it utilizes a smooth proxy gradient, specifically the derivative oftanh⁡\(k​x\)\\tanh\(kx\)wherek=100k=100\(illustrated in[Fig\.8](https://arxiv.org/html/2506.13771#A2.F8)\)\.

[Table6](https://arxiv.org/html/2506.13771#A2.T6)\(OPT\-1\.3B results\) shows SmoothSign yielding better PPL as effective bits decrease, with a notable advantage at 0\.1 BPW\. Due to its superior stability and performance in the ultra\-low bit regime, SmoothSign was adopted for all QAT experiments \(consistent with[Section4\.1](https://arxiv.org/html/2506.13771#S4.SS1)\)\.

\(a\) Forward pass:sign⁡\(x\)\\mathrm\{sign\}\(x\)andtanh⁡\(100​x\)\\tanh\(100x\)

\(b\) Backward pass: gradientdd​x​tanh⁡\(100​x\)\\frac\{d\}\{dx\}\\tanh\(100x\)

Figure 8:The SmoothSign activation technique\. \(a\) Functions relevant to the forward pass: SmoothSign uses thesign⁡\(x\)\\mathrm\{sign\}\(x\)function \(actual forward pass\)\. The plot also showstanh⁡\(100​x\)\\tanh\(100x\), which serves as the basis for deriving the smooth proxy gradient shown in \(b\)\. \(b\) The smooth proxy gradient,dd​x​tanh⁡\(100​x\)\\frac\{d\}\{dx\}\\tanh\(100x\), utilized for the backward pass\.Table 6:Perplexity \(WikiText\-2\) comparison between STE and SmoothSign for OPT\-1\.3B at various effective bits \(BPW\)\. Learning rate was 8e\-5\. Lower PPL is better\.
### B\.3Grouped Query Attention

Table 7:Ablation study on adjusting the latent dimension factor \(rr\) for key \(K\) and value \(V\) projection layers in Llama3\-8B with GQA, targeting an overall effective bit of 0\.1 BPW\. Performance measured by perplexity \(PPL, lower is better\) on WikiText\-2 and C4 validation sets, and average zero\-shot accuracy \(%\) on common sense reasoning tasks \(higher is better\)\. The ’KV Factor’ indicates the multiplier applied to the K/V layers’ latent rankrrcompared to the standard calculation for 0\.1 BPW\.Modern LLMs often use Grouped Query Attention \(GQA\)\[[64](https://arxiv.org/html/2506.13771#bib.bib66)\]or Multi\-Query Attention \(MQA\)\[[67](https://arxiv.org/html/2506.13771#bib.bib67)\], where key \(K\) and value \(V\) projection layers have smaller output dimensions than query \(Q\) projections\. ApplyingLittleBitat ultra\-low bits \(e\.g\., 0\.1 BPW\) to these GQA/MQA models can result in an extremely small latent dimensionrrfor K/V layers \(e\.g\.,r≈20r\\approx 20for Llama3\-8B K/V layers at 0\.1 BPW overall\), potentially creating an information bottleneck and hindering QAT or performance\.

To investigate, we performed an ablation on Llama3\-8B \(∼\\sim0\.1 BPW target\), increasing the latent rankrrfor K/V layers by 2×\\times, 4×\\times, and 8x over the standard calculation \([AppendixD](https://arxiv.org/html/2506.13771#A4)\), while other layers remained unchanged\.[Table7](https://arxiv.org/html/2506.13771#A2.T7)shows that a 4x factor for K/V rank yielded the best PPL on WikiText\-2 and C4, with minimal impact on overall BPW \(e\.g\.,∼\\sim0\.098 BPW to∼\\sim0\.107 BPW for 4×\\times\)\. Average zero\-shot accuracy also remained competitive\. An 8x factor showed diminishing returns\.

Given the significant PPL improvement with the 4×\\timesK/V rank factor \(nearly 1 point on WikiText\-2 vs\. 1×\\times\) at a negligible BPW cost \(<10% relative increase in effective bits for transformer layers, <4% of total parameters\), this strategy was adopted for GQA/MQA models \(Llama3\-8B, Phi\-4, QwQ\-32B\) in our main results\. For these models, the latent rankrrfor K and V projection layers was calculated based on a 4×\\timesmultiplier compared to the standard calculation for the target BPW, while all other layers used the standard rank calculation\. This approach helps mitigate potential information bottlenecks in GQA/MQA layers under extreme quantization without significantly altering the target compression level\.

## Appendix CPerplexity on C4 and PTB

To further assessLittleBit’s generalization, we evaluated its performance on additional benchmarks: the diverse C4 web text corpus and the distinct PTB news corpus \([Table8](https://arxiv.org/html/2506.13771#A3.T8)\)\. Focusing on Llama2\-7B/13B against STBLLM, the results on these datasets confirm the trends observed on WikiText\-2\.LittleBitconsistently achieves lower perplexity than STBLLM at comparable sub\-1\-bit regimes\. This performance advantage becomes more significant below 0\.55 BPW, where STBLLM’s performance degrades considerably, particularly on PTB\.

Notably,LittleBitmaintains robust performance and stability even at the extreme 0\.1 BPW level across all tested datasets\. This consistent ability to maintain strong language modeling capabilities under severe compression highlights the generalizability of our approach, which integrates latent factorization, multi\-scale compensation, and QAT\. It reinforcesLittleBit’s potential for effectively deploying capable LLMs in resource\-limited environments\.

Table 8:Perplexity \(PPL\) comparison on C4 and PTB validation sets for Llama2 models\. Lower PPL indicates better performance\. STBLLM\[[16](https://arxiv.org/html/2506.13771#bib.bib30)\]utilizes N:M sparsity \(ratio in parentheses\)\.
## Appendix DAverage Bits Per Weight Calculation

This section details the calculation of the average effective bits per weight \(which we denote asbbin the following equations\) for a linear layer implemented using theLittleBitarchitecture, including the parameters from the primary \(𝐖^pri\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{pri\}\}\) and residual compensation \(𝐖^res\\widehat\{\\mathbf\{W\}\}\_\{\\mathrm\{res\}\}\) pathways, as described in[Sections3\.1](https://arxiv.org/html/2506.13771#S3.SS1)and[3\.3](https://arxiv.org/html/2506.13771#S3.SS3)\. The calculation demonstrates how to determine the latent rankrrrequired to achieve a target value forbbwithin the range of approximately 0\.1 to 1\.0\.

ALittleBitlinear layer replaces the original weight matrix𝐖∈ℝdout×din\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}with two parallel structures, each comprising:

- •Two binary sign matrices:𝐔sign∈\{±1\}dout×r\\mathbf\{U\}\_\{\\mathrm\{sign\}\}\\in\\\{\\pm 1\\\}^\{d\_\{\\mathrm\{out\}\}\\times r\}and𝐕sign∈\{±1\}din×r\\mathbf\{V\}\_\{\\mathrm\{sign\}\}\\in\\\{\\pm 1\\\}^\{d\_\{\\mathrm\{in\}\}\\times r\}\. These require11bit per element\.
- •Three FP16 scaling vectors:𝐡∈ℝdout\\mathbf\{h\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\},𝐠∈ℝdin\\mathbf\{g\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\}, andℓ∈ℝr\\bm\{\\ell\}\\in\\mathbb\{R\}^\{r\}\. These require1616bits per element\.

Since both the primary and residual paths have this structure, the total number of bits required to store the parameters for aLittleBitlayer is:

Total Bits=2×\(Bits for​𝐔sign\+Bits for​𝐕sign\+Bits for​𝐡\+Bits for​𝐠\+Bits for​ℓ\)\\displaystyle=2\\times\\left\(\\text\{Bits for \}\\mathbf\{U\}\_\{\\mathrm\{sign\}\}\+\\text\{Bits for \}\\mathbf\{V\}\_\{\\mathrm\{sign\}\}\+\\text\{Bits for \}\\mathbf\{h\}\+\\text\{Bits for \}\\mathbf\{g\}\+\\text\{Bits for \}\\bm\{\\ell\}\\right\)=2×\(\(dout×r×1\)\+\(din×r×1\)\+\(dout×16\)\+\(din×16\)\+\(r×16\)\)\\displaystyle=2\\times\\left\(\(d\_\{\\mathrm\{out\}\}\\times r\\times 1\)\+\(d\_\{\\mathrm\{in\}\}\\times r\\times 1\)\+\(d\_\{\\mathrm\{out\}\}\\times 16\)\+\(d\_\{\\mathrm\{in\}\}\\times 16\)\+\(r\\times 16\)\\right\)=2​r​\(dout\+din\)\+32​\(dout\+din\)\+32​r\\displaystyle=2r\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)\+32\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)\+32rNote: If Residual Compensation is not used, the initial factor of 2 should be removed from the calculation\.

The average bits per weight, denoted asbb, is calculated by dividing the total bits by the number of parameters in the original FP16 weight matrix \(dout×dind\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\):

b=2​r​\(dout\+din\)\+32​\(dout\+din\)\+32​rdout×dinb=\\frac\{2r\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)\+32\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)\+32r\}\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}\(16\)
To achieve a target value forbb, we rearrange[Eq\.16](https://arxiv.org/html/2506.13771#A4.E16)to solve for the required latent rankrr:

r=\(b×dout×din\)−32​\(dout\+din\)2​\(dout\+din\)\+32r=\\frac\{\(b\\times d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\)\-32\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)\}\{2\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)\+32\}\(17\)
Sincerrmust be an integer, we typically round the calculated value to the nearest suitable integer\. This chosen integerrrthen determines the actual value ofbbachieved, which will be very close to the target value\.

- •Example 1: Linear Layer \(4,096×4,096$4,096$\\times$4,096$\)Letdout=4,096d\_\{\\mathrm\{out\}\}=$4,096$anddin=4,096d\_\{\\mathrm\{in\}\}=$4,096$\. The original number of parameters is4,096×4,096=16,777,216$4,096$\\times$4,096$=$16,777,216$\.dout\+din=8,192d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}=$8,192$\. For a targetb≈0\.55b\\approx 0\.55: Using[Eq\.17](https://arxiv.org/html/2506.13771#A4.E17)withb=0\.55b=0\.55: r=\(0\.55×16,777,216\)−32​\(8,192\)2​\(8,192\)\+32≈546r=\\frac\{\(0\.55\\times$16,777,216$\)\-32\($8,192$\)\}\{2\($8,192$\)\+32\}\\approx 546We choose integerr=546r=546\. The actual value ofbbforr=546r=546, using[Eq\.16](https://arxiv.org/html/2506.13771#A4.E16), is: b=2​\(546\)​\(8,192\)\+32​\(8,192\)\+32​\(546\)16,777,216≈0\.5498b=\\frac\{2\(546\)\($8,192$\)\+32\($8,192$\)\+32\(546\)\}\{$16,777,216$\}\\approx 0\.5498
- •Example 2: Linear Layer \(4,096×11,008$4,096$\\times$11,008$\)Letdout=4,096d\_\{\\mathrm\{out\}\}=$4,096$anddin=11,008d\_\{\\mathrm\{in\}\}=$11,008$\. The original number of parameters is4,096×11,008=45,090,816$4,096$\\times$11,008$=$45,090,816$\.dout\+din=15,104d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}=$15,104$\. For a targetb≈0\.1b\\approx 0\.1: Using[Eq\.17](https://arxiv.org/html/2506.13771#A4.E17)withb=0\.1b=0\.1: r=\(0\.1×45,090,816\)−32​\(15,104\)2​\(15,104\)\+32≈133r=\\frac\{\(0\.1\\times$45,090,816$\)\-32\($15,104$\)\}\{2\($15,104$\)\+32\}\\approx 133We choose integerr=133r=133\. The actual value ofbbforr=133r=133, using[Eq\.16](https://arxiv.org/html/2506.13771#A4.E16), is: b=2​\(133\)​\(15,104\)\+32​\(15,104\)\+32​\(133\)45,090,816≈0\.0999b=\\frac\{2\(133\)\($15,104$\)\+32\($15,104$\)\+32\(133\)\}\{$45,090,816$\}\\approx 0\.0999

By following this procedure for each linear layer in the model, we can select appropriate ranks \(rr\) to achieve a desired overall value forbb\(average bits per weight\), thereby controlling the trade\-off between model compression and performance\. The specific ranks used to achieve the average bits per weight \(BPW\) figures reported in the main paper \(e\.g\., in[Tables1](https://arxiv.org/html/2506.13771#S4.T1)and[3](https://arxiv.org/html/2506.13771#S4.T3)\) are determined using this calculation methodology forbbfor each layer type within the respective models\.

## Appendix EPruning versus SVD

To select a base compression method to achieve sub\-0\.5 BPW, we compared pruning \(Wanda\[[22](https://arxiv.org/html/2506.13771#bib.bib14)\]\) with SVD\-based compression \(SVD\-LLMv2\[[21](https://arxiv.org/html/2506.13771#bib.bib32)\]\) on Llama\-7B \([Table9](https://arxiv.org/html/2506.13771#A5.T9)\)\. SVD showed superior robustness at extreme compression ratios \(e\.g\., 25% parameter retention\)\.

Consequently, low\-rank factorization was chosen to designLittleBitdue to its performance resilience and the deployment advantages of dense factorized matrices over sparse pruned models \(which mayrequire specialized hardwareor overhead\)\. Although SVD\-LLMv2 itself performs well, we found that initializingLittleBit’s Dual\-SVID with vanilla SVD was sufficient\. The subsequent QAT process effectively recovered performance, rendering the added complexity of SVD\-LLMv2 for initialization unnecessary\. Accordingly, we employ the simpler vanilla SVD within our Dual\-SVID initialization procedure\.

Table 9:Perplexity \(WikiText\-2\) comparison between Pruning \(Wanda\) and SVD\-based compression \(SVD\-LLMv2\) on Llama\-7B at different parameter retention ratios\. Lower perplexity is better\. FP16 PPL is 5\.68\.
## Appendix FAnalysis of Generated Samples

The below generated samples illustrate qualitative differences as the BPW of theLittleBitmodel decreases\. Two main observations are:

1. 1\.Impact on Specificity, Factual Detail, and Coherence: As the effective bits decrease from 0\.8 BPW towards 0\.1 BPW, the generated text exhibits significant changes in detail, factual accuracy, and coherence\. At 0\.8 BPW, the Mona Lisa sample, while attempting to provide details \(e\.g\., "complex facial features"\), becomes verbose and repetitive, and anachronistically refers to Da Vinci and the Mona Lisa in the context of the "20th century\." The Turing sample at this effective bit, however, offers a standard, factually sound definition\. As the effective bit drops to 0\.55 BPW, the Mona Lisa sample loses coherence significantly, expressing uncertainty about the painting’s location and bizarrely shifting to a help\-seeking persona \("Could you help me find the Mona Lisa painting…"\)\. The Turing sample at 0\.55 BPW begins to broaden its definition, losing some precision but remaining somewhat related\.
2. 2\.Degradation in Factual Accuracy and Relevance at Extreme Compression: Extreme compression, especially at 0\.1 BPW, leads to severe degradation in factual accuracy and relevance, as vividly demonstrated by both samples\. The Mona Lisa sample at 0\.1 BPW starts with a generally acceptable statement \(e\.g\., "iconic masterpiece"\) but quickly devolves into hallucination, referring to the Mona Lisa as a "statue" and associating it with "Milan’s fashion world\." Similarly, the Turing sample at 0\.1 BPW deviates drastically from the core definition of computer science, incorrectly defining it as "the study of the theory and methods of how the human mind and the brain work" and "the science of mind\." These examples underscore that whileLittleBitmight maintain superficial grammatical structure at very low bits, the ability to recall and generate accurate, relevant factual information is compromised, often leading to the generation of plausible\-sounding but nonsensical statements\.

### Sample 1: Mona Lisa

Prompt:Leonardo da Vinci painted the Mona Lisa\. The Mona Lisa is