REAL-Q: E2E LLM Quantization via Dynamic Gradient Descent

# REAL-Q: E2E LLM Quantization via Dynamic Gradient Descent
Source: [https://arxiv.org/html/2609.00049](https://arxiv.org/html/2609.00049)
Yaoming LiZhewen TanYanshu WangHeng LuKun SuZongwei LvWenhan YuYongge MaYinjun HanRuikuang LiuTong YangPeking University Northeastern University ZTE CorporationAffiliation:Equal contribution Correspondence:[yangtong@pku\.edu\.cn](mailto:[email protected])

###### Abstract

Post\-training quantization \(PTQ\) is essential for deploying large language models \(LLMs\) under strict resource constraints\. State\-of\-the\-art PTQ methods quantize each layer with a single closed\-form second\-order solver: to remain analytically tractable, they heavily approximate the global loss \(dropping cross\-channel coupling, pooling output rows into groups\), and they then freeze the resulting Hessian across the entire layer, with no way to refresh it as the loss landscape shifts column by column—a phenomenon we callinformation misalignment\. We proposeREAL\-Q\(Real\-timeE2E\-lossAlignedLLMQuantization\), a novel PTQ paradigm that breaks this compromise: instead of diluting the objective for the sake of analytic tractability, REAL\-Q targets an end\-to\-end\-aligned surrogate of the global loss and refines it via fine\-grained, dynamic Block\-wise Gradient Descent applied after every column block \(128 columns\)\. By coupling this fine\-grained correction with a sliding window mechanism for smooth cross\-layer transitions, REAL\-Q effectively mitigates error propagation across the network\. On LLaMA\-3\.1 \(8B and 70B\) and Qwen3 \(0\.6B–32B\) at W4A16, REAL\-Q reduces end\-to\-end KL divergence by up to∼\\sim49% relative to state\-of\-the\-art globally\-guided methods\.

## 1Introduction

Large language models \(LLMs\) have demonstrated remarkable capabilities across a wide range of natural language processing tasks\([Grattafiori et al\., 2024](https://arxiv.org/html/2609.00049#bib.bib14)\)\. However, deploying models with billions of parameters incurs substantial compute and memory costs\. Post\-training quantization \(PTQ\) offers a practical solution by compressing model weights to low\-bit representations without requiring costly retraining\([Gholami et al\., 2021](https://arxiv.org/html/2609.00049#bib.bib13)\)\.

Among PTQ methods, GPTQ\([Frantar et al\., 2023](https://arxiv.org/html/2609.00049#bib.bib12)\)has emerged as the foundational algorithm for weight\-only compression\. Building upon the Optimal Brain Surgeon \(OBS\) framework\([Hassibi et al\., 1993](https://arxiv.org/html/2609.00049#bib.bib15)\), its column\-wise solver has been widely integrated into mainstream inference engines \(e\.g\., vLLM, TensorRT\-LLM\) and extended by a vast lineage of state\-of\-the\-art techniques\([Dettmers et al\., 2024](https://arxiv.org/html/2609.00049#bib.bib7);[Li et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib22);[Kim et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib16)\)\. Despite its success, GPTQ has a fundamental limitation:

Inaccuracy of the Layer\-Local Objective\.GPTQ minimizes a layer\-wise reconstruction MSE‖𝐖𝐱−𝐖^​𝐱‖22\\\|\\mathbf\{W\}\\mathbf\{x\}\-\\hat\{\\mathbf\{W\}\}\\mathbf\{x\}\\\|\_\{2\}^\{2\}, where𝐱\\mathbf\{x\}is the full\-precision activation\. This local objective suffers from a dual misalignment\. Upstream, it assumes full\-precision input activations𝐱\\mathbf\{x\}and ignores the perturbed inputs𝐱^\\hat\{\\mathbf\{x\}\}arriving from previously quantized layers, leaving compounding errors uncorrected\. Downstream, it weights all output\-channel perturbations uniformly, ignoring that the true KL divergence depends on a strongly non\-linear, highly coupled mapping through subsequent transformer blocks and the LM head\. A surrogate that relies on unperturbed inputs and uniformly weights squared errors is therefore misaligned with the true quantization loss\.

Recent works attempt to address these blind spots: GPTAQ\([Li et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib22)\)recalibrates inputs to correct upstream errors, yet its per\-layer loss remains a uniform reconstruction MSE\. Conversely, GuidedQuant\([Kim et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib16)\)incorporates end\-to\-end gradients to capture downstream sensitivity\. However, it still leaves upstream errors uncorrected, and it relies on structural approximations such as ignoring fine\-grained cross\-channel coupling and sharing the Hessian across output groups\. In addition, because the true end\-to\-end Hessian is prohibitively expensive to recompute, the already\-approximated matrix remains static within each layer\. While the true loss landscape shifts as weights are progressively quantized column by column, the misaligned Hessian causes compensation errors to accumulate as the column\-wise sweep progresses\. We refer to this phenomenon asinformation misalignment\. Here, "information" refers to first\- or second\-order information such as the gradient and Hessian\.

The shared limitation of these methods is structural:requiring the column\-wise update to remain analytically solvable forces them to \(i\) heavily approximate the global loss—dropping cross\-channel coupling, pooling output rows into groups, and otherwise compressing the second\-order information until it fits a closed\-form solver—and \(ii\) freeze the resulting Hessian over the entire layer, with no way to refresh it at finer granularity as the loss landscape shifts with each column quantization step\. REAL\-Q \(Real\-timeE2E\-lossAlignedLLMQuantization\) addresses both limitations by interleaving*fine\-grained, dynamic*corrections into the column\-wise sweep, via two components:

An end\-to\-end\-aligned objective: Aggregated Fisher MSE\.REAL\-Q’s surrogate is a transformer\-block\-output Fisher MSE, derived from a second\-order Taylor expansion of the global KL divergence at the current block’s output\. Using the*full*aggregated Fisher matrix preserves the cross\-channel coupling that grouped baselines must discard, while the choice of expansion point keeps backpropagation confined to at most two adjacent blocks\. A loss sliding window smooths the objective across block boundaries to prevent discontinuous jumps in the optimization landscape\.

Fine\-grained, dynamic Block\-wise Gradient Descent\.Rather than committing the entire layer to one frozen second\-order solver, REAL\-Q interleaves a single Adam\([Kingma and Ba, 2014](https://arxiv.org/html/2609.00049#bib.bib18)\)gradient step into the GPTQ sweep*after every column block*\(128 columns\)\. This raises the correction granularity from the per\-layer level used by prior solvers to the column\-block level—roughly two orders of magnitude finer—and lets each step react to the most recent quantized state, since the gradient is recomputed online against the actual partially quantized weights\. Adam’s diagonal preconditioner additionally adapts the per\-coordinate scale on the fly, providing implicit second\-order information without ever inverting a Hessian\.

In summary, our main contributions are as follows:

- •Aggregated Fisher MSE with Loss Sliding Window:Within the GPTQ\-lineage sequential pipeline, we adopt an end\-to\-end\-aligned objective in place of the layer\-local MSE\. We formulate an aggregated Fisher MSE that preserves cross\-channel coupling while keeping the backpropagation overhead confined to localized block boundaries\. To stabilize the sequential quantization process, we introduce aloss sliding windowmechanism that smooths the transition of the objective across consecutive transformer blocks and prevents discontinuous jumps in the loss landscape\.
- •Dynamic Block\-wise Gradient Descent \(Block\-GD\):We address theinformation misalignmentof prior works by replacing static analytical solvers with a dynamic first\-order approach\. By aligning Adam\-driven gradient feedback directly with column\-block updates, this mechanism adapts to the shifting loss landscape and corrects accumulated upstream errors online\.
- •Empirical Performance:Experiments on LLaMA\-3\.1 \(8B and 70B\) and Qwen3 \(0\.6B–32B\) show that REAL\-Q achieves the lowest KL on every model evaluated, reducing KL by up to∼\\sim49% on Qwen3\-1\.7B at W4A16 \(Figure[1](https://arxiv.org/html/2609.00049#S1.F1)\)\.

Figure 1:Headline result\.KL divergence \(×10−2\\times 10^\{\-2\}, lower is better\) on WikiText\-2 \(W4A16, per\-row\)\. REAL\-Q substantially outperforms GuidedQuant, the strongest prior baseline, with exact relative reductions annotated per model\. Full seven\-model results are in Appendix[D\.2](https://arxiv.org/html/2609.00049#A4.SS2)\.
## 2Related Work

#### Second\-order post\-training quantization\.

The Optimal Brain Damage \(OBD\)\([LeCun et al\., 1989](https://arxiv.org/html/2609.00049#bib.bib20)\)and Optimal Brain Surgeon \(OBS\)\([Hassibi et al\., 1993](https://arxiv.org/html/2609.00049#bib.bib15)\)frameworks established the foundation for using second\-order Hessian information in network compression\. GPTQ\([Frantar et al\., 2023](https://arxiv.org/html/2609.00049#bib.bib12)\)successfully scaled OBS to billion\-parameter LLMs\. By processing weight rows concurrently and employing a shared, static inverse Hessian to compensate for quantization errors column by column, GPTQ established the dominant pipeline for modern sequential PTQ\.

#### Global\-aware extensions of GPTQ\.

Recent advancements attempt to enhance the standard sequential GPTQ pipeline by incorporating broader model context into the quantization process\.GPTAQ\([Li et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib22)\)addresses upstream error accumulation by iteratively recalibrating the input activations\. Instead of relying solely on fixed, full\-precision inputs, it dynamically computes calibration activations using the outputs from previously quantized layers, thereby incorporating upstream quantization drift into the layer\-wise reconstruction objective\.GuidedQuant\([Kim et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib16)\)incorporates end\-to\-end downstream sensitivity into the local solver\. It performs an initial backward pass over the calibration set to obtain per\-token saliency scores, which are subsequently used to weight the input covariance matrix\. To maintain the computational tractability of the analytical second\-order update, GuidedQuant partitions the weight matrix rows into groups and computes a shared, static approximated Hessian for each group to drive the column\-wise quantization\.

#### Other related directions\.

Extended discussions on rotation pre\-processing \(e\.g\., QuaRot\([Ashkboos et al\., 2024b](https://arxiv.org/html/2609.00049#bib.bib3)\), which we adopt\), weight\-only representations, quantization\-aware training, and classical Fisher applications are deferred to Appendix[B](https://arxiv.org/html/2609.00049#A2)\. This appendix also provides a detailed comparison with BRECQ\([Li et al\., 2021](https://arxiv.org/html/2609.00049#bib.bib21)\); while BRECQ shares conceptual similarities in block\-level loss modeling, its overall quantization pipeline is fundamentally different from ours\.

## 3Preliminaries

### 3\.1Problem Formulation

Consider a pre\-trained LLM withLLtransformer blocks, where each blockℓ\\ellcontains linear modules \(e\.g\.,𝐖q,𝐖k,𝐖v,𝐖o\\mathbf\{W\}^\{q\},\\mathbf\{W\}^\{k\},\\mathbf\{W\}^\{v\},\\mathbf\{W\}^\{o\}for attention and𝐖up,𝐖gate,𝐖down\\mathbf\{W\}^\{\\text\{up\}\},\\mathbf\{W\}^\{\\text\{gate\}\},\\mathbf\{W\}^\{\\text\{down\}\}for the MLP\)\. For a linear module with weight matrix𝐖∈ℝm×n\\mathbf\{W\}\\in\\mathbb\{R\}^\{m\\times n\}, the goal of PTQ is to find a quantized weight𝐖^\\hat\{\\mathbf\{W\}\}that minimizes the degradation in model output quality\.

Given a calibration dataset𝒟=\{𝐱1,…,𝐱N\}\\mathcal\{D\}=\\\{\\mathbf\{x\}\_\{1\},\\ldots,\\mathbf\{x\}\_\{N\}\\\}, we measure quantization quality via the KL divergence between the original and quantized model outputs:

ℒKL=𝔼𝐱∼𝒟\[DKL\(p𝐖\(⋅∣𝐱\)∥p𝐖^\(⋅∣𝐱\)\)\]\.\\mathcal\{L\}\_\{\\text\{KL\}\}=\\mathbb\{E\}\_\{\\mathbf\{x\}\\sim\\mathcal\{D\}\}\\left\[D\_\{\\text\{KL\}\}\\left\(p\_\{\\mathbf\{W\}\}\(\\cdot\\mid\\mathbf\{x\}\)\\;\\\|\\;p\_\{\\hat\{\\mathbf\{W\}\}\}\(\\cdot\\mid\\mathbf\{x\}\)\\right\)\\right\]\.\(1\)
Following recent studies\([Kong et al\., 2026](https://arxiv.org/html/2609.00049#bib.bib19)\), we adopt KL divergence as our primary evaluation metric, as it captures the full output distributional shift and correlates more faithfully with quantization\-induced behavioral degradation than perplexity\. For comparability with prior works, we also report perplexity and zero\-shot task accuracy as secondary metrics\.

### 3\.2GPTQ Recap

GPTQ\([Frantar et al\., 2023](https://arxiv.org/html/2609.00049#bib.bib12)\)processes linear modules by quantizing weights column by column\. At stepjj, GPTQ quantizes columnjjto𝐰^j\\hat\{\\mathbf\{w\}\}\_\{j\}, computes the residual𝜹j=𝐰j−𝐰^j\\boldsymbol\{\\delta\}\_\{j\}=\\mathbf\{w\}\_\{j\}\-\\hat\{\\mathbf\{w\}\}\_\{j\}, and optimally updates the remaining unquantized columns to compensate:

𝐖:,j\+1:n←𝐖:,j\+1:n−1\[𝐇−1\]j​j𝜹j𝐇−1j,j\+1:n,\\mathbf\{W\}\_\{:,j\+1:n\}\\leftarrow\\mathbf\{W\}\_\{:,j\+1:n\}\-\\frac\{1\}\{\[\\mathbf\{H\}^\{\-1\}\]\_\{jj\}\}\\boldsymbol\{\\delta\}\_\{j\}\\mathbf\{H\}^\{\-1\}\_\{j,j\+1:n\},\(2\)where𝐇=2T​∑t=1T𝐱t​𝐱t⊤\\mathbf\{H\}=\\frac\{2\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbf\{x\}\_\{t\}\\mathbf\{x\}\_\{t\}^\{\\top\}is the Hessian of the localized reconstruction MSE\. To accelerate computation, GPTQ groups columns into blocks of sizeBB\. After applying Eq\.[2](https://arxiv.org/html/2609.00049#S3.E2)within the block, a batch update compensates all trailing columns globally:𝐖:,bend:n←𝐖:,bend:n−𝐄b⋅𝐇−1b,bend:n\\mathbf\{W\}\_\{:,b\_\{\\text\{end\}\}:n\}\\leftarrow\\mathbf\{W\}\_\{:,b\_\{\\text\{end\}\}:n\}\-\\mathbf\{E\}\_\{\\text\{b\}\}\\cdot\\mathbf\{H\}^\{\-1\}\_\{b,b\_\{\\text\{end\}\}:n\}\(bendb\_\{\\text\{end\}\}denotes the end of blockbb\), where𝐄b\\mathbf\{E\}\_\{\\text\{b\}\}is the accumulated block error\.

### 3\.3GuidedQuant: Saliency\-Weighted Hessian

While standard GPTQ relies on a uniform layer\-wise reconstruction objective, GuidedQuant\([Kim et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib16)\)improves this by aligning the Hessian with the end\-to\-end loss\. By performing an initial end\-to\-end backward pass over the calibration set, it obtains per\-token saliency scoresst\(ℓ\)=‖∂ℒ/∂𝐲t\(ℓ\)‖2s\_\{t\}^\{\(\\ell\)\}=\\\|\\partial\\mathcal\{L\}/\\partial\\mathbf\{y\}\_\{t\}^\{\(\\ell\)\}\\\|^\{2\}, which are then used to weight the GPTQ Hessian computation\. To manage computational overhead, the weight matrix rows are partitioned intoNgN\_\{g\}groups \(we useNg=4N\_\{g\}=4in all experiments\), and a separate Hessian is maintained for each group:

𝐇\(ℓ\)g=∑t=1Tst,g\(ℓ\)⋅𝐱t𝐱t⊤,g=1,…,Ng,\\mathbf\{H\}^\{\(\\ell\)\}\_\{g\}=\\sum\_\{t=1\}^\{T\}s\_\{t,g\}^\{\(\\ell\)\}\\cdot\\mathbf\{x\}\_\{t\}\\mathbf\{x\}\_\{t\}^\{\\top\},\\quad g=1,\\ldots,N\_\{g\},\(3\)wherest,g\(ℓ\)s\_\{t,g\}^\{\(\\ell\)\}is the saliency score for tokenttrestricted to the output dimensions in groupgg\. While computationally feasible, this group\-sharing strategy inherently discards exact cross\-channel coupling and degrades the precision of the second\-order matrix\.

## 4REAL\-Q

### 4\.1Overview

REAL\-Qaugments the saliency\-weighted PTQ pipeline with a coarse\-to\-fine optimization hierarchy\. We retain the traditional column\-wise analytical compensation as a preliminary coarse update, but augment the optimization trajectory with: \(1\) anAggregated Fisher\-Weighted MSEproviding end\-to\-end gradient guidance; \(2\)Dynamic Block\-GD, an active first\-order correction applied immediately post\-block to reduce residual errors \(e\.g\., upstream perturbations and structural imprecision\) missed by static solvers; and \(3\) aLoss Sliding Windowto smoothly interpolate objectives across consecutive transformer blocks\. Together, these components correct residual errors left by static second\-order updates\. Figure[2](https://arxiv.org/html/2609.00049#S4.F2)presents a comparison between prior methods and REAL\-Q\. The complete pseudocode is provided in Appendix[A](https://arxiv.org/html/2609.00049#A1)\.

![Refer to caption](https://arxiv.org/html/2609.00049v1/figures/fig2_blockgd_v3.jpg)Figure 2:Comparison of static analytical quantization and REAL\-Q\. In prior static methods \(e\.g\., standard GPTQ or GuidedQuant\), a fixed Hessian approximation is used throughout, allowing quantization errors to accumulate unchecked across column blocks \(red\)\. In REAL\-Q, a dynamic Block\-GD step is applied immediately after each block, actively correcting accumulated structural errors and keeping the end\-to\-end reconstruction loss tightly bounded \(green\)\.
### 4\.2Objective

A challenge in moving beyond static analytical solvers is defining a tractable, high\-fidelity loss function to drive dynamic optimization\. Using the exact end\-to\-end KL divergence \(Eq\.[1](https://arxiv.org/html/2609.00049#S3.E1)\) as an explicit objective would require a full forward pass through all subsequent layers and the LM head at every block step—a computationally prohibitive requirement\.

To break this computational deadlock, we construct a high\-fidelity surrogate: the*Aggregated Fisher MSE loss*\. Consider the KL divergenceℒKL\\mathcal\{L\}\_\{\\text\{KL\}\}as a function of the per\-token transformer block output𝐲t\\mathbf\{y\}\_\{t\}\. By performing a second\-order Taylor expansion around the full\-precision output𝐲t∗\\mathbf\{y\}\_\{t\}^\{\*\}, and noting that the first\-order gradient identically vanishes since𝐲t∗\\mathbf\{y\}\_\{t\}^\{\*\}minimizes the KL divergence \(ℒKL​\(𝐲t∗\)=0\\mathcal\{L\}\_\{\\text\{KL\}\}\(\\mathbf\{y\}\_\{t\}^\{\*\}\)=0\), the expected loss over tokens satisfies:

𝔼t​\[ℒKL​\(𝐲t\)\]≈12​𝔼t​\[Δ​𝐲t⊤​𝐇t​Δ​𝐲t\],\\mathbb\{E\}\_\{t\}\\bigl\[\\mathcal\{L\}\_\{\\text\{KL\}\}\(\\mathbf\{y\}\_\{t\}\)\\bigr\]\\approx\\frac\{1\}\{2\}\\mathbb\{E\}\_\{t\}\\bigl\[\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\\mathbf\{H\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}\\bigr\],\(4\)whereΔ​𝐲t=𝐲t−𝐲t∗\\Delta\\mathbf\{y\}\_\{t\}=\\mathbf\{y\}\_\{t\}\-\\mathbf\{y\}\_\{t\}^\{\*\}is the per\-token output perturbation and𝐇t=∇𝐲t2ℒKL\|𝐲t∗\\mathbf\{H\}\_\{t\}=\\nabla^\{2\}\_\{\\mathbf\{y\}\_\{t\}\}\\mathcal\{L\}\_\{\\text\{KL\}\}\\big\|\_\{\\mathbf\{y\}\_\{t\}^\{\*\}\}is the per\-token Hessian\. The standard Fisher–Hessian identity gives𝔼t​\[𝐇t\]=𝔼t​\[𝐠t​𝐠t⊤\]\\mathbb\{E\}\_\{t\}\[\\mathbf\{H\}\_\{t\}\]=\\mathbb\{E\}\_\{t\}\[\\mathbf\{g\}\_\{t\}\\mathbf\{g\}\_\{t\}^\{\\top\}\]in expectation under the model distribution, where𝐠t=∇𝐲tℒNLL\\mathbf\{g\}\_\{t\}=\\nabla\_\{\\mathbf\{y\}\_\{t\}\}\\mathcal\{L\}\_\{\\text\{NLL\}\}\(noting the shift fromℒKL\\mathcal\{L\}\_\{\\text\{KL\}\}, as Fisher information is formally defined via the NLL score function using target labels sampled from the full\-precision model’s output distributionpp\)\. In contrast to the structural approximations used by prior analytical solvers, we formulate an objective that operates at the*transformer block*level and preserves the cross\-channel coupling structure\.

#### Aggregated Fisher loss \(ours\)\.

We bypass the structural approximations of prior works by moving the expectation*inside*the quadratic form\. We define the aggregated Fisher matrix:

𝐅=𝔼t​\[𝐠t​𝐠t⊤\]≈1T​∑t=1T𝐠t​𝐠t⊤∈ℝd×d,\\mathbf\{F\}=\\mathbb\{E\}\_\{t\}\\bigl\[\\mathbf\{g\}\_\{t\}\\mathbf\{g\}\_\{t\}^\{\\top\}\\bigr\]\\approx\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbf\{g\}\_\{t\}\\mathbf\{g\}\_\{t\}^\{\\top\}\\in\\mathbb\{R\}^\{d\\times d\},\(5\)whereddis the transformer block output dimension\. Our surrogate loss, which acts as the foundational objective for our subsequent Block\-GD updates \(Section[4\.3](https://arxiv.org/html/2609.00049#S4.SS3)\), is formulated as:

ℒFisher=12𝔼t\[Δ𝐲t⊤𝐅Δ𝐲t\]=12⋅1T∑t=1TΔ𝐲t⊤𝐅Δ𝐲t\.\\mathcal\{L\}\_\{\\text\{Fisher\}\}=\\frac\{1\}\{2\}\\mathbb\{E\}\_\{t\}\\bigl\[\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\\mathbf\{F\}\\Delta\\mathbf\{y\}\_\{t\}\\bigr\]=\\frac\{1\}\{2\}\\cdot\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\\mathbf\{F\}\\Delta\\mathbf\{y\}\_\{t\}\.\(6\)Here,𝐅\\mathbf\{F\}is the*full*Fisher matrix \(retaining all cross\-channel covariances\), aggregated once per transformer block over the calibration set\. It requires zero per\-token storage and removes the need for any heuristic grouping hyperparameters\. Details regarding the pre\-computation of𝐅\\mathbf\{F\}and the special handling of the final transformer block are provided in Appendix[D\.1](https://arxiv.org/html/2609.00049#A4.SS1)\.

###### Proposition 4\.1\(Decoupled Fisher Aggregation\)\.

The exact expected second\-order penalty is𝔼t​\[Δ​𝐲t⊤​𝐇t​Δ​𝐲t\]\\mathbb\{E\}\_\{t\}\[\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\\mathbf\{H\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}\]\. By applying a mean\-field approximation to decouple the token\-wise Hessian from the output perturbation, we obtain𝔼t​\[Δ​𝐲t⊤​𝔼t​\[𝐇t\]​Δ​𝐲t\]\\mathbb\{E\}\_\{t\}\[\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\\,\\mathbb\{E\}\_\{t\}\[\\mathbf\{H\}\_\{t\}\]\\,\\Delta\\mathbf\{y\}\_\{t\}\]\. Combined with the Fisher–Hessian identity𝔼t​\[𝐇t\]=𝔼t​\[𝐠t​𝐠t⊤\]≈𝐅\\mathbb\{E\}\_\{t\}\[\\mathbf\{H\}\_\{t\}\]=\\mathbb\{E\}\_\{t\}\[\\mathbf\{g\}\_\{t\}\\mathbf\{g\}\_\{t\}^\{\\top\}\]\\approx\\mathbf\{F\}, Eq\.[6](https://arxiv.org/html/2609.00049#S4.E6)serves as a tractable surrogate\. The approximation error strictly amounts to the element\-wise covariance∑i,jCov​\(\(𝐇t\)i​j,\(Δ​𝐲t​Δ​𝐲t⊤\)i​j\)\\sum\_\{i,j\}\\text\{Cov\}\\bigl\(\(\\mathbf\{H\}\_\{t\}\)\_\{ij\},\(\\Delta\\mathbf\{y\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\)\_\{ij\}\\bigr\)\(see Appendix[C\.1](https://arxiv.org/html/2609.00049#A3.SS1)\)\.

We adopt this decoupling as a deliberate trade\-off: recognizing that cross\-channel covariance affects LLM behavior more strongly than token\-wise correlations, we trade the latter to bypass intractable per\-token Hessians\. This allows materializing the full aggregated Fisher𝐅\\mathbf\{F\}without memory explosion\.

### 4\.3Dynamic Block\-wise Gradient Descent

In REAL\-Q, the saliency\-weighted analytical update serves as a foundational step\. After analytically processing column blockbb, we evaluate our surrogateℒFisher​\(𝐖\(b\)\)\\mathcal\{L\}\_\{\\text\{Fisher\}\}\(\\mathbf\{W\}^\{\(b\)\}\)via a forward pass through the current transformer block using a calibration mini\-batch\. We explicitly correct the static solver’s residual errors by computing the exact gradient with respect to the remaining unquantized weights, updating these trailing columns via a single Adam\([Kingma and Ba, 2014](https://arxiv.org/html/2609.00049#bib.bib18)\)step \(whose states are tracked per\-parameter, as the pool of active unquantized weights continuously shrinks\):

𝐖:,bend:n←𝐖:,bend:n−η⋅Adam\(∇𝐖:,bend:nℒFisher\(𝐖\(b\)\)\),\\mathbf\{W\}\_\{:,b\_\{\\text\{end\}\}:n\}\\leftarrow\\mathbf\{W\}\_\{:,b\_\{\\text\{end\}\}:n\}\-\\eta\\cdot\\text\{Adam\}\\big\(\\nabla\_\{\\mathbf\{W\}\_\{:,b\_\{\\text\{end\}\}:n\}\}\\mathcal\{L\}\_\{\\text\{Fisher\}\}\(\\mathbf\{W\}^\{\(b\)\}\)\\big\),\(7\)whereη\\etais the learning rate\. Already\-quantized weights remain locked\. By relying on this first\-order correction, REAL\-Q limits cross\-block error accumulation and avoids the prohibitive cost of refreshing the analytical Hessian during the quantization loop\.

### 4\.4Loss Sliding Window

Sequential quantization typically suffers from discontinuous objective landscapes when transitioning between transformer blocks\. To resolve this, we introduce a*loss sliding window*\(Figure[4](https://arxiv.org/html/2609.00049#A1.F4)\) that smoothly interpolates exact Fisher objectives\. When quantizing blockℓ\\ellat cumulative column\-block stepbglobalb\_\{\\text\{global\}\}\(out ofBtotalB\_\{\\text\{total\}\}, whereBtotalB\_\{\\text\{total\}\}denotes the total number of column\-block steps within transformer blockℓ\\ell, i\.e\., the sum of column blocks across all linear modules in the block; the per\-linear\-module column\-block indexbbin Eq\.[7](https://arxiv.org/html/2609.00049#S4.E7)is local to its module, whilebglobalb\_\{\\text\{global\}\}accumulates across all modules in the block\), the surrogate is dynamically blended:

ℒslide\(bglobal\)=α\(bglobal\)⋅ℒFisher\(ℓ\)\+\(1−α\(bglobal\)\)⋅ℒFisher\(ℓ\+1\),\\mathcal\{L\}\_\{\\text\{slide\}\}^\{\(b\_\{\\text\{global\}\}\)\}=\\alpha^\{\(b\_\{\\text\{global\}\}\)\}\\cdot\\mathcal\{L\}\_\{\\text\{Fisher\}\}^\{\(\\ell\)\}\+\\bigl\(1\-\\alpha^\{\(b\_\{\\text\{global\}\}\)\}\\bigr\)\\cdot\\mathcal\{L\}\_\{\\text\{Fisher\}\}^\{\(\\ell\+1\)\},\(8\)whereα\(bglobal\)=1−bglobal−1Btotal−1\\alpha^\{\(b\_\{\\text\{global\}\}\)\}=1\-\\frac\{b\_\{\\text\{global\}\}\-1\}\{B\_\{\\text\{total\}\}\-1\}\(assumingBtotal\>1B\_\{\\text\{total\}\}\>1\)\. Mechanically, evaluating the gradient of this blended loss requires backpropagating from the output of blockℓ\+1\\ell\+1into blockℓ\\ell\. While this expands the dynamic correction graph to two transformer blocks, it avoids the overhead of full end\-to\-end unrolling\. This two\-block scope encourages the current layer to anticipate and reduce error propagation into its successor, preventing abrupt jumps in the loss landscape between adjacent blocks and thereby stabilizing the sequential quantization process\.

## 5Theoretical Analysis

We present two theoretical results justifying REAL\-Q: formalizing the inherent error accumulation of static Hessian solvers, and deriving a sufficient cosine condition under which Block\-GD descends on the true global loss\. Full proofs are deferred to Appendix[C\.2](https://arxiv.org/html/2609.00049#A3.SS2)and[C\.3](https://arxiv.org/html/2609.00049#A3.SS3)\.

#### Error Accumulation in Static Solvers\.

Globally\-guided PTQ methods rely on a static approximation of the end\-to\-end Hessian \(𝐇\\mathbf\{H\}\)\. However, as column quantization progresses, the true downstream\-aware Hessian shifts\. LettingΔ​𝐇\(j\)=𝐇ideal\(j\)−𝐇\\Delta\\mathbf\{H\}^\{\(j\)\}=\\mathbf\{H\}\_\{\\mathrm\{ideal\}\}^\{\(j\)\}\-\\mathbf\{H\}denote this gap, the static solver incurs a cumulative penalty\. Assuming a linear structural drift, we prove in Appendix[C\.2](https://arxiv.org/html/2609.00049#A3.SS2)that this expected uncompensated error scales as:

𝔼⁡\[ℰ\]=Ω⁡\(n⋅ϵ¯2⋅‖𝐇‖2\),\\mathbb\{E\}\[\\mathcal\{E\}\]=\\Omega\(n\\cdot\\bar\{\\epsilon\}^\{2\}\\cdot\\\|\\mathbf\{H\}\\\|\_\{2\}\),\(9\)wherennis the column dimension andϵ¯2\\bar\{\\epsilon\}^\{2\}is the expected squared residual\. Under the stated assumption, this linear error growth with respect to module widthnnindicates that static analytical solvers cannot prevent compounding cross\-column errors\.

#### Descent Condition Analysis under First\-Order Updates\.

To overcome this, REAL\-Q dynamically optimizes the surrogateℒFisher\\mathcal\{L\}\_\{\\text\{Fisher\}\}\. Assuming the true end\-to\-end objectiveJ⁡\(𝐖\)J\(\\mathbf\{W\}\)isβ\\beta\-smooth, the true loss after a gradient step𝐖\(\+\)=𝐖−η​𝐠surr\\mathbf\{W\}^\{\(\+\)\}=\\mathbf\{W\}\-\\eta\\mathbf\{g\}\_\{\\text\{surr\}\}satisfies the descent\-lemma upper bound

J⁡\(𝐖\(\+\)\)≤J⁡\(𝐖\)−η​‖∇J​\(𝐖\)‖​‖𝐠surr‖​cos⁡θ\+η2​β2​‖𝐠surr‖2,J\(\\mathbf\{W\}^\{\(\+\)\}\)\\leq J\(\\mathbf\{W\}\)\-\\eta\\\|\\nabla J\(\\mathbf\{W\}\)\\\|\\\|\\mathbf\{g\}\_\{\\text\{surr\}\}\\\|\\cos\\theta\+\\frac\{\\eta^\{2\}\\beta\}\{2\}\\\|\\mathbf\{g\}\_\{\\text\{surr\}\}\\\|^\{2\},\(10\)wherecos⁡θ\\cos\\thetais the cosine similarity between the surrogate gradient𝐠surr\\mathbf\{g\}\_\{\\text\{surr\}\}and the true gradient∇J​\(𝐖\)\\nabla J\(\\mathbf\{W\}\)\. Thus, the true global loss strictly decreases \(J⁡\(𝐖\(\+\)\)<J⁡\(𝐖\)J\(\\mathbf\{W\}^\{\(\+\)\}\)<J\(\\mathbf\{W\}\)\) if:

cos⁡θ\>η​β2⋅‖𝐠surr‖‖∇J​\(𝐖\)‖\.\\cos\\theta\>\\frac\{\\eta\\beta\}\{2\}\\cdot\\frac\{\\\|\\mathbf\{g\}\_\{\\text\{surr\}\}\\\|\}\{\\\|\\nabla J\(\\mathbf\{W\}\)\\\|\}\.\(11\)This establishes that optimization fails under poor gradient approximations \(cos⁡θ∼0\\cos\\theta\\sim 0\)\. Conversely, REAL\-Q maintainscos⁡θ\\cos\\thetawell above this threshold \(Figure[3](https://arxiv.org/html/2609.00049#S6.F3)\)\. The above argument is stated for SGD; for Adam, the preconditioned step direction is no longer aligned with𝐠surr\\mathbf\{g\}\_\{\\text\{surr\}\}, so the inequality does not directly transfer, but we observe consistent empirical descent under the learning rates used in our experiments \(Section[6](https://arxiv.org/html/2609.00049#S6)\)\.

## 6Experiments

### 6\.1Experimental Setup

#### Setup\.

We evaluate REAL\-Q on LLaMA\-3\.1\-8B / LLaMA\-3\.1\-70B\([Grattafiori et al\., 2024](https://arxiv.org/html/2609.00049#bib.bib14)\)and Qwen3\-0\.6B / 1\.7B / 4B / 8B / 32B\([Yang et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib34)\), against RTN, GPTQ\([Frantar et al\., 2023](https://arxiv.org/html/2609.00049#bib.bib12)\), GPTAQ\([Li et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib22)\), and GuidedQuant\([Kim et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib16)\); all methods share QuaRot\([Ashkboos et al\., 2024b](https://arxiv.org/html/2609.00049#bib.bib3)\)rotation\. We report WikiText\-2\([Merity et al\., 2017](https://arxiv.org/html/2609.00049#bib.bib29)\)KL divergence, perplexity, and zero\-shot accuracy on ten tasks\. Quantization is symmetric throughout; W4A16 uses*per\-row*weights \(no grouping\); group size 128 is used only for low\-bit \(W3A16, W2A16\) and weight\-activation \(WxxA4KV4\) settings\. Calibration: 2048 WikiText\-2 samples for W4A16 and W4A4KV4 \(256 for LLaMA\-3\.1\-70B due to cost; 256 for low\-bit / W2A4KV4 / W3A4KV4\)\. We use Adam with a mini\-batch size of 32 and a reverse\-cosine layer\-wise LR schedule\. Full setup and learning rates are detailed in Appendices[D\.1](https://arxiv.org/html/2609.00049#A4.SS1)and[D\.3](https://arxiv.org/html/2609.00049#A4.SS3)\.

#### Why KL is our primary fidelity metric\.

All Qwen3 checkpoints used in our evaluation are post\-trained models, for which the bf16 reference is generally not at the extremum of perplexity or zero\-shot task accuracy\(post\-training optimizes neither metric; cf\. Appendix[D\.1](https://arxiv.org/html/2609.00049#A4.SS1)\)\. Consequently, any quantization\-induced fluctuation in PPL or downstream accuracy—in either direction—does not by itself indicate quantization quality; only the KL divergence between the quantized and full\-precision output distributions directly measures fidelity to the full\-precision model’s behavior\.Furthermore, the calibration set \(2048×2048≈4​M2048\\times 2048\\approx 4\\text\{M\}tokens\) is a small fraction of the full WikiText\-2 corpus, and KL/PPL are evaluated on the held\-out test split, so the consistent KL reduction across every evaluated model reflects generalization to unseen text rather than overfitting to the calibration sample\.

### 6\.2Main Results: W4A16

Table[1](https://arxiv.org/html/2609.00049#S6.T1)reports W4A16 results \(per\-row weight quantization\) for REAL\-Q and four baselines on four representative models—one LLaMA\-3\.1 base model and three Qwen3 sizes\. REAL\-Q achieves the lowest KL divergence on every model\. The full seven\-model table \(additionally covering LLaMA\-3\.1\-70B, Qwen3\-0\.6B / 8B\) is in Appendix[D\.2](https://arxiv.org/html/2609.00049#A4.SS2)\.

Table 1:W4A16 \(per\-row weight quantization\) on four representative models\. KL \(×10−2\\times 10^\{\-2\}\) and PPL are evaluated on WikiText\-2\. Downstream columns report zero\-shot accuracy \(%\) on ten tasks and their average\.KL is the primary fidelity metric; for post\-trained Qwen3 models the bf16 reference is not at the PPL/accuracy optimum, so PPL or downstream\-accuracy fluctuations relative to bf16 do not by themselves indicate quality \(see §[6\.1](https://arxiv.org/html/2609.00049#S6.SS1)and Appendix[D\.1](https://arxiv.org/html/2609.00049#A4.SS1)\)\.
### 6\.3Low\-bit Weight\-only Quantization

Table[2](https://arxiv.org/html/2609.00049#S6.T2)compares baselines and REAL\-Q on Qwen3\-8B at W3A16 and W2A16\. REAL\-Q still beats every baseline including GuidedQuant\.

Table 2:Low\-bit weight\-only quantization on Qwen3\-8B\. Group size 128, calibration sequence length 2048, 256 calibration samples\. KL \(×10−2\\times 10^\{\-2\}\) and PPL are evaluated on WikiText\-2\. Downstream columns report zero\-shot accuracy \(%\) on ten tasks and their average\.
### 6\.4Weight\-Activation Quantization

We evaluate REAL\-Q in the weight\-activation quantization setting on Qwen3\-8B with W4A4KV4, W3A4KV4 and W2A4KV4 configurations\. Activations and KV cache are quantized per\-token with clipping ratiosaclip=kclip=vclip=0\.9a\_\{\\text\{clip\}\}=k\_\{\\text\{clip\}\}=v\_\{\\text\{clip\}\}=0\.9; weights use group size 128\. As described in Appendix[D\.1](https://arxiv.org/html/2609.00049#A4.SS1), the reverse\-cosine learning\-rate schedule is disabled specifically in activation\-aware settings, ensuring that early\-block errors are corrected by gradient descent rather than amplified\.

#### Two activation\-quantization variants\.

For each method we report two activation\-quantization variants\.\(a\) Activation\-unaware:weights are quantized using the full\-precision activations as input; the activations are quantized only*after*all weights are quantized\.\(b\) Activation\-aware:the quantization of weights is performed against already\-quantized activations, so each weight column sees the same input distribution it will face at inference\. GPTAQ and GuidedQuant both natively support activation\-aware quantization, and we therefore enable that variant for those baselines\.

Table 3:Weight\-activation quantization on Qwen3\-8B \(WxxA4KV4\)\. KL \(×10−2\\times 10^\{\-2\}\) and PPL are evaluated on WikiText\-2\. Downstream columns report zero\-shot accuracy \(%\) on ten tasks and their average\. For REAL\-Q we report both the activation\-unaware \(a\-unaware\) and the activation\-aware \(a\-aware\) variants\. GPTQ / GPTAQ / GuidedQuant baselines use their activation\-aware variant \(where supported\)\.

### 6\.5Ablation on Qwen3\-4B

We ablate four REAL\-Q design choices on Qwen3\-4B \(W4A16, per\-row weight quantization, scheduled final learning rate×10−42\\\!\\times\\\!10^\{\-4\}, final\-block LR10−510^\{\-5\}\): the losssliding window\(SW, on/off\); the output\-perturbation clipping ratioaloss\_clipa\_\{\\text\{loss\\\_clip\}\}; the columnblock sizeBB\(smallerBB⇒\\Rightarrowmore frequent gradient corrections\); and the number ofbackward samplesper gradient step \(controls the variance of𝐠\(b\)\\mathbf\{g\}^\{\(b\)\}\)\. The base configuration is SW on,aloss\_clip=0\.95a\_\{\\text\{loss\\\_clip\}\}\{=\}0\.95,B=128B\{=\}128, 32 backward samples per step\.

Table 4:Ablation on Qwen3\-4B \(W4A16, per\-row weight quantization\)\. KL \(×10−2\\times 10^\{\-2\}\) and PPL are evaluated on WikiText\-2; Avg\. is the average zero\-shot accuracy across the ten downstream tasks\. Each row changes one knob from the base; remaining knobs are at the base values\.Notably, the column block sizeBBexhibits the largest single\-knob impact, supporting the role of fine\-grained update granularity \(smallerBB\) in mitigating error accumulation\. We use KL as the primary metric throughout \(§[6\.1](https://arxiv.org/html/2609.00049#S6.SS1)\)\.

### 6\.6Loss Approximation Quality

To empirically assess the quality of our aggregated Fisher MSE as a surrogate, we evaluate the geometric alignment between the surrogate gradients and the true end\-to\-end KL gradient\. Figure[3](https://arxiv.org/html/2609.00049#S6.F3)plots this mean cosine similarity on Qwen3\-0\.6B \(W4A16\)\. REAL\-Q maintains high alignment with the true global trajectory\. By contrast, the layer\-wise MSE \(GPTQ/GPTAQ\) and saliency\-guided MSE \(GuidedQuant\) hover near zero, consistent with their limited ability to capture the true descent direction\. Full experimental setup is detailed in Appendix[D\.4](https://arxiv.org/html/2609.00049#A4.SS4)\.

Figure 3:Mean cosine similarity between each surrogate’s gradient and the true end\-to\-end KL gradient on Qwen3\-0\.6B \(W4A16\)\. REAL\-Q’s full Fisher \(blue\) remains consistently aligned with the true gradient, whereas the saliency\-guided MSE \(GuidedQuant, orange\) and module MSE \(GPTQ / GPTAQ, grey\) sit near zero alignment\.#### Additional results\.

Full W4A16 sweeps, calibration\-seed stability on Qwen3\-0\.6B, and detailed wall\-clock and GPU memory profiling are deferred to Appendices[D\.2](https://arxiv.org/html/2609.00049#A4.SS2),[D\.7](https://arxiv.org/html/2609.00049#A4.SS7),[D\.5](https://arxiv.org/html/2609.00049#A4.SS5), and[D\.6](https://arxiv.org/html/2609.00049#A4.SS6)\.

## 7Discussion, Limitations, and Future Work

REAL\-Q addresses the precision–tractability trade\-off in modern PTQ by replacing static analytical solvers with a dynamic, coarse\-to\-fine optimization hierarchy\. By pairing a full Aggregated Fisher MSE with Block\-GD, our method reduces*information misalignment*and preserves cross\-channel coupling\. Furthermore, our theoretical analysis shows that, under a linear structural\-drift assumption, static solvers incurΩ⁡\(n\)\\Omega\(n\)compounding errors, and gives the cosine condition that, when satisfied, guarantees descent of our dynamic first\-order updates on the true global loss\. Shifting to this active gradient\-based paradigm introduces additional computational and memory overhead during calibration; we expect these overheads can be amortized through future system\-level optimizations, such as custom memory\-efficient backward kernels and intermediate activation offloading\. An extended discussion of limitations and future work is provided in Appendix[E](https://arxiv.org/html/2609.00049#A5)\.

## References

- Amari \(1998\)Shun\-ichi Amari\.Natural gradient works efficiently in learning\.*Neural Computation*, 10\(2\):251–276, 1998\.
- Ashkboos et al\. \(2024a\)Saleh Ashkboos, Maximilian L\. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman\.SliceGPT: Compress large language models by deleting rows and columns\.In*International Conference on Learning Representations \(ICLR\)*, 2024a\.
- Ashkboos et al\. \(2024b\)Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L\. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman\.QuaRot: Outlier\-free 4\-bit inference in rotated LLMs\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024b\.
- Bondarenko et al\. \(2024\)Yelysei Bondarenko, Riccardo Del Chiaro, and Markus Nagel\.LR\-QAT: Low\-rank quantization\-aware training for LLMs\.*arXiv preprint arXiv:2406\.06385*, 2024\.
- Chee et al\. \(2023\)Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa\.QuIP: 2\-bit quantization of large language models with guarantees\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- Chen et al\. \(2025\)Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo\.EfficientQAT: Efficient quantization\-aware training for large language models\.In*Association for Computational Linguistics \(ACL\)*, 2025\.
- Dettmers et al\. \(2024\)Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh\.SpQR: A sparse\-quantized representation for near\-lossless LLM weight compression\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Dong et al\. \(2019\)Zhen Dong, Zhewei Yao, Amir Gholami, Michael W\. Mahoney, and Kurt Keutzer\.HAWQ: Hessian AWare quantization of neural networks with mixed\-precision\.In*International Conference on Computer Vision \(ICCV\)*, 2019\.
- Dong et al\. \(2020\)Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W\. Mahoney, and Kurt Keutzer\.HAWQ\-V2: Hessian aware trace\-weighted quantization of neural networks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2020\.
- Egiazarian et al\. \(2024\)Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh\.AQLM: Extreme compression of large language models via additive quantization\.In*International Conference on Machine Learning \(ICML\)*, 2024\.
- Frantar et al\. \(2022\)Elias Frantar, Sidak Pal Singh, and Dan Alistarh\.Optimal brain compression: A framework for accurate post\-training quantization and pruning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- Frantar et al\. \(2023\)Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh\.GPTQ: Accurate post\-training quantization for generative pre\-trained transformers\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.
- Gholami et al\. \(2021\)Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W\. Mahoney, and Kurt Keutzer\.A survey of quantization methods for efficient neural network inference\.*arXiv preprint arXiv:2103\.13630*, 2021\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Hassibi et al\. \(1993\)Babak Hassibi, David G\. Stork, and Gregory J\. Wolff\.Optimal brain surgeon and general network pruning\.In*IEEE International Conference on Neural Networks*, 1993\.
- Kim et al\. \(2025\)Jinuk Kim, Marwa El Halabi, Wonpyo Park, Clemens J\. S\. Schaefer, Deokjae Lee, Yeonhong Park, Jae W\. Lee, and Hyun Oh Song\.GuidedQuant: Large language model quantization via exploiting end loss guidance\.In*International Conference on Machine Learning \(ICML\)*, 2025\.
- Kim et al\. \(2024\)Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W\. Mahoney, and Kurt Keutzer\.SqueezeLLM: Dense\-and\-sparse quantization\.In*International Conference on Machine Learning \(ICML\)*, 2024\.
- Kingma and Ba \(2014\)Diederik P Kingma and Jimmy Ba\.Adam: A method for stochastic optimization\.*arXiv preprint arXiv:1412\.6980*, 2014\.
- Kong et al\. \(2026\)Jason Kong, Nilesh Prasad Pandey, Flavio Ponzina, and Tajana Rosing\.A KL lens on quantization: Fast, forward\-only sensitivity for mixed\-precision SSM\-transformer models\.*arXiv preprint arXiv:2604\.13440*, 2026\.
- LeCun et al\. \(1989\)Yann LeCun, John S\. Denker, and Sara A\. Solla\.Optimal brain damage\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 1989\.
- Li et al\. \(2021\)Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu\.BRECQ: Pushing the limit of post\-training quantization by block reconstruction\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.
- Li et al\. \(2025\)Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, and Priyadarshini Panda\.GPTAQ: Efficient finetuning\-free quantization for asymmetric calibration\.*arXiv preprint arXiv:2504\.02692*, 2025\.
- Lin et al\. \(2024\)Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei\-Ming Chen, Wei\-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han\.AWQ: Activation\-aware weight quantization for LLM compression and acceleration\.In*MLSys*, 2024\.
- Liu et al\. \(2021\)Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing\-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang\.Group Fisher pruning for practical network compression\.In*International Conference on Machine Learning \(ICML\)*, 2021\.
- Liu et al\. \(2024a\)Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra\.LLM\-QAT: Data\-free quantization aware training for large language models\.In*Findings of the Association for Computational Linguistics \(ACL\)*, 2024a\.
- Liu et al\. \(2024b\)Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort\.SpinQuant: LLM quantization with learned rotations\.*arXiv preprint arXiv:2405\.16406*, 2024b\.
- Malinovskii et al\. \(2024\)Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik\.PV\-Tuning: Beyond straight\-through estimation for extreme LLM compression\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.
- Martens and Grosse \(2015\)James Martens and Roger Grosse\.Optimizing neural networks with Kronecker\-factored approximate curvature\.In*International Conference on Machine Learning \(ICML\)*, 2015\.
- Merity et al\. \(2017\)Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\.Pointer sentinel mixture models\.In*International Conference on Learning Representations \(ICLR\)*, 2017\.
- Shao et al\. \(2024\)Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo\.OmniQuant: Omnidirectionally calibrated quantization for large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Singh and Alistarh \(2020\)Sidak Pal Singh and Dan Alistarh\.WoodFisher: Efficient second\-order approximation for neural network compression\.*Advances in Neural Information Processing Systems*, 33:18098–18109, 2020\.
- Sun et al\. \(2025\)Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, and Jun Yao\.FlatQuant: Flatness matters for LLM quantization\.In*International Conference on Machine Learning \(ICML\)*, 2025\.
- Tseng et al\. \(2024\)Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa\.QuIP\#: Even better LLM quantization with Hadamard incoherence and lattice codebooks\.In*International Conference on Machine Learning \(ICML\)*, 2024\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Zhao et al\. \(2023\)Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien\-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al\.PyTorch FSDP: Experiences on scaling fully sharded data parallel\.*arXiv preprint arXiv:2304\.11277*, 2023\.

## Appendix AREAL\-Q Algorithm Pseudocode and Sliding\-Window Illustration

Algorithm 1REAL\-Q Quantization0:Pre\-trained model with weights

\{𝐖\(ℓ\)\}\\\{\\mathbf\{W\}^\{\(\\ell\)\}\\\}, calibration set

𝒟\\mathcal\{D\}, block size

BB, learning rate

η\\eta
1:Stage 0:Run end\-to\-end backward on

𝒟\\mathcal\{D\}to compute aggregated Fisher matrix

𝐅\(ℓ\)\\mathbf\{F\}^\{\(\\ell\)\}and saliency

𝐬\(ℓ\)\\mathbf\{s\}^\{\(\\ell\)\}for all layers

ℓ\\ell
2:forlayer

ℓ=1,…,L\\ell=1,\\ldots,Ldo

3:foreach linear module in layer

ℓ\\elldo

4:Compute saliency\-weighted Hessian

𝐇g\\mathbf\{H\}\_\{g\}and its inverse

\(𝐇g\)−1\(\\mathbf\{H\}\_\{g\}\)^\{\-1\}for all row\-groups

gg
5:forblock

b=1,…,⌈n/B⌉b=1,\\ldots,\\lceil n/B\\rceildo

6:// Standard GPTQ within block \(executed parallelly across groups\)

7:forcolumn

jjin block

bbdo

8:

𝐰^g,j←quant​\(𝐖g,j\)\\hat\{\\mathbf\{w\}\}\_\{g,j\}\\leftarrow\\text\{quant\}\(\\mathbf\{W\}\_\{g,j\}\);

𝜹g,j←𝐖g,j−𝐰^g,j\(∀g\)\\boldsymbol\{\\delta\}\_\{g,j\}\\leftarrow\\mathbf\{W\}\_\{g,j\}\-\\hat\{\\mathbf\{w\}\}\_\{g,j\}\\quad\(\\forall g\)
9:Update remaining columns in block via Eq\.[2](https://arxiv.org/html/2609.00049#S3.E2)

\(∀g\)\(\\forall g\)
10:endfor

11:// Second\-order compensation for trailing columns

12:

𝐖g,bend:n←𝐖g,bend:n−𝐄g,b\(𝐇g−1\)b,bend:n\(∀g\)\\mathbf\{W\}\_\{g,b\_\{\\text\{end\}\}:n\}\\leftarrow\\mathbf\{W\}\_\{g,b\_\{\\text\{end\}\}:n\}\-\\mathbf\{E\}\_\{g,b\}\\,\(\\mathbf\{H\}\_\{g\}^\{\-1\}\)\_\{b,b\_\{\\text\{end\}\}:n\}\\quad\(\\forall g\)
13:// Block\-wise gradient descent \(jointly updates the full unquantized matrix\)

14:// Note:bglobalb\_\{\\text\{global\}\}is the cumulative column\-block step within the current transformer block, summed across all linear modules in the block \(matches the indexbbin Eq\.[8](https://arxiv.org/html/2609.00049#S4.E8)\)\.

15:if

ℓ=L\\ell=Lthen

16:

ℒ\(b\)←\\mathcal\{L\}^\{\(b\)\}\\leftarrowtrue KL divergence

ℒKL\\mathcal\{L\}\_\{\\text\{KL\}\}against the LM head

17:else

18:

ℒ\(b\)←\\mathcal\{L\}^\{\(b\)\}\\leftarrowsliding\-window loss

ℒslide\(bglobal\)\\mathcal\{L\}\_\{\\text\{slide\}\}^\{\(b\_\{\\text\{global\}\}\)\}\(Eq\.[8](https://arxiv.org/html/2609.00049#S4.E8)\) using Fisher MSE

19:endif

20:

𝐠\(b\)←∇𝐖:,bend:nℒ\(b\)\\mathbf\{g\}^\{\(b\)\}\\leftarrow\\nabla\_\{\\mathbf\{W\}\_\{:,b\_\{\\text\{end\}\}:n\}\}\\mathcal\{L\}^\{\(b\)\}
21:

𝐖:,bend:n←𝐖:,bend:n−η⋅Adam\(𝐠\(b\)\)\\mathbf\{W\}\_\{:,b\_\{\\text\{end\}\}:n\}\\leftarrow\\mathbf\{W\}\_\{:,b\_\{\\text\{end\}\}:n\}\-\\eta\\cdot\\text\{Adam\}\(\\mathbf\{g\}^\{\(b\)\}\)
22:endfor

23:endfor

24:endfor

![Refer to caption](https://arxiv.org/html/2609.00049v1/figures/fig3_slidewindow_v3.png)Figure 4:Illustration of the loss sliding window mechanism \(referenced from Section[4\.4](https://arxiv.org/html/2609.00049#S4.SS4)\)\. When quantizing transformer blockℓ\\ell, the weight ofℒFisher\(ℓ\)\\mathcal\{L\}\_\{\\text\{Fisher\}\}^\{\(\\ell\)\}\(blue\) linearly decreases whileℒFisher\(ℓ\+1\)\\mathcal\{L\}\_\{\\text\{Fisher\}\}^\{\(\\ell\+1\)\}\(orange\) linearly increases across the column blocks of the linear modules in transformer blockℓ\\ell\.Right:without the sliding window, the objective exhibits a sharp discontinuity at the transformer\-block boundary; with it, the transition is a smooth linear interpolation\.
## Appendix BExtended Related Work

#### Weight\-only PTQ methods\.

A complementary line of work targets aggressive low\-bit weight\-only quantization, where activations stay in higher precision\. AWQ\[[Lin et al\., 2024](https://arxiv.org/html/2609.00049#bib.bib23)\]scales salient weight channels prior to quantization to protect them from rounding error; SqueezeLLM\[[Kim et al\., 2024](https://arxiv.org/html/2609.00049#bib.bib17)\]uses sensitivity\-based non\-uniform clustering of weights; SpQR\[[Dettmers et al\., 2024](https://arxiv.org/html/2609.00049#bib.bib7)\]keeps a small number of outlier weights at higher precision and quantizes the rest more aggressively; OmniQuant\[[Shao et al\., 2024](https://arxiv.org/html/2609.00049#bib.bib30)\]learns clipping ranges and weight transformations end\-to\-end on calibration data; AQLM\[[Egiazarian et al\., 2024](https://arxiv.org/html/2609.00049#bib.bib10)\]uses additive vector quantization for sub\-3\-bit weight compression\. These methods are largely orthogonal to REAL\-Q: they design new representations or optimize auxiliary quantization parameters \(scales/shifts\), while REAL\-Q targets the sequential weight\-updating solver itself\. REAL\-Q could in principle be combined with any of these representations\.

#### Incoherence and rotation\-based methods\.

A separate family applies orthogonal transforms to the weight and activation matrices before quantization to reduce outlier magnitudes and produce more uniform distributions\. QuIP\[[Chee et al\., 2023](https://arxiv.org/html/2609.00049#bib.bib5)\]introduces the incoherence\-processing framework with random orthogonal matrices and adaptive rounding; QuIP\#\[[Tseng et al\., 2024](https://arxiv.org/html/2609.00049#bib.bib33)\]extends this with structured \(Hadamard\) rotations and lattice codebooks\. SliceGPT\[[Ashkboos et al\., 2024a](https://arxiv.org/html/2609.00049#bib.bib2)\]uses orthogonal transformations to enable post\-training structural pruning by safely reducing model dimensionality\. Furthermore, methods like SpinQuant\[[Liu et al\., 2024b](https://arxiv.org/html/2609.00049#bib.bib26)\]and FlatQuant\[[Sun et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib32)\]actively learn rotation matrices or affine transformations to optimize weight and activation distributions\. These pre\-processing steps are complementary to the OBS\-style update used by REAL\-Q—we adopt QuaRot\[[Ashkboos et al\., 2024b](https://arxiv.org/html/2609.00049#bib.bib3)\]as a default rotation pre\-processing in all experiments\.

#### Quantization\-aware training and fine\-tuning\.

A different class of methods trades calibration efficiency for additional training: LLM\-QAT\[[Liu et al\., 2024a](https://arxiv.org/html/2609.00049#bib.bib25)\]performs full quantization\-aware training with distillation; LR\-QAT\[[Bondarenko et al\., 2024](https://arxiv.org/html/2609.00049#bib.bib4)\]reduces the cost of QAT via low\-rank adapters; EfficientQAT\[[Chen et al\., 2025](https://arxiv.org/html/2609.00049#bib.bib6)\]uses block\-wise alternating optimization of weights and quantization parameters; PV\-tuning\[[Malinovskii et al\., 2024](https://arxiv.org/html/2609.00049#bib.bib27)\]fine\-tunes the discrete codebook used by extreme low\-bit quantization methods\. These methods generally achieve higher accuracy than PTQ at low bit\-widths but require multiple full epochs through training data and substantially more compute \(typically1010–1000×1000\\timesthat of PTQ\)\. REAL\-Q sits firmly on the PTQ side of this trade\-off: it incorporates first\-order gradient corrections entirely within the single\-pass GPTQ calibration pipeline, never updating dequantized weights globally or requiring full training epochs\.

#### Hessian and Fisher information in compression\.

Second\-order information has been extensively used for network compression\. WoodFisher\[[Singh and Alistarh, 2020](https://arxiv.org/html/2609.00049#bib.bib31)\]demonstrated the effectiveness of using the empirical Fisher matrix as a tractable proxy for the exact Hessian in OBS\-based structural pruning\. Similarly,[Liu et al\. \[2021\]](https://arxiv.org/html/2609.00049#bib.bib24)derives Fisher\-based metrics for structured pruning\. HAWQ\[[Dong et al\., 2019](https://arxiv.org/html/2609.00049#bib.bib8)\]and HAWQ\-V2\[[Dong et al\., 2020](https://arxiv.org/html/2609.00049#bib.bib9)\]use Hutchinson’s algorithm to efficiently estimate Hessian eigenvalues and traces for automated mixed\-precision quantization\. K\-FAC\[[Martens and Grosse, 2015](https://arxiv.org/html/2609.00049#bib.bib28)\]approximates the Fisher matrix as Kronecker products for efficient natural gradient computation\[[Amari, 1998](https://arxiv.org/html/2609.00049#bib.bib1)\]\.

#### Comparison with BRECQ\.

BRECQ\[[Li et al\., 2021](https://arxiv.org/html/2609.00049#bib.bib21)\]was originally proposed for CNN quantization, using a block\-output Fisher\-weighted reconstruction loss that is conceptually similar to our per\-transformer\-block Fisher MSE\. However, BRECQ operates in a fundamentally different continuous optimization regime: it jointly optimizes all weight parameters of a residual block as continuous variables with a regularization penalty, rather than operating within a discrete, sequential GPTQ\-style pipeline\. Moreover, even if BRECQ’s token\-by\-token diagonal Fisher objective were directly adapted for a transformer architecture, it would reduce to the same saliency\-weighted Hessian approximation employed by GuidedQuant\. To make this reduction explicit, BRECQ’s per\-token diagonal\-Fisher reconstruction loss for a linear module𝐖^\\hat\{\\mathbf\{W\}\}takes the form

ℒBRECQ=1T​∑t=1T\(Δ​𝐖𝐱t\)⊤​diag​\(𝐠t⊙𝐠t\)​\(Δ​𝐖𝐱t\),\\mathcal\{L\}\_\{\\text\{BRECQ\}\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\(\\Delta\\mathbf\{W\}\\mathbf\{x\}\_\{t\}\)^\{\\top\}\\mathrm\{diag\}\(\\mathbf\{g\}\_\{t\}\\odot\\mathbf\{g\}\_\{t\}\)\\,\(\\Delta\\mathbf\{W\}\\mathbf\{x\}\_\{t\}\),\(12\)whereΔ​𝐖=𝐖−𝐖^\\Delta\\mathbf\{W\}=\\mathbf\{W\}\-\\hat\{\\mathbf\{W\}\}and𝐠t=∇𝐲tℒtask\\mathbf\{g\}\_\{t\}=\\nabla\_\{\\mathbf\{y\}\_\{t\}\}\\mathcal\{L\}\_\{\\text\{task\}\}is the per\-token output gradient\. Expanding the diagonal weighting and grouping terms by output rowii,

ℒBRECQ=1T∑i=1m∑t=1Tgt,i2\(Δ𝐖i,:𝐱t\)2=∑i=1mΔ𝐖i,:\(1T​∑t=1Tgt,i2​𝐱t​𝐱t⊤\)⏟=:𝐇iΔ𝐖i,:⊤,\\mathcal\{L\}\_\{\\text\{BRECQ\}\}=\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{m\}\\sum\_\{t=1\}^\{T\}g\_\{t,i\}^\{2\}\\,\(\\Delta\\mathbf\{W\}\_\{i,:\}\\mathbf\{x\}\_\{t\}\)^\{2\}=\\sum\_\{i=1\}^\{m\}\\Delta\\mathbf\{W\}\_\{i,:\}\\underbrace\{\\Bigl\(\\tfrac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}g\_\{t,i\}^\{2\}\\,\\mathbf\{x\}\_\{t\}\\mathbf\{x\}\_\{t\}^\{\\top\}\\Bigr\)\}\_\{=:\\,\\mathbf\{H\}\_\{i\}\}\\Delta\\mathbf\{W\}\_\{i,:\}^\{\\top\},\(13\)which is exactly a per\-output\-row, saliency\-weighted Hessian quadratic\. This is theNg→mN\_\{g\}\\\!\\to\\\!mlimit of GuidedQuant’s grouped Hessian \(Eq\.[3](https://arxiv.org/html/2609.00049#S3.E3)\); GuidedQuant simply pools rows intoNgN\_\{g\}groups for tractability\. The structural limitations we discuss for GuidedQuant therefore apply identically, and the gradient correlation of this objective is shown to be low in our experiments \(see Figure[3](https://arxiv.org/html/2609.00049#S6.F3)\)\.

## Appendix CTheoretical Analysis with Proofs

### C\.1Proof of Decoupled Fisher Aggregation \(Proposition[4\.1](https://arxiv.org/html/2609.00049#S4.Thmtheorem1)\)

Here we provide the exact mathematical derivation for the approximation error introduced by the mean\-field decoupling in Proposition[4\.1](https://arxiv.org/html/2609.00049#S4.Thmtheorem1)\.

Using the cyclic property of the trace operator and the linearity of expectation, the exact expected second\-order penalty can be rewritten as:

𝔼t​\[Δ​𝐲t⊤​𝐇t​Δ​𝐲t\]=𝔼t​\[Tr​\(Δ​𝐲t⊤​𝐇t​Δ​𝐲t\)\]=Tr​\(𝔼t​\[𝐇t​Δ​𝐲t​Δ​𝐲t⊤\]\)\.\\mathbb\{E\}\_\{t\}\\bigl\[\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\\mathbf\{H\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}\\bigr\]=\\mathbb\{E\}\_\{t\}\\bigl\[\\text\{Tr\}\(\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\\mathbf\{H\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}\)\\bigr\]=\\text\{Tr\}\\bigl\(\\mathbb\{E\}\_\{t\}\[\\mathbf\{H\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\]\\bigr\)\.\(14\)Similarly, the decoupled expectation used in our surrogate formulation evaluates to:

𝔼t​\[Δ​𝐲t⊤​𝔼t​\[𝐇t\]​Δ​𝐲t\]=Tr​\(𝔼t​\[𝐇t\]⋅𝔼t​\[Δ​𝐲t​Δ​𝐲t⊤\]\)\.\\mathbb\{E\}\_\{t\}\\bigl\[\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\\,\\mathbb\{E\}\_\{t\}\[\\mathbf\{H\}\_\{t\}\]\\,\\Delta\\mathbf\{y\}\_\{t\}\\bigr\]=\\text\{Tr\}\\bigl\(\\mathbb\{E\}\_\{t\}\[\\mathbf\{H\}\_\{t\}\]\\cdot\\mathbb\{E\}\_\{t\}\[\\Delta\\mathbf\{y\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\]\\bigr\)\.\(15\)By utilizing the linearity of the trace operator and the fact that both𝐇t\\mathbf\{H\}\_\{t\}andΔ​𝐲t​Δ​𝐲t⊤\\Delta\\mathbf\{y\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}are symmetric, the trace of their product is equivalent to their Frobenius inner product\. Consequently, the difference between the exact joint expectation and the product of their marginal expectations exactly amounts to the sum of their element\-wise covariances:

Tr​\(𝔼t​\[𝐇t​Δ​𝐲t​Δ​𝐲t⊤\]\)−Tr​\(𝔼t​\[𝐇t\]⋅𝔼t​\[Δ​𝐲t​Δ​𝐲t⊤\]\)=∑i,jCov​\(\(𝐇t\)i​j,\(Δ​𝐲t​Δ​𝐲t⊤\)i​j\)\.\\text\{Tr\}\\bigl\(\\mathbb\{E\}\_\{t\}\[\\mathbf\{H\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\]\\bigr\)\-\\text\{Tr\}\\bigl\(\\mathbb\{E\}\_\{t\}\[\\mathbf\{H\}\_\{t\}\]\\cdot\\mathbb\{E\}\_\{t\}\[\\Delta\\mathbf\{y\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\]\\bigr\)=\\sum\_\{i,j\}\\text\{Cov\}\\bigl\(\(\\mathbf\{H\}\_\{t\}\)\_\{ij\},\(\\Delta\\mathbf\{y\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\)\_\{ij\}\\bigr\)\.\(16\)Combined with the standard Fisher–Hessian identity𝔼t​\[𝐇t\]≈𝐅\\mathbb\{E\}\_\{t\}\[\\mathbf\{H\}\_\{t\}\]\\approx\\mathbf\{F\}, the decoupled formulation directly yields our objectiveℒFisher\\mathcal\{L\}\_\{\\text\{Fisher\}\}in Eq\.[6](https://arxiv.org/html/2609.00049#S4.E6)\. The mathematical gap between the exact expected penalty and the decoupled surrogate is exactly the trace covarianceTr​\(Cov​\(𝐇t,Δ​𝐲t​Δ​𝐲t⊤\)\)\\text\{Tr\}\\bigl\(\\text\{Cov\}\(\\mathbf\{H\}\_\{t\},\\Delta\\mathbf\{y\}\_\{t\}\\Delta\\mathbf\{y\}\_\{t\}^\{\\top\}\)\\bigr\), which we explicitly trade for global cross\-channel structural fidelity as discussed in Section[4\.2](https://arxiv.org/html/2609.00049#S4.SS2)\.□\\square

### C\.2Error Accumulation in Column\-wise Quantization

Consider a single linear layer with weight matrix𝐖∈ℝm×n\\mathbf\{W\}\\in\\mathbb\{R\}^\{m\\times n\}and input𝐱∈ℝn\\mathbf\{x\}\\in\\mathbb\{R\}^\{n\}\. The standard layer\-wise quantization objective is:

min𝐖^⁡𝔼𝐱​\[‖𝐖𝐱−𝐖^​𝐱‖2\]=min𝐖^⁡Tr​\(\(𝐖−𝐖^\)​𝐇​\(𝐖−𝐖^\)⊤\),\\min\_\{\\hat\{\\mathbf\{W\}\}\}\\;\\mathbb\{E\}\_\{\\mathbf\{x\}\}\\left\[\\\|\\mathbf\{W\}\\mathbf\{x\}\-\\hat\{\\mathbf\{W\}\}\\mathbf\{x\}\\\|^\{2\}\\right\]=\\min\_\{\\hat\{\\mathbf\{W\}\}\}\\;\\text\{Tr\}\\left\(\(\\mathbf\{W\}\-\\hat\{\\mathbf\{W\}\}\)\\mathbf\{H\}\(\\mathbf\{W\}\-\\hat\{\\mathbf\{W\}\}\)^\{\\top\}\\right\),\(17\)where𝐇=𝔼⁡\[𝐱𝐱⊤\]∈ℝn×n\\mathbf\{H\}=\\mathbb\{E\}\[\\mathbf\{x\}\\mathbf\{x\}^\{\\top\}\]\\in\\mathbb\{R\}^\{n\\times n\}is the input covariance \(Hessian of the quadratic reconstruction loss\)\.

###### Theorem C\.1\(Reconstruction error of GPTQ; restated from[Frantar et al\. \[2022\]](https://arxiv.org/html/2609.00049#bib.bib11)\)\.

Let𝐖\\mathbf\{W\}be quantized column\-by\-column with GPTQ using a fixed Hessian𝐇\\mathbf\{H\}\. Letϵj∈ℝm\\boldsymbol\{\\epsilon\}\_\{j\}\\in\\mathbb\{R\}^\{m\}denote the per\-column quantization residual at stepjj, defined asϵj=𝐖:,j\(j\)−Q\(𝐖:,j\(j\)\)\\boldsymbol\{\\epsilon\}\_\{j\}=\\mathbf\{W\}^\{\(j\)\}\_\{:,j\}\-Q\(\\mathbf\{W\}^\{\(j\)\}\_\{:,j\}\)\. The total layer\-wise reconstruction error equalsℒGPTQ=∑j=1n‖ϵj‖22/\[𝐇−1\]j​j\\mathcal\{L\}\_\{\\text\{GPTQ\}\}=\\sum\_\{j=1\}^\{n\}\\\|\\boldsymbol\{\\epsilon\}\_\{j\}\\\|\_\{2\}^\{2\}/\[\\mathbf\{H\}^\{\-1\}\]\_\{jj\}, exactly under the fixed\-Hessian objective\.

#### Proof of Theorem[C\.1](https://arxiv.org/html/2609.00049#A3.Thmtheorem1)\.

This follows from the OBS framework\[[Hassibi et al\., 1993](https://arxiv.org/html/2609.00049#bib.bib15),[Frantar et al\., 2022](https://arxiv.org/html/2609.00049#bib.bib11)\]\. At stepjj, quantizing columnjjintroduces errorϵj\\boldsymbol\{\\epsilon\}\_\{j\}and the optimal compensation for remaining columns isδ𝐖:,j\+1:n=−ϵj⋅\[𝐇−1\]j​j−1⋅𝐇j,j\+1:n−1\\delta\\mathbf\{W\}\_\{:,j\+1:n\}=\-\\boldsymbol\{\\epsilon\}\_\{j\}\\cdot\[\\mathbf\{H\}^\{\-1\}\]\_\{jj\}^\{\-1\}\\cdot\\mathbf\{H\}^\{\-1\}\_\{j,j\+1:n\}\. By the Schur complement structure of𝐇−1\\mathbf\{H\}^\{\-1\}, each row independently incurs a loss increase ofϵi​j2/\[𝐇−1\]j​j\\epsilon\_\{ij\}^\{2\}/\[\\mathbf\{H\}^\{\-1\}\]\_\{jj\}\. Summing over allmmrows andnncolumns yields the exact closed\-form expression\.□\\square

###### Lemma C\.2\(Static Approximation Penalty\)\.

Let𝐇ideal\(j\)\\mathbf\{H\}\_\{\\mathrm\{ideal\}\}^\{\(j\)\}denote the true downstream\-aware Hessian at stepjj, andΔ​𝐇\(j\)=𝐇ideal\(j\)−𝐇\\Delta\\mathbf\{H\}^\{\(j\)\}=\\mathbf\{H\}\_\{\\mathrm\{ideal\}\}^\{\(j\)\}\-\\mathbf\{H\}denote the gap of the cached static Hessian\. In GPTQ, the weight update at stepjjspans multiple columns via compensation, forming a rank\-1 perturbationΔ​𝐖\(j\)=ϵj​𝐯j⊤\\Delta\\mathbf\{W\}^\{\(j\)\}=\\boldsymbol\{\\epsilon\}\_\{j\}\\mathbf\{v\}\_\{j\}^\{\\top\}, where𝐯j∈ℝn\\mathbf\{v\}\_\{j\}\\in\\mathbb\{R\}^\{n\}contains the static Hessian dependencies\. Assuming the structural drift makes this static compensation increasingly misaligned, such that the projection gap is lower\-bounded by𝐯j⊤​Δ​𝐇\(j\)​𝐯j≥c⋅\(j/n\)⋅‖𝐇‖2\\mathbf\{v\}\_\{j\}^\{\\top\}\\Delta\\mathbf\{H\}^\{\(j\)\}\\mathbf\{v\}\_\{j\}\\geq c\\cdot\(j/n\)\\cdot\\\|\\mathbf\{H\}\\\|\_\{2\}for some constantc\>0c\>0, the expected cumulative uncompensated penalty scales as:

𝔼⁡\[ℰ\]=Ω⁡\(n⋅ϵ¯2⋅‖𝐇‖2\),\\mathbb\{E\}\[\\mathcal\{E\}\]=\\Omega\(n\\cdot\\bar\{\\epsilon\}^\{2\}\\cdot\\\|\\mathbf\{H\}\\\|\_\{2\}\),\(18\)whereϵ¯2=𝔼⁡\[‖ϵj‖22\]\\bar\{\\epsilon\}^\{2\}=\\mathbb\{E\}\[\\\|\\boldsymbol\{\\epsilon\}\_\{j\}\\\|\_\{2\}^\{2\}\]is the expected squared residual norm\.

#### Proof of Lemma[C\.2](https://arxiv.org/html/2609.00049#A3.Thmtheorem2)\.

At stepjj, quantizing columnjjintroduces an immediate residualϵj=𝐖:,j\(j\)−Q\(𝐖:,j\(j\)\)\\boldsymbol\{\\epsilon\}\_\{j\}=\\mathbf\{W\}^\{\(j\)\}\_\{:,j\}\-Q\(\\mathbf\{W\}^\{\(j\)\}\_\{:,j\}\)\(defined as in Theorem[C\.1](https://arxiv.org/html/2609.00049#A3.Thmtheorem1), with𝐖\(j\)\\mathbf\{W\}^\{\(j\)\}the weight matrix entering stepjj\)\. Following the OBS framework\[[Frantar et al\., 2022](https://arxiv.org/html/2609.00049#bib.bib11)\], GPTQ applies a correlated update to the remaining unquantized columns to compensate\. The exact step\-wise weight perturbation \(new minus old\) is a rank\-1 matrixΔ​𝐖\(j\)=ϵj​𝐯j⊤\\Delta\\mathbf\{W\}^\{\(j\)\}=\\boldsymbol\{\\epsilon\}\_\{j\}\\mathbf\{v\}\_\{j\}^\{\\top\}, where the compensation vector𝐯j∈ℝn\\mathbf\{v\}\_\{j\}\\in\\mathbb\{R\}^\{n\}is strictly defined as:

\[𝐯j\]k=\{0if​k<j−1if​k=j−\[𝐇−1\]j,k/\[𝐇−1\]j​jif​k\>j\[\\mathbf\{v\}\_\{j\}\]\_\{k\}=\\begin\{cases\}0&\\text\{if \}k<j\\\\ \-1&\\text\{if \}k=j\\\\ \-\[\\mathbf\{H\}^\{\-1\}\]\_\{j,k\}/\[\\mathbf\{H\}^\{\-1\}\]\_\{jj\}&\\text\{if \}k\>j\\end\{cases\}\(19\)Evaluating this exact GPTQ update under the true downstream landscape, the actual step\-wise penalty gap introduced by the stale Hessian is the quadratic form:

ξj=Tr​\(Δ​𝐖\(j\)​Δ​𝐇\(j\)​\(Δ​𝐖\(j\)\)⊤\)=\(𝐯j⊤​Δ​𝐇\(j\)​𝐯j\)⋅‖ϵj‖22\.\\xi\_\{j\}=\\text\{Tr\}\\left\(\\Delta\\mathbf\{W\}^\{\(j\)\}\\Delta\\mathbf\{H\}^\{\(j\)\}\(\\Delta\\mathbf\{W\}^\{\(j\)\}\)^\{\\top\}\\right\)=\\left\(\\mathbf\{v\}\_\{j\}^\{\\top\}\\Delta\\mathbf\{H\}^\{\(j\)\}\\mathbf\{v\}\_\{j\}\\right\)\\cdot\\\|\\boldsymbol\{\\epsilon\}\_\{j\}\\\|\_\{2\}^\{2\}\.\(20\)
In deep neural networks with non\-quadratic loss landscapes, the true local Hessian𝐇ideal\(j\)\\mathbf\{H\}\_\{\\mathrm\{ideal\}\}^\{\(j\)\}deviates from the initial static Hessian𝐇\\mathbf\{H\}proportional to the accumulated distance traversed in the weight space\. As the column indexjjprogresses, the accumulated weight displacement grows monotonically\. It is therefore reasonable to assume that the structural drift penalty along the compensation direction,𝐯j⊤​Δ​𝐇\(j\)​𝐯j\\mathbf\{v\}\_\{j\}^\{\\top\}\\Delta\\mathbf\{H\}^\{\(j\)\}\\mathbf\{v\}\_\{j\}, worsens at least linearly with the macroscopic quantization progressj/nj/n\.

Formalizing this motivated assumption \(𝐯j⊤​Δ​𝐇\(j\)​𝐯j≥c​jn​‖𝐇‖2\\mathbf\{v\}\_\{j\}^\{\\top\}\\Delta\\mathbf\{H\}^\{\(j\)\}\\mathbf\{v\}\_\{j\}\\geq c\\frac\{j\}\{n\}\\\|\\mathbf\{H\}\\\|\_\{2\}for somec\>0c\>0\), the step\-wise penalty gap becomesξj≥c​jn​‖𝐇‖2​‖ϵj‖22\\xi\_\{j\}\\geq c\\frac\{j\}\{n\}\\\|\\mathbf\{H\}\\\|\_\{2\}\\\|\\boldsymbol\{\\epsilon\}\_\{j\}\\\|\_\{2\}^\{2\}\.

Recognizing that in sequential quantization, the magnitude of the local quantization residual‖ϵj‖22\\\|\\boldsymbol\{\\epsilon\}\_\{j\}\\\|\_\{2\}^\{2\}is primarily determined by local weight variance, it can be reasonably modeled as statistically independent of the macroscopic structural drift indexjj\. Taking the expectation over the random quantization residuals, we can decouple the summation due to linearity:

𝔼\[∑j=1nξj\]≥c⋅∥𝐇∥2∑j=1njn𝔼\[∥ϵj∥22\]\.\\mathbb\{E\}\\left\[\\sum\_\{j=1\}^\{n\}\\xi\_\{j\}\\right\]\\geq c\\cdot\\\|\\mathbf\{H\}\\\|\_\{2\}\\sum\_\{j=1\}^\{n\}\\frac\{j\}\{n\}\\mathbb\{E\}\\left\[\\\|\\boldsymbol\{\\epsilon\}\_\{j\}\\\|\_\{2\}^\{2\}\\right\]\.\(21\)Lettingϵ¯2=𝔼⁡\[‖ϵj‖22\]\\bar\{\\epsilon\}^\{2\}=\\mathbb\{E\}\[\\\|\\boldsymbol\{\\epsilon\}\_\{j\}\\\|\_\{2\}^\{2\}\]denote the expected squared residual norm, the cumulative expected penalty evaluates to:

c⋅∥𝐇∥2⋅ϵ¯2∑j=1njn=c⋅∥𝐇∥2⋅ϵ¯2⋅n⁡\(n\+1\)2​n\.c\\cdot\\\|\\mathbf\{H\}\\\|\_\{2\}\\cdot\\bar\{\\epsilon\}^\{2\}\\sum\_\{j=1\}^\{n\}\\frac\{j\}\{n\}=c\\cdot\\\|\\mathbf\{H\}\\\|\_\{2\}\\cdot\\bar\{\\epsilon\}^\{2\}\\cdot\\frac\{n\(n\+1\)\}\{2n\}\.\(22\)Sincen\+12≥n2\\frac\{n\+1\}\{2\}\\geq\\frac\{n\}\{2\}, the expected total cumulative penalty is lower\-bounded byc2⋅n⋅ϵ¯2⋅‖𝐇‖2\\frac\{c\}\{2\}\\cdot n\\cdot\\bar\{\\epsilon\}^\{2\}\\cdot\\\|\\mathbf\{H\}\\\|\_\{2\}\. This establishes the asymptotic lower bound𝔼⁡\[ℰ\]=Ω⁡\(n⋅ϵ¯2⋅‖𝐇‖2\)\\mathbb\{E\}\[\\mathcal\{E\}\]=\\Omega\(n\\cdot\\bar\{\\epsilon\}^\{2\}\\cdot\\\|\\mathbf\{H\}\\\|\_\{2\}\), showing that under the stated assumption, static analytical solvers cannot prevent compounding cross\-column errors\.□\\square

### C\.3Descent Condition Analysis under First\-Order Updates

Prior globally\-guided PTQ methods rely on structural approximations to maintain the tractability of analytical solvers\. In this section, we analyze REAL\-Q through the lens of continuous surrogate optimization theory, deriving the condition under which descending our surrogate guarantees a decrease in the true end\-to\-end objective\.

###### Theorem C\.3\(End\-to\-End Descent Condition\)\.

Assume the true end\-to\-end objectiveJ⁡\(𝐖\)J\(\\mathbf\{W\}\)isβ\\beta\-smooth\. We update the trailing un\-quantized weights using a gradient step driven by our surrogate objective:𝐖\(\+\)=𝐖−η​𝐠surr\\mathbf\{W\}^\{\(\+\)\}=\\mathbf\{W\}\-\\eta\\mathbf\{g\}\_\{\\mathrm\{surr\}\}, where𝐠surr=∇J~​\(𝐖\)\\mathbf\{g\}\_\{\\mathrm\{surr\}\}=\\nabla\\tilde\{J\}\(\\mathbf\{W\}\)\(J~\\tilde\{J\}denotes the surrogate objective\)\. To strictly guarantee descent on the true global loss \(J⁡\(𝐖\(\+\)\)<J⁡\(𝐖\)J\(\\mathbf\{W\}^\{\(\+\)\}\)<J\(\\mathbf\{W\}\)\), the cosine similarity \(cos⁡θ\\cos\\theta\) between the surrogate gradient𝐠surr\\mathbf\{g\}\_\{\\mathrm\{surr\}\}and the true end\-to\-end gradient∇J​\(𝐖\)\\nabla J\(\\mathbf\{W\}\)must satisfy the exact strict inequality:

cos⁡θ\>η​β2⋅‖𝐠surr‖‖∇J​\(𝐖\)‖\.\\cos\\theta\>\\frac\{\\eta\\beta\}\{2\}\\cdot\\frac\{\\\|\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\\|\}\{\\\|\\nabla J\(\\mathbf\{W\}\)\\\|\}\.\(23\)

#### Proof of Theorem[C\.3](https://arxiv.org/html/2609.00049#A3.Thmtheorem3)\.

We rely on the standard Descent Lemma forβ\\beta\-smooth functions, which establishes a strict quadratic upper bound on the objective value after a parameter update:

J⁡\(𝐖−η​𝐠surr\)≤J⁡\(𝐖\)−η⁡⟨∇J​\(𝐖\),𝐠surr⟩\+η2​β2​‖𝐠surr‖2\.J\(\\mathbf\{W\}\-\\eta\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\)\\leq J\(\\mathbf\{W\}\)\-\\eta\\langle\\nabla J\(\\mathbf\{W\}\),\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\rangle\+\\frac\{\\eta^\{2\}\\beta\}\{2\}\\\|\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\\|^\{2\}\.\(24\)To ensure strict descent, the true loss after the update must be less than the initial loss \(J⁡\(𝐖−η​𝐠surr\)<J⁡\(𝐖\)J\(\\mathbf\{W\}\-\\eta\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\)<J\(\\mathbf\{W\}\)\)\. This requires the sum of the linear and quadratic terms to be strictly negative:

−η⁡⟨∇J​\(𝐖\),𝐠surr⟩\+η2​β2​‖𝐠surr‖2<0\.\-\\eta\\langle\\nabla J\(\\mathbf\{W\}\),\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\rangle\+\\frac\{\\eta^\{2\}\\beta\}\{2\}\\\|\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\\|^\{2\}<0\.\(25\)Moving the inner product to the right side yields:

η⁡⟨∇J​\(𝐖\),𝐠surr⟩\>η2​β2​‖𝐠surr‖2\.\\eta\\langle\\nabla J\(\\mathbf\{W\}\),\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\rangle\>\\frac\{\\eta^\{2\}\\beta\}\{2\}\\\|\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\\|^\{2\}\.\(26\)We express the inner product using the cosine similarity between the two gradient vectors:⟨∇J​\(𝐖\),𝐠surr⟩=‖∇J​\(𝐖\)‖​‖𝐠surr‖​cos⁡θ\\langle\\nabla J\(\\mathbf\{W\}\),\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\rangle=\\\|\\nabla J\(\\mathbf\{W\}\)\\\|\\\|\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\\|\\cos\\theta\. Substituting this into the inequality:

η​‖∇J​\(𝐖\)‖​‖𝐠surr‖​cos⁡θ\>η2​β2​‖𝐠surr‖2\.\\eta\\\|\\nabla J\(\\mathbf\{W\}\)\\\|\\\|\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\\|\\cos\\theta\>\\frac\{\\eta^\{2\}\\beta\}\{2\}\\\|\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\\|^\{2\}\.\(27\)Since the learning rateη\>0\\eta\>0and assuming a non\-zero surrogate gradient \(‖𝐠surr‖\>0\\\|\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\\|\>0\), we divide both sides byη​‖∇J​\(𝐖\)‖​‖𝐠surr‖\\eta\\\|\\nabla J\(\\mathbf\{W\}\)\\\|\\\|\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\\|to obtain the exact necessary condition:

cos⁡θ\>η​β2⋅‖𝐠surr‖‖∇J​\(𝐖\)‖\.\\cos\\theta\>\\frac\{\\eta\\beta\}\{2\}\\cdot\\frac\{\\\|\\mathbf\{g\}\_\{\\mathrm\{surr\}\}\\\|\}\{\\\|\\nabla J\(\\mathbf\{W\}\)\\\|\}\.\(28\)This inequality follows directly from the standard descent lemma without further relaxations\. As empirically observed in Figure[3](https://arxiv.org/html/2609.00049#S6.F3), REAL\-Q’s full aggregated Fisher matrix yields highcos⁡θ\\cos\\theta, satisfying this condition under the learning rates used in our experiments\.□\\square

## Appendix DAdditional Experimental Details

### D\.1Full Setup \(continuation of Section[6\.1](https://arxiv.org/html/2609.00049#S6.SS1)\)

#### Optimizer\.

We use Adam withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,ϵ=10−8\\epsilon=10^\{\-8\}, with bias correction applied at each block step\. The learning rate depends on model and bit\-width and is reported in Table[6](https://arxiv.org/html/2609.00049#A4.T6); the final transformer block always uses a separate learning rate \(also reported in Table[6](https://arxiv.org/html/2609.00049#A4.T6)\) and optimizes the true KL loss against the LM head directly with Adam\. The learning rate is fixed across all column blocks within a transformer block\.

#### Reverse\-cosine layer\-wise learning\-rate schedule\.

The base learning rate of transformer blockii\(out ofLLtotal transformer blocks\) is set to

ηi=ηbase\+\(ηfinal−ηbase\)⋅sin⁡\(π2⋅i−1L−1\),\\eta\_\{i\}=\\eta\_\{\\text\{base\}\}\+\(\\eta\_\{\\text\{final\}\}\-\\eta\_\{\\text\{base\}\}\)\\cdot\\sin\\\!\\left\(\\frac\{\\pi\}\{2\}\\cdot\\frac\{i\-1\}\{L\-1\}\\right\),\(29\)which interpolates fromηbase\\eta\_\{\\text\{base\}\}at the first block up toηfinal\\eta\_\{\\text\{final\}\}at the last block\. The motivation is that early transformer blocks have not yet accumulated upstream quantization error and therefore require less aggressive correction; using a smaller learning rate at early blocks improves stability and prevents gradient explosions\. For activation\-aware quantization, this schedule is*disabled*\(constantηbase\\eta\_\{\\text\{base\}\}throughout\): once activation quantization is introduced, error accumulates from the very first block, and using a small early\-layer learning rate would let this error compound rather than be corrected\.

#### Activation clipping \(a\_loss\_clip\) for small Qwen3 models\.

On Qwen3\-0\.6B, Qwen3\-1\.7B, and Qwen3\-4B we additionally enable an output\-perturbation clipping \(a\_loss\_clip=0\.95\) when computing the Fisher MSE loss: per\-token output channels whoseΔ​𝐲\\Delta\\mathbf\{y\}magnitude exceeds the 95th percentile are scaled down via a detached scale factor \(so the loss is rescaled by the same factor and the gradient is not zeroed out\)\. This is necessary because smaller post\-trained Qwen3 models exhibit a small number of channels with very largeΔ​𝐲\\Delta\\mathbf\{y\}, which can dominate the Fisher MSE and let the Adam updates be hijacked by a few outlier directions\. Note this is distinct from the Hessian\-side “saliency clipping” described below\.

#### Hessian\-side saliency clipping\.

To prevent extreme gradient outliers \(which can be 10–12 orders of magnitude larger than the median in deep layers\) from collapsing the Hessian to a near rank\-1 matrix, we clip per\-token saliency values to the 99th percentile before computing the saliency\-weighted Hessian\.

#### Note on perplexity for Qwen3 models\.

On the post\-trained Qwen3 models we occasionally observe quantized perplexities slightly*below*the bf16 baseline\. This is expected: post\-training does not optimize perplexity, so the bf16 model is no longer at the PPL minimum, and any perturbation—including quantization—can move PPL in either direction\. KL divergence is the primary metric in all cases since it directly measures how far the quantized model’s output distribution has moved from the full\-precision reference\.

#### Pre\-computation of the aggregated Fisher and last\-transformer\-block handling\.

The aggregated Fisher𝐅\(ℓ\)\\mathbf\{F\}^\{\(\\ell\)\}for each transformer block is computed once via a single end\-to\-end backward pass over the calibration set before quantization begins, with on\-the\-fly outer\-product accumulation \(no per\-token storage;O⁡\(d2\)O\(d^\{2\}\)memory per block\)\. The final transformer block uses the true KL loss against the LM head \(no Fisher surrogate, since there are no downstream blocks to traverse\) with a separate learning rate \(reported in Table[6](https://arxiv.org/html/2609.00049#A4.T6)\), still using Adam\.

#### FSDP multi\-GPU pipeline for large models\.

For LLaMA\-3\.1\-70B the full\-precision model and per\-layer Fisher matrices do not fit in a single GPU\. We use an FSDP\-sharded\[[Zhao et al\., 2023](https://arxiv.org/html/2609.00049#bib.bib35)\]end\-to\-end backward across multiple GPUs \(Stage 0\) to compute and cache Fisher / saliency to disk; Stage 1 then loads the cache and performs the per\-layer quantization with Block\-GD on a sharded model\.

### D\.2Full W4A16 Results \(continuation of Section[6\.2](https://arxiv.org/html/2609.00049#S6.SS2)\)

Table[5](https://arxiv.org/html/2609.00049#A4.T5)reports the full seven\-model W4A16 sweep summarized in Section[6\.2](https://arxiv.org/html/2609.00049#S6.SS2)\. The LLaMA\-3\.1\-70B run uses 256 calibration samples \(rather than 2048\) for tractability of the 70B forward / backward; all other rows use 2048 samples\. REAL\-Q achieves the lowest KL on every model\.

Table 5:Full seven\-model W4A16 \(per\-row weight quantization\) results\. KL \(×10−2\\times 10^\{\-2\}\) and PPL are evaluated on WikiText\-2\. Downstream columns report zero\-shot accuracy \(%\) on ten tasks and their average\. bf16 KL is omitted \(the bf16 model is the reference\)\.
### D\.3REAL\-Q Learning Rates

Table[6](https://arxiv.org/html/2609.00049#A4.T6)reports the non\-final\-block learning\-rate hyperparameter and the final\-transformer\-block learning rate for every REAL\-Q run\. For rows using the reverse\-cosine schedule, the reported value isηfinal\\eta\_\{\\text\{final\}\}, and the first\-block learning rate is fixed to0\.01​ηfinal0\.01\\eta\_\{\\text\{final\}\}before layer\-wise interpolation\. For rows with the schedule disabled, the reported value is used as a constant learning rate throughout the non\-final transformer blocks\.

#### LR selection\.

The per\-model LRs in Table[6](https://arxiv.org/html/2609.00049#A4.T6)were obtained from a coarse log\-scale grid search over\{10−3,10−4,…,10−7\}\\\{10^\{\-3\},10^\{\-4\},\\ldots,10^\{\-7\}\\\}, conducted on a low\-cost proxy of the full pipeline that uses a larger column block size and fewer backward samples per gradient step\. We did not perform fine\-grained per\-model tuning; the LRs in Table[6](https://arxiv.org/html/2609.00049#A4.T6)are simply the best\-performing point on this loose grid\. Even with this coarse, low\-fidelity LR sweep, REAL\-Q already attains the substantial improvements over prior methods reported in Section[6](https://arxiv.org/html/2609.00049#S6), and we expect more careful per\-model LR selection to only further widen the margin\.

Table 6:REAL\-Q learning rates used across experimental settings\. The scheduled final LR isηfinal\\eta\_\{\\text\{final\}\}for reverse\-cosine rows and the constant non\-final\-block LR for rows with the schedule disabled\. The final\-layer LR is used only for the final transformer block, where REAL\-Q optimizes the true KL loss against the LM head\.

### D\.4Experimental Details for Loss Approximation Quality

This section provides the detailed setup for the gradient alignment evaluation presented in Section[6\.6](https://arxiv.org/html/2609.00049#S6.SS6)\(Figure[3](https://arxiv.org/html/2609.00049#S6.F3)\)\.

We evaluate on Qwen3\-0\.6B at RTN W4A16\. The model is processed transformer block by transformer block in the standard sequential quantization order\. Immediately after the internal linear modules of blockℓ\\ellare quantized, we freeze the current model state to compute the exact analytical gradients\. Specifically, for each linear module within blockℓ\\ell, we calculate:

1. \(i\)Thetrue end\-to\-end KL gradientwith respect to the module’s weights, obtained via a full backward pass through all subsequent layers and the language modeling head over the calibration mini\-batch\.
2. \(ii\)Thesurrogate gradientsproduced by the respective proxy losses:REAL\-Q Fisher MSE\(our aggregated full Fisher\),Saliency\-guided MSE\(the saliency\-weighted per\-module MSE optimized by GuidedQuant\), andModule MSE\(the uniform per\-module reconstruction MSE optimized by GPTQ / GPTAQ\)\.

We then compute the cosine similarity between the surrogate gradients and the true KL gradient\. As observed in the per\-module breakdown in the main text \(Figure[3](https://arxiv.org/html/2609.00049#S6.F3)\), while REAL\-Q maintains strong overall alignment, its fidelity degrades slightly on FFN modules \(particularlyup\_projanddown\_proj\) compared to attention modules\. This is primarily due to the SwiGLU non\-linearity, which makes the block\-wise truncated second\-order expansion a locally poorer approximation of the true downstream loss\. This architectural heterogeneity directly motivates Limitation \(1\) discussed in Appendix[E](https://arxiv.org/html/2609.00049#A5)\.

### D\.5GPU\-hour Overhead

All experiments were conducted using NVIDIA RTX Pro 6000 GPUs \(96GB\)\. Table[7](https://arxiv.org/html/2609.00049#A4.T7)reports the total end\-to\-end wall\-clock cost for applying REAL\-Q at W4A16\. This duration sums both the Stage\-0 full\-precision backward pass \(for Fisher and saliency pre\-computation\) and the Stage\-1 sequential quantization loop with Block\-GD\.

Table 7:RTX Pro 6000 end\-to\-end REAL\-Q W4A16 wall\-clock per model\. “\#GPUs” denotes the number of GPUs used for the run; “Hours per GPU” is the per\-GPU wall\-clock time; and “GPU\-hours” represents the total compute budget consumed by the entire REAL\-Q pipeline\.While the inclusion of dynamic gradient corrections inherently introduces additional computational overhead compared to purely static, closed\-form PTQ baselines like vanilla GPTQ, the dominant Stage\-1 cost scales cleanly with the number of \(transformer block×\\timeslinear module×\\timescolumn block\) triples\. Consequently, the overhead grows roughly linearly with model size at a fixed block sizeBB\. Most importantly, the total compute budget remains comfortably within the practical realm of offline PTQ—requiring only a few GPU\-hours for medium\-sized models—which is orders of magnitude more efficient than quantization\-aware training \(QAT\) or fine\-tuning approaches that demand multiple full epochs over the training data\.

### D\.6GPU Memory Footprint for Fisher Aggregation

While REAL\-Q preserves the full cross\-channel structure of the Fisher \(no row grouping\), caching the full empirical Fisher Information Matrix \(𝐅∈ℝd×d\\mathbf\{F\}\\in\\mathbb\{R\}^\{d\\times d\}\) for all transformer blocks requires additional memory footprint during the offline calibration stage\.

Since the Fisher matrix is structured per transformer block and decoupled from the context length, the total memory required to store the aggregated statistics for the entire model is deterministic\. Specifically, for a model withLLtransformer blocks and a hidden dimensiondd, the storage cost usingbfloat16precision \(2 bytes per parameter\) is exactly2×L×d22\\times L\\times d^\{2\}bytes\.

In Table[8](https://arxiv.org/html/2609.00049#A4.T8), we detail the theoretical memory footprint required to cache the full Fisher matrices for all seven models evaluated in our experiments\.

Table 8:GPU Memory usage for storing full Fisher Information Matrices across different models \(inbfloat16precision\)\. The total memory exactly scales with2×L×d22\\times L\\times d^\{2\}\.Model FamilyBlocks \(LL\)Hidden Dim \(dd\)Matrix Size per BlockTotal Memory CostQwen3\-0\.6B2810242\.002\.00MB∼56\\sim 56MBQwen3\-1\.7B2820488\.008\.00MB∼224\\sim 224MBQwen3\-4B36256012\.5012\.50MB∼450\\sim 450MBQwen3\-8B36409632\.0032\.00MB∼1\.13\\sim 1\.13GBQwen3\-32B64512050\.0050\.00MB∼3\.13\\sim 3\.13GBLLaMA\-3\.1\-8B32409632\.0032\.00MB∼1\.00\\sim 1\.00GBLLaMA\-3\.1\-70B808192128\.00128\.00MB∼10\.00\\sim 10\.00GB

Regarding the transient memory required for the end\-to\-end backward pass in Stage 0, we treat it strictly as a standard distributed fine\-tuning forward\-backward step\. By employing Fully Sharded Data Parallel \(FSDP\) combined with activation checkpointing across the calibration GPUs, the parameters, gradients, and optimizer states are sharded\. This makes the per\-GPU memory footprint for the calibration backward pass of massive models \(e\.g\., LLaMA\-3\.1\-70B\) entirely scalable and consistent with standard LLM fine\-tuning pipelines, requiring no exotic hardware beyond conventional multi\-GPU clusters\.

### D\.7Stability Across Random Seeds

REAL\-Q and all baselines are deterministic given a fixed calibration set, so the only stochastic source is the random sampling of WikiText\-2 sequences used to form the calibration set\. To assess the sensitivity of REAL\-Q to this randomness, we re\-run the full W4A16 pipeline on Qwen3\-0\.6B with five different calibration\-sampling seeds while holding all other settings \(model, bit\-width, hyperparameters in Table[6](https://arxiv.org/html/2609.00049#A4.T6), evaluation protocol\) fixed\. Seed 1 corresponds to the run reported in Tables[1](https://arxiv.org/html/2609.00049#S6.T1)and[5](https://arxiv.org/html/2609.00049#A4.T5)\.

Table 9:REAL\-Q W4A16 stability across five calibration\-sampling seeds on Qwen3\-0\.6B\. KL \(×10−2\\times 10^\{\-2\}\) and PPL are evaluated on WikiText\-2\. Seed 1 is the run reported in the main W4A16 tables\. The reported standard deviation is the \(sample\)1​σ1\\sigmaacross the five runs\.The standard deviation is roughly0\.7%0\.7\\%of the mean for KL and0\.2%0\.2\\%for PPL, indicating that the W4A16 result on Qwen3\-0\.6B is stable under calibration\-sample randomness\. Notably, even the worst of the five REAL\-Q seeds \(KL=6\.92=6\.92, PPL=21\.68=21\.68\) remains substantially below the strongest prior baseline on this model \(GuidedQuant: KL=8\.76=8\.76, PPL=22\.31=22\.31; cf\. Table[5](https://arxiv.org/html/2609.00049#A4.T5)\), so the headline ranking is preserved across all seeds\.

## Appendix EDetailed Limitations and Future Directions

The shift from static analytical solvers to dynamic gradient\-based corrections in REAL\-Q opens up several promising avenues for future research to further refine the methodology:

\(1\) Enhancing Surrogate Fidelity for Severe Non\-linearities\.As shown by the cosine alignment results in Appendix[D\.4](https://arxiv.org/html/2609.00049#A4.SS4), while our aggregated Fisher MSE provides accurate gradient guidance for attention modules, its correlation with the true end\-to\-end gradient degrades on certain FFN linear modules \(notablyup\_projanddown\_proj\)\. We attribute this to the SwiGLU non\-linearity, which makes a pure second\-order Taylor truncation a less faithful approximation of the downstream loss landscape\. Future work could explore higher\-order Taylor expansions or non\-linear proxy functions specifically designed to capture FFN activation dynamics\.

\(2\) Cross\-Module Optimizer State Transfer\.Currently, the Adam optimizer is re\-initialized for each new linear module being quantized\. Consequently, its first\- and second\-moment estimates have a very limited number of update steps to warm up before the module’s quantization is finalized\. Developing an optimizer with structured state\-transfer mechanisms across linear modules—or designing custom optimizers tailored for short, non\-stationary sequential trajectories—could significantly accelerate convergence and further improve REAL\-Q’s correction capabilities\.

\(3\) System\-Level Memory Optimizations\.The Stage\-0 pre\-computation inherently requires temporarily keeping the full\-precision model resident and storing per\-layer Fisher matrices \(O⁡\(d2\)O\(d^\{2\}\)each\)\. While we successfully mitigate this for 70B\-class models using an FSDP two\-stage pipeline, the peak memory overhead remains higher than that of vanilla, zero\-shot GPTQ\. We anticipate that this computational footprint can be systematically amortized in the future through custom memory\-efficient backward kernels, intermediate activation offloading, and fused Block\-GD operators, making the method even more accessible for consumer\-grade hardware\.