ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization

# ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization
Source: [https://arxiv.org/html/2608.07019](https://arxiv.org/html/2608.07019)
Yongge Ma1, Guoan Wang2, Feiyu Wang1, Yaoming Li1, Qian Zhang3, Zihan Yan4, Yinjun Han5, Tong Yang1 1School of Computer Science, Peking University 2School of Software and Microelectronics, Peking University 3School of Physics, Peking University 4The Chinese University of Hong Kong, Shenzhen 5Central Research Institute, ZTE Corporation

###### Abstract

Post\-training quantization \(PTQ\) is widely used to reduce the memory and computational cost of large language models\. Existing PTQ methods typically obtain an initial quantized model through heuristic rules or greedy optimization, and once quantization is completed the resulting integer assignments are usually treated as final\. This observation motivates a complementary optimization stage*within*PTQ that keeps quantized weights improvable after an executable quantized model has been produced, while preserving the quantized format\. We introduce ReQuant, a backpropagation\-free fixed\-grid refinement procedure for this stage\. Agnostic to the PTQ initializer, ReQuant takes an existing quantized model as a feasible starting point and iteratively revisits its discrete weight assignments on the fixed quantization grid\. Accepted updates strictly reduce the mean squared reconstruction error and remain on the original grid\. In this way, ReQuant turns the initially fixed PTQ output into an iteratively optimizable discrete solution and serves as a plug\-and\-play post\-processing stage for existing PTQ pipelines\. Experiments across diverse model families, bit\-widths, and downstream tasks show that ReQuant consistently improves quantized models from heterogeneous PTQ initializers, with especially large gains on simple initializers and lower bit\-widths\. Notably, ReQuant can refine a simple round\-to\-nearest initialization across multiple sweeps until it approaches or surpasses GPTAQ under the same quantization format\. These results establish ReQuant as a practical complementary stage for further improving existing PTQ pipelines\.

*Keywords*Post\-training quantization⋅\\cdotLarge language models⋅\\cdotModel quantization⋅\\cdotDiscrete optimization

## 1Introduction

Large language models \(LLMs\) based on the transformer architecture have achieved remarkable performance across a wide range of tasks\[[47](https://arxiv.org/html/2608.07019#bib.bib1),[7](https://arxiv.org/html/2608.07019#bib.bib2),[1](https://arxiv.org/html/2608.07019#bib.bib5),[33](https://arxiv.org/html/2608.07019#bib.bib6),[3](https://arxiv.org/html/2608.07019#bib.bib7)\]\. However, the rapidly growing parameter sizes of LLMs lead to significant deployment costs\[[45](https://arxiv.org/html/2608.07019#bib.bib4),[13](https://arxiv.org/html/2608.07019#bib.bib10)\]\. Modern LLMs often contain tens to hundreds of billions of parameters, with some mixture\-of\-experts models reaching the trillion\-parameter scale, requiring substantial memory for inference\[[10](https://arxiv.org/html/2608.07019#bib.bib3),[17](https://arxiv.org/html/2608.07019#bib.bib38),[16](https://arxiv.org/html/2608.07019#bib.bib39)\]\. During autoregressive decoding, model weights dominate memory consumption, making memory capacity a major bottleneck for inference\.

Quantization addresses this challenge by approximating full\-precision weights or activations with values from a discrete quantization grid, thereby reducing both storage and memory access costs\[[26](https://arxiv.org/html/2608.07019#bib.bib12),[9](https://arxiv.org/html/2608.07019#bib.bib28)\]\. As a result, quantization has become a key technique for improving deployment efficiency without modifying the model architecture\[[32](https://arxiv.org/html/2608.07019#bib.bib9),[49](https://arxiv.org/html/2608.07019#bib.bib8)\]\. Existing quantization methods can be broadly categorized into two paradigms: quantization\-aware training \(QAT\) and post\-training quantization \(PTQ\)\[[20](https://arxiv.org/html/2608.07019#bib.bib15)\]\. QAT simulates low\-precision arithmetic during training with fake\-quantization operations and back\-propagates through the non\-differentiable quantizer via the straight\-through estimator\[[26](https://arxiv.org/html/2608.07019#bib.bib12),[24](https://arxiv.org/html/2608.07019#bib.bib25),[5](https://arxiv.org/html/2608.07019#bib.bib29)\]\. While QAT often achieves the strongest low\-bit accuracy, it requires access to training data and repeated forward–backward optimization, making it prohibitively expensive at LLM scale\[[14](https://arxiv.org/html/2608.07019#bib.bib26),[34](https://arxiv.org/html/2608.07019#bib.bib13),[8](https://arxiv.org/html/2608.07019#bib.bib16),[27](https://arxiv.org/html/2608.07019#bib.bib27),[36](https://arxiv.org/html/2608.07019#bib.bib14)\]\.

In contrast, PTQ directly quantizes pretrained full\-precision models using only a small calibration set and no additional training, making it a practical alternative for large\-scale deployment\[[21](https://arxiv.org/html/2608.07019#bib.bib11),[32](https://arxiv.org/html/2608.07019#bib.bib9),[19](https://arxiv.org/html/2608.07019#bib.bib18),[31](https://arxiv.org/html/2608.07019#bib.bib17),[25](https://arxiv.org/html/2608.07019#bib.bib52)\]\. Existing PTQ methods mainly improve quantization through either reshaping the weight or activation distributions, or minimizing reconstruction error on a calibration dataset\. Distribution\-reshaping methods, such as SmoothQuant\[[49](https://arxiv.org/html/2608.07019#bib.bib8)\], QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\], and SpinQuant\[[35](https://arxiv.org/html/2608.07019#bib.bib24)\], apply scaling or rotation transformations to make weight or activation distributions more compatible with the quantization grid\. AWQ\[[32](https://arxiv.org/html/2608.07019#bib.bib9)\]instead uses activation\-aware scaling to protect salient weights from large quantization errors\. Reconstruction\-based methods, such as GPTQ\[[19](https://arxiv.org/html/2608.07019#bib.bib18)\]and GPTAQ\[[31](https://arxiv.org/html/2608.07019#bib.bib17)\], build on OBQ\-style second\-order approximations and perform greedy column\-wise quantization with error compensation\.

Existing PTQ methods differ in design, yet they typically treat the grid assignments obtained after quantization as final, as shown in Figure[1](https://arxiv.org/html/2608.07019#S3.F1)\. For reconstruction\-based methods such as GPTQ\[[19](https://arxiv.org/html/2608.07019#bib.bib18)\]and GPTAQ\[[31](https://arxiv.org/html/2608.07019#bib.bib17)\], once a column is mapped to the quantization grid, its discrete assignment is fixed, and subsequent steps compensate for the induced error through the remaining columns while leaving earlier decisions unchanged\. From a discrete optimization perspective, the quantized model produced by such methods is a feasible assignment on the fixed quantization grid and remains open to further improvement after quantization\.

This observation motivates us to revisit a complementary stage*within*PTQ: after an upstream method has already produced an executable quantized model, its discrete assignments can still be improved while preserving the deployment format\. We propose ReQuant, a backpropagation\-free fixed\-grid refinement procedure for this stage\. ReQuant is a post\-processing stage inside PTQ that takes a completed PTQ output as input, freezes bit\-width, scales, zero\-points, grid layout, and inference kernels, and directly revisits integer codes on that fixed grid\. Through iterative refinement, ReQuant improves quantized weights from heterogeneous PTQ initializers while preserving the deployed representation\.

Our main contributions are summarized as follows:

- •Method\.ReQuant formulates post\-quantization discrete refinement as a composable stage inside PTQ\. It treats an existing PTQ solution as a feasible initialization and optimizes its integer weight codes on the fixed quantization grid, while preserving the bit\-width, scale, zero\-point, grid layout, and deployment format\.
- •Analysis\.We show that ReQuant reduces the overall reconstruction loss and terminates after a finite number of accepted updates, with per\-sweep complexity comparable to a single GPTAQ pass\.
- •Empirical results\.Across Llama\-3 8B/70B and Qwen3\-14B under W4A16, W4A4, W3A4, and W2A4, ReQuant improves perplexity, KL divergence, and zero\-shot accuracy over four PTQ baselines across the reported settings, with especially large gains for simple initializers and lower bit\-widths; the offline budget is controllable through the number of sweepsTT\. Matched W4A16 pipeline comparisons with FlexRound\[[28](https://arxiv.org/html/2608.07019#bib.bib54)\]and a paired GPTQ±\\pmReQuant study further isolate refinement gains under a backpropagation\-free procedure, and a Qwen3\-235B MoE experiment demonstrates large\-scale feasibility\.

## 2Related Work

#### Post\-training quantization\.

Post\-training quantization \(PTQ\) maps pre\-trained weights, and optionally activations, to a discrete grid using only a small calibration set, without retraining\[[4](https://arxiv.org/html/2608.07019#bib.bib20),[44](https://arxiv.org/html/2608.07019#bib.bib21),[49](https://arxiv.org/html/2608.07019#bib.bib8)\]\. Existing PTQ methods mainly improve quantization by either reshaping weight or activation distributions to better align with the quantization grid, or minimizing reconstruction error to preserve the behavior of the full\-precision model\.

*Distribution\-reshaping*methods make weights or activations more amenable to low\-bit quantization by modifying their numerical distributions before quantization\. SmoothQuant\[[49](https://arxiv.org/html/2608.07019#bib.bib8)\]uses channel\-wise rescaling to migrate activation outliers into weights, thereby reducing activation quantization difficulty\. OmniQuant\[[44](https://arxiv.org/html/2608.07019#bib.bib21)\]learns lightweight affine transformations to reshape weight and activation distributions during calibration\. QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]and SpinQuant\[[35](https://arxiv.org/html/2608.07019#bib.bib24)\]apply orthogonal rotations to redistribute outliers across dimensions, producing representations that are more robust to low\-bit quantization\.

*Optimization\-driven*methods formulate PTQ as a reconstruction\-error minimization problem and solve it through local or greedy optimization procedures\. Building on OBQ\[[18](https://arxiv.org/html/2608.07019#bib.bib19)\], GPTQ\[[19](https://arxiv.org/html/2608.07019#bib.bib18)\]performs sequential column\-wise quantization under a second\-order approximation\. Once a column is mapped to the quantization grid, its assignment is fixed, while the resulting quantization error is compensated by closed\-form updates to the remaining unquantized columns\. GPTAQ\[[31](https://arxiv.org/html/2608.07019#bib.bib17)\]further accounts for the activation mismatch introduced by preceding quantized layers, using the activations actually produced by the quantized model to more accurately evaluate the reconstruction error\. In parallel, AWQ\[[32](https://arxiv.org/html/2608.07019#bib.bib9)\]takes a complementary activation\-aware perspective\. It identifies salient weight channels using activation statistics and applies channel\-wise rescaling to better preserve their contribution after quantization\.

#### Construction\-time continuous relaxation vs\. post\-deployment fixed\-grid refinement\.

A related but distinct line of work optimizes quantization decisions while*constructing*the quantized model from full\-precision weights\. AdaRound\[[39](https://arxiv.org/html/2608.07019#bib.bib51)\]learns continuous rounding variables through gradient\-based optimization; BRECQ\[[30](https://arxiv.org/html/2608.07019#bib.bib53)\]extends reconstruction\-based PTQ to block\-level settings; and FlexRound\[[28](https://arxiv.org/html/2608.07019#bib.bib54)\]jointly learns element\-wise division factors and a quantization\-grid scale via STE\-based backpropagation for Transformers\. AdaQuant\[[25](https://arxiv.org/html/2608.07019#bib.bib52)\]similarly optimizes layer\-level reconstruction with continuous parameters\. These methods introduce continuous surrogate variables, optimize a relaxed objective, and then discretize to obtain the final codes\. They share a reconstruction objective with ReQuant, but operate at a different pipeline stage and optimize different variables\.

ReQuant instead starts*after*an upstream PTQ method has already produced an executable quantized model\. It freezes the inherited bit\-width, scales, zero\-points, quantization grid, storage format, and inference kernels, and directly updates integer codes on that fixed grid with exact discrete loss\-change evaluations, remaining free of model backpropagation, STE, and optimizer states for learnable quantizer parameters\. Consequently, every intermediate solution remains deployable under the original format\. We therefore position ReQuant as a complementary, initializer\-agnostic refinement stage inside PTQ that can be composed with AdaRound/BRECQ/FlexRound\-style construction\-time optimization as well as with standard heuristic or greedy PTQ pipelines\.

## 3Method

![Refer to caption](https://arxiv.org/html/2608.07019v1/x1.png)Figure 1:Overview of ReQuant\.ReQuant refines an initial PTQ model by iteratively updating discrete weight assignments on the fixed quantization grid, improving reconstruction error while preserving the quantized format\.### 3\.1Preliminaries

For a linear layer with full\-precision weights𝐖∈ℝdrow×dcol\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{row\}\}\\times d\_\{\\mathrm\{col\}\}\}and calibration activations𝐗∈ℝdcol×m\\mathbf\{X\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{col\}\}\\times m\}, classical layer\-wise PTQ is commonly formulated as minimizing the reconstruction error

min⁡‖𝐖𝐗−𝐖q​𝐗‖F2\.\\min\\;\\\|\\mathbf\{W\}\\mathbf\{X\}\-\\mathbf\{W\}\_\{q\}\\mathbf\{X\}\\\|\_\{F\}^\{2\}\.\(1\)
However, at inference time, the input to a given layer is produced by preceding quantized layers rather than by the full\-precision model\. To model this activation mismatch, GPTAQ\[[31](https://arxiv.org/html/2608.07019#bib.bib17)\]replaces the full\-precision activations𝐗\\mathbf\{X\}in the quantized branch with the activations𝐗~\\widetilde\{\\mathbf\{X\}\}observed under the quantized prefix, yielding the activation\-aware objective

min⁡ℒ​\(𝐖q\)=‖𝐖𝐗−𝐖q​𝐗~‖F2\.\\min\\;\\mathcal\{L\}\(\\mathbf\{W\}\_\{q\}\)\\;=\\;\\\|\\mathbf\{W\}\\mathbf\{X\}\-\\mathbf\{W\}\_\{q\}\\widetilde\{\\mathbf\{X\}\}\\\|\_\{F\}^\{2\}\.\(2\)Both𝐗\\mathbf\{X\}and𝐗~\\widetilde\{\\mathbf\{X\}\}can be collected by forward passes on the calibration set, using the full\-precision and partially quantized models, respectively\.

#### Notation\.

For a matrix𝐖∈ℝdrow×dcol\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{row\}\}\\times d\_\{\\mathrm\{col\}\}\}, the first subscript indexes rows and the second indexes columns:𝐖i,:\\mathbf\{W\}\_\{i,:\}denotes itsii\-th row and𝐖:,j\\mathbf\{W\}\_\{:,j\}denotes itsjj\-th column\. For an ordered index setRR,𝐖i,R\\mathbf\{W\}\_\{i,R\}denotes the entries of rowiirestricted to the columns inRR, and𝐖R,R\\mathbf\{W\}\_\{R,R\}denotes the corresponding principal submatrix when the matrix is square\. The same convention applies to𝐇~:=𝐗~​𝐗~⊤\\widetilde\{\\mathbf\{H\}\}:=\\widetilde\{\\mathbf\{X\}\}\\widetilde\{\\mathbf\{X\}\}^\{\\top\}and other matrices appearing below\.

### 3\.2ReQuant Method

The framework of ReQuant is shown in Figure[1](https://arxiv.org/html/2608.07019#S3.F1)\. The core idea is that existing PTQ methods typically stop after a single quantization procedure and treat the resulting on\-grid assignment as final, whereas ReQuant uses this assignment as initialization and continues to optimize the discrete weight values on the same quantization grid\.

#### Optimization objective\.

After the quantization parameters \(e\.g\., scale and zero\-point\) are fixed, we consider the optimization problem on the induced discrete grid\. Our objective is to further optimize the quantized weights:

min𝐖q∈𝒢⁡ℒ​\(𝐖q\)=‖𝐖𝐗−𝐖q​𝐗~‖F2\.\\min\_\{\\mathbf\{W\}\_\{q\}\\in\\mathcal\{G\}\}\\mathcal\{L\}\(\\mathbf\{W\}\_\{q\}\)\\;=\\;\\\|\\mathbf\{W\}\\mathbf\{X\}\-\\mathbf\{W\}\_\{q\}\\widetilde\{\\mathbf\{X\}\}\\\|\_\{F\}^\{2\}\.\(3\)where𝒢\\mathcal\{G\}is the fixed quantization grid\. We refer toℒ​\(𝐖q\)\\mathcal\{L\}\(\\mathbf\{W\}\_\{q\}\)as the layer\-wise reconstruction objective\. In this work, we adopt Eq\.[3](https://arxiv.org/html/2608.07019#S3.E3)as the refinement objective and optimize the quantized weights directly over the discrete grid\.

#### Row\-wise decomposition\.

The key observation is that each output row of a linear layer is produced by the corresponding weight row independently\. In particular, theii\-th row of𝐖q\\mathbf\{W\}\_\{q\}only affects theii\-th row of the quantized output𝐖q​𝐗~\\mathbf\{W\}\_\{q\}\\widetilde\{\\mathbf\{X\}\}\. Therefore, the reconstruction loss decomposes as

‖𝐖𝐗−𝐖q​𝐗~‖F2=∑i=1drow‖𝐖i,:​𝐗−𝐖q,i,:​𝐗~‖22\.\\\|\\mathbf\{W\}\\mathbf\{X\}\-\\mathbf\{W\}\_\{q\}\\widetilde\{\\mathbf\{X\}\}\\\|\_\{F\}^\{2\}=\\sum\_\{i=1\}^\{d\_\{\\mathrm\{row\}\}\}\\\|\\mathbf\{W\}\_\{i,:\}\\mathbf\{X\}\-\\mathbf\{W\}\_\{q,i,:\}\\widetilde\{\\mathbf\{X\}\}\\\|\_\{2\}^\{2\}\.\(4\)
Updating a single coordinateWq,i,jW\_\{q,i,j\}affects only theii\-th term in Eq\.[4](https://arxiv.org/html/2608.07019#S3.E4); the losses of all other rows remain unchanged\. Thus, reducingLLfor any single row directly decreases the layer\-level lossℒ​\(𝐖q\)\\mathcal\{L\}\(\\mathbf\{W\}\_\{q\}\), and ReQuant can refine each row independently and in parallel\. In the rest of this section we derive the update rule for a single row and omit the row index for clarity\.

#### Row\-level reconstruction loss\.

Following the row\-wise decomposition above, we focus on a single row and omit the row index\. Let𝐰=𝐖i,:\\mathbf\{w\}=\\mathbf\{W\}\_\{i,:\}denote the full\-precision weight row,𝐪=𝐖q,i,:\\mathbf\{q\}=\\mathbf\{W\}\_\{q,i,:\}denote its current quantized counterpart, and𝐞=𝐰−𝐪\\mathbf\{e\}=\\mathbf\{w\}\-\\mathbf\{q\}denote the corresponding quantization error\. By Eq\.[4](https://arxiv.org/html/2608.07019#S3.E4), the contribution of this row toℒ​\(𝐖q\)\\mathcal\{L\}\(\\mathbf\{W\}\_\{q\}\)is

L​\(𝐞\)=‖𝐰𝐗−𝐪​𝐗~‖22=‖𝐞​𝐗~−𝐰​Δ​𝐗‖22,L\(\\mathbf\{e\}\)=\\\|\\mathbf\{w\}\\mathbf\{X\}\-\\mathbf\{q\}\\widetilde\{\\mathbf\{X\}\}\\\|\_\{2\}^\{2\}=\\\|\\mathbf\{e\}\\widetilde\{\\mathbf\{X\}\}\-\\mathbf\{w\}\\Delta\\mathbf\{X\}\\\|\_\{2\}^\{2\},\(5\)whereΔ​𝐗=𝐗~−𝐗\\Delta\\mathbf\{X\}=\\widetilde\{\\mathbf\{X\}\}\-\\mathbf\{X\}denotes the activation mismatch induced by earlier quantized layers\. Eq\.[5](https://arxiv.org/html/2608.07019#S3.E5)rewrites the row\-level loss as a function of the quantization error𝐞\\mathbf\{e\}, which is the form on which ReQuant operates\.

We define the gradient vector𝐠\\mathbf\{g\}as the gradient ofLLwith respect to𝐞\\mathbf\{e\}:

𝐠=∇𝐞L​\(𝐞\)=2​\(𝐞​𝐇~−𝐰𝐁\),\\mathbf\{g\}=\\nabla\_\{\\mathbf\{e\}\}L\(\\mathbf\{e\}\)=2\(\\mathbf\{e\}\\widetilde\{\\mathbf\{H\}\}\-\\mathbf\{w\}\\mathbf\{B\}\),\(6\)where𝐁=Δ​𝐗​𝐗~⊤\\mathbf\{B\}=\\Delta\\mathbf\{X\}\\widetilde\{\\mathbf\{X\}\}^\{\\top\}and𝐇~=𝐗~​𝐗~⊤⪰0\\widetilde\{\\mathbf\{H\}\}=\\widetilde\{\\mathbf\{X\}\}\\widetilde\{\\mathbf\{X\}\}^\{\\top\}\\succeq 0is the row\-level Hessian ofLLwith respect to𝐞\\mathbf\{e\}\. Consequently,L​\(𝐞\)L\(\\mathbf\{e\}\)is a convex quadratic function of𝐞\\mathbf\{e\}, and the curvature along coordinatejjisH~j​j\\widetilde\{H\}\_\{jj\}\. If the quantized value at coordinatejjis moved by an on\-grid stepΔ​qj\\Delta q\_\{j\}, then the quantization error becomes𝐞′=𝐞−Δ​qj​𝐮j\\mathbf\{e\}^\{\\prime\}=\\mathbf\{e\}\-\\Delta q\_\{j\}\\mathbf\{u\}\_\{j\}, and the loss change is

Δ​L​\(Δ​qj\)=L​\(𝐞′\)−L​\(𝐞\)=−Δ​qj​gj\+\(Δ​qj\)2​H~j​j\.\\Delta L\(\\Delta q\_\{j\}\)=L\(\\mathbf\{e\}^\{\\prime\}\)\-L\(\\mathbf\{e\}\)=\-\\Delta q\_\{j\}\\,g\_\{j\}\+\(\\Delta q\_\{j\}\)^\{2\}\\,\\widetilde\{H\}\_\{jj\}\.\(7\)The derivation is given in Appendix[A\.1](https://arxiv.org/html/2608.07019#A1.SS1)\. This computation reduces the cost of evaluating the reconstruction error\. Once the current row lossLLand vector𝐠\\mathbf\{g\}are computed, the effect of any subsequent coordinate update can be obtained directly from Eq\.[6](https://arxiv.org/html/2608.07019#S3.E6)and Eq\.[7](https://arxiv.org/html/2608.07019#S3.E7)\. Therefore, during refinement, ReQuant only needs to maintain the current row lossLLand vector𝐠\\mathbf\{g\}, which substantially reduces the cost of evaluating candidate updates\.

#### Coordinate update\.

Letsjs\_\{j\},ojo\_\{j\}, andzjz\_\{j\}denote the scale, zero\-point, and quantized integer code of coordinatejj, so thatqj=\(zj−oj\)​sjq\_\{j\}=\(z\_\{j\}\-o\_\{j\}\)s\_\{j\}\. Starting from the current integer codezjz\_\{j\}, ReQuant considers integer offsetskksuch that the updated codezj\+kz\_\{j\}\+kremains within the representable range\. Therefore, any candidate update has the formΔ​qj=k​sj\\Delta q\_\{j\}=ks\_\{j\}, and the feasible set at coordinatejjis

𝒢j=\{k​sj:k∈ℤ,k≠0,zmin≤zj\+k≤zmax\}\.\\mathcal\{G\}\_\{j\}=\\\{\\,ks\_\{j\}:k\\in\\mathbb\{Z\},\\;k\\neq 0,\\;z\_\{\\min\}\\leq z\_\{j\}\+k\\leq z\_\{\\max\}\\,\\\}\.\(8\)In practice we further restrict the search to a local neighborhood of sizeKK,

𝒢j\(K\)=\{k​sj∈𝒢j:1≤\|k\|≤K\},\\mathcal\{G\}\_\{j\}^\{\(K\)\}=\\\{\\,ks\_\{j\}\\in\\mathcal\{G\}\_\{j\}:1\\leq\|k\|\\leq K\\,\\\},and score candidates with Eq\.[7](https://arxiv.org/html/2608.07019#S3.E7), selecting

Δ​qj⋆=arg⁡minΔ​q∈𝒢j\(K\)⁡Δ​L​\(Δ​q\)\.\\Delta q\_\{j\}^\{\\star\}=\\arg\\min\_\{\\Delta q\\in\\mathcal\{G\}\_\{j\}^\{\(K\)\}\}\\Delta L\(\\Delta q\)\.\(9\)This step can be viewed as a discrete coordinate descent update\. With all other coordinates fixed, ReQuant moves coordinatejjto the best feasible grid point in theKK\-neighborhood\.

ReQuant accepts a move whenever it strictly decreases the row loss, i\.e\.,Δ​L​\(Δ​qj⋆\)<0\\Delta L\(\\Delta q\_\{j\}^\{\\star\}\)<0\. Upon acceptance, ReQuant updates coordinatejjand increments the row loss byL←L\+Δ​L​\(Δ​qj⋆\)L\\leftarrow L\+\\Delta L\(\\Delta q\_\{j\}^\{\\star\}\)\. The gradient vector is then refreshed incrementally as

𝐠←𝐠−2​Δ​qj⋆​𝐇~j,:\.\\mathbf\{g\}\\leftarrow\\mathbf\{g\}\-2\\Delta q\_\{j\}^\{\\star\}\\widetilde\{\\mathbf\{H\}\}\_\{j,:\}\.\(10\)This follows from𝐠=2​\(𝐞​𝐇~−𝐰𝐁\)\\mathbf\{g\}=2\(\\mathbf\{e\}\\widetilde\{\\mathbf\{H\}\}\-\\mathbf\{w\}\\mathbf\{B\}\), since an accepted move changes𝐞\\mathbf\{e\}only at coordinatejj\. Therefore,𝐠\\mathbf\{g\}can be updated inO​\(dcol\)O\(d\_\{\\mathrm\{col\}\}\)time using thejj\-th row of𝐇~\\widetilde\{\\mathbf\{H\}\}\.

Because each accepted update changes𝐞\\mathbf\{e\}and hence the score vector𝐠\\mathbf\{g\}, a coordinate that appears locally fixed in one pass can become improvable after other coordinates have changed\. Thus, a single greedy traversal overj=1,…,dcolj=1,\\ldots,d\_\{\\mathrm\{col\}\}leaves additional discrete improvements available for the coupled quadratic objectiveL​\(𝐞\)L\(\\mathbf\{e\}\)\. ReQuant therefore repeats the coordinate cycle for a fixed number of sweepsTT, as summarized in Algorithm[1](https://arxiv.org/html/2608.07019#alg1), providing a controllable compute–accuracy trade\-off\.

Algorithm 1Post\-Quantization Discrete Coordinate Refinement1:Input:full\-precision row

𝐰\\mathbf\{w\}, initial quantized row

𝐪\\mathbf\{q\}, grid information,

𝐇~\\widetilde\{\\mathbf\{H\}\},

𝐁\\mathbf\{B\}, neighborhood size

KK, sweeps

TT
2:

𝐞←𝐰−𝐪\\mathbf\{e\}\\leftarrow\\mathbf\{w\}\-\\mathbf\{q\},

𝐠←2​\(𝐞​𝐇~−𝐰𝐁\)\\mathbf\{g\}\\leftarrow 2\(\\mathbf\{e\}\\widetilde\{\\mathbf\{H\}\}\-\\mathbf\{w\}\\mathbf\{B\}\)
3:

L←‖𝐞​𝐗~−𝐰​Δ​𝐗‖22L\\leftarrow\\\|\\mathbf\{e\}\\widetilde\{\\mathbf\{X\}\}\-\\mathbf\{w\}\\Delta\\mathbf\{X\}\\\|\_\{2\}^\{2\}
4:for

t=1,…,Tt=1,\\dots,Tdo

5:for

j=1,…,dcolj=1,\\dots,d\_\{\\mathrm\{col\}\}do

6:

Δ​qj⋆←arg⁡minΔ​qj∈𝒢j\(K\)⁡Δ​L​\(Δ​qj\)\\Delta q\_\{j\}^\{\\star\}\\leftarrow\\arg\\min\_\{\\Delta q\_\{j\}\\in\\mathcal\{G\}\_\{j\}^\{\(K\)\}\}\\Delta L\(\\Delta q\_\{j\}\)
7:if

Δ​L​\(Δ​qj⋆\)<0\\Delta L\(\\Delta q\_\{j\}^\{\\star\}\)<0then

8:

qj←qj\+Δ​qj⋆q\_\{j\}\\leftarrow q\_\{j\}\+\\Delta q\_\{j\}^\{\\star\}
9:

𝐠←𝐠−2​Δ​qj⋆​𝐇~j,:\\mathbf\{g\}\\leftarrow\\mathbf\{g\}\-2\\Delta q\_\{j\}^\{\\star\}\\widetilde\{\\mathbf\{H\}\}\_\{j,:\}
10:

L←L\+Δ​L​\(Δ​qj⋆\)L\\leftarrow L\+\\Delta L\(\\Delta q\_\{j\}^\{\\star\}\)
11:endif

12:endfor

13:endfor

14:Return:refined quantized row

𝐪\\mathbf\{q\}

### 3\.3Analysis

#### Convergence\.

ReQuant performs discrete coordinate descent over a finite quantization grid, searching theKK\-neighborhood𝒢j\(K\)\\mathcal\{G\}\_\{j\}^\{\(K\)\}at each coordinate\. At each step, a candidate update is accepted wheneverΔ​L​\(Δ​qj⋆\)<0\\Delta L\(\\Delta q\_\{j\}^\{\\star\}\)<0\(Algorithm[1](https://arxiv.org/html/2608.07019#alg1)\)\. Hence, every accepted update strictly decreases the corresponding row\-level lossLL\. Since the layer objective decomposes asℒ​\(𝐖q\)=∑iL​\(𝐞i\)\\mathcal\{L\}\(\\mathbf\{W\}\_\{q\}\)=\\sum\_\{i\}L\(\\mathbf\{e\}\_\{i\}\)and each update affects a single row, it also strictly decreasesℒ​\(𝐖q\)\\mathcal\{L\}\(\\mathbf\{W\}\_\{q\}\)\. The feasible set is finite, and the objective strictly decreases after each accepted update, so each accepted move yields a distinct quantized state with strictly lower reconstruction loss\. If refinement continues until everyKK\-neighborhood move has been exhausted, the returned solution is a coordinate\-wise local optimum within that neighborhood; in practice a fixed budget ofTTsweeps already yields strong empirical improvements while keeping offline cost controllable\. A full proof of the infinite\-sweep case is provided in Appendix[A\.2](https://arxiv.org/html/2608.07019#A1.SS2)\.

As the calibration set increases, the empirical activation statistics in𝐇~\\widetilde\{\\mathbf\{H\}\}and𝐁\\mathbf\{B\}provide increasingly accurate estimates of their population counterparts\. As a result, the empirical refinement objective better approximates the population\-level objective, and the solution obtained by ReQuant becomes better aligned with the population\-level on\-grid optimum\.

#### Reconstruction loss versus downstream metrics\.

The analysis guarantees monotone decrease of the empirical layer\-wise objective along accepted updates, and neighborhood\-local optimality when theKK\-search is run to exhaustion\. Reducing layer reconstruction error is a standard proxy in PTQ\[[39](https://arxiv.org/html/2608.07019#bib.bib51),[30](https://arxiv.org/html/2608.07019#bib.bib53),[19](https://arxiv.org/html/2608.07019#bib.bib18)\]: if the remaining network is locally Lipschitz, smaller layer residuals tighten a bound on later hidden\-state perturbations\. Consistent with this proxy, our experiments show that reconstruction\-driven refinement typically improves held\-out PPL, KL, and zero\-shot accuracy, which we report as empirical outcomes of Algorithm[1](https://arxiv.org/html/2608.07019#alg1)\.

#### Efficiency\.

ReQuant evaluates candidate updates from cached statistics rather than recomputing the full reconstruction loss each time\. Using the row\-wise decomposition and the closed\-form coordinate score in Eq\.[7](https://arxiv.org/html/2608.07019#S3.E7), it scores on\-grid moves efficiently and refreshes the score vector after each accepted move with a rank\-one update\. Since output rows are independent, they can also be refined in parallel while sharing the same activation statistics\. For a layer withTTrefinement sweeps, the dominant cost isO​\(m​dcol2\+\(T\+1\)​drow​dcol2\)O\(md\_\{\\mathrm\{col\}\}^\{2\}\+\(T\+1\)d\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}^\{2\}\)under low\-bit quantization, where each coordinate has a small number of feasible moves\. Appendix[A\.3](https://arxiv.org/html/2608.07019#A1.SS3)provides the full complexity analysis\.

## 4Experiments

### 4\.1Experimental Setup

#### Overview\.

We organize our experiments as follows\. First, we use GPTAQ as the primary baseline and evaluate perplexity and downstream task performance on Qwen3\[[50](https://arxiv.org/html/2608.07019#bib.bib32)\]and Llama\-3\[[22](https://arxiv.org/html/2608.07019#bib.bib31)\]\. We then examine scale, including 8B, 70B, and a 235B MoE setting \(Section[4\.6](https://arxiv.org/html/2608.07019#S4.SS6)\)\. Finally, we evaluate generality across PTQ initializers \(RTN, AWQ\[[32](https://arxiv.org/html/2608.07019#bib.bib9)\], GPTQ\[[19](https://arxiv.org/html/2608.07019#bib.bib18)\], GPTAQ\[[31](https://arxiv.org/html/2608.07019#bib.bib17)\]\), cost–benefit viaTT, and comparisons with FlexRound\.

#### Implementation details\.

We implement ReQuant with Hugging Face Transformers\[[48](https://arxiv.org/html/2608.07019#bib.bib34)\]and PyTorch\[[41](https://arxiv.org/html/2608.07019#bib.bib33)\]\. We use WikiText\-2\[[37](https://arxiv.org/html/2608.07019#bib.bib35)\]as the calibration dataset and sample 512 sequences of length 2048 to collect activation statistics\. Unless otherwise specified, weights are quantized with per\-channel asymmetric quantization, and activations are quantized with per\-tensor asymmetric quantization\. Following GPTAQ\[[31](https://arxiv.org/html/2608.07019#bib.bib17)\], activation quantizers are calibrated and applied before weight quantization, so ReQuant optimizes weights under the activations produced by the quantized model\. By default, we set the number of ReQuant refinement sweeps toT=4T\{=\}4and evaluateK=2K\{=\}2neighboring grid moves in each direction for every coordinate\. Unless noted otherwise, main tables use data\-parallel calibration on eight NVIDIA GeForce RTX 4090 GPUs; Llama\-3 70B uses four NVIDIA B200 GPUs\. FlexRound comparisons \(Section[4\.5](https://arxiv.org/html/2608.07019#S4.SS5)\) are timed on an NVIDIA RTX PRO 6000 for memory\-matched fairness, and the Qwen3\-235B experiment \(Section[4\.6](https://arxiv.org/html/2608.07019#S4.SS6)\) uses eight NVIDIA H200 GPUs\.

#### Evaluation\.

We evaluate language modeling performance by reporting perplexity on WikiText\-2\[[37](https://arxiv.org/html/2608.07019#bib.bib35)\], UltraChat\-2k\[[15](https://arxiv.org/html/2608.07019#bib.bib49)\], and NuminaMath\[[29](https://arxiv.org/html/2608.07019#bib.bib50)\]\. For downstream evaluation, we report accuracy on ten standard commonsense reasoning and language understanding benchmarks, including ARC\-Challenge \(ARC\-C\), ARC\-Easy \(ARC\-E\), BoolQ, CEval, HellaSwag, LAMBADA, OpenBookQA \(OBQA\), PIQA, SocialIQA \(SIQA\), and Winogrande\[[12](https://arxiv.org/html/2608.07019#bib.bib40),[11](https://arxiv.org/html/2608.07019#bib.bib41),[23](https://arxiv.org/html/2608.07019#bib.bib42),[51](https://arxiv.org/html/2608.07019#bib.bib43),[40](https://arxiv.org/html/2608.07019#bib.bib44),[38](https://arxiv.org/html/2608.07019#bib.bib45),[6](https://arxiv.org/html/2608.07019#bib.bib46),[43](https://arxiv.org/html/2608.07019#bib.bib47),[42](https://arxiv.org/html/2608.07019#bib.bib48)\]\. We use the average accuracy across these tasks as the overall downstream score\.

### 4\.2Main results

Table 1:Zero\-shot accuracy \(%\) at W4A16 and W4A4; both settings use QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]rotation \(W4A16 without QuaRot is in Appendix[B\.6](https://arxiv.org/html/2608.07019#A2.SS6)\)\.Avg\.is the mean over ten tasks; the better entry of each pair is bolded\. Corresponding KL/PPL are reported in Table[7](https://arxiv.org/html/2608.07019#A2.T7)\(Appendix[B\.1](https://arxiv.org/html/2608.07019#A2.SS1)\) and Table[8](https://arxiv.org/html/2608.07019#A2.T8)\(Appendix[B\.2](https://arxiv.org/html/2608.07019#A2.SS2)\)\.#### Results on Llama\-3 8B and Qwen3\-14B\.

We evaluate ReQuant on Llama\-3 8B and Qwen3\-14B under W4A16 and W4A4 quantization\. We use four representative PTQ initializers: RTN\[[4](https://arxiv.org/html/2608.07019#bib.bib20)\], AWQ\[[32](https://arxiv.org/html/2608.07019#bib.bib9)\], GPTQ\[[19](https://arxiv.org/html/2608.07019#bib.bib18)\], and GPTAQ\[[31](https://arxiv.org/html/2608.07019#bib.bib17)\], covering both heuristic and greedy reconstruction\-based quantization methods\. In Table[1](https://arxiv.org/html/2608.07019#S4.T1), both W4A16 and W4A4 use QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]rotations; a W4A16 ablation without QuaRot is reported in Appendix[B\.6](https://arxiv.org/html/2608.07019#A2.SS6)\. ReQuant is applied only after the initial PTQ stage: the quantization grid, scale, zero\-point, and bit\-width remain fixed, and only the integer code at each coordinate is updated\. Table[1](https://arxiv.org/html/2608.07019#S4.T1)reports the average zero\-shot accuracy over ten downstream tasks across all model, precision, and initializer combinations\. We defer the corresponding KL divergence and perplexity \(PPL\) results on WikiText\-2, UltraChat\-2k, and NuminaMath to Appendix[B\.1](https://arxiv.org/html/2608.07019#A2.SS1)and Appendix[B\.2](https://arxiv.org/html/2608.07019#A2.SS2), report runtime measurements in Appendix[B\.4](https://arxiv.org/html/2608.07019#A2.SS4), and analyze how WikiText\-2 PPL, KL, and average zero\-shot accuracy vary with the calibration sample count for Llama\-3 8B at W4A4 in Appendix[B\.7](https://arxiv.org/html/2608.07019#A2.SS7)\.

Table[1](https://arxiv.org/html/2608.07019#S4.T1)shows that ReQuant improves average downstream accuracy across both precision settings, both model families, and all four PTQ initializers in the reported pairs\. The gains are larger for simpler initializers such as RTN and AWQ, indicating that many errors left by basic quantizers can still be corrected on the fixed grid\. For example, under W4A16 on Qwen3\-14B, ReQuant improves RTN by\+2\.51\+2\.51average accuracy points\. Under the more challenging W4A4 setting, where both weights and activations are quantized to 4 bits, the improvement becomes more pronounced: on Llama\-3 8B, ReQuant improves RTN by\+8\.61\+8\.61average accuracy points\.

ReQuant also improves stronger initializers such as GPTQ and GPTAQ, confirming that residual fixed\-grid assignment error remains exploitable even after advanced PTQ construction\. The offline budget remains controllable throughTT, so practitioners can allocate more sweeps when larger gains are desired\.

Notably, ReQuant substantially narrows the gap between simple and advanced PTQ methods\. Under W4A4, RTN refined by ReQuant nearly matches GPTAQ on Llama\-3 8B in average accuracy, and surpasses GPTAQ on Qwen3\-14B\. This suggests that a substantial share of the performance gap among PTQ methods comes from correctable discrete assignment errors that ReQuant can recover on the fixed grid\.

Table 2:Zero\-shot accuracy \(%\) on Llama\-3 70B at W4A4 with QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]rotation\.Avg\.is the mean over ten tasks; the better entry of each pair is bolded\. Corresponding KL/PPL are reported in Table[11](https://arxiv.org/html/2608.07019#A2.T11)of Appendix[B\.5](https://arxiv.org/html/2608.07019#A2.SS5)\.
#### Results on Llama\-3 70B at W4A4\.

We further evaluate ReQuant on Llama\-3 70B under the challenging W4A4 setting with QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]rotation\. ReQuant is applied to the same four PTQ initializers: RTN, AWQ, GPTQ, and GPTAQ\. Table[2](https://arxiv.org/html/2608.07019#S4.T2)reports zero\-shot accuracy across ten downstream tasks, and the corresponding KL divergence and perplexity results are provided in Table[11](https://arxiv.org/html/2608.07019#A2.T11)of Appendix[B\.5](https://arxiv.org/html/2608.07019#A2.SS5)\.

At the 70B scale, ReQuant improves all four initializers on average accuracy\. On GPTAQ, average accuracy rises from66\.20%66\.20\\%to67\.01%67\.01\\%\(\+0\.81\+0\.81\), with lower perplexity and KL as well, showing that even a strong activation\-aware initializer retains correctable fixed\-grid error\. Starting from RTN, ReQuant reaches66\.93%66\.93\\%average accuracy—close to GPTAQ\+ReQuant—and obtains better KL/PPL than GPTAQ\+ReQuant on the reported evaluation sets\. Thus fixed\-grid refinement can lift a simple initializer to a competitive final discrete assignment\.

### 4\.3Lower\-bit weight quantization

We further evaluate ReQuant under more aggressive settings, W3A4 and W2A4, on Llama\-3 8B with QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]rotation\. We focus on GPTQ and GPTAQ, the strongest reconstruction\-based initializers at these bit\-widths\. Table[3](https://arxiv.org/html/2608.07019#S4.T3)reports zero\-shot accuracy; the corresponding KL/PPL results are deferred to Appendix[B\.3](https://arxiv.org/html/2608.07019#A2.SS3)\. ReQuant improves both baselines on average accuracy, with larger gains at lower bit\-width\. At W3A4, ReQuant improves GPTQ from57\.1057\.10to58\.6158\.61\(\+1\.51\+1\.51\) and further improves GPTAQ from58\.6558\.65to58\.7158\.71\. At W2A4, the gains are much larger: GPTQ improves from35\.8835\.88to41\.0041\.00\(\+5\.12\+5\.12\), matching GPTAQ \(40\.9040\.90\), while GPTAQ itself rises to41\.9141\.91\. Across both settings, ReQuant narrows the GPTQ–GPTAQ gap and nearly closes it at W2A4, indicating that a large share of the very\-low\-bit gap is recoverable discrete assignment error on the fixed grid\.

Table 3:Zero\-shot accuracy \(%\) of ReQuant on Llama\-3 8B at W3A4 and W2A4 with QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]rotation\.Avg\.is the mean over ten tasks; the better entry of each pair is bolded\. Corresponding KL/PPL are in Table[9](https://arxiv.org/html/2608.07019#A2.T9)\(Appendix[B\.3](https://arxiv.org/html/2608.07019#A2.SS3)\)\.
### 4\.4Effect of refinement sweeps

![Refer to caption](https://arxiv.org/html/2608.07019v1/x2.png)

![Refer to caption](https://arxiv.org/html/2608.07019v1/x3.png)

![Refer to caption](https://arxiv.org/html/2608.07019v1/x4.png)

Figure 2:Effect of the number of ReQuant refinement sweepsTTon \(left\) WikiText\-2 perplexity, \(center\) WikiText\-2 KL divergence to the FP model, and \(right\) average zero\-shot accuracy on Llama\-3 8B\. Three settings are compared: RTN at W4A16 \(no QuaRot\), RTN with QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]rotation at W4A16, and GPTAQ with QuaRot rotation at W4A4\.We next study how ReQuant evolves with the number of refinement sweepsTT\. Figure[2](https://arxiv.org/html/2608.07019#S4.F2)reports WikiText\-2 perplexity, WikiText\-2 KL divergence to the full\-precision model, and average zero\-shot accuracy on Llama\-3 8B for three representative settings: RTN at W4A16 without QuaRot, RTN\+QuaRot at W4A16, and GPTAQ\+QuaRot at W4A4\. All other settings follow Section[4\.1](https://arxiv.org/html/2608.07019#S4.SS1)\. We varyT∈\{0,1,…,8\}T\\in\\\{0,1,\\ldots,8\\\}, whereT=0T=0denotes the initial PTQ output\. The PPL and KL plots also include full\-precision and W4A16 GPTAQ results as horizontal references\.

Three observations emerge from Figure[2](https://arxiv.org/html/2608.07019#S4.F2)\. First, as the number of refinement sweepsTTincreases, PPL and KL generally decrease, while average zero\-shot accuracy improves, consistent with the monotone decrease of the layer\-wise objective along accepted updates \(Section[3\.3](https://arxiv.org/html/2608.07019#S3.SS3.SSS0.Px1)\)\. Second, ReQuant benefits both simple and strong initializers\. For RTN at W4A16 without QuaRot, additional sweeps substantially improve average accuracy, bringing it close to the GPTAQ \+ QuaRot W4A16 reference value of65\.50%65\.50\\%on Llama\-3 8B\. For GPTAQ \+ QuaRot at W4A4 on the same model, refinement further raises the ten\-task mean from61\.97%61\.97\\%to62\.38%62\.38\\%\(Table[1](https://arxiv.org/html/2608.07019#S4.T1)\), showing that residual fixed\-grid error remains exploitable after a strong PTQ start\. Third, QuaRot\-based settings converge faster and achieve better final performance, suggesting that rotation makes fixed\-grid optimization more favorable\.

#### Cost–benefit ofTT\.

Table[4](https://arxiv.org/html/2608.07019#S4.T4)reports the accuracy–runtime trade\-off under W4A16 on Llama\-3 8B for RTN\+QuaRot and GPTQ\+QuaRot with neighborhood sizeK=2K\{=\}2\. On RTN,T=4T\{=\}4uses81\.8081\.80minutes of end\-to\-end offline wall\-clock; on GPTQ,T=4T\{=\}4uses94\.0594\.05minutes\. Most of the held\-out gain appears within the first one or two sweeps\. For RTN\+QuaRot,T=2T\{=\}2recovers a large fraction of theT=8T\{=\}8WikiText PPL/KL improvement at lower cost; for GPTQ\+QuaRot,T=2T\{=\}2already reaches the best average accuracy in this sweep \(65\.66%65\.66\\%\), and additional sweeps further reduce PPL/KL\.

Table 4:Cost–benefit of the number of refinement sweepsTTunder W4A16 on Llama\-3 8B \(K=2K\{=\}2\)\.T=0T\{=\}0is the initializer without ReQuant\. W2/UC/NM denote WikiText\-2, UltraChat\-2k, and NuminaMath\. Time is end\-to\-end offline quantization wall\-clock\. Within each setting, the best entry in each column is bolded\.

### 4\.5Comparison with FlexRound

To situate ReQuant relative to construction\-time adaptive rounding, we compare against FlexRound\[[28](https://arxiv.org/html/2608.07019#bib.bib54)\]under W4A16 with the same WikiText\-2 calibration budget \(512512sequences of length20482048\) and evaluation protocol\. All methods in Table[5](https://arxiv.org/html/2608.07019#S4.T5)were timed on the same NVIDIA RTX PRO 6000 for a hardware\-matched offline\-cost comparison\. FlexRound follows its original reconstruction pipeline; ReQuant pipelines use QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\], which is orthogonal to ReQuant and common in low\-bit Transformer PTQ\. We therefore treat Table[5](https://arxiv.org/html/2608.07019#S4.T5)as a*pipeline\-level*comparison in which metric gaps reflect the full stack \(initializer, rotation, and refinement\)\.

The paired GPTQ\+QuaRot control in Appendix[B\.8](https://arxiv.org/html/2608.07019#A2.SS8)freezes the initializer, grid, scales, and zero\-points and isolates the refinement stage: PPL/KL metrics improve and average accuracy rises from65\.16%65\.16\\%to65\.33%65\.33\\%\. Relative to plain FlexRound, GPTQ\+QuaRot\+ReQuant is stronger on both Llama\-2 7B\[[46](https://arxiv.org/html/2608.07019#bib.bib30)\]and Llama\-3 8B in Table[5](https://arxiv.org/html/2608.07019#S4.T5), with lower measured offline time\. On Llama\-3 8B, RTN\+QuaRot\+ReQuant attains the best average accuracy among the compared pipelines \(65\.42%65\.42\\%\), and GPTQ\+QuaRot\+ReQuant attains the best WikiText/UltraChat PPL–KL and NuminaMath PPL while remaining free of backpropagation and optimizer states\. These results support ReQuant as a practical backpropagation\-free refinement stage within a PTQ pipeline\.

Table 5:Pipeline comparison with FlexRound under W4A16 \(hardware\-matched timing on RTX PRO 6000\)\. ReQuant usesT=8T\{=\}8for Llama\-2 RTN andT=4T\{=\}4otherwise \(K=2K\{=\}2\)\. FlexRound\+QuaRot is reported for Llama\-3 8B\. W2/UC/NM denote WikiText\-2, UltraChat\-2k, and NuminaMath\. Within each model, the best entry in each column is bolded\.
### 4\.6Scalability to Qwen3\-235B MoE

Table[6](https://arxiv.org/html/2608.07019#S4.T6)evaluates W4 quantization of Qwen3\-235B\-A22B\[[50](https://arxiv.org/html/2608.07019#bib.bib32)\]\(235B total / 22B active parameters\) on eight NVIDIA H200 GPUs with expert\-parallel solving, QuaRot, and the same512×2048512\{\\times\}2048WikiText\-2 calibration protocol used elsewhere\. Relative to GPTQ\+QuaRot, four ReQuant sweeps reduce Top\-20 KL on WikiText\-2 / UltraChat / NuminaMath by13\.8%13\.8\\%/13\.1%13\.1\\%/5\.5%5\.5\\%and raise average accuracy from75\.00%75\.00\\%to75\.35%75\.35\\%\. RTN\+QuaRot\+ReQuant completes in321\.27321\.27minutes atT=4T\{=\}4, providing a faster offline path at this scale\. These results show that ReQuant remains effective beyond 100B parameters as a one\-time offline refinement stage\.

Table 6:Qwen3\-235B\-A22B W4 scalability evaluation \(8×\\timesH200\)\. W2/UC/NM denote WikiText\-2, UltraChat\-2k, and NuminaMath\. The best entry in each column is bolded\.

## 5Conclusion

We presented ReQuant, a fixed\-grid discrete refinement stage within PTQ that treats completed PTQ outputs as feasible initializations and continues optimizing their discrete assignments\. ReQuant iteratively revisits integer codes on the fixed quantization grid, accepts loss\-reducing updates, and preserves the original deployment format\. Across model families, bit\-widths, and PTQ initializers—including a Qwen3\-235B MoE setting—ReQuant improves quantized models, with especially large gains for simple initializers and lower bit\-widths; the offline budget is controllable throughTT, and refinement is performed entirely offline with zero impact on serving latency\.

#### Limitations\.

ReQuant performs coordinate\-wise local search on a fixed quantization grid inherited from an upstream PTQ method, so its solution quality depends on the calibration set and the frozen scales/zero\-points\. The offline refinement cost grows with the number of sweepsTTand is most noticeable relative to very fast initializers such as RTN; this cost is incurred once, leaves inference unchanged, and can be reduced by choosing a smallerTT\. Like other reconstruction\-based PTQ methods, ReQuant optimizes a layer\-wise calibration objective, and held\-out metrics are reported empirically\.

#### Future Work\.

Promising directions include more efficient search strategies, extensions to joint weight\-activation optimization, and stronger guarantees on approximate optimality and robustness under distribution shift\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)GPT\-4 Technical Report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p1.1)\.
- \[2\]S\. Ashkboos, A\. Mohtashami, M\. L\. Croci, B\. Li, P\. Cameron, M\. Jaggi, D\. Alistarh, T\. Hoefler, and J\. Hensman\(2024\)QuaRot: outlier\-free 4\-bit inference in rotated llms\.Advances in Neural Information Processing Systems \(NeurIPS\)37\.Cited by:[§B\.5](https://arxiv.org/html/2608.07019#A2.SS5.p1.1),[§B\.7](https://arxiv.org/html/2608.07019#A2.SS7.p1.1),[Table 11](https://arxiv.org/html/2608.07019#A2.T11),[Table 7](https://arxiv.org/html/2608.07019#A2.T7),[Table 8](https://arxiv.org/html/2608.07019#A2.T8),[Table 9](https://arxiv.org/html/2608.07019#A2.T9),[§1](https://arxiv.org/html/2608.07019#S1.p3.1),[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px1.p2.1),[Figure 2](https://arxiv.org/html/2608.07019#S4.F2),[§4\.2](https://arxiv.org/html/2608.07019#S4.SS2.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.07019#S4.SS2.SSS0.Px2.p1.1),[§4\.3](https://arxiv.org/html/2608.07019#S4.SS3.p1.10),[§4\.5](https://arxiv.org/html/2608.07019#S4.SS5.p1.2),[Table 1](https://arxiv.org/html/2608.07019#S4.T1),[Table 2](https://arxiv.org/html/2608.07019#S4.T2),[Table 3](https://arxiv.org/html/2608.07019#S4.T3)\.
- \[3\]J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.\(2023\)Qwen Technical Report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p1.1)\.
- \[4\]R\. Banner, Y\. Nahshan, and D\. Soudry\(2019\)Post\-training 4\-bit quantization of convolutional networks for rapid deployment\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.07019#S4.SS2.SSS0.Px1.p1.1)\.
- \[5\]Y\. Bengio, N\. Léonard, and A\. Courville\(2013\)Estimating or propagating gradients through stochastic neurons for conditional computation\.arXiv preprint arXiv:1308\.3432\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1)\.
- \[6\]Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. Choi\(2020\)PIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.
- \[7\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in Neural Information Processing Systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p1.1)\.
- \[8\]M\. Chen, W\. Shao, P\. Xu, J\. Wang, P\. Gao, K\. Zhang, and P\. Luo\(2025\)EfficientQAT: efficient quantization\-aware training for large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 10081–10100\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1)\.
- \[9\]J\. Choi, Z\. Wang, S\. Venkataramani, P\. I\. Chuang, V\. Srinivasan, and K\. Gopalakrishnan\(2018\)PACT: parameterized clipping activation for quantized neural networks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1)\.
- \[10\]A\. Chowdhery, S\. Narang, J\. Devlin, M\. Bosma, G\. Mishra, A\. Roberts, P\. Barham, H\. W\. Chung, C\. Sutton, S\. Gehrmann,et al\.\(2023\)PaLM: scaling language modeling with pathways\.Journal of Machine Learning Research24\(240\),pp\. 1–113\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p1.1)\.
- \[11\]C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova\(2019\)BoolQ: exploring the surprising difficulty of natural Yes/No questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 2924–2936\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.
- \[12\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.
- \[13\]T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer\(2022\)GPT3\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.Advances in Neural Information Processing Systems35,pp\. 30318–30332\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p1.1)\.
- \[14\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)Qlora: efficient finetuning of quantized llms\.Advances in neural information processing systems36,pp\. 10088–10115\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1)\.
- \[15\]N\. Ding, Y\. Chen, B\. Xu, Y\. Qin, S\. Hu, Z\. Liu, M\. Sun, and B\. Zhou\(2023\)Enhancing chat language models by scaling high\-quality instructional conversations\.arXiv preprint arXiv:2305\.14233\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.
- \[16\]N\. Du, Y\. Huang, A\. M\. Dai, S\. Tong, D\. Lepikhin, Y\. Xu, M\. Krikun, Y\. Zhou, A\. W\. Yu, O\. Firat, B\. Zoph, L\. Fedus, M\. P\. Bosma, Z\. Zhou, T\. Wang, E\. Wang, K\. Webster, M\. Pellat, K\. Robinson, K\. Meier\-Hellstern, T\. Duke, L\. Dixon, K\. Zhang, Q\. V\. Le, Y\. Wu, Z\. Chen, and C\. Cui\(2022\)GLaM: efficient scaling of language models with mixture\-of\-experts\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 5547–5569\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p1.1)\.
- \[17\]W\. Fedus, B\. Zoph, and N\. Shazeer\(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p1.1)\.
- \[18\]E\. Frantar and D\. Alistarh\(2022\)Optimal brain compression: a framework for accurate post\-training quantization and pruning\.Advances in Neural Information Processing Systems35,pp\. 4475–4488\.Cited by:[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px1.p3.1)\.
- \[19\]E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\(2022\)GPTQ: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p3.1),[§1](https://arxiv.org/html/2608.07019#S1.p4.1),[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px1.p3.1),[§3\.3](https://arxiv.org/html/2608.07019#S3.SS3.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.07019#S4.SS2.SSS0.Px1.p1.1)\.
- \[20\]A\. Gholami, S\. Kim, Z\. Dong, Z\. Yao, M\. W\. Mahoney, and K\. Keutzer\(2022\)A survey of quantization methods for efficient neural network inference\.InLow\-Power Computer Vision,pp\. 291–326\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1)\.
- \[21\]R\. Gong, Y\. Ding, Z\. Wang, C\. Lv, X\. Zheng, J\. Du, Y\. Yong, S\. Gu, H\. Qin, J\. Guo,et al\.\(2025\)A survey of low\-bit large language models: basics, systems, and algorithms\.Neural Networks,pp\. 107856\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p3.1)\.
- \[22\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px1.p1.1)\.
- \[23\]Y\. Huang, Y\. Bai, Z\. Zhu, J\. Zhang, J\. Zhang, T\. Su, J\. Liu, C\. Lv, Y\. Zhang, J\. Lei, Y\. Fu, M\. Sun, and J\. He\(2023\)C\-Eval: a multi\-level multi\-discipline chinese evaluation suite for foundation models\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 62991–63010\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.
- \[24\]I\. Hubara, M\. Courbariaux, D\. Soudry, R\. El\-Yaniv, and Y\. Bengio\(2016\)Binarized neural networks\.Advances in neural information processing systems29\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1)\.
- \[25\]I\. Hubara, Y\. Nahshan, Y\. Hanani, R\. Banner, and D\. Soudry\(2021\)Accurate post training quantization with small calibration sets\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 4466–4475\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p3.1),[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px2.p1.1)\.
- \[26\]B\. Jacob, S\. Kligys, B\. Chen, M\. Zhu, M\. Tang, A\. Howard, H\. Adam, and D\. Kalenichenko\(2018\)Quantization and training of neural networks for efficient integer\-arithmetic\-only inference\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 2704–2713\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1)\.
- \[27\]W\. Ke, Z\. Li, D\. Li, L\. Tian, and E\. Barsoum\(2024\)Dl\-qat: weight\-decomposed low\-rank quantization\-aware training for large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,pp\. 113–119\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1)\.
- \[28\]J\. H\. Lee, J\. Kim, S\. J\. Kwon, and D\. Lee\(2023\)FlexRound: learnable rounding based on element\-wise division for post\-training quantization\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 18913–18939\.Cited by:[3rd item](https://arxiv.org/html/2608.07019#S1.I1.i3.p1.2),[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px2.p1.1),[§4\.5](https://arxiv.org/html/2608.07019#S4.SS5.p1.2)\.
- \[29\]J\. Li, E\. Beeching, L\. Tunstall, B\. Lipkin, R\. Soletskyi, S\. C\. Huang, K\. Rasul, L\. Yu, A\. Jiang, Z\. Shen, Z\. Qin, B\. Dong, L\. Zhou, Y\. Fleureau, G\. Lample, and S\. Polu\(2024\)NuminaMath: a large\-scale math reasoning dataset\.Note:[https://huggingface\.co/datasets/AI\-MO/NuminaMath\-CoT](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT)Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.
- \[30\]Y\. Li, R\. Gong, X\. Tan, Y\. Yang, P\. Hu, Q\. Zhang, F\. Yu, W\. Wang, and S\. Gu\(2021\)BRECQ: pushing the limit of post\-training quantization by block reconstruction\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2608.07019#S3.SS3.SSS0.Px2.p1.1)\.
- \[31\]Y\. Li, R\. Yin, D\. Lee, S\. Xiao, and P\. Panda\(2025\)GPTAQ: efficient finetuning\-free quantization for asymmetric calibration\.arXiv preprint arXiv:2504\.02692\.Cited by:[§B\.7](https://arxiv.org/html/2608.07019#A2.SS7.p1.1),[§1](https://arxiv.org/html/2608.07019#S1.p3.1),[§1](https://arxiv.org/html/2608.07019#S1.p4.1),[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px1.p3.1),[§3\.1](https://arxiv.org/html/2608.07019#S3.SS1.p2.2),[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px2.p1.2),[§4\.2](https://arxiv.org/html/2608.07019#S4.SS2.SSS0.Px1.p1.1)\.
- \[32\]J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han\(2024\)AWQ: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of Machine Learning and Systems \(MLSys\)6,pp\. 87–100\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1),[§1](https://arxiv.org/html/2608.07019#S1.p3.1),[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px1.p3.1),[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.07019#S4.SS2.SSS0.Px1.p1.1)\.
- \[33\]A\. Liu, B\. Feng, B\. Wang, B\. Wang, B\. Liu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Guo,et al\.\(2024\)DeepSeek\-V2: a strong, economical, and efficient mixture\-of\-experts language model\.arXiv preprint arXiv:2405\.04434\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p1.1)\.
- \[34\]Z\. Liu, B\. Oguz, C\. Zhao, E\. Chang, P\. Stock, Y\. Mehdad, Y\. Shi, R\. Krishnamoorthi, and V\. Chandra\(2024\)LLM\-QAT: data\-free quantization\-aware training for large language models\.InFindings of the Association for Computational Linguistics: ACL,pp\. 467–484\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1)\.
- \[35\]Z\. Liu, C\. Zhao, I\. Fedorov, B\. Soran, D\. Choudhary, R\. Krishnamoorthi, V\. Chandra, Y\. Tian, and T\. Blankevoort\(2024\)SpinQuant: llm quantization with learned rotations\.arXiv preprint arXiv:2405\.16406\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p3.1),[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px1.p2.1)\.
- \[36\]V\. Malinovskii, D\. Mazur, I\. Ilin, D\. Kuznedelev, K\. Burlachenko, K\. Yi, D\. Alistarh, and P\. Richtárik\(2024\)PV\-Tuning: beyond straight\-through estimation for extreme llm compression\.Advances in Neural Information Processing Systems37,pp\. 5074–5121\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1)\.
- \[37\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\(2016\)Pointer sentinel mixture models\.arXiv preprint arXiv:1609\.07843\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px2.p1.2),[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.
- \[38\]T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal\(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 2381–2391\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.
- \[39\]M\. Nagel, R\. A\. Amjad, M\. Van Baalen, C\. Louizos, and T\. Blankevoort\(2020\)Up or down? adaptive rounding for post\-training quantization\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 7197–7206\.Cited by:[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2608.07019#S3.SS3.SSS0.Px2.p1.1)\.
- \[40\]D\. Paperno, G\. Kruszewski, A\. Lazaridou, N\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. Fernández\(2016\)The LAMBADA dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers,pp\. 1525–1534\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.
- \[41\]A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Kopf, E\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. Chintala\(2019\)PyTorch: an imperative style, high\-performance deep learning library\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.32\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px2.p1.2)\.
- \[42\]K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. Choi\(2021\)WinoGrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.
- \[43\]M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. Choi\(2019\)SocialIQA: commonsense reasoning about social interactions\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 4463–4473\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.
- \[44\]W\. Shao, M\. Chen, Z\. Zhang, P\. Xu, L\. Zhao, Z\. Li, K\. Zhang, P\. Gao, Y\. Qiao, and P\. Luo\(2023\)OmniQuant: omnidirectionally calibrated quantization for large language models\.arXiv preprint arXiv:2308\.13137\.Cited by:[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px1.p2.1)\.
- \[45\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)LLaMA: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p1.1)\.
- \[46\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§4\.5](https://arxiv.org/html/2608.07019#S4.SS5.p2.3)\.
- \[47\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in Neural Information Processing Systems30\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p1.1)\.
- \[48\]T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. Le Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. M\. Rush\(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp\. 38–45\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px2.p1.2)\.
- \[49\]G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han\(2023\)SmoothQuant: accurate and efficient post\-training quantization for large language models\.InInternational Conference on Machine Learning \(ICML\),pp\. 38087–38099\.Cited by:[§1](https://arxiv.org/html/2608.07019#S1.p2.1),[§1](https://arxiv.org/html/2608.07019#S1.p3.1),[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.07019#S2.SS0.SSS0.Px1.p2.1)\.
- \[50\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px1.p1.1),[§4\.6](https://arxiv.org/html/2608.07019#S4.SS6.p1.8)\.
- \[51\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.Cited by:[§4\.1](https://arxiv.org/html/2608.07019#S4.SS1.SSS0.Px3.p1.1)\.

## Appendix AReQuant derivation details

### A\.1ReQuant Coordinate Update Derivation

We derive the row\-wise quantities used by ReQuant and show why every candidate move can be scored from cached statistics\. For one output row, let𝐰\\mathbf\{w\}be the full\-precision row,𝐪\\mathbf\{q\}the current quantized row, and𝐞:=𝐰−𝐪\\mathbf\{e\}:=\\mathbf\{w\}\-\\mathbf\{q\}the quantization error\. WithΔ​𝐗:=𝐗~−𝐗\\Delta\\mathbf\{X\}:=\\widetilde\{\\mathbf\{X\}\}\-\\mathbf\{X\}, the activation\-aware row loss in Eq\.[5](https://arxiv.org/html/2608.07019#S3.E5)can be rewritten as

L​\(𝐞\)\\displaystyle L\(\\mathbf\{e\}\)=‖𝐰𝐗−𝐪​𝐗~‖22\\displaystyle=\\\|\\mathbf\{w\}\\mathbf\{X\}\-\\mathbf\{q\}\\widetilde\{\\mathbf\{X\}\}\\\|\_\{2\}^\{2\}=‖𝐰​\(𝐗−𝐗~\)\+\(𝐰−𝐪\)​𝐗~‖22\\displaystyle=\\\|\\mathbf\{w\}\(\\mathbf\{X\}\-\\widetilde\{\\mathbf\{X\}\}\)\+\(\\mathbf\{w\}\-\\mathbf\{q\}\)\\widetilde\{\\mathbf\{X\}\}\\\|\_\{2\}^\{2\}=‖𝐞​𝐗~−𝐰​Δ​𝐗‖22\.\\displaystyle=\\\|\\mathbf\{e\}\\widetilde\{\\mathbf\{X\}\}\-\\mathbf\{w\}\\Delta\\mathbf\{X\}\\\|\_\{2\}^\{2\}\.ThusLLis a quadratic function of𝐞\\mathbf\{e\}\. Defining

𝐇~:=𝐗~​𝐗~⊤,𝐁:=Δ​𝐗​𝐗~⊤,\\widetilde\{\\mathbf\{H\}\}:=\\widetilde\{\\mathbf\{X\}\}\\widetilde\{\\mathbf\{X\}\}^\{\\top\},\\qquad\\mathbf\{B\}:=\\Delta\\mathbf\{X\}\\widetilde\{\\mathbf\{X\}\}^\{\\top\},and expanding the squared norm gives

L​\(𝐞\)=𝐞​𝐇~​𝐞⊤−2​𝐞𝐁⊤​𝐰⊤\+𝐰​Δ​𝐗​Δ​𝐗⊤​𝐰⊤\.L\(\\mathbf\{e\}\)=\\mathbf\{e\}\\widetilde\{\\mathbf\{H\}\}\\mathbf\{e\}^\{\\top\}\-2\\mathbf\{e\}\\mathbf\{B\}^\{\\top\}\\mathbf\{w\}^\{\\top\}\+\\mathbf\{w\}\\Delta\\mathbf\{X\}\\Delta\\mathbf\{X\}^\{\\top\}\\mathbf\{w\}^\{\\top\}\.The last term is constant with respect to𝐞\\mathbf\{e\}, so the gradient vector used in the main text is

𝐠:=∇𝐞L​\(𝐞\)=2​\(𝐞​𝐇~−𝐰𝐁\)\.\\mathbf\{g\}:=\\nabla\_\{\\mathbf\{e\}\}L\(\\mathbf\{e\}\)=2\\bigl\(\\mathbf\{e\}\\widetilde\{\\mathbf\{H\}\}\-\\mathbf\{w\}\\mathbf\{B\}\\bigr\)\.
Now consider changing coordinatejjof the quantized row by an on\-grid stepΔ​qj\\Delta q\_\{j\}\. Since𝐞=𝐰−𝐪\\mathbf\{e\}=\\mathbf\{w\}\-\\mathbf\{q\}, increasingqjq\_\{j\}byΔ​qj\\Delta q\_\{j\}changes the error to

𝐞′=𝐞−Δ​qj​𝐮j,\\mathbf\{e\}^\{\\prime\}=\\mathbf\{e\}\-\\Delta q\_\{j\}\\mathbf\{u\}\_\{j\},where𝐮j\\mathbf\{u\}\_\{j\}is thejj\-th unit vector\. The exact loss change is

Δ​L​\(Δ​qj\)\\displaystyle\\Delta L\(\\Delta q\_\{j\}\):=L​\(𝐞′\)−L​\(𝐞\)\\displaystyle=L\(\\mathbf\{e\}^\{\\prime\}\)\-L\(\\mathbf\{e\}\)=‖\(𝐞−Δ​qj​𝐮j\)​𝐗~−𝐰​Δ​𝐗‖22−‖𝐞​𝐗~−𝐰​Δ​𝐗‖22\\displaystyle=\\\|\(\\mathbf\{e\}\-\\Delta q\_\{j\}\\mathbf\{u\}\_\{j\}\)\\widetilde\{\\mathbf\{X\}\}\-\\mathbf\{w\}\\Delta\\mathbf\{X\}\\\|\_\{2\}^\{2\}\-\\\|\\mathbf\{e\}\\widetilde\{\\mathbf\{X\}\}\-\\mathbf\{w\}\\Delta\\mathbf\{X\}\\\|\_\{2\}^\{2\}=−2​Δ​qj​𝐮j​𝐗~​\(𝐞​𝐗~−𝐰​Δ​𝐗\)⊤\+\(Δ​qj\)2​‖𝐮j​𝐗~‖22\\displaystyle=\-2\\Delta q\_\{j\}\\,\\mathbf\{u\}\_\{j\}\\widetilde\{\\mathbf\{X\}\}\(\\mathbf\{e\}\\widetilde\{\\mathbf\{X\}\}\-\\mathbf\{w\}\\Delta\\mathbf\{X\}\)^\{\\top\}\+\(\\Delta q\_\{j\}\)^\{2\}\\\|\\mathbf\{u\}\_\{j\}\\widetilde\{\\mathbf\{X\}\}\\\|\_\{2\}^\{2\}=−2​Δ​qj​\(𝐞​𝐇~−𝐰𝐁\)j\+\(Δ​qj\)2​H~j​j\\displaystyle=\-2\\Delta q\_\{j\}\\,\(\\mathbf\{e\}\\widetilde\{\\mathbf\{H\}\}\-\\mathbf\{w\}\\mathbf\{B\}\)\_\{j\}\+\(\\Delta q\_\{j\}\)^\{2\}\\widetilde\{H\}\_\{jj\}=−Δ​qj​gj\+\(Δ​qj\)2​H~j​j\.\\displaystyle=\-\\Delta q\_\{j\}\\,g\_\{j\}\+\(\\Delta q\_\{j\}\)^\{2\}\\widetilde\{H\}\_\{jj\}\.This proves Eq\.[7](https://arxiv.org/html/2608.07019#S3.E7)\. The key point is that evaluating a candidate move requires onlygjg\_\{j\}andH~j​j\\widetilde\{H\}\_\{jj\}, not a recomputation of the row residual\.

The feasible candidates are determined by the integer quantization code\. Ifqj=\(zj−oj\)​sjq\_\{j\}=\(z\_\{j\}\-o\_\{j\}\)s\_\{j\}, any valid move has the formΔ​qj=k​sj\\Delta q\_\{j\}=ks\_\{j\}, wherek∈ℤ∖\{0\}k\\in\\mathbb\{Z\}\\setminus\\\{0\\\}andzmin≤zj\+k≤zmaxz\_\{\\min\}\\leq z\_\{j\}\+k\\leq z\_\{\\max\}\. Therefore the best coordinate move is the finite enumeration in Eq\.[9](https://arxiv.org/html/2608.07019#S3.E9)\. After an accepted move, the state maintained by ReQuant is updated exactly\. The error update is immediate from𝐞′=𝐞−Δ​qj⋆​𝐮j\\mathbf\{e\}^\{\\prime\}=\\mathbf\{e\}\-\\Delta q\_\{j\}^\{\\star\}\\mathbf\{u\}\_\{j\}, and the gradient vector satisfies

𝐠′\\displaystyle\\mathbf\{g\}^\{\\prime\}=2​\(𝐞′​𝐇~−𝐰𝐁\)\\displaystyle=2\\bigl\(\\mathbf\{e\}^\{\\prime\}\\widetilde\{\\mathbf\{H\}\}\-\\mathbf\{w\}\\mathbf\{B\}\\bigr\)=2​\(\(𝐞−Δ​qj⋆​𝐮j\)​𝐇~−𝐰𝐁\)\\displaystyle=2\\bigl\(\(\\mathbf\{e\}\-\\Delta q\_\{j\}^\{\\star\}\\mathbf\{u\}\_\{j\}\)\\widetilde\{\\mathbf\{H\}\}\-\\mathbf\{w\}\\mathbf\{B\}\\bigr\)=𝐠−2​Δ​qj⋆​𝐇~j,:\.\\displaystyle=\\mathbf\{g\}\-2\\Delta q\_\{j\}^\{\\star\}\\widetilde\{\\mathbf\{H\}\}\_\{j,:\}\.Finally, because Eq\.[7](https://arxiv.org/html/2608.07019#S3.E7)gives the exact loss difference, the maintained scalar loss is updated byL←L\+Δ​L​\(Δ​qj⋆\)L\\leftarrow L\+\\Delta L\(\\Delta q\_\{j\}^\{\\star\}\)\. Together these identities justify the incremental updates used in Algorithm[1](https://arxiv.org/html/2608.07019#alg1)\.

### A\.2Finite Termination of ReQuant

We prove finite termination for one row under fixed calibration statistics and a fixed quantization grid; the layer\-wise result follows from the row decomposition in Eq\.[4](https://arxiv.org/html/2608.07019#S3.E4)\. Let the feasible values of coordinatejjbe the finite set𝒬j\\mathcal\{Q\}\_\{j\}\. Underbb\-bit quantization,\|𝒬j\|≤2b\|\\mathcal\{Q\}\_\{j\}\|\\leq 2^\{b\}\. The feasible set for one quantized row is therefore the Cartesian product

𝒬=𝒬1×𝒬2×⋯×𝒬dcol,\\mathcal\{Q\}=\\mathcal\{Q\}\_\{1\}\\times\\mathcal\{Q\}\_\{2\}\\times\\cdots\\times\\mathcal\{Q\}\_\{d\_\{\\mathrm\{col\}\}\},whose cardinality satisfies

\|𝒬\|=∏j=1dcol\|𝒬j\|≤\(2b\)dcol=2b​dcol\.\|\\mathcal\{Q\}\|=\\prod\_\{j=1\}^\{d\_\{\\mathrm\{col\}\}\}\|\\mathcal\{Q\}\_\{j\}\|\\leq\(2^\{b\}\)^\{d\_\{\\mathrm\{col\}\}\}=2^\{bd\_\{\\mathrm\{col\}\}\}\.Hence𝒬\\mathcal\{Q\}is finite\.

Let𝐪\(r\)∈𝒬\\mathbf\{q\}^\{\(r\)\}\\in\\mathcal\{Q\}denote the quantized row after therr\-th accepted update, and let

𝐞\(r\)=𝐰−𝐪\(r\)\\mathbf\{e\}^\{\(r\)\}=\\mathbf\{w\}\-\\mathbf\{q\}^\{\(r\)\}be the corresponding quantization error\. ReQuant accepts an update only if

ΔL\(r\):=L\(𝐞\(r\+1\)\)−L\(𝐞\(r\)\)<0\.\\Delta L^\{\(r\)\}:=L\(\\mathbf\{e\}^\{\(r\+1\)\}\)\-L\(\\mathbf\{e\}^\{\(r\)\}\)<0\.Therefore every accepted update strictly decreases the row objective:

L​\(𝐞\(0\)\)\>L​\(𝐞\(1\)\)\>L​\(𝐞\(2\)\)\>⋯\.L\(\\mathbf\{e\}^\{\(0\)\}\)\>L\(\\mathbf\{e\}^\{\(1\)\}\)\>L\(\\mathbf\{e\}^\{\(2\)\}\)\>\\cdots\.
We now prove by contradiction that infinitely many accepted updates are impossible\. Suppose ReQuant accepted infinitely many updates\. Then it would generate an infinite sequence

𝐪\(0\),𝐪\(1\),𝐪\(2\),…\\mathbf\{q\}^\{\(0\)\},\\mathbf\{q\}^\{\(1\)\},\\mathbf\{q\}^\{\(2\)\},\\ldotswith every𝐪\(r\)∈𝒬\\mathbf\{q\}^\{\(r\)\}\\in\\mathcal\{Q\}\. Since𝒬\\mathcal\{Q\}is finite, any infinite sequence in𝒬\\mathcal\{Q\}must repeat an element\. Hence there exist two indicesa<ba<bsuch that

𝐪\(a\)=𝐪\(b\)\.\\mathbf\{q\}^\{\(a\)\}=\\mathbf\{q\}^\{\(b\)\}\.Because𝐞=𝐰−𝐪\\mathbf\{e\}=\\mathbf\{w\}\-\\mathbf\{q\}, this equality also implies

𝐞\(a\)=𝐰−𝐪\(a\)=𝐰−𝐪\(b\)=𝐞\(b\),\\mathbf\{e\}^\{\(a\)\}=\\mathbf\{w\}\-\\mathbf\{q\}^\{\(a\)\}=\\mathbf\{w\}\-\\mathbf\{q\}^\{\(b\)\}=\\mathbf\{e\}^\{\(b\)\},and thus

L​\(𝐞\(a\)\)=L​\(𝐞\(b\)\)\.L\(\\mathbf\{e\}^\{\(a\)\}\)=L\(\\mathbf\{e\}^\{\(b\)\}\)\.However, the sequence contains at least one accepted update betweenaaandbb, and every accepted update strictly decreasesLL\. Hence

L​\(𝐞\(b\)\)<L​\(𝐞\(a\)\),L\(\\mathbf\{e\}^\{\(b\)\}\)<L\(\\mathbf\{e\}^\{\(a\)\}\),which contradictsL​\(𝐞\(a\)\)=L​\(𝐞\(b\)\)L\(\\mathbf\{e\}^\{\(a\)\}\)=L\(\\mathbf\{e\}^\{\(b\)\}\)\.

Thus ReQuant cannot accept infinitely many updates\. Since each accepted update visits a new quantized row, the number of accepted updates is at most

\|𝒬\|−1≤2b​dcol−1\.\|\\mathcal\{Q\}\|\-1\\leq 2^\{bd\_\{\\mathrm\{col\}\}\}\-1\.This bound is extremely loose, but it is sufficient to establish finite termination under fixed calibration statistics\.

### A\.3Complexity Analysis

We analyze the cost of ReQuant for one linear layer with𝐖∈ℝdrow×dcol\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{row\}\}\\times d\_\{\\mathrm\{col\}\}\}and𝐗,𝐗~∈ℝdcol×m\\mathbf\{X\},\\widetilde\{\\mathbf\{X\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{col\}\}\\times m\}, wheremmis the number of calibration tokens\. LetKKdenote the number of candidate grid moves considered per coordinate\. If all feasible nonzero moves are enumerated forbb\-bit quantization, thenK≤2b−1K\\leq 2^\{b\}\-1; if only nearby grid points are considered,KKis a small constant\.

#### Naive candidate evaluation\.

The most direct way to test whether changing one entryWq,i,jW\_\{q,i,j\}improves the objective is to recompute the full layer reconstruction loss‖𝐖𝐗−𝐖q​𝐗~‖F2\\\|\\mathbf\{W\}\\mathbf\{X\}\-\\mathbf\{W\}\_\{q\}\\widetilde\{\\mathbf\{X\}\}\\\|\_\{F\}^\{2\}\. This requires a matrix multiplication and costsO​\(m​drow​dcol\)O\(md\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}\)per candidate\. Since a layer containsdrow​dcold\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}coordinates and each coordinate hasKKcandidates, naive exhaustive evaluation would cost

O​\(K​m​drow2​dcol2\),O\(K\\,m\\,d\_\{\\mathrm\{row\}\}^\{2\}d\_\{\\mathrm\{col\}\}^\{2\}\),which is prohibitive for LLM layers\.

#### Effect of row\-wise decomposition\.

By Eq\.[4](https://arxiv.org/html/2608.07019#S3.E4), changingWq,i,jW\_\{q,i,j\}affects only theii\-th row loss

Li=‖𝐖i,:​𝐗−𝐖q,i,:​𝐗~‖22\.L\_\{i\}=\\\|\\mathbf\{W\}\_\{i,:\}\\mathbf\{X\}\-\\mathbf\{W\}\_\{q,i,:\}\\widetilde\{\\mathbf\{X\}\}\\\|\_\{2\}^\{2\}\.If this row loss were recomputed from scratch for each candidate, the cost would beO​\(m​dcol\)O\(md\_\{\\mathrm\{col\}\}\)per candidate, reducing the full\-layer factordrowd\_\{\\mathrm\{row\}\}and enabling all rows to be processed independently\. This is still expensive because it repeatedly accesses the calibration activations\.

#### Closed\-form candidate scoring\.

ReQuant instead uses the quadratic form in Eq\.[5](https://arxiv.org/html/2608.07019#S3.E5)\. Once𝐠=2​\(𝐞​𝐇~−𝐰𝐁\)\\mathbf\{g\}=2\(\\mathbf\{e\}\\widetilde\{\\mathbf\{H\}\}\-\\mathbf\{w\}\\mathbf\{B\}\)is available, Eq\.[7](https://arxiv.org/html/2608.07019#S3.E7)scores one candidate using onlyΔ​qj\\Delta q\_\{j\},gjg\_\{j\}, andH~j​j\\widetilde\{H\}\_\{jj\}, so each candidate costsO​\(1\)O\(1\)\. Scanning allKKcandidates at one coordinate costsO​\(K\)O\(K\); scanning all coordinates of all rows in one sweep costs

O​\(drow​dcol​K\)\.O\(d\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}K\)\.

#### Accepted\-update maintenance\.

When a move is accepted, ReQuant refreshes the gradient vector by𝐠←𝐠−2​Δ​qj⋆​𝐇~j,:\\mathbf\{g\}\\leftarrow\\mathbf\{g\}\-2\\Delta q\_\{j\}^\{\\star\}\\widetilde\{\\mathbf\{H\}\}\_\{j,:\}\. This costsO​\(dcol\)O\(d\_\{\\mathrm\{col\}\}\), since𝐇~j,:\\widetilde\{\\mathbf\{H\}\}\_\{j,:\}has lengthdcold\_\{\\mathrm\{col\}\}\. Recomputing𝐠=2​\(𝐞​𝐇~−𝐰𝐁\)\\mathbf\{g\}=2\(\\mathbf\{e\}\\widetilde\{\\mathbf\{H\}\}\-\\mathbf\{w\}\\mathbf\{B\}\)from scratch for one row would costO​\(dcol2\)O\(d\_\{\\mathrm\{col\}\}^\{2\}\)\. IfAAmoves are accepted in one sweep, the state\-maintenance cost is thereforeO​\(A​dcol\)O\(Ad\_\{\\mathrm\{col\}\}\)\. In the worst case,A≤drow​dcolA\\leq d\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}, givingO​\(drow​dcol2\)O\(d\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}^\{2\}\)per sweep for accepted\-update maintenance\.

#### Precomputation and initialization\.

Before refinement, ReQuant computes

𝐇~=𝐗~​𝐗~⊤,𝐁=Δ​𝐗​𝐗~⊤,\\widetilde\{\\mathbf\{H\}\}=\\widetilde\{\\mathbf\{X\}\}\\widetilde\{\\mathbf\{X\}\}^\{\\top\},\\qquad\\mathbf\{B\}=\\Delta\\mathbf\{X\}\\widetilde\{\\mathbf\{X\}\}^\{\\top\},which costsO​\(m​dcol2\)O\(md\_\{\\mathrm\{col\}\}^\{2\}\)and storesO​\(dcol2\)O\(d\_\{\\mathrm\{col\}\}^\{2\}\)statistics shared by all output rows\. Initializing the gradient matrix for all rows,

𝐆=2​\(𝐄​𝐇~−𝐖𝐁\),𝐄=𝐖−𝐐,\\mathbf\{G\}=2\(\\mathbf\{E\}\\widetilde\{\\mathbf\{H\}\}\-\\mathbf\{W\}\\mathbf\{B\}\),\\qquad\\mathbf\{E\}=\\mathbf\{W\}\-\\mathbf\{Q\},costsO​\(drow​dcol2\)O\(d\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}^\{2\}\)\. Initializing all row losses from residuals costsO​\(m​drow​dcol\)O\(md\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}\), which is usually dominated by the matrix\-statistic and gradient\-initialization terms\.

#### Total cost\.

ForTTrefinement sweeps and average accepted updatesAavgA\_\{\\mathrm\{avg\}\}per sweep, the layer\-wise ReQuant complexity is

O​\(m​dcol2\)\+O​\(drow​dcol2\)\+O​\(T​drow​dcol​K\)\+O​\(T​Aavg​dcol\)\.O\(md\_\{\\mathrm\{col\}\}^\{2\}\)\+O\(d\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}^\{2\}\)\+O\(Td\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}K\)\+O\(TA\_\{\\mathrm\{avg\}\}d\_\{\\mathrm\{col\}\}\)\.UsingAavg≤drow​dcolA\_\{\\mathrm\{avg\}\}\\leq d\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}, the worst\-case bound becomes

O​\(m​dcol2\)\+O​\(\(T\+1\)​drow​dcol2\)\+O​\(T​drow​dcol​K\)\.O\(md\_\{\\mathrm\{col\}\}^\{2\}\)\+O\(\(T\+1\)d\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}^\{2\}\)\+O\(Td\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}K\)\.For low\-bit quantizationK≤2b−1K\\leq 2^\{b\}\-1and typicallyK≪dcolK\\ll d\_\{\\mathrm\{col\}\}, so the dominant terms are

O​\(m​dcol2\+\(T\+1\)​drow​dcol2\)\.O\(md\_\{\\mathrm\{col\}\}^\{2\}\+\(T\+1\)d\_\{\\mathrm\{row\}\}d\_\{\\mathrm\{col\}\}^\{2\}\)\.

## Appendix BAdditional experimental results

This appendix collects supplementary experimental results that did not fit in the main text\. All*KL/PPL*tables below are evaluated on WikiText\-2, UltraChat\-2k, and NuminaMath; we abbreviate the three column groups asW2,UC, andNM, matching the headers\.KLis the \(unscaled\) KL divergence to the full\-precision model on each evaluation text\.PPLis standard*token\-level*perplexity on the same text\. Unless noted otherwise, we report PPL to two decimal places and KL to three\. Lower is better for both\. Unless noted otherwise,within each baseline pair\(each PTQ initializer vs\. the same initializer after ReQuant\),the better entry is bolded\.Section[B\.1](https://arxiv.org/html/2608.07019#A2.SS1)reports these metrics for Llama\-3 8B and Qwen3\-14B at W4A16\. Section[B\.2](https://arxiv.org/html/2608.07019#A2.SS2)reports them at W4A4 for the same models\. Section[B\.5](https://arxiv.org/html/2608.07019#A2.SS5)reports them on Llama\-3 70B at W4A4\. Section[B\.6](https://arxiv.org/html/2608.07019#A2.SS6)ablates QuaRot by re\-running the W4A16 experiments without any orthogonal preprocessing\. Section[B\.4](https://arxiv.org/html/2608.07019#A2.SS4)reports the runtime overhead on Qwen3\-14B\. Section[B\.3](https://arxiv.org/html/2608.07019#A2.SS3)reports KL/PPL for the W3A4/W2A4 setting in Section[4\.3](https://arxiv.org/html/2608.07019#S4.SS3)\. Section[B\.7](https://arxiv.org/html/2608.07019#A2.SS7)plots WikiText\-2 PPL/KL and average downstream accuracy as a function of calibration sample size for GPTAQ with QuaRot and ReQuant at W4A4 on Llama\-3 8B\. Sections[B\.8](https://arxiv.org/html/2608.07019#A2.SS8)–[B\.9](https://arxiv.org/html/2608.07019#A2.SS9)add FlexRound/QuaRot controls and search\-procedure ablations \(KK, coordinate order, seed\)\. The Qwen3\-235B MoE scalability study is reported in Section[4\.6](https://arxiv.org/html/2608.07019#S4.SS6)\.

### B\.1W4A16 results on Llama\-3 8B and Qwen3\-14B

At W4A16 under QuaRot and per\-weight–activation quantization, Table[7](https://arxiv.org/html/2608.07019#A2.T7)shows that ReQuant reduces both KL divergence and perplexity versus each initializer\-alone baseline across Llama\-3 8B and Qwen3\-14B and across W2/UC/NM\. The KL/PPL deltas are especially large for simple initial grids \(especially RTN, then AWQ\), and GPTAQ still exhibits consistent KL/PPL improvements in most bins\.

Table 7:KL and PPL under W4A16 quantization on Llama\-3 8B and Qwen3\-14B with QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]rotation\.
### B\.2W4A4 results on Llama\-3 8B and Qwen3\-14B

Under W4A4, Table[8](https://arxiv.org/html/2608.07019#A2.T8)records much larger raw KL and PPL than at W4A16 for the same models, reflecting the added difficulty of quantizing activations\. ReQuant still improves almost every baseline pair on every corpus, with the same qualitative pattern as in the weight\-only regime: naive initializers incur the largest distortions relative to FP, leaving the biggest room for refinement, whereas GPTAQ starts from comparatively low divergence and perplexity\.

Table 8:KL and PPL under W4A4 quantization on Llama\-3 8B and Qwen3\-14B with QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]rotation\.
### B\.3KL/PPL under W3A4 and W2A4 on Llama\-3 8B

Table[9](https://arxiv.org/html/2608.07019#A2.T9)reports WikiText\-2, UltraChat\-2k, and NuminaMath KL divergence and perplexity for the low\-bit setting in Section[4\.3](https://arxiv.org/html/2608.07019#S4.SS3)\. The corresponding zero\-shot accuracies appear in Table[3](https://arxiv.org/html/2608.07019#S4.T3)\.

Table 9:KL divergence and perplexity \(PPL\) on Llama\-3 8B at W3A4 and W2A4 with QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]rotation, evaluated on WikiText\-2 \(W2\), UltraChat\-2k \(UC\), and NuminaMath \(NM\)\.
### B\.4Runtime results on Qwen3\-14B

Table[10](https://arxiv.org/html/2608.07019#A2.T10)reports the end\-to\-end quantization time on Qwen3\-14B for different PTQ initializers with QuaRot, both with and without ReQuant\. For example, RTN\+QuaRot uses15\.615\.6minutes and RTN\+QuaRot\+ReQuant uses117\.1117\.1minutes; GPTAQ\+QuaRot uses35\.335\.3minutes and GPTAQ\+QuaRot\+ReQuant uses128\.6128\.6minutes\. The offline budget is controllable throughTT\(Section[4\.4](https://arxiv.org/html/2608.07019#S4.SS4)\)\.

This cost is incurred once during offline model preparation and leaves the deployed representation and inference latency unchanged\. In our implementation the overhead is mainly computational, because ReQuant stores neither gradients nor optimizer states for learnable quantizer parameters\.

Table 10:Runtime on Qwen3\-14B\.
### B\.5KL divergence and perplexity results on Llama\-3 70B

Table[11](https://arxiv.org/html/2608.07019#A2.T11)focuses on Llama\-3 70B at W4A4 under QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]\. RTN alone exhibits extreme KL/PPL on W2 relative to FP, and ReQuant closes most of that gap while also improving AWQ\. For GPTQ and GPTAQ the initial mismatch is already smaller; ReQuant still tends to help on perplexity and many KL cells, though a few entries favor the unrefined baseline, highlighting that fixed\-grid moves can redistribute error across corpora at large scale\.

Table 11:KL and PPL under W4A4 quantization on Llama\-3 70B with QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\]rotation\.
### B\.6W4A16 results without QuaRot rotation

Tables[12](https://arxiv.org/html/2608.07019#A2.T12)and[13](https://arxiv.org/html/2608.07019#A2.T13)repeat the Llama\-3 8B W4A16 study*without*QuaRot orthogonal preprocessing\. On KL/PPL, ReQuant tends to lower both metrics relative to each initializer alone across W2/UC/NM, matching the qualitative pattern under rotation but with a few small reversals on individual cells\. On downstream accuracy \(mean over ten tasks\), ReQuant improves RTN \(61\.65→63\.2061\.65\\rightarrow 63\.20,\+1\.55\+1\.55\), AWQ \(63\.50→64\.0663\.50\\rightarrow 64\.06\), and GPTAQ \(64\.28→64\.6864\.28\\rightarrow 64\.68\), while for GPTQ the average drops slightly \(64\.63→64\.4164\.63\\rightarrow 64\.41\) even though its WikiText\-2 KL/PPL still improve—illustrating that fixed\-grid refinement can shift the accuracy–perplexity trade\-off when rotation is absent\.

Table 12:KL and PPL under W4A16 on Llama\-3 8B*without*QuaRot rotation\.Table 13:Zero\-shot accuracy \(%\) under W4A16 on Llama\-3 8B*without*QuaRot rotation\.Avg\.is the mean over ten tasks\.
### B\.7Effect of calibration sample size on Llama\-3 8B at W4A4 \(GPTAQ \+ QuaRot \+ ReQuant\)

Figure[3](https://arxiv.org/html/2608.07019#A2.F3)studies the effect of calibration\-set size atW4A4on Llama\-3 8B, with all other settings following Section[4\.1](https://arxiv.org/html/2608.07019#S4.SS1)\. We report WikiText\-2 perplexity, WikiText\-2 KL divergence to the full\-precision model, and average zero\-shot accuracy from lm\-eval\-harness for GPTAQ\[[31](https://arxiv.org/html/2608.07019#bib.bib17)\]with QuaRot\[[2](https://arxiv.org/html/2608.07019#bib.bib23)\], both before and after ReQuant refinement\. Increasing the number of calibration sequences generally improves all three metrics by providing more stable activation statistics, although the gains diminish once the calibration budget becomes sufficiently large\.

Across all calibration\-sample counts shown, ReQuant improves over GPTAQ\+QuaRot under the same W4A4 deployment format: WikiText\-2 perplexity and KL decrease while average zero\-shot accuracy increases\. The gains therefore appear across calibration budgets, not only under a particular sample count\.

![Refer to caption](https://arxiv.org/html/2608.07019v1/x5.png)

![Refer to caption](https://arxiv.org/html/2608.07019v1/x6.png)

![Refer to caption](https://arxiv.org/html/2608.07019v1/x7.png)

Figure 3:W4A4: sensitivity to calibration sample size for GPTAQ \+ QuaRot \+ ReQuant on Llama\-3 8B\. \(left\) WikiText\-2 perplexity, \(center\) WikiText\-2 KL divergence to the FP model, \(right\) average zero\-shot accuracy \(ten\-task mean\)\. Lower is better for PPL and KL; higher is better for accuracy\.
### B\.8Separating QuaRot from FlexRound comparisons

Table[14](https://arxiv.org/html/2608.07019#A2.T14)reports controlled add\-ons around the FlexRound comparison in Section[4\.5](https://arxiv.org/html/2608.07019#S4.SS5)on Llama\-3 8B under W4A16\. Adding QuaRot to FlexRound improves PPL/KL and average accuracy, confirming that rotation is a beneficial and orthogonal pipeline component\. Pairing QuaRot with ReQuant further strengthens the RTN pipeline, and the paired GPTQ\+QuaRot vs\. GPTQ\+QuaRot\+ReQuant comparison isolates ReQuant under identical initialization, grid, scales, and zero\-points: five of six PPL/KL metrics improve and average accuracy rises from65\.16%65\.16\\%to65\.33%65\.33\\%\.

Table 14:Controlled W4A16 comparisons on Llama\-3 8B that separate QuaRot from ReQuant\. GPTQ rows reuse Table[4](https://arxiv.org/html/2608.07019#S4.T4)\.
### B\.9Search\-procedure ablations

Unless noted, experiments below use Llama\-3 8B, W4A16, GPTQ\+QuaRot\+ReQuant,T=4T\{=\}4, and seed0\.

#### Neighborhood sizeKK\.

Candidate integer offsets are\{±1,…,±K\}\\\{\\pm 1,\\ldots,\\pm K\\\}\. Table[15](https://arxiv.org/html/2608.07019#A2.T15)shows thatK∈\{1,2,3\}K\{\\in\}\\\{1,2,3\\\}yields similar quality and runtime\.K=2K\{=\}2is strongest on WikiText PPL/KL for GPTQ and yields the highest RTN average accuracy, whileK=1K\{=\}1is competitive on most PPL/KL metrics for RTN; we therefore useK=2K\{=\}2as the default\.

Table 15:Neighborhood\-size ablation at fixedT=4T\{=\}4on Llama\-3 8B \(W4A16\)\.
#### Coordinate ordering\.

Table[16](https://arxiv.org/html/2608.07019#A2.T16)compares forward \(j=1​…​dcolj\{=\}1\{\\ldots\}d\_\{\\mathrm\{col\}\}\), reverse, and a random\-fixed within\-row permutation reused across sweeps\. WikiText PPL differs by at most0\.00760\.0076and average accuracy by0\.370\.37points\. Ordering changes the local solution while preserving overall quality; random\-fixed is slower due to permutation overhead\.

Table 16:Within\-row coordinate\-order ablation for GPTQ\+QuaRot\+ReQuant \(T=4T\{=\}4,K=2K\{=\}2\) on Llama\-3 8B\.
#### QuaRot seed stability\.

Given fixed calibration data, ReQuant itself is deterministic; randomness enters through QuaRot\. Over five QuaRot seeds with GPTQ\+QuaRot\+ReQuant \(T=4T\{=\}4,K=2K\{=\}2\), WikiText PPL is6\.4277±0\.00406\.4277\{\\pm\}0\.0040, WikiText KL is0\.0414±0\.00040\.0414\{\\pm\}0\.0004, and average accuracy is65\.27±0\.5565\.27\{\\pm\}0\.55\(range64\.6864\.68–66\.0966\.09\)\. PPL/KL are stable across seeds, while downstream accuracy exhibits moderate seed variability\.