ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization
Summary
ReRound is a post-training quantization method that uses a conditional diffusion model to guide rounding of near-midpoint weights, improving small LLM accuracy at 3-bit and 4-bit quantization without inference overhead.
View Cached Full Text
Cached at: 08/13/26, 11:24 PM
Paper page - ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization
Source: https://huggingface.co/papers/2608.11045 Published on Aug 11
·
Submitted byhttps://huggingface.co/He-Yen
Hsiehon Aug 13
Abstract
ReRound uses a conditional diffusion model to guide rounding of near-midpoint weights during low-bit post-training quantization, selecting candidates by matching leading singular values to improve small LLM accuracy without inference overhead.
ReRound (Reconstructive Rounding) is apost-training quantizationmethod that addresses the midpoint ambiguity inherent in standardround-to-nearest(RTN) schemes when quantizing weights near the centers of quantization intervals. Starting from a pretrained LLM, ReRound trains aconditional diffusion modelto produce continuous reconstructions of low-bit weights for the LLM. These reconstructed weights act as a guidance signal to disambiguate the rounding direction of weights located close to interval midpoints. To integrate this reconstruction-guided rounding with conventional RTN, ReRound introduces atolerance metricmeasuring how far the quantized weight (not the final quantized integer) is away from the midpoint: quantized weights within a tolerance region around midpoints are quantized using diffusion-based reconstructions, whereas weights closer to quantization boundaries are quantized with RTN. By sweeping the tolerance parameter, ReRound generates multiple candidate quantized integer weight matrices and selects the de-quantized weight matrix candidate whose leadingsingular valuesmost closely match those of the original full-precision weights. This selected candidate determines the tolerance parameter ReRound uses. ReRound is particularly effective for smaller LLMs. Across a range of such models, it consistently outperforms standard RTN for 3-bit and 4-bit weight quantization. ReRound achieves superior accuracy compared to an extensive set of calibration-free methods, remains competitive with calibration-dependent approaches, and operates entirely offline, introducing no additional overhead during low-bit inference. The ReRound strategy represents a new approach forlow-bit quantization. The method applies to AI models beyond LLMs. This paper focuses on its applications to small LLMs.
View arXiv pageView PDFProject pageGitHub1Add to collection
Get this paper in your agent:
hf papers read 2608\.11045
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.11045 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.11045 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.11045 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Introduces Recurrent Residual Quantization (RRQ), a post-training quantization framework that enables multiple effective precisions from a single LLM checkpoint via additive residual refinement, improving flexibility and construction speed.
ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization
ReQuant introduces a backpropagation-free, fixed-grid discrete refinement stage for post-training quantization (PTQ) that iteratively improves initial quantized models while preserving the quantized format, showing consistent gains across various LLMs and bit-widths.
FAIR-Calib: Frontier-Aware Instability-Reweighted Calibration for Post-Training Quantization of Diffusion Large Language Models
This paper proposes FAIR-Calib, a two-stage post-training quantization framework for diffusion large language models that addresses the instability of token commitments during iterative refinement. It achieves state-of-the-art results on LLaDA and Dream models under low-bit quantization.
REAL-Q: E2E LLM Quantization via Dynamic Gradient Descent
The paper proposes REAL-Q, a novel end-to-end aligned post-training quantization method for large language models that uses dynamic gradient descent to mitigate quantization errors, achieving significant improvements in KL divergence reduction over state-of-the-art methods.
Trainable Smooth-Rotation Transforms with Learned Channel Scales for LLM Quantization
This paper proposes trainable smooth-rotation transforms with quantile-robust scaling and gradient-based optimization to improve post-training quantization of LLMs, achieving significant error reduction on LLaMA-3.2-1B under W4A4 quantization.