Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs
Summary
Quantization-Aware Healing is a method that recovers compressed 4-bit language models by distilling directly from the original uncompressed model, offering faster and more stable performance than Quantization-Aware Training.
View Cached Full Text
Cached at: 08/25/26, 12:35 PM
Paper page - Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs
Source: https://huggingface.co/papers/2608.20953
Abstract
Quantization-aware healing recovers compressed 4-bit language models faster and more stably than quantization-aware training by distilling directly from the original uncompressed model.
Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and quantized to 4 bits. Together these steps degrade reasoning, mathematics, coding, and long-context behavior enough to require a recovery, or healing, stage before deployment. The default recipe,quantization-aware training(QAT), re-fits the compressed, quantized model to hard labels; in our pipeline it converged slowly and collapsed past its peak. We adoptedQuantization-Aware Healing(QAH) instead. Because a structurally compressed model is never independently trained at full precision, itsbfloat16checkpoint is adistillation-recovered approximation of the original; QAH distills the 4-bit student directly from the original, uncompressed model. On aGPT-OSS120B to 60B toMXFP4pipeline, the QAH student matches or beats itsbfloat16source on 7 of 9 benchmarks at roughly 4 times less weight memory and half the teacher’s parameter count, and is released open-weight asHypernova-60B. Against a matched QAT baseline it reaches a comparable peak about 7 times faster and stays stable under continued training, without hand-tuned early stopping. We also report deployment lessons, including a large, reproducible quality gap between distributed-training backends. Our aim is a recipe deployable without a multi-week hyper-parameter search.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.20953
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.20953 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.20953 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.20953 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data
Researchers from AMD propose Recover-LoRA, a method that uses low-rank adaptation with knowledge distillation on synthetic data to recover accuracy lost from aggressive 2-bit quantization of LLMs, achieving 80–95% accuracy recovery on 9 of 12 benchmarks for Qwen3-4B using only 10k synthetic samples.
QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction
QUASAR is a quantization-aware training method that uses loss-aware reconstruction to lower the loss floor, improving low-bit model performance in large language models with significant accuracy gains at 2-4 bits.
From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization
Researchers identify two distinct failure modes in aggressive LLM quantization—Signal Degradation and Computation Collapse—and show that training-free fixes only remedy the former, indicating structural reconstruction is needed for ultra-low-bit models.
Language-Conditional Dequantization: Recovering What Quantization Steals from Non-English Languages
This paper introduces Language-Conditional Dequantization (LCD), a post-hoc method that adds per-language low-rank LoRA corrections to quantized LLMs to recover multilingual performance lost during English-calibrated quantization, showing significant perplexity and accuracy improvements on non-English languages.
MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
MixQuant proposes an adaptive mixed-precision quantization framework for LLMs that handles variable memory budgets by marginalizing layer distortion over random upstream configurations, outperforming existing methods across multiple models and budgets.