TempQuant: Learning Adaptive Temperatures for Quantized Language Models
Summary
TempQuant 提出自适应学习量化语言模型的温度与 top-p 设置,通过校准使量化模型的下一词元概率分布更接近全精度模型,在 3-bit Qwen3-4B 上将 MATH 准确率从 34% 提升至 51%。
View Cached Full Text
Cached at: 11/19/26, 03:36 AM
# 1Introduction
Source: [https://arxiv.org/html/2611.22128](https://arxiv.org/html/2611.22128)
![[Uncaptioned image]](https://arxiv.org/html/2611.22128v1/assets/AMD_logo.png)
TempQuant: Learning Adaptive Temperatures for Quantized Language Models
Mukund Athmaram12♣\\clubsuitLiang Ma1Manoj Rao1
1AMD2Cornell University
♣\\clubsuitCorrespondence to:ma2388@cornell\.edu\.
††footnotetext:Work conducted during an AMD internship\.###### Abstract
Large language models are increasingly deployed at low precision for applications ranging from reasoning to creative writing, yet quantization can degrade their performance\. These applications often use different decoding settings, such as the sampling temperature and top\-pp, which control the randomness of token selection\. We study how quantization interacts with these settings by examining how it distorts the model’s next\-token probabilities, which can change generated responses and task performance\. Measuring entropy during generation across deployed temperatures, we find that quantization warms the model on average, and that temperature matters much more for the performance of a quantized model than for its full\-precision counterpart\. Because quantization and temperature both shape the next\-token distribution, we investigate how much the sampling settings can undo what quantization changes, through our method TempQuant, which learns the quantized model’s temperature and top\-ppto bring its next\-token probabilities closer to the full\-precision model’s\. We learn these settings at two granularities\. The first is a single global setting per requested temperature, which we show finds nearly the same temperatures that tuning on labeled answers would, without any labels\. This matters in practice because model cards typically recommend sampling a quantized model with the same settings as its full\-precision counterpart, while we find that different settings can substantially improve its accuracy\. On 3\-bit Qwen3\-4B, for example, global TempQuant raises MATH accuracy from34%34\\%under the configuration recommended for Qwen’s quantized releases to51%51\\%\. The second sets the temperature and top\-ppfor every token\. It is motivated by our finding that tokens whose rankings flip under quantization tend to need warming, while tokens whose rankings are preserved tend to need cooling, which one temperature cannot do at once\. Finally, we show that when quantization degrades accuracy substantially, as at 3 bits, TempQuant recovers much of it, and when degradation is lighter and there is little accuracy to recover, it still brings the quantized model’s behavior closer to the full\-precision model’s\.
## 1Introduction
At every step of autoregressive generation, large language models generate a probability distribution over the next token\. This probability distribution is a function of the prefix, which is the prompt together with any tokens generated so far\. Based on the chosen decoding method, we select a token from this distribution\[[43](https://arxiv.org/html/2611.22128#bib.bib43)\]\. When sampling, the randomness of token selection is controlled by the temperatureτ\\tau, a user\-chosen deployment setting\. The chosen temperature depends on the use case: higher temperatures generally lead to more diverse outputs, whereas lower temperatures concentrate sampling on the most likely tokens\. Changing the temperature changes the entropy of the distribution, that is, how spread out it is\.
Quantization is a separate form of compression, where we use fewer bits to represent the weights and, in some configurations, activations of the large language model, so it can fit in less memory and run faster on supporting hardware\[[6](https://arxiv.org/html/2611.22128#bib.bib6),[21](https://arxiv.org/html/2611.22128#bib.bib21)\]\. Compressing this information introduces numerical error, so the next\-token probability distributions can diverge from those of the full\-precision model\. These distortions can also change the distribution’s entropy, affecting the sampling randomness that the user controls through temperature\.
Temperature scaling and quantization thus both change the model’s next\-token distribution, and we analyze how the two interact\. How well a decoding setting works depends on the task and on factors such as quantization, and the best results often require extensive tuning\[[43](https://arxiv.org/html/2611.22128#bib.bib43)\], which motivates learning decoding settings automatically\[[46](https://arxiv.org/html/2611.22128#bib.bib46)\]\. We study how much of quantization’s distortion of the next\-token distribution can be corrected by adjusting temperature and top\-pp, and whether doing so recovers accuracy\.
We quantify the difference between the full\-precision and quantized probability distributions through Kullback–Leibler divergence\[[14](https://arxiv.org/html/2611.22128#bib.bib14)\]\. We hypothesize that reducing this divergence can help preserve the full\-precision model’s behavior\. This is motivated by*Accuracy is Not All You Need*, in which[Dutta et al\. \[5\]](https://arxiv.org/html/2611.22128#bib.bib5)argue that quantization should preserve the full\-precision model’s behavior, and that similar aggregate benchmark accuracy does not establish behavioral preservation, because correct\-to\-incorrect and incorrect\-to\-correct “flips” can cancel out\. They motivate distance metrics such as KL divergence to capture the changes that aggregate accuracy hides, and this divergence is what we reduce\.
Minimizing divergence between probability distributions is not new in quantization: LLM\-QAT updates quantized model weights to minimize forward KL for fixed teacher targets\[[21](https://arxiv.org/html/2611.22128#bib.bib21)\]\. However, quantization\-aware training \(QAT\) retrains the language\-model weights, which can be expensive\. We instead reduce this divergence with a lightweight method applied after post\-training quantization \(PTQ\)\. We modify two sampling settings, temperature and top\-pp, rather than the model weights, and call this method TempQuant\. We learn these settings at two granularities: global TempQuant learns one temperature and top\-ppto deploy the quantized model at for each temperature requested of the full\-precision model, and per\-step TempQuant chooses them at every decoding step\. Global TempQuant is the practical version that can be applied today, since it only changes the decoding settings\. We also show why changing the temperature at every token can recover even more of the quantization error, which motivates per\-step TempQuant\.
Figure 1:Illustrative three\-token distributions before and after temperature and top\-ppcorrection\.Figure[1](https://arxiv.org/html/2611.22128#S1.F1)illustrates how decoding settings change token probabilities\. At every token there is a best temperature and top\-pp, and per\-step TempQuant aims to choose it\.
In practice, existing deployments usually sample a quantized model at the settings chosen for its full\-precision counterpart\. Qwen, for example, recommends temperature0\.60\.6and top\-pp0\.950\.95for the full\-precision Qwen3\[[35](https://arxiv.org/html/2611.22128#bib.bib35)\]and repeats the same settings for its own 4\-bit and GGUF releases\[[36](https://arxiv.org/html/2611.22128#bib.bib36),[37](https://arxiv.org/html/2611.22128#bib.bib37)\], and community quantizations down to 3 bits copy these settings unchanged\[[29](https://arxiv.org/html/2611.22128#bib.bib29),[45](https://arxiv.org/html/2611.22128#bib.bib45)\]\. We argue that this leaves much of the quantization error unrecovered, and that choosing different sampling settings for the quantized model, whether at a global or a per\-token granularity, can recover much of it \(Section[4\.6](https://arxiv.org/html/2611.22128#S4.SS6)\)\.
## 2Related Work
##### Adaptive decoding\.
Several methods learn temperatures at a fine granularity: ATS per token for calibration\[[48](https://arxiv.org/html/2611.22128#bib.bib48)\], TempNet per input for task performance\[[34](https://arxiv.org/html/2611.22128#bib.bib34)\], and ReSET per token for quantized reasoning models, with settings tuned on labeled data\[[16](https://arxiv.org/html/2611.22128#bib.bib16)\]\. AutoDeco predicts temperature and top\-ppat every step with two small heads\[[46](https://arxiv.org/html/2611.22128#bib.bib46)\], a design we borrow, but trains them to raise the likelihood of reasoning traces that reach correct answers, so as to replace hand\-tuned decoding\. TempQuant instead learns its settings to match the full\-precision model, so as to undo the effect of quantization\.
##### Decoding\-time fixes\.
[Lotfi et al\. \[22\]](https://arxiv.org/html/2611.22128#bib.bib22)measure the token\-level KL between quantized and full\-precision reasoning models and reduce overthinking with a logit penalty on a curated set of markers\. TempQuant also corrects the quantized model at decoding time, but fits the standard temperature and top\-ppto the full\-precision model\. Other samplers target repetition without a reference model\[[13](https://arxiv.org/html/2611.22128#bib.bib13),[2](https://arxiv.org/html/2611.22128#bib.bib2),[26](https://arxiv.org/html/2611.22128#bib.bib26),[9](https://arxiv.org/html/2611.22128#bib.bib9),[28](https://arxiv.org/html/2611.22128#bib.bib28)\], and[Braverman et al\. \[3\]](https://arxiv.org/html/2611.22128#bib.bib3)correct entropy\-rate drift with a single temperature\-like parameter, a close precedent for global TempQuant\.
##### Temperature calibration and quantization\.
DACA, Thermometer, and Dual\-Align fit temperatures for confidence calibration\[[23](https://arxiv.org/html/2611.22128#bib.bib23),[42](https://arxiv.org/html/2611.22128#bib.bib42),[24](https://arxiv.org/html/2611.22128#bib.bib24)\], while we fit them to keep a quantized model close to its full\-precision version\. Aggressive quantization can lengthen chains of thought and cause repetition\[[22](https://arxiv.org/html/2611.22128#bib.bib22),[17](https://arxiv.org/html/2611.22128#bib.bib17),[20](https://arxiv.org/html/2611.22128#bib.bib20)\], and can flip preferences between candidate tokens\[[47](https://arxiv.org/html/2611.22128#bib.bib47)\]\. We use GPTQ\[[6](https://arxiv.org/html/2611.22128#bib.bib6)\], AWQ\[[19](https://arxiv.org/html/2611.22128#bib.bib19)\], and the MXFP4 format\[[41](https://arxiv.org/html/2611.22128#bib.bib41)\]\. Appendix[A](https://arxiv.org/html/2611.22128#A1)discusses related work in more detail\.
## 3Method
##### Matching the full\-precision distribution\.
Our goal is to bring the probability distribution the quantized model emits at each step as close as possible to what the full\-precision model emits under the user’s requested temperature and top\-pp\. We compare the two models after the same prefix\. Temperature divides the logits before they are turned into probabilities, and top\-ppkeeps the most probable tokens until their cumulative probability reaches the cutoff, then rescales them to sum to one\[[12](https://arxiv.org/html/2611.22128#bib.bib12)\]\.
Letut,zt∈ℝVu\_\{t\},z\_\{t\}\\in\\mathbb\{R\}^\{V\}be the full\-precision and quantized logits at prefixctc\_\{t\}, and let𝒩r\\mathcal\{N\}\_\{r\}denote top\-ppfiltering with cutoffrr\. With requested temperatureτ\>0\\tau\>0and cutoffρ\\rho, the full\-precision model emits
pt=𝒩ρ\(softmax\(utτ\)\),p\_\{t\}=\\mathcal\{N\}\_\{\\rho\}\(\\operatorname\{softmax\}\\\!\\left\(\\frac\{u\_\{t\}\}\{\\tau\}\\right\)\),\(1\)and we fixρ=\.95\\rho=\.95in our experiments\. If we instead sample the quantized model with temperatureTtT\_\{t\}and cutoffρtQ\\rho\_\{t\}^\{Q\}, it emits
qt=𝒩ρtQ\(softmax\(ztTt\)\)\.q\_\{t\}=\\mathcal\{N\}\_\{\\rho\_\{t\}^\{Q\}\}\(\\operatorname\{softmax\}\\\!\\left\(\\frac\{z\_\{t\}\}\{T\_\{t\}\}\\right\)\)\.\(2\)We learnTtT\_\{t\}andρtQ\\rho\_\{t\}^\{Q\}so thatqtq\_\{t\}is close toptp\_\{t\}\. Both forms of TempQuant are learned against the full\-precision model on the same training data, and both need only the quantized model at deployment\. We measure how closeqtq\_\{t\}is toptp\_\{t\}with forward KL,KL\(pt∥qt\)\\operatorname\{KL\}\(p\_\{t\}\\\|q\_\{t\}\), reverse KL,KL\(qt∥pt\)\\operatorname\{KL\}\(q\_\{t\}\\\|p\_\{t\}\), orJSD\(pt,qt\)\\mathrm\{JSD\}\(p\_\{t\},q\_\{t\}\), and Section[4\.3](https://arxiv.org/html/2611.22128#S4.SS3)explains why we use all three\. Both KL losses mix each distribution with a uniform component of weightη=10−4\\eta=10^\{\-4\}so they stay finite when top\-ppremoves a token the other model keeps \(Appendix[F\.2](https://arxiv.org/html/2611.22128#A6.SS2)\)\. Unless noted, divergences in this paper use these smoothed KLs between top\-pp\-filtered distributions\.
##### Global TempQuant\.
For each requested temperature, global TempQuant searches a grid of temperature and top\-ppvalues for the pair with the lowest divergence on the training data \(Appendix[E\.3](https://arxiv.org/html/2611.22128#A5.SS3)\)\. It is fit separately for each quantized model, and is temperature scaling\[[7](https://arxiv.org/html/2611.22128#bib.bib7)\]with the full\-precision model’s next\-token distribution in place of the labels, together with top\-pp\. We also learn two smaller versions: global TempQuant \(top\-pponly\), which keeps the requested temperature, and global TempQuant \(TTonly\), which keepsρtQ=\.95\\rho\_\{t\}^\{Q\}=\.95\. Global TempQuant adds no cost at deployment, since it only changes the two numbers passed to the sampler\.
##### Per\-step TempQuant\.
Per\-step TempQuant chooses the setting at every step\. At inference we no longer have the full\-precision model, so it has to choose from what the quantized model gives us\. At each step, it looks at the quantized logits after adjusting them for the requested temperature, and summarizes them in 42 features: the 32 largest logits, normalized, and ten statistics that describe how spread out the distribution is, such as its entropy and the gaps between the leading tokens\. Two small MLP heads read these features and predict a correctionκt\\kappa\_\{t\}to the requested temperature and a new top\-ppcutoff\. The sampler divides the original quantized logits byTt=τκtT\_\{t\}=\\tau\\kappa\_\{t\}and applies the predicted cutoff\. We borrow this two\-head design from AutoDeco\[[46](https://arxiv.org/html/2611.22128#bib.bib46)\]\. Withxt\(τ\)x\_\{t\}\(\\tau\)denoting the features,
κt=Fθ\(xt\(τ\)\),ρtQ=Gϕ\(xt\(τ\),logκt\),Tt=τκt\.\\kappa\_\{t\}=F\_\{\\theta\}\(x\_\{t\}\(\\tau\)\),\\qquad\\rho\_\{t\}^\{Q\}=G\_\{\\phi\}\(x\_\{t\}\(\\tau\),\\log\\kappa\_\{t\}\),\\qquad T\_\{t\}=\\tau\\kappa\_\{t\}\.\(3\)Appendix Figure[9](https://arxiv.org/html/2611.22128#A3.F9)shows how the heads control the sampler\. We also train per\-step TempQuant \(TTonly\), which only predicts the temperature and keepsρtQ=\.95\\rho\_\{t\}^\{Q\}=\.95\. Where a table lists training losses, “TempQuant” refers to per\-step TempQuant\. Appendix[C](https://arxiv.org/html/2611.22128#A3)gives the exact features, output ranges, and training details\.
##### Training the per\-step predictor\.
Both language models remain frozen, and we train only the two heads, comparing full next\-token distributions at every calibration prefix as in LLM\-QAT\[[21](https://arxiv.org/html/2611.22128#bib.bib21)\], with the sampler’s settings in place of the weights\. The calibration prefixes come from responses by both BF16 and the quantized model \(Appendix[C\.5](https://arxiv.org/html/2611.22128#A3.SS5)\), so the loss covers text from both the reference and the quantized model\. Each prefix is evaluated at requested temperatures0\.30\.3,0\.60\.6, and1\.01\.0, with all prefix and temperature pairs weighted equally\. Hard top\-ppgives no gradient for the cutoff, so training uses a smooth version of the quantized model’s top\-pp, as in AutoDeco, while the BF16 reference keeps hard filtering\. We minimize
ℒD=1M∑j=1MD\(pj,q~j\)\+λDℛ,\\mathcal\{L\}\_\{D\}=\\frac\{1\}\{M\}\\sum\_\{j=1\}^\{M\}D\\big\(p\_\{j\},\\widetilde\{q\}\_\{j\}\\big\)\+\\lambda\_\{D\}\\,\\mathcal\{R\},\(4\)wherejjruns over allMMpairs of a training prefix and a requested temperature,DDis one of the three divergences, andq~j\\widetilde\{q\}\_\{j\}uses the smooth top\-pp\. The penaltyℛ\\mathcal\{R\}measures how much the predicted temperature and top\-ppvary from one training prefix to the next\. We compute it separately at each requested temperature,0\.30\.3,0\.60\.6, and1\.01\.0, and average the three\. It keeps the heads from overfitting the small calibration set, and it also pulls them toward a single setting \(Appendix[C](https://arxiv.org/html/2611.22128#A3)\)\.
## 4Experiments
We first analyze how quantization changes the next\-token distribution \(Sections[4\.1](https://arxiv.org/html/2611.22128#S4.SS1)–[4\.3](https://arxiv.org/html/2611.22128#S4.SS3)\), then evaluate both forms of TempQuant \(Sections[4\.4](https://arxiv.org/html/2611.22128#S4.SS4)–[4\.5](https://arxiv.org/html/2611.22128#S4.SS5)\), their accuracy \(Section[4\.6](https://arxiv.org/html/2611.22128#S4.SS6)\), and how well they preserve the full\-precision model’s behavior \(Section[4\.7](https://arxiv.org/html/2611.22128#S4.SS7)\)\.
We evaluate Qwen3\-4B\[[51](https://arxiv.org/html/2611.22128#bib.bib51)\]quantized to 3\-bit GPTQ \(GPTQ3\) and to native 4\-bit MXFP4, at requested temperatures0\.30\.3,0\.60\.6, the setting Qwen recommends, and1\.01\.0, with further models in Section[4\.6](https://arxiv.org/html/2611.22128#S4.SS6)\. Appendix[D](https://arxiv.org/html/2611.22128#A4)gives the benchmarks, data splits, and statistical tests\.
### 4\.1How quantization changes the temperature
We find that quantized models have higher average entropy than BF16: their next\-token probabilities are more spread out, as if they were sampled at a higher temperature\. GPTQ3 at a requested0\.60\.6has the same mean entropy as BF16 at0\.740\.74, and the gap grows with the requested temperature \(Figure[2](https://arxiv.org/html/2611.22128#S4.F2)a\)\. Adding random noise of the same size to BF16’s logits leaves its effective temperature where it was, so the heating comes from quantization itself \(Appendix[E\.2](https://arxiv.org/html/2611.22128#A5.SS2.SSS0.Px2)\)\. MXFP4, which loses much less accuracy, shows a smaller shift in the same direction \(Figure[2](https://arxiv.org/html/2611.22128#S4.F2)a\)\.
Quantization barely changes the near\-certain tokens that make up most of the text, and the extra entropy comes from tokens where the model is more uncertain \(Figure[2](https://arxiv.org/html/2611.22128#S4.F2)b\)\.
Figure 2:Quantized models against BF16: \(a\) the BF16 temperature with the same mean entropy as GPTQ3, MXFP4, and BF16 with random noise added; \(b\) GPTQ3’s entropy change from BF16 atτ=0\.6\\tau=0\.6, grouped by the average entropy of the two models, with each group’s share of tokens above\.
### 4\.2Which tokens carry the quantization error
Figure 3:GPTQ3 against BF16 atτ=0\.6\\tau=0\.6, on tokens where the rankings are preserved or flipped: \(a\) share of tokens and of each divergence; \(b\) the temperature that best matches BF16 for each token under forward KL\.Since the quantized model runs hotter, it is natural to assume that cooling it would reduce this divergence\. However, our proxy for quantization error is KL divergence, not entropy, and whether warming or cooling reduces KL divergence depends on what quantization does to the token probabilities\. Quantization does two different things to tokens \(Figure[3](https://arxiv.org/html/2611.22128#S4.F3)\)\. We say a token’s rankings are preserved when the quantized model’s most likely next token is the same as BF16’s, and flipped when it is not; less likely tokens may still change order\. On most tokens, quantization keeps BF16’s most likely token and only spreads the probabilities out\. On these tokens, where the rankings are preserved, quantization acts like heating, so they tend to be best matched by cooling to a lower temperature\. On the rest, however, the rankings flip and BF16’s preferred token now ranks lower\. Here only a higher temperature gives it back probability, as in Figure[1](https://arxiv.org/html/2611.22128#S1.F1)b, so raising the temperature tends to reduce the KL divergence \(Figure[3](https://arxiv.org/html/2611.22128#S4.F3)b\)\. These flipped tokens are a small share of tokens, but each costs far more, so they carry most of the error under every divergence we use \(Figure[3](https://arxiv.org/html/2611.22128#S4.F3)a\), at every requested temperature \(Appendix Table[38](https://arxiv.org/html/2611.22128#A6.T38)\)\.
A single temperature therefore has to trade off the two, since warming helps the flipped tokens and cooling helps the preserved ones\. Fitting global TempQuant only on preserved tokens shows this\. Its temperature drops close to the requested one and fits those tokens slightly better, but the error rises on flipped tokens and over all tokens \(Appendix[E\.3](https://arxiv.org/html/2611.22128#A5.SS3)\)\. A per\-step temperature can instead correct each token on its own\.
### 4\.3Choice of divergence
To see how handling this trade\-off differently changes performance, we measure the quantization error with three divergences that weigh the two kinds of token differently\. TempQuant can be seen as a restricted form of knowledge distillation\[[10](https://arxiv.org/html/2611.22128#bib.bib10)\], in which the full\-precision model is the teacher, its quantized version is the student, and the student can lower the divergence only through its sampling settings, without changing its weights\. On\-policy distillation\[[1](https://arxiv.org/html/2611.22128#bib.bib1)\]shows that when a student cannot reproduce its teacher, the divergence decides which errors it accepts\. Inspired by this, we train with three divergences that fail in different ways \(Appendix[F\.1](https://arxiv.org/html/2611.22128#A6.SS1)\)\.
Forward KL is large when the quantized model gives little probability to a token BF16 considers likely\. A flipped token therefore spikes forward KL, so forward KL prioritizes warming to correct the flipped tokens\. Reverse KL, on the other hand, is large when the quantized model puts probability where BF16 does not\. Warming spreads probability onto tokens BF16 gives little probability to, which spikes reverse KL, so reverse KL prefers cooling even for the flipped tokens\. JSD combines the two, weighs both errors, and lands in between, near the requested temperature\. On a typical preserved token every divergence favors cooling, so they part ways mainly on the flipped tokens, as Appendix[B](https://arxiv.org/html/2611.22128#A2)illustrates\. This is the familiar contrast between mass\-covering and mode\-seeking divergences\.
Figure 4:Divergence from BF16 atτ=0\.6\\tau=0\.6when one temperature is used for every token, split into the part from tokens with preserved and flipped rankings\. Dots mark each curve’s minimum, and the dotted line the requested temperature\.Figure 5:Temperatures chosen by per\-step TempQuant for GPTQ3 under each training loss, on 768 held\-out steps\.Comparing the three also addresses a question this trade\-off raises: whether it is better to correct the flipped tokens, which are rare but carry most of the error, or the preserved tokens, which carry less of it but are far more common\. Figures[4](https://arxiv.org/html/2611.22128#S4.F4)and[5](https://arxiv.org/html/2611.22128#S4.F5)show the temperatures each divergence selects for global and per\-step TempQuant, and Appendix[B](https://arxiv.org/html/2611.22128#A2)gives the settings global TempQuant selects at every requested temperature\.
### 4\.4Global TempQuant
BF16 accuracy stays nearly flat across our three temperatures \(Table[2](https://arxiv.org/html/2611.22128#S4.T2)\), but under 3\-bit quantization the deployed temperature decides both accuracy and the divergence from BF16 \(Figure[4](https://arxiv.org/html/2611.22128#S4.F4)\)\. We first show that the best temperature for the quantized model differs from the full\-precision one, as temperature tuning finds, and then that global TempQuant, which does no sweep and only learns the setting that mimics BF16, finds nearly the same temperature\. Native 4\-bit MXFP4 behaves similarly \(Table[29](https://arxiv.org/html/2611.22128#A5.T29)\)\. Global TempQuant turns top\-ppoff at every requested temperature, which by itself removes much of the forward KL, and the learned temperature removes more \(Appendix Table[36](https://arxiv.org/html/2611.22128#A6.T36)\)\.
##### TempQuant versus temperature tuning\.
The usual way to choose a sampler setting is to tune it on labeled data: generate and grade responses at every candidate setting and keep the most accurate one\[[43](https://arxiv.org/html/2611.22128#bib.bib43),[46](https://arxiv.org/html/2611.22128#bib.bib46)\]\. Tuning GPTQ3 this way on 200 MATH questions took about 7,800 graded responses and 170 million generated tokens \(Figure[6](https://arxiv.org/html/2611.22128#S4.F6), Appendix Figure[12](https://arxiv.org/html/2611.22128#A5.F12)\), and has to be repeated for every model and task\. Global TempQuant needs no labels or grading, only the full\-precision model’s next\-token probabilities on a few hundred unlabeled responses, and lands close to the setting tuning finds\. The best setting differs between models, quantization methods, and requested temperatures, so a fixed rule such as sampling every quantized model at a higher temperature does not carry over \(Appendix[E\.3](https://arxiv.org/html/2611.22128#A5.SS3.SSS0.Px5)\)\.
Figure 6:MATH accuracy at fixed temperatures, and the temperature global TempQuant chooses from BF16 without answers for a requested0\.60\.6\(dotted\)\.
##### TempQuant and the recommended recipe\.
Model providers have noticed that quantized models degrade and loop at these temperatures\. Qwen’s cards for its quantized models keep the temperature at0\.60\.6but add a presence penalty of1\.51\.5to stop endless repetition\[[36](https://arxiv.org/html/2611.22128#bib.bib36)\], the failure that costs GPTQ3 most of its accuracy\. Global TempQuant performs better than this recipe, and adding the penalty on top of global TempQuant helps a little further, so the two combine \(Table[1](https://arxiv.org/html/2611.22128#S4.T1)\)\. The penalty pushes the model away from tokens it has already used, while TempQuant moves it toward the full\-precision model\.
Table 1:Qwen3\-4B GPTQ3 atτ=0\.6\\tau=0\.6\(%\)\. Qwen’s recipe adds top\-kk20 and presence penalty 1\.5; strict counts responses at the length limit as wrong\.
### 4\.5Per\-step TempQuant
Per\-step TempQuant chooses the setting for every token to avoid the compromise of Section[4\.2](https://arxiv.org/html/2611.22128#S4.SS2)\. To measure how much of the quantization error this could remove, we test an oracle that knows BF16’s distribution at every token and picks the quantized model’s temperature and top\-pp, from the grid global TempQuant searches, to minimize the divergence\. The oracle removes most of the divergence quantization adds, 84% of the forward KL at the recommended0\.60\.6, almost all of it through the temperature alone \(Appendix Table[36](https://arxiv.org/html/2611.22128#A6.T36)\), and sampling GPTQ3 with its settings brings GSM8K accuracy close to BF16’s and lifts MATH well above TempQuant \(Appendix[F\.5](https://arxiv.org/html/2611.22128#A6.SS5)\)\. Per\-step TempQuant tries to capture this headroom from the quantized model alone, so it has to infer from the quantized logits whether the top token changed \(examples in Appendix[E\.4](https://arxiv.org/html/2611.22128#A5.SS4)\)\.
### 4\.6Downstream accuracy
Table 2:GPTQ3 accuracy, with the share of responses at the length limit and the closeness to BF16 averaged over the four tasks\. Bold marks the best quantized value \(lowest for the length limit\); green marks Holm\-significant gains over GPTQ3\.Table[2](https://arxiv.org/html/2611.22128#S4.T2)measures accuracy on each policy’s own responses\. At0\.30\.3and0\.60\.6, TempQuant raises GSM8K, MATH, and GPQA accuracy far above raw GPTQ3, lifting mean accuracy from10%10\\%to40%40\\%at0\.30\.3\. At1\.01\.0, where raw GPTQ3 loops less, there is less to recover \(ablations and strict grading in Appendix[E\.1](https://arxiv.org/html/2611.22128#A5.SS1)\)\. The correction carries over to 3\-bit Qwen3\-8B and to Gemma 4 31B, a different family, raising Gemma’s MATH accuracy from 70% to 90%, but not to R1\-Distill\-Llama\-8B, which rarely runs into repetition \(Appendix Table[12](https://arxiv.org/html/2611.22128#A5.T12)\)\. Table[2](https://arxiv.org/html/2611.22128#S4.T2)also reports how close each policy’s generated text is to BF16’s with a closeness score: 100 minus the average gap between a policy and BF16 on four statistics of its text, namely word overlap with BF16’s responses, how alike the policy’s own responses to the same prompt are, repeated four\-grams, and the share of responses that hit the length limit \(details in Appendix[E\.7](https://arxiv.org/html/2611.22128#A5.SS7)\)\.
Of the three divergences, forward KL gives per\-step TempQuant the highest accuracy, matching global TempQuant at0\.30\.3and0\.60\.6and exceeding it at1\.01\.0\(Appendix Table[10](https://arxiv.org/html/2611.22128#A4.T10)\)\. Global TempQuant adds no cost at deployment, and per\-step TempQuant adds time per token \(Appendix[G](https://arxiv.org/html/2611.22128#A7)\)\.
### 4\.7Does TempQuant preserve behavior and task performance?
Since similar accuracy can hide large changes in behavior\[[5](https://arxiv.org/html/2611.22128#bib.bib5)\], and matching a reference model need not improve task performance\[[44](https://arxiv.org/html/2611.22128#bib.bib44)\], we report behavior alongside accuracy, using task\-specific measures and the closeness score of Section[4\.6](https://arxiv.org/html/2611.22128#S4.SS6)\. The three divergences behave differently here, and the results separate into two cases: long reasoning under heavy degradation, and shorter outputs or lighter degradation\.
#### 4\.7\.1Long reasoning under heavy degradation
At the requested temperature, GPTQ3’s reasoning collapses into repetition: most of its responses repeat themselves until the length limit, while BF16’s rarely do \(Table[2](https://arxiv.org/html/2611.22128#S4.T2)\)\. Here, preserving BF16’s behavior mostly means stopping these loops, and a response that never finishes cannot be graded correct\. Breaking the loops and giving flipped tokens back their probability both need a higher temperature\. Forward KL, which warms the most, undoes about half of the collapse and recovers the most accuracy, while reverse KL cools, keeps the model on its favorite tokens, and makes the repetition worse \(Table[2](https://arxiv.org/html/2611.22128#S4.T2), Appendix[E\.7](https://arxiv.org/html/2611.22128#A5.SS7)\)\. The same pattern holds on the other 3\-bit models \(Appendix Table[12](https://arxiv.org/html/2611.22128#A5.T12)\)\. In this case the forward KL controller preserves BF16’s behavior best, because it targets the largest source of divergence, the looping\. Quantized reasoning models also overthink\[[22](https://arxiv.org/html/2611.22128#bib.bib22)\], and TempQuant reduces this only in part \(Appendix[E\.8](https://arxiv.org/html/2611.22128#A5.SS8)\)\.
#### 4\.7\.2Short text and light degradation
On short summaries, GPTQ3 does not loop, so there is little accuracy to recover\. Its main error is instead spreading probability onto tokens BF16 would not pick, so at0\.30\.3it varies about as much as BF16 does at0\.60\.6\. Reverse KL penalizes exactly this error, so it cools GPTQ3 back toward BF16’s variety, while forward KL makes nearly every sample different\. The reverse KL controller is closest to BF16 on variety, meaning diversity, summary quality, and closeness at every temperature \(Table[3](https://arxiv.org/html/2611.22128#S4.T3)\)\.
Figure 7:MXFP4 on MATH atτ=0\.6\\tau=0\.6: \(a\) accuracy and \(b\) disagreements with BF16, samples that one answers correctly and the other does not\.MXFP4 is a similar case on reasoning, with lighter degradation\. It loses little accuracy, but similar accuracy can hide many answer flips, samples that the full\-precision model answers correctly and the quantized model does not, or the reverse\[[5](https://arxiv.org/html/2611.22128#bib.bib5)\]\. On MATH at0\.60\.6, the reverse KL and JSD controllers keep accuracy about the same while removing about a third of the extra answer flips MXFP4 has with BF16 \(Figure[7](https://arxiv.org/html/2611.22128#S4.F7)\), and across all four tasks and three temperatures, the JSD controller brings which questions MXFP4 answers correctly closer to BF16’s \(Appendix Table[33](https://arxiv.org/html/2611.22128#A5.T33)\)\.
These results suggest the forward KL controller for more aggressive quantization, the JSD controller for reasoning under light degradation, and the reverse KL controller for short text, though larger experiments are needed before committing to this split\.
Table 3:GPTQ3 XSum summaries, eight per article: share of unique summaries, meaning diversity \(mean embedding distance between samples\), quality \(ROUGE\-2 against human summaries\), and closeness to BF16 \(Section[4\.6](https://arxiv.org/html/2611.22128#S4.SS6)\)\. BF16 rows give its values, andΔ\\Deltarows give each policy’s change from BF16, so values nearer 0 are closer\. Bold marks the closest policy\.
## 5Limitations
Temperature and top\-ppcannot reorder tokens, so errors that flip rankings remain even for the oracle\. The accuracy gains are large only when the quantized model runs into repetition\. None of the per\-step controllers we trained preserves every behavior we measure\. We fit TempQuant on math text only, evaluate reasoning with thinking mode on, use a single reference top\-ppof\.95\.95, and compare with Qwen’s presence penalty on one model\. Global TempQuant adds no cost at deployment, while our unoptimized per\-step implementation adds 18–34% to the time per generated token \(Appendix[G](https://arxiv.org/html/2611.22128#A7)\)\.
## 6Conclusion
Quantized models run hotter, but most of the error sits on the few tokens whose ranking quantization flips, which tend to need warming while the rest tend to need cooling\. Global TempQuant finds without labels the setting that tuning on answers would find, and recovers much of the accuracy lost by 3\-bit models that fall into repetition\. Per\-step TempQuant, which chooses the setting at every token from the quantized model alone, matches this at requested temperatures0\.30\.3and0\.60\.6and is more accurate at1\.01\.0\. The choice of divergence decides what is preserved: forward KL restores accuracy when the model loops, while for a lightly degraded model reverse KL brings short text and JSD brings reasoning closer to BF16’s\. Overall, we show that much of the quantization error can be recovered through temperature and top\-pp, so quantized models should be deployed at different settings from their full\-precision counterparts\.
## References
- \[1\]Agarwal, R\., Vieillard, N\., Zhou, Y\., Stanczyk, P\., Ramos, S\., Geist, M\., and Bachem, O\.On\-policy distillation of language models: Learning from self\-generated mistakes, 2024\.URL[https://arxiv\.org/abs/2306\.13649](https://arxiv.org/abs/2306.13649)\.
- \[2\]Basu, S\., Ramachandran, G\. S\., Keskar, N\. S\., and Varshney, L\. R\.Mirostat: A neural text decoding algorithm that directly controls perplexity, 2021\.URL[https://arxiv\.org/abs/2007\.14966](https://arxiv.org/abs/2007.14966)\.
- \[3\]Braverman, M\., Chen, X\., Kakade, S\. M\., Narasimhan, K\., Zhang, C\., and Zhang, Y\.Calibration, entropy rates, and memory in language models\.In*Proceedings of the 37th International Conference on Machine Learning*, 2020\.URL[https://arxiv\.org/abs/1906\.05664](https://arxiv.org/abs/1906.05664)\.
- \[4\]Cobbe, K\., Kosaraju, V\., Bavarian, M\., Chen, M\., Jun, H\., Kaiser, L\., Plappert, M\., Tworek, J\., Hilton, J\., Nakano, R\., Hesse, C\., and Schulman, J\.Training verifiers to solve math word problems, 2021\.URL[https://arxiv\.org/abs/2110\.14168](https://arxiv.org/abs/2110.14168)\.
- \[5\]Dutta, A\., Krishnan, S\., Kwatra, N\., and Ramjee, R\.Accuracy is not all you need, 2024\.URL[https://arxiv\.org/abs/2407\.09141](https://arxiv.org/abs/2407.09141)\.
- \[6\]Frantar, E\., Ashkboos, S\., Hoefler, T\., and Alistarh, D\.GPTQ: Accurate post\-training quantization for generative pre\-trained transformers, 2023\.URL[https://arxiv\.org/abs/2210\.17323](https://arxiv.org/abs/2210.17323)\.
- \[7\]Guo, C\., Pleiss, G\., Sun, Y\., and Weinberger, K\. Q\.On calibration of modern neural networks, 2017\.URL[https://arxiv\.org/abs/1706\.04599](https://arxiv.org/abs/1706.04599)\.
- \[8\]Hendrycks, D\., Burns, C\., Kadavath, S\., Arora, A\., Basart, S\., Tang, E\., Song, D\., and Steinhardt, J\.Measuring mathematical problem solving with the MATH dataset, 2021\.URL[https://arxiv\.org/abs/2103\.03874](https://arxiv.org/abs/2103.03874)\.
- \[9\]Hewitt, J\., Manning, C\. D\., and Liang, P\.Truncation sampling as language model desmoothing, 2022\.URL[https://arxiv\.org/abs/2210\.15191](https://arxiv.org/abs/2210.15191)\.
- \[10\]Hinton, G\., Vinyals, O\., and Dean, J\.Distilling the knowledge in a neural network, 2015\.URL[https://arxiv\.org/abs/1503\.02531](https://arxiv.org/abs/1503.02531)\.
- \[11\]Holm, S\.A simple sequentially rejective multiple test procedure\.*Scandinavian Journal of Statistics*, 6\(2\):65–70, 1979\.
- \[12\]Holtzman, A\., Buys, J\., Du, L\., Forbes, M\., and Choi, Y\.The curious case of neural text degeneration\.In*International Conference on Learning Representations*, 2020\.URL[https://arxiv\.org/abs/1904\.09751v2](https://arxiv.org/abs/1904.09751v2)\.
- \[13\]Keskar, N\. S\., McCann, B\., Varshney, L\. R\., Xiong, C\., and Socher, R\.CTRL: A conditional transformer language model for controllable generation, 2019\.URL[https://arxiv\.org/abs/1909\.05858](https://arxiv.org/abs/1909.05858)\.
- \[14\]Kullback, S\. and Leibler, R\. A\.On information and sufficiency\.*The Annals of Mathematical Statistics*, 22\(1\):79 – 86, 1951\.doi:10\.1214/aoms/1177729694\.URL[https://doi\.org/10\.1214/aoms/1177729694](https://doi.org/10.1214/aoms/1177729694)\.
- \[15\]Kwon, W\., Li, Z\., Zhuang, S\., Sheng, Y\., Zheng, L\., Yu, C\. H\., Gonzalez, J\. E\., Zhang, H\., and Stoica, I\.Efficient memory management for large language model serving with PagedAttention, 2023\.URL[https://arxiv\.org/abs/2309\.06180](https://arxiv.org/abs/2309.06180)\.
- \[16\]Lee, S\., Lee, J\., Yoo, D\., Kim, J\. G\., Ryu, H\., Ryu, S\., and Choi, J\.ReSET: Accurate latency\-critical NVFP4 reasoning via step\-aware temperature scaling, 2026\.URL[https://arxiv\.org/abs/2606\.13233](https://arxiv.org/abs/2606.13233)\.
- \[17\]Li, S\., Ning, X\., Wang, L\., Liu, T\., Shi, X\., Yan, S\., Dai, G\., Yang, H\., and Wang, Y\.Evaluating quantized large language models, 2024\.URL[https://arxiv\.org/abs/2402\.18158](https://arxiv.org/abs/2402.18158)\.
- \[18\]Lin, C\.\-Y\.ROUGE: A package for automatic evaluation of summaries\.In*Text Summarization Branches Out*, pp\. 74–81, Barcelona, Spain, July 2004\. Association for Computational Linguistics\.URL[https://aclanthology\.org/W04\-1013/](https://aclanthology.org/W04-1013/)\.
- \[19\]Lin, J\., Tang, J\., Tang, H\., Yang, S\., Chen, W\.\-M\., Wang, W\.\-C\., Xiao, G\., Dang, X\., Gan, C\., and Han, S\.AWQ: Activation\-aware weight quantization for LLM compression and acceleration\.In*Proceedings of Machine Learning and Systems*, 2024\.URL[https://arxiv\.org/abs/2306\.00978](https://arxiv.org/abs/2306.00978)\.
- \[20\]Liu, R\., Sun, Y\., Zhang, M\., Bai, H\., Yu, X\., Yu, T\., Yuan, C\., and Hou, L\.Quantization hurts reasoning? An empirical study on quantized reasoning models, 2025\.URL[https://arxiv\.org/abs/2504\.04823](https://arxiv.org/abs/2504.04823)\.
- \[21\]Liu, Z\., Oguz, B\., Zhao, C\., Chang, E\., Stock, P\., Mehdad, Y\., Shi, Y\., Krishnamoorthi, R\., and Chandra, V\.LLM\-QAT: Data\-free quantization aware training for large language models, 2023\.URL[https://arxiv\.org/abs/2305\.17888](https://arxiv.org/abs/2305.17888)\.
- \[22\]Lotfi, S\., Kirichenko, P\., Li, S\., and Liu, Z\.Quantized reasoning models think they need to think longer, but they do not, 2026\.URL[https://arxiv\.org/abs/2606\.00206](https://arxiv.org/abs/2606.00206)\.
- \[23\]Luo, B\., Wang, S\., Li, S\., and Wei, H\.Your pre\-trained LLM is secretly an unsupervised confidence calibrator, 2025\.URL[https://arxiv\.org/abs/2505\.16690](https://arxiv.org/abs/2505.16690)\.
- \[24\]Luo, B\., Wang, C\., Wei, H\., Li, S\., and Du, X\.Unlocking the pre\-trained model as a dual\-alignment calibrator for post\-trained LLMs\.*arXiv preprint arXiv:2601\.04277*, 2026\.URL[https://arxiv\.org/abs/2601\.04277v1](https://arxiv.org/abs/2601.04277v1)\.
- \[25\]Mattei, P\.\-A\. and Loureiro, B\.The well\-tempered classifier: Some elementary properties of temperature scaling, 2026\.URL[https://arxiv\.org/abs/2602\.14862](https://arxiv.org/abs/2602.14862)\.
- \[26\]Meister, C\., Pimentel, T\., Wiher, G\., and Cotterell, R\.Locally typical sampling\.*Transactions of the Association for Computational Linguistics*, 2023\.URL[https://arxiv\.org/abs/2202\.00666](https://arxiv.org/abs/2202.00666)\.
- \[27\]Narayan, S\., Cohen, S\. B\., and Lapata, M\.Don’t give me the details, just the summary\! topic\-aware convolutional neural networks for extreme summarization\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pp\. 1797–1807, Brussels, Belgium, October\-November 2018\. Association for Computational Linguistics\.doi:10\.18653/v1/D18\-1206\.URL[https://aclanthology\.org/D18\-1206/](https://aclanthology.org/D18-1206/)\.
- \[28\]Nguyen, M\. N\., Baker, A\., Neo, C\., Roush, A\., Kirsch, A\., and Shwartz\-Ziv, R\.Turning up the heat: Min\-p sampling for creative and coherent LLM outputs, 2025\.URL[https://arxiv\.org/abs/2407\.01082](https://arxiv.org/abs/2407.01082)\.
- \[29\]Ollama\.qwen3:4b\.[https://ollama\.com/library/qwen3](https://ollama.com/library/qwen3), 2025\.Default parameters of the 4\-bit build; accessed November 2026\.
- \[30\]Papineni, K\., Roukos, S\., Ward, T\., and Zhu, W\.\-J\.Bleu: a method for automatic evaluation of machine translation\.In*Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics*, pp\. 311–318, Philadelphia, Pennsylvania, USA, July 2002\. Association for Computational Linguistics\.doi:10\.3115/1073083\.1073135\.URL[https://aclanthology\.org/P02\-1040/](https://aclanthology.org/P02-1040/)\.
- \[31\]Peeperkorn, M\., Kouwenhoven, T\., Brown, D\., and Jordanous, A\.Is temperature the creativity parameter of large language models?, 2024\.URL[https://arxiv\.org/abs/2405\.00492v1](https://arxiv.org/abs/2405.00492v1)\.
- \[32\]Pillutla, K\., Liu, L\., Thickstun, J\., Welleck, S\., Swayamdipta, S\., Zellers, R\., Oh, S\., Choi, Y\., and Harchaoui, Z\.MAUVE scores for generative models: Theory and practice, 2023\.URL[https://arxiv\.org/abs/2212\.14578](https://arxiv.org/abs/2212.14578)\.
- \[33\]Post, M\.A call for clarity in reporting BLEU scores\.In*Proceedings of the Third Conference on Machine Translation: Research Papers*, pp\. 186–191, Brussels, Belgium, October 2018\. Association for Computational Linguistics\.doi:10\.18653/v1/W18\-6319\.URL[https://aclanthology\.org/W18\-6319/](https://aclanthology.org/W18-6319/)\.
- \[34\]Qiu, Z\.\-H\., Guo, S\., Xu, M\., Zhao, T\., Zhang, L\., and Yang, T\.To cool or not to cool? Temperature network meets large foundation models via DRO, 2024\.URL[https://arxiv\.org/abs/2404\.04575](https://arxiv.org/abs/2404.04575)\.
- \[35\]Qwen Team\.Qwen3\-4B model card\.[https://huggingface\.co/Qwen/Qwen3\-4B](https://huggingface.co/Qwen/Qwen3-4B), 2025a\.Accessed November 2026\.
- \[36\]Qwen Team\.Qwen3\-4B\-AWQ model card\.[https://huggingface\.co/Qwen/Qwen3\-4B\-AWQ](https://huggingface.co/Qwen/Qwen3-4B-AWQ), 2025b\.Accessed November 2026\.
- \[37\]Qwen Team\.Qwen3\-8B\-GGUF model card\.[https://huggingface\.co/Qwen/Qwen3\-8B\-GGUF](https://huggingface.co/Qwen/Qwen3-8B-GGUF), 2025c\.Accessed November 2026\.
- \[38\]Raffel, C\., Shazeer, N\., Roberts, A\., Lee, K\., Narang, S\., Matena, M\., Zhou, Y\., Li, W\., and Liu, P\. J\.Exploring the limits of transfer learning with a unified text\-to\-text transformer\.*Journal of Machine Learning Research*, 2020\.URL[https://arxiv\.org/abs/1910\.10683](https://arxiv.org/abs/1910.10683)\.
- \[39\]Rein, D\., Hou, B\. L\., Stickland, A\. C\., Petty, J\., Pang, R\. Y\., Dirani, J\., Michael, J\., and Bowman, S\. R\.GPQA: A graduate\-level Google\-Proof Q&A benchmark, 2023\.URL[https://arxiv\.org/abs/2311\.12022](https://arxiv.org/abs/2311.12022)\.
- \[40\]Renze, M\. and Guven, E\.The effect of sampling temperature on problem solving in large language models\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pp\. 7346–7356\. Association for Computational Linguistics, 2024\.doi:10\.18653/v1/2024\.findings\-emnlp\.432\.URL[http://dx\.doi\.org/10\.18653/v1/2024\.findings\-emnlp\.432](http://dx.doi.org/10.18653/v1/2024.findings-emnlp.432)\.
- \[41\]Rouhani, B\. D\., Zhao, R\., More, A\., Hall, M\., Khodamoradi, A\., Deng, S\., Choudhary, D\., Cornea, M\., Dellinger, E\., Denolf, K\., Dusan, S\., Elango, V\., Golub, M\., Heinecke, A\., James\-Roxby, P\., Jani, D\., Kolhe, G\., Langhammer, M\., Li, A\., Melnick, L\., Mesmakhosroshahi, M\., Rodriguez, A\., Schulte, M\., Shafipour, R\., Shao, L\., Siu, M\., Dubey, P\., Micikevicius, P\., Naumov, M\., Verrilli, C\., Wittig, R\., Burger, D\., and Chung, E\.Microscaling data formats for deep learning, 2023\.URL[https://arxiv\.org/abs/2310\.10537](https://arxiv.org/abs/2310.10537)\.
- \[42\]Shen, M\., Das, S\., Greenewald, K\., Sattigeri, P\., Wornell, G\., and Ghosh, S\.Thermometer: Towards universal calibration for large language models, 2024\.URL[https://arxiv\.org/abs/2403\.08819](https://arxiv.org/abs/2403.08819)\.
- \[43\]Shi, C\., Yang, H\., Cai, D\., Zhang, Z\., Wang, Y\., Yang, Y\., and Lam, W\.A thorough examination of decoding methods in the era of LLMs, 2024\.URL[https://arxiv\.org/abs/2402\.06925](https://arxiv.org/abs/2402.06925)\.
- \[44\]Stanton, S\., Izmailov, P\., Kirichenko, P\., Alemi, A\. A\., and Wilson, A\. G\.Does knowledge distillation really work?, 2021\.URL[https://arxiv\.org/abs/2106\.05945](https://arxiv.org/abs/2106.05945)\.
- \[45\]Unsloth\.Qwen3\-4B\-GGUF\.[https://huggingface\.co/unsloth/Qwen3\-4B\-GGUF](https://huggingface.co/unsloth/Qwen3-4B-GGUF), 2025\.Accessed November 2026\.
- \[46\]Wang, Z\., Ma, D\., Huang, X\., Cai, D\., Lan, T\., Xu, J\., Mi, H\., Tang, X\., and Wang, Y\.The end of manual decoding: Towards truly end\-to\-end language models, 2025\.URL[https://arxiv\.org/abs/2510\.26697](https://arxiv.org/abs/2510.26697)\.
- \[47\]Wu, Z\., Dhiman, S\., and Koshiyama, A\.Which decisions low\-bit quantization breaks, and how to predict them, 2026\.URL[https://arxiv\.org/abs/2608\.06564](https://arxiv.org/abs/2608.06564)\.
- \[48\]Xie, J\., Chen, A\. S\., Lee, Y\., Mitchell, E\., and Finn, C\.Calibrating language models with adaptive temperature scaling, 2024\.URL[https://arxiv\.org/abs/2409\.19817](https://arxiv.org/abs/2409.19817)\.
- \[49\]Xu, J\., Liu, X\., Yan, J\., Cai, D\., Li, H\., and Li, J\.Learning to break the loop: Analyzing and mitigating repetitions for neural text generation, 2022\.URL[https://arxiv\.org/abs/2206\.02369](https://arxiv.org/abs/2206.02369)\.
- \[50\]Yang, A\., Zhang, B\., Hui, B\., Gao, B\., Yu, B\., Li, C\., Liu, D\., Tu, J\., Zhou, J\., Lin, J\., Lu, K\., Xue, M\., Lin, R\., Liu, T\., Ren, X\., and Zhang, Z\.Qwen2\.5\-Math technical report: Toward mathematical expert model via self\-improvement, 2024\.URL[https://arxiv\.org/abs/2409\.12122](https://arxiv.org/abs/2409.12122)\.
- \[51\]Yang, A\., Li, A\., Yang, B\., Zhang, B\., Hui, B\., Zheng, B\., Yu, B\., Gao, C\., Huang, C\., Lv, C\., Zheng, C\., Liu, D\., Zhou, F\., Huang, F\., Hu, F\., Ge, H\., Wei, H\., Lin, H\., Tang, J\., Yang, J\., Tu, J\., Zhang, J\., Yang, J\., Yang, J\., Zhou, J\., Zhou, J\., Lin, J\., Dang, K\., Bao, K\., Yang, K\., Yu, L\., Deng, L\., Li, M\., Xue, M\., Li, M\., Zhang, P\., Wang, P\., Zhu, Q\., Men, R\., Gao, R\., Liu, S\., Luo, S\., Li, T\., Tang, T\., Yin, W\., Ren, X\., Wang, X\., Zhang, X\., Ren, X\., Fan, Y\., Su, Y\., Zhang, Y\., Zhang, Y\., Wan, Y\., Liu, Y\., Wang, Z\., Cui, Z\., Zhang, Z\., Zhou, Z\., and Qiu, Z\.Qwen3 technical report, 2025\.URL[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\.
## Appendix AExtended related work
##### Adaptive decoding\.
Adaptive temperature scaling \(ATS\) and TempNet also learn temperatures at a fine granularity, but for different reasons\. ATS learns a temperature for each token to improve the model’s confidence calibration\[[48](https://arxiv.org/html/2611.22128#bib.bib48)\], and TempNet learns to predict a temperature for each input to improve prediction and generalization across tasks\[[34](https://arxiv.org/html/2611.22128#bib.bib34)\]\. ReSET also sets the temperature per token, and targets NVFP4\-quantized reasoning models\. It observes that quantization adds sampling errors at confident tokens and concentrates probability too much at uncertain reasoning steps\. It samples high\-entropy tokens at temperature1\.01\.0and the rest at a lower temperature, with an entropy threshold that adapts to the current reasoning step, and calibrates the low temperature for each model and task on labeled held\-out data\[[16](https://arxiv.org/html/2611.22128#bib.bib16)\]\. It does not learn a temperature predictor, and its goal is to balance precision and exploration during reasoning, rather than to imitate the full\-precision model\. Our entropy\-threshold baseline in Appendix[E\.3](https://arxiv.org/html/2611.22128#A5.SS3.SSS0.Px5)is a simpler rule in the spirit of ReSET, and not ReSET itself\. AutoDeco is the closest work to ours architecturally, since it also learns temperature and top\-ppat each generation step, with two small heads on the language model\. It trains them with cross\-entropy to raise the probability of the next token in reasoning traces selected for correct final answers\[[46](https://arxiv.org/html/2611.22128#bib.bib46)\], and we borrow its two\-head design\. TempQuant instead learns these settings so that the quantized model’s next\-token probabilities match those of its full\-precision counterpart, and uses them to measure how much of the quantization error the sampler can remove\.
##### Decoding\-time fixes for degeneration\.
[Lotfi et al\. \[22\]](https://arxiv.org/html/2611.22128#bib.bib22)measure the token\-level KL between quantized and full\-precision reasoning models, find that it is high at uncertain positions, where quantized models oversample overthinking markers such as “wait”, and reduce this with a training\-free logit penalty on a curated set of these markers\. TempQuant also corrects the quantized model at decoding time without retraining it, but fits the standard temperature and top\-ppto the full\-precision model, with no chosen set of tokens\. More general samplers target repetition and incoherence without a reference model: repetition penalties\[[13](https://arxiv.org/html/2611.22128#bib.bib13)\], Mirostat, which holds perplexity at a target\[[2](https://arxiv.org/html/2611.22128#bib.bib2)\], locally typical sampling\[[26](https://arxiv.org/html/2611.22128#bib.bib26)\], truncation sampling\[[9](https://arxiv.org/html/2611.22128#bib.bib9)\], and min\-pp\[[28](https://arxiv.org/html/2611.22128#bib.bib28)\]\. Repetition also reinforces itself once it starts\[[49](https://arxiv.org/html/2611.22128#bib.bib49)\], which is consistent with how quickly a cold 3\-bit model falls into loops\.[Braverman et al\. \[3\]](https://arxiv.org/html/2611.22128#bib.bib3)show that the entropy rate of generated text drifts away from that of real text, and correct this with a single temperature\-like parameter, a close precedent for global TempQuant\. We fit such a parameter against the full\-precision model after quantization\.
##### Temperature calibration\.
Other work learns temperatures to calibrate a model’s confidence\. Temperature scaling fits one temperature on labeled data\[[7](https://arxiv.org/html/2611.22128#bib.bib7)\]\. DACA fits a temperature with a reference\-based forward KL so that a post\-trained model’s confidence matches its pre\-trained model\[[23](https://arxiv.org/html/2611.22128#bib.bib23)\]\. Thermometer predicts one temperature for a task by aggregating unlabeled prompts\[[42](https://arxiv.org/html/2611.22128#bib.bib42)\], and Dual\-Align combines output and layerwise supervision to fit a single temperature\[[24](https://arxiv.org/html/2611.22128#bib.bib24)\]\. We do not select temperature and top\-ppfor confidence calibration, but to preserve the behavior of a quantized model\.
##### Distillation\.
Matching a teacher’s next\-token distribution is the basis of knowledge distillation\[[10](https://arxiv.org/html/2611.22128#bib.bib10)\]\. LLM\-QAT applies it to quantization, updating the quantized weights to minimize forward KL to the full\-precision model\[[21](https://arxiv.org/html/2611.22128#bib.bib21)\], and[Agarwal et al\. \[1\]](https://arxiv.org/html/2611.22128#bib.bib1)study how the choice of divergence affects a student that cannot reproduce its teacher\.
##### Temperature and generation behavior\.
Temperature affects the diversity and coherence of generated text\.[Holtzman et al\. \[12\]](https://arxiv.org/html/2611.22128#bib.bib12)show that low\-temperature sampling can reduce diversity and increase repetitive loops in open\-ended generation\. In a Llama\-2 story\-generation study,[Peeperkorn et al\. \[31\]](https://arxiv.org/html/2611.22128#bib.bib31)find that higher temperature is only weakly associated with novelty and more strongly associated with incoherence\. For full\-precision models, the temperature between00and11has little effect on problem\-solving accuracy\[[40](https://arxiv.org/html/2611.22128#bib.bib40)\]\.
##### Behavioral effects of quantization\.
Quantization can change reasoning behavior as well as accuracy, and similar aggregate accuracy does not establish that behavior is preserved\[[5](https://arxiv.org/html/2611.22128#bib.bib5)\]\.[Lotfi et al\. \[22\]](https://arxiv.org/html/2611.22128#bib.bib22)find that aggressive quantization can lengthen chains of thought and increase overthinking errors, where a model reaches a correct intermediate answer and then abandons it\. In dialogue evaluations,[Li et al\. \[17\]](https://arxiv.org/html/2611.22128#bib.bib17)document sentence\-level repetition under 3\-bit weight or KV\-cache quantization and repeated\-token failures at 2 bits\. The effects vary by configuration:[Liu et al\. \[20\]](https://arxiv.org/html/2611.22128#bib.bib20)find response lengths close to BF16 under several settings, while more aggressive quantization can lengthen outputs\. In our data, 3\-bit responses are several times longer than full\-precision ones, mostly because they run into repetition\. We measure response length and repetition alongside accuracy, using BF16 as our behavioral reference\.
##### Quantization and token probabilities\.
[Wu et al\. \[47\]](https://arxiv.org/html/2611.22128#bib.bib47)find that low\-bit quantization can shrink decision margins, the logit differences between the first tokens of two candidate continuations, and introduce directional biases that reverse model preferences\. A margin describes a particular choice, while entropy describes uncertainty over the full vocabulary, and one does not determine the other\. We study the related question of how quantization changes entropy and which tokens’ rankings it flips\. Our quantized models use GPTQ\[[6](https://arxiv.org/html/2611.22128#bib.bib6)\], AWQ\[[19](https://arxiv.org/html/2611.22128#bib.bib19)\], and the MXFP4 microscaling format\[[41](https://arxiv.org/html/2611.22128#bib.bib41)\]\.
## Appendix BWhy the divergences disagree on flipped tokens
Figure[8](https://arxiv.org/html/2611.22128#A2.F8)uses the three\-token examples of Figure[1](https://arxiv.org/html/2611.22128#S1.F1)to show how cooling and warming the quantized model change forward KL,∑ipilog\(pi/qi\)\\sum\_\{i\}p\_\{i\}\\log\(p\_\{i\}/q\_\{i\}\), and reverse KL,∑iqilog\(qi/pi\)\\sum\_\{i\}q\_\{i\}\\log\(q\_\{i\}/p\_\{i\}\), whereppis the full\-precision distribution andqqthe quantized one\. When the rankings are preserved, quantization only spreads the probabilities, and cooling bringsqqback towardppunder both divergences\. When the rankings flip, cooling sharpensqqaround the wrong token: reverse KL improves, becauseqqmoves its mass off the token BF16 rules out, while forward KL worsens, because BF16’s preferred token loses probability\. Warming does the opposite\. A typical preserved token therefore pulls a single setting the same way under both divergences, and the divergences part ways mainly on the flipped tokens\.
Figure 8:Forward and reverse KL for the three\-token examples of Figure[1](https://arxiv.org/html/2611.22128#S1.F1), before correction and after cooling or warming the quantized model\. Arrows mark whether each divergence falls or rises\.Table[4](https://arxiv.org/html/2611.22128#A2.T4)gives the settings global TempQuant selects under each divergence\. Unlike Figure[4](https://arxiv.org/html/2611.22128#S4.F4), which varies the temperature alone, the search also chooses top\-pp, which lets reverse KL cut more tokens and cool further\.
Table 4:Temperature and top\-pp\(T,p\)\(T,p\)that global TempQuant selects for GPTQ3 under each training divergence\.∗At the lower edge of the search range\.
## Appendix CController architecture and training
We train two controllers per objective with different seeds and average their results\. Figure[9](https://arxiv.org/html/2611.22128#A3.F9)shows the design\.
Figure 9:Per\-step TempQuant’s two prediction heads and their connection to next\-token sampling\.### C\.1Architecture and inputs
The temperature and top\-ppheads share the same 42 logit features but have separate weights\. The temperature head outputs the correction factorκ\\kappa, and the top\-pphead also receiveslogκ\\log\\kappa, so gradients reach both heads\. We passlogκ\\log\\kapparather thanκ\\kappaso that keeping the requested temperature gives zero, and doubling or halving it moves the input by the same amount in opposite directions\. Following AutoDeco, the top\-pphead receives the predicted temperature because changing the temperature changes which tokens a given cutoff keeps\.
##### Model and quantization\.
We use[Qwen/Qwen3\-4B](https://huggingface.co/Qwen/Qwen3-4B/tree/1cfa9a7208912126459214e8b04321603b3df60c)\[[51](https://arxiv.org/html/2611.22128#bib.bib51)\]\. MXFP4 quantizes the linear weights statically and their inputs dynamically, in groups of 32 with shared E8M0 scales\. Embeddings, normalization, the tied vocabulary head, and the KV cache stay in BF16\. We serve models with vLLM\[[15](https://arxiv.org/html/2611.22128#bib.bib15)\]\.
Table 5:Architecture and output ranges of the per\-step predictor\.
### C\.2Input features
Table[6](https://arxiv.org/html/2611.22128#A3.T6)lists the features\. We adjust the logits for the requested temperature first, and normalize each feature with its training mean and standard deviation\. Herez~\\widetilde\{z\}is the logits scaled byτ0/τ\\tau\_\{0\}/\\tauand shifted so the maximum is zero, withτ0=0\.6\\tau\_\{0\}=0\.6,rkr\_\{k\}is itskkth largest value,ssis its standard deviation, andv=softmax\(z~/0\.6\)v=\\operatorname\{softmax\}\(\\widetilde\{z\}/0\.6\)\.
Table 6:The controller’s ten distribution summaries and 32 leading\-logit features, before normalization\.
### C\.3Output ranges and starting values
Each head produces an unbounded number, and a sigmoid maps it into an allowed range\. For head outputsaθ\(x\)a\_\{\\theta\}\(x\)andbϕ\(x,logκ\)b\_\{\\phi\}\(x,\\log\\kappa\),
logκ\\displaystyle\\log\\kappa=logκmin\+σ\(aθ\(x\)\+cT\)log\(κmaxκmin\),\\displaystyle=\\log\\kappa\_\{\\min\}\+\\sigma\(a\_\{\\theta\}\(x\)\+c\_\{T\}\)\\log\\\!\\left\(\\frac\{\\kappa\_\{\\max\}\}\{\\kappa\_\{\\min\}\}\\right\),ρQ\\displaystyle\\rho^\{Q\}=σ\(bϕ\(x,logκ\)\+cp\),\(κmin,κmax\)=\(0\.050\.6,500\.6\)\.\\displaystyle=\\sigma\(b\_\{\\phi\}\(x,\\log\\kappa\)\+c\_\{p\}\),\\qquad\(\\kappa\_\{\\min\},\\kappa\_\{\\max\}\)=\\left\(\\tfrac\{0\.05\}\{0\.6\},\\tfrac\{50\}\{0\.6\}\\right\)\.\(5\)The first line placeslogκ\\log\\kappabetweenlogκmin\\log\\kappa\_\{\\min\}andlogκmax\\log\\kappa\_\{\\max\}, so the correction lies between about0\.0830\.083and8383and is spread evenly on a log scale\. Equivalently, the temperature the head would use for a request of0\.60\.6, which is the scale the features are normalized to, lies between0\.050\.05and5050\. A correction ofκ=1\\kappa=1keeps the requested temperature, and the sampler usesTt=τκtT\_\{t\}=\\tau\\kappa\_\{t\}\. The second line keeps the top\-ppcutoff between00and11\. These areFθF\_\{\\theta\}andGϕG\_\{\\phi\}in Equation[3](https://arxiv.org/html/2611.22128#S3.E3)\.
The constantscTc\_\{T\}andcpc\_\{p\}are set so that, before training, both heads output the global TempQuant setting fitted on the training data, and the final layers start at zero\. Every prefix therefore starts at that global setting, and training learns how far to move away from it at each token\.
### C\.4Training settings
Only the two heads are trained \(Table[7](https://arxiv.org/html/2611.22128#A3.T7)\)\. We use the controller after 120 updates\.
Table 7:Training settings for all three objectives and every training run\.##### Action\-variance regularization\.
A small penalty keeps the predicted settings from varying too much from one training prefix to the next\. At each requested temperature, it adds the variance across prefixes of the log predicted temperature and of the predicted cutoff on the log\-odds scale, which spreads out cutoffs near11, and then averages over the three requested temperatures:
ℛ=13∑τ∈\{0\.3,0\.6,1\.0\}1N∑t=1N\[\(ℓt,τ−ℓ¯τ\)2\+\(ζt,τ−ζ¯τ\)2\],\\mathcal\{R\}=\\frac\{1\}\{3\}\\sum\_\{\\tau\\in\\\{0\.3,0\.6,1\.0\\\}\}\\frac\{1\}\{N\}\\sum\_\{t=1\}^\{N\}\\left\[\(\\ell\_\{t,\\tau\}\-\\bar\{\\ell\}\_\{\\tau\}\)^\{2\}\+\(\\zeta\_\{t,\\tau\}\-\\bar\{\\zeta\}\_\{\\tau\}\)^\{2\}\\right\],\(6\)whereℓt,τ=logκt\(τ\)\\ell\_\{t,\\tau\}=\\log\\kappa\_\{t\}\(\\tau\),ζt,τ=logitρtQ\(τ\)\\zeta\_\{t,\\tau\}=\\operatorname\{logit\}\\rho\_\{t\}^\{Q\}\(\\tau\),N=2,304N=2\{,\}304training prefixes are weighted equally, and bars are means\. The implementation scales the divergence bycDc\_\{D\}and the penalty byλ=\.01\\lambda=\.01, which gives Equation[4](https://arxiv.org/html/2611.22128#S3.E4)withλD=λ/cD\\lambda\_\{D\}=\\lambda/c\_\{D\}\.
##### Smooth top\-pp\.
During training we replace top\-ppwith a smooth version\. Sort the logits in ascending order, letbib\_\{i\}be their probabilities after temperature scaling, and letCi=∑j≤ibjC\_\{i\}=\\sum\_\{j\\leq i\}b\_\{j\}\. Withϵ=0\.01\\epsilon=0\.01,
loggi=logσ\(Ci−\(1−ρQ\)ϵ\)−logσ\(Ciϵ\),q~i=bigi∑jbjgj,\\log g\_\{i\}=\\log\\sigma\\\!\\left\(\\frac\{C\_\{i\}\-\(1\-\\rho^\{Q\}\)\}\{\\epsilon\}\\right\)\-\\log\\sigma\\\!\\left\(\\frac\{C\_\{i\}\}\{\\epsilon\}\\right\),\\qquad\\widetilde\{q\}\_\{i\}=\\frac\{b\_\{i\}g\_\{i\}\}\{\\sum\_\{j\}b\_\{j\}g\_\{j\}\},\(7\)withgi=1g\_\{i\}=1for the top token\. Targets, evaluation, and generation use ordinary top\-pp\. The KL smoothing weight and the sharpness of this smooth top\-ppare choices we make only so that training works, so Table[8](https://arxiv.org/html/2611.22128#A3.T8)tests how much the results depend on them\. Halving or doubling either one changes held\-out forward KL by at most5%5\\%and the mean chosen temperature by at most4%4\\%\. The predictor therefore learns essentially the same settings whatever values we pick, so these choices make training possible without shaping what it learns\. Since the held\-out divergence is measured with ordinary top\-pp, this also shows that training with the smooth version carries over to the hard cutoff used in deployment\.
Table 8:GPTQ3 held\-out forward KL and mean chosen temperature for forward\-KL per\-step TempQuant when the KL smoothing weight or the sharpness of the smooth top\-ppused in training is halved or doubled\. All variants use the same random seed; the chosen top\-ppstays between\.974\.974and\.976\.976\.
### C\.5Training data
We train on 24 GSM8K\[[4](https://arxiv.org/html/2611.22128#bib.bib4)\]and 24 MATH\[[8](https://arxiv.org/html/2611.22128#bib.bib8)\]training questions, and hold out 32 more from each, excluding MATH\-500\. For each question we collect two responses each from BF16, uncorrected quantization, and a preliminary version of per\-step TempQuant, at temperature0\.60\.6and top\-pp0\.950\.95\. We take eight points from each training response and two from each held\-out response, giving 2,304 training steps and 768 held\-out steps, and reuse them at all three requested temperatures\.
## Appendix DEvaluation details
### D\.1Protocol notes for Section[4](https://arxiv.org/html/2611.22128#S4)
Fitting global TempQuant has a one\-time cost: generating the calibration responses and recording both models’ next\-token probabilities on them\. Each requested temperature needs its own fit, on the same data\.
Every learned setting is fit on training questions, and divergence is measured on 768 decoding steps from 64 other questions\. Since we also used these questions during development, we confirm the reductions on 64 fresh questions with the same settings and no refitting: at0\.60\.6, GPTQ3’s forward KL falls from1\.351\.35to0\.350\.35with global TempQuant and0\.370\.37with per\-step TempQuant, and MXFP4’s from0\.790\.79to0\.230\.23and0\.320\.32\. Unless noted, downstream results for per\-step TempQuant average two training runs, and its divergence results on Qwen3\-4B average five\. The comparison with Qwen’s recipe \(Table[1](https://arxiv.org/html/2611.22128#S4.T1)\) and the temperature sweep \(Figure[6](https://arxiv.org/html/2611.22128#S4.F6)\) use 200 MATH questions, and the former also 198 GPQA questions, with one sample each\.
All GPTQ3 models use 3\-bit GPTQ with group size 128 and keep the vocabulary head in BF16\. The Qwen3\-4B GPTQ3 checkpoint is a public third\-party release111[https://huggingface\.co/irish\-quant/Qwen\-Qwen3\-4B\-3bit](https://huggingface.co/irish-quant/Qwen-Qwen3-4B-3bit), revisiond2c2902\., and we quantized Qwen3\-8B, R1\-Distill\-Llama\-8B, and Gemma 4 31B ourselves, calibrating on C4\[[38](https://arxiv.org/html/2611.22128#bib.bib38)\]\. GPTQ3 weights are dequantized for BF16 computation, while MXFP4 runs in native four\-bit computation\.
### D\.2Benchmarks and comparison methods
Table[9](https://arxiv.org/html/2611.22128#A4.T9)lists the tasks\. GSM8K, MATH\-500, and GPQA\-Diamond\[[39](https://arxiv.org/html/2611.22128#bib.bib39)\]use their full test sets\. AIME\-120 is AIME90 plus the 30 problems from 2025\. XSum\[[27](https://arxiv.org/html/2611.22128#bib.bib27)\]uses a fixed validation subset screened for overlap with the training data\. The global TempQuant settings are searched per requested temperature \(Appendix[E\.3](https://arxiv.org/html/2611.22128#A5.SS3)\), and per\-step TempQuant \(TTonly\) keepsp=\.95p=\.95\.
Table 9:Evaluation tasks, responses per item, and output limits at each requested temperature\.
### D\.3Generation and scoring
Seeds match across policies by task, question, and trial\. Accuracy is the average over sampled responses\. Thinking mode is on for reasoning tasks and off for XSum, with no repetition penalty\. GSM8K and AIME use the Qwen2\.5\-Math scorer\[[50](https://arxiv.org/html/2611.22128#bib.bib50)\], MATH\-500 uses Math\-Verify on the last boxed answer, and GPQA uses LM\-eval’s flexible extraction\. XSum quality is ROUGE\-2 F1 against the human summaries\[[18](https://arxiv.org/html/2611.22128#bib.bib18)\], after removing a leading thinking block\. The MXFP4 and GPTQ3 studies each sample their own BF16 reference runs, so BF16 scores differ slightly between their tables\.
### D\.4Behavior measures and uncertainty
Length counts generated tokens, and the cap rate is the share of responses that hit the length limit\. BLEU against BF16\[[30](https://arxiv.org/html/2611.22128#bib.bib30),[33](https://arxiv.org/html/2611.22128#bib.bib33)\]measures word overlap with BF16 responses, splitting each question’s responses into two halves, so BF16 against itself scores below 100\. Self\-BLEU measures how similar a model’s responses to the same question are, so higher means less diverse\. Repetition is the share of repeated four\-grams within a response\.
##### Testing one comparison\.
To compare two policies, for example raw GPTQ3 and global TempQuant, we use the fact that both answer the same questions with the same random seeds, and take the difference in accuracy question by question\. We then resample the questions with replacement many times and recompute the average difference each time\. The middle 95% of these averages gives the interval, and a gain is significant when the interval excludes zero\. We call this a paired bootstrap over questions\. Intervals in the appendix are not adjusted for multiple comparisons unless stated\. When a grade is unknown, we count it as half correct, the midpoint of counting it wrong or right\.
##### Testing many comparisons\.
We compare several policies, each training loss and global TempQuant, on several tasks and temperatures, and with this many comparisons a few would look significant by chance\. The Holm correction\[[11](https://arxiv.org/html/2611.22128#bib.bib11)\]guards against this\. It sorts the comparisons from smallest to largestpp\-value and tests them in that order: withmmcomparisons, the smallest must fall below0\.05/m0\.05/m, the next below0\.05/\(m−1\)0\.05/\(m\-1\), and so on, and testing stops at the first comparison that fails\. The chance that any comparison is declared significant by luck then stays below5%5\\%across all of them together\. For GPTQ3, we apply it across all 75 comparisons of every task, temperature, and policy against raw GPTQ3 and against each other \(Table[10](https://arxiv.org/html/2611.22128#A4.T10)\)\. For MXFP4, we apply it across its 54 comparisons of every task, temperature, and training loss against native MXFP4 \(Table[30](https://arxiv.org/html/2611.22128#A5.T30)\)\. MATH drops 3 questions whose reference answers cannot be graded\.
Table 10:GPTQ3 accuracy differences in percentage points with 95% intervals from a paired bootstrap over questions \(Appendix[D\.4](https://arxiv.org/html/2611.22128#A4.SS4)\)\. Per\-step is forward\-KL per\-step TempQuant\. Green marks gains that remain significant after Holm correction over all 75 tests, which cover every task, temperature, and training loss; the table shows the comparisons with global and per\-step TempQuant\. Differences use paired attempts, so they can differ slightly from Table[2](https://arxiv.org/html/2611.22128#S4.T2)\.
##### Strict grading\.
A response that hits the length limit is cut off before it finishes\. Our graders take the last answer that appears in the text, so a response that reached the right answer and then kept going, often repeating itself until the limit, can still be graded correct\. Atτ=0\.3\\tau=0\.3, for example, raw GPTQ3 hits the limit on 94% of GSM8K responses yet scores 18%\. To check that TempQuant’s gains do not come from grading unfinished responses, Table[11](https://arxiv.org/html/2611.22128#A4.T11)counts every response that hits the length limit as wrong\. The gains from TempQuant atτ=0\.3\\tau=0\.3and0\.60\.6hold under this stricter grading, and much of them comes from more responses finishing: atτ=0\.3\\tau=0\.3, GPTQ3 finishes few responses, while TempQuant finishes most of its GSM8K and GPQA responses and about half of its MATH responses\. Among the responses that finish, GPTQ3 is more accurate, because it only finishes its easiest ones\. Atτ=1\.0\\tau=1\.0, global TempQuant finishes fewer GSM8K responses than GPTQ3, so it falls further behind under strict grading, while per\-step TempQuant stays level or ahead\.
Table 11:Qwen3\-4B GPTQ3 under strict grading\. Strict counts responses that hit the length limit as wrong; Fin\. acc\. is accuracy among responses that finished; Fin\. is the share of responses that finished \(%\)\. Per\-step TempQuant is forward KL, two fits averaged\.
## Appendix EAdditional experimental results
### E\.1Full results for Section[4](https://arxiv.org/html/2611.22128#S4)
Table 12:3\-bit models atτ=0\.6\\tau=0\.6: accuracy \(%\), the share of responses at the length limit before and after global TempQuant, and the forward KL it removes\. Green marks gains over GPTQ3\.In Table[12](https://arxiv.org/html/2611.22128#A5.T12), Qwen3\-4B is evaluated on the full benchmarks and the other models on 300 GSM8K, 300 MATH, and 198 GPQA questions with four samples each\. On Qwen3\-8B and R1\-Distill\-Llama\-8B, the per\-token oracle removes a similar share of the divergence as on Qwen3\-4B \(Table[13](https://arxiv.org/html/2611.22128#A5.T13)\)\.
Table 13:Two further models quantized with GPTQ3, at requested temperatureτ=0\.6\\tau=0\.6, on held\-out steps\. Global TempQuant settings are searched on training steps and given as\(T,p\)\(T,p\); per\-step values average two training runs\. On Qwen3\-4B, the gap between per\-step and global TempQuant comes from the learned cutoff \(Appendix[F\.4](https://arxiv.org/html/2611.22128#A6.SS4)\)\.
### E\.2Details for Figure[2](https://arxiv.org/html/2611.22128#S4.F2)
##### The effective temperature\.
Figure[2](https://arxiv.org/html/2611.22128#S4.F2)a reports, for each requested temperatureτ\\tau, the BF16 temperatureTeq\(τ\)T\_\{\\rm eq\}\(\\tau\)whose mean entropy matches the quantized model’s on the same prefixes:
∑iwiH\(softmax\(uiTeq\(τ\)\)\)=∑iwiH\(softmax\(ziτ\)\)\.\\sum\_\{i\}w\_\{i\}H\\\!\\left\(\\operatorname\{softmax\}\\\!\\left\(\\frac\{u\_\{i\}\}\{T\_\{\\rm eq\}\(\\tau\)\}\\right\)\\right\)=\\sum\_\{i\}w\_\{i\}H\\\!\\left\(\\operatorname\{softmax\}\\\!\\left\(\\frac\{z\_\{i\}\}\{\\tau\}\\right\)\\right\)\.\(8\)We use the 198 BF16\-generated GPQA\-Diamond responses, sampled at0\.60\.6with top\-pp\.95\.95, with questions weighted equally\. GPTQ3 is hotter than BF16 at all three requested temperatures, on these responses and on MXFP4\-generated ones\.
##### Random\-noise control\.
For the noise line, we add Gaussian noise to BF16’s logits at every token, with the same size as GPTQ3’s difference from BF16 at that token, and find the BF16 temperature with the same mean entropy\. The noise gives0\.290\.29,0\.580\.58, and0\.990\.99, against0\.360\.36,0\.740\.74, and1\.261\.26for GPTQ3, with question\-level 95% intervals narrower than±0\.02\\pm 0\.02\.
##### Placing MXFP4\.
MXFP4 was measured on its own set of BF16\-generated GPQA responses, where BF16’s entropy is known at only the three requested temperatures\. We take its mean entropy relative to BF16’s on that text at each requested temperature and find the BF16 temperature with the same ratio on the GPTQ3 responses\. Applied to GPTQ3, the same procedure reproduces its values in the figure exactly\.
##### Grouping in panel \(b\)\.
Panel \(b\) groups tokens by the average entropy of the two models\. Figure[10](https://arxiv.org/html/2611.22128#A5.F10)shows why\. Grouped by BF16’s entropy alone, both GPTQ3 and random noise with the same forward KL look flatter on confident tokens and sharper on uncertain ones, since tokens picked for high BF16 entropy tend to have lower entropy under any other model\. Grouped by the average of the two entropies, the noise is sharper and GPTQ3 flatter almost everywhere\.
Figure 10:Entropy change from BF16 for GPTQ3 and for random noise with the same forward KL, grouped by BF16 entropy \(top\) or by the average of the two entropies \(bottom\)\.
##### Top\-ppand entropy\.
Figure[11](https://arxiv.org/html/2611.22128#A5.F11)shows how the requested top\-ppchanges the gap\. Top\-ppremoves the least likely tokens, which lowers the entropy of both models and also narrows the gap between them: after top\-pp, the gap from BF16 is 13–34% smaller\. Top\-ppalone does not remove it, so the quantized model still runs hotter than BF16 at the user’s settings\.
Figure 11:Quantized\-minus\-BF16 mean entropy on BF16 responses before and after top\-ppat\.95\.95, with 95% intervals\.
### E\.3Global TempQuant
##### How the setting is fit\.
For each requested temperature, global TempQuant tries a coarse grid of temperatures from0\.150\.15to3\.53\.5and top\-ppvalues from\.9\.9to11on the training steps, refines around the best pair, and is scored on held\-out steps from other questions\. Nearby settings give almost the same divergence\. The downstream runs at0\.30\.3and1\.01\.0use\(1\.09,1\)\(1\.09,1\)and\(1\.22,1\)\(1\.22,1\)\. The runs at0\.60\.6use\(1\.01,1\)\(1\.01,1\)from an earlier, coarser search, whose forward KL is within2%2\\%of the refined\(1\.13,1\)\(1\.13,1\)that the divergence tables use\.
##### Do our measurement choices create the improvement?
Plain KL is infinite when top\-ppcuts a token BF16 keeps, so we mix a tiny uniform probability into both distributions \(weight10−410^\{\-4\}\)\. The uncorrected model still pays a large penalty for each cut token, and the penalty grows as the weight shrinks, so part of its divergence comes from this choice\. Table[14](https://arxiv.org/html/2611.22128#A5.T14)varies the weight a thousandfold\. The uncorrected model’s divergence changes with it, because it cuts a BF16 token on about15%15\\%of steps, while global TempQuant turns top\-ppoff, cuts nothing, and stays the same\. TempQuant stays far below the uncorrected model at every weight, so the smoothing changes how large the gap looks, not whether it exists\. The other requested temperatures behave the same way\.
Table 14:GPTQ3 forward KL to BF16 atτ=0\.6\\tau=0\.6for each smoothing weight, and the share of steps where the policy’s top\-ppcuts a token BF16 keeps\.Our target is also BF16 as the user samples it, with top\-ppcutting its least likely tokens\. To check that this choice does not create the gains, Table[15](https://arxiv.org/html/2611.22128#A5.T15)compares against BF16’s full distribution instead\.
Table 15:Share of GPTQ3’s uncorrected forward KL removed, measured against BF16 with top\-pp\.95\.95\(Top\-pp\), as in the rest of the paper, and against the untruncated BF16 distribution \(Full\)\.
##### What drives the fitted temperature up?
Section[4\.2](https://arxiv.org/html/2611.22128#S4.SS2)argues that the flipped tokens push a single temperature up\. Table[16](https://arxiv.org/html/2611.22128#A5.T16)tests this by fitting global TempQuant only on tokens whose rankings are preserved\.
Table 16:Global TempQuant fit on all tokens or only on tokens whose rankings are preserved\. Held\-out GPTQ3 forward KL to BF16 on 768 steps, overall and split by rankings\. Bold marks the lower value in each pair\.
##### Global TempQuant against tuning on answers\.
Global TempQuant needs no labels, while tuning the sampler on answers took about 7,800 graded responses \(Figure[12](https://arxiv.org/html/2611.22128#A5.F12)\)\. One could tune on a small labeled set instead\. Table[17](https://arxiv.org/html/2611.22128#A5.T17)simulates this: we choose the most accurate setting on 10 to 100 random MATH questions and score it on the rest, against global TempQuant on the same questions\. Table[18](https://arxiv.org/html/2611.22128#A5.T18)compares the settings tuned on all 200 questions with global TempQuant on new MATH questions, GSM8K, and GPQA\. In both, global TempQuant reaches about the accuracy of answer tuning without any labels\.
Figure 12:MATH accuracy of GPTQ3 at fixed temperature and top\-ppsettings, and at the settings global TempQuant chooses from BF16 without answers, on the 198 gradable of 200 MATH questions with one sample each\. For Qwen3\-4B,\(1\.0,1\)\(1\.0,1\)shows the mean of two runs \(52\.0 and 49\.0\)\.Tuned minus global TempQuant \(accuracy points\)Labeled questionsMean10th–90th percentileWorse by 3\+ points \(%\)10\+0\.4\[−\-3\.7, \+3\.3\]1620\+0\.8\[−\-3\.4, \+3\.2\]1250\+1\.0\[−\-1\.4, \+3\.2\]6100\+0\.7\[−\-2\.3, \+3\.8\]7Table 17:Tuning GPTQ3’s sampler on a small labeled MATH set atτ=0\.6\\tau=0\.6and scoring it on the remaining questions, minus global TempQuant on the same questions \(4,000 random splits of 198 questions\)\.Qwen3\-4B GPTQ3 settingTuning MATHNew MATHGSM8KGPQAGlobal TempQuant,τ=0\.6\\tau=0\.6,\(1\.01,1\)\(1\.01,1\)49\.0053\.9280\.8025\.63Global TempQuant,τ=0\.3\\tau=0\.3,\(1\.09,1\)\(1\.09,1\)48\.7554\.6781\.7124\.24Tuned on answers\(1\.0,1\)\(1\.0,1\)47\.7555\.8381\.1225\.63Tuned on answers\(1\.2,\.95\)\(1\.2,\.95\)49\.3854\.3380\.4024\.75R1\-Distill\-Llama\-8B GPTQ3 settingTuning MATHNew MATHGSM8KGPQADefault\(0\.6,\.95\)\(0\.6,\.95\)64\.4869\.5076\.5722\.73Global TempQuant\(0\.72,1\)\(0\.72,1\)70\.0369\.1776\.7221\.72Tuned on answers\(0\.8,1\)\(0\.8,1\)71\.5567\.1776\.4227\.27Table 18:Accuracy \(%\) of settings tuned on MATH answers and of global TempQuant, on the 200 tuning questions, 300 new MATH questions, GSM8K, and GPQA\. Global TempQuant’s GSM8K and GPQA scores come from the original runs with four samples per question, and the tuned settings use two to four new samples\.
##### Other ways to choose a setting\.
We also tried 12 rules in the spirit of ReSET\[[16](https://arxiv.org/html/2611.22128#bib.bib16)\]that use a low temperature below an entropy threshold and a higher one above it\. The best reaches49\.5%49\.5\\%on the tuning questions, against53\.0%53\.0\\%for the best fixed setting\. Sampling every quantized model at1\.01\.0with top\-ppoff would be simpler, since raw GPTQ3 already does much better at1\.01\.0than at0\.60\.6\(Table[2](https://arxiv.org/html/2611.22128#S4.T2)\), but the best setting differs between models\. R1\-Distill\-Llama\-8B collapses a little above1\.01\.0, and matching BF16 picks a lower temperature for it, close to its most accurate setting \(Figure[12](https://arxiv.org/html/2611.22128#A5.F12)\), while Qwen’s 4\-bit AWQ release is best matched near0\.70\.7\(Appendix[E\.10](https://arxiv.org/html/2611.22128#A5.SS10)\)\.
### E\.4Per\-step examples
Table[19](https://arxiv.org/html/2611.22128#A5.T19)shows two written prompts\. In the first, GPTQ3 puts all of its probability on “multiple” where BF16 writes “common”, and per\-step TempQuant raises the temperature far above the global one, doubling the probability of “common”\. In the second, GPTQ3 already mostly agrees with BF16, and per\-step TempQuant stays close to the requested temperature, while global TempQuant’s higher temperature moves probability onto other tokens\.
Table 19:GPTQ3: temperature each policy uses and the probability it gives BF16’s next token atτ=0\.6\\tau=0\.6, on written prompts\.
### E\.5Comparing training losses
Lowering the divergence a model is trained on need not lower the others\. Table[20](https://arxiv.org/html/2611.22128#A5.T20)measures every training loss on all four divergences for GPTQ3\. Each loss lowers its own divergence, and none lowers all four at every temperature, which is why we report all three losses instead of treating one divergence as the measure of fidelity\. The same holds for fixed settings\. Among 140 fixed temperature and top\-ppsettings, none lowers all four on the training steps for GPTQ3 at0\.30\.3and0\.60\.6or for MXFP4 at any temperature, and the few that do for GPTQ3 at1\.01\.0lower them by at most 11%\.
Since per\-step TempQuant is a learned predictor, a result could depend on a lucky training run\. Table[21](https://arxiv.org/html/2611.22128#A5.T21)retrains each loss five times with different seeds: the divergences vary by only a few percent between runs, for example from0\.4600\.460to0\.4830\.483for forward KL at0\.60\.6, so a single run is representative\. Unless noted, downstream results average two runs\.
PolicyForward KLReverse KLJSDTotal variationTemperature \(τ\\tau\) = 0\.3Global TempQuant \(forward KL\)\-81\+36\+4\+27Global TempQuant \(JSD\)\-48\+7\-6\+5Global TempQuant \(reverse KL\)\+26\-1\+50Global TempQuant \(top\-pponly, forward KL\)\-570\-20Per\-step TempQuant \(forward KL\)\-75\+33\+2\+25Per\-step TempQuant \(JSD\)\-43\+7\-6\+4Per\-step TempQuant \(reverse KL\)\+23\-1\+40Temperature \(τ\\tau\) = 0\.6Global TempQuant \(forward KL\)\-73\+40\+9\+20Global TempQuant \(JSD\)\-40\+6\-2\+2Global TempQuant \(reverse KL\)\+114\-9\+20\+4Global TempQuant \(top\-pponly, forward KL\)\-63\+3\-1\+1Per\-step TempQuant \(forward KL\)\-68\+56\+14\+29Per\-step TempQuant \(JSD\)\-17\+5\-2\+1Per\-step TempQuant \(reverse KL\)\+62\-7\+90Temperature \(τ\\tau\) = 1\.0Global TempQuant \(forward KL\)\-60\+28\+11\+13Global TempQuant \(JSD\)\-2\-80\-1Global TempQuant \(reverse KL\)\+297\-24\+40\+12Global TempQuant \(top\-pponly, forward KL\)\-58\+8\+3\+4Per\-step TempQuant \(forward KL\)\-45\+35\+12\+15Per\-step TempQuant \(JSD\)\+8\-40\-1Per\-step TempQuant \(reverse KL\)\+124\-19\+11\+1Table 20:Change in GPTQ3 divergence from BF16 relative to uncorrected GPTQ3 \(%\), on held\-out steps\. Bold marks reductions\.Table 21:GPTQ3 divergence from BF16 for five per\-step TempQuant training runs per loss: mean \[minimum, maximum\]\.Table[22](https://arxiv.org/html/2611.22128#A5.T22)shows why the losses trade off\. Forward KL charges most for cutting a token BF16 would use, so the forward\-KL predictor warms and opens its cutoff, cutting almost none of BF16’s probability; the warming also moves probability onto tokens BF16’s own top\-ppremoves, which reverse KL and total variation charge for\. Reverse KL does the opposite: it cools and tightens the cutoff, placing the least probability on tokens BF16 removes while cutting more of BF16’s\. JSD lands in between\.
Table 22:GPTQ3 and per\-step TempQuant trained with each loss, on held\-out steps: probability each side places on tokens the other’s top\-ppremoves, and the mean settings the predictor chooses\. Bold marks the lowest value in each mass column\. Rows average two training runs\.
### E\.6What learning top\-ppadds
Per\-step TempQuant learns a temperature and a top\-ppcutoff\. To see what the cutoff adds, we compare it with per\-step TempQuant \(TTonly\), trained the same way but keepingp=\.95p=\.95\. On divergence, learning the cutoff lowers forward KL at every requested temperature, and all four divergences at0\.60\.6and1\.01\.0\(Table[23](https://arxiv.org/html/2611.22128#A5.T23)\)\. On accuracy, the two versions are close \(Table[24](https://arxiv.org/html/2611.22128#A5.T24)\); the difference on GSM8K at0\.60\.6is\+0\.7\+0\.7points with a 95% interval of\[−0\.1,\+1\.6\]\[\-0\.1,\+1\.6\]\. Global TempQuant that learns only top\-ppreaches much lower accuracy than global TempQuant that also learns the temperature, a mean of28\.3%28\.3\\%against40\.0%40\.0\\%at0\.60\.6\. With its learned top\-ppturned off at1\.01\.0, per\-step TempQuant scores0\.60\.6points lower on GSM8K and1\.31\.3on MATH, with intervals\[−1\.5,\+0\.2\]\[\-1\.5,\+0\.2\]and\[−3\.4,\+0\.8\]\[\-3\.4,\+0\.8\]\. Most of the correction therefore comes from the temperature\.
Table 23:Change in GPTQ3 divergence from BF16 relative to uncorrected GPTQ3 \(%\), on held\-out steps, for forward\-KL per\-step TempQuant with and without a learned top\-pp\. Bold marks the lower of each pair\. Rows average two training runs\.Table 24:GPTQ3 accuracy \(%\) with and without learning top\-pp\. Per\-step rows use forward KL and average two training runs; – marks a comparison not run\.
### E\.7Behavior beyond accuracy
##### Closeness\.
The closeness score summarizes how far a policy’s generated text is from BF16’s on four statistics: word overlap with BF16’s responses to the same question \(BLEU against a different half of BF16’s samples, so BF16 against itself scores below 100\); self\-BLEU, which measures how alike the policy’s own responses to the same question are; the share of repeated four\-grams; and the share of responses that hit the length limit\. For each task it is100−14∑m\|xm−bm\|100\-\\frac\{1\}\{4\}\\sum\_\{m\}\|x\_\{m\}\-b\_\{m\}\|, wherexmx\_\{m\}are the policy’s statistics andbmb\_\{m\}BF16’s, so BF16 scores 100; for reasoning we average it over GSM8K, MATH, AIME, and GPQA\. Table[25](https://arxiv.org/html/2611.22128#A5.T25)shows the four statistics atτ=0\.6\\tau=0\.6\. On reasoning, the gaps in repetition and in responses at the length limit dominate, so the score mostly tracks looping\. On XSum, where nothing loops, word overlap and self\-BLEU dominate, so it tracks variety\. The closest controller in each block follows the split of Section[4\.7](https://arxiv.org/html/2611.22128#S4.SS7)\. Long reasoning under heavy degradation, GPTQ3 on the four reasoning tasks, should favor forward KL, and the forward KL controllers are closest\. Short text should favor reverse KL, and the reverse KL controller is closest on both GPTQ3 and MXFP4 summaries\. Reasoning under light degradation, MXFP4 on GPQA, should favor JSD, and the JSD controller is the closest of the three\.
Table 25:The four statistics behind the closeness score atτ=0\.6\\tau=0\.6\. Closeness is100100minus the average absolute gap from BF16 over the four statistics, computed per task; for reasoning, the statistics and closeness are averaged over GSM8K, MATH, AIME, and GPQA\. Bold marks the quantized value nearest BF16’s\.
##### MAUVE\.
MAUVE\[[32](https://arxiv.org/html/2611.22128#bib.bib32)\]compares the distribution of a policy’s responses with BF16’s \(Table[26](https://arxiv.org/html/2611.22128#A5.T26)\)\. It is computed on 512 responses per task and temperature, with the same questions and seeds for every policy and each response truncated to its first 1,024 GPT\-2 tokens\. At0\.30\.3and0\.60\.6, both forms of TempQuant score above raw GPTQ3 on every task, and all policies stay below the agreement between two halves of BF16’s own samples\.
Table 26:GPTQ3 MAUVE against BF16 \(higher is closer\)\. BF16 vs BF16 compares two halves of BF16’s own samples; per\-step TempQuant uses forward KL\. Bold marks the quantized policy closest to BF16\.
### E\.8Overthinking
Quantized reasoning models tend to overthink, sampling words such as “Wait” and “Alternatively” that start unnecessary rechecking\[[22](https://arxiv.org/html/2611.22128#bib.bib22)\]\. GPTQ3 does the same\. Continuing generation from 14 points where GPTQ3 strongly prefers “Wait” and BF16 does not, chosen to show the effect where it is largest, raw GPTQ3 writes about twice as many overthinking markers as BF16, and the learned settings barely change this \(Table[28](https://arxiv.org/html/2611.22128#A5.T28)\)\. Table[27](https://arxiv.org/html/2611.22128#A5.T27)shows one such point\. GPTQ3 always opens with “Wait” there and BF16 never does; per\-step TempQuant warms sharply at this step and opens with doubt less often than global TempQuant\. Across all 14 points, the learned settings and even the oracle move only part of the way to BF16\. Changing the temperature delays this behavior more than it removes it\.
*Text so far: “…So maybe I should start by finding the LCM of 6 and 10\. Then, once I have that, I can figure out the smallest multiple of that LCM that’s greater than 100\.”*PolicyTTStart of one continuationOpens with doubtBF160\.60Alright, let me recall how to calculate the LCM\. I think the formula is LCM\(a,b\)=ab/GCD\(a,b\)\(a,b\)=ab/\\mathrm\{GCD\}\(a,b\)…0/16GPTQ30\.60Wait, but let me check\. The LCM of 6 and 10…Hmm, 6 and 10\. Let me recall how to calculate the LCM…16/16Global TempQuant1\.13Wait, but let me check\. Let me remember that LCM of 6 and 10\. But 6 and 10 have a common factor…14/16Per\-step TempQuant1\.91Alright, so let’s start with finding the LCM of 6 and 10\. To find the LCM of two numbers, I think I need to find their prime factors…10/16Oracle per\-step TempQuant2\.58Hmm, right\. Let me recall how to find the LCM\. To find the LCM of two numbers, I can use the formula…9/16Table 27:GPTQ3 continuations of one written problem atτ=0\.6\\tau=0\.6from the same seed\. Of 16 sampled continuations per policy, “Opens with doubt” counts those whose first word is “Wait”, “Hmm”, or “Alternatively” \(bold\), which start the rechecking[Lotfi et al\. \[22\]](https://arxiv.org/html/2611.22128#bib.bib22)associate with overthinking\.Table 28:Continuations from 14 points where GPTQ3 prefers “Wait” and BF16 does not, 16 per point and policy, 400 tokens each, with matched seeds\. Opens with doubt counts continuations whose first words are “Wait”, “Hmm”, or “Alternatively”\. Markers are the 50 overthinking markers of[Lotfi et al\. \[22\]](https://arxiv.org/html/2611.22128#bib.bib22)\. Bold marks the best non\-oracle policy\.
### E\.9MXFP4 results
This section collects the MXFP4 versions of the GPTQ3 results\.
##### Divergence\.
Global TempQuant removes about three quarters of MXFP4’s forward KL at every requested temperature \(Table[29](https://arxiv.org/html/2611.22128#A5.T29)\)\. On MXFP4 at0\.60\.6, the per\-token oracle removes about four fifths of the forward KL, and temperature alone about two thirds\.
Table 29:MXFP4 version of Table[36](https://arxiv.org/html/2611.22128#A6.T36): forward KL to BF16 on held\-out steps\. Global TempQuant uses\(T,p\)=\(1\.21,1\)\(T,p\)=\(1\.21,1\),\(1\.27,1\)\(1\.27,1\), and\(1\.43,1\)\(1\.43,1\)\. These MXFP4 next\-token distributions come from a separate collection on the same steps\. On GPTQ3, the gap between per\-step and global TempQuant comes from the learned cutoff \(Appendix[F\.4](https://arxiv.org/html/2611.22128#A6.SS4)\)\.
##### Accuracy\.
Table[30](https://arxiv.org/html/2611.22128#A5.T30)is the MXFP4 version of Table[2](https://arxiv.org/html/2611.22128#S4.T2)\. MXFP4’s closeness on GPQA is in Table[25](https://arxiv.org/html/2611.22128#A5.T25)\.
Table 30:MXFP4 version of Table[2](https://arxiv.org/html/2611.22128#S4.T2): accuracy, and the share of responses that hit the length limit, averaged over the four tasks\. Bold marks the best quantized value \(lowest for the length limit\); green marks gains over MXFP4 that remain significant after Holm correction \(Appendix[D\.4](https://arxiv.org/html/2611.22128#A4.SS4.SSS0.Px2)\)\.
##### Summaries\.
Table[31](https://arxiv.org/html/2611.22128#A5.T31)is the MXFP4 version of Table[3](https://arxiv.org/html/2611.22128#S4.T3), with variety measured by self\-BLEU, how alike a policy’s summaries of the same article are\. The reduction in the reverse KL controller’s self\-BLEU gap from BF16 has a 95% interval below zero at each temperature\.
Table 31:MXFP4 version of Table[3](https://arxiv.org/html/2611.22128#S4.T3): self\-BLEU, how alike a policy’s summaries of the same article are \(lower is more varied\), quality \(ROUGE\-2 against human summaries\), and closeness to BF16 \(Section[4\.6](https://arxiv.org/html/2611.22128#S4.SS6)\)\. BF16 rows give its values, andΔ\\Deltarows give each policy’s change from BF16\. Bold marks the closest policy\.
##### Correctness agreement on MATH\.
Table[32](https://arxiv.org/html/2611.22128#A5.T32)extends Figure[7](https://arxiv.org/html/2611.22128#S4.F7)to all three temperatures, on the 497 MATH questions whose reference answers can be scored, counting both directions of disagreement with BF16\. Two BF16 runs already disagree on about 67, 67, and 53 of 1,988 attempts at the three temperatures, and the best loss removes23%23\\%,37%37\\%, and24%24\\%of the disagreements MXFP4 adds on top of this\.
Table 32:MXFP4 MATH accuracy and correctness disagreements with BF16 on 1,988 scoreable attempts per temperature\. Per\-step rows average two fits\. Bold marks the highest accuracy and the fewest disagreements at each temperature\.
##### Pooled test across tasks\.
Figure[7](https://arxiv.org/html/2611.22128#S4.F7)shows one task at one temperature, where there are few disagreements to count\. To check whether the effect holds more broadly, we combine all four reasoning tasks at all three temperatures into one test, fixed before we computed it\. For each question, we compare how often a policy answers it correctly with how often BF16 does: if BF16 is right on 3 of 4 samples and the policy on 1 of 4, they differ by0\.50\.5\. We average this difference over all questions, and compare native MXFP4 with the JSD and reverse KL controllers\. To decide whether a difference is larger than chance, we resample the questions many times \(Appendix[D\.4](https://arxiv.org/html/2611.22128#A4.SS4)\), correcting for running two tests\. Table[33](https://arxiv.org/html/2611.22128#A5.T33)gives the result\. For reference, two halves of BF16’s own samples differ by0\.0810\.081on this measure\.
Table 33:Average difference between each policy’s and BF16’s share of correct answers per question on MXFP4, over four tasks and three temperatures\. The test compares the JSD and reverse KL controllers with native MXFP4; forward KL is shown for reference\. Bold marks the policy closest to BF16\.
### E\.10Qwen3\-4B AWQ
We apply global TempQuant to Qwen’s official 4\-bit AWQ release of Qwen3\-4B \(revision 74d4bd2, served in float16 with vLLM 0\.33\), which loses little accuracy, and apply the forward KL per\-step predictor trained on GPTQ3 to it unchanged\. Table[34](https://arxiv.org/html/2611.22128#A5.T34)gives the forward KL to BF16 on the held\-out steps atτ=0\.6\\tau=0\.6, and Table[35](https://arxiv.org/html/2611.22128#A5.T35)the accuracy over two samples per question\.
Table 34:Qwen3\-4B AWQ forward KL to BF16 on held\-out steps atτ=0\.6\\tau=0\.6under each setting\.Table 35:Accuracy \(%\) of Qwen3\-4B AWQ atτ=0\.6\\tau=0\.6\. Bold marks the better value in each column \(lower for the length limit\)\.
## Appendix FDivergence measures and oracle analysis
### F\.1Forward KL, reverse KL, and JSD
With natural logarithms andm=p\+q2m=\\frac\{p\+q\}\{2\},
KL\(p∥q\)\\displaystyle\\operatorname\{KL\}\(p\\\|q\)=∑i:pi\>0pilog\(piqi\),\\displaystyle=\\sum\_\{i:p\_\{i\}\>0\}p\_\{i\}\\log\\\!\\left\(\\frac\{p\_\{i\}\}\{q\_\{i\}\}\\right\),DJS\(p,q\)\\displaystyle D\_\{\\mathrm\{JS\}\}\(p,q\)=12KL\(p∥m\)\+12KL\(q∥m\),\\displaystyle=\\frac\{1\}\{2\}\\operatorname\{KL\}\(p\\\|m\)\+\\frac\{1\}\{2\}\\operatorname\{KL\}\(q\\\|m\),\(9\)and reverse KL swapsppandqq\. Forward KL is infinite when the quantized model removes a token BF16 keeps, and reverse KL in the opposite case\. JSD is always between zero andlog2\\log 2\.
### F\.2KL with smoothed probabilities
To keep KL finite, we mix a small uniform distributionUUinto both distributions,Pη=\(1−η\)P\+ηUP\_\{\\eta\}=\(1\-\\eta\)P\+\\eta Uwithη=10−4\\eta=10^\{\-4\}, and compute
DF,η\(P,Q\)=KL\(Pη∥Qη\),DR,η\(P,Q\)=KL\(Qη∥Pη\)\.D\_\{F,\\eta\}\(P,Q\)=\\operatorname\{KL\}\(P\_\{\\eta\}\\\|Q\_\{\\eta\}\),\\qquad D\_\{R,\\eta\}\(P,Q\)=\\operatorname\{KL\}\(Q\_\{\\eta\}\\\|P\_\{\\eta\}\)\.\(10\)If the quantized model removes probability massMMthat BF16 keeps, the penalty grows likeMlog\(1/η\)M\\log\(1/\\eta\)asη\\etashrinks\. Training scales both KL losses by1/41/4; reported values are unscaled\.
### F\.3Temperature\-only oracle
For a targetppandqβ=softmax\(βz\)q\_\{\\beta\}=\\operatorname\{softmax\}\(\\beta z\)withβ=1/T\\beta=1/Tandμ=∑ipizi\\mu=\\sum\_\{i\}p\_\{i\}z\_\{i\},
L\(β\)=log∑ieβzi−βμ−H\(p\),L′\(β\)=𝔼qβ\[z\]−μ,L′′\(β\)=Varqβ\(z\)\.L\(\\beta\)=\\log\\sum\_\{i\}e^\{\\beta z\_\{i\}\}\-\\beta\\mu\-H\(p\),\\qquad L^\{\\prime\}\(\\beta\)=\\mathbb\{E\}\_\{q\_\{\\beta\}\}\[z\]\-\\mu,\\qquad L^\{\\prime\\prime\}\(\\beta\)=\\operatorname\{Var\}\_\{q\_\{\\beta\}\}\(z\)\.\(11\)The problem is convex inβ\\beta, and at an interior optimum𝔼qβ\[z\]=𝔼p\[z\]\\mathbb\{E\}\_\{q\_\{\\beta\}\}\[z\]=\\mathbb\{E\}\_\{p\}\[z\]andH\(q∗\)=H\(p\)\+KL\(p∥q∗\)H\(q^\{\*\}\)=H\(p\)\+\\operatorname\{KL\}\(p\\\|q^\{\*\}\), so the best temperature does not match entropy unless the distributions match exactly\[[25](https://arxiv.org/html/2611.22128#bib.bib25)\]\.
On the full distributions, without top\-pp, the best per\-step temperature removes73%73\\%,53%53\\%, and33%33\\%of GPTQ3’s forward KL at requested temperatures0\.30\.3,0\.60\.6, and1\.01\.0\.
### F\.4Forward\-KL oracle for temperature and top\-pp
Table[36](https://arxiv.org/html/2611.22128#A6.T36)compares the oracle with both forms of TempQuant and their smaller versions\. Per\-step TempQuant’s held\-out forward KL is above global TempQuant’s because its learned cutoff sometimes removes tokens BF16 keeps, which forward KL penalizes heavily; with top\-ppturned off, the two match\. Allowing the oracle any cutoff, down to keeping only the top token, lowers its forward KL by about another 40% at0\.60\.6\.
Policyτ=0\.3\\tau=0\.3τ=0\.6\\tau=0\.6τ=1\.0\\tau=1\.0Uncorrected2\.4091\.4930\.867Global TempQuant \(top\-pponly\)1\.0370\.5480\.364Global TempQuant \(TTonly\)0\.5640\.5130\.474Global TempQuant0\.4510\.3970\.346Oracle per\-step TempQuant \(TTonly\)0\.2880\.2640\.255Oracle per\-step TempQuant0\.2720\.2400\.220Per\-step TempQuant \(TTonly\)0\.7380\.5100\.494Per\-step TempQuant0\.5940\.4700\.484Per\-step TempQuant, top\-ppoff0\.4490\.4000\.344Oracle per\-step TempQuant: share removed \(%\)898475Table 36:GPTQ3 smoothed forward KL to BF16 in nats\. The oracle sees BF16 at every token and picks the best setting from the grid global TempQuant searches\. Per\-step TempQuant, top\-ppoff, keeps the predicted temperature and turns top\-ppoff\. Per\-step values average five training runs, except that row, which averages the two runs evaluated downstream; the two averages differ by under3%3\\%\.For unsmoothed forward KL, the oracle must keep every token BF16 gives positive probability, and keeping more tokens only adds to the loss\. With logits sorted soz1≥⋯≥zVz\_\{1\}\\geq\\cdots\\geq z\_\{V\}andk∗=max\{i:pi\>0\}k^\{\*\}=\\max\\\{i:p\_\{i\}\>0\\\}, the joint problem reduces to minimizing
Lk∗\(β\)=log∑j≤k∗eβzj−β∑ipizi−H\(p\)L\_\{k^\{\*\}\}\(\\beta\)=\\log\\sum\_\{j\\leq k^\{\*\}\}e^\{\\beta z\_\{j\}\}\-\\beta\\sum\_\{i\}p\_\{i\}z\_\{i\}\-H\(p\)over the temperature range and then picking a cutoff that keeps exactly the topk∗k^\{\*\}tokens\. This is again convex inβ\\beta\.
### F\.5The oracle in generation
At every step, the oracle chooses the temperature and top\-ppthat minimize forward KL to BF16 and samples from GPTQ3’s own filtered distribution\. All policies run with the same seeds and code on the same GPU for each question\. On MATH, the oracle is ahead of per\-step TempQuant by21\.921\.9points \[10\.910\.9,32\.832\.8\] and of global TempQuant by25\.025\.0\[14\.114\.1,35\.935\.9\]\. This gap shows how much a per\-step predictor could still gain by better recognizing, from the quantized model alone, when the top token has flipped, which we leave to future work\.
Table 37:GPTQ3 on the first GSM8K and MATH questions atτ=0\.6\\tau=0\.6\(accuracy and responses at the length limit in %\)\. Per\-step TempQuant pools its two training runs; BF16 averages four samples from the original runs\.
### F\.6Limits of correcting token rankings
A positive temperature keeps the logit ranking, and top\-ppkeeps a leading set without reordering it, so a lower\-ranked token can never become the most likely one\. If BF16’s top token has rankrrin the quantized model, any setting gives it probability at most1/r1/r\. GPTQ3 keeps BF16’s most likely token at83%83\\%of steps and MXFP4 at89%89\\%\. When GPTQ3 changes it, BF16’s token is usually second, which happens at10%10\\%of steps, and it falls below fifth at only1\.8%1\.8\\%\. Table[38](https://arxiv.org/html/2611.22128#A6.T38)splits forward KL by these steps at each requested temperature, before and after the oracle\. Table[39](https://arxiv.org/html/2611.22128#A6.T39)splits it further by the rank GPTQ3 gives BF16’s most likely token and, when that rank is 1, by whether GPTQ3 keeps the order of the tokens BF16 keeps after top\-pp\. The largest share of the error sits where BF16’s top token falls to second place\. This is the most common kind of flip, at about10%10\\%of steps, and it holds3737–41%41\\%of the error the oracle leaves at every requested temperature, more than any other group\.
Figure[3](https://arxiv.org/html/2611.22128#S4.F3)uses the training and held\-out steps together atτ=0\.6\\tau=0\.6, and fits the temperature only, with top\-ppoff\.
Top token kept \(83%\)Top token changed \(17%\)Share on changed steps \(%\)τ\\tauGPTQ3OracleGPTQ3OracleGPTQ3Oracle0\.30\.280\.0312\.661\.4590920\.60\.350\.047\.011\.2281871\.00\.310\.053\.551\.027080Table 38:Forward KL per step from GPTQ3 to BF16 on 768 held\-out steps, split by whether GPTQ3 keeps BF16’s most likely token, before and after the per\-step oracle, and the share of the total on steps where it changes\.Steps grouped by GPTQ3 rank of BF16 top tokenSteps\(%\)Raw forwardKL \(%\)Oracleresidual \(%\)Removed byoracle \(%\)Temperature \(τ\\tau\) = 0\.3; oracle removes 89% overallTop token kept, order kept80\.15\.11\.397Top token kept, order changed3\.33\.76\.082BF16 top ranked 2nd9\.944\.441\.290BF16 top ranked 3rd–5th4\.933\.633\.389BF16 top ranked below 5th1\.813\.318\.185Temperature \(τ\\tau\) = 0\.6; oracle removes 84% overallTop token kept, order kept78\.08\.81\.897Top token kept, order changed5\.39\.510\.383BF16 top ranked 2nd9\.930\.739\.779BF16 top ranked 3rd–5th4\.933\.031\.485BF16 top ranked below 5th1\.818\.116\.985Temperature \(τ\\tau\) = 1\.0; oracle removes 75% overallTop token kept, order kept73\.011\.12\.794Top token kept, order changed10\.318\.316\.977BF16 top ranked 2nd9\.928\.337\.367BF16 top ranked 3rd–5th4\.926\.628\.173BF16 top ranked below 5th1\.815\.714\.976Table 39:Where the error left by the per\-step oracle’s temperature and top\-ppsits, for GPTQ3 on 768 held\-out steps, grouped by the rank GPTQ3 gives BF16’s most likely token \(%\)\.
## Appendix GRuntime cost
Table[40](https://arxiv.org/html/2611.22128#A7.T40)measures the cost of each generated token during decoding\. Global TempQuant only changes the two numbers passed to the sampler, so it costs nothing, and turning top\-ppoff makes it slightly faster\. Per\-step TempQuant adds0\.50\.5–22ms, or1818–34%34\\%, per token, and peak memory is the same for all three\. The predictor itself is small; most of this time goes to passes over the full vocabulary to compute its features and apply top\-pp, which our implementation does not optimize and which fused kernels could shorten\. The added time grows with batch size \(Table[40](https://arxiv.org/html/2611.22128#A7.T40)\)\. We time MXFP4; GPTQ3 is served as dequantized BF16 weights, so the predictor adds the same work per token\.
Table 40:Milliseconds per generated token for MXFP4 Qwen3\-4B in vLLM on one AMD Instinct MI350X GPU, with 512 prompt tokens and 1,024 generated tokens, averaged over three repeats\.
## Appendix HUse of AI tools
Generative AI assisted with planning, literature review, code and tests, derivations, writing, and figures\. LLM coding agents were also used extensively to run and monitor experiments, collect model outputs, and process and analyze the resulting data\.
We also tried Sakana AI’s ShinkaEvolve222[https://github\.com/SakanaAI/ShinkaEvolve](https://github.com/SakanaAI/ShinkaEvolve), which evolves code with LLM\-proposed changes, to search for improvements to earlier temperature predictors, scoring each candidate by its held\-out divergence from BF16\. None improved clearly enough over the baseline to be adopted in the final method\.
The authors remain responsible for the experiments, claims, and manuscript\. Automated checks do not replace human verification\.Similar Articles
SoloQ: Calibration-Free Quantization for Diffusion Language Models
SoloQ提出一种免校准的量化框架,将权重和激活映射到归一化旋转基底,实现数据无关的4比特量化,在扩散语言模型上保持精度并最多降低2.61倍峰值内存、加速2.24倍推理。
Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
论文提出 OnQ 流水线,通过 on-policy 蒸馏与候选级 Dynamic KL 恢复低比特量化 LLM 的推理能力,在 Qwen3 系列 2/3 比特量化上取得最优九基准平均成绩,2 比特下 Qwen3-8B 平均提升 10.26 分。
MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
MixQuant proposes an adaptive mixed-precision quantization framework for LLMs that handles variable memory budgets by marginalizing layer distortion over random upstream configurations, outperforming existing methods across multiple models and budgets.
Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models
Introduces Variable Bit-width Quantization (VBQ), a training-time method where each group of 64 weights learns its own bit-width (1,2,4,8) via Gumbel-Softmax relaxation. VBQ discovers a heterogeneous allocation that yields a 'bigger-but-smaller' regime, e.g., a 131M parameter model at 1.82 mean bits beats a 55M FP16 model while using less storage, and a 1.46B model matches a 593M FP16 with ~3.7x less storage.
Post-Training Quantization of Autoregressive Weather Models
A study investigating post-training quantization (PTQ) on pre-trained AI weather emulators (DLWP and FourCastNet) to accelerate autoregressive inference and enable deployment on edge hardware, finding that quantized configurations retain qualitatively meaningful short-range forecasts.