dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats

# dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats
Source: [https://arxiv.org/html/2606.04115](https://arxiv.org/html/2606.04115)
Giuseppe Franco Ian Colbert Pablo Monteagudo\-Lago Felix Marty Nicholas Fraser AMD

###### Abstract

Quantizing large language models \(LLMs\) to low\-precision floating\-point representations is central to efficient deployment, yet applying a single bit\-width uniformly across all layers is sub\-optimal when considering the performance/accuracy trade\-off\. This work introduces dMX, a differentiable mixed\-precision quantization framework for learnable floating\-point bit\-width assignment\. In particular, we study its application for the microscaling floating\-point \(MXFP\) family of data types defined by the Open Compute Project \(OCP\) standard\. The per\-layer bit\-width assignment is formulated as a continuous optimization problem in which each layer’s floating\-point format format is parameterized by a scalar parameter, folding the multi\-variate design space into a single learnable offset\. During training this offset takes continuous values, avoiding sudden oscillations between discrete quantization formats\. A temperature\-based annealing schedule progressively discretizes the learned offsets, ensuring that the final configuration maps to hardware\-compatible MXFP formats without abrupt transitions between training and inference behavior\. A target\-aware regularization term steers the average bit\-width toward a user\-specified budget, serving as a coarse\-grained proxy for inference cost and balancing model quality against deployment efficiency\. We performed experiments on different families of LLMs, such as Llama, Qwen3, and SmolLM2, evaluating perplexity on WikiText\-2 and accuracy on four zero\-shot reasoning benchmarks\. Across these settings, dMX consistently yields Pareto\-dominating models and improves over Kullback\-Leibler \(KL\) divergence\-based layer\-selection heuristics, efficiently navigating trade\-offs between model quality and average bit\-width\.

## 1Introduction

Deploying large language models \(LLMs\) at scale demands substantial reductions in both memory footprint and computational cost, a challenge that grows with each new generation of model architectures\. Quantization is one of the primary tools for achieving this goal: high\-precision parameters are replaced with lower\-precision representations while aiming to preserve model quality\. Full quantization\-aware training \(QAT\) can recover much of the lost accuracy in principle, but it remains prohibitively expensive for modern LLMs\. This practical limitation has motivated the development of more expressive data types that can be used post\-trainingFrantaret al\.\([2022](https://arxiv.org/html/2606.04115#bib.bib55)\); Linet al\.\([2024](https://arxiv.org/html/2606.04115#bib.bib56)\); Xiaoet al\.\([2023](https://arxiv.org/html/2606.04115#bib.bib57)\)\.

![Refer to caption](https://arxiv.org/html/2606.04115v1/images/quantization_grids.png)Figure 1:Comparison of the quantization grids when using continuous values for mantissa and exponent bit\-widths, compared to a discrete E2M1 configuration\.In this context, floating\-point formats such as FP8, FP6, and FP4 offer greater dynamic range than integer formats at the same nominal bit\-widthvan Baalenet al\.\([2023](https://arxiv.org/html/2606.04115#bib.bib51)\)\. Per\-group scaling further improves granularity relative to per\-tensor or per\-channel schemesRouhaniet al\.\([2023](https://arxiv.org/html/2606.04115#bib.bib26)\), and dynamic activation quantization adapts to the statistics of each input at inference time\. The microscaling \(MX\) standard published by the Open Compute Project \(OCP\)Darvish Rouhaniet al\.\([2023](https://arxiv.org/html/2606.04115#bib.bib48)\)formalizes a family of block\-scaled floating\-point formats \(i\.e\., MXFP8, MXFP6, and MXFP4\) that are now supported by recent accelerator architectures\.

Despite these advances, applying a single low\-precision format uniformly across all layers still leads to substantial accuracy loss\. Deploying MXFP4 throughout an entire network, for instance, often degrades model quality beyond acceptable thresholdsEgiazarianet al\.\([2025](https://arxiv.org/html/2606.04115#bib.bib37)\); Sanjeetet al\.\([2026](https://arxiv.org/html/2606.04115#bib.bib1)\)\. This limitation motivates mixed\-precision quantization, in which different layers or operators receive different bit\-widths so that the model can achieve a more favorable balance between accuracy and efficiency\. This approach introduces its own challenge: the bit\-width assignment across layers is a combinatorial optimization problemCohenet al\.\([2024](https://arxiv.org/html/2606.04115#bib.bib60)\); Wuet al\.\([2018](https://arxiv.org/html/2606.04115#bib.bib11)\), and estimating how quantization errors interact across layers is difficultAkbulutet al\.\([2026](https://arxiv.org/html/2606.04115#bib.bib59)\); Ranjan and Savakis \([2025](https://arxiv.org/html/2606.04115#bib.bib58)\)\. Greedy methods typically evaluate the degradation caused by quantizing each layer in isolationDonget al\.\([2019](https://arxiv.org/html/2606.04115#bib.bib6)\), or they approximate cross\-layer interactions only partially through local proxies of sensitivityDonget al\.\([2020](https://arxiv.org/html/2606.04115#bib.bib7)\); Akbulutet al\.\([2026](https://arxiv.org/html/2606.04115#bib.bib59)\)\. Their decisions depend strongly on the chosen sensitivity metric and do not account for the nonlinear accumulation of quantization errors through the network\.

Gradient\-based methods offer a global alternativeWuet al\.\([2018](https://arxiv.org/html/2606.04115#bib.bib11)\); Yang and Jin \([2021](https://arxiv.org/html/2606.04115#bib.bib13)\), as they optimize bit\-width parameters jointly through backpropagation, thereby capturing cross\-layer dependencies that greedy approaches miss\. Learned bit\-width representations have been studied primarily for integer quantizationYang and Jin \([2021](https://arxiv.org/html/2606.04115#bib.bib13)\); Huanget al\.\([2022](https://arxiv.org/html/2606.04115#bib.bib14)\); Yanget al\.\([2021](https://arxiv.org/html/2606.04115#bib.bib15)\); comparatively little work has explored analogous ideas for floating\-point formats\. In the floating\-point setting, optimization must contend with the structure of the data type itself, which involves exponent and mantissa bit\-widths together with related quantities such as the exponent bias\. Current hardware supports only a restricted set of format combinations, typically limited to \(MX\)FP8, \(MX\)FP6, and \(MX\)FP4, with predefined choices for special\-value representation and exponent biasDarvish Rouhaniet al\.\([2023](https://arxiv.org/html/2606.04115#bib.bib48)\)\. These constraints mean that the continuous search space must ultimately collapse to a small discrete set of admissible configurations, making both the parameterization of the search and the discretization mechanism central design decisions\. Although a more thorough analysis of related works will be presented in Sec\.[4](https://arxiv.org/html/2606.04115#S4), to the best of our knowledge, this is the first work that proposes a framework for a gradient\-based bit\-width allocation, focused on the MX format\.

This work introduces a gradient\-based framework for learning per\-layer floating\-point bit\-widths\. An overview of the pipeline is shown in[Figure2](https://arxiv.org/html/2606.04115#S2.F2)\. The main contributions are:

- •A differentiable bit\-width parameterization for floating\-point quantization that enables end\-to\-end gradient\-based optimization of per\-layer precision assignments\.
- •A continuous representation of the floating\-point format of the bit\-width during calibration that produces smoother optimization landscapes compared to discrete alternatives based on rounding and straight\-through estimators \(STEs\)Bengioet al\.\([2013](https://arxiv.org/html/2606.04115#bib.bib38)\)\.
- •A temperature\-based annealing mechanism that progressively discretizes the learned bit\-widths, ensuring convergence to hardware\-compatible formats while minimizing the gap between calibration\-time and inference\-time model behavior\.
- •Support for targeting specific hardware configurations through mixed\-precision quantization for floating\-point quantization\. We focus our experiments on MXFP configurations, in particular on MXFP8/MXFP4 and MXFP6/MXFP4 format pairs\.
- •Compatibility with existing post\-training quantization \(PTQ\) algorithms, enabling integration with gradient\-based weight optimization methods and rotation\-based preprocessing techniquesLiuet al\.\([2024](https://arxiv.org/html/2606.04115#bib.bib36)\)\.

## 2Methodology

![Refer to caption](https://arxiv.org/html/2606.04115v1/x1.png)

Figure 2:Overview of the dMX pipeline\.All blue elements highlight the main contributions of this work\. The pre\-trained LLM \(left\) contains a learned continuous offsetβi\\beta\_\{i\}for each layerii, which parameterizes the bit\-width used in that layer\. During the forward pass these offsets are mapped to discrete format assignmentsβ^=F​\(β,T\)\\hat\{\\beta\}=F\(\\beta,T\)\. A task loss and a user\-defined regularization termℛ\\mathcal\{R\}onβ\\betajointly drive the gradient update \(red loop\), while a temperature annealing schedule progressively sharpens the mapping so that everyβi\\beta\_\{i\}converges to a hardware\-compatible MXFP format \(blue loop\)\.The proposed method rests on three observations\. The first is that bit\-widths need not be integer\-valued during optimization\. The quantization equations remain well defined when the bit\-width is treated as a continuous quantity, and no mathematical constraint forces a discrete bit\-width\.

The second observation follows directly: although continuous bit\-widths are permissible during training, the final assignments must map to discrete, hardware\-supported formats at deployment time\. In the case of floating\-point quantization, this mapping is more restrictive than in the integer setting\. The MX format family supports only a specific set of configurations, each with a fixed partition of bits between exponent and mantissaDarvish Rouhaniet al\.\([2023](https://arxiv.org/html/2606.04115#bib.bib48)\)\.

The third observation is that gradients can be propagated with respect to the bit\-width in the same way as for any other learnable parameter, provided that the bit\-width itself is stored as a floating\-point variable\. This idea has been applied successfully to integer bit\-width learningYang and Jin \([2021](https://arxiv.org/html/2606.04115#bib.bib13)\); Huanget al\.\([2022](https://arxiv.org/html/2606.04115#bib.bib14)\); its extension to floating\-point quantization remains largely unexplored\.

These observations give rise to three design challenges\. The first is how to parameterize the floating\-point representation so that all quantities entering the quantizer remain differentiable with respect to the bit\-width\. The second is how to ensure that the learned bit\-widths converge to hardware\-compatible values by the end of training\. The third is how to constrain the bit\-widths within user\-defined boundaries\. The following subsections address each challenge in turn\.

### 2\.1Continuous floating\-point bit\-width parameterization

The standard formulation of floating\-point quantization represents a floating\-point value as

x\(FP\)=\(−1\)S×2u−b×\(1\+∑i=1mMi⋅2−i\),x^\{\(\\mathrm\{FP\}\)\}=\(\-1\)^\{S\}\\times 2^\{u\-b\}\\times\\left\(1\+\\sum\_\{i=1\}^\{m\}M\_\{i\}\\cdot 2^\{\-i\}\\right\),\(1\)whereS∈\{0,1\}S\\in\\\{0,1\\\}is the sign bit,uuis the unsigned exponent with0≤u<2e0\\leq u<2^\{e\},b=2e−1−1b=2^\{e\-1\}\-1is the exponent bias,eeandmmdenote the number of exponent and mantissa bits respectively, andMi∈\{0,1\}M\_\{i\}\\in\\\{0,1\\\}is theii\-th mantissa bit\. This formulation is adjusted accordingly in case of subnormal values\. The representable range of this format is bounded by

qmax\(FP\)=\(2−2−m\)⋅22e−b−1,qmin\(FP\)=−qmax\(FP\)\.q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}=\(2\-2^\{\-m\}\)\\cdot 2^\{\\,2^\{e\}\-b\-1\},\\qquad q\_\{\\min\}^\{\(\\mathrm\{FP\}\)\}=\-q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\.\(2\)Given a tensor𝐗\\mathbf\{X\}with elementsx∈𝐗x\\in\\mathbf\{X\}and a coarse scales=t/qmax\(FP\)s=t\\,/\\,q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}wheret=max⁡\(\|𝐗\|\)t=\\max\(\|\\mathbf\{X\}\|\), each quantized elementxqx\_\{q\}is obtained by scaling, rounding to the nearest representable value, and clipping:

xq=clip\(𝑠𝑠⌊x/s𝑠𝑠⌉;qmin\(FP\),qmax\(FP\)\),x\_\{q\}=\\operatorname\{clip\}\\\!\\left\(\\mathit\{ss\}\\left\\lfloor\\frac\{x/s\}\{\\mathit\{ss\}\}\\right\\rceil;\\;q\_\{\\min\}^\{\(\\mathrm\{FP\}\)\},\\,q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\\right\),\(3\)where𝑠𝑠\\mathit\{ss\}is a fine\-grained power\-of\-two scale derived from each element’s magnitude and⌊⋅⌉\\lfloor\\cdot\\rceildenotes rounding to the nearest integer\. For a fixed inputxx, the quantized outputxq​\(e,m;x\)x\_\{q\}\(e,m;x\)is differentiable with respect toeeandmmwhen these quantities are treated as real\-valued parameters, since the range bounds and bias vary continuously with them\. The generalization of Eq\.[3](https://arxiv.org/html/2606.04115#S2.E3)to continuous values and the derivation of the bit\-widths is provided in Appendix[A](https://arxiv.org/html/2606.04115#A1), and an example on how the quantization grid changes when using continuous values is shown in[Figure1](https://arxiv.org/html/2606.04115#S1.F1)\.

Although exponent and mantissa bit\-widths could in principle be learned independently, this might produce configurations incompatible with existing hardware\. To avoid this, the format space is simplified by expressing each supported format relative to a shared baseline\. The MXFP8, MXFP6, and MXFP4 representations correspond to E4M3, E2M3, and E2M1, respectivelyDarvish Rouhaniet al\.\([2023](https://arxiv.org/html/2606.04115#bib.bib48)\); the E5M2 configuration for MXFP8 is not considered here, as it is not typically used for the forward passMicikeviciuset al\.\([2022](https://arxiv.org/html/2606.04115#bib.bib25)\)\. We can define a generic MXFP\(β\\beta\) configuration as

E​\(2\+β\)​M​\(1\+β\),E\(2\{\+\}\\beta\)M\(1\{\+\}\\beta\),\(4\)whereβ\\betais a learnable parameter\. Whenβ=2\\beta=2, we obtain the MXFP8 configuration \(i\.e\., E4M3\), whilstβ=0\\beta=0gives us the baseline MXFP4 configuration\. During training,β\\betamay take any continuous value in\[0,2\]\[0,2\], permitting a smooth transition between configurations and making the optimization less sensitive to abrupt changes in the underlying representation\. This can be applied in the context of MXFP8/MXFP4 mixed precision configuration, where botheeandmmare optimized, and for the MXFP6/MXFP4 configuration, where onlymmis optimized\. Finally, tying weights and activations bit\-widths reduces the problem to a single learnable scalar per layer\.

### 2\.2Temperature\-based annealing

A continuous representation during training benefits optimization stability, but inference requires discrete bit\-widths that map to hardware\-supported formats\. The central difficulty lies in managing this transition without introducing instabilities; at the same time, it is important to avoid discontinuities between training\-time and inference\-time behavior\. A naive approach that applies a rounding operator with an STEBengioet al\.\([2013](https://arxiv.org/html/2606.04115#bib.bib38)\)during each forward pass can cause individual layers to oscillate between MXFP4 and MXFP8 throughout training\. Such oscillations create abrupt changes in the loss landscape and generally lead to worse final performance, as the experiments in Section[3\.1](https://arxiv.org/html/2606.04115#S3.SS1)confirm\.

To avoid these oscillations, we propose to smooth the bit\-width offset through a temperature\-regulated sigmoid function\. The design goal is a mapping that behaves approximately as an identity early in training, allowing the optimizer to explore the continuous bit\-width space freely, and that gradually sharpens into a step\-like function toward the end, forcing convergence to one of the two discrete endpoints, matching more closely the inference\-time behaviour\. Our proposed formulation is:

β=F​\(β^,T\)=21\+e−T⋅\(β^−0\.5\),\\beta=F\(\\hat\{\\beta\},T\)=\\frac\{2\}\{1\+e^\{\-T\\cdot\(\\hat\{\\beta\}\-0\.5\)\}\},\(5\)whereβ\\betais the offset parameter used to define the final bit\-width as indicated in Sec\.[2\.1](https://arxiv.org/html/2606.04115#S2.SS1),β^\\hat\{\\beta\}is the raw learned offset, andTTis a temperature coefficient that increases during training\.

WhenTTis small, the sigmoid is nearly linear and the optimizer can adjust bit\-widths with minimal restriction\. As training proceeds,TTincreases and the function sharpens into a step centered atβ=1\\beta=1\. Near the end of training, offsets below 1 are pushed towardβ=0\\beta=0\(MXFP4\), while offsets above 1 are pushed towardβ=2\\beta=2\(MXFP8 or MXFP6, depending on the chosen format range\)\. This annealing scheme ensures that the final precision assignments are compatible with the discrete formats supported by the target hardware, while the gradual transition avoids the abrupt format switches that characterize the rounding\-based alternative\.

### 2\.3Bit\-width regularization

Since higher precision generally reduces quantization error and quantization error may limit how much the model task loss \(e\.g\., cross\-entropy loss\) can be minimised, a regularization term is required to balance model quality against inference efficiency\. We consider two regularization strategies in this work: a simple scaling penalty and a target\-aware penalty\.

##### Simple scaling penalty\.

The first strategy multiplies the current average bit\-width by a constant factorλ\\lambda:

ℛ𝓈=λ⋅b¯current\.\\mathcal\{R\_\{s\}\}=\\lambda\\cdot\\bar\{b\}\_\{\\mathrm\{current\}\}\.\(6\)This form has a straightforward interpretation, with higher values ofλ\\lambdaleading to more aggressive quantization, and vice\-versa\. However, it provides only indirect control over the final model configuration, since it is not possible to predict apriori how a given change inλ\\lambdawill affect the resulting bit\-width allocation\.

##### Target\-aware penalty\.

The second strategy steers optimization toward a specific target bit\-width through the penalty

ℛt=max⁡\(0,λ⋅\(b¯current−b¯target\)\),\\mathcal\{R\}\_\{t\}=\\max\\\!\\left\(0,\\;\\lambda\\cdot\\left\(\\bar\{b\}\_\{\\mathrm\{current\}\}\-\\bar\{b\}\_\{\\mathrm\{target\}\}\\right\)\\right\),\(7\)whereb¯current\\bar\{b\}\_\{\\mathrm\{current\}\}is the current average bit\-width andb¯target\\bar\{b\}\_\{\\mathrm\{target\}\}is the desired target\. When the current average lies below the target, the penalty vanishes and the bit\-width is driven solely by the task loss\. When the current average exceeds the target, the penalty grows in proportion to the gap, providing a direct incentive to reduce precision\. This formulation offers finer control over the resulting configuration\. A practical limitation arises whenb¯current\\bar\{b\}\_\{\\mathrm\{current\}\}andb¯target\\bar\{b\}\_\{\\mathrm\{target\}\}are close: the penalty approaches zero and the gradient signal weakens, making precise convergence to the target dependent on the magnitude ofλ\\lambda\.

Other regularization options could be considered, based on the specific use\-case \(e\.g\., accounting for stricter memory or compute constraints\); we leave this exploration for a future work, while we present the results of the two strategies presented above in Section[3\.1](https://arxiv.org/html/2606.04115#S3.SS1)\.

#### 2\.3\.1Model\-wise representative bit\-width

For both regularization losses, we employ a model\-wise representative bit\-width for the entire model,b¯current\\bar\{b\}\_\{\\mathrm\{current\}\}, for which we propose two formulations\.

##### Simple average\.

The first computes a simple unweighted average of the per\-layer bit\-widths:

b¯w=1N​∑i=1Nbw,i,b¯a=1N​∑i=1Nba,i,b¯current=b¯w\+b¯a2,\\bar\{b\}\_\{w\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}b\_\{w,i\},\\quad\\bar\{b\}\_\{a\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}b\_\{a,i\},\\quad\\bar\{b\}\_\{\\mathrm\{current\}\}=\\frac\{\\bar\{b\}\_\{w\}\+\\bar\{b\}\_\{a\}\}\{2\},\(8\)wherebw,ib\_\{w,i\}andba,ib\_\{a,i\}denote the weight and activation bit\-widths of theii\-th layer, andNNis the total number of quantized layers\.

##### Weighted average\.

The second formulation uses a weighted average in which each layer’s contribution is proportional to the number of elements in its tensor\. Larger layers dominate the overall model footprint, so weighting by size aligns the regularization penalty with actual memory cost:

b¯wwt=∑i=1Nnw,i⋅bw,i∑i=1Nnw,i,b¯awt=∑i=1Nna,i⋅ba,i∑i=1Nna,i,b¯currentwt=b¯wwt\+b¯awt2,\\bar\{b\}\_\{w\}^\{\\,\\text\{wt\}\}=\\frac\{\\sum\_\{i=1\}^\{N\}n\_\{w,i\}\\cdot b\_\{w,i\}\}\{\\sum\_\{i=1\}^\{N\}n\_\{w,i\}\},\\quad\\bar\{b\}\_\{a\}^\{\\,\\text\{wt\}\}=\\frac\{\\sum\_\{i=1\}^\{N\}n\_\{a,i\}\\cdot b\_\{a,i\}\}\{\\sum\_\{i=1\}^\{N\}n\_\{a,i\}\},\\quad\\bar\{b\}\_\{\\mathrm\{current\}\}^\{\\,\\text\{wt\}\}=\\frac\{\\bar\{b\}\_\{w\}^\{\\,\\text\{wt\}\}\+\\bar\{b\}\_\{a\}^\{\\,\\text\{wt\}\}\}\{2\},\(9\)wherenw,in\_\{w,i\}andna,in\_\{a,i\}denote the number of elements in the weight and activation tensors of theii\-th layer, respectively\. The choice between these two formulations affects how the precision budget is distributed: the simple average treats all layers equally, while the weighted average concentrates the penalty on layers with the largest parameter counts\. We compare these approaches in Section[3\.1](https://arxiv.org/html/2606.04115#S3.SS1)\.

## 3Experimental Setting

Experiments are conducted on three language models of comparable scale: Llama 3\.2 1BGrattafioriet al\.\([2024](https://arxiv.org/html/2606.04115#bib.bib39)\), Qwen3 1\.7BYanget al\.\([2025](https://arxiv.org/html/2606.04115#bib.bib40)\), and SmolLM2 1\.7BBen Allalet al\.\([2025](https://arxiv.org/html/2606.04115#bib.bib41)\)\. The training infrastructure we use is similar to the one proposed in SpinQuantLiuet al\.\([2024](https://arxiv.org/html/2606.04115#bib.bib36)\), but we optimize rotations and bit\-widths jointly\. The main difference, alongside all the modifications required to do learned bit\-width assignment, is that the calibration uses 3200 samples drawn from the FineWebPenedoet al\.\([2024](https://arxiv.org/html/2606.04115#bib.bib43)\)dataset and processed with a batch size of 8, yielding 400 optimization steps\. This is a modest increase in training steps compared to the SpinQuant training setup, while still being considerably faster and more convenient than full QAT training\.

We evaluate our approach on perplexity computed on WikiText\-2Merityet al\.\([2017](https://arxiv.org/html/2606.04115#bib.bib42)\), and accuracy averaged over four zero\-shot reasoning benchmarks: ARC\-Challenge and ARC\-EasyClarket al\.\([2018](https://arxiv.org/html/2606.04115#bib.bib44)\), HellaSwagZellerset al\.\([2019](https://arxiv.org/html/2606.04115#bib.bib45)\), and WinoGrandeSakaguchiet al\.\([2020](https://arxiv.org/html/2606.04115#bib.bib46)\), measured with the LightEval frameworkHabibet al\.\([2023](https://arxiv.org/html/2606.04115#bib.bib47)\)\. We quantize all models using BrevitasFrancoet al\.\([2025](https://arxiv.org/html/2606.04115#bib.bib66)\)\.

On the format side, MXFP8/MXFP4 and MXFP6/MXFP4 are evaluated as the two mixed\-precision format configurations\.

We also examine several design choices through controlled ablations\. Tensor\-size\-weighted and simple\-average formulations of the bit\-width regularization \(Section[2\.3](https://arxiv.org/html/2606.04115#S2.SS3)\) are compared, as are the two regularization objectives \(simple scaling versus target\-aware penalty\)\. Finally, dMX is compared against a greedy heuristic for layer selection, in particular KL divergence\-based pre\-selected precision assignments\. Finally, we compare dMX with a discretized variant that applies rounding with an STE during each forward pass \(ROUND\+STE\)\. These results are presented in Appendix[C](https://arxiv.org/html/2606.04115#A3)\. For all the experiments described so far, we use MXFP4 as starting configuration for all layers\.

As baseline, we consider identical quantization configuration \(i\.e\., MXFP4, MXFP6, or MXFP8\) uniformly across all layers\. All baselines use the same calibration setup as the learned mixed\-precision experiments; the only difference is that their bit\-widths remain fixed throughout optimization\.

In all figures, each configuration is plotted against its average bit\-width computed from the per\-layer bit\-widths of both weights and activations\. Unless stated otherwise, average bit\-width refers to the simple average across layers\. Each point corresponds to a different target average bit\-width ranging from4\.14\.1to88\.

##### Hyper\-parameter considerations\.

The proposed formulation introduces several hyper\-parameters\. The temperature schedule of Section[2\.2](https://arxiv.org/html/2606.04115#S2.SS2)requires initial and final values forTT, as well as a ramp profile\. In all experiments, the schedule follows an exponential ramp\. The initial and final values ofTTare chosen to approximate a near\-linear mapping and a near\-step function, respectively\.

A warm\-up phase is also introduced: for a fixed fraction of calibration steps, denotedTratioT\_\{\\mathrm\{ratio\}\}, the temperature remains constant so that the mapping stays approximately linear\. After this phase, the exponential schedule begins and the bit\-widths are progressively discretized\. Experiments with different values ofTratioT\_\{\\mathrm\{ratio\}\}suggest that the method is fairly robust to this choice\.

Across several models, effective settings were found with relatively modest tuning effort\. More careful hyper\-parameter search may yield further improvements\. We defer that exploration to future work\. The details about the hyper\-parameters for our experiments are detailed in Appendix[B](https://arxiv.org/html/2606.04115#A2)\.

### 3\.1Main Results

We first present the results associated to the MXFP8 and MXFP6 mixed precision configurations\. In the following tables we compare against the float baseline, and the relevant MXFP configurations\.

For the sake of brevity, we only report results with target bit\-width equal to4\.5,5,64\.5,5,6, and88for MXFP8/MXFP4 mixed precision configuration, and4\.5,54\.5,5, and66for MXFP6/MXFP4 configurations\. We present the plot with all the different target bit\-widths in the Appendix[C](https://arxiv.org/html/2606.04115#A3)\.

Tables[1](https://arxiv.org/html/2606.04115#S3.T1)\-[2](https://arxiv.org/html/2606.04115#S3.T2)show how even small increase in average bit\-width can lead to consistent improvements in terms of both perplexity and zero\-shot accuracy\. Moreover, we are always able to closely match the desired target bit\-width, with the main exception being the highest bit\-width target; we believe this could be fixed with a more extensive hyper\-parameter search \(e\.g\., adjusting the learning rate\)\.

Table 1:We compare MXFP8/MXFP4 mixed precision versus three baselines: the base BF16 model, and quantization configurations where all layers are quantized to MXFP8 and MXFP4\. When using dMX, we report four different target bit\-widths, using the average bit\-width across all layers\.ModelTypePPLZero\-ShotLlama 3\.2 1BFloatBF168\.9451\.53MXFP44\.011\.6847\.82MXFP88\.09\.1551\.39dMX4\.5711\.0248\.115\.1110\.6049\.276\.049\.8349\.618\.09\.1951\.46SmolLM2 1\.7BFloatBF167\.6159\.52MXFP44\.010\.2853\.42MXFP88\.07\.8658\.69dMX4\.539\.3354\.415\.098\.9255\.366\.068\.2657\.287\.297\.9657\.90Qwen3 1\.7BFloatBF1615\.7454\.12MXFP44\.012\.5751\.15MXFP88\.010\.5054\.08dMX4\.5311\.9152\.205\.0211\.5851\.625\.9011\.0253\.747\.6510\.4754\.09Table 2:We compare MXFP6/MXFP4 mixed precision versus three baselines: the base BF16 model, and quantization configurations where all layers are quantized to MXFP6 and MXFP4\. When using dMX, we report three different target bit\-widths, using the average bit\-width across all layers\.ModelTypePPLZero\-ShotLlama 3\.2 1BFloatBF168\.9451\.53MXFP44\.011\.6847\.82MXFP66\.09\.1651\.39dMX4\.5210\.7649\.435\.0210\.0250\.265\.989\.4850\.86SmolLM2 1\.7BFloatBF167\.6159\.52MXFP44\.010\.2853\.42MXFP66\.07\.8158\.18dMX4\.529\.7856\.405\.068\.5057\.525\.548\.2658\.25Qwen3 1\.7BFloatBF1615\.7454\.12MXFP44\.012\.5751\.15MXFP66\.010\.6754\.05dMX4\.5211\.5352\.905\.0311\.0154\.145\.6910\.6454\.86
### 3\.2Ablation Studies

For the ablation studies, we present the plots with all the different target bit\-widths to have a better understanding of the various comparisons\.

##### Regularization function comparison\.

![Refer to caption](https://arxiv.org/html/2606.04115v1/x2.png)

Figure 3:Comparison of the target\-aware penalty and the simple scaling penalty for MXFP8–MXFP4 mixed\-precision quantization\. Red: target\-aware penalty \(ℛt\\mathcal\{R\}\_\{t\}\)\. Cyan: simple scaling penalty \(ℛs\\mathcal\{R\}\_\{s\}\)\. Black diamonds: homogeneous MXFP baselines\. Top row: perplexity \(lower is better\)\. Bottom row: average zero\-shot accuracy \(higher is better\)\. The target\-aware penalty produces a smoother spread of operating points across the bit\-width range\.[Figure3](https://arxiv.org/html/2606.04115#S3.F3)compares the two regularization objectives described in Section[2\.3](https://arxiv.org/html/2606.04115#S2.SS3)\. The target\-aware penalty \(ℛt\\mathcal\{R\}\_\{t\}\) produces a denser and more evenly distributed set of operating points across the bit\-width range, particularly for Llama 3\.2 1B and SmolLM2 1\.7B\. The simple scaling penalty \(ℛs\\mathcal\{R\}\_\{s\}\) tends to cluster configurations at the extremes of the bit\-width range\.

As expected the target\-aware formulation offers more predictable control over the final bit\-width allocation, and the extra constraints on the target bit\-width does not to negatively impact the training\.

For the rest of the experiments, we use the target\-aware cost function, since it allows a better understanding of the transition between bit\-width configurations\.

##### Cost function comparison\.

As described in Section[2\.3](https://arxiv.org/html/2606.04115#S2.SS3), we investigate two cost function strategies, a simple average or a tensor\-size\-weighted average of the per\-layer bit\-widths\. Under the weighted formulation, larger layers receive a stronger penalty, which is expected to push the optimizer toward assigning them lower precision\.

[Figure4](https://arxiv.org/html/2606.04115#S3.F4)compares the two strategies\. In all panels thexx\-axis reports the same weighted average bit\-width, defined as the average bit\-width across layers, weighted by the weight tensor size in each layer\. The results for the two loss functions results largely overlap, showing how the user can easily adapt the method to match their use cases\.

![Refer to caption](https://arxiv.org/html/2606.04115v1/x3.png)

Figure 4:Comparison of simple\-average and tensor\-size\-weighted bit\-width regularization for MXFP8/MXFP4 mixed\-precision quantization\. Red: simple average\. Cyan: weighted average\. Black diamonds: homogeneous MXFP baselines\. Top row: perplexity \(lower is better\)\. Bottom row: average zero\-shot accuracy \(higher is better\)\. The simple\-average formulation tends to reach lower perplexity and spans a wider range of operating points\.
##### Learned vs\. pre\-selected layer precision\.

An alternative to gradient\-based bit\-width learning is sensitivity\-based selection\. Each layer is quantized to lower precision in isolation, and the KL divergence between the output distributions of the full\-precision model and the single\-layer\-quantized model is measured\. Layers with larger KL divergence are treated as more sensitive and assigned higher precision first\.[Figure5](https://arxiv.org/html/2606.04115#S3.F5)presents the comparison for Llama 3\.2 1B\.

![Refer to caption](https://arxiv.org/html/2606.04115v1/x4.png)

Figure 5:Learned bit\-width optimization vs\. KL divergence\-based pre\-selected layer precision for Llama 3\.2 1B\. Red: learned\. Cyan: pre\-selected\. Black diamonds: homogeneous MXFP baselines\. Thexx\-axis reports the weighted average bit\-width\. Left: perplexity \(lower is better\)\. Right: average zero\-shot accuracy \(higher is better\)\. The learned approach achieves lower perplexity and higher accuracy at intermediate bit\-widths\.At the lowest average bit\-widths, the learned and pre\-selected approaches perform similarly because nearly all layers are forced to MXFP4\. The few layers that are quantized to MXFP8 have a high sensitivity to quantization, which can also be identified through KL divergence\.

At intermediate bit\-widths, roughly between 5\.0 and 7\.0, the sensitivity approximation becomes less reliable, causing the pre\-selected curve to remain relatively flat in this region, while the learned assignment achieves noticeably lower perplexity\. The same trend appears in zero\-shot accuracy, where the learned approach outperforms the pre\-selected baseline across most of the intermediate range\. At high bit\-widths both methods converge toward the MXFP8 baseline\. This advantage at intermediate operating points is consistent with the expectation that end\-to\-end optimization captures cross\-layer dependencies that isolated sensitivity estimates miss\.

## 4Related Work

##### Layer\-wise bit\-width selection\.

Determining an optimal bit\-width allocation across layers is a combinatorial optimization problem whose search space grows exponentially with the number of layers and candidate precisionsWuet al\.\([2018](https://arxiv.org/html/2606.04115#bib.bib11)\); Cohenet al\.\([2024](https://arxiv.org/html/2606.04115#bib.bib60)\)\. Early sensitivity\-based approaches estimate per\-layer importance through proxies such as the layer’s Hessian spectrum in HAWQDonget al\.\([2019](https://arxiv.org/html/2606.04115#bib.bib6)\)or reinforcement\-learning\-based search over hardware\-oriented objectives in HAQWanget al\.\([2019](https://arxiv.org/html/2606.04115#bib.bib9)\)\. Subsequent refinements have replaced or extended these proxies: HAWQ\-V3Yaoet al\.\([2021](https://arxiv.org/html/2606.04115#bib.bib8)\)casts the allocation as an integer linear program, and Chen et al\.Chenet al\.\([2021](https://arxiv.org/html/2606.04115#bib.bib10)\)formulate it as a constrained knapsack problem\. In the context of LLMs, APTQGuanet al\.\([2024](https://arxiv.org/html/2606.04115#bib.bib20)\)combines Hessian\-trace sensitivity with attention\-aware metrics, ResQSaxenaet al\.\([2024](https://arxiv.org/html/2606.04115#bib.bib23)\)separates principal components through PCA to isolate outlier structure\. A common finding across these works is that mixed\-precision quantization is often essential for pushing below four bits per parameter without unacceptable degradation, yet they focus almost entirely on integer formats\.

##### Gradient\-based bit\-width learning\.

Learning bit\-widths through gradient descent has been previously explored for integer quantizationWuet al\.\([2018](https://arxiv.org/html/2606.04115#bib.bib11)\); Cai and Vasconcelos \([2020](https://arxiv.org/html/2606.04115#bib.bib12)\)\. FracBitsYang and Jin \([2021](https://arxiv.org/html/2606.04115#bib.bib13)\)uses linear interpolation between adjacent integer bit\-widths, while requiring a two\-stage process for effective mixed\-precision deployment\. while SDQHuanget al\.\([2022](https://arxiv.org/html/2606.04115#bib.bib14)\)employs auxiliary learned parameters that represents the probability of selecting a specific bit\-width, with no continuous transition between values\. BSQYanget al\.\([2021](https://arxiv.org/html/2606.04115#bib.bib15)\)recasts mixed\-precision assignment as bit\-level sparsity, and Q\-ViTLiet al\.\([2022](https://arxiv.org/html/2606.04115#bib.bib16)\)extends differentiable precision search to vision transformers\. On the discrete\-selection side, HMQHabiet al\.\([2020](https://arxiv.org/html/2606.04115#bib.bib17)\)relies on a Gumbel\-Softmax relaxationJanget al\.\([2016](https://arxiv.org/html/2606.04115#bib.bib5)\); Maddisonet al\.\([2017](https://arxiv.org/html/2606.04115#bib.bib53)\), Bayesian Bitsvan Baalenet al\.\([2020](https://arxiv.org/html/2606.04115#bib.bib18)\)unifies quantization and pruning through stochastic gates, and Uhlich et al\.Uhlichet al\.\([2020](https://arxiv.org/html/2606.04115#bib.bib19)\)demonstrate that the parameterization of the quantizer itself significantly affects optimization stability\. All of these methods target integer formats\. Floating\-point and MX\-based formats are becoming the more prevalent for LLM deployment on recent hardwareRouhaniet al\.\([2023](https://arxiv.org/html/2606.04115#bib.bib26)\); Micikeviciuset al\.\([2022](https://arxiv.org/html/2606.04115#bib.bib25)\), which motivates extending gradient\-based techniques to this setting\.

##### Floating\-point quantization\.

Sensitivity\-based approaches could in principle be extended to floating\-point formats, For example, MicroMixLiuet al\.\([2025](https://arxiv.org/html/2606.04115#bib.bib32)\)performs channel\-wise assignment across MX formats, using an heuristic approach to perform precision assignment\. Gradient\-based methods are harder to transfer directly because of the floating\-point representation\. To the best of current knowledge, no prior work has addressed this problem through a gradient\-based method that learns floating\-point format selection while using a continuous parameterization to interpolate smoothly between hardware\-supported formats during calibration\.

## 5Conclusion and Future Works

This work presented dMX, a differentiable framework for learning per\-layer floating\-point bit\-widths, focusing on the MX format family\. The method parameterizes each layer’s precision through a single continuous offset that interpolates between different hardware\-supported MXFP formats, and uses a temperature\-based annealing schedule to progressively discretize the learned offsets\.

Our experiments on Llama 3\.2 1B, Qwen3 1\.7B, and SmolLM2 1\.7B demonstrated that learned mixed\-precision assignments consistently outperform both homogeneous MXFP baselines and KL divergence\-based heuristics, particularly at intermediate bit\-widths where the precision budget must be allocated carefully across layers\.

Ablation studies examined the roles of initialization strategy, format ranges, and regularization formulations, providing guidance for practical deployment\.

Several directions remain open for future work\. The current evaluation is limited to models in the 1–2 billion parameter range; scaling to larger architectures and to mixture\-of\-experts models, where layer sensitivities can vary more dramaticallyHuanget al\.\([2025](https://arxiv.org/html/2606.04115#bib.bib52)\), is a natural next step\. A more systematic hyper\-parameter study, including alternative temperature schedules and regularization forms, may also improve robustness across a wider range of models and deployment targets\.

Although we have investigated this approach in the context of gradient\-based PTQ, dMX lends itself to applications in other domains, such as parameter\-efficient fine tuning \(PEFT\), or QAT setups\.

## References

- \[1\]M\. E\. Akbulut, H\. H\. Y\. Shalby, F\. Pittorino, and M\. Roveri\(2026\)InfoQ: mixed\-precision quantization via global information flow\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p3.1)\.
- \[2\]L\. Ben Allal, A\. Lozhkov, E\. Bakouch, G\. Martín Blázquez, G\. Penedo, L\. Tunstall, A\. Marafioti, H\. Kydlíček, A\. Piqueres Lajarín, V\. Srivastav,et al\.\(2025\)SmolLM2: when smol goes big – data\-centric training of a small language model\.arXiv preprint arXiv:2502\.02737\.Cited by:[§3](https://arxiv.org/html/2606.04115#S3.p1.1)\.
- \[3\]Y\. Bengio, N\. Léonard, and A\. Courville\(2013\)Estimating or propagating gradients through stochastic neurons for conditional computation\.arXiv preprint arXiv:1308\.3432\.Cited by:[§A\.1](https://arxiv.org/html/2606.04115#A1.SS1.SSS0.Px1.p1.5),[Appendix C](https://arxiv.org/html/2606.04115#A3.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2606.04115#S1.I1.i2.p1.1),[§2\.2](https://arxiv.org/html/2606.04115#S2.SS2.p1.1)\.
- \[4\]Z\. Cai and N\. Vasconcelos\(2020\)Rethinking differentiable search for mixed\-precision neural networks\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 2346–2355\.Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[5\]W\. Chen, P\. Wang, and J\. Cheng\(2021\)Towards mixed\-precision quantization of neural networks via constrained optimization\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 5350–5359\.Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px1.p1.1)\.
- \[6\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§3](https://arxiv.org/html/2606.04115#S3.p2.1)\.
- \[7\]N\. Cohen, O\. Joglekar, D\. Di Castro, V\. Tchuiev, S\. Kozlovsky, and M\. Moshkovitz\(2024\)Gradient\-free training of quantized neural networks\.arXiv preprint arXiv:2410\.09734\.Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p3.1),[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px1.p1.1)\.
- \[8\]B\. Darvish Rouhani, N\. Garegrat, T\. Savell, A\. More, K\. Han, R\. Zhao, M\. Hall, J\. Klar, E\. Chung, Y\. Yu, M\. Schulte, R\. Wittig, I\. Bratt, N\. Stephens, J\. Milanovic, J\. Brothers, P\. Dubey, M\. Cornea, A\. Heinecke, A\. Rodriguez, M\. Langhammer, S\. Deng, M\. Naumov, P\. Micikevicius, M\. Siu, and C\. Verrilli\(2023\-09\)OCP microscaling formats \(MX\) specification\.Technical reportOpen Compute Project\.Note:Version 1\.0External Links:[Link](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf)Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p2.1),[§1](https://arxiv.org/html/2606.04115#S1.p4.1),[§2\.1](https://arxiv.org/html/2606.04115#S2.SS1.p2.1),[§2](https://arxiv.org/html/2606.04115#S2.p2.1)\.
- \[9\]Z\. Dong, Z\. Yao, Y\. Cai, D\. Arfeen, A\. Gholami, M\. W\. Mahoney, and K\. Keutzer\(2020\)HAWQ\-V2: hessian aware trace\-weighted quantization of neural networks\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p3.1)\.
- \[10\]Z\. Dong, Z\. Yao, A\. Gholami, M\. W\. Mahoney, and K\. Keutzer\(2019\)HAWQ: hessian AWare quantization of neural networks with mixed\-precision\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 293–302\.Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p3.1),[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px1.p1.1)\.
- \[11\]V\. Egiazarian, R\. L\. Castro, D\. Kuznedelev, A\. Panferov, E\. Kurtic, S\. Pandit, A\. Marques, M\. Kurtz, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\(2025\)Bridging the gap between promise and performance for microscaling fp4 quantization\.External Links:2509\.23202,[Link](https://arxiv.org/abs/2509.23202)Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p3.1)\.
- \[12\]Xilinx/brevitasExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.3333552),[Link](https://doi.org/10.5281/zenodo.3333552)Cited by:[§3](https://arxiv.org/html/2606.04115#S3.p2.1)\.
- \[13\]E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\(2022\)GPTQ: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p1.1)\.
- \[14\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§3](https://arxiv.org/html/2606.04115#S3.p1.1)\.
- \[15\]Z\. Guan, H\. Huang, Y\. Su, H\. Huang, N\. Wong, and H\. Yu\(2024\)APTQ: attention\-aware post\-training mixed\-precision quantization for large language models\.InProceedings of the 61st IEEE/ACM Design Automation Conference,Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px1.p1.1)\.
- \[16\]H\. V\. Habi, R\. H\. Jennings, and A\. Netzer\(2020\)HMQ: hardware friendly mixed precision quantization block for CNNs\.InComputer Vision – ECCV 2020,pp\. 448–463\.Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[17\]N\. Habib, C\. Fourrier, H\. Kydlíček, T\. Wolf, and L\. Tunstall\(2023\)LightEval: a lightweight framework for LLM evaluation\.External Links:[Link](https://github.com/huggingface/lighteval)Cited by:[§3](https://arxiv.org/html/2606.04115#S3.p2.1)\.
- \[18\]W\. Huang, Y\. Liao, J\. Liu, R\. He, H\. Tan, S\. Zhang, H\. Li, S\. Liu, and X\. Qi\(2025\)Mixture compressor for mixture\-of\-experts LLMs gains more\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=hheFYjOsWO)Cited by:[§5](https://arxiv.org/html/2606.04115#S5.p4.1)\.
- \[19\]X\. Huang, Z\. Shen, S\. Li, Z\. Liu, X\. Hu, J\. Wicaksana, E\. Xing, and K\. Cheng\(2022\)SDQ: stochastic differentiable quantization with mixed precision\.InProceedings of the 39th International Conference on Machine Learning,pp\. 9295–9309\.Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p4.1),[§2](https://arxiv.org/html/2606.04115#S2.p3.1),[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[20\]IEEE\(2019\)IEEE standard for floating\-point arithmetic\.IEEE Std 754\-2019 \(Revision of IEEE 754\-2008\)\(\),pp\. 1–84\.External Links:[Document](https://dx.doi.org/10.1109/IEEESTD.2019.8766229)Cited by:[Appendix A](https://arxiv.org/html/2606.04115#A1.SS0.SSS0.Px2.p1.14)\.
- \[21\]S\. R\. Jain, A\. Gural, M\. Wu, and C\. H\. Dick\(2020\)Trained quantization thresholds for accurate and efficient fixed\-point inference of deep neural networks\.InProceedings of the 3rd Machine Learning and Systems \(MLSys\) Conference,Cited by:[§A\.1](https://arxiv.org/html/2606.04115#A1.SS1.SSS0.Px1.p1.5),[§A\.1](https://arxiv.org/html/2606.04115#A1.SS1.SSS0.Px3.p1.12)\.
- \[22\]E\. Jang, S\. Gu, and B\. Poole\(2016\)Categorical reparameterization with gumbel\-softmax\.arXiv preprint arXiv:1611\.01144\.Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[23\]Z\. Li, T\. Yang, P\. Wang, and J\. Cheng\(2022\)Q\-ViT: fully differentiable quantization for vision transformer\.arXiv preprint arXiv:2201\.07703\.Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[24\]J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han\(2024\)AWQ: activation\-aware weight quantization for on\-device LLM compression and acceleration\.InProceedings of Machine Learning and Systems,Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p1.1)\.
- \[25\]W\. Liu, H\. Meng, Y\. Luo, Y\. Zhao, P\. Zhang, and X\. Ma\(2025\)Micromix: efficient mixed\-precision quantization with microscaling formats for large language models\.arXiv preprint arXiv:2508\.02343\.Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px3.p1.1)\.
- \[26\]Z\. Liu, C\. Zhao, I\. Fedorov, B\. Soran, D\. Choudhary, R\. Krishnamoorthi, V\. Chandra, Y\. Tian, and T\. Blankevoort\(2024\)SpinQuant: llm quantization with learned rotations\.arXiv preprint arXiv:2405\.16406\.Cited by:[§B\.3](https://arxiv.org/html/2606.04115#A2.SS3.p1.3),[5th item](https://arxiv.org/html/2606.04115#S1.I1.i5.p1.1),[§3](https://arxiv.org/html/2606.04115#S3.p1.1)\.
- \[27\]C\. J\. Maddison, A\. Mnih, and Y\. W\. Teh\(2017\)The concrete distribution: a continuous relaxation of discrete random variables\.InProceedings of the International Conference on Learning Representations,Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[28\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\(2017\)Pointer sentinel mixture models\.InProceedings of the International Conference on Learning Representations,Cited by:[§3](https://arxiv.org/html/2606.04115#S3.p2.1)\.
- \[29\]P\. Micikevicius, D\. Stosic, N\. Burgess, M\. Cornea, P\. Dubey, R\. Grisenthwaite, S\. Ha, A\. Heinecke, P\. Judd, J\. Kamalu, N\. Mellempudi, S\. Oberman, M\. Shoeybi, M\. Siu, and H\. Wu\(2022\)FP8 formats for deep learning\.arXiv preprint arXiv:2209\.05433\.Cited by:[§2\.1](https://arxiv.org/html/2606.04115#S2.SS1.p2.1),[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[30\]A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Köpf, E\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. Chintala\(2019\)PyTorch: an imperative style, high\-performance deep learning library\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§A\.1](https://arxiv.org/html/2606.04115#A1.SS1.SSS0.Px6.p1.12)\.
- \[31\]G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, and T\. Wolf\(2024\)The FineWeb datasets: decanting the web for the finest text data at scale\.arXiv preprint arXiv:2406\.17557\.Cited by:[§3](https://arxiv.org/html/2606.04115#S3.p1.1)\.
- \[32\]N\. Ranjan and A\. Savakis\(2025\)Mix\-QSAM: mixed\-precision quantization of the segment anything model\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p3.1)\.
- \[33\]B\. D\. Rouhani, R\. Zhao, A\. More, M\. Hall, A\. Khodamoradi, S\. Deng, D\. Choudhary, M\. Cornea, E\. Dellinger, K\. Denolf,et al\.\(2023\)Microscaling data formats for deep learning\.arXiv preprint arXiv:2310\.10537\.Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p2.1),[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[34\]K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. Choi\(2020\)WinoGrande: an adversarial Winograd schema challenge at scale\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 8732–8740\.Cited by:[§3](https://arxiv.org/html/2606.04115#S3.p2.1)\.
- \[35\]S\. Sanjeet, I\. Colbert, P\. Monteagudo\-Lago, G\. Franco, Y\. Umuroglu, and N\. J\. Fraser\(2026\)MixQuant: pushing the limits of block rotations in post\-training quantization\.arXiv preprint arXiv:2601\.22347\.Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p3.1)\.
- \[36\]U\. Saxena, S\. Sharify, K\. Roy, and X\. Wang\(2024\)ResQ: mixed\-precision quantization of large language models with low\-rank residuals\.arXiv preprint arXiv:2412\.14363\.Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px1.p1.1)\.
- \[37\]S\. Uhlich, L\. Mauch, F\. Cardinaux, K\. Yoshiyama, J\. A\. Garcia, S\. Tiedemann, T\. Kemp, and A\. Nakamura\(2020\)Mixed precision DNNs: all you need is a good parametrization\.InProceedings of the International Conference on Learning Representations,Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[38\]M\. van Baalen, A\. Kuzmin, S\. S\. Nair, Y\. Ren, E\. Mahurin, C\. Patel, S\. Subramanian, S\. Lee, M\. Nagel, J\. Soriaga, and T\. Blankevoort\(2023\)FP8 versus INT8 for efficient deep learning inference\.arXiv preprint arXiv:2303\.17951\.Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p2.1)\.
- \[39\]M\. van Baalen, C\. Louizos, M\. Nagel, R\. A\. Amjad, Y\. Wang, T\. Blankevoort, and M\. Welling\(2020\)Bayesian bits: unifying quantization and pruning\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[40\]K\. Wang, Z\. Liu, Y\. Lin, J\. Lin, and S\. Han\(2019\)HAQ: hardware\-aware automated quantization with mixed precision\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 8612–8620\.Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px1.p1.1)\.
- \[41\]B\. Wu, Y\. Wang, P\. Zhang, Y\. Tian, P\. Vajda, and K\. Keutzer\(2018\)Mixed precision quantization of ConvNets via differentiable neural architecture search\.arXiv preprint arXiv:1812\.00090\.Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p3.1),[§1](https://arxiv.org/html/2606.04115#S1.p4.1),[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[42\]G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han\(2023\)SmoothQuant: accurate and efficient post\-training quantization for large language models\.InProceedings of the 40th International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p1.1)\.
- \[43\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3](https://arxiv.org/html/2606.04115#S3.p1.1)\.
- \[44\]H\. Yang, L\. Duan, Y\. Chen, and H\. Li\(2021\)BSQ: exploring bit\-level sparsity for mixed\-precision neural network quantization\.InProceedings of the International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p4.1),[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[45\]L\. Yang and Q\. Jin\(2021\)FracBits: mixed precision quantization via fractional bit\-widths\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 10612–10620\.Cited by:[§1](https://arxiv.org/html/2606.04115#S1.p4.1),[§2](https://arxiv.org/html/2606.04115#S2.p3.1),[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px2.p1.1)\.
- \[46\]Z\. Yao, Z\. Dong, Z\. Zheng, A\. Gholami, J\. Yu, E\. Tan, L\. Wang, Q\. Huang, Y\. Wang, M\. W\. Mahoney, and K\. Keutzer\(2021\)HAWQ\-V3: dyadic neural network quantization\.InProceedings of the 38th International Conference on Machine Learning,pp\. 11875–11886\.Cited by:[§4](https://arxiv.org/html/2606.04115#S4.SS0.SSS0.Px1.p1.1)\.
- \[47\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Cited by:[§3](https://arxiv.org/html/2606.04115#S3.p2.1)\.

## Appendix AContinuous and Differentiable Float Conversion Function

This appendix specifies how𝑠𝑠\\mathit\{ss\}in Eq\.[3](https://arxiv.org/html/2606.04115#S2.E3)is computed from each element’s magnitude, mantissa bit\-widthmm, and exponent biasbb\. Lettingx~=x/s\\tilde\{x\}=x/sdenote the magnitude\-scaled element, Eqs\.[2](https://arxiv.org/html/2606.04115#S2.E2)and[3](https://arxiv.org/html/2606.04115#S2.E3)are reproduced here for reference:

qmax\(FP\)=\(2−2−m\)⋅22e−b−1,qmin\(FP\)=−qmax\(FP\)\.q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}=\(2\-2^\{\-m\}\)\\cdot 2^\{\\,2^\{e\}\-b\-1\},\\qquad q\_\{\\min\}^\{\(\\mathrm\{FP\}\)\}=\-q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\.\(10\)
xq=clip\(𝑠𝑠\(x~\)⌊x~𝑠𝑠​\(x~\)⌉;qmin\(FP\),qmax\(FP\)\)x\_\{q\}=\\operatorname\{clip\}\\\!\\left\(\\mathit\{ss\}\(\\tilde\{x\}\)\\left\\lfloor\\frac\{\\tilde\{x\}\}\{\\mathit\{ss\}\(\\tilde\{x\}\)\}\\right\\rceil;\\;q\_\{\\min\}^\{\(\\mathrm\{FP\}\)\},\\,q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\\right\)\(11\)
wheressis a coarse scale\.

The purpose of𝑠𝑠\\mathit\{ss\}is to represent the mantissa part of the floating\-point value as an integer, permitting quantization via a standard rounding operator\. The resulting expression is differentiable inmmandeeeverywhere except in the case of rounding operations\. These are handled by straight\-through estimator, formally defined in Section[A\.1](https://arxiv.org/html/2606.04115#A1.SS1)\.

##### Inputs and assumptions\.

Recallx~\\tilde\{x\}from above, with mantissa bit\-widthmm, exponent bit\-widthee, and exponent biasb=2e−1−1b=2^\{e\-1\}\-1\.

The subnormal exponent floor is

emin=1−b−m,e\_\{\\min\}\\;=\\;1\-b\-m,\(12\)the exponent of the unit in the last place \(ULP\) is defined as

η​\(x~\)=max⁡\(⌊log2⁡\(\|x~\|\)⌋−m,emin\),\\eta\(\\tilde\{x\}\)\\;=\\;\\max\\\!\\Bigl\(\\bigl\\lfloor\\log\_\{2\}\(\|\\tilde\{x\}\|\)\\bigr\\rfloor\-m,\\;e\_\{\\min\}\\Bigr\),\(13\)and the per\-element power\-of\-two scale is

𝑠𝑠​\(x~\)=2η​\(x~\)\.\\mathit\{ss\}\(\\tilde\{x\}\)\\;=\\;2^\{\\,\\eta\(\\tilde\{x\}\)\}\.\(14\)
The sign bit and the zero case \(x=0x=0\) are excluded for brevity;

##### Interpretation\.

For nonzerox~\\tilde\{x\}, the quantity⌊log2⁡x~⌋\\lfloor\\log\_\{2\}\\tilde\{x\}\\rflooris the binary exponent ofx~\\tilde\{x\}, so⌊log2⁡x~⌋−m\\lfloor\\log\_\{2\}\\tilde\{x\}\\rfloor\-mis the exponent of the ULP of a normalized floating\-point number withmmmantissa bits at that magnitude\. Clamping the exponent below byemine\_\{\\min\}implements the IEEE\-754\-style\[[20](https://arxiv.org/html/2606.04115#bib.bib49)\]*subnormal*regime: once the true exponent falls below1−b1\-b, the ULP stays fixed at2emin2^\{e\_\{\\min\}\}and mantissa precision degrades toward zero\. The mantissa bit\-widthmmenters𝑠𝑠\\mathit\{ss\}through bothη​\(x~\)\\eta\(\\tilde\{x\}\)andemine\_\{\\min\}, while the exponent biasbbenters throughemine\_\{\\min\}; these are the entry points for the gradient analysis in Section[A\.1](https://arxiv.org/html/2606.04115#A1.SS1)\.

##### From discrete to continuous\.

At deployment, hardware imposes the constraintm∈ℤ\+m\\in\\mathbb\{Z\}^\{\+\}ande∈ℤ\+e\\in\\mathbb\{Z\}^\{\+\}\. However, Eqs\.[10](https://arxiv.org/html/2606.04115#A1.E10)and[11](https://arxiv.org/html/2606.04115#A1.E11)remain well defined form,e∈ℝ\+m,e\\in\\mathbb\{R\}^\{\+\}, and no mathematical constraint forces integer values during training\. Accordingly,mmandeeare treated as real\-valued parameters throughout calibration, subject to the range constraints described in Section[2\.1](https://arxiv.org/html/2606.04115#S2.SS1)\.

##### Continuous floating\-point interpretation\.

The output of the continuous floating\-point conversion function exhibits the following properties:

- •It is a non\-uniform*staircase*function whose distinct quantization levels need not be powers of two\.
- •The number of distinct staircase step sizes, determined by changes in the exponent, likewise need not align with powers of two\.

Asmmincreases, the number of quantization levels within each step\-size increases; aseeincreases, the number of distinct step sizes increases\. See[Figure1](https://arxiv.org/html/2606.04115#S1.F1)for an example grid and panel[6\(a\)](https://arxiv.org/html/2606.04115#A1.F6.sf1)of[Figure6](https://arxiv.org/html/2606.04115#A1.F6)for the forward staircase ate=2e=2,m=2m=2\.

### A\.1Gradients via the straight\-through estimator

We now derive the gradients of the unscaled inner quantizerQ​\(x~\)=xqQ\(\\tilde\{x\}\)=x\_\{q\}with respect to the mantissa bit\-widthmm, the exponent bit\-widthee, and the shared offsetβ\\betathat parameterizes both viam=1\+βm=1\+\\beta,e=2\+βe=2\+\\beta\(Section[2\.1](https://arxiv.org/html/2606.04115#S2.SS1)\)\. For this analysis, we considerssto be constant inmmandee, so the bit\-width gradients ofxqx\_\{q\}and ofQQdiffer only by the outersign⁡\(x\)⋅s\\operatorname\{sign\}\(x\)\\cdot sfactor, and we focus onQQthroughout\.

##### STE convention\.

We adopt the straight\-through estimator of\[[3](https://arxiv.org/html/2606.04115#bib.bib38)\], following the convention used in TQT\[[21](https://arxiv.org/html/2606.04115#bib.bib64)\]:

∂∂u⌊u⌉=∂∂u⌊u⌋=1,but⌊u⌉≠u,⌊u⌋≠uin the backward pass\.\\frac\{\\partial\}\{\\partial u\}\\lfloor u\\rceil\\;=\\;\\frac\{\\partial\}\{\\partial u\}\\lfloor u\\rfloor\\;=\\;1,\\qquad\\text\{but\}\\qquad\\lfloor u\\rceil\\neq u,\\;\\;\\lfloor u\\rfloor\\neq u\\quad\\text\{in the backward pass\.\}\(15\)That is, the derivative of the rounding and floor operators is approximated by11, while their values are preserved\. Crucially, this means the rounded quantityr:=⌊x~/𝑠𝑠\(x~\)⌉r:=\\lfloor\\tilde\{x\}/\\mathit\{ss\}\(\\tilde\{x\}\)\\rceildoes*not*collapse tox~/𝑠𝑠​\(x~\)\\tilde\{x\}/\\mathit\{ss\}\(\\tilde\{x\}\)when applying the chain rule; this distinction is what gives themm\-gradient a non\-trivial form\.

##### Setup\.

Letr:=⌊x~/𝑠𝑠\(x~\)⌉r:=\\lfloor\\tilde\{x\}/\\mathit\{ss\}\(\\tilde\{x\}\)\\rceilandu:=𝑠𝑠​\(x~\)⋅ru:=\\mathit\{ss\}\(\\tilde\{x\}\)\\cdot r, so thatQ​\(x~\)=uQ\(\\tilde\{x\}\)=uon the unclipped range andQ​\(x~\)=qmax\(FP\)Q\(\\tilde\{x\}\)=q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}otherwise\. Recall that𝑠𝑠​\(x~\)=2η​\(x~\)\\mathit\{ss\}\(\\tilde\{x\}\)=2^\{\\eta\(\\tilde\{x\}\)\}withη\\etaas in Eq\.[13](https://arxiv.org/html/2606.04115#A1.E13)andemin=1−b−me\_\{\\min\}=1\-b\-mas in Eq\.[12](https://arxiv.org/html/2606.04115#A1.E12)\. The two regimes ofη\\eta, \(i\.e\., the normal regime whenη=⌊log2⁡x~⌋−m\>emin\\eta=\\lfloor\\log\_\{2\}\\tilde\{x\}\\rfloor\-m\>e\_\{\\min\}and the subnormal regime whenη=emin\\eta=e\_\{\\min\}\), combined with whether the input exceeds the saturating bound, partition the input range into three cases: subnormal, normal\-unclipped, and clipped \(x~\>qmax\(FP\)\\tilde\{x\}\>q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\)\. Each yields a qualitatively different gradient and is kept explicit below\. Forx~\>qmax\(FP\)\\tilde\{x\}\>q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}, the bit\-width gradients∂Q/∂m\\partial Q/\\partial mand∂Q/∂e\\partial Q/\\partial eremain non\-trivial, sinceqmax\(FP\)q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}depends onmmandeethrough Eq\.[10](https://arxiv.org/html/2606.04115#A1.E10)\. The negative branch follows by the symmetryQ​\(−x~\)=−Q​\(x~\)Q\(\-\\tilde\{x\}\)=\-Q\(\\tilde\{x\}\)\. Atx~=0\\tilde\{x\}=0, the gradient is defined as the limit of the difference quotient from either side, taking the common value when these limits agree, and zero otherwise; in our case the bit\-width gradients vanish since the rounding gapx~−u→0\\tilde\{x\}\-u\\to 0\.

##### Gradient with respect tomm\.

The mantissa bit\-widthmmentersQQthroughη\\eta\(via the−m\-mterm in the normal regime and viaemine\_\{\\min\}in the subnormal regime\) and, in the clipped regime, throughqmax\(FP\)q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\. For the unclipped range,∂η/∂m=−1\\partial\\eta/\\partial m=\-1in both the normal and subnormal regimes, so

∂𝑠𝑠∂m=−ln⁡\(2\)⋅𝑠𝑠\.\\frac\{\\partial\\mathit\{ss\}\}\{\\partial m\}\\;=\\;\-\\ln\(2\)\\cdot\\mathit\{ss\}\.\(16\)Applying the STE to the round and using the chain rule onr=⌊x~/𝑠𝑠⌉r=\\lfloor\\tilde\{x\}/\\mathit\{ss\}\\rceilat fixedx~\\tilde\{x\},

∂r∂m=−x~𝑠𝑠2​∂𝑠𝑠∂m=x~​ln⁡\(2\)𝑠𝑠\.\\frac\{\\partial r\}\{\\partial m\}\\;=\\;\-\\frac\{\\tilde\{x\}\}\{\\mathit\{ss\}^\{\\,2\}\}\\,\\frac\{\\partial\\mathit\{ss\}\}\{\\partial m\}\\;=\\;\\frac\{\\tilde\{x\}\\,\\ln\(2\)\}\{\\mathit\{ss\}\}\.\(17\)Combining via the product rule onu=𝑠𝑠⋅ru=\\mathit\{ss\}\\cdot r:

∂Q∂m=∂𝑠𝑠∂m⋅r\+𝑠𝑠⋅∂r∂m=ln⁡\(2\)⋅\(x~−u\)for​\|x~\|∈\(0,qmax\(FP\)\]\.\\frac\{\\partial Q\}\{\\partial m\}\\;=\\;\\frac\{\\partial\\mathit\{ss\}\}\{\\partial m\}\\cdot r\\;\+\\;\\mathit\{ss\}\\cdot\\frac\{\\partial r\}\{\\partial m\}\\;=\\;\\ln\(2\)\\cdot\\bigl\(\\tilde\{x\}\-u\\bigr\)\\qquad\\text\{for \}\|\\tilde\{x\}\|\\in\(0,\\,q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\]\.\(18\)This expression has the same form as TQT’s threshold gradient\[[21](https://arxiv.org/html/2606.04115#bib.bib64), Eq\. 7\], withmmsubstituted for the log\-threshold and the exclusion ofss\.

Forx~\>qmax\(FP\)\\tilde\{x\}\>q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\},Q=qmax\(FP\)Q=q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}and themm\-derivative passes through the clipping bound\. Differentiating Eq\.[2](https://arxiv.org/html/2606.04115#S2.E2),

∂qmax\(FP\)∂m=ln⁡\(2\)⋅2−m⋅22e−b−1=ln⁡\(2\)2m\+1−1⋅qmax\(FP\),\\frac\{\\partial q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\}\{\\partial m\}\\;=\\;\\ln\(2\)\\cdot 2^\{\-m\}\\cdot 2^\{\\,2^\{e\}\-b\-1\}\\;=\\;\\frac\{\\ln\(2\)\}\{2^\{m\+1\}\-1\}\\cdot q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\},\(19\)a positive contribution that pushesmmupward to enlarge the representable range and bring saturating elements inside it\. Combining the two regions yields the complete piecewise gradient

∂Q∂m=\{ln⁡\(2\)⋅\(x~−u\)subnormal/normal regime,​\|x~\|∈\(0,qmax\(FP\)\],ln⁡\(2\)2m\+1−1⋅qmax\(FP\)x~\>qmax\(FP\)\.\\frac\{\\partial Q\}\{\\partial m\}\\;=\\;\\begin\{cases\}\\ln\(2\)\\cdot\\bigl\(\\tilde\{x\}\-u\\bigr\)&\\text\{subnormal/normal regime, \}\|\\tilde\{x\}\|\\in\(0,\\,q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\],\\\\\[4\.30554pt\] \\dfrac\{\\ln\(2\)\}\{2^\{m\+1\}\-1\}\\cdot q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}&\\tilde\{x\}\>q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\.\\end\{cases\}\(20\)

##### Gradient with respect toee\.

The exponent bit\-widtheeentersη\\etaonly throughb=2e−1−1b=2^\{e\-1\}\-1, and only in the subnormal regime, since⌊log2⁡x~⌋−m\\lfloor\\log\_\{2\}\\tilde\{x\}\\rfloor\-mhas is not a function ofee\. Hence∂η/∂e=0\\partial\\eta/\\partial e=0in the normal regime and∂η/∂e=−∂b/∂e=−2e−1​ln⁡\(2\)\\partial\\eta/\\partial e=\-\\partial b/\\partial e=\-2^\{e\-1\}\\ln\(2\)in the subnormal regime\. Following the same chain\-rule derivation as formm, with the extra factor of∂η/∂e\\partial\\eta/\\partial epropagating through𝑠𝑠\\mathit\{ss\}andrr:

∂Q∂e=\{0normal regime,2e−1​\(ln⁡2\)2⋅\(x~−u\)subnormal regime​\(η=emin\),\|x~\|∈\(0,qmax\(FP\)\]\.\\frac\{\\partial Q\}\{\\partial e\}\\;=\\;\\begin\{cases\}0&\\text\{normal regime\},\\\\\[2\.15277pt\] 2^\{e\-1\}\\,\(\\ln 2\)^\{2\}\\cdot\\bigl\(\\tilde\{x\}\-u\\bigr\)&\\text\{subnormal regime \}\(\\eta=e\_\{\\min\}\),\\end\{cases\}\\qquad\|\\tilde\{x\}\|\\in\(0,\\,q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\]\.\(21\)
Forx~\>qmax\(FP\)\\tilde\{x\}\>q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}, the gradient passes throughqmax\(FP\)q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\. Withb=2e−1−1b=2^\{e\-1\}\-1so that2e−b−1=2e−12^\{e\}\-b\-1=2^\{e\-1\}, differentiating Eq\.[2](https://arxiv.org/html/2606.04115#S2.E2)yields

∂qmax\(FP\)∂e=2​\(ln⁡2\)2⋅2e−1⋅22e−b−1​\(1−2−m−1\)=\(ln⁡2\)2⋅2e−1⋅qmax\(FP\)\.\\frac\{\\partial q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\}\{\\partial e\}\\;=\\;2\(\\ln 2\)^\{2\}\\cdot 2^\{e\-1\}\\cdot 2^\{\\,2^\{e\}\-b\-1\}\\bigl\(1\-2^\{\-m\-1\}\\bigr\)\\;=\\;\(\\ln 2\)^\{2\}\\cdot 2^\{e\-1\}\\cdot q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\.\(22\)Combining the two regions yields the complete piecewise gradient

∂Q∂e=\{2e−1​\(ln⁡2\)2⋅\(x~−u\)subnormal regime​\(η=emin\),0normal regime,​\|x~\|∈\(0,qmax\(FP\)\],\(ln⁡2\)2⋅2e−1⋅qmax\(FP\)x~\>qmax\(FP\)\.\\frac\{\\partial Q\}\{\\partial e\}\\;=\\;\\begin\{cases\}2^\{e\-1\}\\,\(\\ln 2\)^\{2\}\\cdot\\bigl\(\\tilde\{x\}\-u\\bigr\)&\\text\{subnormal regime \}\(\\eta=e\_\{\\min\}\),\\\\\[4\.30554pt\] 0&\\text\{normal regime, \}\|\\tilde\{x\}\|\\in\(0,\\,q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\],\\\\\[2\.15277pt\] \(\\ln 2\)^\{2\}\\cdot 2^\{e\-1\}\\cdot q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}&\\tilde\{x\}\>q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\.\\end\{cases\}\(23\)Theee\-gradient is supported at*both*extremes of the input distribution: in the subnormal regime viaemine\_\{\\min\}, and in the clipped regime viaqmax\(FP\)q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}, while in the normal\-unclipped regime the gradient contribution is zero\.

##### Gradient with respect toβ\\beta\.

The shared offset parameterizationm=1\+βm=1\+\\beta,e=2\+βe=2\+\\beta\(Section[2\.1](https://arxiv.org/html/2606.04115#S2.SS1)\) gives∂m/∂β=∂e/∂β=1\\partial m/\\partial\\beta=\\partial e/\\partial\\beta=1, so the chain rule yields

∂Q∂β=∂Q∂m\+∂Q∂e\.\\frac\{\\partial Q\}\{\\partial\\beta\}\\;=\\;\\frac\{\\partial Q\}\{\\partial m\}\\;\+\\;\\frac\{\\partial Q\}\{\\partial e\}\.\(24\)Substituting Eqs\.[20](https://arxiv.org/html/2606.04115#A1.E20)and[23](https://arxiv.org/html/2606.04115#A1.E23)gives

∂Q∂β=\{ln⁡\(2\)​\(1\+2e−1​ln⁡2\)⋅\(x~−u\)subnormal regime​\(η=emin\),ln⁡\(2\)⋅\(x~−u\)normal regime,​\|x~\|∈\(0,qmax\(FP\)\],\[ln⁡\(2\)2m\+1−1\+\(ln⁡2\)2​2e−1\]⋅qmax\(FP\)x~\>qmax\(FP\)\.\\frac\{\\partial Q\}\{\\partial\\beta\}\\;=\\;\\begin\{cases\}\\ln\(2\)\\,\\bigl\(1\+2^\{e\-1\}\\ln 2\\bigr\)\\cdot\\bigl\(\\tilde\{x\}\-u\\bigr\)&\\text\{subnormal regime \}\(\\eta=e\_\{\\min\}\),\\\\\[4\.30554pt\] \\ln\(2\)\\cdot\\bigl\(\\tilde\{x\}\-u\\bigr\)&\\text\{normal regime, \}\|\\tilde\{x\}\|\\in\(0,\\,q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\],\\\\\[2\.15277pt\] \\Bigl\[\\dfrac\{\\ln\(2\)\}\{2^\{m\+1\}\-1\}\\,\+\\,\(\\ln 2\)^\{2\}\\,2^\{e\-1\}\\Bigr\]\\cdot q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}&\\tilde\{x\}\>q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}\.\\end\{cases\}\(25\)Theβ\\beta\-gradient inherits themm\-component’s density across all three regions, picking up additionalee\-component support at the two extremes \(subnormal and clipped\)\. Among the three format parameters\{m,e,β\}\\\{m,e,\\beta\\\}, onlymmandβ\\betacarry a learning signal in the bulk normal\-unclipped regime where most of the input mass typically lies, which motivates theβ\\beta\-only training scheme of the main text\.

##### Remarks\.

Three observations follow from Eqs\.[20](https://arxiv.org/html/2606.04115#A1.E20),[23](https://arxiv.org/html/2606.04115#A1.E23)and[25](https://arxiv.org/html/2606.04115#A1.E25), all of which are illustrated by[Figure6](https://arxiv.org/html/2606.04115#A1.F6)\. First, the chain rule must be applied to the unsimplified product𝑠𝑠⋅r\\mathit\{ss\}\\cdot rrather than to its forward\-equivalentx~\\tilde\{x\}; the STE preserves the rounding gapx~−u\\tilde\{x\}\-u, which is the source of the non\-trivialmm\- andee\-gradients in the unclipped range\. Second, themm\-gradient is non\-zero in all three regions \(subnormal, normal\-unclipped, and clipped\) and is the dominant component of theβ\\beta\-gradient, as visible in panels[6\(b\)](https://arxiv.org/html/2606.04115#A1.F6.sf2)and[6\(d\)](https://arxiv.org/html/2606.04115#A1.F6.sf4)of[Figure6](https://arxiv.org/html/2606.04115#A1.F6)\. Third, theee\-gradient vanishes in the bulk normal\-unclipped regime and is supported at both extremes, i\.e\., below the subnormal cutoff21−b2^\{1\-b\}and above the clipping boundqmax\(FP\)q\_\{\\max\}^\{\(\\mathrm\{FP\}\)\}, so learningeealone requires sufficient mass at one or both extremes of the input distribution, while learning the joint offsetβ\\betadoes not\. In practice these gradients are computed automatically by autodifferentiation; we provide the closed forms here for completeness, having verified them against PyTorch’s autograd\[[30](https://arxiv.org/html/2606.04115#bib.bib65)\], confirmed by the overlap of analytical and autograd curves in[Figure6](https://arxiv.org/html/2606.04115#A1.F6)\.

![Refer to caption](https://arxiv.org/html/2606.04115v1/x5.png)

\(a\)forwardxq​\(x\)x\_\{q\}\(x\)
![Refer to caption](https://arxiv.org/html/2606.04115v1/x6.png)

\(b\)∂xq/∂m\\partial x\_\{q\}/\\partial m
![Refer to caption](https://arxiv.org/html/2606.04115v1/x7.png)

\(c\)∂xq/∂e\\partial x\_\{q\}/\\partial e
![Refer to caption](https://arxiv.org/html/2606.04115v1/x8.png)

\(d\)∂xq/∂β\\partial x\_\{q\}/\\partial\\beta
\(m=1\+βm\\\!=\\\!1\\\!\+\\\!\\beta,e=2\+βe\\\!=\\\!2\\\!\+\\\!\\beta\)

Figure 6:Forward and bitwidth\-gradient transfer curves of the inner MX\-FP quantizer ate=2e=2,m=2m=2\. The closed\-form analytical gradients \(black dashed\) overlay the Brevitas autograd reference \(blue dotted\) across the entire input range\. Panels[6\(a\)](https://arxiv.org/html/2606.04115#A1.F6.sf1)–[6\(d\)](https://arxiv.org/html/2606.04115#A1.F6.sf4)show the forwardxq​\(x\)x\_\{q\}\(x\)and the local gradients with respect tomm,ee, andβ\\beta, respectively\.

## Appendix BHyper\-Parameters

### B\.1Temperature Annealing

We employ a temperature\-based sigmoid function to switch from a linear\-like approximation to a step\-like one during training\. For the scheduling of the temperature, we employ an exponential function between the values of88and400400\. These values were picked because they best represent the linear\-like and the step\-like functions\.

As already mentioned, we also employ a warm\-up phase during which the temperature value is kept constant so that the sigmoid is its linear\-like phase\. After a fixed fraction of calibration steps,TratioT\_\{\\mathrm\{ratio\}\}, the exponential schedule starts\. For our experiments we setTratio=60%T\_\{\\mathrm\{ratio\}\}=60\\%\. We observed that our method is fairly robust to other values ofTratioT\_\{\\mathrm\{ratio\}\}, although adapting this value to a specific model and training pipeline could potentially yield better results\.

### B\.2Bit\-Width Cost Function

When testing the simple scaling penalty defined in Eq\.[6](https://arxiv.org/html/2606.04115#S2.E6), we use 15 different values fromλ\\lambdaranging from1​e−51e^\{\-5\}to77\. Similarly, in all the experiments with Eq\.[7](https://arxiv.org/html/2606.04115#S2.E7), we fixλ\\lambdato55, while testing for 17 different target bit\-widthb¯target\\bar\{b\}\_\{\\mathrm\{target\}\}between4\.14\.1and88, as mentioned in the Sec\.[3](https://arxiv.org/html/2606.04115#S3)\.

### B\.3Training Parameters

We follow a similar set\-up employed in SpinQuant\[[26](https://arxiv.org/html/2606.04115#bib.bib36)\], using the same hyper\-parameters for the optimization of the rotation matrices\. In particular, we use CayleySGD optimizer with a learning rate of1\.51\.5\. We merge the Hadamard rotations whenever possible; these are then optimized during training\. For all layers where they cannot be merged, we leave them as standalone, static rotations\. The main difference is the number of training steps and training samples used, which has been extended to400400and32003200respectively\.

For the bit\-width optimization, we employ SGD as optimized with a learning rate of11\.

## Appendix CAblation Studies

##### Mixed Precision Results

In[Figure7](https://arxiv.org/html/2606.04115#A3.F7)we represent the full set of experiments where we compare MXFP8/MXFP4 mixed precision configurations to MXFP6/MXFP4 ones\. Although both configurations perform well, the MXFP6/MXFP4 configurations are able to achieve better perplexity and average zero\-shot results for similar average bit\-widths\.

![Refer to caption](https://arxiv.org/html/2606.04115v1/x9.png)

Figure 7:Comparison of the MXFP8/MXFP4 mixed precision quantization and the MXFP6/MXFP4 ones\. Red: MXFP8/MXFP4 configurations\. Cyan: MXFP6/MXFP4 configurations\. Black diamonds: homogeneous MXFP baselines\. Top row: perplexity \(lower is better\)\. Bottom row: average zero\-shot accuracy \(higher is better\)\. The MXFP6/MXFP4 configurations are morebit\-width efficient, managing to reach better accuracy for the same average bit\-width\.
##### Continuous vs\. discrete bit\-width representation\.

Several prior approaches store the bit\-width as a continuous variable but discretize it during each forward pass, typically through a rounding operator with an STE in the backward pass\[[3](https://arxiv.org/html/2606.04115#bib.bib38)\]\. While effective for integer quantization where all bit\-widths in a range such as\[2,8\]\[2,8\]are admissible, this strategy may be less suitable for MXFP8–MXFP4 quantization\. The gap between FP8 and FP4 is comparatively large, and abrupt format switches during calibration can destabilize the training process\. The continuous representation used in dMXallows the effective format to evolve gradually, with the temperature\-based annealing of Section[2\.2](https://arxiv.org/html/2606.04115#S2.SS2)managing the transition to discrete inference\-time values\.

![Refer to caption](https://arxiv.org/html/2606.04115v1/x10.png)

Figure 8:Comparison between continuous forward\-pass bit\-width learning and a discretized forward pass with rounding \(ROUND\+STE\)\. Red: continuous representation\. Cyan: ROUND\+STE\. Black diamonds: homogeneous MXFP baselines\. Top row: perplexity \(lower is better\)\. Bottom row: average zero\-shot accuracy \(higher is better\)\. The continuous representation consistently achieves lower perplexity at a given average bit\-width\.As shown in[Figure8](https://arxiv.org/html/2606.04115#A3.F8), the continuous representation consistently achieves better perplexity and zero\-shot than the ROUND\+STE alternative across all three models\. Even when they are comparable, the results show that the ROUND\+STE approach struggles more in matching the desired target bit\-width, thus signaling a reliability issue compared to the continuous representation format\.

We believe that these results support the hypothesis that avoiding abrupt format switches during the forward pass leads to a more stable optimization trajectory\.