MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

# MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
Source: [https://arxiv.org/html/2607.23047](https://arxiv.org/html/2607.23047)
Ashitabh Misra University of Illinois at Urbana\-Champaign misra8@illinois\.edu&Madhav Agrawal University of Illinois at Urbana\-Champaign madhav5@illinois\.edu&Arham Jain University of Illinois at Urbana\-Champaign arhamj3@illinois\.eduTarek Abdelzaher University of Illinois at Urbana\-Champaign zaher@illinois\.edu

###### Abstract

Mixed\-precision quantization improves the accuracy of post\-training quantization by allocating higher bitwidths to sensitive layers, but existing methods solve the allocation for a single fixed memory budget\. In practice the budget varies across deployments and is unknown at calibration time\. Adaptive quantization addresses this with one offline calibration that serves any budget, yet current methods score layer sensitivity in a manner that does not consider its dependency on quantization levels of other layers\. We show that a layer’s sensitivity depends strongly on the bitwidths of its upstream layers and that this dependence shifts the resulting preferred bit allocation\. We proposeMixQuant, a technique\-agnostic adaptive framework that wraps any base quantizer\.MixQuantmarginalizes each layer’s distortion over random quantized upstream configurations to obtain budget\-agnostic scores, calibrates the quantizer’s parameters on plans the allocator itself produces, and penalizes allocations that leave layers at the lowest bitwidths\. A single greedy pass then serves any budget at deployment\. Across Llama\-3\.2\-3B, Llama\-2\-7B, and Mistral\-7B under AWQ and GPTQ,MixQuantoutperforms adaptive and mixed\-precision baselines in every setting, improving average accuracy by up to 8 points and reducing perplexity from 12\.43 to 10\.70 at the tightest budget, while matching an ILP solver at negligible deployment cost\.

## 1Introduction

Large Language Models \(LLMs\) underpin a wide range of applications, from dialog systems and code assistants to search and document understanding\(Touvronet al\.[2023](https://arxiv.org/html/2607.23047#bib.bib151); Lewiset al\.[2020](https://arxiv.org/html/2607.23047#bib.bib155); Islamet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib131)\)\. Their capability scales with size, and so does their cost: serving a model with tens of billions of parameters exceeds the memory and bandwidth of most deployment targets\(Touvronet al\.[2023](https://arxiv.org/html/2607.23047#bib.bib151); Kwonet al\.[2023](https://arxiv.org/html/2607.23047#bib.bib152)\)\. Among compression techniques such as pruning, distillation, and low\-rank factorization\(Liuet al\.[2019](https://arxiv.org/html/2607.23047#bib.bib107); Hintonet al\.[2015](https://arxiv.org/html/2607.23047#bib.bib109); Liu and Parhi[2023](https://arxiv.org/html/2607.23047#bib.bib154)\), post\-training quantization \(PTQ\) is the most widely adopted, reducing weight precision with only a small calibration set and no retraining\(Gholamiet al\.[2021](https://arxiv.org/html/2607.23047#bib.bib129); Rokhet al\.[2023](https://arxiv.org/html/2607.23047#bib.bib130)\)\.

Mixed\-precision quantization extends uniform PTQ by exploiting the fact that layers differ widely in sensitivity: allocating higher bitwidths to sensitive layers and lower bitwidths elsewhere yields better accuracy at the same memory footprint\(Wanget al\.[2019](https://arxiv.org/html/2607.23047#bib.bib56); Xuet al\.[2022](https://arxiv.org/html/2607.23047#bib.bib69)\)\. We show that allocations that have similar memory consumption can differ in quality \(Table[1](https://arxiv.org/html/2607.23047#S1.T1)\)\. Memory budget alone does not determine accuracy\. Where the bits are placed is what matters\. Existing mixed\-precision quantization methods search for an optimal allocation under a*fixed*resource constraint\(Wanget al\.[2019](https://arxiv.org/html/2607.23047#bib.bib56); Koryakovskiyet al\.[2023](https://arxiv.org/html/2607.23047#bib.bib54)\)\. In practice the constraint is neither single nor known: the same model is deployed across many devices with different memory capacities, and new deployment targets appear after calibration is complete\. Moreover, if calibration is slower than the memory availability changes, the allocation is always stale\. Many budgets make per\-budget calibration computationally infeasible; unknown budgets make it impossible\. A solution must therefore calibrate once, producing metrics from which a bit allocation for any budget is derived cheaply at deployment\(Jinet al\.[2020](https://arxiv.org/html/2607.23047#bib.bib49); Yuet al\.[2021](https://arxiv.org/html/2607.23047#bib.bib126)\)\.

Existing adaptive quantization methods, however, have three limitations\. First, adaptive quantization techniques developed for Convolutional Neural Networks rely on retraining to recover accuracy\(Jinet al\.[2020](https://arxiv.org/html/2607.23047#bib.bib49); Bulat and Tzimiropoulos[2021](https://arxiv.org/html/2607.23047#bib.bib48); Shkolniket al\.[2020](https://arxiv.org/html/2607.23047#bib.bib68); Sunet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib52)\), which is infeasible at LLM scale\. Second, post\-training adaptive quantization methods for LLMs, such as\(Dumitruet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib144)\), compute their layer\-wise metrics on inputs generated by upstream layers held in FP16 \(we call the upstream bit allocation the*context*\)\. This context never occurs at deployment, where every layer is quantized, and we show that the choice of context shifts both the metrics and the resulting bit allocation \(Figure[2](https://arxiv.org/html/2607.23047#S1.F2)\)\. Third, methods such as Any\-Precision LLM\(Parket al\.[2024](https://arxiv.org/html/2607.23047#bib.bib146)\)are tied to a single underlying quantization scheme, the non\-uniform codebooks of SqueezeLLM\(Kimet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib139)\), and do not transfer across PTQ techniques\.

![Refer to caption](https://arxiv.org/html/2607.23047v1/x1.png)Figure 1:Overview ofMixQuant\.Stage 1builds a budget\-agnostic distortion table𝐃∈ℝL×\|ℬ\|\\mathbf\{D\}\\in\\mathbb\{R\}^\{L\\times\|\\mathcal\{B\}\|\}by scoring each \(module, bitwidth\) pair underKKrandom quantized upstream contexts \(Section[4\.1](https://arxiv.org/html/2607.23047#S4.SS1)\)\.Stage 2estimates the technique parametersμ\\muon activations induced by anchor plans the greedy solver produces across the feasible budget range, rather than on FP16 activations \(Section[4\.2](https://arxiv.org/html/2607.23047#S4.SS2)\)\.Stage 3adds a tail penaltyγ\\gammathat steers spare budget away from the lowest bitwidths \(Section[4\.3](https://arxiv.org/html/2607.23047#S4.SS3)\)\. At deployment, each budgetMiM\_\{i\}is served by a single greedy solve over𝐃\\mathbf\{D\},μ\\mu, andγ\\gamma\.#### Our Solution\.

We proposeMixQuant, a technique\-agnostic adaptive PTQ framework that wraps any base quantizer, such as AWQ, GPTQ, and others\(Linet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib138); Frantaret al\.[2022](https://arxiv.org/html/2607.23047#bib.bib143)\)\. It addresses these limitations in three stages\.*First*, rather than scoring each layer against an FP16 network,MixQuantmeasures the output distortion of each \(layer, bitwidth\) pair under random mixed\-precision configurations of the layer’s*upstream*layers, the only layers that shape its input\. It then marginalizes this distortion over the random configurations via Monte Carlo sampling\. Marginalization removes the dependence on the unknown context, and the resulting distortion table depends only on the layer and its bitwidth, so it serves every deployment budget\.*Second*,MixQuantre\-introduces the information that is known at calibration time, namely the technique and the solver\. It estimates the technique parameters \(AWQ scales and clipping ranges, GPTQ Hessians\) on activations induced by allocations the solver actually produces, rather than on FP16 activations\.*Third*, because quantization error is irreversible along the forward pass, low\-bit assignments are disproportionately damaging;MixQuantadds a*tail regularization*term that steers spare budget toward layers still at the lowest bitwidths\. Finally, we cast bit allocation as a multiple\-choice knapsack problem and solve it with a greedy algorithm cheap enough to run at deployment time for each new budget\. We empirically show that its allocations match the downstream accuracy of an integer linear programming solver \(Table[5](https://arxiv.org/html/2607.23047#S5.T5)\)\.

In summary, our main contributions are:

- •We identify deployment\-context mismatch in adaptive quantization: per\-layer sensitivity scores computed on the FP16 model do not reflect the fully quantized networks actually deployed\. We propose a mean\-field remedy, budget\-agnostic distortion scores obtained by marginalizing each layer’s error over random quantized configurations of its upstream layers\.
- •We developMixQuant, an end\-to\-end, technique\-agnostic adaptive quantization pipeline that combines decoupled distortion scores, plan\-aware estimation of technique parameters, and tail\-regularized greedy allocation, so that a single offline calibration serves any memory budget at deployment with one cheap greedy solve\.
- •We evaluateMixQuantacross multiple models, datasets, and commonly used calibration metrics, and show that it consistently produces better bit allocations than existing adaptive and mixed\-precision baselines\(Dumitruet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib144); Zhaoet al\.[2026](https://arxiv.org/html/2607.23047#bib.bib145); Donget al\.[2020](https://arxiv.org/html/2607.23047#bib.bib149); Jinet al\.[2020](https://arxiv.org/html/2607.23047#bib.bib49)\)\.

Table 1:Plans sampled from the main experiments \(Tables[2](https://arxiv.org/html/2607.23047#S5.T2),[3](https://arxiv.org/html/2607.23047#S5.T3), and[4](https://arxiv.org/html/2607.23047#S5.T4)\) at a fixed budget per model\. Despite near\-equal memory, neither perplexity nor accuracy tracks the unweighted average bitwidth: the highest\-bitwidth plan is not the most accurate\. How precision is distributed across layers, not the average, determines quality\.![Refer to caption](https://arxiv.org/html/2607.23047v1/x2.png)\(a\)Metric variation across contexts\. For six representative modules of Llama\-3\.2\-3B \(left\) and Llama\-2\-7B \(right\), the relative spread\(max−min\)/min\(\\max\-\\min\)/\\minof NMSE and KL divergence across contexts\. Both metrics move by up to two to three orders of magnitude\.
![Refer to caption](https://arxiv.org/html/2607.23047v1/x3.png)\(b\)Allocation variation across contexts\. Per\-linear\-layer bitwidth spread \(max−min\\max\-\\min\) for Llama\-3\.2\-3B, one plan per context, from the ILP solver on the MCKP objective equation[2](https://arxiv.org/html/2607.23047#S3.E2)\. Thexx\-axis runs over linear layers, labelled by transformer block; theyy\-axis varies the memory bound\. Spread appears across all memory bounds, showing that context choice shifts the bit allocation\.

Figure 2:Effect of upstream context, measured over the same 20 random quantized contexts in both panels\. Both per\-module metrics \(top\) and the downstream plans they induce \(bottom\) vary with the context, so scoring in a single context biases the resulting bit allocation\.MixQuantremoves this dependence by averaging over contexts \(Section[4\.1](https://arxiv.org/html/2607.23047#S4.SS1)\)\.

## 2Related Work

MixQuantbuilds on two lines of prior work: post\-training quantization for LLMs, and adaptive quantization\. We review each in turn\.

### 2\.1Post\-Training Quantization for LLMs

Since retraining billion\-parameter models is prohibitively expensive,*Post\-Training Quantization*\(PTQ\) has become the dominant approach for compressing LLMs\. Scaling\-based methods redistribute quantization difficulty between weights and activations: SmoothQuant\(Xiaoet al\.[2023](https://arxiv.org/html/2607.23047#bib.bib137)\)migrates activation outliers into the weights via a per\-channel equivalent transform, while AWQ\(Linet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib138)\)protects the few salient weight channels identified by activation magnitude\. GPTQ\(Frantaret al\.[2022](https://arxiv.org/html/2607.23047#bib.bib143)\)instead exploits second\-order information, quantizing column by column and using an approximate Hessian to compensate the error introduced at each step\. LQER\(Zhanget al\.[2024](https://arxiv.org/html/2607.23047#bib.bib140)\)takes a corrective route, absorbing the residual quantization error into an activation\-scaled low\-rank term\. Rotation\-based methods instead precondition the network: QuaRot\(Ashkbooset al\.[2024](https://arxiv.org/html/2607.23047#bib.bib141)\)applies randomized Hadamard rotations and SpinQuant\(Liuet al\.[2025](https://arxiv.org/html/2607.23047#bib.bib142)\)learns them, both suppressing outliers before quantization\.*Mixed\-precision*methods instead allocate bits non\-uniformly across the model: HAWQv2\(Donget al\.[2020](https://arxiv.org/html/2607.23047#bib.bib149)\)assigns layer bit\-widths by average Hessian trace, AMQ\(Leeet al\.[2025](https://arxiv.org/html/2607.23047#bib.bib158)\)searches per\-layer precisions to minimize accuracy loss under a budget, and AQLM\(Egiazarianet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib157)\)pushes weights to∼\\sim2 bits via learned additive codebooks, all trading precision against a target budget\. The aforementioned techniques assume the deployment memory budget is known a priori\. In real\-time deployment, where constraints change at runtime, this assumption fails—and re\-solving the allocation for every new budget is impractical\.

### 2\.2Adaptive Quantization

*Adaptive quantization*removes this dependence: a single offline calibration yields per\-layer scores from which an allocation for*any*budget is recovered by one cheap solve at deployment\. Early adaptive quantization methods, developed for CNNs and small transformers, achieve this through quantization\-aware retraining: AdaBits\(Jinet al\.[2020](https://arxiv.org/html/2607.23047#bib.bib49)\)jointly trains a single model to run at any of a fixed set of bit\-widths, BitMixer\(Bulat and Tzimiropoulos[2021](https://arxiv.org/html/2607.23047#bib.bib48)\)trains a meta\-network that selects arbitrary mixed\-precision configurations at runtime, and RobustQuant\(Shkolniket al\.[2020](https://arxiv.org/html/2607.23047#bib.bib68)\)regularizes weight kurtosis so layer distributions stay resilient across precisions\. All recover accuracy through gradient updates, which is infeasible at LLM scale; we therefore focus on retraining\-free adaptive quantization for LLMs\. LIM\(Dumitruet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib144)\)scores each transformer block by the cosine similarity between its input and output hidden states, a single\-pass metric requiring no solver\. CoopQ\(Zhaoet al\.[2026](https://arxiv.org/html/2607.23047#bib.bib145)\)attributes a contribution to each layer via Shapley values and allocates bits with an ILP\. Any\-Precision LLM\(Parket al\.[2024](https://arxiv.org/html/2607.23047#bib.bib146)\)stores one model that serves multiple bit\-widths, built on the non\-uniform codebooks of SqueezeLLM\(Kimet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib139)\)\.

## 3Problem Setup

#### Setup\.

A pretrained LLM contains linear modulesℓ∈\{1,…,L\}\\ell\\in\\\{1,\\dots,L\\\}, where moduleℓ\\ellhasPℓP\_\{\\ell\}parameters\. Each module is quantized by a PTQ techniqueTT, e\.g\., AWQ or GPTQ\(Linet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib138); Frantaret al\.[2022](https://arxiv.org/html/2607.23047#bib.bib143)\)\. The technique has calibration parametersμ\\mu, such as smoothing factors and clipping ranges for AWQ or Hessian estimates for GPTQ\. Bitwidths are chosen from an ordered setℬ\\mathcal\{B\}of admissible values \(ℬ=\{3,…,8\}\\mathcal\{B\}=\\\{3,\\dots,8\\\}in our experiments\), and a*plan*is a vector𝐛=\(b1,…,bL\)∈ℬL\\mathbf\{b\}=\(b\_\{1\},\\dots,b\_\{L\}\)\\in\\mathcal\{B\}^\{L\}\. A plan occupies weight memorymem​\(𝐛\)=18​∑ℓ=1LPℓ​bℓ\\mathrm\{mem\}\(\\mathbf\{b\}\)=\\tfrac\{1\}\{8\}\\sum\_\{\\ell=1\}^\{L\}P\_\{\\ell\}\\,b\_\{\\ell\}\(reported in bytes\)\. Given a memory budgetMM, the deployment objective is

min𝐛,μ⁡ℒ​\(T​\(𝐛,μ\)\)s\.t\.mem​\(𝐛\)≤M,\\min\_\{\\mathbf\{b\},\\,\\mu\}\\;\\;\\mathcal\{L\}\\big\(T\(\\mathbf\{b\},\\mu\)\\big\)\\quad\\text\{s\.t\.\}\\quad\\mathrm\{mem\}\(\\mathbf\{b\}\)\\leq M,\(1\)whereℒ\\mathcal\{L\}is the loss of the quantized model\. In the adaptive setting, the techniqueTTand the allocation solver are fixed before calibration, but the budgetMMis a*runtime*constraint unknown at calibration time\. This variation calls for a shared calibration and precomputed metrics independent ofMM\.

#### Surrogate\.

Optimizing equation[1](https://arxiv.org/html/2607.23047#S3.E1)directly is intractable, sinceℒ\\mathcal\{L\}is a black box over a configuration space that is exponential in the number of modules\. We replaceℒ\\mathcal\{L\}with a sum of per\-module distortion scoresd¯ℓ​\(b\)\\bar\{d\}\_\{\\ell\}\(b\), whered¯ℓ​\(b\)\\bar\{d\}\_\{\\ell\}\(b\)measures the output distortion of quantizing moduleℓ\\elltobbbits\. We formulate the resulting allocation problem as a multiple\-choice knapsack problem \(MCKP\), a formulation shared by prior mixed\-precision methods\(Yaoet al\.[2020](https://arxiv.org/html/2607.23047#bib.bib153); Liet al\.[2023](https://arxiv.org/html/2607.23047#bib.bib2); Misraet al\.[2025](https://arxiv.org/html/2607.23047#bib.bib53)\),

minx​∑ℓ=1L∑b∈ℬd¯ℓ​\(b\)​xℓ,bs\.t\.∑b∈ℬxℓ,b=1​∀ℓ,18​∑ℓ=1L∑b∈ℬPℓ​b​xℓ,b≤M,\\min\_\{x\}\\;\\sum\_\{\\ell=1\}^\{L\}\\sum\_\{b\\in\\mathcal\{B\}\}\\bar\{d\}\_\{\\ell\}\(b\)\\,x\_\{\\ell,b\}\\quad\\text\{s\.t\.\}\\quad\\sum\_\{b\\in\\mathcal\{B\}\}x\_\{\\ell,b\}=1\\;\\;\\forall\\ell,\\qquad\\frac\{1\}\{8\}\\sum\_\{\\ell=1\}^\{L\}\\sum\_\{b\\in\\mathcal\{B\}\}P\_\{\\ell\}\\,b\\,x\_\{\\ell,b\}\\leq M,\(2\)wherexℓ,b∈\{0,1\}x\_\{\\ell,b\}\\in\\\{0,1\\\}indicates the assignment of bitwidthbbto moduleℓ\\ell\. The additive objective presumes each module’s score is independent of the rest of the plan; Section[4\.1](https://arxiv.org/html/2607.23047#S4.SS1)constructsd¯ℓ​\(b\)\\bar\{d\}\_\{\\ell\}\(b\)so that this holds by definition\. The technique parametersμ\\mudo not appear in equation[2](https://arxiv.org/html/2607.23047#S3.E2); they are estimated once during calibration and then held fixed \(Section[4\.2](https://arxiv.org/html/2607.23047#S4.SS2)\)\.

## 4Method

MixQuantsolves the allocation problem equation[2](https://arxiv.org/html/2607.23047#S3.E2)in three stages, which together estimate its two unknowns: the distortion scoresd¯ℓ​\(b\)\\bar\{d\}\_\{\\ell\}\(b\)and the technique parametersμ\\mu\. Figure[1](https://arxiv.org/html/2607.23047#S1.F1)gives an overview\.Stage 1estimatesd¯ℓ​\(b\)\\bar\{d\}\_\{\\ell\}\(b\)\. A module’s distortion depends on its*context*, the bitwidths of the upstream modules that shape its input, and the deployed context is unknown at calibration time\. We therefore average each score over random quantized upstream configurations, which yields a budget\-agnostic distortion table \(Section[4\.1](https://arxiv.org/html/2607.23047#S4.SS1)\)\.Stage 2estimatesμ\\mu\. Unlike the budget, the technique and the solver are known before deployment\. We generate plans across the feasible budget range with the greedy solver, and calibrateμ\\muon the activations these plans induce rather than on FP16 activations \(Section[4\.2](https://arxiv.org/html/2607.23047#S4.SS2)\)\.Stage 3introduces a tail regularizer\. Distortion introduced by a low\-bit module cannot be undone by any downstream module, however high its precision, and the per\-module scores in equation[2](https://arxiv.org/html/2607.23047#S3.E2)do not capture this cost\. The regularizer biases the solver against leaving modules at the lowest bitwidths \(Section[4\.3](https://arxiv.org/html/2607.23047#S4.SS3)\)\.

### 4\.1Stage 1: Mean\-Field Distortion Estimation

Any per\-module distortion score is measured in a*context*: the bitwidths of the upstream modules that shape the input the module receives\. Prior work fixes this context to full precision\. Layer\- and block\-wise PTQ reconstructs each unit from FP16 inputs\(Nagelet al\.[2020](https://arxiv.org/html/2607.23047#bib.bib147); Liet al\.[2021](https://arxiv.org/html/2607.23047#bib.bib148)\)\. Mixed\-precision methods rank the sensitivity of a layer while every other layer is left unquantized\(Donget al\.[2020](https://arxiv.org/html/2607.23047#bib.bib149); Yaoet al\.[2020](https://arxiv.org/html/2607.23047#bib.bib153)\)\. Activation\-aware LLM quantizers calibrate their transforms on FP16 activations\(Linet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib138); Xiaoet al\.[2023](https://arxiv.org/html/2607.23047#bib.bib137)\)\. A fully FP16 upstream is a configuration that never occurs at deployment, where every module is quantized\. A representative score must instead be measured under quantized upstream modules\. The deployed plan is unknown at calibration time, so we construct a score table that does not depend on it and therefore serves every budget\.

#### Context\-dependent distortion\.

The input to moduleℓ\\elldrifts from its FP16 value whenever upstream modules are quantized\. A per\-module score therefore depends on the bitwidth of the module and on its upstream context𝐛<ℓ∈ℬℓ−1\\mathbf\{b\}\_\{<\\ell\}\\in\\mathcal\{B\}^\{\\ell\-1\}\. We instantiatedℓd\_\{\\ell\}as the normalized mean squared error \(NMSE\) of the module output,

dℓ​\(b;𝐛<ℓ\)=𝔼x​‖yℓ\(b,𝐛<ℓ\)​\(x\)−yℓfp16​\(x\)‖2𝔼x​‖yℓfp16​\(x\)‖2,d\_\{\\ell\}\(b;\\mathbf\{b\}\_\{<\\ell\}\)=\\frac\{\\mathbb\{E\}\_\{x\}\\big\\\|y\_\{\\ell\}^\{\(b,\\,\\mathbf\{b\}\_\{<\\ell\}\)\}\(x\)\-y\_\{\\ell\}^\{\\mathrm\{fp16\}\}\(x\)\\big\\\|^\{2\}\}\{\\mathbb\{E\}\_\{x\}\\big\\\|y\_\{\\ell\}^\{\\mathrm\{fp16\}\}\(x\)\\big\\\|^\{2\}\},\(3\)wherexxis a calibration sample,yℓ\(b,𝐛<ℓ\)y\_\{\\ell\}^\{\(b,\\,\\mathbf\{b\}\_\{<\\ell\}\)\}is the output of moduleℓ\\ellquantized tobbbits under context𝐛<ℓ\\mathbf\{b\}\_\{<\\ell\}, andyℓfp16y\_\{\\ell\}^\{\\mathrm\{fp16\}\}is the FP16 reference\. The normalization makes scores comparable across modules\.

This dependence is large in practice, in both the raw metrics and the allocation they induce \(Figure[2](https://arxiv.org/html/2607.23047#S1.F2)\), so a score measured in a single arbitrary context is not representative\.

#### Averaging over quantized contexts\.

The deployed context is unknown at calibration time, so we treat it as random\. We draw the upstream bit\-widths independently and uniformly overℬ\\mathcal\{B\}, writtenπ​\(𝐛<ℓ\)=∏m<ℓUnif​\(ℬ\)\\pi\(\\mathbf\{b\}\_\{<\\ell\}\)=\\prod\_\{m<\\ell\}\\mathrm\{Unif\}\(\\mathcal\{B\}\), and define the*decoupled distortion*d¯ℓ​\(b\)\\bar\{d\}\_\{\\ell\}\(b\)as the expected NMSE under this prior\. We estimate it withKKMonte Carlo draws,

d¯ℓ​\(b\)=𝔼𝐁<ℓ∼π​\[dℓ​\(b;𝐁<ℓ\)\],d^ℓ​\(b\)=1K​∑k=1Kdℓ​\(b;𝐛<ℓ\(k\)\),𝐛<ℓ\(k\)∼π\.\\bar\{d\}\_\{\\ell\}\(b\)=\\mathbb\{E\}\_\{\\mathbf\{B\}\_\{<\\ell\}\\sim\\pi\}\\\!\\big\[d\_\{\\ell\}\(b;\\mathbf\{B\}\_\{<\\ell\}\)\\big\],\\qquad\\hat\{d\}\_\{\\ell\}\(b\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}d\_\{\\ell\}\\big\(b;\\mathbf\{b\}\_\{<\\ell\}^\{\(k\)\}\\big\),\\quad\\mathbf\{b\}\_\{<\\ell\}^\{\(k\)\}\\sim\\pi\.\(4\)
For the sampled context𝐛<ℓ\(k\)\\mathbf\{b\}\_\{<\\ell\}^\{\(k\)\}, the technique parameters \(μ\\mu\) of the quantized modules are recalibrated under that context, so no measurement in the table depends on FP16 calibration\. These per\-context parameters are used only for scoring and are then discarded\. The single set of parameters deployed with the model is estimated in Stage 2\.

The resulting table depends on\(ℓ,b\)\(\\ell,b\)alone and serves every budget\. This averaging is a mean\-field decoupling: the unknown joint configuration of upstream modules is replaced by its average effect under the fixed priorπ\\pi, so each module’s score becomes independent of the decisions made for other modules\.

### 4\.2Stage 2: Plan\-Aware Technique Parameters

Stage 1 removes all plan information from the scores, but not all of it is unknown\. The techniqueTTand the solver are fixed before deployment, and the solver has its own systematic preferences over plans\. We re\-introduce this known structure through the technique parametersμ\\mu, by calibrating them on plans the solver itself produces\.

We partition the feasible memory range\[Mmin,Mmax\]\[M\_\{\\min\},M\_\{\\max\}\]intoIIintervals, whereMminM\_\{\\min\}places every module at the smallest bitwidth andMmaxM\_\{\\max\}at the largest\. From intervaliiwe sampleJJbudgets and solve each with the same allocator used at deployment \(Section[4\.4](https://arxiv.org/html/2607.23047#S4.SS4)\), which yieldsJJanchor plans\. We quantize the network according to each anchor plan, calibrate one parameter set on the activations it induces, and average theJJsets to formμi\\mu\_\{i\}\.

For GPTQ the parameters are the layer Hessians, for AWQ the smoothing factors and clipping ranges\. Averaging is exact for Hessians, since they are second moments of the inputs and their mean equals the Hessian of the mixed activations\. For AWQ it is a heuristic, and we use it uniformly across techniques as a simple way to capture the average behavior the technique and solver induce\. Bucketing exists because plans at opposite ends of the budget range induce different activation statistics, so one parameter set fits neither end well, while within an interval the anchor plans are close enough to share one\. At deployment, a budgetMMfalling in intervaliiis served by one greedy solve and the precomputedμi\\mu\_\{i\}\.

### 4\.3Stage 3: Tail Regularization

Quantization distortion is irreversible along the forward pass\. Once a module’s output is corrupted by a low\-bit assignment, no downstream module can recover the lost information, however high its precision, and the error compounds as it propagates\. The lowest bitwidths should therefore be conceded sparingly\. We make this explicit with a penalty that steers spare budget toward the modules still at the lowest bitwidths, so the lowest precisions are used only when the budget leaves no alternative\.

Letb0b\_\{0\}be the smallest bit\-width inℬ\\mathcal\{B\}, and letb\+b^\{\+\}denote the successor ofbbin the ordered setℬ\\mathcal\{B\}\. Raising a module frombbtob\+b^\{\+\}removes distortion at the cost of additional memory\. We score this upgrade by the distortion removed per additional memory,

eℓ​\(b\)=d¯ℓ​\(b\)−d¯ℓ​\(b\+\)Pℓ​\(b\+−b\),e\_\{\\ell\}\(b\)\\;=\\;\\frac\{\\bar\{d\}\_\{\\ell\}\(b\)\-\\bar\{d\}\_\{\\ell\}\(b^\{\+\}\)\}\{P\_\{\\ell\}\\,\(b^\{\+\}\-b\)\},\(5\)and discount it by how far the module has already been raised above the floor,

e~ℓ​\(b\)=eℓ​\(b\)1\+γ​\(b−b0\),γ≥0\.\\tilde\{e\}\_\{\\ell\}\(b\)\\;=\\;\\frac\{e\_\{\\ell\}\(b\)\}\{1\+\\gamma\\,\(b\-b\_\{0\}\)\},\\qquad\\gamma\\geq 0\.\(6\)The depthb−b0b\-b\_\{0\}measures how far into the tail an upgrade reaches\. Each additional bit granted to an already\-raised module is progressively discounted relative to a module still at the floor, so the budget is pulled toward clearing modules off the lowest bitwidths rather than enriching a few modules deeply\. The strengthγ\\gammacontrols how strongly low bitwidths are discouraged\.

### 4\.4Greedy Allocation

Given a feasible budgetMM, the allocator starts from the uniform floor plan𝐛=\(b0,…,b0\)\\mathbf\{b\}=\(b\_\{0\},\\dots,b\_\{0\}\)and maintains a set of candidate upgrades, one per module: the transition from its current bitwidthbℓb\_\{\\ell\}to the successorbℓ\+b\_\{\\ell\}^\{\+\}, scored by the penalized efficiencye~ℓ​\(bℓ\)\\tilde\{e\}\_\{\\ell\}\(b\_\{\\ell\}\)of equation[6](https://arxiv.org/html/2607.23047#S4.E6)\. At each step the highest\-scoring upgrade that fits the remaining budget is applied\. The upgraded module’s old transition is removed and its next transition, frombℓ\+b\_\{\\ell\}^\{\+\}onward, is inserted\. The procedure ends when no candidate fits the remaining budget\. Algorithm[1](https://arxiv.org/html/2607.23047#alg1)gives the pseudocode\. The solve performs at mostL​\(\|ℬ\|−1\)L\(\|\\mathcal\{B\}\|\-1\)upgrades, each selecting the best of at mostLLcandidates, so a full allocation costsO​\(L​\|ℬ\|​log⁡L\)O\(L\\,\|\\mathcal\{B\}\|\\log L\)with a priority queue, and is negligible at deployment time\.

Algorithm 1Tail\-Regularized Greedy Allocation1:distortion table

d¯\\bar\{d\}, budget

MM, penalty

γ\\gamma
2:

b0←b\_\{0\}\\leftarrowsmallest uniform bitwidth in

ℬ\\mathcal\{B\}
3:

bℓ←b0b\_\{\\ell\}\\leftarrow b\_\{0\}for all

ℓ\\ell;

R←M−mem​\(𝐛\)R\\leftarrow M\-\\mathrm\{mem\}\(\\mathbf\{b\}\)⊳\\trianglerightremaining budget

4:

Q←\{\(e~ℓ​\(bℓ\),ℓ\):bℓ<bmax\}Q\\leftarrow\\\{\(\\tilde\{e\}\_\{\\ell\}\(b\_\{\\ell\}\),\\,\\ell\):b\_\{\\ell\}<b\_\{\\max\}\\\}⊳\\trianglerightcandidate transitions

5:while

QQcontains a transition with cost

Pℓ​\(bℓ\+−bℓ\)≤RP\_\{\\ell\}\\,\(b\_\{\\ell\}^\{\+\}\-b\_\{\\ell\}\)\\leq Rdo

6:pop the highest\-scoring

\(e~ℓ,ℓ\)\(\\tilde\{e\}\_\{\\ell\},\\,\\ell\)with

Pℓ​\(bℓ\+−bℓ\)≤RP\_\{\\ell\}\\,\(b\_\{\\ell\}^\{\+\}\-b\_\{\\ell\}\)\\leq R
7:

R←R−Pℓ​\(bℓ\+−bℓ\)R\\leftarrow R\-P\_\{\\ell\}\\,\(b\_\{\\ell\}^\{\+\}\-b\_\{\\ell\}\);

bℓ←bℓ\+b\_\{\\ell\}\\leftarrow b\_\{\\ell\}^\{\+\}
8:if

bℓ<bmaxb\_\{\\ell\}<b\_\{\\max\}then

9:push

\(e~ℓ​\(bℓ\),ℓ\)\(\\tilde\{e\}\_\{\\ell\}\(b\_\{\\ell\}\),\\,\\ell\)into

QQ
10:endif

11:endwhile

12:return

𝐛\\mathbf\{b\}

## 5Evaluation

#### Setup\.

We evaluateMixQuanton Llama\-3\.2\-3B, Llama\-2\-7B, and Mistral\-7B\-v0\.1\. Plans assign per\-linear\-module bitwidths fromℬ=\{3,…,8\}\\mathcal\{B\}=\\\{3,\\dots,8\\\}using AWQ or GPTQ as the base technique, with 8\-bit activations\. Quantization is simulated \(quantize–dequantize to FP16\); we report memory as packed linear\-weight size, excluding scale overhead\. Distortion tables and technique parameters are calibrated on 128 random windows of length 2048 from the WikiText\-2 training split\. Unless stated otherwise,MixQuantusesK=10K=10Monte Carlo context draws per \(module, bitwidth\) pair \(Section[4\.1](https://arxiv.org/html/2607.23047#S4.SS1)\),I=3I=3budget intervals withJ=5J=5anchor plans each \(Section[4\.2](https://arxiv.org/html/2607.23047#S4.SS2)\), and tail penaltyγ=10\\gamma=10\(Section[4\.3](https://arxiv.org/html/2607.23047#S4.SS3)\)\. We evaluate word\-level perplexity on the WikiText\-2 validation split and accuracy on PIQA, ARC\-Challenge, WinoGrande, and MMLU \(grouped into humanities, social sciences, STEM, and other\), all through the EleutherAI lm\-evaluation\-harness\(Gaoet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib150)\)\.

#### Baselines\.

The baselines span the space of FP16\-prior scoring metrics, alongside existing adaptive pipelines\. The first four are controlled comparisons constructed by us\. Each scores every \(module, bitwidth\) pair against the FP16 model and feeds the resulting table to the same ILP solver, based on the MCKP formulation of equation[2](https://arxiv.org/html/2607.23047#S3.E2)\. The four therefore differ only in the scoring metric\.*Quant\. Err\.*and*KL Div*compare the output activations of each quantized module to their full\-precision counterparts, scoring the pair by the activation error and the KL divergence of the output distributions, respectively\.*HAWQ\-v2*adopts the metric of HAWQ\-V2\(Donget al\.[2020](https://arxiv.org/html/2607.23047#bib.bib149)\), which combines a Hessian\-based importance estimate with the quantization error, so that both the sensitivity of a module and the magnitude of its perturbation enter the score\.*Fisher*uses the Fisher approximation of the Hessian as a pure importance estimate, capturing sensitivity alone\. Together these four cover error, distributional divergence, importance, and their combination, the principal axes along which prior scoring metrics differ\.

The remaining two baselines are existing adaptive quantization methods and run with their own allocation procedures\.*LIM*\(Dumitruet al\.[2024](https://arxiv.org/html/2607.23047#bib.bib144)\)scores modules by the cosine similarity between input and output of each transformer block\. By design, LIM’s methodology does not extend to linear\-layer level granularity\.*CoopQ*\(Zhaoet al\.[2026](https://arxiv.org/html/2607.23047#bib.bib145)\)departs from per\-module scoring altogether, estimating module contributions via Shapley\-value distributions and solving a novel ILP formulation for adaptive quantization\.

#### Accuracy across memory budgets\.

Tables[2](https://arxiv.org/html/2607.23047#S5.T2),[3](https://arxiv.org/html/2607.23047#S5.T3), and[4](https://arxiv.org/html/2607.23047#S5.T4)report perplexity and downstream accuracy under three memory budgets per model\.MixQuantattains the best average accuracy and the lowest perplexity in every \(model, budget, technique\) cell, for both AWQ and GPTQ, which supports the claim that the pipeline is agnostic to the base technique\.

Table 2:Downstream mixed\-precision quantization results on Llama\-3\.2\-3B\. Accuracies \(%\); Wiki\. = word perplexity \(↓\\downarrow\)\. Avg excludes Wiki\.Bold= best,underline= second best per column within each memory budget\.Table 3:Downstream mixed\-precision quantization results on Llama\-2\-7B\. Accuracies \(%\); Wiki\. = word perplexity \(↓\\downarrow\)\. Avg excludes Wiki\.Bold= best,underline= second best per column within each memory budget\.Table 4:Downstream mixed\-precision quantization results on Mistral\-7B\. Accuracies \(%\); Wiki\. = word perplexity \(↓\\downarrow\)\. Avg excludes Wiki\.Bold= best,underline= second best per column within each memory budget\.The margins are largest where quantization is most aggressive, and this is where a context\-aware score should matter most: at tight budgets many modules sit at the lowest bitwidths, upstream contexts drift far from full precision, and the resulting errors are large, irreversible, and propagate downstream, so misallocating even a few modules is costly\. On Llama\-3\.2\-3B at the 1\.25 GB budget, the tightest setting we evaluate,MixQuantimproves average accuracy over the best baseline by 7\.1 points under AWQ \(55\.6 vs\. 48\.5\) and 8\.0 points under GPTQ \(54\.0 vs\. 46\.0\), while reducing perplexity from 12\.43 to 10\.70 and from 13\.35 to 11\.19, respectively\. The gains are not confined to a single task: at this budgetMixQuantimproves WinoGrande by roughly 5 points and the MMLU categories by up to 11 points over the strongest baseline in each column\. The pattern persists at 1\.5 GB, where the GPTQ margin is 9\.1 points\. On Llama\-2\-7B at 3\.0 GB the margin is 4\.2 points under AWQ, and on Mistral\-7B the improvement ranges from 2\.1 to 3\.7 points across budgets and techniques\.

Notably, LIM and CoopQ are not the strongest baselines in most cells\. The FP16\-prior metrics paired with our ILP allocator frequently outperform them\. Both LIM and CoopQ were designed and validated on small bitwidth sets, typically\{2,3,4\}\\\{2,3,4\\\}, where their tendency to push allocations toward the extremes of the set is harmless because the extremes are never far apart\. Our setting expands the choice set toℬ=\{3,…,8\}\\mathcal\{B\}=\\\{3,\\dots,8\\\}, which enlarges the plan space by orders of magnitude and widens the gap between extremes\. Figure[3](https://arxiv.org/html/2607.23047#S5.F3)shows the bitwidth distributions of the resulting plans: LIM and CoopQ concentrate mass at the extremes ofℬ\\mathcal\{B\}, whileMixQuantspreads allocations across the middle bitwidths\.

![Refer to caption](https://arxiv.org/html/2607.23047v1/x4.png)Figure 3:Per\-linear\-layer bitwidth allocations by method\. For Llama\-3\.2\-3B \(1\.25 GB\), Llama\-2\-7B \(3\.0 GB\), and Mistral\-7B \(3\.0 GB\), thexx\-axis runs over linear layers, labelled by transformer block; the rows are scoring methods; shade encodes the bitwidth assigned to each layer\. LIM and CoopQ collapse to the extremes ofℬ\\mathcal\{B\}, while the others spread across the middle bitwidths\.On the Llama models, the margins narrow as the budget approaches the memory of the highest uniform bitwidth: on Llama\-2\-7B the AWQ improvement falls from 4\.2 points at 3\.0 GB to 1\.5 points at 4\.0 GB, and under GPTQ to 0\.8\. This is expected\. A generous budget places most modules at high bitwidths under any reasonable allocation, and high\-bitwidth quantization is robust enough that the remaining decisions carry little weight, so accuracies converge toward the plateau of the underlying technique\. The value of context\-aware scoring is concentrated in the low\-budget regime\.

#### Component ablation\.

Table[5](https://arxiv.org/html/2607.23047#S5.T5)isolates the contribution of the allocator on Llama\-3\.2\-3B at the 1\.25 GB budget\.*Mem\. budget*removes the distortion scores entirely: bits are assigned by an ILP whose only objective is to fill the budget, i\.e\.,min⁡\(M−mem​\(𝐛\)\)\\min\\,\(M\-\\mathrm\{mem\}\(\\mathbf\{b\}\)\)subject tomem​\(𝐛\)≤M\\mathrm\{mem\}\(\\mathbf\{b\}\)\\leq M, so the allocation reflects module sizes alone\.*Greedy \(rev\.\)*runs our greedy allocator with the candidate ordering reversed, applying the*worst*\-scoring upgrade first\.*Stage 1 \+ Stage 2 \+ ILP*keeps the full scoring pipeline but replaces the greedy solver with an ILP solution of equation[2](https://arxiv.org/html/2607.23047#S3.E2)\.MixQuantis the full method: the efficiency\-greedy allocator with the tail penalty of Section[4\.3](https://arxiv.org/html/2607.23047#S4.SS3)\.

Table 5:Component ablation on Llama\-3\.2\-3B at the 1\.25 GB budget\. Accuracies \(%\); Wiki\. = word perplexity \(↓\\downarrow\)\. Avg excludes Wiki\.The score\-free and reversed variants collapse, losing 6–17 average points against the full method, which confirms that the decoupled distortion table carries real signal: allocating without it, or against it, is severely punished at this budget\. The comparison with the ILP variant addresses the solver\. The ILP is the natural upper bound for our surrogate objective, yetMixQuantmatches it on average \(55\.2 vs\. 55\.6 under AWQ, 53\.9 vs\. 55\.3 under GPTQ\) and surpasses it on several individual tasks, including perplexity under both techniques \(10\.75 vs\. 11\.02 and 11\.25 vs\. 11\.33\), PIQA, and WinoGrande under GPTQ\. The greedy solver is also the only variant compatible with the adaptive setting: the ILP took upwards of two hours to converge on some budgets, whereas the greedy solve costsO​\(L​\|ℬ\|​log⁡L\)O\(L\\,\|\\mathcal\{B\}\|\\log L\)and runs in negligible time at deployment, where a new budget must be served on arrival\.

#### Sensitivity to the tail penalty\.

Figure[4](https://arxiv.org/html/2607.23047#S5.F4)sweeps the tail\-penalty strengthγ\\gammaand reports the relative change in perplexity against the unregularized allocator \(γ=0\\gamma=0\) across four \(model, budget\) settings\. Two observations follow\. First, we observe that the penalty helps: no setting is hurt relative toγ=0\\gamma=0, and perplexity typically decreases asγ\\gammagrows from 0\. Second, the effect saturates: beyondγ≈5\\gamma\\approx 5the curves are flat\. Saturation is expected from the mechanism of the penalty\. Onceγ\\gammais large enough to redirect the budget away from the tail\-most upgrades, only a small number of contested transitions remain whose ordering the penalty can still change; increasingγ\\gammafurther re\-ranks nothing, and the allocator returns the same plan\.

![Refer to caption](https://arxiv.org/html/2607.23047v1/x5.png)Figure 4:Effect of the tail penalty\. Relative change in WikiText\-2 perplexity versus the unregularized allocator \(γ=0\\gamma=0\) as the tail\-penalty strengthγ\\gammaincreases, across four \(model, budget\) settings;PPL0\\mathrm\{PPL\}\_\{0\}is theγ=0\\gamma=0perplexity for each curve\. \(Section[4\.3](https://arxiv.org/html/2607.23047#S4.SS3)\)

## 6Conclusion

We presentedMixQuant, a technique\-agnostic adaptive post\-training quantization framework in which one offline calibration serves any deployment budget through a single inexpensive greedy solve\. It starts from a mismatch that prior adaptive quantizers overlook: per\-layer sensitivity measured against an FP16 network does not reflect the fully quantized model that is actually deployed\.MixQuantcloses this gap with two components: mean\-field decoupled distortion scores, which marginalize each layer’s error over random quantized upstream contexts; and plan\-aware technique parameters, which are calibrated on the activations the deployed plan induces rather than on FP16 activations\. It additionally penalizes retaining low\-bit assignments through a tail regularizer that steers spare budget away from layers still stuck at the lowest bitwidths\. Across Llama\-3\.2\-3B, Llama\-2\-7B, and Mistral\-7B, under both AWQ and GPTQ,MixQuantproduces consistently better bit allocations than prior adaptive and mixed\-precision baselines\. Its greedy allocator matches an ILP solver at negligible deployment cost\. Future work includes extendingMixQuantto jointly allocate precision across weights, activations, and KV\-cache, toward maximizing efficiency in adaptive settings on the edge\.

## References

- S\. Ashkboos, A\. Mohtashami, M\. L\. Croci, B\. Li, P\. Cameron, M\. Jaggi, D\. Alistarh, T\. Hoefler, and J\. Hensman \(2024\)QuaRot: outlier\-free 4\-bit inference in rotated llms\.External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/b5b939436789f76f08b9d0da5e81af7c-Abstract-Conference.html)Cited by:[§2\.1](https://arxiv.org/html/2607.23047#S2.SS1.p1.1)\.
- Bit\-mixer: mixed\-precision networks with runtime bit\-width selection\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 5188–5197\.Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.23047#S2.SS2.p1.1)\.
- Z\. Dong, Z\. Yao, D\. Arfeen, A\. Gholami, M\. W\. Mahoney, and K\. Keutzer \(2020\)HAWQ\-V2: hessian aware trace\-weighted quantization of neural networks\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/d77c703536718b95308130ff2e5cf9ee-Abstract.html)Cited by:[3rd item](https://arxiv.org/html/2607.23047#S1.I1.i3.p1.1),[§2\.1](https://arxiv.org/html/2607.23047#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.23047#S4.SS1.p1.1),[§5](https://arxiv.org/html/2607.23047#S5.SS0.SSS0.Px2.p1.1)\.
- R\. Dumitru, V\. Yadav, R\. Maheshwary, P\. Clotan, S\. T\. Madhusudhan, and M\. Surdeanu \(2024\)Layer\-wise quantization: A pragmatic and effective method for quantizing llms beyond integer bit\-levels\.CoRRabs/2406\.17415\.External Links:[Link](https://doi.org/10.48550/arXiv.2406.17415),[Document](https://dx.doi.org/10.48550/ARXIV.2406.17415),2406\.17415Cited by:[3rd item](https://arxiv.org/html/2607.23047#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2607.23047#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.23047#S2.SS2.p1.1),[§5](https://arxiv.org/html/2607.23047#S5.SS0.SSS0.Px2.p2.1)\.
- V\. Egiazarian, A\. Panferov, D\. Kuznedelev, E\. Frantar, A\. Babenko, and D\. Alistarh \(2024\)Extreme compression of large language models via additive quantization\.pp\. 12284–12303\.External Links:[Link](https://proceedings.mlr.press/v235/egiazarian24a.html)Cited by:[§2\.1](https://arxiv.org/html/2607.23047#S2.SS1.p1.1)\.
- E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh \(2022\)GPTQ: accurate post\-training quantization for generative pre\-trained transformers\.CoRRabs/2210\.17323\.External Links:[Link](https://doi.org/10.48550/arXiv.2210.17323),[Document](https://dx.doi.org/10.48550/ARXIV.2210.17323),2210\.17323Cited by:[§1](https://arxiv.org/html/2607.23047#S1.SS0.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2607.23047#S2.SS1.p1.1),[§3](https://arxiv.org/html/2607.23047#S3.SS0.SSS0.Px1.p1.10)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§5](https://arxiv.org/html/2607.23047#S5.SS0.SSS0.Px1.p1.5)\.
- A\. Gholami, S\. Kim, Z\. Dong, Z\. Yao, M\. W\. Mahoney, and K\. Keutzer \(2021\)A survey of quantization methods for efficient neural network inference\.CoRRabs/2103\.13630\.External Links:[Link](https://arxiv.org/abs/2103.13630),2103\.13630Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p1.1)\.
- G\. E\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.CoRRabs/1503\.02531\.External Links:[Link](http://arxiv.org/abs/1503.02531),1503\.02531Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p1.1)\.
- S\. Islam, H\. Elmekki, A\. Elsebai, J\. Bentahar, N\. Drawel, G\. Rjoub, and W\. Pedrycz \(2024\)A comprehensive survey on applications of transformers for deep learning tasks\.Expert Syst\. Appl\.241,pp\. 122666\.External Links:[Link](https://doi.org/10.1016/j.eswa.2023.122666),[Document](https://dx.doi.org/10.1016/J.ESWA.2023.122666)Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p1.1)\.
- Q\. Jin, L\. Yang, and Z\. Liao \(2020\)Adabits: neural network quantization with adaptive bit\-widths\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 2146–2156\.Cited by:[3rd item](https://arxiv.org/html/2607.23047#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2607.23047#S1.p2.1),[§1](https://arxiv.org/html/2607.23047#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.23047#S2.SS2.p1.1)\.
- S\. Kim, C\. Hooper, A\. Gholami, Z\. Dong, X\. Li, S\. Shen, M\. W\. Mahoney, and K\. Keutzer \(2024\)SqueezeLLM: dense\-and\-sparse quantization\.pp\. 23901–23923\.External Links:[Link](https://proceedings.mlr.press/v235/kim24f.html)Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.23047#S2.SS2.p1.1)\.
- I\. Koryakovskiy, A\. Yakovleva, V\. Buchnev, T\. Isaev, and G\. Odinokikh \(2023\)One\-shot model for mixed\-precision quantization\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 7939–7949\.Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p2.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.pp\. 611–626\.External Links:[Link](https://doi.org/10.1145/3600006.3613165),[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p1.1)\.
- S\. Lee, S\. Woo, J\. Jin, C\. Lee, and E\. Park \(2025\)AMQ: enabling automl for mixed\-precision weight\-only quantization of large language models\.pp\. 35532–35550\.External Links:[Link](https://doi.org/10.18653/v1/2025.emnlp-main.1799),[Document](https://dx.doi.org/10.18653/V1/2025.EMNLP-MAIN.1799)Cited by:[§2\.1](https://arxiv.org/html/2607.23047#S2.SS1.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html)Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p1.1)\.
- S\. Li, X\. Ning, K\. Hong, T\. Liu, L\. Wang, X\. Li, K\. Zhong, G\. Dai, H\. Yang, and Y\. Wang \(2023\)LLM\-MQ: mixed\-precision quantization for efficient LLM deployment\.InThe Efficient Natural Language and Speech Processing Workshop with NeurIPS,External Links:[Link](https://neurips.cc/virtual/2023/81141)Cited by:[§3](https://arxiv.org/html/2607.23047#S3.SS0.SSS0.Px2.p1.6)\.
- Y\. Li, R\. Gong, X\. Tan, Y\. Yang, P\. Hu, Q\. Zhang, F\. Yu, W\. Wang, and S\. Gu \(2021\)BRECQ: pushing the limit of post\-training quantization by block reconstruction\.External Links:[Link](https://openreview.net/forum?id=POWv6hDd9XH)Cited by:[§4\.1](https://arxiv.org/html/2607.23047#S4.SS1.p1.1)\.
- J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han \(2024\)AWQ: activation\-aware weight quantization for on\-device LLM compression and acceleration\.External Links:[Link](https://proceedings.mlsys.org/paper%5C_files/paper/2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.23047#S1.SS0.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2607.23047#S2.SS1.p1.1),[§3](https://arxiv.org/html/2607.23047#S3.SS0.SSS0.Px1.p1.10),[§4\.1](https://arxiv.org/html/2607.23047#S4.SS1.p1.1)\.
- X\. Liu and K\. K\. Parhi \(2023\)Tensor decomposition for model reduction in neural networks: A review\.CoRRabs/2304\.13539\.External Links:[Link](https://doi.org/10.48550/arXiv.2304.13539),[Document](https://dx.doi.org/10.48550/ARXIV.2304.13539),2304\.13539Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p1.1)\.
- Z\. Liu, C\. Zhao, I\. Fedorov, B\. Soran, D\. Choudhary, R\. Krishnamoorthi, V\. Chandra, Y\. Tian, and T\. Blankevoort \(2025\)SpinQuant: LLM quantization with learned rotations\.External Links:[Link](https://openreview.net/forum?id=ogO6DGE6FZ)Cited by:[§2\.1](https://arxiv.org/html/2607.23047#S2.SS1.p1.1)\.
- Z\. Liu, M\. Sun, T\. Zhou, G\. Huang, and T\. Darrell \(2019\)Rethinking the value of network pruning\.In7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6\-9, 2019,External Links:[Link](https://openreview.net/forum?id=rJlnB3C5Ym)Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p1.1)\.
- A\. Misra, N\. Saoda, and T\. Abdelzaher \(2025\)Latency\-constrained input\-aware quantization of time series inference workflows at the edge\.InIEEE INFOCOM 2025 \- IEEE Conference on Computer Communications,Vol\.,pp\. 1–10\.External Links:[Document](https://dx.doi.org/10.1109/INFOCOM55648.2025.11044709)Cited by:[§3](https://arxiv.org/html/2607.23047#S3.SS0.SSS0.Px2.p1.6)\.
- M\. Nagel, R\. A\. Amjad, M\. van Baalen, C\. Louizos, and T\. Blankevoort \(2020\)Up or down? adaptive rounding for post\-training quantization\.pp\. 7197–7206\.External Links:[Link](http://proceedings.mlr.press/v119/nagel20a.html)Cited by:[§4\.1](https://arxiv.org/html/2607.23047#S4.SS1.p1.1)\.
- Y\. Park, J\. Hyun, S\. Cho, B\. Sim, and J\. W\. Lee \(2024\)Any\-precision LLM: low\-cost deployment of multiple, different\-sized llms\.pp\. 39682–39701\.External Links:[Link](https://proceedings.mlr.press/v235/park24e.html)Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.23047#S2.SS2.p1.1)\.
- B\. Rokh, A\. Azarpeyvand, and A\. Khanteymoori \(2023\)A comprehensive survey on model quantization for deep neural networks in image classification\.ACM Trans\. Intell\. Syst\. Technol\.14\(6\),pp\. 97:1–97:50\.External Links:[Link](https://doi.org/10.1145/3623402),[Document](https://dx.doi.org/10.1145/3623402)Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p1.1)\.
- M\. Shkolnik, B\. Chmiel, R\. Banner, G\. Shomron, Y\. Nahshan, A\. M\. Bronstein, and U\. C\. Weiser \(2020\)Robust quantization: one model to rule them all\.CoRRabs/2002\.07686\.External Links:[Link](https://arxiv.org/abs/2002.07686),2002\.07686Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.23047#S2.SS2.p1.1)\.
- X\. Sun, R\. Panda, C\. R\. Chen, N\. Wang, B\. Pan, A\. Oliva, R\. Feris, and K\. Saenko \(2024\)Improved techniques for quantizing deep networks with adaptive bit\-widths\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 957–967\.Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p3.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. Canton\-Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. Scialom \(2023\)Llama 2: open foundation and fine\-tuned chat models\.CoRRabs/2307\.09288\.External Links:[Link](https://doi.org/10.48550/arXiv.2307.09288),[Document](https://dx.doi.org/10.48550/ARXIV.2307.09288),2307\.09288Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p1.1)\.
- K\. Wang, Z\. Liu, Y\. Lin, J\. Lin, and S\. Han \(2019\)Haq: hardware\-aware automated quantization with mixed precision\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 8612–8620\.Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p2.1)\.
- G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han \(2023\)SmoothQuant: accurate and efficient post\-training quantization for large language models\.pp\. 38087–38099\.External Links:[Link](https://proceedings.mlr.press/v202/xiao23c.html)Cited by:[§2\.1](https://arxiv.org/html/2607.23047#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.23047#S4.SS1.p1.1)\.
- K\. Xu, Q\. Feng, X\. Zhang, and D\. Wang \(2022\)MultiQuant: training once for multi\-bit quantization of neural networks\.InProceedings of the Thirty\-First International Joint Conference on Artificial Intelligence,IJCAI\-2022\.External Links:[Link](http://dx.doi.org/10.24963/ijcai.2022/504),[Document](https://dx.doi.org/10.24963/ijcai.2022/504)Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p2.1)\.
- Z\. Yao, Z\. Dong, Z\. Zheng, A\. Gholami, J\. Yu, E\. Tan, L\. Wang, Q\. Huang, Y\. Wang, M\. W\. Mahoney, and K\. Keutzer \(2020\)HAWQV3: dyadic neural network quantization\.CoRRabs/2011\.10680\.External Links:[Link](https://arxiv.org/abs/2011.10680),2011\.10680Cited by:[§3](https://arxiv.org/html/2607.23047#S3.SS0.SSS0.Px2.p1.6),[§4\.1](https://arxiv.org/html/2607.23047#S4.SS1.p1.1)\.
- H\. Yu, H\. Li, H\. Shi, T\. S\. Huang, and G\. Hua \(2021\)Any\-precision deep neural networks\.pp\. 10763–10771\.External Links:[Link](https://doi.org/10.1609/aaai.v35i12.17286),[Document](https://dx.doi.org/10.1609/AAAI.V35I12.17286)Cited by:[§1](https://arxiv.org/html/2607.23047#S1.p2.1)\.
- C\. Zhang, J\. Cheng, G\. A\. Constantinides, and Y\. Zhao \(2024\)LQER: low\-rank quantization error reconstruction for llms\.pp\. 58763–58779\.External Links:[Link](https://proceedings.mlr.press/v235/zhang24j.html)Cited by:[§2\.1](https://arxiv.org/html/2607.23047#S2.SS1.p1.1)\.
- J\. Zhao, A\. Derakhshan, J\. K\. Hyman, J\. Dong, S\. A\. Jyothi, and I\. G\. Harris \(2026\)CoopQ: cooperative game inspired layerwise mixed precision quantization for llms\.pp\. 7566–7578\.External Links:[Link](https://aclanthology.org/2026.findings-acl.373/)Cited by:[3rd item](https://arxiv.org/html/2607.23047#S1.I1.i3.p1.1),[§2\.2](https://arxiv.org/html/2607.23047#S2.SS2.p1.1),[§5](https://arxiv.org/html/2607.23047#S5.SS0.SSS0.Px2.p2.1)\.