FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inference

# FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inference
Source: [https://arxiv.org/html/2608.24945](https://arxiv.org/html/2608.24945)
Gongwei LeeThanks:Equal contribution\.Affiliation:School of Computer Science and Technology, Soochow University, Suzhou, ChinaJi LiuThanks:Corresponding author: Ji Liu \(jiliuwork@gmail\.com\)Affiliation:Hithink Research, Hangzhou, ChinaJuncheng JiaJi WuAffiliation:Electronic Engineering, Tsinghua University, Beijing, China

###### Abstract

Recent years have witnessed remarkable achievements of Large Language Models \(LLMs\) in multiple domains, while the excessive resource requirements of LLMs hinder the deployment on resource\-constrained devices\. Although model quantization stands out as an effective approach, conventional quantization approaches typically incur severe performance degradation due to uniform bit\-width or simple heuristic sensitivity evaluation\. In this paper, we propose a novel Fisher information\-based Adaptive Mixed Precision Weight Quantization approach, i\.e\., FAMPWQ, which performs layer\-adaptive weight quantization for effective LLM inference on commodity GPUs\. First, we propose a system model with a novel Fisher information metric to measure the layer\-wise sensitivity to quantization\. Second, we propose a reinforcement learning\-based bit\-width allocator in FAMPWQ, which generates an adaptive bit\-width allocation strategy based on the Fisher information sensitivity metric\. Extensive experiments on 7 models and 5 benchmarks demonstrate that FAMPWQ significantly outperforms 7 baseline approaches in terms of PPL \(up to 3\.39 smaller\), accuracy \(up to 6\.87% higher\), and LLM\-as\-a\-judge comparison \(up to 76% win rate\)\.

## 1Introduction

Recent years have witnessed remarkable progress of Large Language Models \(LLMs\) across a wide range of applications[32](https://arxiv.org/html/2608.24945#bib.bib2);[31](https://arxiv.org/html/2608.24945#bib.bib44);[5](https://arxiv.org/html/2608.24945#bib.bib8)\. Most state\-of\-the\-art LLMs are built upon the Transformer architecture[42](https://arxiv.org/html/2608.24945#bib.bib16), achieving strong performance by scaling model size to hundreds billions[1](https://arxiv.org/html/2608.24945#bib.bib1)or trillions[8](https://arxiv.org/html/2608.24945#bib.bib54)of parameters\.

However, the prohibitive resource requirements of LLMs hinder their deployment on resource\-constrained devices, such as edge devices, consumer GPUs, or even inference GPUs\. For instance, LLaMA[39](https://arxiv.org/html/2608.24945#bib.bib38);[40](https://arxiv.org/html/2608.24945#bib.bib36);[10](https://arxiv.org/html/2608.24945#bib.bib37)spans from 7B to 405B, which may consume from 26GB to 1500GB memory with FP32, and up to 750GB memory with FP16\. Similarly, the scale of Qwen[3](https://arxiv.org/html/2608.24945#bib.bib39);[38](https://arxiv.org/html/2608.24945#bib.bib35);[49](https://arxiv.org/html/2608.24945#bib.bib33)can reach up to 235B corresponding to 435GB memory with FP16\. In addition, DeepSeek\-V4[8](https://arxiv.org/html/2608.24945#bib.bib54)goes even larger: its 1\.6T\-parameter MoE \(49B activated\) requires about 865GB in mixed FP4/FP8, which significantly exceeds the memory of GPUs\. In addition, the memory requirement scales from linear to quadratic with the sequence length\. To deploy LLMs on resource\-constrained devices, model quantization stands out as an effective approach\.

Figure 1:Compression\-frontier behavior on LLaMA\-7B with WikiText\-2\. FAMPWQ keeps PPL lower as the average bit\-width approaches 3 bits, where uniform quantization methods degrade rapidly\.Post\-Training Quantization \(PTQ\) approaches[54](https://arxiv.org/html/2608.24945#bib.bib25)directly quantize pre\-trained LLMs without architectural modifications or retraining, albeit typically incurring performance degradation\. However, existing PTQ approaches generally recognize the heterogeneous importance distribution of model weights[9](https://arxiv.org/html/2608.24945#bib.bib9);[15](https://arxiv.org/html/2608.24945#bib.bib12);[46](https://arxiv.org/html/2608.24945#bib.bib47), with their key differentiation stemming from the statistical approaches exploited to identify and preserve critical weights\. While some existing quantization approaches, e\.g\., GPTQ[12](https://arxiv.org/html/2608.24945#bib.bib11)and AWQ[22](https://arxiv.org/html/2608.24945#bib.bib17), successfully reduce memory consumption through fixed bit\-widths and outlier optimization, they nevertheless suffer from two fundamental limitations\. First, their uniform bit\-width allocation overlooks crucial layer\-wise sensitivity variations, particularly in attention layers\. Second, their localized outlier handling fails to account for global importance patterns across the LLM\. As a consequence, the existing PTQ approaches may bring unacceptably severe performance degradation in real\-life scenarios\.

While some mixed\-precision approaches, e\.g\., OWQ[19](https://arxiv.org/html/2608.24945#bib.bib14)and AMQ[20](https://arxiv.org/html/2608.24945#bib.bib3), attempt to address layer heterogeneity, they rely on coarse heuristics such as weight magnitude or raw gradient norms that fail to faithfully reflect quantization\-induced degradation\.

A fundamental challenge lies in the*heterogeneous sensitivity*of LLM layers to quantization\. Empirically, we find that certain layers \(particularlyattention value projectionsandMLP down\-projections\) are orders of magnitude more sensitive than others\. This reveals even a small number of aggressively quantized sensitive layers can disproportionately degrade model quality, while many redundant layers can tolerate extreme compression with negligible impact\. Accurately identifying which layers are critical therefore becomes the key to effective mixed\-precision quantization\.

Existing sensitivity metrics[11](https://arxiv.org/html/2608.24945#bib.bib10);[22](https://arxiv.org/html/2608.24945#bib.bib17), however, are ill\-suited to this task\. Weight magnitude and gradient norms capture only first\-order statistics and do not reflect the geometry of the loss surface under quantization\-specific perturbations\. Second\-order point estimates, including Hessian\-based[9](https://arxiv.org/html/2608.24945#bib.bib9)and standard Fisher\-based metrics, evaluate curvature only at the unperturbed weights and remain agnostic to the bit\-width\-specific noise that quantization actually injects\. To bridge this gap, we propose a*perturbation\-based Fisher Information*metric that directly injects quantization\-simulating perturbations into layer weights and measures the resulting shift in the FIM\. Different from these point\-estimate metrics, our formulation captures how quantization noise, rather than arbitrary parameter variations, distorts the local loss geometry, providing a principled and quantization\-specific layer sensitivity measure\.

![Refer to caption](https://arxiv.org/html/2608.24945v1/FAMPWQ.png)Figure 2:System model of FAMPWQ\.In this paper, we propose a Fisher Information\-based Adaptive Mixed Precision Weight Quantization \(FAMPWQ\) approach, i\.e\., a weight\-only mixed\-precision PTQ framework for fixed\-memory LLM deployment\. FAMPWQ introduces a quantization\-perturbation Fisher sensitivity metric that estimates per\-layer degradation than magnitude/gradient proxies, and exploits a low\-cost proxy optimizer to allocate layer bitwidths under a storage budget\. As shown in Figure[2](https://arxiv.org/html/2608.24945#S1.F2), FAMPWQ consists of two stages: a perturbation\-based Fisher sensitivity measurement stage and a Reinforcement Learning \(RL\)\-based adaptive bit\-width allocation stage\. As shown in Figure[1](https://arxiv.org/html/2608.24945#S1.F1), while uniform PTQ approaches are competitive around 4 bits, their PPL rises sharply below 3\.5 average bits\. By preserving sensitive layers and compressing tolerant layers more aggressively, FAMPWQ maintains significantly lower degradation below the 3\-bit frontier\.

The major contributions are as follows:

1. 1\.We propose a system model with a novel sensitivity measurement method based on a new Fisher Information metric for layer adaptive quantization\. The Fisher Information metric explicitly injects quantization\-simulating perturbations and measures the resulting Fisher shift to capture layer\-wise sensitivity to quantization loss\.
2. 2\.We propose an adaptive bit\-width allocator in FAMPWQ to enable storage\-constrained mixed\-precision search guided by quantization\-specific loss geometry\. The allocator generates an adaptive bit\-width allocation strategy based on Proximal Policy Optimization \(PPO\) and the quantization perturbation Fisher sensitivity of each layer, for layer\-wise quantization of LLMs\.
3. 3\.We implement FAMPWQ and maintain the compatibility with existing methods, e\.g\., GPTQ or AWQ\. We carry out extensive experiments on 7 models and 5 benchmarks to demonstrate that FAMPWQ significantly outperforms 7 baseline approaches in terms of PPL \(up to 3\.39 smaller\), accuracy \(up to 6\.87% higher\), and LLM\-as\-a\-judge comparison \(up to 76% win rate\)\.

## 2Related Work

Recent LLM post\-training quantization \(PTQ\) studies improve compression by reducing quantization error, correcting outliers, or calibrating quantized weights\. SmoothQuant[48](https://arxiv.org/html/2608.24945#bib.bib48)redistributes quantization difficulty between weights and activations, GPTAQ[21](https://arxiv.org/html/2608.24945#bib.bib15)mitigates error accumulation through calibration, and OmniQuant[36](https://arxiv.org/html/2608.24945#bib.bib45)and ABQ\-LLM[51](https://arxiv.org/html/2608.24945#bib.bib32)further explore adaptive clipping and bit\-balance strategies for low\-bit settings\. Other methods, such as SqueezeLLM[17](https://arxiv.org/html/2608.24945#bib.bib13)and OWQ[19](https://arxiv.org/html/2608.24945#bib.bib14), preserve salient weights or channels at higher precision\. These works show the importance of protecting sensitive parameters, but their adaptation is generally local and does not directly optimize layer\-wise precision under a global memory budget\.

This limitation has motivated mixed\-precision and search\-based allocation\. DeepSeek\-V4 models[8](https://arxiv.org/html/2608.24945#bib.bib54)adopt mixed FP4/FP8 expert\-aware quantization for MoE deployment at expert granularity\. AMQ[20](https://arxiv.org/html/2608.24945#bib.bib3)exploits activation\-guided mixed precision, HAQ[43](https://arxiv.org/html/2608.24945#bib.bib50)employs reinforcement learning for hardware\-aware bit\-width search, COPAL[26](https://arxiv.org/html/2608.24945#bib.bib51)formulates layer\-wise allocation as combinatorial optimization, and BitWeaver[13](https://arxiv.org/html/2608.24945#bib.bib52)investigates hardware\-efficient mixed\-precision layouts\. RL\-PTQ[45](https://arxiv.org/html/2608.24945#bib.bib53)applies reinforcement learning, but depends on repeated model\-level evaluation\. FAMPWQ focuses on layer\-wise sensitivity differences under a global memory budget; it estimates layer sensitivity via Fisher Information and employs a proxy\-guided PPO allocator to efficiently search storage\-constrained bit\-width configurations, achieving superb performance\.

## 3System Model and Problem Formulation

In this section, we present the system model of FAMPWQ, and formulate the problem to address in LLM quantization\.

### 3\.1System Model

While conventional quantization approaches employ identical bit\-widths for all layers, LLM layers exhibit highly uneven tolerance to quantization\. As shown in Figure[3](https://arxiv.org/html/2608.24945#S3.F3)\(a\), the measured layer sensitivity spans orders of magnitude within the same model, indicating that a few critical layers can dominate quantization\-induced degradation\. Figure[3](https://arxiv.org/html/2608.24945#S3.F3)\(b\) further shows that, under a comparable average precision budget, a mixed\-precision allocation achieves lower WikiText\-2 PPL than uniform INT4 by assigning higher bit\-widths to sensitive layers and lower bit\-widths to tolerant ones\. These observations motivate a layer\-adaptive quantization framework that explicitly measures sensitivity and allocates precision under a global storage constraint\.

As shown in Figure[2](https://arxiv.org/html/2608.24945#S1.F2), the system model of FAMPWQ consists of two stages: sensitivity measurement and layer\-adaptive quantization\. In the sensitivity measurement stage, we inject perturbation noise into layer weights and compute the Fisher information change to quantify each layer’s sensitivity \(Section[4\.1](https://arxiv.org/html/2608.24945#S4.SS1)\)\. In the layer\-adaptive quantization stage, an RL\-based method allocates appropriate bit\-widths to each layer based on the computed sensitivity \(Section[4\.2](https://arxiv.org/html/2608.24945#S4.SS2)\)\. The resulting allocation strategy is then applied via any compatible quantization method \(e\.g\., AWQ\)\.

Figure 3:Motivation for adaptive bitwidth quantization\.
### 3\.2Problem Formulation

Let us consider an LLMMMcomposed ofLLlayers\. The sensitivity value of Layerllis denoted bysl,l∈\[1,L\]s\_\{l\},l\\in\[1,L\]\. In order to quantize the LLM, we define a set of available bit\-width options,ℬ\\mathcal\{B\}, comprising one or more discrete choices \(e\.g\.,\{4,16\}\\\{4,16\\\}or\{2,3,4,8\}\\\{2,3,4,8\\\}\)\. We assume that the sensitivity of each layer is independent of its allocated bit\-widthqlq\_\{l\}\. Then, the accuracy degradation incurred by quantizing an individual layerΔ​A​c​cl\\Delta Acc\_\{l\}is proportional to its sensitivity and decreases exponentially with increasing bit\-width[53](https://arxiv.org/html/2608.24945#bib.bib22), as shown in Formula[1](https://arxiv.org/html/2608.24945#S3.E1):

Δ​A​c​cl​\(𝒬\)∝sl⋅e−α⁡\(ql/B\),\\displaystyle\\Delta Acc\_\{l\}\(\\mathcal\{Q\}\)\\propto s\_\{l\}\\cdot e^\{\-\\alpha\(q\_\{l\}/B\)\},\(1\)whereBBrepresents the original \(full\-precision\) bit\-width,𝒬=\{\(1,q1\),…,\(L,qL\)\}\\mathcal\{Q\}=\\\{\(1,q\_\{1\}\),\.\.\.,\(L,q\_\{L\}\)\\\}refers to a quantization bit\-width allocation strategy,α\\alphais a positive constant decay rate, andeeis Euler’s number\. We define the accuracy degradation by normalizing the exponential decay fluctuation induced by bit\-widthqlq\_\{l\}as Formula[2](https://arxiv.org/html/2608.24945#S3.E2)\.

Δ​A​c​cl​\(𝒬\)=A​c​co⋅sl∑i=1Lsi⋅\(e−α⁡\(ql/B\)−e−α\)\(1−e−α\),\\displaystyle\\Delta Acc\_\{l\}\(\\mathcal\{Q\}\)=\\frac\{Acc\_\{o\}\\cdot s\_\{l\}\}\{\\sum\_\{i=1\}^\{L\}s\_\{i\}\}\\cdot\\frac\{\(e^\{\-\\alpha\(q\_\{l\}/B\)\}\-e^\{\-\\alpha\}\)\}\{\(1\-e^\{\-\\alpha\}\)\},\(2\)whereA​c​coAcc\_\{o\}represents the original accuracy of LLMMMwithout quantization\. Afterwards, we can calculate the total accuracy degradation brought by all layers as defined in Formula[3](https://arxiv.org/html/2608.24945#S3.E3)\.

Δ​A​c​c​\(𝒬\)=A​c​co∑i=1Lsi​∑l=1Lsl⋅\(e−α⁡\(ql/B\)−e−α\)\(1−e−α\)\.\\displaystyle\\Delta Acc\(\\mathcal\{Q\}\)=\\frac\{Acc\_\{o\}\}\{\\sum\_\{i=1\}^\{L\}s\_\{i\}\}\\sum\_\{l=1\}^\{L\}s\_\{l\}\\cdot\\frac\{\(e^\{\-\\alpha\(q\_\{l\}/B\)\}\-e^\{\-\\alpha\}\)\}\{\(1\-e^\{\-\\alpha\}\)\}\.\(3\)
The problem we address in this work is how to find a bit\-width allocation strategy𝒬∗\\mathcal\{Q\}^\{\*\}so as to minimize the accuracy degradation while achieving the compression rate target as formulated in Formula[4](https://arxiv.org/html/2608.24945#S3.E4)\.

𝒬∗=argmin𝒬Δ​A​c​c​\(𝒬\),\\displaystyle\\mathcal\{Q\}^\{\*\}=\\operatorname\*\{argmin\}\_\{\\mathcal\{Q\}\}\\Delta Acc\(\\mathcal\{Q\}\),\(4\)s\.t\.\\displaystyle\\text\{s\.t\.\}\{∀\(l,ql\)∈𝒬∗,ql∈ℬ,∑l=1Lpl​ql≤R⋅B⋅∑l=1Lpl,\\displaystyle\\begin\{cases\}\\forall\\quad\(l,q\_\{l\}\)\\in\\mathcal\{Q\}^\{\*\},q\_\{l\}\\in\\mathcal\{B\},\\\\ \\sum\_\{l=1\}^\{L\}p\_\{l\}q\_\{l\}\\leq R\\cdot B\\cdot\\sum\_\{l=1\}^\{L\}p\_\{l\},\\end\{cases\}whereplp\_\{l\}is the number of parameters in Layerll, andRRis the target compression ratio\. This problem definition bridges the accuracy and memory requirement by optimizing bit\-width allocation strategy𝒬\\mathcal\{Q\}, where the objective functionΔ​A​c​c​\(𝒬\)\\Delta Acc\(\\mathcal\{Q\}\)explicitly represents the accuracy degradation, while the compression rate target guarantees hardware compatibility in terms of memory requirement\. This problem is complicated due to severe combinatorial explosion\. The search space grows exponentially as𝒪⁡\(\|ℬ\|L\)\\mathcal\{O\}\(\|\\mathcal\{B\}\|^\{L\}\)\. For instance, the search space reaches3224≈101063^\{224\}\\approx 10^\{106\}for LLaMA3\-8B \(224 layers from 32 blocks × 7 layers\) with only 3 bit\-width options for each layer, rendering exhaustive search computationally prohibitive even for offline quantization\.

## 4FAMPWQ Methodology

In this section, we detail the methodology of FAMPWQ\. We first describe the Fisher information\-based sensitivity measurement for each layer\. Then, we present the adaptive bit\-width allocation method that minimizes accuracy degradation while achieving the compression rate target\.

### 4\.1Fisher Information\-based Sensitivity

Fisher information quantifies the amount of information that observable data carries about unknown model parameters\. We leverage this property to measure the sensitivity of each layer to quantization noise: a layer whose Fisher information changes substantially under perturbation is highly sensitive\. Specifically, we compute the Fisher information of each layer with its original weights and with perturbed weights, and use the difference as the sensitivity measure\.

In order to quantify layer sensitivity, we inject a perturbationδ​θl\\delta\\theta\_\{l\}into the parameters of each Layerlland measure the resulting shift in the FIM\. While a generic perturbation, e\.g\., uniform or magnitude\-proportional noise, only reflects general parameter importance, we need to capture the*specific*noise incurred bybb\-bit quantization\. We therefore exploit Formula[5](https://arxiv.org/html/2608.24945#S4.E5)to generate the perturbation\.

δ​θl=Qb​\(θl\)−θl,\\delta\\theta\_\{l\}=Q\_\{b\}\(\\theta\_\{l\}\)\-\\theta\_\{l\},\(5\)whereQb​\(⋅\)Q\_\{b\}\(\\cdot\)denotes thebb\-bit quantize, i\.e\., the dequantization operator\. This definition ensuresθl\+δ​θl=Qb​\(θl\)\\theta\_\{l\}\+\\delta\\theta\_\{l\}=Q\_\{b\}\(\\theta\_\{l\}\), soδ​θl\\delta\\theta\_\{l\}is exactly the additive rounding perturbation introduced bybb\-bit quantization rather than an arbitrary direction\.

To simplify sensitivity evaluation, we adopt a layer\-independent strategy\. We add the perturbation to only one target layerθl\\theta\_\{l\}, while all other layers remain at their original parameters\. Then, we can get the layer after adding perturbation noise as defined in Formula[6](https://arxiv.org/html/2608.24945#S4.E6)\.

θlpert=θl\+δ​θl,\\theta\_\{l\}^\{\\text\{pert\}\}=\\theta\_\{l\}\+\\delta\\theta\_\{l\},\(6\)whereθl\\theta\_\{l\}refers to the original parameters of Layerllandθlpert\\theta\_\{l\}^\{\\text\{pert\}\}is the parameters with perturbation\.

![Refer to caption](https://arxiv.org/html/2608.24945v1/rl.png)Figure 4:PPO\-based adaptive bit\-width allocation method\.ℓ⁡\(θt−1\)=𝔼⁡\[min⁡\(𝔯t​\(θt−1\)​At,𝔠t​\(θt−1\)​At\)\]\\ell\(\\theta\_\{t\-1\}\)=\\mathbb\{E\}\\left\[\\min\\left\(\\mathfrak\{r\}\_\{t\}\(\\theta\_\{t\-1\}\)A\_\{t\},\\mathfrak\{c\}\_\{t\}\(\\theta\_\{t\-1\}\)A\_\{t\}\\right\)\\right\]andℓ⁡\(ϕ\)=At2\\ell\(\\phi\)=A\_\{t\}^\{2\}, which are exploited in Formulas[4\.2\.1](https://arxiv.org/html/2608.24945#S4.Ex1)and[18](https://arxiv.org/html/2608.24945#S4.E18)\.𝒮=\{s1,…,sL\}\\mathcal\{S\}=\\\{s\_\{1\},\.\.\.,s\_\{L\}\\\}represents the sensitivity\.We can calculate the gradients∇θlℓ​\(x\|θ\)\\nabla\_\{\\theta\_\{l\}\}\\ell\(x\|\\theta\), which denotes the first\-order derivative of the LLM\. Then, we can derive the empirical Fisher Information Matrix \(FIM\) as defined in Formula[7](https://arxiv.org/html/2608.24945#S4.E7)\.

F^​\(θl\)=1N​∑n=1N\(∇θlℓ​\(xn\|θ\)​∇θlℓ​\(xn\|θ\)⊤\)\.\\hat\{F\}\(\\theta\_\{l\}\)=\\frac\{1\}\{N\}\\sum^\{N\}\_\{n=1\}\(\\nabla\_\{\\theta\_\{l\}\}\\ell\(x\_\{n\}\|\\theta\)\\nabla\_\{\\theta\_\{l\}\}\\ell\(x\_\{n\}\|\\theta\)^\{\\top\}\)\.\(7\)We use the diagonal vector of FIM denoted byℱ⁡\(θl\)\\mathcal\{F\}\(\\theta\_\{l\}\)to approximate the original FIM[23](https://arxiv.org/html/2608.24945#bib.bib21)as defined in Formula[8](https://arxiv.org/html/2608.24945#S4.E8)\.

ℱ⁡\(θl\)=diag​\(I\|∇θlℓ​\(x\|θ\)\|⊙F^​\(θl\)\)\.\\mathcal\{F\}\(\\theta\_\{l\}\)=\\text\{diag\}\(I\_\{\|\\nabla\_\{\\theta\_\{l\}\}\\ell\(x\|\\theta\)\|\}\\odot\\hat\{F\}\(\\theta\_\{l\}\)\)\.\(8\)Since the diagonal vector of FIM only depends on the diagonal elements of the original matrix, we can simplify the calculation of Formulas[7](https://arxiv.org/html/2608.24945#S4.E7)and[8](https://arxiv.org/html/2608.24945#S4.E8)\. We can calculate each elementfi​\(θl\)∈ℱ⁡\(θl\)f\_\{i\}\(\\theta\_\{l\}\)\\in\\mathcal\{F\}\(\\theta\_\{l\}\)withiirepresenting theii\-th element inℱ⁡\(θl\)\\mathcal\{F\}\(\\theta\_\{l\}\)as defined in Formula[9](https://arxiv.org/html/2608.24945#S4.E9)\.

fi​\(θl\)=1N​∑n=1N∑j=1J\(∇θlℓ​\(xn\|θ\)\)\(i,j\)2,f\_\{i\}\(\\theta\_\{l\}\)=\\frac\{1\}\{N\}\\sum^\{N\}\_\{n=1\}\\sum^\{J\}\_\{j=1\}\(\\nabla\_\{\\theta\_\{l\}\}\\ell\(x\_\{n\}\|\\theta\)\)\_\{\(i,j\)\}^\{2\},\(9\)where∇θlℓ​\(xn\|θ\)\(i,j\)\\nabla\_\{\\theta\_\{l\}\}\\ell\(x\_\{n\}\|\\theta\)\_\{\(i,j\)\}represents the element with the index \(ii,jj\) in∇θlℓ​\(xn\|θ\)\\nabla\_\{\\theta\_\{l\}\}\\ell\(x\_\{n\}\|\\theta\)andJJrepresents the number of elements inii\-th row of∇θlℓ​\(xn\|θ\)\\nabla\_\{\\theta\_\{l\}\}\\ell\(x\_\{n\}\|\\theta\)\. Similarly, we can calculate the diagonal vector of FIM for the parameters with added perturbation noise denoted byℱ⁡\(θlpert\)\\mathcal\{F\}\(\\theta\_\{l\}^\{\\text\{pert\}\}\)\. See calculation details of FIM in Appendix[A\.2\.2](https://arxiv.org/html/2608.24945#A1.SS2.SSS2)\.

Finally, we take the FIM variation to indicate the sensitivity of a layer, which is defined in Formula[10](https://arxiv.org/html/2608.24945#S4.E10)\.

sl=‖ℱ⁡\(θlpert\)−ℱ⁡\(θl\)‖2‖ℱ⁡\(θl\)‖2,\{s\}\_\{l\}=\\frac\{\\left\\\|\\mathcal\{F\}\(\\theta\_\{l\}^\{\\text\{pert\}\}\)\-\\mathcal\{F\}\(\\theta\_\{l\}\)\\right\\\|\_\{2\}\}\{\\left\\\|\\mathcal\{F\}\(\\theta\_\{l\}\)\\right\\\|\_\{2\}\},\(10\)wheresl\{s\}\_\{l\}represents the sensitivity of Layerll,\|\|⋅\|\|2\|\|\\cdot\|\|\_\{2\}is the Euclidean norm over the diagonal FIM vector\. The resulting scalar sensitivity valuesls\_\{l\}is used in Formula[3](https://arxiv.org/html/2608.24945#S3.E3)\.

### 4\.2Adaptive Bit\-width Allocation

In this section, we present an RL\-based adaptive bit\-width allocation method\. Since the combinatorial problem defined in Formula[4](https://arxiv.org/html/2608.24945#S3.E4)is intractable, we transform it into a single loss function minimization problem as defined in Formula[11](https://arxiv.org/html/2608.24945#S4.E11)\.

min⁡ℒ⁡\(𝒬\)=Δ​A​c​c​\(𝒬\)\+P⁡\(ψ⁡\(𝒬\)\)⋅‖ψ⁡\(𝒬\)‖2,\\min\\mathcal\{L\}\(\\mathcal\{Q\}\)=\\Delta Acc\(\\mathcal\{Q\}\)\+P\(\\psi\(\\mathcal\{Q\}\)\)\\cdot\|\|\\psi\(\\mathcal\{Q\}\)\|\|^\{2\},\(11\)whereP⁡\(ψ⁡\(𝒬\)\)P\(\\psi\(\\mathcal\{Q\}\)\)is a penalty parameter andψ⁡\(𝒬\)\\psi\(\\mathcal\{Q\}\)is the storage loss compared with the compression rate targetRRas defined in Formula[12](https://arxiv.org/html/2608.24945#S4.E12)\.

ψ⁡\(𝒬\)=∑i=1Lpi⋅qi−R⋅B⋅∑i=1Lpi\.\\displaystyle\\psi\(\\mathcal\{Q\}\)=\\sum\_\{i=1\}^\{L\}p\_\{i\}\\cdot q\_\{i\}\-R\\cdot B\\cdot\\sum\_\{i=1\}^\{L\}p\_\{i\}\.\(12\)In addition,P⁡\(ψ⁡\(𝒬\)\)P\(\\psi\(\\mathcal\{Q\}\)\)depends onψ⁡\(𝒬\)\\psi\(\\mathcal\{Q\}\)as defined in Formula[13](https://arxiv.org/html/2608.24945#S4.E13)\.

P⁡\(ψ⁡\(𝒬\)\)=\{Ppenalty\>0,ifψ\(𝒬\)\>0,Preward≤0,otherwise,\\displaystyle P\(\\psi\(\\mathcal\{Q\}\)\)=\\begin\{cases\}P\_\{\\text\{penalty\}\}\>0,\\text\{if \}\\psi\(\\mathcal\{Q\}\)\>0,\\\\ P\_\{\\text\{reward\}\}\\leq 0,\\text\{otherwise\},\\end\{cases\}\(13\)wherePpenaltyP\_\{\\text\{penalty\}\}is the penalty when the quantization does not achieve the targeted compression rate andPrewardP\_\{\\text\{reward\}\}refers to the rewards brought by the extra quantization compression\. BothPpenaltyP\_\{\\text\{penalty\}\}andPrewardP\_\{\\text\{reward\}\}are constant values\.

While RL is an effective approach for complex combinatorial optimization problems[6](https://arxiv.org/html/2608.24945#bib.bib18), we adopt Proximal Policy Optimization \(PPO\)[35](https://arxiv.org/html/2608.24945#bib.bib30)for bit\-width allocation\. As shown in Figure[4](https://arxiv.org/html/2608.24945#S4.F4), the architecture consists of an agent and the environment\. The agent generates the bit\-width allocation strategy while the environment provides feedback through a reward function\. The agent consists of two modules: the actor generates bit\-width allocation strategies and the critic guides policy optimization\. Both the actor and critic modules are implemented as lightweight residual networks \(see architecture details in Appendix[A\.6](https://arxiv.org/html/2608.24945#A1.SS6)\)\. During the quantization phase, both modules are first trained, after which the actor generates the final allocation strategy\. The actor takes the layer ID, the current allocation𝒬\\mathcal\{Q\}, per\-layer parameter counts, per\-layer sensitivity scores, and the candidate bit\-widthsℬ\\mathcal\{B\}as input\. It then outputs the bit\-width for the corresponding layer\. The critic receives the same inputs and produces a scalar value estimate to guide policy optimization\.

#### 4\.2\.1Training Process

The training process contains multiple epochs, each of which consists ofLLsteps\. At the beginning of the training, the bit\-width allocation strategy𝒬\\mathcal\{Q\}is initialized to the highest selectable bit\-width in each layer, i\.e\.,∀\(l,ql\)∈𝒬0,ql∈ℬ\\forall\(l,q\_\{l\}\)\\in\\mathcal\{Q\}\_\{0\},q\_\{l\}\\in\\mathcal\{B\}, which is exploited for the first epoch\. For each epoch, at Steptt, we denote the parameters of the actor network byθt\\theta\_\{t\}and that of the critic network byϕt\\phi\_\{t\}\. We denote the bit\-width for Layerllat Stepttbyqltq\_\{l\}^\{t\}\. Then, the actor network generates the bit\-widthqttq^\{t\}\_\{t\}for Layertt, and update𝒬t−1\\mathcal\{Q\}\_\{t\-1\}to𝒬t\\mathcal\{Q\}\_\{t\}by replacingqtt−1q\_\{t\}^\{t\-1\}byqttq\_\{t\}^\{t\}\. In addition, we denote the scalar value of the critic network byVϕt​\(𝒬t\)V\_\{\\phi\_\{t\}\}\(\\mathcal\{Q\}\_\{t\}\)\. Then, we compute the Temporal\-Difference \(TD\) advantage[34](https://arxiv.org/html/2608.24945#bib.bib31)AtA\_\{t\}at Stepttas defined in Formula[14](https://arxiv.org/html/2608.24945#S4.E14)\.

At=ℒ⁡\(𝒬t−1\)\+γ​Vϕt−1​\(𝒬t\)−Vϕt−1​\(𝒬t−1\),A\_\{t\}=\\mathcal\{L\}\(\\mathcal\{Q\}\_\{t\-1\}\)\+\\gamma V\_\{\\phi\_\{t\-1\}\}\(\\mathcal\{Q\}\_\{t\}\)\-V\_\{\\phi\_\{t\-1\}\}\(\\mathcal\{Q\}\_\{t\-1\}\),\(14\)whereℒ⁡\(𝒬t−1\)\\mathcal\{L\}\(\\mathcal\{Q\}\_\{t\-1\}\)is defined in Formula[11](https://arxiv.org/html/2608.24945#S4.E11),γ∈\(0,1\)\\gamma\\in\(0,1\)is a discount factor that controls the trade\-off between immediate and future rewards\. The actor network is updated by minimizing the clipped surrogate objective[35](https://arxiv.org/html/2608.24945#bib.bib30)while ensuring stable policy improvements as defined in Formula[4\.2\.1](https://arxiv.org/html/2608.24945#S4.Ex1)\.

θt←θt−1−ηθ∇θt−1𝔼\[min\(𝔯t\(θt−1\)At,\\displaystyle\\theta\_\{t\}\\leftarrow\\theta\_\{t\-1\}\-\\eta\_\{\\theta\}\\nabla\_\{\\theta\_\{t\-1\}\}\\mathbb\{E\}\\big\[\\min\\big\(\\mathfrak\{r\}\_\{t\}\(\\theta\_\{t\-1\}\)A\_\{t\},𝔠t\(θt−1\)At\)\]\\displaystyle\\mathfrak\{c\}\_\{t\}\(\\theta\_\{t\-1\}\)A\_\{t\}\\big\)\\big\]\(15\)where𝔼⁡\[⋅\]\\mathbb\{E\}\\left\[\\cdot\\right\]corresponds to the empirical average,ηθ\\eta\_\{\\theta\}is a constant learning rate of the actor network,𝔠t​\(θt−1\)\\mathfrak\{c\}\_\{t\}\(\\theta\_\{t\-1\}\)refers to a clip reward defined in Formula[16](https://arxiv.org/html/2608.24945#S4.E16):

𝔠t​\(θt−1\)=clip​\(𝔯t​\(θt−1\),1−ϵ,1\+ϵ\),\\displaystyle\\mathfrak\{c\}\_\{t\}\(\\theta\_\{t\-1\}\)=\\text\{clip\}\(\\mathfrak\{r\}\_\{t\}\(\\theta\_\{t\-1\}\),1\-\\epsilon,1\+\\epsilon\),\(16\)whereϵ\\epsilonis a small constant controlling the policy update range,𝔯t​\(θ\)\\mathfrak\{r\}\_\{t\}\(\\theta\)represents the policy\-dependent reward as defined in Formula[17](https://arxiv.org/html/2608.24945#S4.E17)\.

𝔯t​\(θt−1\)=πθt​\(qtt\|𝒬t−1\)πθt−1​\(qtt\|𝒬t−1\),\\displaystyle\\mathfrak\{r\}\_\{t\}\(\\theta\_\{t\-1\}\)=\\frac\{\\pi\_\{\\theta\_\{t\}\}\(q\_\{t\}^\{t\}\|\\mathcal\{Q\}\_\{t\-1\}\)\}\{\\pi\_\{\\theta\_\{t\-1\}\}\(q\_\{t\}^\{t\}\|\\mathcal\{Q\}\_\{t\-1\}\)\},\(17\)whereπθt​\(qtt\|𝒬t−1\)\\pi\_\{\\theta\_\{t\}\}\(q\_\{t\}^\{t\}\|\\mathcal\{Q\}\_\{t\-1\}\)represents the probability to generateqttq\_\{t\}^\{t\}with the actor networkθt\\theta\_\{t\}and the allocation strategy𝒬t−1\\mathcal\{Q\}\_\{t\-1\}\.

Simultaneously, the critic network is updated to minimize the squared TD advantage:

ϕt←ϕt−1−ηϕ​∇ϕt−1At2,\\phi\_\{t\}\\leftarrow\\phi\_\{t\-1\}\-\\eta\_\{\\phi\}\\nabla\_\{\\phi\_\{t\-1\}\}A\_\{t\}^\{2\},\(18\)whereηϕ\\eta\_\{\\phi\}is a constant learning rate of the critic network\. See training details in Appendix[A\.6](https://arxiv.org/html/2608.24945#A1.SS6)\.

#### 4\.2\.2Inference Process

The inference process consists ofLLsteps\. Similar to the training process, the bit\-width allocation strategy𝒬0\\mathcal\{Q\}\_\{0\}is initialized to the highest selectable bit\-width\. At each steptt, the actor module generates a bit\-widthqttq\_\{t\}^\{t\}for Layerttand updates𝒬t−1\\mathcal\{Q\}\_\{t\-1\}by replacingqtt−1q\_\{t\}^\{t\-1\}withqttq\_\{t\}^\{t\}\. AfterLLsteps,𝒬L\\mathcal\{Q\}\_\{L\}contains the generated bit\-widths and is used as the adaptive allocation strategy to quantize the LLM\.

## 5Experiments

In this section, we present the experimental results\. We first describe the experimental setup and then compare FAMPWQ with 7 baseline approaches across 7 models and 5 benchmarks\. We implement FAMPWQ in Python while maintaining compatibility with existing quantization backends such as GPTQ, AWQ, and OmniQuant\.

Table 1:PPL↓\\downarrowcomparison on LLaMA\-7B and LLaMA\-13B with 4\-bit and 3\-bit average quantization\.Boldindicates the lowest PPL andunderlinedindicates the second lowest\.### 5\.1Experimental Setup

We take 6 state\-of\-the\-art PTQ approaches, i\.e\., GPTQ[12](https://arxiv.org/html/2608.24945#bib.bib11), GPTQv2[21](https://arxiv.org/html/2608.24945#bib.bib15), AMQ[20](https://arxiv.org/html/2608.24945#bib.bib3), OmniQuant[36](https://arxiv.org/html/2608.24945#bib.bib45), OWQ[19](https://arxiv.org/html/2608.24945#bib.bib14), and AWQ[22](https://arxiv.org/html/2608.24945#bib.bib17)as baseline approaches\. We take a simple quantization approach by mapping floating\-point values to their nearest discrete levels, which is denoted by Round\-To\-Nearest \(RTN\), as a baseline approach\. We evaluate on 7 LLMs, i\.e\., LLaMA\-7B, LLaMA\-13B[39](https://arxiv.org/html/2608.24945#bib.bib38), LLaMA2\-7B\-chat, LLaMA2\-13B\-chat[40](https://arxiv.org/html/2608.24945#bib.bib36), Qwen2\.5\-7B, Qwen2\.5\-14B[50](https://arxiv.org/html/2608.24945#bib.bib4)and Mistral\-7B\-v0\.1[16](https://arxiv.org/html/2608.24945#bib.bib20)\. In addition, we utilize 5 benchmarks: Wikitext\-2 \(Wiki2\)[28](https://arxiv.org/html/2608.24945#bib.bib5), Penn Treebank \(PTB\)[27](https://arxiv.org/html/2608.24945#bib.bib40), C4[33](https://arxiv.org/html/2608.24945#bib.bib6), lm\-evaluation\-harness[14](https://arxiv.org/html/2608.24945#bib.bib19), and Vicuna[7](https://arxiv.org/html/2608.24945#bib.bib29), to evaluate the PPL, the accuracy, and the LLM\-as\-a\-judge comparison of diverse quantization approaches\.

The hyperparameters used in our experiments are shown in Table[A\.3](https://arxiv.org/html/2608.24945#A1.T3)\. All experiments are conducted on NVIDIA 4090 GPUs using PyTorch 2\.0[30](https://arxiv.org/html/2608.24945#bib.bib43)with HuggingFace integration[47](https://arxiv.org/html/2608.24945#bib.bib7), to ensure the consistent comparison with baseline approaches\. For fair evaluation, we maintain identical experimental settings across all quantization approaches, including calibration data \(128 randomly sampled sequences from C4\)\.

Figure 5:Zero\-shot accuracy comparison under 3\-bit quantization on Qwen2\.5\-7B\. FAMPWQ \(red\) consistently outperforms baselines across five reasoning tasks\.
### 5\.2Experimental Results

In this section, we present the experimental results in terms of the PPL with 3 benchmarks, the accuracy on zero\-shot tasks, and the evaluation of FAMPWQ based on LLM\-as\-a\-judge comparison\.

#### 5\.2\.1Perplexity Evaluation

As shown in Table[1](https://arxiv.org/html/2608.24945#S5.T1), FAMPWQ consistently achieves excellent performance in terms of PPL across 2 LLMs and 3 benchmarks when performing 4\-bit and 3\-bit quantization on average\. To mitigate the influence of evaluation randomness, all PPL results reported in this section are averaged over 3 independent runs with different random seeds for calibration sampling, and we report the mean value across runs\. With LLaMA\-7B and 4 average bits, FAMPWQ attains the PPLs of 5\.81 on WikiText\-2 and 10\.34 on PTB, outperforming the strongest baseline \(AWQ\) by 0\.02 and 0\.08, respectively\. While FAMPWQ corresponds to slightly higher \(0\.01\) PPL compared with AWQ and OmniQuant on C4, it still significantly outperforms other baseline approaches \(from 0\.13 to 3\.39\)\. We observe similar results with LLaMA\-13B at 4 bits\. Under 3\-bit quantization, the advantage of FAMPWQ becomes substantially larger: FAMPWQ outperforms all baselines on both models, reducing average PPL by up to 2\.37 over GPTQ and 0\.40 over AWQ on LLaMA\-7B\. In addition, FAMPWQ outperforms baseline approaches \(from 0\.04 to 2\.86 in average PPL\) on Qwen2\.5\-7B, Qwen2\.5\-14B, and Mistral\-7B\-v0\.1 \(see Table[A\.9](https://arxiv.org/html/2608.24945#A1.T9)in Appendix[A\.7\.5](https://arxiv.org/html/2608.24945#A1.SS7.SSS5)\)\.

Table 2:Zero\-shot reasoning accuracy \(↑\\uparrow\) of quantized Qwen2\.5\-7B under 3\-bit quantization\.
#### 5\.2\.2Zero\-shot Reasoning Evaluation

We evaluate zero\-shot reasoning using lm\-evaluation\-harness\. Figure[5](https://arxiv.org/html/2608.24945#S5.F5)visualizes the 3\-bit results on Qwen2\.5\-7B\. Compared with RTN and GPTQ, which it outperforms by 6\.87% and 6\.69% in average accuracy respectively, FAMPWQ avoids the severe shrinkage of the radar profile, indicating better preservation of general reasoning ability under aggressive compression\. Compared with stronger quantized baselines such as AMQ, AWQ and OmniQuant, FAMPWQ expands the outer boundary on most tasks and remains closer to the FP16 reference, especially on BoolQ, ARC\-E, and WinoGrande, demonstrating that layer\-adaptive bit\-width allocation is critical for preserving model quality under aggressive compression\.

Figure 6:LLM\-as\-a\-judge comparison based on GPT\-3\.5\-turbo with 4\-bit quantized LLaMA2\-13B\-chat\.
#### 5\.2\.3LLM\-as\-a\-judge Evaluation

To comprehensively evaluate the performance of FAMPWQ, we compare FAMPWQ with baseline approaches based on the quantized versions of the instruction\-tuned LLaMA2\-13B\-chat model exploiting the Vicuna benchmark[7](https://arxiv.org/html/2608.24945#bib.bib29)\. We use GPT\-3\.5\-turbo[29](https://arxiv.org/html/2608.24945#bib.bib42)as a judge across 80 diverse questions\. We mitigate position bias through bidirectional comparison, which results in 160 trials per comparison\. As shown in Figure[6](https://arxiv.org/html/2608.24945#S5.F6), FAMPWQ achieves substantially higher win rates than all baseline approaches \(76% against RTN, 69% against OWQ, 64% against GPTQ, 72% against GPTQv2, 61% against AWQ and 54% against OmniQuant\), where the win rate excludes tie samples\. A two\-sided binomial test on the head\-to\-head trials confirms that the comparison against the strong AWQ baseline is statistically significant \(p<0\.05p<0\.05\), reducing the risk that the observed judge preference is caused by evaluation noise\. Additional 3\-bit Vicuna\-Bench results are reported in Appendix[A\.7\.11](https://arxiv.org/html/2608.24945#A1.SS7.SSS11)\.

#### 5\.2\.4Inference acceleration

As shown in Figure[7](https://arxiv.org/html/2608.24945#S5.F7), FAMPWQ delivers a clear throughput advantage over FP16 \(up to42%42\\%\) and both intra\-layer \(OWQ\) \(up to69%69\\%\) and activation\-guided \(AMQ\) \(up to28%28\\%\) mixed\-precision baselines, while remaining slower than uniform low\-bit AWQ due to heterogeneous kernel scheduling\. AWQ retains the highest absolute throughput \(2\.44×2\.44\\timeson 7B,2\.10×2\.10\\timeson 13B\) by exploiting uniform 4\-bit kernels, while FAMPWQ delivers consistently higher accuracy at the same or lower average bit\-width as shown in Table[1](https://arxiv.org/html/2608.24945#S5.T1)\.

Figure 7:Inference speedup over FP16 on NVIDIA 4090 for 3\-bit average quantization\.Table 3:PPL↓\\downarrowof LLaMA\-7B and Qwen2\.5\-7B on WikiText\-2 with diverse perturbation strategies\.Boldindicates the lowest PPL\.x%​δ1​θx\\%\\delta\_\{1\}\\thetarefers to theδ1\\delta\_\{1\}strategy withϵ=x%\\epsilon=x\\%\.x%​δ2​θx\\%\\delta\_\{2\}\\thetadenotes theδ2\\delta\_\{2\}strategy withβ=x%\\beta=x\\%\.xxbitδ3​θ\\delta\_\{3\}\\thetarepresents theδ3\\delta\_\{3\}strategy withb=xb=x\.
#### 5\.2\.5Computational Cost

Figure 8:Preprocessing time breakdown of FAMPWQ across 5 models\. Total cost remains below 1 GPU\-hour even for 14B\-scale models\. Fisher sensitivity computation dominates, while RL search takes only 1–3 minutes\.The preprocessing overhead of FAMPWQ consists of three components: Fisher sensitivity computation, RL\-based bit\-width search, and the quantization itself\. As shown in Figure[8](https://arxiv.org/html/2608.24945#S5.F8), the total preprocessing time remains below 1 GPU\-hour for all models tested, including 14B\-scale models\. Fisher sensitivity computation dominates the cost \(24–64 minutes\) and scales with model size\. The RL search is lightweight \(<<5 minutes on a single GPU\), as it operates on a proxy model rather than performing full quantization at each step\. The entire preprocessing is a one\-time offline cost, amortized across all subsequent inference\.

### 5\.3Ablation Study

In this section, we analyze the impact of diverse sensitivity measurement methods and the comparison of diverse bit\-width allocation methods\.

#### 5\.3\.1Artificial Perturbation

We compareδ​θl\\delta\\theta\_\{l\}in Eq\.[5](https://arxiv.org/html/2608.24945#S4.E5)against two generic alternatives: magnitude\-proportional uniform noise \(δ1\\delta\_\{1\}\) and Bernoulli\-masked weight\-proportional noise \(δ2\\delta\_\{2\}\) \(see Appendix[A\.4](https://arxiv.org/html/2608.24945#A1.SS4)for details\)\. As shown in Table[3](https://arxiv.org/html/2608.24945#S5.T3), the quantization perturbation form atb=4b\{=\}4yields the lowest PPL on both models, beatingδ1\\delta\_\{1\}by up to 0\.56 andδ2\\delta\_\{2\}by up to 0\.23\. Onlyδ3\\delta\_\{3\}matches the actualbb\-bit rounding perturbation in both direction and magnitude;δ1\\delta\_\{1\}andδ2\\delta\_\{2\}are agnostic to the target bit\-width and therefore reflect only generic parameter importance\.

#### 5\.3\.2Bit\-width Allocation Strategy

We compare our PPO\-based method with four alternative methods: greedy search, Bayesian optimization, simulated annealing, and a genetic algorithm\. As shown in Table[4](https://arxiv.org/html/2608.24945#S5.T4), the RL\-based adaptive allocation strategy achieves substantially lower average PPL than these alternatives \(up to 1\.50 lower than Greedy, 1\.33 lower than Bayesian optimization, 1\.90 lower than simulated annealing, and 0\.45 lower than the genetic algorithm\), revealing the superb performance of our allocation method\.

#### 5\.3\.3Sensitivity Metric Comparison

![Refer to caption](https://arxiv.org/html/2608.24945v1/sensitivity_metric.png)Figure 9:Pearson correlation \(rr\) between sensitivity metrics and actual quantization degradation \(Oracle\)\.Table 4:PPL↓\\downarrowwith different bit\-width allocation methods\.Boldindicates the lowest PPL\.We compare the FIM\-based sensitivity metric against random allocation, weight magnitude \(‖W‖2\\\|W\\\|\_\{2\}\), and Oracle sensitivity \(actual per\-layer PPL increase\)\. As shown in Figure[9](https://arxiv.org/html/2608.24945#S5.F9), our FIM\-based sensitivity metric achieves a Pearson correlation ofr=0\.91,p<10−14r\{=\}0\.91,p<10^\{\-14\}on LLaMA\-7B andr=0\.88,p<10−17r\{=\}0\.88,p<10^\{\-17\}on Qwen2\.5\-7B with Oracle sensitivity\.This significant correlation directly leads to better quantization performance: at 3\.5\-bit average, the FIM\-based metric limitsΔ\\DeltaPPL to \+0\.42, while weight magnitude yields \+0\.85 and random allocation yields \+1\.19 \(see detals in Appendix Table[A\.14](https://arxiv.org/html/2608.24945#A1.T14)\)\.

#### 5\.3\.4α\\alphaSensitivity Analysis

Figure 10:PPL on WikiText\-2 as a function of decay rateα\\alphafor LLaMA\-7B and Qwen2\.5\-7B under 3\-bit average quantization\. Stars mark the optimalα\\alphafor each model\. The shaded green region indicates the robust zone \(α∈\[15,25\]\\alpha\\in\[15,25\]\) where PPL variation is minimal \(<<0\.3\)\. Dashed lines show FP16 baselines\.As shown in Figure[10](https://arxiv.org/html/2608.24945#S5.F10), the decay rate parameterα\\alphaexhibits a broad optimal region\. For LLaMA\-7B, the optimalα=18\\alpha\{=\}18yields a PPL of 6\.35, while anyα∈\[15,25\]\\alpha\\in\[15,25\]produces PPL within 0\.09 of the optimum\. For Qwen2\.5\-7B,α=20\\alpha\{=\}20is optimal \(PPL 8\.27\), with<<0\.06 variation across the robust zone\. This robustness toα\\alphasimplifies hyperparameter selection and confirms that the exponential decay model in Formula[1](https://arxiv.org/html/2608.24945#S3.E1)is a stable approximation\.

## 6Conclusion

In this work, we propose a novel Fisher information\-based Adaptive Mixed Precision Weight Quantization approach, i\.e\., FAMPWQ\. FAMPWQ introduces a novel perturbation\-based Fisher Information metric to capture layer\-wise quantization\-specific sensitivity\. In addition, FAMPWQ couples the metric with a new PPO\-based allocation method to efficiently generate an adaptive bit\-width allocation strategy with superb performance\. Extensive experiments on 7 models and 5 benchmarks demonstrate that FAMPWQ outperforms 7 baselines in PPL \(up to 3\.39 smaller\), accuracy \(up to 6\.87% higher\), and LLM\-as\-a\-judge comparison \(up to 76% win rate\), with particularly strong advantages at the 3\-bit compression frontier\.

## Limitations

Several limitations of FAMPWQ should be acknowledged\. First, mixed\-precision quantization can reduce inference throughput because heterogeneous bit\-widths are less compatible with optimized uniform\-precision kernels; our focus is therefore memory\-constrained deployment rather than peak tokens\-per\-second\. Second, FAMPWQ currently targets weight\-only quantization \(WxxA16\), leaving joint weight\-activation quantization to future work\. Third, our experiments focus on dense Transformer models, so effectiveness on Mixture\-of\-Experts architectures remains untested\. Finally, Fisher sensitivity estimation is a one\-time offline cost but remains the dominant preprocessing component, motivating lighter sensitivity proxies\.

## Acknowledgements

This work was partially \(for Juncheng Jia\) supported by the Priority Academic Program Development of Jiangsu Higher Education Institutions, Suzhou Frontier Science and Technology Program \(Project SYG202310\)\.

## References

- Agarwalet al\.\(2025\)S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p1.1)\.
- Amari \(1998\)S\. AmariNatural gradient works efficiently in learning\.Neural Computation10\(2\),pp\. 251–276\.External Links:ISSN 0899\-7667,[Document](https://dx.doi.org/10.1162/089976698300017746),LCCN 4Cited by:[§A\.2\.2](https://arxiv.org/html/2608.24945#A1.SS2.SSS2.p1.1)\.
- Baiet al\.\(2023\)J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang,et al\.Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p2.1)\.
- Benestyet al\.\(2009\)J\. Benesty, J\. Chen, Y\. Huang, and I\. CohenNoise reduction in speech processing\.Vol\.2,Springer Science & Business Media\.Cited by:[§A\.3](https://arxiv.org/html/2608.24945#A1.SS3.p1.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, and G\. SastryLanguage models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p1.1)\.
- Cappartet al\.\(2021\)Q\. Cappart, T\. Moisan, L\. Rousseau, I\. Prémont\-Schwarz, and A\. A\. CireCombining reinforcement learning and constraint programming for combinatorial optimization\.InProceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 3677–3687\.Cited by:[§4\.2](https://arxiv.org/html/2608.24945#S4.SS2.p2.1)\.
- Chianget al\.\(2023\)W\. Chiang, Z\. Li, Z\. Lin, Y\. Sheng, Z\. Wu, H\. Zhang, L\. Zheng, S\. Zhuang, Y\. Zhuang, J\. E\. Gonzalez,et al\.Vicuna: an open\-source chatbot impressing gpt\-4 with 90%\* chatgpt quality\.See https://vicuna\. lmsys\. org \(accessed 14 April 2023\)2\(3\),pp\. 6\.Cited by:[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1),[§5\.2\.3](https://arxiv.org/html/2608.24945#S5.SS2.SSS3.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-v4 technical report\.Note:Technical reportExternal Links:[Link](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf)Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p1.1),[§1](https://arxiv.org/html/2608.24945#S1.p2.1),[§2](https://arxiv.org/html/2608.24945#S2.p2.1)\.
- Donget al\.\(2019\)Z\. Dong, Z\. Yao, A\. Gholami, M\. Mahoney, and K\. KeutzerHAWQ: hessian aware quantization of neural networks with mixed\-precision\.In2019 IEEE/CVF International Conference on Computer Vision \(ICCV\),Seoul, Korea \(South\),pp\. 293–302\.External Links:[Document](https://dx.doi.org/10.1109/iccv.2019.00038)Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p3.1),[§1](https://arxiv.org/html/2608.24945#S1.p6.1)\.
- Dubeyet al\.\(2024\)A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.The llama 3 herd of models\.arXiv e\-prints,pp\. arXiv–2407\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p2.1)\.
- Frantar and Alistarh \(2022\)E\. Frantar and D\. AlistarhOptimal brain compression: a framework for accurate post\-training quantization and pruning\.Advances in Neural Information Processing Systems35,pp\. 4475–4488\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p6.1)\.
- Frantaret al\.\(2023\)E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. AlistarhGPTQ: accurate post\-training quantization for generative pre\-trained transformers\.arXiv\.External Links:2210\.17323,[Document](https://dx.doi.org/10.48550/arXiv.2210.17323)Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p3.1),[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Gagnonet al\.\(2025\)G\. Gagnon, S\. Malla, Y\. Kang, and L\. LiuBitWeaver: read\-time truncation in memory\.InProceedings of the 39th ACM International Conference on Supercomputing \(ICS ’25\),pp\. 13–25\.External Links:[Document](https://dx.doi.org/10.1145/3721145.3725779)Cited by:[§2](https://arxiv.org/html/2608.24945#S2.p2.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. ZouA framework for few\-shot language model evaluation\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.10256836),[Link](https://zenodo.org/records/10256836)Cited by:[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Gonget al\.\(2024\)Z\. Gong, J\. Liu, J\. Wang, X\. Cai, D\. Zhao, and R\. YanWhat makes quantization for large language model hard? an empirical study from the lens of perturbation\.Proceedings of the AAAI Conference on Artificial Intelligence38\(16\),pp\. 18082–18089\.External Links:ISSN 2374\-3468,[Document](https://dx.doi.org/10.1609/aaai.v38i16.29765)Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p3.1)\.
- Jianget al\.\(2023\)A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. SayedMistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Kimet al\.\(2024\)S\. Kim, C\. Hooper, A\. Gholami, Z\. Dong, X\. Li, S\. Shen, M\. W\. Mahoney, and K\. KeutzerSqueezeLLM: dense\-and\-sparse quantization\.arXiv\.External Links:2306\.07629,[Document](https://dx.doi.org/10.48550/arXiv.2306.07629)Cited by:[§2](https://arxiv.org/html/2608.24945#S2.p1.1)\.
- Kunstneret al\.\(2019\)F\. Kunstner, P\. Hennig, and L\. BallesLimitations of the empirical fisher approximation for natural gradient descent\.Advances in neural information processing systems \(NeurIPS\)32\.Cited by:[§A\.2\.2](https://arxiv.org/html/2608.24945#A1.SS2.SSS2.p1.2)\.
- Leeet al\.\(2024\)C\. Lee, J\. Jin, T\. Kim, H\. Kim, and E\. ParkOWQ: outlier\-aware weight quantization for efficient fine\-tuning and inference of large language models\.Proceedings of the AAAI Conference on Artificial Intelligence38\(12\),pp\. 13355–13364\.External Links:ISSN 2374\-3468,[Document](https://dx.doi.org/10.1609/aaai.v38i12.29237)Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p4.1),[§2](https://arxiv.org/html/2608.24945#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Leeet al\.\(2025\)S\. Lee, S\. Woo, J\. Jin, C\. Lee, and E\. ParkAmq: enabling automl for mixed\-precision weight\-only quantization of large language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 35520–35538\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p4.1),[§2](https://arxiv.org/html/2608.24945#S2.p2.1),[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Liet al\.\(2025\)Y\. Li, R\. Yin, D\. Lee, S\. Xiao, and P\. PandaGPTAQ: efficient finetuning\-free quantization for asymmetric calibration\.arXiv\.External Links:2504\.02692,[Document](https://dx.doi.org/10.48550/arXiv.2504.02692)Cited by:[§2](https://arxiv.org/html/2608.24945#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Linet al\.\(2024\)J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. HanAWQ: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of Machine Learning and Systems6,pp\. 87–100\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p3.1),[§1](https://arxiv.org/html/2608.24945#S1.p6.1),[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Liuet al\.\(2024\)J\. Liu, J\. Ren, R\. Jin, Z\. Zhang, Y\. Zhou, P\. Valduriez, and D\. DouFisher information\-based efficient curriculum federated learning with large language models\.arXiv\.External Links:2410\.00131,[Document](https://dx.doi.org/10.48550/arXiv.2410.00131)Cited by:[§A\.2\.2](https://arxiv.org/html/2608.24945#A1.SS2.SSS2.p2.1),[§4\.1](https://arxiv.org/html/2608.24945#S4.SS1.p4.2)\.
- Liuet al\.\(2021\)L\. Liu, S\. Zhang, Z\. Kuang, A\. Zhou, J\. Xue, X\. Wang, Y\. Chen, W\. Yang, Q\. Liao, and W\. ZhangGroup fisher pruning for practical network compression\.InProceedings of the 38th International Conference on Machine Learning,pp\. 7021–7032\.External Links:ISSN 2640\-3498Cited by:[§A\.2\.2](https://arxiv.org/html/2608.24945#A1.SS2.SSS2.p2.1)\.
- Lyet al\.\(2017\)A\. Ly, M\. Marsman, J\. Verhagen, R\. P\. P\. P\. Grasman, and E\. WagenmakersA tutorial on fisher information\.Journal of Mathematical Psychology80,pp\. 40–55\.External Links:ISSN 0022\-2496,[Document](https://dx.doi.org/10.1016/j.jmp.2017.05.006),LCCN 4Cited by:[§A\.2\.2](https://arxiv.org/html/2608.24945#A1.SS2.SSS2.p1.1)\.
- Mallaet al\.\(2024\)S\. Malla, J\. H\. Choi, and C\. ChoiCOPAL: continual pruning in large language generative models\.InForty\-first International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2608.24945#S2.p2.1)\.
- Marcuset al\.\(1994\)M\. Marcus, G\. Kim, M\. A\. Marcinkiewicz, R\. MacIntyre, A\. Bies, M\. Ferguson, K\. Katz, and B\. SchasbergerThe penn treebank: annotating predicate argument structure\.InHuman Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 8\-11, 1994,External Links:[Document](https://dx.doi.org/10.3115/1075812.1075835)Cited by:[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Merityet al\.\(2016\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. SocherPointer sentinel mixture models\.arXiv preprint arXiv:1609\.07843\.Cited by:[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- OpenAI \(2025\)OpenAIGPT\-3\.5 Turbo\.Note:[https://platform\.openai\.com/docs/models/gpt\-3\.5\-turbo](https://platform.openai.com/docs/models/gpt-3.5-turbo), accessed 2025\-08Cited by:[§5\.2\.3](https://arxiv.org/html/2608.24945#S5.SS2.SSS3.p1.1)\.
- Paszkeet al\.\(2019\)A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan,et al\.PyTorch: an imperative style, high\-performance deep learning library\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p2.1)\.
- Petroniet al\.\(2019\)F\. Petroni, T\. Rocktäschel, P\. Lewis, A\. Bakhtin, Y\. Wu, A\. H\. Miller, and S\. RiedelLanguage models as knowledge bases?\.arXiv\.External Links:1909\.01066,[Document](https://dx.doi.org/10.48550/arXiv.1909.01066)Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p1.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p1.1)\.
- Raffelet al\.\(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. LiuExploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of Machine Learning Research21\(140\),pp\. 1–67\.External Links:[Link](http://jmlr.org/papers/v21/20-074.html)Cited by:[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Rowlandet al\.\(2024\)M\. Rowland, R\. Munos, M\. G\. Azar, Y\. Tang, G\. Ostrovski, A\. Harutyunyan, K\. Tuyls, M\. G\. Bellemare, and W\. DabneyAn analysis of quantile temporal\-difference learning\.Journal of Machine Learning Research25\(163\),pp\. 1–47\.Cited by:[§4\.2\.1](https://arxiv.org/html/2608.24945#S4.SS2.SSS1.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§4\.2\.1](https://arxiv.org/html/2608.24945#S4.SS2.SSS1.p1.2),[§4\.2](https://arxiv.org/html/2608.24945#S4.SS2.p2.1)\.
- Shaoet al\.\(2024\)W\. Shao, M\. Chen, Z\. Zhang, P\. Xu, L\. Zhao, Z\. Li, K\. Zhang, P\. Gao, Y\. Qiao, and P\. LuoOmniQuant: omnidirectionally calibrated quantization for large language models\.InICLR,Cited by:[§2](https://arxiv.org/html/2608.24945#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Singh and Alistarh \(2020\)S\. P\. Singh and D\. AlistarhWoodFisher: efficient second\-order approximation for neural network compression\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 18098–18109\.Cited by:[§A\.2\.2](https://arxiv.org/html/2608.24945#A1.SS2.SSS2.p2.1)\.
- Team \(2024\)Q\. TeamQwen2 technical report\.arXiv preprint arXiv:2407\.10671\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p2.1)\.
- Touvronet al\.\(2023a\)H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, Lacroix,et al\.LLaMA: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Touvronet al\.\(2023b\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Tuet al\.\(2016\)M\. Tu, V\. Berisha, M\. Woolf, J\. Seo, and Y\. CaoRanking the parameters of deep neural networks using the fisher information\.In2016 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 2647–2651\.External Links:ISSN 2379\-190X,[Document](https://dx.doi.org/10.1109/ICASSP.2016.7472157)Cited by:[§A\.2\.2](https://arxiv.org/html/2608.24945#A1.SS2.SSS2.p2.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p1.1)\.
- Wanget al\.\(2019\)K\. Wang, Z\. Liu, Y\. Lin, J\. Lin, and S\. HanHAQ: hardware\-aware automated quantization with mixed precision\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 8604–8612\.Cited by:[§2](https://arxiv.org/html/2608.24945#S2.p2.1)\.
- Wanget al\.\(2024a\)W\. Wang, W\. Chen, Y\. Luo, Y\. Long, Z\. Lin, L\. Zhang, B\. Lin, D\. Cai, and X\. HeModel compression and efficient inference for large language models: a survey\.arXiv preprint arXiv:2402\.09748\.Cited by:[§A\.2\.1](https://arxiv.org/html/2608.24945#A1.SS2.SSS1.p1.4)\.
- Wanget al\.\(2024b\)Z\. Wang, X\. Liu, H\. Zhang, and W\. ShaoRL\-ptq: reinforcement learning for post\-training quantization\.arXiv preprint arXiv:2405\.17508\.Cited by:[§2](https://arxiv.org/html/2608.24945#S2.p2.1)\.
- Weiet al\.\(2022\)X\. Wei, Y\. Zhang, X\. Zhang, R\. Gong, S\. Zhang, Q\. Zhang, F\. Yu, and X\. LiuOutlier suppression: pushing the limit of low\-bit transformer language models\.Advances in Neural Information Processing Systems35,pp\. 17402–17414\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p3.1)\.
- Wolfet al\.\(2019\)T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz,et al\.Huggingface’s transformers: state\-of\-the\-art natural language processing\.arXiv preprint arXiv:1910\.03771\.Cited by:[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p2.1)\.
- Xiaoet al\.\(2023\)G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. HanSmoothQuant: accurate and efficient post\-training quantization for large language models\.InProceedings of the 40th International Conference on Machine Learning,pp\. 38087–38099\.External Links:ISSN 2640\-3498Cited by:[§2](https://arxiv.org/html/2608.24945#S2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p2.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li,et al\.Qwen2 technical report\.arXiv preprint arXiv:2407\.10671\.Cited by:[§5\.1](https://arxiv.org/html/2608.24945#S5.SS1.p1.1)\.
- Zenget al\.\(2025\)C\. Zeng, S\. Liu, Y\. Xie, H\. Liu, X\. Wang, M\. Wei, S\. Yang, F\. Chen, and X\. MeiABQ\-llm: arbitrary\-bit quantized inference acceleration for large language models\.Proceedings of the AAAI Conference on Artificial Intelligence39\(21\),pp\. 22299–22307\.External Links:ISSN 2374\-3468,[Document](https://dx.doi.org/10.1609/aaai.v39i21.34385)Cited by:[§2](https://arxiv.org/html/2608.24945#S2.p1.1)\.
- Zhenget al\.\(2024\)Z\. Zheng, X\. Song, and C\. LiuMixLLM: llm quantization with global mixed\-precision between output\-features and highly\-efficient system design\.arXiv\.External Links:2412\.14590,[Document](https://dx.doi.org/10.48550/arXiv.2412.14590)Cited by:[§A\.2\.2](https://arxiv.org/html/2608.24945#A1.SS2.SSS2.p1.1)\.
- Zhouet al\.\(2018\)Y\. Zhou, S\. Moosavi\-Dezfooli, N\. Cheung, and P\. FrossardAdaptive quantization for deep neural network\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.32\.Cited by:[§A\.7\.10](https://arxiv.org/html/2608.24945#A1.SS7.SSS10.p2.1),[§3\.2](https://arxiv.org/html/2608.24945#S3.SS2.p1.4)\.
- Zhuet al\.\(2024\)X\. Zhu, J\. Li, Y\. Liu, C\. Ma, and W\. WangA survey on model compression for large language models\.Transactions of the Association for Computational Linguistics12,pp\. 1556–1577\.Cited by:[§1](https://arxiv.org/html/2608.24945#S1.p3.1)\.

Table A\.1:Summary of main notations## Appendix AAppendix

### A\.1Explanation of Notations

The meanings of the notations in this paper are summarized in Table[A\.1](https://arxiv.org/html/2608.24945#A0.T1)\.

### A\.2Preliminary

In this section, we present the quantization preliminary and introduce Fisher information calculation\.

#### A\.2\.1Quantization Preliminary

The quantization process can be classified into uniform and non\-uniform[44](https://arxiv.org/html/2608.24945#bib.bib23)\. The uniform quantization uses uniform and finite intervals \(e\.g\.,2b2^\{b\}intervals forbb\-bit integer\) to represent the original values\. In contrast, non\-uniform quantization utilizes non\-uniformly spaced intervals, and the length of intervals can vary\. Given a weight tensor𝐖\\mathbf\{W\}in a LLM, the quantization and de\-quantization process can be defined as Formula[A\.1](https://arxiv.org/html/2608.24945#A1.E1)\.

𝐖Q=Q\(𝐖\),𝐖~=Q−1\(𝐖Q\),\\displaystyle\\mathbf\{W\}^\{Q\}=Q\(\\mathbf\{W\}\),\\quad\\mathbf\{\\widetilde\{W\}\}=Q^\{\-1\}\(\\mathbf\{W\}^\{Q\}\),\(A\.1\)where𝐖Q\\mathbf\{W\}^\{Q\}is the quantized tensor, and𝐖~\\mathbf\{\\widetilde\{W\}\}is the recovered tensor\. The quantization functionQ⁡\(⋅\)Q\(\\cdot\)of a uniform quantization approach is defined as a rounding\-to\-nearest operation over the scaled input calculated in Formula[A\.2](https://arxiv.org/html/2608.24945#A1.E2)\.

Quni​\(𝐖\)=clip​\(⌊𝐖α⌉\+z,0,2b−1\),\\displaystyle Q\_\{\\text\{uni\}\}\(\\mathbf\{W\}\)=\\text\{clip\}\\left\(\\left\\lfloor\\frac\{\\mathbf\{W\}\}\{\\alpha\}\\right\\rceil\+z;0,2^\{b\}\-1\\right\),\(A\.2\)whereb∈ℕb\\in\\mathbb\{N\}is the bit\-width,α∈ℝ\\alpha\\in\\mathbb\{R\}is the scale factor,z∈ℕz\\in\\mathbb\{N\}is zero\-point or offset value,⌊⋅⌉\\left\\lfloor\\cdot\\right\\rceildenotes the round\-to\-nearest\-integer operator, andc​l​i​p​\(x,m​i​nv​a​l​u​e,m​a​xv​a​l​u​e\)clip\(x,min\_\{value\},max\_\{value\}\)represents a clip function of the inputxxwith the minimum value and the maximum value\. The corresponding de\-quantization function is defined in Formula[A\.3](https://arxiv.org/html/2608.24945#A1.E3)\.

Quni−1​\(𝐖\)=\(𝐖Q−z\)⋅α\.\\displaystyle Q\_\{\\text\{uni\}\}^\{\-1\}\(\\mathbf\{W\}\)=\(\\mathbf\{W\}^\{Q\}\-z\)\\cdot\\alpha\.\(A\.3\)
Uniform quantization can be either symmetric or asymmetric according to the sign of the mapping space\. In this paper, we use the symmetric uniform quantization approach\. The symmetric quantization restricts the zero\-point to 0 as defined in Formula[A\.4](https://arxiv.org/html/2608.24945#A1.E4)\.

Quni\_sym​\(𝐖\)=clip​\(⌊𝐖α⌉,−2b−1,2b−1−1\)\\displaystyle Q\_\{\\text\{uni\\\_sym\}\}\(\\mathbf\{W\}\)=\\text\{clip\}\\left\(\\left\\lfloor\\frac\{\\mathbf\{W\}\}\{\\alpha\}\\right\\rceil;\-2^\{b\-1\},2^\{b\-1\}\-1\\right\)\(A\.4\)
Quantization approaches optimize the global loss as defined in Formula[A\.5](https://arxiv.org/html/2608.24945#A1.E5)\.

argmin𝐖QE\\displaystyle\\operatorname\*\{argmin\}\_\{\\mathbf\{W\}^\{Q\}\}E=argmin𝐖Q‖𝐖​X−𝐖Q​X‖22,\\displaystyle=\\operatorname\*\{argmin\}\_\{\\mathbf\{W\}^\{Q\}\}\\left\\\|\\mathbf\{W\}X\-\\mathbf\{W\}^\{Q\}X\\right\\\|^\{2\}\_\{2\},\(A\.5\)whereXXis the input of the corresponding layer of the LLM\.

#### A\.2\.2Fisher Information

Fisher information quantifies the amount of information that observable data carries about the unknown parameters of a probabilistic model[25](https://arxiv.org/html/2608.24945#bib.bib24);[52](https://arxiv.org/html/2608.24945#bib.bib49)\. Fisher information can be used to evaluate parameter importance in neural networks by quantifying how sensitive the model output is to the noises of each parameterθ\\theta\. We can denote the Fisher Information Matrix \(FIM\) by the expectation of the outer product of score vectors[2](https://arxiv.org/html/2608.24945#bib.bib26)as defined in Formula[A\.6](https://arxiv.org/html/2608.24945#A1.E6)\.

F⁡\(θl\):=𝔼pθ​\(y\|x\)​\[∇θl​log​pθl​\(y\|x\)​∇θl​log​pθl​\(y\|x\)⊤\],F\(\\theta\_\{l\}\):=\\textstyle\\mathbb\{E\}\_\{p\_\{\\theta\}\(y\|x\)\}\\left\[\\nabla\_\{\\theta\_\{l\}\}\\log p\_\{\\theta\_\{l\}\}\(y\|x\)\\nabla\_\{\\theta\_\{l\}\}\\log p\_\{\\theta\_\{l\}\}\(y\|x\)^\{\\top\}\\right\],\(A\.6\)wherepθl​\(y\|x\)p\_\{\\theta\_\{l\}\}\(y\|x\)represents the probability density function of the inference with LLM and parametersθl\\theta\_\{l\}at Layerll,∇θl​log​pθl​\(y\|x\)\\nabla\_\{\\theta\_\{l\}\}\\log p\_\{\\theta\_\{l\}\}\(y\|x\)denotes the first\-order derivative of the LLM, which is calculated via the gradient\. In practice, we use the empirical FIM to approximate the expected one[18](https://arxiv.org/html/2608.24945#bib.bib34)as shown in Formula[A\.7](https://arxiv.org/html/2608.24945#A1.E7)\.

F^​\(θl\)=1N​∑n=1N∇θl​log​pθl​\(yn\|xn\)​∇θl​log​pθl​\(yn\|xn\)⊤,\\hat\{F\}\(\\theta\_\{l\}\)=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\nabla\_\{\\theta\_\{l\}\}\\log p\_\{\\theta\_\{l\}\}\(y\_\{n\}\|x\_\{n\}\)\\nabla\_\{\\theta\_\{l\}\}\\log p\_\{\\theta\_\{l\}\}\(y\_\{n\}\|x\_\{n\}\)^\{\\top\},\(A\.7\)whereNNrepresents the number of samples in the validation dataset𝒟\\mathcal\{D\}\.

Figure A\.1:Visualization of FIM for LLaMA\-7B and LLaMA\-13B\.FIM captures the essential influence of parameters on the likelihood function, where larger FIM values indicate more influential parameters should be preserved for inference\. FIM can be used to evaluate the importance of layer\-wise parameters[41](https://arxiv.org/html/2608.24945#bib.bib28), so as to preserve accuracy while compressing LLMs[37](https://arxiv.org/html/2608.24945#bib.bib46);[24](https://arxiv.org/html/2608.24945#bib.bib27)\. While calculating FIM is computationally expensive with large gradient matrices, we use the diagonal vector of FIM to represent the FIM[23](https://arxiv.org/html/2608.24945#bib.bib21)as defined in Formula[A\.8](https://arxiv.org/html/2608.24945#A1.E8)\.

ℱ⁡\(θl\)=diag​\(I\|∇θl​log​pθl​\(y\|x\)\|⊙F^​\(θl\)\),\\mathcal\{F\}\(\\theta\_\{l\}\)=\\text\{diag\}\(I\_\{\|\\nabla\_\{\\theta\_\{l\}\}\\log p\_\{\\theta\_\{l\}\}\(y\|x\)\|\}\\odot\\hat\{F\}\(\\theta\_\{l\}\)\),\(A\.8\)wherediag​\(⋅\)\\text\{diag\}\(\\cdot\)represents the diagonal vector of a matrix,I\|∇θl​log​pθl​\(y\|x\)\|I\_\{\|\\nabla\_\{\\theta\_\{l\}\}\\log p\_\{\\theta\_\{l\}\}\(y\|x\)\|\}is the identity matrix with the same size of the gradient matrix, and⊙\\odotis element\-wise multiplication\.

### A\.3Pearson Correlation Analysis

In order to verify the correlation between FIM and accuracy degradation, we calculate the Pearson correlation coefficients[4](https://arxiv.org/html/2608.24945#bib.bib41)between the layer\-wise quantization sensitivity and the increase in perplexity \(PPL\) with perturbations \(δ3\\delta\_\{3\}quantization\-simulating perturbations atb=4b\{=\}4\) in each layer\. As shown in Table[A\.2](https://arxiv.org/html/2608.24945#A1.T2), the perturbations and the increase in PPL have a significant positive correlation withR\>0\.5R\>0\.5andP<0\.0001P<0\.0001, indicating that FIM is a reliable proxy for performance degradation\.

Table A\.2:Pearson correlation coefficients between FIM and the increase in PPL among different types of LLaMA\-7B layers\.RRrepresents the related correlation coefficient andPPrefers to the Pearson value\.Figure A\.2:Visualization of FIM for Qwen2\.5\-7B and Qwen2\.5\-14B\.
### A\.4Perturbation Strategy Ablation

In Section[4\.1](https://arxiv.org/html/2608.24945#S4.SS1)we adopt the quantization perturbationδ​θl=Qb​\(θl\)−θl\\delta\\theta\_\{l\}=Q\_\{b\}\(\\theta\_\{l\}\)\-\\theta\_\{l\}as the canonical choice, sinceθl\+δ​θl=Qb​\(θl\)\\theta\_\{l\}\+\\delta\\theta\_\{l\}=Q\_\{b\}\(\\theta\_\{l\}\)and it therefore matches the exact perturbation thatbb\-bit quantization injects\. For completeness, here we provide the full mathematical definitions of two alternative perturbation forms used in the ablation in Section 5\.4\.1:

- •Magnitude\-proportional uniform noise:δ1​θl=ϵ⋅μl\\delta\_\{1\}\\theta\_\{l\}=\\epsilon\\cdot\\mu\_\{l\}, whereμl=𝔼⁡\[θl\]\\mu\_\{l\}=\\mathbb\{E\}\[\\theta\_\{l\}\]andϵ∼𝒰⁡\(−1,1\)\\epsilon\\sim\\mathcal\{U\}\(\-1,1\)\. This form scales uniform noise by each layer’s mean magnitude, preserving relative scale differences across layers but*not*the structure of the quantization perturbation\.
- •Bernoulli\-masked weight\-proportional noise:δ2​θl=β⋅θl⊙𝐦\\delta\_\{2\}\\theta\_\{l\}=\\beta\\cdot\\theta\_\{l\}\\odot\\mathbf\{m\}, withβ∈\[0,1\]\\beta\\in\[0,1\]and𝐦∼Bernoulli​\(0\.5\)\\mathbf\{m\}\\sim\\text\{Bernoulli\}\(0\.5\)\. This form applies sparse, weight\-proportional noise via a random mask\. The fixed ratioβ\\betaensures consistent perturbation intensity but, likeδ1\\delta\_\{1\}, is agnostic to the target bit\-widthbb\.

The corresponding empirical comparison is reported in the main text \(Table[3](https://arxiv.org/html/2608.24945#S5.T3)\); both alternatives are dominated by the quantization perturbationδ3\\delta\_\{3\}atb=4b\{=\}4\.

### A\.5Visualization of Sensitivity

As shown in Figures[A\.1](https://arxiv.org/html/2608.24945#A1.F1)and[A\.2](https://arxiv.org/html/2608.24945#A1.F2), the sensitivity \(FIM\) varies substantially across layers\. Input and output\-adjacent layers tend to be more sensitive because they shape low\-level feature encoding and final predictions, while intermediate layers show more heterogeneous behavior\. FAMPWQ exploits this diversity by preserving high\-sensitivity layers and compressing low\-sensitivity layers more aggressively\.

Algorithm 1Reinforcement Learning \(RL\)\-based Network TrainingInput: EE: The number of training epochs LL: The number of layers in a LLM Output: θ;ϕ\\theta;\\phi: Parameters of the pre\-trained actor and critic network

1:

θ0\\theta\_\{0\};

ϕ0←\\phi\_\{0\}\\leftarrowRandomly initialize the actor and critic network

2:

𝒬0←\[max⁡\(ℬ\)\]L\\mathcal\{Q\}\_\{0\}\\leftarrow\[\\max\(\\mathcal\{B\}\)\]\_\{L\}
3:forEpoch

e=1e=1to

EEdo

4:forStep

t=1t=1to

LLdo

5:

qtt←q^\{t\}\_\{t\}\\leftarrowGenerate the bit\-width for Layer

tt
6:

𝒬t←Update​𝒬t−1\\mathcal\{Q\}\_\{t\}\\leftarrow\\text\{Update \}\\mathcal\{Q\}\_\{t\-1\}by replacing

qtt−1q\_\{t\}^\{t\-1\}by

qttq^\{t\}\_\{t\}
7:

ℒ⁡\(𝒬t−1\)←\\mathcal\{L\}\(\\mathcal\{Q\}\_\{t\-1\}\)\\leftarrowCalculate the loss according to Formula[11](https://arxiv.org/html/2608.24945#S4.E11)

8:

At←A\_\{t\}\\leftarrowCalculate according to Formula[14](https://arxiv.org/html/2608.24945#S4.E14)

9:

𝔯t​\(θt−1\)←\\mathfrak\{r\}\_\{t\}\(\\theta\_\{t\-1\}\)\\leftarrowCalculate

𝔯t​\(θt−1\)\\mathfrak\{r\}\_\{t\}\(\\theta\_\{t\-1\}\)according to Formula[17](https://arxiv.org/html/2608.24945#S4.E17)

10:

𝔠t​\(θt−1\)←\\mathfrak\{c\}\_\{t\}\(\\theta\_\{t\-1\}\)\\leftarrowCalculate

𝔠t​\(θt−1\)\\mathfrak\{c\}\_\{t\}\(\\theta\_\{t\-1\}\)with

𝔯t​\(θt−1\)\\mathfrak\{r\}\_\{t\}\(\\theta\_\{t\-1\}\)and

AtA\_\{t\}according to Formula[16](https://arxiv.org/html/2608.24945#S4.E16)

11:

θt←\\theta\_\{t\}\\leftarrowUpdate

θt−1\\theta\_\{t\-1\}with

AtA\_\{t\},

𝔯t​\(θt−1\)\\mathfrak\{r\}\_\{t\}\(\\theta\_\{t\-1\}\)and

𝔠t​\(θt−1\)\\mathfrak\{c\}\_\{t\}\(\\theta\_\{t\-1\}\)according to Formula[4\.2\.1](https://arxiv.org/html/2608.24945#S4.Ex1)

12:

ϕt←\\phi\_\{t\}\\leftarrowUpdate

ϕt−1\\phi\_\{t\-1\}with

AtA\_\{t\}according to Formula[18](https://arxiv.org/html/2608.24945#S4.E18)

13:endfor

14:endfor

Table A\.3:Hyperparameter values\.ϵ\\epsilonrefers to the hyperparameter in Formula[16](https://arxiv.org/html/2608.24945#S4.E16)\. For different models, recommendationα\\alphais given\.Table A\.4:PPL of Qwen2\.5\-7B and LLaMA\-7B with quantization of 4\-bit and 5\-bit and diverse benchmarks \(Wiki2, PTB, C4\)\. “4\-bit” represents that all the layers are quantized to 4\-bit\. Other rows report the PPL when the corresponding layers are quantized to 5\-bit while other layers remain at 4\-bit\.Boldindicates the lowest PPL\.underlinedindicates the second\-highest\.Table A\.5:PPL of FAMPWQ with varying average quantization bit\-widths\.
### A\.6Reinforcement Learning \(RL\)\-based Network Training

Table A\.6:Quantization time with NVIDIA 4090 GPU\.Table A\.7:FAMPWQ preprocessing time on NVIDIA 4090 GPUs\. Sensitivity is computed using the listed number of GPUs, while RL\-based bit\-width search is performed on a single GPU\.As shown in Algorithm[1](https://arxiv.org/html/2608.24945#alg1), the actor and critic networks are trained in multiple epochs\. First, the actor and critic networks are randomly initialized \(Line 1\), and the initial bit\-width allocation strategy𝒬0\\mathcal\{Q\}\_\{0\}is initialized to the maximum value for each layer \(Line 2\)\. Within each training epoch, the bit\-widthqttq\_\{t\}^\{t\}is generated for each layer \(Lines 4\-5\)\. The bit\-width allocation strategy𝒬t\\mathcal\{Q\}\_\{t\}is updated withqttq\_\{t\}^\{t\}\(Line 6\)\. Then, the loss function corresponding to𝒬t\\mathcal\{Q\}\_\{t\}is computed \(Line 7\)\. Afterwards, the Temporal\-Difference \(TD\) advantage is calculated according to Formula[14](https://arxiv.org/html/2608.24945#S4.E14)\(Line 8\)\. In addition, the clip reward and the policy\-dependent reward are calculated based on Formulas[17](https://arxiv.org/html/2608.24945#S4.E17)and[16](https://arxiv.org/html/2608.24945#S4.E16)\(Lines 9\-10\)\. Finally, the critic networkϕt\\phi\_\{t\}and the actor networkθt\\theta\_\{t\}are updated based on Formulas[4\.2\.1](https://arxiv.org/html/2608.24945#S4.Ex1)and[18](https://arxiv.org/html/2608.24945#S4.E18)\(Lines 11\-12\)\.

### A\.7Experiment Details

In this section, we first present the hyperparameter values in experimental setup\. Then, we present additional experiments, including the PPL with 5\-bit quantization, varying average quantization bit\-widths \(from 3\.1 to 4\.5\), the comparison of time consumption, and the quantization with 5\-bits on average for LLaMA2\-7B\-chat, LLaMa2\-13B\-chat, and Mistral\-7B\-v0\.1\.

#### A\.7\.1Calibration\-Size Robustness

We examine whether the Fisher estimate and the resulting bit\-width allocation are sensitive to the amount of calibration data\. On LLaMA\-7B, we construct nested C4 subsets containing 32, 64, 128, and 256 sequences for each of three independent seeds, and use the 256\-sequence subset from the same seed as the reference\. The analysis covers all 224 quantizable linear modules\. To isolate calibration noise, every setting uses the same bit\-conditioned proxy, candidate set\{2,3,4\}\\\{2,3,4\\\}, parameter\-weighted 3\-bit budget, and deterministic same\-budget allocator\. Table[A\.8](https://arxiv.org/html/2608.24945#A1.T8)reports the rank correlation, overlap among the top 10% most sensitive modules, and the fraction of module assignments that differ from the 256\-sequence reference\.

Table A\.8:Robustness to calibration\-set size on LLaMA\-7B over three seeds\. Each row is compared with the nested 256\-sequence subset from the same seed\.Even with only 32 calibration sequences, the Fisher ranking retains a Spearman correlation of 0\.996 and a 97\.1% top\-10% overlap with the 256\-sequence reference, while only 2\.68% of module assignments change\. At the 128\-sequence setting used in the main experiments, the allocation difference decreases to 1\.34%\. These results show that the sensitivity ranking and budget\-constrained allocation are stable with limited calibration data\. This study uses 512\-token calibration sequences and a deterministic proxy allocator, and does not rerun final\-backend PPL for every calibration size; therefore, it establishes ranking and allocation stability rather than complete invariance of downstream quality\.

#### A\.7\.2PPL with Diverse Quantization Bit\-width

As shown in Table[A\.4](https://arxiv.org/html/2608.24945#A1.T4), the PPL corresponding to the quantization of 4\-bit with one layer quantized to 5 bits can vary across different layers\. MLP layers, especially the down\-projection \(mlp\.down\), correspond to significant PPL drop \(up to 0\.115 PPL drop\)\. Among attention layers, the value projection \(attn\.v\) corresponds to the highest sensitivity \(up to 0\.047\)\. FAMPWQ aligns with this diversity and thus yields strong performance\.

#### A\.7\.3Diverse Average Quantization Bit\-width

FAMPWQ can achieve varying average quantization bit\-widths through adaptive layer\-wise bit\-width allocation strategies\. As shown in Table[A\.5](https://arxiv.org/html/2608.24945#A1.T5), FAMPWQ can achieve average bit\-widths from 3\.1 to 4\.5, with correspondingly decreasing PPL\.

#### A\.7\.4Comparison of Time Consumption

As shown in Table[A\.6](https://arxiv.org/html/2608.24945#A1.T6), the quantization time of FAMPWQ is comparable to several baselines and can be shorter than GPTQ \(up to 37%\), GPTQv2 \(up to 61%\), OWQ \(up to 35%\), and OmniQuant \(up to 66%\)\. Although RTN and AWQ can be faster than FAMPWQ by up to 97% and 46%, respectively, they may incur substantially larger performance degradation under aggressive compression\.

Table A\.9:PPL↓\\downarrowcomparison on Qwen2\.5, Qwen2\.5\-14B, and Mistral with 4\-bit average quantization\. “Avg PPL” denotes the average perplexity over Wiki2, PTB, and C4\.Table A\.10:The accuracy↑\\uparrowof quantized Qwen2\.5\-7B and Qwen2\.5\-13B on zero\-shot reasoning tasks\.Boldindicates the highest accuracy andunderlinedindicates the second\-highest\. “Avg bit” represents the average width\-bit\. “Avg acc” represents the average accuracy of the 5 tasks\.As shown in Table[A\.7](https://arxiv.org/html/2608.24945#A1.T7), the preprocessing stage of FAMPWQ, which consists of per\-layer Fisher sensitivity computation and an RL\-based bit\-width optimization search, is conducted on NVIDIA RTX 4090 GPUs\. The sensitivity computation accounts for the majority of the preprocessing cost and scales with model size, whereas the RL\-based search is lightweight, requiring only tens of seconds on a single GPU\. Overall, the total preprocessing time remains below 70 minutes even for 14B\-scale models, demonstrating the practical efficiency and scalability of FAMPWQ\.

#### A\.7\.5Results of LLaMA2, Qwen2\.5 and Mistral\-7B\-v0\.1 models

Table A\.11:Comparison of perplexity results↓\\downarrowof different 4\-bit quantization approaches with the LLaMA2\-7B\-chat and LLaMA2\-13B\-chat model\. “Avg PPL” represents the average PPL of the 3 benchmarks\.As shown in Table[A\.11](https://arxiv.org/html/2608.24945#A1.T11), FAMPWQ achieves the lowest average PPL under 4\-bit quantization for both LLaMA2\-7B\-chat and LLaMA2\-13B\-chat \(up to 1\.09 lower than RTN, 0\.28 lower than GPTQ, 0\.05 lower than GPTQv2, 0\.13 lower than OmniQuant, 0\.16 lower than OWQ, and 0\.05 lower than AWQ\)\. The average reduction over all quantized baselines is larger on LLaMA2\-7B\-chat \(0\.28 PPL\) than on LLaMA2\-13B\-chat \(0\.10 PPL\), while both model sizes show the best average PPL with FAMPWQ\. Furthermore, FAMPWQ outperforms baseline approaches for the majority of the combinations of the benchmarks and models \(up to 0\.84 for Wiki2 and 1\.29 for PTB in LLaMA2\-7B\-chat; up to 0\.21 for Wiki2, 0\.20 for PTB, and 0\.39 for C4 in LLaMA2\-13B\-chat\)\. While the PPL of FAMPWQ is slightly \(0\.07\) higher than that of GPTQv2 with the combination of C4 and LLaMA2\-7B\-chat, FAMPWQ outperforms other baselines in this setting \(1\.12 lower than RTN, 0\.30 lower than GPTQ, 0\.09 lower than OmniQuant, 0\.21 lower than OWQ, and 0\.03 lower than AWQ\)\.

As shown in Table[A\.9](https://arxiv.org/html/2608.24945#A1.T9), FAMPWQ consistently achieves the best performance in terms of PPL across all 3 LLMs and 3 benchmarks under 4\-bit quantization\. With Qwen2\.5\-7B, FAMPWQ attains an average PPL of 10\.81, which is up to 2\.86 lower than that of RTN, and also lower than GPTQ, GPTQv2, OmniQuant, OWQ, and AWQ by 0\.26, 0\.07, 0\.05, 0\.19, and 0\.04, respectively\. On Qwen2\.5\-14B, FAMPWQ achieves an average PPL of 9\.15, which is 1\.41 lower than RTN, 0\.19 lower than GPTQ, 0\.06 lower than GPTQv2, and 0\.02 lower than both OmniQuant and AWQ\. For Mistral\-7B\-v0\.1, FAMPWQ yields an average PPL of 8\.04, representing reductions of 0\.94, 0\.12, 0\.05, 0\.07, and 0\.01 compared to RTN, GPTQ, GPTQv2, OWQ, and AWQ, respectively\.

As shown in Table[A\.10](https://arxiv.org/html/2608.24945#A1.T10), FAMPWQ significantly outperforms baseline approaches \(from 0\.06% to 4\.55%\) in terms of average accuracy with Qwen2\.5\-7B and Qwen2\.5\-14B under 4\-bit quantization\. As shown in Table[A\.12](https://arxiv.org/html/2608.24945#A1.T12), FAMPWQ achieves the highest average zero\-shot accuracy under 4\-bit quantization with both LLaMA2\-7B\-chat \(up to 1\.63% higher than RTN, 0\.39% higher than GPTQ, and 0\.2% higher than AWQ\) and LLaMA2\-13B\-chat \(up to 2\.87% higher than RTN\), with FAMPWQ surpassing all or most baseline approaches for the majority of tasks\.

Table A\.12:The accuracy↑\\uparrowof quantized LLaMA2\-7B\-chat and LLaMA2\-13B\-chat on 5 zero\-shot reasoning tasks under 4\-bit quantization\.Boldindicates the highest accuracy andunderlinedindicates the second\-highest\. “Avg bit” represents the average width\-bit\. “Avg acc” represents the average accuracy of the 5 tasks\.
#### A\.7\.6Storage reduction

Comparable to single\-precision quantization approaches, e\.g\., RTN, AWQ, GPTQ, GPTQv2, FAMPWQ incurs no extra storage overhead or no additional metadata, while OWQ corresponds to larger storage requirement with extra metadata\. As shown in the Table[A\.13](https://arxiv.org/html/2608.24945#A1.T13), FAMPWQ reduces storage overhead by 1%–3% of the original model size compared to OWQ\. As the quantization bit width of the model weights decreases, the storage space required by each model is almost linearly reduced\.

Table A\.13:Storage requirements of various LLMs under different weight quantization precisions\. Percentages in parentheses indicate the proportion relative to FP16 size\.
#### A\.7\.7Sensitivity Metric Comparison

The sensitivity metric comparison is visualized in the main text \(Figure[9](https://arxiv.org/html/2608.24945#S5.F9)\)\. Table[A\.14](https://arxiv.org/html/2608.24945#A1.T14)provides the same data in an extended format for reference\.

Table A\.14:Sensitivity analysis of different metrics on LLaMA\-7B and Qwen2\.5\-7B \(extended from the main\-text sensitivity comparison\)\.
#### A\.7\.8Packed\-Deployment Memory Accounting

Static model size alone does not capture the complete deployment footprint\. We therefore perform analytical tensor accounting for packed Llama\-2\-7B inference with a 512\-token prompt, 256 generated tokens, and an FP16 KV cache\. The packed static weight footprints are 12\.551 GiB for FP16, 2\.862 GiB at a 3\-bit average, and 3\.622 GiB at a 4\-bit average\. Because FAMPWQ changes only the weight representation, its KV\-cache footprint is identical to AWQ: 0\.375 GiB at batch size 1 and 1\.500 GiB at batch size 4\. The largest per\-layer FP16 materialization is at most 86 MiB \(0\.084 GiB\), and this workspace is reused across sequential layer execution rather than allocated once per layer\.

Table A\.15:Analytical packed\-deployment memory accounting for Llama\-2\-7B\. Values include packed weights, the final FP16 KV cache, and the known reusable workspace; they are not measured runtime peaks\.At the 3\-bit average budget, packing reduces the weight footprint by 9\.689 GiB, whereas the largest known temporary mixed\-bit workspace is only 0\.084 GiB\. The resulting tensor\-accounted saving is therefore at least 9\.605 GiB, and the workspace is only 0\.87% of the static weight saving\. Under this accounting, temporary mixed\-bit storage cannot offset the weight\-memory reduction\. This result is an analytical estimate rather than a measured runtime peak: activation and framework residuals, CUDA\-reserved memory, allocator behavior, and fragmentation still require measurement with an actual packed\-kernel implementation\.

#### A\.7\.9Bit\-width Allocation Visualization

![Refer to caption](https://arxiv.org/html/2608.24945v1/sensitivity_and_allocation.png)Figure A\.3:Layer\-wise sensitivity scores \(top\) and the resulting bit\-width allocations \(bottom\) under the three metrics on LLaMA\-7B \(bt=4b\_\{t\}=4\)\.Fig\.[A\.3](https://arxiv.org/html/2608.24945#A1.F3)visualizes the per\-layer sensitivity scores and the corresponding quantization\-optimized bit\-width allocations\. FIM captures gradient\-level information that reveals additional critical layers overlooked by weight\-only metrics\. Thus, FAMPWQ can generate a structurally distinct allocation that assigns higher precision to the most loss\-sensitive modules\. In this way, FAMPWQ ultimately delivers excellent performance\.

#### A\.7\.10Validation of Modeling Assumptions

Our accuracy\-degradation proxy \(Formulas[1](https://arxiv.org/html/2608.24945#S3.E1)–[3](https://arxiv.org/html/2608.24945#S3.E3)\) rests on two assumptions: \(1\) exponential decay of degradation with bit\-width, and \(2\) approximate layer independence\.

Exponential Decay\.Following Rate\-Distortion Theory[53](https://arxiv.org/html/2608.24945#bib.bib22), quantization error decreases exponentially with allocated bits\. We verify this empirically by measuring per\-layer PPL as a function of bit\-width and fitting exponential curves \(R2\>0\.95R^\{2\}\>0\.95across all layer types for LLaMA\-7B; see Figure[10](https://arxiv.org/html/2608.24945#S5.F10)\)\.

Layer Independence\.To validate the additivity assumption, we perform a controlled test on LLaMA\-7B: we quantize layer pairs\(L10,L11\)\(L\_\{10\},L\_\{11\}\)and\(L5,L25\)\(L\_\{5\},L\_\{25\}\)individually, sum their PPL increases, and compare against the joint quantization\. The relative error between the additive prediction and actual degradation is<<0\.1%, confirming that cross\-layer interaction effects are negligible for the purpose of bit\-width allocation\.

α\\alphaSensitivity\.The decay rateα\\alphain Formula[1](https://arxiv.org/html/2608.24945#S3.E1)is robust across a wide range: as shown in Figure[10](https://arxiv.org/html/2608.24945#S5.F10), PPL varies by less than 0\.3 withinα∈\[15,25\]\\alpha\\in\[15,25\]for both LLaMA\-7B and Qwen2\.5\-7B\.

##### Controlled Error Additivity Test\.

To further validate the layer\-independence assumption used in the proxy model, we quantize two layers individually and jointly while keeping all other layers at FP16\. If cross\-layer interactions are negligible, the joint PPL increase should match the sum of the two individual increases\.

Table A\.16:Controlled error additivity test on LLaMA\-7B under 3\-bit quantization\. Relative error compares the additive predictionΔ⁡\(Li\)\+Δ⁡\(Lj\)\\Delta\(L\_\{i\}\)\+\\Delta\(L\_\{j\}\)against the jointly measured degradationΔ⁡\(Li,Lj\)\\Delta\(L\_\{i\},L\_\{j\}\)\.As shown in Table[A\.16](https://arxiv.org/html/2608.24945#A1.T16), the relative error is below 0\.1% for both adjacent and distant layer pairs, supporting the approximation that cross\-layer interaction effects are small for the purpose of bit\-width allocation\.

Figure A\.4:Accuracy vs\. model size trade\-off for Qwen2\.5\-7B\. At identical storage cost, FAMPWQ achieves \+0\.65 percentage points over AWQ at 4\-bit and \+1\.55 points at 3\-bit, demonstrating better accuracy\-per\-byte efficiency\.

#### A\.7\.11Vicuna\-Bench Generation Quality at 3\-bit

To bridge perplexity and real\-world generation quality, we also evaluate 3\-bit quantized models on Vicuna\-Bench with GPT\-4 as the judge\. Table[A\.17](https://arxiv.org/html/2608.24945#A1.T17)shows that FAMPWQ preserves generation quality better than the strongest uniform baseline under this aggressive compression setting\.

Table A\.17:Vicuna\-Bench generation quality under 3\-bit quantization\. Win rate is measured against FP16 responses\.
#### A\.7\.12Accuracy\-per\-Byte Analysis

To evaluate the deployment efficiency of FAMPWQ, we analyze the accuracy\-per\-byte trade\-off\. As shown in Figure[A\.4](https://arxiv.org/html/2608.24945#A1.F4), at identical storage costs \(same average bit\-width\), FAMPWQ consistently achieves higher accuracy than the best uniform baseline \(AWQ\)\. The advantage grows as compression becomes more aggressive: \+0\.65 percentage points at 4\-bit \(5,197 MB\) and \+1\.55 points at 3\-bit \(4,414 MB\)\. This confirms that FAMPWQ extracts more quality from each byte of storage, making it particularly valuable for memory\-constrained deployment scenarios where the goal is fitting the best possible model into a fixed VRAM budget\.

#### A\.7\.13Qualitative Generation Example at 3\-bit

Table A\.18:Representative Llama\-2\-Chat responses at a 3\-bit average budget; light blue marks the factual error in 3\-bit AWQ\.Aggregate win rates do not reveal which information is lost by a quantized model\. We therefore compare one representative response from FP16, 3\-bit AWQ, and 3\-bit FAMPWQ on Llama\-2\-Chat withmax\_length=200\. The prompt requires a short summary of the cause, customer impact, and revised plan described in a multi\-speaker dialogue:

> Briefly summarize the cause of the delay, the customer impact, and the revised plan described below\. Write one short paragraph\. Maya:Can we still launch on Friday? Leo:No\. Tuesday’s update introduced a payment bug that can charge mobile orders twice\. Seventeen of our 120 beta users were affected; desktop orders were not affected\. I can finish the patch by Thursday\. Nina:QA needs two full days after the patch, so we should move the launch to Monday\. I will notify the beta users and refund the 17 affected users today\. Maya:Agreed\. We will make the final go/no\-go decision on Friday at 3 p\.m\.

Light\-blue text marks AWQ’s factual timeline error: it places the patch on Tuesday, the day the bug was introduced, and omits the two\-day QA period\. FAMPWQ preserves the cause, impact, QA delay, and Monday launch without introducing this contradiction, although it gives less precise patch timing and omits the notification and refund action\. This example illustrates a specific low\-bit failure mode rather than an aggregate claim\.