Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization

# Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization
Source: [https://arxiv.org/html/2605.20295](https://arxiv.org/html/2605.20295)
Jinghe Zhang, Daliang Xu, Chenghua Wang, Weikai Xie, Tao Qi, Yun Ma, Mengwei Xu, Gang Huang

###### Abstract

Large language models \(LLMs\) are increasingly deployed on mobile devices, where Neural Processing Units \(NPUs\) necessitate fully static quantization for optimal inference efficiency\. However, existing post\-training quantization \(PTQ\) methods predominantly rely on dynamic activation quantization, rendering them incompatible with NPU hardware constraints\. To bridge the gap between high\-fidelity PTQ and NPU\-constrained inference, we proposeQuant\.npu, a integer\-only fully static quantization framework\. It incorporates learnable quantization parameters and rotation matrices, enabling low\-bit activation\-weight quantization without runtime quantization parameters re\-computation\. Crucially, we identify that initialization and selective optimization of quantization parameters is pivotal for optimization stability, as improper initialization and naive joint optimization induce gradient instability that disrupts the optimization of rotation matrices\. To address this, we propose a rotation\-and\-bit\-width\-aware initialization tailored to diverse activation profiles and a distribution\-aware selective optimization \(two\-stage quantization pipeline\) tailored to rotated and unrotated tensors\. Furthermore, we introduce a sensitivity\-guided adaptive mixed\-precision scheme to balance accuracy with inference efficiency\. Extensive experiments on real\-world mobile NPUs demonstrate thatQuant\.npuachieves comparable accuracy to state\-of\-the\-art methods, while reducing inference latency by up to 15\.1%\.

## 1Introduction

On\-device LLM inference has emerged as a crucial research direction, primarily driven by its inherent advantages in safeguarding data privacy and enabling low\-latency, network\-independent operations[shafee2025privacy](https://arxiv.org/html/2605.20295#bib.bib33);[pamadi2025edge](https://arxiv.org/html/2605.20295#bib.bib25)\. To enable the energy\-efficient execution on mobile devices, manufacturers integrate Neural Processing Units \(NPUs\), an LLM\-specialized hardware into their System\-on\-Chips \(SoCs\)\. For instance, Qualcomm’s Hexagon NPU features dedicated matrix units for high\-throughput GEMM, vector units for element\-wise processing, and specialized direct memory access \(DMA\) engines to maximize memory bandwidth utilization\. These architectural enhancements deliver more than 10×\\timesperformance and 4×\\timesenergy efficiency than mobile GPUs[xu2024fastondevicellminference](https://arxiv.org/html/2605.20295#bib.bib41)\.

These efficiency gains, however, impose specific architectural constraints: \(i\) Preference for integer arithmetic: NPUs prioritize high\-throughput integer matrix multiplication due to the superior area and energy efficiency of integer units compared to floating\-point counterparts[xu2024fastondevicellminference](https://arxiv.org/html/2605.20295#bib.bib41)\. \(ii\) Requirement for static quantization: NPUs are architected for static quantization[qualcomm\_applyencodings\_2026](https://arxiv.org/html/2605.20295#bib.bib28)to avoid the high computational overhead of dynamic reduction operations \(e\.g\., on\-the\-fly min/max computation\)\. For instance, performing a tree reduction on a 128\-element INT8 vector requires at least 7 vector instruction cycles, which creates a bottleneck\. \(iii\) Inefficiency of fine\-grained quantization: NPUs generally rely on systolic array\-based cores, favoring coarse\-grained quantization[qualcomm\_applyencodings\_2026](https://arxiv.org/html/2605.20295#bib.bib28)\. Fine\-grained methods \(e\.g\., per\-block\) incur frequent de\-quantization overhead, disrupting the pipeline and reducing throughput\. On Hexagon NPUs, standard W4A8 per\-tensor inference is≈\\approx20% faster than W4A16 per\-block\.

However, prevailing quantization paradigms[frantar2023gptqaccurateposttrainingquantization](https://arxiv.org/html/2605.20295#bib.bib13);[xiao2024smoothquantaccurateefficientposttraining](https://arxiv.org/html/2605.20295#bib.bib40);[liu2025spinquantllmquantizationlearned](https://arxiv.org/html/2605.20295#bib.bib21);[sun2025flatquantflatnessmattersllm](https://arxiv.org/html/2605.20295#bib.bib36), which are predominantly tailored forGPUs and dynamic quantization, fundamentally struggle to align with mobile NPUs\. While Post\-Training Quantization \(PTQ\) methods attempt to mitigate accuracy loss through second\-order optimization[frantar2023gptqaccurateposttrainingquantization](https://arxiv.org/html/2605.20295#bib.bib13)and channel\-wise outlier suppression[xiao2024smoothquantaccurateefficientposttraining](https://arxiv.org/html/2605.20295#bib.bib40), their fully static variants remain highly vulnerable to the extreme activation outliers inherent in LLMs, leading to severe accuracy degradation\. To address this, recent rotation\-based approaches[liu2025spinquantllmquantizationlearned](https://arxiv.org/html/2605.20295#bib.bib21);[sun2025flatquantflatnessmattersllm](https://arxiv.org/html/2605.20295#bib.bib36)employ learnable rotations to smooth activation distributions and mitigate the impact of outliers\. However, these methods typically assume support for dynamic activation quantization during deployment and use dynamic quantization simulation during optimization\. This assumption conflicts with the static compilation constraints of mobile NPUs, often causing severe accuracy collapse when models are converted to a static setting\.

To address this, this paper integrates fully static quantization with learnable quantization parameters and rotation matrices, bridging the gap between optimization and deployment to prevent performance degradation\. However, achieving this is non\-trivial\. The primary distinction between static and dynamic quantization lies in the accuracy of quantization parameter estimation\. Inappropriate settings introduce significant quantization error, causing the optimization process to converge slowly or fail\. Our preliminary experiments \([Section˜3](https://arxiv.org/html/2605.20295#S3)\) confirm that these settings critically impact convergence stability: \(i\) Initialization of activation quantization parameters: The initial values determines the clipping range\. Suboptimal initialization leads to either excessive clipping distortion \(if the range is too narrow\) or wasted bit\-width resolution \(if the range is too wide\), creating a poor optimization landscape and insufficient convergence\. \(ii\) Selectivity in learnable quantization parameters: We observe that jointly optimizing the quantization parameters for every tensor can be counterproductive\.

Our solution\.To address these challenges, we proposeQuant\.npu, an algorithm\-system co\-designed system with full static quantization, tailored for practically efficient on\-NPU LLM inference\. It incorporates three key techniques: \(i\) Rotation\-and\-bit\-width\-aware Initialization: We demonstrate that the convergence accuracy of typical initialization strategies \(e\.g\., Max\-Min vs\. Mean\) varies depending on the target bit\-width and the presence of rotations\. Our method adapts the initialization accordingly to ensure a stable starting point\. \(ii\) Distribution\-Aware Selective Optimization \(two\-stage quantization pipeline\): We effectively decouple the optimization of rotated and unrotated distributions\. Since unrotated distributions are harder to optimize and increase complexity, we treat them distinctly\. \(iii\) Layer\-wise Adaptive Mixed\-Precision Strategy: Guided by a quantization sensitivity metric, we selectively assign higher bit\-widths \(e\.g\., 16\-bit\) to sensitive components\. This allowsQuant\.nputo employ high precision where necessary, reducing inference latency overhead\.

We evaluateQuant\.npuon a real mobile NPU \(Qualcomm SM8650\) using 4 mobile\-sized \(1–3B\) LLMs across 3 real scenario and 6 accuracy datasets\. To ensure zero inference overhead, we introduce only two rotation matrices \(R1,R2R\_\{1\},R\_\{2\}\) that are fused into weight matrices offline, as shown in[Figure˜2](https://arxiv.org/html/2605.20295#S4.F2)\. To fully leverage hardware capabilities,Quant\.npuemploys per\-tensor quantization for activations and per\-channel quantization for weights\. At identical inference latency, our method consistently achieves higher accuracy on downstream tasks than existing approaches\. Furthermore, compared to state\-of\-the\-art approaches, which are up to 15\.1% slower than our method,Quant\.npuincurs only a marginal performance gap, with an average accuracy drop of 2\.58% and a PPL increase of 1\.23\.

Our main contributions are summarized as follows:

- •Identification:We find that the optimization stability of fully static quantization is highly sensitive to the initialization and the selection of learnable quantization parameters\.
- •Framework:We propose a mobile NPU\-friendly quantization framework that incorporates novel rotation\-and\-bit\-width\-aware initialization and distribution\-aware selective optimization to effectively align optimization process and deployment under static NPU constraints\.
- •Performance:Our method achieves comparable SOTA accuracy with up to 15\.1% lower latency\.

![Refer to caption](https://arxiv.org/html/2605.20295v1/x1.png)\(a\)Fluctuation of dynamic activation scale forWoW\_\{o\}of the21t​h21^\{th\}layer\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x2.png)\(b\)Loss convergence curves under different optimization settings\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x3.png)\(c\)Different components in Stage One impacts on accuracy and time\.

Figure 1:The influence of scale on training\.[Figure˜1\(a\)](https://arxiv.org/html/2605.20295#S1.F1.sf1)shows large fluctuations in dynamic scale during optimization\.[Figure˜1\(b\)](https://arxiv.org/html/2605.20295#S1.F1.sf2)indicates that the learnability and initialization of quantization parameters are crucial for convergence\. In[Figure˜1\(c\)](https://arxiv.org/html/2605.20295#S1.F1.sf3), "A", "OA", and "W" denote the input activation, output activation, and weights of the linear layers\. "KV" refers to the Key and Value matrices\.
## 2Background

### 2\.1LLM Inference on Mobile NPUs

Modern SoCs increasingly integrate NPUs to accelerate LLM inference[10592049](https://arxiv.org/html/2605.20295#bib.bib31)by optimizing integer arithmetic, offering a2\.0×2\.0\\timesspeedup over standard floating\-point operations \([Table˜3](https://arxiv.org/html/2605.20295#A2.T3)in Appendix\)\. To further maximize efficiency, hardware architectures like Qualcomm NPUs favor coarse\-grained quantization schemes \(per\-tensor activation and per\-channel weight\) over fine\-grained per\-block alternatives, yielding an additional1\.2×1\.2\\timesspeedup\. However, this efficient coarse\-grained scheme causes significant accuracy degradation \(e\.g\., 7\.2% loss on SmolLM2\-1\.7B\) compared to per\-block methods like LPBQ[aimet\_lpbq\_2026](https://arxiv.org/html/2605.20295#bib.bib29)\. To reconcile this trade\-off,Quant\.npuadopts a hardware\-friendly coarse\-grained configuration to accelerate inference while preserving model performance\.

### 2\.2Quantization

Quantization[gholami2021surveyquantizationmethodsefficient](https://arxiv.org/html/2605.20295#bib.bib14);[frantar2023gptqaccurateposttrainingquantization](https://arxiv.org/html/2605.20295#bib.bib13)reduces memory footprint and latency by lowering numerical precision\. For a full\-precision tensorXF​PX\_\{FP\}, uniform affine quantization is defined asXQ=α\(⌊XF​P/α\+β⌉−β\)X\_\{Q\}=\\alpha\(\\lfloor X\_\{FP\}/\\alpha\+\\beta\\rceil\-\\beta\), whereα\\alphais the scale andβ\\betais the zero\-point\. Symmetric schemes setα=max⁡\(\|XF​P\|\)/\(2N−1−1\)\\alpha=\\max\(\|X\_\{FP\}\|\)/\(2^\{N\-1\}\-1\)andβ=0\\beta=0, while asymmetric schemes defineα=\(max⁡\(XF​P\)−min⁡\(XF​P\)\)/\(2N−1\)\\alpha=\(\\max\(X\_\{FP\}\)\-\\min\(X\_\{FP\}\)\)/\(2^\{N\}\-1\)andβ=round​\(−min⁡\(XF​P\)/α\)\\beta=\\text\{round\}\(\-\\min\(X\_\{FP\}\)/\\alpha\), whereNNis the bit\-width\. However, LLMs present a challenge due to activation distributions with significant outliers[dettmers2022llmint88bitmatrixmultiplication](https://arxiv.org/html/2605.20295#bib.bib9);[xiao2024smoothquantaccurateefficientposttraining](https://arxiv.org/html/2605.20295#bib.bib40), which lead to a marked drop in performance\.

Rotation\-based QuantizationPrior works[ashkboos2024quarotoutlierfree4bitinference](https://arxiv.org/html/2605.20295#bib.bib2);[liu2025spinquantllmquantizationlearned](https://arxiv.org/html/2605.20295#bib.bib21)have shown that Hadamard matrices \(H∈\{\+1,−1\}n×nH\\in\\\{\+1,\-1\\\}^\{n\\times n\}\) are particularly effective in mitigating outliers, as they can redistribute extreme values across all channels, thereby suppressing their impact\. Owing to the orthogonality \(H⊤​H=IH^\{\\top\}H=I\), the following equivalence always holds:Y=X​W⊤=\(X​H\)​\(H⊤​W⊤\)Y=XW^\{\\top\}=\(XH\)\(H^\{\\top\}W^\{\\top\}\)\. This property allows activations and weights to be rotated without altering the end\-to\-end inference output of the model\.

Prior work[liu2025spinquantllmquantizationlearned](https://arxiv.org/html/2605.20295#bib.bib21)has explored four types of Hadamard matrices for activations \(Figure[2](https://arxiv.org/html/2605.20295#S4.F2)\):R1R\_\{1\}\(shared across layers\),R2R\_\{2\}\(applied to Value\),R3R\_\{3\}\(applied to Query and Key\), andR4R\_\{4\}\(applied to down\-projection\)\. Among these,R1R\_\{1\}andR2R\_\{2\}can be fused into the weights \(W​HWH\) offline, eliminating runtime overhead\. It further improvesR1R\_\{1\}andR2R\_\{2\}using Cayley optimization to discover superior rotation matrices while maintaining orthogonality\. In contrast,R3R\_\{3\}andR4R\_\{4\}require online floating\-point matrix multiplications during inference, introducing approximately 2×\\timescomputational overhead\.

## 3Motivation

Joint optimization of static parameters and rotation matrices\.Existing optimization\-based methods typically rely on dynamic quantization to simulate errors\. However, as shown in[Figure˜1\(a\)](https://arxiv.org/html/2605.20295#S1.F1.sf1), the dynamic activation scale forWoW\_\{o\}of the21t​h21^\{th\}layer fluctuates significantly during the optimization \(e\.g\., a variance of 0\.58\)\. This instability indicates that dynamic simulation fails to capture the static quantization error\. Consequently, directly converting the optimized model to static quantization for on\-NPU inference leads to notable accuracy degradation \(e\.g\., 15\.61% for the SmolLM2\-1\.7B\-Instruct model\)\. In contrast, fixing quantization parameters before optimization prevents such mismatch but severely hinders convergence, resulting in suboptimal performance as shown in[Figure˜1\(b\)](https://arxiv.org/html/2605.20295#S1.F1.sf2)\(e\.g\., loss variance of 0\.3\)\. This is because the optimal range of quantization parameters continuously shifts as rotation matrices are optimized\. Therefore, the strong coupling between rotation matrices and quantization parameters implies that static calibration prior to joint optimization is inadequate\.

Impact of initialization on convergence\.Our experiments further reveal that quantization parameters initialization is critical for optimization\. As shown in[Figure˜1\(b\)](https://arxiv.org/html/2605.20295#S1.F1.sf2), poor initialization can significantly slow convergence, leading to inferior results \(e\.g\., final loss variance of 0\.8\) or even divergence\.

Selectivity of learnable quantization parameters\.A quantized LLM encompasses a number of static quantization parameters\. Counterintuitively,[Figure˜1\(c\)](https://arxiv.org/html/2605.20295#S1.F1.sf3)shows that increasing the number of learnable quantization parameters does not improve performance\. Specifically, jointly optimizing the input activations, output activations, weight of linear layers and KV tensors results in the lowest accuracy\. In contrast, excluding output activations improves performance, while optimizing only the input activation and weight quantization parameters of linear layers yields the best performance\.

Latency\-accuracy trade\-off in rotation\-based quantization\.Recent rotation\-based quantization methods[ashkboos2024quarotoutlierfree4bitinference](https://arxiv.org/html/2605.20295#bib.bib2);[liu2025spinquantllmquantizationlearned](https://arxiv.org/html/2605.20295#bib.bib21)effectively mitigate outliers by applying Hadamard matrices to activations\. While offline\-fused rotations \(R1,R2R\_\{1\},R\_\{2\}\) incur no latency, online rotations \(R3,R4R\_\{3\},R\_\{4\}\) require floating\-point matrix multiplications during inference, introducing approximately2×2\\timescomputational overhead\. To improve NPU inference efficiency,Quant\.npucompletely eliminates these online rotations\. However, this removal causes the input activations of down\_proj to exhibit severe outliers, making them significantly more difficult to quantize and leading to substantial performance degradation\.

Insights: Motivated by these findings, Quant\.npu jointly optimizes static quantization parameters and rotation matrices through three key strategies: \(i\) rotation\-and\-bit\-width\-aware initialization \([Section˜4\.2](https://arxiv.org/html/2605.20295#S4.SS2)\) to stabilize convergence; \(ii\) distribution\-aware selective optimization \([Section˜4\.3](https://arxiv.org/html/2605.20295#S4.SS3)\) to improve optimization effectiveness and manage complexity; \(iii\) adaptive mixed\-precision \([Section˜4\.4](https://arxiv.org/html/2605.20295#S4.SS4)\) to preserve accuracy while eliminating high\-latency online rotations\.

## 4Method

### 4\.1Overall Framework

In this section, we analyze the distribution differences between rotated and unrotated activations and introduce our integer\-only fully static quantization framework,Quant\.npu\. As illustrated in[Figure˜2](https://arxiv.org/html/2605.20295#S4.F2), it consists of three key components: 1\) a rotation\-and\-bit\-width\-aware initialization strategy \([Section˜4\.2](https://arxiv.org/html/2605.20295#S4.SS2)\), which secures an optimal initial state and stabilizes subsequent optimization; 2\) a distribution\-aware selective optimization \([Section˜4\.3](https://arxiv.org/html/2605.20295#S4.SS3)\), which decouples the quantization into a two\-stage quantization pipeline, isolating sensitive heavy\-tailed tensors from joint optimization, thereby reducing resource overhead and improving model performance; and 3\) an outlier\-aware adaptive mixed\-precision strategy \([Section˜4\.4](https://arxiv.org/html/2605.20295#S4.SS4)\), which mitigates outlier effects introduced by removing online rotation matrices \(R4\) with minimal computational cost while preserving model accuracy\.

### 4\.2Rotation\-and\-bit\-width\-aware Initialization

Analysis of Scale Initialization\.We first analyze two representative methods:Mean\-based[bhalgat2020lsqimprovinglowbitquantization](https://arxiv.org/html/2605.20295#bib.bib4)andMax\-Min\. The former determines quantization parameters from distribution statistics \(mean and variance\), while the latter relies on the full value range:si​n​i​t=m​a​x​\(\|μ−3​σ\|,\|μ\+3​σ\|\)/2b−1s\_\{init\}=max\(\|\\mu\-3\\sigma\|,\|\\mu\+3\\sigma\|\)/2^\{b\-1\}andsi​n​i​t=\(Xm​a​x−Xm​i​n\)/\(2b−1\)s\_\{init\}=\(X\_\{max\}\-X\_\{min\}\)/\(2^\{b\}\-1\), respectively\. Here,bbdenotes the bit\-width, andμ\\muandσ\\sigmaare the mean and standard deviation, andXm​a​xX\_\{max\}/Xm​i​nX\_\{min\}denote the maximum/minimum values of the tensor\. A more comprehensive analysis of additional initialization methods is provided in the Appendix[B](https://arxiv.org/html/2605.20295#A2)\.

As indicated by the equations, theMax\-Minmethod captures a broad dynamic range, ensuring uniform coverage of all data points\. However, this often results in coarser quantization granularity for the dense regions of the data\. In contrast, theMean\-basedmethod provides higher quantization precision for the central distribution, effectively clipping extreme values \(e\.g\., outliers\)\. Consequently,Max\-Mininitialization is preferable for higher bit\-widths \(e\.g\., 8\-bit or 16\-bit\), where the quantization error remains negligible even when the numerical distribution spans a wide range\. Conversely, when the data approximately follows a Gaussian distribution, theMean\-basedmethod yields better performance under low\-bit quantization by minimizing error in the most information\-rich regions\.

Activation Distribution Analysis\.Furthermore, we categorize activations into two types based on rotation:unrotatedandrotated, as illustrated in[Figure˜5](https://arxiv.org/html/2605.20295#A4.F5)\. Rotated activations exhibit a Gaussian distribution, whereas unrotated activations typically follow a heavy\-tailed distribution\. This distinction aligns well with the characteristics of the initialization methods discussed above\. Therefore, for rotated activations \(e\.g\., input activation ofWqW\_\{q\},WoW\_\{o\},Wu​pW\_\{up\}\), we employMean\-basedinitialization\. Conversely, unrotated activations requireMax\-Mininitialization to accommodate their wide range\.

Initialization Rule\.Based on the above analysis, we summarize our choice of initialization methods as follows: \(1\) For rotated activations, we adoptMean\-basedinitialization to support lower\-bit quantization\. \(2\) For unrotated activations, we useMax\-Mininitialization with higher bit\-widths \(e\.g\., 8\-bit or 16\-bit\) to accommodate their wide range, thereby ensuring model accuracy\. Detailed discussion and theoretical analysis of this initialization rule are provided in Appendix[C](https://arxiv.org/html/2605.20295#A3)and Appendix[D](https://arxiv.org/html/2605.20295#A4)\.

![Refer to caption](https://arxiv.org/html/2605.20295v1/x4.png)Figure 2:The Overview ofQuant\.npu\. The quantization is divided into two stages\. In the first stage, we optimize the quantization parameters for the "hot" merged weights and activations while keeping the other components at BF16 precision\. In the second stage, we directly apply static calibration to the remaining activations and weights\. In the diagram, black lines represent BF16, purple represents INT16, blue represents INT8, and green indicates a mixed\-precision scheme of INT8 and INT16\.
### 4\.3Distribution\-Aware Selective Optimization

In this work, we categorize quantization parameters into two types: \(1\) those associated with rotated distributions, such as the input activations and weights of linear layers; and \(2\) those associated with non\-rotated distributions, including the output activations of linear layers and key/value tensors\.

The fundamental distinction between these categories lies in their distribution smoothness and quantization sensitivity\. Rotated distributions are significantly smoother and approximate a Gaussian profile \(see[Figure˜5\(e\)](https://arxiv.org/html/2605.20295#A4.F5.sf5)–[Figure˜5\(h\)](https://arxiv.org/html/2605.20295#A4.F5.sf8)\), allowing their quantization parameters to be stably updated during the optimization\. In contrast, non\-rotated distributions exhibit pronounced heavy\-tailed behavior \(see[Figure˜5\(a\)](https://arxiv.org/html/2605.20295#A4.F5.sf1)–[Figure˜5\(d\)](https://arxiv.org/html/2605.20295#A4.F5.sf4)\), with values concentrated near zero\. These distributions are inherently more sensitive to quantization, making their parameters difficult to optimize and prone to instability\. A more detailed theoretical analysis of the optimization instability is provided in Appendix[E](https://arxiv.org/html/2605.20295#A5)\. As illustrated in[Figure˜1\(c\)](https://arxiv.org/html/2605.20295#S1.F1.sf3), the convergence time is substantially increased without performance improvement\. Furthermore, our initialization analysis in[Section˜4\.2](https://arxiv.org/html/2605.20295#S4.SS2)indicates that non\-rotated tensors require 8\-bit or higher precision\. Given this sufficient representational capacity, directly applying static calibration without further optimization to these components incurs negligible accuracy degradation\.

Leveraging these insights, we propose a selective optimization pipeline that decouples the quantization process into two stages: Gradient\-based Optimization and Static Calibration\. This design significantly reduces optimization complexity and enhances stability without compromising model accuracy\.

#### 4\.3\.1Stage One: Gradient\-based Optimization

In the first stage, we jointly optimize the quantization parameters of input activations and weights for all linears together with the rotation matrices\. Specifically, following prior work[liu2025spinquantllmquantizationlearned](https://arxiv.org/html/2605.20295#bib.bib21), the input activations, output activations, and weights of the lm\_head linear are not quantized in this stage and remain in floating\-point precision\. Following[esser2020learnedstepsizequantization](https://arxiv.org/html/2605.20295#bib.bib12);[bhalgat2020lsqimprovinglowbitquantization](https://arxiv.org/html/2605.20295#bib.bib4), we derive gradients for scale and zero\-point parameters, apply gradient scaling, and introduce a local error loss to improve optimization stability\.

Learnable Scale and Zero\-Point\.We adopt the straight\-through estimator \(STE\)[bengio2013estimatingpropagatinggradientsstochastic](https://arxiv.org/html/2605.20295#bib.bib3)to approximate gradients through the rounding operator, to achieve differentiable optimization of quantitative parameters\. The gradient of the scale parameterssis approximated as:

∂x^∂s=∂x¯∂s​s\+x¯−zp≈\{−\(xs\+zp\)\+⌊xs\+zp⌉,if​qmin<xs\+zp<qmax,qmin−zp​or​qmax−zp,otherwise\.\\frac\{\\partial\\hat\{x\}\}\{\\partial s\}=\\frac\{\\partial\\bar\{x\}\}\{\\partial s\}\\,s\+\\bar\{x\}\-z\_\{p\}\\approx\\begin\{cases\}\-\\left\(\\dfrac\{x\}\{s\}\+z\_\{p\}\\right\)\+\\left\\lfloor\\dfrac\{x\}\{s\}\+z\_\{p\}\\right\\rceil,&\\text\{if \}q\_\{\\min\}<\\dfrac\{x\}\{s\}\+z\_\{p\}<q\_\{\\max\},\\\\\[6\.0pt\] q\_\{\\min\}\-z\_\{p\}\\;\\;\\text\{or\}\\;\\;q\_\{\\max\}\-z\_\{p\},&\\text\{otherwise\}\.\\end\{cases\}\(1\)Similarly, the gradient of the zero\-pointzpz\_\{p\}is:

∂x^∂zp=\(∂x¯∂zp−1\)​s≈\{0,if​qmin<xs\+zp<qmax,−s,otherwise\.\\frac\{\\partial\\hat\{x\}\}\{\\partial z\_\{p\}\}=\\left\(\\frac\{\\partial\\bar\{x\}\}\{\\partial z\_\{p\}\}\-1\\right\)s\\approx\\begin\{cases\}0,&\\text\{if \}q\_\{\\min\}<\\dfrac\{x\}\{s\}\+z\_\{p\}<q\_\{\\max\},\\\\\[6\.0pt\] \-s,&\\text\{otherwise\}\.\\end\{cases\}\(2\)
Gradient Scaling\.During optimization, both rotation matrices and quantization parameters are updated jointly\. However, the gradient magnitude of quantization parameters increases with the number of elements and bit\-width, which leads to the gradient of the quantization parameter being much greater than that of the rotation matrix\. To stabilize optimization, we apply the gradient scaling factorg=1/N⋅Qm​a​xg=1/\\sqrt\{N\\cdot Q\_\{max\}\}proposed in the prior work[esser2020learnedstepsizequantization](https://arxiv.org/html/2605.20295#bib.bib12), whereNNis the number of elements sharing the same quantization parameter, andQm​a​xQ\_\{max\}is the maximum representable integer value\.

Local Quantization Error LossTo further stabilize early\-stage optimization and accelerate convergence, we introduce a local quantization error loss for activation quantizers:

mins,zp⁡‖𝒟​𝒬​\(𝒬​\(𝐱\)\)−𝐱‖22\\displaystyle\\min\_\{s,z\_\{p\}\}\\;\\left\\\|\\mathcal\{DQ\}\(\\mathcal\{Q\}\(\\mathbf\{x\}\)\)\-\\mathbf\{x\}\\right\\\|\_\{2\}^\{2\}\(3\)where𝒬​\(⋅\)\\mathcal\{Q\}\(\\cdot\)and𝒟​𝒬​\(⋅\)\\mathcal\{DQ\}\(\\cdot\)respectively represent quantization and dequantization operators\. This loss directly constrains the reconstruction error between original and dequantized activations, enabling quantization parameters to converge to favorable values early in the optimization process\.

#### 4\.3\.2Stage Two: Static Calibration

After the first stage \(Gradient\-based Optimization\), we directly apply static calibration to the quantization parameters for the remaining operators, without further optimization\. These include the output activations of linear layers, key/value tensors, and non\-linear activation functions such as SiLU\.

### 4\.4Adaptive Mixed\-Precision Quantization

As discussed in[Section˜2\.2](https://arxiv.org/html/2605.20295#S2.SS2), we eliminate online rotation matrices to avoid floating\-point overhead during inference\. Consequently, down\_proj input activations are no longer smoothed by rotation\. Following the principles in[Section˜4\.2](https://arxiv.org/html/2605.20295#S4.SS2), these activations require a minimum precision of 8\-bit\.

However, prior work[xu2024fastondevicellminference](https://arxiv.org/html/2605.20295#bib.bib41)indicates that down\_proj inputs exhibit severe outliers, as illustrated in Appendix[F](https://arxiv.org/html/2605.20295#A6), causing significant accuracy degradation even at 8\-bit quantization \(see[Figure˜4](https://arxiv.org/html/2605.20295#S5.F4)\)\. Further analysis reveals that quantization sensitivity varies significantly across linear layers[xu2024fastondevicellminference](https://arxiv.org/html/2605.20295#bib.bib41)\. We observe that selectively promoting a small fraction of highly sensitive activations across all down\_proj layers from 8\-bit to 16\-bit effectively recovers model accuracy\. Leveraging this insight, we propose an adaptive mixed\-precision strategy that assigns bit\-widths based on a quantization sensitivity metric\.

Quantization Sensitivity Metric\.We identify two distinct activation distribution patterns that lead to quantization\-induced accuracy degradation: \(1\) The activation distribution has a large dynamic range that exceeds the representational capacity of the static quantizer\. \(2\) The activation values are mostly clustered near zero, so quantization preserves these small values but loses the important information carried by a few outliers\. Both cases are characterized by the amplification of quantization error due to outliers\. We therefore define a quantization sensitivity metric based on relative quantization error:

ratio=1N​∑i=1N\|dequantizedi−original\_activationi\|\|original\_activationi\|\+10−8\\displaystyle\\text\{ratio\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{\\left\|\\text\{dequantized\}\_\{i\}\-\\text\{original\\\_activation\}\_\{i\}\\right\|\}\{\\left\|\\text\{original\\\_activation\}\_\{i\}\\right\|\+10^\{\-8\}\}\(4\)whereoriginal\_activationdenotes the original activation tensor,dequantizedis the corresponding dequantized tensor, andNNis the number of elements\. In practice, we quantize activations using low\-bitMax\-Mininitialization and compute this metric\. A ratio close to 1 indicates severe outlier\-induced error and insufficient bit\-width, whereas a ratio near 0 indicates the quantization is nearly lossless\.

Adaptive Mixed\-Precision \(8\-/16\-bit\)\.The adaptive strategy targets down\_proj input activations\. We initially apply 8\-bit quantization withMax–Mininitialization to these, compute sensitivity ratios, and selectively increase the bit\-width of highly sensitive activations to 16\-bit\. As shown in[Section˜5\.4](https://arxiv.org/html/2605.20295#S5.SS4), adjusting only the top 10% of activations by the sensitivity metric \(e\.g\., for a model with 30 layers, only 3 layers are adjusted\) is sufficient to achieve performance comparable to full 16\-bit quantization\.

Table 1:Evaluation on Llama\-3\.2\-3B\-Instruct: Perplexity\(PPL\) and Zero\-shot QA task accuracy results of 4\-bit weight or 8\-bit weight quantized models\.ModelMethodW\-A\-KVW\. Gran\.PPL\(C4\)↓\\downarrowPIQAWinograndeHellaSwagARC\-EARC\-CLAMBADAAvg↑\\uparrowLlama\-3\.2\-3B\-itFP32––16\.850\.75410\.67480\.70020\.55930\.41980\.56960\.613executorch4\-16\-8per\-block18\.190\.73610\.67720\.68780\.53280\.39420\.53290\.5935executorch8\-8\-8per\-channel17\.760\.73720\.66380\.6940\.55180\.42660\.55270\.6044QuaRot8\-8\-8per\-channel17\.060\.74480\.66540\.69580\.54670\.41550\.56390\.6054SpinQuant8\-8\-8per\-channel17\.180\.74050\.66770\.69360\.55010\.41130\.55770\.6035Quant\.npu8\-8\-8per\-channel16\.420\.75410\.68270\.70030\.54460\.41040\.60660\.6165executorch4\-8\-8per\-channel26\.390\.70400\.60460\.6280\.47980\.36350\.38350\.5272QuaRot4\-8\-8per\-channel30\.060\.67080\.56910\.57550\.46090\.32340\.26000\.4766SpinQuant4\-8\-8per\-channel28\.780\.63930\.56510\.56200\.41790\.29780\.29150\.4623Quant\.npu4\-8\-8per\-channel19\.160\.73780\.64960\.67770\.51050\.38910\.53170\.5827executorch4\-4\-8per\-channel6950\.810\.51960\.51380\.24930\.26010\.24740\.00\.2984QuaRot4\-4\-8per\-channel481\.820\.51960\.50910\.27680\.26890\.25170\.00140\.3046SpinQuant4\-4\-8per\-channel528\.970\.51030\.48780\.27310\.26140\.22950\.00190\.2940Quant\.npu4\-4\-8per\-channel21\.760\.72090\.62190\.64200\.49160\.36950\.46890\.5525

## 5Experiments

Evaluation models and datasets\.Our evaluation is conducted on LLaMA\-3\.2\-3B\-Instruct[grattafiori2024llama3herdmodels](https://arxiv.org/html/2605.20295#bib.bib15)\. Following prior work[ashkboos2024quarotoutlierfree4bitinference](https://arxiv.org/html/2605.20295#bib.bib2), we report the perplexity \(PPL\) of language generation tasks on the C4 datasets[raffel2023exploringlimitstransferlearning](https://arxiv.org/html/2605.20295#bib.bib30)\. In addition, we evaluate zero\-shot commonsense reasoning performance on six standard benchmarks: PIQA[bisk2019piqareasoningphysicalcommonsense](https://arxiv.org/html/2605.20295#bib.bib5), WinoGrande[sakaguchi2019winograndeadversarialwinogradschema](https://arxiv.org/html/2605.20295#bib.bib32), HellaSwag[zellers2019hellaswagmachinereallyfinish](https://arxiv.org/html/2605.20295#bib.bib45), ARC\-Easy, ARC\-Challenge[clark2018thinksolvedquestionanswering](https://arxiv.org/html/2605.20295#bib.bib8)and LAMBADA[paperno2016lambadadatasetwordprediction](https://arxiv.org/html/2605.20295#bib.bib26)\. All evaluations are developed with reference to the lm\-eval\-harness framework\.

BaselinesWe compare against three state\-of\-the\-art frameworks: \(i\) ExecuTorch[pytorch\_executorch\_2026](https://arxiv.org/html/2605.20295#bib.bib27), an edge runtime supporting Hexagon NPU inference with per\-block weight PTQ; \(ii\) QuaRot[ashkboos2024quarotoutlierfree4bitinference](https://arxiv.org/html/2605.20295#bib.bib2), a rotation\-based PTQ method with random Hadamard matrices; and \(iii\) SpinQuant[liu2025spinquantllmquantizationlearned](https://arxiv.org/html/2605.20295#bib.bib21), a rotation\-based PTQ method with learnable rotation matrices, utilizing dynamic quantization during optimization\.

Implementation Details\.The optimization stage in experiments employs 4 NVIDIA A40 GPUs \(48 GB\)\. We optimize on WikiText\-2 \(bfloat16, seq\_len 2048\) for 512 steps \(7 epochs\) using a batch size of 2 and 2 gradient accumulation steps\. To stabilize optimization, the local quantization error loss is applied for the first 128 steps \([Section˜4\.3](https://arxiv.org/html/2605.20295#S4.SS3)\)\. We use SGD with cosine decay, setting learning rates to 0\.1 for rotation matrices and 0\.01 for static quantization parameters\.

Quantization\.Quant\.npuadopts pre\-channel symmetric weight quantization and per\-tensor symmetric activation quantization\. For bit\-width assignment, we only vary the quantization precision of input activations and weights of linear layers, while keeping other tensors at fixed precision\. Specifically, KV cache and output activations of linear layers are fixed at 8\-bit precision, whereas remaining operations \(e\.g\., SiLU\) are fixed at 16\-bit precision to preserve model accuracy\. In particular, for the down\_proj layers, the activation bit\-width is assigned using the adaptive mixed\-precision strategy proposed in[Section˜4\.4](https://arxiv.org/html/2605.20295#S4.SS4)\. All static quantization parameters are initialized based on activation characteristics following in[Section˜4\.2](https://arxiv.org/html/2605.20295#S4.SS2)\. Calibration utilizes 16 samples from WikiText\-2[merity2016pointersentinelmixturemodels](https://arxiv.org/html/2605.20295#bib.bib23)\.

### 5\.1Main Results

Setup\.All baselines and our method Quant\.npu are implemented on ExecuTorch \(integer\-only quantization for NPU inference\) to ensure a fair comparison, modifying only the rotation matrices and quantizers\. Specifically, ExecuTorch uses 4\-bit block\-wise weights \(low\-power blockwise quantization, LPBQ[aimet\_lpbq\_2026](https://arxiv.org/html/2605.20295#bib.bib29)\) and 16\-bit activation quantization\. In contrast, another ExecuTorch variant, QuaRot, SpinQuant, andQuant\.npushare the same quantization configuration, including per\-tensor activation quantization, per\-channel weight quantization, and 16\-bit activations for all down\_proj layers\. The key difference is that QuaRot utilizes fixed Hadamard matrices, SpinQuant optimizes only rotation matrices, whereasQuant\.npujointly optimizes rotations and static quantization parameters\. Furthermore,Quant\.npuapplies a two\-stage quantization \([Section˜4\.3](https://arxiv.org/html/2605.20295#S4.SS3)\) for operators like SiLU\.

Results: higher accuracy at iso\-precision\.[Table˜1](https://arxiv.org/html/2605.20295#S4.T1)reports the perplexity \(PPL\) on C4 and the accuracy on six zero\-shot commonsense reasoning benchmarks under W8A8, W4A8, and W4A4 settings\. Across all major benchmarks,Quant\.npuoutperforms state\-of\-the\-art quantization approaches while maintaining equivalent inference efficiency\. Notably, under W8A8 quantization, Llama\-3\.2\-3B\-Instruct achieves zero\-shot accuracy comparable to, or even slightly better than, its FP32 baseline\. The advantage ofQuant\.npubecomes even more pronounced under the challenging W4A8 setting\. For Llama\-3\.2\-3B\-Instruct,Quant\.npu\-W4A8 reduces PPL from 28\.78 to 19\.16 and boosts zero\-shot accuracy from 46\.23% to 58\.27%, representing a 12\.04% improvement\. Furthermore,Quant\.npu\-W4A4 maintains a relatively high accuracy, even in scenarios where the baseline almost entirely fails, exhibiting only about a 6% drop in accuracy compared to the FP32 baseline\. Extensive supplementary experiments are detailed in Appendix[G](https://arxiv.org/html/2605.20295#A7)\. We validate the broad applicability ofQuant\.npuacross newer architectures \(Qwen3\-1\.7B[yang2025qwen3](https://arxiv.org/html/2605.20295#bib.bib43)\) and larger model \(Llama3\-8B[grattafiori2024llama3herdmodels](https://arxiv.org/html/2605.20295#bib.bib15)\) , while also verifying its ability to preserve instruction\-following capabilities\. Additional evaluation on more widely used models, comparisons with MobileQuant[tan2024mobilequant](https://arxiv.org/html/2605.20295#bib.bib37), and profiling of memory and energy consumption on the newer SM8750 NPU are provided to further confirm our method’s superiority\.

### 5\.2End\-to\-end Latency

Setup\.We evaluate inference latency on the SM8650 NPU using HellaSwag[zellers2019hellaswagmachinereallyfinish](https://arxiv.org/html/2605.20295#bib.bib45), Persona\-Chat[jandaghi2023faithfulpersonabasedconversationaldataset](https://arxiv.org/html/2605.20295#bib.bib18), and DroidTask[wen2024autodroidllmpoweredtaskautomation](https://arxiv.org/html/2605.20295#bib.bib39), spanning varied token lengths to show system performance under varying loads\.

Results: faster inference with negligible accuracy loss\.Quant\.npu\-W4A8 yields a up to 15\.1% speedup compared to ExecuTorch\-W4A16 with a negligible accuracy drop of 1% on Llama\-3\.2\-3B\-Instruct \(averaging a 2\.58% drop and 1\.23 PPL increase across all tasks and all models\)\. These results demonstrate that the joint optimization of rotation matrices and static quantization parameters enablesQuant\.nputo adopt lower\-bit and more coarse\-grained quantization than ExecuTorch\-W4A16, thereby improving inference efficiency without significantly compromising performance\.

![Refer to caption](https://arxiv.org/html/2605.20295v1/x5.png)Figure 3:End\-to\-end latency results, where blue bars represent the per\-block weight quantization in ExecuTorch; the yellow and red bars indicate per\-channel weight quantization inQuant\.npu\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x6.png)Figure 4:The trend of perplexity \(PPL\) with respect to the percentage of linear\-layer activations quantized to higher bit\-widths for the Llama\-3\.2\-3B\-Instruct model\.

### 5\.3Adaptive strategy

Setup\.We employ an adaptive mixed\-precision strategy prior to optimization, where activations of down\_proj layers are selectively promoted to 16\-bit precision to better preserve model accuracy\. Consistent with[Section˜5\.1](https://arxiv.org/html/2605.20295#S5.SS1), all evaluations fully utilize a pure integer\-arithmetic pipeline\. We further measure perplexity \(PPL\) on the C4 dataset to validate the effectiveness of the proposed strategy\.

Results\.[Figure˜4](https://arxiv.org/html/2605.20295#S5.F4)illustrates the impact of mixed\-precision ratios of all down\_proj activations on perplexity \(PPL\)\. We observe that increasing the precision of only a small fraction of down\_proj activations yields substantial accuracy gains\. For instance, promoting just 10% of activations to 16\-bit precision results in a PPL increase of only 0\.63 compared to the full 16\-bit baseline, while reducing PPL by 7\.75 relative to the full 8\-bit setting\. Based on this key insight,Quant\.npufurther reduces the use of high\-bit activations, thereby significantly improving inference efficiency in practice with only marginal accuracy degradation compared to well\-established ExecuTorch\-W4A16 baseline\.

Table 2:Ablation study of Quant\.npu’s main components on Llama3\.2\-3B\-Instruct: Perplexity \(PPL\) and zero\-shot QA task accuracy results under different component settings\. Here, "D/S" indicates whether dynamic or static quantization was used during optimization; "IM" refers to the initialization methodology; "Sel\-Opt" denotes the application of our selective optimization strategy; "Adaptive" indicates whether a mixed\-precision strategy was applied, with the value in parentheses representing the percentage of positions where the quantization bit\-width was increased\. The FP32 model does not involve any quantization\-related components \(D/S, Optim, IM, Sel\-Opt, Adaptive\)\. Thus, these fields are marked as "–", and this row represents the performance of the FP32 model\.ModelMethodD/SOptimIM\(Sec\. 4\.2\)Sel\-Opt\(Sec\. 4\.3\)Adaptive \(%\)\(Sec\. 4\.4\)PPL\(C4\)↓\\downarrowWinogrande↑\\uparrowARC\-C↑\\uparrowLAMBADA↑\\uparrowAvg↑\\uparrowLlama\-3\.2\-3B\-itFP32–––––22\.640\.60930\.39930\.47310\.4939SpinQuantDR1187\.690\.49170\.26190\.00040\.2513–SR150\.580\.50360\.21840\.07040\.2641–SR and Scale \(Joint\)69\.650\.51300\.26620\.11140\.2969–SR and Scale \(Joint\)✓36\.420\.56590\.29100\.28200\.3796–SR and Scale \(Joint\)✓✓30\.890\.58090\.31140\.31220\.4015OursSR and Scale \(Joint\)✓✓✓\(10%\)22\.090\.60620\.37030\.44340\.4733–SR and Scale \(Joint\)✓✓✓\(100%\)21\.560\.62190\.34560\.46810\.4785High PrecisionAloneSR and Scale \(Joint\)✓\(100%\)31\.160\.56510\.30550\.32140\.3973

### 5\.4Ablation Study

Setup\.To better demonstrate the effectiveness of our proposed method, we conduct an ablation study on the key components ofQuant\.npu\. All experiments are performed on LlaMA\-3\.2\-3B\-Instruct\. To clearly present the results, we quantize only the weights, input activations, and output activations of the linear layers, while other operators such as SiLU remain in full precision\.

Results\.As summarized in Table[2](https://arxiv.org/html/2605.20295#S5.T2), starting from a naive static quantization baseline, we progressively integrate each component of our method\. In the baseline setting, where both rotation matrices and quantization parameters are directly optimized under static quantization constraints, the model suffers significant degradation, yielding a perplexity \(PPL\) of 69\.65 and an average accuracy of 0\.2969\. Introducing our proposed initialization strategy significantly stabilizes optimization process and improves optimization quality, reducing PPL to 36\.42 and improving average accuracy to 0\.3796\. Building on this, the proposed distribution\-aware selective optimization, implemented through a two\-stage quantization pipeline, further improves model performance, achieving a PPL of 30\.89 and an average accuracy of 0\.4015\. Finally, by incorporating the adaptive mixed\-precision strategy, only 10% of the down\_proj activations \(3 out of 28 layers\) are promoted to 16\-bit precision, while the remaining activations are kept at 8\-bit, achieving a PPL of 22\.09 and an average accuracy of 0\.4733\. Notably, the resulting performance remains nearly identical to that of both the FP32 \(Avg: 0\.4939\) and full 16\-bit \(Avg: 0\.4785\) settings\. These results demonstrate that each proposed component contributes to performance improvement, and thatQuant\.npueffectively maintains performance close to the FP32 model, achieving a favorable trade\-off between accuracy and inference efficiency\.

## 6Conclusions

We have proposedQuant\.npu, an NPU\-friendly fully static quantization framework for on\-device LLMs that integrates learnable quantization parameters and rotation matrices to enable efficient low\-bit inference\. By incorporating a rotation\-and\-bit\-width\-aware initialization strategy together with a decoupled two\-stage selective optimization pipeline,Quant\.npuachieves competitive accuracy while significantly reducing inference latency\. Furthermore,Quant\.npudemonstrates strong generalization across different model architectures and scales, highlighting its scalability in practical deployment scenarios\. Experiments confirm thatQuant\.npuachieves comparable accuracy to state\-of\-the\-art methods while delivering up to a 15\.1% reduction in latency on mobile NPUs\.

## References

- \[1\]Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, and et al\.Smollm2: When smol goes big – data\-centric training of a small language model, 2025\.
- \[2\]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L\. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman\.Quarot: Outlier\-free 4\-bit inference in rotated llms, 2024\.
- \[3\]Yoshua Bengio, Nicholas Léonard, and Aaron Courville\.Estimating or propagating gradients through stochastic neurons for conditional computation, 2013\.
- \[4\]Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak\.Lsq\+: Improving low\-bit quantization through learnable offsets and better initialization, 2020\.
- \[5\]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi\.Piqa: Reasoning about physical commonsense in natural language, 2019\.
- \[6\]Yelysei Bondarenko, Riccardo Del Chiaro, and Markus Nagel\.Low\-rank quantization\-aware training for llms\.arXiv preprint arXiv:2406\.06385, 2024\.
- \[7\]Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo\.Efficientqat: Efficient quantization\-aware training for large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), pages 10081–10100, 2025\.
- \[8\]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord\.Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018\.
- \[9\]Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer\.Llm\.int8\(\): 8\-bit matrix multiplication for transformers at scale, 2022\.
- \[10\]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer\.Qlora: Efficient finetuning of quantized llms\.Advances in neural information processing systems, 36:10088–10115, 2023\.
- \[11\]Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu\.Bitdistiller: Unleashing the potential of sub\-4\-bit llms via self\-distillation\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), pages 102–116, 2024\.
- \[12\]Steven K\. Esser, Jeffrey L\. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S\. Modha\.Learned step size quantization, 2020\.
- \[13\]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh\.Gptq: Accurate post\-training quantization for generative pre\-trained transformers, 2023\.
- \[14\]Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W\. Mahoney, and Kurt Keutzer\.A survey of quantization methods for efficient neural network inference, 2021\.
- \[15\]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et al\.The llama 3 herd of models, 2024\.
- \[16\]Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan\.Deep learning with limited numerical precision\.InInternational conference on machine learning, pages 1737–1746\. PMLR, 2015\.
- \[17\]Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko\.Quantization and training of neural networks for efficient integer\-arithmetic\-only inference\.InProceedings of the IEEE conference on computer vision and pattern recognition, pages 2704–2713, 2018\.
- \[18\]Pegah Jandaghi, XiangHai Sheng, Xinyi Bai, Jay Pujara, and Hakim Sidahmed\.Faithful persona\-based conversational dataset generation with large language models, 2023\.
- \[19\]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei\-Ming Chen, Wei\-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han\.Awq: Activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of machine learning and systems, 6:87–100, 2024\.
- \[20\]Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra\.Llm\-qat: Data\-free quantization aware training for large language models\.InFindings of the Association for Computational Linguistics: ACL 2024, pages 467–484, 2024\.
- \[21\]Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort\.Spinquant: Llm quantization with learned rotations, 2025\.
- \[22\]Xinyin Ma, Gongfan Fang, and Xinchao Wang\.Llm\-pruner: On the structural pruning of large language models, 2023\.
- \[23\]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\.Pointer sentinel mixture models, 2016\.
- \[24\]Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort\.Overcoming oscillations in quantization\-aware training\.InInternational Conference on Machine Learning, pages 16318–16330\. PMLR, 2022\.
- \[25\]Vishesh Narendra Pamadi and Pushpa Singh\.Edge ai vs cloud ai: A comparative study of performance latency and scalability\.International Journal of Research in Modern Engineering & Emerging Technology \(IJRMEET\), 13\(3\):13–35, 2025\.
- \[26\]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández\.The lambada dataset: Word prediction requiring a broad discourse context, 2016\.
- \[27\]PyTorch\.ExecuTorch: On‑Device AI Inference Powered by PyTorch\.GitHub repository, 2026\.Version accessed Jan 2026\.
- \[28\]Qualcomm\.Applyencodings, 2026\.Qualcomm Documentation\. Accessed: 2026\-01\-29\.
- \[29\]Qualcomm Innovation Center, Inc\. \(AIMET\)\.Low\-power blockwise quantization \(lpbq\), 2026\.AIMET Documentation \(Version 2\.19\.0\)\. Accessed: 2026\-01\-29\.
- \[30\]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J\. Liu\.Exploring the limits of transfer learning with a unified text\-to\-text transformer, 2023\.
- \[31\]Alejandro Rico, Satyaprakash Pareek, Javier Cabezas, David Clarke, Baris Ozgul, Francisco Barat, Yao Fu, Stephan Münz, Dylan Stuart, Patrick Schlangen, and et al\.Amd xdna npu in ryzen ai processors\.IEEE Micro, 44\(6\):73–82, 2024\.
- \[32\]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi\.Winogrande: An adversarial winograd schema challenge at scale, 2019\.
- \[33\]Ahmed Shafee, SR Hasan, and Tasneem A Awaad\.Privacy and security vulnerabilities in edge intelligence: An analysis and countermeasures\.Computers and Electrical Engineering, 123:110146, 2025\.
- \[34\]Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo\.Omniquant: Omnidirectionally calibrated quantization for large language models, 2024\.
- \[35\]Mingjie Sun, Zhuang Liu, Anna Bair, and J\. Zico Kolter\.A simple and effective pruning approach for large language models, 2024\.
- \[36\]Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, and Jun Yao\.Flatquant: Flatness matters for llm quantization, 2025\.
- \[37\]Fuwen Tan, Royson Lee, Łukasz Dudziak, Shell Xu Hu, Sourav Bhattacharya, Timothy Hospedales, Georgios Tzimiropoulos, and Brais Martinez\.Mobilequant: Mobile\-friendly quantization for on\-device language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 9761–9771, 2024\.
- \[38\]Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu\.Outlier suppression\+: Accurate quantization of large language models by equivalent and optimal shifting and scaling\.arXiv preprint arXiv:2304\.09145, 2023\.
- \[39\]Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia\-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu\.Autodroid: Llm\-powered task automation in android, 2024\.
- \[40\]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han\.Smoothquant: Accurate and efficient post\-training quantization for large language models, 2024\.
- \[41\]Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu\.Fast on\-device llm inference with npus, 2024\.
- \[42\]Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian\.Qa\-lora: Quantization\-aware low\-rank adaptation of large language models\.arXiv preprint arXiv:2309\.14717, 2023\.
- \[43\]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388, 2025\.
- \[44\]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and et al\.Qwen2\.5 technical report, 2025\.
- \[45\]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi\.Hellaswag: Can a machine really finish your sentence?, 2019\.
- \[46\]Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci\.Plug\-and\-play: An efficient post\-training pruning method for large language models\.2024\.
- \[47\]Yilong Zhao, Chien\-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci\.Atom: Low\-bit quantization for efficient and accurate llm serving\.Proceedings of Machine Learning and Systems, 6:196–209, 2024\.

## Appendix ARelated Work

Quantization is widely recognized as one of the most practical techniques for deploying large language models \(LLMs\) on resource\-constrained platforms, as it effectively compresses model parameters and reduces computational and memory overhead\. Compared to pruning\-based approaches\[[22](https://arxiv.org/html/2605.20295#bib.bib22),[35](https://arxiv.org/html/2605.20295#bib.bib35),[46](https://arxiv.org/html/2605.20295#bib.bib46)\], quantization does not alter the network architecture and often yields superior performance\.

Early studies primarily focused on weight\-only quantization, where model weights are represented with low\-bit precision while activations remain in full precision\. Representative methods such as GPTQ\[[13](https://arxiv.org/html/2605.20295#bib.bib13)\]attain high accuracy at low bit\-width by minimizing layer\-wise reconstruction error\. However, weight\-only quantization suffers from an inherent limitation: during inference, full\-precision activations prevent the use of fully integer matrix multiplication operators for efficient inference acceleration\.

To overcome this bottleneck, recent research has shifted toward weight\-activation quantization, which quantizes both weights and activations to enable fully integer computation\. A key challenge in this setting is that activations in modern LLMs exhibit heavy\-tailed distributions with extreme outliers, leading to substantial accuracy degradation under low\-bit quantization\. To address this issue, numerous methods have been proposed to mitigate the impact of activation outliers\. Existing approaches can be broadly categorized into post\-training quantization \(PTQ\) and quantization\-aware training \(QAT\)\.

Post\-training quantization\.PTQ applies quantization to pretrained models without further training\. One line of work explores mixed\-precision strategies\[[9](https://arxiv.org/html/2605.20295#bib.bib9),[47](https://arxiv.org/html/2605.20295#bib.bib47)\], which assign higher bit\-widths to layers or tensors that are more sensitive to quantization error, particularly those affected by severe outliers\. Another line of research focuses on scaling\-based methods\[[40](https://arxiv.org/html/2605.20295#bib.bib40),[34](https://arxiv.org/html/2605.20295#bib.bib34),[38](https://arxiv.org/html/2605.20295#bib.bib38)\], which balance the magnitudes of weights and activations via per\-channel scaling and shifting to reduce quantization error\. However, such scaling strategies often increase the complexity of weight quantization and tend to degrade significantly under extremely low\-bit settings\. Inspired by SmoothQuant\[[40](https://arxiv.org/html/2605.20295#bib.bib40)\], AWQ\[[19](https://arxiv.org/html/2605.20295#bib.bib19)\]proposes an activation\-aware per\-channel scaling method to reduce weight quantization error\. More recently, rotation\-based approaches\[[2](https://arxiv.org/html/2605.20295#bib.bib2),[21](https://arxiv.org/html/2605.20295#bib.bib21)\]have demonstrated strong effectiveness: by applying orthogonal or Hadamard transformations, activation outliers can be redistributed across channels through matrix multiplication, thereby alleviating extreme values and improving quantization robustness\. Notably, QuaRot\[[2](https://arxiv.org/html/2605.20295#bib.bib2)\]is the first work to introduce Hadamard transforms for LLMs\.

Quantization\-aware training\.Quantization\-Aware Training \(QAT\) methods\[[16](https://arxiv.org/html/2605.20295#bib.bib16),[17](https://arxiv.org/html/2605.20295#bib.bib17),[12](https://arxiv.org/html/2605.20295#bib.bib12),[4](https://arxiv.org/html/2605.20295#bib.bib4),[24](https://arxiv.org/html/2605.20295#bib.bib24)\]simulate quantization effects during the training phase, allowing the model to find more optimal solutions compared to Post\-Training Quantization \(PTQ\)\. However, traditional QAT approaches are severely bottlenecked by prolonged training times, massive memory footprints, and a strict reliance on labeled data and complex hyperparameter tuning\. These formidable computational overheads render them largely impractical for modern Large Language Models \(LLMs\)\. To mitigate this, recent studies\[[20](https://arxiv.org/html/2605.20295#bib.bib20),[11](https://arxiv.org/html/2605.20295#bib.bib11),[7](https://arxiv.org/html/2605.20295#bib.bib7),[10](https://arxiv.org/html/2605.20295#bib.bib10),[42](https://arxiv.org/html/2605.20295#bib.bib42),[6](https://arxiv.org/html/2605.20295#bib.bib6)\]have explored efficient QAT variants tailored for LLMs\.

## Appendix BComparison with Other Initialization Methods

For low\-bit coarse\-grained activation quantization \(e\.g\., 4\-bit per\-tensor\), it is crucial to identify a stable method for initializing quantization parameters in order to provide a favorable starting point for optimization and to promote convergence\. Under such low\-bit and coarse\-grained settings, quantization performance is particularly vulnerable to outliers\. Accordingly, we evaluate initialization methods based on mean statistics as well as various common clipping strategies\. The resulting model perplexity \(PPL\) on the Wikitext2 dataset is reported in[Table˜4](https://arxiv.org/html/2605.20295#A2.T4)\.

We further analyze different clipping algorithms with varying thresholds and observe that fixed thresholds are not universally applicable to all activation distributions\. They may lead to either overly aggressive or insufficient clipping, both of which harm model performance\. In contrast, the Mean\-based initialization method, which dynamically adapts the clipping range to the distribution, achieves the best initialization effect\. This finding also provides empirical support for the motivation behind introducing learnable clipping thresholds in the prior work\[[36](https://arxiv.org/html/2605.20295#bib.bib36)\]\.

Moreover, we observe that under low\-bit coarse\-grained activation quantization, omitting rotation substantially increases the quantization difficulty\. In this case, none of the initialization methods yield satisfactory performance\. In contrast, when rotation is applied, the Mean\-based initialization achieves the best results\. Overall, the Mean\-based initialization method is justified and demonstrates strong effectiveness\.

Table 3:Matrix multiplication speedup under different quantization configurations\.A\. Gran\.A\. Prec\.W\. Gran\.W\. Prec\.Speedup–FP–FP1\.0×1\.0\\timesper\-tensor16\-bitper\-block4\-bit1\.67×1\.67\\timesper\-tensor8\-bitper\-channel4\-bit2\.0×2\.0\\timesper\-tensor8\-bitper\-tensor4\-bit2\.0×2\.0\\timesTable 4:PPL on Wikitext2 for SmolLM2\-1\.7B\-Instruct under 4\-bit per\-tensor activation quantization, comparing Mean\-based initialization with various common clipping thresholds, with and without rotation\.Activation\-ConfigMeanClip\(99%\)Clip\(99\.9%\)Clip\(99\.99%\)4\-bit \+ with rotation30\.272838\.81145\.63183\.154\-bit \+ without rotation6550995\.018466806\.0502306\.3847484904\.0

## Appendix CQuantization Performance under different configurations

As shown in Table[5](https://arxiv.org/html/2605.20295#A3.T5), for Llama3\.2\-3B\-Instruct, we restrict quantization to the input activations of linear layers\. This setting facilitates a systematic analysis of different quantization configurations, such as the presence or absence of rotation and the activation bit\-width allowing a clearer evaluation of how various initialization strategies affect quantized model performance\. Model performance is measured by perplexity \(PPL\) on the C4 dataset\.

Table 5:Perplexity \(PPL\) on C4 for activation quantization under different configurations and initialization methods\.Activation\-ConfigMax\-MinMean8\-bit \+ without rotation33\.2412651\.844\-bit \+ without rotation276217\.2812833\.838\-bit \+ with rotation18\.5525\.374\-bit \+ with rotation548\.2743\.68

We further validate this conclusion on weight quantization by repeating the same experimental settings and evaluations\. The results, also measured by PPL on C4, exhibit the same pattern, and are summarized in Table[6](https://arxiv.org/html/2605.20295#A3.T6)\.

Table 6:Perplexity \(PPL\) on C4 for weight quantization under different configurations and initialization methods\.Weight\-ConfigMax\-MinMean8\-bit \+ without rotation17\.441646\.304\-bit \+ without rotation363402\.223791\.848\-bit \+ with rotation16\.7917\.674\-bit \+ with rotation5157\.1122\.38

## Appendix DTheoretical Explanation of Mean\-Based Initialization with Rotation

In this section, we provide a theoretical explanation and analysis of the quantization parameters initialization methodology introduced in[Section˜4\.2](https://arxiv.org/html/2605.20295#S4.SS2)\. Some conclusions are intuitive and do not require extensive derivation\. For instance, under 8\-bit quantization, the quantizer possesses sufficient representational capacity\. Therefore, whether or not a rotation matrix is introduced, the quantization parameters can be initialized using the Max–Min method\. In contrast, under 4\-bit quantization, the representational capacity is limited, and even with a rotation matrix, Max–Min initialization still leads to substantial quantization error\.

More importantly, we observe a key phenomenon: under both 4\-bit and 8\-bit quantization, the Mean\-based initialization method becomes stable and effective only after the introduction of rotation\. In the following, we provide a theoretical explanation for this observation\.

### D\.1Effect of Orthogonal Rotation on Activation Statistics

###### Assumption D\.1\.

Let the unrotated activation be flattened into a vectorx∈ℝNx\\in\\mathbb\{R\}^\{N\}with zero mean, i\.e\.,mean​\(x\)=0\\mathrm\{mean\}\(x\)=0\.

###### Definition D\.2\(Orthogonal Transformation of Activations\)\.

Letx∈ℝNx\\in\\mathbb\{R\}^\{N\}be a random vector satisfying𝔼​\[xi\]=0\\mathbb\{E\}\[x\_\{i\}\]=0for allii\. LetR∈ℝN×NR\\in\\mathbb\{R\}^\{N\\times N\}be an orthogonal matrix such thatR⊤​R=IR^\{\\top\}R=I, and definey=R​xy=Rx\.

###### Proposition D\.3\.

Under Assumption[D\.1](https://arxiv.org/html/2605.20295#A4.Thmtheorem1)and Definition[D\.2](https://arxiv.org/html/2605.20295#A4.Thmtheorem2), the quantitiesmean​\(y\)±3​std​\(y\)\\mathrm\{mean\}\(y\)\\pm 3\\,\\mathrm\{std\}\(y\)are equal tomean​\(x\)±3​std​\(x\)\\mathrm\{mean\}\(x\)\\pm 3\\,\\mathrm\{std\}\(x\)in expectation\.

###### Proof\.

We first analyze the mean ofyy\. By definition,

mean​\(y\)=1N​𝟏⊤​y=1N​𝟏⊤​R​x\.\\mathrm\{mean\}\(y\)=\\frac\{1\}\{N\}\\mathbf\{1\}^\{\\top\}y=\\frac\{1\}\{N\}\\mathbf\{1\}^\{\\top\}Rx\.Letv=R⊤​𝟏v=R^\{\\top\}\\mathbf\{1\}, so that𝟏⊤​R=v⊤\\mathbf\{1\}^\{\\top\}R=v^\{\\top\}\. Then

mean​\(y\)=1N​v⊤​x\.\\mathrm\{mean\}\(y\)=\\frac\{1\}\{N\}v^\{\\top\}x\.Assuming independence betweenviv\_\{i\}andxix\_\{i\}, together with𝔼​\[xi\]=0\\mathbb\{E\}\[x\_\{i\}\]=0, we obtain

𝔼​\[mean​\(y\)\]=1N​𝔼​\[vT​x\]=1N​∑i=1N𝔼​\[vi​xi\]=1N​∑i=1N𝔼​\[vi\]​𝔼​\[xi\]=0\\mathbb\{E\}\[\\mathrm\{mean\}\(y\)\]=\\frac\{1\}\{N\}\\mathbb\{E\}\[v^\{T\}x\]=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{E\}\[v\_\{i\}x\_\{i\}\]=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{E\}\[v\_\{i\}\]\\mathbb\{E\}\[x\_\{i\}\]=0
Next, we consider the standard deviation\. By orthogonality ofRR, we have

‖y‖22=‖R​x‖22=x⊤​R⊤​R​x=‖x‖22\.\\\|y\\\|\_\{2\}^\{2\}=\\\|Rx\\\|\_\{2\}^\{2\}=x^\{\\top\}R^\{\\top\}Rx=\\\|x\\\|\_\{2\}^\{2\}\.The standard deviation ofxxis given by

std​\(x\)=1N​‖x‖22−mean​\(x\)2\.\\mathrm\{std\}\(x\)=\\sqrt\{\\frac\{1\}\{N\}\\\|x\\\|\_\{2\}^\{2\}\-\\mathrm\{mean\}\(x\)^\{2\}\}\.Since𝔼​\[mean​\(x\)\]=0\\mathbb\{E\}\[\\mathrm\{mean\}\(x\)\]=0, this yields

𝔼​\[std​\(x\)\]=1N​‖x‖22=𝔼​\[std​\(y\)\]\.\\mathbb\{E\}\[\\mathrm\{std\}\(x\)\]=\\sqrt\{\\frac\{1\}\{N\}\\\|x\\\|\_\{2\}^\{2\}\}=\\mathbb\{E\}\[\\mathrm\{std\}\(y\)\]\.
Under the commonly adopted3​σ3\\sigmaapproximation of the dynamic range,

max​\(z\)≈mean​\(z\)\+3​std​\(z\),min​\(z\)≈mean​\(z\)−3​std​\(z\),\\mathrm\{max\}\(z\)\\approx\\mathrm\{mean\}\(z\)\+3\\,\\mathrm\{std\}\(z\),\\quad\\mathrm\{min\}\(z\)\\approx\\mathrm\{mean\}\(z\)\-3\\,\\mathrm\{std\}\(z\),the equality of mean and standard deviation implies that the effective quantization range is preserved in expectation\. ∎

![Refer to caption](https://arxiv.org/html/2605.20295v1/x7.png)\(a\)Unrotated activation distribution forWqW\_\{q\}of the10t​h10^\{th\}transformer layer\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x8.png)\(b\)Unrotated activation distribution forWoW\_\{o\}of the10t​h10^\{th\}transformer layer\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x9.png)\(c\)Unrotated activation distribution forWu​pW\_\{up\}of the10t​h10^\{th\}transformer layer\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x10.png)\(d\)Unrotated activation distribution forWd​o​w​nW\_\{down\}of the10t​h10^\{th\}transformer layer\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x11.png)\(e\)Rotated activation distribution forWqW\_\{q\}of the10t​h10^\{th\}transformer layer\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x12.png)\(f\)Rotated activation distribution forWoW\_\{o\}of the10t​h10^\{th\}transformer layer\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x13.png)\(g\)Rotated activation distribution forWu​pW\_\{up\}of the10t​h10^\{th\}transformer layer\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x14.png)\(h\)Rotated activation distribution forWd​o​w​nW\_\{down\}of the10t​h10^\{th\}transformer layer\.

Figure 5:Activation distributions ofWqW\_\{q\},WoW\_\{o\},Wu​pW\_\{up\}, andWd​o​w​nW\_\{down\}in the10t​h10^\{th\}transformer layer of SmolLM2\-1\.7B\-Instruct\.[Figure˜5\(a\)](https://arxiv.org/html/2605.20295#A4.F5.sf1)\-[Figure˜5\(d\)](https://arxiv.org/html/2605.20295#A4.F5.sf4)show the unrotated distributions, while[Figure˜5\(e\)](https://arxiv.org/html/2605.20295#A4.F5.sf5)\-[Figure˜5\(h\)](https://arxiv.org/html/2605.20295#A4.F5.sf8)show the rotated distributions\. Blue bars represent the activation distributions\. The labelsminandmaxindicate the boundaries of the quantization range \(calculated based on the Mean\-based initialization method\), and the box in the top\-right corner displays themin/maxvalues along with the percentage of activations that fall within this range\.
### D\.2Advantages of Mean\-Based Initialization with Rotation

[˜D\.1](https://arxiv.org/html/2605.20295#A4.Thmtheorem1)requires the activation distribution to have a mean close to zero\. However, by visualizing the pre\-rotation activations of LLaMA\-3\.2\-3B\-Instruct, Qwen2\.5\-3B\-Instruct, and SmolLM2\-1\.7B\-Instruct, we observe that the output activations of the gate\_proj have a mean that significantly deviates from zero, especially in Qwen models\. Aside from this exception, all other activations satisfy the assumptions in[˜D\.1](https://arxiv.org/html/2605.20295#A4.Thmtheorem1)\. Therefore, in our subsequent analysis, we exclude gate\_proj activations from low\-bit quantization and retain them in BF16 precision\.

To provide a more intuitive illustration of[Proposition˜D\.3](https://arxiv.org/html/2605.20295#A4.Thmtheorem3), we visualize the activation distributions ofWqW\_\{q\},WoW\_\{o\},Wu​pW\_\{up\}, andWd​o​w​nW\_\{down\}in the10t​h10^\{th\}transformer layer of SmolLM2\-1\.7B\-Instruct, as shown in[Figure˜5](https://arxiv.org/html/2605.20295#A4.F5)\. From the visualization, we summarize the advantages of combining Mean\-based initialization with rotation as follows\.

For activations within the quantization coverage range, the numerical values are relatively small, allowing the dequantized values to closely approximate the original distribution regardless of whether a rotation is applied\. Consequently, the quantization error for this subset is typically concentrated near zero, indicating high quantization accuracy\. In contrast, activations that fall outside the coverage range \(i\.e\., outliers\) incur substantial quantization errors\. Without a rotation matrix, these outliers have large magnitudes, and the information they carry is severely truncated during quantization, leading to noticeable performance degradation\. Introducing a rotation matrix smooths the distribution, mapping more activations into the coverage range and reducing the magnitude of outliers, thereby mitigating their impact\. Hence, Mean\-based initialization becomes effective only after rotation; applying it directly to the unrotated activations would result in large quantization errors\.

![Refer to caption](https://arxiv.org/html/2605.20295v1/x15.png)\(a\)Rotated activation distribution forWqW\_\{q\}of the19t​h19^\{th\}transformer layer\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x16.png)\(b\)Rotated activation distribution forWoW\_\{o\}of the19t​h19^\{th\}transformer layer\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x17.png)\(c\)Rotated activation distribution forWu​pW\_\{up\}of the19t​h19^\{th\}transformer layer\.
![Refer to caption](https://arxiv.org/html/2605.20295v1/x18.png)\(d\)Unrotated activation distribution forWd​o​w​nW\_\{down\}of the19t​h19^\{th\}transformer layer\.

Figure 6:Activation distributions ofWqW\_\{q\},WoW\_\{o\},Wu​pW\_\{up\}, andWd​o​w​nW\_\{down\}in the19t​h19^\{th\}transformer layer of SmolLM2\-1\.7B\-Instruct\.[Figure˜6\(a\)](https://arxiv.org/html/2605.20295#A4.F6.sf1)\-[Figure˜6\(c\)](https://arxiv.org/html/2605.20295#A4.F6.sf3)show the rotated distributions ofWqW\_\{q\},WoW\_\{o\}, andWu​pW\_\{up\}, while[Figure˜6\(d\)](https://arxiv.org/html/2605.20295#A4.F6.sf4)shows the unrotated distributions ofWd​o​w​nW\_\{down\}\.

## Appendix ETheoretical Analysis on the Failure of Gradient\-Based Optimization for Unrotated Tensors

In[Section˜4\.3](https://arxiv.org/html/2605.20295#S4.SS3), we point out that directly applying gradient\-based optimization to the quantization parameters of unrotated tensors introduces significant instability, often resulting in worse model accuracy than using unoptimized parameters\. In this section, we further analyze the fundamental nature of quantization error optimization and provide a theoretical analysis for the optimization failure caused by the conflict between clipping error and rounding error under heavy\-tailed distributions\.

The optimization of quantization parameters essentially aims to find a theoretical equilibrium point that minimizes the total quantization error, denoted as

Etotal​\(s\)=Erounding​\(s\)\+Eclipping​\(s\)\.\\displaystyle E\_\{\\text\{total\}\}\(s\)=E\_\{\\text\{rounding\}\}\(s\)\+E\_\{\\text\{clipping\}\}\(s\)\.\(5\)For a random input tensor, the total quantization error can be rigorously decomposed into two components: the rounding error, produced when values are mapped to the nearest quantization grid, and the clipping error, caused by truncating values that exceed the representable integer range\. The quantization scalessdirectly governs the trade\-off between these two types of errors\.

For rotated tensors that approximately follow a Gaussian distribution, most values are concentrated around the mean\. Under such distributions, the trade\-off curve betweenEroundingE\_\{\\text\{rounding\}\}andEclippingE\_\{\\text\{clipping\}\}varies smoothly assschanges\. Consequently, the optimizer can relatively easily locate a global optimum\. As a result, the quantization parameters can be optimized in a stable manner in this setting\.

In contrast, unrotated tensors typically exhibit heavy\-tailed distributions, where the majority of values are densely concentrated near zero while a small number of outliers are distributed far from the center\. During optimization, reducingEclippingE\_\{\\text\{clipping\}\}for these extreme outliers forces the optimizer to enlarge the quantization scaless\. However, increasingsscompresses the densely populated central region into only a few quantization bins near zero, leading to a dramatic increase inEroundingE\_\{\\text\{rounding\}\}\. Conversely, if the optimizer reducesssto preserve the quantization precision of the dominant central values and minimizeEroundingE\_\{\\text\{rounding\}\}, the large\-magnitude outliers become severely clipped, significantly increasingEclippingE\_\{\\text\{clipping\}\}\.

Therefore, under the extremely limited representation capacity of low\-bit quantization, the severe polarization inherent in heavy\-tailed distributions causes the loss landscape ofEtotal​\(s\)E\_\{\\text\{total\}\}\(s\)to contain numerous local minima\. Consequently, gradient\-based optimization through backpropagation becomes highly unstable\. This observation further motivates the necessity of adopting a distribution\-aware selective optimization strategy for tensors with different statistical characteristics, thereby mitigating the adverse effects introduced by heavy\-tailed distributions during joint optimization\.

## Appendix FDown\_proj: Outlier\-Dominated Activation Distribution

For the SmolLM2\-1\.7B\-Instruct model, we visualized the distributions of input activations for q\_proj \(k\_proj and v\_proj\), o\_proj, up\_proj \(gate\_proj\), and down\_proj inQuant\.npu\. To avoid the inference latency issue discussed in[Section˜2\.2](https://arxiv.org/html/2605.20295#S2.SS2), we disabled all online rotations \(R4\) in our method\. As a result, the input activations of down\_proj remain in their original, unrotated form, whereas the input activations of the remaining linear layers are smoothed via rotation\.

As shown in[Figure˜6](https://arxiv.org/html/2605.20295#A4.F6), and consistent with prior work\[[41](https://arxiv.org/html/2605.20295#bib.bib41)\], the input activations of down\_proj layers exhibit particularly severe outliers: the numerical range is extensive, and a large portion of values concentrates near zero\. To preserve quantization accuracy for these near\-zero values, important information carried by the outliers may be lost\. This ultimately leads to a substantial drop in model accuracy\.

## Appendix GAdditional Experimental Results

Table 7:Evaluation on additional widely used models \(Qwen2\.5\-3B\-Instruct and SmolLM2\-1\.7B\-Instruct\): Perplexity\(PPL\) and Zero\-shot QA task accuracy results of 4\-bit weight or 8\-bit weight quantized models\.ModelMethodW\-A\-KVW\. Gran\.PPL\(C4\)↓\\downarrowPIQAWinograndeHellaSwagARC\-EARC\-CLAMBADAAvg↑\\uparrowQwen2\.5\-3B\-itFP32––13\.840\.78070\.69140\.73300\.55850\.42490\.58080\.6282executorch4\-16\-8per\-blcok15\.480\.76060\.67880\.71720\.55220\.43170\.55790\.6164executorch8\-8\-8per\-channel14\.680\.74970\.67250\.72030\.56190\.41810\.57070\.6155QuaRot8\-8\-8per\-channel14\.080\.78240\.68980\.73200\.55770\.41640\.56670\.6242SpinQuant8\-8\-8per\-channel14\.070\.77370\.68590\.72940\.54210\.41720\.58240\.6218Quant\.npu8\-8\-8per\-channel13\.820\.77640\.68670\.72780\.55010\.43090\.57500\.6245executorch4\-8\-8per\-channel28901\.540\.50330\.49010\.25180\.24830\.22530\.00\.2865QuaRot4\-8\-8per\-channel19\.960\.72200\.60460\.64500\.48150\.36260\.38270\.5331SpinQuant4\-8\-8per\-channel20\.610\.73340\.61090\.64140\.58080\.40780\.40170\.5627Quant\.npu4\-8\-8per\-channel16\.020\.76330\.66850\.69200\.51010\.40610\.50530\.5906executorch4\-4\-8per\-channel415330\.380\.50000\.51140\.25370\.24240\.23460\.00\.2904QuaRot4\-4\-8per\-channel97\.010\.59190\.50430\.38270\.35650\.25170\.05070\.3563SpinQuant4\-4\-8per\-channel101\.850\.58600\.50200\.36830\.34180\.26280\.07530\.3560Quant\.npu4\-4\-8per\-channel17\.960\.74370\.62270\.66190\.51050\.38740\.44400\.5617SmolLM2\-1\.7B\-itFP32––13\.550\.76220\.68350\.71270\.54170\.41810\.57030\.6148executorch4\-16\-8per\-block15\.350\.75950\.64880\.69330\.53620\.40440\.51780\.5933executorch8\-8\-8per\-channel25\.750\.70620\.61880\.62560\.51940\.39330\.38890\.5420QuaRot8\-8\-8per\-channel13\.930\.76330\.66460\.70470\.54710\.40360\.55250\.6060SpinQuant8\-8\-8per\-channel13\.960\.76280\.66770\.70200\.55510\.41720\.56160\.6111Quant\.npu8\-8\-8per\-channel13\.940\.76010\.65510\.68590\.53960\.39420\.54840\.5972executorch4\-8\-8per\-channel13121503\.00\.62020\.50590\.44700\.41710\.30120\.04440\.3893QuaRot4\-8\-8per\-channel120\.060\.57940\.50040\.34650\.32830\.24740\.11160\.3523SpinQuant4\-8\-8per\-channel67\.050\.63170\.52570\.41050\.36620\.29350\.09740\.3875Quant\.npu4\-8\-8per\-channel18\.610\.73120\.61560\.60960\.48650\.35840\.45620\.5436executorch4\-4\-8per\-channel53274784\.00\.50160\.50040\.25210\.25000\.25680\.00\.2935QuaRot4\-4\-8per\-channel8553\.560\.51630\.50200\.25680\.25630\.25940\.00060\.2986SpinQuant4\-4\-8per\-channel4283\.870\.49890\.48930\.24990\.25290\.22610\.00040\.2862Quant\.npu4\-4\-8per\-channel23\.560\.69480\.57700\.53420\.45410\.32510\.34970\.4898

### G\.1Results on Additional Widely Used Models

To further demonstrate the generalizability ofQuant\.npu, we extend our analysis to other widely deployed models, including Qwen2\.5\-3B\-Instruct\[[44](https://arxiv.org/html/2605.20295#bib.bib44)\]and SmolLM2\-1\.7B\-Instruct\[[1](https://arxiv.org/html/2605.20295#bib.bib1)\]\.[Table˜7](https://arxiv.org/html/2605.20295#A7.T7)reports the perplexity \(PPL\) on C4 and the accuracy on six zero\-shot commonsense reasoning benchmarks under W8A8, W4A8, and W4A4 settings\. Consistent with the results in[Section˜5\.1](https://arxiv.org/html/2605.20295#S5.SS1),Quant\.npuachieves zero\-shot accuracy comparable to the FP32 baseline under W8A8 quantization on Qwen2\.5\-3B\-Instruct\. The performance gains are particularly significant under the W4A8 setting, whereQuant\.npuachieves a maximum accuracy improvement of up to 30\.41% compared to baseline methods\. Moreover,Quant\.npu\-W4A4 maintains a relatively high accuracy, even in scenarios where the baseline almost entirely fails\. For Qwen2\.5\-3B\-Instruct,Quant\.npuexhibits only about a 6% drop in accuracy compared to the FP32 baseline\. Furthermore,Quant\.npudemonstrates broad applicability by yielding comparable performance gains on SmolLM2\-1\.7B\-Instruct\.

Table 8:Evaluation on recent architectures \(Qwen3\-1\.7B\): Perplexity\(PPL\) and Zero\-shot QA task accuracy results of 4\-bit weight or 8\-bit weight quantized models\.ModelMethodW\-A\-KVW\. Gran\.PPL\(C4\)↓\\downarrowPIQAWinograndeHellaSwagARC\-EARC\-CLAMBADAAvg↑\\uparrowQwen3\-1\.7BFP32––22\.640\.71600\.60930\.59320\.54380\.39930\.47310\.5558executorch4\-16\-8per\-block31\.520\.66540\.55960\.53840\.48440\.34730\.31960\.4858executorch8\-8\-8per\-channel29\.990\.65890\.55880\.52940\.49070\.34470\.34680\.4882QuaRot8\-8\-8per\-channel28\.910\.66590\.57060\.54560\.53160\.37030\.38330\.5112SpinQuant8\-8\-8per\-channel28\.640\.66590\.58800\.54680\.52190\.36010\.38040\.5105Quant\.npu8\-8\-8per\-channel22\.740\.68720\.60140\.57100\.53540\.39590\.44300\.5390executorch4\-8\-8per\-channel72\.610\.60940\.50590\.41790\.40610\.27050\.12300\.3888QuaRot4\-8\-8per\-channel429\.090\.57830\.50590\.35460\.34090\.26020\.12500\.3608SpinQuant4\-8\-8per\-channel548\.100\.56370\.50910\.33640\.31730\.24570\.12090\.3489Quant\.npu4\-8\-8per\-channel31\.680\.67790\.56040\.50540\.46510\.33110\.27290\.4688executorch4\-4\-8per\-channel491615\.220\.49240\.50910\.25060\.25510\.24400\.00\.2919QuaRot4\-4\-8per\-channel6898\.980\.49130\.49250\.27170\.26810\.25340\.00990\.2978SpinQuant4\-4\-8per\-channel5018\.610\.51690\.53280\.27600\.27270\.25340\.01510\.3112Quant\.npu4\-4\-8per\-channel36\.250\.62790\.52410\.46770\.46930\.32680\.21990\.4393

### G\.2Results on More Recent Architectures

To further evaluate the generality ofQuant\.npuon recent model architectures, we additionally conduct optimization and evaluation on Qwen3\-1\.7B\[[43](https://arxiv.org/html/2605.20295#bib.bib43)\]and compare our method against relevant baselines\. The detailed results are provided in[Table˜8](https://arxiv.org/html/2605.20295#A7.T8)\.

As shown in[Table˜8](https://arxiv.org/html/2605.20295#A7.T8),Quant\.npuconsistently achieves superior accuracy preservation across different quantization settings\. Under the highly constrained W4A4KV8 per\-channel setting,Quant\.npuachieves an average accuracy of 0\.4393 with a PPL\(C4\) of 36\.25, significantly outperforming the baseline, SpinQuant, which suffers severe degradation \(Avg: 0\.3112, PPL: 5018\.61\)\. Even under the W8A8KV8 setting,Quant\.npuachieves an average accuracy of 0\.5390 and a PPL\(C4\) of 22\.74, closely matching the FP32 model \(Avg: 0\.5558, PPL: 22\.64\)\. These results demonstrate thatQuant\.npucan be effectively generalized to newer architectures without requiring architecture\-specific modifications\.

In addition,Quant\.npuis practical and easy to deploy\. For Qwen3\-1\.7B, the complete quantization pipeline requires only approximately 2\.5 hours on NVIDIA A40 GPUs under the experimental setup described in[Section˜5](https://arxiv.org/html/2605.20295#S5)\. AdaptingQuant\.nputo a new model requires no complex model\-specific engineering, demonstrating the practicality and portability of our method\.

Table 9:Evaluation on larger LLMs \(LLaMA3\-8B\): Perplexity\(PPL\) and Winogrande accuracy results of 4\-bit weight quantized models\.ModelMethodW\-A\-KVW\. Gran\.PPL\(C4\)↓\\downarrowWinogrande↑\\uparrowLLaMA3\-8BFP32––9\.070\.7285executorch4\-16\-8per\-block10\.350\.7182executorch4\-8\-8per\-channel17\.930\.6756QuaRot4\-8\-8per\-channel19\.150\.6819SpinQuant4\-8\-8per\-channel22\.370\.6448Quant\.npu4\-8\-8per\-channel11\.880\.7190

### G\.3Results on Larger Models

Our primary evaluation focuses on 1B–3B models, which align with the target deployment scenario of mobile devices with limited memory budgets \(typically around 4GB available memory\)\. Accordingly, we evaluatesQuant\.npuon several representative mobile\-scale models, including Llama\-3\.2\-3B\-Instruct, Qwen2\.5\-3B\-Instruct, and SmolLM2\-1\.7B\-Instruct\.

To further validate the scalability ofQuant\.npu, we additionally conduct experiments on the substantially larger Llama3\-8B model\[[15](https://arxiv.org/html/2605.20295#bib.bib15)\]\. The detailed results are summarized in[Table˜9](https://arxiv.org/html/2605.20295#A7.T9)\.

As shown in[Table˜9](https://arxiv.org/html/2605.20295#A7.T9),Quant\.npuconsistently maintains strong quantization performance at larger scales\. Under the challenging W4A8KV8 per\-channel configuration, baseline methods suffer significant degradation, with ExecuTorch, QuaRot, and SpinQuant achieving PPL values of 17\.93, 19\.15, and 22\.37, respectively\. In contrast,Quant\.npusubstantially reduces the PPL to 11\.88 while achieving an accuracy of 0\.7190 on Winogrande\. Notably, this performance not only significantly surpasses all other W4A8 per\-channel baselines, but also approaches the higher\-precision ExecuTorch\-W4A16 per\-block setting \(PPL: 10\.35\) and the FP32 model \(PPL: 9\.07\)\.

These results demonstrate thatQuant\.npuremains highly robust across substantially different model scales\. More importantly, as model capacity increases,Quant\.npuexhibits the consistent capability in preserving model accuracy under aggressive NPU\-friendly quantization constraints\.

Table 10:Comparison with MobileQuant: Perplexity \(PPL\) and HellaSwag accuracy results of 4\-bit weight quantization \(w4a8, with per\-channel weight quantization\) on the TinyLlama\-1\.1B\-Chat\-v1\.0 model\.MethodPPL\(C4\)↓\\downarrowHellaSwag↑\\uparrowFP329\.750\.5087MoblieQuant11\.950\.4787Quant\.npu11\.430\.4893

### G\.4Additional Comparison with MobileQuant

Our original evaluation primarily focused on rotation\-based quantization methods, including QuaRot and SpinQuant, since these approaches consistently demonstrate stronger accuracy preservation compared with earlier methods such as SmoothQuant and OmniQuant, upon which MobileQuant\[[37](https://arxiv.org/html/2605.20295#bib.bib37)\]is built\. To provide a more comprehensive evaluation, we additionally conduct a direct comparison betweenQuant\.npuand MobileQuant under the exact same static NPU\-friendly quantization constraints\. We conduct this evaluation on the TinyLlama\-1\.1B\-Chat\-v1\.0 model\. The detailed performance metrics are summarized in Table[10](https://arxiv.org/html/2605.20295#A7.T10)\.

As shown in[Table˜10](https://arxiv.org/html/2605.20295#A7.T10),Quant\.npuconsistently outperforms MobileQuant across both perplexity and downstream task accuracy\. Specifically, MobileQuant achieves a PPL of 11\.95 and an accuracy of 0\.4787 on HellaSwag, whereasQuant\.npuimproves the PPL to 11\.43 and achieves a higher accuracy of 0\.4893 on HellaSwag, substantially narrowing the gap toward the FP32 baseline \(PPL: 9\.75, HellaSwag: 0\.5087\)\. This further highlights the performance advantage ofQuant\.npuunder fully static NPU\-friendly quantization constraints\.

Table 11:Instruction\-following evaluation on Llama\-3\.2\-3B\-Instruct using AlpacaEval 2\.0\. Win rates are computed against the FP16 model to isolate the impact of quantization\.Modellength\_controlled\_winrate↑\\uparrowwin\_rate↑\\uparrowavg\_lengthExecutorch\-w4a89\.099\.592301Quant\.npu\-w4a824\.7622\.432167Quant\.npu\-w8a848\.8253\.592406

### G\.5Instruction\-Following Capability Evaluation

To further evaluate whetherQuant\.npupreserves instruction\-following capability after quantization, we conduct additional experiments on Llama\-3\.2\-3B\-Instruct using AlpacaEval 2\.0\.

Since small models typically exhibit relatively low win rates against the default GPT\-4\-Turbo reference model in AlpacaEval 2\.0, direct comparison against GPT\-4\-Turbo can obscure the effect of quantization\. For a clearer assessment, we instead use the FP16 Llama\-3\.2\-3B\-Instruct model as the reference model and compute the win rates of quantized variants against it\.

As shown in Table[11](https://arxiv.org/html/2605.20295#A7.T11),Quant\.npu\-W8A8 achieves a length\-controlled win rate of 48\.82%, indicating nearly lossless generation quality compared with the FP16 model\. In contrast, the more aggressive W4A8 setting still achieves a 24\.76% length\-controlled win rate, demonstrating thatQuant\.npumaintains reasonable conversational quality even under low\-bit constraints\.

Notably,Quant\.npusignificantly outperforms the ExecuTorch\-W4A8 baseline, which achieves only a 9\.09% length\-controlled win rate\. This substantial gap highlights the effectiveness of our method in preserving instruction\-following capability under aggressive quantization\.

Table 12:Detailed performance comparison between Quant\.npu and ExecuTorch: Prefill and decode speed, memory footprint, and energy consumption of different LLMs across various datasets under different quantization settings\.\(a\)Executorch: 4\-bit per\-block weight quantizationModelDatasetPrefill LenDecode LenPrefill Speed \(TPS\)Decode Speed \(TPS\)Peak Mem \(MB\)Total Energy \(J\)Per\-run Energy \(J\)Per\-run Drain \(μ\\muAh\)Qwen2\.5\-3B\-itHellaSwag6442577\.6129\.232217\.2137\.797227\.55941800Persona\-Chat53646775\.1226\.182219\.7307\.440061\.48804000DroidTask7413808\.1021\.062216\.3184\.736\.92400Llama3\.2\-3B\-itHellaSwag6442674\.2928\.042435\.5339\.292867\.85864400Persona\-Chat53646860\.9927\.812433\.1415\.044083\.00885400DroidTask7413827\.6022\.682433\.6538\.020067\.25254375SmolLM2\-1\.7B\-itHellaSwag6442956\.3548\.861119\.4185\.846423\.23081500Persona\-Chat536461329\.9144\.401119\.1402\.948050\.36853250DroidTask74131343\.9837\.001118\.647\.25009\.4500600Qwen3\-1\.7BHellaSwag6442826\.2043\.971564\.6232\.200046\.44003000Persona\-Chat536461213\.7542\.221567\.9171\.230434\.24612200DroidTask74131177\.2234\.271568\.479\.236015\.84721000

\(b\)Quant\.npu: 4\-bit per\-channel weight quantizationModelDatasetPrefill LenDecode LenPrefill Speed \(TPS\)Decode Speed \(TPS\)Peak Mem \(MB\)Total Energy \(J\)Per\-run Energy \(J\)Per\-run Drain \(μ\\muAh\)Qwen2\.5\-3B\-itHellaSwag6442693\.1329\.231876\.60136\.151027\.23021780Persona\-Chat53646930\.1426\.181877\.85293\.138258\.62763815DroidTask7413969\.7221\.061876\.15158\.077231\.58152055Llama3\.2\-3B\-itHellaSwag6442809\.1528\.041985\.75335\.922867\.18464355Persona\-Chat536461033\.1927\.811984\.55399\.112079\.82245195DroidTask7413993\.1222\.681984\.80461\.268657\.65863750SmolLM2\-1\.7B\-itHellaSwag64421147\.6248\.86994\.70183\.694222\.96181485Persona\-Chat536461595\.8944\.40994\.55386\.721448\.34023120DroidTask74131612\.7837\.00994\.3040\.50618\.1012515Qwen3\-1\.7BHellaSwag6442991\.4443\.971217\.30229\.278345\.85572960Persona\-Chat536461456\.5042\.221218\.95164\.218532\.84372110DroidTask74131412\.6634\.271219\.2068\.108413\.6217860

### G\.6Deployment Performance on SM8750 NPU

While the primary evaluation in[Section˜5\.2](https://arxiv.org/html/2605.20295#S5.SS2)focuses on the Qualcomm SM8650 NPU, this section extends our hardware benchmarking to the newer Qualcomm SM8750 NPU\. To provide a comprehensive profile ofQuant\.npu’s hardware efficiency, we report the inference speed, memory footprint, and energy consumption, and compare these results with the performance of the slower ExecuTorch\-W4A16 on the same hardware\.

Experimental Setup\.We evaluate four representative LLMs: Qwen2\.5\-3B\-Instruct, Llama3\.2\-3B\-Instruct, SmolLM2\-1\.7B\-Instruct, and Qwen3\-1\.7B\. We conduct evaluations on three datasets with distinct workload characteristics: HellaSwag, Persona\-Chat, and DroidTask\. Detailed results are summarized in[Table˜12](https://arxiv.org/html/2605.20295#A7.T12)\.

Inference Speed\.The acceleration improvements observed on the SM8750 are highly consistent with those on the SM8650\.Quant\.npucontinues to demonstrate substantial speedup benefits, particularly during the prefill stage\. For example, on the DroidTask dataset, Qwen2\.5\-3B\-Instruct achieves a prefill throughput of 969\.72 tokens/s, while SmolLM2\-1\.7B\-Instruct reaches 1612\.78 tokens/s\.

Memory Footprint\.Quant\.npuconsistently achieves lower memory consumption compared with baseline methods such as ExecuTorch\. Although the peak memory footprint of on\-device LLM inference is primarily dominated by model weights,Quant\.npufurther reduces activation memory through lower\-bit activation quantization\. Specifically,Quant\.npuadopts W4A8 quantization, whereas the baseline uses W4A16 quantization\. The lower activation precision therefore leads to a reduced overall memory footprint\. As shown in[Table˜12](https://arxiv.org/html/2605.20295#A7.T12), the peak memory usage of 3B models remains around 1876–1985 MB, while 1\.7B models require only around 994–1219 MB, demonstrating the efficiency of our deployment scheme\.

Energy Consumption\.Compared with ExecuTorch\-W4A16,Quant\.npualso achieves lower energy consumption during inference\. The energy reduction mainly comes from two factors\. First,Quant\.npumore efficiently utilizes the computational capability of the underlying NPU hardware\. Second, the reduced inference latency, especially during the prefill stage, shortens the duration of high\-power execution states\. For example, Qwen2\.5\-3B\-Instruct consumes only 27\.23 J per run on the HellaSwag dataset, confirming the exceptional suitability ofQuant\.npufor power\-constrained edge deployment\.

## Appendix HLimitations and Broader Impacts

LimitationsWhileQuant\.npusignificantly advances fully static quantization for mobile NPUs, it has two primary limitations\. First, although matrix multiplications, which dominate inference latency, are already implemented in low\-bit precision, the framework still retains 16\-bit precision for a minor subset of operations \(e\.g\., SiLU activations\) to strictly preserve model perfomance\. This introduces latency bottlenecks compared to a fully low\-bit execution pipeline\. Further reducing reliance on 16\-bit operations without incurring non\-negligible accuracy degradation remains a critical direction for future research\. Second,Quant\.npucurrently does not employ a dedicated strategy for curating the calibration dataset\. Since our parameter optimization process is highly sensitive to activation distributions, the quality of the calibration set plays a critical role\. Designing principled methods to select more representative calibration data could further improve the performance of quantized models\.

Broader ImpactsThe development ofQuant\.npuaddresses a key challenge in deploying large language models on mobile and edge devices, where computational resources and energy budgets are limited\. This framework has practical implications for a wide range of real\-world applications, including privacy\-preserving on\-device AI, mobile natural language understanding and generation, and low\-latency edge computing\. Through its enhanced efficiency and accuracy,Quant\.npuopens new possibilities for deploying powerful LLMs in resource\-constrained environments, fostering the growth of AI in mobile and edge technologies\.