PRQuant: 置换残差量化用于低开销推理

# PRQuant: Permutation Residual Quantization for Low-Overhead Inference
Source: [https://arxiv.org/html/2609.22106](https://arxiv.org/html/2609.22106)
Anqi WangJiaying ZhaoHuiwen YangZhenyu MingRongqian WangYiwu YaoKun TianXin YaoGong ZhangFan YangZhongyi Huang\[2mm\]

###### Abstract

Accuracy of Low\-bit quantization of linear layers is often dominated by a small number of outliers\. Although existing methods, such as smoothing, rotation, or residual\-based approaches, may mitigate this problem, they often introduce new accuracy bottlenecks to weights\. Besides, most of these techniques are implemented as online approaches, which can result in heavy execution overheads\. To address the afore\-mentioned issues, We proposePRQuant\(Permutation Residual Quantization\), a training\-free and low\-overhead framework that combines channel reorganization with static weight\-side residual compensation\. After AWQ\-style scaling,PRQuantidentifies the input channels that contribute most to weight quantization error, permutes them into contiguous tail blocks, and constructs their residual weight sub\-tensors offline\. During inference, this contiguous structure enables the activation side to use tail blocks seamlessly without the expensive online gathering operation, and turns scattered residual compensation into a regular tail\-augmented GEMM, substantially reducing latency\. Experiments demonstrate thatPRQuanteffectively reduces down\-projection reconstruction error\. Ablation studies confirm that smoothing and residual compensation are the primary drivers of numerical improvement, while permutation provides a consistent marginal numerical benefit and, more importantly, enables a hardware\-friendly contiguous layout that eliminates dynamic gathering overhead\. Overall,PRQuantoutperforms default MXFP4 and the evaluated PTQ baselines in average accuracy across five downstream benchmarks, improving over MXFP4 by 1\.24 and 0\.55 on Qwen3\-4B\-Instruct\-2507 and Qwen3\-30B\-A3B\-Instruct\-2507, respectively\.

††footnotetext:1Huawei Technologies Co\., Ltd\.2Tsinghua University\.
∗Equal contribution\.†Corresponding authors\.## 1 Introduction

Large language models \(LLMs\), including mixture\-of\-experts \(MoE\) architectures, have achieved strong performance in language understanding, code generation, and reasoning\. However, their deployment is constrained by memory capacity, bandwidth, and by the cost of large matrix multiplications\([3](https://arxiv.org/html/2609.22106#bib.bib1);[9](https://arxiv.org/html/2609.22106#bib.bib2);[5](https://arxiv.org/html/2609.22106#bib.bib3)\)\. Low\-bit quantization is therefore a central approach to reducing inference cost and improving throughput\.

Recent hardware advances have increased interest in fine\-grained block\-scaled FP4 formats such as MXFP4 and NVFP4\([20](https://arxiv.org/html/2609.22106#bib.bib4);[18](https://arxiv.org/html/2609.22106#bib.bib5)\)\. These formats share a scale within each small block, which confines an outlier’s influence to its local block\. Within that block, however, a single large value can inflate the shared scale and coarsen the quantization of all other values\. This unique block\-wise duality implies that the properties of these formats need to be carefully considered in the design of quantization algorithms\.

Existing post\-training quantization \(PTQ\) strategies address outliers from different perspectives, but most were developed for integer quantization and can behave differently under block\-wise shared scaling\. Rotation\-based methods use orthogonal transformations to redistribute activation outliers\([1](https://arxiv.org/html/2609.22106#bib.bib6)\)\. Under MXFP4, global rotations can spread outlier energy across multiple scaling blocks and increase the quantization error of otherwise regular blocks\([25](https://arxiv.org/html/2609.22106#bib.bib16);[22](https://arxiv.org/html/2609.22106#bib.bib14);[11](https://arxiv.org/html/2609.22106#bib.bib17)\)\. BRQ and BATQuant align their transformations with MXFP4 blocks, but their activation\-side transformations still run online during inference\([25](https://arxiv.org/html/2609.22106#bib.bib16);[11](https://arxiv.org/html/2609.22106#bib.bib17)\)\. SmoothQuant migrates quantization difficulty from activations to weights through a mathematically equivalent per\-channel scaling transformation\([26](https://arxiv.org/html/2609.22106#bib.bib9)\)\. However, with weights also quantized to 4 bits, weight quantization can itself become the new bottleneck\. Residual compensation can partially restore the accuracy lost by 4\-bit quantization, but its fundamental cost lies in the additional computation and memory\-access overhead\. For example, ARCQuant extends the activation matrix with quantized residual channels to preserve a unified low\-precision GEMM path, but it depends on input\-dependent residual channels\([17](https://arxiv.org/html/2609.22106#bib.bib18)\)\. SVDQuant absorbs outliers using a low\-rank branch, but this additional branch requires specialized kernel fusion to avoid extra data movement\([12](https://arxiv.org/html/2609.22106#bib.bib19)\)\.

These limitations motivate a low\-overhead W4A4 method that preserves a regular low\-precision GEMM path\. We proposePRQuant\(Permutation Residual Quantization\), whose overall pipeline is illustrated in Figure[1](https://arxiv.org/html/2609.22106#S1.F1)\. For each candidateα\\alpha,PRQuantfirst applies the AWQ\-style scaling and then selects the top\-kkhardest\-to\-quantize input channels from the scaled weights\. These channels are permuted to the tail, where a static weight\-side residual sub\-tensor is constructed from their MXFP4 quantization residuals\. On the activation side, the corresponding tail activation copy can be obtained either directly or through offline expansion ofWupW\_\{\\mathrm\{up\}\}andWgateW\_\{\\mathrm\{gate\}\}, without constructing activation residuals online\.The main and residual paths can ultimately be merged into a regular augmented GEMM\.

![Refer to caption](https://arxiv.org/html/2609.22106v1/prq1_.png)

![Refer to caption](https://arxiv.org/html/2609.22106v1/prq2.png)

Figure 1:PRQuant framework\.Our contributions are summarized as follows:

- •We identify a new scaled\-weight bottleneck that arises from AWQ\-style scaling under W4A4/MXFP4 quantization: although the activation dynamic range is reduced, a small subset of scaled\-weight columns dominates the output reconstruction error, limiting the accuracy of low\-precision inference\.
- •We proposePRQuant\(Permutation Residual Quantization\), a training\-free PTQ framework that combines AWQ\-style scaling with an offline\-fusible channel permutation\. This permutation reorganizes the problematic channels into contiguous tail blocks, preserving the original linear mapping while isolating the channels that require compensation into a structured layout\.
- •We introduce a weight\-side residual augmentation scheme where the selected weight residuals are precomputed entirely offline and stored in MXFP4\. At inference time, the corresponding tail activations are accessed via contiguous slicing or offline producer expansion, enabling the primary computation and residual compensation to be fused into a single regular MXFP4 GEMM\. This design eliminates the need for online residual construction, irregular gather operations, or specialized mixed\-precision kernels, making it readily compatible with standard low\-precision matrix multiplication paths\.
- •We evaluatePRQuanton both MoE and dense Qwen3 models under mixed MXFP8/MXFP4 quantization\. By modifying only the corresponding FFN down\-projections,PRQuantreduces layer\-wise reconstruction error and improves the average end\-to\-end accuracy over default MXFP4 by 0\.55 and 1\.24 on Qwen3\-30B\-Instruct and Qwen3\-4B\-Instruct, respectively, demonstrating its effectiveness across distinct FFN architectures\.

## 2 Related Work

#### Scaling\-based post\-training quantization\.

SmoothQuant migrates quantization difficulty between activations and weights through an equivalent scaling transformation, suppressing activation outliers while amplifying the corresponding weight channels\([26](https://arxiv.org/html/2609.22106#bib.bib9)\)\. AWQ further uses activation statistics to identify salient weight channels and protects them through equivalent scaling\([15](https://arxiv.org/html/2609.22106#bib.bib10)\), while OmniQuant calibrates weight and activation quantization errors using learnable weight clipping and learnable equivalent transformations\([23](https://arxiv.org/html/2609.22106#bib.bib11)\)\. These methods are highly effective for the quantization of W8A8 or low\-bit weight\-and\-activation\. However, in W4A4 settings, the quantization difficulty migrated from the activation side must still be absorbed by equally constrained 4\-bit weights, potentially creating a new weight\-side bottleneck\.

#### Rotation and block\-aware FP4 quantization\.

QuaRot applies randomized Hadamard rotations to reshape weight and activation distributions, while SpinQuant learns orthogonal rotations using a quantization\-aware objective\([1](https://arxiv.org/html/2609.22106#bib.bib6);[16](https://arxiv.org/html/2609.22106#bib.bib7)\)\. DartQuant further improves rotation calibration by directly regularizing the distribution of rotated activations\([24](https://arxiv.org/html/2609.22106#bib.bib8)\)\. Although effective for integer W4A4 quantization, global channel mixing is not naturally aligned with the fine\-grained block structure of MXFP4, since it can redistribute outlier energy into originally low\-energy blocks, inflate their shared power\-of\-two scales, and accumulate quantization error\([25](https://arxiv.org/html/2609.22106#bib.bib16);[11](https://arxiv.org/html/2609.22106#bib.bib17)\)\. Format\-aware methods therefore localize transformations to the microscaling granularity\. MR\-GPTQ combines GPTQ with MXFP\-aligned Hadamard rotations, static activation reordering, and format\-specific scale optimization\([8](https://arxiv.org/html/2609.22106#bib.bib15)\)\. BRQ applies independent rotations within MXFP4\-aligned blocks, whereas BATQuant replaces fixed rotations with learnable block\-wise affine transformations and block\-wise clipping\([25](https://arxiv.org/html/2609.22106#bib.bib16);[11](https://arxiv.org/html/2609.22106#bib.bib17)\)\. These methods nevertheless retain activation\-side transformations on the inference path, either as explicit online operations in BRQ and BATQuant or through specialized fused kernels in MR\-GPTQ\.

#### Channel reordering and outlier isolation\.

Permutation and channel reordering constitute another class of low\-overhead structural transformations\. RPTQ argues that the difficulty of activation quantization also arises from varying ranges across channels, and mitigates this issue by rearranging channels and quantizing them in clusters\([28](https://arxiv.org/html/2609.22106#bib.bib12)\)\. MixQuant further investigates the effect of permutation on block\-wise quantization from the perspective of block mass redistribution via permutations\([22](https://arxiv.org/html/2609.22106#bib.bib14)\)\. These methods demonstrate that channel layout itself is an important factor that affects both low\-bit quantization error and hardware efficiency\. However, reordering alone is generally insufficient to fully recover the accuracy lost under 4\-bit quantization\.

#### Mixed\-precision and residual compensation\.

Early mixed\-precision methods preserve quantization\-sensitive components using higher precision\. LLM\.int8\(\) isolates outlier feature dimensions into a 16\-bit matrix multiplication, while SpQR retains high\-error weights in a sparse high\-precision component\([6](https://arxiv.org/html/2609.22106#bib.bib20);[7](https://arxiv.org/html/2609.22106#bib.bib21)\)\. For FP4 quantization, OSC identifies token\-persistent outlier positions offline and executes a 4\-bit base GEMM together with a 16\-bit outlier branch\([30](https://arxiv.org/html/2609.22106#bib.bib22)\)\. To preserve unified low\-bit execution, ARCQuant augments the activation matrix with quantized residual channels along the reduction dimension, but it still relies on online residual construction and a corresponding fused preprocessing kernel\([17](https://arxiv.org/html/2609.22106#bib.bib18)\)\. MosaicQuant uses a dense 4\-bit base component with a sparse 4\-bit residual to compensate for error\-critical blocks, but it requires a specialized dense\-sparse execution engine to fuse residual computation\([10](https://arxiv.org/html/2609.22106#bib.bib23)\)\. These approaches improve low\-bit quantization accuracy by adding representational capacity, but this often comes at the cost of non\-trivial runtime overhead\.

## 3 Motivation

In practice, the activations often exhibit substantial channel\-wise variations, with a small fraction of channels containing disproportionately large outliers \(Figure[1](https://arxiv.org/html/2609.22106#S1.F1)\(a\)\)\. Under 4\-bit quantization, these outliers enlarge the quantization range and reduce the effective resolution available to normal channels, exacerbating the overall quantization error\. SmoothQuant mitigates this problem through an equivalent per\-channel scaling transformation, which suppresses activation outliers by transferring the corresponding magnitudes to the weights\. Although this transformation improves activation quantization accuracy, it may amplify a small number of scaled\-weight input channels and create a new weight\-side bottleneck under W4A4 quantization\.

#### Limitations of high\-precision weight\-side compensation\.

SVDQuant demonstrates that merely transferring quantization difficulty between activations and weights is insufficient at 4\-bit precision\. It applies a low\-rank compensation approach to decompose the magnified weights into the sum of a 4\-bit quantized weight matrix and a full\-precision low\-rank residual \(LoRA\)\([12](https://arxiv.org/html/2609.22106#bib.bib19)\)\. Although the low\-rank component captures the dominant singular directions of the magnified weight and effectively preserves accuracy, it introduces significant overhead\. Executing the full\-precision residual incurs extra data movement, whose efficient deployment relies on specialized mixed\-precision kernel fusion\. This makes the approach less compatible with a standard and homogeneous low\-precision GEMM path\.

#### Permutation enables structured weight\-side compensation

Our key observation is that, after scaling, the remaining quantization difficulty is highly concentrated: only a small number of scaled\-weight channels dominate the reconstruction error\. However, as these channels are typically scattered across non\-contiguous regions \(Figure[1](https://arxiv.org/html/2609.22106#S1.F1)\(b\)\), directly compensating for them would require expensive index gatherings\. Therefore, we apply channel permutation to reorder the indices of the scaled weight matrices to consolidate the abnormal channels into contiguous tail blocks\. The permutation yields two benefits\. Numerically, it isolates the large\-magnitude channels from regular ones, preventing outliers from inflating the shared quantization scales of otherwise well\-behaved MXFP4 blocks\. Structurally, it transforms the scattered indices into a dense, contiguous data layout, thereby facilitating coalesced memory access and aligning well with hardware vectorized loading during inference\. As detailed in Section[4](https://arxiv.org/html/2609.22106#S4), we further fuse this operation into adjacent static weights entirely offline, allowing online inference to retain a standard GEMM path without runtime overhead\.

#### Static low\-precision residual augmentation\.

After permutation, we pre\-compute the MXFP4 quantized weights of the identified outlier blocks and construct the corresponding residual entirely offline\. During LLM inference, the tail activations only requires contiguous slicing or offline producer expansion, avoiding the online residual construction and irregular gather operations required by ARCQuant[17](https://arxiv.org/html/2609.22106#bib.bib18)\. Subsequently, we concatenate both the extracted activations and the stored residuals with their primary counterparts along the reduction dimension to evaluate the entire compensated operation in a single, regular MXFP4 GEMM\.

Taken together, these observations suggest that W4A4 quantization is not solely a problem of reducing numerical error, but also one of organizing that error into a hardware\-friendly computational structure\. This motivates the central design principle ofPRQuant: reordering magnified but scattered weights into a structured form for efficient compensation\.

Specifically, we use an offline\-fusible permutation to reorganize the hardest\-to\-quantize scaled\-weight channels into contiguous tail blocks, enabling their quantization residuals to be precomputed and stored entirely offline\. At inference time, the activation side only needs to provide the corresponding contiguous tail activations, allowing the main computation and residual compensation to be unified into a single regular MXFP4 GEMM through augmentation\.

In a nutshell, by integrating AWQ, permutation, and residual compensation \(for scaled weights\),PRQuantconverts an irregular compensation problem into a regular low\-precision computation without relying on online residual construction or specialized mixed\-precision kernels; therefore, it addresses the long\-standing conflict between quantization accuracy and execution efficiency\.

## 4 Method

This section presents thePRQuantpipeline for MoE down\-projection layers, as illustrated in Figure[1](https://arxiv.org/html/2609.22106#S1.F1)\.

#### Offline scaling and channel reordering\.

LetXXdenote the input to an MoE expert\. We useZZto denote the intermediate activations of the gated FFN collected from a calibration set\. Given a scaling strengthα\\alpha, we construct the diagonal scaling matrix asDα=diag⁡\(sα\)D\_\{\\alpha\}=\\operatorname\{diag\}\(s\_\{\\alpha\}\), wheresα,j=\(maxt⁡\|Zt,j\|\+ϵ\)αs\_\{\\alpha,j\}=\(\\max\_\{t\}\|Z\_\{t,j\}\|\+\\epsilon\)^\{\\alpha\}\. We then identify the Top\-kkhardest\-to\-quantize channels inWdown​DαW\_\{\\mathrm\{down\}\}D\_\{\\alpha\}and move them to contiguous tail blocks using a permutation matrixPαP\_\{\\alpha\}\.

The scaling and permutation can be fused offline into the weights of adjacent linear layers\. Specifically,

MLP⁡\(X\)\\displaystyle\\mathrm\{MLP\}\(X\)=\(X​Wup⊤⊙σ⁡\(X​Wgate⊤\)\)​Wdown⊤\\displaystyle=\\left\(XW\_\{\\mathrm\{up\}\}^\{\\top\}\\odot\\sigma\(XW\_\{\\mathrm\{gate\}\}^\{\\top\}\)\\right\)W\_\{\\mathrm\{down\}\}^\{\\top\}\(1\)=\(X​\(Pα⊤​Dα−1​Wup\)⊤⊙σ⁡\(X​\(Pα⊤​Wgate\)⊤\)\)​\(Wdown​Dα​Pα\)⊤\\displaystyle=\\left\(X\(P\_\{\\alpha\}^\{\\top\}D\_\{\\alpha\}^\{\-1\}W\_\{\\mathrm\{up\}\}\)^\{\\top\}\\odot\\sigma\\\!\\left\(X\(P\_\{\\alpha\}^\{\\top\}W\_\{\\mathrm\{gate\}\}\)^\{\\top\}\\right\)\\right\)\(W\_\{\\mathrm\{down\}\}D\_\{\\alpha\}P\_\{\\alpha\}\)^\{\\top\}=Z~α​W~down,α⊤\.\\displaystyle=\\widetilde\{Z\}\_\{\\alpha\}\\widetilde\{W\}\_\{\\mathrm\{down\},\\alpha\}^\{\\top\}\.Here, the lastkkcolumns ofZ~α\\widetilde\{Z\}\_\{\\alpha\}are denoted byZ~α,tail\\widetilde\{Z\}\_\{\\alpha,\\mathrm\{tail\}\}, and the corresponding columns ofW~down,α\\widetilde\{W\}\_\{\\mathrm\{down\},\\alpha\}are denoted byW~down,α,tail\\widetilde\{W\}\_\{\\mathrm\{down\},\\alpha,\\mathrm\{tail\}\}\.

The permutation forces the outlier channels into contiguous tail blocks, providing a structured layout for subsequent residual compensation of the corresponding weights\. Meanwhile, the scaling transformation substantially reduces the dynamic range of the tail activations, making MXFP4 quantization of the main path more favorable\.

For different candidate values ofα\\alpha, we use the down\-projection reconstruction error as the selection criterion:

NRMSEα\\displaystyle\\operatorname\{NRMSE\}\_\{\\alpha\}=‖Y^α−Z​Wdown⊤‖F‖Z​Wdown⊤‖F,\\displaystyle=\\frac\{\\left\\\|\\widehat\{Y\}\_\{\\alpha\}\-ZW\_\{\\mathrm\{down\}\}^\{\\top\}\\right\\\|\_\{F\}\}\{\\left\\\|ZW\_\{\\mathrm\{down\}\}^\{\\top\}\\right\\\|\_\{F\}\},\(2\)Y^α\\displaystyle\\widehat\{Y\}\_\{\\alpha\}=𝒬⁡\(\[Z~α,Z~α,tail\]\)​𝒬​\(\[W~down,α,Rα,W\]\)⊤\.\\displaystyle=\\mathcal\{Q\}\\\!\\left\(\[\\widetilde\{Z\}\_\{\\alpha\},\\widetilde\{Z\}\_\{\\alpha,\\mathrm\{tail\}\}\]\\right\)\\mathcal\{Q\}\\\!\\left\(\[\\widetilde\{W\}\_\{\\mathrm\{down\},\\alpha\},R\_\{\\alpha,W\}\]\\right\)^\{\\top\}\.Here,

Rα,W=W~α,down,tail−𝒬⁡\(W~α,down,tail\),R\_\{\\alpha,W\}=\\widetilde\{W\}\_\{\\alpha,\\mathrm\{down\},\\mathrm\{tail\}\}\-\\mathcal\{Q\}\\\!\\left\(\\widetilde\{W\}\_\{\\alpha,\\mathrm\{down\},\\mathrm\{tail\}\}\\right\),\(3\)and𝒬⁡\(⋅\)\\mathcal\{Q\}\(\\cdot\)denotes the MXFP4 quantize–dequantize operator\. For notational simplicity, we omit the subscriptα\\alphabelow\.

#### Offline weight residual augmentation\.

After scaling and permutation, thekkhardest\-to\-quantize weight channels are concentrated in contiguous tail blocks\. We compute their residualRWR\_\{W\}and append it offline to the original weight\. The resulting down\-projection computation is

Y^\\displaystyle\\widehat\{Y\}=𝒬⁡\(Z~\)​𝒬​\(W~down\)⊤\+𝒬⁡\(Z~tail\)​𝒬​\(RW\)⊤\\displaystyle=\\mathcal\{Q\}\(\\widetilde\{Z\}\)\\mathcal\{Q\}\(\\widetilde\{W\}\_\{\\mathrm\{down\}\}\)^\{\\top\}\+\\mathcal\{Q\}\(\\widetilde\{Z\}\_\{\\mathrm\{tail\}\}\)\\mathcal\{Q\}\(R\_\{W\}\)^\{\\top\}\(4\)≈𝒬⁡\(\[Z~,Z~tail\]\)​𝒬​\(\[W~down,RW\]\)⊤\.\\displaystyle\\approx\\mathcal\{Q\}\\\!\\left\(\[\\widetilde\{Z\},\\widetilde\{Z\}\_\{\\mathrm\{tail\}\}\]\\right\)\\mathcal\{Q\}\\\!\\left\(\[\\widetilde\{W\}\_\{\\mathrm\{down\}\},R\_\{W\}\]\\right\)^\{\\top\}\.Here, the first term represents the standard MXFP4 down\-projection, and the second term compensates for the quantization error exaggerated in the tail weight slice\. Mathematically, this formulation expands the reduction dimension of the GEMM fromCC\(the intermediate dimension of FFN\) toC\+kC\+k\. Sincek≪Ck\\ll C, the computational and memory overhead introduced by this augmentation is negligible\.

#### Activation sub\-tensor construction and quantization\.

For the activation side of Eq\. \([4](https://arxiv.org/html/2609.22106#S4.E4)\), we consider two implementations: online copy and offline copy\.

\(1\) Online copy\.

Since the permutation is fused offline into the weights of adjacent layers, the activation tensorZ~\\widetilde\{Z\}natively contains the reordered channels during runtime\. Consequently, obtainingZ~tail\\widetilde\{Z\}\_\{\\mathrm\{tail\}\}requires merely a zero\-copy, contiguous tail\-slicing operation onZ~\\widetilde\{Z\}:

Z~tail=Z~\[:,C−k:C\],\\widetilde\{Z\}\_\{\\mathrm\{tail\}\}=\\widetilde\{Z\}\[:,\\,C\-k:C\],\(5\)which avoids the expensive online gathering operation in prior work\([17](https://arxiv.org/html/2609.22106#bib.bib18)\)\. The concatenated tensor\[Z~,Z~tail\]\[\\widetilde\{Z\},\\widetilde\{Z\}\_\{\\mathrm\{tail\}\}\]is then jointly quantized as the activation input to the augmented GEMM\.

\(2\) Offline copy\.Alternatively, the copy operation can be moved to the FFN weight\-construction stage\. Specifically, after fusing the AWQ scaling and permutation into the up\- and gate\-projection weights, we duplicate their lastkkoutput channels, such that the FFN directly produces the intermediate activation with duplicated tail channels,\[Z~,Z~tail\]\[\\widetilde\{Z\},\\widetilde\{Z\}\_\{\\mathrm\{tail\}\}\]\. During inference, the required activation sub\-tensor is therefore generated directly by the expanded up\- and gate\-projections\. The corresponding cost is that both the output dimensions of the up/gate projections and the reduction dimension of the down\-projection increase fromCCtoC\+kC\+k\.

The two implementations differ only in howZ~tail\\widetilde\{Z\}\_\{\\mathrm\{tail\}\}is obtained and do not alter the final compensation formulation\. Importantly, we do not construct an online activation residual of the formZ~tail−𝒬⁡\(Z~tail\)\\widetilde\{Z\}\_\{\\mathrm\{tail\}\}\-\\mathcal\{Q\}\(\\widetilde\{Z\}\_\{\\mathrm\{tail\}\}\)\. The tail activations are only duplicated and quantized, while residual compensation is performed exclusively on the weight side\. The online\-copy implementation introduces only a lightweight contiguous slicing operation and therefore incurs minimal additional computation\. The offline\-copy implementation completely eliminates runtime copying, at the cost of a larger but regular GEMM\.

Algorithm 1Calibration and Export Pipeline ofPRQuant0:Weight

WW, calibration activations

ZcalZ\_\{\\rm cal\}, MXFP4 quantizer

𝒬\\mathcal\{Q\}, candidate set

𝒜\\mathcal\{A\}, residual width

kk
1:Compute the per\-channel maxima of

ZcalZ\_\{\\rm cal\}for scaling\.

2:foreach

α∈𝒜\\alpha\\in\\mathcal\{A\}do

3:Construct

DαD\_\{\\alpha\}and the scaled weight

Wdown​DαW\_\{\\mathrm\{down\}\}D\_\{\\alpha\}\.

4:Select the

kkinput channels with the largest column\-wise MXFP4 quantization errors\.

5:Construct

PαP\_\{\\alpha\}to move the selected channels to contiguous tail blocks\.

6:Obtain

Z~α\\widetilde\{Z\}\_\{\\alpha\}and

W~down,α\\widetilde\{W\}\_\{\\mathrm\{down\},\\alpha\}using Equation \([1](https://arxiv.org/html/2609.22106#S4.E1)\)\.

7:Quantize

W~down,α\\widetilde\{W\}\_\{\\mathrm\{down\},\\alpha\}and compute the quantized tail residual

RW,αqR\_\{W,\\alpha\}^\{q\}\.

8:Construct

Wα,augq=\[𝒬⁡\(W~down,α\),RW,αq\]W\_\{\\alpha,\\mathrm\{aug\}\}^\{q\}=\[\\mathcal\{Q\}\(\\widetilde\{W\}\_\{\\mathrm\{down\},\\alpha\}\),R\_\{W,\\alpha\}^\{q\}\]and the augmented activations\.

9:Compute the reconstruction error using Equation \([2](https://arxiv.org/html/2609.22106#S4.E2)\)\.

10:endfor

11:Select

α⋆\\alpha^\{\\star\}with the minimum error and export

\(P,D\)\(P,D\)and

WaugqW\_\{\\mathrm\{aug\}\}^\{q\}\.

12:Fuse the equivalent transform into the producer and down\-projection weights\.

13:Use either an online contiguous slice or offline producer\-channel duplication for the tail activations\.

14:Execute one MXFP4 augmented GEMM on

\[Z~,Z~tail\]\[\\widetilde\{Z\},\\widetilde\{Z\}\_\{\\mathrm\{tail\}\}\]\.

## 5 Experiments

### 5\.1 Experimental Setup

#### Model and quantization configuration\.

We use Qwen3\-4B\-Instruct\-2507 and Qwen3\-30B\-A3B\-Instruct\-2507\([27](https://arxiv.org/html/2609.22106#bib.bib30)\)as our primary evaluation models\. Unless otherwise specified,PRQuantis applied only to the down\-projection layers of MoE experts\. All compared methods use the same model checkpoint and evaluation pipeline\. We evaluate a mixed MXFP quantization configuration, where the attention modules are quantized in MXFP8 and the MoE modules in MXFP4\.

#### Baseline methods\.

We benchmarkPRQuantagainst full\-precision BF16, default MXFP4, and three representative low\-bit quantization methods\. Specifically, BF16 provides the full\-precision reference, whereas default MXFP4 directly quantizes both activations and weights without additional quantization\-aware transformations or compensation\. We further compare against ARCQuant, which performs residual compensation for activation quantization; FlatQuant, which learns affine transformations to reshape weight and activation distributions for low\-bit quantization; and QuaRot, for which we apply random Hadamard rotations to the down\-projection layers\([1](https://arxiv.org/html/2609.22106#bib.bib6)\)\. All quantized methods follow the same mixed\-precision setting, with MXFP8 for the attention modules and MXFP4 for the FFN/MoE modules\.

#### Calibration and benchmarking datasets\.

#### Implementation and environmental setup\.

For a rigorous comparison, all methods are implemented and evaluated under identical environmental conditions on the Ascend A2 NPU\. Specifically, all model checkpoints \(including full\-precision and quantized results\) are deployed using thevllm\-ascend \(0\.20\.2 RC1\)serving framework, and end\-to\-end evaluation is conducted uniformly via OpenCompass\([19](https://arxiv.org/html/2609.22106#bib.bib29)\)\.

### 5\.2 End\-to\-End Accuracy

Table 1:End\-to\-end accuracy \(%\) on Qwen3\-30B\-Instruct and Qwen3\-4B\-Instruct using zero\-shot evaluation\. For quantized configurations, the attention modules use MXFP8 and the MoE modules use MXFP4\. Bold values denote the best result among the quantized configurations for each model\. The average is the unweighted mean over the five benchmarks\.We report the end\-to\-end accuracy in Table[1](https://arxiv.org/html/2609.22106#S5.T1)\. For Qwen3\-30B\-A3B\-Instruct\-2507, default MXFP4 reduces the average accuracy from 90\.62 to 89\.35, corresponding to a 1\.27 degradation from BF16\. By modifying only the expert down\-projection computation,PRQuantimproves the average accuracy to 89\.90, yielding a 0\.55 gain over MXFP4 and recovering approximately 43\.3% of the accuracy gap between MXFP4 and BF16\. It also achieves the highest average accuracy among all evaluated quantized methods, outperforming FlatQuant, QuaRot, and ARCQuant by 0\.09, 1\.02, and 1\.08, respectively\.

The improvement over MXFP4 is observed across all five benchmarks\. In particular,PRQuantimproves WinoGrande and PIQA by 1\.42 and 1\.14, respectively, while providing smaller gains of 0\.19, 0\.03, and 0\.01 on ARC\-E, HellaSwag, and ARC\-C\. Compared with the BF16 reference,PRQuantfully preserves the ARC\-E accuracy and limits the PIQA degradation to only 0\.32\. These empirical results demonstrate that mitigating the reconstruction error in sensitive expert down\-projections improves end\-to\-end accuracy\.

For the dense Qwen3\-4B\-Instruct\-2507, the quantization gap is substantially larger: default MXFP4 reduces the average accuracy from 84\.07 to 80\.96, corresponding to a 3\.11 degradation from BF16\. ApplyingPRQuantto the dense FFN down\-projections improves the average accuracy to 82\.20, yielding a 1\.24 improvement over MXFP4 and recovering approximately 39\.9% of the of the average accuracy gap between MXFP4 and BF16\.PRQuantagain achieves the highest average accuracy among the evaluated quantized methods, outperforming FlatQuant, ARCQuant, and QuaRot by 0\.77, 0\.87, and 0\.98, respectively\.

Compared with MXFP4,PRQuantimproves four of the five benchmarks, with gains of 1\.69 on ARC\-C, 0\.53 on ARC\-E, 3\.68 on HellaSwag, and 0\.76 on PIQA\. The largest improvement is observed on HellaSwag, suggesting that the dense model is particularly sensitive to quantization errors in the FFN down\-projections\. Although individual baselines remain competitive on specific tasks,PRQuantprovides the strongest overall accuracy across the five benchmarks\. These results show that reducing reconstruction error in the targeted down\-projections translates into end\-to\-end accuracy gains for both MoE expert FFNs and dense FFNs, while requiring no modification to the attention computation\.

### 5\.3 Ablation Studies

To assess the contribution of each component ofPRQuant, we conduct layer\-wise ablation studies on the expert down\-projection layers of Qwen3\-30B\. For each layer, we compute the normalized reconstruction error for every expert and report the mean across all experts in that layer \(denoted asyy\-NRMSE\)\. Loweryy\-NRMSE indicates more accurate reconstruction of the down\-projection outputs\.

We compare the completePRQuantwith naive MXFP4 quantization and three ablated variants that remove AWQ scaling \(w/o SQ\), residual compensation \(w/o Res\), and permutation \(w/o Perm\), respectively\. We additionally evaluatePRQuantwith weight\-side E8M0 block\-scale search, denoted asPRQuant\(E8M0\-W\)\. Figure[2](https://arxiv.org/html/2609.22106#S5.F2)presents the layer\-wise results, while Table[2](https://arxiv.org/html/2609.22106#S5.T2)reports the averagedyy\-NRMSE across all evaluated layers\.

![Refer to caption](https://arxiv.org/html/2609.22106v1/fig3_.png)Figure 2:Layer\-wise ablation results for the down\-projection layers of Qwen3\-30B MoE experts\. The vertical axis denotes the meanyy\-NRMSE over all experts in each layer, where lower values indicate smaller reconstruction errors in the down\-projection outputs\.Table 2:Average ablation results on the down\-projection layers of Qwen3\-30B MoE experts\.As shown,PRQuantreduces the average y\-NRMSE from 0\.1861 \(the naive MXFP4 baseline\) to 0\.1678\. Removing AWQ scaling increases the error to 0\.1791, confirming that scaling activation is essential for reducing quantization error\. Similarly, omitting residual compensation leads to a higher error of 0\.1809, demonstrating that MXFP4 quantization alone is insufficient for recovering the magnified weights\. Residual augmentation plays a decisive role in enhancing the final reconstruction quality\.

In addition, removing permutation results in a marginal error increase to 0\.1687\. This observation indicates that while scaling and residual compensation provide the primary accuracy gains, permutation also offers a consistent marginal improvement\. More fundamentally, the primary value of permutation is structural rather than purely numerical: it reorganizes the scattered hard channels into a contiguous tail slice\. This enables the residual compensation to be executed via a standard, hardware\-friendly augmented GEMM, thereby completely eliminating dynamic gathering overhead\. Finally, introducing a local block\-scale search \(PRQuantE8M0\-W\) further suppresses the error to 0\.1658, demonstrating that optimizing weight scaling factors can achieve even higher reconstruction accuracy under MXFP4 quantization\.

## 6 Conclusion

In this work, we presentedPRQuant, a low\-overhead framework for MXFP4 quantization\. Motivated by the observation that activation scaling can shift the quantization bottleneck from activation outliers to the corresponding weight input channels, which remain scattered in the original layout,PRQuantcombines AWQ\-style scaling with an offline\-fusible permutation to reorganize these channels into contiguous tail blocks, enabling the corresponding MXFP4\-quantized weight residuals to be precomputed entirely offline\. At inference, the corresponding tail activations are supplied through contiguous slicing or offline producer expansion, allowing the primary computation and residual compensation to be unified into a single regular MXFP4 GEMM without online residual construction, irregular gather operations, or specialized mixed\-precision kernels\. Experiments on both MoE and dense models show that applyingPRQuantonly to FFN down\-projections improves overall accuracy over default MXFP4, highlighting the importance of organizing quantization difficulty into a hardware\-friendly computational structure\. Future work will explore extendingPRQuantbeyond FFN down\-projections to other Transformer layers, such as attention output and FFN up/gate projections, through layer\-specific offline fusion strategies\.

## Ethics Statement

This work focuses on model compression and inference efficiency and does not involve new data collection or additional risks to user privacy\. Low\-bit quantization may alter model output behavior; therefore, task\-level and safety regression tests should be conducted before deployment\.

## References

- Ashkbooset al\.\(2024\)S\. Ashkboos, A\. Mohtashami, M\. L\. Croci, B\. Li, P\. Cameron, M\. Jaggi, D\. Alistarh, T\. Hoefler, and J\. HensmanQuaRot: outlier\-free 4\-bit inference in rotated LLMs\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 100213–100240\.Cited by:[§1](https://arxiv.org/html/2609.22106#S1.p3.1),[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2609.22106#S5.SS1.SSS0.Px2.p1.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. ChoiPIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§5\.1](https://arxiv.org/html/2609.22106#S5.SS1.SSS0.Px3.p1.1)\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder,et al\.Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2609.22106#S1.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§5\.1](https://arxiv.org/html/2609.22106#S5.SS1.SSS0.Px3.p1.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-AIDeepSeek\-V3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2609.22106#S1.p1.1)\.
- Dettmerset al\.\(2022\)T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. ZettlemoyerLLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 30318–30332\.Cited by:[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px4.p1.1)\.
- Dettmerset al\.\(2024\)T\. Dettmers, R\. Svirschevski, V\. Egiazarian, D\. Kuznedelev, E\. Frantar, S\. Ashkboos, A\. Borzunov, T\. Hoefler, and D\. AlistarhSpQR: a sparse\-quantized representation for near\-lossless LLM weight compression\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px4.p1.1)\.
- Egiazarianet al\.\(2026\)V\. Egiazarian, R\. L\. Castro, D\. Kuznedelev, A\. Panferov, E\. Kurtic, S\. Pandit, A\. Marques, M\. Kurtz, S\. Ashkboos, T\. Hoefler, and D\. AlistarhBridging the gap between promise and performance for microscaling FP4 quantization\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px2.p1.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§1](https://arxiv.org/html/2609.22106#S1.p1.1)\.
- Huet al\.\(2026\)Y\. Hu, H\. Wang, Z\. Hong, Q\. Liu, Q\. Shou, J\. Lin, S\. Guo, X\. Shen, X\. Huang, D\. Wang, and J\. YangMosaicQuant: inlier–outlier disaggregation for unified 4\-bit LLM quantization\.arXiv preprint arXiv:2606\.15652\.Cited by:[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px4.p1.1)\.
- Liet al\.\(2026\)J\. Li, M\. Zhang, X\. Xia, H\. Bao, H\. Bai, Z\. Dong, and X\. YuBATQuant: outlier\-resilient MXFP4 quantization via learnable block\-wise optimization\.arXiv preprint arXiv:2603\.16590\.Cited by:[Appendix A](https://arxiv.org/html/2609.22106#A1.p3.1),[§1](https://arxiv.org/html/2609.22106#S1.p3.1),[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2025\)M\. Li, Y\. Lin, Z\. Zhang, T\. Cai, X\. Li, J\. Guo, E\. Xie, C\. Meng, J\. Zhu, and S\. HanSVDQuant: absorbing outliers by low\-rank components for 4\-bit diffusion models\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2609.22106#A1.p3.1),[§1](https://arxiv.org/html/2609.22106#S1.p3.1),[§3](https://arxiv.org/html/2609.22106#S3.SS0.SSS0.Px1.p1.1)\.
- Linet al\.\(2026\)H\. Lin, X\. Jia, H\. Xu, B\. Yao, X\. Guo, Y\. Wu, Z\. Lu, Y\. Wei, Q\. Zhang, and Z\. SunDuQuant\+\+: fine\-grained rotation enhances microscaling FP4 quantization\.arXiv preprint arXiv:2604\.17789\.Cited by:[Appendix A](https://arxiv.org/html/2609.22106#A1.p3.1)\.
- Linet al\.\(2024a\)H\. Lin, H\. Xu, Y\. Wu, J\. Cui, Y\. Zhang, L\. Mou, L\. Song, Z\. Sun, and Y\. WeiDuQuant: distributing outliers via dual transformation makes stronger quantized LLMs\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 87766–87800\.Cited by:[Appendix A](https://arxiv.org/html/2609.22106#A1.p3.1)\.
- Linet al\.\(2024b\)J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. HanAWQ: activation\-aware weight quantization for on\-device LLM compression and acceleration\.InProceedings of Machine Learning and Systems,Vol\.6,pp\. 87–100\.Cited by:[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Zhao, I\. Fedorov, B\. Soran, D\. Choudhary, R\. Krishnamoorthi, V\. Chandra, Y\. Tian, and T\. BlankevoortSpinQuant: LLM quantization with learned rotations\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px2.p1.1)\.
- Menget al\.\(2026\)H\. Meng, Y\. Luo, Y\. Zhao, W\. Liu, P\. Zhang, and X\. MaARCQuant: boosting NVFP4 quantization with augmented residual channels for LLMs\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8609–8623\.Cited by:[Appendix A](https://arxiv.org/html/2609.22106#A1.p3.1),[§1](https://arxiv.org/html/2609.22106#S1.p3.1),[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2609.22106#S3.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2609.22106#S4.SS0.SSS0.Px3.p3.2)\.
- Open Compute Project \(2023\)Open Compute ProjectOCP microscaling formats \(MX\) specification, version 1\.0\.Technical reportOpen Compute Project\.Note:Final specificationCited by:[§1](https://arxiv.org/html/2609.22106#S1.p2.1)\.
- OpenCompass Contributors \(2023\)OpenCompass ContributorsOpenCompass: a universal evaluation platform for foundation models\.Note:GitHub repository[https://github\.com/open\-compass/opencompass](https://github.com/open-compass/opencompass)Cited by:[§5\.1](https://arxiv.org/html/2609.22106#S5.SS1.SSS0.Px4.p1.1)\.
- Rouhaniet al\.\(2023\)B\. D\. Rouhani, R\. Zhao, A\. More,et al\.Microscaling data formats for deep learning\.arXiv preprint arXiv:2310\.10537\.Cited by:[§1](https://arxiv.org/html/2609.22106#S1.p2.1)\.
- Sakaguchiet al\.\(2021\)K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. ChoiWinoGrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§5\.1](https://arxiv.org/html/2609.22106#S5.SS1.SSS0.Px3.p1.1)\.
- Sanjeetet al\.\(2026\)S\. Sanjeet, I\. Colbert, P\. Monteagudo\-Lago, G\. Franco, Y\. Umuroglu, and N\. J\. FraserMixQuant: pushing the limits of block rotations in post\-training quantization\.arXiv preprint arXiv:2601\.22347\.Cited by:[§1](https://arxiv.org/html/2609.22106#S1.p3.1),[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px3.p1.1)\.
- Shaoet al\.\(2024\)W\. Shao, M\. Chen, Z\. Zhang, P\. Xu, L\. Zhao, Z\. Li, K\. Zhang, P\. Gao, Y\. Qiao, and P\. LuoOmniQuant: omnidirectionally calibrated quantization for large language models\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px1.p1.1)\.
- Shaoet al\.\(2025\)Y\. Shao, Y\. Chen, P\. Wang, J\. Yu, J\. Lin, Y\. Yao, Z\. Wei, and J\. ChengDartQuant: efficient rotational distribution calibration for LLM quantization\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px2.p1.1)\.
- Shaoet al\.\(2026\)Y\. Shao, P\. Wang, Y\. Chen, C\. Xu, Z\. Wei, and J\. ChengBlock rotation is all you need for MXFP4 quantization\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2609.22106#A1.p3.1),[§1](https://arxiv.org/html/2609.22106#S1.p3.1),[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px2.p1.1)\.
- Xiaoet al\.\(2023\)G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. HanSmoothQuant: accurate and efficient post\-training quantization for large language models\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 38087–38099\.Cited by:[§1](https://arxiv.org/html/2609.22106#S1.p3.1),[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§5\.1](https://arxiv.org/html/2609.22106#S5.SS1.SSS0.Px1.p1.1)\.
- Yuanet al\.\(2023\)Z\. Yuan, L\. Niu, J\. Liu, W\. Liu, X\. Wang, Y\. Shang, G\. Sun, Q\. Wu, J\. Wu, and B\. WuRPTQ: reorder\-based post\-training quantization for large language models\.arXiv preprint arXiv:2304\.01089\.Cited by:[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px3.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.Cited by:[§5\.1](https://arxiv.org/html/2609.22106#S5.SS1.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2026\)Z\. Zhang, Y\. Li, Z\. Zou, B\. Du, Y\. Sun, H\. Dong, and H\. WangOSC: hardware efficient W4A4 quantization via outlier separation in channel dimension\.arXiv preprint arXiv:2604\.12782\.Cited by:[Appendix A](https://arxiv.org/html/2609.22106#A1.p3.1),[§2](https://arxiv.org/html/2609.22106#S2.SS0.SSS0.Px4.p1.1)\.

## Appendix AApplicability and Layer Selection

Although this work uses permutation as the primary example, the underlying principle is more general\. Any channel\-side operation acting on a hidden dimension shared by adjacent modules can support a low\-overhead compensation mechanism, provided that it can be absorbed offline into the producer or consumer weights, or that it exposes the required activation sub\-tensor through regular access\. Because different layers provide different fusion opportunities, the applicable operations need not be restricted to permutation\. They may also include diagonal scaling, block permutation, channel duplication, structured augmentation, or structured pruning with residual correction\.

This principle applies most naturally to the second projection of an FFN, including the down\-projection of a dense FFN and the down\-projection of an MoE expert\. For these layers, channel reordering, scaling, and tail duplication can be implemented by modifying adjacent weights offline\. Similar ideas may also be extended to the attention output projection\.

We select the MoE expert down\-projection as the primary experimental setting for two reasons\. First, this layer follows the gated activation, and prior work has observed token\-local massive outliers at FFN down\-projection inputs\. SmoothQuant may also migrate activation\-side difficulty at this position into pronounced weight\-side outliers\([14](https://arxiv.org/html/2609.22106#bib.bib13)\)\. Second, existing MXFP4 methods typically rely on block\-aware rotations, affine transformations, fallback paths, or residual/low\-rank compensation to recover accuracy\. These mechanisms may still introduce online activation transformations, dynamic residual construction, additional high\-precision branches, or specialized kernel fusion\([25](https://arxiv.org/html/2609.22106#bib.bib16);[13](https://arxiv.org/html/2609.22106#bib.bib24);[11](https://arxiv.org/html/2609.22106#bib.bib17);[30](https://arxiv.org/html/2609.22106#bib.bib22);[17](https://arxiv.org/html/2609.22106#bib.bib18);[12](https://arxiv.org/html/2609.22106#bib.bib19)\)\. The MoE expert down\-projection therefore combines substantial quantization difficulty with a suitable channel structure, making it a representative layer for evaluating the proposed method\.