Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs

# Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs
Source: [https://arxiv.org/html/2608.26581](https://arxiv.org/html/2608.26581)
Mehran Taghian JaziYunke PengZhuang MaAnandharaju Durai RajuYao WangXing HuangHei Yi MakShadan GolestanHoang LeYonghan DongWei GuoYaoyuan WangAffiliation:HuaweiE\-mail[\{tanzila\.rahman, pengyunke\}@huawei\.com](mailto:{tanzila.rahman,%20pengyunke}@huawei.com)

###### Abstract

Low\-bit quantization offers a promising avenue for reducing the computational and memory demands of Multimodal Large Language Models \(MLLMs\)\. Recent hardware support for low\-precision formats, ranging from MXFP8 to ultra\-low\-bit formats such as MXFP4 and HiF4, has accelerated research into efficient MLLM training and deployment\. In this work, we present a systematic study of these quantization schemes in representative MLLMs that span both video generation and reasoning tasks\. Our analysis shows that MXFP8 achieves near\-lossless performance, whereas aggressive 4\-bit quantization leads to significant degradation\. Through extensive ablations, we identify activation quantization as the primary source of this performance loss, contributing substantially more than weight quantization\. Motivated by this observation, we proposeResidualFallbackQuantization \(RFQ\), a lightweight activation reconstruction framework that supplements the primary ulta\-low\-bit activation representation with an auxiliary quantized residual pathway\. By explicitly modeling and compensating for quantization errors, RFQ improves activation fidelity while preserving the efficiency advantages of ultra\-low\-bit computation\. RFQ requires no architectural modifications and incurs negligible computational overhead\. Extensive experiments on Wan2\.2 and Qwen3\-VL demonstrate that RFQ consistently recovers a substantial portion of the performance lost under the quantization of MXFP4 and HiF4, significantly narrowing the gap to BF16 baselines across both generation and 4 reasoning benchmarks\. Our findings establish activation quantization as the dominant bottleneck in ultra\-low\-bit MLLMs and highlight residual\-based activation reconstruction as an effective and practical strategy for robust 4\-bit deployment\.

## 1Introduction

Foundation Models \(FMs\), particularly Multimodal Large Language Models \(MLLMs\), have emerged as a dominant paradigm for building generalist AI systems capable of jointly reasoning over language, vision, and other modalities\. This unified modeling paradigm has driven substantial progress in visual understanding\[[24](https://arxiv.org/html/2608.26581#bib.bib3),[7](https://arxiv.org/html/2608.26581#bib.bib4)\], multimodal reasoning\[[43](https://arxiv.org/html/2608.26581#bib.bib5),[22](https://arxiv.org/html/2608.26581#bib.bib6)\], and agentic interaction\[[1](https://arxiv.org/html/2608.26581#bib.bib7),[44](https://arxiv.org/html/2608.26581#bib.bib8)\]\. Despite these advances, the training and deployment of MLLMs remain prohibitively expensive due to their large parameter scales, long\-context computation, and heterogeneous multimodal architectures, all of which impose significant memory and compute demands\[[50](https://arxiv.org/html/2608.26581#bib.bib9),[5](https://arxiv.org/html/2608.26581#bib.bib2)\]\. To improve efficiency, a wide range of techniques have been explored, including sparse attention mechanisms\[[55](https://arxiv.org/html/2608.26581#bib.bib10)\], efficient decoding strategies\[[19](https://arxiv.org/html/2608.26581#bib.bib52)\], and most prominently low\-bit quantization\[[50](https://arxiv.org/html/2608.26581#bib.bib9),[17](https://arxiv.org/html/2608.26581#bib.bib11),[29](https://arxiv.org/html/2608.26581#bib.bib12)\]\. Quantization reduces both memory footprint and computational cost by representing model tensors, including weights, activations, gradients, and optimizer states, with reduced\-precision formats such as FP8\.

![Refer to caption](https://arxiv.org/html/2608.26581v1/teaser_main.png)Figure 1:Overview of the impact of activation outliers and the proposed RFQ framework\. Activation outliers degrade MLLM performance under FP4 quantization\. RFQ reduces outlier\-induced quantization errors through residual fallback correction, recovering generation quality close to the BF16 baseline\.Recent advances in hardware\-supported mixed\-precision training have significantly expanded the applicability of low\-bit computation in large\-scale systems\. Early efforts such as TransformerEngine\[[37](https://arxiv.org/html/2608.26581#bib.bib13)\]demonstrated the effectiveness of matrix multiplications to accelerate linear layers\. FP8\-LM\[[39](https://arxiv.org/html/2608.26581#bib.bib20)\]further extended FP8 quantization to gradients, optimizer states, and communication, reducing memory and bandwidth overhead during training\. More recently, COAT\[[50](https://arxiv.org/html/2608.26581#bib.bib9)\]advanced end\-to\-end FP8 training by additionally quantizing activations and second\-order optimizer states, while introducing tensor\-specific strategies such as dynamic range expansion for optimizer statistics and fine\-grained activation quantization for sensitive nonlinear layers\. Alternatively,\[[6](https://arxiv.org/html/2608.26581#bib.bib23)\]identifies attention\-induced activation outliers in transformers and introduces clipped softmax and gated attention to suppress them, enabling full INT8 quantization without extra fine\-tuning\. These developments highlight that effective low\-precision training requires careful and component\-aware design throughout the training stack\.

Motivated by these gains, we investigate whether further efficiency improvements can be achieved by pushing multimodal training into the more aggressive FP4 regime\. However, the limited dynamic range and precision of FP4 introduce substantial numerical instability, often resulting in degraded convergence and accuracy\. To understand this behavior, we conduct a systematic study on two representative MLLMs: Wan2\.2\[[46](https://arxiv.org/html/2608.26581#bib.bib19)\], a multimodal generative model, and Qwen3\-VL\[[3](https://arxiv.org/html/2608.26581#bib.bib14)\], a multimodal reasoning model\. Following\[[20](https://arxiv.org/html/2608.26581#bib.bib18)\], we also quantize key components spanning vision encoders, text encoders, attention blocks, feed\-forward networks, multimodal projection layers, and mixture\-of\-expert modules\. Our analysis reveals that FP4 degradation is highly heterogeneous across both models and modules\. In Wan2\.2, the most severe performance drop occurs in the visual generation backbone \(*i\.e\.*WanDiT\), while the UMT5\[[12](https://arxiv.org/html/2608.26581#bib.bib21)\]text encoder remains relatively robust\. In contrast, Qwen3\-VL exhibits a heightened sensitivity within its vision encoder relative to the language modules; nonetheless, the language backbone also undergoes non\-trivial architectural degradation under ultra\-low\-bit constraints\. Across both models, we find that activation quantization is the primary driver of accuracy degradation, with certain modules exhibiting heavy\-tailed activation distributions and pronounced outliers that are severely distorted under FP4 precision, leading to amplified downstream errors during training\.

Prior work on outlier\-aware quantization provides partial solutions to the activation instability\. Post\-training quantization methods such as SmoothQuant\[[51](https://arxiv.org/html/2608.26581#bib.bib15)\]and MASQuant\[[20](https://arxiv.org/html/2608.26581#bib.bib18)\]mitigate activation outliers through calibration\-time transformations on frozen models\. Quantization\-aware training approaches, including OCC\[[48](https://arxiv.org/html/2608.26581#bib.bib16)\]and Outlier Fallback\[[57](https://arxiv.org/html/2608.26581#bib.bib17)\], address outliers during training via clipping, compensation, or mixed\-precision execution\. However, these methods are not directly suited for end\-to\-end FP4 MLLM training: PTQ approaches rely on static calibration statistics, while existing QAT techniques are largely designed for text\-only LLMs or higher\-bit regimes and typically apply uniform heuristics across modules\. In contrast, our findings show that the FP4 failure modes in MLLMs are both model\-dependent and module\-dependent, with activation outliers evolving dynamically during training and concentrating in different components in different architectures\.

These observations suggest that stable FP4 training for MLLMs requires fully differentiable and training\-aware activation\-error modeling\. To this end, motivated by\[[57](https://arxiv.org/html/2608.26581#bib.bib17)\], we propose*Residual Fallback Quantization*\(*RFQ*\), a lightweight activation\-aware quantization strategy designed for ultra\-low\-bit MLLM training\. RFQ decomposes activations into a quantized primary FP4 representation and an explicit quantization residual without any threshold\-based selection\. Instead of discarding or selectively preserving values, RFQ computes the full quantization error and reconstructs it through a secondary low\-bit quantization pathway\. This ensures that both the primary activations and residual correction remain compatible with FP4 computation\. RFQ directly addresses FP4 sensitivity in MLLMs while requiring no architectural modifications or full\-precision fallback\.

Our contributions are summarized as follows:

- •We perform the first systematic study of ultra\-low precision quantization in multimodal large language models, analyzing the behavior of MXFP8, MXFP4, and HiF4 across both multimodal generation and reasoning architectures, including Wan2\.2 and Qwen3\-VL\.
- •We show that FP4 sensitivity is highly module\-dependent in MLLMs, with visual components generally exhibiting greater susceptibility to quantization\-induced degradation than language components\. We further identify activation quantization as the dominant source of FP4 degradation, revealing that performance collapse is primarily driven by heavy\-tailed activation distributions and activation outliers rather than weight quantization\.
- •We propose*Residual Fallback Quantization*\(*RFQ*\), a lightweight activation\-aware quantization strategy that compensates for FP4 quantization errors by re\-quantizing activation residuals and accumulating their contributions during GEMM\. RFQ effectively mitigates the accuracy degradation introduced by quantization while preserving FP4 computation for the majority of operations\.
- •Extensive experiments demonstrate that RFQ substantially recovers the accuracy degradation introduced by MXFP4 and HiF4 quantization, achieving near BF16 baseline performance while preserving the efficiency benefits of ultra\-low precision computation\.

## 2Related Work

Quantization of LLMs\.Quantization techniques are broadly split into post\-training quantization \(PTQ\) and quantization\-aware training \(QAT\)\. PTQ compresses a pretrained model using a small calibration dataset without further parameter optimization, making it a popular deployment choice due to low computational overhead\. Representative methods include RTN\[[23](https://arxiv.org/html/2608.26581#bib.bib25)\], GPTQ\[[16](https://arxiv.org/html/2608.26581#bib.bib26)\], AWQ\[[28](https://arxiv.org/html/2608.26581#bib.bib24)\], and SmoothQuant\[[51](https://arxiv.org/html/2608.26581#bib.bib15)\], with recent extensions exploring outlier\-aware allocation\[[25](https://arxiv.org/html/2608.26581#bib.bib27)\], mixed\-precision\[[15](https://arxiv.org/html/2608.26581#bib.bib28)\], and rotation\-based transformations\[[18](https://arxiv.org/html/2608.26581#bib.bib29)\]\. However, PTQ struggles in ultra\-low\-bit regimes \(e\.g\., 4\-bit\) because it cannot adapt parameters to quantization\-induced distribution shifts\.

In contrast, QAT integrates quantization into the training phase, allowing models to adapt to low\-precision noise via gradient\-based optimization\. Recent frameworks like LLM\-QAT\[[32](https://arxiv.org/html/2608.26581#bib.bib30)\], BitDistiller\[[14](https://arxiv.org/html/2608.26581#bib.bib31)\], EfficientQAT\[[10](https://arxiv.org/html/2608.26581#bib.bib32)\], and GETA\[[41](https://arxiv.org/html/2608.26581#bib.bib33)\]leverage distillation and staged optimization to improve stability, while QuEST\[[38](https://arxiv.org/html/2608.26581#bib.bib34)\], DB\-LLM\[[8](https://arxiv.org/html/2608.26581#bib.bib35)\], and BitNet\[[35](https://arxiv.org/html/2608.26581#bib.bib36)\]introduce alternative ternary/binary parameterizations\. Despite this, ultra\-low precision QAT remains challenging due to activation outliers and non\-uniform layer sensitivities, which introduce massive quantization noise and amplify error propagation across deep transformer blocks during backpropagation\.

Quantization of Multimodal LLMs\.Multimodal large language models \(MLLMs\) extend LLMs by integrating vision encoders, language backbones, and cross\-modal fusion modules to enable joint reasoning over heterogeneous inputs such as images and text\. However, this multimodal design introduces additional quantization challenges due to substantial differences in activation distributions across modalities and model components\. As a result, direct application of LLM quantization techniques often leads to uneven degradation across modalities\. To address this issue, recent PTQ methods introduce modality\- and layer\-aware designs\. MBQ\[[26](https://arxiv.org/html/2608.26581#bib.bib37)\]leverages modality\-specific token sensitivities to improve calibration quality\. LUQ\[[5](https://arxiv.org/html/2608.26581#bib.bib2)\]studies ultra\-low\-bit PTQ for MLLMs and highlights strong layer\-wise variation in quantization robustness driven by heterogeneous activation statistics\. MQuant\[[54](https://arxiv.org/html/2608.26581#bib.bib39)\]further shows that cross\-modal distribution mismatch is a key factor in quantization degradation\. MASQuant\[[20](https://arxiv.org/html/2608.26581#bib.bib18)\]identifies smoothing imbalance across modalities, where dominant activation scales can suppress others under channel\-wise smoothing, and proposes modality\-aware compensation\. Other approaches, including Q\-VLM\[[47](https://arxiv.org/html/2608.26581#bib.bib38)\], VLMQ\[[53](https://arxiv.org/html/2608.26581#bib.bib40)\], and Quant Experts\[[21](https://arxiv.org/html/2608.26581#bib.bib41)\], incorporate sensitivity\-aware rounding, token\-level importance, or reconstruction\-based correction to reduce quantization error in vision\-language models\. Beyond PTQ, QAT for MLLMs remains relatively underexplored but increasingly important\. Attn\-QAT\[[56](https://arxiv.org/html/2608.26581#bib.bib42)\]analyzes FP4 attention training and identifies key stability constraints in low\-precision backward computation\. MF\-QAT\[[52](https://arxiv.org/html/2608.26581#bib.bib43)\]proposes multi\-format QAT, enabling a single model to operate across multiple precision formats without retraining\.

Despite these advancements, stable optimization of MLLMs under ultra\-low precision remains an open challenge due to dynamically evolving activation distributions and strong cross\-modal interactions\. Existing PTQ methods rely on static calibration that cannot adapt to shifting activations, while QAT approaches fail to model reconstruction under extreme bit constraints\. These limitations are heavily amplified by the dynamic, modality\-dependent nature of MLLMs statistics\. To address this, we propose a unified, modality\-agnostic quantization strategy that operates directly at the activation level, correcting quantization\-induced distortions via differentiable reconstruction\. By targeting errors at the activation level, our approach eliminates complex, modality\-specific engineering while maintaining robustness under highly heterogeneous distributions\.

## 3Preliminaries

### 3\.1Standard Floating\-Point and Quantization Basics

A standard floating\-point \(F​PFP\)\[[4](https://arxiv.org/html/2608.26581#bib.bib44)\]number is represented by a sign bitss, an exponentee, and a mantissamm\. For a format withEEexponent bits andMMmantissa bits, the real\-valued representation is given by:

v=\(−1\)s×2e−bias×\(1\+m2M\),v=\(\-1\)^\{s\}\\times 2^\{e\-\\text\{bias\}\}\\times\\left\(1\+\\frac\{m\}\{2^\{M\}\}\\right\),\(1\)wherebiasis the exponent bias andm2M\\frac\{m\}\{2^\{M\}\}denotes the fractional contribution of the mantissa\. Standard uniform quantization maps a real\-valued tensorx∈ℝx\\in\\mathbb\{R\}to a discrete set of low\-precision values using a scaling factorSSand integer clipping bounds\[qmin,qmax\]\[q\_\{\\min\},q\_\{\\max\}\]:

x^=S⋅clip​\(round​\(xS\),qmin,qmax\),\\hat\{x\}=S\\cdot\\text\{clip\}\\left\(\\text\{round\}\\left\(\\frac\{x\}\{S\}\\right\),q\_\{\\min\},q\_\{\\max\}\\right\),\(2\)whereround​\(⋅\)\\text\{round\}\(\\cdot\)denotes round\-to\-nearest integer rounding\. While per\-tensor and per\-channel scaling strategies are effective for 8\-bit integer quantization \(INT8\\mathrm\{INT8\}\), they become less reliable at ultra\-low bit\-widths \(e\.g\., 4\-bit\)\. In such regimes, severe dynamic range mismatches and activation outliers lead to significant quantization error, making accurate representation difficult under fixed uniform scaling\.

### 3\.2Microscaling \(MX\) Block Specifications

To mitigate accuracy degradation in ultra\-low bit regimes, the OCP Microscaling Formats \(MX\)\[[42](https://arxiv.org/html/2608.26581#bib.bib45)\]specification introduces a block\-based scaling mechanism\. Instead of applying a single scale factor to an entire tensor, elements are partitioned into small independent blocks of sizeBB\(typicallyB=32B=32\)\. Within each block, all elements share a common scaling factor derived from a shared exponent\. Given a block of high\-precision values𝐗=\{x1,x2,…,xB\}\\mathbf\{X\}=\\\{x\_\{1\},x\_\{2\},\\dots,x\_\{B\}\\\}, the block scale is computed as:

Sblock=2⌊log2⁡\(maxi⁡\|xi\|\)⌋\.S\_\{\\text\{block\}\}=2^\{\\left\\lfloor\\log\_\{2\}\\left\(\\max\_\{i\}\|x\_\{i\}\|\\right\)\\right\\rfloor\}\.\(3\)
Each element is then normalized by the block scale and quantized into a low\-bit representation:

x~i=𝒬format​\(xiSblock\),\\tilde\{x\}\_\{i\}=\\mathcal\{Q\}\_\{\\text\{format\}\}\\left\(\\frac\{x\_\{i\}\}\{S\_\{\\text\{block\}\}\}\\right\),\(4\)where𝒬format​\(⋅\)\\mathcal\{Q\}\_\{\\text\{format\}\}\(\\cdot\)denotes element\-wise quantization to a low\-bit floating\-point or integer code\.

#### MXFP8 \(Microscaling 8\-bit Floating Point\)

The MXFP8 format defines two variants, E4M3 and E5M2, within the microscaling framework to balance precision and dynamic range\. The E4M3 variant consists of 1 sign bit, 4 exponent bits, and 3 mantissa bits, providing higher precision at the cost of a narrower dynamic range, making it well suited for representing activations\. In contrast, the E5M2 variant uses 1 sign bit, 5 exponent bits, and 2 mantissa bits, offering a wider dynamic range and improved robustness to large\-magnitude variations\.

#### MXFP4 \(Microscaling 4\-bit Floating Point\)

MXFP4 further reduces the element\-wise representation to 4 bits, typically using an E2M1 configuration \(1 sign bit, 2 exponent bits, 1 mantissa bit\)\. Due to the extremely limited precision, MXFP4 relies heavily on the shared block exponentSblockS\_\{\\text\{block\}\}to adaptively align the limited representable range with the local distribution of each tensor block\.

### 3\.3HiF4 \(HiFloat4\)

While standard microscaling formats employ a flat, single\-level block scaling factor,HiF4\\mathrm\{HiF4\}\[[34](https://arxiv.org/html/2608.26581#bib.bib46),[45](https://arxiv.org/html/2608.26581#bib.bib47)\]introduces a multi\-level hierarchical scaling paradigm designed for hardware\-efficient acceleration\. HiF4 organizes data into 64\-element blocks with 32 bits of shared metadata, resulting in an amortized overhead of 0\.5 bits per value \(4\.5 bits total per element\)\. The scaling hierarchy consists of a global base scale and two levels of binary micro\-exponents\. The global scale is represented using an unsigned 8\-bitE6M2\\mathrm\{E6M2\}floating\-point format with an exponent bias of 48, defining a coarse block\-level normalization factor:

Smacro=2E⋅\(1\.M\)\.S\_\{\\text\{macro\}\}=2^\{E\}\\cdot\(1\.M\)\.\(5\)
Fine\-grained dynamic range refinement within the block is achieved using two tiers of 1\-bit micro\-exponents,E​1​\_​8E1\\\_8\(an 8\-element vector\) andE​1​\_​16E1\\\_16\(a 16\-element vector\)\. These micro\-exponents provide localized exponent corrections over 8\-element and 4\-element sub\-groups, respectively, effectively mitigating the impact of outliers and suppressing quantization noise\.

Each individual element within the 64\-element block is encoded using a 4\-bitS1P2\\mathrm\{S1P2\}sign\-magnitude format \(1 integer bit and 2 fractional bits\), conceptually equivalent to anE​1​M​2E1M2representation\. The reconstructed value for theii\-th element \(i∈\[1,64\]i\\in\[1,64\]\) is computed as:

Vi=Smacro×2E​1​\_​8⌈i/8⌉\+E​1​\_​16⌈i/4⌉×S1P2i\.V\_\{i\}=S\_\{\\text\{macro\}\}\\times 2^\{E1\\\_8\_\{\\lceil i/8\\rceil\}\+E1\\\_16\_\{\\lceil i/4\\rceil\}\}\\times\\mathrm\{S1P2\}\_\{i\}\.\(6\)
This hierarchical formulation significantly expands the intra\-block dynamic range to 4\.81 binades while maintaining ultra\-low precision storage\. This capability is particularly critical for MLLMs; despite the use of Quantization\-Aware Training, severe activation outliers still persistently emerge in these architectures\. Consequently, the primary focus of our work is to leverage this hierarchical scaling to effectively mitigate these emergent outliers and preserve model accuracy\.

## 4Our Approach

We study quantization\-aware training \(QAT\) for multimodal LLMs under ultra\-low\-bit numerical formats\. Our goal is to enable efficient deployment using aggressive quantization schemes such as MXFP4 and HiF4 while maintaining stable multimodal generation and reasoning performance\. Therefore, we first construct a mixed\-precision QAT framework tailored for MLLMs, where different model components are assigned heterogeneous numerical formats based on their quantization sensitivity\. We then conduct a systematic diagnostic analysis to identify the dominant sources of degradation across modalities, layers, and tensor types \(weights versus activations\)\. Guided by these findings, we propose Residual Fallback Quantization \(RFQ\), an approach that mitigates activation\-induced errors in ultra\-low\-bit regimes\.

### 4\.1Exploration of Ultra\-Low\-Bit Mixed\-Precision QAT for MLLMs

We investigate quantization\-aware training of MLLMs using block\-wise floating\-point formats defined by the Open Compute Project \(OCP\) MX specification, including MXFP8 and MXFP4, together with the hierarchical HiF4 representation\. To evaluate the effectiveness of these formats in large\-scale multimodal settings, we conduct QAT experiments on two representative MLLMs: the generative video modelWan2\.2 5B\[[46](https://arxiv.org/html/2608.26581#bib.bib19)\]and the reasoning\-oriented modelQwen3\-VL 30B\[[2](https://arxiv.org/html/2608.26581#bib.bib53)\]\.

For our preliminary experiments, we adopt a mixed\-precision QAT strategy that allocates numerical formats according to the quantization sensitivity of different model components\. Specifically, the feed\-forward networks \(FFNs\) and linear projection layers within both the vision and language modules are quantized using low\-bit formats \(MXFP8, W4A8, MXFP4, or HiF4\), whereas components that are empirically more sensitive to precision reduction, such as embedding layers and the language modeling head, remain in BF16\. This design aims to maximize compression efficiency while preserving multimodal generation and reasoning capabilities\.

Our experiments reveal a clear separation in performance between 8\-bit and ultra\-low\-bit 4\-bit quantization regimes\. As shown in Table[1](https://arxiv.org/html/2608.26581#S4.T1), MXFP8 preserves performance close to the BF16 baseline after QAT\. We further observe that a mixed W4A8 configuration where weights are compressed to a 4\-bit format while activations remain in 8\-bit MXFP8 introduces only marginal additional degradation\. In contrast, uniformly quantizing both weights and activations to 4\-bit formats \(MXFP4 or HiF4\) results in substantially higher training loss\. These findings suggest that MLLMs exhibit heterogeneous quantization sensitivity across tensors, with activations appearing considerably more vulnerable to aggressive precision reduction than weights\.

Despite this careful allocation of precision, fully 4\-bit quantization still leads to significant degradation, suggesting that format selection alone cannot bridge the performance gap\. This points to the underlying activation distributions as a key source of quantization error, motivating a more fine\-grained analysis of activation behavior in the following subsection\.

Algorithm 1Residual Fallback Quantization \(RFQ\) GEMMInput:Input activationXX, weightYY, FP4 quantizer𝒬\\mathcal\{Q\}, fallback indicatorϕ⁡\(p,r\)\\phi\(p,r\), block sizes\[Ps,Qs,Rs\]\[P\_\{s\},Q\_\{s\},R\_\{s\}\]

Output:Output

ZZ
Partition

XXinto blocks

Xp,rX^\{p,r\}and

YYinto blocks

Yr,qY^\{r,q\};

for*p=0p=0to⌈P/Ps⌉−1\\lceil P/P\_\{s\}\\rceil\-1*do

for*q=0q=0to⌈Q/Qs⌉−1\\lceil Q/Q\_\{s\}\\rceil\-1*do

Zp,q←0Z^\{p,q\}\\leftarrow 0;//Initialize block accumulator

for*r=0r=0to⌈R/Rs⌉−1\\lceil R/R\_\{s\}\\rceil\-1*do

X~p,r←𝒬⁡\(Xp,r\)\\tilde\{X\}^\{p,r\}\\leftarrow\\mathcal\{Q\}\(X^\{p,r\}\);

Y~r,q←𝒬⁡\(Yr,q\)\\tilde\{Y\}^\{r,q\}\\leftarrow\\mathcal\{Q\}\(Y^\{r,q\}\);

Zp,q\+=X~p,r​Y~r,qZ^\{p,q\}\\mathrel\{\+\}=\\tilde\{X\}^\{p,r\}\\tilde\{Y\}^\{r,q\};//Base FP4 GEMM

if*ϕ⁡\(p,r\)=1\\phi\(p,r\)=1*then

Δ​Xp,r←Xp,r−X~p,r\\Delta X^\{p,r\}\\leftarrow X^\{p,r\}\-\\tilde\{X\}^\{p,r\};//Compute RFQ residual error

X^p,r←𝒬⁡\(Δ​Xp,r\)\\widehat\{X\}^\{p,r\}\\leftarrow\\mathcal\{Q\}\(\\Delta X^\{p,r\}\);//Quantize residual to FP4

Zp,q\+=X^p,r​Y~r,qZ^\{p,q\}\\mathrel\{\+\}=\\widehat\{X\}^\{p,r\}\\tilde\{Y\}^\{r,q\};//Accumulate RFQ correction

return

ZZ;

Table 1:Relative training loss increase \(%\) for the BF16 baseline after QAT on models trained with over 5B tokens\. MXFP8 and W4A8 show minimal degradation, while MXFP4 and HiF4 incur larger errors\.
### 4\.2Cross\-Modal Weights–Activation Sensitivity

We further investigate whether the degradation under ultra\-low\-bit quantization is uniformly distributed across modalities or is primarily driven by a specific modality\. In particular, we analyze the relative sensitivity of the vision and language components in both models under identical quantization configurations\. To this end, we perform a controlled sensitivity study where vision and language pathways are quantized in the same numerical formats, allowing us to attribute performance variations to each modality independently\. We conduct experiments on Wan2\.2 and Qwen3\-VL, both of which exhibit tightly coupled vision\-language interactions during generation and reasoning\. In prior work such as MBQ\[[27](https://arxiv.org/html/2608.26581#bib.bib1)\], where vision tokens are generally less sensitive than language tokens under post\-training quantization, we observe a reverse trend in our QAT setting\. Specifically, vision components are more sensitive to aggressive 4\-bit quantization \(MXFP4 and HiF4\), resulting in greater degradation in both generative quality and reasoning consistency\. This pattern is consistent on both models, although the effect is more pronounced in Wan2\.2, which relies heavily on fine\-grained visual detail reconstruction \(see Figure[2](https://arxiv.org/html/2608.26581#S4.F2)\(a\) and \(b\)\)\.

![Refer to caption](https://arxiv.org/html/2608.26581v1/sensitivity_analysis_new.png)Figure 2:Cross\-modal sensitivity analysis under ultra\-low\-bit quantization\.To better understand the source of this degradation, we analyze quantization sensitivity across weights and activations\. As shown in Figure[2](https://arxiv.org/html/2608.26581#S4.F2)\(c\) and \(d\), activation tensors exhibit substantially larger dynamic ranges and more heterogeneous distributions compared to weights\. This effect is particularly pronounced in vision\-related layers, where extreme outliers significantly expand the quantization range under FP4 precision\. As a result, the effective precision allocated to the majority of activation values is reduced, leading to distortion and frequent underflow of small\-magnitude activations, which introduces substantial quantization residuals\. In contrast, weight distributions remain comparatively compact and well\-behaved, making them more amenable to low\-bit quantization\.

Overall, these findings indicate that the dominant failure mode in ultra\-low\-bit MLLM quantization is primarily driven by activation quantization errors shared in both language and vision pathways\. Since both modalities are subject to severe precision constraints in activation propagation, weight\-only mitigation strategies are insufficient to recover the resulting information loss\. This observation motivates the need for activation\-centric correction mechanisms\. To this end, we propose Residual Fallback Quantization \(RFQ\)\. Inspired by the block\-level fallback mechanism of\[[57](https://arxiv.org/html/2608.26581#bib.bib17)\], RFQ extends this idea to ultra\-low\-bit QAT by re\-quantizing activation residuals through an efficient FP4\-to\-FP8 pathway, thereby improving accuracy while maintaining the computational efficiency of FP4 operations\.

### 4\.3Residual Fallback Quantization \(RFQ\)

Motivated by our observation that activation quantization constitutes the dominant source of degradation in ultra\-low\-bit MLLMs, we propose*Residual Fallback Quantization \(RFQ\)*, a residual\-based correction mechanism that improves activation reconstruction while preserving the efficiency of uniform low\-bit computation\.

Consider a matrix multiplication operation

whereX∈ℝP×RX\\in\\mathbb\{R\}^\{P\\times R\}denotes input activations andY∈ℝR×QY\\in\\mathbb\{R\}^\{R\\times Q\}represents the weights\. Following a block\-wise execution scheme, we partitionXXinto blocksXp,r∈ℝPs×RsX^\{p,r\}\\in\\mathbb\{R\}^\{P\_\{s\}\\times R\_\{s\}\}andYYinto blocksYr,q∈ℝRs×QsY^\{r,q\}\\in\\mathbb\{R\}^\{R\_\{s\}\\times Q\_\{s\}\}\.

For each activation and weight block, we first perform conventional low\-bit quantization using a target FP4 quantizer𝒬⁡\(⋅\)\\mathcal\{Q\}\(\\cdot\):

X~p,r=𝒬⁡\(Xp,r\),Y~r,q=𝒬⁡\(Yr,q\)\.\\tilde\{X\}^\{p,r\}=\\mathcal\{Q\}\(X^\{p,r\}\),\\qquad\\tilde\{Y\}^\{r,q\}=\\mathcal\{Q\}\(Y^\{r,q\}\)\.\(8\)
Although this base FP4 approximation is effective for the majority of tokens, severe activation outliers introduce substantial reconstruction errors\. To selectively compensate for these errors without incurring global overhead, RFQ introduces a hardware\-friendly fallback indicatorϕ⁡\(p,r\)∈\{0,1\}\\phi\(p,r\)\\in\\\{0,1\\\}, which identifies activation blocks containing significant outlier magnitudes\. For blocks flagged by the indicator \(ϕ⁡\(p,r\)=1\\phi\(p,r\)=1\), RFQ computes the quantization residual:

Δ​Xp,r=Xp,r−X~p,r\.\\Delta X^\{p,r\}=X^\{p,r\}\-\\tilde\{X\}^\{p,r\}\.\(9\)
Rather than storing this residual in a costly higher\-precision format, we quantize it using the same underlying FP4 representation:

X^p,r=𝒬⁡\(Δ​Xp,r\)\.\\widehat\{X\}^\{p,r\}=\\mathcal\{Q\}\(\\Delta X^\{p,r\}\)\.\(10\)
The final output blockZp,qZ^\{p,q\}is obtained by conditionally incorporating the residual correction term alongside the base GEMM computation:

Zp,q=∑rX~p,r​Y~r,q\+∑rϕ⁡\(p,r\)⋅\(X^p,r​Y~r,q\)\.Z^\{p,q\}=\\sum\_\{r\}\\tilde\{X\}^\{p,r\}\\tilde\{Y\}^\{r,q\}\+\\sum\_\{r\}\\phi\(p,r\)\\cdot\\left\(\\widehat\{X\}^\{p,r\}\\tilde\{Y\}^\{r,q\}\\right\)\.\(11\)
Equivalently, RFQ can be interpreted as dynamically approximating sensitive activation blocks using a two\-stage low\-bit decomposition:

Xp,r≈X~p,r\+ϕ⁡\(p,r\)⋅X^p,r,X^\{p,r\}\\approx\\tilde\{X\}^\{p,r\}\+\\phi\(p,r\)\\cdot\\widehat\{X\}^\{p,r\},\(12\)where both terms are restricted to the same low\-bit numerical format\. The first term captures the dominant signal component across all blocks, while the conditional residual term recovers precision lost to severe outliers only where strictly necessary\.

![Refer to caption](https://arxiv.org/html/2608.26581v1/Figures/rfq_two_stage_histogram_comparison.png)Figure 3:Comparative analysis of activation tail tracking and fallback interception across model modalities\. Top Row highlights the unmitigated quantization noise and overflow elements in standard MXFP4 compared against theB​F​16BF16baseline\. Bottom Row demonstrates how Residual Fallback Quantization \(RFQ\) restores fidelity by snapping extreme outliers\.Compared with global double\-quantization alternatives, RFQ preserves the advantages of uniform low\-bit arithmetic and triggers additional residual accumulation of GEMM whenϕ⁡\(p,r\)=1\\phi\(p,r\)=1\. Unlike traditional fallback frameworks that explicitly load pre\-computed residual tensors from off\-chip DRAM inside the execution loop \(e\.g\.,u⁡\(i,k\)u\(i,k\)in baseline FQ\[[13](https://arxiv.org/html/2608.26581#bib.bib54),[57](https://arxiv.org/html/2608.26581#bib.bib17)\]\), RFQ operates as a streaming kernel that evaluates both the residual errorΔ​Xp,r\\Delta X^\{p,r\}and its subsequent quantization of FP4X^p,r\\widehat\{X\}^\{p,r\}completely on\-the\-fly\. Furthermore, by restricting both the base and residual pathways to the exact same uniform FP4 format𝒬⁡\(⋅\)\\mathcal\{Q\}\(\\cdot\)rather than heterogeneous integer types, RFQ minimizes hardware execution paths\. Crucially, this residual correction mechanism is applied exclusively during the forward pass to avoid compounding memory and gradient overhead during backward propagation, keeping the implementation fully compatible with existing low\-bit hardware tensor primitives and standard quantization\-aware training infrastructure\. The complete execution flow is summarized in Algorithm[1](https://arxiv.org/html/2608.26581#algorithm1)and the RFQ effect is illustrated in Figure[3](https://arxiv.org/html/2608.26581#S4.F3)\.

Table 2:Parameter breakdown \(%\) by functional component with respect to the total paramter size\. Quant\. \(%\) for the visual and textual modules denotes the proportion of parameters quantized within each module\. Projection layers and mergers are included in the “Other” category\.

## 5Experimental Analysis

Table 3:For Wan2\.2, quantitative comparison on VBench\. Values in parentheses indicate the relative change \(%\) with respect to the BF16 baseline\. Blue denotes improvement and red denotes degradation\.Table 4:Accuracy comparison across different datasets and quantization formats on Qwen3\-VL\. Values in parentheses indicate the absolute change with respect to the BF16 baseline\. Blue denotes improvement, while red indicates degradation\.### 5\.1Experimental Setup

We evaluate our proposed RFQ framework under ultra\-low\-bit precision constraints on two representative multimodal architectures: Wan2\.2\-5B and Qwen3VL\-30B\. For both models, we initialize from publicly available pretrained checkpoints and subsequently perform low\-precision quantization\-aware fine\-tuning\. For the Qwen3\-VL supervised fine\-tuning \(SFT\) phase, we utilize the concept\-balanced CC\-3M dataset comprising 595K samples\[[30](https://arxiv.org/html/2608.26581#bib.bib55),[31](https://arxiv.org/html/2608.26581#bib.bib56)\]\. For Wan2\.2, we leverage a subset of the OpenVid\-1M dataset\[[36](https://arxiv.org/html/2608.26581#bib.bib57)\], specifically utilizing Part 1 that contains approximately 26,000 video\-text pairs\. Following the mixed\-precision scheme outlined in Section[4](https://arxiv.org/html/2608.26581#S4), the embedding layers and the final language modeling head are maintained in BF16 precision, whereas all remaining linear, MoE and attention projection layers are quantized using MXFP8, W4A8, MXFP4, or HiF4 formats\. See Table\.[2](https://arxiv.org/html/2608.26581#S4.T2)for the quantized parameter count\. To ensure a fair and rigorous comparison across these diverse numerical configurations, both models are fine\-tuned under identical quantization hyperparameters for approximately 5 billion tokens\. See supplemental for more details\.

For both Wan2\.2 and Qwen3\-VL, we performed SFT using AdamW starting from their respective publicly available pretrained checkpoints\. Wan2\.2 is fine\-tuned with a learning rate of1×10−51\\times 10^\{\-5\}, while Qwen3\-VL uses a learning rate of1×10−71\\times 10^\{\-7\}\. Unless otherwise specified, all other training hyperparameters remain unchanged in different quantization configurations to ensure fair comparisons\. We evaluated Wan2\.2 using VBench on 100 randomly sampled prompts from the MovieGen\[[40](https://arxiv.org/html/2608.26581#bib.bib22)\]benchmark and report downstream performance on different dimensions\. For Qwen3\-VL, we assess multimodal reasoning capabilities on four widely used benchmarks: RealWorldQA\[[49](https://arxiv.org/html/2608.26581#bib.bib48)\], MMStar\[[9](https://arxiv.org/html/2608.26581#bib.bib49)\], MMBench\-EN\[[33](https://arxiv.org/html/2608.26581#bib.bib50)\], and SimpleVQA\[[11](https://arxiv.org/html/2608.26581#bib.bib51)\]\. We report the VQA accuracy on each benchmark\. All evaluations are conducted using the same inference protocol for both BF16 and quantized models\.

### 5\.2Evaluation Performance

![Refer to caption](https://arxiv.org/html/2608.26581v1/qualatative_results.png)Figure 4:Qualitative results for video generation\. While 4\-bit quantization methods \(*i\.e\.*, MXFP4 and HiF4\) degrade generation quality, our proposed RFQ recovers visual fidelity and improves temporal consistency\.Performance of video generation\.Table[3](https://arxiv.org/html/2608.26581#S5.T3)presents quantitative results on Wan2\.2 evaluated using VBench\. Compared with standard MXFP4 quantization, RFQ consistently improves multiple video generation metrics\. In particular, RFQ improves the Dynamic Degree from 43\.00 to 50\.00 while simultaneously improving Aesthetic Quality and maintaining competitive Subject and Background Consistency\. Similar trends are observed in the HiF4 setting, where RFQ improves Subject Consistency from 95\.23 to 95\.86 and Aesthetic Quality from 59\.44 to 59\.54\. We also include qualitative results in Figure[4](https://arxiv.org/html/2608.26581#S5.F4)\. Seesupplementalfor more results\. These results demonstrate that RFQ effectively mitigates the adverse effects of activation quantization in ultra\-low\-bit video generation models\.

Performance of multimodal reasoning\.Table[4](https://arxiv.org/html/2608.26581#S5.T4)reports the results of Qwen3\-VL\. In MXFP4 quantization, RFQ consistently narrows the performance gap with respect to the baseline BF16, improving RealWorldQA from 70\.98 to 72\.16, MMStar from 69\.67 to 70\.73, and SimpleVQA from 15\.16 to 16\.44\. Under the HiF4 configuration, RFQ achieves further gains, improving RealWorldQA from 72\.42 to 72\.81 \(even better than BF16 baseline\) and MMStar from 71\.27 to 71\.47, while recovering the BF16 performance in MMBench\-EN\. These findings indicate that the RFQ generalizes across various multimodal reasoning tasks and effectively improves the robustness of ultra\-low\-bit MLLMs\.

![Refer to caption](https://arxiv.org/html/2608.26581v1/fp4-curves.png)Figure 5:Training loss curves comparing low\-bit FP4 strategies against the BF16 baseline across Qwen3\-VL\-30B and Wan2\.2\-5B\.In addition to downstream performance, Table[5](https://arxiv.org/html/2608.26581#S5.T5)reports the relative training loss difference with respect to the BF16 baseline\. RFQ consistently enhances both generative and reasoning performance under aggressive FP4 quantization schemes, leading to improvements in both SFT and downstream evaluation\. Notably, these gains are obtained without increasing the precision of the underlying representation, demonstrating the effectiveness of residual fallback correction in alleviating activation\-induced quantization errors\.

Table 5:Relative training loss increase \(%\) over the BF16 baseline after QAT with 4\-bit quantization\. RFQ substantially mitigates the degradation caused by MXFP4 and HiF4\.

## 6Conclusion

In this paper, we present a systematic study of ultra\-low\-bit supervised fine\-tuning \(SFT\) for multimodal LLMs\. Our analysis shows that visual modules are generally more sensitive to aggressive quantization than language modules, though quantizing language modules also leads to non\-negligible performance degradation\. These findings suggest that effective accuracy recovery in ultra\-low\-bit settings requires jointly addressing both components\. Based on this observation, we propose RFQ, a simple yet effective approach that exploits residual quantization errors through an auxiliary uniform quantization pathway to compensate for information loss under ultra\-low\-bit quantization\. Extensive experiments demonstrate that RFQ consistently improves training dynamics and downstream performance, recovering accuracy close to the BF16 baseline\. Overall, our results highlight the promise of residual\-based compensation for efficient deployment of multimodal foundation models under tight memory and computation constraints\. In future work, we will explore more aggressive quantization\-aware training settings, including 2\-bit quantization, and extend RFQ to broader multimodal architectures and tasks\.

## References

- \[1\]S\. Agashe, J\. Han, S\. Gan, J\. Yang, A\. Li, and X\. E\. Wang\(2025\)Agent s: an open agentic framework that uses computers like a human\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1)\.
- \[2\]S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. Zhu\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§4\.1](https://arxiv.org/html/2608.26581#S4.SS1.p1.1)\.
- \[3\]S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin\(2025\)Qwen2\.5\-vl technical report\.External Links:2502\.13923Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p3.1)\.
- \[4\]G\. Barrett\(1989\)Formal methods applied to a floating\-point number system\.IEEE transactions on software engineering\.Cited by:[§3\.1](https://arxiv.org/html/2608.26581#S3.SS1.p1.1)\.
- \[5\]S\. Bhatnagar, A\. Xu, K\. Tan, and N\. Ahuja\(2025\)LUQ: layerwise ultra\-low bit quantization for multimodal large language models\.arXiv preprint arXiv:2509\.23729\.Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1),[§2](https://arxiv.org/html/2608.26581#S2.p3.1)\.
- \[6\]Y\. Bondarenko, M\. Nagel, and T\. Blankevoort\(2023\)Quantizable transformers: removing outliers by helping attention heads do nothing\.Advances in Neural Information Processing Systems36,pp\. 75067–75096\.Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p2.1)\.
- \[7\]D\. Campbell, S\. Rane, T\. Giallanza, N\. De Sabbata, K\. Ghods, A\. Joshi, A\. Ku, S\. M\. Frankland, T\. L\. Griffiths, J\. D\. Cohen, and T\. Webb\(2024\)Understanding the limits of vision language models through the lens of the binding problem\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1)\.
- \[8\]H\. Chen, C\. Lv, L\. Ding, H\. Qin, X\. Zhou, Y\. Ding, X\. Liu, M\. Zhang, J\. Guo, X\. Liu,et al\.\(2024\)Db\-llm: accurate dual\-binarization for efficient llms\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 8719–8730\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p2.1)\.
- \[9\]L\. Chen, J\. Li, X\. Dong, P\. Zhang, Y\. Zang, Z\. Chen, H\. Duan, J\. Wang, Y\. Qiao, D\. Lin,et al\.\(2024\)Are we on the right way for evaluating large vision\-language models?\.arXiv preprint arXiv:2403\.20330\.Cited by:[§5\.1](https://arxiv.org/html/2608.26581#S5.SS1.p2.1)\.
- \[10\]M\. Chen, W\. Shao, P\. Xu, J\. Wang, P\. Gao, K\. Zhang, and P\. Luo\(2025\)Efficientqat: efficient quantization\-aware training for large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10081–10100\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p2.1)\.
- \[11\]X\. Cheng, W\. Zhang, S\. Zhang, J\. Yang, X\. Guan, X\. Wu, X\. Li, G\. Zhang, J\. Liu, Y\. Mai, Y\. Zeng, Z\. Wen, K\. Jin, B\. Wang, W\. Zhou, Y\. Lu, T\. Li, W\. Huang, and Z\. Li\(2025\)SimpleVQA: multimodal factuality evaluation for multimodal large language models\.Cited by:[§5\.1](https://arxiv.org/html/2608.26581#S5.SS1.p2.1)\.
- \[12\]H\. W\. Chung, X\. Garcia, A\. Roberts, Y\. Tay, O\. Firat, S\. Narang, and N\. ConstantUniMax: fairer and more effective language sampling for large\-scale multilingual pretraining\.InThe Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p3.1)\.
- \[13\]T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer\(2022\)LLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.arXiv preprint arXiv:2208\.07339\.Cited by:[§4\.3](https://arxiv.org/html/2608.26581#S4.SS3.p8.1)\.
- \[14\]D\. Du, Y\. Zhang, S\. Cao, J\. Guo, T\. Cao, X\. Chu, and N\. Xu\(2024\)Bitdistiller: unleashing the potential of sub\-4\-bit llms via self\-distillation\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 102–116\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p2.1)\.
- \[15\]C\. Feng, G\. Zhou, X\. Wu, and Q\. Zhao\(2025\)PLMQ: piecewise linear mixed\-precision quantization for deep neural networks\.Neurocomputing,pp\. 131000\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p1.1)\.
- \[16\]E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\(2022\)Gptq: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p1.1)\.
- \[17\]E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh\(2023\)OPTQ: accurate quantization for generative pre\-trained transformers\.InThe Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1)\.
- \[18\]L\. He, S\. Zheng, K\. Sun, Y\. Liu, Y\. Zhao, C\. Tan, H\. Yang, Y\. Du, and L\. Du\(2025\)BASE\-q: bias and asymmetric scaling enhanced rotational quantization for large language models\.arXiv preprint arXiv:2506\.15689\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p1.1)\.
- \[19\]C\. Hooper, S\. Kim, H\. Mohammadzadeh, H\. Genc, K\. Keutzer, A\. Gholami, and Y\. Sophia Shao\(2025\)Speed: speculative pipelined execution for efficient decoding\.InEnhancing LLM Performance: Efficacy, Fine\-Tuning, and Inference Techniques,pp\. 19–32\.Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1)\.
- \[20\]L\. Hu, W\. Xiao, X\. Chen, X\. Xu, B\. Xu, K\. Li, and Y\. Tao\(2026\)MASQuant: modality\-aware smoothing quantization for multimodal large language models\.External Links:2603\.04800Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p3.1),[§1](https://arxiv.org/html/2608.26581#S1.p4.1),[§2](https://arxiv.org/html/2608.26581#S2.p3.1)\.
- \[21\]C\. Jia, B\. Li, X\. Zhang, M\. Wei, B\. Lin, and H\. Sun\(2026\)Quant experts: token\-aware adaptive error reconstruction with mixture of experts for large vision\-language models quantization\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 24716–24726\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p3.1)\.
- \[22\]J\. Kil, Z\. Mai, J\. Lee, A\. Chowdhury, Z\. Wang, K\. Cheng, L\. Wang, Y\. Liu, and W\. Chao\(2024\)MLLM\-compbench: a comparative reasoning benchmark for multimodal llms\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1)\.
- \[23\]R\. Krishnamoorthi\(2018\)Quantizing deep convolutional networks for efficient inference: a whitepaper\.arXiv preprint arXiv:1806\.08342\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p1.1)\.
- \[24\]H\. Laurençon, L\. Tronchon, M\. Cord, and V\. Sanh\(2024\)What matters when building vision\-language models?\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1)\.
- \[25\]C\. Lee, J\. Jin, T\. Kim, H\. Kim, and E\. Park\(2024\)Owq: outlier\-aware weight quantization for efficient fine\-tuning and inference of large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 13355–13364\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p1.1)\.
- \[26\]S\. Li, Y\. Hu, X\. Ning, X\. Liu, K\. Hong, X\. Jia, X\. Li, Y\. Yan, P\. Ran, G\. Dai,et al\.\(2025\)Mbq: modality\-balanced quantization for large vision\-language models\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 4167–4177\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p3.1)\.
- \[27\]S\. Li, Y\. Hu, X\. Ning, X\. Liu, K\. Hong, X\. Jia, X\. Li, Y\. Yan, P\. Ran, G\. Dai, S\. Yan, H\. Yang, and Y\. Wang\(2025\)MBQ: modality\-balanced quantization for large vision\-language models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§4\.2](https://arxiv.org/html/2608.26581#S4.SS2.p1.1)\.
- \[28\]J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han\(2024\)Awq: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of machine learning and systems6,pp\. 87–100\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p1.1)\.
- \[29\]J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han\(2024\)AWQ: activation\-aware weight quantization for on\-device llm compression and acceleration\.InProceedings of Machine Learning and Systems,P\. Gibbons, G\. Pekhimenko, and C\. D\. Sa \(Eds\.\),Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1)\.
- \[30\]H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee\(2023\)Improved baselines with visual instruction tuning\.arXiv:2310\.03744\.Cited by:[§5\.1](https://arxiv.org/html/2608.26581#S5.SS1.p1.1)\.
- \[31\]H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee\(2023\)Visual instruction tuning\.InNeurIPS,Cited by:[§5\.1](https://arxiv.org/html/2608.26581#S5.SS1.p1.1)\.
- \[32\]Y\. Liu, J\. Wen, Y\. Wang, S\. Ye, L\. L\. Zhang, T\. Cao, C\. Li, and M\. Yang\(2024\)Vptq: extreme low\-bit vector post\-training quantization for large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8181–8196\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p2.1)\.
- \[33\]Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.\(2024\)Mmbench: is your multi\-modal model an all\-around player?\.InEuropean conference on computer vision,pp\. 216–233\.Cited by:[§5\.1](https://arxiv.org/html/2608.26581#S5.SS1.p2.1)\.
- \[34\]Y\. Luo, J\. Huang, Y\. Cheng, Z\. Yu, K\. Tang, X\. Ma, X\. Wang, A\. Tong, G\. Hu, Y\. Xu,et al\.\(2026\)Hifloat4 format for language model inference\.arXiv preprint arXiv:2602\.11287\.Cited by:[§3\.3](https://arxiv.org/html/2608.26581#S3.SS3.p1.1)\.
- \[35\]S\. Ma, H\. Wang, S\. Huang, X\. Zhang, Y\. Hu, T\. Song, Y\. Xia, and F\. Wei\(2025\)BitNet b1\. 58 2b4t technical report\.arXiv preprint arXiv:2504\.12285\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p2.1)\.
- \[36\]K\. Nan, R\. Xie, P\. Zhou, T\. Fan, Z\. Yang, Z\. Chen, X\. Li, J\. Yang, and Y\. Tai\(2024\)OpenVid\-1m: a large\-scale high\-quality dataset for text\-to\-video generation\.arXiv preprint arXiv:2407\.02371\.Cited by:[§5\.1](https://arxiv.org/html/2608.26581#S5.SS1.p1.1)\.
- \[37\]NVIDIA\(2024\)TransformerEngine: an efficient library for training transformer models\.Note:[https://github\.com/NVIDIA/TransformerEngine](https://github.com/NVIDIA/TransformerEngine)Accessed: 2026\-05\-05Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p2.1)\.
- \[38\]A\. Panferov, J\. Chen, S\. Tabesh, R\. L\. Castro, M\. Nikdan, and D\. Alistarh\(2025\)Quest: stable training of llms with 1\-bit weights and activations\.arXiv preprint arXiv:2502\.05003\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p2.1)\.
- \[39\]H\. Peng, K\. Wu, Y\. Wei, G\. Zhao, Y\. Yang, Z\. Liu, Y\. Xiong, Z\. Yang, B\. Ni, J\. Hu, R\. Li, M\. Zhang, C\. Li, J\. Ning, R\. Wang, Z\. Zhang, S\. Liu, J\. Chau, H\. Hu, and P\. Cheng\(2023\)FP8\-lm: training fp8 large language models\.External Links:2310\.18313Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p2.1)\.
- \[40\]A\. Polyak, A\. Zohar, A\. Brown, A\. Tjandra, A\. Sinha, A\. Lee, A\. Vyas, B\. Shi, C\. Ma, C\. Chuang,et al\.\(2024\)Movie gen: a cast of media foundation models\.arXiv preprint arXiv:2410\.13720\.Cited by:[§5\.1](https://arxiv.org/html/2608.26581#S5.SS1.p2.1)\.
- \[41\]X\. Qu, D\. Aponte, C\. Banbury, D\. P\. Robinson, T\. Ding, K\. Koishida, I\. Zharkov, and T\. Chen\(2025\)Automatic joint structured pruning and quantization for efficient neural network training and compression\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 15234–15244\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p2.1)\.
- \[42\]B\. D\. Rouhani, R\. Zhao, A\. More, M\. Hall, A\. Khodamoradi, S\. Deng, D\. Choudhary, M\. Cornea, E\. Dellinger, K\. Denolf,et al\.\(2023\)Microscaling data formats for deep learning\.arXiv preprint arXiv:2310\.10537\.Cited by:[§3\.2](https://arxiv.org/html/2608.26581#S3.SS2.p1.1)\.
- \[43\]H\. Shao, S\. Qian, H\. Xiao, G\. Song, Z\. Zong, L\. Wang, Y\. Liu, and H\. Li\(2024\)Visual cot: advancing multi\-modal language models with a comprehensive dataset and benchmark for chain\-of\-thought reasoning\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1)\.
- \[44\]A\. Szot, B\. Mazoure, H\. Agrawal, R\. D\. Hjelm, Z\. Kira, and A\. T\. Toshev\(2024\)Grounding multimodal large language models in actions\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1)\.
- \[45\]M\. Taghian, Y\. Peng, X\. Huang, Y\. Wang, Y\. Wang, W\. Guo, Y\. Luo, T\. Hu, J\. Wang, X\. Wang,et al\.\(2026\)HiFloat4 format for language model pre\-training on ascend npus\.arXiv preprint arXiv:2604\.08826\.Cited by:[§3\.3](https://arxiv.org/html/2608.26581#S3.SS3.p1.1)\.
- \[46\]T\. Wan, A\. Wang, B\. Ai, B\. Wen, C\. Mao, C\. Xie, D\. Chen, F\. Yu, H\. Zhao, J\. Yang, J\. Zeng, J\. Wang, J\. Zhang, J\. Zhou, J\. Wang, J\. Chen, K\. Zhu, K\. Zhao, K\. Yan, L\. Huang, M\. Feng, N\. Zhang, P\. Li, P\. Wu, R\. Chu, R\. Feng, S\. Zhang, S\. Sun, T\. Fang, T\. Wang, T\. Gui, T\. Weng, T\. Shen, W\. Lin, W\. Wang, W\. Wang, W\. Zhou, W\. Wang, W\. Shen, W\. Yu, X\. Shi, X\. Huang, X\. Xu, Y\. Kou, Y\. Lv, Y\. Li, Y\. Liu, Y\. Wang, Y\. Zhang, Y\. Huang, Y\. Li, Y\. Wu, Y\. Liu, Y\. Pan, Y\. Zheng, Y\. Hong, Y\. Shi, Y\. Feng, Z\. Jiang, Z\. Han, Z\. Wu, and Z\. Liu\(2025\)Wan: open and advanced large\-scale video generative models\.External Links:2503\.20314Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p3.1),[§4\.1](https://arxiv.org/html/2608.26581#S4.SS1.p1.1)\.
- \[47\]C\. Wang, Z\. Wang, X\. Xu, Y\. Tang, J\. Zhou, and J\. Lu\(2024\)Q\-vlm: post\-training quantization for large vision\-language models\.Advances in Neural Information Processing Systems37,pp\. 114553–114573\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p3.1)\.
- \[48\]R\. Wang, Y\. Gong, X\. Liu, G\. Zhao, Z\. Yang, B\. Guo, Z\. Zha, and P\. CHENG\(2025\)Optimizing large language model training using FP4 quantization\.InForty\-second International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p4.1)\.
- \[49\]xAI\(2024\)RealWorldQA: a benchmark for real\-world spatial understanding\.Note:[https://huggingface\.co/datasets/xai\-org/RealworldQA](https://huggingface.co/datasets/xai-org/RealworldQA)Accessed: 2025\-04\-26Cited by:[§5\.1](https://arxiv.org/html/2608.26581#S5.SS1.p2.1)\.
- \[50\]H\. Xi, H\. Cai, L\. Zhu, Y\. Lu, K\. Keutzer, J\. Chen, and S\. Han\(2025\)COAT: compressing optimizer states and activations for memory\-efficient FP8 training\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1),[§1](https://arxiv.org/html/2608.26581#S1.p2.1)\.
- \[51\]G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han\(2023\)Smoothquant: accurate and efficient post\-training quantization for large language models\.InInternational conference on machine learning,pp\. 38087–38099\.Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p4.1),[§2](https://arxiv.org/html/2608.26581#S2.p1.1)\.
- \[52\]Z\. Xu, S\. Sharify, and H\. Mostafa\(2026\)MF\-qat: multi\-format quantization\-aware training for elastic inference\.arXiv preprint arXiv:2604\.00529\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p3.1)\.
- \[53\]Y\. Xue, Y\. Huang, J\. Shao, and J\. Zhang\(2025\)Vlmq: efficient post\-training quantization for large vision\-language models via hessian augmentation\.arXiv preprint arXiv:2508\.03351\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p3.1)\.
- \[54\]J\. Yu, S\. Zhou, D\. Yang, S\. Li, S\. Wang, X\. Hu, C\. Xu, Z\. Xu, C\. Shu, and Z\. Yuan\(2025\)Mquant: unleashing the inference potential of multimodal large language models via static quantization\.InProceedings of the 33rd ACM International Conference on Multimedia,pp\. 1783–1792\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p3.1)\.
- \[55\]M\. Zaheer, G\. Guruganesh, K\. A\. Dubey, J\. Ainslie, C\. Alberti, S\. Ontanon, P\. Pham, A\. Ravula, Q\. Wang, L\. Yang, and A\. Ahmed\(2020\)Big bird: transformers for longer sequences\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),pp\. 17283–17297\.Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p1.1)\.
- \[56\]P\. Zhang, M\. Noto, W\. Tan, C\. Jiang, W\. Lin, W\. Zhou, and H\. Zhang\(2026\)Attn\-qat: 4\-bit attention with quantization\-aware training\.arXiv preprint arXiv:2603\.00040\.Cited by:[§2](https://arxiv.org/html/2608.26581#S2.p3.1)\.
- \[57\]P\. Zhang, J\. Wei, J\. Zhang, J\. Zhu, and J\. Chen\(2025\)Accurate int8 training through dynamic block\-level fallback\.arXiv preprint arXiv:2503\.08040\.Cited by:[§1](https://arxiv.org/html/2608.26581#S1.p4.1),[§1](https://arxiv.org/html/2608.26581#S1.p5.1),[§4\.2](https://arxiv.org/html/2608.26581#S4.SS2.p3.1),[§4\.3](https://arxiv.org/html/2608.26581#S4.SS3.p8.1)\.