Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
Summary
This paper explains why fully quantizing the recurrent Gated DeltaNet layers in hybrid LLMs to 4-bit NVFP4 preserves accuracy across long-context benchmarks, detailing mechanisms like outlier localization and robust delta-rule dynamics.
View Cached Full Text
Cached at: 09/04/26, 03:58 AM
Paper page - Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
Source: https://huggingface.co/papers/2609.04098
Abstract
Fully quantizing hybrid LLMs—including recurrent Gated DeltaNet layers—to 4-bit NVFP4 preserves accuracy across long-context and reasoning benchmarks by localizing outliers and exploiting robust delta-rule dynamics.
Hybrid LLMs pairsoftmax attentionwithlinear-attentionlayers such asGated DeltaNet(GDN), whose recurrent state summarizes the context in fixed size. Early community4-bit quantizations of Qwen3.8-27B (48 GDN layers, 16 attention layers) left the GDN block in 8- or 16-bit precision -- especially its decay and write-strength gates -- on the intuition that errors in a recurrence accumulate over long contexts. We test that intuition by building Minima:NVFP4W4A4on all 496 linear layers, GDN included. Across perplexity at 4K/32K, MMLU-Pro, GSM8K, AIME’25, GPQA-Diamond, LiveCodeBench, and RULER retrieval to 64K, Minima matches BF16 within seed noise (5-task average -0.52) while being the smallest (17.5 GiB) and fastest-prefill (+14-19%) recipe we compare, and its 32K perplexity gap shrinks with position. A four-part mechanism study explains why: (i)NVFP4’s 16-element block scaling localizes the residual stream’s extreme outliers, equalizing activation error across layer roles; (ii) the supposedly fragile gate projections are the least sensitive -- softplus/exponential and sigmoid parameterizations compress ~11%GEMMerror to ~2% output error; (iii) thedelta-rule recurrenceholds injected noise at a flat plateau over 32K tokens and forgets a state impulse within hundreds of steps, because each write overwrites the state along the current key direction; (iv) the per-token quantization cost washes out with context instead of compounding. We also repair a global-scale mismatch that arises when per-module-calibratedNVFP4checkpoints are served by kernels that fuse those modules into oneGEMM, and show calibratedFP8KV-cachescales are performance-free. The result: a practical recipe -- quantize everything, ship KV scales -- and a mechanistic account of why the recurrent half of a hybrid LLM is the easy half to quantize. Checkpoint: https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2609\.04098
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### minima-ai/mnma_qwen3.8_27b_nvfp4 Text Generation• 27B• Updatedabout 2 hours ago • 2
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.04098 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.04098 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
The paper proposes STEPQuant, a spatial-temporal post-training quantization framework for delta-rule recurrent states in linear attention models, achieving near-FP32 accuracy at 6-bit precision, over 5x state compression, and up to 68.7% serving memory reduction when integrated into SGLang.
The 4-Bitter Lesson: Balancing Stability and Performance in NVFP4 RL
This article presents a recipe for low-precision (NVFP4) RL training that balances throughput and stability, addressing issues from forward and backward pass quantization errors.
High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
This paper identifies that low-bit KV cache quantization degrades LLM accuracy due to structured local misranking of logits, and proposes DGAP, a method that restores the local distribution of top-K candidates, recovering RULER accuracy from 47.8% to 83.2% on Llama-3.1-8B with minimal overhead.
Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floor
This paper decomposes MXFP4 quantization error into three additive components—scale bias, deadzone truncation, and grid noise—and proposes targeted corrections that recover BF16 accuracy to within 0.7 pp on Qwen2.5-3B and 3.0 pp on Qwen3-30B-A3B-Base for LLM reinforcement learning post-training.
Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs
The paper identifies activation quantization as the primary bottleneck in ultra-low-bit quantized multimodal LLMs and proposes ResidualFallbackQuantization (RFQ) to recover performance with minimal overhead.