Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
Summary
This paper identifies a fundamental limitation (shrinkage bias) in non-uniform FP4 quantization formats for LLM pretraining and proposes UFP4, a uniform 4-bit training recipe that outperforms existing E2M1-based methods.
View Cached Full Text
Cached at: 06/20/26, 02:27 PM
Paper page - Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
Source: https://huggingface.co/papers/2606.20381 Authors:
,
,
,
,
,
,
,
,
,
,
Abstract
Uniform 4-bit training with RHT-based quantization outperforms E2M1-based methods by eliminating shrinkage bias and improving training stability across large language model architectures.
FP4 trainingpromises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered onE2M1data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such asE2M1inherently suffer fromShrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across layers and is amplified by theRandom Hadamard Transform(RHT), providing a unified explanation for thetraining instabilityobserved in existingE2M1-based FP4 recipes. In contrast,uniform grids(E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higherquantization quality. Based on this finding, we propose UFP4, a uniform 4-bit training recipe that applies RHT to all three trainingGEMMswhile restricting stochastic rounding to dY alone. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strongE2M1-based baselines, supported byscaling-law analysisandablation studies. Our results suggest that future accelerators should supportE1M2/INT4-style uniform 4-bit grids as first-class training primitives alongsideE2M1.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.20381
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.20381 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.20381 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.20381 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference
SharQ introduces a training-free method combining activation sparsity and FP4 quantization for LLM inference, using sparse-dense decomposition and a unified FP4 weight payload. It achieves significant latency reduction and accuracy recovery over FP4-only baselines.
Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floor
This paper decomposes MXFP4 quantization error into three additive components—scale bias, deadzone truncation, and grid noise—and proposes targeted corrections that recover BF16 accuracy to within 0.7 pp on Qwen2.5-3B and 3.0 pp on Qwen3-30B-A3B-Base for LLM reinforcement learning post-training.
@ArkadiiBessonov: Three main ways to do FP8 in LLM pretraining — and they differ in mainly one thing: how the scale is attached. per-tens…
Explains three main approaches to FP8 scaling in LLM pretraining—per-tensor, blockwise, and MXFP8—focusing on how the scale is attached, and derives tile geometries from the constraint that scale must remain constant along the matmul's contracted dimension.
Format-Aware Fusion for Fast FP4 Pretraining
提出了 format-aware fusion 方法，将 FP4 量化中的 scale 计算、操作数打包与反向状态保存与 GEMM 融合，在 Llama-3 8B 预训练中达到最高 37.9K tokens/s/GPU，MXFP4 路线在略高于 bfloat16 训练损失终点的情况下实现 37.2K tokens/s/GPU（86.3% MFU），显著超过 Transformer Engine 的 27.6K。
@VukRosic99: NVFP4 end-to-end training diverges. Current recipes patch around it with Hadamard transforms, stochastic rounding, high…
Four Over Six (4/6) introduces adaptive block scaling for NVFP4 quantization, reducing quantization error with minimal overhead, improving both training and post-training quantization for large language models.