fp4

NVIDIA released Qwen3.8-Flash-Next-FP4-Local, an FP4-quantized version of Alibaba's Qwen3.8-Flash-Next model, built with quantization-aware distillation via NVIDIA Model Optimizer. The ~96 GiB checkpoint targets local AI hardware like the RTX PRO 6000 Blackwell and DGX Spark, and is validated with vLLM.

miniTE 是一个开源框架，用于诊断并弥合原生 NVFP4（W4A4）强化学习中训练与推理引擎间的数值执行差异，使 log-probability 差距降至约 1e-8，并在 8×B300 GPU 上实现最高 1.39 倍于 BF16 的学习器吞吐量，同时在 Qwen3-30B-A3B 上保持接近 BF16 的长程 RL 稳定性。

提出了 format-aware fusion 方法，将 FP4 量化中的 scale 计算、操作数打包与反向状态保存与 GEMM 融合，在 Llama-3 8B 预训练中达到最高 37.9K tokens/s/GPU，MXFP4 路线在略高于 bfloat16 训练损失终点的情况下实现 37.2K tokens/s/GPU（86.3% MFU），显著超过 Transformer Engine 的 27.6K。

A user reports solid results running a $5400 eBay 8x V100 server for local LLM inference, using nvfp4 NVIDIA checkpoints via a heavily optimized vLLM fork (1CatAI/1Cat-vLLM) that unpacks to fp16 on the fly, achieving over 200 tok/s on a 27B model with TP=4 and strong prefill throughput.

The author criticizes the trend of using FP4 quantization in inference engines, arguing that it severely degrades model quality and causes hallucinations, especially for small dense models.

This paper presents HiFloat4 format and Rollout-ResQ for end-to-end FP4 reinforcement learning post-training of LLMs, achieving a 1.1% accuracy gap to BF16 on Qwen2.5 models.

The FastVideo team releases new FP4 attention kernels for B300, achieving up to 1.69x speedup over FlashAttention 4.

NVIDIA released the NVFP4 quantized Kimi-K2.7-Code, a 1 trillion-parameter Moonshot AI model quantized to FP4 for Blackwell GPUs, preserving accuracy with reduced memory usage.

A llama.cpp fork introduces TurboQuant TQ3_4S quantization that maps to Blackwell FP4 tensor cores, achieving up to 221% faster prompt processing on GB10 while maintaining near Q4 quality at Q3 size.

NVIDIA's blog details how FP4, with the NVFP4 format and Blackwell hardware, has evolved from a compression trick to a practical primitive for training and inference across LLMs and diffusion models, achieving near 16-bit accuracy.

SharQ introduces a training-free method combining activation sparsity and FP4 quantization for LLM inference, using sparse-dense decomposition and a unified FP4 weight payload. It achieves significant latency reduction and accuracy recovery over FP4-only baselines.

Discusses using NVFP4 4-bit floating point weights for maximum performance, achieved via in-house quantization from FP8 using NVIDIA ModelOpt, highlighting the data format's dual scale factors for high dynamic range.

This paper identifies a fundamental limitation (shrinkage bias) in non-uniform FP4 quantization formats for LLM pretraining and proposes UFP4, a uniform 4-bit training recipe that outperforms existing E2M1-based methods.

Converted the Qwen 3.6 35b a3b model to ROCmfp4 format, leveraging MTP benefits for improved performance on AMD hardware.

A page from Modal's LLM Engineer's Almanac that provides an interactive explorer for understanding low-precision floating-point formats like bf16 and fp4.