nvfp4

The article provides the optimal deployment configuration for GLM-5.2 on 8xB200 nodes, showing that NVFP4 with two TP=4 replicas achieves roughly 2x throughput over FP8 TP=8, with detailed performance data and caveats.

This post details the author's setup and benchmarks for running NVFP4-quantized image and video generation models on a GIGABYTE AI TOP ATOM (DGX Spark) workstation, achieving impressive performance with models like FLUX.2, Qwen-Image, and LTX-2.3 for video with synchronized audio.

A user successfully ran nvfp4 quantization on Intel Arc B70s GPUs, achieving nearly double speed and higher accuracy compared to their best int4 configuration, challenging hardware-specific format assumptions.

A bug in vLLM's speculative decoding configuration for GLM-5.2 NVFP4 on four DGX Sparks was fixed, resolving a performance tradeoff and achieving ~24 tok/s at 128K context with MTP4.

A user asks about hardware requirements for serving GLM-5.2 in NVFP4 format, which vLLM now supports with reduced memory footprint and maintained accuracy.

NVIDIA released an NVFP4 quantized checkpoint of GLM-5.2, a 744B MoE model (40B active) optimized for reasoning and coding, with day-0 support in SGLang.

NVIDIA has released an NVFP4 precision version quantized from the Zhipu GLM-5.2 model, available via the Hugging Face free tier API.

NVIDIA released GLM-5.2-NVFP4, a quantized version of ZAI's GLM-5.2 MoE language model optimized for inference on NVIDIA Blackwell GPUs using Model Optimizer.

A tweet thread introduces a visualizer for micro-scaling/block quant formats like NVFP4 and MXFP4, explaining how these low-precision floats work and their use in LLM inference to reduce memory bandwidth demands.

Luke Alonso uploaded an NVFP4 quantized version of GLM 5.2 (467GB) that can fit on 4x DGX Sparks hardware, costing approximately $20k.

NVFP4 KV cache quantization on sm120 significantly improves memory efficiency for large language models, enabling 32GB VRAM systems to achieve ~60 tok/sec inference at 196k context size with Qwen3.6-27B.

A turnkey Docker setup to serve the GLM-5.2-NVFP4-REAP-469B model on 4× RTX PRO 6000 Blackwell GPUs using vLLM, with detailed instructions and configuration options.

@onusoz demonstrates running 16 parallel instances of NVIDIA's quantized Gemma-4-26B-A4B-NVFP4 model on a single DGX Spark with 128GB unified memory, achieving 300 tok/s aggregate, showcasing high concurrency without flashinfer.

A technical comparison between nvfp4 and mxfp4 formats, highlighting that nvfp4 uses an additional tensor-wise scale factor to overcome fp4's range limit, allowing more precision in block-wise scale factors.

A pull request to vLLM adds support for tensor parallelism degree 3 for MiniMax M3 with its NVFP4 quantization, enabling the model to run on 3x DGX Sparks with 87GB memory each.

A tweet highlights that the abliterated, NVFP4 quantized Gemma-4-12B model (7.7 GB) can rival Qwen 3.6-35B in practical tasks while running fast on Blackwell GPUs, demonstrating significant efficiency gains.

ScaleSweep proposes a new block scale initialization method for NVFP4 post-training quantization of LLMs, achieving improved accuracy by sweeping over feasible block scale candidates. Experiments on Llama and Qwen models show it preserves over 93% of full-precision performance under aggressive quantization.

A mixed-precision quantization of Google's Gemma-4-12B-it model using NVFP4 for MLP weights and FP8 for attention layers, achieving 25% smaller footprint and faster throughput while maintaining quality.

A benchmark of NVFP4 on an RTX 5090 with Qwen3.6-27B shows prefill speed gains of 32-42% over equal-bit Q4_K_M and 52-68% over Q6_K, but decode gains are modest (+9% vs Q4) as decode is memory-bandwidth bound. The quality loss compared to Q6 is minimal (-0.8 average), making NVFP4 a good choice for local inference.

The author introduces an open-source GGUF quantizer tool for llama.cpp that creates NVFP4 and MXFP6 quantized models with advanced techniques like RSF, tensor promotion, and dynamic quantization, achieving better quality than existing methods like ModelOpt.