nvfp4

Optimized performance for running the Qwen3.8 27b NVFP4 model on a single AMD Radeon R9700 GPU, achieving up to 153 tokens per second in decode and 3,619 tokens per second in prefill with improved concurrency.

This paper explains why fully quantizing the recurrent Gated DeltaNet layers in hybrid LLMs to 4-bit NVFP4 preserves accuracy across long-context benchmarks, detailing mechanisms like outlier localization and robust delta-rule dynamics.

OCGQuant proposes an outlier-companion grouping method for NVFP4 quantization to reduce activation block errors, enhancing LLM inference efficiency. Experiments on Llama3 and Qwen3 demonstrate its superior performance over existing post-training quantization techniques.

NVIDIA researchers will present at PyTorch Conference North America 2026 on using NVFP4 pretraining recipes to accelerate large-scale LLM training while maintaining quality comparable to BF16, with integration into PyTorch tools like TorchAO and TorchTitan.

RadixArk/Qwen3.8-Flash-Next-NVFP4 is now supported in SGLang-V100, enabling full context operation on 4xV100 GPUs with performance metrics showing high throughput and context handling up to 256k tokens.

A new checkpoint for Qwen3.8 Flash-Next offloads the ngram lookup table to SSD, reducing RAM usage by 48 GB while maintaining inference speed in SGLang.

The benchmark reveals that Qwen3.8-Flash-Next-NVFP4 achieves nearly the highest scores among local models, with superior efficiency in request handling and token generation, and notable speed despite being undertrained.

Shantanu Goel shared a configuration recipe for optimizing Qwen 3.8 Flash Next on a single DGX Spark, tested for practical tasks and plans to benchmark it further.

NVIDIA demonstrates how quantization-aware distillation (QAD) using NVIDIA Model Optimizer improves the Nemotron 3.5 Lightning model, reducing memory usage and increasing throughput while preserving accuracy for agentic benchmarks.

A fully quantized 4-bit NVFP4 version of the Qwen3.8-27B AI model, trained with the QUASAR method to maintain high quality while reducing model size.

The author achieved performance parity between four 2017 Tesla V100 GPUs and a modern RTX 5090 when running the Qwen 3.8 model with NVFP4 precision, using custom software optimizations like the QPN kernel for efficient inference.

A tweet advising owners of NVIDIA's DGX Spark to prioritize using nvfp4 for optimal performance.

The Qwen 3.8 27B model has been released in an NVFP4 quantized version, enabling it to run on a single RTX 5090 GPU with enhancements in coding, agentic tasks, and vision-language understanding.

A newly released large MoE model, Motif 3 (314B with 13B active parameters, NVFP4 available), seems promising; the author asks for community experiences.

Introduces v100-skinny, a custom kernel library enabling fast NVFP4 inference on V100 (sm70) GPUs, achieving 366 t/s for Qwen3.6 27B in best-case extraction, with lower speeds for structured generation and code.

This paper studies quantization-aware distillation for NVFP4 low-precision LLMs and finds that output-matching with KL loss alone can mask internal representational drift. The authors propose CKA-QAD, which preserves internal geometry via CKA-guided alignment, improving reasoning and coding accuracy in compact models.

Release of an NVFP4-quantized uncensored MiniMax-H3 text encoder (Qwen3-VL-32B Heretic) that fits on a single 16GB GPU and serves as a drop-in replacement in ComfyUI workflows.

The article compares three quantized variants of Qwen3.6-27B (NVFP4 from Unsloth and Nvidia, int4-AutoRound from Intel) and requests benchmarks and hallucination data from the community.

A detailed benchmark of Unsloth's Qwen3.6-27B NVFP4 model on RTX 5090 GPUs, showing MTP (multi-token prediction) gives large speedups for single requests at short context but becomes detrimental under batch concurrency or long contexts.

Nota AI releases a 4-bit quantized version of Upstage's Solar Open2 250B MoE model, using proprietary NVFP4 quantization that requires NVIDIA Blackwell GPUs.