nvfp4

This paper studies quantization-aware distillation for NVFP4 low-precision LLMs and finds that output-matching with KL loss alone can mask internal representational drift. The authors propose CKA-QAD, which preserves internal geometry via CKA-guided alignment, improving reasoning and coding accuracy in compact models.

Release of an NVFP4-quantized uncensored MiniMax-H3 text encoder (Qwen3-VL-32B Heretic) that fits on a single 16GB GPU and serves as a drop-in replacement in ComfyUI workflows.

The article compares three quantized variants of Qwen3.6-27B (NVFP4 from Unsloth and Nvidia, int4-AutoRound from Intel) and requests benchmarks and hallucination data from the community.

A detailed benchmark of Unsloth's Qwen3.6-27B NVFP4 model on RTX 5090 GPUs, showing MTP (multi-token prediction) gives large speedups for single requests at short context but becomes detrimental under batch concurrency or long contexts.

Nota AI releases a 4-bit quantized version of Upstage's Solar Open2 250B MoE model, using proprietary NVFP4 quantization that requires NVIDIA Blackwell GPUs.

Baseten releases GLM-5.2-Vision, a vision-language model that adds MoonViT vision encoder to GLM-5.2 via a trained PatchMerger projector, keeping the text backbone and vision tower frozen. The model is quantized to NVFP4 for efficient inference on Blackwell hardware.

This paper proposes QUADS, a method to stabilize NVFP4 reinforcement learning for mixture-of-experts large language models by aligning quantization errors between trainer and rollout sides, achieving BF16-level accuracy and higher throughput than FP8.

MiaAI Lab publishes a recipe to serve Google's Gemma 4 31B IT in NVFP4 quantization via vLLM, enabling 256k context, MTP speculative decoding, agentic reasoning, native tool calling, and image/video support.

MiaAI Lab released an upgraded recipe for serving DeepSeek V4 Flash on two DGX Spark nodes using vLLM with DSpark speculative decoding and NVFP4 KV-cache, achieving up to 153.7 tokens per second with six concurrent sessions.

A detailed report on optimizing a production vLLM serving configuration on NVIDIA's DGX Spark, correcting flags that were costing 34% MTP acceptance after reviewing 90+ official NVIDIA documents and running a 69-scenario tool evaluation.

A detailed comparison of Unsloth's NVFP4 quantized model inference performance between vLLM and llama.cpp, highlighting prefill speed advantages for vLLM but decode and caching advantages for llama.cpp in single-stream agent workloads.

Reports on testing the RTX Pro 4500 GPU with Oculink, focusing on INT4 Autoround and NVFP4 W4A4 quantized models using PrismaQuant.

Benchmark results of the unsloth/Qwen3.6-27B-NVFP4 model running on 4x RTX 5060 Ti GPUs with peer-to-peer and pipeline parallelism at various concurrency levels.

This article presents a recipe for low-precision (NVFP4) RL training that balances throughput and stability, addressing issues from forward and backward pass quantization errors.

Unsloth releases a new version of Qwen3.6 27B NVFP4 quantization scheme, introducing FP8_E4M3 intermediate precision layer and refined weight protection, achieving 2.5x speed improvement on 24GB VRAM, while improving accuracy and tool-calling capabilities.

Unsloth AI releases new NVFP4 quantized Qwen3.6 models that run 2.5x faster on GPUs, with improved accuracy and tool calling capability.

Unsloth releases quantized Qwen3.6 models using NVFP4 format, achieving 2.5x faster inference speeds.

Four Over Six (4/6) introduces adaptive block scaling for NVFP4 quantization, reducing quantization error with minimal overhead, improving both training and post-training quantization for large language models.

A real-time demo shows 16 concurrent users chatting with Qwen3.6-35B on a single DGX Spark, achieving peak 440 tok/s total and 105 tok/s per user using NVFP4 + MTP-3 on vLLM.

Red Hat AI team released quantized checkpoints for GLM-5.2 using NVFP4 and FP8 quantization, reducing model size by over 70% while maintaining high accuracy on GPQA. The quantized model, paired with the DSpark speculator, enables efficient deployment with vLLM.