Tag
Baseten releases GLM-5.2-Vision, a vision-language model that adds MoonViT vision encoder to GLM-5.2 via a trained PatchMerger projector, keeping the text backbone and vision tower frozen. The model is quantized to NVFP4 for efficient inference on Blackwell hardware.
This paper proposes QUADS, a method to stabilize NVFP4 reinforcement learning for mixture-of-experts large language models by aligning quantization errors between trainer and rollout sides, achieving BF16-level accuracy and higher throughput than FP8.
MiaAI Lab publishes a recipe to serve Google's Gemma 4 31B IT in NVFP4 quantization via vLLM, enabling 256k context, MTP speculative decoding, agentic reasoning, native tool calling, and image/video support.
MiaAI Lab released an upgraded recipe for serving DeepSeek V4 Flash on two DGX Spark nodes using vLLM with DSpark speculative decoding and NVFP4 KV-cache, achieving up to 153.7 tokens per second with six concurrent sessions.
A detailed report on optimizing a production vLLM serving configuration on NVIDIA's DGX Spark, correcting flags that were costing 34% MTP acceptance after reviewing 90+ official NVIDIA documents and running a 69-scenario tool evaluation.
A detailed comparison of Unsloth's NVFP4 quantized model inference performance between vLLM and llama.cpp, highlighting prefill speed advantages for vLLM but decode and caching advantages for llama.cpp in single-stream agent workloads.
Reports on testing the RTX Pro 4500 GPU with Oculink, focusing on INT4 Autoround and NVFP4 W4A4 quantized models using PrismaQuant.
Benchmark results of the unsloth/Qwen3.6-27B-NVFP4 model running on 4x RTX 5060 Ti GPUs with peer-to-peer and pipeline parallelism at various concurrency levels.
This article presents a recipe for low-precision (NVFP4) RL training that balances throughput and stability, addressing issues from forward and backward pass quantization errors.
Unsloth releases a new version of Qwen3.6 27B NVFP4 quantization scheme, introducing FP8_E4M3 intermediate precision layer and refined weight protection, achieving 2.5x speed improvement on 24GB VRAM, while improving accuracy and tool-calling capabilities.
Unsloth AI releases new NVFP4 quantized Qwen3.6 models that run 2.5x faster on GPUs, with improved accuracy and tool calling capability.
Unsloth releases quantized Qwen3.6 models using NVFP4 format, achieving 2.5x faster inference speeds.
Four Over Six (4/6) introduces adaptive block scaling for NVFP4 quantization, reducing quantization error with minimal overhead, improving both training and post-training quantization for large language models.
A real-time demo shows 16 concurrent users chatting with Qwen3.6-35B on a single DGX Spark, achieving peak 440 tok/s total and 105 tok/s per user using NVFP4 + MTP-3 on vLLM.
Red Hat AI team released quantized checkpoints for GLM-5.2 using NVFP4 and FP8 quantization, reducing model size by over 70% while maintaining high accuracy on GPQA. The quantized model, paired with the DSpark speculator, enables efficient deployment with vLLM.
The article provides the optimal deployment configuration for GLM-5.2 on 8xB200 nodes, showing that NVFP4 with two TP=4 replicas achieves roughly 2x throughput over FP8 TP=8, with detailed performance data and caveats.
This post details the author's setup and benchmarks for running NVFP4-quantized image and video generation models on a GIGABYTE AI TOP ATOM (DGX Spark) workstation, achieving impressive performance with models like FLUX.2, Qwen-Image, and LTX-2.3 for video with synchronized audio.
A user successfully ran nvfp4 quantization on Intel Arc B70s GPUs, achieving nearly double speed and higher accuracy compared to their best int4 configuration, challenging hardware-specific format assumptions.
A bug in vLLM's speculative decoding configuration for GLM-5.2 NVFP4 on four DGX Sparks was fixed, resolving a performance tradeoff and achieving ~24 tok/s at 128K context with MTP4.
A user asks about hardware requirements for serving GLM-5.2 in NVFP4 format, which vLLM now supports with reduced memory footprint and maintained accuracy.