Tag
A new checkpoint for Qwen3.8 Flash-Next offloads the ngram lookup table to SSD, reducing RAM usage by 48 GB while maintaining inference speed in SGLang.
The benchmark reveals that Qwen3.8-Flash-Next-NVFP4 achieves nearly the highest scores among local models, with superior efficiency in request handling and token generation, and notable speed despite being undertrained.
Shantanu Goel shared a configuration recipe for optimizing Qwen 3.8 Flash Next on a single DGX Spark, tested for practical tasks and plans to benchmark it further.
NVIDIA demonstrates how quantization-aware distillation (QAD) using NVIDIA Model Optimizer improves the Nemotron 3.5 Lightning model, reducing memory usage and increasing throughput while preserving accuracy for agentic benchmarks.
A fully quantized 4-bit NVFP4 version of the Qwen3.8-27B AI model, trained with the QUASAR method to maintain high quality while reducing model size.
The author achieved performance parity between four 2017 Tesla V100 GPUs and a modern RTX 5090 when running the Qwen 3.8 model with NVFP4 precision, using custom software optimizations like the QPN kernel for efficient inference.
A tweet advising owners of NVIDIA's DGX Spark to prioritize using nvfp4 for optimal performance.
The Qwen 3.8 27B model has been released in an NVFP4 quantized version, enabling it to run on a single RTX 5090 GPU with enhancements in coding, agentic tasks, and vision-language understanding.
A newly released large MoE model, Motif 3 (314B with 13B active parameters, NVFP4 available), seems promising; the author asks for community experiences.
Introduces v100-skinny, a custom kernel library enabling fast NVFP4 inference on V100 (sm70) GPUs, achieving 366 t/s for Qwen3.6 27B in best-case extraction, with lower speeds for structured generation and code.
This paper studies quantization-aware distillation for NVFP4 low-precision LLMs and finds that output-matching with KL loss alone can mask internal representational drift. The authors propose CKA-QAD, which preserves internal geometry via CKA-guided alignment, improving reasoning and coding accuracy in compact models.
Release of an NVFP4-quantized uncensored MiniMax-H3 text encoder (Qwen3-VL-32B Heretic) that fits on a single 16GB GPU and serves as a drop-in replacement in ComfyUI workflows.
The article compares three quantized variants of Qwen3.6-27B (NVFP4 from Unsloth and Nvidia, int4-AutoRound from Intel) and requests benchmarks and hallucination data from the community.
A detailed benchmark of Unsloth's Qwen3.6-27B NVFP4 model on RTX 5090 GPUs, showing MTP (multi-token prediction) gives large speedups for single requests at short context but becomes detrimental under batch concurrency or long contexts.
Nota AI releases a 4-bit quantized version of Upstage's Solar Open2 250B MoE model, using proprietary NVFP4 quantization that requires NVIDIA Blackwell GPUs.
Baseten releases GLM-5.2-Vision, a vision-language model that adds MoonViT vision encoder to GLM-5.2 via a trained PatchMerger projector, keeping the text backbone and vision tower frozen. The model is quantized to NVFP4 for efficient inference on Blackwell hardware.
This paper proposes QUADS, a method to stabilize NVFP4 reinforcement learning for mixture-of-experts large language models by aligning quantization errors between trainer and rollout sides, achieving BF16-level accuracy and higher throughput than FP8.
MiaAI Lab publishes a recipe to serve Google's Gemma 4 31B IT in NVFP4 quantization via vLLM, enabling 256k context, MTP speculative decoding, agentic reasoning, native tool calling, and image/video support.
MiaAI Lab released an upgraded recipe for serving DeepSeek V4 Flash on two DGX Spark nodes using vLLM with DSpark speculative decoding and NVFP4 KV-cache, achieving up to 153.7 tokens per second with six concurrent sessions.
A detailed report on optimizing a production vLLM serving configuration on NVIDIA's DGX Spark, correcting flags that were costing 34% MTP acceptance after reviewing 90+ official NVIDIA documents and running a 69-scenario tool evaluation.