Qwen3.6-27B KLDs - INTs and NVFPs
Summary
Reddit post compares quantized Qwen3.6-27B variants (INT4, NVFP4, BF16-INT4) showing trade-offs between memory size and accuracy for different use-cases.
Similar Articles
Qwen 3.6 27B - VLLM Performance Benchmark Results (BF16, FP8, NVFP4)
A detailed benchmark of Qwen 3.6 27B using VLLM across BF16, FP8, and NVFP4 quantizations, showing NVFP4 fastest for token generation but FP8 best for prompt processing, with practical advice on choosing the right quantization for coding tasks.
Qwen3.6-27B Quantization Benchmark
This article benchmarks various Qwen3.6-27B quantizations (Q8 to Q2) using KLD and Same Top P metrics, comparing providers like Unsloth and mradermacher, and offers recommendations for quality-size trade-offs.
UPDATE: Qwen-27B-IQ4_KS and Qwen-27B-IQ_KS_KT for ik_llama.cpp, especially for NVIDIA with 16GB VRAM
New GGUF quantizations of Qwen3.6-27B optimized for 16GB VRAM NVIDIA GPUs, including an experimental Trellis variant, with perplexity benchmarks.
@witcheer: everyone says NVFP4 makes blackwell cards "faster." I benchmarked Qwen3.6-27B three ways on my 5090: >NVFP4 >plain Q4_K…
A benchmark of NVFP4 on an RTX 5090 with Qwen3.6-27B shows prefill speed gains of 32-42% over equal-bit Q4_K_M and 52-68% over Q6_K, but decode gains are modest (+9% vs Q4) as decode is memory-bandwidth bound. The quality loss compared to Q6 is minimal (-0.8 average), making NVFP4 a good choice for local inference.
RedHatAI/Qwen3.6-35B-A3B-NVFP4
Red Hat AI released an NVFP4-quantized 35B MoE version of Qwen3.6 that retains 96.28% GSM8K accuracy while enabling 4-bit inference via vLLM.