We built a calibration-aware Q4_K_M quant of Qwen3.5 0.8B that recovers 96.5% of the BF16 gap vs pure llama.cpp Q4_K_M (SpectralQuant)
Summary
A calibration-aware Q4_K_M quantization of Qwen3.5 0.8B using SpectralQuant recovers 96.5% of the BF16 performance gap compared to the standard llama.cpp Q4_K_M quant.
Similar Articles
Qwen 3.6 27B on 24GB VRAM setup: backend comparisons, quant choice and settings (llama.cpp, ik_llama.cpp, BeeLlama, vllm)
The article compares llama.cpp backends for running Qwen 3.6 27B on an RTX 3090 24GB, finding ik_llama.cpp with IQ4_KS quantization yields the best performance (1261 tok/s prefill, 72.9 tok/s decode).
Qwen3.6-27B Quantization Benchmark
This article benchmarks various Qwen3.6-27B quantizations (Q8 to Q2) using KLD and Same Top P metrics, comparing providers like Unsloth and mradermacher, and offers recommendations for quality-size trade-offs.
I tested Qwen3.8 27B KLD for 21 Models at 9 different KV quants that fit on 16GB VRAM (189 benchmarks).
A community benchmark evaluated 21 Qwen3.8 27B model/KV quantization combinations (189 benchmarks) that fit in 16GB VRAM, finding q5_1/q5_1 is usually optimal and that upping model quant at q4_0/q4_0 beats lower quants at full f16 KV cache.
We quantized Qwen 3.8 27B and compared the quants on an RTX 6000
The team quantized Qwen 3.8 27B into various GGUF formats and benchmarked them on an RTX 6000, finding similar performance across quants with AD-Q6_K recommended for safety.
Qwen3.8-27B Hybrid IQ4_XS quantization for 16GB gang
This is a quantized version of the Qwen3.8-27B AI model using IQ4_XS quantization, optimized for 16GB RAM systems, with instructions for local deployment using various tools like llama.cpp and Ollama.