Introducing cyankiwi AWQ 4-bit Quantization — 26.05 update
Summary
Cyankiwi introduced an updated version of their AWQ 4-bit quantization method that jointly optimizes scales and quantization ranges, achieving lower KL divergence than existing methods on Llama-3 models.
Similar Articles
We built a calibration-aware Q4_K_M quant of Qwen3.5 0.8B that recovers 96.5% of the BF16 gap vs pure llama.cpp Q4_K_M (SpectralQuant)
A calibration-aware Q4_K_M quantization of Qwen3.5 0.8B using SpectralQuant recovers 96.5% of the BF16 performance gap compared to the standard llama.cpp Q4_K_M quant.
Qwen3.6-27B Quantization Benchmark
This article benchmarks various Qwen3.6-27B quantizations (Q8 to Q2) using KLD and Same Top P metrics, comparing providers like Unsloth and mradermacher, and offers recommendations for quality-size trade-offs.
KronQ: LLM Quantization via Kronecker-Factored Hessian
KronQ is a post-training quantization framework that incorporates gradient covariance using a Kronecker-factored Hessian approximation, enabling bidirectional incoherence processing and improved sensitivity metrics for mixed-precision allocation. It achieves low perplexity even at 2-bit weight-only quantization on large models like LLaMA-3-70B.
moar QAT stuff and hairy ticks
The author releases improved GGUF quantized versions of Gemma 4 models (12B and 31B) using a more accurate quantization-aware training process that achieves lower KLD and higher same-top percentage than stock quantizations.
I built a tool to actually test which weights matter before quantizing, instead of guessing (Qwen3.6-27B, 3 builds: Bedrock/Tightrope/Gambit)
A developer built a testing harness that measures KL divergence per weight group during quantization, leading to three custom quantized builds of Qwen3.6-27B (Bedrock, Tightrope, Gambit) with optimized compression. Tool calling is identified as the first capability to degrade under quantization.