Does it make sense to use alternative quantizations of QAT models? [D]
Summary
A discussion on whether it is sensible to use alternative quantization methods on quantization-aware trained (QAT) models like Gemma-4, questioning if unsloth's benchmarks showing closer performance to QAT fine-tunes are beneficial or counterproductive.
Similar Articles
Gemma 4 26B A4B IT QAT Comparison
A user benchmarks three quantized versions of Gemma 4 26B IT (4-bit, 6-bit, and 8-bit QAT) on MMLU_PRO and HumanEval, finding that the QAT 8-bit model performs worse than the 6-bit quant on HumanEval and is not clearly better than 4-bit, questioning the superiority of QAT for this model.
2-bit QAT model releases
A discussion on the potential of 2-bit Quantization Aware Training (QAT) for larger MoE models, comparing their performance to 4-bit QAT and ternary LLMs, and considering feasibility for consumer hardware.
Some contrived tests comparing the accuracy of different Gemma and Qwen quantizations
A user shares benchmark results comparing the accuracy of various quantized Gemma and Qwen models on arithmetic, presidential DOB, and attention tests, highlighting trade-offs between model size and quantization level.
Gemma 4 QAT 31B responds better to KV cache quantization too
The Gemma 4 QAT 31B model demonstrates improved behavior with KV cache quantization, suggesting enhanced inference efficiency.
moar QAT stuff and hairy ticks
The author releases improved GGUF quantized versions of Gemma 4 models (12B and 31B) using a more accurate quantization-aware training process that achieves lower KLD and higher same-top percentage than stock quantizations.