Tag
The post questions why quantization-aware training is not a standard practice for open-weight AI models and asks about potential barriers like compute costs or performance impacts.
This research paper analyzes capability-stratified degradation in ternary quantized language models, showing that while factual knowledge deteriorates significantly, commonsense reasoning and downstream task adaptability are retained, making the models viable for efficient edge deployment.
QUASAR is a quantization-aware training method that uses loss-aware reconstruction to lower the loss floor, improving low-bit model performance in large language models with significant accuracy gains at 2-4 bits.
Commentary on Google's Gemma 4 technical report, noting aggressive optimization via Quantization-Aware Training and Multi-Token Prediction for faster inference.
Introduces Variable Bit-width Quantization (VBQ), a training-time method where each group of 64 weights learns its own bit-width (1,2,4,8) via Gumbel-Softmax relaxation. VBQ discovers a heterogeneous allocation that yields a 'bigger-but-smaller' regime, e.g., a 131M parameter model at 1.82 mean bits beats a 55M FP16 model while using less storage, and a 1.46B model matches a 593M FP16 with ~3.7x less storage.
Release of Gemma4-12B-QAT Uncensored Balanced, a fine-tuned uncensored model with a multi-token-prediction draft head for ~60% faster speculative decoding, optimized for llama.cpp and offering vision support.
Google released Gemma 4 models with quantization-aware training (QAT) at Q4_0 precision on Hugging Face, offering efficient variants from 5B to 33B parameters.
New QAT Gemma 4 checkpoints offer similar performance with ~4x less memory, enabling a 1GB memory footprint for Gemma 4 E2B via a new mobile quantization format.
Google releases Gemma 4 models optimized with Quantization-Aware Training (QAT) to improve efficiency for mobile and laptop deployment, reducing memory footprint to 1GB for the E2B model while preserving quality.
Google DeepMind releases Gemma 4 models optimized with Quantization-Aware Training (QAT) in multiple formats including GGUF, enabling high quality with reduced memory requirements.
This paper systematically studies HiF8 W8A8 quantization-aware training for OpenPangu-Embedded-1B, identifying and addressing failure modes such as amax saturation and catastrophic forgetting, achieving near-lossless performance with a 64-step max-algorithm DTS strategy and a 500-step BF16 warmup.