Tag
This post from NVIDIA explains how to use the NVIDIA Model Optimizer library to quantize a CLIP model to FP8 using post-training quantization, reducing VRAM usage and improving inference performance on consumer GPUs.
Introduces Flatness metric and Bidirectional Diagonal Quantization (BDQ) for post-training quantization of large language models, achieving near-lossless 4-bit weight and activation quantization and substantial improvements at extreme low-bit settings.
This paper investigates the impact of different quantization levels (2-bit to 8-bit) on the performance of eight large language models across reasoning, code comprehension, and reading comprehension tasks, finding that while higher precision generally yields better performance, aggressive quantization often retains acceptable accuracy, with larger models showing greater resilience.
LoopQ is a post-training quantization framework for looped language models that addresses distribution shift, state reuse, and error accumulation. It achieves 68.8% average accuracy improvement under 4-bit weights and activations.
This paper introduces E-PMQ, an expert-guided post-merge quantization framework that addresses the combined deviations from merging and quantization, achieving significant accuracy improvements on multi-task merged models like CLIP-ViT and FLAN-T5.
This paper proposes Saliency-Aware Regularized Quantization Calibration (SARQC), a unified framework that improves Post-Training Quantization (PTQ) for LLMs by adding a regularization term to preserve weight proximity, enhancing generalization and performance.
Researchers identify two distinct failure modes in aggressive LLM quantization—Signal Degradation and Computation Collapse—and show that training-free fixes only remedy the former, indicating structural reconstruction is needed for ultra-low-bit models.