Tag
This paper surveys 200 works on LLM quantization, formalizing the 'Great Inversion' principle that contrasts energy concentration in coding with within-group flattening in quantization, and offers a guide for transform selection based on deployment regimes and formats like MXFP4 and NVFP4.
ExTernD introduces an expanded-rank ternary decomposition for post-training LLM quantization, enabling accuracy approaching bf16 by using a factored representation with free inner rank. It matches Q4_K accuracy at 5.2-5.5 effective bits per weight on models like Gemma-4 and Qwen3.5.
An open dataset on GitHub maps which local LLMs fit various RAM tiers (8GB to 128GB), providing memory sizing rules, per-tier model lists, and Ollama commands, with a JSON API for programmatic access.
ScaleSweep proposes a new block scale initialization method for NVFP4 post-training quantization of LLMs, achieving improved accuracy by sweeping over feasible block scale candidates. Experiments on Llama and Qwen models show it preserves over 93% of full-precision performance under aggressive quantization.
LiftQuant introduces a 'lift-then-project' mechanism enabling continuous (non-integer) bit-width quantization for LLMs, allowing precise fitting to hardware memory budgets. The framework compresses a 70B LLM to 2.4-bit to fit a 24GB GPU, outperforming state-of-the-art 2-bit models.
Introduces QAM-W, a joint 2D codebook quantization method for LLM weights using Hadamard rotation and activation-aware scaling, achieving near BF16 perplexity at 5–6 bits per weight and matching SmoothQuant W8A8 quality with 32% fewer weight bits.
This paper investigates smoothness degradation in extremely quantized Large Language Models, arguing that preserving smoothness is crucial for maintaining performance beyond numerical accuracy.
A user demonstrates successful local inference of a 27B parameter Qwen model across three GTX 1080 Ti GPUs, achieving approximately 28-30 tokens per second using TurboQuant optimization.
Researchers identify two distinct failure modes in aggressive LLM quantization—Signal Degradation and Computation Collapse—and show that training-free fixes only remedy the former, indicating structural reconstruction is needed for ultra-low-bit models.