Has anyone tested how quantization hits different capabilities separately? My results are surprising.

Reddit r/LocalLLaMA News
quantization benchmarking model-capabilities gguf fp16

Summary

The author shares surprising results from systematic tests on how different quantization levels (e.g., Q4_K_M, Q5_K_M) affect model capabilities separately, showing that math accuracy degrades more than knowledge tasks, and calls for more rigorous testing on context decay across quant levels.

I've been running some systematic tests on a few models comparing FP16 vs various GGUF quant levels, and instead of looking at one aggregate benchmark score, I broke it down by capability: math (GSM8K), code (HumanEval), reasoning (ARC-Challenge), and knowledge recall (MMLU-Pro). The results are way more nuanced than "Q4 loses X% quality." For example on one 27B model, Q4_K_M barely moved the needle on conversational/knowledge tasks (under 2% degradation) but dropped multi step math accuracy by almost 9% compared to FP16. Q5_K_M basically eliminated the math gap. So the "right" quant level depends entirely on what you're using the model for. The other thing I've been curious about is context decay. Does anyone know of systematic testing on whether quantized models lose context retrieval accuracy faster than FP16 as the context window fills up? Like, does a Q4 model start hallucinating at 8K context where the FP16 version holds steady until 12K? I've seen scattered anecdotes but nothing rigorous with controlled needle in haystack tests across quant levels. It feels like the community has tons of data on "which model is best" but almost nothing on "which quant of this specific model is best for my use case and hardware." Am I missing something, or is this genuinely a gap?
Original Article

Similar Articles

K-Quantization and its Impact on Output Performance

arXiv cs.CL

This paper investigates the impact of different quantization levels (2-bit to 8-bit) on the performance of eight large language models across reasoning, code comprehension, and reading comprehension tasks, finding that while higher precision generally yields better performance, aggressive quantization often retains acceptable accuracy, with larger models showing greater resilience.