Tag
SemKV introduces a mixed-precision quantization framework for KV cache guided by an empirically measured quality cliff, achieving significant storage reduction without detectable quality loss in long-context LLM inference.
A novel end-to-end framework for LLM compression that jointly optimizes structural pruning and mixed-precision quantization, achieving significant perplexity reductions and speedups over state-of-the-art methods, especially at ultra-low bit precisions.