KVpop -- Key-Value Cache Compression with Predictive Online Pruning

Hugging Face Daily Papers Papers
key-value-cache cache-compression predictive-pruning autoregressive-decoding kv-eviction future-attention

Summary

KVpop introduces a learned KV cache eviction policy supervised by future-attention targets, achieving high compression rates (e.g., 98% performance at 75% compression) on Qwen3 models while maintaining quality.

Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on static heuristics or proxy scores, which poorly track future token utility and cause brittle eviction as relevance shifts. To address this, we introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. The scorer is trained against a novel future-attention target, computed efficiently without materializing dense attention maps. We further introduce a delayed memory-based scorer that, uniquely among learned eviction methods, defers scoring for a fixed number of steps to exploit near-future context. On AIME and HMMT mathematical reasoning, KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines. Qwen3-8B shows even stronger results, reaching near-full teacher performance. These results show that supervising eviction with future-attention signals cuts memory costs while maintaining quality.
Original Article
View Cached Full Text

Cached at: 07/07/26, 10:43 AM

Paper page - KVpop – Key-Value Cache Compression with Predictive Online Pruning

Source: https://huggingface.co/papers/2607.05061 Published on Jul 6

·

Submitted byhttps://huggingface.co/sirluk

Lukason Jul 7

Abstract

KVpop learns optimal key-value cache eviction by directly supervising keep-or-drop decisions using future-attention targets, achieving high performance with reduced memory usage.

Key-value (KV) cache growth is a major bottleneck inautoregressive decoding, as memory and bandwidth scale linearly with context length. ExistingKV evictionmethods often rely on static heuristics or proxy scores, which poorly track future token utility and cause brittle eviction as relevance shifts. To address this, we introduce KVpop, which learns a fixed-budgetKV evictionpolicy by directly supervising the keep-or-drop decision. The scorer is trained against a novelfuture-attention target, computed efficiently without materializing denseattention maps. We further introduce adelayed memory-based scorerthat, uniquely among learned eviction methods, defers scoring for a fixed number of steps to exploit near-future context. On AIME and HMMT mathematical reasoning, KVpop retains 98% of full-attention performance onQwen3-4Bat 75%KV cache compressionand 97% at 88% compression, consistently outperforming established eviction baselines.Qwen3-8Bshows even stronger results, reaching near-full teacher performance. These results show that supervising eviction with future-attention signals cuts memory costs while maintaining quality.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.05061

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.05061 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.05061 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.05061 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility

arXiv cs.LG

Introduces Self-Pruned Key-Value Attention (SP-KV), a mechanism that learns to predict future utility of key-value pairs to dynamically prune the KV cache, reducing memory usage and decoding speed by 3-10x with minimal performance degradation. The model and utility predictor are trained end-to-end using next-token prediction.

KV Cache Compression 900000x Beyond TurboQuant and Per-Vector Shannon Limit

Hacker News Top

A new paper proposes sequential KV cache compression using probabilistic language tries and predictive delta coding, achieving theoretical compression ratios of ~914,000× beyond TurboQuant by exploiting the sequential structure of language model tokens rather than treating vectors independently.