Tag
This paper introduces a diversity-aware, layer-wise scoring method for KV cache eviction in large language models, incorporating attention dispersion and redundancy to improve performance on LongBench datasets.
Random Attention presents a signal-free KV-cache eviction policy for reasoning models that matches or exceeds the performance of learned methods on benchmarks like MATH-500 and LiveCodeBench, while being faster in inference.
The paper proposes Random Attention, a KV cache eviction method that uses random selection instead of scoring, matching selective methods while improving throughput in reasoning tasks.
This paper analyzes KV-cache eviction strategies in sparse-attention models, showing that selecting largest weights is near-optimal and that published margins come from memory and query information, with ContourKV achieving strong performance.
This paper introduces LaProx, a novel KV Cache eviction strategy for long-context LLM inference that reformulates the problem as an output-aware matrix multiplication approximation, achieving high performance with only 5% cache usage.