Tag
TaskPress introduces a query-agnostic KV cache compression framework that uses a task guide as a meta-query and quantization scale factors to prune irrelevant tokens, enabling reusable caches across diverse queries with negligible overhead.
This empirical study examines practical online KV cache compaction for LLM agents, comparing token eviction and attention matching methods under different proxy query sources. It finds that delaying compaction to use future agent queries recovers performance, and token eviction preserves accuracy while reducing KV cache by 80%.
This paper introduces S4R, a low-rank KV cache compression method that combines selective token sampling, subspace construction, and sparse reconstruction to achieve up to 5× compression with near full-cache accuracy on LongBench and RULER benchmarks.
NVIDIA explains how attention architecture choices (group size, head dimension, KV-cache size, parallelism) set the ceiling for long-context inference performance, with guidelines for co-designing models for faster serving.
Cloudflare details how it serves large open-source MoE models like Kimi K2.6 and GLM 5.2 efficiently using FP8 KV cache quantization and weight compression, boosting throughput and cutting costs without accuracy loss.
This tutorial covers how to use a transformer model from training to inference, focusing on autoregressive generation, prefill vs. decode phases, and key-value caching for efficient inference.
ResKV proposes a KV cache compression method that divides a fixed budget into an exact main cache and a compact residual cache to reconstruct omitted attention contributions, improving performance on LongBench and RULER across multiple backbones.
This paper investigates how training-free acceleration can silently change generated content in diffusion-based multimodal large language models, and proposes paired diagnostics and consistency-control methods to mitigate content drift.
This paper introduces Mixture-of-Translators (MoT), a framework for translating KV caches across heterogeneous LLMs, enabling cache reuse between different architectures. Experiments show preserved QA performance and long-context quality across Qwen2.5, GPT-2, and OPT models.
This paper presents a topology-aware data movement orchestrator for disaggregated LLM inference, which dynamically selects optimal transport based on interconnect hierarchy and overlaps KV cache transfer with computation, achieving 3-18x transfer latency reduction over uniform RDMA.
A technical post warns against quantizing the KV cache for DeepSeek V4 Flash, showing significant quality degradation in perplexity, KL divergence, and token probabilities compared to Qwen 397B.
A computer science paper presents a graph-based workflow serving engine that unifies agent operations into a global wGraph, using dynamic graph synthesis and differential KV-cache reuse to boost agent accuracy by 4.95% while cutting GPU memory usage by 4x.
RestoreKV introduces a learned restoration mechanism that complements query-agnostic KV cache eviction, recovering full-cache behavior under aggressive budgets by generating a compact context-conditioned restore cache via a single LoRA-adapted pass, improving performance on four long-context benchmarks.
A paper introducing predictive speculative KV replication to handle bursty LLM inference workloads, with code available on GitHub.
This paper proposes a KV cache eviction strategy that scores tokens by counter-causal surprise, removing past tokens that are well-predicted from future context. The method is training-free, in-distribution, and achieves competitive performance with a fast single-layer approximation.
This paper introduces SparseSpec-L, a training-free self-speculative decoding framework for long-context LLM inference that uses a dynamically sparsified and recallable KV cache along with an entropy-based controller for speculation length, achieving up to 2.79x speedup over autoregressive decoding.
ReTopK is a training-free method that accelerates dynamic Top-K sparse attention for long-context LLMs by reusing historical query–support pairs, avoiding full-context scoring and global Top-K selection. It achieves up to 3.07x speedup over Exact Top-K with only 0.50% perplexity increase at 128K context.
This paper proposes functional reconstruction for converting MHA/GQA checkpoints into MLA draft models for speculative decoding, directly optimizing attention modules to preserve token acceptance. It reports consistent improvements across 192 configurations involving Llama/Qwen models and multiple conversion methods.
New research from UC Santa Barbara and LinkedIn introduces a self-speculating agent that unifies the agent and speculator roles in one model, improving next tool-call prediction accuracy (Hit@1) while preserving task success via joint agent-speculator reinforcement learning.
ReToken introduces a single learnable retrieval token that selects sparse query-relevant visual tokens from a pre-filled visual KV cache, improving long-context vision-language model performance on visual retrieval tasks. It yields consistent gains across image and video benchmarks while enabling efficient long-video inference on a single H100.