LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding
Summary
LazyAttention introduces a novel attention mechanism that defers positional encoding to enable zero-copy, position-agnostic KV cache reuse across multiple requests. The approach reduces time-to-first-token by 1.37× and increases throughput by 1.40× compared to Block-Attention in RAG settings with skewed document distributions.
View Cached Full Text
Cached at: 06/05/26, 02:13 AM
# LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding Source: [https://arxiv.org/abs/2606.04302](https://arxiv.org/abs/2606.04302) [View PDF](https://arxiv.org/pdf/2606.04302) > Abstract:Key\-value \(KV\) caching accelerates inference of large language models \(LLMs\) by reusing past computations for generated tokens\. Its importance becomes even greater in long\-context applications such as retrieval\-augmented generation \(RAG\) and in\-context learning \(ICL\)\. However, conventional KV caching embeds positional information directly into the cache, limiting its reusability\. Existing solutions either restrict reuse to prefixes or require expensive memory materialization for positional re\-encoding\. We introduce LazyAttention, a novel attention mechanism that kernelizes deferred positional encoding to enable zero\-copy, position\-agnostic KV reuse\. By adjusting positional encoding within attention kernels on\-the\-fly, LazyAttention resolves the materialization bottleneck, allowing a single physical KV copy to serve multiple logical requests at arbitrary positions\. Leveraging attention kernels tailored for prefilling and decoding, our system achieves significant efficiency improvements: under skewed document distributions, it reduces time\-to\-first\-token \(TTFT\) by 1\.37$\\times$ and increases inference throughput by 1\.40$\\times$ compared to the state\-of\-the\-art Block\-Attention, while maintaining comparable output quality\. ## Submission history From: Haocheng Xia \[[view email](https://arxiv.org/show-email/dd48bd35/2606.04302)\] **\[v1\]**Wed, 3 Jun 2026 00:12:22 UTC \(103 KB\)
Similar Articles
In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective
This paper studies retrieval-augmented generation as an in-context optimization process, showing that linear self-attention can implement gradient descent on a unified RAG objective. It proposes a lightweight method for frozen RAG LLMs that predicts context-conditioned updates, improving performance across multiple QA benchmarks.
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
The paper proposes Random Attention, a KV cache eviction method that uses random selection instead of scoring, matching selective methods while improving throughput in reasoning tasks.
@ethantsliu: LLMs can control their own attention for long-context! During text generation, LLMs typically read the full KV cache at…
The paper introduces declarative attention, a technique where LLMs explicitly declare which context segments to attend to, reducing token usage by up to 52% with minimal accuracy drops.
@omarsar0: Banger paper from Google DeepMind and colleagues. (bookmark it) A model reads its entire KV cache on every generated to…
This paper introduces Declarative Attention, a protocol that allows language models to declare where to attend in their chain-of-thought, reducing attended tokens by 52% on Gemma-4-31B and 31.1% on Qwen-3.6-27B with minimal accuracy drops.
CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
CompactAttention introduces Block-Union KV Selection to accelerate chunked prefill for long-context LLMs, achieving up to 2.72x attention speedup on LLaMA-3.1-8B at 128K context while maintaining accuracy close to dense attention.