LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding
Summary
LazyAttention introduces a novel attention mechanism that defers positional encoding to enable zero-copy, position-agnostic KV cache reuse across multiple requests. The approach reduces time-to-first-token by 1.37× and increases throughput by 1.40× compared to Block-Attention in RAG settings with skewed document distributions.
View Cached Full Text
Cached at: 06/05/26, 02:13 AM
# LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding Source: [https://arxiv.org/abs/2606.04302](https://arxiv.org/abs/2606.04302) [View PDF](https://arxiv.org/pdf/2606.04302) > Abstract:Key\-value \(KV\) caching accelerates inference of large language models \(LLMs\) by reusing past computations for generated tokens\. Its importance becomes even greater in long\-context applications such as retrieval\-augmented generation \(RAG\) and in\-context learning \(ICL\)\. However, conventional KV caching embeds positional information directly into the cache, limiting its reusability\. Existing solutions either restrict reuse to prefixes or require expensive memory materialization for positional re\-encoding\. We introduce LazyAttention, a novel attention mechanism that kernelizes deferred positional encoding to enable zero\-copy, position\-agnostic KV reuse\. By adjusting positional encoding within attention kernels on\-the\-fly, LazyAttention resolves the materialization bottleneck, allowing a single physical KV copy to serve multiple logical requests at arbitrary positions\. Leveraging attention kernels tailored for prefilling and decoding, our system achieves significant efficiency improvements: under skewed document distributions, it reduces time\-to\-first\-token \(TTFT\) by 1\.37$\\times$ and increases inference throughput by 1\.40$\\times$ compared to the state\-of\-the\-art Block\-Attention, while maintaining comparable output quality\. ## Submission history From: Haocheng Xia \[[view email](https://arxiv.org/show-email/dd48bd35/2606.04302)\] **\[v1\]**Wed, 3 Jun 2026 00:12:22 UTC \(103 KB\)
Similar Articles
In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective
This paper studies retrieval-augmented generation as an in-context optimization process, showing that linear self-attention can implement gradient descent on a unified RAG objective. It proposes a lightweight method for frozen RAG LLMs that predicts context-conditioned updates, improving performance across multiple QA benchmarks.
CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
CompactAttention introduces Block-Union KV Selection to accelerate chunked prefill for long-context LLMs, achieving up to 2.72x attention speedup on LLaMA-3.1-8B at 128K context while maintaining accuracy close to dense attention.
SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
SparDA proposes a decoupled sparse attention architecture that adds a lightweight 'Forecast' projection to predict future KV cache needs, enabling lookahead prefetching from CPU to GPU and reducing selection overhead. On 8B sparse-pretrained models, it achieves up to 1.25× prefill and 1.7× decode speedup, with up to 5.3× higher decode throughput over non-offload baselines.
Dynamic Linear Attention
This paper proposes DLA, a dynamic memory modeling framework for multi-state linear attention that adaptively merges states based on token information variation and maintains a fixed-size state cache, enabling better long-context representation without the quadratic complexity of standard attention.
ART: Attention Run-time Termination for Efficient Large Language Model Decoding
This paper proposes ART, a lightweight run-time mechanism that tracks accumulated attention outputs during LLM decoding and terminates unnecessary KV block accesses, achieving 20% higher generation throughput with comparable accuracy.