SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

# SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
Source: [https://arxiv.org/abs/2606.04511](https://arxiv.org/abs/2606.04511)
[View PDF](https://arxiv.org/pdf/2606.04511)

> Abstract:Sparse attention reduces compute and memory bandwidth for long\-context LLM inference\. However, two key challenges remain: \(1\) KV cache capacity still grows with sequence length, and offloading to CPU memory introduces a PCIe transfer bottleneck; \(2\) the sparse selection step itself retains $O\(T^2\)$ complexity and can dominate attention cost at long contexts\. We propose SparDA, a decoupled sparse attention architecture that introduces a fourth per\-layer projection, the Forecast, alongside Query, Key, and Value\. The Forecast predicts the KV blocks needed by the next layer, enabling lookahead selection that overlaps CPU\-to\-GPU prefetch with current\-layer execution\. Because Forecast is decoupled from the attention query, our GQA implementation uses one Forecast head per GQA group, reducing selection overhead versus the original multi\-head selector\. SparDA adds $<$0\.5% parameters and trains only the Forecast projections by matching the original selector's attention distribution\. On two sparse\-pretrained 8B models, SparDA matches or slightly improves accuracy and delivers up to 1\.25$\\times$ prefill speedup and 1\.7$\\times$ decode speedup over the sparse\-attention offload baseline\. By enabling larger feasible batch sizes on a single GPU, SparDA further reaches up to 5\.3$\\times$ higher decode throughput than the non\-offload sparse baseline\. Our source code is available at[this https URL](https://github.com/NVlabs/SparDA)\.

## Submission history

From: Yaosheng Fu \[[view email](https://arxiv.org/show-email/481e6733/2606.04511)\] **\[v1\]**Wed, 3 Jun 2026 06:42:05 UTC \(328 KB\)