Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
Summary
RTPurbo leverages intrinsic sparsity in full-attention LLMs to achieve efficient long-context inference with minimal training overhead, enabling significant speedups while maintaining near-lossless accuracy.
View Cached Full Text
Cached at: 05/22/26, 06:23 AM
Paper page - Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
Source: https://huggingface.co/papers/2605.16928
Abstract
RTPurbo leverages intrinsic sparsity in full-attention LLMs to achieve efficient long-context inference with minimal training overhead, enabling significant speedups while maintaining near-lossless accuracy.
Long-context inferencein large language models is bottlenecked by the quadratic cost offull attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset ofattention headstruly requires full long-context processing; (2) long-range retrieval is governed primarily by a low-dimensional subspace, allowing relevant tokens to be retrieved efficiently with a 16-dimensional indexer; and (3) the useful token budget is strongly query-dependent, makingdynamic top-p selectionmore suitable than fixed top-k sparsification. Based on these insights, we proposeRTPurbo, which retains the fullKV cacheonly for retrieval heads and introduces a lightweighttoken indexerforsparse attention. By exploiting the model’sintrinsic sparsity,RTPurboachieves sparsification with only a few hundred training steps. Experiments on long-context benchmarks and reasoning tasks show thatRTPurbopreserves near-lossless accuracy while delivering substantial efficiency gains, including up to a 9.36timesprefill speedupat 1M context and about a 2.01timesdecode speedup. These results suggest that strong sparse inference can be obtained from standard full-attention training without expensive native sparse pretraining.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.16928
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.16928 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.16928 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.16928 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
I built a 2.5D visual compiler for AI agents: It separates topology from geometry so LLMs stop generating spaghetti diagrams.
An open-source 2.5D diagram engine in Go that separates topology from geometry to enable LLMs to generate clean architecture diagrams without spatial hallucinations.
SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference
SharQ introduces a training-free method combining activation sparsity and FP4 quantization for LLM inference, using sparse-dense decomposition and a unified FP4 weight payload. It achieves significant latency reduction and accuracy recovery over FP4-only baselines.
Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudication
This benchmark study evaluates 46 large language models against human experts for coding qualitative humanitarian data, finding that LLMs can achieve comparable reliability with structured prompts and reasoning, but require careful oversight for nuanced themes.
Epiphany-Aware KV Cache Eviction Without the Attention Matrix
This paper introduces EpiKV, a KV cache eviction method that scores token importance via changes in internal representations (epiphany score) instead of attention weights, avoiding the need to materialize the attention matrix. It achieves competitive performance on reasoning benchmarks while enabling up to 16× longer context lengths.
Information-Aware KV Cache Compression for Long Reasoning
This paper proposes InfoKV, an entropy-aware KV cache compression framework that combines token-level predictive uncertainty with attention scores to improve long-context reasoning efficiency. Experiments show it outperforms existing attention-based methods on Llama-3.1, Llama-3.2, and DeepSeek-R1.