Inference Time Context Sparsity: Illusion or Opportunity?

# Inference Time Context Sparsity: Illusion or Opportunity?
Source: [https://arxiv.org/html/2605.24168](https://arxiv.org/html/2605.24168)
Sahil Joshiα,∗Prithvi Dixitβ,∗Agniva ChowdhuryαAnshumali Shrivastavaα Joseph E\. GonzalezβIon StoicaβKumar Krishna Agrawalβ,†Aditya Desaiγ,† αRice UniversityβUC BerkeleyγIIT Bombay ###### Abstract Sparsity has long been a central theme in LLM efficiency, but its role in context processing remains unresolved\. As LLM workloads shift toward longer contexts and agentic interactions, the compute and memory bottlenecks of attention become increasingly critical, raising the question of whether these constraints are fundamental\. Our position is that these constraints are artificial and unnecessary, and that the future of LLM inference lies in extreme but principled sparsity along the context dimension\. This position is supported by several strands of empirical and theoretical evidence\. First, we find the insistence on dense attention unreasonable, since in a long context a query effectively projects O\(N\) attention information into a hidden space of dimensiond≪Nd\\ll N, making the process inherently lossy\. Second, we perform an extensive study of sparsity in LLMs spanning 20 models across five model families, varying context lengths, and different sparsity levels\. We empirically demonstrate a strong trend: current LLMs, despite not being trained for context sparsity, are remarkably robust to inference\-time decode sparsity across tasks of varying complexity, including retrieval, multi\-hop QA, mathematical reasoning, and agentic coding\. For instance, Qwen3\.5\-27B can tolerate up to 100× sparsity on benchmarks such as RULER\-HARD and AIME2025 without loss of quality, and up to 50× sparsity on LOFT and SWE with only a small drop in performance\. These results suggest that a transition to complete sparsity may be possible without meaningful loss of capability\. Importantly, we also show that current hardware is already sufficient to realize substantial gains from this sparsity\. For example, our sparse decode kernels accelerate large\-context processing by up to 10× over FlashInfer at 50× sparsity levels on hardware such as the H100\. Overall, these results position extreme context sparsity not as a heuristic, but as a principled foundation for LLM inference, training, and architecture design: one that is both feasible and beneficial, and a compelling direction for future systems\. Code:[https://github\.com/skylight\-org/sparse\-attention\-hub](https://github.com/skylight-org/sparse-attention-hub) Project page:[https://sky\-light\.eecs\.berkeley\.edu](https://sky-light.eecs.berkeley.edu/)

11footnotetext:Equal contribution\.22footnotetext:Co\-lead:kagrawal@berkeley\.edu,apdesai@cse\.iitb\.ac\.in\.## 1Introduction

![Refer to caption](https://arxiv.org/html/2605.24168v1/x1.png)\(a\)Inference\-time attention I/O across decode regimes\.
![Refer to caption](https://arxiv.org/html/2605.24168v1/arxiv_images/fig1_workload_spectrum.png)\(b\)Qwen3\.5\-27B across workloads\.

Figure 1:Inference\-time50×50\\timescontext sparsity is bandwidth\-friendly by construction \(a\) and retains near\-dense quality across diverse workloads on a single model \(b\)\.\(a\)Three decode regimes share an HBM band but read from it differently\.*Dense*reads the full KV cache every step \(𝒪​\(N⋅d\)\\mathcal\{O\}\(N\{\\cdot\}d\)bytes\);*Sparse*routes through a lightweight indexer that selectskkrows \(𝒪​\(k⋅d\)\\mathcal\{O\}\(k\{\\cdot\}d\)bytes,k≪Nk\{\\ll\}N\);*Linear*\(Gated DeltaNet\) reads a fixed\-size recurrent stateSS\(𝒪​\(d2\)\\mathcal\{O\}\(d^\{2\}\)bytes, constant inNN\)\. All three are memory\-bandwidth bound on H100/B200; the contrast is whether per\-step traffic scales with context length\.\(b\)Qwen3\.5\-27B on four workloads ordered by increasing complexity along\(T,D,C\)\(T,D,C\)– turns, decoded tokens per turn, input context per turn \(§[3](https://arxiv.org/html/2605.24168#S3)\)\. Configurations:RULER\-HARD\-32K\(T=1,D<1\(T\{=\}1,\\ D\{<\}1K,C=32,\\ C\{=\}32K\)\);LOFT\-128K\(T=1,D<1\(T\{=\}1,\\ D\{<\}1K,C=128,\\ C\{=\}128K\)\);AIME\-2025\(T=1,D≈25\(T\{=\}1,\\ D\{\\approx\}25K,C<1,\\ C\{<\}1K\)\);SWE\-Bench Django\(T≈67,D∼1\(T\{\\approx\}67,\\ D\{\\sim\}1K per turn,C,\\ Cgrows to\>100\{\>\}100K\)\)\. Sparse within≤2\\leq\\\!2points of dense at50×50\\timeson retrieval \(RULER, LOFT\), reasoning \(AIME\), and agentic coding \(SWE\-BenchS3S\_\{3\}subset,n=58n\{=\}58, App\.[B](https://arxiv.org/html/2605.24168#A2)\)\. Kernel\-level speedup on H100/B200 is deferred to §[4](https://arxiv.org/html/2605.24168#S4)and Tab\.[1](https://arxiv.org/html/2605.24168#S3.T1)\.Sparsity in LLM inference has long been a goal for researchers\. Efforts to sparsify the FFN component of Transformers\[[5](https://arxiv.org/html/2605.24168#bib.bib129),[10](https://arxiv.org/html/2605.24168#bib.bib32),[32](https://arxiv.org/html/2605.24168#bib.bib128),[39](https://arxiv.org/html/2605.24168#bib.bib21)\]have largely converged on Mixture\-of\-Experts \(MoE\) as the de facto architecture across frontier labs\[[2](https://arxiv.org/html/2605.24168#bib.bib135),[27](https://arxiv.org/html/2605.24168#bib.bib133),[40](https://arxiv.org/html/2605.24168#bib.bib136),[46](https://arxiv.org/html/2605.24168#bib.bib132),[54](https://arxiv.org/html/2605.24168#bib.bib134)\]\. In contrast, a similar consensus has yet to emerge for attention mechanisms, or more broadly, context processors\. Research on sparsity in attention dates back several years and has recently re\-emerged in the LLM era, primarily focusing on emergent sparsity in trained models\[[1](https://arxiv.org/html/2605.24168#bib.bib80),[6](https://arxiv.org/html/2605.24168#bib.bib35),[8](https://arxiv.org/html/2605.24168#bib.bib74),[9](https://arxiv.org/html/2605.24168#bib.bib115),[14](https://arxiv.org/html/2605.24168#bib.bib36),[20](https://arxiv.org/html/2605.24168#bib.bib34),[24](https://arxiv.org/html/2605.24168#bib.bib77),[30](https://arxiv.org/html/2605.24168#bib.bib76),[31](https://arxiv.org/html/2605.24168#bib.bib2),[42](https://arxiv.org/html/2605.24168#bib.bib27),[48](https://arxiv.org/html/2605.24168#bib.bib78),[55](https://arxiv.org/html/2605.24168#bib.bib116),[56](https://arxiv.org/html/2605.24168#bib.bib73),[57](https://arxiv.org/html/2605.24168#bib.bib25)\]\. However, such sparsity is rarely adopted in state\-of\-the\-art inference engines, highlighting a limited understanding of its practical value\. There has been some consolidation of this line of work through the adoption of sparse attention during post\-training\[[28](https://arxiv.org/html/2605.24168#bib.bib37),[54](https://arxiv.org/html/2605.24168#bib.bib134)\]\. However, demonstrated gains are largely confined to extremely large model regimes, raising questions about their general applicability\. In parallel, an alternative form of sparsity has emerged through the development of lightweight context processors, such as linear attention\[[7](https://arxiv.org/html/2605.24168#bib.bib38),[21](https://arxiv.org/html/2605.24168#bib.bib69),[22](https://arxiv.org/html/2605.24168#bib.bib40),[34](https://arxiv.org/html/2605.24168#bib.bib71),[53](https://arxiv.org/html/2605.24168#bib.bib70)\], and SSMs\[[12](https://arxiv.org/html/2605.24168#bib.bib39),[23](https://arxiv.org/html/2605.24168#bib.bib68),[50](https://arxiv.org/html/2605.24168#bib.bib66),[52](https://arxiv.org/html/2605.24168#bib.bib67)\]\. While these approaches have seen some adoption, modern architectures still retain full scaled dot\-product attention \(SDPA\) layers, underscoring the limited expressivity of purely SSM\-based models\. This leads to a fundamental question: are the quadratic compute bottleneck during prefill and the linear memory bottleneck during decoding inherent constraints that are here to stay? This question is becoming increasingly important as LLM workloads shift toward substantially longer contexts and generations\. Emerging use cases such as agentic tool use, code generation, retrieval augmented generation, multi document reasoning, long form dialogue, and repository scale software understanding are all driving this trend\[[13](https://arxiv.org/html/2605.24168#bib.bib60),[15](https://arxiv.org/html/2605.24168#bib.bib64),[17](https://arxiv.org/html/2605.24168#bib.bib54),[18](https://arxiv.org/html/2605.24168#bib.bib59),[26](https://arxiv.org/html/2605.24168#bib.bib62),[35](https://arxiv.org/html/2605.24168#bib.bib41),[36](https://arxiv.org/html/2605.24168#bib.bib61),[37](https://arxiv.org/html/2605.24168#bib.bib65),[38](https://arxiv.org/html/2605.24168#bib.bib63),[45](https://arxiv.org/html/2605.24168#bib.bib1),[49](https://arxiv.org/html/2605.24168#bib.bib53)\]\. To ground this in concrete terms, a single 50 page PDF can contain on the order of 33K tokens\. An LLM that aims to condition its responses on corporate or legal documents may therefore need to process hundreds of thousands, if not millions, of tokens within a single context window\. At the same time, expectations for state of the art models continue to increase, with each generation pushing toward larger and more persistent context handling\. Recent examples further highlight this shift\. Systems such as openclaw\[[41](https://arxiv.org/html/2605.24168#bib.bib55)\]have been reported to use system prompts reaching 160K tokens111https://github\.com/openclaw/openclaw/issues/21999, illustrating how rapidly context sizes are expanding in practice\. Importantly, these workloads do not merely involve longer prompts, but also often require long form generation over extended histories\. This challenge is further amplified in agentic settings, where models iteratively generate outputs, incorporate new information, and repeatedly condition on growing interaction traces among agents\. As a result, the question becomes of utmost importance: what does the future of LLM Inference looks like in context dimension?

We take the following position:The future of LLM inference lies in extreme sparsity along the context dimension\.Our position is primarily motivated by our large\-scale empirical findings on inference\-time sparsity, as well as the substantial speedups we demonstrate over FlashInfer even under sparsity patterns that are arguably highly irregular\.

#### Sparsity already exists in new generation models:

In Section[3](https://arxiv.org/html/2605.24168#S3)we observe empirically that sparsity is already emerging in modern architectures\. Larger and more recent models exhibit remarkable robustness to aggressive context sparsification across a wide spectrum of tasks as shown in Section[3](https://arxiv.org/html/2605.24168#S3)\. This holds for relatively simpler benchmarks such as RULER\-HARD, a challenging subset of RULER\[[16](https://arxiv.org/html/2605.24168#bib.bib42)\], and LOFT\[[25](https://arxiv.org/html/2605.24168#bib.bib50)\], as well as for more complex reasoning tasks such as AIME\[[33](https://arxiv.org/html/2605.24168#bib.bib121)\]and real\-world agentic workloads such as SWE\[[19](https://arxiv.org/html/2605.24168#bib.bib51)\]\(50\+ agentic turns\)\. To our knowledge, this is the first study to examine inference\-time sparsity on an agentic workload\. Of course, inducing sparsity during training is ideal, but the emergence of sparsity at inference time reinforces the possibility that models can operate effectively in highly sparse regimes and provides greater confidence in designing training procedures that explicitly target sparsity\. We empirically evaluate five model families, including Llama3\[[11](https://arxiv.org/html/2605.24168#bib.bib97)\], Qwen2\.5\[[47](https://arxiv.org/html/2605.24168#bib.bib95)\], Qwen3\.5\[[44](https://arxiv.org/html/2605.24168#bib.bib130)\], Gemma3\[[43](https://arxiv.org/html/2605.24168#bib.bib96)\], and Ministral3\[[29](https://arxiv.org/html/2605.24168#bib.bib131)\], across four tasks: RULER\-HARD \(32K\), LOFT \(32K and 128K\), AIME \(65K generation length\), and SWE \(50\+ agentic turns\)\. Our results show that the effectiveness of inference\-time sparsity improves with both model scale and the use of hybrid architectures\. In particular, larger models from newer\-generation families such as Qwen3\.5, Gemma3, and Ministral3 sustain quality parity with dense execution even at 50×\\timessparsity\. For smaller standard models, the observed degradation can largely be mitigated through stochastic index selection\[[8](https://arxiv.org/html/2605.24168#bib.bib74)\]\. Notably, these gains are achieved purely at inference time, suggesting that incorporating sparsity during training could yield even greater benefits\.

#### Sparsity can be leveraged for system gains:

The alignment of sparsity with hardware is of paramount importance for fully realizing its benefits\. It is therefore essential to evaluate whether sparsity actually alleviates the underlying bottlenecks\. A common belief is that efficiency gains require block\-structured sparsity\[[58](https://arxiv.org/html/2605.24168#bib.bib58)\]; without it, sparsity is unlikely to translate into real speedups\. We present an argument against this view\. Notably, DeepSeek Attention\[[28](https://arxiv.org/html/2605.24168#bib.bib37)\]demonstrates both training\- and inference\-time speedups with token\-level sparsity during prefill\. This serves as evidence that token\-level sparsity can indeed translate into practical efficiency gains without imposing excessive structure on the model\. We extend this further to decoding\. For decoding, we provide kernels that accelerate inference under per\-token, per\-query, and per\-head sparsity, i\.e\., highly irregular sparsity patterns\. These gains persist even under grouped query attention, where the number of query heads can be a factor \(typically 4\) larger than the number of key\-value heads\. This is possible because the KV cache’s vector dimension provides sufficient contiguous memory to make such sparsity effective on modern hardware\. In particular, our optimized sparse attention kernel, built on top of FlashInfer with a paged KV\-cache backend, achieves up to10×10\\timeskernel speedup at50×50\\timessparsity for large batch sizes\.

#### Non\-existence of a truly dense attention:

Apart from the empirically strong results, our position and research is strongly rooted in the idea that dense attention is incompatible with long context\. We show a simple result that truly dense attention does not exist in practice: full attention is ultimately bottlenecked by the hidden dimension, causing it to collapse unable to distinguish between varying attention distributions\. While the result itself is straightforward, the implications are significant\. It suggests that complete sparsity is not merely a practical approximation but principally a superior objective\. Overall, this paper argues that the community should actively explore extreme sparsity along the context dimension without compromises such as partially retaining full attention layers\. The empirical emergence of sparsity across multiple axes highlights strong potential, and decode\-time sparse kernel analyses in Section[3](https://arxiv.org/html/2605.24168#S3), augmented by results from recent DeepSeek models, demonstrate the significant efficiency gains that can be unlocked\.

## 2Why Dense Attention is not meant for Long Context

We begin by examining whether attention layers can remain truly dense as context length grows, and show that they cannot\.

### 2\.1Dense Attention Collapses Through The Hidden Dimension\.

While sparse attention and SSMs have their own shortcomings, fully dense attention has its own bottleneck: it assigns a weight to every context token, but the layer passes forward only a fixed\-dimensional hidden vector\.

###### Theorem 1\.

LetV∈ℝN×dV\\in\\mathbb\{R\}^\{N\\times d\}be any value matrix, and let the dense attention output beo=V⊤​ao=V^\{\\top\}a, wherea=\(a1,…​aN\)a=\(a\_\{1\},\\dots a\_\{N\}\)is an attention distribution overNNcontext tokens\. Ifd<N−1d<N\-1, then this map is not injective on the attention simplex\. In particular, there exist two distinct dense attention distributionsa,a′a,a^\{\\prime\}such thata≠a′a\\neq a^\{\\prime\}, andV⊤​a=V⊤​a′V^\{\\top\}a=V^\{\\top\}a^\{\\prime\}\. Thus, add\-dimensional post\-attention embedding cannot preserve all dense attention\-score variations over more thand\+1d\+1context tokens\.

###### Corollary 2\.

If all dense attention distributions overNNcontext tokens must remain distinguishable after the attention outputV⊤​aV^\{\\top\}a, then the hidden dimension must satisfyd≥N−1d\\geq N\-1\. Thus, losslessly preserving arbitrary dense attention over a million\-token context requires million\-scale hidden width\.

This is the embedding bottleneck\. Whend≪Nd\\ll N, the mapa↦V⊤​aa\\mapsto V^\{\\top\}acollapses many distinct attention patterns into the same hidden representation; hence not every fine\-grained variation in a dense attention distribution can be carried forward by the post\-attention vector\. This conclusion is consistent with lower bounds for indexed lookup: finite\-precision recurrent models, including RNNs, LSTMs, state\-space models, and recurrent linear attention, require hidden\-state sizeΩ​\(N\)\\Omega\(N\)to recover arbitrary tokens from anNN\-token sequence\[[4](https://arxiv.org/html/2605.24168#bib.bib137)\]\. These observations motivate a complete context\-sparse attention\.

![Refer to caption](https://arxiv.org/html/2605.24168v1/arxiv_images/plot_ruler_main_v2.png)Figure 2:RULER\-HARD\-32K score across families, scales, and sparsity\.Scores are absolute \(averaged over the six RULER\-HARD subtasks\)\. Squares mark dense \(1×1\\times\); filled circles mark sparse \(5×5\\timesand50×50\\timesoracle top\-kk\)\. Each line is one checkpoint; lighter shade within a family means smaller scale\. Panels \(a\) and \(b\) share the x\-axis\.\(a\) Standard families\.Qwen2\.5 \(blue\) and Ministral 3 \(amber\) fan out as sparsity grows: Qwen2\.5\-1\.5B drops from6060at dense to3838at50×50\\times, while Qwen2\.5\-72B holds within33points across the full sweep\.\(b\) Hybrid families\.Gemma\-3 \(green, sliding\-window\) and Qwen3\.5 \(red, linear\-attention\) stay essentially flat at every scale; the small\-checkpoint penalty visible in \(a\) is absent\.\(c\) Qwen3\.5 saturation under extreme top\-KK\.Retention==score/K\{\}\_\{K\}/scoredense\{\}\_\{\\text\{dense\}\}, computed per subtask and then averaged\.KKretrieved tokens with6464sinks always retained,K∈\{1,4,16,64,128\}K\\\!\\in\\\!\\\{1,4,16,64,128\\\}\. Larger Qwen3\.5 scales saturate earlier \(27B reaches near\-parity byK=4K\{=\}4\); 0\.8B is still climbing atK=128K\{=\}128\(0\.87 at the right edge\) — the knee migrates with scale\. Three Qwen3\.5\-27B cells in the parsed CSV were unevaluated and imputed at ceiling continuation of adjacent rates \(nm3@5×\\times,nm2@10×\\times→100\\to 100;qa2@20×\\times→63\\to 63\); effect on the 6\-subtask average is<1<\\\!1point\. LLaMA\-3 omitted: no per\-subtask RULER\-HARD data in the current eval set\.

## 3Emergent Sparsity Observed in State\-of\-the\-art Models

We now turn to empirical evidence showing that even under current training recipes and latest architectures, where attention is not explicitly trained to be sparse, sparsity emerges regardless\. This reinforces our proposal: it suggests that in principle, models can transition to extremely sparse context processing without any loss in capability\. We adopt the following evaluation setup\. Algorithms:We enforce sparse attention at decode time and measure the effect across models, architectures, and tasks\. The primary mechanism is exact*oracle*top\-kkselection \(to eliminate confounds from approximate top\-kkindexers\); in selected experiments we also report stochastic indexing via vAttention\[[8](https://arxiv.org/html/2605.24168#bib.bib74)\]\. Datasets:For long\-context retrieval, we report the average score on RULER\-32K\-HARD, a challenging subset consisting of six tasks from RULER\-32K\[[16](https://arxiv.org/html/2605.24168#bib.bib42)\]\(fwe,qa1,qa2,vt,nm2, andnm3\)\. On the LOFT\[[25](https://arxiv.org/html/2605.24168#bib.bib50)\]benchmark, we evaluate long\-context performance by averaging results over five datasets \(hotpotqa,nq,musique,qampari, andquest\) at both 32K and 128K context lengths\. To assess sparsity in long\-form generation settings, we evaluate the largest hybrid model, Qwen3\.5\-27B\[[44](https://arxiv.org/html/2605.24168#bib.bib130)\], on AIME2025\[[33](https://arxiv.org/html/2605.24168#bib.bib121)\], where we limit the generation length to 65K tokens\. For agentic workloads, we evaluate Qwen3\.5\-27B on the SWE\-Bench Django\[[19](https://arxiv.org/html/2605.24168#bib.bib51)\]subset \(114114tasks,≤100\\leq 100turns per task\) at three sparsity levels: dense,5×5\\timessparsity, and50×50\\timessparsity\. To our knowledge this is the first inference\-time sparsity study on an agentic benchmark\. Models\.We investigate five families, with a total of 20 models: three standard transformers \(Qwen2\.5\[[47](https://arxiv.org/html/2605.24168#bib.bib95)\], Ministral3\[[29](https://arxiv.org/html/2605.24168#bib.bib131)\], and Llama3\[[11](https://arxiv.org/html/2605.24168#bib.bib97)\]\) and two hybrids \(Qwen3\.5\[[44](https://arxiv.org/html/2605.24168#bib.bib130)\]and Gemma3\[[43](https://arxiv.org/html/2605.24168#bib.bib96)\]\), enabling a controlled comparison of sparsity behavior across architectural variants\.

![Refer to caption](https://arxiv.org/html/2605.24168v1/arxiv_images/plot_vattention_1.png)Figure 3:vAttention vs OracleTopK retention at50×50\\timessparsity on RULER\-HARD\-32K\.Bars report relative score \(sparse / dense\); the horizontal line at1\.01\.0marks dense parity\. Values≥1\.0\\geq\\\!1\.0are bold green; values<0\.90<\\\!0\.90are muted red\. Panels separate model family \(\(a\) Qwen2\.5, \(b\) Ministral 3\)\. Traditionally, the failure of top\-kksparse attention has been attributed to the diffusion of attention scores across the context\. However, this does not necessarily imply that attention must remain dense; rather, it suggests the need for a different mechanism for selecting relevant context\. We observe that stochastic index\-selection \(vAttention\) closely tracks dense parity at every Qwen2\.5 and Ministral checkpoint, while deterministic OracleTopK collapses on smaller standard models \(Qwen2\.5\-1\.5B drops to∼\\sim0\.630\.63of dense\)\. This suggests that the primary limitation of conventional top\-kksparsification lies not in sparsity itself, but in the determinism and locality of the selection mechanism\.#### RULER\-32K\(Hard Subset\)

On the hard subset of RULER, we observe two consistent trends across model families in Figure[2](https://arxiv.org/html/2605.24168#S2.F2)\. First, hybrid architectures such as Qwen3\.5 and Gemma3 exhibit significantly greater robustness to context sparsity, maintaining performance even at up to 50× sparsity with little to no degradation in quality\. Interestingly, this robustness appears largely invariant to model scale within these families, suggesting that the inclusion of SSM or linear\-attention layers may inherently improve resilience to sparse context retrieval\. We further evaluate the Qwen3\.5 \(See Figure[2](https://arxiv.org/html/2605.24168#S2.F2)\) family of hybrid models under extreme context sparsity, where the model is restricted to using only a very small number of retrieved tokens \(1–128 tokens\)\. Even in these highly constrained regimes, larger hybrid models exhibit striking robustness to sparsity\. In particular, restricting attention to just 128 retrieved tokens corresponds to approximately 250× sparsity while still preserving strong performance\. Given the broader trajectory of state\-of\-the\-art LLMs toward larger parameter scales and increasingly hybrid architectures, these findings suggest that future inference systems may rely more heavily on extremely sparse context processing rather than dense attention across the entire context window\. Second, for standard dense\-attention architectures such as Llama3, Qwen2\.5, and Ministral3, robustness to sparsity improves steadily with model size as seen in Figure[2](https://arxiv.org/html/2605.24168#S2.F2)\. In particular, the largest models in these families are able to preserve quality even under 50× sparsity\. This suggests that scaling alone may enable stronger implicit retrieval and context localization capabilities, even in the absence of explicit architectural mechanisms for sparse processing\. For smaller standard models, the performance of top\-kkcan be significantly lower than dense model \(see Figure[2](https://arxiv.org/html/2605.24168#S2.F2)\)\. A common explanation for the failure of top\-kksparse attention is that attention mass is often diffusely distributed across the context, making deterministic selection of small number of tokens ineffective\. However, this does not necessarily imply that attention itself must remain dense; rather, it points to the need for more effective context selection mechanisms\. As shown in Figure[3](https://arxiv.org/html/2605.24168#S3.F3), we find that stochastic index\-selection approaches such as vAttention are able to nearly recover the quality of full attention while still operating under extreme sparsity in the context dimension \(up to 50× sparsity in our experiments\)\. These results suggest that the core limitation of conventional top\-kksparsification may stem less from sparsity itself and more from the deterministic and locality\-biased nature of the selection process\.

![Refer to caption](https://arxiv.org/html/2605.24168v1/arxiv_images/plot_loft.png)Figure 4:LOFT subspan\-EM retention under5×5\\timesand50×50\\timesinference\-time sparsity\.Bars report relative score \(sparse Subspan\-EM / dense Subspan\-EM\); the horizontal line at1\.01\.0marks dense parity\. Values≥1\.0\\geq\\\!1\.0\(sparse meets or exceeds dense\) are bold green; values<0\.90<\\\!0\.90\(over a 10% relative drop\) are muted red\. Layout: rows separate context length \(top: 32K, bottom: 128K\); columns separate model family \(Qwen2\.5: 3B–32B, Ministral3: 3B–14B, Qwen3\.5: 2B–27B\)\. At5×5\\timessparsity, retention sits within∼\\sim2%2\\%of dense across nearly every checkpoint; at50×50\\times, mid\-scale standard checkpoints \(Qwen2\.5\-7B, Ministral\-3B\) suffer the largest drops while the hybrid Qwen3\.5 family stays close to parity except at the smallest scales\. Qwen3\.5\-9B at 32K shows a1\.51×1\.51\\timesratio — a small\-denominator effect on a low\-scoring task — consistent with the qualitative picture from RULER\-HARD that sparsity does not degrade quality once a model is large enough\.
#### LOFT\-32K and LOFT\-128K

To evaluate context sparsity beyond synthetic retrieval settings, we additionally study LOFT, a more natural retrieval and question\-answering benchmark\. LOFT appears to be substantially more challenging than RULER\-HARD, with even the dense baseline models achieving relatively modest performance\. Despite this increased difficulty, the qualitative trends with respect to sparsity remain largely consistent with those observed on RULER\-HARD\. To further study the effect of sequence length, we evaluate LOFT under two context regimes: 32K and 128K tokens\. Across both settings, robustness to sparsity remains remarkably stable\. Intuitively, one might expect longer contexts to naturally induce greater effective sparsity, especially for retrieval\-oriented tasks where only a small subset of tokens should be relevant to the query\. However, we do not observe a corresponding increase in inference\-time sparsity as context length grows\. This suggests that current models may not automatically adapt their retrieval behavior with increasing context size, pointing to potential opportunities during training to explicitly encourage sparsity to scale with context length\.

#### AIME25

We use AIME 2025 to evaluate the effect of context sparsity on long\-form autoregressive generation\. While we permit generations of up to 65K tokens, models produce roughly 25K tokens on average across evaluation samples, making this setting particularly sensitive to accumulated approximation errors\. Sparse attention introduces perturbations at every embedding update, raising the concern that such errors may compound not only across layers, but also across thousands of autoregressively generated tokens\. Despite these concerns, the results on AIME demonstrate remarkable robustness to aggressive sparsification\. Even over extremely long generations, model quality remains largely stable, suggesting that sparsity\-induced approximation errors do not significantly destabilize long\-horizon reasoning or autoregressive decoding dynamics\. Furthermore, we observe that increasing sparsity results in only a marginal increase in the average number of generated tokens required to solve a task\. This is an important practical observation: it indicates that improvements in per\-token decoding efficiency can translate into genuine end\-to\-end reductions in task completion time, rather than being offset by substantially longer generations\.

![Refer to caption](https://arxiv.org/html/2605.24168v1/arxiv_images/plot_aime_1.png)
![Refer to caption](https://arxiv.org/html/2605.24168v1/arxiv_images/plot_aime2.png)

Figure 5:We use AIME 2025 for evaluating long\-form generation\. Although we allow generations up to 65K tokens, models generate approximately 25K tokens on average across samples\. Since sparse attention introduces approximation errors at each embedding update, an important concern is whether these errors compound across layers and, more critically, across autoregressively generated tokens\. The results on AIME are therefore particularly promising: they indicate that models remain robust to sparsity\-induced errors even over very long generations, suggesting that aggressive context sparsification does not necessarily destabilize long\-horizon reasoning or autoregressive decoding\. Additionally, we observe that increasing sparsity leads to only a slight increase in the average number of generated tokens per task\. This suggests that the gains observed in decoding speed can translate into meaningful end\-to\-end task completion speedups in practice\.
#### SWE\-Bench \(Django\)

![Refer to caption](https://arxiv.org/html/2605.24168v1/arxiv_images/swe_main_v2.png)Figure 6:SWE\-Bench Django head\-to\-head, Qwen3\.5\-27B under dense,5×5\\times, and50×50\\timessparsity\.Left: resolution rate on two nested subsets \(solid:n=58n\{=\}58where all three produced a valid eval verdict; hatched:n=70n\{=\}70that also admits tasks where any config exhausted its250250\-turn budget\)\. Sparse matches dense within∼\\sim22points on the strict subset; the∼\\sim44\-point gap on the broader subset is driven by sparse runs occasionally collapsing into degenerate command loops\. Middle: mean agent turns per task on the strictn=58n\{=\}58subset \(error bars: SEM\)\. Right: mean prompt tokens per LLM call as a function of turn index, averaged acrossn=58n\{=\}58tasks that reached that turn \(shaded band:±\\pmSEM; tail truncated where fewer than six tasks remain\); the sparse curves track dense closely with a small \(∼\\sim6%6\\%\) per\-call offset that grows with context length\. Full subgroup table, per\-task cost, and outcome composition \(including the step\-limit failure mode\) are in Appendix[B](https://arxiv.org/html/2605.24168#A2)\.SWE\-bench\[[19](https://arxiv.org/html/2605.24168#bib.bib51)\]is a benchmark of2,2942\{,\}294real\-world GitHub issues drawn from1212popular Python repositories\. Each task instance pairs an*issue*\(a bug report or feature request,195195words on average\) with the corresponding repository snapshot at the base commit \(codebases average3,0103\{,\}010non\-test files and438438K lines of code\) and a set of*fail\-to\-pass*tests; the task is to produce a patch that resolves the issue and passes the tests\. Gold patches average32\.832\.8lines spread over1\.71\.7files and33functions, so each instance reduces to identifying a small set of edits within a large body of context\. We evaluate on the114114\-task Django subset of SWE\-Bench Lite, the curated variant of the original benchmark that biases toward single\-file gold patches\. Agentic compute profile:We run Qwen3\.5\-27B as a tool\-using coding agent inside themini\-swe\-agentharness, with a limit of250250turns per task\. At every step, the agent sees its complete conversation history \(issue text, prior tool calls, and their stdout/stderr\), emits one new tool invocation, and the harness appends the tool output to the next prompt\. Consequently, the effective context length grows monotonically from∼\\sim11K tokens of issue text and repository metadata at turn one, to∼\\sim2020K tokens by turn6060on a typical resolved trajectory, and exceeds100100K tokens on the tail of trajectories that hit the step limit\. We compare three densities on the Django subset: dense,5×5\\times\(20%\) sparsity, and50×50\\times\(2%2\\%\) sparsity\. To our knowledge this is the first inference\-time sparsity study on an agentic benchmark\. Resolution rate:A clean three\-way comparison requires controlling for runtime errors \(e\.g\., timeouts, server errors\) that can bury an otherwise competent run as an empty patch\. We report two head\-to\-head subsets \(Figure[6](https://arxiv.org/html/2605.24168#S3.F6), left\): the strictn=58n\{=\}58subset where all three configurations produced a valid evaluation verdict, and a broadern=70n\{=\}70subset that also admits tasks where the agent exhausted its250250\-turn budget \(step\-limit hits, an agent\-side failure mode shared across configs; see Appendix[B](https://arxiv.org/html/2605.24168#A2)\)\. On the valid\-eval subset the three configurations resolve within∼\\sim22points of each other \(77\.6%77\.6\\%dense,79\.3%79\.3\\%at5×5\\times,75\.9%75\.9\\%at50×50\\times\): on tasks that all three configurations finish cleanly, sparse matches dense\. On the step\-limit\-inclusive subset the gap widens to∼\\sim44points \(71\.4%71\.4\\%dense vs67\.1%67\.1\\%at both sparse settings\) because sparse runs collapse into degenerate command loops slightly more often, exhausting the turn budget without submitting\. The full subgroup analysis is in Appendix[B](https://arxiv.org/html/2605.24168#A2)\. Why we report a strict subset \(error analysis\):A non\-trivial fraction of trajectories never enter the patch\-quality evaluation because the agent never emits a parseable diff \(empty\_patch\), or emits one the harness cannot apply \(error\)\. These are not attention\-quality signal, so we attribute them to a root cause before deciding what to drop\. Dense’s1313empty\_patchcases are99CalledProcessError\(the per\-instancedocker runreturned exit125/127125/127*before*the agent could start: a daemon\-side launch failure, not a model failure\) plus44LimitsExceeded\(the agent ran but hit the250250\-turn cap without submitting\)\. Sparseempty\_patchis bigger \(3434at5×5\\times,2626at50×50\\times\) and has a different root cause: it is dominated byInternalServerError\(1717and1111, respectively\) andTimeout\(22and44\), both of which are the vLLM server crashing or stalling mid\-trajectory under the sparse\-attention\-hub backend, an engineering instability of the serving stack rather than a property of sparse attention itself\. The remaining sparseempty\_patchcases areLimitsExceeded\(1111and1111\);50×50\\timesadditionally has33errorcases where the produced patch was rejected bygit apply\. Because these failure modes either pre\-date the model call \(CalledProcessError\) or are mid\-call infrastructure crashes \(InternalServerError,Timeout\), they bury an otherwise competent run as a zero, and we drop them before comparing patch quality\. Then=58n\{=\}58subset is the strict residual where all three configurations emitted a non\-empty patch that the harness scored; then=70n\{=\}70subset additionally admitsLimitsExceededon the rationale that turn\-budget exhaustion is a shared, model\-side failure mode\. The full per\-config outcome×\\timesexit\-status breakdown is in Appendix[B](https://arxiv.org/html/2605.24168#A2)\.

Table 1:Per\-query, per\-head irregular sparse decode is up to76×76\\timesfaster than FlashInfer at extreme sparsitySpeedup over FlashInfer\[[51](https://arxiv.org/html/2605.24168#bib.bib114)\]on H100 80GB HBM3 \(FP16, GQAHq=32H\_\{q\}\{=\}32,Hk​v=8H\_\{kv\}\{=\}8,D=128D\{=\}128, page size 16, NHD, 128K context\)\.S×S\\timessparsity implies each query\-head attends to1/S1/Sfraction of total tokens\. The Speed up of<1×<\\\!1\\timesdenotes overhead\. We generally break\-even at10×10\\timessparsity for every batch with huge speedups of1010–76×76\\timesin the5050–500×500\\timesregime\. Note that no block structure imposed on the sparsity\.Together, these experiments probe sparsity across five key axes, yielding the following generic takeaways:

- •Scale:Performance under top\-kksparsity improves with model size; the gap between sparse and dense decoding largely closes at scale\.
- •Architecture:Hybrid models tolerate sparsity better than standard transformers\. Notably, at high sparsity levels \(e\.g\.,50×50\\timesreduction\), hybrid models maintain strong performance\. In fact for larger hybrid model Qwen3\.5\-27B, even 16\-32 tokens in top\-kkare enough to achieve parity with the dense model on RULER\-HARD\.
- •Context Length:The qualitative behavior of sparsity remains consistent across context lengths\. Observations at 32K and 128K contexts largely align, indicating that sparsity properties generalize to long\-context settings without introducing additional degradation\. A case can be made for more sparsity in longer contexts and it seems like it would need training time modifications to achieve it\.
- •Algorithm:Generally Hybrid models and large models with standard architecture show good results with top\-kksparsity\. However, for smaller models with standard architecture, top\-kksparsity may not suffice\. In such a case, stochastic sparsity with methods such as vAttention can still enable sparsity in attention while maintaining quality\.
- •Task complexity:Qwen3\.5\-27B robustness to sparsity holds across single\-hop retrieval and multi\-hop QA \(RULER\-HARD, LOFT\), mathematical reasoning \(AIME 2025\), and agentic coding \(SWE\-Bench Django\)\. This evaluation showcases that even inference time sparsity does not deteriorate the capability of models in many real world long context applications\.

Table 2:Sparse decode is net\-positive with indexer cost included\. We use DoubleSparsity as an example here\.Speedup over FlashInfer on H100 \(FP16, 128K, page size 16\) using Double Sparsity\[[48](https://arxiv.org/html/2605.24168#bib.bib78)\]\(8×168\\times 16\-bit channels, untuned\) for index selection, under MHA \(Hq=Hk​v=32H\_\{q\}\{=\}H\_\{kv\}\{=\}32\) and GQA \(Hq=32,Hk​v=8H\_\{q\}\{=\}32,H\_\{kv\}\{=\}8\)\. MHA: break\-even at2×2\\timessparsity with4\.17×4\.17\\timesspeedup at100×100\\timessparsity\. GQA: break\-even at10×10\\times, with2\.81×2\.81\\timesspeed up at100×100\\times\. A lighter indexer \(HashAttention, PQCache, low\-precision Double Sparsity\) is expected to widen the margin\. The upper limit can be seen from Table[1](https://arxiv.org/html/2605.24168#S3.T1)

## 4Sparsity and Hardware

A common argument against sparsity concerns its poor alignment with modern hardware, which has led to considerable debate over whether block sparsity is a necessary condition for practical efficiency gains\. DeepSeek\-V3 Attention\[[28](https://arxiv.org/html/2605.24168#bib.bib37)\]demonstrates that token\-level fine\-grained sparsity can yield meaningful efficiency improvements at both prefill and decode time: a result with implications not only for inference but also for training\-time reductions on long\-context workloads\. We take this a step further in the context of decode\-time sparsity\. We show that it is possible to improve upon the state\-of\-the\-art FlashInfer222[flashinfer\.ai](https://arxiv.org/html/2605.24168v1/flashinfer.ai)decoding kernels using an even finer\-grained sparsity pattern: per\-query, per\-query\-head token\-level sparsity\. We find this effective even in challenging settings such as Grouped Query Attention \(GQA\)\[[3](https://arxiv.org/html/2605.24168#bib.bib57)\]where number of query heads can be much larger than key\-value heads\. We benchmark sparse decode backend kernel against a full dense decode baseline on an NVIDIA H100 80GB HBM3 GPU using fp16 precision,Hk​v=8H\_\{kv\}=8,Hq=32H\_\{q\}=32, head dimensionD=128D=128, page size 16, and NHD layout\. Table[1](https://arxiv.org/html/2605.24168#S3.T1)shows the performance of sparse backend which computes the weighted attention given sparse index and associated weights\. It shows that we can leverage even this irregular sparsity\. At5050–100×100\\timessparsity, our backend delivers5\.55\.5–20×20\\timesspeedup over FlashInfer across batch sizes; at extreme500×500\\timessparsity, speedup reaches76×76\\timesat large batch \(Table[1](https://arxiv.org/html/2605.24168#S3.T1)\)\. To include some form of indexing mechanism, we simulate an 8\-channel \(1616\-bit precision\) Double Sparsity\[[48](https://arxiv.org/html/2605.24168#bib.bib78)\]indexer and report results in Table[2](https://arxiv.org/html/2605.24168#S3.T2)\. Double Sparsity achieves up to4\.17×4\.17\\timesspeedup in MHA and2\.81×2\.81\\timesin GQA at100×100\\timessparsity\. MHA crosses break\-even at2×2\\timessparsity; GQA at1010–20×20\\times\. A lighter indexer \(HashAttention\[[9](https://arxiv.org/html/2605.24168#bib.bib115)\], PQCache\[[55](https://arxiv.org/html/2605.24168#bib.bib116)\], or low\-precision Double Sparsity\[[48](https://arxiv.org/html/2605.24168#bib.bib78)\]\) is expected to widen these margins\.

## 5Conclusion

The AI workload landscape is rapidly shifting toward long\-context understanding and long\-form generation, with tasks such as repository\-scale code comprehension, long\-document question answering, and agentic systems becoming increasingly common\. We argue that standard attention mechanisms were not designed for such extreme context lengths\. In particular, attention faces a fundamental embedding bottleneck: a relatively small hidden dimensiond≪Nd\\ll Nforces information from anNN\-dimensional context to collapse into a much lower\-dimensional representation\. Motivated by this limitation, we envision a future in which long\-context LLM inference becomes entirely sparse in the context dimension\. To support this vision, we demonstrate the surprising robustness of new\-generation large models to extreme sparsity, even though these models were not explicitly trained for sparse context processing\. We further show that multiple forms of sparsity can already be exploited effectively on current hardware, while even greater gains may be unlocked through future hardware designs that explicitly acknowledge the inherently sparse nature of context processing\. Overall, we believe the community should treat sparsity as a central principle when designing the next generation of model architectures, inference and training systems, and the hardware platforms on which they operate\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)GPT\-4 Technical Report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[2\]S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.\(2025\)gpt\-oss\-120b & gpt\-oss\-20b Model Card\.arXiv preprint arXiv:2508\.10925\.Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[3\]\(2023\)GQA: training generalized multi\-query transformer models from multi\-head checkpoints\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 4895–4901\.Cited by:[§4](https://arxiv.org/html/2605.24168#S4.p1.16)\.
- \[4\]S\. Bhattamishra, M\. Hahn, P\. Blunsom, and V\. Kanade\(2024\)Separations in the representational capabilities of transformers and recurrent architectures\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,Cited by:[§2\.1](https://arxiv.org/html/2605.24168#S2.SS1.p2.4)\.
- \[5\]B\. Chen, T\. Medini, J\. Farwell, C\. Tai, A\. Shrivastava,et al\.\(2020\)SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large\-Scale Deep Learning Systems\.Proceedings of Machine Learning and Systems2,pp\. 291–306\.External Links:[Link](https://arxiv.org/abs/1903.03129)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[6\]Z\. Chen, R\. Sadhukhan, Z\. Ye, Y\. Zhou, J\. Zhang, N\. Nolte, Y\. Tian, M\. Douze, L\. Bottou, Z\. Jia, and B\. Chen\(2025\)MagicPIG: LSH Sampling for Efficient LLM Generation\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[7\]K\. M\. Choromanski, V\. Likhosherstov, D\. Dohan, X\. Song, A\. Gane, T\. Sarlos, P\. Hawkins, J\. Q\. Davis, A\. Mohiuddin, L\. Kaiser, D\. B\. Belanger, L\. J\. Colwell, and A\. Weller\(2021\)Rethinking Attention with Performers\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Ua6zuk0WRH)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[8\]A\. Desai, K\. K\. Agrawal, S\. Yang, A\. Cuadron, L\. G\. Schroeder, M\. Zaharia, J\. E\. Gonzalez, and I\. Stoica\(2026\)vAttention: Verified Sparse Attention via Sampling\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=zzTDulLys0)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.24168#S1.p1.1),[§3](https://arxiv.org/html/2605.24168#S3.p1.6)\.
- \[9\]A\. Desai, S\. Yang, A\. Cuadron, M\. Zaharia, J\. E\. Gonzalez, and I\. Stoica\(2025\)HashAttention: Semantic Sparsity for Faster Inference\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=Em2oaXd8Dc)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1),[§4](https://arxiv.org/html/2605.24168#S4.p1.16)\.
- \[10\]W\. Fedus, B\. Zoph, and N\. Shazeer\(2022\)Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.External Links:[Link](http://jmlr.org/papers/v23/21-0998.html)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[11\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The Llama 3 Herd of Models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.24168#S3.p1.6)\.
- \[12\]A\. Gu and T\. Dao\(2024\)Mamba: Linear\-Time Sequence Modeling with Selective State Spaces\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=tEYskw1VY2)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[13\]K\. Guu, K\. Lee, Z\. Tung, P\. Pasupat, and M\. Chang\(2020\)REALM: Retrieval\-Augmented Language Model Pre\-Training\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[14\]C\. R\. C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. Maheswaran, S\. Zhao, J\. Paik, M\. W\. Mahoney, K\. Keutzer, and A\. Gholami\(2025\)Squeezed Attention: Accelerating Long Context Length LLM Inference\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[15\]X\. Hou, Y\. Zhao, Y\. Liu, Z\. Yang, K\. Wang, L\. Li, X\. Luo, D\. Lo, J\. Grundy, and H\. Wang\(2024\)Large Language Models for Software Engineering: A Systematic Literature Review\.ACM Transactions on Software Engineering and Methodology33,pp\. 1–79\.External Links:[Link](https://arxiv.org/abs/2308.10620)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[16\]C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, and B\. Ginsburg\(2024\)RULER: what’s the real context size of your long\-context language models?\.InFirst Conference on Language Modeling,Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.24168#S3.p1.6)\.
- \[17\]G\. Izacard and E\. Grave\(2021\)Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,pp\. 874–880\.Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[18\]G\. Izacard, P\. Lewis, M\. Lomeli, L\. Hosseini, F\. Petroni, T\. Schick, J\. Dwivedi\-Yu, A\. Joulin, S\. Riedel, and E\. Grave\(2023\)ATLAS: Few\-Shot Learning with Retrieval Augmented Language Models\.The Journal of Machine Learning Research24\(1\)\.External Links:[Link](https://arxiv.org/abs/2208.03299)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[19\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan\(2024\)SWE\-bench: Can Language Models Resolve Real\-world Github Issues?\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.24168#S3.SS0.SSS0.Px4.p1.53),[§3](https://arxiv.org/html/2605.24168#S3.p1.6)\.
- \[20\]S\. Joshi, A\. Chowdhury, W\. Bellinger, A\. Kanakamedala, E\. Singh, H\. A\. Duy Le, A\. Desai, and A\. Shrivastava\(2026\)SOCKET: SOft Collison Kernel EsTimator for Sparse Attention\.arXiv preprint arXiv:2602\.06283\.External Links:2602\.06283,[Link](https://arxiv.org/abs/2602.06283)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[21\]S\. Joshi, A\. Chowdhury, A\. Kanakamedala, E\. Singh, E\. Tu, and A\. Shrivastava\(2026\)RACE Attention: A Strictly Linear\-Time Attention Layer for Training on Outrageously Large Contexts\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[22\]A\. Katharopoulos, A\. Vyas, N\. Pappas, and F\. Fleuret\(2020\)Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[23\]A\. Lahoti, K\. Li, B\. Chen, C\. Wang, A\. Bick, J\. Z\. Kolter, T\. Dao, and A\. Gu\(2026\)Mamba\-3: Improved Sequence Modeling using State Space Principles\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HwCvaJOiCj)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[24\]X\. Lai, J\. Lu, Y\. Luo, Y\. Ma, and X\. Zhou\(2025\)FlexPrefill: A Context\-Aware Sparse Attention Mechanism for Efficient Long\-Sequence Inference\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=OfjIlbelrT)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[25\]J\. Lee, A\. Chen, Z\. Dai, D\. Dua, D\. S\. Sachan, M\. Boratko, Y\. Luan, S\. M\. R\. Arnold, V\. Perot, S\. Dalmia, H\. Hu, X\. Lin, P\. Pasupat, A\. Amini, J\. R\. Cole, S\. Riedel, I\. Naim, M\. Chang, and K\. Guu\(2024\)Can Long\-Context Language Models Subsume Retrieval, RAG, SQL, and More?\.arXiv preprint arXiv:2406\.13121\.Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.24168#S3.p1.6)\.
- \[26\]Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. Dal Lago,et al\.\(2022\)Competition\-Level Code Generation with AlphaCode\.Science378\(6624\),pp\. 1092–1097\.External Links:[Link](https://arxiv.org/abs/2203.07814)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[27\]A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)DeepSeek\-V3 technical report\.arXiv preprint arXiv:2412\.19437\.External Links:[Link](https://arxiv.org/html/2412.19437v1)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[28\]A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong,et al\.\(2025\)DeepSeek\-V3\.2: Pushing the Frontier of Open Large Language Models\.arXiv preprint arXiv:2512\.02556\.External Links:[Link](https://arxiv.org/abs/2512.02556)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px2.p1.2),[§1](https://arxiv.org/html/2605.24168#S1.p1.1),[§4](https://arxiv.org/html/2605.24168#S4.p1.16)\.
- \[29\]A\. H\. Liu, K\. Khandelwal, S\. Subramanian, V\. Jouault, A\. Rastogi, A\. Sadé, A\. Jeffares, A\. Jiang, A\. Cahill, A\. Gavaudan,et al\.\(2026\)Ministral 3\.arXiv preprint arXiv:2601\.08584\.Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.24168#S3.p1.6)\.
- \[30\]D\. Liu, M\. Chen, B\. Lu, H\. Jiang, Z\. Han, Q\. Zhang, Q\. Chen, C\. Zhang, B\. Ding, K\. Zhang, C\. Chen, F\. Yang, Y\. Yang, and L\. Qiu\(2026\)RetrievalAttention: Accelerating Long\-Context LLM Inference via Vector Retrieval\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=8z3cOVER4z)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[31\]Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava\(2024\)Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time\.Advances in Neural Information Processing Systems36\.Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[32\]Z\. Liu, J\. Wang, T\. Dao, T\. Zhou, B\. Yuan, Z\. Song, A\. Shrivastava, C\. Zhang, Y\. Tian, C\. Re,et al\.\(2023\)Deja vu: contextual sparsity for efficient llms at inference time\.InInternational Conference on Machine Learning,pp\. 22137–22176\.Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[33\]Mathematical Association of America\(2025\)American Invitational Mathematics Examination \(AIME\) 2025\.Note:[https://maa\.org/maa\-invitational\-competitions/](https://maa.org/maa-invitational-competitions/)Accessed: 2026\-05\-20Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.24168#S3.p1.6)\.
- \[34\]H\. Peng, N\. Pappas, D\. Yogatama, R\. Schwartz, N\. Smith, and L\. Kong\(2021\)Random Feature Attention\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=QtTKTdVrFBB)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[35\]O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. Smith, and M\. Lewis\(2023\)Measuring and Narrowing the Compositionality Gap in Language Models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[36\]B\. Rozière, J\. Gehring, F\. Gloeckle, S\. Sootla, I\. Gat, X\. E\. Tan, Y\. Adi, J\. Liu, R\. Sauvestre, T\. Remez,et al\.\(2023\)Code Llama: Open Foundation Models for Code\.arXiv preprint arXiv:2308\.12950\.External Links:[Link](https://arxiv.org/abs/2308.12950)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[37\]L\. Ruan and Q\. Jin\(2022\)Survey: transformer based Video\-Language Pre\-Training\.AI Open3,pp\. 1–13\.External Links:[Link](https://arxiv.org/abs/2109.09920)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[38\]T\. Schick, J\. Dwivedi\-Yu, R\. Dessi, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom\(2023\)Toolformer: Language Models Can Teach Themselves to Use Tools\.InThirty\-seventh Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[39\]N\. Shazeer, \*\. Mirhoseini, \*\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean\(2017\)Outrageously Large Neural Networks: the Sparsely\-Gated Mixture\-of\-Experts Layer\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=B1ckMDqlg)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[40\]A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)OpenAI GPT\-5 System Card\.arXiv preprint arXiv:2601\.03267\.External Links:[Link](https://arxiv.org/abs/2601.03267)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[41\]P\. Steinberger and OpenClaw contributors\(2026\)OpenClaw: Personal AI Assistant\.Note:[https://github\.com/openclaw/openclaw](https://github.com/openclaw/openclaw)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[42\]J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han\(2024\)QUEST: Query\-Aware Sparsity for Efficient Long\-Context LLM Inference\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[43\]G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi,et al\.\(2025\)Gemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.24168#S3.p1.6)\.
- \[44\]Q\. Team\(2026\-02\)Qwen3\.5: accelerating productivity with native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.24168#S3.p1.6)\.
- \[45\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar, A\. Rodriguez, A\. Joulin, E\. Grave, and G\. Lample\(2023\)LLaMA: Open and Efficient Foundation Language Models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[46\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 Technical Report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[47\]A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu\(2025\)Qwen2\.5 Technical Report\.arXiv preprint arXiv:2412\.15115\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2605.24168#S3.p1.6)\.
- \[48\]S\. Yang, Y\. Sheng, J\. E\. Gonzalez, I\. Stoica, and L\. Zheng\(2024\)Post\-Training Sparse Attention with Double Sparsity\.arXiv preprint arXiv:2408\.07092\.External Links:[Link](https://arxiv.org/abs/2408.07092)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1),[Table 2](https://arxiv.org/html/2605.24168#S3.T2),[Table 2](https://arxiv.org/html/2605.24168#S3.T2.18.9.9),[§4](https://arxiv.org/html/2605.24168#S4.p1.16)\.
- \[49\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao\(2023\)ReAct: Synergizing Reasoning and Acting in Language Models\.InThe Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[50\]Z\. Ye, K\. Xia, Y\. Fu, X\. Dong, J\. Hong, X\. Yuan, S\. Diao, J\. Kautz, P\. Molchanov, and Y\. C\. Lin\(2025\)LongMamba: Enhancing Mamba’s Long\-Context Capabilities via Training\-Free Receptive Field Enlargement\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=fMbLszVO1H)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[51\]Z\. Ye, L\. Chen, R\. Lai, W\. Lin, Y\. Zhang, S\. Wang, T\. Chen, B\. Kasikci, V\. Grover, A\. Krishnamurthy, and L\. Ceze\(2024\)FlashInfer documentation\.Note:Accessed: 2025\-05\-27External Links:[Link](https://flashinfer.ai/2024/02/02/introduce-flashinfer.html)Cited by:[Table 1](https://arxiv.org/html/2605.24168#S3.T1),[Table 1](https://arxiv.org/html/2605.24168#S3.T1.24.12.11)\.
- \[52\]A\. Yu and N\. B\. Erichson\(2026\)Block\-Biased Mamba for Long\-Range Sequence Processing\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=5WKEH9LhAQ)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[53\]M\. Zaheer, G\. Guruganesh, A\. Dubey, J\. Ainslie, C\. Alberti, S\. Ontanon, P\. Pham, A\. Ravula, Q\. Wang, L\. Yang, and A\. Ahmed\(2020\)Big Bird: Transformers for Longer Sequences\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[54\]A\. Zeng, X\. Lv, Z\. Hou, Z\. Du, Q\. Zheng, B\. Chen, D\. Yin, C\. Ge, C\. Huang, C\. Xie,et al\.\(2026\)GLM\-5: from Vibe Coding to Agentic Engineering\.arXiv preprint arXiv:2602\.15763\.External Links:[Link](https://arxiv.org/abs/2602.15763)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[55\]H\. Zhang, X\. Ji, Y\. Chen, F\. Fu, X\. Miao, X\. Nie, W\. Chen, and B\. Cui\(2025\)PQCache: Product Quantization\-based KVCache for Long Context LLM Inference\.Proceedings of the ACM on Management of Data3\(3\),pp\. 1–30\.External Links:[Link](https://arxiv.org/abs/2407.12820)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1),[§4](https://arxiv.org/html/2605.24168#S4.p1.16)\.
- \[56\]J\. Zhang, C\. Xiang, H\. Huang, J\. wei, H\. Xi, J\. Zhu, and J\. Chen\(2025\)SpargeAttention: Accurate and Training\-free Sparse Attention Accelerating Any Model Inference\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=74c3Wwk8Tc)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[57\]Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Re, C\. Barrett, Z\. Wang, and B\. Chen\(2023\)H2O: Heavy\-Hitter Oracle for Efficient Generative Inference of Large Language Models\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=RkRrPp7GKO)Cited by:[§1](https://arxiv.org/html/2605.24168#S1.p1.1)\.
- \[58\]D\. Zhu, Z\. Zuo, and M\. M\. Khalili\(2025\)An efficient training algorithm for models with block\-wise sparsity\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2605.24168#S1.SS0.SSS0.Px2.p1.2)\.

## Appendix

## Appendix AProofs

### A\.1Proof of Theorem[1](https://arxiv.org/html/2605.24168#Thmtheorem1)

###### Proof\.

Consider the linear mapT:ℝN→ℝdT:\\mathbb\{R\}^\{N\}\\to\\mathbb\{R\}^\{d\}defined byT​\(a\):=V⊤​aT\(a\):=V^\{\\top\}a\. SinceV⊤∈ℝd×NV^\{\\top\}\\in\\mathbb\{R\}^\{d\\times N\}, we haverank⁡\(T\)≤d\\operatorname\{rank\}\(T\)\\leq d\. Let𝒩​\(T\):=\{z∈ℝN:V⊤​z=0\}\\mathcal\{N\}\(T\):=\\\{z\\in\\mathbb\{R\}^\{N\}:V^\{\\top\}z=0\\\}be the null space ofTT\. By rank\-nullity,

dim𝒩​\(T\)=N−rank⁡\(T\)≥N−d\.\\dim\\mathcal\{N\}\(T\)=N\-\\operatorname\{rank\}\(T\)\\geq N\-d\.
We want a nonzero direction in the null space that also preserves the simplex sum constraint\. Let𝟏:=\(1,…,1\)⊤∈ℝN\\mathbf\{1\}:=\(1,\\dots,1\)^\{\\top\}\\in\\mathbb\{R\}^\{N\}and consider the zero\-sum hyperplaneH:=\{z∈ℝN:𝟏⊤​z=0\}\.H:=\\\{z\\in\\mathbb\{R\}^\{N\}:\\mathbf\{1\}^\{\\top\}z=0\\\}\.This hyperplane has dimensionN−1N\-1\. By the standard dimension formula for subspaces,

dim\(𝒩​\(T\)∩H\)≥dim𝒩​\(T\)\+dimH−N\.\\dim\(\\mathcal\{N\}\(T\)\\cap H\)\\geq\\dim\\mathcal\{N\}\(T\)\+\\dim H\-N\.Sincedim𝒩​\(T\)≥N−d\\dim\\mathcal\{N\}\(T\)\\geq N\-danddimH=N−1\\dim H=N\-1, we get

dim\(𝒩​\(T\)∩H\)≥\(N−d\)\+\(N−1\)−N=N−d−1\>0,\\dim\(\\mathcal\{N\}\(T\)\\cap H\)\\geq\(N\-d\)\+\(N\-1\)\-N=N\-d\-1\>0,where the last inequality usesd<N−1d<N\-1\. Hence there exists a nonzeroz∈𝒩​\(T\)∩Hz\\in\\mathcal\{N\}\(T\)\\cap Hsuch thatV⊤​z=0V^\{\\top\}z=0, and𝟏⊤​z=0\\mathbf\{1\}^\{\\top\}z=0\. Sincez≠0z\\neq 0, we may rescale it asz←z‖z‖∞,z\\leftarrow\\frac\{z\}\{\\\|z\\\|\_\{\\infty\}\},so that‖z‖∞=1\\\|z\\\|\_\{\\infty\}=1\. This rescaling preservesV⊤​z=0V^\{\\top\}z=0and𝟏⊤​z=0\\mathbf\{1\}^\{\\top\}z=0\. Now, leta0:=1N​𝟏a\_\{0\}:=\\frac\{1\}\{N\}\\mathbf\{1\}be the uniform attention distribution, and fix anyβ∈\(0,1\)\\beta\\in\(0,1\)and define

a:=a0\+βN​z,a′:=a0−βN​z\.a:=a\_\{0\}\+\\frac\{\\beta\}\{N\}z,\\qquad a^\{\\prime\}:=a\_\{0\}\-\\frac\{\\beta\}\{N\}z\.Since𝟏⊤​z=0\\mathbf\{1\}^\{\\top\}z=0, bothaaanda′a^\{\\prime\}sum to one\. Moreover, since‖z‖∞=1\\\|z\\\|\_\{\\infty\}=1, each coordinate satisfies

1−βN≤ai,ai′≤1\+βN\.\\frac\{1\-\\beta\}\{N\}\\leq a\_\{i\},a^\{\\prime\}\_\{i\}\\leq\\frac\{1\+\\beta\}\{N\}\.Becauseβ∈\(0,1\)\\beta\\in\(0,1\), all coordinates are strictly positive\. Thusaaanda′a^\{\\prime\}are valid attention distributions, and both have full support\. They are distinct becausez≠0z\\neq 0\.

Finally,

V⊤​a=V⊤​a0\+βN​V⊤​z=V⊤​a0=V⊤​a0−βN​V⊤​z=V⊤​a′\.V^\{\\top\}a=V^\{\\top\}a\_\{0\}\+\\frac\{\\beta\}\{N\}V^\{\\top\}z=V^\{\\top\}a\_\{0\}=V^\{\\top\}a\_\{0\}\-\\frac\{\\beta\}\{N\}V^\{\\top\}z=V^\{\\top\}a^\{\\prime\}\.Thus there exist two distinct full\-support attention distributionsaaanda′a^\{\\prime\}such that

a≠a′,V⊤​a=V⊤​a′\.a\\neq a^\{\\prime\},\\qquad V^\{\\top\}a=V^\{\\top\}a^\{\\prime\}\.Therefore, the mapa↦V⊤​aa\\mapsto V^\{\\top\}ais not injective on the attention simplex whend<N−1d<N\-1\. ∎

## Appendix BSWE\-Bench Django: subgroup, failure\-mode, and cost breakdown

This appendix reports the complete SWE\-Bench Django evaluation behind the headline of Sec\.[3](https://arxiv.org/html/2605.24168#S3): that on tasks the dense baseline can solve, sparse attention matches dense within∼2\\sim\\\!2pp, and the largerS0S\_\{0\}gap is driven by serving\-stack failures rather than attention quality\. We compare three configurations on Qwen3\.5\-27B served via vLLM under themini\-swe\-agentv2\.2\.82\.2\.8harness \(step\_limit=250,cost\_limit=$3,6060s per\-command timeout\):*dense*is full softmax \(100%100\\%density\);*5×5\\times*is Sink\(128128\)\+\+Local\(128128\)\+\+OracleTopK with heavy fraction0\.200\.20\(achieved density∼22%\\sim\\\!22\\%; per\-layer attention\-outputL2L\_\{2\}error∼1\.3%\\sim\\\!1\.3\\%relative to full attention\);*50×50\\times*uses the same scaffold with heavy fraction0\.020\.02\(∼3\.8%\\sim\\\!3\.8\\%density,∼8\.8%\\sim\\\!8\.8\\%error\)\. All three were run on the full114114\-instance Django subset of SWE\-Bench Lite\. The harness graded114/113/110114/113/110instances, respectively; the11\- and44\-instance deficits trace to serving\-stack exit codes documented below\.

#### Outcome definitions \(SWE\-Bench harness\)\.

resolved: patch applied and target tests pass\.unresolved: patch applied, tests fail\.empty\_patch: the agent produced no diff \(or only test\-file changes, which the harness strips before applying\)\.error: a patch was produced butgit applyrejected it \(malformed hunk or wrong line numbers\)\.

#### Exit\-status definitions \(mini\-swe\-agent terminal state\)\.

The harness’s outcome bucket conflates several mechanisms\. Insideempty\_patch, the agent’s terminal state isolates root cause:Submitted– agent reached the submit step and emitted an empty \(or test\-only\) diff;LimitsExceeded– hitstep\_limit=250without submitting \(model\-side: agent could not converge\);InternalServerError– the vLLM server crashed mid\-conversation, attributable to sparse\-attention\-hub instability under sustained decode \(server\-side, sparse\-specific\);Timeout– LiteLLM18001800s connection timeout against a stalled vLLM server \(same root cause asInternalServerError\);CalledProcessError– the per\-instancedocker runreturned125/127125/127before the agent could place its first model call \(pure infra; no sparsity dependence\)\.

Table 3:Resolution rate by subgroup \(resolved/total\)\.S0S\_\{0\}is the unconditional union;S3S\_\{3\}is the head\-to\-head subset where all three configurations emitted a non\-empty patch that the harness scored\. The∼10\\sim\\\!10ppS0S\_\{0\}dense→\\\!\\to\\\!sparse gap collapses to∼2\\sim\\\!2pp onS3S\_\{3\}\(5×5\\timesin fact slightly edges dense\), placing the gap on*whether*the agent emits a parseable patch rather than on*patch quality*\.
#### Where does theS0S\_\{0\}gap come from?

Theempty\_patchbucket determines theS0S\_\{0\}gap, but it is a misleadingly uniform label: the13/34/2613/34/26empty\_patchcases come from very different root causes across configurations \(Tab\.[4](https://arxiv.org/html/2605.24168#A2.T4)\)\. For dense, the dominant root cause isCalledProcessError: the docker container for the instance failed to launch and the agent never placed a model call, so attention played no role in the failure\. For sparse, the dominant root cause isInternalServerError\(plusTimeout\): the vLLM server crashed or stalled mid\-trajectory under sustained sparse\-attention\-hub decode, again with no bearing on attention quality, just on serving\-stack stability\. The only failure mode plausibly attributable to attention isLimitsExceeded– the agent ran out of250250turns without converging – and it scales modestly with sparsity \(4/11/114/11/11\), which is the residual attention\-quality cost of compressing context\. On theresolvedandunresolvedbuckets every trajectory isSubmitted: once the agent reaches submit, sparsity does not change*whether*the patch passes, only*which*tasks the agent reaches submit on\. The threeerrorentries at50×50\\timesaregit applyrejections of malformed hunks; we treat them as the same family asempty\_patch– patches the harness never scored\.

Table 4:Per\-\(outcome×\\timesconfig\) exit\-status counts\.nnis the size of that outcome bucket for that configuration\.*Bold*entries are root causes attributable to infrastructure \(docker daemon, vLLM server crash, or connection timeout\) and are eligible for retry; plain entries are model\-side failures \(Submittedbut tests fail, orLimitsExceeded\)\.configoutcomennexit\-status mixdenseresolved70Submitted:7070denseunresolved31Submitted:3131denseempty\_patch13CalledProcessError:99, LimitsExceeded:445×5\\timesresolved60Submitted:60605×5\\timesunresolved19Submitted:19195×5\\timesempty\_patch34InternalServerError:1717, LimitsExceeded:1111, Submitted:44,Timeout:2250×50\\timesresolved57Submitted:575750×50\\timesunresolved24Submitted:242450×50\\timesempty\_patch26LimitsExceeded:1111,InternalServerError:1111,Timeout:4450×50\\timeserror3Submitted:33\(git applyrejected\)
#### Counterfactual: what would a clean rerun show?

If we retry the infrastructure\-attributable failures and assume each retry resolves at the configuration’s empiricalS3S\_\{3\}rate \(77\.677\.6/79\.379\.3/75\.9%75\.9\\%\), the projected post\-retry rates become dense61\.4→66\.7%61\.4\\\!\\to\\\!66\.7\\%\(∼6\\sim\\\!6of99docker failures recovered\),5×5\\times53\.1→61\.1%53\.1\\\!\\to\\\!61\.1\\%\(∼9\\sim\\\!9of1919vLLM\-side failures recovered\), and50×50\\times51\.8→59\.1%51\.8\\\!\\to\\\!59\.1\\%\(∼8\\sim\\\!8of1515\)\. The sparse\-to\-dense gap narrows by∼4\\sim\\\!4pp but a∼6\\sim\\\!6pp residual remains: this residual is genuine server\-side instability of the sparse\-attention\-hub backend, not attention\-quality loss, and is the right target for follow\-up engineering rather than for re\-evaluating the sparsity claim\.

#### Per\-task compute cost\.

Tab\.[5](https://arxiv.org/html/2605.24168#A2.T5)reports mean turns and total tokens by outcome\. On*resolved*tasks, sparse runs use∼15%\\sim\\\!15\\%fewer turns \(67→57→5567\\\!\\to\\\!57\\\!\\to\\\!55\) and∼15\\sim\\\!15–19%19\\%fewer total tokens \(1\.341\.34M→1\.14\\\!\\to\\\!1\.14M→1\.08\\\!\\to\\\!1\.08M\); on*unresolved*tasks the reduction is larger \(92→67/7092\\\!\\to\\\!67/70turns;2\.662\.66M→1\.56/1\.62\\\!\\to\\\!1\.56/1\.62M tokens\), suggesting the sparse agent commits to or abandons a fix sooner rather than churning\. The two effects compound with the per\-decode kernel speedup \(Tab\.[1](https://arxiv.org/html/2605.24168#S3.T1)\): wall\-clock savings per productive task are the product of fewer turns, fewer tokens per turn, and faster per\-call attention\.

Table 5:Per\-task compute cost by outcome\.Mean agent turns and mean total tokens per task\.nncolumns report instance counts as \(dense /5×5\\times/50×50\\times\)\.empty\_patchanderrorcosts are inflated by stalled trajectories that exhaust the step budget before the harness gives up; they are not comparable to the productive outcomes\.![Refer to caption](https://arxiv.org/html/2605.24168v1/images/swe_failure_modes_v2.png)Figure 7:Empty\-patch root cause changes with attention configuration\.Outcome counts split by terminal exit status \(visual companion to Tab\.[4](https://arxiv.org/html/2605.24168#A2.T4); numerical counts there\)\. The takeaway is the colour composition of theempty\_patchbar: dense’sempty\_patchis almost entirely docker\-launch failures \(blue,CalledProcessError\); sparse’s is almost entirely vLLM crashes and timeouts \(purple/orange,InternalServerError/Timeout\)\. The only stratum that grows with sparsity isLimitsExceeded\(pink,4→11→114\\\!\\to\\\!11\\\!\\to\\\!11\), the model\-side residual\.![Refer to caption](https://arxiv.org/html/2605.24168v1/images/swe_tokens_per_turn_v2.png)Figure 8:Per\-LLM\-call prompt size is unaffected by sparsity on productive outcomes\.Mean prompt tokens per LLM call, by outcome and configuration\. Onresolvedthe three configurations are within0\.30\.3K tokens/call \(17\.3/17\.6/17\.617\.3/17\.6/17\.6K\); onunresolvedthey sit within33K \(24\.1/21\.0/21\.524\.1/21\.0/21\.5K\)\. Sparsity does not change how much context the agent maintains per call – it changes what attention does*with*that context\. The largeempty\_patchbars \(59\.2/33\.1/45\.659\.2/33\.1/45\.6K\) come from the stalled\-trajectory tail \(Fig\.[7](https://arxiv.org/html/2605.24168#A2.F7)\) where the agent burns through the250250\-turn cap on a steadily growing prompt; the elevated tokens reflect the failure mode, not the productive cost\.