SPIN: Shadow Predictive Indexer for Sparse Attention

# Spin : Shadow Predictive Indexer for Sparse Attention
Source: [https://arxiv.org/html/2610.09025](https://arxiv.org/html/2610.09025)
Cyrus ChangRitchie ZhaoBryce LongYueying Li Mahdi KamaniSamkit JainRahul RamanTara SafaviShreya Gupta Parsa Ashrafi FashiMinseok LeeJulien DemouthBita Darvish Rouhani

###### Abstract

Indexer\-based sparse attention reduces the cost of core attention by passing only a fixed, small number of important tokens to it\. However, the indexer must still score the entire KV cache at every decoding step\. This scoring overhead becomes a major bottleneck as the context length grows\. We proposeSpin\(ShadowPredictiveIndexer\) to reduce this indexer overhead\.Spinuses lightweight, history\-based prediction to identify important KV blocks, avoiding the need to score the full KV cache at every decoding step\.Spintreats KV blocks and speculative decoding as first\-class design and implementation considerations\. Across extensive evaluations on long\-context and agentic benchmarks,Spinachieves 30–40% sparsity while preserving task quality\. In end\-to\-end vLLM serving,Spinimproves output throughput by up to 14\.9% and reduces median inter\-token latency by up to 13\.2%\.

## 1Introduction

DeepSeek Sparse Attention\[[11](https://arxiv.org/html/2610.09025#bib.bib11)\]\(DSA\) is a sparse attention architecture that relies on an indexer to score each token in the context\. DSA and its variants have seen significant adoption among frontier open\-source models, including DeepSeek\-V4\[[12](https://arxiv.org/html/2610.09025#bib.bib12)\], GLM\-5\.2\[[26](https://arxiv.org/html/2610.09025#bib.bib26)\], MiniMax\-M3\[[15](https://arxiv.org/html/2610.09025#bib.bib15)\], and LongCat\-2\.0\[[17](https://arxiv.org/html/2610.09025#bib.bib17)\]\. Core attention processes a fixed top\-KKbudget, whereas the indexer scales linearly with context length, making it the dominant bottleneck in long\-context inference\. DeepSeek\-V4’s CSA layer\[[12](https://arxiv.org/html/2610.09025#bib.bib12)\]combines the original DSA with a new token\-level compressor, effectively reducing DSA’s context length by4×4\\times\. This, however, does not change the indexer’s linear scaling with context length; thus, at very long context lengths \(e\.g\., 512K–1M\), the indexer remains a major contributor to DeepSeek\-V4’s decode inter\-token latency \(ITL\)\.

We observe that DSA’s indexer patterns are predictable, similar to temporal patterns previously observed in attention scores\[[23](https://arxiv.org/html/2610.09025#bib.bib23),[24](https://arxiv.org/html/2610.09025#bib.bib24)\]\. Indexer scores can be positively correlated with \(1\) previous DSA layers’ indexer scores in the same decode iteration \(prev\-layer\) and \(2\) the same layer’s scores from previous iterations \(prev\-iter\)\. Our benchmarking results show that prev\-iter signals correlate more strongly with current scores than prev\-layer signals\. This motivatesShadowPredictiveIndexer \(SPIN\), which uses simple per\-request statistics of each DSA layer’s scores from previous decoding iterations to predict token importance and skip less important context tokens, thereby sparsifying the indexer input\. Skipped blocks receive no score updates and may remain excluded even after becoming important\. We address this with random exploration within a fixed block budget to refresh skipped blocks at low overhead\. To enable efficient implementation in LLM serving frameworks,Spinapplies sparsity at the KV\-block level\.Spinis training\-free and introduces no new auxiliary layers or networks\.

We implementSpinfor the DeepSeek\-V4 model class in vLLM\[[14](https://arxiv.org/html/2610.09025#bib.bib14)\]for accuracy evaluation\. Across the long\-context benchmarksLongBench\-v2\[[4](https://arxiv.org/html/2610.09025#bib.bib4)\],AA\-LCR\[[3](https://arxiv.org/html/2610.09025#bib.bib3)\],MRCRv2\[[18](https://arxiv.org/html/2610.09025#bib.bib18)\], andRULER\[[13](https://arxiv.org/html/2610.09025#bib.bib13)\], and the agentic benchmarktau2\-airline\[[25](https://arxiv.org/html/2610.09025#bib.bib25),[7](https://arxiv.org/html/2610.09025#bib.bib7)\],Spinpreserves comparable task quality at 30–40% block sparsity\. We further integrateSpinwith random exploration into vLLM, improving output throughput by up to 14\.9% and reducing median inter\-token latency by up to 13\.2%\.

Our concrete contributions are as follows:

1. 1\.We evaluate various prev\-iter and prev\-layer prediction signals for the indexer in DeepSeek\-V4 and show that prev\-iter indexer scores provide the most accurate predictions\.
2. 2\.We proposeSpinto sparsify the indexer based on prev\-iter scores and an exploration mechanism and ablate various methods to pool the per\-token scores into a block\-level prediction\.
3. 3\.We evaluateSpinon DeepSeek\-V4 across long\-context and agentic benchmarks and integrate it into vLLM, improving output throughput by up to 14\.9% and reducing median inter\-token latency by up to 13\.2%\.
4. 4\.We compare exploration methods onMRCRv2and show that exploration can recover task quality while retaining the same block budget\.
5. 5\.We evaluateSpinduring speculative target\-model verification using DeepSeek\-V4’s native multi\-token prediction \(MTP\) onSPEED\-Bench\[[1](https://arxiv.org/html/2610.09025#bib.bib1)\]\.

## 2DSA Indexer Patterns and Prediction

In each decode iteration, DSA’s indexer computes a scalar score for each context token\. For query tokentt, the indexer projects its representation into a multi\-head queryqtq\_\{t\}\. The keykik\_\{i\}for each context tokeniiis generated earlier and cached during inference\. Rotary positional encodings \(RoPE\)\[[19](https://arxiv.org/html/2610.09025#bib.bib19)\]are applied to half of each head’s dimensions\. The indexer score for context tokeniiiszt​\[i\]=∑h=1nhwh​ReLU​\(qt,h⋅ki,h\)z\_\{t\}\[i\]=\\sum\_\{h=1\}^\{n\_\{h\}\}w\_\{h\}\\,\\text\{ReLU\}\(q\_\{t,h\}\\cdot k\_\{i,h\}\), wherenhn\_\{h\}is the number of heads, andqt,hq\_\{t,h\},ki,hk\_\{i,h\}, andwhw\_\{h\}are the query, key, and weight for headhh\. The top\-KKtokens with the highest scores are selected and passed to core attention, withKKin practice ranging from512512to20482048\.

The DSA indexer exhibits patterns that can be predicted in different ways\. Figure[1](https://arxiv.org/html/2610.09025#S2.F1)visualizes the top\-KKselection mask from a single layer on a prompt fromsummscreenfd\[[8](https://arxiv.org/html/2610.09025#bib.bib8)\]\. The vertical axis represents decode iterations\. Vertical\-line and diagonal\-band patterns are clearly visible\. Vertical lines indicate that the indexer tends to select the same tokens across multiple iterations\. Diagonal bands resemble well\-documented "slash" patterns in some attention heads\[[10](https://arxiv.org/html/2610.09025#bib.bib10),[24](https://arxiv.org/html/2610.09025#bib.bib24)\], which are known to be caused by RoPE\[[6](https://arxiv.org/html/2610.09025#bib.bib6),[9](https://arxiv.org/html/2610.09025#bib.bib9)\]\. Among the DSA models we examine, DeepSeek\-V3\.2 is the only one with prominent diagonal bands; the others exhibit mostly vertical patterns\. Both patterns allow prediction of indexer scores using scores from the previous iteration \(prev\-iterprediction\)\.

DeepSeek\-V3\.2 Indexer top\-KK

![Refer to caption](https://arxiv.org/html/2610.09025v1/figures/fig1/dsv32_pattern.png)

DeepSeek\-V4 Indexer top\-KK

![Refer to caption](https://arxiv.org/html/2610.09025v1/figures/fig1/dsv4_pattern.png)

Figure 1:Indexer top\-KKpatternsfrom a single DSA layer on a prompt fromsummscreenfd\. The y\-axis represents decode iterations\. The prompt is 9\.7K tokens long, and 1,400 context KV slots are visualized along the x\-axis\. DeepSeek\-V3\.2 exhibits prominent diagonal bands and vertical lines, whereas DeepSeek\-V4 exhibits mostly vertical patterns\.Indexer scores can also be predicted using scores from earlier layers in the same decode iteration \(prev\-layerprediction\)\. This is the core idea behind GLM\-5’s IndexCache\[[5](https://arxiv.org/html/2610.09025#bib.bib5)\]and indexer sharing across adjacent layers in GLM\-5\.2\[[26](https://arxiv.org/html/2610.09025#bib.bib26)\]\. We compare four predictors derived from two signal sources—previous layers and previous decode iterations—against a random baseline\. Each method constructs a proxy for the target indexer scores:

1. 1\.Random: Sample indexer scores independently and uniformly at random\. The expected score correlation is zero\.
2. 2\.Prev\-layer: Reuse the indexer scores from the preceding DSA layer in the same decode iteration\.
3. 3\.Prev\-layer \(best\-of\-5 oracle\): Consider the indexer scores from the five preceding DSA layers and report the best of the five\.
4. 4\.Prev\-iter: Reuse the indexer scores from the preceding decode iteration at the same DSA layer\.
5. 5\.Prev\-iter \(offset 1\): Shift the prev\-iter indexer scores by one position to align with the diagonal pattern\.

To test whether the prediction signals generalize across models, tasks, and context lengths, we evaluate them on both DeepSeek\-V3\.2 and DeepSeek\-V4 Flash using two diagnostic prompts: a 9\.7K\-tokensummscreenfdsummarization prompt and a 94K\-tokenAA\-LCR\[[3](https://arxiv.org/html/2610.09025#bib.bib3)\]reasoning prompt\. At each decode iteration, we compare the predicted scores and top\-KKindices with the original indexer outputs, reporting Pearson correlation for score prediction and recall for top\-KKselection\. Forsummscreenfd, we disable thinking and evaluate all DSA layers\. ForAA\-LCR, we enable thinking but evaluate only layers\{0,10,20,30,40,50,60\}\\\{0,10,20,30,40,50,60\\\}and omit the prev\-layer best\-of\-5 oracle to limit evaluation cost\.

Table 1:Comparison of indexer prediction methods— profiling results for one prompt each fromsummscreenfd\(length: 9\.7k\) andAA\-LCR\(length: 94k\)\. Values are averaged over layers and decode iterations\. The best method is shown in bold, and the second\-best method is underlined\.Table[1](https://arxiv.org/html/2610.09025#S2.T1)shows the score correlation and top\-KKrecall for each prediction method, averaged over layers and decode iterations\. Prev\-iter prediction is the most accurate method and also scales well to longer contexts\. FromsummscreenfdtoAA\-LCR, the context length increases by a factor of9\.79\.7, and the random recall baseline drops proportionally\. However, the prev\-iter recall rate is largely maintained\. Prev\-layer prediction works better on DeepSeek\-V4 than on DeepSeek\-V3\.2 but lags prev\-iter on both models\.

Motivated by these results, we designSpinto predict block importance from per\-layer statistics accumulated across previous decode iterations\.

## 3SPIN

### 3\.1Overview

Spinextends naive prev\-iter score prediction in three ways:

1. 1\.Maintain exponential moving averages \(EMAs\) of previous indexer scores along the vertical and diagonal patterns\. This allowsSpinto capture longer\-term signals in both patterns\.
2. 2\.Pool the predicted token\-level scores into blocks\. This allowsSpinto skip entire KV blocks and integrate efficiently with paged\-attention backends\.
3. 3\.Reserve a small fraction of the block budget for random exploration\. This allowsSpinto refresh observations of skipped blocks without increasing the number of blocks scored\.

### 3\.2SpinAlgorithm

Spinoperates over the KV slots visible to the indexer; for DeepSeek\-V4, each slot corresponds to a compressed token\.

To predict the indexer score of slotiiat query positionqtq\_\{t\},Spincombines two EMA statistics:V⁡\[i\]V\[i\], the vertical score associated with absolute slotii, andD⁡\[qt−i\]D\[q\_\{t\}\-i\], the diagonal score associated with relative offsetqt−iq\_\{t\}\-i\. The predicted score is

z^t​\[i\]=max⁡\(V⁡\[i\],D⁡\[qt−i\]\)\.\\hat\{z\}\_\{t\}\[i\]=\\max\\left\(V\[i\],D\[q\_\{t\}\-i\]\\right\)\.\(1\)
Spinpartitions the slots into logical blocks of sizebb, matching the block size used by the attention backend, and definesb⁡\(i\)=⌊i/b⌋b\(i\)=\\lfloor i/b\\rflooras the block containing slotii\. It then max\-pools the predicted scores within each block to produce a predicted block score:

St\[j\]=maxi:b⁡\(i\)=jz^t\[i\]\.S\_\{t\}\[j\]=\\max\_\{i:\\,b\(i\)=j\}\\hat\{z\}\_\{t\}\[i\]\.\(2\)We study alternative pooling methods in Section[4\.6](https://arxiv.org/html/2610.09025#S4.SS6.SSS0.Px1)\.

LetKKdenote the output size of the post\-indexer top\-KKselection\. The target block sparsityρ∈\[0,1\]\\rho\\in\[0,1\]denotes the desired fraction of context blocks to skip\. The resulting number of selected blocks,mtm\_\{t\}, is lower\-bounded by⌈K/b⌉\\lceil K/b\\rceilto provide capacity for at leastKKslots and capped by the number of available blocks\. Themtm\_\{t\}highest\-scoring blocks are passed to the indexer, which computes scores for all slots in those blocks\. The original post\-indexer top\-KKoperator then selectsKKslots from these scores for core attention\. The pre\-indexer block selection operates over one predicted score per block rather than one per slot, reducing the size of the selection problem\.

After the indexer evaluates the selected blocks, their scores update the correspondingVVandDDEMA states\. Because skipped blocks produce no scores,Spinupdates its full EMA state from only partial observations at each decode iteration\. If an important slot lies in a skipped block, its EMA statistics are not updated and may become stale\. We mitigate this failure mode with exploration in the next subsection\.

By design,Spin’s block prediction and EMA update can be scheduled asynchronously and overlapped with the forward pass of other layers\.

Algorithm 1CoreSpinblock\-selection and EMA\-update logic for one decode iterationInput:Indexer query position

qtq\_\{t\}; context length

NtN\_\{t\}in KV slots; state

V,DV,D; block size

bb; post\-indexer selection size

KK; target block sparsity

ρ\\rho; EMA decay

α\\alpha
Output:Post\-indexer top\-

KKslot set

TtT\_\{t\}and updated state

V,DV,D
1

Bt←⌈Nt/b⌉B\_\{t\}\\leftarrow\\lceil N\_\{t\}/b\\rceil;//total number of context blocks

2for*j←0j\\leftarrow 0toBt−1B\_\{t\}\-1*do

3

St​\[j\]←maxj​b≤i<min⁡\{\(j\+1\)​b,Nt\}⁡max⁡\{V⁡\[i\],D⁡\[qt−i\]\}S\_\{t\}\[j\]\\leftarrow\\max\_\{jb\\leq i<\\min\\\{\(j\+1\)b,N\_\{t\}\\\}\}\\max\\\{V\[i\],D\[q\_\{t\}\-i\]\\\};//predicted scores

4

mt←min⁡\{Bt,max⁡\(⌈K/b⌉,⌈\(1−ρ\)​Bt⌉\)\}m\_\{t\}\\leftarrow\\min\\\!\\left\\\{B\_\{t\},\\max\\\!\\left\(\\lceil K/b\\rceil,\\lceil\(1\-\\rho\)B\_\{t\}\\rceil\\right\)\\right\\\};//number of blocks to select

5

MtB←TopK⁡\(\{St​\[0\],…,St​\[Bt−1\]\},mt\)M\_\{t\}^\{B\}\\leftarrow\\operatorname\{TopK\}\(\\\{S\_\{t\}\[0\],\\ldots,S\_\{t\}\[B\_\{t\}\-1\]\\\},m\_\{t\}\);//selected blocks

6

zt←Indexer⁡\(MtB\)z\_\{t\}\\leftarrow\\operatorname\{Indexer\}\(M\_\{t\}^\{B\}\);//indexer scores in logical KV\-slot coordinates

7

Tt←TopK⁡\(zt,K\)T\_\{t\}\\leftarrow\\operatorname\{TopK\}\(z\_\{t\},K\);

8foreach*i∈dom⁡\(zt\)i\\in\\operatorname\{dom\}\(z\_\{t\}\)*do

9

V⁡\[i\]←α​V​\[i\]\+\(1−α\)​zt​\[i\]V\[i\]\\leftarrow\\alpha V\[i\]\+\(1\-\\alpha\)z\_\{t\}\[i\];//EMA updates

10

D⁡\[qt−i\]←α​D​\[qt−i\]\+\(1−α\)​zt​\[i\]D\[q\_\{t\}\-i\]\\leftarrow\\alpha D\[q\_\{t\}\-i\]\+\(1\-\\alpha\)z\_\{t\}\[i\];

Algorithm[1](https://arxiv.org/html/2610.09025#algorithm1)summarizesSpin’s core block\-selection and EMA\-update logic\. For clarity, it omits several practical details\.SpinresetsVVandDDto zero after prefill\. For multi\-turn inference, it repeats this reset at the start of each turn\. In both cases, the first decode iteration runs without block pruning to initialize the EMA state\. We also always retain the sink block and the newest active block\. Exploration is described separately below\.

#### 3\.2\.1Staleness Mitigation

WithSpinenabled, blocks with lower predicted scores may be excluded from the indexer input and therefore receive no EMA updates\. However, these blocks can still become important at later decode steps\. Periodic full\-indexer refresh revisits all blocks, but increases scoring work relative to the usual sparse steps\. We instead reserve part of the existing block budget for exploration\.

At each iteration, a fractionγ\\gammaof the block budgetmtm\_\{t\}is reserved for exploration\. The remaining budget retains the forced blocks and the highest\-scoring blocks following Algorithm[1](https://arxiv.org/html/2610.09025#algorithm1)\. Exploration fills the reserved positions with distinct blocks chosen from those not already selected\. Forced blocks count towardmtm\_\{t\}, and the exploration budget is capped by the remaining capacity\. The total number of selected blocks remains unchanged, preserving the block budget\.

The explored blocks receive fresh indexer scores and EMA updates, allowing previously missed slots to reflect their current importance\. We compare several exploration methods and choose random exploration, which recovers task quality with low implementation overhead\. Details are presented in Section[4\.6](https://arxiv.org/html/2610.09025#S4.SS6.SSS0.Px2)\.

#### 3\.2\.2Speculative Decoding

For a speculative draft of lengthdd, the target model evaluatesd\+1d\+1verifier queries in a single forward pass, indexed by rowsr∈\{0,…,d\}r\\in\\\{0,\\ldots,d\\\}\. Rowrris valid if the firstrrdraft tokens are accepted; row 0 is therefore always valid\. These verifier queries share the same context KV cache but have different query positions and hidden states\.

A potential concern when applyingSpinto speculative decoding is that independently predicted block masks for each query may have a large union, reducing the realized block sparsity\. However, we observe that the block masks selected for adjacent verifier queries overlap substantially \(Section[4\.5](https://arxiv.org/html/2610.09025#S4.SS5)\)\. Based on this observation,Spinpredicts a block mask for each verifier query and uses their union as the shared indexer\-input mask\.

EMA updates are deferred until the verification outcomes are known\. Ifaadraft tokens are accepted, the observations from rows0,…,a0,\\ldots,aare replayed in order to update the EMA state\. The remaining rows are off the committed decode path and are discarded\.

We evaluate this design using DeepSeek\-V4’s native MTP implementation and study its impact on speculative\-decoding acceptance in Section[4\.5](https://arxiv.org/html/2610.09025#S4.SS5)\.

## 4Experimental Evaluation

In this section, we evaluateSpin’s impact on inference quality and performance\. We also conduct microbenchmarks of its key design choices\. Specifically, we answer the following questions:

1. 1\.How doesSpinaffect task quality on long\-context and agentic benchmarks?
2. 2\.How much doesSpinimprove end\-to\-end output throughput and inter\-token latency in vLLM?
3. 3\.How doesSpininteract with speculative decoding, and does applying it during target\-model verification affect average acceptance length?
4. 4\.How effectively does exploration mitigate staleness, and how sensitive isSpinto the block\-pooling method?

### 4\.1Experimental Setup

We evaluateSpinon DeepSeek\-V4 Flash and Pro, using Flash as the primary model\. The evaluation covers long\-context benchmarks \(LongBench\-v2,AA\-LCR,MRCRv2, andRULER\), the agentic benchmarktau2\-airline, and speculative decoding onSPEED\-Bench\. Unless otherwise specified, sparse experiments use logical blocks of 64 compressed indexer slots, max block pooling, and an EMA decay of0\.50\.5\. The LongBench\-v2, AA\-LCR, RULER, tau2\-airline, block\-reduction, and MTP results evaluate the core predictor without exploration\. The MRCRv2 study compares exploration policies, and the end\-to\-end implementation uses random exploration withγ=0\.05\\gamma=0\.05\. We useSpin\(x%x\\%\) to denote a target block sparsity ofx%x\\%, meaning thatx%x\\%is the intended fraction of context blocks to skip\. Realized block sparsity is the measured fraction skipped after budget rounding, forced\-block retention, and, for speculative decoding, mask union\. We compare eachSpinconfiguration with a matched*all\-keep*control, in which no blocks are skipped, using the same model and evaluation protocol\.

To measure mean and p99 miss, we compute dense indexer scores within the DeepSeek\-V4 indexer path and applySpin’s block masks before top\-KKselection\. We report the task\-specific score as the primary quality metric and top\-KKmiss rate \(mean and p99\) as auxiliary diagnostic metrics\. The miss rate is the fraction of ground truth top\-KKindices removed due toSpin’s block\-sparse mask\. In the MRCRv2 exploration study, the indexer scores only the selected blocks, and we report task scores\. Benchmark\-specific setup details and hyperparameters are described in their respective subsections\. End\-to\-end serving performance is measured using the integrated vLLM implementation described in Section[4\.4](https://arxiv.org/html/2610.09025#S4.SS4)\.

### 4\.2Long\-context Benchmarks

##### Cross\-model quality\.

We evaluate whetherSpinpreserves task quality on DeepSeek\-V4 Flash and Pro usingLongBench\-v2\. We follow the official evaluation protocol but increase the maximum output length from 128 to 1,024 tokens to reduce output truncation\. Table[2](https://arxiv.org/html/2610.09025#S4.T2)reports the zero\-shot exact\-match scores for all\-keep andSpinconfigurations\. Across target sparsities from 20% to 50%, allSpinscores differ from the corresponding all\-keep score by at most 2\.6% in relative terms\. These results show thatSpinmaintains comparable task quality on both Flash and Pro with up to 50% target block sparsity\.

Table 2:LongBench\-v2 exact\-match score \(%\) over all 503 examples\.
##### Sensitivity to selection misses\.

We evaluate how sensitive task quality is to the selection misses introduced bySpinonAA\-LCRandRULERusing DeepSeek\-V4 Flash\. TheAA\-LCRevaluation contains 100 questions with mean and maximum prompt lengths of 95\.5K and 115\.2K tokens, respectively\. We repeat the evaluation five times with different random seeds to obtain a more stable score estimate\.RULERcovers 13 tasks at six context lengths from 4K to 128K, with 500 examples per task–length cell, yielding 39,000 examples per configuration\. Given this large evaluation set, we run each configuration once\.

Table 3:AA\-LCR and RULER results for DeepSeek\-V4 Flash\. AA\-LCR scores are averaged over execution\-level runs; RULER scores are macro\-averaged over task–length cells\.Table[3](https://arxiv.org/html/2610.09025#S4.T3)reports the task score, mean miss, and p99 miss for all\-keep and different sparse configurations\. Across target sparsities from 20% to 50%,AA\-LCRscores differ from all\-keep by at most 2\.4 points, or 3\.7% in relative terms, without a monotonic decrease\. OnRULER, scores decrease slightly and monotonically as miss rates increase\.RULERexhibits substantially higher miss rates thanAA\-LCR: at 50% sparsity, its p99 miss reaches 48\.83%, but task\-score degradation remains small—only 1\.24 points, or 1\.4% relative to all\-keep\.

##### Long\-context stress test\.

We evaluateSpinon DeepSeek\-V4 Flash using the 8\-needleMRCRv2setting at 256K, 512K, and 1M context lengths\. Each configuration uses the same 100 official prompts per context bin over 40 seeded repeats, with thinking enabled at maximum reasoning effort and an output limit of 8,192 tokens\. The three examples that exceed the model context limit in the 1M bin retain zero scores\. Table[4](https://arxiv.org/html/2610.09025#S4.T4)comparesSpinwithout exploration and with 5% random exploration\.

Table 4:MRCRv2 task scores without exploration and with 5% random exploration\. Each score averages 40 repeats over 100 fixed prompts per context bin\.Without exploration,Spinshows a clear task\-score gap relative to all\-keep as target sparsity increases beyond 20%\. Random exploration substantially narrows this gap up to 50% target sparsity, despite variation across runs\. We ablate alternative exploration methods in Section[4\.6](https://arxiv.org/html/2610.09025#S4.SS6.SSS0.Px2)\.

### 4\.3Agentic Benchmarks

##### tau2\-airline

We evaluateSpinwith DeepSeek\-V4 Flash ontau2\-airline, a multi\-turn agentic benchmark containing 50 tasks\. We run four seeded trials per task for each configuration, yielding 200 trajectories per configuration\. The target agent uses temperature 0 and top\-pp1\. The hosted GPT\-5\.2 user simulator is fixed across all configurations at temperature 0\.

Table 5:tau2\-airlineresults for DeepSeek\-V4 Flash\. Mean and p99 miss are selection\-fidelity metrics over eligible sparse decode rows and are not applicable to the all\-keep control\.At 30% target sparsity,Spinshows no degradation in task score\. Even at 50% sparsity, the score decreases by only 4\.1% relative to all\-keep, despite the mean miss reaching 25\.7%\.

In a separate evaluation onTerminal\-Bench 2\.1with DeepSeek\-V4 Flash,Spinat 40% target sparsity matches the all\-keep score \(62\.5% for both\), although a few tasks produce substantially longer agent trajectories\.

### 4\.4End\-to\-End vLLM Performance

We integrateSpininto vLLM v0\.27\.0rc2 and evaluate its end\-to\-end serving performance with DeepSeek\-V4 Flash on eight NVIDIA B300 GPUs using tensor parallelism of eight \(TP8\) and expert parallelism of eight \(EP8\)\. These runs use the same block size, max pooling, EMA decay, warmup, and forced\-block policies as the task\-quality evaluations\. We reserve 5% of the block budget for exploration\.

We evaluate a workload with a long cached context, a common setting in agentic tasks\. We pre\-populate one distinct cached prefix for each request before sending the measured wave\. The workload contains 507K cached context tokens and 5K new input tokens\. After one warmup batch, we measure 96 concurrent requests, each generating 16K output tokens\. We report output throughput in tokens per second \(TPS\) and median inter\-token latency \(ITL\)\.

Table 6:End\-to\-end serving performance for DeepSeek\-V4 Flash with 507K cached context tokens and 5K new input tokens\. The batch size is 96, and each request generates 16K tokens\. TPS gain and ITL reduction are computed relative to the matched all\-keep setting\.Table[6](https://arxiv.org/html/2610.09025#S4.T6)reports the end\-to\-end results\.Spinimproves output throughput by 10\.6–14\.9% and reduces median ITL by 9\.2–13\.2%\. Increasing target block sparsity from 40% to 50% increases the output TPS gain by 40\.7%\. This occurs becauseSpin’s overhead is largely fixed\. Once this overhead is paid, the additional reduction in indexer work translates more directly into end\-to\-end speedup\.

### 4\.5Speculative Decoding

We study howSpininteracts with speculative decoding using DeepSeek\-V4’s native multi\-token prediction \(MTP\) with a draft length of three\. ExtendingSpinto multiple verifier queries requires decisions about constructing a shared sparse mask and updating the EMA state after verification\. We first ablate these choices through offline trace replay and then evaluate the selected policy’s impact on speculative\-decoding acceptance online\.

##### Offline policy ablation\.

We collect 60 MTP traces by running 15 curatedAA\-LCRprompts with DeepSeek\-V4 Flash and Pro in both chat and thinking modes\. Each trace records the verifier queries and their acceptance outcomes\.

For mask construction, we compare row\-0 sharing, which shares the row\-0 mask across all verifier rows, with row\-local union, which unions masks predicted separately for each row\. For EMA updating, we compare row\-0 update with prefix\-valid update, which sequentially updates from all prefix\-valid rows after the acceptance outcomes are known\. We replaySpinon the collected traces at seven target block sparsities from 10% to 50% and report the aggregated p99 miss rate\.

Figure 2:MTP policy ablation over 60AA\-LCRtraces with a draft length of three\. Each point reports p99 miss and realized block sparsity for one target sparsity under one mask\-construction and EMA\-update strategy\.Figure[2](https://arxiv.org/html/2610.09025#S4.F2)shows that prefix\-valid update has the larger effect, reducing p99 miss by 5\.9–11\.0% relative to row\-0 update\. Mask construction has a smaller effect\. At matched realized sparsity, row\-local union achieves slightly lower p99 miss than row\-0 sharing\. Mean miss follows the same trend and is therefore omitted from the figure\. We therefore adopt row\-local union with prefix\-valid update asSpin’s default MTP strategy\.

##### Online acceptance behavior\.

We then evaluate the selected strategy online onSPEED\-Benchusing DeepSeek\-V4 Flash\. We compare it with all\-keep MTP and a sparse baseline using row\-0 sharing with row\-0 update\. Both sparse configurations use 40% target block sparsity\. Average acceptance length is the mean number of tokens committed per verification step, including the target\-model token, and therefore ranges from one to four\. Draft\-token acceptance rate is the fraction of proposed draft tokens accepted by the target model\.

We exclude 27 responses flagged for repetition across all three algorithms \(0\.59–0\.85% per configuration\) and ten requests with incomplete mask statistics, leaving 1,499 matched requests\.

Table 7:Online MTP acceptance behavior on a common subset of 1,499SPEED\-Benchrequests after excluding repetition\-flagged responses and requests without complete mask statistics\.Across the three configurations, average acceptance length differs by at most 0\.0049, and draft\-token acceptance rate differs by less than 0\.5% relative to all\-keep\. The selectedSpinstrategy therefore shows no visible change in observed MTP acceptance behavior on this matched subset while retaining 39\.51% realized block sparsity\. Its realized sparsity is only 0\.4% lower than that of the row\-0\-sharing baseline, indicating that verifier rows select largely overlapping KV blocks\.

### 4\.6SpinDesign Ablations

##### Block reduction\.

We ablate two block\-reduction design choices inSpin: where in the prediction algorithm slot\-level signals are reduced to blocks and how values within each block are pooled\. We define three variants, ordered from the latest to the earliest block reduction:

- •A0:Compute predicted slot scores, select top\-KKslots, then expand the selection to blocks\. Every block containing at least one selected slot is selected\.
- •A1 \(Spin’s default\):Compute predicted slot scores, pool them within each block to obtain predicted block scores, and select the top\-mtm\_\{t\}blocks\.
- •A2:Pool indexer scores within each block and use the pooled scores to update block\-levelV/DV/DEMA states\. Use these states to directly predict block scores for top\-mtm\_\{t\}block selection\.

A0 therefore performs block conversion latest, after slot selection\. A1 pools predicted slot scores into predicted block scores while retaining slot\-level EMA states\. A2 performs block reduction earliest, before the EMA predictor\. A0 has no pooling operator\. For A1 and A2, Max takes the largest value in each block, Top\-4 averages up to the four largest values, and Mean averages all values\.

Figure 3:Block\-reduction ablation\.Panel \(a\) compares P99 miss rates for A0, A1\-Max, and A2\-Max; panels \(b\) and \(c\) show relative changes in P99 miss for Mean and Top\-4 compared with Max pooling in A1 and A2\. Results average per\-layer, per\-trace P99 over all 60 curatedAA\-LCRtraces and C4A layers\.Figure 4:Exploration\-policy comparison onMRCRv2at 256K, 512K, and 1M context lengths\. Scores are shown as percentage changes relative to all\-keep at each target block sparsity\. Each point averages 40 repeats over 100 fixed prompts\. All exploration methods reserve 5% of the retained\-block budget\.Figure[3](https://arxiv.org/html/2610.09025#S4.F3)\(a\) shows that A0 and A1\-Max remain close, whereas A2\-Max produces consistently higher p99 miss\. Mean pooling consistently increases p99 miss\. Top\-4 remains close to Max but is not uniformly better\. Within A1, it is slightly worse at 10–20% sparsity, near parity at 25–35%, and modestly better at 40–50%\. We choose A1 because it directly controls target block sparsity, whereas A0 controls a pre\-expansion slot budget and therefore determines realized block sparsity only indirectly\. A1\-Max also achieves lower p99 miss than A2\-Max\. We choose Max over Top\-4 because it is simpler and avoids the low\-sparsity regression while remaining close across the evaluated range\.

##### Staleness mitigation\.

The MRCRv2 results show that random exploration can recover task quality without increasing the selected\-block budget\. We compare it with two alternatives under the same setup\. Staleness\-score exploration combines staleness and predicted importance to rank candidate blocks\. Block\-age exploration selects the least recently observed blocks\. All three methods reserve 5% of the retained\-block budget for exploration\.

Figure[4](https://arxiv.org/html/2610.09025#S4.F4)shows that all three exploration methods yield higher mean task scores than no exploration at 30–50% target sparsity, but no method consistently leads across context lengths and sparsities\. We choose random exploration for its simpler implementation, which requires no age tracking or age\-based selection\.

## 5Related Work

##### Temporal correlation in attention\.

It is well documented that attention scores form vertical and diagonal patterns in certain heads; many papers have leveraged these patterns to sparsify and speed up attention during decode\. H2O\[[27](https://arxiv.org/html/2610.09025#bib.bib27)\]selects heavy\-hitter tokens using accumulated historical attention scores while retaining a separate budget for recent tokens\. StreamingLLM\[[21](https://arxiv.org/html/2610.09025#bib.bib21)\]instead retains attention\-sink tokens together with a sliding window of recent tokens\. Several works also analyze the diagonal pattern and its origin in RoPE positional encoding\[[6](https://arxiv.org/html/2610.09025#bib.bib6),[10](https://arxiv.org/html/2610.09025#bib.bib10),[24](https://arxiv.org/html/2610.09025#bib.bib24)\]\. AttentionPredictor\[[23](https://arxiv.org/html/2610.09025#bib.bib23)\]learns to predict next\-token attention scores from their temporal history\. PRR\[[20](https://arxiv.org/html/2610.09025#bib.bib20)\]applies an EMA block predictor to dynamic sparse attention, using predicted blocks to speculatively execute core attention while the original block selector runs and repairing missed blocks afterward\. Rather than assuming temporal prediction is the right choice, Section[2](https://arxiv.org/html/2610.09025#S2)empirically compares previous\-iteration prediction against random and cross\-layer alternatives and shows that previous\-iteration prediction is more accurate\.

##### Indexer\-based sparse attention and its optimizations\.

Recent work reduces indexer cost through several complementary mechanisms\. Cross\-layer reuse avoids repeated indexing at adjacent layers\. IndexCache\[[5](https://arxiv.org/html/2610.09025#bib.bib5)\], discussed in Section[2](https://arxiv.org/html/2610.09025#S2), studies training\-free and training\-aware methods that reuse top\-KKindices across DSA layers\. GLM\-5\.2 natively deploys this mechanism from mid\-training\[[26](https://arxiv.org/html/2610.09025#bib.bib26)\]\. HISA\[[22](https://arxiv.org/html/2610.09025#bib.bib22)\]reduces indexer cost through hierarchical filtering, running DSA’s original token\-level indexer only within blocks selected by a coarse block\-level stage\. LongCat\-2\.0 later combines cross\-layer reuse and hierarchical filtering with streaming\-aware indexing in LongCat Sparse Attention\[[17](https://arxiv.org/html/2610.09025#bib.bib17)\]\. MiniMax Sparse Attention\[[15](https://arxiv.org/html/2610.09025#bib.bib15)\]instead adopts a native block\-level Index Branch that selects blocks independently for each GQA group\. MISA\[[28](https://arxiv.org/html/2610.09025#bib.bib28)\]routes a query\-dependent subset of indexer heads to perform full\-prefix token\-level scoring, while PIVOT\[[16](https://arxiv.org/html/2610.09025#bib.bib16)\]amortizes a full\-prefix indexer scan across grouped queries\. Closest to our work, TISA\[[2](https://arxiv.org/html/2610.09025#bib.bib2)\]partitions prefix positions into fixed contiguous groups, refreshes one group per decode step in round\-robin order, and reuses cached scores for the remaining groups before running top\-KKover the complete score cache\. Unlike TISA’s fixed refresh schedule,Spinexplicitly models vertical and diagonal temporal patterns with per\-slot and relative\-offset EMA states and selects indexer\-input blocks according to predicted importance\.

Other work targets later stages of sparse attention\. Guess\-Verify\-Refine\[[9](https://arxiv.org/html/2610.09025#bib.bib9)\]uses temporal correlation to accelerate the post\-indexer top\-KKoperator\.Spininstead applies temporal prediction before the indexer, using vertical and diagonal slot\-level EMAs to prune its input\. This introduces partial observations and motivates exploration within the selected\-block budget\.

## 6Limitations

Spinpredicts block importance from historical indexer scores and therefore has limited ability to anticipate patterns not represented in its history, such as sudden attention shifts\. In addition,Spin’s predictions affect their own future observations: once a block is skipped, its indexer scores are not observed, which may create self\-reinforcing prediction errors\. Random exploration gives skipped blocks another opportunity to be observed, but does not guarantee timely discovery or full task\-score recovery\. Incorporating other prediction signals is left for future work\.

Our current evaluation focuses on DeepSeek\-V4 and its native MTP\. Although we expect the observations and designs to apply more broadly, this has not been demonstrated on other indexer\-based sparse\-attention models, such as GLM\-5, MiniMax\-M3, and LongCat\-2\.0, or with other speculative\-decoding methods\.

## 7Conclusion

Spinshows that temporal statistics of indexer scores can serve as an effective proxy for current block importance\. Using vertical and diagonal EMA statistics,Spinskips up to 40% of indexer input blocks while preserving comparable task quality across the evaluated long\-context and agentic benchmarks\. Random exploration mitigates stale observations within the same block budget\. The core predictor also retains block sparsity without a visible change in acceptance behavior under DeepSeek\-V4’s native MTP\. Our integrated vLLM implementation improves output throughput by up to 14\.9% and reduces median inter\-token latency by up to 13\.2%\.Spinremains limited by sudden attention shifts and the feedback between its predictions and future observations\. Combining temporal history with current\-query or cross\-layer signals may reduce worst\-case misses and is a natural direction for future work\.

## References

- \[1\]Talor Abramovich, Maor Ashkenazi, Izzy Putterman, Benjamin Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein, and Yonatan Geifman\.SPEED\-Bench: A unified and diverse benchmark for speculative decoding\.*arXiv preprint*, February 2026\.URL[https://arxiv\.org/abs/2604\.09557](https://arxiv.org/abs/2604.09557)\.
- \[2\]Anonymous\.TISA: Exploiting temporal locality to accelerate indexing sparse attention\.*ACL Rolling Review \(ARR\) submission*, May 2026\.URL[https://openreview\.net/forum?id=6kOo3YtMdu](https://openreview.net/forum?id=6kOo3YtMdu)\.
- \[3\]Artificial Analysis\.Artificial analysis long context reasoning benchmark leaderboard, 2025\.URL[https://artificialanalysis\.ai/evaluations/artificial\-analysis\-long\-context\-reasoning](https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning)\.
- \[4\]Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li\.Longbench v2: Towards deeper understanding and reasoning on realistic long\-context multitasks, 2025\.URL[https://arxiv\.org/abs/2412\.15204](https://arxiv.org/abs/2412.15204)\.
- \[5\]Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, and Juanzi Li\.IndexCache: Accelerating sparse attention via cross\-layer index reuse\.*arXiv preprint*, March 2026\.URL[https://arxiv\.org/abs/2603\.12201](https://arxiv.org/abs/2603.12201)\.
- \[6\]Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković\.Round and round we go\! what makes rotary positional encodings useful?In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=GtvuNrk58a](https://openreview.net/forum?id=GtvuNrk58a)\.
- \[7\]Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan\.τ2\\tau^\{2\}\-bench: Evaluating conversational agents in a dual\-control environment, 2025\.URL[https://arxiv\.org/abs/2506\.07982](https://arxiv.org/abs/2506.07982)\.
- \[8\]Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel\.SummScreen: A dataset for abstractive screenplay summarization\.*Annual Meeting of the Association for Computational Linguistics \(ACL\)*, May 2022\.URL[https://aclanthology\.org/2022\.acl\-long\.589](https://aclanthology.org/2022.acl-long.589)\.
- \[9\]Long Cheng, Ritchie Zhao, Timmy Liu, Mindy Li, Xianjie Qiao, Kefeng Duan, Yu\-Jung Chen, Xiaoming Chen, Bita Darvish Rouhani, and June Yang\.Guess\-Verify\-Refine: Data\-aware top\-K for sparse\-attention decoding on Blackwell via temporal correlation\.*arXiv preprint*, April 2026a\.URL[https://arxiv\.org/abs/2604\.22312](https://arxiv.org/abs/2604.22312)\.
- \[10\]Yuan Cheng, Fengzhuo Zhang, Yunlong Hou, Cunxiao Du, Chao Du, Tianyu Pang, Aixin Sun, and Zhuoran Yang\.Demystifying the slash pattern in attention: The role of RoPE\.*arXiv preprint*, 2026b\.URL[https://arxiv\.org/abs/2601\.08297](https://arxiv.org/abs/2601.08297)\.
- \[11\]DeepSeek\-AI\.DeepSeek\-V3\.2: Pushing the frontier of open large language models\.*arXiv preprint*, December 2025\.URL[https://arxiv\.org/abs/2512\.02556](https://arxiv.org/abs/2512.02556)\.
- \[12\]DeepSeek\-AI\.DeepSeek\-V4: Towards highly efficient million\-token context intelligence\.*arXiv preprint*, April 2026\.URL[https://arxiv\.org/abs/2606\.19348](https://arxiv.org/abs/2606.19348)\.
- \[13\]Cheng\-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg\.RULER: What’s the real context size of your long\-context language models?*Conference on Language Modeling \(COLM\)*, April 2024\.URL[https://arxiv\.org/abs/2404\.06654](https://arxiv.org/abs/2404.06654)\.
- \[14\]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E\. Gonzalez, Hao Zhang, and Ion Stoica\.Efficient memory management for large language model serving with PagedAttention\.*Symposium on Operating Systems Principles \(SOSP\)*, September 2023\.URL[https://arxiv\.org/abs/2309\.06180](https://arxiv.org/abs/2309.06180)\.
- \[15\]Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Jinkai Hu, Jiayao Li, Rui Gao, Zekun Li, Songquan Zhu, Jingkai Zhou, and Pengyu Zhao\.MiniMax sparse attention\.*arXiv preprint*, June 2026\.URL[https://arxiv\.org/abs/2606\.13392](https://arxiv.org/abs/2606.13392)\.
- \[16\]Hong Liu, Yuan Cheng, Lin Niu, Yi Su, Yufei Xue, Anmin Liu, Guanghua Yu, and Jianchen Zhu\.PIVOT: Efficient query\-group indexing for token\-level sparse attention\.*arXiv preprint*, July 2026\.URL[https://arxiv\.org/abs/2607\.24593](https://arxiv.org/abs/2607.24593)\.
- \[17\]LongCat\.Introducing LongCat\-2\.0, June 2026\.URL[https://longcat\.chat/blog/longcat\-2\.0/](https://longcat.chat/blog/longcat-2.0/)\.
- \[18\]OpenAI\.Introducing GPT\-4\.1 in the API, April 2025\.URL[https://openai\.com/index/gpt\-4\-1/](https://openai.com/index/gpt-4-1/)\.
- \[19\]Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu\.RoFormer: Enhanced transformer with rotary position embedding\.*arXiv preprint*, April 2021\.URL[https://arxiv\.org/abs/2104\.09864](https://arxiv.org/abs/2104.09864)\.
- \[20\]Tianyu Wang, Gourav Rattihalli, Aditya Dhakal, Junbo Li, Zhiwei Ren, Dejan Milojicic, and Longfei Shangguan\.Predict, reuse, and repair: Accelerating dynamic sparse attention for long\-context LLM decoding\.*arXiv preprint*, June 2026\.URL[https://arxiv\.org/abs/2606\.30389](https://arxiv.org/abs/2606.30389)\.
- \[21\]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\.Efficient streaming language models with attention sinks\.*International Conference on Learning Representations \(ICLR\)*, September 2024\.URL[https://arxiv\.org/abs/2309\.17453](https://arxiv.org/abs/2309.17453)\.
- \[22\]Yufei Xu, Fanxu Meng, Fan Jiang, Yuxuan Wang, Ruijie Zhou, Zhaohui Wang, Jiexi Wu, Zhixin Pan, Xiaojuan Tang, Wenjie Pei, Tongxuan Liu, Di Yin, Xing Sun, and Muhan Zhang\.HISA: Efficient hierarchical indexing for fine\-grained sparse attention\.*arXiv preprint*, March 2026\.URL[https://arxiv\.org/abs/2603\.28458](https://arxiv.org/abs/2603.28458)\.
- \[23\]Qingyue Yang, Jie Wang, Xing Li, Zhihai Wang, Chen Chen, Lei Chen, Xianzhi Yu, Wulong Liu, Jianye Hao, Mingxuan Yuan, and Bin Li\.AttentionPredictor: Temporal patterns matter for KV cache compression\.*Conference on Neural Information Processing Systems \(NeurIPS\)*, February 2025\.URL[https://arxiv\.org/abs/2502\.04077](https://arxiv.org/abs/2502.04077)\.
- \[24\]Qingyue Yang, Jie Wang, Xing Li, Yinqi Bai, Xialiang Tong, Huiling Zhen, Jianye Hao, Mingxuan Yuan, and Bin Li\.Why attention patterns exist: A unifying temporal perspective analysis\.*International Conference on Learning Representations \(ICLR\)*, 2026\.URL[https://arxiv\.org/abs/2601\.21709](https://arxiv.org/abs/2601.21709)\.
- \[25\]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan\.τ\\tau\-bench: A benchmark for tool\-agent\-user interaction in real\-world domains, 2024\.URL[https://arxiv\.org/abs/2406\.12045](https://arxiv.org/abs/2406.12045)\.
- \[26\]Z\.ai\.GLM\-5\.2: Built for long\-horizon tasks, June 2026\.URL[https://z\.ai/blog/glm\-5\.2](https://z.ai/blog/glm-5.2)\.
- \[27\]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen\.H2O: Heavy\-hitter oracle for efficient generative inference of large language models\.*Conference on Neural Information Processing Systems \(NeurIPS\)*, June 2023\.URL[https://arxiv\.org/abs/2306\.14048](https://arxiv.org/abs/2306.14048)\.
- \[28\]Ruijie Zhou, Fanxu Meng, Yufei Xu, Tongxuan Liu, Guangming Lu, Muhan Zhang, and Wenjie Pei\.MISA: Mixture of indexer sparse attention for long\-context LLM inference\.*arXiv preprint*, May 2026\.URL[https://arxiv\.org/abs/2605\.07363](https://arxiv.org/abs/2605.07363)\.