Dropping the Anchor: Statistical Context Summarization for Distributed Systems via Pulsar Attention

# Statistical Context Summarization for Distributed Systems via Pulsar Attention
Source: [https://arxiv.org/html/2607.20457](https://arxiv.org/html/2607.20457)
###### Abstract

Inference with large language models \(LLMs\) on long sequences is computationally expensive due to the quadratic complexity of self\-attention\. Distributed blockwise methods such as Star Attention reduce this cost by sharding context across hosts, but rely on prepending a static, content\-blind copy of the first block to every host\. We propose Pulsar Attention, which replaces the static anchor with two lightweight, content\-aware components: a small attention\-sink prefix that stabilizes softmax, and compact cross\-block summaries built via a Max\-IDF heuristic that selects chunks containing globally rare tokens\. This reduces the Phase 1 per\-GPU FLOPs by up to 3\.3×\\timesover Star Attention while retaining an identical KV cache footprint\. On RULER and BABILong with Llama\-3\.1\-8B, Pulsar Attention outperforms both Star Attention and dense attention at sequence lengths up to 128K tokens, with absolute gains of up to 4\.7% over the dense baseline\.

Machine Learning, ICML, Deep Learning, Keyword3, Keyword4

## 1Introduction

Large Language Models \(LLMs\) have demonstrated strong performance on tasks requiring long context windows, but scaling inference remains expensive: self\-attention is quadratic in sequence length, and the KV cache grows linearly, quickly exhausting device memory on a single GPU\.

Distributed methods address this by sharding context across GPUs\. Ring Attention\(Liuet al\.,[2023](https://arxiv.org/html/2607.20457#bib.bib3)\)computes exact global attention by circulating KV blocks in a ring, but requires coordinated communication at every layer\. Star Attention\(Acharyaet al\.,[2025](https://arxiv.org/html/2607.20457#bib.bib2)\)eliminates this communication during context encoding by processing independent blocks in parallel, recovering global attention at query time through a lightweight softmax merge\. To prevent softmax collapse, each host prepends a static copy of the first block \(called the anchor block\) to its input\.

The anchor is effective but it duplicates the same tokens regardless of relevance, doubling per\-host sequence length and FLOPs during context encoding\. Because it is fixed to the sequence start, a blockiireceives no information about the intermediate blocks, which widens as sequences grow\.

We propose Pulsar Attention, which replaces the static anchor with two lightweight, content\-aware components: \(i\) a small attention\-sink prefix \(64 tokens\) that stabilizes softmax at a fraction of the anchor’s cost, and \(ii\) compact Max\-IDF block summaries that select contiguous chunks containing globally rare tokens from each preceding block\. Summaries propagate causally and adapt to each block’s content, prioritizing tokens most likely to carry task\-critical information\. Our main contributions are:

- •We replace the static anchor with content\-aware summaries and attention sinks, reducing Phase 1 per\-GPU FLOPs by up to 3\.3×\\timesover Star Attention while retaining an identical KV cache footprint\.
- •We introduce Max\-IDF chunk scoring and validate it against three alternative heuristics on BABILong, showing rare\-token selection best captures cross\-block context effectively\.
- •We show on RULER that Pulsar Attention outperforms both Star Attention and dense attention on context lengths upto 128K, with gains of up to \+4\.7% over the dense baseline\.

## 2Background

Transformer\-based LLMs cache key and value vectors for all previous tokens during autoregressive decoding\. While this avoids redundant computation, KV cache memory grows linearly with sequence length, quickly exhausting the device memory at long contexts\.

Liuet al\.\([2023](https://arxiv.org/html/2607.20457#bib.bib3)\)addresses this by distributing the KV cache across GPUs and circulating blocks in a ring to compute exact global attention\. This enables arbitrarily long sequences but requires cross\-host communication at every layer\.

Acharyaet al\.\([2025](https://arxiv.org/html/2607.20457#bib.bib2)\)eliminates this communication during context encoding with a two\-phase design\. In Phase 1, the context is split into contiguous blocks processed independently on parallel hosts; a static copy of the first block \(the anchor block\) is prepended to each host’s input to prevent softmax collapse\. In Phase 2, the query is broadcast to all hosts, which compute local attention scores and aggregate them via an online softmax merge to recover global attention\. While effective, the anchor duplicates the same tokens regardless of relevance, doubling per\-host sequence length and FLOPs during Phase 1\. Our method addresses this limitation\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2607.20457v1/x1.png)

\(a\)Phase 1: Statistical context encoding\. Hostiiassembles input\[𝒮sink​‖Σ0‖​⋯​‖Σi−1‖​Bi\]\[\\mathcal\{S\}\_\{\\text\{sink\}\}\\parallel\\Sigma\_\{0\}\\parallel\\cdots\\parallel\\Sigma\_\{i\-1\}\\parallel B\_\{i\}\], runs a forward pass, then discards sink and summary KV entries, retaining onlyKV​\(Bi\)\\text\{KV\}\(B\_\{i\}\)\.
![Refer to caption](https://arxiv.org/html/2607.20457v1/x2.png)

\(b\)Phase 2: Global query encoding\. Each host attends over its retained KV shard; anall\_gatherand online softmax merge\(Milakov and Gimelshein,[2018](https://arxiv.org/html/2607.20457#bib.bib17)\)recovers exact global attention without transmitting the full KV cache\.

Figure 1:Pulsar Attention overview\.\(Left\)Phase 1 replaces the static anchor block with a content\-aware prefix: a small attention\-sink and Max\-IDF statistical summaries of all causally preceding blocks\.\(Right\)Phase 2 broadcasts the query to all hosts and merges local attention scores into an exact global output via online softmax\.### 3\.1Phase 1: Statistical Context Encoding

Like Star Attention, Pulsar Attention partitions the input sequence of lengthLLintonncontiguous blocks\{B0,B1,…,Bn−1\}\\\{B\_\{0\},B\_\{1\},\\ldots,B\_\{n\-1\}\\\}, each assigned to a parallel host \([Figure1\(a\)](https://arxiv.org/html/2607.20457#S3.F1.sf1)\)\. Rather than duplicating the entire first block as a static anchor, we replace it with two lightweight, content\-aware components: an attention\-sink prefix and Max\-IDF block summaries\. Context encoding proceeds in four steps\.

Attention sinks\.The firstssinks\_\{\\text\{sink\}\}tokens ofB0B\_\{0\}are extracted as a fixed prefix𝒮sink\\mathcal\{S\}\_\{\\text\{sink\}\}\(defaultssink=64s\_\{\\text\{sink\}\}=64\)\. FollowingXiaoet al\.\([2023](https://arxiv.org/html/2607.20457#bib.bib10)\), these tokens stabilize the softmax distribution during blockwise attention at a fraction of the cost of a full anchor block\.

Summary generation\.Before any neural computation, every host constructs an identical corpus\-level IDF table from the token IDs of the full sequence, anO​\(L\)O\(L\)integer\-only operation that requires no inter\-host communication and no GPU time\. Using this table, each host scores non\-overlapping contiguous chunks ofmmtokens \(defaultm=32m=32\) within each block via a Max\-IDF heuristic:

score​\(C\)=maxt∈C⁡IDF⁡\(t\)\\displaystyle\\text\{score\}\(C\)=\\max\_\{t\\in C\}\\operatorname\{IDF\}\(t\)\(1\)IDF⁡\(t\)=log⁡\(nmax⁡\(df⁡\(t\),1\)\)\\displaystyle\\operatorname\{IDF\}\(t\)=\\log\\left\(\\frac\{n\}\{\\max\(\\operatorname\{df\}\(t\),1\)\}\\right\)\(2\)
wheredf⁡\(t\)\\operatorname\{df\}\(t\)is the number of blocks containing tokentt\. The top\-kkchunks per block are selected and concatenated in positional order to form block summaryΣi\\Sigma\_\{i\}, with a per\-block budget ofσ\\sigmatokens \(defaultσ=0\.125×\|Bi\|\\sigma=0\.125\\times\|B\_\{i\}\|\)\. We choose contiguous chunks rather than individual tokens to preserve local syntactic structure in the resulting key–value representations\. Max\-IDF is preferred over averaging heuristics \(TF\-IDF, BM25\) because retrieval\-critical information like UUIDs, entity names, numerical values are typically concentrated in a single rare token per chunk; averaging dilutes this signal by a factor of1/m1/m\(see Section[5\.1](https://arxiv.org/html/2607.20457#S5.SS1)for an empirical comparison\)\.

Causal Assembly\.Each hostiiprepends only summaries from causally preceding blocks along with𝒮sink\\mathcal\{S\}\_\{\\text\{sink\}\}, forming the augmented Phase 1 input:

Inputi=\{\[B0\]i=0\[𝒮sink​‖Σ0‖​⋯​‖Σi−1‖​Bi\]i\>0\\text\{Input\}\_\{i\}=\\begin\{cases\}\[B\_\{0\}\]&i=0\\\\ \[\\mathcal\{S\}\_\{\\text\{sink\}\}\\\|\\Sigma\_\{0\}\\\|\\cdots\\\|\\Sigma\_\{i\-1\}\\\|B\_\{i\}\]&i\>0\\end\{cases\}\(3\)Each token retains its original global position ID, preserving the inter\-token distances encoded by RoPE \(see Appendix[E](https://arxiv.org/html/2607.20457#A5)for a comparison with contiguous re\-numbering\)\.

KV Discard\.After the forward pass, KV states for𝒮sink\\mathcal\{S\}\_\{\\text\{sink\}\}and\{Σj\}j<i\\\{\\Sigma\_\{j\}\\\}\_\{j<i\}are discarded; only the\|Bi\|\|B\_\{i\}\|entries belonging to the block itself are retained\. The final distributed KV cache spans exactlyLLtokens with no duplication, and peak per\-host memory during Phase 1 is strictly lower than Star Attention’s: each host processes at mostL/n\+ssink\+CL/n\+s\_\{\\text\{sink\}\}\+Ctokens versus Star’s2​L/n2L/n, whereC≪L/nC\\ll L/nis the total summary budget\.

Algorithm 1Pulsar Attention1:Input:Blocks

\{B0,…,Bn−1\}\\\{B\_\{0\},\\dots,B\_\{n\-1\}\\\}, query

QQ,

ss,

mm,

kk,

TT
2:Output:Generated token sequence

3:

𝒮sink←B0\[0:s\]\\mathcal\{S\}\_\{\\text\{sink\}\}\\leftarrow B\_\{0\}\[0\{:\}s\]; build

IDF​\(t\)\\mathrm\{IDF\}\(t\)from

df​\(t\)\\mathrm\{df\}\(t\)across all blocks

4:forhost

i=0,…,n−1i=0,\\dots,n\{\-\}1in paralleldo

5:

Σi←\\Sigma\_\{i\}\\leftarrowtop\-

kkchunks of

BiB\_\{i\}by

maxt∈C⁡IDF​\(t\)\\max\_\{t\\in C\}\\mathrm\{IDF\}\(t\), in positional order

6:

Inputi←\[B0\]\\text\{Input\}\_\{i\}\\leftarrow\[B\_\{0\}\]if

i=0i\{=\}0, else

\[𝒮sink​‖Σ0‖​⋯​‖Σi−1‖​Bi\]\[\\mathcal\{S\}\_\{\\text\{sink\}\}\\\!\\parallel\\\!\\Sigma\_\{0\}\\\!\\parallel\\\!\\cdots\\\!\\parallel\\\!\\Sigma\_\{i\-1\}\\\!\\parallel\\\!B\_\{i\}\]
7:

KVi←ForwardPass​\(Inputi\)\\text\{KV\}\_\{i\}\\leftarrow\\textsc\{ForwardPass\}\(\\text\{Input\}\_\{i\}\); discard non\-

BiB\_\{i\}KV entries

8:endfor

9:for

t=1t=1to

TTdo

10:

\(Ah,ℓh\)←FlashAttn​\(q,KVh\)\(A\_\{h\},\\ell\_\{h\}\)\\leftarrow\\textsc\{FlashAttn\}\(q,\\text\{KV\}\_\{h\}\)for each host

hhin parallel;all\_gather

\{\(Ah,ℓh\)\}\\\{\(A\_\{h\},\\ell\_\{h\}\)\\\}
11:

\(A,ℓ\)←OnlineSoftmaxMerge​\(\{\(Ah,ℓh\)\}h=0n−1\)\(A,\\ell\)\\leftarrow\\textsc\{OnlineSoftmaxMerge\}\\bigl\(\\\{\(A\_\{h\},\\ell\_\{h\}\)\\\}\_\{h=0\}^\{n\-1\}\\bigr\)
12:

q←argmax⁡\(LMHead​\(A\)\)q\\leftarrow\\operatorname\{argmax\}\(\\textsc\{LMHead\}\(A\)\); append

KV​\(q\)\\mathrm\{KV\}\(q\)to

KVn−1\\text\{KV\}\_\{n\-1\};if

q=EOSq=\\texttt\{EOS\}then break

13:endfor

14:returngenerated sequence

### 3\.2Phase 2: Global Query Encoding and Generation

Because Phase 1 produces a clean, non\-overlapping distributed KV cache, Phase 2 is structurally identical to Star Attention\(Acharyaet al\.,[2025](https://arxiv.org/html/2607.20457#bib.bib2)\): the query is broadcast to all hosts, each computes local attention via FlashAttention, and a designated query host aggregates the results through an online softmax merge\(Milakov and Gimelshein,[2018](https://arxiv.org/html/2607.20457#bib.bib17)\)\. The full pseudocode is given in Algorithm[1](https://arxiv.org/html/2607.20457#alg1)\.

## 4Experiments

### 4\.1Experimental Setup

Model\.We evaluate on Meta\-Llama\-3\.1\-8B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.20457#bib.bib1)\), which supports a native 128K context window\. All experiments use bfloat16 precision with FlashAttention\-2\. Generation uses greedy decoding\.

Distributed configuration\.Following Star Attention, the context is partitioned intoB=4B=4equal blocks, with one block assigned per GPU\. All distributed experiments use 4 hosts for all context lengths\. However we switched to 8 blocks for 128K due to memory efficiency\. The summary budget is fixed at 12\.5% of block size \(chunk sizem=32m\{=\}32tokens, with the number of selected chunkskkscaled proportionally to context length\)\. The attention\-sink prefix is fixed at 64 tokens across all settings\.

Benchmarks\.We evaluate on two benchmarks: \(i\) RULER\(Hsiehet al\.,[2024](https://arxiv.org/html/2607.20457#bib.bib18)\), a synthetic long\-context benchmark comprising 13 tasks across five categories \(single\-needle retrieval, multi\-needle retrieval, multi\-hop tracing, aggregation, and question answering\), with 500 samples per task per context length; and \(ii\) BABILong\(Kuratovet al\.,[2024](https://arxiv.org/html/2607.20457#bib.bib19)\), a reasoning benchmark requiring multi\-step inference over supporting facts embedded in long distractor text, with 1,000 samples per task\. We report exact\-match accuracy for both benchmarks\.

Baselines\.We compare against four baselines: \(i\) Full \(Dense\) Attention, computed on a single GPU as the accuracy ceiling; \(ii\) Star Attention\(Acharyaet al\.,[2025](https://arxiv.org/html/2607.20457#bib.bib2)\), the most directly comparable distributed method; \(iii\) StreamingLLM\(Xiaoet al\.,[2023](https://arxiv.org/html/2607.20457#bib.bib10)\), a sparse method combining sink tokens with sliding\-window attention; and \(iv\) MInference\(Jianget al\.,[2024a](https://arxiv.org/html/2607.20457#bib.bib8)\), which dynamically selects per\-head sparse attention patterns\. Baselines marked with†are reproduced fromAcharyaet al\.\([2025](https://arxiv.org/html/2607.20457#bib.bib2)\)using the same model and hardware configuration\.

### 4\.2Results on RULER

Table 1 reports average accuracy across all 13 RULER tasks at context lengths from 16K to 128K\. Pulsar Attention outperforms Star Attention at 32K \(\+2\.8%\), 64K \(\+6\.1%\), and 128K \(\+4\.7%\), with the gap widening at longer contexts\. At 16K, Pulsar trails Star Attention by 0\.9%, likely because the summary budget \(512 tokens from a 4K block\) provides less coverage than Star’s full 4K anchor at short contexts where the anchor’s cost is relatively low\.

Notably, Pulsar also exceeds Full Attention accuracy at 32K \(\+4\.0%\), 64K \(\+4\.7%\), and 128K \(\+2\.8%\)\. We hypothesize that block\-local attention with content\-aware summaries produces sharper softmax distributions than full\-sequence attention, which increasingly dilutes signal across irrelevant tokens at long contexts\.

Table 1:RULER average accuracy \(%\) on Llama\-3\.1\-8B\.†FromAcharyaet al\.\([2025](https://arxiv.org/html/2607.20457#bib.bib2)\), same model and setup\.Bold= best among efficiency\-focused methods\.Among the single\-GPU baselines, MInference performs competitively at 16K\-64K but degrades sharply to 58\.2% at 128K \(\-21\.3 points versus Pulsar\), suggesting that its offline sparse pattern estimation becomes unreliable at extreme lengths\. StreamingLLM collapses past 16K, confirming that recency\-biased context windows cannot substitute for cross\-block information flow\.

### 4\.3Results on BABILong

Table[3](https://arxiv.org/html/2607.20457#S5.T3)reports accuracy on three BABILong tasks \(qa1, qa3, qa5\) at 16K context length usingB=4B=4blocks of 4K tokens each\. Pulsar Attention with Max\-IDF scoring \(s=512s=512\) achieves 37% on qa1, 44% on qa3, and 63% on qa5\. On the multi\-hop tasks \(qa3 and qa5\), Pulsar outperforms Star Attention \(33% and 58% respectively\) by \+11 and \+5 points\. The advantage on qa3 is particularly notable: this task requires chaining three supporting facts that may reside in different blocks, and Max\-IDF’s ability to select chunks containing rare entity tokens provides cross\-block context that Star’s position\-fixed anchor cannot\. We present extended BABILong results across context lengths and scoring heuristics in the ablation study \(Section[5\.1](https://arxiv.org/html/2607.20457#S5.SS1)\)\.

### 4\.4RULER Task\-Category Analysis

Table[2](https://arxiv.org/html/2607.20457#S4.T2)reports per\-task\-category accuracy at 128K tokens on RULER, where the differences between methods are most pronounced\.

Table 2:Per\-task accuracy \(%\) at 128K on Llama\-3\.1\-8B\. NIAH MultiKey and NIAH Single scores are averaged across their three sub\-tasks; QA averages SQuAD and HotpotQA\.Retrieval \(NIAH\)\.Single\-needle retrieval is effectively saturated \( 100%\) across all methods\. On MultiKey and MultiQuery, Pulsar maintains near\-parity with Dense, as Max\-IDF reliably promotes chunks containing rare UUID or keyword needles\. The notable exception is NIAH MultiValue \(63\.5% vs\. Dense 91\.6%\): this task requires retrieving multiple values associated with a single key, where the values may reside in different blocks\. Max\-IDF selects the chunk containing the rare key identifier but may miss value chunks whose tokens are individually less distinctive\.

Aggregation\.The most striking result is Common Words Extraction, where both Full Attention \(0\.04%\) and Star Attention \(0\.04%\) effectively fail at 128K, while Pulsar achieves 81\.2%\. This task requires identifying tokens that appear frequently across the full sequence, precisely the global co\-occurrence signal that diluted dense attention loses at long contexts and that a position\-fixed anchor cannot reconstruct\. Pulsar’s causal summaries propagate recurring tokens from earlier blocks into each host’s local context, restoring partial global frequency information without transmitting the full KV cache\. Frequent Words Extraction shows the same pattern at a smaller magnitude \(Pulsar 80\.7% vs\. Dense 72\.3% vs\. Star 75\.9%\)\.

Multi\-hop reasoning\.Variable Tracking and QA show broad parity across all three methods \(within 4 points\), consistent with the expectation that multi\-hop inference benefits from cross\-block context but ultimately requires denser inter\-block communication than the current summary budget provides\.

## 5Ablations

### 5\.1Scoring Heuristic Comparison

Table[3](https://arxiv.org/html/2607.20457#S5.T3)compares the four scoring heuristics against Even\-Spaced selection and Star Attention’s anchor onBABILong\-16K across three per\-block summary budgets\. Block size is fixed at 4 K tokens\. \(see Appendix[A](https://arxiv.org/html/2607.20457#A1)for task details\)\.

Table 3:BABILong\-16K accuracy \(%\) across three summary budgets \(B=4B\\\!=\\\!4, block size 4 K\)\.Bold: best among the four statistical heuristics per group\. Star Attention uses a fixed anchor block independent of summary budget\.\(1\)Max\-IDFachieves the bestqa5score ats=128s\\\!=\\\!128and is the only heuristic that*improves*with budget on the most discriminative task:qa3rises from 36 to 44 ats=512s\\\!=\\\!512, an 11\-point margin over Star Attention\.

\(2\)Even\-Spacedrecords the highestqa1ats=128s\\\!=\\\!128but degrades monotonically onqa3as budget grows \(36→26→2536\\to 26\\to 25\), confirming that uniform coverage wastes tokens on low\-information distractor spans once the budget is large enough to exhibit selection bias\.

\(3\)Entropyperforms strongly ats=128s\\\!=\\\!128\(qa3: 39;qa5: 65\) but collapses ats=1024s\\\!=\\\!1024\(qa3: 27\), indicating that diversity\-based selection saturates when the budget approaches a large fraction of block size\.

\(4\)All four statistical heuristics outperform Star Attention onqa5at every budget, confirming that scored selection surpasses a static anchor\.

#### Why Max\-IDF excels at multi\-hop tasks\.

Tasksqa3andqa5require joining multiple evidence spans, each mentioning a specific entity\. These entity tokens receive the highest IDF values like proper nouns and task\-specific names appear in only one or two blocks, yieldingIDF​\(t\)≈log⁡N\\mathrm\{IDF\}\(t\)\\approx\\log N\. Max\-IDF guarantees that any chunk containing a rare entity is selected regardless of surrounding distractor tokens\. TF\-IDF and BM25*average*IDF over the chunk, so a single high\-IDF entity among 31 filler tokens contributes only∼3%\\sim\\\!3\\%of the score\. Entropy is blind to global rarity entirely\.

#### Budget Selection\.

We useds=12\.5%s=12\.5\\%of block size \(512 tokens for a 4 K block\), the configuration used in all RULER experiments\. This budget provides sufficient coverage for multi\-hop entity chains while remaining a small fraction of block size, limiting Phase 1 overhead\.

### 5\.2Computational Efficiency

We derive Phase 1 memory and FLOPs analytically for Llama\-3\.1\-8B\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.20457#bib.bib1)\)\(Llayers=32L\_\{\\text\{layers\}\}\\\!=\\\!32,HQ=32H\_\{Q\}\\\!=\\\!32,HK​V=8H\_\{KV\}\\\!=\\\!8,dh=128d\_\{h\}\\\!=\\\!128, fp16\), usingB=4B\\\!=\\\!4,s=512s\\\!=\\\!512,k=64k\\\!=\\\!64\. Under FlashAttention\(Daoet al\.,[2022](https://arxiv.org/html/2607.20457#bib.bib7)\), HBM activation memory scales asO​\(n\)O\(n\)while FLOPs remainΘ​\(n2​d\)\\Theta\(n^\{2\}d\)\.

#### Critical\-path sequence length\.

The worst\-case per\-GPU input \(last block with all prior summaries prepended\) is:

nDense=L,nStar=L2,nPulsar=L4\+k\+\(B−1\)​s⏟=1,600\.n\_\{\\text\{Dense\}\}=L,\\quad n\_\{\\text\{Star\}\}=\\tfrac\{L\}\{2\},\\quad n\_\{\\text\{Pulsar\}\}=\\tfrac\{L\}\{4\}\+\\underbrace\{k\+\(B\{\-\}1\)\\,s\}\_\{=\\,1\{,\}600\}\.\(4\)

#### KV cache memory\.

Each token occupies128​KB128\\,\\text\{KB\}of KV cache\. After Phase 1 KV discard, Star and Pulsar both retain only the local block, giving identical per\-GPU KV cache:

MStar/PulsarKV=LB×128​KB,MDenseKV=L×128​KB\.M^\{\\text\{KV\}\}\_\{\\text\{Star/Pulsar\}\}=\\frac\{L\}\{B\}\\times 128\\,\\text\{KB\},\\quad M^\{\\text\{KV\}\}\_\{\\text\{Dense\}\}=L\\times 128\\,\\text\{KB\}\.\(5\)AtL=64​KL\\\!=\\\!64\\text\{K\}: 2\.15 GB per GPU \(distributed\) vs\. 8\.59 GB for Dense which is a4×4\\timesper\-GPU reduction exact and independent ofss,kk, orLL\.

#### Phase 1 FLOPs\.

FLOPs per layer for GQA attention areℱ=2​n2​\(HQ\+HK​V\)​dh=2​n2×5,120\\mathcal\{F\}=2\\,n^\{2\}\\,\(H\_\{Q\}\+H\_\{KV\}\)\\,d\_\{h\}=2\\,n^\{2\}\\times 5\{,\}120\. Table[4](https://arxiv.org/html/2607.20457#S5.T4)reports concrete values at three context lengths\.

Table 4:Phase 1 computational cost \(Llama\-3\.1\-8B,B=4B\\\!=\\\!4,s=512s\\\!=\\\!512,k=64k\\\!=\\\!64, fp16\)\. Activation memory and FLOPs are for the critical \(last\) block; KV cache is the per\-GPU retained cache after Phase 1 discard\.Star Attention’s constant4×4\\timesFLOPs reduction reflects its anchor always equalling one local block:nStar2/nDense2=1/4n\_\{\\text\{Star\}\}^\{2\}/n\_\{\\text\{Dense\}\}^\{2\}=1/4\. Pulsar’s growing advantage arises because the fixed prefix of 1,600 tokens represents a diminishing fraction of block size asLLgrows:

nStar2nPulsar2=\(2​LL\+6,400\)2→L→∞4×\.\\frac\{n\_\{\\text\{Star\}\}^\{2\}\}\{n\_\{\\text\{Pulsar\}\}^\{2\}\}=\\left\(\\frac\{2L\}\{L\+6\{,\}400\}\\right\)^\{\\\!2\}\\;\\xrightarrow\{L\\to\\infty\}\\;4\\times\.\(6\)AtL=64​KL\\\!=\\\!64\\text\{K\}the ratio is already3\.3×3\.3\\times\.

#### Estimated wall\-clock speedup\.

Star Attention\(Acharyaet al\.,[2025](https://arxiv.org/html/2607.20457#bib.bib2)\)reports wall\-clock speedups of1\.1×1\.1\\times,1\.2×1\.2\\times, and1\.8×1\.8\\timesover Dense at 16K, 32K, and 64K \(B=4B\\\!=\\\!4\)\. Since Pulsar’s Phase 2 is architecturally identical to Star’s, its additional gain derives entirely from Phase 1:

rPulsar≈rStar×nStar2nPulsar2\.r\_\{\\text\{Pulsar\}\}\\;\\approx\\;r\_\{\\text\{Star\}\}\\times\\frac\{n\_\{\\text\{Star\}\}^\{2\}\}\{n\_\{\\text\{Pulsar\}\}^\{2\}\}\.\(7\)
Table 5:Estimated Pulsar wall\-clock speedup over Dense, derived from Eq\. \([7](https://arxiv.org/html/2607.20457#S5.E7)\) with Star Attention’s reported speedups\(Acharyaet al\.,[2025](https://arxiv.org/html/2607.20457#bib.bib2)\)as a calibration baseline\.The4×4\\timesper\-GPU KV cache reduction is exact across allLL\. The Phase 1 FLOPs advantage \(2\.1×2\.1\\times–3\.3×3\.3\\timesover Star\) compounds with Star’s baseline to yield an estimated6\.0×6\.0\\timesspeedup over Dense at 64K\. Summary scoring adds no overhead: IDF table construction requiresO​\(L\)O\(L\)integer operations, a fraction below10−610^\{\-6\}% of attention FLOPs\.

## 6Related Work

#### Distributed long\-context inference\.

Sequence parallelism\(Korthikantiet al\.,[2023](https://arxiv.org/html/2607.20457#bib.bib4)\)shards the sequence dimension with an all\-reduce per layer\. Ring Attention\(Liuet al\.,[2023](https://arxiv.org/html/2607.20457#bib.bib3)\)removes this synchronization by overlapping communication with computation in a ring topology, but still requires cross\-host transfers at every layer\. Star Attention\(Acharyaet al\.,[2025](https://arxiv.org/html/2607.20457#bib.bib2)\)eliminates Phase 1 communication entirely by prepending a static first\-block anchor to each host\. Pulsar inherits Star’s two\-phase structure and communication\-free Phase 1 but replaces the content\-blind anchor with scored statistical summaries, recovering cross\-block information without reintroducing communication\.

#### Sparse and efficient attention\.

Longformer\(Beltagyet al\.,[2020](https://arxiv.org/html/2607.20457#bib.bib5)\)and BigBird\(Zaheeret al\.,[2020](https://arxiv.org/html/2607.20457#bib.bib6)\)combine sliding\-window attention with global tokens atO​\(n\)O\(n\)cost but use fixed structural patterns\. MInference\(Jianget al\.,[2024a](https://arxiv.org/html/2607.20457#bib.bib8)\)dynamically selects sparse attention patterns per head during pre\-filling, achieving strong results up to 64K but degrading at 128K in our experiments \(Table[1](https://arxiv.org/html/2607.20457#S4.T1)\)\. FlexPrefill\(Laiet al\.,[2025](https://arxiv.org/html/2607.20457#bib.bib9)\)extends dynamic sparsity with query\-dependent budget allocation; its gains are orthogonal to our distributed encoding phase\. StreamingLLM\(Xiaoet al\.,[2023](https://arxiv.org/html/2607.20457#bib.bib10)\)preserves attention\-sink tokens for streaming generation; we adopt its sink mechanism as one component of our prefix but pair it with content\-aware summaries rather than a recency\-based sliding window\. FlashAttention\(Daoet al\.,[2022](https://arxiv.org/html/2607.20457#bib.bib7)\)provides the tiled block computation underlying all our Phase 1 forward passes\.

#### KV cache compression and context selection\.

H2O\(Zhanget al\.,[2023](https://arxiv.org/html/2607.20457#bib.bib11)\), SnapKV\(Liet al\.,[2024](https://arxiv.org/html/2607.20457#bib.bib12)\), and PyramidKV\(Caiet al\.,[2024](https://arxiv.org/html/2607.20457#bib.bib13)\)evict low\-importance KV entries during generation; these are orthogonal to Pulsar’s Phase 1 and compatible to Phase 2\. Prompt\-compression methods such as LLMLingua\(Jianget al\.,[2023](https://arxiv.org/html/2607.20457#bib.bib14)\)and LongLLMLingua\(Jianget al\.,[2024b](https://arxiv.org/html/2607.20457#bib.bib15)\)use a proxy LM to score and remove tokens before the main forward pass\. Pulsar’s Max\-IDF scoring is intentionally non\-neural; it relies on corpus\-level token statistics computable in a single CPU pass, adding negligible overhead while drawing on classical IDF\-based retrieval\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2607.20457#bib.bib16)\)\.

## 7Conclusion

We introduced Pulsar Attention, a drop\-in replacement for Star Attention’s static anchor block that uses two lightweight, content\-aware components: a 64\-token attention\-sink prefix and Max\-IDF block summaries at 12\.5% of block size\. On RULER with Llama\-3\.1\-8B\-Instruct, Pulsar outperforms both Dense and Star Attention at 32K–128K tokens, with gains of up to \+4\.7% over Dense and \+6\.1% over Star, while reducing Phase 1 per\-GPU FLOPs by up to 3\.3×\\timesrelative to Star\. The retained KV cache is identical to Star Attention’s, making Pulsar compatible with existing Phase 2 infrastructure and post\-hoc KV compression methods\. The main limitation is on tasks requiring retrieval of multiple values per key \(NIAH MultiValue\), where single\-score chunk ranking can miss value\-bearing chunks that lack globally rare tokens\. Future work will address this through query\-aware or multi\-score selection strategies, and validate the FLOPs\-based speedup estimates with end\-to\-end wall\-clock profiling\.

## References

- S\. Acharya, F\. Jia, and B\. Ginsburg \(2025\)Star attention: efficient LLM inference over long sequences\.InForty\-second International Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2607.20457#S1.p2.1),[§2](https://arxiv.org/html/2607.20457#S2.p3.1),[§3\.2](https://arxiv.org/html/2607.20457#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.20457#S4.SS1.p4.1),[Table 1](https://arxiv.org/html/2607.20457#S4.T1),[Table 1](https://arxiv.org/html/2607.20457#S4.T1.5.2),[§5\.2](https://arxiv.org/html/2607.20457#S5.SS2.SSS0.Px4.p1.4),[Table 5](https://arxiv.org/html/2607.20457#S5.T5),[Table 5](https://arxiv.org/html/2607.20457#S5.T5.18.2),[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px1.p1.1)\.
- I\. Beltagy, M\. E\. Peters, and A\. Cohan \(2020\)Longformer: the long\-document transformer\.arXiv preprint arXiv:2004\.05150\.Cited by:[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px2.p1.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu,et al\.\(2024\)Pyramidkv: dynamic kv cache compression based on pyramidal information funneling\.arXiv preprint arXiv:2406\.02069\.Cited by:[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px3.p1.1)\.
- T\. Dao, D\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)FlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.InAdvances in Neural Information Processing Systems \(NeurIPS 2022\),Vol\.35,pp\. 16344–16359\.Cited by:[§5\.2](https://arxiv.org/html/2607.20457#S5.SS2.p1.9),[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px2.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2607.20457#S4.SS1.p1.1),[§5\.2](https://arxiv.org/html/2607.20457#S5.SS2.p1.9)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.arXiv preprint arXiv:2404\.06654\.Cited by:[§A\.2](https://arxiv.org/html/2607.20457#A1.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.20457#S4.SS1.p3.1)\.
- H\. Jiang, Y\. Li, C\. Zhang, Q\. Wu, X\. He, S\. Garg, Q\. Chen, J\. Jiang, Y\. Wei, L\. Chi,et al\.\(2024a\)MInference 1\.0: accelerating pre\-filling for long\-context LLMs via dynamic sparse attention\.InAdvances in Neural Information Processing Systems \(NeurIPS 2024\),Cited by:[§4\.1](https://arxiv.org/html/2607.20457#S4.SS1.p4.1),[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px2.p1.1)\.
- H\. Jiang, Q\. Wu, C\. Lin, Y\. Yang, and L\. Qiu \(2023\)Llmlingua: compressing prompts for accelerated inference of large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 13358–13376\.Cited by:[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px3.p1.1)\.
- H\. Jiang, Q\. Wu, X\. Luo, D\. Li, C\. Lin, Y\. Yang, and L\. Qiu \(2024b\)Longllmlingua: accelerating and enhancing llms in long context scenarios via prompt compression\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1658–1677\.Cited by:[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px3.p1.1)\.
- V\. A\. Korthikanti, J\. Casper, S\. Lym, L\. McAfee, M\. Andersch, M\. Shoeybi, and B\. Catanzaro \(2023\)Reducing activation recomputation in large transformer models\.Proceedings of Machine Learning and Systems5,pp\. 341–353\.Cited by:[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px1.p1.1)\.
- Y\. Kuratov, A\. Bulatov, P\. Anokhin, I\. Rodkin, D\. Sorokin, A\. Sorokin, and M\. Burtsev \(2024\)BABILong: testing the limits of LLMs with long context reasoning\-in\-a\-haystack\.arXiv preprint arXiv:2406\.10149\.Cited by:[§A\.1](https://arxiv.org/html/2607.20457#A1.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.20457#S4.SS1.p3.1)\.
- X\. Lai, J\. Lu, Y\. Luo, Y\. Ma, and X\. Zhou \(2025\)Flexprefill: a context\-aware sparse attention mechanism for efficient long\-sequence inference\.arXiv preprint arXiv:2502\.20766\.Cited by:[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px2.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)Snapkv: llm knows what you are looking for before generation\.Advances in Neural Information Processing Systems37,pp\. 22947–22970\.Cited by:[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px3.p1.1)\.
- H\. Liu, M\. Zaharia, and P\. Abbeel \(2023\)Ring attention with blockwise transformers for near\-infinite context\.arXiv preprint arXiv:2310\.01889\.Cited by:[§1](https://arxiv.org/html/2607.20457#S1.p2.1),[§2](https://arxiv.org/html/2607.20457#S2.p2.1),[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px1.p1.1)\.
- M\. Milakov and N\. Gimelshein \(2018\)Online normalizer calculation for softmax\.arXiv preprint arXiv:1805\.02867\.Cited by:[Appendix G](https://arxiv.org/html/2607.20457#A7.p1.3),[1\(b\)](https://arxiv.org/html/2607.20457#S3.F1.sf2),[1\(b\)](https://arxiv.org/html/2607.20457#S3.F1.sf2.5.2),[§3\.2](https://arxiv.org/html/2607.20457#S3.SS2.p1.1)\.
- P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. Liang \(2016\)Squad: 100,000\+ questions for machine comprehension of text\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 2383–2392\.Cited by:[Table 7](https://arxiv.org/html/2607.20457#A1.T7.10.13.9.3.1.1)\.
- S\. Robertson and H\. Zaragoza \(2009\)The probabilistic relevance framework: bm25 and beyond\.Vol\.4,Now Publishers Inc\.Cited by:[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px3.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2023\)Efficient streaming language models with attention sinks\.arXiv preprint arXiv:2309\.17453\.Cited by:[§3\.1](https://arxiv.org/html/2607.20457#S3.SS1.p2.4),[§4\.1](https://arxiv.org/html/2607.20457#S4.SS1.p4.1),[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px2.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[Table 7](https://arxiv.org/html/2607.20457#A1.T7.10.14.10.2.1.1)\.
- M\. Zaheer, G\. Guruganesh, K\. A\. Dubey, J\. Ainslie, C\. Alberti, S\. Ontanon, P\. Pham, A\. Ravula, Q\. Wang, L\. Yang, and A\. Ahmed \(2020\)Big Bird: transformers for longer sequences\.InAdvances in Neural Information Processing Systems \(NeurIPS 2020\),Vol\.33,pp\. 17283–17297\.Cited by:[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px2.p1.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett,et al\.\(2023\)H2o: heavy\-hitter oracle for efficient generative inference of large language models\.Advances in Neural Information Processing Systems36,pp\. 34661–34710\.Cited by:[§6](https://arxiv.org/html/2607.20457#S6.SS0.SSS0.Px3.p1.1)\.

## Appendix ADataset Details

### A\.1BABILong

BABILong\(Kuratovet al\.,[2024](https://arxiv.org/html/2607.20457#bib.bib19)\)extends the bAbI reasoning suite by embedding short synthetic reasoning chains inside long distractor text drawn from PG\-19 books\. The reasoning chains are generated from symbolic rules; the model must locate the relevant sentences among thousands of distractor tokens and apply multi\-step logic\. We evaluate on two tasks; Table[6](https://arxiv.org/html/2607.20457#A1.T6)summarises their requirements\.

Table 6:BABILong task descriptions\.Context splits coverL∈\{16​K,32​K,64​K,128​K\}L\\in\\\{16\\text\{K\},\\,32\\text\{K\},\\,64\\text\{K\},\\,128\\text\{K\}\\\}tokens; each split contains 1,000 samples\. We report results forqa1,qa3, andqa5as representative of increasing reasoning depth\.

### A\.2RULER

RULER\(Hsiehet al\.,[2024](https://arxiv.org/html/2607.20457#bib.bib18)\)is a synthetic benchmark designed to test long\-context recall under controlled conditions\. We use all 13 tasks across four context lengths \(L∈\{16​K,32​K,64​K,128​K\}L\\in\\\{16\\text\{K\},\\,32\\text\{K\},\\,64\\text\{K\},\\,128\\text\{K\}\\\}\), 500 samples per task per length\.

Table 7:RULER task descriptions\. Abbreviations:kk= number of unique needle keys;vv= values per key;rr= number of queries\.CategoryTaskDescriptionNIAHSingle\-1Single word key→\\tosingle number value hidden in synthetic noise text\.Single\-2Single word key→\\tosingle number value hidden in Paul Graham essay excerpts\.Single\-3Single word key→\\toUUID value hidden in book text\. UUID tokens are maximally rare \(high IDF\)\.MultiKey\-14 distinct word keys all map to the same number; the model must retrieve the value from any key\.MultiKey\-2An entire line of word keys maps to one value; model must match the full key phrase\.MultiKey\-3UUID keys→\\toUUID values; both key and value are rare tokens\.MultiValue1 key maps to 4 different numbers; the model must recall all of them\.MultiQuery4 queries each asking for the value of a different key from the same set of needles\.TrackingVariableA chain of variable assignments is hidden in noise \(e\.g\.,x=3; y=x; z=y\)\. The model finds all variables assigned a target value\.WordsCWEA numbered list of words is presented; some appear more often\. The model identifies the 10 most common words\.FWEA “coded” text with high\-frequency synthetic words\. The model identifies the 3 most frequent codes\.QAQA1Reading comprehension over SQuAD\(Rajpurkaret al\.,[2016](https://arxiv.org/html/2607.20457#bib.bib20)\)passages concatenated into a long context\.QA2Multi\-hop reading comprehension over HotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2607.20457#bib.bib21)\)documents; requires reasoning across two documents\.

## Appendix BChat Template and Prompt Format

All experiments use Meta\-Llama\-3\.1\-8B\-Instruct with the standard Llama 3 instruct chat template\. The full template skeleton is:

```
<|begin_of_text|>
<|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>

{task_instruction}

{in_context_examples}

{post_prompt}

<context>
{long_context}
</context>

Question: {query}<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
```

#### Context/query split\.

Pulsar Attention \(and Star Attention\) separates the input into two distinct strings before tokenisation:

- •prompt\_context: everything from<\|begin\_of\_text\|\>through the closing</context\>tag, inclusive\. This string is tokenised, split into blocks of sizeBB, and processed by Phase 1\.
- •prompt\_query: theQuestion: \{query\}fragment followed by the assistant header tokens\. This string is tokenised separately and processed by Phase 2\.

#### Stop words\.

Generation halts at any of:<\|end\_of\_text\|\>,<\|eom\_id\|\>,<\|eot\_id\|\>, or aftermax\_new\_tokens\(128 for BABILong; task\-dependent for RULER, see Table[9](https://arxiv.org/html/2607.20457#A4.T9)\)\.

#### Tokenisation\.

Tokens are produced withadd\_special\_tokens=False; the<\|begin\_of\_text\|\>BOS token is included as part of the template string itself and must not be added a second time\.

## Appendix CScoring Heuristic Details

All heuristics operate on non\-overlapping contiguous chunks ofchunk\_sizetokens carved from each block\. Chunks preserve local word order; the top\-num\_chunkschunks by score are selected and returned*in positional order*to maintain natural reading order for the Transformer\.

### C\.1Corpus\-Level IDF Table

Before any scoring, a single pass over all blocks constructs a document\-frequency table\. LetNNbe the total number of blocks anddf​\(t\)\\mathrm\{df\}\(t\)the number of blocks containing tokentt:

IDF​\(t\)=log⁡\(Nmax⁡\(df​\(t\),1\)\)\.\\mathrm\{IDF\}\(t\)=\\log\\\!\\left\(\\frac\{N\}\{\\max\(\\mathrm\{df\}\(t\),\\,1\)\}\\right\)\.\(8\)This is stored as a dense integer\-indexed tensoridf\_tableof shape\(vocab\_size,\)\(\\texttt\{vocab\\\_size\},\)\. All four IDF\-based heuristics perform a single gather from this table; no Python loops over the vocabulary are required\.

### C\.2TF\-IDF

scoreTF\-IDF​\(C\)=1\|C\|​∑p∈CIDF​\(tp\)\.\\text\{score\}\_\{\\text\{TF\-IDF\}\}\(C\)=\\frac\{1\}\{\|C\|\}\\sum\_\{p\\in C\}\\mathrm\{IDF\}\(t\_\{p\}\)\.\(9\)This equals the classical mean TF\-IDF over types, and permits a single vectorized lookup:idf\_table\[chunk\_tokens\]\.sum\(\) / len\(chunk\)\.

Strengths\.Balanced, general\-purpose; promotes chunks with rare but recurrent vocabulary\.Weakness\.A single rare entity among many common tokens is diluted by the average\.

### C\.3BM25

scoreBM25​\(C\)=∑t∈types​\(C\)IDFBM25​\(t\)⋅tf​\(t\)​\(k1\+1\)tf​\(t\)\+k1​\(1−b\+b​\|C\|l¯\),\\text\{score\}\_\{\\text\{BM25\}\}\(C\)=\\sum\_\{t\\in\\text\{types\}\(C\)\}\\mathrm\{IDF\}\_\{\\text\{BM25\}\}\(t\)\\cdot\\frac\{\\mathrm\{tf\}\(t\)\\,\(k\_\{1\}\+1\)\}\{\\mathrm\{tf\}\(t\)\+k\_\{1\}\\\!\\left\(1\-b\+b\\,\\frac\{\|C\|\}\{\\overline\{l\}\}\\right\)\},\(10\)wherek1=1\.2k\_\{1\}\\\!=\\\!1\.2,b=0\.75b\\\!=\\\!0\.75,l¯\\overline\{l\}is the mean chunk length across the corpus, andIDFBM25​\(t\)=log⁡\(N−df​\(t\)\+0\.5df​\(t\)\+0\.5\+1\)\\mathrm\{IDF\}\_\{\\text\{BM25\}\}\(t\)=\\log\\\!\\left\(\\frac\{N\-\\mathrm\{df\}\(t\)\+0\.5\}\{\\mathrm\{df\}\(t\)\+0\.5\}\+1\\right\)\.

Strengths\.More robust than TF\-IDF when chunk lengths vary\.Weakness\.Slightly higher compute; benefits diminish when all chunks are the same length\.

### C\.4Entropy \(Type\-Token Ratio\)

scoreEntropy​\(C\)=\|\{t:t∈C\}\|\|C\|\.\\text\{score\}\_\{\\text\{Entropy\}\}\(C\)=\\frac\{\\bigl\|\\\{t:t\\in C\\\}\\bigr\|\}\{\|C\|\}\.\(11\)Implemented as\(bincount\(C\) \> 0\)\.sum\(\) / len\(C\)\.

Strengths\.Corpus\-independent; no IDF table required\. A chunk of padding tokens or repeated stop\-words scores near 0; an entity\-rich chunk scores near 1\.Weakness\.Ignores global rarity; a chunk with 32 distinct common stop\-words scores identically to one with 32 rare entities\.

### C\.5Max\-IDF

scoreMax\-IDF​\(C\)=maxp∈C⁡IDF​\(tp\)\.\\text\{score\}\_\{\\text\{Max\-IDF\}\}\(C\)=\\max\_\{p\\in C\}\\,\\mathrm\{IDF\}\(t\_\{p\}\)\.\(12\)Implemented asidf\_table\[chunk\_tokens\]\.max\(\)\.

Strengths\.Acts as a pure needle detector: if any token in the chunk appears in only one block, the chunk is guaranteed to be selected\. This is exactly what multi\-hop tasks and NIAH require\.Weakness\.Ignores overall chunk quality; one rare token among 31 uninformative tokens still wins\.

### C\.6Even\-Spaced \(Baseline\)

Selects exactlynum\_chunks×chunk\_size\\texttt\{num\\\_chunks\}\\times\\texttt\{chunk\\\_size\}tokens at uniform intervals across the block usingtorch\.linspace:

indices=⌊linspace​\(0,\|B\|−1,s\)⌋,s=num\_chunks×chunk\_size\.\\text\{indices\}=\\left\\lfloor\\texttt\{linspace\}\(0,\\,\|B\|\-1,\\,s\)\\right\\rfloor,\\quad s=\\texttt\{num\\\_chunks\}\\times\\texttt\{chunk\\\_size\}\.\(13\)Original \(sparse\) position IDs are preserved\.

Strengths\.No corpus statistics required; deterministic\.Weakness\.Wastes budget on low\-information spans as budget grows \(confirmed in ablations:qa3degrades from 36 to 25 asssincreases from 128 to 1024\)\.

### C\.7Heuristic Summary

Table 8:Summary of all scoring heuristics\.

## Appendix DHyperparameter Configuration

Table[9](https://arxiv.org/html/2607.20457#A4.T9)lists all hyperparameters used in our main experiments\. All values are fixed across RULER and BABILong unless noted\.

Table 9:Full hyperparameter configuration\.
## Appendix ERoPE Position ID Assignment

When a block’s Phase 1 input is assembled as\[SINK∣S1,…,Si−1∣Bi\]\[\\text\{SINK\}\\mid S\_\{1\},\\ldots,S\_\{i\-1\}\\mid B\_\{i\}\], the position IDs of the inserted sink and summary tokens can be assigned in two ways\.

#### Sparse \(default\)\.

Every token retains its*original global position*in the full context\. Summary tokens from blockjjkeep the position IDs of their source locations\. The assembled sequence therefore contains*gaps*where non\-selected tokens were dropped\. This preserves the true inter\-token distances that RoPE was trained to encode: a summary token from block 0 is still seen as far from a token in block 3, matching the model’s pre\-training geometry\.

#### Contiguous\.

The assembled sequence is re\-numbered0,1,…,Li−10,1,\\ldots,L\_\{i\}\-1for each blockii’s Phase 1 forward pass \(original Star Attention style\)\. In this mode, Phase 2 query position IDs are offset to start just past the longest assembled Phase 1 length, ensuring RoPE deltas from the query to every block’s KV remain small and in\-distribution\.

Contiguous mode can occasionally help by ensuring that summary tokens and local block tokens are seen at short relative distances, but it distorts the global geometry\. All experiments reported in this paper usesparsemode, which aligns with the model’s pre\-training distribution\.

## Appendix FKV Cache Discard

After each Phase 1 forward pass, the model’s KV cache contains entries for*all*tokens in the assembled input\[SINK∣S1,…,Si−1∣Bi\]\[\\text\{SINK\}\\mid S\_\{1\},\\ldots,S\_\{i\-1\}\\mid B\_\{i\}\]\. The sink and summary KV states are discarded immediately:

Ki\(layer\)←Ki\(layer\)​\[:,:,is\_bi\]\(and same for​Vi​\)K\_\{i\}^\{\\text\{\(layer\)\}\}\\;\\leftarrow\\;K\_\{i\}^\{\\text\{\(layer\)\}\}\[\\,:\\,,\\,:\\,,\\,\\texttt\{is\\\_bi\}\\,\]\\quad\\text\{\(and same for \}V\_\{i\}\\text\{\)\}\(14\)whereis\_bi\[p\]=True\[p\]=\\text\{True\}iffp≥summary\_lenp\\geq\\texttt\{summary\\\_len\}\. Implemented via a boolean mask:

```
is_bi = torch.zeros(total_len, dtype=torch.bool)
is_bi[-block_size:] = True
kv = [[x[0][:,:,is_bi], x[1][:,:,is_bi]] for x in kv]
```

Discarding serves two purposes:\(1\)*No duplicate tokens in Phase 2*: summary tokenSjS\_\{j\}was prepended to blocksj\+1,…,B−1j\{\+\}1,\\ldots,B\{\-\}1; if retained, each KV cache would contain a copy ofSjS\_\{j\}’s key/value vectors, corrupting the softmax normalisation during Phase 2’sall\_gather\.\(2\)*Exact final cache size*: after discarding, each GPU holds exactlyblock\_sizeKV entries, so the total KV cache acrossBBGPUs equals the original context lengthLL\.

The discard can be disabled via\-\-no\_discard\_summary\_kvfor experimental purposes; in that case, sink and summary KV states bake into the final cache and Phase 2 attends over them directly \(though with duplicate tokens\)\.

## Appendix GPhase 2 Distributed Softmax Merge

Phase 2 runs an identical forward pass on allBBGPUs simultaneously\. Each GPU computes a local attention output𝐨h\\mathbf\{o\}\_\{h\}and log\-sum\-expℓh\\ell\_\{h\}over its own KV block\. Afterdist\.all\_gather, the outputs are merged with the numerically stable online\-softmax formula\(Milakov and Gimelshein,[2018](https://arxiv.org/html/2607.20457#bib.bib17)\):

ℓ′\\displaystyle\\ell^\{\\prime\}=ℓ\+log⁡\(1\+eℓh−ℓ\),\\displaystyle=\\ell\+\\log\\\!\\left\(1\+e^\{\\ell\_\{h\}\-\\ell\}\\right\),\(15\)𝐨′\\displaystyle\\mathbf\{o\}^\{\\prime\}=eℓ−ℓ′​𝐨\+eℓh−ℓ′​𝐨h,\\displaystyle=e^\{\\ell\-\\ell^\{\\prime\}\}\\,\\mathbf\{o\}\+e^\{\\ell\_\{h\}\-\\ell^\{\\prime\}\}\\,\\mathbf\{o\}\_\{h\},\(16\)applied iteratively over all gathered outputs\. Equations \([15](https://arxiv.org/html/2607.20457#A7.E15)\)–\([16](https://arxiv.org/html/2607.20457#A7.E16)\) are implemented instar\_flash\_attn/utils\.pyas atorch\.jit\.scriptkernel for efficiency\. The result is mathematically equivalent to running full attention over the concatenated KV cache of allBBGPUs, at the cost of oneall\_gathercommunication round per generation step\.