ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference

# ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
Source: [https://arxiv.org/html/2608.02947](https://arxiv.org/html/2608.02947)
###### Abstract

The attention score with rotary position embeddings \(RoPE\) decomposes exactly into a sum over its 2D\-rotation frequency pairs, and each pair’s wavelength limits how far it can discriminate position\. Aligned with this structure, we propose the per\-RoPE\-wavelength distance window: it prunes the query–key inner\-product terms beyond a wavelength\-proportional distance\. Unlike a sliding window, every key remains reachable, at least through the low\-frequency pairs\. The reduction rate is input\-independent, with a closed form logarithmic in the sequence lengthNN, in contrast to dynamic\-sparse methods like MInference\. Such token\-level selection is orthogonal to our frequency\-level pruning\. The window can therefore be applied on top of those methods\. On Qwen2\.5\-0\.5B and Llama\-3\.2\-3B, the window prunes 37–48% of the query–key inner\-product terms within each model’s native context length\. Relative to full attention, the top\-1 match rate stays at 96–98% and the mean output\-distribution KL at the10−310^\{\-3\}\-nat level on LongBench\-v2 contexts\. We examine absolute scores on long\-context benchmarks such as RULER, OpenAI\-MRCR, LongCodeQA, and∞\\inftyBench: they are broadly preserved\. We implement the window as a slice of the query–key contraction axis, leaving the online\-softmax recurrences untouched, and port it with minimal diffs into the released FlashAttention\-4 prefill and FlashInfer decode\. On RTX PRO 6000 with Llama, both ports outpace stock with gains growing with context length, up to1\.29×1\.29\\timesat 128K\. End to end on Qwen2\.5\-7B\-1M, with 57% of the inner\-product terms pruned, the speedup reaches1\.31×1\.31\\timesat a 1M\-token context\.

## 1Introduction

Inference for large language models \(LLMs\) is increasingly limited by the compute of attention and by the memory bandwidth of the KV cache—the intermediate state kept for past tokens—as contexts grow longer and multi\-agent deployments spread\. FlashAttention\(Dao[2024](https://arxiv.org/html/2608.02947#bib.bib2)\)\(hereafter FA\)\-class kernels are the fastest implementations that carry out this computationexactly, but being exact, they do not reduce the amount of computation itself\. Many approximation methods that cut computation suffer from one of the following: \(a\) the amount of reduction is input\-dependent and cannot be predicted in advance; \(b\) they discard tokens or KV, so the dropped information becomes structurally unreachable; or \(c\) they are tightly coupled to a particular kernel implementation and cannot be carried to other implementations\.

Our starting point is that the attention score with rotary position embedding \(RoPE\)\(Suet al\.[2024](https://arxiv.org/html/2608.02947#bib.bib1)\)decomposesexactlyinto a sum of per\-frequency\-pair contributions \(Section[3](https://arxiv.org/html/2608.02947#S3)\)\. Each pair has its own wavelengthλr\\lambda\_\{r\}; high\-frequency pairs complete one period in a few tokens, and low\-frequency pairs in millions of tokens\. How far each component can carry positional information is determined by model constants \(the RoPE base and dimension\), independent of the input\. Aligned with this structure, imposing a wavelength\-proportional distance windowwr=min⁡\(k​λr,N\)w\_\{r\}=\\min\(k\\lambda\_\{r\},N\)on each frequency pair to prune the terms of the score sum is theper\-RoPE\-wavelength distance window—the core of our proposed method, ATFlash \(Figure[1](https://arxiv.org/html/2608.02947#S1.F1)\)\. Unlike conventional windows that drop tokens, every token continues to be referenced through the low\-frequency pairs wheneverN<k​λmaxN<k\\lambda\_\{\\max\}\.

![Refer to caption](https://arxiv.org/html/2608.02947v1/figs/fig_swa_perfreq_dense_v12.png)Figure 1:Contrast of a sliding window, the per\-RoPE\-wavelength distance window, and full attention \(conceptual diagram, causal attention\)\. \(a\): out\-of\-window tokens disappear \(gray\)\. \(b\): the same row is split vertically by frequency pairrrand truncated by the wavelength\-proportional window\. CASE: actual count of inner\-product terms for the last query row \(Qwen2\.5\-0\.5B,k=2k\{=\}2,N=32768N\{=\}32768\)\.We make four contributions\.

- •Method: We propose a per\-RoPE\-wavelength distance window aligned with the exact pair decomposition of the RoPE score\. The window can be implemented as a slice of the reduction axis of online softmax, and because it discards no token it can be inserted while preserving the skeleton and numerical path of FA\-class kernels\. It has a single free parameter, the number of retained periodskk, and requires neither training nor hand\-tuning \(Section[3](https://arxiv.org/html/2608.02947#S3)\)\.
- •Theory: We derive the reduction rate in closed form\. The reduction is logarithmic inNN, and forN≥k​λmaxN\\geq k\\lambda\_\{\\max\}it asymptotes toO​\(1\)O\(1\)terms per query andO​\(N\)O\(N\)overall\. Measured reduction matches the closed form at every point \(within the 1–2\-point integral\-approximation error\), so reduction is predictable before running \(Section[4](https://arxiv.org/html/2608.02947#S4)\)\.
- •Implementation and portability: We port the window with minimal diffs into the released FlashAttention\-4 \(hereafter FA4\) prefill and the FlashInfer decode, and on RTX PRO 6000 \(sm\_120, GDDR7\) each surpasses its stock implementation at all evaluated Llama context lengths while remaining bit\-identical with the window off\. Recovery of the theoretical ceiling on Llama lines up at 85–96% across the two ports, confirming that the loss in converting theoretical reduction into real time is not implementation\-specific \(Sections[5](https://arxiv.org/html/2608.02947#S5)–[7](https://arxiv.org/html/2608.02947#S7)\)\.
- •Two\-layer robustness verification: We show that the method maintains 37–48% reduction in the non\-extrapolation regime, a Kullback–Leibler \(KL\) divergence on the order of10−310^\{\-3\}, and 100% recall on in\-text needle retrieval, with a two\-order\-of\-magnitude gap over a sliding window at equal compute, at two layers—distribution and capability \(Sections[6\.1](https://arxiv.org/html/2608.02947#S6.SS1)–[6\.2](https://arxiv.org/html/2608.02947#S6.SS2)\)\. Benchmark scores are broadly preserved; the exception is retrieval over structureless strings, wherekkacts as a dial—at uniformk=64k\{=\}64, still pruning 29% of the terms, the worst case returns to the full\-attention score\.

## 2Related Work

Exact FA\-class kernels\.FA computes exact attention without materializing the score matrix, using tile\-wise online softmax\(Milakov and Gimelshein[2018](https://arxiv.org/html/2608.02947#bib.bib5)\)\(sequential updates of a running max and a normalizer\); from FlashAttention\-2\(Dao[2024](https://arxiv.org/html/2608.02947#bib.bib2)\)onward the family has specialized by hardware generation \(Hopper in FA3\(Shahet al\.[2024](https://arxiv.org/html/2608.02947#bib.bib3)\), Blackwell data\-center in FA4\(Zadouriet al\.[2026](https://arxiv.org/html/2608.02947#bib.bib4)\)\)\. This family deepens hardware fit while retaining all inner\-product terms, and is orthogonal to our method, which selectively drops terms—we demonstrate in Section[5](https://arxiv.org/html/2608.02947#S5)that the two can be combined\.

The frequency structure of RoPE\.RoPE\(Suet al\.[2024](https://arxiv.org/html/2608.02947#bib.bib1)\)applies a rotation with a geometrically spaced angular velocityθr\\theta\_\{r\}for each dimension pair\. The structure of the wavelengthλr=2​π/θr\\lambda\_\{r\}=2\\pi/\\theta\_\{r\}—high frequency for short distance, low frequency for long distance—and the division of roles—high frequency for position, low frequency for meaning—are widely observed\(Barberoet al\.[2025](https://arxiv.org/html/2608.02947#bib.bib6)\)\. On the other hand, it is also known that whenq,kq,kare isotropic the expectation of the RoPE contribution does not depend on distance\(Barberoet al\.[2025](https://arxiv.org/html/2608.02947#bib.bib6)\), so one cannot justify a window by assuming adistance decayof the score\. What our method relies on is not decay but thewavelength\(Section[3\.1](https://arxiv.org/html/2608.02947#S3.SS1)\), which is fundamentally different from conventional windows based on a decay assumption\.

Sparse and windowed attention\.The sliding window\(Beltagyet al\.[2020](https://arxiv.org/html/2608.02947#bib.bib15)\), attention sinks with a recent window\(Xiaoet al\.[2024](https://arxiv.org/html/2608.02947#bib.bib16)\), and KV eviction by cumulative scores\(Zhang and others[2023](https://arxiv.org/html/2608.02947#bib.bib17)\)reduce computation, but all impose the same window or selection on every frequency, so the dropped tokens become structurally unreachable\.

Input\-dependent dynamic sparsification\.A training\-free line is active here: MInference\(Jianget al\.[2024](https://arxiv.org/html/2608.02947#bib.bib9)\)is the standard baseline with offline search over vertical\-slash patterns, FlexPrefill\(Laiet al\.[2025](https://arxiv.org/html/2608.02947#bib.bib10)\)and XAttention\(Xuet al\.[2025](https://arxiv.org/html/2608.02947#bib.bib11)\)make the selection online and lighter, and SpargeAttention\(Zhanget al\.[2025](https://arxiv.org/html/2608.02947#bib.bib12)\)combines block sparsity with quantization\. Approaches that use the RoPE frequency structure as a cue have also appeared: FASA\(Wanget al\.[2026](https://arxiv.org/html/2608.02947#bib.bib8)\)predicts token importance from dominant frequency components and dynamically selects KV entries at decode, and TriAttention\(Maoet al\.[2026](https://arxiv.org/html/2608.02947#bib.bib13)\)compresses KV with a trigonometric distance preference; both use frequencies only as aselector, and the surviving tokens are computed at all frequencies\. In all of these the selection is input\-dependent, so the reduction is not fixed before execution\.

Training\-based methods\.DSA\(DeepSeek\-AI[2025](https://arxiv.org/html/2608.02947#bib.bib7)\)builds a learned selector into pretraining, and SFA\(Xieet al\.[2026](https://arxiv.org/html/2608.02947#bib.bib14)\)prunes inner\-product terms with learned codes\. Their units of reduction differ, but both presuppose training the model itself and cannot be retrofitted\.

Our method carries none of the constraints \(a\)–\(c\) of Section[1](https://arxiv.org/html/2608.02947#S1): the window is determined in closed form from the wavelengths and hence input\-independent, and it is training\-free and retrofittable onto the standard attention layers of any pretrained RoPE model\. Like SFA, our method prunes inner\-product terms, but it derives the window analytically from the wavelengths and learns no codes\.

Customizable attention frameworks\.The customization axes of FlexAttention\(Donget al\.[2024](https://arxiv.org/html/2608.02947#bib.bib18)\)and FlashInfer\(Yeet al\.[2025](https://arxiv.org/html/2608.02947#bib.bib19)\)run along the token \(row/column\) direction of the score matrix, whereas our pruning operatesinsideeach inner product—along the frequency\-pair axis of the head dimension—which neither programming model can currently express\. We realize it by minimal\-diff insertion into the released FA4 and FlashInfer\.

## 3Method: Per\-RoPE\-Wavelength Windows

### 3\.1Pair Decomposition of the RoPE Score

For a queryqq\(at positionnn\) and a keykk\(at positionmm\) with head dimensiondd, RoPE applies a two\-dimensional rotation with angular velocityθr=B−2​r/d\\theta\_\{r\}=B^\{\-2r/d\}\(BBis the base\) to each dimension pairr∈\{0,…,d/2−1\}r\\in\\\{0,\\dots,d/2\-1\\\}\. Writing the rotated query and key asq~,k~\\tilde\{q\},\\tilde\{k\}, the score depends only on the relative positionz=n−mz=n\-mand can be writtenexactlyas a sum of per\-pair contributions,

q~⊤​k~d=1d​∑rΔ​xm,r,Δ​xm,r=ar​cos⁡\(z​θr\)\+br​sin⁡\(z​θr\)\\frac\{\\tilde\{q\}^\{\\top\}\\tilde\{k\}\}\{\\sqrt\{d\}\}=\\\!\\frac\{1\}\{\\sqrt\{d\}\}\\\!\\sum\_\{r\}\\Delta x\_\{m,r\},\\;\\Delta x\_\{m,r\}=a\_\{r\}\\cos\(z\\theta\_\{r\}\)\+b\_\{r\}\\sin\(z\\theta\_\{r\}\)\(1\)wherear,bra\_\{r\},b\_\{r\}are coefficients determined by the components ofqqandkk\. Thewavelengthof each pair,

λr=2​π/θr\\lambda\_\{r\}=2\\pi/\\theta\_\{r\}\(2\)is the number of tokens over whichcos,sin\\cos,\\sincomplete one period along the distancezz, and beyond it a single pair can no longer discriminate position \(aliasing\)\. That is, high\-frequency \(smallλr\\lambda\_\{r\}\) pairs carry positional information only nearby, while low\-frequency \(largeλr\\lambda\_\{r\}\) pairs remain informative out to long distances \(Figure[2](https://arxiv.org/html/2608.02947#S3.F2)\)\. The truncation of our method therefore assumes no score decay; it aligns with the upper limit of the distance over which each pair can discriminate position\. We cast this as a hypothesis, that a pair’s term becomes redundant at distances where the pair cannot discriminate position, and verify it on output distributions and capabilities from Section[6\.1](https://arxiv.org/html/2608.02947#S6.SS1)onward\.

![Refer to caption](https://arxiv.org/html/2608.02947v1/figs/rwise_range_log_v22.png)Figure 2:The wavelengthλr=2​π/θr\\lambda\_\{r\}=2\\pi/\\theta\_\{r\}of each pair \(log axis\)\. Qwen2\.5\-0\.5B, Qwen2\.5\-7B\-1M, Llama\-3\.2\-3B \(effectiveλr′\\lambda^\{\\prime\}\_\{r\}after llama3\-style scaling\), and gpt\-oss\-20b \(effectiveλr′\\lambda^\{\\prime\}\_\{r\}under YaRN×\\times32\) are shown together; horizontal lines mark each model’s training length \(dashed\) and the5×5\\timesextrapolation guide \(solid\)\.
### 3\.2Window Rule

We therefore vary the distance window per frequency:

wr=min⁡\(k​λr,N\)w\_\{r\}=\\min\(k\\,\\lambda\_\{r\},\\ N\)\(3\)and at cell\(n,m\)\(n,m\)we keep in the score sum \([1](https://arxiv.org/html/2608.02947#S3.E1)\) only the pairs satisfyingz≤wrz\\leq w\_\{r\}\. The number of retained periodskkis theonly free parameterof this rule, the wavelengthsλr\\lambda\_\{r\}being model constants andNNthe input length; because the dependence onkkis only logarithmic \(Section[4](https://arxiv.org/html/2608.02947#S4)\), we default tok=2k\{=\}2\. The parameter thus acts as a dial between attention accuracy and compute that can be set at deployment time, much as weight precision is chosen per deployment, and its cost is known in closed form before running\. High frequencies get short windows and low frequencies get long windows, producing a staircase\-like profile in which farther cells have fewer active pairs \(fewer inner\-product terms\)—for Qwen2\.5\-0\.5B atN=32768N\{=\}32768,k=2k\{=\}2, from 32 terms atz=0z\{=\}0down to 13 at the farthest distance shown \(Figure[1](https://arxiv.org/html/2608.02947#S1.F1), CASE\)\. Crucially, distant tokens continue to be referenced through the low\-frequency pairs; our methoddrops terms of the score sum per frequency rather than dropping tokens\. Full token retention holds in the rangeN<k​λmaxN<k\\lambda\_\{\\max\}, where the lowest\-frequency window covers the sequence\. A sliding window corresponds to the special case of imposing the highest\-frequency window on all pairs \(Figure[1](https://arxiv.org/html/2608.02947#S1.F1)\(a\)\(b\)\)\.

The window table \(the list of window widths for each pair\) is generated from the effective frequencies in the model config\. Frequency transforms such as YaRN or llama3\-style rope\_scaling are reflected in the window rule through the effectiveθr′\\theta^\{\\prime\}\_\{r\}\(and henceλr′\\lambda^\{\\prime\}\_\{r\}\), with no per\-scaling\-scheme hand\-tuning \(Figure[2](https://arxiv.org/html/2608.02947#S3.F2)\)\.

### 3\.3Implementation as a Slice of the Reduction Axis of Online Softmax

Because the window \([3](https://arxiv.org/html/2608.02947#S3.E3)\) does not depend on the input and is fixed in advance, the set of active pairs per tile is determined statically\. Out\-of\-window pruning is implemented not as a multiplicative mask but as a shortening \(slice\) of the reduction length of the inner product\. As the distancezzgrows, pairs fall out of the window from the high\-frequency side, so the active\-pair set is a contiguous slice on the low\-frequency side; we round its boundary to a multiple of the K\-direction atom \(16 elements\) of the matrix multiply\-accumulate \(MMA\) instruction\. As a result, the tiling, running max, and normalizer of the FA\-class kernel are leftentirely unchanged, and only the effective inner\-product length of each tile shrinks statically \(tokens andVVare loaded in full; nothing is discarded\)\. The low\-frequency terms that remain even at the farthest distance fit within a single MMA tile \(Figure[1](https://arxiv.org/html/2608.02947#S1.F1), CASE\), preserving Tensor Core fill density\.

As in Figure[3](https://arxiv.org/html/2608.02947#S3.F3), the window insertion merely advances the start of the pair loop to the window boundary\. More important is the memory side \(right of the figure\): out\-of\-window pair components not only skip computation but arenot loaded into SRAM at allfor the relevant parts ofK⊤K^\{\\top\}andqq\(gray in the figure\)\. Thus the window cuts the QK compute and the transfer ofK⊤K^\{\\top\}andqqat the same ratio \(the transfer ofVVis not reduced\)\. This “slice of the reduction axis” form underlies the cross\-implementation portability of Section[5](https://arxiv.org/html/2608.02947#S5)\.

![Refer to caption](https://arxiv.org/html/2608.02947v1/figs/fa_decode_i_only_v31.png)Figure 3:FA’s online\-softmax loop and where the per\-RoPE\-wavelength distance window is inserted \(a full view including the double loop of prefill; decode corresponds to the last query rown=Nn\{=\}Nonly\)\. Left: FA’s recurrences \(running maxμ\\mu, normalizerℓ′\\ell^\{\\prime\}, outputo′o^\{\\prime\}\) and the causal mask\. Center: the pair decomposition of the scorexmx\_\{m\}and the window predicate \(gray box\)\. Right: the memory\-side behavior \(gray==out\-of\-window components, not loaded for this reduction\)\.

## 4Theory: Closed\-Form QK Reduction Rate

Because the window is fixed in advance, the amount of reduction is deterministic and can be analyzed without running\. We measure the computational cost by thenumber of terms of the query–key inner productin score computation: whereas full attention requiresd/2d/2pairs of multiply\-accumulate per causal cell, with the window only the pairs withz≤wrz\\leq w\_\{r\}are computed\. We define the reduction rate as “1−1\-\(number of terms retained inside the window\)//\(number of causal cells×d/2\\times\\,d/2\)”\. This definition is a term\-count proxy that excludes softmax, theVVproduct, and memory transfer; wall\-clock time is measured in Section[7](https://arxiv.org/html/2608.02947#S7)\.

Approximating the pair index by a continuous variableρ\\rho, the window width becomes the geometric progressionw​\(ρ\)=wmin​B2​ρ/dw\(\\rho\)=w\_\{\\min\}B^\{2\\rho/d\}\(wmin=2​π​kw\_\{\\min\}=2\\pi k\), and the number of retained terms over the interval\[n0,n1\]\[n\_\{0\},n\_\{1\}\]can be written as the double integral

S​\(n0,n1\)=∫n0n1∫0d/2min⁡\(w​\(ρ\),n\)​𝑑ρ​𝑑nS\(n\_\{0\},n\_\{1\}\)=\\int\_\{n\_\{0\}\}^\{n\_\{1\}\}\\\!\\\!\\int\_\{0\}^\{d/2\}\\min\\bigl\(w\(\\rho\),n\\bigr\)\\,d\\rho\\,dn\(4\)which solves in closed form by splitting the integral at the saturation boundaryρ∗​\(n\)=d2​ln⁡\(n/wmin\)/ln⁡B\\rho^\{\*\}\(n\)=\\tfrac\{d\}\{2\}\\,\\ln\(n/w\_\{\\min\}\)/\\ln B\(wherew​\(ρ∗\)=nw\(\\rho^\{\*\}\)\{=\}n\)\. For the whole prefill \(\[0,N\]\[0,N\],N≫wminN\\gg w\_\{\\min\}\),

reductionprefill≈ln⁡\(N/wmin\)−3/2ln⁡B\\mathrm\{reduction\}\_\{\\mathrm\{prefill\}\}\\approx\\frac\{\\ln\(N/w\_\{\\min\}\)\-3/2\}\{\\ln B\}\(5\)For Qwen2\.5 atN=32N\{=\}32K the substituted value is 46\.1% and the exact discrete sum is 47\.6%; the difference is the integral\-approximation error\. The reduction rate for decode \(the last row\) has the same form, and its difference from the whole prefill is only the constant1/\(2​ln⁡B\)1/\(2\\ln B\)\(3\.6 points forB=106B\{=\}10^\{6\}\)\.

Three structural facts can be read from this closed form\. \(i\) The reduction rate islogarithmicinNN\(slope1/ln⁡B1/\\ln B\) and deepens for longer contexts\. \(ii\) The dependence onkkis also logarithmic; doublingkkcosts onlyln⁡2/ln⁡B\\ln 2/\\ln Bof the reduction—so headroom on the quality side can be bought cheaply\. \(iii\) ForN≥k​λmaxN\\geq k\\lambda\_\{\\max\}themin\\minof every window settles on thek​λrk\\lambda\_\{r\}side \(no window is clipped byNN\), the number of retained terms reaches the constant∑rwr\\sum\_\{r\}w\_\{r\}independent ofNN, and it becomesO​\(1\)O\(1\)terms per query andO​\(N\)O\(N\)over the whole sequence\(Qwen2\.5,k=2k\{=\}2: 64% atN=200N\{=\}200K, 76% reduction at 1M\)\. Note thatk​λmaxk\\lambda\_\{\\max\}is orders of magnitude beyond practical context lengths \(Figure[2](https://arxiv.org/html/2608.02947#S3.F2)\); all experiments in this paper lie in the full\-retention regimeN<k​λmaxN<k\\lambda\_\{\\max\}—theO​\(N\)O\(N\)behavior is an asymptotic guarantee\. Figure[4](https://arxiv.org/html/2608.02947#S4.F4)shows the closed\-form curves; Llama\-3\.2\-3B is counted on its effective wavelengths after llama3\-style scaling, and gpt\-oss\-20b on its full\-attention layers only \(a hybrid architecture interleaving sliding\-window and full\-attention layers\)\. The QK\-pruned column of Table[1](https://arxiv.org/html/2608.02947#S6.T1)\(measured counts on realq,kq,k\) matches the closed form within the integral\-approximation error at every point\.

![Refer to caption](https://arxiv.org/html/2608.02947v1/figs/saving_vs_N_v26.png)Figure 4:QK term reduction against sequence lengthNN: the closed form of Eq\. \([5](https://arxiv.org/html/2608.02947#S4.E5)\); agreement with measurement is in Table[1](https://arxiv.org/html/2608.02947#S6.T1)\. Light blue: the closed form for Qwen2\.5 \(B=106B\{=\}10^\{6\}\) under YaRN at the minimal scale perNN\.
## 5Implementation: Window Ports

We implement the method as minimal\-diff ports into the released implementations: FA4 for prefill and FlashInfer for decode\. The FA4 port changes 4 files \(\+296/−13\+296/\-13lines\) in the forward path of the CuTeDSL implementation\. The base is thereleased upstream main\(public commit5835c73\)\. The insertion truncates the QK reduction of the unmasked band to the leading MMA\-K chunks of the effective inner\-product length per distance band and narrows the asynchronous load of K columns to the same width \(a prefix\-only load for the contiguous slice on the low\-frequency side\); it does not touch the masked band, softmax, the PV product, or the online accumulation\. Because all added branches resolve to compile\-time constants, the window\-off path generates the same trace as upstream—abit\-identical design\.

Because FA4’s decode path drops occupancy at query length 1, the FA4 port targets prefill\. For decode we ported the same window into the decode kernel of FlashInfer 0\.6\.13 \(the released wheel\)\. This kernel partitions the KV sequence across blocks and merges the partial online\-softmax states afterward; the loop inside each split is identical to Figure[3](https://arxiv.org/html/2608.02947#S3.F3)\. The only changes are adding the window condition to the asynchronous K\-load predicate and nullifying the partial QK dot under the same predicate \(about 100 lines in effect\); the window\-off specialization follows the stock code path\. The FlashInfer port likewise resolves at compile time, and we confirm in measurement that both ports are bit\-identical to stock with the window off\.

## 6Evaluation of Output Quality

This section verifies output quality at 2 layers: a relative comparison against full attention \(preservation rates; Section[6\.1](https://arxiv.org/html/2608.02947#S6.SS1)\), and the absolute scores of the benchmarks themselves \(Section[6\.2](https://arxiv.org/html/2608.02947#S6.SS2)\)\. The experiments run on NVIDIA GH200 and H100 GPUs\.

### 6\.1Fidelity: Preserved Output Distributions

Setup\.We embedded the window into PyTorch’s eager attention implementation \(transformers 4\.57\.1; 5\.5\.3 for RULER and gpt\-oss\-20b\) and compared it against full attention \(window off\) on real models and real tasks, in fp32 \(bf16 for gpt\-oss\-20b\)\. The main fidelity metric is the average KL divergence \(nats\) from the full\-attention model’s output distribution, alongside the argmax match rate \(top\-1\)\.

Results\.Table[1](https://arxiv.org/html/2608.02947#S6.T1)shows sequence\-length scaling in the LongBench\-v2 context\. The wavelength\-proportionalk=2k\{=\}2window reduces the number of inner\-product terms by37–48%in thenon\-extrapolation regime\(within each model’s native context; up to 32K for Qwen2\.5\-0\.5B and up to 64K for Llama\) while keeping top\-1 at 96–98% and KL on the order of10−310^\{\-3\}nats\. In the YaRN extrapolation regime the reduction grows to 53\.8% \(Qwen 160K\) and top\-1 declines gently to 88\.1%\. This decline mainly reflects the full\-attention model itself beginning to break down in the extrapolation regime, not a long\-context collapse of the window’s pruning—within the verification region, top\-1 and KL stay nearly flat even as the reduction share grows \(Table[1](https://arxiv.org/html/2608.02947#S6.T1)\)\.

LongBench\-v2NNQK\-prunedtop\-1KL divfulloursQwen2\.5\-0\.5B8K37\.6%97\.4%0\.00200\.360\.3816K42\.6%96\.9%0\.00290\.260\.3032K47\.6%96\.2%0\.00450\.260\.3464K†50\.4%94\.9%0\.01740\.160\.24160K‡53\.8%88\.1%0\.09540\.080\.18Llama\-3\.2\-3B8K38\.5%97\.8%0\.00260\.280\.2216K42\.5%96\.6%0\.00550\.340\.3432K45\.6%96\.4%0\.00560\.240\.2464K48\.1%97\.0%0\.00550\.160\.18Table 1:Sequence\-length scaling \(k=2k\{=\}2, all query positions\)\. QK\-pruned==share of inner\-product terms pruned\. full/ours==accuracy on official LongBench\-v2 4\-choice questions, 50 per band, scored by choice\-letter likelihood with contexts tail\-truncated to eachNN; not comparable to the official generation\-based scoring\.†YaRNs=2s\{=\}2,‡YaRNs=5s\{=\}5\.modelQwen2\.5\-7B\-1M \(d=128d\{=\}128\)Qwen2\.5\-0\.5B \(d=64d\{=\}64\)gpt\-oss\-20bbenchmarkMRCRLongCode∞\\inftyBenchRULERGSM8KsubsetcustomQA10 taskMath\.FindRetr\.KV∗4 taskMK\-NIAH0\-shotcontext66–160K31–245K≤\\leq262K∼\\sim117K∼\\sim169K16K16K∼\\sim1\.2Knn1652331425350350201200300300500kk222\\cellcoloratfgraylt2\\cellcoloratforange16→\\to2\\cellcoloratfgraylt2\\cellcoloratforange16→\\to2\\cellcoloratfpink642\\cellcoloratfgraylt2\\cellcoloratforange16→\\to22QK\-pruned \(ours\)45\.8%46\.3%48\.6%\\cellcoloratfgraylt48\.2%\\cellcoloratforange37\.8%\\cellcoloratfgraylt50\.5%\\cellcoloratforange40\.3%\\cellcoloratfpink29\.0%42\.6%\\cellcoloratfgraylt42\.6%\\cellcoloratforange29\.8%9\.8%full0\.2040\.6220\.3930\.3340\.75020\.696\.70\.908\\rowcoloratfblue ours0\.1810\.6270\.386\\cellcoloratfgraylt0\.334\\cellcoloratforange0\.343\\cellcoloratfgraylt0\.300\\cellcoloratforange0\.350\\cellcoloratfpink0\.75021\.3\\cellcoloratfgraylt93\.7\\cellcoloratforange96\.00\.920MInf0\.2290\.6700\.3890\.3490\.800————\\rowcoloratfblue ours\+\+MInf0\.2180\.640\\cellcolorwhite—\\cellcolorwhite—\\cellcolorwhite—\\cellcoloratfgraylt0\.350\\cellcolorwhite—\\cellcoloratfpink0\.800\\cellcolorwhite—\\cellcolorwhite—\\cellcolorwhite—\\cellcolorwhite—Table 2:Absolute benchmark scores\. Bold==best within each rule\-separated block; the gray/orange cell pairs contrast the fixedk=2k\{=\}2window with the variable\-kkprofile;∗==significant drop for the window\. Conventions and symbols: Section[6\.2](https://arxiv.org/html/2608.02947#S6.SS2)\.
### 6\.2Capability: Absolute Benchmark Scores

Preserved distributions would mean little if capabilities broke\. This section scores the window on benchmarks in absolute terms: if the terms beyond a pair’s wavelength carry noise rather than positional signal \(the hypothesis of Section[3\.1](https://arxiv.org/html/2608.02947#S3.SS1)\), pruning them should not hurt task scores—and we in fact observe scattered cases where scores rise\. We verify this on worst\-caseretrieval\(needle\-in\-a\-haystack, Retr\.KV\),long\-context reference\(MRCR, LongCodeQA,∞\\inftyBench, RULER\), andmathematical reasoning\(GSM8K, chain\-of\-thought\) \(Table[2](https://arxiv.org/html/2608.02947#S6.T2)\)\.

Setup\.Conventions of Table[2](https://arxiv.org/html/2608.02947#S6.T2): RULER uses its 0–100 official metric, shown as the mean of 4 tasks with the multi\-key NIAH broken out; the 5\-task means are 35\.8 / 35\.7\. The∞\\inftyBench columns give the equal\-weight mean of 10 tasks with Math\.Find and Retr\.KV broken out, and the 12\-task means are full 0\.418 / ours 0\.375 / MInf 0\.420\. LongCode is the QA subset of LongCodeBench, exhaustive over the buckets that fit the native context, and these benches score paired window\-ON/OFF runs of the same samples\.

MRCR, LongCodeQA, and the∞\\inftyBench columns are scored with the window injected into a vLLM \(v0\.22\.1\) attention backend of Qwen2\.5\-7B\-Instruct\-1M \(greedy, official scoring\); RULER and GSM8K use the eager\-attention injection of Section[6\.1](https://arxiv.org/html/2608.02947#S6.SS1)\.

MRCR and the 4∞\\inftyBench tasks PassKey, Number, Retr\.KV, and Math\.Find are scored with fp32 attention scores, since we identified and fixed a bf16 score\-rounding issue that flipped greedy selections on low\-margin digit and hex retrieval; the remaining 8 tasks and LongCodeQA are confirmed invariant to the numeric type by a symmetric paired probe with the pre\-fixed criterion\|Δ\|\>2​SE\|\\Delta\|\>2\\,\\mathrm\{SE\}and stay bf16\.

The QK\-pruned row gives the window’s input\-independent term reduction at each column’s median context, example\-weighted for∞\\inftyBench and over the median input\+\+output length for GSM8K\. The GSM8K entry divides by the stock implementation’s actual term count, since gpt\-oss\-20b is a hybrid of sliding\-window and full\-attention layers\.

MInference \(MInf\) values are measured on the official Qwen2\.5\-1M inference stack; within the native regime DCA reduces to identity, so they are effectively MInference alone, and because the serving stack differs, absolute levels are not always directly comparable to full/ours\. The ours\+\+MInf row overlays the window on MInference’s retained cells and prunes a further 43–50% of the retained terms\.

Results\.On LongBench\-v2 4\-choice QA \(Table[1](https://arxiv.org/html/2608.02947#S6.T1)\), the windowed accuracy exceeds full attention in all 5 Qwen2\.5\-0\.5B bands, including the 2 extrapolation bands, while Llama\-3\.2\-3B is higher in 1 of its 4 bands, equal in 2, and lower in 1; given the small per\-band samples we read this as a tendency, not a significant margin\. Across all 6 non\-extrapolation conditions the same\-answer rate is 0\.88–0\.96: decisions, not just distributions, are preserved\.

The worst case for windowing is retrieval of a specific fact placed far away\. Embedding a needle at 5 depths in real LongBench\-v2 text and measuring recall with greedy generation, at bothN=8N\{=\}8K and 16K the per\-wavelength window \(k=1k\{=\}1–88\) achieves100% recall at all policies and all depths\(compute ratio 0\.57–0\.76\), whereas sliding\[256\] \(compute ratio 0\.03\) is0% at all depths\. This gap is not explained by the amount of reduction: even in the iso\-compute control with the compute ratio matched to 0\.45, the sliding window’s KL ismore than two orders of magnitudelarger; within the same budget it reaches only the most recent 44% of the context, whereas the per\-wavelength window covers all distances\. On the official RULER metric the window stays level with full attention; only point\-retrieval MK\-NIAH \(a standard test distinct from the needle experiment above\) drops atk=2k\{=\}2and recovers under the variable\-kkprofile \(Table[2](https://arxiv.org/html/2608.02947#S6.T2)\)\. Applying the window only to gpt\-oss\-20b’s full\-attention layers, GSM8K accuracy stays on par with the unwindowed model through multi\-step reasoning \(Table[2](https://arxiv.org/html/2608.02947#S6.T2); answer match 95\.2%\)\.

Benchmark scoring at the 7B model scale\.We additionally score the window on OpenAI\-MRCR, a multi\-needle co\-reference retrieval benchmark, and on LongCodeQA, a code\-domain 4\-choice benchmark; where the window is inactive—16K—the ON/OFF scores coincide exactly\. Pooled over all bands \(nativen=165n\{=\}165pairs; Table[2](https://arxiv.org/html/2608.02947#S6.T2)\), the window broadly preserves the score \(full 0\.204 / ours 0\.181, not significant\)\. These 165 rows are drawn deterministically and score\-blind from all 583 native\-length rows, and all four arms score the identical set\. Paired tests \(Wilcoxon signed\-rank for continuous scores, exact McNemar for accuracies\) find a significant drop for the window on only 1 task, Retr\.KV \(p=0\.004p\{=\}0\.004\); MRCR \(p=0\.98p\{=\}0\.98\), LongCodeQA, Math\.Find \(p=1\.0p\{=\}1\.0\), and all remaining paired tasks show no significant difference, consistent with score preservation\. This task is a point lookup over sequences with no semantic structure—the boundary case for the hypothesis of Section[3\.1](https://arxiv.org/html/2608.02947#S3.SS1), which presumes positional information contributes\. At thek=2k\{=\}2default this costs accuracy; the ours 0\.30 versus full 0\.75 is not a rounding artifact but the window pruning retrieved keys\. The loss is a compute setting rather than a hard limit: at uniformk=64k\{=\}64, still pruning 29\.0% of the terms, Retr\.KV returns to the full\-attention score \(0\.300→0\.7500\.300\{\\to\}0\.750; McNemarp=0\.004p\{=\}0\.004\), while the variable profile, which keeps low\-frequency pairs neark=2k\{=\}2, does not recover it \(0\.350\)—what matters is long\-range reach, not high\-frequency precision\.

## 7Evaluation of Speed

Setup\.The main measurement platform is the RTX PRO 6000 \(Blackwell generation sm\_120, 96 GB GDDR7\)—a representative single\-GPU inference platform outside the specialization targets of FA3/FA4\. Measurement is a single request \(batch 1\); timing is the median of CUDA events, and decode is set up to eliminate the apparent speedup of L2 residency\. The only FA\-class implementations available as public wheels for this machine are the cuDNN backend of SDPA \(PyTorch’s scaled dot\-product attention\) and FlashInfer \(the public FA2–FA4 wheels do not support sm\_120, but FA4 runs when built from source, and this build—at the same commit as the port base in Section[5](https://arxiv.org/html/2608.02947#S5)—serves as stock FA4\), and we compare against the per\-casemin\\min\(cuDNN, FlashInfer\), thefastest available FA\-class implementation\. Each port is measured as latency normalized to its stock host; because the window\-off path is bit\-identical to stock, this ratio isolates the effect of the window itself\.

Window\-table basis\.The quality experiments \(Sections[6\.1](https://arxiv.org/html/2608.02947#S6.SS1)–[6\.2](https://arxiv.org/html/2608.02947#S6.SS2)\) use the effective wavelengthλr′\\lambda^\{\\prime\}\_\{r\}, whereas the speed measurements use the raw RoPE base\. The two coincide for Qwen2\.5\-7B\-1M, which has no rope scaling; for Llama, which has llama3\-style scaling, this amounts to stronger pruning thank=2k\{=\}2on the effective basis; the resulting end\-to\-end quality and its recovery by one step ofkkare reported in Section[7\.2](https://arxiv.org/html/2608.02947#S7.SS2)\.

### 7\.1Measured Performance of the Ports

![Refer to caption](https://arxiv.org/html/2608.02947v1/figs/fig_llama_ports_v19.png)Figure 5:Latency of the two ports \(Llama\-3\.2\-3B, same run, lower is better\)\. Left: decode; right: prefill\. Bars are relative to each stock implementation \(FlashInfer/FA4\); dashed lines mark stock \(=1\.0=1\.0\)\. Values inside the bars are measured times; arrows give the reduction vs stock\.Decode: the FlashInfer port\.With the window on, FlashInfer decode surpasses stock FlashInfer at all evaluated Llama context lengths \(latency reductions of 6\.0–22\.3%; Figure[5](https://arxiv.org/html/2608.02947#S7.F5), left\)\. At a 160K shape outside the native context, decode keeps growing \(1\.296×1\.296\\timesover stock\) while prefill holds at the 128K level \(1\.193×1\.193\\timesover the per\-case best\)\. On Qwen2\.5\-0\.5B \(2 KV heads,d=64d\{=\}64\) the K read does not dominate decode bandwidth and the window has no effect \(1\.00×1\.00\\times\)\.

Prefill: the FA4 port\.With the window on, FA4 surpasses stock FA4 at all evaluated context lengths, and the reduction \(the arrows in Figure[5](https://arxiv.org/html/2608.02947#S7.F5), right\) grows withNN\(Llama 12\.5→\\to22\.7%\)\. On Llama, moreover, both ports run below cuDNN at every context length \(prefill by 6\.3–16\.4%\)\. On the Qwen2\.5\-7B\-1M shape, the same shape as the 1M run, the picture is sharper still: the windowed kernel is the fastest of the 4 systems at everyNNincluding 8K, with the reduction growing from 10\.4% at 8K to 29\.9% at 512K\. At 512K the gain over stock \(1\.43×1\.43\\times\) exceeds the theoretical ceiling2/\(1\+s\)=1\.352/\(1\+s\)\{=\}1\.35\(ss==the QK slice ratio of the implemented window table\) because the 3 full\-attention systems degrade super\-quadratically there while the window stays quadratic; the window’s intrinsic recovery is 0\.77–0\.83 at 8K–256K\.

Agreement of recovery rates\.Recovery of the theoretical ceiling lines up across the 2 implementation families \(CUDA templates, CuTeDSL\) on Llama—85–91% \(FlashInfer port\) and 92–96% \(FA4 port\)—showing that the same methodology of minimal insertion, bit\-identical window\-off, and prefix\-limited loads converts the bulk of the theoretical reduction into wall\-clock time\.

### 7\.2End\-to\-End Whole\-Request Time \(Llama\)

Setup\.We integrate the two ports into a single inference pipeline, where the KV cache written by the FA4\-port prefill is read by the FlashInfer\-port decode with no conversion, and measure whole\-request time with the window switched on and off in the same binary \(LongBench\-v2 long documents truncated to each KV size,n=8n\{=\}8\)\.

Results\.The saving grows monotonically with context length \(Table[3](https://arxiv.org/html/2608.02947#S7.T3)\)\. On this extreme\-long\-document pool the ON/OFF top\-1 match falls from 0\.90–0\.95 at 8K–64K to 0\.76 at 128K, but one notch of the retained\-period countkkrecovers the quality\. The cost is about 1% of whole\-request time, demonstrating end to end thatkkacts as the deployment\-time dial between accuracy and compute \(Section[3\.2](https://arxiv.org/html/2608.02947#S3.SS2)\)\.

Table 3:Thekkdial \(Llama\-3\.2\-3B, LongBench\-v2,n=8n\{=\}8\): where quality drops, one notch ofkkon the named side recovers it at about 1% of whole\-request time; ratios are OFF/ON\.Table 4:End\-to\-end whole\-request time on Qwen2\.5\-7B\-Instruct\-1M \(window ON/OFF in the same binary; 64K\-chunked prefill;n=6n\{=\}6at everyNN\)\. Ratios are OFF/ON; parentheses give measured seconds \(OFF→\\toON\)\. The maximum per\-request deviation from the median is≤±2\.5\{\\leq\}\\pm 2\.5s; run\-to\-run deviation is≤0\.23%\{\\leq\}0\.23\\%at 256K/512K \(3 runs\)\.
### 7\.3Scaling to 1M Contexts \(Qwen2\.5\-7B\-1M\)

Setup\.To probe the regime where the logarithmic reduction is deepest, we run the same pipeline on Qwen2\.5\-7B\-Instruct\-1M \(native context 262,144, RoPE base10710^\{7\}, no rope scaling\), with the window tables regenerated from its effective frequencies and the prefill executed in 64K chunks \(bit\-identical to single\-shot prefill, both arms\)\.

Results\.On LongBench\-v2 long documents truncated to 256K/512K/1M tokens, the ON/OFF gains grow monotonically with context, reaching a1\.31×1\.31\\timeswhole\-request speedup at 1M \(Table[4](https://arxiv.org/html/2608.02947#S7.T4)\)\. The gap between the attention\-only theoretical ceiling2/\(1\+s\)2/\(1\+s\)and the measured prefill gains reflects non\-attention compute and narrows at longer contexts as attention dominates\. Quality holds at 256K \(native\) and 512K: top\-1 is 0\.996/0\.992, and the LongBench\-v2 answer accuracy at 256K is identical ON and OFF\. 1M lies×\\times3\.85 beyond the native context; top\-1 falls to 0\.77 there, which we report as extrapolation behavior\.

## 8Conclusion

Limitations\.\(i\) The window is fixed in advance, and input adaptation is out of scope\. \(ii\) The theoretical reduction rate is a term\-count proxy \(Section[4](https://arxiv.org/html/2608.02947#S4)\); wall\-clock time is complemented by measurement\. \(iii\) The speed measurements are on a representative platform of a single hardware generation \(sm\_120, GDDR7\); generalization to other generations is unverified\. \(iv\) Quality verification centers on teacher\-forced metrics\. At the defaultk=2k\{=\}2one∞\\inftyBench task degrades significantly \(Retr\.KV\), retrieval over structureless strings, and uniformk=64k\{=\}64—still pruning 29% of the terms—recovers it \(Section[6\.2](https://arxiv.org/html/2608.02947#S6.SS2)\), sokktrades computation against this worst case rather than fixing a hard limit\. \(v\) Kernel speed is median\-based without error bars \(end\-to\-end deviations: Table[4](https://arxiv.org/html/2608.02947#S7.T4)\)\. \(vi\) We do not compare directly with the frequency\-exploiting sparse methods or the sink\-augmented window\(Xiaoet al\.[2024](https://arxiv.org/html/2608.02947#bib.bib16)\)\.

Future work\.Main directions areVV\-side reduction via block\-sparsification and content\-addressed management of unrotated KV toward multi\-request inference\.

Conclusion\.The ports into the released FA4 and FlashInfer surpass their stock implementations at all evaluated Llama context lengths, and end to end the whole\-request speedup reaches1\.31×1\.31\\timesat a 1M\-token context \(Table[4](https://arxiv.org/html/2608.02947#S7.T4)\)\.

## Acknowledgments

This work was supported by the Joint Usage/Research Center for Interdisciplinary Large\-scale Information Infrastructures \(JHPCN\) and the High Performance Computing Infrastructure \(HPCI\) in Japan, under project ID jh260017\. This work was also supported by JST, the Next Generation Edge AI Semiconductor Research and Development Project, Grant Number JPMJES2511\. The experiments in this paper used the supercomputer Miyabi \(OFP\-II\) at the Joint Center for Advanced High Performance Computing \(JCAHPC\), the supercomputer Genkai at the Research Institute for Information Technology, Kyushu University, and the computational resources of R\-CCS Cloud provided by the RIKEN Center for Computational Science \(R\-CCS\)\.

## References

- F\. Barbero, A\. Vitvitskyi, C\. Perivolaropoulos, R\. Pascanu, and P\. Veličković \(2025\)Round and Round We Go\! What Makes Rotary Positional Encodings Useful?\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p2.3)\.
- I\. Beltagy, M\. E\. Peters, and A\. Cohan \(2020\)Longformer: The Long\-Document Transformer\.External Links:2004\.05150Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p3.1)\.
- T\. Dao \(2024\)FlashAttention\-2: Faster Attention with Better Parallelism and Work Partitioning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.02947#S1.p1.1),[§2](https://arxiv.org/html/2608.02947#S2.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-V3\.2: Pushing the Frontier of Open Large Language Models\.External Links:2512\.02556Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p5.1)\.
- J\. Dong, B\. Feng, D\. Guessous, Y\. Liang, and H\. He \(2024\)FlexAttention: A Programming Model for Generating Optimized Attention Kernels\.External Links:2412\.05496Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p7.1)\.
- H\. Jiang, Y\. Li, C\. Zhang, Q\. Wu, X\. Luo, S\. Ahn, Z\. Han, A\. H\. Abdi, D\. Li, C\.\-Y\. Lin, Y\. Yang, and L\. Qiu \(2024\)MInference 1\.0: Accelerating Pre\-filling for Long\-Context LLMs via Dynamic Sparse Attention\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p4.1)\.
- X\. Lai, J\. Lu, Y\. Luo, Y\. Ma, and X\. Zhou \(2025\)FlexPrefill: A Context\-Aware Sparse Attention Mechanism for Efficient Long\-Sequence Inference\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p4.1)\.
- W\. Mao, X\. Lin, W\. Huang, Y\. Xie, T\. Fu, B\. Zhuang, S\. Han, and Y\. Chen \(2026\)TriAttention: Efficient Long Reasoning with Trigonometric KV Compression\.External Links:2604\.04921Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p4.1)\.
- M\. Milakov and N\. Gimelshein \(2018\)Online normalizer calculation for softmax\.External Links:1805\.02867Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p1.1)\.
- J\. Shah, G\. Bikshandi, Y\. Zhang, V\. Thakkar, P\. Ramani, and T\. Dao \(2024\)FlashAttention\-3: Fast and Accurate Attention with Asynchrony and Low\-precision\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p1.1)\.
- J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu \(2024\)RoFormer: Enhanced Transformer with Rotary Position Embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§1](https://arxiv.org/html/2608.02947#S1.p2.3),[§2](https://arxiv.org/html/2608.02947#S2.p2.3)\.
- Q\. Wang, Z\. Yousefijamarani, M\. L\. Heisler, R\. Gu, X\. Bai, Y\. Shan, W\. Zhang, L\. Wang, Y\. Xiong, Y\. Zhang, and Z\. Fan \(2025\)MEPIC: memory efficient position independent caching for LLM serving\.Note:arXiv preprint arXiv:2512\.16822Version 1, December 18, 2025Cited by:[Appendix B](https://arxiv.org/html/2608.02947#A2.p1.1),[Appendix B](https://arxiv.org/html/2608.02947#A2.p3.1)\.
- Y\. Wang, Y\. Wang, Z\. Yue, H\. Zeng, Y\. Wang, I\. Lourentzou, Z\. Tu, X\. Chu, and J\. McAuley \(2026\)FASA: Frequency\-aware Sparse Attention\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p4.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient Streaming Language Models with Attention Sinks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p3.1),[§8](https://arxiv.org/html/2608.02947#S8.p1.4)\.
- Y\. Xie, T\. Wen, T\. Huang, B\. Chen, C\. You, S\. Jegelka, and Y\. Wang \(2026\)Scaling Attention via Feature Sparsity\.External Links:2603\.22300Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p5.1)\.
- R\. Xu, G\. Xiao, H\. Huang, J\. Guo, and S\. Han \(2025\)XAttention: Block Sparse Attention with Antidiagonal Scoring\.InInternational Conference on Machine Learning \(ICML\),pp\. 69819–69831\.Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p4.1)\.
- Z\. Ye, L\. Chen, R\. Lai, W\. Lin, Y\. Zhang, S\. Wang, T\. Chen, B\. Kasikci, V\. Grover, A\. Krishnamurthy, and L\. Ceze \(2025\)FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving\.InProceedings of Machine Learning and Systems \(MLSys\),Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p7.1)\.
- T\. Zadouri, M\. Hoehnerbach, J\. Shah, T\. Liu, V\. Thakkar, and T\. Dao \(2026\)FlashAttention\-4: Algorithm and Kernel Pipelining Co\-Design for Asymmetric Hardware Scaling\.External Links:2603\.05451Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p1.1)\.
- J\. Zhang, C\. Xiang, H\. Huang, J\. Wei, H\. Xi, J\. Zhu, and J\. Chen \(2025\)SpargeAttention: Accurate and Training\-free Sparse Attention Accelerating Any Model Inference\.InInternational Conference on Machine Learning \(ICML\),pp\. 76397–76413\.Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p4.1)\.
- Z\. Zhanget al\.\(2023\)H2O: Heavy\-Hitter Oracle for Efficient Generative Inference of Large Language Models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2608.02947#S2.p3.1)\.

Appendix

This appendix records \(A\) a family of window rules of which the per\-RoPE\-wavelength distance window is one point, \(B\) the distinction between the cache footprint, which position\-independent caching shrinks, and the read\-time rotation count, which the window prunes, \(C\) the closed form of the reduction rate, including the decode case that the main text states without writing out, \(D\) the two implementation routes the pair decomposition admits, and why we take the one that keeps an FA\-class kernel intact, and \(E\) notes on measurement and verification\.

## Appendix AGeneralizing the window rule

We use the notation of the main text throughout \(dd, the pair indexrr, the wavelengthλr\\lambda\_\{r\}, the RoPE baseBB, the context lengthNN, the per\-pair window widthwrw\_\{r\}\)\.

The default windowwr=min⁡\(k​λr,N\)w\_\{r\}=\\min\(k\\lambda\_\{r\},N\)of the main text is one point of a larger family\. Factorizing a window into a retention profilekrk\_\{r\}times a power of the wavelength, clamped between a floor and a ceiling, gives

wr=min⁡\(max⁡\(kr​λrγ\+a​r\+1d/2,β\),W\)w\_\{r\}=\\min\\\!\\bigl\(\\max\(k\_\{r\}\\,\\lambda\_\{r\}^\{\\gamma\}\+a\\,\\tfrac\{r\+1\}\{d/2\},\\ \\beta\),\\ W\\bigr\)\(6\)wherekrk\_\{r\}is the retention profile \(a constantkk, or a gradient that keeps more periods at high frequencies\),γ\\gammais the wavelength exponent,β\\betais a floor in tokens that keeps near distances in full,WWis a ceiling, anda​r\+1d/2a\\,\\tfrac\{r\+1\}\{d/2\}is an additive term that is*linear in the pair indexrr*and carries no wavelength dependence at all; it is0in most settings\. The default window is the special casekr=k,γ=1,β=0,W=N,a=0k\_\{r\}\{=\}k,\\ \\gamma\{=\}1,\\ \\beta\{=\}0,\\ W\{=\}N,\\ a\{=\}0; settingkr=0,a=Wk\_\{r\}\{=\}0,\\ a\{=\}Wdegenerates to a*linear*window, which the figures label as such\. Figure[6](https://arxiv.org/html/2608.02947#A1.F6)illustrates the factorization panel by panel\.

The retention profile is either constant or interpolates from a high\-frequency valuekhik\_\{\\mathrm\{hi\}\}down to a low\-frequency valueklok\_\{\\mathrm\{lo\}\}\. We sweep two interpolations, and the figures name them accordingly:*kk\-arithmetic*decreaseskrk\_\{r\}by a constant amount per pair, linear inrr; the variable\-kkprofile of the main text is this case withkr=16−14d/2−1​rk\_\{r\}=16\-\\tfrac\{14\}\{d/2\-1\}\\,r, running from1616at the highest frequency to22at the lowest\.*kk\-geometric*instead decreaseskrk\_\{r\}by a constant ratio per pair, so it falls faster at high frequencies\. The other labels follow the same equation:c​NcNis a ceiling set as a fraction of the sequence length, andn%n\\%is the fraction of the lowest pairs made position\-free\.*SlidingWW*is the sliding window, the special case in which every pair receives the same width\.

#### Position\-free pairs as a limit\.

A pair whose rotation is removed \(NoPE\) has no wavelength, so no distance renders it positionally uninformative: it is the limitθr→0\\theta\_\{r\}\\\!\\to\\\!0, in whichwr≥Nw\_\{r\}\\geq Nfor everyNNand the pair is retained permanently\. The fraction of low\-frequency pairs treated this way is a separate knob from the window factorization itself\. Architectures that interleave rotated and unrotated attention layers therefore fall inside the same description: the rotated layers are pruned by Eq\. \([6](https://arxiv.org/html/2608.02947#A1.E6)\) and the unrotated ones are not pruned at all\.

![Refer to caption](https://arxiv.org/html/2608.02947v1/figs/fig_param_taxonomy.png)Figure 6:Factorization of the window pattern into a retention profile, a power of the RoPE wavelength, and floor/ceiling terms\.
#### Sweeping the family\.

Figure[7](https://arxiv.org/html/2608.02947#A1.F7)sweeps the knobs of Eq\. \([6](https://arxiv.org/html/2608.02947#A1.E6)\) against the compute they save, measured as the fraction of inner\-product terms pruned, the quantity the main text reports as QK\-pruned\. The sweep runs on Qwen2\.5\-0\.5B atN=8192N\{=\}8192; the error axis is the relative deviation of the windowed attention output from full attention, taken per head as the 95th percentile over query positions and summarized by the median head\. This deviation is measured on the attention output itself; closeness of the model’s output distribution and task scores are established separately in the main text\. Three things are visible\. The sliding window is dominated everywhere: at any level of pruning it carries an error far above every wavelength\-scaled setting\. The wavelength\-scaled families—the exponentγ\\gamma, the floorβ\\beta, the capc​NcN, and a gradient inkrk\_\{r\}—collapse onto a common front over the practical range, so within this setting the additional knobs buy little over the plainγ=1\\gamma\{=\}1default that the main text uses\. Making a fraction of the low\-frequency pairs position\-free costs error at equal compute, which is the expected price of removing the very components that carry long\-range position\.

![Refer to caption](https://arxiv.org/html/2608.02947v1/figs/fig_q50_families.png)Figure 7:Error of the window families of Eq\. \([6](https://arxiv.org/html/2608.02947#A1.E6)\) against the fraction of inner\-product terms pruned \(QK\-pruned\); Qwen2\.5\-0\.5B,N=8192N\{=\}8192\. Left: the whole sweep on a logarithmic error axis; right: the practical range on a linear axis\. Series and marker conventions are given by the in\-figure legends; the families are defined in the text\.
#### Fidelity along the sequence\.

The main text reports the mean KL divergence per band\. Figure[8](https://arxiv.org/html/2608.02947#A1.F8)resolves it along the query position: within the non\-extrapolation regime the curves stay in the10−310^\{\-3\}nat range, and only the two YaRN\-extended settings depart from it\.

![Refer to caption](https://arxiv.org/html/2608.02947v1/figs/fig_pos_profile.png)Figure 8:KL divergence from full attention against query position in the sequence, per context length\. Solid lines give the mean per\-token KL within each position bin; the shaded band above each line extends from that mean to the 95th percentile of the same bin \(the upper tail\)\.

## Appendix BCache footprint versus rotation count

Position\-independent caching \(MEPIC\(Wanget al\.[2025](https://arxiv.org/html/2608.02947#bib.bib20)\)\) keeps the KV cache*unrotated*and fuses the rotation into attention at read time; this shrinks the memory the cache must hold—its*footprint*—because entries carry no position yet and can be shared or deduplicated across requests\. The per\-wavelength window does not shrink the cache\. What it reduces is the number of rotations that have to be applied at read time: out\-of\-window pair components are never brought into on\-chip memory \(SRAM\), so the corresponding rotations—and the trigonometric evaluations or table lookups behind them—are not performed either\.

The two reductions therefore act on different axes of the same runtime\-fused path and can be applied together\. The main text measures the analogous composition with token\-level sparsity: overlaying the window on the cells kept by a dynamic\-sparse method prunes a further 43–50% of the retained terms\.

In the single\-request, single\-GPU setting of our speed measurements the trigonometric work is not the bottleneck, which is why the main text quantifies compute in inner\-product terms and in transferred bytes\. Multi\-request serving by itself does not change this either: per request, the rotation work still grows only linearly\. What changes the picture is the position\-independent cache itself\. There the cache holds*one unrotated entry per token*, shared across the requests that reference it, and the rotation is fused into every read: the trigonometric work then scales not with the tokens written but with the reads of that entry, summed over all sharing requests\(Wanget al\.[2025](https://arxiv.org/html/2608.02947#bib.bib20)\)\. Reads dwarf writes in exactly the settings that motivate such caches, and it is this read\-time rotation that the per\-wavelength window prunes pair by pair\. That regime is outside the scope of this paper\.

## Appendix CDeriving the reduction rate

The main text solves the double integral over rows and pair index for the whole prefill\. It is worth solving it once for an arbitrary row interval, because prefill and decode then become the same expression evaluated on different rows rather than two separate derivations\. We then carry the same integral through the window family of Eq\. \([6](https://arxiv.org/html/2608.02947#A1.E6)\)\.

#### Step 1: an arbitrary row interval\.

Following the main text we relax the discrete pair indexrrto a continuous variableρ\\rho; for the default window \(kr=kk\_\{r\}\{=\}k,γ=1\\gamma\{=\}1\) this givesw​\(ρ\)=wmin​B2​ρ/dw\(\\rho\)=w\_\{\\min\}B^\{2\\rho/d\}withwmin=2​π​kw\_\{\\min\}=2\\pi k\. Lets​\(n\)s\(n\)denote the number of inner\-product terms the window retains in query rownn, the integrand of the main text’s double integralSS\(the letterKKis kept for the truncation order of Appendix[D](https://arxiv.org/html/2608.02947#A4)\):

s​\(n\)=∫0d/2min⁡\(w​\(ρ\),n\)​𝑑ρ\.s\(n\)=\\int\_\{0\}^\{d/2\}\\min\\bigl\(w\(\\rho\),n\\bigr\)\\,d\\rho\.\(7\)Full attention computesd2​n\\tfrac\{d\}\{2\}nterms in the same row\. Forn≥wminn\\geq w\_\{\\min\}the minimum switches at the saturation boundaryρ∗​\(n\)=d2​ln⁡\(n/wmin\)/ln⁡B\\rho^\{\*\}\(n\)=\\tfrac\{d\}\{2\}\\ln\(n/w\_\{\\min\}\)/\\ln B, wherew​\(ρ∗\)=nw\(\\rho^\{\*\}\)\{=\}n, so

s​\(n\)=\(∫0ρ∗w​\(ρ\)​𝑑ρ\)⏟ρ<ρ∗​\(n\)\+\(d2−ρ∗​\(n\)\)​n⏟ρ≥ρ∗​\(n\)\.s\(n\)=\\underbrace\{\\biggl\(\\int\_\{0\}^\{\\rho^\{\*\}\}\\\!\\\!w\(\\rho\)\\,d\\rho\\biggr\)\}\_\{\\rho<\\rho^\{\*\}\(n\)\}\\;\+\\;\\underbrace\{\\Bigl\(\\frac\{d\}\{2\}\-\\rho^\{\*\}\(n\)\\Bigr\)n\}\_\{\\rho\\geq\\rho^\{\*\}\(n\)\}\.\(8\)The first term integrates the window widths of the pairs left of the boundary and evaluates tod2​ln⁡B​\(n−wmin\)\\tfrac\{d\}\{2\\ln B\}\(n\-w\_\{\\min\}\); the second counts the remaining pairs, for which the causal lengthnnbinds instead\. Figure[9](https://arxiv.org/html/2608.02947#A3.F9)draws the split to scale for a single query row, each area labeled by its own term\.

![Refer to caption](https://arxiv.org/html/2608.02947v1/figs/fig_sn_decomp.png)Figure 9:The split of Eq\. \([8](https://arxiv.org/html/2608.02947#A3.E8)\) for a single query row, to scale \(Qwen2\.5\-0\.5B constants,k=2k\{=\}2,n=16384n\{=\}16384\)\. The painted areas equal the two terms: orange under the dashed continuousw​\(ρ\)w\(\\rho\)up toρ∗​\(n\)=16\.6\\rho^\{\*\}\(n\)\{=\}16\.6, yellow beyond; gray: pruned\. The cyan steps mark the discretemin⁡\(wr,n\)\\min\(w\_\{r\},n\)actually counted; their deviation from the curve is the integral\-approximation error \(pruned: 46\.3% discrete, 44\.7% continuous\)\.Integrating the pruned countd2​n−s​\(n\)\\tfrac\{d\}\{2\}n\-s\(n\)over the rows\[n0,n1\]\[n\_\{0\},n\_\{1\}\]and dividing by thed2⋅n12−n022\\tfrac\{d\}\{2\}\\cdot\\tfrac\{n\_\{1\}^\{2\}\-n\_\{0\}^\{2\}\}\{2\}terms that full attention holds there gives the reduction rate of that interval, which we denoteR​\(n0,n1\)R\(n\_\{0\},n\_\{1\}\)\. Writing

F​\(n\)=n2​\(ln⁡nwmin−32\)\+2​wmin​nF\(n\)=n^\{2\}\\Bigl\(\\ln\\frac\{n\}\{w\_\{\\min\}\}\-\\frac\{3\}\{2\}\\Bigr\)\+2w\_\{\\min\}n\(9\)for the primitive that this integration produces \(up to constant factors\),

R​\(n0,n1\)=\[F​\(n\)\]n0n1\(n12−n02\)​ln⁡B\.R\(n\_\{0\},n\_\{1\}\)=\\frac\{\\bigl\[\\,F\(n\)\\,\\bigr\]\_\{n\_\{0\}\}^\{n\_\{1\}\}\}\{\(n\_\{1\}^\{2\}\-n\_\{0\}^\{2\}\)\\,\\ln B\}\\,\.\(10\)

#### Step 2: prefill and decode as substitutions\.

Both cases of the main text are substitutions into Eq\. \([10](https://arxiv.org/html/2608.02947#A3.E10)\), and each constant can be read straight offFF\. The whole prefill is\(n0,n1\)=\(1,N\)\(n\_\{0\},n\_\{1\}\)=\(1,N\), whereF​\(1\)F\(1\)is negligible andF​\(N\)/\(N2​ln⁡B\)F\(N\)/\(N^\{2\}\\ln B\)remains; dropping the2​wmin​N2w\_\{\\min\}Nterm forN≫wminN\\gg w\_\{\\min\}leaves the prefill form with exactly the constant the primitive carries,−32\-\\tfrac\{3\}\{2\}\. Decode is the last row alone,\(n0,n1\)=\(N−1,N\)\(n\_\{0\},n\_\{1\}\)=\(N\{\-\}1,N\): the bracket is then the unit\-step differenceF​\(N\)−F​\(N−1\)F\(N\)\-F\(N\{\-\}1\), which isF′​\(N\)F^\{\\prime\}\(N\)to leading order, with

F′​\(n\)=2​n​\(ln⁡nwmin−1\)\+2​wmin,F^\{\\prime\}\(n\)=2n\\bigl\(\\ln\\tfrac\{n\}\{w\_\{\\min\}\}\-1\\bigr\)\+2w\_\{\\min\},\(11\)and division by the denominator\(2​N−1\)​ln⁡B≃2​N​ln⁡B\(2N\{\-\}1\)\\ln B\\simeq 2N\\ln Bleaves−1\-1\. In the notation of the main text, asN→∞N\\to\\inftybelow saturation,

reductionprefill\\displaystyle\\mathrm\{reduction\}\_\{\\mathrm\{prefill\}\}=R​\(1,N\)\\displaystyle=R\(1,N\)\(12\)=ln⁡\(N/wmin\)−3/2ln⁡B\+O​\(wmin/N\),\\displaystyle=\\frac\{\\ln\(N/w\_\{\\min\}\)\-3/2\}\{\\ln B\}\+O\\bigl\(w\_\{\\min\}/N\\bigr\),reductiondecode\\displaystyle\\mathrm\{reduction\}\_\{\\mathrm\{decode\}\}=R​\(N−1,N\)\\displaystyle=R\(N\{\-\}1,N\)=ln⁡\(N/wmin\)−1ln⁡B\+O​\(wmin/N\)\.\\displaystyle=\\frac\{\\ln\(N/w\_\{\\min\}\)\-1\}\{\\ln B\}\+O\\bigl\(w\_\{\\min\}/N\\bigr\)\.The two constants are therefore not separate results: the whole prefill reads the primitive, decode reads its derivative\. TheO​\(wmin/N\)O\(w\_\{\\min\}/N\)terms can be written out: evaluating the last row directly, with no outer integration, gives1−s​\(N\)/\(d2​N\)=\[ln⁡\(N/wmin\)−1\+wmin/N\]/ln⁡B1\-s\(N\)/\(\\tfrac\{d\}\{2\}N\)=\\bigl\[\\ln\(N/w\_\{\\min\}\)\-1\+w\_\{\\min\}/N\\bigr\]/\\ln B, so the decode error is exactlywmin/\(N​ln⁡B\)w\_\{\\min\}/\(N\\ln B\), and the prefill form carries2​wmin/N2w\_\{\\min\}/Nin the same place\. Neither leading term is a limit value: the rates rise along the logarithmic law until saturation ends it, and the only value the reduction converges to asN→∞N\\to\\inftyis11\(Step 3\)\.

#### The interval itself, not only its two extremes\.

Keeping the general interval has a direct use\. In a long\-running session, a further user message or a returned tool result is appended on top of a context that is already long, and with the existing KV retained, only the new rows are computed\. Each such step evaluates\[n0,n1\]\[n\_\{0\},n\_\{1\}\]withn0n\_\{0\}the length already present, which is Eq\. \([10](https://arxiv.org/html/2608.02947#A3.E10)\) with neither extreme substituted—the whole prefill \(n0=1n\_\{0\}\{=\}1\) and decode \(n1−n0=1n\_\{1\}\{\-\}n\_\{0\}\{=\}1\) are its endpoints\. The reduction rises asn0n\_\{0\}moves up the context, because the short early rows, the ones no window can prune, are no longer counted: over the second half of the context,\[N/2,N\]\[N/2,N\], Eq\. \([10](https://arxiv.org/html/2608.02947#A3.E10)\) gives, to the same order,\(ln⁡\(N/wmin\)−1\.27\)/ln⁡B\\bigl\(\\ln\(N/w\_\{\\min\}\)\-1\.27\\bigr\)/\\ln B, about 1\.7 points above the whole prefill atB=106B\{=\}10^\{6\}, or half of the 3\.6\-point gap to decode\. \(This is distinct from chunked prefill, which splits one input but still computes every row from 1 toNN, so its total remainsR​\(1,N\)R\(1,N\)\.\)

#### Step 3: what converges asNNgrows\.

Two things do\. First, both intervals share the slope1/ln⁡B1/\\ln B, so their*gap*converges to the constant\(32−1\)/ln⁡B=1/\(2​ln⁡B\)\(\\tfrac\{3\}\{2\}\-1\)/\\ln B=1/\(2\\ln B\)—the 3\.6 points quoted in the main text atB=106B\{=\}10^\{6\}\. Decode is the deeper of the two because the prefill value averages in the early rows, which are still short enough for the window to cover them\. Second, the logarithmic law itself holds only until the boundary saturates: atN=k​λmax=wmin​BN=k\\lambda\_\{\\max\}=w\_\{\\min\}Bwe haveρ∗=d/2\\rho^\{\*\}\{=\}d/2, no window is clipped bynn, and Eq\. \([8](https://arxiv.org/html/2608.02947#A3.E8)\) collapses tos=∑rwrs=\\sum\_\{r\}w\_\{r\}, independent ofnn\. The decode reduction reaches1−1/ln⁡B1\-1/\\ln Bthere \(92\.8% atB=106B\{=\}10^\{6\}\) and tends to11beyond it, since the retained count stays fixed while the row keeps growing\.

#### Step 4: the same integral over the general family\.

Carried through the same steps, Eq\. \([6](https://arxiv.org/html/2608.02947#A1.E6)\) shows that the wavelength exponent reduces to the same law\. Withw​\(ρ\)=k​λ​\(ρ\)γw\(\\rho\)=k\\lambda\(\\rho\)^\{\\gamma\}andλ​\(ρ\)=2​π​B2​ρ/d\\lambda\(\\rho\)=2\\pi B^\{2\\rho/d\}, the window is again a geometric progression,w​\(ρ\)=k​\(2​π\)γ​B2​γ​ρ/dw\(\\rho\)=k\(2\\pi\)^\{\\gamma\}B^\{2\\gamma\\rho/d\}, so every expression above survives under the two replacements

ln⁡B⟶γ​ln⁡B,wmin⟶k​\(2​π\)γ\.\\ln B\\ \\longrightarrow\\ \\gamma\\ln B,\\qquad w\_\{\\min\}\\ \\longrightarrow\\ k\(2\\pi\)^\{\\gamma\}\.\(13\)The exponent therefore acts exactly as a rescaling of the RoPE base toBγB^\{\\gamma\}: aγ\\gamma\-window on a model of baseBBprunes like the default window on a model of baseBγB^\{\\gamma\}\. Saturation moves toN=k​λmaxγN=k\\lambda\_\{\\max\}^\{\\gamma\}, where the decode reduction is1−1/\(γ​ln⁡B\)1\-1/\(\\gamma\\ln B\)\.

The other end of the family behaves differently in kind\. The degenerate linear window \(kr=0k\_\{r\}\{=\}0,a=Wa\{=\}W, i\.e\.w​\(ρ\)=W​\(ρ\+1\)/\(d/2\)w\(\\rho\)=W\(\\rho\{\+\}1\)/\(d/2\)\) saturates atρ∗=d2​\(n/W\)−1\\rho^\{\*\}=\\tfrac\{d\}\{2\}\(n/W\)\-1, and the same two integrals give a retention rate1−n/\(2​W\)1\-n/\(2W\)up toO​\(1/d\)O\(1/d\)corrections\. Its reduction is therefore*linear*innnrather than logarithmic: it reaches one half atn=Wn\{=\}Wand then follows1−W/\(2​n\)1\-W/\(2n\)once every window is saturated\. Both ends tend to full pruning, but the wavelength\-scaled family approaches it logarithmically while the linear window approaches it as1/n1/n—which is the structural reason the two are not interchangeable at a fixed compute budget\.

The floorβ\\betaand the ceilingWWkeep the integral elementary; they only add breakpoints wherew​\(ρ\)w\(\\rho\)meets them, splitting\[0,d/2\]\[0,d/2\]into further subintervals of the same two kinds\. The one exception is the mixed case: when the exponential and the additive linear term are both active,w​\(ρ∗\)=nw\(\\rho^\{\*\}\)\{=\}nis transcendental inρ∗\\rho^\{\*\}, and its root is a value of the Lambert functionW0W\_\{0\}\(not the window ceilingWW\)\. The family is thus elementary at either end but not in between\.

## Appendix DTwo routes: slicing and accumulation

The exact pair decomposition admits two implementation routes\. One slices the reduction axis and leaves the surrounding kernel alone; the other turns the sum over pairs into a recurrence and accumulates the exponential along it\. We take only the first, and this section records why\.

#### \(a\) Slicing the reduction axis \(the main text\)\.

Because the window is static, the set of active pairs per tile is known at compile time, so the tiling, the running maximum, and the normalizer of an FA\-class kernel are left untouched and only the effective inner\-product length shrinks\. Nothing about the model is assumed beyond the RoPE constants\.

#### \(b\) Accumulating the exponential in increment form\.

With the partial sum of the score over pairs written ascm,r=cm,r−1\+Δ​xm,rc\_\{m,r\}=c\_\{m,r\-1\}\+\\Delta x\_\{m,r\}, wheremmis the key position andΔ​xm,r\\Delta x\_\{m,r\}the per\-pair contribution of the main text’s score decomposition, and withℓm,r=ecm,r\\ell\_\{m,r\}=e^\{c\_\{m,r\}\}, the exponential law gives the identity

ℓm,r=ℓm,r−1\+ecm,r−1​\(eΔ​xm,r−1\),\\ell\_\{m,r\}=\\ell\_\{m,r\-1\}\+e^\{\\,c\_\{m,r\-1\}\}\\bigl\(e^\{\\,\\Delta x\_\{m,r\}\}\-1\\bigr\),\(14\)which is exact\. Each pair then contributes only through the incrementeΔ​xm,r−1e^\{\\Delta x\_\{m,r\}\}\-1, which for small arguments can be truncated,

eΔ​xm,r−1=Δ​xm,r\+Δ​xm,r22\!\+Δ​xm,r33\!\+⋯,e^\{\\,\\Delta x\_\{m,r\}\}\-1=\\Delta x\_\{m,r\}\+\\frac\{\\Delta x\_\{m,r\}^\{2\}\}\{2\!\}\+\\frac\{\\Delta x\_\{m,r\}^\{3\}\}\{3\!\}\+\\cdots,\(15\)so that the exponential is never evaluated directly\. The catch is the range ofΔ​x\\Delta x, and it is not a matter of degree\. Figure[10](https://arxiv.org/html/2608.02947#A4.F10)contrasts a model that normalizes queries and keys with one that does not\. With normalization the error falls steadily with the order: the median head reaches5\.4×10−45\.4\\times 10^\{\-4\}atK=4K\{=\}4and1\.1×10−81\.1\\times 10^\{\-8\}atK=8K\{=\}8\. Without it the median error is two to three orders of magnitude larger at equal truncation order \(2\.4×10−12\.4\\times 10^\{\-1\}atK=4K\{=\}4\)\. More damaging for a kernel, which must be correct for every head, the worst heads*diverge*as the order grows: the largest head error rises from9\.4×1029\.4\\times 10^\{2\}atK=1K\{=\}1to1\.5×10191\.5\\times 10^\{19\}atK=8K\{=\}8, and one head overflows outright, because the expansion is applied to logits of magnitude∼9\.4×102\\sim\\\!9\.4\\times 10^\{2\}\. Route \(b\) also gives up the running\-maximum subtraction that makes the standard formulation safe, so it is a different kernel architecture rather than a modification of one\.

![Refer to caption](https://arxiv.org/html/2608.02947v1/figs/fig_qknorm_b.png)Figure 10:Relative error of the truncated increment against the truncation order, for a model with query–key normalization \(OLMo\-2\-1B\) and for one without \(Qwen2\.5\-0\.5B\)\.We take route \(a\) because it is model\-agnostic: it requires no assumption on the logit range and it preserves bit\-identity with the host kernel when the window is off\. Route \(b\) remains open, and with it the question that both routes share—how to avoid holdingexe^\{x\}per element at all\.

#### Precision as the other side of the compute–accuracy trade\.

Pruning terms shortens the score sum, which can narrow the margins that decide the arg\-max\. We observed this directly on the digit\-retrieval task Math\.Find: under bf16 the windowed arm’s score was depressed while the full\-attention arm was invariant to the numeric type, and rescoring both arms in fp32 closed the gap; the paired test in the main text returnsp=1\.0p\{=\}1\.0\. Splitting a low\-precision product to recover accuracy is therefore a natural direction for spending part of what the window saves; we make no performance claim here\.

## Appendix ENotes on measurement and verification

These notes record protocol details that the main text compresses for space; none of them changes a reported number\.

#### Answer\-format addendum for retrieval\-style tasks\.

The four retrieval\-style∞\\inftyBench tasks \(PassKey, Number, Retr\.KV, Code\.Run\) are scored under a one\-line prompt addendum that fixes the answer format\.

#### Two\-site execution of∞\\inftyBench\.

Byte\-identity of the∞\\inftyBench data and tokenizer across the main text’s two GPU platforms was verified\.

#### MRCR native sample\.

The recipe behind the main text’s draw of then=165n\{=\}165native rows: within each length bin the rows are sorted by length, and contiguous blocks from a fixed offset are accumulated wave by wave\. In the extrapolation band \(491–496K tokens,n=3n\{=\}3, where the dual\-chunk attention \(DCA\) mechanism of the official serving stack is active\) every method scores near zero\.

#### Choice of the official RULER metric\.

On the broad\-reference tasks—variable tracking \(VT\), common\-words extraction \(CWE\), and frequent\-words extraction \(FWE\)—strict accuracy is zero even for full attention; the official metric is the one under which the baseline itself is measurable\.

#### Implementation checks behind bit\-identity\.

Beyond bit\-identity with the window off, the ports passed: equality of the window\-on path atk=∞k\{=\}\\inftywith the window\-off path; correctness against a reference implementation; hash\-identical kernel outputs before and after rebasing the window patch \(FA4 port\); and, for the FlashInfer port, agreement of output hashes across four configurations on Llama with the window tables rounded, at injection, to that port’s 16\-byte vector\-load granularity \(a load\-path constant, distinct from the slice\-boundary rounding of the main text\)\.

#### Decode timing without L2 residency\.

The decode platform has a nominal bandwidth of 1,792 GB/s and a 128 MiB L2; timed reads cycle through a KV pool of at least 512 MiB\.