Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding

# Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding
Source: [https://arxiv.org/html/2609.00097](https://arxiv.org/html/2609.00097)
1\]Fudan University, Shanghai, China 2\]Shanghai Innovation Institute, Shanghai, China 3\]Harbin Institute of Technology, Harbin, China\\contribution\[†\]Corresponding author\\correspondence253108120105@sii\.edu\.cn,zhangmin2021@hit\.edu\.cn,ziwei\.he@sii\.edu\.cn,xpqiu@fudan\.edu\.cn

Zhiyuan NingRuixiao LiXiaoran LiuYuerong SongMin ZhangZiwei HeXipeng QiuAffiliation:\[Affiliation:\[Affiliation:\[

###### Abstract

The development of long\-context Large Language Models \(LLMs\) is constrained by the memory bandwidth bottleneck and quadratic complexity of the attention mechanism during decoding\. To overcome the inherent trade\-offs between the memory overhead of metadata\-based metrics and the computational inefficiency of adaptive selection strategies, we presentFaster Flash Decoding \(FFD\), a novel hardware\-algorithm co\-design framework designed to break the memory wall in long\-context decoding\. FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content\-aware scanning via low\-bit quantization\. Furthermore, we introduce the top\-δ\\deltastrategy, which dynamically filters blocks to achieve distribution\-adaptive sparsity without global synchronization\. Offering a training\-free and plug\-and\-play solution, FFD also enables the reuse of scanning results for computation, achieving up to 11\.6×\\timeskernel\-level speedup and scaling to 256K context length, with 2\.37×\\timesend\-to\-end throughput improvement\. Empirical validation on RULER and LongBench confirms that FFD maintains model accuracy while delivering high\-ratio sparsity, with code available at[https://github\.com/qluoluo/faster\-flash\-decoding](https://github.com/qluoluo/faster-flash-decoding)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.00097v1/ffa_intro-v3.png)Figure 1:Comparison of selection strategies\. \(a\) Top\-kk\(fixed budget\) is fast but fails to adapt to distribution shape\. \(b\) Top\-ppoffers dynamic cardinality but introduces non\-trivial overhead due to global synchronization and sorting, which breaks the streaming pipeline\. \(c\) Top\-δ\\delta\(ours\) uses a relative thresholdδ\\deltafrom the local max, achieving the adaptivity of top\-ppwith the hardware efficiency of top\-kk\.The capability to process long contexts has become a defining characteristic of modern Large Language Models \(LLMs\)\[[1](https://arxiv.org/html/2609.00097#bib.bib1),[2](https://arxiv.org/html/2609.00097#bib.bib2),[3](https://arxiv.org/html/2609.00097#bib.bib3),[4](https://arxiv.org/html/2609.00097#bib.bib4)\]\. However, long\-context capability comes at a prohibitive cost during the decoding phase\. As the sequence length grows, the standard attention mechanism exhibits quadratic computational complexity and linear memory growth for Key\-Value \(KV\) caches\[[5](https://arxiv.org/html/2609.00097#bib.bib5)\], creating a severe memory wall bottleneck in the decoding stage\[[6](https://arxiv.org/html/2609.00097#bib.bib6),[7](https://arxiv.org/html/2609.00097#bib.bib7)\]\. Unlike the prefill phase where computation dominates and queries are processed in parallel, the autoregressive decode phase is memory\-bandwidth\-bound: each token generation requires reloading the entire KV cache from HBM, making IO the primary bottleneck rather than FLOPs\. Consequently, sparse attention mechanisms, particularly training\-free, plug\-and\-play approaches, have emerged as a critical research direction to reduce computational overhead without retraining the model\[[8](https://arxiv.org/html/2609.00097#bib.bib8),[9](https://arxiv.org/html/2609.00097#bib.bib9),[10](https://arxiv.org/html/2609.00097#bib.bib10),[11](https://arxiv.org/html/2609.00097#bib.bib11),[12](https://arxiv.org/html/2609.00097#bib.bib12)\]\.

Despite the diversity of sparse attention techniques, most approaches adhere to a decoupled Selector\-Computer paradigm\. In this standard workflow, the selector first filters indices based on metrics, and the computer subsequently retrieves data for calculation\. This separation prevents the Computer from reusing the Selector’s intermediate computations, leading to efficiency bottlenecks\. Furthermore, the two core components within this paradigm—the importance metric and the selection strategy—face their own dilemmas\. On one hand, to determine which tokens to retain, existing methods typically rely on extra metadata, such as partial dimensions, cluster centroids, or geometric bounds\[[9](https://arxiv.org/html/2609.00097#bib.bib9),[13](https://arxiv.org/html/2609.00097#bib.bib13),[8](https://arxiv.org/html/2609.00097#bib.bib8)\]\. These approaches result in a metric dilemma: they either incur additional memory overhead to store these metadata structures or suffer from information distortion\. On the other hand, selecting the optimal subset of tokens remains challenging\. As shown in Fig[1](https://arxiv.org/html/2609.00097#S1.F1),top\-kkselectionis computationally efficient but fails to adapt to the varying entropy of attention distributions\[[14](https://arxiv.org/html/2609.00097#bib.bib14),[15](https://arxiv.org/html/2609.00097#bib.bib15)\]\. Comparatively,top\-ppselectionoffers theoretical superiority by adapting to the distribution, but needs a global softmax operation to compute cumulative probabilities, thus limiting its efficiency\[[10](https://arxiv.org/html/2609.00097#bib.bib10)\]\.

In this paper, we proposeFFD,Faster Flash Decoding, a novel sparse attention framework that addresses these challenges through a hardware\-algorithm co\-design\. By integrating the selector and computer, FFD enables the computation stage to directly reuse the scanning results from the selection stage\. We shift from metadata\-based indexing to content\-aware scanning\. Instead of storing extra metadata, we split the K cache into low\-bit quantization and half\-precision residual, taking the former as a compressed representation of the KV cache\. Then, we introduce atop\-δ\\bm\{\\delta\}strategy rooted in numerical precision analysis\. Rather than enforcing a fixed budget \(kk\) or computing global probabilities \(pp\), top\-δ\\deltadynamically filters blocks based on their estimated contribution to the attention score sum and can be approximately parallelized thanks to the properties of attention in LLMs\[[16](https://arxiv.org/html/2609.00097#bib.bib16)\]\. We implement FFD as a fully fused kernel that eliminates reduction bubbles, achieving 11\.6×\\timeskernel\-level speedup\. Our contributions are summarized as follows:

- •Content\-Aware Scanning via Low\-Bit Quantization\. We resolve the “metric dilemma" by replacing metadata\-based indexing with content\-aware scanning\. By partitioning the K\-cache into low\-bit quantized representations and 8\-bit residuals, we eliminate the memory overhead of extra metadata while maintaining high information fidelity\.
- •The Top\-δ\\bm\{\\delta\}Selection Strategy\. We introduce top\-δ\\delta, a novel selection mechanism, unlike top\-kk\(fixed budget\) or top\-pp\(high overhead\), top\-δ\\deltadynamically filters attention blocks based on their contribution to the attention sum, offering distribution\-adaptive sparsity with the efficiency of parallel execution\.
- •Faster Flash Decoding \(FFD\)\. We present FFD, a hardware\-algorithm co\-design framework tailored for the bandwidth\-critical decode phase\. We implement a fused selector\-computer kernel where the calculation phase reuses the selector’s quantized scan, optimizing memory access patterns\. Our implementation achieves up to 11\.6×\\timeskernel\-level speedup and scales to 256K context, with up to 2\.37×\\timesend\-to\-end throughput increase while remaining a plug\-and\-play, training\-free solution\.
- •Extensive Empirical validation\. We demonstrate that FFD maintains the performance of LLMs across various short and long\-context benchmarks, including RULER\[[17](https://arxiv.org/html/2609.00097#bib.bib17)\]and LongBench\[[18](https://arxiv.org/html/2609.00097#bib.bib18)\], proving that high\-ratio sparsity can be achieved without retraining or sacrificing accuracy\.

![Refer to caption](https://arxiv.org/html/2609.00097v1/ffa_main-v5.png)Figure 2:Overview of our FFD implementation\. Inputs include the 16\-bit query as well as the value cache and the key cache split into a 2\-bit quantized thumbnail and an 8\-bit residual\. The attention output is acquired after K quantization, pseudo\-max estimation, top\-δ\\deltaselection, as well as filtering based on the top\-δ\\deltastrategy, and fine\-grained refinement\.#### Conflict of Interest Disclosure

We declare no financial conflicts of interest related to this work\.

## 2Related Work

#### Hardware\-Aware Exact Attention\.

The standard attention mechanism imposes a quadratic cost during prefilling and a linear cost per step during decoding, creating a significant bottleneck for long\-context applications\. FlashAttention\[[6](https://arxiv.org/html/2609.00097#bib.bib6),[19](https://arxiv.org/html/2609.00097#bib.bib19)\]revolutionized this field by introducing IO\-aware optimizations, employing tiling and kernel fusion to minimize High\-Bandwidth Memory \(HBM\) access\. FlashDecoding and FlashDecoding\+\+\[[20](https://arxiv.org/html/2609.00097#bib.bib20)\]further adapted these principles to the generative phase by parallelizing attention computation across the sequence dimension\. While FFD inherits the spirit of kernel fusion and hardware\-algorithm co\-design from these works, existing Flash\-based methods fundamentally remainexactattention mechanisms\. As context length increases, they must still load and scan the entire KV cache, resulting in unavoidable linear growth in latency\. FFD addresses this by integrating sparsity directly into the fused kernel, breaking the linear scanning barrier while preserving hardware efficiency\.

#### Dynamic Sparse Attention

To circumvent the costs of exact attention, various sparse attention mechanisms have been proposed\. These methods generally face specific challenges in importance estimation \(the Metric Dilemma\) and subset filtering \(the Selection Dilemma\)\. Regarding the Metric Dilemma, methods like H2O\[[14](https://arxiv.org/html/2609.00097#bib.bib14)\], Scissorhands\[[21](https://arxiv.org/html/2609.00097#bib.bib21)\], and SnapKV\[[22](https://arxiv.org/html/2609.00097#bib.bib22)\]rely on historical accumulation to identify "heavy hitters" for token eviction\. However, purely history\-based metrics risk discarding tokens that become relevant only in future contexts\. To address this, dynamic retrieval methods such as SparQ\[[9](https://arxiv.org/html/2609.00097#bib.bib9)\]and Quest\[[8](https://arxiv.org/html/2609.00097#bib.bib8)\]employ auxiliary metadata \(e\.g\., mean vectors or Min\-Max bounds\) to estimate block importance\. Similarly, InfLLM\[[13](https://arxiv.org/html/2609.00097#bib.bib13)\]utilizes representative blocks for neighborhood search\. While these metadata\-based approaches reduce scanning costs, they suffer from information distortion and require additional memory management\. Regarding the Selection Dilemma, most approaches \(e\.g\., Quest, SparQ\) enforce a statictop\-kkbudget, which lacks flexibility across varying attention entropy distributions\. Twilight\[[10](https://arxiv.org/html/2609.00097#bib.bib10)\]attempts to adapt to various attention distributions but introduces a global softmax operation, thus limiting its efficiency\. Concurrent to our work, SALE\[[23](https://arxiv.org/html/2609.00097#bib.bib23)\]independently explored low\-bit estimation with anchor\-based selection for sparse attention\. However, SALE targets the prefill stage and employs 4\-bit quantization, doubling the scanning bandwidth compared to our 2\-bit design tailored for bandwidth\-critical decoding\. In contrast, FFD performs content\-aware scanning via a fused Selector\-Computer kernel, bypassing the need for auxiliary structures and enabling a dynamictop\-δ\\deltaselection strategy that adapts to entropy changes on the fly\.

#### KV Cache Quantization

KV cache quantization is primarily employed to alleviate memory capacity constraints\. Methods such as KIVI\[[24](https://arxiv.org/html/2609.00097#bib.bib24)\], Atom\[[25](https://arxiv.org/html/2609.00097#bib.bib25)\], and KVQuant\[[26](https://arxiv.org/html/2609.00097#bib.bib26)\]compress keys and values into low\-precision formats to fit longer contexts into GPU memory\. Typically, these frameworks dequantize data back to higher precision \(FP16/BF16\) before performing the attention computation to preserve accuracy\. FFD fundamentally repurposes quantization\. Instead of treating low\-bit representations solely as a storage compression technique, we utilize the quantized INT2/4 data as a high\-speed proxy for the importance metric itself\. This allows our selector to scan the full context with high fidelity and minimal latency, avoiding the decompression overhead typical of standard quantization frameworks while maintaining superior precision compared to metadata\-based heuristics\.

## 3Method

In this section, we presentFFD\(Faster Flash Decoding\), a hardware\-aware sparse attention framework designed to break the memory wall in long\-context decoding\. FFD is built upon a hardware\-algorithm co\-design that systematically resolves the trade\-offs in sparse attention through two core innovations: Content\-Aware Scanning via Low\-Bit Quantization and top\-𝜹\\bm\{\\delta\}Selection Strategy\. FFD utilizes low\-bit quantization thumbnails of the K cache to perform high\-fidelity importance estimation without additional memory overhead\. To adapt to varying attention distributions, the top\-δ\\deltacriterion is introduced to provide a theoretically grounded, parallelizable alternative to the rigid top\-kkor computationally expensive top\-ppselection\. We implement a fused operator for the selector and computer to optimize the execution; specifically, the computer reuses the selector’s estimation results to continue the computation, thereby accelerating the process while efficiently supporting the adaptive selection\. The overall procedure is summarized in Figure[2](https://arxiv.org/html/2609.00097#S1.F2)and Alg\.[1](https://arxiv.org/html/2609.00097#alg1)\.

The remainder of this section is organized as follows\. First, we detail the efficient quantization\-based scanning mechanism that enables high\-fidelity importance estimation\. Next, we introduce the formulation and parallel implementation of the top\-δ\\deltaselection\. Finally we describe the system\-level optimization and fused kernel design that underpin the high\-speed execution of FFD\.

### 3\.1Content\-Aware Scanning via Low\-Bit Quantization

Inspired by advances in KV cache quantization\[[24](https://arxiv.org/html/2609.00097#bib.bib24),[26](https://arxiv.org/html/2609.00097#bib.bib26),[27](https://arxiv.org/html/2609.00097#bib.bib27)\], we decouple retrieval fidelity from storage overhead by decomposing the K cache as𝒌→\{𝒌quant,𝒌res\}\\bm\{k\}\\to\\\{\\bm\{k\}^\{\\text\{quant\}\},\\bm\{k\}^\{\\text\{res\}\}\\\}\. Here,𝒌quant\\bm\{k\}^\{\\text\{quant\}\}represents 2\-bit quantized thumbnails utilized for high\-throughput similarity scanning, while𝒌res\\bm\{k\}^\{\\text\{res\}\}denotes 8\-bit residuals\.

To maximize information entropy within the limited bit\-width, we employ a symmetric zero\-free \(mid\-rise\) quantization scheme, which ensures that directional information is preserved even for small\-magnitude features \(details in Appendix[A](https://arxiv.org/html/2609.00097#A1)\)\.

This architecture ensures that the scanning phase operates at minimal arithmetic intensity, while the identified candidates are reconstructed to near\-FP16 precision for exact computation\. A formal concentration bound on the scanning error is provided in Appendix[B](https://arxiv.org/html/2609.00097#A2)\.

Algorithm 1FFD Decoding with Hybrid Quantization1:Input:Query

𝒒i\\bm\{q\}\_\{i\}, 2\-bit Keys

𝑲quant\\bm\{K\}^\{\\text\{quant\}\}, 8\-bit Residuals

𝑲res\\bm\{K\}^\{\\text\{res\}\}, Values

𝑽\\bm\{V\}, threshold

δ\\delta, sink indices

SglobalS\_\{\\text\{global\}\}, local indices

SlocalS\_\{\\text\{local\}\}\.

2:Output:Attention output

oto\_\{t\}
3:\# Pseudo\-max estimation

4:

sglobal←maxj∈Sglobal⁡\(𝒒i⊤⋅𝒌jquant\)s\_\{\\text\{global\}\}\\leftarrow\\max\_\{j\\in S\_\{\\text\{global\}\}\}\\left\(\\bm\{q\}\_\{i\}^\{\\top\}\\cdot\\bm\{k\}^\{\\text\{quant\}\}\_\{j\}\\right\)
5:

slocal←maxj∈Slocal⁡\(𝒒i⊤⋅𝒌jquant\)s\_\{\\text\{local\}\}\\leftarrow\\max\_\{j\\in S\_\{\\text\{local\}\}\}\\left\(\\bm\{q\}\_\{i\}^\{\\top\}\\cdot\\bm\{k\}^\{\\text\{quant\}\}\_\{j\}\\right\)
6:

m~i←max⁡\(sglobal,slocal\)\\tilde\{m\}\_\{i\}\\leftarrow\\max\\left\(s\_\{\\text\{global\}\},s\_\{\\text\{local\}\}\\right\)
7:

8:\# Top\-

δ\\deltaselection

9:

ℐselected←∅\\mathcal\{I\}\_\{\\text\{selected\}\}\\leftarrow\\emptyset
10:foreach block index

BBdo

11:

sapprox←𝒒i⋅𝑲quant​\[B\]s^\{\\text\{approx\}\}\\leftarrow\\bm\{q\}\_\{i\}\\cdot\\bm\{K\}^\{\\text\{quant\}\}\[B\]
12:if

max⁡\(sapprox\)≥m~i−δ\\max\(s^\{\\text\{approx\}\}\)\\geq\\tilde\{m\}\_\{i\}\-\\deltathen

13:

ℐselected←ℐselected∪B\\mathcal\{I\}\_\{\\text\{selected\}\}\\leftarrow\\mathcal\{I\}\_\{\\text\{selected\}\}\\cup B
14:endif

15:endfor

16:

17:\# Fine\-grained refinement

18:

logits←\[\]\\text\{logits\}\\leftarrow\[\]
19:foreach block index

BBdo

20:

𝑲full←Dequant​\(𝑲quant​\[B\],𝑲res​\[B\]\)\\bm\{K\}^\{\\text\{full\}\}\\leftarrow\\text\{Dequant\}\(\\bm\{K\}^\{\\text\{quant\}\}\[B\],\\bm\{K\}^\{\\text\{res\}\}\[B\]\)
21:

sfinal←𝒒i⋅𝑲fullds\_\{\\text\{final\}\}\\leftarrow\\dfrac\{\\bm\{q\}\_\{i\}\\cdot\\bm\{K\}^\{\\text\{full\}\}\}\{\\sqrt\{d\}\}
22:

logits\.append​\(sfinal\)\\text\{logits\}\.\\text\{append\}\(s\_\{\\text\{final\}\}\)
23:endfor

24:

ot←Softmax​\(logits\)⋅V⁡\[ℐselected\]o\_\{t\}\\leftarrow\\text\{Softmax\}\(\\text\{logits\}\)\\cdot V\[\\mathcal\{I\}\_\{\\text\{selected\}\}\]

### 3\.2Top\-𝜹\\bm\{\\delta\}Selection Criterion

Letsi​j=𝒒i⊤​𝒌jds\_\{ij\}=\\dfrac\{\\bm\{q\}\_\{i\}^\{\\top\}\\bm\{k\}\_\{j\}\}\{\\sqrt\{d\}\}be the pre\-softmax attention score between query vector𝒒i\\bm\{q\}\_\{i\}and key vector𝒌j\\bm\{k\}\_\{j\}\. Ideally, we aim to retain only those tokens whose attention weight contributes significantly to the distribution\. We define a retention condition based on the relative magnitudeδ\\deltacompared to the maximum attention scoremi=maxj⁡si​jm\_\{i\}=\\max\_\{j\}s\_\{ij\}, and thus only calculate the attention over the KV cache where

si​j≥mi−δ\.s\_\{ij\}\\geq m\_\{i\}\-\\delta\.\(1\)This additive threshold in the log\-space,δ\\delta, translates to a rigorous multiplicative bound in the probability space\. By exponentiating Eq\.[1](https://arxiv.org/html/2609.00097#S3.E1), we observe that a retained tokenjjmust satisfy

exp⁡\(si​j\)exp⁡\(mi\)≥e−δ\.\\frac\{\\exp\(s\_\{ij\}\)\}\{\\exp\(m\_\{i\}\)\}\\geq e^\{\-\\delta\}\.\(2\)This provides a clear physical interpretation thatδ\\deltacontrols the maximum allowable attention drop\-off\. For instance, settingδ=5\\delta=5guarantees that we discard only tokens whose contribution is less thane−5≈0\.67%e^\{\-5\}\\approx 0\.67\\%of the peak attention mass\. This allows our FFD to adaptively vary the retrieval budget based on the sharpness of the attention distribution, rather than a fixed top\-kkand top\-ppas shown in Fig\.[1](https://arxiv.org/html/2609.00097#S1.F1)\.

Implementing Eq\.[1](https://arxiv.org/html/2609.00097#S3.E1)strictly requires computing the global maxmim\_\{i\}, which incurs the same synchronization overhead as softmax\. Fortunately, thanks to the phenomenon that attention distributions are typically dominated by either local context or initial sink tokens\[[16](https://arxiv.org/html/2609.00097#bib.bib16)\], we introduce the pseudo\-max approximationm~i\\tilde\{m\}\_\{i\}by estimating the global maximum using only these accessible subsets, sink tokensSglobalS\_\{\\text\{global\}\}and local contextSlocalS\_\{\\text\{local\}\}, as follows\.

m~i=max⁡\(maxt∈Sglobal⁡si​t,maxt∈Slocal⁡si​t\)\.\\tilde\{m\}\_\{i\}=\\max\{\\left\(\\max\_\{t\\in S\_\{\\text\{global\}\}\}s\_\{it\},\\max\_\{t\\in S\_\{\\text\{local\}\}\}s\_\{it\}\\right\)\}\.\(3\)This approximation allows the thresholdm~i−δ\\tilde\{m\}\_\{i\}\-\\deltato be computed using only locally available data, effectively bypassing the global reduction bottleneck and enabling independent parallel processing of KV blocks\.

### 3\.3Kernel Optimization

We implement FFD as a cooperative pipeline comprising three specialized Triton kernels, illustrated in Fig\.[2](https://arxiv.org/html/2609.00097#S1.F2)\. The execution flow proceeds as follows:

1. 1\.Pseudo\-max estimation\. A lightweight kernel computes the pseudo\-maxm~\\tilde\{m\}using only the sink and local tokens\.
2. 2\.Top\-δ\\bm\{\\delta\}selection\. The main kernel loads 2\-bit keys in streaming chunks\. It computes tentative scores and compares them againstm~−δ\\tilde\{m\}\-\\delta\.
3. 3\.Fine\-grained refinement\. If a block passes the filter, the kernel loads the corresponding 8\-bit residual keys and values\.

The add\-on computation in the refinement stage is shown as follows and ensures that for selected tokens, the attention score is computed with near\-FP16 precision\.

sfinal=𝒒⊤​𝒌quant\+𝒒⊤​𝒌resds\_\{\\text\{final\}\}=\\frac\{\\bm\{q\}^\{\\top\}\\bm\{k\}^\{\\text\{quant\}\}\+\\bm\{q\}^\{\\top\}\\bm\{k\}^\{\\text\{res\}\}\}\{\\sqrt\{d\}\}\(4\)This two\-term form is semantically equivalent to theDequantstep in Algorithm[1](https://arxiv.org/html/2609.00097#alg1)but corresponds more closely to the actual kernel implementation, where the two dot products are accumulated without materializing a full\-precision key tensor\.

To further minimize latency, particularly for decoding with small batch sizes where CPU launch overhead dominates, we implement a full\-chain CUDA graph strategy\. Unlike standard implementations that only capture the attention kernel, we capture the entire decoding step, including MLP, RMSNorm, and residual connections\.

However, standard CUDA Graphs are static and incompatible with the dynamic sequence growth of KV caches\. We address this with two innovations:

- •Block\-wise JIT capture\. We introduce a dynamic graph capture mechanism that triggers re\-capture only when the number of full KV blocks changes\. This amortizes compilation cost over the block size\.
- •Graph\-friendly cache\. We replace standard Python slicing \(which triggers CPU\-GPU synchronization\) with tensor\-based indexing kernels, ensuring the entire cache update process remains device\-side and graph\-capturable\.

This hybrid execution model runs 99% of steps within a CUDA Graph, falling back to eager mode only at block boundaries for memory allocation\.

## 4Experiments

### 4\.1Setup

DeviceWe conduct evaluations on two distinct GPU architectures, NVIDIA H100, representing datacenter\-grade hardware with high memory bandwidth, and NVIDIA GeForce RTX 4090, representing consumer\-grade hardware\. All experiments are implemented using PyTorch 2\.8 and Triton 3\.4 under CUDA 12\.8\.

BaselinesWe compare FFD with full attention accelerated by FlashAttention\-2\[[6](https://arxiv.org/html/2609.00097#bib.bib6)\], Quest\[[8](https://arxiv.org/html/2609.00097#bib.bib8)\], KIVI\[[24](https://arxiv.org/html/2609.00097#bib.bib24)\], and Twilight\[[10](https://arxiv.org/html/2609.00097#bib.bib10)\]\. Twilight does not provide fully open\-source official code at the time of writing; its results are based on our own faithful reproduction following the algorithm described in the original paper\. To ensure a fair system\-level comparison, we align the token budget as 16k and memory access patterns with hardware characteristics\. For Quest, we utilize the official page size ofP=16P=16\. Fortunately, under this configuration, the memory footprint of the indices among these methods is equivalent to that of our 2\-bit data for full scanning, which ensures a fair comparison\.

HyperparameterFor FFD, we evaluate sparsity thresholdsδ∈\{5,7\}\\delta\\in\\\{5,7\\\}\. These values are selected based on their theoretical implications for the probability mass:

- •δ=5\\delta=5implies pruning tokens whose attention weightexp⁡\(si​j\)\\exp\(s\_\{ij\}\)is less thane−δ≈1/148e^\{\-\\delta\}\\approx 1/148of the maximum score\. This represents an aggressive filtering strategy targeting the top∼1%\\sim 1\\%relevant tokens\.
- •δ=7\\delta=7corresponds to a threshold ofe−7≈1/1096e^\{\-7\}\\approx 1/1096, serving as a conservative setting that retains tokens with even∼0\.1%\\sim 0\.1\\%relative importance to prevent tail\-risk information loss\.

Evaluation

We structure our evaluation into two parts: Efficiency and Effectiveness\.

- •Efficiency:We evaluate system performance at two levels\. We useTrace\-Driven Kernel Microbenchmarks\(Section[4\.2](https://arxiv.org/html/2609.00097#S4.SS2)\) to measure operator latency, andEnd\-to\-End Generation\(Section[4\.3](https://arxiv.org/html/2609.00097#S4.SS3)\) to test the throughput of the full model\.
- •Effectiveness:We validate that our method preserves model quality\. We employ standard long\-context benchmarks, specifically LongBench and RULER \(Section[4\.4](https://arxiv.org/html/2609.00097#S4.SS4)\), to assess performance on downstream tasks\.

### 4\.2Kernel Efficiency

We benchmark the latency of our fused Triton kernel on a consumer GPU with sequence lengths ranging from 4K to 256K\. To minimize launch overheads in the low\-latency regime \(<1ms\), we utilize CUDA Graphs for kernel submission\. We compare against the standard PyTorchscaled\_dot\_product\_attention\(which utilizes FlashAttention\-2\)\.

Figure 3:Kernel Latency vs\. Sequence Length \(RTX 4090\)\. Layer\-averaged execution time for FlashAttention\-2 and FFD variants \(batch=1\)\. FFD\-δ=5\\delta=5andδ=7\\delta=7achieve average speedups of7\.33×7\.33\\timesand6\.18×6\.18\\timesrespectively, with a peak reduction of11\.63×11\.63\\times\. Error bands denote the min–max range across layers\.As shown in Figure[3](https://arxiv.org/html/2609.00097#S4.F3), our FFD kernel demonstrates linear scaling with sequence length but with a significantly lower slope; error bands indicate the layer\-wise min–max range for FFD \(δ=5\\delta=5,δ=7\\delta=7\) and FlashAttention\-2\. At 256K context length, FlashAttention\-2 averages 1\.12 ms, while FFD \(δ=5\\delta=5\) and FFD \(δ=7\\delta=7\) average 0\.17 ms and 0\.21 ms, corresponding to 6\.58×\\timesand 5\.33×\\timesspeedups\. This dramatic acceleration is achieved by combining: 1\.2\-bit IO:Reducing memory access from 16 bits to 2 bits for the scanning phase\. 2\.High Sparsity:Theδ=5\\delta=5threshold effectively filters a lot of blocks on real Llama\-3\.1 data\. 3\.CUDA Graphs:Eliminating CPU launch overhead, which is critical when the kernel runtime is sub\-100μ\\mus\.

### 4\.3End\-to\-End Throughput

Beyond kernel\-level microbenchmarks, we evaluate the impact of FFD on end\-to\-end token generation throughput\. Figure[4](https://arxiv.org/html/2609.00097#S4.F4)compares the maximum throughput \(tokens/sec\) achievable by FFD, Quest, FlashAttention\-2, and KIVI across varying context lengths on both NVIDIA RTX 4090 and H100 GPUs\.

On the consumer\-grade RTX 4090, FFD consistently achieves the highest throughput, reaching 61\.0 tokens/s at 4K and maintaining 51\.8 tokens/s at 16K context\. This represents a speedup of up to 2\.37×\\timesover FlashAttention\-2 \(at 16K\)\. Quest also performs well but lags behind FFD \(e\.g\., 47\.8 vs 51\.8 at 16K\)\.

We further scale our evaluation to the datacenter\-grade H100\. FFD demonstrates strong scalability, reaching 110\.6 tokens/s at 4K context and maintaining 87\.0 tokens/s at 16K\. This corresponds to a 1\.96×\\timesspeedup over FlashAttention\-2 \(44\.5 tokens/s\) at 16K context\. Consistent with 4090 results, KIVI continues to underperform dense attention on H100 in this low\-batch regime \(e\.g\., 22\.4 vs 44\.5 at 16K\), highlighting the significant cost of dequantization\.

Figure 4:End\-to\-End Generation Throughput Comparison on RTX 4090 and H100 \(16K Context\)\. FFD outperforms FlashAttention\-2, Quest, and KIVI on both platforms\. On H100, FFD achieves up to 1\.96×\\timesspeedup over the dense baseline and surpasses Quest \(e\.g\., 87\.0 vs 73\.5 tokens/s at 16K\)\. KIVI suffers from dequantization overheads in this single\-batch setting\.Figure 5:Distribution of the approximation error \(mi−m~im\_\{i\}\-\\tilde\{m\}\_\{i\}\) by layer\. The gap is effectively zero for most of conditions, confirming that the maximum attention score is dominated by sinks and local tokens\. This verifies that our Pseudo\-Max approximation serves as a robust safe relaxation\.
### 4\.4Downstream Performance

Table 1:RULER Benchmark Results \(32k Context\)\. Comparison of Llama\-3\.1\-8B\-Instruct Base, StreamingLLM, KIVI \(2\-bit, group=32\), Quest \(ratio=0\.5\), Twilight, and FFD\. We report the aggregated score \(0–100\) for each task category\.Boldindicates the best performance among sparse methods \(excluding Base\)\. Abbreviations: SK \(SingleKey\), MK \(MultiKey\), MQ \(MultiQuery\), MV \(MultiValue\), VT \(Variable Tracking\), CWE \(Common Words Extraction\), FWE \(Frequent Words Extraction\)\. Avg is the simple average over all 13 RULER subtasks\.MethodSK\-1SK\-2SK\-3MK\-1MK\-2MK\-3MQMVVTSQuADHotpotCWEFWEAvgBase100\.0100\.0100\.098\.0100\.099\.098\.598\.599\.667\.056\.067\.893\.090\.6StreamingLLM3\.00\.00\.02\.02\.01\.00\.00\.00\.011\.09\.00\.251\.06\.1KIVI100\.099\.098\.093\.093\.048\.086\.886\.099\.263\.054\.056\.493\.382\.3Quest \(0\.5\)100\.0100\.0100\.096\.064\.06\.098\.396\.399\.465\.049\.015\.871\.073\.9Twilight100\.088\.096\.088\.076\.037\.056\.560\.398\.260\.049\.012\.786\.369\.8FFD \(δ=5\\delta=5\)100\.099\.099\.096\.097\.078\.098\.397\.896\.865\.052\.061\.093\.087\.1FFD \(δ=7\\delta=7\)100\.0100\.099\.096\.098\.095\.099\.598\.598\.466\.052\.067\.092\.789\.4

Table 2:LongBench Results\. Comparison of Base, StreamingLLM, KIVI \(2\-bit\), Quest, Twilight, and FFD\. We report the average score \(%\) for each category\.Boldindicates the best performance among sparse methods \(excluding Base\)\. Twilight results are based on our own reproduction\.MethodS\-DocM\-DocSummFew\-ShotSynCodeAvgBase24\.0215\.2416\.5744\.1032\.7024\.6826\.22StreamingLLM3\.443\.125\.9222\.490\.0222\.559\.59KIVI23\.4614\.4415\.4644\.1730\.4824\.9425\.49Quest23\.1314\.3315\.2943\.6930\.2829\.3626\.01Twilight23\.2515\.5316\.9743\.7332\.3725\.2326\.18FFD \(δ=5\\delta=5\)23\.3615\.3516\.4044\.0930\.0425\.9425\.86FFD \(δ=7\\delta=7\)24\.0015\.7816\.1144\.1931\.8326\.2126\.35To validate the effectiveness of FFD in real\-world long\-context scenarios, we evaluated our method on the RULER benchmark\[[17](https://arxiv.org/html/2609.00097#bib.bib17)\]with a context length of 32K\. Table[1](https://arxiv.org/html/2609.00097#S4.T1)presents the results comparing FFD against the dense Llama\-3\.1\-8B\-Instruct baseline\.

Our method maintains high performance across all categories\. Notably, on the challenging “Needle In A Haystack" \(NIAH\) Single and Multi\-Key retrieval tasks, FFD achieves near\-perfect scores \(e\.g\., 100\.0 on SK\-1/SK\-2, 99\.0 on SK\-3\), matching the dense baseline\. Even on complex reasoning tasks like Variable Tracking \(VT\), the performance degradation is minimal \(98\.4 vs 99\.6\), demonstrating that our 2\-bit high\-fidelity filter successfully preserves the critical information required for multi\-hop reasoning\.

We further evaluate FFD on LongBench\[[18](https://arxiv.org/html/2609.00097#bib.bib18)\], a comprehensive benchmark for long\-context understanding containing 21 datasets across 6 categories\. Table[2](https://arxiv.org/html/2609.00097#S4.T2)reports the category\-level results across all compared methods\.

FFD demonstrates strong performance across all six categories\. With an overall average of 26\.35, FFD \(δ=7\\delta=7\) achieves the highest aggregate score among all methods, reinforcing our hypothesis that 2\-bit content\-aware scanning preserves the semantic fidelity required for diverse long\-context understanding tasks\.

Comparison with Top\-ppSelection\.To provide a direct comparison with the Top\-ppselection paradigm, we additionally benchmark Twilight\[[10](https://arxiv.org/html/2609.00097#bib.bib10)\]under its original operating point \(budget=8192,p=0\.85p=0\.85\)\. As shown in Tables[1](https://arxiv.org/html/2609.00097#S4.T1)and[2](https://arxiv.org/html/2609.00097#S4.T2), FFD consistently outperforms Twilight by substantial margins on both benchmarks, while maintaining a clear advantage over Quest\. The multi\-peak retrieval tasks \(MK\-2, MK\-3\) particularly highlight the weakness of fixed\-budget selectors: Quest drops sharply on MK\-3 \(6\.0\), whereas FFD with top\-δ\\deltaadapts to the distributed attention pattern \(95\.0\)\. We omit throughput comparison with Twilight because its official implementation is not fully open\-source, making a fair system\-level speed comparison infeasible\.

### 4\.5Ablation Study

To disentangle the contributions of our three co\-design axes—scan precision \(FP16 vs\. 2\-bit\), selection rule \(top\-kkvs\. top\-δ\\delta\), and execution path \(split vs\. fused\+fullgraph\)—we conduct a controlled ablation study under matched conditions\. All variants are evaluated with a 16K prefill and 128 decode steps on an RTX 4090 \(batch size 1\)\. The full ablation table and detailed analysis are presented in Appendix[G](https://arxiv.org/html/2609.00097#A7)\. In summary, the fused 2\-bit top\-δ\\deltaconfiguration \(FFD\) achieves 53\.96 tok/s with RULER AVG of 90\.40 and LongBench AVG of 25\.90—a synergistic 2\.2×\\timesthroughput gain over the split\-execution counterpart—demonstrating that the algorithmic and system designs are inseparably linked\.

Selection\-Rule Overhead\.To isolate the algorithmic cost of each selection rule, we calibrate top\-kkand top\-ppto match the average keep ratio of top\-δ\\delta\(∼\\sim27\.3%\)\. Table[3](https://arxiv.org/html/2609.00097#S4.T3)reports the per\-head latency breakdown under matched sparsity\. At nearly identical keep ratios, top\-δ\\deltais2×2\\timesfaster than top\-kkand25×25\\timesfaster than top\-pp, confirming that the top\-δ\\deltaformulation eliminates the global synchronization bottleneck inherent to cumulative probability methods\.

Table 3:Per\-head selection latency at matched keep ratio \(∼\\sim27\.3%\)\.Selection RuleKeep RatioLatency \(ms/head\)top\-δ\\delta0\.2730\.0044top\-kk0\.2750\.0090top\-pp0\.2730\.1115KIVI2 \(dense\)1\.0000\.0186Generalization to Qwen2\.5\.We further validate FFD on Qwen2\.5\-7B\-Instruct to assess generalization beyond the Llama architecture\. Under the sameδ=7\\delta=7configuration, FFD achieves 85\.90 RULER AVG and 30\.11 LongBench AVG, outperforming KIVI \(72\.01 / 28\.34\), Quest \(71\.93 / 29\.49\), and StreamingLLM \(16\.24 / 21\.45\)\. These results confirm that FFD’s content\-aware scanning and top\-δ\\deltaselection generalize effectively across model families without architecture\-specific tuning\.

## 5Discussion

### 5\.1Retrieval Fidelity Analysis

We evaluate the “quality" of the retrieved tokens by measuring two metrics:

1. 1\.Recall:The cosine similarity between the sparse attention output and the dense \(exact\) output\.
2. 2\.LSE Error:The Max Absolute Error of the Log\-Sum\-Exp \(LSE\) value, which proxies the stability of the Softmax distribution\.

Figure 6:Quality vs\. IO Budget on Llama\. FFD maintains lower LSE error while achieving higher recall than Quest under the same retrieval budget, showing that high\-fidelity scanning preserves distributional accuracy and hit rate\.FFD achieves significantly higher recall than Quest across all sparsity ratios, confirming the “Thumbnail vs Bounding Box" hypothesis: 2\-bit quantization preserves the geometric directionality of keys, enabling more precise filtering than min/max bounds\. FFD also minimizes the error in the normalization term \(LSE\), which is critical for preventing “collapse" in the attention distribution\.

Figure 7:Layer\-wise Correlation Trends\. FFD achieves consistently higher Spearman correlations across all layers compared to Quest, indicating better preservation of the attention distribution’s structural properties\.Figure[7](https://arxiv.org/html/2609.00097#S5.F7)shows that FFD consistently maintains higher Spearman correlation than Quest across all layers, indicating that quantization\-based scanning better preserves the relative ordering of attention scores, whereas the bounding\-box approximation suffers from degradation\.

### 5\.2Pseudo\-Max Robustness

FFD avoids a global reduction by estimating the threshold using a Pseudo\-Max computed from sink and local tokens\. Figure[5](https://arxiv.org/html/2609.00097#S4.F5)reports the empirical gap between the global maximum and the Pseudo\-Max in a simulation that mirrors the sink/local bias\.

We analyze the implications of the approximation gap wherem~i<mi\\tilde\{m\}\_\{i\}<m\_\{i\}\. As shown in Figure[5](https://arxiv.org/html/2609.00097#S4.F5), such gaps are rare and numerically negligible\. Crucially, even when they occur, this underestimation represents a fail\-safe mechanism\. Recall our filtering condition:si​j≥m~i−δs\_\{ij\}\\geq\\tilde\{m\}\_\{i\}\-\\delta\. Sincem~i≤mi\\tilde\{m\}\_\{i\}\\leq m\_\{i\}by definition, it follows that our threshold is lower than the ideal threshold:m~i−δ≤mi−δ\\tilde\{m\}\_\{i\}\-\\delta\\leq m\_\{i\}\-\\delta\. Consequently, any token that would be selected by the global max isguaranteedto be selected by the Pseudo\-Max\. In these rare “failure" cases, FFD automatically falls back to a more conservative retrieval policy, preserving slightly more tokens\. This structural asymmetry ensures that approximation errors result only in a minor increase in I/O budget \(efficiency penalty\), rather than the loss of critical information \(accuracy penalty\)\.

Figure 8:Layer\-wise sparsity distribution on LongBench \(Llama\-3\.1\-8B\)\. FFD achieves significant computational savings, maintaining average sparsity levels of 82% and 73% underδ=5\\delta=5andδ=7\\delta=7configurations respectively\.
### 5\.3Sparsity Analysis

FFD exhibits layer\-wise adaptive sparsity governed byδ\\delta, with higher sparsity at smallerδ\\delta\. As shown in Figure[8](https://arxiv.org/html/2609.00097#S5.F8), FFD maintains average sparsity of 82% and 73% underδ=5\\delta=5andδ=7\\delta=7respectively—operating with substantially lower compute budget than dense attention—while Tables[1](https://arxiv.org/html/2609.00097#S4.T1)and[2](https://arxiv.org/html/2609.00097#S4.T2)confirm no accuracy sacrifice\. By mitigating redundant I/O overhead, FFD achieves superior inference speed compared to other approaches\.

### 5\.4Limitations

We identify several limitations that point to future work\. First, while our head\-wise variation analysis \(Appendix[D](https://arxiv.org/html/2609.00097#A4)\) indicates that FFD adapts naturally to diverse per\-head sparsity patterns, the characterization is based on controlled single\-sample statistics and has not been systematically validated across diverse task distributions\. Second, the failure\-case profiling in Appendix[E](https://arxiv.org/html/2609.00097#A5)uses proxy metrics \(top\-1/top\-2 gap and score standard deviation\) rather than direct end\-to\-end task\-failure attribution; a more comprehensive causal attribution remains open\. Third, FFD has been validated on Llama and Qwen architectures employing grouped\-query attention \(GQA\); its applicability to MLA\-style architectures\[[12](https://arxiv.org/html/2609.00097#bib.bib12)\]with fundamentally different key/value factorization has not yet been assessed\. Finally, while our component\-wise ablation isolates the major contributors to the end\-to\-end speedup, a fine\-grained latency breakdown within the fused execution path is deferred to future work\.

## 6Conclusion

We presented FFD, a hardware\-algorithm co\-design that rethinks sparse attention as geometric filtering rather than metadata indexing\. By utilizing 2\-bit quantization for high\-fidelity scanning and attention sinks for adaptive thresholding, we break the dependency on rigid top\-kkbudgets\. Our results suggest that future long\-context inference should prioritize “compute\-for\-IO” trade\-offs—spending cheap FLOPs on low\-bit scanning to save expensive HBM bandwidth\.

## Acknowledgments

This work was supported by the Science and Technology Commission of Shanghai Municipality \(No\. 25DZ3100402\)\.

## Impact Statement

This work improves the decoding efficiency of large language models through algorithmic and kernel optimization\. It does not involve human subjects, personal information, or sensitive content, and we identify no extra ethical concerns specific to our work\.

## References

- \[1\]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al\.Gpt\-4 technical report\.*arXiv preprint arXiv:2303\.08774*, 2023\.
- \[2\]Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al\.Gemini 1\.5: Unlocking multimodal understanding across millions of tokens of context\.*arXiv preprint arXiv:2403\.05530*, 2024\.
- \[3\]Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al\.The llama 3 herd of models\.*arXiv e\-prints*, pages arXiv–2407, 2024\.
- \[4\]Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, et al\.Thus spake long\-context large language model\.*arXiv preprint arXiv:2502\.17129*, 2025a\.
- \[5\]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\.Attention is all you need\.*Advances in neural information processing systems*, 30, 2017\.
- \[6\]Tri Dao\.Flashattention\-2: Faster attention with better parallelism and work partitioning\.*arXiv preprint arXiv:2307\.08691*, 2023\.
- \[7\]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E\. Gonzalez, Hao Zhang, and Ion Stoica\.Efficient memory management for large language model serving with pagedattention, 2023\.URL[https://arxiv\.org/abs/2309\.06180](https://arxiv.org/abs/2309.06180)\.
- \[8\]Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han\.Quest: Query\-aware sparsity for efficient long\-context llm inference\.*arXiv preprint arXiv:2406\.10774*, 2024\.
- \[9\]Luka Ribar, Ivan Chelombiev, Luke Hudlass\-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr\.Sparq attention: Bandwidth\-efficient llm inference\.*arXiv preprint arXiv:2312\.04985*, 2023\.
- \[10\]Chaofan Lin, Jiaming Tang, Shuo Yang, Hanshuo Wang, Tian Tang, Boyu Tian, Ion Stoica, Song Han, and Mingyu Gao\.Twilight: Adaptive attention sparsity with hierarchical top\-pppruning\.*arXiv preprint arXiv:2502\.02770*, 2025\.
- \[11\]Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, et al\.Native sparse attention: Hardware\-aligned and natively trainable sparse attention\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 23078–23097, 2025\.
- \[12\]Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al\.Deepseek\-v3\.2: Pushing the frontier of open large language models\.*arXiv preprint arXiv:2512\.02556*, 2025b\.
- \[13\]Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun\.Infllm: Training\-free long\-context extrapolation for llms with an efficient context memory\.*Advances in Neural Information Processing Systems*, 37:119638–119661, 2024\.
- \[14\]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al\.H2o: Heavy\-hitter oracle for efficient generative inference of large language models\.*Advances in Neural Information Processing Systems*, 36:34661–34710, 2023\.
- \[15\]Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao\.Model tells you what to discard: Adaptive kv cache compression for llms\.*arXiv preprint arXiv:2310\.01801*, 2023\.
- \[16\]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\.Efficient streaming language models with attention sinks\.*arXiv preprint arXiv:2309\.17453*, 2023\.
- \[17\]Cheng\-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg\.Ruler: What’s the real context size of your long\-context language models?*arXiv preprint arXiv:2404\.06654*, 2024\.
- \[18\]Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al\.Longbench: A bilingual, multitask benchmark for long context understanding\.In*Proceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\)*, pages 3119–3137, 2024\.
- \[19\]Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao\.Flashattention\-3: Fast and accurate attention with asynchrony and low\-precision\.*Advances in Neural Information Processing Systems*, 37:68658–68685, 2024\.
- \[20\]Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang\.Flashdecoding\+\+: Faster large language model inference on gpus\.*arXiv preprint arXiv:2311\.01282*, 2023\.
- \[21\]Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava\.Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time\.*Advances in Neural Information Processing Systems*, 36:52342–52364, 2023\.
- \[22\]Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen\.Snapkv: Llm knows what you are looking for before generation\.*Advances in Neural Information Processing Systems*, 37:22947–22970, 2024\.
- \[23\]Xiaodong Ji, Hailin Zhang, Fangcheng Fu, and Bin Cui\.Sale: Low\-bit estimation for efficient sparse attention in long\-context llm prefilling, 2025\.URL[https://arxiv\.org/abs/2505\.24179](https://arxiv.org/abs/2505.24179)\.
- \[24\]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu\.Kivi: A tuning\-free asymmetric 2bit quantization for kv cache\.*arXiv preprint arXiv:2402\.02750*, 2024\.
- \[25\]Yilong Zhao, Chien\-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci\.Atom: Low\-bit quantization for efficient and accurate llm serving\.*Proceedings of Machine Learning and Systems*, 6:196–209, 2024\.
- \[26\]Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami\.Kvquant: Towards 10 million context length llm inference with kv cache quantization\.*Advances in Neural Information Processing Systems*, 37:1270–1303, 2024\.
- \[27\]June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee\.No token left behind: Reliable kv cache compression via importance\-aware mixed precision quantization\.*arXiv preprint arXiv:2402\.18096*, 2024\.

## Appendix ADetails of Quantization Implementation

FFD employs a Symmetric Mid\-rise Uniform Quantization scheme\. Unlike standard integer mapping \(e\.g\.,\{−1,0,1\}\\\{\-1,0,1\\\}\) which wastes quantization bins on a zero state, our approach utilizes a strictly non\-zero grid to maximize the information capacity of the limited 2\-bit budget\.

### A\.1Formulation

We define a 4\-level quantization codebook𝒞\\mathcal\{C\}centered around zero but excluding zero itself:

𝒞=\{−1\.5,−0\.5,\+0\.5,\+1\.5\}\\mathcal\{C\}=\\\{\-1\.5,\-0\.5,\+0\.5,\+1\.5\\\}\(5\)For a given input channel vector𝐱\\mathbf\{x\}, we compute the scaling factorssbased on the absolute maximum value to ensure coverage:

s=max⁡\(\|𝐱\|\)1\.5s=\\frac\{\\max\(\|\\mathbf\{x\}\|\)\}\{1\.5\}\(6\)The quantized representation𝐱q\\mathbf\{x\}\_\{q\}is obtained by projecting the scaled input onto the nearest neighbor in the codebook:

𝐱q=arg⁡minc∈𝒞​\|𝐱s−c\|\\mathbf\{x\}\_\{q\}=\\underset\{c\\in\\mathcal\{C\}\}\{\\arg\\min\}\\left\|\\frac\{\\mathbf\{x\}\}\{s\}\-c\\right\|\(7\)The reconstructed approximation is then𝐱^=𝐱q⋅s\\hat\{\\mathbf\{x\}\}=\\mathbf\{x\}\_\{q\}\\cdot s\.

### A\.2Design Rationale

We explicitly choose a mid\-rise quantizer \(no zero point\) over a mid\-tread quantizer to align with the distributional properties of modern LLMs\. Key vectors, particularly after Rotary Positional Embeddings \(RoPE\), are typically dense with magnitude distributed across dimensions\. A standard zero\-inclusive grid effectively utilizes only 3 states \(log2⁡3≈1\.58\\log\_\{2\}3\\approx 1\.58bits\), wasting roughly 20% of the representational capacity of a 2\-bit system\. By forcing every dimension to take a non\-zero value, our mid\-rise scheme ensures full bit utilization and prevents “information collapse", preserving the directional fidelity required for high\-dimensional dot\-product scanning\.

## Appendix BError Analysis of Quantized Scanning

In this section, we provide a theoretical bound on the approximation error introduced by the 2\-bit quantization during the scanning phase\. We demonstrate that as the head dimensionddincreases, the probability of the approximation error exceeding the selection safety margin decays exponentially\.

### B\.1Quantization Error Bound

Let𝐪,𝐤∈ℝd\\mathbf\{q\},\\mathbf\{k\}\\in\\mathbb\{R\}^\{d\}denote the query and key vectors of dimensiondd\(head dimension\)\. In our FFD framework, the exact pre\-softmax attention score is defined ass=𝐪⊤​𝐤/ds=\\mathbf\{q\}^\{\\top\}\\mathbf\{k\}/\\sqrt\{d\}\. The scanner approximates this using the 2\-bit quantized key𝐤^=𝐤\+ϵ\\hat\{\\mathbf\{k\}\}=\\mathbf\{k\}\+\\mathbf\{\\epsilon\}, whereϵ∈ℝd\\mathbf\{\\epsilon\}\\in\\mathbb\{R\}^\{d\}is the quantization noise vector\. The approximate score is given by:

s^=𝐪⊤​\(𝐤\+ϵ\)d=s\+𝐪⊤​ϵd\\hat\{s\}=\\frac\{\\mathbf\{q\}^\{\\top\}\(\\mathbf\{k\}\+\\mathbf\{\\epsilon\}\)\}\{\\sqrt\{d\}\}=s\+\\frac\{\\mathbf\{q\}^\{\\top\}\\mathbf\{\\epsilon\}\}\{\\sqrt\{d\}\}\(8\)The approximation error is thusea​p​p​r​o​x=𝐪⊤​ϵde\_\{approx\}=\\frac\{\\mathbf\{q\}^\{\\top\}\\mathbf\{\\epsilon\}\}\{\\sqrt\{d\}\}\.

###### Assumption B\.1\(Sub\-Gaussian Quantization Noise\)\.

While standard quantization theory models noise as uniform for high bit\-depths, for 2\-bit quantization, we adopt a more robust assumption: the noise componentsϵi\\epsilon\_\{i\}are independent, mean\-zero, andσ2\\sigma^\{2\}\-sub\-Gaussian\. This means𝔼⁡\[exp⁡\(t​ϵi\)\]≤exp⁡\(σ2​t2/2\)\\mathbb\{E\}\[\\exp\(t\\epsilon\_\{i\}\)\]\\leq\\exp\(\\sigma^\{2\}t^\{2\}/2\)for allt∈ℝt\\in\\mathbb\{R\}\. For a symmetric 4\-level quantizer with step sizeΔ\\Delta, the variance and sub\-Gaussian parameter are determined by the bin widthΔ≈max⁡\(\|𝐤\|\)/1\.5\\Delta\\approx\\max\(\|\\mathbf\{k\}\|\)/1\.5\.

###### Theorem B\.2\(Concentration of Scanning Error\)\.

Let the query vector𝐪\\mathbf\{q\}have a boundedℓ2\\ell\_\{2\}\-norm such that‖𝐪‖2≤α​d\\\|\\mathbf\{q\}\\\|\_\{2\}\\leq\\alpha\\sqrt\{d\}, whereα\\alphais a constant determined by the model’s normalization layers \(e\.g\., RMSNorm\)\. Under[AssumptionB\.1](https://arxiv.org/html/2609.00097#A2.Thmtheorem1), the probability that the magnitude of the scanning error\|ea​p​p​r​o​x\|\|e\_\{approx\}\|exceeds a deviationλ\\lambdais bounded by:

P⁡\(\|ea​p​p​r​o​x\|≥λ\)≤2​exp⁡\(−λ22​σ2​α2\)P\(\|e\_\{approx\}\|\\geq\\lambda\)\\leq 2\\exp\\left\(\-\\frac\{\\lambda^\{2\}\}\{2\\sigma^\{2\}\\alpha^\{2\}\}\\right\)\(9\)

###### Proof\.

The error termea​p​p​r​o​xe\_\{approx\}is a linear combination of independent sub\-Gaussian random variables:

ea​p​p​r​o​x=∑i=1d\(qid\)​ϵie\_\{approx\}=\\sum\_\{i=1\}^\{d\}\\left\(\\frac\{q\_\{i\}\}\{\\sqrt\{d\}\}\\right\)\\epsilon\_\{i\}\(10\)The sum of independent sub\-Gaussian variablesX=∑wi​ϵiX=\\sum w\_\{i\}\\epsilon\_\{i\}is itself sub\-Gaussian with parameter‖𝐰‖22​σ2\\\|\\mathbf\{w\}\\\|\_\{2\}^\{2\}\\sigma^\{2\}\. Here, the weights arewi=qi/dw\_\{i\}=q\_\{i\}/\\sqrt\{d\}\. The effective sub\-Gaussian parameter forea​p​p​r​o​xe\_\{approx\}is:

σe​f​f2=∑i=1d\(qid\)2​σ2=σ2d​‖𝐪‖22\\sigma\_\{eff\}^\{2\}=\\sum\_\{i=1\}^\{d\}\\left\(\\frac\{q\_\{i\}\}\{\\sqrt\{d\}\}\\right\)^\{2\}\\sigma^\{2\}=\\frac\{\\sigma^\{2\}\}\{d\}\\\|\\mathbf\{q\}\\\|\_\{2\}^\{2\}\(11\)Using the‖𝐪‖2≤α​d\\\|\\mathbf\{q\}\\\|\_\{2\}\\leq\\alpha\\sqrt\{d\}bound \(where typicallyα≈1\\alpha\\approx 1in normalized LLMs\), we haveσe​f​f2≤σ2​α2\\sigma\_\{eff\}^\{2\}\\leq\\sigma^\{2\}\\alpha^\{2\}\. Applying the general Hoeffding\-type bound for sub\-Gaussian variables:

P⁡\(\|ea​p​p​r​o​x\|≥λ\)≤2​exp⁡\(−λ22​σe​f​f2\)≤2​exp⁡\(−λ22​σ2​α2\)∎P\(\|e\_\{approx\}\|\\geq\\lambda\)\\leq 2\\exp\\left\(\-\\frac\{\\lambda^\{2\}\}\{2\\sigma\_\{eff\}^\{2\}\}\\right\)\\leq 2\\exp\\left\(\-\\frac\{\\lambda^\{2\}\}\{2\\sigma^\{2\}\\alpha^\{2\}\}\\right\)\\qed\(12\)

### B\.2Reliability of Top\-δ\\deltaSelection

Based on the error bound derived in[TheoremB\.2](https://arxiv.org/html/2609.00097#A2.Thmtheorem2), we can now formally justify the FFD selection strategy\. The primary goal is to minimize the False Negative Rate \(FNR\)—missing a token that is truly significant\.

Letδi​d​e​a​l\\delta\_\{ideal\}be the theoretically optimal threshold in log\-space \(Eq\.[1](https://arxiv.org/html/2609.00097#S3.E1)\)\. In FFD, we set the operational thresholdδ=δi​d​e​a​l\+λm​a​r​g​i​n\\delta=\\delta\_\{ideal\}\+\\lambda\_\{margin\}, whereλm​a​r​g​i​n\\lambda\_\{margin\}acts as a safety buffer\. A False Negative occurs if a token satisfiess≥m−δi​d​e​a​ls\\geq m\-\\delta\_\{ideal\}but is rejected by the scanner:s^<m~−δ\\hat\{s\}<\\tilde\{m\}\-\\delta\.

Givens^=s\+ea​p​p​r​o​x\\hat\{s\}=s\+e\_\{approx\}and our definition ofδ\\delta, the rejection condition implies:

s\+ea​p​p​r​o​x<m~−\(δi​d​e​a​l\+λm​a​r​g​i​n\)s\+e\_\{approx\}<\\tilde\{m\}\-\(\\delta\_\{ideal\}\+\\lambda\_\{margin\}\)\(13\)Usings≥m−δi​d​e​a​ls\\geq m\-\\delta\_\{ideal\}and the fact that the Pseudo\-Maxm~\\tilde\{m\}is a lower bound on the true maxmm\(m~≤m\\tilde\{m\}\\leq m\), we obtain:

ea​p​p​r​o​x<\(m~−m\)−λm​a​r​g​i​n≤−λm​a​r​g​i​ne\_\{approx\}<\(\\tilde\{m\}\-m\)\-\\lambda\_\{margin\}\\leq\-\\lambda\_\{margin\}\(14\)Thus, a False Negative requires the quantization error to exceed the safety margin in the negative direction\. According to[TheoremB\.2](https://arxiv.org/html/2609.00097#A2.Thmtheorem2), this probability is:

P⁡\(False Negative\)≤exp⁡\(−λm​a​r​g​i​n22​σ2​α2\)P\(\\text\{False Negative\}\)\\leq\\exp\\left\(\-\\frac\{\\lambda\_\{margin\}^\{2\}\}\{2\\sigma^\{2\}\\alpha^\{2\}\}\\right\)\(15\)
This result provides a strong theoretical guarantee: the probability of missing a critical token decays exponentially with the square of the safety margin\. In practice, settingδ=5\\delta=5or77provides a sufficiently largeλm​a​r​g​i​n\\lambda\_\{margin\}relative to the noise scaleσ\\sigma, explaining the near\-perfect recall observed in our LongBench and RULER experiments despite the 2\-bit representation\.

## Appendix CBandwidth Proportionality of Quantization Depth

A central design choice in FFD is the 2\-bit thumbnail with 8\-bit residual \(“2\+8”\) configuration\. To justify this choice empirically for the decode setting, we benchmark the decoding kernel latency under three quantization configurations—2\+8, 4\+8, and 8\+8—using real text from the LongBench dataset on an RTX 4090\. Table[4](https://arxiv.org/html/2609.00097#A3.T4)reports the results\.

Table 4:Decode kernel latency under different quantization depths \(RTX 4090, 16k context\)\.ConfigurationLatency \(ms/head\)Speedup vs\. FlashAttn2\+8 \(FFD\)0\.10510\.65×\\times4\+80\.2015\.60×\\times8\+80\.4062\.82×\\timesThe kernel speed is nearly proportional to the scan bandwidth: 2\-bit scanning incurs one\-quarter the I/O of 8\-bit and half that of 4\-bit\. This proportionality is a distinctive characteristic of the decode setting where memory bandwidth dominates; it does not necessarily hold in prefill, where computation is the primary bottleneck and 4\-bit may be a natural choice due to better hardware support for 4\-bit arithmetic\. This finding motivates the 2\+8 design as the bandwidth\-optimal configuration for long\-context decoding\.

## Appendix DHead\-Wise Variation under Fixed Delta

To assess whether FFD implicitly assumes uniform sparsity across attention heads, we conduct a controlled head\-wise analysis on a sample from LongBench withδ=5\\delta=5fixed\. Across10241024layer\-head pairs \(32 layers×\\times32 heads on Llama\-3\.1\-8B\), we measure the per\-head keep ratio and salient hit rate\. Table[5](https://arxiv.org/html/2609.00097#A4.T5)summarizes the results\.

Table 5:Head\-wise variation statistics \(δ=5\\delta=5, single LongBench sample, 1024 layer\-head pairs\)\.StatisticKeep Ratio \(%\)Salient Hit Rate \(%\)Mean6\.4287\.45Std\. dev\.14\.14–Min0\.62–Max100\.0–The keep ratio spans from0\.62%0\.62\\%to100%100\\%with a standard deviation of14\.1414\.14percentage points, indicating that FFD does*not*impose a uniform sparsity structure\. Rather, the sameδ\\delta\-threshold rule allows different heads to exhibit markedly different retention behavior, with the mean salient hit rate remaining at87\.45%87\.45\\%\. Notably, both Quest and Twilight require retaining the first two layers’ full\-precision KV cache to avoid accuracy collapse, whereas FFD requires no such early\-layer exception—suggesting that the content\-aware scanning mechanism naturally captures the distributional properties that lead other methods to rely on architectural hand\-tuning\.

Scope\.These statistics are derived from a single LongBench sample under a fixedδ=5\\delta=5\. They should be interpreted as a mechanistic characterization rather than as claims about worst\-case behavior across all task distributions\.

## Appendix EFailure\-Case Profiling

To characterize when FFD’s approximation is most stressed, we profile failure\-prone conditions using two proxy metrics: the gap between the top\-1 and top\-2 attention scores \(top1\_top2\_gap\) and the standard deviation of attention scores \(score\_std\)\. A layer\-head pair is flagged as a*proxy\-error*case if it falls into the lowest decile \(bottom10%10\\%\) on either metric, and as a*high\-error*case if it falls into the lowest decile on both\. These are proxy indicators of flat or multi\-peak attention distributions where quantization\-based selection is most challenged\.

We profile33923392layer\-head pairs across44attention dumps \(2 models×\\times2 LongBench tasks\)\. Table[6](https://arxiv.org/html/2609.00097#A5.T6)summarizes the results\.

Table 6:Failure\-case profiling across 3392 layer\-head pairs\.Case TypeProportion \(%\)Mean Keep Ratio \(%\)Mean Salient Hit Rate \(%\)Regular80\.842\.4363\.74Proxy\-error19\.1613\.1574\.67The dominant response to challenging distributions is*conservative retrieval*: the mean keep ratio rises from2\.43%2\.43\\%\(regular\) to13\.15%13\.15\\%\(proxy\-error\), while the salient hit rate actually*improves*from63\.74%63\.74\\%to74\.67%74\.67\\%\. This confirms the structural asymmetry discussed in Section 5\.2: when the pseudo\-max approximation deviates from the true global maximum, the resulting threshold is more permissive \(not more restrictive\), so the selection defaults to retaining more blocks rather than dropping salient ones\. The primary consequence of these cases is therefore an efficiency penalty rather than an accuracy failure\. We caution that proxy\-error flags are based on attention\-level statistics and have not been causally linked to downstream task errors\.

## Appendix FCross\-Architecture Generalization: Qwen2\.5

Table[7](https://arxiv.org/html/2609.00097#A6.T7)reports the full per\-subtask RULER results on Qwen2\.5\-7B\-Instruct\.

Table 7:Qwen2\.5\-7B\-Instruct RULER results \(32k context, all 13 subtasks\)\.MethodSK\-1SK\-2SK\-3MK\-1MK\-2MK\-3MQMVVTSQAHOTCWEFWEAvgBase100100100100969297\.894\.399\.4574461\.288\.386\.9KIVI100100869954788\.586\.394\.4494049\.382\.772\.0Quest100100979868496\.392\.899\.6393736\.167\.371\.9Stream\.34685117\.38\.011\.6182615\.088\.316\.2Twilight100969185864466\.373\.380\.4443554\.382\.772\.1FFD \(δ=7\\delta=7\)99999999958798\.393\.598\.6534068\.487\.085\.9Table[8](https://arxiv.org/html/2609.00097#A6.T8)reports the LongBench breakdown\. FFD maintains a clear advantage over all sparse baselines on Qwen, confirming generalization beyond Llama\.

Table 8:Qwen2\.5\-7B\-Instruct LongBench results \(category averages\)\.MethodS\-DocM\-DocSummFew\-ShotSynCodeAvgBase24\.3527\.8824\.3559\.5032\.9911\.7130\.13KIVI24\.4927\.0423\.2659\.0124\.3811\.8928\.34Quest25\.0026\.6822\.7058\.8132\.1811\.6129\.49Stream\.15\.9222\.5118\.9652\.535\.4813\.2921\.45Twilight26\.0730\.5323\.6159\.1932\.6511\.8130\.64FFD24\.6528\.3824\.2359\.3132\.3011\.7930\.11FFD maintains a clear accuracy advantage over sparse baselines on Qwen while generalizing beyond Llama without architecture\-specific tuning\. For Twilight, its stronger LongBench average is largely driven by M\-Doc and Syn categories; however on RULER retrieval tasks, FFD retains a substantial lead across all key subtasks \(e\.g\., MK\-3: 87 vs\. 44, MQ: 98\.3 vs\. 66\.3\)\.

## Appendix GComponent\-wise Ablation Study

To disentangle the contributions of our three co\-design axes—scan precision \(FP16 vs\. 2\-bit\), selection rule \(top\-kkvs\. top\-δ\\delta\), and execution path \(split vs\. fused\+fullgraph\)—we conduct a controlled ablation study under matched conditions\. All variants are evaluated with a 16k prefill and 128 decode steps on an RTX 4090 \(batch size 1\)\. Table[9](https://arxiv.org/html/2609.00097#A7.T9)reports the end\-to\-end throughput and downstream task accuracy for six configurations\.

Table 9:Component\-wise ablation of scan precision, selection rule, and execution path \(16k context, RTX 4090\)\.\#Variant \(Scan – Rule – Execution\)Tok/sRULER AVGLongBench AVG1FP16 – top\-kk– split17\.4386\.8225\.872FP16 – top\-δ\\delta– split21\.6490\.3925\.9032\-bit – top\-kk– split14\.8086\.9525\.8342\-bit – top\-δ\\delta– split24\.6590\.4025\.9052\-bit – top\-kk– fused21\.2186\.9525\.8362\-bit – top\-δ\\delta– fused \(FFD\)53\.9690\.4025\.90The ablation reveals three key findings\. First, replacing top\-kkwith top\-δ\\deltaalone improves both accuracy \(RULER: 86\.82→\\rightarrow90\.39\) and throughput \(17\.4→\\rightarrow21\.6 tok/s\), confirming that top\-δ\\deltaachieves distribution\-adaptive sparsity without the overhead of global synchronization \(V1 vs\. V2\)\. Second, 2\-bit scanning is not beneficial in isolation—under top\-kk, the quantized scan degrades throughput \(V1 vs\. V3: 17\.4→\\rightarrow14\.8\) due to compute\-bound sorting overhead\. However, when paired with top\-δ\\delta, which reduces selection to a scalar comparison, the reduced memory footprint of 2\-bit becomes a strict asset \(V2 vs\. V4: 21\.6→\\rightarrow24\.7\)\. Third, and most importantly, the system\-level fusion and CUDA Graph capture exhibit asynergistic amplificationwith the algorithmic components\. While fusion alone yields a modest 1\.4×\\timesgain under top\-kk\(V3 vs\. V5\), the same system optimizations deliver a 2\.2×\\timesboost when combined with top\-δ\\delta\(V4 vs\. V6\), demonstrating that the algorithmic and system designs are inseparably linked\.

## Appendix HComponent Accuracy Ablation on RULER

Table[10](https://arxiv.org/html/2609.00097#A8.T10)provides the full per\-subtask RULER breakdown for the four non\-fused variants of the component ablation study \(Section 4\.5\)\. The key observation is that CWE degrades severely under the fixed\-budget \(top\-kk\) selector \(scores of 24\.6 and 25\.8\), whereas top\-δ\\deltarestores it to 68\.2–69\.5\. This is because CWE requires the model to count word frequencies across the full context, a task that demands distribution\-aware adaptive sparsity rather than a fixed token budget\.

Table 10:Per\-subtask RULER scores for component ablation variants \(Llama\-3\.1\-8B, 32K, split execution\)\.Subtaskfp\-fixed\-splitfp\-delta\-splitq2\-fixed\-splitq2\-delta\-splitSK\-1100100100100SK\-2100100100100SK\-3100100100100MK\-197989797MK\-297999899MK\-399999699MQ98\.599\.097\.899\.0MV99\.310098\.899\.8VT98\.698\.298\.498\.0SQuAD67686968Hotpot54545655CWE24\.668\.225\.869\.5FWE93\.791\.793\.791\.0Avg86\.890\.487\.090\.4