BLASST just won Best Paper at #MLSys26! In this paper, we introduce a simple, training-free dynamic sparse attention mechanism that uses a single scalar threshold on online softmax statistics to skip negligible attention blocks. Unfortunately I won’t be there in person, but please say hi to my awesome coauthors! Paper: https://arxiv.org/abs/2512.12087

1 Introduction

Source: https://arxiv.org/html/2512.12087 marginparsep has been altered. topmargin has been altered. marginparwidth has been altered. marginparpush has been altered. The page layout violates the ICML style.Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you. We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again.

BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding

Anonymous Authors1

Abstract

The growing demand for long-context inference capabilities in Large Language Models (LLMs) has intensified the computational and memory bottlenecks inherent to the self-attention mechanism. To address this challenge, we introduce BLASST, a drop-in, dynamic sparse attention mechanism that accelerates inference by using only a fixed scalar threshold to skip attention blocks. Our method targets practical inference deployment by removing the barriers to adoption present in existing works. As such, BLASST eliminates training requirements, avoids expensive pre-computation passes, accelerates both prefill and decode across all major attention variants (MHA, GQA, MQA, and MLA), provides optimized support for modern hardware, and easily integrates into existing frameworks. This is achieved by reusing online softmax statistics to identify negligible attention scores, skipping softmax, value block loads, and the subsequent matrix multiplication. We demonstrate the BLASST algorithm by delivering optimized kernels with negligible latency overhead. Our automated threshold calibration procedure reveals a simple inverse relationship between optimal threshold and context length, meaning we require only a single threshold each for prefill and decode per model. Preserving benchmark accuracy, we demonstrate a 1.52×\timesspeedup for prefill at 71.9% sparsity and a 1.48×\timesspeedup for decode at 73.2% sparsity on modern GPUs.

††footnotetext:1Anonymous Institution, Anonymous City, Anonymous Region, Anonymous Country. Correspondence to: Anonymous Author <[email protected]>. Preliminary work. Under review by the Machine Learning and Systems (MLSys) Conference. Do not distribute.Large Language Models (LLMs) have revolutionized natural language processing, achieving remarkable performance across diverse tasks. However, their practical deployment faces a critical bottleneck: the quadratic computational complexity of the attention mechanism. As applications increasingly demand longer context windows—from processing entire codebasesRoziereet al.(2023)to analyzing lengthy documentsZenget al.(2025)and maintaining extended conversationsAchiamet al.(2023)—this bottleneck becomes increasingly severe. Recent models like Deepseek-R1Guoet al.(2025)and Qwen3Yanget al.(2025)support context lengths up to 128K tokens, with some models pushing to 1M tokensComaniciet al.(2025). Yet processing such long sequences remains computationally prohibitive, with attention computation dominating both latency and me-

Figure 1:Overview of BLASST. Blocks along a row of the attention matrix are sequentially processed. We (1) update the running row max (m(j)m^{(j)}) as in FlashAttention, (2) compute the block max (m~(j)\tilde{m}^{(j)}) for eachSjS_{j}block (Q​Kj⊤QK_{j}^{\top}), and (3) skip subsequent work if the block max is lower than the running max by more than the input threshold,ln⁡(λ)\ln(\lambda). Full details can be found in Algorithm1.mory consumption. For a sequence of lengthnn, the attention mechanism requiresO​(n2)O(n^{2})operations and memory accesses, making real-world deployment of long-context models challenging even with state-of-the-art hardware. While FlashAttentionDaoet al.(2022); Zadouriet al.(2026)and its successors have optimized memory bandwidth utilization through tiling and kernel fusion, they still compute the full attention matrix, leaving the fundamental quadratic complexity unaddressed.

Sparse attentionmethods have emerged as a promising solution by computing only a subset of the full attention matrix. While these approaches cleverly determine which attention scores to skip, their added complexity hinders practical use. We identify five key hurdles to their adoption: (1) Some methods require expensive pre-computation to determine sparsity patterns, often negating their theoretical speedupsJianget al.(2024); Xuet al.(2025). (2) Other methods introduce new layers that require model fine-tuningXiaoet al.(2025)or training an entirely new architectureDeepSeek-AI (2025). (3) Most existing works focus exclusively on either the prefill or decode phase, missing opportunities for end-to-end inference acceleration. (4) They lack kernel support for newer GPUs, making it unclear if their speedups translate to the characteristics of modern hardware, like Blackwell and Hopper. (5) These methods often hinder framework integration, requiring intrusive modifications to model architectures or attention interfaces and substantial changes to existing APIs.

To address these hurdles, we present BLASST (BLocked Attention Sparsity via Softmax Thresholding), a simple yet effectivetraining-freesparse attention method that dynamically prunes negligible attention blocks duringboth prefill and decodewithno pre-computation overhead. Our key insight is that during FlashAttention’s block-wise online-softmax, we can identify and skip blocks whose contribution to the final output will be negligible based solely on already-computed information. Specifically, when processing blocks sequentially, we maintain a running maximum of attention scores. As shown in Figure1, if a block’s local maximum score is significantly smaller than this running maximum (by a thresholdλ\lambda), its post-softmax values will be near zero after normalization. We can therefore skip three expensive operations for such blocks: (1) computing the exponential for softmax, (2) loading the corresponding value block from HBM, and (3) multiplication between attention and values. This simple pruning rule requires only a single comparison per block andseamlessly integrates into existing attention APIs, requiring only a single scalar threshold input.

To maximize the practical impact of BLASST, we provideoptimized CUDA kernels for Blackwell and Hopperthat implement our sparse attention algorithm. Our kernels are designed with two key goals: (1) introduce minimal overhead for the block-skipping decision logic by reusing already-computed statistics, and (2) strategically target the bottleneck resources in each phase—reducing CUDA core and tensor core usage in compute-bound prefill, and reducing memory bandwidth consumption in memory-bound decode. Our prefill and decode kernels are tailored to their distinct computational patterns. Our kernels achieve up to 1.52×\timesspeedup for prefill at 71.9% sparsity and 1.48×\timesspeedup for decode at 73.2% sparsity over FlashAttention baselinesShahet al.(2024); Zadouriet al.(2026), while maintaining numerical stability and supporting the common attention variants (MHA, MQA, GQA, MLA).

Beyond the core algorithm and kernel implementation, we develop two key techniques to enhance BLASST’s deployment and performance. First, we propose an automated calibration procedure that determines optimal thresholds for any target sparsity level. Our calibration reveals a robust inverse relationshipλ=a/L\lambda=a/Lbetween threshold and context lengthLL, enabling reliable deployment across diverse scenarios without manual tuning. Second, we explore sparsity-aware training as a natural extension, showing that models can be trained to be inherently more robust to sparse attention patterns. This training approach further pushes the accuracy-sparsity frontier, enabling even higher sparsity levels with minimal loss in accuracy.

Our contributions include:

1. 1.The BLASST algorithm, a drop-in method with no pre-computation overhead and no proxy scores, achieving minimal accuracy loss.
2. 2.Automated hyperparameter selection and sparsity-aware training for robust, flexible, and extensible deployment.
3. 3.Optimized CUDA kernels implementing BLASST for both prefill and decode, available in TensorRT-LLM111GPU kernels and inference framework support can be found at https://github.com/NVIDIA/TensorRT-LLMand FlashInfer.

2Related Works

Effectively exploiting the sparse attention property requires either reducing compute on unimportant interactions or reducing memory footprint (e.g., KV cache) without expensive selection overheads or retraining. Comparing to the following related works, BLASST addresses both dimensions simultaneously, in a training-free manner. Table1summarizes the landscape of existing work.

Table 1:Feature comparison of sparse attention methods. BLASST distinguishes itself as the only method capable of accelerating both prefill and decode phases without requiring training or costly pre-computation steps.MethodAcceleratesPrefillAcceleratesDecodeNoTrainingNo Pre-ComputationH2O✗✓✓✓SnapKV✗✓✓✓RocketKV✗✓✓✗Quest✗✓✓✗DuoAttention✓✓✗✓DSA✓✓✗✓MInference✓✗✓✗SpargeAttention✓✗✓✗XAttention✓✗✓✗BLASST✓✓✓✓### 2.1Compute-Optimized Sparsity

Several approaches reduce attentioncomputeby selecting important interactions. Static pattern methods like Sparse TransformerChildet al.(2019), LongFormerBeltagyet al.(2020), and BigBirdZaheeret al.(2020)reduce complexity through local or block-based attention. Retrieval head-based methodsWuet al.(2025); Xiaoet al.(2025)accelerate model decoding by focusing compute on crucial retrieval heads. Dynamic sparsity methods like MInferenceJianget al.(2024)use pre-computed importance scores, XAttentionXuet al.(2025)ranks anti-diagonal blocks, and FlexPrefillLaiet al.(2025)offers compiler-supported, flexible block patterns; while effective for prefill, their pre-computation and scheduling overheads can limit realized speedups. Training-aided sparsity such as SeerAttentionGaoet al.(2025b)induces high sparsity via (pre)training gates, improving efficiency but adding training cost and showing mixed downstream model performance. FLASH-DAlexandridiset al.(2025)leverages the mathematical properties of online softmax in a similar way as BLASST, but to improve numerical stability and parallelism on custom hardware accelerators.

SpargeAttentionZhanget al.(2025)has the most similar design to BLASST. We differ in three key aspects: (1) BLASST optimizes both prefill and decode with specialized kernels, while SpargeAttention targets prefill only; (2) we make skip decisions directly using already-computed statistics with zero overhead, while SpargeAttention uses a separate prediction step; (3) our decode kernel skips Value loading from HBM, addressing memory-bound bottlenecks on top of compute savings. In addition, we provide automated calibration and sparsity-aware training.

2.2Memory-Optimized Sparsity

Token/KV sparsity focuses on reducingmemoryfootprint and decode-time cost. H2OZhanget al.(2023), TOVAOrenet al.(2024), and InfLLMXiaoet al.(2024a)discard tokens based on query patterns. StreamingLLMXiaoet al.(2024b)retains initial and recent tokens for consistent latency and memory usage. QuestTanget al.(2024)prunes tokens conditioned on the current query, Rectified Sparse AttentionSunet al.(2025)adaptively selects tokens to maintain accuracy at high sparsity, RocketKVBehnamet al.(2025)compresses the KV cache with selective eviction, and recent KV compression for hyper-scalingŁańcuckiet al.(2025)further extends effective context; TidalDecodeYanget al.(2026)stabilizes decode efficiency with position-persistent patterns. We further distinguish reasoning-oriented compression methods such as RPCSonget al.(2025), which prioritize preserving reasoning-critical information under memory constraints; non-eviction methods such as LokiSinghaniaet al.(2024), which avoid explicit KV eviction while reducing effective memory/computation overhead. In general, these methods reduce memory accesses in the decode phase, whereas BLASST reduces compute and memory accesses in both prefill and decode while remaining training-free.

2.3New Attention Variants

Beyond the above methods, alternative mechanisms include Sliding Window AttentionBeltagyet al.(2020), Linear or Gated AttentionQiuet al.(2025), and State-Space Models (SSM)Gu and Dao (2024). Native Sparse Attention (NSA)Yuanet al.(2025)and DeepSeek Sparse Attention (DSA)DeepSeek-AI (2025), while effective in some regimes, often require architectural changes and training. By contrast, BLASST is a training-free method that accelerates both prefill and decode without proxy scores or complex pre-computation, integrating seamlessly with FlashAttention implementations.

3Methodology

3.1Pruning Attention with Running Maximums

The core insight of BLASST lies in the observation that during the computation of attention scores in FlashAttention, many blocks contribute negligibly to the final output after softmax normalization. Our method identifies and skips these blocks dynamically during the forward pass, without requiring pre-computation or proxy scores.

3.1.1Key Insight

In the standard attention mechanism, the softmax operation computes:

Attention​(Q,K,V)=softmax​(Q​K⊤dk)​V\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{\top}}{\sqrt{d_{k}}}\right)V(1) During FlashAttention’s block-wise computation, we maintain a running maximummi(j)m_{i}^{(j)}across blocks. If a block’s local maximumm~i(j)\tilde{m}_{i}^{(j)}is significantly smaller than the current running maximum, i.e.,m~i(j)−mi(j)<ln⁡(λ)\tilde{m}_{i}^{(j)}-m_{i}^{(j)}<\ln(\lambda)for some thresholdλ\lambda, then after exponentiation:

exp⁡(m~i(j)−mi(j))<λ≈0\exp(\tilde{m}_{i}^{(j)}-m_{i}^{(j)})<\lambda\approx 0(2) Since the maximum value is bounded byλ\lambda, the block’s contribution to the final attention output will be negligible, allowing us to skip its computation entirely.

Intuitively, this criterion follows a three-step approximation. First, the ideal importance of each scoreSi​jS_{ij}is its value relative to the (unknown) global maximum. Second, computing the true maximum on-the-fly is too expensive, so we use the running maximum as a tractable proxy and compareSi​jS_{ij}against it. Third, to enable an efficient block-level decision inside the kernel, we replace token-levelSi​jS_{ij}with the block-local maximum, which yields the inexpensive condition(block_max−running_max)<ln⁡(λ)(\text{block\_max}-\text{running\_max})<\ln(\lambda).

3.1.2Algorithm Design

Algorithm1presents our modified FlashAttention forward pass, where the sequence is tiled intoTrT_{r}query blocks andTcT_{c}KV blocks of sizeBcB_{c}each. The key modification is the introduction of a dynamic pruning condition that saves both computation and memory bandwidth.Where We Save:Whenm~i(j)−mi(j)<ln⁡(λ)\tilde{m}_{i}^{(j)}-m_{i}^{(j)}<\ln(\lambda)(line 7), we skip:

1. 1.Compute savings (CUDA cores):The expensiveexp⁡(⋅)\exp(\cdot)operations for computingP~i​j\tilde{P}_{ij}require multiple instructions per element:MUFU.EX2(exponential),FMUL(multiplication), andFADD(addition). We also skip therowsumreduction operations (FADDinstructions) for normalizing attention weights. For a typical block, this saves thousands of CUDA core instructions.
2. 2.Compute savings (Tensor cores)The matrix multiplicationP~i​j​Vj\tilde{P}_{ij}V_{j}. In the prefill phase, where kernels are compute-bound, avoiding these MMA operations provides a substantial speedup.
3. 3.Memory bandwidth savings:Loading the Value blockVjV_{j}from HBM to SRAM. This is particularly critical in decode phase, where attention is memory-bound.

Algorithm 1FlashAttention with BLASST1:Query blocks

{Qi}i=1Tr\{Q_{i}\}_{i=1}^{T_{r}}, Key blocks

{Kj}j=1Tc\{K_{j}\}_{j=1}^{T_{c}}, Value blocks

{Vj}j=1Tc\{V_{j}\}_{j=1}^{T_{c}}, threshold

λ\lambda 2:Output blocks

{Oi}i=1Tr\{O_{i}\}_{i=1}^{T_{r}} 3:for

i=1i=1to

TrT_{r}do

4:Initialize

mi(0)=−∞m_{i}^{(0)}=-\infty,

Oi(0)=0O_{i}^{(0)}=0,

li(0)=0l_{i}^{(0)}=0 5:for

j=1j=1to

TcT_{c}do

6:Compute

Si​j=Qi​Kj⊤S_{ij}=Q_{i}K_{j}^{\top}⊳\trianglerightAttention scores

7:

m~i(j)=rowmax⁡(Si​j)\tilde{m}_{i}^{(j)}=\operatorname{rowmax}(S_{ij})⊳\trianglerightLocal maximum

8:

mi(j)=max⁡(mi(j−1),m~i(j))m_{i}^{(j)}=\max(m_{i}^{(j-1)},\tilde{m}_{i}^{(j)})⊳\trianglerightRunning maximum

9:if

m~i(j)−mi(j)<ln⁡(𝝀)\tilde{m}_{i}^{(j)}-m_{i}^{(j)}<\ln(\boldsymbol{\lambda})then

10:continue⊳\trianglerightSkip this block

11:endif

12:

P~i​j=exp⁡(Si​j−mi(j))\tilde{P}_{ij}=\exp(S_{ij}-m_{i}^{(j)})⊳\trianglerightCompute attn. weights

13:

li(j)=emi(j−1)−mi(j)​li(j−1)+rowsum⁡(P~i​j)l_{i}^{(j)}=e^{m_{i}^{(j-1)}-m_{i}^{(j)}}l_{i}^{(j-1)}+\operatorname{rowsum}(\tilde{P}_{ij}) 14:

Oi(j)=emi(j−1)−mi(j)​Oi(j−1)+P~i​j​VjO_{i}^{(j)}=e^{m_{i}^{(j-1)}-m_{i}^{(j)}}O_{i}^{(j-1)}+\tilde{P}_{ij}V_{j} 15:endfor

16:

Oi=Oi(Tc)/li(Tc)O_{i}=O_{i}^{(T_{c})}/l_{i}^{(T_{c})}⊳\trianglerightFinal normalization

17:endfor

18:return

{Oi}i=1Tr\{O_{i}\}_{i=1}^{T_{r}}

Our approach directly reduces the total amount of computation by dynamically identifying and skipping negligible attention blocks during the forward pass. This simple yet effective modification requires minimal changes to the existing FlashAttention implementation while providing significant computational savings.

3.2Calibration for Optimal Sparsity

A critical challenge in deploying BLASST is selecting the appropriate thresholdλ\lambdathat balances sparsity and accuracy. To understand this relationship, we conducted experiments on Llama-3.1-8B across RULER benchmark challenging subsets (NIAH_MULTI, VT, FWE) with context lengths from 8K to 64K tokens.

Sparsity Determines Accuracy.Figure2(left) shows relative accuracy degradation as a function of the observed sparsity level. We normalize each curve by the full attention result for a fair comparison. Remarkably, all curves exhibit consistent degradation patterns: accuracy remains stable up to∼\sim60–70% sparsity, beyond which accuracy drops sharply. This consistency across diverse tasks and sequence lengths reveals thataccuracy degradation is primarily determined by the sparsity ratio itself, not the type of data set or sequence length.

Threshold Calibration is Essential.For models to achieve consistent accuracy, we must maintain afixed sparsity ratiorather than a fixed threshold. However, Figure2(right) shows that achieving 75% sparsity requiresλ≈1​e​−4\lambda\approx110\-4for 8K contexts but only1​e​−5110-5for 64K contexts. This necessitates adaptive calibration. Importantly, by targeting fixed sparsity through calibration, users can control and foresee the computational speedup, since accuracy gains scale predictably with the observed sparsity level.

Figure 2:(Left) Relative accuracy drop across different datasets and context lengths shows consistent degradation patterns as observed sparsity increases. All curves are normalized to their initial accuracy. (Right) Relationship between threshold and observed sparsity levels across different sequence lengths, demonstrating the need for threshold calibration to maintain fixed sparsity across varying contexts.Through empirical analysis, we find that the optimal threshold follows aninversely proportionalrelationship with context lengthLL:

λ=aL\lambda=\frac{a}{L}(3)whereaais a model-specific scale factor that depends on the target sparsity level. This inverse relationship has theoretical grounding: since attention scores are row-normalized to sum to 1, longer sequences have lower average scores per token, requiring proportionally smaller thresholds. Without calibration, fixed thresholds would cause vastly different sparsity levels across sequence lengths.

Algorithm 2BLASST Calibration1:Calibration dataset

𝒟={(xi,Li)}i=1N\mathcal{D}=\{(x_{i},L_{i})\}_{i=1}^{N}, threshold set

Λ\Lambda, sparsity bounds

smins_{\min},

smaxs_{\max} 2:Calibration parameters

α\alpha,

β\beta 3:Initialize data points

𝒫=∅\mathcal{P}=\emptyset 4:foreach sample

(xi,Li)(x_{i},L_{i})in

𝒟\mathcal{D}do

5:foreach

λj∈Λ\lambda_{j}\in\Lambdado

6:

si​j←MeasureSparsity​(λj,xi)s_{ij}\leftarrow\mathrm{MeasureSparsity}(\lambda_{j},x_{i}) 7:if

smin≤si​j≤smaxs_{\min}\leq s_{ij}\leq s_{\max}then⊳\trianglerightFilter unreliable extremes

8:Add

(λj⋅Li,si​j)(\lambda_{j}\cdot L_{i},\;s_{ij})to

𝒫\mathcal{P} 9:endif

10:endfor

11:endfor

12:Fit exponential model

λ⋅L=α⋅exp⁡(β⋅s)\lambda\cdot L=\alpha\cdot\exp(\beta\cdot s)using

𝒫\mathcal{P} 13:returnparameters

α\alpha,

β\beta

To determineaafor any target sparsity, we propose the calibration procedure detailed in Algorithm2. For each calibration samplexix_{i}of lengthLiL_{i}and each candidate thresholdλj\lambda_{j}, we measure the achieved sparsitysi​js_{ij}and record the scale factorλj⋅Li\lambda_{j}\cdot L_{i}. Since sparsity for all thresholds can be computed from the same attention scores, the entire calibration requires only a single forward pass over𝒟\mathcal{D}. We then fit an exponential modelλ⋅L=α⋅exp⁡(β⋅s)\lambda\cdot L=\alpha\cdot\exp(\beta\cdot s)to all collected data points. The exponential form reflects the heavy-tailed distribution of attention scores: small increases in threshold prune many low-scoring blocks, while further increases yield diminishing returns as only high-scoring blocks remain. In inference with target sparsitySS, the threshold isλ=α⋅exp⁡(β⋅S)/L\lambda=\alpha\cdot\exp(\beta\cdot S)/L, preserving the inverse relationship Eq. (3) while allowing the target sparsity to be adjusted at runtime without recalibration.

More importantly, by targeting fixed sparsity levels, our calibration ensures predictable computational speedup across different context lengths. This is a crucial property for production deployment where consistent performance is required. We provide additional cross-dataset evidence in AppendixA(Table12), showing that the calibrated parameteraayields stable sparsity across diverse tasks without task-specific retuning.

3.3Extensibility to Attention Variants

Because BLASST depends only on tiled online softmax, it is inherently compatible with many existing dense attention variants. MLADeepSeek-AI (2025), for instance, still employs online softmax to compute attention scores within its latent space. Although MLA decoding shifts the bottleneck towards a compute-bound regime, BLASST remains effective because it eliminates both computation and memory accesses, providing benefits regardless of the primary hardware bottleneck.

3.4Sparsity-Aware Training

While BLASST is primarily designed as a training-free inference optimization, we explore sparsity-aware training as a simple extension to further improve the accuracy-sparsity trade-off. The motivation is straightforward: if models learn to concentrate important information in high-scoring attention blocks during training, they should maintain higher accuracy when those blocks are pruned during inference.

Our method is simple: during fine-tuning, we apply BLASST in the forward pass to skip negligible attention blocks based on the threshold criterion. In the backward pass, skipped blocks naturally receive no gradients since they were not computed in the forward pass. This encourages the model to adapt its attention patterns to be more compatible with sparsity, concentrating important information in blocks that pass the threshold test. This approach requires no architectural changes or auxiliary losses—it is simply training with the same sparse attention that will be used at inference time.

4Kernel Design

(a)Normal FlashAttention-4 prefill pipeline schedule. (b)BLASST prefill pipeline schedule with T0 and T1 both skipping loops 1 and 3.

Figure 3:Prefill pipeline schedules for FlashAttention and BLASST at 50% sparsity across 4 loop iterations (L0–L3). Rows are separated based on warp/warpgroup specializations. Darker and lighter hues correspond to ops for different tile rows (T0/T1). The MMA warp’s BMM1 and BMM2 ops are indicated with B1 and B2. The softmax warpgroups are primarily bottlenecked by exponentiation (EX2), but they also perform the skip check, row sum and softmax scaling (not shown). Mainloop iterations are enclosed by solid lines.The BLASST kernels were designed with two primary goals: (1) minimal changes to existing FlashAttention kernel interfaces and implementation structure, and (2) minimal overhead for block skipping decision logic. Our key insight is to reuse statistics computed during the standard FlashAttention algorithm—specifically, the local maximum and running maximum values maintained in every thread during online softmax. Our optimizations are specific to BLASST and cannot be applied to typical FlashAttention kernels.

Skip Decision Implementation.The decision process (line 7 in Algorithm1) requires only a few additional instructions per block: (1) setting a predicate per thread based on the threshold comparison, (2) issuing aVOTEinstruction to determine if all threads within a warp agree to skip, and (3) a singleATOMICinstruction to shared memory issued by one thread per warp to coordinate the block-level decision across the softmax warpgroup. We carefully design the kernel such that the decision-making instructions are hidden behind existing operations, adding negligible latency overhead.

Since prefill and decode phases have fundamentally different performance characteristics, we implement specialized optimizations for each.

4.1Prefill Kernel: Compute-Bound Optimization

Prefill kernels are typically compute-bound, bottlenecked by CUDA core (softmax) and tensor core (matrix multiplication) throughput rather than memory bandwidth. Therefore, our prefill kernel is designed to skip both softmax computation and MMA operations (attention-value multiplication) for pruned blocks.

Figure3illustrates our changes to the pipeline schedule for the BLASST prefill kernel, which is optimized for compute-bound scenarios by overlapping different compute tasks. The pipeline schedules operations across Tensor Cores (math warp/matrix multiplication) and CUDA cores (softmax and correction logic). Figure3(b)shows that even as allQ​K⊤QK^{\top}(BMM1) operations are computed, the kernel dynamically skips compute-heavy softmax and attention-value multiplication (BMM2) for blocks identified as negligible (e.g., loop 1 and loop 3 in Figure3(b)). By skipping these compute operations, the kernel frees up execution units, allowing subsequent operations to be scheduled earlier. This compresses the entire schedule, reducing the total runtime from 18 time units in Figure3(a)to 14 units in Figure3(b).

The Value blocks remain loaded from HBM in the prefill kernel because: (1) memory bandwidth is not the bottleneck, (2) the prefetching pipeline benefits from predictable memory access patterns, and (3) the latency of conditional Value loading would exceed the savings. By focusing on eliminating compute operations, we achieve speedups that scale nearly linearly with sparsity in the compute-bound regime. Our current design prioritizes the common case where prefill is compute-bound on modern GPUs; however, Value loading could be skipped in prefill if future workloads and/or hardware architectures shift to a memory-bandwidth-bound regime.

(a)Normal FlashAttention-4 decode pipeline schedule. (b)BLASST decode pipeline schedule when skipping loops 1, 2, and 4.

Figure 4:Decode pipeline schedules for FlashAttention and BLASST skipping in loops 1, 2, and 4. We focus on the steady state of the first 6 mainloop iterations (L0–L5). The BLASST schedule does consecutive K loads since V cannot be pre-fetched until the skip-check is computed after BMM1. V loads in Figure4(b)finish more quickly because there are fewer simultaneous loads. Arrows indicate scoreboard dependencies from the skip check after BMM1. Note that the MMA warp’s BMM1 and BMM2 ops are indicated with B1 and B2.

4.2Decode Kernel: Memory-Bound Optimization

Decode kernels are typically memory-bound, bottlenecked by the HBM bandwidth required to fetch the KV cache rather than compute, as attention involves only a single Query against all Keys. Our kernel thus focuses on skipping the memory-intensive load of the Value matrixVjV_{j}for pruned blocks, directly addressing this HBM bottleneck. This optimization cuts memory traffic proportionally to the sparsity level, while we overlap the threshold and Key operations with the remaining Value loads to achieve a substantial speedup, reflecting the different performance characteristics of decode versus prefill.

A critical challenge in the decode kernel is that in a naive implementation, the value load and subsequent attention-value multiplication (BMM2) would be issued before the query-key multiplication (BMM1) completes and the skip decision can be made. This would result in wasted memory bandwidth loading values that will ultimately be discarded. By conditionally loading blocks of V we can save memory bandwidth; however, this introduces a scoreboard dependency that stops us from issuing consecutive loads ahead of time. The pipeline becomes serialized and can introduce pipeline bubbles.

To address this, we redesign the decode kernel pipeline to use batched load scheduling. As shown in Figure4(b), instead of processing blocks end-to-end one at a time, we process multiple consecutive query-key products back-to-back (K1⊤​QK_{1}^{\top}Q,K2⊤​QK_{2}^{\top}Q…KB⊤​QK_{B}^{\top}Q). The tradeoff is that we must maintainBBnumber of shared memory buffers forSjS_{j}(fromKj⊤​QK_{j}^{\top}Q), but they are relatively small with a query sequence length of 1. This reordering allows us to issue a batch of loads for onlyVjV_{j}tiles that pass the threshold check, removing the possibility of pipeline bubbles. As a result, Figure4(a)takes 38 time units to complete all V loads, whereas Figure4(b)takes 31 units.

For attention mechanisms like Multi-head Latent Attention (MLA)Liuet al.(2024a)that can be compute-bound even in decode, we also skip softmax operations for pruned blocks, providing further speedup beyond memory savings.

Table 2:Performance of BLASST at different sparsity levels across all models and benchmarks. We evaluate on Llama-3.1-8B and Qwen3-8B across three deployment scenarios: prefill-only optimization (long-context tasks: RULER, LongBench); decode-only optimization (reasoning tasks: MATH500, AIME 2024, GPQA); and combined prefill+decode optimization. Results show minimal accuracy degradation even at∼\sim75% sparsity, with occasional improvements over the dense baseline.ModelTarget SparsityPrefill PhaseDecode PhasePrefill + Decode PhaseRULER-32KLongBenchMATH500AIME2024GPQARULER-32KLongBenchLlama-3.1-8BDense92.3331.4073.4046.6646.7192.3331.4050%91.8131.8073.7146.1546.3191.7932.4075%91.6731.8073.8946.0145.9591.6731.80Qwen3-8BDense91.9033.6095.8775.0061.2191.9033.6050%92.0835.1096.2376.5061.5692.0733.3075%92.1134.4096.0775.3361.5191.7433.10

5Experiments

5.1Experimental Setup

Models.We evaluate BLASST on state-of-the-art language models to demonstrate its effectiveness across different architectures. Our evaluation focuses on two 8B parameter models—Llama-3.1-8B-Instruct and Qwen3-8B-Instruct—both supporting context lengths up to 128K tokens. For long-generation reasoning tasks, we use Llama-3.1-8B-Instruct distilled from DeepSeek-R1Guoet al.(2025), which provides enhanced reasoning capabilities while maintaining compatibility with our sparse attention approach.

Baselines.We compare BLASST against dense attention and SOTA sparse attention methods. For prefill optimization, we compare against MInferenceJianget al.(2024), FlexPrefillLaiet al.(2025), and XAttentionXuet al.(2025). For decode optimization, we evaluate against QuestTanget al.(2024), RocketKVBehnamet al.(2025). For each baseline, we adopt its best-performing configuration as reported in its respective paper to ensure fair comparisons.

Datasets.We evaluate on two categories of benchmarks: (1)Long-context tasks: RULERHsiehet al.(2024)(synthetic retrieval and reasoning from 4K-128K tokens) and LongBench v2Baiet al.(2025)(real-world QA, summarization, and code completion). (2)Reasoning tasks: MATH500 (mathematical problem solving), AIME 2024 (advanced mathematics), GPQA (graduate-level science), and LiveCodeBench (code generation). These reasoning benchmarks test whether sparse attention preserves complex multi-step reasoning capabilities. We use the NVIDIA NeMo-Skills framework222https://github.com/NVIDIA-NeMo/Skillsfor standardized evaluation of reasoning tasks.

Implementation Details.We implement BLASST as optimized CUDA kernels integrated into TensorRT-LLM and FlashInferYeet al.(2025). For calibration (Algorithm2), we sample approximately 1000 sequences from the RULER dataset across different context lengths (4K, 8K, 16K, 32K, 64K) to fit the calibration parametersα\alphaandβ\betafor the threshold relationshipλ=α⋅exp⁡(β⋅S)/L\lambda=\alpha\cdot\exp(\beta\cdot S)/L. For sparsity-aware training, we adopt the curriculum training approach from ProLongGaoet al.(2025a), applying BLASST during the finetuning phase with a fixed sparsity threshold.

For evaluation, we use different sampling strategies depending on the task type. For long-context benchmarks (RULER and LongBench), we use greedy decoding with temperature=0 and perform a single run per example to ensure deterministic and reproducible results. For reasoning tasks that benefit from sampling diversity, we use temperature=0.6 and top-p=0.95. Specifically, we generate 10 samples per problem for MATH500, GPQA, and LiveCodeBench, and 20 samples per problem for AIME 2024 due to its greater difficulty. For these reasoning tasks, we report the best-of-N performance where the final answer is selected using majority voting or self-consistency.

Table 3:Prefill phase comparison on Llama-3.1-8B-Instruct across RULER and LongBench. Best (non-dense) score in each column is denoted in bold. Targeting 50% sparsity, BLASST achieves the best accuracy among all sparse attention methods, closely matching dense attention in addition to being the easiest to use.MethodRULERLongBench4K8K16K32K64KAverageEasyHardShortMediumLongOverallDense Attention96.1695.0794.8092.3387.6993.2129.732.538.328.825.031.4FlexPrefill95.9993.6792.7388.1481.1487.7228.823.824.426.526.225.7MInference96.5494.0691.3785.7983.0384.1528.632.836.730.224.131.2XAttention96.3794.4794.4891.9185.0192.4429.231.538.326.026.930.6BLASST (∼\sim50%)96.1794.7094.6191.8187.0692.8730.732.538.329.825.031.8

Table 4:Decode phase comparison on Qwen3-8B across diverse reasoning and generation tasks. Best (non-dense) score in each column is denoted in bold. Targeting 50% sparsity, BLASST matches or exceeds dense baseline on all benchmarks, including mathematical reasoning (MATH500, AIME 2024), graduate-level science (GPQA), and code generation (LiveCodeBench), while maintaining long-context performance (RULER, LongBench).MethodRULER-32KLongBenchMATH500AIME 2024LiveCodeBenchGPQAAverageDense Attention91.9033.6095.8775.0053.8361.2168.57Quest56.2330.3094.1871.5052.1760.1260.75RocketKV87.8930.6095.8873.5453.1060.5066.91BLASST∼\sim50%91.5533.9096.2376.5054.1561.5168.97

5.2Main Results

Overall Performance.Table2presents the accuracy results of BLASST at 50% and 75% target sparsity levels on Llama-3.1-8B and Qwen3-8B across a diverse set of language benchmarks. We also evaluate larger model variants on LongBench and NIAH (Table9and Table10in AppendixA), and evaluate BLASST on DeepSeek-R1 to demonstrate compatibility with the MLA attention mechanism (Table11in AppendixA). Remarkably, BLASST not only maintains accuracy with minimal degradation but occasionallyoutperformsthe dense baseline. For example, on Qwen3-8B, we observe improvements on MATH500 (96.23 vs. 95.87) and AIME 2024 (76.50 vs 75.00) at 50% sparsity. This counterintuitive result can be attributed to two factors. First, in long-context tasks where information is inherently sparse, pruning low-attention blocks forces the model to concentrate probability mass on the most relevant tokens, effectively acting as implicit denoising. Second, for long-generation reasoning tasks, some intermediate reasoning steps or tokens may be redundant or even detrimentalSuiet al.(2025); by skipping blocks with negligible attention scores, we filter out such distractions, allowing the model to focus on essential reasoning chains. These results show that BLASST is not only computationally efficient, but also improves response quality in certain scenarios.

Prefill Phase Comparison.Table3compares BLASST against state-of-the-art prefill-optimized sparse attention methods on Llama-3.1-8B. Across RULER (4K–64K context lengths) and LongBench, BLASST achieves the best overall accuracy (92.87 RULER average, 31.8 LongBench) among all sparse methods, closely matching dense attention (93.21, 31.4). In particular, BLASST significantly outperforms MInference (84.15 RULER) and FlexPrefill (87.72 RULER), demonstrating the effectiveness of our threshold-based pruning over proxy-based importance estimation.

Decode Phase Comparison.Table4evaluates BLASST on Qwen3-8B across reasoning-intensive tasks. Targeting 50% sparsity, BLASST matches or exceeds dense baseline performance on all benchmarks, while maintaining long-context capabilities. We note that all existing methods employ different optimization strategies and target different deployment scenarios, making direct comparison challenging. We include Quest and RocketKV as reference points to contextualize BLASST’s performance. For instance, RocketKV shows 87.89 RULER and 30.60 LongBench scores, illustrating the trade-offs between aggressive KV cache compression and BLASST’s preservation of critical attention patterns.

Table 5:BLASST speedup over dense baseline on Blackwell and Hopper GPUs. Sparsity values reported are achieved sparsity levels, obtained by varying the threshold (λ\lambda). Prefill: batch size 1, 64K sequence length. B200 decode: batch size 148, 32K sequence length. H200 decode: batch size 128, 16K sequence length.Blackwell (B200)Hopper (H200)SparsitySpeedupSparsitySpeedupPrefill Phase0.0%1.00×\times0.0%1.00×\times38.9%1.25×\times23.8%1.08×\times49.2%1.33×\times49.2%1.27×\times63.0%1.43×\times57.3%1.35×\times71.9%1.52×\times71.0%1.52×\times80.8%1.61×\times79.5%1.64×\times88.9%1.71×\times88.5%1.78×\times94.2%1.77×\times92.0%1.84×\timesDecode Phase0.0%0.98×\times0.0%0.96×\times36.9%1.18×\times23.8%1.08×\times46.7%1.25×\times43.7%1.20×\times61.2%1.34×\times59.4%1.31×\times73.2%1.48×\times70.5%1.40×\times82.6%1.64×\times78.4%1.47×\times87.0%1.71×\times87.5%1.56×\times92.0%1.79×\times——

5.3GPU Kernel Performance

We implement and benchmark highly optimized kernels for both Blackwell (B200) and Hopper (H200) GPU architectures, demonstrating that BLASST achieves substantial real-world speedups. Table5shows performance scaling across increasing sparsity levels for both prefill and decode phases. All speedups are measured against FlashAttention-3 BF16 baselines.

Key Results.At near-lossless accuracy (∼\sim50% target sparsity), we achieve approximately 1.33×\timesspeedup for prefill and 1.25×\timesspeedup for decode on Blackwell. At higher sparsity (∼\sim70%), the speedup increases to 1.52×\timesfor prefill and 1.48×\timesfor decode. On Hopper, prefill achieves up to 1.52×\timesspeedup at 71.0% sparsity. These speedups scale predictably with sparsity: higher sparsity yields greater performance gains, allowing users to choose their preferred accuracy-performance trade-off.

Importantly, we observe no significant performance degradation at 0% sparsity (0.96–1.00×\timesbaseline), verifying the kernels are able to hide the skip check computation behind Tensor Core (prefill) or HBM load (decode) instructions.

For a full picture, the accuracy-performance tradeoffs in Figure5show how effective BLASST is in an inference serving environment. We see a1.1×1.1\timesspeedup in TTFT and TPOT with only a marginal drop in LongBench V1 accuracy.

Figure 5:BLASST shows meaningful end-to-end acceleration at medium to long context lengths. Data shows Qwen3-30B-A3B-Instruct-2507, evaluated on LongBench V1 with H200 and B200 GPUs, collected by stepping target sparsity from 0% to 80%. Average input sequence length is 10K; average output sequence length is 6. Performance is measured in TensorRT-LLM using in-flight batching with concurrency 64, meaning decoding requests may be piggybacked with prefilling requests.

5.4Calibration Results

A key motivation for our calibration approach is that fixed thresholds produce inconsistent sparsity across different context lengths, making deployment unreliable. Table6demonstrates the effectiveness of our calibration method across varying sequence lengths. For a target sparsity of 50%, the fixed threshold approach makes the observed sparsity highly unstable, ranging from 23% at 4K to 75% at 64K, making it impractical for production deployment. In contrast, our calibratedλ=a/L\lambda=a/Lapproach maintains sparsity within a tight range with an average error of only 1.2% from the target. Similar improvements are observed at 70% target sparsity. These results confirm that our calibration enables reliable, predictable sparsity control across diverse sequence lengths without manual tuning.

Table 6:Sparsity stability across context lengths: calibrated vs. fixed threshold on Llama-3.1-8B. Our calibration method maintains consistent sparsity levels across different context lengths, while fixed thresholds produce high variance. Values in parentheses indicate deviation of observed sparsity from the target.Method4K8K16K32K64KTarget Sparsity: 50%Fixedλ=1​e​−3\lambda=110\-323.0937.9252.3865.7274.63(-26.91)(-12.08)(+2.38)(+15.72)(+24.63)Calibratedλ=a/L\lambda=a/L54.2049.7052.2046.9648.75(+4.20)(-0.30)(+2.20)(-3.04)(-1.25)Target Sparsity: 70%Fixedλ=3​e​−3\lambda=310\-342.3557.5469.8379.3684.63(-27.65)(-12.46)(-0.17)(+9.36)(+14.63)Calibratedλ=a/L\lambda=a/L67.9974.6573.6472.5474.63(-2.01)(+4.65)(+3.64)(+2.54)(+4.63)

Beyond context-length stability, we also verify that the calibrated parameteraatransfers across different task types without heavy retuning (Table12in AppendixA).

5.5Sparsity-Aware Training Results

Figure6demonstrates that sparsity-aware training improves the accuracy-sparsity trade-off on RULER benchmarks. At low sparsity levels, sparse-trained models even slightly outperform the dense baseline, suggesting the model learns more robust attention patterns. In the target sparsity range of 50–75%, sparse-trained models achieve substantially better accuracy than applying sparsity training-free, reducing accuracy degradation by up to 1.7×\times. These results confirm that models can be trained to concentrate information in high-scoring attention blocks, making them inherently more compatible with sparse attention patterns and pushing the Pareto frontier of efficient attention.

5.6Ablation Studies

Sparsity Distribution Analysis.Figure7illustrates how sparsity varies across layers and attention heads, revealing the attention patterns produced by the model. We observe substantial heterogeneity: different layers exhibit different sparsity levels, and individual heads within each layer also show significant variance. Crucially, BLASST naturally incorporates this heterogeneity without requiring explicit mechanisms like top-k selection or head pruning—by applying the same threshold across all layers and heads, our method automatically adapts to each layer’s and head’s natural attention distribution, pruning more aggressively where attention is naturally more concentrated and preserving more blocks where attention is more diffuse.

Combination with Other Sparsity Methods.Table7explores the combination of BLASST with other attention sparsity techniques. We find that BLASST can be effectively composed with both prefill-optimized methods (XAttention) and KV cache compression methods (RocketKV). When XAttention (prefill) is combined with BLASST (decode), accuracy degradation remains minimal, demonstrating that the methods are largely orthogonal. Similarly, combining BLASST (prefill) with RocketKV maintains strong performance. These results show that BLASST provides a flexible building block for end-to-end optimization in existing sparse attention pipelines, and show strong potential for composing with other fine-grained channel/head pruning methodsXuet al.(2024).

Figure 6:Sparsity-aware training pushes the accuracy-sparsity frontier. Models fine-tuned with BLASST active during training maintain higher accuracy at aggressive (observed) sparsity levels compared to training-free sparsity application. By training with sparse attention, models learn to concentrate information in high-scoring blocks, making them more robust to pruning.Figure 7:Block sparsity distribution across layers and heads for Llama-8B on 8K context. Taken from NIAH benchmark sample with thresholdλ=0.03\lambda=0.03. Substantial head-level and layer-level variance motivates adaptive thresholding strategies.Table 7:Performance of combining BLASST with other sparsity methods on Qwen 8B. BLASST can be effectively composed with both prefill-optimized methods (XAttention) and KV cache compression methods (RocketKV), providing flexible deployment options. Numbers in parentheses show change from dense baseline.Prefill MethodDecode MethodRULER-16KLongBench-16KDense AttentionDense Attention93.2229.4XAttentionDense Attention92.99 (-0.23)29.1 (-0.3)XAttentionBLASST92.89 (-0.33)28.8 (-0.6)Dense AttentionRocketKV92.72 (-0.50)30.0 (+0.6)BLASSTRocketKV92.60 (-0.62)29.4 (-0.0)

Table 8:Performance on very long sequences with RepoQA benchmark. We evaluate BLASST on code repository understanding tasks at 16K and 200K context lengths. Sparsity (P) and Sparsity (D) denoteachieved sparsityin the prefill and decode phases.ContextAttention ModeSparsity (P)Sparsity (D)AccuracyQwen3-Coder-30B-A3B-Instruct, 16K Context16KFull (Dense)0%0%0.89716KBLASST Prefill64.1%0%0.90416KBLASST Prefill+Decode64.1%48.4%0.882Qwen3-Coder-30B-A3B-Instruct, 200K Context200KFull (Dense)0%0%0.850200KBLASST Prefill57.5%0%0.841200KBLASST Prefill+Decode57.5%40.8%0.838

Very Long Sequence Lengths.We evaluate BLASST on extremely long sequences using the RepoQA benchmarkLiuet al.(2024b). Table8presents results on Qwen3-Coder-30B at 16K and 200K context lengths. At 200K tokens, BLASST achieves a high prefill sparsity (∼\sim58%) with a minimal accuracy drop, and applying sparsity to both prefill and decode phases provides additional computational savings with negligible incremental cost. Notably, longer contexts exhibit higher natural sparsity, making our method increasingly effective for extreme-length scenarios where dense attention becomes impractical.

Figure 8:Accuracy-sparsity trade-off at high achieved sparsity levels on RULER-16K for Qwen3-8B. BLASST shows more stable degradation compared to XAttention, maintaining better accuracy at aggressive sparsity settings. This shows the effectiveness of using actual softmax statistics versus proxy-based importance scores.Extreme Sparsity Analysis.Figure8shows BLASST’s behavior at higher sparsity levels (70–90%) on RULER benchmarks. Compared to XAttention, BLASST shows more stable accuracy degradation across increasing sparsity levels. Although XAttention shows sharper accuracy drops at high sparsity, BLASST’s threshold-based pruning using actual softmax statistics (rather than proxy scores) enables more graceful degradation. This stability makes BLASST more suitable for aggressive sparsity settings where computational efficiency is critical.

Tile Row Reordering.We also investigate whether permuting the tile-row processing order could improve pruning accuracy by establishing a better running maximum earlier. Results in AppendixA(Figure9) show that the effect is dataset-dependent but generally negligible, confirming BLASST’s robustness to processing order.

6Conclusion

We presented BLASST, a simple yet effective sparse attention method that dynamically prunes attention computations by reusing online softmax statistics. BLASST is easy to adopt: it requires no pre-computation or training, supports both prefill and decode phases, is optimized for modern hardware, and is already integrated into multiple inference frameworks. By substantially accelerating the attention mechanism with minimal accuracy degradation, BLASST makes long-context inference significantly more practical. Our automated calibration and sparsity-aware training further enhance its robustness and flexibility, providing a practical foundation for efficient long-context transformers.

Looking forward, we believe that the combination of hardware-aware sparse patterns, learned sparsity through training, and adaptive hybrid methods will be the key to unlocking the full potential of future agentic AI systems.

Acknowledgments

The authors thank InnoMatrix for providing cloud compute resources for kernel benchmarking.

References

• J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat,et al.(2023)GPT-4 technical report.CoRRabs/2303.08774.External Links:DocumentCited by:§1.
• FLASH-D: FlashAttention with hidden softmax division.CoRRabs/2505.14201.External Links:DocumentCited by:§2.1.
• Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong,et al.(2025)LongBench v2: towards deeper understanding and reasoning on realistic long-context multitasks.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,pp. 3639–3664.External Links:DocumentCited by:§5.1.
• P. Behnam, Y. Fu, R. Zhao, P. Tsai, Z. Yu, and A. Tumanov (2025)RocketKV: accelerating long-context LLM inference via two-stage KV cache compression.InProceedings of the 42nd International Conference on Machine Learning,External Links:Link,DocumentCited by:§2.2,§5.1.
• I. Beltagy, M. E. Peters, and A. Cohan (2020)Longformer: the long-document transformer.CoRRabs/2004.05150.External Links:DocumentCited by:§2.1,§2.3.
• R. Child, S. Gray, A. Radford, and I. Sutskever (2019)Generating long sequences with sparse transformers.CoRRabs/1904.10509.External Links:DocumentCited by:§2.1.
• G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen,et al.(2025)Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.CoRRabs/2507.06261.External Links:DocumentCited by:§1.
• T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré (2022)FlashAttention: fast and memory-efficient exact attention with IO-awareness.Advances in Neural Information Processing Systems35,pp. 16344–16359.External Links:Link,DocumentCited by:§1.
• DeepSeek-AI (2025)DeepSeek-V3.2-Exp: boosting long-context efficiency with DeepSeek sparse attention.External Links:LinkCited by:§1,§2.3,§3.3.
• T. Gao, A. Wettig, H. Yen, and D. Chen (2025a)How to train long-context language models (effectively).InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,Vol.1: Long Papers,Vienna, Austria,pp. 7376–7399.External Links:DocumentCited by:§5.1.
• Y. Gao, S. Guo, S. Cao, Y. Xia, Y. Cheng, L. Wang, L. Ma, Y. Sun, T. Ye, L. Dong, H. K. So, Y. Hua, T. Cao, F. Yang, and M. Yang (2025b)SeerAttention-R: Sparse attention adaptation for long reasoning.CoRRabs/2506.08889.External Links:Link,DocumentCited by:§2.1.
• A. Gu and T. Dao (2024)Mamba: linear-time sequence modeling with selective state spaces.InFirst Conference on Language Modeling,External Links:DocumentCited by:§2.3.
• D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi,et al.(2025)DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning.CoRRabs/2501.12948.External Links:DocumentCited by:§1,§5.1.
• C. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg (2024)RULER: what’s the real context size of your long-context language models?.InAdvances in Neural Information Processing Systems,Vol.37,pp. 81829–81847.External Links:DocumentCited by:§5.1.
• H. Jiang, Y. Li, C. Zhang, Q. Wu, X. Luo, S. Ahn, Z. Han, A. H. Abdi, D. Li, C. Lin, Y. Yang, and L. Qiu (2024)MInference 1.0: accelerating pre-filling for long-context LLMs via dynamic sparse attention.InAdvances in Neural Information Processing Systems,Vol.37,pp. 52481–52515.External Links:Link,DocumentCited by:§1,§2.1,§5.1.
• X. Lai, J. Lu, Y. Luo, Y. Ma, and X. Zhou (2025)FlexPrefill: a context-aware sparse attention mechanism for efficient long-sequence inference.CoRRabs/2502.20766.External Links:Link,DocumentCited by:§2.1,§5.1.
• A. Łańcucki, K. Staniszewski, P. Nawrot, and E. M. Ponti (2025)Inference-time hyper-scaling with KV cache compression.CoRRabs/2506.05345.External Links:Link,DocumentCited by:§2.2.
• A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo,et al.(2024a)DeepSeek-V2: a strong, economical, and efficient mixture-of-experts language model.CoRRabs/2405.04434.External Links:DocumentCited by:§4.2.
• J. Liu, J. L. Tian, V. Daita, Y. Wei, Y. Ding, Y. K. Wang, J. Yang, and L. Zhang (2024b)RepoQA: evaluating long context code understanding.InICML 2024 Workshop on Long-Context Foundation Models (LCFM),Vienna, Austria.External Links:Link,DocumentCited by:§5.6.
• M. Oren, M. Hassid, N. Yarden, Y. Adi, and R. Schwartz (2024)Transformers are multi-state RNNs.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp. 18724–18741.External Links:DocumentCited by:§2.2.
• Z. Qiu, Z. Wang, B. Zheng, Z. Huang, K. Wen, S. Yang, R. Men, L. Yu, F. Huang, S. Huang, D. Liu, J. Zhou, and J. Lin (2025)Gated attention for large language models: non-linearity, sparsity, and attention-sink-free.InAdvances in Neural Information Processing Systems,Vol.38.Note:Best Paper AwardExternal Links:Link,DocumentCited by:§2.3.
• B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve (2023)Code Llama: open foundation models for code.CoRRabs/2308.12950.External Links:DocumentCited by:§1.
• J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao (2024)FlashAttention-3: fast and accurate attention with asynchrony and low-precision.InAdvances in Neural Information Processing Systems,Vol.37,pp. 68658–68685.External Links:DocumentCited by:§1.
• P. Singhania, S. Singh, S. He, S. Feizi, and A. Bhatele (2024)Loki: low-rank keys for efficient sparse attention.InAdvances in Neural Information Processing Systems,Vol.37,pp. 16692–16723.External Links:DocumentCited by:§2.2.
• J. Song, D. Jo, Y. Kim, and J. Kim (2025)Reasoning path compression: compressing generation trajectories for efficient LLM reasoning.InAdvances in Neural Information Processing Systems,Vol.38.External Links:Link,DocumentCited by:§2.2.
• Y. Sui, Y. Chuang, G. Wang, J. Zhang, T. Zhang, J. Yuan, H. Liu, A. Wen, S. Zhong, N. Zou,et al.(2025)Stop overthinking: a survey on efficient reasoning for large language models.CoRRabs/2503.16419.External Links:DocumentCited by:§5.2.
• Y. Sun, T. Ye, L. Dong, Y. Xia, J. Chen, Y. Gao, S. Cao, J. Wang, and F. Wei (2025)Rectified sparse attention for efficient long-sequence generation.CoRRabs/2506.04108.External Links:Link,DocumentCited by:§2.2.
• J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han (2024)Quest: query-aware sparsity for efficient long-context LLM inference.InForty-first International Conference on Machine Learning,External Links:DocumentCited by:§2.2,§5.1.
• W. Wu, Y. Wang, G. Xiao, H. Peng, and Y. Fu (2025)Retrieval head mechanistically explains long-context factuality.InThe Thirteenth International Conference on Learning Representations,External Links:LinkCited by:§2.1.
• C. Xiao, P. Zhang, X. Han, G. Xiao, Y. Lin, Z. Zhang, Z. Liu, and M. Sun (2024a)InfLLM: training-free long-context extrapolation for LLMs with an efficient context memory.InAdvances in Neural Information Processing Systems,Vol.37,pp. 119638–119661.External Links:DocumentCited by:§2.2.
• G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y. Fu, and S. Han (2025)DuoAttention: efficient long-context LLM inference with retrieval and streaming heads.InProceedings of the 13th International Conference on Learning Representations,External Links:Link,DocumentCited by:§1,§2.1.
• G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis (2024b)Efficient streaming language models with attention sinks.InThe Twelfth International Conference on Learning Representations,External Links:DocumentCited by:§A.4,§2.2.
• R. Xu, G. Xiao, H. Huang, J. Guo, and S. Han (2025)XAttention: block sparse attention with antidiagonal scoring.CoRRabs/2503.16428.External Links:DocumentCited by:§1,§2.1,§5.1.
• Y. Xu, Z. Jie, H. Dong, L. Wang, X. Lu, A. Zhou, A. Saha, C. Xiong, and D. Sahoo (2024)Think: thinner key cache by query-driven pruning.CoRRabs/2407.21018.Cited by:§5.6.
• A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv,et al.(2025)Qwen3 technical report.CoRRabs/2505.09388.External Links:DocumentCited by:§1.
• L. Yang, Z. Zhang, Z. Chen, Z. Li, and Z. Jia (2026)TidalDecode: fast and accurate LLM decoding with position persistent sparse attention.InThe Fourteenth International Conference on Learning Representations,External Links:Link,DocumentCited by:§2.2.
• Z. Ye, L. Chen, R. Lai, W. Lin, Y. Zhang, S. Wang, T. Chen, B. Kasikci, V. Grover, A. Krishnamurthy, and L. Ceze (2025)FlashInfer: efficient and customizable attention engine for LLM inference serving.InProceedings of the 8th Conference on Machine Learning and Systems,External Links:DocumentCited by:§5.1.
• J. Yuan, H. Gao, D. Dai, J. Luo, L. Zhao, Z. Zhang, Z. Xie, Y. Wei, L. Wang, Z. Xiao,et al.(2025)Native sparse attention: hardware-aligned and natively trainable sparse attention.CoRRabs/2502.11089.External Links:DocumentCited by:§2.3.
• T. Zadouri, M. Hoehnerbach, J. Shah, T. Liu, V. Thakkar, and T. Dao (2026)FlashAttention-4: algorithm and kernel pipelining co-design for asymmetric hardware scaling.CoRRabs/2603.05451.External Links:DocumentCited by:§1,§1.
• M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang,et al.(2020)Big Bird: transformers for longer sequences.InAdvances in Neural Information Processing Systems,Vol.33,pp. 17283–17297.External Links:DocumentCited by:§2.1.
• A. Zeng, X. Lv, Q. Zheng, Z. Hou, B. Chen, C. Xie, C. Wang, D. Yin, H. Zeng, J. Zhang,et al.(2025)GLM-4.5: agentic, reasoning, and coding (ARC) foundation models.CoRRabs/2508.06471.External Links:DocumentCited by:§1.
• J. Zhang, C. Xiang, H. Huang, J. Wei, H. Xi, J. Zhu, and J. Chen (2025)SpargeAttn: accurate sparse attention accelerating any model inference.CoRRabs/2502.18137.External Links:Link,DocumentCited by:§2.1.
• Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, and Z. Wang (2023)H2O: heavy-hitter oracle for efficient generative inference of large language models.InAdvances in Neural Information Processing Systems,Vol.36,pp. 34661–34710.External Links:DocumentCited by:§2.2.

Appendix AAdditional Experimental Results

A.1Large Model Evaluations

To evaluate the scalability and robustness of our method, we measured performance across long-context summarization and retrieval tasks. As shown in Table9and Table10, our method maintains baseline accuracy at extreme sparsity (70–80%) using larger models like Qwen3-30B-A3B-Instruct and Llama-3.1-70B-Instruct.

Table 9:Impact of sparsity on LongBench performance using Qwen3-30B-A3B-Instruct. BLASST maintains accuracy comparable to the dense baseline (0.0 sparsity) even as sparsity increases to 70%, demonstrating robustness in long-context summarization.Target SparsityLongBench V1LongBench V2Overall AccuracyOverall Accuracy0%47.7736.2850%47.4338.1460%47.4739.5370%47.2139.5380%46.5037.2190%45.9737.21Table 10:Accuracy on the RULER hard subset using Llama-3.1-70B-Instruct. The method retains>97%>97\%accuracy on needle-in-a-haystack tasks even at aggressive sparsity levels (up to 80%), confirming effective information retention.Target SparsityRULER-hard-8kRULER-hard-16k0%97.40%99.06%20%97.38%98.98%40%97.31%98.80%60%97.20%98.59%80%97.07%98.28%

A.2MLA Compatibility

Table11demonstrates that BLASST is highly compatible with Multi-Head Latent Attention (MLA). When evaluating DeepSeek-R1 NVFP4 on GPQA Diamond, MMLU Pro, and LiveCodeBench, the model maintains near-baseline accuracy even at 60% sparsity.

Table 11:DeepSeek-R1 NVFP4 using BLASST evaluated on GPQA Diamond, MMLU Pro, and LiveCodeBench at differenttarget sparsitylevels. Minimal accuracy degradation demonstrates that BLASST is compatible with MLA.SparsityGPQA DiamondMMLU ProLiveCodeBench0%0.70710.83020.573550%0.71210.82830.569160%0.71090.82660.5677

A.3Calibration Stability Across Datasets

We evaluate whether the calibrated parameteraatransfers across different task types. Table12reports the achieved sparsity when calibrating on individual dataset subsets with a target sparsity of 50%; this per-dataset breakdown is for illustration purposes only, as in practice we calibrate on a combined, diverse sample dataset. For prefill, all datasets maintain similar achieved sparsity levels, confirming cross-task stability. For decode, two datasets (niah_single and qa) yield noticeably loweraavalues; both tasks involve retrieval-focused decoding where the model attends narrowly to specific relevant spans, producing inherently more concentrated attention distributions that require a smaller threshold to reach the target sparsity. Despite this task-dependent variation inaa, the achieved sparsity remains close to 50% across all datasets, confirming that a single calibration on a mixed dataset is sufficient for robust deployment across diverse workloads.

Table 12:Calibration stability across diverse datasets on Llama-3.1-8B with a target sparsity of 50%. We report the calibrated parameteraa(whereλ=a/L\lambda=a/L) and the resultingachieved sparsityfor both prefill and decode phases. Similar parameter values across tasks confirms that BLASST generalizes without task-specific tuning.DatasetPrefillDecode𝐚\mathbf{a}sparsity𝐚\mathbf{a}sparsityniah_single92049.98%4.649.11%niah_multikey109946.89%11.447.87%niah_multivalue101247.82%11.748.17%niah_multiquery110046.38%9.348.15%cwe102046.69%10.851.04%qa90048.68%5.450.97%

A.4Tile Row Reordering

We investigated whether permuting the tile-row processing order could improve pruning accuracy. This is motivated by the phenomenon observed in StreamingLLMXiaoet al.(2024b), where recent tokens at the end (local window) and sink tokens at the beginning of the sequence tend to have high attention scores. By processing tiles containing the local window first, the running maximummim_{i}can be quickly populated with these high-scoring tokens, establishing a better proxy for the global maximum earlier in the computation. This enables more accurate skip decisions for subsequent blocks. Importantly, BLASST supports such reordering flexibility at the kernel scheduling level with negligible overhead.

Figure9compares standard sequential processing against reordered processing on VT and FWE tasks. The results show dataset-dependent behavior: reordering yields similar performance on VT but provides noticeable improvements on FWE. This suggests that the effectiveness of reordering largely depends on the specific attention patterns of each dataset. Nevertheless, this demonstrates a valuable property of BLASST: the algorithm is robust to different processing orders and can accommodate various optimization strategies. The flexibility to support tile reordering shows the potential for dataset-specific optimizations without requiring fundamental algorithmic changes.

Figure 9:Effect of tile row reordering on the accuracy-sparsity trade-off for Llama 3.1 8B (ctx=8192). We compareStandard Cummax(processing tiles sequentially) withReordered Cummax(processing tiles in reverse order). The plots for both VT and FWE benchmarks show that reordering has a negligible impact on model accuracy at a given sparsity level.

Appendix BError Bound Analysis

We derive an error bound for the output approximation introduced by skipping attention blocks in BLASST.

Consider a single query token with attention output

y=∑j=1Tc∑k=1Bcexp⁡(sj​k−M)​vj​kZ,y=\frac{\sum_{j=1}^{T_{c}}\sum_{k=1}^{B_{c}}\exp(s_{jk}-M)\,v_{jk}}{Z},whereBcB_{c}is the KV block size,sj​ks_{jk}are the attention scores,M=maxj,k⁡sj​kM=\max_{j,k}s_{jk}is the global maximum,Z=∑j,kexp⁡(sj​k−M)Z=\sum_{j,k}\exp(s_{jk}-M)is the softmax normalization constant, andvj​kv_{jk}are the value vectors.

Per-block mass bound.When BLASST skips blockjj, the criterionm~(j)−m(j)<ln⁡λ\tilde{m}^{(j)}-m^{(j)}<\ln\lambdaguaranteesexp⁡(m~(j)−m(j))<λ\exp(\tilde{m}^{(j)}-m^{(j)})<\lambda, wherem~(j)\tilde{m}^{(j)}is the block-local maximum score andm(j)m^{(j)}is the running maximum. Sincem(j)≤Mm^{(j)}\leq M, every score in a skipped block satisfies

exp⁡(sj​k−M)≤exp⁡(m~(j)−M)≤exp⁡(m~(j)−m(j))<λ.\exp(s_{jk}-M)\;\leq\;\exp\!\bigl(\tilde{m}^{(j)}-M\bigr)\;\leq\;\exp\!\bigl(\tilde{m}^{(j)}-m^{(j)}\bigr)\;<\;\lambda.Summing over allBcB_{c}tokens in the block, the total unnormalized attention mass of a single skipped block is

∑k=1Bcexp⁡(sj​k−M)<Bc⋅λ.\sum_{k=1}^{B_{c}}\exp(s_{jk}-M)\;<\;B_{c}\cdot\lambda. Output error bound.Let𝒮\mathcal{S}denote the set of skipped blocks and letVmax=maxj,k⁡‖vj​k‖V_{\max}=\max_{j,k}\|v_{jk}\|. SinceZ≥1Z\geq 1(the element achieving the global maximum contributesexp⁡(0)=1\exp(0)=1), each skipped token’s softmax weight satisfies

pj​k=exp⁡(sj​k−M)Z<λ.p_{jk}\;=\;\frac{\exp(s_{jk}-M)}{Z}\;<\;\lambda.The output error equals the total contribution of skipped tokens:

‖y−y^‖=‖∑j∈𝒮∑k=1Bcpj​k​vj​k‖≤(∑j∈𝒮∑k=1Bcpj​k)⏟δ​Vmax.\|y-\hat{y}\|\;=\;\left\|\sum_{j\in\mathcal{S}}\sum_{k=1}^{B_{c}}p_{jk}\,v_{jk}\right\|\;\leq\;\underbrace{\left(\sum_{j\in\mathcal{S}}\sum_{k=1}^{B_{c}}p_{jk}\right)}_{\delta}V_{\max}.Each skipped token contributes at mostλ​Vmax\lambda\,V_{\max}to this sum. Aggregating over all|𝒮||\mathcal{S}|skipped blocks,

∥y−y^∥≤δVmax<|𝒮|BcλVmax.\boxed{\|y-\hat{y}\|\;\leq\;\delta\,V_{\max}\;<\;|\mathcal{S}|\,B_{c}\,\lambda\,V_{\max}.}In practice, because the approximate outputy^\hat{y}is renormalized over non-skipped blocks only (denominatorZ−Z𝒮Z-Z_{\mathcal{S}}instead ofZZ), a correction of orderδ2​Vmax\delta^{2}V_{\max}arises; this is negligible sinceδ≪1\delta\ll 1.

Appendix CArtifact Appendix

C.1Abstract

This artifact evaluation provides the framework and code necessary to reproduce the kernel-level performance benchmarks for BLASST. The repository focuses on evaluating our custom kernels against a SOTA baseline across both prefill and decode phases. Utilizing automated sweeps across various threshold scale factors, the provided scripts systematically measure exact attention sparsity percentages, execution times, memory bandwidth, and speedups compared to dense baselines. Our work has been integrated into TensorRT-LLM and FlashInfer, and we pull the relevant code from these sources for evaluation. The framework is designed to target and benchmark performance on NVIDIA Hopper (H200) and Blackwell (B200) architectures within a containerized Docker or Singularity environment, handling all necessary installation and measurement.

C.2Artifact check-list (meta-information)

• •Algorithm:BLASST (Skip-Softmax)
• •Compilation:CUDA nvcc builds for kernel templates
• •Binary:Some closed binaries used to measure sparsity
• •Run-time environment:Docker
• •Hardware:H200 and B200 GPUs, many-core x86 CPU, SSD
• •Execution:Python and bash scripts
• •Metrics:Skipping threshold, sparsity, throughput, memory bandwidth
• •Output:Standard output (stdout)
• •Experiments:Single GPU kernel benchmarks and sparsity data collection.
• •How much disk space required:100 GB
• •How much time is needed to prepare workflow:45 minutes
• •How much time is needed to complete experiments:1 hour
• •Publicly available:Yes
• •Code licenses:Apache 2.0
• •Workflow framework used:TensorRT-LLM, FlashInfer
• •Archived:TBD

C.3Description

C.3.1How delivered

The artifact is delivered as an open-source GitHub repository. It can be obtained by cloning the repository and its external submodules viagit clone[email protected]:cameronshinn/blasst-ae-mlsys26.git--recursive.

C.3.2Hardware dependencies

The evaluation requires a host machine equipped with a many-core x86 CPU, an NVIDIA Hopper (H200) GPU, or an NVIDIA Blackwell (B200) GPU (depending on which kernels you want to evaluate). The host system should also have an SSD with approximately 100 GB of available storage space to accommodate the required container images, compiled binaries, and generated benchmark data.

C.3.3Software dependencies

The artifact relies on a containerized run-time environment. The host system must have either Docker (with the NVIDIA Container Toolkit installed) or Singularity available. The provided startup scripts automatically pull and utilize the official TensorRT-LLM release container (nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc6). A compatible Linux host distribution with up-to-date NVIDIA drivers supporting the target Hopper or Blackwell architectures is required.

C.3.4Data sets

The core kernel benchmarks sweep across various threshold scale factors, evaluating throughput, memory bandwidth, and execution time on randomly initialized tensors. The artifact also includes closed sm100 binaries used to measure and collect exact sparsity percentages dynamically during execution.

C.4Installation

To install and prepare the artifact, first clone the repository along with its required submodules:

[email protected]:cameronshinn/blasst-ae-mlsys26.git--recursive

Next, initialize the containerized environment. The repository provides a convenience script to automatically launch the required Docker container (falling back to Singularity if Docker is unavailable) and mount the repository to the/workspacedirectory:

./start_docker.sh

cd/workspace

C.5Experiment workflow

The evaluation workflow is organized by target hardware architecture. For the NVIDIA Hopper architecture, the workflow is further decoupled by attention phase into separate directories. For the NVIDIA Blackwell architecture, both prefill and decode evaluations are consolidated into a single directory. After launching the container and navigating to/workspace, the general workflow proceeds as follows:

1. 1.Navigate to the specific subdirectory corresponding to the available architecture and desired evaluation phase (e.g.,hopper_prefill,hopper_decode, orblackwell).
2. 2.Follow the steps in theREADME.mdfile of that specific subdirectory to compile the kernels and initiate the automated benchmarks.
3. 3.The script will automatically sweep across various threshold scale factors, executing both the BLASST kernels and the dense SOTA baselines.
4. 4.Collected measurements for sparsity, execution time, and memory bandwidth, will be logged directly to standard output.

C.6Evaluation and expected result

We expect our results to align with what’s shown in Table5. EachREADME.mdfile in the folders of our repository contain expected outputs of their associated scripts.

SemiAnalysis (@SemiAnalysis_): Sparse attention mechanisms are finally moving beyond academic benchmarks into production systems, including DeepSeek Sparse Attention, and recently @NousResearch ’s Lighthouse Attention. BLASST by NVIDIA, from paper Dynamic Blocked Attention Sparsity via Softmax Thresholding,