Simplified Sparse Attention via Gist Tokens
Summary
This paper introduces Simplified Sparse Attention (SSA), a method that uses gist tokens during continued pretraining to enable efficient chunk selection at inference without architectural changes, achieving high compression ratios and outperforming baselines on long-context tasks like LongBench and retrieval-augmented generation.
View Cached Full Text
Cached at: 06/29/26, 10:05 PM
Paper page - Simplified Sparse Attention via Gist Tokens
Source: https://huggingface.co/papers/2604.20920
Abstract
Simplified Sparse Attention (SSA) reduces long-context inference costs through gist token-based attention masking during pretraining, enabling efficient chunk selection at inference time without architectural modifications.
Sparse attentioncan reduce the cost of long-context inference, but most variants introduce new architectural components. We introduce SimplifiedSparse Attention(SSA), a simpler approach tosparse attentionthat requires no architectural changes. Concretely, we first performcontinued pretrainingon sequences interleaved withgist tokens. We optimize the standardnext-token lossas usual, but thegist tokensuse anattention maskto restrict what parts of the context the language model can attend to; this teaches the model to pack each chunk’s important information into thegist tokens. At inference time, SSA scores chunks via attention between the current query and the small set ofgist tokens, selectively unfolding the top-k chunks by reintroducing their corresponding raw tokens. Since the query is scored only against thegist tokens, we avoid the memory-bandwidth cost associated with naive scoring against the fullKV cache, without requiring the auxiliaryKV cacheapproach used bysparse attentionmethods. OnLongBench, SSA consistently outperforms compression and inference-time sparse-attention baselines under the same compression ratio. More strikingly, inretrieval-augmented generation, SSA can even outperform full attention aftercontinued pretrainingby over 5.7 points. We attribute this to the ability of SSA’sselective unfolding, which concentrates attention on the query-relevant chunks and effectively filters out noise. SSA further extends to ahierarchical gist-of-gistvariant (H-SSA) that achieveslog-linear decoding complexitywhile maintaining or improving accuracy at high compression ratios up to 32x. The code is available at https://github.com/yuzhenmao/simplified-sparse-attention/.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2604\.20920
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.20920 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2604.20920 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.20920 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
This paper proposes a parameter-free adaptive sparse attention method that uses gzip compression ratios to dynamically select non-redundant blocks for long-range attention, achieving significant perplexity improvements over fixed and learned sparse attention baselines on PG-19 language modeling.
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
SAS introduces a gated sparse attention mechanism that optimizes context ranking end-to-end with language modeling loss, improving performance in reasoning and long-context tasks under tight attention budgets.
Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers
This paper introduces a grammatically-guided sparse attention mechanism for Transformers, aiming to improve efficiency and interpretability by leveraging linguistic structure.
Faster Video Diffusion with Trainable Sparse Attention
This paper introduces Trainable Sparse Attention (VSA), a hardware-efficient sparse attention mechanism that reduces computational costs in video diffusion transformers without compromising performance, enabling more efficient scaling and faster generation.
CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention
CoSA proposes a training-free sparse attention method that co-designs a kernel-aware proxy and an ordered-skipping kernel to accelerate long-context inference, achieving up to 4.93× attention speedup and 2.53× end-to-end TTFT reduction with negligible performance degradation.