Simplified Sparse Attention via Gist Tokens

Hugging Face Daily Papers Papers
simplified-sparse-attention gist-tokens long-context attention-masking efficient-inference retrieval-augmented-generation hierarchical

Summary

This paper introduces Simplified Sparse Attention (SSA), a method that uses gist tokens during continued pretraining to enable efficient chunk selection at inference without architectural changes, achieving high compression ratios and outperforming baselines on long-context tasks like LongBench and retrieval-augmented generation.

Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components. We introduce Simplified Sparse Attention (SSA), a simpler approach to sparse attention that requires no architectural changes. Concretely, we first perform continued pretraining on sequences interleaved with gist tokens. We optimize the standard next-token loss as usual, but the gist tokens use an attention mask to restrict what parts of the context the language model can attend to; this teaches the model to pack each chunk's important information into the gist tokens. At inference time, SSA scores chunks via attention between the current query and the small set of gist tokens, selectively unfolding the top-k chunks by reintroducing their corresponding raw tokens. Since the query is scored only against the gist tokens, we avoid the memory-bandwidth cost associated with naive scoring against the full KV cache, without requiring the auxiliary KV cache approach used by sparse attention methods. On LongBench, SSA consistently outperforms compression and inference-time sparse-attention baselines under the same compression ratio. More strikingly, in retrieval-augmented generation, SSA can even outperform full attention after continued pretraining by over 5.7 points. We attribute this to the ability of SSA's selective unfolding, which concentrates attention on the query-relevant chunks and effectively filters out noise. SSA further extends to a hierarchical gist-of-gist variant (H-SSA) that achieves log-linear decoding complexity while maintaining or improving accuracy at high compression ratios up to 32x. The code is available at https://github.com/yuzhenmao/simplified-sparse-attention/.
Original Article
View Cached Full Text

Cached at: 06/29/26, 10:05 PM

Paper page - Simplified Sparse Attention via Gist Tokens

Source: https://huggingface.co/papers/2604.20920

Abstract

Simplified Sparse Attention (SSA) reduces long-context inference costs through gist token-based attention masking during pretraining, enabling efficient chunk selection at inference time without architectural modifications.

Sparse attentioncan reduce the cost of long-context inference, but most variants introduce new architectural components. We introduce SimplifiedSparse Attention(SSA), a simpler approach tosparse attentionthat requires no architectural changes. Concretely, we first performcontinued pretrainingon sequences interleaved withgist tokens. We optimize the standardnext-token lossas usual, but thegist tokensuse anattention maskto restrict what parts of the context the language model can attend to; this teaches the model to pack each chunk’s important information into thegist tokens. At inference time, SSA scores chunks via attention between the current query and the small set ofgist tokens, selectively unfolding the top-k chunks by reintroducing their corresponding raw tokens. Since the query is scored only against thegist tokens, we avoid the memory-bandwidth cost associated with naive scoring against the fullKV cache, without requiring the auxiliaryKV cacheapproach used bysparse attentionmethods. OnLongBench, SSA consistently outperforms compression and inference-time sparse-attention baselines under the same compression ratio. More strikingly, inretrieval-augmented generation, SSA can even outperform full attention aftercontinued pretrainingby over 5.7 points. We attribute this to the ability of SSA’sselective unfolding, which concentrates attention on the query-relevant chunks and effectively filters out noise. SSA further extends to ahierarchical gist-of-gistvariant (H-SSA) that achieveslog-linear decoding complexitywhile maintaining or improving accuracy at high compression ratios up to 32x. The code is available at https://github.com/yuzhenmao/simplified-sparse-attention/.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2604\.20920

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2604.20920 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2604.20920 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2604.20920 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

New sparse attention method for ultra-long context

Reddit r/LocalLLaMA

The team shares Latent Sparse Attention (LSA), an open research release enabling subquadratic attention that matches full-attention accuracy at ultra-long context, making sparse attention up to 22.7× faster and full attention up to 72.4× faster.