simplified-sparse-attention

This paper introduces Simplified Sparse Attention (SSA), a method that uses gist tokens during continued pretraining to enable efficient chunk selection at inference without architectural changes, achieving high compression ratios and outperforming baselines on long-context tasks like LongBench and retrieval-augmented generation.