Sliding-window beats linear attention
Summary
Sliding window attention with sinks outperforms post-trained linear attention on long-context tasks without retraining, offering a cheaper and more reliable inference solution for LLMs.
View Cached Full Text
Cached at: 09/01/26, 11:59 AM
Paper page - Sliding-window beats linear attention
Source: https://huggingface.co/papers/2608.28444
Abstract
Sliding window attention with sinks outperforms post-trained linear attention on long-context tasks without requiring retraining, offering a cheaper and more reliable inference solution.
Due to the nature ofquadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previous one. For each additional token, the keys and values must be stored in memory indefinitely, which is unsustainable. Several alternatives have been proposed to fix the quadratic scaling problem, one of which is retrofitting LLMs to useLinear Attention. This idea has attracted a lot of attention, given its promise to solve the quadratic scaling problem with state-of-the-art performance at low cost. However, this line of research has not been properly compared to simpler baselines. In this work, we show thatSliding Window Attention(SWA) with sinks performs as well or better than post-trainedLinear Attentionmodels. We observe this across multiple LLMs on various downstream tasks. Forlong-context reasoningtasks (Needle-in-a-HaystackandBABILong), SWA achieves massively higher performance (2 to 10 times higher thanlinear attention). SWA requires no post-training, is extremely fast, and requires low memory; therefore, making it an extremely cheap and reliable solution. To reduce inference memory cost, we strongly recommend switching to SWA instead of post-training linear models.Linear attentionmodels may have shown some promise, but they likely require to be trained from scratch or extensive post-training in order to even match SWA.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.28444
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.28444 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.28444 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.28444 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Applying Sliding Window Attention to pretrained LLMs at inference time [P]
This project implements Sliding Window Attention as a reusable inference layer for pretrained Hugging Face LLMs, using bounded KV caches with attention sinks to dramatically reduce memory usage and improve decoding speed. Benchmarks on Qwen2.5-7B show memory dropping from ~923 MB to ~3.5 MB at 16K context, though tasks requiring distant context may degrade.
NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation
Proposes a training-free NLL-guided method for selecting which layers to retain full attention in hybrid attention models, achieving comparable accuracy with 1/4 full-attention layers against a 1/2 periodic baseline on long-context tasks.
GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference
GLIDE introduces a layer-wise adaptive mechanism that strategically integrates sliding-window softmax attention with linear recurrent aggregation for efficient LLM inference, reducing KV cache I/O and latency for long contexts without compromising quality.
Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoning
This paper introduces SWARR, a two-stage recipe using supervised fine-tuning and reinforcement learning to adapt sliding-window attention models for mathematical reasoning, showing that RL can narrow the performance gap with self-attention while maintaining efficiency.
https://x.com/seclink/status/2072187033263784397
Hybrid Sliding Window Attention (Hybrid SWA) is a mixed attention mechanism in long-context language models that balances computational efficiency with full long-range dependencies. By alternating between local SWA layers and global attention layers, it significantly compresses KV cache while maintaining inference capability. This article details its design principles, application in models such as Gemma and Qwen, and best practices in open-source projects like vLLM and HuggingFace.