hybrid-sliding-window-attention

Hybrid Sliding Window Attention (Hybrid SWA) is a mixed attention mechanism in long-context language models that balances computational efficiency with full long-range dependencies. By alternating between local SWA layers and global attention layers, it significantly compresses KV cache while maintaining inference capability. This article details its design principles, application in models such as Gemma and Qwen, and best practices in open-source projects like vLLM and HuggingFace.