Tag
AoH is a data-free method that identifies retrieval and streaming heads from the spectral geometry of query-key projections, enabling sparse attention without runtime attention scores. At 50% sparsity it retains 96.5% of full-attention performance while reducing prefill/decode latency and KV-cache memory.
OasisKV is a memory-centric LLM inference system that decouples full KV-cache storage from HBM by prefetching sparse, important KV blocks using lookahead tokens from speculative decoding, achieving up to 2.1x throughput gains over dense vLLM with minimal accuracy loss.
Introduces BinaryPC, a training-free hashing-based sparse attention method for long-context LLMs that uses binary principal components to construct hash codes, preserving accuracy while improving decoding throughput by 3.56x over FlashAttention.
ATFlash introduces a per-RoPE-wavelength distance window that prunes query-key inner-product terms proportional to each frequency pair's wavelength, cutting 37-48% of attention compute with minimal quality loss and up to 1.31x speedups on long-context LLM inference.
This paper proposes LTGA, a graph attention layer that learns a per-edge Tsallis entropic index to interpolate between heavy-tailed, softmax, and compact-support attention, offering interpretable sparse attention and competitive performance on graph benchmarks.
Meituan released LongCat-Flash-Lite-Sparse, a sparse-attention model supporting 1M-token context with only ~3B active parameters per token, achieving strong SWE-Bench scores under an MIT license.
ReTopK is a training-free method that accelerates dynamic Top-K sparse attention for long-context LLMs by reusing historical query–support pairs, avoiding full-context scoring and global Top-K selection. It achieves up to 3.07x speedup over Exact Top-K with only 0.50% perplexity increase at 128K context.
CoSA proposes a training-free sparse attention method that co-designs a kernel-aware proxy and an ordered-skipping kernel to accelerate long-context inference, achieving up to 4.93× attention speedup and 2.53× end-to-end TTFT reduction with negligible performance degradation.
Wonder is a general-purpose video world model that enables real-time, camera-controllable world exploration from an image or conditional video. It introduces camera conditioning via dense coordinate fields, a sparse attention memory mechanism, and techniques to improve distillation, allowing minute-scale video generation at 16 FPS.
RIS-Kernel introduces a model-agnostic sparse attention architecture (RIS) that reduces self-attention complexity from O(N^2) to O(N log N) for long-context LLM inference, enabling operation on commodity CPU hardware without GPU acceleration.
This paper proposes a parameter-free adaptive sparse attention method that uses gzip compression ratios to dynamically select non-redundant blocks for long-range attention, achieving significant perplexity improvements over fixed and learned sparse attention baselines on PG-19 language modeling.
This paper tests the assumption that attention weights reveal what a model actually depends on for its output, finding that attention and causal dependence often disagree. They propose using causal evidence sets obtained via intervention masking as supervision for sparse attention routers, achieving near-perfect accuracy on retrieval tasks where attention-distilled routers fail.
Sol-Engine weekly update announces integration of Sol Attention, a training-free sparse attention method for video diffusion, with full paper coming next week.
DWARF-55M-Base is a new language model using a nearly all-sparse attention architecture (DSQG) with a single full causal attention layer, achieving reliable retrieval up to 2048 tokens and extrapolating to 3x that context. It is released as a research prototype for community experimentation.
FVAttn is a training-free sparse attention system that uses runtime load balancing to improve distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism for video generation, achieving up to 4.41x attention speedup and 2.11x inference speedup over FlashAttention on step-distilled Wan2.2 I2V while maintaining competitive video quality.
LiteTopK is a fused indexer-top-k kernel for long-context sparse attention that exploits the curse of dimensionality to reduce memory traffic and improve efficiency, accelerating GLM 5.2 by 1.2x during prefill.
MiniMax has open-sourced its native multimodal large model M3, with approximately 428B total parameters (~23B active), supporting 1M context length, and introducing MiniMax Sparse Attention (MSA) technology to improve long-context efficiency.
Introduces Flash-MSA, the first performant open-source training kernels for MiniMax Sparse Attention on Hopper and Blackwell GPUs, enabling efficient million-token training.
NVIDIA researchers introduced SparDA, a transformer variant that adds a fourth projection (Forecast) to predict next-layer KV blocks, enabling prefetching from CPU memory and reducing selection cost, achieving 1.7x faster decoding and 6.5 point accuracy gain on long reasoning.
MiniCPM-SALA is a 9B-parameter hybrid attention model that interleaves sparse and linear attention to overcome the quadratic compute and large KV cache bottlenecks of long-context Transformers. It achieves 3.5x faster inference than Qwen3-8B at 256K tokens and supports up to 1M tokens on consumer GPUs, with a cost-effective continual training approach that reduces training costs by ~75%.