Tag
The paper presents LightningMoBA, which accelerates Mixture of Block Attention via dynamic relative-threshold routing, KV-centric query grouping for data reuse, and pipelined execution on Hopper GPUs, achieving 1.20–1.60× end-to-end speedups over FlashMoBA with improved RULER scores.
A Reddit post argues that a highly-cited MLSys paper (arXiv:2311.04934) failed to correctly implement its claimed isolated per-chunk KV cache processing, using full causal attention instead — invalidating the experiments that supported its central claim that output quality is unaffected.
The paper diagnoses why sparse attention selectors lose long-context multi-key retrieval accuracy, identifying a small set of 'blind' attention heads, and shows that enabling full attention in just those heads (or using a 4-bit scorer) restores near-full-attention accuracy with minimal TTFT cost across InfLLM-V2, NOSA, SeerAttention-R, NSA, and DashAttention models.
RISA提出了一种面向块级扩散语言模型的可恢复稀疏注意力方法,通过共享读取计划与优先/恢复分层,在不改变骨干网络的情况下,在32K与128K上下文中分别实现最高2.67×和5.53×的注意力加速,同时保持可比的任务性能。
NVIDIA's developer blog presents a practical primer on co-designing model-level sparse attention for efficient long-context inference, using an FP8 kernel to sweep indexer design knobs and compare token-level vs. block-level core attention at 128K token contexts.
Sevren released Embla 1, a phone-optimized reasoning model that runs privately and offline at 4 bits, averaging 57.4 across five benchmarks with 1.77 s end-to-end latency on an iPhone 18 Pro, outperforming larger small models at comparable or lower latency. It uses a custom MoE architecture with sparse attention, shared K/V caches, and a memory table, trained with QAT and multi-stage distillation from Qwen3.5 models.
MMFlash improves multimodal speculative decoding by adaptively upweighting visually dependent tokens and using sparse attention for long video inputs, achieving up to 3.58× lossless decoding speedup and outperforming EAGLE-3 and DFlash.
The paper proposes Latent Sparse Attention (LSA), which hierarchically clusters Keys in latent space so that Queries only need to score against cluster centers. This achieves a 72.4x speedup in Prefill attention and a 42.5x speedup in Decode for a single attention layer of Qwen3.8-27B under 10M ultra-long context, reduces end-to-end first-token latency to about 1/9, and keeps accuracy essentially unchanged. The method includes two variants: distillable D-LSA and natively trained N-LSA, with code and distilled weights open-sourced.
The team shares Latent Sparse Attention (LSA), an open research release enabling subquadratic attention that matches full-attention accuracy at ultra-long context, making sparse attention up to 22.7× faster and full attention up to 72.4× faster.
A Zyphra Research paper introduces Zyphra Hierarchical Attention (ZHA), a subquadratic sparse attention method using a hierarchical tree-structured indexer over the KV cache, which can be adapted to pretrained models via short KL distillation and scales dramatically better beyond 1M context.
A research paper formulating long-context diffusion costs as dual budgets—attention graph edges and activated experts—presenting the DualBudget sparse-expert configuration with graph/compute bounds and evaluations showing gains through 16K context and a 20% cost reduction over DiffuSeq.
A developer shares open-source vLLM patches and Triton kernels that run 8 coding agents with 250K-token contexts concurrently on two RTX 3090s by keeping the full int4 KV cache in system RAM and fetching only ~1% of tokens per attention step via a rank-64 key sketch, reporting ~545-790 tok/s total and 31/32 recall at 250K.
A speculative discussion about the next architectural innovations likely to appear in AI models, including looped transformers, Gated Delta Net 2 variants, attention residuals, and a potential revival of diffusion transformers without quality loss.
HyScale 提出了一种面向长上下文训练的学习型稀疏注意力系统,通过 HyTopK(免分数表的精确 top-k 选择)和选择重放(selection replay)技术,在 1M token 序列上将 GLM-5.1 训练速度提升至 Megatron-LM 的 2.07 倍。
LazyFlash 提出一种前向稠密、后向稀疏的注意力机制,通过梯度感知的块稀疏化在保持模型性能的同时,实现最高 5.7x 的注意力反向加速和 1.7x 的端到端训练加速,适用于长上下文 LLM 微调。
NVIDIA researchers introduce MLIPS, a training-free method that shares a producer layer's broad candidate pool across sparse attention layers, achieving 1.14–1.40× speedups and 14.8–27.4% lower GPU energy on DeepSeek-V4-Flash with minimal accuracy change.
提出 ALLOT:一种训练自由的块稀疏注意力分配方法,通过细胞探测与轨迹校准在延迟预算内优化视频扩散 Transformer 各层-步的稀疏度,在 HunyuanVideo 上实现 11.8 dB PSNR 提升。
MiniMax-M3.1-Flash 是一个原生多模态混合专家(MoE)模型,约 428B 总参数(每 token 激活 23B),支持文本、图像和视频输入及 1M 上下文窗口,面向编码、智能体工作流、计算机使用和网络安全任务,并通过稀疏注意力、量化感知训练和 DSpark 投机解码提升推理效率。
论文提出联合子空间稀疏注意力(JSSA),通过学习共享低维查询-键子空间并设计免gather的索引注意力核,在128K上下文下实现最高6.43倍解码吞吐提升,同时保持接近全KV的精度。
Maxwell-V1.0-Flash is a 35B-total/3B-active Mixture-of-Experts language model from the Maxwell Team with 256K context, featuring Block Sparse Attention, manifold-constrained hyper-connections, and a multilingual tokenizer optimized for Southeast Asian languages, trained on ~26T tokens with a two-stage SFT plus multi-branch RL post-training pipeline.