sparse-attention

Tag

Cards List
#sparse-attention

Accelerating Mixture of Block Attention via Dynamic Routing, KV-Centric Execution and Runtime Pipelining

Hugging Face Daily Papers ↗ · yesterday Cached

The paper presents LightningMoBA, which accelerates Mixture of Block Attention via dynamic relative-threshold routing, KV-centric query grouping for data reuse, and pipelined execution on Hopper GPUs, achieving 1.20–1.60× end-to-end speedups over FlashMoBA with improved RULER scores.

0 favorites 0 likes
#sparse-attention

I found a MLSys paper with 455 citations with a major error in its implementation that invalidates one of its central claims [R]

Reddit r/MachineLearning ↗ · 2d ago

A Reddit post argues that a highly-cited MLSys paper (arXiv:2311.04934) failed to correctly implement its claimed isolated per-chunk KV cache processing, using full causal attention instead — invalidating the experiments that supported its central claim that output quality is unaffected.

0 favorites 0 likes
#sparse-attention

Lost in a Few Heads: Diagnosing and Repairing Block Selection in Sparse Attention

arXiv cs.LG ↗ · 2d ago Cached

The paper diagnoses why sparse attention selectors lose long-context multi-key retrieval accuracy, identifying a small set of 'blind' attention heads, and shows that enabling full attention in just those heads (or using a 4-bit scorer) restores near-full-attention accuracy with minimal TTFT cost across InfLLM-V2, NOSA, SeerAttention-R, NSA, and DashAttention models.

0 favorites 0 likes
#sparse-attention

RISA: Recoverable Sparse Attention for Block-Diffusion Language Models

arXiv cs.CL ↗ · 2d ago Cached

RISA提出了一种面向块级扩散语言模型的可恢复稀疏注意力方法,通过共享读取计划与优先/恢复分层,在不改变骨干网络的情况下,在32K与128K上下文中分别实现最高2.67×和5.53×的注意力加速,同时保持可比的任务性能。

0 favorites 0 likes
#sparse-attention

Co-Designing Sparse Attention for Efficient Long-Context Inference (17 minute read)

TLDR AI ↗ · 2d ago Cached

NVIDIA's developer blog presents a practical primer on co-designing model-level sparse attention for efficient long-context inference, using an FP8 kernel to sweep indexer design knobs and compare token-level vs. block-level core attention at 128K token contexts.

0 favorites 0 likes
#sparse-attention

Embla 1 (5 minute read)

TLDR AI ↗ · 4d ago Cached

Sevren released Embla 1, a phone-optimized reasoning model that runs privately and offline at 4 bits, averaging 57.4 across five benchmarks with 1.77 s end-to-end latency on an iPhone 18 Pro, outperforming larger small models at comparable or lower latency. It uses a custom MoE architecture with sparse attention, shared K/V caches, and a memory table, trained with QAT and multi-stage distillation from Qwen3.5 models.

0 favorites 0 likes
#sparse-attention

MMFlash: Faster Speculative Decoding for Vision-Language Models

Hugging Face Daily Papers ↗ · 2026-11-13 Cached

MMFlash improves multimodal speculative decoding by adaptively upweighting visually dependent tokens and using sparse attention for long video inputs, achieving up to 3.58× lossless decoding speedup and outperforming EAGLE-3 and DFlash.

0 favorites 0 likes
#sparse-attention

@NFT_Chen: Unbelievable! Qwen 3.8 27B's attention layer has been completely broken through! With 10M ultra-long context, Prefill attention is 72.4x faster and Decode is 42.5x faster! End-to-end first-token latency cut to about 1/9! Attention, anyone with idle GPUs: hand this new method to Claude/Codex, …

X AI KOLs Timeline ↗ · 2026-11-03 Cached

The paper proposes Latent Sparse Attention (LSA), which hierarchically clusters Keys in latent space so that Queries only need to score against cluster centers. This achieves a 72.4x speedup in Prefill attention and a 42.5x speedup in Decode for a single attention layer of Qwen3.8-27B under 10M ultra-long context, reduces end-to-end first-token latency to about 1/9, and keeps accuracy essentially unchanged. The method includes two variants: distillable D-LSA and natively trained N-LSA, with code and distilled weights open-sourced.

0 favorites 0 likes
#sparse-attention

New sparse attention method for ultra-long context

Reddit r/LocalLLaMA ↗ · 2026-11-02 Cached

The team shares Latent Sparse Attention (LSA), an open research release enabling subquadratic attention that matches full-attention accuracy at ultra-long context, making sparse attention up to 22.7× faster and full attention up to 72.4× faster.

0 favorites 0 likes
#sparse-attention

Subquadratic Attention with a Hierarchical Tree-Structured Indexer

arXiv cs.CL ↗ · 2026-11-02 Cached

A Zyphra Research paper introduces Zyphra Hierarchical Attention (ZHA), a subquadratic sparse attention method using a hierarchical tree-structured indexer over the KV cache, which can be adapted to pretrained models via short KL distillation and scales dramatically better beyond 1M context.

0 favorites 0 likes
#sparse-attention

Sparse Text Diffusion under Dual Budgets: Context Connectivity and Expert Activation

arXiv cs.CL ↗ · 2026-11-02 Cached

A research paper formulating long-context diffusion costs as dual budgets—attention graph edges and activated experts—presenting the DualBudget sparse-expert configuration with graph/compute bounds and evaluations showing gains through 16K context and a 20% cost reduction over DiffuSeq.

0 favorites 0 likes
#sparse-attention

8 agents × 250K context, concurrently, on two 3090s (~600+ tok/s total)

Reddit r/LocalLLaMA ↗ · 2026-11-02

A developer shares open-source vLLM patches and Triton kernels that run 8 coding agents with 250K-token contexts concurrently on two RTX 3090s by keeping the full int4 KV cache in system RAM and fetching only ~1% of tokens per attention step via a rank-64 key sketch, reporting ~545-790 tok/s total and 31/32 recall at 250K.

0 favorites 0 likes
#sparse-attention

What exciting architectural moves are we looking to see next in AI models?

Reddit r/LocalLLaMA ↗ · 2026-10-31

A speculative discussion about the next architectural innovations likely to appear in AI models, including looped transformers, Gated Delta Net 2 variants, attention residuals, and a potential revival of diffusion transformers without quality loss.

0 favorites 0 likes
#sparse-attention

HyScale: Scaling Long-Context Training for Learned Sparse Attention

arXiv cs.AI ↗ · 2026-10-26

HyScale 提出了一种面向长上下文训练的学习型稀疏注意力系统,通过 HyTopK(免分数表的精确 top-k 选择)和选择重放(selection replay)技术,在 1M token 序列上将 GLM-5.1 训练速度提升至 Megatron-LM 的 2.07 倍。

0 favorites 0 likes
#sparse-attention

LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning

arXiv cs.CL ↗ · 2026-10-26

LazyFlash 提出一种前向稠密、后向稀疏的注意力机制,通过梯度感知的块稀疏化在保持模型性能的同时,实现最高 5.7x 的注意力反向加速和 1.7x 的端到端训练加速,适用于长上下文 LLM 微调。

0 favorites 0 likes
#sparse-attention

MLIPS: Multi-Layer Indexer Pool Sharing

arXiv cs.LG ↗ · 2026-10-21 Cached

NVIDIA researchers introduce MLIPS, a training-free method that shares a producer layer's broad candidate pool across sparse attention layers, achieving 1.14–1.40× speedups and 14.8–27.4% lower GPU energy on DeepSeek-V4-Flash with minimal accuracy change.

0 favorites 0 likes
#sparse-attention

Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers

Hugging Face Daily Papers ↗ · 2026-10-21 Cached

提出 ALLOT:一种训练自由的块稀疏注意力分配方法,通过细胞探测与轨迹校准在延迟预算内优化视频扩散 Transformer 各层-步的稀疏度,在 HunyuanVideo 上实现 11.8 dB PSNR 提升。

0 favorites 0 likes
#sparse-attention

MiniMaxAI/MiniMax-M3.1-Flash

Hugging Face Models Trending ↗ · 2026-10-17 Cached

MiniMax-M3.1-Flash 是一个原生多模态混合专家(MoE)模型,约 428B 总参数(每 token 激活 23B),支持文本、图像和视频输入及 1M 上下文窗口,面向编码、智能体工作流、计算机使用和网络安全任务,并通过稀疏注意力、量化感知训练和 DSpark 投机解码提升推理效率。

0 favorites 0 likes
#sparse-attention

Joint Subspace Sparse Attention for Efficient Long-Context Decoding

arXiv cs.CL ↗ · 2026-10-15 Cached

论文提出联合子空间稀疏注意力(JSSA),通过学习共享低维查询-键子空间并设计免gather的索引注意力核,在128K上下文下实现最高6.43倍解码吞吐提升,同时保持接近全KV的精度。

0 favorites 0 likes
#sparse-attention

maxwell-model/Maxwell-V1-Flash

Hugging Face Models Trending ↗ · 2026-10-13 Cached

Maxwell-V1.0-Flash is a 35B-total/3B-active Mixture-of-Experts language model from the Maxwell Team with 256K context, featuring Block Sparse Attention, manifold-constrained hyper-connections, and a multilingual tokenizer optimized for Southeast Asian languages, trained on ~26T tokens with a two-stage SFT plus multi-branch RL post-training pipeline.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback