MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
Summary
The paper introduces MISA, a method that applies a mixture-of-experts approach to the indexer heads in sparse attention mechanisms, significantly reducing computational costs for long-context LLM inference while maintaining performance.
View Cached Full Text
Cached at: 05/11/26, 02:43 AM
Paper page - MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
Source: https://huggingface.co/papers/2605.07363
Abstract
MISA replaces the dense token-wise indexing in sparse attention with a routed mixture-of-experts approach that reduces computational cost while maintaining performance and handling long contexts effectively.
DeepSeekSparse Attention(DSA) sets the state of the art for fine-grained inference-timesparse attentionby introducing a learnedtoken-wise indexerthat scores every prefix token and selects the most relevant ones for the main attention. To remain expressive, the indexer uses manyquery heads(for example, 64 on DeepSeek-V3.2) that share the same selected token set; this multi-head design is precisely what makes the indexer the dominant cost on long contexts. We propose MISA (Mixture of IndexerSparse Attention), a drop-in replacement for the DSA indexer that treats its indexer heads as a pool ofmixture-of-experts. A lightweightrouteruses cheapblock-level statisticsto pick a query-dependent subset of only a few active heads, and only those heads run the heavy token-level scoring. This preserves the diversity of the original indexer pool while reducing the per-query cost from scoring every prefix token with every head to scoring it with only a handful of routed heads, plus a negligiblerouterterm computed on a small set of pooled keys. We further introduce ahierarchical variantof MISA that uses the routed pass to keep an enlarged candidate set and then re-ranks it with the original DSA indexer to recover the final selected tokens almost exactly. With only eight active heads and no additional training, MISA matches the dense DSA indexer on LongBench across DeepSeek-V3.2 and GLM-5 while running with eight and four times fewer indexer heads respectively, and outperforms HISA on average. It also preserves fully greenNeedle-in-a-Haystackheatmaps up to a 128K-token context and recovers more than 92% of the tokens selected by the DSA indexer per layer. OurTileLang kerneldelivers roughly a 3.82 times speedup over DSA’s original indexer kernel on a single NVIDIA H200 GPU.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.07363
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.07363 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.07363 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.07363 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Introduces HiLS Attention, a chunk-wise sparse attention mechanism for LLMs that learns chunk selection end-to-end via LM loss, achieving performance comparable to full attention while enabling ultra-long-context extrapolation and faster inference.
MiniMax Sparse Attention
MiniMax Sparse Attention introduces a blockwise sparse attention mechanism that achieves significant speedups for ultra-long-context LLMs, reducing per-token attention compute by 28.4x at 1M context with wall-clock speedups of 14.2x for prefill and 7.6x for decoding on H800 GPUs. The method is accompanied by an open-source inference kernel and a publicly released multimodal model.
RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention
RIS-Kernel introduces a model-agnostic sparse attention architecture (RIS) that reduces self-attention complexity from O(N^2) to O(N log N) for long-context LLM inference, enabling operation on commodity CPU hardware without GPU acceleration.
Accurate and Efficient Long-Term Memory for LLM Agents
MOSAIC is a structured, conflict-aware long-term memory framework for LLM agents that uses entity-typed graph storage, hash-accelerated retrieval, and active conflict detection to achieve high accuracy and efficiency on long-conversation QA and factual conflict detection tasks.
Inference Time Context Sparsity: Illusion or Opportunity?
This paper argues that extreme context sparsity is a principled and feasible foundation for LLM inference, showing that current models tolerate up to 100× sparsity without quality loss and that sparse decode kernels can accelerate processing by 10× on existing hardware.