Tag
AB-Attention提出一种免训练的块稀疏注意力方法，通过块内聚类选择、包含概率加权采样和细粒度核收集来近似稠密注意力，在Wan2.1和HunyuanVideo上跳过89-92%注意力计算，同时PSNR提升最高4.4dB，端到端加速最高2.23倍。
FlashSinkhorn 2 introduces a two-stage block-sparse GPU solver for entropic optimal transport that scales to 1.34e8-particle point clouds, solving them on a single A100 to a marginal residual below 0.01 in under 2.5 hours — claimed to be the largest discrete EOT problem solved at this accuracy within hours. An open-source implementation is released.
This paper studies the mismatch between sequence locality and attention-graph reachability in fixed block-sparse causal attention, formalizing boundary artifacts and proposing diagnostic coverage functions and a minimal repair called Boundary Bridge Attention.
Llama Surgery injects learned block-sparse attention topologies into pre-trained Llama 3.1 8B without retraining from scratch, using a Dynamic Topology Router with Gumbel-Softmax routing, temperature annealing, and a Straight-Through Estimator to avoid gradient collapse, achieving stable convergence and coherent output.
This paper introduces Dynamic Ultrametric Attention, a framework where Transformers learn per-head block-sparse routing topologies during training, which are then offloaded to a custom Triton block-sparse kernel at inference time, achieving up to 28x speedup and 98.4% memory reduction over dense attention.
OpenAI releases block-sparse GPU kernels, a tool for efficient sparse matrix multiplication on GPUs that reduces computation and memory requirements for neural network operations.