Faster Video Diffusion with Trainable Sparse Attention

Papers with Code Trending Papers
video-diffusion sparse-attention transformers efficiency acceleration research deep-learning

Summary

This paper introduces Trainable Sparse Attention (VSA), a hardware-efficient sparse attention mechanism that reduces computational costs in video diffusion transformers without compromising performance, enabling more efficient scaling and faster generation.

Scaling video diffusion transformers (DiTs) is limited by their quadratic 3D attention, even though most of the attention mass concentrates on a small subset of positions. We turn this observation into VSA, a trainable, hardware-efficient sparse attention that replaces full attention at both training and inference. In VSA, a lightweight coarse stage pools tokens into tiles and identifies high-weight critical tokens; a fine stage computes token-level attention only inside those tiles subjecting to block computing layout to ensure hard efficiency. This leads to a single differentiable kernel that trains end-to-end, requires no post-hoc profiling, and sustains 85\% of FlashAttention3 MFU. We perform a large sweep of ablation studies and scaling-law experiments by pretraining DiTs from 60M to 1.4B parameters. VSA reaches a Pareto point that cuts training FLOPS by 2.53times with no drop in diffusion loss. Retrofitting the open-source Wan-2.1 model speeds up attention time by 6times and lowers end-to-end generation time from 31s to 18s with comparable quality. These results establish trainable sparse attention as a practical alternative to full attention and a key enabler for further scaling of video diffusion models.
Original Article
View Cached Full Text

Cached at: 08/30/26, 09:26 AM

Paper page - Faster Video Diffusion with Trainable Sparse Attention

Source: https://huggingface.co/papers/2505.13389 Published on May 19, 2025

Abstract

Trainable sparse attention (VSA) reduces computational cost in video diffusion transformers with minimal impact on performance, enabling efficient scaling of the models.

Scalingvideo diffusion transformers(DiTs) is limited by their quadratic 3D attention, even though most of the attention mass concentrates on a small subset of positions. We turn this observation intoVSA, a trainable, hardware-efficient sparse attention that replaces full attention at both training and inference. InVSA, a lightweight coarse stage pools tokens into tiles and identifies high-weight critical tokens; a fine stage computestoken-level attentiononly inside those tiles subjecting toblock computinglayout to ensure hard efficiency. This leads to a single differentiable kernel that trains end-to-end, requires no post-hoc profiling, and sustains 85\% ofFlashAttention3MFU. We perform a large sweep of ablation studies and scaling-law experiments by pretrainingDiTsfrom 60M to 1.4B parameters.VSAreaches a Pareto point that cuts trainingFLOPSby 2.53times with no drop indiffusion loss. Retrofitting the open-sourceWan-2.1 modelspeeds up attention time by 6times and lowers end-to-end generation time from 31s to 18s with comparable quality. These results establish trainable sparse attention as a practical alternative to full attention and a key enabler for further scaling of video diffusion models.

View arXiv pageView PDFGitHub4.16kautoAdd to collection

Get this paper in your agent:

hf papers read 2505\.13389

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper5

#### FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers Text-to-Video• 5B• UpdatedNov 25, 2025 • 92.7k • 68 #### FastVideo/FastWan2.1-T2V-1.3B-Diffusers Text-to-Video• 1B• UpdatedJan 5 • 547 • 22 #### FastVideo/FastWan2.1-T2V-14B-Diffusers UpdatedAug 4, 2025 • 409 • 18 #### FastVideo/Wan2.1-VSA-T2V-14B-720P-Diffusers UpdatedAug 4, 2025 • 16 • 9 Browse 5 models citing this paper## Datasets citing this paper3

#### FastVideo/Wan2.2-Syn-121x704x1280_32k Viewer• UpdatedOct 29, 2025 • 33.3k • 12k • 7 #### Hahshshsshbs/Wan2.2-Syn-121x704x1280_32k Viewer• UpdatedJul 7 • 33.3k • 8.62k • 1 #### johnnyeric/Wan-Syn_77x448x832_600k Viewer• UpdatedNov 27, 2025 • 4.26k • 1.06k

Spaces citing this paper26

Browse 26 spaces citing this paper## Collections including this paper2

Similar Articles

FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation

Hugging Face Daily Papers

FVAttn is a training-free sparse attention system that uses runtime load balancing to improve distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism for video generation, achieving up to 4.41x attention speedup and 2.11x inference speedup over FlashAttention on step-distilled Wan2.2 I2V while maintaining competitive video quality.