Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training

Hugging Face Daily Papers Papers
sparse-attention video-audio-generation multimodal efficiency transformer tencent-hunyuan

Summary

Prism提出一种动态稀疏注意力框架，用于原生2K分辨率下联合视频-音频生成模型的训练，通过时空宏区域划分和音频-视频跨模态信号自适应分配注意力块形状，实现相比全注意力2.5倍训练加速且生成质量更优。

Natively training joint video-audio generation models at higher resolutions empowers them to learn richer visual details and sharper motion dynamics. However, full attention incurs quadratic cost and, as resolution increases, spreads attention over increasingly redundant tokens, diluting learning signals for informative content and disrupting pretrained priors. Existing sparse attention methods either target training-free acceleration or overlook the unique structure of joint video-audio data, where cross-modal interactions are inherently concentrated around sound-producing regions. To address this, we propose Prism, a dynamic sparse attention framework for natively training joint video-audio generation models at 2K. In particular, Prism organizes the token sequence into spatiotemporal macro-zones, enabling the attention structure to adapt to local content. For each zone, it estimates local information structure via video feature variance along the channel and feature norms from the audio-to-video cross-attention, jointly capturing how visual content varies directionally and how strongly audio influences each visual region. Based on these signals, Prism dynamically assigns a tailored block shape to each zone, applying finer partitioning along axes of rapid visual content variation and strong audio-visual coupling. This encourages tokens within each block to remain semantically coherent, allowing block-level features to capture both visual content and joint video-audio interaction patterns. Prism further adopts a hybrid block selection strategy to dynamically determine per-query sparsity. Experiments show that Prism achieves 2.5times training speedup compared to full attention, while surpassing it in generation quality.
Original Article
View Cached Full Text

Cached at: 10/06/26, 04:53 AM

Paper page - Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training

Source: https://huggingface.co/papers/2610.05416 Authors:

,

,

,

,

,

,

,

,

,

Abstract

Nativelytrainingjointvideo-audiogenerationmodelsathigherresolutionsempowersthemtolearnrichervisualdetailsandsharpermotiondynamics.However,fullattentionincursquadraticcostand,asresolutionincreases,spreadsattentionoverincreasinglyredundanttokens,dilutinglearningsignalsforinformativecontentanddisruptingpretrainedpriors.Existingsparseattentionmethodseithertargettraining-freeaccelerationoroverlooktheuniquestructureofjointvideo-audiodata,wherecross-modalinteractionsareinherentlyconcentratedaroundsound-producingregions.Toaddressthis,weproposePrism,adynamicsparseattentionframeworkfornativelytrainingjointvideo-audiogenerationmodelsat2K.Inparticular,Prismorganizesthetokensequenceintospatiotemporalmacro-zones,enablingtheattentionstructuretoadapttolocalcontent.Foreachzone,itestimateslocalinformationstructureviavideofeaturevariancealongthechannelandfeaturenormsfromtheaudio-to-videocross-attention,jointlycapturinghowvisualcontentvariesdirectionallyandhowstronglyaudioinfluenceseachvisualregion.Basedonthesesignals,Prismdynamicallyassignsatailoredblockshapetoeachzone,applyingfinerpartitioningalongaxesofrapidvisualcontentvariationandstrongaudio-visualcoupling.Thisencouragestokenswithineachblocktoremainsemanticallycoherent,allowingblock-levelfeaturestocapturebothvisualcontentandjointvideo-audiointeractionpatterns.Prismfurtheradoptsahybridblockselectionstrategytodynamicallydetermineper-querysparsity.ExperimentsshowthatPrismachieves2.5timestrainingspeedupcomparedtofullattention,whilesurpassingitingenerationquality.

View arXiv pageView PDFProject pageGitHub3Add to collection

Get this paper in your agent:

hf papers read 2610\.05416

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### FrancisRing/Prism Image-to-Video• Updatedabout 3 hours ago • 54

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.05416 in a dataset README.md to link it from this page.

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

FrancisRing/Prism

Hugging Face Models Trending

Prism 是一个动态稀疏注意力框架，用于原生训练 2K 分辨率的联合视频-音频生成模型，通过时空宏区块划分与内容自适应的块形状分配，在实现 2.5 倍训练加速的同时超越全注意力的生成质量，由复旦大学与腾讯混元联合发布。