Dynamic-in-Few-Step: Unifying Dynamic Computation and Few-Step Distillation for Efficient Video Generation

# Dynamic-in-Few-Step: Unifying Dynamic Computation and Few-Step Distillation for Efficient Video Generation
Source: [https://arxiv.org/html/2607.06631](https://arxiv.org/html/2607.06631)
Yu Cheng1,2,3Siyue Yao3,4Zhongang Qi3,†Shanyan Guan3Wei Li3Fajie Yuan2,† 1Zhejiang University2Westlake University3BlueImage, vivo4Xian Jiaotong\-Liverpool University chengyu@westlake\.edu\.cn, qizhongang@vivo\.com, yuanfajie@westlake\.edu\.cn †Corresponding authors

###### Abstract

Video Diffusion Models \(VDMs\) have demonstrated superior generation quality but suffer from prohibitive computational costs\. While recent few\-step distillation techniques significantly accelerate inference, they typically enforce a static model architecture across all denoising stages, ignoring the varying computational demands inherent to different noise levels\. In this work, we propose a novel post\-training acceleration framework that exploits this redundancy by integrating dynamic structural sparsification directly into the distillation process\. Unlike conventional post\-hoc compression applied to a fixed diffusion pipeline, our approach jointly optimizes the denoising steps and structured model sparsity, transforming a pre\-trained VDM into a compact, step\-specific Mixture\-of\-Models \(MoM\)\. To address the training instability arising from this joint optimization, we introduce a Progressive Training Strategy coupled with an Output Rollout Mechanism, which ensures the coherent learning of structural decisions across timesteps\. Furthermore, we develop a specialized inference engine to deploy the resulting MoM efficiently\. Our method is orthogonal to existing acceleration techniques and highly effective: On Wan\-14B, it removes 24% of the per\-step FLOPs on top of 4\-step distillation, adding a 1\.2×\\timeswall\-clock gain and reaching a 30×\\timesspeedup over the 50\-step teacher while preserving competitive generation quality\.

Keywords:Video Generation Speedup · Dynamic Network · Efficient Diffusion

## 1Introduction

Video Diffusion Models \(VDMs\)\[[29](https://arxiv.org/html/2607.06631#bib.bib89),[10](https://arxiv.org/html/2607.06631#bib.bib101),[25](https://arxiv.org/html/2607.06631#bib.bib52),[17](https://arxiv.org/html/2607.06631#bib.bib100)\]have revolutionized video creation, enabling breakthroughs in simulation, advertising, and film production\. However, their practical deployment is severely constrained by high inference latency and substantial computational resource requirements\.

Recent advancements in post\-training acceleration have been significant, focusing on step reduction\[[43](https://arxiv.org/html/2607.06631#bib.bib13),[27](https://arxiv.org/html/2607.06631#bib.bib14),[19](https://arxiv.org/html/2607.06631#bib.bib92)\], attention optimization\[[40](https://arxiv.org/html/2607.06631#bib.bib7),[37](https://arxiv.org/html/2607.06631#bib.bib11)\], quantization\[[15](https://arxiv.org/html/2607.06631#bib.bib18)\], token compression\[[2](https://arxiv.org/html/2607.06631#bib.bib84)\], and streamlined generation\[[36](https://arxiv.org/html/2607.06631#bib.bib128),[13](https://arxiv.org/html/2607.06631#bib.bib129)\]\. Despite notable progress, these methods share a common limitation: they treat the denoising network as a static architecture, applying the same heavy computation uniformly across all timesteps\. However, the diffusion process is inherently heterogeneous\. Existing works\[[6](https://arxiv.org/html/2607.06631#bib.bib131),[42](https://arxiv.org/html/2607.06631#bib.bib127)\]observe that generation proceeds in a coarse\-to\-fine manner, where the model first constructs global structures and later refines high\-frequency details\. This suggests that distinct denoising stages require varying computational capacities, raising a pivotal question:Can we exploit this heterogeneity to dynamically allocate computation across timesteps, removing step\-specific redundancy to accelerate VDMs?

While prior works\[[42](https://arxiv.org/html/2607.06631#bib.bib127),[41](https://arxiv.org/html/2607.06631#bib.bib126)\]have explored dynamic computation, they are typically trained under standard denoising objectives and often become unstable or ineffective when adapted to few\-step distillation\[[42](https://arxiv.org/html/2607.06631#bib.bib127)\]\. Given that few\-step distillation has become the de facto paradigm for efficient VDMs, this incompatibility severely restricts their utility\. Moreover, straightforward decoupled pipelines—either “prune\-then\-distill” or “distill\-then\-prune”—are insufficient to resolve this mismatch\. Specifically, pruning before distillation limits the model capacity required to learn the complex few\-step mapping, while pruning after distillation disrupts the carefully aligned generation trajectory, leading to severe error accumulation, see[Table˜1](https://arxiv.org/html/2607.06631#S5.T1)\.

To translate dynamic computation into practical efficiency, we formulate dynamic pruning and few\-step distillation as a unified optimization problem\. Leveraging distribution matching—a robust objective validated in 3D generation\[[32](https://arxiv.org/html/2607.06631#bib.bib132)\]and attention optimization\[[13](https://arxiv.org/html/2607.06631#bib.bib129),[39](https://arxiv.org/html/2607.06631#bib.bib6)\]—we distill a dynamic structure specifically tailored to the compressed generation trajectory\. This results in a step\-aware Mixture\-of\-Models \(MoM\) framework capable of high\-quality 4\-step inference, where the network architecture is adaptively pruned at each step, inherently bypassing the incompatibility worries associated with post\-hoc distillation\.

Achieving this joint objective is non\-trivial, as simultaneously optimizing for parameter sparsity and step reduction leads to gradient conflicts and training instability\. To overcome this, we introduce a Progressive Training Strategy augmented with an Output Rollout mechanism\. Our core insight is that diffusion is a sequential process where the ultimate objective is the perceptual quality of the final output\. Guided by this, we adopt a reverse\-order curriculum: this strategy first stabilizes the generation quality of the later denoising stages before progressively sparsifying the earlier, noisier stages\. Furthermore, we leverage the output rollout strategy in Distribution Matching Distillation \(DMD\)\[[35](https://arxiv.org/html/2607.06631#bib.bib171),[34](https://arxiv.org/html/2607.06631#bib.bib174)\], enabling supervision based on the final output distribution\. These designs ensure the stable training of highly efficient models\. Finally, to translate theoretical sparsity into practical acceleration, we implement a specialized inference engine that effectively leverages this dynamic mixture of models\.

Our contributions are summarized as follows:

1. 1\.We propose an end\-to\-end framework that jointly optimizes few\-step distillation and dynamic structural sparsification, yielding a step\-aware Mixture\-of\-Models that removes both temporal and parametric redundancy\.
2. 2\.We introduce a progressive training plan combined with an output rollout strategy\. This design effectively coordinates the co\-optimization and ensures convergence to a high\-quality solution\.
3. 3\.Equipped with a specialized inference engine for the MoM architecture, our method removes 24% of per\-step FLOPs on Wan\-14B and adds a 1\.2×\\timesspeedup over 4\-step distillation, giving a 30×\\timesspeedup over the 50\-step teacher with competitive VBench scores\.

## 2Related Work

### 2\.1Efficient Video Diffusion Generation

Acceleration techniques for pretrained Video Diffusion Models can be broadly categorized into training\-free optimization\[[21](https://arxiv.org/html/2607.06631#bib.bib94),[3](https://arxiv.org/html/2607.06631#bib.bib95),[5](https://arxiv.org/html/2607.06631#bib.bib96),[22](https://arxiv.org/html/2607.06631#bib.bib113)\]and post\-training frameworks\. We focus on the latter category in this work\. Existing post\-training approaches explore multiple dimensions of optimization\. Step distillation methods\[[43](https://arxiv.org/html/2607.06631#bib.bib13),[24](https://arxiv.org/html/2607.06631#bib.bib17),[6](https://arxiv.org/html/2607.06631#bib.bib131),[27](https://arxiv.org/html/2607.06631#bib.bib14)\]compress the long sampling trajectory into a few\-step regime \(e\.g\., 4–8 steps\), substantially reducing inference latency\. Attention optimization techniques\[[39](https://arxiv.org/html/2607.06631#bib.bib6),[37](https://arxiv.org/html/2607.06631#bib.bib11),[14](https://arxiv.org/html/2607.06631#bib.bib12)\]alleviate quadratic complexity through linearization or sparsification\. Token reduction strategies\[[2](https://arxiv.org/html/2607.06631#bib.bib84)\]employ fine\-tuning to adapt high\-compression VAEs, effectively shortening spatial\-temporal tokens\. Causal distillation methods\[[13](https://arxiv.org/html/2607.06631#bib.bib129),[36](https://arxiv.org/html/2607.06631#bib.bib128)\]convert slow bidirectional generation into efficient streaming generation\. Quantization\-aware training techniques\[[15](https://arxiv.org/html/2607.06631#bib.bib18)\]utilize low\-bit parameterization to reduce inference overhead\.

These approaches typically treat the denoising network as a static architecture, applying uniform computation across all timesteps\. Our approach offers an orthogonal optimization perspective by targeting the structural redundancy inherent in the multi\-step diffusion process, introducing step\-dependentdynamicoptimization\.

### 2\.2Structural Pruning and Dynamic Networks

Structural pruning removes redundant higher\-level units \(e\.g\., channels, filters, and blocks\) to reduce parameters and FLOPs while preserving accuracy\. Classical structural pruning identifies unimportant units based on magnitude\[[12](https://arxiv.org/html/2607.06631#bib.bib55),[9](https://arxiv.org/html/2607.06631#bib.bib56)\], importance scoring\[[23](https://arxiv.org/html/2607.06631#bib.bib57)\], or gradient\-sensitive criteria\[[28](https://arxiv.org/html/2607.06631#bib.bib58)\], and derives a compact static architecture for deployment\. Dynamic networks extend this paradigm by introducing conditional computation, representative methods like\[[18](https://arxiv.org/html/2607.06631#bib.bib99),[11](https://arxiv.org/html/2607.06631#bib.bib54)\]dynamically adjust network depth or width based on input tokens\. A prominent example is Mixture\-of\-Experts \(MoE\)\[[7](https://arxiv.org/html/2607.06631#bib.bib59),[26](https://arxiv.org/html/2607.06631#bib.bib60)\], which activates subsets of parameters—typically in feed\-forward layers—through learned routing mechanisms\. While effective for scaling model capacity, these token\-dependent dynamic structures impose strict requirements on routing strategies and system\-level implementation to achieve practical wall\-clock acceleration\[[7](https://arxiv.org/html/2607.06631#bib.bib59),[4](https://arxiv.org/html/2607.06631#bib.bib61)\]\.

In diffusion models, the varying noise levels across timesteps introduce a unique dimension of heterogeneity\. DyDiT series\[[42](https://arxiv.org/html/2607.06631#bib.bib127),[41](https://arxiv.org/html/2607.06631#bib.bib126)\]explores this by proposing time\-dependent pruning, learning a specific network structure for each timestep\. As the routing depends only on the diffusion step, efficient batched inference is friendly supported without complex routing implementations\. However, DyDiT is trained under standard multi\-step denoising objectives, lacking the mechanism to co\-optimize with step reduction\. As demonstrated in their original paper\[[42](https://arxiv.org/html/2607.06631#bib.bib127)\], combining it with step distillation leads to model collapse or ineffective training\. This incompatibility with modern few\-step distillation frameworks severely restricts its practical utility\. We address this issue by designing ajoint optimizationmethod\.

### 2\.3Step\-Aware Mixture\-of\-Models

Our method transforms a pretrained VDM into a step\-aware Mixture\-of\-Models \(MoM\), where networks are pruned distinctly for different denoising stages\. Several large\-scale generative models, such as ERNIE\-ViLG 2\.0\[[8](https://arxiv.org/html/2607.06631#bib.bib91)\]and Wan2\.2\[[30](https://arxiv.org/html/2607.06631#bib.bib90)\], also adopt MoM designs by training separate denoisers for different noise intervals\. Their objective is to expand model capacity during pre\-training to improve generation quality\. Similarly, PhasedDMD\[[6](https://arxiv.org/html/2607.06631#bib.bib131)\]introduces a MoM structure within step distillation primarily for increased model expressiveness\.

Conversely, we employ MoM as a post\-training acceleration mechanism\. We exploit the structural heterogeneity across noise levels to compress a pre\-trained model, aiming forefficiencyrather than capacity expansion\.

## 3Preliminaries

We perform step distillation based on the Distribution Matching Distillation method with specific modifications\. Additionally, we apply the tri\-level pruning scheme from\[[33](https://arxiv.org/html/2607.06631#bib.bib19)\]for structural sparsification\. We briefly introduce these methods in this section\.

### 3\.1Modified Distribution Matching Distillation

We employ Decoupled Distribution Matching Distillation \(D\-DMD\)\[[20](https://arxiv.org/html/2607.06631#bib.bib88)\]to distill a multi\-step diffusion model into aTT\-step student generatorGθG\_\{\\theta\}\(we setT=4T=4\)\. The student operates on a discrete set of timesteps𝒯=\{t1,t2,…,tT\}\\mathcal\{T\}=\\\{t\_\{1\},t\_\{2\},\\dots,t\_\{T\}\\\}, wheret1t\_\{1\}denotes the initial highest noise level andtTt\_\{T\}denotes the final lowest noise level prior to the clean data\. The objective is to minimize the Kullback\-Leibler \(KL\) divergence𝒟KL​\(pθ∥preal\)\\mathcal\{D\}\_\{\\text\{KL\}\}\(p\_\{\\theta\}\\\|p\_\{\\text\{real\}\}\)between the student distributionpθp\_\{\\theta\}and the real data distributionprealp\_\{\\text\{real\}\}\. Since direct optimization of this divergence is intractable, DMD\-based methods approximate the gradient∇θ𝒟KL\\nabla\_\{\\theta\}\\mathcal\{D\}\_\{\\text\{KL\}\}with respect to the student parametersθ\\thetausing score estimation \(refer to\[[35](https://arxiv.org/html/2607.06631#bib.bib171)\]for details\)\. Specifically, D\-DMD re\-formulates the gradient by explicitly expanding the teacher’s Classifier\-Free Guidance \(CFG\) term and rearranging the components, resulting the following decoupled form:

∇θ𝒟KL≈𝔼t∈𝒯,τDM,τCA​\[−\(\(𝐬real​\(𝐱τDM,c\)−𝐬fake​\(𝐱τDM,c\)\)⏟Distribution Matching:​ΔDM\+\(α−1\)​\(𝐬real​\(𝐱τCA,c\)−𝐬real​\(𝐱τCA,∅\)\)⏟CFG Augmentation:​ΔCA\)​∂Gθ​\(𝐳t\)∂θ\],\\displaystyle\\nabla\_\{\\theta\}\\mathcal\{D\}\_\{\\scalebox\{0\.6\}\{\\text\{KL\}\}\}\\approx\\mathbb\{E\}\_\{t\\in\\mathcal\{T\},\\tau\_\{\\scalebox\{0\.6\}\{\\text\{DM\}\}\},\\tau\_\{\\scalebox\{0\.6\}\{\\text\{CA\}\}\}\}\\left\[\-\\left\(\\underbrace\{\\left\(\\mathbf\{s\}\_\{\\text\{real\}\}\(\\mathbf\{x\}\_\{\\tau\_\{\\scalebox\{0\.6\}\{\\text\{DM\}\}\}\},c\)\-\\mathbf\{s\}\_\{\\text\{fake\}\}\(\\mathbf\{x\}\_\{\\tau\_\{\\scalebox\{0\.6\}\{\\text\{DM\}\}\}\},c\)\\right\)\}\_\{\\scalebox\{0\.9\}\{\\text\{Distribution Matching:\}\}\\Delta\_\{\\text\{DM\}\}\}\+\\underbrace\{\(\\alpha\-1\)\\left\(\\mathbf\{s\}\_\{\\text\{real\}\}\(\\mathbf\{x\}\_\{\\tau\_\{\\scalebox\{0\.6\}\{\\text\{CA\}\}\}\},c\)\-\\mathbf\{s\}\_\{\\text\{real\}\}\(\\mathbf\{x\}\_\{\\tau\_\{\\scalebox\{0\.6\}\{\\text\{CA\}\}\}\},\\emptyset\)\\right\)\}\_\{\\scalebox\{0\.9\}\{\\text\{CFG Augmentation:\}\}\\Delta\_\{\\text\{CA\}\}\}\\right\)\\frac\{\\partial G\_\{\\theta\}\(\\mathbf\{z\}\_\{t\}\)\}\{\\partial\\theta\}\\right\],

\(1\)
where𝐳t\\mathbf\{z\}\_\{t\}denotes the latent input at few\-step noise levelt∈𝒯t\\in\\mathcal\{T\}, and𝐱τ\\mathbf\{x\}\_\{\\tau\}represents the generator output re\-noised to a continuous noise levelτ\\tau\(τ=0\\tau=0for pure noise,τ=1\\tau=1for clean data\)\. The termsccand∅\\emptysetdenote the text condition and the null condition, respectively\.𝐬real\\mathbf\{s\}\_\{\\text\{real\}\}estimates the score of the real data distribution using the frozen pre\-trained teacher network, and𝐬fake\\mathbf\{s\}\_\{\\text\{fake\}\}approximates the student’s score via a fake network trained concurrently with a denoising objective onGθG\_\{\\theta\}’s generated samples\.

The gradient comprises two components:ΔCA\\Delta\_\{\\text\{CA\}\}encapsulates the CFG pattern \(α\\alphais the CFG guidance scale\), acting as the distillation engine that injects knowledge of teacher model to the student\.ΔDM\\Delta\_\{\\text\{DM\}\}serves as a regularizer, prevents the training process from diverging and ensures the quality of the final output\. Following D\-DMD, we apply decoupled re\-noising schedules:τCA∼𝒰​\[t,1\]\\tau\_\{\\text\{CA\}\}\\sim\\mathcal\{U\}\[t,1\]focuses on refining fine\-grained details in later denoising stages, whereasτDM∼𝒰​\[0,1\]\\tau\_\{\\text\{DM\}\}\\sim\\mathcal\{U\}\[0,1\]provides global regularization across all noise levels\.

Moreover, we remove the ODE trajectory initialization in standard DMD as we found it causes generation instability\. We incorporate the GAN loss suggested in DMD2\[[34](https://arxiv.org/html/2607.06631#bib.bib174)\]to enhance video quality\. Consequently, our modified distillation objective is formulated asℒDMD=𝒟KL\+ℒGAN\\mathcal\{L\}\_\{\\text\{DMD\}\}=\\mathcal\{D\}\_\{\\text\{KL\}\}\+\\mathcal\{L\}\_\{\\text\{GAN\}\}\.

### 3\.2Step\-Aware Tri\-level Structural Pruning

The Video Diffusion Model employed in our work is built upon the Diffusion Transformer \(DiT\) architecture\. Following previous practices\[[33](https://arxiv.org/html/2607.06631#bib.bib19)\], we adopt a tri\-level pruning mechanism operating at the block, attention\-head, and channel levels\. We formulate the pruning scheme using discrete binary gates𝐠∈\{0,1\}\\mathbf\{g\}\\in\\\{0,1\\\}, where0indicates a pruned component and11indicates a retained one\.

To accommodate the sequential nature of few\-step inference, we extend this static pruning into a step\-aware dynamic structure\. Specifically, for each discrete inference stepti∈𝒯t\_\{i\}\\in\\mathcal\{T\}, we maintain a specific pruning scheme𝐠ti\\mathbf\{g\}\_\{t\_\{i\}\}\. Consequently, the gate tensors for the block, head, and feed\-forward network \(FFN\) levels are defined with shapes\[T,Nblock\]\[T,N\_\{\\text\{block\}\}\],\[T,Nhead\]\[T,N\_\{\\text\{head\}\}\], and\[T,Nchannel\]\[T,N\_\{\\text\{channel\}\}\], whereNblockN\_\{\\text\{block\}\},NheadN\_\{\\text\{head\}\}, andNchannelN\_\{\\text\{channel\}\}denote the total number of transformer blocks, multi\-head attention heads, and FFN hidden channels, respectively\.

During training, these discrete gates are applied to the intermediate feature representations via element\-wise multiplication to mask out redundant computations \(see[Fig\.˜1](https://arxiv.org/html/2607.06631#S3.F1)\(a\)\)\. Formally, for a specific architectural componentℱ\\mathcal\{F\}at inference steptit\_\{i\}, the gated output is computed as:

𝐱out=𝐠ti⊙ℱ​\(𝐱in\),\\mathbf\{x\}\_\{\\text\{out\}\}=\\mathbf\{g\}\_\{t\_\{i\}\}\\odot\\mathcal\{F\}\(\\mathbf\{x\}\_\{\\text\{in\}\}\),\(2\)where⊙\\odotdenotes broadcasted element\-wise multiplication\. Once the gates are fully optimized, the resulting sparse architecture can be exported into a dense format\. As each inference step possesses a distinct structural configuration, the final exported model effectively constitutes a compact Mixture\-of\-Models \(MoM\), translating the theoretical sparsity of gates into actual computational speedup during inference\.

![Refer to caption](https://arxiv.org/html/2607.06631v1/x1.png)Figure 1:\(a\)Unified training framework\.We jointly optimize few\-step distribution matching and dynamic structural sparsification, learning step\-adaptive architectures via mask\-based gating\. A single re\-noising levelτ\\tauis shown for simplicity\. \(b\)Output rollout mechanism\.Illustrated usingt1t\_\{1\}as an example: starting from the intermediate noisy state att1t\_\{1\}, the student model is applied iteratively with detached gradients across all subsequent steps to produce the final rollout output𝐱1rollout\\mathbf\{x\}\_\{1\}^\{\\text\{rollout\}\}, which is used to train the fake score network\. \(c\)Specialized inference engine\.Shown withT=2T\{=\}2steps and a model of 7 parameters for clarity\. A unified super\-modelθunion\\theta\_\{\\text\{union\}\}is constructed from the union of all active parameters, while step\-specific index tables dynamically gather the required subsets during inference without redundant weight reloading\.

## 4Method

In this section, we introduce our unified framework\. Our goal is tolearn a step\-adaptive architecture that maximizes computational efficiency while preserving the high\-fidelity generation capability of few\-step distillation\.As illustrated in[Fig\.˜1](https://arxiv.org/html/2607.06631#S3.F1), our framework consists of three key components: \(1\) a joint optimization objective that unifies few\-step distillation with dynamic structural pruning; \(2\) a progressive training strategy augmented with output rollout to stabilize the co\-optimization process; and \(3\) a specialized inference engine designed to fully leverage the acceleration potential of the dynamic\-in\-few\-step MoM architecture\.

### 4\.1Unifying Step Reduction and Dynamic Sparsification

To exploit the temporal redundancy inherent in diffusion generation, we propose learning a dynamic pruning policy that adapts to the specific structural requirements of each denoising step\. Our strategy is to update the pruning gates directly via the few\-step distillation objective\. However, the discrete binary gates𝐠ti\\mathbf\{g\}\_\{t\_\{i\}\}defined in[Section˜3\.2](https://arxiv.org/html/2607.06631#S3.SS2)are non\-differentiable, preventing standard gradient\-based updates\. To resolve this, we introduce continuous learnable structural masks𝐦ti\\mathbf\{m\}\_\{t\_\{i\}\}for each inference stepti∈𝒯t\_\{i\}\\in\\mathcal\{T\}to enable optimization within the distillation framework\.

The mapping from the continuous soft masks𝐦ti\\mathbf\{m\}\_\{t\_\{i\}\}to the discrete hard gates𝐠ti\\mathbf\{g\}\_\{t\_\{i\}\}is tailored to each pruning granularity:

- •Block\-level: the gate is directly determined by an indicator function𝐠ti=𝕀​\(𝐦ti\>0\)\\mathbf\{g\}\_\{t\_\{i\}\}=\\mathbb\{I\}\(\\mathbf\{m\}\_\{t\_\{i\}\}\>0\)\.
- •Attention\-head level: for each head with a feature dimension ofDheadD\_\{\\text\{head\}\}, the entire head is retained \(𝐠ti=1\\mathbf\{g\}\_\{t\_\{i\}\}=1\) if the mean of its correspondingDheadD\_\{\\text\{head\}\}mask parameters is positive\.
- •FFN channel level: We group the hidden channels into sequential chunks of sizeCC; a single scalar mask controls an entire chunk, yielding a block\-wise gate vector𝐠ti=𝕀​\(𝐦ti\>0\)⋅𝟏C\\mathbf\{g\}\_\{t\_\{i\}\}=\\mathbb\{I\}\(\\mathbf\{m\}\_\{t\_\{i\}\}\>0\)\\cdot\\mathbf\{1\}^\{C\}\.

We employ the Straight\-Through Estimator \(STE\) to integrate the hard gates into the computational graph of the distillation task\. Specifically, we construct a differentiable surrogate𝐠~ti\\tilde\{\\mathbf\{g\}\}\_\{t\_\{i\}\}for the forward pass:

𝐠~ti=𝐠ti\+𝐦ti−stop\_grad​\(𝐦ti\)\.\\tilde\{\\mathbf\{g\}\}\_\{t\_\{i\}\}=\\mathbf\{g\}\_\{t\_\{i\}\}\+\\mathbf\{m\}\_\{t\_\{i\}\}\-\\text\{stop\\\_grad\}\(\\mathbf\{m\}\_\{t\_\{i\}\}\)\.\(3\)During training, we substitute the discrete𝐠t\\mathbf\{g\}\_\{t\}with𝐠~t\\tilde\{\\mathbf\{g\}\}\_\{t\}in[Eq\.˜2](https://arxiv.org/html/2607.06631#S3.E2)\. This formulation ensures that the forward pass utilizes the exact discrete gates to mask the corresponding components, while the backward pass routes the distillation task gradients directly to the continuous masks𝐦ti\\mathbf\{m\}\_\{t\_\{i\}\}\.

Furthermore, we introduce a sparsity penaltyℒsparse\\mathcal\{L\}\_\{\\text\{sparse\}\}to encourage structural sparsity and control the overall model capacity\. When optimizing for steptit\_\{i\}, the global retention rate for componentkkcombines the differentiable surrogate𝐠~k,ti\\tilde\{\\mathbf\{g\}\}\_\{k,t\_\{i\}\}and the frozen hard gates𝐠k,tj\\mathbf\{g\}\_\{k,t\_\{j\}\}of other stepstj≠tit\_\{j\}\\neq t\_\{i\}:

ℒsparse​\(𝐠~ti\)=∑k∈\{b,h,f\}ReLU​\(1T​\(μ​\(𝐠~k,ti\)\+∑tj≠tiμ​\(𝐠k,tj\)\)−ηk\),\\mathcal\{L\}\_\{\\text\{sparse\}\}\(\\tilde\{\\mathbf\{g\}\}\_\{t\_\{i\}\}\)=\\sum\_\{k\\in\\\{b,h,f\\\}\}\\text\{ReLU\}\\left\(\\frac\{1\}\{T\}\\left\(\\mu\(\\tilde\{\\mathbf\{g\}\}\_\{k,t\_\{i\}\}\)\+\\sum\_\{t\_\{j\}\\neq t\_\{i\}\}\\mu\(\\mathbf\{g\}\_\{k,t\_\{j\}\}\)\\right\)\-\\eta\_\{k\}\\right\),\(4\)whereμ​\(⋅\)\\mu\(\\cdot\)calculates the spatial mean of the gate tensor for componentkk\(block, head, or FFN\), andηk\\eta\_\{k\}is the corresponding target sparsity ratio\.

The overall training objective combines the distribution matching loss with the sparsity penalty:

ℒtotal=𝔼t∼𝒯​\[ℒDMD​\(θ,𝐠~t\)\+λ​ℒsparse​\(𝐠~t\)\]\.\\mathcal\{L\}\_\{\\text\{total\}\}=\\mathbb\{E\}\_\{t\\sim\\mathcal\{T\}\}\\left\[\\mathcal\{L\}\_\{\\text\{DMD\}\}\(\\theta,\\tilde\{\\mathbf\{g\}\}\_\{t\}\)\+\\lambda\\mathcal\{L\}\_\{\\text\{sparse\}\}\(\\tilde\{\\mathbf\{g\}\}\_\{t\}\)\\right\]\.\(5\)By jointly optimizing the model parametersθ\\thetaand the structural masks𝐦\\mathbf\{m\}, our framework simultaneously uncovers both step and parameter redundancies\. This co\-optimization enables the model to learn specific structural demands across different timestep levels, leading to intriguing architectural findings discussed in[Section˜6](https://arxiv.org/html/2607.06631#S6)\.

### 4\.2Progressive Training with Output Rollout

Directly optimizing the joint objectiveℒtotal\\mathcal\{L\}\_\{\\text\{total\}\}results in unsatisfactory outcomes\. We attribute this to the imbalanced distillation gradient norms across different noise levels\. As the sparsity penalty is applied uniformly, stages with smaller distillation gradients are disproportionately dominated by the sparsity loss, leading to arbitrary over\-pruning rather than learning genuinely step\-adaptive structures\.

To resolve this, we propose a progressive training strategy augmented with an output rollout mechanism\. Our core insight is that diffusion is a sequential process where intermediate errors accumulate, yet the ultimate objective is solely the perceptual quality of the final output\. Therefore, we anchor our co\-optimization to the final generation distribution\.

First, we adopt a reverse\-order curriculum to stabilize the structural search\. We prioritize establishing a robust generation foundation at the later denoising stages \(i\.e\., lower noise levels\) before progressively sparsifying the earlier, noisier stages\. We initially optimize the distill loss and sparsity penalty exclusively fort4t\_\{4\}\(the lowest noise level\)\. Subsequently, we progressively incorporate earlier steps \(t3→t2→t1t\_\{3\}\\rightarrow t\_\{2\}\\rightarrow t\_\{1\}\) into the training pipeline, and apply the sparsity penalty only to the newly introduced step\. Formally, for theTT\-step model, the training proceeds backwards acrossTTprogressive stages\. In Stagekk\(k∈\{1,…,T\}k\\in\\\{1,\\dots,T\\\}\), the active training subset is𝒯k=\{tT,tT−1,…,tT−k\+1\}\\mathcal\{T\}\_\{k\}=\\\{t\_\{T\},t\_\{T\-1\},\\dots,t\_\{T\-k\+1\}\\\}\. The newly introduced step istn​e​w=tT−k\+1t\_\{new\}=t\_\{T\-k\+1\}\. The objective for Stagekkis formulated as:

ℒStage\-​k=𝔼t∼𝒯k​\[ℒDMD​\(θ,𝐠~t\)\]\+λ​ℒsparse​\(𝐠~tn​e​w\)\.\\mathcal\{L\}\_\{\\text\{Stage\-\}k\}=\\mathbb\{E\}\_\{t\\sim\\mathcal\{T\}\_\{k\}\}\\left\[\\mathcal\{L\}\_\{\\text\{DMD\}\}\(\\theta,\\tilde\{\\mathbf\{g\}\}\_\{t\}\)\\right\]\+\\lambda\\mathcal\{L\}\_\{\\text\{sparse\}\}\(\\tilde\{\\mathbf\{g\}\}\_\{t\_\{new\}\}\)\.\(6\)After theTTprogressive training stages, a final stage then jointly trains allTTsteps using the full objectiveℒtotal\\mathcal\{L\}\_\{\\text\{total\}\}\. This curriculum ensures the model secures its output quality before aggressively pruning the highly\-noised input stages\.

Furthermore, we align the training of the fake score network𝐬fake\\mathbf\{s\}\_\{\\text\{fake\}\}with this final\-output\-centric philosophy via an output rollout strategy\. In standard distillation practices,𝐬fake\\mathbf\{s\}\_\{\\text\{fake\}\}is typically trained on the single\-step predicted clean data𝐱^1\\hat\{\\mathbf\{x\}\}\_\{1\}\(computed from the model prediction𝐯t\{\\mathbf\{v\}\}\_\{t\}\) derived directly from an intermediate noisy state\. While effective for static architectures, this intermediate supervision exhibits reduced stability when dynamic structural pruning is introduced\. Therefore, we choose to model the true multi\-step generation outcome for a more reliable constraint\. Specifically, we employ an output rollout mechanism: when the student denoiser operates at an intermediate high\-noise step \(e\.g\.,t1t\_\{1\}\), we iteratively apply the student model—with gradients detached—to sequentially denoise the sample all the way to the final steptTt\_\{T\}\(see[Fig\.˜1](https://arxiv.org/html/2607.06631#S3.F1)\(b\)\)\. This final output𝐱1r​o​l​l​o​u​t\{\\mathbf\{x\}\}\_\{1\}^\{rollout\}is then utilized as the training sample to update𝐬fake\\mathbf\{s\}\_\{\\text\{fake\}\}\. By training the fake model to capture the final generation distribution, we inject a more robust regularization signal, ensuring the stable co\-optimization of our dynamic MoM architecture\.

### 4\.3Specialized Inference Engine for MoM

To translate the learned theoretical sparsity into tangible wall\-clock acceleration, we design a specialized inference engine\. After training, we construct a single unifiedsuper\-modelcontaining the union of all active parameters across all steps:θunion=⋃t∈𝒯\{θi∣mt,i\>0\}\\theta\_\{\\text\{union\}\}=\\bigcup\_\{t\\in\\mathcal\{T\}\}\\\{\\theta\_\{i\}\\mid m\_\{t,i\}\>0\\\}\.

Alongside this super\-model, we maintain a lightweight index tableℐt\\mathcal\{I\}\_\{t\}for each stept∈𝒯t\\in\\mathcal\{T\}, explicitly pointing to the specific subset ofθunion\\theta\_\{\\text\{union\}\}required for that stage\. During inference, the engine efficiently gathers the active dense parameters viaℐt\\mathcal\{I\}\_\{t\}\. This dynamic routing eliminates the latency overhead of reloading different weights and significantly reduces memory consumption compared to storingTTseparate models \(see[Fig\.˜1](https://arxiv.org/html/2607.06631#S3.F1)\(c\)\)\.

## 5Experiments

### 5\.1Implementation Details

Model Architecture and Initialization\.We evaluate our method on Wan2\.1\-14B as the main model\. We also use the smaller Wan2\.1\-1\.3B for baseline comparison, ablation, and architectural analysis, where extensive training sweeps are tractable\. Both models are trained for text\-to\-video generation at480×832480\\times 832, generating 5\-second videos at 16 frames per second \(fps\)\. The teacher and fake\-score network are initialized from the full pre\-trained checkpoint\. The student generator adopts the same initialization but is augmented with our learnable structural masks \(initialized to1\.01\.0\) to enable dynamic pruning\.

Training Configurations\.Our training subset comprises 50,000 high\-quality videos curated from the Koala\-36M dataset\[[31](https://arxiv.org/html/2607.06631#bib.bib182)\]\. We implement the discriminator architecture and GAN loss formulation following\[[1](https://arxiv.org/html/2607.06631#bib.bib93)\]\. The Classifier\-Free Guidance scale is randomly sampled from\[3,5\]\[3,5\]during training\. We optimize the models using AdamW across 16 Kunlun P800 XPUs with a total batch size of 16\. The learning rates are set to1×10−61\\times 10^\{\-6\}for the student model and fake score network, and5×10−75\\times 10^\{\-7\}for the GAN discriminator\. The learning rate for the structural masks warms up from1×10−31\\times 10^\{\-3\}to a peak of1×10−21\\times 10^\{\-2\}\. The total training process spans 2,000 steps\. For sparsity loss, the coefficientλ\\lambdais adaptively determined at each stage by monotonically mapping the corresponding gradient norm to the interval\(0,1\)\(0,1\)\. Additional implementation details are provided in the Appendix[Appendix˜A](https://arxiv.org/html/2607.06631#A1)\.

Progressive Pruning Schedule\.We implement the reverse\-order curriculum described in[Section˜4\.2](https://arxiv.org/html/2607.06631#S4.SS2)using a FLOPs\-driven stage transition criterion\. Specifically, we advance the training to incorporate the next noisier step \(e\.g\., transitioning fromt4t\_\{4\}tot3t\_\{3\}\) whenever the newly introduced step’s FLOPs are reduced by5%5\\%\. We stop training once the average FLOPs across all stepst∈𝒯t\\in\\mathcal\{T\}drop to75%75\\%of the original dense model\.

### 5\.2Evaluation Metrics

We evaluate our method along two dimensions: generation quality and computational efficiency\. For generation quality, we adopt VBench\[[16](https://arxiv.org/html/2607.06631#bib.bib85)\]and report its eight primary dimensions, covering frame\-wise quality \(Imaging Quality, Aesthetic Quality\), temporal quality \(Motion Smoothness, Dynamic Degree, Background Consistency, Subject Consistency, Scene Consistency\), and text alignment \(Overall Consistency\), following\[[14](https://arxiv.org/html/2607.06631#bib.bib12)\]\. The complete 16 metrics and additional qualitative results are provided in the Appendix[Appendix˜B](https://arxiv.org/html/2607.06631#A2)\. We employ augmented prompts provided in VBench and generate five videos per prompt\. For efficiency, we report inference latency \(seconds per video\) measured on a single Kunlun P800 XPU\. The reported runtime corresponds exclusively to the DiT backbone inference, excluding VAE decoding\.

![Refer to caption](https://arxiv.org/html/2607.06631v1/x2.png)Figure 2:Qualitative comparisons on Wan\-14B\.We present our method with the 50\-step Wan\-14B teacher\[[29](https://arxiv.org/html/2607.06631#bib.bib89)\]and the LightX2V acceleration framework\[[3](https://arxiv.org/html/2607.06631#bib.bib95)\]\.
### 5\.3Baselines

Our method exposes a new acceleration axis—per\-step computation redundancy\. Accordingly, our comparisons serve two purposes: first, to test whether this step\-wise pruning can be obtained by a two\-stage pipeline; second, to place our method in the context of existing acceleration frameworks\.

Two\-Stage Optimization Baselines\.These baselines separate step distillation and structural pruning\. They use the same 4\-step distillation recipe as our method, but apply pruning either before or after distillation\. For each setting, we sweep the pruning ratio and report the highest value that does not lead to visual collapse:

Category 1: Prune\-then\-Distill\.The model is first pruned and then distilled into a 4\-step generator\.

\(1\) Static→\\rightarrowDistillation: We apply classical magnitude\-based static pruning\[[12](https://arxiv.org/html/2607.06631#bib.bib55)\]to the FFN layers \(attention head pruning is omitted as we observe it may trigger generation collapse\)\. The sparsity ratio is capped at10%10\\%\.

\(2\) Dynamic→\\rightarrowDistillation: Following DyDiT\[[42](https://arxiv.org/html/2607.06631#bib.bib127)\], we train step\-conditioned masks under the standard denoising objective with a sparsity loss\. The noise schedule is divided into four intervals to match the later 4\-step distillation\.

Category 2: Distill\-then\-Prune\.The model is first distilled into 4 steps and then pruned\.

\(3\) Distillation→\\rightarrowStatic: The reverse pipeline of \(1\), where magnitude\-based pruning is applied to a frozen distilled model\.

\(4\) Distillation→\\rightarrowDynamic: We freeze the pre\-distilled model weights and optimize only the dynamic masks using the same training pipeline described in[Section˜4](https://arxiv.org/html/2607.06631#S4)\.

Distillation Baseline and Teacher\.Ours \(Distill\)is the 4\-step DMD\-distilled model without pruning, trained with the same data and recipe asOurs \(Joint\)\. It is the direct baseline for measuring the extra benefit of our dynamic pruning\. We also report the 50\-step Wan model\[[29](https://arxiv.org/html/2607.06631#bib.bib89)\]as the teacher and quality reference\.

SOTA Acceleration Frameworks\.We report two representative open\-source acceleration frameworks: TurboDiff\[[38](https://arxiv.org/html/2607.06631#bib.bib8)\]\(we evaluate the TurboWan2\.1\-T2V\-1\.3B checkpoint\) and LightX2V\[[3](https://arxiv.org/html/2607.06631#bib.bib95)\]\(the Wan2\.1\-T2V\-14B\-StepDistill\-CfgDistill checkpoint\)\. These methods combine step distillation with other optimizations, such as sparse attention, quantization, and inference\-system acceleration\. Their reported speedups \(marked with∗in[Table˜1](https://arxiv.org/html/2607.06631#S5.T1)\) serve as a holistic reference for the overall acceleration landscape, but they are not direct comparison of our method: we study hidden\-channel pruning, while these frameworks optimize other complementary dimensions\. Because we cannot reproduce their full hardware and deployment environment, we cite their official latency gains; all VBench scores are measured under our own unified evaluation protocol\.

### 5\.4Main Results

Table 1:Quantitative comparison on VBench\.Latency is measured on a single Kunlun P800 XPU; the speedup ratio is computed against the 50\-step teacher*within the same model scale*\. “Δ\\Deltavs\. Distill” isolates the pruning contribution on top of 4\-step distillation\. Speedups marked with∗are taken from the original publications as system\-level references, rather than direct baselines\. VBench scores for all methods are measured under our unified protocol\. Best and second\-best results within each scale arebold/underlined\.MethodLatency\(s\)↓\\downarrowSpeedupvs\. 50\-step↑\\uparrowΔ\\Deltavs\.Distill↑\\uparrowImagingQuality↑\\uparrowAestheticQuality↑\\uparrowMotionSmooth\.↑\\uparrowDynamicDegree↑\\uparrowBackgrnd\.Consist\.↑\\uparrowSubjectConsist\.↑\\uparrowSceneConsist\.↑\\uparrowOverallConsist\.↑\\uparrowWan\-1\.3B \(5 s, 16 fps,480×832480\\times 832\)Wan \(50 step\)325\.841\.00×\\times–67\.0165\.4698\.5265\.1997\.9397\.5645\.0625\.57TurboDiffusion–93\.00×∗\\times^\{\*\}3\.72×∗\\times^\{\*\}69\.4764\.6297\.8086\.1194\.2194\.3443\.3125\.46Static→\\rightarrowDistill12\.9225\.22×\\times1\.01×\\times69\.0566\.4998\.7256\.9495\.9297\.1945\.0625\.57Distill→\\rightarrowStatic12\.9025\.26×\\times1\.01×\\times58\.2163\.4898\.3926\.3995\.1194\.5839\.1024\.88Dynamic→\\rightarrowDistill12\.1426\.84×\\times1\.07×\\times67\.5764\.3798\.7465\.2895\.6895\.3140\.8425\.24Distill→\\rightarrowDynamic12\.4826\.11×\\times1\.04×\\times68\.2355\.6698\.3211\.1192\.5993\.4332\.8522\.40Ours \(Distill\)12\.9325\.20×\\times1\.00×\\times70\.6767\.8198\.6168\.0696\.2696\.3544\.1125\.88Ours \(Joint\)11\.7827\.66×\\times1\.09×\\times71\.3666\.4598\.1780\.5695\.6395\.8043\.6825\.86Wan\-14B \(5 s, 16 fps,480×832480\\times 832\)Wan\-14B \(50 step\)1222\.161\.00×\\times–69\.4366\.0798\.3065\.4698\.0997\.5245\.7525\.91LightX2V–47\.50×∗\\times^\{\*\}1\.90×∗\\times^\{\*\}71\.1568\.6098\.4558\.8096\.7096\.2647\.3126\.17Ours \(Joint\)40\.7230\.01×\\times1\.20×\\times69\.3169\.3698\.9062\.1796\.9897\.1545\.5726\.12

[Table˜1](https://arxiv.org/html/2607.06631#S5.T1)reports the results\. We decompose each method’s speedup into the step\-reduction factor \(relative to the 50\-step teacher\) and the additional pruning factor on top of 4\-step distillation \(“Δ\\Deltavs\. Distill”\)\. Step distillation alone already gives a∼\\sim25×\\timessaving; any further gain must come from making each remaining step cheaper, which theΔ\\Deltacolumn directly measures\.

Speedup on Wan\-14B\.On Wan\-14B, Ours \(Joint\) reduces per\-step FLOPs by 24% and cuts latency from 1222\.16 s \(50\-step teacher\) to 40\.72 s\. This represents a 30\.01×\\timesend\-to\-end speedup over the teacher\. Crucially, while step distillation contributes a 25\.00×\\timesreduction, our dynamic pruning provides an additional 1\.20×\\timeswall\-clock acceleration\. Despite this substantial capacity compression, our method maintains highly competitive VBench scores, confirming that the pruned capacity is genuinely redundant for the few\-step trajectory\.

Decoupled pipelines fail to exploit per\-step redundancy\.As shown in[Table˜1](https://arxiv.org/html/2607.06631#S5.T1), decoupling pruning and distillation fails to deliver a favorable speed\-quality trade\-off\. First,Static→\\rightarrowDistillandDistill→\\rightarrowStaticare limited by the static pruning pattern, yielding a negligible 1\.01×\\timesmarginal speedup\. Second, two\-stage dynamic pipelines similarly falter due to sequential optimization conflicts\. Applying pruning after distillation \(Distill→\\rightarrowDynamic\) severely disrupts the delicately aligned few\-step generation trajectory, resulting in a precipitous degradation in temporal dynamics, most notably Dynamic Degree\. Conversely, applying pruning prior to distillation \(Dynamic→\\rightarrowDistill\) restricts the representational capacity essential for effectively learning the highly compressed 4\-step mapping\. Consequently, it suffers from noticeable performance drops in metrics like Imaging Quality and Dynamic Degree\. Beyond quantitative metrics, our supplementary videos reveal that this variant generates visually inferior texture details and exhibits diminished motion amplitude\. In contrast, our unified framework resolves these capacity bottlenecks by jointly optimizing step reduction and dynamic pruning\. This co\-design achieves superior acceleration while preserving high\-fidelity spatial details and rich, coherent temporal dynamics \(further analysis in Appendix[Appendix˜D](https://arxiv.org/html/2607.06631#A4)confirms the high dynamic degree is free from artifact\-driven flickering\)\.

Comparison with SOTA frameworks\.We position our method alongside integrated open\-source acceleration frameworks, TurboDiffusion and LightX2V, to contextualize its performance\. As demonstrated in[Table˜1](https://arxiv.org/html/2607.06631#S5.T1), our method preserves competitive generation quality across VBench dimensions\. It is important to emphasize that our approach is not designed to directly compete with these holistic, system\-level pipelines, which bundle distillation with sparse attention, quantization, and infrastructure\-level engineering\. Instead, our contribution uncovers an orthogonal algorithmic acceleration axis: the per\-step parametric redundancy along the hidden\-channel dimension, driven by the coarse\-to\-fine dynamics of diffusion\. Because our MoM exports standard dense architectures for each step, it is structurally compatible with these broader optimizations\. We therefore position our method as complementary to, rather than competing with, such pipelines, with the potential to be combined for further gains\.

![Refer to caption](https://arxiv.org/html/2607.06631v1/x3.png)Figure 3:Qualitative ablation of our training techniques\.The top row shows the stable generation of our fullOurs \(Joint\)framework\. The subsequent three groups represent the ablated settings\. For each ablation, we display two states: the generation quality at its maximum viable sparsity \(Upper\), and the catastrophic collapse when pushed beyond this threshold \(Lower\)\. The sparsity ratio \(calculated as pruned parameters over total parameters, averaged across 4 steps for step\-aware models\) is denoted on the right\.
### 5\.5Ablation Study

To validate the efficacy of our proposed training techniques, we conduct a comprehensive ablation study\. Qualitative results are illustrated in[Fig\.˜3](https://arxiv.org/html/2607.06631#S5.F3), with the corresponding quantitative bounds detailed in Appendix[Appendix˜C](https://arxiv.org/html/2607.06631#A3)\. Our experiments include:

Step\-Specific vs\. Shared Mask\.We ablate the step\-aware dynamic structure by enforcing ashared maskacross all inference steps\.

Effect of Progressive Training\.We remove the reverse\-order curriculum, directly optimizing all steps jointly from the very beginning \(i\.e\., exclusively using the Final Stage setup\)\.

Effect of Output Rollout\.We replace our output rollout mechanism and use the standard intermediate prediction𝐱^1\\hat\{\\mathbf\{x\}\}\_\{1\}for the fake score network training\.

It can be observed that the step\-specific configuration demonstrates superiority over the shared mask, as it generally achieves higher sparsity ratios and preserves basic structural fidelity\. Furthermore, a consistent observation across all ablated settings is the existence of a strict sparsity wall\. Without our tailored techniques, the models fail to reach high sparsity levels; forcing the optimization further leads to a cliff\-like drop in structural integrity and a rapid collapse in generation quality\. While the ablated models max out at a stable sparsity of only around 2\.3%, our full framework stably achieves a 20\.1% sparsity without visual degradation, highlighting the absolute indispensability of our training strategy for stabilizing the dynamic MoM co\-optimization\.

## 6Exploration and Findings

![Refer to caption](https://arxiv.org/html/2607.06631v1/x4.png)Figure 4:Heatmap of the learned dynamic pruning policy\.We visualize the retention ratios of different architectural components \(Self\-Attention, Cross\-Attention, and FFN\) across different network depths and inference phases\.By jointly optimizing the structural masks and the distillation objective, our framework automatically discovers how to allocate computation across denoising steps\. Beyond the efficiency gains, the learned policy reveals interpretable structure in how redundancy is distributed both*temporally*\(across steps\) and*spatially*\(across components and depth\), as visualized in[Fig\.˜4](https://arxiv.org/html/2607.06631#S6.F4)for the Wan\-1\.3B model\. The Wan\-14B model follows a similar pattern\.

U\-Shaped Computational Allocation\.Temporally, the learned architecture exhibits a distinct “U\-shaped” capacity demand\. The model retains the highest computational footprint at the two ends of the generation trajectory \(88\.4% and 89\.1% FLOPs retention at𝐭1\\mathbf\{t\}\_\{1\}and𝐭4\\mathbf\{t\}\_\{4\}, respectively\) while aggressively pruning the intermediate steps \(78\.5% and 77\.1% at𝐭2\\mathbf\{t\}\_\{2\}and𝐭3\\mathbf\{t\}\_\{3\}\)\. We further verified \([Fig\.˜8](https://arxiv.org/html/2607.06631#A3.F8)\(e\)–\(h\)\) that the U\-shape emerges progressively during training and persists across a broad range of sparsity levels\. The pattern mirrors the physics of diffusion generation: the initial phase needs ample capacity to construct the global layout from pure noise, the final phase to render high\-frequency visual details, while intermediate phases mostly transition the established semantics and can be heavily compressed\.

Semantic Freezing in Deep Layers\.Spatially, we observe that deeper layers are pruned much more aggressively than shallower layers, especially within the Self/Cross\-Attention modules during𝐭2\\mathbf\{t\}\_\{2\}and𝐭3\\mathbf\{t\}\_\{3\}\. In Transformer\-based diffusion models, deeper layers typically process high\-level semantic representations, while shallower layers handle low\-level textures\. This pruning pattern suggests a semantic freezing phenomenon: high\-level semantics are rapidly decided and frozen in the very early stages of inference\. Consequently, the network automatically learns to shut down the deep\-layer semantic updates in subsequent steps, allocating its limited computational budget to the shallow layers for fine\-grained texture refinement\.

## 7Conclusion

We presented an end\-to\-end framework that unifies few\-step distillation and dynamic structural pruning for accelerating Video Diffusion Models\. By treating step reduction and parameter sparsification as a single joint optimization, we derive a step\-aware Mixture\-of\-Models that allocates computation across denoising stages, and we keep this optimization stable with a reverse\-order progressive curriculum and an output\-rollout fake\-score objective\. On Wan\-14B, our method achieves an 30×\\timesspeedup while maintaining satisfied visual quality and temporal dynamics\. We believe our unified framework and architectural findings will pave the way for designing fundamentally more efficient generative models in the future\.

## References

- \[1\]\(2025\)Flash diffusion: accelerating any conditional diffusion model for few steps image generation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 15686–15695\.Cited by:[§5\.1](https://arxiv.org/html/2607.06631#S5.SS1.p2.7)\.
- \[2\]J\. Chen, W\. He, Y\. Gu, Y\. Zhao, J\. Yu, J\. Chen, D\. Zou, Y\. Lin, Z\. Zhang, M\. Li,et al\.\(2025\)DC\-videogen: efficient video generation with deep compression video autoencoder\.arXiv preprint arXiv:2509\.25182\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.
- \[3\]L\. Contributors\(2025\)LightX2V: light video generation inference framework\.GitHub\.Note:[https://github\.com/ModelTC/lightx2v](https://github.com/ModelTC/lightx2v)Cited by:[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1),[Figure 2](https://arxiv.org/html/2607.06631#S5.F2),[§5\.3](https://arxiv.org/html/2607.06631#S5.SS3.p10.1)\.
- \[4\]D\. Dai, C\. Deng, C\. Zhao, R\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu,et al\.\(2024\)Deepseekmoe: towards ultimate expert specialization in mixture\-of\-experts language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1280–1297\.Cited by:[§2\.2](https://arxiv.org/html/2607.06631#S2.SS2.p1.1)\.
- \[5\]T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré\(2022\)FlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.
- \[6\]X\. Fan, Z\. Qiu, Z\. Wu, F\. Wang, Z\. Lin, T\. Ren, D\. Lin, R\. Gong, and L\. Yang\(2025\)Phased dmd: few\-step distribution matching distillation via score matching within subintervals\.arXiv preprint arXiv:2510\.27684\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2607.06631#S2.SS3.p1.1)\.
- \[7\]W\. Fedus, B\. Zoph, and N\. Shazeer\(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§2\.2](https://arxiv.org/html/2607.06631#S2.SS2.p1.1)\.
- \[8\]Z\. Feng, Z\. Zhang, X\. Yu, Y\. Fang, L\. Li, X\. Chen, Y\. Lu, J\. Liu, W\. Yin, S\. Feng,et al\.\(2023\)Ernie\-vilg 2\.0: improving text\-to\-image diffusion model with knowledge\-enhanced mixture\-of\-denoising\-experts\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10135–10145\.Cited by:[§2\.3](https://arxiv.org/html/2607.06631#S2.SS3.p1.1)\.
- \[9\]D\. Filters’Importance\(2016\)Pruning filters for efficient convnets\.arXiv preprint arXiv:1608\.087103\.Cited by:[§2\.2](https://arxiv.org/html/2607.06631#S2.SS2.p1.1)\.
- \[10\]Y\. Gao, H\. Guo, T\. Hoang, W\. Huang, L\. Jiang, F\. Kong, H\. Li, J\. Li, L\. Li, X\. Li,et al\.\(2025\)Seedance 1\.0: exploring the boundaries of video generation models\.arXiv preprint arXiv:2506\.09113\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p1.1)\.
- \[11\]Y\. Han, Z\. Liu, Z\. Yuan, Y\. Pu, C\. Wang, S\. Song, and G\. Huang\(2024\)Latency\-aware unified dynamic networks for efficient image recognition\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(12\),pp\. 7760–7774\.Cited by:[§2\.2](https://arxiv.org/html/2607.06631#S2.SS2.p1.1)\.
- \[12\]Y\. He, X\. Zhang, and J\. Sun\(2017\)Channel pruning for accelerating very deep neural networks\.InProceedings of the IEEE international conference on computer vision,pp\. 1389–1397\.Cited by:[§2\.2](https://arxiv.org/html/2607.06631#S2.SS2.p1.1),[§5\.3](https://arxiv.org/html/2607.06631#S5.SS3.p4.2)\.
- \[13\]X\. Huang, Z\. Li, G\. He, M\. Zhou, and E\. Shechtman\(2025\)Self forcing: bridging the train\-test gap in autoregressive video diffusion\.arXiv preprint arXiv:2506\.08009\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p2.1),[§1](https://arxiv.org/html/2607.06631#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.
- \[14\]Y\. Huang, X\. Ge, R\. Gong, C\. Lv, and J\. Zhang\(2025\)Linvideo: a post\-training framework towards o \(n\) attention in efficient video generation\.arXiv preprint arXiv:2510\.08318\.Cited by:[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1),[§5\.2](https://arxiv.org/html/2607.06631#S5.SS2.p1.1)\.
- \[15\]Y\. Huang, R\. Gong, J\. Liu, Y\. Ding, C\. Lv, H\. Qin, and J\. Zhang\(2025\)Qvgen: pushing the limit of quantized video generative models\.arXiv preprint arXiv:2505\.11497\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.
- \[16\]Z\. Huang, Y\. He, J\. Yu, F\. Zhang, C\. Si, Y\. Jiang, Y\. Zhang, T\. Wu, Q\. Jin, N\. Chanpaisit,et al\.\(2024\)Vbench: comprehensive benchmark suite for video generative models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 21807–21818\.Cited by:[§5\.2](https://arxiv.org/html/2607.06631#S5.SS2.p1.1)\.
- \[17\]W\. Kong, Q\. Tian, Z\. Zhang, R\. Min, Z\. Dai, J\. Zhou, J\. Xiong, X\. Li, B\. Wu, J\. Zhang,et al\.\(2024\)Hunyuanvideo: a systematic framework for large video generative models\.arXiv preprint arXiv:2412\.03603\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p1.1)\.
- \[18\]T\. Lawson and L\. Aitchison\(2025\)Learning to skip the middle layers of transformers\.arXiv preprint arXiv:2506\.21103\.Cited by:[§2\.2](https://arxiv.org/html/2607.06631#S2.SS2.p1.1)\.
- \[19\]S\. Lin, X\. Xia, Y\. Ren, C\. Yang, X\. Xiao, and L\. Jiang\(2025\)Diffusion adversarial post\-training for one\-step video generation\.arXiv preprint arXiv:2501\.08316\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p2.1)\.
- \[20\]D\. Liu, P\. Gao, D\. Liu, R\. Du, Z\. Li, Q\. Wu, X\. Jin, S\. Cao, S\. Zhang, H\. Li,et al\.\(2025\)Decoupled dmd: cfg augmentation as the spear, distribution matching as the shield\.arXiv preprint arXiv:2511\.22677\.Cited by:[§3\.1](https://arxiv.org/html/2607.06631#S3.SS1.p1.11)\.
- \[21\]F\. Liu, S\. Zhang, X\. Wang, Y\. Wei, H\. Qiu, Y\. Zhao, Y\. Zhang, Q\. Ye, and F\. Wan\(2024\)Timestep embedding tells: it’s time to cache for video diffusion model\.arXiv preprint arXiv:2411\.19108\.Cited by:[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.
- \[22\]C\. Lu, Y\. Zhou, F\. Bao, J\. Chen, C\. Li, and J\. Zhu\(2022\)Dpm\-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps\.Advances in Neural Information Processing Systems35,pp\. 5775–5787\.Cited by:[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.
- \[23\]P\. Molchanov, S\. Tyree, T\. Karras, T\. Aila, and J\. Kautz\(2016\)Pruning convolutional neural networks for resource efficient inference\.arXiv preprint arXiv:1611\.06440\.Cited by:[§2\.2](https://arxiv.org/html/2607.06631#S2.SS2.p1.1)\.
- \[24\]W\. Nie, J\. Berner, N\. Ma, C\. Liu, S\. Xie, and A\. Vahdat\(2026\)Transition matching distillation for fast video generation\.arXiv preprint arXiv:2601\.09881\.Cited by:[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.
- \[25\]OpenAI\(2024\)Sora\.Note:Accessed: \[2024\]External Links:[Link](https://openai.com/index/sora/)Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p1.1)\.
- \[26\]N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean\(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.arXiv preprint arXiv:1701\.06538\.Cited by:[§2\.2](https://arxiv.org/html/2607.06631#S2.SS2.p1.1)\.
- \[27\]Y\. Sun, J\. Wu, Y\. Cao, C\. Xu, Y\. Wang, W\. Cao, D\. Luo, C\. Wang, and Y\. Fu\(2025\)Swiftvideo: a unified framework for few\-step video generation through trajectory\-distribution alignment\.arXiv preprint arXiv:2508\.06082\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.
- \[28\]L\. Theis, I\. Korshunova, A\. Tejani, and F\. Huszár\(2018\)Faster gaze prediction with dense networks and fisher pruning\.arXiv preprint arXiv:1801\.05787\.Cited by:[§2\.2](https://arxiv.org/html/2607.06631#S2.SS2.p1.1)\.
- \[29\]T\. Wan, A\. Wang, B\. Ai, B\. Wen, C\. Mao, C\. Xie, D\. Chen, F\. Yu, H\. Zhao, J\. Yang,et al\.\(2025\)Wan: open and advanced large\-scale video generative models\.arXiv preprint arXiv:2503\.20314\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p1.1),[Figure 2](https://arxiv.org/html/2607.06631#S5.F2),[§5\.3](https://arxiv.org/html/2607.06631#S5.SS3.p9.1)\.
- \[30\]T\. Wan, A\. Wang, B\. Ai, B\. Wen, C\. Mao, C\. Xie, D\. Chen, F\. Yu, H\. Zhao, J\. Yang, J\. Zeng, J\. Wang, J\. Zhang, J\. Zhou, J\. Wang, J\. Chen, K\. Zhu, K\. Zhao, K\. Yan, L\. Huang, M\. Feng, N\. Zhang, P\. Li, P\. Wu, R\. Chu, R\. Feng, S\. Zhang, S\. Sun, T\. Fang, T\. Wang, T\. Gui, T\. Weng, T\. Shen, W\. Lin, W\. Wang, W\. Wang, W\. Zhou, W\. Wang, W\. Shen, W\. Yu, X\. Shi, X\. Huang, X\. Xu, Y\. Kou, Y\. Lv, Y\. Li, Y\. Liu, Y\. Wang, Y\. Zhang, Y\. Huang, Y\. Li, Y\. Wu, Y\. Liu, Y\. Pan, Y\. Zheng, Y\. Hong, Y\. Shi, Y\. Feng, Z\. Jiang, Z\. Han, Z\. Wu, and Z\. Liu\(2025\)Wan: open and advanced large\-scale video generative models\.arXiv preprint arXiv:2503\.20314\.Cited by:[§2\.3](https://arxiv.org/html/2607.06631#S2.SS3.p1.1)\.
- \[31\]Q\. Wang, Y\. Shi, J\. Ou, R\. Chen, K\. Lin, J\. Wang, B\. Jiang, H\. Yang, M\. Zheng, X\. Tao,et al\.\(2025\)Koala\-36m: a large\-scale video dataset improving consistency between fine\-grained conditions and video content\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 8428–8437\.Cited by:[§5\.1](https://arxiv.org/html/2607.06631#S5.SS1.p2.7)\.
- \[32\]Z\. Wang, C\. Lu, Y\. Wang, F\. Bao, C\. Li, H\. Su, and J\. Zhu\(2023\)Prolificdreamer: high\-fidelity and diverse text\-to\-3d generation with variational score distillation\.Advances in neural information processing systems36,pp\. 8406–8441\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p4.1)\.
- \[33\]Y\. Wu, Y\. Li, A\. Kag, I\. Skorokhodov, W\. Menapace, K\. Ma, A\. Sahni, J\. Hu, A\. Siarohin, D\. Sagar,et al\.\(2025\)Taming diffusion transformer for efficient mobile video generation in seconds\.arXiv preprint arXiv:2507\.13343\.Cited by:[§3\.2](https://arxiv.org/html/2607.06631#S3.SS2.p1.3),[§3](https://arxiv.org/html/2607.06631#S3.p1.1)\.
- \[34\]T\. Yin, M\. Gharbi, T\. Park, R\. Zhang, E\. Shechtman, F\. Durand, and W\. T\. Freeman\(2024\)Improved distribution matching distillation for fast image synthesis\.arXiv preprint arXiv:2405\.14867\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p5.1),[§3\.1](https://arxiv.org/html/2607.06631#S3.SS1.p5.1)\.
- \[35\]T\. Yin, M\. Gharbi, R\. Zhang, E\. Shechtman, F\. Durand, W\. T\. Freeman, and T\. Park\(2023\)One\-step diffusion with distribution matching distillation\.arXiv preprint arXiv:2311\.18828\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p5.1),[§3\.1](https://arxiv.org/html/2607.06631#S3.SS1.p1.11)\.
- \[36\]T\. Yin, Q\. Zhang, R\. Zhang, W\. T\. Freeman, F\. Durand, E\. Shechtman, and X\. Huang\(2025\)From slow bidirectional to fast autoregressive video diffusion models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 22963–22974\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.
- \[37\]J\. Zhang, H\. Wang, K\. Jiang, S\. Yang, K\. Zheng, H\. Xi, Z\. Wang, H\. Zhu, M\. Zhao, I\. Stoica,et al\.\(2025\)SLA: beyond sparsity in diffusion transformers via fine\-tunable sparse\-linear attention\.arXiv preprint arXiv:2509\.24006\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.
- \[38\]J\. Zhang, K\. Zheng, K\. Jiang, H\. Wang, I\. Stoica, J\. E\. Gonzalez, J\. Chen, and J\. Zhu\(2025\)TurboDiffusion: accelerating video diffusion models by 100\-200 times\.arXiv preprint arXiv:2512\.16093\.Cited by:[§5\.3](https://arxiv.org/html/2607.06631#S5.SS3.p10.1)\.
- \[39\]P\. Zhang, Y\. Chen, H\. Huang, W\. Lin, Z\. Liu, I\. Stoica, E\. Xing, and H\. Zhang\(2025\)Vsa: faster video diffusion with trainable sparse attention\.arXiv preprint arXiv:2505\.13389\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.
- \[40\]P\. Zhang, Y\. Chen, R\. Su, H\. Ding, I\. Stoica, Z\. Liu, and H\. Zhang\(2025\)Fast video generation with sliding tile attention\.arXiv preprint arXiv:2502\.04507\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p2.1)\.
- \[41\]W\. Zhao, Y\. Han, J\. Tang, K\. Wang, H\. Luo, Y\. Song, G\. Huang, F\. Wang, and Y\. You\(2026\)DyDiT\+\+: diffusion transformers with timestep and spatial dynamics for efficient visual generation\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.06631#S2.SS2.p2.1)\.
- \[42\]W\. Zhao, Y\. Han, J\. Tang, K\. Wang, Y\. Song, G\. Huang, F\. Wang, and Y\. You\(2024\)Dynamic diffusion transformer\.arXiv preprint arXiv:2410\.03456\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p2.1),[§1](https://arxiv.org/html/2607.06631#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.06631#S2.SS2.p2.1),[§5\.3](https://arxiv.org/html/2607.06631#S5.SS3.p5.1)\.
- \[43\]K\. Zheng, Y\. Wang, Q\. Ma, H\. Chen, J\. Zhang, Y\. Balaji, J\. Chen, M\. Liu, J\. Zhu, and Q\. Zhang\(2025\)Large scale diffusion distillation via score\-regularized continuous\-time consistency\.arXiv preprint arXiv:2510\.08431\.Cited by:[§1](https://arxiv.org/html/2607.06631#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.06631#S2.SS1.p1.1)\.

Appendix

This supplementary material provides further details and comprehensive results to support the main paper\. It is organised as follows:

- •Section[A](https://arxiv.org/html/2607.06631#A1)details the extended implementation hyperparameters and training configurations\.
- •Section[B](https://arxiv.org/html/2607.06631#A2)presents the complete set of VBench evaluation metrics alongside additional qualitative visualisations\.
- •Section[C](https://arxiv.org/html/2607.06631#A3)provides the quantitative results for our ablation studies\.
- •Section[D](https://arxiv.org/html/2607.06631#A4)illustrates the training dynamics, the continuous quality\-vs\.\-sparsity trade\-off, and an in\-depth analysis of our final model architecture\.

![Refer to caption](https://arxiv.org/html/2607.06631v1/x5.png)Figure 5:Qualitative comparison with baselines\.Visual results demonstrating the generation quality of our proposed method compared against various baseline configurations\. More example videos for these baselines are provided in the supplementary video material\.## Appendix AExtended Implementation Details

Due to space constraints in the main text, we provide the complete hyperparameter settings and training configurations here\.

Dynamic Sparsity Penalty Weighting\.As briefly discussed in the main text, we dynamically adjust the sparsity penalty weightλ\\lambdafor each denoising step to balance the distillation and pruning objectives\. Specifically, we maintain an Exponential Moving Average \(EMA\) of the task gradient norm, denoted asntn\_\{t\}, for each steptt\. To prevent the sparsity penalty from being overshadowed by large distillation gradients, we map this EMA value to a monotonically increasing scalar in the range\(0,1\)\(0,1\)using the following smooth normalization function:

λt=nt\+ϵnt\+ϵ\+1,\\lambda\_\{t\}=\\frac\{n\_\{t\}\+\\epsilon\}\{n\_\{t\}\+\\epsilon\+1\},\(7\)whereϵ=10−6\\epsilon=10^\{\-6\}is a small constant added for numerical stability\.

FFN Reordering and Chunking\.For the FFN channel\-level pruning, we set the chunk size toC=8C=8\. We perform a pre\-processing step before chunking: we reorder the FFN hidden channels of the pre\-trained base model according to theL2L\_\{2\}norm of their corresponding weights\. This norm\-based sorting maximizes the intra\-chunk homogeneity, allowing the single scalar mask to effectively control the entire chunk without sacrificing representational capacity\.

Memory Optimization and Hardware\.The model is trained using Fully Sharded Data Parallel \(FSDP\) combined with mixed\-precision training \(e\.g\., bfloat16\)\. Additionally, we utilize gradient checkpointing to trade computation for memory, ensuring the joint optimization fits within the GPU memory limits\. The training is conducted on 16 Kunlun P800 \(96GB\) XPUs\.

Table 2:Remaining VBench metrics on Wan 1\.3 B\.The best and second\-best results are highlighted inboldandunderlined, respectively\. Visual comparisons are shown in Figure[5](https://arxiv.org/html/2607.06631#A0.F5)\.BaselineLatency\(s\)↓\\downarrowSpeedupratio↑\\uparrowObjectClass↑\\uparrowMultipleObjects↑\\uparrowColor↑\\uparrowSpatialRelation\.↑\\uparrowTemporalStyle↑\\uparrowHumanAction↑\\uparrowTemporalFlicker\.↑\\uparrowAppearanceStyle↑\\uparrowTotalScore↑\\uparrowWan \(50 step\)325\.841\.00×\\times88\.8174\.8389\.2073\.0423\.1394\.0099\.5521\.8183\.31Static→\\rightarrowDistill12\.9225\.22×\\times87\.9780\.2688\.1979\.2924\.5195\.0099\.0719\.6282\.96Distill→\\rightarrowStatic12\.9025\.26×\\times80\.7069\.8976\.4475\.6723\.5893\.0098\.5221\.4977\.65Dynamic→\\rightarrowDistill12\.1426\.84×\\times88\.2972\.5682\.4672\.2824\.2694\.0098\.8519\.2782\.01Distill→\\rightarrowDynamic12\.4826\.11×\\times81\.3359\.8386\.9769\.7522\.2289\.0098\.1620\.5775\.57Ours \(Distill\)12\.9325\.20×\\times87\.9080\.5682\.1474\.8925\.2596\.0098\.8519\.3983\.64Ours \(Joint\)11\.7827\.66×\\times82\.6776\.5285\.0775\.8724\.8396\.0098\.2419\.7783\.62

## Appendix BFull Evaluation Metrics and Additional Visualizations

The VBench suite comprehensively evaluates video generation models across multiple dimensions, including temporal quality \(Subject Consistency, Background Consistency, Temporal Flickering, Motion Smoothness, Dynamic Degree\), frame\-wise quality \(Imaging Quality, Aesthetic Quality\), semantic consistency \(Object Class, Multiple Objects, Human Action, Color, Spatial Relationship, Scene\), style consistency \(Appearance Style, Temporal Style\), and text alignment \(Overall Consistency\)\.

In the main paper, we reported 8 core dimensions from the VBench suite\. Here, we present the remaining 8 VBench metrics in Table[2](https://arxiv.org/html/2607.06631#A1.T2), along with the Total Score that reflects the comprehensive generation quality aggregated across all 16 dimensions\.

As shown in Table[2](https://arxiv.org/html/2607.06631#A1.T2), consistent with the findings in the main text, our joint optimization method \(Ours \(Joint\)\) achieves a highly competitive Total Score of 83\.62, closely matching our dense distillation baseline \(Ours \(Distill\), 83\.64\) while reaching a 27\.66×\\timesend\-to\-end speedup over the 50\-step teacher on Wan\-1\.3B\. Additional qualitative results ofOurs \(Joint\)are provided in Figure[6](https://arxiv.org/html/2607.06631#A2.F6)and Figure[7](https://arxiv.org/html/2607.06631#A2.F7)\.

![Refer to caption](https://arxiv.org/html/2607.06631v1/x6.png)Figure 6:Additional qualitative results of Ours \(Joint\)\.The text prompts are randomly sampled from the VBench evaluation suite, showcasing the model’s robustness across diverse scenarios without cherry\-picking\.![Refer to caption](https://arxiv.org/html/2607.06631v1/x7.png)Figure 7:Qualitative results on Wan 14B\.Visualisations demonstrate that our method maintains high generation quality with significant inference speedup on large\-scale VDMs\. Inference is conducted on a single P800 XPU using BF16 precision\.
## Appendix CQuantitative Ablation

Ablated variants of our dynamic MoM framework experience rapid generation collapse before reaching meaningful sparsity ratios\.[Table˜3](https://arxiv.org/html/2607.06631#A3.T3)provides the quantitative metrics of these failures on Wan\-1\.3B\.

The results clearly demonstrate that without our proposed progressive training and output rollout mechanisms, the optimization hits a rigid “sparsity wall” at roughly 2\.0%–2\.5%\. Pushing beyond this threshold results in collapsed models that suffer severe degradation across core metrics, particularly in Imaging Quality, Aesthetic Quality, and Dynamic Degree\. While our main experiments demonstrate that joint training is the key to unlocking dynamic acceleration, the ablation study reveals that such co\-optimization is highly non\-trivial\. It highlights the absolute indispensability of our progressive curriculum and rollout strategies for stabilizing the dynamic MoM co\-optimization\.

Table 3:Quantitative ablation on Wan\-1\.3B\.These are the collapsed results beyond the∼\\sim2\.3% stability wall\.VariantSparsityImg\.Q\.↑\\uparrowAes\.Q\.↑\\uparrowMot\.Sm\.↑\\uparrowDyn\.Deg\.↑\\uparrowBg\.Cons\.↑\\uparrowSubj\.Cons\.↑\\uparrowSceneCons\.↑\\uparrowOver\.Cons\.↑\\uparrowOurs \(Joint\)20\.1%71\.3666\.4598\.1780\.5695\.6395\.8043\.6825\.86\(A\) Shared Mask7\.1%56\.1949\.9697\.4356\.6791\.5584\.7127\.1923\.27\(B\) w/o Progressive Training8\.4%52\.4147\.8397\.1836\.9490\.2780\.4924\.6622\.75\(C\) w/o Output Rollout6\.8%54\.8849\.1297\.3641\.2291\.0482\.3625\.3123\.04

![Refer to caption](https://arxiv.org/html/2607.06631v1/x8.png)Figure 8:Evolution of training dynamics and structural analysis\.\(a\)\-\(d\) show the fluctuation of key generation metrics and the steady decrease in computational cost over training steps\. \(e\)\-\(h\) illustrate the increasing sparsity across four different denoising steps,t1t\_\{1\}presents the highest noise level\. \(i\)\-\(l\) provide a detailed component\-level breakdown of the retained structures in our final selected model\.
## Appendix DExtended Findings

We track the evolution of the model’s structural sparsity and its corresponding generation quality throughout the training process, as visualized in Figure[8](https://arxiv.org/html/2607.06631#A3.F8)\(on Wan 1\.3B\)\.

Training Dynamics and the Sparsity\-Quality Trade\-off\.We record the training dynamics starting from the Final State training\. As the training progresses, the structural sparsity continuously increases across all denoising steps \(Figure[8](https://arxiv.org/html/2607.06631#A3.F8)\(e\)\-\(h\)\), leading to a steady decline in the global computation cost \(Figure[8](https://arxiv.org/html/2607.06631#A3.F8)\(d\)\)\. However, thanks to our joint distillation strategy, the generation metrics do not degrade monotonically alongside the parameter reduction\. Instead, the model maintains a highly competitive generation capability, with key metrics such as Imaging Quality and Dynamic Degree exhibiting stable fluctuations and even reaching peak performance at specific intermediate stages \(Figure[8](https://arxiv.org/html/2607.06631#A3.F8)\(a\)\-\(c\)\)\.

Identifying a Favorable Sparsity\-Quality Trade\-off\.Through this trajectory, we observe a practical capacity threshold that yields a highly favorable balance between computational efficiency and generation quality\. The model demonstrates excellent performance when the average FLOPs ratio is reduced to approximately 83%\. At this specific operating point, the model attains a high Dynamic Degree and robust Imaging Quality\. Pushing the sparsity beyond this threshold \(e\.g\., dropping FLOPs below 81%\) results in a noticeable degradation, as excessively aggressive pruning begins to impair the representational capacity required for complex few\-step generation\. Consequently, we select the checkpoint at the∼\\sim83% FLOPs ratio as our final model, as it represents a well\-balanced empirical sweet spot\.

Dynamic Degree Analysis on Wan 1\.3B\.Regarding the Dynamic Degree score of 80\.56, which exceeds the teacher’s 65\.19, it is important to note two contextual factors\. First, Dynamic Degree is an inherently high\-variance metric\. As shown in panels \(a\)–\(c\), it naturally fluctuates over\[51\.4,80\.6\]\[51\.4,80\.6\]across viable training checkpoints\. Second, While standard DMD degrades dynamics, our pipeline mitigates this via GAN training and extra real\-video data: Ours \(Distill\) alone already reaches 68\.06\(\>\>teacher’s 65\.19\)\. TurboDiffusion reaches 86\.11, showing that a student exceeding its teacher on this metric is not anomalous\.

Crucially, our method maintains an exceptionally high Motion Smoothness \(98\.17\), which directly contradicts the possibility of temporal flickering or artefact\-driven motion\. Instead, we hypothesize that the structured stochastic perturbations introduced by our step\-conditioned mask updates counteract distillation’s mode\-seeking tendency, yielding genuinely richer motion dynamics\. This is corroborated by our supplementary videos, which exhibit larger, coherent motion rather than jitter\.

In\-depth Structural Analysis\.To further understand the resulting architecture of our joint optimization, we conduct a detailed component\-level analysis, as shown in[Fig\.˜8](https://arxiv.org/html/2607.06631#A3.F8)\(i\)\-\(l\)\. The retention rates of Self\-Attention, Cross\-Attention, and Feed\-Forward Network \(FFN\) modules exhibit distinct patterns across the denoising steps\. As observed in[Fig\.˜8](https://arxiv.org/html/2607.06631#A3.F8)\(l\), the sparsification of Self/Cross\-Attention components constitutes a substantial portion of the total parameter reduction\. The network learns to aggressively compress these Attention components particularly during the intermediate steps \(t2t\_\{2\}andt3t\_\{3\}\), while maintaining a relatively higher retention rate for FFN modules\. This fine\-grained breakdown empirically illustrates how the joint optimization automatically distributes structural redundancy across different architectural components and inference stages\.