ViRDM: Taming Representation Distribution Matching for Few-Step Causal Video Generation

Hugging Face Daily Papers Papers
video-generation representation-matching few-step causal-modeling rdm post-training ai-research

Summary

ViRDM is a method for few-step causal video generation using representation distribution matching, achieving better results than DMD-based baselines with efficient training on single or multiple GPUs.

Few-step autoregressive (AR) video diffusion enables low-latency streaming generation, but existing post-training methods predominantly rely on Distribution Matching Distillation (DMD), requiring both a large pretrained teacher and an online critic to estimate distributional discrepancies through diffusion scores. In this work, we ask whether this resource-intensive teacher--critic stack can be eliminated by post-training only the generator against a precomputed target distribution. Drawing inspiration from representation distribution matching (RDM) for one-step image generation, we systematically study its transfer to few-step causal video generation and identify three key barriers: a memory-intractable gradient path, a distinct video optimization regime, and representation distributions that underconstrain temporal dynamics. We introduce ViRDM, a teacher- and critic-free video post-training recipe that addresses these barriers sequentially. By coupling RDM with stochastically truncated clean-exit supervision, a lightweight VAE decoder, and staged vector--Jacobian products, ViRDM makes representation distribution matching memory-feasible for multi-step causal video rollouts. We further establish effective generated-population and initialization regimes for video RDM, and introduce lightweight dynamics regularization to compensate for the underconstrained temporal dynamics. ViRDM turns three-network distillation into generator-only post-training, reducing GPU memory use and training time while improving video quality. With only 20 generator updates, the recipe reaches 84.87 on the official VBench evaluation, outperforming the previous best few-step causal baseline by 0.36, while requiring 16 A100 GPU-hours. We additionally report exploratory results demonstrating the potential of the same recipe for lower causal sampling budget and for one-, two-, and four-step bidirectional generation.
Original Article
View Cached Full Text

Cached at: 09/25/26, 03:43 AM

Paper page - ViRDM: Taming Representation Distribution Matching for Few-Step Causal Video Generation

Source: https://huggingface.co/papers/2609.28923 Hi everyone! πŸ‘‹ We’re sharingViRDM: A recipe for RDM-based few-step causal video post-training, along with the pitfalls and lessons learned**(single-GPU training supported πŸ’₯)**

Inspired by the impressive results of FD Loss and RDM in image generation, we explored whether the same idea could work for few-step causal video generation: matching a precomputed representation distribution to update only the generator, without the online diffusion teacher and learned critic commonly used in DMD-based methods such as Self Forcing and Causal Forcing.

The idea was straightforward, but getting it to work for video generation came with quite a few challenges πŸ₯Ή πŸ”Ή**Making training fit in memory.**Compared with one-step image generation, multi-step causal video generation has a much longer backward path, and backpropagating through the video VAE is particularly memory-intensive. We had to tackle OOM before anything else. πŸ”Ή**Rethinking population size and initialization.**Image RDM uses very large generated populations and can start directly from a pretrained bidirectional model. For few-step causal video generation, we couldn’t simply carry over these choices. πŸ”Ή**Closing the motion dynamic gap.**Even with video representation features, matching the feature distributions alone did not fully resolve the degraded dynamics issue.

After working through ablated experiments, we put the working configurations, practical lessons, and pitfalls into a recipe. Hopefully, this saves some time for the community in exploring RDM for video generation πŸ”§

Our final four-step causal recipe achieves better results than the DMD-based baselines evaluated in our paper, with substantially less post-training time and lower peak memory: βœ…84.87 VBench Total βœ…Only 20 generator updates βœ…Approximately 2 hours of post-training on 8Γ—A100.

The recipe also supports training ona single 80GB A100, a setup that is challenging for common DMD pipelines πŸ’₯

We’re releasing: βœ” Training and inference code βœ” Model weights and precomputed reference artifacts βœ” Reproduction configurations and evaluation scripts βœ” A single-80GB-GPU training launcher

🌐Website & demos:https://neu-vi.github.io/ViRDM/ πŸ’»GitHub:https://github.com/neu-vi/ViRDM πŸ“„Paper:https://arxiv.org/abs/2609.28923

Many thanks to the authors of FD Loss, RDM, Self Forcing, Causal Forcing, and the other foundational works we build on πŸ™

Stars 🌟, PRs, issues, and discussions are all welcome. We’d love to hear from others experimenting with ViRDM!πŸ€—

Similar Articles

DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation

Papers with Code Trending

DMAD (Distribution Matching as Adversarial Distillation) recasts distribution matching as a classification problem, learning log-density ratios via discriminator heads to train few-step student models for fast visual generation without auxiliary score fitting. It achieves state-of-the-art results, including FID 1.04 on ImageNet-64x64 with one-step generation and VBench score 85.15 for four-step Wan2.1-T2V-14B.

Representation Distribution Matching for One-Step Visual Generation

Hugging Face Daily Papers

This paper introduces Representation Distribution Matching (RDM), a method for one-step image generation by matching feature distributions under pretrained encoders, achieving state-of-the-art results on ImageNet and enabling post-training of FLUX.2 into a one-step generator with improved performance.

Reinforcing Few-step Generators via Reward-Tilted Distribution Matching

Hugging Face Daily Papers

RTDMD is a two-stage framework combining distribution matching distillation with reward-guided reinforcement learning to improve few-step image generation alignment with human preferences. It achieves state-of-the-art results on multiple models with only 4 inference steps.

Rollout-Marginal Distillation for Long-Horizon Autoregressive Video Generation

Hugging Face Daily Papers

This paper introduces Rollout-Marginal Distillation (RMD) for autoregressive video diffusion, where each generated chunk is scored independently against a chunk teacher to avoid error-compounding from imperfect temporal context, followed by video-level DMD to restore coherence. RMD reportedly maintains high visual quality far beyond its training horizon and outperforms video-level DMD baselines.