Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization
Summary
This paper introduces Lip Forcing, the first autoregressive diffusion method for real-time video-to-video lip synchronization. By distilling a 14B teacher into causal students and using only two denoising steps, it achieves 31 FPS streaming at 1.3B scale, 17.6x faster than same-scale bidirectional models.
View Cached Full Text
Cached at: 06/10/26, 05:44 AM
Paper page - Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization
Source: https://huggingface.co/papers/2606.11180
Abstract
Autoregressive diffusion method for video-to-video lip synchronization achieves real-time performance through distillation and optimized inference schedules.
Diffusion-basedlip synchronizationmodels achieve strong visual quality and audio-visual alignment, but full-sequencebidirectional attentionand manydenoising stepsmake them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method forvideo-to-video(V2V)lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher intocausal students. At inference, the students generate each chunk in only twodenoising stepswithoutinference-time CFG, enabling real-timelip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and aSyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS, 17.6times faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2Vlip synchronization, runs 39.8times faster than its teacher at comparable reference fidelity.Time-to-first-frameis sub-millisecond at both scales, far below every diffusion baseline.
View arXiv pageView PDFProject pageGitHub9Add to collection
Get this paper in your agent:
hf papers read 2606\.11180
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.11180 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.11180 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.11180 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Flex-Forcing: Towards a Unified Autoregressive and Bidirectional Video Diffusion Model
Introduces Flex-Forcing, a unified training and inference framework that allows video diffusion models to operate under both bidirectional and autoregressive regimes via a flexible chunking mechanism over temporal and denoising steps, achieving better video quality, long-video stability, and faster inference.
Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation
Causal Forcing++ presents a novel causal consistency distillation method for frame-wise autoregressive video generation, achieving state-of-the-art quality with reduced latency and training cost.
Flexible Video Diffusion (3 minute read)
Flex-Forcing introduces a unified framework for video diffusion that supports both autoregressive and bidirectional generation modes, offering flexible control for video generation tasks.
LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing
LiveEdit proposes a causal, frame-by-frame streaming video editing framework that achieves real-time performance (12.66 FPS) via a three-stage distillation pipeline and AR-oriented mask cache, enabling stable long-horizon edits.
Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models
This paper introduces Causal-rCM, a unified teacher-forcing and self-forcing framework for autoregressive diffusion distillation in streaming video generation and interactive world models, achieving state-of-the-art performance with fast convergence.