dubbing

X-Dubbing 提出一种联合时间感知语音转文本翻译与视频感知文本转语音的方法，通过语言无关的时间预算实现与源视频同步的跨语言配音，无需配对配音数据即可超越现有基线。

一位开发者开源了本地运行的影视配音工具 dub-anything，利用 OmniVoice、IndexTTS、Chatterbox 和 Whisper 克隆并保留每位演员的原声进行多语言配音，让此前未被翻译的内容变得可看。

The paper introduces Align Then Reason (ATR), a multilingual, reference-free lip-sync judge that first aligns frame-level lip representations with phonetic units of a candidate line, then uses an LLM to reason over local alignment evidence and a global score for dubbing quality control, delivering large AUC gains over Qwen3.5, LLaMA-3.1, and Mistral SFT baselines across seven languages and two downstream dubbing tasks.

Amazon Prime Video has introduced an AI-powered feature that synchronizes actors' lip movements with dubbed audio to enhance dubbing quality, initially available for the series Maxton Hall with plans for expansion.

The article explores AI lip-sync technology as a promising enhancement for video translation, making speakers appear to naturally speak the translated language, with evaluations of tools like SyncSo, Rask, and HeyGen, and discusses its broader implications for content accessibility and commercial uses.

VideoLingo is an open-source video translation, localization, and dubbing tool that uses WhisperX and AI to deliver Netflix-level subtitles and multilingual dubbing. It supports downloading via yt-dlp and multiple TTS options, helping video reposters automate the entire workflow.

Google highlights PhD intern Khoi's research on synchronizing lip movements with dubbed audio to create natural-looking multi-language videos for YouTube.

May saw over $1.8 billion in voice AI funding, led by Sierra's $925M and Hark's $700M rounds, while ElevenLabs launched new models for music generation and dubbing with enhanced control. The newsletter also highlights healthcare deals and India's growing voice market.

pyVideoTrans is an open-source video translation tool that supports automatic speech recognition, subtitle translation, AI dubbing, and video synthesis. It integrates multiple ASR, translation, and TTS engines, making it suitable for cross-language video production and localization.

AI lip-sync tools like sync.so can redraw mouth movements to match dubbed audio, potentially neutralizing the long-standing argument that dubs break immersion due to mismatched mouth movements.

Descript redesigned its translation pipeline using OpenAI reasoning models to optimize multilingual video dubbing at scale, achieving 15% increase in translated video exports and 13-43% improvement in duration adherence across languages by addressing the challenge of matching speech duration to video timing constraints.