X-Dubbing: Speech-to-Speech Translation under Temporal and Visual Constraints

arXiv cs.CL Papers
speech-to-speech-translation dubbing audio-visual speech-synthesis multilingual

Summary

X-Dubbing 提出一种联合时间感知语音转文本翻译与视频感知文本转语音的方法，通过语言无关的时间预算实现与源视频同步的跨语言配音，无需配对配音数据即可超越现有基线。

arXiv:2610.32042v1 Announce Type: new Abstract: Cross-lingual dubbing aims to convert source speech into target-language speech that remains synchronized with the original video. The key to this task is to produce a faithful translation that is naturally speakable within the source timing, which is non-trivial because languages differ in duration, rhythm, and articulation. In this paper, we propose X-Dubbing, which formulates the task as a joint problem of time-aware speech-to-text translation and video-aware text-to-speech. Specifically, our method consists of a Dubbing Translator that generates temporally structured scripts and a Voice Actor that renders them in synchrony with the source video, both grounded in a language-agnostic timing budget. We further refine the translator with dubbing-aware feedback from the synthesized speech, encouraging scripts that yield well-synchronized dubbing. This design requires no paired dubbing data, as the two components are trained on speech-text and monolingual audio-visual data respectively. Extensive experiments show that X-Dubbing outperforms recent baselines across diverse metrics, including isochrony and audio-visual correspondence.
Original Article

Similar Articles

DIAL-S2ST: Learning Latency Control for Simultaneous Speech-to-Speech Translation

arXiv cs.CL

The paper proposes DIAL-S2ST, a post-training method that learns inference-time latency control in simultaneous speech-to-speech translation models via self-distillation from lookahead predictions and a jointly trained LoRA adapter, demonstrating flexible latency control on French-to-English translation using Hibiki with only 10 hours of source speech.

Kolibri: A Sovereign European Model on the Pareto Frontier

arXiv cs.CL

This report introduces Kolibri, an English-German mixture-of-experts transformer with 78.1B total and 3.46B active parameters, released as open weights under Apache 2.0 for sovereign, German-focused deployment in regulated domains. Trained on 24T tokens with over 20% German data and post-trained with SFT plus RL on 1.2M curated tasks, Kolibri lies on the Pareto frontier of quality and serving cost among evaluated open models.

Making Kokoro TTS 50-100x faster without knowing how

Reddit r/LocalLLaMA

A free in-browser TTS tool that runs entirely on-device using a 2.7 MB model on a custom 136 kB WebAssembly engine, making Kokoro-distilled Paradee 12.9x faster while maintaining quality. The author breaks down each optimization step and lets users benchmark it live in their browser.