Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing

Hugging Face Daily Papers Papers
lip-sync multimodal dubbing vision-language alignment benchmark speech-recognition

Summary

The paper introduces Align Then Reason (ATR), a multilingual, reference-free lip-sync judge that first aligns frame-level lip representations with phonetic units of a candidate line, then uses an LLM to reason over local alignment evidence and a global score for dubbing quality control, delivering large AUC gains over Qwen3.5, LLaMA-3.1, and Mistral SFT baselines across seven languages and two downstream dubbing tasks.

Dubbing quality control requires a reference-free judge that can determine whether a candidate text line matches a speaker's visible articulation in both content and timing, using only silent video and text because dubbed audio may not yet exist. Existing visual speech recognizers and video-language models are poorly suited to this setting: even when fine-tuned to recover spoken content from lip motion, they remain largely insensitive to temporal errors. We introduce Align Then Reason (ATR), a multilingual lip-sync judge that first establishes a monotonic alignment between frame-level lip representations and the phonetic units of the candidate line, then reasons over this alignment to make the final judgment. An alignment scorer provides the LLM with both local evidence for each phonetic unit and a calibrated global alignment score, enabling it to reason jointly about content and timing. On a seven-language benchmark, our method improves mean AUC over the corresponding Qwen3.5 SFT baselines by 59.4%, 50.2%, and 50.8% with 2B, 4B, and 9B reasoners, respectively. The gains generalize across LLM families, reaching mean AUC improvements of 45.9% and 46.6% over the best baseline for LLaMA-3.1-8B and Mistral-7B, respectively. They also transfer across datasets to three unseen MuAViC languages. Furthermore, we evaluate on two downstream tasks built from real dubbing lines. On dub-line reranking, ATR-9B outperforms the best lip-reading baseline by 52.0%, while on script-to-clip assignment, ATR-9B improves over the best lip-reading baseline by 17.7%.
Original Article
View Cached Full Text

Cached at: 10/02/26, 04:27 AM

Paper page - Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing

Source: https://huggingface.co/papers/2610.00825

Abstract

Dubbingqualitycontrolrequiresareference-freejudgethatcandeterminewhetheracandidatetextlinematchesaspeaker’svisiblearticulationinbothcontentandtiming,usingonlysilentvideoandtextbecausedubbedaudiomaynotyetexist.Existingvisualspeechrecognizersandvideo-languagemodelsarepoorlysuitedtothissetting:evenwhenfine-tunedtorecoverspokencontentfromlipmotion,theyremainlargelyinsensitivetotemporalerrors.WeintroduceAlignThenReason(ATR),amultilinguallip-syncjudgethatfirstestablishesamonotonicalignmentbetweenframe-levelliprepresentationsandthephoneticunitsofthecandidateline,thenreasonsoverthisalignmenttomakethefinaljudgment.AnalignmentscorerprovidestheLLMwithbothlocalevidenceforeachphoneticunitandacalibratedglobalalignmentscore,enablingittoreasonjointlyaboutcontentandtiming.Onaseven-languagebenchmark,ourmethodimprovesmeanAUCoverthecorrespondingQwen3.5SFTbaselinesby59.4%,50.2%,and50.8%with2B,4B,and9Breasoners,respectively.ThegainsgeneralizeacrossLLMfamilies,reachingmeanAUCimprovementsof45.9%and46.6%overthebestbaselineforLLaMA-3.1-8BandMistral-7B,respectively.TheyalsotransferacrossdatasetstothreeunseenMuAViClanguages.Furthermore,weevaluateontwodownstreamtasksbuiltfromrealdubbinglines.Ondub-linereranking,ATR-9Boutperformsthebestlip-readingbaselineby52.0%,whileonscript-to-clipassignment,ATR-9Bimprovesoverthebestlip-readingbaselineby17.7%.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.00825

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.00825 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.00825 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.00825 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Towards a Phonology-Informed Evaluation of Multilingual TTS

arXiv cs.CL

This paper proposes a classifier-based framework to audit multilingual TTS systems for phonological faithfulness, using Assamese ATR vowel harmony as a case study. It reveals that Meta's MMS TTS frequently misproduces advanced tongue root vowels, a bias absent in human speech.

Is AI lip-sync the missing link in actually good video translation?

Reddit r/artificial

The article explores AI lip-sync technology as a promising enhancement for video translation, making speakers appear to naturally speak the translated language, with evaluations of tools like SyncSo, Rask, and HeyGen, and discusses its broader implications for content accessibility and commercial uses.

Are you speaking my languages? On spoken language adherence in multimodal LLMs

arXiv cs.CL

This paper addresses the problem of spoken language adherence in multimodal LLMs for ASR, proposing a soft prompting approach and novel metric to quantify language violations. It evaluates three mitigation strategies—zero-shot prompting, supervised fine-tuning, and chain-of-thought reasoning—across multiple languages to improve transcription fidelity.