Is AI lip-sync the missing link in actually good video translation?

Reddit r/artificial News

Summary

The article explores AI lip-sync technology as a promising enhancement for video translation, making speakers appear to naturally speak the translated language, with evaluations of tools like SyncSo, Rask, and HeyGen, and discusses its broader implications for content accessibility and commercial uses.

So I had been playing around with AI lip-sync recently, and I'm starting to think that this tech is being overlooked a bit. Now, we have had AI translation and dubbing for a while, but the problem always is that people in the video would still LOOK like they're speaking a completely different language. I don't know about y'all but small inconsistencies like that gets on my nerves! YouTube's autodubbing is probably the easiest example: the voice can sound unnatural, the timing can be off, and the person's mouth is moving in a completely different way compared to what they have said. Once it's noticed, it's really hard to unnotice again. Instagram's translation features basically have the same problem for me. The audio gets translated, but the visuals still give it away.. Of course, what surprised me was how good specialized lip-sync models have gotten! I tested a few recently, SyncSo, Rask and HeyGen. The results weren't consistent across every video, but still, some of them were genuinely pretty good. There were a few clips where people could not tell it was AI generated, and honestly? I'll take that as a plus point. Now that's what got me thinking about the bigger picture:- A huge part of high quality educational and informational content exists in English, while a massive number of people don't consume it because of the language barrier. Good translation already removed one part of the problem, but making the speaker seem like they're speaking your language seems like a much bigger deal than I initially thought. The commercial applications seem pretty obvious to me: movies, online courses, advertising, training videos, interviews, etc. Maybe this unsynced mouth thing is only an ick for me, but I'm curious what y'all have to think about this. So, is AI Lip-sync just another improvement in dubbing, or is it one of those technologies that can make a translated video feel native?
Original Article

Similar Articles