Paper page - AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model

Source: https://huggingface.co/papers/2608.16143

Abstract

AnyTalk generates 3D speech animations for arbitrary characters without animation data by adapting video diffusion models via character-specific fine-tuning and optimizing blendshape parameters from synthesized talking-head videos, with a distilled real-time variant.

We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recentvideo diffusion modelstrained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through ourCharacter-specific Fine-tuning (CsF)technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing “no motion”), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimatingblendshape parametersthrough a proposedoptimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, AnyTalk_{RT}, thereby enablingreal-time performance. By leveragingtalking-head video generation, our method broadens access toaudio-driven speech animationtechnology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2608\.16143

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.16143 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.16143 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.16143 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.