OpenVoice: Versatile Instant Voice Cloning

Papers with Code Trending 12/03/23, 06:41 PM Papers

voice-cloning ai open-source cross-lingual speech-synthesis research

Summary

OpenVoice is a versatile voice cloning method that enables flexible voice style control and zero-shot cross-lingual cloning with high efficiency using a single audio clip.

We introduce OpenVoice, a versatile voice cloning approach that requires only a short audio clip from the reference speaker to replicate their voice and generate speech in multiple languages. OpenVoice represents a significant advancement in addressing the following open challenges in the field: 1) Flexible Voice Style Control. OpenVoice enables granular control over voice styles, including emotion, accent, rhythm, pauses, and intonation, in addition to replicating the tone color of the reference speaker. The voice styles are not directly copied from and constrained by the style of the reference speaker. Previous approaches lacked the ability to flexibly manipulate voice styles after cloning. 2) Zero-Shot Cross-Lingual Voice Cloning. OpenVoice achieves zero-shot cross-lingual voice cloning for languages not included in the massive-speaker training set. Unlike previous approaches, which typically require extensive massive-speaker multi-lingual (MSML) dataset for all languages, OpenVoice can clone voices into a new language without any massive-speaker training data for that language. OpenVoice is also computationally efficient, costing tens of times less than commercially available APIs that offer even inferior performance. To foster further research in the field, we have made the source code and trained model publicly accessible. We also provide qualitative results in our demo website. Prior to its public release, our internal version of OpenVoice was used tens of millions of times by users worldwide between May and October 2023, serving as the backend of MyShell.

Original Article

View Cached Full Text

Cached at: 08/24/26, 10:08 AM

Paper page - OpenVoice: Versatile Instant Voice Cloning

Source: https://huggingface.co/papers/2312.01479

Abstract

OpenVoice enables flexible voice style control and zero-shot cross-lingual voice cloning with high efficiency using a single audio clip.

We introduce OpenVoice, a versatilevoice cloningapproach that requires only a shortaudio clipfrom the reference speaker to replicate their voice and generate speech in multiple languages. OpenVoice represents a significant advancement in addressing the following open challenges in the field: 1) Flexible Voice Style Control. OpenVoice enables granular control over voice styles, includingemotion,accent,rhythm,pauses, andintonation, in addition to replicating thetone colorof the reference speaker. Thevoice stylesare not directly copied from and constrained by the style of the reference speaker. Previous approaches lacked the ability to flexibly manipulatevoice stylesafter cloning. 2) Zero-Shot Cross-LingualVoice Cloning. OpenVoice achieves zero-shot cross-lingualvoice cloningfor languages not included in the massive-speaker training set. Unlike previous approaches, which typically require extensive massive-speaker multi-lingual (MSML) dataset for all languages, OpenVoice can clone voices into a new language without any massive-speaker training data for that language. OpenVoice is also computationally efficient, costing tens of times less than commercially available APIs that offer even inferior performance. To foster further research in the field, we have made thesource codeandtrained modelpublicly accessible. We also provide qualitative results in our demo website. Prior to its public release, our internal version of OpenVoice was used tens of millions of times by users worldwide between May and October 2023, serving as the backend of MyShell.