Shanghai AI Laboratory (OpenMMLab) has completely shattered the barriers of “sound creation” this time. Their open-source Amphion is nothing short of an “all-in-one arsenal” for the audio-visual creation world. From speech synthesis to AI singing voice conversion, this thing can knock out the vast majority of paid voice software.

Basic Info

• Project Name: Amphion
• Origin: OpenMMLab (a team of open-source heavyweights)
• Stars: 4.8k+ (still skyrocketing)
• License: MIT / Apache 2.0 (free for commercial use, extremely friendly)

Key Highlights

• Text-to-Speech (TTS): Input text to directly generate ultra-natural voices — it even reproduces breathing sounds.
• AI Singing Voice Conversion (SVD): Input your own singing and convert it into celebrity/anime vocal lines, making viral derivative content with zero barrier to entry.
• Sound Effects & Music Generation: Use prompts to handle background music and a variety of film-grade sound effects.
• Acoustic Feature Extraction: Great for audio researchers and model training folks — just plug it in and skip months of development time.
• Completely Free + Local Deployment: Freeloaders rejoice — data stays on your own computer, with maximum peace of mind.

Project Portal

GitHub: https://github.com/open-mmlab/Amphion…

open-mmlab/Amphion

Source: https://github.com/open-mmlab/Amphion

Amphion: An Open-Source Audio, Music, and Speech Generation Toolkit

Amphion (/æmˈfaɪən/) is a toolkit for Audio, Music, and Speech Generation. Its purpose is to support reproducible research and help junior researchers and engineers get started in the field of audio, music, and speech generation research and development.

Amphion offers a unique feature: visualizations of classic models or architectures. We believe that these visualizations are beneficial for junior researchers and engineers who wish to gain a better understanding of the model.

The North-Star objective of Amphion is to offer a platform for studying the conversion of any inputs into audio.

Amphion is designed to support individual generation tasks, including but not limited to,

• TTS: Text to Speech (⛳ supported)
• SVS: Singing Voice Synthesis (⛳ supported)
• VC: Voice Conversion (⛳ supported)
• AC: Accent Conversion (⛳ supported)
• SVC: Singing Voice Conversion (⛳ supported)
• TTA: Text to Audio (⛳ supported)
• TTM: Text to Music (👨‍💻 developing)
• more…

In addition to the specific generation tasks, Amphion includes several vocoders and evaluation metrics. A vocoder is an important module for producing high-quality audio signals, while evaluation metrics are critical for ensuring consistent metrics in generation tasks. Moreover, Amphion is dedicated to advancing audio generation in real-world applications, such as building large-scale datasets for speech synthesis.

🚀 News

• 2026/03/25: We release Vevo2, a unified and controllable framework for speech and singing voice generation. Vevo2 bridges controllable speech and singing voice generation via unified prosody learning, supporting TTS, SVS, VC, SVC, Singing Voice Editing, Singing Style Conversion, Melody Control, and more. arXiv (https://arxiv.org/abs/2508.16332) hf (https://huggingface.co/RMSnow/Vevo2) WebPage (https://versasinger.github.io/) readme

• 2025/05/26: We release DualCodec, a low-frame-rate (12.5Hz or 25Hz), semantically-enhanced (with SSL feature) Neural Audio Codec designed to extract discrete tokens for efficient speech generation.paper (http://arxiv.org/abs/2505.13000)Open In Colab (https://colab.research.google.com/drive/1VvUhsDffLdY5TdNuaqlLnYzIoXhvI8MK#scrollTo=Lsos3BK4J-4E)demo page (https://dualcodec.github.io/)code

• 2025/04/12: We release Vevo1.5, which extends Vevo and focuses on unified and controllable generation for both speech and singing voice. Vevo1.5 can be applied into a series of speech and singing voice generation tasks, including VC, TTS, AC, SVS, SVC, Speech/Singing Voice Editing, Singing Style Conversion, and more. blog (https://veiled-army-9c5.notion.site/Vevo1-5-1d2ce17b49a280b5b444d3fa2300c93a)

• 2025/02/26: We release Metis (https://github.com/open-mmlab/Amphion/tree/main/models/tts/metis), a foundation model for unified speech generation. The system supports zero-shot text-to-speech, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech. arXiv (https://arxiv.org/pdf/2502.03128) hf (https://huggingface.co/amphion/metis)

• 2025/02/26: The Emilia-Large dataset, featuring over 200,000 hours of data, is now available!!! Emilia-Large combines the original 101k-hour Emilia dataset (licensed under CC BY-NC 4.0) with the brand-new 114k-hour Emilia-YODAS dataset (licensed under CC BY 4.0). Download at hf (https://huggingface.co/datasets/amphion/Emilia-Dataset). Check details at arXiv (https://arxiv.org/abs/2501.15907).

• 2025/01/30: We release Amphion v0.2 Technical Report (https://arxiv.org/abs/2501.15442), which provides a comprehensive overview of the Amphion updates in 2024. arXiv (https://arxiv.org/abs/2501.15442)

• 2025/01/23: MaskGCT (https://arxiv.org/abs/2409.00750) and Vevo (https://openreview.net/pdf?id=anQDiQZhDP) got accepted by ICLR 2025! 🎉

• 2024/12/22: We release the reproduction of Vevo, a zero-shot voice imitation framework with controllable timbre and style. Vevo can be applied into a series of speech generation tasks, including VC, TTS, AC, and more. The released pre-trained models are trained on Emilia (https://huggingface.co/datasets/amphion/Emilia-Dataset) dataset and achieve SOTA zero-shot VC performance. arXiv (https://openreview.net/pdf?id=anQDiQZhDP) hf (https://huggingface.co/amphion/Vevo) WebPage (https://versavoice.github.io/) readme

• 2024/10/19: We release MaskGCT, a fully non-autoregressive TTS model that eliminates the need for explicit alignment information between text and speech supervision. MaskGCT is trained on Emilia (https://huggingface.co/datasets/amphion/Emilia-Dataset) dataset and achieves SOTA zero-shot TTS performance. arXiv (https://arxiv.org/abs/2409.00750) hf (https://huggingface.co/amphion/maskgct) hf (https://huggingface.co/spaces/amphion/maskgct) ModelScope (https://modelscope.cn/studios/amphion/maskgct) ModelScope (https://modelscope.cn/models/amphion/MaskGCT) readme

• 2024/09/01: Amphion (https://arxiv.org/abs/2312.09911), Emilia (https://arxiv.org/abs/2407.05361) and DSFF-SVC (https://arxiv.org/abs/2310.11160) got accepted by IEEE SLT 2024! 🤗

• 2024/08/28: Welcome to join Amphion’s Discord channel (https://discord.com/invite/drhW7ajqAG) to stay connected and engage with our community!

• 2024/08/20: SingVisio (https://arxiv.org/abs/2402.12660) got accepted by Computers & Graphics, available here (https://www.sciencedirect.com/science/article/pii/S0097849324001936)! 🎉

• 2024/08/27: The Emilia dataset is now publicly available! Discover the most extensive and diverse speech generation dataset with 101k hours of in-the-wild speech data now at hf (https://huggingface.co/datasets/amphion/Emilia-Dataset) or OpenDataLab (https://opendatalab.com/Amphion/Emilia)! 👑👑👑

• 2024/07/01: Amphion now releases Emilia, the first open-source multilingual in-the-wild dataset for speech generation with over 101k hours of speech data, and the Emilia-Pipe, the first open-source preprocessing pipeline designed to transform in-the-wild speech data into high-quality training data with annotations for speech generation! arXiv (https://arxiv.org/abs/2407.05361) hf (https://huggingface.co/datasets/amphion/Emilia) demo (https://emilia-dataset.github.io/Emilia-Demo-Page/) readme

• 2024/03/12: Amphion now support NaturalSpeech3 FACodec and release pretrained checkpoints. arXiv (https://arxiv.org/abs/2403.03100) hf (https://huggingface.co/amphion/naturalspeech3_facodec) hf (https://huggingface.co/spaces/amphion/naturalspeech3_facodec) readme

• 2024/02/22: The first Amphion visualization tool, SingVisio, release. arXiv (https://arxiv.org/abs/2402.12660) openxlab (https://openxlab.org.cn/apps/detail/Amphion/SingVisio) Video (https://drive.google.com/file/d/15097SGhQh-SwUNbdWDYNyWEP–YGLba5/view) readme

• 2023/12/18: Amphion v0.1 release. arXiv (https://arxiv.org/abs/2312.09911) hf (https://huggingface.co/amphion) youtube (https://www.youtube.com/watch?v=1aw0HhcggvQ) readme (https://github.com/open-mmlab/Amphion/pull/39)

• 2023/11/28: Amphion alpha release. readme (https://github.com/open-mmlab/Amphion/pull/2)

⭐ Key Features

TTS: Text to Speech

• Amphion achieves state-of-the-art performance compared to existing open-source repositories on text-to-speech (TTS) systems. It supports the following models or architectures:

• FastSpeech2 (https://arxiv.org/abs/2006.04558): A non-autoregressive TTS architecture that utilizes feed-forward Transformer blocks. code

• VITS (https://arxiv.org/abs/2106.06103): An end-to-end TTS architecture that utilizes conditional variational autoencoder with adversarial learning code

• VALL-E (https://arxiv.org/abs/2301.02111): A zero-shot TTS architecture that uses a neural codec language model with discrete codes. code

• NaturalSpeech2 (https://arxiv.org/abs/2304.09116): An architecture for TTS that utilizes a latent diffusion model to generate natural-sounding voices. code

• Jets: An end-to-end TTS model that jointly trains FastSpeech2 and HiFi-GAN with an alignment module. code

• MaskGCT (https://arxiv.org/abs/2409.00750): A fully non-autoregressive TTS architecture that eliminates the need for explicit alignment information between text and speech supervision. code

• Vevo-TTS (https://openreview.net/pdf?id=anQDiQZhDP): A zero-shot TTS architecture with controllable timbre and style. It consists of an autoregressive transformer and a flow-matching transformer. code

• DualCodec-VALLE: A VALLE model trained on 12.5Hz DualCodec tokens for super fast generation.

VC: Voice Conversion

Amphion supports the following voice conversion models:

• Vevo (https://openreview.net/pdf?id=anQDiQZhDP): A zero-shot voice imitation framework with controllable timbre and style. Vevo-Timbre conducts the style-preserved voice conversion, and Vevo-Voice conducts the style-converted voice conversion. code
• FACodec (https://arxiv.org/abs/2403.03100): FACodec decomposes speech into subspaces representing different attributes like content, prosody, and timbre. It can achieve zero-shot voice conversion. code (https://huggingface.co/amphion/naturalspeech3_facodec)
• Noro (https://arxiv.org/abs/2411.19770): A noise-robust zero-shot voice conversion system. Noro introduces innovative components tailored for VC using noisy reference speeches, including a dual-branch reference encoding module and a noise-agnostic contrastive speaker loss. code

Neural Audio Codec

• DualCodec, a low-frame-rate (12.5Hz or 25Hz), semantically-enhanced (with SSL feature) Neural Audio Codec designed to extract discrete tokens for efficient speech generation.paper (http://arxiv.org/abs/2505.13000)Open In Colab (https://colab.research.google.com/drive/1VvUhsDffLdY5TdNuaqlLnYzIoXhvI8MK#scrollTo=Lsos3BK4J-4E)demo page (https://dualcodec.github.io/)code
• FACodec (https://arxiv.org/abs/2403.03100): FACodec decomposes speech into subspaces representing different attributes like content, prosody, and timbre. code (https://huggingface.co/amphion/naturalspeech3_facodec)

AC: Accent Conversion

• Amphion supports AC with Vevo-Style. Particularly, it can conduct the accent conversion in a zero-shot manner. code

SVC: Singing Voice Conversion

• Vevo2: A unified and controllable framework for speech and singing voice generation. Vevo2 supports style-preserved VC/SVC, style-converted VC/SVC, TTS, SVS, Singing Voice Editing, Singing Style Conversion, Melody Control, and more. arXiv (https://arxiv.org/abs/2508.16332) code
• Vevo1.5: A versatile zero-shot voice imitation framework for unified speech and singing voice generation, supporting VC, TTS, AC, SVS, SVC, Editing, Singing Style Conversion, and more. code
• Ampion supports multiple content-based features from various pretrained models, including WeNet (https://github.com/wenet-e2e/wenet), Whisper (https://github.com/openai/whisper), and ContentVec (https://github.com/auspicious3000/contentvec). Their specific roles in SVC has been investigated in our SLT 2024 paper. arXiv (https://arxiv.org/abs/2310.11160) code
• Amphion implements several state-of-the-art model architectures, including diffusion-, transformer-, VAE- and flow-based models. The diffusion-based architecture uses Bidirectional dilated CNN (https://openreview.net/pdf?id=a-xFK8Ymz5J) as a backend and supports several sampling algorithms such as DDPM (https://arxiv.org/pdf/2006.11239.pdf), DDIM (https://arxiv.org/pdf/2010.02502.pdf), and PNDM (https://arxiv.org/pdf/2202.09778.pdf). Additionally, it supports single-step inference based on the Consistency Model (https://openreview.net/pdf?id=FmqFfMTNnv). code

TTA: Text to Audio

• Amphion supports the TTA with a latent diffusion model. It is designed like AudioLDM (https://arxiv.org/abs/2301.12503), Make-an-Audio (https://arxiv.org/abs/2301.12661), and AUDIT (https://arxiv.org/abs/2304.00830). It is also the official implementation of the text-to-audio generation part of our NeurIPS 2023 paper. arXiv (https://arxiv.org/abs/2304.00830) code

Vocoder

• Amphion supports various widely-used neural vocoders, including:

• GAN-based vocoders: MelGAN (https://arxiv.org/abs/1910.06711), HiFi-GAN (https://arxiv.org/abs/2010.05646), NSF-HiFiGAN (https://github.com/nii-yamagishilab/project-NN-Pytorch-scripts), BigVGAN (https://arxiv.org/abs/2206.04658), APNet (https://arxiv.org/abs/2305.07952).

• Flow-based vocoders: WaveGlow (https://arxiv.org/abs/1811.00002).

• Diffusion-based vocoders: Diffwave (https://arxiv.org/abs/2009.09761).

• Auto-regressive based vocoders: WaveNet (https://arxiv.org/abs/1609.03499), WaveRNN (https://arxiv.org/abs/1802.08435v1).

• Amphion provides the official implementation of Multi-Scale Constant-Q Transform Discriminator (https://arxiv.org/abs/2311.14957) (our ICASSP 2024 paper). It can be used to enhance any architecture GAN-based vocoders during training, and keep the inference stage (such as memory or speed) unchanged. arXiv (https://arxiv.org/abs/2311.14957) code

Evaluation

Amphion provides a comprehensive objective evaluation of the generated audio. code

The supported evaluation metrics contain:

• F0 Modeling: F0 Pearson Coefficients, F0 Periodicity Root Mean Square Error, F0 Root Mean Square Error, Voiced/Unvoiced F1 Score, etc.
• Energy Modeling: Energy Root Mean Square Error, Energy Pearson Coefficients, etc.
• Intelligibility: Character/Word Error Rate, which can be calculated based on Whisper (https://github.com/openai/whisper) and more.
• Spectrogram Distortion: Frechet Audio Distance (FAD), Mel Cepstral Distortion (MCD), Multi-Resolution STFT Distance (MSTFT), Perceptual Evaluation of Speech Quality (PESQ), Short Time Objective Intelligibility (STOI), etc.
• Speaker Similarity: Cosine similarity, which can be calculated based on RawNet3 (https://github.com/Jungjee/RawNet), Resemblyzer (https://github.com/resemble-ai/Resemblyzer), WeSpeaker (https://github.com/wenet-e2e/wespeaker), WavLM (https://github.com/microsoft/unilm/tree/master/wavlm) and more.

Datasets

• Amphion unifies the data preprocess of the open-source datasets including AudioCaps (https://audiocaps.github.io/), LibriTTS (https://www.openslr.org/60/), LJSpeech (https://keithito.com/LJ-Speech-Dataset/), M4Singer (https://github.com/M4Singer/M4Singer), Opencpop (https://wenet.org.cn/opencpop/)