WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling

Papers with Code Trending Papers
zero-shot text-to-speech raw-waveform diffusion-transformer flow-matching generative-model

Summary

WavTTS presents the first raw waveform generative text-to-speech model using flow matching and Diffusion Transformer, achieving performance comparable to latent-space diffusion models while avoiding information loss from compressed representations.

Recently, diffusion models operating on VAE latents or mel-spectrograms have become the dominant paradigm for zero-shot TTS. Although these compressed representations improve generation efficiency, they inevitably suffer from information loss and non-end-to-end training. Theoretically, directly modeling raw waveforms circumvents these issues; however, this direction remains underexplored and is often deemed difficult due to the extremely long sequence length of audio signals. To overcome this, we propose WavTTS, the first raw waveform generative TTS model that substantially narrows the gap with latent-space generative models. Built upon the flow matching with Diffusion Transformer (DiT), WavTTS directly models speech waveforms via a simple patchification strategy, while integrating multi-scale mel-spectrogram supervision to provide perceptual guidance during training. Furthermore, we investigate the impact of prediction targets and noise scheduling in waveform diffusion, and develop an effective schedule design to improve generation quality. Evaluations on open-source benchmarks demonstrate that WavTTS closely approaches the performance of current state-of-the-art latent generative zero-shot TTS models, while substantially outperforming previous end-to-end speech generation models. Our findings demonstrate the feasibility of scaling diffusion-based TTS directly in the waveform space, opening a new direction for end-to-end speech generation.
Original Article
View Cached Full Text

Cached at: 06/04/26, 03:40 AM

Paper page - WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling

Source: https://huggingface.co/papers/2606.03455 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

WavTTS presents the first raw waveform generative text-to-speech model that bridges the performance gap with latent-space diffusion models through flow matching and Diffusion Transformer architecture.

Recently,diffusion modelsoperating onVAE latentsormel-spectrogramshave become the dominant paradigm forzero-shot TTS. Although these compressed representations improve generation efficiency, they inevitably suffer from information loss and non-end-to-end training. Theoretically, directly modelingraw waveformscircumvents these issues; however, this direction remains underexplored and is often deemed difficult due to the extremely long sequence length of audio signals. To overcome this, we propose WavTTS, the first raw waveform generative TTS model that substantially narrows the gap with latent-space generative models. Built upon theflow matchingwithDiffusion Transformer(DiT), WavTTS directly models speech waveforms via a simplepatchification strategy, while integratingmulti-scale mel-spectrogram supervisionto provide perceptual guidance during training. Furthermore, we investigate the impact ofprediction targetsandnoise schedulingin waveform diffusion, and develop an effective schedule design to improve generation quality. Evaluations on open-source benchmarks demonstrate that WavTTS closely approaches the performance of current state-of-the-art latent generativezero-shot TTSmodels, while substantially outperforming previousend-to-end speech generationmodels. Our findings demonstrate the feasibility of scaling diffusion-based TTS directly in the waveform space, opening a new direction forend-to-end speech generation.

View arXiv pageView PDFProject pageGitHub36Add to collection

Get this paper in your agent:

hf papers read 2606\.03455

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper2

#### worstchan/WavTTS Text-to-Speech• Updatedabout 24 hours ago • 3 #### drbaph/WavTTS Text-to-Speech• Updatedabout 2 hours ago • 2

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.03455 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.03455 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

WavFlow: Audio Generation in Waveform Space

Hugging Face Daily Papers

WavFlow generates high-fidelity audio directly in raw waveform space using waveform patchify and amplitude lifting, achieving competitive performance on video-to-audio and text-to-audio benchmarks without intermediate latent representations.

WavFlow Generates Audio Directly in Waveform Space (GitHub Repo)

TLDR AI

WavFlow is a new method from Meta AI that generates synchronized, high-fidelity audio from video and text inputs directly in raw waveform space, bypassing latent compression. It achieves performance on par with latent-based methods on VGGSound and AudioCaps benchmarks.

dots.tts Technical Report

Hugging Face Daily Papers

dots.tts presents a 2B-parameter continuous autoregressive TTS model trained on multilingual data, achieving state-of-the-art performance on benchmarks like Seed-TTS-Eval with low-latency streaming via CFG-aware MeanFlow distillation. The model, code, and checkpoints are released under Apache 2.0.