StepAudio 3 Gen Technical Report
Summary
StepAudio 3 Gen is a general-purpose audio generation model that unifies text-to-speech, voice design, sound effects, and music within a single discrete autoregressive framework using residual vector quantization tokens.
View Cached Full Text
Cached at: 09/14/26, 10:34 AM
Paper page - StepAudio 3 Gen Technical Report
Source: https://huggingface.co/papers/2609.12945 Published on Sep 11
·
Submitted byhttps://huggingface.co/giantPanda0906
Yangon Sep 14
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
StepAudio 3 Gen is a discrete autoregressive audio generation model using residual vector quantization tokens to unify text-to-speech, voice design, sound effects, and music within a single framework.
We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discreteautoregressive generatorthat models audio directly overresidual vector quantization(RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. ItsStepAudio Tokenizerrepresents general audio at 12.5 Hz in a shared 16 times 2048 residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweightcausal Transformercompletes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1)interference-aware progressive pretrainingfor acquiring audio capabilities while preserving the textual abilities of the large language model, (2)RVQ Adaptorfor effectively incorporatingmulti-codebook acoustic representations, and (3)discrete autoregressive modelingover a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2609\.12945
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.12945 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.12945 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.12945 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
StepAudio 3 Music Technical Report
StepAudio 3 Music introduces a large-scale, long-form music generation model with explicit musical planning via ABC-CoT, achieving high scores in audio quality and similarity metrics compared to other systems.
StepAudio 3 Realtime Technical Report
The paper presents StepAudio3Realtime, an audio-language foundation model for real-time spoken interaction, using a continuous listen-converse-think-act loop with Think-While-Speaking to achieve deep reasoning and low latency, with top-tier performance on benchmarks like MMSU and Full-Duplex Bench.
StepAudio 2.5 Technical Report
StepAudio 2.5 is a unified audio-language model that achieves state-of-the-art results across ASR, TTS, and real-time spoken interaction by leveraging task-tailored reinforcement learning from human feedback to optimize shared representations.
Qwen3-TTS Technical Report
The Qwen3-TTS technical report introduces a series of advanced multilingual text-to-speech models with voice cloning and controllable generation, featuring a dual-track LM architecture and specialized tokenizers for low-latency streaming.
Stable Audio 3
Stable Audio 3 introduces a family of fast latent diffusion models for variable-length audio generation and editing, with open-source release of small and medium model weights.