Poly-InstructTTS: Learning In-the-Wild Expressive Speech Synthesis from Open-Ended Instructions

# Poly-InstructTTS: Learning In-the-Wild Expressive Speech Synthesis from Open-Ended Instructions
Source: [https://arxiv.org/html/2608.20387](https://arxiv.org/html/2608.20387)
Zhang Xu Guo Yang Miao Wang Song

###### Abstract

While recent text\-to\-speech \(TTS\) models achieve high naturalness, controlling fine\-grained expression via natural\-language instructions remains challenging\. We introducePoly\-InstructTTS, which learns expressive speech from open\-ended instructions using in\-the\-wild audiovisual data\. We build a scalable multi\-modal pipeline to construct a 1,000\-hour instruction\-annotated corpus covering 1,000\+ fine\-grained emotions and styles\. The framework uses a prompt\-free GPT with attribute\-based thinking tokens, followed by a flow\-matching module that injects timbre from a reference audio\. We also present a speaker fine\-tuning procedure to transfer instruction control to specific speakers while preserving persona\. We further extend InstructTTSEval with broader tasks\. Experiments show that Poly\-InstructTTS delivers strong performance in instruction adherence and expressiveness\. Audio demos and the expanded testset are available on our project page111https://zhangjh915\.github\.io/PolyInstructTTS\-demo/\.

###### keywords:

speech synthesis, instruction following, expressive TTS, multi\-modal data processing

## 1Introduction

For the past few years, GPT\-based text\-to\-speech \(TTS\) models\[[1](https://arxiv.org/html/2608.20387#bib.bib1),[2](https://arxiv.org/html/2608.20387#bib.bib2),[3](https://arxiv.org/html/2608.20387#bib.bib3),[4](https://arxiv.org/html/2608.20387#bib.bib4),[5](https://arxiv.org/html/2608.20387#bib.bib5),[6](https://arxiv.org/html/2608.20387#bib.bib6),[7](https://arxiv.org/html/2608.20387#bib.bib7)\]have developed rapidly, leading to significant improvements in speech naturalness and expressiveness\. The prevalent structure includes a GPT for discrete audio token prediction, a Flow\-Matching \(FM\) acoustic model and a vocoder for waveform generation\. These models perform well on zero\-shot voice cloning, effectively capturing and reproducing the timbre and prosody from a short reference audio\.

Beyond cloning, generating controllable speech using natural language instructions instead of predefined style tags has gained attention\. However, the challenge becomes to acquire deep text\-understanding capabilities to interpret complex instructions\. Furthermore, mainstream zero\-shot TTS systems usually require prompt audio as input to the GPT model, which may introduce conflicts between the prompt and target styles specified by the instruction, degrading the controllability\.

Another challenge is the scarcity of expressive speech training data\. Most open\-source TTS datasets like LibriTTS\[[8](https://arxiv.org/html/2608.20387#bib.bib8)\]and MLS\[[9](https://arxiv.org/html/2608.20387#bib.bib9)\]are dominated by neutral reading tones, and some datasets include emotion labels\[[10](https://arxiv.org/html/2608.20387#bib.bib10),[11](https://arxiv.org/html/2608.20387#bib.bib11)\]only cover a few major categories and lack natural language descriptions\. Consequently, existing instruction\-TTS systems struggle to handle subtle emotional shifts or extreme speaking styles \(e\.g\., hesitation, screaming, whispering\)\.

To address the challenges, we proposePoly\-InstructTTS, a TTS system capable of understanding diverse natural language instructions for expressive speech synthesis\. The model is trained on dataset generated by apoly\-modal data pipelineand the model is capable ofpoly\-style generationfor fine\-grained emotions, styles, and paralinguistic behaviors withpoly\-instruction adherence\. The contributions of this paper are summarized as follows:

- •We design a multi\-modal data processing pipeline to generate diverse instruction\-audio pairs from cinematic and television audiovisual media\.
- •We introduce a TTS framework with a prompt\-free GPT guided by attribute\-based thinking tokens and an FM module for timbre injection, together with a fine\-tuning scheme to transfer controllability to specific speakers\.
- •We expand the InstructTTSEval\[[12](https://arxiv.org/html/2608.20387#bib.bib12)\]testset to include more diverse evaluation tasks for the open\-source community\.

## 2Related Works

### 2\.1Instruction\-Annotated Audio Datasets

The development of instruction\-controllable TTS heavily relies on high\-quality instruction\-text\-audio pairs\. Early efforts\[[13](https://arxiv.org/html/2608.20387#bib.bib13),[14](https://arxiv.org/html/2608.20387#bib.bib14)\]transitioned from simple concatenation of acoustic tags, such as gender, emotion, pitch, etc\., to rule\-based natural language generation\. With the advent of Large Language Models \(LLMs\), more recent works\[[15](https://arxiv.org/html/2608.20387#bib.bib15),[16](https://arxiv.org/html/2608.20387#bib.bib16),[17](https://arxiv.org/html/2608.20387#bib.bib17),[18](https://arxiv.org/html/2608.20387#bib.bib18),[19](https://arxiv.org/html/2608.20387#bib.bib19)\]leverage LLMs to generate diverse, multi\-attribute instruction texts\.

However, most instruction datasets are derived from open\-source datasets, recordings with limited emotions and styles such as audiobooks and podcasts, or synthesized speech from commercial APIs\. Consequently, they are dominated by neutral, standardized reading tones and it's hard to capture paralinguistic phenomena or extreme emotional expressions in real\-world conversations\. In contrast, our work explicitly targets in\-the\-wild cinematic and television data to reach a higher ceiling\.

![Refer to caption](https://arxiv.org/html/2608.20387v1/DataPipeline.jpg)\(a\)Proposed data pipeline\. \* ASR: Automatic Speech Recognition, SD: Speaker Diarization, PL: Paralinguistic Tagging via 11Labs API\[[20](https://arxiv.org/html/2608.20387#bib.bib20)\]
![Refer to caption](https://arxiv.org/html/2608.20387v1/ModelV2.jpg)\(b\)Proposed Poly\-InstructTTS architecture\.

Figure 1:Overview of our proposed Poly\-InstructTTS system\. \(a\) illustrates the data pipeline for constructing the instruction dataset\. \(b\) shows the model architecture of the Poly\-InstructTTS based on GPT\-FM framework\.
### 2\.2Instruction\-Following TTS Systems

Early instruction\-based systems, such as InstructTTS\[[21](https://arxiv.org/html/2608.20387#bib.bib21)\]and PromptTTS\[[13](https://arxiv.org/html/2608.20387#bib.bib13),[14](https://arxiv.org/html/2608.20387#bib.bib14)\], encode descriptions with traditional acoustic models, rather than leveraging GPT\-based semantic modeling\. Recent studies like FLEXIVOICE\[[18](https://arxiv.org/html/2608.20387#bib.bib18)\]concatenates instruction text with content text as GPT input but still requires an audio prompt for inference\. This dependency risks \`\`style leakage", where the prompt's acoustic properties may conflict with the intended instruction\. EmoVoice\[[22](https://arxiv.org/html/2608.20387#bib.bib22)\]eliminates the prompt requirement but restricts its scope to major emotions\. VoxInstruct\[[23](https://arxiv.org/html/2608.20387#bib.bib23)\]introduces an MT5\[[24](https://arxiv.org/html/2608.20387#bib.bib24)\]text encoder in GPT for diverse tasks with a codec language model\. TextrolSpeech\[[25](https://arxiv.org/html/2608.20387#bib.bib25)\]converts text into phonemes as GPT inputs, which may destroy high\-level semantic connections for instruction comprehension\.

Related to our architecture, Qwen3\-TTS\[[7](https://arxiv.org/html/2608.20387#bib.bib7)\]introduces the concept of \`\`thinking tokens" after the text input, yet the technical design and training pipeline remain undisclosed\. OV\-INSTRUCTTTS\[[19](https://arxiv.org/html/2608.20387#bib.bib19)\]adopts a GPT\-FM framework and supervises training with explicit LLM‑generated \`\`thinking text"\. We argue that forcing the acoustic model to predict complex, free\-form thinking text introduces unnecessary optimization difficulties\. In our setting, we observe that such a strategy can harm stability\. In contrast, our system employs attribute\-based thinking tokens, effectively steering the style generation without degrading the model's intrinsic semantic understanding\.

## 3Method

### 3\.1Dataset Construction Pipeline

#### 3\.1\.1Data Source

To capture the rich emotions and paralinguistic behaviors rarely found in traditional reading\-style corpora, we compiled an in\-the\-wild 1,000\-hour dataset from cinematic and television audiovisual media\. These sources contain expressive, context\-driven conversational interactions, abundant emotions, diverse paralinguistic behaviors, and various accents\. The media files are collected from publicly accessible sources and processed strictly for non\-commercial, academic research purposes\. We construct a data pipeline in Figure[1\(a\)](https://arxiv.org/html/2608.20387#S2.F1.sf1)to generate our instruction dataset as detailed described in Section[3\.1\.2](https://arxiv.org/html/2608.20387#S3.SS1.SSS2)and[3\.1\.3](https://arxiv.org/html/2608.20387#S3.SS1.SSS3)\.

Due to copyright restrictions of commercial cinematic media, the raw dataset cannot be publicly distributed\. However, we detail the pipeline and open\-source the LLM prompts for researchers to reproduce on their own corpora\.

#### 3\.1\.2Audio\-Text Segmentation & Alignment

To obtain clean single\-speaker utterances, we propose a cascaded data processing pipeline as Figure[1\(a\)](https://arxiv.org/html/2608.20387#S2.F1.sf1)\. First, we segment the videos into few\-minute clips based on long silences detected by subtitle timestamps and a Voice Activity Detection \(VAD\) tool\[[26](https://arxiv.org/html/2608.20387#bib.bib26)\]\. To eliminate post\-production sound effects and background music, we extract the center channel audio track from MKV files and denoise it with Demucs\[[27](https://arxiv.org/html/2608.20387#bib.bib27)\]and ClearerVoice\[[28](https://arxiv.org/html/2608.20387#bib.bib28)\]sequentially\. Then we use a commercial service, ElevenLabs’ Speech\-to\-Text API\[[20](https://arxiv.org/html/2608.20387#bib.bib20)\], including automatic speech recognition \(ASR\), speaker diarization \(SD\) and paralinguistic tagging \(PL\) to generate single\-speaker sentence\-level transcription with fine\-grained paralinguistic tags\. For example,Anna<breathing\>please,<whisper\>someone is there…</whisper\>Just kidding<laughter\>\!To ensure text accuracy, we employ a rule\-based fuzzy matching strategy between the ASR outputs and subtitles\. Sentences with a low Word Error Rate \(WER\) against the subtitles are replaced with the definitive subtitle text, while segments with a high WER are discarded as misrecognitions of background speech\.

#### 3\.1\.3Multi\-Modal Instruction Generation

With the text\-audio pairs prepared, we utilize a commercial service of multi\-modal LLM, Gemini 2\.5 Pro API\[[29](https://arxiv.org/html/2608.20387#bib.bib29)\], to generate natural language instructions via a three\-stage pipeline\. The prompts for all stages are available in the demo page\.

Stage 1 \- Content Summary:LLM integrates down\-sampled video \(720p\) with audio to extract the narrative structure, character relationships, and global contextual background\.

Stage 2 \- Transcript Analysis:Guided by the Stage\-1 summary, the sentence\-level transcripts and the video clip, the LLM annotates each sentence with multi\-dimensional attributes including gender, accent, emotion and speaking style\. The multi\-modal input is critical for capturing subtle, fine‑grained emotions that are difficult to infer from audio‑only input\.

Stage 3 \- Instruction Generation:Based on the global context and segment\-level attributes, the LLM generates a vivid, diverse natural language instruction for each utterance\.

### 3\.2Poly\-InstructTTS Framework

As illustrated in Figure[1\(b\)](https://arxiv.org/html/2608.20387#S2.F1.sf2), Poly\-InstructTTS follows a GPT\-FM architecture, including an auto\-regressive GPT to translate instruction and content texts into discrete speech tokens, an FM module to convert these tokens into mel\-spectrograms, and a HiFi\-Net\[[30](https://arxiv.org/html/2608.20387#bib.bib30)\]to reconstruct the audio waveform\. From an architectural perspective, the system is designed to route prosody and style to the GPT while injecting timbre only at the acoustic module\. The GPT is prompt\-audio\-free and an FM model from CosyVoice3\[[5](https://arxiv.org/html/2608.20387#bib.bib5)\]injects timbre conditioned on the reference audio\. In this way, the GPT learns to map instructions to content and prosody, and this setup mitigates style leakage between reference and instruction while preserving speaker similarity\.

The GPT first undergoes a pre\-training stage on large\-scale speech corpora of Emilia\[[31](https://arxiv.org/html/2608.20387#bib.bib31)\]and MLS\[[9](https://arxiv.org/html/2608.20387#bib.bib9)\]to acquire foundational TTS capabilities\. For the post\-training stage on our instruction dataset, each sample is serialized as<Instruct\> <Text\> <Thinking Tokens\> <Speech Tokens\>\. For example:<instruct\> He lets out a scream, the words drawn slowly with a desperate plea filled with fear\. </instruct\> <text\> Oh God, please no\! </text\> <think\> <male\> <high\> <shouting\> </think\> <SpeechTokens\>\. Instead of using LLM\-generated thinking text tokens as in OV\-InstructTTS\[[19](https://arxiv.org/html/2608.20387#bib.bib19)\], we adopt a compact set of attribute‑based thinking tokens to bridge instructions and acoustic tokens\. We argue that predicting complex LLM text introduces unnecessary optimization burdens and risks catastrophic forgetting of the content\-understanding abilities acquired during pre\-training\. The thinking token set includesgender,emotion intensity,style, andaccent\. Style and accent are selected as the top labels in the training corpus that occur 100\+ times and all attribute\-based tokens are treated as special tokens for GPT\. We deliberately exclude other attributes like pitch and emotion from the thinking tokens, so that the GPT remains responsible for realizing prosody from the instruction and the thinking tokens\. Practically, thegendertoken can reduce the pitch–timbre mismatches in FM \(e\.g\., high‑pitch prosody on a low‑pitch reference\), improving stability and speaker similarity\.

### 3\.3Instruction\-Conditioned Speaker Fine\-Tuning

While the base Poly\-InstructTTS model is capable of generating zero\-shot style with arbitrary timbres, real\-world applications often require endowing a specific speaker with instruction\-following capabilities while maintaining their intrinsic persona\. For this scenario, we propose an Instruction\-Conditioned Speaker Fine\-Tuning \(SFT\) scheme to transfer the expressive variance from our cinematic dataset to target speakers\. The training format of the GPT remains identical to the base stage, but we prepend a Speaker ID tag to the content text\. For the cinematic data, the tag is set to<Unknown\>, whereas SFT data utilizes specific IDs such as<spk\_01\>\. During inference, simply conditioning the prompt with the target Speaker ID allows the GPT model to seamlessly blend the learned instruction controllability with the fine\-tuned speaker characteristics\.

## 4Experiments

### 4\.1Experimental Setup

#### 4\.1\.1Training Dataset & Splits

Following the proposed pipeline, we processed a raw corpus of approximately 2,500 hours, yielding a highly expressive dataset of 1,000 hours with over 1\.1M utterances, and covers over 200 accents, 800 fine\-grained emotions and 400 stylistic variations annotated by the LLM\. For attribute\-based thinking tokens222full category lists are provided in the supplementary material\., we retainstyleandaccentlabels with 200\+ occurrences\. The instruction–audio match rate exceeds 95% according to human ratings\. We randomly split the corpus into 99% training and 1% validation\. For the subsequent Instruction\-SFT task, we additionally curate 200 hours of speech from 10 distinct speakers\.

#### 4\.1\.2Evaluation Testset

Our primary evaluation is based on the InstructTTSEval\[[12](https://arxiv.org/html/2608.20387#bib.bib12)\]benchmark\. However, this original benchmark under\-represents certain scenarios such as diverse accents, subtle/extreme emotions, and non\-mainstream styles\. To address these gaps, we introduce an expanded testset of 200 samples with the same format as the complementary of the base testset\. We evaluate on both the base and the expanded test sets\. Table[1](https://arxiv.org/html/2608.20387#S4.T1)describes the comparison between these testsets, and the expanded testset is available in our project page\.

Table 1:Comparison of attribute distribution between the base InstructTTSEval and the proposed Expanded Testset\.
#### 4\.1\.3Evaluation Metrics

For objective assessment, we measure the system stability using Word Error Rate \(WER\) and the instruction\-following capabilities using the metrics defined in InstructTTSEval: Acoustic\-Parameter Specification \(APS\) for basic acoustic instruction following, Descriptive\-Style Directive \(DSD\) for style assessment, and Role\-Play \(RP\) for evaluation of role play ability\.

Given that LLM\-based objective evaluators often struggle to reliably assess subtle paralinguistic nuances and extreme human emotions, subjective human evaluation is also conducted using mean opinion score \(MOS\) tests with 20 native listeners\. The evaluation encompasses two dimensions on a 5\-point scale:I\-MOS\(instruction\-following accuracy\) andN\-MOS\(speech naturalness and stability\)\.

Table 2:Comparison of the baselines and proposed systems with ablations on the performance of InstructTTSEvalBaseandExpandedtestsets\. The systems include closed\-source APIs, open\-source models and the proposed system with ablations\.- \*The 95% confidence intervals \(95% CI\) for all reported MOS values are⩽±\\leqslant\\pm0\.10\. Only the mean scores are presented above for brevity\.

#### 4\.1\.4Implementation Details

The GPT backbone is initialized from Qwen2\.5\-0\.5B\-Instruct\[[36](https://arxiv.org/html/2608.20387#bib.bib36)\]\. Speech tokens are extracted using the CosyVoice2\[[5](https://arxiv.org/html/2608.20387#bib.bib5)\]speech tokenizer at 25Hz\. The model is trained on 8 NVIDIA A800 GPUs for 30 epochs and optimized using the AdamW optimizer\[[37](https://arxiv.org/html/2608.20387#bib.bib37)\]with a learning rate of 1e\-4 and a batch duration of 300s\. During inference, the FM and vocoder are consistent with CosyVoice3\[[5](https://arxiv.org/html/2608.20387#bib.bib5)\], and the reference audios are randomly chosen from LibriTTS\[[8](https://arxiv.org/html/2608.20387#bib.bib8)\]\. the overall system achieves a Real\-Time Factor \(RTF\) of roughly 0\.06 on a single RTX 4090 GPU\.

### 4\.2Evaluation Results

We compare Poly\-InstructTTS against the instruction\-following TTS systems including open\-source baselines and closed source systems\. As shown in Tabel[2](https://arxiv.org/html/2608.20387#S4.T2), Poly\-InstructTTS achieves consistently high APS/DSD/RP scores, with the largest gains on RP, likely benefiting from our in\-the\-wild training data and thinking\-token design\. The moderate WER of our system may result from the challenging acoustic dynamics and label noise in the dataset compared to clean speech corpora\. Subjectively, our system achieves top‑1 I\-MOS on the base testset and top‑2 on the expanded set, demonstrating leading instruction adherence and expressiveness\. The optional comments in subjective tests indicate that our system can handle diverse styles and scenarios and produce expressive speech with fine\-grained emotions\.

Limitations and trade\-offs: evaluations also reveal weaknesses under challenging acoustic conditions \(e\.g\., noise, echo\), which appear more related to the FM than to the GPT instruction pathway\. During post\-training, we observe a stability–expressiveness trade\-off that as epochs increase, APS/DSD/RP steadily improve while WER rises\. We hypothesize that training on extreme emotions and paralinguistic behaviors weakens monotonic text–token alignment\. These limitations remain a future work for our system\.

### 4\.3Ablation Studies

To validate the core architectural designs, we conduct ablations in Table[2](https://arxiv.org/html/2608.20387#S4.T2)\. First, removing the attribute‑based thinking tokens lowers subjective scores, indicating they provide a useful prior that helps align instructions with acoustic tokens\. These tokens complement the free‑form instruction text, leaving the GPT to achieve better expressiveness compared to direct generation\. We also evaluate optional pretrained instruction encoders, including FlanT5\[[33](https://arxiv.org/html/2608.20387#bib.bib33)\], instructor\[[34](https://arxiv.org/html/2608.20387#bib.bib34)\]and GTR\-base\[[35](https://arxiv.org/html/2608.20387#bib.bib35)\], by conditioning the GPT on their frozen hidden representations of the instruction text \(dashed lines in Figure[1\(b\)](https://arxiv.org/html/2608.20387#S2.F1.sf2)\)\. Across settings, these variants does not yield positive gains versus the encoder‑free default\. This suggests that feeding raw instruction text directly to the GPT is sufficient for generating styled speech under our training setup\.

### 4\.4Instruction\-Conditioned SFT Analysis

Finally, we evaluate the model's performance on the speaker fine\-tuning to endow specific speakers with instruction control while preserving their personas\. We compared two systems: \(1\) the proposed SFT paradigm, and \(2\) the base Poly‑InstructTTS with only an FM‑stage timbre injection, both of which use the same timbre reference for FM\.

We conduct a subjective study on 10 fine‑tuned speakers in Table[3](https://arxiv.org/html/2608.20387#S4.T3)\. In addition to I‑MOS and N‑MOS, we report P‑MOS to indicate how closely the speech aligns with the speaker’s persona\. Results indicate that the SFT variant attains higher P‑MOS, whereas the base setup yields higher I‑MOS\. This suggests that SFT learns a speaker‑specific mapping from instructions to a suitable stylistic range, and is preferable despite the trade‑off in instruction adherence\. In addition, SFT achieves a higher N‑MOS, likely reflecting the higher quality and consistency of the SFT corpus\.

Table 3:Subjective evaluation comparing the proposed Instruction\-Conditioned SFT versus the base method\.

## 5Conclusion

In this paper, we introduce Poly\-InstructTTS, a highly expressive, instruction\-following TTS framework that bridges semantic understanding and complex acoustic realization\. To address the data scarcity, we build a multi\-modal pipeline to construct a 1,000\-hour in\-the\-wild cinematic corpus covering 1,000\+ fine\-grained emotions and styles\. Architecturally, we adopt a GPT\-FM framework with a prompt\-free GPT with attribute\-based thinking tokens to follow the open\-ended natural language instructions\. We also extend the InstructTTSEval testset\. Experiments show that Poly\-InstructTTS achieves strong performance in both objective and subjective metrics, comparable to the best baseline systems\. Furthermore, our instruction\-conditioned SFT strategy effectively endows target speakers with instruction controllability while preserving speaker personas\.

Future work will focus on better balancing the trade\-off between expressiveness and stability\. We will also improve the FM module to cope with more complex conditions\. Finally, we plan to move beyond reference\-conditioned timbre and enable text\-only, reference\-free voice generation, preserving controllability while removing the need for a timbre reference audio\.

## 6Generative AI Use Disclosure

We used OpenAI's GPT‑5 and Google’s Gemini 3\.0 Pro solely to polish the English language and improve readability during manuscript preparation\. The tools were not used to generate scientific ideas, methods or experimental results\. All technical content, analyses, claims, and conclusions are authored by the human authors\. All AI‑assisted edits were reviewed and verified, and the authors take full responsibility for the final text\. No confidential data or personally identifiable information were shared with the AI services\.

## References

- \[1\]S\. Jain, \`\`Expert delivers talk on motor parameter estimation at sgsits indore, india,''*IEEE Power Electronics Magazine*, vol\. 11, no\. 4, pp\. 109–110, 2024\.
- \[2\]Y\. Wang, H\. Zhan, L\. Liu, R\. Zeng, H\. Guo, J\. Zheng, Q\. Zhang, X\. Zhang, S\. Zhang, and Z\. Wu, \`\`MaskGCT: Zero\-shot text\-to\-speech with masked generative codec transformer,'' in*The Thirteenth International Conference on Learning Representations*, 2025\. \[Online\]\. Available:[https://openreview\.net/forum?id=ExuBFYtCQU](https://openreview.net/forum?id=ExuBFYtCQU)
- \[3\]P\. Anastassiou, J\. Chen, J\. Chen, Y\. Chen, Z\. Chen, Z\. Chen, J\. Cong, L\. Deng, C\. Ding, L\. Gao*et al\.*, \`\`Seed\-tts: A family of high\-quality versatile speech generation models,''*arXiv preprint arXiv:2406\.02430*, 2024\.
- \[4\]Z\. Du, Y\. Wang, Q\. Chen, X\. Shi, X\. Lv, T\. Zhao, Z\. Gao, Y\. Yang, C\. Gao, H\. Wang*et al\.*, \`\`Cosyvoice 2: Scalable streaming speech synthesis with large language models,''*arXiv preprint arXiv:2412\.10117*, 2024\.
- \[5\]Z\. Du, C\. Gao, Y\. Wang, F\. Yu, T\. Zhao, H\. Wang, X\. Lv, H\. Wang, C\. Ni, X\. Shi*et al\.*, \`\`Cosyvoice 3: Towards in\-the\-wild speech generation via scaling\-up and post\-training,''*arXiv preprint arXiv:2505\.17589*, 2025\.
- \[6\]S\. Zhou, Y\. Zhou, Y\. He, X\. Zhou, J\. Wang, W\. Deng, and J\. Shu, \`\`Indextts2: A breakthrough in emotionally expressive and duration\-controlled auto\-regressive zero\-shot text\-to\-speech,''*arXiv preprint arXiv:2506\.21619*, 2025\.
- \[7\]H\. Hu, X\. Zhu, T\. He, D\. Guo, B\. Zhang, X\. Wang, Z\. Guo, Z\. Jiang, H\. Hao, Z\. Guo*et al\.*, \`\`Qwen3\-tts technical report,''*arXiv preprint arXiv:2601\.15621*, 2026\.
- \[8\]H\. Zen, V\. Dang, R\. Clark, Y\. Zhang, R\. J\. Weiss, Y\. Jia, Z\. Chen, and Y\. Wu, \`\`LibriTTS: A Corpus Derived from LibriSpeech for Text\-to\-Speech,'' in*Interspeech 2019*, 2019, pp\. 1526–1530\.
- \[9\]V\. Pratap, Q\. Xu, A\. Sriram, G\. Synnaeve, and R\. Collobert, \`\`Mls: A large\-scale multilingual dataset for speech research,''*ArXiv*, vol\. abs/2012\.03411, 2020\.
- \[10\]A\. Adigwe, N\. Tits, K\. E\. Haddad, S\. Ostadabbas, and T\. Dutoit, \`\`The emotional voices database: Towards controlling the emotion dimension in voice generation systems,''*arXiv preprint arXiv:1806\.09514*, 2018\.
- \[11\]K\. Zhou, B\. Sisman, R\. Liu, and H\. Li, \`\`Emotional voice conversion: Theory, databases and esd,''*Speech Communication*, vol\. 137, pp\. 1–18, 2022\.
- \[12\]K\. Huang, Q\. Tu, L\. Fan, C\. Yang, D\. Zhang, S\. Li, Z\. Fei, Q\. Cheng, and X\. Qiu, \`\`Instructttseval: Benchmarking complex natural\-language instruction following in text\-to\-speech systems,''*arXiv preprint arXiv:2506\.16381*, 2025\.
- \[13\]Z\. Guo, Y\. Leng, Y\. Wu, S\. Zhao, and X\. Tan, \`\`Prompttts: Controllable text\-to\-speech with text descriptions,'' in*ICASSP 2023\-2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*\. IEEE, 2023, pp\. 1–5\.
- \[14\]Y\. Leng, Z\. Guo, K\. Shen, Z\. Ju, X\. Tan, E\. Liu, Y\. Liu, D\. Yang, leying zhang, K\. Song, L\. He, X\. Li, sheng zhao, T\. Qin, and J\. Bian, \`\`PromptTTS 2: Describing and generating voices with text prompt,'' in*The Twelfth International Conference on Learning Representations*, 2024\. \[Online\]\. Available:[https://openreview\.net/forum?id=NsCXDyv2Bn](https://openreview.net/forum?id=NsCXDyv2Bn)
- \[15\]A\. Vyas, B\. Shi, M\. Le, A\. Tjandra, Y\.\-C\. Wu, B\. Guo, J\. Zhang, X\. Zhang, R\. Adkins, W\. Ngan*et al\.*, \`\`Audiobox: Unified audio generation with natural language prompts,''*arXiv preprint arXiv:2312\.15821*, 2023\.
- \[16\]D\. Lyth and S\. King, \`\`Natural language guidance of high\-fidelity text\-to\-speech with synthetic annotations,''*arXiv preprint arXiv:2402\.01912*, 2024\.
- \[17\]Z\. Jin, J\. Jia, Q\. Wang, K\. Li, S\. Zhou, S\. Zhou, X\. Qin, and Z\. Wu, \`\`Speechcraft: A fine\-grained expressive speech dataset with natural language description,'' in*Proceedings of the 32nd ACM International Conference on Multimedia*, 2024, pp\. 1255–1264\.
- \[18\]D\. Chen, X\. Zhang, Y\. Wang, K\. Dai, L\. Ma, and Z\. Wu, \`\`Flexivoice: Enabling flexible style control in zero\-shot tts with natural language instructions,''*arXiv preprint arXiv:2601\.04656*, 2026\.
- \[19\]Y\. Ren, J\. Yi, J\. Tao, H\. Sun, Z\. Wen, H\. Gu, L\. Xu, and Y\. Bai, \`\`Ov\-instructtts: Towards open\-vocabulary instruct text\-to\-speech,''*arXiv preprint arXiv:2601\.01459*, 2026\.
- \[20\]ElevenLabs, \`\`Speech\-to\-text api documentation,''[https://docs\.elevenlabs\.io/](https://docs.elevenlabs.io/), 2026, accessed: May 30, 2025\.
- \[21\]D\. Yang, S\. Liu, R\. Huang, C\. Weng, and H\. Meng, \`\`Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt,''*IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol\. 32, pp\. 2913–2925, 2024\.
- \[22\]G\. Yang, C\. Yang, Q\. Chen, Z\. Ma, W\. Chen, W\. Wang, T\. Wang, Y\. Yang, Z\. Niu, W\. Liu*et al\.*, \`\`Emovoice: Llm\-based emotional text\-to\-speech model with freestyle text prompting,'' in*Proceedings of the 33rd ACM International Conference on Multimedia*, 2025, pp\. 10 748–10 757\.
- \[23\]Y\. Zhou, X\. Qin, Z\. Jin, S\. Zhou, S\. Lei, S\. Zhou, Z\. Wu, and J\. Jia, \`\`Voxinstruct: Expressive human instruction\-to\-speech generation with unified multilingual codec language modelling,'' in*Proceedings of the 32nd ACM International Conference on Multimedia*, 2024, pp\. 554–563\.
- \[24\]L\. Xue, N\. Constant, A\. Roberts, M\. Kale, R\. Al\-Rfou, A\. Siddhant, A\. Barua, and C\. Raffel, \`\`mt5: A massively multilingual pre\-trained text\-to\-text transformer,'' in*Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies*, 2021, pp\. 483–498\.
- \[25\]S\. Ji, J\. Zuo, M\. Fang, Z\. Jiang, F\. Chen, X\. Duan, B\. Huai, and Z\. Zhao, \`\`Textrolspeech: A text style control speech corpus with codec language text\-to\-speech models,'' in*ICASSP 2024\-2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*\. IEEE, 2024, pp\. 10 301–10 305\.
- \[26\]S\. Team, \`\`Silero vad: pre\-trained enterprise\-grade voice activity detector \(vad\), number detector and language classifier,''[https://github\.com/snakers4/silero\-vad](https://github.com/snakers4/silero-vad), 2024\.
- \[27\]A\. Défossez, N\. Usunier, L\. Bottou, and F\. Bach, \`\`Demucs: Deep extractor for music sources with extra unlabeled data remixed,''*arXiv preprint arXiv:1909\.01174*, 2019\.
- \[28\]S\. Zhao, Z\. Pan, and B\. Ma, \`\`Clearervoice\-studio: Bridging advanced speech processing research and practical deployment,''*arXiv preprint arXiv:2506\.19398*, 2025\.
- \[29\]Google, \`\`Gemini api documentation,''[https://ai\.google\.dev/gemini\-api](https://ai.google.dev/gemini-api), 2026, accessed: May 30, 2025\.
- \[30\]Y\. A\. Li, C\. Han, X\. Jiang, and N\. Mesgarani, \`\`Hiftnet: A fast high\-quality neural vocoder with harmonic\-plus\-noise filter and inverse short time fourier transform,''*arXiv preprint arXiv:2309\.09493*, 2023\.
- \[31\]H\. He, Z\. Shang, C\. Wang, X\. Li, Y\. Gu, H\. Hua, L\. Liu, C\. Yang, J\. Li, P\. Shi*et al\.*, \`\`Emilia: An extensive, multilingual, and diverse speech dataset for large\-scale speech generation,'' in*2024 IEEE Spoken Language Technology Workshop \(SLT\)*\. IEEE, 2024, pp\. 885–890\.
- \[32\]D\. Zhang, G\. Wang, J\. Xue, K\. Fang, L\. Zhao, R\. Ma, S\. Ren, S\. Liu, T\. Guo, W\. Zhuang*et al\.*, \`\`Mimo\-audio: Audio language models are few\-shot learners,''*arXiv preprint arXiv:2512\.23808*, 2025\.
- \[33\]S\. Longpre, L\. Hou, T\. Vu, A\. Webson, H\. W\. Chung, Y\. Tay, D\. Zhou, Q\. V\. Le, B\. Zoph, J\. Wei*et al\.*, \`\`The flan collection: Designing data and methods for effective instruction tuning,'' in*International conference on machine learning*\. PMLR, 2023, pp\. 22 631–22 648\.
- \[34\]J\. Ni, C\. Qu, J\. Lu, Z\. Dai, G\. H\. Abrego, J\. Ma, V\. Zhao, Y\. Luan, K\. Hall, M\.\-W\. Chang*et al\.*, \`\`Large dual encoders are generalizable retrievers,'' in*Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, 2022, pp\. 9844–9855\.
- \[35\]H\. Su, W\. Shi, J\. Kasai, Y\. Wang, Y\. Hu, M\. Ostendorf, W\.\-t\. Yih, N\. A\. Smith, L\. Zettlemoyer, and T\. Yu, \`\`One embedder, any task: Instruction\-finetuned text embeddings,'' in*Findings of the Association for Computational Linguistics: ACL 2023*, 2023, pp\. 1102–1121\.
- \[36\]Q\. Team, \`\`Qwen2\.5: A party of foundation models,'' September 2024\. \[Online\]\. Available:[https://qwenlm\.github\.io/blog/qwen2\.5/](https://qwenlm.github.io/blog/qwen2.5/)
- \[37\]D\. P\. Kingma and J\. Ba, \`\`Adam: A method for stochastic optimization,''*arXiv preprint arXiv:1412\.6980*, 2014\.