VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents

# VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents
Source: [https://arxiv.org/html/2608.13831](https://arxiv.org/html/2608.13831)
Casanova Kim Graterol Fuenmayor Hussain Klimkov Mendelev Desta Neekhara Zelasko Chen Rastorgueva Hu Pasad Yang Alja’fari Roy Badlani Roche Li Chen

JaehyeonMarianaShehzeenViacheslavValentinMikyasPaarthPiotrChenElenaKeAnkitaXuesongAyaRajarshiRohanJasonJasonZhehuai

###### Abstract

Spoken dialogue is a natural form of human–computer interaction, yet most speech language models remain limited to turn\-based operation and lack real\-time adaptability, such as user barge\-in\. Recent duplex speech\-to\-speech and speech\-to\-text models reduce latency by replacing multi\-stage pipelines, but often compromise speech quality because accurate ASR, interruption handling, and high\-fidelity synthesis must be optimized jointly\. We propose VoiceChat\-TTS, a low\-latency, continuous, and streamable text\-to\-speech model for interactive agents\. VoiceChat\-TTS is driven directly by LLM text\-token streams, supports explicit interruption via control tokens, and produces silence when no textual input is available\. The model enables always\-on, responsive speech generation while preserving modularity and high speech quality, and it supports mid\-utterance interruptions without resetting the KV cache\.

###### keywords

Speech synthesis, text\-to\-speech, streaming TTS, speech LLMs

††address:NVIDIA Corporation††email:ecasanova@nvidia\.com## 1Introduction

Spoken dialogue is a natural and intuitive modality for human–computer interaction\. However, most existing speech language models remain constrained to turn\-based operation and lack real\-time adaptability, such as support for user barge\-in\. Recent duplex speech\-to\-speech \(S2S\)\[[7](https://arxiv.org/html/2608.13831#bib.bib14),[4](https://arxiv.org/html/2608.13831#bib.bib15),[18](https://arxiv.org/html/2608.13831#bib.bib16)\]and speech\-to\-text\[[14](https://arxiv.org/html/2608.13831#bib.bib21)\]models achieve low latency and simplified deployment by replacing traditional multi\-stage pipelines that rely on separate automatic speech recognition \(ASR\), voice activity detection \(VAD\), and text\-to\-speech \(TTS\) components\. ASR systems transcribe spoken audio into text, VAD detects the presence or absence of speech to guide downstream processing, and TTS generates waveform output from text input\.

While end\-to\-end duplex models are highly promising, they often exhibit degraded speech synthesis quality\. This degradation largely arises from the substantial data and modeling challenges involved in jointly optimizing accurate ASR, robust interruption handling, and high\-quality speech generation within a single architecture\.

Prior work on speech\-to\-speech \(S2S\) models has explored architectures that incorporate dedicated speech decoders connected to a shared backbone through latent representations\[[3](https://arxiv.org/html/2608.13831#bib.bib1),[22](https://arxiv.org/html/2608.13831#bib.bib13)\]\. Although effective in reducing overall interaction latency, these approaches increase architectural complexity and can compromise the modularity and debuggability of traditional pipeline systems\.

Recently, streaming TTS has made significant progress in reducing time\-to\-first\-audio \(TTFA\) and enabling incremental speech generation\. One early example is the streaming speech decoder of Audio Flamingo 3\-Chat\[[5](https://arxiv.org/html/2608.13831#bib.bib17)\], which was designed for low\-latency conversational speech generation by consuming streamable text inputs and producing speech tokens progressively\. This design demonstrated that high\-quality neural speech synthesis can be integrated into interactive speech\-language systems without requiring the complete text response in advance\. Subsequent streaming TTS systems further improved latency and alignment\. For example, VoXtream\[[20](https://arxiv.org/html/2608.13831#bib.bib2)\]uses an incremental decoder\-only Transformer with monotonic alignment and limited look\-ahead to map incoming phonemes to acoustic tokens, while SpeakStream\[[1](https://arxiv.org/html/2608.13831#bib.bib11)\]employs a decoder\-only architecture trained with interleaved text\-and\-speech sequences to absorb streaming text from large language models \(LLMs\)\. In addition, Qwen3\-TTS\[[6](https://arxiv.org/html/2608.13831#bib.bib12)\]explores efficient text\-to\-speech alignment for real\-time applications by directly ingesting LLM tokens to achieve very low TTFA while maintaining high synthesis quality and expressiveness\.

However, these systems primarily address streaming generation within a single response\. They do not natively model the continuous, always\-on behavior required by full\-duplex interactive agents, where the speech decoder must remain active across conversational time, generate silence when no agent text is available, and stop promptly in response to user barge\-in\. VoiceChat\-TTS builds on this line of streaming speech decoders, particularly the Audio Flamingo 3\-Chat speech decoder, and extends it with explicit silence modeling, interruption control tokens, and a unified training formulation for both single\-turn and multi\-turn conversational synthesis\.

In this paper, we propose VoiceChat\-TTS, a continuous, streamable, and low\-latency text\-to\-speech model designed for interactive agents\. VoiceChat\-TTS is driven directly by large language model \(LLM\) text\-token streams, supports explicit interruption through control tokens, and produces silence when no textual input is available\. This design enables always\-on, responsive speech generation while preserving modularity and high synthesis quality\. Furthermore, VoiceChat\-TTS is compatible with both duplex interaction frameworks and speech\-to\-speech pipeline systems, and avoids resetting the KV cache when generation is interrupted mid\-utterance\.

The main contributions of our work are as follows:

- •We introduce VoiceChat\-TTS, a continuous, streamable, and low\-latency TTS model that directly consumes LLM text\-token streams and generates silence when no agent text is available;
- •We propose a reliable control\-token\-based interruption mechanism that halts ongoing speech and transitions the output to silence during mid\-utterance user barge\-ins;
- •We present a unified training strategy that combines high\-quality single\-turn TTS data with complex multi\-turn conversational data while minimizing the distribution mismatch between the two settings;
- •We demonstrate that VoiceChat\-TTS achieves competitive speech quality relative to strong offline and streaming baselines while meeting the latency and interruption\-handling requirements of interactive agents\.

VoiceChat\-TTS code is publicly available in NVIDIA NeMo Speech111https://github\.com/NVIDIA\-NeMo/Speech, and the model checkpoint is available on Hugging Face as part of NVIDIA\-NemotronLabs\-VoiceChat\-11B222https://huggingface\.co/nvidia/NVIDIA\-NemotronLabs\-VoiceChat\-11B\.

## 2VoiceChat\-TTS Model

The proposed architecture builds directly upon the streaming speech decoder of Audio Flamingo 3\-Chat\[[5](https://arxiv.org/html/2608.13831#bib.bib17)\], incorporating several modifications to support full\-duplex interactions\. The Audio Flamingo 3\-Chat streaming speech decoder was designed for low\-latency conversational use cases and real\-time interactions\. It relies on streamable inputs, allowing text to be processed incrementally without requiring the full sequence in advance, and produces streamable outputs by generating speech tokens progressively for immediate audio playback\. However, fully duplex interactions impose additional operational requirements: the system must handle mid\-sentence user interruptions and produce silence while the user is speaking to support natural turn\-taking\. To fulfill these requirements, we implement several modifications on top of the base Audio Flamingo 3\-Chat streaming speech decoder architecture\. Figure[1](https://arxiv.org/html/2608.13831#S2.F1)shows an overview of the VoiceChat\-TTS architecture\.

![Refer to caption](https://arxiv.org/html/2608.13831v1/Images/VoiceChat-TTS.png)

Figure 1:VoiceChat\-TTS architecture overview\.Audio Codec:The audio codec backbone is a fully causal autoencoder composed entirely of convolutional layers, similar to that of\[[5](https://arxiv.org/html/2608.13831#bib.bib17)\]\. In our work, we train the codec to compress 22 kHz waveforms at a frame rate of 12\.5 Hz using 31\-codebook Residual Vector Quantization \(RVQ\) tokens\. In this configuration, each generated audio\-token frame corresponds to an 80 ms waveform chunk\. This frame rate is compatible with recent duplex speech\-to\-speech models such as\[[7](https://arxiv.org/html/2608.13831#bib.bib14),[4](https://arxiv.org/html/2608.13831#bib.bib15)\]\. For efficient streaming codec inference, the network caches the inputs of the convolutional blocks only within their receptive field\.

Text Tokenizer:In contrast to Audio Flamingo 3\-Chat\[[5](https://arxiv.org/html/2608.13831#bib.bib17)\], we use the NVIDIA Nemotron Nano 2 subword tokenizer\[[15](https://arxiv.org/html/2608.13831#bib.bib3)\]and augment it with Beginning\-of\-Sequence \(BOS\) and interruption tokens\. On the LibriTTStest\-cleansubset, the tokenizer produces text tokens at an average rate of 4\.16 Hz, corresponding to approximately one text token for every three acoustic\-token frames\. During training, the text stream is right\-padded to match the length of the acoustic sequence, enabling incremental processing throughout the interaction\. A BOS token marks the beginning of each assistant turn, while an interruption token marks the point at which the model should stop speaking and transition to silence\. We additionally introduce a one\-token delay between the text and audio channels, with the text stream leading the audio stream\. This provides limited linguistic look\-ahead, ensuring that the model observes at least two subword tokens before generating the corresponding speech\.

Character\-Aware Subword Encoder:A prevalent issue in incremental text\-to\-speech models that consume LLM subword tokens is the mismatch between LLM vocabularies and TTS training corpora; many subwords produced by the former are rare or absent in the latter\. To mitigate this problem, we introduce a Character\-Aware Subword Encoder\. Each input subword is first converted into a sequence of characters, which is processed by a shallow Transformer encoder\. We then average\-pool the character\-level outputs to obtain robust character\-aware embeddings that can generalize to unseen subwords\. Figure[2](https://arxiv.org/html/2608.13831#S2.F2)shows an overview of the Character\-Aware Subword Encoder\. In addition, to improve pronunciation accuracy during incremental generation of long or fragmented words, we incorporate a dedicated embedding for subword\-continuation tokens\. This allows the model to process and synthesize multi\-token linguistic units more cohesively\.

![Refer to caption](https://arxiv.org/html/2608.13831v1/Images/CAS.png)

Figure 2:Character\-Aware Subword Encoder architecture overview\.Mixture of Gaussian Estimation Head:To accelerate generation of the deep RVQ token hierarchy, we integrate the Mixture of Gaussian head \(MoGH\)\[[9](https://arxiv.org/html/2608.13831#bib.bib18),[5](https://arxiv.org/html/2608.13831#bib.bib17)\]into the architecture\. Instead of relying on a 31\-step autoregressive decoding pipeline, this head iteratively unmasks the RVQ tokens\. In each iteration, it predicts the continuous embedding vector of the masked RVQ tokens using MoG estimation and subsequently quantizes the vector into progressively unmasked discrete tokens\. This iterative refinement is controllable, and prior results indicate that 4 to 8 iterations are sufficient to achieve high\-fidelity acoustic reconstruction\[[5](https://arxiv.org/html/2608.13831#bib.bib17)\]\.

Audio Prompt Conditioning:The Audio Flamingo 3\-Chat streaming speech decoder does not employ explicit speaker\-reference conditioning\. Instead, it is trained on concatenated utterances from the same speaker and learns to infer speaker identity from preceding acoustic context\. This approach can be less effective at the beginning of generation, when little or no speaker\-informative speech is available, and in conversational settings, where the recent acoustic context may be dominated by silence while the user is speaking\. To provide an explicit speaker cue, we condition VoiceChat\-TTS on a 3\-second reference audio prompt\. During training, the corresponding audio tokens prefill the beginning of the acoustic sequence, and the loss over this prompt region is masked so that the model uses the prompt as conditioning context rather than learning to reconstruct it\. During inference, the same 3\-second prompt initializes the acoustic context before synthesis begins\.

Boundary Embeddings and Gated Fusion:Since our training objective targets multi\-turn conversations, we introduce learnable text embeddings for the BOS and interruption tokens\. These embeddings provide explicit boundary signals that improve the smoothness of conversational turn\-taking\. We also address an architectural instability in the base decoder, where high\-magnitude RVQ embeddings can cause mixed\-precision overflow and destabilize early\-layer activations\. We mitigate this issue by applying a gated fusion mechanism between incoming text embeddings and acoustic speech embeddings, ensuring stable numerical scaling throughout the network\.

The final VoiceChat\-TTS model comprises 977M parameters in total, including a 778M\-parameter Gemma 3\-based streaming TTS module\[[19](https://arxiv.org/html/2608.13831#bib.bib19)\]and a 199M\-parameter codec model\.

## 3Experiments

To train the VoiceChat\-TTS model, we utilize a combination of standard single\-turn text\-to\-speech corpora, synthetically generated multi\-turn dialogues, and real\-world conversational datasets\. This diverse data mixture is essential for ensuring robust acoustic quality while simultaneously teaching the model complex interaction dynamics, such as turn\-taking and interruption\.

### 3\.1Single\-turn data

To establish a robust foundation for acoustic generation, we leverage large\-scale, standard text\-to\-speech corpora for the single\-turn training phase\. Specifically, we use approximately 70,159 hours of English speech sourced from the*train\-clean\-360*and*train\-clean\-100*subsets of LibriTTS\[[24](https://arxiv.org/html/2608.13831#bib.bib6)\], the original HiFiTTS corpus\[[2](https://arxiv.org/html/2608.13831#bib.bib8)\], an extended 36\.7k\-hour subset of HiFiTTS\-2\[[12](https://arxiv.org/html/2608.13831#bib.bib24)\], and a proprietary 62\-hour dataset comprising two high\-fidelity speakers\. We additionally incorporate a 32k\-hour internal dataset derived from high\-quality, publicly available YouTube videos under the Creative Commons license\.

A critical challenge in training duplex models is the distribution shift between single\-turn generation, which typically begins speech synthesis immediately, and multi\-turn interactions, which often begin with extended periods of silence while the user speaks\. To mitigate this modality discrepancy, we apply a targeted data augmentation strategy to 50% of our single\-turn data\. We randomly prepend between 0\.5 and 5\.0 seconds of background silence to the beginning of the audio sequences, shifting the corresponding text alignments accordingly\. This technique effectively simulates a multi\-turn dialogue context where the agent is initially in a listening state\. By bridging this gap, we prevent the model from trivially distinguishing between standard TTS tasks and duplex scenarios\.

### 3\.2Multi\-turn data

Synthetic datasets:To explicitly model complex duplex behaviors, we develop a curated synthetic multi\-turn dataset comprising approximately 2\.5k hours of speech\. Millions of textual dialogue scripts are generated by open\-source LLMs using NeMo Data Designer, with prompts tuned to cover a wide range of topics and situations\. Each dialogue contains up to 15 turns and includes LLM\-generated paralinguistic annotations\. We then synthesize the scripts using a pipeline based on Chatterbox\[[17](https://arxiv.org/html/2608.13831#bib.bib20)\]and Koel\-TTS\[[8](https://arxiv.org/html/2608.13831#bib.bib22)\], producing multi\-speaker dialogues with speaker identities drawn from the LibriTTS training sets\. Crucially, 50% of this synthetic data is designed to simulate overlapping speech and user barge\-ins\. This provides the alignment and explicit control tokens needed for the model to learn interruption handling\. Furthermore, to expand phonetic coverage and improve robustness to out\-of\-vocabulary terms in conversational settings, we synthesize specialized multi\-turn interactions containing complex words and proper names\. In these interactions, each turn consists of four distinct words, systematically increasing the lexical diversity observed during duplex training\.

Real conversation datasets:While synthetic data provides precise control over interaction timing and interruption labels, real conversational data is important for capturing natural dialogue dynamics\. To this end, we incorporate unscripted data from the Fisher corpus alongside a small internal conversational dataset\. These data allow the model to learn nuanced acoustic transitions, spontaneous backchannels, and natural turn\-taking patterns that are difficult to fully reproduce through synthetic generation alone\. Accurately transcribing paralinguistic events in spontaneous speech remains challenging; consequently, these datasets often contain untranscribed acoustic phenomena, such as laughter, sighs, and subtle backchannels, which the model must handle robustly\.

### 3\.3Experimental setup

We train VoiceChat\-TTS in two stages\. First, we pretrain the model for 1\.4M steps using only single\-turn TTS data on 8 NVIDIA A100 GPUs\. This stage provides a strong acoustic and linguistic foundation before introducing more complex conversational behavior\. We then fine\-tune the model for 200k steps on 32 NVIDIA H100 GPUs using the full training mixture, including single\-turn TTS data, word and proper\-name lists, real conversational speech, and synthetic duplex dialogues\.

During fine\-tuning, we use Lhotse\[[23](https://arxiv.org/html/2608.13831#bib.bib23)\]dynamic bucketing to construct batches containing approximately 240 seconds of audio\. The training mixture consists of 40% single\-turn TTS data, 15% word and proper\-name list data, 10% real conversational data, and 35% synthetic duplex conversational data\. To prevent any modality discrepancy during training, single\-turn and multi\-turn samples are processed identically\. In both formats, text tokens are right\-padded with special padding IDs within the text channel until they match the total temporal length of the current conversational turn\[[7](https://arxiv.org/html/2608.13831#bib.bib14),[3](https://arxiv.org/html/2608.13831#bib.bib1)\]\.

For all training stages, we use the AdamW optimizer with a learning rate of4×10−54\\times 10^\{\-5\}\.

### 3\.4Results and Discussion

We evaluate VoiceChat\-TTS in terms of intelligibility, speaker similarity, and predicted overall speech quality, following the evaluation protocol of\[[8](https://arxiv.org/html/2608.13831#bib.bib22)\]\. Intelligibility is measured using ASR\-based character error rate \(CER\) and word error rate \(WER\), where transcriptions are produced by Parakeet\-TDT\[[21](https://arxiv.org/html/2608.13831#bib.bib9)\]\. Speaker similarity is measured as the cosine similarity between speaker embeddings extracted from the synthesized speech and the reference prompt audio usingTitaNet\-Large\[[10](https://arxiv.org/html/2608.13831#bib.bib5)\]\. We denote this metric as speaker embedding cosine similarity \(SECS\)\. Overall speech quality is estimated using Squim\-MOS\[[11](https://arxiv.org/html/2608.13831#bib.bib4)\]; we use it as a non\-intrusive quality estimate rather than as a direct measure of naturalness\.

At inference time, we use multinomial top\-ppsampling withp=0\.95p=0\.95, a classifier\-free guidance \(CFG\) scale of0\.20\.2, and a noise scale of0\.0010\.001during Mixture\-of\-Gaussians \(MoG\) sampling\. We use88MoGH refinement iterations for all quality and latency evaluations\. Because generation is stochastic, each experiment is repeated ten times and we report mean values with 95% confidence intervals\. Fixed reference outputs are evaluated once\.

For unseen\-speaker evaluation, we use the fulltest\-cleansubset of LibriTTS\[[24](https://arxiv.org/html/2608.13831#bib.bib6)\]\. For seen\-speaker evaluation, we use five speakers observed during training\. To evaluate the model in multi\-turn settings, we construct synthetic multi\-turn test conversations by grouping LibriTTS utterances from the same speaker\. This setup allows us to measure whether synthesis quality remains stable as the number of generated turns increases\.

Table 1:Comparison between TTS models\. VoiceChat\-TTS is evaluated with different numbers of consecutive turns\.*Unseen Speakers**Seen Speakers*ModelType\# TurnsCER\(%\)↓\\downarrowWER\(%\)↓\\downarrowSECS↑\\uparrowSquim\-MOS↑\\uparrowCER\(%\)↓\\downarrowWER\(%\)↓\\downarrowSECS↑\\uparrowSquim\-MOS↑\\uparrowGround Truth\-\-0\.480\.481\.401\.400\.8300\.8304\.4574\.457\-\-\-\-Chatterbox\-TTS\[[16](https://arxiv.org/html/2608.13831#bib.bib7)\]Offline\-0\.45±0\.020\.45\\pm 0\.021\.24±0\.021\.24\\pm 0\.020\.887±0\.001\\mathbf\{0\.887\\pm 0\.001\}4\.27±0\.0034\.27\\pm 0\.003\-\-\-\-Audio Flamingo 3\-Chat\[[5](https://arxiv.org/html/2608.13831#bib.bib17)\]Streaming\-2\.85±0\.332\.85\\pm 0\.334\.51±0\.414\.51\\pm 0\.410\.761±0\.0020\.761\\pm 0\.0023\.60±0\.0093\.60\\pm 0\.009\-\-\-\-Qwen3\-TTS\-12Hz\-1\.7B\[[6](https://arxiv.org/html/2608.13831#bib.bib12)\]Streaming\-0\.34±0\.02\\mathbf\{0\.34\\pm 0\.02\}1\.01±0\.05\\mathbf\{1\.01\\pm 0\.05\}0\.827±0\.0010\.827\\pm 0\.0014\.45±0\.006\\mathbf\{4\.45\\pm 0\.006\}\-\-\-\-VoiceChat\-TTSStreaming11\.00±0\.101\.00\\pm 0\.102\.00±0\.102\.00\\pm 0\.100\.757±0\.0040\.757\\pm 0\.0044\.380±0\.0044\.380\\pm 0\.0041\.20±0\.101\.20\\pm 0\.102\.40±0\.102\.40\\pm 0\.100\.785±0\.0010\.785\\pm 0\.0014\.365±0\.0014\.365\\pm 0\.001VoiceChat\-TTSStreaming21\.00±0\.101\.00\\pm 0\.101\.90±0\.101\.90\\pm 0\.100\.710±0\.0010\.710\\pm 0\.0014\.377±0\.0024\.377\\pm 0\.0020\.80±0\.100\.80\\pm 0\.101\.90±0\.101\.90\\pm 0\.100\.789±0\.0050\.789\\pm 0\.0054\.358±0\.0044\.358\\pm 0\.004VoiceChat\-TTSStreaming31\.10±0\.101\.10\\pm 0\.102\.10±0\.102\.10\\pm 0\.100\.696±0\.0010\.696\\pm 0\.0014\.381±0\.0034\.381\\pm 0\.0030\.80±0\.100\.80\\pm 0\.101\.80±0\.101\.80\\pm 0\.100\.778±0\.0010\.778\\pm 0\.0014\.362±0\.0014\.362\\pm 0\.001VoiceChat\-TTSStreaming41\.20±0\.201\.20\\pm 0\.202\.20±0\.202\.20\\pm 0\.200\.685±0\.0050\.685\\pm 0\.0054\.376±0\.0034\.376\\pm 0\.0030\.80±0\.100\.80\\pm 0\.101\.80±0\.101\.80\\pm 0\.100\.778±0\.0010\.778\\pm 0\.0014\.359±0\.0014\.359\\pm 0\.001

Table[1](https://arxiv.org/html/2608.13831#S3.T1)compares VoiceChat\-TTS with strong offline and streaming TTS baselines\. Conventional TTS systems, such as Chatterbox\-TTS and Qwen3\-TTS, achieve the best single\-turn intelligibility, speaker similarity, and predicted overall speech quality\. However, these models are optimized for standard single\-response synthesis and are not designed for continuous duplex interaction, where the decoder must remain active across conversational time, generate silence when no agent text is available, and respond reliably to interruption control signals\.

We also include the Audio Flamingo 3\-Chat streaming speech decoder\[[5](https://arxiv.org/html/2608.13831#bib.bib17)\], the closest architectural baseline to VoiceChat\-TTS\. Compared with this baseline, VoiceChat\-TTS substantially improves intelligibility and predicted overall speech quality\. In the one\-turn setting, VoiceChat\-TTS reduces WER from4\.51%4\.51\\%to2\.00%2\.00\\%and increases Squim\-MOS from3\.603\.60to4\.384\.38, while achieving comparable speaker similarity\. These results show that the complete VoiceChat\-TTS system improves upon the Audio Flamingo 3\-Chat streaming decoder while extending it to continuous multi\-turn generation, silence modeling, and interruption\-aware synthesis\.

VoiceChat\-TTS achieves competitive predicted overall speech quality while supporting the additional requirements of interactive agents\. In the unseen\-speaker setting, the model obtains a Squim\-MOS of approximately4\.384\.38across all evaluated turn counts, close to the strongest baselines\. Intelligibility also remains stable as the number of turns increases from one to four, with WER varying only from2\.00%2\.00\\%to2\.20%2\.20\\%\. For seen speakers, all metrics remain similarly stable across multi\-turn generation: CER and WER remain low, Squim\-MOS stays around4\.364\.36, and SECS remains within a narrow range from0\.7780\.778to0\.7890\.789\. This small seen\-speaker similarity gap indicates that VoiceChat\-TTS can preserve speaker identity consistently across multiple turns when the target speaker is well represented during training\.

For unseen speakers, SECS decreases more noticeably as the number of generated turns increases, from0\.7570\.757for one turn to0\.6850\.685for four turns\. This suggests that maintaining speaker identity over extended continuous generation remains more challenging in zero\-shot or unseen\-speaker settings, particularly when the model alternates between speech and silence states\. Improving long\-context speaker consistency for unseen speakers is therefore an important direction for future work, potentially through stronger prompt conditioning or explicit speaker\-consistency objectives\.

Finally, we compare VoiceChat\-TTS with the PersonaPlex speech\-to\-speech \(S2S\) model\[[18](https://arxiv.org/html/2608.13831#bib.bib16)\]\. PersonaPlex jointly predicts text and audio tokens\. For this comparison, we use its outputs on the Smooth Turn Taking subset of Full\-Duplex\-Bench\[[13](https://arxiv.org/html/2608.13831#bib.bib10)\]\. To isolate the contribution of the speech decoder, we keep the PersonaPlex\-predicted text\-token stream fixed and resynthesize it with VoiceChat\-TTS\. We preserve the original temporal alignment of this stream, including PAD tokens during intervals in which the user is speaking\. VoiceChat\-TTS therefore runs continuously over the complete conversation timeline and is expected to generate silence during these intervals\. CER and WER are computed from the full, untrimmed assistant output; consequently, any intelligible speech generated during an intended silence interval contributes ASR insertion errors\. This setup evaluates both speech generation quality and the model’s ability to remain silent during user speech while preserving the same linguistic content and timing\. Since speaker identities are not matched between systems, we omit speaker\-similarity metrics from this comparison\.

Table 2:Comparison between PersonaPlex and VoiceChat\-TTS on the Smooth Turn Taking subset of Full\-Duplex\-Bench, using the same PersonaPlex\-predicted textual content\.Table[2](https://arxiv.org/html/2608.13831#S3.T2)shows that resynthesizing the PersonaPlex\-predicted text with VoiceChat\-TTS improves both intelligibility and predicted overall speech quality while preserving the original linguistic content and text\-token timing\. VoiceChat\-TTS reduces CER from4\.06%4\.06\\%to2\.05%2\.05\\%and WER from5\.00%5\.00\\%to2\.42%2\.42\\%\. Because these metrics are computed on the full, untrimmed assistant waveform, any intelligible speech produced during PAD\-designated user\-speech intervals contributes ASR insertion errors\. The lower error rates therefore indicate both more intelligible synthesis of the intended content and limited intelligible speech leakage during intervals in which the model is expected to remain silent\. VoiceChat\-TTS also improves Squim\-MOS from4\.0944\.094to4\.2924\.292, indicating higher predicted perceptual quality\. Overall, these results demonstrate the benefit of using VoiceChat\-TTS as a modular, high\-quality speech decoder in continuous S2S pipelines\.

### 3\.5Interruption Evaluation

To evaluate whether VoiceChat\-TTS responds correctly to explicit interruption signals, we construct an FDB\-timed controlled\-interruption benchmark based on the User Interruption subset of Full\-Duplex\-Bench \(FDB\)\. This evaluation should not be interpreted as an official FDB score\. We use FDB only for the user audio and interruption\-timing annotations; because the benchmark does not provide canonical assistant transcripts suitable for TTS\-only evaluation, we generate controlled assistant responses with known reference text\. Following the Full\-Duplex\-Bench v1\.5 stop\-latency protocol\[[13](https://arxiv.org/html/2608.13831#bib.bib10)\], we insert the interruption token at the detected onset of the interrupting user speech and measure how rapidly the active assistant output transitions to silence\.

Speech activity is detected independently in the original FDB user audio and the assistant\-only output generated by VoiceChat\-TTS\. Both waveforms are resampled to 16 kHz and processed using the official FDBevaluation/get\_timing\.pyimplementation, which uses Silero VAD\. Following the official configuration, adjacent user and assistant speech segments separated by less than 0\.6 s and 0\.5 s, respectively, are merged\. We identify the interrupting user segment as the merged user segment with the greatest temporal overlap with the interruption interval provided in the FDB metadata\. The onset of this segment defines the interruption time\. Silero VAD detects active assistant speech at interruption onset in all 200 controlled examples\.

We report three timing\-based metrics\. Interruption Obey Rate at 320 ms \(IOR@320ms\), is the percentage of examples in which the assistant speech segment active at interruption onset ends within 320 ms\. FDB\-v1\.5\-compatible Stop Latency is computed from the user–assistant overlap intervals returned by the officiallatency\_stop\_list; the table reports the mean after pooling these intervals across examples\. Finally, Leakage@1s is the cumulative duration of detected assistant speech during the first second after interruption onset, averaged across examples\.

We additionally report After\-Interruption Character Error Rate \(AI\-CER\) to evaluate whether the decoder can recover from an interruption and synthesize the subsequent assistant turn without resetting its KV cache\. For each example, we isolate only the assistant\-output segment corresponding to the controlled turn following the interruption, beginning at its known BOS timestamp and ending at the turn boundary\. We transcribe this segment, compute CER against the corresponding reference text, and average the resulting CER values across examples\. Audio between interruption onset and the subsequent BOS—including any residual overlap, leakage, or silence—is excluded\. AI\-CER therefore measures the intelligibility and content preservation of the subsequent assistant turn, complementing the stop\-latency and leakage metrics rather than incorporating the interrupted portion of the output\. The evaluation contains all 200 examples from the FDB User Interruption subset, and every example is processed for each reported metric\.

We compare two inference settings\. When*Force Silence*is false, the model must transition to silence solely through its learned response to the interruption token\. When*Force Silence*is true, inference injects a fixed silence audio\-token frame at positions aligned with the interruption token, deterministically steering the generated speech stream toward silence\. To obtain this frame, we encode a prolonged segment of pure silence with the codec and select the most frequently occurring 31\-token codec frame\.

Table 3:Effect of deterministic silence forcing on the FDB\-timed controlled\-interruption benchmark\. Timing and leakage metrics are computed using the FDB v1\.5 Silero VAD pipeline\. AI\-CER is computed only on the subsequent assistant turn and excludes the interruption\-overlap interval\. Values are means over ten stochastic runs; 95% confidence intervals are omitted for compactness\.Table[3](https://arxiv.org/html/2608.13831#S3.T3)shows that VoiceChat\-TTS learns a strong response to the interruption token even without deterministic enforcement, achieving an IOR@320ms of 96\.8%\. Enabling*Force Silence*increases IOR@320ms to 100\.0%, reduces mean FDB\-v1\.5\-compatible Stop Latency from 228\.3 ms to 89\.9 ms, and reduces Leakage@1s from 169\.1 ms to 55\.8 ms\. AI\-CER also decreases from 0\.255% to 0\.095%\. Because AI\-CER excludes all audio preceding the subsequent BOS, this result indicates that deterministic silence forcing does not impair the model’s ability to recover and generate the next assistant turn without resetting its KV cache\. Overall, the results show that the learned interruption behavior is already reliable, while deterministic silence forcing provides stricter interruption compliance and substantially reduces residual speech after interruption\.

### 3\.6Latency

We evaluate streaming latency using inter\-token latency \(ITL\), defined as the wall\-clock time required to produce one frame of acoustic tokens after receiving a streaming text update\. VoiceChat\-TTS operates at 12\.5 Hz, so each acoustic\-token frame corresponds to 80 ms of audio\. Therefore, an ITL below 80 ms indicates that the acoustic\-token predictor runs faster than real time\.

For comparison, we use Qwen3\-TTS\-12Hz, which also operates at 12\.5 Hz and is designed for low\-latency streaming synthesis\[[6](https://arxiv.org/html/2608.13831#bib.bib12)\]\. The Qwen3\-TTS technical report provides latency numbers for its 12 Hz models, including first\-packet latency, tokenizer decoding time, and steady\-state LM time per packet\. However, those measurements are reported using the authors’ internal vLLM engine on a “single typical computational resource”, without specifying a directly comparable GPU configuration\. To obtain a more controlled comparison, we remeasure Qwen3\-TTS\-12Hz on the same RTX A6000 setup used for VoiceChat\-TTS and report latency under the same measurement protocol\. Both models are optimized using the vLLM\-Omni framework\.

The final VoiceChat\-TTS model comprises 977M parameters in total, including a 778M\-parameter Gemma 3\-based streaming TTS module\[[19](https://arxiv.org/html/2608.13831#bib.bib19)\]and a 199M\-parameter codec model\. In contrast, the Qwen3\-TTS\-12Hz model used in this comparison is the 1\.7B variant\[[6](https://arxiv.org/html/2608.13831#bib.bib12)\]\. Thus, VoiceChat\-TTS is smaller while targeting the same 12\.5 Hz streaming regime\.

Table[4](https://arxiv.org/html/2608.13831#S3.T4)reports latency at concurrency levels 1 and 4\. We separately measure acoustic\-token ITL and codec decoding time\. Acoustic\-token ITL measures the time required by the neural decoder to produce the next frame of acoustic tokens\. Codec latency measures the time required to decode one acoustic\-token frame into waveform audio, corresponding to 80 ms of speech\. The total next\-frame latency is computed as the sum of acoustic\-token ITL and codec decoding latency\.

Both codec decoders are optimized for persistent streaming inference\. For the Qwen3\-TTS codec, we retain the transformer KV state and causal convolution histories, cache transposed\-convolution overlap, linearize streaming upsampling operations, and fuse Snake activations using native CUDA kernels with reusable FP16 buffers\. For the VoiceChat\-TTS codec, we use causal convolution caching, fuse the 31 codebook lookups into a single bit\-exact CUDA kernel, replace channel\-wise normalization sequences with native FP32 kernels, and cache the fixed inverse\-STFT constants\.

Table 4:Streaming latency comparison on an RTX A6000 GPU\. ITL denotes the time to produce one frame of acoustic tokens at 12\.5 Hz, corresponding to 80 ms of audio\. Codec denotes the measured time to decode one acoustic\-token frame into waveform audio\.As shown in Table[4](https://arxiv.org/html/2608.13831#S3.T4), VoiceChat\-TTS achieves lower latency than Qwen3\-TTS\-12Hz under the same RTX A6000 measurement setup\. At concurrency 1, VoiceChat\-TTS reduces acoustic\-token ITL from15\.4415\.44ms to7\.167\.16ms, corresponding to a2\.1×2\.1\\timesspeedup\. Its optimized codec is also faster, reducing decoding time from4\.904\.90ms to2\.462\.46ms\. Overall, VoiceChat\-TTS reduces next\-frame latency from20\.3420\.34ms to9\.629\.62ms, a2\.1×2\.1\\timesimprovement\.

At concurrency 4, VoiceChat\-TTS remains faster, reducing acoustic\-token ITL from17\.0917\.09ms to12\.3512\.35ms and codec latency from4\.994\.99ms to2\.892\.89ms\. This reduces total next\-frame latency from22\.0822\.08ms to15\.2415\.24ms\. All measured latencies are well below the 80 ms duration represented by one generated acoustic\-token frame, indicating that both systems can run faster than real time\. However, VoiceChat\-TTS provides a larger latency margin for serving overheads such as batching, scheduling, network transport, playback buffering, and asynchronous codec execution, while using a smaller model\.

## 4Conclusions, Limitations, and Future Work

In this paper, we introduced VoiceChat\-TTS, a low\-latency, continuous, and streamable text\-to\-speech architecture designed specifically for interactive agents\. Driven directly by LLM text\-token streams, VoiceChat\-TTS supports explicit mid\-utterance interruptions through control tokens and generates silence when no textual input is available\. The model enables always\-on, responsive speech generation while preserving modularity and competitive speech quality\. Furthermore, it can be integrated into both duplex interaction frameworks and speech\-to\-speech pipelines\. Our evaluations demonstrate competitive intelligibility and predicted overall speech quality relative to strong offline and streaming baselines, together with effective interruption handling and low streaming latency\. VoiceChat\-TTS already serves as the speech decoder in the open\-source NVIDIA Nemotron VoiceChat\-11B duplex S2S model, demonstrating that the proposed modular decoder can be integrated into a complete interactive speech system and support low\-latency duplex interaction\.

Current limitations include the absence of user\-audio conditioning, which prevents the model from dynamically adapting its prosody to acoustic cues from the user, and the lack of controlled component\-wise ablations\. The proposed architectural changes were designed to address complementary limitations of the underlying streaming decoder, and preliminary experiments during model development indicated that their gains were cumulative\. However, systematically quantifying the contribution of each component remains an important direction for future work\. Because continuous duplex TTS is an emerging setting without standardized TTS\-specific benchmarks, we also plan to develop broader evaluation protocols for long\-horizon continuous generation, direct measurement of speech leakage during intended silence, and recovery across repeated interruptions\.

## References

- \[1\]R\. H\. Bai, Z\. Gu, T\. Likhomanenko, and N\. Jaitly\(2025\)Speakstream: streaming text\-to\-speech with interleaved data\.arXiv preprint arXiv:2505\.19206\.Cited by:[§1](https://arxiv.org/html/2608.13831#S1.p4.1)\.
- \[2\]E\. Bakhturina, V\. Lavrukhin, B\. Ginsburg, and Y\. Zhang\(2021\)Hi\-Fi Multi\-Speaker English TTS Dataset\.InINTERSPEECH,Cited by:[§3\.1](https://arxiv.org/html/2608.13831#S3.SS1.p1.1)\.
- \[3\]E\. Casanova, C\. Chen, K\. Hu, A\. Pasad, E\. Rastorgueva, S\. Lakshmi Narasimhan, S\. Deng, E\. Hosseini Asl, P\. Zelasko, V\. Mendelev, S\. Ghosh, Y\. Peng, Z\. Chen, J\. Li, J\. Balam, V\. Lavrukhin, and B\. Ginsburg\(2025\)Open full\-duplex voice agent with speech\-to\-speech language model\.In2025 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),Cited by:[§1](https://arxiv.org/html/2608.13831#S1.p3.1),[§3\.3](https://arxiv.org/html/2608.13831#S3.SS3.p2.1)\.
- \[4\]A\. Défossez, L\. Mazaré, M\. Orsini, A\. Royer, P\. Pérez, H\. Jégou, E\. Grave, and N\. Zeghidour\(2024\)Moshi: a speech\-text foundation model for real\-time dialogue\.arXiv preprint arXiv:2410\.00037\.Cited by:[§1](https://arxiv.org/html/2608.13831#S1.p1.1),[§2](https://arxiv.org/html/2608.13831#S2.p2.1)\.
- \[5\]A\. Goel, S\. Ghosh, J\. Kim, S\. Kumar, Z\. Kong, S\. Lee, C\. H\. Yang, R\. Duraiswami, D\. Manocha, R\. Valle,et al\.\(2025\)Audio flamingo 3: advancing audio intelligence with fully open large audio language models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.13831#S1.p4.1),[§2](https://arxiv.org/html/2608.13831#S2.p1.1),[§2](https://arxiv.org/html/2608.13831#S2.p2.1),[§2](https://arxiv.org/html/2608.13831#S2.p3.1),[§2](https://arxiv.org/html/2608.13831#S2.p5.1),[§3\.4](https://arxiv.org/html/2608.13831#S3.SS4.p5.1),[Table 1](https://arxiv.org/html/2608.13831#S3.T1.2.1.5.1)\.
- \[6\]H\. Hu, X\. Zhu, T\. He, D\. Guo, B\. Zhang, X\. Wang, Z\. Guo, Z\. Jiang, H\. Hao, Z\. Guo,et al\.\(2026\)Qwen3\-tts technical report\.arXiv preprint arXiv:2601\.15621\.Cited by:[§1](https://arxiv.org/html/2608.13831#S1.p4.1),[§3\.6](https://arxiv.org/html/2608.13831#S3.SS6.p2.1),[§3\.6](https://arxiv.org/html/2608.13831#S3.SS6.p3.1),[Table 1](https://arxiv.org/html/2608.13831#S3.T1.2.1.6.1),[Table 4](https://arxiv.org/html/2608.13831#S3.T4.2.1.3.1),[Table 4](https://arxiv.org/html/2608.13831#S3.T4.2.1.5.1)\.
- \[7\]K\. Hu, E\. Hosseini\-Asl, C\. Chen, E\. Casanova, S\. Ghosh, P\. Żelasko, Z\. Chen, J\. Li, J\. Balam, and B\. Ginsburg\(2025\)SALM\-duplex: efficient and direct duplex modeling for speech\-to\-speech language model\.arXiv preprint arXiv:2505\.15670\.Cited by:[§1](https://arxiv.org/html/2608.13831#S1.p1.1),[§2](https://arxiv.org/html/2608.13831#S2.p2.1),[§3\.3](https://arxiv.org/html/2608.13831#S3.SS3.p2.1)\.
- \[8\]S\. S\. Hussain, P\. Neekhara, X\. Yang, E\. Casanova, S\. Ghosh, R\. Fejgin, M\. T\. Desta, R\. Valle, and J\. Li\(2025\)Koel\-tts: enhancing llm based speech generation with preference alignment and classifier free guidance\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 21230–21245\.Cited by:[§3\.2](https://arxiv.org/html/2608.13831#S3.SS2.p1.1),[§3\.4](https://arxiv.org/html/2608.13831#S3.SS4.p1.1)\.
- \[9\]J\. Kim, T\. Moon, K\. Lee, and J\. Cho\(2025\)Efficient generative modeling with residual vector quantization\-based tokens\.InInternational Conference on Machine Learning,pp\. 30609–30630\.Cited by:[§2](https://arxiv.org/html/2608.13831#S2.p5.1)\.
- \[10\]N\. R\. Koluguri, T\. Park, and B\. Ginsburg\(2022\)Titanet: neural model for speaker representation with 1d depth\-wise separable convolutions and global context\.InICASSP,External Links:[Link](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/titanet_small)Cited by:[§3\.4](https://arxiv.org/html/2608.13831#S3.SS4.p1.1)\.
- \[11\]A\. Kumar, K\. Tan, Z\. Ni, P\. Manocha, X\. Zhang, E\. Henderson, and B\. Xu\(2023\)Torchaudio\-squim: reference\-less speech quality and intelligibility measures in torchaudio\.InICASSP 2023\-2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[§3\.4](https://arxiv.org/html/2608.13831#S3.SS4.p1.1)\.
- \[12\]R\. Langman, X\. Yang, P\. Neekhara, S\. Hussain, E\. Casanova, E\. Bakhturina, and J\. Li\(2025\)Hifitts\-2: a large\-scale high bandwidth speech dataset\.arXiv preprint arXiv:2506\.04152\.Cited by:[§3\.1](https://arxiv.org/html/2608.13831#S3.SS1.p1.1)\.
- \[13\]G\. Lin, S\. S\. Kuan, Q\. Wang, J\. Lian, T\. Li, S\. Watanabe, and H\. Lee\(2025\)Full\-duplex\-bench v1\. 5: evaluating overlap handling for full\-duplex speech models\.arXiv preprint arXiv:2507\.23159\.Cited by:[§3\.4](https://arxiv.org/html/2608.13831#S3.SS4.p8.1),[§3\.5](https://arxiv.org/html/2608.13831#S3.SS5.p1.1)\.
- \[14\]X\. Lu, W\. Xu, H\. Wang, H\. Zhou, H\. Zhao, C\. Zhu, T\. Zhao, and M\. Yang\(2025\)Duplexmamba: enhancing real\-time speech conversations with duplex and streaming capabilities\.InCCF International Conference on Natural Language Processing and Chinese Computing,pp\. 62–74\.Cited by:[§1](https://arxiv.org/html/2608.13831#S1.p1.1)\.
- \[15\]NVIDIA\(2025\)NVIDIA nemotron nano 2: an accurate and efficient hybrid mamba\-transformer reasoning model\.External Links:2508\.14444,[Link](https://arxiv.org/abs/2508.14444)Cited by:[§2](https://arxiv.org/html/2608.13831#S2.p3.1)\.
- \[16\]Resemble AI\(2025\)Chatterbox\-TTS\.Note:[https://github\.com/resemble\-ai/chatterbox](https://github.com/resemble-ai/chatterbox)GitHub repositoryCited by:[Table 1](https://arxiv.org/html/2608.13831#S3.T1.2.1.4.1)\.
- \[17\]resemble\-ai\(2025\)Chatterbox: sota open\-source tts\.GitHub\.Note:[https://github\.com/resemble\-ai/chatterbox](https://github.com/resemble-ai/chatterbox)GitHub repositoryCited by:[§3\.2](https://arxiv.org/html/2608.13831#S3.SS2.p1.1)\.
- \[18\]R\. Roy, J\. Raiman, S\. Lee, T\. Ene, R\. Kirby, S\. Kim, J\. Kim, and B\. Catanzaro\(2026\)PersonaPlex: voice and role control for full duplex conversational speech models\.arXiv preprint arXiv:2602\.06053\.Cited by:[§1](https://arxiv.org/html/2608.13831#S1.p1.1),[§3\.4](https://arxiv.org/html/2608.13831#S3.SS4.p8.1),[Table 2](https://arxiv.org/html/2608.13831#S3.T2.2.1.2.1)\.
- \[19\]G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Feng, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. György, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, N\. Sachdeva, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stanczyk, P\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, K\. Black, N\. Babar, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, Dmitry, Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. Hussenot\(2025\)Gemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§2](https://arxiv.org/html/2608.13831#S2.p8.1),[§3\.6](https://arxiv.org/html/2608.13831#S3.SS6.p3.1)\.
- \[20\]N\. Torgashov, G\. E\. Henter, and G\. Skantze\(2025\)VoXtream: full\-stream text\-to\-speech with extremely low latency\.arXiv preprint arXiv:2509\.15969\.Cited by:[§1](https://arxiv.org/html/2608.13831#S1.p4.1)\.
- \[21\]H\. Xu, F\. Jia, S\. Majumdar, H\. Huang, S\. Watanabe, and B\. Ginsburg\(2023\)Efficient sequence transduction by jointly predicting tokens and durations\.InInternational Conference on Machine Learning,External Links:[Link](https://huggingface.co/nvidia/parakeet-tdt-1.1b)Cited by:[§3\.4](https://arxiv.org/html/2608.13831#S3.SS4.p1.1)\.
- \[22\]J\. Xu, Z\. Guo, H\. Hu, Y\. Chu, X\. Wang, J\. He, Y\. Wang, X\. Shi, T\. He, X\. Zhu,et al\.\(2025\)Qwen3\-omni technical report\.arXiv preprint arXiv:2509\.17765\.Cited by:[§1](https://arxiv.org/html/2608.13831#S1.p3.1)\.
- \[23\]P\. Żelasko, D\. Povey, J\. Trmal, S\. Khudanpur,et al\.\(2021\)Lhotse: a speech data representation library for the modern deep learning ecosystem\.arXiv preprint arXiv:2110\.12561\.Cited by:[§3\.3](https://arxiv.org/html/2608.13831#S3.SS3.p2.1)\.
- \[24\]H\. Zen, V\. Dang, R\. Clark, Y\. Zhang, R\. J\. Weiss, Y\. Jia, Z\. Chen, and Y\. Wu\(2019\)LibriTTS: a corpus derived from librispeech for text\-to\-speech\.INTERSPEECH\.Cited by:[§3\.1](https://arxiv.org/html/2608.13831#S3.SS1.p1.1),[§3\.4](https://arxiv.org/html/2608.13831#S3.SS4.p3.1)\.