Paper page - RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

Source: https://huggingface.co/papers/2610.29434

Abstract

RubatoCodec uses adaptive event-based coding with fixed utterance budgets to improve low-rate speech reconstruction and streaming text-to-speech generation.

Neural speech codecs provide discrete tokens for speech generation. Lower frame rates shorten token sequences, but each token must then carry more linguistic and acoustic information.Variable-frame-rate codecs address this trade-off by adapting token allocation to speech content, but threshold-based allocation lets content determine the token count, so the sequence length a speech generator must produce varies across utterances and datasets at a fixed setting. We present RubatoCodec, a variable-frame-rate speech codec built onX-Codec-2with an exact utterance-level event budget andstreaming decoding. ItsAdaptive Firingmechanism uses changes in codec features and waveform spectra to divide speech into variable-duration events without a learned boundary predictor.Discrete codesand an explicit duration represent each event, allowing a speech generator to predict and decode one event at a time. Controlled experiments show that adaptive allocation outperforms fixed-rate coding inreconstruction fidelityand intelligibility at low event rates. A model trained on large-scalemultilingual speechachieves competitive reconstruction quality across English, Japanese, and Mandarin, with the best intelligibility preservation among the compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates. In a matchedtext-to-speechcomparison, the adaptive representation yields lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity. The speech generator also learns the codec’s utterance-level event budget without an explicit generation-time rate constraint. These results demonstrate that budgeted adaptive speech representations support both high-quality reconstruction and streamingtext-to-speechgeneration. Code and weights are available at https://github.com/third-intelligence/RubatoCodec and https://huggingface.co/third-intelligence/RubatoCodec.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2610\.29434

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### third-intelligence/RubatoCodec Audio-to-Audio• 0.2B• Updatedabout 14 hours ago • 25

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.29434 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.29434 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.