重新思考基于长度的训练：语音标记语言模型中的批次组成与损失归一化

# BATCH COMPOSITIONAND LOSS NORMALIZATION IN SPEECH TOKEN LANGUAGE MODELS
Source: [https://arxiv.org/html/2609.25890](https://arxiv.org/html/2609.25890)
## RETHINKING LENGTH\-BASED TRAINING: BATCH COMPOSITION AND LOSS NORMALIZATION IN SPEECH TOKEN LANGUAGE MODELS

Runwu ShiWeiqiao ShanJiale LuoYujin WangYifei WuChunxiang Jin

###### Abstract

Short\-to\-long training is a simple curriculum for speech models, but its gains can be difficult to interpret\. In speech token language models, length\-based training can change the shuffle policy, batch composition, token retention, and token weights under batch\-mean loss\. We disentangle these factors through matched comparisons\. In the tested settings, short\-to\-long ordering shows no independent benefit when batch composition and token exposure are fixed\. First\-epoch grouping lowers perplexity for Mimi under batch\-mean loss, but this gain is not observed under token\-balanced loss\. The cross\-tokenizer results are consistent with a link between chunk\-length variation and token weighting\. This work provides a systematic analysis protocol for studying length\-based training in variable\-length speech models\.

###### Index Terms:

speech token language models, curriculum learning, data ordering, batching, loss normalization

††address:1Beijing Institute of Technology, Zhuhai;2Institute of Science Tokyo;3Northeastern University
4Sichuan University;5Wuhan University;6Ant Group## 1Introduction

Autoregressive modeling of discrete speech tokens has received increasing attention in speech generation and speech language modeling\[[9](https://arxiv.org/html/2609.25890#bib.bib1),[8](https://arxiv.org/html/2609.25890#bib.bib2),[6](https://arxiv.org/html/2609.25890#bib.bib3)\]\. The length of a speech\-token sequence depends on both utterance duration and tokenizer rate\. This affects batch construction and the number of valid tokens in each update\. SortaGrad presents shorter utterances first in the first epoch and then returns to random minibatch order\[[2](https://arxiv.org/html/2609.25890#bib.bib14)\]\. Recent work studies curriculum schedules with token budgets\[[14](https://arxiv.org/html/2609.25890#bib.bib4)\], difficulty\-based token\-loss weighting\[[10](https://arxiv.org/html/2609.25890#bib.bib5)\], and within\-batch diversity\[[3](https://arxiv.org/html/2609.25890#bib.bib6)\]\. Sequence length also matters in speech data selection\[[12](https://arxiv.org/html/2609.25890#bib.bib7)\]\.

Document packing also changes training: Best\-fit Packing reduces document fragmentation while preserving training efficiency\[[7](https://arxiv.org/html/2609.25890#bib.bib12)\]\. Here, length sorting changes batch membership and, under min\-length truncation, which targets are retained\. Per\-batch mean loss also assigns larger coefficients to token losses in batches with fewer valid targets\. A sorted\-versus\-random comparison can therefore mix presentation order with changes in target exposure and loss normalization\.

Figure 1:Overview of the factors changed by length\-based training\. \(a\) Training policies across epochs\. Composition and Reshuf use the same first\-epoch length grouping but differ in batch order; both return to epoch\-wise shuffling afterward, while Persistent S→\\rightarrowL repeats grouping and ascending order\. \(b\) For an example variable\-length batch, min\-length batching discards valid tokens beyond the shortest sequence, whereas padded batching keeps all valid tokens and masks padding in the loss\.We ask whether the gains from short\-to\-long training come from presentation order or from the changes in batching that accompany length sorting\. Our fixed\-batch comparisons isolate presentation order, and a normalization intervention tests the remaining first\-epoch grouping effect\. The study uses an 87M\-parameter Transformer on LibriSpeech train\-clean\-100, with Mimi for the main analysis and EnCodec and SpeechTokenizer for cross\-tokenizer comparisons\.

With padded evaluation and fixed batch membership, sorted orders do not improve perplexity in the tested settings\. First\-epoch grouping lowers PPL for Mimi under batch\-mean loss, but this gain is not observed under token\-balanced loss\. The normalization change has little effect on Shuffle but raises PPL for Composition\. The controlled comparisons separate presentation order from batch construction and loss normalization\.

## 2Method

### 2\.1Training factors

Let𝐱i=\(xi,1,…,xi,ci\)\\mathbf\{x\}\_\{i\}=\(x\_\{i,1\},\\ldots,x\_\{i,c\_\{i\}\}\)be a stored token chunk\. Its next\-token target length isℓi=ci−1≤M\\ell\_\{i\}=c\_\{i\}\-1\\leq M, withM=256M=256\. Length\-based training changes the factors below\.

Shuffle policy\.A random permutation can be sampled once and reused, or sampled again at every epoch\. We call these settings*fixed random order*and*epoch\-wise shuffle*\. They match the SingleShuffle/RandomShuffle distinction in optimization work\[[1](https://arxiv.org/html/2609.25890#bib.bib13)\]\. Epoch\-wise shuffling also changes which sequences share a batch across epochs\.

Batch composition and order\.Length sorting affects both which sequences appear in the same batch and the order in which batches are processed\. We distinguish these two effects\.*Length grouping*forms batches from sequences with similar lengths, while*short\-to\-long order*presents the resulting batches in ascending length order\. The batch order can instead be randomized while preserving the same length\-grouped composition\.

Token retention and capacity use\.For a processed batchℬ\\mathcal\{B\}, letTℬ=∑iℓiT\_\{\\mathcal\{B\}\}=\\sum\_\{i\}\\ell\_\{i\}before truncation\. Min\-length batching retainsKℬ=\|ℬ\|​mini​ℓiK\_\{\\mathcal\{B\}\}=\|\\mathcal\{B\}\|\\min\_\{i\}\\ell\_\{i\}targets; padded batching retains allTℬT\_\{\\mathcal\{B\}\}targets\. Across batches,

Rret=∑ℬKℬ∑ℬTℬ,R\_\{\\rm ret\}=\\frac\{\\sum\_\{\\mathcal\{B\}\}K\_\{\\mathcal\{B\}\}\}\{\\sum\_\{\\mathcal\{B\}\}T\_\{\\mathcal\{B\}\}\},\(1\)Ucap=∑ℬKℬM​∑ℬ\|ℬ\|\.U\_\{\\rm cap\}=\\frac\{\\sum\_\{\\mathcal\{B\}\}K\_\{\\mathcal\{B\}\}\}\{M\\sum\_\{\\mathcal\{B\}\}\|\\mathcal\{B\}\|\}\.\(2\)RretR\_\{\\rm ret\}is true target retention;UcapU\_\{\\rm cap\}measures use of fixed maximum capacity\. Near\-equal short chunks can have high retention but low capacity use\. Table[2](https://arxiv.org/html/2609.25890#S4.T2)reportsUcapU\_\{\\rm cap\}\.

### 2\.2Training configurations

The factors above are separated through the training configurations in Table[1](https://arxiv.org/html/2609.25890#S2.T1)\. Here, “grouped” denotes length\-homogeneous batches\.

*Composition*and*Reshuf*use the same length\-grouped batches in the first epoch\. Composition presents these batches in random order; Reshuf presents them from short to long\. Both return to epoch\-wise shuffling afterward\. Their comparison isolates first\-epoch batch order while keeping batch composition fixed\. Persistent short\-to\-long training repeats length grouping and ascending order at every epoch\.

Table 1:Training configurations used in the factorization\.The fixed\-batch comparison reuses batch membership and padding masks\.*Batch shuffle*randomizes the batch list at every epoch\.*First\-epoch S→\\rightarrowL*sorts batches by mean length in epoch 0 and shuffles them afterward\. Two persistent conditions repeat ascending or descending batch order\. The targets within each batch are unchanged\.

To examine persistent length grouping separately, we also compare two grouped settings\. In the*static\-grouped*setting, the same length\-grouped batches are reused across epochs\. In the*dynamic\-grouped*setting, length\-grouped batches are reconstructed at each epoch while their global order remains random\. Comparing the two tests whether changes in batch membership, rather than length grouping itself, account for their training behavior\.

### 2\.3Loss normalization

For a batchℬ\\mathcal\{B\}, let𝒱ℬ\\mathcal\{V\}\_\{\\mathcal\{B\}\}denote its set of valid target positions andNℬ=\|𝒱ℬ\|N\_\{\\mathcal\{B\}\}=\|\\mathcal\{V\}\_\{\\mathcal\{B\}\}\|\. The standard training objective used in our initial experiments averages token\-level cross entropy within each batch:

ℒmean\(ℬ\)=−1Nℬ∑\(i,t\)∈𝒱ℬlogpθ\(xi,t∣xi,<t\)\.\\mathcal\{L\}\_\{\\mathrm\{mean\}\}\(\\mathcal\{B\}\)=\-\\frac\{1\}\{N\_\{\\mathcal\{B\}\}\}\\sum\_\{\(i,t\)\\in\\mathcal\{V\}\_\{\\mathcal\{B\}\}\}\\log p\_\{\\theta\}\(x\_\{i,t\}\\mid x\_\{i,<t\}\)\.\(3\)
Each valid token loss has coefficient1/Nℬ1/N\_\{\\mathcal\{B\}\}\. Batches with fewer valid targets assign larger coefficients to individual token losses\.

We use*token\-balanced*loss as a diagnostic intervention,

ℒbal\(ℬ\)=−1Z∑\(i,t\)∈𝒱ℬlogpθ\(xi,t∣xi,<t\)\.\\mathcal\{L\}\_\{\\rm bal\}\(\\mathcal\{B\}\)=\-\\frac\{1\}\{Z\}\\sum\_\{\(i,t\)\\in\\mathcal\{V\}\_\{\\mathcal\{B\}\}\}\\log p\_\{\\theta\}\(x\_\{i,t\}\\mid x\_\{i,<t\}\)\.\(4\)The fixed constantZZis the mean*stored*token count in the reference length\-grouped batches, before next\-token shifting\. It is shared by the compared settings\. Every valid target loss has coefficient1/Z1/Z\. For a fixed batch and model state,ℒbal=\(Nℬ/Z\)​ℒmean\\mathcal\{L\}\_\{\\rm bal\}=\(N\_\{\\mathcal\{B\}\}/Z\)\\mathcal\{L\}\_\{\\rm mean\}\. If valid\-target counts are constant, the two losses differ only by a constant scale\. Loss coefficients are not ratios of AdamW parameter updates, which also depend on gradient moment estimates\.

Figure 2:Grouping and ordering comparisons on Mimi\. \(a\) Padded policies with early stopping \(Table[3](https://arxiv.org/html/2609.25890#S4.T3)\)\. \(b\) Fixed batch membership, 12 epochs, and 3,324 updates \(Table[4](https://arxiv.org/html/2609.25890#S4.T4)A\)\. Bars show mean recorded best PPL over eight seeds; error bars show one standard deviation\. Dashed lines mark each panel’s shuffle baseline\.

## 3Experimental Setup

### 3\.1Data and tokenizers

The main experiments use LibriSpeech train\-clean\-100\[[11](https://arxiv.org/html/2609.25890#bib.bib8)\]with a speaker\-disjoint split \(split seed 0\)\. Mimi\[[5](https://arxiv.org/html/2609.25890#bib.bib11)\]provides the main analysis; EnCodec\[[4](https://arxiv.org/html/2609.25890#bib.bib9)\]and SpeechTokenizer\[[13](https://arxiv.org/html/2609.25890#bib.bib10)\]provide comparisons\. We use one stream without concatenating codebooks: Mimi uses codebook 0 with one quantizer, EnCodec its first codebook, and SpeechTokenizer its semantic stream\. Token indices are used directly\. The vocabulary spans zero through the largest observed index; padded training adds a PAD symbol\.

Chunks contain up to 257 stored tokens, yielding up to 256 next\-token targets\. Long utterances use 257\-token windows at stride 256 and omit incomplete tails; short utterances form one chunk\. Main runs omit the trailing group of at most 64 chunks; the fixed\-batch study keeps every complete training batch\. Padded training uses 257 stored positions\. Chunk\-length variation isCV⁡\(c\)=std⁡\(c\)/mean⁡\(c\)\\mathrm\{CV\}\(c\)=\\mathrm\{std\}\(c\)/\\mathrm\{mean\}\(c\)\.

### 3\.2Model and training

Experiments use an 87M\-parameter autoregressive Transformer with 12 layers, 12 attention heads, and hidden dimension 768\. The batch size is 64\. AdamW uses a peak learning rate of3×10−43\\times 10^\{\-4\}, weight decay 0\.01, and a one\-cycle schedule with 5% warmup\. Gradients are clipped at norm 1\. Main runs use a maximum of 12 epochs, with early\-stopping patience three and an improvement threshold of 0\.02 validation PPL\. The fixed\-batch study runs all 12 epochs without early stopping: 3,324 updates in every run\.

The main comparisons pair eight initialization seeds and share optimizer and schedule settings\. They match the maximum budget, not the executed updates: Mimi Shuffle runs take 1,662–1,939 updates, whereas Composition and Reshuf each take 1,939\. All have 277 updates per epoch\. The normalization intervention usesZ=9217\.39Z=9217\.39in both arms, computed before next\-token shifting\. For 64\-chunk reference batches, the mean target count isZ−64=9153\.39Z\-64=9153\.39\.

### 3\.3Evaluation

Validation uses consecutive full batches: 32 chunks in the main comparisons and 64 in the fixed\-batch study\. Padded evaluation retains their non\-padding targets\. Each comparison uses the same validation targets within its study\. Perplexity is computed over the evaluated targets:

PPL=exp\(−1N∑t=1Nlogpθ\(xt∣x<t\)\)\.\\mathrm\{PPL\}=\\exp\\left\(\-\\frac\{1\}\{N\}\\sum\_\{t=1\}^\{N\}\\log p\_\{\\theta\}\(x\_\{t\}\\mid x\_\{<t\}\)\\right\)\.\(5\)We report mean±\\pmstandard deviation of the recorded best PPL, which is updated only when PPL improves by more than 0\.02\. Composition and fixed\-batch analyses use two\-sided pairedtt\-tests\. Table[3](https://arxiv.org/html/2609.25890#S4.T3)reports rawpp\-values\. Holm correction is applied to the two primary decomposition contrasts and, separately, the two persistent sorted\-order comparisons in Table[4](https://arxiv.org/html/2609.25890#S4.T4)A\. For Table[5](https://arxiv.org/html/2609.25890#S4.T5), we computeIs=Δcomp,sbal−Δcomp,smeanI\_\{s\}=\\Delta\_\{\\rm comp,s\}^\{\\rm bal\}\-\\Delta\_\{\\rm comp,s\}^\{\\rm mean\}within each seed and test its mean against zero using a two\-sided one\-samplett\-test \(n=8n=8, seven degrees of freedom\)\. Interaction and loss\-switchpp\-values are unadjusted\.

## 4Results

### 4\.1Length\-based training changes more than order

Figure[2](https://arxiv.org/html/2609.25890#S2.F2)summarizes the padded comparisons on Mimi\. Panel \(a\) varies grouping and presentation order; panel \(b\) keeps batch membership fixed and changes order alone\. Composition reaches 31\.682 PPL, below 31\.857 for epoch\-wise shuffle, whereas Persistent S→\\rightarrowL reaches 32\.082\. With fixed batches, first\-epoch short\-to\-long ordering changes PPL by only 0\.029 \(p=0\.81p=0\.81\)\. The benefit of first\-epoch grouping is therefore distinct from the effect of presenting shorter batches first\.

Table 2:Mimi batch statistics \(batch size 64,M=256M=256\)\. Count statistics use stored tokens before next\-token shifting\. Batch\-max padding is a diagnostic, not the fixed training shape\.Table[2](https://arxiv.org/html/2609.25890#S4.T2)shows that grouping makes chunks within a batch nearly equal in length\. Batch\-max padding is the mean of1−∑ici/\(\|ℬ\|​maxi​ci\)1\-\\sum\_\{i\}c\_\{i\}/\(\|\\mathcal\{B\}\|\\max\_\{i\}c\_\{i\}\)across batches\. These statistics average nine shuffled batchings \(three seeds, three epochs\); grouping is deterministic\.UcapU\_\{\\rm cap\}uses shuffle seed 1, epoch 1\. The increase from 12\.6% to 55\.8% is greater use of the fixed target capacity, not a true\-retention ratio\. Grouping also raises the between\-batch stored\-token\-count CV from 0\.038 to 0\.309, linking batch construction to loss normalization\.

### 4\.2Separating ordering from batch composition

With padded batching, all valid tokens are retained, allowing batch composition and presentation order to be separated\. Table[3](https://arxiv.org/html/2609.25890#S4.T3)compares epoch\-wise shuffle, Composition, Reshuf, and Persistent S→\\rightarrowL under this setting\.

Table 3:Mimi factorization \(n=8n=8\)\. Differences and rawpp\-values are relative to epoch\-wise shuffle\.LetPSP\_\{S\},PCP\_\{C\}, andPRP\_\{R\}denote the PPL of Shuffle, Composition, and Reshuf\. The total Reshuf effect can be written as

PR−PS=\(PC−PS\)⏟Δcomp\+\(PR−PC\)⏟Δorder\.P\_\{R\}\-P\_\{S\}=\\underbrace\{\(P\_\{C\}\-P\_\{S\}\)\}\_\{\\Delta\_\{\\rm comp\}\}\+\\underbrace\{\(P\_\{R\}\-P\_\{C\}\)\}\_\{\\Delta\_\{\\rm order\}\}\.\(6\)For Mimi,Δcomp=−0\.175\\Delta\_\{\\rm comp\}=\-0\.175, whileΔorder=\+0\.120\\Delta\_\{\\rm order\}=\+0\.120\. Composition lowers PPL under batch\-mean loss\. Adding short\-to\-long order to the same grouped batches raises PPL by 0\.120 \(pHolm=6\.70×10−4p\_\{\\rm Holm\}=6\.70\\times 10^\{\-4\}\), removing part of that gain\. Persistent S→\\rightarrowL is also worse than epoch\-wise shuffle\. The observed gain is associated with first\-epoch batch composition, not its short\-to\-long order\.

Table 4:Mimi order and grouping comparisons \(n=8n=8\)\. A: differences from Batch shuffle; Holm correction for the two persistent sorted orders, rawppfor First\-epoch S→\\rightarrowL\. B: raw paired tests\.Table[4](https://arxiv.org/html/2609.25890#S4.T4)separates batch order from membership\. Relative to Batch shuffle, persistent short\-to\-long and long\-to\-short order raise PPL by 0\.654 and 2\.911\. First\-epoch short\-to\-long order changes PPL by only 0\.029 \(p=0\.81p=0\.81\)\. Neither sorted schedule improves over batch shuffling\. Persistent grouping is also worse whether batches are rebuilt or reused; their difference is not significant \(p=0\.57p=0\.57\)\. The observed gain is specific to first\-epoch grouping under batch\-mean loss\.

### 4\.3Loss normalization and the grouping effect

First\-epoch grouping changes batch token counts and hence the coefficient1/Nℬ1/N\_\{\\mathcal\{B\}\}of each valid target loss\. Token\-balanced loss replaces it with fixed1/Z1/Z\. This intervention changes normalization while keeping the batch construction policy unchanged\.

Table 5:Token\-balanced intervention on Mimi \(n=8n=8\)\.Table[5](https://arxiv.org/html/2609.25890#S4.T5)shows that token balancing affects the two settings differently\. Shuffle changes by only−0\.001\-0\.001PPL, whereas Composition increases by 0\.198 PPL\. The resulting interaction is 0\.199 PPL \(p=5\.8×10−5p=5\.8\\times 10^\{\-5\}\)\. The composition effect changes from−0\.175\-0\.175to\+0\.024\+0\.024PPL and is no longer statistically significant \(p=0\.18p=0\.18\)\. Figure[3](https://arxiv.org/html/2609.25890#S4.F3)shows an upward shift for all eight paired seeds\. The interaction supports a role for batch\-dependent normalization in Mimi’s first\-epoch grouping effect\.

Figure 3:Paired composition effect under batch\-mean and token\-balanced losses \(n=8n=8\)\.
### 4\.4Dependence on sequence\-length variation

Table[6](https://arxiv.org/html/2609.25890#S4.T6)reports within\-tokenizer effects from Eq\.[6](https://arxiv.org/html/2609.25890#S4.E6)\. The coefficient diagnostic uses stored lengthscic\_\{i\}, before shifting\. LetAℬ=∑j∈ℬcjA\_\{\\mathcal\{B\}\}=\\sum\_\{j\\in\\mathcal\{B\}\}c\_\{j\}andai=1/Aℬ⁡\(i\)a\_\{i\}=1/A\_\{\\mathcal\{B\}\(i\)\}\. WithS=\{i:ci≤Q25\}S=\\\{i:c\_\{i\}\\leq Q\_\{25\}\\\}andL=\{i:ci≥Q75\}L=\\\{i:c\_\{i\}\\geq Q\_\{75\}\\\}, computed within the grouped arm,

ρpre=\|S\|−1​∑i∈Sai\|L\|−1​∑i∈Lai\.\\rho\_\{\\rm pre\}=\\frac\{\|S\|^\{\-1\}\\sum\_\{i\\in S\}a\_\{i\}\}\{\|L\|^\{\-1\}\\sum\_\{i\\in L\}a\_\{i\}\}\.\(7\)Means are over chunks\. Percentile cutoffs are inclusive, so ties can make the sets overlap\. This is a pre\-shift diagnostic; training uses1/∑j\(cj−1\)1/\\sum\_\{j\}\(c\_\{j\}\-1\)\. Absolute PPL is not compared across tokenizers\.

Table 6:Cross\-tokenizer results \(n=8n=8\)\.ρpre\\rho\_\{\\rm pre\}is the pre\-shift diagnostic in Eq\.[7](https://arxiv.org/html/2609.25890#S4.E7)\.Only Mimi shows a significant first\-epoch composition gain under batch\-mean loss\. Its chunk\-length CV is 0\.309, compared with 0\.080 for SpeechTokenizer and 0\.026 for EnCodec\. The grouped pre\-shift ratio is 2\.87 for Mimi and 1\.00 for EnCodec\. With many full\-length chunks, inclusive percentile groups need not represent separate quarters\. These statistics and the loss intervention support a role for length variation in batch\-dependent normalization\. The protocol tests ordering with fixed batches, then tests the remaining grouping gain by changing the normalization rule\.

## 5Conclusion

In the tested speech\-token language model settings, short\-to\-long ordering does not improve perplexity when batch composition and token exposure are controlled\. First\-epoch length grouping lowers perplexity for Mimi under batch\-mean loss, but this gain is not observed with token\-balanced normalization\. The interaction supports a role for batch\-dependent loss normalization\. Cross\-tokenizer comparisons are consistent with a role of chunk\-length variation\. This work provides a systematic analysis protocol for separating presentation order, token exposure, and loss normalization in length\-based training\.

## References

- \[1\]K\. Ahn, C\. Yun, and S\. Sra\(2020\)SGD with shuffling: optimal rates without component convexity and large epoch requirements\.InAdvances in Neural Information Processing Systems,Vol\.33\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/cb8acb1dc9821bf74e6ca9068032d623-Abstract.html)Cited by:[§2\.1](https://arxiv.org/html/2609.25890#S2.SS1.p2.1)\.
- \[2\]D\. Amodeiet al\.\(2016\)Deep Speech 2: end\-to\-end speech recognition in English and Mandarin\.InProceedings of the 33rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.48,pp\.173–182\.External Links:[Link](https://proceedings.mlr.press/v48/amodei16.html)Cited by:[§1](https://arxiv.org/html/2609.25890#S1.p1.1)\.
- \[3\]Y\. Dai, Y\. Huang, T\. Yang, Y\. Wang, X\. Zhang, W\. Wu, Q\. Zhao, H\. Li, Y\. Gao, K\. Yap, and S\. Li\(2026\)Demystifying data organization for enhanced LLM training\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\.27358–27375\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1262),[Link](https://aclanthology.org/2026.acl-long.1262/)Cited by:[§1](https://arxiv.org/html/2609.25890#S1.p1.1)\.
- \[4\]A\. Défossez, J\. Copet, G\. Synnaeve, and Y\. Adi\(2023\)High fidelity neural audio compression\.Transactions on Machine Learning Research\.External Links:[Link](https://mlanthology.org/tmlr/2023/defossez2023tmlr-high/)Cited by:[§3\.1](https://arxiv.org/html/2609.25890#S3.SS1.p1.1)\.
- \[5\]A\. Défossez, L\. Mazaré, M\. Orsini, A\. Royer, P\. Pérez, H\. Jégou, E\. Grave, and N\. Zeghidour\(2024\)Moshi: a speech\-text foundation model for real\-time dialogue\.arXiv preprint arXiv:2410\.00037\.External Links:[Link](https://arxiv.org/abs/2410.00037)Cited by:[§3\.1](https://arxiv.org/html/2609.25890#S3.SS1.p1.1)\.
- \[6\]L\. Della Libera, F\. Paissan, C\. Subakan, and M\. Ravanelli\(2025\)FocalCodec: low\-bitrate speech coding via focal modulation networks\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\.26926–26951\.External Links:[Document](https://dx.doi.org/10.52202/085713-0801),[Link](https://papers.neurips.cc/paper_files/paper/2025/hash/220e6dbed88faa83bd9f9e8fa359e43d-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.25890#S1.p1.1)\.
- \[7\]H\. Ding, Z\. Wang, G\. Paolini, V\. Kumar, A\. Deoras, D\. Roth, and S\. Soatto\(2024\)Fewer truncations improve language modeling\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\.11030–11048\.External Links:[Link](https://proceedings.mlr.press/v235/ding24f.html)Cited by:[§1](https://arxiv.org/html/2609.25890#S1.p2.1)\.
- \[8\]S\. Ji, M\. Fang, J\. Zuo, Z\. Jiang, D\. Wang, H\. Wang, H\. Huang, and Z\. Zhao\(2025\)Language\-Codec: bridging discrete codec representations and speech language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\.13332–13345\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.654),[Link](https://aclanthology.org/2025.acl-long.654/)Cited by:[§1](https://arxiv.org/html/2609.25890#S1.p1.1)\.
- \[9\]S\. Ji, Z\. Jiang, W\. Wang, Y\. Chen, M\. Fang, J\. Zuo, Q\. Yang, X\. Cheng, Z\. Wang, R\. Li, Z\. Zhang, X\. Yang, R\. Huang, Y\. Jiang, Q\. Chen, S\. Zheng, and Z\. Zhao\(2025\)WavTokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/ea1f5f0878d43ff4fb8bf64ef4a2326c-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.25890#S1.p1.1)\.
- \[10\]Y\. Jia, C\. Zhang, X\. Diao, X\. Yuan, Z\. Ouyang, C\. Ma, and S\. Vosoughi\(2026\)What makes a good curriculum? disentangling the effects of data ordering on LLM mathematical reasoning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\.34472–34488\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1591),[Link](https://aclanthology.org/2026.acl-long.1591/)Cited by:[§1](https://arxiv.org/html/2609.25890#S1.p1.1)\.
- \[11\]V\. Panayotov, G\. Chen, D\. Povey, and S\. Khudanpur\(2015\)LibriSpeech: an ASR corpus based on public domain audio books\.In2015 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\.5206–5210\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP.2015.7178964)Cited by:[§3\.1](https://arxiv.org/html/2609.25890#S3.SS1.p1.1)\.
- \[12\]R\. Whetten, T\. Parcollet, M\. Dinarelli, and Y\. Estève\(2026\)A study of data selection strategies for pre\-training self\-supervised speech models\.In2026 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),External Links:[Document](https://dx.doi.org/10.1109/ICASSP55912.2026.11463465),[Link](https://www.cmsworkshops.com/ICASSP2026/view_paper.php?PaperNum=13120&bare=1)Cited by:[§1](https://arxiv.org/html/2609.25890#S1.p1.1)\.
- \[13\]X\. Zhang, D\. Zhang, S\. Li, Y\. Zhou, and X\. Qiu\(2024\)SpeechTokenizer: unified speech tokenizer for speech language models\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/86d1ab582afb247ccaa84bec4a7e24f7-Abstract-Conference.html)Cited by:[§3\.1](https://arxiv.org/html/2609.25890#S3.SS1.p1.1)\.
- \[14\]Y\. Zhang, A\. Mohamed, H\. Abdine, G\. Shang, and M\. Vazirgiannis\(2026\)Beyond random sampling: efficient language model pretraining via curriculum learning\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\.5776–5794\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.271),[Link](https://aclanthology.org/2026.eacl-long.271/)Cited by:[§1](https://arxiv.org/html/2609.25890#S1.p1.1)\.