Breaking the Curse ofMultilinguality inMany-to-Many Speech-to-Text Translation via a Resource-AwareMixture of Speech Encoders

# Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
Source: [https://arxiv.org/html/2608.04586](https://arxiv.org/html/2608.04586)
###### Abstract

Multimodal large language models \(MLLMs\) have achieved significant success in speech\-to\-text translation \(S2TT\)\. However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high\-resource performance but substantial degradation on low\-resource speech\. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource\-aware Mixture of Speech Encoders \(MoSE\)\. MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder\. A frozen expert preserves high\-resource language capabilities, while a trainable expert adapts to and specializes in medium\- and low\-resource languages\. We further introduce a five\-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2TT data per language for effective alignment\. We conduct extensive experiments on 45 languages, systematically evaluating all45×4445\\times 44translation directions\. Our 4B\-parameter model achieves state\-of\-the\-art performance, outperforming substantially larger baselines\. Empirical analyses show that MoSE improves high\-, medium\-, and low\-resource languages simultaneously, with the largest gains on low\-resource speech, thereby breaking the curse of multilinguality without compromising high\-resource performance\. To support future multilingual S2TT research, we release our code and models\.

Code—https://github\.com/yxduir/MSRT

## Introduction

Speech\-to\-text translation \(S2TT\) aims to translate speech from a source language into text in a target language\. Traditionally, these systems have utilized cascaded pipelines: an Automatic Speech Recognition \(ASR\) component first extracts a transcript from the speech\(Radfordet al\.[2023](https://arxiv.org/html/2608.04586#bib.bib15)\), which a subsequent Machine Translation \(MT\) model then converts into the desired language\(Shanget al\.[2026](https://arxiv.org/html/2608.04586#bib.bib17)\)\. Recently, there has been a growing focus on integrating speech encoders with large language models \(LLMs\) to endow them with multilingual speech capabilities, thereby advancing ASR\(Maet al\.[2026](https://arxiv.org/html/2608.04586#bib.bib7)\)and S2TT tasks\(Duet al\.[2026](https://arxiv.org/html/2608.04586#bib.bib10)\)\.

![Refer to caption](https://arxiv.org/html/2608.04586v1/Figures/mul.png)Figure 1:Multilingual consistency by source language and resource level\. Consistency is the ratio of a source language’s average translation score \(across 44 target directions\) to the English average\. Higher is better, and dashed vertical lines indicate each model’s average within a resource group\.Despite these advances, current MLLMs remain strongly English\-centric in S2TT\. They perform well on English\-to\-X directions but degrade in many\-to\-many translation from non\-English speech\. As shown in Figure[1](https://arxiv.org/html/2608.04586#Sx1.F1), their consistency scores for low\-resource speech inputs fall substantially below those for English, revealing a pronounced cross\-lingual performance gap\. This disparity is not merely a target\-generation issue: changing the source speech language causes substantial quality variation even when the target set and decoder remain unchanged\. Consequently, English\-centric results can overstate multilingual coverage and obscure failures on multilingual speech inputs\.

One important cause of this inconsistency is that most multilingual MLLMs\(Xuet al\.[2025](https://arxiv.org/html/2608.04586#bib.bib25)\)rely on a single shared speech encoder for all input languages\. As language coverage expands, this architecture suffers from the “curse of multilinguality”\(Conneauet al\.[2020](https://arxiv.org/html/2608.04586#bib.bib3)\): limited encoder capacity forces languages with different levels of available resources to compete for the same representation space\. High\-resource languages require preserving pretrained acoustic knowledge, whereas medium\- and low\-resource languages require greater adaptation\. Joint optimization can therefore overwrite established representations or underfit scarce\-language features\.

Inspired by the Mixture\-of\-Experts architectures in LLMs\(Blevinset al\.[2024](https://arxiv.org/html/2608.04586#bib.bib6)\), we propose MSRT, an S2TT MLLM centered on a resource\-awareMixture of Speech Encoders \(MoSE\)\. Two homogeneous Whisper encoders\(Radfordet al\.[2023](https://arxiv.org/html/2608.04586#bib.bib15)\)assume asymmetric roles: a frozen expert handles high\-resource languages, while a trainable expert adapts to medium\- and low\-resource speech\. A source\-language token routes each utterance to one expert, and a target\-language token controls generation\. This design provides specialization at the inference cost of a single encoder\. We further extend the curriculum strategy\(Duet al\.[2025](https://arxiv.org/html/2608.04586#bib.bib1)\)into five stages of increasing difficulty, progressively establishing speech–text alignment and translation\.

We evaluate MSRT on FLEURS across all45×4445\\times 44non\-identity translation directions, with detailed comparisons on representative languages from 11 language families\. MSRT\-4B achieves the highest average COMET score of 83\.3, outperforming both the Gemini\-3\.5\-Flash\-Lite API and the substantially larger MCAT\-27B\(Duet al\.[2026](https://arxiv.org/html/2608.04586#bib.bib10)\)\. Across the complete direction grid, MSRT obtains COMET scores of at least 80 on 1,552 directions \(78\.4%\), compared with 928 for Gemini\. Ablation studies show that MoSE improves performance across high\-, medium\-, and low\-resource languages, with the largest gains on low\-resource languages, while further strengthening high\-resource performance\. These consistent gains across all resource groups demonstrate that MoSE breaks the curse of multilinguality rather than trading high\-resource performance for low\-resource improvements\. Experiments on CoVoST\-2 further demonstrate robust cross\-dataset generalization\.

Our contributions are summarized as follows:

- •We propose MoSE, a resource\-aware mixture\-of\-speech\-encoder architecture that breaks the curse of multilinguality by preserving high\-resource knowledge while enabling targeted adaptation for medium\- and low\-resource languages\.
- •We introduce MSRT\-4B, a compact 4B\-parameter model that supports all translation directions among 45 languages using only 10 hours of paired S2TT data per language\. We open\-source its code and model to facilitate reproducible research\.
- •We comprehensively evaluate all 1,980 translation directions across 45 languages, demonstrating broad and reliable many\-to\-many translation across diverse language families and resource levels, especially for low\-resource speech in realistic multilingual settings\.

## Related Work

#### Speech\-to\-Text Translation\.

Conventional speech\-to\-text translation \(S2TT\) cascades automatic speech recognition and machine translation\. Although this design can reuse strong task\-specific models, recognition errors propagate downstream and two\-stage decoding increases latency\. End\-to\-end S2TT instead learns a direct mapping from source speech to target text\. SeamlessM4T integrates multilingual speech recognition and translation with text translation in a single model, demonstrating that one architecture can support diverse speech–text tasks and languages\(Barraultet al\.[2023](https://arxiv.org/html/2608.04586#bib.bib8)\)\. Its success, however, still depends heavily on paired supervision\. ZeroSwot reduces this dependence by connecting an ASR\-trained speech encoder to a massively multilingual translation model through CTC compression and optimal\-transport alignment, enabling direct S2TT without paired translation data\(Tsiamaset al\.[2024](https://arxiv.org/html/2608.04586#bib.bib2)\)\. These studies establish speech–text alignment as a central challenge in end\-to\-end translation\.

#### Many\-to\-Many Speech\-to\-Text Translation\.

Many\-to\-many S2TT expands both source\- and target\-language coverage, requiring one model to preserve language\-specific acoustic cues while controlling multilingual generation\. SeamlessM4T demonstrates broad multilingual coverage through shared recognition and translation components\(Barraultet al\.[2023](https://arxiv.org/html/2608.04586#bib.bib8)\)\. General\-purpose multimodal LLMs such as Qwen3\-Omni further show that pretrained language backbones can support multilingual speech understanding and translation\(Xuet al\.[2025](https://arxiv.org/html/2608.04586#bib.bib25)\)\. Specialized MLLMs more directly exploit the translation knowledge of LLMs: LLM\-SRT transfers it to many\-to\-many S2TT through curriculum learning\(Duet al\.[2025](https://arxiv.org/html/2608.04586#bib.bib1)\), while MCAT scales MLLM\-based translation to 70 languages\(Duet al\.[2026](https://arxiv.org/html/2608.04586#bib.bib10)\)\. Nevertheless, increasing language coverage does not eliminate resource imbalance or interference between languages\. This limitation motivates speech\-side mechanisms that allocate adaptation capacity according to the data resources and acoustic variation of different languages\.

#### Mixture of Experts\.

MoE has been explored in multimodal and speech models from several perspectives\. Uni\-MoE activates modality\-specific experts to reduce cross\-modal interference in a unified MLLM\(Liet al\.[2025](https://arxiv.org/html/2608.04586#bib.bib18)\), whereas SimulMega uses expert routers as policies for balancing latency and quality in simultaneous translation\(Leet al\.[2025](https://arxiv.org/html/2608.04586#bib.bib19)\)\. PaM selects heterogeneous audio encoders from task prompts to serve different audio tasks\(Shanet al\.[2025](https://arxiv.org/html/2608.04586#bib.bib20)\), while DialectMoE learns dynamic expert routing for multi\-dialect ASR\(Zhouet al\.[2024](https://arxiv.org/html/2608.04586#bib.bib22)\)\. In contrast, this work explores MoE specifically for multilingual S2TT and focuses on interference induced by imbalanced language resources\. MSRT routes source languages by resource level between two homogeneous Whisper encoders with asymmetric roles: a frozen expert preserves high\-resource representations, and a trainable expert specializes in medium\- and low\-resource speech\. Thus, its explicit language\-level routing investigates multilingual expert specialization without token\-level load balancing or task\-dependent expert selection\.

![Refer to caption](https://arxiv.org/html/2608.04586v1/x1.png)Figure 2:Overview of the MSRT framework\. Based on the source\-language token in the prompt, MoSE routes high\-resource speech to a frozen encoder and medium\- and low\-resource speech to a trainable encoder\.

## Methodology

### Model Architecture

As shown in Figure[2](https://arxiv.org/html/2608.04586#Sx2.F2), MSRT connects a resource\-aware Mixture of Speech Encoders \(MoSE\), a speech adapter, and a pretrained LLM\.

#### Mixture of Speech Encoders\.

MoSE comprises two encoders and an explicit language router\. Instead of sharing one encoder across languages with unequal supervision, it separates high\-resource knowledge preservation from medium\- and low\-resource adaptation while activating one expert per utterance\. This design breaks the curse of multilinguality by improving underrepresented languages without sacrificing high\-resource performance\.

- •Frozen Expert\.The frozen expertEfE\_\{f\}retains the original Whisper encoder with fixed parameters\. It preserves general speech representations for high\-resource languages, prevents catastrophic forgetting, and anchors the shared speech interface during adaptation\.
- •Trainable Expert\.The trainable expertEtE\_\{t\}shares the Whisper initialization but adapts to medium\- and low\-resource speech, capturing underrepresented accents and phonological patterns\. Its architecture matchesEfE\_\{f\}, so both experts produce compatible representations and share one adapter without expert\-specific projections\.
- •Explicit Language Router\.The router maps each source\-language token to its resource\-group expert: high\-resource languages useEfE\_\{f\}, whereas medium\- and low\-resource languages useEtE\_\{t\}\. Table[3](https://arxiv.org/html/2608.04586#Sx4.T3)lists all assignments\. Utterance\-level routing keeps all frames on one acoustic path and requires neither a learned gate nor a balancing loss\.

Overall, letℛ​\(ℓs\)\\mathcal\{R\}\(\\ell\_\{s\}\)denote the expert selected for source languageℓs\\ell\_\{s\}\. The MoSE output is written as

HMoSE​\(x,ℓs\)=Eℛ​\(ℓs\)​\(x\),ℛ​\(ℓs\)∈\{f,t\}\.H\_\{\\mathrm\{MoSE\}\}\(x,\\ell\_\{s\}\)=E\_\{\\mathcal\{R\}\(\\ell\_\{s\}\)\}\(x\),\\quad\\mathcal\{R\}\(\\ell\_\{s\}\)\\in\\\{f,t\\\}\.\(1\)Here,ℛ​\(ℓs\)=f\\mathcal\{R\}\(\\ell\_\{s\}\)=fforℓs∈ℒhigh\\ell\_\{s\}\\in\\mathcal\{L\}\_\{\\mathrm\{high\}\}andℛ​\(ℓs\)=t\\mathcal\{R\}\(\\ell\_\{s\}\)=totherwise, covering the medium\- and low\-resource groups\. This formulation retains the inference cost of a single encoder\.

#### Speech Adapter\.

The selected encoder produces variable\-length features\. A Q\-Former compresses them into a translation\-aware representation, while an MLP projects it into the LLM hidden space\. Together, these modules remove acoustic redundancy while retaining information required for translation\.

#### LLM\.

The pretrained LLM provides multilingual translation knowledge and remains frozen during speech\-side alignment to prevent catastrophic forgetting\. It is unlocked only in the final training stage, where it adapts to the aligned acoustic representation\.

#### Tokenizer Expansion\.

We add dedicated language tokens for the FLEURS languages\(Conneauet al\.[2023](https://arxiv.org/html/2608.04586#bib.bib9)\)\. The source\-language token controls MoSE encoder selection, while the target\-language token specifies the LLM translation direction\. Together, they provide explicit language control and reduce unintended switching\.

#### Text Prompt\.

As shown in Figure[2](https://arxiv.org/html/2608.04586#Sx2.F2)and Table[1](https://arxiv.org/html/2608.04586#Sx3.T1), the prompt explicitly instructs the LLM to translate from the source language into the target language\. The source\-language token also controls encoder selection through the explicit language router, while the target\-language token specifies the desired output language\.

Table 1:Prompt design for the five\-stage curriculum\.

### Five\-Stage Curriculum Learning

FollowingDuet al\.\([2025](https://arxiv.org/html/2608.04586#bib.bib1)\), we organize ASR, speech\-guided machine translation \(SMT\), and joint speech recognition and translation \(SRT\) into the five stages in Table[1](https://arxiv.org/html/2608.04586#Sx3.T1)\. Table[2](https://arxiv.org/html/2608.04586#Sx3.T2)summarizes which modules are optimized at each stage; every stage starts from the preceding checkpoint\.

- •Stage I: ASR alignment\.The Q\-Former and MLP adapter are trained successively on English\-only, high\-resource multilingual, and fully multilingual ASR data, establishing a stable multilingual speech–text interface while both speech experts and the LLM remain frozen\.
- •Stage II: Expert specialization\.MoSE routing is activated for multilingual ASR\. The frozen expertEfE\_\{f\}preserves high\-resource representations, whereasEtE\_\{t\}is optimized for medium\- and low\-resource speech; the shared adapter continues training\.
- •Stage III: Translation activation\.SMT supplies the ground\-truth transcription together with source\- and target\-language tokens\. The adapter is optimized to connect routed speech features to the translation knowledge of the frozen LLM\.
- •Stage IV: End\-to\-end SRT\.The ground\-truth transcription is removed from the input\. With the experts and LLM fixed, the adapter learns to generate both the transcription and translation directly from routed speech\.
- •Stage V: Joint adaptation\.LLM LoRA is enabled and jointly optimized with the adapter for SRT, adapting multilingual generation to acoustic inputs without updating the pretrained LLM weights\.

Table 2:MLLM settings for MSRT\.ModulesStageDetailsSpeech Encoder \(MoSE\)Frozen Expert \(EfE\_\{f\}\)–For high\-resourceTrainable Expert \(EtE\_\{t\}\)IIFor medium\-/low\-resourceLanguage Router–Explicit expert routingSpeech AdapterQ\-FormerAllFeature compressionMLPAllDimension alignmentLLM\+\+Vocabulary–Language tokens\+\+LoRAVr=16r=16,α=32\\alpha=32

## Experimental Settings

#### Datasets\.

We use Common Voice 24\(Ardilaet al\.[2020](https://arxiv.org/html/2608.04586#bib.bib5)\)and FLEURS for multilingual ASR pre\-training, combining broad language coverage with curated speech data\. For S2TT training, we use paired speech–text data from FLEURS, comprising approximately 10 hours of speech per language\. We evaluate the S2TT models on both FLEURS and the CoVoST\-2\(Wanget al\.[2021](https://arxiv.org/html/2608.04586#bib.bib24)\)benchmark\.

#### Training Details\.

As shown in Table[2](https://arxiv.org/html/2608.04586#Sx3.T2), the MLLM comprises a MiLMMT\-4B LLM\(Shanget al\.[2026](https://arxiv.org/html/2608.04586#bib.bib17)\), MoSE, and a speech adapter containing a Q\-Former and an MLP\. The Q\-Former uses 80 learnable queries to compress variable\-length speech into a fixed\-length interface for the LLM\. We train in BF16 with DeepSpeed ZeRO\-0 and AdamW, using a peak learning rate of1×10−41\\times 10^\{\-4\}and 1,000 warmup steps\. Training takes less than three days on eight Ascend 910C NPUs\. The same implementation can run on eight NVIDIA A100 GPUs with comparable accuracy\.

#### Baselines\.

We compare three representative deployment paradigms:

- •API\-based:Gemini\-3\.5\-Flash\-Lite\(Teamet al\.[2023](https://arxiv.org/html/2608.04586#bib.bib21)\)serves as the API baseline\.
- •Cascade:Whisper\-Large\-v3\(Radfordet al\.[2023](https://arxiv.org/html/2608.04586#bib.bib15)\)first transcribes speech, after which NLLB\-200\-3\.3B\(NLLB Teamet al\.[2024](https://arxiv.org/html/2608.04586#bib.bib13)\)translates the transcription\.
- •End\-to\-end:SeamlessM4T\-V2\-Large\(Barraultet al\.[2023](https://arxiv.org/html/2608.04586#bib.bib8)\), Qwen3\-Omni\-30B\(Xuet al\.[2025](https://arxiv.org/html/2608.04586#bib.bib25)\), and MCAT\-27B\(Duet al\.[2026](https://arxiv.org/html/2608.04586#bib.bib10)\)directly generate translations from speech and cover both specialized and general\-purpose multilingual architectures\.

#### Evaluation Metrics\.

We employ COMET\(Reiet al\.[2022](https://arxiv.org/html/2608.04586#bib.bib16)\)and spBLEU\(Post[2018](https://arxiv.org/html/2608.04586#bib.bib14)\)as our evaluation metrics\. Specifically, spBLEU uses the FLORES\-200 tokenizer\. All results use a beam size of 1 unless stated otherwise\.

Table 3:Language support\. Resource levels are defined following SeamlessM4T\(Barraultet al\.[2023](https://arxiv.org/html/2608.04586#bib.bib8)\)\.Table 4:spBLEU / COMET results for11×4411\\times 44and44×1144\\times 11directions on FLEURS\.

## Experiments

### Main Results

#### Language Selection\.

As shown in Table[4](https://arxiv.org/html/2608.04586#Sx4.T4), the 11 languages cover 11 language families: Arabic \(Afro\-Asiatic\), Mandarin Chinese \(Sino\-Tibetan\), English \(Indo\-European\), Hungarian \(Uralic\), Indonesian \(Austronesian\), Japanese \(Japonic\), Korean \(Koreanic\), Tamil \(Dravidian\), Thai \(Kra–Dai\), Turkish \(Turkic\), and Vietnamese \(Austroasiatic\)\.

#### Overall Analysis\.

According to the overall average in Table[4](https://arxiv.org/html/2608.04586#Sx4.T4), MSRT\-4B achieves the highest COMET score of 83\.3, outperforming MCAT\-27B \(80\.9\), Gemini\-3\.5\-Flash\-Lite \(80\.0\), Whisper\+NLLB\-200\-3\.3B \(78\.4\), Qwen3\-Omni \(77\.5\), and SeamlessM4T\-V2\-Large \(72\.5\)\. Moreover, MSRT\-4B ranks first in both the11×4411\\times 44and44×1144\\times 11settings, showing that its overall advantage is sustained across translation directions rather than driven by performance in a single setting\.

#### Translation Direction Analysis\.

Performance differs notably between11×4411\\times 44and44×1144\\times 11directions\. The cascaded system achieves a competitive COMET score of 80\.0 in the former setting but drops to 76\.8 in the latter, indicating that translation from a broader set of source speech languages is more challenging, partly due to accumulated multilingual recognition errors\. In contrast, MSRT\-4B achieves COMET scores of 83\.5 and 83\.2, respectively, demonstrating the strongest performance and robust consistency across translation directions\.

### Eng→\\rightarrow44 S2TT

#### Overall Analysis of Eng→\\rightarrow44 Directions\.

Table[5](https://arxiv.org/html/2608.04586#Sx5.T5)reports COMET performance across all 44 English\-source translation directions, including the newly evaluated Gemini\-3\.5\-Flash\-Lite baseline\. MSRT\-4B achieves the highest average score of87\.9, outperforming MCAT\-27B, Gemini\-3\.5\-Flash\-Lite, Qwen3\-Omni, and SeamlessM4T\-V2\-Large by 0\.8, 1\.3, 2\.2, and 2\.6 points, respectively\. These results demonstrate the strong and consistent English\-to\-many translation performance of MSRT\-4B\.

#### English\-centric vs\. Many\-to\-Many Translation\.

Although optimized for many\-to\-many translation, MSRT\-4B also performs strongly on English\-source directions, achieving an average COMET score of 87\.9 on eng→\\rightarrow44 and an overall score of 83\.3 in Table[4](https://arxiv.org/html/2608.04586#Sx4.T4)\. These results suggest that future speech translation models should emphasize comprehensive multilingual optimization rather than focusing predominantly on English\-centric directions\.

#### High\-Resource vs\. Low\-Resource Languages\.

Table[5](https://arxiv.org/html/2608.04586#Sx5.T5)shows that Qwen3\-Omni performs strongly on high\-resource languages such as Chinese \(cmn\) and Japanese \(jpn\), but degrades markedly on low\-resource languages such as Burmese \(mya\) and Khmer \(khm\)\. In comparison, MSRT\-4B maintains consistently strong performance across both resource regimes\. This advantage stems from MoSE, whose trainable expert specializes in medium\- and low\-resource languages, while its frozen expert preserves high\-resource performance\.

Table 5:COMET results on English→44\\rightarrow 44directions\.Table 6:COMET results across the45×4445\\times 44directions\.![Refer to caption](https://arxiv.org/html/2608.04586v1/x2.png)Figure 3:COMET results for all directions\. Shaded regions highlight scores falling below 80 and 70\. Scores for identical source and target languages, such as eng→\\rightarroweng along the diagonal, are smoothed for visualization\.

### Systematic Analysis on45×4445\\times 44Directions

#### Main Results\.

Table[6](https://arxiv.org/html/2608.04586#Sx5.T6)shows that MSRT\-4B achieves COMET scores of at least 80 on 1,552 of the 1,980 directions\. MCAT\-27B, Whisper\+NLLB\-3\.3B, Gemini\-3\.5\-Flash\-Lite, Qwen3\-Omni\-30B\-Instruct, and SeamlessM4T\-V2\-Large achieve scores of at least 80 on 1,232, 1,038, 928, 881, and 117 directions, respectively\. These results demonstrate the broader and more reliable many\-to\-many translation capabilities of MSRT\-4B\.

#### API Comparison\.

MSRT\-4B outperforms the Gemini\-3\.5\-Flash\-Lite API, achieving COMET scores of at least 80 on 1,552 directions, compared with 928 for Gemini, and scores below 70 on only 69 directions, compared with 237 for Gemini\. This result demonstrates broader language coverage and stronger overall performance than the API\.

#### English\-Centric Bias\.

SeamlessM4T\-V2\-Large is centered on English, with its strongest performance concentrated on directions involving English\. Its quality drops on non\-English\-to\-non\-English translation, yielding only 117 directions with COMET scores of at least 80 and 754 with scores below 70\. In contrast, MSRT\-4B remains consistent across the translation directions\.

#### Parameter Efficiency\.

With only 4B parameters, MSRT\-4B surpasses the much larger Qwen3\-Omni\-30B\-Instruct and MCAT\-27B, achieving COMET scores of at least 80 on 1,552 directions versus 881 and 1,232, respectively\. It also produces fewer low\-scoring directions, demonstrating efficient use of model capacity and broader multilingual coverage without relying on model scale\.

![Refer to caption](https://arxiv.org/html/2608.04586v1/Figures/s2tt_mt_combined.png)Figure 4:MoSE ablation and S2TT–MT comparison across resource levels\. Consistent gains, largest for low\-resource languages, show that MoSE breaks the curse of multilinguality\.Table 7:Ablation results of MSRT\-4B onX→44X\\\!\\rightarrow\\\!44translation\.

### Ablation Study

#### Component Ablations\.

Table[7](https://arxiv.org/html/2608.04586#Sx5.T7)evaluates MSRT\-4B on six relatively low\-performing source languages, with two selected from each resource group, translated into the other 44 languages\.

- •Number of experts\.Three\-way routing, with one expert for each resource group, performs similarly to two\-way routing between high\- and medium/low\-resource languages \(75\.3 vs\. 75\.1\), whereas using a single expert substantially reduces the average COMET score to 70\.0\. These results show that two experts provide sufficient resource\-aware specialization for the current 45 languages, while larger language inventories may benefit from finer\-grained expert partitioning\.
- •LoRA\.Removing LoRA lowers average COMET from 75\.1 to 73\.1, showing that lightweight decoder adaptation complements MoSE by mapping speech representations into the LLM’s text space\.

#### Resource\-Level Analysis\.

Figure[4](https://arxiv.org/html/2608.04586#Sx5.F4)compares MoSE variants and text MT across 45 languages by resource level; each score averages the other 44 target directions\.

- •MoSE gains across 45 languages\.MoSE improves COMET by 3\.0, 2\.2, and 1\.4 points for low\-, medium\-, and high\-resource languages, respectively\. Gains across all groups show that MoSE breaks the curse of multilinguality by substantially improving low\-resource translation while also strengthening high\-resource performance\.
- •Text MT vs\. S2TT\.Text MT with source transcriptions provides an oracle upper bound for S2TT\. Its gap to MSRT\-4B narrows as speech resources increase, identifying speech representation and cross\-modal alignment as the main bottlenecks\.

![Refer to caption](https://arxiv.org/html/2608.04586v1/Figures/scaling_data.png)Figure 5:Data scaling on 11 CoVoST\-2 English\-to\-X directions\. MSRT\-4B and MSRT\-4B\-SFT use 7\.5 hours of FLEURS and 429\.6 hours of CoVoST\-2 speech, respectively; dashed lines denote average COMET\.
#### Data Scaling\.

Figure[5](https://arxiv.org/html/2608.04586#Sx5.F5)shows strong zero\-shot transfer: with only7\.5 hof FLEURS English speech, MSRT achieves scores of at least 80 on all 11 English\-to\-X directions, averaging 83\.3\. Scaling to429\.6 hof CoVoST\-2 speech \(57\.3×57\.3\\times\) raises the average to85\.3, with consistent gains of 1\.3–2\.6 points across directions\.

## Conclusion

We introduced MSRT, a 4B many\-to\-many S2TT framework combining resource\-aware MoSE with five\-stage curriculum learning\. By separating high\-resource preservation from medium\- and low\-resource adaptation, MoSE improves all resource groups, with the largest gains on low\-resource speech\. Evaluation of all 1,980 directions among 45 languages demonstrates strong multilingual consistency, parameter efficiency, and cross\-dataset generalization, showing that MoSE effectively breaks the curse of multilinguality\. Future work will extend MoSE to broader language coverage\.

## Limitations

MSRT derives its S2TT capability from the machine translation knowledge of the pretrained LLM\. Consequently, its translation quality is bounded by the LLM’s underlying MT performance, particularly for languages and directions that are weakly represented during pretraining\.

## References

- R\. Ardila, M\. Branson, K\. Davis, M\. Kohler, J\. Meyer, M\. Henretty, R\. Morais, L\. Saunders, F\. Tyers, and G\. Weber \(2020\)Common voice: a massively\-multilingual speech corpus\.InProceedings of the Twelfth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 4218–4222\(eng\)\.External Links:[Link](https://aclanthology.org/2020.lrec-1.520/),ISBN 979\-10\-95546\-34\-4Cited by:[Datasets\.](https://arxiv.org/html/2608.04586#Sx4.SSx2.SSS0.Px1.p1.1)\.
- L\. Barrault, Y\. Chung, M\. C\. Meglioli, D\. Dale, N\. Dong, P\. Duquenne, H\. Elsahar, H\. Gong, K\. Heffernan, J\. Hoffman,et al\.\(2023\)SeamlessM4T: massively multilingual and multimodal machine translation\.arXiv preprint arXiv:2308\.11596\.Cited by:[Speech\-to\-Text Translation\.](https://arxiv.org/html/2608.04586#Sx2.SS0.SSS0.Px1.p1.1),[Many\-to\-Many Speech\-to\-Text Translation\.](https://arxiv.org/html/2608.04586#Sx2.SS0.SSS0.Px2.p1.1),[3rd item](https://arxiv.org/html/2608.04586#Sx4.I4.i3.p1.1),[Table 3](https://arxiv.org/html/2608.04586#Sx4.T3)\.
- T\. Blevins, T\. Limisiewicz, S\. Gururangan, M\. Li, H\. Gonen, N\. A\. Smith, and L\. Zettlemoyer \(2024\)Breaking the curse of multilinguality with cross\-lingual expert language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 10822–10837\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.604/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.604)Cited by:[Introduction](https://arxiv.org/html/2608.04586#Sx1.p4.1)\.
- A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov \(2020\)Unsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 8440–8451\.Cited by:[Introduction](https://arxiv.org/html/2608.04586#Sx1.p3.1)\.
- A\. Conneau, M\. Ma, S\. Khanuja, Y\. Zhang, V\. Axelrod, S\. Dalmia, J\. Riesa, C\. Rivera, and A\. Bapna \(2023\)FLEURS: few\-shot learning evaluation of universal representations of speech\.In2022 IEEE Spoken Language Technology Workshop,pp\. 798–805\.Cited by:[Tokenizer Expansion\.](https://arxiv.org/html/2608.04586#Sx3.SSx1.SSS0.Px4.p1.1)\.
- Y\. Du, K\. Liu, Y\. Pan, B\. Yang, K\. Deng, X\. Chen, Y\. Xiang, M\. Liu, B\. Qin, and Y\. Wang \(2026\)MCAT: scaling many\-to\-many speech\-to\-text translation with MLLMs to 70 languages\.IEEE/ACM Transactions on Audio, Speech, and Language Processing\.Cited by:[Introduction](https://arxiv.org/html/2608.04586#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.04586#Sx1.p5.1),[Many\-to\-Many Speech\-to\-Text Translation\.](https://arxiv.org/html/2608.04586#Sx2.SS0.SSS0.Px2.p1.1),[3rd item](https://arxiv.org/html/2608.04586#Sx4.I4.i3.p1.1)\.
- Y\. Du, Y\. Pan, Z\. Ma, B\. Yang, Y\. Yang, K\. Deng, X\. Chen, Y\. Xiang, M\. Liu, and B\. Qin \(2025\)Making llms better many\-to\-many speech\-to\-text translators with curriculum learning\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12466–12478\.Cited by:[Introduction](https://arxiv.org/html/2608.04586#Sx1.p4.1),[Many\-to\-Many Speech\-to\-Text Translation\.](https://arxiv.org/html/2608.04586#Sx2.SS0.SSS0.Px2.p1.1),[Five\-Stage Curriculum Learning](https://arxiv.org/html/2608.04586#Sx3.SSx2.p1.1)\.
- C\. Le, B\. Han, J\. Li, S\. Chen, and Y\. Qian \(2025\)Simulmega: moe routers are advanced policy makers for simultaneous speech translation\.Advances in Neural Information Processing Systems38,pp\. 42739–42761\.Cited by:[Mixture of Experts\.](https://arxiv.org/html/2608.04586#Sx2.SS0.SSS0.Px3.p1.1)\.
- Y\. Li, S\. Jiang, B\. Hu, L\. Wang, W\. Zhong, W\. Luo, L\. Ma, and M\. Zhang \(2025\)Uni\-moe: scaling unified multimodal llms with mixture of experts\.IEEE Transactions on Pattern Analysis and Machine Intelligence47\(5\),pp\. 3424–3439\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2025.3532688)Cited by:[Mixture of Experts\.](https://arxiv.org/html/2608.04586#Sx2.SS0.SSS0.Px3.p1.1)\.
- Z\. Ma, G\. Yang, W\. Chen, Z\. Gao, Y\. Du, X\. Li, Z\. Zheng, H\. Zhu, J\. Zhuo, Z\. Song, R\. Xu, T\. Wang, Y\. Yang, Y\. Zhu, Z\. Niu, L\. Xue, Y\. Ma, R\. Yuan, S\. Zhang, K\. Yu, E\. S\. Chng, and X\. Chen \(2026\)SLAM\-llm: a modular, open\-source multimodal large language model framework and best practice for speech, language, audio and music processing\.IEEE Journal of Selected Topics in Signal Processing20\(1\),pp\. 63–76\.External Links:[Document](https://dx.doi.org/10.1109/JSTSP.2026.3653157)Cited by:[Introduction](https://arxiv.org/html/2608.04586#Sx1.p1.1)\.
- NLLB Team, M\. R\. Costa\-jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield, K\. Heffernan, E\. Kalbassi, J\. Lam, D\. Licht,et al\.\(2024\)Scaling neural machine translation to 200 languages\.Nature630\(8018\),pp\. 841–846\.Cited by:[2nd item](https://arxiv.org/html/2608.04586#Sx4.I4.i2.p1.1)\.
- M\. Post \(2018\)A call for clarity in reporting BLEU scores\.InProceedings of the Third Conference on Machine Translation,pp\. 186–191\.Cited by:[Evaluation Metrics\.](https://arxiv.org/html/2608.04586#Sx4.SSx2.SSS0.Px4.p1.1)\.
- A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever \(2023\)Robust speech recognition via large\-scale weak supervision\.InProceedings of the 40th International Conference on Machine Learning,pp\. 28492–28518\.Cited by:[Introduction](https://arxiv.org/html/2608.04586#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.04586#Sx1.p4.1),[2nd item](https://arxiv.org/html/2608.04586#Sx4.I4.i2.p1.1)\.
- R\. Rei, J\. G\. C\. De Souza, D\. Alves, C\. Zerva, A\. C\. Farinha, T\. Glushkova, A\. Lavie, L\. Coheur, and A\. F\. T\. Martins \(2022\)COMET\-22: unbabel\-IST 2022 submission for the metrics shared task\.InProceedings of the Seventh Conference on Machine Translation,pp\. 578–585\.Cited by:[Evaluation Metrics\.](https://arxiv.org/html/2608.04586#Sx4.SSx2.SSS0.Px4.p1.1)\.
- W\. Shan, Y\. Li, Y\. Zhang, Y\. Luo, C\. Xu, X\. Zhao, L\. Meng, Y\. Lu, M\. Zhang, H\. Yang, T\. Xiao, and J\. Zhu \(2025\)Enhancing speech large language models with prompt\-aware mixture of audio encoders\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 19305–19320\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.974/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.974),ISBN 979\-8\-89176\-332\-6Cited by:[Mixture of Experts\.](https://arxiv.org/html/2608.04586#Sx2.SS0.SSS0.Px3.p1.1)\.
- Y\. Shang, P\. Gao, W\. Liu, J\. Luan, and J\. Su \(2026\)Scaling model and data for multilingual machine translation with open large language models\.External Links:2602\.11961Cited by:[Introduction](https://arxiv.org/html/2608.04586#Sx1.p1.1),[Training Details\.](https://arxiv.org/html/2608.04586#Sx4.SSx2.SSS0.Px2.p1.1)\.
- G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.\(2023\)Gemini: a family of highly capable multimodal models\.arXiv preprint arXiv:2312\.11805\.Cited by:[1st item](https://arxiv.org/html/2608.04586#Sx4.I4.i1.p1.1)\.
- I\. Tsiamas, G\. I\. Gállego, J\. A\. Fonollosa, and M\. R\. Costa\-jussà \(2024\)Pushing the limits of zero\-shot end\-to\-end speech translation\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 14245–14267\.Cited by:[Speech\-to\-Text Translation\.](https://arxiv.org/html/2608.04586#Sx2.SS0.SSS0.Px1.p1.1)\.
- C\. Wang, A\. Wu, J\. Gu, and J\. Pino \(2021\)CoVoST 2 and massively multilingual speech\-to\-text translation\.pp\. 2247–2251\.Cited by:[Datasets\.](https://arxiv.org/html/2608.04586#Sx4.SSx2.SSS0.Px1.p1.1)\.
- J\. Xu, Z\. Guo, H\. Hu, Y\. Chu, X\. Wang, J\. He, Y\. Wang, X\. Shi, T\. He, X\. Zhu,et al\.\(2025\)Qwen3\-Omni technical report\.arXiv preprint arXiv:2509\.17765\.Cited by:[Introduction](https://arxiv.org/html/2608.04586#Sx1.p3.1),[Many\-to\-Many Speech\-to\-Text Translation\.](https://arxiv.org/html/2608.04586#Sx2.SS0.SSS0.Px2.p1.1),[3rd item](https://arxiv.org/html/2608.04586#Sx4.I4.i3.p1.1)\.
- J\. Zhou, S\. Gao, Z\. Yu, L\. Dong, and W\. Wang \(2024\)DialectMoE: an end\-to\-end multi\-dialect speech recognition model with mixture\-of\-experts\.InProceedings of the 23rd Chinese National Conference on Computational Linguistics \(Volume 1: Main Conference\),S\. Maosong, L\. Jiye, H\. Xianpei, L\. Zhiyuan, and H\. Yulan \(Eds\.\),Taiyuan, China,pp\. 1148–1159\(eng\)\.External Links:[Link](https://aclanthology.org/2024.ccl-1.89/)Cited by:[Mixture of Experts\.](https://arxiv.org/html/2608.04586#Sx2.SS0.SSS0.Px3.p1.1)\.