From Flat Language Labels to Typological Priors: Structured Language Conditioning for Multilingual Speech-to-Speech Translation

# From Flat Language Labels to Typological Priors: Structured Language Conditioning for Multilingual Speech-to-Speech Translation
Source: [https://arxiv.org/html/2605.16026](https://arxiv.org/html/2605.16026)
Yu Pan, Yang Hou, Xiongfei Wu, Liang Zhang, Yves LE TRAON, Lei Ma, Jianjun ZhaoYu Pan was with the School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819\-0395, Japan, when this work was conducted, and is currently with Recho Inc\., Tokyo, Japan \(e\-mail: panyu\.ztj@gmail\.com\)\.Yang Hou is with the National Institute of Informatics, Tokyo, Japan \(e\-mail: yang\-hou@nii\.ac\.jp\)\.Xiongfei Wu and Yves LE TRAON are with the Interdisciplinary Research Centre on Security, Reliability and Trust \(SnT\), University of Luxembourg, Luxembourg \(e\-mail: xiongfei\.wu\.a94@gmail\.com; yves\.letraon@uni\.lu\)\.Liang Zhang is with Donghua University, Shanghai 201620, China \(e\-mail: zhangliang@dhu\.edu\.cn\)\.Lei Ma is with the Department of Computer Science, The University of Tokyo, Tokyo 113\-8656, Japan, and the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada \(e\-mail: ma\.lei@acm\.org\)\.Jianjun Zhao is with the School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819\-0395, Japan \(e\-mail: zhao@ait\.kyushu\-u\.ac\.jp\)\.

###### Abstract

Compositional speech\-to\-speech translation \(S2ST\) systems built upon speech large language models \(SpeechLLMs\) have recently shown promising performance\. However, existing S2ST systems often either neglect source\-language information or encode it through a*language\-as\-label*paradigm, representing each source language as an independent flat embedding\. Such a design overlooks systematic linguistic structure shared across languages, which may limit data\-efficient multilingual adaptation when supervised S2ST data are scarce\. To address this issue, we proposeS2ST\-Omni 2, a many\-to\-one compositional S2ST framework that systematically reformulates multilingual language conditioning from flat language labels to structured typological priors\. Specifically, S2ST\-Omni 2 revisits language conditioning at three levels: typology\-informed hierarchical language encoding for structured source\-language representation, dynamically\-gated language\-aware Dual\-CTC for content\-adaptive acoustic modulation, and typology\-aware LLM prompting for decoder\-side linguistic guidance\. Experiments on CVSS\-C show that S2ST\-Omni 2 achieves superior average performance among representative S2ST approaches across BLEU, COMET, ASR\-BLEU, and BLASER 2\.0 under the adopted evaluation protocol\. Ablation studies indicate that the proposed representation\-level, acoustic\-level, and decoding\-level strategies provide complementary benefits\. Moreover, controlled data\-budget analyses and a Japanese\-to\-English evaluation using only∼\\sim3 hours of supervised training data suggest that explicit typological priors provide useful inductive biases for data\-efficient multilingual S2ST\.

## IIntroduction

Multilingual speech\-to\-speech translation \(S2ST\) aims to directly translate spoken utterances from one language into speech in another, and is essential for cross\-lingual communication in scenarios such as healthcare, education, and international collaboration\[[24](https://arxiv.org/html/2605.16026#bib.bib6),[27](https://arxiv.org/html/2605.16026#bib.bib1)\]\.

Traditional S2ST systems typically rely on cascaded automatic speech recognition \(ASR\)\[[15](https://arxiv.org/html/2605.16026#bib.bib60),[44](https://arxiv.org/html/2605.16026#bib.bib14)\], machine translation \(MT\)\[[30](https://arxiv.org/html/2605.16026#bib.bib51),[34](https://arxiv.org/html/2605.16026#bib.bib52)\], and text\-to\-speech synthesis \(TTS\)\[[10](https://arxiv.org/html/2605.16026#bib.bib18),[5](https://arxiv.org/html/2605.16026#bib.bib21)\]\. Although effective in practice, such pipelines are prone to error propagation and cannot be optimized globally\. Recent studies have therefore explored end\-to\-end S2ST\[[23](https://arxiv.org/html/2605.16026#bib.bib87),[21](https://arxiv.org/html/2605.16026#bib.bib26),[4](https://arxiv.org/html/2605.16026#bib.bib41)\]and compositional S2ST\[[12](https://arxiv.org/html/2605.16026#bib.bib65),[13](https://arxiv.org/html/2605.16026#bib.bib24),[32](https://arxiv.org/html/2605.16026#bib.bib94)\]\. Among them, compositional S2ST, which combines a speech\-to\-text translation \(S2TT\) frontend with a TTS backend, offers a practical balance between modularity, interpretability, and the ability to leverage speech and text resources independently\.

With the rapid progress of large language models \(LLMs\)\[[1](https://arxiv.org/html/2605.16026#bib.bib59),[3](https://arxiv.org/html/2605.16026#bib.bib58),[40](https://arxiv.org/html/2605.16026#bib.bib57)\], speech\-aware LLMs \(SpeechLLMs\)\[[46](https://arxiv.org/html/2605.16026#bib.bib64),[19](https://arxiv.org/html/2605.16026#bib.bib53)\]have become a promising foundation for multilingual S2ST\[[8](https://arxiv.org/html/2605.16026#bib.bib92),[32](https://arxiv.org/html/2605.16026#bib.bib94),[48](https://arxiv.org/html/2605.16026#bib.bib93)\]\. Along this line, S2ST\-Omni\[[32](https://arxiv.org/html/2605.16026#bib.bib94)\]introduces language\-label conditioning into a SpeechLLM\-based compositional framework, enabling effective many\-to\-one S2ST\. However, its language\-conditioning strategy follows a*language\-as\-label*paradigm, where each source language is represented as an isolated identifier\. Such flat language representations overlook systematic linguistic regularities in morphology, reordering tendencies, and genealogical relatedness, which can affect speech alignment, semantic interpretation, and target\-language generation\[[7](https://arxiv.org/html/2605.16026#bib.bib88),[29](https://arxiv.org/html/2605.16026#bib.bib89),[36](https://arxiv.org/html/2605.16026#bib.bib90),[31](https://arxiv.org/html/2605.16026#bib.bib91)\]\. From this perspective, multilingual S2ST should not only identify*which*language the input belongs to, but also capture*what structural properties*the language exhibits\. Flat language embeddings may therefore be insufficient for exposing structural priors that support data\-efficient multilingual adaptation\[[2](https://arxiv.org/html/2605.16026#bib.bib61),[38](https://arxiv.org/html/2605.16026#bib.bib46)\]\.

In this paper, we proposeS2ST\-Omni 2, a typology\-aware compositional framework for many\-to\-one data\-efficient S2ST\. Built upon S2ST\-Omni, S2ST\-Omni 2 preserves the encoder–adapter–LLM–TTS skeleton while redesigning the language\-conditioning pathway at three levels\. First,typology\-informed hierarchical language encoding \(TI\-HLE\)decomposes source\-language information into morphology\-related, reordering, genealogical\-family, and residual language\-specific channels\. Second, adynamically\-gated language\-aware Dual\-CTCmechanism performs content\-adaptive frame\-wise modulation for multilingual acoustic modeling\. Third,typology\-aware promptinginjects translation\-oriented linguistic priors into LLM decoding\. Together, these components provide a structured, adaptive, and linguistically grounded formulation of multilingual language conditioning\. We evaluate S2ST\-Omni 2 on CVSS\-C\[[22](https://arxiv.org/html/2605.16026#bib.bib2)\]against representative S2ST systems\. Compared with the direct baseline S2ST\-Omni, S2ST\-Omni 2 achieves average relative gains of 5\.8% in BLEU and 4\.6% in ASR\-BLEU, with consistent improvements in COMET and BLASER 2\.0\. Ablation studies show the complementary contributions of the proposed representation\-level, acoustic\-level, and decoding\-level strategies\. Furthermore, controlled data\-budget analyses and a limited\-supervision Japanese\-to\-English evaluation suggest that explicit typological priors are particularly beneficial when supervised data are scarce\.

In summary, this work substantially extends S2ST\-Omni\[[32](https://arxiv.org/html/2605.16026#bib.bib94)\]by reformulating flat language\-label conditioning as structured typological conditioning and by providing a broader empirical evaluation\. The main contributions are as follows:

- •We proposeS2ST\-Omni 2, a typology\-aware compositional S2ST framework that reformulates multilingual language conditioning from flat language labels to structured typological priors\.
- •We introduceTI\-HLE, which decomposes source\-language information into morphology\-related, reordering, genealogical\-family, and residual language\-specific channels\.
- •We propose adynamically\-gated language\-aware Dual\-CTCmechanism andtypology\-aware promptingto inject typological priors into acoustic feature modulation and LLM\-based decoding, respectively\.
- •We conduct extensive experiments on CVSS\-C, including ablation studies, TTS\-backend analysis, data\-budget comparisons, and a Japanese\-to\-English evaluation using∼\\sim3 hours of supervised data, providing empirical evidence for the effectiveness of structured typological conditioning in the evaluated multilingual S2ST setting\.

![Refer to caption](https://arxiv.org/html/2605.16026v1/images/s2st-omni2_zl.png)Figure 1:Overall architecture and two\-stage training pipeline of S2ST\-Omni 2\. LA denotes language\-aware, CE is cross\-entropy, and src/tgt denote source/target\. TI\-HLE and Dynamically\-Gated LA\-Dual\-CTC are training\-time auxiliary modules, whereas typology\-aware prompting is retained during inference\.
## IIMethodology

### II\-ASystem Overview

As shown in Fig\.[1](https://arxiv.org/html/2605.16026#S1.F1),S2ST\-Omni 2follows the compositional design of S2ST\-Omni\[[32](https://arxiv.org/html/2605.16026#bib.bib94)\], consisting of a SpeechLLM\-based S2TT frontend and a plug\-and\-play TTS backend\. The frontend contains five main components: 1\) a frozen Whisper encoder\[[37](https://arxiv.org/html/2605.16026#bib.bib33)\]for frame\-level acoustic–semantic feature extraction; 2\) a hybrid speech adapter inherited from S2ST\-Omni to map speech features into the LLM hidden space; 3\) a*TI\-HLE*module that represents each source language through morphology, reordering, genealogical family, and residual language\-specific factors; 4\) a*Dynamically\-Gated Language\-Aware Dual\-CTC*module that applies typology\-conditioned modulation to intermediate adapter features with auxiliary source\- and target\-side CTC supervision; and 5\) a Qwen3\-4B decoder\[[43](https://arxiv.org/html/2605.16026#bib.bib3)\]guided by a*Typology\-Aware LLM Prompt*for target\-language translation\. The TTS backend is decoupled from the S2TT frontend, allowing different synthesizers to be integrated without retraining\. Following S2ST\-Omni\[[32](https://arxiv.org/html/2605.16026#bib.bib94)\], the source\-language identifier is obtained from ground\-truth labels during training and predicted from Whisper encoder representations during inference\.

The key distinction from S2ST\-Omni lies in the language\-conditioning pathway\. Rather than modifying the overall S2ST backbone, S2ST\-Omni 2 replaces flat language\-label conditioning with structured typological priors injected at the representation, acoustic\-modulation, and LLM\-decoding levels\. Keeping the backbone unchanged reduces architectural confounds and enables a focused examination of linguistically grounded conditioning while preserving the modularity of the original framework\. During inference, the TI\-HLE and dynamically\-gated LA\-Dual\-CTC modules are discarded together with the auxiliary CTC branches; therefore, they act only as training\-time typological inductive biases and introduce no additional acoustic\-side inference cost or change to the encoder–adapter–LLM forward path\. The only inference\-time difference is the typology\-aware prompt selected according to the predicted source language\.

### II\-BHybrid Speech Adapter

We adopt the hybrid adapter from S2ST\-Omni\[[32](https://arxiv.org/html/2605.16026#bib.bib94)\]to bridge the frozen Whisper encoder and the Qwen3 LLM\. This component is kept unchanged to minimize architectural confounds and isolate the effect of the proposed typology\-aware conditioning\. Given Whisper encoder outputsX∈ℝB×T×1280X\\in\\mathbb\{R\}^\{B\\times T\\times 1280\}, the adapter first projects them into adh=1024d\_\{h\}=1024hidden space, applies two local depthwise\-separable convolution blocks with kernel size 7, downsamples the sequence with stride 2, and then uses two global self\-attention blocks to model long\-range dependencies\. We denote the downsampled intermediate adapter features as𝐇down∈ℝB×T′×dh\\mathbf\{H\}\_\{\\mathrm\{down\}\}\\in\\mathbb\{R\}^\{B\\times T^\{\\prime\}\\times d\_\{h\}\}, wheredh=1024d\_\{h\}=1024andT′=⌈T/2⌉T^\{\\prime\}=\\lceil T/2\\rceil\. The final linear projection maps the adapter output to the LLM hidden dimensiondllm=3584d\_\{\\mathrm\{llm\}\}=3584, yielding𝐙∈ℝB×T′×dllm\\mathbf\{Z\}\\in\\mathbb\{R\}^\{B\\times T^\{\\prime\}\\times d\_\{\\mathrm\{llm\}\}\}for Qwen3 decoding\. More details of this inherited adapter can be found in\[[32](https://arxiv.org/html/2605.16026#bib.bib94)\]\.

### II\-CTypology\-Informed Hierarchical Language Encoding

Flat language conditioning treats each source language as an isolated symbol, without explicitly exposing linguistic properties that affect translation behavior\. Motivated by linguistic typology and typology\-based language representations in NLP\[[7](https://arxiv.org/html/2605.16026#bib.bib88),[29](https://arxiv.org/html/2605.16026#bib.bib89),[36](https://arxiv.org/html/2605.16026#bib.bib90),[31](https://arxiv.org/html/2605.16026#bib.bib91)\], we construct a typology\-informed language representation for speech\-side conditioning by decomposing source\-language information into four complementary feature groups: morphology\-related profile, English\-directed reordering profile, genealogical family, and a language\-specific residual channel\. The first three groups provide coarse but interpretable typological priors, whose assignments are summarized in Table[I](https://arxiv.org/html/2605.16026#S2.T1), while the residual channel preserves fine\-grained language\-specific information not captured by these categories\. These assignments are not intended as exhaustive linguistic classifications; rather, they are coarse, translation\-oriented profiles designed to encode recurrent structural tendencies relevant to English\-directed S2ST\.

TABLE I:Typological feature assignment used in S2ST\-Omni 2\.#### II\-C1Typological Feature Encoding

For each source language, we encode the morphology\-related profile with a learnable embedding𝐞m∈ℝd1\\mathbf\{e\}\_\{m\}\\in\\mathbb\{R\}^\{d\_\{1\}\}\. Following standard linguistic typology\[[6](https://arxiv.org/html/2605.16026#bib.bib62),[16](https://arxiv.org/html/2605.16026#bib.bib44)\], French and Spanish are assigned to a fusional profile, German to a fusional\+compounding profile, and Japanese to an agglutinative profile\. This group provides priors for morphologically structured forms, productive compounding, and speech–text correspondence\.

Then, we encode the English\-directed reordering profile with a learnable embedding𝐞w∈ℝd2\\mathbf\{e\}\_\{w\}\\in\\mathbb\{R\}^\{d\_\{2\}\}\. Based on typological word\-order classifications\[[9](https://arxiv.org/html/2605.16026#bib.bib47),[17](https://arxiv.org/html/2605.16026#bib.bib49)\]and the reordering demands of translation into English, French and Spanish are assigned to an SVO\-oriented profile, whereas German and Japanese are assigned to a verb\-/clause\-final reordering profile\. This grouping does not imply that German and Japanese share the same syntactic system; rather, it reflects that both often require stronger clause\-final or verb\-final reordering cues than French and Spanish in English\-directed translation\.

We further encode genealogical family with a learnable embedding𝐞f∈ℝd3\\mathbf\{e\}\_\{f\}\\in\\mathbb\{R\}^\{d\_\{3\}\}\. Motivated by prior NLP work on cross\-lingual structure and genealogical relatedness\[[29](https://arxiv.org/html/2605.16026#bib.bib89),[36](https://arxiv.org/html/2605.16026#bib.bib90)\], French and Spanish share a Romance\-family embedding, while German and Japanese are assigned distinct Germanic and Japonic embeddings\. This design enables explicit sharing for related languages while preserving separate family\-level priors for unrelated languages in the present benchmark\[[28](https://arxiv.org/html/2605.16026#bib.bib50)\]\.

#### II\-C2Language\-Specific Residual Channel

Because typological profiles are necessarily coarse\-grained, we introduce a language\-specific residual channel𝐞r∈ℝd4\\mathbf\{e\}\_\{r\}\\in\\mathbb\{R\}^\{d\_\{4\}\}to preserve information not covered by morphology, reordering, or genealogical family\. Its dimensionality is set to match the flat language\-embedding dimension used in S2ST\-Omni, so that the residual channel retains the original language\-specific capacity while the additional channels explicitly encode typological structure\.

#### II\-C3Multi\-Feature Fusion

The four feature groups are concatenated and projected into a unified language representation:

𝐫lang=GELU​\(LN​\(𝐖f​\[𝐞m;𝐞w;𝐞f;𝐞r\]\+𝐛f\)\),\\mathbf\{r\}\_\{\\mathrm\{lang\}\}=\\mathrm\{GELU\}\\\!\\left\(\\mathrm\{LN\}\\\!\\left\(\\mathbf\{W\}\_\{f\}\[\\mathbf\{e\}\_\{m\};\\mathbf\{e\}\_\{w\};\\mathbf\{e\}\_\{f\};\\mathbf\{e\}\_\{r\}\]\+\\mathbf\{b\}\_\{f\}\\right\)\\right\),\(1\)where𝐖f∈ℝdc×\(d1\+d2\+d3\+d4\)\\mathbf\{W\}\_\{f\}\\in\\mathbb\{R\}^\{d\_\{c\}\\times\(d\_\{1\}\+d\_\{2\}\+d\_\{3\}\+d\_\{4\}\)\},dc=256d\_\{c\}=256, andLN\\mathrm\{LN\}denotes layer normalization\. The resulting representation𝐫lang∈ℝdc\\mathbf\{r\}\_\{\\mathrm\{lang\}\}\\in\\mathbb\{R\}^\{d\_\{c\}\}is used as the conditioning input for both the FiLM generator and the dynamic frame gate\.

### II\-DDynamically\-Gated Language\-Aware Dual\-CTC

To make language conditioning sensitive to both source\-language structure and frame\-level acoustic variation, we introduce a dynamically\-gated Language\-Aware Dual\-CTC module over the downsampled intermediate adapter features𝐇down∈ℝB×T′×dh\\mathbf\{H\}\_\{\\mathrm\{down\}\}\\in\\mathbb\{R\}^\{B\\times T^\{\\prime\}\\times d\_\{h\}\}\. This module consists of a typology\-conditioned source CTC branch and a language\-agnostic target CTC branch, which jointly provide source\-side content preservation and target\-side alignment guidance\.

#### II\-D1FiLM\-Based Language Conditioning

Given the fused language representation𝐫lang∈ℝdc\\mathbf\{r\}\_\{\\mathrm\{lang\}\}\\in\\mathbb\{R\}^\{d\_\{c\}\}, a FiLM generator\[[35](https://arxiv.org/html/2605.16026#bib.bib95),[45](https://arxiv.org/html/2605.16026#bib.bib54)\]predicts feature\-wise affine modulation parameters:

\[𝜸,𝜷\]=split⁡\(tanh⁡\(fFiLM​\(𝐫lang\)\)\),\[\\boldsymbol\{\\gamma\},\\boldsymbol\{\\beta\}\]=\\operatorname\{split\}\\left\(\\tanh\\left\(f\_\{\\mathrm\{FiLM\}\}\(\\mathbf\{r\}\_\{\\mathrm\{lang\}\}\)\\right\)\\right\),\(2\)wherefFiLM:ℝdc→ℝ2​dhf\_\{\\mathrm\{FiLM\}\}:\\mathbb\{R\}^\{d\_\{c\}\}\\rightarrow\\mathbb\{R\}^\{2d\_\{h\}\}is a two\-layer MLP, andsplit⁡\(⋅\)\\operatorname\{split\}\(\\cdot\)evenly divides the output into𝜸,𝜷∈ℝdh\\boldsymbol\{\\gamma\},\\boldsymbol\{\\beta\}\\in\\mathbb\{R\}^\{d\_\{h\}\}\. Thetanh⁡\(⋅\)\\tanh\(\\cdot\)activation bounds the modulation parameters and helps stabilize training\. For each framett, the intermediate adapter feature𝐡tdown∈ℝdh\\mathbf\{h\}^\{\\mathrm\{down\}\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{h\}\}is modulated as:

𝐡~tsrc=\(𝟏\+gt​𝜸\)⊙𝐡tdown\+gt​𝜷,\\widetilde\{\\mathbf\{h\}\}^\{\\mathrm\{src\}\}\_\{t\}=\\left\(\\mathbf\{1\}\+g\_\{t\}\\boldsymbol\{\\gamma\}\\right\)\\odot\\mathbf\{h\}^\{\\mathrm\{down\}\}\_\{t\}\+g\_\{t\}\\boldsymbol\{\\beta\},\(3\)where𝐡~tsrc∈ℝdh\\widetilde\{\\mathbf\{h\}\}^\{\\mathrm\{src\}\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{h\}\}is the modulated source\-side feature,𝟏\\mathbf\{1\}is an all\-ones vector,gt∈\(0,1\)g\_\{t\}\\in\(0,1\)is a scalar gate broadcast along the feature dimension, and⊙\\odotdenotes element\-wise multiplication\.

#### II\-D2Dynamic Frame Gate

Instead of applying a globally shared static gate, we compute a per\-frame gate conditioned on both acoustic content and language representation:

gt=σ​\(fgate​\(\[𝐡tdown;𝐫lang\]\)τ\),g\_\{t\}=\\sigma\\\!\\left\(\\frac\{f\_\{\\mathrm\{gate\}\}\(\[\\mathbf\{h\}^\{\\mathrm\{down\}\}\_\{t\};\\mathbf\{r\}\_\{\\mathrm\{lang\}\}\]\)\}\{\\tau\}\\right\),\(4\)where\[⋅;⋅\]\[\\cdot;\\cdot\]denotes vector concatenation,fgate:ℝdh\+dc→ℝf\_\{\\mathrm\{gate\}\}:\\mathbb\{R\}^\{d\_\{h\}\+d\_\{c\}\}\\rightarrow\\mathbb\{R\}is a two\-layer MLP, andσ​\(⋅\)\\sigma\(\\cdot\)is the sigmoid function\. The temperature is parameterized as

τ=softplus⁡\(τlearn\)\+ϵ,\\tau=\\operatorname\{softplus\}\(\\tau\_\{\\mathrm\{learn\}\}\)\+\\epsilon,\(5\)whereτlearn\\tau\_\{\\mathrm\{learn\}\}is a learnable scalar,softplus⁡\(⋅\)\\operatorname\{softplus\}\(\\cdot\)ensures positivity, andϵ=0\.1\\epsilon=0\.1prevents the temperature from becoming too small\. The bias offgatef\_\{\\mathrm\{gate\}\}is initialized to−2\.0\-2\.0, so that modulation is weak at the beginning of training and gradually increases as stable conditioning patterns emerge\. This design allows typology\-aware modulation to vary across both languages and time frames, rather than being uniformly applied to the entire utterance\.

#### II\-D3Source and Target CTC Branches

The source CTC branch applies the gated FiLM modulation in Eq\.[3](https://arxiv.org/html/2605.16026#S2.E3)and projects the resulting features𝐇~src=\{𝐡~tsrc\}t=1T′\\widetilde\{\\mathbf\{H\}\}^\{\\mathrm\{src\}\}=\\\{\\widetilde\{\\mathbf\{h\}\}^\{\\mathrm\{src\}\}\_\{t\}\\\}\_\{t=1\}^\{T^\{\\prime\}\}to source\-language CTC logits\. It is supervised by the standard CTC loss\[[14](https://arxiv.org/html/2605.16026#bib.bib83)\], encouraging the model to preserve source\-language content under typology\-aware conditioning\. In contrast, the target CTC branch operates directly on the unconditioned intermediate adapter features𝐇down\\mathbf\{H\}\_\{\\mathrm\{down\}\}and predicts English CTC logits, providing an additional target\-side alignment signal without language\-specific modulation\. The source and target CTC losses are combined with the LLM cross\-entropy loss during progressive fine\-tuning\. Following S2ST\-Omni\[[32](https://arxiv.org/html/2605.16026#bib.bib94)\], we use stage\-specific CTC weights rather than a single fixed weighting scheme throughout training\. The detailed two\-stage objectives are described in Section[II\-F](https://arxiv.org/html/2605.16026#S2.SS6)\.

### II\-ETypology\-Aware LLM Prompting

To complement acoustic\-level conditioning, we introduce a typology\-aware LLM prompting strategy\. Each prompt consists of a shared system instruction specifying general translation principles and a language\-specific instruction highlighting major translation challenges of the source language\. The prompts are constructed from coarse typological and linguistic properties relevant to translation, without using sentence\-level annotations, dataset\-specific examples, or utterance\-level information\. Specifically, the German prompt emphasizes compound decomposition and clause\-final\-to\-English reordering; the French and Spanish prompts focus on idiomatic expressions and lexical usage; and the Japanese prompt accounts for SOV\-to\-SVO reordering, omitted\-subject inference, and honorific normalization\. Since the same prompt is applied to all utterances from the same source language, this strategy provides only language\-level prior knowledge and encourages more natural and faithful English translations\.

### II\-FProgressive Fine\-Tuning

Following S2ST\-Omni\[[32](https://arxiv.org/html/2605.16026#bib.bib94)\], we adopt the same two\-stage progressive fine\-tuning strategy to stabilize speech–text alignment before LLM adaptation\. In both stages, the Whisper encoder and the base Qwen3 parameters are frozen, while the hybrid speech adapter, TI\-HLE module, and dynamically\-gated LA\-Dual\-CTC module are updated\.

In Stage I, the model is optimized mainly to establish reliable speech–text alignment using the LLM cross\-entropy loss and dual CTC supervision:

ℒ\(1\)=ℒCE\+λsrc\(1\)​ℒCTCsrc\+λtgt\(1\)​ℒCTCtgt\.\\mathcal\{L\}^\{\(1\)\}=\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\+\\lambda^\{\(1\)\}\_\{\\mathrm\{src\}\}\\mathcal\{L\}^\{\\mathrm\{src\}\}\_\{\\mathrm\{CTC\}\}\+\\lambda^\{\(1\)\}\_\{\\mathrm\{tgt\}\}\\mathcal\{L\}^\{\\mathrm\{tgt\}\}\_\{\\mathrm\{CTC\}\}\.\(6\)
In Stage II, the same modules remain trainable, and LoRA\[[18](https://arxiv.org/html/2605.16026#bib.bib4)\]adapters are inserted into the query and value projections in the self\-attention modules of Qwen3 to enhance translation capability\. The CTC losses are down\-weighted to retain auxiliary alignment regularization:

ℒ\(2\)=ℒCE\+λsrc\(2\)​ℒCTCsrc\+λtgt\(2\)​ℒCTCtgt\.\\mathcal\{L\}^\{\(2\)\}=\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\+\\lambda^\{\(2\)\}\_\{\\mathrm\{src\}\}\\mathcal\{L\}^\{\\mathrm\{src\}\}\_\{\\mathrm\{CTC\}\}\+\\lambda^\{\(2\)\}\_\{\\mathrm\{tgt\}\}\\mathcal\{L\}^\{\\mathrm\{tgt\}\}\_\{\\mathrm\{CTC\}\}\.\(7\)
All stage\-specific loss weights and optimization hyperparameters are kept consistent with S2ST\-Omni, so that the comparison isolates the effect of the proposed typology\-aware language conditioning\.

### II\-GTTS Backend

The TTS backend converts the target text generated by the S2TT frontend into target speech\. Since the S2TT and TTS modules are connected through an explicit text interface, the proposed framework supports plug\-and\-play integration with different state\-of\-the\-art TTS systems, without retraining or task\-specific coupling of the S2TT frontend\. In our experiments, we evaluate six publicly available recent TTS systems\[[49](https://arxiv.org/html/2605.16026#bib.bib74),[11](https://arxiv.org/html/2605.16026#bib.bib81),[43](https://arxiv.org/html/2605.16026#bib.bib3),[42](https://arxiv.org/html/2605.16026#bib.bib70),[51](https://arxiv.org/html/2605.16026#bib.bib73),[50](https://arxiv.org/html/2605.16026#bib.bib77)\]to verify the flexibility and backend interchangeability of this strategy in practical deployment\.

## IIIExperimental Setup

### III\-ADatasets

TABLE II:Overall performance comparison on CVSS\-C\. Best results are shown in bold\. “\-” indicates results not reported or not applicable\. Ground\-truth ASR\-BLEU and Whisper–Qwen S2TT reference results are included only as references and are not considered for best highlighting\.†denotes a single many\-to\-one model evaluated across the three source\-to\-English directions\. Unmarked systems follow their originally reported evaluation settings\.We conduct experiments on the CVSS\-C corpus\[[22](https://arxiv.org/html/2605.16026#bib.bib2)\], a publicly available multilingual S2ST corpus derived from CoVoST 2\[[41](https://arxiv.org/html/2605.16026#bib.bib43)\]\. CVSS\-C provides parallel speech in multiple source languages paired with synthesized English target speech, enabling standardized evaluation of S2ST systems\. Following prior work\[[12](https://arxiv.org/html/2605.16026#bib.bib65),[32](https://arxiv.org/html/2605.16026#bib.bib94)\], we mainly evaluate French→\\rightarrowEnglish, German→\\rightarrowEnglish, and Spanish→\\rightarrowEnglish\. For the main multilingual setting, a single model is jointly trained on the supervised training sets of these three directions, containing approximately 264 hours for French, 184 hours for German, and 113 hours for Spanish, for a total of 561 hours\.

To further examine robustness under limited supervised data for a typologically distant source language, we additionally evaluate Japanese→\\toEnglish using approximately three hours of supervised CVSS\-C training data\. Both S2ST\-Omni and S2ST\-Omni 2 are trained under the same Japanese setting and evaluated with the same TTS backend and metric pipeline\.

### III\-BImplementation Details

We use Whisper\-Large\-V3\[[37](https://arxiv.org/html/2605.16026#bib.bib33)\]as the frozen speech encoder and Qwen3\-4B\[[43](https://arxiv.org/html/2605.16026#bib.bib3)\]as the LLM decoder\. The hybrid speech adapter follows S2ST\-Omni\[[32](https://arxiv.org/html/2605.16026#bib.bib94)\]and the architecture described in Section[II\-B](https://arxiv.org/html/2605.16026#S2.SS2)\. For LLM adaptation, LoRA is applied to the query and value projection layers with rankr=8r\{=\}8, scaling factorα=32\\alpha\{=\}32, and dropout 0\.1\.

For TI\-HLE, the morphology\-related, reordering, genealogical\-family, and residual channels have dimensions of 64, 64, 64, and 128, respectively, and the concatenated 320\-dimensional vector is fused into a 256\-dimensional language representation\. DG\-LA\-Dual\-CTC operates on the intermediate adapter features𝐇down\\mathbf\{H\}\_\{\\mathrm\{down\}\}withdh=1024d\_\{h\}\{=\}1024; the FiLM generator therefore predicts2​dh=20482d\_\{h\}\{=\}2048affine parameters, and the dynamic frame gate uses a hidden dimension of 256\. The source\- and target\-side CTC branches use SentencePiece tokenizers\[[25](https://arxiv.org/html/2605.16026#bib.bib42)\]with vocabularies of 8k and 4k subword units, respectively\. The CTC weights are set to\(λsrc\(1\),λtgt\(1\)\)=\(0\.1,0\.2\)\(\\lambda^\{\(1\)\}\_\{\\mathrm\{src\}\},\\lambda^\{\(1\)\}\_\{\\mathrm\{tgt\}\}\)=\(0\.1,0\.2\)in Stage I and\(λsrc\(2\),λtgt\(2\)\)=\(0\.01,0\.05\)\(\\lambda^\{\(2\)\}\_\{\\mathrm\{src\}\},\\lambda^\{\(2\)\}\_\{\\mathrm\{tgt\}\}\)=\(0\.01,0\.05\)in Stage II\.

Training follows the progressive fine\-tuning strategy described in Section[II\-F](https://arxiv.org/html/2605.16026#S2.SS6)\. All experiments use an effective batch size of 24, implemented with a per\-device batch size of 3 and gradient accumulation of 8\. We use bf16 mixed precision and conduct all experiments on two NVIDIA A6000 GPUs\.

### III\-CBaselines

We compare S2ST\-Omni 2 with representative S2ST approaches covering E2E, compositional, simultaneous, zero\-shot, and SpeechLLM\-based paradigms\. These include Translatotron 2\[[21](https://arxiv.org/html/2605.16026#bib.bib26)\], UnitY\[[20](https://arxiv.org/html/2605.16026#bib.bib27)\], DASpeech\[[13](https://arxiv.org/html/2605.16026#bib.bib24)\], ComSpeech\[[12](https://arxiv.org/html/2605.16026#bib.bib65)\], StreamSpeech\[[47](https://arxiv.org/html/2605.16026#bib.bib23)\], SimulS2S\-LLM\[[8](https://arxiv.org/html/2605.16026#bib.bib92)\], Hibiki\[[26](https://arxiv.org/html/2605.16026#bib.bib63)\], RosettaSpeech\[[48](https://arxiv.org/html/2605.16026#bib.bib93)\], and the direct baseline S2ST\-Omni\[[32](https://arxiv.org/html/2605.16026#bib.bib94)\]\. In addition, we report a Whisper–Qwen S2TT as text\-level reference, which uses Whisper\-Large\-V3 to transcribe the source speech and Qwen3\-4B to translate the resulting source\-language transcript into English\.

### III\-DEvaluation Metrics

Following prior S2ST works\[[22](https://arxiv.org/html/2605.16026#bib.bib2),[12](https://arxiv.org/html/2605.16026#bib.bib65),[32](https://arxiv.org/html/2605.16026#bib.bib94)\], we evaluate our model from two perspectives: text\-level translation quality and E2E speech translation quality\.

For text\-level translation quality, we report BLEU\[[33](https://arxiv.org/html/2605.16026#bib.bib19)\]and COMET\[[39](https://arxiv.org/html/2605.16026#bib.bib55)\], both computed on the text output of the S2TT frontend against the ground\-truth English references\. BLEU measures surface\-level lexical overlap, while COMET provides a complementary semantic\-level evaluation of translation quality\. The Whisper–Qwen S2TT reference is included only as an auxiliary text\-level reference in Table[II](https://arxiv.org/html/2605.16026#S3.T2); therefore, only BLEU is reported for this reference\. Regarding E2E speech translation quality, we report ASR\-BLEU and BLASER 2\.0\. For ASR\-BLEU, the generated speech is first transcribed by a pretrained wav2vec 2\.0 ASR model111[https://dl\.fbaipublicfiles\.com/fairseq/wav2vec/wav2vec\_vox\_960h\_pl\.pt](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_960h_pl.pt), and BLEU is then computed using SacreBLEU222[https://github\.com/mjpost/sacrebleu](https://github.com/mjpost/sacrebleu)with a fixed configuration333SacreBLEU signature:nrefs:1\|case:mixed\|eff:no\|tok:13a\|smooth:exp\|version:2\.6\.1\.dev1\+gf615c7286\. For BLASER 2\.0, we use the reference\-based configuration\[[47](https://arxiv.org/html/2605.16026#bib.bib23)\], i\.e\., BLASER 2\.0\-*Ref*\. For systems whose outputs are reproduced or publicly available, all scores are computed using the same evaluation pipeline\. For prior systems without publicly available generated outputs, we report the values from the corresponding papers when available\.

## IVResults and Discussion

### IV\-AOverall Performance

Tables[II](https://arxiv.org/html/2605.16026#S3.T2),[IV](https://arxiv.org/html/2605.16026#S4.T4), and[V](https://arxiv.org/html/2605.16026#S4.T5)report the overall results on CVSS\-C in terms of BLEU, ASR\-BLEU, COMET, and BLASER 2\.0\. It is worth noting that S2ST\-Omni 2 is evaluated as a unified many\-to\-one multilingual S2TT frontend shared across the three source\-to\-English directions, rather than relying on separate pair\-specific models for each direction\. This setting is more challenging because a single shared frontend must handle source\-language differences within a unified parameter space\. Among all evaluated S2ST systems, S2ST\-Omni 2 achieves the best average performance across the reported metrics under the adopted evaluation protocol, suggesting that the proposed typology\-aware conditioning improves both text\-level translation quality and E2E S2ST evaluation quality\.

Compared with the direct baseline S2ST\-Omni\[[32](https://arxiv.org/html/2605.16026#bib.bib94)\], S2ST\-Omni 2 improves average BLEU from 35\.67 to 37\.73 and average ASR\-BLEU from 33\.45 to 35\.00, corresponding to relative gains of 5\.8% and 4\.6%, respectively\. It also improves average COMET and BLASER 2\.0 by \+1\.29 and \+0\.10\. Moreover, the largest BLEU and ASR\-BLEU improvements are observed on De→\\toEn, which is consistent with the motivation that German involves stronger compound morphology and clause\-level reordering mismatch with English\. This observation is consistent with the hypothesis that structured typological conditioning is beneficial when translation requires stronger structural mediation\.

TABLE III:Ablation study on CVSS\-C\. Each row removes or replaces one component or one feature group from the full S2ST\-Omni 2\. Subscripts in the average columns denote relative degradation with respect to the full model, while the main values report absolute BLEU and ASR\-BLEU scores\.TABLE IV:COMET results on CVSS\-C\. Scores are reported on a 0–100 scale, with the best results highlighted in bold\.TABLE V:BLASER 2\.0 results on CVSS\-C\. Best results are shown in bold\.To further contextualize frontend translation quality, we compare S2ST\-Omni 2 with the Whisper–Qwen S2TT reference, which follows a cascaded ASR–MT pipeline\. As shown in Table[II](https://arxiv.org/html/2605.16026#S3.T2), S2ST\-Omni 2 improves the average BLEU score from 36\.54 to 37\.73, with gains of \+2\.68 on Fr→\\toEn and \+1\.23 on Es→\\toEn, while remaining slightly lower on De→\\toEn by 0\.37 BLEU\. This comparison is informative because S2ST\-Omni 2 performs many\-to\-one S2TT through a unified multilingual SpeechLLM frontend, instead of decomposing the process into ASR and MT\. The higher average BLEU score suggests that typology\-aware speech representations can provide effective guidance for multilingual S2TT, making the unified many\-to\-one frontend competitive with a strong cascaded text\-level reference under the adopted BLEU evaluation protocol\. In addition, S2ST\-Omni 2 also shows clear advantages compared with previous SOTA S2ST methods\. Relative to RosettaSpeech\[[48](https://arxiv.org/html/2605.16026#bib.bib93)\], a recent strong baseline, S2ST\-Omni 2 improves average BLEU and ASR\-BLEU by \+8\.65 and \+7\.32, while also improving average COMET and BLASER 2\.0 by \+3\.09 and \+0\.14\. Additionally, S2ST\-Omni 2 further outperforms other representative systems in all evaluated metrics, showcasing the effectiveness of the proposed approach\.

Overall, these results indicate that the advantage of S2ST\-Omni 2 comes not merely from the SpeechLLM backbone, but from the structured redesign of source\-language conditioning\. They highlight the importance of how language information is represented and injected in many\-to\-one multilingual S2ST\.

### IV\-BAblation Study

To assess the contribution of each component, we conduct systematic ablation experiments, as summarized in Table[III](https://arxiv.org/html/2605.16026#S4.T3)\. “w/o TI\-HLE” replaces the proposed typology\-informed hierarchical language encoding with a 320\-dimensional flat per\-language embedding, while keeping the remaining S2ST\-Omni 2 components unchanged\. This setting evaluates whether structured typological decomposition provides benefits beyond flat language\-label conditioning with matched input dimensionality\. “w/o DG” replaces the dynamic frame gate with a static scalar gate, and “w/o TA\-Prompt” replaces the proposed typology\-aware prompting with the language\-aware prompting used in S2ST\-Omni, which specifies the source language but does not include explicit typological guidance\. In addition, “w/o Morph,” “w/o Reorder,” “w/o Family,” and “w/o Residual” remove the morphology\-related profile, word\-order/reordering profile, genealogical\-family embedding, and language\-specific residual channel, respectively\.

#### IV\-B1Effect of Dynamically\-Gated Language\-Aware Dual\-CTC

Replacing the dynamic frame gate with a static scalar gate \(w/o DG\) causes consistent degradation in all evaluated metrics\. To elaborate, it reduces average BLEU by 0\.77 and average ASR\-BLEU by 0\.93, showing that adaptive modulation remains beneficial once richer language representations are available\. This suggests that typological priors should not be applied uniformly to all frames; instead, their modulation strength should vary according to both acoustic content and source\-language characteristics\.

#### IV\-B2Effect of Typology\-Aware LLM Prompting

Replacing typology\-aware prompting with the original language\-aware prompting used in S2ST\-Omni \(w/o TA\-Prompt\) decreases average BLEU by 0\.93 and average ASR\-BLEU by 1\.04\. This result suggests that the improvement is not merely due to indicating the source language to the LLM, but to providing explicit typology\-aware translation guidance beyond conventional language\-aware prompting\.

TABLE VI:Representative examples comparing S2ST\-Omni 2 with S2ST\-Omni\. Boldface highlights the critical error span in the translation\.System/TypeExample 1Example 2German→\\rightarrowEnglishSource TextBei viel Regen dehnt sich dasRückhaltebeckenenorm aus\.Als Gegenleistungfür diese militärischen Dienste erhielt er die Stadt Madaba\.ReferenceWhen there is a lot of rain, theretention basinexpands enormously\.As a rewardfor these military services, he received the city Madaba\.S2ST\-OmniThereservoiris stretched a lot when there are many rains\.The city of Madabareceived the townas compensation for this military service\.S2ST\-Omni 2When it rains a lot, theretention basinexpands a lot\.As a rewardfor this military service, he received the city of Madaba\.Spanish→\\rightarrowEnglishSource TextAsí, al partido se le asignaroncinco escañosen el parlamento\.A quien mucho miente, le huye la gente\.ReferenceThus the party was assignedfive seatsin the parliament\.From whom much lies people flee\.S2ST\-OmniThusfive scottish memberswere assigned to the party in parliament\.The more you lie, the peoplerun away from your\.S2ST\-Omni 2Thus the party was assignedfive seatsin the parliament\.Those who lie too much willlose their friends\.French→\\rightarrowEnglishSource TextPouvez\-vous me rendre un petit service ?Elle aurait desvertus médicinales\.ReferenceCan youdo me a small favor?Ithasmedicinal properties\.S2ST\-OmniCan yougive me a little service\.Shewould havemedical aspects\.S2ST\-Omni 2Can youdo me a small favor\.Itwould havemedicinal properties\.
#### IV\-B3Effect of TI\-HLE

Removing TI\-HLE \(w/o TI\-HLE\) yields the largest degradation, reducing average BLEU and ASR\-BLEU by 1\.64 and 1\.32, respectively\. Notably, this variant uses a 320\-dimensional flat per\-language embedding, matching the input dimensionality of the proposed hierarchical representation\. The drop therefore cannot be simply attributed to language\-embedding capacity; rather, it indicates that decomposing language information into typological and residual channels provides a more effective conditioning signal than an unstructured flat embedding\. The fine\-grained ablations further show that each feature group contributes to performance\. Removing the residual and morphology\-related channels causes the largest BLEU drops, by 1\.52 and 1\.50 points, followed by genealogical family \(\-1\.29\) and reordering \(\-1\.13\)\. Similar trends are observed for ASR\-BLEU, with drops of 1\.26, 1\.25, 1\.13, and 1\.06 points, respectively\. These results suggest that the residual channel preserves language\-specific capacity, while the explicit typological channels provide complementary structural priors for multilingual adaptation and translation\-oriented feature modulation\.

#### IV\-B4Summary of Ablation Study

Taken together, these ablation results show that all variants underperform the full S2ST\-Omni 2 model, indicating that the observed gains do not stem from a single isolated component\. Rather, S2ST\-Omni 2 benefits from the complementary effects of representation\-level typological decomposition, acoustic\-level adaptive modulation, and decoder\-side typology\-aware prompting\.

### IV\-CQualitative Analysis

To complement the quantitative results, Table[VI](https://arxiv.org/html/2605.16026#S4.T6)presents representative examples comparing S2ST\-Omni 2 with the direct baseline S2ST\-Omni\. This analysis aims to examine how the proposed typology\-informed structured language conditioning affects translation behavior in linguistically challenging cases, including compound morphology, argument\-structure preservation, non\-literal expressions, and context\-dependent lexical choices\. To be specific, S2ST\-Omni 2 better preserves compound morphology and clause\-level semantic relations for German\. It renders*Rückhaltebecken*as “retention basin,” whereas S2ST\-Omni produces the less specific “reservoir” and an unnatural rendering of the rain condition\. It also preserves the intended argument structure in the Madaba example, while S2ST\-Omni incorrectly suggests that the city received the town\. Regarding Spanish, S2ST\-Omni 2 better handles both structural and non\-literal expressions\. In the passive construction with*se le asignaron*, it correctly preserves the meaning of “five seats,” whereas S2ST\-Omni generates the semantically implausible phrase “five scottish members\.” For the proverb\-like expression, S2ST\-Omni 2 produces a more complete and natural paraphrase, avoiding the incomplete literal rendering generated by S2ST\-Omni\. For French, S2ST\-Omni 2 improves context\-dependent lexical choice and natural English phrasing\. It maps*rendre un petit service*to the natural English collocation “do me a small favor,” while S2ST\-Omni follows a less appropriate word\-by\-word translation\. It also better handles the mismatch between French grammatical gender and English reference by translating*Elle*as “it” rather than “she” in the medicinal\-properties example\.

0100100200200300300400400500500600600222226263030343438384040Rel\. gain over S2ST\-Omni30h:\+15\.1%\+15\.1\\%50h:\+12\.7%\+12\.7\\%100h:\+10\.4%\+10\.4\\%200h:\+7\.6%\+7\.6\\%400h:\+6\.4%\+6\.4\\%561h:\+5\.8%\+5\.8\\%3050561BLEUS2ST\-OmniS2ST\-Omni 2

\(a\)Average
01001002002003003004004005005006006002222262630303434383840403050561S2ST\-OmniS2ST\-Omni 2

\(b\)Fr→\\toEn
01001002002003003004004005005006006002222262630303434383840403050561Training Data Duration \(hours\)BLEUS2ST\-OmniS2ST\-Omni 2

\(c\)De→\\toEn
01001002002003003004004005005006006002222262630303434383840403050561Training Data Duration \(hours\)S2ST\-OmniS2ST\-Omni 2

\(d\)Es→\\toEn

Figure 2:BLEU under varying training data budgets for S2ST\-Omni and S2ST\-Omni 2\. \(a\) Average BLEU, with relative gains computed over S2ST\-Omni\. \(b\) Fr→\\toEn\. \(c\) De→\\toEn\. \(d\) Es→\\toEn\. Across all settings, S2ST\-Omni 2 consistently outperforms S2ST\-Omni, and the relative advantage becomes more pronounced as the training data budget decreases\.Overall, these examples illustrate that the proposed structured language conditioning helps S2ST\-Omni 2 produce more faithful and natural translations across typologically different source languages, further supporting the effectiveness of typology\-informed conditioning strategy\.

### IV\-DFurther Analysis

#### IV\-D1Effect of TTS Backend

Table[VII](https://arxiv.org/html/2605.16026#S4.T7)reports ASR\-BLEU results with six publicly available TTS backends while keeping the S2TT frontend fixed\. The average scores range from 33\.87 to 35\.00, with a 1\.13\-point gap between the weakest and strongest backends\. Although backend\-specific factors such as pronunciation fidelity and prosodic realization still affect ASR\-BLEU, the relative stability across backends suggests that, in terms of ASR\-BLEU, the improvements of S2ST\-Omni 2 are not tied to a specific synthesizer under our evaluation protocol\.

TABLE VII:Effect of various TTS backends on ASR\-BLEU\. All configurations use the same S2ST\-Omni 2 S2TT frontend\.
#### IV\-D2Effect of Training Data Budget

To examine data efficiency, we train both S2ST\-Omni and S2ST\-Omni 2 under data budgets ranging from 30 to 561 hours and compare their average BLEU scores\. As shown in Fig\.[2](https://arxiv.org/html/2605.16026#S4.F2), the advantage of S2ST\-Omni 2 increases monotonically as the amount of training data decreases: the absolute gain grows from \+2\.06 BLEU at 561 hours to \+3\.93 BLEU at 30 hours, while the relative improvement correspondingly increases from 5\.8% to approximately 15\.1%\. This trend suggests that typology\-aware conditioning is particularly beneficial under limited supervision\. By explicitly encoding morphology, reordering, and genealogical relatedness, S2ST\-Omni 2 provides structured language priors that can support parameter sharing and data\-efficient multilingual adaptation when training data are scarce\. In higher\-resource settings, part of these regularities may be learned directly from data, leading to a more moderate gain\. In lower\-resource settings, however, the explicit typological priors provide additional guidance for acoustic conditioning and downstream decoding, resulting in a larger advantage over the flat language\-label baseline\.

#### IV\-D3Japanese Extension under Limited Supervision

TABLE VIII:Japanese→\\toEnglish translation results with∼\{\\sim\}3 hours of CVSS\-C supervised training data\.We further evaluate S2ST\-Omni 2 on Japanese→\\rightarrowEnglish translation using only approximately three hours of supervised training data\. As shown in Table[VIII](https://arxiv.org/html/2605.16026#S4.T8), S2ST\-Omni 2 consistently outperforms S2ST\-Omni across all four metrics, improving BLEU by \+2\.39, ASR\-BLEU by \+2\.34, COMET by \+2\.02, and BLASER 2\.0 by \+0\.087\. This result suggests that the proposed typology\-aware conditioning remains beneficial beyond the three European source languages considered in the main CVSS\-C evaluation\. This setting is particularly informative because Japanese is typologically distant from French, German, and Spanish in morphology, reordering profile, and genealogy, while also being evaluated under a highly limited data budget\. The observed gains provide additional evidence that structured typological priors can offer useful guidance under low\-resource and typologically divergent conditions\.

#### IV\-D4Summary of Further Analysis

Overall, these analyses provide complementary evidence for the practical behavior of S2ST\-Omni 2\. The TTS\-backend comparison suggests that ASR\-BLEU gains are not tied to a specific synthesizer, the data\-budget analysis shows that the relative advantage of typology\-aware conditioning increases as supervision decreases, and the Japanese extension further indicates that this advantage can extend to a typologically distant low\-resource source language\. These findings suggest that structured typological priors can serve as useful inductive biases when multilingual S2ST models cannot fully infer language\-specific regularities from abundant supervised data\.

## VConclusion

In this paper, we presented S2ST\-Omni 2, a typology\-aware compositional many\-to\-one S2ST framework that reformulates multilingual language conditioning for SpeechLLM\-based S2ST\. Instead of treating source languages as isolated flat labels, S2ST\-Omni 2 introduces structured typological priors through typology\-informed hierarchical language encoding, dynamically\-gated language\-aware Dual\-CTC, and typology\-aware LLM prompting, thereby enhancing language conditioning at the representation, acoustic\-modulation, and decoding levels\. Extensive experiments on CVSS\-C demonstrate that S2ST\-Omni 2 achieves strong average performance across BLEU, ASR\-BLEU, COMET, and BLASER 2\.0 under the adopted evaluation protocol\. Ablation studies and qualitative analyses further indicate that the proposed components provide complementary benefits\. In addition, the controlled data\-budget analysis and the few\-hour Japanese\-to\-English experiment provide complementary evidence that typology\-informed conditioning is particularly useful under reduced\-supervision conditions, covering both lower data budgets within the main multilingual setting and a typologically distant low\-resource extension\. Overall, these findings highlight structured typological conditioning as a practical and effective inductive bias for data\-efficient multilingual S2ST\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1)\.
- \[2\]\(2023\)Unifying cross\-lingual transfer across scenarios of resource scarcity\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 3980–3995\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1)\.
- \[3\]J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.\(2023\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1)\.
- \[4\]L\. Barrault, Y\. Chung, M\. C\. Meglioli, D\. Dale, N\. Dong, P\. Duquenne, H\. Elsahar, H\. Gong, K\. Heffernan, J\. Hoffman,et al\.\(2023\)SeamlessM4T: massively multilingual & multimodal machine translation\.arXiv preprint arXiv:2308\.11596\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1)\.
- \[5\]S\. Chen, Y\. Feng, L\. He, T\. He, W\. He, Y\. Hu, B\. Lin, Y\. Lin, Y\. Pan, P\. Tan,et al\.\(2024\)Takin: a cohort of superior quality zero\-shot speech generation models\.arXiv preprint arXiv:2409\.12139\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1)\.
- \[6\]B\. Comrie\(1988\)Linguistic typology\.Annual Review of Anthropology17,pp\. 145–159\.Cited by:[§II\-C1](https://arxiv.org/html/2605.16026#S2.SS3.SSS1.p1.1)\.
- \[7\]B\. Comrie\(1989\)Language universals and linguistic typology: syntax and morphology\.University of Chicago press\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1),[§II\-C](https://arxiv.org/html/2605.16026#S2.SS3.p1.1)\.
- \[8\]K\. Deng, W\. Chen, X\. Chen, and P\. Woodland\(2025\)SimulS2S\-llm: unlocking simultaneous inference of speech llms for speech\-to\-speech translation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 16718–16734\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1),[§III\-C](https://arxiv.org/html/2605.16026#S3.SS3.p1.1),[TABLE II](https://arxiv.org/html/2605.16026#S3.T2.8.6.15.9.1)\.
- \[9\]M\. S\. Dryer\(2013\)Order of subject, object and verb\.InThe World Atlas of Language Structures Online,M\. S\. Dryer and M\. Haspelmath \(Eds\.\),External Links:[Link](https://wals.info/chapter/81)Cited by:[§II\-C1](https://arxiv.org/html/2605.16026#S2.SS3.SSS1.p2.1)\.
- \[10\]Z\. Du, Q\. Chen, S\. Zhang, K\. Hu, H\. Lu, Y\. Yang, H\. Hu, S\. Zheng, Y\. Gu, Z\. Ma,et al\.\(2024\)Cosyvoice: a scalable multilingual zero\-shot text\-to\-speech synthesizer based on supervised semantic tokens\.arXiv preprint arXiv:2407\.05407\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1)\.
- \[11\]Z\. Du, C\. Gao, Y\. Wang, F\. Yu, T\. Zhao, H\. Wang, X\. Lv, H\. Wang, C\. Ni, X\. Shi,et al\.\(2025\)Cosyvoice 3: towards in\-the\-wild speech generation via scaling\-up and post\-training\.arXiv preprint arXiv:2505\.17589\.Cited by:[§II\-G](https://arxiv.org/html/2605.16026#S2.SS7.p1.1),[TABLE VII](https://arxiv.org/html/2605.16026#S4.T7.3.3.5.2.1)\.
- \[12\]Q\. Fang, S\. Zhang, Z\. Ma, M\. Zhang, and Y\. Feng\(2024\)Can we achieve high\-quality direct speech\-to\-speech translation without parallel speech data?\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7264–7277\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1),[§III\-A](https://arxiv.org/html/2605.16026#S3.SS1.p1.3),[§III\-C](https://arxiv.org/html/2605.16026#S3.SS3.p1.1),[§III\-D](https://arxiv.org/html/2605.16026#S3.SS4.p1.1),[TABLE II](https://arxiv.org/html/2605.16026#S3.T2.8.6.13.7.1),[TABLE IV](https://arxiv.org/html/2605.16026#S4.T4.3.4.1.1),[TABLE V](https://arxiv.org/html/2605.16026#S4.T5.3.6.3.1)\.
- \[13\]Q\. Fang, Y\. Zhou, and Y\. Feng\(2023\)Daspeech: directed acyclic transformer for fast and high\-quality speech\-to\-speech translation\.Advances in Neural Information Processing Systems36,pp\. 72604–72623\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1),[§III\-C](https://arxiv.org/html/2605.16026#S3.SS3.p1.1),[TABLE II](https://arxiv.org/html/2605.16026#S3.T2.8.6.11.5.1)\.
- \[14\]A\. Graves, S\. Fernández, F\. Gomez, and J\. Schmidhuber\(2006\)Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks\.InICML,Cited by:[§II\-D3](https://arxiv.org/html/2605.16026#S2.SS4.SSS3.p1.2)\.
- \[15\]A\. Gulati, J\. Qin, C\. Chiu, N\. Parmar, Y\. Zhang, J\. Yu, W\. Han, S\. Wang, Z\. Zhang, Y\. Wu, and R\. Pang\(2020\)Conformer: Convolution\-augmented Transformer for Speech Recognition\.InInterspeech 2020,pp\. 5036–5040\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2020-3015),ISSN 2958\-1796Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1)\.
- \[16\]M\. Haspelmath and A\. Sims\(2013\)Understanding morphology\.Routledge\.Cited by:[§II\-C1](https://arxiv.org/html/2605.16026#S2.SS3.SSS1.p1.1)\.
- \[17\]J\. A\. Hawkins\(2014\)Word order universals\.Vol\.3,Elsevier\.Cited by:[§II\-C1](https://arxiv.org/html/2605.16026#S2.SS3.SSS1.p2.1)\.
- \[18\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§II\-F](https://arxiv.org/html/2605.16026#S2.SS6.p3.1)\.
- \[19\]R\. Huang, M\. Li, D\. Yang, J\. Shi, X\. Chang, Z\. Ye, Y\. Wu, Z\. Hong, J\. Huang, J\. Liu,et al\.\(2024\)Audiogpt: understanding and generating speech, music, sound, and talking head\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 23802–23804\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1)\.
- \[20\]H\. Inaguma, S\. Popuri, I\. Kulikov, P\. Chen, C\. Wang, Y\. Chung, Y\. Tang, A\. Lee, S\. Watanabe, and J\. Pino\(2023\)Unity: two\-pass direct speech\-to\-speech translation with discrete units\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15655–15680\.Cited by:[§III\-C](https://arxiv.org/html/2605.16026#S3.SS3.p1.1),[TABLE II](https://arxiv.org/html/2605.16026#S3.T2.8.6.12.6.1),[TABLE V](https://arxiv.org/html/2605.16026#S4.T5.3.4.1.1)\.
- \[21\]Y\. Jia, M\. T\. Ramanovich, T\. Remez, and R\. Pomerantz\(2022\)Translatotron 2: high\-quality direct speech\-to\-speech translation with voice preservation\.InInternational Conference on Machine Learning,pp\. 10120–10134\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1),[§III\-C](https://arxiv.org/html/2605.16026#S3.SS3.p1.1),[TABLE II](https://arxiv.org/html/2605.16026#S3.T2.8.6.10.4.1),[TABLE V](https://arxiv.org/html/2605.16026#S4.T5.3.5.2.1)\.
- \[22\]Y\. Jia, M\. T\. Ramanovich, Q\. Wang, and H\. Zen\(2022\)CVSS corpus and massively multilingual speech\-to\-speech translation\.InProceedings of the thirteenth language resources and evaluation conference,pp\. 6691–6703\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p4.1),[§III\-A](https://arxiv.org/html/2605.16026#S3.SS1.p1.3),[§III\-D](https://arxiv.org/html/2605.16026#S3.SS4.p1.1)\.
- \[23\]Y\. Jia, R\. J\. Weiss, F\. Biadsy, W\. Macherey, M\. Johnson, Z\. Chen, and Y\. Wu\(2019\)Direct Speech\-to\-Speech Translation with a Sequence\-to\-Sequence Model\.InInterspeech 2019,pp\. 1123–1127\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2019-1951),ISSN 2958\-1796Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1)\.
- \[24\]G\. Kikui, E\. Sumita, T\. Takezawa, and S\. Yamamoto\(2003\)Creating corpora for speech\-to\-speech translation\.\.InINTERSPEECH,pp\. 381–384\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p1.1)\.
- \[25\]T\. Kudo and J\. Richardson\(2018\)SentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing\.InProceedings of the 2018 conference on empirical methods in natural language processing: System demonstrations,pp\. 66–71\.Cited by:[§III\-B](https://arxiv.org/html/2605.16026#S3.SS2.p2.5)\.
- \[26\]T\. Labiausse, L\. Mazaré, E\. Grave, A\. Défossez, and N\. Zeghidour\(2025\)High\-fidelity simultaneous speech\-to\-speech translation\.InProceedings of the 42nd International Conference on Machine Learning,pp\. 32116–32129\.Cited by:[§III\-C](https://arxiv.org/html/2605.16026#S3.SS3.p1.1),[TABLE II](https://arxiv.org/html/2605.16026#S3.T2.8.6.16.10.1)\.
- \[27\]A\. Lee, H\. Gong, P\. Duquenne, H\. Schwenk, P\. Chen, C\. Wang, S\. Popuri, Y\. Adi, J\. Pino, J\. Gu,et al\.\(2022\)Textless speech\-to\-speech translation on real data\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 860–872\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p1.1)\.
- \[28\]S\. Lim, T\. Yun, J\. Kim, J\. Choi, and T\. Kim\(2024\)Analysis of multi\-source language training in cross\-lingual transfer\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,pp\. 712–725\.Cited by:[§II\-C1](https://arxiv.org/html/2605.16026#S2.SS3.SSS1.p3.1)\.
- \[29\]P\. Littell, D\. R\. Mortensen, K\. Lin, K\. Kairis, C\. Turner, and L\. Levin\(2017\)URIEL and lang2vec: representing languages as typological, geographical, and phylogenetic vectors\.InProceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers,pp\. 8–14\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1),[§II\-C1](https://arxiv.org/html/2605.16026#S2.SS3.SSS1.p3.1),[§II\-C](https://arxiv.org/html/2605.16026#S2.SS3.p1.1)\.
- \[30\]Y\. Moslem, R\. Haque, J\. Kelleher, and A\. Way\(2023\)Adaptive machine translation with large language models\.InProceedings of the 24th Annual Conference of the European Association for Machine Translation,pp\. 227–237\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1)\.
- \[31\]A\. Oncevay, B\. Haddow, and A\. Birch\(2020\)Bridging linguistic typology and multilingual machine translation with multi\-view language representations\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 2391–2406\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1),[§II\-C](https://arxiv.org/html/2605.16026#S2.SS3.p1.1)\.
- \[32\]Y\. Pan, X\. Wu, Y\. Yang, J\. Yao, C\. Maxime, L\. Ma, and J\. Zhao\(2025\)S2ST\-omni: hierarchical language\-aware speechllm adaptation for multilingual speech\-to\-speech translation\.arXiv preprint arXiv:2506\.11160\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1),[§I](https://arxiv.org/html/2605.16026#S1.p3.1),[§I](https://arxiv.org/html/2605.16026#S1.p5.1),[§II\-A](https://arxiv.org/html/2605.16026#S2.SS1.p1.1),[§II\-B](https://arxiv.org/html/2605.16026#S2.SS2.p1.7),[§II\-D3](https://arxiv.org/html/2605.16026#S2.SS4.SSS3.p1.2),[§II\-F](https://arxiv.org/html/2605.16026#S2.SS6.p1.1),[§III\-A](https://arxiv.org/html/2605.16026#S3.SS1.p1.3),[§III\-B](https://arxiv.org/html/2605.16026#S3.SS2.p1.2),[§III\-C](https://arxiv.org/html/2605.16026#S3.SS3.p1.1),[§III\-D](https://arxiv.org/html/2605.16026#S3.SS4.p1.1),[TABLE II](https://arxiv.org/html/2605.16026#S3.T2.7.5.5.1),[§IV\-A](https://arxiv.org/html/2605.16026#S4.SS1.p2.1),[TABLE IV](https://arxiv.org/html/2605.16026#S4.T4.3.7.4.1),[TABLE V](https://arxiv.org/html/2605.16026#S4.T5.3.9.6.1)\.
- \[33\]K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu\(2002\)Bleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th annual meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[§III\-D](https://arxiv.org/html/2605.16026#S3.SS4.p2.1)\.
- \[34\]K\. Peng, L\. Ding, Q\. Zhong, L\. Shen, X\. Liu, M\. Zhang, Y\. Ouyang, and D\. Tao\(2023\)Towards making the most of chatgpt for machine translation\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 5622–5633\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1)\.
- \[35\]E\. Perez, F\. Strub, H\. De Vries, V\. Dumoulin, and A\. Courville\(2018\)FiLM: visual reasoning with a general conditioning layer\.InAAAI,Cited by:[§II\-D1](https://arxiv.org/html/2605.16026#S2.SS4.SSS1.p1.1)\.
- \[36\]E\. M\. Ponti, H\. O’horan, Y\. Berzak, I\. Vulić, R\. Reichart, T\. Poibeau, E\. Shutova, and A\. Korhonen\(2019\)Modeling language variation and universals: a survey on typological linguistics for natural language processing\.Computational Linguistics45\(3\),pp\. 559–601\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1),[§II\-C1](https://arxiv.org/html/2605.16026#S2.SS3.SSS1.p3.1),[§II\-C](https://arxiv.org/html/2605.16026#S2.SS3.p1.1)\.
- \[37\]A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever\(2023\)Robust speech recognition via large\-scale weak supervision\.InInternational conference on machine learning,pp\. 28492–28518\.Cited by:[§II\-A](https://arxiv.org/html/2605.16026#S2.SS1.p1.1),[§III\-B](https://arxiv.org/html/2605.16026#S3.SS2.p1.2)\.
- \[38\]S\. Rajaee and C\. Monz\(2024\)Analyzing the evaluation of cross\-lingual knowledge transfer in multilingual language models\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2895–2914\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1)\.
- \[39\]R\. Rei, J\. G\. De Souza, D\. Alves, C\. Zerva, A\. C\. Farinha, T\. Glushkova, A\. Lavie, L\. Coheur, and A\. F\. Martins\(2022\)COMET\-22: unbabel\-ist 2022 submission for the metrics shared task\.InProceedings of the Seventh Conference on Machine Translation \(WMT\),pp\. 578–585\.Cited by:[§III\-D](https://arxiv.org/html/2605.16026#S3.SS4.p2.1)\.
- \[40\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1)\.
- \[41\]C\. Wang, A\. Wu, J\. Gu, and J\. Pino\(2021\)CoVoST 2 and Massively Multilingual Speech Translation\.InInterspeech 2021,pp\. 2247–2251\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-2027),ISSN 2958\-1796Cited by:[§III\-A](https://arxiv.org/html/2605.16026#S3.SS1.p1.3)\.
- \[42\]K\. Xie, F\. Shen, J\. Li, F\. Xie, X\. Tang, and Y\. Hu\(2025\)Fireredtts\-2: towards long conversational speech generation for podcast and chatbot\.arXiv preprint arXiv:2509\.02020\.Cited by:[§II\-G](https://arxiv.org/html/2605.16026#S2.SS7.p1.1),[TABLE VII](https://arxiv.org/html/2605.16026#S4.T7.3.3.7.4.1)\.
- \[43\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§II\-A](https://arxiv.org/html/2605.16026#S2.SS1.p1.1),[§II\-G](https://arxiv.org/html/2605.16026#S2.SS7.p1.1),[§III\-B](https://arxiv.org/html/2605.16026#S3.SS2.p1.2),[TABLE VII](https://arxiv.org/html/2605.16026#S4.T7.3.3.6.3.1)\.
- \[44\]Y\. Yang, Y\. Pan, J\. Yin, J\. Han, L\. Ma, and H\. Lu\(2023\)Hybridformer: improving squeezeformer with hybrid attention and nsr mechanism\.InICASSP 2023\-2023 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p2.1)\.
- \[45\]J\. Yao, Y\. Yuguang, Y\. Pan, Z\. Ning, J\. Ye, H\. Zhou, and L\. Xie\(2025\)Stablevc: style controllable zero\-shot voice conversion with conditional flow matching\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 25669–25677\.Cited by:[§II\-D1](https://arxiv.org/html/2605.16026#S2.SS4.SSS1.p1.1)\.
- \[46\]D\. Zhanget al\.\(2023\)SpeechGPT: empowering large language models with intrinsic cross\-modal conversational abilities\.arXiv preprint arXiv:2305\.11000\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1)\.
- \[47\]S\. Zhang, Q\. Fang, S\. Guo, Z\. Ma, M\. Zhang, and Y\. Feng\(2024\)Streamspeech: simultaneous speech\-to\-speech translation with multi\-task learning\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8964–8986\.Cited by:[§III\-C](https://arxiv.org/html/2605.16026#S3.SS3.p1.1),[§III\-D](https://arxiv.org/html/2605.16026#S3.SS4.p2.1),[TABLE II](https://arxiv.org/html/2605.16026#S3.T2.8.6.14.8.1),[TABLE IV](https://arxiv.org/html/2605.16026#S4.T4.3.5.2.1),[TABLE V](https://arxiv.org/html/2605.16026#S4.T5.3.7.4.1)\.
- \[48\]Z\. Zheng, X\. Sun, T\. Dinh, A\. Yanamandra, A\. Jain, Z\. Liu, S\. Hadap, V\. Bhat, M\. Aggarwal, G\. Medioni,et al\.\(2025\)RosettaSpeech: zero\-shot speech\-to\-speech translation from monolingual data\.arXiv preprint arXiv:2511\.20974\.Cited by:[§I](https://arxiv.org/html/2605.16026#S1.p3.1),[§III\-C](https://arxiv.org/html/2605.16026#S3.SS3.p1.1),[TABLE II](https://arxiv.org/html/2605.16026#S3.T2.6.4.4.1),[§IV\-A](https://arxiv.org/html/2605.16026#S4.SS1.p3.3),[TABLE IV](https://arxiv.org/html/2605.16026#S4.T4.3.6.3.1),[TABLE V](https://arxiv.org/html/2605.16026#S4.T5.3.8.5.1)\.
- \[49\]S\. Zhou, Y\. Zhou, Y\. He, X\. Zhou, J\. Wang, W\. Deng, and J\. Shu\(2025\)IndexTTS2: a breakthrough in emotionally expressive and duration\-controlled auto\-regressive zero\-shot text\-to\-speech\.arXiv preprint arXiv:2506\.21619\.Cited by:[§II\-G](https://arxiv.org/html/2605.16026#S2.SS7.p1.1),[TABLE VII](https://arxiv.org/html/2605.16026#S4.T7.3.3.4.1.1)\.
- \[50\]Y\. Zhou, G\. Zeng, X\. Liu, X\. Li, R\. Yu, Z\. Wang, R\. Ye, W\. Sun, J\. Gui, K\. Li,et al\.\(2025\)VoxCPM: tokenizer\-free tts for context\-aware speech generation and true\-to\-life voice cloning\.arXiv preprint arXiv:2509\.24650\.Cited by:[§II\-G](https://arxiv.org/html/2605.16026#S2.SS7.p1.1),[TABLE VII](https://arxiv.org/html/2605.16026#S4.T7.3.3.9.6.1)\.
- \[51\]H\. Zhu, W\. Kang, Z\. Yao, L\. Guo, F\. Kuang, Z\. Li, W\. Zhuang, L\. Lin, and D\. Povey\(2025\)ZipVoice: fast and high\-quality zero\-shot text\-to\-speech with flow matching\.arXiv preprint arXiv:2506\.13053\.Cited by:[§II\-G](https://arxiv.org/html/2605.16026#S2.SS7.p1.1),[TABLE VII](https://arxiv.org/html/2605.16026#S4.T7.3.3.8.5.1)\.