NAVER LABS Europe Submission to the Instruction-following 2026 Short Track

# NAVER LABS Europe Submission to the Instruction-following 2026 Short Track
Source: [https://arxiv.org/html/2607.01960](https://arxiv.org/html/2607.01960)
\\affiliations

1NAVER LABS Europe, France 2IIIT Delhi, India\\correspondingauthormarcely\.zanon\-boito@naverlabs\.com\\teaserfig![[Uncaptioned image]](https://arxiv.org/html/2607.01960v1/training_pipeline.png)\\teasercaptionOur training pipeline\. A speech projector \(SpeechMapper\) \(A\) and text LoRA adapters \(B\) are trained in parallel using ASR and text\-to\-text data, respectively\. These modules are then integrated during a brief multimodal adaptation step \(C\)\.

###### Abstract

In this paper, we describe NAVER LABS Europe’s submission to the instruction\-following speech processing short track at IWSLT 2026\. We participate again in the constrained setting, developing systems capable of jointly performing ASR, ST, and SQA from English speech into Chinese, Italian, and German\. Building on our previous submission, ranked first in last year’s short track, we update our multi\-stage training pipeline by replacing the speech projector withSpeechMapper, a method for learning a speech\-to\-LLM embedding projector using only ASR data\. In addition, we introduce a synthetic SQA dataset,fakACL, composed of artificially generated scientific presentations\. This dataset is built by prompting the LLM backbone, segmenting the generated talks, and synthesizing speech withSeamlessM4T\-v2\-large\. The combination of an improved speech projection mechanism and domain\-specific synthetic data allows our model to outperform last year’s best short\-track system, while being considerably more compact and relying on a weaker LLM backbone\. This year’s results place our system tied for first place in the overall short track ranking\.

\\contributions

## 1Introduction

Fueled by the progress of text\-only LLMs, a growing number of speech\-based assistants have recently been proposed, addressing both semantic\[rubenstein2023audiopalm,tang2023salmonn,defossez2024moshi,hu2024wavllm,ambilduke2025tower,speechmapper\]and acoustic speech tasks\[huang2024audiogpt,nguyen2024spirit,xu2025qwen3\]\. In this context, the second edition of theIWSLT Instruction\-following Speech Processing Track\[adelani\-etal\-2026\-iwslt\]provides a common benchmark for developing speech assistants capable of performing semantic tasks directly from speech\. The challenge proposes to leverage LLMs and speech foundation models \(SFMs\) to build systems that can perform multilingual tasks from English speech input, guided by textual multilingual instructions\.

NAVER LABS Europe \(NLE\) again participates in the constrained setting of theshort track, which includes automatic speech recognition \(ASR\), speech translation \(ST\), multilingual spoken question answering \(SQA\), and a surprise instruction\-following task revealed only at test time\. The target languages for ST and SQA are Chinese, Italian, and German\. The target languages for the surprise task were German and Chinese\. Participants are allowed to use theSeamlessM4T\-v2\-large\[barrault2023seamlessm4t\]speech backbone and theQwen3\-4B\-Instruct\-2507\[qwen3technicalreport\]LLM for both training and data generation\. Building on our previous winning submission\[lee\-etal\-2025\-naver\], we adopt a similar multi\-stage training pipeline to develop a speech LLM capable of following multilingual instructions\.

Our approach trains two components in parallel: \(1\) a speech\-to\-text projector that maps averaged speech representations from theSeamlessM4T\-v2\-largeencoder into the embedding space of a frozen LLM, and \(2\) text\-only LoRA adapters\[hu2022lora\]applied to the same frozen LLM\. These components are subsequently combined through a \(3\) short SFT stage on multimodal multilingual data\. Compared to last year, we introduce two main improvements\. First, we replace the speech projector with an updated version ofSpeechMapper\[speechmapper\], which enables learning a speech\-to\-embedding mapping without requiring LLM forward passes, substantially lowering computational and hardware requirements\. Second, we construct a synthetic scientific dataset,fakACL, to reduce the domain gap between training and evaluation data\.

This system paper is organized as follows\. Section[2](https://arxiv.org/html/2607.01960#S2)describes the preprocessing applied to the data used in this challenge\. Section[3](https://arxiv.org/html/2607.01960#S3)presents the updatedSpeechMappermodel we leverage in this work\. Sections[4](https://arxiv.org/html/2607.01960#S4)and[5](https://arxiv.org/html/2607.01960#S5)describe our training pipeline and experimental settings, respectively\. Section[6](https://arxiv.org/html/2607.01960#S6)presents our experiments and discussion\. Section[7](https://arxiv.org/html/2607.01960#S7)presents the submitted system\. Section[8](https://arxiv.org/html/2607.01960#S8)concludes the paper\.

## 2Data

We leverage all available data from the constrained setting for training, including CoVoST2\[wang2020covost\], EuroParlST\[jairsan2020a\], GigaST\[ye23b\_interspeech\], and LibriSQA\[librisqa\]\. This data is also synthetically augmented to cover additional target languages viaSeamlessM4T\-v2\-largeMT\. In addition, we introducefakACL, a synthetic dataset of scientific presentations generated by prompting the LLM backbone and synthesizing the outputs withSeamlessM4T\-v2\-largeTTS\. The validation sets of EuroParlST, CoVoST2, and LibriSQA are used for model validation, while MCIF\[papi2026mcif\]is used for final model selection\.

We now present our data preprocessing \(Section[2\.1](https://arxiv.org/html/2607.01960#S2.SS1)\) and prompt format \(Section[2\.2](https://arxiv.org/html/2607.01960#S2.SS2)\)\.

### 2\.1Data Preprocessing

We create both speech\-to\-text and text\-to\-text instructions to train our systems\. Appendix Table[5](https://arxiv.org/html/2607.01960#A1.T5)lists our splits and number of samples\. In all cases where synthetic translation is generated from English text, this procedure is done viaSeamlessM4T\-v2\-largeMT followed by filtering of the result using reference\-free COMET111Unbabel/wmt22\-cometkiwi\-da\[rei\-etal\-2022\-comet\]to filter out all translations that did not score at least 0\.85\. Below we detail dataset\-specific preprocessing\.

##### GigaST, CoVoST2 and EuroParlST

CoVoST2 and GigaST cover English to German and Simplified Chinese language directions\. EuroParlST covers English to German and Italian\. ASR splits for CoVoST2 and EuroParlST datasets were built by merging the existing language splits and de\-duplicating the audio files\. For all, language\-specific ST and MT splits were created by aligning translations to English speech and reference transcriptions, respectively\. For GigaST, and because our LLM backbone is particularly weak in Italian \(see Table[1](https://arxiv.org/html/2607.01960#S6.T1)\), we produce additional synthetic Italian translations\.

##### LibriSQA

LibriSQA is built on the 300\-hour split of LibriSpeech\[panayotov2015librispeech\]\. The dataset is divided inPartIandPartII, which share the same speech content, but differ in the format of the questions and answers\.PartIconsists of direct questions to be the answered by the system, whilePartIIpresents multiple\-choice questions \(MCQ\) corresponding to those inPartI\. We performed the following modifications to this dataset:

- •MCQ answer expansion:ForPartII, answers are provided as the letter corresponding to the correct option\. This results in a very limited number of tokens contributing to the loss, which we found to be suboptimal in past experiments\. To address this, we expand the target answers to include the full option text \(e\.g\., instead of predicting “A”, the model is trained to produce “A <option A text\>”\)\.
- •Multilingual SQA/QA:We translate the English text into the three target languages\. ForPartI, we retain only question\-answer pairs whosebothtranslations exceed the quality threshold\. ForPartII, we translate only the questions and their associated answer options, since the correct answer is implicitly defined by the options\. We then apply a parsing step to discard examples where the A–B–C–D structure is altered or lost during translation\. This parser also recovers the correct answer by aligning the translated options with the original references\.
- •MT split:We create an MT split for each target language by exploiting the alignment between questions and answers obtained during the creation of our SQA split\. We do not create a corresponding ST split, as we consider this dataset to be significantly out\-of\-domain for the challenge, particularly in terms of acoustic conditions, and including more of it could be detrimental during training\.
- •Similarity\-based Invalid splits:Some questions in LibriSQA are relatively broad \(e\.g\., “How did the person in the text act?”\), making it difficult to reliably mismatch them with unrelated audio segments\. To address this, we construct mismatched pairs by selecting question–transcript combinations with low semantic similarity\. We first shuffle and randomly pair transcripts and questions, then compute sentence\-level representations using a Sentence Transformer222sentence\-transformers/multi\-qa\-MiniLM\-L6\-cos\-v1to measure cosine similarity between them\. We retain only pairs with low similarity scores\.333Based on manual assessment, we set this to≤0\.1\\leq 0\.1\.This process yields 48,346 and 58,682 mismatched pairs forPartIandPartII, respectively\.
- •Qwen\-based Invalid splits:To address potential noise introduced by our similarity\-based invalid splits we created, we include a more general\-purpose set of invalid questions by promptingQwen3\-4B\-Instruct\-2507\(see Appendix[A\.1](https://arxiv.org/html/2607.01960#A1.SS1)for details\)\. After post\-processing, this yields 62,040 English questions, which are then translated and filtered for quality using the same procedure described above\. Finally, these questions are randomly paired with speech audio segments to construct the invalid splits\.

##### fakACL

In both this and the previous edition of the challenge, the validation data is drawn from scientific paper presentations at ACL conferences\. Motivated by this, we constructed a synthetic dataset targeting this domain to reduce the mismatch between training and test conditions, and to provide additional relevant QA data, as the constrained SQA dataset is relatively distant from the test setting\. We generated fakACL ASR/ST/SQA data by first producing short presentation scripts for NLP papers usingQwen3\-4B\-Instruct\-2507\. These scripts were segmented into chunks of two to three sentences and synthesized into speech usingSeamlessM4T\-v2\-largeTTS\. Each segment transcript was then fed back into the LLM to generate two questions answerable from the corresponding content\. Appendix[A\.2](https://arxiv.org/html/2607.01960#A1.SS2)provides full details of the dataset creation process\.

### 2\.2Prompt Format

The goal of the short track of this challenge is to produce a model that is able to 1\) transcribe English speech; 2\) translate English speech into Italian, German and Chinese; 3\) Answer multilingual questions using English speech as input\. In this setting, the language of the question must match the language of the answer\.

Last year we designed an unified prompt with consistent structure: regardless of the task \(ASR, ST, or SQA\), the user turn begins by encapsulating the speech embeddings within textual tags\. This is followed on a new line by a task\-specific instruction formulated as a question in the target language, and finally, another line containing a common suffix\. Our last year results\[lee\-etal\-2025\-naver\]highlighted that by forcing the instruction \(i\.e\. question\) to be in the target language, we were able to reduce confusion between the tasks of ASR/ST, while also helping the system to generalize to languages it was not originally trained for\.444For instance, in the SpeechMapper paper\[speechmapper\], our IWSLT25 system generalizes to Spanish and French ST via multilingual prompting, despite the fact that its LoRA weights were never trained on these languages\.Therefore, this year we only experiment with this setting, in which the instruction is given in the target language\.

We additionally include a template for multiple choice questions \(MCQ\), for the surprise task, and for zero\-shot generation \(LLM backbone andSpeechMapperin projector\-only mode\)\. The list of updated templates used is available in Appendix Table[7](https://arxiv.org/html/2607.01960#A1.T7)\.

## 3Speech Projector:SpeechMapper

In this work we use an improved version of the originalSpeechMapperprojector\[speechmapper\]\. SpeechMapper is a speech\-to\-LLM embedding projector for semantic information, and training it only requires an LLM’s tokenizer and its embedding layer\. The benefits are twofold: \(1\) the size of LLM does not increase GPU memory required for training;555The only increase in computational cost comes from the embedding dimensionality, which is typically not proportional to the LLM size, keeping the approach computationally efficient\.\(2\) it mitigates prompt overfitting\. We refer the reader to the original paper for further details\.

The decision to replace last year’s simple transformer based projector with the more parameter\-heavySpeechMapperis primarily motivated by practical considerations\. In preliminary experiments, we attempted to reproduce the pipeline fromlee\-etal\-2025\-naver, but observed very poor performance for projector\-only models \(exceeding 200% WER\)\. We attribute this low\-performance compared to last year to the lower dimensionality of the embedding space in this year’s LLM\. We hypothesize that projecting into a smaller embedding space requires higher precision, and therefore greater model capacity, whichSpeechMapperis better able to provide\.

Below, we describe our main modification toSpeechMapperrelative tospeechmapper: the training objective\.

### 3\.1NewSpeechMapperTraining Objective

Our updated version ofSpeechMapperis trained using four loss functions designed to align speech to LLM input embeddings\. Speech is encoded into a sequence of embeddingsZs∈ℝT×dZ\_\{s\}\\in\\mathbb\{R\}^\{T\\times d\}using the frozen SFM and fed to theSpeechMapper\. The corresponding target sentence is processed to a sequence of LLM token embeddingsZt∈ℝT′×dZ\_\{t\}\\in\\mathbb\{R\}^\{T^\{\\prime\}\\times d\}using the frozen LLM embedding layer\. The resulting speech sequence lengthTTis significantly larger than the text sequence lengthT′T^\{\\prime\}\. To mitigate this length mismatch, we pad the target embedding sequence with the LLMs\[PAD\]token embedding to match the speech sequence length \(Eq\.[1](https://arxiv.org/html/2607.01960#S3.E1)\)\.

Zt=\[zt\(1\),…,zt\(T′\),zpad,…,zpad\]∈ℝT×dZ\_\{t\}=\[z\_\{t\}^\{\(1\)\},\\dots,z\_\{t\}^\{\(T^\{\\prime\}\)\},z\_\{\\text\{pad\}\},\\dots,z\_\{\\text\{pad\}\}\]\\in\\mathbb\{R\}^\{T\\times d\}\(1\)
Therefore, the output ofSpeechMapperis semantic embeddings followed by pad embeddings, implicitly capturing sequence length\. We optimize the following objectives\.

##### L1 Alignment Loss

The element wise L1 distance between speech and text embeddings explicitly enforce feature\-level alignment between the two modalities \(Eq\.[2](https://arxiv.org/html/2607.01960#S3.E2)\)\. This loss replaces the MSE loss fromspeechmapper\.

ℒL1=1T​d​∑t=1T‖zs\(t\)−zt\(t\)‖\\mathcal\{L\}\_\{\\text\{L1\}\}=\\frac\{1\}\{Td\}\\sum\_\{t=1\}^\{T\}\\left\\lVert z\_\{s\}^\{\(t\)\}\-z\_\{t\}^\{\(t\)\}\\right\\rVert\(2\)

##### Cosine Similarity Loss

To encourage angular alignment in the embedding space, we keep the original cosine similarity loss from the original paper, and minimize the cosine distance between corresponding speech and text representations \(Eq\.[3](https://arxiv.org/html/2607.01960#S3.E3)\)\.

ℒcos=1T​∑t=1T\(1−zs\(t\)⋅zt\(t\)∥zs\(t\)∥2​∥zt\(t\)∥2\)\\mathcal\{L\}\_\{\\text\{cos\}\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\left\(1\-\\frac\{z\_\{s\}^\{\(t\)\}\\cdot z\_\{t\}^\{\(t\)\}\}\{\\lVert z\_\{s\}^\{\(t\)\}\\rVert\_\{2\}\\lVert z\_\{t\}^\{\(t\)\}\\rVert\_\{2\}\}\\right\)\(3\)

##### Softmax Contrastive Loss

To enforce separation from non target tokens, we use softmax function to maximize similarity to the positive class embeddings and minimize similarity to all other embeddings\. LetE∈ℝV×dE\\in\\mathbb\{R\}^\{V\\times d\}denotes the LLM’s embedding layer, whereVVis the vocabulary size\. For each speech representationzs\(t\)z\_\{s\}^\{\(t\)\}, we compute cosine similarity scores with all embeddings and use these as logits, withst∈ℝVs\_\{t\}\\in\\mathbb\{R\}^\{V\}andyt∈ℝVy\_\{t\}\\in\\mathbb\{R\}^\{V\}denoting respectively the logits and the one\-hot vector corresponding to the target token at positiontt\(Eq\.[4](https://arxiv.org/html/2607.01960#S3.E4)\)\. The negative log likelihood encouragesSpeechMapperoutput embeddings to be close to its correct token embedding while remaining well separated from all other tokens in the vocabulary\.

ℒsoftmax=−1T​∑t=1Tyt⊤​log⁡\(softmax​\(st\)\)\\mathcal\{L\}\_\{\\text\{softmax\}\}=\-\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}y\_\{t\}^\{\\top\}\\log\(\\text\{softmax\}\(s\_\{t\}\)\)\(4\)

##### Connectionist Temporal Classification \(CTC\)

We found that adding CTC\[graves2006ctc\]loss \(ℒctc\\mathcal\{L\}\_\{\\text\{ctc\}\}\) at an intermediate layer helps in stabilizingSpeechMapper’s training and output embeddings quality\.

##### Final Objective

The overall training objective is a weighted sum of the losses:

ℒ=ℒL1\+ℒcos\+0\.1∗ℒsoftmax\+ℒctc\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{L1\}\}\+\\mathcal\{L\}\_\{\\text\{cos\}\}\+0\.1\*\\mathcal\{L\}\_\{\\text\{softmax\}\}\+\\mathcal\{L\}\_\{\\text\{ctc\}\}\(5\)

## 4Training Pipeline

Our training pipeline is illustrated in FigureNAVER LABS Europe Submission to the Instruction\-following 2026 Short Track\. We first train two components in parallel: \(A\) a SpeechMapper speech\-to\-embedding projector using ASR data, and \(B\) text LoRA weights using MT and QA data\. These components are subsequently reloaded and jointly adapted on a mixture of speech and text tasks \(C\)\. In this section, we describe the key components of this training pipeline\.

##### Foundation Models

For speech, we leverageSeamlessM4T\-v2\-largemodel, extracting speech representations for all our audio data from its 24th speech encoder layer \(i\.e\. the last layer\)\. Prior to training, we average every two consecutive frame vectors, reducing significantly the sequence length\. We highlight a minor difference from last year’s setup: we average two consecutive frame vectors instead of three\. Prior experiments withSpeechMappermotivated this choice, indicating that a lighter compression before the projector’s CNN layers yields better performance\. All our models are built on top of a frozenQwen3\-4B\-Instruct\-2507\.

##### SpeechMapper Settings

OurSpeechMapperfollows the architecture described inspeechmapper, comprising two consecutive blocks, each consisting of a CNN,Nself\-attention layers, and a feed\-forward projection to a higher dimensional space\. To accommodate a CTC head at the end of the first block, we modify the convolutional strides from the original setting of 2 in both blocks to 1 and 4 for the first and second blocks, respectively\. We train the model withN=6N=6, using initial, intermediate, and output dimensionalities of10241024,20482048, and25602560, respectively\.

##### LoRA Adapters

LoRA adaptation\[hu2022lora\]is applied to both the self\-attention \(Q/K values, output projection\) and feed\-forward modules, and across all LLM layers\. We user​a​n​k=8,α=16rank=8,\\alpha=16\. We do not use dropout\.

##### Data Sampling Strategy

For training the models in FigureNAVER LABS Europe Submission to the Instruction\-following 2026 Short Track\(B\) and \(C\), we define an epoch asXXsteps across the dataset, whereX=\|speech​\_​examples\|batch​\_​sizeX=\\frac\{\|\\text\{speech\}\\\_\\text\{examples\}\|\}\{\\text\{batch\}\\\_\\text\{size\}\}\. To construct the data for each epoch, batches are sampled by first applying the predefined task\-level sampling ratios, followed by sampling according to domain\-level splits within each task\. In the multimodal training setting \(speech and text tasks mixed\), we consider speech as ourmainmodality, using it for defining epoch size and task sampling ratio\. Whenever a sampled speech task and language pair has a textual counterpart \(e\.g\., ST corresponds to MT; SQA to QA\), we also sample a batch from the corresponding textual task\. In practice, this results in alternating batches, such that a sampled ST en\-de batch is followed by an MT en\-de batch\. We find that incorporating textual data in this manner consistently improves the final model performance\.

## 5Experimental Settings

##### Codebase

We train our models using an internal fork oftorchtune\[torchtune\], which allows us to process interleaved representations of text and high\-dimensional vectors within the user turn during instruction tuning\. We also implement our updated version ofSpeechMapperon this codebase\. For multimodal training, the high\-dimensional vectors pass throughSpeechMapper, while the text prefix and suffix user prompts are processed by the LLM embedding layer\. The obtained speech and text embeddings are both concatenated and fed into the first layer of the LLM which is trained on the masked input with standard cross\-entropy loss\. Different learning rate schedulers and optimizers are employed forSpeechMapperand the LoRA weights, allowing for more controlled and effective training of these distinct model components\.

##### Inference Settings

We perform inference usingtorchtune, with a batch size of 1, greedy decoding, and with the maximum number of new tokens limited to 100\. This decoding strategy was consistently applied across all experimental settings\.

##### Evaluation Metrics

We evaluate our models on speech \(ASR, ST, SQA\) and text \(MT, QA\) tasks when relevant\. For ASR, we score word error rate \(WER\) using HuggingFace evaluate library with default settings and MMS normalization\[pratap2024scaling\]\. For ST/MT we score COMET\[rei\-etal\-2022\-comet\]\.666Unbabel/wmt22\-comet\-daFor SQA/QA, we use LLM\-as\-a\-judge evaluation scripts from thebergenlibrary777[https://github\.com/naver/bergen](https://github.com/naver/bergen)\[rau\-etal\-2024\-bergen\]\. We use their “yes/no” quality assessment evaluation format including the reference question and answer, and generated output\. We report average accuracy across three LLMs:EuroLLM\-9B\-Instruct\[eurollm\],Gemma3\-12B\-Instruct, andGemma3\-27B\-Instruct\[gemma3\]\.

##### Baselines

We compare our results with both backbones we use for training\. We evaluate MT using the reference transcripts andQwen3\-4B\-Instruct\-2507in zero\-shot settings, and we evaluateSeamlessM4T\-v2\-largefor ASR, ST and MT\. Additionally, we present results for last year’s best short track system, referring to it asBEST\-IWSLT25\-IF\[lee\-etal\-2025\-naver\]\.

## 6Experiments

ModelASR \(WER\)ST/MT \(COMET\)SQA/QA \(LLM\-as\-judge\)en\-deen\-iten\-zhPart IPart IIText\-only Models \(zero\-shot LLMs, SFM and LoRA\)SeamlessM4T\-v2\-large\(MT\)\-81\.486\.780\.8\-\-Llama\-3\.1\-8B\-Instruct\-81\.984\.177\.086\.670\.8Qwen3\-4B\-Instruct\-2507\-71\.067\.774\.389\.170\.2Qwen3\-4B\-Instruct\-2507\+ LoRA \(B\)\-80\.786\.984\.789\.982\.6SFM and Projector\-only ModelSeamlessM4T\-v2\-large\(ASR/ST\)5\.978\.376\.978\.0\-\-SpeechMapper\(A\)14\.273\.580\.179\.784\.472\.1Multimodally Trained ModelsBEST\-IWSLT25\-IF7\.377\.384\.280\.282\.063\.0SpeechMapper\+ LoRA \(C\) setup 18\.275\.183\.280\.686\.282\.5SpeechMapper\+ LoRA \(C\) setup 27\.476\.384\.481\.387\.980\.2

Table 1:Results for ASR \(LibriSpeech clean/other, EuroParlST, CoVoST2\), ST/MT \(CoVoST2, EuroParlST\), and SQA/QA \(LibriSQA PartI and PartII\)\. ASR and ST/MT scores are reported as weighted averages across datasets, proportional to the number of samples, while SQA/QA results are averaged across judges and reported as accuracy\.Table 2:Averaged MCIF scores for all speech models\.We now present our results for ASR, ST, and SQA\. Section[6\.1](https://arxiv.org/html/2607.01960#S6.SS1)introduces the models used in our experiments, followed by results and discussion in Section[6\.2](https://arxiv.org/html/2607.01960#S6.SS2)\.

### 6\.1Our Models

##### SpeechMapper

We trainSpeechMapperon ASR data from CoVoST2, EuroParlST, GigaST, and LibriSQA, without applying any up\-sampling\. Training is performed for 500k steps using AdamW with a learning rate of1​e−41e\-4and 50k warm\-up steps\. We employ dynamic batching with gradient accumulation set to 2\. AsQwen3\-4B\-Instruct\-2507does not provide a dedicated padding token, we use the reserved \(untrained\) token151664151664for padding\. The model is trained on4×4\\timesA100\-80GB GPUs for approximately two days\.

##### B\. Text\-only LoRA \(MT/QA\)

The LoRA weights \(B\) are trained on all available real and synthetic data from Appendix Table[5](https://arxiv.org/html/2607.01960#A1.T5), with task\-level sampling ratios of 0\.6 and 0\.4 for MT and SQA, respectively\. For MT, language sampling ratios are set to 0\.4/0\.3/0\.3 for de/it/zh, while for QA they are 0\.2/0\.3/0\.3/0\.2 for en/de/it/zh\. We train for 30k steps using AdamW with learning rate of3​e−43e\-4, weight decay of0\.10\.1, and100100warm\-up steps\. Batch size of1616, and gradient accumulation of88is used\. This model trains for approximately 4 days in a single A100\-80GB\.

##### C\. Multimodal \(A \+ B\)

We restart training by reloading both modules described above\. We explore various combinations of the available datasets and obtain the best performance with the configuration detailed in Appendix Table[6](https://arxiv.org/html/2607.01960#A1.T6)\. We use task\-level sampling ratios of 0\.3/0\.4/0\.3 for ASR, ST, and SQA, respectively\. For ST/MT, language sampling ratios are set to 0\.4/0\.4/0\.2 for de/it/zh\. For SQA/QA, we use uniform language sampling over the four languages, assigning a ratio of 0\.2 each for the valid set and 0\.05 each for the invalid set\. We use learning rate of5​e−55e\-5for theSpeechMapper, and of1​e−51e\-5for the LoRA weights \(setup 1\) or5​e−55e\-5for the LoRA weights \(setup 2\)\. We use a batch size of 8, and gradient accumulation of 6\. This model trains for 3K steps, approximately 2 hours in a single A100\-80GB\.

### 6\.2Results and Discussion

Table[1](https://arxiv.org/html/2607.01960#S6.T1)present results for ASR, ST/MT and SQA/QA for all of our models and baselines over the available test splits\. For ASR, we evaluate using EuroParl, CoVoST and Librispeech clean/other\. For ST/MT, we evaluate on EuroParl and CoVoST, excluding GigaST which we find to be particularly noisy\. SQA/QA results cover only English, and are an average over scores obtained for the different LLM\-as\-judge models\. Table[2](https://arxiv.org/html/2607.01960#S6.T2)present MCIF scores \(ASR, ST and multilingual SQA\) obtained for the speech models\. Below we discuss our main findings\.

##### MT Results

Looking at the MT results for text\-only models \(top portion of Table[1](https://arxiv.org/html/2607.01960#S6.T1)\), we observe that this year’s backbone performs substantially worse thanSeamlessM4T\-v2\-largeandLlama\-3\.1\-8B\-Instruct\. This is expected, as the backbone has roughly half the parameters of last year’s LLM\. Results are particularly poor for Italian, motivating us to up\-sample this language during fine\-tuning\. Finally, we also present results for our best LoRA setting, which considerably improves performance across both MT and QA tasks\.

##### Projector\-only Results

We present results for our bestSpeechMapperconfiguration trained to produceQwen3\-4B\-Instruct\-2507embeddings\. Overall, this setting proves challenging: the LLM appears particularly sensitive to noise in the input embeddings, especially compared to larger backbones tested inspeechmapper\. Our bestSpeechMappersetup performs correctly in in\-domain settings \(Table[1](https://arxiv.org/html/2607.01960#S6.T1)\), but we find it to under\-perform considerably in the MCIF dataset \(Table[2](https://arxiv.org/html/2607.01960#S6.T2)\), where it handles poorly named entities\. In addition, zero\-shot usage ofQwen3\-4B\-Instruct\-2507is difficult, as the model reorganizes and/or rephrases the transcription text in many instances, or responds in an overly verbose manner, both cases negatively affect evaluation scores\. Qualitative examples illustrating these behaviors are provided in Appendix Table[8](https://arxiv.org/html/2607.01960#A1.T8)\.

##### Multimodal Models

We experiment with a wide range of dataset combinations and sampling strategies, and report results only for our best\-performing multimodal models \(bottom portion of Table[1](https://arxiv.org/html/2607.01960#S6.T1)and Table[2](https://arxiv.org/html/2607.01960#S6.T2)\)\. These models are able to match the performance of last year’s system, despite relying on a smaller LLM backbone\. Overall, we find that improving performance jointly on ASR and SQA is more challenging this year, as gains in one task often come at the expense of the other\. This trade\-off is illustrated by Setups 1 and 2: while Setup 1 achieves better results on LibriSQA Part II \(MCQ\) and MCIF’s SQA, Setup 2 yields stronger performance on ASR and ST\.

## 7Submitted Model

We select our model based on its MCIF performance\. Overall, we were unable to produce a single run that achieves the best score across all tasks, as improvements in one task tend to degrade performance in others\. Therefore, we prioritize SQA as our primary task and use it as the key criterion for model selection\. Based on Table[2](https://arxiv.org/html/2607.01960#S6.T2), we submit setup 1 as our main submission\. We also resubmit BEST\-IWSLT\-IF as a contrastive, unconstrained system\. Our competition results are discussed in Appendix Section[A\.3](https://arxiv.org/html/2607.01960#A1.SS3)\.

## 8Conclusion

In this paper, we presented NLE’s submission to the instruction\-following speech processing short track at IWSLT 2026 under the constrained setting\. We developed multimodal models capable of jointly performing ASR, ST, and SQA from English speech into Chinese, German, and Italian\. Building on our previous pipeline\[lee\-etal\-2025\-naver\], we replaced the transformer\-based speech projector with an updatedSpeechMapperprojector\[speechmapper\]\. We also introduced a synthetic dataset of scientific talks,fakACL, to mitigate domain mismatch between training and evaluation\. Despite relying on a substantially smaller LLM backbone, our final system outperforms last year’s best submission in the short track\.

## References

## Appendix AAppendix

### A\.1Qwen\-based SQA Invalid Split

We aim to generate high\-quality invalid questions for training our speech LLM\. To ensure this quality, the generated questions must be unrelated to the speech content\. Since LibriSQA is an audiobook\-based dataset, we therefore focus on creating questions grounded in broad conversational topics\.

We first promptedQwen3\-4B\-Instruct\-2507to produce a large set of conversation topics using:

> “List about 1000 conversation topics, without numbering or adding any comment of explanation\.”

After post\-processing the output, we retained 413 topics\. Next, for each one, we prompted the LLM with:

> “Generate 30 words related toTOPIC\. Produce a comma\-separated list of words, without any explanation\.”

Then, using the resulting list, we finally prompted the model with:

> “Ask 25 different questions aboutWORD\. Do not repeat the question, include no comments, and output only the questions, one per line\.”

The output was automatically filtered to remove sentences that did not end with a question mark, resulting in 62,040 English questions\. Some examples are shown below:

- •What role do animals play in traditional rites?
- •How does still air interact with thermal radiation?
- •Can a revoked loan be reapproved?
- •What is the role of the saw blade in determining cut quality?
- •How do social media platforms influence the perception of friendship and connection?
- •What is the significance of putrefaction in forensic science?

### A\.2fakACL Creation

Our goal in creatingfakACLis to mimic ACL60\-60\[salesky\-etal\-2023\-evaluating\]and MCIF\. These datasets consist of oral presentations that typically begin with the authors introducing themselves and stating the title of the paper, followed by an overview of the main findings\.

Our dataset creation process consists of three stages: \(1\) script creation, \(2\) segmentation and speech synthesis, and \(3\) QA generation\. We now describe these stages\.

#### A\.2\.1Script Creation

The goal of this stage of the dataset creation process is to generate high\-quality presentation scripts on NLP\-related topics\. We started the process by first promptingQwen3\-4B\-Instruct\-2507to:

> “List 60 sub\-fields related to ACL/NLP conferences, one subfield par line\.”

This resulted in a list of 56 ACL conference topics, with entries such asHealthcare NLP,Legal document analysisandEducation\. We do not check for duplicated conference topics\.

Next, we generated a collection of possible paper titles for each one of the topics\. We prompted the LLM with:

> “List 40 paper titles from ACL conferences related to:SUB\-FIELD\. Output only the paper titles, one per line\. Nothing else\.”

The result was a collection of 2,527 paper titles\. Examples of generated paper titles were “Phonological and Morphological Development in Multilingual Children” and “Emotion\-Driven Text\-to\-Speech with Multimodal Inputs”\.

Finally, we prompted the LLM with:

> “Write a 12 sentences oral script for presenting the ACL paper entitled:TITLE\.”

After manual processing, this resulted in 2,497 scripts\. An example of a generated script is given at Table[3](https://arxiv.org/html/2607.01960#A1.T3)\.

Table 3:Example of a synthetic presentation script and QA produced byQwen3\-4B\-Instruct\-2507\.
#### A\.2\.2Segmentation and Speech Synthesis

After the scripts are created, the next step consists of segmenting the content into blocks to be synthesized bySeamlessM4T\-v2\-large\. This stage includes preprocessing steps such as number and symbol normalization \(e\.g\., “30%” becomes “thirty percent”\), as well as text normalization\.

We use the Python libraryspaCyto split the scripts into sentences, normalize the text, and discard sentences containing fewer than 10 characters or fewer than 6 words\. This process results in 21,400 sentences\. We then synthesize the sentences usingSeamlessM4T\-v2\-largeTTS with random speakers\. The average utterance length is 8\.6 seconds\.

#### A\.2\.3QA Generation

Lastly, after obtaining the segmented and normalized scripts, our next goal was to generate valid question–answer pairs\. For this stage, we discarded the first three and the last two sentences of each script, as these tended to be overly generic and usually corresponded to greetings or acknowledgments\.

For each text segment, we promptedQwen3\-4B\-Instruct\-2507with:

> “Generate 2 pairs of question and very short answer\. Generate the question in one line, the answer in the next line, prefix the question by “Q:” and the answer by “A:”\. Generate strictly and solely based on the content:CONTENT\.”

We automatically filtered the outputs based on the question–answer prefixes and subsequently performed manual verification to remove clear hallucinations\. This process yielded a final set of 38,968 questions\.

### A\.3IWLST 2026 IF Short Results

Table[4](https://arxiv.org/html/2607.01960#A1.T4)presents the official results for our primary \(Setup 1 from Table[1](https://arxiv.org/html/2607.01960#S6.T1)\) and contrastive \(BEST\-IWSLT25\-IF\) submissions\. Overall, the results confirm that we were able to build a stronger system than last year’s submission, despite relying on a smaller and weaker LLM backbone\.

Our primary submission outperforms BEST\-IWSLT25\-IF across all languages for ST, while lagging by only 0\.2% WER on ASR\. For QA, this year’s model again improves performance across all languages except Italian, for which we found the backbone to be particularly weak\.

Finally, the inclusion of MCQ training allowed our model to perform the surprise task of Quality Estimation \(QE\-accuracy\) while respecting the proposed zero\-shot prompt \(QE\-format\-accuracy\)\. In contrast, last year’s solution struggled to follow the required format and often generated answers instead of selecting options, making it incapable of correctly answering cases where the correct option did not correspond to the language of the prompt\.

Table 4:Official results for our systems submitted to IWSLT 2026 IF short track\.Table 5:Training sets statistics by task\. For ST/MT sets, target side is duplicated\. \* denotes synthetic text obtained viaSeamlessM4T\-v2\-largeMT;†\{\\dagger\}indicates splits generated withSeamlessM4T\-v2\-largeTTS\. For fakACL, all English textual content is produced viaQwen3\-4B\-Instruct\-2507prompting\.Table 6:List of datasets and splits used for training our presented models\. Statistics are presented in Table[5](https://arxiv.org/html/2607.01960#A1.T5)\.Table 7:The user turn prompt template used for training and/or evaluating models\. For speech tasks, the user prompt is given bySpeech Prefix\+Task\+Suffix, for textual tasks, the user prompt is given byText Prefix\+Task\+Suffix\.ST/MT instructionswere obtained by translating the instruction “Can you translate the Speech content into\[German/Italian/Chinese\]text?” and “Answer only with the translation\. Do not add anything else to your answer\.” to corresponding target languages usingSeamlessM4T\-v2\-large\.SQA/QA instructionsare eitherfree format\(direct answer\) ormultiple\-choice questions\(MCQ\)\. In the case of MCQ, options are included in the same line of the question, and both are translated to the target language in case of multilingual SQA\.Surprise task instructionsare included after replacing the “\\n” by spaces\.Table 8:Some examples of generations forQwen3\-4B\-Instruct\-2507usingSpeechMapper\(A\) embeddings\. In the top, an example of hallucination related to unknown named entities at MCIF; in the middle two examples of ASR rephrasing with EuroParlST; and in the bottom an example of extremely verbose SQA output for LibriSQA\.