面向多语言自动语音识别的智能体式数据采样框架：基于语言学感知的错误驱动优化

# Agentic Data Sampling Framework for Multilingual ASR via Linguistic-Aware Error-Driven Optimization
Source: [https://arxiv.org/html/2610.36538](https://arxiv.org/html/2610.36538)
Zhiheng Qian,††thanks:Equal contribution\. $ˆ†$Corresponding author\.Junhao DuAffiliation:AISpeech Co\., Ltd\.Jing PengAffiliation:Shanghai Jiao Tong UniversityHanqi LiAffiliation:Shanghai Jiao Tong UniversityKunyang PengAffiliation:AISpeech Co\., Ltd\.Feng XueAffiliation:AISpeech Co\., Ltd\.Lu ChenAffiliation:Shanghai Jiao Tong UniversityShuai WangAffiliation:Nanjing UniversityE\-mail[\{n1vnhil,jing\.peng,daqige,chenlusz,kai\.yu\}@sjtu\.edu\.cn](mailto:{n1vnhil,jing.peng,daqige,chenlusz,kai.yu}@sjtu.edu.cn)Kai YuE\-mail[dujunhao0114@gmail\.com](mailto:[email protected])E\-mail[\{kunyang\.peng,feng\.xue\}@aispeech\.com](mailto:{kunyang.peng,feng.xue}@aispeech.com)E\-mail[shuaiwang@nju\.edu\.cn](mailto:[email protected])Affiliation:Shanghai Jiao Tong University

###### Abstract

Multilingual automatic speech recognition \(ASR\) has achieved rapid progress with large\-scale pretrained models and multilingual speech corpora\. Previous studies have proved the pivotal role of data sampling in multilingual ASR performance\. However, existing sampling strategies largely rely on time\-consuming manual, heuristic\-based designs\. To address this, we propose a linguistic\-aware agentic data sampling framework that introduces a data sampling LLM agent into the multilingual ASR training loop\. The agent is driven by a linguistically grounded rubric that analyzes ASR errors from phonological, morphological, and syntactic perspectives, and uses the resulting feedback together with training statistics, budget constraints, and algorithm profiles to select domain\-level, sample\-level, and token\-level sampling strategies\. This design turns evaluation results into adaptive data\-selection decisions rather than treating WER/CER as final endpoints only\. We conduct experiments on twelve Common Voice languages using Whisper\-medium as the ASR backbone\. The results show that our method achieves performance better than full\-data training while using less training data and lower computational resources, and substantially accelerates training convergence compared to conventional sampling strategies\.

###### Keywords:

Multilingual ASR Agentic Data Sampling Linguistic Error Analysis

## 1Introduction

Multilingual automatic speech recognition \(ASR\) aims to recognize multiple languages within a unified model, reducing the cost of developing and maintaining language\-specific systems\. Large\-scale joint training has demonstrated the value of cross\-lingual parameter sharing, particularly for low\-resource languages\[[21](https://arxiv.org/html/2610.36538#bib.bib1)\]\. Cross\-lingual self\-supervised pretraining, weakly supervised sequence\-to\-sequence learning, and massively multilingual scaling have further extended language coverage and transferability\[[2](https://arxiv.org/html/2610.36538#bib.bib2),[23](https://arxiv.org/html/2610.36538#bib.bib3),[22](https://arxiv.org/html/2610.36538#bib.bib4)\]\. Despite this progress, recognition performance remains uneven across languages because multilingual corpora differ substantially in data availability, acoustic conditions, pronunciation systems, morphology, and writing conventions\.

A central challenge is therefore how to use heterogeneous multilingual training data effectively under a limited budget\. Existing approaches typically concatenate corpora, balance languages, follow predefined curricula, or select auxiliary data according to corpus relatedness, historical feedback, or acoustic and phonetic similarity\[[13](https://arxiv.org/html/2610.36538#bib.bib5),[31](https://arxiv.org/html/2610.36538#bib.bib6),[10](https://arxiv.org/html/2610.36538#bib.bib7),[25](https://arxiv.org/html/2610.36538#bib.bib8)\]\. These methods demonstrate that multilingual data are not equally useful\. However, most optimize a predefined criterion at a single granularity, such as language weighting or utterance selection\. They do not jointly adapt language allocation, utterance selection, and token supervision to the model’s evolving error profile\.

Current evaluation practice provides limited support for such adaptation\. Multilingual benchmarks commonly report average and per\-language WER/CER\[[26](https://arxiv.org/html/2610.36538#bib.bib9)\]\. Although essential for system comparison, these metrics are affected by tokenization, writing systems, morphology, and normalization\[[29](https://arxiv.org/html/2610.36538#bib.bib10),[18](https://arxiv.org/html/2610.36538#bib.bib11)\], and do not identify the linguistic source of an error\. Typological studies show that phonological, morphological, and syntactic properties help explain cross\-language performance differences\[[9](https://arxiv.org/html/2610.36538#bib.bib12)\]\. Nevertheless, such analyses are generally performed after training and are rarely converted into decisions about what data the model should learn from next\. This leaves a gap between multilingual ASR error diagnosis and data optimization\.

In this work, we propose a linguistic\-aware agentic data sampling framework that closes this loop\. After each training round, recognition outputs are analyzed through a linguistic rubric covering phonology, morphology, and syntax\. An LLM agent combines these diagnostics with per\-language performance, training history, budget constraints, and algorithm profiles\. It then produces an executable sampling plan consisting of domain\-, sample\-, and token\-level algorithm choices and their parameters for the next round\. Data sampling is thus formulated as an adaptive, feedback\-driven process rather than a fixed preprocessing strategy\.

Our contributions are threefold\. First, we formulate multilingual ASR data sampling as a closed\-loop optimization problem in which evaluation results directly guide subsequent training\-data selection\. Second, we provide a linguistic error rubric and a hierarchical agentic sampler that coordinate language allocation, utterance selection, and token\-level supervision according to the current model state\. Third, extensive experiments on twelve Common Voice languages show that the proposed method outperforms both no\-sampling and random\-sampling baselines in average multilingual recognition performance under the same cumulative training budget, while achieving the best results on most evaluated languages\. Further analyses demonstrate improved training convergence, consistent reductions in phonological, morphological, and function\-word errors, and adaptive changes in the agent’s decisions as the model error profile evolves\.

## 2Related Works

### 2\.1Evaluation and Error Diagnosis in Multilingual ASR

Multilingual ASR evaluation has developed from testing individual languages on separate corpora toward broader and more standardized cross\-lingual benchmarks\. Common Voice provides diverse community\-contributed speech across many resource conditions, while FLEURS enables more controlled comparison through parallel content across languages\[[1](https://arxiv.org/html/2610.36538#bib.bib13),[6](https://arxiv.org/html/2610.36538#bib.bib14)\]\. XTREME\-S and the ML\-SUPERB series further evaluate multilingual speech representations across languages, datasets, adaptation settings, and related speech tasks\[[26](https://arxiv.org/html/2610.36538#bib.bib9),[5](https://arxiv.org/html/2610.36538#bib.bib15),[27](https://arxiv.org/html/2610.36538#bib.bib16)\]\. Recent extensions additionally consider language varieties, accents, dialects, code\-switching, and long\-form recognition\[[4](https://arxiv.org/html/2610.36538#bib.bib17),[33](https://arxiv.org/html/2610.36538#bib.bib18),[28](https://arxiv.org/html/2610.36538#bib.bib19)\]\. These benchmarks have made per\-language performance and cross\-lingual robustness increasingly visible, rather than relying only on a single macro\-average score\.

Despite this broader coverage, multilingual ASR is still evaluated primarily with WER and CER\. These metrics provide reproducible measures of transcription accuracy, but their interpretation varies with writing systems, tokenization conventions, word boundaries, and morphological structure\. CER can offer greater consistency than WER for some multilingual comparisons, but neither metric identifies the linguistic source of an error\[[29](https://arxiv.org/html/2610.36538#bib.bib10)\]\. Reported results are also sensitive to evaluation design: language\-insensitive normalization may remove meaningful orthographic information, while annotation and language\-label inconsistencies in multilingual corpora can distort the measured performance\[[18](https://arxiv.org/html/2610.36538#bib.bib11),[12](https://arxiv.org/html/2610.36538#bib.bib20)\]\.

This limitation is important because similar WER/CER values may reflect different model weaknesses\. Errors may arise from phone confusions and phonotactic patterns, missing inflectional markers, or incorrectly recognized function words\. Typologically informed analyses have shown that phonological, morphological, and syntactic properties help explain cross\-language performance variation in multilingual ASR systems\[[9](https://arxiv.org/html/2610.36538#bib.bib12)\]\. Such analyses therefore offer a more informative view of model behavior than aggregate edit distance alone\.

However, existing evaluation and diagnostic studies are mainly used for benchmarking or post\-hoc interpretation\. They reveal which languages or linguistic properties are difficult, but usually do not determine how the training data should be changed in response\. This leaves a gap between error diagnosis and model optimization\. Our work addresses this gap by converting phonological, morphological, and syntactic error statistics into structured feedback for subsequent domain\-, sample\-, and token\-level data selection\. In this way, multilingual evaluation is used not only to measure the final model, but also to guide the next training round\.

### 2\.2Data Sampling Algorithms

Data sampling and selection improve training efficiency by assigning different importance to data sources, examples, or supervision units\. At the domain level, DoReMi learns mixtures of heterogeneous data sources using proxy\-model excess losses and distributionally robust optimization\[[32](https://arxiv.org/html/2610.36538#bib.bib21)\]\. DoGE estimates the contribution of each domain to target generalization through bilevel optimization and gradient\-based signals\[[8](https://arxiv.org/html/2610.36538#bib.bib22)\], while exponentiated\-gradient reweighting provides a general mechanism for converting domain scores into adaptive sampling distributions\[[17](https://arxiv.org/html/2610.36538#bib.bib23)\]\. These methods show that a fixed data mixture may become suboptimal as training progresses\.

Finer\-grained methods operate on individual examples or tokens\. LESS uses low\-dimensional gradient representations to select examples that align with a target or validation objective\[[30](https://arxiv.org/html/2610.36538#bib.bib24)\]\. RHO\-1 prioritizes under\-learned tokens according to their excess loss relative to a reference model\[[16](https://arxiv.org/html/2610.36538#bib.bib25)\], whereas TOSS uses token\-level risk signals to suppress undesirable supervision\[[14](https://arxiv.org/html/2610.36538#bib.bib26)\]\. Although these methods were mainly developed for language\-model training, they establish complementary mechanisms for domain reweighting, example selection, and token\-level control\.

Multilingual ASR introduces additional selection criteria because the value of speech data depends on both linguistic and acoustic relationships\. Corpus\-relatedness sampling selects auxiliary speech according to source–target similarity\[[13](https://arxiv.org/html/2610.36538#bib.bib5)\], while adversarial meta\-sampling adapts language selection using historical model feedback\[[31](https://arxiv.org/html/2610.36538#bib.bib6)\]\. Other approaches use phoneme overlap and pronunciation information for dynamic multilingual sampling\[[10](https://arxiv.org/html/2610.36538#bib.bib7)\], or estimate positive cross\-lingual transfer through acoustic pseudo\-tokens and clip\-level acoustic similarity\[[25](https://arxiv.org/html/2610.36538#bib.bib8),[19](https://arxiv.org/html/2610.36538#bib.bib27)\]\. These findings show that data quantity or loss alone cannot fully determine the usefulness of multilingual speech\.

Beyond multilingual ASR, recent studies have investigated the use of LLMs and agentic mechanisms for data selection and preparation\. AskLLM leverages the zero\-shot reasoning capability of an instruction\-tuned LLM to directly assess the quality of individual training examples, demonstrating that LLM\-based semantic judgments can serve as effective data\-selection signals\[[24](https://arxiv.org/html/2610.36538#bib.bib28)\]\. Bai et al\. introduce a multi\-actor collaborative framework in which multiple data\-selection strategies independently prioritize samples and a central console dynamically combines their decisions according to the current training state\[[3](https://arxiv.org/html/2610.36538#bib.bib29)\]\. Moving beyond sample scoring, DataFlow introduces an LLM\-driven agent that translates natural\-language data requirements into executable preparation pipelines composed of generation, evaluation, filtering, and refinement operators\[[15](https://arxiv.org/html/2610.36538#bib.bib30)\]\.

These agentic data sampling method has not been adapted to multilingual ASR\. Existing approaches, however, generally employ a predefined selection criterion and operate at one granularity, such as language weighting or utterance selection\. Coordinating language\-level allocation, utterance\-level relevance, and token\-level supervision according to changing multilingual ASR errors remains comparatively underexplored\[[20](https://arxiv.org/html/2610.36538#bib.bib31)\]\.

## 3Method

We propose a linguistic\-aware agentic data sampling framework\. The system use an LLM\-based agent to drive the data sampling process to train a multi\-lingual ASR model\. The overall pipeline is shown in Fig\.[1](https://arxiv.org/html/2610.36538#S3.F1)\.

In summary, the agentic workflow proceeds as an iterative training loop\. At each epoch, the system first samples a batch from the training corpus and uses it to fine\-tune the ASR model\. The fine\-tuned model is then evaluated on the development set, and its recognition outputs are converted into a linguistic rubric report\. Based on this report together with the training statistics, the LLM agent designs a three\-stage data\-sampling plan covering domain\-level, sample\-level, and token\-level decisions\. This plan is then applied to the training corpus to construct the subset used for fine\-tuning in the next epoch\.

![Refer to caption](https://arxiv.org/html/2610.36538v1/framework1.png)

Figure 1:Overview of the proposed agentic data sampling framework\. The blue nodes show the common ASR model training pipeline, while the purple nodes belong to our agentic data sampling workflow\.### 3\.1Linguistic Rubric

We propose a linguistic rubric for our multilingual ASR data sampling agent, which converts raw ASR recognition errors into more detailed statistics from a linguistic perspective, and provides structured feedback for data sampling\. The rubric attributes some ASR errors into three linguistic dimensions: phonology, morphology, and syntax\. The goal of the rubric is not to replace WER or CER, but to explain which linguistic phenomena contribute to the observed errors\. Based on the detailed error distribution in each linguistic dimension, the agent can identify more specific weaknesses and select more targeted sampling strategies\.

##### Phonology\.

The phonology dimension analyzes ASR errors at the phoneme level\. We convert grapheme\-level references and predictions into IPA representations using eSpeak NG\[[7](https://arxiv.org/html/2610.36538#bib.bib32)\], and then identify errors at the granularity of uniphones, biphones, and triphones\. This dimension helps identify which sounds or sound combinations are difficult for the model and supports phone\-aware sample selection\.

##### Morphology\.

The morphology dimension focuses on word\-formation errors, especially inflectional errors and case, gender, number, or tense marker errors\. We compare aligned reference and hypothesis tokens to determine whether the main lexical stem is preserved while a morphological marker changes or disappears\. In morphologically rich languages, a single inflectional error can alter the grammatical or semantic function of a word while causing only a minimal change in character\-level edit distance\.

##### Syntax\.

The syntax dimension focuses on errors involving prepositions, function words, and local word\-order patterns\. Although ASR is an acoustic\-to\-text task, recognition errors often appear as systematic textual mistakes, such as dropped function words or confused short grammatical words\. Syntax\-level analysis therefore provides additional guidance for selecting utterances that contain error\-prone grammatical contexts\.

For each linguistics dimension, the rubric report contains both categorical distribution and short natural\-language summaries\. The report is then passed to the agent together with WER/CER and training statistics\. In this way, the agent can distinguish, for example, a language whose high WER is mainly caused by phoneme confusions from another language whose errors are dominated by inflectional endings or function words\.

### 3\.2Data Sampling Agent

#### Hierarchical Algorithm Pool

We formulate data sampling for multilingual ASR as a hierarchical decision process with three stages: domain, sample, and token\. These stages correspond to three increasingly fine\-grained data sampling level: the language ratios of the training dataset, the utterances to be selected for each language, and operations \(e\.g\. gradient weights or masking\) applied to tokens during optimization\.

The agent designs the data sampling plan using a hirachical algorithm pool\. At each training epoch, the agent observes the linguistic rubric report, training statistics, and budget constraints, then selects one compatible algorithm from each level together with its parameters\. The algorithms selected by the data sampling agent are executed in hierarchical order\.

Table 1:Multi\-level data sampling algorithm pool\.AlgorithmStageDescriptionDoremiDomainUpdates language sampling weights using loss\-based signals\.Exp\-ReweightDomainReweights languages using an agent\-selected score\.DOGEDomainUses gradient similarity for domain\-level weighting\.LESSSampleSelects utterances using training\-validation gradient alignment\.Phone Error DistributionSampleSelects utterances matching phoneme error distributions\.RHO\-1TokenSelects informative tokens using token\-level loss or entropy\.TOSSTokenFilters tokens according to utility or risk scores\.We predefined the algorithm pool using 7 algorithms as shown in Table[1](https://arxiv.org/html/2610.36538#S3.T1)\. The pool is extensible: each algorithm is a self\-contained module that only requires its declared inputs to execute\. Once implemented, registering a new algorithm simply requires providing a profile describing its behaviour and parameter schema\. The agent can then reason about it based on its profile alongside all other registered algorithms\.

#### Data Sampling Procedure

The decision\-making process of agentic data sampling can be formulated as follows\. Given the original multilingual corpus𝒟\\mathcal\{D\}, the agent selects a sequence of three actions:

a=\(ad​o​m​a​i​n,as​a​m​p​l​e,at​o​k​e​n\),a=\(a^\{domain\},a^\{sample\},a^\{token\}\),\(1\)
where each action is itself a pairal=\(ml,θl\)a^\{l\}=\(m^\{l\},\\theta^\{l\}\)consisting of an algorithm choicemlm^\{l\}from a level of the predefined pool and its parametersθl\\theta^\{l\}\. The actions are applied sequentially:

𝒮=at​o​k​e​n​\(as​a​m​p​l​e​\(ad​o​m​a​i​n​\(𝒟\)\)\)\.\\mathcal\{S\}=a^\{token\}\(a^\{sample\}\(a^\{domain\}\(\\mathcal\{D\}\)\)\)\.\(2\)
which first adjusts language\-level sampling weights, then selects utterances under the adjusted weights, and finally applies per\-token operation during optimization\.

At each data\-sampling stage, the agent receives a compact state description that combines the current epoch statistics, including the training\-loss curve and WER/CER with per\-language breakdowns, with the linguistic rubric reports\. This state also includes the available algorithm profiles and their parameter schemas, the execution results from earlier sampling stages in the same epoch, such as the selected algorithm, parameters, and filtered dataset, and the running history of the agent’s decisions in previous epochs\. Given this information, the agent outputs a structured decision for the current stage, specifying the selected algorithm and parameters together with an interpretable rationale for the choice\.

## 4Experiments

### 4\.1Setup

#### Dataset

We employ a 12\-language subset of Common Voice\[[1](https://arxiv.org/html/2610.36538#bib.bib13)\]as our benchmark, covering Belarusian \(be\), Chinese \(zh\), Dutch \(nl\), French \(fr\), German \(de\), Italian \(it\), Japanese \(ja\), Polish \(pl\), Portuguese \(pt\), Russian \(ru\), Spanish \(es\), and Turkish \(tr\)\. This subset comprises a training corpus, a development set, and a test set\. Both the development and test sets are 12 hours, equally distributed with exactly one hour per language\. The training corpus comprises a total duration of 600 hours, with each language contributing an equal share of 50 hours\.

#### Baselines and Implementation

We fine\-tune the Whisper\-medium\[[23](https://arxiv.org/html/2610.36538#bib.bib3)\]model on this dataset using various data sampling strategies to evaluate their impact on automatic speech recognition \(ASR\) performance\. To isolate the effects of the sampling strategies, all methods are assessed under identical model architectures, optimizer configurations, and evaluation protocols\. Specifically, we employ Low\-Rank Adaptation\[[11](https://arxiv.org/html/2610.36538#bib.bib33)\]with a rankr=64r=64,α=64\\alpha=64, and a dropout rate of 0\.05\. The model is optimized using the AdamW optimizer with a learning rate of1×10−41\\times 10^\{\-4\}and a per\-device batch size of 128, distributed across 8 Ascend 910B NPUs\.

To ensure a fair comparison, the total duration of trained data for all models is strictly fixed at 1,200 hours\. We benchmark our proposed agentic data sampling method against two baselines: a no\-sampling baseline and a random sampling baseline\. The no\-sampling baseline utilizes the complete training corpus without any selection strategy; specifically, it was trained on the full 600\-hour data for 2 epochs\. The random sampling method involves randomly selecting 300 hours of data from the training corpus, repeated over 4 iterations\. For our agentic sampling method, we evaluated different sampling durations, extracting batches of 100 and 300 hours of data from the training corpus respectively to reach the target training volume\. Consequently, the agentic system performs 12 and 4 sampling iterations respectively\.

### 4\.2Main Results

![Refer to caption](https://arxiv.org/html/2610.36538v1/figures/test_wer.png)Figure 2:Comparison of overall test WER over training time under no sampling, random sampling, and agentic sampling\. The sampling batch of random and agentic method is 300 hours\.Table 2:Comparison of ASR performance across different data sampling strategies\. The suffix of each method denotes the data duration sampled per iteration\. Performance is measured in Character Error Rate \(CER\) for Japanese \(ja\) and Chinese \(zh\), and Word Error Rate \(WER\) for all other languages\. The best results for each language are highlighted in bold\.Table[2](https://arxiv.org/html/2610.36538#S4.T2)presents the comparative analysis of ASR performance across models trained with varying data sampling strategies\. Our proposed agentic sampling strategies demonstrate superior performance compared to the baselines\. Specifically, the agentic sampling method with 300 hours sampling batch achieves the best average performance with an error rate of 19\.4, significantly outperforming the no\-sampling baseline \(23\.9\) and the random sampling baseline \(22\.5\)\. In terms of specific language performance, it yields the lowest error rates in 9 out of 12 languages and shows significant improvement in Indo\-European languages, especially Romance languages\. The agentic\-300h method reduced the WER for Spanish and Portuguese by 38\.5% and 38\.0% respectively, compared with the no\-sampling baseline\. This may be benefit from our linguistic\-aware rubric as these languages shares similar linguistic features\.

As shown in Figure[2](https://arxiv.org/html/2610.36538#S4.F2), the baseline systems tend to converge after approximately 1200 hours of training, whereas the agentic sampling method continues to reduce the overall WER\. This suggests that agentic sampling selects more informative training data and remains effective beyond the point where the baseline strategies begin to saturate\.

### 4\.3Case Analysis

Table 3:Representative test cases corrected by agentic data sampling\. Red text marks persistent recognition errors, while green text marks the corrected linguistic unit after finetuning\.MorphologyFrenchReference:yvan publie des anthologies de poètes allemands et français et vit de ses traductionsWhisper:yvonpublie des anthologies de poètesallemandet français et vit decestraductionsNo\-sampling:yvonpublie des anthologies de poètesallemandet français et vit de ses traductionsAgentic:yvanpublie des anthologies de poètesallemandset français et vit de ses traductionsSyntaxItalianReference:a causa degli enormi debiti fu costretta a vivere grazie a dei prestitiWhisper:akaoza da gdje normidebitifukustretaa viveregracija da ihprestitiNo\-sampling:acausa degli enormidebitifu costrettaa vivere grazieaiprestitiAgentic:acausa degli enormidebitifu costrettaa viveregrazie a deiprestitiPhonologyPortugueseReference:ivaíWhisper:e vai
No\-sampling:e vai
Agentic:ivaíTable[3](https://arxiv.org/html/2610.36538#S4.T3)presents three representative cases of errors in each linguistic dimension\. The base Whisper model makes the error and no\-sampling finetuning still fails to correct it, while our agentic sampling model produces the correct transcription\. These examples illustrate that the improvement of agentic data sampling is not limited to general lexical recovery, but also appears in linguistically meaningful error categories\.

In the morphology case, an inflectional error occurs in the baseline as it transcribespoètes allemandsaspoètes allemand, dropping the plural agreement marker\-son the adjective\. The agentic model restoresallemands, suggesting that the selected training samples help the model better preserve inflectional information in Romance languages\.

In the syntax error case, the baseline simplifies the prepositional phrasegrazie a dei prestititograzie ai prestiti\. The agentic model, however, recovers the full prepositional construction, demonstrating an improved ability to model function\-word structures rather than merely correcting surface\-level spelling errors\. This enhancement is likely attributable to token\-level data sampling, which effectively resolves uncertainty at the local token level\.

The phonology case shows a clear diagnosis\-action\-test loop together with the rubric traces\. This phone error is identified in the rubric, and the phone error distributiont sampling algorithm is selected in the middle epochs to target such phonological patterns, and the final model resolves a persistent phonological confusion on the test set\.

Overall, these cases suggest that agentic data sampling improves ASR behavior across morphology, syntax, and phonology by selecting training data that is more closely aligned with the model’s observed error patterns\.

## 5Analysis

### 5\.1Agent Decision Behavior

We further examine whether the sampling policy produced by the agent is adaptive and interpretable rather than equivalent to a fixed schedule\. At each round, the agent receives the evaluation history and linguistic error diagnostics from the preceding round, and jointly selects domain\-, sample\-, and token\-level sampling strategies for the next round\.

Table 4:Representative agent\-selected sampling strategies\.Table[4](https://arxiv.org/html/2610.36538#S5.T4)shows a transition from a conservative cold\-start policy to diagnosis\-conditioned decisions\. In the first round, no previous evaluation history is available, so the agent selects broadly applicable loss\-based methods rather than making language\- or error\-specific interventions\. Once diagnostic feedback becomes available, the selected actions change across rounds instead of repeating the initial configuration\. This behavior indicates that the policy is conditioned on the evolving error profile of the ASR model rather than following a predefined sampling schedule\.

More importantly, the selected actions are aligned with the granularity of the observed errors\. When phonological errors become concentrated, the agent introducesPE\-DSTBat the sample level, connecting phone\-level diagnostics to the selection of training utterances\. Meanwhile, the alternation amongExp\-Reweight,DOGE, andDoReMisuggests that the agent distinguishes between multilingual allocation and within\-domain data quality\. Domain\-level actions determine where training emphasis should be placed, whereas sample\- and token\-level actions determine which supervision within those domains is most useful\.

The alternation betweenRHO\-1andTOSSfurther shows that the agent does not treat all difficult tokens as equally valuable\. Some persistent errors may require greater emphasis on under\-learned tokens, while others may benefit from filtering unreliable token\-level signals\. The resulting policy is therefore hierarchical: the agent first identifies the data regions requiring intervention and then selects an appropriate level of filtering or emphasis within those regions\.

Overall, the decisions provide evidence that the agent uses linguistic diagnostics to revise both the target and the granularity of data selection\. These examples do not establish that every selected strategy is globally optimal, but they demonstrate that the decision process is traceable, responsive to model feedback, and distinct from a fixed combination of sampling algorithms\. Together with the linguistic error reductions in Table[5](https://arxiv.org/html/2610.36538#S5.T5), this behavior supports the interpretation of the proposed framework as an adaptive feedback loop between error diagnosis and data selection\.

### 5\.2Linguistic Error Reduction

Beyond aggregate WER/CER, we further examine whether agentic data sampling reduces linguistically meaningful recognition errors\. We compare the original Whisper\-medium model with the model obtained after four epochs of agentic data sampling\. Both models are evaluated on the same Common Voice test set covering 12 languages\. For phonological analysis, we convert each reference and hypothesis into IPA phone sequences using G2P, strip non\-phonemic markers, and count all mismatched 1\-, 2\-, and 3\-phonenn\-gram events over the full test set\. For morphology and syntax, we use rule\-based detectors for inflectional marker errors, preposition/function\-word errors, and word\-order deviations\. The relative reduction is computed with respect to the original Whisper\-medium model\.

Table 5:Linguistic error counts of the original and model trained with agentic data sampling\.CategoryError TypeOriginalAgentic\-trainedReductionPhonologyUniphone124,191114,8319,360 \(7\.5%\)Biphone129,700121,1238,577 \(6\.6%\)Triphone131,243122,9708,273 \(6\.3%\)MorphologyInflectional marker / word\-ending975467508 \(52\.1%\)SyntaxPreposition / function\-word526403123 \(23\.4%\)Word\-order deviation312 \(66\.7%\)Table[5](https://arxiv.org/html/2610.36538#S5.T5)reveals a complementary pattern across linguistic levels\. Phonological errors contribute the largest absolute reductions, reflecting their high frequency in multilingual ASR, while morphological errors exhibit the largest relative improvement\. This suggests that the benefit of agentic data sampling is not confined to frequent acoustic–phonetic confusions: it also extends to less frequent but linguistically informative errors, such as inflectional suffixes and word\-final grammatical markers\. The consistent reductions from uniphone to triphone errors further indicate improved modeling of both individual phonemes and short phonetic contexts, although the smaller gain at highernn\-gram orders suggests that longer context\-dependent confusions remain more difficult\.

These results support the central motivation of our agentic sampling framework\. A uniform sampling strategy treats all training data as equally useful, whereas the agent adjusts the data distribution using historical model performance and linguistic error feedback\. The observed error profile is consistent with the agent allocating additional training exposure to languages or samples containing persistent phonological contrasts, morphologically marked word forms, and grammatical function words\. In particular, the substantial reduction in morphological errors suggests that adaptive sampling can improve phenomena that may be underrepresented in the overall training distribution and therefore receive insufficient attention under uniform sampling\.

The reduction in preposition and function\-word errors also indicates that the improvement is not limited to content\-word recognition, but extends to local grammatical information\. However, the number of word\-order deviations is too small to draw a reliable conclusion about global syntactic modeling\. Overall, the linguistic breakdown provides evidence that agentic data sampling changes not only the aggregate error rate but also the composition of the remaining errors, yielding improvements across acoustic–phonetic and grammatical dimensions that cannot be fully characterized by WER/CER alone\.

## 6Conclusion

We proposed a linguistic\-aware agentic data sampling framework for multilingual ASR that converts phonological, morphological, and syntactic error diagnostics into domain\-, utterance\-, and token\-level sampling decisions\. By placing evaluation inside the training loop, the framework treats data sampling as an adaptive process rather than a fixed preprocessing strategy\.

Experiments on twelve Common Voice languages with Whisper\-medium show that the agentic\-300h setting achieves an average error rate of 19\.4, compared with 23\.9 for no sampling and 22\.5 for random sampling, and obtains the best result on nine languages\. The method also reduces phone\-level, inflectional, and function\-word errors, while the selected strategies change with the evolving model error profile\. These results suggest that multilingual ASR evaluation can serve not only to measure performance, but also to provide actionable feedback for subsequent data selection\. Future work will extend the framework to broader language coverage and conduct finer\-grained ablations of the linguistic rubric and individual sampling decisions\.

## References

- \[1\]R\. Ardila, M\. Branson, K\. Davis, M\. Kohler, J\. Meyer, M\. Henretty, R\. Morais, L\. Saunders, F\. Tyers, and G\. Weber\(2020\)Common voice: a massively\-multilingual speech corpus\.InProceedings of the twelfth language resources and evaluation conference,pp\. 4218–4222\.Cited by:[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.36538#S4.SS1.SSSx1.p1.1)\.
- \[2\]A\. Babu, C\. Wang, A\. Tjandra, K\. Lakhotia, Q\. Xu, N\. Goyal, K\. Singh, P\. von Platen, Y\. Saraf, J\. Pino,et al\.\(2022\)XLS\-r: self\-supervised cross\-lingual speech representation learning at scale\.Interspeech 2022\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p1.1)\.
- \[3\]T\. Bai, L\. Yang, Z\. H\. Wong, F\. Sun, X\. Zhuang, J\. Peng, C\. Zhang, L\. Wu, Q\. Jiantao, W\. Zhang,et al\.\(2025\)Efficient pretraining data selection for language models via multi\-actor collaboration\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9465–9491\.Cited by:[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p4.1)\.
- \[4\]W\. Chen, C\. Meng, J\. Shi, M\. Bartelds, S\. Wang, H\. Wang, R\. Mosquera, S\. Hincapie, D\. Jurafsky, A\. Anastasopoulos,et al\.\(2025\)The ml\-superb 2\.0 challenge: towards inclusive asr benchmarking for all language varieties\.InProc\. Interspeech 2025,pp\. 2093–2097\.Cited by:[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p1.1)\.
- \[5\]A\. Conneau, A\. Bapna, Y\. Zhang, M\. Ma, P\. von Platen, A\. Lozhkov, C\. Cherry, Y\. Jia, C\. Rivera, M\. Kale,et al\.\(2022\)XTREME\-s: evaluating cross\-lingual speech representations\.InProc\. Interspeech 2022,pp\. 3248–3252\.Cited by:[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p1.1)\.
- \[6\]A\. Conneau, M\. Ma, S\. Khanuja, Y\. Zhang, V\. Axelrod, S\. Dalmia, J\. Riesa, C\. Rivera, and A\. Bapna\(2023\)Fleurs: few\-shot learning evaluation of universal representations of speech\.In2022 IEEE Spoken Language Technology Workshop \(SLT\),pp\. 798–805\.Cited by:[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p1.1)\.
- \[7\]eSpeak NG ContributorseSpeak NG text\-to\-speech\.Note:https://github\.com/espeak\-ng/espeak\-ngCited by:[§3\.1](https://arxiv.org/html/2610.36538#S3.SS1.SSS0.Px1.p1.1)\.
- \[8\]S\. Fan, M\. Pagliardini, and M\. Jaggi\(2024\)DOGE: domain reweighting with generalization estimation\.InInternational Conference on Machine Learning,pp\. 12895–12915\.Cited by:[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p1.1)\.
- \[9\]S\. Gonzalez, T\. Hoang, M\. M\. Kim, B\. Donnelly, J\. Biggs, and T\. Cawley\(2025\)Understanding multilingual asr systems: the role of language families and typological features in seamless and whisper\.InProceedings of the 23rd Annual Workshop of the Australasian Language Technology Association,pp\. 90–100\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p3.1)\.
- \[10\]I\. Hsieh, C\. Wu, and Z\. Zhao\(2024\)Dynamic sampling\-based meta\-learning using multilingual acoustic data for under\-resourced speech recognition\.IEEE Access12,pp\. 106070–106083\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p3.1)\.
- \[11\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§4\.1](https://arxiv.org/html/2610.36538#S4.SS1.SSSx2.p1.1)\.
- \[12\]M\. Lau, Q\. Chen, Y\. Fang, T\. Xu, T\. Chen, and P\. Golik\(2025\)Data quality issues in multilingual speech datasets: the need for sociolinguistic awareness and proactive language planning\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7466–7492\.Cited by:[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p2.1)\.
- \[13\]X\. Li, S\. Dalmia, A\. W\. Black, and F\. Metze\(2019\)Multilingual speech recognition with corpus relatedness sampling\.InProc\. Interspeech 2019,pp\. 2120–2124\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p3.1)\.
- \[14\]Y\. Li, Z\. Liu, Z\. Li, Z\. Lin, and J\. ZhangToken\-level data selection for safe llm fine\-tuning\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p2.1)\.
- \[15\]H\. Liang, X\. Ma, Z\. Liu, Z\. H\. Wong, Z\. Zhao, Z\. Meng, R\. He, C\. Shen, Q\. Cai, Z\. Han,et al\.\(2025\)DataFlow: an llm\-driven framework for unified data preparation and workflow automation in the era of data\-centric ai\.arXiv preprint arXiv:2512\.16676\.Cited by:[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p4.1)\.
- \[16\]Z\. Lin, Z\. Gou, Y\. Gong, X\. Liu, Y\. Shen, R\. Xu, C\. Lin, Y\. Yang, J\. Jiao, N\. Duan,et al\.\(2024\)Not all tokens are what you need for pretraining\.InProceedings of the 38th International Conference on Neural Information Processing Systems,pp\. 29029–29063\.Cited by:[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p2.1)\.
- \[17\]N\. Majidi, E\. Amid, H\. Talebi, and M\. K\. Warmuth\(2021\)Exponentiated gradient reweighting for robust training under label noise and beyond\.arXiv preprint arXiv:2104\.01493\.Cited by:[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p1.1)\.
- \[18\]K\. Manohar and L\. G\. Pillai\(2024\)What is lost in normalization? exploring pitfalls in multilingual asr model evaluations\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 10864–10869\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p2.1)\.
- \[19\]S\. Mitsumori, S\. Kashiwagi, K\. Tanaka, and S\. Morishima\(2025\)Cross\-lingual data selection using clip\-level acoustic similarity for enhancing low\-resource automatic speech recognition\.InProc\. Interspeech 2025,pp\. 3314–3318\.Cited by:[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p3.1)\.
- \[20\]J\. Peng, Y\. Wang, B\. Li, Y\. Guo, H\. Wang, Y\. Fang, Y\. Xi, H\. Li, X\. Li, K\. Zhang,et al\.\(2025\)A survey on speech large language models for understanding\.IEEE Journal of Selected Topics in Signal Processing\.Cited by:[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p5.1)\.
- \[21\]V\. Pratap, A\. Sriram, P\. Tomasello, A\. Hannun, V\. Liptchinsky, G\. Synnaeve, and R\. Collobert\(2020\)Massively multilingual asr: 50 languages, 1 model, 1 billion parameters\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p1.1)\.
- \[22\]V\. Pratap, A\. Tjandra, B\. Shi, P\. Tomasello, A\. Babu, S\. Kundu, A\. Elkahky, Z\. Ni, A\. V\. M\. F\. Alexei, B\. Y\. Adi,et al\.\(2024\)Scaling speech technology to 1,000\+ languages\.Journal of Machine Learning Research25,pp\. 1–52\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p1.1)\.
- \[23\]A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever\(2023\)Robust speech recognition via large\-scale weak supervision\.InProceedings of the 40th International Conference on Machine Learning,pp\. 28492–28518\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.36538#S4.SS1.SSSx2.p1.1)\.
- \[24\]N\. Sachdeva, B\. Coleman, W\. Kang, J\. Ni, L\. Hong, E\. H\. Chi, J\. Caverlee, J\. McAuley, and D\. Z\. Cheng\(2024\)How to train data\-efficient llms\.arXiv preprint arXiv:2402\.09668\.Cited by:[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p4.1)\.
- \[25\]N\. San, G\. Paraskevopoulos, A\. Arora, X\. He, P\. Kaur, O\. Adams, and D\. Jurafsky\(2024\)Predicting positive transfer for improved low\-resource speech recognition using acoustic pseudo\-tokens\.InProceedings of the 6th workshop on research in computational linguistic typology and multilingual NLP,pp\. 100–112\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p3.1)\.
- \[26\]J\. Shi, D\. Berrebbi, W\. Chen, E\. Hu, W\. Huang, H\. Chung, X\. Chang, S\. Li, A\. Mohamed, H\. Lee,et al\.\(2023\)ML\-superb: multilingual speech universal performance benchmark\.INTERSPEECH 2023\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p1.1)\.
- \[27\]J\. Shi, S\. Wang, W\. Chen, M\. Bartelds, V\. Bannihatti Kumar, J\. Tian, X\. Chang, D\. Jurafsky, K\. Livescu, H\. Lee,et al\.\(2024\)ML\-superb 2\.0: benchmarking multilingual speech models across modeling constraints, languages, and datasets\.InProc\. Interspeech 2024,pp\. 1230–1234\.Cited by:[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p1.1)\.
- \[28\]V\. Srivastav, S\. Zheng, E\. Bezzam, E\. Le Bihan, N\. Koluguri, P\. Żelasko, S\. Majumdar, A\. Moumen, and S\. Gandhi\(2025\)Open asr leaderboard: towards reproducible and transparent multilingual and long\-form speech recognition evaluation\.arXiv e\-prints,pp\. arXiv–2510\.Cited by:[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p1.1)\.
- \[29\]D\. Thennal, J\. James, D\. P\. Gopinath,et al\.\(2025\)Advocating character error rate for multilingual asr evaluation\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 4926–4935\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p2.1)\.
- \[30\]M\. Xia, S\. Malladi, S\. Gururangan, S\. Arora, and D\. Chen\(2024\)LESS: selecting influential data for targeted instruction tuning\.InProceedings of the 41st International Conference on Machine Learning,pp\. 54104–54132\.Cited by:[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p2.1)\.
- \[31\]Y\. Xiao, K\. Gong, P\. Zhou, G\. Zheng, X\. Liang, and L\. Lin\(2021\)Adversarial meta sampling for multilingual low\-resource speech recognition\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 14112–14120\.Cited by:[§1](https://arxiv.org/html/2610.36538#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p3.1)\.
- \[32\]S\. M\. Xie, H\. Pham, X\. Dong, N\. Du, H\. Liu, Y\. Lu, P\. S\. Liang, Q\. V\. Le, T\. Ma, and A\. W\. Yu\(2023\)Doremi: optimizing data mixtures speeds up language model pretraining\.Advances in Neural Information Processing Systems36,pp\. 69798–69818\.Cited by:[§2\.2](https://arxiv.org/html/2610.36538#S2.SS2.p1.1)\.
- \[33\]B\. Yan, I\. Hamed, S\. Shimizu, V\. S\. Lodagala, W\. Chen, O\. Iakovenko, B\. Talafha, A\. Hussein, A\. Polok, K\. Chang,et al\.\(2025\)CS\-fleurs: a massively multilingual and code\-switched speech dataset\.InProc\. Interspeech 2025,pp\. 743–747\.Cited by:[§2\.1](https://arxiv.org/html/2610.36538#S2.SS1.p1.1)\.

## Appendix 1

Structure of agent decision in each stage:

```
{
  "rationale": {
    "algorithm": "Per-language CER shows high variation...",
    "param": "alpha=0.8 to smooth weights for low-resource languages..."
  }
  "algorithm": "doremi",
  "params": {"alpha": 0.8, "reweight_eta": 2.0, ...},
}
```

## Appendix 2

Example of linguistic rubric:

```
{
  "phonology": {
    "uniphone_errors": [
      {"phone": "i", "error_rate": 0.68, "languages": ["be", "pl", "pt"]},
      ...
    ],
    "biphone_errors": [...],
    "triphone_errors": [...],
    "summary": "High phonological error rates are observed..."
  },
  "morphology": {
    "inflection_errors": [
      {"language": "de", "marker": "er", "error_rate": 0.10, ...}
      ...
    ],
    "summary": "Inflectional errors concentrate on..."
  },
  "syntax": {
    "prep_errors": [
      {"language": "fr", "prep": "de", "error_rate": 0.13, ...}
      ...
    ],
    "summary": "Preposition and function-word errors are frequent..."
  }
}
```

## Appendix 3: Prompt Template

The prompt template used by the LLM agent at each sampling stage is as follows\. Placeholders enclosed in braces are filled by the workflow before each LLM call\.

You are a data sampling agent for multilingual ASR training\.\#\# ContextThe pipeline runs three stages sequentially:domain \(= language\) \-\> sample \(= utterance\) \-\> token \(= subword\)Each stage’s output feeds into the next\.\#\# Current stageYou are configuring \*\*Stage \{stage\_num\} \-\- \{stage\_name\} \(\{stage\_level\}\)\*\*\.Available algorithms: \{available\_algorithms\}This stage runs \*\*\{stage\_phase\}\*\*\.\#\# What you will receive\- \`\`rubric\_diagnosis\`\`: error patterns, per\-language CER, overall CER,phone/grapheme distributions, and summary text\.\- \`\`previous\_stages\`\`: summary of what earlier stages did and their results\.\(empty for the first stage\)\- \`\`selection\_context\`\`: experiment\-level constraints and cross\-epochalgorithm usage history, including under\-used algorithms and recentdomain\-\>sample\-\>token paths\.\#\# Rules1\. Choose exactly ONE algorithm from the available list\.2\. Set specific parameter values using the parameter schemas below\.3\. Prefer conservative settings unless the diagnosis clearly supports aggressive filtering\.4\. This experiment must explore the algorithm pool across epochs\. Preferunder\-used algorithms and new stage combinations when their requiredinputs are available from the rubric or from orchestrator precompute\.If an algorithm has zero or much lower usage than the alternatives at thesame stage, choose it before reusing a heavily repeated stage algorithmunless it is invalid for the current inputs\.Avoid repeating the same stage algorithm for more than two consecutiveepochs unless every alternative is invalid; if you repeat, explain why\.5\. Do not choose an algorithm only because it was used in the previous epoch\.Balance current diagnosis fit with coverage of doremi/exp\_reweight/doge,less/pe\_dstb, and rho1/toss over the full run\.6\. Return valid JSON matching the required schema\.7\. Output schema fields following their original order in the schema \(rationale \-\> algorithm \-\> params\)\.\.\. note::File\-path parameters such as \`\`error\_results\_path\`\`, \`\`loss\_path\`\`, and\`\`score\_path\`\` must refer to real run artifacts when you set them\. Formodel\-based score fields such as DoReMi losses or rho1 token scores, theorchestrator precomputes missing fields before training from the currentepoch checkpoint and the fixed train split\. Algorithms do not fall backto package\-local offline scored JSONL files\.\#\# Algorithm parameter schemas\{algo\_schemas\}

\*