ETC-Bench: A Multilingual Benchmark for Evaluation-Side Transcript Canonicalization

# ETC-Bench: A Multilingual Benchmark for Evaluation-Side Transcript Canonicalization
Source: [https://arxiv.org/html/2610.36527](https://arxiv.org/html/2610.36527)
Junhao DuJing PengHanqi LiKunyang PengFeng XueKai Yu††thanks:\*Equal contribution\.†Corresponding author\. Note:ETC\-Bench and TACS will be released on GitHub soon\.

###### Abstract

Written transcripts of the same speech can differ in ways that are not uniquely determined by the speech signal\. We formulate evaluation\-side transcript canonicalization \(ETC\), which maps multiple surface forms to a single canonical form when their differences are not determined by speech\. To our knowledge, ETC\-Bench is the first and most comprehensive multilingual benchmark for evaluation\-side transcript canonicalization, covering 181 phenomenon types across 12 languages\. Each instance is a minimal pair that shares a sentence context and differs only in a target span exhibiting one phenomenon\. To evaluate both canonicalization success and semantic preservation, we propose Semantic\-preserving Exact\-match Normalization Accuracy \(SENA\), which combines exact agreement between independently canonicalized pair members with a pair\-relative BERTScore constraint against information loss\. We further introduce the Taxonomy\-guided Agent Canonicalization Skill \(TACS\), which provides LLM agents with phenomenon descriptions and explicit canonicalization contracts\. Experiments with five baseline configurations and three TACS\-based agents show that the agents outperform the baseline systems on shared languages\. ETC\-Bench provides a systematic testbed for evaluation\-side transcript canonicalization across languages\.

###### Index Terms:

evaluation\-side transcript canonicalization, speech\-to\-text evaluation, multilingual benchmark, text normalization, speech processing

††address:1X\-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
1MoE Key Lab of Artificial Intelligence,1Jiangsu Key Lab of Language Computing
2AISpeech Ltd, Suzhou, China![Refer to caption](https://arxiv.org/html/2610.36527v1/figs/mane.PNG)Figure 1:Overview of ETC\-Bench: taxonomy, benchmark\-item structure, and evaluation metric\.## 1Introduction

Speech\-to\-text \(STT\) systems\[[17](https://arxiv.org/html/2610.36527#bib.bib23),[6](https://arxiv.org/html/2610.36527#bib.bib24)\]produce text from speech and are commonly evaluated by comparing hypotheses with references\[[11](https://arxiv.org/html/2610.36527#bib.bib15)\]\.However, speech does not always determine a unique written form\. Transcript ambiguity arises when a spoken utterance does not encode the graphemic or symbolic information needed to choose among alternative written forms with the same spoken realization\. For example, a number may be transcribed as “2” or “two”, and a common Chinese name pronounced “/sw@nli/” may have several commonly used character forms, such as “

孙丽” and “

孙立”\. Furthermore, in some languages the same word may have multiple spellings\[[14](https://arxiv.org/html/2610.36527#bib.bib14)\]\. Word error rate \(WER\) counts such differences as errors, conflating model performance with transcript conventions and potentially biasing cross\-lingual comparisons\[[22](https://arxiv.org/html/2610.36527#bib.bib22)\]\.

Text normalization \(TN\) and inverse text normalization \(ITN\) are two major normalization paradigms in speech processing\. They aim at the conversion between written and spoken forms\[[20](https://arxiv.org/html/2610.36527#bib.bib17),[8](https://arxiv.org/html/2610.36527#bib.bib18),[5](https://arxiv.org/html/2610.36527#bib.bib20)\]\. TN maps written expressions to spoken forms, typically for text\-to\-speech \(TTS\)\[[21](https://arxiv.org/html/2610.36527#bib.bib16)\], whereas ITN maps spoken\-form text to conventional written text\. NeMo provides weighted finite\-state grammars\[[10](https://arxiv.org/html/2610.36527#bib.bib19)\]for TN and ITN classes such as numbers, dates, and measurements\[[26](https://arxiv.org/html/2610.36527#bib.bib1)\]\. PolyNorm extends multilingual TN with an LLM\-based pipeline and a written\-to\-spoken taxonomy\[[24](https://arxiv.org/html/2610.36527#bib.bib2)\]\. Although TN and ITN were developed for different application needs, both can serve as canonicalization subroutines, depending on the chosen target representation\. In either case, applying the same mapping independently to each transcript can remove the surface distinction\. However, existing TN and ITN systems cover only a limited subset of the ambiguities considered here\. Many language\-specific orthographic and homophonic variations fall outside the semiotic classes they normalize\.

In practice, the evaluation of automatic speech recognition \(ASR\) already accommodates transcript variation in several ways to reduce the effect of this ambiguity to better measure model performance\. Multi\-reference evaluation accounts for acceptable spelling variation in Egyptian Arabic, but requires additional transcripts\[[1](https://arxiv.org/html/2610.36527#bib.bib5)\]\. Sclite also accommodates transcript variation in its evaluation procedure\[[7](https://arxiv.org/html/2610.36527#bib.bib11)\]\. Whisper provides post\-ASR text normalizers to reduce penalties for transcript variation\[[19](https://arxiv.org/html/2610.36527#bib.bib3)\], but its well\-established normalization tool is limited to English\. Furthermore, its inappropriate behavior can damage semantic information, as shown for Indic scripts\[[13](https://arxiv.org/html/2610.36527#bib.bib6)\]\. Despite these practices, multilingual evaluation\-oriented normalization remains limited in scope\. Recent speech\-evaluation efforts, including SURE and Open\-Bench, likewise emphasize reproducible pipelines and fair comparisons through standardized normalization and scoring\[[16](https://arxiv.org/html/2610.36527#bib.bib26),[15](https://arxiv.org/html/2610.36527#bib.bib25)\]\. Motivated by this principle, we define*evaluation\-side transcript canonicalization*\(ETC\) to map STT transcript variants to a consistent canonical form when their differences cannot be resolved from speech, supporting more trustworthy ASR evaluation and offering a framework relevant to speech\-translation text outputs\. Given a transcriptxxand its languagell, a canonicalizer producesC⁡\(x,l\)C\(x,l\), without access to another transcript of the same utterance\. Leta≡lba\\equiv\_\{l\}bindicate thataaandbbare valid written representations of the same spoken utterance and differ only in distinctions that cannot be recovered from that utterance\. An ETC system should then satisfy:

a≡lb⇒C\(a,l\)=C\(b,l\),a\\equiv\_\{l\}b\\Rightarrow C\(a,l\)=C\(b,l\),\(1\)while preserving the semantics encoded in the speech\. Unlike conventional TN and ITN, which are designed for specific TTS/ASR pre/post\-processing, ETC aims to map transcript variants into a consistent canonical representation for fair WER comparison, where either TN or ITN\-style transformations can be adopted\. Extending this principle to speech and text translation requires task\-specific definitions of admissible variation and semantic\-preservation constraints, which we leave to future work\.

In this work, our contributions are threefold: \(1\) we define evaluation\-side transcript canonicalization as a standalone task; \(2\) we construct ETC\-Bench, comprising 1,098 native\-speaker\-reviewed minimal pairs covering 181 phenomenon types across 12 languages, organized by an extensible taxonomy, and propose Semantic\-preserving Exact\-match Normalization Accuracy \(SENA\); and \(3\) we release the Taxonomy\-guided Agent Canonicalization Skill \(TACS\) and assess conventional and agent\-based canonicalization across languages and phenomenon categories\. Agents using TACS achieve higher SENA than the evaluated conventional baselines on shared languages\.

## 2ETC\-Bench

We construct ETC\-Bench, a multilingual benchmark for evaluating ETC\. The benchmark comprises 1,098 minimal pairs covering 181 distinct canonicalization phenomena across 12 languages: Arabic \(ar\), Bengali \(bn\), German \(de\), English \(en\), Persian \(fa\), French \(fr\), Hindi \(hi\), Indonesian \(id\), Japanese \(ja\), Russian \(ru\), Vietnamese \(vi\), and Mandarin Chinese \(zh\)\.

### 2\.1Phenomenon Taxonomy

Table 1:Representative ETC\-Bench pairs\. Blue highlights distinctions not recoverable from speech\.CategoryPhenomenonExample PairNumericCardinalThe recipe calls for2eggs\.
The recipe calls fortwoeggs\.HomophonicHomophonicNamesSaracalled me yesterday\.
Sarahcalled me yesterday\.AbbreviationContext\-dependentAbbreviationСт\.5 закона …
Статья5 закона …OrthographicUmlautVariantsWir kaufenÖl\.
Wir kaufenOel\.HomophonicThird\-personPronoun他负责那部分任务。
她负责那部分任务。We develop a taxonomy of transcript ambiguities by analyzing the outputs of a multilingual ASR model\[[12](https://arxiv.org/html/2610.36527#bib.bib12)\]\. The taxonomy organizes variation into five categories \(Fig\.[1](https://arxiv.org/html/2610.36527#S0.F1)\)\. A*category*is a coarse\-grained class defined by the source of transcript variation, which groups related variation, while a*phenomenon*specifies a pattern of transcript ambiguity through a natural\-language description and serves as the unit for benchmark construction and analysis\.

We group the five categories according to whether their phenomena are language\-agnostic or language\-specific\. Numeric Expressions and Symbols and Markers contain*language\-agnostic*phenomena: their definitions are shared across languages, although their applicability and realizations may differ\. This inventory adapts and extends PolyNorm\[[24](https://arxiv.org/html/2610.36527#bib.bib2)\]\. In contrast, Abbreviations, Orthographic Variants, and Homophonic Substitutions contain*language\-specific*phenomena whose definitions depend on a language’s lexicon, orthography, or writing system\. Homophonic Substitutions specifically covers identically pronounced expressions whose written distinctions cannot be resolved from speech; these forms need not have identical written meanings\. The hierarchy supports extension to additional languages and phenomena\.

### 2\.2Benchmark Construction

Each benchmark itemiiconsists of a minimal pair of sentencesaia\_\{i\}andbib\_\{i\}, annotated with a languagelil\_\{i\}, a categorycic\_\{i\}, and a phenomenonpip\_\{i\}\. The full sentencesaia\_\{i\}andbib\_\{i\}share a context and differ only in a target span instantiating the phenomenon\. Language\-agnostic pairs use shared templates localized with language\-appropriate contexts and values; language\-specific pairs instantiate phenomenon descriptions with suitable lexical items and contexts\.

We use GPT\-5\.6\-Sol to generate 20 candidate pairs per phenomenon for each applicable language and to filter malformed pairs, followed by manual cleaning\. The candidate pool retains an average of 16\.2 pairs per language\-indexed phenomenon\. We then sample 100 pairs per language, allocating at least one pair to every retained phenomenon to prioritize representative coverage\. Twelve native speakers from Beijing Language and Culture University review both the validity of the phenomenon definitions and the correctness of the sampled pairs\. Experts refine the inventories and clean the samples, yielding the final release of 1,098 pairs\. The released counts reflect this final curation rather than the initial 100\-pair sampling quota\.

### 2\.3Evaluation Metric

We propose SENA as the evaluation metric of our benchmark\. It measures the proportion of pairs satisfying both an exact\-match condition and a semantic\-preservation condition\.

LetC⁡\(x,l\)C\(x,l\)denote a canonicalization system that produces a canonical text using only inputxxand its languagell\. The two transcripts within a benchmark pair are processed independently\. Letsms\_\{m\}denote BERTScoreF1F\_\{1\}under semantic encodermm\[[25](https://arxiv.org/html/2610.36527#bib.bib4),[23](https://arxiv.org/html/2610.36527#bib.bib21)\]\. Each benchmark pair is reviewed to isolate a distinction for which a STT model lacks distinguishing evidence in speech, not necessarily to assert identical written semantics \(e\.g\., homophonic gender\-marked pronouns\)\. Its source\-side similarity therefore supplies a sample\-adaptive reference that allows the targeted written distinction to be removed:

ti\\displaystyle t\_\{i\}=sm​\(ai,bi\)\\displaystyle=s\_\{m\}\(a\_\{i\},b\_\{i\}\)\(2\)fi\\displaystyle f\_\{i\}=minx∈\{ai,bi\}⁡sm​\(x,C⁡\(x,li\)\)\\displaystyle=\\min\_\{x\\in\\\{a\_\{i\},b\_\{i\}\\\}\}s\_\{m\}\\\!\\left\(x,C\(x,l\_\{i\}\)\\right\)\(3\)Here,tit\_\{i\}is the similarity between the two original texts, andfif\_\{i\}is the lower of the two similarities between an input and its canonicalized output\. Both conditions must hold:

SENA=1n∑i=1n𝕀\[C\(ai,li\)=C\(bi,li\)\]𝕀\[fi\+δ≥ti\],\\operatorname\{SENA\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbb\{I\}\[C\(a\_\{i\},l\_\{i\}\)=C\(b\_\{i\},l\_\{i\}\)\]\\mathbb\{I\}\[f\_\{i\}\+\\delta\\geq t\_\{i\}\],\(4\)wherennis the number of evaluated pairs,𝕀\\mathbb\{I\}is an indicator function, andδ=0\.1\\delta=0\.1is a tolerance on the BERTScore differenceti−fit\_\{i\}\-f\_\{i\}, fixed before the experiments\. All SENA results are reported as percentages\. The*exact\-match condition*requires identical canonicalized texts\. The*semantic\-preservation condition*requiresfi\+δ≥tif\_\{i\}\+\\delta\\geq t\_\{i\}, using the source pair as a reference for limiting additional semantic loss\.

## 3TACS: Taxonomy\-Guided Agent Canonicalization Skill

Rule\-based normalizers work well for enumerable patterns such as numbers and dates, but ETC\-Bench also covers context\-dependent abbreviations, orthographic conventions, and homophonic substitutions\. Covering these phenomena with handcrafted rules across languages would require extensive language\-specific implementations\. LLMs can draw on contextual and multilingual knowledge, but this ability alone does not ensure that independently processed variants receive the same output\.

We therefore introduceTACS, a taxonomy\-guided skill for LLM agents\. An agent can load relevant guidance as needed to canonicalize each input independently\. The taxonomy describes which distinctions to recognize\. Canonicalization contracts define consistent output forms by language, category, and phenomenon\. Because the contracts specify targets by phenomenon, TACS can express both TN\-style written\-to\-spoken rewrites and ITN\-style spoken\-to\-written rewrites, as well as canonicalizations beyond their conventional semiotic classes\. The skill also instructs the agent to make local changes while preserving unrelated text\. Table[2](https://arxiv.org/html/2610.36527#S3.T2)illustrates canonical forms for the five phenomena in Table[1](https://arxiv.org/html/2610.36527#S2.T1)\.

Table 2:TACS canonical forms for the phenomena illustrated in Table[1](https://arxiv.org/html/2610.36527#S2.T1)\.LIDCategoryPhenomenonCanonical FormenNumericExpressionsCardinalQuantity in ASCII digits\.enHomophonicSubstitutionsHomophonicNamesContext\-appropriate name inbroad American English IPA\.ruAbbreviationsContext\-dependentAbbreviationContext\-appropriate fullRussian form\.deOrthographicVariantsUmlaut VariantsUse*ä, ö, ü*instead of*ae, oe, ue*\.zhHomophonicSubstitutionsThird\-personPronounBroad Mandarin IPA with tone\.The skill guides the agent to use these resources in two steps\. First, it identifies phenomenon instances using the descriptions and the current utterance, resolving overlapping or competing interpretations\. Second, it generates local replacements according to the corresponding contracts\. The instructions require non\-overlapping replacements, exact preservation of text outside the selected spans, and retention of spans with unresolved ambiguity or no applicable contract\. These steps organize the agent’s reasoning without prescribing separate model calls or intermediate outputs\.

Loading TACS starts withSKILL\.md, which specifies batch I/O and global constraints\. The agent is instructed to solve each record independently and write results to the designated output file, without comparing records or using them as evidence for one another\. The skill contains no benchmark pairs or per\-item targets\.

Table 3:Performance comparison across languages in terms of SENA \(%\)\. – indicates unsupported languages\. Avg\. denotes the macro\-average across supported languages\.SystemarbndeenfafrhiidjaruvizhAvg\.NeMo\-ITN1\.2–2\.25\.7–5\.60\.0–5\.35\.01\.014\.14\.4NeMo\-TN0\.0–2\.118\.2–3\.42\.0–2\.64\.06\.15\.44\.9FunASR\-ITN––2\.25\.7–5\.6–3\.15\.35\.01\.02\.23\.7FunASR\-TN––2\.119\.3–––––4\.0–12\.09\.4Whisper Normalizer0\.00\.00\.040\.95\.05\.60\.08\.23\.94\.07\.10\.06\.2No\-Skill\-GPT0\.024\.00\.00\.00\.00\.018\.00\.00\.00\.00\.00\.03\.5No\-Skill\-DeepSeek29\.12\.08\.60\.08\.81\.10\.00\.00\.06\.00\.00\.04\.6No\-Skill\-Qwen15\.120\.812\.917\.49\.210\.113\.324\.110\.514\.76\.511\.613\.9TACS\-GPT51\.926\.542\.744\.762\.139\.740\.044\.238\.640\.346\.641\.743\.3TACS\-DeepSeek42\.623\.149\.853\.047\.948\.331\.041\.842\.548\.355\.840\.943\.8TACS\-Qwen69\.067\.761\.365\.572\.973\.061\.375\.264\.575\.378\.262\.068\.8
## 4Experiments

### 4\.1Experimental Setup

We evaluate five baseline configurations on ETC\-Bench: NeMo\-ITN and NeMo\-TN\[[26](https://arxiv.org/html/2610.36527#bib.bib1)\], FunASR\-ITN and FunASR\-TN\[[9](https://arxiv.org/html/2610.36527#bib.bib7)\], and Whisper’s normalization module\[[19](https://arxiv.org/html/2610.36527#bib.bib3)\], alongside three TACS\-based systems\. We include conventional TN and ITN baseline because ETC is direction\-agnostic and both TN and ITN can be regard as an implementation of ETC\. Some commercial STT services, such as Azure Speech, also include post\-processing that can resolve some transcript ambiguities\. We did not evaluate these services because their post\-processing is not available as a standalone text\-to\-text normalizer\. Performance is measured using SENA as defined in Section[2\.3](https://arxiv.org/html/2610.36527#S2.SS3), with BERTScore computed using thexlm\-roberta\-largeencoder\[[2](https://arxiv.org/html/2610.36527#bib.bib8)\]\. Whisper usesEnglishTextNormalizerfor English andBasicTextNormalizerfor the other 11 languages\. We instantiate the TACS\-based agent systems with GPT\-5\.6\-Terra, DeepSeek\-V4\-Flash\[[4](https://arxiv.org/html/2610.36527#bib.bib13)\], and Qwen3\.8\-Flash\[[18](https://arxiv.org/html/2610.36527#bib.bib9)\], and refer to them as TACS\-GPT, TACS\-DeepSeek, and TACS\-Qwen, respectively\. GPT\-5\.6\-Terra is accessed through a third\-party API provider, whereas DeepSeek and Qwen use their official endpoints\. For each TACS\-based system, we conduct three runs on the benchmark and report their mean\.

We use DeepSeek\-Harness\[[3](https://arxiv.org/html/2610.36527#bib.bib10)\], with shell execution, file editing and search, skill loading, subagent delegation, and web access enabled\. An initial prompt defines the task and instructs the agent to load the skill, after which execution proceeds automatically without human intervention\. The two sides of each benchmark item are canonicalized in separate fresh agent sessions with independent workspaces to avoid pairwise context sharing\.

### 4\.2Results

Table[3](https://arxiv.org/html/2610.36527#S3.T3)reports language\-wise performance\. The rule\-based systems score 3\.7–9\.4 on average over their supported languages\. FunASR\-TN’s 9\.4 is averaged over four languages\. TACS\-GPT, TACS\-DeepSeek, and TACS\-Qwen achieve macro\-average scores of 43\.3, 43\.8, and 68\.8, respectively, across all 12 languages\. Their language\-wise scores exceed those of the evaluated conventional systems wherever both have results\.

Table 4:Performance comparison across benchmark categories in terms of SENA \(%\)\.SystemNum\.Sym\.Abbr\.Orth\.Homo\.NeMo\-ITN16\.00\.66\.82\.80\.0NeMo\-TN23\.73\.26\.80\.00\.0FunASR\-ITN13\.31\.45\.11\.70\.0FunASR\-TN32\.95\.78\.20\.00\.0Whisper Normalizer6\.71\.03\.88\.17\.0No\-Skill\-GPT0\.30\.50\.07\.71\.9No\-Skill\-DeepSeek0\.00\.00\.08\.35\.3No\-Skill\-Qwen3\.07\.325\.619\.113\.0TACS\-GPT44\.129\.153\.455\.720\.0TACS\-DeepSeek50\.427\.156\.058\.512\.1TACS\-Qwen74\.444\.385\.982\.349\.0Category\-wise results in Table[4](https://arxiv.org/html/2610.36527#S4.T4)reveal a narrow success profile for the four TN/ITN systems\. Their successful cases are concentrated in numeric expressions and related forms\. Successful orthographic cases also include number\-related variants, such as French hyphenated numbers and Chinese financial numerals\. The four systems score only 0\.0–2\.8 on orthographic variants and zero on homophonic substitutions\. This pattern is consistent with their focus on written–spoken conversion within implemented grammar classes, rather than the broader transcript variation in ETC\-Bench\. Whisper shows a different profile\. Its English\-specific normalizer reaches 40\.9 on English, with successful cases including spelling and hyphenation variants\. The basic normalizer obtains modest scores in several other languages through surface transformations: hyphen removal in French and Vietnamese, width normalization in Japanese, lowercasing in Indonesian and Russian, and optional diacritic removal in Persian\. However, producing identical outputs does not always yield SENA credit\. In Bengali and Hindi, destructive removal of informative script marks also makes some pairs identical, but none of these pairs passes the semantic\-preservation condition\.

Across the TACS\-based systems, abbreviations and orthographic variants score highest, while symbols and homophonic substitutions remain more difficult\. TACS\-Qwen leads all five categories, but its scores on symbols and homophonic substitutions still leave substantial room for improvement\. The lower scores on Symbols and Markers and Homophonic Substitutions mainly reflect failures of exact agreement rather than rejection by the semantic\-preservation condition\. Across three TACS\-Qwen runs, the semantic\-preservation condition rejected none of the 262 symbol/marker pair evaluations with identical outputs, and only 8 of the 283 homophonic evaluations with identical outputs\.

To assess the contribution of TACS, we evaluate the same GPT, DeepSeek, and Qwen agent configurations without loading it\. They score 3\.5, 4\.6, and 13\.9, respectively, compared with 43\.3 for TACS\-GPT, 43\.8 for TACS\-DeepSeek, and 68\.8 for TACS\-Qwen\. three agents\. The observed rule\-based failures illustrate the roles of the two metric conditions\. Target variants that remain different fail the exact\-match condition\. Destructive cleanup can instead produce identical outputs while failing the semantic\-preservation condition, as observed in Bengali and Hindi\. In contrast, transformations such as*cent\-trois*/*cent trois*and*Agung*/*agung*unify the target variants without substantial additional changes and receive credit\. Thus, a simple transformation can be effective, but identical outputs alone do not establish successful canonicalization\. At the pre\-fixedδ=0\.10\\delta=0\.10, the semantic\-preservation condition rejects all 81 observed Whisper merges that remove informative script marks in Bengali and Hindi\. Atδ=0\.15\\delta=0\.15, 30\.9% of these merges receive credit\. In one agent run, 99\.8% of merged pairs pass the condition throughoutδ∈\[0\.08,0\.12\]\\delta\\in\[0\.08,0\.12\]\. Table[5](https://arxiv.org/html/2610.36527#S4.T5)illustrates how TACS resolves transcript variants left distinct by Whisper normalization on real ASR transcripts\[[12](https://arxiv.org/html/2610.36527#bib.bib12)\]in evaluation\.

Table 5:TACS\-DeepSeek unifies transcript variants left distinct by Whisper normalization in two real examples in ASR evaluation\.LIDWhisper \(Hyp\. / Ref\.\)TACS\-DeepSeekdeund sehrgewiß daßund sehrgewiss dassund sehrgewiss dassenthinking of youmabelandthinking of you/”meIb@l/andthinking of youmableand

## 5Conclusion

We define evaluation\-side transcript canonicalization as an independent task based on an existing evaluation practice\. We introduce ETC\-Bench, comprising 1,098 minimal pairs covering 181 phenomenon types across 12 languages, together with an extensible taxonomy and construction framework\. Experiments reveal limited coverage in conventional normalizers\. TACS shows how the taxonomy and canonicalization contracts can guide LLM agents toward broader coverage, although homophonic substitutions remain challenging\. We release the benchmark and skill and invite community contributions to extend language coverage, phenomenon inventories, and sample diversity\.

## 6compliance with ethical standards

This is a computational study which releases a new dataset\. Native\-speaker reviewers provided linguistic quality control but were not subjects of the research\. No human or animal subjects were studied, and ethical approval was not required\.

## References

- \[1\]A\. Ali, W\. Magdy, and S\. Renals\(2015\)Multi\-reference evaluation for dialectal speech recognition system: a study for egyptian ASR\.InProceedings of the Second Workshop on Arabic Natural Language Processing,pp\. 118–126\.External Links:[Document](https://dx.doi.org/10.18653/v1/W15-3213),[Link](https://aclanthology.org/W15-3213/)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p3.1)\.
- \[2\]A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov\(2020\)Unsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 8440–8451\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747)Cited by:[§4\.1](https://arxiv.org/html/2610.36527#S4.SS1.p1.1)\.
- \[3\]DeepSeek\-AIDeepSeek Harness: everything is a plugin\.GitHub\.Note:[https://github\.com/deepseek\-ai/deepseek\-harness](https://github.com/deepseek-ai/deepseek-harness)Cited by:[§4\.1](https://arxiv.org/html/2610.36527#S4.SS1.p2.1)\.
- \[4\]DeepSeek\-AI\(2026\)DeepSeek\-V4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.External Links:2606\.19348,[Link](https://arxiv.org/abs/2606.19348)Cited by:[§4\.1](https://arxiv.org/html/2610.36527#S4.SS1.p1.1)\.
- \[5\]A\. Deviyani and A\. W\. Black\(2022\)Text normalization for speech systems for all languages\.InProceedings of the 1st Workshop on Speech for Social Good \(S4SG\),pp\. 20–25\.External Links:[Document](https://dx.doi.org/10.21437/S4SG.2022-5),[Link](https://www.isca-archive.org/s4sg_2022/deviyani22_s4sg.html)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p2.1)\.
- \[6\]Y\. Fang, J\. Peng, X\. Li, Y\. Xi, C\. Zhang, G\. Zhong, and K\. Yu\(2025\)Low\-resource domain adaptation for speech llms via text\-only fine\-tuning\.In2025 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 1–7\.Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p1.1)\.
- \[7\]J\. G\. Fiscus, J\. Ajot, N\. Radde, and C\. Laprun\(2006\)Multiple dimension levenshtein edit distance calculations for evaluating automatic speech recognition systems during simultaneous speech\.InProceedings of the Fifth International Conference on Language Resources and Evaluation \(LREC\),Genoa, Italy\.External Links:[Link](https://aclanthology.org/L06-1107/)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p3.1)\.
- \[8\]E\. Flint, E\. Ford, O\. Thomas, A\. Caines, and P\. Buttery\(2017\)A text normalisation system for non\-standard English words\.InProceedings of the 3rd Workshop on Noisy User\-generated Text,pp\. 107–115\.External Links:[Document](https://dx.doi.org/10.18653/v1/W17-4414),[Link](https://aclanthology.org/W17-4414/)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p2.1)\.
- \[9\]Z\. Gao, Z\. Li, J\. Wang, H\. Luo, X\. Shi, M\. Chen, Y\. Li, L\. Zuo, Z\. Du, and S\. Zhang\(2023\)FunASR: a fundamental end\-to\-end speech recognition toolkit\.InProceedings of Interspeech,pp\. 1593–1597\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2023-1428),[Link](https://www.isca-archive.org/interspeech_2023/gao23g_interspeech.html)Cited by:[§4\.1](https://arxiv.org/html/2610.36527#S4.SS1.p1.1)\.
- \[10\]K\. Gorman\(2016\)Pynini: a Python library for weighted finite\-state grammar compilation\.InProceedings of the SIGFSM Workshop on Statistical NLP and Weighted Automata,pp\. 75–80\.External Links:[Document](https://dx.doi.org/10.18653/v1/W16-2409),[Link](https://aclanthology.org/W16-2409/)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p2.1)\.
- \[11\]S\. Kim, A\. Arora, D\. Le, C\. Yeh, C\. Fuegen, O\. Kalinli, and M\. L\. Seltzer\(2021\)Semantic distance: a new metric for ASR performance analysis towards spoken language understanding\.InProceedings of Interspeech,pp\. 1977–1981\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-1929),[Link](https://www.isca-archive.org/interspeech_2021/kim21e_interspeech.html)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p1.1)\.
- \[12\]J\. Li, J\. Peng, Y\. Fang, S\. Wang, and K\. Yu\(2026\)MOSA: mixtures of simple adapters outperform monolithic approaches in LLM\-based multilingual ASR\.InICASSP 2026\-2026 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 18822–18826\.External Links:[Link](https://arxiv.org/abs/2508.18998)Cited by:[§2\.1](https://arxiv.org/html/2610.36527#S2.SS1.p1.1),[§4\.2](https://arxiv.org/html/2610.36527#S4.SS2.p4.1)\.
- \[13\]K\. Manohar and L\. G\. Pillai\(2024\)What is lost in normalization? exploring pitfalls in multilingual ASR model evaluations\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 10864–10869\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.607),[Link](https://aclanthology.org/2024.emnlp-main.607/)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p3.1)\.
- \[14\]I\. Nigmatulina, T\. Kew, and T\. Samardzic\(2020\)ASR for non\-standardised languages with dialectal variation: the case of Swiss German\.InProceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects,pp\. 15–24\.External Links:[Link](https://aclanthology.org/2020.vardial-1.2/)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p1.3)\.
- \[15\]Open\-Bench\(n\.d\.\)Open\-Bench: Open Bench, Open Innovation\.Note:[https://www\.open\-bench\.net/](https://www.open-bench.net/)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p3.1)\.
- \[16\]J\. Peng, J\. Du, C\. Wang, H\. Li, Y\. Yang, Y\. Wang, X\. Gu, G\. Chen, Y\. Wang, J\. Li,et al\.\(2026\)A unified and reproducible experimentation framework for speech understanding\.arXiv preprint arXiv:2605\.30899\.Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p3.1)\.
- \[17\]J\. Peng, Y\. Wang, B\. Li, Y\. Guo, H\. Wang, Y\. Fang, Y\. Xi, H\. Li, X\. Li, K\. Zhang,et al\.\(2025\)A survey on speech large language models for understanding\.IEEE Journal of Selected Topics in Signal Processing\.Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p1.1)\.
- \[18\]Qwen Team\(2026\)Qwen3\.8\-max: a new bar for coding and cowork\.External Links:[Link](https://qwen.ai/blog?id=qwen3.8)Cited by:[§4\.1](https://arxiv.org/html/2610.36527#S4.SS1.p1.1)\.
- \[19\]A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. Mcleavey, and I\. Sutskever\(2023\)Robust speech recognition via large\-scale weak supervision\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 28492–28518\.External Links:[Link](https://proceedings.mlr.press/v202/radford23a.html)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.36527#S4.SS1.p1.1)\.
- \[20\]R\. Sproat, A\. W\. Black, S\. Chen, S\. Kumar, M\. Ostendorf, and C\. D\. Richards\(2001\)Normalization of non\-standard words\.Computer Speech & Language15\(3\),pp\. 287–333\.External Links:[Document](https://dx.doi.org/10.1006/csla.2001.0169),[Link](https://www.sciencedirect.com/science/article/pii/S088523080190169X)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p2.1)\.
- \[21\]R\. Sproat and N\. Jaitly\(2017\)An RNN model of text normalization\.InProceedings of Interspeech,pp\. 754–758\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2017-35),[Link](https://www.isca-archive.org/interspeech_2017/sproat17_interspeech.html)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p2.1)\.
- \[22\]V\. Srivastav, S\. Zheng, E\. Bezzam, E\. Le Bihan, N\. R\. Koluguri, P\. Żelasko, S\. Majumdar, A\. Moumen, and S\. Gandhi\(2025\)Open ASR leaderboard: towards reproducible and transparent multilingual and long\-form speech recognition evaluation\.arXiv preprint arXiv:2510\.06961\.External Links:2510\.06961,[Document](https://dx.doi.org/10.48550/arXiv.2510.06961),[Link](https://arxiv.org/abs/2510.06961)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p1.3)\.
- \[23\]J\. Tobin, Q\. Li, S\. Venugopalan, K\. Seaver, R\. Cave, and K\. Tomanek\(2022\)Assessing ASR model quality on disordered speech using BERTScore\.InProceedings of the 1st Workshop on Speech for Social Good \(S4SG\),pp\. 26–30\.External Links:[Document](https://dx.doi.org/10.21437/S4SG.2022-6),[Link](https://www.isca-archive.org/s4sg_2022/tobin22_s4sg.html)Cited by:[§2\.3](https://arxiv.org/html/2610.36527#S2.SS3.p2.1)\.
- \[24\]M\. Wong, A\. Alshehri, S\. Kao, and H\. He\(2025\)PolyNorm: few\-shot LLM\-based text normalization for text\-to\-speech\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,pp\. 77–85\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-industry.6),[Link](https://aclanthology.org/2025.emnlp-industry.6/)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.36527#S2.SS1.p2.1)\.
- \[25\]T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi\(2020\)BERTScore: evaluating text generation with BERT\.InProceedings of the International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkeHuCVFDr)Cited by:[§2\.3](https://arxiv.org/html/2610.36527#S2.SS3.p2.1)\.
- \[26\]Y\. Zhang, E\. Bakhturina, K\. Gorman, and B\. Ginsburg\(2021\)NeMo inverse text normalization: from development to production\.InProceedings of Interspeech,pp\. 4468–4472\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-1571),[Link](https://www.isca-archive.org/interspeech_2021/zhang21ga_interspeech.html)Cited by:[§1](https://arxiv.org/html/2610.36527#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.36527#S4.SS1.p1.1)\.