When Multiple Scripts Matter: Evaluating ASR in Clinical Settings

# Evaluating ASR in Clinical Settings
Source: [https://arxiv.org/html/2606.17826](https://arxiv.org/html/2606.17826)
Seo Kim Lee Jung Han Yang

## When Multiple Scripts Matter: Evaluating ASR in Clinical Settings

###### Abstract

Automatic speech recognition \(ASR\) in non\-English clinical settings is challenged by multiscript variability, where the same term may appear in multiple valid orthographic forms\. Conventional string\-matching evaluation metrics often underestimate ASR performance by treating orthographic variants as errors\. To address this issue, we introduceMultiClin, a clinical ASR benchmark designed to evaluate robustness to multiscript variability\. Experiments across diverse ASR models show that multiscript\-aware evaluation provides a fairer assessment of recognition quality than conventional single\-reference evaluation\. We further investigate the impact of script consistency during training and find that inconsistent script mappings increase orthographic uncertainty and hinder model convergence, with a balanced 50% mapping ratio producing the highest entropy\. In contrast, script unification consistently yields the best ASR performance\. Our dataset and code are publicly available at:[https://github\.com/aitrics\-ronaldo/Interspeech\_MultiClin](https://github.com/aitrics-ronaldo/Interspeech_MultiClin)\.

###### keywords:

automatic speech recognition, evaluation, multiscript variability, code\-switching, healthcare

## 1Introduction

Automatic speech recognition \(ASR\) is increasingly adopted in clinical settings to improve workflow efficiency\[xu2025enhancing,alboksmaty2025impact,tran2023automatic\]\. However, domain\-specific terminology and noisy environments continue to challenge clinical ASR\. These difficulties are further amplified in non\-English settings, where English medical terminology frequently coexists with phonetic renderings in local scripts\[agro2025codeswitchingendtoendautomaticspeech\]\. A central obstacle to reliable benchmarking in such environments is*multiscript variability*, where a single spoken term may correspond to multiple valid orthographic forms \(e\.g\., English spelling or a phonetic rendering in the local script\)\. Unlike conventional code\-switching, which involves acoustic alternation between languages, multiscript variability arises from orthographic variation despite an identical acoustic realization\.

Conventional ASR evaluation assumes a single reference transcription per utterance\. However, this assumption often breaks down in non\-English clinical settings, where English\-origin medical terms lack standardized localization guidelines and may be transcribed in multiple valid forms\. This many\-to\-one mapping between orthography and speech invalidates strict string\-based metrics such as word error rate \(WER\), systematically penalizing outputs that are phonetically and semantically correct but orthographically different from the reference\[Mustafa2022CodeSwitchingIA,Srivastava2018HomophoneIA,Chowdhury2020EffectsOD\]\. Moreover, normalization\-based solutions remain impractical due to inconsistent clinical documentation practices and the scarcity of standardized domain\-specific corpora\. While multilingual ASR research has extensively studied code\-switching\[Nakayama2019ZeroShotCA\], prior work has largely focused on modeling and data augmentation\[kumar2021dual,Li2019TowardsCA,Yilmaz2017LanguageDF\]rather than evaluation\. Existing benchmarks typically rely on a single ground\-truth reference\[Hamed2022BenchmarkingEM,Paik2025HiKEHE\], while transliteration\-based approaches\[Emond2018TransliterationBA\]and metrics such as transliterated WER \(T\-WER\)\[Chowdhury2021TowardsOM,Ali2015MultiReferenceEF\]have primarily been evaluated on general\-domain code\-switching and dialectal variation, leaving clinical multiscript settings largely unexplored\.

Table 1:Example of the original, tagged, and translated dialogue from theMultiClindataset\.To address this gap, we introduceMultiClin, a clinical ASR benchmark that provides multiple valid transcription variants for multiscript terminology\. Through a Korean clinical case study, we demonstrate that dynamic multi\-reference evaluation yields a fairer assessment of ASR performance under orthographic variability\.

Table 2:Statistics of theMultiClindataset\.A\. Filtering Stages \(Initial→\\toFinal\)B\. Avg\. tagged instances per dialogueACI Bench126→116126\\to 116MedicalTags44Primock57186→9186\\to 9NumberTags6MTS\-Dialog1,175→1911,175\\to 191UnitTags1Total Dialogues1,487→3161,487\\to 316

## 2MultiClin dataset

We construct theMultiClindataset to reflect real\-world clinical ASR challenges\. Table[1](https://arxiv.org/html/2606.17826#S1.T1)illustrates an example data corresponding to each phase of the annotation process\.

### 2\.1Dataset construction

#### 2\.1\.1Collection

We collect publicly available doctor–patient dialogues from ACIBench\[yim2023acibenchnovelambientclinical\], Primock57\[papadopoulos\-korfiatis\-etal\-2022\-primock57\], and MTS\-Dialog\[mts\-dialog\]\. To ensure natural clinical conversations, we exclude dialogues involving virtual assistants and retain only interactions between doctors and patients, resulting in an initial corpus of 1,487 dialogues\.

#### 2\.1\.2Annotation

The dataset undergoes three processing stages: tagging, translation, and human annotation\. We use gpt\-5\.2111[https://openai\.com/](https://openai.com/)to identify script\-switching instances and assign them to three categories:Medical,Unit, andNumber\.Medicaltags denote English\-origin medical terms appearing either in the Roman alphabet or as phonetic loanwords\.Unittags represent measurement units expressed in native scripts or standardized symbols \(e\.g\., %, cm\), whileNumbertags capture numerical expressions written in native scripts or Arabic numerals\. We then translate the dialogues into Korean using the same model\. Tagged spans preserve their original form and are augmented with Korean\-script \(Hangeul\) renderings, separated by commas without spaces\. For example, \`\`You need an <medical\>injection</medical\>\.'' becomes \`\`<medical\>injection,인젝션</medical\>이 필요합니다\.'' In other words, tagged entities undergo transliteration, preserving lexical identity while changing only the script, whereas the remaining text undergoes full translation into Korean\. Finally, two annotators with nursing backgrounds review all dialogues for orthographic correctness, translation fidelity, and naturalness\. Any disagreements or errors are resolved through consensus, resulting in the final curated dataset\.

#### 2\.1\.3Speech generation

To comply with the Health Insurance Portability and Accountability Act \(HIPAA\) restrictions on releasing real\-world clinical audio, we synthesize dialogues using gpt\-4o\-mini\-tts\. We map speaker roles to distinct speaking styles \(e\.g\., professional tones for doctors and lethargic tones for patients\) and apply accent\-aware prompting to align multiscript spans with native intonation patterns\. To reduce the acoustic mismatch between synthetic and real clinical speech, we incorporate human\-like conversational dynamics, including overlaps and response latencies, and simulate clinical environments using a DSP chain222[https://github\.com/spotify/pedalboard](https://github.com/spotify/pedalboard)\(e\.g\., reverb and HVAC noise\)\. All audio is resampled to16,kHz16,\\text\{kHz\}\.

Table 3:Clinical specialty distribution inMultiClin\.Other∗encompasses 8 minor fields \(e\.g\., Pain Management, Dentistry, Plastic Surgery\)\.

### 2\.2Statistics

Dataset filtering\.From the initial 1,487 dialogues, we retain 1,417 instances containing at least oneMedical,Number, orUnittag\. We then manually remove unnatural or hallucinated conversations, resulting in 316 final dialogues \(Table[2](https://arxiv.org/html/2606.17826#S1.T2)A\)\.

Tag and dialogue statistics\.Medicalterminology dominates script\-switching instances \(Table[2](https://arxiv.org/html/2606.17826#S1.T2)B\)\. Each dialogue contains 34 turns and 68 sentences on average, with per\-speaker utterance statistics reported in Table[2](https://arxiv.org/html/2606.17826#S1.T2)C\.

Speaker composition\.Most dialogues involve a single doctor and a single patient \(Table[2](https://arxiv.org/html/2606.17826#S1.T2)D\)\. In cases where a patient is absent, guardians \(Guest Family\) speak on their behalf\.

Clinical specialty distribution\.All dialogues across the three sources are accompanied by structured clinical notes \(e\.g\., SOAP notes\)\. Using gpt\-5\.2, we infer the primary clinical specialty of each dialogue from this metadata \(Table[3](https://arxiv.org/html/2606.17826#S2.T3)\)\.

Table 4:Performance of baseline models\.

## 3Experiments

We evaluate ASR performance on theMultiClinbenchmark to quantify the impact of multiscript variability\. We analyze zero\-shot inference across diverse architectures and assess the effects of domain\-specific fine\-tuning under different labeling strategies\.

### 3\.1Experimental setup

#### 3\.1\.1Baseline Models

We consider three model families as baselines: \(1\)Whisper\[whisper\]\(large\-v3,v3\-turbo\), implemented via faster\-whisper333[https://github\.com/SYSTRAN/faster\-whisper](https://github.com/SYSTRAN/faster-whisper); \(2\)Qwen3 ASR\[qwen3\-asr\]\(0\.6B,1\.7B\); and \(3\)Gemini\[gemini\]\(2\.5 Flash,2\.5 Pro\), representing frontier multimodal state\-of\-the\-art models\.

#### 3\.1\.2Inference Configuration

We detail the zero\-shot inference configurations for our multimodal baselines to ensure reproducibility\.

Gemini prompting strategy\.We query the Gemini models using a structured zero\-shot prompt\. We instruct the model to act as a professional medical stenographer and produce verbatim transcriptions, explicitly prohibiting speaker diarization, speaker prefixes, and summarization\. To ensure deterministic and parseable outputs, we set the sampling temperature to 0\.0 and enforce a JSON output format, from which we extract the transcript as an array of sentences\.

Qwen inference setting\.For Qwen3 ASR models, we accommodate long clinical dialogues by setting the maximum generation length to 65,536 tokens\. To improve memory efficiency and avoid out\-of\-memory \(OOM\) errors during long\-form audio processing, we limit the maximum inference batch size to 32\.

#### 3\.1\.3Fine\-tuning Configuration

For the fine\-tuning experiments, we train Whisper models using LoRA\[lora\]\. We split theMultiClindataset into a 9:1 ratio to construct an independent test set\. Importantly, we apply a 100% transliteration ratio, in which all taggedMedical,Number, andUnitentities are consistently unified into the local script to maximize labeling consistency\. This setup reduces orthographic ambiguity during the learning phase\. Finally, models are trained for 4 epochs with a batch size of 4\.

#### 3\.1\.4Evaluation Protocol

To enable more accurate evaluation of ASR performance under multiscript settings, we introduce a localized evaluation metric \(Algorithm[1](https://arxiv.org/html/2606.17826#alg1)\) that treats both the original English medical term and its phonetic rendering in the local script as valid references\. Specifically, for each script\-switching entity in the reference transcript, we dynamically extract a 50\-character window from the ASR prediction𝐲^\\mathbf\{\\hat\{y\}\}using a tracking cursor\. To mitigate temporal misalignment, we apply Longest Common Substring \(LCS\) matching between the target entity and the corresponding predicted window\. We then compute local CER and WER within these aligned boundaries, reducing the influence of surrounding transcription errors and enabling a more robust comparison of entity\-level correctness across orthographic variants\.

Algorithm 1Dynamic Multiscript Reference ResolutionInput:Tagged reference𝐲t​a​g\\mathbf\{y\}\_\{tag\}, ASR hypothesis𝐲^\\mathbf\{\\hat\{y\}\}, Window sizeW=50W=50, Mode mappingℳ∈\{original,both\}\\mathcal\{M\}\\in\\\{\\text\{original\},\\text\{both\}\\\} Output:Dynamically resolved reference𝐲f​i​n​a​l\\mathbf\{y\}\_\{final\}

1:

c​u​r​s​o​r←0cursor\\leftarrow 0
2:

𝐲f​i​n​a​l←𝐲t​a​g\\mathbf\{y\}\_\{final\}\\leftarrow\\mathbf\{y\}\_\{tag\}
3:foreach entity tuple

\(t,eo​r​i​g,et​g​t\)\(t,e\_\{orig\},e\_\{tgt\}\)in

𝐲t​a​g\\mathbf\{y\}\_\{tag\}do

4:

m←ℳ​\[t\]m\\leftarrow\\mathcal\{M\}\[t\]\{Fetch evaluation mode for tag type

tt\}

5:if

m=originalm=\\text\{original\}then

6:Replace tag with

eo​r​i​ge\_\{orig\}in

𝐲f​i​n​a​l\\mathbf\{y\}\_\{final\}
7:elseif

m=bothm=\\text\{both\}then

8:if

c​u​r​s​o​r≥\|𝐲^\|cursor\\geq\|\\mathbf\{\\hat\{y\}\}\|then

9:Replace tag with

eo​r​i​ge\_\{orig\}in

𝐲f​i​n​a​l\\mathbf\{y\}\_\{final\}
10:else

11:

𝐲^w​i​n←𝐲^\[cursor:min\(cursor\+W,\|𝐲^\|\)\]\\mathbf\{\\hat\{y\}\}\_\{win\}\\leftarrow\\mathbf\{\\hat\{y\}\}\[cursor:\\min\(cursor\+W,\|\\mathbf\{\\hat\{y\}\}\|\)\]
12:

c​e​ro​r​i​g,o​f​f​s​e​to​r​i​g←LocalCER​\(eo​r​i​g,𝐲^w​i​n\)cer\_\{orig\},offset\_\{orig\}\\leftarrow\\text\{LocalCER\}\(e\_\{orig\},\\mathbf\{\\hat\{y\}\}\_\{win\}\)
13:

c​e​rt​g​t,o​f​f​s​e​tt​g​t←LocalCER​\(et​g​t,𝐲^w​i​n\)cer\_\{tgt\},offset\_\{tgt\}\\leftarrow\\text\{LocalCER\}\(e\_\{tgt\},\\mathbf\{\\hat\{y\}\}\_\{win\}\)
14:\{Priority selection based on minimal local error\}

15:if

c​e​rt​g​t<c​e​ro​r​i​gcer\_\{tgt\}<cer\_\{orig\}then

16:Replace tag with

et​g​te\_\{tgt\}in

𝐲f​i​n​a​l\\mathbf\{y\}\_\{final\}
17:

c​u​r​s​o​r←c​u​r​s​o​r\+o​f​f​s​e​tt​g​tcursor\\leftarrow cursor\+offset\_\{tgt\}
18:else

19:Replace tag with

eo​r​i​ge\_\{orig\}in

𝐲f​i​n​a​l\\mathbf\{y\}\_\{final\}
20:

c​u​r​s​o​r←c​u​r​s​o​r\+o​f​f​s​e​to​r​i​gcursor\\leftarrow cursor\+offset\_\{orig\}
21:endif

22:endif

23:endif

24:endfor

25:return

𝐲f​i​n​a​l\\mathbf\{y\}\_\{final\}
26:Function

LocalCER​\(e,𝐰\)\\text\{LocalCER\}\(e,\\mathbf\{w\}\)
27:

LCS←FindLongestMatch​\(e,𝐰\)\\text\{LCS\}\\leftarrow\\text\{FindLongestMatch\}\(e,\\mathbf\{w\}\)
28:

𝐰s​u​b←𝐰\[LCSs​t​a​r​t:LCSe​n​d\]\\mathbf\{w\}\_\{sub\}\\leftarrow\\mathbf\{w\}\[\\text\{LCS\}\_\{start\}:\\text\{LCS\}\_\{end\}\]
29:

c​e​r←ComputeCER​\(e,𝐰s​u​b\)cer\\leftarrow\\text\{ComputeCER\}\(e,\\mathbf\{w\}\_\{sub\}\)
30:return

c​e​r,LCSe​n​dcer,\\text\{LCS\}\_\{end\}

### 3\.2Inference results

Table[4](https://arxiv.org/html/2606.17826#S2.T4)presents zero\-shot inference performance across different script evaluation settings\. A consistent trend emerges: moving from strict single\-label matching \(original\) to multiscript\-aware evaluation \(both\) yields substantial reductions in error rates across all models\. For instance, Gemini 2\.5 Pro's WER decreases from 28\.28% to 15\.78% when medical terms are evaluated with multiscript flexibility\. These results empirically support the claim that conventional string\-based metrics systematically underestimate ASR performance by failing to account for valid orthographic variation in clinical settings\. Our proposed benchmark exposes this limitation by revealing the true capabilities of ASR models\. WhileMedicaltags contribute most to the observed performance gap, model scale also plays a significant role; Qwen3 ASR 1\.7B achieves a 37\.01% WER under the full multiscript\-aware setting \(both\)\. Among open\-source systems, Whisper v3 Turbo demonstrates the strongest robustness, achieving a 23\.00% WER\. Overall, Gemini 2\.5 Pro attains the best CER of 4\.86%\. By properly accounting for medical\-domain orthographic variation,MultiClinprovides a more fair and informative evaluation framework for multiscript clinical ASR\.

Table 5:Detailed CER \(%\) comparison between pre\-trained and fine\-tuned Whisper models\. Parentheses indicate the absolute reduction in CER after fine\-tuning on theMultiClindataset\.Table 6:Impact of transliteration ratio in the training dataset\. Results show the performances of fine\-tuned Whisper large v3 models on the held\-out test set\.
### 3\.3Fine\-tuning results

Table[5](https://arxiv.org/html/2606.17826#S3.T5)summarizes the performance gains from fine\-tuning on the independent test set\. Training with a 100% transliteration ratio yields substantial improvements across all evaluation settings\. Notably, Whisper\-Large v3 Turbo achieves a best\-in\-class CER of 6\.16%, corresponding to an absolute reduction of 3\.83%p over its pre\-trained baseline\. Even larger gains are observed for the standard Whisper\-Large v3 model, with CER decreasing by up to 6\.33%p under multiscript\-aware evaluation criteria\. These consistent improvements across both architectures empirically demonstrate that full script unification is an effective strategy for mitigating orthographic ambiguity in clinical ASR\.

### 3\.4Impact of labeling consistency

We further investigate the impact of labeling consistency in the training data\. As shown in Table[6](https://arxiv.org/html/2606.17826#S3.T6), the 0% transliteration ratio—where all tagged entities are represented exclusively in the Roman alphabet or Arabic numerals while the rest of the utterance is written in Korean \(Hangeul\)—produces the highest error rates on the held\-out test set \(69\.17% CER and 54\.35% WER\)\. As the transliteration ratio increases, meaning that a larger proportion of entities are represented in Hangeul, performance exhibits a non\-monotonic pattern, with a secondary error peak at the 50% ratio \(57\.47% CER and 48\.50% WER\)\. This performance degradation confirms that inconsistent script mapping introduces orthographic ambiguity, maximizing the conditional entropyH​\(Y\|X\)H\(Y\|X\)for a given acoustic featureXX:

H​\(Y\|X\)=−∑y∈𝒴P​\(y\|X\)​log⁡P​\(y\|X\)H\(Y\|X\)=\-\\sum\_\{y\\in\\mathcal\{Y\}\}P\(y\|X\)\\log P\(y\|X\)At the 50% ratio, the model faces maximum epistemic uncertainty between competing scripts, which disrupts internal alignment and prevents the decoder from forming stable decision boundaries\. Ultimately, the 100% ratio resolves the script alternation complexity, yielding the most robust performance \(7\.66% CER, 17\.48% WER\)\. This validates that full script unification is essential for providing a deterministic learning signal\.

## 4Conclusion

This work introduces theMultiClindataset for fairer evaluation in non\-English clinical ASR\. Our experiments show that multiscript\-aware criteria provide a fairer assessment than traditional single\-label metrics, which often underestimate true model performance\. We further demonstrate that labeling consistency in the training data is essential for better performance\. Future work should examine how these ASR improvements influence downstream clinical tasks, such as entity extraction and SOAP note generation\.

## 5Generative AI Use Disclosure

This work employs Generative AI tools including Google Gemini and OpenAI ChatGPT\. Gemini is utilized for linguistic refinement, including grammatical correction and improving the clarity of the initial manuscript\. Furthermore, both Gemini and ChatGPT were integrated into our data construction process to generate synthetic clinical dialogues for theMultiClindataset, addressing the inherent data scarcity and privacy constraints of the medical domain\. We emphasize that the AI tools are used solely under human supervision\. All AI\-generated datasets are rigorously reviewed and validated by the authors for clinical accuracy and ethical compliance\. We maintain full responsibility for the final content and the integrity of the published work\.

## References