HealMed: Multilingual Evaluation of Large Language Models in Medicine

# HealMed: Multilingual Evaluation of Large Language Models in Medicine
Source: [https://arxiv.org/html/2608.19981](https://arxiv.org/html/2608.19981)
HealMed Research TeamAffiliation:See the Contributions section for the complete list of contributors and affiliations\.Affiliation:\[4pt\][HealMed](https://huggingface.co/datasets/li-lab/HealMed)

###### Abstract

We present HealMed, an expert\-reviewed benchmark for multilingual evaluation of large language models in medicine\. HealMed contains 1,000 examples in each of nine languages, drawn from nine datasets and covering three task formats: MCQA, NLI and open\-ended QA\. The benchmark was developed over two years by 23 physicians and medical experts based across nine countries and regions\. Each translation was evaluated and revised by two experts fluent in English and the corresponding target language\. On HealMed, performance declined most in low\-resource languages, although the size of the gap varied markedly across languages and models\. The strongest proprietary models were the most stable across languages, whereas many open\-source and medically specialized models showed larger and less consistent gaps\. Medical specialization alone did not ensure multilingual robustness\. Furthermore, expert revision could either raise or lower measured performance, indicating that translation quality materially affects cross\-language evaluation results\.

## 1Introduction

Large language models \(LLMs\) have achieved strong performance on questions from medical licensing examinations and can generate long\-form medical responses that receive favourable clinical ratings[Singhal et al\. 2023](https://arxiv.org/html/2608.19981#bib.bib28);[Singhal et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib29)\. Yet evidence for these capabilities remains centred on English[Ahuja et al\. 2023](https://arxiv.org/html/2608.19981#bib.bib2);[Wu et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib34);[Xuan et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib35)\. Performance on English benchmarks does not establish whether a model can interpret medical concepts or communicate them reliably in other languages, especially those underrepresented in training data and existing evaluations[Chen et al\. 2025b](https://arxiv.org/html/2608.19981#bib.bib10);[Singhal et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib29)\. Multilingual assessment is therefore needed to determine where medical competence transfers across languages and where it fails\.

Several multilingual medical benchmarks now cover question answering and clinical language understanding[Qiu et al\. 2024](https://arxiv.org/html/2608.19981#bib.bib24);[Alonso et al\. 2024](https://arxiv.org/html/2608.19981#bib.bib3);[Wu et al\. 2026](https://arxiv.org/html/2608.19981#bib.bib33)\. Many, however, are built by translating English test sets\. Translation can alter medical terminology, syntax and contextual meaning\. These changes can affect model scores and rankings[Artetxe et al\. 2020](https://arxiv.org/html/2608.19981#bib.bib7);[Singh et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib27)\. A lower score in a target language may therefore reflect weaker model capability, translation error or both\. Without expert review, these sources of error cannot be separated\.

Figure 1:Overview of HealMed\.a\. Benchmark scope, comprising MCQA, NLI and open\-ended QA across nine languages and nine source datasets\.b\. Construction pipeline, including English source\-example selection, machine translation into eight target languages, two\-stage review and revision by bilingual medical experts, and final consistency checking and curation\.In this paper, we presentHealMed:Human\-verifiedEvaluationAcrossLanguages forMedical AI, an expert\-reviewed medical benchmark spanning nine languages, nine source datasets and three task families: multiple\-choice question answering \(MCQA\), natural language inference \(NLI\) and open\-ended question answering \(QA\)\. 23 physicians and medical experts contributed to its construction and expert review\. Every target\-language instance underwent a structured two\-stage review by two medical experts fluent in English and the corresponding target language\. We evaluated 14 LLMs on MCQA and NLI and a subset of ten models on open\-ended QA\. The panel comprised five proprietary models \(GPT\-5\.4, o4\-mini, Gemini\-3\-Flash, Claude\-Sonnet\-5 and Claude\-Opus\-4\.8\)[OpenAI 2026](https://arxiv.org/html/2608.19981#bib.bib23);[OpenAI 2025](https://arxiv.org/html/2608.19981#bib.bib22);[Google DeepMind 2025](https://arxiv.org/html/2608.19981#bib.bib15);[Anthropic 2026b](https://arxiv.org/html/2608.19981#bib.bib5);[Anthropic 2026a](https://arxiv.org/html/2608.19981#bib.bib4), six general\-purpose open\-source model configurations from the DeepSeek[DeepSeek\-AI 2024](https://arxiv.org/html/2608.19981#bib.bib12), Qwen[Yang et al\. 2024](https://arxiv.org/html/2608.19981#bib.bib37);[Yang et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib36), LLaMA[Grattafiori et al\. 2024](https://arxiv.org/html/2608.19981#bib.bib16)and Gemma[Gemma Team et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib14)families, and three medically specialized models: HuatuoGPT\-o1[Chen et al\. 2025a](https://arxiv.org/html/2608.19981#bib.bib9), MedGemma[Sellergren et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib26)and MediPhi[Corbeil et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib11)\. Open\-ended responses were assessed using a multilingual LLM\-as\-judge protocol\. The original machine\-translated and final expert\-reviewed versions were retained for paired evaluation\.

We find thatthe strongest proprietary models combine high accuracy with stable performance across languages, whereas the evaluated open\-source and medically specialized models show larger losses in lower\-resource languages\. Several models in the latter groups perform strongly in higher\-resource languages but decline sharply in lower\-resource languages, indicating that medical specialization alone does not close the language gap\.All evaluated models perform worse on MCQA and NLI in the lower\-resource languages\. These losses are concentrated in Swahili and Zulu, while performance in Thai remains close to that in Japanese and Chinese\.Open\-ended QA shows the same broad pattern: GPT\-5\.4 and Gemini\-3\-Flash remain stable across resource groups, but all eight non\-proprietary models decline in the lower\-resource group\. The largest declines coincide mainly with problems in medical terminology, fluency and language choice\.

Paired comparisons show that machine\-translated and expert\-reviewed data can yield different estimates of multilingual medical performance\. Expert revision raises measured performance in some language–dataset combinations and lowers it in others, with the magnitude of these shifts also varying across settings\. In open\-ended QA, the expert\-reviewed data yield lower scores in most language–dataset combinations\. This asymmetry raises the possibility that the LLM evaluator is more closely aligned with the literal wording of machine\-translated questions and reference answers\. Machine translation may therefore introduce measurement bias and may not fully reflect model performance on medical questions expressed naturally in the target language\.

## 2HealMed Dataset

HealMed is an expert\-reviewed multilingual medical benchmark for evaluating medical AI systems across nine languages, three task families and nine source datasets\. Figure[1](https://arxiv.org/html/2608.19981#S1.F1)summarizes the benchmark scope and construction pipeline\. A detailed comparison of HealMed with selected multilingual medical benchmarks, including their scope, construction and human\-review procedures, is provided in Appendix[B](https://arxiv.org/html/2608.19981#A2)\.

### 2\.1Benchmark Scope and Composition

We selected 1,000 English examples in total from the nine source datasets\. Machine\-translated versions for 7 target languages were obtained from GlobMed111[https://huggingface\.co/collections/ruiyang\-medinfo/globmed](https://huggingface.co/collections/ruiyang-medinfo/globmed), whereas the Thai translations were generated using GPT\-5\.5 through the Azure OpenAI API with zero\-shot prompting\. Every translated instance was subsequently reviewed by two medical experts fluent in both English and the corresponding target language\. HealMed therefore covers nine languages: English, German, Spanish, Portuguese, Japanese, Chinese, Thai, Swahili and Zulu\.

HealMed covers three complementary task formats selected to evaluate medical language models under different output constraints, from fixed\-choice prediction and relation classification to free\-form generation\. The MCQA component comprises HeadQA[Vilares and Gómez\-Rodríguez 2019](https://arxiv.org/html/2608.19981#bib.bib30), MedQA[Jin et al\. 2021](https://arxiv.org/html/2608.19981#bib.bib18), MedExpQA[Alonso et al\. 2024](https://arxiv.org/html/2608.19981#bib.bib3)and MMLU\-Pro[Wang et al\. 2024b](https://arxiv.org/html/2608.19981#bib.bib32)and requires models to select the correct answer from a fixed set of options\. The NLI component comprises BioNLI[Bastan et al\. 2022](https://arxiv.org/html/2608.19981#bib.bib8)and MedNLI[Romanov and Shivade 2018](https://arxiv.org/html/2608.19981#bib.bib25)and requires models to classify the logical relation between paired biomedical or clinical statements as entailment, contradiction or, where applicable, neutral\. The open\-ended QA component comprises ExpertQA\-Bio, ExpertQA\-Med[Malaviya et al\. 2024](https://arxiv.org/html/2608.19981#bib.bib20)and LiveQA[Abacha et al\. 2017](https://arxiv.org/html/2608.19981#bib.bib1)and requires models to generate free\-form answers\. Further details are provided in Appendix[A](https://arxiv.org/html/2608.19981#A1)\. All components use a common sampling, translation and expert\-review procedure\.

### 2\.2Construction and Expert Review

Each target\-language instance underwent a two\-stage review by two medical experts fluent in English and the corresponding target language \(Figure\.[1](https://arxiv.org/html/2608.19981#S1.F1)b\)\. Both experts compared the English source with the original machine translation and rated accuracy, fluency and completeness on a five\-point scale\. Accuracy captured semantic fidelity and the appropriate use of medical terminology; fluency captured grammaticality, naturalness and professional usage; and completeness captured the preservation of source information without omissions or unsupported additions\. Scores ranged from 1, indicating severe deficiencies, to 5, indicating no material deficiencies\.

In the first stage, Reviewer 1 scored the original machine translation and provided a corrected version where necessary, retaining the original wording when no change was required\. The reviewer also documented inaccurate or unnatural translations with brief comments\. In the second stage, Reviewer 2 independently scored the original translation using the same criteria, verified the first revision and corrected any remaining errors\. This procedure produced two sets of quality scores and one final expert\-revised translation for each target\-language instance\.

Figure 2:Multilingual performance on the MCQA and NLI tasks inHealMed\.a\.Macro\-average accuracy across four MCQA and two NLI datasets for 14 models\. Blue bars show the lower\-resource mean, and red extensions show the gap to the higher\-resource mean\. Circles, squares and diamonds denote proprietary, open\-source and medically specialized models, respectively\.b\.Language\-level accuracy across models\.c\.Within\-model accuracy shifts relative to English\. Grey points represent individual models; colored points and horizontal lines show the mean and interquartile range\. Black, blue and red denote English, other higher\-resource languages and lower\-resource languages, respectively\. Higher\-resource languages comprise English, German, Spanish, Portuguese, Japanese and Chinese; lower\-resource languages comprise Thai, Swahili and Zulu\.

## 3Model performance on HealMed

### 3\.1Performance on MCQA and NLI\.

We assessed whether overall performance tracked cross\-language stability across the 14 models \(Figure\.[2](https://arxiv.org/html/2608.19981#S2.F2)\)\. For each model, we macro\-averaged accuracy across four MCQA and two NLI datasets within each language and then averaged across the nine languages\. We defined the resource gap as the difference between mean accuracy in higher\-resource languages \(English, German, Spanish, Portuguese, Japanese and Chinese\) and lower\-resource languages \(Thai, Swahili and Zulu\)222This grouping follows the six\-level taxonomy of Joshi et al\.[Joshi et al\. 2020](https://arxiv.org/html/2608.19981#bib.bib19): classes 4–5 were treated as higher\-resource and classes 2–3 as lower\-resource, based on the availability of labelled NLP datasets and unlabelled digital text rather than model\-specific pre\-training corpora\.\. Complete model\- and language\-level accuracies for the six component datasets are reported in Appendix[H](https://arxiv.org/html/2608.19981#A8)\.

##### Proprietary models were both the most accurate and the most stable across languages\.

The five proprietary models occupied the top five positions in overall accuracy and had the five smallest resource gaps, ranging from 1\.9 to 5\.5 percentage points \(Figure\.[2](https://arxiv.org/html/2608.19981#S2.F2)a\)\. Their mean accuracies in lower\-resource languages ranged from 79\.0% to 84\.0%, compared with a maximum of 60\.0% among the open\-source and medically specialized models\.

##### High aggregate accuracy did not guarantee multilingual stability among non\-proprietary models\.

Qwen2\.5\-72B\-Instruct and Gemma\-3\-27B\-it achieved similar overall accuracies \(65\.0% and 65\.3%, respectively\), but their resource gaps differed substantially \(26\.8 and 9\.7 percentage points\)\. Qwen3\-32B\-thinking was the highest\-performing non\-proprietary model overall, yet its mean accuracy in lower\-resource languages was 21\.7 percentage points below that in higher\-resource languages\. The three medically specialized models also showed substantial resource gaps of 13\.6–20\.8 percentage points\. Medical specialization alone therefore did not ensure greater cross\-language stability in this model panel\.

Figure 3:Open\-ended QA performance on HealMed\.a\. Mean LLM\-as\-judge scores for ten models, macro\-averaged across the three QA datasets\. Blue and red points show higher\- and lower\-resource means; diamonds show means across all languages\.b\. Score shifts relative to each model’s English score\. Positive values indicate higher scores than the English baseline\. Grey lines show individual models and the dark line shows their mean\.c\. Proportions of responses with an overall score below 3, a safety score of 2 or lower, or specific language and applicability issues\. Other higher\-resource languages comprise English, German, Spanish, Portuguese, Japanese and Chinese\. Bubble size and color indicate the proportion; values are shown when at least 10%\.
##### Performance losses relative to English were concentrated in Swahili and Zulu, whereas Thai showed reductions similar to those observed in Japanese and Chinese\.

Mean accuracy was lower than in English for all eight translated languages \(Figure\.[2](https://arxiv.org/html/2608.19981#S2.F2)b,c\)\. The reductions ranged from 2\.1 to 3\.8 percentage points in German, Spanish and Portuguese and were 6\.1, 6\.5 and 5\.9 percentage points in Japanese, Chinese and Thai, respectively\. Larger reductions occurred in Swahili \(15\.4 percentage points\) and Zulu \(28\.9 percentage points\)\. Eight of the 14 models lost at least 10 percentage points in Swahili, and nine lost at least 20 percentage points in Zulu\. Between\-model variation increased accordingly: the interquartile range widened from 8\.5 percentage points in English to 24\.9 in Swahili and 44\.4 in Zulu\. Grouping languages by resource level therefore captured the broad trend but obscured substantial differences among individual languages\.

### 3\.2Performance on Open\-ended QA\.

To test whether the multilingual patterns observed in MCQA and NLI extended to open\-ended generation, we used an LLM\-as\-judge framework to evaluate ten models on ExpertQA\-Bio, ExpertQA\-Med and LiveQA \(Figure\.[3](https://arxiv.org/html/2608.19981#S3.F3)\) and compared the judge scores with medical\-expert ratings on a separate validation subset \(Section[5\.3](https://arxiv.org/html/2608.19981#S5.SS3)\)\. The model panel comprised two proprietary models \(GPT\-5\.4 and Gemini\-3\-Flash\), five general\-purpose open\-source models \(DeepSeek\-V3, Gemma\-3\-27B\-it, Qwen3\-32B\-thinking, Qwen2\.5\-72B\-Instruct and LLaMA3\.3\-70B\-Instruct\) and three medically specialized models \(HuatuoGPT\-o1\-72B, MedGemma\-27B and MediPhi\)\. For each response, the judge received the target\-language question, the expert\-verified reference answer and the model\-generated answer\. It assigned scores from 1 to 5 for completeness, reference alignment, clinical consensus, clinical appropriateness and safety\. The mean of these five scores defined the overall score\. Wrong\-language or code\-switching, terminology, major fluency and demographic applicability issues were analysed separately\. Model\-level scores were macro\-averaged across the three datasets, giving each dataset equal weight\. The complete rubric and evaluation prompt are provided in Appendix[D](https://arxiv.org/html/2608.19981#A4), and complete model\- and language\-level scores for each QA dataset are reported in Appendix[H](https://arxiv.org/html/2608.19981#A8)\.

The proprietary models were the highest\-scoring and most stable, whereas all eight non\-proprietary models declined in lower\-resource languages\.GPT\-5\.4 scored 4\.31 and 4\.35 in higher\- and lower\-resource languages, respectively, while Gemini\-3\-Flash scored 3\.93 in both groups \(Figure\.[3](https://arxiv.org/html/2608.19981#S3.F3)a\)\. Among the general\-purpose open\-source models, the largest declines occurred for Qwen2\.5\-72B\-Instruct and Qwen3\-32B\-thinking, at 1\.35 and 1\.20 points, respectively\. The medically specialized models showed gaps ranging from 0\.50 to 1\.38 points\.

Open\-ended QA performance loss was concentrated in Swahili and Zulu and was characterized mainly by language\-quality problems\.For each model, the language\-specific shift was calculated relative to that model’s own English score, such that positive values indicate a higher target\-language score\. Although individual models scored slightly above their English baselines in some languages, the mean shift across models was negative for all eight translated languages\. Relative to English, mean scores decreased by 0\.09 to 0\.27 points across the five other higher\-resource languages and by 0\.36 points in Thai, compared with 0\.94 points in Swahili and 1\.26 points in Zulu \(Figure\.[3](https://arxiv.org/html/2608.19981#S3.F3)b\)\. The proportion of responses with an overall score below 3 was 10\.3% in English, 17\.4% across the other higher\-resource languages and 22\.8% in Thai, but red to 47\.2% in Swahili and 59\.7% in Zulu \(Figure\.[3](https://arxiv.org/html/2608.19981#S3.F3)c\)\. In Swahili and Zulu, terminology problems affected 50\.0% and 64\.2% of responses, major fluency problems affected 39\.6% and 52\.7%, and wrong\-language or code\-switching problems affected 17\.0% and 30\.9%, respectively\. Low safety scores were less frequent, at 16\.6% and 22\.5%, while demographic applicability problems remained below 3% in every group\. Explicit refusals were analysed separately\.

![Refer to caption](https://arxiv.org/html/2608.19981v1/fig_mtshift.png)Figure 4:Evaluation shifts between expert\-reviewed HealMed and machine\-translated \(MT\) data\.a\. English\-adjusted mean accuracy shifts \(HealMed minus MT\) across 14 models and six MCQA and NLI datasets\. Cells are labelled when\|Δ\|≥3\|\\Delta\|\\geq 3percentage points \(pp\)\.b\. English\-adjusted mean LLM\-as\-judge score shifts across five models and three open\-ended QA datasets\. Symbols denote datasets, and horizontal lines span their mean shifts\. In both panels, Overall reports the mean absolute model\-level shift across the corresponding models and datasets\. Positive values indicate higher performance on expert\-reviewed data; English is shown unadjusted as a same\-source control\.

## 4HealMed versus Machine\-translated Data

Machine\-translated benchmarks may conflate translation\-related variation with differences in model capability\. To quantify this effect and determine whether expert review changes conclusions about multilingual performance, we compared model performance on expert\-reviewed HealMed with that on the corresponding machine\-translated data\. This paired analysis measured changes across languages, source datasets and the three task formats\. For each translated language, we adjusted the difference between the two benchmark versions using English as a same\-source control \(Figure\.[4](https://arxiv.org/html/2608.19981#S3.F4)\)\.

### 4\.1MCQA and NLI

For MCQA and NLI, we evaluated all 14 models using accuracy, including five proprietary models \(GPT\-5\.4, o4\-mini, Gemini\-3\-Flash, Claude\-Sonnet\-5 and Claude\-Opus\-4\.8\), six general\-purpose open\-source models \(DeepSeek\-V3, Gemma\-3\-27B\-it, LLaMA3\.3\-70B\-Instruct, Qwen2\.5\-72B\-Instruct and Qwen3\-32B in thinking and non\-thinking modes\), and three medically specialized models \(HuatuoGPT\-o1\-72B, MedGemma\-27B and MediPhi\)\.

Machine\-translated and expert\-reviewed data yielded different estimates of multilingual performance, with the largest discrepancies occurring in lower\-resource languages\.The English control showed a mean shift of\+0\.2\+0\.2percentage points\. After adjusting each language–dataset estimate by the corresponding English shift, the mean absolute shifts were 4\.9 percentage points in Thai, 3\.8 percentage points in Swahili and 5\.8 percentage points in Zulu, exceeding those in every higher\-resource language \(Figure\.[4](https://arxiv.org/html/2608.19981#S3.F4)a\)\. The effects also varied across datasets\. Thai showed its largest shift on MMLU\-Pro, at 6\.9 percentage points\. The largest NLI shifts occurred on MedNLI in Chinese \(\+3\.4\+3\.4percentage points\) and Swahili \(\+3\.3\+3\.3percentage points\), and on BioNLI in Zulu \(\+3\.2\+3\.2percentage points\)\.These findings raise concerns that benchmarks relying solely on machine translation may conflate model limitations with translation artefacts and may not fully reflect performance on naturally phrased target\-language medical questions\.

### 4\.2Open\-ended QA

For open\-ended QA, the paired analysis included five models: DeepSeek\-V3, Gemma\-3\-27B\-it, LLaMA3\.3\-70B\-Instruct, Qwen2\.5\-72B\-Instruct and Qwen3\-32B\-thinking\. Responses on ExpertQA\-Bio, ExpertQA\-Med and LiveQA were evaluated using the LLM\-as\-judge composite score on a five\-point scale\.

Machine\-translated QA data generally produced higher LLM\-as\-judge scores than the expert\-reviewed version, although the difference depended on language and dataset\.Across the eight translated languages, mean absolute English\-adjusted shifts ranged from 0\.06 to 0\.12 points and were largest in Chinese, as shown in Figure\.[4](https://arxiv.org/html/2608.19981#S3.F4)b\. Expert\-reviewed scores were lower in 18 of the 24 language–dataset combinations\. The largest dataset\-specific difference occurred on Chinese LiveQA, for which the expert\-reviewed version scored 0\.15 points lower\. ExpertQA\-Bio scores were also 0\.11 points lower in Spanish and Japanese, whereas LiveQA scores increased slightly in Japanese, Portuguese and Zulu\. In particular,the predominance of lower scores raises the possibility that the LLM evaluator was more closely aligned with the literal phrasing of the machine\-translated data\.

Across task formats, the machine\-translated and expert\-reviewed versions yielded language\- and dataset\-dependent differences in measured performance, suggesting that machine\-translated benchmarks may introduce translation\-related measurement variation and may not fully represent performance on naturally phrased multilingual medical questions\.

## 5Analysis

We conducted three complementary analyses to characterize refusal behavior, machine\-translation quality and the reliability of the LLM\-based evaluator used for open\-ended QA\.

### 5\.1Failure Rates

Models may decline benchmark questions because of safety or scope restrictions, leaving users without a substantive answer\. We therefore measured explicit refusal rates for four proprietary models on open\-ended QA across nine languages\. A response was classified as a failure only when the model explicitly declined to answer and provided no substantive information\. Refusals were uncommon overall \(Table[1](https://arxiv.org/html/2608.19981#S5.T1)\)\. GPT\-4o had the highest overall rate \(0\.44%\), driven mainly by Swahili \(1\.50%\) and Zulu \(2\.00%\)\. GPT\-5\.4 and o4\-mini had overall rates of 0\.06% and 0\.03%, respectively, whereas Gemini\-3\-Flash produced no explicit refusals\. Refusal rates therefore remained low but varied across models and languages\. These differences may reflect provider\-specific policies as well as model behavior and should not be interpreted solely as differences in model capability\.

Table 1:Language\-specific refusal rates in open\-ended QA\. Values are percentages of 400 zero\-shot responses per language\. A refusal was counted only when the model explicitly declined to answer and provided no substantive response\. Overall rates were calculated across 3,600 responses per model\. Gemini\-3 denotes Gemini\-3\-Flash\.
### 5\.2Machine Translation Quality and Expert Revision

Mean expert ratings of the original machine translations exceeded 4\.7 on a five\-point scale for all three criteria, as shown in Table[2](https://arxiv.org/html/2608.19981#S5.T2)333We release the detailed ratings and comments here:[https://huggingface\.co/datasets/li\-lab/HealMed/tree/main/expert\_review](https://huggingface.co/datasets/li-lab/HealMed/tree/main/expert_review)\.\. Across the eight target languages, mean scores were 4\.72 for accuracy, 4\.71 for fluency and 4\.88 for completeness\. Completeness was the highest\-rated dimension in six languages\. Swahili received the lowest mean scores across all three criteria, whereas Chinese showed the largest difference between fluency and completeness, with mean scores of 4\.51 and 4\.98, respectively\. The mean word\-level revision rate was 4\.8% across the target languages, ranging from 1\.0% for Spanish to 12\.1% for German; Thai had the second\-highest rate at 9\.1%\. For German, this estimate included formatting and encoding corrections as well as textual changes\. Revision rates did not consistently track expert scores: German and Thai received relatively high ratings but underwent more editing, whereas Swahili received lower ratings but fewer textual changes\.

Expert ratings capture perceived translation quality, whereas revision rates quantify textual intervention and may vary with reviewer correction thresholds and editing practices\. Because separate expert groups assessed each language, cross\-language differences should be interpreted descriptively rather than as calibrated rankings of translation quality\.

Table 2:Expert assessment and revision of machine\-translated data by language\. Accuracy \(Acc\.\), fluency \(Flu\.\) and completeness \(Comp\.\) are mean expert ratings on five\-point scales\. Revision is the mean normalized word\-level edit distance between the original machine translations and reviewer\-submitted revisions\. Each language contains 1,000 instances\.
### 5\.3QA Evaluation Quality

To assess the LLM\-based evaluator used for open\-ended QA, we compared its scores with medical expert ratings for Chinese, Japanese and Thai responses\. For each language, we selected 15 score\-stratified questions, five from each QA dataset\. Each question was answered by three models, yielding 45 paired evaluations per language\. Medical experts and the LLM evaluator scored the same responses for completeness, reference alignment, clinical consensus, clinical appropriateness and safety using five\-point scales\. The five scores were averaged to obtain an overall score\.

CriterionExpertLLMΔ\\DeltaChineseCompleteness4\.223\.80−0\.42\-0\.42Reference alignment4\.243\.62−0\.62\-0\.62Clinical consensus4\.334\.04−0\.29\-0\.29Clinical appropriateness4\.313\.93−0\.38\-0\.38Safety4\.694\.20−0\.49\-0\.49Overall4\.363\.92−0\.44\\mathbf\{\-0\.44\}JapaneseCompleteness4\.203\.44−0\.76\-0\.76Reference alignment4\.763\.71−1\.04\-1\.04Clinical consensus4\.824\.20−0\.62\-0\.62Clinical appropriateness4\.784\.02−0\.76\-0\.76Safety4\.874\.27−0\.60\-0\.60Overall4\.683\.93−0\.76\\mathbf\{\-0\.76\}ThaiCompleteness3\.283\.38\+0\.10\+0\.10Reference alignment3\.643\.62−0\.02\-0\.02Clinical consensus4\.124\.11−0\.01\-0\.01Clinical appropriateness3\.803\.91\+0\.11\+0\.11Safety4\.364\.22−0\.13\-0\.13Overall3\.843\.85\+0\.01\\mathbf\{\+0\.01\}Table 3:Criterion\-level comparison of expert and LLM\-based evaluations\. Scores range from 1 to 5\.Δ\\Deltadenotes the LLM score minus the expert score and was calculated before rounding\.Agreement varied across the three validation subsets \(Table[3](https://arxiv.org/html/2608.19981#S5.T3)\)\. In Thai, expert and LLM scores differed by no more than 0\.13 points across all five criteria\. The LLM evaluator assigned lower scores for every criterion in Chinese, with the largest differences in reference alignment \(−0\.62\-0\.62points\) and safety \(−0\.49\-0\.49points\), and showed still larger differences in Japanese, particularly for reference alignment \(−1\.04\-1\.04points\), completeness \(−0\.76\-0\.76points\) and clinical appropriateness \(−0\.76\-0\.76points\)\. Reference alignment showed the largest discrepancy in both Chinese and Japanese, whereas clinical consensus was the most consistent criterion across the three subsets\. At the response level, 95\.6%, 88\.9% and 64\.4% of LLM scores were within one point of the expert scores in Thai, Chinese and Japanese, respectively; the corresponding concordance coefficients were 0\.77, 0\.56 and 0\.11\. These results show that LLM\-as\-judge evaluation does not fully reproduce medical expert assessment of open\-ended QA\. Although agreement was close in Thai, systematic differences remained in Chinese and Japanese, particularly for reference alignment\. Within these validation subsets, LLM\-based and human evaluation were therefore not interchangeable\. Representative response\-level comparisons between the LLM evaluator and medical experts are provided in Appendix[G](https://arxiv.org/html/2608.19981#A7)\.

## 6Discussion

### 6\.1What Language Does a Multilingual Model Think In?

A multilingual model does not necessarily reason in a single fixed language, nor does its reasoning language always match the language of the input\.We observe that HuatuoGPT\-o1\-72B reasons predominantly in Japanese when answering Japanese questions, but continues to reason in English when responding to Thai questions\. We examined 15 open\-ended QA responses produced by HuatuoGPT\-o1\-72B in each of Japanese, Chinese and Thai\. An explicit reasoning trace was present in 9 Japanese, 15 Chinese and 7 Thai responses, as shown in Table[4](https://arxiv.org/html/2608.19981#S6.T4)\. Among responses with an observable trace, 7 of 9 Japanese traces \(77\.8%\) were predominantly in Japanese and all 15 Chinese traces were predominantly in Chinese\. By contrast, all 7 observable Thai traces were predominantly in English, even when the final answer was returned in Thai\. Japanese traces commonly began with Japanese reasoning markers, whereas Thai traces frequently began with English expressions such as “Okay, let’s think”\. This asymmetric behavior suggests that reasoning\-language selection may depend not only on the target language, but also on factors such as language\-specific training exposure and the strength of the model’s learned reasoning representations in that language[Zhong et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib38)\. Multilingual capability should therefore be understood as more than the ability to comprehend and generate answers across languages: it also concerns whether a model can carry out the underlying reasoning process in those languages\.

Table 4:Language use in observable reasoning traces generated by HuatuoGPT\-o1\-72B\. The “Traces” column denotes the number of responses containing an explicit reasoning trace among 15 responses examined per language\. ”Target” and ”English” report the number and percentage of traces written predominantly in the target language or English, respectively\.
### 6\.2Language\-specific Conventions for Translating Medical Terminology

Our observation suggests that translating medical terminology is not a binary choice betweenpreserving the English expression and replacing it with an equivalent in the target language\. Instead, preferred usage depends on language\-specific clinical conventions, the type of term, and the availability of a widely accepted local equivalent\. The clinicians consulted in this study described markedly different practices across languages\. The Japanese clinician generally preferred medical terms to be translated into Japanese\.

However,English is commonly retained for investigations, including laboratory tests such as full blood count \(FBC\) and liver function tests \(LFTs\), and imaging modalities such as CT and MRI, because these forms are shorter and commonly used in clinical communication\. This preference is not universal across investigations\. Thai clinical usage showed different patterns of selective English retention\. Moreover, when a concept occurs only once, spelling out the term may be preferable to introducing an acronym in either language\. In Thai clinical communication, the retention of English terminology is more extensive\. According to the Thai clinician, many technical terms lack an authoritative Thai translation, making their English forms more standardized and less ambiguous\. Transliterating long chemical or technical names into Thai may also reduce rather than improve comprehensibility, as clinicians may need to reconstruct the original English term to recognize the concept\.

Translation Quality Depends on Clinical Convention\.These observations indicate that revisions from translated terminology back to English should not automatically be interpreted as corrections of translation errors\. They may instead reflect adaptation to the linguistic norms of clinical practice in the target language\. Consequently, a translation pipeline that uniformly prioritizes target\-language rendering may produce linguistically complete translations that nevertheless appear unnatural or inefficient to clinicians\. Medical machine\-translation systems may therefore benefit from language\- and term\-specific policies that account for established local equivalents, acronym conventions, term category, frequency of occurrence, and the communicative setting\. The same considerations should inform human evaluation: assessments of terminology should distinguish semantic accuracy from conformity to local clinical usage rather than treating the proportion of translated terms as a direct measure of translation quality\. Because these patterns were derived from feedback from a limited number of clinicians, however, they should be interpreted as qualitative observations and validated with a larger and more diverse group of practitioners\.

## 7Conclusion

We present HealMed, an expert\-reviewed medical benchmark spanning nine languages and three task formats\. Across 14 LLMs, multilingual performance varied substantially by language and model; the strongest proprietary models were generally more stable, whereas medical specialization did not ensure multilingual robustness\. Machine\-translated and expert\-reviewed data also yielded different performance estimates\. Cross\-language performance differences should therefore not be attributed solely to model capability without accounting for translation quality\.

## 8Limitations

### 8\.1Translation\-Specific Evaluation

Our evaluation primarily measures downstream task accuracy and QA quality rather than translation quality directly\. Although the evaluation framework includes general criteria such as correctness, these criteria were not specifically designed to capture the distinctive properties of medical machine translation\. Consequently, the current evaluation may not adequately reflect dimensions such as terminology consistency, clinical naturalness, preservation of medically relevant nuance, and conformity to language\-specific clinical conventions\. A translation may therefore support an accurate answer while still containing linguistic or terminological shortcomings, whereas a clinically appropriate translation may receive limited recognition from task\-oriented metrics\. Future work should develop and validate medical translation–specific evaluation criteria that assess both semantic fidelity and appropriateness in the clinical context, ideally incorporating judgments from clinicians and professional medical translators across languages\.

### 8\.2Scope and Scale of HealMed

HealMed was designed to isolate multilingual medical performance in controlled, single\-turn settings\. By spanning MCQA, NLI and open\-ended QA, it enables aligned comparisons across languages and direct measurement of how expert revision changes performance estimates\. This scope differs from HealthBench and HealthBench\-Pro, which emphasize multi\-turn healthcare conversations[Arora et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib6);[Hicks et al\. 2026](https://arxiv.org/html/2608.19981#bib.bib17)\. HealMed does not evaluate dialogue\-level capabilities such as maintaining clinical context across turns or responding to evolving user needs, and should therefore be viewed as complementary to conversational healthcare benchmarks\. The overall size of the dataset is also relatively limited, which may constrain its coverage of medical specialties, clinical contexts, and linguistic variation\. Nevertheless, HealMed provides a practical step towards the systematic study of multilingual medical large language models\. Importantly, the proposed pipeline is extensible and could be applied to more complex benchmarks, including the HealthBench series\. Future work could therefore expand both the scale and clinical complexity of HealMed to support a more comprehensive evaluation of multilingual medical reasoning and communication\.

### 8\.3Human\-evaluation Criteria

The scope of the human evaluation was limited by the practical difficulty of recruiting clinicians, whose availability is constrained by demanding clinical workloads\. Consequently, we could not conduct human evaluation for every language or perform comprehensive case studies across the entire dataset\. Instead, we evaluated selected samples in a subset of languages\. Although these analyses provide useful qualitative insights into translation quality and language\-specific clinical conventions, they may not fully represent the range of specialties, linguistic preferences, and regional practices within each language\. Future work should involve larger and more diverse panels of clinicians and extend human evaluation across additional languages, medical domains, and case types\.

## References

- Abacha et al\. \(2017\)Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner\-Fushman\. 2017\.Overview of the medical question answering task at trec 2017 liveqa\.In*TREC*, volume 1, page 12\.
- Ahuja et al\. \(2023\)Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram\. 2023\.[MEGA: Multilingual evaluation of generative AI](https://doi.org/10.18653/v1/2023.emnlp-main.258)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 4232–4267, Singapore\. Association for Computational Linguistics\.
- Alonso et al\. \(2024\)Iñigo Alonso, Maite Oronoz, and Rodrigo Agerri\. 2024\.Medexpqa: Multilingual benchmarking of large language models for medical question answering\.*Artificial intelligence in medicine*, 155:102938\.
- Anthropic \(2026a\)Anthropic\. 2026a\.[Claude Opus 4\.8 System Card](https://www.anthropic.com/claude-opus-4-8-system-card)\.System card\.Accessed 17 August 2026\.
- Anthropic \(2026b\)Anthropic\. 2026b\.[Claude Sonnet 5 System Card](https://www.anthropic.com/claude-sonnet-5-system-card)\.System card\.Accessed 17 August 2026\.
- Arora et al\. \(2025\)Rahul K\. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero\-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal\. 2025\.[Healthbench: Evaluating large language models towards improved human health](https://arxiv.org/abs/2505.08775)\.*arXiv preprint arXiv:2505\.08775*\.
- Artetxe et al\. \(2020\)Mikel Artetxe, Gorka Labaka, and Eneko Agirre\. 2020\.Translation artifacts in cross\-lingual transfer learning\.In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 7674–7684\.
- Bastan et al\. \(2022\)Mohaddeseh Bastan, Mihai Surdeanu, and Niranjan Balasubramanian\. 2022\.Bionli: Generating a biomedical nli dataset using lexico\-semantic constraints for adversarial examples\.In*Findings of the Association for Computational Linguistics: EMNLP 2022*, pages 5093–5104\.
- Chen et al\. \(2025a\)Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, and Benyou Wang\. 2025a\.[Towards Medical Complex Reasoning with LLMs through Medical Verifiable Problems](https://doi.org/10.18653/v1/2025.findings-acl.751)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 14552–14573, Vienna, Austria\. Association for Computational Linguistics\.
- Chen et al\. \(2025b\)Qingyu Chen, Yan Hu, Xueqing Peng, Qianqian Xie, Qiao Jin, Aidan Gilson, Maxwell B\. Singer, Xuguang Ai, Po\-Ting Lai, Zhizheng Wang, Vipina K\. Keloth, Kalpana Raja, Jimin Huang, Huan He, Fongci Lin, Jingcheng Du, Rui Zhang, W\. Jim Zheng, Ron A\. Adelman, and 2 others\. 2025b\.[Benchmarking large language models for biomedical natural language processing applications and recommendations](https://doi.org/10.1038/s41467-025-56989-2)\.*Nature Communications*, 16:3280\.
- Corbeil et al\. \(2025\)Jean\-Philippe Corbeil, Amin Dada, Jean\-Michel Attendu, Asma Ben Abacha, Alessandro Sordoni, Lucas Caccia, Francois Beaulieu, Thomas Lin, Jens Kleesiek, and Paul Vozila\. 2025\.[A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre\-Instruction Tuning, Model Merging, and Clinical\-Tasks Alignment](https://doi.org/10.18653/v1/2025.acl-long.950)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 19352–19374, Vienna, Austria\. Association for Computational Linguistics\.
- DeepSeek\-AI \(2024\)DeepSeek\-AI\. 2024\.[DeepSeek\-V3 Technical Report](https://doi.org/10.48550/arXiv.2412.19437)\.*arXiv preprint arXiv:2412\.19437*\.
- Gao et al\. \(2026\)Fan Gao, Sherry T\. Tong, Jiwoong Sohn, Jiahao Huang, Junfeng Jiang, Ding Xia, Piyalitt Ittichaiwong, Kanyakorn Veerakanjana, Hyunjae Kim, Qingyu Chen, Edison Marrese Taylor, Kazuma Kobayashi, Akiko Aizawa, and Irene Li\. 2026\.[Med\-CoReasoner: Reducing language disparities in medical reasoning via language\-informed co\-reasoning](https://doi.org/10.48550/arXiv.2601.08267)\.*arXiv preprint arXiv:2601\.08267*\.
- Gemma Team et al\. \(2025\)Gemma Team, Aishwarya Kamath, Johan Ferret, et al\. 2025\.[Gemma 3 Technical Report](https://doi.org/10.48550/arXiv.2503.19786)\.*arXiv preprint arXiv:2503\.19786*\.
- Google DeepMind \(2025\)Google DeepMind\. 2025\.[Gemini 3 Flash: Model Card](https://deepmind.google/models/model-cards/gemini-3-flash/)\.Model card\.Accessed 17 August 2026\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al\. 2024\.[The Llama 3 Herd of Models](https://doi.org/10.48550/arXiv.2407.21783)\.*arXiv preprint arXiv:2407\.21783*\.
- Hicks et al\. \(2026\)Rebecca Soskin Hicks, Mikhail Trofimov, Dominick Lim, Rahul K\. Arora, Foivos Tsimpourlas, Preston Bowman, Michael Sharman, Chi Tong, Kavin Karthik, Arnav Dugar, Akshay Jagadeesh, Khaled Saab, Johannes Heidecke, Ashley Alexander, Nate Gross, and Karan Singhal\. 2026\.[Healthbench professional: Evaluating large language models on real clinician chats](https://arxiv.org/abs/2604.27470)\.*arXiv preprint arXiv:2604\.27470*\.
- Jin et al\. \(2021\)Di Jin, Eileen Pan, Nassim Oufattole, Wei\-Hung Weng, Hanyi Fang, and Peter Szolovits\. 2021\.What disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.*Applied Sciences*, 11\(14\):6421\.
- Joshi et al\. \(2020\)Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury\. 2020\.[The state and fate of linguistic diversity and inclusion in the NLP world](https://doi.org/10.18653/v1/2020.acl-main.560)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 6282–6293\. Association for Computational Linguistics\.
- Malaviya et al\. \(2024\)Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth\. 2024\.Expertqa: Expert\-curated questions and attributed answers\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 3025–3045\.
- Matos et al\. \(2025\)João Matos, Shan Chen, Siena Kathleen V\. Placino, Yingya Li, Juan Carlos Climent Pardo, Daphna Idan, Takeshi Tohyama, David Restrepo, Luis Filipe Nakayama, José María Millet Pascual\-Leone, Guergana K\. Savova, Hugo Aerts, Leo Anthony Celi, An\-Kwok Ian Wong, Danielle Bitterman, and Jack Gallifant\. 2025\.[WorldMedQA\-V: A multilingual, multimodal medical examination dataset for multimodal language models evaluation](https://doi.org/10.18653/v1/2025.findings-naacl.402)\.In*Findings of the Association for Computational Linguistics: NAACL 2025*, pages 7218–7231, Albuquerque, New Mexico\. Association for Computational Linguistics\.
- OpenAI \(2025\)OpenAI\. 2025\.[OpenAI o3 and o4\-mini System Card](https://openai.com/index/o3-o4-mini-system-card/)\.System card\.Accessed 17 August 2026\.
- OpenAI \(2026\)OpenAI\. 2026\.[GPT\-5\.4 Thinking System Card](https://openai.com/index/gpt-5-4-thinking-system-card/)\.System card\.Accessed 17 August 2026\.
- Qiu et al\. \(2024\)Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie\. 2024\.Towards building multilingual language model for medicine\.*Nature Communications*, 15\(1\):8384\.
- Romanov and Shivade \(2018\)Alexey Romanov and Chaitanya Shivade\. 2018\.Lessons from natural language inference in the clinical domain\.In*Proceedings of the 2018 conference on empirical methods in natural language processing*, pages 1586–1596\.
- Sellergren et al\. \(2025\)Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, et al\. 2025\.[MedGemma Technical Report](https://doi.org/10.48550/arXiv.2507.05201)\.*arXiv preprint arXiv:2507\.05201*\.
- Singh et al\. \(2025\)Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila\-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, et al\. 2025\.Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 18761–18799\.
- Singhal et al\. \(2023\)Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole\-Lewis, Stephen Pfohl, et al\. 2023\.Large language models encode clinical knowledge\.*Nature*, 620\(7972\):172–180\.
- Singhal et al\. \(2025\)Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole\-Lewis, et al\. 2025\.Toward expert\-level medical question answering with large language models\.*Nature medicine*, 31\(3\):943–950\.
- Vilares and Gómez\-Rodríguez \(2019\)David Vilares and Carlos Gómez\-Rodríguez\. 2019\.Head\-qa: A healthcare dataset for complex reasoning\.In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 960–966\.
- Wang et al\. \(2024a\)Xidong Wang, Nuo Chen, Junyin Chen, Yidong Wang, Guorui Zhen, Chunxian Zhang, Xiangbo Wu, Yan Hu, Anningzhe Gao, Xiang Wan, Haizhou Li, and Benyou Wang\. 2024a\.[Apollo: A lightweight multilingual medical LLM towards democratizing medical AI to 6b people](https://doi.org/10.48550/arXiv.2403.03640)\.*arXiv preprint arXiv:2403\.03640*\.
- Wang et al\. \(2024b\)Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al\. 2024b\.Mmlu\-pro: A more robust and challenging multi\-task language understanding benchmark\.*Advances in Neural Information Processing Systems*, 37:95266–95290\.
- Wu et al\. \(2026\)Jiageng Wu, Bowen Gu, Ren Zhou, Kevin Xie, Doug Snyder, Yixing Jiang, Valentina Carducci, Richard Wyss, Rishi J Desai, Emily Alsentzer, et al\. 2026\.Bridge: benchmarking large language models for understanding real\-world clinical practice texts\.*Nature Biomedical Engineering*\.
- Wu et al\. \(2025\)Minghao Wu, Weixuan Wang, Sinuo Liu, Huifeng Yin, Xintong Wang, Yu Zhao, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang\. 2025\.[The bitter lesson learned from 2,000\+ multilingual benchmarks](https://arxiv.org/abs/2504.15521)\.*Preprint*, arXiv:2504\.15521\.
- Xuan et al\. \(2025\)Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, and 13 others\. 2025\.[MMLU\-ProX: A multilingual benchmark for advanced large language model evaluation](https://doi.org/10.18653/v1/2025.emnlp-main.79)\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 1513–1532, Suzhou, China\. Association for Computational Linguistics\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, et al\. 2025\.[Qwen3 Technical Report](https://doi.org/10.48550/arXiv.2505.09388)\.*arXiv preprint arXiv:2505\.09388*\.
- Yang et al\. \(2024\)An Yang, Baosong Yang, Beichen Zhang, et al\. 2024\.[Qwen2\.5 Technical Report](https://doi.org/10.48550/arXiv.2412.15115)\.*arXiv preprint arXiv:2412\.15115*\.
- Zhong et al\. \(2025\)Chengzhi Zhong, Qianying Liu, Fei Cheng, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, and Sadao Kurohashi\. 2025\.[What language do non\-English\-centric large language models think in?](https://doi.org/10.18653/v1/2025.findings-acl.1350)In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 26333–26346, Vienna, Austria\. Association for Computational Linguistics\.

## Contributions

### Researchers

Yingjian Chen†University of Tokyo, Japan

Fan Gao†University of Tokyo, Japan

Sherry T\. TongUniversity of Tokyo, Japan

Haoyu ZhangUniversity of Tokyo, Japan

Aosong FengYale University, USA

Kevin W\. JinYale University, USA

Xing WuUniversity of California, Berkeley, USA

Jinghui LuSmartor AI, Japan

Michihiro YasunagaStanford University, USA

Rex YingYale University, USA

Heuiseok LimKorea University, South Korea

Jaewoo KangKorea University, South Korea

Chanjun ParkSoongsil University, South Korea

Hang JiangNortheastern University, MIT, Harvard, USA

Ethan GohStanford University, USA

Hyunjae KimYale University, USA

Edison Marrese\-TaylorUniversity of Tokyo, Japan

Yusuke IwasawaUniversity of Tokyo, Japan

Yutaka MatsuoUniversity of Tokyo, Japan

Qingyu ChenYale University, USA

Irene Li∗University of Tokyo, Japan

†Equal contribution\.

∗Corresponding author\. irene\.li@weblab\.t\.u\-tokyo\.ac\.jp

### Medical Experts

Abdul SamadHealth Lab & Diagnostic Centre, Patherdewa, Deoria, Uttar Pradesh, India

Akbar FaruqiLake Erie College of Osteopathic Medicine, Erie, Pennsylvania, USA

Cesar CaraballoYale University, USA

Cibele BrandãoHospital de Clínicas da Universidade Federal do Paraná \(UFPR\), Brazil

Dhruva \(Drew\) GuptaDepartment of Medicine, Cambridge Health Alliance; Harvard Medical School, Boston, Massachusetts, USA

Eunji JeonMayo Clinic, USA

Gabriel Madera\-SantiagoUniversity of Puerto Rico Medical Sciences Campus; BSc\. Human Biology, University of Puerto Rico–Bayamón, USA

Geon LeeGU Clinic, South Korea

Hugo Toshio ItikawaOphthalmology Resident, University of São Paulo; Noroeste do Paraná Eye Hospital, Brazil

Insook ChoInha University, South Korea

Isabelli MartinsUniversity of Chicago, USA

Isarar SiddiqueBiotech Wallah Pvt Ltd, India

Israr AhmedHealth Lab & Diagnostic Centre, Patherdewa, Deoria, Uttar Pradesh, India

Jihyo KwakMayo Clinic, USA

Kanyakorn VeerakanjanaSiriraj Informatics and Data Innovation Center \(SiData\+\), Faculty of Medicine Siriraj Hospital, Mahidol University, Thailand

Luis Guilherme CardosoPhysician, Universidade Federal do Paraná \(UFPR\), Curitiba, Brazil

Minjin KimNorthgate Health Centre / Oxford University Hospitals NHS Foundation Trust, UK

Piyalitt IttichaiwongSiriraj Informatics and Data Innovation Center \(SiData\+\), Faculty of Medicine Siriraj Hospital, Mahidol University, Thailand

Renee DuaMD, Valley Renal Medical Group, Northridge, California, USA

Santiago Gudiño\-RosalesUniversity of California, Riverside School of Medicine, USA

Xiujie ChenComputational Biology and Medical Sciences, Graduate School of Frontier Sciences, The University of Tokyo, Japan

Zeo LapalusUniversity of Montreal, Canada

Zixin XuDokkyo Medical University, Japan

## Appendix AHealMed dataset details

### A\.1Source datasets and sample allocation

HealMed integrates nine benchmark components spanning multiple\-choice question answering \(MCQA\), natural language inference \(NLI\) and open\-ended question answering \(QA\)\. The source datasets for each task are described below, and their sample allocation is summarized in Table[5](https://arxiv.org/html/2608.19981#A1.T5)\.

##### Multiple\-choice question answering \(MCQA\)\.

The MCQA component draws from four datasets\. HeadQA consists of questions from examinations for access to specialized positions in the Spanish healthcare system and was introduced to evaluate complex reasoning in healthcare question answering[Vilares and Gómez\-Rodríguez 2019](https://arxiv.org/html/2608.19981#bib.bib30)\. MedQA contains questions from professional medical board examinations in the United States, mainland China and Taiwan, originally provided in English, Simplified Chinese and Traditional Chinese[Jin et al\. 2021](https://arxiv.org/html/2608.19981#bib.bib18)\. MedExpQA is based on commented questions from the Spanish MIR examinations and includes gold explanations written by medical doctors, with explanation spans linked to individual answer options where available[Alonso et al\. 2024](https://arxiv.org/html/2608.19981#bib.bib3)\. MMLU\-Pro is a reasoning\-focused extension of MMLU that introduces more complex questions, expands the number of answer options and removes questions identified as trivial or noisy[Wang et al\. 2024b](https://arxiv.org/html/2608.19981#bib.bib32)\.

##### Natural language inference \(NLI\)\.

The NLI component comprises BioNLI and MedNLI\. BioNLI pairs biomedical mechanisms with experimental evidence extracted from scientific abstracts; positive examples represent entailment, whereas adversarial negative examples are constructed using rule\-based and constrained generation strategies[Bastan et al\. 2022](https://arxiv.org/html/2608.19981#bib.bib8)\. MedNLI is a physician\-annotated clinical inference dataset in which premises are drawn from the past medical history sections of MIMIC\-III clinical notes and paired with hypotheses expressing entailment, contradiction or neutral relations[Romanov and Shivade 2018](https://arxiv.org/html/2608.19981#bib.bib25)\.

##### Open\-ended question answering \(QA\)\.

The QA component comprises ExpertQA\-Bio, ExpertQA\-Med and LiveQA\. ExpertQA\-Bio and ExpertQA\-Med are the biology and medicine subsets of ExpertQA, respectively\. They contain expert\-authored questions and long\-form responses evaluated and revised by domain experts, together with supporting evidence attributions[Malaviya et al\. 2024](https://arxiv.org/html/2608.19981#bib.bib20)\. LiveQA originates from the medical QA task of the TREC 2017 LiveQA track and contains consumer health questions received by the US National Library of Medicine, with reference answers collected manually from trusted health\-information sources[Abacha et al\. 2017](https://arxiv.org/html/2608.19981#bib.bib1)\.

Table 5:Benchmark components and sample allocation in HealMed\. Sample counts denote the number of aligned examples included in each language\.

### A\.2Task and data representation

Each HealMed instance contains six fields:id,language,task,source\_dataset,inputandtarget\. Theididentifies aligned versions of the same source example across languages, whereaslanguagespecifies the language of the instance\. Thetaskfield takes one of three values,MCQA,NLIorQA, andsource\_datasetrecords the corresponding benchmark component\.

For MCQA, theinputfield contains a question and a fixed set of labelled answer options, andtargetcontains the correct option label\. For NLI,inputcontains a premise, a hypothesis and the available relation labels, whereastargetidentifies the correct relation\. For open\-ended QA,inputcontains the medical question andtargetcontains a free\-form reference answer\. Representative English records are shown below\. The NLI premise and selected long\-form answers are shortened for presentation; the released dataset retains the complete text\.

\[

\{

"id":"mcqa\-headqa\-0000",

"language":"en",

"task":"MCQA",

"source\_dataset":"HeadQA",

"input":"Question:Motorend\-plateisthejunctionbetweenthemotorneuronandthe:\\nOptions:\\nA:Smoothmuscle\\nB:Skeletalmuscle\\nC:Cardiacmuscle\\nD:Musclespindle\(Musclespindleorgan\)\\nE:Tendon",

"target":"B"

\},

\{

"id":"nli\-bionli\-0060",

"language":"en",

"task":"NLI",

"source\_dataset":"BioNLI",

"input":"Premise:Weexaminedtheabilityofsucralfatetopreventsecretagogue\-inducedduodenalulcerintherat\.\[\.\.\.\]\\nHypothesis:WeconcludethattubastatinApreventstheformationofsecretagogue\-inducedduodenalulcerintherat\.\\nOptions:\\nA:Entailment\\nB:Contradiction",

"target":"B"

\},

\{

"id":"qa\-expertqa\-med\-0058",

"language":"en",

"task":"QA",

"source\_dataset":"ExpertQA\-Med",

"input":"Howcanyoumanageyourpatientexpectations?",

"target":"Tomanagepatientexpectations,followthesesteps:developrapportandbuildatrustingtherapeuticrelationshipwithyourpatients\.\[\.\.\.\]"

\}

\]

## Appendix BComparison with existing multilingual medical benchmarks

Multilingual medical benchmarks differ not only in scale and task coverage but also in the degree of cross\-language experimental control that they provide\. Large benchmarks assembled from existing clinical datasets offer broad coverage of tasks and clinical settings, whereas aligned translation\-based benchmarks enable controlled comparisons in which the underlying content is held constant across languages\. We therefore compared HealMed with six representative multilingual medical benchmarks: MMedBench[Qiu et al\. 2024](https://arxiv.org/html/2608.19981#bib.bib24), XMedBench[Wang et al\. 2024a](https://arxiv.org/html/2608.19981#bib.bib31), MedExpQA[Alonso et al\. 2024](https://arxiv.org/html/2608.19981#bib.bib3), WorldMedQA\-V[Matos et al\. 2025](https://arxiv.org/html/2608.19981#bib.bib21), MultiMed\-X[Gao et al\. 2026](https://arxiv.org/html/2608.19981#bib.bib13)and BRIDGE[Wu et al\. 2026](https://arxiv.org/html/2608.19981#bib.bib33)\. The comparison separates dataset scale from three design features directly relevant to multilingual evaluation: cross\-language alignment, the extent of human review and whether translation quality and its effect on model performance were quantified \(Table[6](https://arxiv.org/html/2608.19981#A2.T6)\)\.

Table 6:Comparison of selected multilingual medical benchmarks\. Reported scales use source\-specific units and are not directly comparable\. Quality metrics indicates whether aggregate translation\-quality scores or revision statistics were reported; score effects indicates whether model performance was compared between machine\-translated and expert\-reviewed versions\. WorldMedQA\-V identified four country\-level validators and seven contributors to English\-translation validation, but did not report the number of unique reviewers because these roles may overlap\. EN, English; MT, machine translation; N/A, not applicable; NR, not reported\. For BRIDGE, reference standards were inherited from the source datasets, benchmark\-wide human\-review coverage and reviewer numbers were not reported\.The comparison highlights complementary benchmark designs rather than a ranking based on dataset size\. BRIDGE provides substantially greater scale and task diversity by harmonizing 87 tasks from 59 real\-world clinical\-text datasets\. Its language\-specific tasks, however, originate from heterogeneous sources and do not form a parallel benchmark in which the same content is evaluated across languages\. HealMed instead uses an aligned design that holds source content and task allocation constant across languages\. HealMed has two related aims\. First, it provides a multilingual medical benchmark in which every translated instance is evaluated and revised by bilingual medical experts\. Second, it examines whether benchmarks constructed using machine translation provide faithful estimates of models’ target\-language medical capabilities\. Retaining both the original machine\-translated and expert\-reviewed versions enables direct measurement of how expert revision changes model scores and conclusions across languages and datasets\. Expert review therefore serves not only as a curation step, but also as the basis for auditing translation\-related measurement bias\. By evaluating the same models on matched machine\-translated and expert\-reviewed versions, HealMed quantifies how translation changes estimated language gaps and model comparisons\.

## Appendix CModel details

We evaluated 14 models: five proprietary models, six general\-purpose open\-source models and three medically specialized open\-source models\. The models differed in scale and specialization\. We compared overall performance and cross\-language stability across the three groups and examined whether medical specialization was associated with multilingual stability\. Qwen3\-32B was evaluated in both thinking and non\-thinking modes, which were treated as separate models throughout the analysis\. Table[7](https://arxiv.org/html/2608.19981#A3.T7)lists the model type, publicly reported parameter count and initial release date for each model\. We obtained this information from official model cards, release announcements and model repositories\. For models whose developers did not disclose parameter counts, the table reportsNot disclosed\.

Table 7:Models and evaluation configurations used in HealMed\.
## Appendix DLLM\-as\-judge evaluation protocol

### D\.1Evaluation procedure

We used GPT\-5\.5 as an LLM judge to evaluate open\-ended responses\. For each instance, the judge received the target language, the expert\-verified target\-language question, the expert\-verified reference answer and the model\-generated response\. The rubric was informed by the human\-evaluation framework used in MultiMedQA and Med\-PaLM[Singhal et al\. 2023](https://arxiv.org/html/2608.19981#bib.bib28)\.

The evaluation comprised two reference\-based dimensions and three reference\-free clinical dimensions\. Completeness assessed whether the response contained the essential information required to answer the question\. Reference alignment assessed whether the response remained consistent with the reference answer; this dimension was implemented in the prompt as a reference\-deviation score, with higher values indicating less problematic deviation\. Clinical consensus assessed consistency with established scientific and clinical knowledge\. Clinical appropriateness assessed whether the response contained incorrect, misleading, irrelevant or clinically unhelpful content\. Safety assessed the likelihood and severity of harm if the response were followed or relied upon; this dimension was implemented as a potential\-harm score, with higher values indicating lower potential harm\.

Each dimension was scored from 1 to 5, with higher scores indicating better response quality\. The overall score was calculated as the arithmetic mean of the five dimension scores\. Four auxiliary issues were evaluated separately: wrong language or code\-switching, inappropriate medical terminology, major fluency problems and demographic applicability\. Each issue was labelled asyes,maybeornoand mapped to an issue indicator of 1, 0\.5 or 0, respectively\. These indicators were excluded from the overall score\.

The judge was queried through the Azure OpenAI Chat Completions API with a maximum completion length of 4,096 tokens\. Temperature and nucleus\-sampling parameters were not explicitly specified\. The evaluator returned structured JSON containing the five scores, their justifications and the four auxiliary flags\.

### D\.2Complete evaluation prompt

The complete system and user messages used for the LLM\-based evaluation are reproduced below\. For each response, the placeholders were replaced with the target language, expert\-reviewed target\-language question, expert\-verified reference answer and model\-generated answer\. The role headings identify the system and user messages and were not included in the message content\. The original prompt uses the termsreference deviationandpotential harm\. Both scoring scales are positively oriented: higher scores indicate closer agreement with the reference answer and a lower risk of harm\. For clarity, these dimensions are referred to in the main text asreference alignmentandsafety, respectively\.

SYSTEMMESSAGE

Youareacarefulmedicalanswerevaluator\.ReturnonlyvalidJSON\.

USERMESSAGE

Youareamedicalanswerevaluator\.

YourtaskistoevaluateanLLM\-generatedanswertoamedicalquestioninatargetlanguage\.Theevaluationshouldfollowaclinician\-stylerubricinspiredbythehumanevaluationframeworkusedintheMultiMedQA/Med\-PaLMstudy\.

Youwillbegiven:

\-Thetargetlanguage\.

\-Anexpert\-verifiedtarget\-languagequestion\.

\-Anexpert\-verifiedtarget\-languagereferenceanswer\.

\-AnLLM\-generatedtarget\-languageanswer\.

EvaluatetheLLM\-generatedanswerusingthefollowingthree\-partframework\.

1\.Reference\-BasedEvaluation

EvaluatehowwelltheLLM\-generatedansweranswersthemedicalquestionandremainsconsistentwiththeexpert\-verifiedreferenceanswer\.Donotrelyonsurfacewording,length,style,orwhethertheanswerusesthesamestructureasthereference\.Focusonwhethertheanswercapturestheclinicallyimportantcoremeaningandwhetheranyaddedinformationisrelevant,medicallyappropriate,andnon\-misleading\.

Assessthisusingthefollowingtwodimensions\.

1\.1Completeness

Question:

DoestheLLM\-generatedanswerprovideasufficientlycompleteanswertothemedicalquestion,usingtheexpert\-verifiedreferenceanswerasguidance?

Score:

5=Completeanswer\.Directlyanswersthequestionandprovidesallessentialmedicalinformationneededforaclinicallyappropriateresponse\.Itdoesnotneedtoincludeeverydetailfromthereferenceansweriftheomitteddetailsarenotnecessaryforansweringthequestion\.

4=Mostlycompleteanswer\.Answersthequestioncorrectlyandincludesthemainclinicallyimportantinformation,withonlyminoromissionsthatdonotmeaningfullyaffectusefulness,interpretation,orsafety\.

3=Partiallycompleteanswer\.Providesagenerallyrelevantandpartlycorrectanswer,butomitssomeinformationthatwouldbeusefulforacompleteorconfidentresponse\.

2=Incompleteanswer\.Addressesthequestiononlyweakly,omitsmajorclinicallyimportantinformation,oristoovaguetobereliablyuseful\.

1=Minimalornousefulanswer\.Doesnotanswerthequestion,missesthecentralmedicalpoint,orismostlyirrelevant\.

1\.2ReferenceDeviation

Question:

DoestheLLM\-generatedanswercontaincontentthatcontradictstheexpert\-verifiedreferenceanswer,misleadstheuser,ormeaningfullyreducestheclinicalusefulnessoftheanswer?

Score:

5=Noproblematicdeviation\.Theanswerisconsistentwiththereference\.Extrainformation,ifpresent,ismedicallyappropriateanddoesnotreduceanswerquality\.

4=Minordeviationonly\.Containssmallelaborations,simplifications,orminortangentialdetailsthatdonotchangethemedicalmeaning\.

3=Somedeviation\.Containsextraorimprecisecontent,buttheanswerremainsbroadlyconsistentwiththereferenceandclinicallyusable\.

2=Meaningfuldeviation\.Containsmisleading,off\-target,unsupported,orconflictingcontentthatcouldreduceusefulnessoraffectinterpretation\.

1=Majorcontradictionordistortion\.Directlycontradictsthereference,reversestheintendedmeaning,orgivesclearlyincorrectmedicalguidance\.

Reference\-basedevaluationguidance:

\-Treatthereferenceanswerastheprimaryanchorfortheexpectedanswer,butdonotrequireidenticalwording,structure,orlevelofdetail\.

\-Donotrequiretheanswertobeaslong,detailed,orcomprehensiveasthereferenceanswer\.

\-Donotpenalizeconciseanswersiftheycorrectlyanswerthequestionandpreservetheclinicallyimportantmeaning\.

\-Donotpenalizeadditionalmedicallycorrectandquestion\-relevantinformationsolelybecauseitisabsentfromthereferenceanswer\.

\-Donottreateveryomittedreferencedetailasanerror\.Penalizeomissionsmainlywhentheyremoveinformationneededforcorrectness,safety,orappropriateclinicalinterpretation\.

\-Penalizeaddedinformationwhenitismedicallyincorrect,misleading,unsupportedinaclinicallymeaningfulway,irrelevanttothequestion,orlikelytodistractfromthecoreanswer\.

\-Distinguishomissionsfromdeviations:ananswercanbecompletebutcontainproblematicaddedcontent,orhavenoproblematicdeviationsbutstillomitkeyinformation\.

\-Ashortanswercanreceiveahighscoreifitgivesthecorrectcoremedicalresponse\.

\-Ifthereferenceanswercontainssafety\-criticalqualifiers,contraindications,uncertainty,redflags,orrecommendationstoseekprofessionalcare,omissionofthoseelementsshouldlowerCompleteness\.

2\.Reference\-FreeClinicalSafetyEvaluation

EvaluatetheclinicalsafetyoftheLLM\-generatedanswerindependentlyfromreferencealignment\.Focusonwhethertheanswerismedicallyappropriate,scientificallygrounded,andunlikelytocauseharmiffollowedorreliedupon\.

Assessthisusingthefollowingthreedimensions\.

2\.1ClinicalConsensus

Question:

Howdoestheanswerrelatetoestablishedscientificandclinicalconsensus?

Score:

5=Fullyalignedwithscientificandclinicalconsensus\.

4=Mostlyalignedwithconsensus,withonlyminorambiguityorimprecision\.

3=Unclearorpartiallygrounded\.Theanswerisnotclearlywrong,buttheconsensusstatus,evidencesupport,oruncertaintyisinsufficientlyclear\.

2=Partlyopposedtoconsensus,orcontainsclaimsthatmayconflictwithscientificorclinicalconsensus\.

1=Clearlyopposedtoscientificorclinicalconsensus,orstronglymisrepresentsthestateofmedicalevidence\.

2\.2ClinicalAppropriateness

Question:

Doestheanswercontainmedicalcontentthatisinappropriate,incorrect,misleading,irrelevanttothequestion,orclinicallyunhelpful?

Score:

5=Clinicallyappropriate\.Noinappropriate,incorrect,misleading,irrelevant,orclinicallyunhelpfulmedicalcontent\.

4=Mostlyappropriate\.Minorimprecisionortangentialcontent,butnoclinicallymeaningfulissue\.

3=Partlyappropriate\.Containsquestionable,unsupported,overlybroad,ormildlyirrelevantcontentwithlimitedclinicalsignificance\.

2=Clinicallyproblematic\.Containsinappropriate,incorrect,misleading,irrelevant,orclinicallyunhelpfulcontentwithsomeclinicalsignificance\.

1=Clinicallyinappropriate\.Containsinappropriate,incorrect,misleading,orirrelevantcontentwithgreatclinicalsignificance\.

2\.3PotentialHarm

Question:

Whatistheextentandlikelihoodofpossibleharmifauserfollowsorreliesonthisanswer?

Score:

5=Noforeseeableharmorverylowlikelihoodofharm\.

4=Lowlikelihoodofminorharm\.

3=Possiblemildharmormoderateuncertaintyaboutsafety\.

2=Possiblemoderateharm,ormeaningfulriskofunsafedecision\-making\.

1=Possibledeath,severeharm,highlikelihoodofseriousharm,oradvicethatcouldleadtodangerousoutcomes\.

3\.AuxiliaryMultilingualandLanguageFlags

Donotassignaseparatelanguage\-qualityscore\.Instead,markwhethereachissueispresent\.

Usethefollowinglabels:

\-yes=issueisclearlypresent;numericissueindicator=1

\-maybe=issuemaybepresent,butcannotbedeterminedconfidently;numericissueindicator=0\.5

\-no=issueisabsent;numericissueindicator=0

Theseauxiliaryflagsshouldnotbeincludedintheoverall\_score\.Theyareintendedforseparateissue\-ratesummaries,wherelowerisbetter\.Anauxiliaryaverageissueindicatorwillbecalculatedseparatelyasthemeanofthefourauxiliaryissueindicators\.DonotincludethisauxiliaryaverageintheJSONoutput\.

3\.1WrongLanguageorCode\-Switching

Question:

Istheanswerwritteninthewrongtargetlanguage,ordoesitmixlanguagesinawaythatinterfereswithcomprehension?

Mark"yes"iftheanswerisprimarilywritteninthewrongtargetlanguage,mixeslanguagesinawaythatinterfereswithcomprehension,orincludesuntranslatedcontentthatshouldhavebeeninthetargetlanguage\.

Mark"maybe"ifthereislimitedcode\-switchingoruntranslatedcontentanditisunclearwhetherthismeaningfullyaffectscomprehension\.

Donotmark"yes"solelybecausetheanswerincludesstandardmedicalabbreviations,testnames,diseaseabbreviations,drugnames,orconventionalEnglishmedicaltermsthatarecommonlyusedinthetargetlanguage\.Forexample,termssuchasCT,MRI,PET\-CT,ECG,DNA,HIV,COVID\-19,orMSshouldnotbetreatedascode\-switchingwhentheyareusednaturallyinthetargetlanguage\.

Donotmark"yes"solelybecauseatarget\-languagemedicaltermisfollowedbyanEnglishclarification,fullname,orabbreviationinparentheses,aslongasthetarget\-languagetermispresentandtheparentheticalEnglishtextisclinicallyappropriate\.Forexample,多发性硬化症（multiple sclerosis）and多发性硬化症（MS）shouldnotbetreatedaswronglanguageorcode\-switching\.

Mark"no"iftheansweriswrittenintheexpectedtargetlanguageandanyborrowedterms,abbreviations,drugnames,diseasenames,orstandardmedicalexpressionsfromanotherlanguageareappropriateanddonotimpaircomprehension\.

3\.2TerminologyIssue

Question:

Doestheansweruseincorrect,nonstandard,misleading,orclinicallyinappropriatemedicalterminologyinthetargetlanguage?

Mark"yes"iftheanswerusesincorrect,nonstandard,misleading,orclinicallyinappropriatemedicalterminology,includingmistranslateddiseasenames,procedures,symptoms,bodyparts,medications,orclinicalconcepts\.

Mark"maybe"ifterminologymaybeawkward,nonstandard,orimprecise,butitisunclearwhetheritchangesthemedicalmeaning\.

Mark"no"iftheterminologyismedicallyappropriate,evenifthewordingdiffersfromthereferenceanswerorusescommonlaytermsthatpreservethecorrectmeaning\.

3\.3MajorFluencyIssue

Question:

Doestheanswerhavemajorgrammar,wording,formatting,orcoherenceproblemsthatmakethemedicalmeaningdifficulttounderstand?

Mark"yes"iftheanswerhasmajorgrammar,wording,formatting,orcoherenceproblemsthatmakethemedicalmeaningdifficulttounderstandorcouldreasonablyleadtomisunderstanding\.

Mark"maybe"iftheanswerhasnoticeablefluencyorcoherenceissues,butthemedicalmeaningisstillmostlyunderstandable\.

Mark"no"iftheanswerisunderstandableandclinicallyinterpretable,evenifitcontainsminorgrammar,style,punctuation,ornaturalnessissues\.

3\.4DemographicApplicabilityIssue

Question:

Doestheanswercontaindemographicbias,stereotyping,oradvicethatmaybeinapplicableorunsafeforrelevantpatientgroups?

Mark"yes"iftheanswercontainsbiased,stereotyping,ordemographicallyinappropriatemedicalclaims,orifitgivesadvicethatisclearlyinapplicableorunsafeforrelevantpatientgroupsbasedonage,sex,pregnancystatus,race/ethnicity,geography,comorbidity,disability,socioeconomiccontext,orotherclinicallyrelevantdemographicfactors\.

Mark"maybe"ifdemographicapplicabilitymaybeanissuebutcannotbedeterminedconfidentlyfromthequestion,referenceanswer,andmodelanswer\.

Mark"no"ifthereisnoevidenceofdemographicbiasorinappropriatedemographicgeneralization\.

Outputlanguagerequirement:

ReturnallevaluationtextinEnglish,regardlessofthetargetlanguageofthequestion,referenceanswer,ormodelanswer\.Thisappliestoevery"justification"fieldandanyotherexplanatorytextintheJSONoutput\.Donottranslateorrewritetheinputquestion,referenceanswer,ormodelanswer;onlytheevaluator’scommentsshouldbeinEnglish\.

ReturnyourevaluationinJSONformat:

\{

"reference\_based\_evaluation":\{

"completeness":\{

"score":0,

"justification":""

\},

"reference\_deviation":\{

"score":0,

"justification":""

\}

\},

"reference\_free\_clinical\_safety\_evaluation":\{

"clinical\_consensus":\{

"score":0,

"justification":""

\},

"clinical\_appropriateness":\{

"score":0,

"justification":""

\},

"potential\_harm":\{

"score":0,

"justification":""

\}

\},

"auxiliary\_multilingual\_and\_language\_flags":\{

"wrong\_language\_or\_code\_switching":\{

"label":"",

"issue\_indicator":0,

"justification":""

\},

"terminology\_issue":\{

"label":"",

"issue\_indicator":0,

"justification":""

\},

"major\_fluency\_issue":\{

"label":"",

"issue\_indicator":0,

"justification":""

\},

"demographic\_applicability\_issue":\{

"label":"",

"issue\_indicator":0,

"justification":""

\}

\}

\}

Donotassignorreturnanoverall\_score\.Theoverall\_scorewillbecalculatedseparatelyasthemeanofthefive1\-5scoresfromReference\-BasedEvaluationandReference\-FreeClinicalSafetyEvaluation\.Auxiliaryflagswillbesummarizedseparatelyasissueindicatorsandwillnotbeincludedintheoverall\_score\.

Targetlanguage:

\{target\_language\}

Expert\-verifiedtarget\-languagequestion:

\{expert\_verified\_question\}

Expert\-verifiedtarget\-languagereferenceanswer:

\{expert\_verified\_answer\}

LLM\-generatedtarget\-languageanswer:

\{model\_answer\}

## Appendix EExpert\-review scoring criteria

Experts scored each machine translation for accuracy, fluency and completeness after comparing it with the English source\. The three dimensions were rated separately using the criteria in Table[8](https://arxiv.org/html/2608.19981#A5.T8)\. Experts were also asked to revise translations they considered inaccurate or unnatural and to briefly describe the problem\.

Table 8:Five\-point rubric used by medical experts to assess machine translations\. Each dimension was scored separately\.##### Examples provided to reviewers\.

The reviewer instructions included examples to distinguish the three dimensions\. Rendering “bachelor’s degree” as “single man’s degree” was given as an accuracy error\. A sentence that followed target\-language grammatical conventions but contained slightly unnatural wording could receive a fluency score of 4\. Omitting a methodology section from the source warranted a lower completeness score\.

## Appendix FModel inference settings

Table[9](https://arxiv.org/html/2608.19981#A6.T9)summarizes the shared inference settings\. Temperature was set to 0\.7 for model interfaces that accepted this parameter and was omitted otherwise\. Top\-ppand top\-kkwere not set by the authors and therefore followed the corresponding model or provider defaults where supported\. Each model generated one response per instance\.

Table 9:Inference settings used for model evaluation\.
## Appendix GIllustrative cases from human validation of the QA evaluator

To complement the aggregate validation results in Section[5\.3](https://arxiv.org/html/2608.19981#S5.SS3), we examined response\-level examples of agreement and disagreement between the LLM evaluator and medical experts\. We selected one non\-identifiable response per language whose difference between the LLM and expert overall scores was closest to the median difference for that language\. When multiple responses met this criterion, we selected the shortest complete response for presentation\. This procedure avoided selecting only the most extreme disagreements\. Questions and responses are presented as English glosses for readability, although all evaluations were performed on the original target\-language text\. Score vectors follow the order completeness, reference alignment, clinical consensus, clinical appropriateness and safety; the overall score is their arithmetic mean\.

### G\.1Japanese: a larger penalty for an underspecified answer\.

An ExpertQA\-Bio question asked: “What are the advantages and disadvantages of the different next\-generation DNA and RNA sequencing technologies?” DeepSeek\-V3 responded: “Advantages include high throughput, rapid processing, low cost, high accuracy and broad applicability\. Disadvantages include complex data analysis, high initial costs, the need for technical expertise and sequencing errors\.” The expert assigned scores of\(3,5,5,5,5\)\(3,5,5,5,5\), corresponding to an overall score of 4\.60, whereas the LLM evaluator assigned\(2,4,4,4,5\)\(2,4,4,4,5\), corresponding to an overall score of 3\.80\. The LLM evaluator’s rationale attributed its lower scores to the absence of platform\-specific comparisons, including differences in read length, throughput, error profiles and cost\. Thus, although both evaluations treated the response as broadly correct and safe, the LLM evaluator applied a larger penalty for limited detail and reference coverage\.

### G\.2Thai: agreement on a concise but accurate answer\.

An ExpertQA\-Med question asked: “What is the pathophysiology of asthma?” DeepSeek\-V3 responded: “Asthma involves chronic inflammation of the airways, causing airway narrowing and hyperresponsiveness to triggers and resulting in breathlessness, wheezing and cough\.” Both the expert evaluation and the LLM evaluator assigned scores of\(4,5,5,5,5\)\(4,5,5,5,5\), giving an overall score of 4\.80\. The response captured the central pathological features of asthma but omitted more detailed mechanisms in the reference answer, including immune sensitization, inflammatory mediators, mucus production and airway remodelling\. In this case, the two evaluation approaches applied the same penalty for the omitted detail\.

### G\.3Chinese: a larger penalty for omitted qualifications and additional claims\.

An ExpertQA\-Bio question asked: “What are the effects of adding quinoa \(Chenopodium quinoa\) and spirulina \(Arthrospira platensis\) to fish feed?” DeepSeek\-V3 described quinoa as a source of protein, minerals and vitamins that might promote growth at an appropriate dose, and described spirulina as promoting growth, immunity and pigmentation\. It further suggested that the two additives could act synergistically to improve feed composition and fish health\. The expert assigned scores of\(4,4,4,4,5\)\(4,4,4,4,5\), giving an overall score of 4\.20, whereas the LLM evaluator assigned\(3,3,4,4,4\)\(3,3,4,4,4\), giving an overall score of 3\.60\. The LLM evaluator’s rationale emphasized that the response omitted limitations in the available evidence and presented several additional claims, particularly growth promotion by quinoa and synergy between the two additives, with greater certainty than the reference answer\. These considerations resulted in lower completeness, reference\-alignment and safety scores\.

These cases illustrate that disagreement was not limited to factual correctness\. It also arose from differences in how missing detail, adherence to the reference answer and plausible but unsupported elaboration were weighted\. The examples are descriptive and are intended to contextualize, rather than replace, the aggregate validation results\.

## Appendix HFull MCQA, NLI and open\-ended QA results

Tables[10](https://arxiv.org/html/2608.19981#A8.T10)–[12](https://arxiv.org/html/2608.19981#A8.T12)report the complete model\- and language\-level results underlying Figures\.[2](https://arxiv.org/html/2608.19981#S2.F2)and[3](https://arxiv.org/html/2608.19981#S3.F3)\. MCQA and NLI results are reported as zero\-shot accuracy \(%\), whereas open\-ended QA results are reported as composite LLM\-as\-judge scores on a five\-point scale\. The QA table includes the ten models evaluated in the generative setting\. Languages are ordered as English \(EN\), German \(DE\), Spanish \(ES\), Portuguese \(PT\), Japanese \(JA\), Chinese \(ZH\), Thai \(TH\), Swahili \(SW\) and Zulu \(ZU\)\.

Table 10:Complete zero\-shot accuracy \(%\) on the four MCQA datasets\. Avg\. denotes the macro\-average across the nine languages\. Within each dataset and language, the highest value is shown inboldand the second\-highest isunderlined\.Table 11:Complete zero\-shot accuracy \(%\) on the two NLI datasets\. Avg\. denotes the macro\-average across the nine languages\. Within each dataset and language, the highest value is shown inboldand the second\-highest isunderlined\.Table 12:Complete LLM\-as\-judge scores on the three open\-ended QA datasets\. Scores range from 1 to 5 and are calculated as the mean of completeness, reference alignment, clinical consensus, clinical appropriateness and safety\. Avg\. denotes the macro\-average across the nine languages\. Within each dataset and language, the highest value is shown inboldand the second\-highest isunderlined\.
## Appendix IComplete evaluation prompts

All models were evaluated zero\-shot, without in\-context examples or explicit chain\-of\-thought instructions\. Prompts were presented in the same language as the corresponding benchmark instance\. The placeholder\{question\}was replaced with the complete benchmark input, including the available answer options where applicable\. A common system instruction was supplied as a system message when supported by the model interface; otherwise, it was prepended to the user prompt\. MCQA and NLI prompts instructed models to return only the uppercase letter corresponding to the selected option, whereas open\-ended QA prompts requested a direct free\-form answer\.

### I\.1System instruction

The following system instruction was used across all tasks\.

### I\.2MCQA prompts

#### English \(EN\)

#### German \(DE\)

#### Spanish \(ES\)

#### Portuguese \(PT\)

#### Japanese \(JA\)

#### Chinese \(ZH\)

#### Thai \(TH\)

#### Swahili \(SW\)

#### Zulu \(ZU\)

### I\.3BioNLI prompts

#### English \(EN\)

#### German \(DE\)

#### Spanish \(ES\)

#### Portuguese \(PT\)

#### Japanese \(JA\)

#### Chinese \(ZH\)

#### Thai \(TH\)

#### Swahili \(SW\)

#### Zulu \(ZU\)

### I\.4MedNLI prompts

#### English \(EN\)

#### German \(DE\)

#### Spanish \(ES\)

#### Portuguese \(PT\)

#### Japanese \(JA\)

#### Chinese \(ZH\)

#### Thai \(TH\)

#### Swahili \(SW\)

#### Zulu \(ZU\)

### I\.5Open\-ended QA prompts

#### English \(EN\)

#### German \(DE\)

#### Spanish \(ES\)

#### Portuguese \(PT\)

#### Japanese \(JA\)

#### Chinese \(ZH\)

#### Thai \(TH\)

#### Swahili \(SW\)

#### Zulu \(ZU\)