MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval

# MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval
Source: [https://arxiv.org/html/2606.24200](https://arxiv.org/html/2606.24200)
Junhyeok Lee1,2,†\\dagger,Han Jang1,3,†\\dagger,Hyeonjin Goh3,Kyu Sung Choi1,2,3,∗\* 1Seoul National University2Seoul National University College of Medicine3Seoul National University Hospital †\\daggerEqual contribution∗\*Corresponding author ent1127@snu\.ac\.kr

###### Abstract

Retrieval\-augmented generation \(RAG\) in clinical settings increasingly requires multilingual retrieval against predominantly English evidence corpora\. Multilingual medical retrieval demands three capabilities: cross\-lingual alignment, concept discrimination, and evidence retrieval\. However, existing benchmarks evaluate these only in isolation, leaving the interaction between biomedical expertise and multilingual coverage unmeasured\. We introduceMMed\-Bench\-IR, a benchmark designed to disentangle these axes across 6 languages and three structurally heterogeneous tasks: \(1\) cross\-lingual medical QA retrieval with 6,127 queries grounded in the Unified Medical Language System \(UMLS\), \(2\) concept discrimination over 4,975 confusion sets at three difficulty tiers, and \(3\) multilingual evidence retrieval for RAG with 2,040 quality\-assured queries\. The three tasks share zero concept and query overlap by design, ensuring that aggregate scores reflect genuine capability breadth\. Evaluation of ten systems across six paradigm families reveals severe cross\-lingual failure: biomedical encoders that score 0\.818 nDCG@10 in English drop to 0\.056 in Japanese, a gap that English\-only benchmarks cannot detect\.

MMed\-Bench\-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval

Junhyeok Lee1,2,†\\dagger, Han Jang1,3,†\\dagger, Hyeonjin Goh3, Kyu Sung Choi1,2,3,∗\*1Seoul National University2Seoul National University College of Medicine3Seoul National University Hospital†\\daggerEqual contribution∗\*Corresponding authorent1127@snu\.ac\.kr

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.24200v1/x1.png)Figure 1:Motivation\.\(a\) English\-only biomedical benchmarks miss non\-English queries\. \(b\) Existing multilingual medical benchmarks cover 2–3 languages and one task\. \(c\) MMed\-Bench\-IR evaluates all three axes across 6 languages and 3 writing systems\.Large language models are increasingly adopted in healthcare\(Sahni and Carrus,[2023](https://arxiv.org/html/2606.24200#bib.bib18)\), and the quality of their clinical responses depends heavily on the retrieval systems that supply supporting evidence\(Qiuet al\.,[2024](https://arxiv.org/html/2606.24200#bib.bib15); Wanget al\.,[2024b](https://arxiv.org/html/2606.24200#bib.bib22)\)\. However, these retrieval systems are predominantly built for and evaluated in English, creating a growing gap for non\-English\-speaking populations\. Language barriers are associated with reduced access to care, higher rates of adverse events, and poorer health outcomes across clinical settings\(Jooet al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib9)\), and recent work warns that AI\-powered clinical tools risk deepening these inequities by exhibiting significant performance disparities between well\-resourced and digitally under\-represented languages\(Anyaegbunaet al\.,[2026](https://arxiv.org/html/2606.24200#bib.bib2); Ortegaet al\.,[2025](https://arxiv.org/html/2606.24200#bib.bib13)\)\.

A key bottleneck lies in the retrieval stage: embedding models and dense retrievers that power these systems often lack the multilingual alignment needed to handle diverse languages\(Yuanet al\.,[2022](https://arxiv.org/html/2606.24200#bib.bib25); Liuet al\.,[2021](https://arxiv.org/html/2606.24200#bib.bib10)\), yet the field has no way to measure this shortcoming systematically\. This multilingual medical retrieval problem decomposes into three distinct capabilities: cross\-lingual alignment so that the same concept in different languages maps to similar representations, concept discrimination so that clinically confusable entities such as Type 1 versus Type 2 diabetes are correctly separated, and evidence retrieval so that queries in any language can surface relevant English\-language passages\. These three capabilities are individually well studied\(Liuet al\.,[2021](https://arxiv.org/html/2606.24200#bib.bib10); Remyet al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib16); Zhanget al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib27)\), yet no existing benchmark evaluates them jointly\(Thakuret al\.,[2021](https://arxiv.org/html/2606.24200#bib.bib19); Muennighoffet al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib12)\), leaving the field unable to measure whether progress on one dimension comes at the cost of another\.

Prior work addresses each axis only partially: biomedical encoders such as SapBERT\(Liuet al\.,[2021](https://arxiv.org/html/2606.24200#bib.bib10)\)and BioLORD\-2023\(Remyet al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib16)\)evaluate only in English, while multilingual benchmarks such as MIRACL\(Zhanget al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib27)\)lack medical concept grounding\. Recent efforts in cross\-lingual medical retrieval\(Athar Sheikhet al\.,[2025](https://arxiv.org/html/2606.24200#bib.bib3); Acharyaet al\.,[2025](https://arxiv.org/html/2606.24200#bib.bib1)\)cover limited language pairs or target multimodal rather than text\-only retrieval\. The intersection of multilingual coverage and biomedical specialization remains uncharted \(Figure[1](https://arxiv.org/html/2606.24200#S1.F1); Section[2](https://arxiv.org/html/2606.24200#S2)\)\.

To address these gaps, we introduceMMed\-Bench\-IR, a structured evaluation suite spanning 6 languages \(English, Spanish, French, Japanese, Chinese, Russian\) and 3 writing systems \(Latin, Cyrillic, CJK\)\. Inspired by the Benchmarking Information Retrieval \(BEIR\) suite\(Thakuret al\.,[2021](https://arxiv.org/html/2606.24200#bib.bib19)\), which showed that heterogeneous evaluation across diverse tasks and domains yields insights invisible in any single benchmark, we design MMed\-Bench\-IR around the same principle applied to multilingual medical information retrieval\.

It comprises three structurally heterogeneous tasks: cross\-lingual medical question answering \(QA\) retrieval with 6,127 queries grounded in the Unified Medical Language System \(UMLS\) Metathesaurus ontologyBodenreider \([2004](https://arxiv.org/html/2606.24200#bib.bib4)\), concept discrimination over 4,975 confusion sets organized into three difficulty levels, and multilingual evidence retrieval for retrieval\-augmented generation \(RAG\) with 2,040 quality\-assured queries against an 80,049\-passage English corpus\. By construction, the three tasks share zero annotated concept overlap, zero query overlap, and minimal corpus vocabulary overlap, ensuring that aggregate scores reflect genuine capability breadth rather than mastery of a single skill\.

Our contributions are as follows:

- •To the best of our knowledge, MMed\-Bench\-IR is the first benchmark to jointly evaluate multilingual alignment and biomedical specialization across 6 languages, 3 writing systems, and three tasks with zero concept and query overlap\.
- •We evaluate ten systems across six paradigm families and reveal a consistent paradigm\-level hierarchy across all tasks, with concept discrimination emerging as the hardest lexical bottleneck at a \+0\.38 gap between BM25 and the best dense model\.
- •We expose severe cross\-lingual failure modes invisible in English\-only evaluation: biomedical encoders that score 0\.818 nDCG@10 in English collapse to 0\.056 in Japanese, with a fairness gap of 0\.76\.

![Refer to caption](https://arxiv.org/html/2606.24200v1/x2.png)Figure 2:Overview of MMed\-Bench\-IR\. Three tasks target distinct retrieval capabilities across 6 languages with zero query and CUI overlap\. Scores aggregate into MMed\-IR and per\-task fairness gaps\.
## 2Related Work and Scope

#### Multilingual and biomedical retrieval models\.

Recent work has made significant progress on multilingual and biomedical retrieval independently, but the two axes remain largely disconnected\. MMedC\(Qiuet al\.,[2024](https://arxiv.org/html/2606.24200#bib.bib15)\)and Apollo\(Wanget al\.,[2024b](https://arxiv.org/html/2606.24200#bib.bib22)\)curate multilingual medical corpora for LLM training but provide no retrieval evaluation\. On the retriever side, BGE\-M3\(Chenet al\.,[2024](https://arxiv.org/html/2606.24200#bib.bib5)\)and Multilingual\-E5\(Wanget al\.,[2024a](https://arxiv.org/html/2606.24200#bib.bib23)\)support over 100 languages but lack medical concept grounding, while BioLORD\-2023\(Remyet al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib16)\)and MedCPT\(Jinet al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib8)\)advance biomedical entity representation but evaluate only in English\. UMLS\-based\(Bodenreider,[2004](https://arxiv.org/html/2606.24200#bib.bib4)\)entity linking work such as SapBERT\(Liuet al\.,[2021](https://arxiv.org/html/2606.24200#bib.bib10)\)and CODER\(Yuanet al\.,[2022](https://arxiv.org/html/2606.24200#bib.bib25)\)tests concept alignment but not passage retrieval\. No existing model jointly addresses both axes\.

#### Retrieval benchmarks\.

Several benchmarks have advanced retrieval evaluation along individual axes, but none jointly targets multilingual and biomedical capabilities\. BEIR\(Thakuret al\.,[2021](https://arxiv.org/html/2606.24200#bib.bib19)\)established that in\-domain performance does not predict out\-of\-domain generalization, motivating heterogeneous evaluation\. MTEB\(Muennighoffet al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib12)\)demonstrated that no single embedding method dominates across diverse tasks, reinforcing the need for multi\-task assessment\. However, neither targets the biomedical domain\. On the multilingual side, MIRACL\(Zhanget al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib27)\)and Mr\. TyDi\(Zhanget al\.,[2021](https://arxiv.org/html/2606.24200#bib.bib26)\)cover diverse languages but do not incorporate medical terminology or ontology structure\. Despite these advances, no existing benchmark can evaluate the interaction between multilingual coverage and biomedical specialization\.

#### Cross\-lingual medical retrieval\.

A small but growing body of work has begun to address cross\-lingual retrieval in the medical domain directly\. The MUCHMORE project\(Volket al\.,[2002](https://arxiv.org/html/2606.24200#bib.bib21)\)explored UMLS\-based cross\-language information retrieval \(CLIR\) over English and German medical abstracts, establishing medical CLIR as a research direction\. More recently, CURE\(Athar Sheikhet al\.,[2025](https://arxiv.org/html/2606.24200#bib.bib3)\)evaluates point\-of\-care passage ranking across English, French\-to\-English, and Spanish\-to\-English conditions but covers only three languages and a single retrieval task\. M3Retrieve\(Acharyaet al\.,[2025](https://arxiv.org/html/2606.24200#bib.bib1)\)targets multimodal medical retrieval rather than multilingual text\-only information retrieval\. MMed\-Bench\-IR differs by jointly evaluating concept alignment, concept discrimination, and evidence retrieval within a single benchmark spanning 6 languages and 3 writing systems\.

## 3Benchmark Design Principles

We design MMed\-Bench\-IR around the principle that benchmark value comes from principled heterogeneity rather than dataset count, and organize it along four selection criteria\.

#### Capability heterogeneity\.

The three tasks isolate distinct retrieval capabilities: cross\-lingual concept alignment \(Task 1\), fine\-grained semantic discrimination \(Task 2\), and cross\-lingual passage retrieval \(Task 3\)\. This separation ensures that a model’s aggregate score cannot be driven by strength in a single capability\.

#### Linguistic heterogeneity\.

Six languages span three language families \(Indo\-European \[en, es, fr, ru\], Sino\-Tibetan \[zh\], Japonic \[ja\]\), three writing systems \(Latin, Cyrillic, CJK\), and varying levels of medical NLP resource availability\. This ensures that benchmark scores reflect genuine multilingual robustness rather than performance on closely related languages\.

#### Difficulty heterogeneity\.

Task 1 spans 500 medical concepts across 5 languages with varying UMLS coverage, creating natural difficulty gradients across languages\. Task 2 includes three difficulty tiers \(synonym, sibling, related\-but\-distinct\) validated by multi\-encoder consensus\. Task 3 uses quality\-assured translations with an 83\.7% retention rate\. The resulting score distributions span a 5×\\timesrange across systems, with no task saturated\.

#### Validation heterogeneity\.

Each task uses a different validation strategy targeting a different error source: UMLS ontology grounding \(Task 1\), multi\-encoder majority vote \(Task 2\), and bilingual concept\-fidelity and back\-translation QA \(Task 3\)\. This ensures that no single validation blind spot propagates across tasks\.

Table 1:MMed\-Bench\-IR benchmark statistics\. The three tasks differ structurally in retrieval setting, corpus scale, query granularity, and relevance definition, ensuring that aggregate scores reflect capability breadth\.

## 4Benchmark Construction and Characterization

### 4\.1Benchmark Statistics

Table[1](https://arxiv.org/html/2606.24200#S3.T1)and Figure[2](https://arxiv.org/html/2606.24200#S1.F2)summarize the benchmark\. The three tasks are structurally complementary: they share zero CUI overlap between Task 1 and Task 2 concept spaces, and the pairwise corpus vocabulary overlap is low \(Jaccard<<0\.14 between Tasks 1/2,<<0\.02 between either and Task 3\)\. Query granularity ranges from short concept terms \(4 to 5 words\) to biomedical questions \(6\.7 words\), and corpus scale spans two orders of magnitude \(2,552 to 80,049 documents\)\.

### 4\.2Task Construction

#### Task 1: Cross\-lingual Medical QA Retrieval\.

From MMedBenchQiuet al\.\([2024](https://arxiv.org/html/2606.24200#bib.bib15)\)test\-split multilingual medical QA data, we extract question stems \(without answer options\) as queries across 5 languages \(French is excluded because fewer than 50 queries passed the CUI\-tagging pipeline at this granularity\)\. Each query is tagged with a UMLS CUI via a cascading tagger that first attempts exact lexical match against UMLS preferred terms and synonyms, then falls back to a biomedical linker for unmatched queries\. Queries sharing the same CUI across languages form cross\-lingual positive groups\. The corpus consists of concept definition texts \(1,290 positives \+ 1,262 hard\-negative concepts from non\-matching CUIs\), yielding 6,127 queries and 2,552 corpus documents across 500 concepts\.

#### Task 2: Concept Discrimination\.

From the UMLS 2025AB Metathesaurus concept file \(MRCONSO\.RRF\)\(Bodenreider,[2004](https://arxiv.org/html/2606.24200#bib.bib4)\), we parse 1\.5M concept atoms grouped by source vocabulary and construct confusion sets at three tiers: Tier 1 \(synonym\) contains the same CUI with different surface forms or languages\. Tier 2 \(sibling\) pairs different CUIs sharing the same source vocabulary and 3\-character code prefix, meaning they are clinically related but distinct \(e\.g\., ICD\-10 codes under the same category\)\. Tier 3 \(related\) pairs different CUIs from the same source vocabulary but different code prefix groups, meaning they belong to the same medical domain but are clearly distinct\. Queries are drawn from English and Chinese surface forms, and positives are different surface forms of the same CUI in any language, with the query’s own form excluded from the corpus\. All 6,340 corpus terms across multiple languages form a shared retrieval pool\. Because each confusion set generates one query per unique CUI it contains, the 4,975 validated sets yield 4,143 evaluation queries\.

#### Task 3: Multilingual Evidence Retrieval for RAG\.

340 English biomedical questions from BioASQ 13b\(Tsatsaroniset al\.,[2015](https://arxiv.org/html/2606.24200#bib.bib20)\)with 80,049 evidence snippets are translated to 5 target languages using NLLB\-200\-3\.3B\(Costa\-Jussàet al\.,[2022](https://arxiv.org/html/2606.24200#bib.bib7)\)\. Translation quality is assured via two checks\. The first is concept fidelity, where each translation is verified against a 587K\-entry UMLS bilingual lexicon plus a set of 87 protected medical abbreviations \(e\.g\., HIV, MRI, HbA1c\) that must be preserved\. The second is back\-translation consistency, where each translation is back\-translated via NLLB\-200 and compared with the original using token\-overlap F1\. Queries passing at least one check form the official evaluation subset \(1,708 of 2,040, 83\.7%\)\.

### 4\.3Heterogeneity Analysis

![Refer to caption](https://arxiv.org/html/2606.24200v1/x3.png)Figure 3:Benchmark heterogeneity\.\(a\) Inter\-task corpus vocabulary overlap is low \(Jaccard≤\\leq0\.14\), confirming that the three tasks target distinct lexical domains\. \(b\) Language distributions vary across tasks: Task 1 is English\-dominant due to UMLS coverage, Task 2 uses English and Chinese only, and Task 3 is uniformly balanced across all six languages\.Figure[3](https://arxiv.org/html/2606.24200#S4.F3)confirms that the three tasks are structurally independent\. Vocabulary overlap between Task 3 \(biomedical passages\) and either term\-level task is below 0\.02 Jaccard, reflecting fundamentally different retrieval regimes\. The language distributions differ by design: Task 1 reflects natural UMLS coverage \(English\-dominant\), Task 2 captures the two most distinct scripts \(en, zh\), and Task 3 is uniformly balanced\. Zero queries overlap between any two tasks, and zero annotated CUIs overlap between Tasks 1 and 2 \(Task 3 uses BioASQ questions without CUI annotation\)\. These properties ensure that a model’s aggregate performance cannot be inflated by excelling on a single lexical domain or language\.

### 4\.4Validation

Benchmark validity rests on three controls, each targeting a different error source\.

#### Task 1\.

Ontology\-grounded positives via UMLS CUI alignment ensure face validity\.

#### Task 2\.

Tier definitions are anchored in UMLS ontology structure \(synonym, sibling, related\-but\-distinct\), providing an encoder\-independent criterion\. A 3\-encoder majority vote using BGE\-M3\(Chenet al\.,[2024](https://arxiv.org/html/2606.24200#bib.bib5)\), E5\-large\(Wanget al\.,[2024a](https://arxiv.org/html/2606.24200#bib.bib23)\), and SapBERT\(Liuet al\.,[2021](https://arxiv.org/html/2606.24200#bib.bib10)\)operationalizes acceptance thresholds based on pairwise cosine similarity \(Tier 1:≥\\geq0\.70, Tier 2: 0\.40–0\.75, Tier 3: 0\.15–0\.45\), requiring agreement from at least two of three encoders\. This process rejects 967 sets \(16\.3%\) and re\-tiers 1,597, yielding 4,975 validated sets\. To verify that the tier ordering is not biased toward the validators, we confirm that BM25 and BioLORD\-2023\(Remyet al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib16)\), neither of which participated in validation, exhibit the same monotonic Tier 1, Tier 2, Tier 3 ordering \(Table[3](https://arxiv.org/html/2606.24200#S6.T3)\), with tier boundaries confirmed by non\-overlapping confidence intervals \(Appendix[A](https://arxiv.org/html/2606.24200#A1)\)\.

#### Task 3\.

Two\-stage translation quality assurance based on concept fidelity and back\-translation consistency retains 83\.7% of queries\. Full thresholds and per\-language breakdowns are in Appendix[A](https://arxiv.org/html/2606.24200#A1)\.

### 4\.5Scoring

The primary scoreMMed\-IRis the macro average of three task scores:

MMed\-IR=13​\(Task1score\+Task2score\+Task3score\)\\text\{MMed\-IR\}=\\frac\{1\}\{3\}\\bigl\(\\text\{Task1\}\_\{\\text\{score\}\}\+\\text\{Task2\}\_\{\\text\{score\}\}\+\\text\{Task3\}\_\{\\text\{score\}\}\\bigr\)\(1\)where Task 1 score==mean\(nDCG@10, R@1\), Task 2 score==mean\(R@1Tier1\{\}\_\{\\text\{Tier1\}\}, R@1Tier2\{\}\_\{\\text\{Tier2\}\}, R@1Tier3\{\}\_\{\\text\{Tier3\}\}\), and Task 3 score==R@5\. The fairness gap for each task ismaxℓ⁡\(nDCG@10ℓ\)−minℓ⁡\(nDCG@10ℓ\)\\max\_\{\\ell\}\(\\text\{nDCG@10\}\_\{\\ell\}\)\-\\min\_\{\\ell\}\(\\text\{nDCG@10\}\_\{\\ell\}\)across languagesℓ\\ell, providing a uniform cross\-lingual equity measure\.

Table 2:MMed\-Bench\-IR main results with query\-level bootstrap 95% confidence intervals \(n=2,000n\{=\}2\{,\}000\)\. All paradigm\-level differences exceed CI widths\. Best inbold, second\-bestunderlined\.†Within\-distribution\.

## 5Experimental Setup

### 5\.1Baselines

We evaluate ten systems spanning six paradigm families to characterize the benchmark’s discriminative properties:

- •Lexical: BM25\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2606.24200#bib.bib17)\)\(Okapi, whitespace tokenization; no CJK\-specific segmentation, which disadvantages BM25 on Japanese/Chinese\)\.
- •Biomedical dense: SapBERT\(Liuet al\.,[2021](https://arxiv.org/html/2606.24200#bib.bib10)\)and BioLORD\-2023\(Remyet al\.,[2023](https://arxiv.org/html/2606.24200#bib.bib16)\), English\-centric biomedical encoders\.
- •Multilingual dense: BGE\-M3\(Chenet al\.,[2024](https://arxiv.org/html/2606.24200#bib.bib5)\)and E5\-large\(Wanget al\.,[2024a](https://arxiv.org/html/2606.24200#bib.bib23)\), general\-purpose encoders supporting over 100 languages\.
- •Late\-interaction: ColBERT\-XM\(Louiset al\.,[2025](https://arxiv.org/html/2606.24200#bib.bib11)\), a multilingual late\-interaction retriever based on X\-MOD\(Pfeifferet al\.,[2022](https://arxiv.org/html/2606.24200#bib.bib14)\)with per\-token scoring, trained on MS\-MARCO\.
- •Hybrid: BM25\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2606.24200#bib.bib17)\)and BGE\-M3\(Chenet al\.,[2024](https://arxiv.org/html/2606.24200#bib.bib5)\)combined via reciprocal rank fusion \(RRF,k=60k\{=\}60\)\.
- •Two\-stage reranker: BGE\-M3\(Chenet al\.,[2024](https://arxiv.org/html/2606.24200#bib.bib5)\)retrieval followed by BGE\-reranker\-v2\-m3 cross\-encoder reranking \(top\-30\)\.
- •Multilingual and medical: MMed\-Embed \(Section[5\.2](https://arxiv.org/html/2606.24200#S5.SS2)\) and MMed\-Embed with Reranker, within\-distribution references\.

MMed\-Embed combines both biomedical specialization and multilingual coverage, providing an approximate within\-distribution ceiling\.

### 5\.2Reference Model: MMed\-Embed

To assess the benchmark’s discriminative range above existing baselines, we fine\-tune BGE\-M3\(Chenet al\.,[2024](https://arxiv.org/html/2606.24200#bib.bib5)\)\(568M parameters, XLM\-RoBERTa\(Conneauet al\.,[2020](https://arxiv.org/html/2606.24200#bib.bib6)\)backbone\) on 137K training rows from MMedBench\(Qiuet al\.,[2024](https://arxiv.org/html/2606.24200#bib.bib15)\)\(40K, 6 languages\), UMLS\(Bodenreider,[2004](https://arxiv.org/html/2606.24200#bib.bib4)\)cross\-lingual pairs \(87K, 5 languages\), and xBioASQ \(10K\)\. No overlap exists with benchmark test sets \(verified: 0% query overlap on Task 3;<<0\.5% trivial term overlap on Tasks 1/2\)\.

#### Training objective\.

ℒret\\mathcal\{L\}\_\{\\text\{ret\}\}is temperature\-scaled cross\-entropy over cosine similarities between each query and all in\-batch positives plusk=15k\{=\}15hard negatives, with temperatureτ=0\.02\\tau\{=\}0\.02\.

#### Training procedure\.

We fine\-tune BGE\-M3 on the full 137K rows with 15 random in\-batch negatives per query for 10 epochs \(effective batch 1,024, lr2×10−62\{\\times\}10^\{\-6\}, 1,330 steps\) using AdamW with linear warmup \(10%\) and linear decay, mean pooling with L2 normalization, max sequence length 256, gradient checkpointing, and mixed precision \(bf16\) on 8×\\timesB200 GPUs\. A 2×\\times2 ablation \(Appendix[B](https://arxiv.org/html/2606.24200#A2)\) shows that neither topology\-aware margin loss nor ANCE\-style hard negative mining\(Xionget al\.,[2020](https://arxiv.org/html/2606.24200#bib.bib24)\)yields statistically significant gains over this baseline recipe; the primary benefit comes from domain\-specific contrastive fine\-tuning on multilingual medical data\. Full hyperparameters are in Appendix[B](https://arxiv.org/html/2606.24200#A2)\.

#### Results\.

MMed\-Embed achieves 0\.369 MMed\-IR, a marginal \+0\.022 over the zero\-shot BGE\-M3 baseline \(0\.347\)\. Combining MMed\-Embed with cross\-encoder reranking yields the best overall score \(0\.377\) and the lowest fairness gap \(0\.170\), showing that domain\-specific first\-stage retrieval and cross\-encoder reranking are complementary\. The modest gain from 137K\-row fine\-tuning suggests that the benchmark’s primary challenge lies in architecture\-level multilingual alignment rather than in training data or loss design; MMed\-Embed scores should be interpreted as an approximate within\-distribution ceiling rather than a fair zero\-shot comparison\.

## 6Results and Analysis

### 6\.1Overall Comparison

Table[2](https://arxiv.org/html/2606.24200#S4.T2)presents the main results across all ten systems\. The spread from BM25 \(0\.075\) to the best system \(0\.377\) spans a 5×\\timesrange, confirming effective discrimination across paradigm families, with substantial headroom remaining\.

### 6\.2Key Findings

#### Finding A: Consistent hierarchy across paradigms and tasks\.

All three tasks produce the same paradigm ordering: lexical methods trail biomedical\-only encoders, which in turn trail multilingual dense retrievers \(Table[2](https://arxiv.org/html/2606.24200#S4.T2)\)\. This ordering holds across all six paradigm families, including ColBERT\-XM \(0\.295\), which falls between biomedical encoders \(SapBERT 0\.201\) and multilingual dense retrievers \(BGE\-M3 0\.347\), consistent with its English\-centric contrastive training\. Hybrid RRF \(0\.163\) underperforms standalone dense retrieval, while cross\-encoder reranking \(0\.371\) and MMed\-Embed with Reranker \(0\.377\) occupy the top tier\. This consistency across tasks with zero query and CUI overlap cannot be attributed to data leakage\.

#### Finding B: Concept discrimination is the hardest lexical bottleneck\.

Task 2 most sharply separates paradigms: BM25 scores 0\.076 versus the best model at 0\.454, a \+0\.38 gap \(Table[2](https://arxiv.org/html/2606.24200#S4.T2)\)\. Hybrid RRF degrades Task 2 performance \(0\.124 vs\. 0\.432 for BGE\-M3 alone\), confirming that lexical signal is counterproductive for concept discrimination\. Tier 3 \(related\-but\-distinct concepts\) remains a genuine frontier: even the strongest model reaches only 3\.6%, and ColBERT\-XM’s token\-level scoring yields only 0\.8% \(Table[3](https://arxiv.org/html/2606.24200#S6.T3)\)\.

#### Finding C: Biomedical specialization does not rescue cross\-lingual failure\.

SapBERT achieves 0\.818 nDCG@10 in English but collapses to 0\.056 in Japanese \(Table[4](https://arxiv.org/html/2606.24200#S6.T4)\), with a fairness gap of 0\.76±\\pm0\.04 versus 0\.24±\\pm0\.05 for BGE\-M3, a 3×\\timesequity difference with non\-overlapping CIs\. ColBERT\-XM exhibits the same pattern: its Task 3 fairness gap \(0\.37\) exceeds BGE\-M3 \(0\.24\), confirming that late\-interaction scoring alone does not rescue cross\-lingual retrieval when contrastive training is English\-centric\. Cross\-encoder reranking achieves the lowest fairness gap \(0\.18±\\pm0\.04\)\. Full per\-task gaps are in Appendix[C](https://arxiv.org/html/2606.24200#A3)\.

### 6\.3Evaluation Impact

Each finding requires the multi\-axis evaluation that MMed\-Bench\-IR provides: the consistent hierarchy emerges only from multi\-task comparison, the lexical bottleneck only from concept\-level retrieval, and the cross\-lingual collapse only from joint multilingual and biomedical evaluation\.

Table 3:Task 2 Recall@1 by difficulty tier with query\-level bootstrap 95% CIs\.Table 4:Task 3 nDCG@10 by query language with bootstrap 95% CIs and fairness gap\. Gap CIs computed via bootstrap of max−\-min across languages\.

## 7Benchmark Validity and Bias Analysis

#### Annotation bias\.

All three tasks use automated relevance judgments rather than human annotations, eliminating inter\-annotator disagreement but introducing potential systematic biases\. For Task 2, tier definitions are anchored in encoder\-independent UMLS ontology relations, and both BM25 and BioLORD\-2023 \(neither of which participated in validation\) exhibit the same monotonic tier ordering \(Table[3](https://arxiv.org/html/2606.24200#S6.T3)\)\. A leave\-one\-validator\-out analysis confirms≥\\geq85\.4% tier stability across all encoder\-pair conditions \(Appendix[A](https://arxiv.org/html/2606.24200#A1)\)\.

#### Translation bias\.

Task 3 uses NLLB\-200 machine translation, which may introduce systematic quality differences across languages\. Model ranking is perfectly preserved between the official \(1,708 queries\) and all\-translated \(2,040\) subsets \(Spearmanρ=1\.0\\rho\{=\}1\.0\), confirming that QA filtering does not introduce ranking artifacts \(Appendix[A](https://arxiv.org/html/2606.24200#A1)\)\.

#### LLM\-based quality audit\.

We conduct a 600\-sample quality audit using Claude Opus 4\.7 \(Anthropic, 2026\) across all 12 task×\\timeslanguage cells \(50 per cell; Table[5](https://arxiv.org/html/2606.24200#S7.T5)\)\. Task 1 CUI alignment accuracy is≥\\geq0\.98 across all languages; Task 2 Tier\-1 synonym accuracy is 1\.00\. For Task 3 translations, ES achieves 1\.00, FR 0\.86, RU 0\.92, while JA and ZH drop to 0\.80 and 0\.78 respectively, consistent with limited UMLS coverage for non\-Latin scripts \(Section[Limitations](https://arxiv.org/html/2606.24200#Sx1)\)\. On a 50\-query French subset, an author independently evaluated translation fidelity under six predefined medical\-translation criteria; inter\-rater agreement with the LLM judge was 98% \(Cohen’sκ=0\.92\\kappa\{=\}0\.92; Gwet’s AC1=0\.97\{=\}0\.97\), with the LLM judge identifying 7 of 8 human\-flagged errors\.

Table 5:LLM\-based quality audit accuracy per task and language \(n=50n\{=\}50per cell\)\. Evaluator: Claude Opus 4\.7 \(Anthropic, 2026\)\.

## 8Release and Societal Considerations

Evaluation scripts, benchmark splits, baseline results, and construction code are provided as supplementary material and will be hosted on HuggingFace Datasets\. Reproducing from raw sources requires UMLS 2025AB \(NLM license\) and BioASQ 13b \(registration\); the released package includes all derived artifacts needed for evaluation\. All data derives from licensed sources with no patient\-identifiable information\. The fairness gap metric operationalizes cross\-lingual equity; benchmark scores measure system capability, not clinical safety\.

## 9Conclusion

In this work, we introduce MMed\-Bench\-IR, a heterogeneous benchmark that jointly evaluates multilingual alignment and biomedical specialization in medical information retrieval across 6 languages and 3 writing systems\. Its three structurally distinct tasks share zero query and concept overlap, ensuring that aggregate scores reflect genuine capability breadth rather than mastery of a single skill\. Evaluation of ten systems across six paradigm families reveals consistent paradigm\-level rankings, identifies concept discrimination as the hardest lexical bottleneck, and exposes the collapse of biomedical models on non\-Latin scripts, failure modes detectable only through multi\-axis evaluation\. We hope that MMed\-Bench\-IR serves as a foundation for developing retrieval systems that work equitably across languages in clinical settings\.

## Limitations

MMed\-Bench\-IR provides the first joint evaluation of multilingual alignment and biomedical specialization in medical retrieval, but several design choices constrain the current version\.

Task 3 results for Japanese and Chinese should be interpreted as conservative estimates\. An LLM\-based audit \(Table[5](https://arxiv.org/html/2606.24200#S7.T5)\) confirms that 78 to 80% of JA/ZH translations preserve clinical meaning, while ES achieves 100% and FR achieves 86%, consistent with limited UMLS coverage for non\-Latin scripts\. Future versions could benefit from ensemble translation or human post\-editing for CJK languages\.

The Task 2 corpus of 6,340 terms constrains negative discrimination difficulty, particularly at Tier 3 where only 325 confusion sets are available\. This limits the statistical power of Tier 3 evaluation, as reflected in the wide bootstrap confidence intervals \(Table[3](https://arxiv.org/html/2606.24200#S6.T3)\)\. Scaling the confusion set corpus with additional UMLS source vocabularies is a priority for future versions\.

Six languages exclude Arabic, Hindi, and other high\-population languages with significant healthcare needs, though the construction framework is designed for community extension\. Task 1 excludes French because fewer than 50 French queries passed the CUI\-tagging pipeline at the 500\-concept granularity\. French is retained in Task 3 via NLLB\-derived translations\.

MMed\-Embed is the only within\-distribution system evaluated, and the current baselines do not include recent LLM\-based embedding models \(e\.g\., GritLM, E5\-Mistral\)\. Community submissions would strengthen evaluation and broaden paradigm coverage beyond the six families tested here\.

All three tasks rely on automated relevance judgments rather than expert human annotation\. While Section[7](https://arxiv.org/html/2606.24200#S7)demonstrates high agreement between LLM\-based audits and human evaluation \(κ=0\.92\\kappa\{=\}0\.92\), systematic biases may still exist for underrepresented languages where reference resources are sparse\. Incorporating expert clinician review across all six languages remains an important direction for future iterations\.

## References

- M3Retrieve: benchmarking multimodal retrieval for medicine\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 15263–15276\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.771/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.771),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p3.1),[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Anyaegbuna, N\. Steele, A\. S\. Liang, S\. P\. Ma, I\. Lopez, N\. Chilukuri, K\. Patel, K\. Schulman, and J\. H\. Chen \(2026\)Artificial intelligence translation in healthcare: an urgent call for evidence\-informed policy frameworks\.BMJ Health & Care Informatics33\(1\),pp\. e102007\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p1.1)\.
- N\. Athar Sheikh, D\. Buades Marcos, A\. Jousse, A\. Oladipo, O\. Rousseau, and J\. Lin \(2025\)CURE: a dataset for clinical understanding & retrieval evaluation\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 5270–5277\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p3.1),[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px3.p1.1)\.
- O\. Bodenreider \(2004\)The unified medical language system \(umls\): integrating biomedical terminology\.Nucleic acids research32\(suppl\_1\),pp\. D267–D270\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p5.1),[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2606.24200#S4.SS2.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2606.24200#S5.SS2.p1.1)\.
- J\. Chen, S\. Xiao, P\. Zhang, K\. Luo, D\. Lian, and Z\. Liu \(2024\)M3\-embedding: multi\-linguality, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 2318–2335\.External Links:[Link](https://aclanthology.org/2024.findings-acl.137/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.137)Cited by:[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px1.p1.1),[§4\.4](https://arxiv.org/html/2606.24200#S4.SS4.SSS0.Px2.p1.1),[3rd item](https://arxiv.org/html/2606.24200#S5.I1.i3.p1.1),[5th item](https://arxiv.org/html/2606.24200#S5.I1.i5.p1.1),[6th item](https://arxiv.org/html/2606.24200#S5.I1.i6.p1.1),[§5\.2](https://arxiv.org/html/2606.24200#S5.SS2.p1.1)\.
- A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov \(2020\)Unsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 8440–8451\.Cited by:[§5\.2](https://arxiv.org/html/2606.24200#S5.SS2.p1.1)\.
- M\. R\. Costa\-Jussà, J\. Cross, O\. Çelebi, M\. Elbayad, K\. Heafield, K\. Heffernan, E\. Kalbassi, J\. Lam, D\. Licht, J\. Maillard,et al\.\(2022\)No language left behind: scaling human\-centered machine translation\.arXiv preprint arXiv:2207\.04672\.Cited by:[§4\.2](https://arxiv.org/html/2606.24200#S4.SS2.SSS0.Px3.p1.1)\.
- Q\. Jin, W\. Kim, Q\. Chen, D\. C\. Comeau, L\. Yeganova, W\. J\. Wilbur, and Z\. Lu \(2023\)Medcpt: contrastive pre\-trained transformers with large\-scale pubmed search logs for zero\-shot biomedical information retrieval\.Bioinformatics39\(11\),pp\. btad651\.Cited by:[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Joo, A\. Fernandez, E\. C\. Wick, G\. Moreno Lepe, and S\. P\. Manuel \(2023\)Association of language barriers with perioperative and surgical outcomes: a systematic review\.JAMA Network Open6\(7\),pp\. e2322743\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p1.1)\.
- F\. Liu, E\. Shareghi, Z\. Meng, M\. Basaldella, and N\. Collier \(2021\)Self\-alignment pretraining for biomedical entity representations\.InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies,pp\. 4228–4238\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p2.1),[§1](https://arxiv.org/html/2606.24200#S1.p3.1),[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px1.p1.1),[§4\.4](https://arxiv.org/html/2606.24200#S4.SS4.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2606.24200#S5.I1.i2.p1.1)\.
- A\. Louis, V\. K\. Saxena, G\. Van Dijck, and G\. Spanakis \(2025\)Colbert\-xm: a modular multi\-vector representation model for zero\-shot multilingual information retrieval\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 4370–4383\.Cited by:[4th item](https://arxiv.org/html/2606.24200#S5.I1.i4.p1.1)\.
- N\. Muennighoff, N\. Tazi, L\. Magne, and N\. Reimers \(2023\)Mteb: massive text embedding benchmark\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 2014–2037\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p2.1),[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px2.p1.1)\.
- P\. Ortega, A\. Miller De Rutté, and M\. Vela \(2025\)Language equity in health technology for patients with non–english language preference\.JAMA Network Open8\(2\),pp\. e2457424\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p1.1)\.
- J\. Pfeiffer, N\. Goyal, X\. V\. Lin, X\. Li, J\. Cross, S\. Riedel, and M\. Artetxe \(2022\)Lifting the curse of multilinguality by pre\-training modular transformers\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 3479–3495\.External Links:[Link](https://aclanthology.org/2022.naacl-main.255/),[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.255)Cited by:[4th item](https://arxiv.org/html/2606.24200#S5.I1.i4.p1.1)\.
- P\. Qiu, C\. Wu, X\. Zhang, W\. Lin, H\. Wang, Y\. Zhang, Y\. Wang, and W\. Xie \(2024\)Towards building multilingual language model for medicine\.Nature Communications15\(1\),pp\. 8384\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p1.1),[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2606.24200#S4.SS2.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2606.24200#S5.SS2.p1.1)\.
- F\. Remy, K\. Demuynck, and T\. Demeester \(2023\)Biolord\-2023: semantic textual representations fusing llm and clinical knowledge graph insights\.arXiv preprint arXiv:2311\.16075\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p2.1),[§1](https://arxiv.org/html/2606.24200#S1.p3.1),[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px1.p1.1),[§4\.4](https://arxiv.org/html/2606.24200#S4.SS4.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2606.24200#S5.I1.i2.p1.1)\.
- S\. Robertson and H\. Zaragoza \(2009\)The probabilistic relevance framework: bm25 and beyond\.Vol\.4,Now Publishers Inc\.Cited by:[1st item](https://arxiv.org/html/2606.24200#S5.I1.i1.p1.1),[5th item](https://arxiv.org/html/2606.24200#S5.I1.i5.p1.1)\.
- N\. R\. Sahni and B\. Carrus \(2023\)Artificial intelligence in us health care delivery\.New England Journal of Medicine389\(4\),pp\. 348–358\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p1.1)\.
- N\. Thakur, N\. Reimers, A\. Rücklé, A\. Srivastava, and I\. Gurevych \(2021\)BEIR: a heterogeneous benchmark for zero\-shot evaluation of information retrieval models\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track \(Round 2\),External Links:[Link](https://openreview.net/forum?id=wCu6T5xFjeJ)Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p2.1),[§1](https://arxiv.org/html/2606.24200#S1.p4.1),[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Tsatsaronis, G\. Balikas, P\. Malakasiotis, I\. Partalas, M\. Zschunke, M\. R\. Alvers, D\. Weissenborn, A\. Krithara, S\. Petridis, D\. Polychronopoulos,et al\.\(2015\)An overview of the bioasq large\-scale biomedical semantic indexing and question answering competition\.BMC bioinformatics16\(1\),pp\. 138\.Cited by:[§4\.2](https://arxiv.org/html/2606.24200#S4.SS2.SSS0.Px3.p1.1)\.
- M\. Volk, B\. Ripplinger, Š\. Vintar, P\. Buitelaar, D\. Raileanu, and B\. Sacaleanu \(2002\)Semantic annotation for concept\-based cross\-language medical information retrieval\.International Journal of Medical Informatics67\(1\-3\),pp\. 97–112\.Cited by:[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px3.p1.1)\.
- L\. Wang, N\. Yang, X\. Huang, L\. Yang, R\. Majumder, and F\. Wei \(2024a\)Multilingual e5 text embeddings: a technical report\.arXiv preprint arXiv:2402\.05672\.Cited by:[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px1.p1.1),[§4\.4](https://arxiv.org/html/2606.24200#S4.SS4.SSS0.Px2.p1.1),[3rd item](https://arxiv.org/html/2606.24200#S5.I1.i3.p1.1)\.
- X\. Wang, N\. Chen, J\. Chen, Y\. Wang, G\. Zhen, C\. Zhang, X\. Wu, Y\. Hu, A\. Gao, X\. Wan,et al\.\(2024b\)Apollo: a lightweight multilingual medical llm towards democratizing medical ai to 6b people\.arXiv preprint arXiv:2403\.03640\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p1.1),[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Xiong, C\. Xiong, Y\. Li, K\. Tang, J\. Liu, P\. Bennett, J\. Ahmed, and A\. Overwijk \(2020\)Approximate nearest neighbor negative contrastive learning for dense text retrieval\.arXiv preprint arXiv:2007\.00808\.Cited by:[Appendix B](https://arxiv.org/html/2606.24200#A2.SS0.SSS0.Px1.p1.3),[§5\.2](https://arxiv.org/html/2606.24200#S5.SS2.SSS0.Px2.p1.3)\.
- Z\. Yuan, Z\. Zhao, H\. Sun, J\. Li, F\. Wang, and S\. Yu \(2022\)CODER: knowledge\-infused cross\-lingual medical term embedding for term normalization\.Journal of biomedical informatics126,pp\. 103983\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p2.1),[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Zhang, X\. Ma, P\. Shi, and J\. Lin \(2021\)Mr\. tydi: a multi\-lingual benchmark for dense retrieval\.InProceedings of the 1st workshop on multilingual representation learning,pp\. 127–137\.Cited by:[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Zhang, N\. Thakur, O\. Ogundepo, E\. Kamalloo, D\. Alfonso\-Hermelo, X\. Li, Q\. Liu, M\. Rezagholizadeh, and J\. Lin \(2023\)Miracl: a multilingual retrieval dataset covering 18 diverse languages\.Transactions of the Association for Computational Linguistics11,pp\. 1114–1131\.Cited by:[§1](https://arxiv.org/html/2606.24200#S1.p2.1),[§1](https://arxiv.org/html/2606.24200#S1.p3.1),[§2](https://arxiv.org/html/2606.24200#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix ABenchmark Construction Details

### A\.1Task 1: Data Statistics

Table A1:Task 1 query distribution by language\.Total corpus: 2,552 documents \(1,290 positive \+ 1,262 hard negatives\)\. All queries grounded with UMLS CUI tags \(100% coverage\)\.

### A\.2Task 2: Multi\-Encoder Validation

Three independent encoders \(BGE\-M3, E5\-large, SapBERT\) compute pairwise cosine similarities for each confusion set\. A set is accepted only if≥\\geq2/3 encoders agree on the tier assignment based on cosine similarity thresholds: Tier 1:≥\\geq0\.70; Tier 2: 0\.40–0\.75; Tier 3: 0\.15–0\.45\.

Table A2:Tier validation statistics\. Mean cosine similarity \(3\-encoder average\) confirms monotonic tier ordering with non\-overlapping confidence intervals\.
### A\.3Task 2: Leave\-One\-Validator\-Out Analysis

To assess whether tier assignments depend on any single validator, we re\-run the validation using each 2\-encoder pair \(requiring 2/2 agreement\) and compare with the full 3\-encoder result\.

Table A3:Leave\-one\-validator\-out sensitivity\. Among sets accepted by each 2\-encoder pair, we report the fraction matching the full 3\-encoder tier assignment\. No set shifts to a non\-adjacent tier in any condition\.The stricter 2/2 agreement threshold naturally reduces the number of accepted sets, but among those accepted, tier assignments are highly stable\. When E5\-large is excluded, 509 sets \(14\.6%\) shift between adjacent tiers \(predominantly Tier 1→\\toTier 2\), reflecting differences in the encoders’ similarity calibration rather than fundamental tier ambiguity\.

### A\.4Task 3: Translation Quality Assurance

#### Concept fidelity\.

A bilingual medical lexicon built from 587K UMLS English terms mapped to target\-language equivalents verifies that medical concepts are preserved in translation\. Standard local\-language medical terminology is accepted, not just English surface forms\.

#### Back\-translation consistency\.

Each translation is back\-translated to English via NLLB\-200 and compared with the original via token\-overlap F1\.

#### Audit status thresholds\.

Pass: term preservation rate≥\\geq0\.9 AND back\-translation F1≥\\geq0\.8\.Fail: term preservation<<0\.7 OR back\-translation F1<<0\.6\.Flag: all other cases \(at least one criterion met but not all\)\. Queries with status “pass” or “flag” form the official evaluation subset\.

#### CUI tagging thresholds\.

Task 1 queries are CUI\-tagged via a cascading pipeline: \(1\) exact lexical match against UMLS preferred terms \(confidence≥\\geq0\.9, substring match requires≥\\geq3 characters\); \(2\) biomedical linker fallback \(confidence≥\\geq0\.7\); \(3\) null if both fail\.

Table A4:Translation QA results by language\. This table covers the 5 translated languages only \(English source excluded\)\. Official subset includespass\+flagqueries\. Total official = 1,368 translated \+ 340 English source = 1,708\.

### A\.5Train/Test Separation Verification

Table A5:Query text overlap between MMed\-Embed training data \(137K rows\) and benchmark test sets\.Table A6:MMed\-Embed 2×\\times2 factorial ablation with bootstrap 95% CIs\. All variants: 137K rows, 10 epochs, identical hyperparameters\.Task 2 overlaps are short UMLS terms \(e\.g\., “ATC”, “airway resistance”\) common to both training vocabulary and confusion set members\. No semantic leakage is present\.

## Appendix BReference Model: MMed\-Embed

Training objective, two\-stage procedure, and key hyperparameters are described in Section[5\.2](https://arxiv.org/html/2606.24200#S5.SS2)\.

#### Hard negative mining\.

Hard negatives are mined using ANCE\-style\(Xionget al\.,[2020](https://arxiv.org/html/2606.24200#bib.bib24)\)retrieval from the Stage\-1 model’s embeddings: for each query, the top\-kkmost similar non\-positive documents are selected \(k=15k\{=\}15\)\. To avoid false negatives, candidates with similarity≥\\geq95% of the positive score are filtered out \(positive\-aware filtering\)\.

#### Ablation study\.

Table[A6](https://arxiv.org/html/2606.24200#A1.T6)presents a 2×\\times2 factorial ablation isolating the contributions of topology\-aware loss \(ℒtopo\\mathcal\{L\}\_\{\\text\{topo\}\}\) and ANCE\-style hard negative mining\. All variants use identical hyperparameters: 137K training rows, 10 epochs, effective batch 1,024, lr2×10−62\{\\times\}10^\{\-6\}, starting from BGE\-M3\.

The dominant effect is fine\-tuning itself: Stage\-1 contrastive training \(A1\) improves MMed\-IR by \+0\.022 over the BGE\-M3 base\. Neitherℒtopo\\mathcal\{L\}\_\{\\text\{topo\}\}nor hard negative mining produces a statistically significant additional gain under controlled conditions: A1 vs\. A2 \(\+ℒtopo\+\\mathcal\{L\}\_\{\\text\{topo\}\}, random neg\):Δ\\DeltaMMed\-IR=−=\-0\.001; A1 vs\. A3 \(mined neg,ℒret\\mathcal\{L\}\_\{\\text\{ret\}\}only\):Δ\\DeltaMMed\-IR=−=\-0\.005\. Hard negative mining shifts the performance profile, improving Task 2 \(\+0\.005\) at the cost of Task 1 \(−\-0\.018\), but does not improve the aggregate\. This suggests that for MMed\-Bench\-IR, the primary benefit comes from domain\-specific contrastive fine\-tuning on multilingual medical data, rather than from loss function design or negative mining strategy\. The marginal gain of fine\-tuning over zero\-shot BGE\-M3 \(\+\+0\.022 MMed\-IR\) further indicates that the benchmark’s core challenge is architecture\-level multilingual alignment, not training data availability\.

Full hyperparameters are in Table[A7](https://arxiv.org/html/2606.24200#A2.T7)\.

Table A7:MMed\-Embed training hyperparameters\.

## Appendix CFull Per\-Model Results

All tables below include all ten evaluated systems across six paradigm families\.

#### Statistical framework\.

All 95% CIs reported in this paper use query\-level bootstrap resampling \(n=2,000n\{=\}2\{,\}000iterations\) applied uniformly across all tables and headline claims\. For each metric \(R@1, R@5, nDCG@10\), per\-query scores are resampled with replacement and the 2\.5th/97\.5th percentiles form the CI\. For composite metrics \(Task 1 score = mean of per\-query nDCG@10 and R@1; Task 2 score = mean of three tier R@1s; MMed\-IR = mean of three task scores\), we resample per\-query scores at each level and propagate through the composition\. Fairness gap CIs are computed by bootstrapping the max−\-min of per\-language means, capturing the full uncertainty in both the maximum and minimum languages\. All CIs are reported as±\\pmhalf\-width of the 95% interval\.

### C\.1Task 1: Per\-Language nDCG@10

Table[A8](https://arxiv.org/html/2606.24200#A3.T8)reports per\-language nDCG@10 for all ten systems\. Biomedical\-only models show near\-zero scores on Japanese, while multilingual models maintain moderate performance across all languages\.

Table A8:Task 1 per\-language nDCG@10 with bootstrap 95% CIs for all ten systems\.
### C\.2Task 2: Per\-Tier Recall@5

Table[A9](https://arxiv.org/html/2606.24200#A3.T9)extends the Recall@1 analysis in the main text to Recall@5\. The tier ordering \(Synonym, Sibling, Related\) is preserved across all models at this cutoff\.

Table A9:Task 2 Recall@5 by tier with bootstrap 95% CIs for all ten systems\.
### C\.3Fairness Gaps

Table[A10](https://arxiv.org/html/2606.24200#A3.T10)reports cross\-language fairness gaps across all three tasks\. MMed\-Embed with Reranker achieves the lowest gap on all tasks, while biomedical\-only models show the largest gaps, consistent with Finding C\.

Table A10:Cross\-language fairness gap \(max−\-min nDCG@10 across languages\) per task for all ten systems\. Lower is more equitable\.
### C\.4Task 3: Translation Sensitivity

To verify that the QA filtering does not shape benchmark outcomes, we compare model rankings under three query subsets: pass\-only \(492 queries, Latin\-script dominated\), official pass\+flag \(1,708\), and all translated \(2,040\)\. This analysis uses five zero\-shot baselines \(excluding hybrid, reranker, and MMed\-Embed\) to isolate the effect of QA filtering on ranking order\.

Table A11:Task 3 weighted nDCG@10 under different QA subsets\. Model ranking is perfectly preserved between official and all\-translated \(ρ=1\.0\\rho\{=\}1\.0\) and nearly so with pass\-only \(ρ=0\.9\\rho\{=\}0\.9\)\.

## Appendix DBenchmark Card

#### Task type\.

Information retrieval \(ranking\)\.

#### Languages\.

English, Spanish, French, Japanese, Chinese, Russian \(6 languages, 3 writing systems\)\.

#### Domains\.

Biomedical \(UMLS\-grounded concepts, BioASQ questions\)\.

#### Data sources\.

UMLS 2025AB \(NLM license required\), BioASQ 13b \(registration required\)\.

#### Annotation method\.

Automatic construction with algorithmic validation: UMLS CUI grounding for Task 1; 3\-encoder majority vote for Task 2; UMLS concept\-fidelity lexicon \+ back\-translation consistency for Task 3\.

#### Known limitations\.

See Section[Limitations](https://arxiv.org/html/2606.24200#Sx1)\. Non\-Latin\-script translation quality is bounded by UMLS coverage for those languages\.

#### Maintenance plan\.

Benchmark hosted on HuggingFace Datasets\. Human audit results and version updates documented in the dataset card\.

#### Intended use\.

Evaluation of multilingual medical retrieval systems\. Not intended as training data\.

#### Prohibited use\.

Clinical decision support without further validation\.

## Appendix EQualitative Examples

Tables[A12](https://arxiv.org/html/2606.24200#A5.T12)–[A14](https://arxiv.org/html/2606.24200#A5.T14)present representative examples from each task, drawn from actual benchmark data\. These examples illustrate the cross\-lingual and concept\-level challenges that MMed\-Bench\-IR is designed to evaluate\.

Table A12:Task 1 qualitative examples\.Two cross\-lingual QA retrieval examples spanning Latin, CJK, and Cyrillic scripts\. All positives share the same UMLS CUI as the English query\.Table A13:Task 2 qualitative examples\.Three difficulty tiers with increasing surface\-form divergence\. Encoder cosine similarities \(BGE\-M3 / E5\-large / SapBERT\) are shown for Tiers 2 and 3\. Tier 3 exhibits near\-zero lexical overlap across languages, rendering BM25 ineffective\.Table A14:Task 3 qualitative example\.EGFR mutation query translated to 6 languages via NLLB\-200 with two\-stage translation QA\. Audit status reflects translation quality gating: PASS \(both checks satisfied\), FLAG \(one check satisfied\)\. Latin\-script translations pass more reliably than CJK translations, consistent with limited UMLS coverage for non\-Latin scripts\.RoleLangTextAuditQueries \(QID: 67da0c0f\)SourceENWhat is the most common EGFR mutation in glioblastoma multiforme?—TranslationES¿Cuál es la mutación EGFR más común en el glioblastoma multiforme?PASSTranslationFRQuelle est la mutation EGFR la plus fréquente dans le glioblastome multiforme ?FLAGTranslationJA膠芽腫で最も一般的なEGFR変異は?FLAGTranslationZH什么是最常见的EGFR突变在多形质母细胞瘤?FLAGTranslationRUKakova naibolee rasprostranennaia mutatsiia EGFR v mul’tiformennoj glioblastome?FLAGGold evidence snippets \(BioASQ 13b\)EvidenceENThe epidermal growth factor receptor variant type III \(EGFRvIII\) is the most common mutation of EGFR in glioblastoma multiforme \(GBM\) and is found in approximately 25% of all GBMs\.—EvidenceENIn this study we determined whether EGFR amplification and expression of the most common mutation in GBMs \(EGFRvIII\) is retained at tumor recurrence\.—