审查用于罕见病诊断的大语言模型系统中的评估伪象与数据泄漏

# Auditing evaluation artifacts and data leakage in large language model systems for rare disease diagnosis
Source: [https://arxiv.org/html/2610.33592](https://arxiv.org/html/2610.33592)
Jian Yang1,\*, Peilin Liu2, Xiaohan Li2, Songda Li2, Yu Cao1, Yuqing Feng1, and Haomin Li1,2,\*Affiliation:\*Corresponding Author: hmli@zju\.edu\.cn \(H\. L\.\); yangjian\_bme@zju\.edu\.cn \(J\. Y\.\)Affiliation:The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, 310027, ChinaAffiliation:The Children’s Hospital, Zhejiang University School of Medicine, Hangzhou, 310052, China

Augmented large language model \(LLM\) systems have reported high benchmark accuracies in rare disease diagnosis, yet their real\-world clinical utility remains unproven\. A recent meta\-analysis flagged augmented LLM diagnostic evaluations as having a high risk of bias due to suspected data leakage; however, the precise impact of these vulnerabilities on model inference has yet to be fully quantified\. In this study, we present a comprehensive empirical audit to quantify evaluation artifacts across prevailing AI paradigms, using a repository of over 60,000 cases from diverse public and independent clinical cohorts\. By reconstructing a state\-of\-the\-art agentic framework, we demonstrate that candidate\-space contamination artificially inflates performance: enforcing strict cross\-dataset isolation resulted in a marked performance decline, with the cross\-dataset mean Recall@1 decreasing from 54\.3% \(under conventional one\-vs\-rest evaluation\) to 40\.3% \(Hedges’g=0\.94g=0\.94\)\. Similar spatial overfitting in supervised classifiers caused diagnostic accuracy to drop from an internal mean of 42\.9% to 16\.7% across external cohorts \(Cohen’sd=2\.01d=2\.01\)\. Furthermore, we quantify how LLMs infer diagnoses by exploiting retrospective therapeutic cues in clinical narratives, and how temporal misalignment in foundational knowledge bases leads to performance fluctuations\. We conclude that prevailing evaluation protocols for augmented LLM systems inadvertently reflect data leakage rather than genuine medical reasoning\. To address these systematic artifacts, we propose a rigorous evaluation framework—coupling strict data\-isolation protocols with standardized benchmarking safeguards—to ensure future diagnostic AI for rare diseases is evaluated on actual clinical robustness\.

## 1Introduction

Rare diseases \(RDs\) collectively affect an estimated 300 million people worldwide[39](https://arxiv.org/html/2610.33592#bib.bib26)\. More than 7,000 RDs have been documented, with nearly 80% having a genetic aetiology[16](https://arxiv.org/html/2610.33592#bib.bib52)\. Owing to the rarity and phenotypic heterogeneity of individual conditions[42](https://arxiv.org/html/2610.33592#bib.bib11), patients typically endure a protracted “diagnostic odyssey” characterized by years of redundant investigations and repeated misdiagnoses[11](https://arxiv.org/html/2610.33592#bib.bib6);[2](https://arxiv.org/html/2610.33592#bib.bib5);[24](https://arxiv.org/html/2610.33592#bib.bib7)\. In recent years, phenotype\-based artificial intelligence \(AI\) for diagnostic decision support has emerged as a promising solution to bridge this diagnostic gap\. The field has rapidly progressed from early knowledge\-driven symbolic reasoning[20](https://arxiv.org/html/2610.33592#bib.bib1);[1](https://arxiv.org/html/2610.33592#bib.bib2);[32](https://arxiv.org/html/2610.33592#bib.bib10);[46](https://arxiv.org/html/2610.33592#bib.bib37)to deep\-learning pattern recognition[23](https://arxiv.org/html/2610.33592#bib.bib20);[45](https://arxiv.org/html/2610.33592#bib.bib19), and more recently, to generative large language models \(LLMs\)[41](https://arxiv.org/html/2610.33592#bib.bib28), domain\-specific fine\-tuned LLMs[50](https://arxiv.org/html/2610.33592#bib.bib29);[36](https://arxiv.org/html/2610.33592#bib.bib46);[43](https://arxiv.org/html/2610.33592#bib.bib47), and agentic systems[47](https://arxiv.org/html/2610.33592#bib.bib12);[5](https://arxiv.org/html/2610.33592#bib.bib48);[6](https://arxiv.org/html/2610.33592#bib.bib49);[51](https://arxiv.org/html/2610.33592#bib.bib14)\. Accompanying this architectural evolution, recent studies have reported high diagnostic accuracies relying solely on phenotypic features or clinical narratives\. These systems frequently report top\-1 accuracies exceeding 50%[51](https://arxiv.org/html/2610.33592#bib.bib14), and sometimes nearly 70%[50](https://arxiv.org/html/2610.33592#bib.bib29), across public benchmarks and internal clinical cohorts\. Despite these advancements, such benchmark performance frequently fails to translate into real\-world clinical utility, raising questions about the actual generalizability of these systems[15](https://arxiv.org/html/2610.33592#bib.bib31)\.

A recent systematic review and meta\-analysis evaluating the diagnostic accuracy of LLM\-based systems for RD diagnosis highlighted substantial vulnerabilities within the current evidence base[26](https://arxiv.org/html/2610.33592#bib.bib50)\. While reporting seemingly high pooled diagnostic accuracies, the meta\-analysis simultaneously revealed extreme performance heterogeneity across LLM\-based systems\. Notably, nearly all of the augmented LLM systems—incorporating retrieval, agent\-based reasoning, or fine\-tuning—were rated at a high risk of bias\. This was primarily due to suspected data leakage, including training corpus overlap, retrieval database contamination, and the use of public benchmark cases that pre\-date LLM training cutoffs\. However, because these vulnerabilities were identified solely through retrospective analyses of metadata and methodological descriptions from published studies, the precise mechanistic impact of these factors on actual model inference has yet to be experimentally quantified\. It remains unclear how much of the reported state\-of\-the\-art \(SOTA\) performance stems from genuine medical reasoning capabilities versus the unintended exploitation of flawed evaluation protocols[14](https://arxiv.org/html/2610.33592#bib.bib25)\.

Motivated by this critical gap, we hypothesized that prevailing architectural paradigms in modern RD diagnosis—especially augmented LLM systems—are particularly vulnerable to specific forms of structural data leakage\. To systematically audit the claims of the prior meta\-analysis, we focused on four dimensions likely to introduce evaluation artifacts: First, current patient\-based retrieval and retrieval\-augmented generation \(RAG\) systems are highly susceptible tocandidate\-space contamination\. These systems frequently retrieve reference cases that share the same publication source, familial pedigree, or cohort origin as the query, thereby matching based on data provenance rather than true phenotypic similarity[8](https://arxiv.org/html/2610.33592#bib.bib24)\. Second, domain\-specific fine\-tuned LLMs as well as supervised classifiers trained and tested within a single clinical cohort often exploitspatial overfitting—memorizing institutional diagnosis\-phenotype profiles, disease prevalence, and recording preferences rather than learning generalizable phenotype\-disease associations[18](https://arxiv.org/html/2610.33592#bib.bib40)\. Third, when evaluating zero\-shot LLMs on clinical narratives, models can exploit retrospectivepost\-hoc cue leakage—such as targeted therapeutic interventions and downstream laboratory results recorded in discharge summaries—to infer the correct diagnosis[9](https://arxiv.org/html/2610.33592#bib.bib39)\. Finally,temporal knowledge biasarises not only when historical cases are evaluated against contemporary knowledge snapshots without chronological synchronization \(inadvertently introducing future\-data leakage\), but also when competing models rely on disparate temporal versions of the underlying knowledge, creating an asymmetric and unfair baseline comparison\.

To empirically address these risk\-of\-bias concerns, we present a comprehensive reanalysis and quantitative audit designed to systematically stress\-test phenotype\-based RD diagnostic capabilities\. Using a multi\-tiered repository of over 60,000 RD cases—spanning eight public literature/registry datasets and two independent clinical cohorts—we reconstructed and evaluated the specific environments of classical knowledge\-driven computational models, supervised classifiers, and contemporary LLM paradigms\. Specifically, we replicated the current SOTA agentic system DeepRare[51](https://arxiv.org/html/2610.33592#bib.bib14)to audit its diagnostic capability across controlled data isolation regimes while additionally validating across alternative foundation LLM backbones to confirm our findings are not model\-specific\. In parallel, we executed an extensive series of cross\-paradigm experiments to isolate and quantify all suspected sources of evaluation artifacts across diverse diagnostic architectures\. By systematically applying controlled ablation setups—enforcing cross\-institutional isolation, PMID\-level deduplication, clinical narrative redaction, and chronological synchronization—we quantitatively dissect how AI models inadvertently exploit structural biases to artificially inflate diagnostic metrics\.

Similar to historical methodological reforms in clinical medicine—such as blinding, randomization, and protocol pre\-registration[25](https://arxiv.org/html/2610.33592#bib.bib41)—we argue that diagnostic AI for RDs requires a similar structural shift in its benchmarking standards, aligning more closely with guidelines like CONSORT\-AI[21](https://arxiv.org/html/2610.33592#bib.bib42)and STARD\-AI[37](https://arxiv.org/html/2610.33592#bib.bib51)\. Synthesizing our systematic audit, we propose a rigorous evaluation framework\. By mitigating data leakage while incorporating standardized benchmarking safeguards, this framework seeks to ensure that future diagnostic systems for RDs are judged not by their susceptibility to evaluation biases, but by their genuine clinical safety, robustness, and real\-world reproducibility\.

## 2Results

### 2\.1Evaluation framework

Augmented LLM systems have demonstrated strong diagnostic performance in RDs, largely by integrating external biomedical knowledge bases \(KBs\), historical case retrieval, as well as domain\-specific fine\-tuning\. Disentangling generalizable clinical reasoning from potential evaluation artifacts is essential for establishing rigorous validation standards in this domain\. To examine these dynamics systematically, we designed a multi\-paradigm evaluation framework\. Specifically, using DeepRare—a representative agentic system—as a primary case study, we audited its reference\-case retrieval mechanisms across controlled candidate bank partitions to quantify potential data overlap\. To provide a comprehensive assessment across diagnostic paradigms, our framework further incorporates traditional supervised classifiers to assess cohort\-level spatial overfitting, and classical knowledge\-driven computational models to evaluate temporal knowledge bias\.

To operationalize this audit across varying levels of annotation density, semantic noise, and potential candidate overlap, we compiled a multi\-source evaluation corpus comprising over 60,000 RD cases\. We structured these datasets into a three\-tiered hierarchy representing diverse data origins and curation practices, enabling the systematic evaluation of artifacts and data\-leakage pathways \(Table[1](https://arxiv.org/html/2610.33592#S2.T1)\):

- •Literature\-derived cohorts: Datasets curated from published biomedical case reports, including the GA4GH Phenopacket Store[10](https://arxiv.org/html/2610.33592#bib.bib15), LIRICAL[32](https://arxiv.org/html/2610.33592#bib.bib10), PhenoDP[45](https://arxiv.org/html/2610.33592#bib.bib19), RDmaster[48](https://arxiv.org/html/2610.33592#bib.bib8), MME[30](https://arxiv.org/html/2610.33592#bib.bib16), and RareArena[3](https://arxiv.org/html/2610.33592#bib.bib13), alongside two published clinical literature cohorts \(HMS[33](https://arxiv.org/html/2610.33592#bib.bib17)and VPI[38](https://arxiv.org/html/2610.33592#bib.bib18)\)\. These cases feature expert\-curated Human Phenotype Ontology \(HPO\)[13](https://arxiv.org/html/2610.33592#bib.bib23)annotations and are structurally indexed to source publications via PubMed identifiers \(PMIDs\)\.
- •Public electronic registries: Clinical records compiled from established online disease repositories, represented by the Rare Metabolic Diseases Database \(RAMEDIS\)[40](https://arxiv.org/html/2610.33592#bib.bib9), serving as a benchmark for evaluating specialty\-specific registry data\.
- •Independent clinical cohorts: Two real\-world clinical datasets from tertiary academic medical centers: the Children’s Hospital of Zhejiang University School of Medicine \(ZJUMCH, in\-house\) cohort and Peking Union Medical College Hospital \(PUMCH\) cohort[23](https://arxiv.org/html/2610.33592#bib.bib20)\. The ZJUMCH cohort features manually curated phenotypes, whereas the PUMCH cohort employs automated phenotypic extraction, reflecting the higher annotation density and semantic redundancy typical of routine electronic health records \(EHRs\)\. As unpublished clinical records, these strictly isolated cohorts provide an essential baseline for assessing zero\-leakage generalizability and evaluating clinical narratives free from publication bias\.

Table 1:Overview of benchmark datasets\.No\.DatasetCountry/RegionCase\-level PMIDTotal CasesUnique DiseasesMean HPO TermsLiterature\-derived cohorts\-RareArenaMulti\-country✓\\checkmark49,7604,59711\.5IPhenopacketMulti\-country✓\\checkmark8,1015169\.3IILIRICALMulti\-country✓\\checkmark38426211\.2IIIPhenoDPMulti\-country✓\\checkmark1307214\.2IVRDmasterMulti\-country✓\\checkmark2016210\.0VMMEMulti\-country✓\\checkmark462011\.9VIHMSGermany662910\.9VIIVPIAustralia47484\.1Public electronic registriesVIIIRAMEDISGermany \(mainly\)802927\.4Independent clinical cohortsIXPUMCHChina1,06340531\.2XZJUMCHChina \(in\-house\)8961389\.4
Note: RareArena is for development use only\.

Using this multi\-tiered framework, our systematic investigation examines four primary dimensions of evaluation artifacts: \(i\) candidate\-space contamination in similar patient retrieval, \(ii\) cohort\-level spatial overfitting, \(iii\) post\-hoc textual leakage in clinical narratives, and \(iv\) temporal misalignment in foundational KBs\. As detailed in the following sections, our findings illustrate how agentic systems and other diagnostic paradigms can inadvertently exploit these structural vulnerabilities, underscoring the necessity for rigorous evaluation standards before AI systems can be reliably translated into clinical practice\.

### 2\.2Baseline auditing on the state\-of\-the\-art agentic system

To assess the clinical robustness of current retrieval\-augmented diagnostic systems, we used DeepRare, a recently proposed SOTA agentic system, as our representative baseline\. DeepRare incorporates typical agentic workflows, including external diagnostic tool integration, similar\-patient retrieval, web\-based knowledge search, and multi\-turn diagnostic reflection\. We fully replicated the system based on its code repository, preserving all core analytical functions\. To ensure our reanalysis reflects the full capabilities of DeepRare and avoids reliance on outdated LLMs, we first operationalized the DeepRare architecture using the recent foundation LLM, DeepSeek\-V4\-Flash\. Because the exact reference\-patient corpus utilized in the original DeepRare study is not fully publicly accessible, we populated the system’s retrieval database using our curated cohorts, specifically focusing on those \(e\.g\., LIRICAL, MME, HMS, and RAMEDIS\) that overlap with the original publication’s evaluation sets\. To this end, we implemented an information content\-based phenotype\-similarity algorithm to drive the patient retrieval module, which we hypothesized to be the primary vector for evaluation artifacts and data leakage\.

Our reanalysis evaluated the system’s diagnostic stability under varying data isolation conditions\. In standard evaluations of such agentic systems, including DeepRare, a one\-versus\-rest \(OVR\) strategy \(or random/temporal splitting\) is typically employed, where only the specific deduplicated test case is withheld from the retrieval corpus\. To rigorously test for structural data leakage, we compared the OVR baseline against a strict “Full Test” strategy\. Under the full\-test condition, we enforced complete spatial and literature\-level isolation: the entire dataset to which the target case belongs was removed from the retrieval corpus, and strict deduplication was applied to ensure no reference cases originating from the same literature source \(identified via PMIDs\) could be retrieved\. This strict isolation genuinely benchmarks the system’s diagnostic generalizability when encountering novel cases/cohorts in real\-world clinical practice\.

Figure 1:Structural data leakage and generalization collapse across rare disease diagnostic AI paradigms\.\(a\)Diagnostic performance of the agentic system DeepRare across six evaluation cohorts under one\-vs\-rest \(OVR\), full\-test, and zero\-shot LLM configurations\.\(b\)Diagnostic Recall@1 of the standalone patient retrieval module across 10 cohorts under candidate bank partitioning schemes, transitioning from OVR throughnn\-fold variants to full\-test\.\(c\)Diagnostic Recall@1 \(Mean±\\pm95% CI\) across four literature\-derived cohorts evaluated under the full\-test setting with and without cross\-dataset PMID\-level deduplication\.\(d\)Diagnostic Recall@1 \(Mean±\\pm95% CI\) of the minimalist RAG prototype across ten datasets evaluated under OVR, full\-test, and standalone LLM without RAG configurations\.\(e–g\)Pairwise cross\-dataset diagnostic Recall@1 for \(e\) linear \(SGDClassifier\), \(f\) non\-linear \(XGBoost\), and \(g\) deep learning \(multilayer perceptron\) models \(datasets I–X correspond to Table[1](https://arxiv.org/html/2610.33592#S2.T1); the internal numbers indicate Recall@1\)\.\(h\)Summary comparison of internal versus external validation accuracy \(Mean±\\pmSD\) across supervised linear, non\-linear, and deep learning classifiers\. Note: \*P<0\.05P<0\.05, \*\*P<0\.01P<0\.01, \*\*\*P<0\.001P<0\.001\.The empirical results reveal a marked decline in generalization when strict isolation is enforced \(Fig\.[1](https://arxiv.org/html/2610.33592#S2.F1)a\)\. Under conventional OVR conditions, our replicated agent achieved benchmark metrics consistent with those reported in the original publication \(e\.g\., Recall@1\>70%\>70\\%on MME and RAMEDIS, and≥50%\\geq 50\\%on LIRICAL\), confirming the successful deployment and functional fidelity of the underlying reasoning pipeline\. However, when subjected to the full\-test isolation, diagnostic performance declined across cohorts: the cross\-dataset mean Recall@1 decreased from54\.3%±18\.2%54\.3\\%\\pm 18\.2\\%under OVR to40\.3%±11\.1%40\.3\\%\\pm 11\.1\\%under the full\-test setting\. At the individual dataset level, Recall@1 on MME dropped from 75% down to 42%, while the clinical cohorts from PUMCH and ZJUMCH experienced reductions from 66% to 42%, and from 39% to 30%, respectively \(allP<0\.05P<0\.05, paired McNemar’s test\)\. To quantify the magnitude of this artifact beyond individual cohort variations, we calculated the standardized effect sizes using Hedges’ggacross all six evaluation cohorts\. Although the agentic architecture yielded genuine gains over the zero\-shot base LLM under full\-test isolation \(Hedges’g=\[0\.77,0\.81,1\.01\]g=\[0\.77,0\.81,1\.01\]for Recall@\[1,3,5\]\[1,3,5\], respectively\), the artificial performance inflation induced by transitioning from strict full\-test isolation to the conventional OVR setup was substantially larger \(Hedges’g=\[0\.94,1\.18,1\.21\]g=\[0\.94,1\.18,1\.21\]for Recall@\[1,3,5\]\[1,3,5\]\)\. This comparative effect size analysis yields a critical insight: while the multi\-step agentic framework does provide a measurable diagnostic advantage over raw foundation LLMs under full\-test isolation, the artificial performance inflation imparted purely by candidate\-space leakage \(OVR artifact\) is consistently larger than the system’s actual reasoning contribution\.

To ensure these findings were not specific to a single foundation LLM, we evaluated the DeepRare agentic architecture using diverse foundation LLMs, including Gemini\-2\.5\-Flash and GPT\-4o\-mini \(Supplementary Fig\. S1\)\. While the absolute baseline accuracies naturally fluctuated depending on the intrinsic medical reasoning capabilities of the respective foundation LLMs, the relative performance degradation remained consistent\. Across all tested LLM backbones, the transition from OVR to full\-test isolation resulted in a comparable degradation in recall metrics\. This cross\-LLM consistency confirms that the observed performance disparity is an architectural artifact—stemming from the retrieval module’s unintended access to spatially or literarily proximate data—rather than a limitation of any single foundation LLM\.

### 2\.3Candidate\-space contamination and spatial overfitting across AI paradigms

To elucidate the mechanistic drivers behind the observed performance degradation in the agentic system, we conducted a detailed audit of its core reference\-patient retrieval module\. Case\-based reasoning systems inherently prioritize diagnostic candidates by matching query cases against historical clinical precedents\. We hypothesized that this mechanism is highly vulnerable to structural data contamination, acting as an algorithmic shortcut when homologous cases are retained in the reference bank\. To quantify this, we permuted the candidate bank partitioning across datasets, transitioning from the OVR setup to increasingly restrictivenn\-fold variants \(where only1/n1/nof the cohort was retained for retrieval\), culminating in the full\-test setting\. Across all configurations, we ensured that the retrieval bank contained no reference cases originating from the same source publication as the query case\. The results demonstrated a “few\-patient surge effect” \(Fig\.[1](https://arxiv.org/html/2610.33592#S2.F1)b\)\. Under the OVR configuration, the standalone retrieval module achieved high Recall@1 across the majority of datasets \(48\.5%±24\.3%48\.5\\%\\pm 24\.3\\%\)\. However, as the fraction of the cohort available for retrieval diminished—withnnscaling from 2 to 10 in thenn\-fold setups—diagnostic performance exhibited a consistent monotonic decline\. Notably, even under the 10\-fold setting—where only 10% of the target cohort was accessible within the reference bank—diagnostic accuracy remained artificially elevated relative to the full\-test setting \(Hedges’g=0\.85g=0\.85\)\. Only under complete full\-test isolation did the diagnostic Recall@1 decrease to its baseline of15\.2%±7\.0%15\.2\\%\\pm 7\.0\\%\. This indicates that even minimal exposure to same\-source cohort data allows patient retrieval modules to exploit center\-specific phenotype recording preferences, local documentation habits, or extraction noise, rather than performing generalized clinical matching\.

Furthermore, our findings demonstrate that even under the full\-test isolation, public benchmarks derived from biomedical literature \(e\.g\., Phenopacket, LIRICAL, PhenoDP, and RDmaster\) remain vulnerable to an overlooked source of literature leakage\. To evaluate the consequence of omitting literature\-level controls, we compared the full\-test configuration with and without cross\-dataset PMID\-level deduplication \(Fig\.[1](https://arxiv.org/html/2610.33592#S2.F1)c\)\. Across all four literature\-derived cohorts, diagnostic recall prior to deduplication was significantly higher than after strict filtering \(P<0\.001P<0\.001, paired one\-sided bootstrap test\), exhibiting a large overall effect size \(Hedges’g=1\.22g=1\.22\)\. Specifically, excluding reference cases sharing the query patient’s exact PMID caused diagnostic recall on LIRICAL to drop from 71% to 18%\. This indicates that without PMID\-level deduplication on literature\-derived datasets, benchmark performance is largely driven by data leakage from matching cases originating from the same publication or even the exact same patient\. Moreover, this vulnerability poses a potential risk for agentic frameworks equipped with web or literature search capabilities \(such as DeepRare\), as they can directly retrieve the query patient’s original source paper when tested on literature cohorts—causing reported performance to reflect search and indexing capability rather than clinical deductive reasoning\.

To determine whether this candidate\-space contamination is specific to complex agentic systems or present in the broader retrieval\-augmented paradigm, we designed a minimalist RAG LLM prototype built upon GPT\-4o\. This prototype used the exact same patient\-similarity retrieval module but removed the complex agentic workflows, substituting them with a few\-shot similar\-case inference framework\. Consistent with DeepRare, this baseline RAG LLM exhibited a comparable performance degradation when transitioning from the OVR setup to the full\-test setting \(Fig\.[1](https://arxiv.org/html/2610.33592#S2.F1)d\)\. Notably, under the OVR strategy, this minimalist RAG prototype achieved diagnostic performance closely comparable to that of the complex DeepRare agent\. This indicates that the artificial performance inflation induced by OVR leakage confounds the evaluation metrics, thereby obscuring the actual contributions of advanced agentic architectures\.

The evaluation artifacts induced by data leakage in retrieval\-augmented and agentic systems fundamentally parallel the classical overfitting phenomenon in supervised learning and LLM fine\-tuning\. To examine the universality of these validation vulnerabilities, we extended our audit framework to classical supervised machine learning and deep learning classifiers \(e\.g\., SGDClassifier, XGBoost, and the multilayer perceptron \(MLP\)\)\. Using a cross\-center validation matrix, we observed substantial spatial overfitting \(Fig\.[1](https://arxiv.org/html/2610.33592#S2.F1)e–g\)\. When trained and evaluated within the same dataset, supervised models demonstrated artificially inflated performance by exploiting center\-specific diagnosis\-phenotype associations\. However, when evaluated across external cohorts, diagnostic generalizability decreased substantially \(Fig\.[1](https://arxiv.org/html/2610.33592#S2.F1)h\): mean accuracy fell from 49\.5% to 20\.0% for the linear SGDClassifier \(Cohen’sd=2\.21d=2\.21\), from 35\.9% to 13\.2% for the non\-linear XGBoost \(Cohen’sd=1\.94d=1\.94\), and from 43\.4% to 16\.9% for the deep learning MLP \(Cohen’sd=2\.04d=2\.04\)\. Aggregated across all three classifiers, the overall mean accuracy declined from 42\.9% internally to 16\.7% externally \(pooled Cohen’sd=2\.01d=2\.01\)\. This out\-of\-distribution \(OOD\) performance degradation is equally evident in domain\-specific fine\-tuned LLMs\. For example, in MINT[43](https://arxiv.org/html/2610.33592#bib.bib47), diagnostic accuracy dropped from 47\.6% on diseases seen during fine\-tuning to only 7\.0% on completely unseen RD categories\. Similarly, RareSeek reported a 29\.2\-percentage\-point reduction in diagnostic accuracy when evaluated on the independent external benchmark RareBench[50](https://arxiv.org/html/2610.33592#bib.bib29)\(39\.2%\) compared to its internal validation set \(68\.4%\)[50](https://arxiv.org/html/2610.33592#bib.bib29)\. This widespread OOD performance degradation demonstrates that evaluating diagnostic models within their training distributions is an inadequate evaluation protocol, as in\-distribution testing merely rewards localized data memorization while obscuring true real\-world failure modes\.

### 2\.4Post\-hoc cue leakage in clinical narratives

In addition to candidate\-space contamination, post\-hoc cue leakage within diagnostic inputs constitutes another major source of evaluation artifacts\. Existing diagnostic architectures generally adopt one of two paradigms: structured HPO\-based reasoning \(utilized by traditional tools like PubCaseFinder[12](https://arxiv.org/html/2610.33592#bib.bib53)and Phenomizer[20](https://arxiv.org/html/2610.33592#bib.bib1), or agentic pipelines like DeepRare that parse text into HPO terms\) versus end\-to\-end reasoning over clinical narratives\. Whereas HPO standardization inherently removes extraneous text, direct inference on clinical narratives remains vulnerable to post\-hoc textual leakage, allowing models to exploit retrospective diagnostic hints rather than performing genuine phenotypic deduction\. To quantify this leakage, we conducted a field\-level ablation study across different sections of inpatient clinical records\. We curated 796 complete inpatient EHRs with confirmed ORPHA diagnoses from the in\-house ZJUMCH cohort\. These structured EHRs comprise multiple standardized clinical fields, which we organized into four chronological stages: chief complaint \(CC, encompassing demographic information and history of present illness\), physical examination \(PE\), ancillary examinations \(AE, containing specialized imaging and laboratory findings\), and clinical treatment \(CT\)\. To eliminate direct label leakage, admission and discharge diagnosis fields were strictly excluded\. Furthermore, any explicit mentions of the ground\-truth diagnosis names within the remaining fields were programmatically redacted using regular expression matching\. In parallel, standardized HPO terms were extracted from the records to serve as an isolated phenotypic representation free from narrative context\. Finally, we supplied these segmented EHR sections \(added chronologically\) as well as the structured HPO terms to a zero\-shot prompted GPT\-4o model for RD differential diagnosis\.

As shown in Fig\.[2](https://arxiv.org/html/2610.33592#S2.F2)a, while adding phenotypic descriptions steadily improved diagnostic performance \(\+PE vs\. CC:Δ​Recall@1=\+10\.18%\\Delta\\text\{Recall@1\}=\+10\.18\\%,paired McNemar’s Odds Ratio \(OR\)=6\.06\\text\{paired McNemar's Odds Ratio \(OR\)\}=6\.06; \+AE vs\. \+PE:Δ​Recall@1=\+7\.79%\\Delta\\text\{Recall@1\}=\+7\.79\\%,OR=3\.14\\text\{OR\}=3\.14\), the largest performance increase occurred when clinical treatment records were appended \(\+CT vs\. \+AE:Δ​Recall@1=\+15\.33%\\Delta\\text\{Recall@1\}=\+15\.33\\%,OR=6\.81\\text\{OR\}=6\.81\)\. From a clinical workflow perspective, diagnosing a patient using a discharge summary that includes treatment information is inconsistent with clinical practice\. Real\-world diagnostic reasoning is a prospective process that occurs before targeted treatments are administered\. The CT sections frequently contain retrospective post\-hoc cues, such as highly specific targeted therapies \(e\.g\., enzyme replacement therapies\) or diagnostic drug\-monitoring laboratory results\. Similarly, AE sections sometimes include disease\-specific ancillary findings—such as characteristic imaging features, histopathological markers, or metabolic panels—that act as diagnostic cues retrospectively\. Fig\.[2](https://arxiv.org/html/2610.33592#S2.F2)b further isolates this artifact: appending raw contextual text \(\+CC, including demographics and medical history\) to structured HPO terms provides situational context that yields a substantial improvement in diagnostic recall \(\+CC vs\. HPO:Δ​Recall@1=\+22\.24%\\Delta\\text\{Recall@1\}=\+22\.24\\%,OR=12\.80\\text\{OR\}=12\.80\)\. In contrast, adding physical \(\+PE\) and ancillary \(\+AE\) examinations produces marginal changes \(\+PE vs\. \+CC:Δ​Recall@1=−0\.25%\\Delta\\text\{Recall@1\}=\-0\.25\\%,OR=0\.96\\text\{OR\}=0\.96; \+AE vs\. \+PE:Δ​Recall@1=\+4\.40%\\Delta\\text\{Recall@1\}=\+4\.40\\%,OR=2\.59\\text\{OR\}=2\.59\), indicating they offer little additional phenotypic value beyond the extracted HPO terms\. However, appending clinical treatment \(\+CT\) information—which similarly lacks new phenotypic data—still generates a notable performance gain \(\+CT vs\. \+AE:Δ​Recall@1=\+7\.54%\\Delta\\text\{Recall@1\}=\+7\.54\\%,OR=2\.54\\text\{OR\}=2\.54\)\. This contrast confirms that the LLM derives substantial diagnostic signals from situational and therapeutic cues rather than genuine phenotypic data enrichment\.

Figure 2:Post\-hoc cue leakage in clinical narratives\.\(a\)Stepwise performance inflation \(Mean±\\pm95% CI\) of zero\-shot LLM diagnostics as retrospective clinical narrative segments—chief complaint \(CC\), physical examination \(PE\), ancillary examinations \(AE\), and clinical treatment \(CT\)—are sequentially appended\.\(b\)Diagnostic recall \(Mean±\\pm95% CI\) with sequential addition of clinical text segments to extracted HPO terms\.These findings highlight a critical dichotomy in evaluating AI diagnostic tools\. While systems operating strictly on structured HPO representations are structurally insulated from post\-hoc cue leakage, systems evaluated on raw, unstructured clinical text are particularly susceptible\. For studies assessing clinical narratives to be clinically valid, evaluation protocols must mandate the redaction of therapeutic interventions, discharge outcomes, and post\-diagnostic administrative markers, ensuring the systems are tested on genuinely prospective clinical presentations\.

### 2\.5Temporal misalignment in foundational knowledge bases

Most augmented LLM systems fundamentally rely on foundational biomedical KBs \(e\.g\., ORPHA[44](https://arxiv.org/html/2610.33592#bib.bib3)and OMIM[17](https://arxiv.org/html/2610.33592#bib.bib4)\)—whether by directly querying KBs via RAG pipelines, indirectly utilizing them through external recommendation tools, or fine\-tuning models on knowledge\-base data\. However, unlike traditional AI benchmarks that emphasize rigorous KB consistency across compared baselines, evaluations of contemporary LLM\-based systems often neglect the synchronization of underlying KB versions\. A prevailing assumption in diagnostic AI is that reference KB versioning fundamentally shapes benchmark metrics, with newer snapshots presumed to yield superior accuracy\. Directly quantifying this temporal effect within complex augmented LLMs is challenging in practice, and is further methodologically confounded by their opaque internal parametric memory and stochastic inference\. To cleanly isolate the impact of KB evolution on diagnostic performance, we deployed purely knowledge\-driven architectures as objective analytical baselines for a longitudinal temporal audit\. Specifically, we evaluated two representative knowledge\-driven computational architectures—a likelihood ratio \(LR\) model[32](https://arxiv.org/html/2610.33592#bib.bib10)and a probabilistic graphical \(PG\) model[1](https://arxiv.org/html/2610.33592#bib.bib2)—across six annual snapshots of ORPHA and OMIM spanning 2021 to 2026\. The diagnostic reasoning of these deterministic models is transparently derived from the underlying ontology and computational logic, ensuring that any observed performance changes reflect the true dynamics of the knowledge graph\.

Figure 3:Longitudinal assessment of clinical diagnostic performance and knowledge base evolution\.\(a\)Recall@1 performance trajectories \(Mean±\\pm95% CI\) of the likelihood ratio \(LR\) and probabilistic graphical \(PG\) models across six annual snapshots \(2021–2026\) of ORPHA on diverse public and clinical datasets\.\(b\)Corresponding temporal evaluation using OMIM\.\(c\)Temporal evolution of disease entities in OMIM and ORPHA\.\(d\)Temporal evolution of disease\-phenotype annotations in OMIM and ORPHA\.Evaluating diagnostic baselines across longitudinal snapshots of foundational KBs revealed substantial performance variations tied directly to database versioning \(Fig\.[3](https://arxiv.org/html/2610.33592#S2.F3)\)\. On literature\-derived datasets, models exhibited marked step\-like shifts corresponding to major curator re\-annotation events; for instance, the LR model’s Recall@1 on Phenopacket rose from 17\.2%–22\.6% \(2021–2024\) to 53\.6% in the 2025 OMIM snapshot, with a parallel increase from 33\.7% to 51\.9% observed on LIRICAL \(Fig\.[3](https://arxiv.org/html/2610.33592#S2.F3)b\)\. This observation demonstrates that a model’s apparent diagnostic capability on literature\-derived datasets can increase substantially solely due to KB evolution, independent of algorithmic advancements\. Conversely, on real\-world clinical cohorts, KB evolution led to non\-monotonic, bidirectional variations—such as LR Recall@1 on PUMCH fluctuating from 17\.0% in 2021 to 6\.9% in 2026—as progressive disease refinement in KBs can introduce semantic mismatches when matched against redundant clinical documentation\. Moreover, different computational models exhibited divergent trajectories on the identical KB snapshots—for instance, in the OMIM benchmark on the HMS dataset, LR and PG models demonstrated contradictory trends—indicating that structural ontology updates impact distinct reasoning architectures unequally\.

These findings challenge the assumption that newer KB versions consistently enhance diagnostic accuracy\. Rather, our results demonstrate that version disparities in the underlying KB induce substantial performance fluctuations for the exact same model\. When comparative benchmarks rely on disparate or unaligned KB snapshots, observed performance gaps may merely reflect ontology version discrepancies rather than genuine model superiority\. Consequently, standardizing a unified KB version across all benchmarked baselines is an essential requirement for fair, controlled, and rigorous diagnostic AI evaluation\.

### 2\.6Auditing data leakage risk in rare disease diagnostic AI

A recent meta\-analysis evaluating LLM\-based systems in RD diagnosis highlighted widespread risks of bias, identifying potential data leakage and over\-optimistic performance in previous publications[26](https://arxiv.org/html/2610.33592#bib.bib50)\. However, this prior study primarily inferred these vulnerabilities from literature without conducting direct experimental re\-analyses or quantitatively characterizing their specific impacts\. Our systematic audit provides empirical evidence to substantiate these concerns\. By isolating each potential evaluation shortcut through controlled ablation experiments, we demonstrate that benchmark inflations are not random variations, but measurable distortions driven by specific structural flaws: candidate\-space and literature contamination \(CC\), spatial overfitting \(SO\), post\-hoc cue leakage \(PCL\), and temporal knowledge bias \(TKB\)\. Applying these audit criteria, we systematically reassessed and re\-annotated the majority of recent LLM\-based systems alongside historical landmark studies, based on the methodological details reported in their original publications\. To ensure annotation rigor, the initial risk assessment was conducted by author J\.Y\. and subsequently verified by an independent co\-author, with any discrepancies resolved through consensus\. The explicit grading rubric, along with the extracted methodological details and specific page numbers from the source publications supporting each judgment, are provided in Supplementary Table S1\.

Table 2:Evaluation artifact audit across rare disease diagnostic AI systems\.System \(year\)AI ParadigmDiagnostic FeaturesScopeCCSOPCLTKBOverall RiskDeepRare\(2026\)[51](https://arxiv.org/html/2610.33592#bib.bib14)AgentHPO Terms\#& WESP\+––\+\+RareAgents\(2026\)[6](https://arxiv.org/html/2610.33592#bib.bib49)AgentHPO Terms & SymptomsP\+––\+\+MINT\(2026\)[43](https://arxiv.org/html/2610.33592#bib.bib47)FT LLMHPO Terms & DemographicsP–\+–\+\+RareSeek\(2025\)[50](https://arxiv.org/html/2610.33592#bib.bib29)RAG & FT LLMClinical TextP–\+\+\+\+Deep\-DxSearch\(2025\)[53](https://arxiv.org/html/2610.33592#bib.bib44)Agent & RLClinical TextP\+\+–\+\+MEDDxAgent\(2025\)[34](https://arxiv.org/html/2610.33592#bib.bib45)AgentHPO TermsR\+––\+\+RareScale\(2025\)[36](https://arxiv.org/html/2610.33592#bib.bib46)RAG & FT LLMClinical TextR\+\+–?\+MAC\(2025\)[5](https://arxiv.org/html/2610.33592#bib.bib48)AgentClinical TextP––\+–\+RDguru\(2025\)[47](https://arxiv.org/html/2610.33592#bib.bib12)AgentHPO Terms\#P–––?–RareBench\(2024\)[7](https://arxiv.org/html/2610.33592#bib.bib27)RAG LLMHPO Terms\#& Clinical TextP\+–\+?\+PhenoBrain\(2025\)[23](https://arxiv.org/html/2610.33592#bib.bib20)EnsembleHPO Terms\#P–––––PhenoDP\(2025\)[45](https://arxiv.org/html/2610.33592#bib.bib19)EnsembleHPO TermsP–––––PheLR\(2023\)[46](https://arxiv.org/html/2610.33592#bib.bib37)Ontology\-drivenHPO Terms & WESP–––––LIRICAL\(2020\)[32](https://arxiv.org/html/2610.33592#bib.bib10)Ontology\-drivenHPO Terms & WESP–––––BOQA\(2012\)[1](https://arxiv.org/html/2610.33592#bib.bib2)Ontology\-drivenHPO TermsP–––––Phenomizer\(2009\)[20](https://arxiv.org/html/2610.33592#bib.bib1)Ontology\-drivenHPO TermsP–––––
AI Paradigm: Agent==agentic system; FT==domain\-specific fine\-tuning; RAG==retrieval\-augmented generation; RL==reinforcement learning; Ensemble==multi\-model ensemble; Ontology\-driven==ontology\-driven computation\. Diagnostic Features: HPO Terms\#==extract HPO terms from text; WES==whole\-exome sequencing\. Scope: P==pan\-rare disease spectrum; R==restricted disease subset\. CC/SO/PCL/TKB/Overall Risk: \+==High risk; –==Low risk; ?==Unclear\.

Our audit reveals distinct, paradigm\-specific patterns of evaluation risks across the surveyed systems \(Table[2](https://arxiv.org/html/2610.33592#S2.T2)\)\. First, candidate\-space/literature contamination is primarily present in architectures incorporating case\-based retrieval modules \(e\.g\., DeepRare, RareAgents, Deep\-DxSearch, MEDDxAgent, RareScale, and RareBench\), where reference databases retain cases from the same source cohort or publication without PMID\-level deduplication\. Second, spatial overfitting predominantly affects fine\-tuned LLMs \(e\.g\., RareSeek, MINT, RareScale\) evaluated on intra\-dataset splits, resulting in degraded performance when tested on external clinical cohorts or unseen disease categories\. Third, post\-hoc cue leakage is observed in models that directly process clinical narratives \(e\.g\., RareSeek, MAC, RareBench\), as retrospective medical records frequently contain downstream management and therapeutic details—a phenomenon corroborated in their original studies\. In contrast, frameworks restricted to standardized HPO terms structurally avoid narrative post\-hoc cue leakage\. Fourth, temporal knowledge bias arises when systems utilizing updated KBs or live search are benchmarked against classical computational baselines without synchronizing reference database versions\. The systems annotated with uncertain temporal knowledge bias largely reflect differing pre\-training knowledge cutoffs across foundation LLMs\. Overall, most recent LLM\-based diagnostic studies exhibit risks in one or more evaluation dimensions\. In contrast, classical ontology\-driven methods \(e\.g\., Phenomizer, BOQA, LIRICAL, PheLR\) maintain explicit KB version control and operate on structured HPO inputs without supervised parameter tuning or patient retrieval, thereby avoiding data leakage risks\. Furthermore, ensemble models integrating ontology computing with deep learning or representation learning \(e\.g\., PhenoBrain, PhenoDP\) demonstrate that deep learning pipelines can avoid these evaluation artifacts when appropriate data isolation, feature standardization, and knowledge version control protocols are enforced\.

### 2\.7Guidelines for trustworthy evaluation of rare disease diagnostic AI

Based on our systematic audit and the risk\-of\-bias landscape of representative AI systems, we identified critical vulnerabilities that introduce evaluation artifacts into the assessment of RD diagnostic AI\. To resolve these structural vulnerabilities, we propose a comprehensive benchmarking framework \(Table[3](https://arxiv.org/html/2610.33592#S2.T3)\) that integrates our auditing results with broader consensus in translational AI evaluation[27](https://arxiv.org/html/2610.33592#bib.bib34)\. This framework establishes rigorous protocols across several dimensions: candidate\-space isolation \(institutional dataset segregation and PMID\-level deduplication\), semantically decoupled input \(redaction of therapeutic and post\-hoc clinical indicators from unstructured narratives\), chronological synchronization \(comparative models are grounded in the same version of the KBs\), cross\-center spatial validation \(using independent external validation cohorts\), as well as established safeguards and general guidelines like protocol pre\-registration, multi\-dataset auditing, and candidate scope harmonization\.

Furthermore, as diagnostic AI systems transition from retrospective benchmarks toward regulatory evaluation, evaluation methodologies need to align with clinical trial standards like CONSORT\-AI[21](https://arxiv.org/html/2610.33592#bib.bib42)and STARD\-AI[37](https://arxiv.org/html/2610.33592#bib.bib51)\. Future clinical validations should prioritize prospective cohort designs\. In such real\-world workflows, AI systems process clinical data chronologically, relying on initial chief complaints, physical examinations, and baseline laboratory panels before definitive targeted therapies are initiated or recorded\. Implementing this rigorous evaluation framework will provide a realistic appraisal of diagnostic performance, ensuring that digital health tools deliver verified clinical utility rather than exploiting evaluation artifacts\.

Table 3:Proposed evaluation guidelines for general rare disease diagnostic AI\.Evaluation ArtifactManifestationProposed Evaluation GuidelineTemporal Knowledge BiasInconsistencies in the KB versions that models rely on lead to substantial performance fluctuations, with diagnostic accuracy shifting based on version differences\.Chronological Synchronization:Enforce a single, chronologically aligned KB version across all models to prevent KB version\-driven evaluation artifacts\.Spatial / Institutional OverfittingModels memorize institutional documentation habits, cohort\-specific diagnosis\-phenotype associations and disease prevalence rather than learning generalizable phenotype\-disease profiles\.Cross\-institutional Validation:Replace standard intra\-dataset splits with mandatory out\-of\-distribution validation using independent, institutionally isolated clinical cohorts\.Candidate\-space & Literature ContaminationRetrieval systems exploit structural data leakage by matching query cases to homologous records originating from the exact same publication, family pedigree, or institution\.Source\-level Deduplication:Exclude structurally and institutionally related cases from the reference bank, enforcing PMID\-level cross\-deduplication for literature\-derived cohorts\.Post\-Hoc Cue LeakageRetrospective clinical narratives embed downstream therapeutic interventions and targeted lab results, allowing LLMs to infer diagnoses retrospectively\.Semantic Decoupling & Sanitization:Benchmark primarily using standardized phenotypic profiles such as HPO terms\. When evaluating raw clinical texts, systematically redact post\-hoc therapeutic and administrative cues\.Post\-hoc Hypothesis ShiftingIterative prompt engineering and hyperparameter tuning directly on the evaluation set cause benchmark overfitting and compromise generalizability\.Blinded Evaluation & Pre\-registration:Lock model weights, prompt templates, and evaluation pipelines prior to testing\. Utilize independent validation sets to prevent test\-set optimization\.Dataset HeterogeneitySubstantial variations in diagnostic difficulty and disease spectrum across case datasets make single\-source evaluations unrepresentative\.Multi\-tiered Cross\-dataset Auditing:Diagnostic systems must be benchmarked across multi\-tiered datasets to reliably verify generalization across diverse data origins and clinical complexities\.Diagnostic Scope InconsistencyInconsistencies in candidate diagnostic spaces across competing models introduce substantial evaluation bias\.Scope Harmonization & Full\-spectrum Coverage:Benchmarks must align the diagnostic candidate space across all evaluated models to ensure fair comparisons\. A general RD diagnostic AI should cover the broadest spectrum of RDs\.
Abbreviations: RD, rare disease; AI, artificial intelligence; KB, knowledge base; EHR, electronic health record; PMID, PubMed identifier; LLM, large language model; HPO, Human Phenotype Ontology\.

## 3Discussion

Despite advancements in phenotype\-based RD diagnostics, high performance metrics reported in recent benchmarks often fail to translate into real\-world clinical utility[26](https://arxiv.org/html/2610.33592#bib.bib50)\. Previously, the extent to which these results were driven by genuine medical reasoning versus evaluation artifacts remained unclear\. In this study, we demonstrate that current AI evaluation protocols are confounded by systematic data leakage and validation biases\. Through a multi\-paradigm audit based on over 60,000 RD cases, we quantified how distinct algorithmic shortcuts—specifically, temporal knowledge misalignment, spatial overfitting, candidate\-space contamination, and post\-hoc textual cues—artificially inflate diagnostic accuracy\. These empirical findings highlight the vulnerabilities embedded in current diagnostic AI benchmarking and provide a clinically aligned framework to ensure the reproducibility and generalizability of future diagnostic systems\.

The confluence of big data and LLMs exacerbates leakage risks The increasing prevalence of these evaluation artifacts is driven by shifts in how RD data is curated and utilized\. Historically, the clinical scarcity of documented cases precluded the development of robust supervised classifiers as well as case\-based retrieval systems\. However, the recent emergence of large\-scale, aggregated patient repositories \(e\.g\., GA4GH Phenopacket Store[10](https://arxiv.org/html/2610.33592#bib.bib15)and RareArena[3](https://arxiv.org/html/2610.33592#bib.bib13)\) has altered this landscape, making the training of supervised classifiers both feasible and seemingly robust\. This data expansion has been accelerated by LLMs, which have lowered the computational and financial costs of extracting, standardizing, and curating heterogeneous patient profiles from unstructured medical literature and EHRs[52](https://arxiv.org/html/2610.33592#bib.bib35);[3](https://arxiv.org/html/2610.33592#bib.bib13)\. Furthermore, these generative tools facilitate the continuous updating of clinical KBs[22](https://arxiv.org/html/2610.33592#bib.bib36)\. While this increased data availability represents a significant advance for RD informatics, the associated risks of data contamination have not been sufficiently recognized by the broader research community[8](https://arxiv.org/html/2610.33592#bib.bib24)\. When researchers apply conventional computer science evaluation protocols—such as random intra\-dataset splitting—to these newly available large\-scale cohorts, they inadvertently introduce the spatial overfitting and candidate\-space contamination identified in our audit\. The large volume of accumulated patient records, combined with frequent knowledge updates, creates conditions where temporal and structural leakages occur without being readily detected\. Consequently, as the development of diagnostic models becomes more accessible, the risk of unknowingly validating them on contaminated, overlapping, or temporally misaligned data increases, necessitating a systematic recalibration of benchmarking standards\.

Re\-evaluating performance claims in contemporary LLM paradigms Performing differential diagnosis across thousands of RDs is computationally complex\. Historically, knowledge\-driven computational models maintained transparency by explicitly documenting their logic and underlying KB versions, enabling reproducible comparisons[20](https://arxiv.org/html/2610.33592#bib.bib1);[1](https://arxiv.org/html/2610.33592#bib.bib2);[32](https://arxiv.org/html/2610.33592#bib.bib10);[46](https://arxiv.org/html/2610.33592#bib.bib37)\. The recent shift toward LLMs has introduced augmented LLM systems, such as the agentic system DeepRare, which reported high capabilities \(e\.g\., achieving an HPO\-based Recall@1 of 57%, outperforming baselines like PhenoBrain’s ¡30%\)\. However, our empirical reanalysis prompts a critical re\-evaluation of these claims\.

The DeepRare study evaluated its patient retrieval module using a one\-vs\-rest \(OVR\) methodology across public datasets, and utilized chronological splitting for intra\-institutional testing\. Although the DeepRare study ablated its patient retrieval module—confirming not only that disabling case retrieval caused a substantial drop in diagnostic accuracy, but also that patient retrieval contributed far more to performance gains than its knowledge search or multi\-turn reflection modules[51](https://arxiv.org/html/2610.33592#bib.bib14)—it did not critically examine the validity of its retrieval strategy\. It remained ambiguous whether its reported performance gains reflected genuine clinical reasoning inherent to the patient\-similarity retrieval paradigm or were artificially induced by candidate\-space leakage\. Our work directly resolves this ambiguity\. As our audit demonstrates, OVR introduces substantial candidate\-space leakage, while chronological splitting fails to resolve spatial overfitting because institutional documentation profiles and extraction noise remain structurally homogeneous over time\. By replicating the DeepRare architecture and subjecting it to strict full\-test isolation, we empirically confirmed that its reported performance advantages are largely artifacts of evaluation design\. Our minimalist RAG prototype experiments further corroborate that this vulnerability is not merely an implementation flaw in DeepRare, but a pervasive evaluation artifact across the entire case\-retrieval RAG paradigm\. Furthermore, DeepRare incorporates automated online web searches, dynamically absorbing contemporary knowledge and introducing a risk of retrieving the exact published case reports of the query patients\. This not only circumvents temporal version control but also creates an asymmetric comparison against chronologically frozen, offline baselines like PhenoBrain\. This conclusion is reinforced by a recent audit of 300 strictly independent cases \(100 each from public registries, PUMCH, and ZJUMCH\)[49](https://arxiv.org/html/2610.33592#bib.bib43), which found DeepRare’s Recall@1 \(32%\) to be lower than GPT\-5\.5 \(39%\) yet higher than PhenoBrain \(27%\)—suggesting its diagnostic advantage is less pronounced than originally reported\.

Analogous evaluation artifacts extend to domain\-specific fine\-tuned LLMs\. For instance, the recently introduced RareSeek model fine\-tuned an LLM on internal clinical records, reporting an in\-distribution Recall@1 of 68\.4%, yet its performance dropped to 39\.2% on the independent external benchmark RareBench\. Confining training and evaluation to the same institutional data distribution induces substantial spatial overfitting and OOD performance degradation, directly mirroring the empirical trends observed in our10×1010\\times 10cross\-cohort analysis on supervised classifiers\. Furthermore, models that perform inference directly over clinical notes are highly vulnerable to post\-hoc cue leakage\. RareSeek utilizes comprehensive inpatient records for diagnostic prediction—encompassing specialized confirmatory test results, targeted therapeutic interventions, and disease trajectory responses\. As demonstrated in our field\-level narrative ablation experiments, processing these retrospective clinical summaries introduces substantial post\-hoc clues that artificially inflate diagnostic accuracy\. This design is inconsistent with prospective clinical workflows, where differential diagnoses must be formulated prior to initiating targeted therapies and observing longitudinal disease trajectories\.

The statistical vulnerability of multi\-dataset aggregation As the RD AI field increasingly adopts multi\-dataset benchmarking, the substantial disparity in cohort sizes—ranging from dozens of cases \(e\.g\., MME and VPI\) to tens of thousands \(e\.g\., Phenopacket and RareArena\)—introduces a statistical vulnerability\. When aggregating diagnostic performance across multiple datasets, standard micro\-averaging \(treating every patient equally\) allows large public registries to disproportionately influence a model’s performance on smaller, specialized, or real\-world clinical test sets\. Conversely, unweighted macro\-averaging across dataset scores can allow small, low\-variance datasets with high accuracy to skew the overall performance assessment of a model\. To ensure transparent reporting, performance metrics should be explicitly reported at the individual cohort level\. When a global summary metric is required, studies should concurrently report both macro\-averaged and micro\-averaged metrics, justifying the chosen aggregation method to prevent cohort size imbalances from masking generalization gaps\.

Limitations and future directions Although our systematic audit provides a methodological framework for identifying evaluation artifacts, it has several limitations\. First, while the public datasets are globally sourced, the independent clinical cohorts \(ZJUMCH and PUMCH\) represent tertiary care centers exclusively in China\. Future validation efforts should incorporate multinational, multi\-ethnic prospective EHR data to investigate whether spatial overfitting manifests differently across distinct regional healthcare systems\. Second, our analysis primarily focuses on phenotype\- and text\-driven diagnostic reasoning\. As the field advances toward multimodal AI—integrating whole\-exome sequencing \(WES\) or medical imaging—new dimensions of data leakage, such as genomic batch effects or imaging equipment footprints, may emerge and require similar auditing\. Finally, the continuous, opaque updating of proprietary LLMs makes it challenging to accurately quantify training data contamination\. Consequently, our findings represent a snapshot of current evaluation vulnerabilities rather than a static metric or best practice pertaining to any single model’s capability\. As the deployment of AI in RD diagnostics progresses, transitioning from conventional computer science benchmarks to clinical\-trial\-inspired validation frameworks will shift the field’s focus from leaderboard performance to clinical safety\. Future studies should address these challenges by developing dynamic, community\-driven benchmarking platforms that enforce temporal synchronization, cross\-cohort deduplication, and cross\-site isolation\. Furthermore, open\-source initiatives should prioritize the programmatic sanitization of clinical narratives to eliminate post\-hoc therapeutic cues from test sets\. By identifying the sources of artificial performance inflation, this work provides the methodological groundwork for building trustworthy, generalizable medical AI—systems capable of navigating clinical uncertainty and reliably reducing diagnostic delays for patients with rare diseases\.

## 4Methods

### 4\.1Historical snapshotting of rare disease knowledge bases

To evaluate temporal knowledge alignment, we curated historical versions of two primary rare disease \(RD\) knowledge bases \(KBs\): Online Mendelian Inheritance in Man \(OMIM\)[17](https://arxiv.org/html/2610.33592#bib.bib4)and Orphanet \(ORPHA\)[44](https://arxiv.org/html/2610.33592#bib.bib3)\. Annual snapshots spanning 2021 to 2026 were retrieved from their official repositories\. Each snapshot preserves the state of disease\-phenotype associations and HPO\-semantic structures at that point in time, serving as a controlled baseline to assess the influence of database updates on diagnostic performance\.

### 4\.2Curation of multi\-source patient repositories

We compiled a dataset of over 60,000 RD cases derived from three primary sources: \(1\) literature\-derived cohorts \(RareArena[3](https://arxiv.org/html/2610.33592#bib.bib13), Phenopacket[10](https://arxiv.org/html/2610.33592#bib.bib15), LIRICAL[32](https://arxiv.org/html/2610.33592#bib.bib10), RDmaster[48](https://arxiv.org/html/2610.33592#bib.bib8), PhenoDP[45](https://arxiv.org/html/2610.33592#bib.bib19), MME[30](https://arxiv.org/html/2610.33592#bib.bib16), VPI[38](https://arxiv.org/html/2610.33592#bib.bib18), and HMS[33](https://arxiv.org/html/2610.33592#bib.bib17)\), \(2\) public electronic registries \(RAMEDIS[40](https://arxiv.org/html/2610.33592#bib.bib9)\), and \(3\) independent clinical EHR cohorts \(ZJUMCH and PUMCH[23](https://arxiv.org/html/2610.33592#bib.bib20)\)\.

To standardize these heterogeneous data sources for computational benchmarking, we implemented a preprocessing pipeline\. For the independent clinical cohorts, we initially collected 1,729 raw EHRs from ZJUMCH\. Phenotypic features from ZJUMCH records were manually extracted and mapped to Human Phenotype Ontology \(HPO\) terms\. Ground\-truth clinical diagnoses were subsequently mapped to canonical OMIM/ORPHA nomenclature\. ZJUMCH cases that could not be mapped to a recognized OMIM/ORPHA diagnosis or contained fewer than three initial phenotypes were excluded\. For the PUMCH cohort, phenotypic features were extracted by the original researchers using an automated annotation tool\. In contrast, the public literature\-derived cohorts and registries were acquired with pre\-curated HPO annotations and standardized diagnostic labels, bypassing the raw text extraction phase\.

To ensure valid cross\-temporal evaluations, all integrated cases underwent a snapshot\-specific filtering process before being evaluated against any historical KB version\. A case was deemed valid for a specific KB snapshot \(e\.g\., ORPHA 2023\) if it met three criteria: \(i\) the ground\-truth OMIM/ORPHA disease existed and had associated phenotype annotations within that specific KB snapshot; \(ii\) the case retained at least two valid HPO terms present in that specific snapshot; and \(iii\) the case passed intra\-dataset deduplication to remove identical patient records\. This filtering protocol controls for data quality and chronological alignment across temporal and spatial evaluations\.

### 4\.3Knowledge\-driven phenotypic diagnostic baselines

We implemented two established phenotype\-based computational algorithms to process a patient HPO query set𝒬=\{t1,t2,…,tn\}\\mathcal\{Q\}=\\\{t\_\{1\},t\_\{2\},\.\.\.,t\_\{n\}\\\}and prioritize candidate diseases within synchronized KB snapshots, following their original implementation specifications\.

- •Likelihood ratio model:Based on the Naive Bayes assumption \(specifically LIRICAL[32](https://arxiv.org/html/2610.33592#bib.bib10)\), this approach treats observed phenotypes as independent tests\. Diagnostic evidence is aggregated by summing the log\-likelihood ratios of individual phenotypes based on disease\-specific frequencies and background prevalences\.
- •Probabilistic graphical model:This approach \(specifically BOQA[1](https://arxiv.org/html/2610.33592#bib.bib2)\) models the HPO taxonomy as a directed acyclic graph \(DAG\)\. It enables the joint modeling of hierarchical inheritance and conditional dependencies among HPO terms, parameterizing false\-positive and false\-negative observation rates to calculate the global marginal posterior probability for candidate diseases\.

### 4\.4Case\-based retrieval diagnosis framework

The case\-based retrieval framework matches a query patient𝒬\\mathcal\{Q\}against a historical case library\. The diagnostic specificity of an HPO termttis quantified via information content \(IC\), calculated as the negative logarithm of its expected prevalence across the annotation space:

IC⁡\(t\)=−log⁡\(∑d∈𝒟Pr⁡\(t∣d\)\|𝒟\|\)\\operatorname\{IC\}\(t\)=\-\\log\\left\(\\frac\{\\sum\_\{d\\in\\mathcal\{D\}\}\\Pr\(t\\mid d\)\}\{\|\\mathcal\{D\}\|\}\\right\)\(1\)wherePr⁡\(t∣d\)\\Pr\(t\\mid d\)is the frequency of phenotypettannotated for diseasedd, and\|𝒟\|\|\\mathcal\{D\}\|denotes the total number of candidate RDs\.

Phenotypic similarity between𝒬\\mathcal\{Q\}and a historical reference case𝒬ref\\mathcal\{Q\}\_\{\\text\{ref\}\}is measured by the average IC across their shared ancestors:

Sim⁡\(𝒬,𝒬ref\)=12​\[∑t∈𝒬∩Anc⁡\(𝒬ref\)IC⁡\(t\)\+∑t′∈𝒬ref∩Anc⁡\(𝒬\)IC⁡\(t′\)\]\\operatorname\{Sim\}\(\\mathcal\{Q\},\\mathcal\{Q\}\_\{\\text\{ref\}\}\)=\\frac\{1\}\{2\}\\left\[\\sum\_\{t\\in\\mathcal\{Q\}\\cap\\operatorname\{Anc\}\(\\mathcal\{Q\}\_\{\\text\{ref\}\}\)\}\\operatorname\{IC\}\(t\)\+\\sum\_\{t^\{\\prime\}\\in\\mathcal\{Q\}\_\{\\text\{ref\}\}\\cap\\operatorname\{Anc\}\(\\mathcal\{Q\}\)\}\\operatorname\{IC\}\(t^\{\\prime\}\)\\right\]\(2\)
A max\-pooling strategy aggregates case\-level similarity scores to generate disease\-level recommendations\. The priority score for candidate diseaseddis determined by its most phenotypically similar historical counterpart\.

### 4\.5Supervised machine learning and deep learning

To perform RD diagnosis on sparse phenotype features, we implemented three representative classes of classifiers: a linear model, a non\-linear tree\-based ensemble, and a deep feed\-forward neural network\.

- •Feature representation:The model input was constructed by mapping query phenotypes onto anMM\-dimensional multi\-hot query vector𝐱∈\{0,1\}M\\mathbf\{x\}\\in\\\{0,1\\\}^\{M\}, whereMMrepresents the total number of active HPO terms \(M≈16,000M\\approx 16,000\)\. Directed ancestor propagation was applied to capture the hierarchical semantic structure of the HPO, setting an index to 1 if the corresponding HPO term or its descendants were present\. The label space consisted ofNNunique classes mapped to OMIM or ORPHA nomenclatures\.
- •Linear classifier:Optimized via stochastic gradient descent \(SGD\)[29](https://arxiv.org/html/2610.33592#bib.bib32)with a log loss function, regularized using anL2L\_\{2\}weight penalty \(regularization coefficientα=1×10−4\\alpha=1\\times 10^\{\-4\}\)\.
- •Non\-linear classifier:An extreme gradient boosting \(XGBoost\)[4](https://arxiv.org/html/2610.33592#bib.bib30)ensemble with a learning rate of 0\.1 and 50 boosting rounds, utilizing a column\-subsampling strategy \(10% of the active feature space per tree\) and a maximum tree depth of 5 to prevent overfitting\.
- •Deep learning classifier:A multilayer perceptron \(MLP\)[35](https://arxiv.org/html/2610.33592#bib.bib33)consisting of an input layer that receives the phenotypic feature vector, followed by a hidden layer with 1,024 units and ReLU activation, a dropout regularization layer, and finally a fully connected output layer for classification\. The model was trained with a batch size of 32 using the AdamW optimizer with an initial learning rate of1×10−41\\times 10^\{\-4\}and a cosine annealing schedule\.
- •Dataset splitting:For cross\-dataset evaluations, a 50/50 split was enforced per target dataset\. Training sets were constructed by merging the remaining 50% with RareArena \(the largest dataset with the broadest disease coverage\)\.

### 4\.6Large language model\-based diagnostic paradigms

We implemented three large language model \(LLM\)\-based diagnostic paradigms: \(i\) a replication of the state\-of\-the\-art agentic system DeepRare, \(ii\) a controlled, simplified patient\-retrieval LLM prototype, and \(iii\) a zero\-shot narrative inference framework designed for stepwise clinical text ablation\.

#### 4\.6\.1Redeployment of the DeepRare agentic system

We replicated the DeepRare framework using its open\-source repository, preserving its core multi\-step reasoning architecture\. The agentic pipeline integrates several key functional modules: \(i\) dynamic external diagnostic tool invocation \(PhenoBrain and PubCaseFinder\), \(ii\) live web\-based biomedical knowledge search, \(iii\) case\-based similar\-patient retrieval, and \(iv\) multi\-turn knowledge\-enhanced reflection and candidate re\-ranking\. Since the original reference patient database used in the DeepRare study is not publicly available, we populated the system’s patient retrieval database using our curated data hierarchy \(Table[1](https://arxiv.org/html/2610.33592#S2.T1)\), specifically incorporating the public and clinical cohorts that overlap with the original study \(e\.g\., LIRICAL, MME, HMS, and RAMEDIS\)\. Without loss of generality, we implemented DeepRare’s similar\-patient retrieval functionality using our proposed IC\-based case retrieval framework\.

#### 4\.6\.2Patient\-retrieval RAG LLM prototype

To evaluate the patient retrieval module independently of complex agentic workflows, we developed a simplified retrieval\-augmented generation \(RAG\) prototype\. This baseline omitted components such as multi\-turn reflection and tool invocation, coupling the foundation LLM directly to the IC\-based case retrieval module\. For a given query𝒬\\mathcal\{Q\}, the patient retrieval framework retrieved the top\-10 most phenotypically similar historical cases\. These exemplars were provided as few\-shot contexts alongside the query case to generate a prioritized differential list\. Importantly, the prompt was structured such that the retrieved cases were presented as supporting clinical references to assist differential reasoning, rather than as a closed multiple\-choice candidate pool\.

#### 4\.6\.3Zero\-shot diagnosis on clinical narratives

To evaluate LLM sensitivity to text\-based contextual biases, we adopted a zero\-shot diagnostic paradigm utilizing clinical narrative text\. We curated 796 de\-identified inpatient EHRs with confirmed ORPHA diagnoses from the in\-house ZJUMCH cohort\. To simulate the chronological progression of real\-world hospital admission workflows, we segmented each raw medical record into four sequential clinical sections: \(i\) chief complaint \(CC\): primary onset symptoms, demographic information, and history of present illness; \(ii\) physical examination \(PE\): objective dysmorphic features and physical signs; \(iii\) ancillary examinations \(AE\): laboratory and radiological test findings; and \(iv\) clinical treatment \(CT\): therapeutic interventions and pharmacological treatments, with occasional inclusion of ancillary examination findings\. To eliminate direct label leakage, any fields explicitly indicating diagnostic outcomes—such as admission and discharge diagnoses—were strictly excluded, and regular expression pattern matching was applied to redact all mentions of ground\-truth diagnosis names across the remaining sections\. We conducted an incremental ablation protocol by sequentially appending these segments to the zero\-shot prompt\. The LLM was instructed to infer a prioritized differential list of candidate RDs\. Additionally, to determine whether narrative text provides additional phenotypic value beyond structured ontologies, we evaluated a hybrid input setting where raw narrative segments were incrementally appended to baseline structured HPO annotations extracted from clinical narratives\.

#### 4\.6\.4Disease entity resolution

To map LLM\-generated free\-text diagnoses to standardized OMIM/ORPHA codes while mitigating identifier hallucinations, we implemented a semantic entity resolution pipeline using BioLORD[31](https://arxiv.org/html/2610.33592#bib.bib21), a pre\-trained biomedical language model\. We computed dense embeddings for both the LLM\-generated diagnoses and all canonical diseases in OMIM/ORPHA, assigning the OMIM/ORPHA code with the highest cosine similarity to the LLM’s raw diagnoses\.

### 4\.7Experimental configurations and evaluation settings

Diagnostic metrics:Diagnostic efficacy was evaluated using Recall@nn\(n∈\[1,10\]n\\in\[1,10\]\) with an “any\-hit” strategy for multiple causative diseases\. A diagnostic session was marked successful if any ground\-truth causative diagnosis ranked within the top\-nnprioritized recommendations\.

KB version control:For temporal evaluations \(2021–2026\), test cases were filtered to ensure ground\-truth diagnoses were annotatable under the specific annual snapshot, with HPO profiles synchronized to the corresponding ontology version of that year\. Within any single snapshot\-based benchmark run, patient cohort labels were restricted to either OMIM or ORPHA\. For static baseline evaluations that did not require temporal knowledge permutation, we used the 2023 ORPHA snapshot as the global reference database\.

LLM configurations:All LLMs utilized across our experimental paradigms were deployed using their official snapshots\. To ensure reproducibility and prevent performance variations introduced by generation\-level optimization, we did not apply any custom hyperparameter adjustments\. All inference was conducted using the default generation parameters \(e\.g\., default temperature, probability threshold for top\-ppsampling, and frequency penalties\) without any task\-specific decoding configurations\.

Case retrieval strategies:For any diagnostic paradigms that include the case retrieval module, evaluations on a target datasetDt​a​r​g​e​tD\_\{target\}were typically conducted under three reference\-bank partitioning configurations: \(i\) Full test:Dt​a​r​g​e​tD\_\{target\}was completely excluded from the retrieval bank; \(ii\)nn\-fold variant:1/n1/nofDt​a​r​g​e​tD\_\{target\}was allocated to the retrieval bank and the remainder used for testing; and \(iii\) one\-vs\-rest \(OVR\): excluding only the identical record of the query patient\. Any datasets in Table[1](https://arxiv.org/html/2610.33592#S2.T1)other thanDt​a​r​g​e​tD\_\{target\}were included in the retrieval bank\. Additionally, because many LIRICAL cases are subsets of the Phenopacket dataset, Phenopacket cases were explicitly excluded from the retrieval bank when evaluating DeepRare on the LIRICAL cohort\.

PMID\-level deduplication:PMID\-level deduplication was implemented in three steps: \(i\) identifying the source PMID associated with each literature\-derived query case, \(ii\) matching this identifier against the publication metadata across the entire reference corpus, and \(iii\) excluding all co\-published reference cases from the candidate bank prior to inference\.

Supervised classifier implementation:The linear classifier was implemented using SGDClassifier \(scikit\-learn v1\.0\.2\)[29](https://arxiv.org/html/2610.33592#bib.bib32), the non\-linear ensemble using XGBoost \(xgboost v2\.0\.3\)[4](https://arxiv.org/html/2610.33592#bib.bib30), and the deep feed\-forward neural network was implemented using PyTorch \(v2\.1\.2\)[28](https://arxiv.org/html/2610.33592#bib.bib38)\.

Statistical analysis:\(i\) For paired binary diagnostic outcomes across DeepRare evaluation settings, statistical significance was evaluated using McNemar’s test\. \(ii\) A paired one\-sided bootstrap test was used to assess the statistical significance of performance differences between methods as well as the 95% confidence intervals \(CIs\)\. For each metric \(e\.g\., Recall@kk\), we generated 2,000 bootstrap samples by sampling test cases with replacement\. The one\-sidedPP\-value was calculated as the fraction of bootstrap iterations where each model achieved an equal or higher recall than a designated reference model, following the methodology described by Koehn[19](https://arxiv.org/html/2610.33592#bib.bib22)\. The 95% CIs were derived using the 2\.5th and 97\.5th percentiles of the bootstrap distribution\. \(iii\) The paired McNemar’s odds ratio was employed to quantify relative diagnostic gains across clinical narrative ablation stages\. \(iv\) To quantify effect sizes across multi\-dataset aggregations, we calculated Hedges’ggwith small\-sample bias correction\. \(v\) Cohen’sddwas calculated to measure the performance degradation between in\-distribution and out\-of\-distribution evaluations in supervised classifiers\.

## Data Availability

Historical versions of the Human Phenotype Ontology and corresponding OMIM/ORPHA disease annotations are publicly accessible at[https://github\.com/obophenotype/human\-phenotype\-ontology](https://github.com/obophenotype/human-phenotype-ontology)\. The multi\-source rare disease patient cohorts utilized in this study are available via their respective URLs, as detailed in Supplementary Table S2\. The primary electronic health record data from the Children’s Hospital of Zhejiang University School of Medicine \(ZJUMCH\) are not publicly available due to institutional data privacy regulations and ethical constraints\. The ZJUMCH patient data analyzed in this work were derived from a prior research study conducted under institutional review board \(IRB\) approval \(ZJUMCH IRB \#2021\-IRB\-165, \#2022\-IRB\-068\), and all clinical records were fully de\-identified prior to computational analysis in the present study\.

## Code Availability

## Acknowledgements

This work is supported by the National Natural Science Foundation of China \(Nos\. 62506329 and 82572377\) and the “Pioneer” and “Leading Goose” R&D Program of Zhejiang \(No\. 2026C01022\)\.

## Author Contributions

J\.Y\. and H\.L\. conceived and designed the study, developed the methodology, managed project administration, and acquired funding\. J\.Y\., P\.L\., X\.L\., and H\.L\. curated and processed the benchmark datasets\. Y\.C\. and Y\.F\. conducted the investigations\. J\.Y\. and P\.L\. performed the formal data analysis\. J\.Y\., X\.L\., Y\.C\., Y\.F\., and H\.L\. performed the validation of results\. H\.L\. provided resources and overall supervision\. J\.Y\. and H\.L\. drafted the initial manuscript\. All authors \(J\.Y\., P\.L\., X\.L\., S\.L\., Y\.C\., Y\.F\., and H\.L\.\) critically reviewed and edited the manuscript, and approved the final version\.

## Competing Interests

The authors declare no competing interests\.

## References

- S\. Bauer, S\. Köhler, M\. H\. Schulz, and P\. N\. RobinsonBayesian ontology querying for accurate and noise\-tolerant semantic searches\.Bioinformatics28\(19\),pp\. 2502–2508\.External Links:ISSN 13674803,[Document](https://dx.doi.org/10.1093/bioinformatics/bts471)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.5),[§2\.5](https://arxiv.org/html/2610.33592#S2.SS5.p1.1.4),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.16.1.1.1.3),[§3](https://arxiv.org/html/2610.33592#S3.p3.1.2),[2nd item](https://arxiv.org/html/2610.33592#S4.I1.i2.p1.1.2)\.
- Bauskiset al\.\(2022\)A\. Bauskis, C\. Strange, C\. Molster, and C\. FisherThe diagnostic odyssey: insights from parents of children living with an undiagnosed condition\.Orphanet Journal of Rare Diseases17\(1\),pp\. 233\.External Links:ISSN 1750\-1172,[Document](https://dx.doi.org/10.1186/s13023-022-02358-x)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.4)\.
- Chenet al\.\(2026a\)H\. Chen, Z\. Zhao, S\. Zhou, S\. Hu, J\. Wang, Y\. Jin, X\. Jin, Y\. C\. Tham, X\. Wang, W\. Ma, H\. Wu, B\. Sheng, S\. Zhang, S\. Yu, and T\. Y\. WongRareArena: a comprehensive benchmark dataset unveiling the potential of large language models in rare disease diagnosis\.The Lancet Digital Health8\(2\)\.External Links:ISSN 2589\-7500,[Document](https://dx.doi.org/10.1016/j.landig.2025.100953)Cited by:[1st item](https://arxiv.org/html/2610.33592#S2.I1.i1.p1.1.7),[§3](https://arxiv.org/html/2610.33592#S3.p2.1.3),[§3](https://arxiv.org/html/2610.33592#S3.p2.1.4),[§4\.2](https://arxiv.org/html/2610.33592#S4.SS2.p1.1.1)\.
- Chen and Guestrin \(2016\)T\. Chen and C\. GuestrinXGBoost: A Scalable Tree Boosting System\.Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 785–794\.External Links:[Document](https://dx.doi.org/10.1145/2939672.2939785)Cited by:[3rd item](https://arxiv.org/html/2610.33592#S4.I2.i3.p1.1.2),[§4\.7](https://arxiv.org/html/2610.33592#S4.SS7.p6.1.3)\.
- Chenet al\.\(2025\)X\. Chen, H\. Yi, M\. You, W\. Liu, L\. Wang, H\. Li, X\. Zhang, Y\. Guo, L\. Fan, G\. Chen, Q\. Lao, W\. Fu, K\. Li, and J\. LiEnhancing diagnostic capability with multi\-agents conversational large language models\.npj Digital Medicine8\(1\),pp\. 159\.External Links:ISSN 2398\-6352,[Document](https://dx.doi.org/10.1038/s41746-025-01550-0)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.9),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.9.1.1.1.3)\.
- Chenet al\.\(2026b\)X\. Chen, Y\. Jin, X\. Mao, L\. Wang, S\. Zhang, and T\. ChenRareAgents: autonomous multi\-disciplinary team for rare disease diagnosis and treatment\.Proceedings of the Fortieth AAAI Conference on Artificial Intelligence and Thirty\-Eighth Conference on Innovative Applications of Artificial Intelligence and Sixteenth Symposium on Educational Advances in Artificial Intelligence\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i1.36969)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.9),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.3.1.1.1.3)\.
- Chenet al\.\(2024\)X\. Chen, X\. Mao, Q\. Guo, L\. Wang, S\. Zhang, and T\. ChenRareBench: Can LLMs Serve as Rare Diseases Specialists?\.ACM SIGKDD Conference on Knowledge Discovery and Data Mining\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2402.06341)Cited by:[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.11.1.1.1.3)\.
- Chenget al\.\(2025\)Y\. Cheng, Y\. Chang, and Y\. WuA Survey on Data Contamination for Large Language Models\.arXiv preprint arXiv:2502\.14425\.Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p3.1.2),[§3](https://arxiv.org/html/2610.33592#S3.p2.1.6)\.
- Choet al\.\(2026\)H\. N\. Cho, S\. Sutari, A\. Lopez, H\. Bow, and K\. ZhengBuilding Safe and Deployable Clinical Natural Language Processing under Temporal Leakage Constraints\.arXiv preprint arXiv:2602\.15852\.Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p3.1.6)\.
- Daniset al\.\(2025\)D\. Danis, M\. J\. Bamshad, Y\. Bridges, A\. Caballero\-Oteyza, P\. Cacheiro, L\. C\. Carmody, L\. Chimirri, J\. X\. Chong, B\. Coleman, R\. Dalgleish, P\. J\. Freeman, A\. S\.L\. Graefe, T\. Groza, P\. Hansen, J\. O\.B\. Jacobsen, A\. Klocperk, M\. Kusters, M\. S\. Ladewig, A\. J\. Marcello, T\. Mattina, C\. J\. Mungall, M\. C\. Munoz\-Torres, J\. T\. Reese, F\. Rehburg, B\. C\.S\. Reis, C\. Schuetz, D\. Smedley, T\. Strauss, J\. C\. Sundaramurthi, S\. Thun, K\. Wissink, J\. F\. Wagstaff, D\. Zocche, M\. A\. Haendel, and P\. N\. RobinsonA corpus of GA4GH phenopackets: Case\-level phenotyping for genomic diagnostics and discovery\.Human Genetics and Genomics Advances6\(1\),pp\. 100371\.External Links:ISSN 2666\-2477,[Document](https://dx.doi.org/10.1016/j.xhgg.2024.100371)Cited by:[1st item](https://arxiv.org/html/2610.33592#S2.I1.i1.p1.1.2),[§3](https://arxiv.org/html/2610.33592#S3.p2.1.2),[§4\.2](https://arxiv.org/html/2610.33592#S4.SS2.p1.1.2)\.
- Fayeet al\.\(2024\)F\. Faye, C\. Crocione, R\. Anido de Peña, S\. Bellagambi, L\. Escati Peñaloza, A\. Hunter, L\. Jensen, C\. Oosterwijk, E\. Schoeters, D\. de Vicente, L\. Faivre, M\. Wilbur, Y\. Le Cam, and J\. DubiefTime to diagnosis and determinants of diagnostic delays of people living with a rare disease: results of a Rare Barometer retrospective patient survey\.European Journal of Human Genetics32\(9\),pp\. 1116–1126\.External Links:ISSN 1476\-5438,[Document](https://dx.doi.org/10.1038/s41431-024-01604-z)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.4)\.
- Fujiwaraet al\.\(2018\)T\. Fujiwara, Y\. Yamamoto, J\. Kim, O\. Buske, and T\. TakagiPubCaseFinder: A Case\-Report\-Based, Phenotype\-Driven Differential\-Diagnosis System for Rare Diseases\.The American Journal of Human Genetics103\(3\),pp\. 389–399\.External Links:ISSN 0002\-9297,[Document](https://dx.doi.org/10.1016/j.ajhg.2018.08.003)Cited by:[§2\.4](https://arxiv.org/html/2610.33592#S2.SS4.p1.1.1)\.
- Garganoet al\.\(2024\)M\. A\. Gargano, N\. Matentzoglu, B\. Coleman, E\. B\. Addo\-Lartey, A\. V\. Anagnostopoulos, J\. Anderton, P\. Avillach, A\. M\. Bagley, E\. Bakštein, J\. P\. Balhoff, G\. Baynam, S\. M\. Bello, M\. Berk, H\. Bertram, S\. Bishop, H\. Blau, D\. F\. Bodenstein, P\. Botas, K\. Boztug, J\. Čady, T\. J\. Callahan, R\. Cameron, S\. J\. Carbon, F\. Castellanos, J\. H\. Caufield, L\. E\. Chan, C\. G\. Chute, J\. Cruz\-Rojo, N\. Dahan\-Oliel, J\. R\. Davids, M\. de Dieuleveult, V\. de Souza, B\. B\. A\. de Vries, E\. de Vries, J\. R\. DePaulo, B\. Derfalvi, F\. Dhombres, C\. Diaz\-Byrd, A\. J\. M\. Dingemans, B\. Donadille, M\. Duyzend, R\. Elfeky, S\. Essaid, C\. Fabrizzi, G\. Fico, H\. V\. Firth, Y\. Freudenberg\-Hua, J\. M\. Fullerton, D\. L\. Gabriel, K\. Gilmour, J\. Giordano, F\. S\. Goes, R\. G\. Moses, I\. Green, M\. Griese, T\. Groza, W\. Gu, J\. Guthrie, B\. Gyori, A\. Hamosh, M\. Hanauer, K\. Hanušová, Y\. \(\. He, H\. Hegde, I\. Helbig, K\. Holasová, C\. T\. Hoyt, S\. Huang, E\. Hurwitz, J\. O\. B\. Jacobsen, X\. Jiang, L\. Joseph, K\. Keramatian, B\. King, K\. Knoflach, D\. A\. Koolen, M\. L\. Kraus, C\. Kroll, M\. Kusters, M\. S\. Ladewig, D\. Lagorce, M\. Lai, P\. Lapunzina, B\. Laraway, D\. Lewis\-Smith, X\. Li, C\. Lucano, M\. Majd, M\. L\. Marazita, V\. Martinez\-Glez, T\. H\. McHenry, M\. G\. McInnis, J\. A\. McMurry, M\. Mihulová, C\. E\. Millett, P\. B\. Mitchell, V\. Moslerová, K\. Narutomi, S\. Nematollahi, J\. Nevado, A\. A\. Nierenberg, N\. N\. Čajbiková, J\. I\. Nurnberger, S\. Ogishima, D\. Olson, A\. Ortiz, H\. Pachajoa, G\. Perez de Nanclares, A\. Peters, T\. Putman, C\. K\. Rapp, A\. Rath, J\. Reese, L\. Rekerle, A\. M\. Roberts, S\. Roy, S\. J\. Sanders, C\. Schuetz, E\. C\. Schulte, T\. G\. Schulze, M\. Schwarz, K\. Scott, D\. Seelow, B\. Seitz, Y\. Shen, M\. N\. Similuk, E\. S\. Simon, B\. Singh, D\. Smedley, C\. L\. Smith, J\. T\. Smolinsky, S\. Sperry, E\. Stafford, R\. Stefancsik, R\. Steinhaus, R\. Strawbridge, J\. C\. Sundaramurthi, P\. Talapova, J\. A\. Tenorio Castano, P\. Tesner, R\. H\. Thomas, A\. Thurm, M\. Turnovec, M\. E\. van Gijn, N\. A\. Vasilevsky, M\. Vlčková, A\. Walden, K\. Wang, R\. Wapner, J\. S\. Ware, A\. A\. Wiafe, S\. A\. Wiafe, L\. D\. Wiggins, A\. E\. Williams, C\. Wu, M\. J\. Wyrwoll, H\. Xiong, N\. Yalin, Y\. Yamamoto, L\. N\. Yatham, A\. K\. Yocum, A\. H\. Young, Z\. Yüksel, P\. P\. Zandi, A\. Zankl, I\. Zarante, M\. Zvolský, S\. Toro, L\. C\. Carmody, N\. L\. Harris, M\. C\. Munoz\-Torres, D\. Danis, C\. J\. Mungall, S\. Köhler, M\. A\. Haendel, and P\. N\. RobinsonThe Human Phenotype Ontology in 2024: phenotypes around the world\.Nucleic Acids Research52\(D1\),pp\. D1333–D1346\.External Links:ISSN 0305\-1048,[Document](https://dx.doi.org/10.1093/nar/gkad1005)Cited by:[1st item](https://arxiv.org/html/2610.33592#S2.I1.i1.p1.1.10)\.
- Ghassemiet al\.\(2021\)M\. Ghassemi, L\. Oakden\-Rayner, and A\. L\. BeamThe false hope of current approaches to explainable artificial intelligence in health care\.The Lancet Digital Health3\(11\),pp\. e745–e750\.External Links:ISSN 2589\-7500,[Document](https://dx.doi.org/10.1016/S2589-7500%2821%2900208-9)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p2.1.2)\.
- Goldenberg and Wiens \(2026\)A\. Goldenberg and J\. WiensIs AI actually improving healthcare?\.Nature Medicine32\(4\),pp\. 1182–1183\.External Links:ISSN 1546\-170X,[Document](https://dx.doi.org/10.1038/s41591-026-04329-2)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.12)\.
- Haendelet al\.\(2020\)M\. Haendel, N\. Vasilevsky, D\. Unni, C\. Bologa, N\. Harris, H\. Rehm, A\. Hamosh, G\. Baynam, T\. Groza, J\. McMurry, H\. Dawkins, A\. Rath, C\. Thaxton, G\. Bocci, M\. P\. Joachimiak, S\. Köhler, P\. N\. Robinson, C\. Mungall, and T\. I\. OpreaHow many rare diseases are there?\.Nature reviews\. Drug discovery19\(2\),pp\. 77–78\(eng\)\.External Links:ISSN 1474\-1784 1474\-1776,[Document](https://dx.doi.org/10.1038/d41573-019-00180-y)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.2)\.
- Hamoshet al\.\(2021\)A\. Hamosh, J\. S\. Amberger, C\. Bocchini, A\. F\. Scott, and S\. A\. RasmussenOnline Mendelian Inheritance in Man \(OMIM®\): Victor McKusick’s magnum opus\.American Journal of Medical Genetics Part A185\(11\),pp\. 3259–3265\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1002/ajmg.a.62407)Cited by:[§2\.5](https://arxiv.org/html/2610.33592#S2.SS5.p1.1.2),[§4\.1](https://arxiv.org/html/2610.33592#S4.SS1.p1.1.1)\.
- Kapoor and Narayanan \(2023\)S\. Kapoor and A\. NarayananLeakage and the reproducibility crisis in machine\-learning\-based science\.Patterns4\(9\),pp\. 100804\.External Links:[Document](https://dx.doi.org/10.1016/j.patter.2023.100804)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p3.1.4)\.
- Koehn \(2004\)P\. KoehnStatistical Significance Tests for Machine Translation Evaluation\.Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing,pp\. 388–395\.Cited by:[§4\.7](https://arxiv.org/html/2610.33592#S4.SS7.p7.1.2)\.
- Köhleret al\.\(2009\)S\. Köhler, M\. H\. Schulz, P\. Krawitz, S\. Bauer, S\. Dölken, C\. E\. Ott, C\. Mundlos, D\. Horn, S\. Mundlos, and P\. N\. RobinsonClinical Diagnostics in Human Genetics with Semantic Similarity Searches in Ontologies\.American Journal of Human Genetics85\(4\),pp\. 457–464\.External Links:ISSN 00029297,[Document](https://dx.doi.org/10.1016/j.ajhg.2009.09.003)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.5),[§2\.4](https://arxiv.org/html/2610.33592#S2.SS4.p1.1.2),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.17.1.1.1.3),[§3](https://arxiv.org/html/2610.33592#S3.p3.1.2)\.
- Liuet al\.\(2020\)X\. Liu, S\. Cruz Rivera, D\. Moher, M\. J\. Calvert, A\. K\. Denniston, A\. Chan, A\. Darzi, C\. Holmes, C\. Yau, H\. Ashrafian, J\. J\. Deeks, L\. Ferrante di Ruffano, L\. Faes, P\. A\. Keane, S\. J\. Vollmer, A\. Y\. Lee, A\. Jonas, A\. Esteva, A\. L\. Beam, A\. Chan, M\. B\. Panico, C\. S\. Lee, C\. Haug, C\. J\. Kelly, C\. Yau, C\. Mulrow, C\. Espinoza, J\. Fletcher, D\. Paltoo, E\. Manna, G\. Price, G\. S\. Collins, H\. Harvey, J\. Matcham, J\. Monteiro, M\. K\. ElZarrad, L\. Ferrante di Ruffano, L\. Oakden\-Rayner, M\. McCradden, P\. A\. Keane, R\. Savage, R\. Golub, R\. Sarkar, S\. Rowley, The SPIRIT\-AI and CONSORT\-AI Working Group, SPIRIT\-AI and CONSORT\-AI Steering Group, and SPIRIT\-AI and CONSORT\-AI Consensus GroupReporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT\-AI extension\.Nature Medicine26\(9\),pp\. 1364–1374\.External Links:[Document](https://dx.doi.org/10.1038/s41591-020-1034-x)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p5.1.2),[§2\.7](https://arxiv.org/html/2610.33592#S2.SS7.p2.1.1)\.
- Manda \(2025\)P\. MandaLarge Language Models in Bio\-Ontology Research: A Review\.Bioengineering12\(11\),pp\. 1260\.External Links:ISSN 2306\-5354,[Document](https://dx.doi.org/10.3390/bioengineering12111260)Cited by:[§3](https://arxiv.org/html/2610.33592#S3.p2.1.5)\.
- Maoet al\.\(2025\)X\. Mao, Y\. Huang, Y\. Jin, L\. Wang, X\. Chen, H\. Liu, X\. Yang, H\. Xu, X\. Luan, Y\. Xiao, S\. Feng, J\. Zhu, X\. Zhang, R\. Jiang, S\. Zhang, and T\. ChenA phenotype\-based AI pipeline outperforms human experts in differentially diagnosing rare diseases using EHRs\.npj Digital Medicine8\(1\),pp\. 68\.External Links:ISSN 2398\-6352,[Document](https://dx.doi.org/10.1038/s41746-025-01452-1)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.6),[3rd item](https://arxiv.org/html/2610.33592#S2.I1.i3.p1.1.2),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.12.1.1.1.3),[§4\.2](https://arxiv.org/html/2610.33592#S4.SS2.p1.1.10)\.
- Michaels\-Igbokweet al\.\(2021\)C\. Michaels\-Igbokwe, B\. McInnes, K\. V\. MacDonald, G\. R\. Currie, F\. Omar, B\. Shewchuk, F\. P\. Bernier, and D\. A\. Marshall\(Un\)standardized testing: the diagnostic odyssey of children with rare genetic disorders in Alberta, Canada\.Genetics in Medicine23\(2\),pp\. 272–279\.External Links:ISSN 1098\-3600,[Document](https://dx.doi.org/10.1038/s41436-020-00975-0)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.4)\.
- Moheret al\.\(2010\)D\. Moher, S\. Hopewell, K\. F\. Schulz, V\. Montori, P\. C\. Gøtzsche, P\. J\. Devereaux, D\. Elbourne, M\. Egger, and D\. G\. AltmanCONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials\.BMJ340,pp\. c869\.External Links:[Document](https://dx.doi.org/10.1136/bmj.c869)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p5.1.1)\.
- Nguyenet al\.\(2026\)M\. Nguyen, C\. Yang, T\. A\. Cassini, F\. Ma, R\. Hamid, L\. Bastarache, J\. F\. Peterson, H\. Xu, L\. Li, S\. Ma, and C\. ShyrDiagnostic accuracy of large language models for rare diseases: a systematic review and meta\-analysis\.npj Digital Medicine\.External Links:ISSN 2398\-6352,[Document](https://dx.doi.org/10.1038/s41746-026-03109-z)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p2.1.1),[§2\.6](https://arxiv.org/html/2610.33592#S2.SS6.p1.1.1),[§3](https://arxiv.org/html/2610.33592#S3.p1.1.1)\.
- Omaret al\.\(2026\)M\. Omar, R\. Agbareia, A\. Gorenshtein, A\. Ramaswamy, A\. Sakhuja, Y\. Barash, D\. S\. W\. Ting, E\. Klang, and G\. N\. NadkarniHow to meaningfully evaluate AI in clinical medicine\.Nature Medicine32\(6\),pp\. 1948–1949\.External Links:ISSN 1546\-170X,[Document](https://dx.doi.org/10.1038/s41591-026-04350-5)Cited by:[§2\.7](https://arxiv.org/html/2610.33592#S2.SS7.p1.1.1)\.
- Paszkeet al\.\(2019\)A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Köpf, E\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. ChintalaPyTorch: an imperative style, high\-performance deep learning library\.Proceedings of the 33rd International Conference on Neural Information Processing Systems\.Cited by:[§4\.7](https://arxiv.org/html/2610.33592#S4.SS7.p6.1.4)\.
- Pedregosaet al\.\(2011\)F\. Pedregosa, G\. Varoquaux, A\. Gramfort, V\. Michel, B\. Thirion, O\. Grisel, M\. Blondel, P\. Prettenhofer, R\. Weiss, V\. Dubourg, J\. Vanderplas, A\. Passos, D\. Cournapeau, M\. Brucher, M\. Perrot, and É\. DuchesnayScikit\-learn: Machine Learning in Python\.The Journal of Machine Learning Research12,pp\. 2825–2830\.External Links:ISSN 1532\-4435Cited by:[2nd item](https://arxiv.org/html/2610.33592#S4.I2.i2.p1.1.2),[§4\.7](https://arxiv.org/html/2610.33592#S4.SS7.p6.1.2)\.
- Philippakiset al\.\(2015\)A\. A\. Philippakis, D\. R\. Azzariti, S\. Beltran, A\. J\. Brookes, C\. A\. Brownstein, M\. Brudno, H\. G\. Brunner, O\. J\. Buske, K\. Carey, C\. Doll, S\. Dumitriu, S\. O\.M\. Dyke, J\. T\. den Dunnen, H\. V\. Firth, R\. A\. Gibbs, M\. Girdea, M\. Gonzalez, M\. A\. Haendel, A\. Hamosh, I\. A\. Holm, L\. Huang, M\. E\. Hurles, B\. Hutton, J\. B\. Krier, A\. Misyura, C\. J\. Mungall, J\. Paschall, B\. Paten, P\. N\. Robinson, F\. Schiettecatte, N\. L\. Sobreira, G\. J\. Swaminathan, P\. E\. Taschner, S\. F\. Terry, N\. L\. Washington, S\. Züchner, K\. M\. Boycott, and H\. L\. RehmThe Matchmaker Exchange: A Platform for Rare Disease Gene Discovery\.Human Mutation36\(10\),pp\. 915–921\.External Links:ISSN 1059\-7794,[Document](https://dx.doi.org/10.1002/humu.22858)Cited by:[1st item](https://arxiv.org/html/2610.33592#S2.I1.i1.p1.1.6),[§4\.2](https://arxiv.org/html/2610.33592#S4.SS2.p1.1.6)\.
- Remyet al\.\(2024\)F\. Remy, K\. Demuynck, and T\. DemeesterBioLORD\-2023: semantic textual representations fusing large language models and clinical knowledge graph insights\.Journal of the American Medical Informatics Association31\(9\),pp\. 1844–1855\.External Links:ISSN 1527\-974X,[Document](https://dx.doi.org/10.1093/jamia/ocae029)Cited by:[§4\.6\.4](https://arxiv.org/html/2610.33592#S4.SS6.SSS4.p1.1.1)\.
- Robinsonet al\.\(2020\)P\. N\. Robinson, V\. Ravanmehr, J\. O\. B\. Jacobsen, D\. Danis, X\. A\. Zhang, L\. C\. Carmody, M\. A\. Gargano, C\. L\. Thaxton, G\. Karlebach, J\. Reese, M\. Holtgrewe, S\. Köhler, J\. A\. McMurry, M\. A\. Haendel, and D\. SmedleyInterpretable Clinical Genomics with a Likelihood Ratio Paradigm\.American Journal of Human Genetics107\(3\),pp\. 403–417\.External Links:ISSN 15376605,[Document](https://dx.doi.org/10.1016/j.ajhg.2020.06.021)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.5),[1st item](https://arxiv.org/html/2610.33592#S2.I1.i1.p1.1.3),[§2\.5](https://arxiv.org/html/2610.33592#S2.SS5.p1.1.3),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.15.1.1.1.3),[§3](https://arxiv.org/html/2610.33592#S3.p3.1.2),[1st item](https://arxiv.org/html/2610.33592#S4.I1.i1.p1.1.2),[§4\.2](https://arxiv.org/html/2610.33592#S4.SS2.p1.1.3)\.
- Ronickeet al\.\(2019\)S\. Ronicke, M\. C\. Hirsch, E\. Türk, K\. Larionov, D\. Tientcheu, and A\. D\. WagnerCan a decision support system accelerate rare disease diagnosis? Evaluating the potential impact of Ada DX in a retrospective study\.Orphanet Journal of Rare Diseases14\(1\),pp\. 69\.External Links:ISSN 1750\-1172,[Document](https://dx.doi.org/10.1186/s13023-019-1040-6)Cited by:[1st item](https://arxiv.org/html/2610.33592#S2.I1.i1.p1.1.8),[§4\.2](https://arxiv.org/html/2610.33592#S4.SS2.p1.1.8)\.
- Roseet al\.\(2025\)D\. Rose, C\. Hung, M\. Lepri, I\. Alqassem, K\. Gashteovski, and C\. LawrenceMEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis\.arXiv preprint arXiv:2502\.19175\.Cited by:[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.7.1.1.1.3)\.
- Rumelhartet al\.\(1986\)D\. E\. Rumelhart, G\. E\. Hinton, and R\. J\. WilliamsLearning representations by back\-propagating errors\.Nature323\(6088\),pp\. 533–536\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/323533a0)Cited by:[4th item](https://arxiv.org/html/2610.33592#S4.I2.i4.p1.1.2)\.
- Schumacheret al\.\(2025\)E\. Schumacher, D\. Naik, and A\. KannanRare Disease Differential Diagnosis with Large Language Models at Scale: From Abdominal Actinomycosis to Wilson’s Disease\.arXiv preprint arXiv:2502\.15069\.Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.8),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.8.1.1.1.3)\.
- Sounderajahet al\.\(2025\)V\. Sounderajah, A\. Guni, X\. Liu, G\. S\. Collins, A\. Karthikesalingam, S\. R\. Markar, R\. M\. Golub, A\. K\. Denniston, S\. Shetty, D\. Moher, P\. M\. Bossuyt, A\. Darzi, H\. Ashrafian, A\. Acharya, B\. A\. Mateen, C\. Kelly, D\. Ting, D\. Treanor, D\. King, F\. Greaves, H\. Harvey, J\. De Fauw, J\. F\. Cohen, J\. Godwin, J\. Pearson\-Stuttard, K\. Moons, L\. Harling, L\. Maier\-Hein, L\. Hooft, M\. D\. McInnes, M\. McCradden, N\. Rifai, N\. Tomasev, P\. Normahani, P\. Whiting, R\. Aggarwal, S\. Vollmer, S\. R\. Markar, T\. Panch, Ben Glocker, D\. Taylor, D\. Moher, D\. Samuel, J\. Ordish, K\. Singh, L\. Celi, P\. Bossuyt, S\. Rose, S\. Saria, STARD\-AI Steering Committee, and STARD\-AI Consensus GroupThe STARD\-AI reporting guideline for diagnostic accuracy studies using artificial intelligence\.Nature Medicine31\(10\),pp\. 3283–3289\.External Links:ISSN 1546\-170X,[Document](https://dx.doi.org/10.1038/s41591-025-03953-8)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p5.1.3),[§2\.7](https://arxiv.org/html/2610.33592#S2.SS7.p2.1.2)\.
- Starket al\.\(2017\)Z\. Stark, H\. Dashnow, S\. Lunke, T\. Y\. Tan, A\. Yeung, S\. Sadedin, N\. Thorne, I\. Macciocca, C\. Gaff, A\. Oshlack, S\. M\. White, P\. A\. James, and Melbourne Genomics Health AllianceA clinically driven variant prioritization framework outperforms purely computational approaches for the diagnostic analysis of singleton WES data\.European Journal of Human Genetics25\(11\),pp\. 1268–1272\.External Links:ISSN 1476\-5438,[Document](https://dx.doi.org/10.1038/ejhg.2017.123)Cited by:[1st item](https://arxiv.org/html/2610.33592#S2.I1.i1.p1.1.9),[§4\.2](https://arxiv.org/html/2610.33592#S4.SS2.p1.1.7)\.
- The Lancet Global Health \(2024\)The Lancet Global HealthThe landscape for rare diseases in 2024\.The Lancet Global Health12\(3\),pp\. e341\.External Links:ISSN 2214\-109X,[Document](https://dx.doi.org/10.1016/S2214-109X%2824%2900056-1)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.1)\.
- Töpelet al\.\(2006\)T\. Töpel, R\. Hofestädt, D\. Scheible, and F\. TrefzRAMEDIS: the rare metabolic diseases database\.Applied Bioinformatics5\(2\),pp\. 115–118\.External Links:ISSN 1175\-5636,[Document](https://dx.doi.org/10.2165/00822942-200605020-00006)Cited by:[2nd item](https://arxiv.org/html/2610.33592#S2.I1.i2.p1.1.2),[§4\.2](https://arxiv.org/html/2610.33592#S4.SS2.p1.1.9)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. u\. Kaiser, and I\. PolosukhinAttention is all you need\.Proceedings of the 31st International Conference on Neural Information Processing Systems30,pp\. 5998–6008\.Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.7)\.
- Wakapet al\.\(2020\)S\. N\. Wakap, D\. M\. Lambert, A\. Olry, C\. Rodwell, C\. Gueydan, V\. Lanneau, D\. Murphy, Y\. L\. Cam, and A\. RathEstimating cumulative point prevalence of rare diseases: analysis of the Orphanet database\.European Journal of Human Genetics28\(2\),pp\. 165–173\.External Links:ISSN 14765438,[Document](https://dx.doi.org/10.1038/s41431-019-0508-0)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.3)\.
- Wanget al\.\(2026\)Z\. Wang, D\. Wu, Q\. M\. Nguyen, Z\. Xu, and K\. WangMultimodal integrated knowledge transfer to large language models through preference optimization with biomedical applications\.Patterns7\(8\),pp\. 101544\.External Links:ISSN 2666\-3899,[Document](https://dx.doi.org/10.1016/j.patter.2026.101544)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.8),[§2\.3](https://arxiv.org/html/2610.33592#S2.SS3.p4.1.1),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.4.1.1.1.3)\.
- Weinreichet al\.\(2008\)S\. S\. Weinreich, R\. Mangon, J\. J\. Sikkens, M\. E\. e\. Teeuw, and M\. C\. CornelOrphanet: a European database for rare diseases\.Nederlands tijdschrift voor geneeskunde152\(9\),pp\. 518–519\.External Links:ISSN 0028\-2162Cited by:[§2\.5](https://arxiv.org/html/2610.33592#S2.SS5.p1.1.1),[§4\.1](https://arxiv.org/html/2610.33592#S4.SS1.p1.1.2)\.
- Wenet al\.\(2025\)B\. Wen, S\. Shi, Y\. Long, Y\. Dang, and W\. TianPhenoDP: leveraging deep learning for phenotype\-based case reporting, disease ranking, and symptom recommendation\.Genome Medicine17\(1\),pp\. 67\.External Links:ISSN 1756\-994X,[Document](https://dx.doi.org/10.1186/s13073-025-01496-8)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.6),[1st item](https://arxiv.org/html/2610.33592#S2.I1.i1.p1.1.4),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.13.1.1.1.3),[§4\.2](https://arxiv.org/html/2610.33592#S4.SS2.p1.1.5)\.
- Yanget al\.\(2023\)J\. Yang, L\. Shu, H\. Duan, and H\. LiA robust phenotype\-driven likelihood ratio analysis approach assisting interpretable clinical diagnosis of rare diseases\.Journal of Biomedical Informatics142,pp\. 104372\.External Links:ISSN 1532\-0464,[Document](https://dx.doi.org/10.1016/j.jbi.2023.104372)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.5),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.14.1.1.1.3),[§3](https://arxiv.org/html/2610.33592#S3.p3.1.2)\.
- Yanget al\.\(2025a\)J\. Yang, L\. Shu, H\. Duan, and H\. LiRDguru: A Conversational Intelligent Agent for Rare Diseases\.IEEE Journal of Biomedical and Health Informatics29\(9\),pp\. 6366–6378\.External Links:[Document](https://dx.doi.org/10.1109/JBHI.2024.3464555)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.9),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.10.1.1.1.3)\.
- Yanget al\.\(2024\)J\. Yang, L\. Shu, M\. Han, J\. Pan, L\. Chen, T\. Yuan, L\. Tan, Q\. Shu, H\. Duan, and H\. LiRDmaster: A novel phenotype\-oriented dialogue system supporting differential diagnosis of rare disease\.Computers in Biology and Medicine169,pp\. 107924\.External Links:ISSN 0010\-4825,[Document](https://dx.doi.org/10.1016/j.compbiomed.2024.107924)Cited by:[1st item](https://arxiv.org/html/2610.33592#S2.I1.i1.p1.1.5),[§4\.2](https://arxiv.org/html/2610.33592#S4.SS2.p1.1.4)\.
- Yang \(2026\)J\. YangFusionRare’s challenge cases\.Zenodo\.Note:[https://doi\.org/10\.5281/zenodo\.20697776](https://doi.org/10.5281/zenodo.20697776)External Links:[Document](https://dx.doi.org/10.5281/zenodo.20697776),[Link](https://doi.org/10.5281/zenodo.20697776)Cited by:[§3](https://arxiv.org/html/2610.33592#S3.p4.1.2)\.
- Yanget al\.\(2025b\)T\. Yang, D\. Huang, Y\. Lin, P\. Wu, Z\. Wu, G\. Ma, Y\. Lu, X\. Dong, D\. Li, J\. Ge, Z\. Zhang, X\. Huang, W\. Nong, Y\. Zhou, H\. Tang, H\. Yang, S\. Zhang, J\. Li, X\. Cao, L\. Yang, X\. Gao, K\. Xu, X\. Gu, W\. Zhang, H\. Xia, L\. Liu, W\. Zhou, and M\. J\. LiA Specialized Large Language Model for Clinical Reasoning and Diagnosis in Rare Diseases\.arXiv preprint arXiv:2511\.14638\.Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.11),[§1](https://arxiv.org/html/2610.33592#S1.p1.1.8),[§2\.3](https://arxiv.org/html/2610.33592#S2.SS3.p4.1.2),[§2\.3](https://arxiv.org/html/2610.33592#S2.SS3.p4.1.3),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.5.1.1.1.3)\.
- Zhaoet al\.\(2026\)W\. Zhao, C\. Wu, Y\. Fan, P\. Qiu, X\. Zhang, Y\. Sun, X\. Zhou, S\. Zhang, Y\. Peng, Y\. Wang, X\. Sun, Y\. Zhang, Y\. Yu, K\. Sun, and W\. XieAn agentic system for rare disease diagnosis with traceable reasoning\.Nature651\(8106\),pp\. 775–784\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-025-10097-9)Cited by:[§1](https://arxiv.org/html/2610.33592#S1.p1.1.10),[§1](https://arxiv.org/html/2610.33592#S1.p1.1.9),[§1](https://arxiv.org/html/2610.33592#S1.p4.1.1),[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.2.1.1.1.3),[§3](https://arxiv.org/html/2610.33592#S3.p4.1.1)\.
- Zhaoet al\.\(2023\)Z\. Zhao, Q\. Jin, F\. Chen, T\. Peng, and S\. YuA large\-scale dataset of patient summaries for retrieval\-based clinical decision support systems\.Scientific Data10\(1\),pp\. 909\.External Links:ISSN 2052\-4463,[Document](https://dx.doi.org/10.1038/s41597-023-02814-8)Cited by:[§3](https://arxiv.org/html/2610.33592#S3.p2.1.4)\.
- Zhenget al\.\(2025\)Q\. Zheng, Y\. Sun, C\. Wu, W\. Zhao, P\. Qiu, Y\. Yu, K\. Sun, J\. Zhang, Y\. Wang, Y\. Zhang, and W\. XieEnd\-to\-End Agentic RAG System Training for Traceable Diagnostic Reasoning\.arXiv preprint arXiv:2508\.15746\.Cited by:[Table 2](https://arxiv.org/html/2610.33592#S2.T2.4.6.1.1.1.3)\.