One Score, Two Decisions: Selective Prediction on the Rare-Disease Tail

# Selective Prediction on the Rare-Disease Tail
Source: [https://arxiv.org/html/2608.14683](https://arxiv.org/html/2608.14683)
## One Score, Two Decisions: Selective Prediction on the Rare\-Disease Tail

Zhaoyang Jiang1Zhizhong Fu3Yunsoo Kim1Zicheng Li4 Xuanqi Peng1Fei Teng1Jiacong Mi2Honghan Wu1 1School of Health & Wellbeing, University of Glasgow, Glasgow, UK 2Department of Respiratory and Critical Care Medicine, Shanghai Sixth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China 3School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China 3167645J@student\.gla\.ac\.uk, Honghan\.Wu@glasgow\.ac\.uk

###### Abstract

Given a patient’s clinical findings, a diagnostic system ranks possible diseases and must decide when to endorse its first prediction or defer it for review\. This decision is usually made by thresholding the top score\. Selective prediction over ranked outputs begins with two checks\. First, the ranker must produce enough correct top\-ranked predictions to make the target feasible\. Across2,0002\{,\}000patient records stratified by disease prevalence, eight small open\-weight LLMs achieve at most4\.6%4\.6\\%Recall@1 on ultra\-rare diseases\. At10%10\\%coverage, even a perfect confidence ranking of their existing predictions therefore cannot reach50%50\\%selective accuracy\. More accurate models pass the same check, showing that the limit is regime\-specific\. Second, the confidence signal must match the decision being made\. For fixed\-candidate rankers, the top\-two margin cancels components shared across candidates\. On phenotype\-only Exomiser, it selects10%10\\%of cases at29\.0%29\.0\\%accuracy, compared with13\.3%13\.3\\%overall, while the top score provides no reliable gate\. Yet that cancellation can remove information needed to detect whether the candidate list contains an answer\. SciFact retrieval and biomedical entity linking confirm this distinction\. Finally, we prove that unlabelled scores alone cannot determine whether switching to the margin will help\.

One Score, Two Decisions: Selective Prediction on the Rare\-Disease Tail

Zhaoyang Jiang1Zhizhong Fu3Yunsoo Kim1Zicheng Li4Xuanqi Peng1Fei Teng1Jiacong Mi2Honghan Wu1††thanks:Corresponding author\.1School of Health & Wellbeing, University of Glasgow, Glasgow, UK2Department of Respiratory and Critical Care Medicine, Shanghai Sixth People’s Hospital,Shanghai Jiao Tong University School of Medicine, Shanghai, China3School of Life Science and Technology,University of Electronic Science and Technology of China, Chengdu, China3167645J@student\.gla\.ac\.uk, Honghan\.Wu@glasgow\.ac\.uk,

![Refer to caption](https://arxiv.org/html/2608.14683v1/x1.png)Figure 1:Two of the paper’s three claims; the first, feasibility, is Eq\. \([1](https://arxiv.org/html/2608.14683#S3.E1)\) and §[5\.2](https://arxiv.org/html/2608.14683#S5.SS2)\.\(a\)Eq\. \([3](https://arxiv.org/html/2608.14683#S3.E3)\) splits each score into a level shared by every candidate for that case and a candidate\-specific residual\. We find correctness read from the gap the residuals leave between the leaders, while presence needs case\-level information that the gap discards and that, by Eq\. \([5](https://arxiv.org/html/2608.14683#S3.E5)\), no zero\-sum contrast over the candidates can recover \(§[5\.6](https://arxiv.org/html/2608.14683#S5.SS6)\)\.\(b\)Two joint laws with the same unlabelled scores and the same base accuracy give gains of opposite sign, so no functional of those scores identifies which gate is better \(Prop\.[1](https://arxiv.org/html/2608.14683#Thmproposition1)\)\.## 1Introduction

Rare\-disease diagnosis is a ranking problem\. Given a patient’s phenotypes \(structured findings such as seizures or short stature\), a diagnostic system must order thousands of possible diseases\. Large language models \(LLMs\) can turn that short input into a broad differential diagnosis\(Singhalet al\.,[2023](https://arxiv.org/html/2608.14683#bib.bib6); Kanjeeet al\.,[2023](https://arxiv.org/html/2608.14683#bib.bib5); Chenet al\.,[2024b](https://arxiv.org/html/2608.14683#bib.bib3)\), which makes them attractive on the ultra\-rare tail, where diagnostic support is needed most\.

A differential only helps if the clinician knows when to trust the system’s first suggestion\. Selective prediction handles this by endorsing the top prediction on high\-confidence cases and deferring the rest for review\(Chow,[1970](https://arxiv.org/html/2608.14683#bib.bib36); El\-Yaniv and Wiener,[2010](https://arxiv.org/html/2608.14683#bib.bib37); Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2608.14683#bib.bib21)\), which in our setting withholds endorsement of the first candidate without hiding the remaining differential: a confident error leads to unnecessary testing and counselling, while excessive abstention makes the system unhelpful\. The same top score is routinely thresholded for two different decisions, whether the top candidate is correct and whether any correct candidate is present\(Hendrycks and Gimpel,[2017](https://arxiv.org/html/2608.14683#bib.bib64); Sevgiliet al\.,[2022](https://arxiv.org/html/2608.14683#bib.bib50)\)\. We show that the two need different checks and different signals\.

The first check is feasibility\. If a predictor is correct on a fractionppof cases and the system answers a fractioncc, even a perfect confidence ranking cannot reach selective accuracy abovemin⁡\(1,p/c\)\\min\(1,p/c\)\. A target above that ceiling cannot be reached by recalibrating or rescoring the same predictions, because confidence can reorder correct answers but cannot create them\. The check matters on distributional tails, where pooled accuracy hides a much lower tail accuracy\(Ransohoff and Feinstein,[1978](https://arxiv.org/html/2608.14683#bib.bib22)\)\.

For predictors that pass, the signal depends on the decision\. A top score can carry a case\-level component shared by every candidate, and the gap to the runner\-up removes it\. That helps in deciding whether the leader is correct, since the gap measures its separation from the nearest alternative\. But the shared level is what indicates whether the candidate set fits the input at all, as when a retriever holds no relevant document or a mention has no entry in the knowledge base, so one subtraction can help correctness and hurt presence\. Nor does any other comparison among the candidates recover it: every zero\-sum contrast cancels the level exactly, so a system that needs it must look outside the contest\. Figure[1](https://arxiv.org/html/2608.14683#S0.F1)summarises the distinction and §[3](https://arxiv.org/html/2608.14683#S3)states it formally\.

We test this on a uniform2,0002\{,\}000\-case sample of10,37410\{,\}374patient records, read by the Orphanet prevalence class of each diagnosis\. Across eight small open\-weight LLMs ultra\-rare Recall@1 is at most4\.6%4\.6\\%, so at10%10\\%coverage even perfect confidence is capped at46%46\\%\. Two of five medical\-specialised models clear a50%50\\%target at the point estimate and across their whole interval, and the three frontier configurations reach15\.815\.8–22\.4%22\.4\\%, making the ceiling non\-binding, so the limit is specific to low\-accuracy predictors rather than to confidence\.

In that regime phenotype rankers, which compare a patient’s findings with curated disease profiles, are the stronger starting point under our leave\-source\-out control\. On phenotype\-only Exomiser the top score yields no usable operating point while the gap selects10%10\\%of ultra\-rare cases at29\.0%29\.0\\%against a13\.3%13\.3\\%base rate\. Subtracting the shared component from a retriever we can recompute lifts its top\-score gate, with a post\-hoc temperature, to the level of its gap, testing the mechanism rather than inferring it\. The gap also beats the raw score for correctness on SciFact retrieval and on entity linking with an unnormalized scorer\. Masking relevant candidates then separates the two: on SciFact the gap becomes the worst of three presence signals, and on entity linking it is never the best\. Finally, we prove that unlabelled scores, even at fixed base accuracy, cannot determine the gain from switching\.

## 2Related Work

#### Selective prediction and deferral\.

Calibration and uncertainty estimation sharpen the confidence attached to a*fixed*set of top\-1 predictions\(Xinet al\.,[2021](https://arxiv.org/html/2608.14683#bib.bib48); Kamathet al\.,[2020](https://arxiv.org/html/2608.14683#bib.bib24); Wenet al\.,[2025](https://arxiv.org/html/2608.14683#bib.bib27); Guoet al\.,[2017](https://arxiv.org/html/2608.14683#bib.bib20); Ovadiaet al\.,[2019](https://arxiv.org/html/2608.14683#bib.bib23); Farquharet al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib29); Kuhnet al\.,[2023](https://arxiv.org/html/2608.14683#bib.bib30); Angelopoulos and Bates,[2023](https://arxiv.org/html/2608.14683#bib.bib31); Varshneyet al\.,[2022](https://arxiv.org/html/2608.14683#bib.bib49)\): they select among existing answers without changing them, so what they reach is bounded by Eq\. \([1](https://arxiv.org/html/2608.14683#S3.E1)\)\. Deferral, cascades and adaptive retrieval work in the other regime, routing a case to a second predictor and changing the base accuracy itself\(Mozannar and Sontag,[2020](https://arxiv.org/html/2608.14683#bib.bib67); Jitkrittumet al\.,[2023](https://arxiv.org/html/2608.14683#bib.bib66); Jeonget al\.,[2024b](https://arxiv.org/html/2608.14683#bib.bib52); Niet al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib40); Fenget al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib47)\)\. We read the oracle ceiling\(Rabanseret al\.,[2023](https://arxiv.org/html/2608.14683#bib.bib81)\)as the test that tells the regimes apart\.

#### Confidence for ranked outputs\.

The top score, the gap between the leaders and trust scores over local geometry are established signals\(Jianget al\.,[2018](https://arxiv.org/html/2608.14683#bib.bib65); Schefferet al\.,[2001](https://arxiv.org/html/2608.14683#bib.bib41); Dalitz,[2009](https://arxiv.org/html/2608.14683#bib.bib42); Lowe,[2004](https://arxiv.org/html/2608.14683#bib.bib59); Lianget al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib51)\), as is subtracting a per\-input nuisance: by query length, an impostor cohort, the number of query terms in phenotype\-driven diagnosis, or a normalization that makes scores comparable across queries\(Zhou and Croft,[2007](https://arxiv.org/html/2608.14683#bib.bib60); Shtoket al\.,[2012](https://arxiv.org/html/2608.14683#bib.bib61); Auckenthaleret al\.,[2000](https://arxiv.org/html/2608.14683#bib.bib63); Schulzet al\.,[2011](https://arxiv.org/html/2608.14683#bib.bib62); Karakoset al\.,[2020](https://arxiv.org/html/2608.14683#bib.bib82); Bogolinet al\.,[2022](https://arxiv.org/html/2608.14683#bib.bib87)\)\. All of it removes the shared level*to rank better*; our question is which decision still needs it, which Eq\. \([3](https://arxiv.org/html/2608.14683#S3.E3)\) makes testable\. ConcurrentlyWagner \([2026](https://arxiv.org/html/2608.14683#bib.bib85)\)separate correctness from answerability with two internal LLM signals, andWanget al\.\([2025](https://arxiv.org/html/2608.14683#bib.bib86)\)gate retrieval on a top\-two margin under a monotone link §[3\.3](https://arxiv.org/html/2608.14683#S3.SS3)shows cannot be chosen from unlabelled scores; we study the split in the scores a ranker already emits\.

#### Long\-tail knowledge\.

That literature defines rarity by corpus frequency\(Kandpalet al\.,[2023](https://arxiv.org/html/2608.14683#bib.bib1); Razeghiet al\.,[2022](https://arxiv.org/html/2608.14683#bib.bib2); Mallenet al\.,[2023](https://arxiv.org/html/2608.14683#bib.bib38); Niet al\.,[2025](https://arxiv.org/html/2608.14683#bib.bib39); Sunet al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib53)\); we stratify by Orphanet prevalence\. Holding corpus frequency fixed leaves the per\-disease gap essentially intact, on too few matched pairs to remove the exposure explanation rather than bound it \(App\.[C\.2](https://arxiv.org/html/2608.14683#A3.SS2)\)\.

#### Rare\-disease diagnosis\.

Phenotype prioritizers rank diseases against curated profiles\(Köhleret al\.,[2009](https://arxiv.org/html/2608.14683#bib.bib7); Smedleyet al\.,[2015](https://arxiv.org/html/2608.14683#bib.bib8); Robinsonet al\.,[2020](https://arxiv.org/html/2608.14683#bib.bib9); Jagadeeshet al\.,[2019](https://arxiv.org/html/2608.14683#bib.bib10)\); general\-purpose LLMs trail them\(Reeseet al\.,[2026](https://arxiv.org/html/2608.14683#bib.bib4)\), and specialised or agentic systems add training, retrieval and tools without prevalence\-stratified or decontaminated evaluation\(Yanget al\.,[2025b](https://arxiv.org/html/2608.14683#bib.bib28); Zhaoet al\.,[2026](https://arxiv.org/html/2608.14683#bib.bib26)\)\. A meta\-analysis calls for the prevalence\-stratified design we adopt\(Nguyenet al\.,[2026](https://arxiv.org/html/2608.14683#bib.bib34)\)\.Elmofty and Leser \([2026](https://arxiv.org/html/2608.14683#bib.bib84)\)find the crossover turning on retrieval*coverage*, whether the answer enters the pool at all; ourppasks whether the predictor already places it first, which is what decides the feasible operating points \(App\.[B](https://arxiv.org/html/2608.14683#A2)\)\.

## 3Selective Prediction over Ranked Outputs

Selective prediction is usually posed as a choice of confidence score\. For ranked outputs that question comes too early: one must first ask whether the predictor has produced enough correct top\-ranked answers to reach the operating point, and then what the gate is meant to certify, that the leading candidate is correct or that the candidate set contains a valid answer at all\. Three steps follow: a feasibility test, a decision\-specific decomposition of the scores, and a limit on what unlabelled scores can settle\.

### 3\.1Feasibility Before Confidence Estimation

ConsiderNNcases\. Letyi=1y\_\{i\}\\\!=\\\!1if the predictor’s top\-ranked answer is correct on caseiiand0otherwise, so its base accuracy isp=1N​∑iyip=\\frac\{1\}\{N\}\\sum\_\{i\}y\_\{i\}\. A confidence ruleqqselects an answered setAqA\_\{q\}with realized coveragec=\|Aq\|/Nc=\|A\_\{q\}\|/N, including any effect of tied scores, and selective accuracyAccsel⁡\(q,c\)=1\|Aq\|​∑i∈Aqyi\\operatorname\{Acc\}\_\{\\mathrm\{sel\}\}\(q,c\)=\\frac\{1\}\{\|A\_\{q\}\|\}\\sum\_\{i\\in A\_\{q\}\}y\_\{i\}\. The answered set cannot contain more correct predictions than exist in the whole sample, nor more than it has cases,

∑i∈Aqyi≤min⁡\(\|Aq\|,∑i=1Nyi\),\\sum\_\{i\\in A\_\{q\}\}y\_\{i\}\\;\\leq\\;\\min\\Big\(\|A\_\{q\}\|,\\ \\textstyle\\sum\_\{i=1\}^\{N\}y\_\{i\}\\Big\),which on dividing by\|Aq\|\|A\_\{q\}\|gives

Accsel⁡\(q,c\)≤min⁡\(1,pc\)\.\\operatorname\{Acc\}\_\{\\mathrm\{sel\}\}\(q,c\)\\;\\leq\\;\\min\\\!\\left\(1,\\frac\{p\}\{c\}\\right\)\.\(1\)This is the standard oracle ceiling behind selective prediction\(Chow,[1970](https://arxiv.org/html/2608.14683#bib.bib36); El\-Yaniv and Wiener,[2010](https://arxiv.org/html/2608.14683#bib.bib37); Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2608.14683#bib.bib21); Rabanseret al\.,[2023](https://arxiv.org/html/2608.14683#bib.bib81)\), used here as a feasibility test rather than claimed as a new bound\. If deployment requires selective accuracyτ\\tauat coveragecc, Eq\. \([1](https://arxiv.org/html/2608.14683#S3.E1)\) implies the necessary condition

p≥τ​c\.p\\;\\geq\\;\\tau c\.\(2\)When it fails, no recalibration, uncertainty estimator or confidence score that leaves the predictor’s answers unchanged can reach the target, because the required correct answers do not exist to be selected\. The remedies are all of a different kind: a more accurate predictor, a larger returned candidate set, or a target the deployment can afford to relax\. When the condition holds the target is merely possible, since the gate must still place the correct cases first\. For a top\-kkdecision the same argument applies withyiy\_\{i\}redefined as whether the gold answer appears in the topkk, soppbecomes Recall@kk\. We read the condition across a range ofτ\\tau\(Whiteheadet al\.,[2022](https://arxiv.org/html/2608.14683#bib.bib54)\), labelling it atτ=50%\\tau\{=\}50\\%where a single point helps\(Vickers and Elkin,[2006](https://arxiv.org/html/2608.14683#bib.bib55)\)\.

### 3\.2Correctness and Candidate Presence

Now let one scorer evaluate a fixed candidate setDD\. For a casexxletd1​\(x\)=arg⁡maxd∈D⁡s​\(d∣x\)d\_\{1\}\(x\)=\\arg\\max\_\{d\\in D\}s\(d\\mid x\)be the leading candidate andd2​\(x\)d\_\{2\}\(x\)the runner\-up, with scoress1​\(x\)s\_\{1\}\(x\)ands2​\(x\)s\_\{2\}\(x\)\. Writingrel⁡\(d,x\)=1\\operatorname\{rel\}\(d,x\)\\\!=\\\!1when candidateddis correct forxx, two labels arise:

ycorr​\(x\)\\displaystyle y\_\{\\mathrm\{corr\}\}\(x\)=rel⁡\(d1​\(x\),x\),\\displaystyle=\\operatorname\{rel\}\\big\(d\_\{1\}\(x\),x\\big\),ypres​\(x\)\\displaystyle y\_\{\\mathrm\{pres\}\}\(x\)=maxd∈D⁡rel⁡\(d,x\)\.\\displaystyle=\\max\\nolimits\_\{d\\in D\}\\operatorname\{rel\}\(d,x\)\.The first asks whether the leader should be endorsed, the second whether the candidate set is usable at all\. Correctness implies presence but not conversely: a retriever can hold a relevant document without ranking it first, and a linker can rank confidently when the right entity is absent from its knowledge base\. Theppof §[3\.1](https://arxiv.org/html/2608.14683#S3.SS1)is the base rate ofycorry\_\{\\mathrm\{corr\}\}\. Both decisions are usually gated by the same number, the top scoreqtop​\(x\)=s1​\(x\)q\_\{\\mathrm\{top\}\}\(x\)=s\_\{1\}\(x\), and that becomes consequential when scores carry a component shared by every candidate for the same case\. Suppose

s​\(d∣x\)=b​\(x\)\+r​\(d∣x\),s\(d\\mid x\)=b\(x\)\+r\(d\\mid x\),\(3\)withb​\(x\)b\(x\)a case\-level offset andr​\(d∣x\)r\(d\\mid x\)what distinguishes candidates\. The top score retains both,qtop​\(x\)=b​\(x\)\+r1​\(x\)q\_\{\\mathrm\{top\}\}\(x\)=b\(x\)\+r\_\{1\}\(x\), whereas the marginqmargin​\(x\)=s1​\(x\)−s2​\(x\)q\_\{\\mathrm\{margin\}\}\(x\)=s\_\{1\}\(x\)\-s\_\{2\}\(x\)removes the shared part:

qmargin​\(x\)=r1​\(x\)−r2​\(x\)\.q\_\{\\mathrm\{margin\}\}\(x\)=r\_\{1\}\(x\)\-r\_\{2\}\(x\)\.\(4\)Nothing here is special to the top two\. For any weightswwoverDDsumming to zero,

∑d∈Dwd​s​\(d∣x\)=∑d∈Dwd​r​\(d∣x\),\\sum\_\{d\\in D\}w\_\{d\}\\,s\(d\\mid x\)\\;=\\;\\sum\_\{d\\in D\}w\_\{d\}\\,r\(d\\mid x\),\(5\)so*every*zero\-sum contrast among one case’s candidates cancelsb​\(x\)b\(x\)identically\. The weights may depend on the candidates’ rank order, whichb​\(x\)b\(x\)does not change, so this covers the margin,w=\(1,−1,0,…,0\)w=\(1,\-1,0,\\dots,0\), and equally the statistic one would reach for to*estimate*the case level, the top score against the mean of the field,w=\(1,−1K−1,…,−1K−1\)w=\(1,\-\\tfrac\{1\}\{K\-1\},\\dots,\-\\tfrac\{1\}\{K\-1\}\)\. The top score of course still carriesb​\(x\)b\(x\), but carries it added tor1​\(x\)r\_\{1\}\(x\), and no comparison among the candidates can separate the two\. Isolating a level that is comparable across cases therefore needs something the contest does not contain: a scale already fixed across cases, as a bounded cosine has, or a case\-side estimate of the offset, for which §[5\.6](https://arxiv.org/html/2608.14683#S5.SS6)uses a query’s own IDF mass\.

Splittingb​\(x\)=ν​\(x\)\+e​\(x\)b\(x\)=\\nu\(x\)\+e\(x\)turns this into a decision\-specific hypothesis rather than a universal ordering: a nuisanceν\\nu, moving for reasons unrelated to whether any candidate is correct, and evidenceeeabout whether one is\. A contrast discards both, which should helpycorry\_\{\\mathrm\{corr\}\}and hurtypresy\_\{\\mathrm\{pres\}\}; a case\-side correction discardsν\\nuand keepsee, so it should help both; the raw score keeps both, and can serveypresy\_\{\\mathrm\{pres\}\}only whereν\\nuis small\. Candidate\-dependent noise survives every subtraction\. Which regime a given scorer is in is an empirical question rather than a consequence of the algebra, and §[5\.5](https://arxiv.org/html/2608.14683#S5.SS5)settles it for one system by removingb​\(x\)b\(x\)from it\.

The same point has a probabilistic reading\. If the scores are unnormalized log\-potentials, so thatPT​\(d∣x\)∝exp⁡\(s​\(d∣x\)/T\)P\_\{T\}\(d\\mid x\)\\propto\\exp\(s\(d\\mid x\)/T\), then

s1​\(x\)−s2​\(x\)=T​log⁡PT​\(d1∣x\)PT​\(d2∣x\),s\_\{1\}\(x\)\-s\_\{2\}\(x\)=T\\log\\frac\{P\_\{T\}\(d\_\{1\}\\mid x\)\}\{P\_\{T\}\(d\_\{2\}\\mid x\)\},\(6\)the log\-odds of the leader against its nearest competitor up to temperature: the normalizer and every candidate\-independent offset cancel, as in conditional logit and partial likelihood\(McFadden,[1974](https://arxiv.org/html/2608.14683#bib.bib56); Cox,[1972](https://arxiv.org/html/2608.14683#bib.bib57); Lianget al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib51); Heng and Soh,[2026](https://arxiv.org/html/2608.14683#bib.bib58)\)\. Unders​\(d∣x\)↦α​s​\(d∣x\)\+β​\(x\)s\(d\\mid x\)\\mapsto\\alpha s\(d\\mid x\)\+\\beta\(x\)withα\>0\\alpha\>0the top score moves with the arbitrary case\-levelβ​\(x\)\\beta\(x\)while the margin keeps its ordering of cases, and so its risk–coverage curve, unchanged \(App\.[B\.1](https://arxiv.org/html/2608.14683#A2.SS1)\)\.

### 3\.3Limits of Label\-Free Gate Selection

WriteG​\(c\)=Accsel⁡\(qmargin,c\)−Accsel⁡\(qtop,c\)G\(c\)=\\operatorname\{Acc\}\_\{\\mathrm\{sel\}\}\(q\_\{\\mathrm\{margin\}\},c\)\-\\operatorname\{Acc\}\_\{\\mathrm\{sel\}\}\(q\_\{\\mathrm\{top\}\},c\)for the gain at coveragecc\. One would like to estimateGGbefore switching, from the scores alone\. That is not a matter of finding the right diagnostic\.

###### Proposition 1\(Unidentifiability from unlabelled scores\)\.

LetPSP\_\{S\}be the law of the candidate scores on unlabelled inputs andTTany functional ofPSP\_\{S\}\. Two joint laws can agree onPSP\_\{S\}and on base accuracy whileG​\(12\)G\(\\tfrac\{1\}\{2\}\)takes opposite signs; no suchTTidentifiesGG\.

###### Proof\.

LetU,VU,Vbe independent and uniform on the unit interval and put\(s1,s2\)=\(U,U−V\)\(s\_\{1\},s\_\{2\}\)=\(U,\\,U\-V\), sos1≥s2s\_\{1\}\\\!\\geq\\\!s\_\{2\}, the margin ism=Vm=V, andmmis independent ofs1s\_\{1\}\. Both laws use these scores, soPSP\_\{S\}andT​\(PS\)T\(P\_\{S\}\)agree\. Let the gold candidate bed1d\_\{1\}exactly whenV\>12V\>\\tfrac\{1\}\{2\}underP\+P\_\{\+\}and exactly whenU\>12U\>\\tfrac\{1\}\{2\}underP−P\_\{\-\}; base accuracy is12\\tfrac\{1\}\{2\}in both\. Atc=12c=\\tfrac\{1\}\{2\}the margin selects with accuracy11underP\+P\_\{\+\}and the top score with12\\tfrac\{1\}\{2\}, soG=\+12G=\+\\tfrac\{1\}\{2\}; underP−P\_\{\-\}the two exchange andG=−12G=\-\\tfrac\{1\}\{2\}\. ∎

Rank\-only statistics are the special case, since candidate order is a functional ofPSP\_\{S\}, as a monotone map shows directly: a strictly increasingfffixes every ranking whilef​\(s1\)−f​\(s2\)f\(s\_\{1\}\)\-f\(s\_\{2\}\)reorders cases\(Stevens,[1946](https://arxiv.org/html/2608.14683#bib.bib73); Wang and Shah,[2018](https://arxiv.org/html/2608.14683#bib.bib75)\)\. The quantifier is the unlabelled score distribution and no wider\. A method that reads the inputs, the candidate texts or the scorer itself, or that intervenes on scores rather than observing them, is outside the statement, which is why §[5\.5](https://arxiv.org/html/2608.14683#S5.SS5)’s mechanism experiment, an intervention, is not forbidden by it\. What is ruled out is reading the answer off the scores a deployed system already returns\.

The margin carries a domain condition of its own: it is meaningful when one predictor scores a common candidate set and the top two are distinct alternatives on a common scale\. Repeated samples from a free\-form generator are not that: the two highest\-scoring samples are usually the identical string, leaving no runner\-up at all \(App\.[B\.6](https://arxiv.org/html/2608.14683#A2.SS6)\)\.

The two results order the questions\. Eq\. \([2](https://arxiv.org/html/2608.14683#S3.E2)\) asks whether a predictor can reach the required operating point at all, and is settled before any confidence estimator is compared\. Eq\. \([3](https://arxiv.org/html/2608.14683#S3.E3)\) then motivates which part each decision should read, a structural hypothesis we test rather than assume, and Prop\.[1](https://arxiv.org/html/2608.14683#Thmproposition1)withholds the size of the gain\.

## 4Data and Experimental Setup

Our benchmark is Phenopacket Store v0\.1\.27\(Jacobsenet al\.,[2022](https://arxiv.org/html/2608.14683#bib.bib11); Daniset al\.,[2025](https://arxiv.org/html/2608.14683#bib.bib12)\), which collects10,37410\{,\}374real patient cases, each a set of observed Human Phenotype Ontology terms\(Köhleret al\.,[2021](https://arxiv.org/html/2608.14683#bib.bib13)\)with the patient’s confirmed OMIM diagnosis\. The780780diagnoses that appear are the evaluation labels; the retriever ranks all8,5538\{,\}553OMIM entries with an HPOA profile, the fixed jointly scored set the margin needs\. App\.[A](https://arxiv.org/html/2608.14683#A1)glosses the clinical\-genetics terms used below\.

To stratify by rarity we map each gold OMIM entry to its Orphanet prevalence class\(Nguengang Wakapet al\.,[2020](https://arxiv.org/html/2608.14683#bib.bib14)\), taking the rarest when an entry carries several\. That gives4,7804\{,\}780ultra\-rare cases \(below1/1061/10^\{6\}, or11–9/1069/10^\{6\}\) and1,1671\{,\}167less\-rare ones \(at least1/1051/10^\{5\}\); a third group whose prevalence Orphanet does not document is analysed separately, only372372of the780780diseases carrying a documented class\. The collapse survives a point\-prevalence rule and a finer four\-band split \(App\.[C\.1](https://arxiv.org/html/2608.14683#A3.SS1)\)\. Every model sees the present HPO terms and returns a ranked top\-5 differential with a verbalized confidence per entry, as free text rather than a selection from a list\.

We test four classes of system, named individually in Table[1](https://arxiv.org/html/2608.14683#S5.T1)\. The first is the deployable regime the collapse claim is about: eight small open\-weight models a hospital can run on\-premises, from the Qwen\(Baiet al\.,[2025](https://arxiv.org/html/2608.14683#bib.bib17); Yanget al\.,[2025a](https://arxiv.org/html/2608.14683#bib.bib16)\), Llama\(Grattafioriet al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib33)\), Mistral\(Labraket al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib18)\), InternLM and Yi families\. The second is medical specialisation: five models\(Jeonget al\.,[2024a](https://arxiv.org/html/2608.14683#bib.bib77); Chenet al\.,[2024a](https://arxiv.org/html/2608.14683#bib.bib78); Sellergrenet al\.,[2025](https://arxiv.org/html/2608.14683#bib.bib79); Douet al\.,[2025](https://arxiv.org/html/2608.14683#bib.bib80)\)at the same prompt, decoding and linker, two of them controlled contrasts, OpenBioLLM\-8B against our Llama\-3\.1\-8B row at fixed base model and OpenBioLLM\-70B against its own88B sibling at fixed recipe \(App\.[K](https://arxiv.org/html/2608.14683#A11)\)\. The collapse claim is stated for the eight\-model baseline group and not for these\. The third is a frontier generality check, DeepSeek\-V4\-Flash/Pro\(Xuet al\.,[2026](https://arxiv.org/html/2608.14683#bib.bib32)\)at284284B/1\.61\.6T parameters \(1313B/4949B active\), open\-weight but API\-served and so off\-regime, with reasoning disabled to isolate scale and enabled to add test\-time compute\. The fourth comprises two phenotype rankers: our own information\-content overlap retriever, which scores all8,5538\{,\}553profiled OMIM entries jointly, and Exomiser\(Smedleyet al\.,[2015](https://arxiv.org/html/2608.14683#bib.bib8)\)in phenotype\-only mode on the same cases, a tool clinicians run, which ranks genes rather than diseases \(App\.[F\.1](https://arxiv.org/html/2608.14683#A6.SS1)\)\. Open models are decoded greedily with vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2608.14683#bib.bib19)\); compute, versions and seeds are in App\.[L](https://arxiv.org/html/2608.14683#A12)\.

The rare\-disease benchmark always contains the gold diagnosis, so it can ask whether the top\-ranked answer is correct but not whether any correct candidate is present\. Two standard ranking tasks supply the second decision\. On SciFact\(Waddenet al\.,[2020](https://arxiv.org/html/2608.14683#bib.bib88)\)the released ColBERTv2 checkpoint\(Santhanamet al\.,[2022](https://arxiv.org/html/2608.14683#bib.bib89)\)ranks5,1835\{,\}183abstracts for300300claims; on BC5CDR\(Liet al\.,[2016](https://arxiv.org/html/2608.14683#bib.bib90)\)and MedMentions\(Mohan and Li,[2019](https://arxiv.org/html/2608.14683#bib.bib91)\)SapBERT and BM25\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2608.14683#bib.bib92)\)rank ontology entries for a mention\. Masking a random slice of the corpus or ontology then leaves some inputs with no valid candidate\(Zhuet al\.,[2023](https://arxiv.org/html/2608.14683#bib.bib83)\), so one run scores both decisions off the same scores; rates and preprocessing are in App\.[I](https://arxiv.org/html/2608.14683#A9)\.

Generated disease names are linked to OMIM with SapBERT\(Liuet al\.,[2021](https://arxiv.org/html/2608.14683#bib.bib15)\); the phenotype rankers return identifiers and use no linker, so the tail comparison does not rest on one\. We report Recall@1/@5; selective accuracy as a function of coverage; AUROC for top\-1 correctness and, on the transfer tasks, for candidate presence; and ECE per bin for the verbalized confidence signals\. App\.[H](https://arxiv.org/html/2608.14683#A8)audits the linker on held\-out surface forms\.

## 5Results

*Convention\.*Numbers below are per\-case \(micro\); per\-disease averaging shrinks the collapse to∼2\\sim\\\!2–6×6\\timesand preserves the tail contrast, established once in App\.[C\.2](https://arxiv.org/html/2608.14683#A3.SS2)\.

### 5\.1Performance across Prevalence Strata

Table[1](https://arxiv.org/html/2608.14683#S5.T1)shows the collapse across eight small open models spanning five families: Recall@1 falls from a∼40%\\sim\\\!40\\%less\-rare anchor to≤5%\\leq\\\!5\\%on the ultra\-rare tail, and it is not Qwen\-specific, the two sharpest ratios being Mistral’s and InternLM’s\. Five medical\-specialised models split22over the feasibility bar of §[5\.2](https://arxiv.org/html/2608.14683#S5.SS2)and33under it, the best reaching7\.6%7\.6\\%, and the one controlled pair we have, OpenBioLLM\-8B against our Llama\-3\.1\-8B row, is−1\.0\-1\.0pp \(App\.[K](https://arxiv.org/html/2608.14683#A11)\)\.

Scale and reasoning move the tail substantially without closing it: DeepSeek\-V4\-Pro reaches18\.6%18\.6\\%and reasoning22\.4%22\.4\\%, enough to lift the feasibility ceiling \(§[5\.2](https://arxiv.org/html/2608.14683#S5.SS2)\) though still below the retriever’s25\.6%25\.6\\%\(§[5\.3](https://arxiv.org/html/2608.14683#S5.SS3)\) and not resolvably so\. A paired disease\-clustered test separates the retriever from77of the1010configurations we can recompute, and the three it does not are exactly the frontier ones, trailing by3\.13\.1to9\.89\.8pp with intervals containing zero\. Against the small models the gap is decisive and in triage \(§[5\.2](https://arxiv.org/html/2608.14683#S5.SS2)\)\. The reasoning columns also need a common denominator: on the849849cases both configurations returned the reasoning gain is\+1\.9\+1\.9pp, half of what the printed columns imply \(App\.[K](https://arxiv.org/html/2608.14683#A11)\)\.

Recall@1 \(%\)ModelFamilyLess\-rareUltra\-rareLess/ultra ratio*small open\-weight models*Qwen2\.5\-VL\-7BQwen36\.51\.721×21\\timesQwen2\.5\-14BQwen40\.54\.29\.8×9\.8\\timesQwen2\.5\-VL\-32BQwen40\.64\.39\.4×9\.4\\timesLlama\-3\.1\-8BLlama39\.24\.68\.6×8\.6\\timesMistral\-7B\-v0\.3Mistral40\.11\.625×25\\timesBioMistral\-7BMistral36\.02\.415×15\\timesInternLM2\.5\-7BInternLM32\.91\.424×24\\timesYi\-1\.5\-9BYi7\.20\.79\.7×9\.7\\times*medical\-specialised models \(on\-premises,88B–7070B\)*OpenBioLLM\-8BLlama36\.53\.610×10\\timesHuatuoGPT\-o1\-8BLlama35\.64\.28×8\\timesMedGemma\-27BGemma37\.43\.710×10\\timesBaichuan\-M2\-32BBaichuan46\.97\.66×6\\timesOpenBioLLM\-70BLlama40\.56\.76×6\\times*frontier MoE models \(API\)*DeepSeek\-V4\-FlashDeepSeek40\.515\.82\.6×2\.6\\timesDeepSeek\-V4\-Pro \(reason\. off\)DeepSeek48\.218\.62\.6×2\.6\\timesDeepSeek\-V4\-Pro \(reason\. on\)DeepSeek49\.522\.42\.2×2\.2\\times*phenotype prioritizers*IC\-overlap retriever \(ours\)–31\.425\.61\.2×1\.2\\timesExomiser \(phenotype\-only\)–21\.213\.31\.6×1\.6\\timesTable 1:Per\-case Recall@1 \(%\) by disease\-prevalence stratum, every system on the same HPO input\. Every generative row and Exomiser are scored on one uniform2,0002\{,\}000\-case sample \(939939ultra\-rare\); the retriever needs no generation and is scored on all10,34510\{,\}345eligible cases, returning the same25\.6%25\.6\\%tail value on the sample \(App\.[F\.2](https://arxiv.org/html/2608.14683#A6.SS2)\)\. Two rows have their own denominator: reasoning\-on returned849849of the939939ultra\-rare cases and Qwen2\.5\-VL\-32B901901, and the dropped cases are the harder ones\. Retriever scores use the leave\-source\-out control; Exomiser ranks genes and the rest diseases \(App\.[K](https://arxiv.org/html/2608.14683#A11)\)\.
### 5\.2Feasibility Before Calibration

Stated confidence does not warn a clinician\. It stays badly*over*confident on the tail while its discrimination runs from near chance to0\.900\.90AUROC for Qwen\-32B \(App\.[E\.2](https://arxiv.org/html/2608.14683#A5.SS2)\), which is what makes it dangerous rather than merely poor\.

Good discrimination still does not deliver the specified operating point \(Eq\. \([1](https://arxiv.org/html/2608.14683#S3.E1)\)\), and this cuts two ways\. Atc=10%c\{=\}10\\%the ceiling is simplyp/cp/c, which for the eight small models runs from7%7\\%to46%46\\%, with three of the five medical models inside that band and two above it\. Any target above46%46\\%is out of reach for those eleven however well their confidence discriminates, and Qwen\-32B, the best of them at ranking its own answers, is capped at43%43\\%\. A conformal selector cannot repair the shortfall either: choosing among the same top\-1 predictions, it is subject to the same counting bound \(App\.[E\.1](https://arxiv.org/html/2608.14683#A5.SS1);Hanselleet al\.,[2025](https://arxiv.org/html/2608.14683#bib.bib74)\)\.

At the frontier the ceiling lifts and the second direction weakens: a tail base rate of15\.815\.8–22\.4%22\.4\\%takes the bound to≈1\.0\\approx\\\!1\.0, which makes the target attainable rather than showing it is attained\. It is: at matched10%10\\%coverage the model’s own confidence and the external gate are not separable on the three frontier configurations \(\+2\.1\+2\.1pp to the external one\)\. The test earns its keep by disqualifying predictors rather than endorsing them \(App\.[E\.1](https://arxiv.org/html/2608.14683#A5.SS1)\)\.

Table 2:Selective accuracy \(%\) on the ultra\-rare stratum for four of the1616generative configurations in Table[1](https://arxiv.org/html/2608.14683#S5.T1)\. Parentheses on the second row give the coverage each LLM’s own confidence actually resolves, which is not10%10\\%; the other two rows are at10%10\\%, and the coverage\-matched comparison is in App\.[E\.2](https://arxiv.org/html/2608.14683#A5.SS2)\. The oracle row ismin⁡\(1,p/c\)\\min\(1,p/c\)\. The retriever row varies across columns only because each model returned parseable output on a slightly different case set \(App\.[K](https://arxiv.org/html/2608.14683#A11)\)\.
### 5\.3Phenotype Retrieval on Ultra\-Rare Cases

Is the tail simply hard? A classical information\-content\-weighted phenotype\-overlap retriever, given the same HPO terms, reaches25\.6%25\.6\\%on the ultra\-rare tail after removing same\-source curation leakage: we drop any gold term supported*only*by the case’s own source publication \(App\.[D\.1](https://arxiv.org/html/2608.14683#A4.SS1)\)\. The tail is therefore not uniformly hard: a ranker built directly on phenotype–disease compatibility keeps far more accuracy there than the LLM, and the ordering survives per\-disease averaging\. Neither is accurate enough to diagnose autonomously, which is why the deployable contribution is triage rather than an accuracy chase\.

The retriever’s absolute accuracy is only*partially identified*: curation is shared between the records and the knowledge base, and a stricter same\-publication control gives a sensitivity range of\[5\.2%,25\.6%\]\[5\.2\\%,25\.6\\%\], where the ordering holds against the five weaker small models but not the frontier\. What is stable across it is the lift over the gate’s own base rate,2\.42\.4–3\.3×3\.3\\times: a triage lift, not a certified operating point, and three external cohorts corroborate its upper half \(App\.[D\.1](https://arxiv.org/html/2608.14683#A4.SS1),[D\.2](https://arxiv.org/html/2608.14683#A4.SS2)\)\.

### 5\.4Top Scores versus Top\-Two Margins

The claim is about how a score is built, not about medicine, so it should hold on a clinical prioritizer and a text retriever alike; we take the clinical ones here and the text ones in §[5\.6](https://arxiv.org/html/2608.14683#S5.SS6)\.

Exomiser\(Smedleyet al\.,[2015](https://arxiv.org/html/2608.14683#bib.bib8)\)shows it on a tool clinicians run\. Its ultra\-rare base Recall@1 is13\.3%13\.3\\%overn=939n\{=\}939cases, and the gap between its first two candidates, from the same output, selects a10%10\\%band at29\.0%29\.0\\%\(disease\-clustered17\.717\.7–42\.7%42\.7\\%\)\. Read its top score and no band can be drawn: the decile falls inside one tie block, so its accuracy moves over\[0\.0,4\.3\]%\[0\.0,4\.3\]\\%on how ties are broken\. Neither that block nor the tool’s own shippedpp\-value accounts for the difference, so this is not a straw target \(App\.[F\.1](https://arxiv.org/html/2608.14683#A6.SS1), Fig\.[3](https://arxiv.org/html/2608.14683#A6.F3);Coopersteinet al\.,[2025](https://arxiv.org/html/2608.14683#bib.bib71); Vestitoet al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib72)\)\. Without variant data nothing breaks a tie once the phenotype score saturates, so we claim the dissociation for phenotype\-only prioritization and no further\.

Our own retriever separates the two further, gating at45\.8%45\.8\\%on its raw score against81\.0%81\.0\\%on its margin, and only the delivered system is a deployment claim \(Table[2](https://arxiv.org/html/2608.14683#S5.T2)\): triaged by that margin it answers the most confident10%10\\%of ultra\-rare cases at7474–81%81\\%, where the bare LLM at the coverage its own confidence resolves reaches22–18%18\\%\. The ordering survives simulated prospective phenotyping, where dropping the most informative terms costs the retriever’s band far less than it costs the LLM’s \(App\.[F\.2](https://arxiv.org/html/2608.14683#A6.SS2)\), and it is retriever\-first rather than hybrid because neither a heuristic combiner nor a validation\-trained one improves on the retriever alone \(App\.[G](https://arxiv.org/html/2608.14683#A7)\)\.

### 5\.5Testing the Shared\-Component Mechanism

Eq\. \([4](https://arxiv.org/html/2608.14683#S3.E4)\) is exact and carries no evidence on its own; the reverse manipulation does\. Subtracting an estimate ofb​\(x\)b\(x\)from our retriever’s scores lifts its raw gate from45\.8%45\.8\\%to80\.3%80\.3\\%, the margin’s own level, though only with a post\-hoc temperature and only for estimates read off the candidates in contention, input\-derived ones reaching at most70\.7%70\.7\\%\(App\.[B\.4](https://arxiv.org/html/2608.14683#A2.SS4)\)\. What no manipulation delivers is a label\-free rule for deciding in advance when the subtraction will help\.

LIRICAL\(Robinsonet al\.,[2020](https://arxiv.org/html/2608.14683#bib.bib9)\)shows why on a deployed tool\. Reading one ranking in the two units it ships moves the margin’s advantage from\+0\.056\+0\.056under the likelihood ratio to−0\.281\-0\.281under the post\-test probability: one monotone map, a swing of0\.3370\.337, and nothing in the scores to choose between them\. Across seven systems the advantage runs from\+0\.310\+0\.310to−0\.026\-0\.026, an ordering and not a uniform gain \(App\.[B\.2](https://arxiv.org/html/2608.14683#A2.SS2),[I](https://arxiv.org/html/2608.14683#A9)\)\.

The controlled version of that swing makes the rest concrete\. Fourteen units of*one*ranking, from the fusion literature’s normalizations to an injected per\-input level at five strengths, leave Recall@1 identical at77\.6%77\.6\\%and the margin’s AUROC inside0\.7380\.738–0\.8040\.804while the raw score’s runs0\.3070\.307–0\.8220\.822, a spread eight times as wide\. Under a purely additive shift the margin’s AUROC is unchanged to machine zero, as Eq\. \([4](https://arxiv.org/html/2608.14683#S3.E4)\) requires, and so is the top score against the mean of the field: the injection arms sweep the between\-input share from0\.2370\.237to0\.9880\.988and cost the raw score0\.2920\.292of AUROC while moving either contrast by at most3×10−63\\times 10^\{\-6\}\. The sweep therefore confirms the predicted invariance in an implemented scorer and shows the raw score’s collapse to be driven by exactly the component the contrasts remove\. How far the shared level tracks the gain across these arms is measurable, but only under an assumption about that level which Prop\.[1](https://arxiv.org/html/2608.14683#Thmproposition1)says the scores cannot certify \(App\.[B\.3](https://arxiv.org/html/2608.14683#A2.SS3)\)\.

### 5\.6Different Confidence Signals for Correctness and Candidate Presence

Concept normalization is the closest neighbour and its survey’s rules threshold the top\-1 score\(Sevgiliet al\.,[2022](https://arxiv.org/html/2608.14683#bib.bib50)\)\. On the12,74112\{,\}741BC5CDR mentions the two are not distinguishable \(−0\.027\-0\.027,\[−0\.075,\+0\.021\]\[\-0\.075,\+0\.021\]clustered on mention strings\), consistent withC=0\.011C\\\!=\\\!0\.011\. Swapping the scorer for BM25 raisesCCto0\.4410\.441on the same mentions and moves the margin from tied to ahead \(0\.7430\.743to0\.7930\.793,\[\+0\.017,\+0\.082\]\[\+0\.017,\+0\.082\]\), a crossover registered before running: the scorer’s normalization, not the presence of text, decides whether the correction is needed\.

Both runs ask only whether the top\-1 is right\. Deployed linkers must also decide whether to link at all, and Eq\. \([4](https://arxiv.org/html/2608.14683#S3.E4)\) sends the two decisions to opposite parts of one score: correctness to the*difference*, an out\-of\-base mention to the*level*it cancels\. Masking a quarter of MEDIC makes25\.8%25\.8\\%of mentions unlinkable and asks both\. Our recommendation is bounded here: under SapBERT the margin is the worse NIL detector at every rate \(−0\.054\-0\.054,\[−0\.088,−0\.023\]\[\-0\.088,\-0\.023\]\), and on MedMentions it is best in none of six cells\. The field’s rule is bounded by the same term: reading presence off the raw score works only where that score is normalized, and under BM25 it must be corrected from outside the candidate set\. The query’s own IDF mass does it \(\+0\.109\+0\.109,\[\+0\.053,\+0\.163\]\[\+0\.053,\+0\.163\]\), while the contrast that estimates the same level from the contenders is the worst of four statistics at every rate \(0\.6360\.636–0\.6570\.657against0\.7670\.767–0\.7750\.775\), behind even the raw score at the heaviest masking\.

The split is not a property of short\-mention linking\. On SciFact the same reading holds for correctness, ColBERTv2 gating at0\.7640\.764by its own MaxSim against0\.8360\.836by its margin, with a case\-level share ofC=0\.374C\\\!=\\\!0\.374that is not query length\. Under the same masking protocol, at every rate that margin is the*worst*of three for whether a relevant abstract survives \(0\.6470\.647–0\.6960\.696against0\.6960\.696–0\.7380\.738\) while beating its raw score on top\-1 correctness \(\+0\.043\+0\.043to\+0\.080\+0\.080\)\. That is the statistic adaptive\-RAG gates threshold to decide whether retrieval is needed at all\(Wanget al\.,[2025](https://arxiv.org/html/2608.14683#bib.bib86)\)\(App\.[I](https://arxiv.org/html/2608.14683#A9)\)\.

## 6Conclusion

Selective prediction cannot rescue a ranker that rarely places the correct answer first\. On ultra\-rare diseases, eight small open\-weight LLMs lack enough correct top\-ranked predictions to reach50%50\\%selective accuracy at10%10\\%coverage, regardless of calibration; more accurate medical\-specialised and frontier LLMs pass the same feasibility check\. Phenotype rankers provide a stronger starting point, but their scores expose another distinction\. For fixed candidate sets with unnormalized scores, the top\-two margin can better indicate whether the leader is correct by removing variation shared across candidates\. That removal can also discard information needed to tell whether the list contains an answer, as our SciFact and entity\-linking experiments show\. Selective prediction over ranked outputs should therefore proceed in order: first test whether base accuracy makes the target feasible, then choose a confidence signal for the decision being made\. Since unlabelled scores cannot reveal the gain from switching signals, the final choice requires labelled validation\.

## Limitations

Our thresholds are retrospective estimates, not guarantees for new patients\. Certifying80%80\\%accuracy at10%10\\%coverage would require about26,60026\{,\}600ultra\-rare cases\. Disease clustering reduces the effective sample further, and conformal calibration misses its promised error rate on diseases absent from the calibration data \(App\.[E\.2](https://arxiv.org/html/2608.14683#A5.SS2)\)\. Because clinical sites will encounter such diseases, the reported thresholds are not clinically certified\.

The feasibility result depends on the delivery rule\. We analyze top\-1 endorsement at a fixed accuracy target and coverage\. If success means that the correct diagnosis appears anywhere in the topkk, Recall@kkreplaces Recall@1 and the ceiling rises\. At50%50\\%accuracy and10%10\\%coverage, all eight small models are ruled out atk=1k\{=\}1, but only four atk=5k\{=\}5\(App\.[E\.3](https://arxiv.org/html/2608.14683#A5.SS3)\)\. Passing the test means only that the target is not ruled out, not that an available confidence score reaches it\. This changes which models fail the feasibility check, but not our model comparisons or margin analyses, which holdkkfixed\.

The retriever’s absolute accuracy remains uncertain because patient records and disease profiles share curation\. Leave\-source\-out removes direct same\-publication overlap but not broader curation effects, and our two controls yield a five\-fold range in base accuracy \(App\.[D\.1](https://arxiv.org/html/2608.14683#A4.SS1)\)\. The benchmark phenotypes were also curated after diagnosis and may be richer than prospective inputs; feature removal is only a proxy for that difference \(App\.[F\.2](https://arxiv.org/html/2608.14683#A6.SS2)\)\. The margin’s lift over the retriever’s own base rate survives these checks, but its absolute deployment accuracy is not established\. Finally, the mechanism is directly tested on one retriever; six other rankers provide supporting evidence but not the same causal test\. New systems still require labelled validation, which Prop\.[1](https://arxiv.org/html/2608.14683#Thmproposition1)shows the scores alone cannot substitute for\.

## Ethics Statement

A triage gate can harm\. On the answered decile one global rule delivers71\.0%71\.0\\%, disease\-clustered interval\[58\.8,79\.9\]\[58\.8,79\.9\], so about three in ten answered patients receive a confidently wrong top\-1 and four in ten is inside the planning range\. That is not a null event: it can trigger confirmatory testing, cascade testing of relatives and counselling\. Most of those errors still point at the right work\-up,73\.0%73\.0\\%at the causative gene, and we grade them rather than count them \(App\.[J](https://arxiv.org/html/2608.14683#A10)\)\. Grading is not reassurance: seven of the fifteen near\-misses pair a purely dominant entity with a purely recessive one, so the assay is right while the recurrence risk, the cascade\-testing targets and the reproductive counselling are wrong, and both gene\-level and disorder\-level grading score that as a near miss\. A third of the answered decile is of unknown prevalence, a group a clinic cannot identify before diagnosis and on which accuracy is lower still\. We intend the gate as a ranking aid for a specialist, never as an autonomous decision, which would also engage FDA SaMD and EU MDR\. The measurements above are ordinary selective prediction, where a deferred case is scored as unanswered\. The deployment we propose is weaker than that metric: a deferred patient still receives the differential the system would have produced anyway, without the confidence flag, so the gate annotates rather than gate\-keeps\. It does not follow that deferral costs nothing\. Withholding a model’s output from clinicians raises missed diagnoses relative to showing no model at all, because an absent prediction is read as evidence of absence rather than as neutrality\(Jabbouret al\.,[2025](https://arxiv.org/html/2608.14683#bib.bib69)\)\. An unflagged case is therefore not a null intervention, and how the absence of a flag is displayed is a design question prior to where the threshold sits\.

The gate is also not equally available, which is a documented hazard of selective classification rather than a quirk of ours\(Joneset al\.,[2021](https://arxiv.org/html/2608.14683#bib.bib46)\)\. Of the217217sampled patients with one or two recorded findings, none is ever flagged, against a flag rate of24\.6%24\.6\\%above fifteen terms, and yet the retriever is barely worse on them than on patients with eight to eleven terms,19\.4%19\.4\\%against24\.0%24\.0\\%\. What collapses is not whether the system helps those patients but whether they can qualify for the marker, because the margin grows with how many terms were recorded\. That closes the gate on patients at first presentation, those seen by generalists, and those in systems where deep phenotype coding is not routine\. App\.[B](https://arxiv.org/html/2608.14683#A2)measures what removing the dependence would cost,70\.7%70\.7\\%against81\.0%81\.0\\%, and App\.[F\.3](https://arxiv.org/html/2608.14683#A6.SS3)characterises the patients the gate does select: an efficiency and equity trade\-off we have measured but not resolved\.

All patient data is secondary use of de\-identified, already\-published records, and we use only structured HPO terms and gold labels\. We did not recruit or interact with patients, and obtaining consent was therefore not ours to do: whatever consent the original case reports rest on was obtained by their authors, outside this study, and the resources that aggregate them do not consistently record it\. We note that rather than treat public availability as consent\. Re\-identification risk is nonzero for any rare\-disease case report, since an unusual combination of findings can be distinctive, and prior publication does not remove it\. Releasing source identifiers makes those reports easier to locate, which is a risk we add rather than one we inherit; we judge it warranted for reproducibility, release no case\-report text or further patient attributes, and ask that the artifacts be used under the original sources’ terms\. App\.[D\.1](https://arxiv.org/html/2608.14683#A4.SS1)quantifies the concentration of curation in one contributor account,98\.7%98\.7\\%of the records, which is a statement about a corpus\-construction process and a limitation of our own estimates, not a criticism of work this paper wholly depends on\.

## References

- A\. N\. Angelopoulos and S\. Bates \(2023\)Conformal prediction: a gentle introduction\.Foundations and Trends in Machine Learning16\(4\),pp\. 494–591\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Auckenthaler, M\. Carey, and H\. Lloyd\-Thomas \(2000\)Score normalization for text\-independent speaker verification systems\.Digital Signal Processing10\(1\),pp\. 42–54\.External Links:ISSN 1051\-2004,[Document](https://dx.doi.org/https%3A//doi.org/10.1006/dspr.1999.0360),[Link](https://www.sciencedirect.com/science/article/pii/S1051200499903603)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p3.6)\.
- S\. Bogolin, I\. Croitoru, H\. Jin, Y\. Liu, and S\. Albanie \(2022\)Cross modal retrieval with querybank normalisation\.In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 5184–5195\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1)\.
- L\. F\. P\. Cattelan and D\. Silva \(2024\)How to fix a broken confidence estimator: evaluating post\-hoc methods for selective classification with deep neural networks\.InUncertainty in Artificial Intelligence \(UAI\),Note:arXiv:2305\.15508Cited by:[§B\.2](https://arxiv.org/html/2608.14683#A2.SS2.p3.1)\.
- H\. Chen, Z\. Zhao, S\. Zhou, S\. Hu, J\. Wang, Y\. Jin, X\. Jin, Y\. C\. Tham, X\. Wang, W\. Ma,et al\.\(2026\)RareArena: a comprehensive benchmark dataset unveiling the potential of large language models in rare disease diagnosis\.The Lancet Digital Health8\(2\)\.Cited by:[§C\.2](https://arxiv.org/html/2608.14683#A3.SS2.p3.8)\.
- J\. Chen, Z\. Cai, K\. Ji, X\. Wang, W\. Liu, R\. Wang, J\. Hou, and B\. Wang \(2024a\)Huatuogpt\-o1, towards medical complex reasoning with llms\.arXiv preprint arXiv:2412\.18925\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p3.6)\.
- X\. Chen, X\. Mao, Q\. Guo, L\. Wang, S\. Zhang, and T\. Chen \(2024b\)RareBench: can llms serve as rare diseases specialists?\.InProceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining,pp\. 4850–4861\.Cited by:[§D\.2](https://arxiv.org/html/2608.14683#A4.SS2.p2.2),[§1](https://arxiv.org/html/2608.14683#S1.p1.1)\.
- C\. Chow \(1970\)On optimum recognition error and reject tradeoff\.IEEE Transactions on Information Theory16\(1\),pp\. 41–46\.External Links:[Document](https://dx.doi.org/10.1109/TIT.1970.1054406)Cited by:[§1](https://arxiv.org/html/2608.14683#S1.p2.1),[§3\.1](https://arxiv.org/html/2608.14683#S3.SS1.p1.12)\.
- I\. B\. Cooperstein, S\. Marwaha, A\. Ward, S\. N\. Kobren, J\. N\. Carter, M\. T\. Wheeler, and G\. T\. Marth \(2025\)An optimized variant prioritization process for rare disease diagnostics: recommendations for exomiser and genomiser\.Genome Medicine17\(1\),pp\. 127\.Cited by:[§5\.4](https://arxiv.org/html/2608.14683#S5.SS4.p2.8)\.
- D\. R\. Cox \(1972\)Regression models and life\-tables\.Journal of the Royal Statistical Society: Series B34\(2\),pp\. 187–202\.External Links:[Document](https://dx.doi.org/10.1111/j.2517-6161.1972.tb00899.x)Cited by:[§B\.1](https://arxiv.org/html/2608.14683#A2.SS1.p3.1),[§3\.2](https://arxiv.org/html/2608.14683#S3.SS2.p3.4)\.
- C\. Dalitz \(2009\)Reject options and confidence measures for knn classifiers\.External Links:[Link](https://api.semanticscholar.org/CorpusID:17508606)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Danis, M\. J\. Bamshad, Y\. Bridges, A\. Caballero\-Oteyza, P\. Cacheiro, L\. C\. Carmody, L\. Chimirri, J\. X\. Chong, B\. Coleman, R\. Dalgleish,et al\.\(2025\)A corpus of ga4gh phenopackets: case\-level phenotyping for genomic diagnostics and discovery\.Human Genetics and Genomics Advances6\(1\)\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p1.3)\.
- C\. Dou, C\. Liu, F\. Yang, F\. Li, J\. Jia, M\. Chen, Q\. Ju, S\. Wang, S\. Dang, T\. Li,et al\.\(2025\)Baichuan\-m2: scaling medical capability with large verifier system\.arXiv preprint arXiv:2509\.02208\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p3.6)\.
- R\. El\-Yaniv and Y\. Wiener \(2010\)On the foundations of noise\-free selective classification\.Journal of Machine Learning Research11\(53\),pp\. 1605–1641\.External Links:[Link](http://jmlr.org/papers/v11/el-yaniv10a.html)Cited by:[§1](https://arxiv.org/html/2608.14683#S1.p2.1),[§3\.1](https://arxiv.org/html/2608.14683#S3.SS1.p1.12)\.
- M\. Elmofty and U\. Leser \(2026\)When does retrieval beat direct llm diagnosis in rare disease? an empirical study of ontology coverage\.InBioNLP 2026,pp\. 508–518\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal \(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630\(8017\),pp\. 625–630\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Feng, W\. Shi, Y\. Wang, W\. Ding, V\. Balachandran, and Y\. Tsvetkov \(2024\)Don’t hallucinate, abstain: identifying LLM knowledge gaps via multi\-LLM collaboration\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 14664–14690\.External Links:[Link](https://aclanthology.org/2024.acl-long.786/)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Geifman and R\. El\-Yaniv \(2017\)Selective classification for deep neural networks\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2608.14683#S1.p2.1),[§3\.1](https://arxiv.org/html/2608.14683#S3.SS1.p1.12)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p3.6)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InInternational conference on machine learning,pp\. 1321–1330\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Hanselle, A\. Javanmardi, T\. F\. Oberkofler, Y\. Sale, and E\. Hüllermeier \(2025\)Conformal prediction without nonconformity scores\.InThe 41st Conference on Uncertainty in Artificial Intelligence,Cited by:[§5\.2](https://arxiv.org/html/2608.14683#S5.SS2.p2.6)\.
- D\. Hendrycks and K\. Gimpel \(2017\)A baseline for detecting misclassified and out\-of\-distribution examples in neural networks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.14683#S1.p2.1)\.
- A\. Heng and H\. Soh \(2026\)Know when to abstain: optimal selective classification with likelihood ratios\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2505\.15008Cited by:[§B\.1](https://arxiv.org/html/2608.14683#A2.SS1.p3.1),[§B\.2](https://arxiv.org/html/2608.14683#A2.SS2.p3.1),[§3\.2](https://arxiv.org/html/2608.14683#S3.SS2.p3.4)\.
- J\. L\. Horowitz and C\. F\. Manski \(1995\)Identification and robustness with contaminated and corrupted data\.Econometrica63\(2\),pp\. 281–302\.External Links:ISSN 00129682, 14680262,[Link](http://www.jstor.org/stable/2951627)Cited by:[§D\.1](https://arxiv.org/html/2608.14683#A4.SS1.p3.7)\.
- S\. Jabbour, D\. Fouhey, N\. Banovic, S\. Shepard, E\. Kazerooni, M\. W\. Sjoding, and J\. Wiens \(2025\)On the limits of selective AI prediction: a case study in clinical decision making\.arXiv preprint arXiv:2508\.07617\.Cited by:[Ethics Statement](https://arxiv.org/html/2608.14683#Sx2.p1.3)\.
- J\. O\. Jacobsen, M\. Baudis, G\. S\. Baynam, J\. S\. Beckmann, S\. Beltran, O\. J\. Buske, T\. J\. Callahan, C\. G\. Chute, M\. Courtot, D\. Danis,et al\.\(2022\)The ga4gh phenopacket schema defines a computable representation of clinical data\.Nature biotechnology40\(6\),pp\. 817–820\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p1.3)\.
- K\. A\. Jagadeesh, J\. Birgmeier, H\. Guturu, C\. A\. Deisseroth, A\. M\. Wenger, J\. A\. Bernstein, and G\. Bejerano \(2019\)Phrank measures phenotype sets similarity to greatly improve mendelian diagnostic disease prioritization\.Genetics in Medicine21\(2\),pp\. 464–470\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px4.p1.1)\.
- D\. P\. Jeong, S\. Garg, Z\. C\. Lipton, and M\. Oberst \(2024a\)Medical adaptation of large language and vision\-language models: are we making progress?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 12143–12170\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p3.6)\.
- S\. Jeong, J\. Baek, S\. Cho, S\. J\. Hwang, and J\. Park \(2024b\)Adaptive\-RAG: learning to adapt retrieval\-augmented large language models through question complexity\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 7036–7050\.External Links:[Link](https://aclanthology.org/2024.naacl-long.389/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.389)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Jiang, B\. Kim, M\. Y\. Guan, and M\. Gupta \(2018\)To trust or not to trust a classifier\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1)\.
- W\. Jitkrittum, N\. Gupta, A\. K\. Menon, H\. Narasimhan, A\. S\. Rawat, and S\. Kumar \(2023\)When does confidence\-based cascade deferral suffice?\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Jones, S\. Sagawa, P\. W\. Koh, A\. Kumar, and P\. Liang \(2021\)Selective classification can magnify disparities across groups\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2010.14134)Cited by:[Ethics Statement](https://arxiv.org/html/2608.14683#Sx2.p2.6)\.
- A\. Kamath, R\. Jia, and P\. Liang \(2020\)Selective question answering under domain shift\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 5684–5696\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Kandpal, H\. Deng, A\. Roberts, E\. Wallace, and C\. Raffel \(2023\)Large language models struggle to learn long\-tail knowledge\.InInternational conference on machine learning,pp\. 15696–15707\.Cited by:[§C\.2](https://arxiv.org/html/2608.14683#A3.SS2.p3.8),[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Kanjee, B\. Crowe, and A\. Rodman \(2023\)Accuracy of a generative artificial intelligence model in a complex diagnostic challenge\.Jama330\(1\),pp\. 78–80\.Cited by:[§1](https://arxiv.org/html/2608.14683#S1.p1.1)\.
- D\. Karakos, R\. Zbib, W\. Hartmann, R\. Schwartz, and J\. Makhoul \(2020\)Reformulating information retrieval from speech and text as a detection problem\.InProceedings of the workshop on Cross\-Language Search and Summarization of Text and Speech \(CLSSTS2020\),pp\. 38–43\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Köhler, M\. Gargano, N\. Matentzoglu, L\. C\. Carmody, D\. Lewis\-Smith, N\. A\. Vasilevsky, D\. Danis, G\. Balagura, G\. Baynam, A\. M\. Brower,et al\.\(2021\)The human phenotype ontology in 2021\.Nucleic acids research49\(D1\),pp\. D1207–D1217\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p1.3)\.
- S\. Köhler, M\. H\. Schulz, P\. Krawitz, S\. Bauer, S\. Dölken, C\. E\. Ott, C\. Mundlos, D\. Horn, S\. Mundlos, and P\. N\. Robinson \(2009\)Clinical diagnostics in human genetics with semantic similarity searches in ontologies\.The American Journal of Human Genetics85\(4\),pp\. 457–464\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px4.p1.1)\.
- L\. Kuhn, Y\. Gal, and S\. Farquhar \(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.arXiv preprint arXiv:2302\.09664\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang,et al\.\(2023\)Efficient memory management for large language model serving with PagedAttention\.InACM Symposium on Operating Systems Principles \(SOSP\),Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p3.6)\.
- Y\. Labrak, A\. Bazoge, E\. Morin, P\. Gourraud, M\. Rouvier, and R\. Dufour \(2024\)Biomistral: a collection of open\-source pretrained large language models for medical domains\.InFindings of the association for computational linguistics: acl 2024,pp\. 5848–5864\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p3.6)\.
- J\. Li, Y\. Sun, R\. J\. Johnson, D\. Sciaky, C\. Wei, R\. Leaman, A\. P\. Davis, C\. J\. Mattingly, T\. C\. Wiegers, and Z\. Lu \(2016\)BioCreative v cdr task corpus: a resource for chemical disease relation extraction\.Database2016\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p4.2)\.
- H\. Liang, L\. Peng, and J\. Sun \(2024\)Selective classification under distribution shifts\.Transactions on Machine Learning Research\.Note:arXiv:2405\.05160Cited by:[§B\.1](https://arxiv.org/html/2608.14683#A2.SS1.p3.1),[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2608.14683#S3.SS2.p3.4)\.
- F\. Liu, E\. Shareghi, Z\. Meng, M\. Basaldella, and N\. Collier \(2021\)Self\-alignment pretraining for biomedical entity representations\.InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies,pp\. 4228–4238\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p5.1)\.
- D\. G\. Lowe \(2004\)Distinctive image features from scale\-invariant keypoints\.Int\. J\. Comput\. Vision60\(2\),pp\. 91–110\.External Links:ISSN 0920\-5691,[Link](https://doi.org/10.1023/B:VISI.0000029664.99615.94),[Document](https://dx.doi.org/10.1023/B%3AVISI.0000029664.99615.94)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Mallen, A\. Asai, V\. Zhong, R\. Das, D\. Khashabi, and H\. Hajishirzi \(2023\)When not to trust language models: investigating effectiveness of parametric and non\-parametric memories\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 9802–9822\.External Links:[Link](https://aclanthology.org/2023.acl-long.546/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.546)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px3.p1.1)\.
- M\. A\. Masten and A\. Poirier \(2020\)Inference on breakdown frontiers\.Quantitative Economics11\(1\),pp\. 41–111\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.3982/QE1288),[Link](https://onlinelibrary.wiley.com/doi/abs/10.3982/QE1288),https://onlinelibrary\.wiley\.com/doi/pdf/10\.3982/QE1288Cited by:[§D\.1](https://arxiv.org/html/2608.14683#A4.SS1.p3.7)\.
- D\. McFadden \(1974\)Conditional logit analysis of qualitative choice behavior\.InFrontiers in Econometrics,P\. Zarembka \(Ed\.\),pp\. 105–142\.Cited by:[§B\.1](https://arxiv.org/html/2608.14683#A2.SS1.p3.1),[§3\.2](https://arxiv.org/html/2608.14683#S3.SS2.p3.4)\.
- F\. McSherry and M\. Najork \(2008\)Computing information retrieval performance measures efficiently in the presence of tied scores\.InEuropean conference on information retrieval,pp\. 414–421\.Cited by:[§F\.1](https://arxiv.org/html/2608.14683#A6.SS1.p3.19)\.
- S\. Mohan and D\. Li \(2019\)Medmentions: a large biomedical corpus annotated with umls concepts\.arXiv preprint arXiv:1902\.09476\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p4.2)\.
- H\. Mozannar and D\. Sontag \(2020\)Consistent estimators for learning to defer to an expert\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Nguengang Wakap, D\. M\. Lambert, A\. Olry, C\. Rodwell, C\. Gueydan, V\. Lanneau, D\. Murphy, Y\. Le Cam, and A\. Rath \(2020\)Estimating cumulative point prevalence of rare diseases: analysis of the orphanet database\.European journal of human genetics28\(2\),pp\. 165–173\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p2.8)\.
- M\. Nguyen, C\. Yang, T\. A\. Cassini, F\. Ma, R\. Hamid, L\. Bastarache, J\. F\. Peterson, H\. Xu, L\. Li, S\. Ma, and C\. Shyr \(2026\)Diagnostic accuracy of large language models for rare diseases: a systematic review and meta\-analysis\.medRxiv\.External Links:[Document](https://dx.doi.org/10.64898/2026.03.26.26349194)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Ni, K\. Bi, J\. Guo, and X\. Cheng \(2024\)When do LLMs need retrieval augmentation? mitigating LLMs’ overconfidence helps retrieval augmentation\.InFindings of the Association for Computational Linguistics: ACL,Note:arXiv:2402\.11457Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Ni, K\. Bi, J\. Guo, and X\. Cheng \(2025\)How knowledge popularity influences and enhances LLM knowledge boundary perception\.arXiv preprint arXiv:2505\.17537\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Ovadia, E\. Fertig, J\. Ren, Z\. Nado, D\. Sculley, S\. Nowozin, J\. Dillon, B\. Lakshminarayanan, and J\. Snoek \(2019\)Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift\.Advances in neural information processing systems32\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Rabanser, A\. Thudi, A\. Guha Thakurta, K\. Dvijotham, and N\. Papernot \(2023\)Training private models that know what they don’t know\.Advances in Neural Information Processing Systems36,pp\. 53711–53727\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2608.14683#S3.SS1.p1.12)\.
- D\. F\. Ransohoff and A\. R\. Feinstein \(1978\)Problems of spectrum and bias in evaluating the efficacy of diagnostic tests\.New England Journal of Medicine299\(17\),pp\. 926–930\.Cited by:[§1](https://arxiv.org/html/2608.14683#S1.p3.3)\.
- Y\. Razeghi, R\. L\. Logan IV, M\. Gardner, and S\. Singh \(2022\)Impact of pretraining term frequencies on few\-shot numerical reasoning\.InFindings of the Association for Computational Linguistics: EMNLP 2022,pp\. 840–854\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px3.p1.1)\.
- J\. T\. Reese, L\. Chimirri, Y\. Bridges, D\. Danis, J\. H\. Caufield, M\. A\. Gargano, C\. Kroll, A\. Schmeder, F\. Liu, K\. Wissink,et al\.\(2026\)Systematic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare\-disease decision support tools\.European Journal of Human Genetics34\(4\),pp\. 498–504\.Cited by:[§F\.1](https://arxiv.org/html/2608.14683#A6.SS1.p1.1),[§F\.1](https://arxiv.org/html/2608.14683#A6.SS1.p2.13),[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Robertson and H\. Zaragoza \(2009\)The probabilistic relevance framework: bm25 and beyond\.Foundations and trends® in information retrieval4\(1\-2\),pp\. 1–174\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p4.2)\.
- P\. N\. Robinson, V\. Ravanmehr, J\. O\. Jacobsen, D\. Danis, X\. A\. Zhang, L\. C\. Carmody, M\. A\. Gargano, C\. L\. Thaxton, G\. Karlebach, J\. Reese,et al\.\(2020\)Interpretable clinical genomics with a likelihood ratio paradigm\.The American Journal of Human Genetics107\(3\),pp\. 403–417\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px4.p1.1),[§5\.5](https://arxiv.org/html/2608.14683#S5.SS5.p2.5)\.
- P\. R\. Rosenbaum \(2002\)Observational studies\.2nd edition,Springer\.Cited by:[§D\.1](https://arxiv.org/html/2608.14683#A4.SS1.p3.7)\.
- K\. Santhanam, O\. Khattab, J\. Saad\-Falcon, C\. Potts, and M\. Zaharia \(2022\)Colbertv2: effective and efficient retrieval via lightweight late interaction\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 3715–3734\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p4.2)\.
- T\. Scheffer, C\. Decomain, and S\. Wrobel \(2001\)Active hidden markov models for information extraction\.InInternational Symposium on Intelligent Data Analysis,External Links:[Link](https://api.semanticscholar.org/CorpusID:38833768)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1)\.
- M\. H\. Schulz, S\. Köhler, S\. Bauer, and P\. N\. Robinson \(2011\)Exact score distribution computation for ontological similarity searches\.BMC Bioinformatics12,pp\. 441 – 441\.External Links:[Link](https://api.semanticscholar.org/CorpusID:14050174)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Sellergren, S\. Kazemzadeh, T\. Jaroensri, A\. Kiraly, M\. Traverse, T\. Kohlberger, S\. Xu, F\. Jamil, C\. Hughes, C\. Lau,et al\.\(2025\)Medgemma technical report\.arXiv preprint arXiv:2507\.05201\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p3.6)\.
- Ö\. Sevgili, A\. Shelmanov, M\. Arkhipov, A\. Panchenko, and C\. Biemann \(2022\)Neural entity linking: a survey of models based on deep learning\.Semantic Web13\(3\),pp\. 527–570\.External Links:[Document](https://dx.doi.org/10.3233/SW-222986)Cited by:[§I\.1](https://arxiv.org/html/2608.14683#A9.SS1.p1.44),[§1](https://arxiv.org/html/2608.14683#S1.p2.1),[§5\.6](https://arxiv.org/html/2608.14683#S5.SS6.p1.9)\.
- A\. Shtok, O\. Kurland, D\. Carmel, F\. Raiber, and G\. Markovits \(2012\)Predicting query performance by query\-drift estimation\.ACM Trans\. Inf\. Syst\.30\(2\)\.External Links:ISSN 1046\-8188,[Link](https://doi.org/10.1145/2180868.2180873),[Document](https://dx.doi.org/10.1145/2180868.2180873)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Singhal, S\. Azizi, T\. Tu, S\. S\. Mahdavi, J\. Wei, H\. W\. Chung, N\. Scales, A\. Tanwani, H\. Cole\-Lewis, S\. Pfohl,et al\.\(2023\)Large language models encode clinical knowledge\.Nature620\(7972\),pp\. 172–180\.Cited by:[§1](https://arxiv.org/html/2608.14683#S1.p1.1)\.
- D\. Smedley, J\. O\. Jacobsen, M\. Jäger, S\. Köhler, M\. Holtgrewe, M\. Schubach, E\. Siragusa, T\. Zemojtel, O\. J\. Buske, N\. L\. Washington,et al\.\(2015\)Next\-generation diagnostics and disease\-gene discovery with the exomiser\.Nature protocols10\(12\),pp\. 2004–2015\.Cited by:[§F\.1](https://arxiv.org/html/2608.14683#A6.SS1.p1.1),[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2608.14683#S4.p3.6),[§5\.4](https://arxiv.org/html/2608.14683#S5.SS4.p2.8)\.
- S\. S\. Stevens \(1946\)On the theory of scales of measurement\.Science103\(2684\),pp\. 677–680\.Cited by:[§3\.3](https://arxiv.org/html/2608.14683#S3.SS3.p2.3)\.
- K\. Sun, Y\. Xu, H\. Zha, Y\. Liu, and X\. L\. Dong \(2024\)Head\-to\-tail: how knowledgeable are large language models \(LLMs\)? A\.K\.A\. will LLMs replace knowledge graphs?\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 311–325\.External Links:[Link](https://aclanthology.org/2024.naacl-long.18/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.18)Cited by:[§C\.2](https://arxiv.org/html/2608.14683#A3.SS2.p3.8),[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Traub, T\. J\. Bungert, C\. T\. Lüth, M\. Baumgartner, K\. H\. Maier\-Hein, L\. Maier\-Hein, and P\. F\. Jaeger \(2024\)Overcoming common flaws in the evaluation of selective classification systems\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§I\.4](https://arxiv.org/html/2608.14683#A9.SS4.p1.18)\.
- N\. Varshney, S\. Mishra, and C\. Baral \(2022\)Investigating selective prediction approaches across several tasks in IID, OOD, and adversarial settings\.InFindings of the Association for Computational Linguistics: ACL 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 1995–2002\.External Links:[Link](https://aclanthology.org/2022.findings-acl.158/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.158)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Vestito, J\. O\. Jacobsen, S\. Walker, V\. Cipriani, N\. L\. Harris, M\. A\. Haendel, C\. J\. Mungall, P\. Robinson, and D\. Smedley \(2024\)Efficient reinterpretation of rare disease cases using exomiser\.NPJ genomic medicine9\(1\),pp\. 65\.Cited by:[§5\.4](https://arxiv.org/html/2608.14683#S5.SS4.p2.8)\.
- A\. J\. Vickers and E\. B\. Elkin \(2006\)Decision curve analysis: a novel method for evaluating prediction models\.Medical Decision Making26\(6\),pp\. 565–574\.External Links:[Document](https://dx.doi.org/10.1177/0272989X06295361)Cited by:[§3\.1](https://arxiv.org/html/2608.14683#S3.SS1.p1.19)\.
- D\. Wadden, S\. Lin, K\. Lo, L\. L\. Wang, M\. van Zuylen, A\. Cohan, and H\. Hajishirzi \(2020\)Fact or fiction: verifying scientific claims\.InProceedings of the 2020 conference on empirical methods in natural language processing \(EMNLP\),pp\. 7534–7550\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p4.2)\.
- B\. J\. Wagner \(2026\)Two axes of llm abstention: answer correctness and question answerability\.arXiv preprint arXiv:2607\.08456\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Wang and N\. B\. Shah \(2018\)Your 2 is my 1, your 3 is my 9: handling arbitrary miscalibrations in ratings\.arXiv preprint arXiv:1806\.05085\.Cited by:[§3\.3](https://arxiv.org/html/2608.14683#S3.SS3.p2.3)\.
- Y\. Wang, H\. Ling,et al\.\(2025\)Retrieval as a decision: training\-free adaptive gating for efficient rag\.arXiv preprint arXiv:2511\.09803\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1),[§5\.6](https://arxiv.org/html/2608.14683#S5.SS6.p3.9)\.
- B\. Wen, J\. Yao, S\. Feng, C\. Xu, Y\. Tsvetkov, B\. Howe, and L\. L\. Wang \(2025\)Know your limits: a survey of abstention in large language models\.Transactions of the Association for Computational Linguistics13,pp\. 529–556\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Whitehead, S\. Petryk, V\. Shakib, J\. Gonzalez, T\. Darrell, A\. Rohrbach, and M\. Rohrbach \(2022\)Reliable visual question answering: abstain rather than answer incorrectly\.InEuropean Conference on Computer Vision \(ECCV\),Note:arXiv:2204\.13631Cited by:[§3\.1](https://arxiv.org/html/2608.14683#S3.SS1.p1.19)\.
- J\. Xin, R\. Tang, Y\. Yu, and J\. Lin \(2021\)The art of abstention: selective prediction and error regularization for natural language processing\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 1040–1051\.External Links:[Link](https://aclanthology.org/2021.acl-long.84/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.84)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling,et al\.\(2026\)Deepseek\-v4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p3.6)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025a\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4](https://arxiv.org/html/2608.14683#S4.p3.6)\.
- T\. Yang, D\. Huang, Y\. Lin, P\. Wu, Z\. Wu, G\. Ma, Y\. Lu, X\. Dong, D\. Li, J\. Ge,et al\.\(2025b\)A specialized large language model for clinical reasoning and diagnosis in rare diseases\.arXiv preprint arXiv:2511\.14638\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px4.p1.1)\.
- W\. Zhao, C\. Wu, Y\. Fan, P\. Qiu, X\. Zhang, Y\. Sun, X\. Zhou, S\. Zhang, Y\. Peng, Y\. Wang,et al\.\(2026\)An agentic system for rare disease diagnosis with traceable reasoning\.Nature651\(8106\),pp\. 775–784\.Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Zhou and W\. B\. Croft \(2007\)Query performance prediction in web search environments\.InProceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,SIGIR ’07,New York, NY, USA,pp\. 543–550\.External Links:ISBN 9781595935977,[Link](https://doi.org/10.1145/1277741.1277835),[Document](https://dx.doi.org/10.1145/1277741.1277835)Cited by:[§2](https://arxiv.org/html/2608.14683#S2.SS0.SSS0.Px2.p1.1)\.
- F\. Zhu, J\. Yu, H\. Jin, L\. Hou, J\. Li, and Z\. Sui \(2023\)Learn to not link: exploring nil prediction in entity linking\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 10846–10860\.Cited by:[§I\.2](https://arxiv.org/html/2608.14683#A9.SS2.p2.3),[§4](https://arxiv.org/html/2608.14683#S4.p4.2)\.

## Appendix ATerminology

We use several terms from clinical genetics and rare\-disease diagnosis\. The definitions below state how each term is used in this paper; they are not intended as complete clinical definitions\.

- •Phenotype\.An observable clinical feature of a patient, such as seizures or short stature, as opposed to the underlying genetic cause\.
- •Human Phenotype Ontology \(HPO\)\.A controlled vocabulary of approximately18,00018\{,\}000phenotypic abnormalities arranged in an*is\-a*hierarchy\. For example, “absence seizure” is a descendant of “seizure”\.
- •Phenopacket\.A machine\-readable patient record containing the HPO terms observed in that patient and the confirmed diagnosis\. We use the structured HPO terms as input; the original free\-text case report is not provided to the models\.
- •Differential diagnosis\.A ranked list of diseases that could explain a patient’s findings\. Our systems return a top\-5 differential, and Recall@1 and Recall@5 measure whether the confirmed diagnosis appears first or anywhere in that list\.
- •OMIM\.Online Mendelian Inheritance in Man, a reference catalogue of Mendelian diseases\. OMIM identifiers define the disease label space in our experiments: the retriever ranks8,5538\{,\}553candidate entries, and the evaluation set contains780780gold diseases\.
- •HPOA\.The annotation resource linking each OMIM disease to the HPO terms reported for it, together with the publications supporting those annotations\. HPOA is the entire knowledge base of our phenotype retriever\.
- •Orphanet prevalence\.An epidemiological classification of rare diseases, defined independently of any language\-model training corpus\. We use it to form the ultra\-rare, less\-rare and undocumented\-prevalence strata\.
- •Ultra\-rare\.The union of the two rarest Orphanet prevalence classes:<1<\\\!1case per million and11–99cases per million\.
- •Prioritizer\.A non\-LLM system that ranks candidate diseases or genes by their compatibility with the patient’s findings\. Exomiser is the clinically deployed example used in this paper\.
- •Phenotype\-only prioritization\.Disease or gene prioritization from HPO terms without genetic\-variant evidence\. Our Exomiser experiment uses this setting and should not be read as an evaluation of its full genome\-aware pipeline\.
- •Information content \(IC\)\.The negative logarithm of a phenotype term’s frequency across diseases\. Rare and specific terms receive more weight than common terms when the retriever compares a patient with a disease profile\.
- •True\-path propagation\.Adding the ancestors of each HPO term to a disease profile, so that a specific patient finding can match a disease annotated only with a broader ancestral term\.
- •Curation leakage\.Overlap created when a patient’s phenotypes and the corresponding disease profile were curated from the same publication or by a shared curation process\. Such overlap can inflate phenotype\-retrieval performance even though no case text is given to the retriever\.
- •Selective prediction\.A system answers only a subset of cases and defers the rest;*coverage*is the fraction answered and*selective accuracy*is accuracy on that subset \(§[3](https://arxiv.org/html/2608.14683#S3)\)\.

## Appendix BMargin Derivations and Mechanism Tests

The main text gives the operational argument for using the top\-two margin\. This appendix separates three claims that require different kinds of support\. First, the margin removes a candidate\-independent case\-level score shift; this is algebra\. Second, the gain from doing so cannot be identified from candidate ranks alone; this is a negative result with one decisive counterexample and one empirical refutation\. Third, invariance to the shift does not by itself guarantee that a confidence signal will work; this needs measurement\. We then test the proposed mechanism on our retriever and derive a scale\-specific prediction for Exomiser\.

### B\.1Score Representation and Invariance

Let a prioritizer assign scoress​\(d∣x\)s\(d\\mid x\)to candidatesd∈Dd\\in Dfor casexx\. Treating these as unnormalized log\-potentials induces

PT​\(d∣x\)=exp⁡\(s​\(d∣x\)/T\)ZT​\(x\),P\_\{T\}\(d\\mid x\)=\\frac\{\\exp\(s\(d\\mid x\)/T\)\}\{Z\_\{T\}\(x\)\},\(7\)with normalizerZT​\(x\)=∑d′∈Dexp⁡\(s​\(d′∣x\)/T\)Z\_\{T\}\(x\)=\\sum\_\{d^\{\\prime\}\\in D\}\\exp\(s\(d^\{\\prime\}\\mid x\)/T\), for a temperatureT\>0T\>0\. Withd1,d2d\_\{1\},d\_\{2\}the highest\- and second\-highest\-scoring candidates and scoress1,s2s\_\{1\},s\_\{2\}, their margin satisfies

s1−s2=T​log⁡PT​\(d1∣x\)PT​\(d2∣x\),s\_\{1\}\-s\_\{2\}=T\\log\\frac\{P\_\{T\}\(d\_\{1\}\\mid x\)\}\{P\_\{T\}\(d\_\{2\}\\mid x\)\},\(8\)or equivalentlyPT​\(d1∣x,\{d1,d2\}\)=σ​\(\(s1−s2\)/T\)P\_\{T\}\(d\_\{1\}\\mid x,\\\{d\_\{1\},d\_\{2\}\\\}\)=\\sigma\\\!\\left\(\(s\_\{1\}\-s\_\{2\}\)/T\\right\)after restricting to those two candidates\. The margin is thus proportional to the log\-odds of the leader against its closest competitor, and the normalizerZT​\(x\)Z\_\{T\}\(x\)cancels\.

Now consider

s​\(d∣x\)⟼α​s​\(d∣x\)\+β​\(x\),α\>0,s\(d\\mid x\)\\longmapsto\\alpha s\(d\\mid x\)\+\\beta\(x\),\\qquad\\alpha\>0,\(9\)withα\\alphaa global rescaling andβ​\(x\)\\beta\(x\)an arbitrary case\-level offset shared by all candidates\. Thens1↦α​s1\+β​\(x\)s\_\{1\}\\mapsto\\alpha s\_\{1\}\+\\beta\(x\)whiles1−s2↦α​\(s1−s2\)s\_\{1\}\-s\_\{2\}\\mapsto\\alpha\(s\_\{1\}\-s\_\{2\}\)\. The top score moves withβ​\(x\)\\beta\(x\)and may reorder cases; the margin removesβ​\(x\)\\beta\(x\), and the remaining positive factor changes neither its ordering of cases nor its risk–coverage curve\.

This is the conditioning argument behind conditional logit\(McFadden,[1974](https://arxiv.org/html/2608.14683#bib.bib56)\)and partial likelihood\(Cox,[1972](https://arxiv.org/html/2608.14683#bib.bib57)\), and related invariance arguments motivate margin\-based selective classification\(Lianget al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib51)\)and likelihood\-ratio rejection rules\(Heng and Soh,[2026](https://arxiv.org/html/2608.14683#bib.bib58)\)\. Our contribution is not the algebra but its use in identifying the failure of top\-score gating in the ultra\-rare regime\.

### B\.2A Deployed Instance, and the Limits of Our Evidence

Proposition[1](https://arxiv.org/html/2608.14683#Thmproposition1)bounds what the unlabelled score distribution can determine, and rank statistics are its special case\. LIRICAL supplies a deployed witness rather than a constructed one\. It reports both a composite likelihood ratio and a post\-test probability for one identical ranking of one identical case set, and the pretest probability is a single value for every disease and case, so the second is a fixed monotone map of the first\. The margin’s advantage nonetheless moves from\+0\.056\+0\.056to−0\.281\-0\.281, a swing of0\.3370\.337\. The map is destructive because it saturates: composite likelihood ratios run past102910^\{29\}, so the posterior is pinned at the top of its range on79\.8%79\.8\\%of cases and its top\-two difference is numerically zero on28\.0%28\.0\\%, while every rank is preserved\. Dropping the77cases where the map stops being strictly increasing in double precision leaves the swing at0\.3430\.343, so saturation is the whole of the effect\. That single swing is as large as the entire spread of the gain across our seven systems, which is also0\.3370\.337; the two are distinct quantities that happen to agree\.

Those seven systems are less independent than the count suggests, which is whyCCis reported as a measured property and no ordering is drawn from them: LIRICAL and Exomiser run on the same939939cases, the two entity linkers on the same mentions, and MaxSim SUM and MaxSim MEAN are one ranking in two units, the second being the first divided by query length and agreeing on every top\-11, itself a small instance of the same point, since that one ranking has gains of\+0\.178\+0\.178and\+0\.077\+0\.077\.

The gap this addresses is real even so\. Post\-hoc normalization of a broken confidence estimator is known to repair selective classification on vision classifiers, but the repair is fitted on labelled data and offers no way to tell in advance which model needs it\(Cattelan and Silva,[2024](https://arxiv.org/html/2608.14683#bib.bib70)\); likelihood\-ratio analysis gives the condition under which a top\-two statistic is optimal, that the distribution concentrate on the leading pair, but not a test of it\(Heng and Soh,[2026](https://arxiv.org/html/2608.14683#bib.bib58)\)\. Calibrating a label\-free test of that condition would need scorers spanning the saturation range, which our seven do not\.

### B\.3One Ranking, Fourteen Units

Every transform below is strictly increasing within a mention, so the ranking, the top\-1 and Recall@1 are identical in all fourteen arms; only the score domain moves\. This is the controlled counterpart of LIRICAL’s two vendor units, inside one system, and it replaces the seven\-system correlation the paper draws no ordering from\.

Table 3:Fourteen units of one ranking on12,74112\{,\}741BC5CDR mentions \(AUROC\); Recall@1 is77\.6%77\.6\\%in every row\.*center*shifts each mention’s scores to zero mean,*scale*divides by their standard deviation,*inject*adds a per\-mention offset of the given size drawn independently of correctness\.CChere is the one\-way ANOVA over the ten contenders, not the top\-two form §[5\.6](https://arxiv.org/html/2608.14683#S5.SS6)quotes; on the untransformed cosine the two read0\.2370\.237and0\.0110\.011on the same mentions\.Table[3](https://arxiv.org/html/2608.14683#A2.T3)separates the two halves of the decomposition\. Under*center*, a purely additive per\-mention shift, the margin’s AUROC is0\.7961000\.796100against the untransformed0\.7961000\.796100, equal in double precision as Eq\. \([4](https://arxiv.org/html/2608.14683#S3.E4)\) requires, while the raw score moves from0\.8220\.822to0\.7900\.790\. Under*scale*, a purely multiplicative one, the margin does move, which the algebra does not forbid\. The injection arms then sweep the between\-input share from0\.3880\.388to0\.9880\.988with the margin’s AUROC fixed at0\.7960\.796throughout: the gain grows from\+0\.023\+0\.023to\+0\.265\+0\.265entirely because the raw score decays from0\.7730\.773to0\.5310\.531, not because the margin improves\.

One observation is worth stating on its own\. The normalizations that fix a per\-row statistic outright, softmax at every temperature,zz\-score and sum, drive the between\-input variance to zero by construction, which is whyCCsits at the estimator’sK=10K\{=\}10floor of−1/9\-1/9in those five arms and in the centering arm\. Min\-max fixes only the two extremes, so its row means still vary a little and it reads−0\.047\-0\.047\. Those transforms can sweep the shared level down and no further, so a family built only from them cannot test whether the share tracks the gain; the injection arms exist to span it upward\. It also means that a system whose scores have already been normalized this way has no shared level left for the margin to cancel, which is the same reading the normalized cosine gets in App\.[I](https://arxiv.org/html/2608.14683#A9)\.

### B\.4Mechanism Test on the Retriever

Offset invariance is necessary for the proposed explanation but not sufficient to select a useful signal\. The mean\-centered top scores1−\|D\|−1​∑d∈Ds​\(d∣x\)s\_\{1\}\-\|D\|^\{\-1\}\\sum\_\{d\\in D\}s\(d\\mid x\)has the same invariance to candidate\-independent shifts as the margin, yet reaches only47\.9%47\.9\\%accuracy at10%10\\%coverage against the margin’s81\.0%81\.0\\%\. The invariance argument identifies an admissible class of signals; it does not show that every member of that class works\.

We test the cancellation on our retriever because it is the only system whose full candidate\-score vector we can recompute\. The experiment uses the complete ultra\-rare stratum \(n=4,780n\{=\}4\{,\}780, base Recall@125\.4%25\.4\\%\) and gates at10%10\\%realized coverage\.

Table 4:Testing the cancellation mechanism on the ultra\-rare retriever output; for entropy, lower is treated as more confident\. The temperature of the normalized log\-score is selected after observing performance, whereas the margin has no fitted parameter\.The normalized log\-score removes the per\-case normalizer and recovers almost all of the margin’s advantage, which supports the mechanism without being an equally deployable replacement: it ranges from34\.9%34\.9\\%to80\.3%80\.3\\%asTTvaries over\[0\.25,50\]\[0\.25,50\], best atT=3T\{=\}3, and that temperature is chosen after the fact\. The family also explains why the margin works\. AsT→0T\\\!\\to\\\!0the normalized log\-score approaches a ranking determined by the top\-two margin; asT→∞T\\\!\\to\\\!\\inftyit approaches the mean\-centered top score\. The low\-temperature region is numerically unstable, since atT=0\.5T\{=\}0\.5some60\.5%60\.5\\%of cases collide because the relevant exponential differences underflow, so the limit should not be read as an empirical temperature result\.

### B\.5Locating the Shared Score Component

Is the removable component determined by the case input or by the candidate\-score distribution? Input\-derived quantities \(the number of recorded HPO terms, the number matched, and summaries of their information content\) explainR2=0\.92R^\{2\}\\\!=\\\!0\.92of the raw top score’s variance, and the raw score correlates with matched\-term count atρ=0\.93\\rho\\\!=\\\!0\.93, so the raw score strongly reflects case composition\. Removing that dependence does not recover the margin’s gate: the best input\-derived normalizer reaches70\.7%70\.7\\%and residualizing on the in\-sample optimal linear combination of all input features reaches66\.5%66\.5\\%, against81\.0%81\.0\\%for the margin\.

Score\-derived normalizers behave differently\. Subtracting the score at increasing rank weakens the gate gradually: rank\-10 gives74\.1%74\.1\\%, rank\-100 gives62\.3%62\.3\\%, and subtracting the median gives47\.1%47\.1\\%, close to the uncorrected top score\. The relevant shared level is therefore set by the few candidates genuinely in contention, not by the bulk of all8,5538\{,\}553candidates nor by the patient description alone, which is why a head\-dominated quantity such aslog⁡ZT​\(x\)\\log Z\_\{T\}\(x\)succeeds where the mean does not\. This refines Eq\. \([3](https://arxiv.org/html/2608.14683#S3.E3)\): the useful approximation to the shared component is case\-specific but is expressed through the head of the competitor distribution, and the margin estimates that level locally using the nearest competitor\.

### B\.6Scope of the Argument

The log\-potential reading is an assumption, not a property guaranteed for every scoring system\. The weaker requirement the cancellation argument actually needs is that candidates for one case share an approximately candidate\-independent additive component; a difference removes such a component even when the scores are not calibrated probabilities\. The margin must also be taken on a scale that is additive over evidence: it is stable under a global positive affine map but not under an arbitrary monotone one, as LIRICAL shows, so choosing the score domain is part of specifying the signal\. Finally, the argument needs a fixed candidate set scored jointly by one predictor; independent samples from a free\-form generator are not the leading candidates of such a set and need not have a meaningful runner\-up \(App\.[I](https://arxiv.org/html/2608.14683#A9)\)\.

The mechanism experiment is conducted on one retriever whose complete score vector is available; Exomiser tests one prediction about score scale, and the remaining systems provide transfer evidence rather than further interventions\. We therefore treat cancellation as the supported mechanism in our main setting, not as a theorem that the margin must beat the top score for every ranker\. Determinism and floating\-point sensitivity are recorded in App\.[L](https://arxiv.org/html/2608.14683#A12)\.

## Appendix CPrevalence Strata and Averaging

Two questions about how the collapse is measured: whether the prevalence labels and their binning create it, and whether per\-case averaging inflates it\.

### C\.1Prevalence Stratification

Bin correction\.Orphanet lists some diseases with prevalence class “unknown”/“not yet documented\.” Pooling these \(truthy strings\) into the less\-rare bin would make57%57\\%of that anchor undocumented\-prevalence; only372/780372/780gold diseases carry a real class\. We route undocumented cases to a separate*unknown*bin\. Because the LLM performs near the tail on undocumented diseases, that routing raises the less\-rare anchor \(∼19%→∼37\\sim\\\!19\\%\\\!\\to\\\!\\sim\\\!37–41%41\\%\) and*widens*the reported collapse\.

Finer dose\-response\.Fig\.[2](https://arxiv.org/html/2608.14683#A3.F2)plots Recall@1 across the ordinal Orphanet classes\. LLM accuracy*peaks*at the moderately\-rare11–9/1059/10^\{5\}class and then declines monotonically across the two ultra\-rare classes to a near\-zero floor \(e\.g\. Qwen2\.5\-VL\-32B45\.8%45\.8\\%at11–9/105→18\.2%9/10^\{5\}\\to 18\.2\\%at11–9/106→0\.4%9/10^\{6\}\\to 0\.4\\%at<1/106\{<\}1/10^\{6\}; that model’s ownn=166/198/703n\{=\}166/198/703, the class sizes924/993/3787924/993/3787being the denominators of the retriever curve below\), confirming a genuine gradient/cliff rather than a two\-bin artifact; the decontaminated retriever is nearly flat over the same classes \(34\.0→26\.7→25\.2%34\.0\\\!\\to\\\!26\.7\\\!\\to\\\!25\.2\\%\)\. The dip at the*more\-common*but sparsely\-populated11–5/1045/10^\{4\}class \(n=243n=243; more common than the11–9/1059/10^\{5\}peak\) is within noise; the non\-monotonicity is at the common end and does not affect the ultra\-rare collapse\. A point\-prevalence \(rather than rarest\-class\) rule leaves the collapse magnitude essentially unchanged\.

![Refer to caption](https://arxiv.org/html/2608.14683v1/x2.png)Figure 2:Recall@1 across ordinal Orphanet prevalence classes \(common→\\torare\)\. LLM accuracy peaks at the moderately\-rare class then collapses across the ultra\-rare classes; the decontaminated phenotype retriever \(black\) stays nearly flat\.
### C\.2Per\-Case and Per\-Disease Averaging

The prevalence bins are dominated by a few high\-frequency, textbook\-famous diseases\. The less\-rare bin holds1,1671\{,\}167cases but only8181diseases; Neurofibromatosis\-1 \(OMIM:162200\) alone is405405cases \(34\.7%34\.7\\%of the bin, retriever Recall@143%43\\%\), and the top\-5 diseases are59\.6%59\.6\\%\. The ultra\-rare bin is far less concentrated \(291291diseases; largest9\.7%9\.7\\%\)\. Per\-case \(micro\) averaging therefore weights the famous less\-rare diseases heavily and inflates both the anchor and the collapse ratio\. Per\-disease \(macro\) averaging removes this weighting \(Table[5](https://arxiv.org/html/2608.14683#A3.T5)\): the less\-rare LLM anchor falls from∼40%\\sim\\\!40\\%\(micro\) to∼10%\\sim\\\!10\\%\(macro\), the LLM collapse ratio falls from8\.68\.6–25\.1×25\.1\\times\(micro\) to1\.91\.9–6\.1×6\.1\\times\(macro\), and the retriever’s per\-case decline \(31\.4%→25\.6%31\.4\\%\\\!\\to\\\!25\.6\\%, itself not significant\)*reverses*to a per\-disease*rise*\(12\.7%→23\.3%12\.7\\%\\\!\\to\\\!23\.3\\%\)\.

Two things follow, and they should be kept apart\. First, the*magnitude*of the collapse is a micro effect: per\-disease the LLM fall is only∼2\\sim\\\!2–6×6\\times\. Second, the tail*contrast*is not\. On the ultra\-rare stratum, which is not concentrated, the retriever holds∼24%\\sim\\\!24\\%under both averagings while the small models reach0\.70\.7–4\.6%4\.6\\%per case and1\.31\.3–4\.9%4\.9\\%per disease\. The deployment recommendation rests on that contrast and not on the height of the less\-rare anchor\.

Table 5:Recall@1 under per\-case \(micro\) and per\-disease \(macro\) averaging\.∗ranges span all eight small models; Yi, weak in both strata at7%7\\%even on less\-rare, is the low outlier\. Retriever micro figures use the full decontaminated set; its macro figures and every LLM figure use theN=2000N\{=\}2000sample\.Prevalence against corpus exposure\.Prevalence here is collinear with pretraining frequency and fame, since the famous less\-rare diseases are also the high\-accuracy ones, so we do not claim to separate the two\(Kandpalet al\.,[2023](https://arxiv.org/html/2608.14683#bib.bib1); Sunet al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib53)\)\. That binning a benchmark by an external popularity variable exposes a hidden tail is itself established\(Sunet al\.,[2024](https://arxiv.org/html/2608.14683#bib.bib53)\); what is specific here is §[5\.3](https://arxiv.org/html/2608.14683#S5.SS3), where the discriminating evidence is present in the input, so this is a failure to*use*evidence rather than an exposure gap\.Chenet al\.\([2026](https://arxiv.org/html/2608.14683#bib.bib25)\)run their own Orphanet\-mapped subgroup analysis and report no decline with prevalence, attributing their gradient to PubMed mention counts instead\. We take that seriously: their cases are PMC case reports drawn from the same literature the models are pretrained on, so case availability there is decoupled from epidemiological rarity by construction and their prevalence axis is not ours\. Neither study identifies prevalence net of frequency\. We cannot measure pretraining exposure, so we hold a proxy fixed instead: the number of cases of that disease in this corpus, which is what curator attention and the published literature jointly produced\. Two estimators, since neither alone convinces\. Binning diseases by exact corpus frequency and comparing strata*within*each bin assumes no model of the frequency effect and leaves the per\-disease gap at\+0\.151\+0\.151against a raw\+0\.159\+0\.159, i\.e\.95%95\\%retained\. Matching each less\-rare disease to the ultra\-rare disease nearest in log frequency, without replacement and within a caliper, gives\+0\.169\+0\.169, i\.e\.107%107\\%\. The residual is positive for all77models under both\. The matched design is small \(1313pairs, minimum detectable difference0\.1180\.118\)\. These controls therefore bound the exposure explanation rather than identifying a prevalence effect independent of frequency\.

## Appendix DCuration Leakage and External Validation

The retriever’s absolute accuracy is partially identified\. This section defines the two controls that bracket it, measures what neither can see, and prices the residual from outside the corpus\.

### D\.1Leave\-Source\-Out Decontamination

Phenopacket case identifiers carry their source publication \(PMID\), and each HPO annotation in HPOA carries the PMID\(s\) supporting it\. The fraction of cases whose own source PMID also appears as a source of the gold disease’s HPOA profile is74\.5%74\.5\\%\(less\-rare\),60\.6%60\.6\\%\(ultra\-rare\), and89\.9%89\.9\\%\(unknown\-prevalence\): for most cases the retriever matches a patient against a profile transcribed in part from the same publication\. To decontaminate, for each case we drop from the gold profile any HPO term whose*only*supporting PMID is that case’s own source \(a conservative leave\-source\-out keeping independently corroborated terms\), re\-propagate over the ontology, and re\-score with the identical tie\-aware expected Recall@1; all other diseases are untouched\. Recall@1 falls: less\-rare41\.9%→31\.4%41\.9\\%\\\!\\to\\\!31\.4\\%, ultra\-rare49\.4%→25\.6%49\.4\\%\\\!\\to\\\!25\.6\\%, unknown67\.5%→18\.0%67\.5\\%\\\!\\to\\\!18\.0\\%\(N=10,345N\{=\}10\{,\}345; the2929cases whose gold disease carries no HPOA profile are absent from the candidate set and are excluded from every retriever score, decontaminated or not\)\. A tie\-tolerant double\-precision reimplementation gives28\.1/24\.4/16\.6%28\.1/24\.4/16\.6\\%; we quote that range where the exact value matters, as float\-exact tie detection moves the less\-rare figure between26\.526\.5and32\.5%32\.5\\%\.

Residual curator\-level coupling\.98\.7%98\.7\\%of the phenopackets were created by a single curator account, and that same curator contributed∼21,800\{\\sim\}21\{,\}800HPOA disease\-phenotype rows, so patient record and knowledge base are not independently authored, and a PMID\-based control cannot see this\. Of the ultra\-rare patient terms matching the gold profile,51\.3%51\.3\\%were biocurated by the person who wrote the patient record and49\.6%49\.6\\%by that person*from the same publication*; our control removes23\.0%23\.0\\%and leaves26\.6%26\.6\\%\. The terms that survive do so by “independent corroboration”, but for91\.2%91\.2\\%of them the corroborating publication itself contributes cases of the same disease to this corpus, and for29\.9%29\.9\\%of ultra\-rare cases the gold’s HPOA frequency denominator exactly equals that disease’s case count here \(2\.1×2\.1\\timesa case\-weighted permutation null; a joint disease\-level permutation, the correct unit, gives3\.1×3\.1\\times,p<0\.001p<0\.001\), i\.e\. the annotation was tabulated from these patients\. A strict control that drops*any*gold term the case’s own publication supports gives ultra\-rare5\.2%5\.2\\%\(less\-rare6\.1%6\.1\\%\)\.

Partial identification of retriever accuracy\.The two controls bound the estimand from opposite sides under a monotone\-bias argument: the strict control discards terms that other publications genuinely establish, so it is biased*down*; leave\-source\-out retains terms written by the record’s own curator from other papers, so it is biased*up*\. Neither is the estimand, and HPOA records PMIDs but not cohort identity, so nothing in the metadata closes the gap\. We deliberately do*not*invoke the contaminated\-data bounds ofHorowitz and Manski \([1995](https://arxiv.org/html/2608.14683#bib.bib43)\): those require the contamination*rate*to be known and leave the contaminating distribution free, which is the reverse of our situation, we know the channel and not the rate, and with the rate unrestricted their sharp bounds are vacuous\. What we run is a sensitivity analysis indexed by that unknown rate, in the sense ofRosenbaum \([2002](https://arxiv.org/html/2608.14683#bib.bib44)\), and we report the breakdown point at which each conclusion fails\(Masten and Poirier,[2020](https://arxiv.org/html/2608.14683#bib.bib45)\)\. We accordingly report\[5\.2%,24\.4%\]\[5\.2\\%,\\,24\.4\\%\]as a*sensitivity range*, not a confidence interval\. The same range at the published implementation’s precision is\[5\.2%,25\.6%\]\[5\.2\\%,25\.6\\%\], the1\.21\.2pp being the tie\-handling gap above, and within\-control sampling error is separate, and disease\-clustered at the upper end gives24\.4%24\.4\\%\[17\.1,33\.0\]\[17\.1,33\.0\]\. The strict end is not knowledge\-base deletion: gold profiles retain67\.2%67\.2\\%of their annotations and only6\.8%6\.8\\%of ultra\-rare profiles empty\.

Stability across the sensitivity range\.Sweeping the whole interval, the gate’s*lift*over its own base stays in\[2\.4,3\.3\]×\[2\.4,3\.3\]\\times, it is non\-monotone, peaking at3\.3×3\.3\\timesin the interior before falling to2\.4×2\.4\\timesat the strict end, so the interval must be swept and not merely evaluated at its endpoints, and its AUROC falls monotonically from0\.800\.80to0\.650\.65\. The margin continues to outrank the best predictor\-free gate at every point, though its advantage narrows from1\.9×1\.9\\timesto1\.3×1\.3\\times; we state that as a dominance, not as a level, because the predictor\-free gate’s own level moves across the sweep too and we did not register it endpoint by endpoint\. What is*not*invariant is the absolute level, and with it the comparison against the frontier model, whose breakdown point is only≈6%\{\\approx\}6\\%residual contamination, so we rest nothing on it\. The small\-model comparison is not invariant either, and we do not claim it is: at the strict endpoint the retriever’s5\.2%5\.2\\%clears55of the88small models, but the strongest three \(4\.24\.2–4\.6%4\.6\\%\) are not separable from it at this sample size\. We therefore rest the gate claim on its lift over its own base rather than on the level, and state the accuracy comparison as holding under leave\-source\-out and against the five weaker small models under every control\.*Two asymmetries matter when reading the bracket\.*The LLMs are not decontaminated at all, the same case reports are in their pretraining data, so a maximally\-stripped retriever against an unstripped LLM compares a lower bound with an upper bound rather than measuring which predictor is better\. And at the strict end the retriever falls below the frontier model, so “no LLM surpasses it on the tail” holds against the frontier only under leave\-source\-out, and at the strict end is a statement about the five weaker small models\. Note that the vocabulary ablations below rebut the*near\-unique\-key*mechanism and are computed off the leave\-source\-out retriever; they do*not*address shared authorship, which no vocabulary restriction can remove\.

Selection\-based check and its confounds\.One can avoid the over\-correction objection entirely by*selecting*rather than deleting: keep only tail cases for which no matched gold term is supported by the patient’s own publication\. That leaves2,441/4,7802\{,\}441/4\{,\}780cases, on which the retriever falls25\.4%→9\.7%25\.4\\%\\\!\\to\\\!9\.7\\%while the frontier model barely moves \(22\.4→19\.3%22\.4\\\!\\to\\\!19\.3\\%\), i\.e\. on that subpopulation the frontier model is ahead\. Two confounds inflate this\. Those cases have far thinner gold profiles \(40\.240\.2vs\.84\.084\.0HPOA terms\), and the flag itself is mechanically tied to how much matched:P​\(flagged\)P\(\\text\{flagged\}\)rises from0%0\\%at11–33matched terms to59%59\\%at≥26\{\\geq\}26, while matching more terms independently predicts being correct\. Stratifying on matched\-term count does not explain the gap away: it shrinks to\+11\.1\+11\.1pp among cases matching between1010and2525terms \(n=1,223n\{=\}1\{,\}223\), but in the≥26\{\\geq\}26stratum, which holds63%63\\%of the tail, it is\+34\.8\+34\.8pp against the\+32\.3\+32\.3pp unstratified gap\. We therefore read this as corroborating the bracket, own\-source support does carry real signal beyond case difficulty, rather than as a clean estimate of either endpoint\.

A near\-unique\-key artifact does not operate here\.A canonical field that identifies an entity outright, and does so more often for rarer entities, can manufacture a tail gradient on its own\. Term specificity does rise with rarity \(17\.7→24\.9%17\.7\\\!\\to\\\!24\.9\\%of diseases own a globally\-private term\), but deleting*every*such term from both patient and knowledge base leaves the tail unchanged \(24\.4→24\.4%24\.4\\\!\\to\\\!24\.4\\%, gate79\.7→78\.5%79\.7\\\!\\to\\\!78\.5\\%\): private keys carry only13\.9%13\.9\\%of the tail’s Recall@1 mass\. Removing the patient’s single highest\-information term costs4\.44\.4pp, and restricting the vocabulary to phenotypes shared by more than100100diseases still leaves the tail at16\.8%16\.8\\%, above every small model\. An HPO profile is1010–4040graded\-specificity terms, not one canonical field, which is why that failure mode has no analogue here\. The gate is the most robust component: margin AUROC stays at0\.780\.78–0\.870\.87across all of these ablations and at0\.650\.65under the strict control\. Its*operating point*, however, does not survive that control: with a5\.2%5\.2\\%base the confident decile delivers12\.3%12\.3\\%\(lift2\.4×2\.4\\times\)\. Note this configuration*passes*p≥τ​cp\\\!\\geq\\\!\\tau c\(5\.2%5\.2\\%against a5%5\\%floor\) while delivering nowhere nearτ\\tau, a clean illustration that the criterion is necessary and not sufficient \(§[5\.2](https://arxiv.org/html/2608.14683#S5.SS2)\), and that at the strict end the gate would be a directional signal rather than a safe operating point\. App\.[D\.2](https://arxiv.org/html/2608.14683#A4.SS2)is why we do not read the strict end as the operating truth\.

The apparent “flat/higher on the tail” shape \(§[5\.3](https://arxiv.org/html/2608.14683#S5.SS3)\) is thus partly curation leakage\. We do*not*claim a direction for the decontaminated per\-case gradient: the less\-rare bin holds1,1671\{,\}167cases but only8181diseases \(neurofibromatosis type 1 alone is34\.7%34\.7\\%of it, the top five59\.6%59\.6\\%\), and a disease\-clustered bootstrap puts the gradient at−3\.7\-3\.7pp with95%95\\%CI\[−14\.5,\+14\.6\]\[\-14\.5,\+14\.6\], which per\-disease macro\-averaging flips to\+12\.8\+12\.8pp \(App\.[C\.2](https://arxiv.org/html/2608.14683#A3.SS2)\)\. Absolute retriever % is an upper bound for two*distinct*reasons: same\-source leakage, which we measure and remove here, and post\-hoc HPO*completeness*, a distribution shift decontamination does not touch and which we quantify separately \(App\.[F\.2](https://arxiv.org/html/2608.14683#A6.SS2)\), not because the retriever beats real tools \(§[4](https://arxiv.org/html/2608.14683#S4)\); all triage signals \(§[5\.4](https://arxiv.org/html/2608.14683#S5.SS4)\) are computed on this decontaminated retriever\.

### D\.2Evaluation on External Rare\-Disease Cohorts

App\.[D\.1](https://arxiv.org/html/2608.14683#A4.SS1)leaves the tail retriever partially identified because both of our controls are proxies for the same unobserved quantity: how much of the retriever’s accuracy comes from one curator having written both the patient records and much of the knowledge base\. A PMID\-based control cannot see curator\-level coupling, and the strict control over\-corrects by construction, since a gold term the case’s publication happens to mention is not thereby a leaked term\. No amount of further decontamination of*this*corpus can separate the two\.

We therefore price the leak from outside\. RareBench\(Chenet al\.,[2024b](https://arxiv.org/html/2608.14683#bib.bib3)\)aggregates five rare\-disease cohorts none of which is Phenopacket Store; three are curated by groups with no relation to it: RAMEDIS \(Bielefeld University and the Reutlingen children’s hospital\), HMS \(Germany\), and MME \(CHEO/SickKids/Toronto\)\. We exclude RareBench’s fourth cohort, LIRICAL, on purpose: its cases derive from published case reports inside the same Monarch/HPO ecosystem, so it is not a clean independence test\. Applying the paper’s own prevalence rule \(App\.[C\.1](https://arxiv.org/html/2608.14683#A3.SS1)\) to the remaining cases gives243243ultra\-rare cases over6363diseases\. We run the*identical*retriever, same code, same HPOA index, same IC weighting, no refitting; Table[6](https://arxiv.org/html/2608.14683#A4.T6)reports it\.

Table 6:The identical retriever on three rare\-disease cohorts curated independently of Phenopacket Store, with no refitting\. Intervals are clustered on disease; the disease\-disjoint row keeps only cases whose diagnosis never appears in our own corpus\.External estimates and the sensitivity range\.Disease\-clustered, the external interval sits at the upper end; clustered at the*cohort*level, the right level, since a deployment draws a cohort and not a disease, it contains both endpoints, and aχ2\\chi^\{2\}test rejects a common rate across the three \(p<10−6p<10^\{\-6\}\)\. We therefore report corroboration, not identification\. Reporting the cohort we excluded makes the same point: LIRICAL, whose cases come from published case reports inside the HPO ecosystem, scores52\.0%52\.0\\%\(n=244n\{=\}244\), twice the independent cohorts and close to the undecontaminated Phenopacket Store tail \(49\.4%49\.4\\%\), which is what the leakage account predicts\. The strict control is thus over\-conservative rather than merely conservative: deleting every gold term the case’s own publication supports removes genuine phenotype signal, not only leakage\. Restricting to the216216cases whose disease does not appear in Phenopacket Store at all, so that neither the case nor the disease is shared with our corpus, gives19\.4%19\.4\\%, still excluding the strict endpoint\. Per\-disease averaging gives29\.2%29\.2\\%, i\.e\. the estimate does not depend on case concentration\.

Limits of the external\-cohort evidence\.Between\-cohort spread \(0\.00\.0–53\.8%53\.8\\%\) is far wider than the sampling uncertainty within any one of them, so∼23%\{\\sim\}23\\%is an estimate for*this mixture*of independent cohorts, not a universal constant; a deployment in a population resembling HMS should expect much less\. HMS’s zero is partly structural,33of its cases have a gold disease with no phenotypic HPOA annotation at all, hence unretrievable by any phenotype method, but most of it is genuine failure\.n=243n\{=\}243is small against the main benchmark’s4,7804\{,\}780\. Finally, RareBench’s HPO codings were produced by its own authors; if that mapping consulted HPOA disease profiles, a weaker second\-order coupling survives that these files cannot test\. We therefore treat the external estimate as pricing the*first\-order*curator confound, which is the one that made the interval five\-fold wide, and continue to report the sweep in App\.[D\.1](https://arxiv.org/html/2608.14683#A4.SS1)rather than replacing it\.

## Appendix EFeasibility, Calibration and Delivery Rules

What the feasibility bound forbids, what the incumbent confidence signals can actually resolve, and how both move with the delivery rule\.

### E\.1Auditing the Feasibility Criterion

The criterion is one line of algebra, so its interest is in whether it*discriminates*when applied broadly\. We read it across every \(predictor, stratum\) cell we have, ten LLMs by prevalence bin, our retriever, Exomiser by its own score and by its shippedpp\-value, and the SapBERT and BM25 entity linkers by concept frequency, atc=10%c\{=\}10\\%\.

Most cells are non\-binding\.In2121of the2929the base rate is high enough that the ceiling is1\.01\.0, so noτ\\tauforbids them; only88are informative\. Atτ=50%\\tau\{=\}50\\%,77of those88forbid\. Among the2222permitting cells the incumbent’s own confidence reaches the target in1010, but only1515of them are measurable at all, because in the other77the confidence signal is too tie\-collapsed to resolve a10%10\\%coverage\. We report that denominator rather than scoring an unmeasurable cell as a failure\.

The forbidding cells, read at realised coverage\.A cell’s ceiling is computed atc=10%c\{=\}10\\%, but a tie\-collapsed signal answers fewer cases than that, and a smaller coverage raises the ceiling\. Each cell must therefore be read at the coverage its own signal actually resolves:44of the77forbidding cells are*not*forbidden there, and in22more the signal resolves no coverage at all, so exactly*one*of the seven is a measured forbidding cell\. That reading answers a different question from the one the criterion poses, since a smaller coverage raises the ceiling, so it is a diagnostic about the signals’ resolution, not a rescue of the cells\. The ceiling itself cannot be violated:sel​\-​acc​\(c\)≤min⁡\(1,p/c\)\\mathrm\{sel\\text\{\-\}acc\}\(c\)\\leq\\min\(1,p/c\)is a counting identity, so no cell could contradict it and none does\. What is measured rather than entailed is the single forbidding cell whose signal actually operates atc=10%c\{=\}10\\%\.

Interpretation of the feasibility audit\.It shows the criterion is not vacuous: it forbids where base rates have collapsed and permits where they have not, along a boundary that matches the rest of the paper, every forbidding cell is on the ultra\-rare stratum of the rare\-disease task, while the entity\-linking cells have base rates of6565–84%84\\%and are never forbidden\. It does*not*show seven independent rescues: the external arm in those seven cells is the same retriever margin on the same ultra\-rare cases against seven different incumbents, so behind them lie only22distinct external measurements\.

### E\.2Confidence Calibration and Risk–Coverage

We score three confidence signals \(verbalized\-overall, verbalized\-top1, mean token log\-probability\) by AUROC\-of\-correctness and ECE per bin, with bootstrap95%95\\%CIs\. \(Correctness here is nearest\-name cosine≥0\.90\\geq\\\!0\.90; Tables[1](https://arxiv.org/html/2608.14683#S5.T1),[2](https://arxiv.org/html/2608.14683#S5.T2)use exact OMIM\-id and the two agree qualitatively\.\) Verbalized confidence’s tail discrimination splits by scale: the*smallest*models are near chance \(77B ultra\-rare own\-answer AUROC0\.5710\.571, wide CI,0\.6040\.604on the full set; Llama\-8B0\.5090\.509\), whereas the two larger models rank their own tail answers*well*\(1414B0\.7770\.777,3232B0\.9050\.905\)\. Good ranking does*not*rescue them: by the base\-accuracy ceiling \(§[5\.2](https://arxiv.org/html/2608.14683#S5.SS2)\) even Qwen\-32B’s near\-oracle discrimination leaves its most\-confident10%10\\%far below a safe accuracy \(Table[2](https://arxiv.org/html/2608.14683#S5.T2)\)\. Read as a coverage, the same bound says a small model can answer at mostp/τp/\\tauof the tail at accuracyτ\\tau, which atτ=50%\\tau\{=\}50\\%is1\.51\.5–9\.2%9\.2\\%of it, so raising coverage toc=25%c\{=\}25\\%is missed by more than2\.7×2\.7\\times\. This is because so few tail answers are correct to begin with, so we do*not*read the collapse as a discrimination failure\. ECE worsens sharply on the tail \(3232B less\-rare0\.42→0\.42\\\!\\to\\\!ultra\-rare0\.730\.73, corrected bins\), and the3232B mean\-logprob AUROC*inverts*\(0\.79→0\.360\.79\\\!\\to\\\!0\.36, CI below0\.50\.5\), so likelihood\-based abstention would prefer wrong answers\.Comparison across feasibility regimes\.Ifp≥τ​cp\\\!\\geq\\\!\\tau cis doing work rather than decorating an identity, the external gate’s advantage should decline as the incumbent’s base accuracy rises\. It does\. Table[7](https://arxiv.org/html/2608.14683#A5.T7)reads it at matched10%10\\%coverage, with the LLM’s confidence given its*most favourable*tie\-breaking\.

configurationpp\(tail base\)oracle ceilingfeasible?margin@10%10\\%advantageYi\-1\.5\-9B0\.7%0\.7\\%7%7\\%no82\.8%82\.8\\%\+79\.6\+79\.6InternLM2\.5\-7B1\.4%1\.4\\%14%14\\%no81\.7%81\.7\\%\+76\.3\+76\.3Mistral\-7B\-v0\.31\.6%1\.6\\%16%16\\%no77\.4%77\.4\\%\+62\.4\+62\.4Qwen2\.5\-VL\-7B1\.7%1\.7\\%17%17\\%no80\.6%80\.6\\%\+66\.7\+66\.7Qwen2\.5\-14B4\.2%4\.2\\%42%42\\%no74\.2%74\.2\\%\+46\.2\+46\.2Qwen2\.5\-VL\-32B4\.3%4\.3\\%43%43\\%no75\.6%75\.6\\%\+51\.1\+51\.1Llama\-3\.1\-8B4\.6%4\.6\\%46%46\\%no79\.6%79\.6\\%\+66\.7\+66\.7DeepSeek\-V4\-Flash15\.8%15\.8\\%100%100\\%yes77\.4%77\.4\\%\+8\.6\+8\.6V4\-Pro \(reason\. off\)18\.6%18\.6\\%100%100\\%yes76\.3%76\.3\\%\+0\.0\\mathbf\{\+0\.0\}V4\-Pro \(reason\. on\)22\.4%22\.4\\%100%100\\%yes76\.2%76\.2\\%−2\.4\\mathbf\{\-2\.4\}Table 7:The external gate’s advantage against the incumbent’s own confidence, at matched10%10\\%coverage and with the LLM’s confidence given its most favourable tie\-breaking\.*feasible?*is whetherp≥τ​cp\\geq\\tau cholds atτ=50%\\tau\{=\}50\\%\. The advantage collapses exactly where the ceiling stops binding\.Mean advantage\+64\.1\+64\.1pp in the seven ceiling\-bound configurations and\+2\.1\+2\.1pp in the three where it has lifted\.

Do not read a dose\-response off this table\.The margin band is nearly flat \(7474–83%83\\%\), so the advantage is essentially a constant minus the confidence band, and that band is bounded bymin⁡\(1,p/c\)\\min\(1,p/c\), a deterministic function ofpp\. Substituting*any*quality of confidence ranker reproduces the observedr=−0\.970r\{=\}\{\-\}0\.970to within0\.030\.03, and residualising on the ceiling leaves−0\.15\-0\.15\. The apparent trend is the ceiling identity restated and we claim nothing from it\.

A lifted ceiling is not sufficient\.Sweeping*coverage*on a fixed model moves the ceiling withppheld exactly constant, which separates the ceiling from model quality\. Llama\-3\.1\-8B \(p=4\.6%p\{=\}4\.6\\%\) has a ceiling of1\.01\.0atc=2c\{=\}2–3%3\\%and its confidence band there is10\.710\.7–11\.1%11\.1\\%against the margin’s77\.877\.8–82\.1%82\.1\\%: an advantage of\+67\+67to\+71\+71pp in exactly the regime where the criterion no longer binds\. The three frontier configurations, where the two gates do converge, are one base model at three settings from one vendor, and their per\-configuration advantages all straddle zero\. We therefore claimp≥τ​cp\\\!\\geq\\\!\\tau cas a necessary condition, failing it forecloses every confidence policy, and claim nothing from passing it\.

Two further limits on this table\. The configurations cluster atp≤4\.6%p\\\!\\leq\\\!4\.6\\%andp≥15\.8%p\\\!\\geq\\\!15\.8\\%, soτ​c=5%\\tau c\{=\}5\\%falls in an empty gap and the boundary’s*location*is not identified by our data\. And the confidence bands use the tie\-breaking most favourable to the LLM; under random tie\-breaking they fall further, which widens the margin’s advantage\. It does*not*leave everything unchanged: the go/no\-go table’s permitting cells depend on it, falling from three of three to one of three \(§[5\.2](https://arxiv.org/html/2608.14683#S5.SS2)\)\.

Deployment risk–coverage\.none of these AUROCs buys a safe operating point\. Gating each deployable system by its own signal, a retriever\-first system answers the top10%10\\%of ultra\-rare cases at7474–83%83\\%accuracy, versus22–18%18\\%for the bare LLM under its own confidence \(base Recall@1≤5%\\leq\\\!5\\%\)\. Both numbers are levels, and the level is the part of this result that is*not*invariant over the leakage bracket: at the strict endpoint the retriever\-first band falls to12\.3%12\.3\\%, below Qwen\-14B’s observed18%18\\%\. What survives the bracket is the*lift*over each system’s own base rate \(2\.42\.4–3\.3×3\.3\\timesfor the margin\)\. A lift is what our claim rests on and it is not an operating point: siting a threshold needs the level, which the bracket does not fix\. The ordering of levels holds under leave\-source\-out and not under the strict control\.

Certifying the operating point\.Our threshold is chosen, not certified, and this appendix records what certifying it would take\. Split\-conformal selective prediction calibrates a margin threshold on a held\-out half so that empirical error on the answered part is at mostα\\alpha, then reports the coverage achieved on the test half; the guarantee is distribution\-free and finite\-sample, so it holds whatever the absolute accuracy is, which is why the identification bracket of App\.[D\.1](https://arxiv.org/html/2608.14683#A4.SS1)changes how many patients can be served at a given promise rather than breaking the promise\. Atα=20%\\alpha\{=\}20\\%andδ=0\.10\\delta\{=\}0\.10the calibration half needs≈2,660\{\\approx\}2\{,\}660answered cases, which at10%10\\%coverage is≈26,600\{\\approx\}26\{,\}600patients;α=30%\\alpha\{=\}30\\%needs≈50\{\\approx\}50andα=40%\\alpha\{=\}40\\%≈20\{\\approx\}20\. The binding problem is not sample size but dependence\. Correctness is clustered within disease over207207diagnoses\. A one\-way random\-effects ANOVA on those cases gives ICC0\.5530\.553and a design effect of2\.912\.91, an effectivennof322322; a second estimator archived with the audit gives0\.7040\.704and268268\. We quote the range,268268–322322independent cases, and a cluster\-aware certifier evaluated on*held\-out diseases*breaches its nominal risk on1111–44%44\\%of splits against1313–23%23\\%under i\.i\.d\. case splits\. A deploying site routinely meets diagnoses absent from any calibration set it could have assembled, so the held\-out\-disease split is the regime that matters\.

### E\.3Recall atkk

Every accuracy number in this paper is top\-1, and the feasibility criterion is a top\-1 argument, but the systems emit a ranked top\-5 differential and a clinician reads one\. The two are not interchangeable, so we measure the gap rather than concede it\. Each of the five generated names is linked to OMIM by the same SapBERT nearest\-candidate rule used for top\-1, so a name is scored at rank55exactly as it is scored at rank11\. One caveat belongs here rather than in a footnote: BioMistral\-7B emits a mean of1\.151\.15names per case and exactly one in90%90\\%of them, so its “k=5k\{=\}5” column isk=1k\{=\}1by output\-format failure\. Its ceiling is belowτ\\taueither way, so the44\-of\-88count does not change, but one of those four is a degenerate cell and the table should not be read as though it were not\.

Table 8:Recall at thekkthe deliverable is defined at, and the feasibility ceilingmin⁡\(1,p/c\)\\min\(1,p/c\)recomputed from the base rate at thatkk\.*Forbids*marks the cells where the ceiling still lies belowτ=0\.5\\tau\{=\}0\.5atc=10%c\{=\}10\\%, i\.e\. where the ceiling’s arithmetic still applies\. It does so for all88small models atk=1k\{=\}1but only44atk=5k\{=\}5\.Two things follow, and they cut in opposite directions\. The collapse itself is robust tokk: the small models’ ultra\-rare Recall@5 is22–8%8\\%, against a less\-rare anchor that stays near40%40\\%, so the collapse is not a top\-1 artifact\. The infeasibility verdict is more fragile\. The ceiling forbids theτ=50%\\tau\{=\}50\\%,c=10%c\{=\}10\\%operating point for all88small models atk=1k\{=\}1but for only44atk=5k\{=\}5: Qwen2\.5\-14B, Qwen2\.5\-VL\-32B, Llama\-3\.1\-8B and Mistral\-7B\-v0\.3 pass out of the forbidden region once success means gold\-in\-top\-5\. For those four the question stops being arithmetic and becomes empirical, we have not shown that any confidence policy on them*does*reach the operating point, only that the ceiling no longer rules it out\. The recommendation itself is unaffected: the gate’s advantage atk=5k\{=\}5is measured separately \(App\.[F\.3](https://arxiv.org/html/2608.14683#A6.SS3)\) and survives\.

## Appendix FPhenotype\-Ranker Analyses

Whether a deployed tool reproduces the margin result, whether it survives incomplete phenotyping, and which patients the gate ends up selecting\.

### F\.1Exomiser in Phenotype\-Only Mode

![Refer to caption](https://arxiv.org/html/2608.14683v1/x3.png)Figure 3:Selective accuracy at different coverage levels for phenotype\-only Exomiser \(left\) and our phenotype retriever \(right\), selecting by the top score or the top\-two margin; the horizontal line is accuracy when all cases are answered\. On Exomiser the top score never rises meaningfully above answering everything, so no coverage makes it a usable gate\.We do not apply leave\-source\-out to Exomiser, so its recall belongs beside our undecontaminated figures rather than beside the retriever’s headline\. To anchor our reimplemented retriever against a tool clinicians actually run, we execute Exomiser 14\.0\.0\(Smedleyet al\.,[2015](https://arxiv.org/html/2608.14683#bib.bib8)\)\(2402 data release\) in*phenotype\-only*mode, the mode used byReeseet al\.\([2026](https://arxiv.org/html/2608.14683#bib.bib4)\), on the identicalN=2000N\{=\}2000sample: each case’s HPO terms as a phenopacket with no variants \(a shared empty hg38 VCF satisfies the CLI; thephenotype\-onlypreset scores no variants\)\. Exomiser ranks genes by hiPHIVE phenotype similarity; we scoregene\-levelRecall@1, did it rank the case’s causative gene \(from the phenopacket\) first, which for these Mendelian cases proxies disease\-level but is Exomiser’s native output and*not directly comparable*to the disease\-level LLM/retriever numbers\.

Accuracy\.Gene\-R@1 is21\.2%21\.2\\%\(less\-rare\) /13\.3%13\.3\\%\(ultra\-rare\) /6\.0%6\.0\\%\(unknown\); R@54242/2828/13%13\\%\. This is comparable to our retriever and2\.92\.9–17\.9×17\.9\\timesthe bare LLM’s tail across the small models, but*lower*than the35\.5%35\.5\\%Reeseet al\.\([2026](https://arxiv.org/html/2608.14683#bib.bib4)\)report, expected, given the gene\-level metric \(the causative gene must rank first among∼20,000\{\\sim\}20\{,\}000, a different candidate space\), the phenopacket\-store case mix, and phenotype\-only prioritization without variant evidence\. It confirms the paradigm\-level point \(phenotype tools≫\\ggbare LLM on the tail\) without a horse race, and the modest absolute ceiling \(1313–24%24\\%for both tools\) is exactly why the deployable contribution is triage, not accuracy\.

Margin\-based triage with Exomiser\.Exomiser’s top\-1 combined*score*cannot define a10%10\\%operating point on this tail at all:*no*case is strictly above the cut, and the whole decile is drawn by sort order from a single228228\-case tie block, so the2\.2%2\.2\\%it appears to deliver is an artifact of that ordering and not a gradient; we score ties by their expected contribution rather than by an arbitrary break\(McSherry and Najork,[2008](https://arxiv.org/html/2608.14683#bib.bib76)\), and report the range the break spans where it matters\. But ranking by the top1−\-top2*margin*, the top\-10%10\\%reach29\.0%29\.0\\%\(ultra\-rare\) /40\.9%40\.9\\%\(less\-rare\), well above base\. So the margin\-triage property of §[5\.4](https://arxiv.org/html/2608.14683#S5.SS4)is not an artifact of our reimplementation: a real deployed tool’s own margin also yields a usable operating point on the tail\. Two alternative readings of that result do not survive\. It is not the tie block: on the711711cases whose phenotype score has not saturated, the margin’s decile still beats the raw score’s by\+28\.5\+28\.5pp, disease\-clustered\[\+15\.4,\+42\.3\]\[\+15\.4,\+42\.3\]pp, though the raw decile is only nominally below its own base rate there, so we claim of it that it buys nothing rather than that it hurts\. Nor is the raw score a straw target: over the whole tail it gates at AUROC0\.3830\.383,0\.5070\.507once the block is removed\. The empiricalpp\-value Exomiser ships, a principled normalization already used in clinical filtering and the statistic a deployment reaches for first, does no better: AUROC0\.4150\.415, and as a triage signal its most confident10%10\\%delivers7\.5%7\.5\\%, below the13\.3%13\.3\\%base rate\. That is the ordering App\.[B\.5](https://arxiv.org/html/2608.14683#A2.SS5)predicts, since a normalization derived from the input rather than from the competitors in contention does not recover the gate\. Usable, not safe:29\.0%29\.0\\%is far below theτ=50%\\tau\{=\}50\\%this paper uses as its worked example, and it is the margin, not the raw score, that yields it\.

### F\.2Robustness to Phenotype Incompleteness

The benchmark’s HPO terms are curated*post\-diagnosis*, so they are more complete and precise than a prospective work\-up, which could inflate the phenotype\-only retriever\. To test how far our conclusions survive incomplete phenotyping, we drop a fractionρ\\rhoof each case’s*present*HPO terms, at random, and, as an*adversarial worst case*for a phenotype\-overlap retriever,*highest information\-content first*\(dropping the rare, distinctive findings it most relies on; the opposite regime, missing common low\-IC terms, would hurt it less\), and re\-run*both*predictors on the*identical*degraded input: the retriever is re\-scored, and the frontier DeepSeek\-V4\-Pro is re\-queried on the degraded prompts \(same seed\-0N=2000N\{=\}2000sample\)\. Atρ=0\\rho\{=\}0the harness reproduces the retriever’s clean tail number on this sample \(25\.6%25\.6\\%;25\.6%25\.6\\%on the full decontaminated set\)\.

Table 9:Ultra\-rare Recall@1 under simulated incomplete phenotyping, dropping the highest\-information findings first\. Both predictors are re\-run on the*identical*degraded input; the LLM arm is DeepSeek\-V4\-Pro with reasoning off, held per\-case\.Effects on the main conclusions\.*The tail ranking is robust*: “no LLM surpasses the retriever on the tail” survives realistic incompleteness, because the frontier LLM degrades at least as fast on the tail \(decisively for reasoning\-off; a partialn=127n\{=\}127check for reasoning\-on\), so the predictor\-selection recommendation stands, and the anticipated failure \(the LLM’s parametric priors letting it overtake the retriever under noisy input\) does*not*occur\. The deployment\-relevant comparison survives too: under the same dropout the retriever’s margin band still triages the top\-10%10\\%far above the frontier LLM’s own confidence band \(62\.4%62\.4\\%vs\.21\.5%21\.5\\%atρ=0\.30\\rho\{=\}0\.30,39\.8%39\.8\\%vs\.12\.9%12\.9\\%atρ=0\.50\\rho\{=\}0\.50; the LLM band on the looser cosine\-correctness of App\.[E\.2](https://arxiv.org/html/2608.14683#A5.SS2), generous to the LLM\), because the LLM’s cratered tail base rate re\-imposes the ceiling on its own confidence\.*The absolute operating point is not robust*: the retriever’s tail accuracy falls, and its margin\-triage safeguard weakens with it \(the*retriever’s own*top\-10%10\\%accuracy,82\.8%→62\.4%→39\.8%82\.8\\%\\\!\\to\\\!62\.4\\%\\\!\\to\\\!39\.8\\%atρ=0,0\.30,0\.50\\rho\{=\}0,0\.30,0\.50; Table[2](https://arxiv.org/html/2608.14683#S5.T2)’s7474–81%81\\%is the delivered\-hybrid figure\), so the “safe” band is itself conditional on phenotyping quality, a caveat we state where the claim is made \(§[5\.4](https://arxiv.org/html/2608.14683#S5.SS4), Limitations\)\. The base\-accuracy ceiling and the small\-model negative are*un*affected: degrading phenotypes only lowers the tail base rate, tightening the ceiling\.

### F\.3Characteristics of Answered Cases

A margin gate would be of little clinical interest if it fired only on cases carrying one unmistakable, disease\-defining finding, the cases a specialist resolves without help\. We test this directly on the full ultra\-rare tail \(n=4,780n\{=\}4\{,\}780\): rank cases by the decontaminated retriever’s top1−\-top2 margin, take the answered top\-10%10\\%\(n=478n\{=\}478\), and compare them with the deferred remainder\.

Table 10:Characteristics of the cases the margin gate answers against those it defers\.*share of evidence*is the single strongest matched finding’s contribution, given raw and relative to the uniform1/n1/nshare; the frequency proxy is that disease’s case count in this corpus\.The answered cases are not “one give\-away finding” cases\. They are more richly phenotyped and are decided by roughly twice as many converging findings, and the single strongest finding carries the*same*relative share of the evidence in both groups \(2\.7×2\.7\\timesagainst2\.5×2\.5\\timesthe uniform share\), so the raw difference \(5\.3%5\.3\\%against10\.4%10\.4\\%\) follows mechanically from having more matched findings rather than from greater peakiness\. What distinguishes the answered set is the breadth of converging evidence\. The gate also fires on*less*frequently represented diseases \(51\.951\.9against111\.0111\.0cases per disease\), so it is not concentrating on the corpus’s famous entities\. These are properties of case*structure*, a proxy for where multi\-finding integration is hard; we do not measure clinician performance\.

Per\-disease\.The answered decile spans9494diseases \(top disease13\.2%13\.2\\%of it\), so we also report it macro\-averaged:81\.0%81\.0\\%micro becomes64\.6%64\.6\\%per\-disease \(disease\-clustered95%95\\%CI\[54\.7,73\.5\]\[54\.7,73\.5\]\), still2\.6×2\.6\\timesits own base\. The gate’s advantage is not an artifact of a few well\-represented diseases\.

## Appendix GRetriever–LLM Hybrids and Trained Fusion

All numbers use the decontaminated retriever \(App\.[D\.1](https://arxiv.org/html/2608.14683#A4.SS1)\) and theN=2000N\{=\}2000per\-model sample, with20002000\-resample bootstrap intervals\.

Hybrid\-gate performance\.We gate the delivered answer of a hybrid predictor \(keep the LLM top\-1 when the retriever ranks it in its top\-5, else output the retriever’s top\-1\) by each signal\. On every tail bin the retriever’s own margin dominates\. Over Table[2](https://arxiv.org/html/2608.14683#S5.T2)’s four small configurations the ultra\-rare AUROC is0\.740\.74–0\.770\.77for the margin against0\.540\.54–0\.560\.56for LLM confidence and0\.480\.48–0\.530\.53for agreement, and0\.730\.73–0\.740\.74against0\.450\.45–0\.520\.52and0\.490\.49–0\.580\.58on unknown\-prevalence cases\. Only on less\-rare disease does agreement lead \(0\.790\.79–0\.850\.85\)\. Relaxing the operating point from10%10\\%to25%25\\%coverage shows the usual selective\-prediction trade: on the ultra\-rare tail the margin band falls from7474–83%83\\%to5050–57%57\\%across all ten gated configurations \(Table[2](https://arxiv.org/html/2608.14683#S5.T2)’s four span7474–81%81\\%\), so by a quarter coverage the answered set already carries4343–50%50\\%errors\. This is why we describe the gate as a triage safeguard rather than a cure\.

Agreement as an LLM reliability signal\.Whether retriever–LLM agreement predicts the LLM’s*own*top\-1 correctness is a separate question from whether it should gate the delivered answer, and it is worth separating because agreement could be a restatement of case difficulty\. It is not: Table[11](https://arxiv.org/html/2608.14683#A7.T11)shows agreement predicting LLM correctness far above the “the retriever itself solved the case” difficulty proxy on all1818bins, including the ultra\-rare tail \(3232B tail0\.9190\.919against0\.7870\.787\)\. Agreement is therefore a genuine, non\-circular reliability signal, but once a system already defers to the retriever, the LLM’s contribution to the*delivered*tail answer vanishes\.

Table 11:Predicting the LLM’s*own*top\-1 correctness \(AUROC, decontaminated retriever; six models incl\. the two frontier ones\)\. Retriever–LLM agreement \(left\) is far above the trivial “did the retriever’s own top\-1 match gold” difficulty proxy \(right\) on every bin, so agreement carries reliability information beyond case difficulty \(non\-circular\)\. This is the*estimator*view; for*deployment*the retriever’s own margin is the better gate \(Table[2](https://arxiv.org/html/2608.14683#S5.T2)\)\.Trained fusion\.Our deployable rules are heuristics, so the negative in §[5\.4](https://arxiv.org/html/2608.14683#S5.SS4)could be an artifact of choosing the wrong rule, especially since an oracle over\{\\\{LLM, retriever\}\\\}leaves real headroom on the tail\. We therefore fit the obvious harvester: a per\-case “trust the LLM?” classifier on the deployable signals \(LLM verbalised confidence, LLM mean token log\-probability where available, the retriever’s top1−\-top2 margin, and the retriever’s rank of the LLM’s top\-1\), delivering the LLM’s top\-1 when it fires and the retriever’s otherwise, scored on held\-out folds \(55\-fold stratified CV, so no case is decided by a classifier that saw it\)\. We report the better of logistic regression and gradient boosting\. The complementarity is real but not harvestable\. At the frontier an oracle over the two predictors would reach39\.2%39\.2\\%against the retriever’s25\.4%25\.4\\%, yet the trained fusion captures\+0\.2\+0\.2pp and no configuration yields a significant gain; for the small models the classifier correctly learns never to trust the LLM, so the fusion reduces exactly to the retriever\. The estimate is generous to the fusion, being fitted and evaluated on the same tail distribution with no deployment shift, and it still does not help\. That closes the objection that our negative is specific to the two heuristic rules of §[5\.4](https://arxiv.org/html/2608.14683#S5.SS4)\.

Table 12:Retriever alone against two ways of combining it with the LLM, on the ultra\-rare tail\.Δ\\Deltais against retriever\-alone by exact paired McNemar; the oracle column is the ceiling a perfect per\-case choice between the two predictors would reach\.
## Appendix HAudit of SapBERT\-to\-OMIM Linking

Every LLM number here passes through one instrument: a free\-text disease name linked to OMIM by SapBERT nearest\-candidate cosine\. If that instrument degraded on ultra\-rare names, long, eponymous, numbered, synonym rich, it would*manufacture*the collapse\. Two tests bound this, and neither needs annotation\.

Held\-out surface form\.The test must give the linker a form it has not already been handed, or it measures nothing: feeding back a candidate’s own name scores100%100\\%by construction, since the query embedding*is*the candidate embedding\. We use an Orphanet synonym that is not the OMIM title and that exactly one entry claims\. On6,7766\{,\}776such probes the ultra\-rare tail resolves at39\.6%39\.6\\%, against49\.9%49\.9\\%on less\-rare disease \(n=680n\{=\}680; the Wilson intervals do not overlap\)\. Every eligible probe is used and the hash seed is pinned, since sampling probes from an unordered set moves the result by up to33pp\. The linker does degrade on the tail, so we do not rest the argument on it\. The weight falls on the next test, whose alias arm is string matching and does not pass through SapBERT at all\.

Generous re\-scoring\.Credit a model if*any*of its five names links to the gold*or*matches an alias of it, the gold’s OMIM title or an Orphanet synonym, after case, accent and punctuation normalisation\. Orphanet attaches a group’s name to every OMIM entry it references, so any normalised form claimed by more than one entry is discarded \(2,8952\{,\}895of21,16121\{,\}161;9191diseases lose every unambiguous form\)\. Without that guard the rule silently scores at group level: stripping “syndrome”/“type” and sorting tokens leaves one form covering108108unrelated neurodevelopmental entries, and*every*extra credit the retriever received came from a different OMIM entry rather than an alias of the gold\.

Under the corrected rule the small models reach22–8%8\\%on the tail, against≤5%\\leq\\\!5\\%strict, the alias arm adds almost nothing beyond top\-5, i\.e\. the linker was already seeing what it could see\. Applied to the retriever the same rule adds exactly0\.0%0\.0\\%, but that is an identity, not a measurement: after the guard no surviving form is owned by more than one entry, so for an identifier\-predicting system the generous rule reduces to the strict one\. The rule is therefore*one\-sided*: it can credit a name\-generating system and cannot credit an identifier\-predicting one\. That asymmetry is why the arithmetic below, not this0\.0%0\.0\\%, is what closes the objection\.

What closes the objection is arithmetic rather than either test on its own\. Take the best small model’s strict tail Recall@1 \(4\.6%4\.6\\%\) and inflate it by the linker’s own held\-out resolution rate on that stratum, as though*every*miss the linker makes were a correct answer thrown away:11\.6%11\.6\\%\. That is a deliberately over\-generous correction and it still leaves the best small model less than half the retriever’s25\.6%25\.6\\%\. What none of this establishes is the linker’s*precision*, whether a link goes to the right disease rather than a plausible neighbour\. That needs a genetics\-literate adjudicator and we do not report one; it is a different quantity from the one the objection turns on\.

## Appendix ICross\-Task Transfer and Boundary Conditions

This appendix carries the two decisions to standard ranking tasks, in the order the framework poses them: which statistic reads top\-1 correctness, which reads candidate presence, what conditions the margin needs before either question is well posed, and whether the comparisons survive being read off the whole risk–coverage curve rather than one operating point\.

### I\.1Top\-1 Correctness across Scorers

Entity linking’s standard NIL rule thresholds the top\-1 score rather than the margin\(Sevgiliet al\.,[2022](https://arxiv.org/html/2608.14683#bib.bib50)\), exactly the quantity that fails for our prioritizers\. On12,74112\{,\}741BC5CDR disease mentions linked to MEDIC with SapBERT the two signals are*not distinguishable*: at the10%10\\%operating point this paper uses throughout the margin is nominally ahead \(92\.9%92\.9\\%vs\.92\.2%92\.2\\%overall;96\.6%96\.6\\%vs\.93\.2%93\.2\\%on the rare\-concept tail,n=591n\{=\}591\) while by AUROC the raw cosine is nominally ahead \(0\.8220\.822vs\.0\.7960\.796\), and neither gap survives a bootstrap clustered on the3,2013\{,\}201unique mention strings the corpus actually contains \(Δ\\DeltaAUROC−0\.027\-0\.027,95%95\\%CI\[−0\.075,\+0\.021\]\[\-0\.075,\+0\.021\]; the sign is the margin minus the raw score, so the point estimate favours the raw score and neither arm is separated\)\. The tie is what the screen calls for rather than a failure of it:63\.7%63\.7\\%of mentions are exact string matches, so the raw cosine is saturated; on the4,6294\{,\}629mentions where it is not \(base53\.4%53\.4\\%\) the AUROC ordering*reverses*, margin0\.7700\.770against raw0\.7640\.764\. That is a subgroup observation and not a screen, since §[3\.3](https://arxiv.org/html/2608.14683#S3.SS3)rules out screens of this kind, and it is reported here because the split is diagnostic rather than predictive: where the score has spare resolution the margin is ahead, and where it has saturated the comparison is decided by ties\. The raw arm’s92\.2%92\.2\\%is in fact the most favourable of its tie\-breaks, with808808mentions tied at the10%10\\%cut against the margin’s4949, so the operating point here is as tie\-sensitive as the one we criticise in Exomiser\. The reversal on the unsaturated subset is not itself significant\. What we can say is that the recommendation is established for*unnormalized accumulation*scores, IC\-weighted overlap and Exomiser’s combined score, and is untested for already\-normalized similarities\. The test the scope claim calls for holds the corpus and the candidates fixed and changes only the scorer’s normalization: replacing SapBERT cosine with BM25, an unnormalized accumulation of IDF\-weighted matches, makes the margin significantly better than the raw score by AUROC \(0\.7930\.793vs\.0\.7430\.743; bootstrap clustered on unique mention strings,Δ\\Delta95%95\\%CI\[\+0\.017,\+0\.082\]\[\+0\.017,\+0\.082\]\), with the same ordering on the rare\-concept tail \(0\.8000\.800vs\.0\.7630\.763\); subtracting the per\-mention nuisance the score accumulates, its query IDF mass, is better still \(0\.8080\.808\)\. The crossover is in the predicted direction and was registered before the run\. What it establishes is a*ranking*result: at the10%10\\%operating point the raw score is nominally ahead by8\.28\.2pp, but that gap is not resolved \(95%95\\%CI−0\.6\-0\.6–\+17\.5\+17\.5, straddling zero\), so we claim the crossover for the ranking and nothing for the operating point\. A third arm runs a deployed late\-interaction retriever\. We score300300SciFact queries with the released ColBERTv2 checkpoint under its own configuration \(128128\-dimensional projection, queries padded to3232tokens with\[MASK\], documents truncated to180180with punctuation zeroed, cosine MaxSim\), so the scoring function is the model’s own and not a reimplementation\. Its top score carries a per\-query level \(C=0\.374C\\\!=\\\!0\.374\), gates at AUROC0\.7640\.764against a base of54\.3%54\.3\\%, and its own margin reaches0\.8360\.836\.

The SUM\-versus\-MEAN pair reported below runs on our own encoder rather than this one, and the reason is structural\. On our encoder the top score’s scale tracks query length atρ=0\.97\\rho\\\!=\\\!0\.97and query\-length normalization recovers about half of what the margin recovers\. The deployed model has no such dependence,ρ=\+0\.003\\rho\\\!=\\\!\+0\.003, because ColBERT’s query augmentation pads every query to the same3232tokens, so its sum runs over a constant number of terms and dividing by query length divides by a constant\. The two aggregations are then the same ranking and the same score up to scale, so on the deployed model there is no SUM\-versus\-MEAN contrast to draw\. On our own encoder there is, and it is a controlled manipulation of normalization with the corpus, the candidates and the encoder held fixed, one of the two controlled pairs §[5\.5](https://arxiv.org/html/2608.14683#S5.SS5)uses to testCC\. The two arms make different points: the pair isolates normalization, and the deployed run shows a large between\-input share and a large margin advantage in a system we did not build\.

### I\.2Candidate Presence

Link or NIL: the same score read the other way\.Every BC5CDR mention above has a gold concept in MEDIC, so those arms measure only whether the top\-1 is correct\. The decision a deployed linker also faces is whether to link at all, and the decomposition of Eq\. \([3](https://arxiv.org/html/2608.14683#S3.E3)\) sends the two decisions to different statistics: correctness is a question aboutr1−r2r\_\{1\}\-r\_\{2\}, whereas a mention whose gold concept is absent is marked by the whole candidate set scoring low, which isb​\(x\)b\(x\), the term the margin cancels\. We therefore predicted, before running it, that the margin would*lose*the NIL decision under both scorers, and that the raw top score would be a good NIL detector only for the scorer whoseb​\(x\)b\(x\)is small\.

FollowingZhuet al\.\([2023](https://arxiv.org/html/2608.14683#bib.bib83)\), we mask a random slice of MEDIC out of the candidate space, so mentions whose gold concepts are all masked become genuinely unlinkable; masking10/25/50%10/25/50\\%of concepts makes10\.0/25\.8/44\.7%10\.0/25\.8/44\.7\\%of the12,74112\{,\}741mentions NIL\. Both questions are then asked of one run, with bootstrap intervals clustered on the mention string\.

rawmargingap@1010level*QNIL: is the gold concept in the KB?*SapBERTmask10%10\\%0\.842\\mathbf\{0\.842\}0\.7670\.7670\.8250\.825–mask25%25\\%0\.841\\mathbf\{0\.841\}0\.7870\.7870\.8320\.832–mask50%50\\%0\.8430\.8430\.7970\.7970\.845\\mathbf\{0\.845\}–BM25mask10%10\\%0\.6730\.6730\.7250\.7250\.6360\.6360\.775\\mathbf\{0\.775\}mask25%25\\%0\.6570\.6570\.7210\.7210\.6450\.6450\.767\\mathbf\{0\.767\}mask50%50\\%0\.7020\.7020\.6890\.6890\.6570\.6570\.771\\mathbf\{0\.771\}*Qcorrect: among linkable, is the top\-1 right?*SapBERTmask25%25\\%0\.869\\mathbf\{0\.869\}0\.8310\.8310\.8620\.862–BM25mask25%25\\%0\.7600\.7600\.8080\.8080\.7610\.7610\.850\\mathbf\{0\.850\}Table 13:Two abstention decisions on one corpus \(AUROC\)\.*margin*and*gap@1010*are both zero\-sum contrasts over the candidates,w=\(1,−1,0,…\)w=\(1,\-1,0,\\dots\)andw=\(1,−19,…,−19\)w=\(1,\-\\tfrac\{1\}\{9\},\\dots,\-\\tfrac\{1\}\{9\}\), so by Eq\. \([5](https://arxiv.org/html/2608.14683#S3.E5)\) each cancelsb​\(x\)b\(x\)exactly\.*level*is instead the top score with a case\-side nuisance estimate removed, the query’s own IDF mass, which only BM25 admits: SapBERT’s cosine is bounded and its raw score is already the level, hence the dashes\.Table[13](https://arxiv.org/html/2608.14683#A9.T13)reports it\. Under SapBERT the raw cosine is the better NIL detector at every masking rate and the margin is worse by a paired−0\.054\-0\.054\(95%95\\%CI\[−0\.088,−0\.023\]\[\-0\.088,\-0\.023\]at25%25\\%\), as predicted\. Under BM25 the raw score is a far weaker NIL detector,0\.6570\.657–0\.7020\.702against the cosine’s0\.8410\.841–0\.8430\.843, because its level is dominated by the query’s own IDF mass rather than by evidence about the knowledge base; subtracting that mass beats the raw score by\+0\.102/\+0\.109/\+0\.067\+0\.102/\+0\.109/\+0\.067across the three rates, every interval excluding zero\. BM25 is the one scorer here that carries both families on one run, so the two can be told apart rather than argued about: alongside that case\-side level sits the top score against the mean of the field, a zero\-sum contrast by Eq\. \([5](https://arxiv.org/html/2608.14683#S3.E5)\), and it is the*worst*of the four statistics at every rate \(0\.6360\.636–0\.6570\.657against the level’s0\.7670\.767–0\.7750\.775\), falling behind even the raw score at50%50\\%masking \(−0\.045\-0\.045,\[−0\.076,−0\.015\]\[\-0\.076,\-0\.015\]\)\. Estimating the case level off the contenders does not approximate it; it removes it\. The same protocol on MedMentions replicates the core and is reported here rather than only counted\. Disease\-mention linking against UMLS on a PMID\-disjoint split \(n=3,831n\{=\}3\{,\}831evaluation mentions\) gives a knowledge base with2\.12\.1surface forms per concept against MEDIC’s seven, so the aliases the cosine relies on are far thinner\. Over the same66\(masking rate×\\timesscorer\) cells the margin is the bestQNILQ\_\{\\mathrm\{NIL\}\}statistic in none and is resolvedly behind the raw score in33: at10%10\\%masking under SapBERT the raw cosine reaches0\.9020\.902against the margin’s0\.6840\.684, a paired−0\.216\-0\.216\(\[−0\.320,−0\.117\]\[\-0\.320,\-0\.117\]clustered on the mention string\), three times the size of the same quantity on BC5CDR\. What does not travel is the BM25 half, and the thin alias base is why\.

Our registered prediction had two clauses and one of them failed\. The margin is never the bestQNILQ\_\{\\mathrm\{NIL\}\}statistic under either scorer, and it stays ahead of the raw score onQcorrectQ\_\{\\mathrm\{correct\}\}under BM25; that dissociation is what the decomposition predicts, and it holds in all six cells here and all six on MedMentions\. But we also predicted the margin would lose to the raw score under*both*scorers, and under BM25 it does not: it is ahead at two of the three masking rates\. That clause was wrong, for a reason the decomposition itself supplies: BM25’s raw score is not a clean reading ofb​\(x\)b\(x\), being dominated by the query’s own IDF mass, so beating it is not evidence aboutbbat all\. Subtracting that mass restores the predicted ordering at every rate\. What the decomposition licenses is that whichever statistic readsb​\(x\)b\(x\)cleanly wins the presence question, not that the raw score is that statistic in a scorer where the two come apart\.

One artifact of the protocol works against the margin and we state it rather than argue it away: masking removes competitors at random, which perturbs the runner\-up and so adds noise to the margin specifically, whereas a linkable mention’s gold survives by construction and its top score does not move\. The three reported rates are nested prefixes of one permutation, so they are one draw and not three replicates, and we do not read a trend across them\. What we do read is that the margin is behind at all three rates and behind under both scorers, and that the case\-side level, which is perturbed by the same masking, is ahead\.

The same split on passage retrieval\.A reviewer may read the dissociation as a property of short\-mention linking rather than of ranking scores, so the masking protocol is transplanted unchanged to passage retrieval\. Masking a random slice of SciFact’s5,1835\{,\}183abstracts makes a query unanswerable when every one of its qrel\-positive abstracts is gone;1010/2525/50%50\\%masking leaves8\.78\.7/23\.723\.7/50\.3%50\.3\\%of the300300queries unanswerable\. ColBERTv2 scores what remains\. No case\-side nuisance estimate is available for a late\-interaction score, so the third statistic here is the top score against the mean of the nine candidates behind it, which Eq\. \([5](https://arxiv.org/html/2608.14683#S3.E5)\) places in the same cancelling family as the margin rather than alongside BM25’s level\.

Table 14:The dissociation on passage retrieval \(AUROC, ColBERTv2 on SciFact\)\.*gap@1010*is the top score against the mean of the field, a zero\-sum contrast like the margin\. The margin is the*worst*of the three for whether a relevant abstract survives, at every masking rate, and beats the raw score on whether the top\-1 is right, at every masking rate\.Table[14](https://arxiv.org/html/2608.14683#A9.T14)reports it\. The margin is the bestQexistsQ\_\{\\mathrm\{exists\}\}statistic at none of the three rates, being in fact the worst of the three at all of them, and it beats the raw score onQcorrectQ\_\{\\mathrm\{correct\}\}at all of them, by\+0\.043\+0\.043to\+0\.080\+0\.080\. The third column is the one to read carefully\. It was registered as a level, and Eq\. \([5](https://arxiv.org/html/2608.14683#S3.E5)\) says it is not one: it cancelsb​\(x\)b\(x\)exactly as the margin does, so the framework gives it no claim on the presence question\. At the lightest masking it duly loses to the raw score \(0\.6730\.673against0\.6960\.696\); at2525and50%50\\%it passes, and what passes there is not a recovered level but a field that has been thinned until its own flatness is informative\. BM25 is where the two readings separate, because only there is a genuine case\-side estimate available to compete: the same contrast is the worst of four at every rate while the case\-side level is the best\. What the decomposition forbids is a contrast winning the presence question against a statistic that readsb​\(x\)b\(x\), and no cell here or on the two linking corpora violates it\.

### I\.3Boundary Conditions

A top1−\-top2 margin measures competition between candidates, so it is informative only where the top two are genuinely different answers\. That is a condition on the candidate set and not on the task label: SciFact is passage retrieval and the margin gains\+0\.072\+0\.072AUROC there, and its top two abstracts are genuine alternatives rather than paraphrases: they share7\.3%7\.3\\%of their content words against3\.0%3\.0\\%for a random same\-corpus pair, and0%0\\%of queries have a near\-duplicate pair\. Where the top two*are*near\-duplicates,r1≈r2r\_\{1\}\\\!\\approx\\\!r\_\{2\}by construction and their difference is noise: sampled generation fails the condition outright\. Our own domain satisfies the condition by construction, one gold OMIM disease among8,5538\{,\}553, as do entity linking and concept normalization against a fixed knowledge base\. Before transferring the recommendation the question to ask is therefore whether a system’s runner\-up is a rival answer or a paraphrase of the leader; where it is a paraphrase, de\-duplicating the candidate list restores the condition rather than defeating it\.

The margin also needs a ranked candidate set scored by one scorer, and repeated samples from a generator are not that\. Drawingk=10k\{=\}10independent continuations per case from an instruction\-tuned77B model \(n=600n\{=\}600, base17\.3%17\.3\\%\), the margin between the best and second\-best sample carries no usable signal \(AUROC0\.4090\.409, disease\-clustered CI0\.3780\.378–0\.5240\.524\) while dividing the sequence log\-probability by its length reaches0\.8720\.872\. The reason is structural: in87\.7%87\.7\\%of cases the two highest\-scoring samples are the identical string, so there is no runner\-up to measure a gap against\. Nor is the length divisor removing a nuisance there, the selected sample’s length and its raw score are uncorrelated \(ρ=−0\.03\\rho\\\!=\\\!\-0\.03\), but longer answers happen to be likelier correct \(length alone gives AUROC0\.7660\.766\), so dividing by it injects a task\-specific cue rather than cancelling a scale\. The same operation is nuisance\-removal in one setting and signal\-injection in the other, which is why the recommendation has to name the structure it needs\. The base\-accuracy criterion carries no such restriction; it is the*gate*that does\.

### I\.4Whole\-Curve Evaluation

A comparison made at one operating point need not survive at another, andTraubet al\.\([2024](https://arxiv.org/html/2608.14683#bib.bib68)\)propose the area under the generalized risk–coverage curve \(the mean over thresholds ofP​\(fail and accept\)P\(\\text\{fail and accept\}\), which unlike the ordinary risk–coverage integral does not condition on the accepted set\) as the threshold\-free alternative\. It cannot serve as an independent check of our comparisons\. Writingppfor base accuracy, that area satisfiesAUGRC=p​\(1−p\)​\(1−AUROC\)\+12​\(1−p\)2\\mathrm\{AUGRC\}=p\(1\-p\)\\,\(1\-\\mathrm\{AUROC\}\)\+\\tfrac\{1\}\{2\}\(1\-p\)^\{2\}; the raw score and the margin are two readings of one predictor’s output and therefore shareppexactly, so at fixedppthe area is a decreasing affine function of AUROC and the two metrics order the pair identically\. We verified the identity on our own runs, to within10−510^\{\-5\}of the empirical area\. We therefore report AUROC beside the operating point and claim nothing further from the threshold\-free version\. Our retriever on the ultra\-rare tail:0\.3170\.317for the margin against0\.3430\.343for the raw score, paired difference\[−0\.037,−0\.017\]\[\-0\.037,\-0\.017\]clustered on disease\. Phenotype\-only Exomiser:0\.4070\.407against0\.4470\.447on the ultra\-rare stratum\. BC5CDR with BM25:0\.1020\.102against0\.1130\.113\. BC5CDR with SapBERT, where we report a tie:0\.0610\.061against0\.0560\.056, paired\[−0\.004,\+0\.012\]\[\-0\.004,\+0\.012\], which straddles zero\. Every cell that the operating point calls for the margin the whole curve also calls for the margin, and the one tie stays a tie\. The reverse does not hold everywhere: under BM25 the whole curve favours the margin while the10%10\\%operating point favours the raw score by8\.28\.2pp, an unresolved gap either way, which is why the entity\-linking claim in §[5\.6](https://arxiv.org/html/2608.14683#S5.SS6)is stated by AUROC and not at an operating point\.

## Appendix JClinical Grading of Residual Errors

Exact\-match accuracy treats every wrong answer alike\. A clinician does not: a different genetic subtype of the disorder the patient actually has sends them to the same panel, and a disease of another organ system sends them elsewhere\. We grade each wrong top\-1 on three axes, none of which any system here is scored on or optimises for\.Same\-test: the predicted and true disease share a causative gene, or map to the same Orphanet disorder\.Same\-class: they share an ICD\-10 category\.Unrelated: measurable on at least one axis, related on none\.Unmeasurableis kept as its own bucket and never folded into “unrelated”,9\.6%9\.6\\%of the candidate space carries no gene, no Orphanet mapping and no ICD\-10 code, and calling that unrelated would manufacture harm\.

Decontamination applies to every candidate\.Leave\-source\-out drops terms sourced solely to the case’s own publication, and it is applied*symmetrically*, to the gold profile and to every competitor\. Stripping only the gold would leave a same\-gene sibling annotated from this very patient, routine, since one paper often reports several allelic entries, holding evidence the gold had lost; such a candidate wins on curation provenance and not on clinical adjacency\. The leakage sweep of App\.[D\.1](https://arxiv.org/html/2608.14683#A4.SS1)retains the gold\-only rule, so its bracket is on that rule and we do not claim otherwise\. Per\-bin Recall@1 moves by up to0\.90\.9pp between runs and between float32 and float64, so we grade the gates and report no tail\-accuracy delta\.

Three design choices decide whether these numbers mean anything\.*\(i\)*ICD\-10 is taken strictly: rare syndromes pile into residual buckets\. Q87 “other specified congenital malformation syndromes” and kin, which we identify from the data as the categories carrying more than150150distinct disorders, so a shared category counts only if it is neither a catch\-all nor a “\.8/\.9” residual subdivision\. The loose rule is computed alongside and would move137137of the deferred “unrelated” errors into same\-class; none of the answered ones\. Strictness is the conservative direction, since it makes*unrelated*larger\.*\(ii\)*Phenotypic overlap is deliberately excluded as a grading axis for the retriever: it ranks by IC\-weighted phenotype overlap, so grading its errors that way is circular\. \(For the record it goes the same way,0\.6310\.631answered vs\.0\.5200\.520deferred; we do not use it\.\)*\(iii\)*The gene table ships with Exomiser’s2402release and predates the corpus, which alone put237237gold diseases in the unmeasurable bucket, detectable because their median OMIM identifier is visibly newer\. Phenopacket Store records each case’s causative gene as its directory, so we merge those293293pairs in\. Coverage stays asymmetric, gold\-side is near\-complete, prediction\-side is not, which biases*against*finding same\-test, so the reported share is a lower bound\.

Two worked pairs, so the grading can be checked by eye, and so the limit of it is visible\. In one answered case the true diagnosis is*Greig cephalopolysyndactyly*and the retriever answers*postaxial polydactyly, types A1 and B*; exact match scores this zero, both are*GLI3*, both are dominant, and the test the suggestion triggers is the test that finds the answer\. In another the truth is*Robinow syndrome, autosomal recessive*and the answer is*Robinow syndrome, autosomal dominant 1*: the same Orphanet disorder, so a panel still finds it, but the inheritance is inverted and the recurrence risk a family would be quoted is wrong\. The first pair is the case for the finding; the second is why it must be scoped\.

Table 15:How wrong the wrong answers are, under symmetric leave\-source\-out, for the cases the gate answers against those it defers\. The LLM column is Qwen2\.5\-VL\-7B, fifth of our eight small models by tail Recall@1 \(1\.7%1\.7\\%\), on its wrong top\-1 in the ultra\-rare stratum\.Table[15](https://arxiv.org/html/2608.14683#A10.T15)gives the result, on the globally answered decile,200200of the20002000sampled cases,*not*the ultra\-rare\-stratified top\-10%10\\%\(n=478n\{=\}478\) of App\.[F\.3](https://arxiv.org/html/2608.14683#A6.SS3)\. The account predicts the opposite sign, a large top1–top2 margin means the runner\-up is far, so a near\-neighbour true diagnosis should itself have scored high and shrunk the margin, so this contradicted our expectation rather than confirming it\.

Robustness across grading specifications\.The shift is positive under every specification we tried, but its size and its significance are not stable:\+15\.8\+15\.8pp as reported,\+14\.8\+14\.8pp if the tie rule is applied consistently \(below\),\+14\.4\+14\.4pp with no tie exclusion at all, and\+10\.1\+10\.1pp per\-disease\. Only the first two exclude zero\. We therefore claim a*direction*and not a resolved effect\.

Ties are excluded on both sides\. No top\-1 is committed where candidates tie at the maximum, so those cases are excluded from the grading:281281where the true diagnosis is among the tied and a further301301where it is not\. Both groups sit entirely in the comparison baseline and both have same\-test rates about twice it, so excluding only the first would inflate the shift\. The symmetric exclusion is the\+14\.8\+14\.8pp figure above\.

The same test does not imply the same counselling\.Of the1515same\-test errors,44share a causative gene and1111share only the Orphanet disorder, locus heterogeneity, covered by a panel or exome but not necessarily by a single\-gene test\. More consequentially,77of the1515pair a purely dominant entity with a purely recessive one \(Robinow, distal renal tubular acidosis\)\. For those patients the assay is right and the recurrence risk, the relatives selected for cascade testing and the reproductive counselling are all wrong, the three actions our own Ethics Statement names as the harm channel\. The grading axes are blind to inheritance by construction; we added the check only after the fact, and it removes about half of the comfort the same\-test bucket appears to offer\.

Exomiser admits the same grading at gene level, relating two genes through the disorders they cause\. Its answered decile is10\.3%10\.3\\%same\-test,1\.7%1\.7\\%same\-class,62\.1%62\.1\\%unrelated and25\.9%25\.9\\%unmeasurable \(n=58n=58\)\. The last figure is eightfold the retriever’s3\.4%3\.4\\%, so the two are not on a comparable denominator and we do not read the comparison as a ranking; renormalised to measurable errors it is14\.0%14\.0\\%against the retriever’s26\.8%26\.8\\%\. A phenotype\-only run of a genome\-aware tool should be expected to do worse here, which is one more reason not to read our Exomiser numbers as a verdict on Exomiser\.

Margin confidence and general case difficulty\.Every row above is a difficulty proxy, so a sharper version of the objection survives: perhaps the margin identifies cases that are easy for*any*predictor, in which case “retriever\-first” would weaken to “gate externally, then let either predictor deliver\.” We test it by scoring the*bare LLM*on exactly the cases the margin answers \(Table[16](https://arxiv.org/html/2608.14683#A10.T16),N=2000N\{=\}2000per model\)\. Both predictors do improve inside the band, so a shared\-difficulty component is real, but they do not converge: the retriever reaches81\.181\.1–83\.9%83\.9\\%against the LLM’s1\.11\.1–44\.0%44\.0\\%, an advantage of\+39\+39to\+83\+83pp for every one of the ten configurations, including DeepSeek\-V4\-Pro with reasoning*on*\(44\.0%44\.0\\%vs\.83\.3%83\.3\\%\)\. For two of the weaker models the margin is mildly*anti*\-correlated with LLM correctness \(1\.7→1\.1%1\.7\\\!\\to\\\!1\.1\\%,1\.6→1\.1%1\.6\\\!\\to\\\!1\.1\\%\)\. The gate therefore selects cases*the retriever*solves, not cases that are easy in general, which is exactly the predictor\-selection reading\.

Table 16:Recall@1 \(%\) on the ultra\-rare tail, overall and restricted to the decile the retriever’s own margin answers\. The retriever column varies only with the case set a given model produced parseable output for; BioMistral is omitted, emitting a well\-formed differential in∼0\.4%\{\\sim\}0\.4\\%of cases\.
## Appendix KTable Notes

Table[1](https://arxiv.org/html/2608.14683#S5.T1)\.“less/ultra ratio”==less\-rare over ultra\-rare per\-case R@1;*per\-disease the LLM collapse is only∼2\\sim\\\!2–6×6\\times*\(App\.[C\.2](https://arxiv.org/html/2608.14683#A3.SS2)\)\. Yi is weak on both bins, so its ratio does not describe a fall from a working anchor; we report it rather than dropping it\. The frontier model edges just above Exomiser’s \(gene\-level, not directly comparable\) score\. For the retriever row, a stricter same\-publication control brackets the tail at5\.25\.2–25\.6%25\.6\\%, where it no longer clears the frontier model \(App\.[D\.1](https://arxiv.org/html/2608.14683#A4.SS1)\), and its per\-case1\.2×1\.2\\times*reverses*to a per\-disease rise \(App\.[C\.2](https://arxiv.org/html/2608.14683#A3.SS2)\)\. BioMistral\-7B is medically adapted and follows the requested output format poorly \(App\.[L](https://arxiv.org/html/2608.14683#A12)\)\. Exomiser’s gene\-level score is not directly comparable to the disease\-level rows: it dwarfs the small\-model tail and is comparable to our retriever \(App\.[F\.1](https://arxiv.org/html/2608.14683#A6.SS1)\)\.

Table[2](https://arxiv.org/html/2608.14683#S5.T2)\.The ceiling ismin⁡\(1,p/c\)\\min\(1,p/c\)at the LLM’s own base rate, so no confidence estimator, verbalized, log\-probability, semantic entropy or conformal, can close it \(§[5\.2](https://arxiv.org/html/2608.14683#S5.SS2)\)\. The LLM’s*observed*band is reported at the coverage its confidence can actually resolve: that signal takes only55–1414distinct values here, so a10%10\\%band does not exist for it \(for Qwen\-32B*no*case is strictly above the cut, all9090would come from one140140\-case tie block\), whereas the margin has682682values and resolves10%10\\%exactly\. The mid\-size models rank their own answers well \(own\-answer AUROC0\.780\.78/0\.900\.90; App\.[E\.2](https://arxiv.org/html/2608.14683#A5.SS2)\) and still cannot reach a safe point: the barrier is base rate, not discrimination\. The retriever\-first row is the deployable hybrid of §[5\.4](https://arxiv.org/html/2608.14683#S5.SS4), so its spread is the LLM’s marginal effect on the*same*cases \(≈94\{\\approx\}94cases, per\-cell95%95\\%CI≈±9\{\\approx\}\\pm 9pp; the margin\-vs\-ceiling separation far exceeds that; per\-disease the band is64\.6%64\.6\\%, App\.[F\.3](https://arxiv.org/html/2608.14683#A6.SS3)\)\. Under the strict leakage control of App\.[D\.1](https://arxiv.org/html/2608.14683#A4.SS1)the retriever’s base falls to5\.2%5\.2\\%and this band with it \(12\.3%12\.3\\%\)\. The25%25\\%\-coverage point and the dominated agreement gate are in App\.[G](https://arxiv.org/html/2608.14683#A7)\. Scored under the tie\-breaking most favourable to the LLM, the margin’s band on the small models is7474–83%83\\%against at most33–28%28\\%for verbalized confidence at matched10%10\\%coverage\.

## Appendix LReproducibility

Compute\.All local runs are on one machine with four NVIDIA RTX60006000Ada Generation GPUs \(4848GB each\), driver570\.195\.03570\.195\.03, CUDA12\.812\.8\. Open\-weight inference is single\-GPU except the3232B AWQ model, which is tensor\-parallel across four\. The frontier configurations are API calls and consume no local GPU time, and the decontamination, entity\-linking and retrieval analyses are CPU\- or single\-GPU jobs\. We release the per\-run logs rather than a single total, since the runs were made incrementally over the project\.

Software\.Python3\.113\.11, PyTorch2\.112\.11built against CUDA12\.812\.8, vLLM for open\-weight decoding, andtransformersfor SapBERT\. All data are public: Phenopacket Store v0\.1\.27, HPO/HPOA release 2026\-06\-23, and Orphanet cross\-references\. We release the full pipeline \(parsing, per\-model inference, stratified analysis, the true\-path\-propagated retriever with source\-decontamination, the rerank probe with its shuffle control, and the retriever\-margin triage and hybrid analysis\), the exact prompts, model revisions, the vLLM version, and per\-case outputs including the matched OMIM labels\. Free\-text names are linked by nearest\-neighbor cosine over the OMIM vocabulary with SapBERT \(cambridgeltl/SapBERT\-from\-PubMedBERT\-fulltext\); this linking is sensitive to the transformers/model version, which we pin, it degenerates undertransformers5\.0, so we fix the version under which linking is stable \(the retriever and rerank numbers use no name matching and reproduce exactly\)\. Decoding is greedy \(temperature0\)\. The three Qwen models and Llama\-3\.1\-8B emit a well\-formed five\-item differential in\>99%\{\>\}99\\%of cases; BioMistral does so in only∼0\.4%\\sim\\\!0\.4\\%\(and a parseable overall confidence in∼1\.5%\\sim\\\!1\.5\\%\), so its low recall partly reflects instruction\-following failure and we do not lean on it\.Frontier models\.DeepSeek\-V4\-Flash and DeepSeek\-V4\-Pro are queried through the DeepSeek API on the identicalN=2000N\{=\}2000sample and prompt, temperature0, with reasoning*disabled*\(thinking:\{type:disabled\}\) so the comparison to the greedy open models is fair; both emit a well\-formed differential in100%100\\%of cases\. The API returns no usable token log\-probabilities, so the log\-probability signal \(§[5\.2](https://arxiv.org/html/2608.14683#S5.SS2)\) is reported for the open\-weight models only; all other signals \(verbalized confidence, retriever margin/agreement\) are computed the same way for every model\. The*reasoning\-on*run uses the same endpoint and prompt withthinkingenabled;90%90\\%of cases return well\-formed output within the token budget \(exhaustion on the hardest cases\), and reasoning\-on numbers are computed on that subset\.

Determinism and seeds\.TheN=2000N\{=\}2000evaluation sample is drawn once with seed0and reused by every system, so all rows are scored on the same cases \(Table[1](https://arxiv.org/html/2608.14683#S5.T1)states the two exceptions, which are parsing failures rather than different samples\)\. Masking in the entity\-linking and retrieval arms uses seed0, and the three rates are nested prefixes of one permutation rather than three independent draws\. Retriever figures throughout usePYTHONHASHSEED=0\. Changes in floating\-point summation order still move the base rate and the raw\-score gate by about0\.50\.5pp, and the margin gate reads79\.7%79\.7\\%or81\.0%81\.0\\%according to which decontamination implementation produced the run, a1\.31\.3pp spread we quote rather than average\. Numbers sensitive to this are the retriever’s absolute levels; the margin\-versus\-raw ordering is not, since both are computed from the same run\.

Artifacts and terms\.Every corpus we use is a publicly released research resource: Phenopacket Store, HPO and HPOA, Orphanet, SciFact, BC5CDR and MedMentions, each used under its own published terms\. The model weights are the vendors’ public releases at the revisions we pin\. We release code, prompts, the parsing and linking pipeline, and per\-case derived outputs including matched OMIM identifiers; we redistribute no source case\-report text and no patient attributes beyond the structured HPO terms and gold labels the corpora already publish\.