Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

# Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
Source: [https://arxiv.org/html/2608.17994](https://arxiv.org/html/2608.17994)
Ali EmamiAffiliation:Emory University\{sh545346,hsajjad\}@dal\.caali\.emami@emory\.eduHassan SajjadAffiliation:Dalhousie UniversityAffiliation:New York University Abu Dhabi

###### Abstract

Using LLMs as judges has become standard practice for evaluating model outputs at scale\. This is particularly common for subjective, open\-ended tasks such as assessing helpfulness or alignment, where no single reference answer exists\. However, objective tasks introduce a distinct reliability challenge for reference\-free LLM judging\. In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or through tool augmentation\. Although the former enables efficient evaluation, the judge may hallucinate or lack sufficient evidence for its verdict\. Conversely, tool augmentation can provide additional evidence but introduces extra computational cost and requires an appropriate mechanism to determine when and how that evidence should be used reliably\. More importantly, neither approach alone provides formal control over the risk of accepted verdicts or guarantees their reliability at a specified level\. We propose a risk\-controlled framework that calibrates uncertainty thresholds on a held\-out set so that the false discovery rate among accepted verdicts remains below a user\-specified levelα\\alphawith high probability, using finite\-sample Clopper–Pearson intervals\. When the parametric mode is not sufficiently confident, the instance is routed to a retrieval\-augmented mode, where the judge gathers web evidence and re\-evaluates the instance under a second calibrated threshold\. The finite\-sample guarantee carries over to this two\-threshold routing without additional assumptions\. Across open\-domain QA benchmarks and judges of varying scales, the framework maintains the target error rate while achieving substantially higher coverage than single\-mode baselines\.

## 1Introduction

LLM\-as\-a\-judge has emerged as a scalable alternative to human evaluation for assessing language model outputs\([39](https://arxiv.org/html/2608.17994#bib.bib1)\)\. It enables practitioners to evaluate thousands of model outputs at a fraction of the cost and time required for human annotation\([11](https://arxiv.org/html/2608.17994#bib.bib2);[29](https://arxiv.org/html/2608.17994#bib.bib3)\)\. However, most existing work employs LLM judges for subjective evaluation\([6](https://arxiv.org/html/2608.17994#bib.bib7)\), such as scoring outputs for helpfulness, harmlessness, or style in reward modeling, where no single correct answer exists and approximate agreement suffices\([38](https://arxiv.org/html/2608.17994#bib.bib8);[19](https://arxiv.org/html/2608.17994#bib.bib10)\)\. Extending LLM\-as\-a\-judge to objective, factual evaluation remains an open challenge\.

In factual and open\-domain QA, answers are either correct or incorrect, and the judge must make this determination reliably\. However, obtaining ground\-truth annotations for this purpose is expensive at scale\([9](https://arxiv.org/html/2608.17994#bib.bib4);[22](https://arxiv.org/html/2608.17994#bib.bib5)\), requires domain expertise for specialized questions, and is entirely infeasible when LLMs operate as autonomous agents answering open\-ended queries in real\-world environments where no pre\-collected labels exist\. LLM\-as\-a\-judge offers a reference\-free alternative that eliminates this dependency\. Nevertheless, utilizing LLM judges to assess correctness without explicit references introduces two key reliability challenges\([26](https://arxiv.org/html/2608.17994#bib.bib9);[7](https://arxiv.org/html/2608.17994#bib.bib6)\)\.

First, LLM judges are bounded by their training data and may lack knowledge of recent events, rare entities, or specialized domains\([36](https://arxiv.org/html/2608.17994#bib.bib17)\)\. Such limitations can prevent reliable distinction between correct answers and plausible fabrications\. Second, even if the judge has the relevant knowledge, hallucinations may still cause it to produce confident but incorrect verdicts\([12](https://arxiv.org/html/2608.17994#bib.bib18)\)\. In both cases, the failures are silent as the judge provides no indication that its verdict is unreliable\. Without a mechanism to quantify and act on this uncertainty, incorrect evaluations can propagate undetected into downstream decisions\.

Selective evaluation provides a natural mitigation strategy by allowing the judge to abstain when uncertain and produce a verdict only when it is likely to be correct\. Uncertainty quantification methods can identify unreliable verdicts\([27](https://arxiv.org/html/2608.17994#bib.bib21)\)\. However, applying a heuristic threshold to these scores provides no formal guarantee on the error rate among accepted evaluations\([5](https://arxiv.org/html/2608.17994#bib.bib11)\)\. Recent work on risk\-controlled selective prediction addresses this gap by calibrating statistically valid thresholds that bound the False Discovery Rate \(FDR\) among selected outputs with high probability\([30](https://arxiv.org/html/2608.17994#bib.bib14);[33](https://arxiv.org/html/2608.17994#bib.bib13)\)\. However, existing methods are designed to filter examinee answers in Question\-Answering \(QA\) tasks rather than control errors in a meta\-evaluation setting\. Moreover, earlier methods treat abstention as the only recourse for uncertain instances\. This discards cases where the judge’s uncertainty stems from a knowledge gap or hallucination that could be resolved with additional evidence\([4](https://arxiv.org/html/2608.17994#bib.bib22)\)\.

In this paper, we propose a risk\-controlled selective evaluation framework for LLM\-as\-a\-judge that addresses the above limitations\. Inspired by LLM\-based agents that leverage external tools to overcome their parametric limitations, we equip the judge with web search as a tool to bridge its knowledge gaps\. Our framework operates in two modes\. In Mode 1, the judge model evaluates using only its parametric knowledge without any tool\-augmentation\. If the judge is confident, the verdict is accepted\. If uncertain, the instance is routed to Mode 2 where the judge retrieves relevant evidence from the web and re\-evaluates with this augmented context\. If the judge remains uncertain after retrieval, it abstains and flags the instance for human review\. Both thresholds are jointly calibrated on a held\-out set with known ground\-truth labels via Clopper–Pearson upper confidence bounds\([10](https://arxiv.org/html/2608.17994#bib.bib15)\)\. We prove that the finite\-sample FDR guarantee for a fixed threshold pair extends to the two\-threshold routing setting without additional distributional assumptions\. Our contributions are:

- •We formulate risk\-controlled selective evaluation for LLM\-as\-a\-judge on factual QA\.
- •We introduce a two\-mode routing mechanism that equips the judge with adaptive web retrieval to resolve uncertainty before resorting to abstention\.
- •We show that the Bernoulli structure required for the Clopper–Pearson bound extends to the two\-threshold routing setting\.
- •Experiments across open\-domain QA benchmarks and judges of varying scales demonstrate that adaptive retrieval recovers substantial coverage over single\-mode baselines while controlling the judge’s error rate\.

## 2Related Work

##### LLM\-as\-a\-Judge\.

Using LLMs to evaluate the outputs of other models has gained widespread adoption as a cost\-effective alternative to human annotation\. Strong LLMs can approximate human preferences on subjective dimensions\([19](https://arxiv.org/html/2608.17994#bib.bib10)\)\. As a result, they have been adopted for reward modeling and reinforcement learning\([28](https://arxiv.org/html/2608.17994#bib.bib23)\)\. More recently, LLM judges have been extended to objective tasks such as factual evaluation\. SAFE\([35](https://arxiv.org/html/2608.17994#bib.bib16)\)uses an LLM agent with web search to verify individual facts in long\-form responses\. Similarly, SAGE\([4](https://arxiv.org/html/2608.17994#bib.bib22)\)employs a tool\-augmented agent that iteratively retrieves and synthesizes external evidence for reference\-free QA evaluation\. These works improve judge accuracy but provide no formal guarantee on the reliability of verdicts\.

##### Uncertainty quantification for LLM evaluation\.

Existing methods generally fall into two categories\. White\-box methods estimate uncertainty from the judge’s token probabilities\. The most common choice is the predictive entropy of the output distribution\([20](https://arxiv.org/html/2608.17994#bib.bib25)\)\. These probabilities are often well calibrated with respect to correctness\([16](https://arxiv.org/html/2608.17994#bib.bib24)\)\. Black\-box methods, in contrast, estimate uncertainty by querying the judge repeatedly and measuring how consistent its verdicts are\([31](https://arxiv.org/html/2608.17994#bib.bib26);[32](https://arxiv.org/html/2608.17994#bib.bib27)\)\. Both signals correlate with judge accuracy and can flag low\-confidence verdicts\([5](https://arxiv.org/html/2608.17994#bib.bib11)\)\. However, they remain heuristic\. For instance, a threshold chosen by hand or tuned on a validation set gives no formal guarantee on the error rate among accepted verdicts\.

##### Uncertainty\-based routing and tool use\.

A parallel line of work uses uncertainty to determine when additional resources are needed\. FrugalGPT\([8](https://arxiv.org/html/2608.17994#bib.bib36)\)and AutoMix\([1](https://arxiv.org/html/2608.17994#bib.bib37)\)use confidence to route queries across models with different capabilities, trading off cost against quality\. In contrast, FLARE\([13](https://arxiv.org/html/2608.17994#bib.bib38)\)uses token\-level uncertainty to decide when to retrieve external information during generation\. More recently, SAGE\([4](https://arxiv.org/html/2608.17994#bib.bib22)\)introduced search\-augmented retrieval for LLM judges, where the judge acts as an agent that gathers targeted evidence when evaluating free\-form answers\. However, these approaches often rely on unconditional retrieval, which introduces unnecessary computation\. Furthermore, they do not provide formal guarantees on the error rate of the resulting evaluations\.

##### Risk\-controlled selective evaluation\.

Conformal prediction provides distribution\-free, finite\-sample coverage guarantees by constructing prediction sets calibrated on held\-out data\([2](https://arxiv.org/html/2608.17994#bib.bib19)\)\. Conformal risk control generalizes this beyond coverage to bound arbitrary loss functions, including the FDR, at user\-specified levels\([3](https://arxiv.org/html/2608.17994#bib.bib20)\)\. Recent work has applied these guarantees to question answering\. SConU builds prediction sets that cover admissible answers with high probability\([34](https://arxiv.org/html/2608.17994#bib.bib28)\)\. Similarly, COIN\([33](https://arxiv.org/html/2608.17994#bib.bib13)\)calibrates an uncertainty threshold that bounds the FDR among accepted answers using Clopper–Pearson upper confidence bounds\([10](https://arxiv.org/html/2608.17994#bib.bib15)\)\. Closer to our setting, Trust or Escalate\([15](https://arxiv.org/html/2608.17994#bib.bib12)\)and SCOPE\([5](https://arxiv.org/html/2608.17994#bib.bib11)\)adapt risk\-controlled selective prediction to pairwise evaluation and provide guarantee on human agreement among accepted judgments\. However, both target subjective preference tasks where the judge compares two responses rather than evaluating factual correctness \(i\.e\., objective evaluation\)\.

Our framework introduces risk\-controlled retrieval\-augmented evaluation, where retrieval is selectively activated based on calibrated uncertainty thresholds\. Unlike unconditional retrieval approaches, it provides formal FDR guarantees for the accepted verdicts\.

## 3Methodology

We propose a risk\-controlled framework for LLM\-as\-a\-judge evaluation\. The framework calibrates uncertainty thresholds on a held\-out set with human labels so that, among all instances the judge evaluates, the probability of producing an incorrect verdict is bounded by a user\-specified risk levelα\\alpha\([2](https://arxiv.org/html/2608.17994#bib.bib19);[3](https://arxiv.org/html/2608.17994#bib.bib20)\)\. When the judge is uncertain, the framework first attempts to improve the evaluation through web retrieval; if the judge remains uncertain after retrieval, it abstains and flags the instance for human review111We do not implement human review; the term indicates that these instances are excluded from automated evaluation\.\.

### 3\.1Problem Formulation

##### Task\.

LetF:𝒳→𝒴F\\\!:\\\!\\mathcal\{X\}\\\!\\to\\\!\\mathcal\{Y\}be an examinee LLM that, given a question𝐱∈𝒳\\mathbf\{x\}\\in\\mathcal\{X\}, generates a candidate answery^∈𝒴\\hat\{y\}\\in\\mathcal\{Y\}, and lety∗∈𝒴y^\{\*\}\\in\\mathcal\{Y\}be the corresponding ground\-truth answer\. A judge LLM𝒥:𝒳×𝒴→\{0,1\}\\mathcal\{J\}\\\!:\\\!\\mathcal\{X\}\\\!\\times\\\!\\mathcal\{Y\}\\\!\\to\\\!\\\{0,1\\\}evaluates whethery^\\hat\{y\}correctly answers𝐱\\mathbf\{x\}, producing a binary verdictv∈\{0,1\}v\\in\\\{0,1\\\}, wherev=1v\\\!=\\\!1means the judge deemsy^\\hat\{y\}correct\.

##### Admission function\.

The ground\-truth evaluation labele∗∈\{0,1\}e^\{\*\}\\in\\\{0,1\\\}indicates whethery^\\hat\{y\}is truly correct\. It can be obtained from human annotation, or by applying a task\-specific relevance function𝒜ref:𝒴×𝒴→\[0,1\]\\mathcal\{A\}\_\{\\mathrm\{ref\}\}\\\!:\\\!\\mathcal\{Y\}\\\!\\times\\\!\\mathcal\{Y\}\\\!\\to\\\!\[0,1\]with a thresholdλA\\lambda\_\{A\}:

e∗=\[𝒜ref\(y^,y∗\)≥λA\]\.e^\{\*\}=\\mathbf\{1\}\\\!\\left\[\\mathcal\{A\}\_\{\\mathrm\{ref\}\}\(\\hat\{y\},\\,y^\{\*\}\)\\geq\\lambda\_\{A\}\\right\]\.\(1\)A verdictvvis admissible if it matches the ground truth:

𝒜\(v,e∗\)=\[v=e∗\]\.\\mathcal\{A\}\(v,\\,e^\{\*\}\)=\\mathbf\{1\}\\\!\\left\[v=e^\{\*\}\\right\]\.\(2\)

##### Selective evaluation\.

Rather than always outputting a verdict, the judge is allowed to abstain when it is uncertain\. We define a failure indicatorW=𝟏\[v≠e∗\]W=\\mathbf\{1\}\[v\\neq e^\{\*\}\]\. Given an uncertainty scoreUU\(defined in §[3\.2](https://arxiv.org/html/2608.17994#S3.SS2)\), the judge outputs a verdict only whenU≤tU\\leq tfor some thresholdtt\. The false discovery rate \(FDR\) is:

R⁡\(t\)=𝔼⁡\[W∣U≤t\],R\(t\)=\\mathbb\{E\}\\\!\\left\[W\\mid U\\leq t\\right\],\(3\)i\.e\., the error rate among instances the judge chooses to evaluate\.

##### Goal\.

For a user\-specified risk levelα∈\(0,1\)\\alpha\\in\(0,1\)and significance levelδ\\delta\(e\.g\.,0\.050\.05\), we seek a thresholdttsuch that:

Pr⁡\(R⁡\(t\)≤α\)≥1−δ,\\Pr\\\!\\left\(R\(t\)\\leq\\alpha\\right\)\\geq 1\-\\delta,\(4\)where the probability is over the randomness of the calibration data\. With high confidence \(−δ1\\\!\-\\\!\\delta\), the error rate among the judge’s non\-abstained outputs is at mostα\\alpha\.

##### Calibration data\.

We hold outNNcalibration samples𝒟cal=\{\(𝐱i,y^i,yi∗\)\}i=1N\\mathcal\{D\}\_\{\\mathrm\{cal\}\}=\\\{\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\},y^\{\*\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}with known ground\-truth labelsei∗e\_\{i\}^\{\*\}, drawn i\.i\.d\. from the data\-generating distribution𝒟\\mathcal\{D\}\.

### 3\.2Uncertainty Quantification \(UQ\)

Accurate uncertainty estimation is critical for selective evaluation: the better the uncertainty score separates correct from incorrect verdicts, the higher the coverage \(fewer abstentions\) at a given risk level\.

### 3\.3Risk\-Bounded Selective Evaluation

We now describe how to calibrate the uncertainty threshold to satisfy the guarantee in Eq\. \([4](https://arxiv.org/html/2608.17994#S3.E4)\)\. This section presents the base case where the judge uses only its parametric knowledge \(no retrieval\); we extend to retrieval\-augmented judging in §[3\.4](https://arxiv.org/html/2608.17994#S3.SS4)\.

##### Empirical failure analysis\.

For each calibration instance, we obtain the judge’s verdictviv\_\{i\}and uncertainty scoreUiU\_\{i\}\. Given a candidate thresholdtt, we compute the number of instances the judge would evaluate \(selection size\):

m^cal\(t\)=∑i=1N\{Ui≤t\},\\hat\{m\}\_\{\\mathrm\{cal\}\}\(t\)=\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\\\!\\left\\\{U\_\{i\}\\leq t\\right\\\},\(5\)and the number of errors among them \(error count\):

w^cal\(t\)=∑i=1N\{Ui≤t\}⋅\(1−ci\),\\hat\{w\}\_\{\\mathrm\{cal\}\}\(t\)=\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\\\!\\left\\\{U\_\{i\}\\leq t\\right\\\}\\\!\\cdot\\\!\(1\-c\_\{i\}\),\(6\)whereci=𝒜⁡\(vi,ei∗\)c\_\{i\}=\\mathcal\{A\}\(v\_\{i\},e^\{\*\}\_\{i\}\)is the correctness indicator\. The empirical FDR isr^cal​\(t\)=w^cal​\(t\)/m^cal​\(t\)\\hat\{r\}\_\{\\mathrm\{cal\}\}\(t\)=\\hat\{w\}\_\{\\mathrm\{cal\}\}\(t\)/\\hat\{m\}\_\{\\mathrm\{cal\}\}\(t\)\.

##### Upper confidence bound \(UCB\)\.

The empirical FDR is an unbiased point estimate of the true failure rate, but may underestimate it on any given calibration set due to sampling variability\. To obtain a rigorous bound, we leverage the Clopper–Pearson exact method\. Since the calibration instances are i\.i\.d\. and the selection event\{Ui≤t\}\\\{U\_\{i\}\\leq t\\\}is a deterministic function of\(𝐱i,y^i\)\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\}\), the failure indicators of selected instances are i\.i\.d\. Bernoulli with parameterR⁡\(t\)R\(t\)\. We can therefore apply the one\-sided Clopper–Pearson bound:

R^1−δupper​\(t\)=BetaInv⁡\(1−δ,w^cal​\(t\)\+1,m^cal​\(t\)−w^cal​\(t\)\),\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\)=\\mathrm\{BetaInv\}\\\!\\big\(1\-\\delta;\\;\\hat\{w\}\_\{\\mathrm\{cal\}\}\(t\)\+1,\\;\\hat\{m\}\_\{\\mathrm\{cal\}\}\(t\)\-\\hat\{w\}\_\{\\mathrm\{cal\}\}\(t\)\\big\),\(7\)which satisfiesPr⁡\(R⁡\(t\)≤R^1−δupper​\(t\)\)≥1−δ\\Pr\(R\(t\)\\leq\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\)\)\\geq 1\-\\delta\(Appendix[A\.1](https://arxiv.org/html/2608.17994#A1.SS1)\)\.

##### Threshold selection\.

We select the largest threshold whose UCB stays below the risk levelα\\alpha:

t^=sup\{t:R^1−δupper​\(t′\)≤α​for all​t′≤t\}\.\\hat\{t\}=\\sup\\big\\\{t:\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t^\{\\prime\}\)\\leq\\alpha\\;\\text\{for all\}\\;t^\{\\prime\}\\leq t\\big\\\}\.\(8\)This maximizes the number of instances the judge evaluates while targeting Eq\. \([4](https://arxiv.org/html/2608.17994#S3.E4)\)\.

### 3\.4Adaptive Retrieval via Two\-Mode Routing

Inspired by LLM\-based agents that leverage external tools to overcome the base model limitations, we equip the judge with web search as a tool to bridge its knowledge gaps\. When the judge is not confident based on its parametric knowledge alone, it retrieves relevant evidence from the web and re\-evaluates with this augmented context\. This recovers coverage on exactly the instances that the single\-mode framework would abstain on, turning abstention into an opportunity for informed evaluation\.

##### Two evaluation modes\.

For each instance\(𝐱i,y^i\)\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\}\), we define:

- •Mode 1\(direct\):𝒥\\mathcal\{J\}evaluates using only its internal knowledge, producing verdictv1\(i\)v\_\{1\}^\{\(i\)\}with uncertaintyU1\(i\)U\_\{1\}^\{\(i\)\}\.
- •Mode 2\(retrieval\-augmented\): The system queries a web search engine using the question𝐱i\\mathbf\{x\}\_\{i\}, retrieving the top\-kkresults, each containing a title, text snippet, and source URL\. These are concatenated into an evidence passageℰi\\mathcal\{E\}\_\{i\}appended to the judge’s prompt\. The judge𝒥\\mathcal\{J\}re\-evaluates with this context, producingv2\(i\)v\_\{2\}^\{\(i\)\}with uncertaintyU2\(i\)U\_\{2\}^\{\(i\)\}\.

We denote the correctness of each mode asck\(i\)=𝒜⁡\(vk\(i\),ei∗\)c\_\{k\}^\{\(i\)\}=\\mathcal\{A\}\(v\_\{k\}^\{\(i\)\},e^\{\*\}\_\{i\}\)fork∈\{1,2\}k\\in\\\{1,2\\\}\.

##### Routing logic\.

At test time, given calibrated thresholds\(t^1,t^2\)\(\\hat\{t\}\_\{1\},\\hat\{t\}\_\{2\}\):

output=\{v1,if​U1≤t^1,v2,if​U1\>t^1​and​U2≤t^2,abstain,otherwise\.\\text\{output\}=\\begin\{cases\}v\_\{1\},&\\text\{if \}U\_\{1\}\\leq\\hat\{t\}\_\{1\},\\\\ v\_\{2\},&\\text\{if \}U\_\{1\}\>\\hat\{t\}\_\{1\}\\;\\text\{and\}\\;U\_\{2\}\\leq\\hat\{t\}\_\{2\},\\\\ \\textsc\{abstain\},&\\text\{otherwise\.\}\\end\{cases\}\(9\)Retrieval is invoked only when Mode 1 is uncertain, so confident instances incur zero retrieval cost\.

##### Joint threshold calibration\.

During calibration, we pre\-compute both modes for allNNinstances in𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}\. For a candidate pair\(t1,t2\)\(t\_\{1\},t\_\{2\}\), the selection size counts instances accepted via either mode:

m^cal\(t1,t2\)=∑i=1N\[\{U1\(i\)≤t1\}\+\{U1\(i\)\>t1andU2\(i\)≤t2\}\]\.\\hat\{m\}\_\{\\mathrm\{cal\}\}\(t\_\{1\},t\_\{2\}\)=\\sum\_\{i=1\}^\{N\}\\Big\[\\mathbf\{1\}\\\!\\left\\\{U\_\{1\}^\{\(i\)\}\\leq t\_\{1\}\\right\\\}\+\\mathbf\{1\}\\\!\\big\\\{U\_\{1\}^\{\(i\)\}\>t\_\{1\}\\;\\text\{and\}\\;U\_\{2\}^\{\(i\)\}\\leq t\_\{2\}\\big\\\}\\Big\]\.\(10\)The error count among selected instances is:

w^cal\(t1,t2\)=∑i=1N\[\{U1\(i\)≤t1\}\(1−c1\(i\)\)\+\{U1\(i\)\>t1∧U2\(i\)≤t2\}\(1−c2\(i\)\)\]\.\\hat\{w\}\_\{\\mathrm\{cal\}\}\(t\_\{1\},t\_\{2\}\)=\\sum\_\{i=1\}^\{N\}\\Big\[\\mathbf\{1\}\\\!\\\{U\_\{1\}^\{\(i\)\}\\leq t\_\{1\}\\\}\(1\-c\_\{1\}^\{\(i\)\}\)\+\\mathbf\{1\}\\\!\\\{U\_\{1\}^\{\(i\)\}\>t\_\{1\}\\wedge U\_\{2\}^\{\(i\)\}\\leq t\_\{2\}\\\}\(1\-c\_\{2\}^\{\(i\)\}\)\\Big\]\.\(11\)The empirical FDR and UCB follow as before:

r^cal​\(t1,t2\)\\displaystyle\\hat\{r\}\_\{\\mathrm\{cal\}\}\(t\_\{1\},t\_\{2\}\)=w^cal​\(t1,t2\)m^cal​\(t1,t2\),\\displaystyle=\\frac\{\\hat\{w\}\_\{\\mathrm\{cal\}\}\(t\_\{1\},t\_\{2\}\)\}\{\\hat\{m\}\_\{\\mathrm\{cal\}\}\(t\_\{1\},t\_\{2\}\)\},\(12\)R^1−δupper​\(t1,t2\)\\displaystyle\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\_\{1\},t\_\{2\}\)=BetaInv⁡\(1−δ,w^cal\+1,m^cal−w^cal\)\.\\displaystyle=\\mathrm\{BetaInv\}\\\!\\big\(1\-\\delta;\\;\\hat\{w\}\_\{\\mathrm\{cal\}\}\+1,\\;\\hat\{m\}\_\{\\mathrm\{cal\}\}\-\\hat\{w\}\_\{\\mathrm\{cal\}\}\\big\)\.\(13\)We select the threshold pair that maximizes coverage subject to the risk constraint:

\(t^1,t^2\)=arg​maxt1,t2⁡m^cal​\(t1,t2\)s\.t\.R^1−δupper​\(t1,t2\)≤α\.\(\\hat\{t\}\_\{1\},\\hat\{t\}\_\{2\}\)=\\argmax\_\{t\_\{1\},\\,t\_\{2\}\}\\hat\{m\}\_\{\\mathrm\{cal\}\}\(t\_\{1\},t\_\{2\}\)\\quad\\text\{s\.t\.\}\\quad\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\_\{1\},t\_\{2\}\)\\leq\\alpha\.\(14\)In practice, this is solved by search over the sorted unique uncertainty values on the calibration set\. For each candidatet1t\_\{1\}, cumulative sums enable evaluating allt2t\_\{2\}candidates inO⁡\(\|𝒯2\|\)O\(\|\\mathcal\{T\}\_\{2\}\|\)time, yieldingO⁡\(\|𝒯1\|⋅\|𝒯2\|\)O\(\|\\mathcal\{T\}\_\{1\}\|\\cdot\|\\mathcal\{T\}\_\{2\}\|\)total complexity \(Algorithm[1](https://arxiv.org/html/2608.17994#alg1)\)\. Because this search evaluates many threshold pairs, we additionally report a conservative variant that corrects for it via a Bonferroni adjustment \(δ′=δ/\(\|𝒯1\|​\|𝒯2\|\)\\delta^\{\\prime\}=\\delta/\(\|\\mathcal\{T\}\_\{1\}\|\\,\|\\mathcal\{T\}\_\{2\}\|\)\) \(see Appendix[D\.4](https://arxiv.org/html/2608.17994#A4.SS4)\)\.

### 3\.5Theoretical Guarantee

The central theoretical question is whether the single\-mode Clopper–Pearson guarantee \(Eq\.[4](https://arxiv.org/html/2608.17994#S3.E4)\) extends to the two\-mode routing setting\. It does, because the routing decision for each instance is entirely determined by its uncertainty scores and the fixed thresholds, introducing no additional randomness\.

###### Lemma 1\(Bernoulli structure under routing\)\.

Let\(𝐱i,y^i,yi∗\)\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\},y^\{\*\}\_\{i\}\)be i\.i\.d\. samples from𝒟\\mathcal\{D\}, and let\(t1,t2\)\(t\_\{1\},t\_\{2\}\)be fixed thresholds\. Under the routing policy \(Eq\.[9](https://arxiv.org/html/2608.17994#S3.E9)\), the failure indicators\{Wi\}\\\{W\_\{i\}\\\}over the selected subset are i\.i\.d\.Bernoulli⁡\(R⁡\(t1,t2\)\)\\mathrm\{Bernoulli\}\(R\(t\_\{1\},t\_\{2\}\)\)\.

The proof \(Appendix[A\.2](https://arxiv.org/html/2608.17994#A1.SS2)\) shows that the selection indicator and the correctness of selected instances are both deterministic functions of\(𝐱i,y^i,yi∗\)\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\},y^\{\*\}\_\{i\}\)\. Since the original data is i\.i\.d\., conditioning on selection preserves independence and identical distribution\. Combined with the Clopper–Pearson bound, this yields a guarantee for any fixed threshold pair:

Pr⁡\(R⁡\(t1,t2\)≤R^1−δupper​\(t1,t2\)\)≥1−δ\.\\boxed\{\\;\\Pr\\\!\\left\(R\(t\_\{1\},t\_\{2\}\)\\leq\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\_\{1\},t\_\{2\}\)\\right\)\\geq 1\-\\delta\.\\;\}\(15\)In particular, any fixed pair whose bound satisfiesR^1−δupper​\(t1,t2\)≤α\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\_\{1\},t\_\{2\}\)\\leq\\alphacontrols the error rate atα\\alphawith confidence1−δ1\-\\delta\.

Algorithm 1Calibration \(offline\)1:Calibration set

𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}, judge

𝒥\\mathcal\{J\}
2:Risk level

α\\alpha, significance level

δ\\delta
3:Calibrated thresholds

t^1\\hat\{t\}\_\{1\},

t^2\\hat\{t\}\_\{2\}
4:

5:foreach

\(𝐱i,y^i,yi∗\)∈𝒟cal\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\},y\_\{i\}^\{\*\}\)\\in\\mathcal\{D\}\_\{\\mathrm\{cal\}\}do

6:

v1\(i\),U1\(i\)←𝒥⁡\(𝐱i,y^i\)v\_\{1\}^\{\(i\)\},U\_\{1\}^\{\(i\)\}\\leftarrow\\mathcal\{J\}\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\}\)
7:

ℰi←Retrieve​\(𝐱i\)\\mathcal\{E\}\_\{i\}\\leftarrow\\textsc\{Retrieve\}\(\\mathbf\{x\}\_\{i\}\)
8:

v2\(i\),U2\(i\)←𝒥⁡\(𝐱i,y^i,ℰi\)v\_\{2\}^\{\(i\)\},U\_\{2\}^\{\(i\)\}\\leftarrow\\mathcal\{J\}\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\},\\mathcal\{E\}\_\{i\}\)
9:

c1\(i\)←𝒜⁡\(v1\(i\),ei∗\)c\_\{1\}^\{\(i\)\}\\\!\\leftarrow\\\!\\mathcal\{A\}\(v\_\{1\}^\{\(i\)\},e\_\{i\}^\{\*\}\);

c2\(i\)←𝒜⁡\(v2\(i\),ei∗\)c\_\{2\}^\{\(i\)\}\\\!\\leftarrow\\\!\\mathcal\{A\}\(v\_\{2\}^\{\(i\)\},e\_\{i\}^\{\*\}\)
10:endfor

11:

12:

𝒯1←\\mathcal\{T\}\_\{1\}\\leftarrowsorted unique

\{U1\(i\)\}i=1N\\\{U\_\{1\}^\{\(i\)\}\\\}\_\{i=1\}^\{N\}
13:

𝒯2←\\mathcal\{T\}\_\{2\}\\leftarrowsorted unique

\{U2\(i\)\}i=1N\\\{U\_\{2\}^\{\(i\)\}\\\}\_\{i=1\}^\{N\}
14:

t^1,t^2←null\\hat\{t\}\_\{1\},\\hat\{t\}\_\{2\}\\leftarrow\\textsc\{null\};

m∗←0m^\{\*\}\\leftarrow 0
15:for

t1t\_\{1\}in

𝒯1\\mathcal\{T\}\_\{1\}do

16:for

t2t\_\{2\}in

𝒯2\\mathcal\{T\}\_\{2\}do

17:Compute

m^cal,w^cal\\hat\{m\}\_\{\\mathrm\{cal\}\},\\,\\hat\{w\}\_\{\\mathrm\{cal\}\}
18:

R\+←BetaInv⁡\(−δ,w^cal\+1,m^cal−w^cal\)R^\{\+\}\\\!\\leftarrow\\\!\\mathrm\{BetaInv\}\(1\\\!\-\\\!\\delta;\\,\\hat\{w\}\_\{\\mathrm\{cal\}\}\\\!\+\\\!1,\\,\\hat\{m\}\_\{\\mathrm\{cal\}\}\\\!\-\\\!\\hat\{w\}\_\{\\mathrm\{cal\}\}\)
19:if

R\+≤αR^\{\+\}\\leq\\alphaand

m^cal\>m∗\\hat\{m\}\_\{\\mathrm\{cal\}\}\>m^\{\*\}then

20:

t^1←t1\\hat\{t\}\_\{1\}\\\!\\leftarrow\\\!t\_\{1\};

t^2←t2\\hat\{t\}\_\{2\}\\\!\\leftarrow\\\!t\_\{2\};

m∗←m^calm^\{\*\}\\\!\\leftarrow\\\!\\hat\{m\}\_\{\\mathrm\{cal\}\}
21:endif

22:endfor

23:endfor

24:return

t^1\\hat\{t\}\_\{1\},

t^2\\hat\{t\}\_\{2\}

Cost analysis\.Letppbe the fraction of instances accepted by Mode 1\. Test\-time retrieval is invoked for only\(−p\)\(1\\\!\-\\\!p\)of instances\. Atα=0\.20\\alpha\\\!=\\\!0\.20with Qwen3\-14B,p≈0\.45p\\\!\\approx\\\!0\.45on TriviaQA, so nearly half of evaluations incur zero retrieval cost\. On harder benchmarksppdecreases, but the framework still avoids retrieval for the most confident instances\.

Algorithm 2Test\-time evaluation \(online\)1:Instance

\(𝐱,y^\)\(\\mathbf\{x\},\\hat\{y\}\), judge

𝒥\\mathcal\{J\}
2:Thresholds

t^1\\hat\{t\}\_\{1\},

t^2\\hat\{t\}\_\{2\}
3:Verdict

v∈\{0,1\}v\\in\\\{0,1\\\}orabstain

4:

5:

v1,U1←𝒥⁡\(𝐱,y^\)v\_\{1\},U\_\{1\}\\leftarrow\\mathcal\{J\}\(\\mathbf\{x\},\\hat\{y\}\)
6:if

U1≤t^1U\_\{1\}\\leq\\hat\{t\}\_\{1\}then

7:return

v1v\_\{1\}⊳\\trianglerightConfident

8:endif

9:

10:

ℰ←Retrieve​\(𝐱\)\\mathcal\{E\}\\leftarrow\\textsc\{Retrieve\}\(\\mathbf\{x\}\)
11:

v2,U2←𝒥⁡\(𝐱,y^,ℰ\)v\_\{2\},U\_\{2\}\\leftarrow\\mathcal\{J\}\(\\mathbf\{x\},\\hat\{y\},\\mathcal\{E\}\)
12:if

U2≤t^2U\_\{2\}\\leq\\hat\{t\}\_\{2\}then

13:return

v2v\_\{2\}⊳\\trianglerightRetrieval helped

14:endif

15:

16:returnabstain⊳\\trianglerightHuman review

## 4Experiments

##### Datasets and models\.

We evaluate on four open\-domain QA benchmarks: TriviaQA\([14](https://arxiv.org/html/2608.17994#bib.bib31)\), Natural Questions \(NQ\)\([18](https://arxiv.org/html/2608.17994#bib.bib30)\), HotpotQA\([37](https://arxiv.org/html/2608.17994#bib.bib29)\), and PopQA\([21](https://arxiv.org/html/2608.17994#bib.bib32)\)\. From each benchmark’s standard held\-out split, we sample 2,000 instances using seed 42\.

We use two candidate models: Qwen3\-8B\([25](https://arxiv.org/html/2608.17994#bib.bib33)\)and LLaMA\-3\.1\-70B\([23](https://arxiv.org/html/2608.17994#bib.bib34)\)\. For judges, we employ four models: Qwen3\-4B, Qwen3\-8B, and Qwen3\-14B\([25](https://arxiv.org/html/2608.17994#bib.bib33)\)and LLaMA\-3\.1\-8B\-Instruct\([23](https://arxiv.org/html/2608.17994#bib.bib34)\)\. Each judge outputs a binary verdict \(True/False\) and a brief explanation\.

##### Admission functions\.

To evaluate whether the judge’s verdict is correct, we first determine whether the candidate’s answer is actually right by comparing it against the reference answer\. This ground\-truth labele∗e^\{\*\}is computed using three admission functions: \(1\) Exact Match \(EM\), \(2\) Token F1 \(threshold≥0\.5\\geq 0\.5\), and \(3\) LLM Evaluator\([7](https://arxiv.org/html/2608.17994#bib.bib6);[6](https://arxiv.org/html/2608.17994#bib.bib7)\), which uses Qwen2\.5\-7B\-Instruct\([24](https://arxiv.org/html/2608.17994#bib.bib35)\)as a reference\-based evaluator to judge semantic equivalence\.

##### Uncertainty quantification\.

We use predictive entropy \(PE\)\([20](https://arxiv.org/html/2608.17994#bib.bib25)\)as the uncertainty measure\. PE is computed from the judge’s token\-level log\-probabilities at the verdict position via a single forward pass\([16](https://arxiv.org/html/2608.17994#bib.bib24)\)\. When the judge’s hidden states are available, PE can be replaced by supervised scores without changing the guarantee\. We report a probe and a LoRA\+Prompt variant in Appendix[D\.2](https://arxiv.org/html/2608.17994#A4.SS2)\.

##### Retrieval\.

For Mode 2, we query a web search engine using the original question \(i\.e\., the question provided to the candidate\) and retrieve the top\-kkresults, withk=3k=3as the default setting\. The retrieved results are concatenated into a single passage and appended to the judge prompt as additional context\. We further study the effect of retrieval depth by varyingk∈\{3,6,9\}k\\in\\\{3,6,9\\\}on one candidate–judge pair \(LLaMA\-3\.1\-70B evaluated by Qwen3\-8B\)\.

##### Calibration protocol\.

We run 100 random 50/50 calibration/test splits using the Clopper–Pearson exact binomial bound\. We setδ=0\.05\\delta\\\!=\\\!0\.05and evaluateα∈\{0\.05,0\.10,0\.15,0\.20,0\.25\}\\alpha\\in\\\{0\.05,0\.10,0\.15,0\.20,0\.25\\\}\. Appendix[D\.1](https://arxiv.org/html/2608.17994#A4.SS1)reports sensitivity to bothδ\\deltaand the calibration\-set size\.

##### Evaluation metrics\.

We report: \(1\) FDR, the fraction of accepted instances with incorrect verdicts \(wtest/mtestw\_\{\\text\{test\}\}/m\_\{\\text\{test\}\}\); \(2\) Coverage, the fraction of instances accepted rather than abstained on \(mtest/Ntestm\_\{\\text\{test\}\}/N\_\{\\text\{test\}\}\); and \(3\) Routing distribution which is the split across Mode 1, Mode 2, and abstention\.

## 5Results

Figure 1:Test\-time FDR \(mean±\\pmstd over 100 splits\) across all 32 configurations\. Rows correspond to the two candidate models; columns to the four datasets\. Each line represents a different judge \(see legend\)\. The dashed diagonal marks FDR=α=\\alpha\.We evaluate whether the framework delivers on its two central promises: \(1\) the FDR guarantee holds empirically across judges, candidates, and datasets, and \(2\) adaptive retrieval recovers coverage that would otherwise be lost to abstention\. We use predictive entropy and LLM evaluator for the main results, and additionally analyze the sensitivity of our framework to the choice of admission function and retrieval depth \(§[D\.1](https://arxiv.org/html/2608.17994#A4.SS1)\)\.

### 5\.1FDR control holds across all configurations

The most basic requirement of our framework is that the empirical FDR remains at or below the targetα\\alpha\. Figure[1](https://arxiv.org/html/2608.17994#S5.F1)tests this across all candidate–judge–dataset configurations\. In every case, the FDR stays at or belowα\\alpha, satisfying the finite\-sample guarantee \(Eq\.[15](https://arxiv.org/html/2608.17994#S3.E15)\)\.

Three observations are noteworthy\. First, all four judges provide valid FDR control regardless of their parameter count, showing that the guarantee does not require a strong judge\. Second, FDR tracks close toα\\alphawith narrow confidence bands \(±\\pm0\.01–0\.02\), which indicates stable calibration across 100 random splits\. Third, the gap between FDR andα\\alphais smallest on TriviaQA and PopQA, which are relatively easy tasks where judges are most accurate\. This demonstrates that the framework is tightest when the judge has sufficient knowledge to evaluate\.

### 5\.2Coverage scales with risk tolerance and judge capability

Table[1](https://arxiv.org/html/2608.17994#S5.T1)reports coverage across five risk levels\. Coverage scales smoothly withα\\alphaand is governed by two factors: dataset difficulty and judge capability\. On easy benchmarks \(TriviaQA, PopQA\), even the smallest judge reaches near\-full coverage byα=0\.25\\alpha\\\!=\\\!0\.25, while on knowledge\-intensive tasks \(NQ\-Open, HotpotQA\) only the larger judges \(Qwen\-14B, Llama\-8B\) surpass 80% atα=0\.20\\alpha\\\!=\\\!0\.20for the Qwen\-8B candidate — Qwen\-4B remains below 15% on these same benchmarks\. Evaluating the stronger Llama\-70B candidate is consistently harder: coverage drops across all judges compared to Qwen\-8B, with the best judge reaching only 53% on HotpotQA atα=0\.20\\alpha\\\!=\\\!0\.20\. This gap reflects both the answer diversity of the 70B model and the increased knowledge demands it places on the judge\.

Table 1:Coverage of the joint two\-mode framework \(two\-threshold routing with retrieval\) atα∈\{\.05,\.10,\.15,\.20,\.25\}\\alpha\\in\\\{\.05,\.10,\.15,\.20,\.25\\\}\(mean over 100 splits\) across all datasets\.
### 5\.3Adaptive retrieval recovers substantial coverage

Figure[2](https://arxiv.org/html/2608.17994#S5.F2)breaks down accepted instances by routing mode\. It is depicted that retrieval is the dominant acceptance mechanism\. Across various configurations, Mode 2 accounts for the majority of non\-abstained verdicts\. This does not necessarily mean the judge lacks the knowledge to evaluate correctly, but rather that its uncertainty signal in Mode 1 is not confident enough to satisfy the calibrated threshold\. Crucially, these are instances where a single\-mode framework would abstain entirely\. By grounding the judge in retrieved evidence, Mode 2 reduces uncertainty below the second threshold and converts these abstentions into accepted verdicts\. This trade\-off also depends on the underlying judge model\. Models with better\-calibrated uncertainty estimates allow a larger fraction of instances to be resolved by Mode 1 directly\. For instance, atα=0\.20\\alpha\\\!=\\\!0\.20on TriviaQA, Qwen3\-14B resolves 45% of evaluations parametrically versus 9% for Qwen3\-4B\. On harder tasks like HotpotQA and NQ\-Open, all judges rely predominantly on retrieved evidence\.

Although retrieval substantially expands coverage, it does not always improve the judge’s confidence\. For 14–29% of instances, retrieval increases rather than decreases uncertainty \(Appendix[E\.1](https://arxiv.org/html/2608.17994#A5.SS1)\)\. The second calibrated threshold identifies these cases and routes them to abstention\. Consequently, only retrieval\-based verdicts that satisfy the desired risk level are accepted\.

Figure 2:Routing distribution across configurations\. Within each group of four bars, judges are ordered left to right: Qwen3\-4B, Qwen3\-8B, Qwen3\-14B, Llama\-3\.1\-8B\.
### 5\.4Single\-mode vs\. joint two\-mode routing

Table[2](https://arxiv.org/html/2608.17994#S5.T2)isolates the contribution of adaptive retrieval by comparing Mode 1 Only against the full joint framework\. On TriviaQA, Mode 1 already achieves high coverage for strong judges \(87% for Qwen3\-14B\), so retrieval provides a modest but consistent gain\. The effect is more pronounced: on NQ\-Open, coverage for Qwen3\-8B increases from 7% to 82%; on HotpotQA, coverage for Llama\-8B increases from 40% to 85%\. Similarly, the weakest judge \(Qwen3\-4B\) benefits substantially on PopQA, with coverage increasing from 4% to 100%\.

Table 2:Coverage atα=0\.20\\alpha\\\!=\\\!0\.20: Mode 1 Only vs\. Joint \(two\-threshold routing with retrieval\)\.
### 5\.5Robustness to retrieval non\-stationarity

Web search results change over time, which §[3\.5](https://arxiv.org/html/2608.17994#S3.SS5)identified as the main practical threat to the determinism the guarantee assumes for Mode 2\. We measure the effect directly on the main configuration \(Qwen3\-8B candidate, Qwen3\-8B judge, LLM evaluator,k=3k\\\!=\\\!3\)\. We re\-query the same 2,000 questions on TriviaQA and HotpotQA roughly three months after the original snapshot and recompute the Mode 2 verdicts on the fresh evidence\. The retrieved pages turn over substantially\. The mean per\-item Jaccard overlap between the original \(τ0\\tau\_\{0\}\) and re\-queried \(τ1\\tau\_\{1\}\) ranked top\-33URL lists is0\.300\.30for TriviaQA and0\.240\.24for HotpotQA\. When ignoring the retrieval rank and considering only URL membership, the corresponding position\-blind top\-33overlaps are0\.470\.47and0\.400\.40, respectively\.

To evaluate whether the calibrated thresholds remain valid after retrieval drift, we calibrate\(t^1,t^2\)\(\\hat\{t\}\_\{1\},\\hat\{t\}\_\{2\}\)on the original data and apply them to test instances evaluated with the updated Mode 2 verdicts \(while keeping Mode 1 unchanged\)\. We repeat this evaluation over 100 random 50/50 splits\. Table[3](https://arxiv.org/html/2608.17994#S5.T3)summarizes the results\. Across all settings, the empirical FDR remains below the target levelα\\alpha\. Similarly, coverage remains largely stable after redeployment\. This indicates that drift changes which pages are returned more than the overall quality of the evidence\. Therefore, the calibrated thresholds remain effective after redeployment\.

Table 3:FDR and coverage under retrieval drift from the original \(τ0\\tau\_\{0\}\) to the re\-queried snapshot \(τ1\\tau\_\{1\},∼\\sim3 months later\)\.Δ=τ1−τ0\\Delta=\\tau\_\{1\}\-\\tau\_\{0\}\.

## 6Conclusion

We introduced a risk\-controlled selective evaluation framework for LLM\-as\-a\-judge with finite\-sample guarantees on the reliability of accepted verdicts\. The framework uses a calibrated two\-mode policy that routes uncertain instances to a second evaluation mode and achieves higher coverage than single\-mode abstention\. More importantly, the Clopper–Pearson FDR guarantee remains valid under the joint two\-threshold policy without requiring additional distributional assumptions\. Across all tested configurations, the observed FDR remains at or below the specified target\. Future work will consider graded and subjective evaluation settings, as well as black\-box judges without access to logits\.

## Acknowledgments

We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada \(NSERC\), the Canada Foundation for Innovation \(CFI\), and Research Nova Scotia\. Advanced computing resources are provided by ACENET, the regional partner in Atlantic Canada, and the Digital Research Alliance of Canada\.

## References

- Aggarwalet al\.\(2024\)P\. Aggarwal, A\. Madaan, A\. Anand, S\. P\. Potharaju, S\. Mishra, P\. Zhou, A\. Gupta, D\. Rajagopal, K\. Kappaganthu, Y\. Yang, S\. Upadhyay, M\. Faruqui, and M\. MausamAutoMix: automatically mixing language models\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 131000–131034\.External Links:[Document](https://dx.doi.org/10.52202/079017-4164),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/ecda225cb187b40ea8edc1f46b03ffda-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px3.p1.1)\.
- Angelopoulos and Bates \(2023\)A\. N\. Angelopoulos and S\. BatesA gentle introduction to conformal prediction and distribution\-free uncertainty quantification\.Foundations and Trends® in Machine Learning16\(4\),pp\. 494–591\.Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2608.17994#S3.p1.1)\.
- Angelopouloset al\.\(2024\)A\. N\. Angelopoulos, S\. Bates, A\. Fisch, L\. Lei, and T\. SchusterConformal risk control\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=33XGfHLtZg)Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2608.17994#S3.p1.1)\.
- Badshahet al\.\(2026a\)S\. Badshah, A\. Emami, and H\. SajjadSAGE: a Search\-AuGmented Evaluation of large language models on free\-form QA\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 1466–1491\.External Links:[Link](https://aclanthology.org/2026.acl-long.66/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.66),ISBN 979\-8\-89176\-390\-6Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p4.1),[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px3.p1.1)\.
- Badshahet al\.\(2026b\)S\. Badshah, A\. Emami, and H\. SajjadSCOPE: Selective Conformal Optimized Pairwise LLM Judging\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306\.Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p4.1),[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px4.p1.1)\.
- Badshahet al\.\(2025\)S\. Badshah, M\. Moustafa, and H\. SajjadCLEV: LLM\-based evaluation through lightweight efficient voting for free\-form question\-answering\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,K\. Inui, S\. Sakti, H\. Wang, D\. F\. Wong, P\. Bhattacharyya, B\. Banerjee, A\. Ekbal, T\. Chakraborty, and D\. P\. Singh \(Eds\.\),Mumbai, India,pp\. 1513–1531\.External Links:[Link](https://aclanthology.org/2025.findings-ijcnlp.93/),ISBN 979\-8\-89176\-303\-6Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p1.1),[§4](https://arxiv.org/html/2608.17994#S4.SS0.SSS0.Px2.p1.1)\.
- Badshah and Sajjad \(2025\)S\. Badshah and H\. SajjadReference\-guided verdict: LLMs\-as\-judges in automatic evaluation of free\-form QA\.InProceedings of the 9th Widening NLP Workshop,C\. Zhang, E\. Allaway, H\. Shen, L\. Miculicich, Y\. Li, M\. M’hamdi, P\. Limkonchotiwat, R\. H\. Bai, S\. T\.y\.s\.s\., S\. S\. Han, S\. Thapa, and W\. B\. Rim \(Eds\.\),Suzhou, China,pp\. 251–267\.External Links:[Link](https://aclanthology.org/2025.winlp-main.37/),[Document](https://dx.doi.org/10.18653/v1/2025.winlp-main.37),ISBN 979\-8\-89176\-351\-7Cited by:[§B\.4](https://arxiv.org/html/2608.17994#A2.SS4.p1.1),[§1](https://arxiv.org/html/2608.17994#S1.p2.1),[§4](https://arxiv.org/html/2608.17994#S4.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2023\)L\. Chen, M\. Zaharia, and J\. ZouFrugalGPT: how to use large language models while reducing cost and improving performance\.External Links:2305\.05176,[Link](https://arxiv.org/abs/2305.05176)Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px3.p1.1)\.
- Chiang and Lee \(2023\)C\. Chiang and H\. LeeCan large language models be an alternative to human evaluations?\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 15607–15631\.External Links:[Link](https://aclanthology.org/2023.acl-long.870/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.870)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p2.1)\.
- Clopper and Pearson \(1934\)C\. J\. Clopper and E\. S\. PearsonThe use of confidence or fiducial limits illustrated in the case of the binomial\.Biometrika26\(4\),pp\. 404–413\.External Links:ISSN 00063444, 14643510,[Link](http://www.jstor.org/stable/2331986)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p5.1),[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px4.p1.1)\.
- Guet al\.\(2024\)J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu,et al\.A survey on llm\-as\-a\-judge\.The Innovation\.Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p1.1)\.
- Huanget al\.\(2025\)L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin, and T\. LiuA survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Trans\. Inf\. Syst\.43\(2\)\.External Links:ISSN 1046\-8188,[Link](https://doi.org/10.1145/3703155),[Document](https://dx.doi.org/10.1145/3703155)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p3.1)\.
- Jianget al\.\(2023\)Z\. Jiang, F\. Xu, L\. Gao, Z\. Sun, Q\. Liu, J\. Dwivedi\-Yu, Y\. Yang, J\. Callan, and G\. NeubigActive retrieval augmented generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 7969–7992\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.495/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.495)Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px3.p1.1)\.
- Joshiet al\.\(2017\)M\. Joshi, E\. Choi, D\. S\. Weld, and L\. ZettlemoyerTriviaQA: a large scale distantly supervised challenge dataset for reading comprehension\.External Links:1705\.03551,[Link](https://arxiv.org/abs/1705.03551)Cited by:[§4](https://arxiv.org/html/2608.17994#S4.SS0.SSS0.Px1.p1.1)\.
- Junget al\.\(2025\)J\. Jung, F\. Brahman, and Y\. ChoiTrust or escalate: LLM judges with provable guarantees for human agreement\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=UHPnqSTBPO)Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px4.p1.1)\.
- Kadavathet al\.\(2022\)S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson, S\. Johnston, S\. El\-Showk, A\. Jones, N\. Elhage, T\. Hume, A\. Chen, Y\. Bai, S\. Bowman, S\. Fort, D\. Ganguli, D\. Hernandez, J\. Jacobson, J\. Kernion, S\. Kravec, L\. Lovitt, K\. Ndousse, C\. Olsson, S\. Ringer, D\. Amodei, T\. Brown, J\. Clark, N\. Joseph, B\. Mann, S\. McCandlish, C\. Olah, and J\. KaplanLanguage models \(mostly\) know what they know\.External Links:2207\.05221,[Link](https://arxiv.org/abs/2207.05221)Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2608.17994#S4.SS0.SSS0.Px3.p1.1)\.
- Kapooret al\.\(2024\)S\. Kapoor, N\. Gruver, M\. Roberts, K\. Collins, A\. Pal, U\. Bhatt, A\. Weller, S\. Dooley, M\. Goldblum, and A\. WilsonLarge language models must be taught to know what they don’t know\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 85932–85972\.External Links:[Document](https://dx.doi.org/10.52202/079017-2729),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/9c20f16b05f5e5e70fa07e2a4364b80e-Paper-Conference.pdf)Cited by:[§D\.2](https://arxiv.org/html/2608.17994#A4.SS2.p1.1)\.
- Kwiatkowskiet al\.\(2019\)T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. PetrovNatural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 452–466\.External Links:[Link](https://aclanthology.org/Q19-1026),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276)Cited by:[§4](https://arxiv.org/html/2608.17994#S4.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-eval: NLG evaluation using gpt\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 2511–2522\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.153/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p1.1),[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px1.p1.1)\.
- Malinin and Gales \(2021\)A\. Malinin and M\. GalesUncertainty estimation in autoregressive structured prediction\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=jN5y-zb5Q7m)Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2608.17994#S4.SS0.SSS0.Px3.p1.1)\.
- Mallenet al\.\(2022\)A\. Mallen, A\. Asai, V\. Zhong, R\. Das, H\. Hajishirzi, and D\. KhashabiWhen not to trust language models: investigating effectiveness and limitations of parametric and non\-parametric memories\.arXiv preprint\.Cited by:[§4](https://arxiv.org/html/2608.17994#S4.SS0.SSS0.Px1.p1.1)\.
- Mañaset al\.\(2024\)O\. Mañas, B\. Krojer, and A\. AgrawalImproving automatic vqa evaluation using large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 4171–4179\.External Links:[Link](https://arxiv.org/pdf/2310.02567v2)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p2.1)\.
- Meta AI \(2024\)Meta AIThe llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4](https://arxiv.org/html/2608.17994#S4.SS0.SSS0.Px1.p2.1)\.
- Qwen Team \(2025a\)Qwen TeamQwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4](https://arxiv.org/html/2608.17994#S4.SS0.SSS0.Px2.p1.1)\.
- Qwen Team \(2025b\)Qwen TeamQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4](https://arxiv.org/html/2608.17994#S4.SS0.SSS0.Px1.p2.1)\.
- Schroeder and Wood\-Doughty \(2025\)K\. Schroeder and Z\. Wood\-DoughtyCan you trust llm judgments? reliability of llm\-as\-a\-judge\.External Links:2412\.12509,[Link](https://arxiv.org/abs/2412.12509)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p2.1)\.
- Shorinwaet al\.\(2025\)O\. Shorinwa, Z\. Mei, J\. Lidard, A\. Z\. Ren, and A\. MajumdarA survey on uncertainty quantification of large language models: taxonomy, open research challenges, and future directions\.ACM Comput\. Surv\.58\(3\)\.External Links:ISSN 0360\-0300,[Link](https://doi.org/10.1145/3744238),[Document](https://dx.doi.org/10.1145/3744238)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p4.1)\.
- Sonet al\.\(2024\)G\. Son, H\. Ko, H\. Lee, Y\. Kim, and S\. HongLLM\-as\-a\-judge and reward model: what they can and cannot do\.External Links:2409\.11239,[Link](https://arxiv.org/abs/2409.11239)Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px1.p1.1)\.
- Tanet al\.\(2025\)S\. Tan, S\. Zhuang, K\. Montgomery, W\. Y\. Tang, A\. Cuadron, C\. Wang, R\. A\. Popa, and I\. StoicaJudgeBench: a benchmark for evaluating llm\-based judges\.External Links:2410\.12784,[Link](https://arxiv.org/abs/2410.12784)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p1.1)\.
- Wanget al\.\(2026\)Q\. Wang, Y\. Fan, and X\. WangSAFER: risk\-constrained sample\-then\-filter in large language models\.InInternational Conference on Learning Representations,C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(Eds\.\),Vol\.2026,pp\. 53723–53747\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/582e9771ac8527cb6390e5e9444a0fee-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p4.1)\.
- Wanget al\.\(2023\)X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. ZhouSelf\-consistency improves chain of thought reasoning in language models\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/forum?id=1PL1NIMMrw)Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024\)Z\. Wang, J\. Duan, L\. Cheng, Y\. Zhang, Q\. Wang, X\. Shi, K\. Xu, H\. T\. Shen, and X\. ZhuConU: conformal uncertainty in large language models with correctness coverage guarantees\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 6886–6898\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.404/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.404)Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025a\)Z\. Wang, J\. Duan, Q\. Wang, X\. Zhu, T\. Chen, X\. Shi, and K\. XuCOIN: uncertainty\-guarding selective question answering for foundation models with provable risk guarantees\.External Links:2506\.20178,[Link](https://arxiv.org/abs/2506.20178)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p4.1),[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px4.p1.1)\.
- Wanget al\.\(2025b\)Z\. Wang, Q\. Wang, Y\. Zhang, T\. Chen, X\. Zhu, X\. Shi, and K\. XuSConU: selective conformal uncertainty in large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 19052–19075\.External Links:[Link](https://aclanthology.org/2025.acl-long.934/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.934),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px4.p1.1)\.
- Weiet al\.\(2024\)J\. Wei, C\. Yang, X\. Song, Y\. Lu, N\. Hu, J\. Huang, D\. Tran, D\. Peng, R\. Liu, D\. Huang, C\. Du, and Q\. V\. LeLong\-form factuality in large language models\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 80756–80827\.External Links:[Document](https://dx.doi.org/10.52202/079017-2567),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/937ae0e83eb08d2cb8627fe1def8c751-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2608.17994#S2.SS0.SSS0.Px1.p1.1)\.
- Wenet al\.\(2025\)B\. Wen, J\. Yao, S\. Feng, C\. Xu, Y\. Tsvetkov, B\. Howe, and L\. L\. WangKnow your limits: a survey of abstention in large language models\.Transactions of the Association for Computational Linguistics13,pp\. 529–556\.External Links:ISSN 2307\-387X,[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00754),[Link](https://doi.org/10.1162/tacl_a_00754),https://direct\.mit\.edu/tacl/article\-pdf/doi/10\.1162/tacl\_a\_00754/2534960/tacl\_a\_00754\.pdfCited by:[§1](https://arxiv.org/html/2608.17994#S1.p3.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.External Links:1809\.09600,[Link](https://arxiv.org/abs/1809.09600)Cited by:[§4](https://arxiv.org/html/2608.17994#S4.SS0.SSS0.Px1.p1.1)\.
- Yuanet al\.\(2024\)T\. Yuan, Z\. He, L\. Dong, Y\. Wang, R\. Zhao, T\. Xia, L\. Xu, B\. Zhou, F\. Li, Z\. Zhang, R\. Wang, and G\. LiuR\-judge: benchmarking safety risk awareness for LLM agents\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1467–1490\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.79/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.79)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 46595–46623\.External Links:[Link](https://openreview.net/forum?id=uccHPGDlao)Cited by:[§1](https://arxiv.org/html/2608.17994#S1.p1.1)\.

## Appendix AProofs

### A\.1Clopper–Pearson UCB Validity

We show thatR^1−δupper​\(t\)\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\)in Eq\. \([7](https://arxiv.org/html/2608.17994#S3.E7)\) satisfiesPr⁡\(R⁡\(t\)≤R^1−δupper​\(t\)\)≥1−δ\\Pr\(R\(t\)\\leq\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\)\)\\geq 1\-\\delta\.

###### Proof\.

Under the Bernoulli structure, the observed failure count amongm^cal​\(t\)\\hat\{m\}\_\{\\mathrm\{cal\}\}\(t\)selected instances followsw^cal​\(t\)∼Binomial⁡\(m^cal​\(t\),R⁡\(t\)\)\\hat\{w\}\_\{\\mathrm\{cal\}\}\(t\)\\sim\\mathrm\{Binomial\}\(\\hat\{m\}\_\{\\mathrm\{cal\}\}\(t\),\\,R\(t\)\)\. Define the CDF of the empirical failure rate:

D⁡\(r∣R⁡\(t\)\)=Pr⁡\(R^​\(t\)≤r∣R⁡\(t\)\),D\(r\\mid R\(t\)\)=\\Pr\\\!\\left\(\\hat\{R\}\(t\)\\leq r\\mid R\(t\)\\right\),\(16\)whereR^​\(t\)=w^cal​\(t\)/m^cal​\(t\)\\hat\{R\}\(t\)=\\hat\{w\}\_\{\\mathrm\{cal\}\}\(t\)/\\hat\{m\}\_\{\\mathrm\{cal\}\}\(t\)\. By definition:

R^1−δupper​\(t\)=sup\{R∈\[0,1\]:D⁡\(r^cal​\(t\)∣R\)≥δ\}\.\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\)=\\sup\\\!\\left\\\{R\\in\[0,1\]:D\(\\hat\{r\}\_\{\\mathrm\{cal\}\}\(t\)\\mid R\)\\geq\\delta\\right\\\}\.\(17\)SinceD⁡\(r∣R\)D\(r\\mid R\)is monotonically decreasing inRR\(higher true rate makes low empirical rates less likely\),R⁡\(t\)\>R^1−δupper​\(t\)R\(t\)\>\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\)impliesD⁡\(r^cal​\(t\)∣R⁡\(t\)\)<δD\(\\hat\{r\}\_\{\\mathrm\{cal\}\}\(t\)\\mid R\(t\)\)<\\delta\. Therefore:

Pr⁡\(R⁡\(t\)\>R^1−δupper​\(t\)\)≤Pr⁡\(D⁡\(r^cal​\(t\)∣R⁡\(t\)\)<δ\)≤δ,\\Pr\\\!\\left\(R\(t\)\>\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\)\\right\)\\leq\\Pr\\\!\\left\(D\(\\hat\{r\}\_\{\\mathrm\{cal\}\}\(t\)\\mid R\(t\)\)<\\delta\\right\)\\leq\\delta,\(18\)where the last step uses the super\-uniformity property:Pr⁡\(D⁡\(r^cal​\(t\)∣R⁡\(t\)\)<δ\)≤δ\\Pr\(D\(\\hat\{r\}\_\{\\mathrm\{cal\}\}\(t\)\\mid R\(t\)\)<\\delta\)\\leq\\deltafor discrete distributions\. HencePr⁡\(R⁡\(t\)≤R^1−δupper​\(t\)\)≥1−δ\\Pr\(R\(t\)\\leq\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\)\)\\geq 1\-\\delta\. ∎

### A\.2Bernoulli Structure under Two\-Mode Routing

###### Proof of Lemma[1](https://arxiv.org/html/2608.17994#Thmlemma1)\.

For fixed thresholds\(t1,t2\)\(t\_\{1\},t\_\{2\}\), we define the selection indicator:

Ii:=\{U1\(i\)≤t1\}\+\{U1\(i\)\>t1andU2\(i\)≤t2\}\.I\_\{i\}:=\\mathbf\{1\}\\\!\\left\\\{U\_\{1\}^\{\(i\)\}\\\!\\leq\\\!t\_\{1\}\\right\\\}\+\\mathbf\{1\}\\\!\\left\\\{U\_\{1\}^\{\(i\)\}\\\!\>\\\!t\_\{1\}\\;\\text\{and\}\\;U\_\{2\}^\{\(i\)\}\\\!\\leq\\\!t\_\{2\}\\right\\\}\.\(19\)BothU1\(i\)=U⁡\(𝒥⁡\(𝐱i,y^i\)\)U\_\{1\}^\{\(i\)\}=U\(\\mathcal\{J\}\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\}\)\)andU2\(i\)=U⁡\(𝒥⁡\(𝐱i,y^i,ℰi\)\)U\_\{2\}^\{\(i\)\}=U\(\\mathcal\{J\}\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\},\\mathcal\{E\}\_\{i\}\)\)are deterministic functions of\(𝐱i,y^i\)\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\}\): the judge’s computation and the retrieval processℰi\\mathcal\{E\}\_\{i\}are both determined by the input\. HenceIiI\_\{i\}is a deterministic function of\(𝐱i,y^i\)\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\}\)\.

The correctness of the selected verdict is:

ci\\displaystyle c\_\{i\}=\{U1\(i\)≤t1\}c1\(i\)\\displaystyle=\\mathbf\{1\}\\\!\\left\\\{U\_\{1\}^\{\(i\)\}\\\!\\leq\\\!t\_\{1\}\\right\\\}c\_\{1\}^\{\(i\)\}\+\{U1\(i\)\>t1∧U2\(i\)≤t2\}c2\(i\),\\displaystyle\\;\+\\mathbf\{1\}\\\!\\left\\\{U\_\{1\}^\{\(i\)\}\\\!\>\\\!t\_\{1\}\\,\\wedge\\,U\_\{2\}^\{\(i\)\}\\\!\\leq\\\!t\_\{2\}\\right\\\}c\_\{2\}^\{\(i\)\},\(20\)which is a deterministic function of\(𝐱i,y^i,yi∗\)\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\},y^\{\*\}\_\{i\}\), since the routing decision depends on\(𝐱i,y^i\)\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\}\)and correctness additionally depends onyi∗y^\{\*\}\_\{i\}\.

Since\(𝐱i,y^i,yi∗\)\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\},y^\{\*\}\_\{i\}\)are i\.i\.d\. from𝒟\\mathcal\{D\}:

1. 1\.Independence: The selection events\{Ii=1\}\\\{I\_\{i\}=1\\\}depend on disjoint instances, so they are mutually independent\. Conditioning on selection preserves independence across instances\.
2. 2\.Identical distribution: Each selected instance satisfies\(𝐱i,y^i,yi∗\)∼𝒟t1,t2\(\\mathbf\{x\}\_\{i\},\\hat\{y\}\_\{i\},y^\{\*\}\_\{i\}\)\\sim\\mathcal\{D\}\_\{t\_\{1\},t\_\{2\}\}, where𝒟t1,t2:=𝒟\|I=1\\mathcal\{D\}\_\{t\_\{1\},t\_\{2\}\}:=\\mathcal\{D\}\\mid I=1is the conditional distribution\.

Note thatIi∈\{0,1\}I\_\{i\}\\in\\\{0,1\\\}because the two indicator terms are mutually exclusive: the second requiresU1\(i\)\>t1U\_\{1\}^\{\(i\)\}\>t\_\{1\}, which precludes the first\.

The failure indicatorsWi=1−ciW\_\{i\}=1\-c\_\{i\}over the selected subset are therefore i\.i\.d\.Bernoulli⁡\(R⁡\(t1,t2\)\)\\mathrm\{Bernoulli\}\(R\(t\_\{1\},t\_\{2\}\)\), whereR⁡\(t1,t2\)=𝔼⁡\[W∣I=1\]R\(t\_\{1\},t\_\{2\}\)=\\mathbb\{E\}\[W\\mid I=1\]\. ∎

##### Scope and limitations\.

The guarantee is tied to the retrieval snapshot used during calibration\. Because retrieval relies on a non\-stationary web index, a query may return different evidence at deployment than it did at calibration\. Such retrieval drift affects the guarantee only if it changes the conditional error rate of Mode 2 verdicts\. As long as this error rate remains stable, the calibrated thresholds continue to transfer\. We evaluate this scenario in Section[5\.5](https://arxiv.org/html/2608.17994#S5.SS5)by re\-querying the web several months after calibration and find that FDR control is preserved\. If the quality of the retrieved evidence changes substantially, recalibration restores the guarantee\. Likewise, any change to the task, candidate model, or judge model violates the exchangeability assumption and requires recalibration\.

## Appendix BExperimental Details

### B\.1Notation

Table[4](https://arxiv.org/html/2608.17994#A2.T4)provides a complete reference of all symbols\.

Table 4:Notation reference\.
### B\.2Implementation

##### Candidate generation\.

We generate candidate answers using greedy decoding for both Qwen3\-8B and LLaMA\-3\.1\-70B\. For Qwen3\-8B, thinking mode is disabled \(enable\_thinking=False\) to prevent the<think\>block from consuming the generation budget\.

##### Judge inference\.

For predictive entropy \(PE\), we perform a single forward pass at the verdict token position and extract the log\-probabilities for theTrueandFalsetokens\. PE is computed as binary entropy:−ptrue​log⁡ptrue−\(1−ptrue\)​log⁡\(1−ptrue\)\-p\_\{\\text\{true\}\}\\log p\_\{\\text\{true\}\}\-\(1\-p\_\{\\text\{true\}\}\)\\log\(1\-p\_\{\\text\{true\}\}\)\. This requires no additional inference beyond the greedy decoding pass used to produce the verdict\.

##### Retrieval\.

We query the Serper API with the original question and retrieve the top\-kkresults\. Each result contains a title, text snippet, and source URL\. Results are concatenated into a single evidence passage appended to the Mode 2 judge prompt\.

##### Calibration\.

We run 100 random 50/50 calibration/test splits\. For each split, we perform a grid search over all unique uncertainty values on the calibration set to find the threshold pair\(t^1,t^2\)\(\\hat\{t\}\_\{1\},\\hat\{t\}\_\{2\}\)that maximizes correctly accepted instances subject to the Clopper–Pearson UCB constraintR^1−δupper​\(t1,t2\)≤α\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\(t\_\{1\},t\_\{2\}\)\\leq\\alpha\. The grid search uses cumulative sums forO⁡\(\|𝒯1\|⋅\|𝒯2\|\)O\(\|\\mathcal\{T\}\_\{1\}\|\\cdot\|\\mathcal\{T\}\_\{2\}\|\)complexity\.

### B\.3Prompt Templates

#### B\.3\.1Judge Prompts

##### Mode 1 \(parametric evaluation\)\.

The judge receives a system prompt and a user message containing the question and candidate answer\. The prompt includes a one\-shot example to establish the output format\.

Mode 1: Judge System PromptYou are a helpful assistant acting as an impartial judge\. You will be given a Question and a Candidate Answer\. Your task is to judge whether the Candidate Answer is correct using your own knowledge\. If the Candidate Answer is correct, choose ‘True’, otherwise, choose ‘False’\. Provide a brief explanation for your decision\.Provide your response in the following format: Decision: \[True/False\] Explanation: \[Your brief explanation\]Example: Question: What is the capital of France? Candidate Answer: Paris is the capital of France\. Decision: True Explanation: Paris is indeed the capital of France\.

Mode 1: User MessageQuestion:\{question\}Candidate Answer:\{candidate\_answer\}

##### Mode 2 \(retrieval\-augmented evaluation\)\.

The judge additionally receives retrieved web search results\. Each result is formatted as:\[i\] Title / Snippet / Source: URL\.

Mode 2: Judge System PromptYou are a helpful assistant acting as an impartial judge\. You will be given a Question, a Candidate Answer, and Web Search Results\. Your task is to judge whether the Candidate Answer is correct using the provided search results\. If the search results support the Candidate Answer, choose ‘True’\. If the search results contradict it, choose ‘False’\. If the search results are inconclusive, use your best judgment\. Provide a brief explanation for your decision\.Provide your response in the following format: Decision: \[True/False\] Explanation: \[Your brief explanation\]

Mode 2: User MessageQuestion:\{question\}Candidate Answer:\{candidate\_answer\}Web Search Results:\{evidence\}

### B\.4LLM Evaluator Prompt

The LLM evaluator \(Qwen2\.5\-7B\-Instruct\) determines whether the candidate answer is semantically equivalent to any reference answer\([7](https://arxiv.org/html/2608.17994#bib.bib6)\), serving as the admission function for computing the ground\-truth labele∗e^\{\*\}\.

LLM Evaluator \(Admission Function\)You are a helpful assistant acting as an impartial evaluator\. Given a question, a candidate answer, and reference answers, determine if the candidate answer is correct\.The candidate answer is correct if it conveys the same meaning as ANY of the reference answers, even if the candidate answer is longer, contains additional explanation, uses synonyms, abbreviations, or alternate names\. Focus on whether the final answer or conclusion matches a reference, not on surface form\.Question:\{question\}Candidate Answer:\{candidate\_answer\}Reference Answers:\{references\}Is the candidate answer correct? Respond with ONLY “True” or “False”\.

## Appendix CAdditional Results

This appendix provides additional results that complement the main paper, including raw judge accuracy, an analysis of the joint framework across different risk levels, and per\-dataset coverage and FDR curves\.

### C\.1Judge Verdict Accuracy

Table[5](https://arxiv.org/html/2608.17994#A3.T5)reports the raw verdict accuracy of each judge in Mode 1 \(parametric\) and Mode 2 \(retrieval\-augmented\) before any calibration or selective prediction\. Retrieval consistently improves accuracy across all configurations, with the largest gains on PopQA \(up to 44 percentage points\), showing that retrieval provides useful evidence\.

Table 5:Judge verdict accuracy \(%\) before calibration\. Mode 1 uses parametric knowledge only whereas Mode 2 uses retrieval\-augmented evaluation \(k=3k\\\!=\\\!3\)\. LLM evaluator as admission function\.
### C\.2FDR and Coverage Across Risk Levels

Table[6](https://arxiv.org/html/2608.17994#A3.T6)reports FDR and coverage for the joint two\-mode framework \(two\-threshold routing with retrieval\) at all evaluatedα\\alphalevels for the Qwen\-8B candidate across all four datasets and all four judges \(LLM evaluator, PE,k=3k\\\!=\\\!3\)\.

Table 6:Full results for the joint two\-mode framework at allα\\alphalevels across all four datasets\. Candidate: Qwen3\-8B, LLM evaluator, PE,k=3k\\\!=\\\!3\.#### C\.2\.1Coverage Curves

Figure[3](https://arxiv.org/html/2608.17994#A3.F3)shows coverage as a function ofα\\alphaacross all 32 configurations\.

Figure 3:Coverage \(mean±\\pmstd over 100 splits\) across all 32 configurations\. Rows: candidates; columns: datasets\.
#### C\.2\.2Per\-Dataset FDR

Figures[4](https://arxiv.org/html/2608.17994#A3.F4)–[7](https://arxiv.org/html/2608.17994#A3.F7)provide per\-dataset FDR plots for detailed examination\.

Figure 4:FDR control on TriviaQA\.Figure 5:FDR control on NQ\-Open\.Figure 6:FDR control on PopQA\.Figure 7:FDR control on HotpotQA\.

## Appendix DAblations and Robustness

We evaluate the framework across several dimensions\. This includes pipeline design choices, the calibration procedure, the uncertainty score, and the choice of no\-retrieval baseline\. In all cases, the finite\-sample FDR guarantee is preserved, with conservative choices reducing coverage rather than compromising risk control\.

### D\.1Sensitivity Analyses

##### Correctness criterion \(admission function\)\.

Table[7](https://arxiv.org/html/2608.17994#A4.T7)compares three admission functions across three risk levels on a representative pair \(Qwen3\-8B candidate, Qwen3\-14B judge\)\. The LLM evaluator consistently achieves the highest coverage, with the gap most pronounced on knowledge\-intensive benchmarks: on NQ\-Open atα=0\.15\\alpha\\\!=\\\!0\.15, it reaches 56% while both exact match and token F1 remain at 0%\. This gap arises because string\-matching criteria penalize semantically correct but surface\-different answers, inflating the judge’s apparent error rate\. FDR control holds under all three criteria\.

Table 7:Sensitivity to admission function\. Candidate: Qwen3\-8B, Judge: Qwen3\-14B, UQ: PE \(white\-box\)\. FDR and coverage atα∈\{\.15,\.20,\.25\}\\alpha\\in\\\{\.15,\.20,\.25\\\}\.
##### Retrieval depth\.

Table[8](https://arxiv.org/html/2608.17994#A4.T8)ablates the number of retrieved passages \(k∈\{3,6,9\}k\\in\\\{3,6,9\\\}\) on a representative pair \(Llama\-70B candidate, Qwen3\-8B judge\)\. Coverage and FDR are stable across retrieval depths, with differences within 1–2 percentage points\.k=3k\\\!=\\\!3already provides sufficient evidence\.

Table 8:Retrieval depth ablation atα=0\.20\\alpha\\\!=\\\!0\.20\. Candidate: Llama\-70B, Judge: Qwen3\-8B, LLM evaluator\.
##### Confidence levelδ\\delta\.

Table[9](https://arxiv.org/html/2608.17994#A4.T9)varies the confidence level\. Tighteningδ\\deltafrom0\.050\.05to0\.010\.01trades coverage for confidence, with the cost concentrated at lowα\\alphaand on the harder dataset; FDR stays controlled throughout\.

Table 9:Sensitivity to the confidence levelδ\\delta\(coverage\)\. Candidate: Qwen3\-8B, Judge: Qwen3\-8B, LLM evaluator,k=3k\\\!=\\\!3, PE\. FDR is controlled at the targetα\\alphaunder both settings\.
##### Calibration\-set size\.

Table[10](https://arxiv.org/html/2608.17994#A4.T10)sweeps the calibration fractionncal/nn\_\{\\mathrm\{cal\}\}/nover the 2,000 items per dataset\. Coverage on the easier TriviaQA is stable across the range, while on HotpotQA it degrades gracefully as the calibration set shrinks; FDR is controlled at every size\.

Table 10:Sensitivity to calibration\-set size \(coverage\)\.ncal/nn\_\{\\mathrm\{cal\}\}/nis the fraction of the 2,000 items used for calibration\. Candidate: Qwen3\-8B, Judge: Qwen3\-8B, LLM evaluator,k=3k\\\!=\\\!3, PE\. FDR is controlled at the targetα\\alphafor every size\.

### D\.2Alternative Uncertainty Estimators

The FDR guarantee depends only on the binary correctness indicator\. As a result, the choice of uncertainty score affects coverage but not whether the target risk level is controlled\. In addition to the white\-box predictive entropy \(PE\) used in the main paper, we evaluate two supervised uncertainty scores derived from the judge’s last\-layer hidden state at the verdict position\. Following[17](https://arxiv.org/html/2608.17994#bib.bib39), we consider a small MLP \(*Probe*\) and a*LoRA\+Prompt*adapter fine\-tuned to predict verdict correctness\.

Both models are trained using 5\-fold cross\-validation on the calibration set so that every calibration instance receives an out\-of\-sample uncertainty score before conformal calibration\. Table[11](https://arxiv.org/html/2608.17994#A4.T11)reports the resulting coverage\. All three uncertainty scores satisfy the target FDR level across every setting\.

LoRA\+Prompt consistently achieves the highest coverage, particularly on datasets where PE is less informative\. For example, coverage increases from0\.040\.04to0\.440\.44on HotpotQA atα=0\.10\\alpha\\\!=\\\!0\.10and from0\.000\.00to0\.710\.71on PopQA atα=0\.05\\alpha\\\!=\\\!0\.05\. The Probe also improves over PE on more challenging datasets at moderate risk levels \(e\.g\., NQ\-Open atα=0\.15\\alpha\\\!=\\\!0\.15,0\.24→0\.420\.24\\rightarrow 0\.42\), although it performs worse at the lowest risk levels\.

Table 11:Coverage under three uncertainty scores: the white\-box predictive entropy \(PE\) used in the main paper and two supervised scores, a*Probe*and a*LoRA\+Prompt*adapter on the judge’s hidden states\. Candidate: Qwen3\-8B, Judge: Qwen3\-8B, LLM evaluator,k=3k\\\!=\\\!3\. FDR is controlled at the targetα\\alphafor all three\.
### D\.3Baseline Comparison

To isolate the contribution of retrieved evidence, Table[12](https://arxiv.org/html/2608.17994#A4.T12)compares our joint two\-mode policy against three controls that keep the identical calibration but change what the accepted verdict is based on\.*Direct*accepts only Mode 1 \(no retrieval\)\.*Self\-eval*adds a second Mode 1 pass in which the judge is asked whether its own initial verdict is correct, using no new evidence\.*Empty*runs the Mode 2 prompt with an empty evidence slot\. FDR is controlled for every variant\. The joint policy dominates all three at nearly every operating point, and the gap is largest on the knowledge\-intensive HotpotQA \(0\.740\.74versus at most0\.070\.07for the no\-retrieval controls atα=0\.20\\alpha\\\!=\\\!0\.20\)\. A second evaluation pass without new evidence \(Self\-eval\) barely helps, confirming that the coverage gains come from the retrieved evidence rather than from simply re\-querying the judge\.

Table 12:Baseline comparison \(coverage\)\. Candidate: Qwen3\-8B, Judge: Qwen3\-8B, LLM evaluator,k=3k\\\!=\\\!3, PE\. FDR is controlled at the targetα\\alphafor every variant\.
### D\.4Conservative Joint Calibration

We selected the deployed threshold pair\(t^1,t^2\)\(\\hat\{t\}\_\{1\},\\hat\{t\}\_\{2\}\)using the calibration data rather than fixing them beforehand\. Specifically, we choose the pair that maximizes coverage while satisfyingR^1−δupper≤α\\hat\{R\}^\{\\mathrm\{upper\}\}\_\{1\-\\delta\}\\leq\\alpha\(Eq\.[14](https://arxiv.org/html/2608.17994#S3.E14)\)\. Although the Clopper–Pearson bound holds for each candidate pair on its own, certifying the selected pair requires accounting for this search\. The conservative variant addresses this directly by evaluating all\|𝒯1\|​\|𝒯2\|\|\\mathcal\{T\}\_\{1\}\|\\,\|\\mathcal\{T\}\_\{2\}\|candidate pairs using the corrected confidence levelδ′=δ/\(\|𝒯1\|​\|𝒯2\|\)\\delta^\{\\prime\}=\\delta/\(\|\\mathcal\{T\}\_\{1\}\|\\,\|\\mathcal\{T\}\_\{2\}\|\)\. By a union\-bound argument, the guarantee then holds simultaneously for every candidate pair, implying that the selected pair satisfiesPr⁡\(R⁡\(t^1,t^2\)≤α\)≥1−δ\\Pr\(R\(\\hat\{t\}\_\{1\},\\hat\{t\}\_\{2\}\)\\leq\\alpha\)\\geq 1\-\\delta\. In practice, however, we find that the simpler uncorrected selection already provides effective risk control \(see Section[5](https://arxiv.org/html/2608.17994#S5)\)\. For this reason, we use the coverage\-maximizing thresholds in our main experiments, while treating the conservative procedure as a theoretically valid alternative\.

One subtlety is that the grid is data\-dependent, since𝒯1\\mathcal\{T\}\_\{1\}and𝒯2\\mathcal\{T\}\_\{2\}are the observed uncertainty values, making both the candidate pairs and their count random\. Because each mode contributes at mostNNdistinct thresholds, replacing\|𝒯1\|​\|𝒯2\|\|\\mathcal\{T\}\_\{1\}\|\\,\|\\mathcal\{T\}\_\{2\}\|with the data\-independent boundN2N^\{2\}\(equivalently, correcting at levelδ/N2\\delta/N^\{2\}\) makes the union bound hold over a fixed family and leaves the conclusion unchanged at negligible cost; pre\-specifying the grid removes the dependence altogether\.

Table[13](https://arxiv.org/html/2608.17994#A4.T13)compares the coverage of the main\-paper calibration \(*Pointwise*, per\-pair levelδ\\delta\) with this*Conservative*variant \(grid\-corrected levelδ′\\delta^\{\\prime\}\)\. FDR is controlled under both\. The correction is nearly free on the easier datasets \(TriviaQA, PopQA\) but costs substantial coverage on the harder ones \(NQ\-Open, HotpotQA\), where the empirical FDR sits close toα\\alpha\. Bonferroni is a worst\-case union bound and adjacent grid thresholds select almost the same instances, so it is very loose\. The truly attainable jointly\-valid coverage lies between the two rows which is much closer to the pointwise figures\.

Table 13:Conservative joint calibration \(coverage\)\. Per dataset,*Point\.*and*Cons\.*denote the pointwise and conservative coverage, andΔ=Cons\.−Point\.\\Delta=\\text\{Cons\.\}\-\\text\{Point\.\}\(computed on unrounded values\)\. Candidate: Qwen3\-8B, Judge: Qwen3\-8B, LLM evaluator,k=3k\\\!=\\\!3, PE\. FDR is controlled at the targetα\\alphaunder both variants\.

## Appendix EAnalysis

This section moves beyond aggregate metrics to analyze how retrieval improves evaluation, including its impact on judge uncertainty, examples where it corrects incorrect verdicts, and the residual failure modes\.

### E\.1Effect of Retrieval on Uncertainty

Retrieval does not lower the judge’s uncertainty uniformly\. Table[14](https://arxiv.org/html/2608.17994#A5.T14)reports, per dataset, the fraction of items on which Mode 2 uncertainty is higher than Mode 1 \(U2\>U1U\_\{2\}\>U\_\{1\}\) versus lower, with the mean signed changeΔ​U=U2−U1\\Delta U=U\_\{2\}\-U\_\{1\}within each group\. On most items retrieval reduces uncertainty, but on 14–29% it raises it, concentrated on the multi\-hop HotpotQA and NQ\-Open, where retrieved snippets are only partially relevant or sources disagree\. These are exactly the items the second thresholdt^2\\hat\{t\}\_\{2\}routes to abstention when retrieval fails to resolve the judge’s uncertainty\.

Table 14:Effect of retrieval on judge uncertainty\. Results are obtained with Qwen3\-8B as both candidate and judge, the LLM evaluator, andk=3k\\\!=\\\!3retrieval using predictive entropy \(PE\)\. The “raised” and “lowered” columns report the fraction of instances where retrieval increases or decreases uncertainty, respectively\. MeanΔ​U\\Delta Uis averaged within each group\.
### E\.2Qualitative Examples

Tables[15](https://arxiv.org/html/2608.17994#A5.T15)and[16](https://arxiv.org/html/2608.17994#A5.T16)show representative cases where the judge issues an incorrect verdict in Mode 1 but corrects it after retrieval in Mode 2\. In each case, Mode 1 confidently hallucinated while retrieved evidence grounded the judge in verifiable facts\.

Table 15:Examples where retrieval corrects the judge’s verdict\. Candidate: Qwen3\-8B, Judge: Qwen3\-14B\.Table 16:Retrieved evidence \(k=3k\\\!=\\\!3\) for the examples in Table[15](https://arxiv.org/html/2608.17994#A5.T15)\. The first result in each case directly contains the correct answer\.
### E\.3Failure Modes

When retrieval does not fix an incorrect Mode 1 verdict, the residual errors fall into four qualitative classes\. \(i\)*Off\-topic retrieval*: an ambiguous or under\-specified question yields search queries that miss the relevant evidence\. \(ii\)*Insufficient evidence*: the retrieved snippets are only partially relevant and lack the specific fact needed to verify the candidate\. \(iii\)*Judge reasoning error*: the snippets contain the relevant evidence, but the judge still returns a wrong verdict\. For example defaulting toFalsein the absence of an exact lexical match\. \(iv\)*Admission function disagreement*: the candidate is semantically close to the gold answer but the admission function scores it as different\. The two\-threshold policy sends many such cases to abstention rather than accepting them, which is why coverage rather than FDR absorbs the difficulty on the hardest datasets\.