Empirical Bayes Conformal Prediction for Vision and Language Models

# Empirical Bayes Conformal Prediction for Vision and Language Models
Source: [https://arxiv.org/html/2605.23189](https://arxiv.org/html/2605.23189)
Jiapeng Zeng1Yogesh Prabhu2Zhanpeng Zeng3Michael A\. Newton1Vikas Singh1 1University of Wisconsin–Madison 2University of California San Diego 3Xiamen University

###### Abstract

Conformal prediction \(CP\) gives distribution\-free coverage for modern vision and language models, but it is often forced to make a ranking decision from a single unstable nonconformity score\. Standard CP uses one realization, while average\-then\-calibrate variants smooth multiple realizations into a point estimate\. Both options discard the inconsistency that can help identify whether a candidate is indeed stable\. A weak answer can enter the conformal set even if the evidence is not strong, simply because one posterior sample or prompt phrasing made it look strong\. But variability can help distinguish a stable signal from noise\-driven fluctuations\. We describe an empirical Bayes conformal prediction framework that usesrr\-values to convert score variability into an uncertainty informed nonconformity score\. The resultingrr\-value estimates how likely a candidate’s latent score belongs to the top\-ranked group after accounting for both its mean score and its uncertainty\. It admits both a closed\-form Normal\-Normal empirical Bayes estimator and a nonparametric posterior\-sampling estimator\. Using therr\-value as the nonconformity score preserves the target conformal coverage while provably reducing the inclusion of high variance false candidates under mild regularity conditions\. Across image classification, CLIP\-based VLM benchmarks, and LLMs, we show thatrr\-value conformal prediction preserves target coverage while improving ranking stability and reducing set size when variability is informative, and reverting to CP\-like behavior when variability vanishes\.

## 1Introduction

Large language models \(LLMs\) and vision models such as Vision Transformers \(ViTs\)\[[8](https://arxiv.org/html/2605.23189#bib.bib54),[31](https://arxiv.org/html/2605.23189#bib.bib21),[18](https://arxiv.org/html/2605.23189#bib.bib22)\]are increasingly being used to inform decision making, e\.g\., in clinical deployments and finance where quantifying uncertainty and mitigating risk are essential\. Bayesian neural networks \(BNNs\)\[[15](https://arxiv.org/html/2605.23189#bib.bib9)\], MC\-dropout\[[9](https://arxiv.org/html/2605.23189#bib.bib39)\]and Deep Ensembles \(DE\)\[[17](https://arxiv.org/html/2605.23189#bib.bib10)\]are important ideas in linking trustworthiness and model variability\. But applying these methods directly to large pre\-trained models can be expensive\[[1](https://arxiv.org/html/2605.23189#bib.bib11)\]\. Conformal prediction \(CP\{\\rm CP\}\)\[[29](https://arxiv.org/html/2605.23189#bib.bib7)\]provides an alternative framework where instead of estimating uncertainty inside the model, we wrap the model’s scores with a calibrated prediction set\. Rather than attaching uncertainty to a single prediction, CP returns a set guaranteed to contain the true output with user\-specified probability under exchangeability\[[4](https://arxiv.org/html/2605.23189#bib.bib1)\]\. These properties have made CP a promising tool for both vision and language models\[[30](https://arxiv.org/html/2605.23189#bib.bib17),[27](https://arxiv.org/html/2605.23189#bib.bib18)\]\.

The ranking problem in Conformal Prediction\.Despite its distribution\-free coverage guarantee,CP\{\\rm CP\}is not designed to directly use epistemic uncertainty from model parameters or training randomness\. Standard CP relies on a single model output/score\. The guarantee controls whether the true label is included, but the usefulness of the set relies on how well this score orders true candidates ahead of false ones\. If several candidate labels receive similar scores, small fluctuations can change their order and therefore change the conformal set\. This ranking instability becomes more pronounced under model or data heterogeneity, where score variability is no longer negligible\. Existing results\[[26](https://arxiv.org/html/2605.23189#bib.bib4)\]show thatCP\{\\rm CP\}can struggle under statistical heterogeneity, resulting in unreliable coverage\. Fig\.[1](https://arxiv.org/html/2605.23189#S1.F1)shows an example\. For the same input image, different model instances sampled from the same posterior distribution of the model can produce different conformal prediction sets\. The takeaway is that score variability should not simply be ignored\. Instead, it can provide useful information for constructing more stable and efficient conformal rankings\.

![Refer to caption](https://arxiv.org/html/2605.23189v1/AISTATS/figures/set_sizes_vary.png)Figure 1:Posterior sampled models yield different CP sets for the same image, revealing epistemic instability\.This instability is not just due to posterior model sampling\. In vision models, it may arise from posterior uncertainty over model parameters; in VLMs and LLMs, it may arise from prompt paraphrasing or model based evaluation of candidate responses\. A high score can mean two very different things: stable evidence for a candidate or a fluctuation of an otherwise weak one \(which is when high\-variance candidates enter the conformal set\)\.CP\{\\rm CP\}sees only a single noisy realization, whileCPavg\{\\rm CP\}\_\{\\text\{avg\}\}averages realizations before calibration and smoothing this variability\. Instead of eliminating variability, our goal is to check what it tells us about the reliability of a candidate’s rank\.

Thecontributionof this work is to utilize what is considered a weakness of large models, namely their inconsistency\[[6](https://arxiv.org/html/2605.23189#bib.bib38),[22](https://arxiv.org/html/2605.23189#bib.bib27)\], as a signal for conformal efficiency\. We introduce an empirical Bayes conformal prediction framework \(CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}\) based on therr\-value\[[11](https://arxiv.org/html/2605.23189#bib.bib5)\]by measuring how likely a candidate’s latent score belongs to the top\-ranked group after accounting for both its estimated score and its uncertainty\. Theoretically, under exchangeability, using therr\-value as the nonconformity score preserves the target conformal coverage, while it reduces the inclusion probability of high variance false candidates and leads to smaller expected prediction sets\. We estimate this quantity either with a Normal–Normal empirical Bayes model, which gives a closed\-form expression and theoretical insight, or with an assumption\-free posterior\-sampling estimator\. The nonparametric version is a Monte Carlo estimator of the same posterior tail probability defining the theoreticalrr\-value\. Across image classification, VLM, and LLM tasks,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}behaves similarly toCP\{\\rm CP\}orCPavg\{\\rm CP\}\_\{\\text\{avg\}\}when variability is small, but produces smaller and more stable conformal sets when the variability is informative\.

## 2Preliminaries

We briefly review conformal prediction and then show, with examples, why relying on a single pre\-trained model, while neglecting model uncertainty, can impactCP\{\\rm CP\}reliability\.

Terminology/Notations\.Consider a dataset\{\(xi,yi\)\}i=1n\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}, wherexix\_\{i\}is an input such as an image, a question, or a prompt, andyiy\_\{i\}is the corresponding label or response\. For a given inputxx, let𝒰​\(x\)=\{u1,…,uK\}\\mathcal\{U\}\(x\)=\\\{u\_\{1\},\\ldots,u\_\{K\}\\\}denote the candidate outputs, such as class labels in image classification or candidate responses in language tasks\. The model, or an external evaluator, assigns each candidateuju\_\{j\}a numerical scoref​\(x\)jf\(x\)\_\{j\}\. For image classification,f​\(x\)jf\(x\)\_\{j\}can be the logit or probability of classjj; for LLM tasks, it can be a likelihood or a quality score assigned by another model\. We use the term "candidate" throughout for readability, which corresponds to the “unit” terminology in\[[11](https://arxiv.org/html/2605.23189#bib.bib5)\]\.

### 2\.1Conformal Prediction: Mechanism and Coverage

CP\{\\rm CP\}constructs a prediction set by comparing test scores to a calibration threshold\. Once a nonconformity score is fixed,CP\{\\rm CP\}provides a distribution free coverage guarantee under exchangeability\. Here, the size of the prediction set depends on how this score ranks candidate outputs\.

Setup\.Let\{\(xi,yi\)\}i=1n\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}be a calibration set\. For each calibration example, we compute a nonconformity scoreSi=S​\(xi,yi\)S\_\{i\}=S\(x\_\{i\},y\_\{i\}\)for the true output\. For classification, one choice isSi=1−softmax​\(f​\(xi\)\)yiS\_\{i\}=1\-\\mathrm\{softmax\}\\bigl\(f\(x\_\{i\}\)\\bigr\)\_\{y\_\{i\}\}, where smaller scores indicate more conforming labels\. For a significance levelα\\alpha, CP sets the thresholdBBto be the⌈\(n\+1\)​\(1−α\)⌉\\lceil\(n\+1\)\(1\-\\alpha\)\\rceil\-th smallest calibration score\. For a test inputxtestx\_\{\\mathrm\{test\}\}, we computeS​\(xtest,uj\)S\(x\_\{\\mathrm\{test\}\},u\_\{j\}\)for every candidateuj∈𝒰​\(xtest\)u\_\{j\}\\in\\mathcal\{U\}\(x\_\{\\mathrm\{test\}\}\)and include candidates whose scores are below the calibration threshold:𝒞​\(xtest\)=\{uj:S​\(xtest,uj\)<B\}\.\\mathcal\{C\}\(x\_\{\\mathrm\{test\}\}\)=\\\{u\_\{j\}:S\(x\_\{\\mathrm\{test\}\},u\_\{j\}\)<B\\\}\.In classification, the set contains distinct labels \(possible responses/answer options for language tasks\)\.

Coverage\.Under exchangeability between calibration and test data, the rank of the true test score among the calibration scores is approximately uniform\. Therefore,ℙ​\(ytest∈𝒞​\(xtest\)\)≥1−α\.\\mathbb\{P\}\\Bigl\(y\_\{\\mathrm\{test\}\}\\in\\mathcal\{C\}\(x\_\{\\mathrm\{test\}\}\)\\Bigr\)\\geq 1\-\\alpha\.This guarantee does not require a correctly specified model, but it does not by itself give small sets\. Set size depends on whether the nonconformity score ranks true candidates ahead of incorrect ones, which is where score variability becomes important\.

### 2\.2Impact of Variability on Conformal Set Selection

Modern models often produce variable scores for the same or similar input\. In vision models, this variability may come from posterior uncertainty over model parameters; in VLMs and LLMs, it may come from prompt paraphrasing or model based evaluation of candidate responses\.

Effect of variability\.To see why the variability matters, consider a toy setting with candidate logits

p∼𝒩​\(1,1\)andq∼𝒩​\(0,1000\)\.p\\sim\\mathcal\{N\}\(1,1\)\\quad\\text\{and\}\\quad q\\sim\\mathcal\{N\}\(0,1000\)\.Althoughpphas the larger mean, the variance ofqqis so large thatqqcan exceedppwith probability close to one half \(ℙ​\(q\>p\)=48\.47%\\mathbb\{P\}\(q\>p\)=48\.47\\%\)\. Thus, a high variance candidate can occasionally appear highly confident even when its stable signal is weak\. This is an issue forCP\{\\rm CP\}becauseCP\{\\rm CP\}observes only one realized score\. More generally, letg​\(x\)g\(x\)denote the distribution of model scores for inputxx, and letgM​\(x\)g\_\{M\}\(x\)be a realization draw fromg​\(x\)g\(x\)\. Since nonconformity scores are often functions of this realized score vector, variability ing​\(x\)g\(x\)can change candidate rankings and therefore alter the conformal set\. Figure[1](https://arxiv.org/html/2605.23189#S1.F1)shows this effect that different model instances sampled from the same approximate posterior can produce different conformal sets for the same image\. A natural alternative isCPavg\{\\rm CP\}\_\{\\text\{avg\}\}, which averages scores across multiple realizations\[[29](https://arxiv.org/html/2605.23189#bib.bib7)\]before applyingCP\{\\rm CP\}\. Averaging, however, removes information about whether a high score is stable or noise\-driven\. To construct efficient conformal sets, we need a score that uses both the estimated value and its variability\.

## 3What is anrr\-value?

The above section identifies the main limitation of standard conformal rankings:CP\{\\rm CP\}uses one realized score, whileCPavg\{\\rm CP\}\_\{\\text\{avg\}\}averages realizations before calibration\. However, neither approach directly uses score variability as a ranking signal\. We instead treat the ranking step as an empirical Bayes problem and use therr\-value as an uncertainty\-aware nonconformity score\. Here, we define therr\-value, give parametric and assumption\-free estimators, and show that the resulting conformal procedure preserves coverage while reducing set size under mild regularity conditions\.

Setup\.For a fixed inputxx, suppose each candidateuiu\_\{i\}has an unobserved latent scoreθi\\theta\_\{i\}, interpreted as its stable signal, or long\-run expected score, under the model or evaluation procedure\. The observed scoref​\(x\)if\(x\)\_\{i\}is a noisy realization of this signal, with noise level allowed to vary across candidates\. Therr\-value measures how likelyθi\\theta\_\{i\}is to lie in the top\-ranked group after accounting for both the observed score and its uncertainty\. We use this quantity as the nonconformity score inCP\{\\rm CP\}\.

### 3\.1Parametricrr\-value under a Normal–Normal model

We first present a parametric version of therr\-value under a Normal–Normal empirical Bayes model\. This model is mainly used to make the uncertainty\-aware ranking explicit: it gives a closed\-form expression for therr\-value and describes how score magnitude and score variability trade off\. We use it as an analytic approximation\. For logit scores, approximately Gaussian fluctuations are plausible when posterior samples, bootstrap samples, or lightweight adapter perturbations satisfy standard posterior asymptotics \(Bernstein–von Mises behavior\)\. They are also consistent with Gaussian\-process approximations to wide neural networks\. For each candidateuiu\_\{i\}, assume

θi​∼iid​𝒩​\(μ,τ2\),f​\(x\)i∣θi∼𝒩​\(θi,σi2\)\.\\theta\_\{i\}\\overset\{\\mathrm\{iid\}\}\{\\sim\}\\mathcal\{N\}\(\\mu,\\tau^\{2\}\),\\qquad f\(x\)\_\{i\}\\mid\\theta\_\{i\}\\sim\\mathcal\{N\}\(\\theta\_\{i\},\\sigma\_\{i\}^\{2\}\)\.Hereθi\\theta\_\{i\}represents the candidate’s stable latent score, whileσi2\\sigma\_\{i\}^\{2\}measures the variability of the observed score across posterior or perturbation samples\.

Under the Normal–Normal conjugate model, the posterior distribution ofθi\\theta\_\{i\}is

θi∣f​\(x\)i,σi2∼𝒩​\(μθi,σθi2\),μθi=τ2​f​\(x\)i\+σi2​μτ2\+σi2,σθi2=τ2​σi2τ2\+σi2\.\\theta\_\{i\}\\mid f\(x\)\_\{i\},\\sigma\_\{i\}^\{2\}\\sim\\mathcal\{N\}\(\\mu\_\{\\theta\_\{i\}\},\\sigma\_\{\\theta\_\{i\}\}^\{2\}\),\\quad\\mu\_\{\\theta\_\{i\}\}=\\frac\{\\tau^\{2\}f\(x\)\_\{i\}\+\\sigma\_\{i\}^\{2\}\\mu\}\{\\tau^\{2\}\+\\sigma\_\{i\}^\{2\}\},\\quad\\sigma\_\{\\theta\_\{i\}\}^\{2\}=\\frac\{\\tau^\{2\}\\sigma\_\{i\}^\{2\}\}\{\\tau^\{2\}\+\\sigma\_\{i\}^\{2\}\}\.\(1\)
The posterior mean shrinks the observed score toward the empirical Bayes centerμ\\mu, with stronger shrinkage for candidates whose scores have big variance\. Thus, the posterior distribution in Eq\.[1](https://arxiv.org/html/2605.23189#S3.E1)encodes both the estimated latent score and the uncertainty in that estimate\. In practice, the hyperparametersμ\\muandτ2\\tau^\{2\}can be estimated from data by empirical Bayes\.

For a fractionβ∈\(0,1\)\\beta\\in\(0,1\), letθβ\\theta\_\{\\beta\}denote the\(1−β\)\(1\-\\beta\)\-quantile of the latent score distribution\. The posterior tail probability

Vβ​\(f​\(x\)i,σi2\)=P​\{θi≥θβ∣f​\(x\)i,σi2\}V\_\{\\beta\}\\bigl\(f\(x\)\_\{i\},\\sigma\_\{i\}^\{2\}\\bigr\)=P\\bigl\\\{\\theta\_\{i\}\\geq\\theta\_\{\\beta\}\\mid f\(x\)\_\{i\},\\sigma\_\{i\}^\{2\}\\bigr\\\}is the probability, after observing the noisy score and its variability, that candidateiibelongs to the topβ\\betafraction of latent scores\. Thus,VβV\_\{\\beta\}is asking whether the observed score is large and whether the candidate is likely to remain near the top after uncertainty is taken into account\.

Under the Normal–Normal model, the corresponding optimal threshold for selecting the topβ\\betafraction has the closed form

tβ∗​\(σi2\)=θβ​\(1\+σi2τ2\)−μ​\(σi2τ2\)−zβ​σi​σi2\+τ2τ,t\_\{\\beta\}^\{\*\}\(\\sigma\_\{i\}^\{2\}\)=\\theta\_\{\\beta\}\\Bigl\(1\+\\frac\{\\sigma\_\{i\}^\{2\}\}\{\\tau^\{2\}\}\\Bigr\)\-\\mu\\Bigl\(\\frac\{\\sigma\_\{i\}^\{2\}\}\{\\tau^\{2\}\}\\Bigr\)\-\\frac\{z\_\{\\beta\}\\sigma\_\{i\}\\sqrt\{\\sigma\_\{i\}^\{2\}\+\\tau^\{2\}\}\}\{\\tau\},\(2\)wherezβz\_\{\\beta\}is chosen so that the selected fraction isβ\\beta\. The key point is that the threshold depends onσi2\\sigma\_\{i\}^\{2\}: two candidates with the same observed score can be ranked differently if one score is much less stable\. The full derivation is given in Appendix[C](https://arxiv.org/html/2605.23189#A3)\.

### 3\.2From Posterior Tail Probability torr\-values

The quantityVβV\_\{\\beta\}evaluates membership in a fixed top fractionβ\\beta\. Therr\-value combines these comparisons across allβ\\betalevels into a single ranking score\. Letλβ=1−Φ​\(zβ\)\\lambda\_\{\\beta\}=1\-\\Phi\(z\_\{\\beta\}\)\. Under the parametric threshold family, define

r​\(f​\(x\)i,σi2\)=inf\{β:Vβ​\(f​\(x\)i,σi2\)≥λβ\}=inf\{β:f​\(x\)i≥tβ∗​\(σi2\)\}\.r\\bigl\(f\(x\)\_\{i\},\\sigma\_\{i\}^\{2\}\\bigr\)=\\inf\\Bigl\\\{\\beta:V\_\{\\beta\}\\bigl\(f\(x\)\_\{i\},\\sigma\_\{i\}^\{2\}\\bigr\)\\geq\\lambda\_\{\\beta\}\\Bigr\\\}=\\inf\\Bigl\\\{\\beta:f\(x\)\_\{i\}\\geq t\_\{\\beta\}^\{\*\}\(\\sigma\_\{i\}^\{2\}\)\\Bigr\\\}\.
A smallerrr\-value means that the candidate passes a more selective top\-fraction threshold and is therefore ranked higher\. Unlike ranking byf​\(x\)if\(x\)\_\{i\}alone, therr\-value ranks candidates by how confidently their latent scores belong near the top after accounting for score variability\.

### 3\.3Non\-parametric estimation ofrr\-values

The Normal–Normal model is useful for logit scores and theoretical analysis, but it is not appropriate for all score types\. Probabilities are bounded in\[0,1\]\[0,1\], and VLM/LLM evaluator scores may be discrete, heavy\-tailed, or non\-Gaussian\. In these cases, we estimate the same posterior tail probabilityVβV\_\{\\beta\}directly from posterior or perturbation samples\. Thus, the parametric and nonparametric constructions are two estimators of the samerr\-value quantity\.

Suppose we haveMMposterior or perturbation samples\. For each posterior sample, we rank allKKcandidates\. Forβ=k/K\\beta=k/K, we estimate

Vk/K​\(Di\)≈1M​∑m=1M𝟏​\{ui​appears among the top​k​candidates in sample​m\},V\_\{k/K\}\(D\_\{i\}\)\\approx\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\mathbf\{1\}\\\{u\_\{i\}\\text\{ appears among the top \}k\\text\{ candidates in sample \}m\\\},whereDiD\_\{i\}denotes the posterior\-sample information for candidateii\. This frequency estimates how consistently candidateiiappears in the topkkpositions across posterior samples\. Candidates that are repeatedly ranked near the top receive smallrr\-values, while candidates that appear near the top occasionally are penalized\.

This assumption\-free method is a Monte Carlo estimator of the same posterior tail probabilityVβV\_\{\\beta\}that defines the theoreticalrr\-value\. Therefore, both estimators target the same uncertainty\-aware ranking quantity\. The parametric version gives closed form insight when the Gaussian approximation is appropriate, while the nonparametric version remains applicable without Normality assumptions\.

### 3\.4Coverage/Set‑Size Efficiency of CP withrr\-values

Algorithm 1Coverage Set Selection withrr\-value0:Calibration set

\{\(xi,yi\)\}i=1n\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}, test input

xnewx\_\{\\mathrm\{new\}\}, candidate outputs

\{u1,…,uK\}\\\{u\_\{1\},\\ldots,u\_\{K\}\\\}, significance level

α\\alpha
0:Prediction set

Cr​\(xnew\)C\_\{r\}\(x\_\{\\mathrm\{new\}\}\)
1:Estimate score variability using posterior samples, such as WBB adapters, model ensembles, MC\-dropout, or prompt paraphrases\.

2:Compute the calibration

rr\-values

ri=r​\(xi,yi\)r\_\{i\}=r\(x\_\{i\},y\_\{i\}\)for

i=1,…,ni=1,\\ldots,n\.

3:Set

r^\\hat\{r\}to be the

⌈\(n\+1\)​\(1−α\)⌉\\lceil\(n\+1\)\(1\-\\alpha\)\\rceil\-th smallest value among

\{ri\}i=1n\\\{r\_\{i\}\\\}\_\{i=1\}^\{n\}\.

4:For each candidate

uju\_\{j\}of

xnewx\_\{\\mathrm\{new\}\}, compute

r​\(xnew,uj\)r\(x\_\{\\mathrm\{new\}\},u\_\{j\}\)\.

5:Return

Cr​\(xnew\)=\{uj:r​\(xnew,uj\)<r^\}C\_\{r\}\(x\_\{\\mathrm\{new\}\}\)=\\\{u\_\{j\}:r\(x\_\{\\mathrm\{new\}\},u\_\{j\}\)<\\hat\{r\}\\\}\.

Coverage guarantee\.LetCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\},CP\{\\rm CP\}, andCPavg\{\\rm CP\}\_\{\\text\{avg\}\}denote conformal prediction usingrr\-values, standard CP, and average\-then\-CP, respectively\. For each calibration example\(xi,yi\)\(x\_\{i\},y\_\{i\}\), compute therr\-valueri=r​\(xi,yi\)r\_\{i\}=r\(x\_\{i\},y\_\{i\}\)of the true label\. Setr^\\hat\{r\}to be the⌈\(n\+1\)​\(1−α\)⌉\\lceil\(n\+1\)\(1\-\\alpha\)\\rceil\-th smallest calibrationrr\-value\. For a new inputxnewx\_\{\\mathrm\{new\}\}with candidates\{u1,…,uK\}\\\{u\_\{1\},\\ldots,u\_\{K\}\\\}, defineCr​\(xnew\)=\{uj:r​\(xnew,uj\)<r^\}\.C\_\{r\}\(x\_\{\\mathrm\{new\}\}\)=\\\{u\_\{j\}:r\(x\_\{\\mathrm\{new\}\},u\_\{j\}\)<\\hat\{r\}\\\}\.Under exchangeability,P​\(ynew∉Cr​\(xnew\)\)=P​\(rn\+1\>r^\)≤α\.P\\bigl\(y\_\{\\mathrm\{new\}\}\\notin C\_\{r\}\(x\_\{\\mathrm\{new\}\}\)\\bigr\)=P\(r\_\{n\+1\}\>\\hat\{r\}\)\\leq\\alpha\.

###### Theorem 3\.1\.

Consider a new test inputxnewx\_\{\\text\{new \}\}\. LetCr​\(xnew\)C\_\{r\}\(x\_\{\\mathrm\{new\}\}\),Cstd​\(xnew\)C\_\{\\mathrm\{std\}\}\(x\_\{\\mathrm\{new\}\}\), andCavg​\(xnew\)C\_\{\\mathrm\{avg\}\}\(x\_\{\\mathrm\{new\}\}\)denote the conformal sets produced byCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\},CP\{\\rm CP\}, andCPavg\{\\rm CP\}\_\{\\text\{avg\}\}, respectively\. For any false unit/labelu′u^\{\\prime\}, define the conditional inclusion probabilities:

Pinclstd​\(σ2∣μ0\)\\displaystyle P\_\{\\mathrm\{incl\}\}^\{\\mathrm\{std\}\}\\left\(\\sigma^\{2\}\\mid\\mu\_\{0\}\\right\)=P\(u′∈Cstd∣θu′=μ0,σu′2=σ2,Tstd\)\\displaystyle=P\\left\(u^\{\\prime\}\\in C\_\{\\mathrm\{std\}\}\\mid\\theta\_\{u^\{\\prime\}\}=\\mu\_\{0\},\\sigma\_\{u^\{\\prime\}\}^\{2\}=\\sigma^\{2\},T\_\{\\mathrm\{std\}\}\\right\)Pinclr​\(σ2∣μ0\)\\displaystyle P\_\{\\mathrm\{incl\}\}^\{r\}\\left\(\\sigma^\{2\}\\mid\\mu\_\{0\}\\right\)=P\(u′∈Cr∣θu′=μ0,σu′2=σ2,r∗\)\\displaystyle=P\\left\(u^\{\\prime\}\\in C\_\{r\}\\mid\\theta\_\{u^\{\\prime\}\}=\\mu\_\{0\},\\sigma\_\{u^\{\\prime\}\}^\{2\}=\\sigma^\{2\},r^\{\*\}\\right\)whereTstdT\_\{\\text\{std\}\}andr∗r^\{\*\}are the⌈\(n\+1\)​\(1−α\)⌉n\\frac\{\\lceil\(n\+1\)\(1\-\\alpha\)\\rceil\}\{n\}\-quantiles of corresponding non\-conformity scores of the calibration data, independent ofxnewx\_\{\\text\{new \}\}\. Then,

1. \(1\)If for any moderately small varianceσ2\\sigma^\{2\}, we havePinclr​\(σ2∣μ0\)≤Pinclstd​\(σ2∣μ0\)P\_\{\\mathrm\{incl\}\}^\{r\}\\left\(\\sigma^\{2\}\\mid\\mu\_\{0\}\\right\)\\leq P\_\{\\mathrm\{incl\}\}^\{\\mathrm\{std\}\}\\left\(\\sigma^\{2\}\\mid\\mu\_\{0\}\\right\), then𝔼​\[\|Cr\|\]<𝔼​\[\|Cstd\|\]\\mathbb\{E\}\\left\[\\left\|C\_\{r\}\\right\|\\right\]<\\mathbb\{E\}\\left\[\\left\|C\_\{\\text\{std \}\}\\right\|\\right\]\.
2. \(2\)There exists a constants∈\[0,∞\)s\\in\[0,\\infty\)s\.t\. if allσi2≥s\\sigma\_\{i\}^\{2\}\\geq sfori=1,…,Ki=1,\\ldots,K, then𝔼​\[\|Cr\|\]<𝔼​\[\|Cavg\|\]\\mathbb\{E\}\\\!\\bigl\[\\,\|C\_\{r\}\|\\,\\bigr\]\\;<\\;\\mathbb\{E\}\\\!\\bigl\[\\,\|C\_\{\\text\{\\rm avg\}\}\|\\,\\bigr\], whereCavgC\_\{\\text\{\\rm avg\}\}is obtained by settingσi2→0\\sigma\_\{i\}^\{2\}\\rightarrow 0\.

Smaller set size\.The coverage guarantee follows from conformal calibration; the efficiency gain comes from the ranking induced by therr\-value\. Eq\.[1](https://arxiv.org/html/2605.23189#S3.E1)converts the observed scoref​\(x\)if\(x\)\_\{i\}into a posterior distribution over the latent scoreθi\\theta\_\{i\}, so the ranking depends not only on the posterior meanμθi\\mu\_\{\\theta\_\{i\}\}but also on the posterior uncertaintyσθi2\\sigma\_\{\\theta\_\{i\}\}^\{2\}\. Eq\.[2](https://arxiv.org/html/2605.23189#S3.E2)shows the same dependence from the threshold angle: the optimal cutoff varies with the candidate\-specific varianceσi2\\sigma\_\{i\}^\{2\}\. Thus,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}uses the distribution of scores across posterior/perturbation samples \(CPavg\{\\rm CP\}\_\{\\text\{avg\}\}retains only the average\)\.

When variability vanishes, the distinction betweenCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}and first\-order conformal scores disappears\. In the limitσi2→0\\sigma\_\{i\}^\{2\}\\to 0, Eq\.[1](https://arxiv.org/html/2605.23189#S3.E1)givesμθi→f​\(x\)i\\mu\_\{\\theta\_\{i\}\}\\to f\(x\)\_\{i\}andσθi2→0\\sigma\_\{\\theta\_\{i\}\}^\{2\}\\to 0, while Eq\.[2](https://arxiv.org/html/2605.23189#S3.E2)reduces totβ∗​\(0\)=θβt\_\{\\beta\}^\{\*\}\(0\)=\\theta\_\{\\beta\}\. Thus, therr\-value reduces to a point\-estimate ranking, matching the behavior of average\-then\-CP when many posterior samples are averaged\. Conversely, when variability is non\-negligible and informative,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}uses it as a second\-order signal: unstable false labels are penalized, while stable high\-scoring candidates remain near the top\. Under the conditions of Theorem[3\.1](https://arxiv.org/html/2605.23189#S3.Thmtheorem1)\(2\), this yields smaller prediction sets under the same conformal coverage guarantee\.

We see thatCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}enjoys the same coverage property asCP\{\\rm CP\}andCPavg\{\\rm CP\}\_\{\\text\{avg\}\}, but it also returns smaller coverage sets\. The remaining question is how to obtain the variability estimates in practice\. Next, we instantiate the framework in two settings for vision and language models\.

## 4Conformal Coverage usingrr\-value for Vision Models

![Refer to caption](https://arxiv.org/html/2605.23189v1/AISTATS/figures/img_exp.png)
![Refer to caption](https://arxiv.org/html/2605.23189v1/AISTATS/figures/vit_b_plots_temp.png)

Figure 2:Left:WBB approximates model uncertainty efficiently by training adapter modules\[[13](https://arxiv.org/html/2605.23189#bib.bib15)\]instead of fully retraining the model\.Right:Comparison ofCP\{\\rm CP\}andCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}on ViT\-Base image classification in probability and logit settings\.In this section, we studyCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}in a vision setting where epistemic variability can be estimated from posterior model samples\. Our goal is to verify the predictions of Section 3\. When score variability is small,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}should behave similarly toCP\{\\rm CP\}orCPavg\{\\rm CP\}\_\{\\text\{avg\}\}; when variability is informative, therr\-value ranking should reduce unstable false label inclusion and produce smaller and more stable conformal sets\.

Experiment questions\.Our experimental evaluations for image classification focus on the following questions\(a\)DoesCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}preserve target coverage on standard vision backbones?\(b\)DoesCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}reduce toCP\{\\rm CP\}\-like behavior when variability is small?\(c\)When multiple posterior samples are available, doesCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}produce a more stable and efficient set thanCP\{\\rm CP\}orCPavg\{\\rm CP\}\_\{\\text\{avg\}\}?

Setup\.We evaluate ImageNet classification using ViT\-Large, ViT\-Base, ResNet\-50, and ResNet\-18\. For each inputxxand candidate classuiu\_\{i\}, the scoref​\(x\)if\(x\)\_\{i\}is either the pre\-softmax logit or the post\-softmax probability\. For logits, we computerr\-values using both the parametric and the nonparametric method\. For probabilities, which are bounded in\[0,1\]\[0,1\], we use only the nonparametric estimator to avoid imposing a misspecified Gaussian model\.

Variability under Image Classification\.Figure[2](https://arxiv.org/html/2605.23189#S4.F2)\(left\) illustrates how we estimate posterior score variability\. Models obtained by weighted bootstrapping approximate the posterior given the training data\[[23](https://arxiv.org/html/2605.23189#bib.bib3)\]\. However, repeatedly retraining large neural networks is expensive\. We freeze the pretrained backbone and attach lightweight adapter modules following the adapter design of\[[13](https://arxiv.org/html/2605.23189#bib.bib15)\]\. We then train these adapters under the Weighted Bayesian Bootstrap \(WBB\) framework\[[23](https://arxiv.org/html/2605.23189#bib.bib3)\]\. Each WBB adapter provides one approximate posterior draw of the model parameters given the training data\. The training cost remains modest because the adapters are lightweight and can be optimized in parallel\. In our experiments, we train up to 1,800 ViT\-Base adapters on a single A100 40GB GPU in about half an hour\. Thus, WBB adapters provide a computationally feasible way to approximate posterior variability for large\-scale models while keeping the pretrained model frozen\.

\(A\)rr\-value reduces toCP\{\\rm CP\}when variability is small\.The ViT\-Base results in Figure[2](https://arxiv.org/html/2605.23189#S4.F2)\(right\) show thatCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}matchesCP\{\\rm CP\}in both coverage and set size for probability and logit scores\. This is a sanity check rather than a failure case\. Here, WBB adapters estimate posterior score variability around a fixed pretrained backbone\. The number of adapters controls the precision of the estimate, while the variability itself reflects how much the posterior samples perturb the model outputs\. Since the pretrained backbone is strong and the lightweight adapters induce only modest perturbations, class\-wise variability is small\. As predicted by Theorem[3\.1](https://arxiv.org/html/2605.23189#S3.Thmtheorem1), when variability vanishes, therr\-value reduces to a first order ranking based on the point estimate\. Thus,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}does not distort the usual CP ranking when there is little uncertainty to exploit\. The same pattern appears for ViT\-Large, ResNet\-50, and ResNet\-18 \(see Appendix[D](https://arxiv.org/html/2605.23189#A4)\)\.

\(B\) How doesCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}differ?AlthoughCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}andCP\{\\rm CP\}can have similar aggregate coverage and set sizes, their rankings can differ on individual images\. Figure[3](https://arxiv.org/html/2605.23189#S4.F3)\(left\) shows examples where both methods include the true class but order the selected classes differently\. This ranking matters because conformal sets are often used as ordered lists of plausible labels\.

CP\{\\rm CP\}ranks classes using a single realized score, so a high but unstable score can outrank a more stable class\. In contrast,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}uses posterior score variability to measure how consistently each class remains near the top across WBB posterior samples\. As a result,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}can move the correct class upward;CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}often places the correct class higher in the coverage set \(see Appendix[D](https://arxiv.org/html/2605.23189#A4)\)\. This illustrates the benefit of using variability as a ranking signal\.

![Refer to caption](https://arxiv.org/html/2605.23189v1/AISTATS/figures/test.png)
![Refer to caption](https://arxiv.org/html/2605.23189v1/AISTATS/figures/custom_image.png)

Figure 3:Left:Single\-image comparison ofCP\{\\rm CP\}andCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}, whereCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}incorporates model variability to often rank the correct class higher; smallerrr\-values are better\.Right:Multi\-model comparison showing thatCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}produces smaller, more stable coverage sets thanCP\{\\rm CP\}\. See Appendix[D](https://arxiv.org/html/2605.23189#A4)\.\(C\) Informative variability leads to smaller and more stable sets\.The advantage ofCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}becomes pronounced when the variability across WBB posterior samples is informative\. ApplyingCP\{\\rm CP\}separately to each posterior sample can produce different conformal sets for the same image, with different sizes and compositions \(Figure[3](https://arxiv.org/html/2605.23189#S4.F3)\(right\)\)\. Since all posterior samples are plausible draws from the same approximate posterior, it is unclear whichCP\{\\rm CP\}set should be trusted\.CPavg\{\\rm CP\}\_\{\\text\{avg\}\}reduces this ambiguity by averaging scores before calibration, but it also discards class specific variability\. Therr\-value instead uses posterior samples as evidence of ranking stability\. Rather than choosing one posterior sample or averaging all scores,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}asks whether a candidate repeatedly appears among the top ranked classes\. As shown in Figure[3](https://arxiv.org/html/2605.23189#S4.F3)\(right\),CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}gives a more stable and smaller prediction set\. This matches the mechanism in Section 3 that high variance false labels are less likely to survive therr\-value ranking, while stable high latent score classes remain included\.

Summary\.The vision experiments support two conclusions\. First,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}preserves the coverage behavior ofCP\{\\rm CP\}that when WBB posterior variability is small, it behaves similarly toCP\{\\rm CP\}orCPavg\{\\rm CP\}\_\{\\text\{avg\}\}, as predicted by the zero variance limit\. Second, the same posterior samples reveal that sample specificCP\{\\rm CP\}sets can vary in size and composition\. Rather than choosing one posterior draw or averaging variability away,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}integrates the samples into a single uncertainty aware ranking, yielding more stable sets and reducing unstable false\-label inclusion\. Detailed timing benchmarks are provided in Appendix[B](https://arxiv.org/html/2605.23189#A2)\.

## 5Conformal Coverage usingrr\-value for VLMs and LLMs

![Refer to caption](https://arxiv.org/html/2605.23189v1/AISTATS/figures/variability.png)Figure 4:Pipeline for generating score variability via paraphrasing\. For each discrete input, either an LLM prompt or a CLIP label description, we generateppparaphrases using GPT\-4\.1\. We then score each paraphrase \(LLaMA: paraphrased prompt vs all candidate answers; CLIP: image vs\. paraphrased labels\), yielding a distribution of similarity scores reflecting the score’s variability\.Our experiments with VLMs and LLMs focus on the following questions:\(a\)How can we quantify variability in language\-related tasks?\(b\)When doesCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}improve overCP\{\\rm CP\}andCPavg\{\\rm CP\}\_\{\\text\{avg\}\}on VLM tasks?\(c\)How does the benefit ofCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}change as model accuracy increases and epistemic variability decreases?\(d\)CanCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}be applied to LLMs, especially for open\-ended response tasks? What are its limitations?

Setup\.We evaluate CLIPs on CIFAR\-10, CIFAR\-100, ImageNet, ImageNet\-A, ImageNet\-R, and EuroSAT onCP\{\\rm CP\},CPavg\{\\rm CP\}\_\{\\text\{avg\}\}, andCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}\. For CLIP Base\-Patch16, we use calibration/test splits of 250/250 for CIFAR\-10/100, 1500/1500 for ImageNet, and 500/500 for ImageNet\-A/R and EuroSAT, repeated over 100 random splits\. We also evaluate SigLIP2 and MobileCLIP2\. In particular, MobileCLIP2 S2\-S4 provide the same family sequence with increasing accuracy, allowing us to study howCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}behaves as models become stronger and more stable\.

Table 1:Coverage and Set Size \(α=0\.05\\alpha=0\.05\) on various datasets under CLIP\.CoverageSet SizeDatasetCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}CIFAR\-1095\.0%±1\.8%\\mathbf\{95\.0\\%\\\!\\pm\\\!1\.8\\%\}95\.7%±1\.7%95\.7\\%\\\!\\pm\\\!1\.7\\%95\.4%±1\.8%95\.4\\%\\\!\\pm\\\!1\.8\\%1\.3±0\.1\\mathbf\{1\.3\\\!\\pm\\\!0\.1\}1\.4±0\.21\.4\\\!\\pm\\\!0\.2\(−7\.1%\-7\.1\\%\)1\.4±0\.21\.4\\\!\\pm\\\!0\.2\(−7\.1%\-7\.1\\%\)CIFAR\-10095\.2%±1\.9%\\mathbf\{95\.2\\%\\\!\\pm\\\!1\.9\\%\}95\.8%±1\.9%95\.8\\%\\\!\\pm\\\!1\.9\\%95\.5%±1\.6%95\.5\\%\\\!\\pm\\\!1\.6\\%7\.8±0\.7\\mathbf\{7\.8\\\!\\pm\\\!0\.7\}8\.1±1\.98\.1\\\!\\pm\\\!1\.9\(−3\.7%\-3\.7\\%\)11\.3±2\.311\.3\\\!\\pm\\\!2\.3\(−31\.0%\-31\.0\\%\)ImageNet95\.1%±1\.5%\\mathbf\{95\.1\\%\\\!\\pm\\\!1\.5\\%\}95\.2%±1\.8%95\.2\\%\\\!\\pm\\\!1\.8\\%95\.6%±1\.9%95\.6\\%\\\!\\pm\\\!1\.9\\%5\.7±0\.9\\mathbf\{5\.7\\\!\\pm\\\!0\.9\}6\.9±1\.36\.9\\\!\\pm\\\!1\.3\(−17\.4%\-17\.4\\%\)10\.1±1\.710\.1\\\!\\pm\\\!1\.7\(−43\.6%\-43\.6\\%\)ImageNet\-A94\.8%±1\.6%\\mathbf\{94\.8\\%\\\!\\pm\\\!1\.6\\%\}95\.5%±1\.7%95\.5\\%\\\!\\pm\\\!1\.7\\%94\.7%±1\.5%94\.7\\%\\\!\\pm\\\!1\.5\\%21\.5±2\.6\\mathbf\{21\.5\\\!\\pm\\\!2\.6\}23\.5±3\.823\.5\\\!\\pm\\\!3\.8\(−8\.5%\-8\.5\\%\)25\.6±4\.125\.6\\\!\\pm\\\!4\.1\(−16\.0%\-16\.0\\%\)ImageNet\-R95\.1%±1\.7%\\mathbf\{95\.1\\%\\\!\\pm\\\!1\.7\\%\}95\.4%±1\.9%95\.4\\%\\\!\\pm\\\!1\.9\\%95\.6%±1\.9%95\.6\\%\\\!\\pm\\\!1\.9\\%3\.6±0\.4\\mathbf\{3\.6\\\!\\pm\\\!0\.4\}4\.1±0\.54\.1\\\!\\pm\\\!0\.5\(−12\.2%\-12\.2\\%\)4\.6±0\.44\.6\\\!\\pm\\\!0\.4\(−21\.7%\-21\.7\\%\)EuroSAT94\.9%±1\.7%\\mathbf\{94\.9\\%\\\!\\pm\\\!1\.7\\%\}94\.7%±1\.5%94\.7\\%\\\!\\pm\\\!1\.5\\%95\.2%±1\.2%95\.2\\%\\\!\\pm\\\!1\.2\\%4\.9±0\.2\\mathbf\{4\.9\\\!\\pm\\\!0\.2\}5\.2±0\.25\.2\\\!\\pm\\\!0\.2\(−5\.8%\-5\.8\\%\)5\.6±0\.35\.6\\\!\\pm\\\!0\.3\(−12\.5%\-12\.5\\%\)

Table 2:ImageNet coverage and set size atα=0\.05\\alpha=0\.05\. Accuracy and coverage are reported in percent\. Parentheses show relative set size reductions ofCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}over each baseline\.Coverage \(%\)Set sizeModelAccuracyCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}SigLIP2\-B/3265\.665\.694\.9±1\.3\\mathbf\{94\.9\\pm 1\.3\}95\.1±1\.495\.1\\pm 1\.495\.1±1\.295\.1\\pm 1\.213\.0±1\.6\\mathbf\{13\.0\\pm 1\.6\}14\.4±2\.614\.4\\pm 2\.6\(\-9\.9%\)19\.6±2\.819\.6\\pm 2\.8\(\-33\.5%\)SigLIP2\-B/1672\.172\.194\.9±1\.3\\mathbf\{94\.9\\pm 1\.3\}95\.0±1\.295\.0\\pm 1\.295\.4±1\.495\.4\\pm 1\.47\.3±1\.6\\mathbf\{7\.3\\pm 1\.6\}8\.3±1\.38\.3\\pm 1\.3\(\-13\.0%\)10\.7±2\.810\.7\\pm 2\.8\(\-32\.3%\)SigLIP2\-L/1673\.673\.694\.9±1\.4\\mathbf\{94\.9\\pm 1\.4\}95\.3±1\.295\.3\\pm 1\.295\.0±1\.395\.0\\pm 1\.38\.8±1\.6\\mathbf\{8\.8\\pm 1\.6\}9\.8±1\.99\.8\\pm 1\.9\(\-10\.6%\)12\.3±2\.312\.3\\pm 2\.3\(\-28\.5%\)SigLIP2\-SO400M/1675\.175\.194\.8±1\.3\\mathbf\{94\.8\\pm 1\.3\}95\.3±1\.295\.3\\pm 1\.295\.3±1\.395\.3\\pm 1\.38\.8±1\.2\\mathbf\{8\.8\\pm 1\.2\}9\.5±1\.39\.5\\pm 1\.3\(\-7\.2%\)10\.9±2\.110\.9\\pm 2\.1\(\-19\.7%\)MobileCLIP2\-S275\.875\.895\.0±1\.2\\mathbf\{95\.0\\pm 1\.2\}95\.1±1\.495\.1\\pm 1\.495\.3±1\.495\.3\\pm 1\.44\.0±0\.5\\mathbf\{4\.0\\pm 0\.5\}4\.4±0\.64\.4\\pm 0\.6\(\-8\.2%\)4\.4±0\.74\.4\\pm 0\.7\(\-8\.0%\)MobileCLIP2\-B76\.876\.895\.3±1\.1\\mathbf\{95\.3\\pm 1\.1\}95\.3±1\.295\.3\\pm 1\.295\.2±1\.395\.2\\pm 1\.33\.5±0\.7\\mathbf\{3\.5\\pm 0\.7\}3\.5±0\.53\.5\\pm 0\.5\(\-0\.5%\)3\.6±0\.83\.6\\pm 0\.8\(\-4\.2%\)MobileCLIP2\-S378\.678\.695\.1±1\.2\\mathbf\{95\.1\\pm 1\.2\}95\.4±1\.495\.4\\pm 1\.495\.2±1\.395\.2\\pm 1\.33\.1±0\.4\\mathbf\{3\.1\\pm 0\.4\}3\.2±0\.43\.2\\pm 0\.4\(\-1\.7%\)3\.3±0\.43\.3\\pm 0\.4\(\-6\.1%\)MobileCLIP2\-S479\.579\.595\.1±1\.2\\mathbf\{95\.1\\pm 1\.2\}95\.0±1\.495\.0\\pm 1\.495\.1±1\.495\.1\\pm 1\.43\.0±0\.4\\mathbf\{3\.0\\pm 0\.4\}3\.0±0\.43\.0\\pm 0\.4\(\-0\.4%\)3\.1±0\.73\.1\\pm 0\.7\(\-4\.0%\)\(A\) Variability in VLMs and LLMs\.Variability in VLMs and LLMs differs from that in image models\. In image classification, variability mainly comes from uncertainty in model training, whereas in VLMs and LLMs it also comes from how prompts are written\. Since prompt format can significantly affect model responses\[[10](https://arxiv.org/html/2605.23189#bib.bib16)\], we estimate this variability by paraphrasing prompts or descriptions and then scoring each candidate across paraphrases\. As shown in Figure[4](https://arxiv.org/html/2605.23189#S5.F4), we use GPT\-4\.1 to generate multiple paraphrases for each prompt and evaluate candidates across these variations\. The resulting score distribution captures prompt induced variability in model outputs\. More details and examples of paraphrasing are provided in Appendix[F](https://arxiv.org/html/2605.23189#A6)and Appendix[H](https://arxiv.org/html/2605.23189#A8)\.

\(B\)CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}helps when variability is informative\.Under CLIP, we compareCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}withCP\{\\rm CP\}andCPavg\{\\rm CP\}\_\{\\text\{avg\}\}across datasets\. Since CP coverage admits both lower and upper bounds\[[4](https://arxiv.org/html/2605.23189#bib.bib1)\], coverage closer to1−α1\-\\alphaindicates less conservative calibration\. Table[1](https://arxiv.org/html/2605.23189#S5.T1)shows thatCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}achieves near\-target coverage while consistently reducing set size\. On ImageNet, the average conformal set size underCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}is reduced by nearly half compared toCP\{\\rm CP\}\. These results indicate that variability provides useful ranking information, makingCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}a viable alternative when such variability is available\.

\(C\) Gains shrink as accuracy increases\.We next evaluate eight VLM variants spanning SigLIP2 and MobileCLIP2, as shown in Table[2](https://arxiv.org/html/2605.23189#S5.T2)\. The results show a clear trend in efficiency gains that models with lower accuracy tend to benefit more fromCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}, while stronger models obtain smaller but still consistent reductions\. This is consistent with Theorem[3\.1](https://arxiv.org/html/2605.23189#S3.Thmtheorem1)\. As models become more accurate and stable, conformal sets become smaller, leaving fewer unstable false labels forCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}to remove\.

This trend is visible in the percentage reductions reported in Table[2](https://arxiv.org/html/2605.23189#S5.T2), full coverage and set size results are in Appendix[E](https://arxiv.org/html/2605.23189#A5)\. Moving from the lowest accuracy model to the strongest model, default accuracy increases from65\.6%65\.6\\%to79\.5%79\.5\\%, while the relative gain ofCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}overCP\{\\rm CP\}decreases from33\.5%33\.5\\%to4\.0%4\.0\\%\. The gain overCPavg\{\\rm CP\}\_\{\\text\{avg\}\}shows the same pattern, decreasing from9\.9%9\.9\\%to0\.4%0\.4\\%\. Although the trend is not perfectly monotone for every intermediate model, the overall trend is clear\. Stronger models already produce smaller baseline conformal sets, leaving fewer unstable false labels forCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}to remove\. This matches Theorem[3\.1](https://arxiv.org/html/2605.23189#S3.Thmtheorem1)thatCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}gives the largest gains when epistemic variability is informative and naturally approachesCP\{\\rm CP\}\-like behavior when variability is small\.

\(D\) LLM tasks\.For close\-ended LLM tasks, we evaluate MMLU using LLaMA 1B atα=0\.05\\alpha=0\.05\. Since MMLU has only four answer options, ranking variability is limited, andCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}usually performs similarly toCPavg\{\\rm CP\}\_\{\\text\{avg\}\}\. Gains appear in subjects with more variability\. Average set size decreases from 2\.865 to 2\.754 for Marketing and from 3\.422 to 3\.307 for Clinical Knowledge\. Because MMLU has only four answer options, the absolute gains are modest\. This experiment is mainly a sanity check for applyingCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}to discrete LLM candidate sets\. Appendix[G](https://arxiv.org/html/2605.23189#A7)provides additional comparisons with Conformal Language Modeling\[[27](https://arxiv.org/html/2605.23189#bib.bib18)\]and further GPQA results\.

For open\-ended tasks, standard conformal sets are difficult to define because the output space is effectively unbounded and correctness is not binary\.CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}does not remove this limitation, but it can improve ranking resolution\. When evaluator scores are coarse, e\.g\., 0\-10, many responses have tied scores\. Variability across paraphrases or repeated evaluations helps break these ties and yields a more informative ordering \(see Appendix[F](https://arxiv.org/html/2605.23189#A6)\)\.

Summary\.Across VLM and LLM tasks,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}preserves conformal coverage and improves efficiency when variability is informative\. Its gains are large in higher variability regimes such as SigLIP2 and smaller for stronger, more stable models such as MobileCLIP2\. This matches the theory thatCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}reduces toCP\{\\rm CP\}\-like behavior when variability is negligible and removes unstable false candidates when variability is useful, makingCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}a strong alternative toCP\{\\rm CP\}\.

## 6Related Work

Conformal prediction provides distribution\-free uncertainty quantification and has been applied to vision models, diffusion models, and LLMs\[[29](https://arxiv.org/html/2605.23189#bib.bib7),[4](https://arxiv.org/html/2605.23189#bib.bib1),[12](https://arxiv.org/html/2605.23189#bib.bib24),[27](https://arxiv.org/html/2605.23189#bib.bib18),[30](https://arxiv.org/html/2605.23189#bib.bib17)\]\. We focus on work most related to improving conformal efficiency and using uncertainty information in conformal prediction\.

#### Risk aware and ambiguous label CP\.

Risk controlling and conformal risk control methods extend CP from 0/1 miscoverage to user\-defined losses or expected risk control\[[2](https://arxiv.org/html/2605.23189#bib.bib29),[3](https://arxiv.org/html/2605.23189#bib.bib42)\]\. Separately, ambiguous label methods model uncertainty in the observed label caused by annotator disagreement or distribution valued labels\[[14](https://arxiv.org/html/2605.23189#bib.bib43),[5](https://arxiv.org/html/2605.23189#bib.bib44),[28](https://arxiv.org/html/2605.23189#bib.bib52)\]\. These works address different sources of uncertainty\. Our method keeps the standard marginal coverage target and treats the observed label as the calibration target while using epistemic variability to improve candidate ranking\.

#### Set size optimization and aggregation\.

Other works reduce conformal set size by aggregating conformity scores or optimizing prediction set length under validity constraints\[[20](https://arxiv.org/html/2605.23189#bib.bib46),[16](https://arxiv.org/html/2605.23189#bib.bib45)\]\. Recent ensemble conformal methods further improve efficiency by aggregating score vectors or prediction regions across multiple models\[[25](https://arxiv.org/html/2605.23189#bib.bib49)\]\. These approaches use multiple scores or multiple models to construct a stronger first order conformity score, or to select shorter valid sets after optimization\. In contrast, our method does not learn aggregation weights, optimize a separate set size objective, or combine already constructed prediction sets\. It uses posterior samples to estimate the stability of each candidate’s rank and then uses therr\-value as an uncertainty aware nonconformity score\. Thus, variability is used as a second order ranking signal rather than being averaged before calibration\.

#### Uncertainty aware conformal prediction\.

Several uncertainty aware CP methods use predictive distributions, local variance estimates, or ensemble outputs to adjust conformity scores\[[24](https://arxiv.org/html/2605.23189#bib.bib47),[7](https://arxiv.org/html/2605.23189#bib.bib48),[25](https://arxiv.org/html/2605.23189#bib.bib49)\]\. These methods mainly target regression, continuous outcomes, or ensemble score aggregation\. Recent rank based CP methods use model induced label orderings as conformity scores\[[21](https://arxiv.org/html/2605.23189#bib.bib53)\]\. Our method is related in that candidate ordering is central, but differs in how the ordering is obtained\. Instead of relying on a single model induced rank, we estimate rank stability across posterior samples to penalize high scoring but unstable candidates\. We use posterior tail probabilities to rank discrete candidates in classification, VLM, and LLM tasks, directly targeting ranking instability when labels or responses have similar scores but different epistemic stability\.

## 7Conclusion

Our work introducesCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}, an empirical Bayes conformal prediction framework for vision and language models\. By using score variability as a second order signal,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}preserves theCP\{\\rm CP\}coverage guarantee while producing more stable and efficient sets\. When variability is small,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}behaves similarly toCP\{\\rm CP\}orCPavg\{\\rm CP\}\_\{\\text\{avg\}\}; when variability is informative, it reduces unstable false candidates and yields smaller sets\. This effect is especially pronounced in VLM tasks, where prompt induced variability provides useful ranking information\. Our code will be publicly available\.

Limitations\.The effectiveness ofCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}depends on the quality of variability estimation\. Adapter based WBB provides an efficient way to approximate posterior variability, but it may introduce a small drop in raw accuracy, especially for high performing models\. In addition, the gains ofCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}are largest when epistemic variability is non negligible\. For highly stable models or tasks with few candidate options, the method may behave similarly toCPavg\{\\rm CP\}\_\{\\text\{avg\}\}and yield only modest improvements\.

## References

- \[1\]M\. Abdar, F\. Pourpanah, S\. Hussain, D\. Rezazadegan, L\. Liu, M\. Ghavamzadeh, P\. Fieguth, X\. Cao, A\. Khosravi, U\. R\. Acharya, V\. Makarenkov, and S\. Nahavandi\(2021\-05\)A review of uncertainty quantification in deep learning: Techniques, applications and challenges\.Information Fusion76,pp\. 243–297\.External Links:[Document](https://dx.doi.org/10.1016/j.inffus.2021.05.008),[Link](https://arxiv.org/abs/2011.06225)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p1.1)\.
- \[2\]\(2021\-10\)Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control\.External Links:[Link](https://arxiv.org/abs/2110.01052)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px1.p1.1)\.
- \[3\]A\. N\. Angelopoulos, S\. Bates, A\. Fisch, L\. Lei, and T\. Schuster\(2024\)Conformal Risk Control\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=33XGfHLtZg)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px1.p1.1)\.
- \[4\]A\. N\. Angelopoulos and S\. Bates\(2023\-01\)Conformal prediction: A gentle introduction\.Foundations and Trends® in Machine Learning16\(4\),pp\. 494–591\.External Links:[Document](https://dx.doi.org/10.1561/2200000101),[Link](https://doi.org/10.1561/2200000101)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p1.1),[§5](https://arxiv.org/html/2605.23189#S5.p4.9),[§6](https://arxiv.org/html/2605.23189#S6.p1.1)\.
- \[5\]M\. Caprio, D\. Stutz, S\. Li, and A\. Doucet\(2025\)Conformalized Credal Regions for Classification with Ambiguous Ground Truth\.Note:OpenReviewExternal Links:[Link](https://openreview.net/forum?id=L7sQ8CW2FY)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px1.p1.1)\.
- \[6\]A\. Chen, J\. Phang, A\. Parrish, V\. Padmakumar, C\. Zhao, S\. R\. Bowman, and K\. Cho\(2023\-05\)Two failures of Self\-Consistency in the Multi\-Step reasoning of LLMs\.External Links:[Link](https://arxiv.org/abs/2305.14279)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p4.7)\.
- \[7\]V\. Chernozhukov, K\. Wüthrich, and Y\. Zhu\(2021\)Distributional Conformal Prediction\.Proceedings of the National Academy of Sciences118\(48\),pp\. e2107794118\.External Links:[Document](https://dx.doi.org/10.1073/pnas.2107794118),[Link](https://www.pnas.org/doi/10.1073/pnas.2107794118)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px3.p1.1)\.
- \[8\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby\(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=YicbFdNTTy)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p1.1)\.
- \[9\]Y\. Gal and Z\. Ghahramani\(2015\-06\)Dropout as a Bayesian approximation: representing model uncertainty in deep learning\.External Links:[Link](https://arxiv.org/abs/1506.02142)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p1.1)\.
- \[10\]J\. He, M\. Rungta, D\. Koleczek, A\. Sekhon, F\. X\. Wang, and S\. Hasan\(2024\)Does prompt formatting have any impact on llm performance?\.External Links:2411\.10541,[Link](https://arxiv.org/abs/2411.10541)Cited by:[§5](https://arxiv.org/html/2605.23189#S5.p3.1)\.
- \[11\]N\. C\. Henderson and M\. A\. Newton\(2015\-11\)Making the cut: Improved ranking and selection for Large\-Scale inference\.Journal of the Royal Statistical Society Series B \(Statistical Methodology\)78\(4\),pp\. 781–804\.External Links:[Document](https://dx.doi.org/10.1111/rssb.12131),[Link](https://doi.org/10.1111/rssb.12131)Cited by:[§C\.1](https://arxiv.org/html/2605.23189#A3.SS1.p1.7),[§C\.1](https://arxiv.org/html/2605.23189#A3.SS1.p5.2),[§1](https://arxiv.org/html/2605.23189#S1.p4.7),[§2](https://arxiv.org/html/2605.23189#S2.p2.9)\.
- \[12\]E\. Horwitz and Y\. Hoshen\(2022\-11\)Conffusion: confidence intervals for diffusion models\.External Links:[Link](https://arxiv.org/abs/2211.09795)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.p1.1)\.
- \[13\]N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, D\. L\. Quentin, A\. Gesmundo, M\. Attariyan, and S\. Gelly\(2019\-02\)Parameter\-Efficient Transfer Learning for NLP\.External Links:[Link](https://arxiv.org/abs/1902.00751)Cited by:[Figure 2](https://arxiv.org/html/2605.23189#S4.F2),[Figure 2](https://arxiv.org/html/2605.23189#S4.F2.6.2.3),[§4](https://arxiv.org/html/2605.23189#S4.p4.1)\.
- \[14\]A\. Javanmardi, D\. Stutz, and E\. Hüllermeier\(2024\)Conformalized Credal Set Predictors\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/d42a8bf2f40555d4a5120300f98c88f6-Paper-Conference.pdf)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px1.p1.1)\.
- \[15\]L\. V\. Jospin, H\. Laga, F\. Boussaid, W\. Buntine, and M\. Bennamoun\(2022\-04\)Hands\-On Bayesian Neural Networks—A tutorial for deep learning users\.IEEE Computational Intelligence Magazine17\(2\),pp\. 29–48\.External Links:[Document](https://dx.doi.org/10.1109/mci.2022.3155327),[Link](https://doi.org/10.1109/mci.2022.3155327)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p1.1)\.
- \[16\]S\. Kiyani, G\. Pappas, and H\. Hassani\(2024\)Length Optimization in Conformal Prediction\.External Links:[Link](https://arxiv.org/abs/2406.18814)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px2.p1.1)\.
- \[17\]B\. Lakshminarayanan, A\. Pritzel, and C\. Blundell\(2016\-12\)Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles\.External Links:[Link](https://arxiv.org/abs/1612.01474)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p1.1)\.
- \[18\]Z\. Li, X\. Wu, H\. Du, H\. Nghiem, and G\. Shi\(2025\-01\)Benchmark Evaluations, Applications, and Challenges of large Vision Language Models: a survey\.External Links:[Link](https://arxiv.org/abs/2501.02189)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p1.1)\.
- \[19\]S\. Lin, J\. Hilton, and O\. Evans\(2022\)TruthfulQA: measuring how models mimic human falsehoods\.External Links:2109\.07958,[Link](https://arxiv.org/abs/2109.07958)Cited by:[Appendix F](https://arxiv.org/html/2605.23189#A6.p3.4)\.
- \[20\]R\. Luo and Z\. Zhou\(2025\)Conformity Score Averaging for Classification\.InProceedings of the 42nd International Conference on Machine Learning,External Links:[Link](https://proceedings.mlr.press/v267/luo25v.html)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px2.p1.1)\.
- \[21\]R\. Luo and Z\. Zhou\(2026\)Trustworthy classification through rank\-based conformal prediction sets\.Pattern Recognition172,pp\. 112330\.External Links:[Document](https://dx.doi.org/10.1016/j.patcog.2025.112330)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px3.p1.1)\.
- \[22\]E\. Mitchell, J\. J\. Noh, S\. Li, W\. S\. Armstrong, A\. Agarwal, P\. Liu, C\. Finn, and C\. D\. Manning\(2022\-11\)Enhancing Self\-Consistency and Performance of Pre\-Trained Language Models through Natural Language Inference\.External Links:[Link](https://arxiv.org/abs/2211.11875)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p4.7)\.
- \[23\]M\. A\. Newton, N\. G\. Polson, and J\. Xu\(2020\)Weighted bayesian bootstrap for scalable posterior distributions\.Canadian Journal of Statistics49\(2\),pp\. 421–437\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1002/cjs.11570)Cited by:[§4](https://arxiv.org/html/2605.23189#S4.p4.1)\.
- \[24\]D\. Nolte, S\. Ghosh, and R\. Pal\(2024\)Efficient Normalized Conformal Prediction and Uncertainty Quantification for Anti\-Cancer Drug Sensitivity Prediction with Deep Regression Forests\.External Links:[Link](https://arxiv.org/abs/2402.14080)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px3.p1.1)\.
- \[25\]E\. Ochoa Rivera, Y\. Patel, and A\. Tewari\(2025\)Conformal Prediction for Ensembles: Improving Efficiency via Score\-Based Aggregation\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=sNTqqdPVSv)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px3.p1.1)\.
- \[26\]V\. Plassier, N\. Kotelevskii, A\. Rubashevskii, F\. Noskov, M\. Velikanov, A\. Fishkov, S\. Horvath, M\. Takác, É\. Moulines, M\. Panov, L\. Mathematics, C\. R\. Center, P\. CMAP, S\. I\. of Science, Technology, M\. HSE University, T\. I\. Institute, and A\. D\. Mohamed bin Zayed University of Artificial Intelligence\(2024\)Efficient Conformal Prediction under Data Heterogeneity\.Technical reportVol\.238\.External Links:[Link](https://proceedings.mlr.press/v238/plassier24a/plassier24a.pdf)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p2.2)\.
- \[27\]V\. Quach, A\. Fisch, T\. Schuster, A\. Yala, J\. H\. Sohn, T\. S\. Jaakkola, and R\. Barzilay\(2023\-06\)Conformal language modeling\.External Links:[Link](https://arxiv.org/abs/2306.10193)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p1.1),[§5](https://arxiv.org/html/2605.23189#S5.p7.4),[§6](https://arxiv.org/html/2605.23189#S6.p1.1)\.
- \[28\]D\. Stutz, A\. Guha Roy, T\. Matejovicova, P\. Strachan, A\. T\. Cemgil, and A\. Doucet\(2023\)Conformal prediction under ambiguous ground truth\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=CAd6V2qXxc)Cited by:[§6](https://arxiv.org/html/2605.23189#S6.SS0.SSS0.Px1.p1.1)\.
- \[29\]V\. Vovk, A\. Gammerman, and G\. Shafer\(2005\)Algorithmic learning in a random world\.Springer\.External Links:[Link](https://link.springer.com/book/10.1007/b98835)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.23189#S2.SS2.p2.14),[§6](https://arxiv.org/html/2605.23189#S6.p1.1)\.
- \[30\]Y\. A\. Yadkori, I\. Kuzborskij, D\. Stutz, A\. György, A\. Fisch, A\. Doucet, I\. Beloshapka, W\. Weng, Y\. Yang, C\. Szepesvári, A\. T\. Cemgil, and N\. Tomasev\(2024\-04\)Mitigating LLM hallucinations via conformal abstention\.External Links:[Link](https://arxiv.org/abs/2405.01563)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p1.1),[§6](https://arxiv.org/html/2605.23189#S6.p1.1)\.
- \[31\]X\. Zhai, A\. Kolesnikov, N\. Houlsby, and L\. Beyer\(2021\-06\)Scaling vision transformers\.External Links:[Link](https://arxiv.org/abs/2106.04560)Cited by:[§1](https://arxiv.org/html/2605.23189#S1.p1.1)\.

## Appendix ACode Availability

We provide the full implementation of therr\-value method and demo for CLIP at:https://github\.com/Yogesh914/conformal\-rvalue

## Appendix BComputation Efficiency Analysis

Table 3:Training time for 1,000 adapters across different models\.ModelTraining timeResNet\-1820 minResNet\-5026 minViT\-B28 minViT\-L1 h 42 minEfficient AnalysisEstimating score variability for therr\-value requires multiple model outputs and therefore introduces additional computation beyond standardCP\{\\rm CP\}\. This overhead has two main components\. The first is the cost of obtaining posterior samples, such as WBB adapters or paraphrased prompts, which determines the precision of the estimated mean, variance, or empirical rank distribution\. The second is the cost of computing therr\-values and conformal sets over the calibration and test samples\.

Cost for posterior samples\.For vision models, the posterior\-sampling cost remains modest because we only train lightweight adapters while keeping the pretrained backbone frozen\. In our experiments, a single A100 40GB GPU can train up to 1,800 adapters across four vision backbones\. For 1,000 adapters, ResNet\-18, ResNet\-50, and ViT\-B each require less than 30 minutes, while ViT\-L requires about 1\.5 hours\. Since adapters are lightweight and can be trained in parallel, this is substantially cheaper than repeatedly retraining the full model\.

Cost for paraphrasingFor the multi prompt paraphrasing approach in LLM experiments, the additional cost comes from two steps\. Ones is generating paraphrased questions, and the other one is scoring candidate answers under each paraphrase\. In our implementation, generating and evaluating 20 paraphrases for one question takes approximately 2\-3 seconds with GPT\-4\.1\. Although this is more expensive than a single forward pass, it is much smaller than the posterior sampling cost used in the vision experiments, where we train hundreds to thousands of WBB adapters\. At the dataset level, the cost can be further reduced through batched inference over paraphrases and candidate answers\. Thus, we view this overhead as a practical deployment consideration rather than a fundamental limitation of the method\. Moreover, in many LLM workflows, repeated prompting, self consistency, or evaluator based scoring is already used\. In such cases, therr\-value can reuse these repeated outputs to estimate score variability instead of requiring a separate source of randomness\.

Cost for applyingrr\-value\.The cost of computing conformal sets after the model outputs are obtained is also moderate\. With 1,000 adapters and 10,000 ImageNet test samples, the parametric Normal\-Normal implementation takes about 2 seconds to compute therr\-values and conformal sets, which is comparable to standard CP\. For the nonparametric estimator, computing the conformal set alone takes about 0\.3 seconds, while computing both the nonparametricrr\-values and the resulting conformal sets takes about 1 minute\. This additional cost comes from estimating rank consistency across posterior samples\. Our current implementation is not optimized, so further speedups are possible through batching, vectorization, and parallel computation\.

Overall, the additional computation is modest for vision models relative to the cost of posterior sample generation and is feasible in the settings we study\. For VLMs and LLMs, multiple inference runs or paraphrased evaluations are increasingly standard in practice, and therr\-value framework can use these existing repeated outputs to obtain uncertainty aware rankings rather than treating them only as averaged predictions\.

## Appendix CProofs

### C\.1Derivation of the Threshold Function under the Normal–Normal Model

We acknowledge the work in\[[11](https://arxiv.org/html/2605.23189#bib.bib5)\], which established that in the continuous model, a necessary condition for the functiontβ∗t\_\{\\beta\}^\{\*\}to be optimal within the class of continuously differentiable threshold functions is as follows\. In the following work, we will derive the threshold function used in the parametricrr\-value construction\. The purpose of this threshold is to decide, for a fixed top fractionβ\\beta, whether a candidate with observed scoreXiX\_\{i\}and uncertaintyσi2\\sigma\_\{i\}^\{2\}should be treated as belonging to the topβ\\betafraction of latent scores\. The key point is that the decision should not depend only on the observed scoreXiX\_\{i\}, but also on how reliable that score is\. A candidate with a large observed score but large variance should require stronger evidence than a candidate with a similar score and smaller variance\.

Let

θi∼𝒩​\(μ,τ2\),Xi∣θi,σi2∼𝒩​\(θi,σi2\)\.\\theta\_\{i\}\\sim\\mathcal\{N\}\(\\mu,\\tau^\{2\}\),\\qquad X\_\{i\}\\mid\\theta\_\{i\},\\sigma\_\{i\}^\{2\}\\sim\\mathcal\{N\}\(\\theta\_\{i\},\\sigma\_\{i\}^\{2\}\)\.Hereθi\\theta\_\{i\}is the latent stable score of candidateii, whileXiX\_\{i\}is the observed score\. The varianceσi2\\sigma\_\{i\}^\{2\}measures how much the observed score fluctuates around the latent score\. For notational simplicity, writes=σi2s=\\sigma\_\{i\}^\{2\}\.

Under the Normal\-Normal model, the posterior distribution ofθi\\theta\_\{i\}given\(Xi,s\)\(X\_\{i\},s\)is

θi∣Xi,s∼𝒩​\{m​\(Xi,s\),v​\(s\)\},\\theta\_\{i\}\\mid X\_\{i\},s\\sim\\mathcal\{N\}\\\{m\(X\_\{i\},s\),v\(s\)\\\},where

m​\(Xi,s\)=τ2​Xi\+s​μτ2\+s,v​\(s\)=τ2​sτ2\+s\.m\(X\_\{i\},s\)=\\frac\{\\tau^\{2\}X\_\{i\}\+s\\mu\}\{\\tau^\{2\}\+s\},\\qquad v\(s\)=\\frac\{\\tau^\{2\}s\}\{\\tau^\{2\}\+s\}\.This posterior distribution is the object used by therr\-value\. Instead of asking whether the observed scoreXiX\_\{i\}is large, we ask whether the latent scoreθi\\theta\_\{i\}is likely to lie in the top group after accounting for uncertainty\.

Letθβ\\theta\_\{\\beta\}denote the\(1−β\)\(1\-\\beta\)\-quantile of the prior distribution ofθi\\theta\_\{i\}, so that

P​\(θi≥θβ\)=β\.P\(\\theta\_\{i\}\\geq\\theta\_\{\\beta\}\)=\\beta\.For a candidate with data\(Xi,s\)\(X\_\{i\},s\), define the posterior tail probability

Vβ​\(Xi,s\)=P​\(θi≥θβ∣Xi,s\)=1−Φ​\(θβ−m​\(Xi,s\)v​\(s\)\)\.V\_\{\\beta\}\(X\_\{i\},s\)=P\(\\theta\_\{i\}\\geq\\theta\_\{\\beta\}\\mid X\_\{i\},s\)=1\-\\Phi\\left\(\\frac\{\\theta\_\{\\beta\}\-m\(X\_\{i\},s\)\}\{\\sqrt\{v\(s\)\}\}\\right\)\.This quantity measures how likely candidateiiis to belong to the topβ\\betafraction of latent scores\.

Following the thresholding principle in\[[11](https://arxiv.org/html/2605.23189#bib.bib5)\], the optimal boundary at levelβ\\betais characterized by a constant posterior tail probability along the boundary\. That is, the thresholdtβ∗​\(s\)t\_\{\\beta\}^\{\*\}\(s\)is defined so that candidates exactly on the boundary satisfy

P\(θi≥θβ∣Xi=tβ∗\(s\),s\)=λβ,P\(\\theta\_\{i\}\\geq\\theta\_\{\\beta\}\\mid X\_\{i\}=t\_\{\\beta\}^\{\*\}\(s\),s\)=\\lambda\_\{\\beta\},whereλβ\\lambda\_\{\\beta\}is a level\-specific constant chosen to satisfy the marginal selection constraint\. Equivalently, if

zβ=Φ−1​\(1−λβ\),z\_\{\\beta\}=\\Phi^\{\-1\}\(1\-\\lambda\_\{\\beta\}\),then the boundary condition becomes

θβ−m​\(tβ∗​\(s\),s\)v​\(s\)=zβ\.\\frac\{\\theta\_\{\\beta\}\-m\(t\_\{\\beta\}^\{\*\}\(s\),s\)\}\{\\sqrt\{v\(s\)\}\}=z\_\{\\beta\}\.Substituting the posterior mean and variance gives

θβ−τ2​tβ∗​\(s\)\+s​μτ2\+s=zβ​τ​sτ2\+s\.\\theta\_\{\\beta\}\-\\frac\{\\tau^\{2\}t\_\{\\beta\}^\{\*\}\(s\)\+s\\mu\}\{\\tau^\{2\}\+s\}=z\_\{\\beta\}\\frac\{\\tau\\sqrt\{s\}\}\{\\sqrt\{\\tau^\{2\}\+s\}\}\.Solving this equation fortβ∗​\(s\)t\_\{\\beta\}^\{\*\}\(s\)yields

tβ∗​\(s\)=θβ​\(1\+sτ2\)−μ​\(sτ2\)−zβ​s​τ2\+sτ\.t\_\{\\beta\}^\{\*\}\(s\)=\\theta\_\{\\beta\}\\left\(1\+\\frac\{s\}\{\\tau^\{2\}\}\\right\)\-\\mu\\left\(\\frac\{s\}\{\\tau^\{2\}\}\\right\)\-z\_\{\\beta\}\\frac\{\\sqrt\{s\}\\sqrt\{\\tau^\{2\}\+s\}\}\{\\tau\}\.Returning tos=σi2s=\\sigma\_\{i\}^\{2\}, we obtain

tβ∗​\(σi2\)=θβ​\(1\+σi2τ2\)−μ​\(σi2τ2\)−zβ​σi​τ2\+σi2τ\.t\_\{\\beta\}^\{\*\}\(\\sigma\_\{i\}^\{2\}\)=\\theta\_\{\\beta\}\\left\(1\+\\frac\{\\sigma\_\{i\}^\{2\}\}\{\\tau^\{2\}\}\\right\)\-\\mu\\left\(\\frac\{\\sigma\_\{i\}^\{2\}\}\{\\tau^\{2\}\}\\right\)\-z\_\{\\beta\}\\frac\{\\sigma\_\{i\}\\sqrt\{\\tau^\{2\}\+\\sigma\_\{i\}^\{2\}\}\}\{\\tau\}\.
It remains to specify howzβz\_\{\\beta\}is determined\. The threshold should select a marginal fractionβ\\betaof candidates\. Since

Xi∣s∼𝒩​\(μ,τ2\+s\),X\_\{i\}\\mid s\\sim\\mathcal\{N\}\(\\mu,\\tau^\{2\}\+s\),the marginal selection constraint is

P​\{Xi≥tβ∗​\(s\)\}=∫\[1−Φ​\(tβ∗​\(s\)−μτ2\+s\)\]​g​\(s\)​𝑑s=β,P\\\{X\_\{i\}\\geq t\_\{\\beta\}^\{\*\}\(s\)\\\}=\\int\\left\[1\-\\Phi\\left\(\\frac\{t\_\{\\beta\}^\{\*\}\(s\)\-\\mu\}\{\\sqrt\{\\tau^\{2\}\+s\}\}\\right\)\\right\]g\(s\)\\,ds=\\beta,whereg​\(s\)g\(s\)is the distribution of the variances=σi2s=\\sigma\_\{i\}^\{2\}\. Substituting the expression fortβ∗​\(s\)t\_\{\\beta\}^\{\*\}\(s\)gives

∫\[1−Φ​\(\(θβ−μ\)​τ2\+sτ2−zβ​sτ\)\]​g​\(s\)​𝑑s=β\.\\int\\left\[1\-\\Phi\\left\(\\frac\{\(\\theta\_\{\\beta\}\-\\mu\)\\sqrt\{\\tau^\{2\}\+s\}\}\{\\tau^\{2\}\}\-z\_\{\\beta\}\\frac\{\\sqrt\{s\}\}\{\\tau\}\\right\)\\right\]g\(s\)\\,ds=\\beta\.This equation determineszβz\_\{\\beta\}, or equivalentlyλβ\\lambda\_\{\\beta\}\.

Therefore, under the Normal\-Normal model, the threshold function is

tβ∗​\(σi2\)=θβ​\(1\+σi2τ2\)−μ​\(σi2τ2\)−zβ​σi​τ2\+σi2τ,t\_\{\\beta\}^\{\*\}\(\\sigma\_\{i\}^\{2\}\)=\\theta\_\{\\beta\}\\left\(1\+\\frac\{\\sigma\_\{i\}^\{2\}\}\{\\tau^\{2\}\}\\right\)\-\\mu\\left\(\\frac\{\\sigma\_\{i\}^\{2\}\}\{\\tau^\{2\}\}\\right\)\-z\_\{\\beta\}\\frac\{\\sigma\_\{i\}\\sqrt\{\\tau^\{2\}\+\\sigma\_\{i\}^\{2\}\}\}\{\\tau\},wherezβz\_\{\\beta\}is chosen so that the threshold selects a marginal fractionβ\\betaof candidates\. This expression shows explicitly how the selection boundary depends on both the observed score and its uncertainty\. In particular, candidates with different variances are compared using different thresholds, which is the mechanism by which therr\-value penalizes unstable high\-score candidates\.

### C\.2Proof of Theorem 1

We break Theorem 1 into the following components with distinct properties and subsidiary theorems, to provide clearer explanations and more detailed proofs\. All the following proofs assume thatθα\>0\\theta\_\{\\alpha\}\>0\.

For a given test instancext​e​s​tx\_\{test\}, letiiindex theii\-th possible label\. Then the prediction of the model for theii\-th label isf​\(xt​e​s​t\)if\(x\_\{test\}\)\_\{i\}\(denotedf​\(x\)if\(x\)\_\{i\}\), which is a realization of the normal distributionN​\(θi,σi2\)N\(\\theta\_\{i\},\\sigma\_\{i\}^\{2\}\), where, WLOG,θi\\theta\_\{i\}is the true unobserved quality parameter for labeliiand is drawn from a prior distributionN​\(0,1\)N\(0,1\), andσi2\\sigma\_\{i\}^\{2\}is the variance of the prediction, which is also drawn from a prior distributiong​\(σ2\)g\(\\sigma^\{2\}\)\. \(Note thatg​\(σ2\)g\(\\sigma^\{2\}\)is not a point mass atσ2=0\\sigma^\{2\}=0i\.e\.,P​\(σ2\>0\)\>0P\(\\sigma^\{2\}\>0\)\>0\.\) In general,θi∼N​\(μ,τ\)\\theta\_\{i\}\\sim N\(\\mu,\\tau\)but we can standardize it to a standard normal distribution\.

Recall that both methods aim to construct a conformal set that provides a statistical guarantee\.

LetCr=\{i:ri≤r∗\}≡\{i:f​\(x\)i≥tr∗​\(σi2\)\}C\_\{r\}=\\left\\\{i:r\_\{i\}\\leq r^\{\*\}\\right\\\}\\equiv\\left\\\{i:f\(x\)\_\{i\}\\geq t\_\{r^\{\*\}\}\(\\sigma\_\{i\}^\{2\}\)\\right\\\}denote the conformal set underrr\-value, where nonconformity score forii\-th class is defined asri=inf\{α:f​\(x\)i≥tα​\(σi2\)\}r\_\{i\}=\\inf\\\{\\alpha:f\(x\)\_\{i\}\\geq t\_\{\\alpha\}\(\\sigma\_\{i\}^\{2\}\)\\\}andr∗r^\{\*\}is the⌈\(n\+1\)​\(1−α\)⌉n\\frac\{\\lceil\(n\+1\)\(1\-\\alpha\)\\rceil\}\{n\}quantile determined by calibration data and independent ofxt​e​s​tx\_\{test\}\. Heretα​\(σi2\)=θα​\(σ2\+1\)−zα​σ4\+σ2t\_\{\\alpha\}\(\\sigma\_\{i\}^\{2\}\)=\\theta\_\{\\alpha\}\\,\(\\sigma^\{2\}\+1\)\-z\_\{\\alpha\}\\,\\sqrt\{\\sigma^\{4\}\+\\sigma^\{2\}\}is a threshold function whereθα\\theta\_\{\\alpha\}is the1−α1\-\\alpha\- quantile ofN​\(0,1\)N\(0,1\)andzαz\_\{\\alpha\}is anα\\alphadependent constant that ensureEσs2∼g​\[Φ​\(θα​σs2\+1−zα​σs\)\]=1−αE\_\{\\sigma\_\{s\}^\{2\}\\sim g\}\\left\[\\Phi\\left\(\\theta\_\{\\alpha\}\\sqrt\{\\sigma\_\{s\}^\{2\}\+1\}\-z\_\{\\alpha\}\\sigma\_\{s\}\\right\)\\right\]=1\-\\alpha\. Similarly, letCs​t​d=\{i:f​\(x\)i≥Ts​t​d\}C\_\{std\}=\\left\\\{i:f\(x\)\_\{i\}\\geq T\_\{std\}\\right\\\}denote the conformal set under standard CP, where the non\-conformity score forii\-th class is usually defined as−f​\(x\)i\-f\(x\)\_\{i\}\(smaller is better\)\. And again,Ts​t​dT\_\{std\}is the⌈\(n\+1\)​\(1−α\)⌉n\\frac\{\\lceil\(n\+1\)\(1\-\\alpha\)\\rceil\}\{n\}quantile determined by calibration data and independent ofxt​e​s​tx\_\{test\}\.

###### Proposition C\.1\(Monotonicity oftαt\_\{\\alpha\}\)\.

For a fixedα\\alpha, let

tα​\(σ2\)=θα​\(σ2\+1\)−zα​σ4\+σ2,t\_\{\\alpha\}\(\\sigma^\{2\}\)\\;=\\;\\theta\_\{\\alpha\}\\,\(\\sigma^\{2\}\+1\)\\;\-\\;z\_\{\\alpha\}\\,\\sqrt\{\\sigma^\{4\}\+\\sigma^\{2\}\},and define

s⋆=12​\(θαθα2−zα2−1\),s0=zα2θα2−zα2\(s0\>s⋆\)\.s\_\{\\star\}\\;=\\;\\frac\{1\}\{2\}\\left\(\\frac\{\\theta\_\{\\alpha\}\}\{\\sqrt\{\\theta\_\{\\alpha\}^\{2\}\-z\_\{\\alpha\}^\{2\}\}\}\-1\\right\),\\qquad s\_\{0\}\\;=\\;\\frac\{z\_\{\\alpha\}^\{2\}\}\{\\theta\_\{\\alpha\}^\{2\}\-z\_\{\\alpha\}^\{2\}\}\\quad\\text\{\($s\_\{0\}\>s\_\{\\star\}$\)\}\.
Then, if0<zα<θα0<z\_\{\\alpha\}<\\theta\_\{\\alpha\}:

1. 1\.tαt\_\{\\alpha\}is strictly decreasing on\[0,s⋆\]\[0,s\_\{\\star\}\]and strictly increasing on\[s⋆,∞\)\[s\_\{\\star\},\\infty\)with respect toσ2\\sigma^\{2\}\.
2. 2\.tα​\(0\)=tα​\(s0\)=θαt\_\{\\alpha\}\(0\)=t\_\{\\alpha\}\(s\_\{0\}\)=\\theta\_\{\\alpha\}\.

Ifzα≤0<θαz\_\{\\alpha\}\\leq 0<\\theta\_\{\\alpha\}:

1. 1\.tαt\_\{\\alpha\}is strictly increasing on\[0,∞\)\[0,\\infty\)\.

###### Proof\.

\(0\)zαz\_\{\\alpha\}is strictly less thanθα\\theta\_\{\\alpha\}

LetF​\(u\)=𝔼σ2∼g​\[Φ​\(θα​σ2\+1−u​σ\)\]F\(u\)=\\mathbb\{E\}\_\{\\sigma^\{2\}\\sim g\}\\left\[\\Phi\\left\(\\theta\_\{\\alpha\}\\sqrt\{\\sigma^\{2\}\+1\}\-u\\sigma\\right\)\\right\]foru∈ℝu\\in\\mathbb\{R\}\. Then, by definition,zαz\_\{\\alpha\}must satisfy the constraint such thatF​\(zα\)=1−αF\(z\_\{\\alpha\}\)=1\-\\alpha\.

By the Dominated Convergence Theorem,

d​F​\(u\)d​u\\displaystyle\\frac\{dF\(u\)\}\{du\}=dd​u​𝔼σ2∼g​\[Φ​\(θα​σ2\+1−u​σ\)\]\\displaystyle=\\frac\{d\}\{du\}\\mathbb\{E\}\_\{\\sigma^\{2\}\\sim g\}\\left\[\\Phi\\left\(\\theta\_\{\\alpha\}\\sqrt\{\\sigma^\{2\}\+1\}\-u\\sigma\\right\)\\right\]=𝔼σ2∼g​\[dd​u​Φ​\(θα​σ2\+1−u​σ\)\]\\displaystyle=\\mathbb\{E\}\_\{\\sigma^\{2\}\\sim g\}\\left\[\\frac\{d\}\{du\}\\Phi\\left\(\\theta\_\{\\alpha\}\\sqrt\{\\sigma^\{2\}\+1\}\-u\\sigma\\right\)\\right\]=𝔼σ2∼g​\[ϕ​\(θα​σ2\+1−u​σ\)⋅\(−σ\)\]<0,\\displaystyle=\\mathbb\{E\}\_\{\\sigma^\{2\}\\sim g\}\\left\[\\phi\\left\(\\theta\_\{\\alpha\}\\sqrt\{\\sigma^\{2\}\+1\}\-u\\sigma\\right\)\\cdot\(\-\\sigma\)\\right\]<0,whereϕ​\(⋅\)\\phi\(\\cdot\)is the standard normal PDF\.

Sinced​F​\(u\)d​u<0\\frac\{dF\(u\)\}\{du\}<0for alluu, it follows thatF​\(u\)F\(u\)is strictly decreasing inuu\. BecauseΦ​\(θα\)=1−α\\Phi\(\\theta\_\{\\alpha\}\)=1\-\\alpha, we find that

F​\(θα\)\\displaystyle F\(\\theta\_\{\\alpha\}\)=𝔼σ2∼g​\[Φ​\(θα​σ2\+1−θα​σ\)\]\\displaystyle=\\mathbb\{E\}\_\{\\sigma^\{2\}\\sim g\}\\left\[\\Phi\\left\(\\theta\_\{\\alpha\}\\sqrt\{\\sigma^\{2\}\+1\}\-\\theta\_\{\\alpha\}\\sigma\\right\)\\right\]=𝔼σ2∼g​\[Φ​\(θα⋅1σ2\+1\+σ\)\]\\displaystyle=\\mathbb\{E\}\_\{\\sigma^\{2\}\\sim g\}\\left\[\\Phi\\left\(\\theta\_\{\\alpha\}\\cdot\\frac\{1\}\{\\sqrt\{\\sigma^\{2\}\+1\}\+\\sigma\}\\right\)\\right\]<𝔼σ2∼g​\[Φ​\(θα\)\]=1−α=F​\(zα\),\\displaystyle<\\mathbb\{E\}\_\{\\sigma^\{2\}\\sim g\}\\left\[\\Phi\\left\(\\theta\_\{\\alpha\}\\right\)\\right\]=1\-\\alpha=F\(z\_\{\\alpha\}\),which implies thatθα\>zα\\theta\_\{\\alpha\}\>z\_\{\\alpha\}due to the monotonicity ofF​\(⋅\)F\(\\cdot\)\.

\(1\) Monotonicity

For notational simplicity, letx=σ2x=\\sigma^\{2\}wherex≥0x\\geq 0\. Then

tα​\(x\)=θα​\(x\+1\)−zα​x2\+x,tα′​\(x\)=θα−zα⋅2​x\+12​x2\+x\.t\_\{\\alpha\}\(x\)=\\theta\_\{\\alpha\}\(x\+1\)\-z\_\{\\alpha\}\\sqrt\{x^\{2\}\+x\},\\quad t\_\{\\alpha\}^\{\\prime\}\(x\)=\\theta\_\{\\alpha\}\-z\_\{\\alpha\}\\cdot\\frac\{2x\+1\}\{2\\sqrt\{x^\{2\}\+x\}\}\.
To study the monotonicity oftα​\(x\)t\_\{\\alpha\}\(x\), we analyze its derivative\. Ifzα≤0z\_\{\\alpha\}\\leq 0, thentα′​\(x\)\>0t\_\{\\alpha\}^\{\\prime\}\(x\)\>0for allx≥0x\\geq 0, sinceθα\>0\\theta\_\{\\alpha\}\>0and the second term is non\-positive\. This implies thattα​\(x\)t\_\{\\alpha\}\(x\)is strictly increasing on\[0,∞\)\[0,\\infty\), and no critical point exists\.

Ifθα\>zα\>0\\theta\_\{\\alpha\}\>z\_\{\\alpha\}\>0, settα′​\(x\)=0t\_\{\\alpha\}^\{\\prime\}\(x\)=0\. Then the critical point is:

x=s⋆=12​\(θαθα2−zα2−1\)\.x=s\_\{\\star\}=\\frac\{1\}\{2\}\\left\(\\frac\{\\theta\_\{\\alpha\}\}\{\\sqrt\{\\theta\_\{\\alpha\}^\{2\}\-z\_\{\\alpha\}^\{2\}\}\}\-1\\right\)\.
Sincezα<θαz\_\{\\alpha\}<\\theta\_\{\\alpha\}, it follows that\(4​θα2−4​zα2\)\>0\(4\\theta\_\{\\alpha\}^\{2\}\-4z\_\{\\alpha\}^\{2\}\)\>0\. The derivativetα′​\(x\)t\_\{\\alpha\}^\{\\prime\}\(x\)changes from negative to positive atx=s⋆x=s\_\{\\star\}, indicating thattαt\_\{\\alpha\}achieves a minimum atx=s⋆x=s\_\{\\star\}\. Hence,tαt\_\{\\alpha\}is strictly decreasing on\[0,s⋆\]\[0,s\_\{\\star\}\]and strictly increasing on\[s⋆,∞\)\[s\_\{\\star\},\\infty\)\.

\(2\) Equality at endpoints

We have shown that ifθα\>zα\>0\\theta\_\{\\alpha\}\>z\_\{\\alpha\}\>0, thentα​\(⋅\)t\_\{\\alpha\}\(\\cdot\)is strictly decreasing on\[0,s⋆\]\[0,s\_\{\\star\}\]and strictly increasing on\[s⋆,∞\)\[s\_\{\\star\},\\infty\)\. We now show thattα​\(s0\)=tα​\(0\)t\_\{\\alpha\}\(s\_\{0\}\)=t\_\{\\alpha\}\(0\)\.

Since

tα​\(s0\)\\displaystyle t\_\{\\alpha\}\(s\_\{0\}\)=θα​\(s0\+1\)−zα​s02\+s0\\displaystyle=\\theta\_\{\\alpha\}\(s\_\{0\}\+1\)\-z\_\{\\alpha\}\\sqrt\{s\_\{0\}^\{2\}\+s\_\{0\}\}=θα⋅θα2θα2−zα2−zα⋅zα​θαθα2−zα2\\displaystyle=\\theta\_\{\\alpha\}\\cdot\\frac\{\\theta\_\{\\alpha\}^\{2\}\}\{\\theta\_\{\\alpha\}^\{2\}\-z\_\{\\alpha\}^\{2\}\}\-z\_\{\\alpha\}\\cdot\\frac\{z\_\{\\alpha\}\\theta\_\{\\alpha\}\}\{\\theta\_\{\\alpha\}^\{2\}\-z\_\{\\alpha\}^\{2\}\}=θα3−θα​zα2θα2−zα2=θα,\\displaystyle=\\frac\{\\theta\_\{\\alpha\}^\{3\}\-\\theta\_\{\\alpha\}z\_\{\\alpha\}^\{2\}\}\{\\theta\_\{\\alpha\}^\{2\}\-z\_\{\\alpha\}^\{2\}\}=\\theta\_\{\\alpha\},and

tα​\(0\)=θα​\(0\+1\)−zα⋅0\+0=θα,t\_\{\\alpha\}\(0\)=\\theta\_\{\\alpha\}\(0\+1\)\-z\_\{\\alpha\}\\cdot\\sqrt\{0\+0\}=\\theta\_\{\\alpha\},we conclude thattα​\(s0\)=tα​\(0\)t\_\{\\alpha\}\(s\_\{0\}\)=t\_\{\\alpha\}\(0\)\.

∎

###### Proposition C\.2\(Monotone tail\)\.

Define the probability thatf​\(x\)if\(x\)\_\{i\}surpasses the threshold functiontα​\(s\)t\_\{\\alpha\}\(s\)for any variances=σi2≥0s=\\sigma\_\{i\}^\{2\}\\geq 0at a fixed levelα\\alphaas

pi​\(s\):=P​\{f​\(x\)i≥tα​\(s\)\},for​i∈\{1,…,K\}\.p\_\{i\}\(s\):=P\\bigl\\\{f\(x\)\_\{i\}\\geq t\_\{\\alpha\}\(s\)\\bigr\\\},\\quad\\text\{for \}i\\in\\\{1,\\dots,K\\\}\.Undertα​\(s\)t\_\{\\alpha\}\(s\), the mappings↦pi​\(s\)s\\mapsto p\_\{i\}\(s\)is non\-increasing on\[sconj,∞\)\[s\_\{\\text\{conj\}\},\\infty\), where

sconj:=\{zα2θα2−zα2if​0<zα<θα,0if​zα≤0<θα\.s\_\{\\text\{conj\}\}:=\\begin\{cases\}\\dfrac\{z\_\{\\alpha\}^\{2\}\}\{\\theta\_\{\\alpha\}^\{2\}\-z\_\{\\alpha\}^\{2\}\}&\\text\{if \}0<z\_\{\\alpha\}<\\theta\_\{\\alpha\},\\\\\[6\.0pt\] 0&\\text\{if \}z\_\{\\alpha\}\\leq 0<\\theta\_\{\\alpha\}\.\\end\{cases\}Hence,

pi​\(s\)≤pi​\(0\)for all​s≥sconj\.p\_\{i\}\(s\)\\leq p\_\{i\}\(0\)\\quad\\text\{for all \}s\\geq s\_\{\\text\{conj\}\}\.

###### Proof\.

The result inherit from proposition 1\. ∎

###### Proposition C\.3\.

\(Average\-then\-CP is the special case with variance 0\)

Letf¯​\(x\)i=1M​∑m=1Mf​\(x\)i\(m\)\\,\\bar\{f\}\(x\)\_\{i\}=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}f\(x\)\_\{i\}^\{\(m\)\}\\,be the ensemble average of*i\.i\.d\.*model predictionsf​\(x\)i\(m\)∼𝒩​\(θi,σi2\)f\(x\)\_\{i\}^\{\(m\)\}\\sim\\mathcal\{N\}\(\\theta\_\{i\},\\sigma\_\{i\}^\{2\}\)fori∈\{1,…,K\}i\\in\\\{1,\\dots,K\\\}andm∈\{1,…,M\}m\\in\\\{1,\\dots,M\\\}\.

Define the two conformal prediction sets:

Cavg:=\{i:f¯​\(x\)i≥tα​\(σi2M\)\},Cr\(0\):=\{i:f​\(x\)i≥tα​\(0\)\}\.C\_\{\\mathrm\{avg\}\}:=\\left\\\{i:\\bar\{f\}\(x\)\_\{i\}\\geq t\_\{\\alpha\}\\left\(\\tfrac\{\\sigma\_\{i\}^\{2\}\}\{M\}\\right\)\\right\\\},\\quad C\_\{r\}^\{\(0\)\}:=\\left\\\{i:f\(x\)\_\{i\}\\geq t\_\{\\alpha\}\\left\(0\\right\)\\right\\\}\.Then,

Cavg=Cr\(0\)almost surely\.C\_\{\\mathrm\{avg\}\}=C\_\{r\}^\{\(0\)\}\\quad\\text\{almost surely\.\}

###### Proof\.

We want to show thatCavg=Cr\(0\)C\_\{\\mathrm\{avg\}\}=C\_\{r\}^\{\(0\)\}almost surely asM→∞M\\rightarrow\\infty\.

Note that

f​\(x\)i\(m\)∼i\.i\.d\.𝒩​\(θi,σi2\),andf¯​\(x\)i=1M​∑m=1Mf​\(x\)i\(m\)\.f\(x\)\_\{i\}^\{\(m\)\}\\stackrel\{\{\\scriptstyle\\text\{i\.i\.d\.\}\}\}\{\{\\sim\}\}\\mathcal\{N\}\(\\theta\_\{i\},\\sigma\_\{i\}^\{2\}\),\\quad\\text\{and\}\\quad\\bar\{f\}\(x\)\_\{i\}=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}f\(x\)\_\{i\}^\{\(m\)\}\.Then, by properties of the normal distribution,

f¯​\(x\)i∼𝒩​\(θi,σi2M\),whereVar​\(f¯​\(x\)i\)=σi2M→M→∞0\.\\bar\{f\}\(x\)\_\{i\}\\sim\\mathcal\{N\}\\left\(\\theta\_\{i\},\\frac\{\\sigma\_\{i\}^\{2\}\}\{M\}\\right\),\\quad\\text\{where\}\\quad\\mathrm\{Var\}\(\\bar\{f\}\(x\)\_\{i\}\)=\\frac\{\\sigma\_\{i\}^\{2\}\}\{M\}\\xrightarrow\[M\\to\\infty\]\{\}0\.
By the strong law of large numbers, we havef¯​\(x\)i→a\.s\.θi\\bar\{f\}\(x\)\_\{i\}\\xrightarrow\{\\text\{a\.s\.\}\}\\theta\_\{i\}\. In particular, whenσi2=0\\sigma\_\{i\}^\{2\}=0, thenf​\(x\)i=θif\(x\)\_\{i\}=\\theta\_\{i\}deterministically, and sincetα​\(0\)=θαt\_\{\\alpha\}\(0\)=\\theta\_\{\\alpha\}, we conclude that for sufficiently largeMM,

f¯​\(x\)i≥tα​\(σi2M\)⟺θi≥θαalmost surely\.\\bar\{f\}\(x\)\_\{i\}\\geq t\_\{\\alpha\}\\left\(\\frac\{\\sigma\_\{i\}^\{2\}\}\{M\}\\right\)\\;\\Longleftrightarrow\\;\\theta\_\{i\}\\geq\\theta\_\{\\alpha\}\\quad\\text\{almost surely\.\}
This implies that the setsCavgC\_\{\\mathrm\{avg\}\}andCr\(0\)C\_\{r\}^\{\(0\)\}are equivalent almost surely\. ∎

###### Theorem C\.4\(R\-value selection\)\.

Assume thattα​\(σi2\)=θα​\(σi2\+1\)−zα​σi4\+σi2t\_\{\\alpha\}\(\\sigma\_\{i\}^\{2\}\)=\\theta\_\{\\alpha\}\\,\(\\sigma\_\{i\}^\{2\}\+1\)\-z\_\{\\alpha\}\\,\\sqrt\{\\sigma\_\{i\}^\{4\}\+\\sigma\_\{i\}^\{2\}\}andσi2≥sconj\\sigma\_\{i\}^\{2\}\\geq s\_\{\\text\{conj\}\}for all classesii\.

Letxnewx\_\{\\text\{new\}\}be a new test input with unknown true labelytruey\_\{\\text\{true\}\}\. LetCr​\(xnew\)C\_\{r\}\(x\_\{\\text\{new\}\}\)denote the conformal set selected by therr\-value method, and letCavg​\(xnew\)C\_\{\\text\{avg\}\}\(x\_\{\\text\{new\}\}\)denote the conformal set selected by the ’average\-then\-CP’ method\. When the number of models is sufficiently large, then:

1. 1\.Coverage: P​\{ytrue∉Cr​\(xnew\)\}≤α,andP​\{ytrue∉Cavg​\(xnew\)\}≤α\.P\\left\\\{y\_\{\\text\{true\}\}\\notin C\_\{r\}\(x\_\{\\text\{new\}\}\)\\right\\\}\\leq\\alpha,\\quad\\text\{and\}\\quad P\\left\\\{y\_\{\\text\{true\}\}\\notin C\_\{\\text\{avg\}\}\(x\_\{\\text\{new\}\}\)\\right\\\}\\leq\\alpha\.
2. 2\.Set size: 𝔼​\[\|Cr\|\]≤𝔼​\[\|Cavg\|\],\\mathbb\{E\}\\left\[\\,\|C\_\{r\}\|\\,\\right\]\\;\\leq\\;\\mathbb\{E\}\\left\[\\,\|C\_\{\\text\{avg\}\}\|\\,\\right\],whereCavgC\_\{\\text\{avg\}\}is defined by settingσi2→0\\sigma\_\{i\}^\{2\}\\to 0\(’average\-then\-CP’\)\.

###### Proof\.

1\. The proof follows from the theory of conformal prediction\.

2\. For any selected conformal setCC, the expected size of the set can be decomposed as:

𝔼​\[\|C\|\]\\displaystyle\\mathbb\{E\}\[\|C\|\]=𝔼X​\[𝟏​\(ytrue∈C\)\+∑y′≠ytrue𝟏​\(y′∈C\)\]\\displaystyle=\\mathbb\{E\}\_\{X\}\\left\[\\mathbf\{1\}\\left\(y\_\{\\text\{true\}\}\\in C\\right\)\+\\sum\_\{y^\{\\prime\}\\neq y\_\{\\text\{true\}\}\}\\mathbf\{1\}\\left\(y^\{\\prime\}\\in C\\right\)\\right\]=Pr⁡\(ytrue∈C\)\+∑y′≠ytruePr⁡\(y′∈C\),\\displaystyle=\\Pr\\left\(y\_\{\\text\{true\}\}\\in C\\right\)\+\\sum\_\{y^\{\\prime\}\\neq y\_\{\\text\{true\}\}\}\\Pr\\left\(y^\{\\prime\}\\in C\\right\),where𝟏​\(⋅\)\\mathbf\{1\}\(\\cdot\)is the indicator function andytruey\_\{\\text\{true\}\}is the true label\.

We want to show that

𝔼​\[\|Cr\|\]−𝔼​\[\|Cavg\|\]=\(P​\(ytrue∈Cr\)−P​\(ytrue∈Cavg\)\)\+∑y′≠ytrue\(P​\(y′∈Cr\)−P​\(y′∈Cavg\)\)\\mathbb\{E\}\[\|C\_\{r\}\|\]\-\\mathbb\{E\}\[\|C\_\{\\text\{avg\}\}\|\]=\\left\(P\\left\(y\_\{\\text\{true\}\}\\in C\_\{r\}\\right\)\-P\\left\(y\_\{\\text\{true\}\}\\in C\_\{\\text\{avg\}\}\\right\)\\right\)\+\\sum\_\{y^\{\\prime\}\\neq y\_\{\\text\{true\}\}\}\\left\(P\\left\(y^\{\\prime\}\\in C\_\{r\}\\right\)\-P\\left\(y^\{\\prime\}\\in C\_\{\\text\{avg\}\}\\right\)\\right\)is non\-positive\.

From part \(1\), bothCrC\_\{r\}andCavgC\_\{\\text\{avg\}\}satisfy marginal coverage guarantees:

P​\(ytrue∈Cr\)≈P​\(ytrue∈Cavg\)\.P\\left\(y\_\{\\text\{true\}\}\\in C\_\{r\}\\right\)\\approx P\\left\(y\_\{\\text\{true\}\}\\in C\_\{\\text\{avg\}\}\\right\)\.Hence, it suffices to show that

∑y′≠ytrue\(P​\(y′∈Cr\)−P​\(y′∈Cavg\)\)≤0\.\\sum\_\{y^\{\\prime\}\\neq y\_\{\\text\{true\}\}\}\\left\(P\\left\(y^\{\\prime\}\\in C\_\{r\}\\right\)\-P\\left\(y^\{\\prime\}\\in C\_\{\\text\{avg\}\}\\right\)\\right\)\\leq 0\.
By assumption,σi2≥sconj\\sigma\_\{i\}^\{2\}\\geq s\_\{\\text\{conj\}\}, which satisfies the condition of Proposition 2\. Then for anyy′y^\{\\prime\}, we have:

P​\(y′∈Cr\)−P​\(y′∈Cavg\)​=Prop 3​P​\{f​\(x\)y′≥tr∗​\(σy′2\)\}−P​\{f​\(x\)y′≥tr∗​\(0\)\}​≤Prop 2​0,P\\left\(y^\{\\prime\}\\in C\_\{r\}\\right\)\-P\\left\(y^\{\\prime\}\\in C\_\{\\text\{avg\}\}\\right\)\\overset\{\\text\{Prop 3\}\}\{=\}P\\left\\\{f\(x\)\_\{y^\{\\prime\}\}\\geq t\_\{r^\{\*\}\}\(\\sigma\_\{y^\{\\prime\}\}^\{2\}\)\\right\\\}\-P\\left\\\{f\(x\)\_\{y^\{\\prime\}\}\\geq t\_\{r^\{\*\}\}\(0\)\\right\\\}\\overset\{\\text\{Prop 2\}\}\{\\leq\}0,wherer∗r^\{\*\}is therr\-value threshold determined by the calibration data\. Thus, the total difference is non\-positive, and the result is proved\. ∎

###### Theorem C\.5\(Asymptotic rejection of high variance\)\.

Assume the data\-generating mechanism and notation introduced in the setup, and suppose0<θα<∞0<\\theta\_\{\\alpha\}<\\infty\.

Lety′y^\{\\prime\}be a non\-true label whose latent quality is fixed atθy′=μ0\\theta\_\{y^\{\\prime\}\}=\\mu\_\{0\}, whereμ0\\mu\_\{0\}is sampled fromN​\(0,1\)N\(0,1\), and letσy′2=σ2\>0\\sigma\_\{y^\{\\prime\}\}^\{2\}=\\sigma^\{2\}\>0\.

For anyσ2\\sigma^\{2\}, define the conditional inclusion probabilities:

Pinclstd​\(σ2∣μ0\)\\displaystyle P\_\{\\mathrm\{incl\}\}^\{\\mathrm\{std\}\}\(\\sigma^\{2\}\\mid\\mu\_\{0\}\)=P\(y′∈Cstd\|θy′=μ0,σy′2=σ2,Tstd\),\\displaystyle=P\\left\(y^\{\\prime\}\\in C\_\{\\mathrm\{std\}\}\\,\\middle\|\\,\\theta\_\{y^\{\\prime\}\}=\\mu\_\{0\},\\;\\sigma\_\{y^\{\\prime\}\}^\{2\}=\\sigma^\{2\},\\;T\_\{\\mathrm\{std\}\}\\right\),Pinclr​\(σ2∣μ0\)\\displaystyle P\_\{\\mathrm\{incl\}\}^\{r\}\(\\sigma^\{2\}\\mid\\mu\_\{0\}\)=P\(y′∈Cr\|θy′=μ0,σy′2=σ2,r∗\),\\displaystyle=P\\left\(y^\{\\prime\}\\in C\_\{r\}\\,\\middle\|\\,\\theta\_\{y^\{\\prime\}\}=\\mu\_\{0\},\\;\\sigma\_\{y^\{\\prime\}\}^\{2\}=\\sigma^\{2\},\\;r^\{\*\}\\right\),whereTstdT\_\{\\mathrm\{std\}\}andr∗r^\{\*\}are thresholds computed from calibration data independent ofxtestx\_\{\\text\{test\}\}, and are treated as fixed constants\.

1. 1\.Standard CP: limσ→∞Pinclstd​\(σ2∣μ0\)=12\.\\lim\_\{\\sigma\\rightarrow\\infty\}P\_\{\\mathrm\{incl\}\}^\{\\mathrm\{std\}\}\(\\sigma^\{2\}\\mid\\mu\_\{0\}\)=\\frac\{1\}\{2\}\.
2. 2\.rr\-value CP: limσ→∞Pinclr​\(σ2∣μ0\)=0\.\\lim\_\{\\sigma\\rightarrow\\infty\}P\_\{\\mathrm\{incl\}\}^\{r\}\(\\sigma^\{2\}\\mid\\mu\_\{0\}\)=0\.

###### Proof\.

For any labely′y^\{\\prime\}such thatf​\(x\)y′∼𝒩​\(μ0,σ2\)f\(x\)\_\{y^\{\\prime\}\}\\sim\\mathcal\{N\}\(\\mu\_\{0\},\\sigma^\{2\}\), by definition, the probability that the false labely′y^\{\\prime\}is included inCstdC\_\{\\text\{std\}\}is:

Pinclstd​\(σ2∣μ0\)\\displaystyle P\_\{\\text\{incl\}\}^\{\\text\{std\}\}\\left\(\\sigma^\{2\}\\mid\\mu\_\{0\}\\right\)=P\(y′∈Cstd\|θy′=μ0,σy′2=σ2,Tstd\)\\displaystyle=P\\left\(y^\{\\prime\}\\in C\_\{\\text\{std\}\}\\,\\middle\|\\,\\theta\_\{y^\{\\prime\}\}=\\mu\_\{0\},\\,\\sigma\_\{y^\{\\prime\}\}^\{2\}=\\sigma^\{2\},\\,T\_\{\\text\{std\}\}\\right\)=P​\(f​\(x\)y′\>Tstd∣μ0,σ2,Tstd\)=1−Φ​\(Tstd−μ0σ\)\.\\displaystyle=P\\left\(f\(x\)\_\{y^\{\\prime\}\}\>T\_\{\\text\{std\}\}\\mid\\mu\_\{0\},\\sigma^\{2\},T\_\{\\text\{std\}\}\\right\)=1\-\\Phi\\left\(\\frac\{T\_\{\\text\{std\}\}\-\\mu\_\{0\}\}\{\\sigma\}\\right\)\.
Similarly, the probability thaty′y^\{\\prime\}is included inCrC\_\{r\}is:

Pinclr​\(σ2∣μ0\)\\displaystyle P\_\{\\text\{incl\}\}^\{r\}\\left\(\\sigma^\{2\}\\mid\\mu\_\{0\}\\right\)=P\(y′∈Cr\|θy′=μ0,σy′2=σ2,r∗\)\\displaystyle=P\\left\(y^\{\\prime\}\\in C\_\{r\}\\,\\middle\|\\,\\theta\_\{y^\{\\prime\}\}=\\mu\_\{0\},\\,\\sigma\_\{y^\{\\prime\}\}^\{2\}=\\sigma^\{2\},\\,r^\{\*\}\\right\)=P​\(f​\(x\)y′−μ0σ≥tr∗​\(σ2\)−μ0σ\)=1−Φ​\(tr∗​\(σ2\)−μ0σ\)\.\\displaystyle=P\\left\(\\frac\{f\(x\)\_\{y^\{\\prime\}\}\-\\mu\_\{0\}\}\{\\sigma\}\\geq\\frac\{t\_\{r^\{\*\}\}\(\\sigma^\{2\}\)\-\\mu\_\{0\}\}\{\\sigma\}\\right\)=1\-\\Phi\\left\(\\frac\{t\_\{r^\{\*\}\}\(\\sigma^\{2\}\)\-\\mu\_\{0\}\}\{\\sigma\}\\right\)\.
Define

Astd​\(σ2\):=Tstd−μ0σ,Ar​\(σ2\):=tr∗​\(σ2\)−μ0σ\.A\_\{\\text\{std\}\}\(\\sigma^\{2\}\):=\\frac\{T\_\{\\text\{std\}\}\-\\mu\_\{0\}\}\{\\sigma\},\\quad A\_\{r\}\(\\sigma^\{2\}\):=\\frac\{t\_\{r^\{\*\}\}\(\\sigma^\{2\}\)\-\\mu\_\{0\}\}\{\\sigma\}\.Then the inclusion probabilities are functions ofσ\\sigmaand become:

Pinclstd​\(σ2∣μ0\)\\displaystyle P\_\{\\text\{incl\}\}^\{\\text\{std\}\}\(\\sigma^\{2\}\\mid\\mu\_\{0\}\)=1−Φ​\(Astd​\(σ2\)\),\\displaystyle=1\-\\Phi\\left\(A\_\{\\text\{std\}\}\(\\sigma^\{2\}\)\\right\),\(3\)Pinclr​\(σ2∣μ0\)\\displaystyle P\_\{\\text\{incl\}\}^\{r\}\(\\sigma^\{2\}\\mid\\mu\_\{0\}\)=1−Φ​\(Ar​\(σ2\)\)\.\\displaystyle=1\-\\Phi\\left\(A\_\{r\}\(\\sigma^\{2\}\)\\right\)\.\(4\)
Note thatTstdT\_\{\\text\{std\}\}andμ0\\mu\_\{0\}are constants\. However, the behavior oftr∗​\(σ2\)t\_\{r^\{\*\}\}\(\\sigma^\{2\}\)as a function ofσ\\sigmais unclear\. Recall:

tr∗​\(σ2\)=θr∗​\(σ2\+1\)−zr∗​σ4\+σ2\.t\_\{r^\{\*\}\}\(\\sigma^\{2\}\)=\\theta\_\{r^\{\*\}\}\(\\sigma^\{2\}\+1\)\-z\_\{r^\{\*\}\}\\sqrt\{\\sigma^\{4\}\+\\sigma^\{2\}\}\.By Taylor expansion,1\+1σ2=1\+12​σ2\+o​\(1σ2\)\\sqrt\{1\+\\frac\{1\}\{\\sigma^\{2\}\}\}=1\+\\frac\{1\}\{2\\sigma^\{2\}\}\+o\\left\(\\frac\{1\}\{\\sigma^\{2\}\}\\right\), then we get:

Ar​\(σ2\)\\displaystyle A\_\{r\}\(\\sigma^\{2\}\)=θr∗​\(σ2\+1\)−zr∗​σ4\+σ2−μ0σ\\displaystyle=\\frac\{\\theta\_\{r^\{\*\}\}\(\\sigma^\{2\}\+1\)\-z\_\{r^\{\*\}\}\\sqrt\{\\sigma^\{4\}\+\\sigma^\{2\}\}\-\\mu\_\{0\}\}\{\\sigma\}=θr∗​\(σ\+1σ\)−zr∗​σ​\(1\+12​σ2\+o​\(1σ2\)\)−μ0σ\\displaystyle=\\theta\_\{r^\{\*\}\}\\left\(\\sigma\+\\frac\{1\}\{\\sigma\}\\right\)\-z\_\{r^\{\*\}\}\\sigma\\left\(1\+\\frac\{1\}\{2\\sigma^\{2\}\}\+o\\left\(\\frac\{1\}\{\\sigma^\{2\}\}\\right\)\\right\)\-\\frac\{\\mu\_\{0\}\}\{\\sigma\}=\(θr∗−zr∗\)​σ\+1σ​\(θr∗−zr∗2−μ0\)\+o​\(1σ\)\.\\displaystyle=\(\\theta\_\{r^\{\*\}\}\-z\_\{r^\{\*\}\}\)\\sigma\+\\frac\{1\}\{\\sigma\}\\left\(\\theta\_\{r^\{\*\}\}\-\\frac\{z\_\{r^\{\*\}\}\}\{2\}\-\\mu\_\{0\}\\right\)\+o\\left\(\\frac\{1\}\{\\sigma\}\\right\)\.
From Proposition 1, we know thatθr∗−zr∗\>0\\theta\_\{r^\{\*\}\}\-z\_\{r^\{\*\}\}\>0\. Therefore,

limσ→∞Pinclstd​\(σ2∣μ0\)\\displaystyle\\lim\_\{\\sigma\\to\\infty\}P\_\{\\text\{incl\}\}^\{\\text\{std\}\}\(\\sigma^\{2\}\\mid\\mu\_\{0\}\)=1−Φ​\(limσ→∞Tstd−μ0σ\)=1−Φ​\(0\)=12,\\displaystyle=1\-\\Phi\\left\(\\lim\_\{\\sigma\\to\\infty\}\\frac\{T\_\{\\text\{std\}\}\-\\mu\_\{0\}\}\{\\sigma\}\\right\)=1\-\\Phi\(0\)=\\frac\{1\}\{2\},limσ→∞Pinclr​\(σ2∣μ0\)\\displaystyle\\lim\_\{\\sigma\\to\\infty\}P\_\{\\text\{incl\}\}^\{r\}\(\\sigma^\{2\}\\mid\\mu\_\{0\}\)=1−Φ​\(limσ→∞Ar​\(σ2\)\)=1−Φ​\(∞\)=0\.\\displaystyle=1\-\\Phi\\left\(\\lim\_\{\\sigma\\to\\infty\}A\_\{r\}\(\\sigma^\{2\}\)\\right\)=1\-\\Phi\(\\infty\)=0\.Thus proved\. ∎

###### Theorem C\.6\.

Let the data\-generating assumptions and notation of Theorem 2 hold\. Suppose, in addition, that:

1. 1\.Uniform dominance of inclusion probabilities\.For every moderately smallσ2≥0\\sigma^\{2\}\\geq 0and every false labely′y^\{\\prime\}withθy′=μ0\\theta\_\{y^\{\\prime\}\}=\\mu\_\{0\}, Pinclr​\(σ2∣μ0\)≤Pinclstd​\(σ2∣μ0\)\.P\_\{\\mathrm\{incl\}\}^\{r\}\\left\(\\sigma^\{2\}\\mid\\mu\_\{0\}\\right\)\\leq P\_\{\\mathrm\{incl\}\}^\{\\mathrm\{std\}\}\\left\(\\sigma^\{2\}\\mid\\mu\_\{0\}\\right\)\.

Then therr\-value conformal predictor produces, on average, a smaller prediction set than standard conformal prediction:

𝔼​\[\|Cr\|\]≤𝔼​\[\|Cstd\|\]\.\\mathbb\{E\}\\left\[\\,\|C\_\{r\}\|\\,\\right\]\\leq\\mathbb\{E\}\\left\[\\,\|C\_\{\\mathrm\{std\}\}\|\\,\\right\]\.

###### Proof\.

LetYtrueY\_\{\\mathrm\{true\}\}denote the true label, and lety′≠Ytruey^\{\\prime\}\\neq Y\_\{\\mathrm\{true\}\}denote any false label\. The expected size of a conformal prediction setCCcan be decomposed as:

𝔼​\[\|C\|\]=𝔼​\[𝟏​\{Ytrue∈C\}\]\+∑y′≠Ytrue𝔼​\[𝟏​\{y′∈C\}\]=ℙ​\(Ytrue∈C\)\+∑y′≠Ytrueℙ​\(y′∈C\)\.\\mathbb\{E\}\[\|C\|\]=\\mathbb\{E\}\[\\mathbf\{1\}\\\{Y\_\{\\mathrm\{true\}\}\\in C\\\}\]\+\\sum\_\{y^\{\\prime\}\\neq Y\_\{\\mathrm\{true\}\}\}\\mathbb\{E\}\[\\mathbf\{1\}\\\{y^\{\\prime\}\\in C\\\}\]=\\mathbb\{P\}\(Y\_\{\\mathrm\{true\}\}\\in C\)\+\\sum\_\{y^\{\\prime\}\\neq Y\_\{\\mathrm\{true\}\}\}\\mathbb\{P\}\(y^\{\\prime\}\\in C\)\.
Since bothCrC\_\{r\}andCstdC\_\{\\mathrm\{std\}\}guarantee marginal coverage, we have:

ℙ​\(Ytrue∈Cr\)=ℙ​\(Ytrue∈Cstd\)≥1−α\.\\mathbb\{P\}\(Y\_\{\\mathrm\{true\}\}\\in C\_\{r\}\)=\\mathbb\{P\}\(Y\_\{\\mathrm\{true\}\}\\in C\_\{\\mathrm\{std\}\}\)\\geq 1\-\\alpha\.
Therefore, following the same idea of proof of Theorem 2, the difference in expected set sizes is governed by the false label terms:

𝔼​\[\|Cr\|\]−𝔼​\[\|Cstd\|\]=∑y′≠Ytrue\[ℙ​\(y′∈Cr\)−ℙ​\(y′∈Cstd\)\]\.\\mathbb\{E\}\[\|C\_\{r\}\|\]\-\\mathbb\{E\}\[\|C\_\{\\mathrm\{std\}\}\|\]=\\sum\_\{y^\{\\prime\}\\neq Y\_\{\\mathrm\{true\}\}\}\\left\[\\mathbb\{P\}\(y^\{\\prime\}\\in C\_\{r\}\)\-\\mathbb\{P\}\(y^\{\\prime\}\\in C\_\{\\mathrm\{std\}\}\)\\right\]\.
Letp​\(σ2\)p\(\\sigma^\{2\}\)be the probability density function \(pdf\) ofσ2\\sigma^\{2\}, and letμ∼𝒩​\(0,1\)\\mu\\sim\\mathcal\{N\}\(0,1\)with pdf denoted byq​\(μ\)q\(\\mu\)\. Then for any conformal setCC,

ℙ​\(y′∈C\)=∫−∞∞∫0∞PinclC​\(σ2∣μ\)⋅p​\(σ2\)​q​\(μ\)​𝑑σ2​𝑑μ\.\\mathbb\{P\}\(y^\{\\prime\}\\in C\)=\\int\_\{\-\\infty\}^\{\\infty\}\\int\_\{0\}^\{\\infty\}P\_\{\\mathrm\{incl\}\}^\{C\}\(\\sigma^\{2\}\\mid\\mu\)\\cdot p\(\\sigma^\{2\}\)\\,q\(\\mu\)\\,d\\sigma^\{2\}\\,d\\mu\.
Define the difference of inclusion probabilities:

Δ​P​\(σ2∣μ\):=Pinclr​\(σ2∣μ\)−Pinclstd​\(σ2∣μ\)\.\\Delta P\(\\sigma^\{2\}\\mid\\mu\):=P\_\{\\mathrm\{incl\}\}^\{r\}\(\\sigma^\{2\}\\mid\\mu\)\-P\_\{\\mathrm\{incl\}\}^\{\\mathrm\{std\}\}\(\\sigma^\{2\}\\mid\\mu\)\.Then,

ℙ​\(y′∈Cr\)−ℙ​\(y′∈Cstd\)=∫−∞∞∫0∞Δ​P​\(σ2∣μ\)⋅p​\(σ2\)​q​\(μ\)​𝑑σ2​𝑑μ\.\\mathbb\{P\}\(y^\{\\prime\}\\in C\_\{r\}\)\-\\mathbb\{P\}\(y^\{\\prime\}\\in C\_\{\\mathrm\{std\}\}\)=\\int\_\{\-\\infty\}^\{\\infty\}\\int\_\{0\}^\{\\infty\}\\Delta P\(\\sigma^\{2\}\\mid\\mu\)\\cdot p\(\\sigma^\{2\}\)\\,q\(\\mu\)\\,d\\sigma^\{2\}\\,d\\mu\.
Theorem 3 implies thatΔ​P​\(σ2∣μ\)<0\\Delta P\(\\sigma^\{2\}\\mid\\mu\)<0for sufficiently largeσ2\\sigma^\{2\}\. Moreover, by the assumption, we haveΔ​P​\(σ2∣μ\)≤0\\Delta P\(\\sigma^\{2\}\\mid\\mu\)\\leq 0for moderately smallσ2\\sigma^\{2\}\. Putting both together, we conclude that:

Δ​P​\(σ2∣μ\)≤0for all​σ2\.\\Delta P\(\\sigma^\{2\}\\mid\\mu\)\\leq 0\\quad\\text\{for all \}\\sigma^\{2\}\.
Hence,

𝔼​\[\|Cr\|\]≤𝔼​\[\|Cstd\|\],\\mathbb\{E\}\[\|C\_\{r\}\|\]\\leq\\mathbb\{E\}\[\|C\_\{\\mathrm\{std\}\}\|\],as desired\.

Furthermore, if we make the additional assumption thatp​\(σ2\)\>0p\(\\sigma^\{2\}\)\>0on any interval\[0,∞\)\[0,\\infty\), then the inequality is strictly negative:

ℙ​\(y′∈Cr\)−ℙ​\(y′∈Cstd\)<0,\\mathbb\{P\}\(y^\{\\prime\}\\in C\_\{r\}\)\-\\mathbb\{P\}\(y^\{\\prime\}\\in C\_\{\\mathrm\{std\}\}\)<0,and we have:

𝔼​\[\|Cr\|\]<𝔼​\[\|Cstd\|\]\.\\mathbb\{E\}\[\|C\_\{r\}\|\]<\\mathbb\{E\}\[\|C\_\{\\mathrm\{std\}\}\|\]\.∎

In the following, we present two extensions that relax the assumptions required in our main result\. Both results allow a proportion of variances to violate the original assumptions while still guaranteeing the efficiency of theCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}method\. We start by showing thatCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}does not produce a larger conformal set in expectation thanCPavg\{\\rm CP\}\_\{\\text\{avg\}\}if the number of high\-variance labels is large\. This result becomes stronger as the number of possible false labels increases\.

###### Theorem C\.7\(R\-value selection under weaker assumption\)\.

Following the notation of Theorem 2\. Let

𝐇:=\{i:σi2\>sconj\},𝐋:=\{i:σi2≤sconj\},k:=\|𝐇\|\+\|𝐋\|\\mathbf\{H\}:=\\bigl\\\{i:\\sigma\_\{i\}^\{2\}\>s\_\{\\mathrm\{conj\}\}\\bigr\\\},\\qquad\\mathbf\{L\}:=\\bigl\\\{i:\\sigma\_\{i\}^\{2\}\\leq s\_\{\\mathrm\{conj\}\}\\bigr\\\},\\qquad k:=\|\\mathbf\{H\}\|\+\|\\mathbf\{L\}\|denote the sets of high\- and low\-variance labels, respectively, for a new test input, wherekkis the total number of possible false labels\. For each labelii, letpiavgp\_\{i\}^\{\\mathrm\{avg\}\}andpirp\_\{i\}^\{r\}denote the inclusion probabilities underCPavg\{\\rm CP\}\_\{\\text\{avg\}\}andCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}, respectively\. Define

Δ:=mini∈𝐇⁡\(piavg−pir\),ε:=maxi∈𝐋⁡\(pir−piavg\),f​\(δ\):=Δ​\(1−δ\)−ε​δ,\\Delta:=\\min\_\{i\\in\\mathbf\{H\}\}\\\!\\bigl\(p\_\{i\}^\{\\mathrm\{avg\}\}\-p\_\{i\}^\{r\}\\bigr\),\\qquad\\varepsilon:=\\max\_\{i\\in\\mathbf\{L\}\}\\\!\\bigl\(p\_\{i\}^\{r\}\-p\_\{i\}^\{\\mathrm\{avg\}\}\\bigr\),\\qquad f\(\\delta\):=\\Delta\(1\-\\delta\)\-\\varepsilon\\delta,whereδ≥\|𝐋\|k\\delta\\geq\\frac\{\|\\mathbf\{L\}\|\}\{k\}is an upper bound on the proportion of low\-variance labels\.

1. 1\.Expectation\-level guarantee\.If δ≤ΔΔ\+ε⟺\|𝐇\|\|𝐋\|≥εΔ,\\delta\\leq\\frac\{\\Delta\}\{\\Delta\+\\varepsilon\}\\quad\\Longleftrightarrow\\quad\\frac\{\|\\mathbf\{H\}\|\}\{\|\\mathbf\{L\}\|\}\\geq\\frac\{\\varepsilon\}\{\\Delta\},then, for allk≥1k\\geq 1, 𝔼​\[\|Cr\|\]≤𝔼​\[\|Cavg\|\]\.\\mathbb\{E\}\\bigl\[\|C\_\{r\}\|\\bigr\]\\leq\\mathbb\{E\}\\bigl\[\|C\_\{\\mathrm\{avg\}\}\|\\bigr\]\.
2. 2\.High\-probability guarantee\.Fix any confidence levelη∈\(0,1\)\\eta\\in\(0,1\)\. If k≥2​log⁡\(1/η\)f​\(δ\)2,k\\geq\\frac\{2\\log\(1/\\eta\)\}\{f\(\\delta\)^\{2\}\},then with probability at least1−η1\-\\eta, \|Cr\|<\|Cavg\|\.\|C\_\{r\}\|<\|C\_\{\\mathrm\{avg\}\}\|\.

###### Proof\.

Since at most aδ\\deltafraction of labels have variance less than or equal tosconjs\_\{\\text\{conj\}\}, we have set sizes\|𝐋\|≤δ​k\|\\mathbf\{L\}\|\\leq\\delta kand\|𝐇\|≥\(1−δ\)​k\|\\mathbf\{H\}\|\\geq\(1\-\\delta\)k\.

Define

Gk:=∑i=1k\[𝟏​\(i∈Cavg\)−𝟏​\(i∈Cr\)\],G\_\{k\}:=\\sum\_\{i=1\}^\{k\}\\left\[\\mathbf\{1\}\(i\\in C\_\{\\mathrm\{avg\}\}\)\-\\mathbf\{1\}\(i\\in C\_\{r\}\)\\right\],as the difference of number of labels selected in two sets, and letpiavgp\_\{i\}^\{\\mathrm\{avg\}\}andpirp\_\{i\}^\{r\}denote the probabilities that labeliiis included inCavgC\_\{\\mathrm\{avg\}\}andCrC\_\{r\}, respectively\. Then,

𝔼​\[Gk\]=∑i=1k\(piavg−pir\)=∑i∈𝐇\(piavg−pir\)\+∑i∈𝐋\(piavg−pir\)\.\\mathbb\{E\}\[G\_\{k\}\]=\\sum\_\{i=1\}^\{k\}\\left\(p\_\{i\}^\{\\mathrm\{avg\}\}\-p\_\{i\}^\{r\}\\right\)=\\sum\_\{i\\in\\mathbf\{H\}\}\(p\_\{i\}^\{\\mathrm\{avg\}\}\-p\_\{i\}^\{r\}\)\+\\sum\_\{i\\in\\mathbf\{L\}\}\(p\_\{i\}^\{\\mathrm\{avg\}\}\-p\_\{i\}^\{r\}\)\.
Fori∈𝐇i\\in\\mathbf\{H\}, Proposition 2 guaranteespiavg−pir≥Δ=mini∈𝐇⁡\(piavg−pir\)≥0p\_\{i\}^\{\\mathrm\{avg\}\}\-p\_\{i\}^\{r\}\\geq\\Delta=\\min\_\{i\\in\\mathbf\{H\}\}\\\!\\bigl\(p\_\{i\}^\{\\mathrm\{avg\}\}\-p\_\{i\}^\{r\}\\bigr\)\\geq 0\. Therefore,

∑i∈𝐇\(piavg−pir\)≥Δ​\|𝐇\|≥Δ​\(1−δ\)​k\.\\sum\_\{i\\in\\mathbf\{H\}\}\(p\_\{i\}^\{\\mathrm\{avg\}\}\-p\_\{i\}^\{r\}\)\\geq\\Delta\|\\mathbf\{H\}\|\\geq\\Delta\(1\-\\delta\)k\.
Fori∈𝐋i\\in\\mathbf\{L\}, by the Extreme Value Theorem, there existsε=maxi∈𝐋⁡\(pir−piavg\)\>0\\varepsilon=\\max\_\{i\\in\\mathbf\{L\}\}\\\!\\bigl\(p\_\{i\}^\{r\}\-p\_\{i\}^\{\\mathrm\{avg\}\}\\bigr\)\>0such that

\|piavg−pir\|≤ε⇒∑i∈𝐋\(piavg−pir\)≥−ε​\|𝐋\|≥−ε​δ​k\.\|p\_\{i\}^\{\\mathrm\{avg\}\}\-p\_\{i\}^\{r\}\|\\leq\\varepsilon\\quad\\Rightarrow\\quad\\sum\_\{i\\in\\mathbf\{L\}\}\(p\_\{i\}^\{\\mathrm\{avg\}\}\-p\_\{i\}^\{r\}\)\\geq\-\\varepsilon\|\\mathbf\{L\}\|\\geq\-\\varepsilon\\delta k\.
Combining both bounds:

𝔼​\[Gk\]≥Δ​\|H\|−ϵ​\|L\|=Δ​\(1−δ\)​k−ε​δ​k=k⋅f​\(δ\),where​f​\(δ\):=Δ​\(1−δ\)−ε​δ\.\\mathbb\{E\}\[G\_\{k\}\]\\geq\\Delta\|H\|\-\\epsilon\|L\|=\\Delta\(1\-\\delta\)k\-\\varepsilon\\delta k=k\\cdot f\(\\delta\),\\quad\\text\{where \}f\(\\delta\):=\\Delta\(1\-\\delta\)\-\\varepsilon\\delta\.
1. 1\.Expectation\-level guarantee\.Iff​\(δ\)≥0f\(\\delta\)\\geq 0, then𝔼​\[Gk\]≥0\\mathbb\{E\}\[G\_\{k\}\]\\geq 0\. This implies 𝔼​\[\|Cavg\|\]≥𝔼​\[\|Cr\|\]\.\\mathbb\{E\}\\left\[\|C\_\{\\mathrm\{avg\}\}\|\\right\]\\geq\\mathbb\{E\}\\left\[\|C\_\{r\}\|\\right\]\.
2. 2\.High\-probability guarantee\.Note that for eachii, the difference𝟏​\(i∈Cavg\)−𝟏​\(i∈Cr\)\\mathbf\{1\}\(i\\in C\_\{\\mathrm\{avg\}\}\)\-\\mathbf\{1\}\(i\\in C\_\{r\}\)lies in\[−1,1\]\[\-1,1\]\. Since these are independent acrossii, we may apply Hoeffding’s inequality: ℙ​\(Gk≤0\)\\displaystyle\\mathbb\{P\}\(G\_\{k\}\\leq 0\)=ℙ​\(Gk−𝔼​\[Gk\]≤−𝔼​\[Gk\]\)\\displaystyle=\\mathbb\{P\}\\left\(G\_\{k\}\-\\mathbb\{E\}\[G\_\{k\}\]\\leq\-\\mathbb\{E\}\[G\_\{k\}\]\\right\)≤exp⁡\(−2​𝔼​\[Gk\]24​k\)\\displaystyle\\leq\\exp\\left\(\-\\frac\{2\\,\\mathbb\{E\}\[G\_\{k\}\]^\{2\}\}\{4k\}\\right\)≤exp⁡\(−2​\(k​f​\(δ\)\)24​k\)=exp⁡\(−12​k​f​\(δ\)2\)\.\\displaystyle\\leq\\exp\\left\(\-\\frac\{2\(kf\(\\delta\)\)^\{2\}\}\{4k\}\\right\)=\\exp\\left\(\-\\frac\{1\}\{2\}kf\(\\delta\)^\{2\}\\right\)\. To ensure ℙ​\(\|Cr\|<\|Cavg\|\)≥1−η,\\mathbb\{P\}\\left\(\|C\_\{r\}\|<\|C\_\{\\mathrm\{avg\}\}\|\\right\)\\geq 1\-\\eta,we require: exp⁡\(−12​k​f​\(δ\)2\)≤η⟺k≥2​log⁡\(1/η\)f​\(δ\)2\.\\exp\\left\(\-\\frac\{1\}\{2\}kf\(\\delta\)^\{2\}\\right\)\\leq\\eta\\quad\\Longleftrightarrow\\quad k\\geq\\frac\{2\\log\(1/\\eta\)\}\{f\(\\delta\)^\{2\}\}\. Thus, the condition in the theorem statement guarantees the result\.

∎

Interpretation\.When high\-variance labels outnumber low\-variance byε/Δ\\varepsilon/\\Delta,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}always produce a conformal set that is not larger thanCPavg\{\\rm CP\}\_\{\\text\{avg\}\}in expectation for anykk\. The probability ofCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}generating a strictly smaller conformal set thanCPavg\{\\rm CP\}\_\{\\text\{avg\}\}converges to 1 at an exponential rate inkk, which matches the empirical finding thatCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}perform better with more possible labels where variability becomes an important signal \(see Table 1\)\.

We then show that even ifCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}is only on average less likely to include false labels thanCP\{\\rm CP\}, it still guarantees a smaller expected conformal set\.

###### Theorem C\.8\(Weaker assumption for Theorem 4\)\.

Follow the setup and notation of Theorem 4\. For every false labely′y^\{\\prime\}with latent meanθy′=μ0\\theta\_\{y^\{\\prime\}\}=\\mu\_\{0\}, let

Δ​P​\(σ2∣μ0\)=Pinclr​\(σ2∣μ0\)−Pinclstd​\(σ2∣μ0\),p​\(σ2\)\\Delta P\(\\sigma^\{2\}\\mid\\mu\_\{0\}\)\\;=\\;P\_\{\\mathrm\{incl\}\}^\{r\}\\\!\\left\(\\sigma^\{2\}\\mid\\mu\_\{0\}\\right\)\\;\-\\;P\_\{\\mathrm\{incl\}\}^\{\\mathrm\{std\}\}\\\!\\left\(\\sigma^\{2\}\\mid\\mu\_\{0\}\\right\),\\quad p\(\\sigma^\{2\}\)denote the inclusion\-probability gap and the pdf of the varianceσ2\\sigma^\{2\}\. Suppose that the average gap is negative:

∫0∞Δ​P​\(σ2∣μ0\)​p​\(σ2\)​𝑑σ2<0\.\\int\_\{0\}^\{\\infty\}\\\!\\Delta P\(\\sigma^\{2\}\\mid\\mu\_\{0\}\)\\;p\(\\sigma^\{2\}\)\\,d\\sigma^\{2\}\\;<\\;0\.\(A\)Then therr\-value conformal predictor yields a smaller expected prediction set than standard conformal prediction:

𝔼​\[\|Cr\|\]≤𝔼​\[\|Cstd\|\]\.\\mathbb\{E\}\\\!\\left\[\\,\|C\_\{r\}\|\\,\\right\]\\;\\leq\\;\\mathbb\{E\}\\\!\\left\[\\,\|C\_\{\\mathrm\{std\}\}\|\\,\\right\]\.

###### Proof\.

LetΔ​E:=𝔼​\[\|Cr\|\]−𝔼​\[\|Cstd\|\]\\Delta E:=\\mathbb\{E\}\[\|C\_\{r\}\|\]\-\\mathbb\{E\}\[\|C\_\{\\mathrm\{std\}\}\|\]\. Because both methods have the same marginal coverage for the true label, we follow the same idea of proof of Theorem 4\. The difference arises only from false labels:

∫−∞∞∫0∞Δ​P​\(σ2∣μ\)⋅p​\(σ2\)​q​\(μ\)​𝑑σ2​𝑑μ,\\int\_\{\-\\infty\}^\{\\infty\}\\int\_\{0\}^\{\\infty\}\\Delta P\(\\sigma^\{2\}\\mid\\mu\)\\cdot p\(\\sigma^\{2\}\)\\,q\(\\mu\)\\,d\\sigma^\{2\}\\,d\\mu,which is less than or equal to 0 due to Assumption A andp​\(σ2\)p\(\\sigma^\{2\}\)is nonnegative, which implies that

𝔼​\[\|Cr\|\]≤𝔼​\[\|Cstd\|\]\.\\mathbb\{E\}\[\|C\_\{r\}\|\]\\leq\\mathbb\{E\}\[\|C\_\{\\mathrm\{std\}\}\|\]\.Similarly, if we further assume thatp​\(σ2\)\>0p\(\\sigma^\{2\}\)\>0on any interval\[0,∞\)\[0,\\infty\), then the inequality is strictly negative and we have

𝔼​\[\|Cr\|\]<𝔼​\[\|Cstd\|\]\.\\mathbb\{E\}\[\|C\_\{r\}\|\]<\\mathbb\{E\}\[\|C\_\{\\mathrm\{std\}\}\|\]\.∎

## Appendix DAdditional Image Classification Results

Comparable performance in the single\-backbone setting\.In Figure[5](https://arxiv.org/html/2605.23189#A4.F5), we compare standard conformal prediction andCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}on ImageNet using ResNet18, ResNet50, ViT\-Base, and ViT\-Large\. We evaluate both probability and logit settings and report coverage and set size across different significance levels\. In most cases,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}achieves performance comparable to standard CP\. This behavior is expected in the single backbone setting\. Although we use WBB adapters to obtain posterior samples around the pretrained backbone, the backbone itself is frozen, and the adapters are lightweight\. Therefore, the induced variability is relatively modest\. When the variability signal is small, therr\-value ranking naturally becomes close to the usual first order ranking used by CP orCPavg\{\\rm CP\}\_\{\\text\{avg\}\}\. Thus, the similarity betweenCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}and CP in Figure[5](https://arxiv.org/html/2605.23189#A4.F5)should be interpreted as a sanity check rather than a failure case\.

![Refer to caption](https://arxiv.org/html/2605.23189v1/AISTATS/figures/Untitled_design-2.png)Figure 5:Comparison of conformal prediction andCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}on image classification using ResNet18, ResNet50, ViT\-Base, and ViT\-Large\. We evaluate both probability and logit settings, analyzing coverage and set size across significance levels\. In the single\-backbone setting, WBB adapters introduce only modest posterior variability, soCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}often behaves similarly to standard CP\.Accuracy trade\-off from adapter training\.Using adapters provides an efficient way to estimate model variability, but it also introduces a accuracy trade off\. Since only a small fraction of the model parameters are trainable while the pretrained backbone is frozen, the adapter augmented models may not fully preserve the accuracy of the original backbone\. Table[4](https://arxiv.org/html/2605.23189#A4.T4)reports this effect\. The degradation is modest across all models\. For ViT models, the drop is around11\-1\.5%1\.5\\%, while for ResNet50, it is much smaller\. This accuracy loss should be viewed as the cost of obtaining an efficient approximation of epistemic variability\. It is not an inherent limitation of therr\-value framework itself\.

ModelWithout AdaptersWith Adapters \(Average\)ViT\-L/1685\.15%83\.73%ViT\-B/1681\.16%79\.87%ResNet5079\.26%79\.05%ResNet1871\.12%69\.95%Table 4:Raw classification accuracy before and after introducing WBB\-trained adapters\. The adapter\-based posterior samples provide an efficient estimate of epistemic variability, but the lightweight parameterization can introduce a modest accuracy drop\.Effect of variability quality\.The effectiveness ofCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}depends on the quality of the estimated variability\. In our current implementation, we use adapters because they make WBB posterior sampling computationally feasible for large vision models\. This choice is efficient, but the resulting posterior samples may only capture part of the full model variability\. As a result, the variance signal can be weak in the single backbone setting, which makesCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}behave similarly to CP\. This is consistent with our theoretical analysis\. When variability vanishes or is uninformative, therr\-value reduces to a first order ranking\. When variability is informative,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}can use it to penalize unstable false labels and reduce the conformal set size\.

This also suggests that better variability estimation can further improve the empirical performance ofCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}\. Our method is not tied to shallow adapters\. Other efficient posterior\-sampling or uncertainty\-estimation methods can be used as long as they provide meaningful posterior score samples\. For example, MC\-dropout, Deep Ensembles, or better calibrated paraphrase sampling in VLM and LLM settings may provide higher quality variability estimates\. In such cases, therr\-value ranking may better distinguish stable high score labels from labels that only appear high due to noise\. \(More experimental comparisons will be discussed in the next subsection\.\)

Ranking quality within the conformal set\.Beyond coverage and set size, we also examine where the true label appears inside the selected conformal set\. Table[5](https://arxiv.org/html/2605.23189#A4.T5)reports the average index of the true label\. A smaller value means that the true label appears earlier in the ranked set\. For ResNet18, ResNet50, and ViT\-Base, therr\-value rankings place the true label earlier than standard CP on average\. This suggests that even when aggregate coverage and set size are similar,CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}can still improve the ordering of plausible labels\. ViT\-Large is an exception\. We suggest that it is because ViT\-Large is trained to overfit; the attached adapters might break this trend\. In this case, all methods produce very small sets, and CP has the smallest average index\. This is consistent with the high accuracy and low variability regime, where there is less room for ranking improvement andCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}naturally approaches CP\-like behavior\.

MethodResNet18ResNet50ViT\-BViT\-LConformal Prediction1\.1590\.4970\.2940\.158rr\-value Under Logits1\.0920\.3510\.2720\.197rr\-value Under Logits \(with Assumption\)1\.0820\.3480\.2780\.164Table 5:Average index of the true label within the selected conformal set\. Smaller values indicate that the true label appears earlier in the ranked set\.
## Appendix EAdditional VLM Results

#### Effect of paraphrase quality\.

The quality of the variability source also affects the benefit ofCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}\. To study this, we compare using all 30 randomly generated paraphrases with using the top 20 rephrased prompts selected from them\. Both settings maintain coverage close to the nominal level, but the Top\-20 setting often gives slightly smaller conformal sets or larger relative reductions, especially for MobileCLIP2 variants\. For example, under MobileCLIP2\-S2, the gain overCPavg\{\\rm CP\}\_\{\\text\{avg\}\}increases from1\.4%1\.4\\%with 30 paraphrases to5\.8%5\.8\\%with Top\-20 paraphrases, and the gain overCP\{\\rm CP\}increases from3\.8%3\.8\\%to7\.6%7\.6\\%\. Similarly, MobileCLIP2\-S4 improves from4\.9%4\.9\\%to6\.2%6\.2\\%overCPavg\{\\rm CP\}\_\{\\text\{avg\}\}and from5\.2%5\.2\\%to6\.6%6\.6\\%overCP\{\\rm CP\}\.

This supports the intuition thatCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}benefits not only from having multiple perturbations, but also from having informative perturbations\. Low quality or noisy paraphrases may introduce variability that is less aligned with meaningful epistemic uncertainty, weakening the rank stability signal\. In contrast, higher quality paraphrases provide a cleaner estimate of which candidates remain consistently strong across prompt variations, allowingCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}to better penalize unstable false candidates while preserving coverage\.

Table 6:Model\-level macro\-average coverage and set size across datasets atα=0\.05\\alpha=0\.05\(30 paraphrases\)\. Set\-size parentheses report the relative change ofrr\-value versus the corresponding CP baseline, computed from the unrounded set\-size means\.Coverage \(%\)Set sizeModelCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}SigLIP2\-B/1695\.2±1\.5\\mathbf\{95\.2\\pm 1\.5\}95\.4±1\.495\.4\\pm 1\.495\.5±1\.495\.5\\pm 1\.44\.0±0\.5\\mathbf\{4\.0\\pm 0\.5\}4\.6±0\.64\.6\\pm 0\.6\(\-13\.1%\)5\.3±0\.95\.3\\pm 0\.9\(\-23\.9%\)SigLIP2\-B/3295\.1±1\.6\\mathbf\{95\.1\\pm 1\.6\}95\.3±1\.595\.3\\pm 1\.595\.4±1\.495\.4\\pm 1\.411\.8±1\.1\\mathbf\{11\.8\\pm 1\.1\}13\.1±1\.413\.1\\pm 1\.4\(\-10\.3%\)14\.1±1\.514\.1\\pm 1\.5\(\-16\.8%\)SigLIP2\-L/1695\.2±1\.5\\mathbf\{95\.2\\pm 1\.5\}95\.4±1\.395\.4\\pm 1\.395\.3±1\.495\.3\\pm 1\.43\.8±0\.4\\mathbf\{3\.8\\pm 0\.4\}4\.2±0\.54\.2\\pm 0\.5\(\-9\.6%\)4\.5±0\.64\.5\\pm 0\.6\(\-15\.9%\)SigLIP2\-SO400M/1695\.0±1\.5\\mathbf\{95\.0\\pm 1\.5\}95\.6±1\.395\.6\\pm 1\.395\.5±1\.495\.5\\pm 1\.43\.3±0\.3\\mathbf\{3\.3\\pm 0\.3\}3\.7±0\.53\.7\\pm 0\.5\(\-11\.0%\)4\.0±0\.54\.0\\pm 0\.5\(\-16\.4%\)MobileCLIP2\-S295\.2±1\.5\\mathbf\{95\.2\\pm 1\.5\}95\.7±1\.395\.7\\pm 1\.395\.7±1\.295\.7\\pm 1\.25\.9±0\.5\\mathbf\{5\.9\\pm 0\.5\}6\.0±0\.66\.0\\pm 0\.6\(\-1\.4%\)6\.1±0\.66\.1\\pm 0\.6\(\-3\.8%\)MobileCLIP2\-S395\.3±1\.5\\mathbf\{95\.3\\pm 1\.5\}95\.7±1\.395\.7\\pm 1\.395\.6±1\.395\.6\\pm 1\.34\.0±0\.3\\mathbf\{4\.0\\pm 0\.3\}4\.2±0\.44\.2\\pm 0\.4\(\-6\.3%\)4\.4±0\.34\.4\\pm 0\.3\(\-10\.3%\)MobileCLIP2\-S495\.3±1\.5\\mathbf\{95\.3\\pm 1\.5\}95\.6±1\.395\.6\\pm 1\.395\.6±1\.395\.6\\pm 1\.33\.2±0\.3\\mathbf\{3\.2\\pm 0\.3\}3\.4±0\.33\.4\\pm 0\.3\(\-4\.9%\)3\.4±0\.43\.4\\pm 0\.4\(\-5\.2%\)MobileCLIP2\-B95\.2±1\.5\\mathbf\{95\.2\\pm 1\.5\}95\.6±1\.395\.6\\pm 1\.395\.7±1\.395\.7\\pm 1\.34\.7±0\.6\\mathbf\{4\.7\\pm 0\.6\}4\.8±0\.54\.8\\pm 0\.5\(\-3\.4%\)5\.2±0\.55\.2\\pm 0\.5\(\-10\.0%\)Table 7:Model\-level macro\-average coverage and set size across datasets atα=0\.05\\alpha=0\.05\(Top\-20 rephrased\)\. Set\-size parentheses report the relative change ofrr\-value versus the corresponding CP baseline, computed from the unrounded set\-size means\.Coverage \(%\)Set sizeModelCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}SigLIP2\-B/1695\.1±1\.6\\mathbf\{95\.1\\pm 1\.6\}95\.4±1\.495\.4\\pm 1\.495\.5±1\.495\.5\\pm 1\.44\.0±0\.5\\mathbf\{4\.0\\pm 0\.5\}4\.5±0\.64\.5\\pm 0\.6\(\-11\.3%\)5\.3±0\.95\.3\\pm 0\.9\(\-23\.6%\)SigLIP2\-B/3295\.1±1\.5\\mathbf\{95\.1\\pm 1\.5\}95\.3±1\.595\.3\\pm 1\.595\.4±1\.495\.4\\pm 1\.411\.9±1\.1\\mathbf\{11\.9\\pm 1\.1\}13\.2±1\.413\.2\\pm 1\.4\(\-10\.0%\)14\.1±1\.514\.1\\pm 1\.5\(\-15\.6%\)SigLIP2\-L/1695\.0±1\.5\\mathbf\{95\.0\\pm 1\.5\}95\.4±1\.495\.4\\pm 1\.495\.3±1\.495\.3\\pm 1\.43\.7±0\.4\\mathbf\{3\.7\\pm 0\.4\}4\.2±0\.64\.2\\pm 0\.6\(\-11\.4%\)4\.5±0\.64\.5\\pm 0\.6\(\-16\.6%\)SigLIP2\-SO400M/1694\.9±1\.6\\mathbf\{94\.9\\pm 1\.6\}95\.6±1\.395\.6\\pm 1\.395\.5±1\.495\.5\\pm 1\.43\.3±0\.3\\mathbf\{3\.3\\pm 0\.3\}3\.6±0\.43\.6\\pm 0\.4\(\-8\.7%\)3\.9±0\.53\.9\\pm 0\.5\(\-17\.4%\)MobileCLIP2\-S295\.1±1\.5\\mathbf\{95\.1\\pm 1\.5\}95\.7±1\.395\.7\\pm 1\.395\.7±1\.395\.7\\pm 1\.35\.6±0\.5\\mathbf\{5\.6\\pm 0\.5\}6\.0±0\.66\.0\\pm 0\.6\(\-5\.8%\)6\.1±0\.66\.1\\pm 0\.6\(\-7\.6%\)MobileCLIP2\-S395\.2±1\.5\\mathbf\{95\.2\\pm 1\.5\}95\.7±1\.395\.7\\pm 1\.395\.7±1\.295\.7\\pm 1\.24\.0±0\.4\\mathbf\{4\.0\\pm 0\.4\}4\.2±0\.34\.2\\pm 0\.3\(\-4\.9%\)4\.4±0\.34\.4\\pm 0\.3\(\-9\.7%\)MobileCLIP2\-S495\.2±1\.5\\mathbf\{95\.2\\pm 1\.5\}95\.6±1\.395\.6\\pm 1\.395\.6±1\.395\.6\\pm 1\.33\.2±0\.3\\mathbf\{3\.2\\pm 0\.3\}3\.4±0\.33\.4\\pm 0\.3\(\-6\.2%\)3\.4±0\.43\.4\\pm 0\.4\(\-6\.6%\)MobileCLIP2\-B95\.2±1\.5\\mathbf\{95\.2\\pm 1\.5\}95\.7±1\.495\.7\\pm 1\.495\.7±1\.395\.7\\pm 1\.34\.7±0\.6\\mathbf\{4\.7\\pm 0\.6\}4\.9±0\.54\.9\\pm 0\.5\(\-4\.1%\)5\.2±0\.55\.2\\pm 0\.5\(\-10\.1%\)Table 8:Full coverage and set\-size results atα=0\.05\\alpha=0\.05\(30 paraphrases\)\. Set\-size parentheses report the relative change ofrr\-value versus the corresponding CP baseline, computed from the unrounded set\-size means\.ModelDatasetCoverage \(%\)Set sizeCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}CLIP\-B/16CIFAR\-1095\.5±2\.0\\mathbf\{95\.5\\pm 2\.0\}95\.6±1\.395\.6\\pm 1\.395\.5±1\.695\.5\\pm 1\.61\.3±0\.1\\mathbf\{1\.3\\pm 0\.1\}1\.4±0\.11\.4\\pm 0\.1\(\-6\.6%\)1\.4±0\.11\.4\\pm 0\.1\(\-8\.3%\)CIFAR\-10095\.1±1\.7\\mathbf\{95\.1\\pm 1\.7\}95\.8±1\.895\.8\\pm 1\.895\.4±1\.795\.4\\pm 1\.77\.8±0\.6\\mathbf\{7\.8\\pm 0\.6\}8\.2±1\.28\.2\\pm 1\.2\(\-5\.5%\)8\.3±1\.18\.3\\pm 1\.1\(\-7\.1%\)ImageNet95\.2±1\.4\\mathbf\{95\.2\\pm 1\.4\}95\.3±1\.295\.3\\pm 1\.295\.1±1\.495\.1\\pm 1\.48\.6±1\.1\\mathbf\{8\.6\\pm 1\.1\}8\.9±1\.28\.9\\pm 1\.2\(\-3\.1%\)10\.2±2\.410\.2\\pm 2\.4\(\-16\.0%\)ImageNet\-A95\.1±1\.3\\mathbf\{95\.1\\pm 1\.3\}95\.5±1\.495\.5\\pm 1\.495\.2±1\.495\.2\\pm 1\.422\.5±3\.3\\mathbf\{22\.5\\pm 3\.3\}23\.5±2\.523\.5\\pm 2\.5\(\-4\.6%\)24\.7±2\.724\.7\\pm 2\.7\(\-9\.0%\)ImageNet\-R95\.0±1\.1\\mathbf\{95\.0\\pm 1\.1\}95\.1±1\.395\.1\\pm 1\.394\.9±1\.494\.9\\pm 1\.47\.2±1\.4\\mathbf\{7\.2\\pm 1\.4\}7\.3±1\.47\.3\\pm 1\.4\(\-2\.3%\)8\.4±1\.58\.4\\pm 1\.5\(\-14\.7%\)EuroSAT94\.6±1\.4\\mathbf\{94\.6\\pm 1\.4\}95\.2±1\.595\.2\\pm 1\.595\.3±1\.395\.3\\pm 1\.35\.6±0\.2\\mathbf\{5\.6\\pm 0\.2\}5\.9±0\.35\.9\\pm 0\.3\(\-5\.6%\)6\.0±0\.26\.0\\pm 0\.2\(\-6\.4%\)CLIP\-B/32CIFAR\-1095\.3±2\.2\\mathbf\{95\.3\\pm 2\.2\}95\.8±1\.795\.8\\pm 1\.795\.8±1\.595\.8\\pm 1\.51\.2±0\.1\\mathbf\{1\.2\\pm 0\.1\}1\.2±0\.11\.2\\pm 0\.1\(\-4\.0%\)1\.2±0\.11\.2\\pm 0\.1\(\-3\.1%\)CIFAR\-10095\.5±1\.5\\mathbf\{95\.5\\pm 1\.5\}95\.4±2\.095\.4\\pm 2\.095\.7±1\.695\.7\\pm 1\.67\.3±0\.7\\mathbf\{7\.3\\pm 0\.7\}7\.7±1\.07\.7\\pm 1\.0\(\-4\.5%\)9\.3±1\.89\.3\\pm 1\.8\(\-21\.6%\)ImageNet95\.1±1\.5\\mathbf\{95\.1\\pm 1\.5\}95\.3±1\.495\.3\\pm 1\.495\.0±1\.695\.0\\pm 1\.612\.7±2\.1\\mathbf\{12\.7\\pm 2\.1\}12\.7±1\.512\.7\\pm 1\.5\(0\.0%\)14\.1±2\.414\.1\\pm 2\.4\(\-10\.4%\)ImageNet\-A95\.2±1\.5\\mathbf\{95\.2\\pm 1\.5\}95\.3±1\.495\.3\\pm 1\.495\.0±1\.295\.0\\pm 1\.247\.5±4\.8\\mathbf\{47\.5\\pm 4\.8\}47\.6±4\.047\.6\\pm 4\.0\(\-0\.2%\)60\.4±7\.360\.4\\pm 7\.3\(\-21\.4%\)ImageNet\-R95\.1±1\.3\\mathbf\{95\.1\\pm 1\.3\}95\.0±1\.395\.0\\pm 1\.395\.1±1\.395\.1\\pm 1\.313\.6±3\.0\\mathbf\{13\.6\\pm 3\.0\}14\.0±2\.214\.0\\pm 2\.2\(\-2\.8%\)14\.9±2\.414\.9\\pm 2\.4\(\-8\.9%\)EuroSAT94\.9±1\.5\\mathbf\{94\.9\\pm 1\.5\}95\.4±1\.395\.4\\pm 1\.395\.2±1\.495\.2\\pm 1\.46\.2±0\.4\\mathbf\{6\.2\\pm 0\.4\}6\.2±0\.46\.2\\pm 0\.4\(\-0\.2%\)7\.0±0\.37\.0\\pm 0\.3\(\-11\.9%\)SigLIP2\-B/16CIFAR\-1095\.6±1\.7\\mathbf\{95\.6\\pm 1\.7\}95\.9±1\.695\.9\\pm 1\.696\.3±1\.596\.3\\pm 1\.51\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.1±0\.01\.1\\pm 0\.0\(\-2\.5%\)1\.0±0\.01\.0\\pm 0\.0\(\-0\.1%\)CIFAR\-10095\.3±2\.0\\mathbf\{95\.3\\pm 2\.0\}95\.5±2\.095\.5\\pm 2\.095\.6±1\.795\.6\\pm 1\.74\.6±0\.8\\mathbf\{4\.6\\pm 0\.8\}5\.0±0\.85\.0\\pm 0\.8\(\-7\.6%\)5\.0±1\.05\.0\\pm 1\.0\(\-7\.8%\)ImageNet94\.9±1\.3\\mathbf\{94\.9\\pm 1\.3\}95\.0±1\.195\.0\\pm 1\.195\.4±1\.495\.4\\pm 1\.47\.0±1\.5\\mathbf\{7\.0\\pm 1\.5\}8\.6±1\.38\.6\\pm 1\.3\(\-18\.4%\)10\.7±2\.810\.7\\pm 2\.8\(\-34\.8%\)ImageNet\-A95\.2±1\.2\\mathbf\{95\.2\\pm 1\.2\}95\.3±1\.295\.3\\pm 1\.295\.1±1\.495\.1\\pm 1\.45\.3±0\.6\\mathbf\{5\.3\\pm 0\.6\}5\.9±0\.65\.9\\pm 0\.6\(\-10\.4%\)6\.9±1\.06\.9\\pm 1\.0\(\-23\.3%\)ImageNet\-R95\.3±1\.6\\mathbf\{95\.3\\pm 1\.6\}95\.4±1\.395\.4\\pm 1\.395\.3±1\.295\.3\\pm 1\.21\.1±0\.0\\mathbf\{1\.1\\pm 0\.0\}2\.2±0\.62\.2\\pm 0\.6\(\-47\.7%\)1\.8±0\.31\.8\\pm 0\.3\(\-36\.5%\)EuroSAT94\.9±1\.4\\mathbf\{94\.9\\pm 1\.4\}95\.2±1\.495\.2\\pm 1\.495\.3±1\.195\.3\\pm 1\.15\.1±0\.2\\mathbf\{5\.1\\pm 0\.2\}5\.1±0\.15\.1\\pm 0\.1\(\-0\.5%\)6\.2±0\.26\.2\\pm 0\.2\(\-19\.1%\)SigLIP2\-B/32CIFAR\-1095\.4±1\.7\\mathbf\{95\.4\\pm 1\.7\}95\.6±1\.895\.6\\pm 1\.896\.0±1\.896\.0\\pm 1\.81\.1±0\.0\\mathbf\{1\.1\\pm 0\.0\}1\.1±0\.01\.1\\pm 0\.0\(\-0\.9%\)1\.1±0\.01\.1\\pm 0\.0\(\-2\.7%\)CIFAR\-10095\.3±2\.0\\mathbf\{95\.3\\pm 2\.0\}95\.6±1\.795\.6\\pm 1\.795\.7±1\.595\.7\\pm 1\.55\.1±0\.8\\mathbf\{5\.1\\pm 0\.8\}5\.8±1\.25\.8\\pm 1\.2\(\-13\.0%\)5\.4±1\.05\.4\\pm 1\.0\(\-5\.6%\)ImageNet94\.8±1\.3\\mathbf\{94\.8\\pm 1\.3\}95\.1±1\.495\.1\\pm 1\.495\.1±1\.295\.1\\pm 1\.213\.0±2\.1\\mathbf\{13\.0\\pm 2\.1\}14\.3±2\.314\.3\\pm 2\.3\(\-9\.3%\)19\.6±2\.819\.6\\pm 2\.8\(\-33\.6%\)ImageNet\-A95\.0±1\.5\\mathbf\{95\.0\\pm 1\.5\}94\.9±1\.494\.9\\pm 1\.495\.0±1\.495\.0\\pm 1\.443\.5±3\.4\\mathbf\{43\.5\\pm 3\.4\}46\.2±3\.346\.2\\pm 3\.3\(\-6\.0%\)48\.6±4\.748\.6\\pm 4\.7\(\-10\.6%\)ImageNet\-R95\.0±1\.4\\mathbf\{95\.0\\pm 1\.4\}94\.9±1\.394\.9\\pm 1\.395\.3±1\.295\.3\\pm 1\.22\.2±0\.3\\mathbf\{2\.2\\pm 0\.3\}5\.4±1\.25\.4\\pm 1\.2\(\-59\.6%\)3\.6±0\.53\.6\\pm 0\.5\(\-38\.2%\)EuroSAT95\.1±1\.4\\mathbf\{95\.1\\pm 1\.4\}95\.3±1\.395\.3\\pm 1\.395\.3±1\.395\.3\\pm 1\.35\.8±0\.2\\mathbf\{5\.8\\pm 0\.2\}5\.8±0\.35\.8\\pm 0\.3\(\-0\.5%\)6\.6±0\.26\.6\\pm 0\.2\(\-12\.4%\)SigLIP2\-L/16CIFAR\-1095\.7±1\.7\\mathbf\{95\.7\\pm 1\.7\}96\.0±1\.296\.0\\pm 1\.295\.5±1\.795\.5\\pm 1\.71\.1±0\.0\\mathbf\{1\.1\\pm 0\.0\}1\.1±0\.01\.1\\pm 0\.0\(\-1\.5%\)1\.1±0\.01\.1\\pm 0\.0\(\-3\.2%\)CIFAR\-10095\.3±1\.8\\mathbf\{95\.3\\pm 1\.8\}95\.5±1\.895\.5\\pm 1\.895\.4±1\.895\.4\\pm 1\.84\.7±0\.5\\mathbf\{4\.7\\pm 0\.5\}6\.3±1\.16\.3\\pm 1\.1\(\-25\.9%\)5\.0±0\.85\.0\\pm 0\.8\(\-6\.3%\)ImageNet94\.9±1\.3\\mathbf\{94\.9\\pm 1\.3\}95\.3±1\.395\.3\\pm 1\.395\.0±1\.395\.0\\pm 1\.39\.2±1\.6\\mathbf\{9\.2\\pm 1\.6\}9\.2±1\.59\.2\\pm 1\.5\(\-0\.6%\)12\.3±2\.312\.3\\pm 2\.3\(\-25\.3%\)ImageNet\-A95\.0±1\.3\\mathbf\{95\.0\\pm 1\.3\}95\.3±1\.295\.3\\pm 1\.295\.3±1\.395\.3\\pm 1\.32\.2±0\.2\\mathbf\{2\.2\\pm 0\.2\}2\.3±0\.22\.3\\pm 0\.2\(\-6\.8%\)2\.5±0\.42\.5\\pm 0\.4\(\-14\.4%\)ImageNet\-R95\.1±1\.3\\mathbf\{95\.1\\pm 1\.3\}95\.4±1\.095\.4\\pm 1\.095\.4±0\.995\.4\\pm 0\.91\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.2±0\.11\.2\\pm 0\.1\(\-14\.0%\)1\.2±0\.11\.2\\pm 0\.1\(\-12\.0%\)EuroSAT95\.0±1\.4\\mathbf\{95\.0\\pm 1\.4\}94\.9±1\.494\.9\\pm 1\.495\.2±1\.395\.2\\pm 1\.34\.7±0\.2\\mathbf\{4\.7\\pm 0\.2\}5\.1±0\.15\.1\\pm 0\.1\(\-7\.7%\)5\.1±0\.25\.1\\pm 0\.2\(\-7\.0%\)SigLIP2\-SO400M/16CIFAR\-1095\.2±1\.9\\mathbf\{95\.2\\pm 1\.9\}97\.0±0\.997\.0\\pm 0\.996\.0±1\.396\.0\\pm 1\.31\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.0±0\.01\.0\\pm 0\.0\(\-3\.3%\)1\.0±0\.01\.0\\pm 0\.0\(\-3\.9%\)CIFAR\-10095\.3±1\.8\\mathbf\{95\.3\\pm 1\.8\}95\.4±2\.095\.4\\pm 2\.095\.3±2\.195\.3\\pm 2\.12\.6±0\.4\\mathbf\{2\.6\\pm 0\.4\}3\.6±0\.63\.6\\pm 0\.6\(\-28\.6%\)3\.0±0\.53\.0\\pm 0\.5\(\-14\.7%\)ImageNet94\.9±1\.3\\mathbf\{94\.9\\pm 1\.3\}95\.3±1\.495\.3\\pm 1\.495\.3±1\.395\.3\\pm 1\.38\.8±1\.1\\mathbf\{8\.8\\pm 1\.1\}9\.4±1\.79\.4\\pm 1\.7\(\-6\.6%\)10\.9±2\.110\.9\\pm 2\.1\(\-20\.1%\)ImageNet\-A94\.9±1\.3\\mathbf\{94\.9\\pm 1\.3\}95\.0±1\.495\.0\\pm 1\.495\.4±1\.295\.4\\pm 1\.22\.0±0\.2\\mathbf\{2\.0\\pm 0\.2\}2\.0±0\.32\.0\\pm 0\.3\(\-1\.6%\)2\.1±0\.32\.1\\pm 0\.3\(\-7\.5%\)ImageNet\-R94\.8±1\.4\\mathbf\{94\.8\\pm 1\.4\}95\.4±1\.195\.4\\pm 1\.195\.7±1\.095\.7\\pm 1\.01\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.3±0\.11\.3\\pm 0\.1\(\-19\.6%\)1\.2±0\.21\.2\\pm 0\.2\(\-18\.9%\)EuroSAT94\.6±1\.4\\mathbf\{94\.6\\pm 1\.4\}95\.4±1\.295\.4\\pm 1\.295\.3±1\.295\.3\\pm 1\.24\.6±0\.2\\mathbf\{4\.6\\pm 0\.2\}5\.1±0\.25\.1\\pm 0\.2\(\-10\.0%\)5\.4±0\.25\.4\\pm 0\.2\(\-15\.2%\)MobileCLIP2\-S2CIFAR\-1095\.1±2\.2\\mathbf\{95\.1\\pm 2\.2\}97\.6±0\.797\.6\\pm 0\.797\.8±0\.697\.8\\pm 0\.61\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.0±0\.01\.0\\pm 0\.0\(\-3\.8%\)1\.0±0\.01\.0\\pm 0\.0\(\-3\.8%\)CIFAR\-10095\.4±1\.9\\mathbf\{95\.4\\pm 1\.9\}95\.7±1\.895\.7\\pm 1\.895\.6±1\.995\.6\\pm 1\.91\.6±0\.1\\mathbf\{1\.6\\pm 0\.1\}1\.7±0\.11\.7\\pm 0\.1\(\-4\.4%\)1\.6±0\.21\.6\\pm 0\.2\(\-2\.6%\)ImageNet95\.1±1\.2\\mathbf\{95\.1\\pm 1\.2\}95\.1±1\.395\.1\\pm 1\.395\.3±1\.495\.3\\pm 1\.44\.1±0\.6\\mathbf\{4\.1\\pm 0\.6\}4\.3±0\.64\.3\\pm 0\.6\(\-2\.9%\)4\.4±0\.74\.4\\pm 0\.7\(\-5\.0%\)ImageNet\-A95\.3±1\.3\\mathbf\{95\.3\\pm 1\.3\}95\.4±1\.395\.4\\pm 1\.395\.0±1\.295\.0\\pm 1\.221\.9±2\.0\\mathbf\{21\.9\\pm 2\.0\}22\.1±2\.322\.1\\pm 2\.3\(\-0\.8%\)22\.0±2\.522\.0\\pm 2\.5\(\-0\.6%\)ImageNet\-R95\.3±1\.3\\mathbf\{95\.3\\pm 1\.3\}95\.3±1\.495\.3\\pm 1\.495\.1±1\.195\.1\\pm 1\.11\.9±0\.2\\mathbf\{1\.9\\pm 0\.2\}1\.9±0\.31\.9\\pm 0\.3\(\-2\.1%\)1\.9±0\.21\.9\\pm 0\.2\(\-1\.6%\)EuroSAT95\.1±1\.3\\mathbf\{95\.1\\pm 1\.3\}95\.2±1\.495\.2\\pm 1\.495\.3±1\.295\.3\\pm 1\.24\.8±0\.2\\mathbf\{4\.8\\pm 0\.2\}4\.8±0\.24\.8\\pm 0\.2\(\-1\.4%\)5\.7±0\.25\.7\\pm 0\.2\(\-16\.4%\)MobileCLIP2\-S3CIFAR\-1095\.7±1\.9\\mathbf\{95\.7\\pm 1\.9\}97\.6±0\.797\.6\\pm 0\.797\.6±0\.797\.6\\pm 0\.71\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.0±0\.01\.0\\pm 0\.0\(\-3\.1%\)1\.0±0\.01\.0\\pm 0\.0\(\-3\.1%\)CIFAR\-10095\.4±1\.8\\mathbf\{95\.4\\pm 1\.8\}95\.5±1\.895\.5\\pm 1\.895\.7±1\.895\.7\\pm 1\.81\.6±0\.2\\mathbf\{1\.6\\pm 0\.2\}1\.6±0\.21\.6\\pm 0\.2\(\-0\.1%\)1\.6±0\.21\.6\\pm 0\.2\(\-0\.4%\)ImageNet95\.1±1\.2\\mathbf\{95\.1\\pm 1\.2\}95\.4±1\.495\.4\\pm 1\.495\.2±1\.395\.2\\pm 1\.33\.2±0\.4\\mathbf\{3\.2\\pm 0\.4\}3\.3±0\.43\.3\\pm 0\.4\(\-2\.7%\)3\.3±0\.43\.3\\pm 0\.4\(\-4\.4%\)ImageNet\-A95\.0±1\.4\\mathbf\{95\.0\\pm 1\.4\}95\.1±1\.395\.1\\pm 1\.395\.0±1\.295\.0\\pm 1\.212\.0±1\.1\\mathbf\{12\.0\\pm 1\.1\}13\.3±1\.513\.3\\pm 1\.5\(\-9\.3%\)13\.8±1\.113\.8\\pm 1\.1\(\-12\.8%\)ImageNet\-R95\.5±1\.3\\mathbf\{95\.5\\pm 1\.3\}95\.4±1\.195\.4\\pm 1\.195\.0±1\.495\.0\\pm 1\.41\.3±0\.1\\mathbf\{1\.3\\pm 0\.1\}1\.4±0\.11\.4\\pm 0\.1\(\-3\.2%\)1\.4±0\.11\.4\\pm 0\.1\(\-2\.4%\)EuroSAT95\.1±1\.2\\mathbf\{95\.1\\pm 1\.2\}95\.2±1\.395\.2\\pm 1\.395\.2±1\.195\.2\\pm 1\.14\.6±0\.2\\mathbf\{4\.6\\pm 0\.2\}4\.8±0\.24\.8\\pm 0\.2\(\-4\.0%\)5\.3±0\.25\.3\\pm 0\.2\(\-13\.7%\)MobileCLIP2\-S4CIFAR\-1095\.6±1\.8\\mathbf\{95\.6\\pm 1\.8\}97\.4±0\.797\.4\\pm 0\.797\.7±0\.797\.7\\pm 0\.71\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.0±0\.01\.0\\pm 0\.0\(\-3\.6%\)1\.0±0\.01\.0\\pm 0\.0\(\-3\.6%\)CIFAR\-10095\.5±2\.0\\mathbf\{95\.5\\pm 2\.0\}95\.7±1\.895\.7\\pm 1\.895\.5±1\.795\.5\\pm 1\.71\.3±0\.1\\mathbf\{1\.3\\pm 0\.1\}1\.3±0\.11\.3\\pm 0\.1\(\-1\.9%\)1\.3±0\.11\.3\\pm 0\.1\(\-0\.4%\)ImageNet95\.1±1\.1\\mathbf\{95\.1\\pm 1\.1\}95\.0±1\.495\.0\\pm 1\.495\.1±1\.495\.1\\pm 1\.43\.1±0\.5\\mathbf\{3\.1\\pm 0\.5\}3\.1±0\.33\.1\\pm 0\.3\(\-0\.7%\)3\.1±0\.73\.1\\pm 0\.7\(\-2\.3%\)ImageNet\-A95\.1±1\.3\\mathbf\{95\.1\\pm 1\.3\}95\.1±1\.395\.1\\pm 1\.394\.9±1\.294\.9\\pm 1\.28\.7±0\.8\\mathbf\{8\.7\\pm 0\.8\}9\.4±1\.29\.4\\pm 1\.2\(\-7\.4%\)9\.2±1\.29\.2\\pm 1\.2\(\-4\.7%\)ImageNet\-R95\.2±1\.4\\mathbf\{95\.2\\pm 1\.4\}95\.1±1\.295\.1\\pm 1\.295\.2±1\.395\.2\\pm 1\.31\.2±0\.1\\mathbf\{1\.2\\pm 0\.1\}1\.3±0\.11\.3\\pm 0\.1\(\-1\.6%\)1\.3±0\.11\.3\\pm 0\.1\(\-1\.4%\)EuroSAT95\.0±1\.5\\mathbf\{95\.0\\pm 1\.5\}95\.2±1\.295\.2\\pm 1\.295\.1±1\.595\.1\\pm 1\.54\.0±0\.1\\mathbf\{4\.0\\pm 0\.1\}4\.3±0\.24\.3\\pm 0\.2\(\-4\.8%\)4\.5±0\.34\.5\\pm 0\.3\(\-10\.9%\)MobileCLIP2\-BCIFAR\-1095\.6±1\.9\\mathbf\{95\.6\\pm 1\.9\}97\.5±0\.797\.5\\pm 0\.797\.1±0\.797\.1\\pm 0\.71\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.0±0\.01\.0\\pm 0\.0\(\-3\.5%\)1\.0±0\.01\.0\\pm 0\.0\(\-3\.5%\)CIFAR\-10095\.6±1\.8\\mathbf\{95\.6\\pm 1\.8\}95\.6±2\.095\.6\\pm 2\.096\.0±1\.796\.0\\pm 1\.71\.5±0\.1\\mathbf\{1\.5\\pm 0\.1\}1\.5±0\.11\.5\\pm 0\.1\(\-0\.2%\)1\.6±0\.21\.6\\pm 0\.2\(\-2\.0%\)ImageNet95\.3±1\.1\\mathbf\{95\.3\\pm 1\.1\}95\.3±1\.395\.3\\pm 1\.395\.2±1\.395\.2\\pm 1\.33\.5±0\.5\\mathbf\{3\.5\\pm 0\.5\}3\.5±0\.73\.5\\pm 0\.7\(\-0\.6%\)3\.6±0\.83\.6\\pm 0\.8\(\-3\.4%\)ImageNet\-A95\.2±1\.5\\mathbf\{95\.2\\pm 1\.5\}95\.0±1\.595\.0\\pm 1\.595\.3±1\.395\.3\\pm 1\.316\.5±2\.7\\mathbf\{16\.5\\pm 2\.7\}17\.0±1\.617\.0\\pm 1\.6\(\-3\.2%\)18\.6±1\.618\.6\\pm 1\.6\(\-11\.3%\)ImageNet\-R94\.6±1\.4\\mathbf\{94\.6\\pm 1\.4\}95\.0±1\.295\.0\\pm 1\.295\.2±1\.395\.2\\pm 1\.31\.5±0\.1\\mathbf\{1\.5\\pm 0\.1\}1\.5±0\.11\.5\\pm 0\.1\(\-6\.1%\)1\.5±0\.11\.5\\pm 0\.1\(\-1\.9%\)EuroSAT95\.1±1\.4\\mathbf\{95\.1\\pm 1\.4\}95\.0±1\.395\.0\\pm 1\.395\.5±1\.495\.5\\pm 1\.44\.1±0\.2\\mathbf\{4\.1\\pm 0\.2\}4\.4±0\.24\.4\\pm 0\.2\(\-6\.6%\)4\.9±0\.24\.9\\pm 0\.2\(\-16\.4%\)Table 8:Full coverage and set\-size results atα=0\.05\\alpha=0\.05\(Regular 30, continued\)\.Table 9:Full coverage and set\-size results atα=0\.05\\alpha=0\.05\(Top\-20 rephrased\)\. Set\-size parentheses report the relative change ofrr\-value versus the corresponding CP baseline, computed from the unrounded set\-size means\.ModelDatasetCoverage \(%\)Set sizeCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}CPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}CLIP\-B/16CIFAR\-1095\.6±1\.9\\mathbf\{95\.6\\pm 1\.9\}95\.6±1\.495\.6\\pm 1\.495\.5±1\.695\.5\\pm 1\.61\.2±0\.1\\mathbf\{1\.2\\pm 0\.1\}1\.3±0\.11\.3\\pm 0\.1\(\-3\.9%\)1\.4±0\.11\.4\\pm 0\.1\(\-10\.2%\)CIFAR\-10095\.0±1\.8\\mathbf\{95\.0\\pm 1\.8\}95\.8±1\.895\.8\\pm 1\.895\.4±1\.795\.4\\pm 1\.77\.6±0\.7\\mathbf\{7\.6\\pm 0\.7\}8\.2±0\.98\.2\\pm 0\.9\(\-7\.4%\)8\.3±1\.18\.3\\pm 1\.1\(\-8\.5%\)ImageNet95\.1±1\.5\\mathbf\{95\.1\\pm 1\.5\}95\.3±1\.295\.3\\pm 1\.295\.1±1\.495\.1\\pm 1\.48\.5±1\.2\\mathbf\{8\.5\\pm 1\.2\}8\.7±1\.28\.7\\pm 1\.2\(\-1\.5%\)10\.2±2\.410\.2\\pm 2\.4\(\-16\.5%\)ImageNet\-A95\.1±1\.3\\mathbf\{95\.1\\pm 1\.3\}95\.5±1\.495\.5\\pm 1\.495\.2±1\.495\.2\\pm 1\.421\.4±2\.4\\mathbf\{21\.4\\pm 2\.4\}21\.6±2\.821\.6\\pm 2\.8\(\-0\.8%\)24\.7±2\.724\.7\\pm 2\.7\(\-13\.3%\)ImageNet\-R94\.8±1\.4\\mathbf\{94\.8\\pm 1\.4\}95\.2±1\.295\.2\\pm 1\.295\.2±1\.295\.2\\pm 1\.27\.2±1\.5\\mathbf\{7\.2\\pm 1\.5\}7\.6±1\.57\.6\\pm 1\.5\(\-6\.5%\)8\.8±1\.68\.8\\pm 1\.6\(\-18\.6%\)EuroSAT94\.4±1\.6\\mathbf\{94\.4\\pm 1\.6\}95\.2±1\.495\.2\\pm 1\.495\.3±1\.395\.3\\pm 1\.35\.5±0\.3\\mathbf\{5\.5\\pm 0\.3\}5\.7±0\.25\.7\\pm 0\.2\(\-3\.4%\)6\.0±0\.26\.0\\pm 0\.2\(\-7\.7%\)CLIP\-B/32CIFAR\-1095\.2±2\.2\\mathbf\{95\.2\\pm 2\.2\}95\.8±1\.695\.8\\pm 1\.695\.8±1\.595\.8\\pm 1\.51\.2±0\.1\\mathbf\{1\.2\\pm 0\.1\}1\.2±0\.11\.2\\pm 0\.1\(\-3\.1%\)1\.2±0\.11\.2\\pm 0\.1\(\-4\.1%\)CIFAR\-10095\.5±1\.6\\mathbf\{95\.5\\pm 1\.6\}95\.3±2\.195\.3\\pm 2\.195\.7±1\.695\.7\\pm 1\.67\.4±0\.8\\mathbf\{7\.4\\pm 0\.8\}7\.7±1\.27\.7\\pm 1\.2\(\-4\.4%\)9\.3±1\.89\.3\\pm 1\.8\(\-20\.9%\)ImageNet95\.0±1\.5\\mathbf\{95\.0\\pm 1\.5\}95\.3±1\.295\.3\\pm 1\.295\.0±1\.695\.0\\pm 1\.612\.5±1\.8\\mathbf\{12\.5\\pm 1\.8\}12\.6±1\.512\.6\\pm 1\.5\(\-0\.3%\)14\.1±2\.414\.1\\pm 2\.4\(\-11\.2%\)ImageNet\-A95\.2±1\.4\\mathbf\{95\.2\\pm 1\.4\}95\.2±1\.495\.2\\pm 1\.495\.0±1\.295\.0\\pm 1\.247\.2±3\.8\\mathbf\{47\.2\\pm 3\.8\}49\.4±4\.249\.4\\pm 4\.2\(\-4\.5%\)60\.4±7\.360\.4\\pm 7\.3\(\-21\.9%\)ImageNet\-R94\.7±1\.4\\mathbf\{94\.7\\pm 1\.4\}95\.1±1\.395\.1\\pm 1\.395\.3±1\.395\.3\\pm 1\.312\.6±1\.8\\mathbf\{12\.6\\pm 1\.8\}14\.3±2\.714\.3\\pm 2\.7\(\-11\.8%\)15\.3±2\.515\.3\\pm 2\.5\(\-17\.3%\)EuroSAT94\.7±1\.6\\mathbf\{94\.7\\pm 1\.6\}95\.5±1\.295\.5\\pm 1\.295\.2±1\.495\.2\\pm 1\.45\.9±0\.4\\mathbf\{5\.9\\pm 0\.4\}6\.2±0\.36\.2\\pm 0\.3\(\-4\.7%\)7\.0±0\.37\.0\\pm 0\.3\(\-16\.1%\)SigLIP2\-B/16CIFAR\-1095\.5±1\.7\\mathbf\{95\.5\\pm 1\.7\}95\.8±1\.695\.8\\pm 1\.696\.3±1\.596\.3\\pm 1\.51\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.1±0\.01\.1\\pm 0\.0\(\-1\.9%\)1\.0±0\.01\.0\\pm 0\.0\(\-1\.0%\)CIFAR\-10095\.4±2\.0\\mathbf\{95\.4\\pm 2\.0\}95\.6±2\.095\.6\\pm 2\.095\.6±1\.795\.6\\pm 1\.74\.7±0\.8\\mathbf\{4\.7\\pm 0\.8\}5\.2±1\.05\.2\\pm 1\.0\(\-8\.4%\)5\.0±1\.05\.0\\pm 1\.0\(\-5\.7%\)ImageNet94\.9±1\.3\\mathbf\{94\.9\\pm 1\.3\}95\.0±1\.295\.0\\pm 1\.295\.4±1\.495\.4\\pm 1\.47\.3±1\.6\\mathbf\{7\.3\\pm 1\.6\}8\.3±1\.38\.3\\pm 1\.3\(\-13\.0%\)10\.7±2\.810\.7\\pm 2\.8\(\-32\.3%\)ImageNet\-A95\.0±1\.3\\mathbf\{95\.0\\pm 1\.3\}95\.3±1\.295\.3\\pm 1\.295\.1±1\.495\.1\\pm 1\.45\.2±0\.6\\mathbf\{5\.2\\pm 0\.6\}5\.6±0\.55\.6\\pm 0\.5\(\-5\.9%\)6\.9±1\.06\.9\\pm 1\.0\(\-23\.7%\)ImageNet\-R95\.2±1\.6\\mathbf\{95\.2\\pm 1\.6\}95\.4±1\.295\.4\\pm 1\.295\.3±1\.295\.3\\pm 1\.21\.1±0\.0\\mathbf\{1\.1\\pm 0\.0\}2\.1±0\.52\.1\\pm 0\.5\(\-46\.2%\)1\.7±0\.21\.7\\pm 0\.2\(\-33\.7%\)EuroSAT94\.4±1\.5\\mathbf\{94\.4\\pm 1\.5\}95\.3±1\.295\.3\\pm 1\.295\.3±1\.195\.3\\pm 1\.14\.8±0\.2\\mathbf\{4\.8\\pm 0\.2\}5\.0±0\.25\.0\\pm 0\.2\(\-4\.5%\)6\.2±0\.26\.2\\pm 0\.2\(\-23\.9%\)SigLIP2\-B/32CIFAR\-1095\.5±1\.7\\mathbf\{95\.5\\pm 1\.7\}95\.7±1\.795\.7\\pm 1\.796\.0±1\.896\.0\\pm 1\.81\.1±0\.0\\mathbf\{1\.1\\pm 0\.0\}1\.1±0\.01\.1\\pm 0\.0\(\-0\.9%\)1\.1±0\.01\.1\\pm 0\.0\(\-4\.4%\)CIFAR\-10095\.4±2\.0\\mathbf\{95\.4\\pm 2\.0\}95\.6±1\.795\.6\\pm 1\.795\.7±1\.595\.7\\pm 1\.55\.1±0\.8\\mathbf\{5\.1\\pm 0\.8\}6\.1±1\.26\.1\\pm 1\.2\(\-17\.3%\)5\.4±1\.05\.4\\pm 1\.0\(\-6\.6%\)ImageNet94\.9±1\.3\\mathbf\{94\.9\\pm 1\.3\}95\.1±1\.495\.1\\pm 1\.495\.1±1\.295\.1\\pm 1\.213\.0±1\.6\\mathbf\{13\.0\\pm 1\.6\}14\.4±2\.614\.4\\pm 2\.6\(\-9\.9%\)19\.6±2\.819\.6\\pm 2\.8\(\-33\.5%\)ImageNet\-A94\.9±1\.4\\mathbf\{94\.9\\pm 1\.4\}94\.9±1\.494\.9\\pm 1\.495\.0±1\.495\.0\\pm 1\.444\.6±3\.6\\mathbf\{44\.6\\pm 3\.6\}46\.8±3\.146\.8\\pm 3\.1\(\-4\.7%\)48\.6±4\.748\.6\\pm 4\.7\(\-8\.3%\)ImageNet\-R95\.0±1\.4\\mathbf\{95\.0\\pm 1\.4\}94\.8±1\.494\.8\\pm 1\.495\.3±1\.295\.3\\pm 1\.22\.2±0\.3\\mathbf\{2\.2\\pm 0\.3\}5\.2±1\.25\.2\\pm 1\.2\(\-57\.8%\)3\.3±0\.53\.3\\pm 0\.5\(\-32\.6%\)EuroSAT94\.9±1\.3\\mathbf\{94\.9\\pm 1\.3\}95\.4±1\.395\.4\\pm 1\.395\.3±1\.395\.3\\pm 1\.35\.5±0\.2\\mathbf\{5\.5\\pm 0\.2\}5\.7±0\.25\.7\\pm 0\.2\(\-3\.7%\)6\.6±0\.26\.6\\pm 0\.2\(\-17\.5%\)SigLIP2\-L/16CIFAR\-1095\.6±1\.7\\mathbf\{95\.6\\pm 1\.7\}95\.9±1\.595\.9\\pm 1\.595\.5±1\.795\.5\\pm 1\.71\.1±0\.0\\mathbf\{1\.1\\pm 0\.0\}1\.1±0\.01\.1\\pm 0\.0\(\-1\.0%\)1\.1±0\.01\.1\\pm 0\.0\(\-4\.5%\)CIFAR\-10095\.2±1\.9\\mathbf\{95\.2\\pm 1\.9\}95\.5±1\.995\.5\\pm 1\.995\.4±1\.895\.4\\pm 1\.84\.7±0\.5\\mathbf\{4\.7\\pm 0\.5\}6\.0±1\.26\.0\\pm 1\.2\(\-21\.6%\)4\.9±0\.84\.9\\pm 0\.8\(\-3\.4%\)ImageNet94\.9±1\.4\\mathbf\{94\.9\\pm 1\.4\}95\.3±1\.295\.3\\pm 1\.295\.0±1\.395\.0\\pm 1\.38\.8±1\.6\\mathbf\{8\.8\\pm 1\.6\}9\.8±1\.99\.8\\pm 1\.9\(\-10\.6%\)12\.3±2\.312\.3\\pm 2\.3\(\-28\.5%\)ImageNet\-A94\.8±1\.4\\mathbf\{94\.8\\pm 1\.4\}95\.3±1\.295\.3\\pm 1\.295\.3±1\.395\.3\\pm 1\.32\.1±0\.2\\mathbf\{2\.1\\pm 0\.2\}2\.2±0\.22\.2\\pm 0\.2\(\-5\.7%\)2\.5±0\.42\.5\\pm 0\.4\(\-19\.3%\)ImageNet\-R95\.0±1\.5\\mathbf\{95\.0\\pm 1\.5\}95\.5±1\.095\.5\\pm 1\.095\.4±0\.995\.4\\pm 0\.91\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.2±0\.11\.2\\pm 0\.1\(\-13\.0%\)1\.1±0\.11\.1\\pm 0\.1\(\-11\.0%\)EuroSAT94\.5±1\.4\\mathbf\{94\.5\\pm 1\.4\}94\.8±1\.594\.8\\pm 1\.595\.2±1\.395\.2\\pm 1\.34\.7±0\.2\\mathbf\{4\.7\\pm 0\.2\}5\.0±0\.25\.0\\pm 0\.2\(\-5\.1%\)4\.8±0\.24\.8\\pm 0\.2\(\-2\.1%\)SigLIP2\-SO400M/16CIFAR\-1095\.2±2\.0\\mathbf\{95\.2\\pm 2\.0\}96\.8±1\.096\.8\\pm 1\.096\.0±1\.396\.0\\pm 1\.31\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.0±0\.01\.0\\pm 0\.0\(\-3\.0%\)1\.0±0\.01\.0\\pm 0\.0\(\-3\.9%\)CIFAR\-10095\.1±1\.9\\mathbf\{95\.1\\pm 1\.9\}95\.4±2\.195\.4\\pm 2\.195\.3±2\.195\.3\\pm 2\.12\.5±0\.3\\mathbf\{2\.5\\pm 0\.3\}3\.3±0\.43\.3\\pm 0\.4\(\-23\.9%\)3\.0±0\.53\.0\\pm 0\.5\(\-17\.4%\)ImageNet94\.8±1\.3\\mathbf\{94\.8\\pm 1\.3\}95\.3±1\.295\.3\\pm 1\.295\.3±1\.395\.3\\pm 1\.38\.8±1\.2\\mathbf\{8\.8\\pm 1\.2\}9\.5±1\.39\.5\\pm 1\.3\(\-7\.2%\)10\.9±2\.110\.9\\pm 2\.1\(\-19\.7%\)ImageNet\-A94\.8±1\.3\\mathbf\{94\.8\\pm 1\.3\}95\.0±1\.495\.0\\pm 1\.495\.4±1\.295\.4\\pm 1\.21\.9±0\.2\\mathbf\{1\.9\\pm 0\.2\}1\.9±0\.31\.9\\pm 0\.3\(\-3\.1%\)2\.1±0\.32\.1\\pm 0\.3\(\-12\.2%\)ImageNet\-R94\.8±1\.5\\mathbf\{94\.8\\pm 1\.5\}95\.4±1\.195\.4\\pm 1\.195\.7±1\.095\.7\\pm 1\.01\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.2±0\.11\.2\\pm 0\.1\(\-18\.2%\)1\.2±0\.11\.2\\pm 0\.1\(\-15\.1%\)EuroSAT94\.4±1\.6\\mathbf\{94\.4\\pm 1\.6\}95\.4±1\.395\.4\\pm 1\.395\.3±1\.295\.3\\pm 1\.24\.5±0\.3\\mathbf\{4\.5\\pm 0\.3\}4\.6±0\.24\.6\\pm 0\.2\(\-2\.2%\)5\.4±0\.25\.4\\pm 0\.2\(\-17\.8%\)MobileCLIP2\-S2CIFAR\-1095\.1±2\.2\\mathbf\{95\.1\\pm 2\.2\}97\.8±0\.797\.8\\pm 0\.797\.8±0\.697\.8\\pm 0\.61\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.0±0\.01\.0\\pm 0\.0\(\-4\.0%\)1\.0±0\.01\.0\\pm 0\.0\(\-4\.0%\)CIFAR\-10095\.4±1\.9\\mathbf\{95\.4\\pm 1\.9\}95\.7±1\.895\.7\\pm 1\.895\.6±1\.995\.6\\pm 1\.91\.6±0\.1\\mathbf\{1\.6\\pm 0\.1\}1\.6±0\.11\.6\\pm 0\.1\(\-0\.9%\)1\.6±0\.21\.6\\pm 0\.2\(\-1\.5%\)ImageNet95\.0±1\.2\\mathbf\{95\.0\\pm 1\.2\}95\.1±1\.495\.1\\pm 1\.495\.3±1\.495\.3\\pm 1\.44\.0±0\.5\\mathbf\{4\.0\\pm 0\.5\}4\.4±0\.64\.4\\pm 0\.6\(\-8\.2%\)4\.4±0\.74\.4\\pm 0\.7\(\-8\.0%\)ImageNet\-A95\.2±1\.2\\mathbf\{95\.2\\pm 1\.2\}95\.3±1\.395\.3\\pm 1\.395\.0±1\.295\.0\\pm 1\.220\.8±2\.1\\mathbf\{20\.8\\pm 2\.1\}22\.1±2\.322\.1\\pm 2\.3\(\-6\.2%\)22\.0±2\.522\.0\\pm 2\.5\(\-5\.7%\)ImageNet\-R95\.2±1\.4\\mathbf\{95\.2\\pm 1\.4\}95\.3±1\.495\.3\\pm 1\.495\.0±1\.495\.0\\pm 1\.41\.8±0\.2\\mathbf\{1\.8\\pm 0\.2\}2\.0±0\.32\.0\\pm 0\.3\(\-6\.4%\)1\.9±0\.21\.9\\pm 0\.2\(\-0\.6%\)EuroSAT95\.1±1\.3\\mathbf\{95\.1\\pm 1\.3\}95\.2±1\.495\.2\\pm 1\.495\.3±1\.295\.3\\pm 1\.24\.6±0\.2\\mathbf\{4\.6\\pm 0\.2\}4\.8±0\.24\.8\\pm 0\.2\(\-3\.4%\)5\.7±0\.25\.7\\pm 0\.2\(\-19\.1%\)MobileCLIP2\-S3CIFAR\-1095\.7±1\.8\\mathbf\{95\.7\\pm 1\.8\}97\.8±0\.797\.8\\pm 0\.797\.6±0\.797\.6\\pm 0\.71\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.0±0\.01\.0\\pm 0\.0\(\-3\.4%\)1\.0±0\.01\.0\\pm 0\.0\(\-3\.4%\)CIFAR\-10095\.4±1\.8\\mathbf\{95\.4\\pm 1\.8\}95\.5±1\.895\.5\\pm 1\.895\.7±1\.895\.7\\pm 1\.81\.5±0\.2\\mathbf\{1\.5\\pm 0\.2\}1\.6±0\.21\.6\\pm 0\.2\(\-3\.4%\)1\.6±0\.21\.6\\pm 0\.2\(\-3\.4%\)ImageNet95\.1±1\.2\\mathbf\{95\.1\\pm 1\.2\}95\.4±1\.495\.4\\pm 1\.495\.2±1\.395\.2\\pm 1\.33\.1±0\.4\\mathbf\{3\.1\\pm 0\.4\}3\.2±0\.43\.2\\pm 0\.4\(\-1\.7%\)3\.3±0\.43\.3\\pm 0\.4\(\-6\.1%\)ImageNet\-A95\.0±1\.4\\mathbf\{95\.0\\pm 1\.4\}95\.1±1\.395\.1\\pm 1\.395\.0±1\.295\.0\\pm 1\.212\.6±1\.3\\mathbf\{12\.6\\pm 1\.3\}13\.0±1\.213\.0\\pm 1\.2\(\-3\.6%\)13\.8±1\.113\.8\\pm 1\.1\(\-9\.1%\)ImageNet\-R95\.3±1\.3\\mathbf\{95\.3\\pm 1\.3\}95\.2±1\.295\.2\\pm 1\.295\.2±1\.295\.2\\pm 1\.21\.3±0\.1\\mathbf\{1\.3\\pm 0\.1\}1\.4±0\.11\.4\\pm 0\.1\(\-1\.3%\)1\.4±0\.11\.4\\pm 0\.1\(\-1\.4%\)EuroSAT95\.0±1\.2\\mathbf\{95\.0\\pm 1\.2\}95\.1±1\.395\.1\\pm 1\.395\.2±1\.195\.2\\pm 1\.14\.3±0\.2\\mathbf\{4\.3\\pm 0\.2\}4\.9±0\.24\.9\\pm 0\.2\(\-12\.0%\)5\.3±0\.25\.3\\pm 0\.2\(\-18\.5%\)MobileCLIP2\-S4CIFAR\-1095\.5±1\.9\\mathbf\{95\.5\\pm 1\.9\}97\.5±0\.797\.5\\pm 0\.797\.7±0\.797\.7\\pm 0\.71\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.0±0\.01\.0\\pm 0\.0\(\-3\.9%\)1\.0±0\.01\.0\\pm 0\.0\(\-3\.9%\)CIFAR\-10095\.5±2\.1\\mathbf\{95\.5\\pm 2\.1\}95\.6±1\.995\.6\\pm 1\.995\.5±1\.795\.5\\pm 1\.71\.3±0\.1\\mathbf\{1\.3\\pm 0\.1\}1\.3±0\.11\.3\\pm 0\.1\(\-1\.2%\)1\.3±0\.11\.3\\pm 0\.1\(\-0\.3%\)ImageNet95\.1±1\.2\\mathbf\{95\.1\\pm 1\.2\}95\.0±1\.495\.0\\pm 1\.495\.1±1\.495\.1\\pm 1\.43\.0±0\.4\\mathbf\{3\.0\\pm 0\.4\}3\.0±0\.43\.0\\pm 0\.4\(\-0\.4%\)3\.1±0\.73\.1\\pm 0\.7\(\-4\.0%\)ImageNet\-A94\.9±1\.3\\mathbf\{94\.9\\pm 1\.3\}95\.2±1\.395\.2\\pm 1\.394\.9±1\.294\.9\\pm 1\.28\.5±0\.9\\mathbf\{8\.5\\pm 0\.9\}9\.3±1\.39\.3\\pm 1\.3\(\-9\.1%\)9\.2±1\.29\.2\\pm 1\.2\(\-7\.5%\)ImageNet\-R95\.1±1\.2\\mathbf\{95\.1\\pm 1\.2\}95\.3±1\.295\.3\\pm 1\.295\.2±1\.495\.2\\pm 1\.41\.2±0\.1\\mathbf\{1\.2\\pm 0\.1\}1\.2±0\.11\.2\\pm 0\.1\(\-2\.1%\)1\.2±0\.11\.2\\pm 0\.1\(\-1\.7%\)EuroSAT95\.0±1\.5\\mathbf\{95\.0\\pm 1\.5\}95\.2±1\.295\.2\\pm 1\.295\.1±1\.595\.1\\pm 1\.54\.1±0\.2\\mathbf\{4\.1\\pm 0\.2\}4\.4±0\.24\.4\\pm 0\.2\(\-7\.0%\)4\.5±0\.34\.5\\pm 0\.3\(\-10\.2%\)MobileCLIP2\-BCIFAR\-1095\.5±2\.0\\mathbf\{95\.5\\pm 2\.0\}97\.7±0\.697\.7\\pm 0\.697\.1±0\.797\.1\\pm 0\.71\.0±0\.0\\mathbf\{1\.0\\pm 0\.0\}1\.0±0\.01\.0\\pm 0\.0\(\-3\.9%\)1\.0±0\.01\.0\\pm 0\.0\(\-3\.9%\)CIFAR\-10095\.5±1\.8\\mathbf\{95\.5\\pm 1\.8\}95\.7±2\.095\.7\\pm 2\.096\.0±1\.796\.0\\pm 1\.71\.5±0\.1\\mathbf\{1\.5\\pm 0\.1\}1\.5±0\.11\.5\\pm 0\.1\(\-2\.5%\)1\.6±0\.21\.6\\pm 0\.2\(\-4\.1%\)ImageNet95\.3±1\.1\\mathbf\{95\.3\\pm 1\.1\}95\.3±1\.295\.3\\pm 1\.295\.2±1\.395\.2\\pm 1\.33\.5±0\.7\\mathbf\{3\.5\\pm 0\.7\}3\.5±0\.53\.5\\pm 0\.5\(\-0\.5%\)3\.6±0\.83\.6\\pm 0\.8\(\-4\.2%\)ImageNet\-A95\.1±1\.4\\mathbf\{95\.1\\pm 1\.4\}95\.1±1\.595\.1\\pm 1\.595\.3±1\.395\.3\\pm 1\.316\.6±2\.4\\mathbf\{16\.6\\pm 2\.4\}17\.1±1\.917\.1\\pm 1\.9\(\-3\.2%\)18\.6±1\.618\.6\\pm 1\.6\(\-10\.6%\)ImageNet\-R94\.9±1\.4\\mathbf\{94\.9\\pm 1\.4\}95\.1±1\.595\.1\\pm 1\.595\.2±1\.495\.2\\pm 1\.41\.5±0\.2\\mathbf\{1\.5\\pm 0\.2\}1\.5±0\.21\.5\\pm 0\.2\(\-6\.3%\)1\.5±0\.11\.5\\pm 0\.1\(\-1\.9%\)EuroSAT94\.9±1\.4\\mathbf\{94\.9\\pm 1\.4\}95\.0±1\.395\.0\\pm 1\.395\.5±1\.495\.5\\pm 1\.44\.0±0\.2\\mathbf\{4\.0\\pm 0\.2\}4\.5±0\.24\.5\\pm 0\.2\(\-10\.4%\)4\.9±0\.24\.9\\pm 0\.2\(\-18\.3%\)Table 9:Full coverage and set\-size results atα=0\.05\\alpha=0\.05\(Top\-20 rephrased, continued\)\.

## Appendix FOpen\-ended Response Tasks with LLMs

Motivation\.Applying conformal prediction to open\-ended LLM responses is more difficult than applying it to classification or multiple\-choice tasks\. The output space is large and the correctness of a response is not always binary\. A common practical approach is to use a stronger model as an evaluator and assign each response a quality score\. However, these evaluator scores are often coarse\. For example, if the judge model scores answers on a scale from 0 to 10 with only one decimal place, many calibration examples can receive the same score\. This creates many ties in the conformal ranking and makes it difficult to obtain a fine ordering of responses near the calibration threshold\.

Therr\-value is useful in this setting because it does not rely only on the average evaluator score\. Instead, it also uses the variability of the score across different prompt formulations or evaluation perspectives\. Thus, two responses with the same average score can still receive differentrr\-values if one response is evaluated more stably than the other\. This does not remove the fundamental difficulty of open\-ended conformal prediction, but it provides a more informative ranking when the original score is coarse or tied\.

Dataset preparation and question diversification\.We use TruthfulQA\[[19](https://arxiv.org/html/2605.23189#bib.bib12)\]to study this issue\. For each original questionQQ, we generateppsemantically equivalent rephrased questions\{q1,…,qp\}\\\{q\_\{1\},\\ldots,q\_\{p\}\\\}using GPT\-4o\. To ensure that the rephrased questions preserve the original meaning, we compute the sentence similarity between the original question and each rephrased version using the all\-MiniLM\-L6\-v2 sentence embedding model\. We keep only rephrased questions whose cosine similarity with the original question is above a thresholdτ\\tau\. Each original question and its retained rephrases are associated with the validated ground truth answers provided by TruthfulQA\.

Response generation and selection\.For each question variant, we use LLaMA 3B with temperature sampling to generate multiple candidate responses\. We then compare each generated response with the ground truth answers using embedding\-based cosine similarity\. The selected responseA∗A^\{\*\}is the one with the largest average similarity to the ground truth answers,

A∗=arg⁡maxA⁡1\|G​T\|​∑g∈G​TCosSim​\(embed​\(A\),embed​\(g\)\),A^\{\*\}=\\arg\\max\_\{A\}\\frac\{1\}\{\|GT\|\}\\sum\_\{g\\in GT\}\\mathrm\{CosSim\}\\bigl\(\\mathrm\{embed\}\(A\),\\mathrm\{embed\}\(g\)\\bigr\),whereG​TGTdenotes the set of ground truth answers\. This step gives a representative response for each question while still allowing the generation process to capture variation across sampled answers\.

Multi\-perspective evaluation\.We then evaluate each selected answer using GPT\-4o as a judge model\. For a selected answerA∗A^\{\*\}, we present the original question and its rephrased versions to the judge\. This produces a collection of evaluation scores

\{S0,S1,…,Sp\},\\\{S\_\{0\},S\_\{1\},\\ldots,S\_\{p\}\\\},where each score is on a scale from 0 to 10 and reflects the judged quality of the same answer under a slightly different formulation of the question\. These scores provide a simple way to estimate both the average quality of the answer and the uncertainty of the evaluator score\.

Computing therr\-value\.For each response, we compute the mean evaluator score

μ=1p\+1​∑i=0pSi\\mu=\\frac\{1\}\{p\+1\}\\sum\_\{i=0\}^\{p\}S\_\{i\}and the empirical score variance across rephrased evaluations\. Therr\-value is then computed using the empirical Bayes formulation described in the main text\. In this experiment, the role of variability is to distinguish responses that have the same or nearly the same average score but different levels of evaluation stability\. Responses with high and stable scores receive better rankings, while responses whose high scores are less stable are penalized\.

rr\-value helps with tied calibration scores\.Table[10](https://arxiv.org/html/2605.23189#A6.T10)shows an example from the TruthfulQA experiment\. Many responses receive identical or nearly identical evaluator scores\. For example, several responses have score10\.010\.0and therefore share the same score rank\. Standard score\-based conformal prediction cannot distinguish these responses further\. In contrast, therr\-value uses the estimated score variability to refine the ranking\. Responses with the same score can receive differentrr\-values because their evaluation stability differs\.

This result should be interpreted as a ranking improvement rather than a complete solution to open\-ended conformal prediction\. The conformal guarantee still depends on the calibration setup and exchangeability, and open\-ended response evaluation remains inherently difficult\. Nevertheless, the experiment shows thatrr\-values can provide useful additional resolution when evaluator scores are discrete, coarse, or heavily tied\.

Table 10:rr\-value refines the ranking when evaluator scores are tied or nearly tied\. Smallerrr\-values indicate more stable high\-quality responses\.rr\-valuerr\-value RankScore RankScoreSE0\.0335709116\.010\.00\.7120\.0354204726\.010\.00\.5570\.0359014636\.010\.00\.5300\.0368035546\.010\.01\.3400\.03691267521\.09\.90\.9020\.03697749621\.09\.90\.8400\.03728661721\.09\.90\.7680\.03811422821\.09\.90\.6730\.03829784921\.09\.90\.6570\.039330261021\.09\.91\.255![Refer to caption](https://arxiv.org/html/2605.23189v1/AISTATS/figures/llama_judged.png)Figure 6:Smoothed distribution of judge scores for the selected answerA∗A^\{\*\}and responses generated under temperature sampling for a TruthfulQA prompt\. The score distribution illustrates that evaluator scores can vary across response samples and prompt formulations\.
## Appendix GClosed\-ended Response Tasks with LLMs

#### Comparison with Conformal Language Model

CLM significantly advances open\-ended text generation by clustering sampled candidate responses\. However, our LLM experiments focus on tasks like MMLU, which have a finite/discrete set of candidate answers\. This setting meaningfully differs from open\-ended generation\. We conducted experiments adapting CLM to MMLU and confirmed the difficulties; CLM’s sampling\-based procedure struggles with a small fixed set of responses, leading to an increased risk of returning a null set \(Algorithm 1 in CLM paper\)\. By setting the similarity between options to 0 \(to reduce the risk\), we find that CLM does not produce valid prediction sets\. We acknowledge that methods for open\-ended tasks could be extendable to finite sets but would require significant modifications\.

Table 11:Average conformal set size comparison across models\.ModelCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}CP\{\\rm CP\}LLaMA 3B2\.39\\mathbf\{2\.39\}2\.55​\(−6\.43%\)2\.55\\;\(\-6\.43\\%\)2\.56​\(−7\.15%\)2\.56\\;\(\-7\.15\\%\)Mistral2\.30\\mathbf\{2\.30\}2\.42​\(−5\.49%\)2\.42\\;\(\-5\.49\\%\)2\.47​\(−7\.49%\)2\.47\\;\(\-7\.49\\%\)Phi 3\.52\.17\\mathbf\{2\.17\}2\.33​\(−7\.48%\)2\.33\\;\(\-7\.48\\%\)2\.24​\(−3\.51%\)2\.24\\;\(\-3\.51\\%\)Qwen 7B1\.94\\mathbf\{1\.94\}2\.08​\(−6\.85%\)2\.08\\;\(\-6\.85\\%\)2\.07​\(−6\.44%\)2\.07\\;\(\-6\.44\\%\)Table 12:Set sizes for variousg subjects of MMLU atα=0\.05\\alpha=0\.05\.SubjectCPr​\-value\{\\rm CP\}\_\{r\\text\{\-value\}\}CPavg\{\\rm CP\}\_\{\\text\{avg\}\}computer security3\.6753\.684high school computer science3\.8023\.806college computer science3\.8863\.888machine learning3\.8253\.814formal logic3\.8463\.868high school biology3\.4903\.492anatomy3\.3073\.340clinical knowledge3\.3073\.422college medicine3\.5933\.620professional medicine3\.6133\.615college chemistry3\.7723\.813marketing2\.7542\.865public relations3\.5463\.552management3\.3623\.381business ethics3\.7763\.771professional accounting3\.7663\.758

## Appendix HPrompts

VLM Rephrasing PromptSystem prompt\.You are a creative AI tasked with generating rephrasings of short image descriptions\. Your task is to:1\.Read the description carefully\.2\.Generate\{num\_rephrases\}diverse and natural\-sounding rephrasings of the description\.3\.Keep the core meaning the same\.4\.Return a JSON object with the keyrephrased\_descriptions, containing a list of strings\.User prompt\.Original description:\{input\_image\_description\}GPQA Rephrasing PromptSystem prompt\.You are an expert exam question rephraser\. Your task is to:1\.Read the question and its answer options carefully\.2\.Generate different rephrased versions of the question while maintaining relevance to the answer options\.3\.Return the rephrased versions as a JSON object with the keyrephrased\_questions, containing a list of strings\.4\.Return exactly the requested number of rephrased questions\.User prompt\.Rephrase the following question while considering the answer options\.Question:\{input\_text\}Answer options:\{options\_text\}Number of rephrased questions to return:\{num\_rephrases\}TruthfulQA Evaluation PromptSystem prompt\.You are an expert evaluator assessing how well answers address questions\. Your task is to:1\.Read each question and the provided answer carefully\.2\.Evaluate how directly and completely the answer addresses each question\.3\.Consider factors such as accuracy, relevance, and comprehensiveness\.4\.Assign a score from 0 to 10, with one decimal place, for each question\.5\.Return the scores as a JSON object with ascoresarray\.6\.Use the full range of decimal scores, avoid round numbers, and avoid assigning the same score to every question\.User prompt\.Evaluate how well the following answer addresses each question\. Return only a JSON object with ascoresarray containing scores from 0 to 10 with one decimal place\.Answer to evaluate:\{answer\}Questions:\{formatted\_questions\}
## Appendix IBroader Impact

Positive ImpactOur proposedrr\-value conformal prediction method provides ordered prediction sets with a formal coverage guarantee, which ranks the most likely labels first and can potentially contribute in high\-stakes domains\. For example, in healthcare, clinicians can focus on the top\-ranked diagnoses, which can save critical time and reduce diagnostic error; in finance, investigators can classify transactions by predicted fraud risk, focusing efforts on the highest\-risk cases; and in legal settings, risk assessment tools can list factors in order of importance, making decisions clearer and easier to explain\.

Negative ImpactIf one relies only on the hightest\-ranked labels and ignore the rest, they might miss other valid options further down the list\. In addition, the performance of our method depends on the quality of the data and also how accurate the variability estimation is\. If either is poor, then the statistical guarantee may be violated, or the ranking may be misleading and make the result not trustful\.