Conf-Gen: Conformal Uncertainty Quantification for Generative Models

# Conf-Gen: Conformal Uncertainty Quantification for Generative Models
Source: [https://arxiv.org/html/2605.28920](https://arxiv.org/html/2605.28920)
###### Abstract

Conformal prediction \(CP\) and its extension, conformal risk control \(CRC\), are established frameworks for quantifying uncertainty in supervised machine learning through formal guarantees\. However, recent breakthroughs in artificial intelligence \(AI\) have been driven by unsupervised generative models, such as large language models \(LLMs\) and image generators, which are not directly compatible with CP or CRC\. In this work we introduce*conformal generation*\(Conf\-Gen\), a general framework adapting CRC to generative tasks while relaxing its theoretical assumptions\. Conf\-Gen unifies and generalizes previous attempts to apply CP to LLMs, and extends conformal methodology to entirely new domains\. We demonstrate the flexibility of Conf\-Gen through some novel applications, including obtaining conformal guarantees on: image generators producing non\-memorized images, conversational AI systems having asked enough clarifying questions, and the output of AI agents being correct\.

Generative Models, Uncertainty Quantification, Conformal Methods

## Section 1Introduction

Significant empirical breakthroughs have been achieved in artificial intelligence \(AI\) in recent years, from the advent of large language models\(LLMs; Brown et al\.,[2020](https://arxiv.org/html/2605.28920#bib.bib8); OpenAI,[2024](https://arxiv.org/html/2605.28920#bib.bib33)\), to realistic image generators\(Ramesh et al\.,[2022](https://arxiv.org/html/2605.28920#bib.bib41); Rombach et al\.,[2022](https://arxiv.org/html/2605.28920#bib.bib43)\), agentic systems\(Park et al\.,[2023](https://arxiv.org/html/2605.28920#bib.bib36)\), and more\(Silver et al\.,[2016](https://arxiv.org/html/2605.28920#bib.bib47); Potapenko et al\.,[2021](https://arxiv.org/html/2605.28920#bib.bib38)\)\. Most of these developments have moved away from supervised learning and instead leverage generative models\. Despite these strides, uncertainty quantification \(UQ\) remains a critical challenge in AI, particularly for generative models, thereby hindering their deployment in high\-stakes applications such as healthcare\(Begoli et al\.,[2019](https://arxiv.org/html/2605.28920#bib.bib6)\)and scientific discovery\(Wang et al\.,[2023a](https://arxiv.org/html/2605.28920#bib.bib55)\)\.

Conformal prediction\(CP; Gammerman et al\.,[1998](https://arxiv.org/html/2605.28920#bib.bib12); Vovk et al\.,[1999](https://arxiv.org/html/2605.28920#bib.bib53); Saunders et al\.,[1999](https://arxiv.org/html/2605.28920#bib.bib46); Gammerman & Vovk,[2007](https://arxiv.org/html/2605.28920#bib.bib11)\)is a principled approach for UQ in supervised learning\. CP quantifies uncertainty by using featuresXXto construct a set𝒞​\(X\)\\mathcal\{C\}\(X\)of predictions guaranteed to contain the ground truthYGTY\_\{\\text\{GT\}\}with a user\-specified probability\. Conformal risk control\(CRC; Angelopoulos et al\.,[2024b](https://arxiv.org/html/2605.28920#bib.bib4)\)extends CP by lower\-bounding the expectation of a more general utility function,U​\(𝒞​\(X\),YGT\)U\(\\mathcal\{C\}\(X\),Y\_\{\\text\{GT\}\}\), of which CP is a special case\. Because conformal guarantees are distribution\-free, they are particularly appealing as an approach for UQ capable of handling the complex distributions represented by generative models\. Yet, despite its generality, CRC is not directly applicable to generative tasks\. As a consequence, work leveraging conformal ideas in the context of generative models tends to be highly task\-specific\.

![Refer to caption](https://arxiv.org/html/2605.28920v1/results/main/confgen.png)Figure 1:Simplified illustration of our method\. The selection function𝐂λ​\(x,𝐲\)\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)processes an inputxxand a sequence of generations𝐲\\mathbf\{y\}to yield an output that becomes increasingly conservative asλ\\lambdaincreases\. Conf\-Gen identifies a valueλ^\\hat\{\\lambda\}that formally ensures the output of𝐂λ^\\mathbf\{C\}\_\{\\hat\{\\lambda\}\}satisfies a target*admissibility*requirement in expectation\.Top: when𝐲\\mathbf\{y\}consists of*i\.i\.d\.*responses to a queryxx, admissibility ensures the output contains at least one correct answer\.Bottom: when the images in𝐲\\mathbf\{y\}are generated by progressively modifying the promptxx, admissibility ensures the outputted image is not memorized\.In this work we make the following contributions:

1. \(a\)Framework\.We propose*conformal generation*\(Conf\-Gen, depicted in[Figure 1](https://arxiv.org/html/2605.28920#S1.F1)\), a framework extending CRC for rigorous UQ in generative modelling\. Conf\-Gen can be applied to structures beyond sets, such as sequences, and formally lower bounds the expected admissibility \(the analogue ofUUfor generative tasks\) of the selected structures\. We show that existing task\-specific conformal methods can be recovered as special cases of Conf\-Gen\.
2. \(b\)Theory\.Conf\-Gen relaxes the theoretical assumptions of CRC\. Most importantly, we weaken a monotonicity assumption and we derive a more general reverse \(upper\) bound on the expected utility/admissibility\.
3. \(c\)Python Package\.Naïvely carrying out the computations required by Conf\-Gen is sometimes intractable\. We identify patterns in the design space of Conf\-Gen components which enable efficient computations, and provide a flexible Python package \([https://github\.com/layer6ai\-labs/conf\-gen](https://github.com/layer6ai-labs/conf-gen)\) supporting the full range of Conf\-Gen applications\.
4. \(d\)Empirical Validation\.We demonstrate that Conf\-Gen outperforms state\-of\-the\-art conformal baselines in LLM question answering and showcase its versatility across various novel tasks\. Specifically, we show conformal guarantees for: generating non\-memorized images; ensuring a conversational LLM has asked enough clarifying questions; producing a sequence of agentic AI outputs containing a correct solution; and selecting a subset of trees from a random forest\(Breiman,[2001](https://arxiv.org/html/2605.28920#bib.bib7)\)containing enough trees making the correct prediction\.

## Section 2Conformal Prediction and Risk Control

Split Conformal Prediction\.We present a short summary of key concepts from CP that are relevant to Conf\-Gen; for more thorough presentations, we refer the reader toVovk et al\. \([2005](https://arxiv.org/html/2605.28920#bib.bib54)\),Angelopoulos & Bates \([2021](https://arxiv.org/html/2605.28920#bib.bib2)\), andAngelopoulos et al\. \([2024a](https://arxiv.org/html/2605.28920#bib.bib3)\)\. We focus on split conformal prediction\(Papadopoulos et al\.,[2002](https://arxiv.org/html/2605.28920#bib.bib35)\), a particular instance of CP; henceforth, we will use the abbreviation CP to refer to split conformal prediction\. In CP, we assume access to a calibration dataset containing featuresX\(i\)∈𝒳X^\{\(i\)\}\\in\\mathcal\{X\}and their corresponding ground truth targetYGT\(i\)∈𝒴GTY^\{\(i\)\}\_\{\\text\{GT\}\}\\in\\mathcal\{Y\}\_\{\\text\{GT\}\}fori=1,…,ni=1,\\dots,n\. Then, given a newX\(n\+1\)X^\{\(n\+1\)\}, the goal of CP is to use the calibration dataset to produce a set𝒞​\(X\(n\+1\)\)⊆𝒴GT\\mathcal\{C\}\(X^\{\(n\+1\)\}\)\\subseteq\\mathcal\{Y\}\_\{\\text\{GT\}\}which containsYGT\(n\+1\)Y^\{\(n\+1\)\}\_\{\\text\{GT\}\}with probability at leastγ\\gamma, whereγ\\gammais a user\-specified quantity\. To achieve this goal, CP uses a score functionS↓:𝒳×𝒴GT→ℝS^\{\\downarrow\}:\\mathcal\{X\}\\times\\mathcal\{Y\}\_\{\\text\{GT\}\}\\rightarrow\\mathbb\{R\}, with large values ofS↓​\(x,y\)S^\{\\downarrow\}\(x,y\)conveying low confidence thatyyis compatible withxx\. For example, in a classification setting, we could haveS↓​\(x,y\)=1−S↑​\(x,y\)S^\{\\downarrow\}\(x,y\)=1\-S^\{\\uparrow\}\(x,y\), whereS↑​\(x,y\)S^\{\\uparrow\}\(x,y\)is the probability assigned by a pre\-trained classifier toyywhen givenxxas input\. For a givenλ∈ℝ\\lambda\\in\\mathbb\{R\}, CP considers sets of the form:

𝒞λ​\(x\)≔\{y∈𝒴GT:S↓​\(x,y\)≤λ\}\.\\mathcal\{C\}\_\{\\lambda\}\(x\)\\coloneqq\\\{y\\in\\mathcal\{Y\}\_\{\\text\{GT\}\}:S^\{\\downarrow\}\(x,y\)\\leq\\lambda\\\}\.\(1\)CP performs*calibration*by settingλ^\\hat\{\\lambda\}to be the⌈\(n\+1\)​γ⌉n\\tfrac\{\\lceil\(n\+1\)\\gamma\\rceil\}\{n\}quantile ofS↓​\(X\(1\),YGT\(1\)\),…,S↓​\(X\(n\),YGT\(n\)\)S^\{\\downarrow\}\(X^\{\(1\)\},Y^\{\(1\)\}\_\{\\text\{GT\}\}\),\\dots,S^\{\\downarrow\}\(X^\{\(n\)\},Y^\{\(n\)\}\_\{\\text\{GT\}\}\)\. Then,𝒞λ^​\(X\(n\+1\)\)\\mathcal\{C\}\_\{\\hat\{\\lambda\}\}\(X^\{\(n\+1\)\}\)satisfies the CP desiderata:

###### Theorem 1\(Vovk et al\. \([1999](https://arxiv.org/html/2605.28920#bib.bib53)\)\)\.

Assume\(X\(1\),YGT\(1\)\),…,\(X\(n\+1\),YGT\(n\+1\)\)\(X^\{\(1\)\},Y^\{\(1\)\}\_\{\\textnormal\{GT\}\}\),\\dots,\(X^\{\(n\+1\)\},Y^\{\(n\+1\)\}\_\{\\textnormal\{GT\}\}\)are exchangeable\. Then, for𝒞λ\\mathcal\{C\}\_\{\\lambda\}andλ^\\hat\{\\lambda\}as defined above:

ℙ​\(YGT\(n\+1\)∈𝒞λ^​\(X\(n\+1\)\)\)≥γ\.\{\\mathbb\{P\}\}\\left\(Y^\{\(n\+1\)\}\_\{\\textnormal\{GT\}\}\\in\\mathcal\{C\}\_\{\\hat\{\\lambda\}\}\\left\(X^\{\(n\+1\)\}\\right\)\\right\)\\geq\\gamma\.\(2\)

Recall that the random variablesZ\(1\),…,Z\(n\+1\)Z^\{\(1\)\},\\dots,Z^\{\(n\+1\)\}are*exchangeable*if their joint distribution is invariant under any permutation of their indices\. Exchangeability is a remarkably flexible assumption, as it is strictly weaker than the*i\.i\.d\.*requirement\. Two remarks are worth making about[Theorem 1](https://arxiv.org/html/2605.28920#Thmtheorem1)\. First, note that the probability in[Equation 2](https://arxiv.org/html/2605.28920#S2.E2)is taken with respect to all random quantities inside it, including the calibration dataset itself \(upon whichλ^\\hat\{\\lambda\}depends\)\. Second, although the conformal guarantee in[Equation 2](https://arxiv.org/html/2605.28920#S2.E2)holds regardless of the underlying data\-generating distribution and the choice ofS↓S^\{\\downarrow\}, the practical usefulness of CP does depend onS↓S^\{\\downarrow\}; for example, a poor choice ofS↓S^\{\\downarrow\}could lead to𝒞λ^​\(X\(n\+1\)\)=𝒴GT\\mathcal\{C\}\_\{\\hat\{\\lambda\}\}\(X^\{\(n\+1\)\}\)=\\mathcal\{Y\}\_\{\\text\{GT\}\}, which would render the conformal guarantee vacuously true\.

Conformal Risk Control \(CRC\)\.Angelopoulos et al\. \([2024b](https://arxiv.org/html/2605.28920#bib.bib4)\)extended CP to a more general setting where rather than controlling for coverage, the expectation of an arbitrary utility function is controlled instead\.111CRC is usually formulated as upper\-bounding a loss function\. Lower\-bounding a utility function is equivalent, and will facilitate comparisons to our method\.Let𝒞λ:𝒳→2𝒴GT\\mathcal\{C\}\_\{\\lambda\}:\\mathcal\{X\}\\rightarrow 2^\{\\mathcal\{Y\}\_\{\\text\{GT\}\}\}be a family of functions, indexed byλ∈Λ≔\[λmin,λmax\]\\lambda\\in\\Lambda\\coloneqq\[\\lambda\_\{\\text\{min\}\},\\lambda\_\{\\text\{max\}\}\], producing prediction sets, and letU:2𝒴GT×𝒴GT→\[0,∞\]U:2^\{\\mathcal\{Y\}\_\{\\text\{GT\}\}\}\\times\\mathcal\{Y\}\_\{\\text\{GT\}\}\\rightarrow\[0,\\infty\]be a utility function on prediction sets\. Intuitively, larger values ofλ\\lambdashould result in more conservative prediction sets, i\.e\., ones achieving higher utility\. CRC performs calibration by finding the smallestλ\\lambdasuch that the average utility on the calibration set exceedsn\+1n​γ\\tfrac\{n\+1\}\{n\}\\gamma\. Formally,

λ^=inf\{λ∈Λ:U¯n​\(λ\)≥n\+1n​γ\}∧λmax,\\hat\{\\lambda\}=\\inf\\left\\\{\\lambda\\in\\Lambda:\\bar\{U\}\_\{n\}\(\\lambda\)\\geq\\dfrac\{n\+1\}\{n\}\\gamma\\right\\\}\\wedge\\lambda\_\{\\text\{max\}\},\(3\)whereU¯n​\(λ\)≔1n​∑i=1nU\(i\)​\(λ\)\\bar\{U\}\_\{n\}\(\\lambda\)\\coloneqq\\tfrac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}U^\{\(i\)\}\(\\lambda\)andU\(i\)​\(λ\)≔U​\(𝒞λ​\(X\(i\)\),YGT\(i\)\)U^\{\(i\)\}\(\\lambda\)\\coloneqq U\(\\mathcal\{C\}\_\{\\lambda\}\(X^\{\(i\)\}\),Y^\{\(i\)\}\_\{\\text\{GT\}\}\)\.222Recall that∧\\wedgeand∨\\veedenote the minimum and maximum between two numbers, respectively, and thatinf∅=∞\\inf\\emptyset=\\infty; the minimum in[Equation 3](https://arxiv.org/html/2605.28920#S2.E3)handles this case\.Under some regularity conditions, a conformal guarantee follows:

###### Theorem 2\(Angelopoulos et al\. \([2024b](https://arxiv.org/html/2605.28920#bib.bib4)\)\)\.

Assume the functionsU\(1\),…,U\(n\+1\)U^\{\(1\)\},\\dots,U^\{\(n\+1\)\}are exchangeable, and that they are right\-continuous and non\-decreasing onΛ\\Lambda, almost surely\. Assume also thatU\(i\)​\(λmax\)≥γU^\{\(i\)\}\(\\lambda\_\{\\max\}\)\\geq\\gamma, almost surely\. Then, forλ^\\hat\{\\lambda\}defined as in[Equation 3](https://arxiv.org/html/2605.28920#S2.E3), the following inequality holds:

𝔼​\[U\(n\+1\)​\(λ^\)\]≥γ\.\\mathbb\{E\}\\left\[U^\{\(n\+1\)\}\(\\hat\{\\lambda\}\)\\right\]\\geq\\gamma\.\(4\)

Note that the functionsU\(1\),…,U\(n\+1\)U^\{\(1\)\},\\dots,U^\{\(n\+1\)\}are exchangeable whenever\(X\(1\),YGT\(1\)\),…,\(X\(n\+1\),YGT\(n\+1\)\)\(X^\{\(1\)\},Y^\{\(1\)\}\_\{\\textnormal\{GT\}\}\),\\dots,\(X^\{\(n\+1\)\},Y^\{\(n\+1\)\}\_\{\\textnormal\{GT\}\}\)are exchangeable, providedUUand𝒞λ\\mathcal\{C\}\_\{\\lambda\}are either deterministic, or independent of the features and ground truth variables\. Once again, the expectation in[Equation 4](https://arxiv.org/html/2605.28920#S2.E4)is taken with respect to all random quantities inside it\.

The lower bounds in[Theorem 1](https://arxiv.org/html/2605.28920#Thmtheorem1)and[Theorem 2](https://arxiv.org/html/2605.28920#Thmtheorem2)represent the foundational guarantees of CP and CRC, respectively\. Corresponding upper bounds quantifying the degree of over\-coverage in the resulting sets also exist; we summarize them in[Appendix A](https://arxiv.org/html/2605.28920#A1)\. Note that CRC generalizes CP; indeed,[Equation 4](https://arxiv.org/html/2605.28920#S2.E4)recovers[Equation 2](https://arxiv.org/html/2605.28920#S2.E2)whenU​\(𝒞,y\)=𝟙​\(y∈𝒞\)U\(\\mathcal\{C\},y\)=\\mathds\{1\}\(y\\in\\mathcal\{C\}\)\.

## Section 3Conformal Generation

Motivation\.The goal of this section is to extend conformal methodology to generative tasks in a highly general way\. We begin by identifying several practical limitations in the current application of CRC\. First, while[Theorem 2](https://arxiv.org/html/2605.28920#Thmtheorem2)does not explicitly restrict𝒳\\mathcal\{X\}to the feature space of a supervised learning problem,Angelopoulos et al\. \([2024b](https://arxiv.org/html/2605.28920#bib.bib4)\)focus exclusively on this setting\. Second,𝒞λ\\mathcal\{C\}\_\{\\lambda\}always outputs sets, despite this not being a prerequisite to maintain exchangeability\. Third, current implementations assumeUUis a callable function, which precludes utilities defined by procedures such as human evaluation\. Finally, the requirement thatU\(i\)U^\{\(i\)\}be non\-decreasing is overly restrictive for some generative tasks\. We will see that Conf\-Gen addresses all these limitations\.

Notation\.We use calligraphic font to denote sets \(e\.g\.,𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}\), and we write the space of finite𝒴\\mathcal\{Y\}\-valued sequences as𝒴∗\\mathcal\{Y\}^\{\*\}, i\.e\.,𝒴∗≔∪t=0∞𝒴t\\mathcal\{Y\}^\{\*\}\\coloneqq\\cup\_\{t=0\}^\{\\infty\}\\mathcal\{Y\}^\{t\}\. We use lower\-case letters to denote fixed elements of these sets \(e\.g\.,x∈𝒳x\\in\\mathcal\{X\}andy∈𝒴y\\in\\mathcal\{Y\}\), upper\-case letters to denote random variables \(e\.g\.,X∈𝒳X\\in\\mathcal\{X\}andY∈𝒴Y\\in\\mathcal\{Y\}\), and bold font to denote tuples and sequences \(e\.g\., a random sequence𝐘=\(Y1,…,YT\)∈𝒴∗\\mathbf\{Y\}=\(Y\_\{1\},\\dots,Y\_\{T\}\)\\in\\mathcal\{Y\}^\{\*\}\)\. We denote the length of a sequence𝐲\\mathbf\{y\}as\|𝐲\|\|\\mathbf\{y\}\|, and its subsequence given by\(y1,…,yt\)\(y\_\{1\},\\dots,y\_\{t\}\)as𝐲:t\\mathbf\{y\}\_\{:t\}\.

Setting\.In Conf\-Gen we observe samples𝐆=\(X,𝐘,YGT\)\\mathbf\{G\}=\(X,\\mathbf\{Y\},Y\_\{\\text\{GT\}\}\)from some distributionℙ𝐆\\mathbb\{P\}\_\{\\mathbf\{G\}\}on𝒳×𝒴∗×𝒴GT\\mathcal\{X\}\\times\\mathcal\{Y\}^\{\*\}\\times\\mathcal\{Y\}\_\{\\text\{GT\}\}\. Here,XXcorresponds to a conditioning variable or “input” of the generative model,𝐘\\mathbf\{Y\}is a sequence of generations or “outputs” from the model, andYGTY\_\{\\text\{GT\}\}is an optional ground truth corresponding toXX\. For example,𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}could both be the space of text, withXXcorresponding to a question,𝐘\\mathbf\{Y\}to a sequence of answers toXXgenerated by an LLM, andYGTY\_\{\\text\{GT\}\}to a ground truth answer; we will use this as a running example throughout our paper, but we nonetheless highlight that our method is much more general\. The goal of Conf\-Gen will be to use\(X,𝐘\)\(X,\\mathbf\{Y\}\), which plays the role of the “features” in CRC, to produce a set \(or more generally, a structured output, such as a sequence\) guaranteed to have large expected admissibility, which expands the notion of utility in CRC to the generative realm\. Conf\-Gen provides partial control overℙ𝐆\\mathbb\{P\}\_\{\\mathbf\{G\}\}, whereas CRC lacks control over its data\-generating distribution\. For instance, in our running example, we can choose how each individual answerYtY\_\{t\}in𝐘\\mathbf\{Y\}is generated \(e\.g\., they could be generated independently, or we could also enforce the LLM to provide a different answer than its previous ones,Y1,…,Yt−1Y\_\{1\},\\dots,Y\_\{t\-1\}\)\.

The Conf\-Gen framework is comprised by three components: a parameterized family of selection functions, an admissibility function, and a calibration dataset\. We define and discuss these components below\.

Selection Functions\.We assume access to a family of potentially stochastic functions,𝐂λ:𝒳×𝒴∗→𝒮\\mathbf\{C\}\_\{\\lambda\}:\\mathcal\{X\}\\times\\mathcal\{Y\}^\{\*\}\\rightarrow\{\\mathcal\{S\}\}, indexed byλ∈Λ\\lambda\\in\\Lambda, whereΛ⊆\[−∞,∞\]\\Lambda\\subseteq\[\-\\infty,\\infty\]is such that∞∈Λ\\infty\\in\\Lambda\. Here,𝒮\{\\mathcal\{S\}\}is the output space over which we will provide a conformal guarantee; it is precisely the freedom to choose𝒮\{\\mathcal\{S\}\}that allows Conf\-Gen to extend CRC beyond sets\. Natural choices of𝒮\{\\mathcal\{S\}\}include𝒮=2𝒴\{\\mathcal\{S\}\}=2^\{\\mathcal\{Y\}\}and𝒮=𝒴∗\{\\mathcal\{S\}\}=\\mathcal\{Y\}^\{\*\}; most examples in our paper will use the latter, and we hence write𝐂λ\\mathbf\{C\}\_\{\\lambda\}in bold font\. Intuitively, larger values ofλ\\lambdashould correspond to𝐂λ\\mathbf\{C\}\_\{\\lambda\}producing more conservative outputs, i\.e\., less useful but achieving larger admissibility values\.𝐂λ\\mathbf\{C\}\_\{\\lambda\}is analogous to𝒞λ\\mathcal\{C\}\_\{\\lambda\}from CP and CRC, but it takes a sequence of generations as an additional input\. In our running example,𝐂λ​\(x,𝐲\)\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)could be given by the subsequence of𝐲\\mathbf\{y\}containing only the elementsyty\_\{t\}such thatS↓​\(x,yt\)≤λS^\{\\downarrow\}\(x,y\_\{t\}\)\\leq\\lambda, whereS↓:𝒳×𝒴→ℝS^\{\\downarrow\}:\\mathcal\{X\}\\times\\mathcal\{Y\}\\rightarrow\\mathbb\{R\}is a score function, e\.g\.,S↓​\(x,y\)=−S↑​\(x,y\)S^\{\\downarrow\}\(x,y\)=\-S^\{\\uparrow\}\(x,y\), whereS↑​\(x,y\)S^\{\\uparrow\}\(x,y\)is obtained by asking the LLM to provide a scalar value expressing its confidence thatyyis a correct answer to the questionxx, or by having access to the likelihood function of the LLM\.333We will use bothS↓S^\{\\downarrow\}andS↑S^\{\\uparrow\}to denote score functions, with the convention that smaller values ofS↓S^\{\\downarrow\}are “better” \(i\.e\., expected to have large instance\-level admissibility, see[Section 4\.1](https://arxiv.org/html/2605.28920#S4.SS1); in our LLM example, this means correctness of the answer\), whereas larger values ofS↑S^\{\\uparrow\}are “better”\.

Admissibility Function\.We also assume the existence of a potentially stochastic admissibility functionA:𝒳×𝒮×𝒴GT→\[0,∞\]A:\\mathcal\{X\}\\times\{\\mathcal\{S\}\}\\times\\mathcal\{Y\}\_\{\\text\{GT\}\}\\rightarrow\[0,\\infty\], whereA​\(X,𝐂λ​\(X,𝐘\),YGT\)A\(X,\\mathbf\{C\}\_\{\\lambda\}\(X,\\mathbf\{Y\}\),Y\_\{\\text\{GT\}\}\)generalizes the role ofU​\(𝒞λ​\(X\),YGT\)U\(\\mathcal\{C\}\_\{\\lambda\}\(X\),Y\_\{\\text\{GT\}\}\)in CRC\. Intuitively, larger values ofAAare more desirable, but the outputs of𝐂λ\\mathbf\{C\}\_\{\\lambda\}must become more conservative to increaseAA\. In our running example,A​\(x,𝐲,yGT\)A\(x,\\mathbf\{y\},y\_\{\\text\{GT\}\}\)will be a binary variable indicating whether at least one element of𝐲\\mathbf\{y\}is a correct answer toxx\(other choices, e\.g\., a scalar indicating the quality of the best answer in𝐲\\mathbf\{y\}, could also be possible\)\. When the ground truth is well\-defined, thisAAcould be given by an LLM call checking the semantic equivalence betweenyGTy\_\{\\text\{GT\}\}and each element of𝐲\\mathbf\{y\}\. Importantly though, we do not assume thatAAis callable\. For example, the questionxxbeing asked could so open\-ended that it has no corresponding ground truth answeryGTy\_\{\\text\{GT\}\}\. In this case,AAcould instead be given by human evaluation without access toyGTy\_\{\\text\{GT\}\}; mathematically, this corresponds toA​\(x,𝐲,yGT\)A\(x,\\mathbf\{y\},y\_\{\\text\{GT\}\}\)ignoringyGTy\_\{\\text\{GT\}\}, and thus Conf\-Gen does not require a well\-defined ground truth\.

Calibration Dataset\.Let us consider a sequence𝐆:n≔\(\(X\(i\),𝐘\(i\),YGT\(i\)\)\)i=1n\\mathbf\{G\}\_\{:n\}\\coloneqq\(\(X^\{\(i\)\},\\mathbf\{Y\}^\{\(i\)\},Y\_\{\\text\{GT\}\}^\{\(i\)\}\)\)\_\{i=1\}^\{n\}containingnnsamples fromℙ𝐆\\mathbb\{P\}\_\{\\mathbf\{G\}\}\. The last component for Conf\-Gen is a calibration dataset,𝐃:n\\mathbf\{D\}\_\{:n\}, which contains the admissibility of every tuple in𝐆:n\\mathbf\{G\}\_\{:n\}for everyλ∈Λ\\lambda\\in\\Lambda, i\.e\.,

𝐃:n≔\(\{A\(i\)​\(λ\)\}λ∈Λ\)i=1n,\\mathbf\{D\}\_\{:n\}\\coloneqq\\left\(\\left\\\{A^\{\(i\)\}\(\\lambda\)\\right\\\}\_\{\\lambda\\in\\Lambda\}\\right\)\_\{i=1\}^\{n\},\(5\)whereA\(i\)​\(λ\)≔A​\(X\(i\),𝐂λ​\(X\(i\),𝐘\(i\)\),YGT\(i\)\)A^\{\(i\)\}\(\\lambda\)\\coloneqq A\(X^\{\(i\)\},\\mathbf\{C\}\_\{\\lambda\}\(X^\{\(i\)\},\\mathbf\{Y\}^\{\(i\)\}\),Y\_\{\\text\{GT\}\}^\{\(i\)\}\)\. Note that the generated sequences in𝐆:n\\mathbf\{G\}\_\{:n\}need not have the same length; we make this explicit by writing𝐘\(i\)=\(Y1\(i\),…,YTi\(i\)\)\\mathbf\{Y\}^\{\(i\)\}=\(Y\_\{1\}^\{\(i\)\},\\dots,Y^\{\(i\)\}\_\{T\_\{i\}\}\)\. We also highlight that whileAAneed not be callable, we do require access to𝐃:n\\mathbf\{D\}\_\{:n\}\. In our running LLM example, this can be achieved—even whenAAis not callable—by evaluating the correctness of every answer in𝐘\(i\)\\mathbf\{Y\}^\{\(i\)\}for everyi=1,…,ni=1,\\dots,n\. We will cover this point in more detail and generality in[Section 4\.1](https://arxiv.org/html/2605.28920#S4.SS1)\.

Conformal Generation\.For a given instance of Conf\-Gen composed of the above components, the goal is to leverage the calibration dataset to findλ^\\hat\{\\lambda\}such that𝐂λ^\\mathbf\{C\}\_\{\\hat\{\\lambda\}\}produces outputs that achieve large admissibility values\. In our running example, this means𝐂λ^​\(X\(n\+1\),𝐘\(n\+1\)\)\\mathbf\{C\}\_\{\\hat\{\\lambda\}\}\(X^\{\(n\+1\)\},\\mathbf\{Y\}^\{\(n\+1\)\}\)contains a correct answer with high probability\. In analogy to CRC, calibration is performed by findingλ^\\hat\{\\lambda\}, the smallestλ\\lambdasuch that the average admissibility on the calibration dataset is at leastn\+1n​γ\\tfrac\{n\+1\}\{n\}\\gamma\. More formally, forγ≥0\\gamma\\geq 0, we define

λ^≔inf𝒱​\(𝐃:n,n\+1n​γ\),\\hat\{\\lambda\}\\coloneqq\\inf\\mathcal\{V\}\\left\(\\mathbf\{D\}\_\{:n\},\\dfrac\{n\+1\}\{n\}\\gamma\\right\),\(6\)where

𝒱​\(𝐃:n,γ′\)≔\{λ∈Λ:A¯n​\(λ\)≥γ′\}∪\{∞\}\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\},\\gamma^\{\\prime\}\)\\coloneqq\\left\\\{\\lambda\\in\\Lambda:\\bar\{A\}\_\{n\}\(\\lambda\)\\geq\\gamma^\{\\prime\}\\right\\\}\\cup\\\{\\infty\\\}\(7\)andA¯n​\(λ\)≔1n​∑i=1nA\(i\)​\(λ\)\\bar\{A\}\_\{n\}\(\\lambda\)\\coloneqq\\tfrac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}A^\{\(i\)\}\(\\lambda\)\. We now introduce the notion of a*γ\\gamma\-sensible*instance of Conf\-Gen, which provides the requirements for our conformal guarantees\.

###### Definition 1\.

We say that an instance of Conf\-Gen is*γ\\gamma\-sensible*if the following properties hold:

1. \(a\)The functionsA\(1\),…,A\(n\+1\)A^\{\(1\)\},\\dots,A^\{\(n\+1\)\}are exchangeable\.
2. \(b\)For everyγ′,γ′′≥0\\gamma^\{\\prime\},\\gamma^\{\\prime\\prime\}\\geq 0, ifλ′=inf𝒱​\(𝐃:n,γ′\)\\lambda^\{\\prime\}=\\inf\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\},\\gamma^\{\\prime\}\)andλ′′=inf𝒱​\(𝐃:n\+1,γ′′\)\\lambda^\{\\prime\\prime\}=\\inf\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\+1\},\\gamma^\{\\prime\\prime\}\), then: 1. \(b1\)The map given byλ↦𝔼​\[A\(n\+1\)​\(λ\)∣λ′,λ′′\]\\lambda\\mapsto\\mathbb\{E\}\[A^\{\(n\+1\)\}\(\\lambda\)\\mid\\lambda^\{\\prime\},\\lambda^\{\\prime\\prime\}\]is non\-decreasing onΛ\\Lambda, almost surely\. 2. \(b2\)λ′∈𝒱​\(𝐃:n,γ′\)\\lambda^\{\\prime\}\\in\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\},\\gamma^\{\\prime\}\)andλ′′∈𝒱​\(𝐃:n\+1,γ′′\)\\lambda^\{\\prime\\prime\}\\in\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\+1\},\\gamma^\{\\prime\\prime\}\), almost surely\.
3. \(c\)A​\(X,𝐂∞​\(X,𝐘\),YGT\)≥γA\(X,\\mathbf\{C\}\_\{\\infty\}\(X,\\mathbf\{Y\}\),Y\_\{\\textnormal\{GT\}\}\)\\geq\\gamma, almost surely\.444In practice, this property can always be achieved\. For example, we can add an element,Abstain, to𝒴\\mathcal\{Y\}, defineA​\(x,𝐲,yGT\)A\(x,\\mathbf\{y\},y\_\{\\text\{GT\}\}\)to beγ\\gamma\(or any larger scalar\) wheneverAbstainis an element of𝐲\\mathbf\{y\}, and define𝐂∞​\(x,𝐲\)\\mathbf\{C\}\_\{\\infty\}\(x,\\mathbf\{y\}\)to always haveAbstainas an element\. Intuitively, a sequence containingAbstainrepresents that we abstain from making a non\-vacuous conformal guarantee, analogously to having𝒞λ^​\(X\(n\+1\)\)=𝒴GT\\mathcal\{C\}\_\{\\hat\{\\lambda\}\}\(X^\{\(n\+1\)\}\)=\\mathcal\{Y\}\_\{\\text\{GT\}\}in CP\.

Property\(a\)\(a\), which is identical to the exchangeability assumption of CRC, holds whenever𝐆:n\+1\\mathbf\{G\}\_\{:n\+1\}is exchangeable and bothAAand𝐂λ\\mathbf\{C\}\_\{\\lambda\}are independent of𝐆:n\+1\\mathbf\{G\}\_\{:n\+1\}\. Note that in our running example this property does not require tokens within an answer, nor answers within a sequence of answers, to be exchangeable; we only require different sequences of answers themselves to be exchangeable\. Intuitively, property\(b​1\)\(b1\)weakens the strict monotonicity required by CRC to “sufficient monotonicity”, and\(b​2\)\(b2\)relaxes the right\-continuity assumption of CRC\. Lastly,\(c\)\(c\)is analogous to the condition thatU​\(𝒞λmax​\(X\),YGT\)≥γU\(\\mathcal\{C\}\_\{\\lambda\_\{\\max\}\}\(X\),Y\_\{\\text\{GT\}\}\)\\geq\\gammain CRC, although we make it explicit thatΛ\\Lambdaneed not be bounded by defaulting its maximal element to∞\\inftyand requiring that∞∈Λ\\infty\\in\\Lambda\.

###### Theorem 3\.

Consider aγ\\gamma\-sensible instance of Conf\-Gen\. Then, forλ^\\hat\{\\lambda\}as defined in[Equation 6](https://arxiv.org/html/2605.28920#S3.E6),

𝔼​\[A\(n\+1\)​\(λ^\)\]≥γ\.\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\hat\{\\lambda\}\)\\right\]\\geq\\gamma\.\(8\)Additionally, ifA:𝒳×𝒮×𝒴GT→\[0,amax\]A:\\mathcal\{X\}\\times\{\\mathcal\{S\}\}\\times\\mathcal\{Y\}\_\{\\textnormal\{GT\}\}\\rightarrow\[0,a\_\{\\textnormal\{max\}\}\], then

𝔼​\[A\(n\+1\)​\(λ^\)\]≤γ\+amaxn\+1\+𝔼​\[H\],\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\hat\{\\lambda\}\)\\right\]\\leq\\gamma\+\\dfrac\{a\_\{\\textnormal\{max\}\}\}\{n\+1\}\+\\mathbb\{E\}\[H\],\(9\)whereλ∗∗≔inf𝒱​\(𝐃:n\+1,γ\+amaxn\+1\)\\lambda^\{\*\*\}\\coloneqq\\inf\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\+1\},\\gamma\+\\tfrac\{a\_\{\\textnormal\{max\}\}\}\{n\+1\}\)and

H≔A¯n\+1​\(λ∗∗\)−sup\{A¯n\+1​\(λ\):λ∈Λ,λ<λ∗∗\}∨0\.\\scalebox\{0\.86\}\{ $H\\coloneqq\\bar\{A\}\_\{n\+1\}\(\\lambda^\{\*\*\}\)\-\\sup\\left\\\{\\bar\{A\}\_\{n\+1\}\(\\lambda\):\\lambda\\in\\Lambda,\\lambda<\\lambda^\{\*\*\}\\right\\\}\\vee 0$\}\.\(10\)

Conf\-Gen vs CRC\.Despite[Definition 1](https://arxiv.org/html/2605.28920#Thmdefinition1)being relatively technical, Conf\-Gen relies on strictly weaker assumptions than CRC\. Most notably, it relaxes the monotonicity assumption from an almost sure condition to one in conditional expectation\. It also enables the construction of conformal objects beyond sets \(𝒮=2𝒴\{\\mathcal\{S\}\}=2^\{\\mathcal\{Y\}\}\), such as sequences \(𝒮=𝒴∗\{\\mathcal\{S\}\}=\\mathcal\{Y\}^\{\*\}\), along with other minor technical refinements\. Conf\-Gen serves as a strict generalization of CRC, where any instance of the latter can be recovered as a specific case of the former \(by having𝐂λ\\mathbf\{C\}\_\{\\lambda\}ignore𝐘\\mathbf\{Y\}, andAAignoreXX\)\. The gap between the lower bound in[Equation 8](https://arxiv.org/html/2605.28920#S3.E8)and the upper bound in[Equation 9](https://arxiv.org/html/2605.28920#S3.E9)is determined by𝔼​\[H\]\\mathbb\{E\}\[H\], where, intuitively,HHquantifies the size of the jumps of the average admissibility,A¯n\+1\\bar\{A\}\_\{n\+1\}\. Importantly,[Equation 9](https://arxiv.org/html/2605.28920#S3.E9)generalizes the existing upper bound for CRC\. A detailed discussion of the theoretical assumptions underpinning Conf\-Gen, as well as a comprehensive comparison to CRC, is provided in[Appendix C](https://arxiv.org/html/2605.28920#A3)\.

## Section 4Computational Considerations

In this section we outline common patterns in the selection and admissibility functions which enable efficient calibration and inference within Conf\-Gen\. Our accompanying Python package supports all the instances and patterns we mention, as well as the straightforward customization of all Conf\-Gen components\. Beyond the examples presented here, our package includes a broader suite of utilities designed to support a wide range of Conf\-Gen applications\.

### Section 4\.1Efficient Calibration

In practice, the procedure for findingλ^\\hat\{\\lambda\}must be tailored to the particular Conf\-Gen instance at hand\. IfA¯n\\bar\{A\}\_\{n\}is suitable to gradient\-based optimization,λ^\\hat\{\\lambda\}can be obtained by calling a solver to minimizeλ\\lambdasubject toA¯n​\(λ\)≥n\+1n​γ\\bar\{A\}\_\{n\}\(\\lambda\)\\geq\\tfrac\{n\+1\}\{n\}\\gamma\. IfΛ\\Lambdais finite, constructing𝐃:n\\mathbf\{D\}\_\{:n\}from𝐆:n\\mathbf\{G\}\_\{:n\}requires evaluating the admissibility functionn×\|Λ\|n\\times\|\\Lambda\|times, after which computingλ^\\hat\{\\lambda\}is straightforward; this process could range from trivially cheap to prohibitively expensive, depending on the cost of evaluatingAA\. IfΛ\\Lambdais infinite andAAis not a callable function \(e\.g\., if it is given by human evaluation\), calibration might seem out of reach since calling a solver is not feasible and constructing𝐃:n\\mathbf\{D\}\_\{:n\}appears impossible\.

Score\-Based Selection Functions\.Specific choices of𝐂λ\\mathbf\{C\}\_\{\\lambda\}enable calibration even in this challenging scenario\. Our key insight here is that if𝐂λ\\mathbf\{C\}\_\{\\lambda\}has a finite image as a function ofλ\\lambda\(i\.e\., if\{𝐂λ​\(X,𝐘\):λ∈Λ\}\\\{\\mathbf\{C\}\_\{\\lambda\}\(X,\\mathbf\{Y\}\):\\lambda\\in\\Lambda\\\}is finite, almost surely\), then we can characterize𝐃:n\\mathbf\{D\}\_\{:n\}—even if the setΛ\\Lambdais infinite—by evaluatingA\(i\)A^\{\(i\)\}at the finitely\-many possible outputs of𝐂λ​\(X\(i\),𝐘\(i\)\)\\mathbf\{C\}\_\{\\lambda\}\(X^\{\(i\)\},\\mathbf\{Y\}^\{\(i\)\}\), for everyi=1,…,ni=1,\\dots,n\. A finite image can be achieved by defining𝐂λ​\(x,𝐲\)\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)as a subsequence of𝐲\\mathbf\{y\}derived from scores assigned to each element in𝐲\\mathbf\{y\}\. In the example from[Section 3](https://arxiv.org/html/2605.28920#S3), the selection function uses scoresSt↓=S↓​\(x,yt\)S\_\{t\}^\{\\downarrow\}=S^\{\\downarrow\}\(x,y\_\{t\}\)to form the subsequence𝐂λ\(x,𝐲\)=\(yt:St↓≤λ\)\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)=\(y\_\{t\}:S\_\{t\}^\{\\downarrow\}\\leq\\lambda\)\. Other constructions are also viable; for example,𝐂λ​\(x,𝐲\)\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)could be defined as𝐲:τ​\(x,𝐲,λ\)\\mathbf\{y\}\_\{:\\tau\(x,\\mathbf\{y\},\\lambda\)\}, whereτ​\(x,𝐲,λ\)\\tau\(x,\\mathbf\{y\},\\lambda\)is the first indexttat which the accumulated score of𝐲:t\\mathbf\{y\}\_\{:t\}exceedsλ\\lambda\. More formally,

τ​\(x,𝐲,λ\)≔inf\{t:accum​\(S1↑,…,St↑\)\>λ\}∧\|𝐲\|,\\tau\(x,\\mathbf\{y\},\\lambda\)\\coloneqq\\inf\\left\\\{t:\\texttt\{accum\}\\left\(S\_\{1\}^\{\\uparrow\},\\dots,S\_\{t\}^\{\\uparrow\}\\right\)\>\\lambda\\right\\\}\\wedge\|\\mathbf\{y\}\|,

\(11\)whereSt↑=S↑​\(x,yt\)S\_\{t\}^\{\\uparrow\}=S^\{\\uparrow\}\(x,y\_\{t\}\)andaccum:ℝ∗→ℝ\\texttt\{accum\}:\\mathbb\{R\}^\{\*\}\\rightarrow\\mathbb\{R\}represents an accumulation function such as a sum or a maximum\. A sum would be appropriate when the individual scores are non\-negative, and in the context of our running example, if the individual answers tend to be varied\. Conversely, a maximum might be preferable if duplicate answers are frequent, as it prevents redundant answers from inflating the accumulated score\. These examples are illustrative rather than exhaustive, for instance: the rules for generating a subsequence from individual scores are flexible, the score at timettcould depend on the entire prefix𝐲:t\\mathbf\{y\}\_\{:t\}rather than just the individual elementyty\_\{t\}, and the finite image property can still be satisfied when𝐂λ​\(x,𝐲\)\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)is not a subsequence of𝐲\\mathbf\{y\}\.

Admissibility Evaluation\.Without additional structure beyond a finite image, calibration requires evaluatingAAacross all potential outputs of𝐂λ\\mathbf\{C\}\_\{\\lambda\}; this can be suboptimal because the evaluation ofAAbecomes tied to the specific choice of𝐂λ\\mathbf\{C\}\_\{\\lambda\}\. For example, testing a new selection function would require a completely new set of evaluations𝐃:n\\mathbf\{D\}\_\{:n\}, even if𝐆:n\\mathbf\{G\}\_\{:n\}remains fixed\. To address this issue, we can often identify an instance\-level admissibility function,A′:𝒳×𝒴×𝒴GT→\[0,∞\]A^\{\\prime\}:\\mathcal\{X\}\\times\\mathcal\{Y\}\\times\\mathcal\{Y\}\_\{\\text\{GT\}\}\\rightarrow\[0,\\infty\], which allowsAAto be decomposed as

A​\(x,𝐲,yGT\)=agg​\(A1′,…,AT′\),A\\left\(x,\\mathbf\{y\},y\_\{\\text\{GT\}\}\\right\)=\\texttt\{agg\}\\left\(A^\{\\prime\}\_\{1\},\\dots,A^\{\\prime\}\_\{T\}\\right\),\(12\)whereAt′≔A′​\(x,yt,yGT\)A^\{\\prime\}\_\{t\}\\coloneqq A^\{\\prime\}\(x,y\_\{t\},y\_\{\\text\{GT\}\}\)andagg:ℝ∗→ℝ\\texttt\{agg\}:\\mathbb\{R\}^\{\*\}\\rightarrow\\mathbb\{R\}is an aggregation function, such as the minimum or maximum\. In our running example,aggis the maximum andAt′A\_\{t\}^\{\\prime\}represents the correctness ofyty\_\{t\}as an answer toxx\. Under this decomposition, when𝐂λ​\(x,𝐲\)\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)is a subsequence of𝐲\\mathbf\{y\}, evaluatingA′​\(X\(i\),Yt\(i\),YGT\(i\)\)A^\{\\prime\}\(X^\{\(i\)\},Y\_\{t\}^\{\(i\)\},Y\_\{\\text\{GT\}\}^\{\(i\)\}\)for allt=1,…,Tit=1,\\dots,T\_\{i\}andi=1,…,ni=1,\\dots,nis sufficient for calibration\. This process decouples the evaluation ofAAfrom the choice of𝐂λ\\mathbf\{C\}\_\{\\lambda\}, as it requires evaluatingA′A^\{\\prime\}over the same fixed set of∑i=1nTi\\sum\_\{i=1\}^\{n\}T\_\{i\}points, regardless of𝐂λ\\mathbf\{C\}\_\{\\lambda\}\. Lastly, as with selection functions, this example is not exhaustive, e\.g\.,At′A\_\{t\}^\{\\prime\}could depend on the prefix𝐲:t\\mathbf\{y\}\_\{:t\}rather than just the single elementyty\_\{t\}\.

### Section 4\.2Efficient Inference through Partial Generation

A priori, once calibration has been performed, computing𝐂λ^​\(X\(n\+1\),𝐘\(n\+1\)\)\\mathbf\{C\}\_\{\\hat\{\\lambda\}\}\(X^\{\(n\+1\)\},\\mathbf\{Y\}^\{\(n\+1\)\}\)at test time requires access to bothX\(n\+1\)X^\{\(n\+1\)\}and𝐘\(n\+1\)\\mathbf\{Y\}^\{\(n\+1\)\}; however, many scenarios only necessitate partial access to these quantities\. We refer to these cases as having the capability for*partial generation*\. For instance, there may exist a stopping timeτn\+1≤Tn\+1\\tau\_\{n\+1\}\\leq T\_\{n\+1\}such that𝐘:τn\+1\(n\+1\)\\mathbf\{Y\}\_\{:\\tau\_\{n\+1\}\}^\{\(n\+1\)\}contains all the information needed from𝐘\(n\+1\)\\mathbf\{Y\}^\{\(n\+1\)\}to compute𝐂λ^​\(X\(n\+1\),𝐘\(n\+1\)\)\\mathbf\{C\}\_\{\\hat\{\\lambda\}\}\(X^\{\(n\+1\)\},\\mathbf\{Y\}^\{\(n\+1\)\}\)\. When this is the case, we can sequentially generateY1\(n\+1\),…,Yτn\+1\(n\+1\)Y\_\{1\}^\{\(n\+1\)\},\\dots,Y\_\{\\tau\_\{n\+1\}\}^\{\(n\+1\)\}and stop, thus avoiding having to generate the remainingTn\+1−τn\+1T\_\{n\+1\}\-\\tau\_\{n\+1\}elements; this can save significant compute when generation is expensive\. In the case where𝐂λ​\(x,𝐲\)=𝐲:τ​\(x,𝐲,λ\)\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)=\\mathbf\{y\}\_\{:\\tau\(x,\\mathbf\{y\},\\lambda\)\}forτ​\(x,𝐲,λ\)\\tau\(x,\\mathbf\{y\},\\lambda\)defined as in[Equation 11](https://arxiv.org/html/2605.28920#S4.E11), we have thatτn\+1=τ​\(X\(n\+1\),𝐘\(n\+1\),λ^\)\\tau\_\{n\+1\}=\\tau\(X^\{\(n\+1\)\},\\mathbf\{Y\}^\{\(n\+1\)\},\\hat\{\\lambda\}\)\. Notably, the potential for partial generation extends beyond this single instance\.

### Section 4\.3Examples

Not all design choices within Conf\-Gen are mutually compatible, i\.e\., they do not always result inγ\\gamma\-sensibility holding\.[Table 1](https://arxiv.org/html/2605.28920#S4.T1)presents various score\-based selection functions alongside candidate admissibility functions with which they are compatible\. These entries, which also detail partial generation capabilities, assumeΛ=\[−∞,∞\]\\Lambda=\[\-\\infty,\\infty\]and the instance\-level decomposition defined in[Equation 12](https://arxiv.org/html/2605.28920#S4.E12)\. Further examples are provided in[Appendix D](https://arxiv.org/html/2605.28920#A4)\.

The first two rows of[Table 1](https://arxiv.org/html/2605.28920#S4.T1)are suited for scenarios where the goal is to ensure that at least one element in𝐂λ\\mathbf\{C\}\_\{\\lambda\}possesses high admissibility \(agg=max\\text\{agg\}=\\max\)\. In the second row, the nesting property,𝐂λ⊆𝐂λ′\\mathbf\{C\}\_\{\\lambda\}\\subseteq\\mathbf\{C\}\_\{\\lambda^\{\\prime\}\}forλ≤λ′\\lambda\\leq\\lambda^\{\\prime\}, holds inherently\. For the first row, this nesting is satisfied provided thataccum=max\\texttt\{accum\}=\\maxor the score function is non\-negative, as either condition ensures thatτ​\(x,𝐲,λ\)\\tau\(x,\\mathbf\{y\},\\lambda\)is non\-decreasing inλ\\lambda\. In both cases, this nesting behavior ensures that the output of𝐂λ\\mathbf\{C\}\_\{\\lambda\}becomes more conservative asλ\\lambdaincreases: since appending elements to a sequence cannot decrease the admissibility of its most admissible member, the mappingλ↦A​\(λ\)≔A​\(X,𝐂λ​\(X,𝐘\),YGT\)\\lambda\\mapsto A\(\\lambda\)\\coloneqq A\(X,\\mathbf\{C\}\_\{\\lambda\}\(X,\\mathbf\{Y\}\),Y\_\{\\text\{GT\}\}\)is non\-decreasing, and thus property\(b​1\)\(b1\)of theγ\\gamma\-sensibility requirements \([Definition 1](https://arxiv.org/html/2605.28920#Thmdefinition1)\) is satisfied\.

The third row is appropriate when the goal is for all elements in𝐂λ\\mathbf\{C\}\_\{\\lambda\}to have large admissibility \(agg=min\\texttt\{agg\}=\\min\)\. Here, the nesting property is reversed:𝐂λ′⊆𝐂λ\\mathbf\{C\}\_\{\\lambda^\{\\prime\}\}\\subseteq\\mathbf\{C\}\_\{\\lambda\}wheneverλ≤λ′\\lambda\\leq\\lambda^\{\\prime\}\. Nevertheless, the outputs of𝐂λ\\mathbf\{C\}\_\{\\lambda\}still become more conservative asλ\\lambdaincreases because removing elements from a set cannot decrease the admissibility of its least admissible member; consequently,λ↦A​\(λ\)\\lambda\\mapsto A\(\\lambda\)remains non\-decreasing\. The conformal factuality framework\(Mohri & Hashimoto,[2024](https://arxiv.org/html/2605.28920#bib.bib31)\), discussed further in[Section 5](https://arxiv.org/html/2605.28920#S5), constitutes an instance of this configuration\.

In the fourth row,𝐂λ\\mathbf\{C\}\_\{\\lambda\}always outputs a single element\(𝒮=𝒴\)\(\{\\mathcal\{S\}\}=\\mathcal\{Y\}\); this is appropriate when the goal is for this element to have high admissibility \(the aggregation function becomes irrelevant as long asagg​\(y\)=y\\texttt\{agg\}\(y\)=y\)\. Here, since nesting does not hold,𝐂λ\\mathbf\{C\}\_\{\\lambda\}becoming increasingly conservative asλ\\lambdagrows becomes contingent onℙ𝐆\\mathbb\{P\}\_\{\\mathbf\{G\}\}; thus, formally verifying property\(b​1\)\(b1\)from[Definition 1](https://arxiv.org/html/2605.28920#Thmdefinition1)is harder\. Regardless,γ\\gamma\-sensibility should still hold provided thatτ​\(x,𝐲,λ\)\\tau\(x,\\mathbf\{y\},\\lambda\)is non\-decreasing inλ\\lambdaand thatℙ𝐆\\mathbb\{P\}\_\{\\mathbf\{G\}\}was chosen appropriately\. For example, if𝐘\\mathbf\{Y\}consists of responses generated by progressively more capable LLMs,𝐂λ\\mathbf\{C\}\_\{\\lambda\}would return the first response believed to be correct\. While a stronger LLM might occasionally produce a worse answer than a weaker one—preventingλ↦A​\(λ\)\\lambda\\mapsto A\(\\lambda\)from being almost surely non\-decreasing—the required monotonicity may still hold in conditional expectation\.

Notably, all examples presented here are specifically constructed to ensure the right\-continuity of the mappingλ↦A​\(λ\)\\lambda\\mapsto A\(\\lambda\), thus satisfying property\(b​2\)\(b2\)of[Definition 1](https://arxiv.org/html/2605.28920#Thmdefinition1)\. This property is delicate; for instance, replacing strict inequalities with non\-strict ones \(or vice versa\) within our definitions would generally violate this requirement\.

Table 1:Representative configurations of Conf\-Gen\.Selection functionCompatible with𝐂λ​\(x,𝐲\)\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)accumaggPartial generation𝐲:τ​\(x,𝐲,λ\)\\mathbf\{y\}\_\{:\\tau\(x,\\mathbf\{y\},\\lambda\)\}max\\max/summax\\maxYes\{yt∣St↓≤λ\}\\\{y\_\{t\}\\mid S\_\{t\}^\{\\downarrow\}\\leq\\lambda\\\}N/Amax\\maxNo\{yt∣St↑\>λ\}\\\{y\_\{t\}\\mid S\_\{t\}^\{\\uparrow\}\>\\lambda\\\}N/Amin\\minNoyτ​\(x,𝐲,λ\)y\_\{\\tau\(x,\\mathbf\{y\},\\lambda\)\}max\\maxmax\\max/min\\minYes

## Section 5Related Work as Instances of Conf\-Gen

UQ remains a more open problem for generative models than for supervised learning, with current research largely concentrated on LLMs\. One prominent direction involves*verbalized uncertainty*, where the model is prompted to provide a scalar estimate of its own confidence\(Kadavath et al\.,[2022](https://arxiv.org/html/2605.28920#bib.bib21); Yang et al\.,[2024b](https://arxiv.org/html/2605.28920#bib.bib62)\)\. Another category is based on having the LLM generate multiple answers and aggregating the variability of these answers into a scalar measure of uncertainty\(Wang et al\.,[2023b](https://arxiv.org/html/2605.28920#bib.bib56); Kuhn et al\.,[2023](https://arxiv.org/html/2605.28920#bib.bib24); Lin et al\.,[2024](https://arxiv.org/html/2605.28920#bib.bib29); Grewal et al\.,[2024](https://arxiv.org/html/2605.28920#bib.bib14); Hou et al\.,[2024](https://arxiv.org/html/2605.28920#bib.bib18); Yang et al\.,[2024a](https://arxiv.org/html/2605.28920#bib.bib61); Gao et al\.,[2024](https://arxiv.org/html/2605.28920#bib.bib13); Qiu & Miikkulainen,[2024](https://arxiv.org/html/2605.28920#bib.bib39); Nikitin et al\.,[2024](https://arxiv.org/html/2605.28920#bib.bib32); Wang & Holmes,[2025](https://arxiv.org/html/2605.28920#bib.bib57); Ross et al\.,[2026](https://arxiv.org/html/2605.28920#bib.bib45)\)\. These methods provide no formal guarantees and are orthogonal to Conf\-Gen; indeed, they can in principle be used to define score functions within Conf\-Gen\.

Several works have applied conformal ideas to generative models\.Teneggi et al\. \([2023](https://arxiv.org/html/2605.28920#bib.bib50)\)apply CRC to image\-to\-image regression tasks with diffusion models\(Sohl\-Dickstein et al\.,[2015](https://arxiv.org/html/2605.28920#bib.bib48); Ho et al\.,[2020](https://arxiv.org/html/2605.28920#bib.bib17); Song et al\.,[2021](https://arxiv.org/html/2605.28920#bib.bib49)\), andQuach et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib40)\)andKladny et al\. \([2025](https://arxiv.org/html/2605.28920#bib.bib23)\)apply it to LLMs as in our running example from the previous section, i\.e\., they aim to produce a set of answers to a questionXXsuch that with probability at leastγ\\gamma, the set contains at least one correct answer\. We highlight a few key differences between these two works and ours\. First, their admissibility function does not depend onXX\. They thus always assume access to a ground truth answer, and their admissibility function is limited to asking an LLM whether the generated answers match the ground truth\. Second, these works sequentially filter generated sets of answers, e\.g\., to ensure that each answer has high likelihood and that no two answers in the set are too similar \(as measured by distance on some embedding space\)\. This choice effectively parameterizes𝐂λ\\mathbf\{C\}\_\{\\lambda\}with multiple values, one per filter, rather than a single scalarλ\\lambda\. Because each parameter must be calibrated independently on a disjoint subset of the data in an overly conservative way, the procedure ultimately compounds into the production of unnecessarily large conformal sets\. Although each of these fragmented steps are themselves instances of Conf\-Gen, our experiments in[Section 6](https://arxiv.org/html/2605.28920#S6)show that an instance of Conf\-Gen sidestepping this partitioning yields smaller calibrated sets\.

Mohri & Hashimoto \([2024](https://arxiv.org/html/2605.28920#bib.bib31)\)proposed*conformal factuality*, where an LLM answerY′Y^\{\\prime\}to a questionXXis decomposed into a sequence of claims𝐘=\(Y1,…,YT\)\\mathbf\{Y\}=\(Y\_\{1\},\\dots,Y\_\{T\}\)via an auxiliary LLM call\. Their algorithm produces a subsequence𝐂λ^​\(X,𝐘\)\\mathbf\{C\}\_\{\\hat\{\\lambda\}\}\(X,\\mathbf\{Y\}\)where all selected claims are factually correct with probability at leastγ\\gamma\.Kuwahara et al\. \([2025](https://arxiv.org/html/2605.28920#bib.bib25)\)adapted this framework to*conformal summarization*, where𝐘\\mathbf\{Y\}is now a sequence of sentences making up a document, and their goal is to ensure that some fractionβ\\betaof all important sentences in𝐘\\mathbf\{Y\}is retained with probability at leastγ\\gamma\. As noted in[Section 4\.3](https://arxiv.org/html/2605.28920#S4.SS3), conformal factuality is an instance of Conf\-Gen\. Conformal summarization can be similarly recovered as an instance of Conf\-Gen where the admissibility function is defined as an indicator that the fraction of recovered admissible elements meets the thresholdβ\\beta\(see[Appendix D](https://arxiv.org/html/2605.28920#A4)\)\. We also highlight that both methodologies depend on a nesting property that corresponds to the monotonicity requirement in CRC, which is relaxed in Conf\-Gen\.

Lastly, in*conformal agent error attribution*\(Feng et al\.,[2026](https://arxiv.org/html/2605.28920#bib.bib10)\),XXis a task for an AI agent,𝐘\\mathbf\{Y\}corresponds to a sequence of actions taken by the agent which failed to complete taskXX, andYGTY\_\{\\text\{GT\}\}indicates the step of the trajectory on which the agent made its first mistake\.Feng et al\. \([2026](https://arxiv.org/html/2605.28920#bib.bib10)\)show how to obtain a contiguous subsequence of𝐘\\mathbf\{Y\}guaranteed to contain the first mistake with probability at leastγ\\gamma, thus allowing to rollback the agent to the time of its first mistake: this is another instance of Conf\-Gen, where a sequence is admissible if it contains the first mistake\.

## Section 6Experiments

We validate our approach on a diverse set of tasks\. Additional details for each experiment, including the precise formulation of every task as an instance of Conf\-Gen, are provided in[Appendix E](https://arxiv.org/html/2605.28920#A5)\. Except for the experiments in[Section 6\.1](https://arxiv.org/html/2605.28920#S6.SS1), all tasks considered here are novel, representing the first instance of conformal guarantees being provided for these settings\. These tasks include both binary and scalar\-valued admissibility functions, some of which are not callable\. We also consider a case whereλ↦A​\(λ\)\\lambda\\mapsto A\(\\lambda\)is not non\-decreasing almost surely, but whereγ\\gamma\-sensibility is expected to hold in practice nonetheless\. This variety highlights the wide applicability of Conf\-Gen and of our Python package—which we used across all our experiments—as well as the practical relevance of our relaxed theoretical requirements\. Code to reproduce all our results is included alongside the package\.

All figures in this section contain two panels\. Left panels show the average admissibility on a test set as a function ofγ\\gamma, alongside the corresponding lower bound as a diagonal line\. These plots aim to empirically verify the conformal guarantee; we highlight that the few small dips below the diagonal lines do not contradict[Theorem 3](https://arxiv.org/html/2605.28920#Thmtheorem3)as we can only plot empirical averages over a finite test set for a single calibration dataset, not true expectations\. Since the conformal guarantee can be trivially achieved by always being maximally conservative, the practical usefulness of Conf\-Gen also needs to be empirically verified for each task: this is the goal of the right panels\.

### Section 6\.1Conformal Open\-Domain Question Answering

Following our running example, we consider a question answering task \(as in the top panel of[Figure 1](https://arxiv.org/html/2605.28920#S1.F1)\) where responses are generated independently\. We utilize the selection function described in the first row of[Table 1](https://arxiv.org/html/2605.28920#S4.T1)\(withaccum=max\\texttt\{accum\}=\\max\), except we remove all duplicate entries from𝐲:τ​\(x,𝐲,λ\)\\mathbf\{y\}\_\{:\\tau\(x,\\mathbf\{y\},\\lambda\)\}\. We benchmark Conf\-Gen against the method ofQuach et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib40)\)on the TriviaQA dataset\(Joshi et al\.,[2017](https://arxiv.org/html/2605.28920#bib.bib20)\)\. For a fair comparison, we adopt their experimental configuration: we use LLaMA\-13B\(Touvron et al\.,[2023](https://arxiv.org/html/2605.28920#bib.bib51)\)as the base LLM and its corresponding length\-normalized likelihood\(Johnson et al\.,[2017](https://arxiv.org/html/2605.28920#bib.bib19)\)as the score functionS↑S^\{\\uparrow\}\. Recall that here, admissibility means that the output contains at least one correct response; consequently, the conformal guarantee ensures that the selected sequence includes such a response with probability at leastγ\\gamma\.

Results are displayed in[Figure 2](https://arxiv.org/html/2605.28920#S6.F2): the left plot confirms that both Conf\-Gen and the baseline satisfy the conformal guarantee, while the right plot demonstrates that our method yields more useful outputs by producing shorter sequences across most values ofγ\\gamma\. Additionally, thanks to partial generation, this means that our method also requires fewer LLM calls to produce its output\. We provide additional details and experiments for this task in[sectionE\.1](https://arxiv.org/html/2605.28920#A5.SS1)\.

![Refer to caption](https://arxiv.org/html/2605.28920v1/x1.png)Figure 2:Results on test data for the question answering task\.
### Section 6\.2Conformal Generation of Non\-Memorized Images

Image generation models such as Stable Diffusion\(Rombach et al\.,[2022](https://arxiv.org/html/2605.28920#bib.bib43)\)have been shown to memorize some of their training data\(Wen et al\.,[2023](https://arxiv.org/html/2605.28920#bib.bib59)\)\. Reproducing memorized images during inference can pose a copyright infringement risk for model providers\(Orrick,[2023](https://arxiv.org/html/2605.28920#bib.bib34)\): this makes obtaining conformal guarantees on generated images being non\-memorized particularly appealing\.

In this task, the input is a sequence𝐗=\(X1,…,XT\)\\mathbf\{X\}=\(X\_\{1\},\\dots,X\_\{T\}\), whereX1X\_\{1\}is a prompt of lengthT−1T\-1describing an imageYGTY\_\{\\text\{GT\}\}that is memorized by Stable Diffusion v1\.5, i\.e\., when givenX1X\_\{1\}, the model produces a near\-duplicate ofYGTY\_\{\\text\{GT\}\}\. Subsequent promptsXtX\_\{t\}\(fort=2,…,Tt=2,\\dots,T\) are generated by an LLM instructed to modifyt−1t\-1tokens ofX1X\_\{1\}; we follow the procedure ofRoss et al\. \([2025](https://arxiv.org/html/2605.28920#bib.bib44)\), which selects tokens deemed to contribute more strongly to memorization first\. The sequence𝐘\\mathbf\{Y\}contains images generated from these perturbed prompts\. We employ the selection function from the final row of[Table 1](https://arxiv.org/html/2605.28920#S4.T1), with the scoreS↑S^\{\\uparrow\}defined as the negative average norm of the classifier\-free guidance term\(Ho & Salimans,[2022](https://arxiv.org/html/2605.28920#bib.bib16)\), which has been shown to be indicative of memorization\(Wen et al\.,[2023](https://arxiv.org/html/2605.28920#bib.bib59)\)\.

![Refer to caption](https://arxiv.org/html/2605.28920v1/x2.png)Figure 3:Results on test data for generation of non\-memorized images\.We rely on the memorized images fromWebster \([2023](https://arxiv.org/html/2605.28920#bib.bib58)\)and had their associated generations annotated by human evaluators\. Specifically, we had1010human evaluators assess each generated image and label it as either “good”, “medium”, or “bad”\. Images labeled as “good” are those which retain the semantic content of the ground truth image, but did not memorize it; those labeled as “medium”, while not memorized, are simply unrelated to the ground truth image; and those labeled as “bad” memorized the ground truth image\. The admissibilityAAis a scalar given by the fraction of1010human evaluators who classify the generated image as non\-memorized relative to the ground truth \(i\.e\., “good” or “medium”\)\. Consequently, the conformal guarantee ensures that, in expectation, the selected image would be classified as non\-memorized by at least a fractionγ\\gammaof human evaluators\. The bottom panel of[Figure 1](https://arxiv.org/html/2605.28920#S1.F1)provides a simplified illustration of this task\.

Note that in this task,λ↦A​\(λ\)\\lambda\\mapsto A\(\\lambda\)need not be non\-decreasing, as an image corresponding to a more heavily modified prompt could, in principle, be deemed more memorized than one corresponding to a less modified prompt\. We should of course not expect this to happen often, and thus theγ\\gamma\-sensibility assumption remains reasonable\.

Results are shown in[Figure 3](https://arxiv.org/html/2605.28920#S6.F3)\. The left panel verifies the conformal guarantee\. The right panel displays the average fraction of human evaluators who assessed the selected image as too different from the ground truth \(i\.e\., “medium”\)\. The gap between the left and right plots corresponds to the percentage of “good” images\. This gap being large highlights the usefulness of Conf\-Gen: it avoids trivially achieving large admissibility by always generating “medium” images\. More details are provided in[sectionE\.2](https://arxiv.org/html/2605.28920#A5.SS2)\.

### Section 6\.3Conformal Conversational AI Chatbot

In this task, the input is also a sequence𝐗=\(X1,…,XT\)\\mathbf\{X\}=\(X\_\{1\},\\dots,X\_\{T\}\), whereX1X\_\{1\}is now a potentially ambiguous question asked to an AI chatbot\. At each steptt, an LLM generates a candidate answerYtY\_\{t\}toXtX\_\{t\}; the chatbot then either providesYtY\_\{t\}to the user or requests a clarified version of the prompt,Xt\+1X\_\{t\+1\}, which incorporates an additional piece of information\. We use the selection function given by the last row of[Table 1](https://arxiv.org/html/2605.28920#S4.T1): we interpret the output beingyτ​\(x,𝐲,λ\)y\_\{\\tau\(x,\\mathbf\{y\},\\lambda\)\}as meaning that the chatbot answered the question in stepτ​\(x,𝐲,λ\)\\tau\(x,\\mathbf\{y\},\\lambda\)and requested clarification in all previous steps\. The score is obtained by instructing an LLM to provide a value quantifying how ambiguous a question is\. The admissibility of a conversation is a binary label equal to11if the last question in the conversation is unambiguous\. Thus, our conformal guarantee ensures that, with probability at leastγ\\gamma, the chatbot has asked enough clarifying questions before answering\.

We use ClariQ\(Aliannejadi et al\.,[2021](https://arxiv.org/html/2605.28920#bib.bib1)\), a dataset of questions and clarifications\. The results are presented in[Figure 4](https://arxiv.org/html/2605.28920#S6.F4)\. The left panel verifies the empirical coverage of our conformal guarantee\. The right panel illustrates the expected trade\-off: asγ\\gammaincreases, the chatbot needs to ask more clarifications to ensure admissibility\. Importantly, the chatbot does not default to always asking the maximal number of clarifications, indicating the usefulness of Conf\-Gen\. More details are provided in[sectionE\.3](https://arxiv.org/html/2605.28920#A5.SS3)\.

![Refer to caption](https://arxiv.org/html/2605.28920v1/x3.png)Figure 4:Results on test data for the conversational task\.
### Section 6\.4Conformal Agentic AI for Web\-Based Tasks

We consider the WebVoyager dataset\(He et al\.,[2024](https://arxiv.org/html/2605.28920#bib.bib15)\), whereXXis a task for an AI agent with access to the web\. These tasks simulate common user objectives, such as locating a research paper on arXiv, reserving a hotel on booking\.com, or searching for a specific place on Google\. Each generation𝐘t\\mathbf\{Y\}\_\{t\}is itself a trajectory of actions taken by the agent, with each𝐘t\\mathbf\{Y\}\_\{t\}being generated independently\. We use the selection function from the first row of[Table 1](https://arxiv.org/html/2605.28920#S4.T1)\(withaccum=max\\texttt\{accum\}=\\max\), where the score functionS↑​\(x,𝐲t\)S^\{\\uparrow\}\(x,\\mathbf\{y\}\_\{t\}\)is obtained by instructing an LLM to assess the percentage at which𝐲t\\mathbf\{y\}\_\{t\}has completed taskxx\. The admissibility function is a binary indicator of whether the task is successfully completed by the end of the trajectory \(determined by an LLM judge\)\. Thus, the conformal guarantee ensures that the selected sequence contains at least one successful trajectory with probability at leastγ\\gamma\(as assessed by the LLM judge, whichHe et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib15)\)tested thoroughly\)\.

We followHe et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib15)\)to instantiate the AI agent, and report results in[Figure 5](https://arxiv.org/html/2605.28920#S6.F5)\. While the agent achieves a single\-attempt success rate near60%60\\%, Conf\-Gen formally guarantees above65%65\\%success averaging under two attempts, rising to nearly80%80\\%when more are permitted\. More details are provided in[sectionE\.4](https://arxiv.org/html/2605.28920#A5.SS4)\.

We believe that this task could help enhance the reliability of AI agents\. For example, the agent could execute all its trajectories in a simulated environment, so that each trajectory corresponds to a plan\. The user could then be shown the corresponding subset of selected trajectories and choose which plan to execute, if any\. Such a procedure could help ensure that, for example, an agent tasked with booking a hotel online does not incur a financial loss when it makes a mistake\.

![Refer to caption](https://arxiv.org/html/2605.28920v1/x4.png)Figure 5:Results on test data for the agentic AI tasks\.
### Section 6\.5Conformal Random Forests

To further demonstrate the generality of Conf\-Gen, we consider a task in supervised learning beyond the scope of generative models\. We assume access to a pre\-trained random forest classifier withTTtrees, whereXXis a feature vector,YtY\_\{t\}contains the prediction from thett\-th tree, andYGTY\_\{\\text\{GT\}\}is the ground truth label\. Our selection function outputs the smallest subset of trees whose accumulated score \(withaccum=sum\\texttt\{accum\}=\\text\{sum\}\) exceedsλ\\lambda, where the score functionS↑S^\{\\uparrow\}is given by the weighted tree sample count\. The selected trees are then used to make a prediction, ignoring the other trees\. Admissibility is defined via the binary variableAt′A^\{\\prime\}\_\{t\}, which indicates whether thett\-th tree correctly predictsYGTY\_\{\\text\{GT\}\}forXX, and the aggregation functionagg​\(𝐚′\)=𝟙​\(∑a′∈𝐚′a′≥k\)\\texttt\{agg\}\(\\mathbf\{a\}^\{\\prime\}\)=\\mathds\{1\}\(\\sum\_\{a^\{\\prime\}\\in\\mathbf\{a^\{\\prime\}\}\}a^\{\\prime\}\\geq k\)for a user\-specifiedkk\. Consequently, Conf\-Gen guarantees that the probability of the selected subset containing at leastkktrees which make the correct prediction is at leastγ\\gamma\. This guarantee is particularly meaningful when the selected set contains at most2​k−12k\-1trees, as it then ensures the correctness of the majority\-vote prediction\.

While defining admissibility directly as the correctness of the final prediction might seem more intuitive—as the resulting conformal guarantee would then ensure that the prediction is correct with probability at leastγ\\gamma—such a formulation fails to ensure the monotonicity ofλ↦A​\(λ\)\\lambda\\mapsto A\(\\lambda\), even in conditional expectation\. Our chosen definition ofAAavoids this issue, and satisfies the necessary theoretical properties while still providing a meaningful conformal guarantee\.

We pre\-train a random forest withT=100T=100trees on the Click\_prediction\_small dataset\(Coutinho,[2022](https://arxiv.org/html/2605.28920#bib.bib9)\)and setk=30k=30\. The results in[Figure 6](https://arxiv.org/html/2605.28920#S7.F6)demonstrate that the conformal guarantee is empirically satisfied and that the average number of selected trees remains below2​k−1=592k\-1=59across a wide range ofγ\\gammavalues\. Additional details, experiments on more datasets, and ablations overkkare provided in[sectionE\.5](https://arxiv.org/html/2605.28920#A5.SS5)\.

## Section 7Conclusion and Future Work

In this work, we presented Conf\-Gen, a framework that extends CRC to generative tasks while relaxing its underlying theoretical assumptions\. We also provide a Python package to facilitate the deployment of Conf\-Gen across diverse settings\. Our experiments demonstrate that Conf\-Gen not only outperforms current state\-of\-the\-art conformal methods in open\-domain question answering but also establishes a foundation for a wide array of novel conformal applications\.

We hope that our work will catalyze the adoption of conformal guarantees for UQ in generative modelling\. Future work may focus on refining the data\-generating processes, score functions, and selection functions we proposed for our considered tasks\. Beyond these potential technical improvements, exploring new use cases for Conf\-Gen remains a compelling research direction\.

![Refer to caption](https://arxiv.org/html/2605.28920v1/x5.png)Figure 6:Results on test data for the random forest\.
## Impact Statement

Our work presents a step towards improved uncertainty quantification in generative models\. These type of improvements could have various societal consequences\. On the positive side, they could increase the overall reliability of AI systems, help deploy them in safety\-critical applications, and increase trust in them; indeed, these are some of the key motivations behind our work\. On the negative side, uncertainty quantification can also improve the reliability of malicious AI applications\. Overall, we believe that uncertainty quantification for generative models is a meaningful research topic precisely because it is likely to have more positive than negative impact\. Although we do not believe that any of the experiments in this paper are likely to have direct societal consequences, future applications might\.

## References

- Aliannejadi et al\. \(2021\)Aliannejadi, M\., Kiseleva, J\., Chuklin, A\., Dalton, J\., and Burtsev, M\.Building and evaluating open\-domain dialogue corpora with clarifying questions\.In*Conference on Empirical Methods in Natural Language Processing*, 2021\.
- Angelopoulos & Bates \(2021\)Angelopoulos, A\. N\. and Bates, S\.A gentle introduction to conformal prediction and distribution\-free uncertainty quantification\.*arXiv:2107\.07511*, 2021\.
- Angelopoulos et al\. \(2024a\)Angelopoulos, A\. N\., Barber, R\. F\., and Bates, S\.Theoretical foundations of conformal prediction\.*arXiv:2411\.11824*, 2024a\.
- Angelopoulos et al\. \(2024b\)Angelopoulos, A\. N\., Bates, S\., Fisch, A\., Lei, L\., and Schuster, T\.Conformal risk control\.In*International Conference on Learning Representations*, 2024b\.
- Becker & Kohavi \(1996\)Becker, B\. and Kohavi, R\.Adult\.UCI Machine Learning Repository, 1996\.
- Begoli et al\. \(2019\)Begoli, E\., Bhattacharya, T\., and Kusnezov, D\.The need for uncertainty quantification in machine\-assisted medical decision making\.In*Nature Machine Intelligence*, volume 1, pp\. 20–23, 2019\.
- Breiman \(2001\)Breiman, L\.Random forests\.In*Machine learning*, volume 45, pp\. 5–32\. Springer, 2001\.
- Brown et al\. \(2020\)Brown, T\., Mann, B\., Ryder, N\., Subbiah, M\., Kaplan, J\. D\., Dhariwal, P\., Neelakantan, A\., Shyam, P\., Sastry, G\., Askell, A\., Agarwal, S\., Herbert\-Voss, A\., Krueger, G\., Henighan, T\., Child, R\., Ramesh, A\., Ziegler, D\., Wu, J\., Winter, C\., Hesse, C\., Chen, M\., Sigler, E\., Litwin, M\., Gray, S\., Chess, B\., Clark, J\., Berner, C\., McCandlish, S\., Radford, A\., Sutskever, I\., and Amodei, D\.Language models are few\-shot learners\.In*Advances in Neural Information Processing Systems*, 2020\.
- Coutinho \(2022\)Coutinho, F\.Click prediction\.[https://kaggle\.com/competitions/click\-prediction\-cds](https://kaggle.com/competitions/click-prediction-cds), 2022\.Kaggle\.
- Feng et al\. \(2026\)Feng, N\., Sui, Y\., Hou, S\., Wu, G\., and Cresswell, J\. C\.Conformal agent error attribution\.*arXiv:2605\.06788*, 2026\.
- Gammerman & Vovk \(2007\)Gammerman, A\. and Vovk, V\.Hedging predictions in machine learning\.*The Computer Journal*, 50\(2\):151–163, 2007\.
- Gammerman et al\. \(1998\)Gammerman, A\., Vovk, V\., and Vapnik, V\.Learning by transduction\.In*Conference on Uncertainty in Artificial Intelligence*, 1998\.
- Gao et al\. \(2024\)Gao, X\., Zhang, J\., Mouatadid, L\., and Das, K\.SPUQ: Perturbation\-based uncertainty quantification for large language models\.In*Conference of the European Chapter of the Association for Computational Linguistics*, 2024\.
- Grewal et al\. \(2024\)Grewal, Y\. S\., Bonilla, E\. V\., and Bui, T\. D\.Improving uncertainty quantification in large language models via semantic embeddings\.*arXiv:2410\.22685*, 2024\.
- He et al\. \(2024\)He, H\., Yao, W\., Ma, K\., Yu, W\., Dai, Y\., Zhang, H\., Lan, Z\., and Yu, D\.WebVoyager: Building an end\-to\-end web agent with large multimodal models\.In*Annual Meeting of the Association for Computational Linguistics*, pp\. 6864–6890, 2024\.
- Ho & Salimans \(2022\)Ho, J\. and Salimans, T\.Classifier\-free diffusion guidance\.*arXiv:2207\.12598*, 2022\.
- Ho et al\. \(2020\)Ho, J\., Jain, A\., and Abbeel, P\.Denoising diffusion probabilistic models\.In*Advances in Neural Information Processing Systems*, 2020\.
- Hou et al\. \(2024\)Hou, B\., Liu, Y\., Qian, K\., Andreas, J\., Chang, S\., and Zhang, Y\.Decomposing uncertainty for large language models through input clarification ensembling\.In*International Conference on Machine Learning*, 2024\.
- Johnson et al\. \(2017\)Johnson, M\., Schuster, M\., Le, Q\. V\., Krikun, M\., Wu, Y\., Chen, Z\., Thorat, N\., Viégas, F\., Wattenberg, M\., Corrado, G\., Hughes, M\., and Dean, J\.Google’s multilingual neural machine translation system: Enabling zero\-shot translation\.*Transactions of the Association for Computational Linguistics*, 5:339–351, 2017\.
- Joshi et al\. \(2017\)Joshi, M\., Choi, E\., Weld, D\. S\., and Zettlemoyer, L\.Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension\.*arXiv:1705\.03551*, 2017\.
- Kadavath et al\. \(2022\)Kadavath, S\., Conerly, T\., Askell, A\., Henighan, T\., Drain, D\., Perez, E\., Schiefer, N\., Hatfield\-Dodds, Z\., DasSarma, N\., Tran\-Johnson, E\., et al\.Language models \(mostly\) know what they know\.*arXiv:2207\.05221*, 2022\.
- Kamkari et al\. \(2024\)Kamkari, H\., Ross, B\. L\., Hosseinzadeh, R\., Cresswell, J\. C\., and Loaiza\-Ganem, G\.A geometric view of data complexity: Efficient local intrinsic dimension estimation with diffusion models\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.
- Kladny et al\. \(2025\)Kladny, K\.\-R\., Schölkopf, B\., and Muehlebach, M\.Conformal generative modeling with improved sample efficiency through sequential greedy filtering\.In*International Conference on Learning Representations*, 2025\.
- Kuhn et al\. \(2023\)Kuhn, L\., Gal, Y\., and Farquhar, S\.Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation\.In*International Conference on Learning Representations*, 2023\.
- Kuwahara et al\. \(2025\)Kuwahara, B\., Lin, C\.\-Y\., Huang, X\. S\., Leung, K\. K\., Yapeter, J\. A\., Stanevich, I\., Perez, F\., and Cresswell, J\. C\.Document summarization with conformal importance guarantees\.*arXiv:2509\.20461*, 2025\.
- Laufer\-Goldshtein et al\. \(2023\)Laufer\-Goldshtein, B\., Fisch, A\., Barzilay, R\., and Jaakkola, T\. S\.Efficiently controlling multiple risks with pareto testing\.In*International Conference on Learning Representations*, 2023\.
- Lei et al\. \(2018\)Lei, J\., G’Sell, M\., Rinaldo, A\., Tibshirani, R\. J\., and Wasserman, L\.Distribution\-free predictive inference for regression\.In*Journal of the American Statistical Association*, volume 113, pp\. 1094–1111, 2018\.
- Leung et al\. \(2025\)Leung, K\. K\., Hosseinzadeh, R\., and Loaiza\-Ganem, G\.On convolutions, intrinsic dimension, and diffusion models\.In*Transactions on Machine Learning Research*, 2025\.
- Lin et al\. \(2024\)Lin, Z\., Trivedi, S\., and Sun, J\.Generating with confidence: Uncertainty quantification for black\-box large language models\.In*Transactions on Machine Learning Research*, 2024\.
- Madeo et al\. \(2013\)Madeo, R\., Wagner, P\., and Peres, S\.Gesture Phase Segmentation\.UCI Machine Learning Repository, 2013\.
- Mohri & Hashimoto \(2024\)Mohri, C\. and Hashimoto, T\.Language models with conformal factuality guarantees\.In*International Conference on Machine Learning*, 2024\.
- Nikitin et al\. \(2024\)Nikitin, A\., Kossen, J\., Gal, Y\., and Marttinen, P\.Kernel language entropy: Fine\-grained uncertainty quantification for LLMs from semantic similarities\.In*Advances in Neural Information Processing Systems*, 2024\.
- OpenAI \(2024\)OpenAI\.Gpt\-4 technical report\.*arXiv:2303\.08774*, 2024\.
- Orrick \(2023\)Orrick, W\. H\.Andersen v\. Stability AI Ltd\., 2023\.URL[https://casetext\.com/case/andersen\-v\-stability\-ai\-ltd](https://casetext.com/case/andersen-v-stability-ai-ltd)\.
- Papadopoulos et al\. \(2002\)Papadopoulos, H\., Proedrou, K\., Vovk, V\., and Gammerman, A\.Inductive confidence machines for regression\.In*European Conference on Machine Learning*, 2002\.
- Park et al\. \(2023\)Park, J\. S\., O’Brien, J\., Cai, C\. J\., Morris, M\. R\., Liang, P\., and Bernstein, M\. S\.Generative agents: Interactive simulacra of human behavior\.In*Proceedings of the annual ACM symposium on user interface software and technology*, 2023\.
- Pedregosa et al\. \(2011\)Pedregosa, F\., Varoquaux, G\., Gramfort, A\., Michel, V\., Thirion, B\., Grisel, O\., Blondel, M\., Prettenhofer, P\., Weiss, R\., Dubourg, V\., Vanderplas, J\., Passos, A\., Cournapeau, D\., Brucher, M\., Perrot, M\., and Duchesnay, E\.Scikit\-learn: Machine learning in Python\.*Journal of Machine Learning Research*, 12:2825–2830, 2011\.
- Potapenko et al\. \(2021\)Potapenko, A\. B\., Meyer, C\., Kohl, S\. A\., Ballard, A\. J\., Cowie, A\., Romera\-Paredes, B\., Nikolov, S\., Jain, R\., and Hassabis, D\.Highly accurate protein structure prediction with AlphaFold\.In*Nature*, volume 596, pp\. 583–589, 2021\.
- Qiu & Miikkulainen \(2024\)Qiu, X\. and Miikkulainen, R\.Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space\.In*Advances in Neural Information Processing Systems*, 2024\.
- Quach et al\. \(2024\)Quach, V\., Fisch, A\., Schuster, T\., Yala, A\., Sohn, J\. H\., Jaakkola, T\. S\., and Barzilay, R\.Conformal language modeling\.In*International Conference on Learning Representations*, 2024\.
- Ramesh et al\. \(2022\)Ramesh, A\., Dhariwal, P\., Nichol, A\., Chu, C\., and Chen, M\.Hierarchical text\-conditional image generation with CLIP latents\.*arXiv:2204\.06125*, 2022\.
- Roe \(2005\)Roe, B\.MiniBooNE particle identification\.UCI Machine Learning Repository, 2005\.
- Rombach et al\. \(2022\)Rombach, R\., Blattmann, A\., Lorenz, D\., Esser, P\., and Ommer, B\.High\-resolution image synthesis with latent diffusion models\.In*Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 2022\.
- Ross et al\. \(2025\)Ross, B\. L\., Kamkari, H\., Wu, T\., Hosseinzadeh, R\., Liu, Z\., Stein, G\., Cresswell, J\. C\., and Loaiza\-Ganem, G\.A geometric framework for understanding memorization in generative models\.In*International Conference on Learning Representations*, 2025\.
- Ross et al\. \(2026\)Ross, B\. L\., Vouitsis, N\., Ghomi, A\. A\., Hosseinzadeh, R\., Xin, J\., Liu, Z\., Sui, Y\., Hou, S\., Leung, K\. K\., Loaiza\-Ganem, G\., and Cresswell, J\. C\.Textual Bayes: Quantifying prompt uncertainty in LLM\-based systems\.In*International Conference on Learning Representations*, 2026\.
- Saunders et al\. \(1999\)Saunders, C\., Gammerman, A\., and Vovk, V\.Transduction with confidence and credibility\.In*International Joint Conference on Artificial Intelligence*, 1999\.
- Silver et al\. \(2016\)Silver, D\., Huang, A\., Maddison, C\. J\., Guez, A\., Sifre, L\., van den Driessche, G\., Schrittwieser, J\., Antonoglou, I\., Panneershelvam, V\., Lanctot, M\., Dieleman, S\., Grewe, D\., Nham, J\., Kalchbrenner, N\., Sutskever, I\., Lillicrap, T\., Leach, M\., Kavukcuoglu, K\., Graepel, T\., and Hassabis, D\.Mastering the game of Go with deep neural networks and tree search\.In*Nature*, volume 529, pp\. 484–489, 2016\.
- Sohl\-Dickstein et al\. \(2015\)Sohl\-Dickstein, J\., Weiss, E\., Maheswaranathan, N\., and Ganguli, S\.Deep unsupervised learning using nonequilibrium thermodynamics\.In*International Conference on Machine Learning*, 2015\.
- Song et al\. \(2021\)Song, Y\., Sohl\-Dickstein, J\., Kingma, D\. P\., Kumar, A\., Ermon, S\., and Poole, B\.Score\-based generative modeling through stochastic differential equations\.In*International Conference on Learning Representations*, 2021\.
- Teneggi et al\. \(2023\)Teneggi, J\., Tivnan, M\., Stayman, W\., and Sulam, J\.How to trust your diffusion model: A convex optimization approach to conformal risk control\.In*International Conference on Machine Learning*, 2023\.
- Touvron et al\. \(2023\)Touvron, H\., Lavril, T\., Izacard, G\., Martinet, X\., Lachaux, M\.\-A\., Lacroix, T\., Rozière, B\., Goyal, N\., Hambro, E\., Azhar, F\., et al\.Llama: Open and efficient foundation language models\.*arXiv:2302\.13971*, 2023\.
- U\.S\. Census Bureau \(2000\)U\.S\. Census Bureau\.Census\-Income \(KDD\)\.UCI Machine Learning Repository, 2000\.
- Vovk et al\. \(1999\)Vovk, V\., Gammerman, A\., and Saunders, C\.Machine\-learning applications of algorithmic randomness\.In*International Conference on Machine Learning*, 1999\.
- Vovk et al\. \(2005\)Vovk, V\., Gammerman, A\., and Shafer, G\.*Algorithmic learning in a random world*\.Springer, 2005\.
- Wang et al\. \(2023a\)Wang, H\., Fu, T\., Du, Y\., Gao, W\., Huang, K\., Liu, Z\., Chandak, P\., Liu, S\., Van Katwyk, P\., Deac, A\., Anandkumar, A\., Bergen, K\., Gomes, C\., Ho, S\., Kohli, P\., Lasenby, J\., Leskovec, J\., Liu, T\., Manrai, A\., Marks, D\., Ramsundar, B\., Song, L\., Sun, J\., Tang, J\., Veličković, P\., Welling, M\., Zhang, L\., Coley, C\., Bengio, Y\., and Zitnik, M\.Scientific discovery in the age of artificial intelligence\.In*Nature*, volume 620, pp\. 47–60, 2023a\.
- Wang et al\. \(2023b\)Wang, X\., Aitchison, L\., and Rudolph, M\.LoRa ensembles for large language model fine\-tuning\.*arXiv:2310\.00035*, 2023b\.
- Wang & Holmes \(2025\)Wang, Z\. and Holmes, C\.On subjective uncertainty quantification and calibration in natural language generation\.In*International Conference on Learning Representations*, 2025\.
- Webster \(2023\)Webster, R\.A reproducible extraction of training images from diffusion models\.*arXiv:2305\.08694*, 2023\.
- Wen et al\. \(2023\)Wen, Y\., Liu, Y\., Chen, C\., and Lyu, L\.Detecting, explaining, and mitigating memorization in diffusion models\.In*The Twelfth International Conference on Learning Representations*, 2023\.
- Wu et al\. \(2016\)Wu, Y\., Schuster, M\., Chen, Z\., Le, Q\. V\., Norouzi, M\., Macherey, W\., Krikun, M\., Cao, Y\., Gao, Q\., Macherey, K\., Klingner, J\., Shah, A\., Johnson, M\., Liu, X\., Łukasz Kaiser, Gouws, S\., Kato, Y\., Kudo, T\., Kazawa, H\., Stevens, K\., Kurian, G\., Patil, N\., Wang, W\., Young, C\., Smith, J\., Riesa, J\., Rudnick, A\., Vinyals, O\., Corrado, G\., Hughes, M\., and Dean, J\.Google’s neural machine translation system: Bridging the gap between human and machine translation\.*arXiv:2302\.13971*, 2016\.
- Yang et al\. \(2024a\)Yang, A\. X\., Robeyns, M\., Wang, X\., and Aitchison, L\.Bayesian low\-rank adaptation for large language models\.In*International Conference on Learning Representations*, 2024a\.
- Yang et al\. \(2024b\)Yang, D\., Tsai, Y\.\-H\. H\., and Yamada, M\.On verbalized confidence scores for LLMs\.*arXiv:2412\.14737*, 2024b\.

## Appendix AReview of Existing Upper Bounds

As mentioned in the main text, there are accompanying results to those in[Theorem 1](https://arxiv.org/html/2605.28920#Thmtheorem1)and[Theorem 2](https://arxiv.org/html/2605.28920#Thmtheorem2)which provide upper bounds\. We begin by reviewing the upper bound for CP\.

###### Theorem 4\(Angelopoulos et al\. \([2024a](https://arxiv.org/html/2605.28920#bib.bib3)\)\)\.

Under the conditions of[Theorem 1](https://arxiv.org/html/2605.28920#Thmtheorem1), we have that

ℙ​\(YGT\(n\+1\)∈𝒞λ^​\(X\(n\+1\)\)\)≤γ\+1n\+1\+ϵ,\{\\mathbb\{P\}\}\\left\(Y^\{\(n\+1\)\}\_\{\\textnormal\{GT\}\}\\in\\mathcal\{C\}\_\{\\hat\{\\lambda\}\}\\left\(X^\{\(n\+1\)\}\\right\)\\right\)\\leq\\gamma\+\\dfrac\{1\}\{n\+1\}\+\\epsilon,\(13\)whereϵ\\epsilonis the probability of the\(n\+1\)t​h\(n\+1\)^\{th\}score being equal to another score, i\.e\.,

ϵ≔ℙ​\(S↓​\(X\(j\),YGT\(j\)\)=S↓​\(X\(n\+1\),YGT\(n\+1\)\)​for some​j∈\{1,…,n\}\)\.\\epsilon\\coloneqq\{\\mathbb\{P\}\}\\left\(S^\{\\downarrow\}\\left\(X^\{\(j\)\},Y\_\{\\textnormal\{GT\}\}^\{\(j\)\}\\right\)=S^\{\\downarrow\}\\left\(X^\{\(n\+1\)\},Y\_\{\\textnormal\{GT\}\}^\{\(n\+1\)\}\\right\)\\textnormal\{ for some \}j\\in\\\{1,\\dots,n\\\}\\right\)\.\(14\)

The first such upper bound was proved byLei et al\. \([2018](https://arxiv.org/html/2605.28920#bib.bib27)\)under the assumption that the scores follow a continuous joint distribution, meaning thatϵ=0\\epsilon=0\.

CRC also admits an upper bound, which requires the notion of a càdlàg function and its jumps\.

###### Definition 2\.

ConsiderΛ=\[λmin,λmax\]\\Lambda=\[\\lambda\_\{\\textnormal\{min\}\},\\lambda\_\{\\textnormal\{max\}\}\]\. The functionf:Λ→ℝf:\\Lambda\\rightarrow\\mathbb\{R\}is called*càdlàg*if it is right\-continuous andlimϵ→0\+f​\(λ−ϵ\)\\lim\_\{\\epsilon\\rightarrow 0^\{\+\}\}f\(\\lambda\-\\epsilon\)exists for everyλ∈\(λmin,λmax\]\\lambda\\in\(\\lambda\_\{\\textnormal\{min\}\},\\lambda\_\{\\textnormal\{max\}\}\]\. In this case, the jump offfatλ∈Λ\\lambda\\in\\Lambdais defined as

h​\(f,λ\)≔\{f​\(λ\)−limϵ→0\+f​\(λ−ϵ\),if​λ∈\(λmin,λmax\]\.f​\(λ\),if​λ=λmin\.h\(f,\\lambda\)\\coloneqq\\begin\{cases\}\\displaystyle f\(\\lambda\)\-\\lim\_\{\\epsilon\\rightarrow 0^\{\+\}\}f\(\\lambda\-\\epsilon\),&\\text\{if \}\\lambda\\in\(\\lambda\_\{\\textnormal\{min\}\},\\lambda\_\{\\textnormal\{max\}\}\]\.\\\\ f\(\\lambda\),&\\text\{if \}\\lambda=\\lambda\_\{\\textnormal\{min\}\}\.\\end\{cases\}\(15\)

Note that any right\-continuous, non\-decreasing function is càdlàg, so that under the assumptions of[Theorem 2](https://arxiv.org/html/2605.28920#Thmtheorem2),U\(i\)U^\{\(i\)\}is càdlàg almost surely fori=1,…,n\+1i=1,\\dots,n\+1\. We now state the upper bound for CRC\.

###### Theorem 5\(Angelopoulos et al\. \([2024b](https://arxiv.org/html/2605.28920#bib.bib4)\)\)\.

Assume that the requirements of[Theorem 2](https://arxiv.org/html/2605.28920#Thmtheorem2)hold\. Additionally, assume thatU:2𝒴GT×𝒴GT→\[0,umax\]U:2^\{\\mathcal\{Y\}\_\{\\textnormal\{GT\}\}\}\\times\\mathcal\{Y\}\_\{\\textnormal\{GT\}\}\\rightarrow\[0,u\_\{\\textnormal\{max\}\}\], that fori=1,…,n\+1i=1,\\dots,n\+1, the random functionsU\(i\)U^\{\(i\)\}are*i\.i\.d\.*, and thatℙ​\(h​\(U\(i\),λ\)\>0\)=0\\mathbb\{P\}\(h\(U^\{\(i\)\},\\lambda\)\>0\)=0for everyλ∈Λ\\lambda\\in\\Lambda\. Then,

𝔼​\[U\(n\+1\)​\(λ^\)\]≤γ\+2​umaxn\+1\.\\mathbb\{E\}\\left\[U^\{\(n\+1\)\}\(\\hat\{\\lambda\}\)\\right\]\\leq\\gamma\+\\dfrac\{2u\_\{\\textnormal\{max\}\}\}\{n\+1\}\.\(16\)

Note that originally,Angelopoulos et al\. \([2024b](https://arxiv.org/html/2605.28920#bib.bib4)\)provided a lower bound for a loss function; the restatement we wrote above corresponds to the case where the loss is given byumaxu\_\{\\text\{max\}\}minus the utility function\. Note also that, rather than usingγ\\gammadirectly,Angelopoulos et al\. \([2024b](https://arxiv.org/html/2605.28920#bib.bib4)\)write the bound in terms ofα≔umax−γ\\alpha\\coloneqq u\_\{\\text\{max\}\}\-\\gamma\. Once again, writing the bound for CRC in terms of a utility function rather than a loss function facilitates comparing between CRC and CG\. Note also that the assumption thatℙ​\(h​\(U\(i\),λ\)\>0\)=0\\mathbb\{P\}\(h\(U^\{\(i\)\},\\lambda\)\>0\)=0for everyλ∈Λ\\lambda\\in\\Lambdais analogous to the scores having a continuous distribution in CP\. Lastly, we also note that whenU​\(𝒞,y\)=𝟙​\(y∈𝒞\)U\(\\mathcal\{C\},y\)=\\mathds\{1\}\(y\\in\\mathcal\{C\}\), the bound obtained in CRC is weaker than the corresponding bound in CP by a factor of22; this is discussed in more detail byAngelopoulos et al\. \([2024b](https://arxiv.org/html/2605.28920#bib.bib4)\)\.

## Appendix BProof of[Theorem 3](https://arxiv.org/html/2605.28920#Thmtheorem3)

We begin by restating[Definition 1](https://arxiv.org/html/2605.28920#Thmdefinition1)and[Theorem 3](https://arxiv.org/html/2605.28920#Thmtheorem3)for convenience\.

See[1](https://arxiv.org/html/2605.28920#Thmdefinition1)See[3](https://arxiv.org/html/2605.28920#Thmtheorem3)

###### Proof\.

We first prove the lower bound\. Let

λ∗≔inf𝒱​\(𝐃:n\+1,γ\)\.\\lambda^\{\*\}\\coloneqq\\inf\\mathcal\{V\}\\left\(\\mathbf\{D\}\_\{:n\+1\},\\gamma\\right\)\.\(17\)Note thatλ∗\\lambda^\{\*\}cannot be computed in practice since we do not have access to the\(n\+1\)\(n\+1\)\-th datapoint; we merely use this quantity as a theoretical construct in our proof\. Sinceλ∗∈𝒱​\(𝐃:n\+1,γ\)⊆Λ\\lambda^\{\*\}\\in\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\+1\},\\gamma\)\\subseteq\\Lambda, almost surely, thenℙ​\(λ∗∈Λ\)=1\{\\mathbb\{P\}\}\(\\lambda^\{\*\}\\in\\Lambda\)=1, so thatA\(i\)​\(λ∗\)A^\{\(i\)\}\(\\lambda^\{\*\}\)is almost surely well\-defined fori=1,…,n\+1i=1,\\dots,n\+1\. Additionally, due to the considered instance of Conf\-Gen beingγ\\gamma\-sensible,A¯n\+1​\(λ∗\)≥γ\\bar\{A\}\_\{n\+1\}\(\\lambda^\{\*\}\)\\geq\\gamma, almost surely\. Note thatλ∗\\lambda^\{\*\}is a symmetric and deterministic function ofA\(1\),…,A\(n\+1\)A^\{\(1\)\},\\dots,A^\{\(n\+1\)\}, which are exchangeable by assumption\. By standard properties of exchangeability\(Angelopoulos et al\.,[2024a](https://arxiv.org/html/2605.28920#bib.bib3)\), it follows that

𝔼​\[A\(n\+1\)​\(λ∗\)\]=𝔼​\[A\(i\)​\(λ∗\)\]\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\lambda^\{\*\}\)\\right\]=\\mathbb\{E\}\\left\[A^\{\(i\)\}\(\\lambda^\{\*\}\)\\right\]\(18\)for everyi=1,…,n\+1i=1,\\dots,n\+1\. We then have that

𝔼​\[A\(n\+1\)​\(λ∗\)\]=1n\+1​∑i=1n\+1𝔼​\[A\(i\)​\(λ∗\)\]=𝔼​\[1n\+1​∑i=1n\+1A\(i\)​\(λ∗\)\]=𝔼​\[A¯n\+1​\(λ∗\)\]≥γ\.\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\lambda^\{\*\}\)\\right\]=\\dfrac\{1\}\{n\+1\}\\sum\_\{i=1\}^\{n\+1\}\\mathbb\{E\}\\left\[A^\{\(i\)\}\(\\lambda^\{\*\}\)\\right\]=\\mathbb\{E\}\\left\[\\dfrac\{1\}\{n\+1\}\\sum\_\{i=1\}^\{n\+1\}A^\{\(i\)\}\(\\lambda^\{\*\}\)\\right\]=\\mathbb\{E\}\\left\[\\bar\{A\}\_\{n\+1\}\(\\lambda^\{\*\}\)\\right\]\\geq\\gamma\.\(19\)We now want to show thatλ∗≤λ^\\lambda^\{\*\}\\leq\\hat\{\\lambda\}, almost surely\. There are two cases: eitherλ^=∞\\hat\{\\lambda\}=\\infty, orλ^<∞\\hat\{\\lambda\}<\\infty\. In the former case, the inequality clearly holds\. Otherwise, ifλ^<∞\\hat\{\\lambda\}<\\infty, we have thatλ^∈𝒱​\(𝐃:n,n\+1n​γ\)∖\{∞\}\\hat\{\\lambda\}\\in\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\},\\tfrac\{n\+1\}\{n\}\\gamma\)\\setminus\\\{\\infty\\\}, and thus

1n​∑i=1nA\(i\)​\(λ^\)≥n\+1n​γ\\displaystyle\\dfrac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}A^\{\(i\)\}\(\\hat\{\\lambda\}\)\\geq\\dfrac\{n\+1\}\{n\}\\gamma⟹∑i=1nA\(i\)​\(λ^\)≥\(n\+1\)​γ\\displaystyle\\implies\\sum\_\{i=1\}^\{n\}A^\{\(i\)\}\(\\hat\{\\lambda\}\)\\geq\(n\+1\)\\gamma\(20\)⟹∑i=1n\+1A\(i\)​\(λ^\)≥\(n\+1\)​γ\\displaystyle\\implies\\sum\_\{i=1\}^\{n\+1\}A^\{\(i\)\}\(\\hat\{\\lambda\}\)\\geq\(n\+1\)\\gamma\(21\)⟹1n\+1​∑i=1n\+1A\(i\)​\(λ^\)≥γ\\displaystyle\\implies\\dfrac\{1\}\{n\+1\}\\sum\_\{i=1\}^\{n\+1\}A^\{\(i\)\}\(\\hat\{\\lambda\}\)\\geq\\gamma\(22\)⟹λ^∈𝒱​\(𝐃:n\+1,γ\),\\displaystyle\\implies\\hat\{\\lambda\}\\in\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\+1\},\\gamma\),\(23\)almost surely, where the second implication follows from the non\-negativity ofAA\. Sinceλ∗=inf𝒱​\(𝐃:n\+1,γ\)\\lambda^\{\*\}=\\inf\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\+1\},\\gamma\), it follows thatλ∗≤λ^\\lambda^\{\*\}\\leq\\hat\{\\lambda\}also holds in this case\. Now, we denoteB​\(λ\)≔𝔼​\[A\(n\+1\)​\(λ\)∣λ^,λ∗\]B\(\\lambda\)\\coloneqq\\mathbb\{E\}\[A^\{\(n\+1\)\}\(\\lambda\)\\mid\\hat\{\\lambda\},\\lambda^\{\*\}\]\. Lastly, since by assumptionBBis almost surely non\-decreasing, we have that

𝔼​\[A\(n\+1\)​\(λ^\)\]\\displaystyle\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\hat\{\\lambda\}\)\\right\]=𝔼\[𝔼\[A\(n\+1\)\(λ^\)\|λ^,λ∗\]\]=𝔼\[B\(λ^\)\]≥𝔼\[B\(λ∗\)\]\\displaystyle=\\mathbb\{E\}\\left\[\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\hat\{\\lambda\}\)\\;\\middle\|\\;\\hat\{\\lambda\},\\lambda^\{\*\}\\right\]\\right\]=\\mathbb\{E\}\[B\(\\hat\{\\lambda\}\)\]\\geq\\mathbb\{E\}\[B\(\\lambda^\{\*\}\)\]\(24\)=𝔼\[𝔼\[A\(n\+1\)\(λ∗\)\|λ^,λ∗\]\]=𝔼\[A\(n\+1\)\(λ∗\)\]≥γ,\\displaystyle=\\mathbb\{E\}\\left\[\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\lambda^\{\*\}\)\\;\\middle\|\\;\\hat\{\\lambda\},\\lambda^\{\*\}\\right\]\\right\]=\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\lambda^\{\*\}\)\\right\]\\geq\\gamma,\(25\)where the second and third equalities follow fromB​\(λ\)B\(\\lambda\)beingσ​\(λ^,λ∗\)\\sigma\(\\hat\{\\lambda\},\\lambda^\{\*\}\)\-measurable for everyλ∈Λ\\lambda\\in\\Lambda, whereσ​\(λ^,λ∗\)\\sigma\(\\hat\{\\lambda\},\\lambda^\{\*\}\)denotes theσ\\sigma\-algebra generated byλ^\\hat\{\\lambda\}andλ∗\\lambda^\{\*\}\. This concludes the proof of the lower bound\.

We now prove the upper bound and assume thatAAis upper\-bounded byamaxa\_\{\\text\{max\}\}\. Note that, by definition ofλ∗∗\\lambda^\{\*\*\},

∀λ<λ∗∗,1n\+1​∑i=1n\+1A\(i\)​\(λ\)<γ\+amaxn\+1\.\\forall\\;\\lambda<\\lambda^\{\*\*\},\\quad\\frac\{1\}\{n\+1\}\\sum\_\{i=1\}^\{n\+1\}A^\{\(i\)\}\(\\lambda\)<\\gamma\+\\frac\{a\_\{\\text\{max\}\}\}\{n\+1\}\.\(26\)Therefore, for everyλ<λ∗∗\\lambda<\\lambda^\{\*\*\},

1n\+1​∑i=1n\+1A\(i\)​\(λ∗∗\)−1n\+1​∑i=1n\+1A\(i\)​\(λ\)\>1n\+1​∑i=1n\+1A\(i\)​\(λ∗∗\)−γ−amaxn\+1,\\displaystyle\\frac\{1\}\{n\+1\}\\sum\_\{i=1\}^\{n\+1\}A^\{\(i\)\}\(\\lambda^\{\*\*\}\)\-\\frac\{1\}\{n\+1\}\\sum\_\{i=1\}^\{n\+1\}A^\{\(i\)\}\(\\lambda\)\>\\frac\{1\}\{n\+1\}\\sum\_\{i=1\}^\{n\+1\}A^\{\(i\)\}\(\\lambda^\{\*\*\}\)\-\\gamma\-\\frac\{a\_\{\\text\{max\}\}\}\{n\+1\},\(27\)which then implies that

H≥1n\+1​∑i=1n\+1A\(i\)​\(λ∗∗\)−γ−amaxn\+1H\\geq\\frac\{1\}\{n\+1\}\\sum\_\{i=1\}^\{n\+1\}A^\{\(i\)\}\(\\lambda^\{\*\*\}\)\-\\gamma\-\\frac\{a\_\{\\text\{max\}\}\}\{n\+1\}\(28\)holds almost surely\. It follows that

𝔼​\[A\(n\+1\)​\(λ∗∗\)\]=𝔼​\[1n\+1​∑i=1n\+1A\(i\)​\(λ∗∗\)\]≤γ\+amaxn\+1\+𝔼​\[H\],\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\lambda^\{\*\*\}\)\\right\]=\\mathbb\{E\}\\left\[\\frac\{1\}\{n\+1\}\\sum\_\{i=1\}^\{n\+1\}A^\{\(i\)\}\(\\lambda^\{\*\*\}\)\\right\]\\leq\\gamma\+\\frac\{a\_\{\\text\{max\}\}\}\{n\+1\}\+\\mathbb\{E\}\[H\],\(29\)where the first equality holds due to exchangeability \(this is analogous to the reasoning for the lower bound\)\. Now, note that the following implications hold:

1n\+1​∑i=1n\+1A\(i\)​\(λ∗∗\)≥γ\+amaxn\+1⟹1n​∑i=1n\+1A\(i\)​\(λ∗∗\)≥\(n\+1\)​γn\+amaxn\\displaystyle\\frac\{1\}\{n\+1\}\\sum\_\{i=1\}^\{n\+1\}A^\{\(i\)\}\(\\lambda^\{\*\*\}\)\\geq\\gamma\+\\frac\{a\_\{\\text\{max\}\}\}\{n\+1\}\\implies\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\+1\}A^\{\(i\)\}\(\\lambda^\{\*\*\}\)\\geq\\frac\{\(n\+1\)\\gamma\}\{n\}\+\\frac\{a\_\{\\text\{max\}\}\}\{n\}\(30\)⟹1n​∑i=1nA\(i\)​\(λ∗∗\)≥\(n\+1\)​γn\+1n​\(amax−A\(n\+1\)​\(λ∗∗\)\)\\displaystyle\\implies\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}A^\{\(i\)\}\(\\lambda^\{\*\*\}\)\\geq\\frac\{\(n\+1\)\\gamma\}\{n\}\+\\frac\{1\}\{n\}\\left\(a\_\{\\text\{max\}\}\-A^\{\(n\+1\)\}\(\\lambda^\{\*\*\}\)\\right\)\(31\)⟹1n​∑i=1nA\(i\)​\(λ∗∗\)≥\(n\+1\)​γn⟹λ∗∗∈𝒱​\(𝐃:n,n\+1n​γ\)\.\\displaystyle\\implies\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}A^\{\(i\)\}\(\\lambda^\{\*\*\}\)\\geq\\frac\{\(n\+1\)\\gamma\}\{n\}\\implies\\lambda^\{\*\*\}\\in\\mathcal\{V\}\\left\(\\mathbf\{D\}\_\{:n\},\\dfrac\{n\+1\}\{n\}\\gamma\\right\)\.\(32\)Following once again an analogous reasoning to the one for the lower bound, we have thatλ^≤λ∗∗\\hat\{\\lambda\}\\leq\\lambda^\{\*\*\}holds almost surely and that

𝔼​\[A\(n\+1\)​\(λ^\)\]\\displaystyle\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\hat\{\\lambda\}\)\\right\]=𝔼\[𝔼\[A\(n\+1\)\(λ^\)\|λ^,λ∗∗\]\]≤𝔼\[𝔼\[A\(n\+1\)\(λ∗∗\)\|λ^,λ∗∗\]\]\\displaystyle=\\mathbb\{E\}\\left\[\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\hat\{\\lambda\}\)\\;\\middle\|\\;\\hat\{\\lambda\},\\lambda^\{\*\*\}\\right\]\\right\]\\leq\\mathbb\{E\}\\left\[\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\lambda^\{\*\*\}\)\\;\\middle\|\\;\\hat\{\\lambda\},\\lambda^\{\*\*\}\\right\]\\right\]\(33\)=𝔼​\[A\(n\+1\)​\(λ∗∗\)\]≤γ\+amaxn\+1\+𝔼​\[H\]\.\\displaystyle=\\mathbb\{E\}\\left\[A^\{\(n\+1\)\}\(\\lambda^\{\*\*\}\)\\right\]\\leq\\gamma\+\\frac\{a\_\{\\text\{max\}\}\}\{n\+1\}\+\\mathbb\{E\}\[H\]\.\(34\)∎

## Appendix CAdditional Discussion of Conf\-Gen: Assumptions and Comparison to CRC

We now provide an extended discussion of the assumptions of Conf\-Gen, as well as its similarities and differences with CRC\.

First, we discuss the monotonicity assumption in[Definition 1](https://arxiv.org/html/2605.28920#Thmdefinition1), which we note we only use withγ′=n\+1n​γ\\gamma^\{\\prime\}=\\tfrac\{n\+1\}\{n\}\\gammaandγ′′=γ\\gamma^\{\\prime\\prime\}=\\gammawhen proving the lower bound, i\.e\., thatλ↦𝔼​\[A\(n\+1\)​\(λ\)∣λ^,λ∗\]\\lambda\\mapsto\\mathbb\{E\}\[A^\{\(n\+1\)\}\(\\lambda\)\\mid\\hat\{\\lambda\},\\lambda^\{\*\}\]is non\-decreasing, almost surely\. This assumption is used in the proof of[Theorem 3](https://arxiv.org/html/2605.28920#Thmtheorem3)in[Equation 24](https://arxiv.org/html/2605.28920#A2.E24)to establish that𝔼​\[A\(n\+1\)​\(λ^\)\]≥𝔼​\[A\(n\+1\)​\(λ∗\)\]\\mathbb\{E\}\[A^\{\(n\+1\)\}\(\\hat\{\\lambda\}\)\]\\geq\\mathbb\{E\}\[A^\{\(n\+1\)\}\(\\lambda^\{\*\}\)\]follows fromλ∗≤λ^\\lambda^\{\*\}\\leq\\hat\{\\lambda\}\. Note that the assumption of monotonicity only in \(total\) expectation—i\.e\., thatλ↦𝔼​\[A\(n\+1\)​\(λ\)\]\\lambda\\mapsto\\mathbb\{E\}\[A^\{\(n\+1\)\}\(\\lambda\)\]is non\-decreasing—is not enough to conclude the desired inequality holds, due toλ^\\hat\{\\lambda\}not being independent fromAAand𝐂λ\\mathbf\{C\}\_\{\\lambda\}, andλ∗\\lambda^\{\*\}also not being independent from\(X\(n\+1\),𝐘\(n\+1\),YGT\(n\+1\)\)\(X^\{\(n\+1\)\},\\mathbf\{Y\}^\{\(n\+1\)\},Y\_\{\\text\{GT\}\}^\{\(n\+1\)\}\)\. Note also that the assumption of monotonicity in total expectation follows from monotonicity in conditional expectation\. Thus, our monotonicity condition is stronger than monotonicty in total expectation, but weaker thanλ↦A\(n\+1\)​\(λ\)\\lambda\\mapsto A^\{\(n\+1\)\}\(\\lambda\)being non\-decreasing almost surely, as required by CRC\. Our condition can be understood as strengthening the assumption of monotonicity in total expectation by requiring that knowledge ofλ^\\hat\{\\lambda\}andλ∗\\lambda^\{\*\}must not alter the distribution ofAA,𝐂λ\\mathbf\{C\}\_\{\\lambda\}, and\(X\(n\+1\),𝐘\(n\+1\),YGT\(n\+1\)\)\(X^\{\(n\+1\)\},\\mathbf\{Y\}^\{\(n\+1\)\},Y\_\{\\text\{GT\}\}^\{\(n\+1\)\}\)so drastically so as to violate monotonicity in expectation\. Intuitively, we can expect monotonicity in conditional expectation to hold whenever monotonicity in total expectation holds and the dependence betweenA\(n\+1\)​\(λ\)A^\{\(n\+1\)\}\(\\lambda\)and\(λ^,λ∗\)\(\\hat\{\\lambda\},\\lambda^\{\*\}\)is sufficiently weak\.

Second, the assumption in Conf\-Gen thatAAbe non\-negative can be understood as simply requiringAAto be lower\-bounded \(and the same is true for CRC\)\. More precisely, assume we have an admissibility functionA~\\tilde\{A\}which is lower\-bounded byamin∈ℝa\_\{\\text\{min\}\}\\in\\mathbb\{R\}rather than0, and say we are given a scalar admissibility targetγ~≥amin\\tilde\{\\gamma\}\\geq a\_\{\\text\{min\}\}\. We can then defineA≔A~−aminA\\coloneqq\\tilde\{A\}\-a\_\{\\text\{min\}\}andγ≔γ~−amin\\gamma\\coloneqq\\tilde\{\\gamma\}\-a\_\{\\text\{min\}\}, and by non\-negativity ofAA,[Theorem 3](https://arxiv.org/html/2605.28920#Thmtheorem3)holds\. Addingamina\_\{\\text\{min\}\}to both sides of[Equation 8](https://arxiv.org/html/2605.28920#S3.E8)then yields the desired conformal guarantee:

𝔼​\[A~​\(X\(n\+1\),𝐂λ^​\(X\(n\+1\),𝐘\(n\+1\)\),YGT\(n\+1\)\)\]≥γ~\.\\mathbb\{E\}\\left\[\\tilde\{A\}\\left\(X^\{\(n\+1\)\},\\mathbf\{C\}\_\{\\hat\{\\lambda\}\}\\left\(X^\{\(n\+1\)\},\\mathbf\{Y\}^\{\(n\+1\)\}\\right\),Y\_\{\\text\{GT\}\}^\{\(n\+1\)\}\\right\)\\right\]\\geq\\tilde\{\\gamma\}\.\(35\)Note that in this case,λ^\\hat\{\\lambda\}as defined in[Equation 6](https://arxiv.org/html/2605.28920#S3.E6)can be re\-written in terms ofA~\\tilde\{A\}andγ~\\tilde\{\\gamma\}as

λ^=inf\{λ∈Λ\|1n​∑i=1nA~​\(X\(i\),𝐂λ​\(X\(i\),𝐘\(i\)\),YGT\(n\+1\)\)≥n\+1n​γ~−aminn\}\.\\hat\{\\lambda\}=\\inf\\left\\\{\\lambda\\in\\Lambda\\;\\middle\|\\;\\dfrac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\tilde\{A\}\\left\(X^\{\(i\)\},\\mathbf\{C\}\_\{\\lambda\}\\left\(X^\{\(i\)\},\\mathbf\{Y\}^\{\(i\)\}\\right\),Y\_\{\\text\{GT\}\}^\{\(n\+1\)\}\\right\)\\geq\\dfrac\{n\+1\}\{n\}\\tilde\{\\gamma\}\-\\dfrac\{a\_\{\\text\{min\}\}\}\{n\}\\right\\\}\.\(36\)
Third, the proof of[Theorem 3](https://arxiv.org/html/2605.28920#Thmtheorem3)requiresλ^∈𝒱​\(𝐃:n,n\+1n​γ\)\\hat\{\\lambda\}\\in\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\},\\tfrac\{n\+1\}\{n\}\\gamma\)andλ∗∈𝒱​\(𝐃:n,γ\)\\lambda^\{\*\}\\in\\mathcal\{V\}\(\\mathbf\{D\}\_\{:n\},\\gamma\)to hold almost surely: this happens wheneverλ↦A​\(X,𝐂λ​\(X,𝐘\),YGT\)\\lambda\\mapsto A\(X,\\mathbf\{C\}\_\{\\lambda\}\(X,\\mathbf\{Y\}\),Y\_\{\\text\{GT\}\}\)is right\-continuous andΛ\\Lambdais a closed interval, as required in CRC—yet it can also hold in other settings, e\.g\., ifΛ\\Lambdais finite\.

Fourth, Conf\-Gen deals withΛ\\Lambdamore cleanly than CRC by making it clear that this set can be unbounded and that it need not be an interval\.

Lastly, the upper bound in[Equation 9](https://arxiv.org/html/2605.28920#S3.E9)improves upon the corresponding bound for CRC from[Equation 16](https://arxiv.org/html/2605.28920#A1.E16)in various ways: it has the same weaker assumptions than CRC for the lower bound, it does not require independence nor the càdlàg property, and its use of𝔼​\[H\]\\mathbb\{E\}\[H\]is somewhat analogous toϵ\\epsilonin the CP upper bound \([Theorem 4](https://arxiv.org/html/2605.28920#Thmtheorem4)\), whereas the CRC bound \([Theorem 5](https://arxiv.org/html/2605.28920#Thmtheorem5)\) has no such analogue\. To see that our bound indeed generalizes the CRC one, note that if the functionsA\(i\)A^\{\(i\)\}are càdlàg, thenH≤h​\(A¯n\+1,λ∗∗\)H\\leq h\(\\bar\{A\}\_\{n\+1\},\\lambda^\{\*\*\}\), and if additionally these functions are independent andℙ​\(h​\(A\(i\),λ\)\>0\)=0\{\\mathbb\{P\}\}\(h\(A^\{\(i\)\},\\lambda\)\>0\)=0for everyλ∈Λ\\lambda\\in\\Lambda\(as required by CRC\), then no more than a single jump can happen at the same time, which yields thath​\(A¯n\+1,λ∗∗\)≤amaxn\+1h\(\\bar\{A\}\_\{n\+1\},\\lambda^\{\*\*\}\)\\leq\\frac\{a\_\{\\text\{max\}\}\}\{n\+1\}almost surely \(seeAngelopoulos et al\. \([2024b](https://arxiv.org/html/2605.28920#bib.bib4)\)for a formal derivation of this fact\)\. In other words, our bound implies the CRC bound when we add the càdlàg and independence assumptions used by CRC, yet it remains valid without them\. For example, if the functionsA\(i\)A^\{\(i\)\}are continuous andℙ​\(λ∗∗=infΛ\)=0\{\\mathbb\{P\}\}\(\\lambda^\{\*\*\}=\\inf\\Lambda\)=0, thenH=0H=0almost surely, in which case our upper bound improves toγ\+amaxn\+1\\gamma\+\\frac\{a\_\{\\text\{max\}\}\}\{n\+1\}\.

## Appendix DAdditional Examples

[Table 2](https://arxiv.org/html/2605.28920#A4.T2)is an extended version of[Table 1](https://arxiv.org/html/2605.28920#S4.T1), including the discussion below and the names of each sequence selector as it appears in our Python package\. We invite the reader to consult the package documentation for a complete list of features and configurations beyond those highlighted in[Table 2](https://arxiv.org/html/2605.28920#A4.T2)\.

Here we provide additional examples of Conf\-Gen configurations beyond those presented in[section4\.3](https://arxiv.org/html/2605.28920#S4.SS3)\. First, note that we can always apply a post\-processing step to the selection function outputs from[Table 1](https://arxiv.org/html/2605.28920#S4.T1); formally this means applying a function to this output\. For example, in the experiment from[section6\.1](https://arxiv.org/html/2605.28920#S6.SS1), the selection function isf​\(𝐲:τ​\(x,𝐲,λ\)\)f\(\\mathbf\{y\}\_\{:\\tau\(x,\\mathbf\{y\},\\lambda\)\}\), wheref:𝒴∗→𝒴∗f:\\mathcal\{Y\}^\{\*\}\\rightarrow\\mathcal\{Y\}^\{\*\}is a deterministic function which removes all exact duplicates from its input sequence\. More complex choices of post\-processing functions are also possible\. For instance, we might want to remove all “semantic duplicates”, which could be achieved by outputtingF​\(𝐲:τ​\(x,𝐲,λ\)\)F\(\\mathbf\{y\}\_\{:\\tau\(x,\\mathbf\{y\},\\lambda\)\}\), whereFFis an LLM call instructing the LLM to remove responses with the same meaning, even if they are not identical\. Note that in general, the output of the selection function𝐂λ​\(x,𝐲\)\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)in this example need not be a subsequence of𝐲\\mathbf\{y\}, although the finite image property remains nonetheless valid\.

We also mentioned in[section5](https://arxiv.org/html/2605.28920#S5)that the conformal summarization method ofKuwahara et al\. \([2025](https://arxiv.org/html/2605.28920#bib.bib25)\)is an instance of Conf\-Gen\. Like conformal factuality, this corresponds to the third row of[Table 2](https://arxiv.org/html/2605.28920#A4.T2), except this time the aggregation function is different\. Here, there is no ground truth variable, and we defineAt′′​\(x,yt\)A\_\{t\}^\{\\prime\\prime\}\(x,y\_\{t\}\)as an indicator of whether sentenceyty\_\{t\}is relevant within documentxx\. We also defineAt′​\(x,𝐲\)A\_\{t\}^\{\\prime\}\(x,\\mathbf\{y\}\)as1N​\(x\)​∑tAt′′​\(x,yt\)\\tfrac\{1\}\{N\(x\)\}\\sum\_\{t\}A\_\{t\}^\{\\prime\\prime\}\(x,y\_\{t\}\), whereN​\(x\)N\(x\)is the number of relevant sentences in documentxx\. The aggregation function is then given by

agg​\(𝐚′\)=𝟙​\(∑a′∈𝐚′a′≥β\)\.\\texttt\{agg\}\(\\mathbf\{a\}^\{\\prime\}\)=\\mathds\{1\}\\left\(\\sum\_\{a^\{\\prime\}\\in\\mathbf\{a\}^\{\\prime\}\}a^\{\\prime\}\\geq\\beta\\right\)\.\(37\)Note that Conf\-Gen also allows us to easily modify this setup\. For example, if we wanted the output to be a proper summary rather than a subset of important sentences, we could change the output from\{yt\|St↑\>λ\}\\\{y\_\{t\}\|S\_\{t\}^\{\\uparrow\}\>\\lambda\\\}toF​\(\{yt\|St↑\>λ\}\)F\(\\\{y\_\{t\}\|S\_\{t\}^\{\\uparrow\}\>\\lambda\\\}\), whereFFis an LLM call where the LLM is instructed to summarize all the sentences given to it as input\.

Lastly, in[section6\.5](https://arxiv.org/html/2605.28920#S6.SS5)we used a selection function which is not displayed in[Table 1](https://arxiv.org/html/2605.28920#S4.T1): the smallest subset such that the accumulated score exceedsλ\\lambda\. Formally, for a sequence𝐲\\mathbf\{y\}with corresponding scores𝐒↑\\mathbf\{S\}^\{\\uparrow\}, this selection function is given by

𝐂λ​\(x,𝐲\)∈arg​min𝒰⊆𝐲​\|∑S↑∈𝐒𝒰↑S↑\>​λ⁡\|𝒰\|\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)\\in\\operatorname\*\{arg\\,min\}\_\{\\mathcal\{U\}\\subseteq\\mathbf\{y\}\|\\sum\_\{S^\{\\uparrow\}\\in\\mathbf\{S\}\_\{\\mathcal\{U\}\}^\{\\uparrow\}\}S^\{\\uparrow\}\>\\lambda\}\|\\mathcal\{U\}\|\(38\)whenever thearg​min\\operatorname\*\{arg\\,min\}is not empty, and𝐂λ​\(x,𝐲\)=𝐲\\mathbf\{C\}\_\{\\lambda\}\(x,\\mathbf\{y\}\)=\\mathbf\{y\}otherwise, where𝐒𝒰↑\\mathbf\{S\}\_\{\\mathcal\{U\}\}^\{\\uparrow\}contains the entries in𝐒↑\\mathbf\{S\}^\{\\uparrow\}corresponding to𝒰\\mathcal\{U\}\. Operationally, this selection function is equivalent to the first row of[Table 2](https://arxiv.org/html/2605.28920#A4.T2), except the sequence𝐲\\mathbf\{y\}is sorted beforehand on decreasing order of its corresponding scores \(and the output is a set rather than a sequence\)\. Note that the sorting operation requires access to the entire sequence𝐲\\mathbf\{y\}, and thus this selection function is no longer compatible with partial generation\. Note also that this selection function behaves similarly to the second row from[Table 2](https://arxiv.org/html/2605.28920#A4.T2), except it does not include all entries with matching scores at once\. As an example, consider a sequence𝐲=\(y1,y2,y3\)\\mathbf\{y\}=\(y\_\{1\},y\_\{2\},y\_\{3\}\)with corresponding scores𝐬↑=\(1,1,2\)\\mathbf\{s\}^\{\\uparrow\}=\(1,1,2\), and assume that𝐬↓=−𝐬↑\\mathbf\{s\}^\{\\downarrow\}=\-\\mathbf\{s\}^\{\\uparrow\}\. This selection function would produce\{y3\}\\\{y\_\{3\}\\\},\{y3,y1\}\\\{y\_\{3\},y\_\{1\}\\\}, and\{y3,y2,y1\}\\\{y\_\{3\},y\_\{2\},y\_\{1\}\\\}\(assuming the sorting algorithm produces\(y3,y1,y2\)\(y\_\{3\},y\_\{1\},y\_\{2\}\)and not\(y3,y2,y1\)\(y\_\{3\},y\_\{2\},y\_\{1\}\)\), with the jumps happening atλ=2,3\\lambda=2,3\. The selection function from the second row of[Table 2](https://arxiv.org/html/2605.28920#A4.T2)would produce∅\\emptyset,\{y3\}\\\{y\_\{3\}\\\}, and\{y3,y2,y1\}\\\{y\_\{3\},y\_\{2\},y\_\{1\}\\\}, with the jumps happening atλ=−2,−1\\lambda=\-2,\-1: this latter selection function always outputsy1y\_\{1\}andy2y\_\{2\}together as they have the same score\.

Table 2:Representative configurations of Conf\-Gen\.
## Appendix EExperiments

### Section E\.1Conformal Open\-Domain Question Answering

Dataset Splits\.The TriviaQA dataset\(Joshi et al\.,[2017](https://arxiv.org/html/2605.28920#bib.bib20)\)contains trivia questions such as*“What is Harrison Ford’s real name?”*or*“What is Mel Gibson’s middle name?”*for which there exist a small number of correct answers \(often only one\)\. We use the same TriviaQA dataset used byQuach et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib40)\)composed of17,94417,944trivia questions and their answers\.Quach et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib40)\)decompose their dataset as follows:2,0002,000questions are used for training,2,0002,000questions are used for validation, and the rest is used for test\. We use their training set as our calibration set, and our test set is the same as theirs\.

Protocol\.We follow the experimental protocol fromQuach et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib40)\)by using LLaMA\-13B embeddings\(Touvron et al\.,[2023](https://arxiv.org/html/2605.28920#bib.bib51)\)\. For each questionxx, we sampleT=20T=20answers\. We defineS↑​\(x,y\)S^\{\\uparrow\}\(x,y\)as the likelihood function of the LLaMA model with length\-normalization\(Wu et al\.,[2016](https://arxiv.org/html/2605.28920#bib.bib60)\)for the questionxxand the answeryy\. The admissibility functionA′​\(x,y,yGT\)A^\{\\prime\}\(x,y,y\_\{\\text\{GT\}\}\)is11ifyymatchesyGTy\_\{\\text\{GT\}\}, and0otherwise\.

Sampling Strategies\.In the main paper, we consider that our sampling strategy is*i\.i\.d\.*, which means that the LLM can generate the same answer multiple times\. As an ablation study, we propose here an alternative sampling strategy \(that we call*diverse*\) that forces the LLM to generate an answer that is different from all the answers that were previously generated\. To this end, and due to the nature of the dataset and of the ground truth answers that are often short, we found that an easily reproducible way of generating different answers was to add previously generated answers to the list of*bad words*\.555See[https://huggingface\.co/docs/transformers/v4\.46\.0/en/internal/generation\_utils](https://huggingface.co/docs/transformers/v4.46.0/en/internal/generation_utils)for details\. In practice, this corresponds to a list of token ids that are not allowed to be generated\.When explicitly asked in the prompt to generate different answers, LLaMa tends to generate the same answer multiple times because of the high certainty of the model for its first generated answer; this is why we used the list of bad words instead\.

Results\.[Figure 2](https://arxiv.org/html/2605.28920#S6.F2)illustrates the average test admissibility score and the average number of answers generated for each question byQuach et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib40)\)with the*max scoring functions*that they propose, and by our method with the selection function from the first row of[Table 1](https://arxiv.org/html/2605.28920#S4.T1)withaccum=max\\texttt\{accum\}=\\maxbut with duplicates removed, as a function ofγ∈\[0\.62,0\.86\]\\gamma\\in\[0\.62,0\.86\]\(as done byQuach et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib40)\)\)\. As discussed in the main paper, our approach is cheaper at inference time since it requires fewer LLM calls for a givenγ≤0\.84\\gamma\\leq 0\.84\. Whenγ≥0\.85\\gamma\\geq 0\.85, the value of ourλ^\\hat\{\\lambda\}is\+∞\+\\infty, and the size of our sequence becomes as large as possible\.Quach et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib40)\)avoid this degenerate case by using different filters with the Pareto testing method\(Laufer\-Goldshtein et al\.,[2023](https://arxiv.org/html/2605.28920#bib.bib26)\)to cross\-validate hyperparameters that our approach does not use, but this cross\-validation process also incurs additional cost during calibration, and it results in larger sets whenγ≤0\.84\\gamma\\leq 0\.84\.

[Figure 7](https://arxiv.org/html/2605.28920#A5.F7)illustrates the average sequence length of the baseline and of our method with*i\.i\.d\.*sampling and with the*diverse*strategy described above with different sequence selectors described in[Table 2](https://arxiv.org/html/2605.28920#A4.T2)\. Since the first answer generated by the model is often correct and there are few possible correct answers in this task, the*diverse*sampling strategy underperforms both the strategy with*i\.i\.d\.*sampling and the baseline\. This demonstrates that the choice of the*sequence generation*strategy is a crucial step of our approach and may be impacted by the nature of the task and of the dataset\. On the other hand, the choice of the sequence selector function has less impact on the performance in this task\. We also report the results obtained by the baseline with different scoring functions called*First\-K*,*First\-K\+reject*,*Max*, and*Sum*\(see Quach et al\.,[2024](https://arxiv.org/html/2605.28920#bib.bib40), Section 5\.2\)\.

![Refer to caption](https://arxiv.org/html/2605.28920v1/x6.png)Figure 7:Average sequence length as a function ofγ\\gammawith different sampling strategies\. Our proposed strategy results in shorter average sequence lengths than the baseline, resulting in a smaller number of LLM calls\.
### Section E\.2Conformal Generation of Non\-Memorized Images

Score Function\.We now specify the score function we used for the experiment in[section6\.2](https://arxiv.org/html/2605.28920#S6.SS2)in more detail\. Recall that diffusion models\(Sohl\-Dickstein et al\.,[2015](https://arxiv.org/html/2605.28920#bib.bib48); Ho et al\.,[2020](https://arxiv.org/html/2605.28920#bib.bib17); Song et al\.,[2021](https://arxiv.org/html/2605.28920#bib.bib49)\)such as Stable Diffusion generate an image by first sampling Gaussian noise and iteratively refining the noise into an image\. Since Stable Diffusion is a latent diffusion model, this refinement happens in latent space, and a decoderggmust be applied to obtain an image from a latent\. Formally, thett\-th generation does not only contain the final generated latent, but the entire trajectory of noisy latents used by the diffusion model during generation:𝐘t=\(Yt​\(1\),…,Yt​\(m\)\)\\mathbf\{Y\}\_\{t\}=\(Y\_\{t\}\(1\),\\dots,Y\_\{t\}\(m\)\), whereYt​\(1\)Y\_\{t\}\(1\)corresponds to the final generated latent \(i\.e\.,g​\(Yt​\(1\)\)g\(Y\_\{t\}\(1\)\)is the final generated image\), with eachYt​\(j\)Y\_\{t\}\(j\)becoming progressively nosier asjjincreases, and withYt​\(m\)Y\_\{t\}\(m\)corresponding to pure Gaussian noise\. Recall that diffusion models provide a score functionss,666This function corresponds to the Stein score, and has nothing to do with our use of the term “score function”\.and the classifier guidance term corresponding toYt​\(j\)Y\_\{t\}\(j\)is

s​\(Yt​\(j\),j,Xt\)−s​\(Yt​\(j\),j,∅\),s\\left\(Y\_\{t\}\(j\),j,X\_\{t\}\\right\)\-s\\left\(Y\_\{t\}\(j\),j,\\emptyset\\right\),\(39\)i\.e\., it is the difference between the score function conditioned on the input promptXtX\_\{t\}, and the score function without conditioning on any prompt\. Our score function is given bySt↑=−St↓S\_\{t\}^\{\\uparrow\}=\-S^\{\\downarrow\}\_\{t\}, where

St↓=S↓​\(Xt,𝐘t\)=1m​∑j=1mσ2​\(j\)​‖s​\(Yt​\(j\),j,Xt\)−s​\(Yt​\(j\),j,∅\)‖22,S^\{\\downarrow\}\_\{t\}=S^\{\\downarrow\}\(X\_\{t\},\\mathbf\{Y\}\_\{t\}\)=\\dfrac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\sigma^\{2\}\(j\)\\\|s\\left\(Y\_\{t\}\(j\),j,X\_\{t\}\\right\)\-s\\left\(Y\_\{t\}\(j\),j,\\emptyset\\right\)\\\|\_\{2\}^\{2\},\(40\)whereσ​\(j\)\\sigma\(j\)is a known coefficient from the diffusion process\. As mentioned in the main text,Wen et al\. \([2023](https://arxiv.org/html/2605.28920#bib.bib59)\)found this score indicative of memorization, and we used the procedure proposed byRoss et al\. \([2025](https://arxiv.org/html/2605.28920#bib.bib44)\), which we now describe\.𝐘1\\mathbf\{Y\}\_\{1\}is produced directly fromX1X\_\{1\}, and then the gradient ofS↓​\(X1,𝐘1\)S^\{\\downarrow\}\(X\_\{1\},\\mathbf\{Y\}\_\{1\}\)is computed for every token inX1X\_\{1\}\. The token achieving the largest gradient norm is selected, and an LLM is instructed to produceX2X\_\{2\}by changing the selected token fromX1X\_\{1\}, while keeping the entire prompt semantically meaningful\. Then,𝐘2\\mathbf\{Y\}\_\{2\}is generated usingX2X\_\{2\}as the prompt, and this process is repeated sequentially to produceX3X\_\{3\}based onS↓​\(X2,𝐘2\)S^\{\\downarrow\}\(X\_\{2\},\\mathbf\{Y\}\_\{2\}\), and so on\.Ross et al\. \([2025](https://arxiv.org/html/2605.28920#bib.bib44)\)proposed this procedure as a way to generate a sequence of progressively less memorized images; our work can be understood as conformalizing the choice of which of these images to select\.Ross et al\. \([2025](https://arxiv.org/html/2605.28920#bib.bib44)\)found that a different score based on local intrinsic dimension estimates\(Kamkari et al\.,[2024](https://arxiv.org/html/2605.28920#bib.bib22); Leung et al\.,[2025](https://arxiv.org/html/2605.28920#bib.bib28)\)performed slightly better than the one in[Equation 40](https://arxiv.org/html/2605.28920#A5.E40), but this score is more computationally taxing to compute and resulted in out\-of\-memory errors on our hardware\.

Dataset\.We used the code ofRoss et al\. \([2025](https://arxiv.org/html/2605.28920#bib.bib44)\), which uses the samples thatWebster \([2023](https://arxiv.org/html/2605.28920#bib.bib58)\)identified as memorized by Stable Diffusion v1\.5\. The codebase does not directly provide images, but rather URLs containing the images\. Some of the URLs were no longer valid, and some of the valid URLs contained duplicate images\. After de\-duplication, we were left with5252images, which we split into a calibration and a test set each containing2626images\. For each one of these ground truth memorized imagesYGT\(i\)Y\_\{\\text\{GT\}\}^\{\(i\)\}and each corresponding generated imageg​\(Yt\(i\)​\(1\)\)g\(Y\_\{t\}^\{\(i\)\}\(1\)\), we asked1010human evaluators to choose one of three options:777Note that this is an example where the elements of𝐘\\mathbf\{Y\}do not live in the same space asYGTY\_\{\\text\{GT\}\}: the former are latents, and the later is an image\. In other words, this example highlights that Conf\-Gen remains valid when𝒴≠𝒴GT\\mathcal\{Y\}\\neq\\mathcal\{Y\}\_\{\\text\{GT\}\}\.\(a\)\(a\)“bad”, meaning that the generated image is a copy or near\-copy of the real image;\(b\)\(b\)“medium”, meaning that the generated image is “too different” or completely unrelated to the real one; and\(c\)\(c\)“good”, meaning that the generated image is similar in content and style to the real one, but not close enough to be a copy\. The admissibilityAt′A\_\{t\}^\{\\prime\}is then given by the percentage of evaluators who assessed the corresponding generated image as “good” or “medium”\. In[Figure 8](https://arxiv.org/html/2605.28920#A5.F8),[Figure 9](https://arxiv.org/html/2605.28920#A5.F9), and[Figure 10](https://arxiv.org/html/2605.28920#A5.F10), we display examples shown to the human evaluators, corresponding to a “bad”, a “medium”, and a “good” generation, respectively\.

Discussion\.Note that this experiment is an example where the functionλ↦A​\(λ\)\\lambda\\mapsto A\(\\lambda\)is not non\-decreasing, almost surely: it is possible , however unlikely, that by changing more tokens in the prompt, the corresponding generated image becomes more memorized\. Nonetheless, since this event should be expected to have very low probability, it is reasonable to believe that monotonicity still holds in conditional expectation, as required by[Definition 1](https://arxiv.org/html/2605.28920#Thmdefinition1)\. This example underscores the practical relevance of our relaxed theoretical framework over the standard requirements of CRC\.

Human Evaluators\.We used the Prolific platform to recruit our anonymous human evaluators\. We recruited a total of4040participants; the only restrictions we enforced were self\-reported fluency in English \(in order to understand instructions\), and a minimum of a technical college degree, so as to maximize the probability of true English fluency\. We manually reviewed every image before running the experiment to ensure no disturbing content would be shown to the participants\. We paid an average hourly rate above £1212, which is significantly more than the Prolific\-recommended amount, £99\. Some institutions have an institutional review board \(IRB\) which sets the ethical standards that its members must follow when conducting experiments with humans, and whose approval is required prior to running such experiments\. Our institution has no IRB, and we thus followed the aforementioned steps to ensure our experiment was run ethically\.

![Refer to caption](https://arxiv.org/html/2605.28920v1/results/memorization/bad_example.png)Figure 8:Example of a screen displayed to human evaluators, corresponding to a “bad” example\.![Refer to caption](https://arxiv.org/html/2605.28920v1/results/memorization/medium_example.png)Figure 9:Example of a screen displayed to human evaluators, corresponding to a “medium” example\.![Refer to caption](https://arxiv.org/html/2605.28920v1/results/memorization/good_example.png)Figure 10:Example of a screen displayed to human evaluators, corresponding to a “good” example\.
### Section E\.3Conversational Conversational AI Chatbot

Dataset Specification\.We utilize the multi\-turn human\-generated conversations from the ClariQ dataset for our experiments\. It contains498498human\-labeled conversations containing initial questions and further clarifications\. Certain conversations include responses which provide little information \(e\.g\., simple responses such as*“No”*,*“I don’t know”*,*“I am not sure”*, etc\.\)\. To ensure the quality of the data, we filtered out conversations including such responses, resulting in a high\-quality dataset of316316conversations\. We use a50−5050\{\-\}50split for the calibration and test sets\.

Dataset Preprocessing: Question Consolidation\.Each conversation in the dataset is obtained as follows\. First, a user asks an ambiguous question,X1X\_\{1\}\. An LLM then generates a follow\-up questionY1′Y\_\{1\}^\{\\prime\}asking for clarification at which point the user provides an answerX2′X\_\{2\}^\{\\prime\}toY1′Y\_\{1\}^\{\\prime\}\. The LLM then asks another follow\-up question,Y2′Y\_\{2\}^\{\\prime\}, and the user provides an answerX3′X\_\{3\}^\{\\prime\}\. This process happens once more to produceY3′Y\_\{3\}^\{\\prime\}andX4′X\_\{4\}^\{\\prime\}, and results in the sequences𝐘′=\(Y1′,Y2′,Y3′\)\\mathbf\{Y\}^\{\\prime\}=\(Y\_\{1\}^\{\\prime\},Y\_\{2\}^\{\\prime\},Y\_\{3\}^\{\\prime\}\)and𝐗′=\(X2′,X3′,X4′\)\\mathbf\{X\}^\{\\prime\}=\(X\_\{2\}^\{\\prime\},X\_\{3\}^\{\\prime\},X\_\{4\}^\{\\prime\}\)\. The ClariQ dataset containsX1X\_\{1\},𝐘′\\mathbf\{Y\}^\{\\prime\}, and𝐗′\\mathbf\{X\}^\{\\prime\}; and all the conversations in the dataset are such that at the end of the conversation, the original question has been disambiguated\. We then consolidate the conversation into a series of progressively disambiguated questions:

Xt=CONSOLIDATE​\(X1,𝐗:t′,𝐘:t−1′\)X\_\{t\}=\\text\{CONSOLIDATE\}\(X\_\{1\},\\mathbf\{X\}\_\{:t\}^\{\\prime\},\\mathbf\{Y\}\_\{:t\-1\}^\{\\prime\}\)\(41\)fort=2,3,4t=2,3,4, whereCONSOLIDATEis an LLM call where the LLM is instructed to distill the original question and all provided clarifications into a single question\. As a result, the consolidated questionXtX\_\{t\}represents the entire context of the conversation up to steptt\. We also make another LLM call to obtain an answerYtY\_\{t\}to the consolidated questionXtX\_\{t\}, and the scoreSt↑=S↑​\(Xt,Yt\)S\_\{t\}^\{\\uparrow\}=S^\{\\uparrow\}\(X\_\{t\},Y\_\{t\}\)is obtained through another LLM call where the LLM quantifies how clearXtX\_\{t\}is as a question \(although the score function could only takeXtX\_\{t\}as input, we also passYtY\_\{t\}\)\.

Furthermore, to diversify the question sequence lengths over the conversation dataset, we evenly divide the ClariQ dataset \(the multi\-turn human\-generated portion\) into44subsets, such that the length of the question sequences \(i\.e\.𝐗\\mathbf\{X\}\) ranges from11to44for each of the subsets\. We refer to these subsets as*0\-turn questions*,*1\-turn questions*,*2\-turn questions*, and*3\-turn questions*, respectively\. To obtain these subsets, for each44\-question sequence we obtained from the consolidation stage, we drop the questions in the middle, such that the initial ambiguous questionX1X\_\{1\}and the final fully\-clarified questionX4X\_\{4\}are always maintained \(with the exception for the*0\-turn questions*, for which we only keepX4X\_\{4\}\)\. As a result, the number of follow\-up questions required to obtain an unambiguous question across the test dataset is, on average,*2\.5*per conversation\. For these conversations, the admissibilityAt′A\_\{t\}^\{\\prime\}at thett\-th turn of the conversation is an indicator of whether the original question has been fully disambiguated, i\.e\., it is0unlessttis the final turn of the conversation\. Thus, the conformal guarantee ensures that the conversational chatbot stops asking clarifying questions past the turn where the question has been fully disambiguated with probability at leastγ\\gamma\.

Discussion\.Recall that the selection function is given by the fourth row of[Table 1](https://arxiv.org/html/2605.28920#S4.T1), which does not always automatically satisfy the monotonicity requirement in[Definition 1](https://arxiv.org/html/2605.28920#Thmdefinition1)\. Note that for this task,At′A\_\{t\}^\{\\prime\}is always non\-decreasing inttby construction\. As a consequence,λ↦A​\(λ\)\\lambda\\mapsto A\(\\lambda\)is non\-decreasing, and Conf\-Gen can thus be applied\.

LLM Prompts for Consolidation, Generation, and Scoring\.We provide in[Figure 11](https://arxiv.org/html/2605.28920#A5.F11)the system prompt we used for consolidation step\. The LLM prompt for the response generation given a user question is provided in[Figure 12](https://arxiv.org/html/2605.28920#A5.F12)\. Lastly, the LLM prompt for the scoring function is provided in[Figure 13](https://arxiv.org/html/2605.28920#A5.F13)and[Figure 14](https://arxiv.org/html/2605.28920#A5.F14)\. We use*GPT\-4o*for the LLM models for question consolidation, response generation, and scoring\.

consolidation\_prompt: \|

You condense a multi\-turn passage \(alternating lines starting with "User:" and "Assistant:"\) into a single,

fully specified question that reflects the user’s final clarified intent\.

Rules:

\- Integrate all details; prioritize the latest user clarifications\.

\- Resolve pronouns from earlier context when obvious \(e\.g\., “his” → “Michael Jordan”\)\.

\- Output ONLY the consolidated question as one sentence, no preamble or extra text\.

\- Be concise \(<= 25 words\) and specific\.

consolidation\_examples: \|

Example Passages:

Passage:

User: How do I bake a cake?

Assistant: What type of cake are you interested in?

User: A chocolate cake with minimal sugar\.

Consolidated Question:

How do I bake a low\-sugar chocolate cake?

\-\-\-

Passage:

User: Can you tell me about the latest iPhone?

Assistant: Do you want specifications, price, or features?

User: Just the new features and improvements\.

Consolidated Question:

What are the new features and improvements in the latest iPhone?

\-\-\-

Passage:

User: I want to plan a vacation\.

Assistant: Where are you thinking of going?

User: Somewhere in Europe, ideally with historical landmarks\.

Consolidated Question:

How can I plan a European vacation focused on historical landmarks?

\-\-\-

Passage:

User: Explain machine learning\.

Assistant: Do you want a general overview or examples of applications?

User: Focus on real\-world applications in healthcare\.

Consolidated Question:

What are real\-world applications of machine learning in healthcare?

\-\-\-

Passage:

User: I need help with my laptop\.

Assistant: What problem are you experiencing?

User: It keeps overheating when I run multiple programs\.

Consolidated Question:

How can I prevent my laptop from overheating when running multiple programs?

\-\-\-

Passage:

User: I like Italian food\.

Assistant: Are you looking for recipes or restaurants?

User: Recipes, preferably easy ones I can make at home\.

Consolidated Question:

What are easy Italian recipes I can prepare at home?

\-\-\-

Passage:

User: I want to learn photography\.

Assistant: Do you mean digital or film photography?

User: Digital, with tips for beginners on portraits\.

Consolidated Question:

How can a beginner learn digital portrait photography effectively?

\-\-\-

Passage:

User: I’m interested in running a marathon\.

Assistant: Do you need training plans or diet advice?

User: Both, especially training for a first marathon\.

Consolidated Question:

What training plan and diet advice are recommended for someone running their first marathon?

Figure 11:Prompt used for the LLM to consolidate questions in ClariQ\.generation\_prompt: \|

You are a concise evaluator and answer generator\.

Your task is to decide whether the user’s question is clear enough to answer directly, and then respond appropriately\.

\-\-\-

Decision rules \(in order\):

1\. If the question is truly vague, open\-ended, or underspecified, respond exactly:

"I cannot answer because the question is not clear enough\."

\{internallinenumbers\*\}2\. If the question is reasonably specific and factual — even if it asks for multiple related details \(e\.g\., biography \+ contributions, or recipes for a specific dish\) — treat it as clear and answer concisely \(1\-2 sentences\)\.

Only refuse if the intent or essential details are genuinely ambiguous\.

3\. If the question is clear but requires real\-time data, private information, or external access, respond exactly:

"The question is specific and clear, but I cannot answer due to information or access limitations\."

\-\-\-

Output rules:

\- Respond with only the answer text — no commentary, explanations, or code formatting\.

\- Keep answers short \(1\-2 sentences\), factual, and strictly on\-topic\.

\- Consider multi\-part factual questions as clear if the meaning is specific and understandable\.

generation\_examples: \|

\*\*Examples:\*\*

Q: "Tell me about animals\."

A: "I cannot answer because the question is not clear enough\."

\-\-\-

Q: "What are some good ways to train a dog?"

A: "Positive reinforcement, consistency, and short daily sessions are effective ways to train a dog\."

\-\-\-

Q: "What are some effective exercises to strengthen core muscles and improve posture for office workers?"

A: "Planks, bridges, and seated stretches help strengthen core muscles and improve posture for office workers\."

\-\-\-

Q: "Who won the FIFA World Cup in 2023?"

A: "The question is specific and clear, but I cannot answer due to information or access limitations\."

\-\-\-

Q: "Can you tell me about Leonardo da Vinci?"

A: "Leonardo da Vinci was a Renaissance artist, inventor, and scientist, known for works like the Mona Lisa and The Last Supper\."

\-\-\-

Q: "What are some quick, healthy recipes for dinner under 30 minutes?"

A: "Grilled salmon with vegetables or stir\-fried chicken with broccoli are quick, healthy dinner options under 30 minutes\."

\-\-\-

Q: "Explain AI\."

A: "I cannot answer because the question is not clear enough\."

\-\-\-

Q: "What is the current price of Bitcoin?"

A: "The question is specific and clear, but I cannot answer due to information or access limitations\."

\-\-\-

Q: "What did Marie Curie contribute to science, and what is her biography?"

\{internallinenumbers\*\}A: "Marie Curie discovered radium and polonium, conducted pioneering research on radioactivity, and was the first woman to win a Nobel Prize; she was born in 1867 and died in 1934\."

\-\-\-

Q: "Give me travel tips\."

A: "I cannot answer because the question is not clear enough\."

\-\-\-

Q: "What are some recommended beginner\-friendly books to learn Python programming?"

A: "’Automate the Boring Stuff with Python’ and ’Python Crash Course’ are beginner\-friendly books to learn Python programming\."

\-\-\-

Q: "Who is the current president of France?"

A: "The question is specific and clear, but I cannot answer due to information or access limitations\."

Figure 12:Prompt used for the LLM to generate responses to consolidated questions from ClariQ\.scoring\_prompt: \|

You are evaluating a question\-answer pair produced by a separate generation step\.

The question has already been consolidated into a single, clear form\.

Your primary goal is to assign a \*\*continuous clarity and specificity score between 0\.0 and 1\.0\*\* that reflects

\*\*how specific, detailed, and well\-defined\*\* the question is\.

\#\#\# Scoring Philosophy:

\- The score must vary smoothly and \*\*not be limited to round multiples of 0\.1\.\*\*

\- Use the full numerical continuum — fine\-grained differences \(e\.g\., 0\.33 vs\. 0\.37\) are encouraged\.

\- Imagine the score as a "clarity percentage": 0\.00 = completely unclear, 1\.00 = perfectly precise\.

\#\#\# Main Criteria:

1\. \*\*Clarity & Specificity:\*\*

\- Questions with more explicit context, constraints, or detail score higher\.

\- Ambiguous or open\-ended questions score lower\.

2\. \*\*Answer as Evidence:\*\*

\- The answer serves only as indirect evidence of how clear the question is\.

\- A clear question tends to produce a short, precise answer\.

A vague question tends to produce refusal or generalities\.

\- However, never inflate the score if the question itself is vague\.

3\. \*\*Refusals & Access Limits:\*\*

\- Refusal because of \*vagueness\* → low score\.

\- Refusal because of \*access limitation\* → high score if question itself is specific\.

\#\#\# Interpretation Range:

\| Range \| Meaning \|

\|\-\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\|

\| 0\.00\-0\.20 \| Extremely vague or incoherent \|

\| 0\.21\-0\.40 \| Somewhat vague, lacks focus \|

\| 0\.41\-0\.60 \| Moderately clear but missing key details \|

\| 0\.61\-0\.80 \| Mostly clear and specific, minor ambiguity \|

\| 0\.81\-1\.00 \| Very clear, precise, and narrowly scoped \|

\#\#\# Output format \(strict\):

Respond with a single JSON object containing exactly two keys:

\- "score": a continuous float between 0\.0 and 1\.0 \(number, not string\)\.

\- "rationale": one concise sentence explaining why the score was chosen\.

\*\*Rules:\*\*

\- Use precise decimal values \(e\.g\., 0\.37, 0\.64, 0\.82\) — avoid rounding to tenths\.

\- Output only the JSON object \(no code fences, labels, or commentary\)\.

\- Strings inside the JSON must escape all internal quotes \(\\\\"\)\.

\- The score must reflect the question’s clarity, not how "good" the answer is\.

Figure 13:Prompt used for the LLM to generate responses to consolidated questions from ClariQ\.scoring\_examples: \|

\*\*Examples:\*\*

\# ====== Very Low Clarity \(0\.0\-0\.2\) ======

Q: "Tell me something interesting\."

A: "I cannot answer because the question is not clear enough\."

Score: 0\.05

Rationale: No topic or constraint; completely open\-ended and vague\.

Q: "Explain\."

A: "I cannot answer because the question is not clear enough\."

Score: 0\.02

Rationale: Incoherent and contextless; provides no subject to address\.

Q: "Talk about AI\."

A: "I cannot answer because the question is not clear enough\."

Score: 0\.12

Rationale: Mentions a topic but lacks any direction, focus, or goal\.

Q: "Tell me about history\."

A: "I cannot answer because the question is not clear enough\."

Score: 0\.15

Rationale: Very broad; no timeframe, region, or event specified\.

Q: "How are things?"

A: "I cannot answer because the question is not clear enough\."

Score: 0\.08

Rationale: Conversational but meaningless as a factual query\.

\# ====== Low\-Moderate Clarity \(0\.21\-0\.4\) ======

Q: "What are some technologies used today?"

A: "Computers, smartphones, and the internet are common technologies today\."

Score: 0\.27

Rationale: Some topical relevance but extremely broad; lacks domain or purpose\.

Q: "Tell me about AI in healthcare\."

A: "AI in healthcare includes diagnosis support and predictive analytics\."

Score: 0\.34

Rationale: Slightly more focused but still too general and undefined in scope\.

Q: "What are ways to improve health?"

A: "Exercise, balanced diet, and adequate sleep improve health\."

Score: 0\.31

Rationale: Basic but non\-specific; does not define population or goal\.

Q: "Why is the sky blue?"

A: "Because of Rayleigh scattering of sunlight in the atmosphere\."

Score: 0\.39

Rationale: Simple but specific physical question; focused yet shallow in scope\.

\# ====== Moderate Clarity \(0\.41\-0\.6\) ======

Q: "What are some ways to reduce carbon emissions?"

\{internallinenumbers\*\}A: "Renewable energy, energy efficiency, and carbon taxes can reduce emissions\."

Score: 0\.57

\{internallinenumbers\*\}Rationale: Moderately specific; identifies a theme but lacks timeframe or region\.

Q: "How does machine learning work?"

A: "It learns patterns from data to make predictions or decisions\."

Score: 0\.45

Rationale: Clear topic but broad and missing key context or examples\.

Q: "What are the causes of climate change?"

A: "Main causes include burning fossil fuels, deforestation, and agriculture\."

Score: 0\.53

Rationale: Understandable but still general; no particular framing or focus\.

Q: "What are benefits of meditation?"

A: "It reduces stress and improves focus and emotional well\-being\."

Score: 0\.49

\{internallinenumbers\*\}Rationale: Clear but not specific; lacks context such as duration or demographic\.

Q: "Explain the difference between a virus and bacteria\."

A: "Viruses need a host to reproduce, while bacteria can live independently\."

Score: 0\.58

Rationale: Narrow topic but not framed in detail \(e\.g\., structure, examples\)\.

\# ====== Fair\-High Clarity \(0\.61\-0\.8\) ======

Q: "What are common side effects of taking antibiotics?"

A: "Nausea, diarrhea, and allergic reactions are common side effects\."

Score: 0\.69

Rationale: Fairly clear and focused but lacks mention of drug type or duration\.

Q: "How can students improve concentration while studying?"

\{internallinenumbers\*\}A: "Taking breaks, avoiding distractions, and maintaining good sleep can help\."

Score: 0\.64

\{internallinenumbers\*\}Rationale: Clear practical goal but lacks specificity \(e\.g\., age group or context\)\.

Q: "What are the main causes of inflation in developed economies?"

A: "Supply chain issues, monetary policy, and demand surges are main causes\."

Score: 0\.77

Rationale: Clear economic scope and focus; moderately well\-framed\.

\{internallinenumbers\*\}Q: "What are key differences between renewable and non\-renewable energy sources?"

\{internallinenumbers\*\}A: "Renewables are replenishable \(solar, wind\), while non\-renewables are finite \(coal, oil\)\."

Score: 0\.73

Rationale: Well\-scoped and comparative but lacks application context\.

Q: "What are early symptoms of diabetes in adults?"

A: "Frequent urination, increased thirst, and fatigue are early symptoms\."

Score: 0\.81

Rationale: Highly focused medical question with clear population definition\.

Figure 14:Prompt used for the LLM to score questions from ClariQ\.\# ====== High Clarity \(0\.81\-0\.95\) ======

Q: "What are the top three renewable energy strategies for reducing carbon emissions in urban areas by 2030?"

A: "Solar rooftops, electric public transport, and building efficiency improvements\."

Score: 0\.92

Rationale: Very specific — defined goal, context, timeframe, and scope\.

Q: "How can a beginner learn Python for data analysis within three months?"

A: "Take structured courses, practice datasets, and complete small projects\."

Score: 0\.87

Rationale: Clear, focused, and practical; specifies skill level, purpose, and timeframe\.

Q: "What exercises are recommended for improving cardiovascular health in adults over 50?"

A: "Brisk walking, cycling, and swimming are recommended\."

Score: 0\.93

Rationale: Clear, medically targeted, and demographically scoped\.

Q: "What are the differences between supervised and unsupervised learning, with examples for each?"

A: "Supervised learning uses labeled data \(classification\); unsupervised finds clusters\."

Score: 0\.96

Rationale: Exceptionally precise; question explicitly defines scope and deliverables\.

Q: "What were the economic effects of the 2008 financial crisis on European employment between 2009 and 2012?"

A: "Unemployment rates rose significantly, especially among young workers\."

Score: 0\.94

Rationale: Detailed timeframe, region, and economic focus yield very high clarity\.

\# ====== Perfect Clarity \(1\.0\) ======

Q: "What is the acceleration due to gravity on Earth at sea level, in meters per second squared?"

A: "Approximately 9\.81 m/s²\."

Score: 1\.0

Rationale: Fully specific, measurable, and unambiguous; perfect clarity\.

Q: "In what year did the Apollo 11 mission land on the Moon?"

A: "1969\."

Score: 0\.99

Rationale: Exact factual target, zero ambiguity\.

Q: "Which programming language introduced the concept of object\-oriented programming first?"

A: "Simula, introduced in the 1960s, was the first object\-oriented language\."

Score: 0\.98

Rationale: Clear, historical, and narrowly defined\.

Q: "What is the melting point of pure gold in degrees Celsius?"

A: "1064°C\."

Score: 0\.99

Rationale: Precise scientific measurement with no ambiguity\.

\# ====== Calibration \(fine granularity guidance\) ======

The following examples illustrate subtle scoring differences between similar questions, to encourage

non\-round, continuous scoring behavior\.

\# ====== Fine\-grained calibration examples ======

Q: "Tell me about renewable energy\."

A: "I cannot answer because the question is not clear enough\."

Score: 0\.25

Rationale: Broad topic without context or goal; only slightly better than incoherent because it names a concept\.

Q: "What are types of renewable energy sources?"

A: "Solar, wind, hydro, and geothermal are main renewable energy types\."

Score: 0\.37

Rationale: Slightly clearer; enumerative but still lacks focus, timeframe, or purpose\.

Q: "What are the main advantages of renewable energy?"

A: "It’s sustainable, reduces emissions, and lowers long\-term costs\."

Score: 0\.44

Rationale: Defines direction \(“advantages”\), improving specificity, though still broad in scope\.

Q: "What are the main advantages of solar energy in residential use?"

A: "It reduces electricity bills and reliance on the grid\."

Score: 0\.68

Rationale: Sharply more focused; defines technology \(solar\) and context \(residential\)\.

Q: "What are the main advantages of rooftop solar systems in reducing household electricity costs in Canada?"

A: "They provide stable long\-term savings and reduce grid dependency\."

Score: 0\.79

Rationale: Very specific geographic and functional framing; near high clarity but still not numerical or time\-bounded\.

Q: "What is the estimated average payback period, in years, for a 5kW rooftop solar system in Ontario under current energy prices?"

A: "Around 8 to 10 years, depending on usage and incentives\."

Score: 0\.93

Rationale: Fully concrete parameters \(location, system size, metric\) make this almost perfectly precise\.

Figure 15:Prompt used for the LLM to score questions from ClariQ \(continued\)\.
### Section E\.4Conformal Agentic AI for Web\-Based Tasks

Dataset and Task Setup\.We utilize the benchmark dataset proposed byHe et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib15)\), which initially comprises 643 tasks spanning 15 diverse websites, including Amazon, BBC News, Google Flights, and Wolfram Alpha\. To ensure a reliable evaluation environment, we filter out tasks rendered infeasible by external factors such as unremovable ads, mandatory login requests, or strict anti\-bot policies\. This filtering process results in a curated set of 542 tasks\. We partition these tasks into a calibration set \(80%\) and a test set \(20%\)\.

Sampling Strategy\.For each valid task in our dataset, we generate up to 10 candidate action trajectories using an iterative agentic framework\. More specifically, we always attempt to generate 10 trajectories, but due to occasional API call errors, some tasks end up with 8 or 9 trajectories\. To generate an action trajectory, we employ GPT\-4\.1 in an iterative process\. At each step, it is presented with a screenshot of the current browser view and the task context\. The model then samples a specific UI action \(e\.g\., clicking a coordinate, scrolling, or typing\)\. This action is executed by the backend browser, updating the state for the next step\. Detailed specifications of the action space and browser environment follow those fromHe et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib15)\)\.

Score Function\.To obtain the score function, we require the LLM to self\-assess its progress at every step of the action trajectory\. The model outputs a floating\-point number in the range\[0,1\]\[0,1\], quantifying the percentage of the task it believes has been completed \(with 1 indicating full success\)\. We achieve this by appending a specific scoring instruction to the original prompt template fromHe et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib15)\)\. The instruction isCompletion: \{A float between 0 and 1 indicating how much of the task has been completed until now\. Zero means no progress has been made and one means full completion\.\}\. Placing this instruction at the end allows us to extract the score while preserving the integrity of the original generation context\. Ultimately, we define the score of a full trajectory as the self\-assessment value provided at its final step\.

Admissibility Function\.To implement our admissibility function, we adopt the automated evaluation protocol established byHe et al\. \([2024](https://arxiv.org/html/2605.28920#bib.bib15)\)\. We employ GPT\-4\.1 as an automated judge, denoted byA′A^\{\\prime\}, to evaluate the performance of action trajectories\. Specifically, for a taskxxand a trajectory of actions𝐲t\\mathbf\{y\}\_\{t\}, the functionA′​\(x,𝐲t\)A^\{\\prime\}\(x,\\mathbf\{y\}\_\{t\}\)assigns a binary label of11\(success\) or0\(failure\) to𝐲t\\mathbf\{y\}\_\{t\}by analyzing the task description alongside the complete history of screenshots and text outputs; note that there is no ground truth here\. We adhere strictly to the assessment prompts and criteria detailed in the original study\.

### Section E\.5Conformal Random Forests

Detailed Method\.We trainedTTtreesF1,…,FTF\_\{1\},\\ldots,F\_\{T\}using a training dataset\. HereFt:𝒳→△C−1F\_\{t\}:\\mathcal\{X\}\\rightarrow\\triangle^\{C\-1\}, i\.e\., given a feature vectorXX, each tree will output a probability vector overCCpossible classes\. The prediction of the random forest modelFFwill beF​\(X\)=1T​∑t=1TFt​\(X\)F\(X\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}F\_\{t\}\(X\)\.

In our setting, given a treeFtF\_\{t\}, one can retrieve the predictionFt​\(X\)F\_\{t\}\(X\)\. As our scoring function should reflect confidence in the correctness ofFt​\(X\)F\_\{t\}\(X\), we use the weighted number of samples of the predicted leaf ofXXin the treeFtF\_\{t\}as a proxy\. This can be achieved by using theweighted\_n\_node\_samplesin theTreeclass in thesklearnPython package\(Pedregosa et al\.,[2011](https://arxiv.org/html/2605.28920#bib.bib37)\)\. LetWt​\(X\)W\_\{t\}\(X\)be the weighted number of samples of the predicted leaf ofXXin the treeFtF\_\{t\}\. Then we can letYt=\(Ft​\(X\),Wt​\(X\)\)Y\_\{t\}=\(F\_\{t\}\(X\),W\_\{t\}\(X\)\)\. Then the score functionS↑​\(X,Yt\)=Wt​\(X\)S^\{\\uparrow\}\(X,Y\_\{t\}\)=W\_\{t\}\(X\)\. We used theSmallestSubsetSumselection function described in[Appendix D](https://arxiv.org/html/2605.28920#A4)\.

The admissibilityAt′∈\{0,1\}A^\{\\prime\}\_\{t\}\\in\\\{0,1\\\}is11if and only ifFtF\_\{t\}predicted the correct class, i\.e\.,arg​max⁡Ft​\(X\)=YG​T\\operatorname\*\{arg\\,max\}F\_\{t\}\(X\)=Y\_\{GT\}\.888Note that this is an example where the elements of𝐘\\mathbf\{Y\}do not live in the same space asYGTY\_\{\\text\{GT\}\}: the former are pairs\(Ft​\(X\),Wt​\(X\)\)\(F\_\{t\}\(X\),W\_\{t\}\(X\)\), and the later is a class label\. In other words, this example highlights that Conf\-Gen remains valid when𝒴≠𝒴GT\\mathcal\{Y\}\\neq\\mathcal\{Y\}\_\{\\text\{GT\}\}\.Theaggfunction is11if the number of trees predicting the correct class is at leastkk, wherekkis a hyperparameter, and0otherwise\. This choice corresponds to[Equation 37](https://arxiv.org/html/2605.28920#A4.E37)withβ=k\\beta=k\(exceptAt′A\_\{t\}^\{\\prime\}is different in this experiment than in the example in[Appendix D](https://arxiv.org/html/2605.28920#A4)\)\. Thus the conformal guarantee is that at leastkktrees predicted the correct class\.

The dataset Click\_prediction\_small contains 35,953 train instances and 3,995 test instances, with 11 features and 2 classes\. This data is derived from the 2012 KDD Cup\. The data is subsampled to 0\.1% of the original number of instances, downsampling the majority class \(click=0\) so that the target feature is reasonably balanced \(5 to 1\)\. The data is about advertisements shown alongside search results in a search engine, and whether or not people clicked on these ads\. The task is to build the best possible model to predict whether a user will click on a given ad\.

In addition to the dataset Click\_prediction\_small, we performed experiments on four more datasets; we show the results below\.

Dataset GesturePhaseProcessed\.The dataset GesturePhaseProcessed\(Madeo et al\.,[2013](https://arxiv.org/html/2605.28920#bib.bib30)\)contains 8,885 train instances and 988 test instances, with 32 features and 5 classes\. The dataset is composed of features extracted from 7 videos with people gesticulating, and aims at studying gesture phase segmentation\. Each video is represented by two files: a raw file, which contains the position of hands, wrists, head, and spine of the user in each frame; and a processed file, which contains velocity and acceleration of hands and wrists\. We trained the random forest model on the train instances to achieve an AUC of0\.90270\.9027\.[Figure 16](https://arxiv.org/html/2605.28920#A5.F16)illustrates the average admissibility of the selected conformal sets on GesturePhaseProcessed\. The diagonal line represents the theoretical lower bound established in[Theorem 3](https://arxiv.org/html/2605.28920#Thmtheorem3), which is empirically satisfied for all values ofγ∈\[0,1\]\\gamma\\in\[0,1\]\.[Figure 16](https://arxiv.org/html/2605.28920#A5.F16)\(right\) plots the average number of trees used as a function ofγ\\gamma\.

Dataset Adult\.The dataset Adult\(Becker & Kohavi,[1996](https://arxiv.org/html/2605.28920#bib.bib5)\)contains 43,957 train instances and 4,885 test instances, with1414features and22classes\. The dataset prediction task is to determine whether a person makes over5050K a year\. Extraction was done by Barry Becker from the 1994 Census database\. We trained the random forest model on the train instances to achieve an AUC of0\.91040\.9104\.[Figure 17](https://arxiv.org/html/2605.28920#A5.F17)illustrates the average admissibility of the selected conformal sets on Adult\. The diagonal line represents the theoretical lower bound established in[Theorem 3](https://arxiv.org/html/2605.28920#Thmtheorem3), which is empirically satisfied for all values ofγ∈\[0,1\]\\gamma\\in\[0,1\]\.[Figure 17](https://arxiv.org/html/2605.28920#A5.F17)\(right\) plots the average number of trees used as a function ofγ\\gamma\.

Dataset Census\-Income\.The dataset Census\-Income\(U\.S\. Census Bureau,[2000](https://arxiv.org/html/2605.28920#bib.bib52)\)contains 269,356 train instances and 29,929 test instances, with4141features and22classes\. This dataset contains weighted census data extracted from the19941994and19951995“Current Population Surveys” conducted by the U\.S\. Census Bureau\. The data contains4141demographic and employment related variables\. We trained the random forest model on the train instances to achieve an AUC of0\.94780\.9478\.[Figure 18](https://arxiv.org/html/2605.28920#A5.F18)illustrates the average admissibility of the selected conformal sets on Census\-Income\. The diagonal line represents the theoretical lower bound established in[Theorem 3](https://arxiv.org/html/2605.28920#Thmtheorem3), which is empirically satisfied for all values ofγ∈\[0,1\]\\gamma\\in\[0,1\]\.[Figure 18](https://arxiv.org/html/2605.28920#A5.F18)\(right\) plots the average number of trees used as a function ofγ\\gamma\.

Dataset MiniBooNE\.The dataset MiniBooNE\(Roe,[2005](https://arxiv.org/html/2605.28920#bib.bib42)\)contains 117,057 train instances and 13,007 test instances, with5050features and22classes\. This dataset is taken from the MiniBooNE experiment and is used to distinguish electron neutrinos \(signal\) from muon neutrinos \(background\)\. We trained the random forest model on the train instances to achieve an AUC of0\.97900\.9790\.[Figure 19](https://arxiv.org/html/2605.28920#A5.F19)illustrates the average admissibility of the selected conformal sets on MiniBooNE\. The diagonal line represents the theoretical lower bound established in[Theorem 3](https://arxiv.org/html/2605.28920#Thmtheorem3), which is empirically satisfied for all values ofγ∈\[0,1\]\\gamma\\in\[0,1\]\.[Figure 19](https://arxiv.org/html/2605.28920#A5.F19)\(right\) plots the average number of trees used as a function ofγ\\gamma\.

Difference in performance\.We perform experiments on varying the strength of the conformal guaranteekk, and note their difference in the performance of the model by selecting the best subset of the100100trees\. To this end, for eachkkfrom55to5050, with an increment of55, and for eachγ\\gamma, we perform calibration on 100 instances in the test set\. Then, we perform conformal generation on the remaining instances in the test dataset and compute the performance using AUC with respect to the ground truth label\. For eachkk, we pick theγ\\gammawith the highest AUC\.[Figure 20](https://arxiv.org/html/2605.28920#A5.F20)shows the change in AUC for eachkk\. We note that for most datasets, ifkkis in a reasonable range, the change in AUC is minimal\. For Click\_prediction\_small, there is a minor increase in the AUC using this method\. This shows that it is possible to gain AUC “for free” by using an existing model and just a few data instances on the calibration dataset\.

![Refer to caption](https://arxiv.org/html/2605.28920v1/x7.png)Figure 16:Conformal generation admissibility and average sequence length as a function ofγ\\gammaof our conformal random forest on GesturePhaseProcessed\.![Refer to caption](https://arxiv.org/html/2605.28920v1/x8.png)Figure 17:Conformal generation admissibility and average sequence length as a function ofγ\\gammaof our conformal random forest on Adult\.![Refer to caption](https://arxiv.org/html/2605.28920v1/x9.png)Figure 18:Conformal generation admissibility and average sequence length as a function ofγ\\gammaof our conformal random forest on Census\-Income\.![Refer to caption](https://arxiv.org/html/2605.28920v1/x10.png)Figure 19:Conformal generation admissibility and average sequence length as a function ofγ\\gammaof our conformal random forest on MiniBooNE\.![Refer to caption](https://arxiv.org/html/2605.28920v1/x11.png)Figure 20:Change in performance \(AUC\) of our conformal random forests for various datasets andkkvalues\.