各有不足：在不同证据状态下校准语言模型

# Each Insufficient in Its Own Way: Calibrating Language Models Across Varying Evidence States
Source: [https://arxiv.org/html/2610.33237](https://arxiv.org/html/2610.33237)
Dennis L\. Shung⋄Bradly C\. Stadie⋆Affiliation:⋆Northwestern University⋄Mayo Clinic

###### Abstract

How can a language model know whether it has enough evidence to answer? Language models tend to answer even when the evidence they have is insufficient\. Deciding whether to answer or gather more evidence requires a calibrated estimate of how likely an answer is to be correct with the available evidence\. However, most calibration work treats the input as fixed, without asking how this estimate should change with the evidence\. We introduce*evidence\-state calibration*, which requires a model’s confidence before answering to match its probability of answering correctly and to change accordingly whenever the evidence changes this probability\. We propose SuffiCE \(SufficiencyConfidenceEstimation\), which fine\-tunes the confidence already present in a model’s next\-token distribution to match how often the model answers correctly from each evidence state\. Across clinical diagnosis and multi\-hop question answering with four models from two families, SuffiCE reduces the Brier score by up to 60% and increases AUROC by up to 0\.16\. Its confidence matches the empirical success rate and responds not only to the amount of evidence but also to which evidence is present, showing that models can learn to judge whether their current evidence is enough to answer\.

## 1Introduction

Before answering a question, one must judge whether the information at hand is enough to answer it\. In high\-stakes settings, a physician who diagnoses before gathering enough information may miss the real disease\. For example, a patient who has been tired for two months may seem iron deficient, but a careful clinician would order more blood tests, which may reveal celiac disease, an autoimmune cause of poor iron absorption\.

![Refer to caption](https://arxiv.org/html/2610.33237v1/figure/calibration.png)Figure 1:Given a question with two different sets of evidence, the model’s probability of answering correctly \(gray\) differs, and its confidence before answering \(yellow\) should match it for each\. \(a\) Calibrated confidence matches both\. \(b\) Miscalibrated confidence mismatches the dynamic of the true probability\.Language models facing insufficient information, however, often reach confident but incorrect conclusions\([Kirichenko et al\., 2025](https://arxiv.org/html/2610.33237#bib.bib1);[Yin et al\., 2023](https://arxiv.org/html/2610.33237#bib.bib2)\)\. A common remedy for such overconfidence is to abstain below a confidence threshold\([Wen et al\., 2025](https://arxiv.org/html/2610.33237#bib.bib3)\)\. This works only if the model is calibrated, so that answers given x% confidence are correct about x% of the time\([Guo et al\., 2017](https://arxiv.org/html/2610.33237#bib.bib11);[Geng et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib12)\)\. To achieve this, existing methods train language models to provide calibrated confidence estimates for generated answers\([Kapoor et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib13);[Bani\-Harouni et al\., 2026b](https://arxiv.org/html/2610.33237#bib.bib14)\)\.

However, most calibration work holds the evidence fixed for each question, so it can check whether confidence is calibrated but not whether it varies correctly as a function of the evidence\. In interactive settings, the available information evolves as models ask clarifying questions\([Zhang and Choi, 2025](https://arxiv.org/html/2610.33237#bib.bib25)\)or seek additional information\([Li et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib26)\)\. At each turn, the model must decide whether to answer or keep gathering evidence\.

To make this decision, a model needs a confidence estimated from the available evidence alone, before it produces any answer\. We call the evidence available to a model at a given moment its evidence state\. We then define*evidence\-state calibration*as requiring the confidence before answering to match the probability of answering correctly, so that whenever the evidence changes this probability, the confidence changes by the same amount \(Figure[1](https://arxiv.org/html/2610.33237#S1.F1)\)\.

Testing this requires observing the same problem under different evidence states\. Rather than contrasting the extremes of complete and missing information\([Kirichenko et al\., 2025](https://arxiv.org/html/2610.33237#bib.bib1)\)or following a single fixed sequence of accumulating evidence\([Zhang et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib15)\), we construct granular states for each problem, spanning minimal to complete evidence and including different combinations of information\. Granularity matters because sufficient evidence states are alike, but each insufficient one is insufficient in its own way\.111After the opening line of Leo Tolstoy’s*Anna Karenina*\(1878\): “Happy families are all alike; every unhappy family is unhappy in its own way\.”Some insufficient states can still be answered from what the model already knows\([Joren et al\., 2025](https://arxiv.org/html/2610.33237#bib.bib4)\), while others lack the one piece of evidence the answer depends on\. We therefore measure each state’s sufficiency by how often the model answers correctly from it, not by whether the required facts are present, and teach the model to anticipate this success rate before answering\.

With these success rates as targets, we propose SuffiCE \(SufficiencyConfidenceEstimation\), to our knowledge the first method that trains a model’s confidence before answering to be evidence\-state calibrated\. It reads the confidence directly from the model’s own next\-token distribution\. The confidence is the model’s probability of choosing to answer instead of asking for more information\. No additional classifier or generated number is needed\. This avoids an ambiguity in verbalized confidence\. When a model states that it is 80% confident, it is unclear whether to trust the stated number or the probability the model assigned to generating it\. We instead train the model’s own probability directly to match the success rate across our diverse evidence states\. The result is a model that can tell, before answering, whether what it currently knows is enough to answer correctly\.

We evaluate on two clinical diagnosis datasets, MIMIC\([Hager et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib32)\)and MedQA\([Jin et al\., 2021](https://arxiv.org/html/2610.33237#bib.bib33)\), and the multi\-hop question answering dataset MuSiQue\([Trivedi et al\., 2022](https://arxiv.org/html/2610.33237#bib.bib34)\)\. Across three Qwen models from 4B to 27B parameters and Kimi K2\.6, SuffiCE improves calibration for every model and dataset, reducing the Brier score of the base logit by up to 60% and increasing its AUROC by up to 0\.16\. Its confidence also follows the evidence state\. An estimator trained on complete evidence alone falls short on every metric, so these gains come from training across evidence states\. In practice, answering only when SuffiCE is confident gives fewer wrong answers than any baseline\.

## 2Related Work

Deciding whether to answer or gather more evidence requires a calibrated judgment of whether the current evidence suffices\. This connects three lines of work: interactive information seeking explores how models acquire evidence, information sufficiency and unanswerability examine whether an input supports an answer, and confidence calibration studies whether confidence reflects actual accuracy\.

#### Interactive information seeking\.

In an interactive setting, a model can gather more information before answering by asking clarifying questions\([Zhang and Choi, 2025](https://arxiv.org/html/2610.33237#bib.bib25);[Li et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib26)\), retrieving documents\([Jiang et al\., 2023](https://arxiv.org/html/2610.33237#bib.bib28);[Asai et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib30)\), or ordering diagnostic tests and searching for medical evidence\([Schmidgall et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib31);[Dang et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib29)\)\. Deciding when enough has been gathered typically relies on a confidence or uncertainty signal\([Jiang et al\., 2023](https://arxiv.org/html/2610.33237#bib.bib28);[Zhang and Choi, 2025](https://arxiv.org/html/2610.33237#bib.bib25);[Li et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib26)\), and MediQ shows that improving confidence estimation can enhance interactive diagnostic accuracy\([Li et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib26)\)\. Our approach aims to provide such a signal, estimating before answering how likely the current evidence is to yield a correct answer\. Closest to our work, LA\-CDM trains multi\-turn diagnosis with a verbalized confidence that guides whether to request another test, but this confidence evaluates a hypothesis the model has already generated\([Bani\-Harouni et al\., 2026a](https://arxiv.org/html/2610.33237#bib.bib27)\)\. We instead estimate success before generating an answer and train this estimate across controlled evidence states of the same problem, capturing how changes in evidence affect the likelihood of success\.

#### Information sufficiency and unanswerability\.

Whether an input contains enough information to answer a question has been widely studied through unanswerability\([Rajpurkar et al\., 2018](https://arxiv.org/html/2610.33237#bib.bib5);[Yin et al\., 2023](https://arxiv.org/html/2610.33237#bib.bib2)\)\. Benchmarks test whether models abstain when information is insufficient\([Kirichenko et al\., 2025](https://arxiv.org/html/2610.33237#bib.bib1)\), including settings where key evidence is removed from otherwise answerable questions\([Trivedi et al\., 2022](https://arxiv.org/html/2610.33237#bib.bib34);[Machcha et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib6)\)\. Detection methods assess sufficiency using model hidden states\([Slobodkin et al\., 2023](https://arxiv.org/html/2610.33237#bib.bib7);[Lavi et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib8)\)or by identifying missing information\([Jain and Garimella, 2026](https://arxiv.org/html/2610.33237#bib.bib9)\), and training methods teach models to abstain when information is insufficient\([Zhai et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib10)\)\. A shared limitation is that sufficiency is a binary label attached to the input, so a partially informative input counts as either sufficient or not, regardless of which model answers it\.[Joren et al\. \(2025\)](https://arxiv.org/html/2610.33237#bib.bib4)take a first step, showing that models often answer correctly from insufficient context\. Whether evidence is enough depends on the model answering, so we define sufficiency by the model’s success, how often it answers correctly from each controlled evidence state\.

#### Confidence calibration\.

A model is calibrated when its confidence matches its accuracy\([Guo et al\., 2017](https://arxiv.org/html/2610.33237#bib.bib11)\)\. Training\-free estimates based on verbalized judgments\([Tian et al\., 2023](https://arxiv.org/html/2610.33237#bib.bib16);[Xiong et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib17)\), sampled\-answer consistency\([Kuhn et al\., 2023](https://arxiv.org/html/2610.33237#bib.bib18)\), or token probabilities\([Kadavath et al\., 2022](https://arxiv.org/html/2610.33237#bib.bib19)\)can be informative, but calibration varies across tasks\([Vashurin et al\., 2025](https://arxiv.org/html/2610.33237#bib.bib20)\)\. Trained methods estimate answer correctness\([Kapoor et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib13)\), verbalize calibrated confidence\([Lin et al\., 2022](https://arxiv.org/html/2610.33237#bib.bib21);[Bani\-Harouni et al\., 2026b](https://arxiv.org/html/2610.33237#bib.bib14);[Damani et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib22);[Li et al\., 2025](https://arxiv.org/html/2610.33237#bib.bib24)\), or predict success before answering\([Kadavath et al\., 2022](https://arxiv.org/html/2610.33237#bib.bib19);[Lugoloobi et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib23)\)\. However, they do not explicitly train confidence to track success across changing evidence states\.[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.33237#bib.bib15)evaluate confidence as hints accumulate in a fixed sequence but do not train on these changing states\. We instead train and evaluate confidence across controlled evidence states, varying both the amount and type of evidence to track changes in empirical success \(Appendix[E](https://arxiv.org/html/2610.33237#A5)\)\.

## 3Method

Our method, SuffiCE, has three stages\. We first construct multiple evidence states for each problem \(§[3\.2](https://arxiv.org/html/2610.33237#S3.SS2)\), then estimate a fixed answering policy’s success rate on each state from repeated answers \(§[3\.3](https://arxiv.org/html/2610.33237#S3.SS3)\), and finally train an estimator to predict this rate from the state alone, before answering \(§[3\.4](https://arxiv.org/html/2610.33237#S3.SS4)\)\.

### 3\.1Problem Formulation

Problemiiconsists of a questionqiq\_\{i\}, a pool of evidenceEiE\_\{i\}, and a gold answeryi∗y^\{\*\}\_\{i\}\. An*evidence state*x=\(qi,I\)x=\(q\_\{i\},I\)pairs the question with the evidenceI⊆EiI\\subseteq E\_\{i\}available to the model\.

A fixed answering policyπ0\\pi\_\{0\}samples an answery^∼π0\(⋅∣x\)\\hat\{y\}\\sim\\pi\_\{0\}\(\\cdot\\mid x\), which a grader scores asg⁡\(y^,yi∗\)∈\{0,1\}g\(\\hat\{y\},y^\{\*\}\_\{i\}\)\\in\\\{0,1\\\}\. The*success probability*of an evidence state is

p∗\(x\)=Pry^∼π0\(⋅∣x\)\[g\(y^,yi∗\)=1\]\.p^\{\*\}\(x\)=\\Pr\_\{\\hat\{y\}\\sim\\pi\_\{0\}\(\\cdot\\mid x\)\}\\bigl\[g\(\\hat\{y\},y^\{\*\}\_\{i\}\)=1\\bigr\]\.\(1\)We measure how sufficient a state is byp∗​\(x\)p^\{\*\}\(x\), because the decision to answer or gather more information hinges on how likely an answer is to be correct\. Sufficiency therefore depends on the model as well as the information\. A state is sufficient when its information, together with what the model already knows, lets the model reach the correct answer, whether or not the information itself is complete\. The same state can thus be sufficient for one model and insufficient for another\.

![Refer to caption](https://arxiv.org/html/2610.33237v1/figure/main_overview.png)Figure 2:Overview of SuffiCE\. Each evidence statex=\(q,I\)x=\(q,I\)pairs an input questionqqwith the available evidenceII\.Top:A frozen answering policy samplesKKanswers per evidence state, and the fraction graded correct is the empirical success ratep^​\(x\)\\hat\{p\}\(x\)\.Bottom:The estimator reads its confidence from the softmax over the logits ofA\(Answer\) andS\(Seek\), and is trained with soft\-label binary cross\-entropy to matchp^​\(x\)\\hat\{p\}\(x\)\.
### 3\.2Evidence States

For each problemii, we constructmim\_\{i\}evidence statesxi,s=\(qi,Ii,s\)x\_\{i,s\}=\(q\_\{i\},I\_\{i,s\}\),s=1,…,mis=1,\\ldots,m\_\{i\}, each pairing the question with a subsetIi,s⊆EiI\_\{i,s\}\\subseteq E\_\{i\}of its evidence\. We writexxfor a generic state when the indices are not needed\. All states of a problem share the question and gold answer and differ only in the evidence shown, so differences in success between them are attributable to the evidence\. The model is never told which state it is in and sees only the question and the evidence, so it must judge from their content whether the evidence is enough\. Training on the same problem under different evidence teaches it which evidence makes a correct answer reachable\.

Evidence states vary in both the amount and the composition of evidence\. Since success depends on which evidence is present, not on the order it was gathered, states can combine evidence freely, including states that hold the amount fixed and vary the composition \(§[5\.3](https://arxiv.org/html/2610.33237#S5.SS3)\)\. The states are specific to each dataset \(§[4](https://arxiv.org/html/2610.33237#S4.SS0.SSS0.Px1)\)\.

#### Evidence\-state calibration\.

We seek a confidencecθ​\(x\)∈\[0,1\]c\_\{\\theta\}\(x\)\\in\[0,1\]that a model with parametersθ\\thetacomputes from the statexxalone, before any answer is generated\. Evidence\-state calibration requires it to satisfy two criteria, where expectations are taken over the constructed evidence states\.

1\. Calibration\.Across all states assigned the same confidencecc, the average success probability iscc\([Guo et al\., 2017](https://arxiv.org/html/2610.33237#bib.bib11)\):

𝔼⁡\[p∗​\(x\)∣cθ​\(x\)=c\]=c∀c∈range⁡\(cθ\)\.\\mathbb\{E\}\\bigl\[p^\{\*\}\(x\)\\mid c\_\{\\theta\}\(x\)=c\\bigr\]=c\\quad\\forall c\\in\\operatorname\{range\}\(c\_\{\\theta\}\)\.\(2\)
2\. Evidence sensitivity\.A change in evidence moves confidence as much as it moves success:

cθ​\(xi,t\)−cθ​\(xi,s\)≈p∗​\(xi,t\)−p∗​\(xi,s\)\.c\_\{\\theta\}\(x\_\{i,t\}\)\-c\_\{\\theta\}\(x\_\{i,s\}\)\\approx p^\{\*\}\(x\_\{i,t\}\)\-p^\{\*\}\(x\_\{i,s\}\)\.\(3\)Herexi,sx\_\{i,s\}andxi,tx\_\{i,t\}denote any two evidence statesssandttof the same problemii, so the difference in confidence should match the difference in success\.

To see why calibration alone is not enough, we split the squared error of a confidence by problem\. For problemiiwithmim\_\{i\}states, letc¯θ,i\\bar\{c\}\_\{\\theta,i\}andp¯i∗\\bar\{p\}^\{\*\}\_\{i\}denote the averages ofcθc\_\{\\theta\}andp∗p^\{\*\}over its states, and for any two of its statess,t∈\{1,…,mi\}s,t\\in\\\{1,\\ldots,m\_\{i\}\\\}, letδs​t=\(cθ​\(xi,t\)−cθ​\(xi,s\)\)−\(p∗​\(xi,t\)−p∗​\(xi,s\)\)\\delta\_\{st\}=\\bigl\(c\_\{\\theta\}\(x\_\{i,t\}\)\-c\_\{\\theta\}\(x\_\{i,s\}\)\\bigr\)\-\\bigl\(p^\{\*\}\(x\_\{i,t\}\)\-p^\{\*\}\(x\_\{i,s\}\)\\bigr\)be the gap between the change in confidence and the change in success fromsstott, which is zero when Eq\. \([3](https://arxiv.org/html/2610.33237#S3.E3)\) holds exactly\.

Error Decomposition###### Proposition 1\. For any confidencecθc\_\{\\theta\},𝔼⁡\[\(cθ−p∗\)2\]=𝔼i​\[\(c¯θ,i−p¯i∗\)2\]\+𝔼i​\[12​mi2​∑s,tδs​t2\],\\begin\{split\}\\mathbb\{E\}\\bigl\[\(c\_\{\\theta\}\-p^\{\*\}\)^\{2\}\\bigr\]&=\\mathbb\{E\}\_\{i\}\\bigl\[\(\\bar\{c\}\_\{\\theta,i\}\-\\bar\{p\}^\{\*\}\_\{i\}\)^\{2\}\\bigr\]\\\\ &\\quad\+\\mathbb\{E\}\_\{i\}\\Bigl\[\\tfrac\{1\}\{2m\_\{i\}^\{2\}\}\\textstyle\\sum\_\{s,t\}\\delta\_\{st\}^\{2\}\\Bigr\],\\end\{split\}\(4\)where the sum runs over all ordered pairs of states of problemii, and each problem is weighted by its number of states\.

The first term is the error in each problem’s average success, and the second is the error within each problem, which is zero exactly when[criterion 2](https://arxiv.org/html/2610.33237#S3.SS2.SSS0.Px1)holds with equality \(proof in Appendix[D\.1](https://arxiv.org/html/2610.33237#A4.SS1)\)\. Calibration does not constrain the second term\. Giving every state its problem’s average success satisfies[criterion 1](https://arxiv.org/html/2610.33237#S3.SS2.SSS0.Px1)exactly but leaves this term at the variance of success within each problem, which only following the evidence can reduce\. Since both terms are nonnegative, we seek a confidence that drives the left side to zero and thus satisfies both criteria\.

### 3\.3Monte Carlo Estimation of Success Probability

Because computing the true success probabilityp∗​\(x\)p^\{\*\}\(x\)exactly would require grading every answer the policy could generate, we estimate it by sampling from the fixed answering policy \(Figure[2](https://arxiv.org/html/2610.33237#S3.F2), top\)\. A single answer is only correct or incorrect, so we drawKKindependent answers per state and average their grades\. For each evidence statexxof problemii, we sampley^\(1\),…,y^\(K\)∼π0\(⋅∣x\)\\hat\{y\}^\{\(1\)\},\\ldots,\\hat\{y\}^\{\(K\)\}\\sim\\pi\_\{0\}\(\\cdot\\mid x\), grade each asr\(k\)=g⁡\(y^\(k\),yi∗\)r^\{\(k\)\}=g\(\\hat\{y\}^\{\(k\)\},y^\{\*\}\_\{i\}\), and define the empirical success rate as

p^​\(x\)=1K​∑k=1Kr\(k\)\.\\hat\{p\}\(x\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}r^\{\(k\)\}\.\(5\)Sincer\(k\)∼Bernoulli⁡\(p∗​\(x\)\)r^\{\(k\)\}\\sim\\mathrm\{Bernoulli\}\(p^\{\*\}\(x\)\),𝔼⁡\[p^​\(x\)∣x\]=p∗​\(x\)\\mathbb\{E\}\[\\hat\{p\}\(x\)\\mid x\]=p^\{\*\}\(x\), sop^​\(x\)\\hat\{p\}\(x\)is an unbiased Monte Carlo estimate of the frozen policy’s success probability\.

The estimatep^​\(x\)\\hat\{p\}\(x\)is specific to the answering policyπ0\\pi\_\{0\}, so the sufficiency of a statexxalways refers to whetherπ0\\pi\_\{0\}can answer correctly given the information inxx, and the confidence we train is calibrated to this policy\. Since computingp^​\(x\)\\hat\{p\}\(x\)requires the gold answer, which is unavailable at decision time, we use it only as a training target\.

### 3\.4Sufficiency Confidence Estimator

At decision time, the model must therefore predictp^​\(x\)\\hat\{p\}\(x\)from the evidence state alone\. We read this prediction from the model’s own next\-token distribution rather than adding a separate classifier head \(Figure[2](https://arxiv.org/html/2610.33237#S3.F2), bottom\)\. Each evidence state is placed in a prompt that asks the model, without answering the task, whether the currently available information is sufficient for it to answer correctly, and defines two actions,Ato answer now because the information is sufficient, andSto seek more information because it is not\. Rather than letting the model generate a response, we read the probabilities it assigns toAandSas the first token of its reply \(prompt in Appendix[B\.1](https://arxiv.org/html/2610.33237#A2.SS1)\)\.

LetℓA​\(x\)\\ell\_\{A\}\(x\)andℓS​\(x\)\\ell\_\{S\}\(x\)denote the logits assigned toAandSat the first response position from a single forward pass\. We take the softmax over these two logits, givingcθ​\(x\)=Pθ​\(A∣x\)=exp⁡\(ℓA\)/\(exp⁡\(ℓA\)\+exp⁡\(ℓS\)\)c\_\{\\theta\}\(x\)=P\_\{\\theta\}\(A\\mid x\)=\\exp\(\\ell\_\{A\}\)/\\bigl\(\\exp\(\\ell\_\{A\}\)\+\\exp\(\\ell\_\{S\}\)\\bigr\)\. We callcθ​\(x\)c\_\{\\theta\}\(x\)the*sufficiency confidence*of statexx, and the model that produces it the*sufficiency confidence estimator*\.

We traincθ​\(x\)c\_\{\\theta\}\(x\)to match the empirical success rate using soft\-label binary cross\-entropy:

ℒ⁡\(x\)=−p^​\(x\)​log⁡cθ​\(x\)−\(1−p^​\(x\)\)​log⁡\(1−cθ​\(x\)\)\.\\begin\{split\}\\mathcal\{L\}\(x\)=\{\}&\-\\hat\{p\}\(x\)\\log c\_\{\\theta\}\(x\)\\\\ &\-\\bigl\(1\-\\hat\{p\}\(x\)\\bigr\)\\log\\bigl\(1\-c\_\{\\theta\}\(x\)\\bigr\)\.\\end\{split\}\(6\)
Binary cross\-entropy is a strictly proper scoring rule\([Gneiting and Raftery, 2007](https://arxiv.org/html/2610.33237#bib.bib36)\)and is linear in the target, so with the unbiased labels of §[3\.3](https://arxiv.org/html/2610.33237#S3.SS3)the expected loss at each state is minimized bycθ​\(x\)=p∗​\(x\)c\_\{\\theta\}\(x\)=p^\{\*\}\(x\)despite sampling noise inp^​\(x\)\\hat\{p\}\(x\)\. Training on this loss also controls both terms of Proposition[1](https://arxiv.org/html/2610.33237#Thmproposition1)\.

###### Corollary 1\.

For any confidencecθc\_\{\\theta\}, let𝔼⁡\[ℒ\]\\mathbb\{E\}\[\\mathcal\{L\}\]be the loss of Eq\. \([6](https://arxiv.org/html/2610.33237#S3.E6)\) averaged over all evidence states and over theKKsampled answers that determinep^​\(x\)\\hat\{p\}\(x\), andℒ∗\\mathcal\{L\}^\{\*\}the minimum expected loss over all confidences, reached exactly atcθ=p∗c\_\{\\theta\}=p^\{\*\}\. Then

𝔼⁡\[\(cθ−p∗\)2\]≤12​\(𝔼⁡\[ℒ\]−ℒ∗\),\\mathbb\{E\}\\bigl\[\(c\_\{\\theta\}\-p^\{\*\}\)^\{2\}\\bigr\]\\leq\\tfrac\{1\}\{2\}\\bigl\(\\mathbb\{E\}\[\\mathcal\{L\}\]\-\\mathcal\{L\}^\{\*\}\\bigr\),\(7\)so each term of Eq\. \([4](https://arxiv.org/html/2610.33237#S3.E4)\) is at most12​\(𝔼​\[ℒ\]−ℒ∗\)\\tfrac\{1\}\{2\}\\bigl\(\\mathbb\{E\}\[\\mathcal\{L\}\]\-\\mathcal\{L\}^\{\*\}\\bigr\)\.

Lowering the loss therefore lowers an upper bound on both terms, and the bound reaches zero exactly atcθ=p∗c\_\{\\theta\}=p^\{\*\}\(proof in Appendix[D\.2](https://arxiv.org/html/2610.33237#A4.SS2)\)\.

## 4Experimental Setup

#### Datasets\.

We evaluate on two clinical diagnosis datasets and one multi\-hop question answering dataset, which range from mostly answered correctly to mostly not \(details in Appendix[A\.1](https://arxiv.org/html/2610.33237#A1.SS1)\)\. MIMIC\([Hager et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib32)\)consists of real emergency\-department admissions for four abdominal diseases, and the model chooses among the four\. MedQA\([Jin et al\., 2021](https://arxiv.org/html/2610.33237#bib.bib33)\)contains questions from the United States Medical Licensing Examination, and the model names the diagnosis in free text\. Both are written as structured clinical cases following AgentClinic\([Schmidgall et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib31)\)\. MuSiQue\([Trivedi et al\., 2022](https://arxiv.org/html/2610.33237#bib.bib34)\)composes multi\-hop questions over Wikipedia from single\-hop questions, each answered by one supporting paragraph, and the model gives a short answer\. All states of a problem are assigned to the same split to avoid leakage between training and test\.

#### Evidence states\.

Clinical cases are divided into history, physical examination, imaging, and other tests such as laboratory and pathology\. Each state adds a different subset of the remaining sections to the history\.

Table 1:Results on the three datasets for Kimi K2\.6, Qwen3\.8\-27B, Qwen3\.5\-9B, and Qwen3\.5\-4B, comparing our method with baselines\. Results are mean±\\pmSD over three seeds\. The best result for each model is in bold\.Figure 3:Evidence states of MuSiQue for chain and branching questions\. Hatched cells give the number of distractors padding each state to three paragraphs\.In MuSiQue, we use 3\-hop questions, whose evidence units are the three supporting paragraphs, one per sub\-question\. In chain questions, each answer is needed to ask the next sub\-question \(e1→e2→e3e\_\{1\}\\rightarrow e\_\{2\}\\rightarrow e\_\{3\}\), while in branching questions,e1e\_\{1\}ande2e\_\{2\}answer independent sub\-questions that both feed the final one,e3e\_\{3\}\. States range from the question alone and distractors only \(related paragraphs that MuSiQue retrieves but that do not help answer the question\) to one, two, or all three supporting paragraphs \(Figure[3](https://arxiv.org/html/2610.33237#S4.F3), examples in Appendix[F\.2](https://arxiv.org/html/2610.33237#A6.SS2)\)\. Every state but the question\-only one shows three paragraphs, padded with distractors in one shuffled order per question, so neither the number nor the position of paragraphs reveals the state\.

#### Baselines\.

We compare against three baselines on the same untrained model\.*Base logit*\(*Base*\) is our sufficiency confidence estimator before training, readingP⁡\(A∣x\)P\(A\\mid x\)under the prompt of §[3\.4](https://arxiv.org/html/2610.33237#S3.SS4)\.*Verbalized*asks the model to state the probability that it could answer correctly from the available information\([Tian et al\., 2023](https://arxiv.org/html/2610.33237#bib.bib16)\)\.*Platt*scaling\([Platt, 1999](https://arxiv.org/html/2610.33237#bib.bib37)\)rescales the base logit to fit the success rates in the training set, so it can recalibrate the confidence but not reorder states \(prompts in Appendix[B](https://arxiv.org/html/2610.33237#A2)\)\.

#### Implementation details\.

We evaluate four answering models, three Qwen models of increasing size \(Qwen3\.5\-4B, Qwen3\.5\-9B, and Qwen3\.8\-27B\) and Kimi K2\.6 from a different model family\. The answering policyπ0\\pi\_\{0\}can be any frozen policy\. Our main experiments use the estimator’s own base model, to study how well a model can judge its own evidence state\. Success rates are estimated fromK=10K=10sampled answers per state\. Further details are in Appendix[A\.3](https://arxiv.org/html/2610.33237#A1.SS3)\.

#### Metrics\.

Letci,s=cθ​\(xi,s\)c\_\{i,s\}=c\_\{\\theta\}\(x\_\{i,s\}\)be the sufficiency confidence of statessof problemiiandp^i,s\\hat\{p\}\_\{i,s\}its empirical success rate\. The*Brier*score, the mean of\(ci,s−p^i,s\)2\(c\_\{i,s\}\-\\hat\{p\}\_\{i,s\}\)^\{2\}over the test set, measures overall error\.*AUROC*, averaged over theKKsampled answers, measures how well confidence ranks correct answers above incorrect ones\. Both pool the whole test set, so they reward knowing which problems are hard\. The*change correlation*ρΔ=Corr⁡\(ci,t−ci,s,p^i,t−p^i,s\)\\rho\_\{\\Delta\}=\\operatorname\{Corr\}\(c\_\{i,t\}\-c\_\{i,s\},\\,\\hat\{p\}\_\{i,t\}\-\\hat\{p\}\_\{i,s\}\), computed over all pairs of states of the same problem from a statessto a statettwith more evidence \(Appendix[C\.4](https://arxiv.org/html/2610.33237#A3.SS4)\), instead tests whether confidence follows success as the evidence state changes, as[criterion 2](https://arxiv.org/html/2610.33237#S3.SS2.SSS0.Px1)requires\.

## 5Results

### 5\.1Overall Performance

SuffiCE achieves the best result on every metric in every setting \(Table[1](https://arxiv.org/html/2610.33237#S4.T1)\)\. Each model is calibrated to its own success rates, so methods are compared only for the same model \(Appendix[A\.2](https://arxiv.org/html/2610.33237#A1.SS2)\)\. SuffiCE lowers the Brier score by 20–60% relative to the base logit and by 5–28% relative to Platt scaling, the strongest baseline\. SuffiCE not only calibrates better but also ranks states better\. Platt scaling fits the same labels but only rescales the base confidence, leaving the order of states, and thus AUROC and selective accuracy, unchanged, whereas SuffiCE raises AUROC by up to 0\.16\. SuffiCE also raises the change correlationρΔ\\rho\_\{\\Delta\}above every baseline, so its confidence moves with success as the evidence changes, which §[5\.3](https://arxiv.org/html/2610.33237#S5.SS3)examines in detail\.

Figure 4:SuffiCE brings confidence closer to perfect calibration on every dataset\. Calibration curves on the test set \(a\) before and \(b\) after SuffiCE\.#### Calibration across datasets\.

The base logit is miscalibrated by different amounts across datasets, and SuffiCE improves calibration on all three, moving each curve toward the ideal diagonal \(Figure[4](https://arxiv.org/html/2610.33237#S5.F4), Qwen3\.5\-9B\)\. The other models follow the same pattern \(Appendix[C\.1](https://arxiv.org/html/2610.33237#A3.SS1)\)\.

#### Selective answering\.

If the model answers only the questions it is most confident about and abstains on the rest, SuffiCE is the most accurate at nearly every percentage of questions answered \(Table[2](https://arxiv.org/html/2610.33237#S5.T2)and Appendix[C\.2](https://arxiv.org/html/2610.33237#A3.SS2)\)\. On MIMIC, the questions it is most confident about are almost always answered correctly\.

Table 2:Accuracy \(%\) when the model answers the given percentage of questions it is most confident about\.

### 5\.2Ablation

To test whether the estimator needs diverse evidence states, we train Qwen3\.5\-9B only on the full state of each training problem and evaluate it on the whole test set \(Figure[5](https://arxiv.org/html/2610.33237#S5.F5)\)\. Success on full states still varies across problems, so this estimator learns which problems are hard but never sees missing evidence\. SuffiCE outperforms it on every metric and dataset, both with the same number of training epochs and with the same number of steps \(Appendix[C\.3](https://arxiv.org/html/2610.33237#A3.SS3)\)\.

Figure 5:Training across evidence states improves both calibration and evidence sensitivity\. Estimators trained for the same number of epochs on full states only versus on all evidence states \(SuffiCE\)\.
### 5\.3Evidence\-State Analysis

#### Confidence rises with the decisive evidence, not with more evidence\.

SuffiCE closely tracks the empirical success rate across states, rising only when new evidence makes the question answerable, whereas the base logit rises with every paragraph added \(Figure[6](https://arxiv.org/html/2610.33237#S5.F6), more measures in Appendix[C\.4](https://arxiv.org/html/2610.33237#A3.SS4)\)\. For both question types, success stays near zero until the final hope3e\_\{3\}is shown\. On chain questions, the base logit treats each earlier hop as progress and is confident enough to answer given\{e1,e2\}\\\{e\_\{1\},e\_\{2\}\\\}\(0\.71\), although the model almost never succeeds \(0\.06\), while SuffiCE stays low \(0\.16\) untile3e\_\{3\}arrives\. On branching questions, the base logit rises with each branch, while SuffiCE stays low for either branch alone and for both, since neither helps withoute3e\_\{3\}\. SuffiCE is never told the question type or which paragraph fills which hop, so it judges from the content alone whether the reasoning chain is complete\. It thus separates evidence that makes a question answerable from evidence that only adds information \(Appendix[F\.1](https://arxiv.org/html/2610.33237#A6.SS1)\)\.

Figure 6:SuffiCE tracks the empirical success rate as supporting paragraphs are added, while the base logit rises with every paragraph\. MuSiQue test set, for \(a\) chain and \(b\) branching questions\.Figure 7:With the same amount of evidence, SuffiCE tells which states can be answered\. Each MuSiQue state shows two of the three supporting paragraphs, for \(a\) chain and \(b\) branching questions\. Only SuffiCE stays low until the final hope3e\_\{3\}is shown\.
#### Holding the amount fixed, content decides success\.

Even when every state has the same amount of evidence, SuffiCE distinguishes which states can be answered \(Figure[7](https://arxiv.org/html/2610.33237#S5.F7)\)\. Each state shows two of the three supporting paragraphs, and success is high only when the final hope3e\_\{3\}is among them\. The base logit is similarly confident on all three states, responding to how much evidence there is, whereas SuffiCE stays low on\{e1,e2\}\\\{e\_\{1\},e\_\{2\}\\\}and rises for both states withe3e\_\{3\}\.

#### Confidence separates decisive from suggestive evidence\.

In clinical diagnosis, the decisive evidence differs from patient to patient\. Table[3](https://arxiv.org/html/2610.33237#S5.T3)shows a MedQA case of Hirschsprung disease\. The X\-ray is the pivot of the workup\. It reveals a distal obstruction and prompts the rectal biopsy, but it cannot tell Hirschsprung disease from other causes of obstruction\. Only the biopsy, which finds no ganglion cells, confirms the diagnosis\. The base logit is nearly certain in every state, whereas SuffiCE stays uncertain with the X\-ray in state\{1,4\}\\\{1,4\\\}and becomes confident only once the biopsy is added in state\{1,3,4\}\\\{1,3,4\\\}\. With a suitable confidence threshold, SuffiCE would therefore seek more information before answering\. Unlike the base logit, SuffiCE thus recognizes which evidence settles the diagnosis and which only signals that more information is needed\.

Table 3:A MedQA case of Hirschsprung disease \(Qwen3\.5\-9B\)\. Each state is the set of evidence shown\. Success is the fraction of ten sampled answers that are correct\. Base and SuffiCE give the sufficiency confidence\.

### 5\.4Cross\-Policy Confidence Estimation

To test whether the estimator must be the answering model, we fix the answering policy to Kimi K2\.6 and train Qwen estimators of three sizes on its success rates \(Figure[8](https://arxiv.org/html/2610.33237#S5.F8)\)\. A large enough estimator predicts another model’s success as well as that model does itself\. The 27B estimator matches or exceeds Kimi K2\.6’s own estimate on every dataset in Brier score, and inρΔ\\rho\_\{\\Delta\}on MedQA and MuSiQue\. Trained only on Kimi K2\.6’s success rates and never shown its answers, the estimator thus learns to anticipate from the evidence state alone how well another model can answer\. Smaller estimators are less accurate, and the 4B estimator falls well short of Kimi K2\.6’s own estimate\.

Figure 8:Qwen estimators of three sizes are trained to predict the success rates of Kimi K2\.6, compared with Kimi K2\.6 predicting its own \(self\)\.

## 6Conclusion

To our knowledge, this is the first work to train a model’s confidence before answering to be calibrated across varying evidence states\. The decision to answer or gather more evidence is made before any answer exists, so a model is only as reliable as this judgment\. We formalized it as*evidence\-state calibration*, requiring confidence to be calibrated and to change as much as the chance of success whenever the evidence changes, which calibration alone does not guarantee\. SuffiCE outperforms every baseline, and its confidence rises with the decisive evidence, not with every piece added\. Such confidence is a step toward language models that can judge, in high\-stakes settings, when the evidence at hand is enough to answer\.

## Limitations

SuffiCE shows strong calibration and evidence sensitivity across four answering models and three datasets, but it has three limitations\. First, we preconstruct evidence states rather than gather evidence in live multi\-turn interactions, a controlled setting that lets us attribute differences in success and confidence to the evidence itself\. Second, to limit cost, we estimate each target from ten sampled answers per state, and more samples would give more precise targets\. Third, SuffiCE has not been tested in real clinical scenarios, and its confidence should not replace clinical judgment\.

## References

- Asaiet al\.\(2024\)A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. HajishirziSelf\-rag: learning to retrieve, generate, and critique through self\-reflection\.InICLR,Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px1.p1.1)\.
- Bani\-Harouniet al\.\(2026a\)D\. Bani\-Harouni, C\. Pellegrini, E\. Özsoy, N\. Navab, and M\. KeicherLanguage agents for hypothesis\-driven clinical decision making with reinforcement learning\.InICLR,Cited by:[§D\.2](https://arxiv.org/html/2610.33237#A4.SS2.SSS0.Px2.p3.1.6.1),[Appendix E](https://arxiv.org/html/2610.33237#A5.p1.2),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px1.p1.1)\.
- Bani\-Harouniet al\.\(2026b\)D\. Bani\-Harouni, C\. Pellegrini, P\. Stangel, E\. Özsoy, K\. Zaripova, N\. Navab, and M\. KeicherRewarding doubt: A reinforcement learning approach to calibrated confidence expression of large language models\.InICLR,Cited by:[§D\.2](https://arxiv.org/html/2610.33237#A4.SS2.SSS0.Px2.p3.1.5.1),[Appendix E](https://arxiv.org/html/2610.33237#A5.p1.2),[§1](https://arxiv.org/html/2610.33237#S1.p2.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1)\.
- Damaniet al\.\(2026\)M\. Damani, I\. Puri, S\. Slocum, I\. Shenfeld, L\. Choshen, Y\. Kim, and J\. AndreasBeyond binary rewards: training lms to reason about their uncertainty\.InICLR,Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1)\.
- Danget al\.\(2026\)X\. Dang, K\. Chen, X\. Su, A\. Noori, I\. Arango, L\. Vittor, X\. Long, Y\. Du, M\. Zitnik, and P\. HengKnowGuard: knowledge\-driven abstention for multi\-round clinical reasoning\.InICLR,Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px1.p1.1)\.
- Genget al\.\(2024\)J\. Geng, F\. Cai, Y\. Wang, H\. Koeppl, P\. Nakov, and I\. GurevychA survey of confidence estimation and calibration in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 6577–6595\.External Links:[Link](https://aclanthology.org/2024.naacl-long.366/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.366)Cited by:[§1](https://arxiv.org/html/2610.33237#S1.p2.1)\.
- Gneiting and Raftery \(2007\)T\. Gneiting and A\. E\. RafteryStrictly proper scoring rules, prediction, and estimation\.Journal of the American Statistical Association102\(477\),pp\. 359–378\.External Links:[Document](https://dx.doi.org/10.1198/016214506000001437)Cited by:[§3\.4](https://arxiv.org/html/2610.33237#S3.SS4.p4.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InICML,Proceedings of Machine Learning Research, Vol\.70,pp\. 1321–1330\.Cited by:[§1](https://arxiv.org/html/2610.33237#S1.p2.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2610.33237#S3.SS2.SSS0.Px1.p2.1)\.
- Hageret al\.\(2024\)P\. Hager, F\. Jungmann, R\. Holland, K\. Bhagat, I\. Hubrecht, M\. Knauer, J\. Vielhauer, M\. Makowski, R\. Braren, G\. Kaissis, and D\. RueckertEvaluation and mitigation of the limitations of large language models in clinical decision\-making\.Nature Medicine30\(9\),pp\. 2613–2622\.External Links:[Document](https://dx.doi.org/10.1038/s41591-024-03097-1)Cited by:[§A\.1](https://arxiv.org/html/2610.33237#A1.SS1.SSS0.Px1.p1.1),[Appendix G](https://arxiv.org/html/2610.33237#A7.p1.1),[§1](https://arxiv.org/html/2610.33237#S1.p7.1),[§4](https://arxiv.org/html/2610.33237#S4.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§A\.3](https://arxiv.org/html/2610.33237#A1.SS3.p1.1)\.
- Jain and Garimella \(2026\)A\. Jain and A\. GarimellaKnowing what’s missing: assessing information sufficiency in question answering\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 4163–4174\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.217/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.217),ISBN 979\-8\-89176\-386\-9Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px2.p1.1)\.
- Jianget al\.\(2023\)Z\. Jiang, F\. Xu, L\. Gao, Z\. Sun, Q\. Liu, J\. Dwivedi\-Yu, Y\. Yang, J\. Callan, and G\. NeubigActive retrieval augmented generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 7969–7992\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.495/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.495)Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px1.p1.1)\.
- Jinet al\.\(2021\)D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. SzolovitsWhat disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.Applied Sciences11\(14\),pp\. 6421\.External Links:[Document](https://dx.doi.org/10.3390/app11146421)Cited by:[§A\.1](https://arxiv.org/html/2610.33237#A1.SS1.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.33237#S1.p7.1),[§4](https://arxiv.org/html/2610.33237#S4.SS0.SSS0.Px1.p1.1)\.
- Jorenet al\.\(2025\)H\. Joren, J\. Zhang, C\. Ferng, D\. Juan, A\. Taly, and C\. RashtchianSufficient context: A new lens on retrieval augmented generation systems\.InICLR,Cited by:[§1](https://arxiv.org/html/2610.33237#S1.p5.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px2.p1.1)\.
- Kadavathet al\.\(2022\)S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson, S\. Johnston, S\. El\-Showk, A\. Jones, N\. Elhage, T\. Hume, A\. Chen, Y\. Bai, S\. Bowman, S\. Fort, D\. Ganguli, D\. Hernandez, J\. Jacobson, J\. Kernion, S\. Kravec, L\. Lovitt, K\. Ndousse, C\. Olsson, S\. Ringer, D\. Amodei, T\. Brown, J\. Clark, N\. Joseph, B\. Mann, S\. McCandlish, C\. Olah, and J\. KaplanLanguage models \(mostly\) know what they know\.External Links:2207\.05221,[Link](https://arxiv.org/abs/2207.05221)Cited by:[§D\.2](https://arxiv.org/html/2610.33237#A4.SS2.SSS0.Px2.p3.1.2.1),[Appendix E](https://arxiv.org/html/2610.33237#A5.p1.2),[Appendix E](https://arxiv.org/html/2610.33237#A5.p2.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1)\.
- Kapooret al\.\(2024\)S\. Kapoor, N\. Gruver, M\. Roberts, K\. Collins, A\. Pal, U\. Bhatt, A\. Weller, S\. Dooley, M\. Goldblum, and A\. G\. WilsonLarge language models must be taught to know what they don’t know\.InNeurIPS,Cited by:[§D\.2](https://arxiv.org/html/2610.33237#A4.SS2.SSS0.Px2.p3.1.4.1),[Appendix E](https://arxiv.org/html/2610.33237#A5.p1.2),[Appendix E](https://arxiv.org/html/2610.33237#A5.p2.1),[§1](https://arxiv.org/html/2610.33237#S1.p2.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1)\.
- Kingma and Ba \(2015\)D\. P\. Kingma and J\. BaAdam: a method for stochastic optimization\.InInternational Conference on Learning Representations,Cited by:[Table 7](https://arxiv.org/html/2610.33237#A1.T7.2.10.2)\.
- Kirichenkoet al\.\(2025\)P\. Kirichenko, M\. Ibrahim, K\. Chaudhuri, and S\. J\. BellAbstentionBench: reasoning llms fail on unanswerable questions\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2610.33237#S1.p2.1),[§1](https://arxiv.org/html/2610.33237#S1.p5.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px2.p1.1)\.
- Kuhnet al\.\(2023\)L\. Kuhn, Y\. Gal, and S\. FarquharSemantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InICLR,Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1)\.
- Laviet al\.\(2026\)M\. J\. Lavi, T\. Milo, and M\. GevaDetecting \(un\)answerability in large language models with linear directions\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 682–699\.External Links:[Link](https://aclanthology.org/2026.eacl-long.29/),[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.29),ISBN 979\-8\-89176\-380\-7Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2024\)S\. S\. Li, V\. Balachandran, S\. Feng, J\. Ilgen, E\. Pierson, P\. W\. Koh, and Y\. TsvetkovMediQ: question\-asking llms and a benchmark for reliable interactive clinical reasoning\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2610.33237#S1.p3.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2025\)Y\. Li, M\. Xiong, J\. Wu, and B\. HooiConfTuner: training large language models to express their confidence verbally\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1)\.
- Linet al\.\(2022\)S\. Lin, J\. Hilton, and O\. EvansTeaching models to express their uncertainty in words\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=8s8K2UZGTZ)Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1)\.
- Lugoloobiet al\.\(2026\)W\. Lugoloobi, T\. Foster, W\. Bankes, and C\. RussellLLMs encode their failures: predicting success from pre\-generation activations\.InConference on Language Modeling,Cited by:[§D\.2](https://arxiv.org/html/2610.33237#A4.SS2.SSS0.Px2.p3.1.3.1),[Appendix E](https://arxiv.org/html/2610.33237#A5.p1.2),[Appendix E](https://arxiv.org/html/2610.33237#A5.p2.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1)\.
- Machchaet al\.\(2026\)S\. Machcha, S\. Yerra, S\. Gupta, A\. Sahoo, S\. Sultana, H\. Yu, and Z\. YaoKnowing when to abstain: medical LLMs under clinical uncertainty\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 6153–6182\.External Links:[Link](https://aclanthology.org/2026.eacl-long.291/),[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.291),ISBN 979\-8\-89176\-380\-7Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px2.p1.1)\.
- Platt \(1999\)J\. PlattProbabilistic outputs for support vector machines and comparisons to regularized likelihood methods\.InAdvances in Large Margin Classifiers,pp\. 61–74\.Cited by:[§4](https://arxiv.org/html/2610.33237#S4.SS0.SSS0.Px3.p1.1)\.
- Rajpurkaret al\.\(2018\)P\. Rajpurkar, R\. Jia, and P\. LiangKnow what you don’t know: unanswerable questions for SQuAD\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),I\. Gurevych and Y\. Miyao \(Eds\.\),Melbourne, Australia,pp\. 784–789\.External Links:[Link](https://aclanthology.org/P18-2124/),[Document](https://dx.doi.org/10.18653/v1/P18-2124)Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px2.p1.1)\.
- Schmidgallet al\.\(2026\)S\. Schmidgall, R\. Ziaei, C\. Harris, J\. W\. Kim, E\. P\. Reis, J\. Jopling, and M\. MoorAgentClinic: a multimodal benchmark for tool\-using clinical AI agents\.npj Digit\. Medicine9\(1\),pp\. 499\.External Links:[Document](https://dx.doi.org/10.1038/s41746-026-02674-7)Cited by:[§A\.1](https://arxiv.org/html/2610.33237#A1.SS1.SSS0.Px1.p1.1),[§A\.1](https://arxiv.org/html/2610.33237#A1.SS1.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2610.33237#S4.SS0.SSS0.Px1.p1.1)\.
- Slobodkinet al\.\(2023\)A\. Slobodkin, O\. Goldman, A\. Caciularu, I\. Dagan, and S\. RavfogelThe curious case of hallucinatory \(un\)answerability: finding truths in the hidden states of over\-confident large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 3607–3625\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.220/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.220)Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px2.p1.1)\.
- Tianet al\.\(2023\)K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. ManningJust ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5433–5442\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.330/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.330)Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2610.33237#S4.SS0.SSS0.Px3.p1.1)\.
- Trivediet al\.\(2022\)H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. SabharwalMuSiQue: multihop questions via single\-hop question composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.External Links:[Link](https://aclanthology.org/2022.tacl-1.31/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00475)Cited by:[§A\.1](https://arxiv.org/html/2610.33237#A1.SS1.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.33237#S1.p7.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2610.33237#S4.SS0.SSS0.Px1.p1.1)\.
- Vashurinet al\.\(2025\)R\. Vashurin, E\. Fadeeva, A\. Vazhentsev, L\. Rvanova, D\. Vasilev, A\. Tsvigun, S\. Petrakov, R\. Xing, A\. Sadallah, K\. Grishchenkov, A\. Panchenko, T\. Baldwin, P\. Nakov, M\. Panov, and A\. ShelmanovBenchmarking uncertainty quantification methods for large language models with LM\-polygraph\.Transactions of the Association for Computational Linguistics13,pp\. 220–248\.External Links:[Link](https://aclanthology.org/2025.tacl-1.11/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00737)Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1)\.
- Wenet al\.\(2025\)B\. Wen, J\. Yao, S\. Feng, C\. Xu, Y\. Tsvetkov, B\. Howe, and L\. L\. WangKnow your limits: a survey of abstention in large language models\.Transactions of the Association for Computational Linguistics13,pp\. 529–556\.External Links:[Link](https://aclanthology.org/2025.tacl-1.26/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00754)Cited by:[§1](https://arxiv.org/html/2610.33237#S1.p2.1)\.
- Xionget al\.\(2024\)M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. HooiCan llms express their uncertainty? an empirical evaluation of confidence elicitation in llms\.InICLR,Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1)\.
- Yinet al\.\(2023\)Z\. Yin, Q\. Sun, Q\. Guo, J\. Wu, X\. Qiu, and X\. HuangDo large language models know what they don’t know?\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 8653–8665\.External Links:[Link](https://aclanthology.org/2023.findings-acl.551/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.551)Cited by:[§1](https://arxiv.org/html/2610.33237#S1.p2.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px2.p1.1)\.
- Zhaiet al\.\(2026\)H\. Zhai, J\. Liang, and D\. KangAbstain\-r1: calibrated abstention and post\-refusal clarification via verifiable RL\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 19674–19695\.External Links:[Link](https://aclanthology.org/2026.findings-acl.985/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.985),ISBN 979\-8\-89176\-395\-1Cited by:[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026\)C\. Zhang, R\. Yang, X\. Zhu, C\. Li, T\. Hu, Y\. R\. Dong, D\. Yang, and N\. CollierConfidence estimation for LLMs in multi\-turn interactions\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 25661–25676\.External Links:[Link](https://aclanthology.org/2026.findings-acl.1280/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1280),ISBN 979\-8\-89176\-395\-1Cited by:[§D\.2](https://arxiv.org/html/2610.33237#A4.SS2.SSS0.Px2.p3.1.7.1),[Appendix E](https://arxiv.org/html/2610.33237#A5.p1.2),[Appendix E](https://arxiv.org/html/2610.33237#A5.p2.1),[§1](https://arxiv.org/html/2610.33237#S1.p5.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px3.p1.1)\.
- Zhang and Choi \(2025\)M\. J\. Zhang and E\. ChoiClarify when necessary: resolving ambiguity through interaction with LMs\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 5541–5558\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.306/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.306),ISBN 979\-8\-89176\-195\-7Cited by:[§1](https://arxiv.org/html/2610.33237#S1.p3.1),[§2](https://arxiv.org/html/2610.33237#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix Contents

## Appendix AExperimental Details

### A\.1Datasets

The three datasets range widely in difficulty \(Table[4](https://arxiv.org/html/2610.33237#A1.T4)\)\. For Qwen3\.5\-9B, the mean success rate on the test set is 0\.81 on MIMIC, 0\.50 on MedQA, and 0\.25 on MuSiQue, so we test the confidence both where most states are answered correctly and where most are not\.

Table 4:Dataset statistics, given as train / test\. The mean success ratep^\\hat\{p\}of each answering model is taken over all evidence states\.#### MIMIC\.

We use emergency admissions from MIMIC\-IV\-Ext Clinical Decision Making\([Hager et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib32)\)\. We sample 50 admissions for each of four abdominal diseases, appendicitis, cholecystitis, diverticulitis, and pancreatitis, and the task is to choose one of the four\. We convert each admission to the OSCE layout of AgentClinic\([Schmidgall et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib31)\)\. The history, laboratory results, and radiology reports are copied verbatim, and an LLM \(GPT\-5\.1\) writes only a one\-sentence objective naming the presenting complaints and splits the free\-text physical examination into sections without changing its values\. Microbiology results are never shown, since cultures return after the diagnostic decision\. We split admissions into training and test sets stratified by disease\.

#### MedQA\.

We use the OSCE\-style cases of the extended AgentClinic\-MedQA set\([Schmidgall et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib31)\), which are derived from USMLE questions\([Jin et al\., 2021](https://arxiv.org/html/2610.33237#bib.bib33)\), and the task is to name the diagnosis in free text\. We exclude 5 of its 214 cases, four with no test results and one with a nonstandard format, which leaves 209\. States that show only imaging or only the other tests exist for the 97 cases that have both kinds of test\. We split cases by diagnosis, which keeps near\-duplicate vignettes on the same side and makes every test diagnosis unseen during training\.

#### MuSiQue\.

We use 3\-hop questions from MuSiQue\-Ans\([Trivedi et al\., 2022](https://arxiv.org/html/2610.33237#bib.bib34)\), and the task is to give a short answer\. For each question, we sample three distractors from its non\-supporting paragraphs\. Its three supporting paragraphs and three distractors are shuffled once with a seed derived from the question, and every state shows its paragraphs in that order\. Training questions come from the MuSiQue training set and test questions from its development set, since its test set has no released answers, with no single\-hop question shared between them\. MuSiQue’s own decompositions distinguish two question types \(Table[5](https://arxiv.org/html/2610.33237#A1.T5)\)\. In a chain question each hop needs the answer of the previous one, whereas in a branching question the first two hops are independent and only the final hop combines them\. A branching question therefore has a state withe2e\_\{2\}alone, while in a chaine2e\_\{2\}is only useful aftere1e\_\{1\}\. The full inputs of the branching example appear in Table[16](https://arxiv.org/html/2610.33237#A6.T16)\.

Table 5:The two MuSiQue question types, with sub\-questions from MuSiQue’s decompositions \(capitalization normalized\)\.\#kkstands for the answer of hopkk, supported by paragrapheke\_\{k\}\. In the chain, each hop depends on the previous one, so each stage has its own color\. In the branching question,e1e\_\{1\}ande2e\_\{2\}are independent and share a color, ande3e\_\{3\}combines their answers\.

### A\.2Answering Policies

The main experiments study self\-calibration\. Each model in Table[1](https://arxiv.org/html/2610.33237#S4.T1)serves as both the answering policy and the estimator, so it estimates its own chance of success, and every method for that model is compared against the exact same targets\.

#### Targets for self\-calibration\.

For each model, the frozen model with thinking disabled samplesK=10K=10answers per evidence state, and the fraction graded correct is the success rate of that state\. These success rates are the only targets used for that model\. The base logit readsP⁡\(A∣x\)P\(A\\mid x\)from this model before training, and verbalized confidence prompts this model for the probability that it could answer correctly\. Platt scaling fits its two parameters to this model’s success rates on the training set, and SuffiCE trains a LoRA adapter on a copy of this model to predict the same success rates\. All four are evaluated against the success rates of this model on the test set\. Methods are therefore compared only for the same model, since each model has its own targets\. The cross\-policy experiment of §[5\.4](https://arxiv.org/html/2610.33237#S5.SS4)is the only exception, where Qwen estimators are trained on and evaluated against the success rates of Kimi K2\.6\.

#### Difficulty\.

Success rates differ across answering models, little on MIMIC but widely on MedQA and MuSiQue \(Table[4](https://arxiv.org/html/2610.33237#A1.T4)\), so the same evidence state has a different target for each answering policy\.

#### Precision of the targets\.

Each target is estimated fromK=10K=10answers, so even a perfect estimator that outputs the true success ratep∗p^\{\*\}has a nonzero Brier score against it\. This*noise floor*is the expected binomial sampling variancep∗​\(1−p∗\)/Kp^\{\*\}\(1\-p^\{\*\}\)/K, which we estimate without bias byp^​\(1−p^\)/\(K−1\)\\hat\{p\}\(1\-\\hat\{p\}\)/\(K\-1\)averaged over the test set\. On every dataset, the Brier score of SuffiCE is over 13 times this floor \(Table[6](https://arxiv.org/html/2610.33237#A1.T6)\)\. Sampling noise therefore accounts for only a small part of any method’s error, and ten answers per state are enough to compare estimators\.

Table 6:Noise floor, the expected Brier score of a perfect estimator against ten\-answer success rates, compared with the Brier score of SuffiCE \(Table[1](https://arxiv.org/html/2610.33237#S4.T1)\)\. Both are averaged over the four answering models, on the test set\.

### A\.3Training Details

The settings shared by all models and datasets are listed in Table[7](https://arxiv.org/html/2610.33237#A1.T7)\. All models are run with thinking disabled\. Answers are sampled at temperature 1\.0 without top\-ppor top\-kktruncation\. The estimator is a rank\-32 LoRA adapter\([Hu et al\., 2022](https://arxiv.org/html/2610.33237#bib.bib35)\)on a copy of the estimator’s base model, which is the answering model itself except in the cross\-policy experiment \(§[5\.4](https://arxiv.org/html/2610.33237#S5.SS4)\), initialized to have no effect, so before training it gives the base logit\. All training and sampling run on Tinker\.

#### Grading\.

Each sampled answer is graded as correct or incorrect against the reference\. On MIMIC, an answer is correct if, after lowercasing and removing everything but letters, it is exactly the name of the correct disease among the four options, and any other answer is incorrect\. On MedQA, GPT\-5\.6 Luna judges whether a physician would accept the answer as the same diagnosis as the reference\. On MuSiQue, an answer is correct if it matches the reference or one of its aliases after SQuAD normalization \(lowercasing and removing punctuation and articles\), and otherwise GPT\-5\.6 Luna judges it given the question and the reference\. Both judges decode greedily, and their prompts are in Appendix[B\.4](https://arxiv.org/html/2610.33237#A2.SS4)\.

Table 7:Settings shared by all models and datasets\.

## Appendix BPrompts

We list every prompt verbatim, grouped by role\. Placeholders are in braces\.\{question\}is the MuSiQue question or the clinical case’s objective,\{context\}and\{case\}are the evidence shown in the current state, and\{task line\}is the dataset\-specific line of Table[8](https://arxiv.org/html/2610.33237#A2.T8)\. All prompts are given as a single user message with thinking disabled, except the judges, which also receive a system message\.

Table 8:The dataset\-specific task line of the sufficiency and verbalized prompts\.### B\.1Sufficiency Confidence

The sufficiency confidence estimator and the base logit read the probabilities ofAandSas the first token of the reply \(§[3\.4](https://arxiv.org/html/2610.33237#S3.SS4)\)\.

Decidewhetherthecurrentlyavailableinformationissufficientforyoutoanswerthetaskcorrectly\.Donotanswerthetaskitself\.

A=theinformationissufficient;answernow\.

S=theinformationisinsufficient;seekmoreinformation\.

Task:

\{question\}

\{taskline\}

Availableinformation:

\{context\}

Istheavailableinformationsufficient?Respondwithexactlyoneletter:AorS\.

### B\.2Answering

The frozen answering policy samples ten answers per state with these prompts, from which the empirical success rate is computed \(§[3\.3](https://arxiv.org/html/2610.33237#S3.SS3)\)\.

\{case\}

Thisisalltheinformationavailableaboutthepatient\.

Basedonthisinformation,whichofthefollowingisthemostlikelydiagnosis?

appendicitis

cholecystitis

diverticulitis

pancreatitis

Respondwithonlyoneofthesefourwords,nothingelse\.

\{case\}

Thisisalltheinformationavailableaboutthepatient\.

Basedonthisinformation,whatisthemostlikelydiagnosis?

Respondwithonlythediagnosis,nothingelse\.

Answerthequestionusingtheprovidedinformationandyourownknowledge\.

Ifyouareunsure,giveyourbestguess\.

Giveonlythefinalanswer,withnoexplanation,inatmost100characters\.

\{case\}

Answer:

### B\.3Verbalized Baseline

The first number in the reply is taken as the confidence, with “85%” read as 0\.85\.

Basedonthecurrentlyavailableinformation,estimatetheprobabilitythatyoucouldanswerthetaskcorrectlywithoutseekinganyadditionalinformation\.

Donotanswerthetaskitself\.

Respondwithasinglenumberbetween0and1,where0meansthecurrentinformationiscertainlyinsufficientforacorrectanswerand1meansitiscertainlysufficient\.

Task:

\{question\}

\{taskline\}

Availableinformation:

\{context\}

Probability:

### B\.4LLM Judges

MedQA answers are graded by the judge below\. MuSiQue answers that are not a normalized exact match with the reference or one of its aliases are graded by the MuSiQue judge\. MIMIC answers are graded by exact match\. All methods for a model are evaluated against the same grades, so differences between methods do not come from differences in grading\.

Youareanexpertphysiciangradingadiagnosticansweragainstareferencediagnosis\.

Firstexplainyourreasoningbriefly,thengiveagradeof1\(correct\)or0\(incorrect\)\.

Correctdiagnosis:\{correct\_diagnosis\}

Doctor’sdiagnosis:\{predicted\_diagnosis\}

Grade1ifaphysicianwouldacceptthedoctor’sdiagnosisasthesamediagnosisasthecorrectone\.Acceptsynonyms,abbreviations,alternatespellings,historicalormodernterminology,anddiagnosesatanequivalentlevelofspecificity\.Amorespecificformofthecorrectdiagnosisisalsoacceptable\.

Grade0ifthedoctor’sdiagnosis:

\-isadifferentcondition,evenacloselyrelatedone,

\-isonlyabroaderdiseasecategorythatcontainsthecorrectdiagnosisbutdoesnotidentifythemorespecificdiagnosisthecorrectonerequires,

\-listsseveralpossibilitieswithoutcommittingtothecorrectone,or

\-ismissing\.

Respondinexactlythisformat:

<reasoning\>Atmostthreesentencescomparingthetwodiagnoses\.</reasoning\>

<grade\>1or0</grade\>

Youaregradingshortanswerstomulti\-hopquestionsagainstareferenceanswer\.

Firstexplainyourreasoningbriefly,thengiveagradeof1\(correct\)or0\(incorrect\)\.

Question:\{question\}

Referenceanswer:\{answer\}

Modelanswer:\{predicted\}

Grade1onlyiftheresponseclearlycommitstoananswerthatissemanticallycorrectaccordingtothereference\.Grade0ifitgivesmultiplealternativeanswerswithoutclearlycommittingtothecorrectone,orifitgivesawronganswertothequestionalongsidethecorrectone\.Ignoreextradetailsthatdonotanswerthequestion,eveniftheyareinaccurate\.Differencesinwordingorformattingareacceptablewhentheypreservethesameunderlyinganswer\.Abroaderplace,category,orentity,orarelated\-but\-differentevent,isnotequivalenttothereference\.Datesaretheonlyexception:differencesindateprecisionareacceptablewhentheypreservethecorrecttimeperiod,soanexactdate,month/year,year,ordecademaybetreatedasequivalentwhenconsistentwiththereference\.Extremelybroadperiodssuchascenturiesarenotsufficientsubstitutesforaspecificyearordate\.

Respondinexactlythisformat:

<reasoning\>Atmosttwosentences\.</reasoning\>

<grade\>1or0</grade\>

## Appendix CAdditional Results

### C\.1Calibration Curves for All Models

We extend the calibration curves of Figure[4](https://arxiv.org/html/2610.33237#S5.F4)to all four answering models \(Figure[9](https://arxiv.org/html/2610.33237#A3.F9)\)\.

Figure 9:Calibration curves of Qwen3\.5\-4B, Qwen3\.5\-9B, Qwen3\.8\-27B, and Kimi K2\.6 on the test set, \(a\) before and \(b\) after SuffiCE\.#### The base confidence errs in no consistent direction\.

Under the same prompt, Qwen3\.5\-9B is overconfident on every dataset, Qwen3\.8\-27B and Kimi K2\.6 are underconfident on every dataset, and Qwen3\.5\-4B is overconfident on MedQA but underconfident on MIMIC and MuSiQue\. No single correction therefore repairs the base confidence across models and datasets\. Many base curves are also not monotone, so a state with higher confidence is not reliably more answerable\. Platt scaling can shift such a curve toward the diagonal but cannot straighten it, since it keeps the order of states\.

#### SuffiCE moves every curve toward the diagonal\.

After training, every Qwen curve lies closer to the diagonal\. The confidence also adapts to the range of each dataset, spanning low values on MuSiQue, where most states cannot be answered, and concentrating near the top on MIMIC, where most are answered correctly\. The same prompt and objective produce both, so the estimator learns the scale of each dataset from its targets rather than imposing a fixed one\.

#### SuffiCE carries over to another model family\.

Before training, Kimi K2\.6 is strongly underconfident on every dataset\. SuffiCE moves its curve toward the diagonal on each\.

### C\.2Selective Answering for All Models

Across all four answering models and three datasets, SuffiCE has the highest accuracy at most percentages of questions answered, with the smallest margin for Qwen3\.5\-4B on the clinical datasets \(Figure[10](https://arxiv.org/html/2610.33237#A3.F10), extending Table[2](https://arxiv.org/html/2610.33237#S5.T2)\)\.

Figure 10:Selective answering on the test set of every dataset \(rows\) and model \(columns\)\. Each curve plots the accuracy of the answered questions against the percentage answered, answering the questions with the highest confidence first\. Higher is better\. Platt scaling preserves the ranking of the base logit, so their curves coincide\.Table 9:Step\-matched ablation for Qwen3\.5\-9B\. The full\-only estimator is trained for the same number of steps as SuffiCE\.Table 10:Evidence\-state analysis for every answering model\.Δ\\DeltaMAE compares changes in confidence with changes in empirical success between states of the same problem, andρwithin\\rho\_\{\\text\{within\}\}correlates confidence with success among problems that share a state type\. The best result for each model is in bold\.
### C\.3Step\-Matched Ablation

The ablation of §[5\.2](https://arxiv.org/html/2610.33237#S5.SS2)trains the full\-only estimator for the same number of epochs as SuffiCE\. Since it has one state per problem instead of four to eight, it then takes far fewer gradient steps\. Here we instead train it for the same number of steps as SuffiCE, with all other hyperparameters unchanged \(Table[9](https://arxiv.org/html/2610.33237#A3.T9)\)\. SuffiCE remains better on every metric and dataset, so the gap comes from training across evidence states, not from the number of updates\.

Table 11:Cross\-domain transfer between the clinical datasets \(Qwen3\.5\-9B\)\. Cross\-domain and in\-domain are SuffiCE trained on the other dataset and on the test dataset itself\. The best result in each row is in bold and the second best in italics\.
### C\.4Additional Evidence\-State Analysis

#### Ordering of pairs inρΔ\\rho\_\{\\Delta\}\.

Each pair of states enters once, ordered from less to more evidence\. The correlation subtracts the average change, so confidence that rises equally whenever evidence is added gets no credit, andρΔ\\rho\_\{\\Delta\}rewards confidence that rises more for the evidence that raises success more\.

We report two measures of evidence sensitivity beyondρΔ\\rho\_\{\\Delta\}for every answering model \(Table[10](https://arxiv.org/html/2610.33237#A3.T10)\)\.

- •Δ\\DeltaMAE\(lower is better\) tests whether confidence changes by the right amount as the evidence changes\. For every pair of distinct statess,ts,tof the same problemii, the same pairs asρΔ\\rho\_\{\\Delta\}, it takes the absolute gap between the change in confidence and the change in empirical success,\|\(ci,t−ci,s\)−\(p^i,t−p^i,s\)\|\\bigl\|\(c\_\{i,t\}\-c\_\{i,s\}\)\-\(\\hat\{p\}\_\{i,t\}\-\\hat\{p\}\_\{i,s\}\)\\bigr\|, and averages over all pairs in the test set\. It is zero when confidence rises and falls by exactly as much as success\. WhereρΔ\\rho\_\{\\Delta\}tests the direction of each change,Δ\\DeltaMAE tests its size\.
- •ρwithin\\rho\_\{\\text\{within\}\}\(higher is better\) tests whether confidence separates more and less answerable problems that have the same kind of evidence\. It groups the test set by state type, such as allHistorystates \(on MuSiQue, by question type and state\), computes the Pearson correlation between confidence and empirical success within each group, and averages over groups\. Every state in a group holds the same kind of evidence, so recognizing the state type alone does not raise it\.

SuffiCE has the lowestΔ\\DeltaMAE in 11 of the 12 combinations of model and dataset, so its confidence changes by about as much as success does, not only in the same direction\. The one exception is Kimi K2\.6 on MIMIC, where verbalized confidence is slightly lower\. For Kimi K2\.6, success on MIMIC changes by at most 0\.1 for about three quarters of the pairs of states, so a confidence that barely moves has a lowΔ\\DeltaMAE, and verbalized confidence moves about half as much as SuffiCE\. OnρΔ\\rho\_\{\\Delta\}, which gives no credit for staying still, SuffiCE is far better \(Table[1](https://arxiv.org/html/2610.33237#S4.T1)\)\. SuffiCE also has the highestρwithin\\rho\_\{\\text\{within\}\}for every model and dataset, with the largest margin on MIMIC, where whether the evidence suffices depends on the particular patient, so it does not merely learn an average confidence for each state type\.

### C\.5Cross\-Domain Transfer

To test whether what SuffiCE learns is specific to one dataset, we train it on one clinical dataset and evaluate it on the other without further training \(Table[11](https://arxiv.org/html/2610.33237#A3.T11)\)\. MIMIC asks for one of four diseases and MedQA for a free\-text diagnosis, and success is far more common on MIMIC\.

#### Which evidence matters transfers\.

In both directions, the transferred estimator ranks states better than the base logit in AUROC andρΔ\\rho\_\{\\Delta\}\. It also better separates more and less answerable cases that share a state type, recovering about half of the in\-domain gain inρwithin\\rho\_\{\\text\{within\}\}\(§[C\.4](https://arxiv.org/html/2610.33237#A3.SS4)\) over the base logit in both directions\. What SuffiCE learns about which evidence settles a case thus carries over across datasets\.

#### The overall success level does not transfer\.

The transferred confidence stays close to the success rate of the training dataset\. Trained on MIMIC, where most states succeed, it remains too confident on MedQA, though less so than the base logit\. Trained on MedQA, it is too cautious on MIMIC and is worse calibrated than the base logit\. This error shifts the level of confidence, not the order of states, so ranking transfers while calibration does not\.

## Appendix DProofs

Proposition[1](https://arxiv.org/html/2610.33237#Thmproposition1)says that the error of a confidence splits into two parts: how far off it is on average for each problem, and how far its changes between states of the same problem are from the changes in success\. Calibration constrains only the first part\. Corollary[1](https://arxiv.org/html/2610.33237#Thmcorollary1)shows that the training loss bounds both parts\. We first define the notation shared by both results, then restate and prove Proposition[1](https://arxiv.org/html/2610.33237#Thmproposition1)\(§[D\.1](https://arxiv.org/html/2610.33237#A4.SS1)\) and Corollary[1](https://arxiv.org/html/2610.33237#Thmcorollary1)\(§[D\.2](https://arxiv.org/html/2610.33237#A4.SS2)\)\.

#### Setup\.

Let there bennproblems, indexed byi=1,…,ni=1,\\ldots,n\. Problemiiconsists of a questionqiq\_\{i\}, a pool of evidenceEiE\_\{i\}, and a gold answeryi∗y^\{\*\}\_\{i\}\(§[3\.1](https://arxiv.org/html/2610.33237#S3.SS1)\)\. It hasmi≥1m\_\{i\}\\geq 1evidence states, indexed bys=1,…,mis=1,\\ldots,m\_\{i\}\. Statesspairs the question with a subsetIi,s⊆EiI\_\{i,s\}\\subseteq E\_\{i\}of the evidence,xi,s=\(qi,Ii,s\)x\_\{i,s\}=\(q\_\{i\},I\_\{i,s\}\)\. LetN=∑i=1nmiN=\\sum\_\{i=1\}^\{n\}m\_\{i\}be the total number of states\. For a statexx, letp∗​\(x\)∈\[0,1\]p^\{\*\}\(x\)\\in\[0,1\]be the success probability of the answering policyπ0\\pi\_\{0\}\(Eq\. \([1](https://arxiv.org/html/2610.33237#S3.E1)\)\) andcθ​\(x\)∈\[0,1\]c\_\{\\theta\}\(x\)\\in\[0,1\]the confidence of the estimator\.

Within problemii, we abbreviatecs=cθ​\(xi,s\)c\_\{s\}=c\_\{\\theta\}\(x\_\{i,s\}\)andps=p∗​\(xi,s\)p\_\{s\}=p^\{\*\}\(x\_\{i,s\}\), and define the error at statessasϵs=cs−ps\\epsilon\_\{s\}=c\_\{s\}\-p\_\{s\}\. The problem averages of the confidence, the success probability, and the error are

c¯θ,i=1mi​∑s=1mics,p¯i∗=1mi​∑s=1mips,\\displaystyle\\bar\{c\}\_\{\\theta,i\}=\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s=1\}^\{m\_\{i\}\}c\_\{s\},\\qquad\\bar\{p\}^\{\*\}\_\{i\}=\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s=1\}^\{m\_\{i\}\}p\_\{s\},ϵ¯=c¯θ,i−p¯i∗\.\\displaystyle\\bar\{\\epsilon\}=\\bar\{c\}\_\{\\theta,i\}\-\\bar\{p\}^\{\*\}\_\{i\}\.For anys,t∈\{1,…,mi\}s,t\\in\\\{1,\\ldots,m\_\{i\}\\\}, the change gap

δs​t=\(ct−cs\)−\(pt−ps\)\\delta\_\{st\}=\(c\_\{t\}\-c\_\{s\}\)\-\(p\_\{t\}\-p\_\{s\}\)is the difference between the change in confidence and the change in success probability from statessto statett\. Finally,𝔼\\mathbb\{E\}denotes the uniform average over allNNstates, and𝔼i\\mathbb\{E\}\_\{i\}the average of a per\-problem quantity over problems, with problemiiweighted bymi/Nm\_\{i\}/N, its share of all states\. With this weight, averaging any functionffof a state over all states equals first averaging it within each problem and then applying𝔼i\\mathbb\{E\}\_\{i\},

𝔼⁡\[f\]=1N​∑i=1n∑s=1mif⁡\(xi,s\)=∑i=1nmiN⋅1mi∑s=1mif\(xi,s\)=∑i=1nmiN​f¯i=𝔼i​\[f¯i\],\\begin\{split\}\\mathbb\{E\}\[f\]&=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{n\}\\sum\_\{s=1\}^\{m\_\{i\}\}f\(x\_\{i,s\}\)\\\\ &=\\sum\_\{i=1\}^\{n\}\\frac\{m\_\{i\}\}\{N\}\\cdot\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s=1\}^\{m\_\{i\}\}f\(x\_\{i,s\}\)\\\\ &=\\sum\_\{i=1\}^\{n\}\\frac\{m\_\{i\}\}\{N\}\\,\\bar\{f\}\_\{i\}=\\mathbb\{E\}\_\{i\}\\bigl\[\\bar\{f\}\_\{i\}\\bigr\],\\end\{split\}\(8\)wheref¯i=1mi​∑sf⁡\(xi,s\)\\bar\{f\}\_\{i\}=\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s\}f\(x\_\{i,s\}\)is the average offfover the states of problemii\. A plain average over problems,1n​∑if¯i\\frac\{1\}\{n\}\\sum\_\{i\}\\bar\{f\}\_\{i\}, agrees with𝔼⁡\[f\]\\mathbb\{E\}\[f\]only when all problems have the same number of states\.

### D\.1Proof of Proposition[1](https://arxiv.org/html/2610.33237#Thmproposition1)

#### Proposition[1](https://arxiv.org/html/2610.33237#Thmproposition1)\(restated\)\.

For any confidencecθc\_\{\\theta\},

𝔼⁡\[\(cθ−p∗\)2\]=𝔼i​\[\(c¯θ,i−p¯i∗\)2\]\+𝔼i​\[12​mi2​∑s=1mi∑t=1miδs​t2\]\.\\begin\{split\}\\mathbb\{E\}\\bigl\[\(c\_\{\\theta\}\-p^\{\*\}\)^\{2\}\\bigr\]&=\\mathbb\{E\}\_\{i\}\\bigl\[\(\\bar\{c\}\_\{\\theta,i\}\-\\bar\{p\}^\{\*\}\_\{i\}\)^\{2\}\\bigr\]\\\\ &\\quad\+\\mathbb\{E\}\_\{i\}\\Bigl\[\\tfrac\{1\}\{2m\_\{i\}^\{2\}\}\\textstyle\\sum\_\{s=1\}^\{m\_\{i\}\}\\sum\_\{t=1\}^\{m\_\{i\}\}\\delta\_\{st\}^\{2\}\\Bigr\]\.\\end\{split\}The left side is the mean squared error over all states\. The first term on the right is the error across problems, in each problem’s average success\. The second is the error within each problem, half the average squared change gap over all ordered pairs of its states\.

#### Proof\.

We prove the identity for a single problemiiand then average over problems\. All sums overssandttrun from11tomim\_\{i\}\.

Step 1: split the error into its average and its spread\.Writingϵs=ϵ¯\+\(ϵs−ϵ¯\)\\epsilon\_\{s\}=\\bar\{\\epsilon\}\+\(\\epsilon\_\{s\}\-\\bar\{\\epsilon\}\)and expanding the square,

1mi​∑sϵs2=ϵ¯2\+2​ϵ¯mi​∑s\(ϵs−ϵ¯\)\+1mi∑s\(ϵs−ϵ¯\)2\.\\begin\{split\}\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s\}\\epsilon\_\{s\}^\{2\}&=\\bar\{\\epsilon\}^\{\\,2\}\+\\frac\{2\\bar\{\\epsilon\}\}\{m\_\{i\}\}\\sum\_\{s\}\(\\epsilon\_\{s\}\-\\bar\{\\epsilon\}\)\\\\ &\\quad\+\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s\}\(\\epsilon\_\{s\}\-\\bar\{\\epsilon\}\)^\{2\}\.\\end\{split\}The middle term is zero\. Sinceϵ¯=c¯θ,i−p¯i∗=1mi​∑sϵs\\bar\{\\epsilon\}=\\bar\{c\}\_\{\\theta,i\}\-\\bar\{p\}^\{\*\}\_\{i\}=\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s\}\\epsilon\_\{s\}is the average of the errors, the errors above the average exactly cancel those below it,

∑s\(ϵs−ϵ¯\)=∑sϵs−mi​ϵ¯=mi​ϵ¯−mi​ϵ¯=0\.\\sum\_\{s\}\(\\epsilon\_\{s\}\-\\bar\{\\epsilon\}\)=\\sum\_\{s\}\\epsilon\_\{s\}\-m\_\{i\}\\bar\{\\epsilon\}=m\_\{i\}\\bar\{\\epsilon\}\-m\_\{i\}\\bar\{\\epsilon\}=0\.
Step 2: rewrite the spread with pairs of states\.The spread is the last term of Step 1,1mi​∑s\(ϵs−ϵ¯\)2\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s\}\(\\epsilon\_\{s\}\-\\bar\{\\epsilon\}\)^\{2\}, the average squared deviation of the errors from their mean, that is, their variance over the states of problemii\. It is zero exactly when every state of the problem has the same error\. We now show that it equals half the average squared difference between the errors of two states\. Letds=ϵs−ϵ¯d\_\{s\}=\\epsilon\_\{s\}\-\\bar\{\\epsilon\}, so∑sds=0\\sum\_\{s\}d\_\{s\}=0by Step 1\. Expanding\(dt−ds\)2=dt2\+ds2−2​ds​dt\(d\_\{t\}\-d\_\{s\}\)^\{2\}=d\_\{t\}^\{2\}\+d\_\{s\}^\{2\}\-2d\_\{s\}d\_\{t\}and summing over allssandtt,

∑s,t\(ϵt−ϵs\)2=∑s,t\(dt−ds\)2=2​mi​∑sds2−2​\(∑sds\)2=2​mi​∑sds2=2​mi​∑s\(ϵs−ϵ¯\)2\.\\begin\{split\}\\sum\_\{s,t\}\(\\epsilon\_\{t\}\-\\epsilon\_\{s\}\)^\{2\}&=\\sum\_\{s,t\}\(d\_\{t\}\-d\_\{s\}\)^\{2\}\\\\ &=2m\_\{i\}\\sum\_\{s\}d\_\{s\}^\{2\}\-2\\Bigl\(\\sum\_\{s\}d\_\{s\}\\Bigr\)^\{2\}\\\\ &=2m\_\{i\}\\sum\_\{s\}d\_\{s\}^\{2\}\\\\ &=2m\_\{i\}\\sum\_\{s\}\(\\epsilon\_\{s\}\-\\bar\{\\epsilon\}\)^\{2\}\.\\end\{split\}Dividing by2​mi22m\_\{i\}^\{2\}gives1mi​∑s\(ϵs−ϵ¯\)2=12​mi2​∑s,t\(ϵt−ϵs\)2\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s\}\(\\epsilon\_\{s\}\-\\bar\{\\epsilon\}\)^\{2\}=\\frac\{1\}\{2m\_\{i\}^\{2\}\}\\sum\_\{s,t\}\(\\epsilon\_\{t\}\-\\epsilon\_\{s\}\)^\{2\}\.

Step 3: identify the change gap\.Rearranging,ϵt−ϵs=\(ct−pt\)−\(cs−ps\)=\(ct−cs\)−\(pt−ps\)=δs​t\\epsilon\_\{t\}\-\\epsilon\_\{s\}=\(c\_\{t\}\-p\_\{t\}\)\-\(c\_\{s\}\-p\_\{s\}\)=\(c\_\{t\}\-c\_\{s\}\)\-\(p\_\{t\}\-p\_\{s\}\)=\\delta\_\{st\}\. Combining the three steps, for every problemii,

1mi​∑sϵs2=Step 1​ϵ¯2\+1mi​∑s\(ϵs−ϵ¯\)2=Step 2​ϵ¯2\+12​mi2​∑s,t\(ϵt−ϵs\)2=Step 3​\(c¯θ,i−p¯i∗\)2\+12​mi2​∑s,tδs​t2,\\begin\{split\}\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s\}\\epsilon\_\{s\}^\{2\}&\\;\\overset\{\\text\{Step 1\}\}\{=\}\\;\\bar\{\\epsilon\}^\{\\,2\}\+\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s\}\(\\epsilon\_\{s\}\-\\bar\{\\epsilon\}\)^\{2\}\\\\ &\\;\\overset\{\\text\{Step 2\}\}\{=\}\\;\\bar\{\\epsilon\}^\{\\,2\}\+\\frac\{1\}\{2m\_\{i\}^\{2\}\}\\sum\_\{s,t\}\(\\epsilon\_\{t\}\-\\epsilon\_\{s\}\)^\{2\}\\\\ &\\;\\overset\{\\text\{Step 3\}\}\{=\}\\;\(\\bar\{c\}\_\{\\theta,i\}\-\\bar\{p\}^\{\*\}\_\{i\}\)^\{2\}\+\\frac\{1\}\{2m\_\{i\}^\{2\}\}\\sum\_\{s,t\}\\delta\_\{st\}^\{2\},\\end\{split\}where the last line substitutesϵ¯=c¯θ,i−p¯i∗\\bar\{\\epsilon\}=\\bar\{c\}\_\{\\theta,i\}\-\\bar\{p\}^\{\*\}\_\{i\}andϵt−ϵs=δs​t\\epsilon\_\{t\}\-\\epsilon\_\{s\}=\\delta\_\{st\}\. The left side is the mean squared error of problemii\. On the right, the first term is the squared error of the problem’s average confidence, and the second is half the average squared change gap over its ordered pairs of states\.

Step 4: average over problems\.Steps 1–3 give the identity for one problem\. The proposition averages over all states, so we group the states by problem and weight each problem by its share of states, using Eq\. \([8](https://arxiv.org/html/2610.33237#A4.E8)\) withf=\(cθ−p∗\)2f=\(c\_\{\\theta\}\-p^\{\*\}\)^\{2\}\. Substituting the per\-problem identity for each problem’s mean squared error and splitting the sum,

𝔼⁡\[\(cθ−p∗\)2\]=∑imiN​\[1mi​∑sϵs2\]=∑imiN​\[\(c¯θ,i−p¯i∗\)2\+12​mi2​∑s,tδs​t2\]=∑imiN\(c¯θ,i−p¯∗i\)2\+∑imiN⋅12​mi2∑s,tδs​t2=𝔼i​\[\(c¯θ,i−p¯i∗\)2\]\+𝔼i​\[12​mi2​∑s,tδs​t2\],\\begin\{split\}&\\mathbb\{E\}\\bigl\[\(c\_\{\\theta\}\-p^\{\*\}\)^\{2\}\\bigr\]=\\sum\_\{i\}\\frac\{m\_\{i\}\}\{N\}\\Bigl\[\\frac\{1\}\{m\_\{i\}\}\\sum\_\{s\}\\epsilon\_\{s\}^\{2\}\\Bigr\]\\\\ &=\\sum\_\{i\}\\frac\{m\_\{i\}\}\{N\}\\Bigl\[\(\\bar\{c\}\_\{\\theta,i\}\-\\bar\{p\}^\{\*\}\_\{i\}\)^\{2\}\+\\frac\{1\}\{2m\_\{i\}^\{2\}\}\\sum\_\{s,t\}\\delta\_\{st\}^\{2\}\\Bigr\]\\\\ &=\\sum\_\{i\}\\frac\{m\_\{i\}\}\{N\}\(\\bar\{c\}\_\{\\theta,i\}\-\\bar\{p\}^\{\*\}\_\{i\}\)^\{2\}\+\\sum\_\{i\}\\frac\{m\_\{i\}\}\{N\}\\cdot\\frac\{1\}\{2m\_\{i\}^\{2\}\}\\sum\_\{s,t\}\\delta\_\{st\}^\{2\}\\\\ &=\\mathbb\{E\}\_\{i\}\\bigl\[\(\\bar\{c\}\_\{\\theta,i\}\-\\bar\{p\}^\{\*\}\_\{i\}\)^\{2\}\\bigr\]\+\\mathbb\{E\}\_\{i\}\\Bigl\[\\tfrac\{1\}\{2m\_\{i\}^\{2\}\}\\textstyle\\sum\_\{s,t\}\\delta\_\{st\}^\{2\}\\Bigr\],\\end\{split\}where the last line uses the definition of𝔼i\\mathbb\{E\}\_\{i\}as the average over problems with weightmi/Nm\_\{i\}/N\. This is Eq\. \([4](https://arxiv.org/html/2610.33237#S3.E4)\)\.□\\square

#### Remarks\.

Calibration alone does not remove the second term\.Supposecθc\_\{\\theta\}gives every state of a problem the same value, for example the problem’s average successp¯i∗\\bar\{p\}^\{\*\}\_\{i\}\. Thenct−cs=0c\_\{t\}\-c\_\{s\}=0, soδs​t=−\(pt−ps\)\\delta\_\{st\}=\-\(p\_\{t\}\-p\_\{s\}\), and the second term becomes12​mi2​∑s,t\(pt−ps\)2\\frac\{1\}\{2m\_\{i\}^\{2\}\}\\sum\_\{s,t\}\(p\_\{t\}\-p\_\{s\}\)^\{2\}, which by the identity of Step 2, applied topsp\_\{s\}in place ofϵs\\epsilon\_\{s\}, equals the variance ofp∗p^\{\*\}over the states of problemii\. This value does not depend on which constant is chosen, so a confidence that ignores the evidence cannot reduce it, even if it is perfectly calibrated\.

The Brier score measures the same error\.The Brier score \(§[4](https://arxiv.org/html/2610.33237#S4.SS0.SSS0.Px5)\) compares the confidence with the empirical success ratep^​\(x\)\\hat\{p\}\(x\)rather than withp∗​\(x\)p^\{\*\}\(x\)\. Sincep^​\(x\)\\hat\{p\}\(x\)averagesKKindependent grades, each equal to11with probabilityp∗​\(x\)p^\{\*\}\(x\), it satisfies𝔼⁡\[p^​\(x\)∣x\]=p∗​\(x\)\\mathbb\{E\}\[\\hat\{p\}\(x\)\\mid x\]=p^\{\*\}\(x\)andVar⁡\(p^​\(x\)∣x\)=p∗​\(x\)​\(1−p∗​\(x\)\)/K\\mathrm\{Var\}\(\\hat\{p\}\(x\)\\mid x\)=p^\{\*\}\(x\)\(1\-p^\{\*\}\(x\)\)/K\. For a confidence computed fromxxalone, the cross term vanishes, and

𝔼⁡\[\(cθ−p^\)2\]=𝔼⁡\[\(cθ−p∗\)2\]\+1K​𝔼​\[p∗​\(1−p∗\)\],\\begin\{split\}\\mathbb\{E\}\\bigl\[\(c\_\{\\theta\}\-\\hat\{p\}\)^\{2\}\\bigr\]&=\\mathbb\{E\}\\bigl\[\(c\_\{\\theta\}\-p^\{\*\}\)^\{2\}\\bigr\]\\\\ &\\quad\+\\tfrac\{1\}\{K\}\\,\\mathbb\{E\}\\bigl\[p^\{\*\}\(1\-p^\{\*\}\)\\bigr\],\\end\{split\}where the expectation on the left also averages over the sampled grades\. The last term does not depend on the confidence, so differences in Brier score between methods reflect differences in the error that Proposition[1](https://arxiv.org/html/2610.33237#Thmproposition1)decomposes\.

### D\.2Proof of Corollary[1](https://arxiv.org/html/2610.33237#Thmcorollary1)

#### Corollary[1](https://arxiv.org/html/2610.33237#Thmcorollary1)\(restated\)\.

For any confidencecθc\_\{\\theta\}, let𝔼⁡\[ℒ\]\\mathbb\{E\}\[\\mathcal\{L\}\]be the loss of Eq\. \([6](https://arxiv.org/html/2610.33237#S3.E6)\) averaged over all evidence states and over theKKsampled answers that determinep^​\(x\)\\hat\{p\}\(x\), andℒ∗\\mathcal\{L\}^\{\*\}the minimum expected loss over all confidences, reached exactly atcθ=p∗c\_\{\\theta\}=p^\{\*\}\. Then

𝔼⁡\[\(cθ−p∗\)2\]≤12​\(𝔼⁡\[ℒ\]−ℒ∗\),\\mathbb\{E\}\\bigl\[\(c\_\{\\theta\}\-p^\{\*\}\)^\{2\}\\bigr\]\\leq\\tfrac\{1\}\{2\}\\bigl\(\\mathbb\{E\}\[\\mathcal\{L\}\]\-\\mathcal\{L\}^\{\*\}\\bigr\),so each term of Proposition[1](https://arxiv.org/html/2610.33237#Thmproposition1)is at most12​\(𝔼​\[ℒ\]−ℒ∗\)\\frac\{1\}\{2\}\(\\mathbb\{E\}\[\\mathcal\{L\}\]\-\\mathcal\{L\}^\{\*\}\)\.

#### Proof\.

Recall the soft\-label binary cross\-entropy of Eq\. \([6](https://arxiv.org/html/2610.33237#S3.E6)\), at each evidence statexx,

ℒ⁡\(x\)=−p^​\(x\)​log⁡cθ​\(x\)−\(1−p^​\(x\)\)​log⁡\(1−cθ​\(x\)\)\.\\begin\{split\}\\mathcal\{L\}\(x\)=\{\}&\-\\hat\{p\}\(x\)\\log c\_\{\\theta\}\(x\)\\\\ &\-\\bigl\(1\-\\hat\{p\}\(x\)\\bigr\)\\log\\bigl\(1\-c\_\{\\theta\}\(x\)\\bigr\)\.\\end\{split\}LetH⁡\(p\)=−p​log⁡p−\(1−p\)​log⁡\(1−p\)H\(p\)=\-p\\log p\-\(1\-p\)\\log\(1\-p\)be the binary entropy andKL\(p∥c\)=plogpc\+\(1−p\)log1−p1−c\\mathrm\{KL\}\(p\\,\\\|\\,c\)=p\\log\\frac\{p\}\{c\}\+\(1\-p\)\\log\\frac\{1\-p\}\{1\-c\}the divergence between Bernoulli distributions with meansppandcc, with the convention0​log⁡0=00\\log 0=0\. The confidence of SuffiCE is a softmax and so lies strictly between 0 and 1, which keeps every logarithm finite\. The loss is linear inp^​\(x\)\\hat\{p\}\(x\), so taking the expectation over the sampled grades replacesp^​\(x\)\\hat\{p\}\(x\)by𝔼⁡\[p^​\(x\)∣x\]=p∗​\(x\)\\mathbb\{E\}\[\\hat\{p\}\(x\)\\mid x\]=p^\{\*\}\(x\)\. At each state, writingp∗p^\{\*\}andcθc\_\{\\theta\}forp∗​\(x\)p^\{\*\}\(x\)andcθ​\(x\)c\_\{\\theta\}\(x\), the expected loss is therefore

𝔼⁡\[ℒ⁡\(x\)∣x\]=−p∗​log⁡cθ−\(1−p∗\)​log⁡\(1−cθ\)\.\\mathbb\{E\}\\bigl\[\\mathcal\{L\}\(x\)\\mid x\\bigr\]=\-p^\{\*\}\\log c\_\{\\theta\}\-\(1\-p^\{\*\}\)\\log\(1\-c\_\{\\theta\}\)\.To separate the part that depends oncθc\_\{\\theta\}from the part that does not, we use the entropy of the target,H⁡\(p∗\)=−p∗​log⁡p∗−\(1−p∗\)​log⁡\(1−p∗\)H\(p^\{\*\}\)=\-p^\{\*\}\\log p^\{\*\}\-\(1\-p^\{\*\}\)\\log\(1\-p^\{\*\}\)\. By this definition,H⁡\(p∗\)\+p∗​log⁡p∗\+\(1−p∗\)​log⁡\(1−p∗\)=0H\(p^\{\*\}\)\+p^\{\*\}\\log p^\{\*\}\+\(1\-p^\{\*\}\)\\log\(1\-p^\{\*\}\)=0, so adding these terms to the expected loss leaves it unchanged:

𝔼⁡\[ℒ⁡\(x\)∣x\]=H⁡\(p∗\)\+p∗​log⁡p∗\+\(1−p∗\)​log⁡\(1−p∗\)−p∗​log⁡cθ−\(1−p∗\)​log⁡\(1−cθ\)\.\\begin\{split\}\\mathbb\{E\}\\bigl\[\\mathcal\{L\}\(x\)\\mid x\\bigr\]&=H\(p^\{\*\}\)\+p^\{\*\}\\log p^\{\*\}\\\\ &\\quad\+\(1\-p^\{\*\}\)\\log\(1\-p^\{\*\}\)\\\\ &\\quad\-p^\{\*\}\\log c\_\{\\theta\}\\\\ &\\quad\-\(1\-p^\{\*\}\)\\log\(1\-c\_\{\\theta\}\)\.\\end\{split\}Grouping the two terms with the factorp∗p^\{\*\}and the two with the factor1−p∗1\-p^\{\*\},

𝔼⁡\[ℒ⁡\(x\)∣x\]=H⁡\(p∗\)\+p∗​\(log⁡p∗−log⁡cθ\)\+\(1−p∗\)​\(log⁡\(1−p∗\)CLOSEOPEN−log⁡\(1−cθ\)\),\\begin\{split\}\\mathbb\{E\}\\bigl\[\\mathcal\{L\}\(x\)\\mid x\\bigr\]&=H\(p^\{\*\}\)\+p^\{\*\}\\bigl\(\\log p^\{\*\}\-\\log c\_\{\\theta\}\\bigr\)\\\\ &\\quad\+\(1\-p^\{\*\}\)\\bigl\(\\log\(1\-p^\{\*\}\)\\\\ &\\qquad\-\\log\(1\-c\_\{\\theta\}\)\\bigr\),\\end\{split\}and writing each difference of logarithms as the logarithm of a ratio,

𝔼⁡\[ℒ⁡\(x\)∣x\]=H⁡\(p∗\)\+p∗​log⁡p∗cθ\+\(1−p∗\)​log⁡1−p∗1−cθ=H\(p∗\)\+KL\(p∗∥cθ\)\.\\begin\{split\}\\mathbb\{E\}\\bigl\[\\mathcal\{L\}\(x\)\\mid x\\bigr\]&=H\(p^\{\*\}\)\+p^\{\*\}\\log\\frac\{p^\{\*\}\}\{c\_\{\\theta\}\}\\\\ &\\quad\+\(1\-p^\{\*\}\)\\log\\frac\{1\-p^\{\*\}\}\{1\-c\_\{\\theta\}\}\\\\ &=H\(p^\{\*\}\)\+\\mathrm\{KL\}\(p^\{\*\}\\,\\\|\\,c\_\{\\theta\}\)\.\\end\{split\}Averaging this identity over all evidence states gives

𝔼\[ℒ\]=𝔼\[H\(p∗\)\]\+𝔼\[KL\(p∗∥cθ\)\]\.\\mathbb\{E\}\[\\mathcal\{L\}\]=\\mathbb\{E\}\\bigl\[H\(p^\{\*\}\)\\bigr\]\+\\mathbb\{E\}\\bigl\[\\mathrm\{KL\}\(p^\{\*\}\\,\\\|\\,c\_\{\\theta\}\)\\bigr\]\.The first term depends only on the answering policy, not on the confidence, so no choice ofcθc\_\{\\theta\}can change it\. The second term is an average of divergences, each nonnegative and zero only whencθ​\(x\)=p∗​\(x\)c\_\{\\theta\}\(x\)=p^\{\*\}\(x\)\. Every confidence therefore has𝔼⁡\[ℒ\]≥𝔼⁡\[H⁡\(p∗\)\]\\mathbb\{E\}\[\\mathcal\{L\}\]\\geq\\mathbb\{E\}\[H\(p^\{\*\}\)\], with equality exactly whencθ​\(x\)=p∗​\(x\)c\_\{\\theta\}\(x\)=p^\{\*\}\(x\)at every state\. Hence the minimum expected loss isℒ∗=𝔼⁡\[H⁡\(p∗\)\]\\mathcal\{L\}^\{\*\}=\\mathbb\{E\}\[H\(p^\{\*\}\)\], reached exactly atcθ=p∗c\_\{\\theta\}=p^\{\*\}, and

𝔼\[ℒ\]−ℒ∗=𝔼\[KL\(p∗∥cθ\)\]\.\\mathbb\{E\}\[\\mathcal\{L\}\]\-\\mathcal\{L\}^\{\*\}=\\mathbb\{E\}\\bigl\[\\mathrm\{KL\}\(p^\{\*\}\\,\\\|\\,c\_\{\\theta\}\)\\bigr\]\.Pinsker’s inequality for Bernoulli distributions statesKL\(p∥c\)≥2\(p−c\)2\\mathrm\{KL\}\(p\\,\\\|\\,c\)\\geq 2\(p\-c\)^\{2\}\. Applying it at every state and averaging,

𝔼⁡\[ℒ\]−ℒ∗=𝔼\[KL\(p∗∥cθ\)\]≥2​𝔼​\[\(cθ−p∗\)2\],\\begin\{split\}\\mathbb\{E\}\[\\mathcal\{L\}\]\-\\mathcal\{L\}^\{\*\}&=\\mathbb\{E\}\\bigl\[\\mathrm\{KL\}\(p^\{\*\}\\,\\\|\\,c\_\{\\theta\}\)\\bigr\]\\\\ &\\geq 2\\,\\mathbb\{E\}\\bigl\[\(c\_\{\\theta\}\-p^\{\*\}\)^\{2\}\\bigr\],\\end\{split\}which is Eq\. \([7](https://arxiv.org/html/2610.33237#S3.E7)\)\. By Proposition[1](https://arxiv.org/html/2610.33237#Thmproposition1), the two terms are nonnegative and sum to𝔼⁡\[\(cθ−p∗\)2\]\\mathbb\{E\}\[\(c\_\{\\theta\}\-p^\{\*\}\)^\{2\}\], so each is at most12​\(𝔼​\[ℒ\]−ℒ∗\)\\frac\{1\}\{2\}\(\\mathbb\{E\}\[\\mathcal\{L\}\]\-\\mathcal\{L\}^\{\*\}\)\.□\\square

For SuffiCE, this means that lowering the training loss lowers an upper bound on both terms of Proposition[1](https://arxiv.org/html/2610.33237#Thmproposition1), the error in each problem’s average success and the error in how confidence changes between states of the same problem\. The bound holds only over the states the loss is averaged over, which is why SuffiCE trains on many evidence states of each problem rather than on full states alone \(§[5\.2](https://arxiv.org/html/2610.33237#S5.SS2)\)\.

Table 12:Comparison with the closest prior work along the five properties defined below\. ✓ yes,–no\.

## Appendix EComparison with Related Work

We compare our setting with the closest prior work on confidence estimation along five properties \(Table[D\.2](https://arxiv.org/html/2610.33237#A4.SS2.SSS0.Px2)\)\.

- •Pre\-answer: the confidence is produced before any answer is generated, so it can decide whether to answer or to gather more evidence\.
- •Trained: the confidence estimator is trained, rather than read from an untrained model or elicited by prompting\.
- •Controlled evidence subsets: the same problem is observed with different combinations of evidence, not only with evidence that accumulates in a fixed order\.
- •Within\-problem evaluation: the evaluation tests whether changes in confidence follow changes in success across states of the same problem\.
- •Own token probability: the confidence is the model’s next\-token probability, with no added classifier head or verbalized number\.

P\(IK\)\([Kadavath et al\., 2022](https://arxiv.org/html/2610.33237#bib.bib19)\)trains an added value head to predict, from the question alone, whether the model will answer correctly\. It adds source documents and hints only at evaluation and does not relate the change in P\(IK\) to the change in accuracy\. Pre\-generation probes\([Lugoloobi et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib23)\)train linear probes on the model’s activations before generation to predict success, with each problem shown in a single fixed prompt\. Calibration tuning\([Kapoor et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib13)\)fine\-tunes the model to judge whether a proposed answer is correct, read from its token probability\. Rewarding Doubt\([Bani\-Harouni et al\., 2026b](https://arxiv.org/html/2610.33237#bib.bib14)\)uses reinforcement learning to train the model to state a confidence from 0 to 10 in an answer it has already generated, with each question asked without context\. LA\-CDM\([Bani\-Harouni et al\., 2026a](https://arxiv.org/html/2610.33237#bib.bib27)\)trains a stated confidence in a diagnosis the agent has already proposed, and the evidence is limited to the tests its agent chose to request\. P\(Sufficient\)\([Zhang et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib15)\)prompts the model for the probability that the information so far entails its answer\. Its clues arrive in a fixed order, and its evaluation measures per\-turn calibration and whether confidence rises over turns\.

Each prior method shares some of these properties, and ours is the only one that has all five\. No prior method trains the model’s own token probability before an answer exists\. P\(IK\)\([Kadavath et al\., 2022](https://arxiv.org/html/2610.33237#bib.bib19)\)and pre\-generation probes\([Lugoloobi et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib23)\)estimate success before answering but read it from an added head or probe, while calibration tuning\([Kapoor et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib13)\)and P\(Sufficient\)\([Zhang et al\., 2026](https://arxiv.org/html/2610.33237#bib.bib15)\)use the model’s token probability but score an answer it has already given\. SuffiCE instead trains the model’s own probability of choosing to answer, so the same next\-token probability that would decide whether to answer or gather more evidence is calibrated, with no extra head and no answer to generate first\. We choose this formulation because the model’s next\-token distribution already assigns a probability to answering versus seeking more evidence, so no classifier head is added and the confidence is the model’s own preference between the two actions\. A full\-only estimator with the same formulation falls short of SuffiCE \(§[5\.2](https://arxiv.org/html/2610.33237#S5.SS2)\), so the gains come from training across evidence states\. Two further properties set SuffiCE apart, and no prior method has either\. Training on controlled evidence subsets of the same problem lets the estimator learn which evidence decides success, rather than only which problems are hard \(§[5\.2](https://arxiv.org/html/2610.33237#S5.SS2)\)\. Evaluating within each problem tests this directly, showing that confidence rises with the decisive evidence and not with every piece added \(§[5\.3](https://arxiv.org/html/2610.33237#S5.SS3)\)\.

## Appendix FCase Studies

### F\.1MuSiQue Case Studies

We examine two branching \(3hop2\) questions from the MuSiQue test set in detail \(Tables[13](https://arxiv.org/html/2610.33237#A6.T13)and[14](https://arxiv.org/html/2610.33237#A6.T14)\)\. In both, the branchese1e\_\{1\}ande2e\_\{2\}answer two independent sub\-questions, and only the final hope3e\_\{3\}combines their answers into the requested date\.

#### Case 1: When the Republicans took control of the Senate\.

The branches establish that Alaska supports the Republicans \(e1e\_\{1\}\) and that the Senate sets its own rules \(e2e\_\{2\}\), but onlye3e\_\{3\}gives the date on which the Republicans regained the Senate\. With both branches shown, no sampled answer is correct\. The base logit nonetheless jumps from at most 0\.02 to 0\.50, treating the two branches as progress despite the lack of sampled success\. SuffiCE stays at or below 0\.02, matching this low success rate\. Oncee3e\_\{3\}is shown, 20% of sampled answers are correct\. SuffiCE rises to 0\.19, closely matching this rate, while the base logit falls to 0\.11 and becomes underconfident\.

#### Case 2: When the Dutch reached New Zealand\.

The branches establish that The Lord of the Rings was filmed in New Zealand \(e1e\_\{1\}\) and that the Dutch first brought enslaved people to New York \(e2e\_\{2\}\), but onlye3e\_\{3\}gives the date on which the Dutch navigator Abel Tasman reached New Zealand\. Again no sampled answer is correct with both branches, yet the base logit climbs with each branch to 0\.98\. SuffiCE reaches only 0\.16\. With full evidence, 30% of sampled answers are correct\. SuffiCE gives 0\.34, again close to this rate, while the base logit stays at 0\.96 and is far overconfident\.

#### Comparing the two full states\.

The two questions share the same structure and both show all three supporting paragraphs, yet they differ in how often they are answered correctly\. SuffiCE follows this difference, matching each success rate within 0\.04, whereas the base logit misses in opposite directions\. The confidence of SuffiCE is not fixed by the evidence being complete, so it judges not only whether the reasoning chain is complete but also how likely the answering model is to succeed once it is\.

Table 13:Case study on a 3hop2 MuSiQue question \(Qwen3\.5\-9B\), where the branchese1e\_\{1\}ande2e\_\{2\}together lead to the final hope3e\_\{3\}\. Columns are evidence states, marked by which supporting facts are shown \(∅\\varnothingfor none\)\.
Table 14:A second case study on a 3hop2 MuSiQue question \(Qwen3\.5\-9B\), in the format of Table[13](https://arxiv.org/html/2610.33237#A6.T13)\. The shaded state has both branches but not the final hop\.

### F\.2Full Evidence States of the Case Studies

We give the complete inputs behind the clinical case study of §[5\.3](https://arxiv.org/html/2610.33237#S5.SS3)and the first MuSiQue case study of §[F\.1](https://arxiv.org/html/2610.33237#A6.SS1)word for word, with the evidence states of each example and the evidence they contain \(Tables[15](https://arxiv.org/html/2610.33237#A6.T15)and[16](https://arxiv.org/html/2610.33237#A6.T16)\)\. The state names and evidence labels, such as\{1,4\}\\\{1,4\\\}ore1e\_\{1\}, are defined only for analysis and are never shown to the model\. The model sees only the question and the evidence as it appears in the input, with the section headings of the clinical case but no indication of which state it is in\.

Table 15:Full input of the MedQA case of Table[3](https://arxiv.org/html/2610.33237#S5.T3)and the evidence states shown there\. Each state shows the objective followed by the listed sections, verbatim\.
Table 16:Full input and all eight evidence states of the 3hop2 MuSiQue question of Table[13](https://arxiv.org/html/2610.33237#A6.T13)\. Supporting paragraphse1e\_\{1\}–e3e\_\{3\}and distractorsd1d\_\{1\}–d3d\_\{3\}are shown verbatim\. The six paragraphs of each question are shuffled once into a fixed order, heree2,d1,d2,e1,e3,d3e\_\{2\},d\_\{1\},d\_\{2\},e\_\{1\},e\_\{3\},d\_\{3\}, and every state shows its paragraphs in that order, so a paragraph’s position does not reveal the state\. Every state except the question\-only one shows three paragraphs, padded with distractors\.

## Appendix GEthical Considerations

Our clinical experiments use MIMIC\-IV\-Ext Clinical Decision Making\([Hager et al\., 2024](https://arxiv.org/html/2610.33237#bib.bib32)\), a de\-identified dataset accessed through PhysioNet under its data use agreement, and we do not release its data\. SuffiCE is a research method and is not intended for clinical use\. We used Claude and ChatGPT as AI assistants in preparing this manuscript, in accordance with the ACL policy on AI writing assistance\.