From token probabilities to calibrated confidence: An empirical study of mathematical question answering

# From token probabilities to calibrated confidence: An empirical study of mathematical question answering
Source: [https://arxiv.org/html/2608.07827](https://arxiv.org/html/2608.07827)
###### Abstract

Confidence estimation for large language models \(LLMs\) aims to estimate the probability that a generated answer is correct, while calibration aligns these estimates with empirical accuracy\. Prior work has shown that token probabilities are often overconfident, we investigate whether these readily available signals can nevertheless provide well\-calibrated confidence estimation for mathematical question answering\. We compare single\-pass estimators, which reuse token probabilities from the original generation, with multi\-pass estimators, which obtain additional confidence signals through verification or stochastic forward passes\. While individual token probabilities can be highly saturated, we find that aggregating token probabilities over the full sequence captures small but consistent differences between correct and incorrect generations, yielding more informative confidence estimates\. Multi\-pass methods can yield calibrated confidence estimates\. We study two such approaches: self\-verification through re\-prompting, including a lower\-cost in\-situ variant, and Monte Carlo Dropout, which derives confidence from variation across stochastic forward passes\. We further evaluate two post\-hoc calibration methods, Platt scaling and isotonic regression, both of which substantially reduce in\-domain calibration error\. However, their data efficiency varies with dataset difficulty, and the calibration mappings often transfer asymmetrically across datasets and models\.

From token probabilities to calibrated confidence: An empirical study of mathematical question answering

Avery Ma Lorne Schell Vin Bhaskara Leila PishdadRBC Borealis

## 1Introduction

Large language models \(LLMs\) have achieved remarkable success on a broad range of complex, multi\-step tasks\(Achiamet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib4); Teamet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib5)\)\. As they are increasingly deployed in high\-stakes domains such as financial forecasting\(Wuet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib12); Yuet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib13)\), legal services\(Caiet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib18); Luoet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib19)\), and medical decision support\(Ravendaet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib16); Bartelset al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib17)\), generating a correct response alone is no longer sufficient\. In these settings, it is equally important to know when a model’s output can be trusted, motivating the need for accurate, well\-calibrated estimates of response correctness\.

A natural basis for confidence estimation is the token probabilities produced by LLMs during inference\. However,rawtoken probabilities often provide unreliable and saturated confidence signals\(Mielkeet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib22); Kadavathet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib8); Chhikara,[2025](https://arxiv.org/html/2608.07827#bib.bib21)\), because LLMs are trained to optimize next\-token prediction\(Ouyanget al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib23); Stiennonet al\.,[2020](https://arxiv.org/html/2608.07827#bib.bib24)\)rather than to estimate whether a completed response is correct\. This raises a fundamental question: whether token probabilities contain useful signals for confidence estimation and how effectively these signals can be calibrated to reflect empirical correctness\.

In this work, we conduct an empirical study of token\-probability\-based confidence estimation and post\-hoc calibration for mathematical question answering\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.07827#bib.bib25); Patelet al\.,[2021](https://arxiv.org/html/2608.07827#bib.bib27); Gaoet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib26)\)\. We organize confidence estimators by the number of inference passes they require\. For single\-pass estimators, we analyze how confidence depends on the subset of generated tokens considered and the method used to aggregate their probabilities\. For multi\-pass methods, we study self\-verification\(Kadavathet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib8)\), evaluate a lower\-cost in\-situ variant that appends the verification query directly to the original generation, and Monte Carlo \(MC\) Dropout\(Gal and Ghahramani,[2016](https://arxiv.org/html/2608.07827#bib.bib10)\)\. Finally, we evaluate isotonic regression\(Zadrozny and Elkan,[2002](https://arxiv.org/html/2608.07827#bib.bib28)\)and Platt scaling\(Platt and others,[1999](https://arxiv.org/html/2608.07827#bib.bib29)\), examining how post\-hoc calibration aligns confidence estimates with empirical accuracy\. Our main contributions are:

- •We systematically compare single\-pass confidence estimators derived from different subsets and aggregations of token probabilities\. We find that the overconfidence issue documented in prior work is particularly pronounced for answer\-token estimators\. By contrast, aggregating probabilities over the full reasoning trajectory yields discriminative confidence estimates\.
- •For multi\-pass estimation, we evaluate an efficientin\-situself\-verification variant that appends the verification query directly to the original response\. It achieves calibration comparable to standard model re\-prompting while reducing token\-processing overhead by 88%, indicating that verification\-based confidence can be estimated without re\-encoding the full question–response pair and may therefore be particularly efficient for long reasoning outputs\.
- •We further analyze MC Dropout for multi\-pass confidence estimation\. We find that variation in answer\-token predictive distributions captures uncertainty not reflected in the decoded output, even when the generated answer remains unchanged across stochastic forward passes\.
- •We evaluate Platt scaling and isotonic regression and find that both generally reduce in\-domain calibration error, with isotonic regression often effective using as few as 50 examples\. Together with our cross\-domain analysis, these results show that calibration efficiency depends on dataset difficulty, while transfer depends on the specific source and target model–dataset pairs\.

## 2Related Work

Confidence estimation methods for LLMsare typically categorized as white\-box or black\-box\(Genget al\.,[2024](https://arxiv.org/html/2608.07827#bib.bib67)\)\. Black\-box methods derive confidence from generated outputs, including verbalized confidence\(Xionget al\.,[2024](https://arxiv.org/html/2608.07827#bib.bib64)\)and consistency across sampled generations\(Lyuet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib50)\)\. White\-box methods instead leverage internal representations\(Linet al\.,[2024b](https://arxiv.org/html/2608.07827#bib.bib35)\)or logit\-based signals\(Huanget al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib41); Vazhentsevet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib68); Kuhnet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib49)\)\. Token probabilities are particularly appealing because they are available during inference and lie in\[0,1\]\[0,1\], making them natural candidates for confidence scores\. However, they are often overconfident, concentrated near one, and poorly calibrated for answer correctness\(Mielkeet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib22); Chhikara,[2025](https://arxiv.org/html/2608.07827#bib.bib21)\)\. Motivated by these limitations, we examine how token selection, aggregation, and post\-hoc calibration affect their usefulness for confidence estimation\.

Multi\-pass methods obtain additional confidence signals through repeated forward passes\. Thep​\(True\)p\(\\mathrm\{True\}\)approach re\-prompts the model to judge its generated answer and computes confidence by normalizing the probabilities ofTrueandFalse\(Kadavathet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib8)\)\. This requires processing the full question–answer context again\.

MC Dropout estimates uncertainty from variation across stochastic forward passes\(Gal and Ghahramani,[2016](https://arxiv.org/html/2608.07827#bib.bib10)\)and has been studied in classification\(Ficsor and Berend,[2025](https://arxiv.org/html/2608.07827#bib.bib75)\), summarization\(Zablotskaiaet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib74)\), and generative QA\(Mora\-Cross and Calderon\-Ramirez,[2024](https://arxiv.org/html/2608.07827#bib.bib77)\)\. More recently,Zhanget al\.\([2026](https://arxiv.org/html/2608.07827#bib.bib78)\)estimate token\-level uncertainty conditioned on a fixed reasoning response using weight perturbations and aggregate it over the full sequence\. We instead apply standard MC Dropout to a fixed model\-generated reasoning trajectory and measure variation in final\-answer\-token distributions, evaluating whether it yields calibrated estimates of answer correctness\.

Confidence calibrationaims to align confidence scores with empirical correctness\. Recent work has explored increasingly complex calibration methods for LLMs, including auxiliary LLM\-based calibrators\(Tanet al\.,[2026](https://arxiv.org/html/2608.07827#bib.bib79); Ulmeret al\.,[2024](https://arxiv.org/html/2608.07827#bib.bib82)\), layer\-specific calibration\(Joshiet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib81); Stolfoet al\.,[2024](https://arxiv.org/html/2608.07827#bib.bib85)\), and fine\-tuning\-based approachesLiet al\.\([2025](https://arxiv.org/html/2608.07827#bib.bib83)\); Linet al\.\([2024a](https://arxiv.org/html/2608.07827#bib.bib84)\)\. In contrast, we revisit two classic post\-hoc approaches, Platt scaling\(Platt and others,[1999](https://arxiv.org/html/2608.07827#bib.bib29)\)and isotonic regression\(Zadrozny and Elkan,[2002](https://arxiv.org/html/2608.07827#bib.bib28)\), and show that they can substantially improve token\-probability\-based confidence estimates\. Their simplicity lets us study in\-domain and cross\-domain calibration, along with calibration\-data efficiency\. We describe calibration metrics in Section[5](https://arxiv.org/html/2608.07827#S5)\.

## 3Preliminaries

In this section, we introduce the notation and problem setting for our study of confidence estimation and calibration in LLMs\.

### 3\.1LLM Inference

LLMs are neural networks that model sequential data by predicting the next token in a sequence\(Vaswani,[2017](https://arxiv.org/html/2608.07827#bib.bib1)\)\. In this work, we focus on LLMs trained for text generation\(Brownet al\.,[2020](https://arxiv.org/html/2608.07827#bib.bib2); Achiamet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib4); Touvronet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib3)\)\.

Let𝒱\\mathcal\{V\}denote a finite vocabulary\. Given an input sequence𝒙=\(x1,…,xm\)∈𝒱m\\bm\{x\}=\(x\_\{1\},\\dotsc,x\_\{m\}\)\\in\\mathcal\{V\}^\{m\}, an LLM defines a probability distributionℙ\(⋅∣𝒙\)∈Δ\(𝒱\)\\mathbb\{P\}\(\\cdot\\mid\\bm\{x\}\)\\in\\Delta\(\\mathcal\{V\}\)over the next token, whereΔ​\(𝒱\)\\Delta\(\\mathcal\{V\}\)denotes the probability simplex over𝒱\\mathcal\{V\}\. For an output sequence𝒚=\(y1,…,yn\)∈𝒱n\\bm\{y\}=\(y\_\{1\},\\dotsc,y\_\{n\}\)\\in\\mathcal\{V\}^\{n\}, the autoregressive factorization is given byℙ​\(𝒚∣𝒙\)=∏i=1nℙ​\(yi∣𝒙,𝒚<i\),\\mathbb\{P\}\(\\bm\{y\}\\mid\\bm\{x\}\)=\\prod\_\{i=1\}^\{n\}\\mathbb\{P\}\(y\_\{i\}\\mid\\bm\{x\},\\bm\{y\}\_\{<i\}\),where𝒚<i=\(y1,…,yi−1\)\\bm\{y\}\_\{<i\}=\(y\_\{1\},\\dotsc,y\_\{i\-1\}\)\.

### 3\.2Mathematical Question Answering with Verifiable Final Answers

Confidence estimation requires a notion of output correctness\. For open\-ended dialogue\(Liet al\.,[2017](https://arxiv.org/html/2608.07827#bib.bib32)\), correctness can be ambiguous and task\-dependent, making confidence difficult to define and evaluate\(Liuet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib33); Mendonçaet al\.,[2024](https://arxiv.org/html/2608.07827#bib.bib34)\)\. We therefore focus on mathematical question answering\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.07827#bib.bib25); Patelet al\.,[2021](https://arxiv.org/html/2608.07827#bib.bib27); Gaoet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib26)\), which provides a*structured*and*verifiable*setting: model outputs typically contain multi\-step reasoning followed by a final answer\(Weiet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib39)\), and correctness can be evaluated against a ground\-truth answer\.

We decompose each generated sequence as𝒚=\(𝒚reason,𝒚ans\),\\bm\{y\}=\(\\bm\{y\}^\{\\text\{reason\}\},\\bm\{y\}^\{\\text\{ans\}\}\),where𝒚reason\\bm\{y\}^\{\\text\{reason\}\}denotes the intermediate reasoning tokens and𝒚ans\\bm\{y\}^\{\\text\{ans\}\}denotes the final answer tokens\. We refer to𝒚ans\\bm\{y\}^\{\\text\{ans\}\}as theanswer tokens\. Given a ground\-truth answer𝒚∗\\bm\{y\}^\{\*\}, we define a correctness function𝕀​\(𝒚ans,𝒚∗\)∈\{0,1\}\\mathbb\{I\}\(\\bm\{y\}^\{\\text\{ans\}\},\\bm\{y\}^\{\*\}\)\\in\\\{0,1\\\}, which evaluates correctness under any chosen metric \(e\.g\., exact match, semantic equivalence\(Peineltet al\.,[2020](https://arxiv.org/html/2608.07827#bib.bib46)\), or LLM\-as\-a\-Judge\(Perezet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib44)\)\)\.

Because token probabilities are available for both the reasoning trajectory and the final answer during inference, confidence can be constructed from different token subsets and aggregation rules\. Prior work has considered several such choices\(Linet al\.,[2024b](https://arxiv.org/html/2608.07827#bib.bib35); Guptaet al\.,[2024](https://arxiv.org/html/2608.07827#bib.bib36); Huanget al\.,[2024](https://arxiv.org/html/2608.07827#bib.bib37); Jianget al\.,[2020](https://arxiv.org/html/2608.07827#bib.bib42); Huanget al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib41); Orgadet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib47)\), often as baselines for error detection\. We instead examine how these choices affect both the discrimination and calibration of confidence estimates for mathematical question answering\.

## 4Confidence Estimation

Our goal is to systematically study confidence estimators derived from token probabilities and their calibration behavior\. We distinguish confidence estimation from calibration: confidence estimation constructs a scalar score to reflect answer correctness, while calibration adjusts this score to better align with empirical accuracy\. In this section, we focus on the estimation step\.

### 4\.1Problem Formulation

Confidence is commonly interpreted as the probability that a model’s output is correct\(Guoet al\.,[2017](https://arxiv.org/html/2608.07827#bib.bib7)\)\. Consider a generated response𝒚^\\hat\{\\bm\{y\}\}\. A confidence estimatorc​\(𝒙,𝒚^\)c\(\\bm\{x\},\\hat\{\\bm\{y\}\}\)aims to approximate the probability that the generated answer is correct:c​\(𝒙,𝒚^\)≜ℙ​\(𝕀​\(𝒚ans,𝒚∗\)=1∣𝒙,𝒚^\)c\(\\bm\{x\},\\hat\{\\bm\{y\}\}\)\\triangleq\\mathbb\{P\}\\big\(\\mathbb\{I\}\(\\bm\{y\}^\{\\text\{ans\}\},\\bm\{y\}^\{\*\}\)=1\\mid\\bm\{x\},\\hat\{\\bm\{y\}\}\\big\)\. This confidence scoreccis different from the sequence likelihoodℙ​\(𝒚∣𝒙\)\\mathbb\{P\}\(\\bm\{y\}\\mid\\bm\{x\}\)defined in Section[3\.1](https://arxiv.org/html/2608.07827#S3.SS1)\. These quantities are often not the same: answer tokens can be highly likely when conditioned on an incorrect reasoning trajectory, while low\-probability tokens along the trajectory may signal potential errors, as we later show in Section[6](https://arxiv.org/html/2608.07827#S6)\. Thus, estimating confidence from token probabilities requires two choices: which generated tokens to consider and how to aggregate their probabilities\.

### 4\.2Single\-pass Estimation

We group confidence estimators based on the number of forward passes required\. Single\-pass estimators reuse token probabilities from the original generation and therefore incur no additional forward passes\. We instantiate the two choices above using either the full generated sequence\(𝒚reason,𝒚ans\)\(\\bm\{y\}^\{\\text\{reason\}\},\\bm\{y\}^\{\\text\{ans\}\}\)or only the answer tokens𝒚ans\\bm\{y\}^\{\\text\{ans\}\}, and aggregating their probabilities using either the arithmetic mean\(Huanget al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib41); Jianget al\.,[2020](https://arxiv.org/html/2608.07827#bib.bib42); Orgadet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib47)\)or the length\-normalized joint probability, equivalent to the geometric mean of token probabilities\(Wuet al\.,[2016](https://arxiv.org/html/2608.07827#bib.bib40); Malinin and Gales,[2021](https://arxiv.org/html/2608.07827#bib.bib38)\)\. Combining these choices yields four estimators: sequence joint \(Seq Joint\), sequence average \(Seq Avg\), answer joint \(Ans Joint\), and answer average \(Ans Avg\)\. Length normalization reduces the joint probability’s inherent preference for shorter sequences\.

MethodDefinitionSeq Joint\(∏i=1\|𝒚^\|ℙ​\(yi∣𝒙,𝒚^<i\)\)1/\|𝒚^\|\\left\(\\prod\_\{i=1\}^\{\|\\hat\{\\bm\{y\}\}\|\}\\mathbb\{P\}\(y\_\{i\}\\mid\\bm\{x\},\\hat\{\\bm\{y\}\}\_\{<i\}\)\\right\)^\{1/\|\\hat\{\\bm\{y\}\}\|\}Seq Avg1\|𝒚^\|​∑i=1\|𝒚^\|ℙ​\(yi∣𝒙,𝒚^<i\)\\frac\{1\}\{\|\\hat\{\\bm\{y\}\}\|\}\\sum\_\{i=1\}^\{\|\\hat\{\\bm\{y\}\}\|\}\\mathbb\{P\}\(y\_\{i\}\\mid\\bm\{x\},\\hat\{\\bm\{y\}\}\_\{<i\}\)Ans Joint\(∏j=1\|𝒚ans\|ℙ​\(yjans∣𝒙,𝒚reason,𝒚<jans\)\)1/\|𝒚ans\|\\left\(\\prod\_\{j=1\}^\{\|\\bm\{y\}^\{\\text\{ans\}\}\|\}\\mathbb\{P\}\(y^\{\\mathrm\{ans\}\}\_\{j\}\\mid\\bm\{x\},\\bm\{y\}^\{\\text\{reason\}\},\\bm\{y\}^\{\\text\{ans\}\}\_\{<j\}\)\\right\)^\{1/\|\\bm\{y\}^\{\\text\{ans\}\}\|\}Ans Avg1\|𝒚ans\|​∑j=1\|𝒚ans\|ℙ​\(yjans∣𝒙,𝒚reason,𝒚<jans\)\\frac\{1\}\{\|\\bm\{y\}^\{\\text\{ans\}\}\|\}\\sum\_\{j=1\}^\{\|\\bm\{y\}^\{\\text\{ans\}\}\|\}\\mathbb\{P\}\(y^\{\\mathrm\{ans\}\}\_\{j\}\\mid\\bm\{x\},\\bm\{y\}^\{\\text\{reason\}\},\\bm\{y\}^\{\\text\{ans\}\}\_\{<j\}\)p​\(True\)p\(\\mathrm\{True\}\)ℙ​\(y𝚃𝚛𝚞𝚎∣𝒙ver\)ℙ​\(y𝚃𝚛𝚞𝚎∣𝒙ver\)\+ℙ​\(y𝙵𝚊𝚕𝚜𝚎∣𝒙ver\)\\frac\{\\mathbb\{P\}\(y\_\{\\mathtt\{True\}\}\\mid\\bm\{x\}^\{\\mathrm\{ver\}\}\)\}\{\\mathbb\{P\}\(y\_\{\\mathtt\{True\}\}\\mid\\bm\{x\}^\{\\mathrm\{ver\}\}\)\+\\mathbb\{P\}\(y\_\{\\mathtt\{False\}\}\\mid\\bm\{x\}^\{\\mathrm\{ver\}\}\)\}cBALDc\_\{\\mathrm\{BALD\}\}1\|𝒚ans\|​∑j=1\|𝒚ans\|ϕ​\(BALDj\)\\frac\{1\}\{\|\\bm\{y\}^\{\\text\{ans\}\}\|\}\\sum\_\{j=1\}^\{\|\\bm\{y\}^\{\\text\{ans\}\}\|\}\\phi\(\\mathrm\{BALD\}\_\{j\}\)Table 1:Confidence estimation using token probabilities\.Single\-pass estimators use token probabilities from the original generation: sequence\-based estimators aggregate probabilities over the full response, whereas answer\-based estimators use the final\-answer tokens\. Multi\-pass estimators obtain additional confidence signals throughp​\(True\)p\(\\mathrm\{True\}\)\-based self\-verification\(Kadavathet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib8)\)orcBALDc\_\{\\mathrm\{BALD\}\}computed from stochastic forward passes\(Houlsbyet al\.,[2011](https://arxiv.org/html/2608.07827#bib.bib86)\)\.
### 4\.3Multi\-pass Estimation

The single\-pass estimators above use only token probabilities produced during the original generation\. We next consider two multi\-pass approaches that obtain additional confidence signals through repeated forward passes\.

Self\-verification via re\-prompting:Prior work has used verification\-style prompting to elicit confidence from language models\(Kadavathet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib8)\)\. After generating a response, the model is queried with a verification prompt𝒙ver=g​\(𝒙,𝒚^\)\\bm\{x\}^\{\\mathrm\{ver\}\}=g\(\\bm\{x\},\\hat\{\\bm\{y\}\}\)constructed from the original input and generated response\. As shown in Table[1](https://arxiv.org/html/2608.07827#S4.T1),p​\(True\)p\(\\mathrm\{True\}\)computes confidence by normalizing the probabilities assigned to theTrueandFalsetokens\. Althoughp​\(True\)p\(\\mathrm\{True\}\)is a widely used baseline\(Orgadet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib47); Linet al\.,[2024b](https://arxiv.org/html/2608.07827#bib.bib35); Mahautet al\.,[2024](https://arxiv.org/html/2608.07827#bib.bib48); Lyuet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib50)\), it requires processing the question–response context again\. Its token\-processing overhead therefore increases with response length and can become substantial for long reasoning trajectories\. In Section[6](https://arxiv.org/html/2608.07827#S6), we evaluate the calibration ofp​\(True\)p\(\\mathrm\{True\}\)and a simplein\-situvariant that appends the verification prompt to the original generation trajectory, avoiding re\-encoding the full context\.

Confidence estimation via MC Dropout:We next evaluate MC Dropout, a widely used approximate Bayesian approach for estimating uncertainty through stochastic forward passes\(Gal and Ghahramani,[2016](https://arxiv.org/html/2608.07827#bib.bib10); Gal and others,[2017](https://arxiv.org/html/2608.07827#bib.bib11)\)\. Prior work has applied MC dropout to uncertainty estimation in language models across several settings\(Gaoet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib52); Zhanget al\.,[2026](https://arxiv.org/html/2608.07827#bib.bib78)\)\. Our analysis measures variation in the predictive distributions of answer tokens conditioned on a fixed reasoning trajectory\. This design avoids the cost of repeatedly generating complete alternative reasoning trajectories, while leveraging the structured outputs of mathematical question answering, where a reasoning trajectory is followed by a compact, verifiable final answer\.

We first generate a response with dropout disabled, then fix the trajectory\(𝒚reason,𝒚ans\)\(\\bm\{y\}^\{\\text\{reason\}\},\\bm\{y\}^\{\\text\{ans\}\}\)and enable dropout during inference\. Importantly, we do not regenerate the response under each dropout mask\. Instead, for each stochastic forward passt=1,…,Tt=1,\\dotsc,Tand answer\-token positionjj, we evaluate the next\-token predictive distribution conditioned on the same fixed prefix:ℙt,j\(⋅\)≔ℙt\(⋅∣𝒙,𝒚reason,𝒚<jans\)\\mathbb\{P\}\_\{t,j\}\(\\cdot\)\\coloneqq\\mathbb\{P\}\_\{t\}\\left\(\\cdot\\mid\\bm\{x\},\\bm\{y\}^\{\\text\{reason\}\},\\bm\{y\}^\{\\text\{ans\}\}\_\{<j\}\\right\)\. If the prediction at an answer position is stable under dropout, these distributions should be similar across stochastic passes\. Greater disagreement indicates higher uncertainty about that answer token given the generated reasoning trajectory\. Letℙ¯j​\(⋅\)≔1T​∑t=1Tℙt,j​\(⋅\)\\bar\{\\mathbb\{P\}\}\_\{j\}\(\\cdot\)\\coloneqq\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{P\}\_\{t,j\}\(\\cdot\)denote the predictive distribution averaged across dropout samples\. We quantify disagreement at answer\-token positionjjusing Bayesian Active Learning by Disagreement \(BALD\)\(Houlsbyet al\.,[2011](https://arxiv.org/html/2608.07827#bib.bib86); Atighehchianet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib9)\):

BALDj=H​\(ℙ¯j\)⏟predictive entropy−1T​∑t=1TH​\(ℙt,j\)⏟expected conditionalentropy\.\\mathrm\{BALD\}\_\{j\}=\\underbrace\{H\\\!\\left\(\\bar\{\\mathbb\{P\}\}\_\{j\}\\right\)\}\_\{\\text\{predictive entropy\}\}\-\\underbrace\{\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}H\\\!\\left\(\\mathbb\{P\}\_\{t,j\}\\right\)\}\_\{\\begin\{subarray\}\{c\}\\text\{expected conditional\}\\\\ \\text\{entropy\}\\end\{subarray\}\}\.\(1\)
The first term measures the entropy of the averaged predictive distribution across dropout samples, while the second term measures the average entropy within individual samples\. A higher BALD score indicates greater variation among predictive distributions at the answer\-token position, suggesting lower confidence in the answer conditioned on the fixed generated trajectory\.

Because BALD is an uncertainty score rather than a probability of correctness, we convert each token\-level score into a confidence score using a fixed monotone decreasing transformationϕ:ℝ→\[0,1\]\\phi:\\mathbb\{R\}\\rightarrow\[0,1\]and average across answer\-token positions:cBALD=1\|𝒚ans\|​∑j=1\|𝒚ans\|ϕ​\(BALDj\)\.c\_\{\\mathrm\{BALD\}\}=\\frac\{1\}\{\|\\bm\{y\}^\{\\text\{ans\}\}\|\}\\sum\_\{j=1\}^\{\|\\bm\{y\}^\{\\text\{ans\}\}\|\}\\phi\\\!\\left\(\\mathrm\{BALD\}\_\{j\}\\right\)\.The resulting estimator depends on the dropout rate, the layers in which dropout is enabled, the number of stochastic passes, and the choice ofϕ\\phi; these implementation details are provided in Appendix[B\.1](https://arxiv.org/html/2608.07827#A2.SS1)\.

Likep​\(True\)p\(\\mathrm\{True\}\),cBALDc\_\{\\mathrm\{BALD\}\}requires additional forward passes\. Together, these methods allow us to compare confidence derived directly from the original generation, self\-verification, and stochastic predictive disagreement\.

## 5Confidence Calibration

Calibration aims to align confidence estimates with empirical correctness\(Guoet al\.,[2017](https://arxiv.org/html/2608.07827#bib.bib7)\)\. A confidence estimatorc​\(𝒙,𝒚^\)c\(\\bm\{x\},\\hat\{\\bm\{y\}\}\)isperfectly calibratedifℙ​\(𝕀​\(𝒚^ans,𝒚∗\)=1∣c​\(𝒙,𝒚^\)=p\)=p\\mathbb\{P\}\\\!\\left\(\\mathbb\{I\}\(\\hat\{\\bm\{y\}\}^\{\\mathrm\{ans\}\},\\bm\{y\}^\{\*\}\)=1\\mid c\(\\bm\{x\},\\hat\{\\bm\{y\}\}\)=p\\right\)=pfor allp∈\[0,1\]p\\in\[0,1\]\. Sincec​\(𝒙,𝒚^\)c\(\\bm\{x\},\\hat\{\\bm\{y\}\}\)is typically continuous, this condition is evaluated empirically by partitioning predictions into confidence bins\.

Metrics:Qualitatively, reliability diagrams plot empirical accuracy against mean confidence across bins; perfect calibration lies on the diagonal\(Niculescu\-Mizil and Caruana,[2005](https://arxiv.org/html/2608.07827#bib.bib54)\)\. Quantitatively, Expected Calibration Error \(ECE\) and Maximum Calibration Error \(MCE\)\(Naeiniet al\.,[2015](https://arxiv.org/html/2608.07827#bib.bib55)\)measure the average and worst\-case discrepancies across these bins\. We defer results with Brier scores\(Brier and others,[1950](https://arxiv.org/html/2608.07827#bib.bib56)\)to Appendix[D](https://arxiv.org/html/2608.07827#A4)\.

Post\-hoc calibrationlearns a mappingffon a held\-out calibration set so thatf​\(c​\(𝒙,𝒚^\)\)f\(c\(\\bm\{x\},\\hat\{\\bm\{y\}\}\)\)better approximates the probability of answer correctness\. Following prior work\(Guoet al\.,[2017](https://arxiv.org/html/2608.07827#bib.bib7)\), we evaluate two simple and widely used approaches, isotonic regression and Platt scaling, and study their behavior across calibration\-set sizes, dataset difficulty, and cross\-dataset transfer\.Isotonic regression\(Zadrozny and Elkan,[2002](https://arxiv.org/html/2608.07827#bib.bib28)\)learns a flexible, non\-parametric monotonic mapping\. Because it requires only confidence scores and correctness labels, it can fit non\-linear calibration curves and is applied to all estimators in our study\.Platt scaling\(Platt and others,[1999](https://arxiv.org/html/2608.07827#bib.bib29)\)applies a parametric global adjustment to logits\. Since correctness labels are available only for final answers, we apply temperature scaling for answer\-token estimators and Platt scaling for the binary verification score\.

## 6Experiments

In this section, we evaluate confidence estimators and analyze the factors driving their calibration performance\. We further study post\-hoc calibration, including its data efficiency, sensitivity to dataset difficulty, and transfer across datasets\.

### 6\.1Experiment Setup

Model Selection:We focus on open\-source language models, including Llama\-3\.1\-8B\-Instruct, Llama\-3\.2\-3B\-Instruct\(Dubeyet al\.,[2024](https://arxiv.org/html/2608.07827#bib.bib60)\), Qwen3\-8B\-Thinking, Qwen3\-4B\-Instruct\(Yanget al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib61)\), and Deepseek\-R1\-distill\-llama\-8b\(Guoet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib62)\)\. This collection spans a range of model sizes, architectures, and includes instruction\-tuned, distilled, and reasoning models\.

Datasets:We evaluate on GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.07827#bib.bib25)\), GSMHard\(Gaoet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib26)\), and SVAMP\(Patelet al\.,[2021](https://arxiv.org/html/2608.07827#bib.bib27)\), three mathematical question answering benchmarks with varying difficulty and over two thousand evaluation examples in total\. These tasks elicit structured outputs with intermediate reasoning followed by a final numeric answer\(Weiet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib39)\), enabling verifiable answer extraction for evaluating correctness\.

Estimation:We use each model’s recommended chat template and report the decoding configuration in Table[6](https://arxiv.org/html/2608.07827#A3.T6)\. Forp​\(True\)p\(\\text\{True\}\), we follow the re\-prompting template\(Kadavathet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib8)\)and identify the token indices corresponding toTrueandFalsefor each model\. For MC Dropout estimation, we select the dropout rate separately for each model through grid search and apply it to all supported dropout modules during stochastic forward passes\. Exploring layer\-specific dropout configurations is left for future work\.

### 6\.2Confidence Estimation

ModelDatasetAcc\.Seq AvgSeq JointAns AvgAns Jointp​\(True\)p\(\\mathrm\{True\}\)cBALDc\_\{\\mathrm\{BALD\}\}ECEMCEECEMCEECEMCEECEMCEECEMCEECEMCELlama\-3\.1\-8BGSM8K0\.8240\.0530\.4810\.1170\.1950\.1660\.5290\.1650\.4690\.0970\.6250\.1490\.667GSMHard0\.3500\.4570\.6000\.3200\.4140\.6220\.7430\.6150\.7530\.3330\.6270\.1520\.424SVAMP0\.8270\.0570\.3090\.1700\.3320\.1690\.7970\.1710\.7650\.0990\.4500\.0950\.633Llama\-3\.2\-3BGSM8K0\.7750\.1240\.5870\.0920\.3640\.2180\.6350\.2180\.6230\.0550\.3040\.1080\.389GSMHard0\.2580\.6360\.7730\.5970\.6760\.7250\.8570\.7220\.7680\.2510\.4060\.0780\.202SVAMP0\.8200\.0680\.7830\.0350\.6760\.1750\.8400\.1750\.8370\.1040\.4230\.0970\.610DeepSeek\-R1\-8BGSM8K0\.6500\.2530\.3840\.2190\.4760\.3040\.3610\.3020\.3560\.3150\.4840\.1680\.732GSMHard0\.3200\.5880\.6460\.5560\.6110\.6370\.6740\.6310\.6900\.6340\.6730\.1020\.606SVAMP0\.7280\.1640\.7840\.1270\.5250\.2440\.6170\.2420\.6060\.2410\.5360\.1740\.979Qwen3\-8B\-ThinkingGSM8K0\.9580\.0060\.0700\.0180\.0270\.0420\.0420\.0420\.0420\.0350\.8290\.0430\.545GSMHard0\.8100\.1440\.7810\.1280\.7930\.1900\.1900\.1900\.1900\.1830\.3730\.0280\.365SVAMP0\.9720\.0170\.0170\.0330\.0340\.0280\.0280\.0280\.0280\.0200\.3480\.0260\.133Qwen3\-4B\-InstructGSM8K0\.9380\.0350\.4780\.0280\.7900\.0620\.0620\.0620\.0620\.0620\.6520\.0610\.650GSMHard0\.7010\.2660\.7240\.2560\.7870\.2980\.2990\.2980\.2990\.2850\.9850\.1880\.451SVAMP0\.9430\.0320\.3660\.0250\.2670\.0570\.0570\.0570\.0570\.0580\.9990\.0610\.313Table 2:Confidence estimation results using token probabilities\.We report accuracy \(Acc\.\) and calibration errors \(ECE, MCE; lower is better\)\. For single\-pass methods, sequence\-based estimators achieve lower ECE than answer\-based estimators across all model–dataset pairs\. For multi\-pass methods, MC Dropout\-basedcBALDc\_\{\\mathrm\{BALD\}\}often achieves lower ECE thanp​\(True\)p\(\\mathrm\{True\}\), particularly on GSMHard\.Boldindicates the lowest ECE within each row\.We investigate the effectiveness of token probability in confidence estimation\. Table[2](https://arxiv.org/html/2608.07827#S6.T2)summarizes results for single\-pass and multi\-pass confidence estimators together with model accuracy on each dataset\. Full results, including Brier scores and bootstrap confidence intervals, are reported in Table[7](https://arxiv.org/html/2608.07827#A4.T7)in the appendix\.

Finding 1 \(Trajectory\-level Confidence\):Calibrated confidence arises from aggregating weak but consistent probability differences across the reasoning trajectory\.

Our analysis of single\-pass estimators considers two design choices: the subset of generated tokens used for estimation and the method used to aggregate their probabilities\. Across all model–dataset pairs, sequence\-based estimators achieve lower ECE than answer\-only estimators, although the confidence intervals overlap in some settings\.

Using probabilities from the full generated sequence substantially improves calibration\. In several settings, sequence\-based estimators also achieve lower ECE than the more computationally expensive multi\-pass estimators\. To understand this, we examine the trajectory of token probabilities throughout generation\. In Figure[1](https://arxiv.org/html/2608.07827#S6.F1)\(a\), we plot the moving average of token probabilities as a function of normalized token position\. This can be viewed as aSeq Avgestimator computed with an increasing window size\. The curves initially overlap because many responses share similar opening phrases\. As generation progresses, however, the curves separate: average probabilities tend to increase for responses with correct final answers and decrease for those with incorrect answers\.

This pattern suggests that the useful signal does not necessarily arise from any single token\. Rather, small probability differences accumulate across the reasoning trajectory and become more apparent after sequence\-level aggregation\.

Within each token subset, length\-normalized joint probability often outperforms arithmetic averaging, particularly on GSMHard\. Because the joint probability is more sensitive to low\-probability tokens, this result suggests that occasional low\-probability steps can provide useful evidence of answer incorrectness\.

Actionable takeaway\.Full\-trajectory aggregation provides a strong single\-pass default, suggesting that useful confidence signals are distributed across the reasoning trajectory\. Because global aggregation obscures where these signals arise, local\-window methods offer a natural extension\. Recent work similarly exploits localized confidence patterns, but primarily to improve test\-time reasoning efficiency rather than confidence calibration\(Fuet al\.,[2026](https://arxiv.org/html/2608.07827#bib.bib87)\)\. Extending such local\-window approaches to produce calibrated estimates of correctness is a promising direction for future work\.

![Refer to caption](https://arxiv.org/html/2608.07827v1/x1.png)Figure 1:Confidence estimation analysis\.\(a\)Moving average of token probabilities across the generated sequence\.Confidence\-relevant information arises from the accumulation of small token\-level probability differences across the reasoning trajectory\. \(b\)BALD decomposition under MC Dropout\.Predictive entropy increases and stabilizes after roughly five samples, indicating that MC Dropout primarily captures distributional uncertainty across stochastic inferences\. \(c\)In\-situ verification\.In\-situp​\(True\)p\(\\mathrm\{True\}\)achieves calibration comparable to standardp​\(True\)p\(\\mathrm\{True\}\)while avoiding full re\-prompting\. Results are from Llama\-3\.1\-8B, with \(a\) evaluated on GSM8K\.Finding 2 \(Distributional Uncertainty\):MC Dropout captures uncertainty from variation in predictive distributions, but requires dropout\-rate selection for each model–dataset pair and multiple stochastic passes\.

We next compare two multi\-pass confidence estimators:p​\(True\)p\(\\mathrm\{True\}\), which re\-prompts the model to verify its generated answer, andcBALDc\_\{\\mathrm\{BALD\}\}, which measures dropout\-induced variation across token predictive distributions\. As shown in Table[2](https://arxiv.org/html/2608.07827#S6.T2),cBALDc\_\{\\mathrm\{BALD\}\}often achieves lower calibration error thanp​\(True\)p\(\\mathrm\{True\}\), with its advantage being particularly pronounced on GSMHard\. However, these gains come with substantial computational and tuning costs\.

Both methods introduce additional computational overhead compared to single\-pass estimators\. Forp​\(True\)p\(\\mathrm\{True\}\), the extra cost is roughly proportional to the length of the re\-encoded question–answer pair\. ForcBALDc\_\{\\mathrm\{BALD\}\}, inference cost grows approximately linearly with the number of stochastic passesTT\. Moreover, its performance is sensitive to the dropout rate, which we select separately for each model–dataset pair\. The search space, criteria, and the selected dropout rates are reported in Appendix[B\.2](https://arxiv.org/html/2608.07827#A2.SS2)\.

To examine the trade\-off in the number of stochastic passes, Figure[1](https://arxiv.org/html/2608.07827#S6.F1)\(b\) plots the two terms of Equation \([1](https://arxiv.org/html/2608.07827#S4.E1)\) asTTincreases\. The expected conditional entropy,𝔼θ​\[H​\(yj∣x,y<j,θ\)\]\\mathbb\{E\}\_\{\\theta\}\[H\(y\_\{j\}\\mid x,y\_\{<j\},\\theta\)\], remains relatively stable, indicating that individual dropout\-induced forward passes produce similarly sharp distributions\. In contrast, the predictive entropy,H​\(yj∣x,y<j\)H\(y\_\{j\}\\mid x,y\_\{<j\}\), increases withTTand stabilizes after roughly five samples, resulting in about a five\-fold overhead relative to a single forward pass for a reliable estimation\.

Actionable takeaway\.MC Dropout is most appropriate when its calibration gains justify substantial tuning and inference costs\. Recent approaches reduce repeated inference\(Gaoet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib52)\), suggesting a promising direction toward more efficient confidence estimation\.

Finding 3 \(In\-situ Self\-verification\):In\-situ self\-verification achieves calibration comparable to standardp​\(True\)p\(\\mathrm\{True\}\)with substantially lower token\-processing overhead\.

The performance ofp​\(True\)p\(\\mathrm\{True\}\)raises a natural question: why is it often better calibrated thanAns Avg, despite both relying on only a few token probabilities? One possible explanation is thatp​\(True\)p\(\\mathrm\{True\}\)reframes confidence estimation as binary self\-verification, a setting in which language models have been shown to provide informative confidence signals\(Kadavathet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib8)\)\.

Motivated by this observation, we evaluate anin\-situvariant that appends the verification query to the original response rather than re\-prompting the model from scratch\. The resulting verifier remains conditioned on the original prompt and generated trajectory, including the system prompt, demonstration, question, reasoning, and final answer\. The prompt formats are compared in Figure[4](https://arxiv.org/html/2608.07827#A4.F4)\.

As shown in Figure[1](https://arxiv.org/html/2608.07827#S6.F1)\(c\), in\-situ verification achieves ECE comparable to standardp​\(True\)p\(\\mathrm\{True\}\)\. By reusing the original generation, it avoids reprocessing the full question–response sequence and reduces token\-processing overhead by 88% on average across all settings\. Full results are reported in Table[11](https://arxiv.org/html/2608.07827#A4.T11)\. This saving is particularly relevant for long outputs\(Baiet al\.,[2025](https://arxiv.org/html/2608.07827#bib.bib66)\)\.

Actionable takeaway\.When the original generation state can be retained, in\-situ verification provides a substantially cheaper alternative to standardp​\(True\)p\(\\mathrm\{True\}\), especially for long outputs\.

ModelDatasetIsotonic RegressionPlatt ScalingAns AvgAns Jointp​\(True\)p\(\\mathrm\{True\}\)ttAns AvgAns Joint\(a,b\)\(a,b\)p​\(True\)p\(\\mathrm\{True\}\)Llama\-3\.1\-8BGSM8K0\.056 \(\+66%\)0\.054 \(\+67%\)0\.058 \(\+40%\)1\.820\.080 \(\+52%\)0\.080 \(\+51%\)\(0\.53, 0\.48\)0\.059 \(\+39%\)GSMHard0\.064 \(\+90%\)0\.065 \(\+89%\)0\.057 \(\+83%\)2\.370\.091 \(\+85%\)0\.045 \(\+93%\)\(0\.57, \-1\.74\)0\.087 \(\+74%\)SVAMP0\.020 \(\+88%\)0\.022 \(\+87%\)0\.045 \(\+54%\)1\.910\.084 \(\+50%\)0\.083 \(\+52%\)\(0\.54, 0\.43\)0\.039 \(\+60%\)Llama\-3\.2\-3BGSM8K0\.074 \(\+66%\)0\.075 \(\+65%\)0\.076 \(\-39%\)1\.740\.104 \(\+52%\)0\.103 \(\+53%\)\(0\.65, 1\.07\)0\.044 \(\+20%\)GSMHard0\.027 \(\+96%\)0\.028 \(\+96%\)0\.031 \(\+88%\)2\.270\.105 \(\+85%\)0\.057 \(\+92%\)\(0\.79, \-1\.36\)0\.047 \(\+81%\)SVAMP0\.036 \(\+79%\)0\.036 \(\+80%\)0\.055 \(\+48%\)1\.790\.044 \(\+75%\)0\.041 \(\+76%\)\(0\.61, 0\.80\)0\.029 \(\+72%\)DeepSeek\-R1\-8BGSM8K0\.012 \(\+96%\)0\.013 \(\+96%\)0\.017 \(\+95%\)1\.680\.071 \(\+77%\)0\.070 \(\+77%\)\(0\.054, 0\.46\)0\.016 \(\+95%\)GSMHard0\.068 \(\+89%\)0\.071 \(\+89%\)0\.066 \(\+90%\)2\.150\.100 \(\+84%\)0\.082 \(\+87%\)\(0\.065, \-1\.29\)0\.065 \(\+90%\)SVAMP0\.016 \(\+93%\)0\.024 \(\+90%\)0\.034 \(\+86%\)1\.670\.061 \(\+75%\)0\.066 \(\+73%\)\(0\.022, 0\.83\)0\.013 \(\+95%\)Qwen3\-8B\-ThinkingGSM8K0\.015 \(\+63%\)0\.015 \(\+63%\)0\.015 \(\+58%\)3\.190\.030 \(\+29%\)0\.030 \(\+28%\)\(0\.39, 1\.54\)0\.016 \(\+54%\)GSMHard0\.034 \(\+82%\)0\.033 \(\+83%\)0\.049 \(\+73%\)3\.620\.114 \(\+40%\)0\.114 \(\+40%\)\(0\.52, \-1\.38\)0\.078 \(\+57%\)SVAMP0\.003 \(\+91%\)0\.003 \(\+91%\)0\.007 \(\+66%\)3\.220\.022 \(\+20%\)0\.022 \(\+21%\)\(0\.46, 1\.22\)0\.004 \(\+79%\)Qwen3\-4B\-InstructGSM8K0\.019 \(\+69%\)0\.019 \(\+69%\)0\.020 \(\+68%\)3\.900\.044 \(\+29%\)0\.044 \(\+29%\)\(0\.18, 1\.70\)0\.019 \(\+70%\)GSMHard0\.077 \(\+74%\)0\.077 \(\+74%\)0\.091 \(\+68%\)4\.480\.131 \(\+56%\)0\.144 \(\+52%\)\(0\.11, \-0\.18\)0\.092 \(\+68%\)SVAMP0\.035 \(\+38%\)0\.035 \(\+38%\)0\.032 \(\+44%\)3\.880\.043 \(\+24%\)0\.044 \(\+22%\)\(\-0\.30, 6\.02\)0\.032 \(\+45%\)

Table 3:Post\-hoc calibration results\.We report calibrated ECE forAns Avg,Ans Joint, andp​\(True\)p\(\\mathrm\{True\}\)after isotonic regression and Platt scaling, with relative improvements over uncalibrated estimators shown in parentheses\. For Platt scaling, we also report the learned temperaturettfor answer\-based estimators and learned parameters\(a,b\)\(a,b\)forp​\(True\)p\(\\mathrm\{True\}\)\. For each estimator and dataset,boldindicates the lower ECE between the two calibration methods\.
### 6\.3Confidence Calibration

We next evaluate post\-hoc calibration across confidence estimators\. We consider isotonic regression and Platt scaling\. Table[3](https://arxiv.org/html/2608.07827#S6.T3)reports the calibrated ECE forAns Avg,Ans Joint, andp​\(True\)p\(\\mathrm\{True\}\), with relative improvements over the corresponding uncalibrated estimators shown in parentheses\. Complete results with confidence intervals are deferred to Table[8](https://arxiv.org/html/2608.07827#A4.T8)in the appendix\.

Both methods generally reduce calibration error, including the strongly overconfident answer\-token estimators\. Isotonic regression often produces the largest reductions\. As illustrated in Figure[2](https://arxiv.org/html/2608.07827#S6.F2), its flexible monotonic mapping can correct nonlinear reliability curves that cannot be captured by the global logit rescaling of temperature scaling\. These results show that high raw ECE does not necessarily preclude effective post\-hoc correction when labeled calibration data are available\.

Finding 4 \(Difficulty\-dependent Calibration and Transfer\):Calibration can be more data\-efficient on harder datasets under random sampling, but learned mappings transfer asymmetrically across datasets and models\.

We first study how calibration\-set size affects performance\. Prior work often characterizes calibration\-data requirements in terms of the calibration method\(Guoet al\.,[2017](https://arxiv.org/html/2608.07827#bib.bib7)\)or label noise\(Zhaoet al\.,[2020](https://arxiv.org/html/2608.07827#bib.bib73)\)\. We complement these analyses by examining the role of dataset difficulty\.

![Refer to caption](https://arxiv.org/html/2608.07827v1/x2.png)Figure 2:Post\-hoc confidence calibration analysis\.Reliability diagrams for Ans Avg: both \(a\)isotonic regressionand \(b\)Platt scalingimprove calibration, with isotonic regression fitting a flexible monotonic mapping and Platt scaling applying a global parametric correction\. \(c\)Effect of calibration set size on Ans Avg\.Calibration error decreases quickly with calibration set size, with 50 examples already yielding substantial improvements across datasets\. Results are from Llama\-3\.1\-8B, with \(a\) and \(b\) evaluated on GSMHard\.Figure[2](https://arxiv.org/html/2608.07827#S6.F2)\(c\) plots the ECE ofAns Avgafter isotonic regression and Platt scaling as the calibration set size varies\. Calibration examples are randomly sampled from the calibration split\. Overall, calibration error decreases rapidly with more calibration data, with 50 examples already yielding substantial improvements across datasets\. However, GSMHard benefits from calibration with fewer examples, whereas GSM8K and SVAMP require larger calibration sets for stable improvement\.

This is because the model is more accurate on GSM8K and SVAMP: under random sampling, small calibration sets contain many high\-confidence correct examples, making overconfidence harder to estimate\. Harder datasets, however, expose high\-confidence errors more frequently, allowing effective calibration with fewer examples\. This suggests an important direction for future work: calibration\-set construction need not rely on uniform random sampling, and weighted sampling strategies that emphasize incorrect validation examples may improve calibration data efficiency\.

Calibration Source→\\rightarrowEvaluation TargetIsotonicPlattLlama: GSM8K→\\rightarrowLlama: GSMHard−\.209\\bm\{\-\.209\}−\.083\\bm\{\-\.083\}Llama: GSMHard→\\rightarrowLlama: GSM8K\+\.189\+\.189\+\.213\+\.213Qwen: GSM8K→\\rightarrowQwen: GSMHard−\.034\\bm\{\-\.034\}−\.054\\bm\{\-\.054\}Qwen: GSMHard→\\rightarrowQwen: GSM8K\+\.242\+\.242−\.011\-\.011Llama: GSM8K→\\rightarrowQwen:GSM8K−\.003\-\.003−\.002\-\.002Qwen: GSM8K→\\rightarrowLlama:GSM8K\+\.458\+\.458\+\.658\+\.658Llama: GSMHard→\\rightarrowQwen:GSMHard−\.185\\bm\{\-\.185\}−\.003\\bm\{\-\.003\}Qwen: GSMHard→\\rightarrowLlama:GSMHard−\.527\\bm\{\-\.527\}−\.318\\bm\{\-\.318\}Table 4:Off\-domain calibration transfer forAns Avg\.Each entry reportsΔ​ECE\\Delta\\mathrm\{ECE\}relative to the uncalibrated target; negative values indicate improvement\. Full results are included in Table[10](https://arxiv.org/html/2608.07827#A4.T10)in the appendix\.We further examine calibration transfer across datasets and models\. ForAns Avg, we fit calibrators using Llama\-3\.1\-8B\-Instruct and Qwen3\-4B\-Instruct on either GSM8K or GSMHard, then apply each mapping unchanged to a different dataset or model\. Table[4](https://arxiv.org/html/2608.07827#S6.T4)reports the change in target ECE relative to the uncalibrated estimator\.

Cross\-dataset transfer is strongly asymmetric: mappings learned on GSM8K consistently improve calibration on GSMHard, whereas the reverse direction generally worsens it\. Cross\-model transfer is similarly target\-dependent\. Both directions improve calibration on GSMHard, while transfer on GSM8K is asymmetric and can severely degrade calibration\. These results suggest that calibration mappings depend on both model\-specific score distributions and dataset difficulty, limiting their reliability under transfer\.

Actionable takeaway\.Calibration data should match the target model and difficulty level whenever possible\. When the model is known to be overconfident and calibration labels are limited, difficulty\-aware sampling may improve efficiency by ensuring sufficient coverage of model errors\.

## 7Conclusion

In this work, we study token\-probability\-based confidence estimation and calibration for mathematical question answering\. We compare single\-pass and multi\-pass confidence estimators, then evaluate post\-hoc calibration\. Our findings show that isotonic regression and Platt scaling generally reduce in\-domain calibration error, although their data efficiency and transferability depend on the model and dataset\. Overall, our results show that token selection, aggregation, inference cost, and calibration data all affect the quality and practicality of token\-probability\-based confidence estimates\.

## Limitations

Our study evaluates open\-weight LLMs for which token probabilities and dropout controls are accessible, limiting direct evaluation of proprietary models\. Our study focuses on mathematical question answering tasks with verifiable final answers, but open\-ended dialogue requires handling more ambiguous and task\-dependent notions of correctness\. We consider two classical post\-hoc calibration families, isotonic regression and parametric scaling\. Although they reveal important challenges in calibration efficiency and transfer, more expressive calibration methods may behave differently\.

The MC Dropout results also depend on several design choices: conditioning on a fixed reasoning trajectory, measuring uncertainty only at answer\-token positions, and selecting the dropout rate, enabled layers, number of stochastic passes, and normalization function\. This formulation reduces computation but does not capture uncertainty over alternative reasoning trajectories\. Correctness evaluation relies on regex\-based answer extraction using a fixed delimiter; malformed or nonconforming outputs may therefore introduce extraction errors\. Finally, our experiments use controlled prompts, whereas real deployments may involve long contexts, multi\-turn histories, and external evidence, whose effects on token\-probability\-based confidence estimation remain open\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p1.1),[§3\.1](https://arxiv.org/html/2608.07827#S3.SS1.p1.1)\.
- P\. Atighehchian, F\. Branchaud\-Charron, J\. Freyberg, R\. Pardinas, L\. Schell, and G\. Pearse \(2022\)BAAL, a Bayesian active learning library\.Note:[https://github\.com/baal\-org/baal/](https://github.com/baal-org/baal/)Cited by:[Appendix E](https://arxiv.org/html/2608.07827#A5.p1.1),[§4\.3](https://arxiv.org/html/2608.07827#S4.SS3.p4.6)\.
- Y\. Bai, J\. Zhang, X\. Lv, L\. Zheng, S\. Zhu, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2025\)Longwriter: unleashing 10,000\+ word generation from long context LLMs\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6\.2](https://arxiv.org/html/2608.07827#S6.SS2.p16.1)\.
- D\. Bartels, D\. Gupta, and D\. Demner\-Fushman \(2025\)Can large language models accurately generate answer keys for health\-related questions?\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p1.1)\.
- G\. Brieret al\.\(1950\)Verification of forecasts expressed in terms of probability\.Monthly Weather Review\.Cited by:[§5](https://arxiv.org/html/2608.07827#S5.p2.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§3\.1](https://arxiv.org/html/2608.07827#S3.SS1.p1.1)\.
- H\. Cai, S\. Zhao, L\. Zhang, X\. Shen, Q\. Xu, W\. Shen, Z\. Wen, and T\. Ban \(2025\)Unilaw\-r1: a large language model for legal reasoning with reinforcement learning and iterative inference\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p1.1)\.
- P\. Chhikara \(2025\)Mind the confidence gap: overconfidence, calibration, and distractor effects in large language models\.InTransactions on Machine Learning Research \(TMLR\),Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p2.1),[§2](https://arxiv.org/html/2608.07827#S2.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Appendix E](https://arxiv.org/html/2608.07827#A5.p1.1),[§1](https://arxiv.org/html/2608.07827#S1.p3.1),[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p1.1),[§6\.1](https://arxiv.org/html/2608.07827#S6.SS1.p2.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§6\.1](https://arxiv.org/html/2608.07827#S6.SS1.p1.1)\.
- T\. Ficsor and G\. Berend \(2025\)SUE: sparsity\-based uncertainty estimation via sparse dictionary learning\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p3.1)\.
- Y\. Fu, X\. Wang, H\. Zhang, Y\. Tian, and J\. Zhao \(2026\)Deep think with confidence\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6\.2](https://arxiv.org/html/2608.07827#S6.SS2.p7.1)\.
- Y\. Gal and Z\. Ghahramani \(2016\)Dropout as a bayesian approximation: representing model uncertainty in deep learning\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§B\.1](https://arxiv.org/html/2608.07827#A2.SS1.p1.4),[§1](https://arxiv.org/html/2608.07827#S1.p3.1),[§2](https://arxiv.org/html/2608.07827#S2.p3.1),[§4\.3](https://arxiv.org/html/2608.07827#S4.SS3.p3.1)\.
- Y\. Galet al\.\(2017\)Uncertainty in Deep Learning\.University of Cambridge Cambridge\.Cited by:[§4\.3](https://arxiv.org/html/2608.07827#S4.SS3.p3.1)\.
- L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)PAL: program\-aided language models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[Appendix E](https://arxiv.org/html/2608.07827#A5.p1.1),[§1](https://arxiv.org/html/2608.07827#S1.p3.1),[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p1.1),[§6\.1](https://arxiv.org/html/2608.07827#S6.SS1.p2.1)\.
- S\. Gao, T\. Gong, Z\. Lin, R\. Xu, H\. Zhou, and J\. Li \(2025\)Flue: streamlined uncertainty estimation for large language models\.InAAAI Conference on Artificial Intelligence,Cited by:[§4\.3](https://arxiv.org/html/2608.07827#S4.SS3.p3.1),[§6\.2](https://arxiv.org/html/2608.07827#S6.SS2.p12.1)\.
- J\. Geng, F\. Cai, Y\. Wang, H\. Koeppl, P\. Nakov, and I\. Gurevych \(2024\)A survey of confidence estimation and calibration in large language models\.InProceedings of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics \(NAACL\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InInternational Conference on Machine Learning \(ICML\),Cited by:[Appendix C](https://arxiv.org/html/2608.07827#A3.p6.1),[§4\.1](https://arxiv.org/html/2608.07827#S4.SS1.p1.5),[§5](https://arxiv.org/html/2608.07827#S5.p1.4),[§5](https://arxiv.org/html/2608.07827#S5.p3.2),[§6\.3](https://arxiv.org/html/2608.07827#S6.SS3.p4.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§6\.1](https://arxiv.org/html/2608.07827#S6.SS1.p1.1)\.
- N\. Gupta, H\. Narasimhan, W\. Jitkrittum, A\. S\. Rawat, A\. K\. Menon, and S\. Kumar \(2024\)Language model cascades: token\-level uncertainty and beyond\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p3.1)\.
- N\. Houlsby, F\. Huszár, Z\. Ghahramani, and M\. Lengyel \(2011\)Bayesian active learning for classification and preference learning\.arXiv preprint arXiv:1112\.5745\.Cited by:[§4\.3](https://arxiv.org/html/2608.07827#S4.SS3.p4.6),[Table 1](https://arxiv.org/html/2608.07827#S4.T1)\.
- Y\. Huang, J\. Song, Z\. Wang, S\. Zhao, H\. Chen, F\. Juefei\-Xu, and L\. Ma \(2023\)Look before you leap: an exploratory study of uncertainty measurement for large language models\.arXiv preprint arXiv:2307\.10236\.Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p1.1),[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p3.1),[§4\.2](https://arxiv.org/html/2608.07827#S4.SS2.p1.2)\.
- Y\. Huang, Y\. Liu, R\. Thirukovalluru, A\. Cohan, and B\. Dhingra \(2024\)Calibrating long\-form generations from large language models\.InFindings of the Association for Computational Linguistics \(EMNLP Findings\),Cited by:[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p3.1)\.
- Z\. Jiang, F\. F\. Xu, J\. Araki, and G\. Neubig \(2020\)How can we know what language models know?\.Transactions of the Association for Computational Linguistics\.Cited by:[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p3.1),[§4\.2](https://arxiv.org/html/2608.07827#S4.SS2.p1.2)\.
- A\. Joshi, A\. Ahmad, and A\. Modi \(2025\)Calibration across layers: understanding calibration evolution in llms\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p4.1)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson,et al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p2.1),[§1](https://arxiv.org/html/2608.07827#S1.p3.1),[§2](https://arxiv.org/html/2608.07827#S2.p2.1),[§4\.3](https://arxiv.org/html/2608.07827#S4.SS3.p2.4),[Table 1](https://arxiv.org/html/2608.07827#S4.T1),[§6\.1](https://arxiv.org/html/2608.07827#S6.SS1.p3.1),[§6\.2](https://arxiv.org/html/2608.07827#S6.SS2.p14.2)\.
- L\. Kuhn, Y\. Gal, and S\. Farquhar \(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p1.1)\.
- Y\. Li, H\. Su, X\. Shen, W\. Li, Z\. Cao, and S\. Niu \(2017\)DailyDialog: a manually labelled multi\-turn dialogue dataset\.InProceedings of the International Joint Conference on Natural Language Processing \(IJCNLP\),Cited by:[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p1.1)\.
- Y\. Li, M\. Xiong, J\. Wu, and B\. Hooi \(2025\)Conftuner: training large language models to express their confidence verbally\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p4.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2024a\)Teaching models to express their uncertainty in words\.InTransactions on Machine Learning Research \(TMLR\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p4.1)\.
- Z\. Lin, S\. Trivedi, and J\. Sun \(2024b\)Contextualized sequence likelihood: enhanced confidence scores for natural language generation\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p1.1),[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p3.1),[§4\.3](https://arxiv.org/html/2608.07827#S4.SS3.p2.4)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p1.1)\.
- K\. Luo, Q\. Huang, C\. Jiang, and Y\. Feng \(2025\)Automating legal interpretation with LLMs: retrieval, generation, and evaluation\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p1.1)\.
- Q\. Lyu, K\. Shridhar, C\. Malaviya, L\. Zhang, Y\. Elazar, N\. Tandon, M\. Apidianaki, M\. Sachan, and C\. Callison\-Burch \(2025\)Calibrating large language models with sample consistency\.InAAAI Conference on Artificial Intelligence,Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p1.1),[§4\.3](https://arxiv.org/html/2608.07827#S4.SS3.p2.4)\.
- M\. Mahaut, L\. Aina, P\. Czarnowska, M\. Hardalov, T\. Müller, and L\. Màrquez \(2024\)Factual confidence of LLMs: on reliability and robustness of current estimators\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§4\.3](https://arxiv.org/html/2608.07827#S4.SS3.p2.4)\.
- A\. Malinin and M\. Gales \(2021\)Uncertainty estimation in autoregressive structured prediction\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4\.2](https://arxiv.org/html/2608.07827#S4.SS2.p1.2)\.
- J\. Mendonça, A\. Lavie, and I\. Trancoso \(2024\)On the benchmarking of LLMs for open\-domain dialogue evaluation\.InProceedings of the 6th Workshop on NLP for Conversational AI \(NLP4ConvAI 2024\),Cited by:[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p1.1)\.
- S\. J\. Mielke, A\. Szlam, E\. Dinan, and Y\. Boureau \(2022\)Reducing conversational agents’ overconfidence through linguistic calibration\.Transactions of the Association for Computational Linguistics\.Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p2.1),[§2](https://arxiv.org/html/2608.07827#S2.p1.1)\.
- M\. Mora\-Cross and S\. Calderon\-Ramirez \(2024\)Uncertainty estimation in large language models to support biodiversity conservation\.InProceedings of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Industry Track \(NAACL Industry Track\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p3.1)\.
- M\. P\. Naeini, G\. Cooper, and M\. Hauskrecht \(2015\)Obtaining well calibrated probabilities using Bayesian binning\.InAAAI Conference on Artificial Intelligence,Cited by:[§5](https://arxiv.org/html/2608.07827#S5.p2.1)\.
- A\. Niculescu\-Mizil and R\. Caruana \(2005\)Predicting good probabilities with supervised learning\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§5](https://arxiv.org/html/2608.07827#S5.p2.1)\.
- H\. Orgad, M\. Toker, Z\. Gekhman, R\. Reichart, I\. Szpektor, H\. Kotek, and Y\. Belinkov \(2025\)LLMs know more than they show: on the intrinsic representation of llm hallucinations\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p3.1),[§4\.2](https://arxiv.org/html/2608.07827#S4.SS2.p1.2),[§4\.3](https://arxiv.org/html/2608.07827#S4.SS3.p2.4)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in Neural Information Processing Systems \(NeurIPS\)\.Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p2.1)\.
- A\. Patel, S\. Bhattamishra, and N\. Goyal \(2021\)Are NLP models really able to solve simple math word problems?\.InProceedings of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics \(NAACL\),Cited by:[Appendix E](https://arxiv.org/html/2608.07827#A5.p1.1),[§1](https://arxiv.org/html/2608.07827#S1.p3.1),[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p1.1),[§6\.1](https://arxiv.org/html/2608.07827#S6.SS1.p2.1)\.
- N\. Peinelt, D\. Nguyen, and M\. Liakata \(2020\)tBERT: topic models and BERT joining forces for semantic similarity detection\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p2.6)\.
- E\. Perez, S\. Huang, F\. Song, T\. Cai, R\. Ring, J\. Aslanides, A\. Glaese, N\. McAleese, and G\. Irving \(2022\)Red teaming language models with language models\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p2.6)\.
- J\. Plattet al\.\(1999\)Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods\.Advances in Large Margin Classifiers\.Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p3.1),[§2](https://arxiv.org/html/2608.07827#S2.p4.1),[§5](https://arxiv.org/html/2608.07827#S5.p3.2)\.
- F\. Ravenda, S\. A\. Bahrainian, A\. Raballo, A\. Mira, and N\. Kando \(2025\)Are LLMs effective psychological assessors? leveraging adaptive RAG for interpretable mental health screening through psychometric practice\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p1.1)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano \(2020\)Learning to summarize with human feedback\.Advances in Neural Information Processing Systems \(NeurIPS\)\.Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p2.1)\.
- A\. Stolfo, B\. Wu, W\. Gurnee, Y\. Belinkov, X\. Song, M\. Sachan, and N\. Nanda \(2024\)Confidence regulation neurons in language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p4.1)\.
- H\. Tan, W\. Yang, J\. Zhang, X\. Chen, R\. Tang, D\. Su, J\. Wang, Y\. Wang, F\. Sun, and X\. Cheng \(2026\)BaseCal: unsupervised confidence calibration via base model signals\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p4.1)\.
- G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.\(2023\)Gemini: a family of highly capable multimodal models\.arXiv preprint arXiv:2312\.11805\.Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§3\.1](https://arxiv.org/html/2608.07827#S3.SS1.p1.1)\.
- D\. Ulmer, M\. Gubri, H\. Lee, S\. Yun, and S\. J\. Oh \(2024\)Calibrating large language models using their generations only\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p4.1)\.
- A\. Vaswani \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§3\.1](https://arxiv.org/html/2608.07827#S3.SS1.p1.1)\.
- A\. Vazhentsev, A\. Tsvigun, R\. Vashurin, S\. Petrakov, D\. Vasilev, M\. Panov, A\. Panchenko, and A\. Shelmanov \(2023\)Efficient out\-of\-domain detection for sequence to sequence models\.InFindings of the Association for Computational Linguistics \(ACL Findings\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in Neural Information Processing Systems \(NeurIPS\)\.Cited by:[§3\.2](https://arxiv.org/html/2608.07827#S3.SS2.p1.1),[§6\.1](https://arxiv.org/html/2608.07827#S6.SS1.p2.1)\.
- S\. Wu, O\. Irsoy, S\. Lu, V\. Dabravolski, M\. Dredze, S\. Gehrmann, P\. Kambadur, D\. Rosenberg, and G\. Mann \(2023\)BloombergGPT: a large language model for finance\.arXiv preprint arXiv:2303\.17564\.Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p1.1)\.
- Y\. Wu, M\. Schuster, Z\. Chen, Q\. V\. Le, M\. Norouzi, W\. Macherey, M\. Krikun, Y\. Cao, Q\. Gao, K\. Macherey,et al\.\(2016\)Google’s neural machine translation system: bridging the gap between human and machine translation\.arXiv preprint arXiv:1609\.08144\.Cited by:[§4\.2](https://arxiv.org/html/2608.07827#S4.SS2.p1.2)\.
- M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. Hooi \(2024\)Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§6\.1](https://arxiv.org/html/2608.07827#S6.SS1.p1.1)\.
- X\. Yu, Z\. Chen, and Y\. Lu \(2023\)Harnessing LLMs for temporal data\-a study on explainable financial time series forecasting\.InProceedings of the Conference on Empirical Methods in Natural Language Processing: Industry Track \(EMNLP Industry Track\),Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p1.1)\.
- P\. Zablotskaia, D\. Phan, J\. Maynez, S\. Narayan, J\. Ren, and J\. Liu \(2023\)On uncertainty calibration and selective generation in probabilistic neural summarization: a benchmark study\.InFindings of the Association for Computational Linguistics \(EMNLP Findings\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p3.1)\.
- B\. Zadrozny and C\. Elkan \(2002\)Transforming classifier scores into accurate multiclass probability estimates\.InProceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining \(KDD\),Cited by:[§1](https://arxiv.org/html/2608.07827#S1.p3.1),[§2](https://arxiv.org/html/2608.07827#S2.p4.1),[§5](https://arxiv.org/html/2608.07827#S5.p3.2)\.
- T\. Zhang, H\. Shi, Y\. Wang, H\. Wang, X\. He, Z\. Li, H\. Chen, L\. Han, K\. Xu, H\. Zhang,et al\.\(2026\)TokUR: token\-level uncertainty estimation for large language model reasoning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2608.07827#S2.p3.1),[§4\.3](https://arxiv.org/html/2608.07827#S4.SS3.p3.1)\.
- Y\. Zhao, J\. Chen, and S\. Oymak \(2020\)On the role of dataset quality and heterogeneity in model confidence\.arXiv preprint arXiv:2002\.09831\.Cited by:[§6\.3](https://arxiv.org/html/2608.07827#S6.SS3.p4.1)\.

## Appendix ASummary of the Supplementary Material

The supplementary material is organized as follows\. In Appendix[B\.1](https://arxiv.org/html/2608.07827#A2.SS1), we provide additional details on how the BALD uncertainty score is converted into the confidence estimatorcBALDc\_\{\\mathrm\{BALD\}\}, including the normalization strategies we considered and the implementation choices used in our main experiments\. In Appendix[B\.2](https://arxiv.org/html/2608.07827#A2.SS2), we discuss the dataset, the selection criteria, and present the dropout sweep results for all model–dataset pairs\. In Appendix[C](https://arxiv.org/html/2608.07827#A3), we describe additional experimental details, including dataset splits, answer extraction, prompting, calibration metric computation, bin\-count analysis, and model\-specific inference settings\. Finally, Appendix[D](https://arxiv.org/html/2608.07827#A4)presents additional experimental results, including full confidence\-estimation and post\-hoc calibration results with bootstrap confidence intervals, in\-situp​\(True\)p\(\\mathrm\{True\}\)results, bin\-count ablations, and prompt comparisons\.

## Appendix BConfidence Estimation via MC Dropout

We provide additional details on confidence estimation using MC Dropout\.

### B\.1Converting BALD score tocBALDc\_\{\\mathrm\{BALD\}\}

Since BALD is an uncertainty score\(Gal and Ghahramani,[2016](https://arxiv.org/html/2608.07827#bib.bib10)\), we convert token\-level BALD values into confidence by applying a monotone decreasing normalization functionϕ\\phiand averaging over answer\-token positions:cBALD=1\|𝒚ans\|​∑j=1\|𝒚ans\|ϕ​\(BALDj\),c\_\{\\mathrm\{BALD\}\}=\\frac\{1\}\{\|\\bm\{y\}^\{\\mathrm\{ans\}\}\|\}\\sum\_\{j=1\}^\{\|\\bm\{y\}^\{\\mathrm\{ans\}\}\|\}\\phi\(\\mathrm\{BALD\}\_\{j\}\),whereϕ\\phimaps BALD values to\[0,1\]\[0,1\]with larger BALD corresponding to lower confidence\.

In practice, choosingϕ\\phiis non\-trivial because BALD is not naturally bounded in\[0,1\]\[0,1\]and its scale depends on the support over which entropy is computed\. We experimented withthree normalization strategies\. The first directly maps uncertainty to confidence asϕ​\(BALD\)=clip​\(1−BALD,0,1\)\\phi\(\\mathrm\{BALD\}\)=\\mathrm\{clip\}\(1\-\\mathrm\{BALD\},0,1\)\. The second normalizes BALD over an effective top\-kksupport: for each MC Dropout sample, we identify thekktokens with highest probability, take the union of these tokens across allTTsamples, and normalize the BALD score bylog⁡\(keff\)\\log\(k\_\{\\mathrm\{eff\}\}\), wherek≤keff≤T​kk\\leq k\_\{\\mathrm\{eff\}\}\\leq Tk\. The third uses a top\-ppsupport, where the effective support is determined by the smallest set of tokens whose cumulative probability exceeds a thresholdpp\. This can produce a wider range of effective support sizes than the top\-kkapproach\.

Across them, we found the resulting confidence scores to be sensitive to the dropout rate\. This sensitivity is expected from the BALD decomposition: the predictive entropy term reflects the entropy of the averaged distribution across dropout samples and therefore increases as dropout induces more variation among predictive distributions\. The expected conditional entropy term, however, remains relatively stable in our experiments, suggesting that individual dropout samples still tend to produce sharp token distributions\. Therefore, changing the dropout rate primarily changes the amount of disagreement captured by the averaged predictive distribution, which directly affects the scale of BALD and motivates the need for careful normalization\.

In addition to the dropout rate, the top\-kkand top\-ppnormalization strategies introduce extra hyperparameters that must be selected\. Although these approaches are more principled in that they normalize BALD with respect to an estimated effective support size, they add another source of tuning and sensitivity\. For simplicity and reproducibility, we therefore use the direct clipping normalization,ϕ​\(BALD\)=clip​\(1−BALD,0,1\)\\phi\(\\mathrm\{BALD\}\)=\\mathrm\{clip\}\(1\-\\mathrm\{BALD\},0,1\), in our main experiments\.

ModelDatasetDropout rateECEMCEBrierLlama\-3\.1\-8BGSM8K0\.100\.1590\.8050\.1790\.150\.5100\.9740\.4360\.200\.5760\.8690\.475GSMHard0\.100\.3080\.5680\.3400\.150\.1430\.5260\.2490\.200\.1410\.4130\.230SVAMP0\.100\.1330\.5490\.1720\.150\.4800\.9400\.4070\.200\.5690\.7470\.495Llama\-3\.2\-3BGSM8K0\.100\.1190\.8900\.1710\.150\.5520\.8280\.4700\.200\.5540\.8730\.464GSMHard0\.100\.2790\.9490\.2500\.150\.1490\.5620\.2450\.200\.1060\.2760\.205SVAMP0\.100\.0791\.0000\.1610\.150\.4760\.6190\.3880\.200\.5410\.7380\.459DeepSeek\-R1\-8BGSM8K0\.100\.1210\.9290\.2320\.150\.4120\.7420\.4030\.200\.4440\.5590\.436GSMHard0\.100\.4130\.8090\.4080\.150\.3490\.7260\.3500\.200\.1870\.7230\.295SVAMP0\.100\.2110\.9280\.2440\.150\.4300\.7670\.3930\.200\.5390\.7260\.486Qwen3\-8B\-ThinkingGSM8K0\.100\.0440\.1510\.0410\.150\.1190\.5670\.0640\.200\.3350\.5420\.167GSMHard0\.100\.1510\.2470\.1610\.150\.0670\.5480\.1430\.200\.2440\.7240\.203SVAMP0\.100\.0540\.2500\.0510\.150\.1520\.5470\.0790\.200\.3160\.8250\.160Qwen3\-4B\-InstructGSM8K0\.100\.0980\.6760\.0580\.150\.4640\.8640\.2680\.200\.5590\.9470\.369GSMHard0\.100\.1550\.5800\.2260\.150\.1980\.9060\.2430\.200\.2940\.9690\.313SVAMP0\.100\.0380\.2480\.0450\.150\.4130\.9320\.2390\.200\.4870\.9800\.301Table 5:Grid search for MC\-dropout rate selection\.We report ECE, MCE, and Brier score on the selection split for each model–dataset pair\. Lower values are better\.Boldindicates the selected dropout rate\.
### B\.2Dropout Rate Selection

Although fixingϕ​\(BALD\)\\phi\(\\mathrm\{BALD\}\)reduces the hyperparameter search space, MC Dropout remains sensitive to the dropout rate\. GSM8K and SVAMP provide training \(calibration\) and test set splits\. GSMHard does not include a training split, so we randomly split the dataset into 70% for calibration and 30% for evaluation\. From the calibration split, we randomly sample 100 examples to search over dropout rates\{0\.10,0\.15,0\.20\}\\\{0\.10,0\.15,0\.20\\\}for each model–dataset pair\. The selected rate is then fixed and evaluated on the disjoint evaluation split in Table[2](https://arxiv.org/html/2608.07827#S6.T2)\.

Table[5](https://arxiv.org/html/2608.07827#A2.T5)reports performance on the 100\-example tuning subset\. The selected rates generally transfer well to the evaluation split, but vary across models and datasets\. Notably, GSMHard favors a higher dropout rate than GSM8K and SVAMP for four of the five models, suggesting that more challenging datasets may require stronger stochastic perturbations to expose useful predictive variation\. This sensitivity nevertheless introduces an additional tuning cost for MC Dropout\.

## Appendix CImplementation Details

In addition to the experiment setups described in Sec\.[6\.1](https://arxiv.org/html/2608.07827#S6.SS1), we provide other implementation details\.

Datasets:GSM8K and SVAMP provide training splits, which we use as calibration sets for fitting post\-hoc calibrators, and we evaluate on their test sets\. GSMHard does not include a training split, so we randomly split the dataset into 70% for calibration and 30% for evaluation\. The resulting evaluation sets contain 1319, 400, and 300 examples for GSM8K, GSMHard, and SVAMP, respectively, while the calibration sets contain 7473, 919, and 700 examples, respectively\. Although these calibration sets are relatively large, Figure[2](https://arxiv.org/html/2608.07827#S6.F2)shows that as few as 50 calibration examples are already effective in many settings\.

Answer extraction:We employ a regex\-based extraction method, instructing the model to provide the final numerical answer after the\#\#\#\#delimiter\. Figure[3](https://arxiv.org/html/2608.07827#A3.F3)illustrates a representative interaction, including the system instructions, special tokens, and generated reasoning steps for Llama\-3\.1\-8B\.

Not all models were fine\-tuned to follow this specific answer format, i\.e\.,\#\#\#\#\. In particular, DeepSeek\-R1\-8B was fine\-tuned to produce final answers in the\\boxed\{\}format, and we posit that this format mismatch is the main reason for its degraded performance relative to reported results\.

One\-shot Demonstration\.Our input prompt includes a one\-shot demonstration \(see Figure[3](https://arxiv.org/html/2608.07827#A3.F3)\)\. We found this example to be essential for improving model accuracy and ensuring formatting consistency; specifically, it significantly increased the success rate of our regex\-based answer extraction by anchoring the model to the\#\#\#\#delimiter\.

Calibration:For reliability diagrams and the computation of ECE and MCE, we followGuoet al\.\([2017](https://arxiv.org/html/2608.07827#bib.bib7)\)and use 10 bins\. In Table[12](https://arxiv.org/html/2608.07827#A4.T12), we study the effect of the number of bins on calibration for Llama\-3\.1\-8B and Qwen3\-8B\-Thinking, reporting ECE, MCE, and Brier score across datasets and confidence estimators\. We observe that ECE remains relatively stable as the number of bins increases, while Brier score is unchanged because it does not depend on binning\. MCE is more sensitive to the binning choice\. This is expected because MCE measures the largest bin\-wise calibration gap: as bins become smaller, localized calibration errors are less averaged out and can produce larger maximum deviations\.

Inference:All inferences are performed using each model’s recommended configuration, including temperature, top\-pp, top\-kk, and system prompts\. In Table[6](https://arxiv.org/html/2608.07827#A3.T6), we summarize those settings\.

SettingLlama\-3\.1\-8BLlama\-3\.2\-3BDeepSeek\-R1\-8BQwen3\-8B\-ThinkingQwen3\-4B\-InstructTemperature1\.00\.60\.60\.60\.7Top\-pp1\.00\.90\.950\.950\.8Top\-kk\-1\-1\-12020Min\-pp00000System promptYesYesNoYesYesTable 6:Recommended inference settings\.We report the decoding and prompt settings used for each model\.[⬇](data:text/plain;base64,PHxiZWdpbl9vZl90ZXh0fD48fHN0YXJ0X2hlYWRlcl9pZHw+c3lzdGVtPHxlbmRfaGVhZGVyX2lkfD4KCllvdSBhcmUgYSBoZWxwZnVsLCByZXNwZWN0ZnVsLCBhbmQgaG9uZXN0IGFzc2lzdGFudC4gQWx3YXlzIGFuc3dlciBhcyBoZWxwZnVsbHkgYXMgcG9zc2libGUsIHdoaWxlIGJlaW5nIHNhZmUuIFlvdXIgYW5zd2VycyBzaG91bGQgbm90IGluY2x1ZGUgYW55IGhhcm1mdWwsIHVuZXRoaWNhbCwgcmFjaXN0LCBzZXhpc3QsIHRveGljLCBkYW5nZXJvdXMsIG9yIGlsbGVnYWwgY29udGVudC4gUGxlYXNlIGVuc3VyZSB0aGF0IHlvdXIgcmVzcG9uc2VzIGFyZSBzb2NpYWxseSB1bmJpYXNlZCBhbmQgcG9zaXRpdmUgaW4gbmF0dXJlLgoKPHxzdGFydF9oZWFkZXJfaWR8PnVzZXI8fGVuZF9oZWFkZXJfaWR8PgoKWW91IGFyZSBhIG1hdGhlbWF0aWNzIGV4cGVydC4gRmlyc3QsIHN0dWR5IHRoaXMgZXhhbXBsZToKUXVlc3Rpb246IFNhcmFoIGhhcyAxMCBtYXJibGVzLCBnaXZlcyAzIGF3YXkuCkFuc3dlcjogU2FyYWggd291bGQgaGF2ZSAxMCAtIDMgPSA8PDEwLTM9Nz4+NyBtYXJibGVzIHJlbWFpbmluZy4KIyMjIyA3CgpOZXh0LCBzb2x2ZToKTWFyeSBpcyBiYWtpbmcgYSBjYWtlLiBUaGUgcmVjaXBlIGNhbGxzIGZvciA2IGN1cHMgb2YgZmxvdXIsIDggY3VwcyBvZiBzdWdhciwKYW5kIDcgY3VwcyBvZiBzYWx0LiBTaGUgYWxyZWFkeSBwdXQgaW4gNSBjdXBzIG9mIGZsb3VyLgpIb3cgbWFueSBtb3JlIGN1cHMgb2Ygc3VnYXIgdGhhbiBjdXBzIG9mIHNhbHQgZG9lcyBzaGUgbmVlZCB0byBhZGQgbm93PwoKQXQgdGhlIGVuZCwgd3JpdGUgdGhlIGFuc3dlciBhcyBhbiBpbnRlZ2VyIGFmdGVyICcjIyMjICcuCgo8fHN0YXJ0X2hlYWRlcl9pZHw+YXNzaXN0YW50PHxlbmRfaGVhZGVyX2lkfD4KClRvIHNvbHZlIHRoaXMgcHJvYmxlbSwgbGV0J3MgYnJlYWsgaXQgZG93biBzdGVwIGJ5IHN0ZXA6CjEuIFRoZSByZWNpcGUgY2FsbHMgZm9yIDggY3VwcyBvZiBzdWdhciBhbmQgNyBjdXBzIG9mIHNhbHQuCjIuIERpZmZlcmVuY2UgPSA4IC0gNyA9IDEuCgojIyMjIDE=)<\|begin\_of\_text\|\><\|start\_header\_id\|\>system<\|end\_header\_id\|\>Youareahelpful,respectful,andhonestassistant\.Alwaysanswerashelpfullyaspossible,whilebeingsafe\.Youranswersshouldnotincludeanyharmful,unethical,racist,sexist,toxic,dangerous,orillegalcontent\.Pleaseensurethatyourresponsesaresociallyunbiasedandpositiveinnature\.<\|start\_header\_id\|\>user<\|end\_header\_id\|\>Youareamathematicsexpert\.First,studythisexample:Question:Sarahhas10marbles,gives3away\.Answer:Sarahwouldhave10\-3=<<10\-3=7\>\>7marblesremaining\.\#\#\#\#7Next,solve:Maryisbakingacake\.Therecipecallsfor6cupsofflour,8cupsofsugar,and7cupsofsalt\.Shealreadyputin5cupsofflour\.Howmanymorecupsofsugarthancupsofsaltdoessheneedtoaddnow?Attheend,writetheanswerasanintegerafter’\#\#\#\#’\.<\|start\_header\_id\|\>assistant<\|end\_header\_id\|\>Tosolvethisproblem,let’sbreakitdownstepbystep:1\.Therecipecallsfor8cupsofsugarand7cupsofsalt\.2\.Difference=8\-7=1\.\#\#\#\#1Figure 3:Example input–output exchangeon GSM8K with Llama\-3\.1\-8B\. The one\-shot example is crucial in improving the success rate of our regex\-based answer extraction\.
## Appendix DAdditional Experiment Results

#### Bootstrap confidence intervals\.

Tables[7](https://arxiv.org/html/2608.07827#A4.T7),[8](https://arxiv.org/html/2608.07827#A4.T8), and[9](https://arxiv.org/html/2608.07827#A4.T9)report mean estimates with 95% percentile confidence intervals obtained from 5000 bootstrap resamples of the evaluation set\. Because ECE and MCE depend on bin assignments, their bootstrap distributions may be asymmetric, and MCE intervals can be comparatively wide due to variation in the worst\-calibrated bin\. Nevertheless, the point estimates and confidence intervals are broadly consistent with the qualitative conclusions\.

ModelDatasetAcc\.Seq AvgSeq JointAns AvgAns JointECEMCEBrierECEMCEBrierECEMCEBrierECEMCEBrierLlama\-3\.1\-8BGSM8K\.824\[\.804,\.845\]\.824\_\{\\scriptscriptstyle\[\.804,\.845\]\}\.053\[\.035,\.072\]\\mathbf\{\.053\}\_\{\\scriptscriptstyle\[\.035,\.072\]\}\.481\[\.417,\.575\]\.481\_\{\\scriptscriptstyle\[\.417,\.575\]\}\.131\[\.119,\.144\]\.131\_\{\\scriptscriptstyle\[\.119,\.144\]\}\.117\[\.101,\.137\]\.117\_\{\\scriptscriptstyle\[\.101,\.137\]\}\.195\[\.139,\.294\]\.195\_\{\\scriptscriptstyle\[\.139,\.294\]\}\.138\[\.130,\.147\]\.138\_\{\\scriptscriptstyle\[\.130,\.147\]\}\.166\[\.147,\.187\]\.166\_\{\\scriptscriptstyle\[\.147,\.187\]\}\.529\[\.357,\.818\]\.529\_\{\\scriptscriptstyle\[\.357,\.818\]\}\.167\[\.148,\.187\]\.167\_\{\\scriptscriptstyle\[\.148,\.187\]\}\.165\[\.146,\.187\]\.165\_\{\\scriptscriptstyle\[\.146,\.187\]\}\.469\[\.345,\.818\]\.469\_\{\\scriptscriptstyle\[\.345,\.818\]\}\.167\[\.147,\.187\]\.167\_\{\\scriptscriptstyle\[\.147,\.187\]\}GSMHard\.350\[\.302,\.398\]\.350\_\{\\scriptscriptstyle\[\.302,\.398\]\}\.457\[\.411,\.502\]\.457\_\{\\scriptscriptstyle\[\.411,\.502\]\}\.600\[\.556,\.665\]\.600\_\{\\scriptscriptstyle\[\.556,\.665\]\}\.416\[\.386,\.446\]\.416\_\{\\scriptscriptstyle\[\.386,\.446\]\}\.320\[\.277,\.363\]\.320\_\{\\scriptscriptstyle\[\.277,\.363\]\}\.414\[\.374,\.518\]\.414\_\{\\scriptscriptstyle\[\.374,\.518\]\}\.299\[\.276,\.321\]\.299\_\{\\scriptscriptstyle\[\.276,\.321\]\}\.622\[\.576,\.668\]\.622\_\{\\scriptscriptstyle\[\.576,\.668\]\}\.743\[\.709,\.856\]\.743\_\{\\scriptscriptstyle\[\.709,\.856\]\}\.606\[\.561,\.648\]\.606\_\{\\scriptscriptstyle\[\.561,\.648\]\}\.615\[\.568,\.661\]\.615\_\{\\scriptscriptstyle\[\.568,\.661\]\}\.753\[\.733,\.850\]\.753\_\{\\scriptscriptstyle\[\.733,\.850\]\}\.596\[\.551,\.639\]\.596\_\{\\scriptscriptstyle\[\.551,\.639\]\}SVAMP\.827\[\.780,\.870\]\.827\_\{\\scriptscriptstyle\[\.780,\.870\]\}\.057\[\.030,\.100\]\\mathbf\{\.057\}\_\{\\scriptscriptstyle\[\.030,\.100\]\}\.309\[\.148,\.492\]\.309\_\{\\scriptscriptstyle\[\.148,\.492\]\}\.133\[\.110,\.159\]\.133\_\{\\scriptscriptstyle\[\.110,\.159\]\}\.170\[\.136,\.213\]\.170\_\{\\scriptscriptstyle\[\.136,\.213\]\}\.332\[\.265,\.743\]\.332\_\{\\scriptscriptstyle\[\.265,\.743\]\}\.160\[\.143,\.178\]\.160\_\{\\scriptscriptstyle\[\.143,\.178\]\}\.169\[\.128,\.214\]\.169\_\{\\scriptscriptstyle\[\.128,\.214\]\}\.797\[\.148,\.817\]\.797\_\{\\scriptscriptstyle\[\.148,\.817\]\}\.169\[\.130,\.211\]\.169\_\{\\scriptscriptstyle\[\.130,\.211\]\}\.171\[\.130,\.215\]\.171\_\{\\scriptscriptstyle\[\.130,\.215\]\}\.765\[\.157,\.798\]\.765\_\{\\scriptscriptstyle\[\.157,\.798\]\}\.169\[\.128,\.212\]\.169\_\{\\scriptscriptstyle\[\.128,\.212\]\}Llama\-3\.2\-3BGSM8K\.775\[\.753,\.798\]\.775\_\{\\scriptscriptstyle\[\.753,\.798\]\}\.124\[\.102,\.146\]\.124\_\{\\scriptscriptstyle\[\.102,\.146\]\}\.587\[\.289,\.793\]\.587\_\{\\scriptscriptstyle\[\.289,\.793\]\}\.182\[\.165,\.200\]\.182\_\{\\scriptscriptstyle\[\.165,\.200\]\}\.092\[\.072,\.116\]\.092\_\{\\scriptscriptstyle\[\.072,\.116\]\}\.364\[\.269,\.655\]\.364\_\{\\scriptscriptstyle\[\.269,\.655\]\}\.173\[\.157,\.189\]\.173\_\{\\scriptscriptstyle\[\.157,\.189\]\}\.218\[\.196,\.241\]\.218\_\{\\scriptscriptstyle\[\.196,\.241\]\}\.635\[\.445,\.763\]\.635\_\{\\scriptscriptstyle\[\.445,\.763\]\}\.218\[\.196,\.240\]\.218\_\{\\scriptscriptstyle\[\.196,\.240\]\}\.218\[\.197,\.240\]\.218\_\{\\scriptscriptstyle\[\.197,\.240\]\}\.623\[\.426,\.758\]\.623\_\{\\scriptscriptstyle\[\.426,\.758\]\}\.217\[\.196,\.240\]\.217\_\{\\scriptscriptstyle\[\.196,\.240\]\}GSMHard\.258\[\.215,\.300\]\.258\_\{\\scriptscriptstyle\[\.215,\.300\]\}\.636\[\.594,\.678\]\.636\_\{\\scriptscriptstyle\[\.594,\.678\]\}\.773\[\.751,\.791\]\.773\_\{\\scriptscriptstyle\[\.751,\.791\]\}\.589\[\.556,\.623\]\.589\_\{\\scriptscriptstyle\[\.556,\.623\]\}\.597\[\.555,\.638\]\.597\_\{\\scriptscriptstyle\[\.555,\.638\]\}\.676\[\.648,\.744\]\.676\_\{\\scriptscriptstyle\[\.648,\.744\]\}\.538\[\.506,\.569\]\.538\_\{\\scriptscriptstyle\[\.506,\.569\]\}\.725\[\.682,\.766\]\.725\_\{\\scriptscriptstyle\[\.682,\.766\]\}\.857\[\.842,\.870\]\.857\_\{\\scriptscriptstyle\[\.842,\.870\]\}\.712\[\.670,\.753\]\.712\_\{\\scriptscriptstyle\[\.670,\.753\]\}\.722\[\.677,\.766\]\.722\_\{\\scriptscriptstyle\[\.677,\.766\]\}\.768\[\.749,\.862\]\.768\_\{\\scriptscriptstyle\[\.749,\.862\]\}\.708\[\.666,\.748\]\.708\_\{\\scriptscriptstyle\[\.666,\.748\]\}SVAMP\.820\[\.777,\.863\]\.820\_\{\\scriptscriptstyle\[\.777,\.863\]\}\.068\[\.036,\.113\]\.068\_\{\\scriptscriptstyle\[\.036,\.113\]\}\.783\[\.773,\.792\]\.783\_\{\\scriptscriptstyle\[\.773,\.792\]\}\.145\[\.113,\.177\]\.145\_\{\\scriptscriptstyle\[\.113,\.177\]\}\.035\[\.016,\.081\]\\mathbf\{\.035\}\_\{\\scriptscriptstyle\[\.016,\.081\]\}\.676\[\.048,\.676\]\.676\_\{\\scriptscriptstyle\[\.048,\.676\]\}\.139\[\.112,\.169\]\.139\_\{\\scriptscriptstyle\[\.112,\.169\]\}\.175\[\.135,\.218\]\.175\_\{\\scriptscriptstyle\[\.135,\.218\]\}\.840\[\.734,\.871\]\.840\_\{\\scriptscriptstyle\[\.734,\.871\]\}\.170\[\.128,\.213\]\.170\_\{\\scriptscriptstyle\[\.128,\.213\]\}\.175\[\.133,\.218\]\.175\_\{\\scriptscriptstyle\[\.133,\.218\]\}\.837\[\.731,\.866\]\.837\_\{\\scriptscriptstyle\[\.731,\.866\]\}\.170\[\.129,\.212\]\.170\_\{\\scriptscriptstyle\[\.129,\.212\]\}DeepSeek\-R1\-8BGSM8K\.650\[\.624,\.676\]\.650\_\{\\scriptscriptstyle\[\.624,\.676\]\}\.253\[\.227,\.278\]\.253\_\{\\scriptscriptstyle\[\.227,\.278\]\}\.384\[\.341,\.427\]\.384\_\{\\scriptscriptstyle\[\.341,\.427\]\}\.285\[\.265,\.305\]\.285\_\{\\scriptscriptstyle\[\.265,\.305\]\}\.219\[\.194,\.245\]\.219\_\{\\scriptscriptstyle\[\.194,\.245\]\}\.476\[\.330,\.615\]\.476\_\{\\scriptscriptstyle\[\.330,\.615\]\}\.267\[\.249,\.286\]\.267\_\{\\scriptscriptstyle\[\.249,\.286\]\}\.304\[\.279,\.330\]\.304\_\{\\scriptscriptstyle\[\.279,\.330\]\}\.361\[\.354,\.572\]\.361\_\{\\scriptscriptstyle\[\.354,\.572\]\}\.310\[\.286,\.334\]\.310\_\{\\scriptscriptstyle\[\.286,\.334\]\}\.302\[\.277,\.329\]\.302\_\{\\scriptscriptstyle\[\.277,\.329\]\}\.356\[\.345,\.561\]\.356\_\{\\scriptscriptstyle\[\.345,\.561\]\}\.309\[\.285,\.333\]\.309\_\{\\scriptscriptstyle\[\.285,\.333\]\}GSMHard\.320\[\.273,\.365\]\.320\_\{\\scriptscriptstyle\[\.273,\.365\]\}\.588\[\.543,\.635\]\.588\_\{\\scriptscriptstyle\[\.543,\.635\]\}\.646\[\.578,\.718\]\.646\_\{\\scriptscriptstyle\[\.578,\.718\]\}\.561\[\.525,\.598\]\.561\_\{\\scriptscriptstyle\[\.525,\.598\]\}\.556\[\.509,\.598\]\.556\_\{\\scriptscriptstyle\[\.509,\.598\]\}\.611\[\.546,\.786\]\.611\_\{\\scriptscriptstyle\[\.546,\.786\]\}\.523\[\.487,\.556\]\.523\_\{\\scriptscriptstyle\[\.487,\.556\]\}\.637\[\.591,\.683\]\.637\_\{\\scriptscriptstyle\[\.591,\.683\]\}\.674\[\.628,\.779\]\.674\_\{\\scriptscriptstyle\[\.628,\.779\]\}\.620\[\.578,\.662\]\.620\_\{\\scriptscriptstyle\[\.578,\.662\]\}\.631\[\.583,\.676\]\.631\_\{\\scriptscriptstyle\[\.583,\.676\]\}\.690\[\.652,\.807\]\.690\_\{\\scriptscriptstyle\[\.652,\.807\]\}\.613\[\.569,\.656\]\.613\_\{\\scriptscriptstyle\[\.569,\.656\]\}SVAMP\.728\[\.676,\.779\]\.728\_\{\\scriptscriptstyle\[\.676,\.779\]\}\.164\[\.116,\.215\]\.164\_\{\\scriptscriptstyle\[\.116,\.215\]\}\.784\[\.155,\.784\]\.784\_\{\\scriptscriptstyle\[\.155,\.784\]\}\.219\[\.182,\.258\]\.219\_\{\\scriptscriptstyle\[\.182,\.258\]\}\.127\[\.078,\.180\]\\mathbf\{\.127\}\_\{\\scriptscriptstyle\[\.078,\.180\]\}\.525\[\.314,\.715\]\.525\_\{\\scriptscriptstyle\[\.314,\.715\]\}\.207\[\.171,\.244\]\.207\_\{\\scriptscriptstyle\[\.171,\.244\]\}\.244\[\.196,\.294\]\.244\_\{\\scriptscriptstyle\[\.196,\.294\]\}\.617\[\.603,\.789\]\.617\_\{\\scriptscriptstyle\[\.603,\.789\]\}\.244\[\.196,\.291\]\.244\_\{\\scriptscriptstyle\[\.196,\.291\]\}\.242\[\.195,\.295\]\.242\_\{\\scriptscriptstyle\[\.195,\.295\]\}\.606\[\.506,\.789\]\.606\_\{\\scriptscriptstyle\[\.506,\.789\]\}\.242\[\.196,\.290\]\.242\_\{\\scriptscriptstyle\[\.196,\.290\]\}Qwen3\-8B\-ThinkingGSM8K\.958\[\.947,\.969\]\.958\_\{\\scriptscriptstyle\[\.947,\.969\]\}\.006\[\.001,\.017\]\\mathbf\{\.006\}\_\{\\scriptscriptstyle\[\.001,\.017\]\}\.070\[\.006,\.231\]\.070\_\{\\scriptscriptstyle\[\.006,\.231\]\}\.039\[\.030,\.050\]\.039\_\{\\scriptscriptstyle\[\.030,\.050\]\}\.018\[\.008,\.029\]\.018\_\{\\scriptscriptstyle\[\.008,\.029\]\}\.027\[\.012,\.080\]\.027\_\{\\scriptscriptstyle\[\.012,\.080\]\}\.040\[\.031,\.049\]\.040\_\{\\scriptscriptstyle\[\.031,\.049\]\}\.042\[\.031,\.053\]\.042\_\{\\scriptscriptstyle\[\.031,\.053\]\}\.042\[\.032,\.053\]\.042\_\{\\scriptscriptstyle\[\.032,\.053\]\}\.042\[\.031,\.053\]\.042\_\{\\scriptscriptstyle\[\.031,\.053\]\}\.042\[\.032,\.053\]\.042\_\{\\scriptscriptstyle\[\.032,\.053\]\}\.042\[\.032,\.053\]\.042\_\{\\scriptscriptstyle\[\.032,\.053\]\}\.042\[\.031,\.053\]\.042\_\{\\scriptscriptstyle\[\.031,\.053\]\}GSMHard\.810\[\.762,\.854\]\.810\_\{\\scriptscriptstyle\[\.762,\.854\]\}\.144\[\.101,\.189\]\.144\_\{\\scriptscriptstyle\[\.101,\.189\]\}\.781\[\.545,\.888\]\.781\_\{\\scriptscriptstyle\[\.545,\.888\]\}\.169\[\.130,\.210\]\.169\_\{\\scriptscriptstyle\[\.130,\.210\]\}\.128\[\.085,\.172\]\.128\_\{\\scriptscriptstyle\[\.085,\.172\]\}\.793\[\.287,\.793\]\.793\_\{\\scriptscriptstyle\[\.287,\.793\]\}\.163\[\.126,\.203\]\.163\_\{\\scriptscriptstyle\[\.126,\.203\]\}\.190\[\.149,\.238\]\.190\_\{\\scriptscriptstyle\[\.149,\.238\]\}\.190\[\.149,\.235\]\.190\_\{\\scriptscriptstyle\[\.149,\.235\]\}\.190\[\.146,\.235\]\.190\_\{\\scriptscriptstyle\[\.146,\.235\]\}\.190\[\.149,\.234\]\.190\_\{\\scriptscriptstyle\[\.149,\.234\]\}\.190\[\.146,\.238\]\.190\_\{\\scriptscriptstyle\[\.146,\.238\]\}\.190\[\.146,\.235\]\.190\_\{\\scriptscriptstyle\[\.146,\.235\]\}SVAMP\.972\[\.951,\.990\]\.972\_\{\\scriptscriptstyle\[\.951,\.990\]\}\.017\[\.003,\.036\]\\mathbf\{\.017\}\_\{\\scriptscriptstyle\[\.003,\.036\]\}\.017\[\.011,\.281\]\.017\_\{\\scriptscriptstyle\[\.011,\.281\]\}\.026\[\.011,\.044\]\.026\_\{\\scriptscriptstyle\[\.011,\.044\]\}\.033\[\.018,\.052\]\.033\_\{\\scriptscriptstyle\[\.018,\.052\]\}\.034\[\.023,\.193\]\.034\_\{\\scriptscriptstyle\[\.023,\.193\]\}\.027\[\.012,\.044\]\.027\_\{\\scriptscriptstyle\[\.012,\.044\]\}\.028\[\.010,\.049\]\.028\_\{\\scriptscriptstyle\[\.010,\.049\]\}\.028\[\.010,\.049\]\.028\_\{\\scriptscriptstyle\[\.010,\.049\]\}\.028\[\.010,\.049\]\.028\_\{\\scriptscriptstyle\[\.010,\.049\]\}\.028\[\.010,\.049\]\.028\_\{\\scriptscriptstyle\[\.010,\.049\]\}\.028\[\.010,\.049\]\.028\_\{\\scriptscriptstyle\[\.010,\.049\]\}\.028\[\.010,\.049\]\.028\_\{\\scriptscriptstyle\[\.010,\.049\]\}Qwen3\-4B\-InstructGSM8K\.938\[\.925,\.951\]\.938\_\{\\scriptscriptstyle\[\.925,\.951\]\}\.035\[\.023,\.048\]\.035\_\{\\scriptscriptstyle\[\.023,\.048\]\}\.478\[\.151,\.777\]\.478\_\{\\scriptscriptstyle\[\.151,\.777\]\}\.057\[\.045,\.069\]\.057\_\{\\scriptscriptstyle\[\.045,\.069\]\}\.028\[\.016,\.041\]\\mathbf\{\.028\}\_\{\\scriptscriptstyle\[\.016,\.041\]\}\.790\[\.176,\.792\]\.790\_\{\\scriptscriptstyle\[\.176,\.792\]\}\.056\[\.045,\.067\]\.056\_\{\\scriptscriptstyle\[\.045,\.067\]\}\.062\[\.049,\.075\]\.062\_\{\\scriptscriptstyle\[\.049,\.075\]\}\.062\[\.049,\.075\]\.062\_\{\\scriptscriptstyle\[\.049,\.075\]\}\.062\[\.049,\.075\]\.062\_\{\\scriptscriptstyle\[\.049,\.075\]\}\.062\[\.049,\.075\]\.062\_\{\\scriptscriptstyle\[\.049,\.075\]\}\.062\[\.049,\.075\]\.062\_\{\\scriptscriptstyle\[\.049,\.075\]\}\.062\[\.049,\.075\]\.062\_\{\\scriptscriptstyle\[\.049,\.075\]\}GSMHard\.701\[\.656,\.744\]\.701\_\{\\scriptscriptstyle\[\.656,\.744\]\}\.266\[\.220,\.310\]\.266\_\{\\scriptscriptstyle\[\.220,\.310\]\}\.724\[\.575,\.845\]\.724\_\{\\scriptscriptstyle\[\.575,\.845\]\}\.267\[\.225,\.309\]\.267\_\{\\scriptscriptstyle\[\.225,\.309\]\}\.256\[\.211,\.301\]\.256\_\{\\scriptscriptstyle\[\.211,\.301\]\}\.787\[\.548,\.793\]\.787\_\{\\scriptscriptstyle\[\.548,\.793\]\}\.258\[\.219,\.298\]\.258\_\{\\scriptscriptstyle\[\.219,\.298\]\}\.298\[\.253,\.346\]\.298\_\{\\scriptscriptstyle\[\.253,\.346\]\}\.299\[\.254,\.345\]\.299\_\{\\scriptscriptstyle\[\.254,\.345\]\}\.298\[\.251,\.346\]\.298\_\{\\scriptscriptstyle\[\.251,\.346\]\}\.298\[\.253,\.344\]\.298\_\{\\scriptscriptstyle\[\.253,\.344\]\}\.299\[\.252,\.347\]\.299\_\{\\scriptscriptstyle\[\.252,\.347\]\}\.298\[\.251,\.346\]\.298\_\{\\scriptscriptstyle\[\.251,\.346\]\}SVAMP\.943\[\.913,\.967\]\.943\_\{\\scriptscriptstyle\[\.913,\.967\]\}\.032\[\.010,\.059\]\.032\_\{\\scriptscriptstyle\[\.010,\.059\]\}\.366\[\.036,\.733\]\.366\_\{\\scriptscriptstyle\[\.036,\.733\]\}\.051\[\.030,\.074\]\.051\_\{\\scriptscriptstyle\[\.030,\.074\]\}\.025\[\.008,\.052\]\\mathbf\{\.025\}\_\{\\scriptscriptstyle\[\.008,\.052\]\}\.267\[\.093,\.753\]\.267\_\{\\scriptscriptstyle\[\.093,\.753\]\}\.049\[\.029,\.074\]\.049\_\{\\scriptscriptstyle\[\.029,\.074\]\}\.057\[\.033,\.084\]\.057\_\{\\scriptscriptstyle\[\.033,\.084\]\}\.057\[\.030,\.084\]\.057\_\{\\scriptscriptstyle\[\.030,\.084\]\}\.057\[\.033,\.084\]\.057\_\{\\scriptscriptstyle\[\.033,\.084\]\}\.057\[\.033,\.084\]\.057\_\{\\scriptscriptstyle\[\.033,\.084\]\}\.057\[\.033,\.084\]\.057\_\{\\scriptscriptstyle\[\.033,\.084\]\}\.057\[\.033,\.084\]\.057\_\{\\scriptscriptstyle\[\.033,\.084\]\}

\(a\) Single\-pass estimators

ModelDatasetp​\(True\)p\(\\mathrm\{True\}\)cBALDc\_\{\\mathrm\{BALD\}\}ECEMCEBrierECEMCEBrierLlama\-3\.1\-8BGSM8K\.097\[\.083,\.120\]\.097\_\{\\scriptscriptstyle\[\.083,\.120\]\}\.625\[\.228,\.893\]\.625\_\{\\scriptscriptstyle\[\.228,\.893\]\}\.143\[\.127,\.159\]\.143\_\{\\scriptscriptstyle\[\.127,\.159\]\}\.149\[\.129,\.170\]\.149\_\{\\scriptscriptstyle\[\.129,\.170\]\}\.667\[\.521,1\.000\]\.667\_\{\\scriptscriptstyle\[\.521,1\.000\]\}\.182\[\.171,\.194\]\.182\_\{\\scriptscriptstyle\[\.171,\.194\]\}GSMHard\.333\[\.294,\.381\]\.333\_\{\\scriptscriptstyle\[\.294,\.381\]\}\.627\[\.556,\.720\]\.627\_\{\\scriptscriptstyle\[\.556,\.720\]\}\.319\[\.289,\.350\]\.319\_\{\\scriptscriptstyle\[\.289,\.350\]\}\.152\[\.118,\.203\]\\mathbf\{\.152\}\_\{\\scriptscriptstyle\[\.118,\.203\]\}\.424\[\.256,\.558\]\.424\_\{\\scriptscriptstyle\[\.256,\.558\]\}\.252\[\.222,\.282\]\.252\_\{\\scriptscriptstyle\[\.222,\.282\]\}SVAMP\.099\[\.071,\.147\]\.099\_\{\\scriptscriptstyle\[\.071,\.147\]\}\.450\[\.440,\.972\]\.450\_\{\\scriptscriptstyle\[\.440,\.972\]\}\.139\[\.108,\.173\]\.139\_\{\\scriptscriptstyle\[\.108,\.173\]\}\.095\[\.074,\.147\]\.095\_\{\\scriptscriptstyle\[\.074,\.147\]\}\.633\[\.621,1\.000\]\.633\_\{\\scriptscriptstyle\[\.621,1\.000\]\}\.158\[\.134,\.184\]\.158\_\{\\scriptscriptstyle\[\.134,\.184\]\}Llama\-3\.2\-3BGSM8K\.055\[\.039,\.077\]\\mathbf\{\.055\}\_\{\\scriptscriptstyle\[\.039,\.077\]\}\.304\[\.138,\.634\]\.304\_\{\\scriptscriptstyle\[\.138,\.634\]\}\.161\[\.150,\.173\]\.161\_\{\\scriptscriptstyle\[\.150,\.173\]\}\.108\[\.089,\.132\]\.108\_\{\\scriptscriptstyle\[\.089,\.132\]\}\.389\[\.302,\.651\]\.389\_\{\\scriptscriptstyle\[\.302,\.651\]\}\.186\[\.173,\.200\]\.186\_\{\\scriptscriptstyle\[\.173,\.200\]\}GSMHard\.251\[\.221,\.295\]\.251\_\{\\scriptscriptstyle\[\.221,\.295\]\}\.406\[\.356,\.542\]\.406\_\{\\scriptscriptstyle\[\.356,\.542\]\}\.229\[\.210,\.248\]\.229\_\{\\scriptscriptstyle\[\.210,\.248\]\}\.078\[\.049,\.125\]\\mathbf\{\.078\}\_\{\\scriptscriptstyle\[\.049,\.125\]\}\.202\[\.126,\.512\]\.202\_\{\\scriptscriptstyle\[\.126,\.512\]\}\.196\[\.171,\.221\]\.196\_\{\\scriptscriptstyle\[\.171,\.221\]\}SVAMP\.104\[\.078,\.149\]\.104\_\{\\scriptscriptstyle\[\.078,\.149\]\}\.423\[\.326,\.828\]\.423\_\{\\scriptscriptstyle\[\.326,\.828\]\}\.148\[\.125,\.173\]\.148\_\{\\scriptscriptstyle\[\.125,\.173\]\}\.097\[\.072,\.149\]\.097\_\{\\scriptscriptstyle\[\.072,\.149\]\}\.610\[\.427,\.896\]\.610\_\{\\scriptscriptstyle\[\.427,\.896\]\}\.167\[\.142,\.194\]\.167\_\{\\scriptscriptstyle\[\.142,\.194\]\}DeepSeek\-R1\-8BGSM8K\.315\[\.289,\.341\]\.315\_\{\\scriptscriptstyle\[\.289,\.341\]\}\.484\[\.326,\.972\]\.484\_\{\\scriptscriptstyle\[\.326,\.972\]\}\.326\[\.303,\.350\]\.326\_\{\\scriptscriptstyle\[\.303,\.350\]\}\.168\[\.142,\.195\]\\mathbf\{\.168\}\_\{\\scriptscriptstyle\[\.142,\.195\]\}\.732\[\.510,\.937\]\.732\_\{\\scriptscriptstyle\[\.510,\.937\]\}\.265\[\.252,\.278\]\.265\_\{\\scriptscriptstyle\[\.252,\.278\]\}GSMHard\.634\[\.588,\.680\]\.634\_\{\\scriptscriptstyle\[\.588,\.680\]\}\.673\[\.664,\.817\]\.673\_\{\\scriptscriptstyle\[\.664,\.817\]\}\.618\[\.576,\.661\]\.618\_\{\\scriptscriptstyle\[\.576,\.661\]\}\.102\[\.075,\.154\]\\mathbf\{\.102\}\_\{\\scriptscriptstyle\[\.075,\.154\]\}\.606\[\.443,\.610\]\.606\_\{\\scriptscriptstyle\[\.443,\.610\]\}\.239\[\.214,\.265\]\.239\_\{\\scriptscriptstyle\[\.214,\.265\]\}SVAMP\.241\[\.194,\.293\]\.241\_\{\\scriptscriptstyle\[\.194,\.293\]\}\.536\[\.360,\.680\]\.536\_\{\\scriptscriptstyle\[\.360,\.680\]\}\.252\[\.207,\.299\]\.252\_\{\\scriptscriptstyle\[\.207,\.299\]\}\.174\[\.133,\.231\]\.174\_\{\\scriptscriptstyle\[\.133,\.231\]\}\.979\[\.361,\.979\]\.979\_\{\\scriptscriptstyle\[\.361,\.979\]\}\.240\[\.214,\.270\]\.240\_\{\\scriptscriptstyle\[\.214,\.270\]\}Qwen3\-8B\-ThinkingGSM8K\.035\[\.025,\.047\]\.035\_\{\\scriptscriptstyle\[\.025,\.047\]\}\.829\[\.246,\.829\]\.829\_\{\\scriptscriptstyle\[\.246,\.829\]\}\.042\[\.032,\.053\]\.042\_\{\\scriptscriptstyle\[\.032,\.053\]\}\.043\[\.032,\.054\]\.043\_\{\\scriptscriptstyle\[\.032,\.054\]\}\.545\[\.193,\.692\]\.545\_\{\\scriptscriptstyle\[\.193,\.692\]\}\.042\[\.032,\.053\]\.042\_\{\\scriptscriptstyle\[\.032,\.053\]\}GSMHard\.183\[\.141,\.229\]\.183\_\{\\scriptscriptstyle\[\.141,\.229\]\}\.373\[\.179,\.897\]\.373\_\{\\scriptscriptstyle\[\.179,\.897\]\}\.183\[\.141,\.227\]\.183\_\{\\scriptscriptstyle\[\.141,\.227\]\}\.028\[\.022,\.082\]\\mathbf\{\.028\}\_\{\\scriptscriptstyle\[\.022,\.082\]\}\.365\[\.107,\.665\]\.365\_\{\\scriptscriptstyle\[\.107,\.665\]\}\.148\[\.122,\.175\]\.148\_\{\\scriptscriptstyle\[\.122,\.175\]\}SVAMP\.020\[\.005,\.041\]\.020\_\{\\scriptscriptstyle\[\.005,\.041\]\}\.348\[\.123,\.773\]\.348\_\{\\scriptscriptstyle\[\.123,\.773\]\}\.027\[\.011,\.047\]\.027\_\{\\scriptscriptstyle\[\.011,\.047\]\}\.026\[\.010,\.048\]\.026\_\{\\scriptscriptstyle\[\.010,\.048\]\}\.133\[\.110,\.260\]\.133\_\{\\scriptscriptstyle\[\.110,\.260\]\}\.026\[\.011,\.048\]\.026\_\{\\scriptscriptstyle\[\.011,\.048\]\}Qwen3\-4B\-InstructGSM8K\.062\[\.050,\.076\]\.062\_\{\\scriptscriptstyle\[\.050,\.076\]\}\.652\[\.626,1\.000\]\.652\_\{\\scriptscriptstyle\[\.626,1\.000\]\}\.063\[\.050,\.077\]\.063\_\{\\scriptscriptstyle\[\.050,\.077\]\}\.061\[\.048,\.075\]\.061\_\{\\scriptscriptstyle\[\.048,\.075\]\}\.650\[\.624,\.697\]\.650\_\{\\scriptscriptstyle\[\.624,\.697\]\}\.069\[\.058,\.080\]\.069\_\{\\scriptscriptstyle\[\.058,\.080\]\}GSMHard\.285\[\.242,\.332\]\.285\_\{\\scriptscriptstyle\[\.242,\.332\]\}\.985\[\.453,\.985\]\.985\_\{\\scriptscriptstyle\[\.453,\.985\]\}\.286\[\.242,\.331\]\.286\_\{\\scriptscriptstyle\[\.242,\.331\]\}\.188\[\.150,\.241\]\\mathbf\{\.188\}\_\{\\scriptscriptstyle\[\.150,\.241\]\}\.451\[\.315,\.464\]\.451\_\{\\scriptscriptstyle\[\.315,\.464\]\}\.248\[\.212,\.284\]\.248\_\{\\scriptscriptstyle\[\.212,\.284\]\}SVAMP\.058\[\.033,\.087\]\.058\_\{\\scriptscriptstyle\[\.033,\.087\]\}\.999\[\.109,\.999\]\.999\_\{\\scriptscriptstyle\[\.109,\.999\]\}\.060\[\.034,\.088\]\.060\_\{\\scriptscriptstyle\[\.034,\.088\]\}\.061\[\.039,\.091\]\.061\_\{\\scriptscriptstyle\[\.039,\.091\]\}\.313\[\.246,\.477\]\.313\_\{\\scriptscriptstyle\[\.246,\.477\]\}\.061\[\.040,\.086\]\.061\_\{\\scriptscriptstyle\[\.040,\.086\]\}

\(b\) Multi\-pass estimators

Table 7:Full confidence\-estimation results using token probabilities\.We report point estimates with 95% percentile confidence intervals from 5,000 bootstrap resamples shown as subscripts\. Lower values are better for ECE, MCE, and Brier score\. Bold indicates the lowest ECE point estimate across all estimators for each model–dataset pair\.ModelDatasetIsotonic RegressionPlatt ScalingAns AvgAns Jointp​\(True\)p\(\\mathrm\{True\}\)ttAns AvgAns Joint\(a,b\)\(a,b\)p​\(True\)p\(\\mathrm\{True\}\)Llama\-3\.1\-8BGSM8K0\.056\[0\.038,0\.076\]​\(\+𝟔𝟔%\)\\mathbf\{0\.056\}\_\{\\scriptscriptstyle\[0\.038,0\.076\]\}\\,\\mathbf\{\(\+66\\%\)\}0\.054\[0\.036,0\.075\]​\(\+𝟔𝟕%\)\\mathbf\{0\.054\}\_\{\\scriptscriptstyle\[0\.036,0\.075\]\}\\,\\mathbf\{\(\+67\\%\)\}0\.058\[0\.041,0\.079\]​\(\+𝟒𝟎%\)\\mathbf\{0\.058\}\_\{\\scriptscriptstyle\[0\.041,0\.079\]\}\\,\\mathbf\{\(\+40\\%\)\}1\.820\.081\[0\.063,0\.101\]​\(\+52%\)0\.081\_\{\\scriptscriptstyle\[0\.063,0\.101\]\}\\,\(\+52\\%\)0\.080\[0\.064,0\.101\]​\(\+51%\)0\.080\_\{\\scriptscriptstyle\[0\.064,0\.101\]\}\\,\(\+51\\%\)\(0\.53,0\.48\)\(0\.53,0\.48\)0\.059\[0\.041,0\.079\]​\(\+39%\)0\.059\_\{\\scriptscriptstyle\[0\.041,0\.079\]\}\\,\(\+39\\%\)GSMHard0\.064\[0\.036,0\.110\]​\(\+𝟗𝟎%\)\\mathbf\{0\.064\}\_\{\\scriptscriptstyle\[0\.036,0\.110\]\}\\,\\mathbf\{\(\+90\\%\)\}0\.065\[0\.038,0\.110\]​\(\+89%\)0\.065\_\{\\scriptscriptstyle\[0\.038,0\.110\]\}\\,\(\+89\\%\)0\.057\[0\.037,0\.105\]​\(\+𝟖𝟑%\)\\mathbf\{0\.057\}\_\{\\scriptscriptstyle\[0\.037,0\.105\]\}\\,\\mathbf\{\(\+83\\%\)\}2\.370\.091\[0\.062,0\.141\]​\(\+85%\)0\.091\_\{\\scriptscriptstyle\[0\.062,0\.141\]\}\\,\(\+85\\%\)0\.045\[0\.036,0\.099\]​\(\+𝟗𝟑%\)\\mathbf\{0\.045\}\_\{\\scriptscriptstyle\[0\.036,0\.099\]\}\\,\\mathbf\{\(\+93\\%\)\}\(0\.57,−1\.74\)\(0\.57,\-1\.74\)0\.087\[0\.063,0\.131\]​\(\+74%\)0\.087\_\{\\scriptscriptstyle\[0\.063,0\.131\]\}\\,\(\+74\\%\)SVAMP0\.020\[0\.019,0\.070\]​\(\+𝟖𝟖%\)\\mathbf\{0\.020\}\_\{\\scriptscriptstyle\[0\.019,0\.070\]\}\\,\\mathbf\{\(\+88\\%\)\}0\.022\[0\.021,0\.070\]​\(\+𝟖𝟕%\)\\mathbf\{0\.022\}\_\{\\scriptscriptstyle\[0\.021,0\.070\]\}\\,\\mathbf\{\(\+87\\%\)\}0\.045\[0\.029,0\.093\]​\(\+54%\)0\.045\_\{\\scriptscriptstyle\[0\.029,0\.093\]\}\\,\(\+54\\%\)1\.910\.084\[0\.055,0\.125\]​\(\+50%\)0\.084\_\{\\scriptscriptstyle\[0\.055,0\.125\]\}\\,\(\+50\\%\)0\.083\[0\.054,0\.123\]​\(\+52%\)0\.083\_\{\\scriptscriptstyle\[0\.054,0\.123\]\}\\,\(\+52\\%\)\(0\.54,0\.43\)\(0\.54,0\.43\)0\.039\[0\.027,0\.089\]​\(\+𝟔𝟎%\)\\mathbf\{0\.039\}\_\{\\scriptscriptstyle\[0\.027,0\.089\]\}\\,\\mathbf\{\(\+60\\%\)\}Llama\-3\.2\-3BGSM8K0\.074\[0\.054,0\.096\]​\(\+𝟔𝟔%\)\\mathbf\{0\.074\}\_\{\\scriptscriptstyle\[0\.054,0\.096\]\}\\,\\mathbf\{\(\+66\\%\)\}0\.075\[0\.056,0\.097\]​\(\+𝟔𝟓%\)\\mathbf\{0\.075\}\_\{\\scriptscriptstyle\[0\.056,0\.097\]\}\\,\\mathbf\{\(\+65\\%\)\}0\.076\[0\.056,0\.098\]​\(−39%\)0\.076\_\{\\scriptscriptstyle\[0\.056,0\.098\]\}\\,\(\-39\\%\)1\.740\.104\[0\.084,0\.127\]​\(\+52%\)0\.104\_\{\\scriptscriptstyle\[0\.084,0\.127\]\}\\,\(\+52\\%\)0\.103\[0\.082,0\.124\]​\(\+53%\)0\.103\_\{\\scriptscriptstyle\[0\.082,0\.124\]\}\\,\(\+53\\%\)\(0\.65,1\.07\)\(0\.65,1\.07\)0\.044\[0\.034,0\.096\]​\(\+𝟐𝟎%\)\\mathbf\{0\.044\}\_\{\\scriptscriptstyle\[0\.034,0\.096\]\}\\,\\mathbf\{\(\+20\\%\)\}GSMHard0\.027\[0\.017,0\.073\]​\(\+𝟗𝟔%\)\\mathbf\{0\.027\}\_\{\\scriptscriptstyle\[0\.017,0\.073\]\}\\,\\mathbf\{\(\+96\\%\)\}0\.028\[0\.017,0\.075\]​\(\+𝟗𝟔%\)\\mathbf\{0\.028\}\_\{\\scriptscriptstyle\[0\.017,0\.075\]\}\\,\\mathbf\{\(\+96\\%\)\}0\.031\[0\.028,0\.077\]​\(\+𝟖𝟖%\)\\mathbf\{0\.031\}\_\{\\scriptscriptstyle\[0\.028,0\.077\]\}\\,\\mathbf\{\(\+88\\%\)\}2\.270\.105\[0\.079,0\.150\]​\(\+85%\)0\.105\_\{\\scriptscriptstyle\[0\.079,0\.150\]\}\\,\(\+85\\%\)0\.057\[0\.037,0\.102\]​\(\+92%\)0\.057\_\{\\scriptscriptstyle\[0\.037,0\.102\]\}\\,\(\+92\\%\)\(0\.79,−1\.36\)\(0\.79,\-1\.36\)0\.047\[0\.036,0\.091\]​\(\+81%\)0\.047\_\{\\scriptscriptstyle\[0\.036,0\.091\]\}\\,\(\+81\\%\)SVAMP0\.036\[0\.030,0\.081\]​\(\+𝟕𝟗%\)\\mathbf\{0\.036\}\_\{\\scriptscriptstyle\[0\.030,0\.081\]\}\\,\\mathbf\{\(\+79\\%\)\}0\.036\[0\.029,0\.081\]​\(\+𝟖𝟎%\)\\mathbf\{0\.036\}\_\{\\scriptscriptstyle\[0\.029,0\.081\]\}\\,\\mathbf\{\(\+80\\%\)\}0\.055\[0\.034,0\.099\]​\(\+48%\)0\.055\_\{\\scriptscriptstyle\[0\.034,0\.099\]\}\\,\(\+48\\%\)1\.790\.044\[0\.031,0\.090\]​\(\+75%\)0\.044\_\{\\scriptscriptstyle\[0\.031,0\.090\]\}\\,\(\+75\\%\)0\.041\[0\.029,0\.087\]​\(\+76%\)0\.041\_\{\\scriptscriptstyle\[0\.029,0\.087\]\}\\,\(\+76\\%\)\(0\.61,0\.80\)\(0\.61,0\.80\)0\.029\[0\.025,0\.083\]​\(\+𝟕𝟐%\)\\mathbf\{0\.029\}\_\{\\scriptscriptstyle\[0\.025,0\.083\]\}\\,\\mathbf\{\(\+72\\%\)\}DeepSeek\-R1\-8BGSM8K0\.012\[0\.008,0\.042\]​\(\+𝟗𝟔%\)\\mathbf\{0\.012\}\_\{\\scriptscriptstyle\[0\.008,0\.042\]\}\\,\\mathbf\{\(\+96\\%\)\}0\.013\[0\.007,0\.042\]​\(\+𝟗𝟔%\)\\mathbf\{0\.013\}\_\{\\scriptscriptstyle\[0\.007,0\.042\]\}\\,\\mathbf\{\(\+96\\%\)\}0\.017\[0\.006,0\.044\]​\(\+95%\)0\.017\_\{\\scriptscriptstyle\[0\.006,0\.044\]\}\\,\(\+95\\%\)1\.680\.071\[0\.053,0\.099\]​\(\+77%\)0\.071\_\{\\scriptscriptstyle\[0\.053,0\.099\]\}\\,\(\+77\\%\)0\.070\[0\.053,0\.099\]​\(\+77%\)0\.070\_\{\\scriptscriptstyle\[0\.053,0\.099\]\}\\,\(\+77\\%\)\(0\.054,0\.46\)\(0\.054,0\.46\)0\.016\[0\.003,0\.043\]​\(\+𝟗𝟓%\)\\mathbf\{0\.016\}\_\{\\scriptscriptstyle\[0\.003,0\.043\]\}\\,\\mathbf\{\(\+95\\%\)\}GSMHard0\.068\[0\.037,0\.115\]​\(\+𝟖𝟗%\)\\mathbf\{0\.068\}\_\{\\scriptscriptstyle\[0\.037,0\.115\]\}\\,\\mathbf\{\(\+89\\%\)\}0\.071\[0\.040,0\.120\]​\(\+𝟖𝟗%\)\\mathbf\{0\.071\}\_\{\\scriptscriptstyle\[0\.040,0\.120\]\}\\,\\mathbf\{\(\+89\\%\)\}0\.066\[0\.031,0\.117\]​\(\+90%\)0\.066\_\{\\scriptscriptstyle\[0\.031,0\.117\]\}\\,\(\+90\\%\)2\.150\.100\[0\.080,0\.154\]​\(\+84%\)0\.100\_\{\\scriptscriptstyle\[0\.080,0\.154\]\}\\,\(\+84\\%\)0\.082\[0\.057,0\.134\]​\(\+87%\)0\.082\_\{\\scriptscriptstyle\[0\.057,0\.134\]\}\\,\(\+87\\%\)\(0\.065,−1\.29\)\(0\.065,\-1\.29\)0\.065\[0\.020,0\.110\]​\(\+𝟗𝟎%\)\\mathbf\{0\.065\}\_\{\\scriptscriptstyle\[0\.020,0\.110\]\}\\,\\mathbf\{\(\+90\\%\)\}SVAMP0\.016\[0\.015,0\.076\]​\(\+𝟗𝟑%\)\\mathbf\{0\.016\}\_\{\\scriptscriptstyle\[0\.015,0\.076\]\}\\,\\mathbf\{\(\+93\\%\)\}0\.024\[0\.019,0\.082\]​\(\+𝟗𝟎%\)\\mathbf\{0\.024\}\_\{\\scriptscriptstyle\[0\.019,0\.082\]\}\\,\\mathbf\{\(\+90\\%\)\}0\.034\[0\.014,0\.087\]​\(\+86%\)0\.034\_\{\\scriptscriptstyle\[0\.014,0\.087\]\}\\,\(\+86\\%\)1\.670\.061\[0\.044,0\.122\]​\(\+75%\)0\.061\_\{\\scriptscriptstyle\[0\.044,0\.122\]\}\\,\(\+75\\%\)0\.066\[0\.046,0\.126\]​\(\+73%\)0\.066\_\{\\scriptscriptstyle\[0\.046,0\.126\]\}\\,\(\+73\\%\)\(0\.022,0\.83\)\(0\.022,0\.83\)0\.013\[0\.002,0\.064\]​\(\+𝟗𝟓%\)\\mathbf\{0\.013\}\_\{\\scriptscriptstyle\[0\.002,0\.064\]\}\\,\\mathbf\{\(\+95\\%\)\}Qwen3\-8B\-ThinkingGSM8K0\.015\[0\.005,0\.027\]​\(\+𝟔𝟑%\)\\mathbf\{0\.015\}\_\{\\scriptscriptstyle\[0\.005,0\.027\]\}\\,\\mathbf\{\(\+63\\%\)\}0\.015\[0\.005,0\.027\]​\(\+𝟔𝟑%\)\\mathbf\{0\.015\}\_\{\\scriptscriptstyle\[0\.005,0\.027\]\}\\,\\mathbf\{\(\+63\\%\)\}0\.015\[0\.004,0\.026\]​\(\+𝟓𝟖%\)\\mathbf\{0\.015\}\_\{\\scriptscriptstyle\[0\.004,0\.026\]\}\\,\\mathbf\{\(\+58\\%\)\}3\.190\.030\[0\.020,0\.042\]​\(\+29%\)0\.030\_\{\\scriptscriptstyle\[0\.020,0\.042\]\}\\,\(\+29\\%\)0\.030\[0\.020,0\.042\]​\(\+28%\)0\.030\_\{\\scriptscriptstyle\[0\.020,0\.042\]\}\\,\(\+28\\%\)\(0\.39,1\.54\)\(0\.39,1\.54\)0\.016\[0\.006,0\.028\]​\(\+54%\)0\.016\_\{\\scriptscriptstyle\[0\.006,0\.028\]\}\\,\(\+54\\%\)GSMHard0\.034\[0\.009,0\.077\]​\(\+𝟖𝟐%\)\\mathbf\{0\.034\}\_\{\\scriptscriptstyle\[0\.009,0\.077\]\}\\,\\mathbf\{\(\+82\\%\)\}0\.033\[0\.009,0\.076\]​\(\+𝟖𝟑%\)\\mathbf\{0\.033\}\_\{\\scriptscriptstyle\[0\.009,0\.076\]\}\\,\\mathbf\{\(\+83\\%\)\}0\.049\[0\.028,0\.094\]​\(\+𝟕𝟑%\)\\mathbf\{0\.049\}\_\{\\scriptscriptstyle\[0\.028,0\.094\]\}\\,\\mathbf\{\(\+73\\%\)\}3\.620\.114\[0\.078,0\.161\]​\(\+40%\)0\.114\_\{\\scriptscriptstyle\[0\.078,0\.161\]\}\\,\(\+40\\%\)0\.114\[0\.079,0\.162\]​\(\+40%\)0\.114\_\{\\scriptscriptstyle\[0\.079,0\.162\]\}\\,\(\+40\\%\)\(0\.52,−1\.38\)\(0\.52,\-1\.38\)0\.078\[0\.055,0\.124\]​\(\+57%\)0\.078\_\{\\scriptscriptstyle\[0\.055,0\.124\]\}\\,\(\+57\\%\)SVAMP0\.003\[0\.001,0\.022\]​\(\+𝟗𝟏%\)\\mathbf\{0\.003\}\_\{\\scriptscriptstyle\[0\.001,0\.022\]\}\\,\\mathbf\{\(\+91\\%\)\}0\.003\[0\.001,0\.022\]​\(\+𝟗𝟏%\)\\mathbf\{0\.003\}\_\{\\scriptscriptstyle\[0\.001,0\.022\]\}\\,\\mathbf\{\(\+91\\%\)\}0\.007\[0\.003,0\.026\]​\(\+66%\)0\.007\_\{\\scriptscriptstyle\[0\.003,0\.026\]\}\\,\(\+66\\%\)3\.220\.022\[0\.005,0\.042\]​\(\+20%\)0\.022\_\{\\scriptscriptstyle\[0\.005,0\.042\]\}\\,\(\+20\\%\)0\.022\[0\.005,0\.043\]​\(\+21%\)0\.022\_\{\\scriptscriptstyle\[0\.005,0\.043\]\}\\,\(\+21\\%\)\(0\.46,1\.22\)\(0\.46,1\.22\)0\.004\[0\.001,0\.024\]​\(\+𝟕𝟗%\)\\mathbf\{0\.004\}\_\{\\scriptscriptstyle\[0\.001,0\.024\]\}\\,\\mathbf\{\(\+79\\%\)\}Qwen3\-4B\-InstructGSM8K0\.019\[0\.006,0\.033\]​\(\+𝟔𝟗%\)\\mathbf\{0\.019\}\_\{\\scriptscriptstyle\[0\.006,0\.033\]\}\\,\\mathbf\{\(\+69\\%\)\}0\.019\[0\.006,0\.033\]​\(\+𝟔𝟗%\)\\mathbf\{0\.019\}\_\{\\scriptscriptstyle\[0\.006,0\.033\]\}\\,\\mathbf\{\(\+69\\%\)\}0\.020\[0\.008,0\.033\]​\(\+68%\)0\.020\_\{\\scriptscriptstyle\[0\.008,0\.033\]\}\\,\(\+68\\%\)3\.900\.044\[0\.031,0\.057\]​\(\+29%\)0\.044\_\{\\scriptscriptstyle\[0\.031,0\.057\]\}\\,\(\+29\\%\)0\.044\[0\.031,0\.057\]​\(\+29%\)0\.044\_\{\\scriptscriptstyle\[0\.031,0\.057\]\}\\,\(\+29\\%\)\(0\.18,1\.70\)\(0\.18,1\.70\)0\.019\[0\.007,0\.032\]​\(\+𝟕𝟎%\)\\mathbf\{0\.019\}\_\{\\scriptscriptstyle\[0\.007,0\.032\]\}\\,\\mathbf\{\(\+70\\%\)\}GSMHard0\.077\[0\.032,0\.123\]​\(\+𝟕𝟒%\)\\mathbf\{0\.077\}\_\{\\scriptscriptstyle\[0\.032,0\.123\]\}\\,\\mathbf\{\(\+74\\%\)\}0\.077\[0\.032,0\.124\]​\(\+𝟕𝟒%\)\\mathbf\{0\.077\}\_\{\\scriptscriptstyle\[0\.032,0\.124\]\}\\,\\mathbf\{\(\+74\\%\)\}0\.091\[0\.060,0\.140\]​\(\+𝟔𝟖%\)\\mathbf\{0\.091\}\_\{\\scriptscriptstyle\[0\.060,0\.140\]\}\\,\\mathbf\{\(\+68\\%\)\}4\.480\.131\[0\.097,0\.181\]​\(\+56%\)0\.131\_\{\\scriptscriptstyle\[0\.097,0\.181\]\}\\,\(\+56\\%\)0\.144\[0\.109,0\.193\]​\(\+52%\)0\.144\_\{\\scriptscriptstyle\[0\.109,0\.193\]\}\\,\(\+52\\%\)\(0\.11,−0\.18\)\(0\.11,\-0\.18\)0\.092\[0\.067,0\.146\]​\(\+68%\)0\.092\_\{\\scriptscriptstyle\[0\.067,0\.146\]\}\\,\(\+68\\%\)SVAMP0\.035\[0\.012,0\.062\]​\(\+𝟑𝟖%\)\\mathbf\{0\.035\}\_\{\\scriptscriptstyle\[0\.012,0\.062\]\}\\,\\mathbf\{\(\+38\\%\)\}0\.035\[0\.012,0\.062\]​\(\+𝟑𝟖%\)\\mathbf\{0\.035\}\_\{\\scriptscriptstyle\[0\.012,0\.062\]\}\\,\\mathbf\{\(\+38\\%\)\}0\.032\[0\.009,0\.059\]​\(\+44%\)0\.032\_\{\\scriptscriptstyle\[0\.009,0\.059\]\}\\,\(\+44\\%\)3\.880\.043\[0\.020,0\.070\]​\(\+24%\)0\.043\_\{\\scriptscriptstyle\[0\.020,0\.070\]\}\\,\(\+24\\%\)0\.044\[0\.020,0\.071\]​\(\+22%\)0\.044\_\{\\scriptscriptstyle\[0\.020,0\.071\]\}\\,\(\+22\\%\)\(−0\.30,6\.02\)\(\-0\.30,6\.02\)0\.032\[0\.008,0\.060\]​\(\+𝟒𝟓%\)\\mathbf\{0\.032\}\_\{\\scriptscriptstyle\[0\.008,0\.060\]\}\\,\\mathbf\{\(\+45\\%\)\}

Table 8:Post\-hoc calibration results\.We report calibrated ECE with 95% percentile bootstrap confidence intervals shown as subscripts forAns Avg,Ans Joint, andp​\(True\)p\(\\mathrm\{True\}\)after isotonic regression and Platt scaling\. Relative improvements over the corresponding uncalibrated estimators are shown in parentheses\. For Platt scaling, we also report the learned temperaturettfor answer\-based estimators and the learned parameters\(a,b\)\(a,b\)forp​\(True\)p\(\\mathrm\{True\}\)\. For each estimator and dataset,boldindicates the lower ECE between the two calibration methods\.ModelDatasetSeq AvgSeq JointAns AvgAns JointLlama\-3\.1\-8BGSM8K0\.054\[0\.036,0\.073\]0\.054\_\{\\scriptscriptstyle\[0\.036,0\.073\]\}\(−1%\)\(\-1\\%\)0\.056\[0\.039,0\.076\]0\.056\_\{\\scriptscriptstyle\[0\.039,0\.076\]\}\(\+52%\)\(\+52\\%\)0\.056\[0\.038,0\.076\]0\.056\_\{\\scriptscriptstyle\[0\.038,0\.076\]\}\(\+66%\)\(\+66\\%\)0\.054\[0\.036,0\.075\]\\mathbf\{0\.054\}\_\{\\scriptscriptstyle\[0\.036,0\.075\]\}\(\+67%\)\(\+67\\%\)GSMHard0\.055\[0\.029,0\.104\]0\.055\_\{\\scriptscriptstyle\[0\.029,0\.104\]\}\(\+88%\)\(\+88\\%\)0\.060\[0\.035,0\.107\]0\.060\_\{\\scriptscriptstyle\[0\.035,0\.107\]\}\(\+81%\)\(\+81\\%\)0\.064\[0\.036,0\.110\]\\mathbf\{0\.064\}\_\{\\scriptscriptstyle\[0\.036,0\.110\]\}\(\+90%\)\(\+90\\%\)0\.065\[0\.038,0\.110\]0\.065\_\{\\scriptscriptstyle\[0\.038,0\.110\]\}\(\+89%\)\(\+89\\%\)SVAMP0\.033\[0\.023,0\.081\]0\.033\_\{\\scriptscriptstyle\[0\.023,0\.081\]\}\(\+42%\)\(\+42\\%\)0\.030\[0\.017,0\.080\]0\.030\_\{\\scriptscriptstyle\[0\.017,0\.080\]\}\(\+82%\)\(\+82\\%\)0\.020\[0\.019,0\.070\]\\mathbf\{0\.020\}\_\{\\scriptscriptstyle\[0\.019,0\.070\]\}\(\+88%\)\(\+88\\%\)0\.022\[0\.021,0\.070\]0\.022\_\{\\scriptscriptstyle\[0\.021,0\.070\]\}\(\+87%\)\(\+87\\%\)Llama\-3\.2\-3BGSM8K0\.074\[0\.054,0\.096\]0\.074\_\{\\scriptscriptstyle\[0\.054,0\.096\]\}\(\+40%\)\(\+40\\%\)0\.074\[0\.053,0\.096\]0\.074\_\{\\scriptscriptstyle\[0\.053,0\.096\]\}\(\+20%\)\(\+20\\%\)0\.074\[0\.054,0\.096\]\\mathbf\{0\.074\}\_\{\\scriptscriptstyle\[0\.054,0\.096\]\}\(\+66%\)\(\+66\\%\)0\.075\[0\.056,0\.097\]0\.075\_\{\\scriptscriptstyle\[0\.056,0\.097\]\}\(\+65%\)\(\+65\\%\)GSMHard0\.032\[0\.021,0\.080\]0\.032\_\{\\scriptscriptstyle\[0\.021,0\.080\]\}\(\+95%\)\(\+95\\%\)0\.042\[0\.030,0\.091\]0\.042\_\{\\scriptscriptstyle\[0\.030,0\.091\]\}\(\+93%\)\(\+93\\%\)0\.027\[0\.017,0\.073\]\\mathbf\{0\.027\}\_\{\\scriptscriptstyle\[0\.017,0\.073\]\}\(\+96%\)\(\+96\\%\)0\.028\[0\.017,0\.075\]\\mathbf\{0\.028\}\_\{\\scriptscriptstyle\[0\.017,0\.075\]\}\(\+96%\)\(\+96\\%\)SVAMP0\.029\[0\.019,0\.075\]0\.029\_\{\\scriptscriptstyle\[0\.019,0\.075\]\}\(\+57%\)\(\+57\\%\)0\.040\[0\.022,0\.083\]0\.040\_\{\\scriptscriptstyle\[0\.022,0\.083\]\}\(−13%\)\(\-13\\%\)0\.036\[0\.030,0\.081\]0\.036\_\{\\scriptscriptstyle\[0\.030,0\.081\]\}\(\+79%\)\(\+79\\%\)0\.036\[0\.029,0\.081\]\\mathbf\{0\.036\}\_\{\\scriptscriptstyle\[0\.029,0\.081\]\}\(\+80%\)\(\+80\\%\)DeepSeek\-R1\-8BGSM8K0\.032\[0\.021,0\.060\]0\.032\_\{\\scriptscriptstyle\[0\.021,0\.060\]\}\(\+88%\)\(\+88\\%\)0\.031\[0\.022,0\.062\]0\.031\_\{\\scriptscriptstyle\[0\.022,0\.062\]\}\(\+86%\)\(\+86\\%\)0\.012\[0\.008,0\.042\]\\mathbf\{0\.012\}\_\{\\scriptscriptstyle\[0\.008,0\.042\]\}\(\+96%\)\(\+96\\%\)0\.013\[0\.007,0\.042\]\\mathbf\{0\.013\}\_\{\\scriptscriptstyle\[0\.007,0\.042\]\}\(\+96%\)\(\+96\\%\)GSMHard0\.060\[0\.032,0\.110\]0\.060\_\{\\scriptscriptstyle\[0\.032,0\.110\]\}\(\+90%\)\(\+90\\%\)0\.053\[0\.027,0\.101\]\\mathbf\{0\.053\}\_\{\\scriptscriptstyle\[0\.027,0\.101\]\}\(\+91%\)\(\+91\\%\)0\.068\[0\.037,0\.115\]0\.068\_\{\\scriptscriptstyle\[0\.037,0\.115\]\}\(\+89%\)\(\+89\\%\)0\.071\[0\.040,0\.120\]0\.071\_\{\\scriptscriptstyle\[0\.040,0\.120\]\}\(\+89%\)\(\+89\\%\)SVAMP0\.049\[0\.030,0\.101\]0\.049\_\{\\scriptscriptstyle\[0\.030,0\.101\]\}\(\+70%\)\(\+70\\%\)0\.068\[0\.037,0\.117\]0\.068\_\{\\scriptscriptstyle\[0\.037,0\.117\]\}\(\+47%\)\(\+47\\%\)0\.016\[0\.015,0\.076\]\\mathbf\{0\.016\}\_\{\\scriptscriptstyle\[0\.015,0\.076\]\}\(\+93%\)\(\+93\\%\)0\.024\[0\.019,0\.082\]0\.024\_\{\\scriptscriptstyle\[0\.019,0\.082\]\}\(\+90%\)\(\+90\\%\)Qwen3\-8B\-ThinkingGSM8K0\.007\[0\.006,0\.029\]0\.007\_\{\\scriptscriptstyle\[0\.006,0\.029\]\}\(−15%\)\(\-15\\%\)0\.017\[0\.007,0\.029\]0\.017\_\{\\scriptscriptstyle\[0\.007,0\.029\]\}\(\+8%\)\(\+8\\%\)0\.015\[0\.005,0\.027\]0\.015\_\{\\scriptscriptstyle\[0\.005,0\.027\]\}\(\+63%\)\(\+63\\%\)0\.015\[0\.005,0\.027\]0\.015\_\{\\scriptscriptstyle\[0\.005,0\.027\]\}\(\+63%\)\(\+63\\%\)GSMHard0\.019\[0\.014,0\.067\]\\mathbf\{0\.019\}\_\{\\scriptscriptstyle\[0\.014,0\.067\]\}\(\+87%\)\(\+87\\%\)0\.036\[0\.023,0\.082\]0\.036\_\{\\scriptscriptstyle\[0\.023,0\.082\]\}\(\+71%\)\(\+71\\%\)0\.034\[0\.009,0\.077\]0\.034\_\{\\scriptscriptstyle\[0\.009,0\.077\]\}\(\+82%\)\(\+82\\%\)0\.033\[0\.009,0\.076\]0\.033\_\{\\scriptscriptstyle\[0\.009,0\.076\]\}\(\+83%\)\(\+83\\%\)SVAMP0\.005\[0\.004,0\.027\]0\.005\_\{\\scriptscriptstyle\[0\.004,0\.027\]\}\(\+70%\)\(\+70\\%\)0\.008\[0\.005,0\.031\]0\.008\_\{\\scriptscriptstyle\[0\.005,0\.031\]\}\(\+75%\)\(\+75\\%\)0\.003\[0\.001,0\.022\]\\mathbf\{0\.003\}\_\{\\scriptscriptstyle\[0\.001,0\.022\]\}\(\+91%\)\(\+91\\%\)0\.003\[0\.001,0\.022\]\\mathbf\{0\.003\}\_\{\\scriptscriptstyle\[0\.001,0\.022\]\}\(\+91%\)\(\+91\\%\)Qwen3\-4B\-InstructGSM8K0\.014\[0\.006,0\.027\]0\.014\_\{\\scriptscriptstyle\[0\.006,0\.027\]\}\(\+60%\)\(\+60\\%\)0\.012\[0\.005,0\.025\]0\.012\_\{\\scriptscriptstyle\[0\.005,0\.025\]\}\(\+59%\)\(\+59\\%\)0\.019\[0\.006,0\.033\]\\mathbf\{0\.019\}\_\{\\scriptscriptstyle\[0\.006,0\.033\]\}\(\+69%\)\(\+69\\%\)0\.019\[0\.006,0\.033\]\\mathbf\{0\.019\}\_\{\\scriptscriptstyle\[0\.006,0\.033\]\}\(\+69%\)\(\+69\\%\)GSMHard0\.050\[0\.033,0\.095\]\\mathbf\{0\.050\}\_\{\\scriptscriptstyle\[0\.033,0\.095\]\}\(\+81%\)\(\+81\\%\)0\.052\[0\.041,0\.099\]0\.052\_\{\\scriptscriptstyle\[0\.041,0\.099\]\}\(\+80%\)\(\+80\\%\)0\.077\[0\.032,0\.123\]0\.077\_\{\\scriptscriptstyle\[0\.032,0\.123\]\}\(\+74%\)\(\+74\\%\)0\.077\[0\.032,0\.124\]0\.077\_\{\\scriptscriptstyle\[0\.032,0\.124\]\}\(\+74%\)\(\+74\\%\)SVAMP0\.036\[0\.015,0\.061\]0\.036\_\{\\scriptscriptstyle\[0\.015,0\.061\]\}\(−12%\)\(\-12\\%\)0\.035\[0\.014,0\.060\]0\.035\_\{\\scriptscriptstyle\[0\.014,0\.060\]\}\(−38%\)\(\-38\\%\)0\.035\[0\.012,0\.062\]0\.035\_\{\\scriptscriptstyle\[0\.012,0\.062\]\}\(\+38%\)\(\+38\\%\)0\.035\[0\.012,0\.062\]0\.035\_\{\\scriptscriptstyle\[0\.012,0\.062\]\}\(\+38%\)\(\+38\\%\)

\(a\) Single\-pass estimators

ModelDatasetp​\(True\)p\(\\mathrm\{True\}\)cBALDc\_\{\\mathrm\{BALD\}\}Llama\-3\.1\-8BGSM8K0\.058\[0\.041,0\.079\]0\.058\_\{\\scriptscriptstyle\[0\.041,0\.079\]\}\(\+40%\)\(\+40\\%\)0\.062\[0\.043,0\.084\]0\.062\_\{\\scriptscriptstyle\[0\.043,0\.084\]\}\(\+58%\)\(\+58\\%\)GSMHard0\.057\[0\.037,0\.105\]0\.057\_\{\\scriptscriptstyle\[0\.037,0\.105\]\}\(\+83%\)\(\+83\\%\)0\.122\[0\.080,0\.170\]0\.122\_\{\\scriptscriptstyle\[0\.080,0\.170\]\}\(\+20%\)\(\+20\\%\)SVAMP0\.045\[0\.029,0\.093\]0\.045\_\{\\scriptscriptstyle\[0\.029,0\.093\]\}\(\+54%\)\(\+54\\%\)0\.040\[0\.025,0\.086\]0\.040\_\{\\scriptscriptstyle\[0\.025,0\.086\]\}\(\+58%\)\(\+58\\%\)Llama\-3\.2\-3BGSM8K0\.076\[0\.056,0\.098\]0\.076\_\{\\scriptscriptstyle\[0\.056,0\.098\]\}\(−39%\)\(\-39\\%\)0\.048\[0\.026,0\.072\]0\.048\_\{\\scriptscriptstyle\[0\.026,0\.072\]\}\(\+56%\)\(\+56\\%\)GSMHard0\.031\[0\.028,0\.077\]0\.031\_\{\\scriptscriptstyle\[0\.028,0\.077\]\}\(\+88%\)\(\+88\\%\)0\.067\[0\.021,0\.105\]0\.067\_\{\\scriptscriptstyle\[0\.021,0\.105\]\}\(\+14%\)\(\+14\\%\)SVAMP0\.055\[0\.034,0\.099\]0\.055\_\{\\scriptscriptstyle\[0\.034,0\.099\]\}\(\+48%\)\(\+48\\%\)0\.060\[0\.027,0\.103\]0\.060\_\{\\scriptscriptstyle\[0\.027,0\.103\]\}\(\+38%\)\(\+38\\%\)DeepSeek\-R1\-8BGSM8K0\.017\[0\.006,0\.044\]0\.017\_\{\\scriptscriptstyle\[0\.006,0\.044\]\}\(\+95%\)\(\+95\\%\)0\.095\[0\.075,0\.123\]0\.095\_\{\\scriptscriptstyle\[0\.075,0\.123\]\}\(\+43%\)\(\+43\\%\)GSMHard0\.066\[0\.031,0\.117\]0\.066\_\{\\scriptscriptstyle\[0\.031,0\.117\]\}\(\+90%\)\(\+90\\%\)0\.088\[0\.056,0\.133\]0\.088\_\{\\scriptscriptstyle\[0\.056,0\.133\]\}\(\+14%\)\(\+14\\%\)SVAMP0\.034\[0\.014,0\.087\]0\.034\_\{\\scriptscriptstyle\[0\.014,0\.087\]\}\(\+86%\)\(\+86\\%\)0\.053\[0\.016,0\.110\]0\.053\_\{\\scriptscriptstyle\[0\.016,0\.110\]\}\(\+70%\)\(\+70\\%\)Qwen3\-8B\-ThinkingGSM8K0\.015\[0\.004,0\.026\]0\.015\_\{\\scriptscriptstyle\[0\.004,0\.026\]\}\(\+58%\)\(\+58\\%\)0\.004\[0\.000,0\.015\]\\mathbf\{0\.004\}\_\{\\scriptscriptstyle\[0\.000,0\.015\]\}\(\+92%\)\(\+92\\%\)GSMHard0\.049\[0\.028,0\.094\]0\.049\_\{\\scriptscriptstyle\[0\.028,0\.094\]\}\(\+73%\)\(\+73\\%\)0\.032\[0\.015,0\.081\]0\.032\_\{\\scriptscriptstyle\[0\.015,0\.081\]\}\(−14%\)\(\-14\\%\)SVAMP0\.007\[0\.003,0\.026\]0\.007\_\{\\scriptscriptstyle\[0\.003,0\.026\]\}\(\+66%\)\(\+66\\%\)0\.031\[0\.024,0\.055\]0\.031\_\{\\scriptscriptstyle\[0\.024,0\.055\]\}\(−20%\)\(\-20\\%\)Qwen3\-4B\-InstructGSM8K0\.020\[0\.008,0\.033\]0\.020\_\{\\scriptscriptstyle\[0\.008,0\.033\]\}\(\+68%\)\(\+68\\%\)0\.032\[0\.020,0\.045\]0\.032\_\{\\scriptscriptstyle\[0\.020,0\.045\]\}\(\+47%\)\(\+47\\%\)GSMHard0\.091\[0\.060,0\.140\]0\.091\_\{\\scriptscriptstyle\[0\.060,0\.140\]\}\(\+68%\)\(\+68\\%\)0\.088\[0\.051,0\.137\]0\.088\_\{\\scriptscriptstyle\[0\.051,0\.137\]\}\(\+53%\)\(\+53\\%\)SVAMP0\.032\[0\.009,0\.059\]0\.032\_\{\\scriptscriptstyle\[0\.009,0\.059\]\}\(\+44%\)\(\+44\\%\)0\.029\[0\.010,0\.058\]\\mathbf\{0\.029\}\_\{\\scriptscriptstyle\[0\.010,0\.058\]\}\(\+53%\)\(\+53\\%\)

\(b\) Multi\-pass estimators

Table 9:Calibration performance after applying isotonic regression to confidence estimators\.We report ECE with 95% percentile bootstrap confidence intervals shown as subscripts and relative improvements over the corresponding uncalibrated estimators shown in parentheses\. Bold indicates the largest relative improvement across all estimators for each model–dataset pair; ties at the reported precision are all bolded\.
#### Cross\-domain calibration transfer\.

Table[10](https://arxiv.org/html/2608.07827#A4.T10)expands the transfer results in Table[4](https://arxiv.org/html/2608.07827#S6.T4)by reporting ECE, MCE, and Brier for each source–target pair\. We notice that cross\-domain calibration is strongly asymmetric: mappings fitted on GSM8K improve calibration on GSMHard for both models and calibration methods, whereas transfer from GSMHard to GSM8K generally degrades calibration\.

Similarly, cross\-model transfer is also target\-dependent\. On GSMHard, transferring between Llama and Qwen improves ECE in both directions, whereas on GSM8K, transfer from Llama to Qwen produces little change and the reverse direction substantially worsens calibration\.

#### In\-situ self\-verification\.

We compare in\-situp​\(True\)p\(\\mathrm\{True\}\)with standardp​\(True\)p\(\\mathrm\{True\}\)andAns Avgin Table[11](https://arxiv.org/html/2608.07827#A4.T11)\. Compared with standardp​\(True\)p\(\\mathrm\{True\}\), it matches or improves ECE in 10 of the 15 settings\.

CalibrationSource\\\\backslashTargetLlama / GSM8KLlama / GSMHardQwen / GSM8KQwen / GSMHardECEMCEBrierECEMCEBrierECEMCEBrierECEMCEBrierUncalibrated–0\.1660\.5290\.1670\.6220\.7430\.6060\.0620\.0620\.0620\.2980\.2990\.298IsotonicLlama / GSM8K0\.0560\.5000\.1340\.4130\.4900\.3620\.0620\.0620\.0620\.2930\.2950\.295Llama / GSMHard0\.3550\.6450\.2630\.0640\.1630\.1870\.3410\.3410\.1740\.1130\.6530\.224Qwen / GSM8K0\.6240\.6240\.5340\.1500\.1500\.2500\.0190\.8000\.0590\.2640\.7890\.279Qwen / GSMHard0\.3450\.3610\.2550\.0950\.1680\.2040\.3040\.5000\.1500\.0770\.2880\.217PlattLlama / GSM8K0\.0800\.2130\.1340\.5390\.6630\.4830\.0620\.5860\.0620\.2980\.2980\.298Llama / GSMHard0\.3790\.4450\.2640\.0910\.1750\.2190\.0620\.5670\.0620\.2950\.8740\.295Qwen / GSM8K0\.8240\.8240\.8150\.2950\.2950\.2990\.0440\.7160\.0590\.2440\.7590\.260Qwen / GSMHard0\.8300\.8300\.8270\.3040\.3040\.3060\.0510\.8240\.0580\.1310\.2140\.210

Table 10:Cross\-domain calibration transfer forAns Avg\.Each calibrator is fitted on the source model–dataset pair shown in the rows and applied without refitting to the target pair shown in the columns\. We report ECE, MCE, and Brier score; lower values are better\. Bold indicates in\-domain calibration, where the source and target pairs match\. Llama denotes Llama\-3\.1\-8B, and Qwen denotes Qwen3\-4B\-Instruct\.ModelDatasetAns AvgIn\-situp​\(True\)p\(\\mathrm\{True\}\)p​\(True\)p\(\\mathrm\{True\}\)Reduction \(%\)Llama\-3\.1\-8BGSM8K0\.1660\.0300\.09788\.75GSMHard0\.6220\.4650\.33389\.50SVAMP0\.1690\.0610\.09986\.13Llama\-3\.2\-3BGSM8K0\.2180\.0920\.05586\.58GSMHard0\.7250\.4810\.25189\.01SVAMP0\.1750\.1540\.10485\.50DeepSeek\-R1\-8BGSM8K0\.3040\.2700\.31591\.43GSMHard0\.6370\.6080\.63491\.92SVAMP0\.2440\.1920\.24189\.36Qwen3\-8B\-ThinkingGSM8K0\.0420\.0270\.03591\.27GSMHard0\.1900\.1680\.18393\.08SVAMP0\.0280\.0100\.02089\.48Qwen3\-4B\-InstructGSM8K0\.0620\.0620\.06283\.48GSMHard0\.2980\.2870\.28586\.87SVAMP0\.0570\.0560\.05878\.77Table 11:In\-situp​\(True\)p\(\\mathrm\{True\}\)calibration and token\-processing reduction\.We compareAns Avg, in\-situp​\(True\)p\(\\mathrm\{True\}\), and standardp​\(True\)p\(\\mathrm\{True\}\)in terms of ECE\. The last column reports the percentage reduction in token\-processing overhead for in\-situp​\(True\)p\(\\mathrm\{True\}\)relative to standardp​\(True\)p\(\\mathrm\{True\}\)\.ModelDatasetBinsSeq AvgSeq JointAns AvgAns JointECEMCEBrierECEMCEBrierECEMCEBrierECEMCEBrierLlama\-3\.1\-8BGSM8K100\.0530\.4810\.1310\.1170\.1950\.1380\.1660\.5290\.1670\.1650\.4690\.167200\.0530\.5160\.1310\.1230\.2750\.1380\.1670\.6410\.1670\.1650\.4690\.167300\.0540\.5550\.1310\.1220\.3120\.1380\.1670\.6410\.1670\.1660\.5180\.167GSMHard100\.4570\.6000\.4160\.3200\.4140\.2990\.6220\.7430\.6060\.6150\.7530\.596200\.4570\.6830\.4160\.3200\.5260\.2990\.6220\.8280\.6060\.6150\.8320\.596300\.4570\.7200\.4160\.3200\.5490\.2990\.6220\.8510\.6060\.6150\.8810\.596SVAMP100\.0570\.3090\.1330\.1700\.3320\.1600\.1690\.7970\.1690\.1710\.7650\.169200\.0610\.4830\.1330\.1760\.4860\.1600\.1700\.7970\.1690\.1710\.7980\.169300\.0660\.4900\.1330\.1810\.4860\.1600\.1700\.9320\.1690\.1710\.9290\.169Qwen3\-8B\-ThinkingGSM8K100\.0060\.0700\.0390\.0180\.0270\.0400\.0420\.0420\.0420\.0420\.0420\.042200\.0090\.0700\.0390\.0210\.1050\.0400\.0420\.0420\.0420\.0420\.0420\.042300\.0070\.1380\.0390\.0210\.1740\.0400\.0420\.0420\.0420\.0420\.0420\.042GSMHard100\.1440\.7810\.1690\.1280\.7930\.1630\.1900\.1900\.1900\.1900\.1900\.190200\.1440\.8420\.1690\.1280\.8330\.1630\.1900\.1900\.1900\.1910\.1910\.190300\.1440\.8540\.1690\.1280\.8240\.1630\.1900\.1910\.1900\.1900\.1920\.190SVAMP100\.0170\.0170\.0260\.0330\.0340\.0270\.0280\.0280\.0280\.0280\.0280\.028200\.0170\.0200\.0260\.0330\.0800\.0270\.0280\.0280\.0280\.0280\.0280\.028300\.0310\.8580\.0260\.0400\.3210\.0270\.0280\.0280\.0280\.0280\.0280\.028

Table 12:Effect of bin count on calibration metrics\.We report ECE, MCE, and Brier score for different bin counts across datasets and confidence estimators\.Standardp​\(True\)p\(\\mathrm\{True\}\)[⬇](data:text/plain;base64,PHxiZWdpbl9vZl90ZXh0fD48fHN0YXJ0X2hlYWRlcl9pZHw+c3lzdGVtPHxlbmRfaGVhZGVyX2lkfD4KCllvdSBhcmUgYSBoZWxwZnVsLCByZXNwZWN0ZnVsLCBhbmQgaG9uZXN0IGFzc2lzdGFudC4gQWx3YXlzIGFuc3dlciBhcyBoZWxwZnVsbHkgYXMgcG9zc2libGUsIHdoaWxlIGJlaW5nIHNhZmUuIC4uLgoKPHxzdGFydF9oZWFkZXJfaWR8PnVzZXI8fGVuZF9oZWFkZXJfaWR8PgpRdWVzdGlvbjoKTWFyeSBpcyBiYWtpbmcgYSBjYWtlLiBUaGUgcmVjaXBlIGNhbGxzIGZvciA2IGN1cHMgb2YgZmxvdXIsIDggY3VwcyBvZiBzdWdhciwKYW5kIDcgY3VwcyBvZiBzYWx0LiBTaGUgYWxyZWFkeSBwdXQgaW4gNSBjdXBzIG9mIGZsb3VyLgpIb3cgbWFueSBtb3JlIGN1cHMgb2Ygc3VnYXIgdGhhbiBjdXBzIG9mIHNhbHQgZG9lcyBzaGUgbmVlZCB0byBhZGQgbm93PwoKUHJvcG9zZWQgYW5zd2VyOgoKVG8gc29sdmUgdGhpcyBwcm9ibGVtLCBsZXQncyBicmVhayBpdCBkb3duIHN0ZXAgYnkgc3RlcDoKMS4gVGhlIHJlY2lwZSBjYWxscyBmb3IgOCBjdXBzIG9mIHN1Z2FyIGFuZCA3IGN1cHMgb2Ygc2FsdC4KMi4gRGlmZmVyZW5jZSA9IDggLSA3ID0gMS4KCiMjIyMgMQoKSXMgdGhlIHByb3Bvc2VkIGFuc3dlcjoKQTogVHJ1ZQpCOiBGYWxzZQpZb3UgbXVzdCBhbnN3ZXIgd2l0aCBleGFjdGx5IG9uZSBjaGFyYWN0ZXI6IGVpdGhlciBgQWAgb3IgYEJgLgoKVGhlIHByb3Bvc2VkIGFuc3dlciBpczoKCg==)<\|begin\_of\_text\|\><\|start\_header\_id\|\>system<\|end\_header\_id\|\>Youareahelpful,respectful,andhonestassistant\.Alwaysanswerashelpfullyaspossible,whilebeingsafe\.\.\.\.<\|start\_header\_id\|\>user<\|end\_header\_id\|\>Question:Maryisbakingacake\.Therecipecallsfor6cupsofflour,8cupsofsugar,and7cupsofsalt\.Shealreadyputin5cupsofflour\.Howmanymorecupsofsugarthancupsofsaltdoessheneedtoaddnow?Proposedanswer:Tosolvethisproblem,let’sbreakitdownstepbystep:1\.Therecipecallsfor8cupsofsugarand7cupsofsalt\.2\.Difference=8\-7=1\.\#\#\#\#1Istheproposedanswer:A:TrueB:FalseYoumustanswerwithexactlyonecharacter:either‘A‘or‘B‘\.Theproposedansweris:

In\-situp​\(True\)p\(\\mathrm\{True\}\)[⬇](data:text/plain;base64,PHxiZWdpbl9vZl90ZXh0fD48fHN0YXJ0X2hlYWRlcl9pZHw+c3lzdGVtPHxlbmRfaGVhZGVyX2lkfD4KCllvdSBhcmUgYSBoZWxwZnVsLCByZXNwZWN0ZnVsLCBhbmQgaG9uZXN0IGFzc2lzdGFudC4gQWx3YXlzIGFuc3dlciBhcyBoZWxwZnVsbHkgYXMgcG9zc2libGUsIHdoaWxlIGJlaW5nIHNhZmUuIC4uLgoKPHxzdGFydF9oZWFkZXJfaWR8PnVzZXI8fGVuZF9oZWFkZXJfaWR8PgoKWW91IGFyZSBhIG1hdGhlbWF0aWNzIGV4cGVydC4gRmlyc3QsIHN0dWR5IHRoaXMgZXhhbXBsZToKUXVlc3Rpb246IFNhcmFoIGhhcyAxMCBtYXJibGVzLCBnaXZlcyAzIGF3YXkuCkFuc3dlcjogU2FyYWggd291bGQgaGF2ZSAxMCAtIDMgPSA8PDEwLTM9Nz4+NyBtYXJibGVzIHJlbWFpbmluZy4KIyMjIyA3CgpOZXh0LCBzb2x2ZToKTWFyeSBpcyBiYWtpbmcgYSBjYWtlLiBUaGUgcmVjaXBlIGNhbGxzIGZvciA2IGN1cHMgb2YgZmxvdXIsIDggY3VwcyBvZiBzdWdhciwKYW5kIDcgY3VwcyBvZiBzYWx0LiBTaGUgYWxyZWFkeSBwdXQgaW4gNSBjdXBzIG9mIGZsb3VyLgpIb3cgbWFueSBtb3JlIGN1cHMgb2Ygc3VnYXIgdGhhbiBjdXBzIG9mIHNhbHQgZG9lcyBzaGUgbmVlZCB0byBhZGQgbm93PwoKQXQgdGhlIGVuZCwgd3JpdGUgdGhlIGFuc3dlciBhcyBhbiBpbnRlZ2VyIGFmdGVyICcjIyMjICcuCgo8fHN0YXJ0X2hlYWRlcl9pZHw+YXNzaXN0YW50PHxlbmRfaGVhZGVyX2lkfD4KClRvIHNvbHZlIHRoaXMgcHJvYmxlbSwgbGV0J3MgYnJlYWsgaXQgZG93biBzdGVwIGJ5IHN0ZXA6Ci4uLgojIyMjIDEKCklzIHRoZSBwcm9wb3NlZCBhbnN3ZXI6CkE6IFRydWUKQjogRmFsc2UKWW91IG11c3QgYW5zd2VyIHdpdGggZXhhY3RseSBvbmUgY2hhcmFjdGVyOiBlaXRoZXIgYEFgIG9yIGBCYC4KClRoZSBwcm9wb3NlZCBhbnN3ZXIgaXM6)<\|begin\_of\_text\|\><\|start\_header\_id\|\>system<\|end\_header\_id\|\>Youareahelpful,respectful,andhonestassistant\.Alwaysanswerashelpfullyaspossible,whilebeingsafe\.\.\.\.<\|start\_header\_id\|\>user<\|end\_header\_id\|\>Youareamathematicsexpert\.First,studythisexample:Question:Sarahhas10marbles,gives3away\.Answer:Sarahwouldhave10\-3=<<10\-3=7\>\>7marblesremaining\.\#\#\#\#7Next,solve:Maryisbakingacake\.Therecipecallsfor6cupsofflour,8cupsofsugar,and7cupsofsalt\.Shealreadyputin5cupsofflour\.Howmanymorecupsofsugarthancupsofsaltdoessheneedtoaddnow?Attheend,writetheanswerasanintegerafter’\#\#\#\#’\.<\|start\_header\_id\|\>assistant<\|end\_header\_id\|\>Tosolvethisproblem,let’sbreakitdownstepbystep:\.\.\.\#\#\#\#1Istheproposedanswer:A:TrueB:FalseYoumustanswerwithexactlyonecharacter:either‘A‘or‘B‘\.Theproposedansweris:

Figure 4:Standardp​\(True\)p\(\\mathrm\{True\}\)vs\. in\-situp​\(True\)p\(\\mathrm\{True\}\)on GSM8K with Llama\-3\.1\-8B\. Left: standardp​\(True\)p\(\\mathrm\{True\}\)self\-verification via re\-prompting, where the full question–answer pair is re\-encoded\. Right: in\-situp​\(True\)p\(\\mathrm\{True\}\), where the verification prompt is appended directly to the original generation trajectory and only the appended verification tokens require additional encoding\.

## Appendix ELicensing of Artifacts

Our experiments use GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.07827#bib.bib25)\), GSMHard\(Gaoet al\.,[2023](https://arxiv.org/html/2608.07827#bib.bib26)\), and SVAMP\(Patelet al\.,[2021](https://arxiv.org/html/2608.07827#bib.bib27)\), three publicly available mathematical question\-answering benchmarks released for research use\. Our BALD\-score computation is adapted from the publicly available BAAL implementation\(Atighehchianet al\.,[2022](https://arxiv.org/html/2608.07827#bib.bib9)\), which is distributed under the Apache License 2\.0\. To support reproducibility, we will release our implementation under the GPL\-3\.0 license\.

## Appendix FUse of AI Assistants

We used LLMs, including GPT, only to polish the writing \(e\.g\., grammar and phrasing\)\. All research ideas, methods, experiments, and substantive content were conceived and written by the authors\.