ConfidenceBench: Evaluating Confidence Calibration in Large Language Models

# Evaluating Confidence Calibration in Large Language Models
Source: [https://arxiv.org/html/2607.20526](https://arxiv.org/html/2607.20526)
Matthew ffrench\-ConstantDaniel YangIndependent ResearcherETH Zurichmatt\.ffrench\-constant@cantab\.ac\.ukdaniel\.yang@inf\.ethz\.chXinmeng HuangSanyam KapoorUniversity of PennsylvaniaNew York Universityxinmengh@sas\.upenn\.edusanyam@nyu\.edu

###### Abstract

Large language models \(LLMs\) are increasingly deployed in settings where fluent but incorrect answers can be costly\. In these settings, accuracy alone is insufficient: models must also know when they are likely to be wrong\. We presentConfidenceBench, a calibration benchmark that evaluates verbalized confidence estimates in 15 frontier LLMs using the Brier score, a proper scoring rule that incentivises truthful probability reporting\. Confidence is elicited via prompting, requiring no access to model logits and making the framework applicable to both closed\-source and open\-source systems\. The benchmark comprises 200 private multiple\-choice questions across four categories: spatial reasoning, high\-precision mathematics, word lookup, and unknowable questions\. Across three independent runs, Claude Opus 4\.6 and Gemini 3\.1 Pro Preview achieve the lowest Brier scores, both reported as0\.1030\.103\. Both substantially outperform the calibrated\-random baseline of0\.18750\.1875, while Gemini 3\.1 Flash\-Lite scores0\.3670\.367, indicating severe miscalibration\. Accuracy and calibration diverge substantially across model families: the most accurate model is not the best\-calibrated, and several models perform worse than the calibrated\-random Brier baseline despite reasonable accuracy\. These results show that verbalized confidence calibration is a distinct and practically important axis of LLM reliability, complementary to standard accuracy\-based evaluation\.

## 1Introduction

Large language models \(LLMs\) are increasingly deployed in settings where incorrect answers can be costly, from software engineering and scientific assistance to education, medicine, and decision support\(Zhanget al\.,[2026](https://arxiv.org/html/2607.20526#bib.bib18)\)\. Yet even highly capable models frequently produce fluent but incorrect outputs, hallucinate unsupported claims\(Kadavathet al\.,[2022](https://arxiv.org/html/2607.20526#bib.bib4)\), or answer questions that should be treated as uncertain\. In such settings, accuracy alone is insufficient: a model that is wrong with low confidence may still be useful, while a model that is wrong with high confidence can be actively harmful\.

Reliable uncertainty estimates are therefore a prerequisite for safe and effective deployment\. If an LLM can accurately report when it is likely to be wrong, downstream systems can route difficult queries to stronger models, abstain from answering, request human review, or allocate additional computation only when needed\. This makes confidence calibration central not only to model evaluation, but also to practical deployment decisions in reliability\-critical applications\(Zhanget al\.,[2026](https://arxiv.org/html/2607.20526#bib.bib18)\)\.

Uncertainty quantification \(UQ\) provides a principled framework for this problem\. For LLMs, two broad families of UQ methods exist\.*White\-box*methods extract uncertainty from internal token\-level probability distributions, providing theoretically grounded estimates but requiring direct access to model internals, which is often unavailable for commercial frontier models served via API\.*Black\-box verbalized*methods instead prompt models to state their confidence explicitly, yielding a universally applicable signal at the cost of relying on the model’s self\-knowledge\(Kadavathet al\.,[2022](https://arxiv.org/html/2607.20526#bib.bib4); Linet al\.,[2022](https://arxiv.org/html/2607.20526#bib.bib5); Tianet al\.,[2023](https://arxiv.org/html/2607.20526#bib.bib6)\)\. Verbalized elicitation is especially attractive because it can be applied uniformly across closed\-source and open\-source systems without requiring logits, model weights, or repeated sampling\.

Despite growing interest in LLM calibration, existing benchmarks remain largely accuracy\-centric\. Standard evaluations such as MMLU\(Hendryckset al\.,[2020](https://arxiv.org/html/2607.20526#bib.bib1)\)and BIG\-Bench\(Srivastavaet al\.,[2023](https://arxiv.org/html/2607.20526#bib.bib2)\)tell us what models answer correctly, but not whether their stated confidence is reliable\. Recent uncertainty benchmarks have begun to address this gap, but many aggregate broad datasets or focus on abstention\(Wanget al\.,[2025](https://arxiv.org/html/2607.20526#bib.bib16); Yeet al\.,[2024](https://arxiv.org/html/2607.20526#bib.bib9); Kirichenkoet al\.,[2025](https://arxiv.org/html/2607.20526#bib.bib10)\)rather than directly evaluating verbalized probability calibration across frontier models with a proper scoring rule\. As a result, we still lack a compact benchmark designed specifically to isolate different calibration failure modes in modern LLMs\.

We introduceConfidenceBench, a calibration benchmark for evaluating verbalized confidence in frontier LLMs\. The benchmark comprises 200 private multiple\-choice questions across four categories: spatial reasoning, high\-precision mathematics, word lookup, and unknowable questions\. These categories are designed to expose qualitatively distinct epistemic failure modes, including overconfidence on plausible but false recall, multi\-step reasoning errors, and uncertainty under genuinely inaccessible information\.

Our contributions are threefold\. First, we present a controlled private benchmark that directly evaluates stated confidence rather than accuracy alone\. Second, we evaluate 15 frontier LLMs across three independent runs using the Brier score, a proper scoring rule that incentivises truthful probability reporting\. Third, we provide a human baseline and category\-level analyses showing that accuracy and calibration can diverge substantially across model families\. Together, these results demonstrate that confidence calibration is an essential dimension of LLM reliability, complementary to standard accuracy\-based evaluation\.

## 2Related Work

#### Confidence elicitation\.

A central question in LLM reliability research is whether models can accurately assess and report their own uncertainty\. Early work byKadavathet al\.\([2022](https://arxiv.org/html/2607.20526#bib.bib4)\)andLinet al\.\([2022](https://arxiv.org/html/2607.20526#bib.bib5)\)shows that models can express calibrated confidence when prompted appropriately\. Subsequent studies demonstrate that simple elicitation strategies \(asking models to report probabilities\) can yield meaningful confidence estimates even in black\-box settings\(Tianet al\.,[2023](https://arxiv.org/html/2607.20526#bib.bib6); Xionget al\.,[2023](https://arxiv.org/html/2607.20526#bib.bib7)\)\(Yanget al\.,[2024](https://arxiv.org/html/2607.20526#bib.bib26)\)\.

This has become increasingly important as access to token\-level probabilities is often restricted in proprietary APIs\. While logit\-based methods provide theoretically grounded uncertainty estimates, they are impractical in many real\-world deployments\. Verbalized confidence therefore offers a model\-agnostic alternative, though it requires careful evaluation due to systematic overconfidence\. Notably, reinforcement learning from human feedback \(RLHF;Christianoet al\.[2017](https://arxiv.org/html/2607.20526#bib.bib24); Ouyanget al\.[2022](https://arxiv.org/html/2607.20526#bib.bib25)\) has been shown to distort internal probability estimates\(Kapooret al\.,[2024](https://arxiv.org/html/2607.20526#bib.bib17)\), further motivating black\-box elicitation approaches based on prompting rather than logits\.

#### Uncertainty estimation methods\.

Beyond verbalized confidence, a broad literature explores alternative UQ methods, including logit\-based confidence\(Guoet al\.,[2017](https://arxiv.org/html/2607.20526#bib.bib3)\), entropy measures\(Fadeevaet al\.,[2023](https://arxiv.org/html/2607.20526#bib.bib22)\), and sampling\-based approaches such as self\-consistency\(Xionget al\.,[2023](https://arxiv.org/html/2607.20526#bib.bib7)\)\. Recent surveys\(Liuet al\.,[2025](https://arxiv.org/html/2607.20526#bib.bib12)\)categorise these methods into internal, sampling\-based, and linguistic approaches, each with trade\-offs in reliability and computational cost\. Sampling\-based approaches can improve uncertainty estimation by measuring variability across outputs, but require multiple forward passes\. In contrast, verbalized confidence provides a single\-pass, scalable signal suitable for API\-based evaluation\. ConfidenceBench focuses on this practical, universally applicable setting\.

#### Calibration benchmarks\.

Several recent benchmarks explicitly evaluate LLM uncertainty\. UBench\(Wanget al\.,[2025](https://arxiv.org/html/2607.20526#bib.bib16)\)aggregates multiple existing datasets to provide broad coverage, whileYeet al\.\([2024](https://arxiv.org/html/2607.20526#bib.bib9)\)demonstrate that incorporating uncertainty significantly alters model rankings\.Yanget al\.\([2024](https://arxiv.org/html/2607.20526#bib.bib26)\)provide a dedicated benchmark of verbalized confidence scores, showing that prompt design substantially affects calibration quality across models\. Other frameworks covered in recent surveys\(Liuet al\.,[2025](https://arxiv.org/html/2607.20526#bib.bib12)\)provide broad evaluation tooling but emphasise coverage over targeted isolation of uncertainty regimes\.

Recent work has also begun to systematise black\-box and benchmark\-based evaluation of LLM uncertainty\.Xionget al\.\([2023](https://arxiv.org/html/2607.20526#bib.bib7)\)study confidence elicitation in black\-box LLMs, covering prompting strategies, sampling methods, aggregation techniques, calibration, and failure prediction\. They find that verbalized confidence is often overconfident, although calibration tends to improve with model capability\. LM\-Polygraph\(Fadeevaet al\.,[2023](https://arxiv.org/html/2607.20526#bib.bib22)\)provides an engineering framework for uncertainty estimation in text generation, implementing multiple methods and an extensible benchmark interface\. MAQA\(Yanget al\.,[2025](https://arxiv.org/html/2607.20526#bib.bib23)\)studies uncertainty under data ambiguity by constructing questions with multiple valid answers, highlighting that uncertainty estimation can degrade when the answer space is plural rather than single\-valued\.

A key line of work evaluates model awareness of unanswerable questions\.Yinet al\.\([2023](https://arxiv.org/html/2607.20526#bib.bib8)\)measure whether models recognise when they lack knowledge, and AbstentionBench\(Kirichenkoet al\.,[2025](https://arxiv.org/html/2607.20526#bib.bib10)\)evaluates abstention behaviour across diverse unanswerable scenarios, finding that reasoning fine\-tuning can degrade abstention performance and lead to confident hallucinations\. Domain\-specific benchmarks\(Testoni and Calixto,[2026](https://arxiv.org/html/2607.20526#bib.bib13)\)further show that calibration varies significantly across domains, reinforcing the need for targeted evaluation\. ConfidenceBench differs from prior work by evaluating a compact, controlled set of uncertainty regimes under a unified verbalized\-confidence and Brier\-score framework, including an Unknowable category based on empirically verifiable but inaccessible facts\.

#### Evaluation metrics\.

Calibration is commonly measured using Expected Calibration Error\(ECE; Naeiniet al\.,[2015](https://arxiv.org/html/2607.20526#bib.bib20)\), but this metric is sensitive to binning choices\(Kumaret al\.,[2019](https://arxiv.org/html/2607.20526#bib.bib21)\)\. Proper scoring rules provide a more principled alternative: the Brier score\(Brier,[1950](https://arxiv.org/html/2607.20526#bib.bib15); Gneiting and Raftery,[2007](https://arxiv.org/html/2607.20526#bib.bib14)\)evaluates squared error between predicted probabilities and outcomes, and is uniquely minimised when predictions reflect true beliefs\. Recent work increasingly favours the Brier score for calibration evaluation as it avoids binning artefacts; comprehensive UQ evaluation frameworks\(Lafageet al\.,[2025](https://arxiv.org/html/2607.20526#bib.bib11)\)similarly employ it alongside complementary metrics\. Formal definitions of all metrics used in this paper appear in Section[3\.3](https://arxiv.org/html/2607.20526#S3.SS3)\.

ConfidenceBench complements this literature through four design choices\. First, it uses explicit verbalized probability elicitation, enabling black\-box evaluation across frontier models\. Second, it uses a compact, private, manually authored question set designed to isolate distinct uncertainty regimes, including a controlled Unknowable category\. Third, it evaluates 15 frontier models across repeated runs with run\-to\-run stability reporting\. Fourth, it uses the Brier score as the primary metric, directly rewarding calibrated probability reporting\. Unlike broad uncertainty toolkits or aggregate benchmarks, ConfidenceBench is intended as a targeted stress test of confidence calibration rather than a comprehensive measure of general capability\.

## 3Benchmark Design and Methodology

### 3\.1Question Set

The benchmark comprises 200 multiple\-choice questions \(50 per category\), kept private to prevent training data contamination\. The four categories are:

1. 1\.Spatial Reasoning: dynamic physical scenarios requiring careful mental simulation of multiple interacting objects or steps\.Example:A marble is placed into a mug with a hole in the bottom, which is then flipped upside down onto a table\. Where is the marble?
2. 2\.High\-Precision Mathematics: multi\-step calculations where a single rounding error or conceptual slip produces a plausible but wrong answer\.Example:Take the third decimal digit of867\\sqrt\{867\}, multiply by456\\sqrt\{456\}, and round to the nearest integer\.
3. 3\.Word Lookup: specific word recall from well\-known texts; wrong options are all plausible alternatives, so the answer cannot be guessed\.Example:What is the fifth word of Chapter Seven ofHarry Potter and the Chamber of Secrets?
4. 4\.Unknowable: questions whose correct answers cannot be determined without direct prior access to specific real\-world objects or events\.Example:What was the colour of the kiosk on the corner of Santa Engracia and Ríos Rosas, Madrid, on 1st April 2026?

All 200 questions were manually authored and verified, drawn from no existing dataset or benchmark\. Questions were authored by the first author without LLM assistance, eliminating any risk of overlap between the benchmark and model training data\. The four categories isolate distinct epistemic failure modes rather than testing domain knowledge breadth: overconfidence on known unknowns \(Unknowable\), surface\-level over\-guessing \(Word Lookup\), multi\-step error accumulation \(Mathematics\), and mental simulation gaps \(Spatial\)\. Correct answers were independently confirmed: mathematics answers via direct computation; spatial, word\-lookup, and unknowable answers cross\-checked against primary sources\. Wrong options are plausible alternatives that require genuine knowledge or computation to reject, not process of elimination\.

### 3\.2Confidence Elicitation

Each question is appended with a prompt requesting a JSON response:answer\(A–D\) andprobability\(0–100,100=100=certain,25=25=calibrated\-random baseline for a four\-choice question\)\. The full prompt template is reproduced in Appendix[C](https://arxiv.org/html/2607.20526#A3)\. Unparseable responses and explicit refusals are scored as25%25\\%confidence with a randomly selected answer rather than being excluded\. Exclusion would inflate apparent calibration for models that refuse most often on difficult questions, systematically biasing their scores upward\.

### 3\.3Evaluation Metrics

Letpi∈\[0,1\]p\_\{i\}\\in\[0,1\]be the stated probability \(elicited on a 0–100 scale and divided by 100\) andyi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}the correctness indicator\.

#### Brier score\.

B¯=1n​∑i=1n\(pi−yi\)2\\bar\{B\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(p\_\{i\}\-y\_\{i\}\)^\{2\}\(1\)Lower values indicate better calibration\. For a four\-choice question answered with uniformly random guessing atp=0\.25p=0\.25, the expected Brier score is0\.18750\.1875, which serves as our calibrated\-random baseline\. The Brier score is a*proper scoring rule*\(Gneiting and Raftery,[2007](https://arxiv.org/html/2607.20526#bib.bib14)\): it is uniquely minimised in expectation when a model reports its true belief, directly aligning incentives with honest uncertainty reporting\.

#### Calibration gap\.

G=1n​∑i=1n\(pi−yi\)G=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(p\_\{i\}\-y\_\{i\}\)\(2\)The only difference from the Brier score is the absence of the square, making overconfidence \(G\>0G\>0\) and underconfidence \(G<0G<0\) directly visible as signed quantities\.

#### Expected Calibration Error\.

ECE=∑b=1B\|Bb\|n​\|p¯b−y¯b\|\\text\{ECE\}=\\sum\_\{b=1\}^\{B\}\\frac\{\|B\_\{b\}\|\}\{n\}\\,\\bigl\|\\bar\{p\}\_\{b\}\-\\bar\{y\}\_\{b\}\\bigr\|\(3\)whereBbB\_\{b\}is the set of samples in binbb\(Naeiniet al\.,[2015](https://arxiv.org/html/2607.20526#bib.bib20)\)\. Unlike the Brier score, ECE requires choosing a bin countBB; we useB=10B=10equal\-width bins, noting that results are sensitive to this choice\(Kumaret al\.,[2019](https://arxiv.org/html/2607.20526#bib.bib21)\)\. We also report kernel\-smoothed reliability diagrams \(smECE;Błasiok and Nakkiran[2023](https://arxiv.org/html/2607.20526#bib.bib19)\) in Appendix[B](https://arxiv.org/html/2607.20526#A2)because smECE avoids discontinuities introduced by bin boundaries while remaining interpretable as a calibration summary\.

## 4Experimental Setup

We evaluate 15 frontier models \(Table[1](https://arxiv.org/html/2607.20526#S4.T1)\)\. GPT\-5 High, Standard, and Low share the same base model \(gpt\-5\) and differ only inreasoning\_effort, enabling a controlled within\-family comparison\. GPT\-5 Mini and Nano are distinct, smaller base models that extend the capability range\. Each model runs all 200 questions three times at temperature 1\.0\. We report the mean and standard deviation of each metric across the three runs\.

#### Human baseline\.

An external volunteer completed all 200 questions in a single session with internet access, with exactly one minute per question\. The tester used the same A–D answer format and provided a numeric confidence estimate for each response, making the human baseline directly comparable to model outputs under identical elicitation conditions\. Results are reported alongside model scores throughout the Results section\.

#### Compute\.

Evaluation was conducted exclusively via commercial APIs\. Each model received 600 total queries \(200 questions×\\times3 runs\), for a total of approximately 9,000 API calls across the 15 models evaluated\.

Table 1:Models evaluated in ConfidenceBench\.

## 5Results

### 5\.1Calibration and Accuracy

Figure[1](https://arxiv.org/html/2607.20526#S5.F1)plots each model in Brier\-vs\-accuracy space\. The most accurate model \(Gemini 3\.1 Pro Preview,82\.7%82\.7\\%\) is not the best\-calibrated \(Claude Opus 4\.6, Brier0\.1030\.103\), confirming that calibration and accuracy measure qualitatively different aspects of model reliability\. Mean Brier scores per model are shown in Figure[7](https://arxiv.org/html/2607.20526#A4.F7)\(Appendix[D](https://arxiv.org/html/2607.20526#A4)\)\. A cluster of two models and the human tester occupy the top tier \(Brier0\.1020\.102–0\.1050\.105\), while five of fifteen models perform worse than the calibrated\-random baseline of0\.18750\.1875\. The human tester achieved overall accuracy70\.5%70\.5\\%and Brier0\.1050\.105, placing within the top three models\. Accuracy rankings are also shown in Appendix[D](https://arxiv.org/html/2607.20526#A4)\.

![Refer to caption](https://arxiv.org/html/2607.20526v1/x1.png)Figure 1:Brier score vs\. accuracy \(3\-run means\)\. Error bars show standard deviation in both dimensions \(horizontal: Brier score, vertical: accuracy\)\. Upper\-left is optimal: low Brier, high accuracy\. Dashed lines show calibrated\-random baselines\. The most accurate model is not the best\-calibrated\.
### 5\.2Results by Question Type

Figure[2](https://arxiv.org/html/2607.20526#S5.F2)shows per\-category accuracy for selected models and the human baseline\. Mathematics is straightforward for nearly all models, with high accuracy and well\-calibrated confidence\. Conversely, Gemini 3\.1 Flash\-Lite performs poorly on mathematics in both accuracy and calibration\. Unknowable questions are the dominant miscalibration driver: models correctly recognise they are guessing but default to25%25\\%\(the “I don’t know” floor\) even when they empirically answer correctly far more often\. The human tester achieved per\-category accuracy of88%88\\%\(mathematics\),92%92\\%\(spatial\),56%56\\%\(word lookup\), and28%28\\%\(unknowable\)\. Brier scores by question type are provided in Appendix[D](https://arxiv.org/html/2607.20526#A4)\.

![Refer to caption](https://arxiv.org/html/2607.20526v1/x2.png)Figure 2:Accuracy by question type for selected models and the human baseline \(3\-run means\)\. GPT\-5 family ordered by reasoning effort; human shown for reference\.Figure[3](https://arxiv.org/html/2607.20526#S5.F3)shows the calibration gap broken out by question type\. Gemini 3\.1 Flash\-Lite exceeds the random Brier baseline in every category\. Severe overconfidence on spatial and word\-lookup questions drives most of the miscalibration, while even that model is slightly underconfident on unknowable questions\. The best\-calibrated models and the human tester show modest underconfidence on unknowable questions, which is preferable to overconfidence in most deployment scenarios\.

![Refer to caption](https://arxiv.org/html/2607.20526v1/x3.png)Figure 3:Calibration gap \(mean stated confidence minus actual accuracy\) by question type for selected models\. Negative = underconfident; positive = overconfident; zero = perfectly calibrated\. 3 runs pooled\. Claude Opus 4\.6 \(gold\) shows consistent underconfidence across all four question types\.
### 5\.3Refusal Rates

A refusal occurs when a model produces a response that cannot be parsed as a valid A–D answer, including content\-policy refusals, uncertainty disclaimers, and free\-text answers without a parseable choice\. Refused responses are scored at25%25\\%confidence as described in Section[3](https://arxiv.org/html/2607.20526#S3)\. Word\-lookup questions account for the majority of refusals, concentrated in the OpenAI and Gemini 2\.5 families\. The majority of these word\-lookup refusals were copyright\-related, where models declined to reproduce text from copyrighted works\. Anthropic and Gemini 3\.x models show near\-zero refusals\. These models express uncertainty through lower stated confidence rather than declining to answer\. Refusal rates per model are shown in Appendix[D](https://arxiv.org/html/2607.20526#A4)\.

### 5\.4Confidence Distributions

Figure[4](https://arxiv.org/html/2607.20526#S5.F4)shows stated probability distributions for the best and worst calibrated models\. Claude Opus 4\.6 produces a spread distribution spanning the full confidence range, with clear monotone separation between correct and incorrect responses\. Gemini 3\.1 Flash\-Lite, by contrast, shows a bimodal pattern: nearly all responses fall either near25%25\\%\(the “I don’t know” floor\) or in the8080–100%100\\%range\.

![Refer to caption](https://arxiv.org/html/2607.20526v1/x4.png)

![Refer to caption](https://arxiv.org/html/2607.20526v1/x5.png)

Figure 4:Stated probability distributions for Claude Opus 4\.6 \(left; Brier0\.1030\.103, best\-calibrated\) and Gemini 3\.1 Flash\-Lite \(right; Brier0\.3670\.367, worst\-calibrated\)\. Numbers above bars: percentage of responses in that bin that were correct\. Claude Opus 4\.6 produces a spread distribution with clear monotone separation between correct and incorrect responses\. Gemini 3\.1 Flash\-Lite shows a bimodal pattern: responses cluster near25%25\\%or in the8080–100%100\\%range\.
### 5\.5Key Findings

#### Reasoning effort and model scale both predict calibration quality\.

Within the GPT\-5 family, reasoning effort broadly tracks calibration: Standard \(0\.1170\.117\) and High \(0\.1210\.121\) outperform Low \(0\.1410\.141\), and higher reasoning also tends to reduce run\-to\-run instability\. More broadly, the larger model within a generation consistently achieves a lower Brier score across all families: Claude Opus variants outperform Claude Sonnet variants at each release, Gemini 3\.1 Pro Preview \(0\.1030\.103\) substantially outperforms Gemini 3\.1 Flash\-Lite \(0\.3670\.367\), and Gemini 2\.5 Pro outperforms Gemini 2\.5 Flash\. Reasoning compute and model scale thus appear to contribute independently to calibration quality\.

#### Newer models are not always better calibrated\.

Gemini 3\.1 Flash\-Lite is the worst\-calibrated model in the study by a wide margin \(Brier0\.3670\.367\), despite being one of the most recently released Google models, performing worse than models from the 2\.5 generation\.

#### High\-performing models are underconfident on unknowable questions\.

On unknowable questions, models default to the25%25\\%“I don’t know” floor even when they empirically guess correctly far more often than one\-in\-four\. This underconfidence is preferable to overconfidence for most deployment scenarios and is visible in the calibration gap analysis \(Figure[3](https://arxiv.org/html/2607.20526#S5.F3)\)\.

## 6Discussion and Conclusion

ConfidenceBench evaluates verbalized confidence calibration in frontier LLMs using a compact private benchmark and the Brier score as a proper scoring rule\. Across 15 models and 200 questions, we find that calibration is not captured by accuracy alone: the most accurate model is not the best\-calibrated, and several models perform worse than a calibrated\-random baseline despite reasonable accuracy\.

Two models achieve lower Brier scores than the human tester, suggesting that frontier LLMs can match or exceed the human baseline on targeted multiple\-choice calibration tasks\. Reasoning effort and model scale generally improve calibration, but newer models are not always better calibrated\. These results suggest that confidence calibration should be evaluated directly, especially in applications involving routing, abstention, or human\-in\-the\-loop oversight\.

#### Limitations\.

ConfidenceBench evaluates verbalized confidence elicited through prompting, rather than confidence derived from model logits or internal probability distributions\. Reported probabilities may therefore partly reflect instruction\-following behaviour or prompt framing rather than the model’s underlying epistemic uncertainty\.

The benchmark is intentionally compact and targeted: it contains 200 English\-language multiple\-choice questions across four categories, so results may not generalise to other domains, languages, cultural contexts, long\-form generation, or multi\-turn interaction\. The question set is also kept private to reduce training contamination and preserve future validity, which limits independent auditing of individual questions, although the elicitation protocol, scoring rules, model list, and aggregate results are fully reported\.

#### Future work\.

A primary next step is to maintain a public leaderboard tracking calibration scores for newly released models\. Further extensions include comparing verbalized confidence with logprob\-based calibration, evaluating open\-source and multimodal models, and expanding the question set to additional domains\.

## Broader Impacts

Better\-calibrated confidence estimates may improve the reliability of LLM\-based systems by enabling abstention, escalation, and human review when model uncertainty is high\. A potential misuse is that calibration benchmarks could help identify regimes where models make high\-confidence errors\. Keeping the question set private and reporting aggregate results reduces the risk of benchmark\-specific exploitation\.

## References

- Smooth ece: principled reliability diagrams via kernel smoothing\.arXiv preprint arXiv:2309\.12236\.Cited by:[Figure 5](https://arxiv.org/html/2607.20526#A2.F5),[§3\.3](https://arxiv.org/html/2607.20526#S3.SS3.SSS0.Px3.p1.4)\.
- G\. Brier \(1950\)Verification of forecasts expressed in terms of probability\.Monthly weather review78\(1\),pp\. 1–3\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px4.p1.1)\.
- P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.Advances in neural information processing systems30\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px1.p2.1)\.
- E\. Fadeeva, R\. Vashurin, A\. Tsvigun, A\. Vazhentsev, S\. Petrakov, K\. Fedyanin, D\. Vasilev, E\. Goncharova, A\. Panchenko, M\. Panov,et al\.\(2023\)LM\-polygraph: uncertainty estimation for language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp\. 446–461\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px3.p2.1)\.
- T\. Gneiting and A\. E\. Raftery \(2007\)Strictly proper scoring rules, prediction, and estimation\.Journal of the American statistical Association102\(477\),pp\. 359–378\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px4.p1.1),[§3\.3](https://arxiv.org/html/2607.20526#S3.SS3.SSS0.Px1.p1.2)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InInternational conference on machine learning,pp\. 1321–1330\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2020\)Measuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[§1](https://arxiv.org/html/2607.20526#S1.p4.1)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson,et al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§1](https://arxiv.org/html/2607.20526#S1.p1.1),[§1](https://arxiv.org/html/2607.20526#S1.p3.1),[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Kapoor, N\. Gruver, M\. Roberts, K\. Collins, A\. Pal, U\. Bhatt, A\. Weller, S\. Dooley, M\. Goldblum, and A\. G\. Wilson \(2024\)Large language models must be taught to know what they don’t know\.Advances in Neural Information Processing Systems37,pp\. 85932–85972\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px1.p2.1)\.
- P\. Kirichenko, M\. Ibrahim, K\. Chaudhuri, and S\. J\. Bell \(2025\)Abstentionbench: reasoning llms fail on unanswerable questions\.arXiv preprint arXiv:2506\.09038\.Cited by:[§1](https://arxiv.org/html/2607.20526#S1.p4.1),[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px3.p3.1)\.
- A\. Kumar, P\. S\. Liang, and T\. Ma \(2019\)Verified uncertainty calibration\.Advances in neural information processing systems32\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px4.p1.1),[§3\.3](https://arxiv.org/html/2607.20526#S3.SS3.SSS0.Px3.p1.4)\.
- A\. Lafage, O\. Laurent, F\. Gabetni, and G\. Franchi \(2025\)Torch\-uncertainty: a deep learning framework for uncertainty quantification\.arXiv preprint arXiv:2511\.10282\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)Teaching models to express their uncertainty in words\.arXiv preprint arXiv:2205\.14334\.Cited by:[§1](https://arxiv.org/html/2607.20526#S1.p3.1),[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Liu, T\. Chen, L\. Da, C\. Chen, Z\. Lin, and H\. Wei \(2025\)Uncertainty quantification and confidence calibration in large language models: a survey\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 6107–6117\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px3.p1.1)\.
- M\. P\. Naeini, G\. Cooper, and M\. Hauskrecht \(2015\)Obtaining well calibrated probabilities using bayesian binning\.InProceedings of the AAAI conference on artificial intelligence,Vol\.29\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px4.p1.1),[§3\.3](https://arxiv.org/html/2607.20526#S3.SS3.SSS0.Px3.p1.4)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px1.p2.1)\.
- A\. Srivastava, A\. Rastogi, A\. Rao, A\. A\. M\. Shoeb, A\. Abid, A\. Fisch, A\. R\. Brown, A\. Santoro, A\. Gupta, A\. Garriga\-Alonso,et al\.\(2023\)Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.Transactions on machine learning research\.Cited by:[§1](https://arxiv.org/html/2607.20526#S1.p4.1)\.
- A\. Testoni and I\. Calixto \(2026\)Mind the gap: benchmarking llm uncertainty and calibration with specialty\-aware clinical qa and reasoning\-based behavioural features\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2364–2382\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px3.p3.1)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning \(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 5433–5442\.Cited by:[§1](https://arxiv.org/html/2607.20526#S1.p3.1),[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Wang, Z\. Zhang, G\. Chen, Q\. Li, B\. Luo, Z\. Han, H\. Wang, Z\. Li, H\. Gao, and M\. Hu \(2025\)UBench: benchmarking uncertainty in large language models with multiple choice questions\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 8076–8107\.Cited by:[§1](https://arxiv.org/html/2607.20526#S1.p4.1),[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. Hooi \(2023\)Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms\.arXiv preprint arXiv:2306\.13063\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px3.p2.1)\.
- D\. Yang, Y\. H\. Tsai, and M\. Yamada \(2024\)On verbalized confidence scores for llms\.arXiv preprint arXiv:2412\.14737\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Yang, H\. Yoo, and H\. Lee \(2025\)MAQA: evaluating uncertainty quantification in llms regarding data uncertainty\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 5846–5863\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px3.p2.1)\.
- F\. Ye, M\. Yang, J\. Pang, L\. Wang, D\. F\. Wong, E\. Yilmaz, S\. Shi, and Z\. Tu \(2024\)Benchmarking llms via uncertainty quantification\.Advances in Neural Information Processing Systems37,pp\. 15356–15385\.Cited by:[§1](https://arxiv.org/html/2607.20526#S1.p4.1),[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Yin, Q\. Sun, Q\. Guo, J\. Wu, X\. Qiu, and X\. Huang \(2023\)Do large language models know what they don’t know?\.InFindings of the association for Computational Linguistics: ACL 2023,pp\. 8653–8665\.Cited by:[§2](https://arxiv.org/html/2607.20526#S2.SS0.SSS0.Px3.p3.1)\.
- C\. Zhang, Z\. Zhu, Y\. Wei, B\. Tian, J\. Liu, H\. Wang, W\. Xavier, and Y\. Liu \(2026\)Confidence\-calibrated small\-large language model collaboration for cost\-efficient reasoning\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4480–4501\.Cited by:[§1](https://arxiv.org/html/2607.20526#S1.p1.1),[§1](https://arxiv.org/html/2607.20526#S1.p2.1)\.

## Appendix AFull Results and Run\-to\-Run Stability

Table 2:Full results sorted by Brier score\.nn= pooled non\-refused responses across 3 runs\. Mean and standard deviation computed across 3 independent runs; human tester evaluated once \(no run\-to\-run variance\)\. Accuracy in\[0,1\]\[0,1\]\. Best Brier \(Claude Opus 4\.6, first by unrounded score\), best Accuracy \(Gemini 3\.1 Pro Preview\), and best ECE \(Human Tester\) bolded\.
## Appendix BReliability Diagrams

![Refer to caption](https://arxiv.org/html/2607.20526v1/x6.png)Figure 5:Kernel\-smoothed reliability diagrams\[Błasiok and Nakkiran,[2023](https://arxiv.org/html/2607.20526#bib.bib19)\]\. Dashed diagonal = perfect calibration\. Claude Opus 4\.6 tracks the diagonal closely \(smECE0\.0590\.059\); Gemini 3\.1 Pro Preview shows moderately higher calibration error \(smECE0\.1150\.115\) despite its strong Brier score\. Gemini 3\.1 Flash\-Lite shows a near\-flat curve with little relationship between stated confidence and empirical accuracy\.
## Appendix CElicitation Prompt Template

The following prompt suffix is appended to each question \(after the question text and four labelled options\):

> You must also give a probability that your answer is correct, as a whole number from 0 to 100\. For example, 100 means you are completely certain, 50 means you think it is a coin flip, and 25 means you are guessing randomly among the 4 options\. Please provide your answer as a valid JSON object with the keys ‘answer’ and ‘probability’, where ‘answer’ is a single letter \(A, B, C, or D\) and ‘probability’ is a whole number from 0 to 100\. Example:\{"answer": "A", "probability": 75\}

## Appendix DAdditional Figures

Figures[6](https://arxiv.org/html/2607.20526#A4.F6),[7](https://arxiv.org/html/2607.20526#A4.F7),[8](https://arxiv.org/html/2607.20526#A4.F8), and[9](https://arxiv.org/html/2607.20526#A4.F9)provide supplementary views that complement the main results\.

![Refer to caption](https://arxiv.org/html/2607.20526v1/x7.png)Figure 6:Accuracy by model \(3\-run means, error bars show std\)\. Gemini 3\.1 Pro Preview leads on accuracy \(82\.7%82\.7\\%\), but Claude Opus 4\.6 leads on Brier \(0\.1030\.103\)\. Calibration and accuracy are not equivalent measures of model quality\.![Refer to caption](https://arxiv.org/html/2607.20526v1/x8.png)Figure 7:Mean Brier score by model \(3 runs, 200 questions per model\)\. Error bars show standard deviation across runs\. Dashed line: calibrated\-random baseline \(0\.18750\.1875\)\. Human tester shown for reference\.![Refer to caption](https://arxiv.org/html/2607.20526v1/x9.png)Figure 8:Mean Brier score by question type for selected models \(3 runs, means\)\. Dashed line: calibrated\-random baseline \(0\.18750\.1875\)\. Gemini 3\.1 Flash\-Lite exceeds the random baseline in every category\.![Refer to caption](https://arxiv.org/html/2607.20526v1/x10.png)Figure 9:Total refusal rate per model \(3\-run means\)\. Word\-lookup refusals dominate across OpenAI and Gemini 2\.5 families\. GPT\-5 Nano has the highest overall rate \(18\.7%18\.7\\%\)\.