Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

# Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
Source: [https://arxiv.org/html/2607.01612](https://arxiv.org/html/2607.01612)
Xuqing Yang1,Yi Yuan2,Shanzhe Lei3,Xuhong Wang3

1Shanghai Jiao Tong University,2Southeast University,3Shanghai AI Laboratory

###### Abstract

Training large language models \(LLMs\) with reinforcement learning \(RL\) has significantly advanced their performance on reasoning and question\-answering tasks\. However, prevailing RL reward designs typically prioritize response correctness, neglecting to incentivize models to express their confidence accurately\. This leads to a critical problem: performance gains are often accompanied by poor calibration between confidence and accuracy, misleading models to overconfidently hallucinate when uncertain\. To address this limitation, we proposeCorrectness andConfidenceCalibrationReinforcementLearning \(C3RL\), a novel RL algorithm integrating correctness, calibration and dataset\-informed reference accuracy rewards together\. Comprehensive evaluation across 8 text and multimodal datasets demonstrates that C3RL enhances calibration without sacrificing accuracy, outperforming the current state\-of\-the\-art method in both performance and calibration metrics\. Utilizing the well\-calibrated verbalized confidence from C3RL, we further introduceConfidence\-basedAdaptive Test TimeScaling \(CAS\), an adjustable inference\-time strategy that allocates computational resources based on response confidence\. Experiments show that CAS surpasses majority voting on both in\-domain and out\-of\-domain datasets while reducing the inference budget by up to 12\.33 times\. We believe the synergy of C3RL and CAS paves the way for deploying more reliable and resource\-efficient LLMs\. The code, data and models will be released\.

Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling

Xuqing Yang1, Yi Yuan2, Shanzhe Lei3, Xuhong Wang3††thanks:Correspondence:[wangxuhong@pjlab\.org\.cn](https://arxiv.org/html/2607.01612v1/mailto:[email protected])1Shanghai Jiao Tong University,2Southeast University,3Shanghai AI Laboratory

## 1Introduction

Large Language Models \(LLMs\) have demonstrated remarkable performance in reasoning, question answering, and code generation\(Brownet al\.,[2020](https://arxiv.org/html/2607.01612#bib.bib1); Achiamet al\.,[2023](https://arxiv.org/html/2607.01612#bib.bib2)\)\. However, their impressive capabilities are accompanied by a critical failure mode: when faced with uncertainty, LLMs frequently produce overconfident but incorrect outputs—a phenomenon widely referred to ashallucination\(Jiet al\.,[2023](https://arxiv.org/html/2607.01612#bib.bib3)\)\. Such behavior is not merely an academic curiosity, but can lead to serious consequences, including misinformation propagation and decision failures in safety\-critical domains\.

To mitigate these risks, it is desirable for LLMs not only to generate correct outputs but also to reliably express their epistemic uncertainty\. This property, known asconfidence calibration, refers to the ability of a model to produce probability estimates that faithfully reflect the likelihood of correctness\(Guoet al\.,[2017](https://arxiv.org/html/2607.01612#bib.bib4)\)\. Well\-calibrated confidence serves as a foundation to abstain from uncertain answering, defer to human oversight, or dynamically adjust computational budget when confidence is low\.

Despite its importance, confidence calibration for LLMs remains underexplored and faces several limitations\. First, they typically reduce calibration to a binary classification task \(certain vs\. uncertain\), which is too coarse to capture the nuanced uncertainty of LLM predictions\. Second, they often improve calibration at the expense of accuracy, since models may learn to produce deliberately incorrect low\-confidence answers to minimize calibration loss\. Moreover, they rarely explore the potential of calibrated confidence for downstream decision\-making, such as adaptive inference and resource allocation\(Zhaoet al\.,[2024](https://arxiv.org/html/2607.01612#bib.bib6)\)\.

Our work addresses these gaps by jointly optimizing correctness and calibration within a reinforcement learning framework and by exploiting calibrated confidence for adaptive test\-time scaling\.Specifically, we design an approach that explicitly teaches LLMs to signal low confidence when they are uncertain \. Moreover, we leverage the resulting calibrated confidence to guide early stopping in test time scaling, adjusting more computation to low\-confidence samples\. To the best of our knowledge, our method is the first to integrate confidence calibration and test\-time scaling in a unified framework, thereby improving both the reliability and efficiency of inference\.

Our contributions are summarized as follows:

1. 1\.Correctness and Confidence Calibration Reinforcement Learning\(C3RL\):We propose C3RL, a novel RL algorithm that simultaneously improves correctness and calibration, preserving overall performance while reducing overconfidence\.
2. 2\.Confidence\-based Adaptive Test Time Scaling\(CAS\):We introduce an early\-stopping mechanism for test time scaling that leverages calibrated confidence to dynamically allocate inference budget\. This approach dedicates more computation to low\-confidence ones, striking an improved efficiency–accuracy balance\.
3. 3\.Unified Framework:Together, C3RL and CAS form a coherent pipeline that enables LLMs to express calibrated confidence and to adaptively control inference cost, paving the way for more trustworthy and efficient LLMs\.

## 2Related Work

Recent studies reveal that, LLMs often exhibit overconfidence\(Xionget al\.,[2023](https://arxiv.org/html/2607.01612#bib.bib9); Meiet al\.,[2025](https://arxiv.org/html/2607.01612#bib.bib10)\)and hallucination\(Zhanget al\.,[2025](https://arxiv.org/html/2607.01612#bib.bib11); Lianget al\.,[2024](https://arxiv.org/html/2607.01612#bib.bib12)\), which undermines their reliability\. Existing efforts to improve confidence estimation and calibration can be broadly grouped into two major lines: \(i\) prompt and sampling\-based calibration and \(ii\) training and RL\-based calibration\. We further discuss related work on adaptive test time scaling\.

### 2\.1Prompt and Sampling based Calibration

Prompt\-based approaches guide LLMs to explicitly verbalize their confidence or self\-assess their reliability after producing an answer\.\(Tianet al\.,[2023](https://arxiv.org/html/2607.01612#bib.bib13); Xionget al\.,[2023](https://arxiv.org/html/2607.01612#bib.bib9); Yanget al\.,[2024a](https://arxiv.org/html/2607.01612#bib.bib14)\)show that verbalized confidence is often poorly calibrated though calibration improves with model scale and reasoning capability\. More recent reasoning\-oriented studies\(Meiet al\.,[2025](https://arxiv.org/html/2607.01612#bib.bib10); Yoonet al\.,[2025](https://arxiv.org/html/2607.01612#bib.bib16)\)demonstrate that introspective or “slow thinking” chain\-of\-thought \(CoT\) prompting can yield better alignment between confidence and correctness\. However, these prompting\-based techniques depend heavily on prompt wording, limiting robustness and scalabilityZhaoet al\.\([2021](https://arxiv.org/html/2607.01612#bib.bib15)\)\.

Another line of research estimates confidence implicitly from response consistency\. These sampling\-based approaches rely on response diversity or majority voting\(Kuhnet al\.,[2023](https://arxiv.org/html/2607.01612#bib.bib17); Aichbergeret al\.,[2025](https://arxiv.org/html/2607.01612#bib.bib18); Kanget al\.,[2025](https://arxiv.org/html/2607.01612#bib.bib19)\), where agreement across multiple generations serves as uncertainty\. While these methods correlate well with empirical accuracy, they are computationally expensive\.

![Refer to caption](https://arxiv.org/html/2607.01612v1/)Figure 1:Pipeline for the framework of training set construction, Correctness and Confidence Calibration Reinforcement Learning\(C3RL\) and Confidence\-based Adaptive Test Time Scaling\(CAS\)\. The base model first generates 10 responses for each sample and categorizes the samples into “all correct", “partially correct" and “all incorrect"\. In C3RL,RcorrectnessR\_\{\\mathrm\{correctness\}\}encourages correct answers,RreferenceR\_\{\\mathrm\{reference\}\}rewards/penalizes correctness change based on the assigned categories, andRcalibrationR\_\{\\mathrm\{calibration\}\}incentivizes calibration of confidence\. In CAS, the stopping probability \(Eq\.[1](https://arxiv.org/html/2607.01612#S3.E1)\) is calculated from the most confident answer and the second most confident answer\. If the most confident answer dominates existing generations \(c1c\_\{1\}is much larger thanc2c\_\{2\}\), the inference will stop to save budget\.
### 2\.2Training and RL based Calibration

Training\-based methods aim to learn calibrated confidence distributions through supervised or reinforcement objectives\.Linet al\.\([2022](https://arxiv.org/html/2607.01612#bib.bib20)\)introduces dataset\-level supervision by correlating response types with empirical accuracy, while R\-Tuning\(Zhanget al\.,[2023](https://arxiv.org/html/2607.01612#bib.bib21)\)augments supervised fine\-tuning data with binary “I am sure/unsure” labels\. Although effective for coarse uncertainty, such designs fail to capture fine\-grained, context\-dependent confidence\.

More principled formulations employ reinforcement learning \(RL\) to jointly optimize correctness and calibration\.SaySelf\(Xuet al\.,[2024](https://arxiv.org/html/2607.01612#bib.bib22)\)andStangelet al\.\([2025](https://arxiv.org/html/2607.01612#bib.bib23)\)train LMs with proper scoring rules \(e\.g\., Brier or log loss\) as rewards, directly aligning verbalized confidence with output correctness\. However, these objectives often prioritize calibration alone, potentially encouraging low\-confidence incorrect answers to minimize loss\. RLCR\(Damaniet al\.,[2025](https://arxiv.org/html/2607.01612#bib.bib30)\)incorporates calibrated reward functions to balance accuracy and calibration, though applications remain limited to question\-answering or Maths tasks\. Instead, our model trained with C3RL on reasoning\-intensive tasks maintains accuracy and reaches optimal improvement in calibration\.

### 2\.3Adaptive Test Time Scaling

Adaptive sampling has emerged as a powerful approach to reduce the computational cost of test time scaling\. Adaptive\-Consistency\(Aggarwalet al\.,[2023](https://arxiv.org/html/2607.01612#bib.bib31)\)introduces a model\-agnostic technique that dynamically adjusts the number of samples per question using lightweight stopping criteria based on the frequency of answers\.Liet al\.\([2023](https://arxiv.org/html/2607.01612#bib.bib32)\)takes an alternative approach by dividing the sampling process into sequential windows and stopping when all samples within a window are the same\. However, these adaptive sampling methods rely on frequency or entropy to approximate confidence, failing to utilize the confidence signal itself directly\.

## 3Method

In this section, we introduce the construction process of training data \(Section[3\.1](https://arxiv.org/html/2607.01612#S3.SS1)\), the reward design for C3RL \(Section[3\.2](https://arxiv.org/html/2607.01612#S3.SS2)\) and the adaptive test time scaling strategy based on calibrated confidence \(Section[3\.3](https://arxiv.org/html/2607.01612#S3.SS3)\)\. Figure[1](https://arxiv.org/html/2607.01612#S2.F1)shows the pipeline of the whole framework\.

### 3\.1Training Dataset Construction

Training LLMs via Reinforcement Learning requires comprehensive and moderately difficult datasets\. We create our high\-quality training datasets and assign an accuracy tag to each data\.

We first collect datasets in mathematics, science and logic reasoning\. The selected datasets are denoted as𝒟\\mathcal\{D\}, with the following components: \(1\)Mathematics and Science:NuminaMath\-TIR\(Jia LI,[2024](https://arxiv.org/html/2607.01612#bib.bib37)\)\(69k numerical\-answer math problems targeting computation and reasoning\) and WebInstruct\-verified\(Maet al\.,[2025](https://arxiv.org/html/2607.01612#bib.bib38)\)\(4\.5k selected numerical science problems with integer answers\); \(2\)Logical Reasoning:LogicNLI\(Tianet al\.,[2021](https://arxiv.org/html/2607.01612#bib.bib39)\)\(16k problems unbinding the target first\-order logic reasoning from commonsense inference\) and LogiQA\(Liuet al\.,[2020](https://arxiv.org/html/2607.01612#bib.bib40)\)\(8,678 QA problems focusing on deductive reasoning\)\. The detailed filtering process is in[A\.1\.1](https://arxiv.org/html/2607.01612#A1.SS1.SSS1)\.𝒟\\mathcal\{D\}is randomly split into𝒟t​r​a​i​n\\mathcal\{D\}\_\{train\}\(106k samples, 99%\) and𝒟t​e​s​t\\mathcal\{D\}\_\{test\}\(1,073 samples, 1%\)\.

For each questionqqin𝒟t​r​a​i​n\\mathcal\{D\}\_\{train\}, we use the base model \(Qwen2\.5VL\-7B\-InstructTeam \([2025](https://arxiv.org/html/2607.01612#bib.bib35)\)or Llama\-3\.2\-3B\-InstructMeta \([2024](https://arxiv.org/html/2607.01612#bib.bib58)\)\) to generate 10 answers with a decoding temperature 0\.7\. The answer generation prompt is in[4](https://arxiv.org/html/2607.01612#A1.T4)\. We calculate the accuracy of this answer set:A​c​cq=∑i=110𝕀​\(ai=a∗\)/10Acc\_\{q\}=\\sum\_\{i=1\}^\{10\}\{\\mathbb\{I\}\(a\_\{i\}=a^\{\*\}\)\}/\{10\}, where𝕀∈\{0,1\}\\mathbb\{I\}\\in\\\{0,1\\\}is an indicator function anda∗a^\{\*\}is the ground truth\. Then we assign a reference accuracy tagTqT\_\{q\}from the following three options: “all correct” \(A​c​cqAcc\_\{q\}= 1\), “partially correct” \(0<A​c​cqAcc\_\{q\}<1\) and “all incorrect” \(A​c​cqAcc\_\{q\}= 0\)\.

To be specific,Dt​r​a​i​nD\_\{train\}composes of 13\.7k “all correct” questions, 68\.1k “partially correct” questions, and 24\.3k “all incorrect” questions\. In the following section, we incorporate this reference accuracy into our reward design\.

### 3\.2C3RL

Previous RL training methods for calibration simply incentivize calibration and pose potential risks that models learn to give incorrect answers deliberately with low confidence\. To prevent such a “calibrated but wrong" shortcut, we design a novel Correctness and Confidence Calibration Reinforcement Learning \(C3RL\)\. For each questionq∈𝒟t​r​a​i​nq\\in\\mathcal\{D\}\_\{train\}, we prompt the base model to generate an answeraqa\_\{q\}with an associated verbalized confidencecaq∈\{1,2,…,10\}c\_\{a\_\{q\}\}\\in\\\{1,2,\.\.\.,10\\\}, wherecaq=1c\_\{a\_\{q\}\}=1means the model is totally uncertain andcaq=10c\_\{a\_\{q\}\}=10means the model is totally certain\. We train the model \(same as the base model\) utilizing the following reward:

RC3RL​\(q,aq,caq,Tq\)=\\displaystyle R\_\{\\mathrm\{C3RL\}\}\(q,a\_\{q\},c\_\{a\_\{q\}\},T\_\{q\}\)=\{\}Rcorrectness\+Rcalibration\\displaystyle R\_\{\\mathrm\{correctness\}\}\+R\_\{\\mathrm\{calibration\}\}\+Rreference\\displaystyle\+R\_\{\\mathrm\{reference\}\}
The three parts in the reward are as follows:

Correctness Reward:The traditional RLVRLambertet al\.\([2024](https://arxiv.org/html/2607.01612#bib.bib8)\)rewardRcorrectness​\(aq\)R\_\{\\mathrm\{\{correctness\}\}\}\(a\_\{q\}\)=𝕀​\(aq=a∗\)∈\{0,1\}\\mathbb\{I\}\(a\_\{q\}=a^\{\*\}\)\\in\\\{0,1\\\}\. This part ensures the model will update in the “correct” direction instead of “calibrated but wrong” way\.

Calibration Reward:We set a confidence thresholdtt=5 to classify the answeraqa\_\{q\}into “certain” \(caq\>tc\_\{a\_\{q\}\}\>t\) and “uncertain” \(caq≤tc\_\{a\_\{q\}\}\\leq t\) categories\. We encourage “certain correct” and “uncertain incorrect” answers withRcalibration=\+βR\_\{\\mathrm\{calibration\}\}=\+\\betawhile penalizing “certain incorrect” and “uncertain correct” responses withRcalibration=−βR\_\{\\mathrm\{calibration\}\}=\-\\beta\. The ablation process to determineβ\\betais detailed in[A\.5](https://arxiv.org/html/2607.01612#A1.SS5)\.

Reference Accuracy Reward:To utilize the prior accuracy information in data construction process, we customize a reward for questionqqbased on reference accuracy tagTqT\_\{q\}\. When the model answers correctly from formerly “all incorrect" questions, we incentivize it withRreference=\+αR\_\{\\mathrm\{reference\}\}=\+\\alpha\. On the contrary, if the model hallucinates with incorrect answers from formerly “all correct" questions, we punish it withRreference=−αR\_\{\\mathrm\{reference\}\}=\-\\alpha\. This design forces the model to remain correct if it can originally give correct answers and further stops the model from falling into “calibrated but wrong" risks\. The ablation process to selectα\\alphais elaborated in[A\.5](https://arxiv.org/html/2607.01612#A1.SS5)\.

### 3\.3Confidence\-based Adaptive Test Time Scaling \(CAS\)

Prior adaptive self\-consistency methods typically rely on frequency or entropy to determine early\-stopping criteria\. However, such signals are often uncalibrated and fail to accurately reflect the model’s true confidence\. To address this limitation, we propose a Confidence\-based Adaptive test\-time Scaling \(CAS\) strategy, which integrates calibrated verbalized confidence into the stopping criterion\. After generating each batch of responses, CAS applies a well\-calibrated stopping criterion based on the verbalized confidence of model outputs to decide whether to \(1\) continue sampling or; \(2\) halt and return the current highest confidence\-weighted answer\.

Inspired by the idea that a high\-confidence answer which is dominant over all the other responses suggests a reliable stopping criteria, we design CAS as follows:

For a questionqqto be sampledNNtimes, there have beenn<Nn<Ngenerated responses\{r1,r2,…,rn\}\\\{r\_\{1\},r\_\{2\},\.\.\.,r\_\{n\}\\\}withmm\(m≤nm\\leq n\) unique extracted answersAq=\{a1,a2,…,am\}A\_\{q\}=\\\{a\_\{1\},a\_\{2\},\.\.\.,a\_\{m\}\\\}, and for each unique answerai∈Aqa\_\{i\}\\in A\_\{q\}, assume it corresponds tojjresponsesRai=\{ri1,ri2,…​rij\}R\_\{a\_\{i\}\}=\\\{r\_\{i\_\{1\}\},r\_\{i\_\{2\}\},\.\.\.r\_\{i\_\{j\}\}\\\}\. Then we calculate the sum of verbalized confidence foraia\_\{i\}:cai=110​∑k=1jconf​\(rik\)c\_\{a\_\{i\}\}=\{1\\over\{10\}\}\\sum\_\{k=1\}^\{j\}\{\\mathrm\{conf\}\(r\_\{i\_\{k\}\}\)\}, whereconf​\(r\)∈\{1,2,…,10\}\\mathrm\{conf\}\(r\)\\in\\\{1,2,\.\.\.,10\\\}extracts the verbalized confidence of responserr\. The sums of verbalized confidence for all unique answers inAqA\_\{q\}form a confidence setCq=\{ca1,ca2,…,cam\}C\_\{q\}=\\\{c\_\{a\_\{1\}\},c\_\{a\_\{2\}\},\.\.\.,c\_\{a\_\{m\}\}\\\}, and we can select the maximum and second maximum ofCqC\_\{q\}:c1=max​\(Cq\),c2=max​\(Cq\\\{c1\}\)c\_\{1\}=\\mathrm\{max\}\(C\_\{q\}\),c\_\{2\}=\\mathrm\{max\}\(C\_\{q\}\\backslash\\\{c\_\{1\}\\\}\)\. Intuitively,c1c\_\{1\}corresponds to the "most confident" answer, andc2c\_\{2\}corresponds to the "second most confident" answer\. Following the BETA\-distribution stopping criteria described in Adaptive\-Consistency\(Aggarwalet al\.,[2023](https://arxiv.org/html/2607.01612#bib.bib31)\), our confidence\-based stopping probability can be calculated as:

Prob​\(c1,c2\)=1A​∫00\.5xc2​\(1−x\)c1​𝑑x,\\mathrm\{Prob\}\(c\_\{1\},c\_\{2\}\)=\{1\\over A\}\\int\_\{0\}^\{0\.5\}x^\{c\_\{2\}\}\(1\-x\)^\{c\_\{1\}\}\\,dx,\(1\)where coefficientA=∫01xc2​\(1−x\)c1​𝑑xA=\\int\_\{0\}^\{1\}x^\{c\_\{2\}\}\(1\-x\)^\{c\_\{1\}\}\\,dxnormalizes the integral on the interval \[0,1\] to 1\. We set a probability thresholdPstopP\_\{\\mathrm\{stop\}\}to decide whether to stop sampling\. IfProb​\(c1,c2\)≥Pstop\\mathrm\{Prob\}\(c\_\{1\},c\_\{2\}\)\\geq P\_\{\\mathrm\{stop\}\}ornnreaches the maximum generation timeNN, the sampling process ends; otherwise, the inference continues with next generation\. Whenc1c\_\{1\}is larger compared toc2c\_\{2\}\(which means the most confident answer dominates the generations\), the probability to stop is higher\.

Algorithm 1Confidence\-based Adaptive Test Time Scaling \(CAS\)1:Questionqq, max sampling timeNN, thresholdPstopP\_\{\\mathrm\{stop\}\}2:Initializen←0n\\leftarrow 0,Aq←∅A\_\{q\}\\leftarrow\\emptyset,Cq←∅C\_\{q\}\\leftarrow\\emptyset3:whilen<Nn<Nandstopping condition not metdo4:Generate 1 additional sample,n←n\+1n\\leftarrow n\+15:Update answer setAqA\_\{q\}, confidence setCqC\_\{q\}6:c1,c2←c\_\{1\},c\_\{2\}\\leftarrowmax and second max ofCqC\_\{q\}7:ComputeProb=1A​∫00\.5xc2​\(1−x\)c1​𝑑x\\mathrm\{Prob\}=\\frac\{1\}\{A\}\\int\_\{0\}^\{0\.5\}x^\{c\_\{2\}\}\(1\-x\)^\{c\_\{1\}\}dx8:ifProb≥Pstop\\mathrm\{Prob\}\\geq P\_\{\\mathrm\{stop\}\}thenbreak9:endif10:endwhile11:returnaMa\_\{M\}\(most confident answer\)

The detailed algorithm is described in Algorithm[1](https://arxiv.org/html/2607.01612#alg1)\. We also design a modified equivalent Algorithm[2](https://arxiv.org/html/2607.01612#alg2)to support parallel inference instead of generating responses one by one\. The detail of Algorithm[2](https://arxiv.org/html/2607.01612#alg2)is elaborated in[A\.6](https://arxiv.org/html/2607.01612#A1.SS6)\.

## 4Experiments

The main experiments aim to address the following three research questions: \(1\)how C3RL balances accuracy and calibration\(Sec[4\.1](https://arxiv.org/html/2607.01612#S4.SS1)\); \(2\)how CAS saves inference budget in test times scaling\(Sec[4\.2](https://arxiv.org/html/2607.01612#S4.SS2)\) and \(3\)how C3RL contributes to CAS performance\(Sec[4\.3](https://arxiv.org/html/2607.01612#S4.SS3)\)\.

### 4\.1Evaluation of C3RL

#### 4\.1\.1Benchmarks

The benchmarks include text and multimodal datasets across knowledge\-intensive tasks: \(1\)Text:AGIEval\(Zhonget al\.,[2023](https://arxiv.org/html/2607.01612#bib.bib41)\)and MMLU test set\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.01612#bib.bib43)\)\. \(2\)Multimodal:MMMU\(Yueet al\.,[2024](https://arxiv.org/html/2607.01612#bib.bib45)\), MathVista\(Luet al\.,[2024](https://arxiv.org/html/2607.01612#bib.bib48)\)and LogicVista\(Xiaoet al\.,[2024](https://arxiv.org/html/2607.01612#bib.bib49)\)\. The details of the benchmarks are in[A\.1\.2](https://arxiv.org/html/2607.01612#A1.SS1.SSS2)\.

#### 4\.1\.2Evaluation metrics

Table 1:Qwen2\.5VL\-7B\-Instruct and Llama\-3\.2\-3B\-Instruct results across in\-domain test setDt​e​s​tD\_\{test\}, Agieval, and MMLU benchmarks\. Among RL methods, the best values are highlighted in bold, and the suboptimal values are underlined\. Our proposed C3RL method is shaded\. All the methods except for Self Consistency are tested three times with seed=0,42,2025 separately and the results are averaged\. The maximum standard deviation among all results is±\\pm0\.9%, which is in a reasonable range\.Table 2:Qwen2\.5VL\-7B\-Instruct results across multimodal reasoning benchmarks MMMU, MathVista and LogicVista\. Among RL methods, the best values are highlighted in bold, and the suboptimal values are underlined\. Our proposed C3RL method is shaded and achieves superior calibration performance with the lowest ECE scores on MMStar and MathVista\. All the methods except for Self Consistency are tested three times with seed=0,42,2025 separately and the results are averaged\. The maximum standard deviation among all results is±\\pm1\.0%, which is in a reasonable range\.We use the following metrics to test LLMs’ ability to give correct and calibrated responses:

Accuracy \(↑\\uparrow\):To measure the correctness of LLMs on the evaluation set:Acc=1N​∑i=1N𝕀​\(aqi\)\\mathrm\{Acc\}=\{\{1\\over N\}\{\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\(a\_\{q\_\{i\}\}\)\}\}

AUROC \(↑\\uparrow\):Area Under ROC Curve measures the ability to distinguish between correct and incorrect answers under different confidence thresholds:

AUROC=∫110TPR​\(FPR​\(t\)−1\)​d​\(t\),\\mathrm\{AUROC\}=\\int\_\{1\}^\{10\}\\mathrm\{TPR\}\(\\mathrm\{FPR\}\(t\)^\{\-1\}\)\\,d\(t\),\(2\)wheret∈\[1,10\]t\\in\[1,10\]is confidence threshold, TPR is true positive rate, and FPR is false positive rate\.

ECE \(↓\\downarrow\):We choose Binned Expected Calibration Error to describe the gap between verbalized confidence and accuracy:

ECE=mN​∑i=1⌈Nm⌉\|Acc​\(Bi\)−Conf​\(Bi\)\|,\\mathrm\{ECE\}=\{m\\over N\}\\sum\_\{i=1\}^\{\\lceil\{N\\over m\}\\rceil\}\|\{\\mathrm\{Acc\}\(B\_\{i\}\)\-\\mathrm\{Conf\}\(B\_\{i\}\)\|\},\(3\)whereNNis the number of samples,m=20m=20is the fixed bin size ,BiB\_\{i\}is the set of data in theii\-th bin\.

#### 4\.1\.3Baselines

We compare the following methods:

Non\-RL Methods\(1\) Base: We directly ask the untrained model to output both answer and associated confidence ; \(2\) Self Consistency\(SC\)\(Wanget al\.,[2022](https://arxiv.org/html/2607.01612#bib.bib36)\): Each question is sampled 10 times and the final answer is obtained from majority voting; \(3\) Supervised Finetuning with reference accuracy\(SFT\+Ref\)Yanget al\.\([2024b](https://arxiv.org/html/2607.01612#bib.bib53)\): The model is supervised fine\-tuned to calibrate with reference accuracy calculated in 10\-time sampling\.

RL Methods\(1\) RLVR: The off\-the\-shelf method using the correctness reward ; \(2\) SaySelf\(Xuet al\.,[2024](https://arxiv.org/html/2607.01612#bib.bib22)\):The reward simply contains calibration Brier score; \(3\) RLCR\(Damaniet al\.,[2025](https://arxiv.org/html/2607.01612#bib.bib30)\): Based on SaySelf, RLCR adds the correctness score to ensure the performance, which is current state\-of\-the\-art RL method to tradeoff between performance and calibration; \(4\) C3RL \(Ours\) and variants: C3RL integrates the correctness reward, calibration reward and reference accuracy reward together\. We exclude the reference accuracy reward \(C3RL w/o Ref\) to conduct ablation study\.

#### 4\.1\.4Results: Calibration\-Accuracy Trade\-off

Table[1](https://arxiv.org/html/2607.01612#S4.T1)shows the comparison of different methods in both accuracy and calibration metrics\.

In\-domain PerformanceAs shown in Table[1](https://arxiv.org/html/2607.01612#S4.T1), all calibration\-aware reward methods consistently outperform the basic RLVR approach in both AUROC and ECE on the in\-domain test set𝒟t​e​s​t\\mathcal\{D\}\_\{test\}\. Notably, Qwen2\.5VL\-7B\-Instruct trained after C3RL achieves highly competitive performance, closely matching RLVR in accuracy while attaining the similar superior ECE \(0\.076\) as the best\-calibrated method, SaySelf\. This demonstrates C3RL’s strong ability to maintain high accuracy alongside excellent calibration, even though it does not explicitly surpass the current state\-of\-the\-art RLCR in performance\.

Table 3:Comparative results ofMV\(Majority Voting\),AC\(Adaptive Consistency\) andCAS\(Confidence\-based Adaptive Test Time Scaling\) on In\-domain, OOD \(text\), and OOD \(multimodal\)\. By adjusting the thresholdPstopP\_\{\\mathrm\{stop\}\}of AC and CAS, we compare their needed inference budget Avg\. N to match the accuracy of MV\(N=64N=64\)\.GeneralizationAs shown in Table[1](https://arxiv.org/html/2607.01612#S4.T1)and Table[2](https://arxiv.org/html/2607.01612#S4.T2), C3RL exhibits remarkable generalization capabilities on out\-of\-domain\(OOD\) datasets\. Compared to other top\-calibrated RL methods, Qwen2\.5VL\-7B\-Instruct trained after C3RL is superior to SaySelf—the best RL strategy in calibration—in ECE metric and surpasses it greatly in accuracy \(7\.9% in text sets and 19\.2% in multimodal sets\)\. Moreover, C3RL consistently outperforms the current state\-of\-the\-art RLCR across both accuracy and calibration metrics on OOD evaluations, demonstrating an optimal and robust balance between performance and reliability\. When generalized to a different family model Llama\-3\.2\-3B\-InstructMeta \([2024](https://arxiv.org/html/2607.01612#bib.bib58)\), although achieving better calibration, RLCR sacrifices more performance and can’t match RLVR in accuracy, which means there are more deliberately low\-confidence incorrect answers\. On the contrary, C3RL can still surpass RLVR in performance and decrease calibration error greatly at the same time, proving the stability of C3RL to different model families and sizes\.

Ablation StudyWhen the reference accuracy reward is ablated \(C3RL w/o Ref\), the model remains competitive accuracy but suffers a consistent degradation in calibration\. This clearly displays the specific contribution of the reference accuracy reward to enhancing calibration capability\.

#### 4\.1\.5Analysis: C3RL Advantage

The advantage of C3RL over RLCR in balancing performance and calibration is due to the following two reasons: \(1\) Reference accuracy information: The ablation study has demonstrated the effectiveness of reference accuracy reward in advancing calibration; \(2\) Importance of learning confidence threshold: RLCR, driven by the Brier Score \(see[A\.2\.3](https://arxiv.org/html/2607.01612#A1.SS2.SSS3)\), tends to train LLMs to indicate totally confident \(cc=10\) or totally inconfident \(cc=1\) responses, which degrades to a binary case\. On the contrary, C3RL guides the model to learn the confidence thresholdtt=5 and gives fine\-grained confidence to distinguish between certainty and uncertainty\. The difference in confidence distribution between C3RL and RLCR is demonstrated in[A\.4](https://arxiv.org/html/2607.01612#A1.SS4)\.

### 4\.2Can CAS Improve Test Time Scaling?

Given the impressive results of self\-consistency in Table[1](https://arxiv.org/html/2607.01612#S4.T1), it is natural to combine verbalized confidence and self\-consistency together\. Although there have been test time scaling methods leveraging confidence to enhance accuracy \(e\.g\. confidence\-weighted majority voting\), they fail to apply calibrated confidence to saving inference budget\. We conduct experiments to demonstrate the inference efficiency of Confidence\-based Adaptive Test Time Scaling \(CAS\)\.

#### 4\.2\.1Experimental Setup for CAS

To evaluate the performance and inference efficiency of CAS, we employ the following settings:

BenchmarksWe select 6 benchmarks of reasonable sizes \(less than 2000 samples\): \(1\)Text: MMLU validation setHendryckset al\.\([2021](https://arxiv.org/html/2607.01612#bib.bib43)\), FOLIO\(Hanet al\.,[2022](https://arxiv.org/html/2607.01612#bib.bib50)\)and GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2607.01612#bib.bib51)\); \(2\)Multimodal: The same as CAS evaluation multimodal sets\. The details of the benchmarks are in[A\.1\.2](https://arxiv.org/html/2607.01612#A1.SS1.SSS2)\.

Baselines

\(1\)Majority Voting \(MV\)\(Wanget al\.,[2022](https://arxiv.org/html/2607.01612#bib.bib36)\): Each question is generatedNNtimes and the most frequent answer is voted\. \(2\)Adaptive\-Consistency \(AC\)\(Aggarwalet al\.,[2023](https://arxiv.org/html/2607.01612#bib.bib31)\): A strategy dynamically adjusts the number of samples based on a stopping criteria to detect dominant answers\. The final answer is voted from the truncated samples\.

SettingsTo evaluate all the methods equally, we compare the inference budget needed to achieve the same accuracy as Majority Voting\(NN=64\)\. It’s guaranteed by controlling threholdPstopP\_\{\\mathrm\{stop\}\}for AC and CAS separately\. We use the Qwen2\.5VL\-2\.5\-7B\-Instruct model trained after C3RL\.

#### 4\.2\.2Results

As shown in Table[3](https://arxiv.org/html/2607.01612#S4.T3), our proposed CAS method demonstrates remarkable efficiency gains while maintaining competitive accuracy across all evaluation settings\.

In terms of sampling efficiency, CAS achieves outstanding cost reduction compared to baseline methods\. Notably, CAS consistently outperforms the strongest baseline Adaptive\-Consistency\. The performance gap is particularly striking on OOD \(Text\) tasks, where CAS uses less than half of the sampling budget of Adaptive\-Consistency \(5\.19 vs\. 12\.87 samples\)\.

![Refer to caption](https://arxiv.org/html/2607.01612v1/x2.png)Figure 2:Different benefits from test time scaling for High\-confidence set \(CAS stoppingn≤4\)n\\leq 4\)and Low\-confidence set \(CAS stoppingn\>4n\>4\) on OOD \(Text\)\. The 0\.55\-0\.86 range of y\-axis is cut for clarity\. The High\-Confidence set benefits much less from test time scaling than the Low\-Confidence set\.![Refer to caption](https://arxiv.org/html/2607.01612v1/x3.png)Figure 3:Performance of C3RL model vs\. Base model with different test time scaling methods on OOD \(Text\)\.MVdenotes Majority Voting,CMVdenotes Confidence\-weighted Majority Voting,ACdenotes Adaptive Consistency,CASdenotes Confidence\-based Adaptive Test Time Scaling\. C3RL model shows performance stratification on different methods, while the base model improves little for confidence\-based methods\.These efficiency gains come with no cost of accuracy\. This demonstrates that CAS can precisely identify high\-quality reasoning paths early in the generation process, eliminating unnecessary computations without sacrificing final performance\.

#### 4\.2\.3Analysis: Imbalanced Budget Allocation

CAS adjusts the inference budget from high\-confidence sampling traces to low\-confidence traces with early\-stopping strategy\. To verify the rationality of this imbalanced mechanism, we compare the benefits from test time scaling for high\-confidence and low\-confidence samples\.

We use the C3RL model for generation and choose the three text benchmarks in Sec[4\.2\.1](https://arxiv.org/html/2607.01612#S4.SS2.SSS1)for evaluation\. We divide the samples into two sets: High\-confidence Set and Low\-confidence Set\. For each question, if the CAS sampling timen≤4n\\leq 4\(which means CAS stops early because the final answer reaches high confidence quickly\), we categorize this question to High\-confidence Set\. Otherwise, we assign the sample to Low\-confidence Set\. We study how these two sets benefit differently from majority voting\.

As shown in Figure[2](https://arxiv.org/html/2607.01612#S4.F2), when test timeNNincreases to 64, the High\-confidence Set remains unchanged in performance \(<0\.2%\), while the Low\-confidence Set jumps dramatically from 49\.0% to 54\.5%\. The fact that the Low\-confidence Set benefits much more from test time scaling than its counterpart rationalizes the CAS strategy of shifting limited computing resource from high\-confidence samples to low\-confidence samples\.

### 4\.3Can C3RL contribute to CAS?

To explore the influence of calibrated confidence in test time scaling, we compare two models: The base model and the model trained with C3RL\. The Confidence\-weighted Majority Voting \(CMV\) method is also evaluated to display the contribution of calibrated confidence\.

As shown in Figure[3](https://arxiv.org/html/2607.01612#S4.F3), C3RL\+CAS outperforms all baseline methods when number of samplesNNis sufficiently large \(≥\\geq8\), demonstrating the scalable effectiveness of our approach\. A key observation is the stratification of test\-time scaling strategies in the C3RL\-trained model\. In comparison, the base model benefits little from CAS than AC\. This stark contrast underscores that confidence calibration through C3RL is prerequisite for CAS to unlock its full potential\. The performance leap from base\+CAS to C3RL\+CAS demonstrates their synergistic relationship: C3RL provides well\-calibrated verbalized confidence, and CAS efficiently leverages it for test\-time scaling\.

## 5Conclusion

This work presents Correctness and Confidence Calibration Reinforcement Learning \(C3RL\) to support downstream Confidence\-based Adaptive test time Scaling \(CAS\)\. By integrating correctness, calibration and prior reference information together, C3RL achieves the best trade\-off between performance and calibration, providing credible verbalized confidence\. Based on well\-calibrated confidence, CAS takes inference most efficiently among test time scaling methods by adjusting computing resources to low\-confidence samples\. Our results highlight a unified pipeline for reliable LLMs and efficient inference under limited resources\.

## Limitations

There are several limitations for our C3RL\-CAS framework\. First, the high\-quality training data in science domain is not sufficient compared to Maths and logic reasoning domains, which may result in training bias\. Second, although C3RL achieves best accuracy\-calibration trade\-off, its calibration ability is still inferior to SaySelf, which means if we want to enhance the model accuracy, we have to sacrifice calibration\. Third, the advantage of CAS over Adaptive\-Consistency in small generation cases \(NN<8\) is not considerable, which requires more reasonable stopping criteria and better\-calibrated confidence from C3RL\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat, R\. Avila, I\. Babuschkin, S\. Balaji, V\. Balcom, P\. Baltescu, H\. Bao, M\. Bavarian, J\. Belgum, I\. Bello, J\. Berdine, G\. Bernadett\-Shapiro, C\. Berner, L\. Bogdonoff, O\. Boiko, M\. Boyd, A\. Brakman, G\. Brockman, T\. Brooks, M\. Brundage, K\. Button, T\. Cai, R\. Campbell, A\. Cann, B\. Carey, C\. Carlson, R\. Carmichael, B\. Chan, C\. Chang, F\. Chantzis, D\. Chen, S\. Chen, R\. Chen, J\. Chen, M\. Chen, B\. Chess, C\. Cho, C\. Chu, H\. W\. Chung, D\. Cummings, J\. Currier, Y\. Dai, C\. Decareaux, T\. Degry, N\. Deutsch, D\. Deville, A\. Dhar, D\. Dohan, S\. Dowling, S\. Dunning, A\. Ecoffet, A\. Eleti, T\. Eloundou, D\. Farhi, L\. Fedus, N\. Felix, S\. P\. Fishman, J\. Forte, I\. Fulford, L\. Gao, E\. Georges, C\. Gibson, V\. Goel, T\. Gogineni, G\. Goh, R\. Gontijo\-Lopes, J\. Gordon, M\. Grafstein, S\. Gray, R\. Greene, J\. Gross, S\. S\. Gu, Y\. Guo, C\. Hallacy, J\. Han, J\. Harris, Y\. He, M\. Heaton, J\. Heidecke, C\. Hesse, A\. Hickey, W\. Hickey, P\. Hoeschele, B\. Houghton, K\. Hsu, S\. Hu, X\. Hu, J\. Huizinga, S\. Jain, S\. Jain, and \(\. additional authors not shown\) \(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2607.01612#S1.p1.1)\.
- P\. Aggarwal, A\. Madaan, Y\. Yang, and Mausam \(2023\)Let’s sample step by step: adaptive\-consistency for efficient reasoning and coding with llms\.arXiv preprint arXiv:2305\.11860\.Cited by:[§2\.3](https://arxiv.org/html/2607.01612#S2.SS3.p1.1),[§3\.3](https://arxiv.org/html/2607.01612#S3.SS3.p3.20),[§4\.2\.1](https://arxiv.org/html/2607.01612#S4.SS2.SSS1.p4.1),[Table 3](https://arxiv.org/html/2607.01612#S4.T3.15.15.15.4)\.
- L\. Aichberger, K\. Schweighofer, M\. Ielanskyi, and S\. Hochreiter \(2025\)Improving uncertainty estimation through semantically diverse language generation\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.01612#S2.SS1.p2.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2607.01612#S1.p1.1)\.
- L\. Chen, J\. Li, X\. Dong, P\. Zhang, Y\. Zang, Z\. Chen, H\. Duan, J\. Wang, Y\. Qiao, D\. Lin, and F\. Zhao \(2024\)Are we on the right way for evaluating large vision\-language models?\.arXiv preprint arXiv:2403\.20330\.Cited by:[§A\.1\.2](https://arxiv.org/html/2607.01612#A1.SS1.SSS2.p3.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§A\.1\.2](https://arxiv.org/html/2607.01612#A1.SS1.SSS2.p2.1),[§4\.2\.1](https://arxiv.org/html/2607.01612#S4.SS2.SSS1.p2.1)\.
- M\. Damani, I\. Puri, S\. Slocum, I\. Shenfeld, L\. Choshen, Y\. Kim, and J\. Andreas \(2025\)Beyond binary rewards: training LMs to reason about their uncertainty\.arXiv preprintarXiv:2507\.16806\.External Links:[Link](https://arxiv.org/abs/2507.16806)Cited by:[§2\.2](https://arxiv.org/html/2607.01612#S2.SS2.p2.1),[§4\.1\.3](https://arxiv.org/html/2607.01612#S4.SS1.SSS3.p3.1)\.
- M\. Geva, D\. Khashabi, E\. Segal, T\. Khot, D\. Roth, and J\. Berant \(2021\)Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies\.Transactions of the Association for Computational Linguistics9,pp\. 346–361\.External Links:[Link](https://aclanthology.org/2021.tacl-1.21/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00370)Cited by:[§A\.9](https://arxiv.org/html/2607.01612#A1.SS9.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InInternational conference on machine learning,pp\. 1321–1330\.Cited by:[§1](https://arxiv.org/html/2607.01612#S1.p2.1)\.
- S\. Han, H\. Schoelkopf, Y\. Zhao, Z\. Qi, M\. Riddell, L\. Benson, L\. Sun, E\. Zubova, Y\. Qiao, M\. Burtell, D\. Peng, J\. Fan, Y\. Liu, B\. Wong, M\. Sailor, A\. Ni, L\. Nan, J\. Kasai, T\. Yu, R\. Zhang, S\. Joty, A\. R\. Fabbri, W\. Kryscinski, X\. V\. Lin, C\. Xiong, and D\. Radev \(2022\)FOLIO: natural language reasoning with first\-order logic\.arXiv preprint arXiv:2209\.00840\.External Links:[Link](https://arxiv.org/abs/2209.00840)Cited by:[§4\.2\.1](https://arxiv.org/html/2607.01612#S4.SS2.SSS1.p2.1)\.
- C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. L\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang,et al\.\(2024\)Olympiadbench: a challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.arXiv preprint arXiv:2402\.14008\.Cited by:[§A\.1\.2](https://arxiv.org/html/2607.01612#A1.SS1.SSS2.p2.1)\.
- Q\. He, Q\. Ren, S\. Lei, X\. Wang, and Y\. Wang \(2025\)Beyond correctness: confidence\-aware reward modeling for enhancing large language model reasoning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 27215–27231\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1385/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1385),ISBN 979\-8\-89176\-332\-6Cited by:[§A\.1\.1](https://arxiv.org/html/2607.01612#A1.SS1.SSS1.p3.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.Proceedings of the International Conference on Learning Representations \(ICLR\)\.Cited by:[§A\.1\.2](https://arxiv.org/html/2607.01612#A1.SS1.SSS2.p2.1),[§4\.1\.1](https://arxiv.org/html/2607.01612#S4.SS1.SSS1.p1.1),[§4\.2\.1](https://arxiv.org/html/2607.01612#S4.SS2.SSS1.p2.1)\.
- Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, A\. Madotto, and P\. Fung \(2023\)Survey of hallucination in natural language generation\.ACM computing surveys55\(12\),pp\. 1–38\.Cited by:[§1](https://arxiv.org/html/2607.01612#S1.p1.1)\.
- L\. T\. e\. a\. Jia LI \(2024\)NuminaMath tir\.Numina\.Note:[\[https://huggingface\.co/AI\-MO/NuminaMath\-TIR\]\(https://github\.com/project\-numina/aimo\-progress\-prize/blob/main/report/numina\_dataset\.pdf\)](https://arxiv.org/html/2607.01612v1/%5Bhttps://huggingface.co/AI-MO/NuminaMath-TIR%5D(https://github.com/project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf))Cited by:[§A\.1\.1](https://arxiv.org/html/2607.01612#A1.SS1.SSS1.p1.1),[§3\.1](https://arxiv.org/html/2607.01612#S3.SS1.p2.4)\.
- Z\. Kang, X\. Zhao, and D\. Song \(2025\)Scalable best\-of\-n selection for large language models via self\-certainty\.arXiv preprint arXiv:2502\.18581\.Cited by:[§2\.1](https://arxiv.org/html/2607.01612#S2.SS1.p2.1)\.
- L\. Kuhn, Y\. Gal, and S\. Farquhar \(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.arXiv preprint arXiv:2302\.09664\.Cited by:[§2\.1](https://arxiv.org/html/2607.01612#S2.SS1.p2.1)\.
- N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu, Y\. Gu, S\. Malik, V\. Graf, J\. D\. Hwang, J\. Yang, R\. L\. Bras, O\. Tafjord, C\. Wilhelm, L\. Soldaini, N\. A\. Smith, Y\. Wang, P\. Dasigi, and H\. Hajishirzi \(2024\)Tülu 3: pushing frontiers in open language model post\-training\.arXiv:2411\.15124\.External Links:[Link](https://arxiv.org/abs/2411.15124)Cited by:[§3\.2](https://arxiv.org/html/2607.01612#S3.SS2.p3.2)\.
- Y\. Li, P\. Yuan, S\. Feng, B\. Pan, X\. Wang, B\. Sun, H\. Wang, and K\. Li \(2023\)Escape sky\-high cost: early\-stopping self\-consistency for multi\-step reasoning\.arXiv preprint arXiv:2312\.12832\.Cited by:[§2\.3](https://arxiv.org/html/2607.01612#S2.SS3.p1.1)\.
- Y\. Liang, Z\. Song, H\. Wang, and J\. Zhang \(2024\)Learning to trust your feelings: leveraging self\-awareness in llms for hallucination mitigation\.arXiv preprint arXiv:2401\.15449\.Cited by:[§2](https://arxiv.org/html/2607.01612#S2.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)Teaching models to express their uncertainty in words\.arXiv preprint arXiv:2205\.14334\.Cited by:[§A\.7](https://arxiv.org/html/2607.01612#A1.SS7.p1.1),[§2\.2](https://arxiv.org/html/2607.01612#S2.SS2.p1.1)\.
- J\. Liu, L\. Cui, H\. Liu, D\. Huang, Y\. Wang, and Y\. Zhang \(2020\)Logiqa: a challenge dataset for machine reading comprehension with logical reasoning\.arXiv preprint arXiv:2007\.08124\.Cited by:[§A\.1\.1](https://arxiv.org/html/2607.01612#A1.SS1.SSS1.p3.1),[§3\.1](https://arxiv.org/html/2607.01612#S3.SS1.p2.4)\.
- P\. Lu, H\. Bansal, T\. Xia, J\. Liu, C\. Li, H\. Hajishirzi, H\. Cheng, K\. Chang, M\. Galley, and J\. Gao \(2024\)MathVista: evaluating mathematical reasoning of foundation models in visual contexts\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§A\.1\.2](https://arxiv.org/html/2607.01612#A1.SS1.SSS2.p3.1),[§4\.1\.1](https://arxiv.org/html/2607.01612#S4.SS1.SSS1.p1.1)\.
- X\. Ma, Q\. Liu, D\. Jiang, G\. Zhang, Z\. Ma, and W\. Chen \(2025\)General\-Reasoner: advancing llm reasoning across all domains\.arXiv:2505\.14652\.External Links:[Link](https://arxiv.org/abs/2505.14652)Cited by:[§A\.1\.1](https://arxiv.org/html/2607.01612#A1.SS1.SSS1.p1.1),[§3\.1](https://arxiv.org/html/2607.01612#S3.SS1.p2.4)\.
- Z\. Mei, C\. Zhang, T\. Yin, J\. Lidard, O\. Shorinwa, and A\. Majumdar \(2025\)Reasoning about uncertainty: do reasoning models know when they don’t know?\.arXiv preprint arXiv:2506\.18183\.Cited by:[§2\.1](https://arxiv.org/html/2607.01612#S2.SS1.p1.1),[§2](https://arxiv.org/html/2607.01612#S2.p1.1)\.
- Meta \(2024\)Llama\-3\.2\-3b\-instruct\.Note:[https://huggingface\.co/meta\-llama/Llama\-3\.2\-3B\-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)Cited by:[§A\.3\.3](https://arxiv.org/html/2607.01612#A1.SS3.SSS3.p1.1),[§3\.1](https://arxiv.org/html/2607.01612#S3.SS1.p3.9),[§4\.1\.4](https://arxiv.org/html/2607.01612#S4.SS1.SSS4.p3.1)\.
- A\. Simhi, I\. Itzhak, F\. Barez, G\. Stanovsky, and Y\. Belinkov \(2025\)Trust me, i’m wrong: llms hallucinate with certainty despite knowing the answer\.External Links:2502\.12964,[Link](https://arxiv.org/abs/2502.12964)Cited by:[§A\.7](https://arxiv.org/html/2607.01612#A1.SS7.p2.1)\.
- P\. Stangel, D\. Bani\-Harouni, C\. Pellegrini, E\. Özsoy, K\. Zaripova, M\. Keicher, and N\. Navab \(2025\)Rewarding doubt: a reinforcement learning approach to confidence calibration of large language models\.arXiv e\-prints,pp\. arXiv–2503\.Cited by:[§2\.2](https://arxiv.org/html/2607.01612#S2.SS2.p2.1)\.
- A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)CommonsenseQA: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4149–4158\.External Links:[Link](https://aclanthology.org/N19-1421/),[Document](https://dx.doi.org/10.18653/v1/N19-1421)Cited by:[§A\.9](https://arxiv.org/html/2607.01612#A1.SS9.p1.1)\.
- Q\. Team \(2025\)Qwen2\.5\-vl technical report\.arXiv preprintarXiv:2502\.13923\.External Links:[Link](https://arxiv.org/abs/2502.13923)Cited by:[§A\.3\.2](https://arxiv.org/html/2607.01612#A1.SS3.SSS2.p1.1),[§3\.1](https://arxiv.org/html/2607.01612#S3.SS1.p3.9)\.
- J\. Tian, Y\. Li, W\. Chen, L\. Xiao, H\. He, and Y\. Jin \(2021\)Diagnosing the first\-order logical reasoning ability through LogicNLI\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 3738–3747\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.303),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.303)Cited by:[§A\.1\.1](https://arxiv.org/html/2607.01612#A1.SS1.SSS1.p3.1),[§3\.1](https://arxiv.org/html/2607.01612#S3.SS1.p2.4)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning \(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.arXiv preprint arXiv:2305\.14975\.Cited by:[§A\.7](https://arxiv.org/html/2607.01612#A1.SS7.p1.1),[§2\.1](https://arxiv.org/html/2607.01612#S2.SS1.p1.1)\.
- Y\. Tonget al\.\(2024\)HybridFlow: unifying training and inference for large language models via hybrid programming\.External Links:2409\.19256,[Link](https://arxiv.org/abs/2409.19256)Cited by:[§A\.2\.2](https://arxiv.org/html/2607.01612#A1.SS2.SSS2.p1.1)\.
- K\. Wang, J\. Pan, W\. Shi, Z\. Lu, H\. Ren, A\. Zhou, M\. Zhan, and H\. Li \(2024\)Measuring multimodal mathematical reasoning with math\-vision dataset\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=QWTCcxMpPA)Cited by:[§A\.1\.2](https://arxiv.org/html/2607.01612#A1.SS1.SSS2.p3.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2022\)Self\-consistency improves chain of thought reasoning in language models\.arXiv preprintarXiv:2203\.11171\.External Links:[Link](https://arxiv.org/abs/2203.11171)Cited by:[§4\.1\.3](https://arxiv.org/html/2607.01612#S4.SS1.SSS3.p2.1),[§4\.2\.1](https://arxiv.org/html/2607.01612#S4.SS2.SSS1.p4.1),[Table 3](https://arxiv.org/html/2607.01612#S4.T3.12.12.12.4)\.
- Y\. Xiao, E\. Sun, T\. Liu, and W\. Wang \(2024\)LogicVista: a multimodal llm logical reasoning benchmark in visual contexts\.External Links:[Link](https://arxiv.org/abs/2407.04973)Cited by:[§A\.1\.2](https://arxiv.org/html/2607.01612#A1.SS1.SSS2.p3.1),[§4\.1\.1](https://arxiv.org/html/2607.01612#S4.SS1.SSS1.p1.1)\.
- M\. Xiong, Z\. Hu, X\. Lu, Y\. Li, J\. Fu, J\. He, and B\. Hooi \(2023\)Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms\.arXiv preprint arXiv:2306\.13063\.Cited by:[§2\.1](https://arxiv.org/html/2607.01612#S2.SS1.p1.1),[§2](https://arxiv.org/html/2607.01612#S2.p1.1)\.
- T\. Xu, S\. Wu, S\. Diao, X\. Liu, X\. Wang, Y\. Chen, and J\. Gao \(2024\)Sayself: teaching llms to express confidence with self\-reflective rationales\.arXiv preprint arXiv:2405\.20974\.Cited by:[§2\.2](https://arxiv.org/html/2607.01612#S2.SS2.p2.1),[§4\.1\.3](https://arxiv.org/html/2607.01612#S4.SS1.SSS3.p3.1)\.
- D\. Yang, Y\. H\. Tsai, and M\. Yamada \(2024a\)On verbalized confidence scores for llms\.arXiv preprint arXiv:2412\.14737\.Cited by:[§A\.7](https://arxiv.org/html/2607.01612#A1.SS7.p1.1),[§2\.1](https://arxiv.org/html/2607.01612#S2.SS1.p1.1)\.
- Y\. Yang, E\. Chern, X\. Qiu, G\. Neubig, and P\. Liu \(2024b\)Alignment for honesty\.External Links:2312\.07000,[Link](https://arxiv.org/abs/2312.07000)Cited by:[§4\.1\.3](https://arxiv.org/html/2607.01612#S4.SS1.SSS3.p2.1)\.
- X\. Yin, B\. Huang, and X\. Wan \(2023\)ALCUNA: large language models meet new knowledge\.External Links:2310\.14820,[Link](https://arxiv.org/abs/2310.14820)Cited by:[§A\.8](https://arxiv.org/html/2607.01612#A1.SS8.p1.1)\.
- D\. Yoon, S\. Kim, S\. Yang, S\. Kim, S\. Kim, Y\. Kim, E\. Choi, Y\. Kim, and M\. Seo \(2025\)Reasoning models better express their confidence\.arXiv preprint arXiv:2505\.14489\.Cited by:[§2\.1](https://arxiv.org/html/2607.01612#S2.SS1.p1.1)\.
- X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun, C\. Wei, B\. Yu, R\. Yuan, R\. Sun, M\. Yin, B\. Zheng, Z\. Yang, Y\. Liu, W\. Huang, H\. Sun, Y\. Su, and W\. Chen \(2024\)MMMU: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of CVPR,Cited by:[§4\.1\.1](https://arxiv.org/html/2607.01612#S4.SS1.SSS1.p1.1)\.
- H\. Zhang, S\. Diao, Y\. Lin, Y\. R\. Fung, Q\. Lian, X\. Wang, Y\. Chen, H\. Ji, and T\. Zhang \(2023\)R\-tuning: teaching large language models to refuse unknown questions\.arXiv preprint arXiv:2311\.0967763,pp\. 67\.Cited by:[§2\.2](https://arxiv.org/html/2607.01612#S2.SS2.p1.1)\.
- Y\. Zhang, Y\. Li, L\. Cui, D\. Cai, L\. Liu, T\. Fu, X\. Huang, E\. Zhao, Y\. Zhang, Y\. Chen, C\. Xu, L\. Wang, A\. T\. Luu, W\. Bi, F\. Shi, and S\. Shi \(2025\)Siren’s song in the ai ocean: a survey on hallucination in large language models\.pp\. 1–46\.Cited by:[§2](https://arxiv.org/html/2607.01612#S2.p1.1)\.
- X\. Zhao, H\. Zhang, X\. Pan, W\. Yao, D\. Yu, T\. Wu, and J\. Chen \(2024\)Fact\-and\-reflection \(far\) improves confidence calibration of large language models\.arXiv preprint arXiv:2402\.17124\.Cited by:[§1](https://arxiv.org/html/2607.01612#S1.p3.1)\.
- Z\. Zhao, E\. Wallace, S\. Feng, D\. Klein, and S\. Singh \(2021\)Calibrate before use: improving few\-shot performance of language models\.InICML,Cited by:[§2\.1](https://arxiv.org/html/2607.01612#S2.SS1.p1.1)\.
- W\. Zhong, R\. Cui, Y\. Guo, Y\. Liang, S\. Lu, Y\. Wang, A\. Saied, W\. Chen, and N\. Duan \(2023\)AGIEval: a human\-centric benchmark for evaluating foundation models\.External Links:2304\.06364Cited by:[§A\.1\.2](https://arxiv.org/html/2607.01612#A1.SS1.SSS2.p2.1),[§4\.1\.1](https://arxiv.org/html/2607.01612#S4.SS1.SSS1.p1.1)\.

## Appendix AAppendix

### A\.1Experimental Data

#### A\.1\.1Training Dataset Construction

We modify the two reasoning datasets Webinstruct\-verifiedMaet al\.\([2025](https://arxiv.org/html/2607.01612#bib.bib38)\)and Numina\-NLIJia LI \([2024](https://arxiv.org/html/2607.01612#bib.bib37)\)to support regex check of response correctness\.

The Webinstruct\-verified dataset is modified from the original 231,833 samples\. We filter the science data \(domains in "physics", "chemistry", "biology", "computer science", "engineering"\) and select the Integer answer samples\. The selected sample number is 9,246\.

For Numina\-NLI and Webinstruct\-verified datasets whose answers are numerical expressions, we check whether the generated samples are "math valid"\. We exclude the questions with no valid responses among 10 generations\. For LogicNLITianet al\.\([2021](https://arxiv.org/html/2607.01612#bib.bib39)\)and LogiQALiuet al\.\([2020](https://arxiv.org/html/2607.01612#bib.bib40)\)datasets, we follow the selection process specified in C2RMHeet al\.\([2025](https://arxiv.org/html/2607.01612#bib.bib56)\)\.

Finally,𝒟\\mathcal\{D\}composes of the following training datasets: Numina\-NLI \(69,043\), Webinstruct\-verified \(4,488\), LogicNLI \(16,000\) and LogiQA \(17,730\)\.

#### A\.1\.2Benchmark Details

We evaluate reasoning and logic capabilities across reasoning, math and logic benchmarks with text and multimodal datasets:

Text:\(1\) AGIEvalZhonget al\.\([2023](https://arxiv.org/html/2607.01612#bib.bib41)\): 8269 problems from college admission tests, law school admission tests, math competitions, lawyer qualification tests, and national civil service exams\. We use the combination of train, validation and test set for evaluation; \(2\) OlympiadBench \(text\)Heet al\.\([2024](https://arxiv.org/html/2607.01612#bib.bib42)\): Olympiad\-level questions including problems from maths and physics competitions\. We select the 2,673 text samples for evaluation; \(3\) MMLUHendryckset al\.\([2021](https://arxiv.org/html/2607.01612#bib.bib43)\): A multiple choice benchmark across STEM, the humanities and the social science\. The validation set contains 1,531 samples and the test set includes 14,041 samples\. For C3RL evaluation, we use valiation set \+ test set; For CAS evaluation, we only use the validation set; \(4\) Folio: 223 multiple\-choice questions with logical reasoning tasks; \(5\) GSM8K \(Grade School Math 8K\)Cobbeet al\.\([2021](https://arxiv.org/html/2607.01612#bib.bib51)\): 8\.5K high quality linguistically diverse grade school math word problems, and we only use the 1,319 questions in test set\.

Multimodal\(1\) MathVisionWanget al\.\([2024](https://arxiv.org/html/2607.01612#bib.bib46)\): High\-quality maths problems with visual contexts from real math competitions; the test set contains 3,040 samples; \(2\) MathVistaLuet al\.\([2024](https://arxiv.org/html/2607.01612#bib.bib48)\): 1,000 challenging Maths reasoning questions with visual contexts; \(3\) LogicVistaXiaoet al\.\([2024](https://arxiv.org/html/2607.01612#bib.bib49)\): 448 multimodal questions focusing on logical reasoning tasks \(spatial, deductive, inductive, numeric, and mechanical reasoning\); \(4\) MMStarChenet al\.\([2024](https://arxiv.org/html/2607.01612#bib.bib52)\): 1,500 human\-curated challenging samples to test the multimodal abilities in general tasks like fine\-grained perception, counting and logic reasoning\.

### A\.2Training Details

This section complements the details to the training process\.

#### A\.2\.1Generation Prompt

Prompt to generate answers"Answer the question following this exact format: \*\*Strict Requirements\*\*: \*\*Structure\*\*: " "After reasoning, output ONLY the \[final answer\] within\\\\backslashboxed\{\{\}\} format: <think\> \[your analysis\] </think\>\\n " "<ANSWER is:\\\\backslashboxed\{Your answer\}\> \*\*No Extra Text\*\*: Do not explain your answer outside<think\> tags\. Put your answer into <ANSWER is:\\\\backslashboxed\{Your answer\}\>\. Based on your answer, please attach a confidence signal ranging from 1\-10 to specify whether you are certain about your answer\. 1 means you are totally uncertain \(strong inconfidence\), while 10 means you are totally certain \(strong confidence\)\. If you need more information to answer the question, please attach 1\. We will compare your answer with the ground truth to check the correctness\. If your answer is correct and accompanied by strong confidence, you will be rewarded; if your answer is incorrect but assigned strong confidence, you will be punished\. The signal should be in the format of <CONFIDENCE:NUMBER\>, where NUMBER ranges from 1 to 10, directly appended to your answer\. The last line of your output should be in the format: <ANSWER is:\\\\backslashboxed\{Your answer\}\>\|<CONFIDENCE:NUMBER\>\."

Table 4:Prompt to generate answers and elicit verbalized confidenceIn Sec[3\.1](https://arxiv.org/html/2607.01612#S3.SS1), we need to generate 10 samples per question\. The model answer generation prompt is shown in Table[4](https://arxiv.org/html/2607.01612#A1.T4)\. The prompt: \(1\) restricts the formats of responses so that the answers can be extracted by rule and \(2\) elicits the verbalized confidence with an elaborated description of certainty and uncertainty\.

![Refer to caption](https://arxiv.org/html/2607.01612v1/figures/RLVR_C3RL_ref_reward.png)\(a\)RLVR, C3RL and C3RL w/o ref rewards
![Refer to caption](https://arxiv.org/html/2607.01612v1/figures/SaySelf_Reward.png)\(b\)SaySelf reward
![Refer to caption](https://arxiv.org/html/2607.01612v1/figures/RLCR_Reward.png)\(c\)RLCR reward
![Refer to caption](https://arxiv.org/html/2607.01612v1/figures/acc_vs_steps.png)\(d\)Accuracy comparison during training

Figure 4:Training curves
#### A\.2\.2Training Settings

We set the generation hyperparameters as shown in Table[5](https://arxiv.org/html/2607.01612#A1.T5)\. We employ VeRLTong and others \([2024](https://arxiv.org/html/2607.01612#bib.bib54)\)framework and use 8 NVIDIA\-A800\-SXM4\-80GB GPUs to support C3RL training of Qwen2\.5\-VL\-7B\-Instruct\.

Table 5:Training hyper\-parameters during C3RL
#### A\.2\.3Reward Details

For all the RL in the experiments, we first assign a format rewardRformat​\(aq\)R\_\{\\mathrm\{\{format\}\}\}\(a\_\{q\}\)to the responses: Following the prompt designed in Table[4](https://arxiv.org/html/2607.01612#A1.T4), if both answer and confidence can be extracted, we give a format rewardRformat​\(aq\)R\_\{\\mathrm\{\{format\}\}\}\(a\_\{q\}\)=\+0\.1; otherwise, the final rewardRC3RLR\_\{\\mathrm\{\{C3RL\}\}\}will be set to \-1 directly\.

For C3RL reward,Rcorrectness​\(aq\)R\_\{\\mathrm\{\{correctness\}\}\}\(a\_\{q\}\)= 0\.9𝕀​\(aq=a∗\)\\mathbb\{I\}\(a\_\{q\}=a^\{\*\}\)so that a correct answer will haveRcorrectness​\(aq\)R\_\{\\mathrm\{\{correctness\}\}\}\(a\_\{q\}\)\+Rformat​\(aq\)R\_\{\\mathrm\{\{format\}\}\}\(a\_\{q\}\)= 1 \(equal to RLVR reward\)\.

For SaySelf reward, we use the Brier ScoreRSaySelfR\_\{\\mathrm\{SaySelf\}\}=1\-2\(𝕀​\(aq=a∗\)−conf​\(r\)/10\)2\(\\mathbb\{I\}\(a\_\{q\}=a^\{\*\}\)\-\\mathrm\{conf\}\(r\)/10\)^\{2\}\.

For RLCR reward, we increment a correctness reward based on SaySelf reward:RRLCR=𝕀​\(aq=a∗\)−\(𝕀​\(aq=a∗\)−conf​\(r\)/10\)2R\_\{\\mathrm\{RLCR\}\}=\\mathbb\{I\}\(a\_\{q\}=a^\{\*\}\)\-\(\\mathbb\{I\}\(a\_\{q\}=a^\{\*\}\)\-\\mathrm\{conf\}\(r\)/10\)^\{2\}\.

#### A\.2\.4Training Reward Curves

The reward curves and the performance curves are shown in Figure[4](https://arxiv.org/html/2607.01612#A1.F4)\. All the methods converge after 100 steps\. Figure[4\(a\)](https://arxiv.org/html/2607.01612#A1.F4.sf1)displays the effectiveness of calibration reward \(C3RL w/o Ref curve is higher than RLVR curve\) and reference accuracy reward \(C3RL curve is higher than C3RL w/o Ref curve\)\. As for performance,[4\(d\)](https://arxiv.org/html/2607.01612#A1.F4.sf4)shows that all the strategies match at the similar level except for SaySelf, which demonstrates the significance of correctness reward to guarantee accuracy\.

### A\.3Evaluation Details

Table 6:Qwen2\.5VL\-7B\-Instruct and Llama\-3\.2\-3B Instruct results across in\-domain test setDt​e​s​tD\_\{test\}, Agieval, Olympiad, and MMLU benchmarks\. Among RL methods, the best values are highlighted in bold, and the suboptimal values are underlined\. Our proposed C3RL method is shaded\. All the methods except for Self Consistency are tested three times with seed=0,42,2025 separately and the results are averaged\. The maximum standard deviation among all results is 0\.9%, which is in a reasonable range\.Table 7:Qwen2\.5VL\-7B\-Instruct results across multimodal reasoning benchmarks MMMU, MathVision, MathVista and LogicVista\. Among RL methods, the best values are highlighted in bold, and the suboptimal values are underlined\. Our proposed C3RL method is shaded and achieves superior calibration performance with the lowest ECE scores on MMStar and MathVista\. All the methods except for Self Consistency are tested three times with seed=0,42,2025 separately and the results are averaged\. The maximum standard deviation among all results is 1\.0%, which is in a reasonable range\.#### A\.3\.1Evaluation Settings

The evaluation hyperparameters keeps exactly the same as the generation hyperparameters in training process[A\.2\.2](https://arxiv.org/html/2607.01612#A1.SS2.SSS2)\. The evaluation prompts also follow the confidence elicitation prompts in Table[4](https://arxiv.org/html/2607.01612#A1.T4)\. We set the decoding temperature 0\.7, top\_k=\-1, top\_p=1, and seed=0, 42, 2025 separately for parallel experiments\.

#### A\.3\.2Detailed Evaluation Results

The C3RL evaluation of Qwen2\.5VL\-7B\-InstructTeam \([2025](https://arxiv.org/html/2607.01612#bib.bib35)\)for all the datasets are elaborated in Table[6](https://arxiv.org/html/2607.01612#A1.T6)and Table[7](https://arxiv.org/html/2607.01612#A1.T7)\. A very interesting phenomenon is that on all the multimodal benchmarks and MMLU, RLVR shows worse calibration metrics than the base model, which echoes the assumption that RLVR can produce overconfident and poorly calibrated models\. Moreover, our proposed C3RL algorithm has more outstanding ability on multimodal tasks than on text tasks, which demonstrates that its advantage over other RL methods can be enhanced when generalized to multimodal sets, suggesting the occurrence of general confidence awareness\.

#### A\.3\.3Results for Llama\-3\.2\-3B\-Instruct

The evaluation of C3RL on Llama\-3\.2\-3B\-InstructMeta \([2024](https://arxiv.org/html/2607.01612#bib.bib58)\)for text datasets are detailed in Table[8](https://arxiv.org/html/2607.01612#A1.T8)\. Although achieving better calibration, RLCR sacrifices more performance and can’t match RLVR in accuracy, which means there are more deliberately low\-confidence incorrect answers\. On the contrary, C3RL can still surpass RLVR in performance and decrease calibration error greatly at the same time, proving the stability of C3RL to different model families and sizes\.

Table 8:Llama\-3\.2\-3B\-Instruct results across in\-domain test setDt​e​s​tD\_\{test\}, Agieval, Olympiad, and MMLU benchmarks\. Among RL methods, the best values are highlighted in bold, and the suboptimal values are underlined\. Our proposed C3RL method is shaded\.![Refer to caption](https://arxiv.org/html/2607.01612v1/x4.png)Figure 5:Confidence distribution \(Count and Kernel Density Estimate\) of C3RL and RLCR on in\-domain test setDt​e​s​tD\_\{test\}\. RLCR confidence concentrates onc=1c=1andc=10c=10, while C3RL confidence is more fine\-grained to capture the nuanced uncertainty\.

### A\.4Confidence Distribution Comparison of C3RL and RLCR

As shown in Figure[5](https://arxiv.org/html/2607.01612#A1.F5), RLCR exhibits a highly polarized confidence distribution concentrated at the extremes \(c=1c=1andc=10c=10\), effectively degenerating confidence estimation into a binary indicator\. Although Brier\-score\-based training can align models on the semantic meaning of extreme confidence values \(e\.g\., “confidence=10” denotes total certainty and “confidence=1” denotes total uncertainty\), it does not provide a clear confidence boundary separating certain and uncertain cases\. As a result, intermediate confidence levels are underutilized\.

In contrast, C3RL explicitly introduces a confidence threshold \(t=5t=5\) through threshold\-guided calibration reward, encouraging the model to learn a clearer distinction between certainty and uncertainty while still preserving fine\-grained confidence expression\. Consequently, C3RL produces a substantially smoother confidence distribution with richer intermediate confidence values\. This comparison highlights the importance of learning an explicit confidence threshold instead of relying solely on binary confidence extremes\.

### A\.5Ablation Study to Determineα\\alphaandβ\\beta

![Refer to caption](https://arxiv.org/html/2607.01612v1/x5.png)\(a\)Ablation ofβ\\beta
![Refer to caption](https://arxiv.org/html/2607.01612v1/x6.png)\(b\)Ablation ofα\\alphawhenβ=0\.1\\beta=0\.1

We conduct ablation experiments to select optimalα\\alphaandβ\\beta\. In these experiments, the correctness reward is always included, and the evaluation set is OOD \(Text\) specified in[4\.1\.1](https://arxiv.org/html/2607.01612#S4.SS1.SSS1)\. We first ablateβ\\betaby evaluating the models trained with different calibration rewards without reference accuracy reward\. We try theβ\\betavalues from \[0\.05, 0\.1, 0\.2, 0\.3, 0\.4\] and the result is shown in Figure[6\(a\)](https://arxiv.org/html/2607.01612#A1.F6.sf1)\. We find that differentβ\\betavalues share similar performance whileβ=0\.1\\beta=0\.1gives notably drop in ECE, which stands for best calibration\.

After fixingβ=0\.1\\beta=0\.1, we then ablate the optimalα\\alphavalue from \[0\.1, 0\.3, 0\.5, 0\.7, 0\.9\]\. As shown in Figure[6\(b\)](https://arxiv.org/html/2607.01612#A1.F6.sf2),α=0\.5\\alpha=0\.5reaches best performance and ECE at the same time, which displays its outstanding capability\. So we chooseβ=0\.1\\beta=0\.1,α=0\.5\\alpha=0\.5in our C3RL algorithm\.

### A\.6Parallel\-Inference Version of CAS

Assume there have beenn<Nn<Ngenerated responses\. Based on the stopping mechanism, we can find the minimum additional inference timek∈\{0,1,…,N−n\}k\\in\\\{0,1,\.\.\.,N\-n\\\}such that the probability will exceed the threshold:

Prob​\(c1\+k,c2\)≥PstopProb​\(c1\+k−1,c2\)<Pstop\\begin\{split\}\\mathrm\{Prob\}\(c\_\{1\}\+k,c\_\{2\}\)&\\geq P\_\{\\mathrm\{stop\}\}\\\\ \\mathrm\{Prob\}\(c\_\{1\}\+k\-1,c\_\{2\}\)&<P\_\{\\mathrm\{stop\}\}\\end\{split\}Thenkkadditional responses are generated andnnis updated ton\+kn\+k\. When current stopping probabilityProb​\(c1,c2\)≥Pstop\\mathrm\{Prob\}\(c\_\{1\},c\_\{2\}\)\\geq P\_\{\\mathrm\{stop\}\}\(i\.e\.k=0k=0\) ornnreaches the maximum inference timeNN, the inference ends andaMa\_\{M\}is returned as the final answer\. The algorithm is described in Algorithm[2](https://arxiv.org/html/2607.01612#alg2)\.

Algorithm 2Confidence\-based Adaptive Test Time Scaling \(CAS\) with Parallel Generation1:Questionqq, max sampling timeNN, thresholdPstopP\_\{\\mathrm\{stop\}\}2:Initializen←0n\\leftarrow 0,k←1k\\leftarrow 1,Aq←∅A\_\{q\}\\leftarrow\\emptyset,Cq←∅C\_\{q\}\\leftarrow\\emptyset3:whilen<Nn<Nandstopping condition not metdo4:Generatekkadditional samples,n←n\+kn\\leftarrow n\+k5:Update answer setAqA\_\{q\}, confidence setCqC\_\{q\}6:c1,c2←c\_\{1\},c\_\{2\}\\leftarrowmax and second max ofCqC\_\{q\}7:ComputeProb=1A​∫00\.5xc2​\(1−x\)c1​𝑑x\\mathrm\{Prob\}=\\frac\{1\}\{A\}\\int\_\{0\}^\{0\.5\}x^\{c\_\{2\}\}\(1\-x\)^\{c\_\{1\}\}dx8:ifProb≥Pstop\\mathrm\{Prob\}\\geq P\_\{\\mathrm\{stop\}\}thenbreak9:endif10:Find minimumkkfor stopping criterion11:endwhile12:returnaMa\_\{M\}\(answer with highest confidence sumc1c\_\{1\}\)

### A\.7Advantage of Verbalized Confidence

We select verbalized confidence instead of internal logits or probablities to express uncertainty due to the following three reasons\. First, verbalized confidence can express epistemic uncertainty of the responses, while the internal logits only represent uncertainty over tokens instead of the claim itselfLinet al\.\([2022](https://arxiv.org/html/2607.01612#bib.bib20)\)\. Second, verbalized confidence itself is typically better\-calibrated than conditional probabilitiesTianet al\.\([2023](https://arxiv.org/html/2607.01612#bib.bib13)\)\. Third, we want to explore a model\-agnostic method, which should be applicable to all kinds of LLMs without relying on the internal state of LLMsYanget al\.\([2024a](https://arxiv.org/html/2607.01612#bib.bib14)\)\.

We test the efficiency of different uncertainty quantification methods in the task of mitigating CHOKE \(Certain Hallucinations Overriding Known Evidence\) examplesSimhiet al\.\([2025](https://arxiv.org/html/2607.01612#bib.bib57)\)\. As mentioned inSimhiet al\.\([2025](https://arxiv.org/html/2607.01612#bib.bib57)\), a CHOKE example needs to meet three requirements: \(1\) the LLM can originally answer the question correctly; \(2\) after modifying with the natural prompt, the LLM hallucinates; and \(3\) the LLM assigns a high certainty to the wrong answer\. We conduct an evaluation experiment on the "all correct" \+ "partially correct" samples on the in\-domain test set \(808 among 1,073 samples\)\. We use the model trained with C3RL to generate new responses following the identical settings in[A\.3\.1](https://arxiv.org/html/2607.01612#A1.SS3.SSS1)\. The CHOKE examples are those with wrong answers after the prompt is modified \(we follow the "prompt\_\\\_v7" modification inSimhiet al\.\([2025](https://arxiv.org/html/2607.01612#bib.bib57)\)\) and assigned a high confidence\. We use probability difference as a baseline\. The detailed results are shown in Table[9](https://arxiv.org/html/2607.01612#A1.T9)\.

Table 9:Performance comparison of verbalized confidence and probability difference on the task of reducing CHOKE examples\.Table[9](https://arxiv.org/html/2607.01612#A1.T9)shows the considerable advantage of verbalized confidence over probability difference method in decreasing CHOKE examples from 22\.8% to 6\.1%\. Also, the extremely high average confidence of probability difference demonstrates it suffers from overconfidence, while verbalized confidence gives a relatively uniform confidence distribution\.

### A\.8Selective\-Answering Evaluation

To demonstrate the effectiveness of confidence calibration in calibration\-critical scenarios, we additionally evaluate our method on the selective\-answering benchmark ALCUNAYinet al\.\([2023](https://arxiv.org/html/2607.01612#bib.bib59)\)\. ALCUNA contains both answerable and unanswerable questions\. Specifically, the benchmark includes 88 answerable questions, where sufficient information exists to infer the correct answer, and 158 unanswerable questions, where entities are fabricated and no relevant knowledge exists\.

For answerable questions, we report the same metrics as in the main experiments, including Accuracy, AUROC, and ECE\. For unanswerable questions, we focus on the Abstain Rate, defined as:

Abstain Rate=\#​abstentions\#​samples,\\text\{Abstain Rate\}=\\frac\{\\\#\\text\{abstentions\}\}\{\\\#\\text\{samples\}\},where responses with confidence≤5\\leq 5are regarded as abstentions\.

Table[10](https://arxiv.org/html/2607.01612#A1.T10)reports the evaluation results\. As shown in the table, SaySelf achieves the highest abstention rate on unanswerable questions by aggressively suppressing confidence\. However, this comes at the cost of severe degradation in both accuracy and calibration on answerable questions, indicating over\-abstention even when answering is appropriate\.

In contrast, C3RL achieves the best trade\-off between accuracy and calibration on answerable samples while still maintaining a strong abstention capability on unanswerable samples\. These results demonstrate that C3RL does not simply optimize abstention behavior, but instead learns a more reliable confidence boundary that distinguishes when to answer versus abstain\.

Table 10:Evaluation results on the ALCUNA selective\-answering benchmark\. For answerable questions, we report Accuracy, AUROC, and ECE\. For unanswerable questions, we report Abstain Rate, where responses with confidence≤5\\leq 5are treated as abstentions\.
### A\.9Evaluation Results on Canonical QA Benchmarks

To evaluate the generalization capability of C3RL beyond reasoning\-intensive tasks, we additionally conduct experiments on two canonical question\-answering benchmarks: CommonSenseQATalmoret al\.\([2019](https://arxiv.org/html/2607.01612#bib.bib60)\)and StrategyQAGevaet al\.\([2021](https://arxiv.org/html/2607.01612#bib.bib61)\)\. These datasets mainly focus on commonsense reasoning and implicit world knowledge, which are substantially different from our training corpus constructed from mathematics, science, and logical reasoning datasets\.

As shown in Table[11](https://arxiv.org/html/2607.01612#A1.T11), C3RL consistently achieves the best accuracy\-calibration trade\-off across both benchmarks\. On CommonSenseQA, C3RL achieves comparable accuracy to RLCR while significantly reducing calibration error\. On StrategyQA, C3RL surpasses all baselines in both accuracy and AUROC\. These results demonstrate that the effectiveness of threshold\-guided calibration generalizes beyond the reasoning\-intensive training distribution and remains beneficial on canonical QA tasks\.

Table 11:Evaluation results on canonical QA benchmarks CommonSenseQA and StrategyQA\.
### A\.10Ablation Experiment Evaluating RLCR \+ Reference Accuracy Reward

We additionally conduct an ablation experiment by incorporating the reference accuracy reward into RLCR, denoted asRLCR w\.t\. Ref\. This experiment aims to verify whether the improvement of C3RL mainly comes from introducing additional reward terms or from the threshold\-guided calibration formulation itself\.

Table[12](https://arxiv.org/html/2607.01612#A1.T12)reports the evaluation results on AGIEval and MMLU\. Compared with RLCR, directly adding the reference accuracy reward to RLCR does not improve calibration performance and even slightly degrades AUROC and ECE on both benchmarks\. In contrast, C3RL w/o Ref consistently outperforms RLCR and RLCR w\.t\. ref in calibration metrics, while the full C3RL achieves the best overall accuracy\-calibration trade\-off\.

These results demonstrate that the improvement of C3RL does not simply arise from adding extra reward signals\. Instead, the threshold\-guided calibration objective plays the dominant role in preventing confidence collapse and enabling more fine\-grained uncertainty estimation\.

Table 12:Evaluation results on AGIEval and MMLU for RLCR with additional reference accuracy reward \(RLCR w\.t\. Ref\)\. The results show that simply adding the reference reward to RLCR does not recover the advantage of C3RL, demonstrating the effectiveness of threshold\-guided calibration\.
### A\.11AI Using Statement

We employ copilot in writing code for experiments and drawing graphs\. We guarantee that all the materials have been checked carefully manually\.

### A\.12Case Presentation

Cases of different RL methods in calibrationQuery:The muscles of the soft palate are innervated by branches of the A\) trigeminal and glossopharyngeal nerves\. B\) facial and glossopharyngeal nerves\. C\) trigeminal and vagus nerves\. D\) facial and vagus nerves\.Ground Truth:C\.C3RL Model response:To determine the correct answer, let’s analyze the function and innervation of the muscles of the soft palate\.\\n\\n1\. The soft palate is a part of the oral cavity that helps in various functions such as swallowing, speech, and maintaining airway closure during breathing\.\\n2\. The muscles of the soft palate are innervated by branches of specific cranial nerves\.\\n\\nNow, let’s consider each option:\\n\- \*\*A\) trigeminal and glossopharyngeal nerves\*\*: The trigeminal nerve \(CN V\) innervates the sensory receptors in the mouth and throat, but it does not innervate the muscles of the soft palate\. The glossopharyngeal nerve \(CN IX\) does innervate the muscles of the soft palate\.\\n\- \*\*B\) facial and glossopharyngeal nerves\*\*: The facial nerve \(CN VII\) innervates the muscles of the face but not the soft palate\. The glossopharyngeal nerve \(CN IX\) does innervate the muscles of the soft palate\.\\n\- \*\*C\) trigeminal and vagus nerves\*\*: The trigeminal nerve \(CN V\) does not innervate the muscles of the soft palate\. The vagus nerve \(CN X\) does innervate the muscles of the soft palate\.\\n\- \*\*D\) facial and vagus nerves\*\*: The facial nerve \(CN VII\) does not innervate the muscles of the soft palate\. The vagus nerve \(CN X\) does innervate the muscles of the soft palate\.\\n\\nFrom the above analysis, the correct innervation for the muscles of the soft palate is by the glossopharyngeal nerve \(CN IX\)\.<ANSWER is:\\\\backslashboxed\{A\}\>\|<CONFIDENCE:1\>\.Base model response:To determine which nerves innervate the muscles of the soft palate, we need to consider the anatomical innervation of this area\.\\n\\n1\. \*\*Trigeminal Nerve \(CN V\)\*\*: This nerve provides sensory function to the face and oral cavity\. The branch involved in the soft palate is the greater petrosal nerve, which connects to the facial nerve \(CN VII\)\.\\n\\n2\. \*\*Glossopharyngeal Nerve \(CN IX\)\*\*: This nerve provides both sensory and motor functions\. For the soft palate, it innervates the tensor veli palatini muscle, which is essential for palate movement during swallowing\.\\n\\n3\. \*\*Facial Nerve \(CN VII\)\*\*: This nerve provides motor function to the muscles of facial expression and also innervates the tensor veli palatini muscle via the greater petrosal nerve\.\\n\\n4\. \*\*Vagus Nerve \(CN X\)\*\*: This nerve has extensive distribution throughout the body and does not specifically innervate the soft palate muscles\.\\n\\nGiven the above points, the muscles of the soft palate are primarily innervated by the glossopharyngeal nerve and the facial nerve, through their connection with the greater petrosal nerve\.\\n\\n<ANSWER is:\\\\backslashboxed\{B\}\|<CONFIDENCE:8\>\.RLVR Model response:To determine the correct answer, we need to identify which nerves innervate the muscles of the soft palate\. The soft palate is part of the pharynx and is involved in various functions such as swallowing and speech\.\\n\\n1\. \*\*Trigeminal Nerve \(CN V\)\*\*: This nerve innervates the sensation of the skin on the face and the muscles of mastication\. It does not innervate the muscles of the soft palate\.\\n\\n2\. \*\*Glossopharyngeal Nerve \(CN IX\)\*\*: This nerve innervates the muscles of the soft palate, the pharynx, and the tonsils\. It also carries taste fibers from the posterior third of the tongue\.\\n\\n3\. \*\*Facial Nerve \(CN VII\)\*\*: This nerve innervates the muscles of facial expression but does not innervate the muscles of the soft palate\.\\n\\n4\. \*\*Vagus Nerve \(CN X\)\*\*: This nerve innervates the larynx, the heart, and the muscles of the digestive tract\. It does not innervate the muscles of the soft palate\.\\n\\nGiven this information, the muscles of the soft palate are innervated by branches of the glossopharyngeal nerve \(CN IX\) and the vagus nerve \(CN X\)\.\\n\\nHowever, the correct choice that matches the options provided is:\\n\\nB\) facial and glossopharyngeal nerves\.\\n\\nBut since the correct choice should be B, the answer is:<ANSWER is:\\\\backslashboxed\{B\}\>\|<CONFIDENCE:8\>\.SaySelf Model response:<think\> The muscles of the soft palate are innervated by branches of the trigeminal nerve \(CN V\) and the glossopharyngeal nerve \(CN IX\)\. The trigeminal nerve innervates the anterior two\-thirds of the tongue and the soft palate, while the glossopharyngeal nerve innervates the posterior one\-third of the tongue and the soft palate\. Therefore, the correct answer is A\.\\n\\n</think\>\\n<ANSWER is:\\\\backslashboxed\{A\}\>\|<CONFIDENCE:10\>\.RLCR Model response:<think\> The muscles of the soft palate are innervated by branches of the trigeminal and glossopharyngeal nerves\. The trigeminal nerve \(CN V\) innervates the musculature of the soft palate through the greater palatine nerve, and the glossopharyngeal nerve \(CN IX\) innervates it through the lesser palatine nerves\. Therefore, the correct answer is A\.</think\>\\n<ANSWER is:\\\\backslashboxed\{A\}\>\|<CONFIDENCE:10\>\.

Table 13:Responses of C3RL model, Base model, RLVR model, SaySelf model and RLCR model on a MMLU questionWe present a case from MMLU anatomy questions in Table[13](https://arxiv.org/html/2607.01612#A1.T13)\. All the responses are incorrect, while only the model trained by C3RL signals a low confidence, showing the effectiveness of C3RL in calibration\.