DirEAG: Dirichlet Evidence Aggregation for Calibrating Verbalized Confidence in Mathematical Reasoning

arXiv cs.AI Papers
confidence-estimation mathematical-reasoning large-language-models calibration uncertainty Dirichlet-method

Summary

DirEAG is a method for calibrating verbalized confidence in large language models during mathematical reasoning by aggregating Dirichlet evidence from multiple prompts, showing improved calibration on benchmarks like GSM8K.

arXiv:2608.20717v1 Announce Type: new Abstract: Reliable confidence estimation is essential for using large language models in mathematical reasoning, but black-box verbalized confidence is difficult to calibrate. When the same problem is queried under multiple confidence-steering prompts, the resulting answer-confidence observations contain useful uncertainty information, yet their scales may shift across steering levels, models, and datasets. Existing black-box uncertainty methods often rely on answer agreement, sample consistency, or entropy, which describe output variation but do not model the numerical meaning of self-reported confidence. Conversely, direct averaging or heuristic aggregation of elicited confidence cannot learn prompt- and task-dependent bias. We propose DirEAG, a Dirichlet Evidence Aggregation method that converts each elicited answer-confidence observation into calibrated soft evidence over generated candidate answers and an additional null state, allowing the model to represent cases where none of the candidates is correct. Experiments on GSM8K, SVAMP, and GSM-Hard with Qwen, Mistral, and Gemma models show that, compared with direct confidence averaging and heuristic confidence-steering aggregation, DirEAG often achieves better calibration while maintaining competitive answer selection. Ablations further reveal that evidence aggregation and final binary calibration address distinct parts of the calibration problem.
Original Article
View Cached Full Text

Cached at: 08/24/26, 04:21 AM 

# DirEAG: Dirichlet Evidence Aggregation for Calibrating Verbalized Confidence in Mathematical Reasoning
Source: [https://arxiv.org/abs/2608.20717](https://arxiv.org/abs/2608.20717)
[View PDF](https://arxiv.org/pdf/2608.20717)

> Abstract:Reliable confidence estimation is essential for using large language models in mathematical reasoning, but black\-box verbalized confidence is difficult to calibrate\. When the same problem is queried under multiple confidence\-steering prompts, the resulting answer\-confidence observations contain useful uncertainty information, yet their scales may shift across steering levels, models, and datasets\. Existing black\-box uncertainty methods often rely on answer agreement, sample consistency, or entropy, which describe output variation but do not model the numerical meaning of self\-reported confidence\. Conversely, direct averaging or heuristic aggregation of elicited confidence cannot learn prompt\- and task\-dependent bias\. We propose DirEAG, a Dirichlet Evidence Aggregation method that converts each elicited answer\-confidence observation into calibrated soft evidence over generated candidate answers and an additional null state, allowing the model to represent cases where none of the candidates is correct\. Experiments on GSM8K, SVAMP, and GSM\-Hard with Qwen, Mistral, and Gemma models show that, compared with direct confidence averaging and heuristic confidence\-steering aggregation, DirEAG often achieves better calibration while maintaining competitive answer selection\. Ablations further reveal that evidence aggregation and final binary calibration address distinct parts of the calibration problem\.

## Submission history

From: Haorui Xu \[[view email](https://arxiv.org/show-email/8b76010f/2608.20717)\] **\[v1\]**Fri, 21 Aug 2026 03:48:25 UTC \(114 KB\)

Similar Articles

CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

arXiv cs.CL

The paper introduces CALIBER, a method for calibrating confidence in reasoning language models by eliciting confidence estimates both before and after reasoning, with supervision targets matched to the information state. It achieves significant reductions in Expected Calibration Error (up to 52.5%) and strong Brier scores and AUROC across multiple benchmarks.

VeriEvol: Scaling Multimodal Mathematical Reasoning via Verifiable Evol-Instruct

Hugging Face Daily Papers

VeriEvol is a novel framework for scaling reinforcement learning in visual mathematical reasoning by ensuring reliable reward labels through a two-axis approach separating prompt difficulty from answer reliability, using evolutionary operators and hypothesis-testing verification. It achieves significant accuracy gains on a five-benchmark visual-math suite.

Ask-E: An Environment for Calibrated Question Generation

arXiv cs.CL

Ask-E is a new benchmark and training environment that evaluates and trains models on generating questions calibrated to specific skill levels, defined by the capabilities of two existing language models. Frontier models score below 50% on calibration, and training on Ask-E improves downstream math benchmarks without new math data or correctness-based rewards.

Variational Inference for Evidential Deep Learning

arXiv cs.LG

A mathematically principled framework, Variational Inference Evidential Deep Learning (VI-EDL), is proposed to address limitations in conventional Evidential Deep Learning by reformulating it through variational inference, deriving an Evidence Lower Bound, establishing a generalization bound, and achieving state-of-the-art performance on visual and medical datasets.

Evidence-RL: Towards Evidence-intensive Visual Reasoning

Hugging Face Daily Papers

This paper introduces Counterfactual Evidence Disentanglement (CED), a training-time method that makes vision-language models rely on concrete image evidence rather than language priors or shortcuts, improving visual reasoning grounding across benchmarks.