confidence-estimation

This paper empirically studies token-probability-based confidence estimation and calibration for LLMs in mathematical question answering, comparing single-pass and multi-pass estimators and evaluating post-hoc calibration methods.

This paper studies confidence estimation for financial vision-language models in chart and document understanding, evaluating seven estimators across five LVLMs. It finds that calibration, not ranking, is the scarce property, and only trained probes produce thresholdable scores for safe deferral to human reviewers.

Google's Gemma 4 model has been enhanced to recognize when it is incorrect, improving its calibration and reliability.

This paper introduces FALCON-Discover, a post-hoc framework for discovering regions where model predictions are confidently wrong, shifting calibration focus from aggregate metrics to sample-level dangerous failures.

This article benchmarks various blackbox and whitebox methods for LLM confidence estimation, including verbalized confidence, linguistic uncertainty, reasoning-length, P(Answer), P(True), and self-consistency, comparing their effectiveness for tasks like active learning and safety classification.

The article introduces CARE-PPO, a reinforcement learning framework that integrates confidence estimation with PPO fine-tuning for language-based quantitative prediction, enabling models to produce both accurate numerical estimates and reliable confidence signals, demonstrated in healthcare and finance tasks.

Introduces Critic Experience Bank (CEB), a self-evolving critic framework for step-level confidence estimation in LLM agents that leverages a memory bank of past judgments and consequences to improve calibration without training.

This paper audits whether self-consistency and cross-model agreement are reliable indicators of correctness in LLMs, finding that agreement is a weak, regime-dependent proxy and that frontier models exhibit overconfidence.

This paper investigates how confidence-related information evolves during LLM answer generation and introduces future confidence distillation, which trains predictors on pre-solution hidden representations using post-solution correctness probes to achieve reliable and sample-efficient confidence estimation.

SpanUQ introduces a lightweight probe for span-level uncertainty quantification in LLMs, using a DETR-style decoder and mixture of Beta distributions, achieving superior error localization and faster inference than sampling-based methods.

AB-RAG is a training-free, backbone-agnostic framework that adaptively retrieves passages for question answering by estimating answer confidence, improving efficiency and accuracy across multiple backbones and datasets.

ExtractConf is a confidence estimation method for LLM-based document field extraction that uses two structurally different calls (field-guided and document-guided) to derive disagreement signals, achieving 0.928 ROC AUC on DocILE invoices and enabling reliable selective prediction for high-stakes automation.

The paper introduces CALIBER, a method for calibrating confidence in reasoning language models by eliciting confidence estimates both before and after reasoning, with supervision targets matched to the information state. It achieves significant reductions in Expected Calibration Error (up to 52.5%) and strong Brier scores and AUROC across multiple benchmarks.

This paper explores Large Language Models' inability to recognize their knowledge limits on structured clinical data, proposing a cross-model attribution divergence method to detect epistemic blind spots. The approach improves calibration and accuracy without training by combining few-shot examples and SHAP-derived feature evidence.

This paper addresses the challenge of robust checkpoint selection for multimodal LLMs under evaluation uncertainty, proposing a multi-stage framework that integrates curated real-world data, LLM-based judgment, and ranking protocols with confidence estimation.

This paper proposes Retrieval-Augmented Linguistic Calibration (RALC), a post-hoc pipeline for calibrating confidence signals in LLMs by modeling linguistic confidence as a distribution and using retrieval-augmented rewriting. It introduces Faithfulness Divergence metric and shows significant improvements across benchmarks.

Introduces Stepwise Confidence Attribution (SCA), a framework for assigning step-level confidence to reasoning traces from black-box LLMs without internal access, using the Information Bottleneck principle to distinguish legitimate variability from errors. Experiments show SCA reliably identifies low-confidence steps and improves self-correction success rates by up to 13.5% over answer-level feedback.

This paper introduces a margin-based confidence ranking method for LLM-as-a-judge systems, learning a dedicated estimator to ensure monotonicity between confidence and human-disagreement risk, with generalization guarantees and improved ranking accuracy across datasets.

This paper proposes a metacognitive harness that separates monitoring from reasoning in LLMs, using pre-solve feeling-of-knowing and post-solve judgment-of-learning signals to control when to trust, retry, or aggregate answers, improving accuracy on text, code, and multimodal benchmarks without parameter updates.