Tag
该论文将评估器引导的测试时推理建模为一个计算感知的三动作控制器（trust、fallback、expand），通过学习并校准动作特定的置信度估计来优化推理计算分配，在 Qwen3-8B 上实现约 30% 和 16% 的计算节省。
该论文提出一种在多步 LLM 智能体任务中，利用最终结果（而非单步正确性）来校准智能体自报置信度的强化学习方法。在 ALFWorld、TextWorld 和 WebShop 环境中，步级置信度对最终结果的 AUROC 从接近随机提升至 0.68–0.85，校准误差显著下降，且不损害任务成功率。
MARGIN提出了一种运行时置信度校准方法，无需重训练或预留校准集即可从观测结果中学习模型特定的置信度修正，用于异构基础模型的多智能体协调决策，在代码生成和问答任务上相比未校准的置信度加权提升了答案选择准确率。
DSPy 3.4.0 has been released, featuring native support for Jev and System One models along with a new optimizer, ReAnchor, for calibrating outputs with confidence.
This paper documents an open-source toolkit for testing language model confidence by fine-tuning on a fabricated corpus of arithmetic facts, focusing on hallucination and symbol grounding without reporting specific results.
This paper evaluates intrinsic sequence-likelihood confidence in retrieval-dominated extractive QA, finding that confidence signals are insufficient for model control and that retrieval alone achieves high accuracy under pre-specified criteria.
The article presents R2VC, a modular fact-checking system that combines retrieval, verification, and confidence calibration to improve accuracy and reliability in automated fact checking, achieving a 13.74% accuracy increase on the FEVER benchmark.
An experiment with two AI memory instances revealed that a contradiction can permanently erase both a truth and a falsehood, highlighting a flaw in multi-agent setups where less informed agents display higher confidence.
An LLM judge consistently returned a fixed confidence score of 0.72 in evaluations, but switching to categorical labels improved score distribution, showing that models are better at classification than numerical estimation for assessments.
The author questions how to evaluate AI agent decisions beyond simple pass/fail metrics, proposing cost-weighted evaluations and confidence calibration.
A user shares concerns about AI models presenting thin or ambiguous data with the same confidence as well-supported findings, citing a case where a complaint appearing only twice in 200 comments was ranked as a top concern. The piece questions whether this is a fixable prompting issue or a fundamental limitation requiring manual verification.
This paper investigates the 'knowing-saying gap' in language models, showing that linear probes can detect corrupted context with near-perfect accuracy yet fail to predict final answer errors, with implications for deployment monitoring and intervention strategies.
CalTwin introduces a Fisher-Information-based regularization to improve robustness to covariate shift and confidence misalignment in medical world models, achieving modest improvements on the PhysioNet 2019 Sepsis Challenge dataset.
This paper evaluates how small open-weight vision-language models (Qwen2-VL-2B and SmolVLM) handle realistic image degradations, finding that their verbalized confidence is unreliable while internal token probability provides much better error detection, though both fail under severe low-light conditions.
ConfidenceBench is a new benchmark that evaluates verbalized confidence estimates in large language models using Brier scores, revealing that accuracy and calibration diverge and that even highly accurate models can be severely miscalibrated.
This paper presents the first comprehensive overview of metacognition in LLMs, arguing that behaviors like confidence calibration and self-verification are facets of a unified metacognitive ability, and taxonomizes methods and benchmarks for evaluating and improving these abilities to enhance LLM reliability and transparency.
This paper presents a task-specific two-agent architecture for multimodal question answering in the QANTA 2026 shared challenge, using GPT-4.1-mini for tossup questions with confidence calibration and GPT-4.1 for bonus questions with structured reasoning. The system achieved the highest overall leaderboard score, demonstrating the effectiveness of efficient reasoning strategies.
The author presents a method to make LLMs verbalize calibrated confidence by using a linear probe on mid-layer states and a small trained bridge to confidence logits, requiring only 200 labeled examples and no weight modification. This is linked to Anthropic's global workspace paper explaining the know-say gap.
The paper proposes C3RL, a reinforcement learning algorithm that calibrates LLM confidence while maintaining accuracy, and CAS, a confidence-based adaptive test-time scaling strategy that reduces inference costs by up to 12.33 times.
New paper proposes a meta-agent that monitors AI agents and corrects their mistakes, achieving ~2x more task completion autonomously.