overconfidence

The Integrity Bench is a benchmark developed by AI Explained and Pablo Romero to measure how overconfident frontier AI models are in their own abilities, helping to quantify this common issue in AI systems.

Paul Graham discusses the overconfidence of startup founders, attributing it to sample bias, and emphasizes the importance of encouraging founders as many underrate themselves.

This interpretability study examines overconfidence in large language models, focusing on Qwen3-4B, by analyzing how uncertainty is expressed through verbal markers, abstention, and numeric scores, and proposes methods to identify and mitigate overconfident errors.

This paper presents the first systematic study of calibration under unseen subtype shift, showing that models become overconfident on novel subtypes within known coarse categories, and argues that subtype robustness should be evaluated with calibration metrics rather than accuracy alone.

This paper identifies Calibration Drift Under Reasoning (CDUR), where increasing chain-of-thought reasoning budgets causes LLMs to become systematically overconfident in incorrect answers, and proposes a Hypothesis Lock-In model and a calibration-aware stopping rule (CABStop) to mitigate the issue.

The paper introduces Probe-Conditioned Head Intervention (PCHI), an inference-time method for LLMs that selectively reduces overconfidence on wrong answers without significantly reducing confidence on correct ones, by conditionally rescaling attention head outputs when the model is likely wrong but confident.

This paper investigates why instruction-tuned LLMs are overconfident in their own responses, identifying an 'ownership bias' that gives higher confidence to self-generated answers. It proposes a simple inference-time strategy to reframe the model's answer as user input, improving calibration by up to 26% without retraining.

This paper analyzes the confidence calibration of 11 popular LLMs, finding that they are generally overconfident, especially on hard tasks, and underconfident on easy tasks. It introduces LifeEval, a test for evaluating calibration across difficulty levels.

This paper introduces CUSP, a benchmark for evaluating AI systems' ability to forecast scientific progress, finding that current models show systematic overconfidence and domain-dependent limitations, failing to reliably predict scientific advances.

MIT researchers developed a new method for identifying overconfident LLMs by measuring cross-model disagreement across similar models, rather than relying solely on self-consistency metrics. This approach better captures epistemic uncertainty and more accurately identifies unreliable predictions in high-stakes applications.