Integrity Bench by AI Explained and Pablo Romero - Measuring how overconfident a model is
Summary
The Integrity Bench is a benchmark developed by AI Explained and Pablo Romero to measure how overconfident frontier AI models are in their own abilities, helping to quantify this common issue in AI systems.
Similar Articles
MoralityBench.ai: Morality Leaderboard for AI
MoralityBench.ai is a new benchmark adapting moral psychology tests to evaluate AI models, finding that most models' moral answers show considerable variance across runs — suggesting morality is not deterministic in LLMs.
ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
ConfidenceBench is a new benchmark that evaluates verbalized confidence estimates in large language models using Brier scores, revealing that accuracy and calibration diverge and that even highly accurate models can be severely miscalibrated.
EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models
This paper introduces EvalDetectBench, an open benchmark and pipeline for measuring evaluation awareness in frontier language models, addressing biases in existing methods to improve AI safety assessments.
Unsteady Metrics and Benchmarking Cultures of AI Model Builders
This paper introduces Benchmarking-Cultures-25, a dataset analyzing how AI model builders selectively highlight benchmarks in press releases. It finds a fragmented evaluation landscape with limited cross-model comparability, arguing that benchmarks are used as narrative devices for market positioning rather than standardized scientific measurement.
ObserverBench: Testing Mechanistic Estimates for Intervention and Control
ObserverBench is a benchmark framework designed to evaluate whether internal estimators in AI models are adequate for guiding interventions, control, or safety tasks by reporting both estimation accuracy and downstream action performance.