When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs
Summary
This paper demonstrates that global calibration metrics like Expected Calibration Error are confounded by model accuracy, and proposes ACE, an accuracy-controlled evaluation framework for fair comparison of large language models.
View Cached Full Text
Cached at: 07/01/26, 05:31 AM
# When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs Source: [https://arxiv.org/abs/2606.30814](https://arxiv.org/abs/2606.30814) [View PDF](https://arxiv.org/pdf/2606.30814) > Abstract:Calibration evaluates whether a model confidence aligns with its empirical accuracy\. Existing studies often compare the calibration of different large language models using global calibration metrics such as Expected Calibration Error and Brier Score\. We begin by showing, both theoretically and empirically, that such comparisons are confounded by differences in model accuracy\. For fairer cross\-model comparison, we then propose ACE, an accuracy\-controlled evaluation framework with three complementary views: Instance\-Aligned, Distribution\-Aligned, and Candidate\-Aligned calibration\. Across multiple benchmarks, model families, and confidence elicitation methods, we use ACE to study two practically important comparison axes, small versus large models and thinking versus non\-thinking models\. We find that many previously reported calibration advantages under raw global metrics weaken substantially after accuracy control\. We also find that ranking reversal is frequent: models favored by raw metrics often cease to be favored once accuracy is controlled\. Our results show that raw global calibration metrics are not robust for cross\-model comparison, and that fair calibration comparison requires accuracy\-aware evaluation\. ## Submission history From: Ruihan Yang \[[view email](https://arxiv.org/show-email/6c7a7840/2606.30814)\] **\[v1\]**Mon, 29 Jun 2026 18:37:48 UTC \(9,358 KB\)
Similar Articles
Confidence Calibration in Large Language Models
This paper analyzes the confidence calibration of 11 popular LLMs, finding that they are generally overconfident, especially on hard tasks, and underconfident on easy tasks. It introduces LifeEval, a test for evaluating calibration across difficulty levels.
Stable Miscalibration in Large Language Models: A Practical View of High-Confidence Errors
This paper studies stable miscalibration in large language models, where high-confidence errors remain locally stable under perturbations, using diagnostics like audit scores and probes to assess calibration and internal sensitivity.
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
This paper proposes a scalable, domain-agnostic framework for automated LLM evaluation that uses pairwise comparisons by multiple LLMs and an Elo rating system to approximate expert judgments, reducing the need for human intervention.
ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
ConfidenceBench is a new benchmark that evaluates verbalized confidence estimates in large language models using Brier scores, revealing that accuracy and calibration diverge and that even highly accurate models can be severely miscalibrated.
Human-Alignment, Calibration, and Activation Patterns in Large Language Model Uncertainty
This paper investigates how similar large language model uncertainty is to human uncertainty, exploring alignment, calibration, and activation patterns in LLMs across multiple datasets and the impact of instruction fine-tuning.