Auditing LLM Benchmarks with Item Response Theory
Summary
This paper introduces an Item Response Theory-based method to detect mislabeled examples in LLM benchmarks at 95% precision, tracing errors to labeling heuristics and annotation issues.
View Cached Full Text
Cached at: 06/01/26, 09:24 AM
# Auditing LLM Benchmarks with Item Response Theory Source: [https://arxiv.org/abs/2605.30504](https://arxiv.org/abs/2605.30504) [View PDF](https://arxiv.org/pdf/2605.30504) > Abstract:LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all\. We introduce an Item Response Theory\-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple\-choice benchmarks using responses from 114 models, outperforming a supervised classifier\. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label\. The same model fit reveals that reward models specialize in stylistic preference rather than factual knowledge, and identifies one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with benchmark contamination or benchmark\-specific over\-optimization\. ## Submission history From: Sander Land \[[view email](https://arxiv.org/show-email/607baf0b/2605.30504)\] **\[v1\]**Thu, 28 May 2026 19:38:44 UTC \(421 KB\)
Similar Articles
BenchMIRT: What are LLM benchmarks actually measuring?
BenchMIRT introduces a method to audit LLM benchmarks at the individual prompt level using multidimensional item response theory, separating underlying capabilities like safety and general reasoning to reveal what benchmarks actually measure.
Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
This paper investigates whether standard benchmarks underestimate LLM performance by re-evaluating hallucination detection datasets using an LLM-first, human-adjudicated assessment method. The study finds that incorporating LLM reasoning into the adjudication process improves agreement and suggests that model-assisted re-evaluation yields more reliable benchmarks for ambiguity-prone tasks.
Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks
This paper introduces LLM-NRM, an option-level psychometric framework for multiple-choice benchmarks that models the full distribution over answer choices rather than binary correctness, showing that incorrect responses carry useful measurement information and improving ability estimation and benchmarking efficiency.
Benchmarking Different Methods of LLM Confidence Estimation
This article benchmarks various blackbox and whitebox methods for LLM confidence estimation, including verbalized confidence, linguistic uncertainty, reasoning-length, P(Answer), P(True), and self-consistency, comparing their effectiveness for tasks like active learning and safety classification.
Quantifying Ranking Uncertainty in LLM Benchmarks
This paper analyzes sources of ranking uncertainty in LLM benchmarks like MMLU, proposing modifications to hypothesis tests for constructing rank confidence intervals, and shows that variability across subjects is substantial.