Auditing LLM Benchmarks with Item Response Theory

arXiv cs.CL Papers
llm-benchmarks item-response-theory benchmark-auditing mislabel-detection ai-safety evaluation llm-evaluation

Summary

This paper introduces an Item Response Theory-based method to detect mislabeled examples in LLM benchmarks at 95% precision, tracing errors to labeling heuristics and annotation issues.

arXiv:2605.30504v1 Announce Type: new Abstract: LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all. We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple-choice benchmarks using responses from 114 models, outperforming a supervised classifier. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label. The same model fit reveals that reward models specialize in stylistic preference rather than factual knowledge, and identifies one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with benchmark contamination or benchmark-specific over-optimization.
Original Article
View Cached Full Text

Cached at: 06/01/26, 09:24 AM 

# Auditing LLM Benchmarks with Item Response Theory
Source: [https://arxiv.org/abs/2605.30504](https://arxiv.org/abs/2605.30504)
[View PDF](https://arxiv.org/pdf/2605.30504)

> Abstract:LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all\. We introduce an Item Response Theory\-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple\-choice benchmarks using responses from 114 models, outperforming a supervised classifier\. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label\. The same model fit reveals that reward models specialize in stylistic preference rather than factual knowledge, and identifies one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with benchmark contamination or benchmark\-specific over\-optimization\.

## Submission history

From: Sander Land \[[view email](https://arxiv.org/show-email/607baf0b/2605.30504)\] **\[v1\]**Thu, 28 May 2026 19:38:44 UTC \(421 KB\)

Similar Articles

Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment

arXiv cs.CL

This paper investigates whether standard benchmarks underestimate LLM performance by re-evaluating hallucination detection datasets using an LLM-first, human-adjudicated assessment method. The study finds that incorporating LLM reasoning into the adjudication process improves agreement and suggests that model-assisted re-evaluation yields more reliable benchmarks for ambiguity-prone tasks.

Benchmarking Different Methods of LLM Confidence Estimation

Reddit r/artificial

This article benchmarks various blackbox and whitebox methods for LLM confidence estimation, including verbalized confidence, linguistic uncertainty, reasoning-length, P(Answer), P(True), and self-consistency, comparing their effectiveness for tasks like active learning and safety classification.

Quantifying Ranking Uncertainty in LLM Benchmarks

arXiv cs.LG

This paper analyzes sources of ranking uncertainty in LLM benchmarks like MMLU, proposing modifications to hypothesis tests for constructing rank confidence intervals, and shows that variability across subjects is substantial.