[PAPER] Major benchmarks are found to be polluted, with up to 12% of questions broken
Summary
This paper audits major LLM benchmarks (GPQA, MMLU-Pro, MMMU-Pro) and finds 4.5–11.8% of items have wrong answer keys or are malformed, releasing corrected 'Clean' versions for drop-in use.
View Cached Full Text
Cached at: 07/28/26, 08:28 PM
adamallcock/answer-key-audit
Source: https://github.com/adamallcock/answer-key-audit
When the Answer Key Is Wrong
A Calibrated Audit of GPQA, MMLU-Pro, and MMMU-Pro — with Corrected Releases
Adam Allcock, 2026
📄 Read the paper (PDF) · 
· arXiv: forthcoming
Public LLM leaderboards rank models to the tenth of a percent — against answer keys that are themselves wrong. Auditing four of the most-quoted multiple-choice benchmarks under one taxonomy (wrong keys, malformed items, and questions with more than one defensible answer), 4.5–11.8% of items can’t fairly score a model, against a near-zero rate on a saturated MATH-500 control.
We don’t just diagnose — we ship the fix: corrected drop-in “-Clean” releases of all four, with per-item confidence tiers, the full flagged-candidate ledger, and dual original-vs-cleaned scoring, so an existing eval harness rescores with no code changes.
Two results travel beyond these sets: our model-adjudicated verdicts track GPQA’s own recorded expert-disagreement (external validation, not circular), and the standard single-pass disagreement method undercounts the rate by 2.3×. An audit is itself a measurement instrument, so we calibrate ours against injected defects and never-flagged controls and report rates as bands, not points. Model-adjudicated and calibrated — not human-certified.
Corrected drop-in releases (the -Clean family)
| Release | Source | Cleaned | License |
|---|---|---|---|
| gpqa-diamond-clean | GPQA Diamond | 198 → 189 | CC BY 4.0 |
| gpqa-ext-complement-clean | GPQA Extended (minus Diamond) | 348 → 309 | CC BY 4.0 |
| mmlu-pro-clean | MMLU-Pro | 12,032 → 10,689 | MIT |
| mmmu-pro-clean | MMMU-Pro (standard, 10-option) | 1,730 → 1,526 | Apache-2.0 |
Each is a drop-in: a harness that runs the original runs the -Clean set unchanged, and every removal is auditable item by item.
Citation
Allcock, A. (2026). When the Answer Key Is Wrong: A Calibrated Audit of GPQA, MMLU-Pro, and MMMU-Pro, with Corrected Releases. Zenodo. https://doi.org/10.5281/zenodo.21613590
@misc{allcock2026answerkey,
title = {When the Answer Key Is Wrong: A Calibrated Audit of GPQA, MMLU-Pro, and MMMU-Pro, with Corrected Releases},
author = {Allcock, Adam},
year = {2026},
doi = {10.5281/zenodo.21613590},
publisher = {Zenodo}
}
Similar Articles
Quantifying Ranking Uncertainty in LLM Benchmarks
This paper analyzes sources of ranking uncertainty in LLM benchmarks like MMLU, proposing modifications to hypothesis tests for constructing rank confidence intervals, and shows that variability across subjects is substantial.
The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer
This paper reveals a 'benchmark illusion' where pruned LLMs perform well on multiple-choice benchmarks but fail to answer the same questions in open generation, suggesting that compressed models should be tested on generative tasks rather than just recognition tasks.
Benchmarking LLM Competence on Logical Inference over Probability Operators
This paper introduces a benchmark of 14,320 procedurally-generated prompts for evaluating LLMs on logical inference over probability operators like 'probably', 'might', and 'must'. Testing 29 models, the authors find systematic answer biases and show that only 9 exceed random chance.
Auditing LLM Benchmarks with Item Response Theory
This paper introduces an Item Response Theory-based method to detect mislabeled examples in LLM benchmarks at 95% precision, tracing errors to labeling heuristics and annotation issues.
LLM Performance on a Real, Double-Marked GCSE Benchmark
Introduces a dataset of 32,534 double-marked GCSE student responses across five subjects, finding that top-performing LLMs agree with examiners more closely than examiners agree with each other, including on handwritten and subjective tasks.