@OpenAI: To audit SWE-Bench Pro, we used model-based investigator agents alongside independent reviews from five independent exp…
Summary
OpenAI describes its audit of SWE-Bench Pro using model-based investigator agents and independent reviews from experienced software engineers to ensure thorough evaluation at scale.
View Cached Full Text
Cached at: 07/09/26, 07:43 AM
To audit SWE-Bench Pro, we used model-based investigator agents alongside independent reviews from five independent experienced software engineers.
That helped us examine tasks at scale while keeping expert judgment at the center. https://t.co/3PNbk57uvF
Similar Articles
@OpenAI: We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures fr…
OpenAI audited the SWE-Bench Pro coding benchmark and found approximately 30% of tasks broken, retracting their previous recommendation for its use as a leading coding evaluation.
Introducing SWE-bench Verified
OpenAI is releasing SWE-bench Verified, a human-validated subset of the SWE-bench benchmark designed to more reliably evaluate AI models' ability to autonomously solve real-world software engineering tasks. The release addresses issues with overly specific or irrelevant unit tests that caused correct solutions to be incorrectly rejected.
Someone did an audit on the new DeepSWE, the results aren't pretty
DeepSWE is a new benchmark for evaluating AI coding agents on real-world software engineering tasks from active open-source repositories, comprising 113 tasks across TypeScript, Go, Python, JavaScript, and Rust with isolated environments and program-based verifiers.
Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
Senior SWE-Bench is an open-source benchmark that evaluates AI agents on software engineering tasks requiring senior-level skills.
Separating signal from noise in coding evaluations
OpenAI audited SWE-Bench Pro, a coding benchmark, and found approximately 30% of tasks are broken due to issues like overly strict tests and underspecified prompts, advising model developers to carefully examine results.