SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents

Hugging Face Daily Papers Papers
software-engineering benchmark ai-agents reliability evaluation

Summary

SWE-Bench Pro Verified addresses reliability issues in the SWE-Bench Pro benchmark by eliminating reward hacking and improving task quality, providing a more accurate evaluation for software engineering agents.

SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present SWE-Bench Pro Verified, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines anti-hacking safeguards that eliminate major leakage channels without disrupting normal agent functionality, with task refinement that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously reported, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.
Original Article
View Cached Full Text

Cached at: 09/10/26, 10:10 AM

Paper page - SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents

Source: https://huggingface.co/papers/2609.08149

Abstract

SWE-BenchProhasemergedasastandardbenchmarkforevaluatingsoftwareengineeringagentsonchallengingrepository-leveltasks.However,ouranalysisworkshowthatitsevaluationisunderminedbytwosourcesofunreliability:rewardhacking,enabledbyleakageofgoldsolutionsorhiddenevaluationinformation,andtaskqualityissues,includingmisleadingproblemstatementsandimproperlyscopedtests.Theseissuescaninflatebenchmarkperformanceandobscureagents’truecodingability.WepresentSWE-BenchProVerified,averifiedversionofSWE-BenchProthataddressesbothproblems.Ourapproachcombinesanti-hackingsafeguardsthateliminatemajorleakagechannelswithoutdisruptingnormalagentfunctionality,withtaskrefinementthatminimallycorrectsinconsistencieswithinflawedinstances.EvaluationsonSWE-BenchProVerifiedrevealthatsomemodelsperformsubstantiallyworsethanpreviouslyreported,suggestingthatexistingresultsonSWE-BenchPromayoverestimaterealsoftwareengineeringcapability.SWE-BenchProVerifiedoffersamoretrustworthybenchmarkforassessingsoftwareengineeringagents.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.08149

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.08149 in a model README.md to link it from this page.

Datasets citing this paper1

#### opencompass/SWEBench-Pro-Verified Viewer• Updated31 minutes ago • 731 • 68

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.08149 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Introducing SWE-bench Verified

OpenAI Blog

OpenAI is releasing SWE-bench Verified, a human-validated subset of the SWE-bench benchmark designed to more reliably evaluate AI models' ability to autonomously solve real-world software engineering tasks. The release addresses issues with overly specific or irrelevant unit tests that caused correct solutions to be incorrectly rejected.