@_akhaliq: SWE-Bench ProMax Benchmarking Agents on Large-Scale Multilingual Code Refactoring paper: https://huggingface.co/papers/…
Summary
Introduces SWE-Bench ProMax, a multilingual code refactoring benchmark of 170 instances across 7 programming languages to evaluate AI coding agents. Frontier models achieve only 41.2% resolve rate, confirming it as a challenging and unsaturated benchmark.
View Cached Full Text
Cached at: 08/11/26, 05:45 AM
SWE-Bench ProMax
Benchmarking Agents on Large-Scale Multilingual Code Refactoring
paper: https://t.co/2DO56pDC95 https://t.co/JtZpMSoTNF
Paper page - SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
Source: https://huggingface.co/papers/2608.09802 Published on Aug 10
·
Submitted byhttps://huggingface.co/YerbaPage
Yulingon Aug 11
#1 Paper of the day Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
AsAIcodingagentstakeonincreasinglycomplex,long-horizonsoftwareengineeringtasks,existingbenchmarksarerapidlysaturatingandtheirevaluationqualityhascomeunderseriousscrutiny:arecentauditfoundthatnearly60%ofunsolvedSWE-benchVerifiedinstancescontainflawedtests--eitheroverlynarrowteststhatrejectcorrectsolutionsoroverlybroadteststhatcheckunstatedrequirements--andthatfrontiermodelscanverbatimreproducegoldpatchesfromtrainingdata.Coderefactoring,whichrequirescoordinated,behavior-preservingchangesacrossmanyfiles,offersasubstantiallyharderandmorerealistictestofagentcapability,yetremainsunderservedbycurrentbenchmarks.WeintroduceSWE-BenchProMax,anexpert-curated,multilingualcoderefactoringbenchmarkof170instancesdrawnfromrealcommitsacrosssevenprogramminglanguages(Python,Java,TypeScript,Go,C,C++,andRust).Everyinstanceundergoesrigorous,multi-stagecurationthatdirectlyaddressesthequalityproblemsidentifiedinpriorbenchmarks:issuedescriptionsarerewrittenfromscratchtoprovideprecise,unambiguousspecifications,andtestsuitesaremanuallyreviewedtoremoveoverlynarrowandoverlybroadtests.Taskswithinsufficientcomplexityorlimitedcross-filescopearefilteredout,yieldingabenchmarkofchallenging,large-scalerefactoringtasksthataverage11.4modifiedfilesand261.6linesofcodeperinstance,substantiallyexceedingthescaleofexistingbenchmarks.Experimentswithfrontiermodelsundertwoagentscaffoldsshowthatthebestmodelachievesonly41.2%resolverate,confirmingthatSWE-BenchProMaxpresentsameaningfulandunsaturatedchallengeforcurrentAIcodingagents.Ourbenchmarkisavailableathttps://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2608\.09802
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.09802 in a model README.md to link it from this page.
Datasets citing this paper1
#### swe-bench-promax/SWE-Bench-ProMax Viewer• Updatedabout 2 hours ago • 170
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.09802 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
SWE Context Bench just proved something I think a lot of coding agent users already feel
A new benchmark paper 'SWE Context Bench' tests whether coding agents can reuse knowledge across tasks, highlighting a gap in existing benchmarks that only evaluate isolated problem-solving. The author discusses solutions like external memory and mentions tools such as langmem, mem0, supermemory, and Greplica.
@swyx: It's finally out!!! @METR_Evals found that more than half of SWEBench results is unmergeable slop. FrontierCode represe…
FrontierCode is a new coding benchmark from METR and Cognition that evaluates AI models on code maintainability and quality, revealing that many models produce unmergeable code. It includes over 1000 hours of work and shows that even top models struggle, with Opus 4.8 achieving only 13.8% on the hardest tier.
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
This paper audits three performance-optimization benchmarks (GSO, SWE-Perf, SWE-efficiency) for coding agents, finding that runtime instability, scoring rules, and task coverage significantly affect reliability, and that many tasks are already solved by at least one public submission.
ProgramBench (5 minute read)
ProgramBench is a new benchmark that evaluates AI agents' ability to reconstruct complete software projects from compiled binaries and documentation without access to source code or decompilation tools.
TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework
TensorBench is a benchmark of 199 feature-addition and refactoring tasks on a compiler-based tensor framework, evaluating seven coding agents with pass rates ranging from 22.1% to 64.8%.