Hy3 Benchmark Roundup: from SWE-Bench Pro to 312 real-world workflow tasks
Summary
A roundup of benchmarks including SWE-Bench Pro and 312 real-world workflow tasks, likely evaluating AI performance on software engineering challenges.
Similar Articles
SWE-rebench leaderboard update: GLM-5.2, Qwen3.6-27B, Qwen3.6-35B-A3B, Gemma 4 31B and more + improved UI
SWE-rebench leaderboard updated with new models (GLM-5.2, Qwen3.6, Gemma 4 31B, etc.) and an improved UI, showing performance rankings on software engineering tasks.
Senior SWE Bench: a new benchmark focussed on realistically underspecified feature tasks
Senior SWE-Bench is a new open-source benchmark designed to evaluate AI agents on realistic, underspecified software engineering tasks, emphasizing skills like intent alignment and code quality rather than overly detailed specifications.
CursorBench 3.1
CursorBench 3.1 introduces new benchmark tasks focused on codebase understanding, bugfinding, planning, and code review, and presents updated scores and cost comparisons for various AI models.
Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
Senior SWE-Bench is an open-source benchmark that evaluates AI agents on software engineering tasks requiring senior-level skills.
Someone did an audit on the new DeepSWE, the results aren't pretty
DeepSWE is a new benchmark for evaluating AI coding agents on real-world software engineering tasks from active open-source repositories, comprising 113 tasks across TypeScript, Go, Python, JavaScript, and Rust with isolated environments and program-based verifiers.