@ArizePhoenix: The result: a 50% improvement over 5.2 on their in-house Code Bench, Terminal-Bench 3.0 from 4.6 → 28.3, DeepSWE v1.1 f…

X AI KOLs Following Models
ai-benchmarks code-generation software-engineering open-weights sota cyber-capability

Summary

DeepSWE v1.1 shows a 50% improvement on in-house Code Bench and sets new state-of-the-art scores on Terminal-Bench 3.0 and Agents' Last Exam, with emerging cyber capabilities advancing faster than expected.

The result: a 50% improvement over 5.2 on their in-house Code Bench, Terminal-Bench 3.0 from 4.6 → 28.3, DeepSWE v1.1 from 46.2 → 66.9, and open-weights SOTA on Terminal-Bench 3.0 and Agents' Last Exam. Cyber capability emerged faster than they expected.
Original Article

Similar Articles

Someone did an audit on the new DeepSWE, the results aren't pretty

Reddit r/singularity

DeepSWE is a new benchmark for evaluating AI coding agents on real-world software engineering tasks from active open-source repositories, comprising 113 tasks across TypeScript, Go, Python, JavaScript, and Rust with isolated environments and program-based verifiers.

SWE-Bench Pro V2 (9 minute read)

TLDR AI

SWE-Bench Pro V2 is an updated benchmark for evaluating AI agents in software engineering, featuring 642 tasks across 11 repositories with improved evaluation protocols and contamination controls.