DeepSWE-mini a 16 instance subset of DeepSWE that replicates the rankings of the leaderboard
Summary
Released a new dataset called DeepSWE-mini, a 16-instance subset of DeepSWE designed for efficient benchmarking of local AI models by replicating leaderboard rankings.
Similar Articles
@xdotli: mini-swe-agent is impressive. 100 lines, one bash tool, same prompt for every model tops on DeepSWE by @datacurve where…
mini-swe-agent is a minimal, open-source SWE-agent implementation that tops DeepSWE benchmarks with just 100 lines of code and a single bash tool. The team also open-sourced mini-swe-code for interactive use and mini-swe-acp for evaluation harness across benchmarks.
Someone did an audit on the new DeepSWE, the results aren't pretty
DeepSWE is a new benchmark for evaluating AI coding agents on real-world software engineering tasks from active open-source repositories, comprising 113 tasks across TypeScript, Go, Python, JavaScript, and Rust with isolated environments and program-based verifiers.
DeepSWE for GPT-5.6
DeepSWE is a specialized variant of GPT-5.6 tailored for software engineering tasks.
DeepSWE Opus 4.8 results have been released.
The results of DeepSWE Opus 4.8 have been released, showcasing its performance on benchmarks.
Real-SWE Benchmark (new)
Real-SWE is a new benchmark that evaluates frontier AI models on private, real-world enterprise codebases, focusing on complex software engineering tasks with business context.