Tag
Claude Opus 5.5 (xhigh) resolves 18.5% of ProgramBench's 200 from-scratch program rebuild tasks, up from 0% at the benchmark's May launch, per Jack Clark's Import AI. Clark predicts the benchmark could saturate by end of 2027, raising questions about whether it truly measures software engineering ability or just reverse-engineering persistence.
ProgramBench results show Fable 5 achieving double the performance of Opus 4.8, even with fallback to 4.8 in 99% of runs.
A study from the Opus 4.8 card shows that while multi-agent systems do not achieve better results on ProgramBench, they reach mediocre solutions twice as fast.
GPT5.5 achieved the first solve on the difficult ProgramBench SWE benchmark, significantly outperforming Opus 4.7.