@gdb: benchmarks get saturated very quickly these days
Summary
A tweet notes that benchmarks quickly become saturated, citing the example of a model called GPT-5.6 Sol Pro scoring 91/99 on prinzbench, with two questions remaining unsolved.
View Cached Full Text
Cached at: 07/17/26, 12:19 AM
benchmarks get saturated very quickly these days
prinz (@deredleritt3r): Added to prinzbench: GPT-5.6 Sol Pro.
As previewed a few days ago, this model has saturated my benchmark, with a total score of 91/99.
For context, prinzbench contains two questions that no model tested to date has ever been able to solve (one requires extremely thorough
Similar Articles
@BenjaminDEKR: So it's just over? GPT5.6 Sol Ultra scores 91.9% on TerminalBench Coding is approaching solved, the same way arithmetic…
GPT5.6 Sol Ultra achieves 91.9% on TerminalBench coding benchmark, suggesting coding tasks are approaching solved.
On SWEBench Pro, 68.5% of GPT 5.5’s failures were caused by broken or incorrect test cases, totaling 28.9% of the entire benchmark
An analysis reveals that 28.9% of GPT 5.5's failures on SWEBench Pro are due to broken or incorrect test cases, and similar issues affect other major AI benchmarks, raising concerns about the accuracy of current evaluation methods.
Introducing BenchBench (5 minute read)
Introduces BenchBench, a benchmark that tests AI models' ability to create effective benchmarks for other models, with GPT 5.2 being the only successful winner so far while frontier models like GPT 5.5 and Opus 4.6 struggled.
GPT-5.6 Sol hits the ZeroBench human baseline at pass@5 without tools
GPT-5.6 Sol reportedly hits the ZeroBench human baseline at pass@5 without tools, meaning at least one of five attempts succeeds on the benchmark.
GPT-5.6 Sol preview is out and the benchmark gap is wider than I expected
OpenAI released a preview of GPT-5.6 Sol, showing a larger benchmark gap than anticipated.