@jerryjliu0: An interesting thing I'm observing from the blog/system card is that on a good chunk of the reported benchmarks (~20-30…
Summary
An observation that Claude Opus 5's max thinking leads to performance degradation on ~20-30% of benchmarks compared to xhigh, contrary to the expectation that more test-time compute improves performance.
View Cached Full Text
Cached at: 07/25/26, 08:03 AM
An interesting thing I’m observing from the blog/system card is that on a good chunk of the reported benchmarks (~20-30% from a skim), Opus 5 max thinking leads to a degradation in performance compared to xhigh.
Usually you would assume that as you increase thinking and test-time compute, performance goes up. Some of these results contradict that assumption. I wonder if this is a natural emergent property of smaller models or a posttraining issue.
Claude (@claudeai): Introducing Claude Opus 5.
It’s a thoughtful and proactive model that comes close to the frontier intelligence of Fable 5 at half the price.
Similar Articles
@no_stp_on_snek: someone will wave the card at me: the 9B crushes its base on the coding benchmarks (SWE-bench 69 vs 53). true. but on m…
A commentator discusses the performance of a 9B model on coding benchmarks, noting that while it beats its base on SWE-bench (69 vs 53), the advantage narrows on behavioral and long-horizon tests, suggesting limited gains outside benchmark distributions.
@elliotarledge: Claude Fable 5 [max] on KernelBench-Hard. The main kernel that impressed me was a B200 fp8 GEMM: it HAND WROTE raw SM10…
Claude Fable 5 achieves top results on KernelBench-Hard by hand-writing PTX code for B200 fp8 GEMM, outperforming other models and reaching 44-59% of peak performance on compute-bound shapes.
@polynoamial: https://x.com/polynoamial/status/2064210146558136827
This article argues that LLM benchmark performance is increasingly a function of test-time compute, and that current evaluation methods fail to capture capability improvements when controlling for inference budget. It advocates for plotting performance vs. tokens, cost, or time, and discusses implications for safety evaluations.
@injaneity: https://x.com/injaneity/status/2075659478096376158
This article explains how batching and parallel operations improve latency and efficiency in AI computer use systems, highlighting open-source implementations like pi-computer-use and cua-driver that achieved significant performance gains before similar features appeared in Codex.
@KLieret: Very interesting study from Opus 4.8 card: Multi-agents do not deliver better results on ProgramBench, but they get to …
A study from the Opus 4.8 card shows that while multi-agent systems do not achieve better results on ProgramBench, they reach mediocre solutions twice as fast.