@stanine: Hmm. Today, we ran our 2100 scored runs with Grok 4.6. Versus 4.5, the pass rate regressed from 87.3% to 85.9%, and nea…
Summary
The article reports on performance regression in Grok 4.6 compared to 4.5, with lower pass rate and higher latency, affecting practical business tasks.
View Cached Full Text
Cached at: 08/15/26, 01:49 AM
Hmm. Today, we ran our 2100 scored runs with Grok 4.6. Versus 4.5, the pass rate regressed from 87.3% to 85.9%, and nearly doubled the median latency (71s → 131s). Grok 4.6 also errored out more (“Something went wrong”), refused to answer benign questions (e.g., “Who in Sales has access to Google Workspace”), and produced incomplete outputs but claimed success (returning 54% of the required fields, but asserting 100%).
Could be a matter of prompt tuning, but it feels like we’re at the Pareto front (at least for now) on some of these models with everyday business tasks.
Read the original post for details on the shape of data and tasks RipplingBench represents; it’s very practical.
Matt MacInnis (@stanine): New: we ran ~2100 scored runs on open-weight models and lab models and found some big surprises. Grok (@spacexai) was the value leader, and GLM (@Zai_org) is super close to the frontier. I guess this is why the ‘labs’ are nervous…
Similar Articles
@ericzakariasson: and here’s the grok 4.7 model card. a few jumps vs 4.6 that stood out: - Terminal-Bench: 20.3% → 38.0% - SWE-Marathon: …
The tweet shares the Grok 4.7 model card, highlighting significant performance improvements over version 4.6 on benchmarks like Terminal-Bench, SWE-Marathon, and HealthBench Pro, with the same price point.
@gregpr07: Grok 4.6 is within 1 point of Opus 5 at ~40% lower cost. Across 7 runs, it solved 105/106 hard browser tasks. A little …
Grok 4.6 achieves near-Opus 5 performance at a 40% lower cost, solving 105/106 hard browser tasks, indicating potential for further advancement with reinforcement learning.
@corbin_braun: W
Grok 4.7 has been released, offering a notable improvement over Grok 4.6 while maintaining the same price and speed.
@larsencc: Grok 4.7 is insanely good. The model is not the bottleneck anymore. I believe harness reliability and everything around…
The author praises Grok 4.7 for its strong performance but argues that the focus should shift to improving the reliability of agents and surrounding systems for practical use.
Grok 4.7 benchmarks
This article likely discusses the benchmark results for the Grok 4.7 AI model, comparing its performance across various tasks.