Tag
DeepSeek's new V4-Flash-0731 model is now far outperforming the V4-Pro-Preview in benchmarks, marking a significant improvement in the model family.
A comparison shows that LLM performance on the 2026 IMO varies dramatically depending on the evaluation harness, with structured multi-agent setups achieving far higher scores than simple web UI, indicating that current gains are absorbed at the frontier by better orchestration.
Discussion of dynamic workflows for test-time compute, including their limited use cases, benefits for research experiments, and the need for better benchmarks. Mentions models like Mythos and Opus 4.8 for agent orchestration.
The author describes losing faith in public AI model benchmarks due to vendor-created metrics, self-reported parameters, and lack of independent verification, and advocates for building custom evaluation sets from real production traffic to make more relevant model comparisons.
Qwen3.7 Max ranks 5th on Artificial Analysis benchmarks, matching GPT-5.4 and outperforming Gemini 3.5 Flash, while Qwen3.6 27B trails significantly.
Google DeepMind researcher Lun Wang leaves the company and writes a post criticizing the current AI evaluation system, arguing that it lags behind model evolution and cannot predict new capabilities, leaving the industry in a state of 'flying blind'.