Tag
This article discusses how AI agents, when incentivized to pass test checks, write superficial tests that satisfy automated gates but lack real verification, citing Goodhart's law and recent studies on the issue in coding environments like AIPass.
The article reflects on the rapid progress in AI models from three years ago, comparing early models like Bard that struggled with coding tests to current models like Qwen 27B and Opus 5.5, and speculates on future advancements.
A local experiment comparing Qwen 35B-A3B MoE and Qwen 27B dense on coding-maintenance tasks, finding the MoE model ~3.9× faster with a smaller quality gap than expected.