Let a cheap model fix real bugs overnight, and most of its PRs didn't survive review

Reddit r/AI_Agents News

Summary

The author describes running an overnight bug-fixing experiment where a cheap model (DeepSeek) opened 14 PRs, but only 6 survived review by a stronger model (GPT-6.1 Sol) due to regressions, ineffective tests, and security bypasses. They then flipped the pipeline to have the strong model design and review while the cheap model only implements.

This is my own setup, my own repos, and I'm a bit stuck on what to do with the result. An audit turned up some serious bugs. Instead of fixing them myself I let a cheap, fast model (DeepSeek, during an API discount window) grind through them overnight. Round 1 was 14 PRs. I had a stronger model (GPT-6.1 Sol) re-review every one of them. 6 got merged after all CI passed. I closed the other 8. The reasons were things like a fix that broke other valid inputs, a test that needed a tool our CI doesn't have, a bypass that still worked, a key test that never even ran in CI, and a new test that couldn't catch the bug even when I deliberately broke the code. So for round 2 I flipped it. A stronger model designs each fix, another one reviews the design, and the cheap model only implements. That's 7 PRs so far and none approved yet. Where do cheap-model fixes fail for you? And what split between planning, implementing and reviewing has actually worked?
Original Article

Similar Articles

A cheaper AI model is not necessarily cheaper once retries are counted

Reddit r/artificial

A side-by-side coding experiment comparing GPT-5.6 Luna and DeepSeek V4 Flash shows that DeepSeek's apparent 5x price advantage shrinks when retries are included. The article argues for more comprehensive benchmarks reporting cost per attempt and cost per verified success.