Let a cheap model fix real bugs overnight, and most of its PRs didn't survive review
Summary
The author describes running an overnight bug-fixing experiment where a cheap model (DeepSeek) opened 14 PRs, but only 6 survived review by a stronger model (GPT-6.1 Sol) due to regressions, ineffective tests, and security bypasses. They then flipped the pipeline to have the strong model design and review while the cheap model only implements.
Similar Articles
Built a 6x cheaper CodeRabbit alternative using open source models
Built a cheaper alternative to CodeRabbit using open source models, achieving similar or better PR review accuracy at 6x lower cost, with features like auto-fix and prompt-based bug fixing.
more models more better. one expensive model is losing to three cheap ones, and there's a paper on it
A mixture-of-agents paper (arxiv 2406.04692) shows that a committee of cheap open models can outperform GPT-4o on AlpacaEval 2.0 by leveraging decorrelated errors, and the author shares similar real-world findings where multiple cheap models catch more bugs than a single expensive model.
A cheaper AI model is not necessarily cheaper once retries are counted
A side-by-side coding experiment comparing GPT-5.6 Luna and DeepSeek V4 Flash shows that DeepSeek's apparent 5x price advantage shrinks when retries are included. The article argues for more comprehensive benchmarks reporting cost per attempt and cost per verified success.
GPT-5.6 Luna vs GPT-6 Astra: is a $1.20 model good enough for code review?
Benchmarked GPT-5.6 Luna vs GPT-6 Astra across 50 real PRs, showing Astra found 92 bugs vs Luna's 69, with Luna catching 75% of bugs at 3.6% of the cost. Includes detailed evaluation breakdown and upcoming comparison with Fable 5.1.
Built an AI PR review workflow that generates actual fix PRs instead of just comments
The author built an AI-powered PR review workflow that generates actual fix PRs instead of just comments, claiming to be 6x cheaper than CodeRabbit and more accurate on large PRs.