@paulg: Paweł Huryn tested models' ability to find bugs planted in code. Cost increases exponentially with performance (note th…
Summary
Paweł Huryn tested AI models' ability to find planted bugs in code, revealing that performance improvements come with exponentially increasing costs, as shown in a chart with a log scale.
View Cached Full Text
Cached at: 09/23/26, 04:08 PM
Paweł Huryn tested models’ ability to find bugs planted in code. Cost increases exponentially with performance (note the log scale on the x axis). I had ChatGPT superimpose a trend line. Bargain = above the line. https://t.co/tI616AZGt2
Similar Articles
17 bugs in 10 weeks from AI security scanning
AI-based security scanning has discovered 17 bugs in Perfetto's trace processor over 10 weeks, highlighting the potential for AI to uncover vulnerabilities in long-tail code that previously received little attention.
under 2% quality gap but 10x cost difference: tested 5 models on identical tool calling tasks[D]
A developer tested five AI models on tool calling tasks and found that cheaper models perform within 2% of expensive models like Opus, with Tencent's Hunyuan under $1.50 vs Opus's $15, leading to a daily cost reduction from $40 to $9 by routing simpler tasks to cheaper models.
@IntuitMachine: Your AI coding agent just burned $2 on a single bug fix. You thought it was "cheap automation." Here's what 16,000 prod…
An analysis of AI coding agent costs reveals that agentic workflows can use up to 3,500x more tokens than a simple ChatGPT call, with most waste coming from redundant context loading. The article suggests tracking repeated file actions and using efficient models to cut costs.
more models more better. one expensive model is losing to three cheap ones, and there's a paper on it
A mixture-of-agents paper (arxiv 2406.04692) shows that a committee of cheap open models can outperform GPT-4o on AlpacaEval 2.0 by leveraging decorrelated errors, and the author shares similar real-world findings where multiple cheap models catch more bugs than a single expensive model.
You can just choose how many bugs you want now
The article discusses how AI coding tools enable nearly free bug-finding in software, highlighting challenges like increased workload, trade-offs in fixing bugs, and the need for better management strategies to maintain software quality.