I tested AI agents on fixing real security bugs. Here's what I found.

A lot of buzz around AI and security lately. As a researcher, I put myself to indepently test if agents can actually fix real security vulnerabilities. So I built a benchmark. 20 real vulnerabilities from real-world projects (Pillow, GitPython, yt-dlp, urllib3, and others; Python focused, sorry 🙏), 5 models (or what my budget could pay for sorry again 🙏), each agent sandboxed with a constrained toolset and scored against hidden tests. Three things stood out: * **Fixing security bugs is still hard.** The best solve rate was 50%. These are frontier models with full access to the codebase and a clear task. Honestly, I didn't expect this low. * **Expensive models aren't worth it.** gpt-5.5 costs 12× more per run than gpt-5.4-mini for statistically equivalent outcomes. The deliberation doesn't translate into better fixes. * **The dangerous failure is a convincing wrong answer.** Agents would edit the right file, run their own tests, see them pass, and stop. Fixes made sense. But, testing against a hidden grading script showed that vulnerability was still there . That last one is what keeps me up at night when I think about deploying agents on anything security-critical. Full writeup and traces in the comments. Code and result traces are open-sourced.