A boring agent doing a boring job — triaging security scanner noise. But it actually works.

Since there is so much talk about AI agents, I decided to build my own one - something small, measurable, and cheap enough to run for real (yeah, right — more on that below), so I could think about numbers instead of marketing. Static analysis tools (Semgrep, Snyk, CodeQL, gosec) flag hundreds of potential vulnerabilities and most of them are false positives. Someone has to open the code behind each finding, follow the data flow, and decide whether it's real. That's the job the agent does. All those scanners emit a standard SARIF 2.1.0 file, so it doesn't care which one you use. Now many of them are also shipped with AI agents, so it isn't something very new, although you can use any model you want, including self-hosted ones. Btw, the agent itself is written in Go. Basically, the model gets two read-only tools — read_file and grep_repo — and decides for itself which files to open, what to grep for, and when it has enough to rule. A typical finding takes 3–8 turns: read the sink, grep for the source, follow the assignment chain, then rule. The agent doesn't create or fix any code. I ran it against OWASP BenchmarkJava — a deliberately vulnerable Java app that ships a CSV of ground truth. So the verdicts get compared against published answers rather than my judgment. I ran three models — Claude Sonnet, DeepSeek-V4-Pro and Kimi k3 — across 50 vulnerable files, which produced 61 scored findings. I tested a subset of issues in BenchmarkJava, just to keep things quite cheap. DeepSeek-V4-Pro — 37 exploitable, 15 benign, 9 uncertain. (2.7M in / 109k out tokens): | triage verdict | actually vulnerable | safe by design | |-------------------------------|---------------------|----------------------| | exploitable — fails the build | 37 ✅ caught | 0 ❌ blocked in error | | benign — suppressed, unseen | 3 ❌ missed | 12 ✅ cleared | | uncertain — left for a human | 9 — parked | 0 — parked | DeepSeek were uncertain about 9 of them(needs manual review). And here we already see what marketing slides won't tell - it missed 3 real ones marking them as safe. So, looks like at least Deepseek wasn't trained with that specific OWASP BenchmarkJava code. Once it is run - SAST Triage agent will create a PR with findings, so it is there for review. So yes, it doesn't magically fix everything, humans are very much needed in this process. Kimi k3 — 49 exploitable, 12 benign, 0 uncertain. (504k in / 55k out tokens): | triage verdict | actually vulnerable | safe by design | |-------------------------------|---------------------|----------------------| | exploitable — fails the build | 49 ✅ caught | 0 ❌ blocked in error | | benign — suppressed, unseen | 0 ❌ missed | 12 ✅ cleared | | uncertain — left for a human | 0 — parked | 0 — parked | Kimi k3 is straight up impressive and cheap, but I need to run against a bigger set. It still will miss some things, but man, not only it is cheap to use - it clears noise so well(I tried with some of my own projects, but numbers aren't ready yet). And below is the expensive one. Claude Sonnet 5 — 47 exploitable, 9 benign, 5 uncertain. (2.2M in / 65k out tokens): | triage verdict | actually vulnerable | safe by design | |-------------------------------|---------------------|----------------------| | exploitable — fails the build | 45 ✅ caught | 2 ❌ blocked in error | | benign — suppressed, unseen | 2 ❌ missed | 7 ✅ cleared | | uncertain — left for a human | 2 — parked | 3 — parked | I was reluctant to run Claude Opus as Sonnet spent $5 on this single run alone. SAST Triage supports caching, so the second run will be ~0, but still. Running Claude Sonnet on all Opengrep findings (about 2350 of them) will cost ~$220 and just about $7 for DeepSeek. Keep in mind that the agent doesn't need to run across the whole codebase, which is approximately 200k LoC for BenchmarkJava, that would blow the cost even when using very cheap models. It runs against vulnerable code snippets + code which uses it only. Below are some observations after using it myself with my own github repos. A DevEx part of the agent is important. The agent just creates clean PRs or adds a single clean commit to existing one. Basically, this AI Agent is just another tool here you need to know how to work with, not something you drop in and can totally forget about. The availability is the problem for all LLM providers seems to be. It is quite annoying to run the agent with an expensive(Anthropic, OpenAI) model, only to get an issue before I the agent finishes the whole set of vulnerabilities No amount of prompt or loop design will fix that. I think having proper infrastructure around agents is what's needed most right now.