@pilvar222: We ran DeepSeek v4 Pro 0813 on our cybersecurity benchmark, it outperformed EVERY (!) other model at finding vulnerabil…

We ran DeepSeek v4 Pro 0813 on our cybersecurity benchmark, it outperformed EVERY (!) other model at finding vulnerabilities

• At pass@3, it rediscovered 87.5% of the benchmark CVEs. Far above Opus 5 and Qwen 3.8 at 81.3%

• The tradeoff is precision. Only 65.6% of vulnerabilities reported by DeepSeek were valid. Far below GPT-5.6-Sol’s 86.4%

• The model can also be unpredictable. It only finds an average of 58.3% of vulnerabilities per run. It is strongest when combining its runs’ findings.

@deepseek_ai is amazing. They just outperformed every other labs with an open model

1/3

An interesting pattern to observe is that DeepSeek is much more exploratory than other models.

While it is not consistent in finding vulnerabilities, it tends to go much deeper in understanding certain aspects of the features it is finding vulnerabilities in.

This behavior helps find deep vulnerabilities, but penalizes the model on being consistent on the easier ones.

The model focuses on depth over breadth.

2/3

The new DeepSeek model is a great recall model, especially for its price. But it has poor precision.

Its use can be very powerful for finding deep vulnerabilities, but other models are preferable for exhaustive security audits.

Try our harness: https://aikido.dev/code/code-audit

3/3

Grok 4.6 out as well!

nice catch, thank you!

Indeed, lots of politics that make this hard for them. Just look at Hugging Face, they had to use chines models for defense

Yeah good point, our strategy for this does not require too many tokens on this phase fortunately, also DeepSeek, even if it has a low precision, is not the model that reported the most findings!

So the full cost is the one reflected in the graph/table :)

Interesting theory, any other benchmarks that reflect that?

Indeed, harness matters quite a bit

Indeed, it’s quite unique!

Agree with your points, our benchmark was ran based on our final product, so it makes it more realistic (more details here: https://aikido.dev/blog/benchmarking-ai-models-known-cves…)

The harness is as important as the model :)

We have our ways, can’t share all :)

Hopefully soon!

Agreed, though FNs are much worse than FPs

No internet for agents, CVEs from last month (not in training data)

Fair point, although our goal is to reflect what our customers will see after running our product. We do run multiple passes and have ways to reduce/eliminate FPs, so cost and recall is what matters for us :)

The cost displayed is the avg per run for each model, you can also find a few more stats in the 3rd thread post image :)

indeed, very big potential tho

That’s what we do for our customer! :)

Yes you can???? That would make 100% sense, and when you pick the cheaper model, then you can usually also run a lot more runs for the same amount.

And as @pilvar222 said, DSV4 Pro prioritizes depth over breadth (similar to the approach dfs-large1 developed through RL.