)

# DeepsecBench: evaluating model performance in finding cybersecurity vulnerabilities Source: [https://vercel.com/blog/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-vulnerabilities](https://vercel.com/blog/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-vulnerabilities) Last week, OpenAI evaluated two models on an exploit benchmark within an isolated sandbox\. Guardrails were reduced for testing, and the models found a vulnerability in their environment, accessed the internet, and reached Hugging Face's production database\. No human directed the action, but the breach is a clear example of how much more capable malicious attackers are when equipped with powerful AI models\. But defenders have the same tools, and a clear advantage: knowledge of their own codebase\. Hacks are initiated from the outside, so the single best defense is finding vulnerabilities from the inside before attackers do\. Today we're releasing[DeepsecBench](https://vercel.com/ai-gateway/leaderboards/deepsecbench), a benchmark that evaluates how well different models find cybersecurity vulnerabilities in application code\. For each model the report includes recall, precision, cost, and total time, and combines recall and precision into a single benchmark score\. Here is a sample of model performance from the leaderboard: We built[deepsec](https://deepsec.sh/)to make scanning as easy as possible\. Now you can use the benchmark report to build a security scanning program that fits your budget and the complexity of your codebase, choosing the right mix of models to run and how often to run them\. ## [Copy link to heading](https://vercel.com/blog/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-vulnerabilities#how-the-benchmark-works)How the benchmark works DeepsecBench runs on an open\-source codebase at a commit state just before a large number of vulnerabilities were fixed\. We selected 50 entry\-point files and built a golden set of 231 human\-judged findings\. Each model's score is a recall\-weighted F2 \(Score = 100 × 5PR/\(4P\+R\)\), weighting recall \(R\) twice as much as precision \(P\), because missed vulnerabilities will go unfixed, while false positives don't make your codebase less secure\. Findings beyond the golden set are classified by a judge model as real or false, and count for or against the model in the precision measure \(recall measures only the 231 known findings\)\. The benchmark is run three times and the data published in the report is the median of the three runs\. The construction of the benchmark stays secret\. We don't disclose the repository, the commit, the files, or the findings, so there is nothing for models to train against\. A model reciting memorized fixes would score near\-total recall\. Instead, the best run finds 30\.7%, and 20 of the 25 runs come in under 20%\. ## [Copy link to heading](https://vercel.com/blog/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-vulnerabilities#the-cost-of-capable-analysis-is-falling)The cost of capable analysis is falling When we[introduced`deepsec`](https://vercel.com/blog/introducing-deepsec-find-and-fix-vulnerabilities-in-your-code-base), thorough scanning on production codebases required using the most capable models, and they were expensive to run\. Frontier models from OpenAI and Anthropic still score highest, but open\-weight models and more efficient reasoning options are closing that gap, making comprehensive scanning far more cost\-efficient\. Score vs cost for each model run in the benchmark\. Higher and farther right is better\. Interact with and download these charts on the DeepsecBench page\. Today, higher price does not buy proportionally more\.[Kimi K3](https://vercel.com/ai-gateway/models/kimi-k3), from Moonshot AI, ranks eighth at a score of 17\.56 for $12\.38 on the high setting, half the top score for about a fifth of the cost\.[Grok 4\.5](https://vercel.com/ai-gateway/models/grok-4.5)set to high delivers near\-Kimi performance for less than half the cost, scoring 15\.58 for only $5\.60\.[GPT\-5\.6 Sol](https://vercel.com/ai-gateway/models/gpt-5.6-sol?__vercel_draft=1)set to medium offers the best score\-to\-cost balance of the top\-performing models, taking fifth place with a score of 25\.10 at a cost of $17\.95\. You can interact with and download these charts on the[DeepsecBench page](https://vercel.com/ai-gateway/leaderboards/deepsecbench)\. Anthropic's most capable model, Fable 5, is absent because it declines security work, including defensive tasks\. We will add security\-enabled versions to the benchmark when they are made available\. The costs reported in the benchmark cover 50 files\. A production codebase can run on the order of 100 times that, pricing a full pass at roughly $1,200 for a Kimi K3 sweep, or over $5,000 for the top\-scoring frontier model from OpenAI\. ## [Copy link to heading](https://vercel.com/blog/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-vulnerabilities#building-a-scanning-program-across-multiple-models)Building a scanning program across multiple models Because security scanning is recurring work, the question is which model to use for which task, and when\. For example, frontier models can be used for periodic deep audits, while cheaper models like Kimi K3 or Grok 4\.5 run scans at a higher cadence\. Turning GPT\-5\.6 Sol's reasoning down from xhigh to medium moves it from 35\.58 in 3 hours, 39 minutes to 25\.10 in just over 30 minutes, fast enough for reviewing new features before they are pushed to production\. A startup might scan every merge with Grok 4\.5 and save more expensive audits for milestone releases\. A large enterprise might run frontier audits on its critical services, the same model at lower reasoning on key pull requests, and a continuous Kimi\-class or Grok sweep across the rest of the codebase\. ## [Copy link to heading](https://vercel.com/blog/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-vulnerabilities#running-the-benchmark-on-one-endpoint)Running the benchmark on one endpoint Security scans are spiky workloads that consume a large number of tokens in a short time window, and that burst capacity is hard to buy from any single provider directly\. Every DeepsecBench run goes through[AI Gateway](https://vercel.com/ai-gateway)\. The gateway gives us a single endpoint to reach each model\. During runs, routing, retries, and failover happen automatically, without any per\-provider keys or rate limits to manage\. The same routing we use for the benchmark can run your security scanning program\. Pass`provider/model`to`deepsec`, and one`AI\_GATEWAY\_API\_KEY`in your environment covers every model on the board \(or on a linked Vercel project, access AI Gateway via OIDC\)\. ``` pnpm deepsec process --project-id my-app --agent pi --model xai/grok-4.5 ``` Use AI Gateway to balance model power and cost for different deepsec security scans\. ## [Copy link to heading](https://vercel.com/blog/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-vulnerabilities#find-your-vulnerabilities-first)Find your vulnerabilities first Attackers can use AI models, but they act from the outside\. Defenders can see their entire system: the source, the architecture, the history\. That visibility makes the same models more powerful in your hands than in theirs, because a model that can read the source finds what an attacker can only probe for\. The advantage is real, but it only counts if you use it first\. We will continue to update[DeepsecBench](https://vercel.com/ai-gateway/leaderboards/deepsecbench)with new models as they are released and measured\. [**Run a security scan with deepsec** Build a security scanning program with deepsec and AI Gateway\. Get started](https://github.com/vercel-labs/deepsec/blob/main/docs/getting-started.md)