Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Hugging Face Daily Papers Papers
security evaluation cost-aware offensive defensive ai-agents benchmark

Summary

This paper introduces a cost-aware evaluation framework for security AI agents, measuring not just success rate but also inference and tool costs. It finds that offensive CTF performance scales with compute, while defensive SOC tasks depend more on disciplined tool use than raw reasoning budget.

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.
Original Article
View Cached Full Text

Cached at: 07/21/26, 01:28 AM

Paper page - Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Source: https://huggingface.co/papers/2607.15263

Abstract

Security-agentevaluationscommonlymeasurepeakoffensivecapabilityundergenerousinferencebudgets,emphasizingvulnerabilitydiscovery,exploitdevelopment,penetrationtesting,andCTFcompletion.Suchmeasurementsareusefulbutincomplete:inoperationalsecurity,everyreasoningstep,toolcall,telemetryquery,andenrichmentrequestconsumesbudget.Weevaluatelanguage-modelsecurityagentsthroughthiscost-successlensonoffensiveCybenchchallengesanddefensiveSplunkBOTSv1investigationchallenges.Insteadofreportingonlybest-casesuccess,wecomparemodelsatfixedcostlevelsanddecomposeperformancebyinferencespendandtoolspend.Ourresultsshowdistinctscalingregimesforred-andblue-teamtasks.OffensiveCTFperformanceimproveswithadditionaltest-timecompute,andscaledopen-weightmodelscanapproachfrontierproprietarysystemswhileremainingcost-competitive.DefensiveSOCinvestigationdoesnotscaleinthesameway:successdependsmoreheavilyondisciplinedtooluse,telemetrynavigation,andselectiveenrichmentthanonrawreasoningbudgetalone.Wearguethatsecurity-agentbenchmarksshouldmeasureeconomicefficiencyandoperationalfitalongsidetasksuccess.Cost-aware,SOC-nativeevaluationsprovideaclearerpictureofwhichmodelsarepracticallyusefultodayandwheredefensiveagentsstillneedtoimprove.Wepresentaninteractivewebsitewithourresultshttps://evals.frontier.security.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.15263

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.15263 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.15263 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.15263 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

Hugging Face Daily Papers

This paper presents a practical evaluation protocol for assessing AI pentesting agents in realistic, complex targets rather than simplified benchmarks. It uses LLM-based semantic matching, bipartite resolution, and continuous ground-truth to score vulnerabilities discovered, and releases expert-annotated ground truth and code.

AI Agent Intelligence tool - Incident debugging, Cost spike detection

Reddit r/AI_Agents

Building a tool for AI Agent incident debugging and cost spike detection without additional instrumentation, covering issues like prompt injection, reasoning loops, and data exfiltration. Asking if customers in production environments see this as a pain point worth paying for.

)

TLDR AI

Vercel releases DeepsecBench, a benchmark for evaluating AI models' ability to find cybersecurity vulnerabilities in application code, with findings that open-weight models are becoming more cost-effective for security scanning.