StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents

Hugging Face Daily Papers Papers
benchmark autonomous-agent offensive-security operational-stealth llm-evaluation opsec cybersecurity

Summary

StealthBench measures operational stealth in autonomous offensive-security agents across six OPSEC dimensions, using a 3-model LLM judge panel. Results show no model exceeds 54% safe success rate, indicating systematic OPSEC failures.

Stealth, the discipline of achieving an objective without revealing your presence, capabilities, or collected intelligence, is what separates sophisticated operators from detectable ones. Elite security researchers and advanced persistent threats achieve their objectives unnoticed; autonomous agents increasingly inherit the same offensive tasks, but do they inherit the tradecraft? We introduce StealthBench,a benchmark that measures operational stealth in autonomous offensive-security agents across six operational security (OPSEC) dimensions. We extract 11 hand-verified OPSEC incidents from real bug-bounty and red-team trajectories, expanded into 14 dockerized task scenarios, where agents, despite finding real vulnerabilities, committed stealth failures inconsistent with standard operational tradecraft: embedding credentials in public uploads, deleting production resources to prove access, force-adding uninvolved users to demonstrate a race condition. We evaluate agent trajectories using a 3-model large language model (LLM) judge panel with majority-vote aggregation, measuring safe success rate (solved and stealthy), Stealth@Solve (tradecraft quality among successful solves), and reckless solve rate (solved but cover blown). Our results show that no model exceeds 54% safe success rate (the compound metric requiring both task completion and stealth), confirming that OPSEC failures are systematic across model families. We release StealthBench as a public benchmark to support both the development of stealth-aware agents and automated OPSEC monitoring for autonomous offensive-security deployments. The interactive leaderboard, evaluation harness, and dataset are available at https://stealthbench.com.
Original Article
View Cached Full Text

Cached at: 07/30/26, 05:46 AM

Paper page - StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents

Source: https://huggingface.co/papers/2607.26314

Abstract

Stealth,thedisciplineofachievinganobjectivewithoutrevealingyourpresence,capabilities,orcollectedintelligence,iswhatseparatessophisticatedoperatorsfromdetectableones.Elitesecurityresearchersandadvancedpersistentthreatsachievetheirobjectivesunnoticed;autonomousagentsincreasinglyinheritthesameoffensivetasks,butdotheyinheritthetradecraft?WeintroduceStealthBench,abenchmarkthatmeasuresoperationalstealthinautonomousoffensive-securityagentsacrosssixoperationalsecurity(OPSEC)dimensions.Weextract11hand-verifiedOPSECincidentsfromrealbug-bountyandred-teamtrajectories,expandedinto14dockerizedtaskscenarios,whereagents,despitefindingrealvulnerabilities,committedstealthfailuresinconsistentwithstandardoperationaltradecraft:embeddingcredentialsinpublicuploads,deletingproductionresourcestoproveaccess,force-addinguninvolveduserstodemonstratearacecondition.Weevaluateagenttrajectoriesusinga3-modellargelanguagemodel(LLM)judgepanelwithmajority-voteaggregation,measuringsafesuccessrate(solvedandstealthy),Stealth@Solve(tradecraftqualityamongsuccessfulsolves),andrecklesssolverate(solvedbutcoverblown).Ourresultsshowthatnomodelexceeds54%safesuccessrate(thecompoundmetricrequiringbothtaskcompletionandstealth),confirmingthatOPSECfailuresaresystematicacrossmodelfamilies.WereleaseStealthBenchasapublicbenchmarktosupportboththedevelopmentofstealth-awareagentsandautomatedOPSECmonitoringforautonomousoffensive-securitydeployments.Theinteractiveleaderboard,evaluationharness,anddatasetareavailableathttps://stealthbench.com.

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2607\.26314

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.26314 in a model README.md to link it from this page.

Datasets citing this paper1

#### 0xmoose/stealthbench Preview• Updatedabout 5 hours ago

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.26314 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents

arXiv cs.AI

POLAR-Bench is a diagnostic benchmark that evaluates the privacy-utility trade-off in LLM agents by testing their ability to follow privacy policies while being adversarially probed by third-party models. Results show frontier models protect over 99% of protected attributes but smaller open-weight models leak over half, highlighting gaps in intent-following.

What Benchmarks Don't Measure: The Case for Evaluating Abstention Competence in Autonomous Agents

arXiv cs.AI

This paper argues that current benchmarks for autonomous agents fail to evaluate whether an agent should have proceeded at all, introducing a 'compliance bias'. The authors propose a taxonomy of abstention-warranted scenarios and new evaluation protocols (Safety Rate, Usability Rate, Informed Refusal Rate) with preliminary results showing tunable safety–usability tradeoffs across model families.