From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

Hugging Face Daily Papers Papers
ai-pentesting cyber-security evaluation-protocol benchmark llm-agents vulnerability-discovery

Summary

This paper presents a practical evaluation protocol for assessing AI pentesting agents in realistic, complex targets rather than simplified benchmarks. It uses LLM-based semantic matching, bipartite resolution, and continuous ground-truth to score vulnerabilities discovered, and releases expert-annotated ground truth and code.

AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perform best in real-world targets. Existing evaluation protocols assess and optimize for predefined goals such as capture-the-flag, remote code execution, exploit reproduction, or trajectory similarity, in simplified or narrow settings. These tools are valuable for measuring bounded capabilities, yet they do not adequately capture the complexity, open-ended exploration, and strategic decision-making required in realistic pentesting. In this paper, we present a practical evaluation protocol that shifts assessment from task completion to validated vulnerability discovery, allowing evaluation in sufficiently complex targets spanning multiple attack surfaces and vulnerability classes. The protocol combines structured ground-truth with LLM-based semantic matching to identify vulnerabilities, bipartite resolution to score findings under realistic ambiguity, continuous ground-truth maintenance, repeated and cumulative evaluation of stochastic agents, efficiency metrics, and reduced-suite selection for sustainable experimentation. This protocol extends the state of the art by enabling a more realistic, operationally informative comparison of AI pentesting agents. To enable reproducibility, we also release expert-annotated ground truth and code for the proposed evaluation protocol: https://github.com/ethiack/ethibench.
Original Article
View Cached Full Text

Cached at: 07/16/26, 01:43 PM

Paper page - From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

Source: https://huggingface.co/papers/2605.10834

Abstract

AIpentestingagentsareincreasinglycredibleasoffensivesecuritysystems,butcurrentbenchmarksstillprovidelimitedguidanceonwhichwillperformbestinreal-worldtargets.Existingevaluationprotocolsassessandoptimizeforpredefinedgoalssuchascapture-the-flag,remotecodeexecution,exploitreproduction,ortrajectorysimilarity,insimplifiedornarrowsettings.Thesetoolsarevaluableformeasuringboundedcapabilities,yettheydonotadequatelycapturethecomplexity,open-endedexploration,andstrategicdecision-makingrequiredinrealisticpentesting.Inthispaper,wepresentapracticalevaluationprotocolthatshiftsassessmentfromtaskcompletiontovalidatedvulnerabilitydiscovery,allowingevaluationinsufficientlycomplextargetsspanningmultipleattacksurfacesandvulnerabilityclasses.Theprotocolcombinesstructuredground-truthwithLLM-basedsemanticmatchingtoidentifyvulnerabilities,bipartiteresolutiontoscorefindingsunderrealisticambiguity,continuousground-truthmaintenance,repeatedandcumulativeevaluationofstochasticagents,efficiencymetrics,andreduced-suiteselectionforsustainableexperimentation.Thisprotocolextendsthestateoftheartbyenablingamorerealistic,operationallyinformativecomparisonofAIpentestingagents.Toenablereproducibility,wealsoreleaseexpert-annotatedgroundtruthandcodefortheproposedevaluationprotocol:https://github.com/ethiack/ethibench.

View arXiv pageView PDFGitHub64Add to collection

Get this paper in your agent:

hf papers read 2605\.10834

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.10834 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.10834 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.10834 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Free AI Agent Security Assessment

Reddit r/AI_Agents

Antitech is offering free early-access security assessments for AI agents, testing against attack vectors like prompt injection, tool abuse, and data leakage, providing a vulnerability report and discounts for participants.

An Empirical Study of Automating Agent Evaluation

arXiv cs.CL

This paper introduces EvalAgent, a system that automates the evaluation of AI agents by encoding domain-specific expertise, addressing the limitations of standard coding assistants in this task. It also presents AgentEvalBench, a benchmark for testing evaluation pipelines, and demonstrates significant improvements in evaluation reliability.