Paper page - CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat

Source: https://huggingface.co/papers/2610.19005 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

CyberArena is a dynamic adversarial benchmark that evaluates AI agents through concurrent multi-round attack-and-defense combat in evolving environments, measuring capabilities via a dynamical scoring model.

AI agents are demonstrating increasingly strong offensive and defensive cybersecurity capabilities in the real world. Current benchmarks mostly evaluate agents in isolation against static targets. Real cyber operations, however, are dynamic and coupled: security postures constantly change as attackers and defenders adapt to each other in real time. Static benchmarks quickly saturate and miss an agent’s ability to attack and defend simultaneously under live pressure. To address this gap, we presentCyberArena, a dynamic, symmetricadversarial benchmarkbuilt aroundmulti-round attack-and-defensecombat. Inisolated sandboxes, two agents compete under identical rules, acting concurrently as both attackers and defenders over continuously changing services. This converts static targets into evolving, semi-open challenges, directly testing an agent’s speed and precision in discovering, exploiting, and patching vulnerabilities. To sustain the benchmark’s longevity, anautomated agentic pipelinecontinuously generates, reviews, and validates new environments. Furthermore, because pairwise contests are notoriously hard to quantify, we introduce adynamical modelthat maps match dynamics into unified,multidimensional scoresacross offensive, defensive, and domain-specific capabilities. Evaluating 12 frontierLLM agentsacross 40 challenges in 466 matches,CyberArenaseparates model capabilities where single-agent baselines hit a ceiling. The trajectories read like a fight rather than an exam: defenders patch one hole and miss the next and certify repairs on stale evidence, while capable agents think a move ahead, planting backdoors for later rounds and, in one case, writing their own packet sniffer to watch the opponent. Beyond a benchmark,CyberArenaserves as an adaptable, scalable experimental platform for future research oncybersecurity agents.

View arXiv pageView PDFProject pageGitHub7Add to collection

Get this paper in your agent:

hf papers read 2610\.19005

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.19005 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.19005 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.19005 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.