Which LLM is actually best at pentesting? benchmark to find out

# Which AI is the best at penetration testing? — HunterBench Source: [https://www.hunterbench.com/](https://www.hunterbench.com/) Autonomous pentest benchmark We run frontier and open LLMs as autonomous pentesters on real infrastructure — their own recon, hunting and exploitation\. Leaderboard Two labs, two axes\.**Halcyon**\(coverage\) is a broad enterprise surface — a Node API, a Flask service, a legacy PHP app — seeded with the bread\-and\-butter of a real assessment: SQLi, IDOR, secrets and env dumps in the open, path traversal, broken auth\. How much of it does the model find?**Meridian**\(exploitation\) is a live fintech SaaS with real accounts and roles, where the bugs don't stand alone but*chain*: SSRF into an internal service, JWT alg\-confusion into account takeover, a pickle sink into RCE\. How far can it push a low\-priv foothold toward shell or admin? Each lab is out of 500;**Overall**is the two added, out of 1000\. We run each model**three times per lab**\(R1 · R2 · R3\) and take the**mean**— one run gets lucky or unlucky, three averages that out\.**Cost**and**Time**are per run, reported for value; they never touch the score\. Coverage matrix ### What each model finds Who finds what\. Every row is one planted vuln or chain step; every cell is the**share of a model's three runs**that reached it —**100**= all three, blank = never\. The easy stuff sits at the top; the deep bugs almost nobody cracks sink to the bottom\. Halcyon · coverage — % of each model's 3 runs that found the vuln Meridian · exploitation — which chain steps & gems each model reached Methodology ### How a model is scored A**run**is one full engagement: the model gets a target and a scope, does its own recon, then hunts and exploits — one autonomous pass\. Because a single pass is noisy, a result is the**mean of three runs**per lab \(R1 · R2 · R3\), not one\. Depth is never self\-graded: it's verified by secret markers the agent can only recover by completing each step of a chain\. Each lab is out of 500; the two add to an**Overall /1000**\. Modelthe brain3 RUNS · R1 · R2 · R3Run 1Run 2Run 3Match vsanswer keymeanof 3 runsLab score/ 500 **Each lab scores out of 500 — the mean of 3 runs \(R1 · R2 · R3\)\.**A model runs the target three times; we average the three because a single run gets lucky or unlucky\. A model's**Overall /1000**is its two lab scores added together — cost and time are reported alongside, never scored\. Coverage /500 500 · recall · precision The share of the planted surface a model finds, scaled by how clean its reports are — averaged over the 3 runs\. Exploitation /500 500 · \(90% chain\-weight \+ 10% logic\-bugs\) How much of the attack\-chain*weight*the model recovers — finishing the hard chain scores far more than the easy one — averaged over the 3 runs\. Business\-logic bugs are a capped bonus\. [Read the full methodology →](https://www.hunterbench.com/blog/how-we-test)·[How we score, out of 1000 →](https://www.hunterbench.com/blog/how-we-score-models) The engine ### How the hunt runs Every scan runs in a locked, disposable sandbox: all Linux capabilities dropped except`NET\_RAW`\(so its scanner works\), under a deny\-by\-default firewall whose only scan\-reachable host is the in\-scope target — plus the model's own API endpoint and DNS\. Inside, an autonomous agent maps the surface, then hunts: it runs its own tools, chases leads, and writes up each vulnerability it can prove\. Target\+ scope \(RoE\)ISOLATED SANDBOXdisposable · no privileges · deny\-by\-default egressReconmap services · portsHuntautonomous agent · exploitFindingsonly what it provesScoredvs answer key **The results cross the boundary, the methods don't\.**Commands, payloads, prompts and the agent's reasoning stay inside the sandbox — only proven findings are published\. [Read how the engine works →](https://www.hunterbench.com/blog/how-the-engine-works) The labs ### Real infra, not a CTF Halcyon and Meridianaren't toy targets with sixty planted flags\. They're full applications we built and run ourselves — a multi\-service enterprise back\-end, and a fintech SaaS with user accounts, roles and real business logic — seeded with the kind of vulnerabilities we've actually hit on engagements as pentesters: the misconfigurations, the injection into a forgotten endpoint, the auth you can forge, the SSRF that reaches something it shouldn't\. It reads like a real company, because that's what a model faces in the field\. We test the models on our own labs, and we keep the labs private\. That's it\. We also hold ourselves to it: we audited our own answer keys at the source\-code level and proved every lab is fully solvable\. The ground truth is audited, not assumed\. [Read about the targets & ground truth →](https://www.hunterbench.com/blog/the-target) Limitations ### What this does — and doesn't — measure A benchmark that hides its limits is the thing we're trying not to build\. So, plainly: #### Two labs isn't a population A model strong here may be weak on a different stack\. Read the scores as capability on controlled targets — not production readiness\. More labs are on the roadmap\. #### Lab is not the field Agents that ace labs still drop sharply on real, unstructured engagements\. This measures hunting and exploitation on known\-vulnerable targets, nothing more\. #### Small samples, real variance Results move run to run; we average the valid runs, report the reliable floor, discard and re\-run degraded ones, and mark a result provisional until it has enough valid runs\. Small gaps between models are likely noise, not signal\. [Full limitations & reproducibility →](https://www.hunterbench.com/blog/how-we-score-models)·[How we keep the harness honest →](https://www.hunterbench.com/blog/keeping-the-benchmark-honest)