这篇文章看完后，我真正注意到的点不是比哪个模型更厉害。作者拿AI跑了一圈实际的安全研究测试。Semgrep直接没找到。Strix接GLM 5.1跑了12小时，花了接近6000万tokens，还是没抓到关键漏洞。Cursor配GPT 5.5偶尔能碰上，但结果不稳定，有时候有有时候没有。真正每次都稳稳打中的，是本地模型加上他自己写的那个harness。这个做法确实有意思。他没有让agent一口把整个仓库全审完，而是把项目拆开，一个源码文件一个源码文件地喂给模型，然后把报告收回来汇总。PHPIPAM的LFI漏洞就这样每次都能跑出来。后来他又拿myVesta试了试，大概8小时左右就挖到了一个authenticated RCE。我看这才是本地AI做安全研究最有价值的地方。不是说本地模型现在已经全面超过Claude或者GPT了。而是它不怕把token烧光钱包，不用把代码丢给第三方，也没有云端模型动不动就拒答的限制。在安全场景里，能自己掌握主动权，这点真的重要。

原文 https://projectblack.io/blog/local-ai-for-cyber-security/…

#LocalAI #CyberSecurity #LLM #AIAgent #Pentest #CodeReview #SecurityResearch

Local AI for Penetration Testing & Research

Source: https://projectblack.io/blog/local-ai-for-cyber-security/ Model intelligence and tradecraft have progressed a lot in the year that’s passed since I last triedsomething similar.

There’s a lot of hype around the research Anthropic is publishing; however, cost and privacy are still problems. When there’s no guarantee that a thorough job was performed, this turns assurance work into something that feels more like gambling.

0:00

/0:03

Just one more run! “Make no mistakes, be thorough”

So I put together a small test.

I benchmarkedfour different approachesto identify a known-to-me vulnerability in order to evaluate how effectively each approach could find it.

The Benchmark Vulnerability - PHPIPAM Authenticated LFI

The benchmark vulnerability is a classic\.phpAuthenticated Local File Inclusion (LFI) issue.

Thecontroller nameis taken from user input and directly concatenated intorequire\_oncewith no sanitisation being performed.

If the API is enabled (disabled by default) and you have valid credentials to obtain an API token. It’s possible to include/execute any file on the file system accessible by the web server that ends with a\.phpextension.

Second request includes aphpinfo\(\);file I planted to demonstrate the issue.⚠️

This issue is being publicly disclosed without a patch as it’s relatively low risk and no response has been received from the PHPIPAM maintainers.

The vulnerability only affects instances where the API is enabled (not enabled by default). Default impact is also limited, there doesn’t appear to be a way to upload arbitrary\.phpfiles to the web server, nor are there any default files to include/execute that are interesting.

If you run PHPIPAM and wish to take action yourself immediately, I would recommend disabling the API in the interim. This is tracked asCVE-2026-12194.

The Results

**TL;DR:**Of the four approaches I tested, the most successful one made something clear: the harness/approach matters more than the model.

If you’re only interested in that approach, feel free to skip aheadhere.

| Approach                       | Did it find the vulnerability? |
|--------------------------------|--------------------------------|
| Semgrep                        | No                             |
| Cloud GLM 5.1 + Strix AI Agent | No *                           |
| Cloud SOTA + Code Review Skill | Sometimes                      |
| Local AI + Custom Harness      | Yes                            |

Semgrep

Semgrep, run with justsemgrep scan \-\-config auto, did not identify the vulnerability.

Of course, it’s possible to write custom rules to detect this specific pattern. However, this has always been one of the core challenges with traditional SAST tooling: you can only write rules for dangerous patterns that you already know exist.

GLM 5.1 + Strix Fully Agentic AI Pentest

Next, I looked atStrix: a fully agentic AI workflow for penetration testing.

Open-source AI hackers to find and fix your app’s vulnerabilities.

https://github.com/usestrix/strix#open-source-ai-hackers-to-find-and-fix-your-apps-vulnerabilitiesWith over 25,000 GitHub stars, I was keen to see what it could do/whether it could identify the bug.

💡

* NOTE: Their documentation calls for using GPT-5.4, Claude Sonnet 4.6, or Gemini 3 Pro.

I wasn’t keen on handing over an unsigned cheque so I opted for GLM 5.1 so I could observe token consumption using a cheaper model first.

GLM 5.1 benchmarks similarly to Strix’s recommended models but take this result with agrain of salt.

It was genuinely exciting to watch in action.

It cloned the repository, explored the codebase, and even installed the application by itself so it could dynamically validate potential findings.

~12 hours later, close to60 million tokenshad been used and areport\.mdfile was ready for me.

$Itdidn’tfind our LFI. Kind of reminded me of this tweet.

GPT-5 just refactored my entire codebase in one call. 25 tool invocations. 3,000+ new lines. 12 brand new files. It modularized everything. Broke up monoliths. Cleaned up spaghetti. None of it worked. But boy was it beautiful.pic.twitter.com/RCTGK1DE9H — vas (@vasuman)August 7, 2025

Actual spend with GLM 5.1 was ~$30 USD (as much as possible, requests were also routed to free inference providers).

If I had used Sonnet this would’ve costed somewhere between $180 USD and $300 USD. Opus would’ve been double that again.

I wasn’t particularly interested in spending that kind of money to try again with a more expensive model.

**Generated:** 2026-05-22 04:33:37 UTC

# Executive Summary

An external penetration test of phpIPAM (IP Address Management) version 1.8.1 identified multiple security weaknesses that, if exploited, could result in sensitive data exposure, stored cross-site scripting, server-side request forgery, and privilege escalation.

Overall risk posture: High.

Key findings include:
- Sensitive user data exposure via API (password hashes, tokens, 2FA secrets accessible to any authenticated user)
- Stored XSS via API (HTML escaping disabled, allowing persistent script injection)
- SSRF in vault certificate fetch (internal network scanning and cloud metadata access)
- Broken function-level authorization in API (non-admin users can create admin resources)
- Multiple CSRF vulnerabilities on state-changing admin endpoints
- Second-order SQL injection in custom field reorder functionality
- Weak CSRF token validation using loose comparison instead of timing-safe comparison

Some of these issues are by design per PHPIPAM’s threat model. Others looked like false positives/didn’t look very interesting/have already been fixed.

Cloud SOTA + Skill Based Code Review

Next I tried a different approach: AI skills.

For those unfamiliar with the concept, AI skills are effectively Markdown files that provide an agent with specific instructions, workflows, andexpertise.

I used a community contributed security-review skill as a starting point.

awesome-copilot/skills/security-review/SKILL.md at main · github/awesome-copilot Community-contributed instructions, agents, skills, and configurations to help you make the most of GitHub Copilot. - github/awesome-copilot GitHubgithub The changes I made include:

• Removing Dependency Audit, Secrets Scan, and Proposed Patch steps.
• Fanned out each vulnerability category into its own dedicated sub-agent.
• Expanded the Injection Flaws section with additional guidance around Local File Inclusion (LFI).

Ultimately, what I found was that the results werehighly inconsistent. The scan wouldsometimesfind the vulnerability and sometimes miss it entirely.

Claude Code + Opus 4.8

Trying to use the ‘Pro’ plan in Claude Code with this skill resulted in a pretty terrible result.

Reading through the thinking, it had decided to not spawn sub-agents to handle each vulnerability class.

Asking it to explicitly spawn subagents resulted in hitting my 5 hour session limit instantly...

#### Cursor + GPT 5.5 (Medium) - ~5-10 USD if Using Open Router

After a few more tries with various models here’s one run which found the issue using Cursor and GPT 5.5.

#### The Problem

What I realised pretty quickly was that, in a reasonably large codebase, the reviews weren’tthorough. Whether the issue was found often came down to which files the agent decided to read and what terms it chose to grep for.

The interesting part was that once pointed at the correct file, almost every model identified the vulnerability immediately.

Be More Thorough! Asking with a Skill

Trying this approach with a skill immediately resulted in refusals. Looking at the model’s reasoning, it would often conclude that the task was too large to complete in a single pass.

nooo.Pretty please?

trying again.Even if I got past the refusal, this approach would likely be prohibitively expensive.

But what if a local model is sufficient when given an individual source code file + context, rather than the much harder task of reviewing an entire codebase in one shot?

Local AI Model + a Custom Harness

Our next approach swaps the single big review for a small local harness. Rather than asking one agent to reason about the whole codebase at once, the harness walks a local model over the project one source file at a time, handing it a single file plus the context it needs on each pass.

For each source file:
|
+--> Local model reviews a single file (+ context)
|
+--> Writes a structured report
|
v
Collect all reports
|
v
???
|
v
Profit

High level diagram.

This approach found our benchmark vulnerabilityevery singlerun.

# Security Vulnerability Report: api/index.php

## High Risk: Path Traversal / Arbitrary File Inclusion via Controller Parameter

### Description

The API entry point constructs file inclusion paths using unsanitized user-supplied
input from the `controller` parameter. The value flows from `$_GET`, `$_POST`, 
JSON body, or XML body directly into `require_once()` without path validation, 
enabling directory traversal (`../`) to include arbitrary PHP files from the 
filesystem.

--- SNIP ---

For this code base, I’d estimate that roughly120 million tokenswere consumed reviewing around 800 source code files.

Limitations

• Token intensive - expensive if you can’t run the model locally.- Fortunately for us,Project Blackalready maintains a hashcat rig for penetration tests wherepassword crackingis required. As it turns out, this hardware is also sufficient to run Qwen 3.6 27b with ~170k context.
• False positives - lots of them.- As this approach is exclusively code review based, it can result in false positives, you could feed the output of this stage into more AI tooling to validate exploitability but that increases token burn even further.
• Lack of threat model/broader application context understanding.- From my testing against other known issues, it seemed to struggle with identifying more complexBroken Access Controlissues where there’s more nuanced differences in assumptions.

In any case,once could be a fluke. Lets see if it can find something I don’t already know about?

myVesta Authenticated RCE

Just as I was wrapping up with the benchmarks, I received this message from a friend who usesmyVesta- a web server control panel like cPanel.

good timing. target acquired.8 hours later, it found Authenticated RCE.

For context, web server control panels like cPanel are used to give randoms on the internet the ability to perform limited administrative tasks on shared web servers.An authenticated RCE vulnerability in such software means that you could sign up for an account with a hosting provider and potentially take over the server. This command executes as the higher privilegedadminuser on the web server rather than your own user.

As it turned out, a bit of legacy code in the FTP username deletion function allowed the Username parameter to be passed directly intoexec.

Creating a sample request.Replacing username in the request with a POC.Checking the server for the created file.Thanks to the myvesta team for responding quickly with this one! This is tracked asCVE-2026-12195.

web\edit\web\index.php fix · myvesta/vesta@95d7e43 myVESTA Control Panel. Contribute to myvesta/vesta development by creating an account on GitHub. GitHubmyvesta Patched here.

Can We Keep Going?

As it turns out, yes.

There aremorefindings currently cooking.

We’re holding the details to give the affected projects time to patch. They’ll surface in their own time but there’s more to come.

Conclusion

As local AI models keep improving, these capabilities are going to land in more hands, not fewer. I reckon that’s a good thing overall.

On our end, we’re experimenting to see how we can incorporate technologies like this into how we work which could potentially mean quicker and more thoroughpenetration tests for our customers. Lots of potential here, and we’ll keep tinkering to see where local AI earns its keep.