@seclink: Share
Summary
This post shares a GitHub repository, featuring a curated set of security vulnerability samples and benchmarks, to evaluate the capabilities of static analysis tools and large models in vulnerability mining and secure code generation, covering multiple languages and the latest research findings.
View Cached Full Text
Cached at: 08/24/26, 05:55 PM
Share https://t.co/dEnO9bI6fB
XiaomingX/awesome-security-benchmark
Source: https://github.com/XiaomingX/awesome-security-benchmark
Awesome Security Benchmark Samples 🛡️
A curated list of security vulnerability samples / benchmarks (Security Vulnerability Benchmark & Samples) for evaluating the vulnerability discovery capabilities and secure code generation capabilities of Static Application Security Testing (SAST) tools and Large Language Models (LLMs).
Covers Java · Node.js/JavaScript · Golang · Next.js/React and real-world multi-language CVE datasets, including latest research benchmarks from 2025–2026 (ordered newest first).
This list is compiled and deduplicated from multiple comparative surveys, with all repository links verified for validity.
📈 Latest Research & Benchmarks (2025–2026)
2026
| Benchmark | Languages | Description | Link |
|---|---|---|---|
| SecCodeBench-V2 (Alibaba, arXiv:2602.15485) | Java · C · Python · Go · Node.js | 98 anonymized real-world vulnerability scenarios, 22 CWEs; Docker sandbox dynamic execution of PoC tests, functional and security dual-dimensional scoring, LLM-as-judge for semantic vulnerabilities (weak encryption/hardcoded credentials) | github.com/alibaba/sec-code-bench (https://github.com/alibaba/sec-code-bench) · arXiv (https://arxiv.org/abs/2602.15485) |
| SecLens (mattersec-labs) | General | Evaluates vulnerability detection capabilities of 12 frontier models by role (Engineer/CISO), proposes Role Divergence Index; no single model leads in all categories | github.com/mattersec-labs/seclens (https://github.com/mattersec-labs/seclens) |
| Veracode 2026 GenAI Code Security Report | Multi-language | Average security pass rate for 100+ models stagnates at 56%; reasoning models maintain a stable lead (56% vs 51%); Java is most dangerous (30%), Python is safest (63%) | veracode.com (https://www.veracode.com/news/llms-are-getting-smarter-but-not-safer-veracode-2026-genai-code-security-report-finds-ai-generated-code-security-has-stalled-at-56%25-pass-rate) |
2025
| Benchmark | Languages | Description | Link |
|---|---|---|---|
| SecureCode v2.0 (arXiv:2512.18542) | Go · Python · JavaScript · Java etc. | 1215 production-level secure code generation training entries, each with real-world security incident CVE traceability and vuln/safe dual implementations, covering OWASP Top 10:2025 | arXiv (https://arxiv.org/abs/2512.18542) |
| PATCHEVAL (arXiv:2511.11019) | Go · JavaScript · Python | 1000 real-world CVEs (2015–2025), 65 CWEs, automated vulnerability remediation benchmark; 230 entries include sandbox runtime environments for simultaneous verification of security and functionality | arXiv (https://arxiv.org/abs/2511.11019) |
| SafeGenBench | 12 languages (incl. JS/Go) | 558 prompts, 44 vulnerability classes (OWASP/CWE taxonomy), dual judge system with Semgrep static analysis + LLM (DeepSeek-R1) | SafeGenBench (https://www.emergentmind.com/topics/safegenbench) |
| PrimeVul (ICSE 2025, arXiv:2403.18624) | C/C++ | ~7k vulnerable functions + ~229k benign functions (97% benign, close to real-world distribution), 140+ CWEs; high-precision labeling + time-split to prevent contamination + VD-Score; evaluations show mainstream LLMs approach random performance under strict settings | github.com/DLVulDet/PrimeVul (https://github.com/DLVulDet/PrimeVul) |
2024 and Earlier (Foundational LLM Evaluation Methodologies)
| Benchmark | Languages | Description | Link |
|---|---|---|---|
| CyberSecEval 3 (Meta, arXiv:2408.01605) | General | LLM cybersecurity risk/capability evaluation suite (red teaming, exploitation, malicious code generation, prompt injection); conclusion: no evidence that LLMs surpass traditional tools in real-scale vulnerability discovery | github.com/meta-llama/PurpleLlama (https://github.com/meta-llama/PurpleLlama) |
| SVEN (ICSE 2024) | C/Python | Function-level vuln/safe minimal-difference pairs annotated with CodeQL, simultaneously evaluating security and functional correctness | github.com/eth-sri/sven (https://github.com/eth-sri/sven) |
| LLMSecEval (MSR 2023) | Python | 150 natural language requirements → secure code generation, dual scoring with security and functional unit tests | github.com/tuhh-softsec/LLMSecEval (https://github.com/tuhh-softsec/LLMSecEval) |
☕ Java (De facto Standard for SAST / Vulnerability Detection)
| Sample Set | Scale / Features | Link |
|---|---|---|
| OWASP Benchmark v1.2 | De facto standard for SAST. 2740 compilable cases, 11 categories (sqli/xss/cmdi/pathtraver/weakrand/crypto…), vuln:safe≈1:1, includes expectedresults for TPR/FPR curve calculation | github.com/OWASP-Benchmark/BenchmarkJava (https://github.com/OWASP-Benchmark/BenchmarkJava) |
| NIST Juliet Test Suite | 112 CWEs, 40k+ cases, bad()/good() paired, servlet/standalone dual forms, dozens of data-flow/control-flow variants per sink | UnitTestBot 112-CWE Complete Mirror (https://github.com/UnitTestBot/juliet-java-test-suite) · find-sec-bugs 10-CWE Lightweight Mirror (https://github.com/find-sec-bugs/juliet-test-suite) · NIST Official Zip (https://samate.nist.gov/SRD/testsuite.php) |
| securibench-micro | 125 taint analysis micro-benchmarks, 12 taint challenge categories (aliasing/arrays/context/inter-procedural/sanitizers…), Javadoc @should_taint expected labels, remains the litmus test for taint engines | github.com/too4words/securibench-micro (https://github.com/too4words/securibench-micro) |
| Vul4J (ISSTA 2024) | 262 reproducible real-world Java CVEs (including famous projects like fastjson), trigger tests that fail before patch / pass after patch, ground truth executable verification, supports detection/localization/remediation tasks | github.com/tuhh-softsec/Vul4J (https://github.com/tuhh-secure/Vul4J) |
| java-sec-code | Spring Boot common vulnerabilities vs. secure code examples (35+ categories), educational benchmark | github.com/JoyChou93/java-sec-code (https://github.com/JoyChou93/java-sec-code) |
| WebGoat | OWASP classic educational vulnerable web application | github.com/WebGoat/WebGoat (https://github.com/WebGoat/WebGoat) |
| find-sec-bugs | SpotBugs security plugin, 140+ detector rules, real-world Java sink rule library (command injection/deserialization/SSRF/XSS/crypto…) | github.com/find-sec-bugs/find-sec-bugs (https://github.com/find-sec-bugs/find-sec-bugs) |
| ysoserial | Java deserialization gadget chain exploit payload generator | github.com/frohoff/ysoserial (https://github.com/frohoff/ysoserial) |
| JYso (qi4L) | Modern gadget / JNDI exploitation framework, including fastjson exploit chains for various versions | github.com/qi4L/JYso (https://github.com/qi4L/JYso) |
🟢 Node.js / JavaScript
| Sample Set | Scale / Features | Link |
|---|---|---|
| SecBench.js (AttackStrings) | Node.js-specific vulnerability benchmark. 600 public vulnerabilities, 5 categories: Prototype Pollution 192 / Path Traversal 169 / Command Injection 101 / ReDoS 98 / Arbitrary Code Injection 40; each entry includes Jest tests + CVE/fix commit/sink location metadata | github.com/MartinDetloff/AttackStrings (https://github.com/MartinDetloff/AttackStrings) |
| OpenSSF CVE Benchmark | 200+ reproducible code and metadata for real-world JS/TS CVEs, with SAST tool (ESLint/NodeJSScan/CodeQL) evaluation report toolchain | github.com/ossf-cve-benchmark/ossf-cve-benchmark (https://github.com/ossf-cve-benchmark/ossf-cve-benchmark) |
| OWASP Juice Shop | Most modern “insecure” web application (Node.js/Express), for education and CTF | github.com/juice-shop/juice-shop (https://github.com/juice-shop/juice-shop) |
| DVNA (Damn Vulnerable NodeJS Application) | Intentionally vulnerable Node.js application (★779) | github.com/appsecco/dvna (https://github.com/appsecco/dvna) |
| nodejs-goof | Intentionally vulnerable Node.js todo app by Snyk | github.com/snyk-labs/nodejs-goof (https://github.com/snyk-labs/nodejs-goof) |
🐹 Golang
| Sample Set | Scale / Features | Link |
|---|---|---|
| go-test-bench (Contrast Security) | Intentionally vulnerable Go web applications, covering OWASP Top 10, multiple framework variants (net/http · gin · chi · go-swagger · httprouter) | github.com/Contrast-Security-OSS/go-test-bench (https://github.com/Contrast-Security-OSS/go-test-bench) |
| GoVulnDB | Official Go vulnerability database (golang.org/x/vulndb mirror), CVE + affected/fixed versions, for use with govulncheck | github.com/golang/vulndb (https://github.com/golang/vulndb) |
▲ Next.js / React
| Sample Set | Scale / Features | Link |
|---|---|---|
| React2Shell Lab (CVE-2025-55182) | Intentionally vulnerable Next.js application based on App Router + Server Actions, demonstrates insecure deserialization in RSC “Flight” protocol → pre-auth RCE (affects React 19.0.0–19.2.0 / Next.js 14.3+, for isolated environment research only) | github.com/subzer0x0/React2Shell (https://github.com/subzer0x0/React2Shell) |
| ai-code-security-golden (HuggingFace) | 118 hand-labeled code samples (65 are Next.js App Router/API snippets), 34 vulnerability classes, includes “safe-twin” security pairs for simultaneous recall and precision testing | huggingface.co/datasets/axyr/ai-code-security-golden (https://huggingface.co/datasets/axyr/ai-code-security-golden) |
| oss-oopssec-store | First React + Next.js CTF e-commerce store (XSS/CSRF/IDOR/JWT/Path Traversal/SQLi), CI regression testing to prevent accidental vulnerability fixes | Docker Hub (https://hub.docker.com/r/leogra/oss-oopssec-store) |
🌐 Multi-language Real-World CVE Datasets
| Sample Set | Description | Link |
|---|---|---|
| CVEfixes (MSR 2021) | All-language CVE→remediation commit relationship database (SQLite published via Zenodo), real-world historical vulnerability distribution, supports localization/patch evaluation | github.com/secureIT-project/CVEfixes (https://github.com/secureIT-project/CVEfixes) |
| VulnGym (Tencent) | Project-level white-box benchmark, entry_point→trace→critical_operation three-part structure, 60 L2 categories (12 business logic + 1 sub-category), multi-language (primarily JS/TS) | github.com/Tencent/VulnGym (https://github.com/Tencent/VulnGym) |
📌 Selection Guide
- Evaluating SAST tool speed/accuracy → OWASP Benchmark + Juliet + find-sec-bugs
- Evaluating taint analysis engines → securibench-micro
- Executable evaluation for real-world Java vulnerabilities → Vul4J
- Vulnerability detection in Node.js ecosystem → SecBench.js + OpenSSF CVE Benchmark
- Security testing for Go services → go-test-bench + GoVulnDB
- Security evaluation for Next.js/App Router → ai-code-security-golden + React2Shell Lab
- LLM vulnerability discovery/secure generation evaluation → SecCodeBench-V2 (2026), PrimeVul, SVEN, LLMSecEval, CyberSecEval 3
📚 References
This list is compiled and deduplicated from multiple comparative surveys of open-source benchmarks for static analysis and LLM vulnerability detection, covering three generations of benchmark evolution (educational label period → real CVE period → LLM paired samples / executable ground truth period).
Contributions: Welcome to supplement authoritative sample sets via Issue / PR (please provide verifiable GitHub links).
License
Similar Articles
@seclink: It seems we can create benchmarks for more languages. Overnight, everyone is rushing to evaluate the cybersecurity capabilities of large language models, to see if this is real emergence or just hype. Next, we need to continue adding high-quality samples for next.js, rust, golang, c/c++, python...
This article discusses creating benchmarks for more programming languages to evaluate the cybersecurity capabilities of large language models, and announces the release of JSEF v1.3.0, a Java security teaching framework and benchmark for measuring the vulnerability detection capabilities of SAST tools and LLMs.
@AdamShao: Officially open-sourcing my vulnerability discovery tool: http://flounders.xyz This is an AI Agent-based fully automated vulnerability discovery workflow. You just tell the AI which project's vulnerabilities you want to find, and it will automatically download code and documentation, deeply audit the code, discover suspicious vulnerabilities, automatically verify them locally and online…
Flounder is an open-source AI agent-based tool that automates vulnerability discovery in codebases. Users describe the target and the tool autonomously downloads code, conducts deep code audits, tests vulnerabilities locally and online, and generates reports.
@seclink: Google believes that relying solely on large models themselves for vulnerability mining is not enough; the key is to combine harness ...
Google points out that relying solely on large language models in vulnerability mining is not enough; the key is to combine an agent-based vulnerability discovery framework. For example, its team used this tool to discover multiple critical vulnerabilities in a short time.
@GitHub_Daily: When doing security testing, dozens of vulnerability types like SQL injection, XSS, and SSRF, manually checking each one is very time-consuming. Deep Eye uses AI-driven penetration testing, can integrate with more than 10 model services, and automatically generates test plans. Covers more than 45 vulnerability detection types, first identifying what technology and protection the target site uses...
Deep Eye is an AI-driven penetration testing tool that can integrate with multiple model services, automatically generate test plans, cover over 45 vulnerability detection types, and support compliance report generation.
@seclink: Actually, Alibaba internally had over 10,000 deduplicated, manually annotated high-quality third-party component vulnerability datasets in 2020-2021, including both public 1-day and undisclosed 0-day vulnerabilities. All contain source code + vulnerability sink points (…
Alibaba internally possessed over 10,000 high-quality vulnerability datasets in 2020-2021, which can be used for AI model training to enhance cybersecurity capabilities. The article discusses its potential value and compares it with other datasets.