AI Security Leaderboard: benchmarking model robustness [P]

Reddit r/MachineLearning Papers
ai-security benchmarking jailbreak robustness leaderboard

Summary

The authors introduce an AI Security Leaderboard that benchmarks frontier model robustness by running models through 1500 automated jailbreak attempts, highlighting gaps in security across models and inviting community feedback on methodology and next steps.

We developed a leaderboard ranking frontier model security. There's no shortage of model capability rankings, but we didn't find anything comparable for model security. Yet security is becoming increasingly critical to deployment decisions: from the USG making developers pull models for cybersecurity jailbreaks to developers holding back on AI agent deployments due to risks of adversarial attacks. We developed an automated test suite that runs models through 1500 automatically generated jailbreak attempts and measures the number of universal jailbreaks: prompts that elicit compliant, detailed responses to >75% clearly harmful questions within a domain (like offensive cybersecurity). We find a big gap between the most and least robust models in our technical report. This is v1.0 and we'd really appreciate input from this subreddit on next steps, as well as feedback on the metholodogy. Areas we're considering include: 1. Adding open-weight models. How to fairly compare them to proprietary models as open-weight models naturally have a larger attack surface through perturbing weights (refusal abliteration, helpfulness fine-tuning)? 2. Adding new domains beyond CBRNE and cybersecurity, e.g. agent hijacking or harmful manipulation. 3. Increasing the realism of the domains, e.g. agentic cyber tasks rather than specific questions. 4. Stronger attacks: we aimed to keep these pretty basic for initial release but could add more powerful adaptive optimization attacks like boundary point jailbreaking. We'd also love to hear ways we could make this benchmark more useful in your work. If you're an adversarial robustness researcher, are there artifacts such as datasets or evaluation rubrics you'd like to re-use?
Original Article

Similar Articles

Gate AI: LLM Security Benchmark Evaluation Methodology and Results

arXiv cs.LG

This paper presents an evaluation methodology for LLM security detectors that addresses systematic weaknesses like per-dataset threshold tuning and undisclosed operating points. The framework uses cross-validation across 16 benchmarks, selects a single global operating point, and includes multiple diagnostics for generalization.

)

TLDR AI

Vercel releases DeepsecBench, a benchmark for evaluating AI models' ability to find cybersecurity vulnerabilities in application code, with findings that open-weight models are becoming more cost-effective for security scanning.

Evaluating potential cybersecurity threats of advanced AI

Google DeepMind Blog

DeepMind published a comprehensive framework for evaluating offensive cybersecurity capabilities of advanced AI models, analyzing over 12,000 real-world AI-powered cyberattack attempts across 20 countries and creating a 50-challenge benchmark covering the entire attack chain to help defenders prioritize security resources.