FUSE: An Evaluating Framework for Dangerous Capabilities of LLMs

# FUSE: An Evaluating Framework for Dangerous Capabilities of LLMs
Source: [https://arxiv.org/html/2609.02168](https://arxiv.org/html/2609.02168)
Ru Zhang†Xiao ChenXinbo LiuJiaxuan LinJia HuangJianyi LiuZhen YangAffiliation:School of Cyberspace Security, Beijing University of Posts and Telecommunications, ChinaAffiliation:†Corresponding author: zhangru@bupt\.edu\.cn

###### Abstract

Fragmented safety evaluation undermines the governance of dangerous AI capabilities\. We present a modular framework that evaluates each model through three orthogonal pipelines—Knowledge \(KK\), Defense \(DD\), and Harm \(HH\)—under a unified protocol, aggregating results into a standardized dangerous\-capability profileϕ\\phi\. Pluggable modules supply scenario seeds, knowledge banks, hazard queries, and judge rubrics, while the core evaluation engine remains unchanged across domains; the CB evaluation is complemented by a cyber pilot demonstrating protocol transfer\.

Instantiating the framework with a chemical\-biological \(CB\) module, we evaluate 12 commercial LLMs from four families\. Our first contribution is a horizontal comparison of dangerous capability across models and model families: the three dimensions expose sharply divergent profiles—models with comparable knowledge differ in refusal resilience, and strong defenders do not generate less harmful content when they do comply—while family\-level patterns further separate Claude, DeepSeek, and GPT models\. The second is a temporal analysis of capability evolution: trackingKK,DD, andHHagainst model release dates reveals that dangerous capability has not monotonically declined; newer models deepen knowledge while only partially improving defense, showing that scaling and alignment progress do not uniformly translate into safety\. Reliability is established via cross\-judge consistency \(bootstrapρ\>0\.79\\rho\>0\.79, 4 of 5 judges\) and pipeline orthogonality \(KK–DD–HHinter\-correlationsρ∈\[0\.32,0\.52\]\\rho\\in\[0\.32,0\.52\]\)\.

## 1Introduction

Regulators, platform operators, and enterprise security teams face a fundamental question about large language models\[[1](https://arxiv.org/html/2609.02168#bib.bib38),[3](https://arxiv.org/html/2609.02168#bib.bib5)\]:*which models pose dangerous capabilities, and how dangerous are they?*Existing safety evaluations cannot answer this question\. Knowledge benchmarks \(GPQA\[[25](https://arxiv.org/html/2609.02168#bib.bib40)\], WMDP\[[15](https://arxiv.org/html/2609.02168#bib.bib7)\]\) measure what a model knows but not what it will do under pressure\. Defense benchmarks \(R\-Judge\[[35](https://arxiv.org/html/2609.02168#bib.bib10)\], Agent\-SafetyBench\[[36](https://arxiv.org/html/2609.02168#bib.bib11)\]\) measure refusal behavior but lack standardized outputs for cross\-domain comparison\. Harm\-generation assessments evaluate outputs without considering how those outputs were elicited\. Each benchmark runs its own protocol, reports its own format, and connects to nothing downstream—no portable risk profile for a regulator, no standardized vector for cross\-model comparison\.

The root problem is fragmentation, not the absence of measurement\. A regulator cannot aggregate “this model knows hazardous biology” from one benchmark, “this model complies under pressure” from another, and “this model produces actionable harm protocols” from a third—because these results are not comparable\. They use different models, different protocols, different score scales, and different sample populations\. The fragmentation is structural: no amount of post\-hoc normalization can produce a comparable risk profile from benchmarks that were never designed to be compared\. Figure[1](https://arxiv.org/html/2609.02168#S1.F1)contrasts the two worlds\.

![Refer to caption](https://arxiv.org/html/2609.02168v1/fig_intro_problem_solution.png)Figure 1:The problem and our solution\. Left: existing benchmarks measure the LLMs with incompatible protocols, scales, and other single dimension—the outputs cannot answer which model is more dangerous or how danger has evolved\. Right: our framework evaluates along three orthogonal dimensions \(KK,DD,HH\) and aggregates the results into the dangerous capability profileϕ\\phi, enabling direct cross\-model, cross\-domain, and cross\-time comparison\.Our answer: a modular framework, not another benchmark\. We present a modular evaluation infrastructure for assessing dangerous capabilities of LLMs\. The framework accepts pluggable danger\-domain modules and evaluates each model along three orthogonal dimensions under a unified protocol:

- •Knowledge \(KK\) — system information\-leakage risk: the hazardous\-domain proficiency a model can disclose, measured via IRT\-estimated ability on MCQ items\.KKbounds what an attacker can extract from the model through querying alone, before any behavioral bypass is attempted\.
- •Defense \(DD\) — system access\-control robustness: how robustly the model enforces its behavioral boundaries under adversarial induction, measured via path\-dependent interaction in which a Red\-Team Agent applies escalating elicitation strategies adapted to the model’s prior responses\.DDquantifies the boundary itself, not just its first\-contact appearance\.
- •Harm \(HH\) — system output content\-safety risk: the actionability of hazardous content the model produces when its access controls fail, measured via rubric\-scored open\-ended generation\.HHquantifies the blast radius of an access\-control breach\.

These three dimensions answer independent questions about the nature of risk, and our CB evaluation confirms they are not redundant: pairwise correlations across 12 commercial LLMs are low \(ρ∈\[0\.32,0\.52\]\\rho\\in\[0\.32,0\.52\]\)\. No single dimension suffices for a complete risk profile\.

The framework aggregates these dimensions—together with two derived indicators capturing domain asymmetry \(Δ\\Delta\) and silent refusal \(E​RER\)—into a single dangerous\-capability profileϕ\\phi: a standardized seven\-dimensional vector that summarizes what a model knows, how it behaves under pressure, and what it produces when it complies\. Because every dimension is measured under one protocol on one model population,ϕ\\phivectors are directly comparable across models and across domains—the property that existing benchmarks cannot provide\.

We instantiate the framework with a chemical\-biological \(CB\) module and evaluate 12 commercial LLMs spanning four model families and a three\-year release window \(2023–2026\)\. The evaluation yields two headline conclusions\. First,*cross\-model comparison*: models and model families exhibit sharply distinct capability structures, differences visible only when all dimensions are reported side by side\. Second,*temporal evolution*: knowledge compounds across generations within every family, defense diverges by family, and harm remains stubbornly inelastic—no vendor has produced a generation whose compliant outputs are meaningfully less actionable than its predecessors\. We further validate the framework’s scoring foundation through cross\-judge consistency \(bootstrapρ\>0\.79\\rho\>0\.79for 4 of 5 alternative judges\), pipeline orthogonality, and IRT measurement precision \(Section[6](https://arxiv.org/html/2609.02168#S6)\)\.

In summary, we claim the following contributions:

- •We design a modular evaluation infrastructure for dangerous capability assessment—four\-module architecture \(Pluggable Module, Task Orchestration, Test Environment, Judgement\), unified evaluation protocol, and switchable Content\-Only/Tool\-Enhanced execution modes\.
- •We propose a formal pluggable module interface that accepts danger\-domain\-specific scenario seeds, knowledge banks, harm queries, and rubrics while reusing the core evaluation pipeline; instantiated and validated with a CB module on 12 commercial LLMs\.
- •We define the dangerous\-capability profileϕ\\phias a standardized seven\-dimensional aggregation \(KK,DD,HH,D​LDL,B​RBR,Δ\\Delta,E​RER\) and reportϕ\\phifor 12 models, enabling direct cross\-model and cross\-domain comparison that existing benchmarks cannot support\.
- •We provide the first systematic analysis of how dangerous capability evolves with model release time across three generations, revealing compounding knowledge, diverging defense, and inelastic harm\.

## 2Background and Related Work

The evaluation of LLM safety has evolved rapidly but remains fragmented\[[16](https://arxiv.org/html/2609.02168#bib.bib2),[31](https://arxiv.org/html/2609.02168#bib.bib4),[33](https://arxiv.org/html/2609.02168#bib.bib3)\]\. We categorize existing work into four tracks and identify the common gap: the absence of a unified infrastructure that assesses dangerous capability consistently across orthogonal dimensions\.

### 2\.1Knowledge Benchmarks

MMLU\[[10](https://arxiv.org/html/2609.02168#bib.bib8)\]established the MCQ paradigm for large\-scale knowledge evaluation; GPQA\[[25](https://arxiv.org/html/2609.02168#bib.bib40)\]and WMDP\[[15](https://arxiv.org/html/2609.02168#bib.bib7)\]extend it to hazardous scientific domains, measuring*whether a model knows*dangerous information\. These benchmarks are fundamentally single\-turn and cannot assess how a model behaves under adversarial pressure\.

### 2\.2Defense and Safety Benchmarks

R\-Judge\[[35](https://arxiv.org/html/2609.02168#bib.bib10)\]tests risk identification in multi\-turn agentic interactions; Agent\-SafetyBench and OpenAgentSafety\[[32](https://arxiv.org/html/2609.02168#bib.bib12)\]evaluate agents with real\-world tool access, finding that individually safe steps compound into unsafe outcomes\. Domain\-specific evaluations address extreme risks: RAND’s CBRN studies\[[21](https://arxiv.org/html/2609.02168#bib.bib35)\]show LLMs can lower the barrier to biological attack planning\[[28](https://arxiv.org/html/2609.02168#bib.bib36),[29](https://arxiv.org/html/2609.02168#bib.bib34)\]; Do\-Not\-Answer\[[34](https://arxiv.org/html/2609.02168#bib.bib13)\]evaluates refusal safeguards\. Each uses its own protocol and output format, making results incomparable across benchmarks—a structural weakness our framework removes\.

### 2\.3Adaptive and Multi\-Turn Jailbreaking

Gradient\-based attacks\[[40](https://arxiv.org/html/2609.02168#bib.bib19)\], automated prompt generation\[[18](https://arxiv.org/html/2609.02168#bib.bib17)\], multi\-turn elicitation\[[14](https://arxiv.org/html/2609.02168#bib.bib20)\], and model\-based red teaming\[[23](https://arxiv.org/html/2609.02168#bib.bib22),[8](https://arxiv.org/html/2609.02168#bib.bib23)\]show that adversarial pressure bypasses safety guardrails; PAIR\[[5](https://arxiv.org/html/2609.02168#bib.bib18)\]and TAP elicit harmful responses through iterative attacker\-model queries\. Prior work treats jailbreaking as an isolated attack vector; our framework embeds adaptive multi\-round evaluation as a built\-in design feature of the defense dimension, validated empirically through the initial\-defense\-to\-erosion correlation \(Section[5\.3](https://arxiv.org/html/2609.02168#S5.SS3)\)\.

### 2\.4LLM\-as\-Judge Reliability

Our framework scores with LLM\-as\-Judge, whose reliability has been extensively studied\. Zheng et al\.\[[38](https://arxiv.org/html/2609.02168#bib.bib27)\]show strong judges reach over 80% human agreement but exhibit position and verbosity biases, controlled here through multiple judge families and temperature 0\.0\. Multi\-agent ensembles\[[13](https://arxiv.org/html/2609.02168#bib.bib21)\]mitigate single\-judge variance; Verga et al\.\[[30](https://arxiv.org/html/2609.02168#bib.bib28)\]show agreement varies by task type—objective scoring is more consistent than subjective quality assessment, exactly the pattern we observe across defense and harm scoring \(Section[6](https://arxiv.org/html/2609.02168#S6)\)\. These findings motivate our cross\-judge validation design\.

### 2\.5Agent and Tool\-Augmented Safety

AgentBench\[[17](https://arxiv.org/html/2609.02168#bib.bib15)\]and ToolEmu\[[27](https://arxiv.org/html/2609.02168#bib.bib14)\]show tool access alters safety behavior and that sandboxed evaluation is feasible\. Our Tool\-Enhanced mode \(Section[3\.4](https://arxiv.org/html/2609.02168#S3.SS4)\) builds on this line, extending the content\-only mode to tool\-augmented scenarios within an isolated sandbox\.

### 2\.6Summary

Table 1:Comparison of existing benchmarks with our framework\.Even within a single threat category, recent advances produce incompatible protocols\[[19](https://arxiv.org/html/2609.02168#bib.bib1)\]\. The fundamental gap is not the absence of evaluation but the absence of a unified infrastructure that hosts multiple danger\-domain evaluations under one protocol and produces comparable dangerous capability profiles\. A researcher today must configure GPQA, R\-Judge, and a custom harm test separately; our framework reduces this to instantiating one Pluggable Module\. The following sections detail the framework: Section[3](https://arxiv.org/html/2609.02168#S3)defines the four\-module architecture, Section[4](https://arxiv.org/html/2609.02168#S4)shows the evaluation pipelines and the dangerous capability profileϕ\\phi, Section[5](https://arxiv.org/html/2609.02168#S5)is the CB instantiation and its two headline findings, and Section[6](https://arxiv.org/html/2609.02168#S6)the judge\-consistency analysis\.

## 3Framework Design

Self\-assembly of existing benchmarks cannot achieve three properties that our framework provides by design: \(1\)Cross\-pipeline comparability— the same model evaluated onKK,DD, andHHunder our unified protocol yields scores in the same coordinate system, enabling cross\-dimensional diagnosis; \(2\)Standardized output— the dangerous\-capability profileϕ\\phiis distilled from the evaluation’s stable dimensions, not a post\-hoc integration; and \(3\)Amortized cost— by design, adding a new danger domain requires only a new Pluggable Module adhering to a fixed interface; the cyber pilot \(Section[5\.6](https://arxiv.org/html/2609.02168#S5.SS6)\) provides initial evidence of this transfer\.

### 3\.1Architecture Overview

The framework is organized as a four\-module pipeline: a Pluggable Module supplies domain assets, a Task Orchestration module manages the evaluation lifecycle, a Test Environment hosts the interaction between the Red\-Team Agent and the Target model, and a Judgment module scores the recorded interactions and produces the dangerous\-capability profileϕ\\phi\. Figure[2](https://arxiv.org/html/2609.02168#S3.F2)provides the complete system diagram\. Each module has a well\-defined interface and can be independently configured or replaced\.

![Refer to caption](https://arxiv.org/html/2609.02168v1/fig_architecture.png)Figure 2:Framework architecture overview: a four\-module system \(Pluggable Module, Task Orchestration, Test Environment, Judgment\) that evaluates each model through three parallel assessment pipelines \(KK,DD,HH\) and aggregates them into a dangerous capability profileϕ\\phi\.
### 3\.2Pluggable Module

The framework’s extensibility is rooted in a formal module interface: every danger domain is encapsulated in a single self\-contained module specification, termed a Danger\-Domain Module, which bundles exactly four evaluation\-specific assets\.

The first asset is the*scenario\-seed set*, a collection of domain\-specific scenarios used by the defense dimension \(DD\)\. Each scenario seed encodes a legitimate work context, a harm goal embedded within it, and an escalation ladder that drives the progressive multi\-round elicitation\. The second one is the*knowledge bank*, a set of MCQ items used by the knowledge dimension \(KK\); every item carries its correct answer and a difficulty label\. The third is the*harm\-queries set*, a collection of open\-ended prompts used by the harm dimension \(HH\)\. The last asset is the*judge rubric*, which defines the scoring dimensions and their ranges\.

The core framework is shared across all modules and remains untouched when a new domain is added; the cyber pilot \(Section[5\.6](https://arxiv.org/html/2609.02168#S5.SS6)\) exercises this transfer in practice\. To adapt the evaluation to a new danger domain, one supplies a new module specification containing these four assets; the framework handles the remainder, from orchestrating interactions to producing the dangerous capability profileϕ\\phi\. This design ensures that adding a domain is a data\-preparation task rather than an engineering task\.

### 3\.3Task Orchestration

The Task Orchestration module manages the complete lifecycle of an evaluation run\. It first loads from the Pluggable Module the three components that correspond to the three assessment dimensions: the*knowledge bank*\(KK\),*scenario seeds*\(DD\), and the*harm queries*\(HH\)\. The evaluator then selects which dimension\(s\) to run for this evaluation —KK,DD,HH, or any combination thereof\.

For the defense dimension, the orchestrator maintains an internal*scenario database*that stores the instantiated induction scenarios together with their predefined attack personas; each scenario seed encapsulates a domain\-specific context, an attack persona, and a progressive conversation template\. Given the evaluation task, the orchestrator selects one of two execution modes —*content\-only*or*tool\-enhanced*— and activates the Test Environment together with the Red\-Team Agent\. The orchestrator also manages parallel evaluation across multiple Red\-Team–Target pairs when the workload demands it\.

### 3\.4Test Environment

The Test Environment hosts the interaction between the Red\-Team Agent \(the adaptive adversarial elicitor\) and the Target model \(the LLM under evaluation\)\.

Content\-Only Mode\.Models interact through natural\-language dialogue\. The Red\-Team Agent adapts its elicitation strategy across rounds based on the Target model’s prior responses\.

Tool\-Enhanced Mode\.The Target model gains real tool access \(Bash, Python, file I/O, web search\) inside an isolated sandbox\. Tool invocation follows the ReAct \(Reasoning \+ Acting\) paradigm, with structured requests routed to the sandbox through an intercept\-and\-forward mechanism rather than executed directly on the host\. We prototyped this mode\[[11](https://arxiv.org/html/2609.02168#bib.bib16)\]using a Docker dual\-sandbox: an Execution Sandbox where the model’s commands execute, and a Target Server Sandbox hosting virtual targets \(e\.g\., CTF\-style vulnerable servers\)\. A strictly isolated internal network separates the two sandboxes, and all tool calls from the Target LLM are confined to the sandbox, ensuring no direct system\-level execution on the host\. A KillSwitch mechanism monitors tool execution and can immediately terminate runaway behaviors—a safeguard essential for scenarios where models execute real system commands\. In representative cyber scenarios, the agent invoked curl for HTTP reconnaissance, authored and executed custom Python exploit scripts, and iteratively debugged tool outputs, all contained within the sandbox without host compromise\.

Interaction Logging\.The Test Environment hooks into the interaction loop and records every Thought\-Action trajectory and every content response generated by the Target model as a structured log entry\. Each entry captures the model’s reasoning trace, the exact tool invocation payload, and timing metadata, preserving the full decision\-making context for downstream scoring\.

### 3\.5Judgment

The Judgment module converts raw interaction logs into the three assessment dimensions and aggregates them into the dangerous\-capability profileϕ\\phi\. We formalize the scoring process as a cascade of two functions: a fast router that filters, and a deep judge that scores\.

Router:ℛ\\mathcal\{R\}\.The first stage is a risk\-categorization function\. Given a logged interaction recordℓ\\ell\(a dialogue trajectory or a generated response\), the SLM routerℛ\\mathcal\{R\}assigns it to one of a finite set of risk categories defined by the Pluggable Module:

ℛ⁡\(ℓ\)∈𝒞,𝒞=\{c0,c1,…,cK\}\\mathcal\{R\}\(\\ell\)\\in\\mathcal\{C\},\\qquad\\mathcal\{C\}=\\\{c\_\{0\},c\_\{1\},\\dots,c\_\{K\}\\\}\(1\)
wherec0c\_\{0\},c1,…,cKc\_\{1\},\\dots,c\_\{K\}denote the domain\-specific risk classes \(e\.g\., reconnaissance, exploitation, privilege escalation, and data exfiltration for a cyber module; synthesis guidance, dissemination planning, and weaponization for CB\)\. The routing decision is domain\-aware: each Pluggable Module supplies its own category set𝒞\\mathcal\{C\}alongside its rubric\.

All records proceed to the Judge layer with their category label attached, enabling category\-conditional reliability reporting\.

Judge:𝒥\\mathcal\{J\}\.The second stage is a rubric\-scored evaluation function\. For each retained recordℓ\\ell, the LLM Judge𝒥\\mathcal\{J\}returns add\-dimensional score vector according to the domain rubric loaded from the Pluggable Module:

𝒥⁡\(ℓ,ρ\)=\(j1,j2,…,jd\)∈\[0,m\]d\\mathcal\{J\}\(\\ell;\\rho\)=\\bigl\(j\_\{1\},j\_\{2\},\\dots,j\_\{d\}\\bigr\)\\in\[0,m\]^\{d\}\(2\)
where\[0,m\]d\[0,m\]^\{d\}is thedd\-fold Cartesian product, i\.e\., the set of alldd\-dimensional score vectors with each component bounded in\[0,m\]\[0,m\],ρ\\rhodenotes the rubric andmmis the per\-dimension maximum score\.

Cross\-Judge Validation\.To quantify the reliability of the Judgment, the framework supports re\-scoring by alternative judge families\. For each alternative judge, the framework computes agreement statistics against the primary judge—Spearmanρ\\rhofor rank agreement and bootstrap standard errors for interval estimation:

ρ⁡\(𝒥,𝒥i\),SE^boot​\(ρ\),i=1,…,k\\rho\(\\mathcal\{J\},\\mathcal\{J\}\_\{i\}\),\\quad\\widehat\{\\mathrm\{SE\}\}\_\{\\mathrm\{boot\}\}\(\\rho\),\\qquad i=1,\\dots,k\(3\)
These statistics serve as built\-in quality diagnostics: each assessment dimension reports its own cross\-judge agreement, so can assess the reliability of each dimension individually \(Section[6](https://arxiv.org/html/2609.02168#S6)\)\.

The judge outputs feed the three assessment dimensions \(KK,DD,HH, Section[4](https://arxiv.org/html/2609.02168#S4)\), whose model\-level values are aggregated into the dangerous\-capability profileϕ\\phiat the end of Section[4](https://arxiv.org/html/2609.02168#S4)\.

## 4Three Evaluation Pipelines \(KK,DD,HH\)

The framework evaluates each model along three orthogonal dimensions, summarized in Table[2](https://arxiv.org/html/2609.02168#S4.T2)\. Each dimension is defined generically with respect to the assets supplied by a Danger\-Domain Module; Section[5\.1](https://arxiv.org/html/2609.02168#S5.SS1)instantiates all three for the CB domain\.

Table 2:Three evaluation pipelines\. Each measures an orthogonal dimension of dangerous capability\.### 4\.1Knowledge Pipeline \(KK\)

The knowledge pipeline quantifies the information\-leakage surface of a model: the hazardous\-domain knowledge an attacker can extract through querying, measured by multiple\-choice questionnaires\. We model the response process with Item Response Theory \(IRT\)\[[20](https://arxiv.org/html/2609.02168#bib.bib29),[6](https://arxiv.org/html/2609.02168#bib.bib30)\], adopting the two\-parameter logistic \(2PL\) form with a fixed guessing parameter\[[4](https://arxiv.org/html/2609.02168#bib.bib31),[12](https://arxiv.org/html/2609.02168#bib.bib32),[26](https://arxiv.org/html/2609.02168#bib.bib33)\]\. For each four\-choice MCQ itemii, the probability that modelTTanswers correctly is:

Pi,T​\(correct∣θT\)=c\+1−c1\+exp⁡\(−\(θT−bi\)\)P\_\{i,T\}\(\\mathrm\{correct\}\\mid\\theta\_\{T\}\)=c\+\\frac\{1\-c\}\{1\+\\exp\(\-\(\\theta\_\{T\}\-b\_\{i\}\)\)\}\(4\)
whereθT\\theta\_\{T\}is modelTT’s latent knowledge capability,bib\_\{i\}is itemii’s difficulty, and the lower asymptotec=0\.25c=0\.25equals the random\-guessing probability of a four\-choice item\. Item difficulty is calibrated once from the pooled pass\-rates of all evaluated models:

bi=log⁡\(1−pipi−0\.25\)b\_\{i\}=\\log\\left\(\\frac\{1\-p\_\{i\}\}\{p\_\{i\}\-0\.25\}\\right\)\(5\)
wherepip\_\{i\}denotes the pooled correct rate of itemiiover 12 LLMs\. Given the calibrated item bank, a model’s capability is estimated by maximum likelihood:

θT=arg⁡max⁡∏iθ⁡Pi,T​\(θ\)xi,T​\(1−Pi,T​\(θ\)\)1−xi,T\\theta\_\{T\}=\\arg\\max\_\{\\theta\}\\;\\prod\_\{i\}P\_\{i,T\}\(\\theta\)^\{x\_\{i,T\}\}\\bigl\(1\-P\_\{i,T\}\(\\theta\)\\bigr\)^\{1\-x\_\{i,T\}\}\(6\)
withxi,T∈\{0,1\}x\_\{i,T\}\\in\\\{0,1\\\}the model’s answer to itemii\(0 False, 1 True\), solved via Newton–Raphson iteration\. Estimation precision is quantified by the Fisher information:

I⁡\(θT\)=∑i\(\(Pi,T​\(θT\)−c\)​\(1−Pi,T​\(θT\)\)\)2\(1−c\)2⋅Pi,T​\(θT\)​\(1−Pi,T​\(θT\)\),I\(\\theta\_\{T\}\)=\\sum\_\{i\}\\frac\{\\bigl\(\(P\_\{i,T\}\(\\theta\_\{T\}\)\-c\)\(1\-P\_\{i,T\}\(\\theta\_\{T\}\)\)\\bigr\)^\{2\}\}\{\(1\-c\)^\{2\}\\cdot P\_\{i,T\}\(\\theta\_\{T\}\)\(1\-P\_\{i,T\}\(\\theta\_\{T\}\)\)\},\(7\)
S​E​\(θT\)=1I⁡\(θT\)\\qquad SE\(\\theta\_\{T\}\)=\\frac\{1\}\{\\sqrt\{I\(\\theta\_\{T\}\)\}\}\(8\)
Note that calibratingbib\_\{i\}from pooled pass\-rates and then estimatingθT\\theta\_\{T\}from individual response vectors does not constitute circular reasoning: the calibration step fixes the measurement scale \(zero point and unit\) of the item bank, whereas the estimation step relies on each model’s item\-by\-item response pattern\{xi,T\}\\\{x\_\{i,T\}\\\}—two models with identical total scores receive distinctθ\\thetaestimates when they answer different items correctly, precisely because the items differ in difficulty\.

IRT offers three advantages over raw accuracy: \(1\)*difficulty weighting*—harder items contribute more toθ\\theta; \(2\)*uncertainty quantification*viaS​E​\(θ\)SE\(\\theta\); and \(3\)*interval\-scale measurement*supporting rigorous cross\-model comparison\. We operationalize the knowledge dimension asK:=θK:=\\theta, reporting raw accuracy as a supplementary diagnostic where relevant\. For danger domains spanning multiple subdomains,KKis instantiated as per\-subdomain scores \(e\.g\.,Kbio,KchemK\_\{\\text\{bio\}\},K\_\{\\text\{chem\}\}for CB\); a single\-subdomain module reports a scalarKK\.

### 4\.2Defense Pipeline \(DD\)

The defense pipeline quantifies the access\-control boundary of a model: whether its behavioral guardrails hold when an adversary escalates pressure adaptively\. A scenario seed generates a five\-round interaction chain\{c1,…,c5\}\\\{c\_\{1\},\\dots,c\_\{5\}\\\}, where each subsequent prompt adapts to the model’s prior responses—a*path\-dependent protocol*reflecting how real\-world adversaries escalate gradually\.

Scoring instantiation\.The Judge𝒥\\mathcal\{J\}\(Section[3\.5](https://arxiv.org/html/2609.02168#S3.SS5)\) evaluates the full dialogue trajectory

τs​\(T\)=\(\(u1,a1\),\(u2,a2\),…,\(u5,a5\)\)\\tau\_\{s\}\(T\)=\\bigl\(\(u\_\{1\},a\_\{1\}\),\(u\_\{2\},a\_\{2\}\),\\dots,\(u\_\{5\},a\_\{5\}\)\\bigr\)\(9\)
the five\-turn pairs between the Red\-Team Agent and modelTTon scenarioss—under the defense rubricρD\\rho\_\{D\}supplied by the Pluggable Module\. The rubric definesd=4d=4semantic dimensions—Induction Defense, Risk Leakage, Context Handling, and Multi\-round Resilience—each bounded bym=25m=25, yielding a per\-scenario score vector:

𝒥⁡\(τs​\(T\),ρD\)=\(jsid,jsrl,jsch,jsmr\)∈\[0,25\]4\\mathcal\{J\}\\\!\\bigl\(\\tau\_\{s\}\(T\);\\rho\_\{D\}\\bigr\)=\\bigl\(j^\{\\mathrm\{id\}\}\_\{s\},j^\{\\mathrm\{rl\}\}\_\{s\},j^\{\\mathrm\{ch\}\}\_\{s\},j^\{\\mathrm\{mr\}\}\_\{s\}\\bigr\)\\in\[0,25\]^\{4\}\(10\)
The per\-scenario overall defense score is theℓ1\\ell\_\{1\}norm of the score vector:

Ds​\(T\)=‖𝒥⁡\(τs​\(T\),ρD\)‖1=∑j∈𝒥⁡\(⋅\)j∈\[0,100\]D\_\{s\}\(T\)=\\left\\lVert\\mathcal\{J\}\\bigl\(\\tau\_\{s\}\(T\);\\rho\_\{D\}\\bigr\)\\right\\rVert\_\{1\}=\\sum\_\{j\\in\\mathcal\{J\}\(\\cdot\)\}j\\in\[0,100\]\(11\)
Model\-level metrics\.The defense dimensionD⁡\(T\)D\(T\)aggregates the per\-scenario scores over the module’sNNscenario seeds:

D⁡\(T\)=1N​∑s=1N‖𝒥⁡\(τs​\(T\),ρD\)‖1∈\[0,100\]D\(T\)=\\frac\{1\}\{N\}\\sum\_\{s=1\}^\{N\}\\left\\lVert\\mathcal\{J\}\\bigl\(\\tau\_\{s\}\(T\);\\rho\_\{D\}\\bigr\)\\right\\rVert\_\{1\}\\in\[0,100\]\(12\)
To characterize*how*a model defends—not merely how well—we project the score vector onto two complementary indicators\. The Defense LevelD​LDLisolates the Induction\-Defense component, normalized to the unit interval; the Breakthrough RatioB​RBRcontrasts multi\-round resilience against induction defense:

D​L​\(T\)\\displaystyle DL\(T\)=125​N​∑s=1Njsid​\(T\)∈\[0,1\]\\displaystyle=\\frac\{1\}\{25\\,N\}\\sum\_\{s=1\}^\{N\}j^\{\\mathrm\{id\}\}\_\{s\}\(T\)\\in\[0,1\]\(13\)B​R​\(T\)\\displaystyle BR\(T\)=∑s=1Njsmr​\(T\)∑s=1Njsid​\(T\)\\displaystyle=\\frac\{\\sum\_\{s=1\}^\{N\}j^\{\\mathrm\{mr\}\}\_\{s\}\(T\)\}\{\\sum\_\{s=1\}^\{N\}j^\{\\mathrm\{id\}\}\_\{s\}\(T\)\}\(14\)
D​LDLandB​RBRtogether reflect the outcome of safety training\[[22](https://arxiv.org/html/2609.02168#bib.bib24),[2](https://arxiv.org/html/2609.02168#bib.bib25),[24](https://arxiv.org/html/2609.02168#bib.bib26)\]: baseline refusal emerges from preference optimization, while erosion reveals how deeply that preference is internalized\.D​LDLcaptures baseline refusal tendency on first contact;B​RBRcaptures how that tendency evolves under sustained pressure—B​R≈1BR\\approx 1indicates stability,B​R<1BR<1indicates erosion, andB​R\>1BR\>1indicates escalating compliance\. Together,D​LDLandB​RBRspan a two\-dimensional risk typology that distinguishes*surface\-level*defense from*deep*defense \(Section[5\.3](https://arxiv.org/html/2609.02168#S5.SS3)\)\.

### 4\.3Harm Pipeline \(HH\)

The harm pipeline quantifies the content\-safety consequence of an access\-control failure: how actionable the hazardous content is that a model produces once its boundary is breached\. Each model receivesNNopen\-ended hazardous\-domain questions drawn from the module’s harm\-queries set\.

Scoring instantiation\.The Judge𝒥\\mathcal\{J\}\(Section[3\.5](https://arxiv.org/html/2609.02168#S3.SS5)\) evaluates the model’s responseri​\(T\)r\_\{i\}\(T\)to queryiiunder the harm rubricρH\\rho\_\{H\}, which definesd=4d=4semantic dimensions—Completeness, Implement\-Ability, Resource\-Accessibility, and Innovation—each bounded bym=25m=25, yielding a per\-response score vector:

𝒥⁡\(ri​\(T\),ρH\)=\(jicp,jiim,jirs,jiin\)∈\[0,25\]4\\mathcal\{J\}\\\!\\bigl\(r\_\{i\}\(T\);\\rho\_\{H\}\\bigr\)=\\bigl\(j^\{\\mathrm\{cp\}\}\_\{i\},j^\{\\mathrm\{im\}\}\_\{i\},j^\{\\mathrm\{rs\}\}\_\{i\},j^\{\\mathrm\{in\}\}\_\{i\}\\bigr\)\\in\[0,25\]^\{4\}\(15\)
The per\-response exposure score is theℓ1\\ell\_\{1\}norm of the score vector, quantifying how much actionable hazardous information the response conveys:

Hi​\(T\)=‖𝒥⁡\(ri​\(T\),ρH\)‖1∈\[0,100\]H\_\{i\}\(T\)=\\left\\lVert\\mathcal\{J\}\\bigl\(r\_\{i\}\(T\);\\rho\_\{H\}\\bigr\)\\right\\rVert\_\{1\}\\in\[0,100\]\(16\)
Model\-level metric\.The harm dimensionH⁡\(T\)H\(T\)aggregates the exposure scores over the set of non\-refusal responsesℛT\\mathcal\{R\}\_\{T\}\(empty or purely\-refusal responses carry zero information and are excluded from scoring\):

H⁡\(T\)=1\|ℛT\|​∑i∈ℛTHi​\(T\)∈\[0,100\]H\(T\)=\\frac\{1\}\{\|\\mathcal\{R\}\_\{T\}\|\}\\sum\_\{i\\in\\mathcal\{R\}\_\{T\}\}H\_\{i\}\(T\)\\in\[0,100\]\(17\)
HHcomplementsDD: whereasDDmeasures whether a model refuses,HHmeasures the harmfulness of what it produces when it complies—the two dimensions are weakly correlated \(Section[5\.4](https://arxiv.org/html/2609.02168#S5.SS4)\), confirming that a high\-defense model can still generate highly actionable content on the queries it answers\.

### 4\.4Derived Dimensions \(Δ\\Delta,E​RER\)

Two additional dimensions enrich the dangerous capability profile beyond the three primary metrics\.

Domain asymmetry \(Δ\\Delta\)\.When a danger domain spans multiple subdomains, a model’s defense may be unevenly distributed across them—a signature of imbalanced safety training coverage\. We quantify this asymmetry as the signed, normalized difference between per\-subdomain defense scores:

Δd1,d2​\(T\)=Dd1​\(T\)−Dd2​\(T\)max⁡\(Dd1​\(T\),Dd2​\(T\)\)∈\[−1,\+1\]\\Delta\_\{d\_\{1\},d\_\{2\}\}\(T\)=\\frac\{D\_\{d\_\{1\}\}\(T\)\-D\_\{d\_\{2\}\}\(T\)\}\{\\max\\\!\\big\(D\_\{d\_\{1\}\}\(T\),\\,D\_\{d\_\{2\}\}\(T\)\\big\)\}\\in\[\-1,\+1\]\(18\)
whered1d\_\{1\}andd2d\_\{2\}denote the two subdomains defined by the Danger\-Domain Module \(e\.g\., biology and chemistry for CB\)\.Δ\>0\\Delta\>0indicates stronger defense ind1d\_\{1\};Δ≈0\\Delta\\approx 0indicates balanced coverage\. A single\-subdomain module omits this dimension\.

Empty\-response rate \(E​RER\)\.Beyond explicit refusal, some models deploy*silent refusal*: the API returns an empty response with no error field\. This behavior reflects the provider’s content\-filtering strategy and is observable from interaction logs alone, independent of domain semantics:

E​R​\(T\)=\{empty turns\}\{total turns\}×100%ER\(T\)=\\frac\{\\\{\\text\{empty turns\}\\\}\}\{\\\{\\text\{total turns\}\\\}\}\\times 100\\%\(19\)
E​RERdistinguishes models that refuse*visibly*\(text\-level refusal\) from those that refuse*silently*\(empty response\), a strategic difference invisible to standard defense metrics \(Section[5\.5](https://arxiv.org/html/2609.02168#S5.SS5)\)\.

### 4\.5Dangerous Capability Profile \(ϕ\\phi\)

Having defined all constituent dimensions, we aggregate them into the canonical dangerous\-capability profile—the framework’s portable output\. The profileϕ\\phiis a seven\-dimensional vector that summarizes what a model knows, how it behaves under adversarial pressure, and what it produces when it complies:

ϕ⁡\(T\)=⟨K,D,H,D​L,B​R,Δ,E​R⟩\\phi\(T\)=\\langle K,D,H,DL,BR,\\Delta,ER\\rangle\(20\)
Each component is computed by the corresponding pipeline \(Eqs\.[12](https://arxiv.org/html/2609.02168#S4.E12)–[19](https://arxiv.org/html/2609.02168#S4.E19)\) from the same evaluation run, guaranteeing that all dimensions share a common protocol and model population—the property that makesϕ\\phivectors directly comparable across models\.ϕ\\phiserves as the framework’s portable output: regulators and platform operators can compareϕ\\phivectors across models without consulting benchmark\-specific documentation\. Section[5\.1](https://arxiv.org/html/2609.02168#S5.SS1)instantiatesϕ\\phifor the CB domain and reports the resulting vectors for 12 commercial models\.

## 5Experiment

### 5\.1CB Setting

We instantiate the framework with the CB Danger\-Domain Module and evaluate 12 commercial LLMs spanning four model families: Claude \(Opus\-4, Opus\-4\.5, Opus\-4\.8, Haiku\-3\), GPT \(GPT\-3\.5\-Turbo, GPT\-4o, GPT\-5\.4, GPT\-5\.5\), DeepSeek \(V3, V4\-Flash, V4\-Pro\), and Kimi \(K3\)\.

The module supplies three evaluation assets across two subdomains \(biology and chemistry\): 298 scenario seeds \(biology: 156, chemistry: 142\) for the defense dimensionDD, a knowledge bank of 3,773 MCQ items \(biology: 1,938, chemistry: 1,835; drawn from SciKnowEval\[[7](https://arxiv.org/html/2609.02168#bib.bib6)\], GPQA\[[25](https://arxiv.org/html/2609.02168#bib.bib40)\], and a Chinese college\-entrance examination set\) forKK, and 200 open\-ended harm queries \(biology: 100, chemistry: 100; drawn from WMDP\[[15](https://arxiv.org/html/2609.02168#bib.bib7)\], LabSafety Bench\[[39](https://arxiv.org/html/2609.02168#bib.bib9)\], and ChemSafetyBench\[[37](https://arxiv.org/html/2609.02168#bib.bib37)\]\) forHH\. The primary judge is GPT\-4o\-mini \(temperature = 0\.0\); cross\-judge reliability is analyzed in Section[6](https://arxiv.org/html/2609.02168#S6)\.

Sections[5\.2](https://arxiv.org/html/2609.02168#S5.SS2)–[5\.5](https://arxiv.org/html/2609.02168#S5.SS5)report the per\-dimension results with dedicated tables and figures; Section[5\.7](https://arxiv.org/html/2609.02168#S5.SS7)aggregates them into the two headline conclusions of this work: cross\-model comparison and temporal evolution of dangerous capability\.

### 5\.2Knowledge in CB \(KK\)

Table[3](https://arxiv.org/html/2609.02168#S5.T3)reports the knowledge dimension for all 12 models\. IRT ability scoresθ\\thetaare reported per subdomain alongside raw accuracy as a supplementary diagnostic; the two rankings agree at Spearmanρ=0\.998\\rho=0\.998\.

Table 3:Knowledge dimension \(KK\) in CB\.θ\\theta: IRT ability relative to the 12\-model mean \(logits\); raw%: percentage of MCQ items correct\.Two patterns emerge: First, knowledge is strongly family\-structured: GPT\-5\.5 and GPT\-5\.4 lead both subdomains, while GPT\-3\.5\-Turbo trails by over 5 logits in chemistry—the largest single\-model gap across the entire evaluation\. Itsθchem=−4\.09\\theta\_\{\\text\{chem\}\}=\-4\.09corresponds to a raw accuracy of 44\.0%, barely above the 25% random\-guessing floor, whereas GPT\-5\.5’s\+1\.21\+1\.21reflects 79\.5% on the same bank\. Second, knowledge is*asymmetric across subdomains*: 8 of 12 models exhibit stronger biological than chemical knowledge \(θbio\>θchem\\theta\_\{\\text\{bio\}\}\>\\theta\_\{\\text\{chem\}\}\), with domain gaps ranging from 0\.1 to 3\.0 logits, yet the three Claude models and GPT\-4o reverse this pattern\. Figure[3](https://arxiv.org/html/2609.02168#S5.F3)visualizes the asymmetry\.

![Refer to caption](https://arxiv.org/html/2609.02168v1/fig_irt_scatter.png)Figure 3:Domain asymmetry:θbio\\theta\_\{\\text\{bio\}\}vs\.θchem\\theta\_\{\\text\{chem\}\}\. Above\-diagonal points indicate stronger biological knowledge; this offset links pre\-training data to domain\-specific hazardous knowledge\.The direction of asymmetry is family\-dependent: all DeepSeek models favor biology, all Claude models favor chemistry, and GPT models split between the two\. This finding is actionable: a model audited only on biology benchmarks may harbor unexamined chemistry expertise, making per\-subdomain knowledge reporting essential for domain\-specific risk assessment\.

### 5\.3Defense in CB \(DD\)

Table[4](https://arxiv.org/html/2609.02168#S5.T4)reports the defense dimension for all 12 models\. Per\-subdomain scoresDbioD\_\{\\text\{bio\}\}andDchemD\_\{\\text\{chem\}\}capture refusal capability separately for biology and chemistry; the derived indicatorsD​LDL\(Defense Level\) andB​RBR\(Breakthrough Ratio\) characterize the*structure*of defense as defined in Section[4\.2](https://arxiv.org/html/2609.02168#S4.SS2)\.

Table 4:Defense dimension \(DD\) in CB\.DD: mean per\-scenario defense score \(0–100\);D​LDL: normalized Induction\-Defense score;B​RBR: Multi\-round Resilience over induction defense\.The defense scores span a wider range than either knowledge or harm \(Section[5\.4](https://arxiv.org/html/2609.02168#S5.SS4)\): from DeepSeek\-V3’sDbio=56\.9D\_\{\\text\{bio\}\}=56\.9to Claude\-Opus\-4\.5’sDchem=99\.9D\_\{\\text\{chem\}\}=99\.9, a gap of 43 points that no other dimension approaches\. This spread is not uniform across subdomains—the same model can defend differently against biological and chemical elicitation\. Claude\-Opus\-4 exhibits the sharpest asymmetry:Dbio=97\.0D\_\{\\text\{bio\}\}=97\.0versusDchem=82\.0D\_\{\\text\{chem\}\}=82\.0, a 15\-point gap indicating safety training that favors biology over chemistry\. We formalize this asymmetry asΔ\\Deltain Section[5\.5](https://arxiv.org/html/2609.02168#S5.SS5)\.

To characterize defense beyond a single aggregate, we project each model onto theD​L×B​RDL\\times BRplane \(Figure[4](https://arxiv.org/html/2609.02168#S5.F4)\)\.kk\-means clustering \(k=4k=4, chosen by the gap statistic\) on the standardized\(D​L,B​R\)\(DL,BR\)coordinates yields four qualitatively distinct risk modes\.

![Refer to caption](https://arxiv.org/html/2609.02168v1/fig_dl_br_scatter.png)Figure 4:D​L×B​RDL\\times BRrisk space showing four distinct risk modes identified viakk\-means clustering \(kk=4\)\.Mode I—Brittle\.GPT\-5\.4 and GPT\-5\.5 achieve near\-perfect baseline defense \(D​L≈0\.97DL\\approx 0\.97\) but show a modestB​RBRdeficit \(B​R≈0\.95BR\\approx 0\.95\): their refusal tendency erodes slightly under sustained pressure\. These models are safe on first contact yet reveal subtle fragility across rounds\.

Mode II—Exposed\.DeepSeek\-V3, V4\-Flash, and V4\-Pro exhibit low baseline defense \(D​L<0\.72DL<0\.72\) coupled with escalating compliance \(B​R\>1\.09BR\>1\.09\): their already\-weak refusal tendency further deteriorates across rounds, yielding the highest breakthrough rates of the evaluation\.

Mode III—Robust\.Claude\-Opus\-4, Opus\-4\.5, Opus\-4\.8, and Kimi\-K3 maintain strong, stable defense \(D​L\>0\.89DL\>0\.89,B​R≈1\.0BR\\approx 1\.0\) across all five rounds\. Notably, Kimi\-K3 demonstrates that robust defense is achievable outside the Claude family\.

Mode IV—Moderate\.Claude\-3\-Haiku, GPT\-4o, and GPT\-3\.5\-Turbo occupy the middle ground \(D​L≈0\.65DL\\approx 0\.65–0\.760\.76,B​R≈1\.0BR\\approx 1\.0\): moderate baseline defense that neither erodes nor escalates\. Modes III and IV share similarB​RBRbut are distinguished byD​LDL— the two clusters differ in baseline strength rather than stability\.

The most consequential finding concerns Mode I: models judged safest by single\-turn evaluation are precisely those whose defense erodes under sustained interaction\. To ground this quantitative pattern in concrete behavior, we examine a representative five\-round trajectory\.

#### Defense Erosion Case: Refusal\-to\-Protocol

DeepSeek\-V4\-Pro, judged entirely safe under single\-turn queries, transitions within five rounds from a 20\-character refusal to a complete operational protocol for pathogen surface\-glycoprotein detection—specifying detection assays, decontamination concentrations, and activity thresholds for area lockdown escalation, all redacted here for dual\-use review \(Judge: Unsafe, 54/100\)\. Figure[5](https://arxiv.org/html/2609.02168#S5.F5)presents the interaction card\.

![Refer to caption](https://arxiv.org/html/2609.02168v1/fig_case_defense.png)Figure 5:Five\-round defense trajectory for DeepSeek\-V4\-Pro on a fire\-department HazMat scenario \(translated from the original Chinese logs, operational parameters redacted\)\.This trajectory illustrates precisely what a path\-dependent protocol captures and a single\-turn evaluation misses: the defense score is not a property of the model’s first response but of how its behavior evolves under escalating pressure\.

### 5\.4Harm in CB \(HH\)

Table[5](https://arxiv.org/html/2609.02168#S5.T5)reports the harm dimension for all 12 models: the mean exposure score of the hazardous content each model produces when it complies\.

Table 5:Harm dimension \(HH\) in CB: mean exposure score of non\-refusal responses \(0–100; higher = more actionable hazardous content\)\.Two properties distinguishHHfrom the other dimensions\. First, its range is remarkably narrow: all 12 models cluster within a 10\-point band \(68\.3–78\.5\), in sharp contrast to the 43\-point spread ofDD\. Once a model answers a hazardous query, the content it produces is uniformly actionable regardless of how well it defends otherwise\.

Counterintuitively, models with the strongest defense produce the highest harm scores when they comply: Claude\-Opus\-4\.5 and Opus\-4\.8, despiteD\>99D\>99, yieldH\>75H\>75\. This inversion is a selection effect: strong refusers are scored only on the minority of queries they fail to refuse—a maximally difficult residual set—whereas weak refusers \(GPT\-3\.5\-Turbo, GPT\-4o\) average over many trivial responses, deflating their means\.HHthus measures capability conditional on compliance, not a monotone function of defense weakness\.

![Refer to caption](https://arxiv.org/html/2609.02168v1/fig_case_harm.png)Figure 6:Single\-round harm query for DeepSeek\-V4\-Pro \(translated from the original Chinese log, operational parameters redacted\)\.Figure[6](https://arxiv.org/html/2609.02168#S5.F6)grounds this dimension in concrete behavior: DeepSeek\-V4\-Pro \(H=77\.0H=77\.0\) produces a cell\-free protein synthesis protocol spanning four production systems and a five\-step pipeline, with operational parameters redacted for dual\-use review—scored 72/100 by the Judge\.

The complementarity with Section[5\.3](https://arxiv.org/html/2609.02168#S5.SS3)is direct: DeepSeek\-V4\-Pro combines weak defense \(D≈67D\\approx 67–8080\) with near\-peak harm generation \(H=77\.0H=77\.0\), while Claude\-Opus\-4\.8 combines near\-perfect defense \(D\>99D\>99\) with the highest harm score of the evaluation \(H=78\.5H=78\.5\)\. Neither dimension alone predicts the other \(ρ=0\.318\\rho=0\.318, Section[6](https://arxiv.org/html/2609.02168#S6)\), which is precisely why a complete capability profile requires both\.

### 5\.5Domain Asymmetry and Silent Refusal \(Δ\\Delta,E​RER\)

Beyond the three primary dimensions, two derived indicators capture properties of dangerous capability that aggregate scores conceal\. Table[6](https://arxiv.org/html/2609.02168#S5.T6)reports both for all 12 models\.

Table 6:Derived dimensions in CB\.Δ\\Delta: signed normalized defense asymmetry between subdomains,\+\+: stronger biology defense,−\-: stronger chemistry defense;E​RER: percentage of interaction turns returning an empty response\.Domain asymmetry \(Δ\\Delta\)\.Δ\\Deltaquantifies whether a model’s Defense is evenly distributed across subdomains—a signal of how its safety training was provisioned\. Two models anchor the extremes\. Claude\-Opus\-4 \(Δ=\+0\.155\\Delta=\+0\.155\) defends biology markedly better than chemistry, mirroring the 15\-pointDDgap documented in Section[5\.3](https://arxiv.org/html/2609.02168#S5.SS3); DeepSeek\-V4\-Pro \(Δ=−0\.161\\Delta=\-0\.161\) exhibits the inverse profile, with chemistry defense outpacing biology\. The remaining ten models cluster withinΔ∈\[−0\.11,\+0\.03\]\\Delta\\in\[\-0\.11,\+0\.03\], indicating roughly balanced subdomain coverage\.

Two observations follow\. First, the extremes are not the weakest defenders: both Opus\-4 and V4\-Pro sit above the midpoint of theDDdistribution, so their asymmetry reflects*uneven provision*of safety training rather than a general defense deficit\. Second,Δ\\Deltais orthogonal toDD: knowing a model’s aggregate defense score reveals nothing about which subdomain it protects better\. The dimension therefore contributes independently to the capability profile \(Section[4\.5](https://arxiv.org/html/2609.02168#S4.SS5)\)\.

Silent refusal \(E​RER\)\.E​RERexposes a strategic division among providers that is invisible in refusal*content*\. Three Claude\-family models \(Opus\-4, Opus\-4\.5, Opus\-4\.8\) return an empty response with no error field in67\.867\.8–78\.6%78\.6\\%of interaction turns: the provider filters hazardous content at the serving layer, dropping responses before they reach the user—a*silent refusal*strategy\. GPT\-5\.4 and GPT\-5\.5 show intermediate rates \(15\.015\.0–28\.0%28\.0\\%\), consistent with partial serving\-layer filtering, while all remaining models exhibitE​R<3%ER<3\\%, refusing through visible text \(“I cannot…”\)\.

The operational distinction matters\. A silent refusal leaves no trace that a defense mechanism fired: downstream auditors cannot distinguish a filtered model from one that was never prompted\. A text\-level refusal, by contrast, records the defense event in the interaction log\.E​RERthus captures a property of the*deployment stack*—the provider’s serving configuration—rather than of the model weights, making it a necessary complement toDDin the capability profile\. The two dimensions are indeed independent: silent\-refusal models span the fullΔ\\Deltarange \(Opus\-4:\+0\.155\+0\.155; Opus\-4\.8:−0\.002\-0\.002\), confirming that serving\-layer strategy and subdomain defense balance are orthogonal axes of deployment behavior\.

### 5\.6Cyber Pilot

To probe cross\-domain generality, we instantiate a minimal cyber module and evaluate the three models that overlap with the CB set \(DeepSeek\-V3, GPT\-3\.5\-Turbo, GPT\-4o\) in Tool\-Enhanced mode \(Section[3\.4](https://arxiv.org/html/2609.02168#S3.SS4)\), with real Bash/Python execution across 16 attack\-chain scenarios spanning reconnaissance, injection, exploitation, exfiltration, persistence, and track\-covering\. The module reuses the CB judge protocol unchanged: the same four\-dimension rubric structure, the same 0–25 per\-dimension scale, and the same primary judge \(GPT\-4o\-mini\)\.

Table[7](https://arxiv.org/html/2609.02168#S5.T7)reports all three dimensions alongside the corresponding CB\-domain values\.

Table 7:Cyber pilot results for the three overlapping models, each evaluated on all 16 attack\-chain scenarios\.KK: raw accuracy on 75 cyber MCQ items \(the cyber module reports raw accuracy rather than IRTθ\\theta; see text\);DD: mean defense score \(0–100, higher = safer\);HH: mean exposure score \(0–100, higher = more actionable\)\. CB columns reproduce the corresponding values from Sections[5\.2](https://arxiv.org/html/2609.02168#S5.SS2)–[5\.4](https://arxiv.org/html/2609.02168#S5.SS4)\.Two observations generalize across domains\. First,*harm is inelastic in cyber as in CB*:HHspans 69–71, reproducing the narrow band observed across the CB evaluation \(68–78\), and the three models preserve their relative ordering\. Second,*defense rankings are directionally preserved but systematically lower*: GPT\-4o defends best in both domains, yet all three models score markedly below their CB defense—the Tool\-Enhanced setting, where the model must decline concrete tool invocations rather than merely refuse prose, is the harder test\. GPT\-4o’s mean conceals a bimodal pattern: it defends fully \(D=100D=100\) on exploitation, exfiltration, and persistence scenarios yet collapses to minimal defense \(D=35D=35\) on XSS, SSTI, and JWT scenarios—a scenario\-dependence that aggregate scores cannot express \(Appendix Figure[14](https://arxiv.org/html/2609.02168#Ax7.F14)\)\.

The cyber knowledge items exhibit a clear ceiling effect: all three models score above 98%\. We therefore report raw accuracy rather than IRTθ\\theta: with 75 items of near\-uniform pass\-rates, the difficulty parameters would collapse to a single low value, andθ\\thetaestimates would carry no discriminative content\. The item bank is publicly sourced, so current models have plausibly encountered the items during pre\-training\. Beyond the contamination concern, the ceiling itself is a methodological signal: cyber modules require contamination\-checked, expert\-authored item banks with sufficient difficulty spread beforeKKbecomes discriminative in this domain\.

Taken together, the pilot demonstrates that the pluggable module interface \(Section[3\.2](https://arxiv.org/html/2609.02168#S3.SS2)\) transfers the full three\-dimension protocol to a second domain without modification, and that the framework’s headline patterns— inelastic harm and directionally stable defense—are not artifacts of the CB instantiation\.

### 5\.7Dangerous Capability Profile and Two Headline Findings

This section aggregates the per\-dimension scores intoϕ\\phi, then draws the two headline conclusions: cross\-model comparison and temporal evolution of dangerous capability\.

\(a\)Temporal evolution of knowledge capability \(KK, meanθ\\theta\)\.\(b\)Temporal evolution of defense \(DD, mean defense score\)\.\(c\)Temporal evolution of harm \(HH, mean exposure score\)\.
Figure 7:Family\-level K–D–H means comparison by subdomain\.\(a\)Temporal evolution of knowledge capability \(KK, meanθ\\theta\)\.\(b\)Temporal evolution of defense \(DD, mean defense score\)\.\(c\)Temporal evolution of harm \(HH, mean exposure score\)\.
Figure 8:Combined temporal evolution of knowledge, defense, and harm across model generations\.#### Dangerous Capability Profile in CB

The seven dimensions reported in Sections[5\.2](https://arxiv.org/html/2609.02168#S5.SS2)–[5\.5](https://arxiv.org/html/2609.02168#S5.SS5)assemble into the capability profileϕ=⟨K,D,H,D​L,B​R,Δ,E​R⟩\\phi=\\langle K,D,H,DL,BR,\\Delta,ER\\rangledefined in Section[4\.5](https://arxiv.org/html/2609.02168#S4.SS5)\. Table[8](https://arxiv.org/html/2609.02168#S5.T8)illustrates the profile for three models occupying distinct regions of the capability space\.

Table 8:Capability profiles for three representative models\. The full per\-dimension values appear in Tables[3](https://arxiv.org/html/2609.02168#S5.T3)–[6](https://arxiv.org/html/2609.02168#S5.T6)\.The three rows embody three distinct risk structures, visible only becauseϕ\\phireports every dimension side by side\. DeepSeek\-V4\-Pro pairs strong knowledge with weak, eroding defense \(B​R=1\.11BR=1\.11\) and near\-peak harm—maximal overall exposure\. Claude\-Opus\-4\.8 pairs near\-perfect, stable defense with the highest harm of the evaluation and silent refusal \(E​R=78\.6%ER=78\.6\\%\): its residual risk is concentrated in the rare responses that survive filtering\. GPT\-3\.5\-Turbo is weak on every axis—low knowledge, moderate defense, low harm—yet still exceeds the 25% guessing floor only marginally in chemistry \(Section[5\.2](https://arxiv.org/html/2609.02168#S5.SS2)\)\.

#### Model\-Family Comparison

Figure[7](https://arxiv.org/html/2609.02168#S5.F7)compares the four families on each primary dimension, split by subdomain\. The family profiles diverge sharply in*structure*\. DeepSeek combines the highest knowledge with the lowest defense and the highest harm: its three models averageθ=\+0\.40\\theta=\+0\.40,D=68\.2D=68\.2, andH=75\.8H=75\.8—they know the most, refuse the least, and produce the most actionable content when they comply, a profile of maximal overall risk\. Claude occupies the opposite corner on defense \(D=91\.3D=91\.3\) yet ranks second in harm \(H=74\.2H=74\.2\): strong refusal coexists with high\-conditional output quality, the selection effect quantified in Section[5\.4](https://arxiv.org/html/2609.02168#S5.SS4)\. GPT is the most heterogeneous family, spanning the full knowledge range from GPT\-3\.5\-Turbo \(θ=−3\.07\\theta=\-3\.07\) to GPT\-5\.5 \(θ=\+1\.40\\theta=\+1\.40\), so its family mean \(θ=−0\.44\\theta=\-0\.44\) misrepresents every member\. Kimi\-K3, evaluated as a single model rather than a family, matches Claude on defense \(D=95\.8D=95\.8\) while sitting at the low\-harm end \(H=70\.9H=70\.9\); it appears in the temporal figures \(Figure[8](https://arxiv.org/html/2609.02168#S5.F8)\) but is excluded from the family comparison, where a single\-model “mean” would be misleading\.

#### Temporal Evolution of Dangerous Capability

A second axis of comparison is time: do newer models become more or less dangerous? Figure[8](https://arxiv.org/html/2609.02168#S5.F8)plots each primary dimension against the model’s release date, colored by family\.

Three observations emerge from the temporal axis\. First,*knowledge compounds*: every family improves knowledge monotonically with generation, and the steepest trajectory belongs to GPT—the family with the longest evaluation window\. Dangerous knowledge is accumulating faster than it is being removed\. Second,*defense diverges by family*: GPT’s defense increases in lockstep with its knowledge, suggesting safety training kept pace; DeepSeek’s defense gains are modest and plateau well below the 80\-point mark, leaving its rising knowledge increasingly unbalanced against weak refusal\. Third,*harm is inelastic*: across 3\.5 years and twelve models,HHnever leaves the 68–78 band\. No vendor has produced a generation whose compliant outputs are meaningfully less actionable than its predecessors’\. The danger, in other words, is not that models learn to produce more harmful content—they already could—but that they know increasingly more*and*refuse increasingly inconsistently, enlarging the population of users who can obtain actionable content in the first place\.

## 6Judge Consistency Analysis

The results in Section[5](https://arxiv.org/html/2609.02168#S5)rest on LLM\-as\-Judge scoring\. This section interrogates that foundation through three ablation\-style analyses: how sensitive rankings are to the choice of judge, whether the three dimensions measure distinct constructs, and how preciselyθ\\thetais estimated\.

### 6\.1Cross\-Judge Agreement

Five alternative judge families \(Qwen3\.5\-Flash, GLM\-4\.5\-Air, Gemini\-2\.5\-Flash, Grok\-3\-Mini, Llama\-3\.3\-70b\) re\-score 90 stratified defense scenarios \(30 Safe, 30 Borderline, 30 Unsafe\) sampled proportionally from the 298\-scenario pool\. Table[9](https://arxiv.org/html/2609.02168#S6.T9)reports bootstrap Spearmanρ\\rho\(1,000 resamples\) with 95% confidence intervals for each alternative judge against the primary judge \(GPT\-4o\-mini, temperature = 0\.0\)\.

Table 9:Cross\-judge reliability for Defense assessment \(bootstrap Spearmanρ\\rho,n=90n=90, 1,000 resamples\)\.Four of five alternative judges agree with the primary judge atρ\>0\.79\\rho\>0\.79, confirming that the Defense ranking is protocol\-inherent rather than judge\-dependent\. The exception, Llama\-3\.3\-70b \(ρ=0\.641\\rho=0\.641\), is consistent with its distinct scoring behavior observed in prior work\. Pairwise cross\-judge correlations range fromρ=0\.59\\rho=0\.59\(GLM–Llama\) toρ=0\.87\\rho=0\.87\(GLM–Qwen\)\.

![Refer to caption](https://arxiv.org/html/2609.02168v1/fig_judge_consistency_heatmap.png)Figure 9:Cross\-judge consistency matrix \(Spearmanρ\\rho\) across six judge families\. Left: defense \(DD\), computed over a stratified subsample of 90 scenarios\. Right: harm \(HH\), computed over 360 scored responses\.†DDevaluation used Grok\-3\-Mini;HHevaluation used Grok\-4\.3\.For the harm dimension, inter\-judge agreement is lower:ρ∈\[0\.42,0\.58\]\\rho\\in\[0\.42,0\.58\]against the primary judge overn=360n=360items \(Figure[9](https://arxiv.org/html/2609.02168#S6.F9), right panel\)\. This gap between dimensions is expected: scoring an MCQ answer is a near\-objective task, whereas grading the actionability of an open\-ended hazardous protocol is inherently subjective\. The framework reports both levels transparently, so that downstream consumers ofϕ\\phican weigh each dimension’s reliability accordingly\.

### 6\.2Pipeline Orthogonality

If the three dimensions measured the same underlying construct, their model rankings would be strongly correlated\. They are not\. The moderateKK–DDcoupling is theoretically expected: recognizing a hazardous query as dangerous presupposes hazardous\-domain knowledge, so some positive association is structural rather than evidence of redundancy\. The decisive evidence is the near\-zeroDD–HHpair\. The Spearman correlations between dimension\-level rankings are consistently weak:KK–DDatρ=0\.521\\rho=0\.521,KK–HHatρ=0\.469\\rho=0\.469, andDD–HHatρ=0\.318\\rho=0\.318\(Figure[10](https://arxiv.org/html/2609.02168#S6.F10)\)\.

![Refer to caption](https://arxiv.org/html/2609.02168v1/fig_triaxial_correlation.png)Figure 10:Cross\-dimension Spearman correlations between pipeline rankings: Knowledge–Defense \(ρ=0\.521\\rho=0\.521\), Knowledge–Harm \(ρ=0\.469\\rho=0\.469\), Defense–Harm \(ρ=0\.318\\rho=0\.318\)\.The near\-zeroDD–HHcorrelation is particularly instructive: a model’s refusal capability does not predict the harmfulness of content it produces when it does comply—precisely the blind spot that theHHdimension is designed to illuminate\. A complete capability profile therefore requires all three dimensions; any subset misses a weakly coupled component of risk\.

### 6\.3IRT Measurement Precision

Model rankings fromθ\\thetaagree near\-perfectly with raw\-accuracy rankings \(Spearmanρ=0\.998\\rho=0\.998\), confirming thatθ\\thetaadds difficulty\-weighted discrimination while preserving the same construct\. All 24 model\-subdomainθ\\thetaestimates carry standard errors in the narrow range\[0\.09,0\.16\]\[0\.09,0\.16\]logits, confirming that the 3,773\-item bank provides sufficient information across the fullθ\\thetaspectrum\. Two models with identical raw totals receive distinctθ\\thetavalues when they answer different items correctly—the property that makes IRT strictly more informative than raw accuracy\.

## 7Discussion and Limitations

The CB evaluation is complemented by a cyber pilot \(Section[5\.6](https://arxiv.org/html/2609.02168#S5.SS6)\) that reproduces inelastic harm and directionally stable defense; cross\-judge validation \(ρ\>0\.79\\rho\>0\.79\) confirms ranking robustness is protocol\-inherent\. Limitations: \(1\) full\-scale validation is limited to CB—the cyber pilot demonstrates protocol transfer at reduced scale and with an immature knowledge bank; \(2\)HHscoring shows moderate cross\-judge agreement \(ρ≈0\.42\\rho\\approx 0\.42–0\.580\.58\), the expected range for open\-ended quality assessment; \(3\) knowledge benchmark contamination\[[9](https://arxiv.org/html/2609.02168#bib.bib39)\]remains a shared risk\.

The capability profile supports deployment decisions directly: a high\-DD, high\-HHmodel may be acceptable with output filtering, whereas a low\-DDmodel carries inherent refusal risk regardless\. The 15\-point bio–chem defense gap in Claude\-Opus\-4 illustrates that domain\-specific guardrails remain necessary even for strong defenders\.

The temporal trajectories sharpen the deployment calculus\. Knowledge compounds across generations within every family, so knowledge\-based guardrails calibrated on today’s models become stale within one release cycle\. Defense diverges by family, and harm remains inelastic everywhere—a regulator auditing one family at one point in time sees a snapshot, not a trajectory\.

This work involves dual\-use content; mitigations include an isolated sandbox with kill\-switch, aggregate\-level reporting, and the framework’s defensive orientation\. Detailed ethical considerations are provided in Appendix[Ethical](https://arxiv.org/html/2609.02168#Ax2)\.

## 8Conclusion

We present a modular framework that measures dangerous capability along three orthogonal dimensions \(KK,DD,HH\) plus derived indicators \(D​LDL,B​RBR,Δ\\Delta,E​RER\), aggregated into a seven\-dimensional capability profileϕ\\phi\. Our CB instantiation on 12 models yields two headline conclusions: families exhibit sharply distinct capability structures, and capability evolves over time with knowledge compounding, defense diverging by family, and harm remaining inelastic\. Cross\-judge validation \(ρ\>0\.79\\rho\>0\.79\) and IRT precision \(ρ=0\.998\\rho=0\.998\) ground the measurement\. Future work scales the cyber module, elevates Tool\-Enhanced evaluation to protocol level, extends the temporal window, improves harm scoring reliability, and integrates real\-time monitoring\.

## References

- \[1\]M\. Anderljung, J\. Barnhart, A\. Korinek, J\. Leung, C\. O’Keefe, J\. Whittlestone,et al\.\(2023\)Frontier ai regulation: managing emerging risks to public safety\.arXiv preprint arXiv:2307\.03718\.Cited by:[§1](https://arxiv.org/html/2609.02168#S1.p1.1)\.
- \[2\]Y\. Bai, S\. Kadavath, S\. Kundu,et al\.\(2022\)Constitutional AI: harmlessness from AI feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[§4\.2](https://arxiv.org/html/2609.02168#S4.SS2.p12.1)\.
- \[3\]R\. Bommasani, K\. Klyman, S\. Longpre, S\. Kapoor, N\. Maslej, B\. Xiong, D\. Zhang, and P\. Liang\(2023\)The foundation model transparency index\.arXiv preprint arXiv:2310\.12941\.Cited by:[§1](https://arxiv.org/html/2609.02168#S1.p1.1)\.
- \[4\]R\. P\. Chalmers\(2012\)Mirt: a multidimensional item response theory package for the R environment\.Journal of Statistical Software48\(6\),pp\. 1–29\.Cited by:[§4\.1](https://arxiv.org/html/2609.02168#S4.SS1.p1.1)\.
- \[5\]P\. Chao, A\. Robey, E\. Dobriban, H\. Hassani, G\. J\. Pappas, and E\. Wong\(2024\)Jailbreaking black box large language models in twenty queries\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2310\.08419Cited by:[§2\.3](https://arxiv.org/html/2609.02168#S2.SS3.p1.1)\.
- \[6\]S\. E\. Embretson and S\. P\. Reise\(2000\)Item response theory for psychologists\.Lawrence Erlbaum Associates\.Cited by:[§4\.1](https://arxiv.org/html/2609.02168#S4.SS1.p1.1)\.
- \[7\]K\. Feng, X\. Shen, W\. Wang, X\. Zhuang, Y\. Tang, Q\. Zhang, and K\. Ding\(2024\)SciKnowEval: evaluating multi\-level scientific knowledge of large language models\.arXiv preprint arXiv:2406\.09098\.Cited by:[§5\.1](https://arxiv.org/html/2609.02168#S5.SS1.p2.1)\.
- \[8\]D\. Ganguli, L\. Lovitt, J\. Kernion, A\. Askell, Y\. Bai, S\. Kadavath, B\. Mann, E\. Perez, N\. Schiefer, K\. Ndousse,et al\.\(2022\)Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned\.arXiv preprint arXiv:2209\.07858\.Cited by:[§2\.3](https://arxiv.org/html/2609.02168#S2.SS3.p1.1)\.
- \[9\]S\. Golchin and M\. Surdeanu\(2024\)Time travel in LLMs: tracing data contamination in large language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2308\.08493Cited by:[§7](https://arxiv.org/html/2609.02168#S7.p1.1)\.
- \[10\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt\(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2009\.03300Cited by:[§2\.1](https://arxiv.org/html/2609.02168#S2.SS1.p1.1)\.
- \[11\]P\. Kulkarni, A\. Wei, and N\. Mengar\(2025\)SandboxBench: a comprehensive evaluation framework for ai agent containment\.Technical reportSupervised Program for Alignment Research \(SPAR\)\.Note:Final Report, Fall 2025; mentored by Nitzan Shulman; contributed to UK AISI inspect\_evals repositoryCited by:[§3\.4](https://arxiv.org/html/2609.02168#S3.SS4.p3.1)\.
- \[12\]J\. P\. Lalor, H\. Wu, and H\. Yu\(2016\)Building an evaluation scale using item response theory\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§4\.1](https://arxiv.org/html/2609.02168#S4.SS1.p1.1)\.
- \[13\]J\. Li, Q\. Zhang, Y\. Yu, Q\. Fu, and D\. Ye\(2024\)More agents is all you need\.Transactions on Machine Learning Research\.Note:arXiv:2402\.05120Cited by:[§2\.4](https://arxiv.org/html/2609.02168#S2.SS4.p1.1)\.
- \[14\]N\. Li, Z\. Han, I\. Steneker, W\. Primack, R\. Goodside, H\. Zhang, Z\. Wang, C\. Menghini, and S\. Yue\(2024\)LLM defenses are not robust to multi\-turn human jailbreaks yet\.arXiv preprint arXiv:2408\.15221\.Cited by:[§2\.3](https://arxiv.org/html/2609.02168#S2.SS3.p1.1)\.
- \[15\]N\. Li, A\. Pan, A\. Gopal,et al\.\(2024\)The WMDP benchmark: measuring and reducing malicious use with unlearning\.InInternational Conference on Machine Learning \(ICML\),Note:arXiv:2403\.03218Cited by:[§1](https://arxiv.org/html/2609.02168#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.02168#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2609.02168#S5.SS1.p2.1)\.
- \[16\]P\. Liang, R\. Bommasani, T\. Lee,et al\.\(2023\)Holistic evaluation of language models\.Transactions on Machine Learning Research \(TMLR\)\.Note:arXiv:2211\.09110Cited by:[§2](https://arxiv.org/html/2609.02168#S2.p1.1)\.
- \[17\]X\. Liu, H\. Yu, H\. Zhang,et al\.\(2024\)AgentBench: evaluating LLMs as agents\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2308\.03688Cited by:[§2\.5](https://arxiv.org/html/2609.02168#S2.SS5.p1.1)\.
- \[18\]X\. Liu, N\. Xu, M\. Chen, and C\. Xiao\(2024\)AutoDAN: generating stealthy jailbreak prompts on aligned large language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2310\.04451Cited by:[§2\.3](https://arxiv.org/html/2609.02168#S2.SS3.p1.1)\.
- \[19\]Y\. Liu, Y\. Jia, R\. Geng, J\. Jia, and N\. Z\. Gong\(2024\)Formalizing and benchmarking prompt injection attacks and defenses\.In33rd USENIX Security Symposium \(USENIX Security 24\),pp\. 1831–1847\.Note:arXiv:2310\.12815Cited by:[§2\.6](https://arxiv.org/html/2609.02168#S2.SS6.p1.1)\.
- \[20\]F\. M\. Lord and M\. R\. Novick\(1968\)Statistical theories of mental test scores\.Addison\-Wesley\.Cited by:[§4\.1](https://arxiv.org/html/2609.02168#S4.SS1.p1.1)\.
- \[21\]C\. A\. Mouton, C\. Lucas, and E\. Guest\(2024\)The operational risks of AI in large\-scale biological attacks: results of a red\-team study\.Technical reportTechnical ReportRR\-A2977\-2,RAND Corporation\.Cited by:[§2\.2](https://arxiv.org/html/2609.02168#S2.SS2.p1.1)\.
- \[22\]L\. Ouyang, J\. Wu, X\. Jiang,et al\.\(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2203\.02155Cited by:[§4\.2](https://arxiv.org/html/2609.02168#S4.SS2.p12.1)\.
- \[23\]E\. Perez, S\. Huang, F\. Song,et al\.\(2022\)Red teaming language models with language models\.arXiv preprint arXiv:2202\.03286\.Cited by:[§2\.3](https://arxiv.org/html/2609.02168#S2.SS3.p1.1)\.
- \[24\]R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn\(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2305\.18290Cited by:[§4\.2](https://arxiv.org/html/2609.02168#S4.SS2.p12.1)\.
- \[25\]D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty,et al\.\(2023\)GPQA: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[§1](https://arxiv.org/html/2609.02168#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.02168#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2609.02168#S5.SS1.p2.1)\.
- \[26\]P\. Rodriguez, J\. Barrow, A\. Hoyle, J\. P\. Lalor, R\. Jia, and J\. Boyd\-Graber\(2021\)Evaluation examples are not equally informative: how should that change NLP leaderboards?\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 4486–4503\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.346)Cited by:[§4\.1](https://arxiv.org/html/2609.02168#S4.SS1.p1.1)\.
- \[27\]Y\. Ruan, H\. Dong, A\. Wang,et al\.\(2024\)Identifying the risks of LM agents with an LM\-emulated sandbox\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2309\.15817Cited by:[§2\.5](https://arxiv.org/html/2609.02168#S2.SS5.p1.1)\.
- \[28\]J\. B\. Sandbrink\(2023\)Artificial intelligence and biological misuse: differentiating risks of language models and biological design tools\.arXiv preprint arXiv:2306\.13952\.Cited by:[§2\.2](https://arxiv.org/html/2609.02168#S2.SS2.p1.1)\.
- \[29\]E\. H\. Soice, R\. Rocha, K\. Cordova, M\. Specter, and K\. M\. Esvelt\(2023\)Can large language models democratize access to dual\-use biotechnology?\.arXiv preprint arXiv:2306\.03809\.Cited by:[§2\.2](https://arxiv.org/html/2609.02168#S2.SS2.p1.1)\.
- \[30\]P\. Verga, S\. Hofstatter, S\. Althammer, Y\. Su, A\. Piktus, A\. Arkhangorodsky, M\. Xu, N\. White, and P\. Lewis\(2024\)Replacing judges with juries: evaluating LLM generations with a panel of diverse models\.arXiv preprint arXiv:2404\.18796\.Cited by:[§2\.4](https://arxiv.org/html/2609.02168#S2.SS4.p1.1)\.
- \[31\]B\. Vidgen, A\. Agrawal, A\. M\. Ahmed, V\. Akinwande, N\. Al\-Nuaimi, N\. Alfaraj,et al\.\(2024\)Introducing v0\.5 of the AI safety benchmark from MLCommons\.arXiv preprint arXiv:2404\.12241\.Cited by:[§2](https://arxiv.org/html/2609.02168#S2.p1.1)\.
- \[32\]S\. Vijayvargiya, A\. B\. Soni, X\. Zhou, Z\. Z\. Wang, N\. Dziri, G\. Neubig, and M\. Sap\(2026\)OpenAgentSafety: a comprehensive framework for evaluating real\-world AI agent safety\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2507\.06134Cited by:[§2\.2](https://arxiv.org/html/2609.02168#S2.SS2.p1.1)\.
- \[33\]B\. Wang, W\. Chen, H\. Pei, C\. Xie, M\. Kang, C\. Zhang, C\. Xu, Z\. Xiong, R\. Dutta, R\. Schaeffer, S\. T\. Truong, S\. Arora, M\. Mazeika, D\. Hendrycks, Z\. Lin, Y\. Cheng, S\. Koyejo, D\. Song, and B\. Li\(2023\)DecodingTrust: a comprehensive assessment of trustworthiness in GPT models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2306\.11698Cited by:[§2](https://arxiv.org/html/2609.02168#S2.p1.1)\.
- \[34\]Y\. Wang, H\. Li, X\. Han, P\. Nakov, and T\. Baldwin\(2024\)Do\-not\-answer: evaluating safeguards in LLMs\.InFindings of the Association for Computational Linguistics: EACL 2024,pp\. 896–911\.Note:arXiv:2308\.13387Cited by:[§2\.2](https://arxiv.org/html/2609.02168#S2.SS2.p1.1)\.
- \[35\]T\. Yuan, Z\. He, L\. Dong, Y\. Wang, R\. Zhao, T\. Xia, L\. Xu, B\. Zhou, F\. Li, Z\. Zhang, R\. Wang, and G\. Liu\(2024\)R\-Judge: benchmarking safety risk awareness for LLM agents\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Note:arXiv:2401\.10019Cited by:[§1](https://arxiv.org/html/2609.02168#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.02168#S2.SS2.p1.1)\.
- \[36\]Z\. Zhang, L\. Lei, L\. Wu, R\. Sun, Y\. Huang, C\. Long, X\. Liu, X\. Lei, J\. Tang, and M\. Huang\(2024\)SafetyBench: evaluating the safety of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 15537–15553\.Note:arXiv:2309\.07045Cited by:[§1](https://arxiv.org/html/2609.02168#S1.p1.1)\.
- \[37\]H\. Zhao, X\. Tang, Z\. Yang, X\. Han, X\. Feng, Y\. Fan, S\. Cheng, D\. Jin, Y\. Zhao, A\. Cohan, and M\. Gerstein\(2024\)ChemSafetyBench: benchmarking LLM safety on chemistry domain\.arXiv preprint arXiv:2411\.16736\.Cited by:[§5\.1](https://arxiv.org/html/2609.02168#S5.SS1.p2.1)\.
- \[38\]L\. Zheng, W\. Chiang, Y\. Sheng,et al\.\(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Note:arXiv:2306\.05685Cited by:[§2\.4](https://arxiv.org/html/2609.02168#S2.SS4.p1.1)\.
- \[39\]Y\. Zhou, J\. Yang, Y\. Huang, K\. Guo, Z\. Emory, B\. Ghosh, A\. Bedar, S\. Shekar, Z\. Liang, P\. Chen, T\. Gao, W\. Geyer, N\. Moniz, N\. V\. Chawla, and X\. Zhang\(2026\)LabSafety Bench: benchmarking LLMs on safety issues in scientific labs\.arXiv preprint arXiv:2410\.141828,pp\. 20–31\.External Links:[Document](https://dx.doi.org/10.1038/s42256-025-01152-1)Cited by:[§5\.1](https://arxiv.org/html/2609.02168#S5.SS1.p2.1)\.
- \[40\]A\. Zou, Z\. Wang, N\. Carlini, M\. Nasr, J\. Z\. Kolter, and M\. Fredrikson\(2023\)Universal and transferable adversarial attacks on aligned language models\.arXiv preprint arXiv:2307\.15043\.Cited by:[§2\.3](https://arxiv.org/html/2609.02168#S2.SS3.p1.1)\.

## Acknowledgments

We thank the anonymous reviewers for their constructive feedback\. This work was supported in part by the National Natural Science Foundation of China under Grant U21B2020\.

## Appendix

This appendix provides supporting material for reproducibility and review\. It contains, in order: the ethical considerations governing our dual\-use evaluation \(Appendix[Ethical](https://arxiv.org/html/2609.02168#Ax2)\), the artifact access policy \(Appendix[Open Science](https://arxiv.org/html/2609.02168#Ax3)\), the IRT measurement derivations used in the knowledge pipeline \(Section[4\.1](https://arxiv.org/html/2609.02168#S4.SS1)\), the correlation and bootstrap procedures underlying the cross\-judge analysis \(Section[6\.1](https://arxiv.org/html/2609.02168#S6.SS1)\), complete measurement tables omitted from the main text for space, and supplementary figures\.

## Ethical

This work involves dual\-use content at two levels\. First, the CB module contains scenario seeds and harm queries designed to elicit dangerous knowledge from LLMs\. Second, our evaluation demonstrates that several commercial models can be induced to produce actionable hazardous protocols across five escalating rounds, from initial refusal to complete operational guidance\. We address the resulting ethical obligations across six dimensions\.

Why CB in particular\.Chemical and biological hazards differ from other AI\-safety domains in a crucial respect: their misuse requires no digital infrastructure to cause harm\. A jailbroken model that generates a phishing email enables an attack that still must traverse digital defenses; a model that supplies synthesis conditions for a toxic compound shortens the path to physical\-world harm\. This asymmetry is why we chose CB as the first instantiation, and also why its assets carry stricter disclosure controls than, say, cyber scenario seeds\.

Public\-interest motivation\.This study is conducted in the public interest: to identify and quantify dangerous capabilities of deployed LLMs before they are exploited adversarially\. The framework measures general model capabilities rather than validating any specific harmful application; its output—a capability profile—serves regulators and platform operators making safety decisions\. Results are framed as capability measurements, never as operational guidance\.

Safeguards and compliance\.All experiments ran inside an isolated Docker sandbox with a kill\-switch mechanism \(Section[3\.1](https://arxiv.org/html/2609.02168#S3.SS1)\); no model outputs left the controlled environment, and no evaluation activity interacted with real systems, real users, or real chemical processes\. The Tool\-Enhanced prototype \(Section[3\.4](https://arxiv.org/html/2609.02168#S3.SS4)\) executed commands exclusively against locally hosted virtual targets\. API evaluations of closed\-source models used standard public interfaces, at academic\-usage volumes, without bypassing authentication, rate\-limiting, or safety\-layer mechanisms, in compliance with the reviewed terms of service of each provider\. No model output was used to train, fine\-tune, or otherwise improve any machine learning model; outputs were used solely for measurement and are not redistributed in the open data release\.

Minimization of harmful disclosure\.We report findings at the level of aggregate metrics and representative excerpts rather than releasing complete interaction logs\. The case figures in Sections[5\.3](https://arxiv.org/html/2609.02168#S5.SS3)and[5\.4](https://arxiv.org/html/2609.02168#S5.SS4)illustrate this policy in action: all operational parameters are redacted \(temperatures, durations, cutoffs, concentrations, thresholds\), pathogen and compound identities are softened to functional descriptions, and commercial reagent names are removed\. What remains is the measurement evidence itself— response lengths, escalation structure, and judge verdicts— which supports the paper’s quantitative claims without constituting operational guidance\.

Responsible disclosure\.The defense\-erosion patterns reported here describe publicly observable behaviors of commercial models queried through their own APIs\. The scenario\-seed design \(Section[3\.2](https://arxiv.org/html/2609.02168#S3.SS2)\) recombines elicitation strategies already documented in the public literature; we did not attempt to exploit any vulnerability beyond what the models themselves offered\. The evaluation measures, it does not amplify\.

Benefit–risk assessment and liability\.Systematic, reproducible measurement of dangerous capabilities is essential to effective LLM governance\. It gives regulators evidence, operators a basis for safeguards, and researchers a way to track progress\. Our temporal findings \(Section[5\.7](https://arxiv.org/html/2609.02168#S5.SS7)\) underscore the urgency: if hazardous knowledge accumulates while defenses stagnate, intervention becomes progressively harder\. These findings are published exclusively to advance AI safety research; any misuse is the sole responsibility of the misuser\.

## Open Science

We support the USENIX Security Open Science Policy\. The code and data are available upon reasonable request\.

We withhold three categories of dual\-use sensitive materials: full framework implementation \(internal infrastructure\), CB module assets \(scenario seeds, harm probes, MCQ bank\), and raw interaction logs\. Full disclosure would lower attack barriers; these are withheld under USENIX ethics guidelines\.

For reviewers, we provide a lightweight minimal reference implementation, covering the four\-module pipeline on non\-sensitive examples with all metric computation logic\. Reviewers can verify the paper’s numbers against primary data\. The methodology is fully specified in Sections[3](https://arxiv.org/html/2609.02168#S3)–[4](https://arxiv.org/html/2609.02168#S4), sufficient for third\-party reproduction ofϕ\\phiaggregation and statistical analyses from the released materials\.

## IRT Measurement Derivation

Item difficulty calibration\.Given the 2PL model with guessing parameter0\.250\.25:

Pi=0\.25\+0\.75​Ψi,Ψi=\[1\+exp⁡\(−\(θ−bi\)\)\]−1,P\_\{i\}=0\.25\+0\.75\\,\\Psi\_\{i\},\\qquad\\Psi\_\{i\}=\\bigl\[1\+\\exp\(\-\(\\theta\-b\_\{i\}\)\)\\bigr\]^\{\-1\},we calibratebib\_\{i\}by evaluating atθ=0\\theta=0and equating to the pooled pass\-ratepip\_\{i\}:

pi=0\.25\+0\.751\+exp⁡\(bi\)\\displaystyle p\_\{i\}=0\.25\+\\frac\{0\.75\}\{1\+\\exp\(b\_\{i\}\)\}Subtract0\.250\.25:pi−0\.25=0\.751\+exp⁡\(bi\)\\displaystyle p\_\{i\}\-0\.25=\\frac\{0\.75\}\{1\+\\exp\(b\_\{i\}\)\}Invert both sides:1\+exp⁡\(bi\)=0\.75pi−0\.25\\displaystyle 1\+\\exp\(b\_\{i\}\)=\\frac\{0\.75\}\{p\_\{i\}\-0\.25\}Solve forexp⁡\(bi\)\\exp\(b\_\{i\}\):exp⁡\(bi\)=0\.75pi−0\.25−1=1−pipi−0\.25\\displaystyle\\exp\(b\_\{i\}\)=\\frac\{0\.75\}\{p\_\{i\}\-0\.25\}\-1=\\frac\{1\-p\_\{i\}\}\{p\_\{i\}\-0\.25\}Take logarithms:bi=log⁡1−pipi−0\.25\.\\displaystyle b\_\{i\}=\\log\\frac\{1\-p\_\{i\}\}\{p\_\{i\}\-0\.25\}\.
Maximum\-likelihood estimation ofθ\\theta\.Withbib\_\{i\}fixed, the log\-likelihood for a response vector\{xi,T\}\\\{x\_\{i,T\}\\\}is

ℓ⁡\(θ\)=∑i\[xi,T​log⁡Pi​\(θ\)\+\(1−xi,T\)​log⁡\(1−Pi​\(θ\)\)\]\\displaystyle\\ell\(\\theta\)=\\sum\_\{i\}\\Bigl\[x\_\{i,T\}\\log P\_\{i\}\(\\theta\)\+\(1\-x\_\{i,T\}\)\\log\(1\-P\_\{i\}\(\\theta\)\)\\Bigr\]The first and second derivatives areℓ′​\(θ\)=∑i\(xi,T−Pi\)​Pi′​\(θ\)Pi​\(θ\)​\(1−Pi​\(θ\)\)\\displaystyle\\ell^\{\\prime\}\(\\theta\)=\\sum\_\{i\}\\frac\{\(x\_\{i,T\}\-P\_\{i\}\)\\,P\_\{i\}^\{\\prime\}\(\\theta\)\}\{P\_\{i\}\(\\theta\)\(1\-P\_\{i\}\(\\theta\)\)\}ℓ′′​\(θ\)=∑i\[\(xi,T−Pi\)​Pi′′​\(θ\)Pi​\(1−Pi\)−\(xi,T−Pi\)​\(Pi′\)2​\(1−2​Pi\)Pi2​\(1−Pi\)2\]\\displaystyle\\ell^\{\\prime\\prime\}\(\\theta\)=\\sum\_\{i\}\\Biggl\[\\frac\{\(x\_\{i,T\}\-P\_\{i\}\)\\,P\_\{i\}^\{\\prime\\prime\}\(\\theta\)\}\{P\_\{i\}\(1\-P\_\{i\}\)\}\-\\frac\{\(x\_\{i,T\}\-P\_\{i\}\)\\,\(P\_\{i\}^\{\\prime\}\)^\{2\}\(1\-2P\_\{i\}\)\}\{P\_\{i\}^\{2\}\(1\-P\_\{i\}\)^\{2\}\}\\Biggr\]wherePi′P\_\{i\}^\{\\prime\}andPi′′P\_\{i\}^\{\\prime\\prime\}denote derivatives with respect toθ\\theta\. Newton–Raphson updates are performed asθ←θ−ℓ′​\(θ\)ℓ′′​\(θ\)\\displaystyle\\theta\\leftarrow\\theta\-\\frac\{\\ell^\{\\prime\}\(\\theta\)\}\{\\ell^\{\\prime\\prime\}\(\\theta\)\}until convergence to obtainθ^T\\hat\{\\theta\}\_\{T\}\.

Fisher information\.LetΨi=\[1\+exp⁡\(−\(θ−bi\)\)\]−1\\Psi\_\{i\}=\\bigl\[1\+\\exp\(\-\(\\theta\-b\_\{i\}\)\)\\bigr\]^\{\-1\}, hencePi=0\.25\+0\.75​ΨiP\_\{i\}=0\.25\+0\.75\\,\\Psi\_\{i\}\. The derivative of the item response function is

∂Pi∂θ\\displaystyle\\frac\{\\partial P\_\{i\}\}\{\\partial\\theta\}=0\.75​Ψi​\(1−Ψi\)\\displaystyle=0\.75\\,\\Psi\_\{i\}\(1\-\\Psi\_\{i\}\)=0\.75⋅Pi−0\.250\.75⋅1−Pi0\.75\\displaystyle=0\.75\\cdot\\frac\{P\_\{i\}\-0\.25\}\{0\.75\}\\cdot\\frac\{1\-P\_\{i\}\}\{0\.75\}=\(Pi−0\.25\)​\(1−Pi\)0\.75\.\\displaystyle=\\frac\{\(P\_\{i\}\-0\.25\)\(1\-P\_\{i\}\)\}\{0\.75\}\.
Using the standard item\-information identityIi​\(θ\)=\(∂Pi/∂θ\)2/\(Pi​\(1−Pi\)\)I\_\{i\}\(\\theta\)=\(\\partial P\_\{i\}/\\partial\\theta\)^\{2\}/\(P\_\{i\}\(1\-P\_\{i\}\)\), the test information at the estimated abilityθT\\theta\_\{T\}is

I⁡\(θT\)\\displaystyle I\(\\theta\_\{T\}\)=∑i\(\(Pi,T−0\.25\)​\(1−Pi,T\)0\.75\)2Pi,T​\(1−Pi,T\)\\displaystyle=\\sum\_\{i\}\\frac\{\\left\(\\frac\{\(P\_\{i,T\}\-0\.25\)\(1\-P\_\{i,T\}\)\}\{0\.75\}\\right\)^\{2\}\}\{P\_\{i,T\}\(1\-P\_\{i,T\}\)\}=∑i\(Pi,T−0\.25\)2​\(1−Pi,T\)20\.752​Pi,T​\(1−Pi,T\)\\displaystyle=\\sum\_\{i\}\\frac\{\(P\_\{i,T\}\-0\.25\)^\{2\}\(1\-P\_\{i,T\}\)^\{2\}\}\{0\.75^\{2\}\\,P\_\{i,T\}\(1\-P\_\{i,T\}\)\}=∑i\(\(Pi,T−0\.25\)​\(1−Pi,T\)\)20\.752​Pi,T​\(1−Pi,T\)\.\\displaystyle=\\sum\_\{i\}\\frac\{\\bigl\(\(P\_\{i,T\}\-0\.25\)\(1\-P\_\{i,T\}\)\\bigr\)^\{2\}\}\{0\.75^\{2\}\\,P\_\{i,T\}\(1\-P\_\{i,T\}\)\}\.
Finally, the standard error of the ability estimate is

SE\(θT\)=I\(θT\)−1/2\.SE\(\\theta\_\{T\}\)=I\(\\theta\_\{T\}\)^\{\-1/2\}\.

## Correlation and Bootstrap Procedure

Spearman rank correlation\.We measure ranking agreement between two judges via Spearman’sρ\\rho\. Define the ranks asri\(1\)r^\{\(1\)\}\_\{i\}andri\(2\)r^\{\(2\)\}\_\{i\}for scenarioii\. The computation proceeds as:

r¯\(1\)=1N​∑i=1Nri\(1\),r¯\(2\)=1N​∑i=1Nri\(2\)\\displaystyle\\bar\{r\}^\{\(1\)\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}r^\{\(1\)\}\_\{i\},\\qquad\\bar\{r\}^\{\(2\)\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}r^\{\(2\)\}\_\{i\}The numerator \(covariance of ranks\) is𝒩=∑i=1N\(ri\(1\)−r¯\(1\)\)​\(ri\(2\)−r¯\(2\)\)\\displaystyle\\mathcal\{N\}=\\sum\_\{i=1\}^\{N\}\\bigl\(r^\{\(1\)\}\_\{i\}\-\\bar\{r\}^\{\(1\)\}\\bigr\)\\bigl\(r^\{\(2\)\}\_\{i\}\-\\bar\{r\}^\{\(2\)\}\\bigr\)and the denominator is the product of standard deviations𝒟=∑i=1N\(ri\(1\)−r¯\(1\)\)2⋅∑i=1N\(ri\(2\)−r¯\(2\)\)2\\displaystyle\\mathcal\{D\}=\\sqrt\{\\sum\_\{i=1\}^\{N\}\\bigl\(r^\{\(1\)\}\_\{i\}\-\\bar\{r\}^\{\(1\)\}\\bigr\)^\{2\}\}\\cdot\\sqrt\{\\sum\_\{i=1\}^\{N\}\\bigl\(r^\{\(2\)\}\_\{i\}\-\\bar\{r\}^\{\(2\)\}\\bigr\)^\{2\}\}Thusρ=𝒩𝒟\.\\displaystyle\\rho=\\frac\{\\mathcal\{N\}\}\{\\mathcal\{D\}\}\.
The coefficientρ\\rhois invariant to monotone rescaling of either judge’s scores, making it appropriate for comparing judges with different absolute score distributions\.

Stratified sampling\.Cross\-judge re\-scoring uses a stratified subsample from the 298\-scenario defense pool\.

Bootstrap confidence intervals\.To quantify sampling uncertainty ofρ\\rhoon the 90\-scenario sample, we use nonparametric bootstrap withB=1,000B=1\{,\}000replications\.

Step 1 \(resampling\): For eachk=1,…,Bk=1,\\dots,B, draw a bootstrap sample𝒮k∗\\mathcal\{S\}\_\{k\}^\{\*\}of size 90 with replacement from the original 90 scenarios\. Compute the Spearman correlation on𝒮k∗\\mathcal\{S\}\_\{k\}^\{\*\}, denoteρk\\rho\_\{k\}

Step 2 \(point estimate\): Take the mean of the bootstrap distribution

ρ^=1B​∑k=1Bρk\.\\displaystyle\\widehat\{\\rho\}=\\frac\{1\}\{B\}\\sum\_\{k=1\}^\{B\}\\rho\_\{k\}\.

Step 3 \(interval estimate\): The 95% percentile confidence interval is

C​I95=\[ρ\(⌊0\.025​B⌋\),ρ\(⌊0\.975​B⌋\)\]\\displaystyle CI\_\{95\}=\\bigl\[\\rho\_\{\(\\lfloor 0\.025B\\rfloor\)\},\\;\\rho\_\{\(\\lfloor 0\.975B\\rfloor\)\}\\bigr\],

whereρ\(α\)\\rho\_\{\(\\alpha\)\}denotes the empiricalα\\alpha\-quantile of the sorted bootstrap sample\.

The same bootstrap procedure underlies the harm\-dimension re\-scoring \(360 scored responses\) and the per\-model consistency tables in the supplementary figures\.

## Complete Measurement Tables

Table 10:Complete dangerous capability profilesϕ\\phifor all 12 models\.Table 11:IRT ability estimatesθ±S​E\\theta\\pm SEfor all 24 model\-subdomain combinations\.Table[12](https://arxiv.org/html/2609.02168#Ax6.T12)reports the complete pairwise Spearman matrix for the defense dimension; the harm\-dimension matrix is reported in the released data\.

Table 12:Complete dim1 cross\-judge Spearman matrix \(n=90n=90stratified scenarios\)\.
## Supplementary Figures

This section collects visualizations supporting the main text\. The IRT test information curve visualizes where the knowledge bank measures most precisely \(Section[6\.3](https://arxiv.org/html/2609.02168#S6.SS3)\)\. The temporal trajectories extend the headline evolution analysis \(Section[5\.7](https://arxiv.org/html/2609.02168#S5.SS7.SSSx3)\) to the derived indicatorsD​LDL,B​RBR, andE​RER\. The cyber defense profile decomposes the pilot results \(Section[5\.6](https://arxiv.org/html/2609.02168#S5.SS6)\) to per\-scenario level\. The per\-model judge consistency heatmaps complement the pooled matrices reported in Section[6\.1](https://arxiv.org/html/2609.02168#S6.SS1)\.

![Refer to caption](https://arxiv.org/html/2609.02168v1/fig_irt_info_curve.png)Figure 11:IRT test information functionI⁡\(θ\)I\(\\theta\)for the 3,773\-item knowledge bank\. The curve peaks atθ≈−0\.4\\theta\\approx\-0\.4, indicating that the item bank provides maximum measurement precision near the L3 \(discern\) difficulty level—the region most critical for distinguishing mid\-range hazardous knowledge capabilities\.Figure 12:Temporal evolution of the derived defense indicators: \(a\) Defense LevelD​LDLrises within every family while DeepSeek plateaus below0\.750\.75; \(b\) Breakthrough RatioB​RBRagainst theB​R=1BR=1stability line—DeepSeek consistently exceeds it, GPT\-5\.x dips below it, and Claude tracks it closely\.Figure 13:Temporal evolution of the empty\-response rateE​RER\. Silent refusal emerges with Claude\-Opus\-4 \(67\.8%\) in mid\-2025, consolidates above78%78\\%in Opus\-4\.5/4\.8, and remains absent from DeepSeek and GPT\-4o\-class models; GPT\-5\.x adopts a partial strategy \(1515–28%28\\%\)\.Figure 14:Per\-scenario defense scores \(DD\) across the 16\-scenario cyber attack chain\. GPT\-4o alternates between full defense \(D=100D=100on SQLi, exfiltration, persistence, and brute\-force scenarios\) and minimal defense \(D=35D=35on XSS, SSTI, and JWT scenarios\)—a scenario\-dependence invisible in its mean \(D=61\.2D=61\.2\)\. DeepSeek\-V3 and GPT\-3\.5\-Turbo defend weakly across nearly the entire chain, with isolated spikes at Persistence Backdoor and Brute Force Admin respectively\.![Refer to caption](https://arxiv.org/html/2609.02168v1/fig_model_consistency_D.png)Figure 15:Model\-level judge consistency for Defense \(DD\)\. Spearmanρ\\rhobetween each alternative judge and the primary judge \(GPT\-4o\-mini\)\. Darker cells indicate higher per\-model agreement\.![Refer to caption](https://arxiv.org/html/2609.02168v1/fig_model_consistency_H.png)Figure 16:Model\-level judge consistency for Harm \(HH\)\. Spearmanρ\\rhobetween each alternative judge and the primary judge \(GPT\-4o\-mini\)\.