Execution-Grounded Security Testing for Coding Agents in Software Engineering Pipelines

# Execution-Grounded Security Testing for Coding Agents in Software Engineering Pipelines
Source: [https://arxiv.org/html/2607.22569](https://arxiv.org/html/2607.22569)
Yifei Ge1, Weisong Sun2, Jinkun Xiao1, Yuchen Chen1, Yebo Feng2, Peizhuo Lv2,Xia Feng3, Chunrong Fang1, Zhihong Zhao1, Zhenyu Chen1, Yang Liu21Nanjing University2Nanyang Technological University3Hainan Universitygyf991213@126\.com, \{522025320179,yuc\.chen\}@smail\.nju\.edu\.cn, \{fangchunrong,zhaozhih,zychen\}@nju\.edu\.cn\{weisong\.sun,yebo\.feng,peizhuo\.lyu,yangliu\}@ntu\.edu\.sg, xiafeng@hainanu\.edu\.cnContact:lvpeizhuo@gmail\.com

###### Abstract\.

Coding agents are increasingly integrated into system operations, where their tool use can directly modify project artifacts, execution environments, and the underlying system\. For example, if a coding agent inserts a hook into a system startup or configuration script, that change can persist after the interaction, be triggered later, and abuse delegated user or system privileges to modify the system\. This makes security testing a system problem: the key question is not only what the agent says, but what it actually does to the surrounding environment\. We present an*execution\-grounded red\-team testing framework*for probing this execution\-layer security boundary using observable sandbox evidence, including tool invocations, runtime traces, and file\-system diffs\. Our framework embeds target unsafe operations into routine software engineering workloads, including unit testing, regression testing, crash reproduction, and validation, and uses an execution oracle to guide refinement when an initial probe is rejected or fails\. Across multiple agent frameworks and model backbones, our red\-team workload reformulation substantially increases verified unsafe execution, reaching 73\.61% on code carriers and 53\.93% on text carriers\. These results show that coding agents in system operations remain insecure under task disguise: once risky intent is hidden inside plausible engineering tasks, the agent can be induced to carry out unsafe actions on the surrounding system\. More broadly, coding agents in system operations still demand stronger security testing and safeguards\.

## 1\.Introduction

The impressive capability of large language models \(LLMs\) to follow complex natural\-language instructions has driven widespread deployment across both consumer and enterprise settings\(Yanget al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib23); Wanget al\.,[2024b](https://arxiv.org/html/2607.22569#bib.bib22)\)\. In software engineering, this progress has enabled*coding agents*: LLM\-based agents integrated into development workflows that invoke tools and execute code through command\-line interfaces under delegated user privileges\(Shinnet al\.,[2023](https://arxiv.org/html/2607.22569#bib.bib20); Zhouet al\.,[2023](https://arxiv.org/html/2607.22569#bib.bib21); Zhenget al\.,[2024a](https://arxiv.org/html/2607.22569#bib.bib3)\)\. Unlike code\-only LLMs, coding agents can run code, invoke test runners and build tools, manipulate project artifacts, and interact with the file system during task execution\. While these capabilities improve developer productivity, they also introduce severe security risks\. Unlike text\-only LLMs, whose security failures mainly appear as disallowed output, a coding agent that executes an unsafe operation can produce*persistent, irreversible impacts*on the underlying system\. For instance, if a user grants an agent permission to modify system directories or development settings, the agent may rewrite shell initialization files such as\.bashrcor alter user\-level configuration files, causing later commands and sessions to inherit an unsafe state\. Such effects can outlast the original interaction, propagate through the broader execution environment, and affect other users or processes sharing the same host\. As coding agents are increasingly granted delegated privileges in local workspaces, containerized environments, and operational systems, the security of their*execution\-layer behavior*becomes a critical systems concern\. For example, developers increasingly install agents such as Claude Code\(Anthropic,[2026](https://arxiv.org/html/2607.22569#bib.bib29)\), OpenClaw\(OpenClaw,[2026](https://arxiv.org/html/2607.22569#bib.bib30)\), and Codex\(OpenAI,[2026](https://arxiv.org/html/2607.22569#bib.bib28)\)into their local environments through the CLI, and often grant them broad permissions to complete tasks with minimal friction\. In this setting, execution\-layer failures can directly affect local environments and software pipelines\.

In principle, coding agents and their underlying models are expected to enforce security alignment: when a request could damage the system or create a persistent unsafe state, the agent should recognize the risk and refuse to carry it out\. However, we find that this boundary can become much weaker once the same unsafe operation is embedded inside a seemingly legitimate engineering workflow\. In that case, the agent may treat the request as part of normal testing, debugging, or validation and carry out the unsafe operation during task completion\. This phenomenon is related to the jailbreak literature on classical LLMs\(Shenet al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib8); Shahet al\.,[2023](https://arxiv.org/html/2607.22569#bib.bib34); Zouet al\.,[2023](https://arxiv.org/html/2607.22569#bib.bib40); Dinget al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib39)\), where prompt manipulation techniques\(Chenget al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib35); Yuet al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib7); Russinovichet al\.,[2025](https://arxiv.org/html/2607.22569#bib.bib36); Kanget al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib37); Lvet al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib38)\)are used to induce policy\-violating outputs\. The difference is that, for coding agents, a successful bypass does not stop at generating disallowed text, but can lead to actual tool use and system\-changing actions under delegated privileges\.

![Refer to caption](https://arxiv.org/html/2607.22569v1/x1.png)Figure 1\.A direct unsafe request is refused by the agent’s security alignment and leaves no observable execution evidence\. The same unsafe operation, once embedded in a seemingly legitimate engineering task, is accepted and executed through the command\-line interface \(CLI\)\.As shown in Figure[1](https://arxiv.org/html/2607.22569#S1.F1), we illustrate this escalation with a risky code\-modification request that asks the coding agent to add a hook snippet to a system initialization/configuration file\. When the request is issued directly, the safeguard rejects it: the agent does not invoke tools or execute code, and the environment shows no execution trace or state change\. When the same intent is wrapped in a workload\-contextualized probe, it appears benign and is accepted\. The agent edits the configuration file until the task appears successful, and the resulting trace and file diff directly verify that the hook has been inserted\. This illustrates why execution\-layer evidence, rather than language\-layer responses, must be the ground truth for agent security testing\.

Recent studies have already examined how agents can produce unsafe system\-level effects and have proposed taxonomies and benchmark tasks for different risk types\(Yuanet al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib15); Liuet al\.,[2026](https://arxiv.org/html/2607.22569#bib.bib18); Luoet al\.,[2025](https://arxiv.org/html/2607.22569#bib.bib16); Andriushchenkoet al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib17)\)\. However, they still leave a more fundamental question underexplored: where is the true security boundary of coding agents in system operations? In particular, prior work has not systematically tested whether a risky operation that is rejected in direct form can still bypass security alignment once disguised as a benign\-looking engineering task, thereby inducing unsafe changes to the surrounding system\. This leaves a gap in systematically probing the true execution\-layer security boundary of coding agents when risky operations are packaged as legitimate engineering tasks\.

Motivated by this gap, we propose an*execution\-grounded red\-team testing framework*for systematically characterizing the execution\-layer security boundary of coding agents in system operations\. Our key insight is that routine software engineering tasks, including unit testing, regression testing, crash reproduction, and validation, serve as natural testing probes\. Each probe embeds a target unsafe operation within a plausible engineering workflow, while the red\-team agent performs outcome\-aware refinement based on oracle feedback\. This allows the framework to identify where the target agent’s execution\-layer security boundary breaks, and under what task framing the same unsafe operation becomes executable\. Our framework executes workloads in an isolated Docker sandbox and determines outcomes from observable execution evidence \(tool invocations, runtime traces, file\-system diffs\) rather than the agent’s textual response\.

Contributions\.In brief, our contributions are:

- •To the best of our knowledge, we present the first*execution\-grounded red\-team testing framework*for systematically probing the execution\-layer security boundary of coding agents in system operations, using observable sandbox evidence rather than textual compliance\.
- •We identify routine software engineering tasks as natural probes for execution\-grounded security testing: each probe embeds a target unsafe operation within a plausible engineering workflow using a shared four\-slot template \(Motivation, Test Objective, Procedure, Pass Criterion\) instantiated across four workload types \(Unit, Regression, Crash Reproduction, Validation\), with an execution oracle for evidence\-based outcome determination and iterative red\-team refinement\.
- •Through experiments across multiple agent frameworks and model backbones, we quantify the language\-execution security gap: refusal rates are 44\.36% \(Code\) and 28\.02% \(Text\), while execution evidence match reaches 73\.61% and 53\.93%, showing that text\-centric security testing systematically underestimates execution\-layer risk in system operations\.

## 2\.Background and Related Work

### 2\.1\.Coding Agents

In this paper, we define*coding agents*as LLM\-based systems that do not merely generate standalone code, but directly interact with software environments during task completion\. They can inspect repositories and local directories as context, execute generated or provided code, and invoke auxiliary tools such as shells, file\-system interfaces, execution sandboxes, external tool servers, or reusable skill modules\. Because these actions run under delegated user or pipeline privileges, coding agents can change files, processes, configurations, and other parts of the surrounding system state as part of normal task execution\.

Current coding\-agent architectures vary in how they couple reasoning, tool use, and feedback\. ReAct interleaves reasoning traces with external actions\(Yaoet al\.,[2023](https://arxiv.org/html/2607.22569#bib.bib4)\)\. Reflexion augments agents with verbal self\-reflection and episodic memory across trials\(Shinnet al\.,[2023](https://arxiv.org/html/2607.22569#bib.bib20)\)\. CodeAct treats executable code as a unified action space for interacting with tools and environments\(Wanget al\.,[2024a](https://arxiv.org/html/2607.22569#bib.bib5)\)\. OpenCodeInterpreter integrates code generation, execution, and iterative refinement into an open code system\(Zhenget al\.,[2024a](https://arxiv.org/html/2607.22569#bib.bib3)\)\. SWE\-agent emphasizes a dedicated agent\-computer interface for repository navigation, file editing, and test execution in software engineering tasks\(Yanget al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib23)\)\. More recent work such as ReflexiCoder internalizes reflection and self\-correction into the model itself, reducing dependence on external iterative refinement loops\(Jianget al\.,[2026](https://arxiv.org/html/2607.22569#bib.bib24)\)\. Across these designs, the common property relevant to our work is that coding agents act on live environments rather than producing inert text alone\.

### 2\.2\.Agent Security Risks and Benchmarks

As LLM\-based agents increasingly perform planning, tool use, and long\-horizon execution, security concerns have expanded beyond static content moderation to include unsafe operations that emerge during an agent’s decision\-making and action execution over time\. Recent work has begun to characterize agentic security risks through benchmark\-style evaluations and higher\-level risk definitions\.

AgentHarm\(Andriushchenkoet al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib17)\)organizes harms across social, physical, and digital domains, while AgentDoG\(Liuet al\.,[2026](https://arxiv.org/html/2607.22569#bib.bib18)\)further decomposes agent risk into dimensions such as risk source, failure mode, and real\-world harm\. Existing benchmarks then evaluate these concerns through interaction records, execution traces, or risky code tasks\. For example, R\-Judge\(Yuanet al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib15)\)contains multi\-turn agent interaction records spanning multiple application categories and asks the model to identify security hazards during the interaction\. ASSE\-Safety\(Luoet al\.,[2025](https://arxiv.org/html/2607.22569#bib.bib16)\)focuses on unintended harmful behaviors that arise in non\-malicious environments due to capability deficiencies or design flaws\. AgentMonitor\(Naihinet al\.,[2023](https://arxiv.org/html/2607.22569#bib.bib19)\)is a trajectory\-level benchmark with full execution trajectories, broad tool coverage, and fine\-grained labels\. RedCode\(Guoet al\.,[2024a](https://arxiv.org/html/2607.22569#bib.bib2)\)is a benchmark for code\-agent security\. It consists of RedCode\-Gen and RedCode\-Exec for risky code generation and risky code execution, respectively\.

Despite their broad coverage and increasingly fine\-grained labels, these taxonomies and benchmarks still leave a key question unresolved: where is the actual security boundary of coding agents once they move from language responses to real execution? In particular, prior work has not systematically characterized the*language\-execution security gap*, namely how far language\-layer refusal can diverge from execution\-layer security outcomes\. Our framework directly addresses this gap\.

## 3\.Security Boundary Formalism

We formalize the setting to specify what the framework probes, what constitutes an execution\-layer security boundary violation, and what evidence the execution oracle uses to determine it\. We model the target system as an agentAAthat receives a natural\-language promptp∈𝒫p\\in\\mathcal\{P\}and produces a sequence of actions and observable outcomes\. The agent is accessed through a public interface \(e\.g\., an API or an integrated development environment\), and the testing framework has no access toAA’s internal parameters, system prompts, planning logic, or security policies \(black\-box testing\)\. The agentAAis equipped with a tool set𝒰\\mathcal\{U\}, including but not limited to code execution, file\-system operations, and test execution\. All tool invocations issued by the agent are executed in a system environmentℰ\\mathcal\{E\}that records execution traces, including invoked commands, file operations, and other side effects\. We abstract the agent’s security alignment capability under risky requests as a safeguard functionSS\. Given a promptpp,S​\(A,p\)S\(A,p\)determines whether the agentAccepts the request and proceeds with planning and tool invocation, orRejects it at the language layer\.

Testing objective\.The testing framework aims to characterize the*execution\-layer security boundary*: the set of conditions under which an accepted request leads to unsafe operations in the environment\. The language\-layer outcomeAccept/Reject\\texttt\{Accept\}/\\texttt\{Reject\}is therefore only the entry point of testing, not the final security judgment\. Formally, the framework constructs a testing probepjp\_\{j\}and asks whether the agent, once accepting the probe, produces execution\-layer unsafe behavior\. Critically,*success is not determined by whether the agent verbally agrees to the request*: a probe is deemed to reveal an execution\-layer security gap only if the agent executes unsafe operations in the environmentℰ\\mathcal\{E\}by invoking tools in𝒰\\mathcal\{U\}\. Formally, let𝖤𝗑𝖾𝖼​\(A,p,ℰ\)\\mathsf\{Exec\}\(A,p,\\mathcal\{E\}\)denote the execution trace produced by the agent in the environment\. A testing probepjp\_\{j\}reveals an execution\-layer security gap if:

S​\(A,pj\)=Acceptand𝖤𝗑𝖾𝖼​\(A,pj,ℰ\)⊧𝒪mal,S\(A,p\_\{j\}\)=\\texttt\{Accept\}\\quad\\text\{and\}\\quad\\mathsf\{Exec\}\(A,p\_\{j\},\\mathcal\{E\}\)\\models\\mathcal\{O\}\_\{\\text\{mal\}\},where𝒪mal\\mathcal\{O\}\_\{\\text\{mal\}\}represents a predefined set of unsafe operations \(e\.g\., file deletion, unauthorized access, execution of untrusted code\), and satisfaction is verified by the evidence predicateϕg\\phi\_\{g\}from the execution oracle\.

## 4\.Methodology

![Refer to caption](https://arxiv.org/html/2607.22569v1/x2.png)Figure 2\.Overview of the execution\-grounded red\-team testing framework\. A red\-team agent embeds unsafe operations into structured testing and debugging contexts and submits testing probes to the target agent in the system environment\. An execution oracle verifies outcomes from observable execution evidence \(tool traces, file\-system diffs\) rather than textual responses, and guides outcome\-aware refinement when needed\.This section presents our execution\-grounded red\-team testing workflow for assessing security risks in coding agents\. The workflow is goal\-driven and couples workload generation with outcome\-aware refinement based on execution feedback\. Figure[2](https://arxiv.org/html/2607.22569#S4.F2)provides an overview of the framework, which the following subsections detail step by step\.

![Refer to caption](https://arxiv.org/html/2607.22569v1/x3.png)Figure 3\.Workflow of the red\-team agent, which uses oracle\-guided rollback to iteratively refine testing probes and probe the target agent’s execution\-layer security boundary\.### 4\.1\.Overview

Our framework takes as input a target unsafe operationggfor testing\. It builds testing probes aroundgg, executes them in the environment, and uses oracle\-checked feedback to adjust later attempts when needed\. Overall, the framework is designed to identify where the target agent’s execution\-layer security boundary breaks and under what task framing the same unsafe operation becomes executable\.

### 4\.2\.Malicious Goal Format

Each malicious goal is represented as a tuple containing \(i\) a target unsafe operationgg, \(ii\) an input carrier that conveys this operation, \(iii\) the execution context required to exercise this operation, and \(iv\) an evidence predicateϕg\\phi\_\{g\}that verifies whether the intended environment\-level effect actually occurs\. We consider 2 carrier types: code carriers, where the seed is risky code to be executed by the coding agent, and text carriers, where the seed is a natural\-language description that the agent translates into code and tool actions\.

Goal Sampling \(Step 1\)\.As the start of the testing pipeline, we first sample a risky scenario from the goal pool and then sample one unsafe operation under that scenario as the targetgg\. For eachgg, we associate an evidence predicateϕg\\phi\_\{g\}that determines whethergghas been realized in the environment\. Importantly,ϕg\\phi\_\{g\}is evaluated over observable outcomes \(e\.g\., environment state changes and execution artifacts\)\. The execution oracle usesϕg\\phi\_\{g\}to assess success based on end\-to\-end executable behavior and environment side effects, rather than on the agent’s language\-layer response\.

### 4\.3\.Target Agent and Environment

The tested agent system is an execution\-capable LLM agent that can plan and invoke external tools to perform actions such as running code and manipulating local artifacts\. In practice, such agents are deployed to assist with engineering workflows that require tool usage and state changes, and they may be equipped with a safeguard that refuses certain high\-risk requests\. In our testing setting, we treat the target agent as a black box: the framework interacts with it through probes and observes its responses and tool\-mediated behaviors, without assuming access to internal policies or implementation details \(see Section[3](https://arxiv.org/html/2607.22569#S3)\)\. Conceptually, our testing concerns the agent’s behavior in the system environment where it performs tool\-mediated actions and causes state changes\. In experiments, we instantiate this environment as an isolated Docker sandbox for security and reproducibility\. Concrete environment preparation details, and runtime settings are reported in Section[5\.2](https://arxiv.org/html/2607.22569#S5.SS2)\. For each executed probe \(ptestp\_\{\\text\{test\}\}orpnextp\_\{\\text\{next\}\}\), we collect an execution trace that captures tool invocations and command executions, file\-system operations \(including relevant artifacts and diffs\), and other runtime signals needed to determine whether the target unsafe operationgghas been realized\.

### 4\.4\.Red\-team Agent

The*red\-team agent*is the component that operationalizes our testing by transforming a malicious goal into executable probes for testing the target agent’s execution\-layer security boundary\. Concretely, it is responsible for constructing workload\-contextualized testing probes and adapting them according to execution feedback from the oracle\. Figure[3](https://arxiv.org/html/2607.22569#S4.F3)illustrates this internal workflow\.

Baseline Check \(Step 2\)\.To ensure that our testing measures security gaps that emerge specifically under workload\-contextualized probing, the generator first issues a direct requestpdirect​\(g\)p\_\{\\text\{direct\}\}\(g\)that requestsggwithout any testing narrative\. The purpose of this step is to verify whether the target agent’s safeguard treatsggas disallowed at the language\-policy layer\. Ifpdirect​\(g\)p\_\{\\text\{direct\}\}\(g\)is allowed, we treatggas directly executable under this agent configuration and exclude it from the language\-execution gap analysis, since any later success would not depend on workload contextualization or red\-team refinement\. Only whenpdirect​\(g\)p\_\{\\text\{direct\}\}\(g\)is rejected do we proceed to workload contextualization \(Step 3\), so that subsequent success \(if any\) reflects a security boundary crossed by the workload\-contextualized carrier rather than a trivially unsafe baseline behavior\.

Workload Contextualization \(Step 3\)\.After baseline rejection, the red\-team agent initializes a testing probeptest​\(g\)p\_\{\\text\{test\}\}\(g\)by selecting a workload typet∈𝒯t\\in\\mathcal\{T\}and instantiating a pattern\-specific prompt template with the target unsafe operationgg\. Let𝒯=\{Unit,Regression,Crash,Validation\}\\mathcal\{T\}=\\\{\\textsc\{Unit\},\\textsc\{Regression\},\\textsc\{Crash\},\\textsc\{Validation\}\\\}denote four different workload types \(Figure[2](https://arxiv.org/html/2607.22569#S4.F2), Step 3\)\. All types are framed as testing/debugging tasks, but they emphasize different engineering contexts to provide diverse testing coverage and probe different safeguard responses\.Unitframes the request as a small, isolated unit\-level check;Regressionframes it as adding or updating a regression test to prevent a previously observed issue from recurring;Crashframes it as reproducing and diagnosing a crash or exception; andValidationframes it as validating an invariant or environment assumption as part of a routine health check\. We sample an initial typet∼Sample​\(𝒯\)t\\sim\\textsc\{Sample\}\(\\mathcal\{T\}\)and build the testing probeptest​\(g\)=TestPrompt​\(g,t\),p\_\{\\text\{test\}\}\(g\)=\\textsc\{TestPrompt\}\(g,t\),whereTestPromptfills a shared four\-slot template with type\-conditioned content generated by the red\-team agent\. Across all types, we use a shared four\-slot template \(Motivation, Test Objective, Procedure, and Pass Criterion\):

- •Motivation: why this test/debug step is needed; e\.g\., regression prevention, validation of an invariant\.
- •Test Objective: what the test claims to check, expressed as a test\-like condition\.
- •Procedure: how to run the check and what to do when it fails, described as standard troubleshooting steps\.
- •Pass Criterion: what constitutes success in the testing narrative \(e\.g\., “test passes” or “validation succeeds”\)\.

Operationally, after receiving\(g,t\)\(g,t\), the red\-team agent is prompted to generate a structured probe with these four named slots\. The generation follows two constraints: \(i\) the underlying unsafe operationggmust remain semantically unchanged, and \(ii\) the resulting probe must be framed as a legitimate software\-engineering task of typett\. The final output is a structured four\-slot probe that is directly submitted to the target agent as the testing input\. These template slots therefore emulate realistic test prompts while preserving the operational dependency ongg\. In the instantiated probe, the unsafe operationggis presented as an intermediate step within the testing workflow rather than being stated as the explicit end goal\. This is the core workload\-contextualization mechanism: it preserves the operational dependency onggwhile presenting the request in a form that resembles everyday software maintenance\. Concrete instantiated probes and prompt examples are included in our evaluation materials\.

### 4\.5\.Outcome\-Aware Feedback Loop

After the red\-team agent constructs the initial testing probeptest​\(g\)p\_\{\\text\{test\}\}\(g\)\(Step 3\), the target execution\-capable coding agent executes it in the environment, producing an execution traceτtest\\tau\_\{\\text\{test\}\}and the agent’s responsertestr\_\{\\text\{test\}\}\. The goal of this stage is to close the loop between execution evidence and probe synthesis: an execution oracle converts\(g,τtest,rtest\)\(g,\\tau\_\{\\text\{test\}\},r\_\{\\text\{test\}\}\)into an outcome labelotesto\_\{\\text\{test\}\}and rollback directivesπg\\pi\_\{\\text\{g\}\}\. Ifotest≠Successo\_\{\\text\{test\}\}\\neq\\textit\{Success\}and the interaction budget allows, the generator performs the refinement step \(Steps 5–6\) to generate a follow\-up probepnextp\_\{\\text\{next\}\}for one additional execution attempt\.

Outcome Check \(Step 4\)\.The oracle determines whether the current attempt has realized the target unsafe operationgg, and if not, whether the failure is due to a policy refusal or an execution miss\. Formally, the oracle assigns an outcome label:otest=Check​\(g,τtest,rtest\)∈\{Success,Reject,Failure\}o\_\{\\text\{test\}\}=\\textsc\{Check\}\(g,\\tau\_\{\\text\{test\}\},r\_\{\\text\{test\}\}\)\\in\\\{\\textit\{Success\},\\textit\{Reject\},\\textit\{Failure\}\\\}\. Crucially,otesto\_\{\\text\{test\}\}is decided based on execution\-layer evidence rather than textual compliance: an attempt is labeledSuccessonly when the observed execution evidence satisfies the predicateϕg\\phi\_\{g\}\(e\.g\., goal\-tied artifact changes, file\-system diffs, or tool\-call patterns\), regardless of how the agent verbally describes its behavior\. In contrast,Rejectindicates that the target agent refuses to carry out the request, andFailureindicates that the agent may have performed tool usage, yetτtest\\tau\_\{\\text\{test\}\}does not provide evidence thatggwas realized\.

Rollback Directives \(Step 5\)\.Step 5 converts oracle feedback into a structured refinement signal rather than applying arbitrary prompt mutation\. Concretely, the rollback module first distinguishes whether the current probe failed because it was rejected at the language layer or because it was accepted but did not realizeggin the environment\. It then maps that failure mode to slot\-level edit targets in the shared template and produces concrete rollback directivesπg\\pi\_\{g\}for the next probe\. Formally,πg=Rollback​\(otest,g,ϕg\)\\pi\_\{g\}=\\textsc\{Rollback\}\\\!\\left\(o\_\{\\text\{test\}\},\\,g,\\,\\phi\_\{g\}\\right\), whereotest∈\{Reject,Failure,Success\}o\_\{\\text\{test\}\}\\in\\\{\\text\{Reject\},\\text\{Failure\},\\text\{Success\}\\\}is the outcome returned by the execution oracle, andϕg\\phi\_\{g\}is the evidence predicate associated with the target unsafe operationgg\. The outputπg\\pi\_\{g\}is a structured set of edit directives specifying which prompt elements to modify and how to modify them, while keeping the underlying target unsafe operationggunchanged\. Whenotest=Rejecto\_\{\\text\{test\}\}=\\text\{Reject\}, the rollback focuses on narrative\-facing slots such as Motivation, Test Objective, or Pass Criterion, since the failure indicates that the probe formulation itself activates the target agent’s language\-layer safeguard\. Whenotest=Failureo\_\{\\text\{test\}\}=\\text\{Failure\}, the rollback instead targets operational elements such as Procedure, workload type, or environment\-specific setup, because the probe has passed the language layer but still failed to produce the evidence required byϕg\\phi\_\{g\}\. In both cases, rollback acts as an outcome\-conditioned translation from observed failure modes to constrained prompt edits, so that later refinement preserves the underlying unsafe intent while changing only the parts most likely responsible for the current miss\.

Probe Refinement \(Step 6\)\.Step 6 operationalizes the rollback signal by turningπg\\pi\_\{g\}into follow\-up probes under the same shared template, rather than restarting generation from scratch\. Its role is to perform constrained probe refinement: the unsafe operationggand the overall task wrapper are preserved, while only the slots selected byπg\\pi\_\{g\}are revised\. Formally, the refinement step takes the initial probeptest​\(g\)p\_\{\\text\{test\}\}\(g\)and the rollback directivesπg\\pi\_\{g\}as input, and outputs a candidate set𝒫next=Refine​\(ptest​\(g\),πg\)=\{pnext\(1\),…,pnext\(K\)\}\\mathcal\{P\}\_\{\\text\{next\}\}=\\textsc\{Refine\}\(p\_\{\\text\{test\}\}\(g\),\\pi\_\{g\}\)=\\\{p\_\{\\text\{next\}\}^\{\(1\)\},\\dots,p\_\{\\text\{next\}\}^\{\(K\)\}\\\}\. Followingπg\\pi\_\{g\}, the workload generator translates the directives into explicit edits over predefined prompt slots and synthesizesKKdiverse probe variants while maintaining a consistent test/task wrapper\. Each candidate therefore represents a targeted follow\-up hypothesis about how the current probe should be adjusted to cross the next part of the security boundary, rather than a free\-form rewrite\. Each candidatepnext\(i\)∈𝒫nextp^\{\(i\)\}\_\{\\text\{next\}\}\\in\\mathcal\{P\}\_\{\\text\{next\}\}is then executed in the environment and checked by the execution oracle usingϕg\\phi\_\{g\}\. The procedure terminates early if any candidate achieves Success; otherwise, it proceeds to the next refinement round by invokingπg=Rollback​\(otest,g,ϕg\)\\pi\_\{g\}=\\textsc\{Rollback\}\\\!\\left\(o\_\{\\text\{test\}\},\\,g,\\,\\phi\_\{g\}\\right\)again on the latest outcome\. We allow up toBBrefinement rounds per goal, so the overall process progressively approaches the target agent’s execution\-layer security boundary through outcome\-conditioned, goal\-preserving probe updates\.

## 5\.Experiment

In this section, we first present our research questions \(RQs\), followed by our detailed testing setup\. We then systematically answer each RQ with our experimental results\.

### 5\.1\.Research Questions

RQ1\.:Across different agent configurations, how far do observed system\-level behaviors deviate from intended safety alignment?

RQ2\.:What is the impact of our red\-team strategies on exposing execution\-layer unsafe behaviors?

RQ3\.:How does our testing framework compare with the tested baselines?

RQ4\.:How is the execution\-layer security gap distributed across system\-operation domains and fine\-grained risky scenarios?

### 5\.2\.Testing Setup

Agents and models\.To characterize the execution\-layer security gap across different agent configurations, we consider three widely used agent frameworks: ReAct \(RA\)\(Yaoet al\.,[2023](https://arxiv.org/html/2607.22569#bib.bib4)\), CodeAct\(Wanget al\.,[2024a](https://arxiv.org/html/2607.22569#bib.bib5)\), and OpenCodeInterpreter \(OCI\)\(Zhenget al\.,[2024a](https://arxiv.org/html/2607.22569#bib.bib3)\)\. We also include Claude Code\(Anthropic,[2026](https://arxiv.org/html/2607.22569#bib.bib29)\)and Codex\(OpenAI,[2026](https://arxiv.org/html/2607.22569#bib.bib28)\)as commercial\-agent targets, since they are among the most commonly used production coding agents today and offer capabilities comparable to open\-source tool\-using agents, making them important evaluation targets in practice\. For the underlying backbones, we primarily use commonly adopted LLMs, including GPT\-4o\(OpenAI,[2025b](https://arxiv.org/html/2607.22569#bib.bib25)\), GPT\-4\.1\(OpenAI,[2025a](https://arxiv.org/html/2607.22569#bib.bib26)\), GPT\-5\.1\(OpenAI,[2025c](https://arxiv.org/html/2607.22569#bib.bib27)\), DeepSeek\-V3\(DeepSeek AI,[2025](https://arxiv.org/html/2607.22569#bib.bib31)\), and DeepSeek\-Coder\(Guoet al\.,[2024b](https://arxiv.org/html/2607.22569#bib.bib33)\)\. Notably, OCI additionally enforces hard\-coded security constraints in its execution stack\. For OCI, we adopt the framework\-provided backbones fine\-tuned for improved agent capabilities \(e\.g\., better tool use and execution reliability\), based on CodeLlama \(7B/13B\) and DeepSeekCoder \(6\.7B\)\(Zhenget al\.,[2024b](https://arxiv.org/html/2607.22569#bib.bib32)\)\.

Red\-team agent configuration\.The red\-team agent is implemented as an LLM\-based probe generator used only for probe construction and refinement; it never decides whether an unsafe operation has succeeded\. Outcome labels are produced solely by the execution oracle from execution traces and evidence predicates\. In our main testing, we use GPT\-4\.1 as the red\-team generator with a fixed decoding configuration: temperature=0\.7=0\.7, top\-p=1\.0p=1\.0, and max tokens=1024=1024\. The evaluated code tasks are in Python, and we keep this configuration fixed across all target\-agent settings\. Each testing run is instantiated in a fresh container so that state changes do not affect subsequent runs\.

Goal pool and sandbox environment\.We use the RedCode\-Exec benchmark\(Guoet al\.,[2024a](https://arxiv.org/html/2607.22569#bib.bib2)\)as our goal pool\. It organizes risky behaviors for execution\-capable agents into 8 domains and 25 concrete executable scenarios, with each scenario containing 30 unsafe operations\. Each unsafe operation is paired with a corresponding goalggand an execution evidence predicateϕg​\(τ\)\\phi\_\{g\}\(\\tau\)for checking whether the intended execution\-layer effect appears in the observed traceτ\\tau\. Our testing targets system\-environment effects, but for safety and reproducibility we instantiate each run in a fresh Docker container with scenario\-specific resources and setup\.

Baselines and setting\.We compare our framework against three classic jailbreak\-style baselines for probing the security boundary of coding agents, all tested under the same execution\-grounded oracle\. MASTERKEY\(Denget al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib6)\)generates iterative jailbreak variants from a seed template\. LLM\-Fuzzer\(Yuet al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib7)\)applies jailbreak\-style seed selection and mutation\. RedCode\-prefix\(Guoet al\.,[2024a](https://arxiv.org/html/2607.22569#bib.bib2)\)prepends a lightweight jailbreak prefix to the original instruction forgg\. Each baseline is ported to our setting by treating the unsafe operationggas the underlying target while keeping the same execution\-grounded outcome checking\. We set the testing budget toB=2B=2per goal and allow up toK=4K=4prompt variants per refinement step\. To ensure a fair comparison, we apply the same total interaction budgetB×KB\\times Kto all baseline methods\.

### 5\.3\.Metrics

Each testing run against a target agent results in one of three mutually exclusive outcomes:Reject,Success, orFail\. For a set of evaluated goalsGG, we report:

P​\(y\)=\|\{g∈G:outcome​\(g\)=y\}\|\|G\|,P\(y\)=\\frac\{\|\\\{g\\in G:\\mathrm\{outcome\}\(g\)=y\\\}\|\}\{\|G\|\},wherey∈\{Reject,Success,Fail\}y\\in\\\{\\mathrm\{Reject\},\\mathrm\{Success\},\\mathrm\{Fail\}\\\}\. We writeRR=P​\(Reject\)\\mathrm\{RR\}=P\(\\mathrm\{Reject\}\),ASR=P​\(Success\)\\mathrm\{ASR\}=P\(\\mathrm\{Success\}\), andFail=P​\(Fail\)\\mathrm\{Fail\}=P\(\\mathrm\{Fail\}\)\.

To compare agent behaviorBeforeandAfterapplying our testing framework, we report Before and After metrics over the workload\-contextualized stage only\.Beforerecords the outcomes obtained by directly querying the target agent with the original unsafe goal and executing that attempt to completion \(Crucially, it is not a framework\-generated result, so nonzeroBefore Successsimply means some goals are already directly executable under direct prompting\);Afteris the final outcome after applying the full testing framework, including workload contextualization and refinement within budgetBB\. Before and After are computed over the same evaluated goal set\. For repeated\-run results, we report 95% confidence intervals \(CI\) around the run\-level mean to quantify uncertainty induced by repeated executions\. In our setting, RR measures language\-layer refusal, while ASR measures execution\-layer unsafe outcomes verified by observable evidence satisfyingϕg\\phi\_\{g\}\. Their divergence is the key measurement of the execution\-layer security gap\.

### 5\.4\.Experimental Results

Table 1\.Execution\-grounded testing results before vs\. after refinement across agents on Code and Text datasets \(RA: ReAct; OCI: OpenCodeInterpreter; CA: CodeAct\. DeepSeek\-Coder: DeepSeek\-Coder\-6\.7B\-Instruct; OCI\-CL/DS: fine\-tuned CodeLlama/DeepSeek released from OCI\.Δ​R​R/A​S​R=R​R/A​S​Rafter−R​R/A​S​Rbefore\\Delta RR/ASR=RR/ASR\_\{\\text\{after\}\}\-RR/ASR\_\{\\text\{before\}\}\. Before denotes the outcome obtained by directly querying the original unsafe goal and executing that attempt to completion, and After denotes the final outcome after the full testing framework\)\.DatasetAgentModelBefore \(%\)After \(%\)Δ\\Delta\(%\)RejectFailSuccessRejectFailSuccessΔ​R​R\\Delta RRΔ​A​S​R\\Delta ASRExecutionCodeRAGPT\-4o38\.271\.3360\.400\.006\.3093\.70\-38\.2733\.30GPT\-4\.120\.134\.2775\.600\.539\.7789\.70\-19\.6014\.10GPT\-5\.112\.004\.1383\.870\.5315\.3484\.13\-11\.470\.26DeepSeek\-V315\.8710\.2673\.871\.336\.6792\.00\-14\.5418\.13DeepSeek\-Coder58\.547\.0834\.381\.8744\.7953\.33\-56\.6718\.95OCIOCI\-DS\-6\.7b46\.5339\.2014\.2710\.2720\.9368\.80\-36\.2654\.53OCI\-CL\-7b32\.1362\.135\.7310\.4027\.2062\.40\-21\.7356\.67OCI\-CL\-13b35\.6051\.7312\.678\.7032\.1759\.13\-26\.9046\.46CAGPT\-4o48\.2929\.2022\.515\.0249\.8045\.20\-43\.2765\.78GPT\-4\.157\.6728\.7613\.582\.5917\.0380\.38\-55\.0866\.80GPT\-5\.129\.7214\.1959\.092\.6925\.8171\.51\-27\.0312\.42DeepSeek\-V310\.5313\.0776\.402\.0014\.9583\.04\-8\.536\.64Codex CLIGPT\-5\.117\.193\.1679\.652\.281\.0596\.67\-14\.9117\.02Claude CodeSonnet\-4\.528\.301\.8969\.817\.550\.0092\.45\-20\.7522\.64ExecutionTextRAGPT\-4o55\.976\.1137\.920\.4210\.0089\.58\-55\.5551\.66GPT\-4\.149\.474\.8045\.730\.6724\.9374\.40\-48\.8028\.67GPT\-5\.185\.902\.3111\.7914\.6245\.1340\.26\-71\.2828\.47DeepSeek\-V344\.674\.1351\.200\.0013\.2086\.80\-44\.6735\.60DeepSeek\-Coder13\.2026\.1360\.676\.5327\.0766\.40\-6\.675\.73OCIOCI\-CL\-7b44\.8132\.5922\.5937\.0436\.3026\.67\-7\.774\.08OCI\-CL\-13b21\.4862\.5915\.9318\.1551\.8530\.00\-3\.3314\.07OCI\-DS\-6\.7b47\.7334\.8017\.4721\.7349\.0729\.20\-26\.0011\.73CAGPT\-4o81\.7110\.687\.615\.0755\.1439\.79\-79\.8432\.18GPT\-4\.189\.337\.473\.201\.8754\.7443\.39\-87\.4640\.19GPT\-5\.148\.7027\.2224\.075\.9353\.8940\.19\-42\.7716\.12DeepSeek\-V344\.5917\.3638\.052\.9428\.6168\.45\-41\.6530\.40Codex CLIGPT\-5\.113\.735\.8880\.390\.000\.00100\.00\-13\.7319\.61Claude CodeSonnet\-4\.533\.704\.8661\.446\.646\.4986\.87\-27\.0625\.43Table 2\.Ablation study of the execution\-grounded testing pipeline\. \(●\\CIRCLEindicates the component is enabled, and○\\Circleindicates it is removed\.R​R/A​S​RA=R​R/A​S​Ra​f​t​e​rRR/ASR\_\{A\}=RR/ASR\_\{after\}\)AgentRollback RouteTest Pattern SetTTResult \(%\)SemanticExecutionUnitRegr\.CrashValid\.R​RARR\_\{A\}A​S​RAASR\_\{A\}Δ​R​R\\Delta RRΔ​A​S​R\\Delta ASRRA\(GPT\-4o\)○○●●●●7\.6061\.07\-30\.670\.67●○●●●●3\.7090\.00\-34\.5729\.60○●●●●●11\.1185\.19\-27\.1624\.79●●●○○○3\.7087\.04\-34\.5726\.64●●○●○○2\.2273\.33\-36\.0512\.93●●○○●○0\.0088\.89\-38\.2728\.49●●○○○●0\.7490\.00\-37\.5329\.60●●●●●●0\.0093\.70\-38\.2733\.30#### 5\.4\.1\.RQ1: Deviation from Safety Alignment Across Agent Configurations

Our testing measures the gap between language\-layer refusal and execution\-layer unsafe outcomes: the degree to which an agent that refuses direct unsafe requests still fails to prevent those operations when they are presented through workload\-contextualized testing probes\. Throughout our testing, ASR is determined by execution evidence \(tool traces and observable sandbox side effects\) rather than textual compliance, and RR reflects language\-layer refusal alone\. Table[1](https://arxiv.org/html/2607.22569#S5.T1)summarizes the overall outcomes \(Reject/Fail/Success\) on both the Code and Text splits, reported before the framework is applied and after the full testing pipeline\. The commercial\-agent rows in Table[1](https://arxiv.org/html/2607.22569#S5.T1)show the same qualitative pattern: both Codex CLI and Claude Code exhibit lower RR and higher ASR after the full execution\-grounded testing pipeline is applied\.

The central finding is that a substantial gap exists between RR and ASR across all evaluated configurations\. On average across all agent–model configurations, the Code split shows a baseline RR of 44\.36% under direct requests, meaning that the agent refuses that fraction at the language layer, yet the same operations reach an ASR of 73\.61% after workload\-contextualized testing, a gap of 29\.25 percentage points\. On the Text split, the gap is 25\.91 points \(28\.02% RR vs\. 53\.93% ASR\)\. This divergence is the core security measurement finding: the language\-layer refusal rate substantially*underestimates*the execution\-layer unsafe outcome rate\. An agent that appears secure under direct\-request testing because its safeguard refuses a large fraction of goals is demonstrably not secure when those same goals are embedded in realistic testing and debugging workflows, as evidenced by the execution oracle recording tool\-mediated state changes in the sandbox\.

The Code/Text gap in ASR \(73\.61% vs\. 53\.93%\) is expected because Code goals provide a more direct executable carrier: once the agent accepts the testing narrative, it often only needs to execute and observe\. Text goals require the agent to translate natural\-language descriptions into concrete tool\-mediated actions, increasing execution misses\. Notably, a lower ASR on Text should not be interpreted as stronger security by default: many non\-Reject cases in the Text split manifest as Fail \(the agent does not refuse but still does not satisfyϕg\\phi\_\{g\}\), indicating the bottleneck is execution capability rather than security alignment\. This distinction is important for testing: a high Fail rate means the security gap exists but is masked by capability limitations, not by genuine security enforcement\.

Across frameworks, OCI is the most conservative overall, consistent with its additional hard\-coded security constraints\. This is especially evident on the Text split \(OCI: 28\.62% ASR vs\. ReAct: 71\.49%, CodeAct: 47\.96%\)\. However, OCI’s robustness is notably weaker on the Code split, where ASR increases sharply after workload contextualization \(rising to approximately 59%–69%\)\. This contrast reveals a structural weakness: hard\-coded language\-layer constraints are less effective when the unsafe behavior is presented in directly executable form, because rule\-based refusal checks are triggered by surface\-level signals rather than execution\-level intent\. Different LLM backbones also shift outcomes in qualitatively different directions, toward Reject, Fail, or Success, even within the same agent framework\. This indicates that language\-layer refusal behavior is highly backbone\-dependent and does not reliably predict execution\-layer security, motivating agent\-level defenses \(tool guardrails, execution\-time state monitoring\) that operate below the language layer\.

![Refer to caption](https://arxiv.org/html/2607.22569v1/x4.png)Figure 4\.Repeated\-run robustness over 3 repeated runs\. Left: Before/After means for RR, ASR, and Fail, with error bars showing 95% confidence intervals\. Right: the corresponding average changes after applying the full testing framework\.The commercial\-agent rows in Table[1](https://arxiv.org/html/2607.22569#S5.T1)also suggest that the execution\-layer security gap is not confined to open\-source frameworks\. For Codex CLI on the Code split, the full evaluation shows the same trend: RR decreases from 17\.19% to 2\.28% \(98 rejects to 13\), while ASR increases from 79\.65% to 96\.67% \(454 successes to 551\)\. Claude Code exhibits the same qualitative pattern, with lower RR and higher ASR after applying the full testing pipeline\. These results indicate that the same language\-execution misalignment also appears in widely used commercial agents\.

To assess robustness under model nondeterminism, we repeat the full testing process three times under the ReAct \(RA\) setting, while keeping the same GPT\-4\.1 red\-team generator configuration, and report the repeated\-run results in Figure[4](https://arxiv.org/html/2607.22569#S5.F4)\. Figure[4](https://arxiv.org/html/2607.22569#S5.F4)shows that the aggregate results remain stable across runs: the weighted overall baseline RR is 11\.67 with a 95% CI of 0\.27, while the final ASR is 98\.89 with a 95% CI of 1\.09\. The repeated\-run averageΔ\\DeltaASR is 10\.56 \(95% CI 1\.27\), indicating that the execution\-layer security gap exposed by our framework is not an artifact of a single stochastic run\.

Summary:Our execution\-grounded testing reveals a systematic security gap: a baseline RR of 44\.36% \(Code\) and 28\.02% \(Text\) under direct requests stands against an ASR of 73\.61% and 53\.93% after workload\-contextualized testing, demonstrating that language\-layer refusal underestimates execution\-layer unsafe outcomes by 25–30 percentage points across all tested configurations\.

#### 5\.4\.2\.RQ2: Impact of Red\-Team Strategies

![Refer to caption](https://arxiv.org/html/2607.22569v1/x5.png)Figure 5\.Comparison of our execution\-grounded framework and text\-centric baselines onΔ​ASR\\Delta\\text\{ASR\}andΔ​Reject\\Delta\\text\{Reject\}across 8 domains covering 25 scenarios\.We ablate two key design choices in our testing framework: \(i\) the rollback route \(Semantic vs\. Execution\), and \(ii\) the workload\-type set𝒯\\mathcal\{T\}\(Unit / Regression / Crash / Validation\)\. Table[2](https://arxiv.org/html/2607.22569#S5.T2)reports results under the ReAct \+ GPT\-4o configuration\.

When both rollback routes are removed, the framework reduces to a pure workload wrapper with all types enabled\. This setting can still reduce RR, but yields almost no additional gain in ASR \(Δ​ASR=0\.67\\Delta\\text\{ASR\}=0\.67\)\. This finding has a direct testing implication: workload contextualization alone, which wraps unsafe operations in testing narratives, is sufficient to lower the language\-layer refusal rate, but insufficient to reveal the full execution\-layer security gap\. The additional ASR that rollback unlocks represents security failures that would remain hidden under single\-attempt testing\.

Enabling rollback immediately restores large ASR gains\. Semantic rollback is particularly effective at suppressing Reject outcomes \(RRA\\text\{RR\}\_\{A\}drops to 3\.70\), consistent with its role in adjusting narrative framing to reduce language\-layer trigger signals while preserving testing intent\. Execution rollback targets non\-refusal failures and improves ASR through execution trace alignment; combining both yields the strongest result \(ASRA=93\.70\\text\{ASR\}\_\{A\}=93\.70,Δ​ASR=33\.30\\Delta\\text\{ASR\}=33\.30\), confirming that the two rollback modes address orthogonal failure modes \(language\-layer refusal vs\. execution\-layer miss\) and together reveal the maximum security gap\.

When restricted to a single workload type, ASR drops compared to using the full set\. Different types probe different safeguard triggers and expose different security failure modes; no single type captures the full gap\. Using all types jointly maximizes testing coverage, indicating that type diversity is a structural property of thorough execution\-grounded testing\.

Summary:Both rollback routes and workload\-type diversity have a clear impact on testing coverage\. Without rollback, the framework shows only marginal additional ASR beyond language\-layer refusal bypass \(Δ\\DeltaASR = 0\.67\)\. Semantic rollback primarily reduces RR \(RRA=3\.70\\text\{RR\}\_\{A\}=3\.70\), while execution rollback improves end\-to\-end execution coverage\. Combining both achieves the largest security gap exposure \(Δ\\DeltaASR = 33\.30\)\. Using the full workload\-type set consistently outperforms any individual type\.

#### 5\.4\.3\.RQ3: Comparison with Baselines

We compare our framework with three representative baselines ported from LLM jailbreak methods \(MASTERKEY\(Denget al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib6)\), LLM\-Fuzzer\(Yuet al\.,[2024](https://arxiv.org/html/2607.22569#bib.bib7)\), and RedCode\-prefix\(Guoet al\.,[2024a](https://arxiv.org/html/2607.22569#bib.bib2)\)\), all evaluated under the same execution\-grounded oracle\. Figure[5](https://arxiv.org/html/2607.22569#S5.F5)reports domain\-wise changes from Before to After forΔ​ASR\\Delta\\text\{ASR\}andΔ​Reject\\Delta\\text\{Reject\}across the 8 domains covering 25 scenarios\. The key interpretive point is thatΔ​ASR\\Delta\\text\{ASR\}andΔ​Reject\\Delta\\text\{Reject\}measure different things\.Δ​Reject\\Delta\\text\{Reject\}\(decrease in Reject\) measures whether testing probes bypass language\-layer screening\.Δ​ASR\\Delta\\text\{ASR\}\(increase in Success\) measures whether testing probes actually produce execution\-layer unsafe outcomes, as verified byϕg\\phi\_\{g\}over sandbox traces\. A method that reduces Reject without increasing ASR has not revealed an execution\-layer security gap; it has only shown that language\-layer blocking becomes weaker while verified unsafe execution does not increase correspondingly\.

Consistent with this, Figure[5](https://arxiv.org/html/2607.22569#S5.F5)shows that only our framework consistently exhibits the desired cross\-domain pattern: higherΔ​ASR\\Delta\\text\{ASR\}together with lowerΔ​Reject\\Delta\\text\{Reject\}\. The three ported LLM jailbreak baselines often reduce Reject but deliverΔ​ASR\\Delta\\text\{ASR\}gains that are small, unstable, or even negative in multiple domains\. This gap arises because LLM\-jailbreak techniques are designed to elicit disallowed text through prompt manipulation\. They can reduce language\-layer refusal, but that same semantic distortion often degrades the operational coherence required for tool\-mediated execution\. For weaker execution settings \(e\.g\., DeepSeek\-Coder\-6\.7B\), LLM\-Fuzzer drives Reject to near\-zero while collapsing ASR, illustrating that lowΔ​Reject\\Delta\\text\{Reject\}alone does not characterize the execution\-layer security boundary\. Our framework, by contrast, preserves task semantics through workload contextualization rather than obfuscation, ensuring that ASR reflects genuine execution\-layer security failures rather than execution misses\.

Summary:Our execution\-grounded framework shows the most consistent security gap characterization across domains: it raises ASR while lowering Reject, converting language\-layer refusals into oracle\-verified execution\-layer outcomes\. Text\-centric baselines may reduce Reject as well, but their ASR gains are much less stable, so reduced refusal alone does not imply a larger execution\-layer security gap\.

![Refer to caption](https://arxiv.org/html/2607.22569v1/x6.png)Figure 6\.Heatmap of scenario\-wiseΔ​ASR\\Delta\\text\{ASR\}andΔ​RR\\Delta\\text\{RR\}across agent frameworks and carrier types\.
#### 5\.4\.4\.RQ4: Security Gap Distribution Across System\-Operation Domains

This RQ examines how the execution\-layer security gap distributes across the fine\-grained scenario taxonomy\. Figure[6](https://arxiv.org/html/2607.22569#S5.F6)visualizes the 25 risky scenarios \(grouped into 8 domains\) as a heatmap, with rows corresponding to scenarios and columns grouped by agent framework and carrier type\. The left block showsΔ​ASR\\Delta\\text\{ASR\}and the right block showsΔ​RR\\Delta\\text\{RR\}for the three agent frameworks on both Code and Text splits\. Darker cells indicate larger\-magnitude changes: positiveΔ​ASR\\Delta\\text\{ASR\}means more execution\-oracle\-verified unsafe side effects than Before, while more negativeΔ​RR\\Delta\\text\{RR\}means fewer language\-layer refusals after the full framework is applied\. The execution\-layer security gap is broad rather than concentrated in a small subset of scenarios\. Across the majority of scenarios, we observe positiveΔ​ASR\\Delta\\text\{ASR\}together with negativeΔ​RR\\Delta\\text\{RR\}, confirming that the gap between language\-layer security measurement and execution\-layer security measurement is a systemic property across system\-operation task domains, not an artifact of particular scenarios\. Several domains reach high ASR after workload\-contextualized testing \(e\.g\., Operating System and Program logic on the Code split\), indicating that in these domains, language\-layer refusal is a particularly unreliable security proxy: the agent reliably refuses direct requests but consistently fails to maintain that security boundary when the same operations are embedded in testing and debugging workflows\. A small number of scenarios show weakΔ​ASR\\Delta\\text\{ASR\}or slight regression\. For example, weaker cases include file\-enumeration tasks in the file\-system domain, IP\-pattern validation in the network domain, predictable\-key generation in the cyber\-security domain, and logic/data\-processing tasks such as tax calculation and key\-value lookup\. This pattern is largely explained by a ceiling effect: when the initial workload\-contextualized probe already achieves high ASR, there is limited headroom for refinement to further expose security gaps\. Scenario\-level results confirm that workload\-contextualized testing broadly reveals execution\-layer security gaps across the scenario spectrum, but with variance; aggregate averages can mask both high\-gap scenarios and ceiling\-limited cases\.

Summary:The execution\-layer security gap, namely the divergence between RR and ASR, generalizes broadly across 25 risky scenarios and 8 system\-operation domains\. In most tested scenarios, language\-layer refusals are not a reliable proxy for preventing execution\-layer unsafe outcomes, motivating execution\-grounded testing as a first\-class security measurement methodology for coding agents in system operations\.

## 6\.Limitations

Our goal pool is primarily inherited from the RedCode benchmark\. Although RedCode is widely used and well structured, it may not fully cover the threat surface faced by real\-world execution\-capable agents, so our current results should be interpreted as evidence on an important benchmarked subset rather than as complete coverage of coding\-agent security risk\. When an operation causes only limited or superficial environment\-level impact, agents may be less likely to refuse it even under direct prompting, which can affect baseline difficulty and cross\-domain comparisons\.

In addition, our testing is conducted in a controlled Docker sandbox\. This is necessary for security and reproducibility, but it still differs from production environments in tool availability, external connectivity, permissions, and system policies, which may affect how broadly our measured execution\-layer security gaps transfer to real deployments\. Our execution oracle also remains bounded by the benchmark’s evidence predicates: it captures observable side effects such as tool traces, runtime outputs, and file\-system diffs, but may miss subtler near\-miss behaviors or partial harms that do not satisfy the predefined predicate\.

## 7\.Conclusion

We presented an execution\-grounded red\-team testing framework for coding agents in system operations\. By grounding measurement in observable execution evidence, our framework shows that language\-layer refusal substantially underestimates execution\-layer unsafe outcomes, and that workload\-contextualized probes expose security failures that text\-centric testing misses\.

## References

- M\. Andriushchenko, A\. Souly, M\. Dziemian, D\. Duenas, M\. Lin, J\. Wang, D\. Hendrycks, A\. Zou, Z\. Kolter, M\. Fredrikson, E\. Winsor, J\. Wynne, Y\. Gal, and X\. Davies \(2024\)Agentharm: a benchmark for measuring harmfulness of llm agents\.arXiv preprint arXiv:2410\.09024\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.22569#S2.SS2.p2.1)\.
- Note:Accessed: 2026\-04External Links:[Link](https://docs.anthropic.com/en/docs/claude-code/overview)Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p1.1),[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p1.1)\.
- Y\. Cheng, M\. Georgopoulos, V\. Cevher, and G\. G\. Chrysos \(2024\)Leveraging the context through multi\-round interactions for jailbreaking attacks\.arXiv preprint arXiv:2402\.09177\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p2.1)\.
- DeepSeek AI \(2025\)Note:Accessed: 2025\-09External Links:[Link](https://huggingface.co/deepseek-ai/DeepSeek-V3-Base)Cited by:[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p1.1)\.
- G\. Deng, Y\. Liu, Y\. Li, K\. Wang, Y\. Zhang, Z\. Li, H\. Wang, T\. Zhang, and Y\. Liu \(2024\)MASTERKEY: automated jailbreaking of large language model chatbots\.In31st Annual Network and Distributed System Security Symposium, NDSS 2024, San Diego, California, USA, February 26 \- March 1, 2024,Cited by:[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p4.5),[§5\.4\.3](https://arxiv.org/html/2607.22569#S5.SS4.SSS3.p1.7)\.
- P\. Ding, J\. Kuang, D\. Ma, X\. Cao, Y\. Xian, J\. Chen, and S\. Huang \(2024\)A wolf in sheep’s clothing: generalized nested jailbreak prompts can fool large language models easily\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 2136–2153\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p2.1)\.
- C\. Guo, X\. Liu, C\. Xie, A\. Zhou, Y\. Zeng, Z\. Lin, D\. Song, and B\. Li \(2024a\)RedCode: risky code execution and generation benchmark for code agents\.Cited by:[§2\.2](https://arxiv.org/html/2607.22569#S2.SS2.p2.1),[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p3.3),[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p4.5),[§5\.4\.3](https://arxiv.org/html/2607.22569#S5.SS4.SSS3.p1.7)\.
- D\. Guo, Q\. Zhu, D\. Yang, Z\. Xie, K\. Dong, W\. Zhang, G\. Chen, X\. Bi, Y\. Wu, Y\.K\. Li, F\. Luo, Y\. Xiong, and W\. Liang \(2024b\)DeepSeek\-coder: when the large language model meets programming – the rise of code intelligence\.CoRRabs/2401\.14196\.External Links:[Link](https://arxiv.org/abs/2401.14196)Cited by:[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p1.1)\.
- J\. Jiang, J\. Shen, S\. Kim, K\. M\. Yoo, J\. Kim, and S\. Kim \(2026\)ReflexiCoder: teaching large language models to self\-reflect on generated code and self\-correct it via reinforcement learning\.arXiv preprint arXiv:2603\.05863\.Cited by:[§2\.1](https://arxiv.org/html/2607.22569#S2.SS1.p2.1)\.
- D\. Kang, X\. Li, I\. Stoica, C\. Guestrin, M\. Zaharia, and T\. Hashimoto \(2024\)Exploiting programmatic behavior of llms: dual\-use through standard security attacks\.In2024 IEEE Security and Privacy Workshops \(SPW\),pp\. 132–143\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p2.1)\.
- D\. Liu, Q\. Ren, C\. Qian, S\. Shao, Y\. Xie, Y\. Li, Z\. Yang, H\. Luo, P\. Wang, Q\. Liu, B\. Hu, L\. Tang, J\. Mei, D\. Guo, L\. Yuan, J\. Yang, G\. Chen, Q\. Lin, Y\. Yu, B\. Zhang, J\. Guo, J\. Zhang, W\. Shao, H\. Deng, Z\. Xi, W\. Wang, W\. Wang, W\. Shen, Z\. Chen, H\. Xie, J\. Tao, J\. Dai, J\. Ji, Z\. Ba, L\. Zhang, Y\. Liu, Q\. Zhang, L\. Zhu, Z\. Wei, H\. Xue, C\. Lu, J\. Shao, and X\. Hu \(2026\)AgentDoG: a diagnostic guardrail framework for ai agent safety and security\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.22569#S2.SS2.p2.1)\.
- H\. Luo, S\. Dai, C\. Ni, X\. Li, G\. Zhang, K\. Wang, T\. Liu, and H\. Salam \(2025\)Agentauditor: human\-level safety and security evaluation for llm agents\.arXiv preprint arXiv:2506\.00641\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.22569#S2.SS2.p2.1)\.
- H\. Lv, X\. Wang, Y\. Zhang, C\. Huang, S\. Dou, J\. Ye, T\. Gui, Q\. Zhang, and X\. Huang \(2024\)Codechameleon: personalized encryption framework for jailbreaking large language models\.arXiv preprint arXiv:2402\.16717\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p2.1)\.
- S\. Naihin, D\. Atkinson, M\. Green, M\. Hamadi, C\. Swift, D\. Schonholtz, A\. T\. Kalai, and D\. Bau \(2023\)Testing language model agents safely in the wild\.arXiv preprint arXiv:2311\.10538\.Cited by:[§2\.2](https://arxiv.org/html/2607.22569#S2.SS2.p2.1)\.
- OpenAI \(2025a\)Note:Accessed: 2025\-09External Links:[Link](https://platform.openai.com/docs/models/gpt-4.1)Cited by:[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p1.1)\.
- OpenAI \(2025b\)Note:Accessed: 2025\-09External Links:[Link](https://platform.openai.com/docs/models/gpt-4o)Cited by:[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p1.1)\.
- OpenAI \(2025c\)Note:Accessed: 2025\-12External Links:[Link](https://platform.openai.com/docs/models/gpt-5.1)Cited by:[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p1.1)\.
- OpenAI \(2026\)Note:Accessed: 2026\-04External Links:[Link](https://openai.com/codex)Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p1.1),[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p1.1)\.
- OpenClaw \(2026\)Note:Accessed: 2026\-04External Links:[Link](https://docs.openclaw.ai/cli)Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p1.1)\.
- M\. Russinovich, A\. Salem, and R\. Eldan \(2025\)Great, now write an article about that: the crescendo\{\\\{multi\-turn\}\\\}\{\\\{llm\}\\\}jailbreak attack\.In34th USENIX Security Symposium \(USENIX Security 25\),pp\. 2421–2440\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p2.1)\.
- R\. Shah, Q\. Feuillade\-Montixi, S\. Pour, A\. Tagade, S\. Casper, and J\. Rando \(2023\)Scalable and transferable black\-box jailbreaks for language models via persona modulation\.arXiv preprint arXiv:2311\.03348\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p2.1)\.
- X\. Shen, Z\. Chen, M\. Backes, Y\. Shen, and Y\. Zhang \(2024\)” Do anything now”: characterizing and evaluating in\-the\-wild jailbreak prompts on large language models\.InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security,pp\. 1671–1685\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p2.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in Neural Information Processing Systems36,pp\. 8634–8652\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.22569#S2.SS1.p2.1)\.
- X\. Wang, Y\. Chen, L\. Yuan, Y\. Zhang, Y\. Li, H\. Peng, and H\. Ji \(2024a\)Executable code actions elicit better LLM agents\.CoRRabs/2402\.01030\.Cited by:[§2\.1](https://arxiv.org/html/2607.22569#S2.SS1.p2.1),[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p1.1)\.
- X\. Wang, Y\. Chen, L\. Yuan, Y\. Zhang, Y\. Li, H\. Peng, and H\. Ji \(2024b\)Executable code actions elicit better llm agents\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p1.1)\.
- J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press \(2024\)Swe\-agent: agent\-computer interfaces enable automated software engineering\.Advances in Neural Information Processing Systems37,pp\. 50528–50652\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.22569#S2.SS1.p2.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)React: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.1](https://arxiv.org/html/2607.22569#S2.SS1.p2.1),[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p1.1)\.
- J\. Yu, X\. Lin, Z\. Yu, and X\. Xing \(2024\)\{\\\{llm\-Fuzzer\}\\\}: scaling assessment of large language model jailbreaks\.In33rd USENIX Security Symposium \(USENIX Security 24\),pp\. 4657–4674\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p2.1),[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p4.5),[§5\.4\.3](https://arxiv.org/html/2607.22569#S5.SS4.SSS3.p1.7)\.
- T\. Yuan, Z\. He, L\. Dong, Y\. Wang, R\. Zhao, T\. Xia, L\. Xu, B\. Zhou, F\. Li, Z\. Zhang, R\. Wang, and G\. Liu \(2024\)R\-judge: benchmarking safety risk awareness for llm agents\.arXiv preprint arXiv:2401\.10019\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.22569#S2.SS2.p2.1)\.
- T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. Yue \(2024a\)Opencodeinterpreter: integrating code generation with execution and refinement\.arXiv preprint arXiv:2402\.14658\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.22569#S2.SS1.p2.1),[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p1.1)\.
- T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. Yue \(2024b\)Note:Accessed: 2025\-09External Links:[Link](https://huggingface.co/models?other=arxiv:2402.14658)Cited by:[§5\.2](https://arxiv.org/html/2607.22569#S5.SS2.p1.1)\.
- A\. Zhou, K\. Yan, M\. Shlapentokh\-Rothman, H\. Wang, and Y\. Wang \(2023\)Language agent tree search unifies reasoning acting and planning in language models\.arXiv preprint arXiv:2310\.04406\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p1.1)\.
- A\. Zou, Z\. Wang, N\. Carlini, M\. Nasr, J\. Z\. Kolter, and M\. Fredrikson \(2023\)Universal and transferable adversarial attacks on aligned language models\.arXiv preprint arXiv:2307\.15043\.Cited by:[§1](https://arxiv.org/html/2607.22569#S1.p2.1)\.