Safeguarding LLM Agents from Misalignment through Provenance Analysis

# Safeguarding LLM Agents from Misalignment through Provenance Analysis
Source: [https://arxiv.org/html/2607.01236](https://arxiv.org/html/2607.01236)
###### Abstract\.

As LLM agents gain increasing access to powerful tools, ensuring that their actions are aligned with the user’s intent becomes critical\. When an agent’s proposed tool invocation deviates from the user’s intent—a phenomenon called*misalignment*—it may lead to harmful consequences that are difficult to undo\. Existing runtime guardrails rely on an*LLM\-as\-a\-judge paradigm*that lacks a systematic framework for reasoning about alignment, often producing judgments that are inconsistent or difficult to audit\. Motivated by*provenance analysis*, we propose a provenance\-based conceptual framework that formalizes misalignment detection as determining whether a proposed tool call is supported by traceable evidence in the agent’s context\. Building on this framework, we propose\\tool, a multi\-stage pipeline that analyzes the agent’s action for three types of misalignment before the selected tool is executed and only allows the action to take place when it is considered aligned with the user’s input query\. We evaluated our proposed approach on two different benchmarks, Agent\-SafetyBench and WorkBench, across 10 backbone LLMs\. Compared to the LLM\-as\-a\-judge baseline,\\toolreduces error rate on misaligned traces from 42\.9% to 1\.8% on Agent\-SafetyBench and from 32\.1% to 17\.3% on WorkBench, while reducing intervention burden on task\-successful traces from 30\.5% to 12\.8% and introducing no statistically significant increase in unnecessary interventions on aligned traces\. These results demonstrate that structured, provenance\-based reasoning provides an effective and practical foundation for safeguarding LLM agents from misalignment\.

## 1\.Introduction

The use of large language models \(LLMs\) is rapidly advancing from conversational assistants toward autonomous agentic systems\. These systems equip LLMs with external tools that allow them to interact directly with the real world: sending emails, modifying files, controlling smart home devices, and executing code\(Yaoet al\.,[2023](https://arxiv.org/html/2607.01236#bib.bib33); OpenClaw,[2026](https://arxiv.org/html/2607.01236#bib.bib38); Anysphere, Inc\.,[2026](https://arxiv.org/html/2607.01236#bib.bib35); Anthropic,[2026](https://arxiv.org/html/2607.01236#bib.bib41); Birkmoseet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib42)\)\. Unlike chatbots that simply produce text for human review, agents can perform consequential and irreversible actions on the user’s behalf\. As agents gain greater autonomy, their potential impact and associated risks increase dramatically\(Suet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib32); Denget al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib40); Maet al\.,[2026](https://arxiv.org/html/2607.01236#bib.bib39); penligent,[2026](https://arxiv.org/html/2607.01236#bib.bib36)\)\.

A major risk in agentic systems is that agents may execute actions that deviate from what the user actually intended, a phenomenon we refer to as*misalignment*\. For example, a user who asks an agent to “request $30 from Daniel” may find that the agent instead calls a “send money” tool, paying the amount to Daniel, an action contrary to the user’s request\. The term “misalignment” has sometimes been used to refer to*malicious*agent behavior\(Lynchet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib10); Betleyet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib46); Naiket al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib8)\), an equally important but distinct concern\. In this work, we focus on cases where agents act without malicious intent but on a misunderstanding of or unjustified assumptions about the user’s intent that is expressed through a prompt\. Such misalignment can cause significant unintended harm, particularly when the resulting actions are irreversible\. Detecting and preventing misalignment before tool execution is therefore crucial for deploying agents in practice\.

Existing research has explored multiple approaches to mitigating the risks of agent misalignment\. Training\-time alignment methods, such as preference optimization and supervised fine\-tuning\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.01236#bib.bib43); Baiet al\.,[2022](https://arxiv.org/html/2607.01236#bib.bib45)\), can improve the general behavior of an agent, but cannot cover the full range of tool configurations, environment states, and task contexts encountered at deployment\. Prompt\-level controls, including carefully designed system instructions and reasoning scaffolds, can further guide behavior, but they still rely on the same underlying model whose judgments they seek to improve\(Sahooet al\.,[2024](https://arxiv.org/html/2607.01236#bib.bib12); Shinnet al\.,[2023](https://arxiv.org/html/2607.01236#bib.bib47); Madaanet al\.,[2023](https://arxiv.org/html/2607.01236#bib.bib48)\)\. Sandboxing and capability restriction can reduce harm by isolating tool execution and limiting an agent’s privileges, but they primarily contain the consequences of an action rather than determine whether the action faithfully reflects the user’s intent\(Zhanget al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib13); Wuet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib49)\)\. Post\-hoc verification and recovery methods provide useful auditing or corrections after actions are taken, but their abilities to prevent harm are limited once an irreversible tool invocation has already occurred\(Ruanet al\.,[2024](https://arxiv.org/html/2607.01236#bib.bib14); Vuddantiet al\.,[2026](https://arxiv.org/html/2607.01236#bib.bib50)\)\. Together, these limitations motivate the need for pre\-execution runtime*guardrails*in LLM agents\(Wanget al\.,[2025a](https://arxiv.org/html/2607.01236#bib.bib15); Xianget al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib17)\)\. Positioned between action generation and tool execution, these guardrails serve as an independent decision layer that can detect and prevent misaligned actions before they produce irreversible effects on the environment\(Fanget al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib7)\)\.

However, existing runtime guardrails often rely on an*LLM\-as\-a\-judge*mechanism to assess whether a proposed action should be allowed\(Zhenget al\.,[2023](https://arxiv.org/html/2607.01236#bib.bib31); Luoet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib54); Wanget al\.,[2026](https://arxiv.org/html/2607.01236#bib.bib55)\)\. Without a systematic framework for reasoning about alignment, such judgments remain largely subjective and unguided\(Yeadonet al\.,[2026](https://arxiv.org/html/2607.01236#bib.bib52)\)\. They lack explicit decision criteria, which can lead to inconsistent standards between runs and models\(Weiet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib51); Shiet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib53); Leeet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib56)\)\. Consequently, the underlying rationale for a judgment is more difficult to interpret and audit\.

![Refer to caption](https://arxiv.org/html/2607.01236v1/figures/example3.png)Figure 1\.Examples for 3 types of misalignment\.In this paper, we propose a systematic framework to detect misaligned agent actions motivated by*provenance analysis*, a well\-established discipline that traces the origin and derivation history of an artifact\(Cheneyet al\.,[2009](https://arxiv.org/html/2607.01236#bib.bib4); Davidson and Freire,[2008](https://arxiv.org/html/2607.01236#bib.bib5)\)\. The key intuition behind our approach is thatwhether or not an action is aligned with the user’s intent should be justifiable by traceable evidence in the context, i\.e\., the user’s request, the available tool documentation, and the history of prior interactions\. We develop this idea at the level of tool invocation bydistinguishing three types of misalignment\(Fig\.[1](https://arxiv.org/html/2607.01236#S1.F1)\): \(1\)*tool\-level misalignment*: selecting a wrong tool, \(2\)*parameter\-level misalignment*: selecting tool parameters that are not traceable to the context or not appropriate to address user’s query, and \(3\)*interpretation\-level misalignment*: selecting one of multiple possible interpretations of the user’s query that is*underspecified*in the given context\. We provide a conceptual framework that formalizes the notion of provenance in agentic settings and the definition of the above types of misalignment\. Our framework serves as a theoretical basis for implementing guardrails to check whether or not a tool action is considered aligned or misaligned based on explicit, auditable provenance links between the action and the given context\.

Based on the proposed framework, we developed\\tool, a multi\-stage pipeline implementation of the framework\. The pipeline performs a three\-stage analysis, with separate stages for tool\-level, parameter\-level, and interpretation\-level misalignment\. We evaluated\\toolon two benchmarks, Agent\-SafetyBench\(Zhanget al\.,[2024b](https://arxiv.org/html/2607.01236#bib.bib1)\)and WorkBench\(Styleset al\.,[2024](https://arxiv.org/html/2607.01236#bib.bib2)\), against an LLM\-as\-a\-judge baseline and a single\-step variant of\\tool, across 10 backbone LLMs\.

Our experimental results show that the proposed provenance\-based conceptual framework itself substantially improves misalignment detection over direct LLM\-as\-a\-judge prompting, and that the multi\-stage design yields significant additional gains, especially on Agent\-SafetyBench\. Specifically,\\toolreduces average error rate on misaligned traces from 42\.9% to 1\.8% on Agent\-SafetyBench and from 32\.1% to 17\.3% on WorkBench\. At the same time, these gains do not come with a comparable increase in intervention cost\. Compared to the baseline,\\tooldoes not introduce a statistically significant rise in unnecessary interventions on aligned traces, and it lowers intervention rate on task\-successful WorkBench traces from 30\.5% to 12\.8%\. These results suggest thatstructured, provenance\-based reasoning provides an effective and practical foundation for safeguarding LLM agents against misaligned actions\.

The main contributions of this paper are as follows:

- •A taxonomy of agent action misalignment, distinguishing tool\-level, parameter\-level, and interpretation\-level misalignments \(Sec\.[3](https://arxiv.org/html/2607.01236#S3)\)\.
- •A provenance\-based conceptual framework for detecting misaligned agent actions, which frames alignment checking as determining whether proposed tool calls are grounded by traceable evidence in the agent’s context \(Sec\.[4](https://arxiv.org/html/2607.01236#S4)\)\.
- •\\tool , a multi\-stage pipeline implementation of the framework that operates as a runtime guardrail \(Sec\.[4\.3](https://arxiv.org/html/2607.01236#S4.SS3)\), together with an evaluation on two benchmarks across 10 backbone LLMs \(Sec\.[5](https://arxiv.org/html/2607.01236#S5)&[6](https://arxiv.org/html/2607.01236#S6)\)\.
- •A manually annotated benchmark of agent execution traces derived from Agent\-SafetyBench \(Sec\.[5\.1](https://arxiv.org/html/2607.01236#S5.SS1)\)\.

## 2\.Preliminaries

### 2\.1\.LLM Agent Workflow

We consider a typical LLM agent setting in which an LLM interacts with a set of pre\-defined tools to complete user tasks \(Fig\.[2](https://arxiv.org/html/2607.01236#S2.F2)\)\. At the beginning of an interaction, a user submits a natural language query to the agent\. The LLM serves as the decision\-making core of the agent\. Based on the query and the current context, the LLM determines which tool to invoke and how to parameterize the tool call in order to access information or modify the environment\(Yaoet al\.,[2023](https://arxiv.org/html/2607.01236#bib.bib33)\)\.

![Refer to caption](https://arxiv.org/html/2607.01236v1/x1.png)Figure 2\.Workflow of an LLM agent with an external guardrail\. The guardrail monitors every proposed action before execution\.After a tool is invoked, the resulting output is returned to the LLM\. The model then decides the next step: It may invoke additional tools, ask the user for clarification, or return a final response once the task is completed\. Through this iterative loop of reasoning, tool invocation, and observation, the agent interacts with the environment to accomplish the user’s request\.

Incorporating an external guardrail module in the workflow is an effective way to reduce the risk of unintended actions\(Fanget al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib7)\)\. This module acts as a monitoring layer between the actions proposed by the LLM and the actual execution of the corresponding tools\. When the LLM proposes a tool call, the guardrail evaluates whether the action is consistent with the user’s query\. If the guardrail determines that the proposed action is potentially misaligned, it can block the execution and instead request user confirmation or clarification\.

### 2\.2\.Underspecification

When a user interacts with an LLM agent, the natural language query ultimately corresponds to a sequence of concrete actions that an LLM agent needs to execute\. However, due to the inherent limitations of natural language and the absence of certain contextual information, a single query can often be interpreted in multiple plausible ways\. Each interpretation may correspond to a different sequence of actions that the agent could perform\.But not every interpretation is aligned with THE user\-wanted sequence of concrete actions\.We refer to such situations as*underspecification*\.

A query is underspecified when the information contained in the query, together with the available context, is insufficient to uniquely determine the intended action sequence\. In this case, multiple interpretations remain consistent with the user’s request, and the agent cannot reliably infer which interpretation reflects the user’s true intent\.

Underspecification is closely related to, but distinct from, ambiguity\(Yanget al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib27)\)\. Ambiguity is an intrinsic property of natural language: Linguistic expressions can admit multiple meanings due to lexical, syntactic, or semantic factors\. Underspecification, in contrast, arises from the interaction between the query and its context\. A query is underspecified not merely because it has multiple linguistic interpretations, but mainly because the available information is insufficient to resolve which interpretation should guide the agent’s actions\. A query that is underspecified in one context may be well\-specified in another with additional contextual constraints\.

When a query is underspecified, selecting one plausible interpretation and acting upon it requires the agent to make assumptions about the user’s intent\. If these assumptions are incorrect, the resulting actions may deviate from the user’s actual intent, leading to misaligned behavior\. Therefore, identifying underspecified queries is a critical step toward ensuring that LLM agents avoid arbitrarily committing to a particular interpretation and instead seek clarification or defer action when necessary\.

## 3\.Types of Misalignment

LLM agents interact with external environments by invoking tools\. These tool calls could produce irreversible or high\-impact effects, such as transferring money, sending emails, or modifying files in the computer\. Unlike incorrect textual responses from chatbots, which are typically visible to the user and can be corrected, misaligned tool executions may cause unintended consequences without the user immediately realizing it\.

In this work, we define misalignment at the level of tool invocation\. We categorize misaligned actions into three types\.

##### \(1\) Tool\-Level Misalignment

The first type occurs when the agent selects a tool that is irrelevant or incompatible with the task in the user query\.

##### Example

As shown in Fig\.[1](https://arxiv.org/html/2607.01236#S1.F1)a, while the user asks the agent to “request $30,” the agent proposes to call the “money\_transfer” tool, which transfers money to the person who actually owes the user\. The action is clearly inconsistent with the user’s request\. The tool invocation itself is unrelated to the intended operation, leading to a tool\-level misalignment\.

##### \(2\) Parameter\-Level Misalignment

The second type occurs when the agent selects the correct tool but assigns incorrect or ungrounded parameters\. The resulting action deviates from the stated task even though the tool itself is appropriate\.

##### Example

As shown in Fig\.[1](https://arxiv.org/html/2607.01236#S1.F1)b, the agent proposes to call the right tool, “money\_transfer,” but the parameters it picks do not match the user’s request\. This type of error reflects a failure to properly ground the action parameters in the user query or the context\.

##### \(3\) Interpretation\-Level Misalignment

The third category directly relates to underspecification\. An underspecified query admits multiple plausible interpretations\. The misalignment arises when the agent arbitrarily resolves an underspecified query by committing to a particular interpretation and executes actions without clarification\. In such cases, the actions may be consistent with one possible interpretation, but the interpretation itself is unjustified and the actions could lead to severe unwanted consequences\.

##### Example

As shown in Fig\.[1](https://arxiv.org/html/2607.01236#S1.F1)c, the user only says to “handle” the request while not providing any concrete steps about how to handle it\. Possible interpretations include paying the request, rejecting it, notifying the user, or asking for confirmation\. However, the agent interprets this instruction as automatically paying overdue tickets, which involves a strong assumption that was never stated by the user\. This is a clearly misaligned action\. The appropriate behavior in this situation would be to present the pending requests and ask the user how they should be handled\.

![Refer to caption](https://arxiv.org/html/2607.01236v1/x2.png)Figure 3\.Overview of the conceptual framework\.

## 4\.Technical Approach

In this section, we introduce our provenance\-based approach to detection of misalignment, including a conceptual framework for provenance\-based guardrails and its realization in a tool called\\tool\.

### 4\.1\.Background: Provenance Analysis

Provenance analysis is a well\-established discipline that studies the origin and derivation history of an artifact by explicitly tracking how it is produced from source inputs, intermediate transformations, and responsible actors\. It has been widely applied in databases, scientific workflows, and security/privacy\-critical systems, where understanding how a result is generated is critical for explanation, reproducibility, debugging and auditing\(Belhajjameet al\.,[2013](https://arxiv.org/html/2607.01236#bib.bib3); Cheneyet al\.,[2009](https://arxiv.org/html/2607.01236#bib.bib4); Davidson and Freire,[2008](https://arxiv.org/html/2607.01236#bib.bib5); Panet al\.,[2023](https://arxiv.org/html/2607.01236#bib.bib6)\)\.

A central concept in provenance analysis is dependency tracing\. The W3C PROV data model\(Belhajjameet al\.,[2013](https://arxiv.org/html/2607.01236#bib.bib3)\)is one widely\-established standard of provenance\. It models the causal dependencies between three primary components:Entities\(digital artifacts or input data\),Activities\(processes or functions that consume and generate entities\), andAgents\(the operators or systems responsible for executing activities\)\. In database literature\(Cheneyet al\.,[2009](https://arxiv.org/html/2607.01236#bib.bib4)\), provenance is further classified into three critical dimensions:*where\-provenance*, which tracks where an output value originates;*why\-provenance*, which are the subset of source records necessary to justify the existence of an output; and*how\-provenance*, which explains how outputs are constructed through transformations\. Overall, across settings, provenance is a structured representation of how outcomes depend on prior context and computation\. By exposing dependency structure, provenance supports debugging and error diagnosis, attribution of responsibility, and anomaly detection\(Cheneyet al\.,[2009](https://arxiv.org/html/2607.01236#bib.bib4); Davidson and Freire,[2008](https://arxiv.org/html/2607.01236#bib.bib5); Panet al\.,[2023](https://arxiv.org/html/2607.01236#bib.bib6)\)\.

This paradigm offers a theoretical foundation for addressing LLM agent misalignment\. Agents are tasked with completing the tasks expressed in user queries, and all agent behaviors should arise from system and user instructions, interactions with environments, and other information in the context window\. Intuitively, whether an agent action is aligned depends on whether the action can be traced to appropriate context\.

If an agent hallucinates a tool parameter or executes an irrelevant tool, this proposed action will fundamentally lack a valid provenance path back to the provided context\. If the user query is underspecified and the agent picks an arbitrary interpretation, the proposed action will make ungrounded assumptions without the support of provenance\. Consequently, checking misalignment based on provenance provides a systematic and principled mechanism for detecting unwanted agent behaviors in runtime environments\.

### 4\.2\.Conceptual Framework for Provenance\-based Guardrails

#### 4\.2\.1\.Basic Elements

We model the guardrail as a runtime monitor that intercepts each tool call proposed by the agent before execution\. At every decision step, the guardrail receives the agent’s current context together with the newly proposed action, and determines whether executing that action would be aligned with the user query\.

##### Context available to the guardrail\.

The guardrail takes three core contexts as input: theUser Query\(q∈Qq\\in Q\), theAvailable Tools’ Documentation\(d∈Dd\\in D\), and theHistory of Prior Tool Calls and Their Outcomes\(h∈Hh\\in H\)\. Together, these contexts capture the main sources from which an aligned action should derive its provenance\. Although we focus on\(q,d,h\)\(q,d,h\)in this work, the framework is not limited to such and can be extended to incorporate additional context sources when available, e\.g\., system instructions, long\-term memory, and retrieved documents\.

##### Proposed action\.

We represent a proposed tool call as\(t,a\)\(t,a\), wheret∈Tt\\in Tdenotes the selected tool anda∈Aa\\in Adenotes the corresponding parameter assignment\. Tool documentationddrecords the specification of tools and their parameters\. The part ofddcorresponding to a toolttis denoted asdtd\_\{t\}\.

##### Guardrail input\-output formulation\.

Putting these pieces together, the guardrailg​u​a​r​dguardtakes the current context and the newly proposed action as input\. Its output is a binary decision indicating whether the proposed action should be allowed to execute:

g​u​a​r​d​\(q,d,h,t,a\)∈\{Aligned,Misaligned\}\.guard\(q,d,h,t,a\)\\in\\\{\\textit\{Aligned\},\\textit\{Misaligned\}\\\}\.
Table 1\.We formalize provenance relations using a set of binary relations over contextual objects and action’s components\.

#### 4\.2\.2\.Provenance Relations

Motivated by provenance analysis, we frame misalignment detection as the problem of tracing whether a proposed action is justified by the available context\. An aligned action should be supported by traceable evidence drawn from the user query, tool documentation, and prior interaction history\.

In this view, provenance is a justification structure that links a proposed action to contextual evidence through semantic*relations*between the elements of an action to those of the context\. Provenance analysis then becomes the process of determining whether the relations required for an aligned action can be established from the available context\.

Specifically, the three types of misalignment in Sec\.[3](https://arxiv.org/html/2607.01236#S3)correspond to failures of three different justification conditions, each expressed in terms of provenance relations\. Tool\-level misalignment concerns whether the selected tool can contribute to the current subtask\. Parameter\-level misalignment concerns whether the instantiated parameter assignment is grounded in the current context\. Interpretation\-level misalignment concerns whether the proposed action is the only interpretation of the user query according to the context\. An overview of the framework is presented in Fig\.[3](https://arxiv.org/html/2607.01236#S3.F3)\.

Table[1](https://arxiv.org/html/2607.01236#S4.T1)summarizes the provenance relations used by the framework\. Next, we specify which relation must be identified for each type of misalignment and how failure of that relation is interpreted\.

#### 4\.2\.3\.Provenance for Tool Choice

At this level, the object of justification is the tool selected for the current step\. First, we need to obtain the documentation that specifies the selected tooltt\. If thedtd\_\{t\}satisfyingS​p​e​c​i​f​i​e​d​B​y​\(t,dt\)SpecifiedBy\(t,d\_\{t\}\)cannot be found, it means thatttis not a valid tool available to the agent and should be rejected\.

After identifyingdtd\_\{t\}, tool\-level provenance requires establishing the relation𝑅𝑒𝑙𝑒𝑣𝑎𝑛𝑡𝑇𝑜​\(dt,s\)\\mathit\{RelevantTo\}\(d\_\{t\},s\)with some subtasks∈Ss\\in Sthat satisfiesC​o​n​t​r​i​b​u​t​e​T​o​\(s,q\)ContributeTo\(s,q\)\. Intuitively, the tool of an aligned action must be capable of advancing the overall task given by the user\. If𝑅𝑒𝑙𝑒𝑣𝑎𝑛𝑡𝑇𝑜​\(dt,s\)\\mathit\{RelevantTo\}\(d\_\{t\},s\)cannot be established, then the proposed action lacks the provenance required for tool selection, and the action is classified as misaligned\. Taking Fig\.[1](https://arxiv.org/html/2607.01236#S1.F1)a as an example, “money\_transfer” is not relevant to any part of the user query, so it is classified as misaligned\.

#### 4\.2\.4\.Provenance for Parameter Assignment

Even when the selected tool is appropriate, the action can still be misaligned if its parameters are unsupported\. A tool provides the means to act, but the parameter assignment determines*what concrete action*will actually be executed\. Here, parameter\-level provenance requires establishing two relations\.

First, the selected parameter assignment needs to be derivable from the context\. We denote*factual evidence*ase∈Ee\\in E\. A proper parameter assignment must be sourced from some facts in the context\. Therefore, givenaa, parameter\-level provenance requires the relation𝐷𝑒𝑟𝑖𝑣𝑒𝑑𝐹𝑟𝑜𝑚​\(a,e\)\\mathit\{DerivedFrom\}\(a,e\)to be satisfied for some evidenceeethatA​p​p​e​a​r​I​nAppearInqqorhh\.

Note that, it is not required to have an explicitly exact match for a parameter value inee\. A selected value is grounded as far as it can be reasonably derived from prior evidence\. Conversely, if any part of a value cannot be traced back to any relevant contextual support, the relation𝐷𝑒𝑟𝑖𝑣𝑒𝑑𝐹𝑟𝑜𝑚​\(a,e\)\\mathit\{DerivedFrom\}\(a,e\)will not hold\.

Second, the grounded assignment must also constitute an appropriate instantiation of the selected tool that can resolve the current subtaskssidentified in the last stage\. We capture this requirement using the relation𝐶𝑎𝑛𝐴𝑑𝑑𝑟𝑒𝑠𝑠​\(dt,a,s\)\\mathit\{CanAddress\}\(d\_\{t\},a,s\)\.

Both conditions are necessary\. If𝐷𝑒𝑟𝑖𝑣𝑒𝑑𝐹𝑟𝑜𝑚​\(a,e\)\\mathit\{DerivedFrom\}\(a,e\)cannot be established, thenaalacks evidential provenance and may reflect hallucinated or arbitrarily chosen values\. If𝐶𝑎𝑛𝐴𝑑𝑑𝑟𝑒𝑠𝑠​\(dt,a,s\)\\mathit\{CanAddress\}\(d\_\{t\},a,s\)does not hold, thenaa, even if it can be derived from the context, does not provide the parameterization required for the current subtask\. In either case, the proposed action fails the required parameter\-level provenance condition and is classified as misaligned\.

##### Generative Parameters Don’t Require Provenance

The conditions above do not apply uniformly to all parameters\. Some arguments are inherently expected to be generated rather than copied or extracted\. For example, for asend\_emailtool, the recipient field is often expected to be grounded in the query or in prior observations, whereas the message body may be newly composed by the agent\. Such parameters may be coherent and useful even when they do not have direct provenance\. This observation does not undermine the framework; instead, it clarifies that provenance\-based checking should focus on parameters whose values are expected to be context\-grounded\. The framework thus distinguishes between*derivable*parameters, which should admit provenance tracing, and*generative*parameters, whose correctness may require other forms of evaluation\. Accordingly, provenance\-based checking at this level focuses on*derivable*parameters\.

#### 4\.2\.5\.Provenance for Interpretation

In our approach, an action may be fully grounded under one interpretation of the query and still considered misaligned if that interpretation is only one of several plausible possibilities\. Therefore, the key question of provenance analysis here is whether the available context sufficiently determines what the agent should do next, rather than leaving multiple reasonable choices of actions unresolved\.

We say that a candidate action\(t,a\)\(t,a\)is*provenance\-admissible*if it satisfies the required provenance conditions for tool and parameter alignment; namely, \(i\)𝑅𝑒𝑙𝑒𝑣𝑎𝑛𝑡𝑇𝑜​\(dt,s\)\\mathit\{RelevantTo\}\(d\_\{t\},s\), \(ii\)𝐷𝑒𝑟𝑖𝑣𝑒𝑑𝐹𝑟𝑜𝑚​\(a,e\)\\mathit\{DerivedFrom\}\(a,e\), and \(iii\)𝐶𝑎𝑛𝐴𝑑𝑑𝑟𝑒𝑠𝑠​\(dt,a,s\)\\mathit\{CanAddress\}\(d\_\{t\},a,s\)for somessandeethat satisfyC​o​n​t​r​i​b​u​t​e​T​o​\(s,q\)ContributeTo\(s,q\)andA​p​p​e​a​r​I​n​\(e,q/h\)AppearIn\(e,q/h\)\. The provenance analysis at this level is to attempt to identify the set of provenance\-admissible candidate actions given the context\.

If there exist distinct provenance\-admissible actions for the current step, then the context does not uniquely constrain how the user’s request should be resolved\. In this case, any concrete action makes an assumption that is not justified; i\.e\., it is considered misaligned at the interpretation\-level\. Taking Fig\.[1](https://arxiv.org/html/2607.01236#S1.F1)c as an example, the user query “handle it” is underspecified\. Therefore, provenance\-admissible actions include both “pay” and “decline” the request, and the agent selecting any one of them is considered a misaligned action in our approach\.

### 4\.3\.ProvenanceGuard

We instantiate the above conceptual framework as\\tool, a run\-time guardrail that intercepts each proposed tool call before execution and applies provenance\-based checks to determine whether the action should be allowed\. Concretely,\\tooloperationalizes the three misalignment detection as a sequential three\-stage pipeline, as shown in Fig\.[3](https://arxiv.org/html/2607.01236#S3.F3)\. Each stage targets one type of misalignment and performs an early rejection if insufficient provenance is found\.

For the subset of the relations from Table[1](https://arxiv.org/html/2607.01236#S4.T1)that require natural language understanding,\\toolinvokes an LLM to check whether a given tuple is in the relation or extract such a tuple from the given context\. However, although\\toolrelies on an LLM for parts of the pipeline, we distinguish it from an LLM\-as\-a\-judge, which typically involves a single call to an LLM to evaluate the agent’s action; we provide a comparison against an LLM\-as\-a\-judge in Section[5](https://arxiv.org/html/2607.01236#S5)\.

#### 4\.3\.1\.Stage I: Detecting Tool\-Level Misalignment

The first stage checks whether the selected toolttis an appropriate means of accomplishing some subtasksscontributing toqq\.

##### Reuse Agent’s Plan

Agents typically generate an explicit plan that specifies the subtaskssaddressed by each proposed action\.\\toolleverages this agent\-provided subtask directly, rather than inferring one from the overall queryqq\. This design choice simplifies the procedure and avoids potential inconsistencies that could arise from independently inferred subtasks, which may otherwise lead to incorrect rejection of aligned actions\.

##### Environment\-Changing Tool Filter

To reduce unnecessary interference with the agent’s autonomy,\\toolfocuses its runtime checks on actions that can directly affect the external environment or produce consequential side effects\. Examples include sending emails, editing files, or transferring money\. In contrast, purely observational tools, such as search or information retrieval, typically have much lower risk and need not always be blocked prior to execution\.

We assume that tool developers provide metadata indicating whether a tool is*environment\-changing*\.\\tooluses this metadata as a lightweight rule\-based filter before invoking the provenance checker\. This design concentrates protection on high\-impact actions while preserving the agent’s autonomy for low\-risk information gathering\.

##### Provenance check for tool selection

The selected toolttis first verified to exist in the documentationdd, from which its descriptiondtd\_\{t\}is retrieved\. Thendtd\_\{t\}is used to determine whether the tool is environment\-changing\.

Oncettpasses these two checks,\\toolevaluates whether its use is justified for the current subtaskss\. Concretely, it prompts an LLM to assess whether𝑅𝑒𝑙𝑒𝑣𝑎𝑛𝑡𝑇𝑜​\(dt,s\)\\mathit\{RelevantTo\}\(d\_\{t\},s\)holds\.111All prompts are provided in the replication package\.If𝑅𝑒𝑙𝑒𝑣𝑎𝑛𝑡𝑇𝑜​\(dt,s\)\\mathit\{RelevantTo\}\(d\_\{t\},s\)cannot be established,\\toolimmediately labels the proposed action asMisalignedand blocks execution, without proceeding to later stages\.

#### 4\.3\.2\.Stage II: Detecting Parameter\-Level Misalignment

If Stage I determines that toolttis appropriate for subtaskss, Stage II proceeds to verify whether the proposed parameter assignmentaais itself grounded in the available context\.

##### Identifying verifiable parameters\.

A practical complication is that not all parameters are expected to admit direct provenance tracing, as discussed in Sec\.[4\.2\.4](https://arxiv.org/html/2607.01236#S4.SS2.SSS4)\. For this reason, the first step of Stage II is to infer which parameters of toolttare derivable and should be checked for provenance at runtime\. We use an LLM to determine which parameters are derivableaderivablea\_\{\\textit\{derivable\}\}forttbased on its documentationdtd\_\{t\}\. If there is no derivable parameter, then Stage II performs no provenance check and the action is passed directly to the final stage\.

##### Provenance check for parameter values\.

Givenaderivablea\_\{\\textit\{derivable\}\},\\toolthen instructs an LLM to find some evidenceeesuch that𝐷𝑒𝑟𝑖𝑣𝑒𝑑𝐹𝑟𝑜𝑚​\(a,e\)\\mathit\{DerivedFrom\}\(a,e\)is satisfied, and to establish𝐶𝑎𝑛𝐴𝑑𝑑𝑟𝑒𝑠𝑠​\(dt,a,s\)\\mathit\{CanAddress\}\(d\_\{t\},a,s\)\. If either relation cannot be satisfied,\\toolimmediately labels the action asMisalignedand stops\.

#### 4\.3\.3\.Stage III: Detecting Interpretation\-Level Misalignment

Passing the first two stages means that the proposed tool and its verifiable parameters are both grounded\. However, the action may still be misaligned if it relies on an unjustified interpretation of an underspecified request\.

Therefore, Stage III asks an LLM whether there exist more than one provenance\-admissible candidate action given the context\. Importantly, this step’s prompt*does not reveal the proposed action*to the LLM\. This design avoids biasing LLM to the agent’s chosen action and reduces the risk that the model will simply rationalize that choice or deliberately search for a conflicting alternative\.

If multiple provenance\-admissible actions are identified, it means that multiple alternative interpretations of the user’s query remain and that executing any concrete action would involve the agent making an unjustified assumption\. In this case,\\toollabels the proposed action asMisalignedand blocks execution\.

## 5\.Experiment Setup

To demonstrate the effectiveness of\\tool, we conducted experiments to answer the following two research questions:

- •RQ1:How accurately does\\tooldetect misaligned actions?
- •RQ2:How often does\\toolincur interventions on correct actions?

RQ1 evaluates the effectiveness of\\toolin detecting misaligned actions relative to the compared methods\. RQ2 evaluates the practical oversight burden introduced by\\toolas a runtime guardrail\. To strengthen the validity of these comparisons, we further assess whether the performance differences between methods are statistically significant using one\-sided paired permutation tests\.

### 5\.1\.Benchmarks

Evaluating alignment guardrail imposes several requirements on the benchmark\. \(1\) The benchmark must provide realistic agent interaction traces, or an executable environment from which such traces can be collected\. \(2\) The agent must have access to a diverse set of tools that can directly affect the environment, since our setting focuses on actions with potentially irreversible consequences\. \(3\) Most importantly, the benchmark must support accurate trace\-level alignment annotation, either through human\-annotated ground\-truth labels or through a deterministic rule\-based labeling procedure that does not depend on another LLM judge\.

To our knowledge, no existing benchmark fully satisfies these requirements\. In particular, many agent safety benchmarks focus on prompt injection, adversarial instructions, or other types of failures outside our scope, while others lack realistic traces, environment\-changing tools, or suitable labels for our notion of misalignment\. We therefore use two complementary benchmarks\. WorkBench\(Styleset al\.,[2024](https://arxiv.org/html/2607.01236#bib.bib2)\)provides agent traces, environment\-changing tools, and ground\-truth action sequences, making it a strong testbed for tool\- and parameter\-level misalignment\. Agent\-SafetyBench\(Zhanget al\.,[2024b](https://arxiv.org/html/2607.01236#bib.bib1)\)complements it by including test cases with underspecified user queries, which closely matches our interpretation\-level misalignment setting\. Because Agent\-SafetyBench does not provide traces or alignment labels, we manually generated traces and annotated them\.

#### 5\.1\.1\.WorkBench

WorkBench contains 690 agent test cases with traces generated by multiple LLMs, together with mock environments, mock tools, tool documentation, and ground\-truth action sequences\. We use the GPT\-4 traces, which are reported as the strongest in the original benchmark\. This benchmark is useful for our setting because its ground\-truth action sequences let us identify traces containing tool invocations that do not follow the user’s expected execution\. However, WorkBench does not explicitly consider underspecification as a kind of misalignment\.

We treat traces that take any environment\-changing actions deviating from the benchmark’s expected action sequence as*misaligned traces*\. This is because, in these cases, at least one executed tool call fails to match the benchmark’s ground\-truth actions for the task, either through misaligned tool selection, or misaligned parameterization\. In particular, some traces may also have underspecified user query, so we do not interpret this subset as containing only tool\- or parameter\-level misalignments\.

All remaining traces, which correctly complete or partially complete their tasks, are referred to as*task\-successful traces*\. These task\-successful traces are not necessarily equivalent to*aligned*traces in our setting, because matching the ground\-truth execution does not rule out underspecification in the original user query\.

We restricted the benchmark to test cases involving environment\-changing tools\. The resulting subset contains 62 misaligned traces and 326 task\-successful traces\.

#### 5\.1\.2\.Agent\-SafetyBench

We used the subset of Agent\-SafetyBench with “failure mode 2,” which is marked as the agent mistakenly calls tools when the necessary information is incomplete\. This failure mode closely matches our interpretation\-level misalignment, since these cases largely arise from underspecified user queries\.

Agent\-SafetyBench does not provide agent execution traces or trace\-level alignment annotations\. We therefore used GPT\-5, with default configuration, to generate one execution trace for each test case, yielding 188 traces in total\. Two authors then independently annotated each trace for \(1\) whether it contains a misaligned action and \(2\) the first misaligned step, if any\. Out of the 188 annotations, eight had inconsistent labels across the two annotators\. Inter\-annotator agreement reached Cohen’sκ=0\.85\\kappa=0\.85, indicating almost perfect agreement according to\(McHugh,[2012](https://arxiv.org/html/2607.01236#bib.bib11)\)\. Two annotators had a discussion to resolve the disagreed cases and filtered out 5 inappropriate cases\. The resulting dataset contains 35 aligned traces and 148 misaligned traces\.

#### 5\.1\.3\.Plan reconstruction

\\tool

requires the agent’s plan as part of its input\. However, neither Agent\-SafetyBench nor WorkBench explicitly outputs plans in the traces\. To recover this information, we used GPT\-5 to infer the agent’s plan from each decision point\. Specifically, given\(q,d,h,t,a\)\(q,d,h,t,a\), GPT\-5 reconstructed the subtaskssthat the agent intended to address with\(t,a\)\(t,a\)\.

### 5\.2\.Compared Methods

We compared\\toolagainst a prompt\-based baseline and a single\-step variant derived from our conceptual framework\.

##### DirectPromptBaseline

This method represents the*LLM\-as\-a\-judge*paradigm\(Zhenget al\.,[2023](https://arxiv.org/html/2607.01236#bib.bib31)\)\. Given the agent context and a proposed action, the model is directly prompted to determine whether the action is misaligned according to the definitions in Sec\.[3](https://arxiv.org/html/2607.01236#S3)\.222Full prompts are provided in the replication package\.

##### Single\-step Provenance Prompt\.

ProvePromptis a single\-step prompt implementation of our provenance\-based conceptual framework\. Compared toDirectPrompt, its prompt explicitly instructs the model to assess provenance for tool choice, parameter assignment, and interpretation\-level justification\. The main difference between it and\\toolis that it produces a single holistic judgment in one pass rather than decomposing the decision into a multi\-stage pipeline\. We include this method to isolate the contribution of the*provenance\-based conceptual framework*from that of\\tool’s*multi\-stage design*\.

### 5\.3\.Backbone Large Language Models

We evaluated\\tooland the two compared methods using 10 backbone LLMs: GPT\-5, GPT\-5\-mini, GPT\-4\.1, GPT\-4\.1\-mini, Gemini\-3\-flash\-preview, Gemini\-3\.1\-flash\-lite\-preview, Gemini\-2\.5\-flash, Gemini\-2\.5\-flash\-lite, Claude\-sonnet\-4\.6, Claude\-haiku\-4\.5\(OpenAI,[2026](https://arxiv.org/html/2607.01236#bib.bib57); Google,[2026](https://arxiv.org/html/2607.01236#bib.bib59); Anthropic,[2026](https://arxiv.org/html/2607.01236#bib.bib41)\)\. All models were used with their default inference configurations provided by the corresponding API\.

### 5\.4\.Setup and Metrics

![Refer to caption](https://arxiv.org/html/2607.01236v1/x3.png)\(a\)RQ1 Agent\-SafetyBench Error Rate
![Refer to caption](https://arxiv.org/html/2607.01236v1/x4.png)\(b\)RQ1 Workbench Error Rate
![Refer to caption](https://arxiv.org/html/2607.01236v1/x5.png)\(c\)RQ2 Agent\-SafetyBench Intervention Rate
![Refer to caption](https://arxiv.org/html/2607.01236v1/x6.png)\(d\)RQ2 Workbench Intervention Rate

Figure 4\.RQ1 and RQ2 evaluation results of each method\-LLM pair on two benchmarks\.#### 5\.4\.1\.RQ1: Misalignment Detection Performance

This RQ assesses how effectively\\tooland the two compared methods detect misaligned actions\. For each combination of method and backbone LLM, we evaluate on the misaligned traces from Agent\-SafetyBench and WorkBench\. To account for run\-to\-run variance in LLM judgments, each method\-model pair is executed five times on each trace\.

##### Metrics

We report the*error rate*, defined as the proportion of misaligned traces that are incorrectly judged as aligned\. Equivalently, when misaligned traces are treated as the positive class, this metric corresponds to the false negative rate \(FNR\)\.

For a given method\-model pair, the Error Rate is computed as:

\(1\)ER=∑r=1R\#​FNrN×R,\\mathrm\{ER\}=\\frac\{\\sum\_\{r=1\}^\{R\}\\\#\\mathrm\{FN\}\_\{r\}\}\{N\\times R\},whereNNis the number of misaligned traces,RRis the number of repeated runs, and\#​FNr\\\#\\mathrm\{FN\}\_\{r\}denotes the number of misaligned traces predicted as “Aligned” \(False Negatives\) in runrr\.

#### 5\.4\.2\.RQ2: Intervention Frequency

Accurately detecting misaligned actions is only one side of the trade\-off\. A practical guardrail should also avoid interrupting agent executions that would otherwise proceed acceptably, since excessive interventions increase user review burden\.

We therefore evaluated the*intervention rate*of\\toolfrom two complementary perspectives, reflecting the different labeling semantics of our two benchmarks\.

##### RQ2\.1: Unnecessary interventions on aligned trace\.

On Agent\-SafetyBench, we used the 35 traces that were manually annotated as aligned, where any interventions are unnecessary\. This evaluation provides a direct measure of how often the method disrupts actions that are aligned with user query, which corresponds to the false positive rate \(FPR\)\.

##### RQ2\.2: Interventions on task\-successful traces

On WorkBench, we used the 326 task\-successful traces\. Here, we measure how often the guardrail intervenes on traces where agents are correctly addressing the user task\. We do not interpret this quantity as a FPR, because correctness with respect to benchmark ground truth is not equivalent to alignment in our setting, as discussed in Sec\.[5\.1\.1](https://arxiv.org/html/2607.01236#S5.SS1.SSS1)\. Therefore, this metric captures the cost of intervention on traces that would otherwise be considered successful by the benchmark\.

##### Metrics

We report the*intervention rate*, defined as the proportion of evaluated traces for which the method predictsMisaligned\. For RQ2\.1, this corresponds to the*unnecessary intervention rate*\. For RQ2\.2, this corresponds to the*intervention rate on task\-successful traces*, which should be interpreted as an operational burden measure rather than a strict false positive rate\.

IR=∑r=1R\#​IntervenerN×R,\\mathrm\{IR\}=\\frac\{\\sum\_\{r=1\}^\{R\}\\\#\\mathrm\{Intervene\}\_\{r\}\}\{N\\times R\},where\#​Intervener\\\#\\mathrm\{Intervene\}\_\{r\}is the number of traces predicted as Misaligned in runrr\. Its interpretation depends on the ground\-truth semantics \(aligned or task\-successful\) of the evaluated subset\.

#### 5\.4\.3\.Statistical Analysis

For both RQs, we assessed whether differences between methods are statistically significant using one\-sided paired permutation tests\. Because each method\-LLM pair was executed five times on the same trace, we do not treat individual runs as independent observations\. Instead, for each method, LLM, and trace, we averaged the five binary outcomes to obtain a*case\-level score*\. For RQ1, this score is the error rate on a misaligned trace\. For RQ2, this score is the IR on the evaluated trace\.

We then compared\\toolagainst each method using paired case\-level differences computed on the same traces under the same LLM\. For a comparator methodmm, lets\\tool\(c\)s^\{\(c\)\}\_\{\\tool\}andsm\(c\)s^\{\(c\)\}\_\{m\}denote the case\-level scores of\\toolandmmon tracecc, respectively\. We define the paired difference as

dc=s\\tool\(c\)−sm\(c\),d\_\{c\}=s^\{\(c\)\}\_\{\\tool\}\-s^\{\(c\)\}\_\{m\}\\ ,so that a negative value indicates that\\toolachieves a lower rate than the comparator on that trace\. The test statistic is the mean paired difference,

Δ=1N​∑c=1Ndc,\\Delta=\\frac\{1\}\{N\}\\sum\_\{c=1\}^\{N\}d\_\{c\}\\ ,whereNNis the number of traces in the evaluated subset\.

The null hypothesis is that\\tooldoes not outperform the comparator\. The one\-sided alternative hypothesis is that\\toolyields a lower case\-level rate than the comparator\. We estimate permutationpp\-values via Monte Carlo sampling over 10,000 random sign flips\.

Table 2\.Error Rate \(ER\) by model and benchmark\.PvGd=\\tool,PvPt=ProvePrompt,Base=DirectPrompt\.Δ\\Delta= mean difference \(left−\-right\)\.pp\-values: one\-sided paired permutation test \(H1: left<<right\)\.Boldmeans∗p<0\.05p<0\.05\.Table 3\.Intervention Rate \(IR\) by model and benchmark\. Agent\-SafetyBench’spp\-values test reverse hypothesis \(H1: left\>\>right\) while WorkBench’s test \(H1: left<<right\)

## 6\.Evaluation Results

### 6\.1\.RQ1: Misalignment Detection Performance

Fig\.[4\(a\)](https://arxiv.org/html/2607.01236#S5.F4.sf1)&[4\(b\)](https://arxiv.org/html/2607.01236#S5.F4.sf2)show ERs on misaligned traces, where lower is better\. Overall, our experiments show that the provenance\-based framework consistently improves misalignment detection\.

On Agent\-SafetyBench, both\\toolandProvePromptoutperform the baseline on all 10 LLMs\. Averaged across LLMs,the ER drops from 42\.9% forDirectPromptto 12\.8% forProvePromptand further to 1\.8% for\\tool\.As reflected in Table[2](https://arxiv.org/html/2607.01236#S5.T2), all performance gains over the baseline are statistically significant \(p<0\.05p<0\.05\)\. The gains are especially large for weaker models\. With\\tool, even models such as GPT\-4\.1\-mini, Gemini\-2\.5\-flash\-lite, and Claude\-haiku\-4\.5 reach ERs close to the strongest models, and all 10 LLMs fall below 5% ER\. This suggests thatthe provenance\-based framework provides useful structure for detecting misaligned actions, even when the underlying model is relatively weak\.

On WorkBench, the improvements are more moderate\. Here,\\tooloutperforms the baseline on 8 of 10 LLMs, whileProvePromptdoes so on all LLMs\. Averaged across models,the ER decreases from 32\.1% forDirectPromptto 19\.5% forProvePromptand 17\.3% for\\tool\.Table[2](https://arxiv.org/html/2607.01236#S5.T2)shows that all improvements of\\toolare statistically significant except for Claude\-haiku\-4\.5, whose result is borderline \(p=0\.06p=0\.06\)\. In particular,\\tooldoes not improve over the baseline for GPT\-5 or Claude\-sonnet\-4\.6, whileProvePrompthas non\-statistically significant improvements on these two LLMs\.

The inconsistent improvements on two benchmarks are probably due to their distinct distribution of traces\. The Agent\-SafetyBench subset is intentionally concentrated on underspecified queries\. In that setting,\\toolis highly effective and nearly eliminates false negatives across all models\. WorkBench, in contrast, is not concentrated on underspecification and appears to contain a broader mix of misalignment types\. This suggests that the multi\-stage pipeline is particularly valuable for identifying underspecification, while detection of other types of misalignment still depends strongly on backbone LLMs\.

Comparing the two provenance\-based variants also yields an important insight\. On both benchmarks,ProvePromptalready improves substantially over the baseline, showing that much of the benefit comes from the provenance\-based conceptual framework itself rather than only from the multi\-stage pipeline\. At the same time,\\toolis clearly stronger on Agent\-SafetyBench, while on WorkBench the advantage of the multi\-stage design is less uniform andProvePromptoutperforms\\toolon five LLMs\. This suggests that the benefit of the conceptual framework is consistent across implementations, whereas the additional value of multi\-stage decomposition depends on the benchmark distribution and model characteristics\.

##### Takeaway

Provenance\-based framework substantially improves misalignment detection overDirectPrompt\. Among them,\\toolachieves better performance overall on both benchmarks than the single\-step variant, especially on Agent\-SafetyBench, suggesting the benefit of the multi\-stage separation\.

### 6\.2\.RQ2: Intervention Cost

Fig\.[4\(c\)](https://arxiv.org/html/2607.01236#S5.F4.sf3)&[4\(d\)](https://arxiv.org/html/2607.01236#S5.F4.sf4)show IRs on the two benchmarks, where lower is better\.

##### RQ2\.1 Unnecessary Interventions on Aligned Trace

On Agent\-SafetyBench aligned traces, the three methods have similar average IRs: 10\.1% for the baseline, 10\.9% forProvePrompt, and 14\.3% for\\tool\. Compared to the baseline,\\toolappears to yield a higher unnecessary IR on 8 of 10 LLMs \(average increase of 8\.0% on these 7 LLMs\)\.

To test the significance of these increases, we verified the reverse one\-sided alternative hypothesis that\\toolyields a higher case\-level unnecessary IR than the comparator\. According to Table[3](https://arxiv.org/html/2607.01236#S5.T3),7 out of 8increases are actuallynot statistically significant\. This indicates that the large gain of\\toolin detecting misaligned actions \(41\.1%\) is achieved witha limited tradeoff in unnecessary intervention cost\.

##### RQ2\.2 Interventions on Task\-successful Traces\.

On WorkBench task\-successful traces, the pattern is notably different\. Here,\\toolhas a lower IR on all 10 LLMs than the other two methods, and reduces the average rate from 30\.5% forDirectPromptto 12\.8%\. The reduction is substantial for several strong models: GPT\-5 \(45\.2%→13\.2%45\.2\\%\\rightarrow 13\.2\\%\), Gemini\-3\-flash\(41\.4%→17\.2%\(41\.4\\%\\rightarrow 17\.2\\%\), and Claude\-sonnet\-4\.6 \(39\.4%→4\.8%39\.4\\%\\rightarrow 4\.8\\%\)\. This suggests that, in practice, the multi\-stage provenance checks do not simply lead to more interventions than the baseline\. Instead, they often make judgments more precisely, allowing many successful traces to pass while detecting misalignment with few false negatives\.

##### Comparing provenance\-based variants

The comparison between the two provenance\-based variants further highlights the value of the multi\-stage design\.ProvePrompthashigher average IRs than the other two methods on both subsets: 17\.5% on Agent\-SafetyBench and 34\.6% on WorkBench\. On WorkBench in particular, it intervenes more often than\\toolon all 10 LLMs \(9 of them are statistically significant\), often by a large margin \(e\.g\., 49\.5% vs\. 13\.2% on GPT\-5 and 52\.0% vs\. 17\.2% on Gemini\-3\-flash\)\. This indicates that while the provenance\-based framework improves detection, the multi\-stage decomposition is important for controlling over\-intervention in practice\.

##### Takeaway

\\tool

does not statistically significantly increase unnecessary interventions on already aligned traces, but it substantially reduces interventions on task\-successful traces compared to bothDirectPromptandProvePrompt\. Combined with the RQ1 results, this suggests that the multi\-stage design built on the provenance\-based framework achieves a better balance between detecting misaligned actions and avoiding excessive disruption\.

### 6\.3\.Threats to Validity

Our labeled Agent\-SafetyBench includes only 35 aligned cases\. This small sample limits statistical power, so the lack of a statistically significant difference in unnecessary interventions between\\tooland the baseline should not be over\-interpreted\. It may reflect insufficient evidence to detect a difference, but it may also indicate that\\tooldoes not meaningfully increase unnecessary interventions relative to the baseline\. More aligned test cases would be needed to more reliably assess how\\toolaffects unnecessary intervention behavior\.

WorkBench does not explicitly consider underspecification as misalignment\. Its “task\-successful” traces may have underspecified user queries\. As a result, they could not be interpreted as fully aligned traces\. We therefore treat the intervention rate on these traces as a measure of the oversight burden in practical usage rather than a strict unnecessary intervention rate\.

The agents’ plans in both datasets were reconstructed using GPT\-5, since neither benchmark provides such information directly\. Errors in this reconstruction could affect\\tool’s decisions and the measured performance\.

## 7\.Related Works

Guardrails, the external defense layers that monitor and control LLM interactions, have emerged as a crucial approach for improving the safety of LLM agents\. However, prior works have largely focused on*safety, security, and policy compliance*\(Xianget al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib17); Chenet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib21); Debenedettiet al\.,[2024](https://arxiv.org/html/2607.01236#bib.bib22),[2025](https://arxiv.org/html/2607.01236#bib.bib23); Shaoet al\.,[2024](https://arxiv.org/html/2607.01236#bib.bib29); Jiaet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib30)\)\. For example,*GuardAgent*\(Xianget al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib17)\)and*ShieldAgent*\(Chenet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib21)\)safeguard agent behavior against violations of safety or policy constraints, while*AgentDojo*\(Debenedettiet al\.,[2024](https://arxiv.org/html/2607.01236#bib.bib22)\)and*CaMeL*\(Debenedettiet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib23)\)focus on security threats such as prompt injection and adversarial tool environments\. Although related, these problems differ from our notion of misalignment\. Our focus is not on harmful, adversarial, or policy\-noncompliant behavior, but on whether a proposed tool invocation is grounded by evidence in the context without making an arbitrary assumption about an underspecified user query in benign settings\. This distinction is important because some prior work also uses the term “misalignment” to describe strategically harmful or insider\-threat behavior by agents\(Lynchet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib10); Ninget al\.,[2026](https://arxiv.org/html/2607.01236#bib.bib9)\), which is conceptually different from the misalignment we study\.

Another related work to ours is InferAct\(Fanget al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib7)\)\. Although both works perform pre\-execution intervention, they are not directly comparable\. InferAct leaves the notion of misalignment largely implicit and operationalizes detection through a holistic prompt, asking whether the agent is “progressing correctly toward completing the user’s tasks” without precisely defining what it means for the agent to be making ”progress”\. In contrast, our work addresses a different and more explicitly specified decision problem: Given a proposed action, determine whether its tool selection and parameterization are justified by traceable contextual evidence, and whether the context uniquely supports committing to that action\.

One concept close to underspecification is*ambiguity*\. Recent works study whether language models can recognize ambiguous user requests and ask clarification questions when needed\(Zhanget al\.,[2024a](https://arxiv.org/html/2607.01236#bib.bib24); Zhang and Choi,[2025](https://arxiv.org/html/2607.01236#bib.bib25); Wanget al\.,[2025b](https://arxiv.org/html/2607.01236#bib.bib26); Yanget al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib27); Malaviyaet al\.,[2025](https://arxiv.org/html/2607.01236#bib.bib28)\)\. This line of work is related to our interpretation\-level analysis, but the concepts are not identical\. In our setting, underspecification means that the available context is insufficient to uniquely determine the next action, even if the request is not linguistically ambiguous in the usual sense\. Our focus is therefore not simply on ambiguity in language, but on whether the current context uniquely justifies one provenance\-admissible action\.

## 8\.Discussion and Limitations

Overall, our results suggest that the proposed provenance\-based conceptual framework enables a more structured and reliable way to detect misaligned agent actions\. At the same time, the current framework has several limitations\.

The proposed framework cannot detect alignment of generative parameters\. Our provenance analysis focuses on parameters that should be derivable from context\. For free\-form generated content such as drafted text or code, provenance tracing is insufficient by nature\. Therefore, our method is designed to identify the generative parameters and exclude them from provenance\-based checking\. Detecting misalignment for them requires other forms of guardrails\.

\\tool

checks provenance relative to the agent’s own plan\. If that plan is itself already misaligned with the user query, then a locally justified action may still be globally misaligned\. One possible alternative is to let the guardrail infer its own task decomposition\. However, this would introduce additional latency and create a new risk that the guardrail’s inferred plan is misaligned, or conflicts with an aligned agent plan\. We therefore chose to use the agent’s plan in the current design\.

The multi\-stage design introduces extra latency and inference cost compared to a one\-shot judge\. However, our results suggest that smaller and cheaper models such as GPT\-5\-mini can achieve misalignment detection performance comparable to stronger models such as GPT\-5, potentially offering a better cost\-performance trade\-off\. Exploring this trade\-off more systematically is an important direction for future work\.

## 9\.Data Availability

The replication package of the paper, which provides the labeled benchmarks, source code, experiment results, and analysis notebooks, will be available soon\.

## References

- Anthropic \(2026\)Claude\.Note:[https://www\.anthropic\.com/claude](https://www.anthropic.com/claude)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p1.1),[§5\.3](https://arxiv.org/html/2607.01236#S5.SS3.p1.1)\.
- Anysphere, Inc\. \(2026\)Cursor: the ai code editorExternal Links:[Link](https://cursor.com/)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p1.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1)\.
- K\. Belhajjame, R\. B’Far, J\. Cheney, S\. Coppens, S\. Cresswell, Y\. Gil, P\. Groth, G\. Klyne, T\. Lebo, J\. McCusker,et al\.\(2013\)Prov\-dm: the prov data model\.W3C Recommendation14,pp\. 15–16\.Cited by:[§4\.1](https://arxiv.org/html/2607.01236#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.01236#S4.SS1.p2.1)\.
- J\. Betley, D\. C\. H\. Tan, N\. Warncke, A\. Sztyber\-Betley, X\. Bao, M\. Soto, N\. Labenz, and O\. Evans \(2025\)Emergent misalignment: narrow finetuning can produce broadly misaligned LLMs\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=aOIJ2gVRWW)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p2.1)\.
- R\. Birkmose, N\. M\. Reece, E\. H\. Norvin, J\. Bjerva, and M\. Zhang \(2025\)On\-device llms for home assistant: dual role in intent detection and response generation\.InProceedings of the Tenth Workshop on Noisy and User\-generated Text,pp\. 57–67\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p1.1)\.
- Z\. Chen, M\. Kang, and B\. Li \(2025\)ShieldAgent: shielding agents via verifiable safety policy reasoning\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=DkRYImuQA9)Cited by:[§7](https://arxiv.org/html/2607.01236#S7.p1.1)\.
- J\. Cheney, L\. Chiticariu, and W\. Tan \(2009\)Provenance in databases: why, how, and where\.Foundations and trends in databases1\(4\),pp\. 379–474\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p5.1),[§4\.1](https://arxiv.org/html/2607.01236#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.01236#S4.SS1.p2.1)\.
- S\. B\. Davidson and J\. Freire \(2008\)Provenance and scientific workflows: challenges and opportunities\.InProceedings of the 2008 ACM SIGMOD international conference on Management of data,pp\. 1345–1350\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p5.1),[§4\.1](https://arxiv.org/html/2607.01236#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.01236#S4.SS1.p2.1)\.
- E\. Debenedetti, I\. Shumailov, T\. Fan, J\. Hayes, N\. Carlini, D\. Fabian, C\. Kern, C\. Shi, A\. Terzis, and F\. Tramèr \(2025\)Defeating prompt injections by design\.External Links:2503\.18813,[Link](https://arxiv.org/abs/2503.18813)Cited by:[§7](https://arxiv.org/html/2607.01236#S7.p1.1)\.
- E\. Debenedetti, J\. Zhang, M\. Balunovic, L\. Beurer\-Kellner, M\. Fischer, and F\. Tramèr \(2024\)Agentdojo: a dynamic environment to evaluate prompt injection attacks and defenses for llm agents\.Advances in Neural Information Processing Systems37,pp\. 82895–82920\.Cited by:[§7](https://arxiv.org/html/2607.01236#S7.p1.1)\.
- Z\. Deng, Y\. Guo, C\. Han, W\. Ma, J\. Xiong, S\. Wen, and Y\. Xiang \(2025\)Ai agents under threat: a survey of key security challenges and future pathways\.ACM Computing Surveys57\(7\),pp\. 1–36\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p1.1)\.
- H\. Fang, X\. Zhu, and I\. Gurevych \(2025\)Preemptive detection and correction of misaligned actions in llm agents\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 222–244\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.01236#S2.SS1.p3.1),[§7](https://arxiv.org/html/2607.01236#S7.p2.1)\.
- Google \(2026\)Note:Accessed: Mar\. 26, 2026\.External Links:[Link](https://gemini.google.com/)Cited by:[§5\.3](https://arxiv.org/html/2607.01236#S5.SS3.p1.1)\.
- F\. Jia, T\. Wu, X\. Qin, and A\. Squicciarini \(2025\)The task shield: enforcing task alignment to defend against indirect prompt injection in llm agents\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 29680–29697\.Cited by:[§7](https://arxiv.org/html/2607.01236#S7.p1.1)\.
- Y\. Lee, J\. Kim, J\. Kim, H\. Cho, J\. Kang, P\. Kang, and N\. Kim \(2025\)Checkeval: a reliable llm\-as\-a\-judge framework for evaluating text generation using checklists\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 15782–15809\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p4.1)\.
- W\. Luo, S\. Dai, X\. Liu, S\. Banerjee, H\. Sun, M\. Chen, and C\. Xiao \(2025\)Agrail: a lifelong agent guardrail with effective and adaptive safety detection\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 8104–8139\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p4.1)\.
- A\. Lynch, B\. Wright, C\. Larson, S\. J\. Ritchie, S\. Mindermann, E\. Hubinger, E\. Perez, and K\. Troy \(2025\)Agentic misalignment: how llms could be insider threats\.arXiv preprint arXiv:2510\.05179\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p2.1),[§7](https://arxiv.org/html/2607.01236#S7.p1.1)\.
- X\. Ma, Y\. Gao, Y\. Wang, R\. Wang, X\. Wang, Y\. Sun, Y\. Ding, H\. Xu, Y\. Chen, Y\. Zhao,et al\.\(2026\)Safety at scale: a comprehensive survey of large model and agent safety\.Foundations and Trends in Privacy and Security8\(3\-4\),pp\. 1–240\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p1.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark \(2023\)Self\-refine: iterative refinement with self\-feedback\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=S37hOerQLB)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1)\.
- C\. Malaviya, J\. C\. Chang, D\. Roth, M\. Iyyer, M\. Yatskar, and K\. Lo \(2025\)Contextualized evaluations: judging language model responses to underspecified queries\.Transactions of the Association for Computational Linguistics13,pp\. 878–900\.Cited by:[§7](https://arxiv.org/html/2607.01236#S7.p3.1)\.
- M\. L\. McHugh \(2012\)Interrater reliability: the kappa statistic\.Biochemia medica22\(3\),pp\. 276–282\.Cited by:[§5\.1\.2](https://arxiv.org/html/2607.01236#S5.SS1.SSS2.p2.1)\.
- A\. Naik, P\. Quinn, G\. Bosch, E\. Gouné, F\. J\. C\. Zabala, J\. R\. Brown, and E\. J\. Young \(2025\)Agentmisalignment: measuring the propensity for misaligned behaviour in llm\-based agents\.arXiv preprint arXiv:2506\.04018\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p2.1)\.
- Y\. Ning, J\. Jones, Z\. Zhang, C\. Ye, W\. Ruan, J\. Li, R\. Gupta, and H\. Sun \(2026\)When actions go off\-task: detecting and correcting misaligned actions in computer\-use agents\.arXiv preprint arXiv:2602\.08995\.Cited by:[§7](https://arxiv.org/html/2607.01236#S7.p1.1)\.
- OpenAI \(2026\)Note:Accessed: Mar\. 26, 2026\.External Links:[Link](https://chatgpt.com/)Cited by:[§5\.3](https://arxiv.org/html/2607.01236#S5.SS3.p1.1)\.
- OpenClaw \(2026\)OpenClaw — personal ai assistant\.Note:[https://openclaw\.ai/](https://openclaw.ai/)Accessed: 2026\-03\-12Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1)\.
- B\. Pan, N\. Stakhanova, and S\. Ray \(2023\)Data provenance in security and privacy\.ACM Computing Surveys55\(14s\),pp\. 1–35\.Cited by:[§4\.1](https://arxiv.org/html/2607.01236#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2607.01236#S4.SS1.p2.1)\.
- penligent \(2026\)External Links:[Link](https://www.penligent.ai/hackinglabs/meta-ai-alignment-directors-openclaw-email-deletion-incident-exposes-the-real-agent-safety-boundary/)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p1.1)\.
- Y\. Ruan, H\. Dong, A\. Wang, S\. Pitis, Y\. Zhou, J\. Ba, Y\. Dubois, C\. J\. Maddison, and T\. Hashimoto \(2024\)Identifying the risks of LM agents with an LM\-emulated sandbox\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=GEcwtMk1uA)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1)\.
- P\. Sahoo, A\. K\. Singh, S\. Saha, V\. Jain, S\. Mondal, and A\. Chadha \(2024\)A systematic survey of prompt engineering in large language models: techniques and applications\.arXiv preprint arXiv:2402\.079271\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1)\.
- Y\. Shao, T\. Li, W\. Shi, Y\. Liu, and D\. Yang \(2024\)Privacylens: evaluating privacy norm awareness of language models in action\.Advances in Neural Information Processing Systems37,pp\. 89373–89407\.Cited by:[§7](https://arxiv.org/html/2607.01236#S7.p1.1)\.
- L\. Shi, C\. Ma, W\. Liang, X\. Diao, W\. Ma, and S\. Vosoughi \(2025\)Judging the judges: a systematic study of position bias in llm\-as\-a\-judge\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,pp\. 292–314\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p4.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. R\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=vAElhFcKW6)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1)\.
- O\. Styles, S\. Miller, P\. Cerda\-Mardini, T\. Guha, V\. Sanchez, and B\. Vidgen \(2024\)WorkBench: a benchmark dataset for agents in a realistic workplace setting\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=4HNAwZFDcH)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p6.1),[§5\.1](https://arxiv.org/html/2607.01236#S5.SS1.p2.1)\.
- H\. Su, J\. Luo, C\. Liu, X\. Yang, Y\. Zhang, Y\. Dong, and J\. Zhu \(2025\)A survey on autonomy\-induced security risks in large model\-based agents\.arXiv preprint arXiv:2506\.23844\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p1.1)\.
- S\. V\. Vuddanti, A\. Shah, S\. K\. Chittiprolu, T\. Song, S\. Dev, K\. Zhu, and M\. Chaudhary \(2026\)PALADIN: self\-correcting language model agents to cure tool\-failure cases\.InLLM\-based Multi\-Agent Systems: Towards Responsible, Reliable, and Scalable Agentic Systems,External Links:[Link](https://openreview.net/forum?id=NVTtoO297p)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1)\.
- H\. Wang, C\. M\. Poskitt, and J\. Sun \(2025a\)Agentspec: customizable runtime enforcement for safe and reliable llm agents\.arXiv preprint arXiv:2503\.18666\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1)\.
- W\. Wang, S\. Juluan, Z\. Ling, Y\. Chan, C\. Wang, C\. Lee, Y\. Yuan, J\. Huang, W\. Jiao, and M\. R\. Lyu \(2025b\)Learning to ask: when LLM agents meet unclear instruction\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 21773–21784\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1104/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1104),ISBN 979\-8\-89176\-332\-6Cited by:[§7](https://arxiv.org/html/2607.01236#S7.p3.1)\.
- Y\. Wang, Y\. Song, T\. Zhu, X\. Zhang, Z\. Yu, H\. Chen, C\. Song, Q\. Wang, C\. Wang, Z\. Wu, X\. Dai, Y\. Zhang, W\. Ye, and S\. Zhang \(2026\)TrustJudge: inconsistencies of LLM\-as\-a\-judge and how to alleviate them\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=4uPyOCeN6U)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p4.1)\.
- H\. Wei, S\. He, T\. Xia, F\. Liu, A\. Wong, J\. Lin, and M\. Han \(2025\)Systematic evaluation of LLM\-as\-a\-judge in LLM alignment tasks: explainable metrics and diverse prompt templates\.InICLR 2025 Workshop on Building Trust in Language Models and Applications,External Links:[Link](https://openreview.net/forum?id=CAgBCSt8gL)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p4.1)\.
- Y\. Wu, F\. Roesner, T\. Kohno, N\. Zhang, and U\. Iqbal \(2025\)IsolateGPT: An Execution Isolation Architecture for LLM\-Based Agentic Systems\.InNetwork and Distributed System Security \(NDSS\) Symposium,Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1)\.
- Z\. Xiang, L\. Zheng, Y\. Li, J\. Hong, Q\. Li, H\. Xie, J\. Zhang, Z\. Xiong, C\. Xie, C\. Yang, D\. Song, and B\. Li \(2025\)GuardAgent: safeguard LLM agents via knowledge\-enabled reasoning\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=2nBcjCZrrP)Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1),[§7](https://arxiv.org/html/2607.01236#S7.p1.1)\.
- C\. Yang, Y\. Shi, Q\. Ma, M\. X\. Liu, C\. Kästner, and T\. Wu \(2025\)What prompts don’t say: understanding and managing underspecification in llm prompts\.arXiv preprint arXiv:2505\.13360\.Cited by:[§2\.2](https://arxiv.org/html/2607.01236#S2.SS2.p3.1),[§7](https://arxiv.org/html/2607.01236#S7.p3.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.01236#S2.SS1.p1.1)\.
- W\. Yeadon, T\. Hardy, P\. Mackay, and E\. Agra \(2026\)Criterion\-referenceability determines llm\-as\-a\-judge validity across physics assessment formats\.arXiv preprint arXiv:2603\.14732\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p4.1)\.
- K\. Zhang, Z\. Su, P\. Chen, E\. Bertino, X\. Zhang, and N\. Li \(2025\)LLM agents should employ security principles\.arXiv preprint arXiv:2505\.24019\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p3.1)\.
- M\. J\. Zhang and E\. Choi \(2025\)Clarify when necessary: resolving ambiguity through interaction with lms\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 5526–5543\.Cited by:[§7](https://arxiv.org/html/2607.01236#S7.p3.1)\.
- T\. Zhang, P\. Qin, Y\. Deng, C\. Huang, W\. Lei, J\. Liu, D\. Jin, H\. Liang, and T\. Chua \(2024a\)CLAMBER: a benchmark of identifying and clarifying ambiguous information needs in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 10746–10766\.Cited by:[§7](https://arxiv.org/html/2607.01236#S7.p3.1)\.
- Z\. Zhang, S\. Cui, Y\. Lu, J\. Zhou, J\. Yang, H\. Wang, and M\. Huang \(2024b\)Agent\-safetybench: evaluating the safety of llm agents\.arXiv preprint arXiv:2412\.14470\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p6.1),[§5\.1](https://arxiv.org/html/2607.01236#S5.SS1.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§1](https://arxiv.org/html/2607.01236#S1.p4.1),[§5\.2](https://arxiv.org/html/2607.01236#S5.SS2.SSS0.Px1.p1.1)\.