When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry

# When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry
Source: [https://arxiv.org/html/2608.14680](https://arxiv.org/html/2608.14680)
###### Abstract\.

Reliability in LLM\-based agentic systems is a property of the whole execution \(its tool calls, model calls, guardrails, and inter\-agent messages\), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails\. We presentAgentChaosBench, a benchmark for detecting and localizing runtime faults in agentic systems from their execution telemetry\. We run five heterogeneous applications that coordinate agents over the Agent\-to\-Agent protocol and call tools through the Model Context Protocol, and inject ten types of operational fault \(unavailable or slow tools, corrupted or oversized responses, and delayed, looped, or misrouted delegations and bypassed guardrails\) at their tool, model, guardrail, and inter\-agent boundaries, alongside a no\-fault control\. The resulting dataset contains275275sanitized traces:250250faulty executions spanning ten fault types and2525no\-fault controls\. Each faulty trace is aligned with the no\-fault execution of the same input; fault\-type labels and, where applicable, location labels are held out from diagnosis\. On structured single\-trace inputs, a first set of zero\-shot LLM baselines shows the task is far from solved: local detectors up to1414B parameters reach only13\.613\.6–19\.2%19\.2\\%top\-1 fault\-type accuracy and the frontier DeepSeek\-v4\-pro only24\.8%24\.8\\%, while jointly identifying the fault type and its location tops out at22%22\\%; reference\-dependent faults \(above all a bypassed guardrail\) stay near\-unsolved from a single trace\. An aligned reference improves selected relative faults but does not resolve guardrail bypass\. The held\-out labels and compact prediction format support reproducible comparison of LLM\-based and non\-LLM diagnosis methods\.

## 1\.Introduction

Large language model \(LLM\) agents are evolving from conversational interfaces into software systems that plan, invoke tools, maintain state, and coordinate with other agents\(He et al\.,[2025](https://arxiv.org/html/2608.14680#bib.bib9); Zhao et al\.,[2026](https://arxiv.org/html/2608.14680#bib.bib18)\)\. Benchmarks such as AgentBench demonstrate the breadth of tasks that can be addressed through multi\-step interaction with external environments\(Liu et al\.,[2024](https://arxiv.org/html/2608.14680#bib.bib12)\)\. More recent AI\-native systems further combine agents with standardized tool and communication protocols, including the Model Context Protocol \(MCP\) and Agent\-to\-Agent \(A2A\) communication, making system behavior depend on orchestration logic and networked services in addition to the underlying model\(Wang et al\.,[2026b](https://arxiv.org/html/2608.14680#bib.bib16)\)\. As these dependencies grow, evaluating only the final task outcome provides little insight into how reliably the system operates or why a particular execution fails\.

Task \+ faultconfigurationInstrumentedagentic executionRaw traceLangfuse JSONSanitized tracecase\.jsonStructured viewone line per spanDiagnosis methodfault type \+ locationScoreprediction vs\. held\-out labelRaw viewFigure 1\.TheAgentChaosBenchdata\-generation and diagnosis pipeline\.A task and fault configuration drives an instrumented agentic execution, producing a raw Langfuse trace\. Ground\-truth fields are removed to produce a sanitized trace, which is used directly or summarized into a structured view for diagnosis\. Predictions are scored against separately held\-out labels\.Production\-like agent executions expose a broad fault surface\. A remote tool may become unavailable, time out, or return a plausible but corrupted response; an LLM call may exceed its context budget; and an inter\-agent request may be delayed or routed to the wrong agent\. These faults are not necessarily caused by an incorrect reasoning step, yet they can propagate through a long execution and eventually appear as an incorrect answer, excessive latency, or a stalled workflow\. Initial work has therefore adapted fault injection and chaos\-engineering ideas to agentic systems, studying tool unavailability, latency, hangs, incorrect responses, task perturbations, and tool or API failures\(Iannillo,[2025](https://arxiv.org/html/2608.14680#bib.bib10); Gupta,[2026](https://arxiv.org/html/2608.14680#bib.bib8)\)\. More broadly, a recent benchmark of real\-world agent\-system maintenance found that state\-of\-the\-art software\-engineering agents resolved only0\.670\.67–4\.67%4\.67\\%of the studied issues\(Rahardja et al\.,[2025](https://arxiv.org/html/2608.14680#bib.bib14)\), underscoring how difficult agent\-specific failures remain to repair\. These efforts establish the importance of testing and maintaining agents under stress, but provide limited support for diagnosing faults across heterogeneous tool, model, guardrail, and inter\-agent boundaries\.

In parallel, failure\-attribution research has shown that diagnosing agentic systems is itself difficult\. Existing studies catalog recurring behavioral failure modes in multi\-agent systems and use techniques such as counterfactual replay and programmatic perturbation to identify failure\-inducing agents and steps in long trajectories\(Cemri et al\.,[2025](https://arxiv.org/html/2608.14680#bib.bib4); Zhang et al\.,[2025](https://arxiv.org/html/2608.14680#bib.bib17)\)\. This line of work primarily attributes failures to agent actions or reasoning behavior\(Deshpande et al\.,[2025](https://arxiv.org/html/2608.14680#bib.bib5); Epperson et al\.,[2025](https://arxiv.org/html/2608.14680#bib.bib6); Wang et al\.,[2026a](https://arxiv.org/html/2608.14680#bib.bib15); Zhu et al\.,[2025](https://arxiv.org/html/2608.14680#bib.bib20); Mazhar et al\.,[2026](https://arxiv.org/html/2608.14680#bib.bib13)\)\. It does not directly address the complementary operational question considered in this paper: given telemetry from an agentic execution, can a detector distinguish normal behavior from an externally induced runtime fault, identify the fault type, and localize the affected system component? Answering this question requires executions with controlled faults, system\-level telemetry, and ground truth that is independent of a detector’s interpretation of the final answer\.

We introduceAgentChaosBench, a fault\-injection benchmark for evaluating the diagnosability of LLM\-based agentic systems\.111The experimental code and a sample of the benchmark data are available at[https://github\.com/kevinzck8k/agentic\-fault\-diagnosis](https://github.com/kevinzck8k/agentic-fault-diagnosis)\.We use agent applications derived from AI\-NativeBench\(Wang et al\.,[2026b](https://arxiv.org/html/2608.14680#bib.bib16)\)as executable workloads and inject faults at multiple boundaries of their execution, including tool calls, LLM calls, guardrails, and agent\-to\-agent interactions\. Each run is executed with a locally deployed LLM and instrumented through Langfuse to capture its distributed trace, including agent, model, and tool spans and their timing, inputs, outputs, and status metadata\. Fault\-injection markers and labels are removed before diagnosis; the injected fault type and affected location are retained separately as ground truth\.

The resulting benchmark spans five heterogeneous agentic systems and ten fault types \(including tool failure, excessive tool or inter\-agent latency, context overflow, tool output corruption, tool and agent misrouting, and guardrail bypass\) plus a no\-fault control, for275275controlled executions with same\-input alignment between faulty and fault\-free runs\. The benchmark formulates diagnosis as two related tasks: fault\-type classification and localization of the responsible span or component\. Because explicit failures and subtle semantic corruptions coexist in the same taxonomy, the benchmark tests both direct recognition of error signals and contextual reasoning over an execution trace\. Its held\-out labels and compact prediction format also support reproducible comparison of LLM\-based and non\-LLM diagnosis methods\. A first set of LLM baselines shows the task is far from solved: local detectors up to1414B parameters stay at13\.613\.6–19\.2%19\.2\\%top\-1 accuracy over the ten fault types, and even the frontier DeepSeek\-v4\-pro reaches only24\.8%24\.8\\%\. Component localization also remains unreliable: top\-1 accuracy reaches only31%31\\%, and getting type and location jointly right stays at88–22%22\\%\. The reference\-dependent faults most relevant to trustworthiness \(above all a bypassed guardrail\) remain near\-unsolved from a single trace even for the frontier model\. An aligned fault\-free reference raises context\-overflow recall by up to5555points and helps selected latency and routing faults, but its effect is fault\-specific and a bypassed guardrail remains unresolved\.

In summary, this paper makes the following contributions:

- •We define and implement a production\-oriented fault model spanning tool, model, guardrail, and inter\-agent boundaries\.
- •We construct275275sanitized traces across five systems and ten fault types plus no\-fault controls, with aligned references and verified type and location labels\.
- •We provide an automated diagnosis protocol and LLM baselines showing low fault\-type and localization accuracy and fault\-specific gains from paired references\.

## 2\.Benchmark Design and Construction

An agentic system is not a single model, but a composition of probabilistic language models, orchestration logic, local and remote tools, guardrails, and networked services\. Modern systems increasingly connect these components through the Model Context Protocol \(MCP\) for tool access and the Agent\-to\-Agent \(A2A\) protocol for inter\-agent communication\(Wang et al\.,[2026b](https://arxiv.org/html/2608.14680#bib.bib16)\)\. This composition makes reliability a property of the complete execution rather than of the model or final output alone\. Assuring agentic systems therefore requires examining*how*an execution unfolded in addition to*what*it produced\.

AgentChaosBenchsupports this form of analysis through controlled fault injection and trace\-based diagnosis\. We execute five heterogeneous agentic systems, inject faults into their model calls, tools, guardrails, and inter\-agent interactions, and capture the resulting telemetry as structured traces\. A diagnosis method receives a trace with all injection markers and fault labels removed, and must classify the fault and localize the affected span or component\. To make this task both realistic and verifiable, injected faults must model plausible operational failures and produce evidence in the collected telemetry that grounds their assigned fault type and location\. Figure[1](https://arxiv.org/html/2608.14680#S1.F1)summarizes the resulting pipeline\.

### 2\.1\.Design Goals

The benchmark design follows six goals:

- •Production\-relevant faults\.Faults model concrete operational failures \(unreachable or slow tools, corrupted responses, context\-budget exhaustion, delayed or misrouted delegations, bypassed guardrails\) rather than synthetic reasoning perturbations\.
- •Coverage across system components and interactions\.The fault taxonomy spans LLM calls, tools, guardrails, individual agents, and inter\-agent communication across multiple agent frameworks\.
- •Reproducibility\.Every case is produced by a fixed set of task inputs executed against a locally deployed model under a controlled injection configuration, so results do not depend on a proprietary API whose behavior may drift\.
- •No ground\-truth leakage\.Injection markers, fault labels, and other attributes that directly reveal the injected fault are removed from the diagnosis input; a method receives only telemetry available through the observability pipeline\.
- •Trace\-grounded labels\.Each case has a held\-out fault type and affected span or component\. Quality\-control checks verify that the injected fault produces a distinguishing signal in the trace relative to an aligned fault\-free run of the same input \(§[2\.7](https://arxiv.org/html/2608.14680#S2.SS7)\)\.
- •Automated scoring\.A machine\-readable prediction format supports scoring LLM and non\-LLM methods without human adjudication\.

Table 1\.Selected AI\-NativeBench\(Wang et al\.,[2026b](https://arxiv.org/html/2608.14680#bib.bib16)\)workloads used inAgentChaosBench\.
### 2\.2\.Agentic Systems and Task Selection

We buildAgentChaosBenchon five agentic applications from AI\-NativeBench\(Wang et al\.,[2026b](https://arxiv.org/html/2608.14680#bib.bib16)\), selected so that the benchmark exercises the boundaries our fault taxonomy targets\. Concretely, we required each selected system to \(i\) use real tool or MCP interaction, \(ii\) delegate work across agents over the A2A protocol, \(iii\) produce multi\-step executions long enough for faults to propagate, and \(iv\) admit a deterministic success condition\. We use the heterogeneous A2A \(H\-A2A\) variant of each system\. In this variant, agents implemented with CrewAI, LangGraph, and AutoGen communicate through A2A and access external tools through MCP\. Because this protocol stack is common to all five systems, Table[1](https://arxiv.org/html/2608.14680#S2.T1)instead shows how agent roles are assigned to frameworks in each workflow\.

### 2\.3\.Deployment and Execution Environment

Each system is deployed as a set of independent A2A services \(one process per participating agent\) exposing HTTP/JSON\-RPC endpoints, together with the tool and MCP servers the workflow requires\. An orchestrator issues a task input and drives the workflow to completion or failure\.

To make executions reproducible and independent of any external LLM provider, all agents are served by a single locally deployed model, Qwen3\.5\-9B, hosted with the vLLM inference server behind an OpenAI\-compatible endpoint\. All agents in a run share this endpoint, so behavior differences between the fault\-free and fault\-injected runs of the same input are attributable to the injected fault rather than to model variation\. Decoding is greedy \(temperature0\), and tool calling uses vLLM’s Hermes\-style parser\. Bounded timeouts and retries are applied at the A2A and tool boundaries, matching how such systems behave in deployment; these policies are themselves part of the observed behavior a detector must reason about\. For example, bounded retries make repeated attempts visible as recurring calls in the trace\.

### 2\.4\.Fault Model

Our fault model is intended to cover major operational failure modes rather than serve as an exhaustive ontology\. It spans four ways the environment can perturb an agentic execution:*availability failures*make a tool or A2A invocation fail explicitly \(Tool Failure,A2A Timeout\);*performance faults*delay an otherwise valid interaction \(Tool Latency,A2A Latency\);*control and routing faults*alter what is invoked, repeated, or delegated \(Infinite Loop,Tool Misroute,Agent Misroute\); and*data and policy faults*alter exchanged content or enforcement decisions \(Context Overflow,Output Corruption,Guardrail Bypass\)\. Together, these classes exercise tool, LLM, agent, inter\-agent, and guardrail boundaries\. They describe the injected mechanism, not its expected diagnosis difficulty\.

Table[2](https://arxiv.org/html/2608.14680#S2.T2)defines the ten fault types together with the boundary they target, the observable signal they are designed to leave in the trace, and the intended ground\-truth location\. A no\-fault control accompanies every input\. For readability, we use spaced display names in the paper \(e\.g\.,Tool Failure\); released labels use the corresponding snake\_case identifiers \(e\.g\.,tool\_failure\)\. We distinguish an*injected runtime fault*from an*intrinsic reasoning error*: the benchmark injects faults into the execution environment \(tools, delegations, guardrails, model I/O\) while holding the model and task input fixed\. For every input, we first collect a*clean*execution with no injection active; each fault\-injected execution then reuses that input, yielding an aligned clean execution that serves as a fault\-free reference\. This pairing establishes that the diagnosed fault reflects an environmental condition rather than a model mistake, and provides the basis for our quality control \(§[2\.7](https://arxiv.org/html/2608.14680#S2.SS7)\)\.

Table 2\.Fault types, target boundaries, observable trace signals, and expected diagnosis difficulty, grouped by the four fault\-mechanism classes of §[2\.4](https://arxiv.org/html/2608.14680#S2.SS4)\.
### 2\.5\.Fault\-Injection Mechanism

Faults are introduced by interceptors placed at the boundaries between an agent and the resource it invokes \(its tools, its model calls, its guardrail checks, and its A2A delegations\) so that injection is transparent to agent logic and identical in placement across the five systems\. An injection configuration names a target boundary, an activation condition \(which input, which call index\), and an operator\. The operators mirror the fault families of Table[2](https://arxiv.org/html/2608.14680#S2.T2): delaying a response, raising an exception, replacing a payload with a corrupted or oversized one, forcing a bounded retry to repeat an identical call, overriding a guardrail verdict, or redirecting a delegation to a different agent endpoint\.

Two properties of the mechanism are essential to the benchmark’s validity\. First, injection is*argument\-isolated*where a fault could otherwise perturb the model: for control and routing faults such asInfinite Loopwe drive the repeated call through a dedicated anchor argument signature, so the loop is observable as repeated identical calls without destabilizing the surrounding generation\. Second, each fault is realized against a*genuine*target: forTool Misroute, the misrouted call must reach a tool whose behavior truly differs from the intended one, so that the fault produces a real output mismatch rather than a no\-op\. When a fault fires, the injector records the activation \(the fault type, the affected span, and its location\) into a separate ground\-truth record\. This record is used to constructlabels\.jsonland is never written into the trace the detector sees\.

### 2\.6\.Observability and Trace Collection

A useful diagnosis trace must expose more than LLM and tool calls: it must also reveal agent identity, guardrail decisions, and A2A interactions with per\-step timing and status\(Balusu,[2026](https://arxiv.org/html/2608.14680#bib.bib2)\)\. We instrument every run with Langfuse, which records a hierarchical trace of spans covering the execution\. Each span carries an identifier, a span kind, a level/status, start time and duration, and, subject to filtering, its input and output\. Observed span kinds includeAGENT\(an agent’s activity\),LLM\_CALL,TOOL\_CALL,MEMORY, and the A2A delegation spans that connect agents, alongside structuralCHAIN/SPANnodes; guardrail decisions surface as their own spans\. A typical execution comprises on the order of tens to a few hundred spans \(e\.g\.,6464spans for a representative SQLAssistant run\)\.

Before release, traces are normalized and filtered: injection markers and any fields that would leak the label are removed, and the directory path that organizes a case by fault type \(used only for storage\) is explicitly*not*part of the detector input\. What remains is the span hierarchy with timing, status, identity, and content that a production observability stack would expose\.

### 2\.7\.Dataset Construction and Specification

Case Generation and Quality Control\.Cases are generated by crossing the five systems with the ten fault types plus a no\-fault control, over a fixed set of five task inputs per condition, with inputs*aligned*across conditions so that every faulty case has a same\-input fault\-free reference\. This yields5×11×5=2755\\times 11\\times 5=275cases \(5555per system:5050faulty and55no\-fault\)\.

Because the controlled design fixes the input set, we do not discard or resample cases to obtain clean\-looking numbers; instead every case must be made valid by ensuring the intended fault actually fired and left its signal\. We enforce this with an automated audit that, for each faulty case, compares the trace against its aligned clean execution and checks that the fault’s expected signal \(Table[2](https://arxiv.org/html/2608.14680#S2.T2)\) is present, for example, thatInfinite Loopproduces at least three calls with the same tool name and input, thatContext Overflowproduces an output more than 1\.3 times the size of the corresponding clean output, and thatGuardrail Bypassrecords a*pass*decision for a policy\-violating request\. Cases that did not exhibit their signal were traced to the underlying injection mechanism and fixed at the source rather than masked\. For the availability, control/routing, and data/policy faults this yields an unambiguous signal in every released case\. The two performance faults differ\.Tool Latencyis clean: the injected∼15\{\\sim\}15s lands on a short tool span, and its measured delta is\+15\.0\+15\.0s at the median\.A2A Latencyis inherently softer: the same delay is added to a delegation span that already brackets the downstream agent’s variable work, so the measured delta ranges from−3\{\-\}3s to\+26\{\+\}26s across systems; the injection fires, but its signal is often masked\. We surface this difficulty rather than mask it, and it is reflected in the low diagnosis rates forA2A Latency\.

Trace Representation\.Each released casecase\_NN\.jsoncontains the sanitized trace provided to the detector: a list of spans, each with an identifier, name, span kind, level, start time, duration, and filtered input/output\. Because real multi\-agent traces are large \(in our data the full\-trace serialization has a median of∼108\{\\sim\}108K tokens and reaches∼638\{\\sim\}638K tokens for the largest case\), feeding the raw trace to a detector is often impractical\. We therefore also define a compact*structured view*that renders one line per span exposing the quantitative features a diagnosis needs: span id, kind, name, level, duration, output size, a repeat count of identical \(name\+input\) calls, and a short output preview \(Figure[2](https://arxiv.org/html/2608.14680#S2.F2)\)\. The structured view reduces the largest traces by roughly an order of magnitude while surfacing \(not computing\) standard observability features; it deliberately does not flag anomalies or compare against a baseline, as that would perform the diagnosis for the detector\. For the single outlier, a BookWriterContext Overflowrun containing1,5911\{,\}591identical framework event\-queue polls, we collapse identical \(name\+input\) spans into one row while retaining their true repeat count in thexNcolumn, bringing the view within the local detectors’ context windows without changing the other274274views\.

Raw span description\(13,802 JSON characters;∼100\{\\sim\}100spans per trace\)

\{"id":"85f8974d\.\.","name":"Crew\.\.kickoff","duration\_ms":132501,"input":"\{chapter\_title:\.\.,goal:\.\.\}","output":"\{title: The Nature and Power ofIllusions, content: \# \.\.\.\}" \(12,012 chars\)\}

Structured span description\(one 245\-character line\)

\[85f8974d\] span Crew\.\.kickoff \| DEFAULT \|132501 \| 12012 \| x1 \| \{title: The Natureand Power of Illusions, content: \# \.\.\.

Figure 2\.Raw and structured representations of one trace span\.The raw representation shows a gray box containing multiline JSON\. The structured representation shows a smaller gray box containing one line with the span identifier, kind, name, level, duration, output size, repeat count, and output preview\.Ground Truth and Dataset Summary\.Ground\-truth answers are held out inlabels\.jsonl, keyed by case UID\. Each record gives the fault type and a numeric fault id, the required span kind, the detection signal, and the affected location \(span kind, component name, and call index\); for routing faults it additionally records the misrouted target\. Table[2](https://arxiv.org/html/2608.14680#S2.T2)lists the per\-fault detection signal and required kind\. The benchmark and evaluation kit, including the audit and scoring scripts, will be released publicly\.

## 3\.Evaluation

We ask whether a diagnosis method can infer a fault’s presence, type, and responsible component from sanitized telemetry, and how that depends on the fault class and detector capacity\. We report a first set of LLM baselines, organized around four research questions:

- •RQ1 \(Fault\-type diagnosis and trace representation\)\.How accurately can a detector identify the fault type from a single sanitized trace, and how do the structured and raw views affect coverage and accuracy?
- •RQ2 \(Fault\-specific difficulty\)\.How does diagnosability vary across fault types and the four fault\-model classes?
- •RQ3 \(Localization\)\.Beyond naming the fault type, can a detector reliably identify the responsible component and jointly predict its location and fault type?
- •RQ4 \(Reference condition\)\.Does providing a comparable fault\-free reference trace improve diagnosis over the single\-trace setting, and for which fault types?

### 3\.1\.Diagnosis Task and Prediction Format

In the primary single\-trace setting, a method receives one execution trace of unknown status \(which may be fault\-free\) and must return a ranked list of candidate fault types drawn from the eleven conditions \(the ten faults plusNo Fault\), each with a short textual justification citing spans\. From the ranking we score top\-1 and top\-3 correctness\. The optional paired\-trace setting \(RQ4\) additionally supplies a known\-normal execution of the same input; because inputs and external task state are held fixed across conditions \(§[2\.3](https://arxiv.org/html/2608.14680#S2.SS3)\), the two traces differ only in the injected fault\.

### 3\.2\.Baselines

Our baselines are general\-purpose instruction\-tuned LLMs applied zero\-shot\. To span an order of magnitude in detector capacity we use four open models \(Qwen3\-1\.7B, Qwen3\-4B, Qwen3\.5\-9B, and Qwen3\-14B\), each served locally with vLLM, and we add the frontier DeepSeek\-v4\-pro \(served via API\) as a strong upper reference\. Every detector is run under both trace representations where its context window permits \(§[2\.7](https://arxiv.org/html/2608.14680#S2.SS7)\), so the same model is compared on the structured and raw views\. All detectors share an identical prompt that asks for a ranked list of candidate fault types with span\-citing justifications\. A rule\-based detector over the observable signals in Table[2](https://arxiv.org/html/2608.14680#S2.T2)is a natural non\-LLM baseline left to future work\.

### 3\.3\.Metrics

For fault\-type diagnosis we report top\-1 accuracy \(AC@1\) and top\-3 accuracy \(AC@3\) over the faulty cases, and separately the rate at which no\-fault controls are correctly identified asNo Fault\(a proxy for the false\-positive behavior of a detector\)\. We report accuracy overall and per fault type; because the eleven conditions are balanced by construction, a uniform\-random detector scores1/11≈9%1/11\\approx 9\\%AC@1\. For localization we report, over the225225cases whose label carries a location \(the2525Context Overflowfaults and2525No Faultcontrols have none\), component\-location AC@1 and AC@3: the top\-1, respectively any top\-3, cited span must resolve to the ground\-truth component after normalizing span\-kind and delegation\-call name decorations\. Type\-and\-location AC@1 requires both the top\-ranked fault type and the predicted component to be correct\.

### 3\.4\.Experimental Setup

All baselines run against the correctedAgentChaosBenchdataset of275275cases, with directory names and held\-out fields excluded from the input and greedy decoding throughout\. On the structured view every detector covers all275275cases, so all cross\-detector comparisons use identical samples\. Outputs that yield no parseable ranking are counted as incorrect, and their rate varies sharply by detector:0%0\\%for DeepSeek\-v4\-pro and2%2\\%for Qwen3\-14B, but5%5\\%,17%17\\%, and33%33\\%for Qwen3\-1\.7B, Qwen3\-4B, and Qwen3\.5\-9B\. These are truncations \(finish reason*length*\) in which a reasoning model exhausts its output budget mid\-deliberation; we show in §[5](https://arxiv.org/html/2608.14680#S5)that doubling the budget leaves the accuracies unchanged, so they are not an artifact of the budget\. The raw view is only attempted where the context window allows: it is infeasible for the≤40960\{\\leq\}40960\-token Qwen3 detectors, Qwen3\.5\-9B runs raw within its262262K native window \(covering the225/275225/275cases that fit\), and DeepSeek\-v4\-pro within its11M window \(all275275\)\. Scoring is fully automated againstlabels\.jsonlby case UID\. DeepSeek\-v4\-pro’s localization scores come from a separate API run whose fault\-type accuracy \(25\.2%/37\.6%25\.2\\%/37\.6\\%AC@1/AC@3\) differs from Table[3](https://arxiv.org/html/2608.14680#S3.T3)by less than four percentage points\.

### 3\.5\.Results

Table[3](https://arxiv.org/html/2608.14680#S3.T3)reports AC@1 and AC@3 on the250250faulty cases for the structured and raw trace representations; a dash indicates that the detector could not ingest the raw trace within its context window\. Table[4](https://arxiv.org/html/2608.14680#S3.T4)reports per\-fault\-type top\-3 recall on structured traces\. Each entry is the number of cases whose true fault type appears in the detector’s top three, out of2525cases \(55systems×\\times55inputs\); the final row instead counts no\-fault controls correctly recognized as clean\.

Table 3\.Fault\-type diagnosis accuracy\.Table 4\.Per\-fault\-type top\-3 recall on structured traces\.RQ1: fault\-type diagnosis remains difficult across trace representations\.Table[3](https://arxiv.org/html/2608.14680#S3.T3)shows that fault\-type diagnosis from a single sanitized trace is difficult\. The local Qwen detectors sit at13\.613\.6–19\.2%19\.2\\%AC@1, barely above the9%9\\%random baseline and essentially flat across an order of magnitude of model size; and even DeepSeek\-v4\-pro, a frontier model, reaches only24\.8%24\.8\\%AC@1 \(34\.0%34\.0\\%AC@3\)\. Larger models do get the correct fault into their top\-3 more often, but for most cases they cannot commit to it as the first choice\. Inspecting the1414B detector, the true fault is absent from the top\-3 entirely in65%65\\%of faulty cases, so the AC@1–AC@3 gap reflects genuine misses rather than mere ranking noise, and only2%2\\%of outputs are malformed: this is a capability gap, not a formatting artifact\. Recognition of no\-fault controls follows a different trend: it rises from4%4\\%for Qwen3\-1\.7B to96%96\\%for Qwen3\-14B \(and88%88\\%for DeepSeek\-v4\-pro\), indicating that scale helps detectors avoid false alarms even though fault\-type classification remains weak\.

Trace representation also determines whether diagnosis is computationally feasible\. The local detectors with context windows of at most131131K tokens cannot ingest raw traces that reach∼638\{\\sim\}638K tokens\. Qwen3\.5\-9B can process the82%82\\%of cases that fit its262262K window, but its raw\-view accuracy is lower than its structured\-view accuracy \(15\.2%/22\.5%15\.2\\%/22\.5\\%versus19\.2%/28\.0%19\.2\\%/28\.0\\%AC@1/AC@3\), and40%40\\%of its raw\-view outputs contain no parseable ranking\. DeepSeek\-v4\-pro can ingest all raw traces; its raw and structured results are similar \(21\.6%/36\.0%21\.6\\%/36\.0\\%versus24\.8%/34\.0%24\.8\\%/34\.0\\%\), while the raw run costs approximately8\.6×8\.6\\timesmore \($16\.45 versus $1\.91 for the275275cases\) because it contains∼14×\{\\sim\}14\\timesas many prompt tokens\. The structured view therefore expands detector coverage and reduces inference cost without increasing the strongest detector’s accuracy artificially\.

RQ2: mechanism class does not by itself determine diagnosability\.Table[4](https://arxiv.org/html/2608.14680#S3.T4)follows the fault taxonomy of Table[2](https://arxiv.org/html/2608.14680#S2.T2)\. Availability failures are the easiest overall:Tool Failurereaches25/2525/25andA2A Timeout23/2523/25\. Performance faults andInfinite Loopcarry directly observable signals \(a large duration or repeated call\) but detectors use them unevenly \(0/250/25to13/2513/25\), showing that visible evidence does not guarantee correct classification\. The other control/routing faults require knowledge of the intended destination: local detectors remain near chance onTool MisrouteandAgent Misroute, although the frontier model raises the latter to40%40\\%\. Data and policy faults are similarly difficult because their traces can remain well formed\. In particular,Guardrail Bypassis usually predicted asNo Fault: a forged*pass*verdict is indistinguishable from a legitimate approval without knowing that the request should have been rejected\. The smallest detector’s apparent edge onContext Overflow\(8/258/25\) is a calibration artifact rather than skill: it over\-predicts that condition on unrelated cases, buying recall with a matching false\-positive rate while almost never recognizing a clean run \(no\-fault recognition itself scales sharply, from1/251/25at1\.71\.7B to24/2524/25at1414B\)\. Thus, across the four mechanism classes, the more predictive divide is whether the evidence can be interpreted directly or requires a reference for the*expected*behavior\.

Table 5\.Component\-location and type\-and\-location accuracy on structured traces\.RQ3: reliable localization remains difficult\.Table[5](https://arxiv.org/html/2608.14680#S3.T5)scores whether a detector points to the responsible component, over the225225faulty cases that carry a location\. Top\-1 localization remains between20%20\\%and31%31\\%, so even the frontier detector identifies the wrong component in more than two thirds of cases\. Allowing three candidates raises localization to27%27\\%–45%45\\%, but no detector localizes a majority of cases\. Type\-and\-location AC@1 is lower still:8%8\\%–16%16\\%for the local detectors and22%22\\%for DeepSeek\-v4\-pro\. Scale improves location AC@3 more consistently than fault\-type AC@1, but the absolute results show that both component localization and type\-and\-location diagnosis remain open problems\.

RQ4: aligned references improve selected fault types\.We test the reference condition directly: each faulty trace is paired with the alignedNo Faultrun of the same input, given to the detector as a known\-normal baseline \(§[2\.7](https://arxiv.org/html/2608.14680#S2.SS7)\)\. Figure[3](https://arxiv.org/html/2608.14680#S3.F3)reports the change in top\-3 recall,Δ=\\Delta=paired−\-single, on matched cases\. The reference helps whenΔ\\Deltais positive\. It lifts overall AC@3 for every detector, and the lift grows with capability \(\+3\.9\+3\.9,\+4\.9\+4\.9, and\+10\.7\+10\.7percentage points for1\.71\.7B,44B, and1414B\)\. The largest gains occur for faults defined by a deviation from normal behavior: for Qwen3\-14B, the reference raisesContext Overflowrecall by5555points andTool Latencyby3535points, while routing faults improve for some detectors\. These changes show that an aligned execution supplies useful comparison signals for output size, duration, and destination, but do not imply a general solution to single\-trace diagnosis\.

Two exceptions sharpen the picture\.Guardrail Bypassis unmoved or slightly worse \(−5\-5to0points\): on a benign input the reference guard also returns*pass*, so a forged pass is indistinguishable even side by side\.Output Corruptionalso decreases by55–1515points because a plausible but incorrect output does not expose a consistent textual difference that the detectors can interpret as corruption\. Using a reference also demands capacity: Qwen3\.5\-9B reasons past its output budget on the doubled input and fails to emit a valid ranking on over half of paired cases even at81928192output tokens, so we omit it\. Overall, paired traces improve several faults with measurable baselines, but they do not help faults that require an external policy or semantic judgment\.

Context OverflowAgent MisrouteTool LatencyInfinite LoopTool MisrouteA2A LatencyA2A TimeoutTool FailureOutput CorruptionGuardrail Bypass−20\-200202040406060higher recall withpaired traces↑\\uparrow↓\\downarrowhigher recall witha single traceΔ\\Deltatop\-3 recall \(percentage points\)Qwen3\-1\.7BQwen3\-4BQwen3\-14BFigure 3\.Change in top\-3 recall after adding a fault\-free reference trace \(relative to single\-trace diagnosis\)\.A grouped bar chart showing the change in top\-3 recall between paired and single\-trace diagnosis for ten fault types and three Qwen detectors\. Positive bars indicate higher recall with paired traces; negative bars indicate higher recall with a single trace\.Implications\.Two findings bear on assurance\. A diagnosis pipeline must first solve a representation problem, since raw traces reach hundreds of thousands of tokens \(§[2\.7](https://arxiv.org/html/2608.14680#S2.SS7)\)\. And the faults that matter most for trustworthiness, above all a bypassed guardrail and a silent context overflow, are exactly the ones single\-trace detectors miss at every scale, motivating the reference\-based and specialized methods the benchmark is designed to evaluate\.

## 4\.Related Work

Agentic systems and observability\.Benchmarks such as AgentBench established the breadth of tasks LLM agents can solve through multi\-step environment interaction\(Liu et al\.,[2024](https://arxiv.org/html/2608.14680#bib.bib12)\), and AI\-native suites like AI\-NativeBench extend this to white\-box systems that combine agents with MCP tools and A2A communication\(Wang et al\.,[2026b](https://arxiv.org/html/2608.14680#bib.bib16)\)\. As these systems grow into distributed software, operating them reliably requires telemetry beyond final\-answer checking\. AgentTelemetry argues that useful agent observability must expose agent identity, guardrails, delegation, and timing/status metadata, and provides a fault\-detection toolkit over such telemetry\(Balusu,[2026](https://arxiv.org/html/2608.14680#bib.bib2)\)\.AgentChaosBenchadopts this observability stance and pairs it with controlled fault injection and held\-out ground truth\.

Fault injection and stress testing for agents\.Chaos\-engineering ideas have begun to reach agents\(Basiri et al\.,[2016](https://arxiv.org/html/2608.14680#bib.bib3)\)\. ChaosLLM injects unreachable, slow, hanging, and incorrect responses at the tool boundary of a ReAct agent\(Iannillo,[2025](https://arxiv.org/html/2608.14680#bib.bib10)\), and ReliabilityBench evaluates consistency, robustness to task perturbations, and tolerance of tool/API faults under production\-like stress\(Gupta,[2026](https://arxiv.org/html/2608.14680#bib.bib8)\)\. These works establish the value of stressing agents but focus on single\-agent tool faults and on outcome robustness rather than on*diagnosing*a fault across heterogeneous boundaries\.AgentChaosBenchinjects faults at the tool, LLM, guardrail, agent, and inter\-agent boundaries of multi\-framework, multi\-agent systems, and evaluates diagnosis from traces stripped of injection metadata and labels\.

Failure attribution in multi\-agent systems\.A parallel line studies who or what caused a failure\. MAST catalogs recurring behavioral failure modes of multi\-agent systems\(Cemri et al\.,[2025](https://arxiv.org/html/2608.14680#bib.bib4)\); AgenTracer uses counterfactual replay and programmatic perturbation to attribute failures to agents and steps in long trajectories\(Zhang et al\.,[2025](https://arxiv.org/html/2608.14680#bib.bib17)\); spectrum\-based methods adapt fault\-localization statistics to agent executions\(Ge et al\.,[2025](https://arxiv.org/html/2608.14680#bib.bib7)\); and recent work rethinks attribution from multiple perspectives\(In et al\.,[2026](https://arxiv.org/html/2608.14680#bib.bib11)\)\. LLMGuard targets fault diagnosis for language\-model services\(Zhong et al\.,[2026](https://arxiv.org/html/2608.14680#bib.bib19)\)\. This body of work primarily attributes failures to*internal*agent actions or reasoning\.AgentChaosBenchis complementary: it targets*external*, environmentally induced runtime faults, with ground truth that is independent of the final answer and of any detector’s interpretation, and it explicitly includes a no\-fault control so that false positives are measurable\.

## 5\.Discussion

### 5\.1\.Limitations

The current benchmark uses five systems with five aligned inputs per condition; while this yields275275verified cases and balanced per\-fault coverage, broader task and input diversity would strengthen external validity, and we plan to scale both\. Our baselines are zero\-shot general\-purpose LLMs \(four local Qwen models and the frontier DeepSeek\-v4\-pro\) evaluated on both the structured and raw trace views; a rule\-based non\-LLM detector, few\-shot prompting, and specialized attribution methods remain to be evaluated\. Because DeepSeek\-v4\-pro is served through a commercial API, its outputs are not reproducible even at temperature0: a re\-run disagrees on nearly half \(48%48\\%\) of top\-1 predictions, though aggregate accuracy is stable to within a point, so we read its per\-case results as a single sample while the local Qwen detectors are fully reproducible\. We report fault\-type diagnosis and component\-level localization; exact\-span scoring and richer localization metrics \(e\.g\., MRR\) remain to be added\. Finally, all runs use a single locally served model to hold generation fixed across conditions, which aids reproducibility but leaves cross\-model generalization of the*workloads*\(as opposed to the detectors\) for future study\.

### 5\.2\.Threats to Validity

Internal validity\.Four factors could distort our scores; we address each\. First, an injected fault must be real and its identity hidden: we remove injection markers and label\-bearing fields from the released trace and verify that every faulty case exhibits its intended signal against an aligned no\-fault reference \(§[2\.7](https://arxiv.org/html/2608.14680#S2.SS7)\), fixing cases that failed this check at the injection source rather than dropping them, which would bias the set toward easy cases\. Second, because faulty and fault\-free runs share the same input and model endpoint, observed trace differences are attributable to the injected fault\. Third, trace representation is a potential confound, but the structured view only surfaces observability features without computing anomaly judgments, and the one model able to read both views scores the same on each \(Table[3](https://arxiv.org/html/2608.14680#S3.T3)\), so the compact view does not inflate accuracy, while the paired\-trace experiment \(RQ4, Figure[3](https://arxiv.org/html/2608.14680#S3.F3)\) shows that the benefit of a reference is concentrated in specific fault types rather than uniformly improving diagnosis\. Fourth, the Qwen reasoning detectors are sometimes truncated before emitting a ranking \(up to33%33\\%for Qwen3\.5\-9B,≤2%\\leq 2\\%for Qwen3\-14B and DeepSeek\-v4\-pro\); doubling the output budget \(4096→81924096\\to 8192tokens\) leaves both truncation \(37%→33%37\\%\\to 33\\%for Qwen3\.5\-9B\) and accuracy \(its AC@118\.4→19\.218\.4\\to 19\.2\) essentially unchanged, so these are hard cases on which deliberation diverges rather than answers a larger budget would recover, and scoring them incorrect does not understate ability\.

External validity\.The systems, faults, and single\-trace protocol approximate but do not exhaust production conditions; real deployments mix faults, vary load, and evolve over time\. The taxonomy targets common operational failures at standard agent boundaries, and the injection architecture is framework\-agnostic by placement, which we expect to transfer, but confirming this on additional systems is future work\.

## 6\.Conclusion and Future Work

We presentedAgentChaosBench, a benchmark of275275sanitized traces for classifying and localizing ten runtime fault types across five multi\-agent systems\. Its controlled injections, aligned no\-fault executions, verified labels, and automated scoring support reproducible evaluation of agent\-system diagnosis\.

The evaluated LLMs remain unreliable: single\-trace fault\-type AC@1 reaches at most24\.8%24\.8\\%, component\-location AC@1 at most31%31\\%, and type\-and\-location AC@1 at most22%22\\%\. A compact structured view makes long traces tractable without improving the strongest detector artificially\. Aligned references improve context overflow \(by up to5555points\) and selected latency and routing faults, but not faults requiring semantic or policy judgment, including guardrail bypass\. These results motivate reference\-based and specialized detectors, non\-LLM baselines, and exact\-span localization, all of which the released benchmark is designed to support\.

## References

- \(1\)
- Balusu \(2026\)Krishna Chaitanya Balusu\. 2026\.AgentTelemetry: A Fault Detection Benchmark and Toolkit for LLM Agent Observability\. In*Proceedings of the 3rd ACM International Conference on AI\-Powered Software*\. 380–387\.
- Basiri et al\.\(2016\)Ali Basiri, Niosha Behnam, Ruud De Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, and Casey Rosenthal\. 2016\.Chaos Engineering\.*IEEE Software*33, 3 \(2016\), 35–41\.
- Cemri et al\.\(2025\)Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al\.2025\.Why Do Multi\-Agent LLM Systems Fail?\. In*Advances in Neural Information Processing Systems*\.
- Deshpande et al\.\(2025\)Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian\. 2025\.TRAIL: Trace Reasoning and Agentic Issue Localization\.*arXiv preprint arXiv:2505\.08638*\(2025\)\.
- Epperson et al\.\(2025\)Will Epperson, Gagan Bansal, Victor C Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, and Saleema Amershi\. 2025\.Interactive Debugging and Steering of Multi\-Agent AI Systems\. In*Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems*\.
- Ge et al\.\(2025\)Yu Ge, Linna Xie, Zhong Li, Yu Pei, and Tian Zhang\. 2025\.Who Is Introducing the Failure? Automatically Attributing Failures of Multi\-Agent Systems via Spectrum Analysis\.*arXiv preprint arXiv:2509\.13782*\(2025\)\.
- Gupta \(2026\)Aayush Gupta\. 2026\.ReliabilityBench: Evaluating LLM Agent Reliability Under Production\-Like Stress Conditions\.*arXiv preprint arXiv:2601\.06112*\(2026\)\.
- He et al\.\(2025\)Junda He, Christoph Treude, and David Lo\. 2025\.LLM\-Based Multi\-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead\.*ACM Transactions on Software Engineering and Methodology*\(2025\)\.
- Iannillo \(2025\)Antonio Ken Iannillo\. 2025\.ChaosLLM: A Dependability Testing Approach for Tool\-Calling Agents\. In*2025 IEEE 36th International Symposium on Software Reliability Engineering Workshops \(ISSREW\)*\.
- In et al\.\(2026\)Yeonjun In, Mehrab Tanjim, Jayakumar Subramanian, Sungchul Kim, Uttaran Bhattacharya, Wonjoong Kim, Sangwu Park, Somdeb Sarkhel, and Chanyoung Park\. 2026\.Rethinking Failure Attribution in Multi\-Agent Systems: A Multi\-Perspective Benchmark and Evaluation\.*arXiv preprint arXiv:2603\.25001*\(2026\)\.
- Liu et al\.\(2024\)Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al\.2024\.AgentBench: Evaluating LLMs as Agents\. In*International Conference on Learning Representations*\.
- Mazhar et al\.\(2026\)Anna Mazhar, Huzaifa Suri, and Sainyam Galhotra\. 2026\.Trace\-Level Analysis of Information Contamination in Multi\-Agent Systems\. In*Proceedings of the ACM Conference on AI and Agentic Systems*\.
- Rahardja et al\.\(2025\)Alfin Wijaya Rahardja, Junwei Liu, Weitong Chen, Zhenpeng Chen, and Yiling Lou\. 2025\.Can Agents Fix Agent Issues?\. In*NeurIPS*\.
- Wang et al\.\(2026a\)Yawen Wang, Wenjie Wu, Junjie Wang, and Qing Wang\. 2026a\.From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM\-Based Multi\-Agent Systems\.*arXiv preprint arXiv:2602\.23701*\(2026\)\.
- Wang et al\.\(2026b\)Zirui Wang, Guangba Yu, and Michael R Lyu\. 2026b\.AI\-NativeBench: An Open\-Source White\-Box Agentic Benchmark Suite for AI\-Native Systems\.*arXiv preprint arXiv:2601\.09393*\(2026\)\.
- Zhang et al\.\(2025\)Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, and Shuicheng Yan\. 2025\.AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?*arXiv preprint arXiv:2509\.03312*\(2025\)\.
- Zhao et al\.\(2026\)Chenyu Zhao, Shenglin Zhang, Yihang Lin, Wenwei Gu, Zhimin Chen, Yongqian Sun, Dan Pei, Chetan Bansal, Saravan Rajmohan, and Minghua Ma\. 2026\.Debugging the Debuggers: Failure\-Anchored Structured Recovery for Software Engineering Agents\.*arXiv preprint arXiv:2605\.08717*\(2026\)\.
- Zhong et al\.\(2026\)Yuedong Zhong, Guangba Yu, Yujie Huang, QunChao Fu, Rui Ren, Cong Feng, Yongqiang Yang, and Michael Lyu\. 2026\.LLMGuard: Multi\-Agent Fault Diagnosis for Reliable Language\-Model\-as\-a\-Service\. In*2026 56th Annual IEEE International Conference on Dependable Systems and Networks \(DSN\)*\.
- Zhu et al\.\(2025\)Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingxuan Yang, Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, et al\.2025\.Where LLM Agents Fail and How They Can Learn from Failures\.*arXiv preprint arXiv:2509\.25370*\(2025\)\.