Beyond LLM-Based Reasoning: Lightweight GNNs for Agent Failure Attribution

# Lightweight GNNs for Agent Failure Attribution
Source: [https://arxiv.org/html/2608.18575](https://arxiv.org/html/2608.18575)
## Beyond LLM\-Based Reasoning: Lightweight GNNs for Agent Failure Attribution

Ting\-Wei LiAffiliation:University of IllinoisAffiliation:Urbana\-Champaign, IL USAEmail:[twli@illinois\.edu](mailto:)Yuanchen BeiAffiliation:University of IllinoisAffiliation:Urbana\-Champaign, IL USAEmail:[bei4@illinois\.edu](mailto:)Xiao LinAffiliation:University of IllinoisAffiliation:Urbana\-Champaign, IL USAEmail:[xiaol13@illinois\.edu](mailto:)Hanghang TongAffiliation:University of IllinoisAffiliation:Urbana\-Champaign, IL USAEmail:[htong@illinois\.edu](mailto:)

###### Abstract

Large language model \(LLM\)\-based multi\-agent systems \(MAS\) often exhibit complex failure modes, which frequently cause agents to produce incorrect outcomes\. This motivates the task ofAgent Failure Attribution \(AFA\): given a failed multi\-agent trajectory, identify the faulty agents and their corresponding error types\. Existing approaches predominantly rely on LLMs to perform failure attribution, either through direct prompting, fine\-tuning on synthetic data or complex agentic pipelines\. While effective, these methods incur substantial computational overhead due to long\-context processing, expensive post\-training and handcrafted workflows\. Moreover, empirical evidence shows that even state\-of\-the\-art models achieve limited accuracy on existing benchmarks, suggesting that scaling model size alone is insufficient\. In this work, we revisit this task and question the necessity of such expensive generative solutions\. We introduceAFANet, a lightweight graph\-based framework that models interaction trajectories through step\-level semantic signals and agent\-level relationships\. We show that with significantly fewer parameters and near\-zero inference cost,AFANet\(i\) matches or outperforms LLM\-based baselines, including fine\-tuned models on in\-domain benchmarks, \(ii\) maintains robust performance across different GNN architectures and \(iii\) can be further improved with inexpensive test\-time adaptation on the OOD benchmark\. Our results suggest that effective agent failure attribution does not require heavy LLM reasoning and a lightweight, structured approach can achieve strong performance\.

Figure 1:Attribution accuracy and efficiency comparison between our proposedAFANetand LLMs\.AFANetcan achieve similar or better performance against LLMs while incurring significantly lower training and inference cost\. We consider the base models:Qwen\-2\.5\-7B/14B\-Instruct, denoted as7B/14B, respectively\. We also include their post\-trained variants obtained through SFT \(S\) and subsequent GRPO training \(G\), denoted by7B\+S/14B\+Sand7B\+S\+G/14B\+S\+G\.## 1Introduction

LLM\-based multi\-agent systems\[[11](https://arxiv.org/html/2608.18575#bib.bib2),[7](https://arxiv.org/html/2608.18575#bib.bib3),[25](https://arxiv.org/html/2608.18575#bib.bib1)\]have emerged as a powerful paradigm for solving complex tasks through coordinated reasoning, tool use, and interaction across multiple agents\. Despite their strong capabilities, these systems often exhibit*complex failure modes*\[[3](https://arxiv.org/html/2608.18575#bib.bib6),[31](https://arxiv.org/html/2608.18575#bib.bib7),[23](https://arxiv.org/html/2608.18575#bib.bib19),[12](https://arxiv.org/html/2608.18575#bib.bib20)\], where errors introduced by one agent propagate through subsequent interactions and ultimately lead to incorrect outcomes\. This motivates the task ofAgent Failure Attribution \(AFA\)\[[3](https://arxiv.org/html/2608.18575#bib.bib6),[31](https://arxiv.org/html/2608.18575#bib.bib7),[5](https://arxiv.org/html/2608.18575#bib.bib8),[10](https://arxiv.org/html/2608.18575#bib.bib9)\]: given a failed multi\-agent interaction trajectory, identify the faulty agents and their corresponding error types responsible for the failure\. However, this task is inherently challenging due to long interaction horizons, complex dependencies across agents and the ambiguous nature of error propagation\. Even state\-of\-the\-art reasoning models or powerful proprietary models achieve limited accuracy on this task, highlighting the difficulty of identifying root causes within long trajectories\[[10](https://arxiv.org/html/2608.18575#bib.bib9)\]\.

Existing approaches predominantly treat failure attribution as a*generative reasoning problem*, relying on LLMs to analyze interaction trajectories and directly generate the answers\. These methods typically involve direct prompting\[[31](https://arxiv.org/html/2608.18575#bib.bib7)\], fine\-tuning on synthetic failure data\[[10](https://arxiv.org/html/2608.18575#bib.bib9),[30](https://arxiv.org/html/2608.18575#bib.bib11),[29](https://arxiv.org/html/2608.18575#bib.bib10)\]or complex multi\-stage agentic pipelines\[[28](https://arxiv.org/html/2608.18575#bib.bib12),[2](https://arxiv.org/html/2608.18575#bib.bib13),[24](https://arxiv.org/html/2608.18575#bib.bib14),[18](https://arxiv.org/html/2608.18575#bib.bib15)\]\. While these approaches have improved performance, they suffer from several fundamental limitations\. Firstly\(Efficiency Bottleneck\), they incur substantial computational overhead due to long\-context processing, repeated inference and expensive training procedures\. Secondly\(Architectural Complexity\), they introduce significant system complexity through multi\-stage workflows and handcrafted pipelines\. Lastly\(Systematical Ineffectiveness\), despite these efforts, their performance remains limited, sometimes defeated by random baselines, suggesting that scaling model size alone is insufficient for solving this task\. These observations raise a fundamental question:

> *Is heavy LLM reasoning necessary for agent failure attribution?*

In this work, we challenge this prevailing paradigm and argue that agent failure attribution can be solved much efficiently with lightweight models\. Instead of relying on expensive LLM\-based reasoning, we proposeAFANet, a lightweight graph neural network \(GNN\)\[[26](https://arxiv.org/html/2608.18575#bib.bib16)\]that models interaction trajectories through step\-level semantic and agent\-level structural signals\. By explicitly encoding temporal dependencies and inter\-agent interactions,AFANetcaptures the underlying structure of failure propagation\. Specifically,AFANetaddresses the core challenges of agentic failure attribution in a principled manner\. First, graph\-based representations naturally model error propagation and inter\-agent dependencies, enabling accurate identification of root causes\. Second, the model operates with significantly reduced computational cost, avoiding both long\-context inference and expensive post\-training\. Through extensive experiments, we demonstrate thatAFANetmatches or outperforms strong LLM\-based baselines, including supervised fine\-tuned and RL\-enhanced models, while using significantly fewer parameters and achieving near\-zero inference cost \(as shown in Figure[1](https://arxiv.org/html/2608.18575#S0.F1)\)\. We further show thatAFANetis robust to architectural choices and can be improved through inexpensive test\-time adaptation on the OOD dataset\. We summarize our contributions are as follows:

- •We introduce a new perspective on agent failure attribution: instead of LLM reasoning, graph\-based modeling of failure trajectories is sufficient for effective attribution\.
- •We proposeAFANet, a lightweight framework that models multi\-agent failure propagation through a turn\-level conversation graph, integrating turn\-level semantics and temporal/intra\-agent dependencies via graph message passing\.
- •We demonstrate that with substantially lower training and inference cost,AFANetachieves competitive performance compared to LLMs\.
- •We conduct comprehensive ablation study and show thatAFANetremains strong under different architectural choices and can be improved via inexpensive test\-time adaptation\.

## 2Preliminaries

### 2\.1Multi\-Agent System Trajectories

Let𝒜=\{a1,a2,…,aM\}\\mathcal\{A\}=\\\{a\_\{1\},a\_\{2\},\\dots,a\_\{M\}\\\}denote a set of LLM\-based agents\. Given a user queryq∈𝒬q\\in\\mathcal\{Q\}, a multi\-agent system attempts to solve the task and produces an interaction trajectory𝒯=\{\(ait,xt\)\}t=1T\\mathcal\{T\}=\\\{\(a\_\{i\_\{t\}\},x\_\{t\}\)\\\}\_\{t=1\}^\{T\}, whereTTis the trajectory length,ait∈𝒜a\_\{i\_\{t\}\}\\in\\mathcal\{A\}is the agent selected at steptt, andxt∈𝒳x\_\{t\}\\in\\mathcal\{X\}is the content generated by that agent\. Each trajectory𝒯\\mathcal\{T\}is associated with an outcome variableo=g⁡\(𝒯,q\)∈\{0,1\}o=g\(\\mathcal\{T\},q\)\\in\\\{0,1\\\}, whereggis the outcome verifier ando=0/1o=0/1indicates task failure/success, respectively\.

### 2\.2Agent Failure Attribution \(AFA\)

Letℰ=\{e1,e2,…,eK\}\\mathcal\{E\}=\\\{e\_\{1\},e\_\{2\},\\dots,e\_\{K\}\\\}denote a predefined set of error types andKKis the number of error types\. We define the label space as the Cartesian product over agent space and error space:ℒ=𝒜×ℰ\\mathcal\{L\}=\\mathcal\{A\}\\times\\mathcal\{E\}\. Given a failed trajectory𝒯\\mathcal\{T\}, the task ofAgent Failure Attribution\(AFA\) is to predict a subset of labels𝒴⊆ℒ\\mathcal\{Y\}\\subseteq\\mathcal\{L\}, where each\(aj,ek\)∈𝒴\(a\_\{j\},e\_\{k\}\)\\in\\mathcal\{Y\}indicates that agentaja\_\{j\}commits an error of typeeke\_\{k\}that contributes to the failure\. We thus formulate AFA as a multi\-way classification problem for each agent and our goal is to learn a mapping functionh:𝒯→\{0,1\}M×\(K\+1\)h:\\mathcal\{T\}\\rightarrow\\\{0,1\\\}^\{M\\times\(K\+1\)\}where the first class corresponds to thecleanstate and the remainingKKclasses correspond to each error type\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2608.18575v1/ErrorMAS-method.png)Figure 2:Overall pipeline\. Given a failed multi\-agent system trajectory, we first transform it into a conversation graph with turn\-level textual features and temporal/agent\-level connections\. The graph will then be passed through our proposedAFANetfor agent failure prediction\.In this section, we propose a lightweight graph\-based model,AFANet, to tackle agent failure attribution\. Given a conversation, we first construct a turn\-level graph with temporal and intra\-agent connections \(Sec\.[3\.1](https://arxiv.org/html/2608.18575#S3.SS1)\)\. We then introduce our proposed model,AFANet, a simple GNN that combines deviation/statistical signals and contextual semantics to produce agent\-level representations \(Sec\.[3\.2](https://arxiv.org/html/2608.18575#S3.SS2)\)\. Then, we detail the learning objective \(Sec\.[3\.3](https://arxiv.org/html/2608.18575#S3.SS3)\) and finally describe the inference procedure ofAFANet\(Sec\.[3\.4](https://arxiv.org/html/2608.18575#S3.SS4)\)\. The overall pipeline is detailed in Figure[2](https://arxiv.org/html/2608.18575#S3.F2)\.

### 3\.1Conversation Graph Construction

We first transform each multi\-agent conversation into a heterogeneous graph with turn\-level nodes\. The key intuition is that failures in multi\-agent systems are not only reflected in the semantic content of individual turns, but also in how an agent’s contributions evolve over time and interact with other agents\. Therefore, we represent it as a heterogeneous graph whose nodes are conversation turns and whose edges encode temporal and intra\-agent dependencies\. We detail the construction of node features and edges in the following paragraphs\.

#### Turn node representation\.

Given a failed trajectory𝒯=\{\(ait,xt\)\}t=1T\\mathcal\{T\}=\\\{\(a\_\{i\_\{t\}\},x\_\{t\}\)\\\}\_\{t=1\}^\{T\}, we construct a graph𝒢=\(𝒱,ℰ\)\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{E\}\), where each nodevt∈𝒱v\_\{t\}\\in\\mathcal\{V\}corresponds to one turn\(ait,xt\)\(a\_\{i\_\{t\}\},x\_\{t\}\)in the conversation and the number of nodes is\|𝒱\|=T\|\\mathcal\{V\}\|=T\. Inspired by recent anomaly detection literature\[[19](https://arxiv.org/html/2608.18575#bib.bib4),[17](https://arxiv.org/html/2608.18575#bib.bib5)\], we aim to capture abnormal deviation and statistical patterns to derive useful turn\-level features for solving AFA\. The key intuition is that faulty agent behavior is often reflected through inconsistencies in interaction dynamics rather than only the semantic content of individual turns\.

To obtain such signals efficiently, we first construct per\-conversation TF\-IDF representations by treating each turn as a document and the entire trajectory as the corpus \(more details in Appendix[A](https://arxiv.org/html/2608.18575#A1)\)\. We then apply truncated SVD to obtain low\-dimensional dense representations:𝐗svd∈ℝT×r\\mathbf\{X\}\_\{\\mathrm\{svd\}\}\\in\\mathbb\{R\}^\{T\\times r\}, whererris the rank size\. Based on𝐗svd\\mathbf\{X\}\_\{\\mathrm\{svd\}\}, we compute a set of deviation features that captures how each turn diverges from the contexts in the conversation\. We concatenate these deviation features to obtain𝐱tdev\\mathbf\{x\}^\{\\mathrm\{dev\}\}\_\{t\}for each turn, and stack them across all turns:𝐗dev=\[𝐱dev1​\|⋯\|​𝐱devT\]∈ℝT×ddev\\mathbf\{X\}\_\{\\mathrm\{dev\}\}=\[\\mathbf\{x\}\_\{\\mathrm\{dev\}\}^\{1\}\|\\cdots\|\\mathbf\{x\}\_\{\\mathrm\{dev\}\}^\{T\}\]\\in\\mathbb\{R\}^\{T\\times d\_\{\\mathrm\{dev\}\}\}, whereddevd\_\{\\mathrm\{dev\}\}is the number of deviation features\.

In addition, we consider statistical features𝐗stat∈ℝT×dstat\\mathbf\{X\}\_\{\\mathrm\{stat\}\}\\in\\mathbb\{R\}^\{T\\times d\_\{\\mathrm\{stat\}\}\}\(dstatd\_\{\\mathrm\{stat\}\}denotes the number of statistical features\) that encode non\-semantic properties such as positional encoding\. Finally, we consider dense semantic features obtained from pretrained sentence encoders \(e\.g\., sentence\-BERT\[[16](https://arxiv.org/html/2608.18575#bib.bib22)\]\) to capture richer contextual meaning, leading to𝐗dense∈ℝT×ddense\\mathbf\{X\}\_\{\\mathrm\{dense\}\}\\in\\mathbb\{R\}^\{T\\times d\_\{\\mathrm\{dense\}\}\}, whereddensed\_\{\\mathrm\{dense\}\}is the embedding dimension\. Stacking these components together, we obtain the initial feature matrix for turn nodes:𝐗=\[𝐗dev​\|𝐗stat\|​𝐗dense\]∈ℝT×d\\mathbf\{X\}=\\big\[\\mathbf\{X\}\_\{\\mathrm\{dev\}\}\\;\\big\|\\;\\mathbf\{X\}\_\{\\mathrm\{stat\}\}\\;\\big\|\\;\\mathbf\{X\}\_\{\\mathrm\{dense\}\}\\big\]\\in\\mathbb\{R\}^\{T\\times d\}, whered=ddev\+dstat\+ddensed=d\_\{\\mathrm\{dev\}\}\+d\_\{\\mathrm\{stat\}\}\+d\_\{\\mathrm\{dense\}\}is the number of feature dimension\. Each turn node is therefore associated with a feature vector𝐱t=𝐗\[t,:\]∈ℝd\\mathbf\{x\}\_\{t\}=\\mathbf\{X\}\[t,:\]\\in\\mathbb\{R\}^\{d\}\. The formal definitions of deviation and statistical features are deferred to Appendix[A](https://arxiv.org/html/2608.18575#A1)\.

#### Edge construction\.

We construct two types of connections:temporal progressionandintra\-agent dependency\. Firstly, to preserve the sequential nature of the conversation, we add bidirectional temporal edges between adjacent turns:\(vt,vt\+1\)∈ℰseq→,\(vt\+1,vt\)∈ℰseq←\(v\_\{t\},v\_\{t\+1\}\)\\in\\mathcal\{E\}\_\{\\mathrm\{seq\}\}^\{\\rightarrow\},\(v\_\{t\+1\},v\_\{t\}\)\\in\\mathcal\{E\}\_\{\\mathrm\{seq\}\}^\{\\leftarrow\}\. Secondly, to capture long\-range consistency across non\-adjacent turns induced by the same agent, we connect turns produced by the same agent\. For two turnsu<vu<vsuch thataiu=aiva\_\{i\_\{u\}\}=a\_\{i\_\{v\}\}, we add\(vu,vv\)∈ℰagent→,\(vv,vu\)∈ℰagent←\(v\_\{u\},v\_\{v\}\)\\in\\mathcal\{E\}\_\{\\mathrm\{agent\}\}^\{\\rightarrow\},\(v\_\{v\},v\_\{u\}\)\\in\\mathcal\{E\}\_\{\\mathrm\{agent\}\}^\{\\leftarrow\}\. Therefore, the full edge set isℰ=ℰseq→∪ℰseq←∪ℰagent→∪ℰagent←\\mathcal\{E\}=\\mathcal\{E\}\_\{\\mathrm\{seq\}\}^\{\\rightarrow\}\\cup\\mathcal\{E\}\_\{\\mathrm\{seq\}\}^\{\\leftarrow\}\\cup\\mathcal\{E\}\_\{\\mathrm\{agent\}\}^\{\\rightarrow\}\\cup\\mathcal\{E\}\_\{\\mathrm\{agent\}\}^\{\\leftarrow\}\.

### 3\.2AFANet: A Lightweight GNN for Agent Failure Attribution

After constructing the turn\-level graph𝒢\\mathcal\{G\}, we introduceAFANet, a model that captures both structural and semantic signals to solve AFA\. Specifically, turn features are first projected into a hidden space , followed by graph neural network layers that propagate neighboring turn information along the conversation structure\. Finally, agent\-level representations are obtained and mapped to prediction scores that quantify the likelihood of each agent being associated with certain error types\. We detail the procedure ofAFANetas follows\.

#### Input projection\.

We represent each turn nodettin the conversation graph with a concatenated feature vector𝐱t∈ℝd\\mathbf\{x\}\_\{t\}\\in\\mathbb\{R\}^\{d\}\. Before graph message passing, we project these initial node features into a hidden dimensiondhiddend\_\{\\mathrm\{hidden\}\}\. This is achieved via a projection module consisting of a linear transformation, a ReLU activation, and layer normalization:

𝐡t\(0\)=LN⁡\(ReLU⁡\(𝐖in​𝐱t\+𝐛in\)\),\\mathbf\{h\}\_\{t\}^\{\(0\)\}=\\operatorname\{LN\}\(\\operatorname\{ReLU\}\(\\mathbf\{W\}\_\{\\mathrm\{in\}\}\\mathbf\{x\}\_\{t\}\+\\mathbf\{b\}\_\{\\mathrm\{in\}\}\)\),where𝐖in\\mathbf\{W\}\_\{\\mathrm\{in\}\}is learnable weight matrix,𝐛in\\mathbf\{b\}\_\{\\mathrm\{in\}\}is learnable bias andLN\\operatorname\{LN\}denotes layer normalization\.

#### Graph Neural Network blocks\.

To capture contextual dependencies and the structural flow of the conversation, we applyLLlayers of message passing over the conversation graph\. We also incorporate residual skip connections at each layer\. The node representation at layerℓ\+1\\ell\+1is computed as:

𝐡t\(ℓ\+1\)=𝐡t\(ℓ\)\+GNN\(ℓ\)⁡\(𝐡t\(ℓ\),\{𝐡u\(ℓ\):\(u,t\)∈ℰ\}\),\\mathbf\{h\}\_\{t\}^\{\(\\ell\+1\)\}=\\mathbf\{h\}\_\{t\}^\{\(\\ell\)\}\+\\operatorname\{GNN\}^\{\(\\ell\)\}\\big\(\\mathbf\{h\}\_\{t\}^\{\(\\ell\)\},\\\{\\mathbf\{h\}\_\{u\}^\{\(\\ell\)\}:\(u,t\)\\in\\mathcal\{E\}\\\}\\big\),whereℰ\\mathcal\{E\}represents the set of edges in the graph\. AfterLLlayers, we obtain the per\-turn node representation𝐡tgnn=𝐡t\(L\)∈ℝdhidden\\mathbf\{h\}\_\{t\}^\{\\mathrm\{gnn\}\}=\\mathbf\{h\}\_\{t\}^\{\(L\)\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{hidden\}\}\}\. Note that theGNN\\operatorname\{GNN\}layer can be instantiated with any architecture \(e\.g\. GCN\[[9](https://arxiv.org/html/2608.18575#bib.bib21)\], GAT\[[21](https://arxiv.org/html/2608.18575#bib.bib31)\]and GraphSAGE\[[8](https://arxiv.org/html/2608.18575#bib.bib32)\]\)\.

#### Agent\-level pooling\.

Since the final classification objective is performed at the agent level, we aggregate the turn\-level representations belonging to each respective agent\. For a given agentaja\_\{j\}with the corresponding turn index setℐj\\mathcal\{I\}\_\{j\}, we apply a dual pooling strategy that concatenates both the mean\-pooled and max\-pooled representations for better expressiveness:

𝐳j=\[1\|ℐj\|∑t∈ℐj𝐡tgnn∥maxt∈ℐj𝐡tgnn\]\.\\mathbf\{z\}\_\{j\}=\\left\[\\frac\{1\}\{\|\\mathcal\{I\}\_\{j\}\|\}\\sum\_\{t\\in\\mathcal\{I\}\_\{j\}\}\\mathbf\{h\}\_\{t\}^\{\\mathrm\{gnn\}\}\\;\\\|\\;\\max\_\{t\\in\\mathcal\{I\}\_\{j\}\}\\mathbf\{h\}\_\{t\}^\{\\mathrm\{gnn\}\}\\right\]\.

#### Prediction head\.

Given the pooled agent representation𝐳j∈ℝ2​dhidden\\mathbf\{z\}\_\{j\}\\in\\mathbb\{R\}^\{2d\_\{\\mathrm\{hidden\}\}\}, we map it to the prediction space through a multi\-layer projection head\. We first obtain an intermediate hidden representation for each agent:

𝐡j=LN⁡\(ReLU⁡\(𝐖1​𝐳j\+𝐛1\)\),\\mathbf\{h\}\_\{j\}=\\operatorname\{LN\}\\\!\\left\(\\operatorname\{ReLU\}\(\\mathbf\{W\}\_\{1\}\\mathbf\{z\}\_\{j\}\+\\mathbf\{b\}\_\{1\}\)\\right\),where𝐡j∈ℝdhidden\\mathbf\{h\}\_\{j\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{hidden\}\}\}\. We then apply a bottleneck projection followed by a non\-linear activation and dropout:

𝐡~j=Dropout⁡\(ReLU⁡\(𝐖mid​𝐡j\+𝐛mid\)\),\\tilde\{\\mathbf\{h\}\}\_\{j\}=\\operatorname\{Dropout\}\\\!\\left\(\\operatorname\{ReLU\}\(\\mathbf\{W\}\_\{\\mathrm\{mid\}\}\\mathbf\{h\}\_\{j\}\+\\mathbf\{b\}\_\{\\mathrm\{mid\}\}\)\\right\),where𝐡~j∈ℝdmid\\tilde\{\\mathbf\{h\}\}\_\{j\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{mid\}\}\}anddmidd\_\{\\mathrm\{mid\}\}is the bottleneck dimension size\. Finally, we map to the prediction logits:

𝐬j=𝐖out​𝐡~j\+𝐛out,\\mathbf\{s\}\_\{j\}=\\mathbf\{W\}\_\{\\mathrm\{out\}\}\\tilde\{\\mathbf\{h\}\}\_\{j\}\+\\mathbf\{b\}\_\{\\mathrm\{out\}\},where𝐬j∈ℝK\+1\\mathbf\{s\}\_\{j\}\\in\\mathbb\{R\}^\{K\+1\}represents the un\-normalized scores overKKclasses\. TheK\+1K\+1classes correspond to class00for a clean type \(i\.e\. no error occurs\) and classes1,…,K−11,\\dots,K\-1for different error types\.

### 3\.3Training Objectives

Given the agent\-level representations\{𝐡j\}j=1M\\\{\\mathbf\{h\}\_\{j\}\\\}\_\{j=1\}^\{M\}, we optimizeAFANetusing a two\-term objective that combines \(i\) agent\-level fault detection and \(ii\) error\-type classification\.

#### Agent\-level objective\.

We first derive a binary fault logitaja\_\{j\}from the prediction scores𝐬j∈ℝK\+1\\mathbf\{s\}\_\{j\}\\in\\mathbb\{R\}^\{K\+1\}for each agentjj:

aj=log∑k=1Kexp\(sj,k\)−logexp\(sj,0\)\.a\_\{j\}=\\log\\sum\_\{k=1\}^\{K\}\\exp\(s\_\{j,k\}\)\-\\log\\exp\(s\_\{j,0\}\)\.We then apply a weighted binary cross\-entropy loss with logits:

ℒagent=1M​∑j=1MBCEWithLogits⁡\(aj,yjbinary,w\+\),\\mathcal\{L\}\_\{\\mathrm\{agent\}\}=\\frac\{1\}\{M\}\\sum\_\{j=1\}^\{M\}\\operatorname\{BCEWithLogits\}\(a\_\{j\},y\_\{j\}^\{\\mathrm\{binary\}\};\\,w^\{\+\}\),whereyjbinary=𝕀\[yj≠0\]y\_\{j\}^\{\\mathrm\{binary\}\}=\\mathbb\{I\}\[y\_\{j\}\\neq 0\]denotes whether agentjjis faulty and the positive\-class weight is defined asw\+=ncleannfaultyw^\{\+\}=\\frac\{n\_\{\\text\{clean\}\}\}\{n\_\{\\text\{faulty\}\}\}, wherenclean,nfaultyn\_\{\\text\{clean\}\},n\_\{\\text\{faulty\}\}is the number of non\-faulty and faulty agents summing over the conversations in the batch\.

#### Error\-level objective\.

For faulty agentsℱ=\{j∣yj≠0\}\\mathcal\{F\}=\\\{j\\mid y\_\{j\}\\neq 0\\\}, we apply a multi\-class cross\-entropy loss over the error\-type subspace:

ℒfm=1\|ℱ\|​∑j∈ℱCE⁡\(\[sj,1,…,sj,K\],yj−1\)\.\\mathcal\{L\}\_\{\\mathrm\{fm\}\}=\\frac\{1\}\{\|\\mathcal\{F\}\|\}\\sum\_\{j\\in\\mathcal\{F\}\}\\operatorname\{CE\}\(\[s\_\{j,1\},\\dots,s\_\{j,K\}\],y\_\{j\}\-1\)\.

#### Overall objective\.

The final loss is simply the weighted sum over the above two terms:ℒ=λagent​ℒagent\+λfm​ℒfm\\mathcal\{L\}=\\lambda\_\{\\mathrm\{agent\}\}\\mathcal\{L\}\_\{\\mathrm\{agent\}\}\+\\lambda\_\{\\mathrm\{fm\}\}\\mathcal\{L\}\_\{\\mathrm\{fm\}\}, whereλagent,λfm\\lambda\_\{\\mathrm\{agent\}\},\\lambda\_\{\\mathrm\{fm\}\}are hyper\-parameters\.

### 3\.4Inference withAFANet

We first compute the score vector𝐬j∈ℝK\+1\\mathbf\{s\}\_\{j\}\\in\\mathbb\{R\}^\{K\+1\}for each agent, wheresj,0s\_\{j,0\}denotes the*clean*class\. We derive an agent\-level fault score: for each agentjj,scorejagent=ℙ⁡\[agentj​is faulty\]=∑k=1Kexp⁡\(sj,k\)∑k=0Kexp⁡\(sj,k\)\\text\{score\}^\{\\mathrm\{agent\}\}\_\{j\}=\\mathbb\{P\}\[\\mathrm\{agent\_\{j\}\\text\{\\,is faulty\}\}\]=\\frac\{\\sum\_\{k=1\}^\{K\}\\exp\(s\_\{j,k\}\)\}\{\\sum\_\{k=0\}^\{K\}\\exp\(s\_\{j,k\}\)\}\. During inference, we consider both threshold\-based and ranking\-based decoding\. Forthreshold\-based decoding, we utilize the validation set to sweep the best thresholdτ\\tauthat maximizes agent\-level metric, and then fixτ\\tauto selectτfm\\tau\_\{\\mathrm\{fm\}\}that maximizes error\-level metric\. We predict faulty agents asℱ^=\{j∣scorejagent≥τ\}\\hat\{\\mathcal\{F\}\}=\\\{j\\mid\\text\{score\}^\{\\mathrm\{agent\}\}\_\{j\}\\geq\\tau\\\}\. For eachj∈ℱ^j\\in\\hat\{\\mathcal\{F\}\}, we assigne^j=arg⁡maxk≥1⁡sj,k\\hat\{e\}\_\{j\}=\\arg\\max\_\{k\\geq 1\}s\_\{j,k\}and retain\(aj,e^j\)\(a\_\{j\},\\hat\{e\}\_\{j\}\)only ifsj,e^j≥τfms\_\{j,\\hat\{e\}\_\{j\}\}\\geq\\tau\_\{\\mathrm\{fm\}\}\. Forranking\-based decoding, we rank agents byscorejagent\\mathrm\{score\}^\{\\mathrm\{agent\}\}\_\{j\}and directly select the top\-ranked agent\(s\), assigning each selected agent the error type with the largest score\.

## 4Experiment

Table 1:Performance comparison on AEGIS\-Bench and Who&When\. Best and second\-best results are inboldandunderline\. All scores are percentages \(%\)\. For the Qwen\-based models,Instructare abbreviated asIt, whileNon\-ThinkingandThinkingmodes are abbreviated asNTandT\. Their SFT and subsequent GRPO\-trained variants are denoted by\+Sand\+S\+G\. Rows marked with∗∗indicate models re\-implemented and evaluated by us under the same setting, while the remaining rows are adapted from\[[10](https://arxiv.org/html/2608.18575#bib.bib9)\]\.CategoryMethodAEGIS\-BenchWho&WhenAvg\.AgentErrorPairAgentErrorPairμ\\muF1MF1μ\\muF1MF1μ\\muF1MF1μ\\muF1MF1μ\\muF1MF1μ\\muF1MF1BaselineRandom4\.543\.5611\.2311\.150\.330\.211\.060\.838\.747\.140\.110\.054\.08Pre\-trainedLLMsQwen2\.5\-7B\-It27\.5514\.4914\.9611\.365\.022\.5240\.9223\.503\.641\.772\.311\.1412\.43Qwen2\.5\-7B\-It∗∗26\.3816\.1515\.1411\.543\.671\.7646\.2537\.853\.261\.942\.062\.4414\.04Qwen2\.5\-14B\-It35\.7812\.7120\.245\.915\.472\.2049\.8833\.191\.561\.350\.000\.0013\.99Qwen2\.5\-14B\-It∗∗36\.7219\.6218\.5916\.495\.862\.6145\.4744\.830\.000\.000\.000\.0015\.85Qwen3\-8B\-NT21\.348\.1615\.8113\.893\.961\.4027\.7817\.643\.881\.913\.881\.8110\.12Qwen3\-8B\-T34\.639\.0117\.4814\.314\.421\.5237\.9127\.584\.652\.211\.951\.1013\.06Fine\-tunedLLMsQwen2\.5\-7B\-It\+S60\.0322\.7019\.6116\.905\.052\.8043\.5132\.516\.774\.201\.260\.5217\.99Qwen2\.5\-7B\-It\+S∗∗37\.9322\.1417\.6913\.854\.482\.0256\.7765\.815\.424\.753\.944\.5819\.97Qwen2\.5\-7B\-It\+S\+G35\.4314\.8617\.2110\.547\.112\.7750\.7730\.143\.862\.302\.311\.1914\.87Qwen2\.5\-14B\-It\+S76\.5347\.9727\.5327\.6616\.629\.9951\.1436\.949\.877\.774\.032\.0826\.51Qwen2\.5\-14B\-It\+S∗∗41\.0421\.8017\.1613\.834\.382\.0249\.3764\.901\.771\.730\.410\.2518\.22Qwen2\.5\-14B\-It\+S\+G49\.7418\.3821\.1916\.106\.842\.5554\.4340\.884\.152\.672\.451\.4918\.41Qwen3\-8B\-NT\+S64\.7938\.9620\.3720\.369\.685\.7345\.4830\.778\.005\.295\.172\.3321\.41Qwen3\-8B\-NT\+S\+G45\.9117\.3920\.8915\.156\.942\.8250\.9438\.262\.211\.682\.211\.4517\.15Qwen3\-8B\-T\+G36\.1115\.7317\.9412\.034\.411\.6653\.1240\.5211\.256\.918\.103\.1917\.58ProprietaryLLMsGPT\-4\.137\.4811\.1220\.6515\.757\.442\.2742\.2928\.937\.005\.843\.361\.1615\.27GPT\-4o\-mini38\.5414\.7219\.9516\.025\.761\.6347\.4234\.215\.263\.332\.110\.9815\.83o340\.3123\.2722\.3716\.767\.862\.2753\.1042\.5514\.888\.637\.413\.9820\.24Gemini\-2\.5\-Flash42\.0216\.4523\.4719\.856\.992\.7655\.5636\.9811\.947\.967\.323\.3319\.55Gemini\-2\.5\-Pro41\.3216\.1519\.9316\.296\.962\.8853\.1134\.9211\.078\.116\.812\.6918\.35Claude\-Sonnet\-440\.7315\.5121\.2116\.557\.682\.3444\.7637\.2313\.339\.236\.772\.6618\.16OursAFANet74\.1647\.8627\.0125\.9617\.4216\.3537\.9320\.2613\.796\.056\.904\.1624\.82

In this section, we conduct extensive experiments to evaluate the effectiveness ofAFANetagainst LLM\-based baselines\. Our experiments are designed to answer the following research questions:

- •\(RQ1\): How doesAFANetcompare to LLM\-based methods w\.r\.t\. efficacy and efficiency?
- •\(RQ2\): How doesAFANetperform under different GNN backbones and hyper\-parameters?
- •\(RQ3\): How doesAFANetperform across domains and how to improve it?

### 4\.1Experimental Setup

#### Datasets\.

We conduct experiments on the AEGIS\-Bench\[[10](https://arxiv.org/html/2608.18575#bib.bib9)\]111[https://huggingface\.co/datasets/Fancylalala/AEGIS](https://huggingface.co/datasets/Fancylalala/AEGIS)and the Who&When\[[31](https://arxiv.org/html/2608.18575#bib.bib7)\]datasets222[https://huggingface\.co/datasets/Kevin355/Who\_and\_When](https://huggingface.co/datasets/Kevin355/Who_and_When)\. AEGIS\-Bench serves as the in\-domain dataset and has pre\-defined train/val/test splits\. On the other hand, Who&When is utilized as the out\-of\-distribution datasets and has exactly one faulty agent per conversation\. During training phase, the training and validation splits of AEGIS\-Bench are used to train and validateAFANet; the test split of AEGIS\-Bench and the entire Who&When are used to report the evaluation performances\. The data statistics is in Appendix[B](https://arxiv.org/html/2608.18575#A2)\.

#### Baselines\.

We primarily compareAFANetagainst a broad range of LLM\-based methods, including pre\-trained, fine\-tuned and proprietary LLMs\. Following[10](https://arxiv.org/html/2608.18575#bib.bib9), we adopt theAll\-at\-Onceprompting strategy for all LLM baselines, where the model is provided with the user query together with the complete failure trajectory, and is directly asked to identify the faulty agent as well as the corresponding error typeall at once\. The evaluated LLM backbones includeQwen2\.5\[[20](https://arxiv.org/html/2608.18575#bib.bib23)\],Qwen3\[[27](https://arxiv.org/html/2608.18575#bib.bib24)\],GPT\-4\.1\[[14](https://arxiv.org/html/2608.18575#bib.bib25)\],GPT\-4o\-mini\[[13](https://arxiv.org/html/2608.18575#bib.bib27)\],o3\[[15](https://arxiv.org/html/2608.18575#bib.bib26)\],Gemini\-2\.5\-Flash/Pro\[[4](https://arxiv.org/html/2608.18575#bib.bib28)\]andClaude\-Sonnet\-4\[[1](https://arxiv.org/html/2608.18575#bib.bib29)\]\. We also consider the supervised fine\-tuning \(\+S\) and subsequent GRPO\-trained \(\+S\+G\) variants on small\-scaleQwen2\.5andQwen3models\.

#### Evaluation Metrics\.

Following[10](https://arxiv.org/html/2608.18575#bib.bib9), we evaluate the attribution accuracy at three levels of granularity:Pair\-level\(correct agent\-error pairs\),Agent\-level\(correct faulty agents, ignoring the error types\) andError\-level\(correct error types, ignoring faulty agents\)\. For each level,Micro\-F1\(μ\\muF1\) andMacro\-F1\(MF1\) are reported\. Micro\-F1 pools predictions over all samples to compute a single global score\. In contrast, Macro\-F1 calculates the F1 score separately for each class \(e\.g\., each of the error types\) and then averages them uniformly\. The details of the error type taxonomy is in Appendix[C](https://arxiv.org/html/2608.18575#A3)\.

#### Implementation Details\.

ForAFANet, we default its backbone GNN to a 2\-layer GCN\[[9](https://arxiv.org/html/2608.18575#bib.bib21)\]with a hidden dimension of 64 and a bottleneck dimension of 32\. The model is trained using the Adam optimizer with a learning rate of1×10−21\\times 10^\{\-2\}for up to 100 epochs, with early stopping based on validation performance using a patience of 10 epochs\. We set the batch size to 2000 and apply dropout rate 0\.1\. We employall\-MiniLM\-L6\-v2\[[16](https://arxiv.org/html/2608.18575#bib.bib22)\]as the embedding model to encode turn node contexts\. The best model is selected based on the validation pair\-level Micro\-F1 score\. ForAFANetinference, we utilize Micro\-F1 to select agent thresholdτ\\tauand error thresholdτf​m\\tau\_\{fm\}on AEGIS\-Bench; we utilize ranking\-based decoding on Who&When to capture the single error nature\. All experiments forAFANetare conducted on an NVIDIA V100 GPU\. For the LLM\-based methods, we fine\-tuneQwen2\.5\-7B\-Instruct&Qwen2\.5\-14B\-Instructourselves and evaluate them under the same setting\. The versions we re\-implement are marked with∗∗in Table[1](https://arxiv.org/html/2608.18575#S4.T1)\. The prompts and the evaluation scripts for the re\-implementation are from the AEGIS\[[10](https://arxiv.org/html/2608.18575#bib.bib9)\]codebase333[https://github\.com/kfq20/AEGIS](https://github.com/kfq20/AEGIS)\.

### 4\.2Main Results

Table 2:Efficiency comparison betweenAFANetand LLM\-based baselines\. ForAFANet, preprocessing time summed over train/validation/test graph construction time\. Inference time is reported as in\-domain / OOD, respectively\.MethodTraining TimeInference TimePreprocessing TimeTrainable Params7B SFT6 hrs199s / 108s–7B7B SFT \+ GRPO\> 26 hrs199s / 108s–7B14B SFT8\.8 hrs367s / 231s–14B14B SFT \+ GRPO\> 74 hrs367s / 231s–14BAFANet1\.1 hrs1\.16s / 0\.37s80\.8s65K#### AFANetis comparable to or outperforms LLMs\.

We present our main results in Table[1](https://arxiv.org/html/2608.18575#S4.T1)\.AFANetachieves competitive or superior average performance compared to pretrained, post\-trained and proprietary LLM baselines over two in\-domain and OOD datasets\. In particular, despite not relying on LLM\-based reasoning,AFANetis highly effective w\.r\.t\. the pair\-level metrics\.

We note that pair\-level attribution is the most challenging and practically important, since it requires simultaneously identifying both the faulty agent and the correct error type\. On AEGIS\-Bench,AFANetachieves top\-1 pair\-level performances\. More importantly, under the OOD setting on Who&When,AFANetremains highly\-ranked on pair\-level attribution, demonstrating strong robustness under distribution shift\. We hypothesize that this advantage mainly comes from modeling structural interaction dynamics and consistency patterns, which remain relatively stable across trajectories; however, post\-trained approaches may overfit to surface\-level semantic patterns or dataset\-specific failure distributions\.

#### AFANetis much more efficient than LLMs\.

As shown in Figure[1](https://arxiv.org/html/2608.18575#S0.F1)and Table[2](https://arxiv.org/html/2608.18575#S4.T2),AFANetachieves significantly lower training time, inference time and trainable parameters compared to LLM counterpart\. The preprocessing time ofAFANetis also neglectable \(i\.e\. the time for conversation graph construction over all splits\)\.

### 4\.3Discussions

#### Backbone sensitivity\.

Table 3:Backbone sensitivity study ofAFANet\. Best and second\-best results are inboldandunderline\. We show thatAFANetremains robust across different GNN architectures and model depths\. All scores are percentages \(%\)\. The reference model uses a 2\-layer GCN with both edge types\.BackboneAEGIS\-BenchWho&WhenAvg\.AgentErrorPairAgentErrorPairμ\\muF1MF1μ\\muF1MF1μ\\muF1MF1μ\\muF1MF1μ\\muF1MF1μ\\muF1MF1Main ModelGCN \(2\-layer, all edges\)74\.1647\.8627\.0125\.9617\.4216\.3537\.9320\.2613\.796\.056\.904\.1624\.82GNN BackboneGAT \(2\-layer, all edges\)71\.9243\.3124\.2323\.1316\.5713\.3834\.4825\.577\.473\.670\.460\.2522\.04GraphSAGE \(2\-layer, all edges\)74\.3450\.7624\.9424\.6816\.4214\.3030\.4623\.698\.054\.533\.451\.5523\.10GNN DepthGCN \(1\-layer, all edges\)71\.1742\.1723\.2321\.8816\.4614\.3934\.4824\.785\.173\.171\.150\.8521\.58GCN \(3\-layer, all edges\)72\.8444\.2525\.0524\.6117\.9614\.2231\.6120\.459\.775\.623\.451\.8522\.64

We present backbone sensitivity in Table[4](https://arxiv.org/html/2608.18575#S4.T4)to evaluate the robustness ofAFANetunder different architectural choices\. We observe that the overall performance remains relatively stable across different GNN architectures and layer configurations\. These observations suggest that the effectiveness ofAFANetis not tied to a specific GNN implementation or carefully tuned depth, indicating good architectural robustness\.

#### Ablation\.

Table 4:Ablation study ofAFANet\. Best and second\-best results are inboldandunderline\. We show that graph structure, edge design and each feature component all contribute toAFANet’s performance\. All scores are percentages \(%\)\. The reference model uses a 2\-layer GCN with both edge types\.SettingAEGIS\-BenchWho&WhenAvg\.AgentErrorPairAgentErrorPairμ\\muF1MF1μ\\muF1MF1μ\\muF1MF1μ\\muF1MF1μ\\muF1MF1μ\\muF1MF1Main ModelGCN \(all edges\)74\.1647\.8627\.0125\.9617\.4216\.3537\.9320\.2613\.796\.056\.904\.1624\.82Edge Type AblationGCN \(no edges\)71\.2643\.2124\.6523\.5418\.4915\.0332\.1825\.108\.625\.553\.452\.3922\.78GCN \(only same\-agent edges\)70\.5342\.7422\.3221\.5816\.3112\.2336\.7826\.216\.322\.954\.603\.1622\.14GCN \(only temporal edges\)74\.8253\.6024\.8224\.3917\.5416\.5235\.0620\.415\.753\.414\.601\.9923\.58Model Design AblationW/o dev/stat features65\.9035\.4723\.5922\.2514\.009\.8037\.9332\.005\.752\.891\.721\.1421\.04W/o sentence embedding73\.5848\.1823\.6222\.1414\.7810\.1637\.9321\.768\.055\.442\.301\.3322\.44W/o GNN71\.1143\.8026\.3425\.6117\.8114\.9326\.4418\.8611\.495\.682\.871\.8922\.24

In Table[4](https://arxiv.org/html/2608.18575#S4.T4), we further study the impact of connection types and some key model designs\. Firstly, removing all edges consistently degrades performance, demonstrating that relational structure is important for agent failure attribution\. Secondly, using only same\-agent edges or only temporal edges also leads to noticeable performance drops compared to the full graph, suggesting that both temporal interaction patterns and intra\-agent behavioral consistency provide complementary signals\.

For model design ablation, we observe that removing deviation/statistical features causes large overall performance degradation, particularly on pair\-level metrics, supporting our hypothesis that faulty agents are often characterized by abnormal interaction dynamics and consistency violations\. Removing the GNN module also harms performance, showing the importance of explicit structured modeling\.

#### Generalization study\.

Table 5:Test\-time adaptation results on Who&When\. Best and second\-best results are inboldandunderline\. We observe that pair\-level adaptation achieves the strongest overall gains\.CategoryMethodAgentErrorPairAvg\.μ\\muF1MF1μ\\muF1MF1μ\\muF1MF1OriginalBase38\.5122\.379\.775\.055\.172\.9213\.96TTAAgent35\.0621\.4214\.375\.566\.324\.4014\.52Pair35\.0620\.1514\.375\.777\.474\.9614\.63Pair\-Faulty39\.0822\.4711\.495\.066\.324\.0614\.75

WhileAFANetdemonstrates strong performance on the in\-domain benchmark, generalizing agent failure attribution under severe distribution shift remains an inherently challenging problem\. As a case study, we investigate whether lightweight test\-time adaptation \(TTA\) can improveAFANetunder OOD settings without requiring retraining or additional supervision\.

Motivated by entropy\-minimization \(EM\) based test\-time adaptation such as TENT\[[22](https://arxiv.org/html/2608.18575#bib.bib30)\], we explore whetherAFANetcan adapt to OOD trajectories by encouraging more confident predictions at inference time\. We consider three adaptation objectives at different prediction granularities as follows: \(i\)Agent\-level EMminimizes the binary entropy of agent faulty probability; \(ii\)Pair\-level EMencourages confident predictions over pairwise label space; and \(iii\)Faulty\-agent pair\-level EMapplies pair\-level entropy minimization only to agents that are likely to be faulty\. The TTA procedure and the detailed formulas of these strategies are in Appendix[D](https://arxiv.org/html/2608.18575#A4)\. Table[5](https://arxiv.org/html/2608.18575#S4.T5)shows that lightweight test\-time adaptation can consistently improve OOD performance\.

## 5Related Work

#### Direct prompting\.

Early approaches to agent failure attribution primarily rely on prompting pretrained LLMs to directly analyze interaction trajectories\[[31](https://arxiv.org/html/2608.18575#bib.bib7),[5](https://arxiv.org/html/2608.18575#bib.bib8)\]\. These methods treat attribution as a reasoning task over long execution logs\. However, these approaches rely on costly LLM inference and often require carefully designed prompting strategies\.

#### Finetuning\-based methods\.

Another line of work focuses on training specialized attribution models using synthetic data with labeled faulty trajectories\[[30](https://arxiv.org/html/2608.18575#bib.bib11),[29](https://arxiv.org/html/2608.18575#bib.bib10),[10](https://arxiv.org/html/2608.18575#bib.bib9)\]\. These methods generate large\-scale annotated failure trajectories through techniques such as error injection, counterfactual replay, or graph\-guided synthesis, and subsequently post\-train models using supervised fine\-tuning and reinforcement learning\. However, these methods introduce significant overhead in training cost and they still inherit the limitations of direct LLM inference at test time\.

#### Agentic systems\.

Some works explores more complex failure attribution frameworks\. In this category, some approaches employ more sophisticate prompting techniques\[[32](https://arxiv.org/html/2608.18575#bib.bib18)\], some construct structured representations such as causal graphs\[[24](https://arxiv.org/html/2608.18575#bib.bib14)\], hierarchical context models\[[2](https://arxiv.org/html/2608.18575#bib.bib13)\]and complex patterns\[[6](https://arxiv.org/html/2608.18575#bib.bib17),[18](https://arxiv.org/html/2608.18575#bib.bib15)\]to better capture inter\-agent dependencies, while others incorporate memory mechanisms to reuse previously observed failure patterns\[[28](https://arxiv.org/html/2608.18575#bib.bib12)\]\.

## 6Conclusion

We revisit agent failure attribution in LLM\-based multi\-agent systems and show that heavy generative approaches are not necessary for strong performance\. We proposeAFANet, a lightweight graph\-based framework that models interaction trajectories through simple step\-level signals and agent\-level structure, achieving competitive or superior results against LLM\-based methods with minimal computational cost\. These findings suggest that structured modeling of agent interactions is sufficient for effective attribution\. Future work includes designing more tailored aggregation operators and conducting in\-depth analysis of generalization under distribution shifts\.

## References

- \[1\]\(2025\)System card: claude opus 4 & claude sonnet 4\.Claude\-4 Model Card\.Cited by:[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px2.p1.1)\.
- \[2\]A\. Banerjee, A\. Nair, and T\. Borogovac\(2025\)Where did it all go wrong? a hierarchical look into multi\-agent error attribution\.arXiv preprint arXiv:2510\.04886\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p2.1),[§5](https://arxiv.org/html/2608.18575#S5.SS0.SSS0.Px3.p1.1)\.
- \[3\]M\. Cemri, M\. Z\. Pan, S\. Yang, L\. A\. Agrawal, B\. Chopra, R\. Tiwari, K\. Keutzer, A\. Parameswaran, D\. Klein, K\. Ramchandran,et al\.\(2025\)Why do multi\-agent llm systems fail?\.arXiv preprint arXiv:2503\.13657\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p1.1)\.
- \[4\]G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px2.p1.1)\.
- \[5\]D\. Deshpande, V\. Gangal, H\. Mehta, J\. Krishnan, A\. Kannappan, and R\. Qian\(2025\)Trail: trace reasoning and agentic issue localization\.arXiv preprint arXiv:2505\.08638\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p1.1),[§5](https://arxiv.org/html/2608.18575#S5.SS0.SSS0.Px1.p1.1)\.
- \[6\]Y\. Ge, L\. Xie, Z\. Li, Y\. Pei, and T\. Zhang\(2025\)Who is introducing the failure? automatically attributing failures of multi\-agent systems via spectrum analysis\.arXiv preprint arXiv:2509\.13782\.Cited by:[§5](https://arxiv.org/html/2608.18575#S5.SS0.SSS0.Px3.p1.1)\.
- \[7\]T\. Guo, X\. Chen, Y\. Wang, R\. Chang, S\. Pei, N\. V\. Chawla, O\. Wiest, and X\. Zhang\(2024\)Large language model based multi\-agents: a survey of progress and challenges\.arXiv preprint arXiv:2402\.01680\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p1.1)\.
- \[8\]W\. Hamilton, Z\. Ying, and J\. Leskovec\(2017\)Inductive representation learning on large graphs\.Advances in neural information processing systems30\.Cited by:[§3\.2](https://arxiv.org/html/2608.18575#S3.SS2.SSS0.Px2.p1.2)\.
- \[9\]T\. N\. Kipf and M\. Welling\(2016\)Semi\-supervised classification with graph convolutional networks\.arXiv preprint arXiv:1609\.02907\.Cited by:[§3\.2](https://arxiv.org/html/2608.18575#S3.SS2.SSS0.Px2.p1.2),[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px4.p1.1)\.
- \[10\]F\. Kong, R\. Zhang, H\. Yin, G\. Zhang, X\. Zhang, Z\. Chen, Z\. Zhang, X\. Zhang, S\. Zhu, and X\. Feng\(2025\)Aegis: automated error generation and attribution for multi\-agent systems\.arXiv preprint arXiv:2509\.14295\.Cited by:[Table 6](https://arxiv.org/html/2608.18575#A2.T6.2.1.1),[Appendix C](https://arxiv.org/html/2608.18575#A3.p1.1),[§1](https://arxiv.org/html/2608.18575#S1.p1.1),[§1](https://arxiv.org/html/2608.18575#S1.p2.1),[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px4.p1.1),[Table 1](https://arxiv.org/html/2608.18575#S4.T1),[§5](https://arxiv.org/html/2608.18575#S5.SS0.SSS0.Px2.p1.1)\.
- \[11\]X\. Li, S\. Wang, S\. Zeng, Y\. Wu, and Y\. Yang\(2024\)A survey on llm\-based multi\-agent systems: workflow, infrastructure, and challenges\.Vicinagearth1\(1\),pp\. 9\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p1.1)\.
- \[12\]X\. Ma, X\. Xie, Y\. Wang, J\. Wang, B\. Wu, M\. Li, and Q\. Wang\(2025\)Diagnosing failure root causes in platform\-orchestrated agentic systems: dataset, taxonomy, and benchmark\.arXiv preprint arXiv:2509\.23735\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p1.1)\.
- \[13\]OpenAI\(2024\)GPT\-4o\-mini\.Note:[https://openai\.com/index/gpt\-4o\-system\-card/](https://openai.com/index/gpt-4o-system-card/)Cited by:[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px2.p1.1)\.
- \[14\]OpenAI\(2025\)GPT\-4\.1\.Note:[https://openai\.com/index/gpt\-4\-1/](https://openai.com/index/gpt-4-1/)Cited by:[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px2.p1.1)\.
- \[15\]OpenAI\(2025\)O3\.Note:[https://openai\.com/index/introducing\-o3\-and\-o4\-mini/](https://openai.com/index/introducing-o3-and-o4-mini/)Cited by:[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px2.p1.1)\.
- \[16\]N\. Reimers and I\. Gurevych\(2019\)Sentence\-bert: sentence embeddings using siamese bert\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://arxiv.org/abs/1908.10084)Cited by:[§3\.1](https://arxiv.org/html/2608.18575#S3.SS1.SSS0.Px1.p3.1),[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px4.p1.1)\.
- \[17\]A\. Roy, J\. Shu, J\. Li, C\. Yang, O\. Elshocht, J\. Smeets, and P\. Li\(2024\)Gad\-nr: graph anomaly detection via neighborhood reconstruction\.InProceedings of the 17th ACM international conference on web search and data mining,pp\. 576–585\.Cited by:[§3\.1](https://arxiv.org/html/2608.18575#S3.SS1.SSS0.Px1.p1.1)\.
- \[18\]K\. Sun, W\. Li, B\. Dong, Y\. Lin, J\. Zhang, and B\. Shi\(2026\)Scope delineation before localization: a two\-stage framework for enhancing failure attribution in multi\-agent systems\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 33108–33116\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p2.1),[§5](https://arxiv.org/html/2608.18575#S5.SS0.SSS0.Px3.p1.1)\.
- \[19\]J\. Tang, J\. Li, Z\. Gao, and J\. Li\(2022\)Rethinking graph neural networks for anomaly detection\.InInternational conference on machine learning,pp\. 21076–21089\.Cited by:[§3\.1](https://arxiv.org/html/2608.18575#S3.SS1.SSS0.Px1.p1.1)\.
- \[20\]Q\. Team\(2024\)Qwen2\.5: a party of foundation models\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5/)Cited by:[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px2.p1.1)\.
- \[21\]P\. Veličković, G\. Cucurull, A\. Casanova, A\. Romero, P\. Lio, and Y\. Bengio\(2017\)Graph attention networks\.arXiv preprint arXiv:1710\.10903\.Cited by:[§3\.2](https://arxiv.org/html/2608.18575#S3.SS2.SSS0.Px2.p1.2)\.
- \[22\]D\. Wang, E\. Shelhamer, S\. Liu, B\. Olshausen, and T\. Darrell\(2020\)Tent: fully test\-time adaptation by entropy minimization\.arXiv preprint arXiv:2006\.10726\.Cited by:[§4\.3](https://arxiv.org/html/2608.18575#S4.SS3.SSS0.Px3.p2.1)\.
- \[23\]J\. Wang, Y\. Wang, M\. Chen, X\. Xie, C\. Chen, F\. Mu, Z\. Liu, and Q\. Wang\(2026\)A survey for llm agent trajectory analysis: from failure attribution to enhancement\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p1.1)\.
- \[24\]Y\. Wang, W\. Wu, J\. Wang, and Q\. Wang\(2026\)From flat logs to causal graphs: hierarchical failure attribution for llm\-based multi\-agent systems\.arXiv preprint arXiv:2602\.23701\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p2.1),[§5](https://arxiv.org/html/2608.18575#S5.SS0.SSS0.Px3.p1.1)\.
- \[25\]T\. Wei, T\. Li, Z\. Liu, X\. Ning, Z\. Yang, J\. Zou, Z\. Zeng, R\. Qiu, X\. Lin, D\. Fu,et al\.\(2026\)Agentic reasoning for large language models\.arXiv preprint arXiv:2601\.12538\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p1.1)\.
- \[26\]Z\. Wu, S\. Pan, F\. Chen, G\. Long, C\. Zhang, and P\. S\. Yu\(2020\)A comprehensive survey on graph neural networks\.IEEE transactions on neural networks and learning systems32\(1\),pp\. 4–24\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p3.1)\.
- \[27\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px2.p1.1)\.
- \[28\]Y\. Yu, M\. Li, S\. Xu, J\. Fu, X\. Hou, F\. Lai, and B\. Wang\(2025\)CORRECT: condensed error recognition via knowledge transfer in multi\-agent systems\.arXiv preprint arXiv:2509\.24088\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p2.1),[§5](https://arxiv.org/html/2608.18575#S5.SS0.SSS0.Px3.p1.1)\.
- \[29\]G\. Zhang, J\. Wang, J\. Chen, W\. Zhou, K\. Wang, and S\. Yan\(2025\)AgenTracer: who is inducing failure in the llm agentic systems?\.arXiv preprint arXiv:2509\.03312\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p2.1),[§5](https://arxiv.org/html/2608.18575#S5.SS0.SSS0.Px2.p1.1)\.
- \[30\]H\. Zhang, Y\. Shi, X\. Gu, H\. You, Z\. Zhang, L\. Gan, Y\. Yuan, and J\. Huang\(2025\)GraphTracer: graph\-guided failure tracing in llm agents for robust multi\-turn deep search\.arXiv preprint arXiv:2510\.10581\.Cited by:[§1](https://arxiv.org/html/2608.18575#S1.p2.1),[§5](https://arxiv.org/html/2608.18575#S5.SS0.SSS0.Px2.p1.1)\.
- \[31\]S\. Zhang, M\. Yin, J\. Zhang, J\. Liu, Z\. Han, J\. Zhang, B\. Li, C\. Wang, H\. Wang, Y\. Chen,et al\.\(2025\)Which agent causes task failures and when? on automated failure attribution of llm multi\-agent systems\.arXiv preprint arXiv:2505\.00212\.Cited by:[Table 6](https://arxiv.org/html/2608.18575#A2.T6.2.1.2),[§1](https://arxiv.org/html/2608.18575#S1.p1.1),[§1](https://arxiv.org/html/2608.18575#S1.p2.1),[§4\.1](https://arxiv.org/html/2608.18575#S4.SS1.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.18575#S5.SS0.SSS0.Px1.p1.1)\.
- \[32\]C\. Zhu, S\. Hong, J\. Wu, K\. Chawla, Y\. Tang, Y\. Yin, N\. Wolfe, E\. Babinsky, and D\. Liu\(2026\)Raffles: reasoning\-based attribution of faults for llm systems\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7659–7688\.Cited by:[§5](https://arxiv.org/html/2608.18575#S5.SS0.SSS0.Px3.p1.1)\.

## Appendix AGraph construction

#### TF\-IDF computation\.

For each conversation, we first construct a local vocabulary from all turns after lowercasing and tokenization\. Based on this vocabulary, we compute turn\-level TF\-IDF representations using standard term frequency and smoothed inverse document frequency, followed by row\-wiseℓ2\\ell\_\{2\}normalization\. Let𝐗TF​\-​IDF∈ℝT×\|𝒱\|\\mathbf\{X\}\_\{\\mathrm\{TF\\mbox\{\-\}IDF\}\}\\in\\mathbb\{R\}^\{T\\times\|\\mathcal\{V\}\|\}denote the resulting TF\-IDF matrix, whereTTis the number of turns and\|𝒱\|\|\\mathcal\{V\}\|is the vocabulary size\.

To obtain compact dense representations that capture the dominant vocabulary structure within the conversation, we apply truncated SVD:

𝐙svd=SVDr⁡\(𝐗TF​\-​IDF\)∈ℝT×r,\\mathbf\{Z\}\_\{\\mathrm\{svd\}\}=\\operatorname\{SVD\}\_\{r\}\(\\mathbf\{X\}\_\{\\mathrm\{TF\\mbox\{\-\}IDF\}\}\)\\in\\mathbb\{R\}^\{T\\times r\},whererrdenotes the retained rank size\. Each row

𝐳svdt=𝐙svd\[t,:\]∈ℝr\\mathbf\{z\}\_\{\\mathrm\{svd\}\}^\{t\}=\\mathbf\{Z\}\_\{\\mathrm\{svd\}\}\[t,:\]\\in\\mathbb\{R\}^\{r\}provides anrr\-dimensional representation for turntt\. In our implementation, we user=16r=16\.

#### Deviation features\.

Letϕt=𝐳svdt\\phi\_\{t\}=\\mathbf\{z\}^\{t\}\_\{\\mathrm\{svd\}\}denote the TF\-IDF SVD feature of turntt, and letℐa=\{t∣ait=a\}\\mathcal\{I\}\_\{a\}=\\\{t\\mid a\_\{i\_\{t\}\}=a\\\}denote the set of turns generated by agentaa\.

- •Sequential deviation\.We measure how abruptly a turn deviates from the immediately preceding turn: fseq​\(t\)=1−cos⁡\(ϕt,ϕt−1\)\.f\_\{\\mathrm\{seq\}\}\(t\)=1\-\\cos\(\\phi\_\{t\},\\phi\_\{t\-1\}\)\.
- •Self inconsistency\.We measure deviation from the historical behavior of the same agent: μait=1\|ℐait\|​∑s∈ℐaitϕs,\\mu\_\{a\_\{i\_\{t\}\}\}=\\frac\{1\}\{\|\\mathcal\{I\}\_\{a\_\{i\_\{t\}\}\}\|\}\\sum\_\{s\\in\\mathcal\{I\}\_\{a\_\{i\_\{t\}\}\}\}\\phi\_\{s\},fself​\(t\)=1−cos⁡\(ϕt,μait\)\.f\_\{\\mathrm\{self\}\}\(t\)=1\-\\cos\(\\phi\_\{t\},\\mu\_\{a\_\{i\_\{t\}\}\}\)\.
- •Conversation consensus deviation\.We compute deviation from the global conversation centroid: μ𝒯=1T​∑s=1Tϕs,\\mu\_\{\\mathcal\{T\}\}=\\frac\{1\}\{T\}\\sum\_\{s=1\}^\{T\}\\phi\_\{s\},fcons​\(t\)=1−cos⁡\(ϕt,μ𝒯\)\.f\_\{\\mathrm\{cons\}\}\(t\)=1\-\\cos\(\\phi\_\{t\},\\mu\_\{\\mathcal\{T\}\}\)\.
- •Cross\-agent deviation\.We measure deviation from the average representation of all other agents: μ¬ait=1T−\|ℐait\|​∑s∉ℐaitϕs,\\mu\_\{\\neg a\_\{i\_\{t\}\}\}=\\frac\{1\}\{T\-\|\\mathcal\{I\}\_\{a\_\{i\_\{t\}\}\}\|\}\\sum\_\{s\\notin\\mathcal\{I\}\_\{a\_\{i\_\{t\}\}\}\}\\phi\_\{s\},fother​\(t\)=1−cos⁡\(ϕt,μ¬ait\)\.f\_\{\\mathrm\{other\}\}\(t\)=1\-\\cos\(\\phi\_\{t\},\\mu\_\{\\neg a\_\{i\_\{t\}\}\}\)\.
- •Agent internal consistency\.We compute the average pairwise similarity among all turns generated by the same agent: fwithin​\(a\)=1\|ℐa\|​\(\|ℐa\|−1\)​∑s,u∈ℐas≠ucos⁡\(ϕs,ϕu\)\.f\_\{\\mathrm\{within\}\}\(a\)=\\frac\{1\}\{\|\\mathcal\{I\}\_\{a\}\|\(\|\\mathcal\{I\}\_\{a\}\|\-1\)\}\\sum\_\{\\begin\{subarray\}\{c\}s,u\\in\\mathcal\{I\}\_\{a\}\\\\ s\\neq u\\end\{subarray\}\}\\cos\(\\phi\_\{s\},\\phi\_\{u\}\)\.
- •Same\-agent temporal consistency\.We measure similarity to the previous turn generated by the same agent: p⁡\(t\)=max⁡\{s<t∣ais=ait\},p\(t\)=\\max\\\{s<t\\mid a\_\{i\_\{s\}\}=a\_\{i\_\{t\}\}\\\},fprev​\(t\)=cos⁡\(ϕt,ϕp⁡\(t\)\)\.f\_\{\\mathrm\{prev\}\}\(t\)=\\cos\(\\phi\_\{t\},\\phi\_\{p\(t\)\}\)\.
- •Vocabulary stability\.Let𝒱t\\mathcal\{V\}\_\{t\}denote the token set of turntt, and let 𝒱ait\(−t\)=⋃s∈ℐait,s≠t𝒱s\.\\mathcal\{V\}\_\{a\_\{i\_\{t\}\}\}^\{\(\-t\)\}=\\bigcup\_\{s\\in\\mathcal\{I\}\_\{a\_\{i\_\{t\}\}\},\\,s\\neq t\}\\mathcal\{V\}\_\{s\}\.We measure the overlap ratio between the current turn and the remaining turns from the same agent: fvocab​\(t\)=\|𝒱t∩𝒱ait\(−t\)\|\|𝒱t\|\.f\_\{\\mathrm\{vocab\}\}\(t\)=\\frac\{\|\\mathcal\{V\}\_\{t\}\\cap\\mathcal\{V\}\_\{a\_\{i\_\{t\}\}\}^\{\(\-t\)\}\|\}\{\|\\mathcal\{V\}\_\{t\}\|\}\.
- •Problem alignment\.We measure similarity between the current turn and the initial problem statement: fprob​\(t\)=cos⁡\(ϕt,ϕ1\)\.f\_\{\\mathrm\{prob\}\}\(t\)=\\cos\(\\phi\_\{t\},\\phi\_\{1\}\)\.

#### Statistical features\.

- •Numerical statistics\.We include lightweight numerical indicators, including whether the turn contains numeric tokens: fnum​\(t\)=𝕀⁡\[∃digit token in​xt\],f\_\{\\mathrm\{num\}\}\(t\)=\\mathbb\{I\}\[\\exists\\text\{ digit token in \}x\_\{t\}\],as well as the normalized mean and standard deviation of word lengths within the conversation: fwmean​\(t\)=mean​\_​len⁡\(xt\)−μlenσlen,fwstd​\(t\)=std​\_​len⁡\(xt\)−μstdσstd\.f\_\{\\mathrm\{wmean\}\}\(t\)=\\frac\{\\operatorname\{mean\\\_len\}\(x\_\{t\}\)\-\\mu\_\{\\mathrm\{len\}\}\}\{\\sigma\_\{\\mathrm\{len\}\}\},\\qquad f\_\{\\mathrm\{wstd\}\}\(t\)=\\frac\{\\operatorname\{std\\\_len\}\(x\_\{t\}\)\-\\mu\_\{\\mathrm\{std\}\}\}\{\\sigma\_\{\\mathrm\{std\}\}\}\.
- •Position statistics\.We include normalized turn position fpos​\(t\)=t−1T−1,f\_\{\\mathrm\{pos\}\}\(t\)=\\frac\{t\-1\}\{T\-1\},binary indicators for whether the turn is the first or last turn: ffirst\(t\)=𝕀\[t=1\],flast\(t\)=𝕀\[t=T\],f\_\{\\mathrm\{first\}\}\(t\)=\\mathbb\{I\}\[t=1\],\\qquad f\_\{\\mathrm\{last\}\}\(t\)=\\mathbb\{I\}\[t=T\],and the cumulative fraction of turns authored by the same agent up to steptt: fagent​\_​frac​\(t\)=\|\{s≤t∣ais=ait\}\|t\.f\_\{\\mathrm\{agent\\\_frac\}\}\(t\)=\\frac\{\|\\\{s\\leq t\\mid a\_\{i\_\{s\}\}=a\_\{i\_\{t\}\}\\\}\|\}\{t\}\.
- •Length statistics\.Letℓt\\ell\_\{t\}denote the token length of turntt\. We include normalized turn length: flen​\(t\)=ℓtmaxs⁡ℓs,f\_\{\\mathrm\{len\}\}\(t\)=\\frac\{\\ell\_\{t\}\}\{\\max\_\{s\}\\ell\_\{s\}\},together with the conversation\-level z\-normalized length: fzlen​\(t\)=ℓt−μℓσℓ\.f\_\{\\mathrm\{zlen\}\}\(t\)=\\frac\{\\ell\_\{t\}\-\\mu\_\{\\ell\}\}\{\\sigma\_\{\\ell\}\}\.
- •Structural statistics\.We additionally include the total participation ratio of the corresponding agent: fpart​\(t\)=\|ℐait\|T,f\_\{\\mathrm\{part\}\}\(t\)=\\frac\{\|\\mathcal\{I\}\_\{a\_\{i\_\{t\}\}\}\|\}\{T\},and the spread of the agent’s participation positions: fspread​\(a\)=std⁡\(\{s−1T−1∣s∈ℐa\}\)\.f\_\{\\mathrm\{spread\}\}\(a\)=\\operatorname\{std\}\\left\(\\left\\\{\\frac\{s\-1\}\{T\-1\}\\mid s\\in\\mathcal\{I\}\_\{a\}\\right\\\}\\right\)\.

## Appendix BDataset Details

Both datasets ae under MIT licenses\.

AEGIS\-Bench\[[10](https://arxiv.org/html/2608.18575#bib.bib9)\]Who&When\[[31](https://arxiv.org/html/2608.18575#bib.bib7)\]TrainValidationTestTest71461787600184Table 6:Dataset size statistics used in our experiments\.
## Appendix CError Types

Adopted from\[[10](https://arxiv.org/html/2608.18575#bib.bib9)\], we consider the following 14 error types in agent failure attribution:

- •Specification Issues - –FM\-1\.1: Task specification deviation - –FM\-1\.2: Role specification deviation - –FM\-1\.3: Add redundant steps - –FM\-1\.4: Remove conversation history - –FM\-1\.5: Remove termination conditions
- •Inter\-Agent Misalignment - –FM\-2\.1: Repeat handled tasks - –FM\-2\.2: Make request ambiguous - –FM\-2\.3: Deviate from main goal - –FM\-2\.4: Hide important information - –FM\-2\.5: Ignore other agents - –FM\-2\.6: Inconsistent reasoning
- •Task Verification Failures - –FM\-3\.1: Premature termination - –FM\-3\.2: Remove verification steps - –FM\-3\.3: Incorrect verification

## Appendix DTest\-Time Adaptation forAFANet

Below we detail the loss objective for each TTA strategy\.

#### Agent\-level entropy minimization\.

Letaia\_\{i\}denote the scalar faulty\-agent logit for agentii, and letpi=σ⁡\(ai\)p\_\{i\}=\\sigma\(a\_\{i\}\)\. We define

ℒagent=1NA∑i=1NAHbin\(pi\)=−1NA∑i=1NA\[pilogpi\+\(1−pi\)log\(1−pi\)\]\.\\mathcal\{L\}\_\{\\mathrm\{agent\}\}=\\frac\{1\}\{N\_\{A\}\}\\sum\_\{i=1\}^\{N\_\{A\}\}H\_\{\\mathrm\{bin\}\}\(p\_\{i\}\)=\-\\frac\{1\}\{N\_\{A\}\}\\sum\_\{i=1\}^\{N\_\{A\}\}\\left\[p\_\{i\}\\log p\_\{i\}\+\(1\-p\_\{i\}\)\\log\(1\-p\_\{i\}\)\\right\]\.

#### Pair\-level entropy minimization\.

Let𝐬i∈ℝK\+1\\mathbf\{s\}\_\{i\}\\in\\mathbb\{R\}^\{K\+1\}denote the pair logits for agentii, where the first class corresponds to the clean class and the remainingKKclasses correspond to error types\. Withp^i​c=softmax​\(𝐬i\)c\\hat\{p\}\_\{ic\}=\\mathrm\{softmax\}\(\\mathbf\{s\}\_\{i\}\)\_\{c\}, we minimize

ℒpair=1NA∑i=1NAHcat\(𝐬i\)=−1NA∑i=1NA∑c=0Kp^i​clogp^i​c\.\\mathcal\{L\}\_\{\\mathrm\{pair\}\}=\\frac\{1\}\{N\_\{A\}\}\\sum\_\{i=1\}^\{N\_\{A\}\}H\_\{\\mathrm\{cat\}\}\(\\mathbf\{s\}\_\{i\}\)=\-\\frac\{1\}\{N\_\{A\}\}\\sum\_\{i=1\}^\{N\_\{A\}\}\\sum\_\{c=0\}^\{K\}\\hat\{p\}\_\{ic\}\\log\\hat\{p\}\_\{ic\}\.

#### Faulty\-agent pair\-level minimization\.

We first estimate the number of faulty agents ask=max\(1,∑i=1NA𝕀\[ai\>0\]\)k=\\max\\left\(1,\\sum\_\{i=1\}^\{N\_\{A\}\}\\mathbb\{I\}\[a\_\{i\}\>0\]\\right\)and select the top\-kkagents according to the faulty\-agent logitsaia\_\{i\}\. The adaptation objective is then

ℒpair​\-​faulty=1k∑i∈Top​\-​k​\(a\)Hcat\(𝐬i\)=−1k∑i∈Top​\-​k​\(a\)∑c=0Kp^i​clogp^i​c\.\\mathcal\{L\}\_\{\\mathrm\{pair\\text\{\-\}faulty\}\}=\\frac\{1\}\{k\}\\sum\_\{i\\in\\mathrm\{Top\}\\text\{\-\}k\(a\)\}H\_\{\\mathrm\{cat\}\}\(\\mathbf\{s\}\_\{i\}\)=\-\\frac\{1\}\{k\}\\sum\_\{i\\in\\mathrm\{Top\}\\text\{\-\}k\(a\)\}\\sum\_\{c=0\}^\{K\}\\hat\{p\}\_\{ic\}\\log\\hat\{p\}\_\{ic\}\.
For the TTA procedure, we freeze all parameters except the LayerNorm affine parameters across all layers inAFANetand minimize the above three losses\. We use Adam optimizer with learning rate1​e−31e\-3for 30 gradient steps over the entire OOD batch\. No OOD labels are used\. The final checkpoint is taken as the final model for evaluation\.

## Appendix ELimitation

- •OOD generalization\.AlthoughAFANetdemonstrates certain robustness under distribution shift, generalization across substantially different multi\-agent settings remains challenging\. Future work may include designing customized message passing operators and adaptive graph propagation mechanisms that better capture failure propagation patterns\.
- •Limited reasoning capability\.While lightweight structural modeling is effective for our studied attribution scenarios, certain failures may still require deeper semantic understanding and long\-horizon reasoning\. An important future direction is to combine graph\-based modeling with LLM reasoning to jointly leverage structural consistency and semantic inference\.

## Appendix FImpact Statement

This paper discusses the advancement of the field of Large Language Model and Graph Machine Learning\. While there are potential societal consequence of our work, none of which we feel must be hightlighted\.

## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: Our main contributions are summarized as four bullet points in Section 1 and each of them are supported by claims in introduction or experiment section\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification: We discuss potential limitations of our presented work in Appendix[E](https://arxiv.org/html/2608.18575#A5)\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[N/A\]
14. Justification: We don’t include theoretical results\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification: We provide all experimental details needed in Section[4\.1](https://arxiv.org/html/2608.18575#S4.SS1)\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[Yes\]
24. Justification: All the benchmarks used in this work is public\. We will provide the code package during submission and make the code available upon acceptance\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification: We provide all experimental details needed in Section[4\.1](https://arxiv.org/html/2608.18575#S4.SS1)\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[No\]
34. Justification: Our conclusions are well supported by consistent trends across multiple datasets, evaluation metrics, model backbones and ablation settings\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification: We provide all experimental details needed in Section[4\.1](https://arxiv.org/html/2608.18575#S4.SS1)\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: We confirm that this work is conducted with the NeurIPS Code of Ethics\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[Yes\]
49. Justification: We provide related discussion in Appendix[F](https://arxiv.org/html/2608.18575#A6)\.
50. Guidelines: - •The answer\[Yes\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification: We confirm that this work does not pose safety risks\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification: We detail them in Appendix[B](https://arxiv.org/html/2608.18575#A2)\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[N/A\]
64. Justification: No new assets introduced\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification: This paper does not involve human subjects\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: This paper does not involve human subjects\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[N/A\]
79. Justification: The proposed method is entirely LLM\-free and does not use LLMs as part of its core methodology\. Although some comparison baselines are LLM\-based methods, they are only included for evaluation purposes and are not key components of the proposed framework\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.