ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems

# ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems
Source: [https://arxiv.org/html/2610.11334](https://arxiv.org/html/2610.11334)
Weilin JinMingyu WangAffiliation:Peking University, Beijing, ChinaAffiliation:Joy Future Academy, Beijing, ChinaTaiyu ZhuAffiliation:Peking University, Beijing, ChinaZiqi ZhouAffiliation:Tsinghua University, Beijing, ChinaWenbo Li††thanks:Project leader\.Affiliation:Joy Future Academy, Beijing, ChinaHaoyang HuangAffiliation:Joy Future Academy, Beijing, ChinaNan DuanAffiliation:Joy Future Academy, Beijing, ChinaYifan Wu††thanks:Corresponding authors\.Affiliation:Peking University, Beijing, ChinaYing Li22footnotemark:2Affiliation:Peking University, Beijing, ChinaZhonghai WuAffiliation:Peking University, Beijing, China

###### Abstract

In LLM\-based agent systems, failures can originate from early steps whose effects propagate through subsequent interactions, making their origins difficult to identify\. To trace such failures back to their origin, failure attribution has been formulated as the task of identifying the earliest step responsible for the failure\. Recent methods leverage LLM internal signals for failure attribution, typically using hidden states as step representations\. We therefore conduct an empirical study to evaluate how effectively these representations distinguish root\-cause steps from other steps and find limited separation\. Motivated by this observation, we proposeReCast, a step representation learning method that transforms hidden states from a frozen LLM into attribution\-oriented step representations\. ReCast first selects attribution\-relevant layers, then constructs complementary pattern and deviation features, and finally learns contextualized step representations through an encoder trained with contrastive and ranking objectives\. We also introduce ReCast\-2K, a training dataset for failure attribution\. ReCast achieves the best Hit@1 across four benchmarks, surpassing the strongest baseline by 5\.65 and 9\.19 pp on Who&When Algorithm and Handcrafted, respectively\. Code is available at[https://anonymous\.4open\.science/r/ReCast\-5FB6](https://anonymous.4open.science/r/ReCast-5FB6)\.

## 1Introduction

Large language model \(LLM\) based multi\-agent systems \(MASs\) coordinate specialized agents, external tools, and environment interactions to solve increasingly complex tasks\([Wu et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib19);[Fourney et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib10);[Song et al\., 2025](https://arxiv.org/html/2610.11334#bib.bib11)\)\. Each execution produces a trace of agent activity, such as messages and tool calls, providing a basis for understanding system behavior\. Despite their growing capabilities, MASs remain vulnerable to errors such as reasoning errors and incorrect tool use\([Liu et al\., 2026a](https://arxiv.org/html/2610.11334#bib.bib12)\)\. These errors can propagate through subsequent interactions and ultimately cause task failure, limiting the reliability of MASs in real\-world applications\.

Diagnosing why a multi\-agent execution failed is considerably harder than detecting the failure itself\. A task may admit many valid solution paths, and stochastic agent decisions can produce substantially different traces even under the same system and task\. More importantly, the root cause may initially appear plausible, since its consequences emerge only after several downstream steps\. Failure attribution therefore requires identifying the earliest decisive error from a long sequence of actions, rather than merely detecting anomalous content or locating the final visible symptom\([Zhang et al\., 2025a](https://arxiv.org/html/2610.11334#bib.bib8)\)\. Accurate step\-level attribution is essential for debugging agent behaviors, improving orchestration strategies, and constructing more reliable multi\-agent systems\.

Existing methods such as CHIEF\([Wang et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib3)\)use LLMs to analyze execution traces, but autoregressive decoding can incur substantial inference latency\. To reduce this overhead, recent methods explore failure attribution using LLM internal signals: MASPrism\([Liu et al\., 2026b](https://arxiv.org/html/2610.11334#bib.bib5)\)derives negative log\-likelihood from hidden states and combines it with attention\. StepFinder\([Zhu et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib7)\)instead learns attribution from text embeddings\. These approaches involve different choices about how to represent each step for attribution\. However, a key question remains underexplored:*what step representations are most suitable for failure attribution?*To investigate this question, we empirically assess how well text embeddings and hidden states distinguish the root\-cause step from other steps\. Figure[1](https://arxiv.org/html/2610.11334#S1.F1)reveals limited separation in these representations, motivating the construction and learning of attribution\-oriented step representations\.

Figure 1:Comparison of text embeddings and last\-layer hidden\-state representations on failed trajectories from ReCast\-2K and the Handcrafted and Algorithm subsets of Who&When\. Lower ARS/GRS and higher SI indicate better separation\. See Appendix[A](https://arxiv.org/html/2610.11334#A1)for metric definitions\.To address this gap, we proposeReCast, a step representation learning method that integrates feature selection, feature construction, and contextual learning for failure attribution\. The method consists of three modules\. First, an attribution\-aware probe selects internal features according to their ability to distinguish the root\-cause step from other steps\. Second, a fixed sparse Johnson–Lindenstrauss projection\([Li et al\., 2006](https://arxiv.org/html/2610.11334#bib.bib28)\)transforms the selected features into complementary pattern and deviation representations, capturing each step’s internal activation structure and its departure from successful execution behavior\. Third, a bidirectional encoder learns contextualized step representations from the complete trajectory\. Training jointly optimizes root and non\-root contrastive losses alongside a trajectory\-level ranking loss\. Together, these three stages determine which internal signals are retained, how they are represented, and how they are learned for attribution\.

We also introduceReCast\-2K, a training dataset containing1,0441\{,\}044successful and1,4961\{,\}496failed trajectories\. Experiments on four benchmarks—Who&When\([Zhang et al\., 2025a](https://arxiv.org/html/2610.11334#bib.bib8)\), TraceElephant\([Chen et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib13)\), AFTraj\-2K\([Zhang et al\., 2026a](https://arxiv.org/html/2610.11334#bib.bib2)\), and Who&When Pro\([Liu et al\., 2026a](https://arxiv.org/html/2610.11334#bib.bib12)\)—show that ReCast consistently achieves the best Hit@1, surpassing the strongest baseline by 5\.65 and 9\.19 pp on Who&When Algorithm and Handcrafted, respectively\. Ablation studies further support the contributions of its main components\. Our main contributions are as follows:

- •To the best of our knowledge, we are the first to systematically study the discriminative ability of step representations for multi\-agent failure attribution, using quantitative metrics to assess separation within and across trajectories\.
- •We propose ReCast, a step representation learning method that combines feature selection, feature construction, and contextual learning to produce attribution\-oriented step representations\.
- •We construct ReCast\-2K, a diverse source training dataset, and evaluate ReCast on four benchmarks, where it outperforms all compared baselines in Hit@1 accuracy\. ReCast also enables efficient attribution from precomputed features\.

## 2Related Work

Black\-box Failure Attribution Methods\.These methods attribute failures from observable trajectories through structured reasoning or learned models\. Among training\-free methods, CHIEF\([Wang et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib3)\)constructs hierarchical causal graphs from execution logs for oracle\-guided backtracking and counterfactual attribution\. FALAT\([Rafi et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib4)\)formulates attribution as hierarchical dependency\-guided search\. Distinguishing root causes from noisy or propagated symptoms remains challenging for these methods\.

Learning\-based methods train attribution models from trajectories\. AgenTracer\([Zhang et al\., 2026b](https://arxiv.org/html/2610.11334#bib.bib1)\)generates annotated failures through counterfactual replay and fault injection and trains AgenTracer\-8B with reinforcement learning\. AgentForesight\([Zhang et al\., 2026a](https://arxiv.org/html/2610.11334#bib.bib2)\)detects decisive errors from trajectory prefixes for online auditing\. StepFinder\([Zhu et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib7)\)models cross\-step dependencies in semantic sequences for root\-cause ranking\. ASCon\([Jiang et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib33)\)jointly models step context and agent interactions for failure attribution\. These methods primarily use external trajectory representations, leaving internal diagnostic signals underexplored\.

White\-box Failure Attribution Methods\.These methods exploit LLM internal signals\. MASPrism\([Liu et al\., 2026b](https://arxiv.org/html/2610.11334#bib.bib5)\)identifies symptom steps using negative log\-likelihood \(NLL\) derived from prefill\-stage hidden states, then traces potential root causes using attention weights\. OAT\([Yeh et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib6)\)learns successful hidden\-state dynamics with neural controlled differential equations and attributes failures to steps that deviate from these dynamics\. Rather than directly scoring internal signals or modeling successful dynamics, ReCast learns attribution\-oriented step representations\.

## 3PROBLEM FORMULATION

### 3\.1Definition

Background\.Consider a multi\-agent system with an agent setℳ=\{m1,…,mN\}\\mathcal\{M\}=\\\{m\_\{1\},\\ldots,m\_\{N\}\\\}interacting with an environmentℰ\\mathcal\{E\}\. Given a task queryq∈𝒬q\\in\\mathcal\{Q\}, the system produces an execution trace

τ=\(q,s0,a1,s1,a2,…,at,st,…,aT,sT\),\\tau=\(q,s\_\{0\},a\_\{1\},s\_\{1\},a\_\{2\},\\ldots,a\_\{t\},s\_\{t\},\\ldots,a\_\{T\},s\_\{T\}\),\(1\)wherests\_\{t\}is the system state after steptt,ata\_\{t\}is the candidate model action, andTTis the final step of the execution\. Here, we only consider model actions as candidates for failure attribution\. LetBBindex a batch of traces, withτi\\tau\_\{i\}denoting the trace indexed byi∈Bi\\in BandTiT\_\{i\}its terminal step\. LetZ⁡\(τ\)∈\{0,1\}Z\(\\tau\)\\in\\\{0,1\\\}denote the task\-specific success evaluator for taskqq, whereZ⁡\(τ\)=1Z\(\\tau\)=1indicates successful completion\. We then define the successful and failed trace sets as𝒯succ=\{τ∣Z⁡\(τ\)=1\}\\mathcal\{T\}^\{\\mathrm\{succ\}\}=\\\{\\tau\\mid Z\(\\tau\)=1\\\}and𝒯fail=\{τ∣Z⁡\(τ\)=0\}\\mathcal\{T\}^\{\\mathrm\{fail\}\}=\\\{\\tau\\mid Z\(\\tau\)=0\\\}\.

Objective\.Following Who&When\([Zhang et al\., 2025a](https://arxiv.org/html/2610.11334#bib.bib8)\), consider a failed traceτi\\tau\_\{i\}withZ⁡\(τi\)=0Z\(\\tau\_\{i\}\)=0\. Suppose that the actionata\_\{t\}taken by an active agent at stepttis replaced with a feasible corrective actiona~t\\tilde\{a\}\_\{t\}, while the execution history before stepttremains fixed\. The system then re\-executes the trajectory from stepttunder the corrected action, producing a counterfactual traceτ~i\\tilde\{\\tau\}\_\{i\}\. IfZ⁡\(τ~i\)=1Z\(\\tilde\{\\tau\}\_\{i\}\)=1, timestampttis defined as a*decisive error*\. We use the earliest decisive error as the unique*root\-cause*target for each failed traceτi\\tau\_\{i\}:

ti∗=min⁡\{t∈\{1,…,Ti\}:t​is a decisive error in​τi\}\.t\_\{i\}^\{\*\}=\\min\\\{t\\in\\\{1,\\ldots,T\_\{i\}\\\}:t\\text\{ is a decisive error in \}\\tau\_\{i\}\\\}\.\(2\)The root\-cause timestampti∗t\_\{i\}^\{\*\}is unique by definition\. We call all remaining candidate timestamps*non\-root*steps and write𝒩⁡\(τi\)=\{1,…,Ti\}∖\{ti∗\}\\mathcal\{N\}\(\\tau\_\{i\}\)=\\\{1,\\ldots,T\_\{i\}\\\}\\setminus\\\{t\_\{i\}^\{\*\}\\\}\. This label identifies steps other than the attribution target and does not imply that they are error\-free\. A successful trace has no root cause, so all its candidate timestamps belong to𝒩⁡\(τi\)\\mathcal\{N\}\(\\tau\_\{i\}\)\. Failure attribution ranks candidate timestamps to identify the unique targetti∗t\_\{i\}^\{\*\}\.

### 3\.2Observation

Some existing failure attribution methods use step representations derived from text embeddings or LLM hidden states\. We conduct an empirical study to evaluate how effectively these representations distinguish the root\-cause step from other steps\. Specifically, we examine Qwen3 text embeddings\([Zhang et al\., 2025b](https://arxiv.org/html/2610.11334#bib.bib29)\), following StepFinder\([Zhu et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib7)\), and last\-layer hidden states from a frozen LLM, motivated by MASPrism’s use of internal signals\([Liu et al\., 2026b](https://arxiv.org/html/2610.11334#bib.bib5)\)\.

We assess these representations from two perspectives:\(1\) Within\-trace separation\.The root\-cause step should be distinguishable from non\-root\-cause steps within the same trace\.\(2\) Cross\-trace separation\.Root\-cause representations should also be distinguishable from non\-root representations across traces\. Based on these criteria, we define three metrics\. Adjacent Root Similarity \(ARS\) and Global Root Similarity \(GRS\) measure local and overall within\-trace separation, respectively, while the cosine Silhouette Index \(SI\) measures cross\-trace separation\. Using these metrics, we compare Qwen3 text embeddings with mean\-pooled and last\-token representations from the final hidden\-state layer on the failed trajectories in our training dataset and the Handcrafted and Algorithm subsets of Who&When\. See Appendix[A](https://arxiv.org/html/2610.11334#A1)for metric and representation details\.

Figure[1](https://arxiv.org/html/2610.11334#S1.F1)shows that the evaluated representations exhibit limited separation under the proposed metrics\. High ARS and GRS indicate that the root\-cause step remains similar to its predecessor and other steps in the same trace, while low SI indicates weak separation across traces\. Although their relative performance varies, these representations exhibit limited separation between the root\-cause step and other steps under these metrics\. This observation motivates investigating whether attribution\-oriented step representation learning can improve failure attribution\.

## 4METHODOLOGY

Figure 2:Overview of ReCast, which learns attribution\-oriented step representations from frozen LLM hidden states for failure attribution\.Motivated by our empirical study, we develop ReCast to learn attribution\-oriented step representations from hidden states extracted by prefilling trajectories with a frozen LLM\. As shown in Figure[2](https://arxiv.org/html/2610.11334#S4.F2), ReCast selects attribution\-relevant layers and constructs complementary pattern and deviation features\. An encoder is then jointly trained with a scoring head under root and non\-root contrastive losses and a trajectory\-level ranking loss to learn contextualized step representations\.

### 4\.1Feature Selection and Construction

Hidden\-state extraction\.We use a frozen LLM with hidden\-state dimensiondhd\_\{h\}to extract step representations by prefilling the trajectory\. For each model\-generated actionai,ta\_\{i,t\}in trajectoryτi\\tau\_\{i\}, we obtain the layer\-ℓ\\ellstep feature by mean\-pooling itsdhd\_\{h\}\-dimensional token\-level hidden states, conditioned on the preceding execution prefix:

𝐡i,t\(ℓ\)=MeanPooltokens⁡\(ai,t\)​\(LLM\(ℓ\)​\(ai,t∣τi,<t\)\),𝐡i,t\(ℓ\)∈ℝdh\.\\mathbf\{h\}\_\{i,t\}^\{\(\\ell\)\}=\\mathrm\{MeanPool\}\_\{\\mathrm\{tokens\}\(a\_\{i,t\}\)\}\\left\(\\mathrm\{LLM\}^\{\(\\ell\)\}\(a\_\{i,t\}\\mid\\tau\_\{i,<t\}\)\\right\),\\qquad\\mathbf\{h\}\_\{i,t\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\_\{h\}\}\.\(3\)
Attribution\-aware layer selection\.Transformer layers encode information at different levels of abstraction and are not equally informative for failure attribution\. We therefore use a lightweight probe to select layers according to their ability to distinguish the root\-cause step from all other steps\. For each layer, we apply layer normalization \(LN\) to obtain normalized step features𝐮i,t\(ℓ\)\\mathbf\{u\}\_\{i,t\}^\{\(\\ell\)\}\. We then construct a difference vector for each failed trajectory by subtracting the mean non\-root feature from the root\-cause feature:

𝐮i,t\(ℓ\)=LN\(𝐡i,t\(ℓ\)\),𝐝i\(ℓ\)=𝐮i,ti∗\(ℓ\)−1\|𝒩⁡\(τi\)\|∑t∈𝒩⁡\(τi\)𝐮i,t\(ℓ\),𝐮i,t\(ℓ\),𝐝i\(ℓ\)∈ℝdh\.\\mathbf\{u\}\_\{i,t\}^\{\(\\ell\)\}=\\mathrm\{LN\}\\left\(\\mathbf\{h\}\_\{i,t\}^\{\(\\ell\)\}\\right\),\\qquad\\mathbf\{d\}\_\{i\}^\{\(\\ell\)\}=\\mathbf\{u\}\_\{i,t\_\{i\}^\{\*\}\}^\{\(\\ell\)\}\-\\frac\{1\}\{\|\\mathcal\{N\}\(\\tau\_\{i\}\)\|\}\\sum\_\{t\\in\\mathcal\{N\}\(\\tau\_\{i\}\)\}\\mathbf\{u\}\_\{i,t\}^\{\(\\ell\)\},\\qquad\\mathbf\{u\}\_\{i,t\}^\{\(\\ell\)\},\\mathbf\{d\}\_\{i\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\_\{h\}\}\.\(4\)Using the difference vectors𝐝i\(ℓ\)\\mathbf\{d\}\_\{i\}^\{\(\\ell\)\}, we assess each layer’s attribution capability through five\-fold cross\-validation on failed training trajectories\. In each round, we compute𝝁\(ℓ\),𝐯\(ℓ\)∈ℝdh\\bm\{\\mu\}^\{\(\\ell\)\},\\mathbf\{v\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\_\{h\}\}as the mean and variance vectors of𝐝i\(ℓ\)\\mathbf\{d\}\_\{i\}^\{\(\\ell\)\}over the trajectories in four folds\. These statistics define a scoring direction that emphasizes large and consistent differences:

𝐰\(ℓ\)=𝝁\(ℓ\)/\(𝐯\(ℓ\)\+η\(ℓ\)\)‖𝝁\(ℓ\)/\(𝐯\(ℓ\)\+η\(ℓ\)\)‖2,𝐰\(ℓ\)∈ℝdh\.\\mathbf\{w\}^\{\(\\ell\)\}=\\frac\{\\bm\{\\mu\}^\{\(\\ell\)\}/\(\\mathbf\{v\}^\{\(\\ell\)\}\+\\eta^\{\(\\ell\)\}\)\}\{\\left\\\|\\bm\{\\mu\}^\{\(\\ell\)\}/\(\\mathbf\{v\}^\{\(\\ell\)\}\+\\eta^\{\(\\ell\)\}\)\\right\\\|\_\{2\}\},\\qquad\\mathbf\{w\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\_\{h\}\}\.\(5\)Hereη\(ℓ\)=λridge​dh−1​∑j=1dhvj\(ℓ\)\\eta^\{\(\\ell\)\}=\\lambda\_\{\\mathrm\{ridge\}\}d\_\{h\}^\{\-1\}\\sum\_\{j=1\}^\{d\_\{h\}\}v\_\{j\}^\{\(\\ell\)\}is a scalar ridge term\. We then evaluate this direction on the remaining fold by scoring each candidate step asri,t\(ℓ\)=⟨𝐮i,t\(ℓ\),𝐰\(ℓ\)⟩r\_\{i,t\}^\{\(\\ell\)\}=\\langle\\mathbf\{u\}\_\{i,t\}^\{\(\\ell\)\},\\mathbf\{w\}^\{\(\\ell\)\}\\rangle\. We aggregate the evaluation results across all five folds and rank layers by their root\-cause identification performance\. Detailed selection criteria are provided in Appendix[B\.1](https://arxiv.org/html/2610.11334#A2.SS1)\.

Using this ranking, we selectkklayers while ensuring coverage across the model depth\. We partition all layers intokkcontiguous groups and retain the highest\-ranked layer from each group\. Letℓ1,…,ℓk\\ell\_\{1\},\\ldots,\\ell\_\{k\}denote these layers in depth order; their step features are concatenated as

𝐡i,t=\[𝐡i,t\(ℓ1\);…;𝐡i,t\(ℓk\)\],𝐡i,t∈ℝk​dh\.\\mathbf\{h\}\_\{i,t\}=\\left\[\\mathbf\{h\}\_\{i,t\}^\{\(\\ell\_\{1\}\)\};\\ldots;\\mathbf\{h\}\_\{i,t\}^\{\(\\ell\_\{k\}\)\}\\right\],\\qquad\\mathbf\{h\}\_\{i,t\}\\in\\mathbb\{R\}^\{kd\_\{h\}\}\.\(6\)
Dual\-branch projection\.To reduce the dimensionality of the concatenated hidden state, we construct two complementary branches using a fixed sparse Johnson–Lindenstrauss projection\([Li et al\., 2006](https://arxiv.org/html/2610.11334#bib.bib28)\)\. The pattern branch applies layer\-wise normalization to preserve relative activation patterns\. For the deviation branch, we introduce successful trajectories as a reference to capture how each step differs from successful execution\. The branch outputs are computed as

𝐱i,tpat=LN⁡\(𝐖JL​\[LN⁡\(𝐡i,t\(ℓj\)\)\]j=1k\)∈ℝdp,𝐱i,tdev=LN⁡\(𝐖JL​\(𝐡i,t−𝝁succ\)\)∈ℝdp\.\\mathbf\{x\}\_\{i,t\}^\{\\mathrm\{pat\}\}\\\!=\\\!\\mathrm\{LN\}\\\!\\left\(\\mathbf\{W\}\_\{\\mathrm\{JL\}\}\\left\[\\mathrm\{LN\}\\\!\\left\(\\mathbf\{h\}\_\{i,t\}^\{\(\\ell\_\{j\}\)\}\\right\)\\right\]\_\{j=1\}^\{k\}\\right\)\\in\\mathbb\{R\}^\{d\_\{p\}\},\\qquad\\mathbf\{x\}\_\{i,t\}^\{\\mathrm\{dev\}\}\\\!=\\\!\\mathrm\{LN\}\\\!\\left\(\\mathbf\{W\}\_\{\\mathrm\{JL\}\}\\left\(\\mathbf\{h\}\_\{i,t\}\-\\bm\{\\mu\}^\{\\mathrm\{succ\}\}\\right\)\\right\)\\in\\mathbb\{R\}^\{d\_\{p\}\}\.\(7\)Here,𝝁succ∈ℝk​dh\\bm\{\\mu\}^\{\\mathrm\{succ\}\}\\in\\mathbb\{R\}^\{kd\_\{h\}\}is the mean concatenated hidden state over all steps in successful training trajectories, and𝐖JL∈ℝdp×k​dh\\mathbf\{W\}\_\{\\mathrm\{JL\}\}\\in\\mathbb\{R\}^\{d\_\{p\}\\times kd\_\{h\}\}is the sparse projection matrix shared by both branches\. In the pattern branch, normalization is applied separately to each selected layer before concatenation; the deviation branch operates directly on the concatenated hidden state\. The outputs are concatenated as𝐱i,t=\[𝐱i,tpat;𝐱i,tdev\]∈ℝ2​dp\\mathbf\{x\}\_\{i,t\}=\[\\mathbf\{x\}\_\{i,t\}^\{\\mathrm\{pat\}\};\\mathbf\{x\}\_\{i,t\}^\{\\mathrm\{dev\}\}\]\\in\\mathbb\{R\}^\{2d\_\{p\}\}\. Both the projection matrix and reference mean are fixed during training\. Further details are provided in Appendix[B\.2](https://arxiv.org/html/2610.11334#A2.SS2)\.

### 4\.2Attribution\-Oriented Step Representation Learning

View augmentation\.Before encoding, we independently augment each projected trajectory𝐗i=\(𝐱i,t\)t=1Ti\\mathbf\{X\}\_\{i\}=\{\\bigl\(\\mathbf\{x\}\_\{i,t\}\\bigr\)\}\_\{t=1\}^\{T\_\{i\}\}twice to obtain two aligned views𝐗~iv\\widetilde\{\\mathbf\{X\}\}\_\{i\}^\{\\,v\}, wherev∈\{f1,f2\}v\\in\\\{\\mathrm\{f\}\_\{1\},\\mathrm\{f\}\_\{2\}\\\}\. The input and augmented views share the same dimensions:𝐗i,𝐗~iv∈ℝTi×2​dp\\mathbf\{X\}\_\{i\},\\widetilde\{\\mathbf\{X\}\}\_\{i\}^\{\\,v\}\\in\\mathbb\{R\}^\{T\_\{i\}\\times 2d\_\{p\}\}\. For timestampttand feature coordinated∈\{1,…,2​dp\}d\\in\\\{1,\\ldots,2d\_\{p\}\\\}, the augmented feature is

x~i,t,dv=ki,tv​ϕi,t,dv​xi,t,d\+ϵi,t,dv,ki,tv,ϕi,t,dv∈\{0,1\},ϵi,t,dv∈ℝ,\\widetilde\{x\}\_\{i,t,d\}^\{\\,v\}=k\_\{i,t\}^\{v\}\\phi\_\{i,t,d\}^\{v\}x\_\{i,t,d\}\+\\epsilon\_\{i,t,d\}^\{v\},\\qquad k\_\{i,t\}^\{v\},\\phi\_\{i,t,d\}^\{v\}\\in\\\{0,1\\\},\\qquad\\epsilon\_\{i,t,d\}^\{v\}\\in\\mathbb\{R\},\(8\)whereki,tvk\_\{i,t\}^\{v\}andϕi,t,dv\\phi\_\{i,t,d\}^\{v\}are the step and feature masks, respectively, andϵi,t,dv\\epsilon\_\{i,t,d\}^\{v\}is additive Gaussian noise\. These perturbations are sampled independently for the two views\. We protect the root\-cause step atti∗t\_\{i\}^\{\*\}from step masking while allowing feature\-level perturbations\. The augmentation settings are reported in Appendix[E\.2](https://arxiv.org/html/2610.11334#A5.SS2)\.

Encoding\.A bidirectional encoderfθf\_\{\\theta\}maps each augmented view todzd\_\{z\}\-dimensional step representations using the complete trajectory:

𝐙iv=fθ​\(𝐗~iv\)=\(𝐳i,tv\)t=1Ti∈ℝTi×dz\.\\mathbf\{Z\}\_\{i\}^\{v\}=f\_\{\\theta\}\(\\widetilde\{\\mathbf\{X\}\}\_\{i\}^\{\\,v\}\)=\\left\(\\mathbf\{z\}\_\{i,t\}^\{v\}\\right\)\_\{t=1\}^\{T\_\{i\}\}\\in\\mathbb\{R\}^\{T\_\{i\}\\times d\_\{z\}\}\.\(9\)A scoring headgψ:ℝdz→ℝg\_\{\\psi\}:\\mathbb\{R\}^\{d\_\{z\}\}\\to\\mathbb\{R\}produces the attribution scoreαi,tv=gψ​\(𝐳i,tv\)\\alpha\_\{i,t\}^\{v\}=g\_\{\\psi\}\(\\mathbf\{z\}\_\{i,t\}^\{v\}\)from each step representation𝐳i,tv∈ℝdz\\mathbf\{z\}\_\{i,t\}^\{v\}\\in\\mathbb\{R\}^\{d\_\{z\}\}, providing step\-level supervision during encoder training\.

Root\-cause contrastive objective\.For each root\-cause step, we use its representation in viewf1\\mathrm\{f\}\_\{1\}as the anchor and the corresponding representation in viewf2\\mathrm\{f\}\_\{2\}as the sole positive, since both represent the same underlying step under different perturbations\. All non\-root steps in viewf2\\mathrm\{f\}\_\{2\}across the batch serve as negatives, promoting separation from diverse non\-root contexts\. The positive and negative cosine similarities are

ρi,ti∗\+=cos⁡\(𝐳i,ti∗f1,𝐳i,ti∗f2\),ρi,ti∗,j,t−=cos⁡\(𝐳i,ti∗f1,𝐳j,tf2\)\.\\rho\_\{i,t\_\{i\}^\{\*\}\}^\{\+\}=\\cos\(\\mathbf\{z\}\_\{i,t\_\{i\}^\{\*\}\}^\{\\mathrm\{f\}\_\{1\}\},\\mathbf\{z\}\_\{i,t\_\{i\}^\{\*\}\}^\{\\mathrm\{f\}\_\{2\}\}\),\\qquad\\rho\_\{i,t\_\{i\}^\{\*\},j,t\}^\{\-\}=\\cos\(\\mathbf\{z\}\_\{i,t\_\{i\}^\{\*\}\}^\{\\mathrm\{f\}\_\{1\}\},\\mathbf\{z\}\_\{j,t\}^\{\\mathrm\{f\}\_\{2\}\}\)\.\(10\)LetBfail=\{i∈B:τi∈𝒯fail\}B\_\{\\mathrm\{fail\}\}=\\\{i\\in B:\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\}\\\}index the failed trajectories in the batch\. We optimize these pairs with an InfoNCE objective:

ℒroot=−1\|Bfail\|∑i∈Bfaillogexp⁡\(ρi,ti∗\+/κ\)exp⁡\(ρi,ti∗\+/κ\)\+∑j∈B∑t∈𝒩⁡\(τj\)exp⁡\(ρi,ti∗,j,t−/κ\)\.\\mathcal\{L\}\_\{\\mathrm\{root\}\}=\-\\frac\{1\}\{\|B\_\{\\mathrm\{fail\}\}\|\}\\sum\_\{i\\in B\_\{\\mathrm\{fail\}\}\}\\log\\frac\{\\exp\(\\rho\_\{i,t\_\{i\}^\{\*\}\}^\{\+\}/\\kappa\)\}\{\\exp\(\\rho\_\{i,t\_\{i\}^\{\*\}\}^\{\+\}/\\kappa\)\+\\sum\_\{j\\in B\}\\sum\_\{t\\in\\mathcal\{N\}\(\\tau\_\{j\}\)\}\\exp\(\\rho\_\{i,t\_\{i\}^\{\*\},j,t\}^\{\-\}/\\kappa\)\}\.\(11\)Hereκ\>0\\kappa\>0is the temperature parameter\. For successful trajectories,𝒩⁡\(τi\)=\{1,…,Ti\}\\mathcal\{N\}\(\\tau\_\{i\}\)=\\\{1,\\ldots,T\_\{i\}\\\}, so all their candidate steps contribute to the negative pool\.

Non\-root\-cause contrastive objective\.To learn discriminative non\-root representations, we select anchors from the steps preceding the root cause\. For each failed trajectory, let𝒫i=\{t∈𝒩⁡\(τi\):t<ti∗\}\\mathcal\{P\}\_\{i\}=\\\{t\\in\\mathcal\{N\}\(\\tau\_\{i\}\):t<t\_\{i\}^\{\*\}\\\}\. When𝒫i\\mathcal\{P\}\_\{i\}is nonempty, the anchor timestamps are selected as

α¯i,t=αi,tf1\+αi,tf22,𝒮i=\{max⁡𝒫i,arg​maxt∈𝒫i⁡α¯i,t\}\.\\bar\{\\alpha\}\_\{i,t\}=\\frac\{\\alpha\_\{i,t\}^\{\\mathrm\{f\}\_\{1\}\}\+\\alpha\_\{i,t\}^\{\\mathrm\{f\}\_\{2\}\}\}\{2\},\\qquad\\mathcal\{S\}\_\{i\}=\\left\\\{\\max\\mathcal\{P\}\_\{i\},\\ \\operatorname\*\{arg\\,max\}\_\{t\\in\\mathcal\{P\}\_\{i\}\}\\bar\{\\alpha\}\_\{i,t\}\\right\\\}\.\(12\)The latest step provides a nearby reference, while the highest\-scoring step is most likely to be mistaken for the root cause\. We retain one anchor when selections coincide and choose the earliest step when scores tie\. For eacht∈𝒮it\\in\\mathcal\{S\}\_\{i\}, viewf1\\mathrm\{f\}\_\{1\}supplies the anchor, while viewf2\\mathrm\{f\}\_\{2\}supplies the same step as the positive and that trajectory’s root\-cause step as the sole negative:

ρi,t\+=cos⁡\(𝐳i,tf1,𝐳i,tf2\),ρi,t−=cos⁡\(𝐳i,tf1,𝐳i,ti∗f2\)\.\\rho\_\{i,t\}^\{\+\}=\\cos\(\\mathbf\{z\}\_\{i,t\}^\{\\mathrm\{f\}\_\{1\}\},\\mathbf\{z\}\_\{i,t\}^\{\\mathrm\{f\}\_\{2\}\}\),\\qquad\\rho\_\{i,t\}^\{\-\}=\\cos\(\\mathbf\{z\}\_\{i,t\}^\{\\mathrm\{f\}\_\{1\}\},\\mathbf\{z\}\_\{i,t\_\{i\}^\{\*\}\}^\{\\mathrm\{f\}\_\{2\}\}\)\.\(13\)LetBpre=\{i∈Bfail:𝒫i≠∅\}B\_\{\\mathrm\{pre\}\}=\\\{i\\in B\_\{\\mathrm\{fail\}\}:\\mathcal\{P\}\_\{i\}\\neq\\emptyset\\\}index the eligible failed trajectories\. We average the InfoNCE losses first over the selected anchors within each trajectory and then over these trajectories:

ℒnon​\-​root=−1\|Bpre\|∑i∈Bpre1\|𝒮i\|∑t∈𝒮ilogexp⁡\(ρi,t\+/κ\)exp⁡\(ρi,t\+/κ\)\+exp⁡\(ρi,t−/κ\)\.\\mathcal\{L\}\_\{\\mathrm\{non\\text\{\-\}root\}\}=\-\\frac\{1\}\{\|B\_\{\\mathrm\{pre\}\}\|\}\\sum\_\{i\\in B\_\{\\mathrm\{pre\}\}\}\\frac\{1\}\{\|\\mathcal\{S\}\_\{i\}\|\}\\sum\_\{t\\in\\mathcal\{S\}\_\{i\}\}\\log\\frac\{\\exp\(\\rho\_\{i,t\}^\{\+\}/\\kappa\)\}\{\\exp\(\\rho\_\{i,t\}^\{\+\}/\\kappa\)\+\\exp\(\\rho\_\{i,t\}^\{\-\}/\\kappa\)\}\.\(14\)This reduction gives each eligible trajectory equal weight, regardless of whether it contributes one or two anchors\. IfBpreB\_\{\\mathrm\{pre\}\}is empty, the loss is zero\.

Listwise responsibility\.For eachi∈Bfaili\\in B\_\{\\mathrm\{fail\}\}and viewvv, we normalize step scores and maximize the root\-cause probability atti∗t\_\{i\}^\{\*\}\. The ranking loss averages the two view\-specific losses:

ℒrankv=−1\|Bfail\|∑i∈Bfaillogexp⁡\(αi,ti∗v\)∑t=1Tiexp⁡\(αi,tv\),ℒrank=12\(ℒrankf1\+ℒrankf2\)\.\\mathcal\{L\}\_\{\\mathrm\{rank\}\}^\{v\}=\-\\frac\{1\}\{\|B\_\{\\mathrm\{fail\}\}\|\}\\sum\_\{i\\in B\_\{\\mathrm\{fail\}\}\}\\log\\frac\{\\exp\(\\alpha\_\{i,t\_\{i\}^\{\*\}\}^\{v\}\)\}\{\\sum\_\{t=1\}^\{T\_\{i\}\}\\exp\(\\alpha\_\{i,t\}^\{v\}\)\},\\qquad\\mathcal\{L\}\_\{\\mathrm\{rank\}\}=\\frac\{1\}\{2\}\\left\(\\mathcal\{L\}\_\{\\mathrm\{rank\}\}^\{\\mathrm\{f\}\_\{1\}\}\+\\mathcal\{L\}\_\{\\mathrm\{rank\}\}^\{\\mathrm\{f\}\_\{2\}\}\\right\)\.\(15\)
Training objective\.The final encoder objective combines the trajectory\-level ranking loss, the root\-cause contrastive objective, and the non\-root\-cause contrastive objective:

ℒ=λrank​ℒrank\+λroot​ℒroot\+λnon​\-​root​ℒnon​\-​root,\\mathcal\{L\}=\\lambda\_\{\\mathrm\{rank\}\}\\mathcal\{L\}\_\{\\mathrm\{rank\}\}\+\\lambda\_\{\\mathrm\{root\}\}\\mathcal\{L\}\_\{\\mathrm\{root\}\}\+\\lambda\_\{\\mathrm\{non\\text\{\-\}root\}\}\\mathcal\{L\}\_\{\\mathrm\{non\\text\{\-\}root\}\},\(16\)whereλrank\\lambda\_\{\\mathrm\{rank\}\},λroot\\lambda\_\{\\mathrm\{root\}\}, andλnon​\-​root\\lambda\_\{\\mathrm\{non\\text\{\-\}root\}\}are the loss weights\. At inference, we encode the unaugmented trajectory and use the scoring head to identify the highest\-scoring step as the predicted root cause\. The specific optimization settings are reported in Appendix[E\.2](https://arxiv.org/html/2610.11334#A5.SS2)\.

## 5Experiments

### 5\.1Experimental Setup

Benchmarks\.We evaluate our method on four failure\-attribution benchmarks: Who&When\([Zhang et al\., 2025a](https://arxiv.org/html/2610.11334#bib.bib8)\), Who&When Pro\([Liu et al\., 2026a](https://arxiv.org/html/2610.11334#bib.bib12)\), AFTraj\-2K\([Zhang et al\., 2026a](https://arxiv.org/html/2610.11334#bib.bib2)\), and TraceElephant\([Chen et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib13)\)\. Our evaluation includes 124 algorithm\-generated \(AG\) and 58 handcrafted \(HC\) trajectories from Who&When, 1,251 test trajectories from an 8:2 task\-grouped split of the Who&When Pro text subset, 163 failed test trajectories from AFTraj\-2K, and 175 valid trajectories from the Captain\-Agent \(85\) and Magentic\-One \(90\) subsets of TraceElephant\. All TraceElephant results use these two subsets\. Further benchmark details are provided in Appendix[C](https://arxiv.org/html/2610.11334#A3)\.

Training datasets\.We use different training datasets for different target benchmarks\. Specifically, for Who&When and TraceElephant, we train on our ReCast\-2K dataset\. For AFTraj\-2K, we train on its official training split\. For Who&When Pro, we use the training partition described above\. See Appendix[D](https://arxiv.org/html/2610.11334#A4)for construction and annotation details\.

Evaluation protocol\.We report*Hit@kk*fork∈\{1,3,5\}k\\in\\\{1,3,5\\\},*Mean Reciprocal Rank \(MRR\)*,*AUROC*, and*AUPRC*\. Hit@kkmeasures whether the annotated root\-cause step appears among the top\-kkpredictions, while MRR captures its overall ranking position\. AUROC and AUPRC assess score\-level separation between root\-cause and non\-root steps, with AUPRC being particularly informative under class imbalance\. All methods use the same three seeds, and we report their mean and standard deviation\. See Appendix[E\.1](https://arxiv.org/html/2610.11334#A5.SS1)for detailed definitions\.

Baselines\.We group the baselines into three categories\.\(1\) Heuristic baselines\.First selects the first candidate step as the predicted root cause, whereas Random uniformly samples one candidate step\.\(2\) LLM\-based methods\.We evaluate the all\-at\-once and step\-by\-step protocols from Who&When\([Zhang et al\., 2025a](https://arxiv.org/html/2610.11334#bib.bib8)\)using Qwen3\.5\-27B\([Qwen Team, 2026](https://arxiv.org/html/2610.11334#bib.bib31)\), GPT\-5\.6 Luna\([OpenAI, 2026b](https://arxiv.org/html/2610.11334#bib.bib32)\), and DeepSeek V4 Flash\([Xu et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib23)\), including both open and closed models\. We also evaluate CHIEF\([Wang et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib3)\)with DeepSeek V4 Flash and include the reported results of AgenTracer\([Zhang et al\., 2026b](https://arxiv.org/html/2610.11334#bib.bib1)\)\.\(3\) Feature\-based methods\.We compare with OAT\([Yeh et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib6)\), StepFinder\([Zhu et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib7)\), MASPrism\([Liu et al\., 2026b](https://arxiv.org/html/2610.11334#bib.bib5)\), and ASCon\([Jiang et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib33)\)\. In each in\-domain setting, the supervised feature\-based baselines use the same failed\-trajectory training split as our method\. OAT instead follows its original positive\-only training protocol\. Detailed baseline settings are provided in Appendix[F](https://arxiv.org/html/2610.11334#A6)\.

Implementation details\.We extract step representations using Qwen3\.5\-27B and set the projection dimension of each branch todp=2,048d\_\{p\}=2\{,\}048\. The bidirectional Transformer encoder uses four layers withdz=128d\_\{z\}=128\. Training uses AdamW for up to 50 epochs with a batch size of 32 and a learning rate of2×10−42\\times 10^\{\-4\}\. The InfoNCE temperature isκ=0\.12\\kappa=0\.12, and the loss weights areλrank=1\.0\\lambda\_\{\\mathrm\{rank\}\}=1\.0,λroot=0\.95\\lambda\_\{\\mathrm\{root\}\}=0\.95, andλnon​\-​root=0\.25\\lambda\_\{\\mathrm\{non\\text\{\-\}root\}\}=0\.25\. Additional settings are provided in Appendix[E\.2](https://arxiv.org/html/2610.11334#A5.SS2)\.

Table 1:Hit@1 attribution accuracy \(%\) on Who&When and TraceElephant\. Results from three random seeds are summarized as mean±\\pmpopulation standard deviation\. The LLMs are Qwen3\.5\-27B \(Qwen\), DeepSeek V4 Flash \(DS\), and GPT\-5\.6 Luna \(GPT\)\. Best results are shown inbold, and second\-best results areunderlined\.†AgenTracer\-8Bis not publicly available; therefore, we cannot report multi\-seed results\.

### 5\.2Main Results

Comparison on Who&When and TraceElephant\.As shown in Table[1](https://arxiv.org/html/2610.11334#S5.T1), our method achieves the best Hit@1 accuracy across all four subsets of Who&When and TraceElephant\. On Who&When, it reaches45\.43%45\.43\\%on Algorithm and33\.33%33\.33\\%on Handcrafted, exceeding the second\-best method, CHIEF, by 5\.65 and 9\.19 percentage points, respectively\. On TraceElephant, our method reaches35\.69%35\.69\\%on Captain\-Agent and27\.04%27\.04\\%on Magentic\-One, outperforming the second\-best method, ASCon, by 4\.71 and 1\.11 percentage points, respectively\. These improvements support the effectiveness of learning attribution\-oriented step representations for failure attribution\.

Table 2:Hit@1 attribution accuracy \(%\) on the two in\-domain benchmark settings\. Results from three random seeds are summarized as mean±\\pmpopulation standard deviation\. Best results are shown inbold, and second\-best results areunderlined\.Comparison on AFTraj\-2K and Who&When Pro\.Table[2](https://arxiv.org/html/2610.11334#S5.T2)reports the Hit@1 results on AFTraj\-2K and Who&When Pro\. On AFTraj\-2K, our method achieves84\.05%84\.05\\%Hit@1, outperforming GPT\-5\.6 Luna with Step\-by\-Step and ASCon by 29\.65 and 1\.02 percentage points, respectively\. On Who&When Pro, it reaches91\.42%91\.42\\%, exceeding the same baselines by 27\.84 and 8\.71 points\. These gains show that our method remains effective when trained on other datasets, suggesting that its benefits are not limited to ReCast\-2K\.

Figure 3:Cumulative Hit@kkaccuracy on Who&When and TraceElephant\. Stacked segments denote ranks 1, 2–3, and 4–5; error bars show the population standard deviation over three seeds\.Hit@kkcomparison\.Figure[3](https://arxiv.org/html/2610.11334#S5.F3)compares cumulative Hit@1, Hit@3, and Hit@5 across methods on Who&When and TraceElephant\. We make two observations:\(1\) Strong top\-kkcoverage\.Our method achieves the highest Hit@5 accuracy across all four benchmark settings, maintaining its advantage under a more relaxed ranking criterion\.\(2\) Better top\-rank prioritization\.On Captain\-Agent, our method reaches78\.04%78\.04\\%Hit@5, exceeding MASPrism’s75\.29%75\.29\\%, while achieving a substantially higher Hit@1 of35\.69%35\.69\\%compared with18\.82%18\.82\\%for MASPrism\. These results indicate that our method ranks the root\-cause step more effectively, placing it first more often while maintaining strong top\-five coverage\.

Learned representation separation\.To assess whether our method addresses the limited separation identified in Section[3\.2](https://arxiv.org/html/2610.11334#S3.SS2), we evaluate the learned step representations using the same metrics\. Compared with the basic representations, the learned representations achieve lower ARS and GRS and higher SI on average across seeds\. On Handcrafted and Algorithm, mean SI improves from the best basic\-representation values of0\.170\.17and0\.030\.03to0\.590\.59and0\.290\.29, respectively\. These results indicate improved within\-trace and cross\-trace separation, supporting our goal of learning attribution\-oriented step representations\. See Appendix[A\.3](https://arxiv.org/html/2610.11334#A1.SS3)for the full comparison\.

## 6Analysis

Table 3:Ablation results on Who&When\. Hit@1 \(%\) over all trajectories is reported as mean±\\pmstandard deviation across three seeds\.Δ\\DeltaHit@1 reports the change from the full\-model mean in percentage points \(pp\)\.ComponentVariantHit@1Δ\\DeltaHit@1 \(pp\)NoneFull model41\.58±\\pm1\.04—Layer selectionAll 64 layers38\.83±\\pm2\.30−2\.75\-2\.75Uniform 838\.10±\\pm2\.55−3\.48\-3\.48Last 837\.55±\\pm0\.93−4\.03\-4\.03Random 837\.18±\\pm0\.69−4\.40\-4\.40Global probe top\-838\.83±\\pm1\.13−2\.75\-2\.75Projection branchesPattern only40\.11±\\pm1\.96−1\.47\-1\.47Deviation only41\.03±\\pm2\.92−0\.55\-0\.55Encoder configurationPer\-step MLP40\.48±\\pm2\.12−1\.10\-1\.10No augmentation37\.36±\\pm0\.90−4\.22\-4\.22Non\-root anchor selectionLatest only39\.93±\\pm0\.93−1\.65\-1\.65Highest\-score only39\.74±\\pm0\.52−1\.84\-1\.84ObjectiveRank only37\.73±\\pm0\.26−3\.85\-3\.85Rank \+ root40\.48±\\pm0\.69−1\.10\-1\.10Rank \+ non\-root37\.73±\\pm1\.04−3\.85\-3\.85Ablation Study\.As summarized in Table[3](https://arxiv.org/html/2610.11334#S6.T3), we conduct ablation experiments on the Who&When benchmark\. Each ablation varies one component while keeping the remaining settings fixed\. The full model achieves the highest mean Hit@1 of41\.58%41\.58\\%\. Attribution\-aware layer selection outperforms five alternatives by 2\.75–4\.40 percentage points, while the joint loss improves over ranking\-only training by 3\.85 points\. Removing augmentation reduces Hit@1 by 4\.22 points, highlighting its contribution to representation learning\. Ablations of the projection branches, encoder, and non\-root anchor selection also yield lower accuracy, supporting the overall design\.

Backbone and efficiency analyses\.In Appendix[G\.3](https://arxiv.org/html/2610.11334#A7.SS3), we compare Qwen3\.5 backbones with 0\.8B, 9B, and 27B parameters while keeping the downstream architecture fixed\. The 27B model achieves the highest Hit@1 on Who&When, AFTraj\-2K, and Who&When Pro, whereas 0\.8B and 9B perform best on Captain\-Agent and Magentic\-One, respectively\. These results indicate that larger backbones can improve attribution accuracy, but increasing model size does not consistently yield better performance\. Appendix[G\.4](https://arxiv.org/html/2610.11334#A7.SS4)further compares accuracy and processing time, demonstrating efficient attribution with ReCast when step features are precomputed\.

## 7Conclusion

This work explores step representation learning for failure attribution in LLM\-based agent systems\. To understand how well existing representations distinguish root\-cause steps from other steps, we conduct an empirical study of text embeddings and hidden states, finding limited separation under the evaluated metrics\. Motivated by this observation, ReCast transforms frozen LLM hidden states into attribution\-oriented step representations through layer selection, complementary pattern and deviation feature construction, and contextual representation learning with contrastive and ranking objectives\. We introduce ReCast\-2K to support training\. ReCast achieves the best Hit@1 across four benchmarks, while representation analysis shows improved within\-trace and cross\-trace separation\. These results highlight the value of learning step representations tailored to failure attribution\.

### AI use statement

In this work, we used generative AI tools for dataset construction and method implementation\. We have not used generative AI tools for writing mathematical proofs, and tasks involving critical ingredients for proofs are not applicable to this work\. Additionally, we used generative AI tools for drafting sections of the manuscript, language editing, and reference formatting\. We have reviewed all AI\-assisted work\. We checked the accuracy and consistency of AI\-assisted content against experimental records and the implementation\. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI\.

### Reproducibility statement

We provide our code to support reproducibility\. Appendices[C](https://arxiv.org/html/2610.11334#A3)and[D](https://arxiv.org/html/2610.11334#A4)describe the datasets, candidate\-step processing, and training\-data construction\. Method implementation and evaluation settings, including metrics, hyperparameters, and random seeds, are detailed in Appendices[B](https://arxiv.org/html/2610.11334#A2)and[E](https://arxiv.org/html/2610.11334#A5)\. Baseline configurations and LLM attribution prompts appear in Appendices[F](https://arxiv.org/html/2610.11334#A6)and[I](https://arxiv.org/html/2610.11334#A9), respectively\.

## References

- Anthropic \(2026\)AnthropicIntroducing claude opus 4\.6\.Note:Accessed: 2026\-09\-15External Links:[Link](https://www.anthropic.com/news/claude-opus-4-6)Cited by:[§D\.2](https://arxiv.org/html/2610.11334#A4.SS2.p2.1)\.
- Chenet al\.\(2026\)M\. Chen, J\. Wang, F\. Mu, Y\. Wang, Z\. Liu, H\. Feng, and Q\. WangSeeing the whole elephant: a benchmark for failure attribution in llm\-based multi\-agent systems\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 19888–19905\.Cited by:[§C\.4](https://arxiv.org/html/2610.11334#A3.SS4.p1.1),[§D\.1](https://arxiv.org/html/2610.11334#A4.SS1.p1.1),[§1](https://arxiv.org/html/2610.11334#S1.p5.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p1.1)\.
- Fourneyet al\.\(2024\)A\. Fourney, G\. Bansal, H\. Mozannar, C\. Tan, E\. Salinas, F\. Niedtner, G\. Proebsting, G\. Bassman, J\. Gerrits, J\. Alber,et al\.Magentic\-one: a generalist multi\-agent system for solving complex tasks\.arXiv preprint arXiv:2411\.04468\.Cited by:[§C\.1](https://arxiv.org/html/2610.11334#A3.SS1.p1.1),[§C\.4](https://arxiv.org/html/2610.11334#A3.SS4.p1.1),[§D\.1](https://arxiv.org/html/2610.11334#A4.SS1.p1.1),[§1](https://arxiv.org/html/2610.11334#S1.p1.1)\.
- Google \(2025\)GoogleIntroducing gemini 3\.Note:Accessed: 2026\-09\-15External Links:[Link](https://blog.google/products-and-platforms/products/gemini/gemini-3/)Cited by:[§D\.2](https://arxiv.org/html/2610.11334#A4.SS2.p2.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§C\.3](https://arxiv.org/html/2610.11334#A3.SS3.p1.1)\.
- Honget al\.\(2024\)S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, J\. Wang, C\. Zhang, Z\. Wang, S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. SchmidhuberMetaGPT: meta programming for a multi\-agent collaborative framework\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 23247–23275\.Cited by:[§C\.3](https://arxiv.org/html/2610.11334#A3.SS3.p1.1)\.
- Hurstet al\.\(2024\)A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§C\.1](https://arxiv.org/html/2610.11334#A3.SS1.p1.1),[§D\.1](https://arxiv.org/html/2610.11334#A4.SS1.p1.1)\.
- Jianget al\.\(2026\)S\. Jiang, Y\. Ran, K\. Xu, X\. Chen, Y\. Zhang, H\. Ren, R\. Tang, and T\. ZhangASCon: a direction\-aware reciprocal agent–step contextualization model for failure attribution in multi\-agent systems\.arXiv preprint arXiv:2608\.10646\.Cited by:[§F\.3](https://arxiv.org/html/2610.11334#A6.SS3.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2610.11334#S2.p2.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p4.1)\.
- Liet al\.\(2006\)P\. Li, T\. J\. Hastie, and K\. W\. ChurchVery sparse random projections\.InProceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining,pp\. 287–296\.Cited by:[§1](https://arxiv.org/html/2610.11334#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.11334#S4.SS1.p4.1)\.
- Liuet al\.\(2026a\)J\. Liu, H\. Xi, S\. Zhang, Y\. Zeng, T\. Yue, C\. Wang, J\. Kang, Q\. Wu, and H\. WangWho&When pro: can llms really attribute failures in ai agents?\.arXiv preprint arXiv:2607\.09996\.Cited by:[§C\.2](https://arxiv.org/html/2610.11334#A3.SS2.p1.1),[§1](https://arxiv.org/html/2610.11334#S1.p1.1),[§1](https://arxiv.org/html/2610.11334#S1.p5.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p1.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 21558–21572\.Cited by:[§C\.3](https://arxiv.org/html/2610.11334#A3.SS3.p1.1)\.
- Liuet al\.\(2026b\)Y\. Liu, H\. Feng, J\. Pu, and Z\. ChenMASPrism: lightweight failure attribution for multi\-agent systems using prefill\-stage signals\.arXiv preprint arXiv:2605\.07509\.Cited by:[2nd item](https://arxiv.org/html/2610.11334#A1.I1.i2.p1.1),[§F\.3](https://arxiv.org/html/2610.11334#A6.SS3.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.11334#S1.p3.1),[§2](https://arxiv.org/html/2610.11334#S2.p3.1),[§3\.2](https://arxiv.org/html/2610.11334#S3.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p4.1)\.
- Mialonet al\.\(2024\)G\. Mialon, C\. Fourrier, T\. Wolf, Y\. LeCun, and T\. ScialomGaia: a benchmark for general ai assistants\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9025–9049\.Cited by:[§C\.1](https://arxiv.org/html/2610.11334#A3.SS1.p1.1),[§C\.3](https://arxiv.org/html/2610.11334#A3.SS3.p1.1),[§C\.4](https://arxiv.org/html/2610.11334#A3.SS4.p1.1),[§D\.1](https://arxiv.org/html/2610.11334#A4.SS1.p1.1)\.
- OpenAI \(2026a\)OpenAIGPT\-5\.5 system card\.Note:Accessed: 2026\-09\-15External Links:[Link](https://openai.com/index/gpt-5-5-system-card/)Cited by:[§D\.2](https://arxiv.org/html/2610.11334#A4.SS2.p2.1)\.
- OpenAI \(2026b\)OpenAIGPT\-5\.6 system card\.Note:Published July 9, 2026External Links:[Link](https://deploymentsafety.openai.com/gpt-5-6)Cited by:[§F\.2](https://arxiv.org/html/2610.11334#A6.SS2.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p4.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§E\.2](https://arxiv.org/html/2610.11334#A5.SS2.p1.1),[§F\.2](https://arxiv.org/html/2610.11334#A6.SS2.SSS0.Px2.p1.1),[§F\.3](https://arxiv.org/html/2610.11334#A6.SS3.SSS0.Px4.p1.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p4.1)\.
- Rafiet al\.\(2026\)M\. N\. Rafi, M\. Ahasanuzzaman, D\. J\. Kim, Z\. Wang, and T\. ChenFALAT: tracing failures in llm agent trajectories via dependency\-guided search\.arXiv preprint arXiv:2606\.00765\.Cited by:[§2](https://arxiv.org/html/2610.11334#S2.p1.1)\.
- Roucheret al\.\(2025\)A\. Roucher, A\. V\. del Moral, T\. Wolf, L\. von Werra, and E\. Kaunismäkismolagents: a smol library to build great agentic systems\.Cited by:[§C\.3](https://arxiv.org/html/2610.11334#A3.SS3.p1.1)\.
- Rousseeuw \(1987\)P\. J\. RousseeuwSilhouettes: a graphical aid to the interpretation and validation of cluster analysis\.Journal of computational and applied mathematics20,pp\. 53–65\.Cited by:[§A\.1](https://arxiv.org/html/2610.11334#A1.SS1.p1.4)\.
- Songet al\.\(2025\)L\. Song, J\. Liu, J\. Zhang, S\. Zhang, A\. Luo, S\. Wang, Q\. Wu, and C\. WangAdaptive in\-conversation team building for language model agents\.arXiv preprint arXiv:2405\.19425\.Cited by:[§C\.4](https://arxiv.org/html/2610.11334#A3.SS4.p1.1),[§D\.1](https://arxiv.org/html/2610.11334#A4.SS1.p1.1),[§1](https://arxiv.org/html/2610.11334#S1.p1.1)\.
- Wanget al\.\(2024\)C\. Wang, Q\. Wu, and the AG2 CommunityAG2: open\-source agentos for ai agents\.Note:Available at https://docs\.ag2\.ai/External Links:[Link](https://github.com/ag2ai/ag2-classic)Cited by:[§C\.1](https://arxiv.org/html/2610.11334#A3.SS1.p1.1)\.
- Wanget al\.\(2026\)Y\. Wang, W\. Wu, J\. Wang, and Q\. WangFrom flat logs to causal graphs: hierarchical failure attribution for llm\-based multi\-agent systems\.arXiv preprint arXiv:2602\.23701\.Cited by:[§F\.2](https://arxiv.org/html/2610.11334#A6.SS2.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.11334#S1.p3.1),[§2](https://arxiv.org/html/2610.11334#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p4.1)\.
- Wuet al\.\(2024\)Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu,et al\.AutoGen: enabling next\-gen llm applications via multi\-agent conversation\.InFirst Conference on Language Modeling,Cited by:[§C\.3](https://arxiv.org/html/2610.11334#A3.SS3.p1.1),[§1](https://arxiv.org/html/2610.11334#S1.p1.1)\.
- Xuet al\.\(2026\)A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling,et al\.Deepseek\-v4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.Cited by:[§D\.1](https://arxiv.org/html/2610.11334#A4.SS1.p1.1),[§F\.2](https://arxiv.org/html/2610.11334#A6.SS2.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p4.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§D\.1](https://arxiv.org/html/2610.11334#A4.SS1.p1.1),[§F\.3](https://arxiv.org/html/2610.11334#A6.SS3.SSS0.Px3.p2.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[§C\.3](https://arxiv.org/html/2610.11334#A3.SS3.p1.1)\.
- Yehet al\.\(2026\)S\. Yeh, Y\. Zhu, S\. Deep, and S\. LiTracing agentic failure from the flow of success\.arXiv preprint arXiv:2607\.12747\.Cited by:[§F\.3](https://arxiv.org/html/2610.11334#A6.SS3.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.11334#S2.p3.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p4.1)\.
- Yoranet al\.\(2024\)O\. Yoran, S\. J\. Amouyal, C\. Malaviya, B\. Bogin, O\. Press, and J\. BerantAssistantbench: can web agents solve realistic and time\-consuming tasks?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8938–8968\.Cited by:[§C\.1](https://arxiv.org/html/2610.11334#A3.SS1.p1.1),[§C\.4](https://arxiv.org/html/2610.11334#A3.SS4.p1.1),[§D\.1](https://arxiv.org/html/2610.11334#A4.SS1.p1.1)\.
- Zhanget al\.\(2026a\)B\. Zhang, J\. Zhu, Z\. Shi, D\. Liu, and R\. TangAgentForesight: online auditing for early failure prediction in multi\-agent systems\.arXiv preprint arXiv:2605\.08715\.Cited by:[§C\.3](https://arxiv.org/html/2610.11334#A3.SS3.p1.1),[§1](https://arxiv.org/html/2610.11334#S1.p5.1),[§2](https://arxiv.org/html/2610.11334#S2.p2.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p1.1)\.
- Zhanget al\.\(2026b\)G\. Zhang, J\. Wang, J\. Chen, W\. Zhou, K\. Wang, and S\. YanAgentracer: who is inducing failure in the llm agentic systems?\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 11377–11399\.Cited by:[§F\.2](https://arxiv.org/html/2610.11334#A6.SS2.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2610.11334#S2.p2.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p4.1)\.
- Zhanget al\.\(2025a\)S\. Zhang, M\. Yin, J\. Zhang, J\. Liu, Z\. Han, J\. Zhang, B\. Li, C\. Wang, H\. Wang, Y\. Chen, and Q\. WuWhich agent causes task failures and when? on automated failure attribution of llm multi\-agent systems\.InProceedings of the 42nd International Conference on Machine Learning,Cited by:[§C\.1](https://arxiv.org/html/2610.11334#A3.SS1.p1.1),[§F\.2](https://arxiv.org/html/2610.11334#A6.SS2.SSS0.Px1.p1.1),[Appendix I](https://arxiv.org/html/2610.11334#A9.p1.1),[§1](https://arxiv.org/html/2610.11334#S1.p2.1),[§1](https://arxiv.org/html/2610.11334#S1.p5.1),[§3\.1](https://arxiv.org/html/2610.11334#S3.SS1.p2.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p4.1)\.
- Zhanget al\.\(2025b\)Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin,et al\.Qwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[§F\.3](https://arxiv.org/html/2610.11334#A6.SS3.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.11334#S3.SS2.p1.1)\.
- Zhuet al\.\(2026\)T\. Zhu, Y\. Wu, W\. Jin, Y\. Li, and G\. HuangStepFinder: a temporal semantic framework for failure attribution in multi\-agent systems\.InProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2,pp\. 6946–6957\.Cited by:[§F\.3](https://arxiv.org/html/2610.11334#A6.SS3.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.11334#S1.p3.1),[§2](https://arxiv.org/html/2610.11334#S2.p2.1),[§3\.2](https://arxiv.org/html/2610.11334#S3.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.11334#S5.SS1.p4.1)\.

Appendix

## Contents

## Appendix AEmpirical Similarity Study

### A\.1Evaluation Metrics

In this subsection,𝐱i,t\\mathbf\{x\}\_\{i,t\}denotes the representation being evaluated for stepttin failed traceτi\\tau\_\{i\}, including text embeddings, hidden states, projected features, or learned representations\. We use the root\-cause timestampsti∗t\_\{i\}^\{\*\}and non\-root sets𝒩⁡\(τi\)\\mathcal\{N\}\(\\tau\_\{i\}\)defined in Section[3\.1](https://arxiv.org/html/2610.11334#S3.SS1)\. Let𝒯pre=\{τi∈𝒯fail:ti∗\>1\}\\mathcal\{T\}\_\{\\mathrm\{pre\}\}=\\\{\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\}:t\_\{i\}^\{\*\}\>1\\\}contain the traces whose root cause has a preceding step\. We define Adjacent Root Similarity \(ARS\) as

ARS=1\|𝒯pre\|​∑τi∈𝒯precos⁡\(𝐱i,ti∗,𝐱i,ti∗−1\)\.\\mathrm\{ARS\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{\\mathrm\{pre\}\}\|\}\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{T\}\_\{\\mathrm\{pre\}\}\}\\cos\\\!\\left\(\\mathbf\{x\}\_\{i,t\_\{i\}^\{\*\}\},\\mathbf\{x\}\_\{i,t\_\{i\}^\{\*\}\-1\}\\right\)\.\(17\)Global Root Similarity \(GRS\) is defined as

GRS=1\|𝒯fail\|​∑τi∈𝒯fail1\|𝒩⁡\(τi\)\|​∑t∈𝒩⁡\(τi\)cos⁡\(𝐱i,ti∗,𝐱i,t\)\.\\mathrm\{GRS\}=\\frac\{1\}\{\|\\mathcal\{T\}^\{\\mathrm\{fail\}\}\|\}\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\}\}\\frac\{1\}\{\|\\mathcal\{N\}\(\\tau\_\{i\}\)\|\}\\sum\_\{t\\in\\mathcal\{N\}\(\\tau\_\{i\}\)\}\\cos\\\!\\left\(\\mathbf\{x\}\_\{i,t\_\{i\}^\{\*\}\},\\mathbf\{x\}\_\{i,t\}\\right\)\.\(18\)For the cosine Silhouette Index \(SI\)\([Rousseeuw, 1987](https://arxiv.org/html/2610.11334#bib.bib30)\), we pool the root\-cause representations into𝒳root=\{𝐱i,ti∗:τi∈𝒯fail\}\\mathcal\{X\}\_\{\\mathrm\{root\}\}=\\\{\\mathbf\{x\}\_\{i,t\_\{i\}^\{\*\}\}:\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\}\\\}and the non\-root representations into𝒳non=\{𝐱i,t:τi∈𝒯fail,t∈𝒩\(τi\)\}\\mathcal\{X\}\_\{\\mathrm\{non\}\}=\\\{\\mathbf\{x\}\_\{i,t\}:\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\},\\,t\\in\\mathcal\{N\}\(\\tau\_\{i\}\)\\\}\. For each representation𝐱\\mathbf\{x\}, we define its same\-class and opposite\-class sets as

\(𝒮⁡\(𝐱\),𝒪⁡\(𝐱\)\)=\{\(𝒳root,𝒳non\),𝐱∈𝒳root,\(𝒳non,𝒳root\),𝐱∈𝒳non\.\\bigl\(\\mathcal\{S\}\(\\mathbf\{x\}\),\\mathcal\{O\}\(\\mathbf\{x\}\)\\bigr\)=\\begin\{cases\}\\bigl\(\\mathcal\{X\}\_\{\\mathrm\{root\}\},\\mathcal\{X\}\_\{\\mathrm\{non\}\}\\bigr\),&\\mathbf\{x\}\\in\\mathcal\{X\}\_\{\\mathrm\{root\}\},\\\\ \\bigl\(\\mathcal\{X\}\_\{\\mathrm\{non\}\},\\mathcal\{X\}\_\{\\mathrm\{root\}\}\\bigr\),&\\mathbf\{x\}\\in\\mathcal\{X\}\_\{\\mathrm\{non\}\}\.\\end\{cases\}Using cosine distancedcos​\(𝐱,𝐳\)=1−cos⁡\(𝐱,𝐳\)d\_\{\\cos\}\(\\mathbf\{x\},\\mathbf\{z\}\)=1\-\\cos\(\\mathbf\{x\},\\mathbf\{z\}\), we compute

a⁡\(𝐱\)\\displaystyle a\(\\mathbf\{x\}\)=1\|𝒮⁡\(𝐱\)\|−1​∑𝐳∈𝒮⁡\(𝐱\)∖\{𝐱\}dcos​\(𝐱,𝐳\),\\displaystyle=\\frac\{1\}\{\|\\mathcal\{S\}\(\\mathbf\{x\}\)\|\-1\}\\sum\_\{\\mathbf\{z\}\\in\\mathcal\{S\}\(\\mathbf\{x\}\)\\setminus\\\{\\mathbf\{x\}\\\}\}d\_\{\\cos\}\\\!\\left\(\\mathbf\{x\},\\mathbf\{z\}\\right\),\(19\)b⁡\(𝐱\)\\displaystyle b\(\\mathbf\{x\}\)=1\|𝒪⁡\(𝐱\)\|​∑𝐳∈𝒪⁡\(𝐱\)dcos​\(𝐱,𝐳\)\.\\displaystyle=\\frac\{1\}\{\|\\mathcal\{O\}\(\\mathbf\{x\}\)\|\}\\sum\_\{\\mathbf\{z\}\\in\\mathcal\{O\}\(\\mathbf\{x\}\)\}d\_\{\\cos\}\\\!\\left\(\\mathbf\{x\},\\mathbf\{z\}\\right\)\.The resulting SI is

SI=1\|𝒳root\|\+\|𝒳non\|​∑𝐱∈𝒳root∪𝒳nonb⁡\(𝐱\)−a⁡\(𝐱\)max⁡\(a⁡\(𝐱\),b⁡\(𝐱\)\)\.\\mathrm\{SI\}=\\frac\{1\}\{\|\\mathcal\{X\}\_\{\\mathrm\{root\}\}\|\+\|\\mathcal\{X\}\_\{\\mathrm\{non\}\}\|\}\\sum\_\{\\mathbf\{x\}\\in\\mathcal\{X\}\_\{\\mathrm\{root\}\}\\cup\\mathcal\{X\}\_\{\\mathrm\{non\}\}\}\\frac\{b\(\\mathbf\{x\}\)\-a\(\\mathbf\{x\}\)\}\{\\max\\\!\\left\(a\(\\mathbf\{x\}\),b\(\\mathbf\{x\}\)\\right\)\}\.\(20\)Together, these metrics assess two aspects of representation quality: within\-trace and cross\-trace separation\. Within a trace, ARS measures similarity between the root\-cause step and its immediately preceding step, while GRS measures its average similarity to all other steps; lower values indicate stronger separation\. Across traces, SI measures the separation between root\-cause and non\-root representations, with higher values indicating better separation\.

### A\.2Representations

We compare the following three representations of candidate model\-action steps:

- •Text embedding\.Following StepFinder, we encode the serialized text of each candidate model\-action step with Qwen3\-Embedding\-0\.6B, producing a1,0241\{,\}024\-dimensional vector\.
- •Mean last\-layer hidden state\.Following MASPrism\([Liu et al\., 2026b](https://arxiv.org/html/2610.11334#bib.bib5)\)in using LLM internal signals, we average the Qwen3\.5\-27B last\-layer hidden states over action tokens at each candidate timestamp, producing a5,1205\{,\}120\-dimensional vector\.
- •Last\-token last\-layer hidden state\.We use the Qwen3\.5\-27B last\-layer hidden state of the final token at each candidate timestamp, producing a5,1205\{,\}120\-dimensional vector\.

### A\.3Complete Results

Table 4:Separation of step representations measured by ARS, GRS, and SI\. ReCast results show mean±\\pmpopulation standard deviation across seeds66,2020, and4242; other representations are fixed\. Best results are bolded for each dataset and metric, based on mean values for ReCast\.We evaluate the three basic representations alongside our encoder inputs and learned step representations on the same failed trajectories using the metrics defined above\. In Table[4](https://arxiv.org/html/2610.11334#A1.T4),*Projected features*denotes𝐱i,t∈ℝ2​dp\\mathbf\{x\}\_\{i,t\}\\in\\mathbb\{R\}^\{2d\_\{p\}\}, the concatenated pattern and deviation features from Eq\.[7](https://arxiv.org/html/2610.11334#S4.E7)before augmentation and encoding, with2​dp=4,0962d\_\{p\}=4\{,\}096\.*ReCast*denotes the learned contextualized step representations before the scoring head, withdz=128d\_\{z\}=128\.

Among the basic representations, text embeddings perform best in most comparisons, although their advantage varies by dataset and metric\. All three representations exhibit substantial within\-trace similarity, reflected in high ARS and GRS, and limited cross\-trace separation, reflected in low SI\. This pattern is consistent with the limited separation observed under these metrics in Section[3\.2](https://arxiv.org/html/2610.11334#S3.SS2)\.

In contrast, ReCast achieves the lowest mean ARS and GRS and the highest mean SI on every dataset\. Its mean SI reaches0\.59130\.5913on Handcrafted and0\.28730\.2873on Algorithm, exceeding the best basic\-representation values of0\.17350\.1735and0\.03090\.0309, respectively\. The gains over projected features further show that representation learning improves separation beyond feature construction\. These improvements within and across trajectories provide quantitative support for learning attribution\-oriented step representations\.

## Appendix BMethod Implementation Details

### B\.1Attribution\-Aware Layer Selection

Section[4\.1](https://arxiv.org/html/2610.11334#S4.SS1)presents the overall attribution\-aware layer\-selection procedure\. Here, we detail fold construction, probe statistics, and layer ranking based on held\-out evaluation results\. We partition the failed training trajectories𝒯fail\\mathcal\{T\}^\{\\mathrm\{fail\}\}into five folds using a deterministic hash of each trace identifier\. For foldff, letℐf\\mathcal\{I\}^\{f\}denote the held\-out trajectories andℐ−f=𝒯fail∖ℐf\\mathcal\{I\}^\{\-f\}=\\mathcal\{T\}^\{\\mathrm\{fail\}\}\\setminus\\mathcal\{I\}^\{f\}the trajectories in the other four folds\. The probe is fitted onℐ−f\\mathcal\{I\}^\{\-f\}and evaluated onℐf\\mathcal\{I\}^\{f\}\. The mean and sample\-variance vectors used in Eq\.[5](https://arxiv.org/html/2610.11334#S4.E5)are

𝝁\(ℓ\)=1\|ℐ−f\|​∑τi∈ℐ−f𝐝i\(ℓ\),𝐯\(ℓ\)=1\|ℐ−f\|−1​∑τi∈ℐ−f\(𝐝i\(ℓ\)−𝝁\(ℓ\)\)⊙\(𝐝i\(ℓ\)−𝝁\(ℓ\)\)\.\\bm\{\\mu\}^\{\(\\ell\)\}=\\frac\{1\}\{\|\\mathcal\{I\}^\{\-f\}\|\}\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{I\}^\{\-f\}\}\\mathbf\{d\}\_\{i\}^\{\(\\ell\)\},\\qquad\\mathbf\{v\}^\{\(\\ell\)\}=\\frac\{1\}\{\|\\mathcal\{I\}^\{\-f\}\|\-1\}\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{I\}^\{\-f\}\}\\left\(\\mathbf\{d\}\_\{i\}^\{\(\\ell\)\}\-\\bm\{\\mu\}^\{\(\\ell\)\}\\right\)\\odot\\left\(\\mathbf\{d\}\_\{i\}^\{\(\\ell\)\}\-\\bm\{\\mu\}^\{\(\\ell\)\}\\right\)\.\(21\)Using the scoreri,t\(ℓ\)r\_\{i,t\}^\{\(\\ell\)\}defined in the main text, the within\-trajectory rank of the labeled root cause is

ranki\(ℓ\)=1\+∑t∈𝒩⁡\(τi\)𝟏\[ri,t\(ℓ\)\>ri,ti∗\(ℓ\)\],∀τi∈ℐf\.\\operatorname\{rank\}\_\{i\}^\{\(\\ell\)\}=1\+\\sum\_\{t\\in\\mathcal\{N\}\(\\tau\_\{i\}\)\}\\mathbf\{1\}\\left\[r\_\{i,t\}^\{\(\\ell\)\}\>r\_\{i,t\_\{i\}^\{\*\}\}^\{\(\\ell\)\}\\right\],\\qquad\\forall\\,\\tau\_\{i\}\\in\\mathcal\{I\}^\{f\}\.\(22\)Thus,ranki\(ℓ\)=1\\operatorname\{rank\}\_\{i\}^\{\(\\ell\)\}=1when no non\-root step has a strictly larger score than the root cause, with tied scores leaving the rank unchanged\. We pool the out\-of\-fold results to evaluate each layer by Hit@1, MRR, and mean root margin:

Hit​@​1\(ℓ\)\\displaystyle\\mathrm\{Hit@1\}^\{\(\\ell\)\}=1∑f=15\|ℐf\|∑f=15∑τi∈ℐf𝟏\[ranki\(ℓ\)=1\],\\displaystyle=\\frac\{1\}\{\\sum\_\{f=1\}^\{5\}\|\\mathcal\{I\}^\{f\}\|\}\\sum\_\{f=1\}^\{5\}\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{I\}^\{f\}\}\\mathbf\{1\}\\left\[\\operatorname\{rank\}\_\{i\}^\{\(\\ell\)\}=1\\right\],\(23\)MRR\(ℓ\)\\displaystyle\\mathrm\{MRR\}^\{\(\\ell\)\}=1∑f=15\|ℐf\|​∑f=15∑τi∈ℐf1ranki\(ℓ\),\\displaystyle=\\frac\{1\}\{\\sum\_\{f=1\}^\{5\}\|\\mathcal\{I\}^\{f\}\|\}\\sum\_\{f=1\}^\{5\}\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{I\}^\{f\}\}\\frac\{1\}\{\\operatorname\{rank\}\_\{i\}^\{\(\\ell\)\}\},\(24\)Margin\(ℓ\)\\displaystyle\\mathrm\{Margin\}^\{\(\\ell\)\}=1∑f=15\|ℐf\|​∑f=15∑τi∈ℐf\(ri,ti∗\(ℓ\)−maxt∈𝒩⁡\(τi\)⁡ri,t\(ℓ\)\)\.\\displaystyle=\\frac\{1\}\{\\sum\_\{f=1\}^\{5\}\|\\mathcal\{I\}^\{f\}\|\}\\sum\_\{f=1\}^\{5\}\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{I\}^\{f\}\}\\left\(r\_\{i,t\_\{i\}^\{\*\}\}^\{\(\\ell\)\}\-\\max\_\{t\\in\\mathcal\{N\}\(\\tau\_\{i\}\)\}r\_\{i,t\}^\{\(\\ell\)\}\\right\)\.\(25\)To ensure coverage across model depth, we partition theLLlayers intokkcontiguous groups:

𝒢b=\{⌊\(b−1\)​Lk⌋\+1,…,⌊b​Lk⌋\},b=1,…,k\.\\mathcal\{G\}\_\{b\}=\\left\\\{\\left\\lfloor\\frac\{\(b\-1\)L\}\{k\}\\right\\rfloor\+1,\\ldots,\\left\\lfloor\\frac\{bL\}\{k\}\\right\\rfloor\\right\\\},\\quad b=1,\\ldots,k\.\(26\)Within each group, we rank layers by Hit@1, breaking ties by MRR and then mean root margin, and select the highest\-ranked layer:

ℓb=arg​maxℓ∈𝒢b⁡\(Hit​@​1\(ℓ\),MRR\(ℓ\),Margin\(ℓ\)\)\.\\ell\_\{b\}=\\operatorname\*\{arg\\,max\}\_\{\\ell\\in\\mathcal\{G\}\_\{b\}\}\\left\(\\mathrm\{Hit@1\}^\{\(\\ell\)\},\\mathrm\{MRR\}^\{\(\\ell\)\},\\mathrm\{Margin\}^\{\(\\ell\)\}\\right\)\.\(27\)The resulting layers and probe settings are reported in Appendix[E\.2](https://arxiv.org/html/2610.11334#A5.SS2)\.

### B\.2Dual\-branch Projection

Equation[7](https://arxiv.org/html/2610.11334#S4.E7)defines the pattern and deviation branches\. Here we provide the implementation details not specified in the main text: how the sparse projection is sampled and the exact normalization and reference statistics\. We sample a sparse projection over allLLlayers and retain the columns corresponding to the selected layers, ensuring that layer\-selection variants share the same projection coordinates\. The random entries follow

\(𝐖JL\)a,b∼\{1δ​dp,with probability​δ2,0,with probability​1−δ,−1δ​dp,with probability​δ2\.δ=1L​dh\.\\left\(\\mathbf\{W\}\_\{\\mathrm\{JL\}\}\\right\)\_\{a,b\}\\sim\\begin\{cases\}\\dfrac\{1\}\{\\sqrt\{\\delta d\_\{p\}\}\},&\\text\{with probability \}\\dfrac\{\\delta\}\{2\},\\\\\[5\.0pt\] 0,&\\text\{with probability \}1\-\\delta,\\\\\[5\.0pt\] \-\\dfrac\{1\}\{\\sqrt\{\\delta d\_\{p\}\}\},&\\text\{with probability \}\\dfrac\{\\delta\}\{2\}\.\\end\{cases\}\\qquad\\delta=\\frac\{1\}\{\\sqrt\{Ld\_\{h\}\}\}\.\(28\)Here,𝐖JL∈ℝdp×k​dh\\mathbf\{W\}\_\{\\mathrm\{JL\}\}\\in\\mathbb\{R\}^\{d\_\{p\}\\times kd\_\{h\}\}, whereLLis the total number of backbone layers \(L=64L=64for Qwen3\.5\-27B\),dhd\_\{h\}is the feature dimension of one hidden layer,dpd\_\{p\}is the projection dimension, andδ\\deltais the projection density, i\.e\., the probability that an entry of𝐖JL\\mathbf\{W\}\_\{\\mathrm\{JL\}\}is nonzero\. The full\-layer matrix has shapedp×L​dhd\_\{p\}\\times Ld\_\{h\}and is generated with random state 42\. Its random entries are sampled independently of both the attribution labels and the hidden\-state covariance, and the retained matrix is fixed and shared by both branches and across training seeds\.

All instances ofLN\\mathrm\{LN\}in Eq\.[7](https://arxiv.org/html/2610.11334#S4.E7)use parameter\-free normalization\. For a vector𝐯∈ℝn\\mathbf\{v\}\\in\\mathbb\{R\}^\{n\}, wheren=dhn=d\_\{h\}for a layer feature andn=dpn=d\_\{p\}for a projected branch output, we compute

LN⁡\(𝐯\)=𝐯−𝔼⁡\[𝐯\]Var⁡\(𝐯\)\+ϵ\.\\mathrm\{LN\}\(\\mathbf\{v\}\)=\\frac\{\\mathbf\{v\}\-\\mathbb\{E\}\[\\mathbf\{v\}\]\}\{\\sqrt\{\\operatorname\{Var\}\(\\mathbf\{v\}\)\+\\epsilon\}\}\.\(29\)Here, the expectation and variance are computed independently for each step over itsnnfeature coordinates, andϵ\\epsilonis a numerical\-stability constant\. The normalization has no learned scale or bias\.

The successful\-reference mean𝝁succ∈ℝk​dh\\bm\{\\mu\}^\{\\mathrm\{succ\}\}\\in\\mathbb\{R\}^\{kd\_\{h\}\}used in Eq\.[7](https://arxiv.org/html/2610.11334#S4.E7)is computed over all candidate steps in the successful training trajectories:

𝝁succ=∑τi∈𝒯succ∑t=1Ti𝐡i,t∑τi∈𝒯succTi\.\\bm\{\\mu\}^\{\\mathrm\{succ\}\}=\\frac\{\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{succ\}\}\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\mathbf\{h\}\_\{i,t\}\}\{\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{succ\}\}\}T\_\{i\}\}\.\(30\)Using the fixed projection matrix𝐖JL\\mathbf\{W\}\_\{\\mathrm\{JL\}\}and successful\-reference mean𝝁succ\\bm\{\\mu\}^\{\\mathrm\{succ\}\}, Eq\.[7](https://arxiv.org/html/2610.11334#S4.E7)produces the pattern and deviation features𝐱i,tpat\\mathbf\{x\}\_\{i,t\}^\{\\mathrm\{pat\}\}and𝐱i,tdev\\mathbf\{x\}\_\{i,t\}^\{\\mathrm\{dev\}\}, which are concatenated directly without additional weighting or branch\-specific scoring heads\. The remaining dimensions and hyperparameters are provided in Appendix[E\.2](https://arxiv.org/html/2610.11334#A5.SS2)\.

## Appendix CBenchmark Details

### C\.1Who&When

Who&When\([Zhang et al\., 2025a](https://arxiv.org/html/2610.11334#bib.bib8)\)contains 184 failed trajectories generated under two MAS construction settings and evaluated on two task benchmarks\. The algorithm\-generated \(AG\) setting uses CaptainAgent from the AG2 library\([Wang et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib9)\)to construct a task\-specific MAS for each query, whereas the handcrafted \(HC\) setting uses the fixed, manually designed Magentic\-One system\([Fourney et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib10)\); both settings use GPT\-4o\([Hurst et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib16)\)as the base language model\. The underlying tasks are drawn from GAIA\([Mialon et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib14)\), which evaluates agents on real\-world questions requiring reasoning and tool use, and AssistantBench\([Yoran et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib15)\), which focuses on realistic and time\-consuming web tasks\.

The AG subset contains 126 failed trajectories, including 98 generated on GAIA and 28 on AssistantBench\. The HC subset contains 58 failed trajectories produced by Magentic\-One, including 30 trajectories from GAIA and 28 from AssistantBench\. Each trajectory is annotated with the failure\-responsible agent, the decisive error step, and a natural\-language explanation\. Because our candidate set is restricted to model\-generated actions, we exclude two AG trajectories whose annotated decisive errors correspond to outputs from theComputer\_terminaltool rather than model\-action steps, leaving 124 AG trajectories and all 58 HC trajectories for evaluation\.

### C\.2Who&When Pro

Who&When Pro\([Liu et al\., 2026a](https://arxiv.org/html/2610.11334#bib.bib12)\)is a large\-scale failure\-attribution benchmark containing12,32612\{,\}326failed trajectories drawn from 26 source benchmarks, nine task categories, and 15 agent frameworks\. It covers text, image, and video modalities as well as both single\-agent and multi\-agent execution settings\. The benchmark defines 18 error modes spanning perception, reasoning, planning, action, verification, and coordination failures, and provides gold\-standard labels for the responsible agent, the decisive error step, and the corresponding error mode\.

Rather than inferring failure causes retrospectively, Who&When Pro constructs each failed trajectory by injecting a controlled error into a successful execution and retaining only interventions that change the final outcome from success to failure, providing causally grounded labels for the responsible agent and decisive step\. Our experiments use all6,2576\{,\}257text trajectories, which span 16 source benchmarks, eight agent frameworks, and 14 represented error modes; Table[5](https://arxiv.org/html/2610.11334#A3.T5)summarizes their category\-level composition and trajectory lengths\.

To keep trajectories from the same task together, we group them by benchmark, task query, and reference answer and assign entire groups to an 8:2 training–test split using seed 42\. The partitions contain5,0065\{,\}006and1,2511\{,\}251trajectories, respectively, with no shared task groups or normalized queries\. Since the training partition contains only failed trajectories, we use1,0441\{,\}044successful trajectories from ReCast\-2K to compute the deviation reference mean and provide contrastive negatives\.

Table 5:Composition of the Who&When Pro text\-modality subset used in our experiments\.Step counts refer to aligned model\-action steps, and token counts are computed over the corresponding inputs\.
### C\.3AFTraj\-2K

AFTraj\-2K\([Zhang et al\., 2026a](https://arxiv.org/html/2610.11334#bib.bib2)\)contains2,2762\{,\}276multi\-agent trajectories across the Coding, Math, and Agentic domains, including1,1621\{,\}162successful trajectories and1,1141\{,\}114failed trajectories with decisive\-error annotations\. Each domain pairs representative task benchmarks with suitable multi\-agent frameworks\. The Coding domain uses HumanEval\+ and MBPP\+\([Liu et al\., 2023](https://arxiv.org/html/2610.11334#bib.bib17)\), with trajectories generated by AutoGen\([Wu et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib19)\)and MetaGPT\([Hong et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib20)\)\. The Math domain uses MATH\-500\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.11334#bib.bib18)\)with AutoGen, while the Agentic domain uses GAIA\([Mialon et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib14)\)and HotpotQA\([Yang et al\., 2018](https://arxiv.org/html/2610.11334#bib.bib21)\)with Smolagents\([Roucher et al\., 2025](https://arxiv.org/html/2610.11334#bib.bib22)\)\. Across the three domains, trajectories contain 11\.5 turns on average\.

Within AFTraj\-2K, successful trajectories are selected through strict checks for outcome correctness, execution integrity, and step coherence\. Failed trajectories come from two sources: controlled error injections and naturally failed runs\. For injected failures, the modified agent and step serve as direct labels; for natural failures, the earliest decisive error is identified through multi\-LLM proposal and verification\. The dataset is divided into1,9441\{,\}944training and 332 test trajectories, with each successful trajectory and its injected variants kept in the same partition\. For the in\-domain setting, we train on the AFTraj\-2K training split and evaluate on its test split\.

### C\.4TraceElephant

TraceElephant\([Chen et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib13)\)is a failure\-attribution benchmark with fully observable execution traces and reproducible environments\. For every agent action, it records the input context, model output, and associated tool interactions, together with trace\-level information such as the task instruction, agent configuration, and system architecture\. Our evaluation uses two multi\-agent system subsets\. Captain\-Agent\([Song et al\., 2025](https://arxiv.org/html/2610.11334#bib.bib11)\)dynamically assembles task\-specific teams and contributes 85 failed trajectories, including 73 from GAIA\([Mialon et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib14)\)and 12 from AssistantBench\([Yoran et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib15)\)\. Magentic\-One\([Fourney et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib10)\)uses a fixed multi\-agent architecture; after excluding one AssistantBench trajectory lacking a root\-cause label, its evaluation subset contains 90 failed trajectories, including 74 from GAIA and 16 from AssistantBench\. Together, these two subsets contain 175 failed trajectories, with an average of 25\.1 and a maximum of 59 candidate steps per trajectory\. None of these trajectories exactly overlaps with the failed trajectories in our training corpus\.

### C\.5Data Processing

To standardize the heterogeneous trace formats across benchmarks, we use the recorded step roles to retain model\-generated actions as attribution candidates and exclude environment observations, tool\-result messages, and other non\-model events\. Among the failed AFTraj\-2K test trajectories, this processing reduces the pooled candidate population from 1,711 to 1,548 steps while retaining all 163 trajectories and their annotated root\-cause steps\. All compared feature\-based methods use the same processed candidate set\.

## Appendix DTraining Dataset Construction

Table 6:Composition of our source training dataset\.Initial runs are the aligned executions before filtering\. The final total consists of retained successful trajectories and failed trajectories with root\-cause annotations\. Additional generation runs are grouped under Others\.### D\.1Task Sampling and Trajectory Generation

We construct ReCast\-2K using the TraceElephant data\-collection pipeline\([Chen et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib13)\)\. Captain\-Agent\([Song et al\., 2025](https://arxiv.org/html/2610.11334#bib.bib11)\)trajectories are collected on GAIA\([Mialon et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib14)\)and AssistantBench\([Yoran et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib15)\), and Magentic\-One\([Fourney et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib10)\)trajectories on GAIA\. Tasks are drawn from the validation splits\. Executions primarily use DeepSeek V4 Flash\([Xu et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib23)\), GPT\-4o\-0806\([Hurst et al\., 2024](https://arxiv.org/html/2610.11334#bib.bib16)\), and Qwen3\-235B\([Yang et al\., 2025](https://arxiv.org/html/2610.11334#bib.bib24)\)\. The collection contains3,2223\{,\}222trajectories before filtering, as summarized in Table[6](https://arxiv.org/html/2610.11334#A4.T6)\.

#### Captain\-Agent generation\.

Captain\-Agent dynamically constructs a task\-specific team for every execution\. Its builder is instructed to include capabilities for information retrieval, Python\-based file processing and calculation, and answer verification, and selects concrete experts from the Captain\-Agent agent library\. Each run is allowed at most 30 group\-chat rounds\. Generation uses temperature 0\.1, top\-pp0\.95, and a maximum of 4,096 output tokens per model call; failed or empty executions are retried up to five times\.

#### Magentic\-One generation\.

Magentic\-One uses its fixed centrally orchestrated team: an Orchestrator maintains the plan and delegates to WebSurfer, FileSurfer, Coder, and ComputerTerminal\. WebSurfer operates a browser, FileSurfer reads local artifacts, Coder proposes programs, and ComputerTerminal executes them in an isolated run directory\. As with Captain\-Agent, attached GAIA files are exposed through their resolved local paths, and each execution is capped at 30 rounds\.

### D\.2Root\-Cause Annotation

Before annotation, we filter incomplete or malformed traces, normalize agent identities and metadata, and extract the final answer from each run\. We determine execution outcomes by matching the extracted answers against benchmark references using both exact and semantic criteria\. Failure preprocessing removes traces without a usable answer or with at most one step and applies answer\-based deduplication within each task and source configuration\. Table[6](https://arxiv.org/html/2610.11334#A4.T6)reports the final retained trajectories\.

For each retained failed trajectory, we apply an all\-at\-once annotation protocol using three independent LLM judges: GPT\-5\.5\([OpenAI, 2026a](https://arxiv.org/html/2610.11334#bib.bib25)\), Gemini 3 Pro Preview\([Google, 2025](https://arxiv.org/html/2610.11334#bib.bib27)\), and Claude Opus 4\.6\([Anthropic, 2026](https://arxiv.org/html/2610.11334#bib.bib26)\)\. Each judge predicts a root\-cause step, and we aggregate valid returned predictions by step\-level voting\. In the final corpus, 1,112 trajectories carry consensus labels: 766 have unanimous agreement among valid predictions, and 346 have a two\-of\-three majority\. The remaining 384 trajectories undergo manual review to determine their final root\-cause labels\. Together, these groups account for all 1,496 failed trajectories\. The responsible agent is the agent acting at the root\-cause step\.

### D\.3Final Dataset Statistics

Table[7](https://arxiv.org/html/2610.11334#A4.T7)summarizes the final dataset by generation model and trajectory type\. It contains1,0441\{,\}044successful and1,4961\{,\}496failed trajectories with substantial variation in step and token counts\. Two patterns emerge from the collected data:\(1\) Failed trajectories are generally longer than successful ones\.They average 25\.9 versus 17\.2 steps and8,346\.68\{,\}346\.6versus5,605\.55\{,\}605\.5tokens\.\(2\) Trajectory characteristics vary noticeably across generation models\.GPT\-4o\-0806 produces failed trajectories with the highest average number of steps \(32\.0\), whereas DeepSeek V4 Flash yields the highest average and maximum token counts among failed trajectories\. Together, these variations provide broad coverage of trajectory lengths and execution patterns in the training data\.

Table 7:Statistics of ReCast\-2K\.Steps count recorded model responses, while token counts cover the question prefix and serialized trajectory using the Qwen3\.5\-27B tokenizer\. Overall averages are computed over all trajectories\.ModelType\# Trajectories\# Steps\# TokensAvg\.MaxAvg\.MaxDeepSeek V4 FlashSuccessful45415\.7617,005\.559,153Failed50021\.65510,603\.574,773GPT\-4o\-0806Successful22518\.5604,460\.022,757Failed51532\.0617,996\.129,094Qwen3\-235BSuccessful35918\.5624,568\.521,378Failed48124\.0626,375\.823,621OthersSuccessful64\.874,677\.56,729OverallSuccessful1,04417\.2625,605\.559,153Failed1,49625\.9628,346\.674,773

## Appendix EEvaluation Details and Hyperparameters

### E\.1Evaluation Metrics

Using the root\-cause timestampti∗t\_\{i\}^\{\*\}defined in Section[3\.1](https://arxiv.org/html/2610.11334#S3.SS1), letranki⁡\(ti∗\)\\operatorname\{rank\}\_\{i\}\(t\_\{i\}^\{\*\}\)denote the predicted rank of the root cause in failed trajectoryτi\\tau\_\{i\}\. Hit@kkand MRR over the evaluated failed trajectories are computed as

Hit​@k=1\|𝒯fail\|∑τi∈𝒯fail𝕀\[ranki\(ti∗\)≤k\],MRR=1\|𝒯fail\|∑τi∈𝒯fail1ranki⁡\(ti∗\)\.\\operatorname\{Hit@\}k=\\frac\{1\}\{\|\\mathcal\{T\}^\{\\mathrm\{fail\}\}\|\}\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\}\}\\mathbb\{I\}\\\!\\left\[\\operatorname\{rank\}\_\{i\}\(t\_\{i\}^\{\*\}\)\\leq k\\right\],\\qquad\\operatorname\{MRR\}=\\frac\{1\}\{\|\\mathcal\{T\}^\{\\mathrm\{fail\}\}\|\}\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\}\}\\frac\{1\}\{\\operatorname\{rank\}\_\{i\}\(t\_\{i\}^\{\*\}\)\}\.We report Hit@kkfork∈\{1,3,5\}k\\in\\\{1,3,5\\\}\. Results are reported as mean±\\pmpopulation standard deviation over three seeds\.

For score\-based evaluation, we use the attribution scoreαi,t\\alpha\_\{i,t\}assigned to timestamptt\. The root\-cause stepti∗t\_\{i\}^\{\*\}is the sole positive candidate; all timestamps in𝒩⁡\(τi\)\\mathcal\{N\}\(\\tau\_\{i\}\)are negative candidates\. Their total numbers over the evaluated failed trajectories are

C\+=\|𝒯fail\|,C−=∑τi∈𝒯fail\|𝒩⁡\(τi\)\|\.C\_\{\+\}=\|\\mathcal\{T\}^\{\\mathrm\{fail\}\}\|,\\qquad C\_\{\-\}=\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\}\}\|\\mathcal\{N\}\(\\tau\_\{i\}\)\|\.AUROC and AUPRC pool these candidate\-level predictions across trajectories rather than averaging per\-trajectory values\. AUROC compares every annotated root\-cause step with every non\-root step across the evaluated trajectories:

AUROC=1C\+​C−​∑τi∈𝒯fail∑τj∈𝒯fail∑t∈𝒩⁡\(τj\)\[𝕀⁡\(αi,ti∗\>αj,t\)\+12​𝕀​\(αi,ti∗=αj,t\)\]\.\\operatorname\{AUROC\}=\\frac\{1\}\{C\_\{\+\}C\_\{\-\}\}\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\}\}\\sum\_\{\\tau\_\{j\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\}\}\\sum\_\{t\\in\\mathcal\{N\}\(\\tau\_\{j\}\)\}\\left\[\\mathbb\{I\}\(\\alpha\_\{i,t\_\{i\}^\{\*\}\}\>\\alpha\_\{j,t\}\)\+\\frac\{1\}\{2\}\\mathbb\{I\}\(\\alpha\_\{i,t\_\{i\}^\{\*\}\}=\\alpha\_\{j,t\}\)\\right\]\.\(31\)
For AUPRC, we aggregate the attribution scores of all candidate steps across the evaluated trajectories and sort theJJdistinct scores in descending order asθ1\>θ2\>⋯\>θJ\\theta\_\{1\}\>\\theta\_\{2\}\>\\cdots\>\\theta\_\{J\}\. At thresholdθj\\theta\_\{j\}, every candidate step withαi,t≥θj\\alpha\_\{i,t\}\\geq\\theta\_\{j\}is treated as a positive prediction\. Such a prediction is a true positive ift=ti∗t=t\_\{i\}^\{\*\}and a false positive otherwise\. The corresponding counts are

TPj=∑τi∈𝒯fail𝕀⁡\(αi,ti∗≥θj\),FPj=∑τi∈𝒯fail∑t∈𝒩⁡\(τi\)𝕀⁡\(αi,t≥θj\)\.\\operatorname\{TP\}\_\{j\}=\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\}\}\\mathbb\{I\}\(\\alpha\_\{i,t\_\{i\}^\{\*\}\}\\geq\\theta\_\{j\}\),\\qquad\\operatorname\{FP\}\_\{j\}=\\sum\_\{\\tau\_\{i\}\\in\\mathcal\{T\}^\{\\mathrm\{fail\}\}\}\\sum\_\{t\\in\\mathcal\{N\}\(\\tau\_\{i\}\)\}\\mathbb\{I\}\(\\alpha\_\{i,t\}\\geq\\theta\_\{j\}\)\.Precision and recall are

Precj=TPjTPj\+FPj,Recj=TPjC\+\.\\operatorname\{Prec\}\_\{j\}=\\frac\{\\operatorname\{TP\}\_\{j\}\}\{\\operatorname\{TP\}\_\{j\}\+\\operatorname\{FP\}\_\{j\}\},\\qquad\\operatorname\{Rec\}\_\{j\}=\\frac\{\\operatorname\{TP\}\_\{j\}\}\{C\_\{\+\}\}\.We compute AUPRC using the non\-interpolated average\-precision formulation:

AUPRC=∑j=1J\(Recj−Recj−1\)​Precj,Rec0=0\.\\operatorname\{AUPRC\}=\\sum\_\{j=1\}^\{J\}\\left\(\\operatorname\{Rec\}\_\{j\}\-\\operatorname\{Rec\}\_\{j\-1\}\\right\)\\operatorname\{Prec\}\_\{j\},\\qquad\\operatorname\{Rec\}\_\{0\}=0\.\(32\)

### E\.2Hyperparameter Settings

Table 8:Hyperparameter settings for our method\.ComponentHyperparameterValueBackboneHidden\-state dimensiondhd\_\{h\}5,120Layer selectionCross\-validation folds5Layer budgetkk8Ridge ratio0\.10Selected layers\{7,15,19,32,33,47,49,57\}\\\{7,15,19,32,33,47,49,57\\\}ProjectionDimension per branchdpd\_\{p\}2,048Final step\-feature dimension2​dp2d\_\{p\}4,096Random state42EncoderTransformer layers / attention heads4 / 4Hidden dimension128Output dimensiondzd\_\{z\}128Feed\-forward dimension384Maximum sequence length \(steps\)512Dropout0\.10AugmentationStep / feature masking probability0\.10 / 0\.10Gaussian\-noise standard deviation0\.01ObjectiveInfoNCE temperatureκ\\kappa0\.12\(λrank,λroot,λnon​\-​root\)\(\\lambda\_\{\\mathrm\{rank\}\},\\lambda\_\{\\mathrm\{root\}\},\\lambda\_\{\\mathrm\{non\\text\{\-\}root\}\}\)\(1\.0,0\.95,0\.25\)\(1\.0,0\.95,0\.25\)OptimizationOptimizerAdamWLearning rate / weight decay2×10−42\\times 10^\{\-4\}/10−410^\{\-4\}Batch size32Maximum epochs50Gradient clipping1\.0Training seeds6, 20, 42Table[8](https://arxiv.org/html/2610.11334#A5.T8)summarizes the settings used with Qwen3\.5\-27B\([Qwen Team, 2026](https://arxiv.org/html/2610.11334#bib.bib31)\)as the hidden\-state extraction backbone\. The selected eight layers provide features for the pattern and deviation branches, each with dimensiondp=2,048d\_\{p\}=2\{,\}048\. A four\-layer bidirectional Transformer maps their concatenation todz=128d\_\{z\}=128step representations\. The backbone, projection matrix, and successful\-reference mean remain fixed while the encoder and scoring head are jointly optimized\.

## Appendix FBaseline Settings

### F\.1Heuristic Baselines

#### First\.

First is a deterministic position\-only baseline that ranks candidate steps by their step order and selects the earliest step as the predicted root cause\. It has no trainable parameters and does not use the trajectory text, agent identity, or benchmark labels\. Its standard deviation across seeds is zero because the ranking is deterministic\.

#### Random\.

Random produces a complete ranking by uniformly shuffling all candidate steps\. For a fair comparison, we evaluate it with the same three seeds as the other methods and report the mean and population standard deviation\. Both First and Random operate directly on each benchmark’s original candidate set\.

### F\.2LLM\-Based Methods

#### Common protocol\.

We adopt two standard LLM\-based attribution protocols from Who&When\([Zhang et al\., 2025a](https://arxiv.org/html/2610.11334#bib.bib8)\)\. Appendix[I](https://arxiv.org/html/2610.11334#A9)provides the complete prompts used in all evaluations, including the task instruction,ground\-truth answer, and serialized multi\-agent conversation\. Each run returns at most one predicted step\. A prediction is counted as correct only when the parsed step maps to a valid candidate and matches the annotated root\-cause step exactly; an unparsable response, an out\-of\-candidate step, or a run that finds no error is counted as incorrect\. Since these protocols return a single step rather than a complete ranking, we report only Hit@1\. The two protocols operate as follows:

- •All\-at\-Once\.The model receives the complete conversation in one request and is asked to return the responsible agent, the first erroneous step, and a short explanation\.
- •Step\-by\-Step\.The model processes growing trajectory prefixes in step order\. At each step, the prompt asks whether the most recent action contains an error that could derail the execution\. Evaluation stops at the first response parsed as “Yes,” and that step becomes the prediction\. If all steps receive “No” or no valid decision is produced, the trajectory is marked as having no predicted root cause\. This protocol uses more calls than All\-at\-Once but prevents later trajectory content from influencing an earlier decision\.

#### Models and inference settings\.

We evaluate both protocols with Qwen3\.5\-27B\([Qwen Team, 2026](https://arxiv.org/html/2610.11334#bib.bib31)\), DeepSeek V4 Flash\([Xu et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib23)\), and GPT\-5\.6 Luna\([OpenAI, 2026b](https://arxiv.org/html/2610.11334#bib.bib32)\)\. Qwen3\.5\-27B is served locally with vLLM using bfloat16 weights, greedy decoding with temperature zero, thinking disabled, and a configured 131K\-token context window\. Across all four benchmarks, all three models use a maximum output length of 16,384 tokens under both protocols\. DeepSeek V4 Flash uses explicitly enabled thinking, while GPT\-5\.6 Luna retains its default configuration\. Both protocols are evaluated with each of the three models using inference seeds\{6,20,42\}\\\{6,20,42\\\}, and results are reported as mean±\\pmpopulation standard deviation\.

#### AgenTracer\.

AgenTracer\([Zhang et al\., 2026b](https://arxiv.org/html/2610.11334#bib.bib1)\)constructs attribution training data through counterfactual replay and fault injection and trains an 8B attribution model with reinforcement learning\. Since the model is not publicly available, we directly use its reported Who&When results, without re\-evaluation on our filtered Algorithm subset\. These are single reported values and do not have a three\-seed standard deviation\.

#### CHIEF\.

CHIEF\([Wang et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib3)\)constructs hierarchical causal graphs for failure attribution\. We follow its released six\-stage pipeline, retaining the original prompts, parsers, and retrieval setup\. Inference uses DeepSeek V4 Flash with thinking enabled, a maximum output length of 16,384 tokens per call, and seeds\{6,20,42\}\\\{6,20,42\\\}\. Retrieval uses the supplied knowledge bases and all\-MiniLM\-L6\-v2 embeddings\. Following the original protocol, prompts include task reference answers but not root\-cause annotations\. We evaluate the final predicted step using Hit@1\.

### F\.3Feature\-Based Methods

Table 9:Optimization settings for trainable feature\-based baselines\.#### OAT\.

OAT\([Yeh et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib6)\)formulates unsupervised failure attribution as one\-class learning: it learns the normal execution dynamics of successful trajectories and treats deviations from these dynamics as failure signals\. We extract the query and candidate\-action representations from the final layer of Qwen3\.5\-27B, using mean pooling over the corresponding tokens\. For each seed, the successful trajectories are randomly divided into 80% training and 20% validation partitions\. PCA is fitted only on the training partition to reduce the 5,120\-dimensional representations to 64 dimensions; the projected features are then standardized by subtracting the training\-partition mean and dividing by its standard deviation\. The query representation initializes the trajectory dynamics, and step timestamps are normalized to\[0,1\]\[0,1\]\.

The predictor models each successful trajectory with a Neural CDE whose projected input and latent\-state dimensions are both 64\. We directly use the released OAT implementation, which employs a three\-layer MLP vector field, cubic interpolation, the Euler solver, and a four\-layer control gate with hidden width 12\. The query\-conditioned initializer and hidden\-state bridge are three\-layer MLPs with hidden width 64; adjoint computation and initial\-state norm regularization are disabled\. We optimize the model with AdamW using the settings in Table[9](https://arxiv.org/html/2610.11334#A6.T9)\. At evaluation time, the squared reconstruction erroret=∥𝐡t−𝐡^t∥22e\_\{t\}=\\lVert\\mathbf\{h\}\_\{t\}\-\\widehat\{\\mathbf\{h\}\}\_\{t\}\\rVert\_\{2\}^\{2\}is used as the anomaly score for steptt, and steps with larger scores are ranked earlier\.

#### StepFinder\.

StepFinder\([Zhu et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib7)\)is a supervised lightweight attribution model that represents a trajectory as parallel sequences of step\-content and agent\-identity embeddings\. We separately encode the content and agent name of each candidate step with Qwen3\-Embedding\-0\.6B\([Zhang et al\., 2025b](https://arxiv.org/html/2610.11334#bib.bib29)\), using last\-token pooling and a maximum input length of 8,192 tokens, and retain 128 and 32 dimensions, respectively\. A two\-layer BiLSTM with hidden size 64 per direction and dropout 0\.5 captures the evolution of the trajectory\. The resulting step representations are processed by a two\-head agent\-aware interaction module with head dimension 32\. This module augments global step attention with an agent\-similarity bias and a trajectory\-level agent gate, both controlled byα=0\.1\\alpha=0\.1\.

StepFinder first predicts a learned error logit for each step and then refines it using normalized differences between steps at scales\{1,2\}\\\{1,2\\\}and a linearly decaying position bias that favors earlier root causes\. We set the corresponding weights toβ=0\.9\\beta=0\.9andγ=0\.4\\gamma=0\.4\. Training minimizes trajectory\-level cross\-entropy over the candidate steps together with an auxiliary loss for predicting the next\-step representation; the auxiliary loss weight isλ=0\.9\\lambda=0\.9\. We use AdamW, gradient clipping at 1\.0, and the optimization settings in Table[9](https://arxiv.org/html/2610.11334#A6.T9)\. The same configuration is used across all reported benchmarks\. Candidate steps are ranked by the resulting refined error scores\.

#### MASPrism\.

MASPrism\([Liu et al\., 2026b](https://arxiv.org/html/2610.11334#bib.bib5)\)is a training\-free and oracle\-free attribution method that extracts diagnostic signals during two prefill passes of a frozen language model\. We follow its original two\-stage scoring protocol without training an attribution head or using root\-cause annotations\. The Filtering stage uses step\-level negative log\-likelihood to identify the top 20% downstream symptom steps and attention from these symptoms to retain up to five earlier candidate sources\. The Diagnosis stage restores the selected symptom and candidate contents, recomputes the prefill signals, and combines normalized attention, NLL contrast, and multi\-symptom consensus to rank the candidate steps\. We use a consensus top\-55cutoff and set the consensus coefficient to 0\.3\.

We use the frozen Qwen3\-0\.6B\([Yang et al\., 2025](https://arxiv.org/html/2610.11334#bib.bib24)\)in bfloat16\. Each trajectory is serialized with the model’s chat template without the ground\-truth answer, and the maximum input length is set to 16,384 tokens\. Following the original layer\-selection rule, we average attention over all heads and the final 20% of the 28 Transformer blocks, corresponding to layers 22–27 under zero\-based indexing\. The Filtering stage retains up to 512 prefix tokens per step\. During Diagnosis, the selected symptom and candidate steps are restored in full, while each remaining ordinary step retains up to 128 prefix tokens\. These larger input budgets reduce truncation as a potential confound\. The method performs only two prefill passes, requires no decoding, and is deterministic; the three reported seeds therefore produce identical predictions\.

#### ASCon\.

ASCon\([Jiang et al\., 2026](https://arxiv.org/html/2610.11334#bib.bib33)\)jointly contextualizes step and agent representations to support failure attribution\. We use its step\-level predictions under the failure attribution setting to localize the root\-cause step\. In our implementation, candidate\-step text and each unique agent name are encoded separately with Qwen3\.5\-27B\([Qwen Team, 2026](https://arxiv.org/html/2610.11334#bib.bib31)\)using last\-token pooling to obtain 5,120\-dimensional embeddings\. Feature extraction uses bfloat16 weights, a maximum input length of 1,024 tokens, and an embedding batch size of 4\. Consecutive candidate steps form a directed step graph, while transitions between their recorded actors form a directed agent graph\. Under our unified candidate representation, the graphs contain only the retained candidate\-step nodes and their corresponding actor nodes; the user query and user role are not instantiated as additional graph nodes\. A two\-layer direction\-aware graph attention network separately aggregates preceding, succeeding, and self information to obtain contextualized step representations\.

ASCon next aggregates the contextualized representations of each agent’s steps through masked step\-to\-agent attention, producing a behavior\-aware agent representation\. It then incorporates the agent identity and global trajectory context and propagates information over the directed agent graph\. The resulting representation of each agent is fused back into the representations of its steps, allowing step\-level predictions to incorporate agent\-specific behavioral context\. We use two direction\-aware graph attention layers, 256\-dimensional step representations, 128\-dimensional agent representations, a 128\-dimensional step\-to\-agent attention space, and dropout 0\.1\. Separate step and agent classifiers are trained jointly with class\-balanced binary cross\-entropy using loss weightsλstep=0\.3\\lambda\_\{\\mathrm\{step\}\}=0\.3andλagent=1\.0\\lambda\_\{\\mathrm\{agent\}\}=1\.0\. Optimization uses AdamW for 12 epochs with gradient clipping at 1\.0 and the remaining settings in Table[9](https://arxiv.org/html/2610.11334#A6.T9)\. Candidate steps are ranked by the step\-classifier logits\.

## Appendix GAdditional Experimental Results

This section collects additional experimental results omitted from the main text, including complete evaluation metrics, detailed subset analyses, backbone comparisons, and time efficiency analysis\.

### G\.1MRR Across Rank Cutoffs

Figure 4:MRR@KKacross rank cutoffsK=1,…,5K=1,\\ldots,5on the four benchmarks\. Curves show the cumulative reciprocal\-rank score as the cutoff increases; error bars denote the population standard deviation over three seeds\.MRR@KKcomplements Hit@KKby assigning a reciprocal\-rank score to the root cause when it appears within the topKKpositions and zero otherwise, rewarding methods that place the root cause earlier within a fixed inspection budget\. Figure[4](https://arxiv.org/html/2610.11334#A7.F4)reports this metric for rank cutoffs from 1 to 5\. ReCast achieves the highest MRR@KKamong the compared methods at every cutoff across all four benchmarks\. On TraceElephant, ReCast reaches MRR@1 of 0\.3124 and MRR@5 of 0\.4341, compared with 0\.2838 and 0\.3860 for ASCon\.

### G\.2Score\-Level Discrimination

The rank\-based results above measure whether each trajectory’s root cause is prioritized over the other steps in the same trajectory\. We additionally evaluate whether the raw attribution scores consistently separate root\-cause steps from non\-root steps after pooling all candidate steps within each benchmark’s evaluation population\. Figures[5](https://arxiv.org/html/2610.11334#A7.F5)and[6](https://arxiv.org/html/2610.11334#A7.F6)show the resulting ROC and PR curves, respectively\. The corresponding AUROC and AUPRC follow the definitions in Appendix[E\.1](https://arxiv.org/html/2610.11334#A5.SS1)\.

![Refer to caption](https://arxiv.org/html/2610.11334v1/four_benchmark_roc_ascon_gold.png)Figure 5:Receiver operating characteristic \(ROC\) curves on the four benchmarks\. Each curve is computed by pooling the candidate\-step predictions within a benchmark\. Solid lines show the mean over three seeds, and shaded regions indicate the population standard deviation\.Figure 6:Precision–recall \(PR\) curves on the four benchmarks\. Each curve is computed by pooling the candidate\-step predictions within a benchmark\. Solid lines show the mean over three seeds, and shaded regions indicate the population standard deviation\.ReCast achieves the highest AUROC and AUPRC on all four benchmarks\. Its advantage in AUPRC is particularly meaningful under class imbalance, as PR curves capture the trade\-off between recovering root\-cause steps and avoiding false positives\. On Who&When and TraceElephant, it obtains AUROC values of0\.81390\.8139and0\.75830\.7583and AUPRC values of0\.29640\.2964and0\.21250\.2125, respectively\. On AFTraj\-2K and Who&When Pro, ReCast reaches AUROC values of0\.97750\.9775and0\.98870\.9887together with AUPRC values of0\.90820\.9082and0\.96670\.9667\.

Together, the MRR@KK, AUROC, and AUPRC results support the effectiveness of ReCast for failure attribution\. Its advantage extends from prioritizing the root\-cause step within each trajectory to distinguishing root\-cause steps from other candidates across trajectories\.

### G\.3Hidden\-State Backbone Comparison

Table 10:Hidden\-state backbone configurations\. Each backbone retains eight layers selected independently using the same source\-only procedure\.#### Protocol\.

To examine whether attribution performance scales monotonically with backbone size, we compare Qwen3\.5 models with 0\.8B, 9B, and 27B parameters while holding the downstream ReCast pipeline fixed\. For each backbone, we independently select eight layers using attribution\-aware layer selection on ReCast\-2K, set the projection dimension of each branch todp=2,048d\_\{p\}=2\{,\}048, and retain the same encoder architecture\. Training and evaluation follow the same settings as the main experiments\. Results are reported as mean±\\pmpopulation standard deviation over seeds 6, 20, and 42\. Table[10](https://arxiv.org/html/2610.11334#A7.T10)summarizes the backbone configurations and selected layers\.

Table 11:Effect of the hidden\-state extraction backbone\. Results report mean±\\pmpopulation standard deviation over three encoder seeds\. Bold indicates the best mean within each evaluation population, determined before rounding\.
#### Results\.

Table[11](https://arxiv.org/html/2610.11334#A7.T11)shows that the benefits of a larger backbone vary across benchmarks and metrics\. Qwen3\.5\-27B achieves the highest Hit@1 on both Who&When subsets, AFTraj\-2K, and Who&When Pro, with all six metrics improving with backbone size on Who&When Pro\. However, smaller backbones remain competitive: 0\.8B and 9B achieve the highest Hit@1 on Captain\-Agent and Magentic\-One, respectively\. On AFTraj\-2K, their Hit@1 scores are within 1\.23 percentage points of 27B, and 9B achieves the highest Hit@5 and AUROC\. These results suggest that larger backbones can improve attribution performance, although the gains vary across benchmarks and metrics\. With attribution\-aware layer selection and representation learning, smaller backbones can also provide effective signals for failure attribution\.

### G\.4Time Efficiency Analysis

Figure 7:Hit@1 accuracy \(%\) versus measured time per trajectory on Who&When and TraceElephant\. Points show three\-seed means; time is in seconds on a logarithmic scale\. Feature\-based methods report attribution time excluding feature extraction, while LLM methods report inference time excluding backend initialization\. Upper\-left positions indicate higher accuracy at lower measured cost\. A and S denote All\-at\-Once and Step\-by\-Step\.#### Protocol\.

We compare Hit@1 accuracy with the average processing time per trajectory on Who&When and TraceElephant\. ReCast, StepFinder, ASCon, and OAT use precomputed features, excluding feature extraction and embedding\-cache preparation from attribution time\. For MASPrism, we report attribution time excluding model initialization, tokenization, and prefill\-stage signal extraction\. For LLM\-based methods, inference time excludes model initialization but includes prompt preparation and API delays and retries where applicable\. We divide the total measured time by the number of evaluated trajectories and report the mean over three seeds in Figure[7](https://arxiv.org/html/2610.11334#A7.F7)\.

#### Results\.

ReCast achieves the highest Hit@1 across all four subsets with mean attribution times ranging from 0\.11 to 8\.28 ms per trajectory\. It has the lowest mean attribution time among the compared feature\-based methods on HC, AG, and Magentic\-One\. On Captain\-Agent, StepFinder and ASCon are faster, but ReCast achieves higher accuracy\. In contrast, the LLM\-based baselines shown in Figure[7](https://arxiv.org/html/2610.11334#A7.F7)require 2\.88–30\.50 seconds per trajectory under the measured configurations while attaining lower Hit@1\. This comparison highlights the efficiency of ReCast’s attribution stage when step features are available\.

## Appendix HCase Study

Figure 8:Step\-level attribution scores on three HC trajectories \(a–c\) and three AG trajectories \(d–f\)\. Original zero\-based step indices are retained; gaps indicate events without candidate scores\.Figure[8](https://arxiv.org/html/2610.11334#A8.F8)presents six cases from the Handcrafted and Algorithm subsets of Who&When\. The curves show ReCast’s attribution scores for all candidate steps, with higher scores indicating higher ranks within each trajectory\. In these selected cases, ReCast ranks the annotated root\-cause step above other high\-scoring candidates, providing qualitative support for the effectiveness of its learned step representations in failure attribution\.

HC\-50, restaurant search \(Figure[8](https://arxiv.org/html/2610.11334#A8.F8)\(a\)\)\.The task asks for dine\-in restaurants within one block of Washington Square Park that serve vegan main dishes for under $15\. At step 5, a location\-only search returns general listings without verifying the remaining constraints; the dataset annotates this retrieval as the root cause\. Later steps also receive elevated scores\. For example, the summary at step 106 presents proximity and dine\-in service as confirmed while leaving vegan options and prices for further verification\. ReCast nevertheless assigns its highest score to step 5, ranking the earlier annotated error above this later competing peak\.

HC\-37, missing a source constraint \(Figure[8](https://arxiv.org/html/2610.11334#A8.F8)\(b\)\)\.The task asks for an organism’s maximum body length in meters, as reported by the Monterey Bay Aquarium website\. At step 4, the agent searches for National Geographic videos without including the required source, an omission annotated as the root cause\. At step 22, it opens a*Human Origins*video whose displayed content does not provide the requested measurement\. Despite other local score peaks, ReCast ranks step 4 first, prioritizing the earlier annotated omission over the later browsing event\.

HC\-42, premature completion \(Figure[8](https://arxiv.org/html/2610.11334#A8.F8)\(c\)\)\.The task asks which word was deleted in a federal rule amendment documented on Cornell Law School’s website\. Browsing the Federal Rules index at step 16 and the Witnesses article at step 24 produces local score peaks\. At step 29, however, the Orchestrator declares the task complete while acknowledging that the retrieved page does not identify the requested word\. This premature completion is the annotated root cause\. ReCast assigns its highest score to step 29, ranking the unsupported completion decision above the earlier browsing steps\.

AG\-116, substituting simulated data \(Figure[8](https://arxiv.org/html/2610.11334#A8.F8)\(d\)\)\.The task asks for the lowest sale price of a single\-family house in Queen Anne in January 2023\. With the transaction file unavailable, step 3 proposes analysis using an assumed CSV path and receives an elevated score\. At step 7, the agent constructs simulated transactions instead of obtaining the actual records; the dataset annotates this substitution as the root cause\. ReCast assigns its highest score to step 7, ranking the use of simulated data above the earlier analysis based on an assumed file path\.

AG\-109, incorrect distance conversion \(Figure[8](https://arxiv.org/html/2610.11334#A8.F8)\(e\)\)\.The task seeks supermarkets within two blocks of Lincoln Park, Chicago, that offer ready\-to\-eat salads for under $15\. At step 1, the agent converts miles to blocks using an assumed rate of 20 blocks per mile, which the dataset identifies as incorrect for Chicago\. A later discussion of candidate stores at step 5 also produces a local score peak\. ReCast nevertheless ranks step 1 highest, selecting the annotated conversion error rather than the subsequent store recommendations\.

AG\-120, unverified restaurant claims \(Figure[8](https://arxiv.org/html/2610.11334#A8.F8)\(f\)\)\.AG\-120 addresses the same restaurant\-search task as HC\-50 through a different execution\. At step 7, a missing Google Maps API key blocks automated verification, and the agent proposes manual checks\. At step 8, another agent presents distances, dining options, and menu prices as verified without performing the required checks, which the dataset annotates as the root cause\. ReCast’s score rises at step 7 but peaks at step 8, prioritizing the unsupported claims over the preceding verification obstacle\.

## Appendix IPrompt Templates

We directly use the Who&When implementation\([Zhang et al\., 2025a](https://arxiv.org/html/2610.11334#bib.bib8)\)and present its prompt templates below\. The red fields denote trajectory\-specific values inserted at inference time\. The\{ground\_truth\}field contains the benchmark\-provided task answer, not the root\-cause annotation\. Line wrapping below is typographical and does not change the prompt content\.

### I\.1All\-at\-Once Prompt

All\-at\-Once presents the complete trajectory in one request and asks the model to identify the responsible agent and the first erroneous step\.

All\-at\-Once Evaluation Template

SYSTEM MESSAGEYou are an AI assistant skilled in analyzing conversations\.USER MESSAGEYou are an AI assistant tasked with analyzing a multi\-agent conversation history when solving a real world problem\. The problem is:\{problem\}The Answer for the problem is:\{ground\_truth\}Identify which agent made an error, at which step, and explain the reason for the error\. Here’s the conversation:\{conversation\}Based on this conversation, please predict the following:1\. The name of the agent who made a mistake that should be directly responsible for the wrong solution to the real world problem\. If there are no agents that make obvious mistakes, decide one single agent in your mind\. Directly output the name of the Expert\.2\. In which step the mistake agent first made mistake\. For example, in a conversation structured as follows:\{ "agent a": "xx", "agent b": "xxxx", "agent c": "xxxxx", "agent a": "xxxxxxx" \},each entry represents a ‘step’ where an agent provides input\. The ‘x’ symbolizes the speech of each agent\. If the mistake is in agent c’s speech, the step number is 2\. If the second speech by ‘agent a’ contains the mistake, the step number is 3, and so on\. Please determine the step number where the first mistake occurred\.3\. The reason for your prediction\.Please answer in the format:Agent Name: \(Your prediction\) Step Number: \(Your prediction\) Reason for Mistake:

### I\.2Step\-by\-Step Prompt

Step\-by\-Step evaluates growing trajectory prefixes\. At each step, the model receives the history up to and including that step\. Evaluation terminates at the first response parsed as “Yes”; otherwise, it advances to the next prefix\.

Step\-by\-Step Evaluation Template

SYSTEM MESSAGEYou are a precise step\-by\-step conversation evaluator\.USER MESSAGE AT STEP\{step\}You are an AI assistant tasked with evaluating the correctness of each step in an ongoing multi\-agent conversation aimed at solving a real\-world problem\. The problem being addressed is:\{problem\}\. The Answer for the problem is:\{ground\_truth\}Here is the conversation history up to the current step:\{conversation\}The most recent step \(\{step\}\) was by ’\{agent\}’\.Your task is to determine whether this most recent agent’s action \(Step\{step\}\) contains an error that could hinder the problem\-solving process or lead to an incorrect solution\. Please respond with ’Yes’ or ’No’ and provide a clear explanation for your judgment\. Note: Please avoid being overly critical in your evaluation\. Focus on errors that clearly derail the process\.Respond ONLY in the format:1\. Yes/No\. 2\. Reason: \[Your explanation here\]