LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures

Hugging Face Daily Papers Papers
failure-diagnosis long-horizon-agents root-cause-analysis benchmark attribution

Summary

LongRCA Bench introduces a benchmark for diagnosing failures in long-horizon agent trajectories, and the RCTA method improves responsible role and root-cause step attribution.

When a long-horizon agent execution fails, outcome-level evaluation reveals the unsuccessful result but not where the decisive error entered the trajectory. Developers must then inspect the full execution to identify the responsible role and localize the earliest decisive root-cause step. Existing failure-attribution benchmarks largely focus on shorter traces, leaving diagnosis across hundreds of recorded steps underexplored. We introduce LongRCA Bench, comprising 1,140 failed trajectories across five domains without injected errors. It provides independently scored human labels for the responsible role and earliest decisive root-cause step. The median trajectory contains 145 steps, and the strongest baseline reaches only 13.2% exact root-step accuracy. We further present Root-Cause Trajectory Attribution (RCTA), a training-free method that retrieves candidate error steps from segment summaries and traces them to available earlier handoff instructions. Using the same backbone, benchmark instances, and scoring protocol, RCTA reaches 51.1% responsible-role accuracy and 24.1% exact root-step accuracy. These results highlight the need to evaluate responsible-role attribution and exact root-step localization as separate targets in long-trajectory failure diagnosis.
Original Article
View Cached Full Text

Cached at: 08/26/26, 03:17 AM

Paper page - LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures

Source: https://huggingface.co/papers/2608.15242 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

LongRCA Bench evaluates failure diagnosis across lengthy agent trajectories, and the training-free RCTA method improves attribution of responsible roles and root-cause steps.

When a long-horizon agent execution fails, outcome-level evaluation reveals the unsuccessful result but not where the decisive error entered the trajectory. Developers must then inspect the full execution to identify theresponsible roleand localize the earliest decisiveroot-cause step. Existingfailure-attributionbenchmarks largely focus on shorter traces, leaving diagnosis across hundreds of recorded steps underexplored. We introduceLongRCA Bench, comprising 1,140 failed trajectories across five domains without injected errors. It provides independently scored human labels for theresponsible roleand earliest decisiveroot-cause step. The median trajectory contains 145 steps, and the strongest baseline reaches only 13.2% exact root-step accuracy. We further presentRoot-Cause Trajectory Attribution(RCTA), a training-free method that retrieves candidate error steps fromsegment summariesand traces them to available earlierhandoff instructions. Using the same backbone, benchmark instances, and scoring protocol,RCTAreaches 51.1% responsible-role accuracy and 24.1% exact root-step accuracy. These results highlight the need to evaluate responsible-role attribution and exact root-step localization as separate targets in long-trajectory failure diagnosis.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.15242

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.15242 in a model README.md to link it from this page.

Datasets citing this paper1

#### CLoud5-real/longrca-bench Viewer• Updated1 day ago • 1.14k • 55 • 2

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.15242 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

How Far Can Root Cause Analysis Go on Real-World Telemetry Data?

arXiv cs.AI

This paper studies root cause analysis on real-world telemetry data using the OpenRCA benchmark, showing that existing classical and LLM-based methods fail and proposing a Structured Multi-Agent RCA pipeline that substantially outperforms them. It further reveals through reverse reasoning that the primary bottleneck is reasoning capability rather than data access, and introduces automated rule mining to reduce reliance on manual domain knowledge.

StableRCA: Robust Graph-Agnostic Mechanism-Level Root Cause Analysis

arXiv cs.LG

StableRCA is a novel root cause analysis framework that identifies intervention targets by estimating local Markov boundaries and detecting conditional distribution shifts, avoiding the need for global causal graph discovery and demonstrating robustness across synthetic and real-world datasets.

Long-Horizon Agent Trajectory Attribution: A Unified Benchmark and Fine-Grained Annotation Framework

arXiv cs.AI

This paper introduces a unified benchmark and fine-grained annotation framework for long-horizon agent trajectory attribution, enabling evaluation of primary attribution localization and attribution-chain recovery across diverse settings. It provides over 1,300 annotated trajectories from existing agent benchmarks and releases a reusable annotation skill for standardizing future trajectories.

LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis

arXiv cs.LG

Introduces LongDS, a benchmark for evaluating LLM agents on long-horizon, multi-turn data analysis tasks. Evaluations show that even the best models achieve only 48.45% accuracy, with performance dropping sharply over turns, highlighting that maintaining analytical state is the key bottleneck.