Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

Hugging Face Daily Papers Papers
llm-agents dataset failure-analysis error-recovery fine-tuning post-training

Summary

论文提出 Agent Error Dataset（AED），包含 50,228 个错误-诊断对，覆盖 33 个环境与 23 个策略模型，并通过 AET 流水线用于失败分析与错误感知的后训练，显著提升修正通过率与诊断准确率。

An unsuccessful LLM agent rollout contains more information than its final reward: the observations available to the agent, the actions it chose, and the environment's responses. Reusing this experience for learning requires identifying a decision to revise and testing a concrete alternative. We introduce the Agent Error Dataset (AED), comprising 50,228 error-diagnosis pairs from 9,961 source tasks across 33 environments, 19 harness families, and 23 policy models in text-based agent systems. We retain source traces and execution metadata to support cross-setting failure analysis and re-diagnosis without repeating the original rollout. Our five-stage Agentic Error-to-Training (AET) pipeline collects natural failures, generates diagnoses and proposed corrections, and checks them against recorded evidence. Where replay is supported, we compare corrections with original-action retries from the same checkpoint under matched execution settings. We then construct separate training views for diagnosis and actor recovery. Across 3,062 matched replay pairs, first-proposal corrections raise verifier pass rates from 18.4% to 51.1%, a gain of 32.7 percentage points. Using a separately frozen diagnosis release, full-diagnosis fine-tuning on 1,656 source tasks raises Qwen3-8B's exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds on a 943-case holdout. The strongest prompted reference in this comparison scores 54.7%, and mean agreement improves at each of four increasing training-set sizes. In a single-seed comparison of actor-training recipes, action-only repair training scores 6.67 percentage points higher on WebShop-lite than success-only training.
Original Article
View Cached Full Text

Cached at: 10/01/26, 04:20 AM

Paper page - Agent Error Dataset: Scaling 50,000 Error–Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

Source: https://huggingface.co/papers/2609.40111

Abstract

AnunsuccessfulLLMagentrolloutcontainsmoreinformationthanitsfinalreward:theobservationsavailabletotheagent,theactionsitchose,andtheenvironment’sresponses.Reusingthisexperienceforlearningrequiresidentifyingadecisiontoreviseandtestingaconcretealternative.WeintroducetheAgentErrorDataset(AED),comprising50,228error-diagnosispairsfrom9,961sourcetasksacross33environments,19harnessfamilies,and23policymodelsintext-basedagentsystems.Weretainsourcetracesandexecutionmetadatatosupportcross-settingfailureanalysisandre-diagnosiswithoutrepeatingtheoriginalrollout.Ourfive-stageAgenticError-to-Training(AET)pipelinecollectsnaturalfailures,generatesdiagnosesandproposedcorrections,andchecksthemagainstrecordedevidence.Wherereplayissupported,wecomparecorrectionswithoriginal-actionretriesfromthesamecheckpointundermatchedexecutionsettings.Wethenconstructseparatetrainingviewsfordiagnosisandactorrecovery.Across3,062matchedreplaypairs,first-proposalcorrectionsraiseverifierpassratesfrom18.4%to51.1%,againof32.7percentagepoints.Usingaseparatelyfrozendiagnosisrelease,full-diagnosisfine-tuningon1,656sourcetasksraisesQwen3-8B’sexact-stepagreementwithinternalteacherlabelsfrom47.2%to63.6%,averagedoverthreeseedsona943-caseholdout.Thestrongestpromptedreferenceinthiscomparisonscores54.7%,andmeanagreementimprovesateachoffourincreasingtraining-setsizes.Inasingle-seedcomparisonofactor-trainingrecipes,action-onlyrepairtrainingscores6.67percentagepointshigheronWebShop-litethansuccess-onlytraining.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.40111

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.40111 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.40111 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.40111 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

The Silent Failures in Agentic Data Science

arXiv cs.AI

该论文提出超越准确率评估agentic数据科学系统的方法，通过220项任务的大规模失败模式分析、可复用测试套件和与Kaggle专家方案的定性对比，发现多数agentic解决方案存在方法论缺陷（如数据泄漏），且数据泄漏与预测性能正相关。

AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems

arXiv cs.CL

This paper introduces AgentForesight, a framework for online auditing and early failure prediction in LLM-based multi-agent systems. It presents a new dataset, AFTraj-22K, and a specialized model, AgentForesight-7B, which outperforms leading proprietary models in detecting decisive errors during trajectory execution.

ARIA: Runtime Inspection and Amendment for Multi-Agent Systems

arXiv cs.AI

ARIA 提出一个用于大模型多智能体系统的运行时检查与修复框架，构建了首个大规模配对失败轨迹与执行验证修复的数据集（3,953 个修复对），并训练紧凑模型在执行过程中自主识别并修正错误，可泛化到未见过的框架和宿主模型。

I analyzed how 50+ AI teams debug production agent failures and got surprised

Reddit r/AI_Agents

Based on interviews with 50+ AI teams, the author highlights that production agent failures often stem from minor prompt or configuration issues rather than deep model problems. The article advocates for adopting software engineering practices like versioning, A/B testing, and experiment tracking to improve reliability.