ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents

# ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents
Source: [https://arxiv.org/html/2609.12345](https://arxiv.org/html/2609.12345)
Bowen GuanAffiliation:School of Computer Science and Technology, Dalian University of Technology, ChinaEmail:[20201071138@mail\.dlut\.edu\.cn](mailto:)Zhentao YinAffiliation:School of Computer Science and Technology, Dalian University of Technology, ChinaEmail:[yinzhentao2003@mail\.dlut\.edu\.cn](mailto:)Yanming Shen11footnotemark:1Affiliation:School of Computer Science and Technology, Dalian University of Technology, ChinaEmail:[shen@dlut\.edu\.cn](mailto:)

###### Abstract

Existing agent benchmarks mainly evaluate final task success or tool\-call correctness, providing limited insight into whether agents can reliably diagnose and recover from intermediate execution failures\. This limitation becomes particularly critical in multi\-turn parallel tool\-use scenarios, where errors may propagate across dependent branches and trigger cascading failures\. We introduce ParaRecover, a process\-level benchmark for evaluating error localization and recovery in multi\-turn parallel tool\-use agents\. Built upon a fine\-grained taxonomy of 14 error types covering planning dependencies, tool selection, and argument matching, the benchmark comprises 10,626 instances spanning two difficulty levels\. To enable fine\-grained, process\-oriented evaluation, we further propose the SDE rubric, which measures structural integrity, diagnostic reasoning, and evolutionary strategy during agent execution\. Experiments across more than ten mainstream LLMs reveal that even state\-of\-the\-art models still struggle with multi\-turn error propagation, implicit tool\-use failures, and precise replanning\. Moreover, we demonstrate that the SDE rubric provides effective supervision signals for improving agents’ reflective recovery capabilities\. Our data and code are available at[https://github\.com/gbw206/ParaRecover](https://github.com/gbw206/ParaRecover)\.

## 1Introduction

BenchmarkStructuredTrajectory Eval\.ParallelCallsError\-StateEval\.ReflectiveLocalizationReplanningDecisionProcess\-LevelMetricsAPI\-Bank✗✗✓✗✗△\\triangleTool\-Bench✗✗△\\triangle✗✗△\\triangleBFCL✗✓✗✗✗△\\triangleToolSandbox✓✓✗✗✗✓TRAJECT\-Bench✓✓✗✗✗✓ParaRecover✓✓✓✓✓✓Table 1:Comparison between previous agent benchmarks and ParaRecover in evaluation dimensions\.✓denotes full support,✗denotes no support, and△\\triangledenotes indirect support\.Large language model agents are increasingly used to solve complex tasks through planning, tool use, and interaction with external environments\([Yao et al\., 2022](https://arxiv.org/html/2609.12345#bib.bib14);[Qin et al\., 2023](https://arxiv.org/html/2609.12345#bib.bib7);[Huang et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib11)\)\. In real\-world scenarios such as scientific research, code generation, and data analysis\([Ma et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib12);[Yang et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib37);[Hu et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib13)\), tasks are often not simple sequential processes\. Instead, they are usually multi\-turn parallel processes, which involve multiple parallel subgoals, branching tool calls, and cross\-step dependencies\([Huang et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib11);[Wang et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib19)\)\. Therefore, systematically evaluating agents’ planning, decision\-making, and error\-recovery abilities in multi\-turn parallel tool\-use scenarios has become an important problem\.

Existing benchmarks for agent and tool use mainly focus on final task outcomes, such as answer correctness and overall task completion rate\([Mialon et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib15);[Yao et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib16);[Liu et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib17)\)\. However, successful task completion does not necessarily imply reliable agent behavior\. An agent may still complete the task through redundant calls, repeated trial and error, or incorrect recovery strategies, leading to high cost in real systems\. These end\-to\-end metrics are useful for measuring final performance, but they provide limited insight into whether agents can correctly localize failures, diagnose their causes, and recover from corrupted intermediate states\.

As shown in Table[1](https://arxiv.org/html/2609.12345#S1.T1), although some recent studies consider process\-level behaviors, they typically focus on tool\-call success rate or provide only coarse\-grained or indirect judgments of intermediate decisions or format\([Li et al\., 2023](https://arxiv.org/html/2609.12345#bib.bib6);[Qin et al\., 2023](https://arxiv.org/html/2609.12345#bib.bib7);[Patil et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib8);[He et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib31);[Lu et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib45)\)\. As a result, they may fail to capture the fine\-grained behaviors required in multi\-turn parallel execution, such as deciding concurrent subtasks, identifying cross\-step dependencies, assessing the impact of failures on downstream branches, and replanning after errors\. Existing evaluation paradigms rarely characterize these recovery challenges in a systematic way, making it difficult to understand how and where agents fail\. In practical applications, however, robust error diagnosis, reflection, and replanning capabilities are crucial for reliable long\-horizon agent execution\.

Motivated by these limitations, we construct a fine\-grained error taxonomy based on agent execution trajectories in real\-world multi\-turn parallel scenarios, covering 14 error types across planning dependencies, tool selection, and argument matching\. Building upon this taxonomy, we develop ParaRecover, a benchmark designed for complex multi\-turn parallel agent execution scenarios\. ParaRecover covers two difficulty levels: LEVEL\-1 only includes errors introduced in the most recent execution, while LEVEL\-2 involves multi\-turn propagation and various error types, comprehensively evaluating the agent’s error localization and replanning capabilities\. To further refine the evaluation, we introduce theSDEevaluation rubric, which measures agent behavior from three dimensions:Structural Integrity,Diagnostic Reasoning, andEvolutionary Strategy\. Our main contributions are as follows:

- •We introduce a process\-level benchmark for multi\-turn parallel tool use, comprising 10,626 instances across two difficulty levels, explicitly evaluating agents’ intermediate decision\-making, error localization, and corrective replanning\.
- •We develop and validate a fine\-grained error taxonomy for parallel agent execution\. Based on real rollout trajectories from multiple models, we further characterize how these errors emerge and distribute in practice\.
- •We propose the SDE Rubric, which evaluates agent from Structural Integrity, Diagnostic Reasoning, and Evolutionary Strategy, enabling fine\-grained assessment of reflection, backtracking, and decision capability after errors\.
- •We reveal a substantial gap between final task success and process reliability in current agents\. Although state\-of\-the\-art models often achieve high task completion rates, they still exhibit severe weaknesses in dependency reasoning, implicit failure localization, and efficient recovery under multi\-turn error propagation\.

![Refer to caption](https://arxiv.org/html/2609.12345v1/ParaRecover.png)Figure 1:Overview of the ParaRecover BenchmarkFamilyModel NameLEVEL\-1LEVEL\-2SIDRESAvg\.Pass@1SIDRESAvg\.Pass@1OpenAIGPT\-4o56\.0859\.1039\.1651\.4592\.2147\.9053\.7332\.5944\.7491\.50GPT\-4o mini49\.9146\.2029\.8341\.9889\.3244\.8540\.4125\.4536\.9088\.84GPT\-5 nano43\.8855\.3125\.0741\.4289\.5040\.5351\.6122\.5638\.2388\.92GPT\-5\.257\.5181\.8141\.2160\.2492\.6155\.2480\.7541\.6159\.2091\.84AnthropicClaude Sonnet 4\.669\.1274\.3650\.2764\.5894\.1065\.8471\.2248\.6361\.9093\.50Claude Opus 4\.673\.5678\.9455\.1869\.2394\.7870\.3876\.4153\.7666\.8594\.05GoogleGemini 3\.1 flash68\.3565\.8747\.1960\.4793\.8263\.5061\.1345\.0456\.5693\.25Gemini 3\.1 pro66\.9061\.7248\.2358\.9593\.6863\.2262\.0847\.1857\.4993\.22QwenQwen 3 8B47\.6157\.0825\.7243\.4791\.8445\.4151\.1226\.6941\.0791\.37Qwen 3 32B52\.2656\.9431\.1746\.8092\.0850\.4254\.5531\.9445\.6491\.86Qwen 3\.5 9B56\.5872\.7838\.3555\.9092\.2153\.1268\.6537\.2752\.9892\.14Qwen 3\.5 122B\-A10B72\.4877\.8752\.2867\.5494\.5069\.1274\.7751\.3465\.0893\.04ZHIPUGLM 4\.5 Air62\.4661\.0342\.6255\.3793\.5058\.4359\.3341\.6753\.1492\.72GLM 4\.666\.2055\.9147\.8056\.6493\.8163\.4252\.6848\.4754\.8693\.06DeepSeekDeepSeek\-V4\-flash73\.7475\.8054\.5768\.0494\.1371\.0473\.8254\.2366\.3693\.17DeepSeek\-V4\-flash\*74\.2374\.5356\.4968\.4294\.6771\.5072\.9055\.3066\.5793\.86DeepSeek\-V4\-pro74\.3072\.3558\.2168\.2994\.1371\.4771\.1057\.0366\.5093\.01Table 2:Performance of different models on LEVEL\-1 and LEVEL\-2 tasks
## 2Problem Definition

We consider real\-world tasks, which contain a substantial number of subtasks that are conditionally independent and therefore can be executed in parallel\. Although parallel execution can significantly improve efficiency, it also imposes substantially greater demands on agent planning and decision\-making capabilities\. To capture this more realistic and challenging setting, we formulate agent execution as amulti\-turn parallel decision process\.

To formalize parallelism, we represent the execution process as a DAGG=\(V,E\)G=\(V,E\), where each nodevi∈Vv\_\{i\}\\in Vdenotes a subtask or tool\-use operation, and each edge\(vi,vj\)∈E\(v\_\{i\},v\_\{j\}\)\\in Eindicates thatvjv\_\{j\}depends on the completion ofviv\_\{i\}\. Under this formulation, the prerequisite set of nodevjv\_\{j\}can be written as

D​e​p​\(vj\)=\{vi∈V\|\(vi,vj\)∈E\}\.Dep\(v\_\{j\}\)=\\\{v\_\{i\}\\in V\|\(v\_\{i\},v\_\{j\}\)\\in E\\\}\.\(1\)A nodevjv\_\{j\}is executable only when all of its dependencies have been satisfied, i\.e\.,

∀vi∈D​e​p​\(vj\),s⁡\(vi\)=1,\\forall v\_\{i\}\\in Dep\(v\_\{j\}\),\\quad s\(v\_\{i\}\)=1,\(2\)wheres⁡\(vi\)∈\{0,1\}s\(v\_\{i\}\)\\in\\\{0,1\\\}represents the state of nodeviv\_\{i\}, indicating whether it has been completed\. Consequently, tool invocations are executed layer\-wise strictly along the topological structure of the DAG\. Originating from the root, the graph naturally partitions into hierarchical layers: nodes directly dependent on the root constitute the first layer, nodes dependent on the first layer form the second, and so forth\.

Since the topological depth reached in any given round is non\-deterministic, the agent must dynamically infer the execution state\. At each interaction steptt, the agent receives the partially executed DAGG\(t−1\)G^\{\(t\-1\)\}from the previous round, along with the execution observationsO\(t\)O^\{\(t\)\}from the recently invoked tools\. A critical capability required of the agent is to deduce the executable frontier for the current round\. This frontier, denoted asF\(t\)F^\{\(t\)\}, comprises all pending nodes whose prerequisite dependencies have been strictly satisfied\.

The core objective of the model at stepttis multifaceted: it must critically analyze the structural correctness of the preceding DAG, identify the target nodes withinF\(t\)F^\{\(t\)\}, and evaluate the observationsO\(t\)O^\{\(t\)\}to diagnose which tool invocations succeeded or failed\. To formalize this diagnostic reflection and decision\-making process, we constrain the agent to follow a Chain\-of\-Thought \(CoT\)\([Wei et al\., 2023](https://arxiv.org/html/2609.12345#bib.bib2)\)reasoning paradigm\. Specifically, the agentAθA\_\{\\theta\}is prompted to generate a textual reflectionτ\(t\)\\tau^\{\(t\)\}\(thought\) followed by an updated graphG\(t\)G^\{\(t\)\}, defined as the mapping:

\(τ\(t\),G\(t\)\)=Aθ​\(G\(t−1\),O\(t\)\)\.\(\\tau^\{\(t\)\},G^\{\(t\)\}\)=A\_\{\\theta\}\(G^\{\(t\-1\)\},O^\{\(t\)\}\)\.\(3\)The overall problem definition is shown in the middle part of Fig\.[1](https://arxiv.org/html/2609.12345#S1.F1)\.

### 2\.1ParaRecover

As shown in Fig\.[1](https://arxiv.org/html/2609.12345#S1.F1), ParaRecover is a process\-level benchmark evaluating error localization and recovery in parallel tool\-use agents via DAG\.

#### 2\.1\.1Task Source

We source our initial tasks from BUTTON\([Chen et al\., 2024a](https://arxiv.org/html/2609.12345#bib.bib1)\), which constructs complex trajectories by introducing dependencies among diverse atomic tasks\. This compositional structure exhibits a natural correspondence with the edges and subtask nodes inherent in DAG planning\. Regarding the tool execution environment, we follow[Chen et al\. \(2024a\)](https://arxiv.org/html/2609.12345#bib.bib1)and adopt the tool agent\-based simulation paradigm\. Since our primary objective is to evaluate the model’s capacity for reflection and decision\-making amidst various intermediate error states rather than assessing the stability of external APIs, this controlled simulated environment serves as an ideal execution backend\.

#### 2\.1\.2Error Taxonomy

A core challenge is to determine what types of errors should be recovered\. Previous work has identified several common failure modes\([Zhuang et al\., 2023](https://arxiv.org/html/2609.12345#bib.bib21);[Chen et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib20)\), such as incorrect tool names, invalid arguments, and non\-executable calls\. However, these error types mainly capture local mistakes at the function\-call level and do not fully characterize the failures that emerge in multi\-turn parallel execution\. In a DAG\-based agent workflow, the model must additionally maintain valid dependency structures, identify which nodes can be executed in parallel and update the global plan after partial completion\.

Therefore, beyond inherited tool\-level errors, we extract additional error categories from large\-scale rollout traces, including parallelization errors, dependency errors, and so on\. Finally, we construct a fine\-grained error taxonomy consisting of three dimensions and 14 specific error types\. Specific error types and their descriptions are provided in Appendix[C](https://arxiv.org/html/2609.12345#A3)\. We further report the empirical distribution of these errors across models of different parameter scales in Appendix[D](https://arxiv.org/html/2609.12345#A4), showing that they are not artificial annotations but naturally occurring failure patterns in realistic agent executions\. We then use this unified error taxonomy as the basis for subsequent trajectory perturbation and error\-aware data augmentation\.

Error TypeLEVEL\-1LEVEL\-2DRSIESDRSIESStructural\-related ErrorsDependence\_Error34\.8946\.8925\.2233\.71↓3\.4%33\.71\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 3\.4\\%\}\}43\.80↓6\.6%43\.80\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 6\.6\\%\}\}24\.85↓1\.5%24\.85\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 1\.5\\%\}\}Dependence\_Null39\.3448\.1522\.2034\.61↓12\.0%34\.61\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 12\.0\\%\}\}44\.07↓8\.5%44\.07\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 8\.5\\%\}\}23\.05↑3\.8%23\.05\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 3\.8\\%\}\}Parallelism\_Error34\.1954\.2629\.7932\.53↓4\.9%32\.53\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 4\.9\\%\}\}47\.48↓12\.5%47\.48\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 12\.5\\%\}\}26\.83↓9\.9%26\.83\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 9\.9\\%\}\}Serial\_Error41\.2649\.7826\.2433\.99↓17\.6%33\.99\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 17\.6\\%\}\}48\.09↓3\.4%48\.09\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 3\.4\\%\}\}26\.26↑0\.01%26\.26\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 0\.01\\%\}\}Parameter\-related ErrorsParameter\_Error57\.3551\.9532\.8048\.82↓14\.9%48\.82\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 14\.9\\%\}\}44\.88↓13\.6%44\.88\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 13\.6\\%\}\}27\.05↓17\.5%27\.05\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 17\.5\\%\}\}Parameter\_Invalid49\.6349\.2129\.9545\.67↓8\.0%45\.67\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 8\.0\\%\}\}44\.82↓8\.9%44\.82\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 8\.9\\%\}\}28\.51↓4\.8%28\.51\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 4\.8\\%\}\}Parameter\_Missing57\.9349\.6331\.9048\.53↓16\.2%48\.53\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 16\.2\\%\}\}45\.86↓7\.6%45\.86\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 7\.6\\%\}\}29\.81↓6\.6%29\.81\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 6\.6\\%\}\}Type\_Error55\.1949\.0431\.5646\.68↓15\.4%46\.68\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 15\.4\\%\}\}47\.71↓2\.7%47\.71\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 2\.7\\%\}\}31\.18↓1\.2%31\.18\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 1\.2\\%\}\}Tool\-related ErrorsNull\_Error46\.1244\.4823\.8837\.74↓18\.2%37\.74\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 18\.2\\%\}\}37\.20↓16\.4%37\.20\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 16\.4\\%\}\}17\.14↓28\.2%17\.14\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 28\.2\\%\}\}Timeout\_Error44\.1741\.9821\.5737\.28↓15\.6%37\.28\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 15\.6\\%\}\}37\.27↓11\.2%37\.27\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 11\.2\\%\}\}16\.39↓24\.0%16\.39\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 24\.0\\%\}\}Tool\_name\_Error41\.9135\.0316\.8035\.61↓15\.0%35\.61\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 15\.0\\%\}\}34\.99↓0\.1%34\.99\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 0\.1\\%\}\}17\.72↑5\.5%17\.72\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 5\.5\\%\}\}Tool\_select\_Error42\.0752\.6628\.9840\.38↓4\.0%40\.38\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 4\.0\\%\}\}44\.64↓15\.2%44\.64\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 15\.2\\%\}\}22\.64↓21\.9%22\.64\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 21\.9\\%\}\}Toolcall\_Missing35\.8154\.3526\.6537\.11↑3\.6%37\.11\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 3\.6\\%\}\}32\.36↓40\.5%32\.36\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 40\.5\\%\}\}16\.91↓36\.6%16\.91\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 36\.6\\%\}\}Toolcall\_Redundancy31\.8552\.0127\.0627\.99↓12\.1%27\.99\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 12\.1\\%\}\}29\.79↓42\.7%29\.79\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 42\.7\\%\}\}18\.36↓32\.2%18\.36\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 32\.2\\%\}\}Table 3:Performance of GPT\-4o\-mini across different error types
#### 2\.1\.3Data Composition

Our benchmark consists of two complementary sources: manually curated real rollout trajectories with errors and augmented error trajectories constructed by injecting the 14 predefined error types into correct trajectories\. The former ensures that the benchmark reflects naturally occurring failure patterns in realistic agent rollouts, while the latter enables controlled expansion of error diversity and difficulty\.

Based on the propagation range of errors, we organize the benchmark into two difficulty levels, namely LEVEL\-1 and LEVEL\-2\. LEVEL\-1 contains both manually selected and augmented single\-round error trajectories, where the error is restricted to the immediately preceding round, either in the planning result or in the tool return\. In this setting, the error has not yet propagated across multiple rounds, and the agent is mainly required to identify and correct a recent error\.

In contrast, LEVEL\-2 is designed to capture more difficult failure cases\. In practice, early errors are often not detected immediately\. Instead, they continue to affect later execution\. Moreover, an agent may introduce new errors in different rounds, and the accumulation of errors can jointly cause the final failure\. LEVEL\-2 is intended to model such conditions: multiple co\-existing errors and cross\-round error propagation\. Specifically, from real rollout trajectories, we extract instances where errors are not immediately resolved but propagate to subsequent rounds, or where new errors are introduced later\. For augmented data, we randomly inject various error types into the downstream steps of already flawed trajectories\. So this level evaluates not only whether the agent can detect an error, but also recover under long\-horizon corrupted contexts\. Detailed data distributions are provided in Appendix[F](https://arxiv.org/html/2609.12345#A6)\.

#### 2\.1\.4The SDE Rubric

To overcome the limitations of traditional binary metrics in evaluating complex agent trajectories, we propose a fine\-grained, multi\-dimensional evaluation framework: theSDE\(Structural,Diagnostic, andEvolutionary\) Rubric\. These three sub\-dimensions correspond respectively to multi\-turn parallel execution, error localization, and replanning capabilities\.

Structural Integrity \(S\)This dimension primarily evaluates the formal correctness and executability of the generated DAG, and is divided into 5 sub\-dimensions \(S​1−S​5S1\-S5\)\.

S1: Thought\-DAG Consistency \(1/0\.5/0\): Evaluates whether the reasoning expressed in thoughtτ\\tauis perfectly mapped onto the generated DAGθ⁡\(G\)\\theta\(G\)\.

S2: Topological Dependency \(1/0\.5/0\): Assesses the correctness of node dependencies, ensuring that parallel nodes have no interdependencies and sequential nodes have valid causal links\.

S3: Tool Call Legality \(1/0\.5/0\): Verifies the validity of the invoked tools and their parameters\.

S4: State Advancement Correctness \(1/0\.5/0\): Checks whether the status attributes of all nodes are accurately updated based on the prior execution feedback\.

S5:Topological Parsability \(1/0\): Determines whether the generated DAG structure can be successfully parsed by the execution engine\.

Error\_TypeLEVEL\-1LEVEL\-2DRSIESDRSIESStructural\-related ErrorsDependence\_Error72\.5679\.2253\.3771\.71↓1\.2%71\.71\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 1\.2\\%\}\}73\.45↓7\.3%73\.45\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 7\.3\\%\}\}52\.23↓2\.1%52\.23\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 2\.1\\%\}\}Dependence\_Null74\.0276\.0950\.8772\.75↓1\.7%72\.75\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 1\.7\\%\}\}70\.88↓6\.8%70\.88\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 6\.8\\%\}\}51\.60↑1\.4%51\.60\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 1\.4\\%\}\}Parallelism\_Error71\.1281\.0656\.1269\.46↓2\.3%69\.46\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 2\.3\\%\}\}73\.14↓9\.8%73\.14\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 9\.8\\%\}\}55\.44↓1\.2%55\.44\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 1\.2\\%\}\}Serial\_Error73\.5578\.4255\.8671\.77↓2\.4%71\.77\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 2\.4\\%\}\}70\.74↓9\.8%70\.74\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 9\.8\\%\}\}54\.09↓3\.2%54\.09\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 3\.2\\%\}\}Parameter\-related ErrorsParameter\_Error84\.3267\.1154\.3980\.47↓4\.6%80\.47\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 4\.6\\%\}\}70\.08↑4\.4%70\.08\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 4\.4\\%\}\}55\.50↑2\.0%55\.50\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 2\.0\\%\}\}Parameter\_Invalid79\.7472\.1154\.2179\.03↓0\.9%79\.03\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 0\.9\\%\}\}72\.26↑0\.2%72\.26\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 0\.2\\%\}\}56\.82↑4\.8%56\.82\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 4\.8\\%\}\}Parameter\_Missing81\.8063\.9252\.5979\.84↓2\.4%79\.84\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 2\.4\\%\}\}68\.22↑6\.7%68\.22\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 6\.7\\%\}\}54\.36↑3\.4%54\.36\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 3\.4\\%\}\}Type\_Error83\.1068\.4055\.2479\.68↓4\.1%79\.68\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 4\.1\\%\}\}67\.15↓1\.8%67\.15\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 1\.8\\%\}\}56\.04↑1\.4%56\.04\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 1\.4\\%\}\}Tool\-related ErrorsNull\_Error73\.1372\.2454\.8170\.77↓3\.2%70\.77\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 3\.2\\%\}\}70\.71↓2\.1%70\.71\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 2\.1\\%\}\}51\.21↓6\.6%51\.21\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 6\.6\\%\}\}Timeout\_Error71\.5168\.0753\.2170\.74↓1\.1%70\.74\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 1\.1\\%\}\}71\.57↑5\.1%71\.57\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 5\.1\\%\}\}52\.50↓1\.3%52\.50\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 1\.3\\%\}\}Tool\_name\_Error76\.8963\.7752\.7574\.71↓2\.8%74\.71\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 2\.8\\%\}\}64\.35↑0\.9%64\.35\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 0\.9\\%\}\}52\.84↑0\.2%52\.84\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 0\.2\\%\}\}Tool\_select\_Error75\.8581\.7056\.0173\.80↓2\.7%73\.80\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 2\.7\\%\}\}72\.21↓11\.6%72\.21\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 11\.6\\%\}\}53\.74↓4\.1%53\.74\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 4\.1\\%\}\}Toolcall\_Missing74\.0381\.3458\.9872\.39↓2\.2%72\.39\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 2\.2\\%\}\}75\.86↓6\.7%75\.86\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 6\.7\\%\}\}53\.98↓8\.5%53\.98\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 8\.5\\%\}\}Toolcall\_Redundancy69\.3183\.4455\.5366\.13↓4\.6%66\.13\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 4\.6\\%\}\}77\.41↓7\.2%77\.41\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 7\.2\\%\}\}52\.84↓4\.9%52\.84\_\{\{\\color\[rgb\]\{1,0,0\}\\downarrow 4\.9\\%\}\}Table 4:Performance of DeepSeek\-V4\-flash across different error typesDiagnostic Reasoning \(D\)This dimension assesses the depth and accuracy of the agent’s failure analysis, and is divided into 5 sub\-dimensions \(D​1−D​5D1\-D5\)\.

D1: Plan Evidence Anchoring \(1/0\.5/0\): Evaluates whether the thoughtτ\\taudemonstrates a correct understanding and accurate referencing of the previous round’s planning graph\.

D2: Tool Evidence Anchoring \(1/0\.5/0\): Assesses whether the thoughtτ\\tauaccurately interprets the methods and execution results of previous tool calls\.

D3: Root Cause Localization Depth \(1/0\.5/0\): Determines if the reasoning identifies the fundamental root cause of the failure rather than merely describing surface\-level symptoms\.

D4: Impact Scope Recognition \(1/0\.5/0\): Checks if the agent successfully identifies downstream nodes and dependency chains affected by a specific error\.

D5: Evidence Sufficiency \(1/0\.5/0\): Evaluates whether the reasoning provides adequate justification while avoiding over\-inference\.

Evolutionary Strategy \(E\)This dimension measures the efficiency and precision of the agent’s replanning, and is divided into 5 sub\-dimensions \(E​1−E​5E1\-E5\)\.

E1: Modification Minimality \(1/0\.5/0\): Assesses whether the updated plan only alters necessary components without introducing irrelevant revisions\.

E2: Recovery Closure \(1/0\.5/0\): Evaluates whether the agent not only fixed the explicit point of failure, but also the previous or subsequent faulty workflows associated with it\.

E3: Goal Persistence \(1/0\.5/0\): Verifies that the new plan remains strictly aligned with the original task objective without goal drift\.

E4: Strategy Necessity \(1/0\.5/0\): Determines whether the new step has a clear and reasonable necessity\.

E5: Problem\-Solving Efficacy: Quantifies the practical utility of the updated plan when executed in a real\-world environment\. It measures whether the task can be completed within finite steps and evaluates the margin of difference from the optimal trajectory\. For each queryqq, we define the following variables:GqG\_\{q\}, which is the optimal number of rounds required by the ground truth,TqT\_\{q\}, which is the actual number of rounds required to complete the task after substituting the current plan into the subsequent workflow, andSq∈\{0,1\}S\_\{q\}\\in\\\{0,1\\\}, which is the final success status, where 1 indicates success and 0 indicates failure\. The resulting scoreS​c​o​r​e​\(E​5\)Score\(E5\)for the query is defined as:

S​c​o​r​e​\(E​5\)=Sq⋅min⁡\(1,GqTq\)\.Score\(E5\)=S\_\{q\}\\cdot\\min\\left\(1,\\frac\{G\_\{q\}\}\{T\_\{q\}\}\\right\)\.\(4\)

## 3Experiment

We comprehensively evaluate agents’ reflection and decision\-making capabilities under erroneous states on our benchmark\. Specifically, we focus on the following research questions:

RQ1: How do mainstream LLMs perform on our benchmark?

RQ2: How does agent performance vary across different types of errors?

RQ3: Is it reliable to evaluate abstract rubrics using a LLM\-as\-a\-Judge paradigm?

RQ4: Can fine\-grained evaluation metrics provide actionable guidance for improving model capabilities?

### 3\.1RQ1: How do mainstream large language models perform on our benchmark?

We benchmark six major model families: \(1\) the OpenAI models, including GPT\-4o\([Hurst et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib3)\), GPT\-4o\-mini\([Hurst et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib3)\), GPT\-5\-nano\([Singh et al\., 2026](https://arxiv.org/html/2609.12345#bib.bib4)\), and GPT\-5\.2\([OpenAI, 2025](https://arxiv.org/html/2609.12345#bib.bib5)\); \(2\) the Anthropic models, including Claude Sonnet 4\.6\([Anthropic, 2026b](https://arxiv.org/html/2609.12345#bib.bib29)\)and Claude Opus 4\.6\([Anthropic, 2026a](https://arxiv.org/html/2609.12345#bib.bib30)\); \(3\) the Google Gemini models, including Gemini\-3\.1\-Flash\([DeepMind, 2026a](https://arxiv.org/html/2609.12345#bib.bib28)\)and Gemini\-3\.1\-Pro\([DeepMind, 2026b](https://arxiv.org/html/2609.12345#bib.bib27)\); \(4\) the Qwen models, including Qwen3\-8B, Qwen3\-32B\([Yang et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib25)\), Qwen3\.5\-9B, and Qwen3\.5\-122B\-A10B\([Team, 2026](https://arxiv.org/html/2609.12345#bib.bib26)\); \(5\) the Zhipu GLM models, including GLM\-4\.5\-Air\([Zeng et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib23)\)and GLM\-4\.6\([Organization, 2025](https://arxiv.org/html/2609.12345#bib.bib24)\); and \(6\) the DeepSeek models, including DeepSeek\-V4\-Flash, DeepSeek\-V4\-Flash\*\(thinking mode\), and DeepSeek\-V4\-Pro\([DeepSeek\-AI, 2026](https://arxiv.org/html/2609.12345#bib.bib22)\)\.

Settings\.For all models, we adopt a consistent CoT prompt \(see Appendix[E](https://arxiv.org/html/2609.12345#A5)\) to guide model reasoning and decision\-making\. Evaluations are conducted under the SDE rubric, with results reported in Table[2](https://arxiv.org/html/2609.12345#S1.T2)\. In the table, SI, DR, and ES denote the average scores for Structural Integrity, Diagnostic Reasoning, and Evolutionary Strategy, respectively\. The five sub\-dimensions within each category are equally weighted\. All reported scores are normalized to the 0˜100 scale for ease of interpretation\. Meanwhile, we report the pass@1 metric, where the updated plan will be adopted in subsequent rounds\. A score of 1 is assigned upon successful task completion within specified steps \(see Appendix[H](https://arxiv.org/html/2609.12345#A8)\), and 0 otherwise\.

The results show that mainstream LLMs achieve average scores below 70 on our benchmark\. Moreover, models with larger parameter sizes substantially outperform smaller ones, indicating that our evaluation effectively captures differences in model capability\. The generally low scores also suggest that state\-of\-the\-art models still struggle with attribution and decision\-making in multi\-turn parallel tasks involving errors\. Models consistently achieve higher scores across all dimensions on LEVEL\-1 than on LEVEL\-2, demonstrating that multi\-turn errors pose a greater challenge than single\-turn errors\. Among all models, Claude Opus 4\.6 achieves the highest average scores on both levels, obtaining 69\.23 and 66\.85, respectively\. Further analysis reveals that low scores are most frequently observed in the Evolutionary Strategy dimension, which measures a model’s ability to replan effectively in response to errors\. While models tend to be relatively proficient at localizing errors, they show limited capacity for generating precise and efficient plan updates\.

DimensionEMAMAEHuman MeanLLM MeanDiagnostic ReasoningD10\.82250\.10620\.63630\.6700D20\.81250\.11000\.55500\.5725D30\.82750\.11000\.57250\.5775D40\.82500\.11620\.53000\.5688D50\.81000\.09500\.32500\.3950Evolutionary StrategyE10\.81250\.11500\.44750\.4625E20\.80250\.09880\.23500\.2463E30\.79500\.10880\.42380\.3925E40\.79250\.13380\.49630\.5100Structural IntegrityS10\.82750\.10750\.51250\.5150S20\.78750\.14250\.47500\.4725S30\.78750\.15620\.49380\.4775S40\.78500\.17000\.45380\.3963Overall0\.80670\.12080\.47360\.4813Table 5:Consistency between human annotations and LLM\-as\-a\-JudgeWe can also observe that Pass@1 of all models is above 89%, indicating that the plans generated by agents can mostly complete the tasks in subsequent rounds\. However, all models score poorly on ES \(where E5 measures the deviation from optimal execution rounds\), revealing severe inefficiencies caused by redundant calls and execution errors\. This highlights that final\-outcome metrics alone fail to capture intermediate planning quality, whereas our evaluation system successfully identifies these process\-level deficiencies\. For further details, please refer to Appendix[B](https://arxiv.org/html/2609.12345#A2)

### 3\.2RQ2: How does agent performance vary across different types of errors?

To investigate which types are difficult to localize or recover, we evaluate two representative models with contrasting parameter scales—GPT\-4o\-mini \(small\) and DeepSeek\-V4\-flash \(large\)—on tasks labeled with different error types\. Experimental results are presented in Table[3](https://arxiv.org/html/2609.12345#S2.T3)and Table[4](https://arxiv.org/html/2609.12345#S2.T4), reporting overall dataset metrics for LEVEL\-1 \(single\-turn\) and LEVEL\-2 \(multi\-turn\)\. Red/green arrows indicate the relative change of metrics on LEVEL\-2 compared to LEVEL\-1\.

Overall, multi\-turn errors are more challenging than single\-turn errors for both small and large models\. Notably, DeepSeek\-V4\-flash shows relatively small changes from LEVEL\-1 to LEVEL\-2 across most metrics, with some dimensions even showing slight improvements, whereas GPT\-4o\-mini suffers substantial drops on most dimensions\. This suggests that models with stronger reasoning capabilities can better handle multi\-turn error scenarios and remain robust\.

For both models, Toolcall\_Redundancy has the lowest scores in the DR dimension, indicating that redundant call is the hardest to identify\. Unlike other errors, it typically does not cause explicit execution failures but manifest as “plausible yet ineffective” implicit errors, making them difficult to detect—a common challenge for agents\.

ModelLEVEL\-1LEVEL\-2DeepSeekGPT\-4oDeepSeekGPT\-4oGPT\-4o51\.4552\.3144\.7445\.41Claude Opus 4\.669\.2369\.2066\.8567\.24Qwen3\-8B43\.4743\.7641\.0741\.34DeepSeek\-V4\-flash68\.0468\.4766\.3667\.67DeepSeek\-V4\-pro68\.2968\.8766\.5067\.36Table 6:Cross\-judge comparison of average SDE scores using DeepSeek\-V4\-flash thinking and GPT\-4o as judges\.Different error types impose distinct challenges\. Parameter\-related errors yield relatively high scores for two models\. However, when difficulty escalates to LEVEL\-2, GPT\-4o\-mini experiences a broad decline, while DeepSeek\-V4\-flash shows few declines and even improvements\. This indicates that large model has become insensitive to this error type, effectively localizing and correcting them in both simple and complex calls\.

Structure\-related errors and tool\-related errors pose significant challenges for both models\. Models score relatively higher on parameter\-related explicit errors\. In real\-world scenarios, explicit failures are often easy to resolve, whereas errors that accumulate implicitly—such as redundant calls, state drift, and unreasonable planning—remain difficult to handle\.

ModelLEVEL\-1LEVEL\-2SIDRESAvg\.SIDRESAvg\.Base\(Qwen3\-8B\)47\.3557\.7526\.4843\.8646\.3351\.0627\.7241\.70Few\-shot48\.7557\.8226\.8344\.4747\.6251\.2027\.9142\.24SFT49\.3558\.4227\.0544\.9448\.5351\.7327\.8542\.70ParaRecover\-DPO53\.4262\.7131\.1649\.10↑5\.24\\textbf\{49\.10\}\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 5\.24\}\}53\.0255\.2730\.9546\.41↑4\.71\\textbf\{46\.41\}\_\{\{\\color\[rgb\]\{0,1,0\}\\uparrow 4\.71\}\}Table 7:Model Performance Comparison
### 3\.3RQ3: Is it reliable to evaluate abstract rubrics using a LLM\-as\-a\-Judge paradigm?

Many dimensions in our evaluation rubric rely on abstract criteria, making manual annotation impractical\. We therefore adopt LLM\-as\-a\-Judge, leveraging the reasoning capability of large language models to automatically score these abstract dimensions\. Prior work shows that fine\-grained, low\-cardinality, and easily discernible rules substantially improve accuracy and consistency\([Hashemi et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib41);[Zhuge et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib42)\), and our three\-level \(1/0\.5/0\) scoring per dimension aligns well with this principle\.

After evaluating multiple LLMs in preliminary experiments, we selected DeepSeek\-V4\-flash thinking, which achieved the highest initial agreement with human annotations\. For each of the 13 LLM\-judged dimensions, we randomly sampled 400 instances, conducted multiple rounds of human annotation, and iteratively refined the evaluation prompt based on this standard\. Four metrics are used to measure human\-model agreement: EMA, MAE, Human Mean, and Model Mean\.

EMA \(Exact Match Accuracy\) is defined as:

EMA=1N​∑i=1N𝟏y^i=yi,\\text\{EMA\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\_\{\\hat\{y\}\_\{i\}=y\_\{i\}\},\(5\)whereNNis the total number of samples,y^i\\hat\{y\}\_\{i\}andyiy\_\{i\}are the model\-predicted and human\-annotated scores, and𝟏\\mathbf\{1\}is the indicator function\. MAE \(Mean Absolute Error\) is defined as:

MAE=1N​∑i=1N\|y^i−yi\|\.\\text\{MAE\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\|\\hat\{y\}\_\{i\}\-y\_\{i\}\\right\|\.\(6\)
As reported in Table[5](https://arxiv.org/html/2609.12345#S3.T5), the average EMA across all dimensions is 0\.8067, and the average MAE is 0\.1208, demonstrating that the LLM evaluations achieve a trustworthy level of agreement\. The evaluation prompt is provided in the Appendix[E](https://arxiv.org/html/2609.12345#A5)\.

ModelLEVEL\-1LEVEL\-2Tool Use Eff\.↑\\uparrowE5↑\\uparrowInvalid DAG Rate↓\\downarrowPass@1↑\\uparrowTool Use Eff\.↑\\uparrowE5↑\\uparrowInvalid DAG Rate↓\\downarrowPass@1↑\\uparrowBase \(Qwen3\-8B\)20\.4336\.5010\.2591\.8417\.0730\.2113\.2291\.37SFT21\.2937\.126\.7192\.4218\.4231\.649\.4192\.02ParaRecover\-DPO26\.2040\.855\.9094\.3021\.3133\.109\.0893\.97

Table 8:Judge\-free evaluation of Base, SFT, and ParaRecover\-DPO models on LEVEL\-1 and LEVEL\-2\. Higher is better for Tool Use Efficiency, E5, and Pass@1, while lower is better for Invalid DAG Rate\.To further examine the sensitivity to judge choice and potential same\-family preference, we additionally calibrate prompt of GPT\-4o as an independent judge and use it to re\-evaluate representative models\. As shown in Table[6](https://arxiv.org/html/2609.12345#S3.T6), the two judges produce highly consistent scores and overall performance patterns across model families\. The LEVEL\-1 ranking remains unchanged, while LEVEL\-2 shows only minor rank permutations among the top\-performing models\. These results suggest that our main conclusions are robust to the choice of judge and are unlikely to be driven by same\-family preference\.

### 3\.4RQ4: Can fine\-grained evaluation metrics provide actionable guidance for improving model capabilities?

To evaluate whether the proposed fine\-grained intermediate reflection and decision metrics can provide effective supervision to improve agent performance, we use Qwen3\-8B as the base model and conduct SFT and DPO\([Rafailov et al\., 2023](https://arxiv.org/html/2609.12345#bib.bib43)\), respectively\. We first randomly construct a fixed test set\. These test instances are excluded from all subsequent training data construction to avoid data leakage\. The remaining tasks are used to construct the training data\.

Few\-shot Prompting\.To examine how much capability can be elicited through prompting alone, we additionally introduce a few\-shot baseline by augmenting the original prompt with three high\-quality demonstrations\.

Supervised Fine\-Tuning\.For SFT, we collect trajectories generated by multiple models and score them using the SDE evaluation metrics\. Trajectories with an average score greater than 10 \(15 sub\-dimensions in total, with a total score of 15\) are selected as candidate high\-quality trajectories and are further manually cleaned\. This process results in 4,924 SFT training instances\.

ParaRecover\-DPO\.Our SDE evaluation framework, as a point\-wise scoring rubric, allows us to construct preferences based on score differences, thus making it compatible with preference\-learning objectives such as DPO \(and variants\)\. Based on the SFT model, we further train with DPO\. For DPO, we construct preference pairs from trajectories generated for the same task\. The trajectory with a higher total score is treated as the preferred response, while the lower\-scoring trajectory is treated as the rejected response\. To ensure that the preference signal is reliable, we require all trajectories to satisfy a minimum threshold 3 and the score gap between the preferred and rejected trajectories to be greater than 5\. After manual cleaning, we obtain 3,187 preference pairs for DPO training\.

As shown in Tab\.[7](https://arxiv.org/html/2609.12345#S3.T7), few\-shot prompting provides moderate improvements over the base model, achieving average scores of 44\.47 and 42\.24 on LEVEL\-1 and LEVEL\-2, respectively\. This suggests that high\-quality demonstrations can partially elicit better recovery behavior without parameter updates\. Also, both SFT and ParaRecover\-DPO improve the SDE scores, with ParaRecover\-DPO achieving the best performance on both LEVEL\-1 and LEVEL\-2\.

Since the training data are constructed using the SDE rubric, however, improvements under the same rubric alone may not provide fully independent evidence of enhanced recovery capability\. We therefore further evaluate the models using judge\-free, execution\-based metrics, including tool\-use efficiency, execution\-round efficiency \(E5\), invalid DAG rate, and Pass@1\. These metrics are computed directly from execution traces or environment outcomes and do not involve the LLM judge\.

As shown in Tab\.[8](https://arxiv.org/html/2609.12345#S3.T8), ParaRecover\-DPO consistently improves all judge\-free metrics across both difficulty levels\. On LEVEL\-1, tool\-use efficiency increases from 20\.43 to 26\.20 and E5 from 36\.50 to 40\.85, while the invalid DAG rate decreases from 10\.25 to 5\.90 and Pass@1 rises from 91\.84 to 94\.30\. Similar gains are also observed on LEVEL\-2\. These judge\-independent results provide further evidence that SDE\-guided preference learning improves actual execution and recovery behavior rather than merely fitting the judge’s evaluation preferences\.

## 4Conclusion

We present ParaRecover, a novel process\-level benchmark for evaluating error localization and recovery in parallel tool\-use agents\. Moving beyond previous benchmarks, our framework introduces a fine\-grained error taxonomy, challenging error trajectories, and the SDE rubric to measure agent capabilities\. Experimental results reveal that high task completion rates do not necessarily imply reliable recovery and replanning capabilities, especially under multi\-turn error propagation\. Furthermore, we demonstrate that SDE\-based supervision can effectively improve agents’ reflective recovery behaviors\.

## 5Limitation

While ParaRecover offers a controlled and fine\-grained framework for evaluating process\-level error recovery in parallel tool\-use agents, it has several limitations\. First, our benchmark is built on a simulated execution environment rather than live real\-world APIs\. Although this design ensures reproducibility and controlled error injection, it may not fully capture the unpredictability, latency, and dynamic state changes inherent in real\-world systems\. Second, our SDE rubric employs an LLM\-as\-a\-Judge paradigm to evaluate abstract reasoning dimensions\. Despite achieving relatively high agreement with human annotations, this approach may still introduce bias in complex or ambiguous cases\. Finally, ParaRecover models agent execution using DAG\-structured workflows\. While this formulation effectively captures dependency relationships in parallel multi\-turn scenarios, it does not fully represent more dynamic real\-world patterns such as loops, conditional branching, or asynchronous interactions\. Future work may further improve ParaRecover through more realistic execution environments, hybrid human\-AI evaluation protocols, and more general execution structures\.

## 6Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 62276044, and also in part by the 2025 Scientific Research Projects of the General Administration of Customs of China under Grant No\. 2025HK184 and 2025HK209\.

## References

- AnthropicClaude Opus 4\.6\.Note:[https://www\.anthropic\.com/news/claude\-opus\-4\-6](https://www.anthropic.com/news/claude-opus-4-6)Accessed: 2026\-05\-16Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- Anthropic \(2026b\)AnthropicClaude Sonnet 4\.6\.Note:[https://www\.anthropic\.com/claude/sonnet](https://www.anthropic.com/claude/sonnet)Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- Chenet al\.\(2025\)C\. Chen, X\. Hao, W\. Liu, X\. Huang, X\. Zeng, S\. Yu, D\. Li, Y\. Huang, X\. Liu, W\. Xinzhi, and W\. LiuACEBench: a comprehensive evaluation of LLM tool usage\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 12970–12998\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.697/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.697),ISBN 979\-8\-89176\-335\-7Cited by:[§2\.1\.2](https://arxiv.org/html/2609.12345#S2.SS1.SSS2.p1.1)\.
- Chenet al\.\(2024a\)M\. Chen, H\. Sun, T\. Li, F\. Yang, H\. Liang, K\. Lu, B\. Cui, W\. Zhang, Z\. Zhou, and W\. ChenFacilitating multi\-turn function calling for llms via compositional instruction tuning\.arXiv preprint arXiv:2410\.12952\.Cited by:[§2\.1\.1](https://arxiv.org/html/2609.12345#S2.SS1.SSS1.p1.1)\.
- Chenet al\.\(2024b\)Z\. Chen, W\. Du, W\. Zhang, K\. Liu, J\. Liu, M\. Zheng, J\. Zhuo, S\. Zhang, D\. Lin, K\. Chen, and F\. ZhaoT\-eval: evaluating the tool utilization capability of large language models step by step\.External Links:2312\.14033,[Link](https://arxiv.org/abs/2312.14033)Cited by:[§A\.2](https://arxiv.org/html/2609.12345#A1.SS2.p2.1)\.
- DeepMind \(2026a\)G\. DeepMindGemini 3\.1 Flash\.Note:[https://deepmind\.google/models/gemini/flash/](https://deepmind.google/models/gemini/flash/)Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- DeepMind \(2026b\)G\. DeepMindGemini 3\.1 Pro\.Note:[https://deepmind\.google/models/gemini/pro/](https://deepmind.google/models/gemini/pro/)Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-v4 collection\.Note:[https://huggingface\.co/collections/deepseek\-ai/deepseek\-v4](https://huggingface.co/collections/deepseek-ai/deepseek-v4)Accessed: 2026\-05\-16Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- Guoet al\.\(2026\)Z\. Guo, B\. Xu, C\. Zhu, W\. Hong, X\. Wang, and Z\. MaoMcp\-agentbench: evaluating real\-world language agent performance with mcp\-mediated tools\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 30888–30896\.Cited by:[§A\.2](https://arxiv.org/html/2609.12345#A1.SS2.p1.1)\.
- Hashemiet al\.\(2024\)H\. Hashemi, J\. Eisner, C\. Rosset, B\. Van Durme, and C\. KedzieLlm\-rubric: a multidimensional, calibrated approach to automated evaluation of natural language texts\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13806–13834\.Cited by:[§3\.3](https://arxiv.org/html/2609.12345#S3.SS3.p1.1)\.
- Heet al\.\(2024\)H\. He, W\. Yao, K\. Ma, W\. Yu, Y\. Dai, H\. Zhang, Z\. Lan, and D\. YuWebvoyager: building an end\-to\-end web agent with large multimodal models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 6864–6890\.Cited by:[§A\.1](https://arxiv.org/html/2609.12345#A1.SS1.p1.1)\.
- Heet al\.\(2025\)P\. He, Z\. Dai, B\. He, H\. Liu, X\. Tang, H\. Lu, J\. Li, J\. Ding, S\. Mukherjee, S\. Wang, Y\. Xing, J\. Tang, and B\. DumoulinTRAJECT\-bench:a trajectory\-aware benchmark for evaluating agentic tool use\.External Links:2510\.04550,[Link](https://arxiv.org/abs/2510.04550)Cited by:[§A\.2](https://arxiv.org/html/2609.12345#A1.SS2.p2.1),[§1](https://arxiv.org/html/2609.12345#S1.p3.1)\.
- Honget al\.\(2024\)S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, C\. Zhang, J\. Wang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin, L\. Zhou, C\. Ran, L\. Xiao, C\. Wu, and J\. SchmidhuberMetaGPT: meta programming for a multi\-agent collaborative framework\.External Links:2308\.00352,[Link](https://arxiv.org/abs/2308.00352)Cited by:[§A\.1](https://arxiv.org/html/2609.12345#A1.SS1.p1.1)\.
- Huet al\.\(2021\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.External Links:2106\.09685,[Link](https://arxiv.org/abs/2106.09685)Cited by:[Appendix G](https://arxiv.org/html/2609.12345#A7.p1.1)\.
- Huet al\.\(2024\)X\. Hu, Z\. Zhao, S\. Wei, Z\. Chai, Q\. Ma, G\. Wang, X\. Wang, J\. Su, J\. Xu, M\. Zhu, Y\. Cheng, J\. Yuan, J\. Li, K\. Kuang, Y\. Yang, H\. Yang, and F\. WuInfiAgent\-dabench: evaluating agents on data analysis tasks\.External Links:2401\.05507,[Link](https://arxiv.org/abs/2401.05507)Cited by:[§1](https://arxiv.org/html/2609.12345#S1.p1.1)\.
- Huanget al\.\(2024\)X\. Huang, W\. Liu, X\. Chen, X\. Wang, H\. Wang, D\. Lian, Y\. Wang, R\. Tang, and E\. ChenUnderstanding the planning of llm agents: a survey\.arXiv preprint arXiv:2402\.02716\.Cited by:[§1](https://arxiv.org/html/2609.12345#S1.p1.1)\.
- Hurstet al\.\(2024\)A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford, A\. Mądry, A\. Baker\-Whitcomb, A\. Beutel, A\. Borzunov, A\. Carney, A\. Chow, A\. Kirillov, A\. Nichol, A\. Paino, A\. Renzin, A\. T\. Passos, A\. Kirillov, A\. Christakis, A\. Conneau, A\. Kamali, A\. Jabri, A\. Moyer, A\. Tam, A\. Crookes, A\. Tootoochian, A\. Tootoonchian, A\. Kumar, A\. Vallone, A\. Karpathy, A\. Braunstein, A\. Cann, A\. Codispoti, A\. Galu, A\. Kondrich, A\. Tulloch, A\. Mishchenko, A\. Baek, A\. Jiang, A\. Pelisse, A\. Woodford, A\. Gosalia, A\. Dhar, A\. Pantuliano, A\. Nayak, A\. Oliver, B\. Zoph, B\. Ghorbani, B\. Leimberger, B\. Rossen, B\. Sokolowsky, B\. Wang, B\. Zweig, B\. Hoover, B\. Samic, B\. McGrew, B\. Spero, B\. Giertler, B\. Cheng, B\. Lightcap, B\. Walkin, B\. Quinn, B\. Guarraci, B\. Hsu, B\. Kellogg, B\. Eastman, C\. Lugaresi, C\. Wainwright, C\. Bassin, C\. Hudson, C\. Chu, C\. Nelson, C\. Li, C\. J\. Shern, C\. Conger, C\. Barette, C\. Voss, C\. Ding, C\. Lu, C\. Zhang, C\. Beaumont, C\. Hallacy, C\. Koch, C\. Gibson, C\. Kim, C\. Choi, C\. McLeavey, C\. Hesse, C\. Fischer, C\. Winter, C\. Czarnecki, C\. Jarvis, C\. Wei, C\. Koumouzelis, D\. Sherburn, D\. Kappler, D\. Levin, D\. Levy, D\. Carr, D\. Farhi, D\. Mely, D\. Robinson, D\. Sasaki, D\. Jin, D\. Valladares, D\. Tsipras, D\. Li, D\. P\. Nguyen, D\. Findlay, E\. Oiwoh, E\. Wong, E\. Asdar, E\. Proehl, E\. Yang, E\. Antonow, E\. Kramer, E\. Peterson, E\. Sigler, E\. Wallace, E\. Brevdo, E\. Mays, F\. Khorasani, F\. P\. Such, F\. Raso, F\. Zhang, F\. von Lohmann, F\. Sulit, G\. Goh, G\. Oden, G\. Salmon, G\. Starace, G\. Brockman, H\. Salman, H\. Bao, H\. Hu, H\. Wong, H\. Wang, H\. Schmidt, H\. Whitney, H\. Jun, H\. Kirchner, H\. P\. de Oliveira Pinto, H\. Ren, H\. Chang, H\. W\. Chung, I\. Kivlichan, I\. O’Connell, I\. O’Connell, I\. Osband, I\. Silber, I\. Sohl, I\. Okuyucu, I\. Lan, I\. Kostrikov, I\. Sutskever, I\. Kanitscheider, I\. Gulrajani, J\. Coxon, J\. Menick, J\. Pachocki, J\. Aung, J\. Betker, J\. Crooks, J\. Lennon, J\. Kiros, J\. Leike, J\. Park, J\. Kwon, J\. Phang, J\. Teplitz, J\. Wei, J\. Wolfe, J\. Chen, J\. Harris, J\. Varavva, J\. G\. Lee, J\. Shieh, J\. Lin, J\. Yu, J\. Weng, J\. Tang, J\. Yu, J\. Jang, J\. Q\. Candela, J\. Beutler, J\. Landers, J\. Parish, J\. Heidecke, J\. Schulman, J\. Lachman, J\. McKay, J\. Uesato, J\. Ward, J\. W\. Kim, J\. Huizinga, J\. Sitkin, J\. Kraaijeveld, J\. Gross, J\. Kaplan, J\. Snyder, J\. Achiam, J\. Jiao, J\. Lee, J\. Zhuang, J\. Harriman, K\. Fricke, K\. Hayashi, K\. Singhal, K\. Shi, K\. Karthik, K\. Wood, K\. Rimbach, K\. Hsu, K\. Nguyen, K\. Gu\-Lemberg, K\. Button, K\. Liu, K\. Howe, K\. Muthukumar, K\. Luther, L\. Ahmad, L\. Kai, L\. Itow, L\. Workman, L\. Pathak, L\. Chen, L\. Jing, L\. Guy, L\. Fedus, L\. Zhou, L\. Mamitsuka, L\. Weng, L\. McCallum, L\. Held, L\. Ouyang, L\. Feuvrier, L\. Zhang, L\. Kondraciuk, L\. Kaiser, L\. Hewitt, L\. Metz, L\. Doshi, M\. Aflak, M\. Simens, M\. Boyd, M\. Thompson, M\. Dukhan, M\. Chen, M\. Gray, M\. Hudnall, M\. Zhang, M\. Aljubeh, M\. Litwin, M\. Zeng, M\. Johnson, M\. Shetty, M\. Gupta, M\. Shah, M\. Yatbaz, M\. J\. Yang, M\. Zhong, M\. Glaese, M\. Chen, M\. Janner, M\. Lampe, M\. Petrov, M\. Wu, M\. Wang, M\. Fradin, M\. Pokrass, M\. Castro, M\. O\. T\. de Castro, M\. Pavlov, M\. Brundage, M\. Wang, M\. Khan, M\. Murati, M\. Bavarian, M\. Lin, M\. Yesildal, N\. Soto, N\. Gimelshein, N\. Cone, N\. Staudacher, N\. Summers, N\. LaFontaine, N\. Chowdhury, N\. Ryder, N\. Stathas, N\. Turley, N\. Tezak, N\. Felix, N\. Kudige, N\. Keskar, N\. Deutsch, N\. Bundick, N\. Puckett, O\. Nachum, O\. Okelola, O\. Boiko, O\. Murk, O\. Jaffe, O\. Watkins, O\. Godement, O\. Campbell\-Moore, P\. Chao, P\. McMillan, P\. Belov, P\. Su, P\. Bak, P\. Bakkum, P\. Deng, P\. Dolan, P\. Hoeschele, P\. Welinder, P\. Tillet, P\. Pronin, P\. Tillet, P\. Dhariwal, Q\. Yuan, R\. Dias, R\. Lim, R\. Arora, R\. Troll, R\. Lin, R\. G\. Lopes, R\. Puri, R\. Miyara, R\. Leike, R\. Gaubert, R\. Zamani, R\. Wang, R\. Donnelly, R\. Honsby, R\. Smith, R\. Sahai, R\. Ramchandani, R\. Huet, R\. Carmichael, R\. Zellers, R\. Chen, R\. Chen, R\. Nigmatullin, R\. Cheu, S\. Jain, S\. Altman, S\. Schoenholz, S\. Toizer, S\. Miserendino, S\. Agarwal, S\. Culver, S\. Ethersmith, S\. Gray, S\. Grove, S\. Metzger, S\. Hermani, S\. Jain, S\. Zhao, S\. Wu, S\. Jomoto, S\. Wu, Shuaiqi, Xia, S\. Phene, S\. Papay, S\. Narayanan, S\. Coffey, S\. Lee, S\. Hall, S\. Balaji, T\. Broda, T\. Stramer, T\. Xu, T\. Gogineni, T\. Christianson, T\. Sanders, T\. Patwardhan, T\. Cunninghman, T\. Degry, T\. Dimson, T\. Raoux, T\. Shadwell, T\. Zheng, T\. Underwood, T\. Markov, T\. Sherbakov, T\. Rubin, T\. Stasi, T\. Kaftan, T\. Heywood, T\. Peterson, T\. Walters, T\. Eloundou, V\. Qi, V\. Moeller, V\. Monaco, V\. Kuo, V\. Fomenko, W\. Chang, W\. Zheng, W\. Zhou, W\. Manassra, W\. Sheu, W\. Zaremba, Y\. Patil, Y\. Qian, Y\. Kim, Y\. Cheng, Y\. Zhang, Y\. He, Y\. Zhang, Y\. Jin, Y\. Dai, and Y\. MalkovGPT\-4o system card\.External Links:2410\.21276,[Link](https://arxiv.org/abs/2410.21276)Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- Liet al\.\(2023\)M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. LiApi\-bank: a comprehensive benchmark for tool\-augmented llms\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 3102–3116\.Cited by:[§A\.2](https://arxiv.org/html/2609.12345#A1.SS2.p1.1),[§1](https://arxiv.org/html/2609.12345#S1.p3.1)\.
- Liuet al\.\(2025\)X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. TangAgentBench: evaluating llms as agents\.External Links:2308\.03688,[Link](https://arxiv.org/abs/2308.03688)Cited by:[§1](https://arxiv.org/html/2609.12345#S1.p2.1)\.
- Luet al\.\(2025\)J\. Lu, T\. Holleis, Y\. Zhang, B\. Aumayer, F\. Nan, F\. Bai, S\. Ma, S\. Ma, M\. Li, G\. Yin, Z\. Wang, and R\. PangToolSandbox: a stateful, conversational, interactive evaluation benchmark for llm tool use capabilities\.External Links:2408\.04682,[Link](https://arxiv.org/abs/2408.04682)Cited by:[§1](https://arxiv.org/html/2609.12345#S1.p3.1)\.
- Lùet al\.\(2025\)X\. H\. Lù, A\. Kazemnejad, N\. Meade, A\. Patel, D\. Shin, A\. Zambrano, K\. Stańczak, P\. Shaw, C\. J\. Pal, and S\. ReddyAgentrewardbench: evaluating automatic evaluations of web agent trajectories\.arXiv preprint arXiv:2504\.08942\.Cited by:[§A\.2](https://arxiv.org/html/2609.12345#A1.SS2.p2.1)\.
- Maet al\.\(2024\)Y\. Ma, Z\. Gou, J\. Hao, R\. Xu, S\. Wang, L\. Pan, Y\. Yang, Y\. Cao, and A\. SunSciagent: tool\-augmented language models for scientific reasoning\.InProceedings of the 2024 conference on empirical methods in natural language processing,pp\. 15701–15736\.Cited by:[§1](https://arxiv.org/html/2609.12345#S1.p1.1)\.
- Mialonet al\.\(2024\)G\. Mialon, C\. Fourrier, T\. Wolf, Y\. LeCun, and T\. ScialomGaia: a benchmark for general ai assistants\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9025–9049\.Cited by:[§A\.2](https://arxiv.org/html/2609.12345#A1.SS2.p1.1),[§1](https://arxiv.org/html/2609.12345#S1.p2.1)\.
- OpenAI \(2025\)OpenAIIntroducing gpt\-5\.2\.Note:[https://openai\.com/zh\-Hans\-CN/index/introducing\-gpt\-5\-2/](https://openai.com/zh-Hans-CN/index/introducing-gpt-5-2/)Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- Organization \(2025\)Z\. OrganizationGLM\-4\.6\.Note:[https://huggingface\.co/zai\-org/GLM\-4\.6](https://huggingface.co/zai-org/GLM-4.6)Accessed: 2026\-05\-16Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- Patilet al\.\(2025\)S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. GonzalezThe berkeley function calling leaderboard \(bfcl\): from tool use to agentic evaluation of large language models\.InForty\-second International Conference on Machine Learning,Cited by:[§A\.2](https://arxiv.org/html/2609.12345#A1.SS2.p1.1),[§1](https://arxiv.org/html/2609.12345#S1.p3.1)\.
- Qinet al\.\(2023\)Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian, S\. Zhao, L\. Hong, R\. Tian, R\. Xie, J\. Zhou, M\. Gerstein, D\. Li, Z\. Liu, and M\. SunToolLLM: facilitating large language models to master 16000\+ real\-world apis\.External Links:2307\.16789,[Link](https://arxiv.org/abs/2307.16789)Cited by:[§A\.1](https://arxiv.org/html/2609.12345#A1.SS1.p1.1),[§A\.2](https://arxiv.org/html/2609.12345#A1.SS2.p1.1),[§1](https://arxiv.org/html/2609.12345#S1.p1.1),[§1](https://arxiv.org/html/2609.12345#S1.p3.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§3\.4](https://arxiv.org/html/2609.12345#S3.SS4.p1.1)\.
- Renet al\.\(2025\)S\. Ren, C\. Xie, P\. Jian, Z\. Ren, C\. Leng, and J\. ZhangTowards scientific intelligence: a survey of llm\-based scientific agents\.arXiv preprint arXiv:2503\.24047\.Cited by:[§A\.1](https://arxiv.org/html/2609.12345#A1.SS1.p1.1)\.
- Schicket al\.\(2023\)T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. ScialomToolformer: language models can teach themselves to use tools\.Advances in neural information processing systems36,pp\. 68539–68551\.Cited by:[§A\.1](https://arxiv.org/html/2609.12345#A1.SS1.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§A\.1](https://arxiv.org/html/2609.12345#A1.SS1.p1.1)\.
- Singhet al\.\(2026\)A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram, A\. Nathan, A\. Luo, A\. Helyar, A\. Madry, A\. Efremov, A\. Spyra, A\. Baker\-Whitcomb, A\. Beutel, A\. Karpenko, A\. Makelov, A\. Neitz, A\. Wei, A\. Barr, A\. Kirchmeyer, A\. Ivanov, A\. Christakis, A\. Gillespie, A\. Tam, A\. Bennett, A\. Wan, A\. Huang, A\. M\. Sandjideh, A\. Yang, A\. Kumar, A\. Saraiva, A\. Vallone, A\. Gheorghe, A\. G\. Garcia, A\. Braunstein, A\. Liu, A\. Schmidt, A\. Mereskin, A\. Mishchenko, A\. Applebaum, A\. Rogerson, A\. Rajan, A\. Wei, A\. Kotha, A\. Srivastava, A\. Agrawal, A\. Vijayvergiya, A\. Tyra, A\. Nair, A\. Nayak, B\. Eggers, B\. Ji, B\. Hoover, B\. Chen, B\. Chen, B\. Barak, B\. Minaiev, B\. Hao, B\. Baker, B\. Lightcap, B\. McKinzie, B\. Wang, B\. Quinn, B\. Fioca, B\. Hsu, B\. Yang, B\. Yu, B\. Zhang, B\. Brenner, C\. R\. Zetino, C\. Raymond, C\. Lugaresi, C\. Paz, C\. Hudson, C\. Whitney, C\. Li, C\. Chen, C\. Cole, C\. Voss, C\. Ding, C\. Shen, C\. Huang, C\. Colby, C\. Hallacy, C\. Koch, C\. Lu, C\. Kaplan, C\. Kim, C\. Minott\-Henriques, C\. Frey, C\. Yu, C\. Czarnecki, C\. Reid, C\. Wei, C\. Decareaux, C\. Scheau, C\. Zhang, C\. Forbes, D\. Tang, D\. Goldberg, D\. Roberts, D\. Palmie, D\. Kappler, D\. Levine, D\. Wright, D\. Leo, D\. Lin, D\. Robinson, D\. Grabb, D\. Chen, D\. Lim, D\. Salama, D\. Bhattacharjee, D\. Tsipras, D\. Li, D\. Yu, D\. Strouse, D\. Williams, D\. Hunn, E\. Bayes, E\. Arbus, E\. Akyurek, E\. Y\. Le, E\. Widmann, E\. Yani, E\. Proehl, E\. Sert, E\. Cheung, E\. Schwartz, E\. Han, E\. Jiang, E\. Mitchell, E\. Sigler, E\. Wallace, E\. Ritter, E\. Kavanaugh, E\. Mays, E\. Nikishin, F\. Li, F\. P\. Such, F\. de Avila Belbute Peres, F\. Raso, F\. Bekerman, F\. Tsimpourlas, F\. Chantzis, F\. Song, F\. Zhang, G\. Raila, G\. McGrath, G\. Briggs, G\. Yang, G\. Parascandolo, G\. Chabot, G\. Kim, G\. Zhao, G\. Valiant, G\. Leclerc, H\. Salman, H\. Wang, H\. Sheng, H\. Jiang, H\. Wang, H\. Jin, H\. Sikchi, H\. Schmidt, H\. Aspegren, H\. Chen, H\. Qiu, H\. Lightman, I\. Covert, I\. Kivlichan, I\. Silber, I\. Sohl, I\. Hammoud, I\. Clavera, I\. Lan, I\. Akkaya, I\. Kostrikov, I\. Kofman, I\. Etinger, I\. Singal, J\. Hehir, J\. Huh, J\. Pan, J\. Wilczynski, J\. Pachocki, J\. Lee, J\. Quinn, J\. Kiros, J\. Kalra, J\. Samaroo, J\. Wang, J\. Wolfe, J\. Chen, J\. Wang, J\. Harb, J\. Han, J\. Wang, J\. Zhao, J\. Chen, J\. Yang, J\. Tworek, J\. Chand, J\. Landon, J\. Liang, J\. Lin, J\. Liu, J\. Wang, J\. Tang, J\. Yin, J\. Jang, J\. Morris, J\. Flynn, J\. Ferstad, J\. Heidecke, J\. Fishbein, J\. Hallman, J\. Grant, J\. Chien, J\. Gordon, J\. Park, J\. Liss, J\. Kraaijeveld, J\. Guay, J\. Mo, J\. Lawson, J\. McGrath, J\. Vendrow, J\. Jiao, J\. Lee, J\. Steele, J\. Wang, J\. Mao, K\. Chen, K\. Hayashi, K\. Xiao, K\. Salahi, K\. Wu, K\. Sekhri, K\. Sharma, K\. Singhal, K\. Li, K\. Nguyen, K\. Gu\-Lemberg, K\. King, K\. Liu, K\. Stone, K\. Yu, K\. Ying, K\. Georgiev, K\. Lim, K\. Tirumala, K\. Miller, L\. Ahmad, L\. Lv, L\. Clare, L\. Fauconnet, L\. Itow, L\. Yang, L\. Romaniuk, L\. Anise, L\. Byron, L\. Pathak, L\. Maksin, L\. Lo, L\. Ho, L\. Jing, L\. Wu, L\. Xiong, L\. Mamitsuka, L\. Yang, L\. McCallum, L\. Held, L\. Bourgeois, L\. Engstrom, L\. Kuhn, L\. Feuvrier, L\. Zhang, L\. Switzer, L\. Kondraciuk, L\. Kaiser, M\. Joglekar, M\. Singh, M\. Shah, M\. Stratta, M\. Williams, M\. Chen, M\. Sun, M\. Cayton, M\. Li, M\. Zhang, M\. Aljubeh, M\. Nichols, M\. Haines, M\. Schwarzer, M\. Gupta, M\. Shah, M\. Y\. Guan, M\. Huang, M\. Dong, M\. Wang, M\. Glaese, M\. Carroll, M\. Lampe, M\. Malek, M\. Sharman, M\. Zhang, M\. Wang, M\. Pokrass, M\. Florian, M\. Pavlov, M\. Wang, M\. Chen, M\. Wang, M\. Feng, M\. Bavarian, M\. Lin, M\. Abdool, M\. Rohaninejad, N\. Soto, N\. Staudacher, N\. LaFontaine, N\. Marwell, N\. Liu, N\. Preston, N\. Turley, N\. Ansman, N\. Blades, N\. Pancha, N\. Mikhaylin, N\. Felix, N\. Handa, N\. Rai, N\. Keskar, N\. Brown, O\. Nachum, O\. Boiko, O\. Murk, O\. Watkins, O\. Gleeson, P\. Mishkin, P\. Lesiewicz, P\. Baltescu, P\. Belov, P\. Zhokhov, P\. Pronin, P\. Guo, P\. Thacker, Q\. Liu, Q\. Yuan, Q\. Liu, R\. Dias, R\. Puckett, R\. Arora, R\. T\. Mullapudi, R\. Gaon, R\. Miyara, R\. Song, R\. Aggarwal, R\. Marsan, R\. Yemiru, R\. Xiong, R\. Kshirsagar, R\. Nuttall, R\. Tsiupa, R\. Eldan, R\. Wang, R\. James, R\. Ziv, R\. Shu, R\. Nigmatullin, S\. Jain, S\. Talaie, S\. Altman, S\. Arnesen, S\. Toizer, S\. Toyer, S\. Miserendino, S\. Agarwal, S\. Yoo, S\. Heon, S\. Ethersmith, S\. Grove, S\. Taylor, S\. Bubeck, S\. Banesiu, S\. Amdo, S\. Zhao, S\. Wu, S\. Santurkar, S\. Zhao, S\. R\. Chaudhuri, S\. Krishnaswamy, Shuaiqi, Xia, S\. Cheng, S\. Anadkat, S\. P\. Fishman, S\. Tobin, S\. Fu, S\. Jain, S\. Mei, S\. Egoian, S\. Kim, S\. Golden, S\. Mah, S\. Lin, S\. Imm, S\. Sharpe, S\. Yadlowsky, S\. Choudhry, S\. Eum, S\. Sanjeev, T\. Khan, T\. Stramer, T\. Wang, T\. Xin, T\. Gogineni, T\. Christianson, T\. Sanders, T\. Patwardhan, T\. Degry, T\. Shadwell, T\. Fu, T\. Gao, T\. Garipov, T\. Sriskandarajah, T\. Sherbakov, T\. Korbak, T\. Kaftan, T\. Hiratsuka, T\. Wang, T\. Song, T\. Zhao, T\. Peterson, V\. Kharitonov, V\. Chernova, V\. Kosaraju, V\. Kuo, V\. Pong, V\. Verma, V\. Petrov, W\. Jiang, W\. Zhang, W\. Zhou, W\. Xie, W\. Zhan, W\. McCabe, W\. DePue, W\. Ellsworth, W\. Bain, W\. Thompson, X\. Chen, X\. Qi, X\. Xiang, X\. Shi, Y\. Dubois, Y\. Yu, Y\. Khakbaz, Y\. Wu, Y\. Qian, Y\. T\. Lee, Y\. Chen, Y\. Zhang, Y\. Xiong, Y\. Tian, Y\. Cha, Y\. Bai, Y\. Yang, Y\. Yuan, Y\. Li, Y\. Zhang, Y\. Yang, Y\. Jin, Y\. Jiang, Y\. Wang, Y\. Wang, Y\. Liu, Z\. Stubenvoll, Z\. Dou, Z\. Wu, and Z\. WangOpenAI gpt\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- Team \(2026\)Q\. TeamQwen3\.5: Towards Native Multimodal Agents\.Note:[https://qwen\.ai/blog?id=qwen3\.5](https://qwen.ai/blog?id=qwen3.5)Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- Wanget al\.\(2026\)J\. Wang, Y\. Hu, W\. Yang, Z\. Pan, X\. Li, and L\. GuoAligning agents via planning: a benchmark for trajectory\-level reward modeling\.arXiv preprint arXiv:2604\.08178\.Cited by:[§A\.2](https://arxiv.org/html/2609.12345#A1.SS2.p2.1)\.
- Wanget al\.\(2024\)L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin, W\. X\. Zhao, Z\. Wei, and J\. WenA survey on large language model based autonomous agents\.Frontiers of Computer Science18\(6\)\.External Links:ISSN 2095\-2236,[Link](http://dx.doi.org/10.1007/s11704-024-40231-1),[Document](https://dx.doi.org/10.1007/s11704-024-40231-1)Cited by:[§1](https://arxiv.org/html/2609.12345#S1.p1.1)\.
- Wanget al\.\(2025\)Z\. Wang, Q\. Chang, H\. Patel, S\. Biju, C\. Wu, Q\. Liu, A\. Ding, A\. Rezazadeh, A\. Shah, Y\. Bao, and E\. SiowMCP\-bench: benchmarking tool\-using llm agents with complex real\-world tasks via mcp servers\.External Links:2508\.20453,[Link](https://arxiv.org/abs/2508.20453)Cited by:[§A\.2](https://arxiv.org/html/2609.12345#A1.SS2.p1.1)\.
- Weiet al\.\(2023\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.External Links:2201\.11903,[Link](https://arxiv.org/abs/2201.11903)Cited by:[§2](https://arxiv.org/html/2609.12345#S2.p4.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- Yanget al\.\(2024\)J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. PressSwe\-agent: agent\-computer interfaces enable automated software engineering\.Advances in Neural Information Processing Systems37,pp\. 50528–50652\.Cited by:[§A\.1](https://arxiv.org/html/2609.12345#A1.SS1.p1.1),[§1](https://arxiv.org/html/2609.12345#S1.p1.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[§1](https://arxiv.org/html/2609.12345#S1.p2.1)\.
- Yaoet al\.\(2022\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReact: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§A\.1](https://arxiv.org/html/2609.12345#A1.SS1.p1.1),[§1](https://arxiv.org/html/2609.12345#S1.p1.1)\.
- Zenget al\.\(2025\)A\. Zeng, X\. Lv, Q\. Zheng, Z\. Hou, B\. Chen, C\. Xie, C\. Wang, D\. Yin, H\. Zeng, J\. Zhang, K\. Wang, L\. Zhong, M\. Liu, R\. Lu, S\. Cao, X\. Zhang, X\. Huang, Y\. Wei, Y\. Cheng, Y\. An, Y\. Niu, Y\. Wen, Y\. Bai, Z\. Du, Z\. Wang, Z\. Zhu, B\. Zhang, B\. Wen, B\. Wu, B\. Xu, C\. Huang, C\. Zhao, C\. Cai, C\. Yu, C\. Li, C\. Ge, C\. Huang, C\. Zhang, C\. Xu, C\. Zhu, C\. Li, C\. Yin, D\. Lin, D\. Yang, D\. Jiang, D\. Ai, E\. Zhu, F\. Wang, G\. Pan, G\. Wang, H\. Sun, H\. Li, H\. Li, H\. Hu, H\. Zhang, H\. Peng, H\. Tai, H\. Zhang, H\. Wang, H\. Yang, H\. Liu, H\. Zhao, H\. Liu, H\. Yan, H\. Liu, H\. Chen, J\. Li, J\. Zhao, J\. Ren, J\. Jiao, J\. Zhao, J\. Yan, J\. Wang, J\. Gui, J\. Zhao, J\. Liu, J\. Li, J\. Li, J\. Lu, J\. Wang, J\. Yuan, J\. Li, J\. Du, J\. Du, J\. Liu, J\. Zhi, J\. Gao, K\. Wang, L\. Yang, L\. Xu, L\. Fan, L\. Wu, L\. Ding, L\. Wang, M\. Zhang, M\. Li, M\. Xu, M\. Zhao, M\. Zhai, P\. Du, Q\. Dong, S\. Lei, S\. Tu, S\. Yang, S\. Lu, S\. Li, S\. Li, Shuang\-Li, S\. Yang, S\. Yi, T\. Yu, W\. Tian, W\. Wang, W\. Yu, W\. L\. Tam, W\. Liang, W\. Liu, X\. Wang, X\. Jia, X\. Gu, X\. Ling, X\. Wang, X\. Fan, X\. Pan, X\. Zhang, X\. Zhang, X\. Fu, X\. Zhang, Y\. Xu, Y\. Wu, Y\. Lu, Y\. Wang, Y\. Zhou, Y\. Pan, Y\. Zhang, Y\. Wang, Y\. Li, Y\. Su, Y\. Geng, Y\. Zhu, Y\. Yang, Y\. Li, Y\. Wu, Y\. Li, Y\. Liu, Y\. Wang, Y\. Li, Y\. Zhang, Z\. Liu, Z\. Yang, Z\. Zhou, Z\. Qiao, Z\. Feng, Z\. Liu, Z\. Zhang, Z\. Wang, Z\. Yao, Z\. Wang, Z\. Liu, Z\. Chai, Z\. Li, Z\. Zhao, W\. Chen, J\. Zhai, B\. Xu, M\. Huang, H\. Wang, J\. Li, Y\. Dong, and J\. TangGLM\-4\.5: agentic, reasoning, and coding \(arc\) foundation models\.External Links:2508\.06471,[Link](https://arxiv.org/abs/2508.06471)Cited by:[§3\.1](https://arxiv.org/html/2609.12345#S3.SS1.p1.1)\.
- Zhouet al\.\(2024\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. NeubigWebArena: a realistic web environment for building autonomous agents\.External Links:2307\.13854,[Link](https://arxiv.org/abs/2307.13854)Cited by:[§A\.1](https://arxiv.org/html/2609.12345#A1.SS1.p1.1)\.
- Zhuanget al\.\(2023\)Y\. Zhuang, Y\. Yu, K\. Wang, H\. Sun, and C\. ZhangToolqa: a dataset for llm question answering with external tools\.Advances in Neural Information Processing Systems36,pp\. 50117–50143\.Cited by:[§2\.1\.2](https://arxiv.org/html/2609.12345#S2.SS1.SSS2.p1.1)\.
- Zhugeet al\.\(2024\)M\. Zhuge, C\. Zhao, D\. Ashley, W\. Wang, D\. Khizbullin, Y\. Xiong, Z\. Liu, E\. Chang, R\. Krishnamoorthi, Y\. Tian, Y\. Shi, V\. Chandra, and J\. SchmidhuberAgent\-as\-a\-judge: evaluate agents with agents\.External Links:2410\.10934,[Link](https://arxiv.org/abs/2410.10934)Cited by:[§3\.3](https://arxiv.org/html/2609.12345#S3.SS3.p1.1)\.

## Appendix ARelated Work

### A\.1LLM Agents

LLM agents extend language models from text generators to goal\-driven systems that can plan, act and interact with external environments\([Yao et al\., 2022](https://arxiv.org/html/2609.12345#bib.bib14);[Shinn et al\., 2023](https://arxiv.org/html/2609.12345#bib.bib34)\)\. An important direction is tool\-augmented agents, where LLMs learn to invoke external search engines, calculators or domain\-specific tools to overcome the limitations of parametric knowledge\([Schick et al\., 2023](https://arxiv.org/html/2609.12345#bib.bib35);[Qin et al\., 2023](https://arxiv.org/html/2609.12345#bib.bib7)\)\. These design have made agents effective and flexible across domains\. For example, research agents automate literature collection, scientific experiments, and paper writing\([Ren et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib36)\); Code agents iteratively edit and debug in complex projects\([Yang et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib37);[Hong et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib38)\); Web agents interact with browsers and operating systems to complete complex tasks\([Zhou et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib39);[He et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib40)\)\. These scenarios involve complex tool calls and long\-range decision\-making, therefore establishing high\-quality benchmarks is crucial for expanding the capabilities of LLM agents\.

### A\.2Agent Tool\-use and Trajectory Evaluation\.

Existing tool\-use benchmarks mainly evaluate tool selection, parameter filling, and task completion\. API\-Bank and Tool\-Bench evaluate API retrieval, and execution over tool\-use tasks\([Li et al\., 2023](https://arxiv.org/html/2609.12345#bib.bib6);[Qin et al\., 2023](https://arxiv.org/html/2609.12345#bib.bib7)\), while BFCL focuses on function\-call correctness across real\-world APIs, including serial and parallel function calls\([Patil et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib8)\)\. More recent benchmarks such as GAIA, MCP\-Bench and MCP\-AgentBench extend evaluation to multi\-step tool use, cross\-tool coordination, protocol\-mediated environments, and realistic agent workflows\([Mialon et al\., 2024](https://arxiv.org/html/2609.12345#bib.bib15);[Wang et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib9);[Guo et al\., 2026](https://arxiv.org/html/2609.12345#bib.bib10)\)\. However, these benchmarks are still mainly outcome\-oriented or call\-level oriented, making it difficult to diagnose where an agent fails within a long trajectory\.

Recent trajectory\-aware benchmarks attempt to provide more fine\-grained evaluation\. T\-Eval decomposes tool utilization into instruction following, planning, reasoning, retrieval, understanding, and review\([Chen et al\., 2024b](https://arxiv.org/html/2609.12345#bib.bib18)\)\. TRAJECT\-Bench evaluates whether tools are selected, parameterized, and ordered correctly along trajectories\([He et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib31)\)\. Reward\-oriented benchmarks such as AgentRewardBench and Plan\-RewardBench evaluate agent trajectory preferences\([Lù et al\., 2025](https://arxiv.org/html/2609.12345#bib.bib32);[Wang et al\., 2026](https://arxiv.org/html/2609.12345#bib.bib33)\)\. Although these work moves beyond final\-answer accuracy, they mainly focus on step correctness, tool\-use correctness or trajectory\-level preference judgment\.

As shown in Table[1](https://arxiv.org/html/2609.12345#S1.T1), our work differs by targeting error\-aware trajectory evaluation in parallel tool\-calling scenarios\. Instead of only asking whether a tool call is correct, whether the final answer is correct, or which trajectory is preferred, our benchmark evaluates whether an agent can inspect flawed intermediate trajectories, localize erroneous steps, and decide how to recover and replan\. This setting complements existing tool\-use and trajectory\-evaluation benchmarks by focusing on the agent’s reflection and recovery decision ability under multi\-turn parallel tool\-use failures\.

ModelLEVEL\-1LEVEL\-2E5Pass@1E5Pass@1GPT\-4o42\.1992\.2140\.2691\.50GPT\-4o\-mini36\.1789\.3230\.0588\.84Claude Opus 4\.660\.6194\.7857\.5094\.05Gemini 3\.1 pro52\.8193\.6849\.3293\.22Qwen 3 8B36\.5091\.8430\.2191\.37Qwen 3 32B39\.9492\.0834\.1091\.86DeepSeek\-V4\-flash57\.4294\.1355\.6893\.17Table 9:E5 score and pass@1 performance of different models on LEVEL\-1 and LEVEL\-2

## Appendix BAnalysis of the Discrepancy Between Pass@1 and ES Scores

Table[9](https://arxiv.org/html/2609.12345#A1.T9)presents the disparity between E5 scores and Pass@1 performance for several representative models across different task levels\. We observe that while Pass@1 metrics remain consistently high \(over 88%\) across all models, E5 scores are generally low \(the scores are shown after normalization to 0˜100\), with significant performance gaps among models of different parameter sizes\. Since the E5 metric quantifies the deviation of actual execution rounds from ground\-truth rounds, the contrasting trends of these two metrics indicate that although models can complete our benchmark tasks within a limited number of rounds \(see Appendix[H](https://arxiv.org/html/2609.12345#A8)\), their execution processes are plagued by erroneous planning and redundant tool calls, leading to severe inefficiency and high costs\.

The case study in Fig\.[2](https://arxiv.org/html/2609.12345#A2.F2)illustrates this phenomenon\. In this relatively simple task, Node 3 returns an empty response due to an incorrect parameter type\. However, instead of performing valid error attribution and retrying the call, the agent misattributes the failure to an upstream task\. Consequently, it creates Node 7 to retry the entire workflow, triggering a cascade of subsequent errors\. The actual execution takes 18 rounds, compared to the ground\-truth requirement of 6 rounds, resulting in an E5 score of 0\.33\. Such cases are prevalent, demonstrating that agents struggle with effective error attribution in multi\-turn parallel scenarios, which frequently generates redundant steps\. This reveals the limitations of relying solely on final task completion rates, underscoring the critical importance of evaluating agents’ error localization and replanning capabilities when handling flawed trajectories\.

![Refer to caption](https://arxiv.org/html/2609.12345v1/CaseStudy.png)Figure 2:Case study
## Appendix CError Description

Table[12](https://arxiv.org/html/2609.12345#A7.T12)shows the 14 error types and their corresponding descriptions\.

## Appendix DError Type Distribution

Figure 3:Comparative analysis of error type occurrences across GPT\-4o\-mini, GPT\-4o, and GPT\-5\.1\. The horizontal bars represent the frequency of each error category on a logarithmic scale\.As shown in Fig\.[3](https://arxiv.org/html/2609.12345#A4.F3), we analyze the error distributions of three different models during real\-world execution\. The most frequent errors encountered are incorrect parameter types, invalid parameter values, redundant tool calls, and missing critical tool calls\. Notably, models with larger parameter counts and stronger reasoning capabilities consistently exhibit lower error rates compared to smaller models\. This distribution also demonstrates that all error categories within our taxonomy occur in real\-world agent execution scenarios\.

## Appendix ERelated Prompts

In this section, we present prompts used across various stages of our experiments\. Specifically, these prompts comprise the DAG\-based CoT agent execution planning prompt, the tool agent prompt used to simulate tool execution results in our controlled environment, and the LLM\-as\-a\-Judge evaluation prompts across the three dimensions of the SDE rubric\.

Prompt for AgentYou are an expert at leveraging various tools to solve problems\. You focus on observing the \[Front\-Wheel Plan\] and \[Front\-Wheel Tool Call Results\], and then make appropriate reflections and plans based on the \[Original User Query\] and \[Available Tools List\]\. \#\# Your task: 1\. Analyze the user’s original query, combine it with the DAG list of the previous plan and the results of the previous tool calls, and consider what needs to be done to complete the task\. 2\. Based on the tools that still need to be called, update the current complete task list to guide the entire subsequent process\. \#\# Output requirements \- First, output the <thought\> section\. Include reflections on the original user task and perform a requirement analysis: first, identify which subtasks have already been completed toward answering the user’s question and which subtasks still need to be completed\. Then, provide a detailed sequence of steps required to accomplish the original task, along with a sound justification for the rationale behind each step\. In your reasoning, include a detailed analysis of the reasonableness of the previous round’s planning, as well as a detailed analysis of the reasonableness of the previous round’s tool invocation arrangements \(including the tools used and their parameters\)\. \- Second, output a parseable JSON representing the task chain list of tool calls\. 1\. Clearly specify the dependency relationships for each subtask\. A dependency exists only if the return result of a preceding task is required to supplement or complete the parameters of a subsequent tool call\. For any subtask that does not require preceding information, its dependency should be set to "root"\. 2\. For subtask nodes that have not yet been executed, you are free to modify, add, or delete them as needed to better invoke tools and complete the task\. Note that subtask nodes which have already been executed must not be altered\. 3\. The first subtask node has been determined as the "root" node, with the subtask set to: \{"id": "root", \# Root node, representing the starting point of the task\. "desc": \{query\}, \# Original query content provided by the user\. "dep": \[\], \# The root node does not need to depend on other nodes\. "status": True \# The root node has already been executed by default\.\} The last subtask node has been determined as the "response" node, with the subtask set to: \{"id": "response", \# The final node indicates the end point of the task\. "desc": "Analyze and summarize, and then respond\.", \# Summarize and analyze the results of all subtasks, and then respond\. "dep": \[…\], \# All nodes that are not dependent on by other subtasks\. "status": False \# The last response node has not been executed by default\.\} \#\#The output format is as follows \(strictly output the reasoning section and the task list JSON\. Do not include comments\. Please output a compact JSON format, removing all unnecessary line breaks and indentation spaces, while preserving necessary spaces within string content\): <thought\>Output your thoughts on the user’s current query; you need to re\-analyze and rethink it\. Conduct a requirements analysis, first analyzing what you have already accomplished to complete the user’s original query, and what tools are still needed\. Then, detail the steps required to answer the user’s query, and reasonably explain the rationale for each step\. Your thinking should include a detailed analysis of the rationality of the previous planning, as well as the rationality of the tool usage arrangement \(including tools used and parameters\)\. Note that your thinking should not exceed 500 words\.</thought\>\{"tasks": \[ \# A list of subtasks, where each element is a tool call subtask node, containing the task ID, the name of the called tool, the task description, and dependencies\. All subtask nodes must be provided \(regardless of whether they have been executed\), including the root node and the response subtask node\. \{"id": string, \# A unique identifier for a subtask, which is identified by a number "n", starting from "1"\. "name": string, \# The name of the tool called in the subtask\. "para": \[string\], \# The parameters used in the tool call are in the format \{"parameter1 name":"parameter1 value","parameter2 name":"parameter2 value", ……\}\. "desc": string, \# Description of task nodes "dep": \[string\], \# The subtask dependencies, where each element in the list is the id of the dependent subtask\. "status": bool \# This indicates whether the subtask has been executed\. True means it has been executed, and False means it has not been executed\.\}, ……\]\}\#\#Original User Query \{query\} \#\#Available Tools List \{tool\_list\} \#\#Front\-Wheel Plan \{plan\_dag\} \#\#Front\-Wheel Tool Call Results \{tool\_call\_dag\}

Prompt for Tool AgentYou are an expert skilled in simulating agent tool invocation\. You need to generate simulated execution results based on the User Task and Available Tool List\. \#\#Your Task Carefully identify the pending executable nodes in the previous planned DAG graph, namely nodes marked False with all dependency requirements satisfied\. Verify the validity of tool invocations in the DAG graph\. Check whether tool names, parameter names and parameter types comply with standard specifications defined in the Available Tool List\. Return reasonable error prompts immediately if any irregularities are detected\. If the invocation is valid, generate plausible returned results corresponding to the description and assigned tool of each task node\. All simulated results shall be logical and consistent with real\-world scenarios\. Deliver accurate calculation outputs for computational tools\. Return reasonable summarized search contents for retrieval tools\. Realistic fabricated values associated with the given task are acceptable\. \#\#DAG Graph Description The previous plan is presented as a DAG graph starting from the root node and ending at the response node\. DAG structure format: \{"tasks": \[\{"id": string, "name": string, "para": \{\}, "desc": string, "dep": \[string\], "status": bool\}, …\]\} Nodes ready for execution are unexecuted nodes with False status, whose all dependent predecessor nodes have been fully completed with True status\. Output Format First write your reasoning process within <thought\></thought\>\.Then output tool responses in standard JSON array format: ”’json \[ \{ "name": string, \# Consistent with the name field in DAG graph "arguments": \{"Parameter1": "Value1"\}, \# Tool calling parameters "results": string \# Simulated tool execution feedback \}, … \] \#\#User Task \{query\} \#\#Available Tool List \{tool\_list\} \#\#Previous Planned DAG Graph \{replan\_dag\}

Prompt for Structural Integrity Evaluation\#\#Role You are a highly strict, conservative, and fine\-grained evaluation expert for replanner decision\-making\. \#\#General Requirements You must evaluate whether the given replanner’s thought and replan DAG are truly high\-quality\.You must default to strict scoring rather than moderate or high scores by default\. \#\#Evaluation Dimension: Structural Integrity Strictly evaluate the following 4 sub\-items judged by LLM from the perspective of plan structure and executability consistency\.An additional script\-based judgment item will be automatically supplemented in the program: \#\#S1 Thought\-DAG Consistency: Whether the actions, repairs, and progress claimed in the thought are truly reflected in the replan DAG\. 1: Key repair actions, downstream processing, and DAG nodes in the thought correspond one\-to\-one\. 0\.5: Partially consistent, but with omissions, misalignments, or claims only in the thought without implementation in the DAG\. 0: Clearly inconsistent\. S2 Topological Dependency & Overall Executability: Whether node dependencies, execution order, and upstream/downstream relationships are correct, and whether the overall structure forms a real executable DAG\. 1: Correct and reasonable dependencies with strong overall executability\. 0\.5: Minor dependency issues or execution risks exist, but the main chain is generally understandable\. 0: Obvious dependency errors or overall difficulty in execution\. S3 Tool Call Legitimacy: Whether tool names, parameter names, and parameter value formats strictly comply with the constraints in the tool list\. 1: All tool names, parameter names, and parameter value formats are legal and accurate\. 0\.5: Minor non\-standard formats or potential risks exist, but not completely unusable\. 0: Invalid tools, wrong parameter names, missing key parameters, or obviously illegal values\. S4 State Progression Correctness: Whether the statuses of executed nodes, unexecuted nodes, retained nodes after failure, and response nodes are reasonable\. 1: State annotations are consistent with the actual execution status\. 0\.5: Minor non\-standard state annotations\. 0: Obvious state errors that will mislead the executor\. \#\#Output Format Output ONLY a JSON object, NO extra content: ”’json \{ "S1\_score": "1 / 0\.5 / 0", "S1\_reason": "Reason for this score", "S2\_score": "1 / 0\.5 / 0", "S2\_reason": "Reason for this score", "S3\_score": "1 / 0\.5 / 0", "S3\_reason": "Reason for this score", "S4\_score": "1 / 0\.5 / 0", "S4\_reason": "Reason for this score", \} ”’ \#\#Input \#\#Original User Task \{query\} \#\#Available Tool List \{tool\_list\} \#\#Previous Plan \{prev\_plan\} \#\#Execution Results & Tool Responses of Previous Plan \{tool\_result\} \#\#Output of ReplannerThought \{thought\} \#\#Updated Plan \{new\_plan\}

Prompt for Diagnostic Reasoning Evaluation\#\#Role You are a highly strict, conservative, and fine\-grained evaluation expert for replanner decision\-making\. \#\#General Requirements You must evaluate whether the given replanner’s thought and replan DAG are truly high\-quality\.You must default to strict scoring rather than moderate or high scores by default\. \#\#Evaluation Dimension: Diagnostic Reasoning Strictly assess the following five sub\-indicators from the perspective of diagnostic quality\. D1 Plan Evidence AnchoringWhether the thought explicitly references and accurately understands specific nodes, dependencies, statuses or steps in the previous plan\. 1: Clearly cites verifiable details such as node IDs, upstream and downstream nodes and status values, and conducts problem analysis based on valid evidence\. 0\.5: Contains partial references but fails to specify verifiable node IDs, dependencies or statuses, or lacks precise supporting evidence\. 0: Barely relies on plan evidence and only provides general descriptions\. D2 Tool Evidence AnchoringWhether the thought explicitly references and correctly interprets specific error messages, return values, parameter names or tool behaviors from tool results\. 1: Precisely specifies exact tool names, parameter names, error fields or return values, and explains their implications\. 0\.5: Detects existing tool abnormalities but fails to reference specific parameters or error details, or presents incomplete evidence\. 0: Makes no reference to concrete tool outputs or contains obvious misinterpretation\. D3 Root Cause Localization AccuracyWhether the actual fundamental cause is identified instead of merely describing superficial symptoms\. 1: Points out failure manifestations and accurately locates essential causes including invalid parameters, faulty dependencies and improper state transition\. 0\.5: Roughly pinpoints problematic sections but stays at superficial descriptions such as failed execution and parameter adjustment demand\. 0: Delivers incorrect root cause judgment or fails to identify underlying reasons\. D4 Impact Scope RecognitionWhether subsequent nodes, dependency chains, state transitions and response generation affected by errors are recognized\. 1: Clearly names at least one affected downstream node, relevant dependency chain or abnormal response progress\. 0\.5: Acknowledges potential impacts but fails to specify affected nodes or covers incomplete influence scope\. 0: Fails to identify any affected scope\. D5 Evidence Sufficiency and RestraintWhether reasoning is sufficiently supported by facts without fabricating non\-existent information\. 1: Draws conclusions based on at least two independent verifiable evidence sources such as plan and tool results; actively detects remaining flaws and risks; distinguishes authentic root causes from irrelevant misleading explanations\. 0\.5: Partially meets requirements but shows noticeable deficiencies in evidence adequacy, false cause elimination or residual risk detection\. 0: Lacks solid supporting grounds, contains obvious speculation and fabrication, or lacks rigorous verification awareness\. \#\#Output Format Output only a single JSON object with no extra content\. ”’json \{ "D1\_score": "1 / 0\.5 / 0", "D1\_reason": "State the reason for the score", "D2\_score": "1 / 0\.5 / 0", "D2\_reason": "State the reason for the score", "D3\_score": "1 / 0\.5 / 0", "D3\_reason": "State the reason for the score", "D4\_score": "1 / 0\.5 / 0", "D4\_reason": "State the reason for the score", "D5\_score": "1 / 0\.5 / 0", "D5\_reason": "State the reason for the score", \} ”’ \#\#Input \#\#Original User Task \{query\} \#\#Available Tool List \{tool\_list\} \#\#Previous Plan \{prev\_plan\} \#\#Execution Results & Tool Responses of Previous Plan \{tool\_result\} \#\#Output of ReplannerThought \{thought\} \#\#Updated Plan \{new\_plan\}

Prompt for Diagnostic Evolutionary Strategy\#\#Role You are a highly strict, conservative, and fine\-grained evaluation expert for replanner decision\-making\. \#\#General Requirements You must evaluate whether the given replanner’s thought and replan DAG are truly high\-quality\.You must default to strict scoring rather than moderate or high scores by default\. \#\#Evaluation Dimension: Evolutionary Strategy Strictly assess the following five sub\-items in terms of the quality of revision and improvement strategies\. E1 Minimal ModificationWhether the replan only revises necessary parts without irrelevant rewriting\. 1: Only essential nodes and dependencies are adjusted, and changes directly correspond to specific errors\. 0\.5: Revisions follow a general correct direction yet remain superficial, with redundant or excessive adjustments lacking sufficient justification\. 0: Uncontrolled modifications made or key issues left unaddressed\. E2 Closed\-loop RevisionWhether the revision fixes current errors and complements relevant downstream nodes, parameters, statuses and response chains\. 1: A complete closed loop is formed with explicit handling of affected downstream nodes, parameters and statuses\. 0\.5: Partial errors get fixed, yet gaps remain in downstream links, status transition and response chains\. 0: No executable closed loop established\. E3 Goal ConsistencyWhether the updated plan still faithfully fulfills the original user task without deviation, incomplete delivery or missing core objectives\. 1: Original goals fully retained with complete pathways reserved for final response generation\. 0\.5: Basic goals maintained but partial coverage missing or minor deviation exists, and complete final response loop cannot be guaranteed\. 0: Noticeable deviation or omission of core objectives\. E4 Strategy RationalityWhether added, removed or reordered steps and dependencies possess clear practical necessity\. 1: All strategic adjustments are fully justified and necessary\. 0\.5: Most adjustments are reasonable while a few changes lack adequate necessity\. 0: Critical adjustments serve no practical purpose and appear as mechanical rewriting\.\#\#Output Format Output only a single JSON object with no extra content\. ”’json \{ "E1\_score": "1 / 0\.5 / 0", "E1\_reason": "State the reason for the score", "E2\_score": "1 / 0\.5 / 0", "E2\_reason": "State the reason for the score", "E3\_score": "1 / 0\.5 / 0", "E3\_reason": "State the reason for the score", "E4\_score": "1 / 0\.5 / 0", "E4\_reason": "State the reason for the score", \} \#\#Input \#\#Original User Task \{query\} \#\#Available Tool List \{tool\_list\} \#\#Previous Plan \{prev\_plan\} \#\#Execution Results & Tool Responses of Previous Plan \{tool\_result\} \#\#Output of ReplannerThought \{thought\} \#\#Updated Plan \{new\_plan\}

## Appendix FData Distribution

Fig\.[4](https://arxiv.org/html/2609.12345#A6.F4)illustrates the data distribution of our constructed benchmark\. The benchmark comprises a total of 10,626 instances categorized into two levels: LEVEL\-1 and LEVEL\-2\. Specifically, LEVEL\-1 contains 4,033 instances, including 1,421 error instances collected from real\-world rollouts and 2,612 instances generated via synthetic error augmentation on correct trajectories\. LEVEL\-2 consists of 6,593 instances, comprising 1,391 error instances from real\-world rollouts and 5,202 instances derived through error augmentation on correct trajectories\. Table[10](https://arxiv.org/html/2609.12345#A6.T10)and Table[11](https://arxiv.org/html/2609.12345#A6.T11)present the data distribution across different error types within the LEVEL\-1 and LEVEL\-2 tasks of ParaRecover\. Specifically, LEVEL\-1 tasks contain only single\-round errors, whereas LEVEL\-2 tasks encompass multi\-turn error types\.

![Refer to caption](https://arxiv.org/html/2609.12345v1/number.png)Figure 4:Data distributionError TypeFrequencyRatio\(%\)Parameter\_Missing3333338\.268\.26Parameter\_Invalid3413418\.468\.46Parameter\_Error2762766\.846\.84Type\_Error3223227\.987\.98Tool\_name\_Error2582586\.406\.40Tool\_select\_Error2662666\.606\.60Toolcall\_Missing3073077\.617\.61Toolcall\_Redundancy2962967\.347\.34Dependence\_Error2782786\.896\.89Dependence\_Null2522526\.256\.25Serial\_Error40740710\.0910\.09Parallelism\_Error2682686\.656\.65Null\_Error2102105\.205\.20Timeout\_Error2192195\.435\.43Total40334033100\.00100\.00Table 10:Error type statistics in LEVEL\-1Error TypeRound1Ratio\(%\)Round2Ratio\(%\)Round3Ratio\(%\)Parameter\_Missing4296\.514436\.721987\.20Parameter\_Invalid5288\.015448\.252047\.42Parameter\_Error4286\.494166\.311876\.80Type\_Error5989\.075538\.391977\.16Tool\_name\_Error5548\.404857\.361816\.58Tool\_select\_Error4356\.603705\.611776\.44Toolcall\_Missing5588\.465548\.402037\.38Toolcall\_Redundancy5408\.195308\.042067\.49Dependence\_Error4496\.814837\.332087\.56Dependence\_Null3955\.994176\.322037\.38Serial\_Error5067\.674947\.491916\.95Parallelism\_Error3535\.353976\.022147\.78Null\_Error3945\.984566\.921896\.87Timeout\_Error4266\.464516\.841916\.95Total6593100\.006593100\.002750100\.00Table 11:Error type statistics in LEVEL\-2
## Appendix GTraining Configuration

All experiments were conducted on a single server equipped with 8 NVIDIA RTX 3090 GPUs\. We adopted Low\-Rank Adaptation \(LoRA\)\([Hu et al\., 2021](https://arxiv.org/html/2609.12345#bib.bib44)\)for both the SFT and DPO stages\. The model used is Qwen3\-8B\. The LoRA hyperparameters were uniformly set to a rank ofr=32r=32, a scaling factor ofα=64\\alpha=64, and a dropout rate of0\.050\.05\. For the SFT stage, we trained the base model for 3 epochs with a maximum sequence length of 4096 tokens to accommodate multi\-turn agent execution trajectories\. We employed a cosine learning rate scheduler with a peak learning rate of1×10−41\\times 10^\{\-4\}and a 3% warmup ratio\. The global batch size was set to 64, achieved through a per\-device batch size of 2 and 4 gradient accumulation steps\. For the DPO stage, we initialized the policy and reference models with the resulting SFT weights\. To prevent overfitting on the preference data, DPO training was limited to 1 epoch\. The learning rate was reduced to5×10−65\\times 10^\{\-6\}with a cosine scheduler and a 10% warmup ratio\. The KL penalty coefficient was set toβ=0\.1\\beta=0\.1, and the global batch size was maintained at 64\.

Error TypeDescriptionDependence\_ErrorDependency Relation Error: In the unfinished tasks of the updated DAG generated by the agent, some dependencies between nodes are incorrectly specified\. For example, node 3 should depend on node 2, but is mistakenly represented as depending on node 1\.Dependence\_NullMissing Dependency Error: In the unfinished tasks of the updated DAG generated by the agent, some nodes have missing dependencies\.Parallelism\_ErrorIncorrect Parallelization: In the unfinished tasks of the updated DAG generated by the agent, some nodes are incorrectly arranged in parallel\. Specifically, two nodes that should be executed sequentially are treated as parallel\. For example, node 2 requires the execution result of node 1 as its input parameter, but the agent schedules them to run in parallel\.Serial\_ErrorIncorrect Serialization: In the unfinished tasks of the updated DAG generated by the agent, some nodes are incorrectly arranged sequentially\. Specifically, two nodes have no dependency on each other and can be executed in parallel, but they are mistakenly represented as having a sequential dependency in the updated DAG\.Parameter\_ErrorParameter Name Error: In the unfinished tasks of the updated DAG generated by the agent, some tool\-call nodes contain invalid parameters that are not defined in the current tool’s parameter list\. For example, the supported parameter in the parameter list isb​e​g​i​n​\_​d​a​t​ebegin\\\_date, but the tool\-call node incorrectly providesb​e​g​i​n−d​a​t​ebegin\-date\.Parameter\_InvalidInvalid Parameter Value: In the unfinished tasks of the updated DAG generated by the agent, some tool\-call nodes assign parameter values with the correct type but invalid content\. For example, the tool requires ad​a​t​edateparameter, but the call provides an invalid value such as "2026/02/32"\.Parameter\_MissingMissing Required Parameter: Each tool is associated with a parameter list and a set of required parameters\. In the unfinished tasks of the updated DAG generated by the agent, some tool\-call nodes fail to provide required parameters\.Type\_ErrorParameter Type Error: Each parameter in a tool’s parameter list has a predefined value type, such as "int" or "string"\. In the unfinished tasks of the updated DAG generated by the agent, some tool\-call nodes assign values with incorrect types\. For example, a parameter requires the integer value1616, but the node provides the string value\`​\`​16​"\`\`16"\.Null\_ErrorEmpty Tool Response Error: During tool execution, the tool returns an empty result for some reason\.Timeout\_ErrorTimeout Error: During tool execution, the tool call fails with a timeout error due to issues such as network instability\.Tool\_name\_ErrorInvalid Tool Name Error: In the unfinished tasks of the updated DAG generated by the agent, some tool\-call nodes invoke an invalid tool name that is not included in the current tool list\.Tool\_select\_ErrorIncorrect Tool Selection: Some tools in the tool list have similar or easily confused functions\. In the unfinished tasks of the updated DAG generated by the agent, some tool\-call nodes select an inappropriate tool for the intended operation\.Toolcall\_MissingMissing Required Tool Call: In the updated plan DAG generated by the agent, some tool calls required to complete the task are missing\.Toolcall\_RedundancyTool call redundancy: In the unfinished tasks of the updated DAG generated by the agent, there are some unnecessary tool calls\.Table 12:Error Types and Descriptions
## Appendix HMaximum Number of Execution Rounds

In our experiments, an agent is considered to have successfully completed a task if it reaches the response node within the maximum number of execution rounds,rr\. We setr=40r=40because preliminary experiments show that trajectories exceeding 40 rounds typically indicate either an inherent bias in the agent’s task comprehension or an infinite loop\. Consequently, such tasks are highly unlikely to be completed even if additional rounds are allowed\.