AgentOrch: How Close Can Global Orchestration Come to Step-by-Step Interaction?

# 1Introduction
Source: [https://arxiv.org/html/2610.33375](https://arxiv.org/html/2610.33375)
marginparsep has been altered\. topmargin has been altered\. marginparpush has been altered\.

The page layout violates the ICML style\.

Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you\.

We’re not able to reliably undo arbitrary changes to the style\. Please remove the offending package\(s\), or layout\-changing commands and try again\.

AgentOrch: How Close Can Global Orchestration Come to Step\-by\-Step Interaction?Zijie Liu1,🖂\{\}^\{1,\\textrm\{\\Letter\}\},Qi Yang Huang2,Jinhao Duan1,Yueh Che Lee2,Thomas A\. Joseph2,Zayne Turner2,Adam Seligman2,Tianlong Chen1, andZhaozhuo Xu21University of North Carolina at Chapel Hill2WorkatoEnterprise workflow agents typically rely on step\-by\-step interaction \(Step\-by\-Step\): the agent executes an action, observes its result, and invokes a large language model \(LLM\) again to decide the next step\. Global orchestration \(Global\) instead generates a complete executable workflow in advance and executes it without further LLM calls, substantially reducing inference overhead, but it remains less accurate thanStep\-by\-Step\. We study how closeGlobalcan come toStep\-by\-Stepand what narrows the remaining gap\. We introduceAgentOrch, a benchmark that directly compares global orchestration with step\-by\-step interaction on 650 enterprise SOPs and 3,220 runtime variants under matched tools, environments, and evaluation\.Step\-by\-Stepremains more accurate for every evaluated model, but the gap has shrunk by about 79% from early models to the strongest recent model\. We identify three routes to further narrowing this gap\. Incremental workflow construction improvesGlobalaccuracy by an average of 9\.98 percentage points \(pp\) without runtime execution feedback\. Revising complete workflows with execution feedback improves all evaluated models, including 6\.13 pp on held\-out runtime variants\. Targeted training on complete workflows improves accuracy by an average of 12\.18 pp across all five trained models\. Although a gap remains, global orchestration offers a promising path toward enterprise agents that deliver higher accuracy with fewer LLM calls\.

00footnotetext:🖂\{\}^\{\\textrm\{\\Letter\}\}Zijie Liu:[jesseliu@cs\.unc\.edu](mailto:[email protected])\. Codebase:[github\.com/workato\-devs/AgentOrch](https://github.com/workato-devs/AgentOrch)\.### 1Introduction

Enterprise workflows often span multiple software tools and services, with later operations depending on the outputs of earlier ones\([Li et al\., 2026](https://arxiv.org/html/2610.33375#bib.bib15);[Wang et al\., 2026](https://arxiv.org/html/2610.33375#bib.bib43);[Lu et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib21);[Trivedi et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib40);[Xie et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib44);[Boisvert et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib1);[Drouin et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib6)\)\. Completing these workflows requires an LLM agent to select tools\([Qin et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib25);[Song et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib36);[Shen et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib30);[Schick et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib28)\)and coordinate their execution order, data dependencies, and control flow\([Liu et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib20);[Nath et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib22);[Li et al\., 2026](https://arxiv.org/html/2610.33375#bib.bib15)\)\. We refer to this capability as*workflow orchestration*\.

Existing work commonly realizes workflow orchestration through step\-by\-step interaction \(Step\-by\-Step\): the model selects an action, observes its execution result, and then determines the next action\([Shi et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib33);[Yao et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib47);[Zhou et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib53);[Liu et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib20)\)\. This interaction allows subsequent decisions to incorporate execution feedback\([Liu et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib20);[Shinn et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib34)\), but requires repeated LLM inference throughout execution\. Yet dependence on intermediate results does not inherently require the model itself to make a new decision after every action\. Data dependencies, branching, loops, and other control flow can instead be specified before execution, allowing intermediate results to be consumed by a preconstructed workflow\([Liang et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib19);[Singh et al\., 2022](https://arxiv.org/html/2610.33375#bib.bib35);[Surís et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib38)\)\. We refer to this paradigm as*global orchestration*\(Global\): the LLM constructs a complete executable workflow before observing concrete runtime inputs, after which the workflow executes without further LLM calls\. As shown in Figure[1](https://arxiv.org/html/2610.33375#S1.F1), across the three models in the cost panels,Step\-by\-Stepmakes 23\.2 times as many LLM calls and generates 3\.64 times as many completion tokens asGlobal, on average\. As LLMs become more capable at planning and tool execution\([Kim et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib11);[Wang et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib42);[Zhou et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib52)\), it becomes increasingly important to understand how much orchestration can be completed before execution\. Prior work has explored generating executable workflows or plans before execution\([Fan et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib7);[Qiao et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib24);[Zhang et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib50)\), but typically evaluates them in isolation rather than against step\-by\-step interaction on the same workflows, tools, execution environment, and criteria\. We therefore ask:

Question:How close can Global Orchestration come to Step\-by\-Step Interaction?

We first make an initial comparison betweenGlobalandStep\-by\-Stepacross model generations, shown in Figure[1](https://arxiv.org/html/2610.33375#S1.F1)\.Step\-by\-Stepachieves higher Orchestration Accuracy for every evaluated model, but the magnitude of this advantage changes substantially across generations\. From the earlier baselines through the 2024 and 2025 models, both settings improve without a consistent reduction in the gap\. The 2026 models show clearer convergence, with gaps of 15\.12 percentage points \(pp\) for GPT\-5\.6 Sol Thinking and 8\.53 pp for Claude Opus 5 Thinking, compared with 40\.67 pp for GPT\-3\.5 Turbo, a reduction of about 79%\. Thus, global orchestration has become substantially stronger and the gap narrows in the latest model generation, butStep\-by\-Stepcontinues to retain an accuracy advantage across all evaluated models\.

![Refer to caption](https://arxiv.org/html/2610.33375v1/teaser_image.png)Figure 1:Global orchestration reduces inference overhead but remains less accurate than step\-by\-step interaction\.Left:Step\-by\-Stepuses on average 23\.2×\\timesmore LLM calls per SOP and 3\.64×\\timesmore completion tokens per runtime variant thanGlobal\.Right:Step\-by\-Stepachieves higher Orchestration Accuracy across all evaluated models, while the gap narrows to approximately 8 percentage points for Claude Opus 5T\.Then to systematically study this gap, we introduceAgentOrch, a benchmark grounded in 650 enterprise standard operating procedures \(SOPs\) and 3,220 runtime variants\. Each SOP is associated with an executable reference workflow and multiple runtime variants, across which the same generated workflow is evaluated\. Importantly,AgentOrchevaluatesGlobalandStep\-by\-Stepon the same SOPs, tool interfaces, execution environment, and evaluation criteria, enabling a direct and controlled comparison between the two orchestration protocols\.

UsingAgentOrch, we first examine what determines how closeGlobalcan come toStep\-by\-Step\. Increasing model scale does not consistently narrow the gap, while newer model generations generally improve global orchestration and reasoning produces mixed effects\. We then identify three routes to narrowing the gap, each drawing on progressively more signal\. Incremental workflow construction improvesGlobalOrchestration Accuracy by an average of 9\.98 pp even without access to runtime execution feedback\. We next revise theGlobalworkflow using feedback from its own executions, without adding LLM calls at execution time\. This improves Orchestration Accuracy for all eight evaluated models by an average of 6\.18 pp, and by 6\.13 pp on held\-out runtime variants never used for feedback\. Finally, we study whether stronger global orchestration can be internalized into model weights\. We construct a separate SOP\-grounded training set from 3,000 source recipes, with each sample pairing a task specification with a complete reference workflow\. Targeted training improves Orchestration Accuracy by an average of 12\.18 pp \(6\.84–19\.88 pp\) across all five trained models\. Together, these results show that global orchestration can be substantially improved through incremental workflow construction, execution\-feedback revision, and targeted training, whileStep\-by\-Stepcontinues to retain an accuracy advantage across the evaluated models\. Our contributions are summarized as follows:

- •Orchestration Benchmark\.We introduceAgentOrch, an execution\-grounded benchmark comparingGlobalandStep\-by\-Stepon 650 enterprise SOPs and 3,220 runtime variants under shared tools, environments, and evaluation criteria\.
- •Test\-time Orchestration Improvement\.We evaluate incremental workflow construction and execution\-feedback revision as routes to strongerGlobalorchestration\. Construction improves several models without runtime execution feedback; revision improves all eight evaluated models, including on held\-out variants\.
- •Training for Global Orchestration\.We train models on 3,000 separate source recipes to generate complete workflows\. Taking the better of SFT and RL for each base model, Orchestration Accuracy improves by an average of 12\.18 pp across all five trained models\.

![Refer to caption](https://arxiv.org/html/2610.33375v1/main_figure.png)Figure 2:Overview of theAgentOrchbenchmark construction and model evaluation pipeline\.Left:Reference recipes define the benchmark SOPs\. We extract their tool operations, map them to a fixed tool bank, construct task specifications and runtime variants, and validate the resulting workflow logic and fields\. Reference executions provide the behavior used for evaluation\.Right:UnderGlobal, the model generates one complete workflow before execution and reuses it across runtime variants\. UnderStep\-by\-Step, the model selects tool calls sequentially using intermediate execution results\. Both protocols are executed in the same simulated environment and evaluated against the corresponding reference behavior\.
### 2Related Work

Interactive software\-use benchmarks\.A broad line of work evaluates language agents through step\-by\-step interaction with software environments\. WebShop\([Yao et al\., 2022](https://arxiv.org/html/2610.33375#bib.bib46)\), Mind2Web\([Deng et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib5)\), WebArena\([Zhou et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib53)\), and VisualWebArena\([Koh et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib12)\)study realistic websites, while WorkArena\([Drouin et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib6)\)and WorkArena\+\+\([Boisvert et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib1)\)target enterprise knowledge work\. OSWorld\([Xie et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib44)\), AndroidWorld\([Rawles et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib27)\), AgentBench\([Liu et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib20)\), SWE\-bench\([Jimenez et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib10)\), and TheAgentCompany\([Xu et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib45)\)extend evaluation to desktop, mobile, coding, and simulated workplace settings\. These benchmarks generally allow agents to repeatedly act on and adapt to execution feedback\. Our work instead asks how close global orchestration can come to step\-by\-step interaction by directly evaluating both paradigms on the same enterprise workflows\.

Tool\-use and API\-agent benchmarks\.API\-Bank\([Li et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib16)\), ToolLLM\([Qin et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib25)\), Gorilla\([Patil et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib23)\), ToolAlpaca\([Tang et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib39)\), RestGPT\([Song et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib36)\), StableToolBench\([Guo et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib8)\), TaskBench\([Shen et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib31)\), UltraTool\([Huang et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib9)\),τ\\tau\-bench\([Yao et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib48)\), and WorkBench\([Styles et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib37)\)evaluate tool retrieval, action selection, argument prediction, planning, multi\-step execution, and user\-agent interaction\. While these capabilities are central to enterprise orchestration, they are typically evaluated under a fixed execution protocol\. In contrast,AgentOrchevaluatesGlobalandStep\-by\-Stepunder matched tasks, tools, execution environments, and evaluation criteria, enabling a direct comparison of the two orchestration paradigms\.

Summary\.Table[1](https://arxiv.org/html/2610.33375#S2.T1)summarizes these differences\. Existing benchmarks typically evaluate either step\-by\-step interaction or complete workflow generation, whereasAgentOrchsupports both under the same executable setting with reference workflows, state grading, and side\-effect evaluation\. This enables direct comparison betweenGlobalandStep\-by\-Step\. Additional related work is discussed in Appendix[C](https://arxiv.org/html/2610.33375#A3)\.

Table 1:Comparison of evaluation settings across agent and workflow benchmarks\.E2E Exec\.indicates whether generated actions or workflows are executed end\-to\-end in an environment\.Step\-by\-Stepindicates whether the model can make new orchestration decisions using intermediate execution observations\.Globalindicates whether the model can generate a complete workflow before execution\.Reference Workflowindicates whether executable reference workflow structures are available for evaluation\.State Gradingevaluates the resulting environment state\.Side\-Effect Checkevaluates external effects produced during execution\. ✓ = yes,△\\triangle= partial, and ✗ = no\.BenchmarkDomainInterfaceExecutionGlobal OrchestrationOutcome EvaluationE2E Exec\.Step\-by\-StepGlobalReference WorkflowState GradingSide\-Effect CheckWebArena\([Zhou et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib53)\)Web / GUIBrowser✓✓✗✗✓△\\triangleWorkArena\+\+\([Boisvert et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib1)\)Enterprise / GUIBrowser / GUI✓✓✗✗✓△\\triangleOSWorld\([Xie et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib44)\)OS / GUIDesktop GUI✓✓✗✗△\\triangle△\\triangleAppWorld\([Trivedi et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib40)\)API / ToolAPIs✓✓✗✗✓✓τ\\tau\-bench\([Yao et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib48)\)API / ToolTools \+ User✓✓✗✗✓✓ToolBench\([Qin et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib25)\)API / ToolAPIs△\\triangle✓✗✗△\\triangle✗AutomationBench\([Shepard & Salimans, 2026](https://arxiv.org/html/2610.33375#bib.bib32)\)Workflow AgentSaaS APIs✓✓△\\triangle△\\triangle✓✓WorFBench\([Qiao et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib24)\)Workflow GenerationWorkflow Graph✗✗✓✓✗✗WorkflowBench\([Fan et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib7)\)Workflow GenerationAPIs \+ Workflow△\\triangle△\\triangle✓✓△\\triangle✗T\-Eval\([Chen et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib4)\)General PlanningText / Tools△\\triangle△\\triangle✓△\\triangle✗✗AgentOrchEnterprise WorkflowiPaaS Simulator✓✓✓✓✓✓

### 3AgentOrch: Benchmarking Enterprise Workflow Orchestration

AgentOrchis designed to quantify the gap between global and step\-by\-step orchestration and to study the factors that determine whether this gap can be narrowed\. As illustrated in Figure[2](https://arxiv.org/html/2610.33375#S1.F2), we construct the benchmark from executable enterprise workflows and evaluate orchestration methods under matched task specifications, tool interfaces, runtime variants, execution environments, and evaluation criteria\. Each workflow is evaluated under multiple runtime conditions, allowing us to compare how the same task is handled when orchestration is completed before execution versus through step\-by\-step interaction\.

Benchmark Overview\.AgentOrchcontains 650 executable standard operating procedures \(SOPs\) and 3,220 runtime variants\. Each source recipe defines one SOP and serves as its reference workflow\. For each SOP, we construct a natural\-language task description \(Appendix[A\.3](https://arxiv.org/html/2610.33375#A1.SS3)\) and multiple runtime variants\. The task description and reference workflow remain fixed, while variants instantiate different inputs and execution conditions\. This design allows the same workflow task to be evaluated under bothGlobalandStep\-by\-Steporchestration\.

Task Specification\.Each SOP has a fixedtask specificationcontaining the task description, public input/output schemas, execution context, workspace resources, communication templates when applicable, and 64 available tool interfaces\. Requirements are presented in non\-causal order, requiring the model to infer tool calls, dependencies, control flow, and argument bindings\. The specification excludes the reference workflow, reference traces, hidden environment state, fault settings, and concrete runtime inputs\.

#### 3\.1Execution Environment and Runtime Variants

Environment Construction\.We map recipe operations to a fixed catalog of 64 tool interfaces spanning services such as Slack, Jira, Salesforce, Google Sheets, Google Drive, Snowflake, and Workato\. We implement real and simulated backends behind the same interfaces: real connectors are used to collect reference behavior, while model\-generated workflows are evaluated in simulation\. The simulated backend reproduces the benchmark\-relevant connector behavior while enabling isolated and reproducible evaluation without external API rate limits, quotas, or mutable service state\.

Runtime Variants\.Each SOP is associated with multiple runtime variants that instantiate concrete inputs and execution conditions\. These variants exercise different control\-flow paths, loop cardinalities, and error conditions while preserving the same task description and reference workflow\. Executions are isolated across variants\. Construction details are provided in Appendix[A\.4](https://arxiv.org/html/2610.33375#A1.SS4)\.

Reference Workflow and Ground Truth\.For each SOP, the source recipe serves as the reference workflow\. We execute it on each runtime variant using the real connector backend and record the execution trace, intermediate tool results, and terminal output\. These executions define the reference behavior used for evaluation\.

#### 3\.2Orchestration Protocols

We compareGlobalandStep\-by\-Stepas the two primary orchestration protocols\. We additionally studyLocalfor incremental workflow construction andReplanfor execution\-feedback improvement\.

Global\.The model generates one complete workflow before execution and before observing concrete runtime inputs\. The workflow is then fixed and reused across all variants of the SOP without further LLM interaction\.

Step\-by\-Step\.The protocol uses an action–observation loop related to ReAct\([Yao et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib47)\)\. Each variant is an independent episode\. The model receives the task specification and concrete runtime input, produces one tool call, observes its result, and selects the next action until termination or the execution limit\.

GlobalandStep\-by\-Stepdiffer in the information available during execution:Step\-by\-Stepreceives the concrete runtime input and intermediate tool results, whereasGlobalconstructs its workflow before observing either\. We therefore use their performance difference as the gap between the two orchestration settings, rather than attributing it solely to execution feedback\.

Local\.Localreceives the same task specification asGlobaland observes neither concrete runtime inputs nor execution results, but constructs the workflow incrementally through an append\-only editor\. After each proposed edit, a validator either accepts the node and returns the updated workflow draft or rejects it with an error\. Thus,Localtests whether incremental construction with validator feedback improves planning without runtime execution feedback\.

Replan\.Replanstudies feedback\-driven self\-improvement without model\-weight updates, a principle also used by Reflexion\([Shinn et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib34)\)\. Starting from theGlobalworkflow, we execute it on runtime variants and provide the model with its workflow and execution feedback, including tool calls, arguments, results or errors, control\-flow outcomes, and terminal returns\. The model then generates a revised complete workflow\. Reference workflows, reference executions, and evaluation scores are never provided\.

#### 3\.3Evaluation Metrics

Orchestration Accuracy\.Our primary metric is the average of Execution Trace Accuracy and Side\-Effect Accuracy for each runtime variant\.Execution Trace Accuracy\.Execution Trace Accuracy measures agreement with the reference execution trace and averages five binary components: tool selection, tool order, tool execution, required arguments, and control path\.Side\-Effect Accuracy\.Side\-Effect Accuracy measures whether execution produces the required state changes without missing, duplicated, or unintended effects\.

Final Accuracy\.Final Accuracy measures whether both the required side effects and terminal output are correct\. It is counted as correct only when both Side\-Effect Accuracy and Terminal Output Accuracy are correct\. Terminal Output Accuracy checks whether the terminal response matches the expected output when the SOP specifies one\.

Error Rate\.Error Rate measures the fraction of runtime variants that do not produce a scoreable execution because of planning, model, infrastructure, or evaluation failures\. These variants remain part of the evaluation and receive zero for the accuracy metrics rather than being excluded\.

#### 3\.4Training for Global Orchestration

We next study whether global orchestration can be improved through targeted training\. To align training with theGlobalsetting, each training example requires the model to generate one complete workflow for an SOP without using any concrete runtime variant\. We construct these examples from a separate set of 3,000 source recipes\. The 3,000 training recipes are disjoint from theAgentOrchbenchmark by source recipe ID\.

Training Data ConstructionFor each source recipe, we construct a task specification following the same procedure used forAgentOrch\. The source recipe serves as the reference workflow\. This produces one training pair per recipe: the task specification is the model input, and the complete workflow is the target output\. The target preserves the full workflow structure of the source recipe, including conditional branches, loops, exception handling, tool calls, and dependencies between tool results and later operations\. Runtime variants are not used to specialize the target\. The resulting training set contains 3,000 training pairs\. Across the training set, the reference workflows use 1,033 tool operations spanning 305 namespaces\.

Figure 3:Scaling does not consistently narrow the GLOBAL–STEP\-BY\-STEP gap\.Across Qwen2\.5 and Gemma\-3, the differences in Orchestration Accuracy, Final Accuracy, and Error Rate do not decrease monotonically with model size\.
#### 3\.5Training Methods

Supervised Fine\-Tuning\.We first apply supervised fine\-tuning to train the model to generate the reference workflow from the task specification\. The training objective is applied to the generated workflow, while the task specification is provided as input context\. We use the same training formulation across all evaluated base models\.

Reinforcement Learning\.We further apply reinforcement learning to optimize generated workflows against the reference workflow\. For each sampled workflow, the reward measures whether the output is valid and how closely it matches the reference workflow in tool calls, tool arguments, and workflow structure\. The reward does not execute the generated workflow or use the simulated connector backend\. We optimize the model using Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib29)\)\. Training configurations for both methods are provided in Appendix[A\.7](https://arxiv.org/html/2610.33375#A1.SS7)\.

### 4Experiments

We first characterize what does and does not explain theGlobal–Step\-by\-Stepgap\. We then study three routes for narrowing it, each drawing on a stronger signal than the last: incremental workflow construction without runtime execution feedback, workflow revision using execution feedback, and targeted training on reference workflows\. Additional evaluation details are provided in Appendix[A\.5](https://arxiv.org/html/2610.33375#A1.SS5)\.

RQ1:Do improvements in model capability translate into stronger global orchestration and a smaller gap to step\-by\-step interaction?

Improvements in model capability may raise bothGlobalandStep\-by\-Stepperformance without reducing the difference between them\. We therefore distinguish two outcomes: absolute improvement inGlobalperformance and reduction in theGlobal–Step\-by\-Stepgap\. We examine three model\-side factors: scale, model generation, and reasoning\. Full results are provided in Appendix[B\.1](https://arxiv.org/html/2610.33375#A2.SS1)\.

Model Scale Does Not Consistently Improve Global Orchestration or Narrow the Gap\.Figure[3](https://arxiv.org/html/2610.33375#S3.F3)comparesGlobalandStep\-by\-Stepacross model sizes within Qwen2\.5 and Gemma\-3\. NeitherGlobalOrchestration Accuracy nor the gap changes monotonically with scale\. For Qwen2\.5,GlobalOrchestration Accuracy drops from 16\.18% at 32B to 9\.91% at 72B, and the gap decreases from 36\.33 pp at 0\.5B to 27\.87 pp at 7B before rising to 36\.66 pp at 72B; for Gemma\-3, the gap grows from 26\.60 pp at 4B to 32\.52 pp at 27B\.Step\-by\-Stepalso retains higher Final Accuracy and lower Error Rate at every evaluated size\.

Table 2:Conditioning global orchestration on concrete runtime inputs does not narrow the gap\.Orchestration Accuracy \(%\);Δ\\DeltaisInput\-Conditioned GlobalminusGlobalin percentage points\.ModelGlobalInput\-Cond\. GlobalStep\-by\-Step𝚫\\bm\{\\Delta\}Qwen3\-4B13\.7513\.0441\.84\-0\.71Gemma\-4\-12B14\.2612\.7349\.83\-1\.53Gemma\-4\-26B4\.053\.6652\.10\-0\.39Claude Opus 546\.4448\.7963\.52\+2\.35DeepSeek\-V427\.0125\.6258\.99\-1\.39Qwen3\.8\-27B28\.5729\.1953\.31\+0\.62Average22\.3522\.1853\.27\-0\.18

Newer Generations Improve Global Orchestration in Most Comparisons, but Do Not Consistently Narrow the Gap\.Across the generation pairs \(Table[3](https://arxiv.org/html/2610.33375#S4.T3)\), newer generations improveGlobalOrchestration Accuracy in four pairs, by 5\.80–17\.98 pp, with only Llama decreasing slightly \(−0\.36\-0\.36pp\)\. However, the gap narrows only for Qwen \(3\.88 pp\) and GPT \(18\.64 pp\), and widens for Gemma, Llama, and GLM \(1\.39–6\.18 pp\)\. The Final Accuracy gap narrows only for GPT, whileGlobalError Rate decreases in all five pairs\. Thus, newer generations generally strengthenGlobal, but stronger absolute performance does not consistently translate into a smaller gap\.

Reasoning Does Not Consistently Improve Global Orchestration or Narrow the Gap\.Enabling reasoning increasesGlobalOrchestration Accuracy for four of eight models and decreases it for the other four \(Table[4](https://arxiv.org/html/2610.33375#S4.T4), Table[11](https://arxiv.org/html/2610.33375#A2.T11)\), with changes ranging from \+13\.44 pp for Claude Opus 5 to−9\.87\-9\.87pp for DeepSeek\-V4\-Flash\. The corresponding gap narrows for only three models and widens for five\. Thus, reasoning provides neither a consistent absolute benefit toGlobalnor a consistent reduction in the gap\.

Table 3:Newer model generations generally improve global orchestration, but do not consistently narrow the gap to step\-by\-step interaction\.Scores are reported in percent; Gap andΔ\\Deltaare reported in percentage points \(pp\)\. For accuracy metrics, Gap isStep\-by\-StepminusGlobal; for Error Rate, Gap isGlobalminusStep\-by\-Step\.Δ\\Deltadenotes newer generation minus earlier generation\.QwenGemmaLlamaGPTGLMMetricSettingQwen2\.53BQwen34BΔ\\DeltaGemma\-312BGemma\-412BΔ\\DeltaLlama\-3\.170BLlama\-3\.370BΔ\\DeltaGPT\-3\.5GPT\-4oΔ\\DeltaGLM\-4\.5FP8GLM\-5\.2FP8Δ\\DeltaOrch\.↑\\uparrowGlobal2\.1913\.75\+11\.568\.4614\.26\+5\.8010\.8110\.45\-0\.3610\.1128\.09\+17\.9812\.9319\.23\+6\.30Step\-by\-Step34\.1641\.84\+7\.6837\.8549\.83\+11\.9838\.9242\.98\+4\.0650\.7850\.12\-0\.6654\.4762\.16\+7\.69Gap31\.9728\.09\-3\.8829\.3935\.57\+6\.1828\.1132\.53\+4\.4240\.6722\.03\-18\.6441\.5442\.93\+1\.39Final↑\\uparrowGlobal0\.593\.91\+3\.321\.834\.07\+2\.243\.354\.29\+0\.944\.3210\.34\+6\.025\.198\.57\+3\.38Step\-by\-Step9\.5017\.30\+7\.8013\.0727\.76\+14\.6914\.4120\.31\+5\.9030\.0629\.13\-0\.9338\.2948\.20\+9\.91Gap8\.9113\.39\+4\.4811\.2423\.69\+12\.4511\.0616\.02\+4\.9625\.7418\.79\-6\.9533\.1039\.63\+6\.53Error↓\\downarrowGlobal94\.7861\.96\-32\.8277\.1765\.78\-11\.3972\.2471\.46\-0\.7875\.0935\.40\-39\.6975\.6861\.34\-14\.34Step\-by\-Step0\.500\.06\-0\.440\.030\.16\+0\.130\.030\.00\-0\.030\.190\.16\-0\.030\.090\.12\+0\.03Gap94\.2861\.90\-32\.3877\.1465\.62\-11\.5272\.2171\.46\-0\.7574\.9035\.24\-39\.6675\.5961\.22\-14\.37

Table 4:Reasoning does not consistently improve global orchestration\.We report global\-mode performance for base and thinking variants across models\. For each base–thinking pair, the better\-performing score is highlighted and bolded \(higher is better for accuracy metrics; lower is better for error rate\)\.Qwen3\-4BQwen3\-30BQwen3\.8\-27BDeepSeek\-V4GLM\-4\.5\-AirGLM\-4\.5\-FP8GLM\-5\.2\-FP8Claude\-Opus\-5MetricBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkOrch\.13\.75%10\.25%8\.35%14\.75%28\.57%33\.02%27\.01%17\.14%12\.28%10\.72%12\.93%20\.09%19\.23%17\.75%46\.44%59\.88%Final3\.91%2\.70%2\.95%5\.19%12\.67%17\.05%11\.43%9\.66%4\.78%4\.25%5\.19%8\.48%8\.57%9\.29%23\.54%32\.33%Side16\.40%13\.04%9\.44%16\.83%31\.80%35\.71%29\.94%17\.52%14\.22%11\.61%13\.88%21\.46%21\.15%18\.66%50\.16%63\.98%Trace11\.11%7\.45%7\.26%12\.68%25\.33%30\.33%24\.07%16\.76%10\.34%9\.83%11\.99%18\.72%17\.31%16\.84%42\.72%55\.78%Term\.16\.30%8\.42%13\.73%20\.12%28\.98%35\.16%28\.98%20\.06%11\.71%13\.45%10\.81%19\.88%18\.94%14\.32%42\.24%48\.29%Mean Acc\.↑\\uparrow12\.29%8\.37%8\.35%13\.91%25\.47%30\.25%24\.29%16\.23%10\.67%9\.97%10\.96%17\.73%17\.04%15\.37%41\.02%52\.05%Error↓\\downarrow61\.96%77\.76%76\.43%60\.81%43\.88%39\.47%43\.51%67\.67%73\.76%74\.63%75\.68%58\.35%61\.34%73\.01%19\.16%6\.21%

Ablation: Access to Runtime Inputs Alone Does Not Explain the GapWe next test whether access to concrete runtime inputs accounts for the protocol gap\.Input\-Conditioned Globalreceives the runtime input but generates the complete workflow before execution, without intermediate execution results\. Across six models, this changes Orchestration Accuracy by−0\.18\-0\.18pp on average; the mean gap toStep\-by\-Stepchanges from 30\.92 pp to 31\.09 pp \(Table[2](https://arxiv.org/html/2610.33375#S4.T2)\)\. Providing the runtime input alone therefore leaves the average gap largely unchanged\. The comparison still jointly varies sequential decision making and intermediate execution information, so it does not isolate their contributions\.

RQ2:Can incremental workflow construction improve global orchestration without runtime execution feedback?

Table 5:Incremental plan construction can improve global orchestration without execution feedback\.Orchestration Accuracy \(%\);Δ\\DeltaisLocalminusGlobal\.ModelGlobalLocal𝚫\\bm\{\\Delta\}Llama\-3\.3\-70B10\.434\.9\+24\.5Qwen2\.5\-3B2\.222\.8\+20\.6Mistral\-Small\-3\.1\-24B14\.127\.3\+13\.1Qwen3\.8\-27B\-Thinking33\.041\.5\+8\.5Qwen3\-30B\-A3B8\.48\.5\+0\.1Qwen3\-4B13\.86\.9\-6\.9

Incremental Workflow Construction Improves Global Orchestration for Most Evaluated Models\.Since stronger models alone do not reliably narrow the gap, we next change how the workflow is constructed while keeping the available information fixed\.LocalandGlobalreceive the same task specification and observe neither concrete runtime inputs nor execution results; they differ only in whether the workflow is generated in one shot or constructed incrementally with validation after each edit \(Section[3\.2](https://arxiv.org/html/2610.33375#S3.SS2)\)\.Localimproves Orchestration Accuracy by 9\.98 pp on average \(Table[5](https://arxiv.org/html/2610.33375#S4.T5)\), with the largest gains for Llama\-3\.3\-70B \(\+24\.5 pp\) and Qwen2\.5\-3B \(\+20\.6 pp\)\. Qwen3\-4B is the only model that declines \(−6\.9\-6\.9pp\)\.

RQ3:Can execution feedback improve global workflows, and where do the gains come from?

Execution\-Feedback Revision Improves Global Orchestration\.RQ2 improvesGlobalwithout observing any execution\. We next allow the model to observe how its workflow executes\.Replanexecutes theGlobalworkflow, returns the execution feedback to the model, and asks it to revise the complete workflow\. The revised workflow still executes without further LLM calls\. Over three rounds,Replanimproves Orchestration Accuracy for all eight evaluated models, by 6\.18 pp on average, and reduces Error Rate for every model \(Figure[4](https://arxiv.org/html/2610.33375#S4.F4)\)\.

The Gains Transfer to Unseen Runtime Variants\.We next examine whether the improvement is limited to the runtime variants used to construct the feedback\. We provide feedback from only two variants of each SOP and evaluate on the remaining held\-out variants\. The average held\-out gain is 6\.13 pp, nearly identical to the 6\.18 pp obtained with feedback from all variants \(Table[7](https://arxiv.org/html/2610.33375#S4.T7)\), and far more individual held\-out variants improve than regress \(Appendix[B\.2](https://arxiv.org/html/2610.33375#A2.SS2)\)\.Replantherefore improves the workflow itself, not only its behavior on the feedback executions\.

Table 6:Decomposition ofReplangains\.Repair and Valid: gains from initially invalid and initially valid workflows\. Validation\-only: revision using pre\-execution validation errors without execution\. Recovery=Δ/\(Repair\+Valid\)=\\Delta/\(\\mathrm\{Repair\}\+\\mathrm\{Valid\}\)\.ReplanValidation\-onlyModelRepairValidShare𝚫\\bm\{\\Delta\}RecoveryR–SMistral\-3\.1\-24B\+7\.09\+0\.5293\.2%\+6\.1080\.2%\+1\.51Qwen3\-30B\-A3B\+2\.75\+0\.6680\.6%\+1\.7049\.9%\+1\.71Qwen3\-4B\+4\.880\.00100\.0%\+4\.2086\.1%\+0\.68Across models\+4\.91\+0\.3992\.6%\+4\.0075\.5%\+1\.30

Most Gains Come from Fixing How Tools Are Called\.Across the three models in the decomposition, revisions that turn invalid Round\-0 workflows into valid ones account for 92\.6% of the aggregateReplangain \(Table[6](https://arxiv.org/html/2610.33375#S4.T6)\)\. In the eight\-model analysis, most repaired workflows initially parse successfully but fail schema, type, or reference checks governing tool arguments and references between steps \(Appendix[B\.2](https://arxiv.org/html/2610.33375#A2.SS2)\)\.Globalmust specify these arguments and references before execution, whileStep\-by\-Stepcan use observed results to construct subsequent calls\. All initial failures in the repaired workflows are detectable before execution\. Revising the same Round\-0 workflows using only pre\-execution validation errors recovers 75\.5% of the aggregateReplangain across the three models\.

Together, these results show that global orchestration requires not only selecting the right tools, but also specifying before execution how each tool is called and how data flows between calls\. Getting these specifications right is a major bottleneck of global orchestration\.

![Refer to caption](https://arxiv.org/html/2610.33375v1/rq2_a.png)Figure 4:Replanconsistently improves global orchestration across models\.Performance over Rounds 0–3 for eight models when execution feedback is collected from all runtime variants of each SOP\. Accuracy metrics generally increase across revision rounds, while Error Rate decreases\.Table 7:Replanimprovements generalize to unseen runtime variants\.We report performance beforeReplan\(Base, Round 0\) and after threeReplanrounds \(Final, Round 3\)\. At each round, the model receives execution feedback from only the two random runtime variants of each SOP, while evaluation is performed on the remaining held\-out variants that are never used for feedback\. For Mean Accuracy and Error Rate, we additionally report the percentage\-point change \(Δ\\Delta\)\. Mean Accuracy averages Orchestration, Final, Side\-effect, Execution\-trace, and Terminal\-output Accuracy\. Higher accuracy and lower Error Rate are better\. No model weights are updated\.ModelOrch\.↑\\uparrowFinal↑\\uparrowSide↑\\uparrowTrace↑\\uparrowTerm\.↑\\uparrowMean Acc\.↑\\uparrowError↓\\downarrowBaseFinalBaseFinalBaseFinalBaseFinalBaseFinalBaseFinal𝚫\\bm\{\\Delta\}BaseFinal𝚫\\bm\{\\Delta\}Qwen3\.8\-27B29\.1140\.0714\.3722\.2233\.0244\.3425\.1935\.7928\.7338\.8126\.0836\.25\+10\.1645\.2231\.58\-13\.64DeepSeek\-V4\-Flash27\.2135\.4612\.3516\.6930\.9040\.6223\.5130\.2928\.4833\.1824\.4931\.25\+6\.7644\.8631\.27\-13\.59Mistral\-Small\-3\.1\-24B14\.5221\.896\.939\.8218\.2926\.7210\.7417\.0616\.3824\.5513\.3720\.01\+6\.6467\.6051\.21\-16\.39Gemma\-4\-26B\-A4B3\.829\.621\.914\.294\.0310\.653\.618\.594\.969\.923\.678\.61\+4\.9590\.6579\.59\-11\.06Gemma\-4\-12B14\.4720\.244\.346\.6716\.9023\.9812\.0416\.5112\.8717\.5712\.1216\.99\+4\.8766\.9354\.68\-12\.25Qwen3\-4B14\.2419\.134\.816\.6117\.3123\.6211\.1714\.6515\.7121\.5512\.6517\.11\+4\.4663\.6250\.49\-13\.13Qwen3\-30B\-A3B8\.5011\.143\.315\.2710\.0312\.666\.989\.6112\.9217\.988\.3511\.33\+2\.9877\.2671\.11\-6\.15Gemma\-3\-27B7\.8011\.133\.154\.3910\.9015\.304\.706\.9610\.9615\.507\.5010\.66\+3\.1578\.5569\.15\-9\.40

RQ4:Can stronger global orchestration be learned through targeted training?

Training on Complete Workflows Improves Global Orchestration for All Models\.RQ2 and RQ3 improveGlobalat test time, and RQ3 shows that its main obstacle is producing a workflow with correct arguments and references\. We next ask whether training can teach models to produce such workflows directly\. We train five models on 3,000 source recipes disjoint fromAgentOrch, each pairing a task specification with a complete reference workflow \(Section[3\.4](https://arxiv.org/html/2610.33375#S3.SS4)\)\. Supervised fine\-tuning \(SFT\) improves Orchestration Accuracy and Final Accuracy for all five models, and taking the better of SFT and RL for each model yields an average Orchestration Accuracy gain of 12\.18 pp \(Table[8](https://arxiv.org/html/2610.33375#S4.T8)\)\. Error Rate drops even more sharply, by 13\.02–40\.99 pp; Qwen3\.8\-27B falls from 43\.88% to 17\.08%, below the 19\.16% of Claude Opus 5 underGlobal\.

Reinforcement Learning Adds Gains for Some Models\.We next examine whether reinforcement learning \(RL\) further improves the SFT models\. As shown in Table[8](https://arxiv.org/html/2610.33375#S4.T8), RL further increases Orchestration Accuracy for three of five models: Qwen3\-4B by 1\.70 pp, Gemma\-4\-26B\-A4B by 4\.04 pp, and Gemma\-4\-12B by 3\.74 pp\. Error Rate also decreases for four models, including reductions of 8\.38 pp for Gemma\-4\-26B\-A4B and 11\.18 pp for Gemma\-4\-12B\. However, RL does not improve Orchestration Accuracy for Qwen3\.8\-27B or Gemma\-4\-31B, and Final Accuracy improves for only two models\. Overall, RL can provide additional gains after SFT, but the benefit is not consistent across models or metrics\.

Table 8:Targeted training improves global orchestration\.Results underGlobal;Δ\\Deltacompares the better of SFT and RL with Base \(pp\)\.Orch\. Acc\. \(%\)↑\\uparrowFinal Acc\. \(%\)↑\\uparrowTrace Acc\. \(%\)↑\\uparrowError Rate \(%\)↓\\downarrowModelBaseSFTRL𝚫\\bm\{\\Delta\}BaseSFTRL𝚫\\bm\{\\Delta\}BaseSFTRL𝚫\\bm\{\\Delta\}BaseSFTRL𝚫\\bm\{\\Delta\}Qwen3\.8\-27B28\.5741\.9641\.14\+13\.3912\.6718\.4818\.26\+5\.8125\.3335\.8735\.24\+10\.5443\.8817\.0819\.38−26\.80\-26\.80Qwen3\-4B13\.7525\.4627\.16\+13\.413\.9110\.7810\.71\+6\.8711\.1119\.8021\.43\+10\.3261\.9640\.9936\.12−25\.84\-25\.84Gemma\-4\-26B\-A4B4\.0519\.8923\.93\+19\.881\.808\.459\.35\+7\.553\.9915\.4018\.66\+14\.6789\.2256\.6148\.23−40\.99\-40\.99Gemma\-4\-31B12\.7119\.5519\.10\+6\.844\.789\.789\.22\+5\.0012\.0716\.0315\.93\+3\.9674\.9457\.1756\.55−18\.39\-18\.39Gemma\-4\-12B14\.2617\.9221\.66\+7\.404\.078\.518\.54\+4\.4712\.4414\.2917\.33\+4\.8965\.7863\.9452\.76−13\.02\-13\.02

### 5Conclusion

We introducedAgentOrchto study how close global orchestration can come to step\-by\-step interaction on the same enterprise workflows\.Step\-by\-Stepremains more accurate for every evaluated model\. The gap has shrunk by about 79% from early models to the strongest recent model, but this narrowing is concentrated in the latest generation: neither model scale, newer generations, nor reasoning alone consistently narrows it\. We identified three routes that narrow the gap further, each drawing on a stronger signal than the last: incremental workflow construction without execution feedback, revision with execution feedback that transfers to unseen runtime variants, and targeted training on reference workflows that improves every trained model\. Our analysis further shows that global orchestration requires not only selecting the right tools, but also specifying before execution how each tool is called and how data flows between calls; getting these specifications right is a key lever for future work\. The gap has not closed, but it is closing: global orchestration offers a promising path toward enterprise agents that approach step\-by\-step accuracy with fewer LLM calls\.

### References

- Boisvert et al\. \(2024\)Boisvert, L\., Thakkar, M\., Gasse, M\., Caccia, M\., De Chezelles, T\. L\., Cappart, Q\., Chapados, N\., Lacoste, A\., and Drouin, A\.Workarena\+\+: Towards compositional planning and reasoning\-based common knowledge work tasks\.*Advances in Neural Information Processing Systems*, 37:5996–6051, 2024\.
- Busch & Leopold \(2024\)Busch, K\. and Leopold, H\.Towards a benchmark for large language models for business process management tasks\.*arXiv preprint arXiv:2410\.03255*, 2024\.
- Chaurasia & Mooney \(2017\)Chaurasia, S\. and Mooney, R\.Dialog for language to code\.In*Proceedings of the Eighth International Joint Conference on Natural Language Processing \(Volume 2: Short Papers\)*, pp\. 175–180, 2017\.
- Chen et al\. \(2024\)Chen, Z\., Du, W\., Zhang, W\., Liu, K\., Liu, J\., Zheng, M\., Zhuo, J\., Zhang, S\., Lin, D\., Chen, K\., and Zhao, F\.T\-eval: Evaluating the tool utilization capability of large language models step by step\.In Ku, L\., Martins, A\., and Srikumar, V\. \(eds\.\),*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024*, pp\. 9510–9529\. Association for Computational Linguistics, 2024\.doi:10\.18653/V1/2024\.ACL\-LONG\.515\.URL[https://doi\.org/10\.18653/v1/2024\.acl\-long\.515](https://doi.org/10.18653/v1/2024.acl-long.515)\.
- Deng et al\. \(2023\)Deng, X\., Gu, Y\., Zheng, B\., Chen, S\., Stevens, S\., Wang, B\., Sun, H\., and Su, Y\.Mind2web: Towards a generalist agent for the web\.*Advances in Neural Information Processing Systems*, 36:28091–28114, 2023\.
- Drouin et al\. \(2024\)Drouin, A\., Gasse, M\., Caccia, M\., Laradji, I\. H\., Del Verme, M\., Marty, T\., Boisvert, L\., Thakkar, M\., Cappart, Q\., Vazquez, D\., et al\.Workarena: How capable are web agents at solving common knowledge work tasks?*arXiv preprint arXiv:2403\.07718*, 2024\.
- Fan et al\. \(2025\)Fan, S\., Cong, X\., Fu, Y\., Zhang, Z\., Zhang, S\., Liu, Y\., Wu, Y\., Lin, Y\., Liu, Z\., and Sun, M\.Workflowllm: Enhancing workflow orchestration capability of large language models\.In*International Conference on Learning Representations*, volume 2025, pp\. 24498–24525, 2025\.
- Guo et al\. \(2024\)Guo, Z\., Cheng, S\., Wang, H\., Liang, S\., Qin, Y\., Li, P\., Liu, Z\., Sun, M\., and Liu, Y\.Stabletoolbench: Towards stable large\-scale benchmarking on tool learning of large language models\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pp\. 11143–11156, 2024\.
- Huang et al\. \(2024\)Huang, S\., Zhong, W\., Lu, J\., Zhu, Q\., Gao, J\., Liu, W\., Hou, Y\., Zeng, X\., Wang, Y\., Shang, L\., et al\.Planning, creation, usage: Benchmarking llms for comprehensive tool utilization in real\-world complex scenarios\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pp\. 4363–4400, 2024\.
- Jimenez et al\. \(2024\)Jimenez, C\. E\., Yang, J\., Wettig, A\., Yao, S\., Pei, K\., Press, O\., and Narasimhan, K\.Swe\-bench: Can language models resolve real\-world github issues?In*International Conference on Learning Representations*, volume 2024, pp\. 54107–54157, 2024\.
- Kim et al\. \(2023\)Kim, S\., Moon, S\., Tabrizi, R\., Lee, N\., Mahoney, M\. W\., Keutzer, K\., and Gholami, A\.An llm compiler for parallel function calling\.*arXiv preprint arXiv:2312\.04511*, 2023\.
- Koh et al\. \(2024\)Koh, J\. Y\., Lo, R\., Jang, L\., Duvvur, V\., Lim, M\., Huang, P\.\-Y\., Neubig, G\., Zhou, S\., Salakhutdinov, R\., and Fried, D\.Visualwebarena: Evaluating multimodal agents on realistic visual web tasks\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 881–905, 2024\.
- Kourani et al\. \(2024\)Kourani, H\., Berti, A\., Schuster, D\., and van der Aalst, W\. M\.Process modeling with large language models\.In*International Conference on Business Process Modeling, Development and Support*, pp\. 229–244\. Springer, 2024\.
- Kourani et al\. \(2026\)Kourani, H\., Berti, A\., Schuster, D\., and van der Aalst, W\. M\. P\.Evaluating large language models on business process modeling: framework, benchmark, and self\-improvement analysis\.*Softw\. Syst\. Model\.*, 25\(4\):1151–1186, 2026\.doi:10\.1007/S10270\-025\-01318\-W\.URL[https://doi\.org/10\.1007/s10270\-025\-01318\-w](https://doi.org/10.1007/s10270-025-01318-w)\.
- Li et al\. \(2026\)Li, J\., Zhao, W\., Zhao, J\., Zeng, W\., Wu, H\., Wang, X\., Ge, R\., Cao, Y\., Huang, Y\., Liu, W\., et al\.The tool decathlon: Benchmarking language agents for diverse, realistic, and long\-horizon task execution\.In*International Conference on Learning Representations*, volume 2026, pp\. 53359–53394, 2026\.
- Li et al\. \(2023\)Li, M\., Zhao, Y\., Yu, B\., Song, F\., Li, H\., Yu, H\., Li, Z\., Huang, F\., and Li, Y\.Api\-bank: A comprehensive benchmark for tool\-augmented llms\.In*Proceedings of the 2023 conference on empirical methods in natural language processing*, pp\. 3102–3116, 2023\.
- Li et al\. \(2025\)Li, X\., Ni, L\., Wang, X\., Yitong, T\., Li, R\., Liu, J\., and Wang, Z\.Llm\-based business process models generation from textual descriptions\.In Inui, K\., Sakti, S\., Wang, H\., Wong, D\. F\., Bhattacharyya, P\., Banerjee, B\., Ekbal, A\., Chakraborty, T\., and Singh, D\. P\. \(eds\.\),*Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics, IJCNLP\-AACL 2025, Mumbai, India, December 20\-24, 2025*, pp\. 523–533\. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics, 2025\.doi:10\.18653/V1/2025\.FINDINGS\-IJCNLP\.31\.URL[https://doi\.org/10\.18653/v1/2025\.findings\-ijcnlp\.31](https://doi.org/10.18653/v1/2025.findings-ijcnlp.31)\.
- Li et al\. \(2024\)Li, Z\., Xu, S\., Mei, K\., Hua, W\., Rama, B\., Raheja, O\., Wang, H\., Zhu, H\., and Zhang, Y\.Autoflow: Automated workflow generation for large language model agents\.*arXiv preprint arXiv:2407\.12821*, 2024\.
- Liang et al\. \(2023\)Liang, J\., Huang, W\., Xia, F\., Xu, P\., Hausman, K\., Ichter, B\., Florence, P\., and Zeng, A\.Code as policies: Language model programs for embodied control\.In*2023 IEEE International conference on robotics and automation \(ICRA\)*, pp\. 9493–9500\. IEEE, 2023\.
- Liu et al\. \(2024\)Liu, X\., Yu, H\., Zhang, H\., Xu, Y\., Lei, X\., Lai, H\., Gu, Y\., Ding, H\., Men, K\., Yang, K\., et al\.Agentbench: Evaluating llms as agents\.In*International Conference on Learning Representations*, volume 2024, pp\. 52989–53046, 2024\.
- Lu et al\. \(2025\)Lu, J\., Holleis, T\., Zhang, Y\., Aumayer, B\., Nan, F\., Bai, H\., Ma, S\., Ma, S\., Li, M\., Yin, G\., et al\.Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities\.In*Findings of the Association for Computational Linguistics: NAACL 2025*, pp\. 1160–1183, 2025\.
- Nath et al\. \(2025\)Nath, V\., Raja, P\., Yoon, C\., and Hendryx, S\.Toolcomp: A multi\-tool reasoning & process supervision benchmark\.*arXiv preprint arXiv:2501\.01290*, 2025\.
- Patil et al\. \(2024\)Patil, S\. G\., Zhang, T\., Wang, X\., and Gonzalez, J\. E\.Gorilla: Large language model connected with massive apis\.*Advances in Neural Information Processing Systems*, 37:126544–126565, 2024\.
- Qiao et al\. \(2025\)Qiao, S\., Fang, R\., Qiu, Z\., Wang, X\., Zhang, N\., Jiang, Y\., Xie, P\., Huang, F\., and Chen, H\.Benchmarking agentic workflow generation\.In*International Conference on Learning Representations*, volume 2025, pp\. 69679–69703, 2025\.
- Qin et al\. \(2024\)Qin, Y\., Liang, S\., Ye, Y\., Zhu, K\., Yan, L\., Lu, Y\., Lin, Y\., Cong, X\., Tang, X\., Qian, B\., et al\.Toolllm: Facilitating large language models to master 16000\+ real\-world apis\.In*International Conference on Learning Representations*, volume 2024, pp\. 9695–9717, 2024\.
- Quirk et al\. \(2015\)Quirk, C\., Mooney, R\., and Galley, M\.Language to code: Learning semantic parsers for if\-this\-then\-that recipes\.In*Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pp\. 878–888, 2015\.
- Rawles et al\. \(2025\)Rawles, C\., Clinckemaillie, S\., Chang, Y\., Waltz, J\., Lau, G\., Fair, M\., Li, A\., Bishop, W\., Li, W\., Campbell\-Ajala, F\., et al\.Androidworld: A dynamic benchmarking environment for autonomous agents\.In*International Conference on Learning Representations*, volume 2025, pp\. 406–441, 2025\.
- Schick et al\. \(2023\)Schick, T\., Dwivedi\-Yu, J\., Dessì, R\., Raileanu, R\., Lomeli, M\., Hambro, E\., Zettlemoyer, L\., Cancedda, N\., and Scialom, T\.Toolformer: Language models can teach themselves to use tools\.*Advances in neural information processing systems*, 36:68539–68551, 2023\.
- Shao et al\. \(2024\)Shao, Z\., Wang, P\., Zhu, Q\., Xu, R\., Song, J\., Bi, X\., Zhang, H\., Zhang, M\., Li, Y\., Wu, Y\., et al\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Shen et al\. \(2023\)Shen, Y\., Song, K\., Tan, X\., Li, D\., Lu, W\., and Zhuang, Y\.Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face\.*Advances in Neural Information Processing Systems*, 36:38154–38180, 2023\.
- Shen et al\. \(2024\)Shen, Y\., Song, K\., Tan, X\., Zhang, W\., Ren, K\., Yuan, S\., Lu, W\., Li, D\., and Zhuang, Y\.Taskbench: Benchmarking large language models for task automation\.*Advances in Neural Information Processing Systems*, 37:4540–4574, 2024\.
- Shepard & Salimans \(2026\)Shepard, D\. and Salimans, R\.Automationbench\.*arXiv preprint arXiv:2604\.18934*, 2026\.
- Shi et al\. \(2024\)Shi, Z\., Gao, S\., Chen, X\., Feng, Y\., Yan, L\., Shi, H\., Yin, D\., Ren, P\., Verberne, S\., and Ren, Z\.Learning to use tools via cooperative and interactive agents\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pp\. 10642–10657, 2024\.
- Shinn et al\. \(2023\)Shinn, N\., Cassano, F\., Gopinath, A\., Narasimhan, K\., and Yao, S\.Reflexion: Language agents with verbal reinforcement learning\.*Advances in neural information processing systems*, 36:8634–8652, 2023\.
- Singh et al\. \(2022\)Singh, I\., Blukis, V\., Mousavian, A\., Goyal, A\., Xu, D\., Tremblay, J\., Fox, D\., Thomason, J\., and Garg, A\.Progprompt: Generating situated robot task plans using large language models\.*arXiv preprint arXiv:2209\.11302*, 2022\.
- Song et al\. \(2023\)Song, Y\., Xiong, W\., Zhu, D\., Wu, W\., Qian, H\., Song, M\., Huang, H\., Li, C\., Wang, K\., Yao, R\., et al\.Restgpt: Connecting large language models with real\-world restful apis\.*arXiv preprint arXiv:2306\.06624*, 2023\.
- Styles et al\. \(2024\)Styles, O\., Miller, S\., Cerda\-Mardini, P\., Guha, T\., Sanchez, V\., and Vidgen, B\.Workbench: a benchmark dataset for agents in a realistic workplace setting\.*arXiv preprint arXiv:2405\.00823*, 2024\.
- Surís et al\. \(2023\)Surís, D\., Menon, S\., and Vondrick, C\.Vipergpt: Visual inference via python execution for reasoning\.In*2023 IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pp\. 11854–11864\. IEEE, 2023\.
- Tang et al\. \(2023\)Tang, Q\., Deng, Z\., Lin, H\., Han, X\., Liang, Q\., Cao, B\., and Sun, L\.Toolalpaca: Generalized tool learning for language models with 3000 simulated cases\.*arXiv preprint arXiv:2306\.05301*, 2023\.
- Trivedi et al\. \(2024\)Trivedi, H\., Khot, T\., Hartmann, M\., Manku, R\., Dong, V\., Li, E\., Gupta, S\., Sabharwal, A\., and Balasubramanian, N\.Appworld: A controllable world of apps and people for benchmarking interactive coding agents\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 16022–16076, 2024\.
- Ur et al\. \(2016\)Ur, B\., Pak Yong Ho, M\., Brawner, S\., Lee, J\., Mennicken, S\., Picard, N\., Schulze, D\., and Littman, M\. L\.Trigger\-action programming in the wild: An analysis of 200,000 ifttt recipes\.In*Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems*, pp\. 3227–3231, 2016\.
- Wang et al\. \(2023\)Wang, Z\., Cai, S\., Chen, G\., Liu, A\., Ma, X\. S\., and Liang, Y\.Describe, explain, plan and select: interactive planning with llms enables open\-world multi\-task agents\.*Advances in Neural Information Processing Systems*, 36:34153–34189, 2023\.
- Wang et al\. \(2026\)Wang, Z\., Chang, Q\., Patel, H\., Biju, S\., Wu, C\.\-E\., Liu, Q\., Ding, A\., Rezazadeh, A\., Shah, A\. P\., Bao, Y\., et al\.Mcp\-bench: Benchmarking tool\-using llm agents with complex real\-world tasks via mcp servers\.In*International Conference on Learning Representations*, volume 2026, pp\. 97377–97425, 2026\.
- Xie et al\. \(2024\)Xie, T\., Zhang, D\., Chen, J\., Li, X\., Zhao, S\., Cao, R\., Hua, T\. J\., Cheng, Z\., Shin, D\., Lei, F\., et al\.Osworld: Benchmarking multimodal agents for open\-ended tasks in real computer environments\.*Advances in Neural Information Processing Systems*, 37:52040–52094, 2024\.
- Xu et al\. \(2025\)Xu, F\. F\., Song, Y\., Li, B\., Tang, Y\., Jain, K\., Bao, M\., Wang, Z\. Z\., Zhou, X\., Guo, Z\., Cao, M\., Yang, M\., Lu, H\. Y\., Martin, A\., Su, Z\., Maben, L\., Mehta, R\., Chi, W\., Jang, L\., Xie, Y\., Zhou, S\., and Neubig, G\.Theagentcompany: Benchmarking LLM agents on consequential real world tasks\.In Belgrave, D\., Zhang, C\., Montoya, L\. N\., Lin, H\., Pascanu, R\., Koniusz, P\., Ghassemi, M\., Chen, N\., Ruíz, I\. V\. M\., and Loaiza\-Bonilla, A\. \(eds\.\),*Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025 / Mexico City, Mexico, November 30 \- December 5, 2025*, 2025\.URL[http://papers\.nips\.cc/paper\_files/paper/2025/hash/0d744742f6fac4d1134c019b7cef3c8a\-Abstract\-Datasets\_and\_Benchmarks\_Track\.html](http://papers.nips.cc/paper_files/paper/2025/hash/0d744742f6fac4d1134c019b7cef3c8a-Abstract-Datasets_and_Benchmarks_Track.html)\.
- Yao et al\. \(2022\)Yao, S\., Chen, H\., Yang, J\., and Narasimhan, K\.Webshop: Towards scalable real\-world web interaction with grounded language agents\.*Advances in Neural Information Processing Systems*, 35:20744–20757, 2022\.
- Yao et al\. \(2023\)Yao, S\., Zhao, J\., Yu, D\., Du, N\., Shafran, I\., Narasimhan, K\. R\., and Cao, Y\.React: Synergizing reasoning and acting in language models\.In*The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023*\. OpenReview\.net, 2023\.URL[https://openreview\.net/forum?id=WE\_vluYUL\-X](https://openreview.net/forum?id=WE_vluYUL-X)\.
- Yao et al\. \(2024\)Yao, S\., Shinn, N\., Razavi, P\., and Narasimhan, K\.τ\\tau\-bench: A benchmark for tool\-agent\-user interaction in real\-world domains\.*arXiv preprint arXiv:2406\.12045*, 2024\.
- Zeng et al\. \(2023\)Zeng, Z\., Watson, W\., Cho, N\., Rahimi, S\., Reynolds, S\., Balch, T\., and Veloso, M\.Flowmind: automatic workflow generation with llms\.In*Proceedings of the Fourth ACM International Conference on AI in Finance*, pp\. 73–81, 2023\.
- Zhang et al\. \(2025\)Zhang, J\., Xiang, J\., Yu, Z\., Teng, F\., Chen, X\., Chen, J\., Zhuge, M\., Cheng, X\., Hong, S\., Wang, J\., et al\.Aflow: Automating agentic workflow generation\.In*International Conference on Learning Representations*, volume 2025, pp\. 34040–34077, 2025\.
- Zhong et al\. \(2026\)Zhong, Y\., Xu, B\., Wang, Y\., Shan, Z\., Qiao, S\., Zheng, G\., and Zhang, N\.Chat2workflow: A benchmark for generating executable visual workflows with natural language\.*arXiv preprint arXiv:2604\.19667*, 2026\.
- Zhou et al\. \(2023\)Zhou, A\., Yan, K\., Shlapentokh\-Rothman, M\., Wang, H\., and Wang, Y\.\-X\.Language agent tree search unifies reasoning acting and planning in language models\.*arXiv preprint arXiv:2310\.04406*, 2023\.
- Zhou et al\. \(2024\)Zhou, S\., Xu, F\. F\., Zhu, H\., Zhou, X\., Lo, R\., Sridhar, A\., Cheng, X\., Ou, T\., Bisk, Y\., Fried, D\., et al\.Webarena: A realistic web environment for building autonomous agents\.In*International Conference on Learning Representations*, volume 2024, pp\. 15585–15606, 2024\.

## Appendix

### Appendix ABenchmark and Data Construction

#### A\.1Source Recipes and Filtering

We start from a sanitized collection of 803 executable Workato workspace recipes\. We exclude 153 recipes with fewer than two reachable enabled actions, resulting in the 650 SOPs used inAgentOrch\. Each retained source recipe defines one SOP and serves as its reference workflow\.

The benchmark split is defined at the source\-recipe level\. Each recipe appears at most once in the benchmark, while multiple runtime variants may be associated with the same recipe\. The resulting benchmark contains 650 SOPs and 3,220 runtime variants\.

#### A\.2Execution Environment Construction

We construct the execution environment from the software operations used by the source recipes\. These operations are mapped to the fixed set of 64 tool interfaces exposed byAgentOrch\. The same tool names and argument interfaces are used across the benchmark\.

Real connector backend\.The real connector backend invokes the corresponding external software services where supported\. During benchmark construction, we execute each reference workflow using the real connector backend under its runtime variants\. We record the active execution trace, intermediate tool results, and terminal output produced by these executions\. These records are used to construct the corresponding reference execution data\.

Simulated connector backend\.For scalable model evaluation, we implement a local simulated backend for the connector operations exercised by AGENTORCH\. The simulated backend exposes the same tool names, argument schemas, and return interfaces as the corresponding real connectors, and reproduces the benchmark\-relevant execution semantics required by the reference workflows, including state transitions, observable side effects, and supported error behavior\.

The simulated backend is used as an evaluation substitute for the real connectors rather than as a different task environment\. Evaluating every model\-generated workflow directly against live services would be impractical at benchmark scale because external connectors impose API rate limits and quotas, incur service\-dependent latency, and operate over mutable external state\. The simulated backend instead provides deterministic, isolated, and reproducible executions while preserving the connector behavior exercised by the benchmark\.

Simulation is also necessary for safety: executing model\-generated workflows against live enterprise services would send real messages, modify real records, and produce side effects that cannot be undone\.

Each runtime variant initializes a separate simulated connector state\. State changes produced by one tool call are visible to subsequent calls within the same execution, and the environment is reset between executions so that different models, SOPs, and runtime variants do not interfere with one another\.

All model evaluations reported in this paper execute model\-generated workflows using the simulated connector backend\. The real connector backend is used during benchmark construction to collect reference execution results rather than to execute the evaluated model\.

#### A\.3Task Description Generation

Following WorkflowLLM[Fan et al\. \(2025\)](https://arxiv.org/html/2610.33375#bib.bib7), we use a bottom\-up workflow\-to\-description procedure to construct the natural\-language task description for each SOP\. LetArA\_\{r\}denote the structured representation of reference workflowrr, and letDrD\_\{r\}denote the associated connector documentation and tool schemas\. The construction follows

Ar\+Dr→Cr→Pr→\{Qr\(k\)\}k=1KQ→Qr,A\_\{r\}\+D\_\{r\}\\xrightarrow\{\\hskip 8\.19447pt\}C\_\{r\}\\xrightarrow\{\\hskip 8\.19447pt\}P\_\{r\}\\xrightarrow\{\\hskip 8\.19447pt\}\\\{Q\_\{r\}^\{\(k\)\}\\\}\_\{k=1\}^\{K\_\{Q\}\}\\xrightarrow\{\\hskip 8\.19447pt\}Q\_\{r\},whereCrC\_\{r\}contains action\-level semantic annotations,PrP\_\{r\}contains the workflow requirements across its control\-flow paths,\{Qr\(k\)\}\\\{Q\_\{r\}^\{\(k\)\}\\\}contains candidate task descriptions, andQrQ\_\{r\}is the selected task description\.

1. 1\.Workflow representation\.We deterministically transcribe reference workflowrrinto a structured representationArA\_\{r\}\. The representation retains the executable actions and their arguments, literal values, formula expressions, branch conditions, loops, try/catch blocks, and terminal behavior\. Actions without a documented tool interface are identified during this stage\.
2. 2\.Action\-level annotation\.A language model produces an annotation for each workflow node usingArA\_\{r\}andDrD\_\{r\}\. The annotation describes the corresponding operation, its arguments, dependencies on preceding results, branch conditions, and error\-handling behavior\. Each annotation is required to be grounded in information contained inArA\_\{r\}orDrD\_\{r\}\.
3. 3\.Task\-plan construction\.The action\-level annotations are combined into a task planPrP\_\{r\}covering the requirements represented by the workflow\. The plan includes the public inputs, branch conditions, loop behavior, fallback behavior, and exception handling required by the reference workflow\. Runtime\-variant values and hidden connector state are not included\.
4. 4\.Candidate task\-description generation\.UsingArA\_\{r\},CrC\_\{r\}, andPrP\_\{r\}, the model generatesKQ=3K\_\{Q\}=3candidate task descriptions\. Each candidate describes the workflow objective, required inputs, conditions, and expected behavior without exposing recipe node identifiers or the reference execution trace\. The requirements are not presented in the execution order of the reference workflow, so their textual presentation does not reveal the solution sequence\. However, ordering requirements prescribed by the source SOP, including dependencies and temporal constraints between operations, are retained in the task description\. Thus, the model must infer the prescribed execution order from the stated workflow requirements rather than from their presentation order\.
5. 5\.Consistency checking and selection\.We extract a semantic ledgerLrL\_\{r\}fromArA\_\{r\}containing the public inputs, operational requirements, and terminal behavior represented by the workflow\. A model\-based critic compares each candidate task description withLrL\_\{r\}for two forms of consistency: whether required information from the workflow is represented in the description, and whether the description introduces requirements not supported by the workflow\. The selected candidate becomesQrQ\_\{r\}\.
6. 6\.Variant invariance and hidden\-information filtering\.Each SOP uses one task descriptionQrQ\_\{r\}across all of its runtime variants\. The description does not contain concrete variant inputs, hidden connector state, fault\-injection settings, reference execution traces, or expected tool\-call sequences\.

The selected task descriptionQrQ\_\{r\}is included in the task specification described in Section[3](https://arxiv.org/html/2610.33375#S3)\. The remaining components of the task specification, including schemas, workspace resources, communication templates, and tool interfaces, are supplied separately\. Runtime information is provided according to the orchestration protocol described in Section[3\.2](https://arxiv.org/html/2610.33375#S3.SS2); in particular,Globaldoes not receive the concrete runtime input when generating its workflow\.

#### A\.4Runtime Variant Generation

Each SOP is associated with multiple runtime variants that instantiate different runtime inputs and execution conditions while keeping the task specification and reference workflow fixed\. A runtime variant contains a public runtime input together with the environment configuration required to execute that input\. The environment configuration may include hidden connector state and controlled tool failures\.

Most benchmark variants were generated through the model\-based variant\-construction pipeline\. Nine additional variants were constructed through coverage synthesis\. The accepted variants are stored as part of the released benchmark\.

Control\-flow coverage\.Variant construction tracks coverage obligations derived from the reference workflow\. For each conditional, the construction targets both branch outcomes\. For each try/catch block, it targets execution both without an exception and through the catch path\. For loops, it targets zero, one, and multiple iterations when these behaviors are represented by the workflow and variant construction\.

Fault injection\.Variants may include a controlled failure for an operation inside an error\-handling region\. A fault is matched to an executed operation using the tool name together with a literal argument selector when available\. If this does not uniquely identify the target call, matching falls back to the ordinal occurrence of that tool\. The same fault registry and matching procedure are used when executing different orchestration protocols\.

Execution isolation\.Before each execution, the corresponding connector state is initialized from the runtime variant\. Each execution uses an independent state, and the environment is reset before the next execution\.

#### A\.5Evaluation Details

All orchestration protocols are scored from their realized executions using the same evaluator\. For each runtime variant, the evaluator receives the executed tool trajectory, execution failures, resulting simulated connector state, externally recorded effects where applicable, and terminal output\.

Reference workflows represent the prescribed enterprise SOPs rather than arbitrary implementations of underspecified tasks\. Accordingly, evaluation measures fidelity to the prescribed workflow behavior, including its required operation order, as well as the resulting state and outputs\.

Why evaluate fidelity to the prescribed SOP\.Enterprise SOPs prescribe the procedure as well as the outcome\. For example, an approval may need to precede a payment, or a duplicate check may need to precede record creation\. An execution that reaches the correct final state through a different procedure can still violate compliance or audit requirements, so we treat the prescribed procedure as part of the task\. The required order is not hidden from the model: task descriptions retain the ordering constraints of the source SOP \(Appendix[A\.3](https://arxiv.org/html/2610.33375#A1.SS3)\)\. Execution Trace Accuracy contributes only half of Orchestration Accuracy; the other half, Side\-Effect Accuracy, evaluates the resulting state\. Final Accuracy evaluates only side effects and terminal output, independent of the execution trajectory\. The evaluator further normalizes runtime\-generated identifiers and compares supported queries by result semantics rather than exact form\.

Execution Trace Accuracy\.Execution Trace Accuracy is the mean of five binary components: tool selection, tool order, tool execution, required arguments, and control path\.

Tool selection requires the executed tool identities and call counts to match the reference SOP\. Tool order measures whether the realized execution follows the prescribed operation order of the reference SOP and requires equality between the executed tool\-name sequence and the reference tool\-name sequence\. Tool execution requires tool selection and tool order to pass and additionally requires that no unexpected execution failure occur\. Required arguments evaluates the required arguments associated with the reference tool calls\. Control path checks whether the realized execution follows the prescribed branch, error\-handling, and termination behavior\.

These components are evaluated jointly from the realized execution\. For example, an additional tool call may affect tool selection, tool order, tool execution, and control\-path agreement\.

Required arguments\.Argument scoring is performed on the arguments of executed tool calls after runtime values have been resolved\. Runtime\-generated identifiers covered by the benchmark normalization procedure are normalized before comparison\. For standard tool arguments, the evaluator checks the required argument fields against the corresponding reference values\.

The evaluator includes operation\-specific handling for supported Salesforce and Snowflake query arguments, where query behavior can be compared using result or state semantics\.

Side\-Effect Accuracy\.Side\-Effect Accuracy is binary and is evaluated over the state produced in the simulated connector backend together with externally recorded effects used by the corresponding verifier\. The verifier checks that required changes are present and that prohibited or unintended changes covered by the verifier are absent\.

Side\-Effect Accuracy evaluates the resulting state rather than the complete sequence of intermediate connector states\. This complements Execution Trace Accuracy, which evaluates the realized tool trajectory\.

Termination and partial executions\.Executions that terminate because of a tool failure, formatting limit, or execution budget are scored using the trajectory and simulated connector state accumulated before termination\.

Aggregation\.Each of the five Execution Trace Accuracy components is binary, and their mean forms the per\-variant Execution Trace Accuracy\. Side\-Effect Accuracy is binary\. Orchestration Accuracy is the average of Execution Trace Accuracy and Side\-Effect Accuracy for the same runtime variant\. We then average the per\-variant score across the benchmark\. Terminal\-output accuracy is reported separately\.

#### A\.6Training Data Construction

We construct the training data from a separate set of 3,000 source recipes\. Recipes are separated from theAgentOrchbenchmark by source recipe ID\.

For each source recipe, we construct a task specification following the same task\-description procedure used for the benchmark\. The source recipe serves as the reference workflow\. We deterministically convert the complete reference workflow into the serialized workflow target used for training\. Each recipe produces one training example; concrete runtime variants are not included in the training input and do not specialize the target\.

Complete workflow targets\.The training target retains the full workflow structure represented by the source recipe\. This includes both outcomes of conditional branches, loop nodes and their loop bodies, try and catch regions, tool calls, dependencies on runtime inputs and preceding tool results, and terminal control behavior\. Loops are represented as workflow structure rather than being expanded into the number of iterations observed in one runtime execution\.

Target validation\.Each target is parsed and normalized before inclusion\. The validation checks the workflow syntax and structure, references between workflow nodes, availability of the tools used by the target, and the argument schemas of those tools\. Compilation, schema, planner\-protocol, and tool\-surface checks are fail\-closed\.

These checks do not execute the serialized training target on runtime variants\. Replay of the original source recipe is not an inclusion criterion for the active 3,000\-example training set\. We therefore use structural and interface validation, rather than execution validation, to describe the validation of the training targets\.

Tool exposure\.Across the training corpus, the tool\-card pool contains 1,036 distinct tool cards spanning 306 namespaces\. Of these, 1,033 tool cards spanning 305 namespaces occur in the reference workflow targets\. Each individual training example exposes 64 tool cards\. The 64\-card set contains the tools required by that recipe together with sampled distractor tools and therefore varies across training examples\.

Train–benchmark separation\.Training and benchmark sets are disjoint at the source\-recipe level: no source recipe used in the AGENTORCH benchmark is used for training\. Both sets are drawn from the same enterprise automation environment and therefore intentionally share the same tool ecosystem and may contain recurring workflow motifs\. The split is designed to evaluate generalization to unseen source recipes and task instances, rather than transfer to unseen tools or a different automation domain\. Thus, all reported training results are evaluated on source recipes that are not observed during training\.

#### A\.7Training Details

Supervised Fine\-Tuning\.We apply supervised fine\-tuning to five base models using the training pairs described in Appendix[A\.6](https://arxiv.org/html/2610.33375#A1.SS6)\. The model receives the task specification and is trained to generate the complete reference workflow\. The loss is applied only to the workflow response; tokens in the input prompt are masked from the training loss\. We do not supervise chain\-of\-thought explanations or intermediate reasoning\. All reported SFT runs use LoRA adaptation\.

Reinforcement Learning\.We further train models using Group Relative Policy Optimization \(GRPO\)\. The reward is computed by comparing a sampled workflow directly with the serialized reference workflow\. Reward computation does not execute the sampled workflow and does not invoke either connector backend\.

A sampled output must first be parsed as a workflow\. If parsing fails, the total reward is zero\. For valid outputs, the reward combines five components:

- •Validity\(0\.15\): whether the sampled response can be parsed as a valid workflow\.
- •Tool calls\(0\.25\): we recursively flatten tool\-call nodes from the generated and reference workflows and compute the longest\-common\-subsequence similarity between their tool\-name sequences\.
- •Arguments\(0\.35\): tool calls are aligned using the tool\-name sequence alignment, after which their arguments are compared after canonical JSON serialization\. The component is normalized by the number of reference tool calls\.
- •Workflow structure\(0\.15\): we flatten the workflow\-node types, including call, conditional, loop, try/catch, transformation, and termination nodes, and compute longest\-common\-subsequence similarity between the generated and reference node\-type sequences\.
- •Exact match\(0\.10\): the complete generated workflow receives this component when its canonical JSON serialization exactly matches the reference target\.

The reward contains no execution\-based component\. In particular, it does not use connector observations, simulated state, Side\-Effect Accuracy, or the benchmark evaluator\. It also contains no separate response\-length reward\. Outputs that cannot be parsed, including outputs truncated before a valid workflow is completed, receive zero reward through the validity check\.

For GRPO, we sample eight responses per prompt and normalize advantages within each group\. The RL dataset uses 1,000 training prompts selected by prompt length to fit the configured generation limits\. Model\-specific LoRA ranks, learning rates, KL coefficients, generation limits, and optimization settings are provided in Table[9](https://arxiv.org/html/2610.33375#A1.T9)\.

Table 9:Training hyperparameters\.SettingSFTQwen RLGemma RLMethodLoRA \(rr=16,α\\alpha=32\)GRPOGRPO \+ LoRA \(rr=16,α\\alpha=32\)Learning rate1×10−41\\times 10^\{\-4\}3×10−63\\times 10^\{\-6\}\(27B\)1×10−51\\times 10^\{\-5\}Batch / rollouts16 seq\.16 / 8 rollouts8 prompts×\\times8 gen\.Max\. length81,92065,536 / 6,1444,096 completionKL coefficient–0\.0010\.001

### Appendix BMore Experimental Analysis

#### B\.1Full Results for RQ1

We provide full results of our RQ1 below:

Table 10:Scaling does not consistently narrow theGlobal–Step\-by\-Stepgap\.We report performance under theGlobalandStep\-by\-Step\(react\) settings\.Δ\\DeltadenotesStep\-by\-StepminusGlobalin percentage points \(pp\)\.Orchestration AccuracyFinal AccuracyError RateFamilyScaleGlobalStep\-by\-StepΔ\\DeltaGlobalStep\-by\-StepΔ\\DeltaGlobalStep\-by\-StepΔ\\DeltaQwen2\.50\.5B0\.13%36\.46%\+36\.330\.00%12\.45%\+12\.4599\.78%0\.22%−99\.56\-99\.561\.5B1\.66%31\.41%\+29\.750\.22%8\.91%\+8\.6996\.43%0\.25%−96\.18\-96\.187B2\.63%30\.50%\+27\.870\.47%7\.42%\+6\.9593\.45%1\.09%−92\.36\-92\.3614B5\.33%41\.54%\+36\.211\.61%21\.46%\+19\.8585\.40%0\.03%−85\.37\-85\.3732B16\.18%45\.64%\+29\.466\.40%24\.41%\+18\.0158\.14%0\.34%−57\.80\-57\.8072B9\.91%46\.57%\+36\.663\.91%25\.96%\+22\.0573\.57%0\.25%−73\.32\-73\.32Gemma\-34B6\.15%32\.75%\+26\.601\.61%10\.65%\+9\.0482\.39%0\.28%−82\.11\-82\.1112B8\.46%37\.85%\+29\.391\.83%13\.07%\+11\.2477\.17%0\.03%−77\.14\-77\.1427B7\.41%39\.93%\+32\.522\.67%17\.02%\+14\.3578\.14%0\.09%−78\.05\-78\.05

Table 11:Reasoning does not consistently improve orchestration performance\.We compare base and thinking variants under bothGlobalandStep\-by\-Step\(react\) execution\. Within each base–thinking pair and execution setting, the better\-performing score is highlighted and bolded \(higher is better for accuracy metrics; lower is better for error rate\)\. Mean accuracy is the average of orchestration, final, side\-effect, execution\-trace, and terminal\-output accuracy\.\(a\)Global

Qwen3\-4BQwen3\-30BQwen3\.8\-27BDeepSeek\-V4GLM\-4\.5\-AirGLM\-4\.5\-FP8GLM\-5\.2\-FP8Claude\-Opus\-5MetricBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkOrch\.13\.75%10\.25%8\.35%14\.75%28\.57%33\.02%27\.01%17\.14%12\.28%10\.72%12\.93%20\.09%19\.23%17\.75%46\.44%59\.88%Final3\.91%2\.70%2\.95%5\.19%12\.67%17\.05%11\.43%9\.66%4\.78%4\.25%5\.19%8\.48%8\.57%9\.29%23\.54%32\.33%Side16\.40%13\.04%9\.44%16\.83%31\.80%35\.71%29\.94%17\.52%14\.22%11\.61%13\.88%21\.46%21\.15%18\.66%50\.16%63\.98%Trace11\.11%7\.45%7\.26%12\.68%25\.33%30\.33%24\.07%16\.76%10\.34%9\.83%11\.99%18\.72%17\.31%16\.84%42\.72%55\.78%Term\.16\.30%8\.42%13\.73%20\.12%28\.98%35\.16%28\.98%20\.06%11\.71%13\.45%10\.81%19\.88%18\.94%14\.32%42\.24%48\.29%Mean Acc\.↑\\uparrow12\.29%8\.37%8\.35%13\.91%25\.47%30\.25%24\.29%16\.23%10\.67%9\.97%10\.96%17\.73%17\.04%15\.37%41\.02%52\.05%Error↓\\downarrow61\.96%77\.76%76\.43%60\.81%43\.88%39\.47%43\.51%67\.67%73\.76%74\.63%75\.68%58\.35%61\.34%73\.01%19\.16%6\.21%

\(b\)Step\-by\-Step\(react\)

Qwen3\-4BQwen3\-30BQwen3\.8\-27BDeepSeek\-V4GLM\-4\.5\-AirGLM\-4\.5\-FP8GLM\-5\.2\-FP8Claude\-Opus\-5MetricBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkBaseThinkOrch\.41\.84%47\.34%47\.81%52\.09%53\.31%63\.08%58\.99%58\.85%48\.59%49\.49%54\.47%54\.68%62\.16%62\.66%63\.52%68\.41%Final17\.30%22\.17%26\.68%28\.17%31\.77%49\.41%44\.57%45\.43%28\.88%30\.06%38\.29%37\.92%48\.20%48\.91%47\.14%55\.31%Side51\.37%57\.05%58\.57%63\.35%64\.91%72\.39%68\.26%67\.67%59\.69%59\.88%64\.13%64\.10%71\.58%72\.08%70\.09%74\.57%Trace32\.32%37\.62%37\.04%40\.82%41\.71%53\.76%49\.73%50\.03%37\.48%39\.11%44\.81%45\.26%52\.73%53\.24%56\.95%62\.25%Term\.52\.17%53\.70%57\.24%56\.55%60\.06%72\.42%71\.21%71\.43%59\.88%58\.14%67\.89%67\.08%72\.98%72\.48%72\.36%77\.55%Mean Acc\.↑\\uparrow39\.00%43\.58%45\.47%48\.20%50\.35%62\.21%58\.55%58\.68%46\.90%47\.34%53\.92%53\.81%61\.53%61\.87%62\.01%67\.62%Error↓\\downarrow0\.06%0\.16%0\.19%0\.06%0\.16%0\.71%0\.19%1\.68%0\.12%4\.04%0\.09%0\.19%0\.12%0\.53%0\.00%0\.00%

#### B\.2Additional Analyses for RQ3

We examine how feedback scope affectsReplangains, how these gains extend to held\-out runtime variants, and which failures are resolved by workflow revision\. Mean Accuracy averages Orchestration, Final, Side\-Effect, Execution Trace, and Terminal Output Accuracy\.

Two\-Variant Feedback Produces Similar Average Gains to All\-Variant Feedback\.We compare feedback from two variants of each SOP with feedback from all variants, evaluating both settings on all variants\. Two\-variant feedback improves Mean Accuracy by 5\.49 pp on average, 0\.05 pp more than all\-variant feedback\. It reduces Error Rate by 12\.07 pp, 0\.17 pp less than all\-variant feedback \(Table[12](https://arxiv.org/html/2610.33375#A2.T12)\)\.

Table 12:Effect of limitingReplanfeedback to two runtime variants per SOP\.All columns evaluate on all runtime variants\. Gaps compare two\-variant feedback with all\-variant feedback\. Error reduction isErrorR​0−ErrorR​3\\text\{Error\}\_\{R0\}\-\\text\{Error\}\_\{R3\}\. All values are in pp\.ModelTwo\-Var\.Δ\\DeltaMean Acc\.Gap vs\. All\-Var\.Two\-Var\. Error Red\.Gap vs\. All\-Var\.Qwen3\.8\-27B\+10\.42\+0\.2114\.00\-0\.53DeepSeek\-V4\-Flash\+6\.88\-0\.0213\.32\-0\.07Mistral\-Small\-3\.1\-24B\+7\.02\+0\.1617\.08\+0\.28Gemma\-4\-26B\-A4B\+5\.12\+0\.2711\.55\-0\.62Gemma\-4\-12B\+4\.70\+0\.5012\.52\+1\.31Qwen3\-4B\+4\.13\-0\.0213\.14\-0\.34Qwen3\-30B\-A3B\+2\.72\-0\.735\.81\-1\.37Gemma\-3\-27B\+2\.90\+0\.039\.16\+0\.00Average\+5\.49\+0\.0512\.07\-0\.17

Held\-Out Variants Improve as Much as Feedback Variants\.Under two\-variant feedback, we compare the improvement on the variants that provided feedback with that on held\-out variants\. The average held\-out minus seen difference is\+0\.03\+0\.03pp for Mean Accuracy and−0\.31\-0\.31pp for Error Rate reduction \(Table[13](https://arxiv.org/html/2610.33375#A2.T13)\)\. The revised workflow thus improves as much on variants it never observed as on those used for feedback\.

Table 13:Transfer from feedback variants to held\-out variants\.Replanfeedback uses two variants of each SOP\. The transfer gap is the held\-out value minus the seen value; values near zero indicate similar improvement on both subsets\.ModelSeenΔ\\DeltaMean Acc\.Held\-out–Seen GapSeen Error Red\.Held\-out–Seen GapQwen3\.8\-27B\+10\.81\-0\.6514\.55\-0\.91DeepSeek\-V4\-Flash\+7\.07\-0\.3112\.92\+0\.67Mistral\-Small\-3\.1\-24B\+7\.59\-0\.9618\.13\-1\.74Gemma\-4\-26B\-A4B\+5\.38\-0\.4312\.29\-1\.23Gemma\-4\-12B\+4\.45\+0\.4212\.92\-0\.67Qwen3\-4B\+3\.63\+0\.8313\.16\-0\.03Qwen3\-30B\-A3B\+2\.32\+0\.665\.30\+0\.85Gemma\-3\-27B\+2\.52\+0\.648\.79\+0\.61Average\+5\.47\+0\.0312\.26\-0\.31

Gains Also Hold on Two Held\-Out Variants per SOP\.Evaluating on a subset of two held\-out variants per SOP, Mean Accuracy improves for all eight models by 5\.34 pp on average, and Error Rate decreases by 12\.31 pp \(Table[14](https://arxiv.org/html/2610.33375#A2.T14)\)\. The transfer result therefore does not depend on aggregating over all held\-out variants\.

Table 14:Replanperformance on a subset of two held\-out variants per SOP\.Feedback uses two variants of each SOP; evaluation uses two other variants never used for feedback\. Base is Round 0 and Final is Round 3\.ModelMean Accuracy↑\\uparrowError Rate↓\\downarrowBaseFinal𝚫\\bm\{\\Delta\}BaseFinal𝚫\\bm\{\\Delta\}Qwen3\.8\-27B26\.2736\.13\+9\.8642\.9628\.43\-14\.53DeepSeek\-V4\-Flash24\.7231\.05\+6\.3242\.8729\.69\-13\.18Mistral\-Small\-3\.1\-24B13\.3120\.40\+7\.0967\.3548\.79\-18\.56Gemma\-4\-26B\-A4B4\.129\.20\+5\.0988\.8877\.31\-11\.57Gemma\-4\-12B12\.1716\.89\+4\.7265\.6553\.09\-12\.56Qwen3\-4B12\.8016\.92\+4\.1260\.8147\.26\-13\.55Qwen3\-30B\-A3B8\.3210\.93\+2\.6175\.9670\.67\-5\.29Gemma\-3\-27B6\.939\.83\+2\.9179\.1069\.87\-9\.23Average13\.5818\.92\+5\.3465\.4553\.14\-12\.31

Improvements Are Widespread Across Held\-Out Variants\.To check whether the average gain is driven by a few variants, we count held\-out variants whose score changes from Round 0 to Round 3\. For every model, improvements outnumber regressions on both Orchestration and Final Accuracy\. In total across models, Orchestration Accuracy improves on 2,214 held\-out variants and decreases on 165, while Final Accuracy improves on 555 and decreases on 75 \(Table[15](https://arxiv.org/html/2610.33375#A2.T15)\)\.

Table 15:Variant\-level improvements and regressions on held\-out variants\.Counts of held\-out variants whose score increases \(Up\) or decreases \(Down\) from Round 0 to Round 3 under two\-variant feedback\. Unchanged variants are omitted\.ModelOrch\. UpOrch\. DownFinal UpFinal DownQwen3\.8\-27B428311619DeepSeek\-V4\-Flash3355111026Mistral\-Small\-3\.1\-24B357316812Gemma\-4\-26B\-A4B240125610Gemma\-4\-12B2646494Qwen3\-4B24912416Qwen3\-30B\-A3B15512435Gemma\-3\-27B18610273Total221416555575Most Repaired Workflows Initially Fail Schema, Type, or Reference Checks\.Across the eight models, 641 workflows are invalid at Round 0 and become valid by Round 3\. Of these, 538 \(83\.9%\) initially parse successfully but fail schema, type, or reference validation\. These checks cover constraints on tool arguments and references between workflow steps\. The remaining 103 workflows \(16\.1%\) fail response\-parsing or plan\-structure checks \(Table[16](https://arxiv.org/html/2610.33375#A2.T16)\)\. The share of parsing or plan\-structure failures is highest for Gemma\-4\-26B\-A4B, at 39\.8%\. Together, these checks detect the initial failures in all 641 repaired workflows before execution\.

Table 16:Failure types of invalid workflows repaired byReplan\.Repaired workflows are invalid at Round 0 and valid by Round 3\. Parse/Shape: failures in response parsing or plan\-structure checks\. Validator: workflows that parse successfully but fail schema, type, or reference validation\. Together, these checks detect all 641 repaired workflows before execution\.ModelRepairedParse / ShapeValidatorValidator \(%\)Qwen3\.8\-27B97168183\.5DeepSeek\-V4\-Flash84127285\.7Mistral\-Small\-3\.1\-24B1171310488\.9Gemma\-4\-12B7997088\.6Qwen3\-4B85137284\.7Qwen3\-30B\-A3B4053587\.5Gemma\-3\-27B5625496\.4Gemma\-4\-26B\-A4B83335060\.2All models64110353883\.9##### Summary\.

Replangains require feedback from only two variants per SOP, extend equally to variants never used for feedback, and are spread across most held\-out variants rather than concentrated in a few\. The invalid workflows it repairs mostly parse correctly but specify tool arguments or references between steps incorrectly, and all of these defects are detectable before execution\. Consistent with the main text, these results identify workflow validity, particularly correct arguments and references, as a major bottleneck in global orchestration\.

### Appendix CAdditional Related Work

Workflow generation and process automation\.The closest work studies workflow generation and automation\. WorfBench\([Qiao et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib24)\), WorkflowLLM\([Fan et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib7)\), AutoFlow\([Li et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib18)\), AFlow\([Zhang et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib50)\), FlowMind\([Zeng et al\., 2023](https://arxiv.org/html/2610.33375#bib.bib49)\), Chat2Workflow\([Zhong et al\., 2026](https://arxiv.org/html/2610.33375#bib.bib51)\), and AutomationBench\([Shepard & Salimans, 2026](https://arxiv.org/html/2610.33375#bib.bib32)\)consider agentic pipelines, visual workflows, RPA\-style procedures, or cross\-application automation\. Earlier work studies IFTTT\-style trigger\-action programming\([Quirk et al\., 2015](https://arxiv.org/html/2610.33375#bib.bib26);[Chaurasia & Mooney, 2017](https://arxiv.org/html/2610.33375#bib.bib3);[Ur et al\., 2016](https://arxiv.org/html/2610.33375#bib.bib41)\), while recent business\-process research explores language\-based process modeling and generation\([Kourani et al\., 2024](https://arxiv.org/html/2610.33375#bib.bib13);[Busch & Leopold, 2024](https://arxiv.org/html/2610.33375#bib.bib2);[Kourani et al\., 2026](https://arxiv.org/html/2610.33375#bib.bib14);[Li et al\., 2025](https://arxiv.org/html/2610.33375#bib.bib17)\)\. Our work differs by treating complete workflow generation as theGlobalorchestration paradigm and directly comparing it withStep\-by\-Stepinteraction on the same executable enterprise workflows, including connector contracts, parameter bindings, dependencies, transformations, branching, looping, and observable side effects\. We further study how global orchestration can be improved through execution feedback and SOP\-grounded supervision\.

### Appendix DSimulator Fidelity via Reference\-Action Replay

Model\-generated workflows inAgentOrchare evaluated using the simulated connector backend, while reference executions are collected from the corresponding real connectors where supported\. We therefore validate whether the simulator preserves the benchmark\-relevant behavior observed in the real\-backend reference executions\.

Specifically, we replay the recorded reference tool calls from each real\-backend execution one action at a time in the simulated backend, without any model planning\. The resulting executions are scored using the same evaluation pipeline as model\-generated runs, including the same normalization of dynamic identifiers, timestamps, argument values, and query semantics\.

Table[17](https://arxiv.org/html/2610.33375#A4.T17)reports the agreement between these simulated replays and the stored real\-backend reference executions\. The simulator achieves 97\.20% agreement on tool selection, tool order, tool execution, and control path, 99\.97% on required arguments, 97\.75% Execution Trace Accuracy, and 99\.32% Side\-Effect Accuracy\. The resulting Orchestration Accuracy is 98\.54%\.

Table 17:Simulator fidelity under reference\-action replay\. Recorded tool calls from the real\-backend reference executions are replayed one action at a time in the simulated backend and evaluated using the same benchmark evaluator as model\-generated runs\.MetricReference\-Action ReplayTool Selection97\.20%Tool Order97\.20%Tool Execution97\.20%Required Arguments99\.97%Control Path97\.20%Execution Trace Accuracy97\.75%Side\-Effect Accuracy99\.32%Orchestration Accuracy98\.54%These results show that the simulated backend reproduces the benchmark\-relevant execution behavior of the recorded real\-backend references with high fidelity\. This makes broad simulator–reference mismatch an unlikely explanation for the observed performance gap betweenGlobalandStep\-by\-Steporchestration\.

### Appendix EPerformance Across Workflow Characteristics

We next examine whether the gap betweenGlobalandStep\-by\-Stepis concentrated in particular types of workflows\. We characterize each SOP using its reference workflow\. Workflow size is measured by the number of tool\-call nodes, with short, medium, and long workflows defined as 1–2, 3–5, and at least 6 calls, respectively, based on the quartiles of the benchmark distribution\. We additionally record whether the workflow contains conditional branches, loops, or try/catch regions, and measure its maximum nesting depth\. To characterize cross\-application workflows, we count the number of distinct application families used by tool\-call nodes in the reference workflow, excluding runtime\-helper tools\. Fault injection is defined at the runtime\-variant level\.

We report results over the 31 models for which bothGlobalandStep\-by\-Stepevaluations are complete\. Fine\-tuned checkpoints are excluded from this analysis\. Failed executions remain in the evaluation and receive zero accuracy, following the main benchmark evaluation\. The gap is defined asStep\-by\-StepOrchestration Accuracy minusGlobalOrchestration Accuracy\. Confidence intervals are computed using a bootstrap clustered by SOP\.

Table 18:Performance across workflow characteristics\. Orchestration Accuracy and Error Rate are reported in percent\. The gap isStep\-by\-StepminusGlobalOrchestration Accuracy in percentage points\. Confidence intervals are 95% bootstrap intervals clustered by SOP\. Workflow characteristics are derived from the reference workflow, except fault injection, which is defined per runtime variant\.Subset\# SOP\# Var\.GLOBALSTEP\-BY\-STEPGap95% CIGLOBAL Err\.STEP Err\.All workflowsAll650322013\.846\.4\+32\.6\[31\.2, 34\.2\]70\.70\.5Workflow sizeShort \(1–2 calls\)234103518\.845\.2\+26\.3\[24\.5, 28\.4\]60\.90\.5Medium \(3–5 calls\)251114914\.247\.3\+33\.1\[30\.9, 35\.1\]69\.30\.5Long \(≥\\geq6 calls\)16510368\.346\.6\+38\.3\[35\.1, 41\.3\]82\.00\.7Number of application families1230116718\.851\.2\+32\.4\[30\.3, 34\.7\]65\.40\.42360167611\.744\.2\+32\.5\[30\.4, 34\.8\]72\.10\.63\+553525\.939\.4\+33\.4\[28\.5, 38\.3\]82\.40\.7Conditional branchesNo conditional303123914\.240\.7\+26\.5\[24\.6, 28\.3\]66\.50\.6Has conditional347198113\.650\.0\+36\.4\[34\.3, 38\.4\]73\.30\.5LoopsNo loop511241315\.647\.1\+31\.5\[29\.9, 33\.1\]67\.90\.4Has loop1398078\.544\.3\+35\.7\[32\.5, 39\.2\]78\.90\.8Try/catch regionsNo try/catch413192913\.743\.2\+29\.5\[27\.7, 31\.4\]67\.90\.6Has try/catch237129114\.051\.2\+37\.2\[34\.7, 39\.6\]74\.80\.5Control\-flow compositionLinear18678114\.439\.7\+25\.3\[22\.7, 28\.0\]64\.20\.5Branch only13268814\.948\.6\+33\.6\[30\.2, 36\.9\]69\.40\.5Loop only361488\.930\.9\+22\.0\[18\.6, 26\.2\]70\.40\.7Error handling only7829516\.648\.8\+32\.2\[28\.6, 36\.1\]69\.60\.5Multiple control\-flow types218130812\.850\.4\+37\.7\[35\.2, 40\.4\]75\.50\.6Maximum nesting depth118678114\.439\.7\+25\.3\[22\.9, 28\.0\]64\.20\.52235103015\.047\.8\+32\.8\[30\.4, 35\.3\]69\.30\.43\+229140912\.649\.0\+36\.5\[34\.2, 38\.8\]75\.30\.6Fault injectionNo injected fault650295613\.645\.8\+32\.2\[30\.7, 33\.8\]70\.40\.5Injected fault22426416\.452\.8\+36\.5\[33\.8, 39\.3\]74\.20\.6

##### The gap persists across workflow characteristics\.

As shown in Table[18](https://arxiv.org/html/2610.33375#A5.T18),Step\-by\-Stepachieves higher Orchestration Accuracy thanGlobalin every reported subgroup\. This pattern also holds at the model level: all 45 evaluated models have a positiveStep\-by\-Step–Globalgap in each reported subgroup\. Thus, the aggregate gap is not attributable to a single workflow\-size, application\-count, or control\-flow subgroup\.

##### The gap increases with workflow size\.

The Orchestration Accuracy gap increases from 26\.3 percentage points \(pp\) for short workflows to 33\.1 pp for medium workflows and 38\.3 pp for long workflows\. This increase is driven primarily by lowerGlobalperformance as workflow size increases:Globaldecreases from 18\.8% on short workflows to 8\.3% on long workflows, whileStep\-by\-Stepremains between 45\.2% and 47\.3% across the three size groups\. TheGlobalError Rate also increases from 60\.9% to 82\.0% across these groups, whereas theStep\-by\-StepError Rate remains below 1%\.

##### Control\-flow structure shows a similar pattern\.

Workflows containing conditional branches have a 36\.4 pp gap, compared with 26\.5 pp for workflows without conditionals\. The corresponding gaps are 35\.7 pp versus 31\.5 pp for workflows with and without loops, and 37\.2 pp versus 29\.5 pp for workflows with and without try/catch regions\. When workflows are grouped by control\-flow composition, linear workflows have a 25\.3 pp gap, while workflows containing multiple control\-flow types have a 37\.7 pp gap\. The same pattern appears with nesting depth: the gap is 25\.3 pp at depth 1, 32\.8 pp at depth 2, and 36\.5 pp at depth 3 or greater\. Together, these stratifications show that theStep\-by\-Step–Globalgap is larger in workflows with greater structural complexity, as measured by workflow size, control\-flow composition, and nesting depth\.

##### Application count does not explain the gap\.

The gap is 32\.4 pp for workflows using one application family, 32\.5 pp for two application families, and 33\.4 pp for three or more application families\. Both protocols obtain lower Orchestration Accuracy as the number of application families increases, but their difference remains similar across these groups\. Thus, the larger gaps observed for longer and more structurally nested workflows are not reproduced by application count alone\.

##### The gap also persists under fault injection\.

For runtime variants without injected faults, the gap is 32\.2 pp, compared with 36\.5 pp for fault\-injected variants\. Both settings therefore preserve the main comparison betweenGlobalandStep\-by\-Step\. We do not interpret the higher absolute accuracy on fault\-injected variants as an effect of fault injection because the two variant groups differ in the workflows and execution paths they contain\.

Overall, these results show that the advantage ofStep\-by\-StepoverGlobalis not concentrated in a particular workflow subgroup\. The gap is present across workflow sizes, application counts, control\-flow structures, nesting depths, and fault\-injection settings, and is larger under several measures of workflow structural complexity\.

### Appendix FInference Cost

Globalconstructs a complete workflow before execution, whereasStep\-by\-Steprepeatedly invokes the model as execution progresses\. Table[19](https://arxiv.org/html/2610.33375#A6.T19)compares the resulting inference usage across open\-weight and closed\-source models\.

LLM calls\.The reduction in repeated inference is consistent across model families\. Across the models in Table[19](https://arxiv.org/html/2610.33375#A6.T19),Step\-by\-Stepuses15\.4×15\.4\\times–35\.4×35\.4\\timesmore LLM calls thanGlobal\. For example, Qwen3\-30B\-A3B increases from 650 calls underGlobalto 22,997 underStep\-by\-Step, while Claude Opus 5 Thinking increases from 650 to 11,964 calls\. On a per\-variant basis,Globaluses approximately 0\.2 calls, compared with 3\.1–7\.1 calls forStep\-by\-Stepamong the models shown\.

Completion tokens\.The larger number of calls also increases total generation\. Across the models in Table[19](https://arxiv.org/html/2610.33375#A6.T19),Step\-by\-Stepgenerates1\.42×1\.42\\times–5\.29×5\.29\\timesmore completion tokens thanGlobal\. For Qwen3\.8\-27B, completion\-token usage increases from 1\.07M underGlobalto 5\.67M underStep\-by\-Step\. Claude Opus 5 Thinking increases from 0\.91M to 1\.30M, while GPT\-5\.6 Sol Thinking increases from 0\.59M to 1\.40M\.

IndividualGlobalcalls are substantially longer because each call generates a complete workflow\. Qwen3\.8\-27B, for example, generates 1,652 completion tokens per call underGlobal, compared with 323 underStep\-by\-Step\. This pattern holds broadly across the models in the table:Globaluses fewer, longer calls, whereasStep\-by\-Stepuses many more, shorter calls\. The substantially larger number ofStep\-by\-Stepcalls nevertheless results in higher total completion\-token usage\.

Summary\.Globalsubstantially reduces inference usage: across the evaluated models,Step\-by\-Steprequires15\.4×15\.4\\times–35\.4×35\.4\\timesmore LLM calls and1\.42×1\.42\\times–5\.29×5\.29\\timesmore completion tokens\.

Total LLM CallsCalls / VariantCompletion Tokens \(M\)Tokens / VariantTokens / CallModelGlobalStep×\\timesGlobalStepGlobalStep×\\timesGlobalStepGlobalStepOpen\-weight modelsGemma\-3\-12B64912,09818\.6×\\times0\.203\.760\.541\.923\.55×\\times167\.8595\.7833159Gemma\-3\-27B64914,25322\.0×\\times0\.204\.430\.952\.492\.62×\\times295\.8774\.31,468175Gemma\-4\-12B64913,88821\.4×\\times0\.204\.311\.183\.292\.79×\\times365\.11,020\.31,812237Gemma\-4\-26B\-A4B64917,19826\.5×\\times0\.205\.340\.954\.104\.30×\\times296\.21,272\.91,470238Gemma\-4\-31B64912,48419\.2×\\times0\.203\.881\.162\.892\.48×\\times360\.9896\.31,791231GLM\-4\.5\-Air65022,33834\.4×\\times0\.206\.940\.854\.214\.98×\\times262\.61,307\.11,301188GLM\-4\.565015,48823\.8×\\times0\.204\.810\.883\.393\.87×\\times272\.01,053\.71,348219GLM\-5\.265015,22823\.4×\\times0\.204\.730\.812\.553\.13×\\times252\.8791\.81,252167Llama\-3\.3\-70B65014,44022\.2×\\times0\.204\.480\.721\.592\.23×\\times222\.5495\.31,102110Qwen2\.5\-72B65017,29026\.6×\\times0\.205\.370\.992\.212\.24×\\times307\.1686\.91,521128Qwen3\-30B\-A3B65022,99735\.4×\\times0\.207\.141\.133\.713\.28×\\times350\.71,151\.31,737161Qwen3\.8\-27B64917,53927\.0×\\times0\.205\.451\.075\.675\.29×\\times332\.91,759\.61,652323Closed\-source modelsGPT\-3\.5 Turbo65012,74219\.6×\\times0\.203\.960\.451\.934\.30×\\times139\.7600\.1692152GPT\-4o65010,04215\.4×\\times0\.203\.120\.681\.081\.60×\\times210\.5336\.11,043108Claude Opus 5 Thinking65011,96418\.4×\\times0\.203\.720\.911\.301\.42×\\times284\.0402\.91,407108GPT\-5\.6 Sol Thinking64711,31217\.5×\\times0\.203\.510\.591\.402\.37×\\times184\.3436\.2917124

Table 19:Inference usage under global and step\-by\-step orchestration\.The×\\timescolumns reportStep\-by\-Step/Global\. For closed\-source models, completion\-token counts are estimated from retained output text using median characters\-per\-token ratios measured from the corresponding non\-thinking runs; all other quantities are directly measured\.
### Appendix GHuman Evaluation\.

Source Recipes\.AgentOrchis constructed from sanitized executable workflows instantiated in a Workato automation environment prior to benchmark construction\. Customer\-identifying information is removed before benchmark artifacts are derived from these workflows\.

Human Review of Task Descriptions\.Four authors participated in an iterative manual validation process covering all benchmark task descriptions\. In each round, task descriptions were reviewed against their corresponding source workflows for omitted workflow requirements, unsupported requirements, and disclosure of information excluded from the model’s task specification\. Reviews were cross\-checked across authors, and descriptions with identified issues were regenerated and subsequently re\-evaluated\. In the first review round, approximately 57% of the task descriptions required regeneration\. We repeated this review–regeneration–validation procedure for five rounds, until all task descriptions passed the review criteria\.

Execution\-Based and Human Validation of Runtime Variants\.For every generated runtime variant, we executed the corresponding reference workflow in the simulated environment and manually inspected the resulting execution behavior\. The execution\-based check verified that the runtime input and environment configuration supported the intended workflow execution, while human review examined the resulting tool outputs, state changes, control\-flow behavior, and terminal outputs for consistency with the source workflow and intended runtime condition\. In the first validation round, approximately 38\.5% of generated runtime variants passed these checks\. Variants that failed either the execution\-based or human validation criteria were regenerated and subsequently re\-executed and re\-reviewed\. This generate–execute–review cycle was repeated until all retained runtime variants passed the predefined validation criteria\. Because the benchmark infrastructure provides a local simulated execution environment with the same tool interfaces and benchmark\-relevant connector semantics as the corresponding real connectors, we were able to inspect the execution outcome of every runtime variant rather than relying only on static input checks\.

### Appendix HLimitations

Scope\.AgentOrchtargets structured enterprise workflows, where procedures are prescribed in advance and the same workflow is executed repeatedly on different inputs\. This is the setting in which global orchestration is most practical\. In GUI\-based or open\-ended environments, a complete executable workflow often cannot be specified before execution, and we leave the comparison in such settings to future work\. Our workflows are derived from sanitized, executable Workato recipes spanning 64 tool interfaces across services such as Slack, Jira, Salesforce, and Snowflake; real enterprise workflows of this kind are rarely publicly available\. Training and benchmark sets are disjoint by source recipe but drawn from the same enterprise automation environment\. This mirrors deployment, where an organization trains on its existing workflows and applies the model to new workflows over the same tool ecosystem\. The training set is also broader than the benchmark, covering 1,033 tool operations across 305 namespaces with only partial overlap with the benchmark tools\. Transfer to unrelated automation domains remains to be studied\.

Protocol comparison\.We compareGlobalandStep\-by\-Stepas deployed paradigms, so they differ in both when decisions are made and what information is available\. Our additional protocols vary these factors separately: providing concrete runtime inputs alone leaves the gap largely unchanged, incremental construction adds step\-wise decisions without execution information, and execution\-feedback revision adds execution information without runtime decisions\. A complete separation of the two factors is not possible, because using intermediate results at runtime without new decisions is precisely what a global workflow does through references between steps\.

Evaluation\.Our primary metric measures fidelity to the prescribed SOP, including its operation order, because enterprise procedures constrain how a task is carried out, not only its outcome\. Final Accuracy, which evaluates only side effects and terminal output, complements this metric, and the gap and the gains from revision and training also hold under it\. Model\-generated workflows are executed in a simulated backend, since executing them against live enterprise services would produce irreversible side effects; reference\-action replay shows high agreement between the simulator and the real connectors \(Appendix[D](https://arxiv.org/html/2610.33375#A4)\)\.

### Appendix IFull Results for Figure[1](https://arxiv.org/html/2610.33375#S1.F1)

Table[20](https://arxiv.org/html/2610.33375#A9.T20)reports the Orchestration Accuracy of all 31 models shown in Figure[1](https://arxiv.org/html/2610.33375#S1.F1)underGlobalorchestration andStep\-by\-Stepinteraction, from which the per\-model gaps in the figure are computed\.

Table 20:Orchestration accuracy under GLOBAL orchestration and STEP\-BY\-STEP \(ReAct\) interaction\. Results are reported as percentages\.ModelGLOBAL \(%\)STEP\-BY\-STEP / ReAct \(%\)GPT\-3\.5 Turbo10\.1150\.78GPT\-4o28\.0950\.12Llama 3\.1 8B4\.4534\.82Llama 3\.1 70B10\.8138\.92Mistral Nemo2\.2634\.27Command R6\.6435\.97Qwen 2\.5 0\.5B0\.1336\.46Qwen 2\.5 1\.5B1\.6631\.41Qwen 2\.5 3B2\.1934\.16Qwen 2\.5 7B2\.6330\.50Qwen 2\.5 14B5\.3341\.54Qwen 2\.5 32B16\.1845\.64Qwen 2\.5 72B9\.9146\.57Llama 3\.2 1B0\.0036\.93Llama 3\.2 3B2\.0735\.50Granite 3\.1 2B6\.3031\.83Granite 3\.1 8B6\.5234\.19Llama 3\.3 70B10\.4542\.98Gemma 3 4B6\.1532\.75Gemma 3 12B8\.4637\.85Gemma 3 27B7\.4139\.93Mistral Small 3\.1 24B14\.1544\.71Qwen 3 4B13\.7541\.84Qwen 3 30B8\.3547\.81GLM\-4\.5 Air FP812\.2848\.59GLM\-4\.5 FP812\.9354\.47DeepSeek V4 Flash27\.0158\.99Qwen 3\.8 27B FP828\.5753\.31Claude Opus 546\.4463\.52GPT\-5\.6 Sol Thinking51\.7666\.88Claude Opus 5 Thinking59\.8868\.41