Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

# Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems
Source: [https://arxiv.org/html/2607.08010](https://arxiv.org/html/2607.08010)
Kalle Kujanpää Ning Liu Shahnawaz Alam Yeshwanth Reddy Sura Tianyu Yang Kristina Klinkner Shervin Malmasi

Amazon, Fulfillment Technologies & Robotics \{kujanpaa,malmasi\}@amazon\.com

###### Abstract

Production LLM agents often waste latency and reliability by regenerating code for the same procedural steps on every request\. We replace this inference\-time coding loop with an agentic tool\-making pipeline that compiles repeated SOP steps into validated, versioned tools before deployment\. The tool\-maker grounds synthesis in the live environment as it collects execution traces, observes backend schemas and values, generates candidate tools, and repairs them against labeled cases\. At runtime, the production agent calls these tools directly and falls back to code generation only when needed\. We deploy the approach in a Fulfillment Center alarm\-triage system, where an agent diagnoses alarms against a 44\-node SOP over heterogeneous metric backends\. In production, tool calls reduce p50 latency by 42%\. On 1,500 historical alarms, they reduce end\-to\-end error rate by up to 53% by suppressing run\-to\-run variance in repeated steps\. Because tools return compact structured verdicts, they also enable a simpler direct\-call architecture, reducing p50 latency by a further 62% in a controlled ablation\. Versioned tools also improve auditability and expose specification gaps and upstream data drift\. Our results show that self\-evolving agents can make industrial LLM systems faster, more reliable, and easier to operate\.

Tool\-Making and Self\-Evolving LLM Agents in Low\-Latency Systems

Kalle Kujanpää Ning Liu Shahnawaz Alam Yeshwanth Reddy SuraTianyu Yang Kristina Klinkner Shervin MalmasiAmazon, Fulfillment Technologies & Robotics\{kujanpaa,malmasi\}@amazon\.com

## 1Introduction

LLM coding agents are becoming a practical interface to complex software and operational systems\. Instead of a fixed API or hand\-written automation per task, these agents read natural\-language instructions, write and execute code, and iterate until they produce an answer or action\. This flexibility has enabled deployments in on\-call incident triage\(Chenet al\.,[2024b](https://arxiv.org/html/2607.08010#bib.bib8); Wanget al\.,[2024b](https://arxiv.org/html/2607.08010#bib.bib9)\), workflow\-guided operations\(Yeet al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib10)\), and general code\-based tool use\(Wanget al\.,[2024a](https://arxiv.org/html/2607.08010#bib.bib11)\)\. In industry settings, many procedures are documented as standard operating procedures \(SOPs\), while the data needed to execute them resides in metric stores, logs, dashboards, and ticketing systems\.

Online: agent runtimeBeforeAfterTriggerMain agentSub\-agent3–10 per alarmActionTriggerMain agentTool3–10 per alarmActionOffline tool makingSOP nodespec textData\-collectorsub\-agentTool\-makerLLMTestlabeled casesReflectorLLMToolPassRepairdeployedFigure 1:Offline \(top\), each SOP node is compiled into a tool: a data\-collector sub\-agent runs against MCP, producing a trace; a tool\-maker LLM writes a candidate from it; and a reflector–tool\-maker loop repairs the candidate against labeled cases\. Online \(bottom\), the baseline sub\-agent writes fresh MCP\-querying code per node over 10–20 LLM turns, whereas our agent calls the compiled tools inline, one call per node\.This flexibility, however, creates an efficiency and reliability bottleneck\. In the prevailing CodeAct\-style paradigm\(Wanget al\.,[2024a](https://arxiv.org/html/2607.08010#bib.bib11)\), the agent generates and executes fresh code for each request at inference time\. When the same workflow is repeated against a stable backend, the agent re\-interprets the same instruction, rediscovers the same schema, and regenerates similar code, raising latency, cost, and run\-to\-run variance\. Prior benchmarks show agents still struggle with workflow\-guided tasks\(Nandiet al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib12); Wanget al\.,[2025b](https://arxiv.org/html/2607.08010#bib.bib13); Riddellet al\.,[2026](https://arxiv.org/html/2607.08010#bib.bib14)\), and even SOP\-enhanced multi\-agent systems leave a substantial accuracy gap on operational diagnostics\(Peiet al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib15)\)\. In our setting, most latency and correctness errors are caused by translating underspecified SOP text into a concrete query against a production metric backend\.

This has motivated a shift from inference\-time coding agents to self\-evolving agents: systems that improve their own action space over time by constructing reusable tools, skills, or programs before they are needed in production\. Recent work explores this through single\-pass tool synthesis from examples or task descriptions\(Caiet al\.,[2024](https://arxiv.org/html/2607.08010#bib.bib17); Yuanet al\.,[2024](https://arxiv.org/html/2607.08010#bib.bib18); Wanget al\.,[2024c](https://arxiv.org/html/2607.08010#bib.bib19)\), iterative build\-time construction with environmental or test\-based validation\(Wanget al\.,[2023](https://arxiv.org/html/2607.08010#bib.bib20),[2024d](https://arxiv.org/html/2607.08010#bib.bib21); Wölfleinet al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib23)\), and adjacent operational documents such as policies\(Zwerdlinget al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib24)\)and troubleshooting guides\(Maoet al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib25)\)\. The key idea is to amortize cost: a procedural step executed repeatedly is compiled into a tested tool once, then invoked cheaply and consistently at runtime\.

We study this approach in a production agent at a Fulfillment Center \(FC\), where SOPs prescribe how alarms on robotic induct, conveyance, and sortation subsystems are diagnosed and acted upon\. These SOPs branch on thresholds from heterogeneous metric backends and select interventions from low\-cost flow reallocations to high\-cost staffing changes, ticket escalations, and equipment standbys\. Latency and correctness are tightly coupled: failures cascade across subsystems within minutes, while an expensive action taken against the wrong cause is difficult to reverse\. Manual execution by floor operators is slow under concurrent alarms and error\-prone given the number of metrics, schemas, and conditional branches\.

We report the design and production deployment of an agentic tool\-making pipeline for this setting, triaging outbound dock alarms against a written SOP\. The system has two modes \(Figure[1](https://arxiv.org/html/2607.08010#S1.F1)\)\. At build time, a tool\-maker constructs a library of per\-SOP\-step tools against the production environment, observing schemas and execution traces and validating candidates against labeled cases rather than relying on SOP text alone\. At inference time, the agent invokes these tools per alarm, falling back to code generation only when a tool is unavailable or fails\. This preserves the flexibility of coding agents while moving repeated, latency\-sensitive work into a reusable and auditable tool library\.

We make two main contributions: an agentic tool\-making pipeline that compiles repeated SOP steps into validated tools, and its integration into a deployed alarm\-triage agent, where it cuts p50 latency by 42%, enables an architectural redesign that cuts it a further 62% in a controlled ablation, and reduces the end\-to\-end error rate by up to 53%\. We also report three findings\. The data\-collection trace and test\-repair loop are both necessary\. Residual errors then concentrate on underspecified SOP steps, where targeted fixes raise pass@1 from 94\.5% to 99\.9%\. Versioned tools with stable inputs and outputs reveal pre\-existing upstream inconsistencies that run\-to\-run variation otherwise hides\. Our results suggest that agents that compile their own experience into reusable tools are a practical path for bringing coding\-agent flexibility to latency\-sensitive industrial workflows\.

## 2Production Setting and Task

An outbound dock is the final operation in a fulfillment center, where packages are sorted and loaded before they leave for downstream logistics\. The dock is tightly coupled: a degraded subsystem backs up the operations feeding it and cascades across the dock within minutes, amplifying a localized fault into an outsized delay that threatens delivery commitments\. The agent triages alarms such as robotic stations idling due to upstream conveyance blockages, or producing elevated error rates from camera, sensor, label, or packaging defects\.

The agent follows an SOP encoded as a decision tree of 44 decision nodes and 19 action nodes\. Decision nodes are of three kinds\. Observations test whether a metric crosses an alert threshold and serve as alarm entry points, root causes identify the underlying failure, and constraints gate the most expensive actions\. Actions range from cheap \(process alerts, flow reallocation\) to costly \(equipment shutdown, staffing changes\)\. Each node specifies which data to query and how to process it\.

The agent evaluates observations to localize the failure, reaches candidate root causes, and executes the action\. Before shutting down a robot with a faulty suction cup, for example, constraint nodes verify that the shutdown is either safe \(remaining capacity covers the item flow\) or unavoidable \(the error rate makes operation worse\)\.

## 3Method

We build on the hierarchical multi\-agent architecture of Eluna\(Liuet al\.,[2026b](https://arxiv.org/html/2607.08010#bib.bib56)\)where a main agent traverses the decision tree and delegates each evaluated node to a CodeAct\-style sub\-agent\(Wanget al\.,[2024a](https://arxiv.org/html/2607.08010#bib.bib11)\)via a free\-text question\. The sub\-agent generates and executes code that queries metrics through MCP \(Model Context Protocol\), iterating until it returns a free\-text answer, which the main agent parses into a verdict \(true/false/no data\) to inform subsequent traversal\. Delegation keeps the multi\-turn data\-processing context within the sub\-agent, and sub\-agent calls within a decision\-tree layer run in parallel\. Per alarm, the agent evaluates 3–10 nodes, and the sub\-agent loop can exceed 100 LLM turns in the most complex cases\.

The dominant per\-alarm cost in this architecture is the sub\-agent’s loop, as most of its turns translate the same SOP text into the same MCP queries against an unchanging interface\. We replace this loop with per\-node tools, each a build\-time compilation of the loop for one node into a single deterministic call that returns the verdict directly at inference\. Tools are model\-agnostic and regenerable independently of agent fine\-tuning\.

### 3\.1Tool\-Making Pipeline

SOPs are authored by humans, and the SOP text omits execution details that a human reader would resolve from domain knowledge or by inspecting the data, such as field names, datatypes, null handling, and boundary\-case conventions\. These omissions cause persistent failures when a tool\-maker is given only the SOP text\. The pipeline closes the gap with two grounding mechanisms \(Figure[1](https://arxiv.org/html/2607.08010#S1.F1), top\): a*data\-collection trace*of real execution against the live schema, and a*test–repair loop*that corrects each candidate against labeled cases\. Each tool is a function with a uniform signature mapping a \(warehouse, timestamp, context\) input to the node’s verdict \(true, false, or no data\) and supporting detail\. The context carries site\-specific parameters and verdicts from parent nodes\. Each node has a labeled training set, consisting of \(input, verdict\) pairs from an offline labeler whose logic was specified by subject\-matter experts\.

The pipeline has three model components, shown left\-to\-right in Figure[1](https://arxiv.org/html/2607.08010#S1.F1)\. The*data\-collector sub\-agent*is the baseline CodeAct sub\-agent serving as the inference\-time fallback, with access to the production MCP\. For each node, it is run on three sampled training cases balanced across verdict labels, writing and executing query code and iterating to a final verdict that is graded against the case label\. The resulting trace, passed to the tool\-maker, contains the query code, the MCP responses and observed schema \(field names, datatypes, value ranges\), and the graded verdict, supplying the execution details the SOP omits\.

The*tool\-maker LLM*produces a candidate tool conditioned on the SOP node text, the node’s position in the decision tree \(its parent and child nodes\), and the data\-collector’s traces\. The candidate is then tested against the full labeled training set, and disagreements are passed to the test–repair loop\. The*reflector LLM*writes a short diagnosis of the failures, after which the tool\-maker does a rewrite conditioned on the diagnosis and the disagreements\. The reflector receives the failing \(input, expected, predicted\) triples and the previous candidate, and the tool\-maker receives the same and the reflector’s diagnosis\. The loop runs for a budget of three rounds and terminates early once a candidate passes the full training set\. The candidate with the highest training\-set pass rate is deployed\. Appendix[D](https://arxiv.org/html/2607.08010#A4)shows an example from a generated tool\.

At inference, tools are used in one of two configurations: the main agent delegates a node to a sub\-agent that calls the node’s tool, or the main agent calls the tool directly \(Figure[1](https://arxiv.org/html/2607.08010#S1.F1), bottom\)\. Each tool returns the verdict, the observed value, the threshold, and a textual explanation\. "No data" is reserved for missing metrics, and exceptions trigger fallback to the CodeAct sub\-agent of the baseline\.

### 3\.2Agent Fine\-Tuning

Production latency constraints cap the deployed alarm\-triage agent at a small model size, and small off\-the\-shelf models fail to follow the SOP reliably\. We therefore distill a larger teacher into the small student model: the teacher generates trajectories on the alarm\-triage task, rejection sampling retains only those whose extracted observations, root causes, and actions all match ground truth\(Yuanet al\.,[2023](https://arxiv.org/html/2607.08010#bib.bib49)\), and the student is fine\-tuned on them with LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.08010#bib.bib55)\)\. To keep the student robust when tools are missing or fail, the teacher generates trajectories under varied tool availability: in 75% of trajectories, the generated tools are available but 25% of tools are randomly disabled per trajectory, and in the remaining 25% all node tools are removed so the teacher falls back to the sub\-agent for every node\.

## 4Tool\-Making Experiments

![Refer to caption](https://arxiv.org/html/2607.08010v1/x1.png)Figure 2:Per\-node pass@1 \(left\) and generation cost in tokens \(right\) for each ablation configuration on GLM\-4\.7\. \+D adds the data\-collection trace \(Section[3\.1](https://arxiv.org/html/2607.08010#S3.SS1)\); \+R adds the test\-repair loop\. Only the deployed configuration \(SOP\+D\+R\) exceeds 90% pass@1\.Table 1:Per\-node pass@1 \(%\) by tool\-maker and configuration\. \+D adds the data\-collection trace; \+R adds the test\-repair loop\. The deployed SOP\+D\+R column is in bold, with GLM\-4\.7 deployed\. Confidence intervals and significance tests are in Table[9](https://arxiv.org/html/2607.08010#A6.T9)\.### 4\.1Experimental Setup

#### Data\.

The pipeline is evaluated on 44 decision nodes\. Each node has a labeled training set used by the pipeline and a separate held\-out evaluation set, constructed as a temporal split\. Per\-node case counts range from 100 to 200, totaling 8,011 training cases and 8,022 evaluation cases\. Data labels are evenly distributed\.

#### Models\.

We compare the following four models as the tool\-making agent \(Section[3\.1](https://arxiv.org/html/2607.08010#S3.SS1)\): GLM\-4\.7\(Zenget al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib26)\), GLM\-5\(Zenget al\.,[2026](https://arxiv.org/html/2607.08010#bib.bib50)\), Qwen3 235B\(Yanget al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib53)\), and GLM\-4\.7 Flash\(Zenget al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib26)\)\. We use Claude Opus 4\.6 for additional ablations\(Anthropic,[2026](https://arxiv.org/html/2607.08010#bib.bib51)\)to prevent model capacity from confounding the comparison\. When reflection is used, the reflector LLM is the same as the tool\-maker LLM\. The deployed alarm\-triage agent uses a separately fine\-tuned model \(Section[3\.2](https://arxiv.org/html/2607.08010#S3.SS2)\)\.

#### Evaluation\.

We evaluate the build\-time pipeline by per\-node pass@1: the probability that one full execution of the pipeline produces a tool that passes the held\-out evaluation set for that node, where a tool passes only if it returns the correct verdict on every case in the set\. Pass@1 is averaged across the 44 decision nodes\. Confidence intervals and significance tests are reported in Appendix[F](https://arxiv.org/html/2607.08010#A6)\.

### 4\.2Main Results

Figure[2](https://arxiv.org/html/2607.08010#S4.F2)shows the per\-node pass@1 and generation cost of each configuration on GLM\-4\.7, and Table[1](https://arxiv.org/html/2607.08010#S4.T1)reports pass@1 for all four tool\-makers\. The full configuration adds the data\-collection trace \(\+D\) and the test\-repair loop \(\+R\) to the SOP text, reaching 94\.5% pass@1 on GLM\-4\.7 and 93\.6% on GLM\-5\. Both components are necessary: removing the repair loop reduces pass@1 by 9\.6pp and removing the trace by 5\.2pp, and only the full SOP\+D\+R configuration exceeds 90%\. The two strong tool\-makers, GLM\-4\.7 and GLM\-5, behave alike, with the trace and the repair loop each adding a significant gain, whereas the weaker Qwen3 235B and GLM\-4\.7 Flash gain less and do not improve monotonically\. We deploy GLM\-4\.7, which attains the highest pass@1 of the four\. The appendices report further results \(Appendix[B](https://arxiv.org/html/2607.08010#A2)\)\. Repair compute cannot substitute for the trace, whose components each contribute a gain, and reflection\-based repair outperforms repairing from raw failures only when the trace is present\. The pipeline also approaches the full\-data ceiling from a handful of labeled cases per node, with a label\-free variant within a few points \(Appendix[C](https://arxiv.org/html/2607.08010#A3)\)\.

### 4\.3Specification Quality as the Bottleneck

At this 94\.5% ceiling, four of the 44 nodes account for almost all remaining errors\. Three have edge cases \(negative denominators, null MCP values\) that neither the SOP nor the training labels cover, so no failing case reaches the repair loop\. The fourth has an ambiguous averaging window that the repair loop overfits away from the SOP \(Appendix[D](https://arxiv.org/html/2607.08010#A4)\)\. Two undeployed Opus runs confirm the cause: augmenting the training data raises pass@1 to 96\.6%, and clarifying the ambiguous SOP raises it to 99\.9% \(Table[2](https://arxiv.org/html/2607.08010#S4.T2)\)\. The headline 94\.5% pass@1 in Section[4\.2](https://arxiv.org/html/2607.08010#S4.SS2)is unaffected\.

Table 2:Effect of targeted specification and data improvements on the four residual nodes \(Opus\)\. Augmentation adds labeled cases for the edge\-case nodes\.

## 5Agent Experiments and Deployment

The deployed alarm\-triage agent uses small language models to meet production latency targets and calls the GLM\-4\.7\-generated tool library from Section[4](https://arxiv.org/html/2607.08010#S4)\. We report runs with two such models \(Qwen3 32B\(Yanget al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib53)\)and GLM\-4\.5\-Air\(Zenget al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib26)\)\), while the tool\-making pipeline \([Section˜3\.1](https://arxiv.org/html/2607.08010#S3.SS1)\) runs offline with stronger models, paying their reasoning cost once per tool and amortizing it across every production call\. We evaluate end\-to\-end correctness on 1,500 historical alarm scenarios with ground\-truth observations, root causes, and actions, and latency from production\. Non\-deployed configurations are evaluated by offline ablation\. Before a candidate agent is promoted to production, it runs in shadow: it executes on live alarms in parallel with the production agent without affecting outcomes, and discrepancies between the two are human reviewed\. Since deployment, our tools have been invoked across over 5,000 production alarms\.

### 5\.1Latency

Table 3:Production p50/p99 latency, relative to the no\-tools sub\-agent baseline \(set to 100, lower is better\)\. Rows 1 and 2 differ only in whether the agent uses tools\. Row 3 additionally changes the architecture to direct calls and the model to GLM\-4\.5\-Air\.Table[3](https://arxiv.org/html/2607.08010#S5.T3)reports production p50 and p99 latency across three configurations\. Replacing the sub\-agent’s CodeAct loop with tool calls cuts per\-alarm production p50 by 42% on Qwen3 32B under the original two\-tier sub\-agent architecture \(rows 1–2\)\. The same comparison reduces per\-alarm output tokens by 58% \(14,998 to 6,225\) and sub\-agent turns by 45% \(33\.7 to 18\.7\)\. A per\-component breakdown \(Appendix[E](https://arxiv.org/html/2607.08010#A5)\) shows the reduction is dominated by an 80% fall in sub\-agent code\-generation tokens, with main\-agent tokens also falling\.

Because tool outputs are small \(a few tokens\), reading them inline costs little, so the main agent can call tools without a sub\-agent, removing the original rationale for the sub\-agent layer \([Section˜3](https://arxiv.org/html/2607.08010#S3)\)\. This direct\-call configuration’s p50 is a further 55% below row 2, but it also runs GLM\-4\.5\-Air, confounding the two effects\. To isolate the architecture, we run an offline ablation that holds the model and inference framework fixed \(GLM\-4\.5\-Air, three seeds, 1,500 alarms\): direct calls alone reduce p50 by 62% and p99 by 30% relative to the sub\-agent architecture\.

### 5\.2End\-to\-End Agent Evaluation

Table 4:End\-to\-end agent accuracy \(%,N=1500N\{=\}1500\)\. "No tools": the sub\-agent writes MCP code, fine\-tuned on full\-tool trajectories\. "Sub\-agent" and "Main\-agent" both use generated tools, the latter called directly by the main agent\. Significance vs\. no\-tools baseline:p∗∗<0\.01\{\}^\{\*\*\}p<0\.01,p∗⁣∗∗<0\.001\{\}^\{\*\*\*\}p<0\.001\(Tables[12](https://arxiv.org/html/2607.08010#A6.T12)–[14](https://arxiv.org/html/2607.08010#A6.T14)\)\.Table[4](https://arxiv.org/html/2607.08010#S5.T4)reports end\-to\-end agent accuracy on the 1,500 historical alarms across the same three configurations\. Tools reduce the error rate from 2\.8% to 1\.8% on Qwen3 32B and from 1\.7% to 0\.8% on GLM\-4\.5\-Air, relative reductions of 36% and 53%\. Manual inspection of trajectories on which the sub\-agent and tool configurations disagree reveals two recurring sub\-agent failure modes\. First, it deviates from the SOP, using a similar but different metric or aggregating where the SOP specifies a per\-entity check\. Second, it reaches the correct conclusion but reports it incorrectly to the main agent\. Deterministic tools eliminate both\.

The agent is fine\-tuned on a mixture of trajectories with and without tools \(Section[3\.2](https://arxiv.org/html/2607.08010#S3.SS2)\) so that it can fall back to manual reasoning when a tool is unavailable\. When the Qwen3 32B agent fine\-tuned with tools is evaluated with tools disabled, its action accuracy \(97\.4%\) and overall accuracy \(96\.9%\) are statistically indistinguishable from the 97\.4% and 97\.2% obtained without tools, so mixed\-trajectory fine\-tuning preserves the fallback\.

## 6Discussion and Production Insights

#### Tools transfer across agents and models with a one\-time generation cost\.

The library generated by GLM\-4\.7 serves both the Qwen3 32B and GLM\-4\.5\-Air agents without regenerating any tool\. Generating the full 44\-node library is a one\-time cost of800800K tokens \(Figure[2](https://arxiv.org/html/2607.08010#S4.F2)\)\.

#### Parameterized tools cover a family of deployments\.

A generated tool generalizes across deployments when the quantities that vary between them are tool inputs\. In our case these are numeric thresholds, which differ across physical sites with alarm preferences and baseline robot error rates\. Each tool resolves them from a runtime parameter store, so one library can serve multiple sites\.

#### Logging tool outputs makes upstream data and environment inconsistencies visible\.

Because a tool returns the same value on the same input, its logs reveal environment issues that run\-to\-run variation in generated code hides\. Every invocation logs the tool version, inputs, and outputs, and a monitoring agent reviews these logs in batch, surfacing drift that no single trajectory would reveal\. A flagged tool is corrected by rerunning the generate\-test\-repair loop and promoting a new version once it passes the held\-out set and manual review\. Because the agent is decoupled from any specific tool version, deploying a fix or reverting is a configuration change requiring no retraining, and no rollback has been needed since deployment\.

This mechanism has revealed three pre\-existing inconsistencies in the upstream environment\. First, one field differed in type between the offline and production MCPs\. Generated code silently adapted, whereas the tool raised a reproducible exception that localized the issue\. Second, a node’s MCP reported cumulative events while the SOP referred to currently active ones: the tool passed offline evaluation, and the monitoring agent flagged the resulting high positive rate in production\. Third, an endpoint silently changed its return format from percentages \(75\) to decimals \(0\.75\), which the monitoring agent caught from the shifted value distribution\.

## 7Related Work

#### Tool use and tool\-making\.

Early LLM tool use focused on calling fixed APIs\(Schicket al\.,[2023](https://arxiv.org/html/2607.08010#bib.bib28); Patilet al\.,[2024](https://arxiv.org/html/2607.08010#bib.bib29)\)\. A parallel line gave agents an action space of executable code\(Gaoet al\.,[2023](https://arxiv.org/html/2607.08010#bib.bib30); Yaoet al\.,[2022](https://arxiv.org/html/2607.08010#bib.bib31); Wanget al\.,[2024a](https://arxiv.org/html/2607.08010#bib.bib11)\)\. A more recent line of work has the agent construct its own tools\. These include single\-pass synthesis methods\(Caiet al\.,[2024](https://arxiv.org/html/2607.08010#bib.bib17); Qianet al\.,[2023](https://arxiv.org/html/2607.08010#bib.bib16); Yuanet al\.,[2024](https://arxiv.org/html/2607.08010#bib.bib18); Wanget al\.,[2024c](https://arxiv.org/html/2607.08010#bib.bib19)\)and iterative build\-time construction with test\- or execution\-based validation\(Wanget al\.,[2023](https://arxiv.org/html/2607.08010#bib.bib20); Zhenget al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib22); Wölfleinet al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib23)\)\. Operational documents can also be transformed into tools\(Zwerdlinget al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib24); Maoet al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib25); Yeet al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib10)\)\. Recent research confirms that prompting\-based agents fail to follow SOPs reliably when evaluated against the original SOP code as a verifier\(Liet al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib32); Nandiet al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib12)\), with recurring failure modes including route blindness and calculation errors\(Wanget al\.,[2025b](https://arxiv.org/html/2607.08010#bib.bib13)\)\. Recent industrial deployments include frameworks for alert triage and root cause analysis in production systems\(Liuet al\.,[2026a](https://arxiv.org/html/2607.08010#bib.bib34); Bharadwaj and Tu,[2026](https://arxiv.org/html/2607.08010#bib.bib33)\)\. The closest peer work for SOP\-to\-code isPeiet al\.\([2025](https://arxiv.org/html/2607.08010#bib.bib15)\), but it is only evaluated on a benchmark\. We contribute to this line of work by compiling SOPs into tools validated against labeled production cases\.

#### Iterative refinement and data grounding\.

Our work is closely related to the execution\-feedback refinement family\(Chenet al\.,[2024a](https://arxiv.org/html/2607.08010#bib.bib37); Zhanget al\.,[2023](https://arxiv.org/html/2607.08010#bib.bib38); Zhonget al\.,[2024](https://arxiv.org/html/2607.08010#bib.bib39); Ridniket al\.,[2024](https://arxiv.org/html/2607.08010#bib.bib40)\)\.Olaussonet al\.\([2024](https://arxiv.org/html/2607.08010#bib.bib27)\)conclude that the success of self\-repair depends on the feedback\. Our finding that the pipeline plateaus without environmental grounding is consistent with this\. The text\-to\-SQL literature also finds that schema alone hits a ceiling and grounding in database values is required\(Liet al\.,[2023](https://arxiv.org/html/2607.08010#bib.bib41); Wanget al\.,[2025a](https://arxiv.org/html/2607.08010#bib.bib42); Caoet al\.,[2026](https://arxiv.org/html/2607.08010#bib.bib43)\)\. Our findings on the importance of specification quality for tool generation echo what has been reported earlier in the code and SQL generation literature\(Wretbladet al\.,[2024](https://arxiv.org/html/2607.08010#bib.bib47); Jinet al\.,[2026](https://arxiv.org/html/2607.08010#bib.bib48)\)\.

## 8Conclusion

We show that repeated inference\-time code generation can be optimized for production LLM agents\. By compiling SOP nodes into environment\-grounded, validated, and versioned tools, our system moves recurring procedural work out of the latency\-critical path while preserving fallback flexibility for failures\. In a deployed fulfillment\-center alarm\-triage agent, this reduces latency and improves end\-to\-end accuracy\.

The broader contribution is an operating pattern for self\-evolving industrial agents: LLMs can build and repair their own action space offline, under test, rather than improvising code on every request\. Once grounded tools are in place, the limiting factor becomes the quality of the underlying SOP specification instead of the agent’s ability to rediscover schemas or rewrite boilerplate\. This makes agent behavior faster, more consistent, and more auditable, suggesting a practical route to reliable low\-latency LLM systems in operational workflows\.

## Limitations

Our agentic tool\-making loop has been deployed on one application, outbound dock alarm triage\. Generalization to other application domains, such as free\-form runbooks, is not established\. The setting nonetheless shares properties with typical agentic workflows: a written procedure that the agent must follow and that is executed against a stable backend, and latency or auditability constraints that make inference\-time code generation less appealing as an alternative\.

The improvement loop retains human oversight for production safety\. It already automates parts of the cycle, such as drift detection and candidate tool generation, but we cannot yet guarantee that every failure would be caught without human review, so fully automating it remains open\. Clarifying an underspecified SOP is inherently a human task, because the SOP encodes the author’s intent for how the procedure should behave\. Creating a new tool still benefits from labeled cases\. Our label\-free configuration trails the labeled ceiling by less than 3pp, and a fully label\-free pipeline would remove the need for hand\-labeled cases when adding new nodes\.

## References

- Claude opus 4\.6 system card\.Note:Accessed: 2026\-06\-04External Links:[Link](https://www-cdn.anthropic.com/0dd865075ad3132672ee0ab40b05a53f14cf5288.pdf)Cited by:[§4\.1](https://arxiv.org/html/2607.08010#S4.SS1.SSS0.Px2.p1.1)\.
- A\. Bharadwaj and K\. Tu \(2026\)Agentic observability: automated alert triage for adobe e\-commerce\.arXiv preprint arXiv:2602\.02585\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- T\. Cai, X\. Wang, T\. Ma, X\. Chen, and D\. Zhou \(2024\)Large language models as tool makers\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54067–54089\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p3.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- B\. Cao, W\. Liao, Y\. Sun, D\. Fang, H\. Li, and W\. Lam \(2026\)APEX\-sql: talking to the data via agentic exploration for text\-to\-sql\.arXiv preprint arXiv:2602\.16720\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px2.p1.1)\.
- X\. Chen, M\. Lin, N\. Schärli, and D\. Zhou \(2024a\)Teaching large language models to self\-debug\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 8746–8825\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px2.p1.1)\.
- Y\. Chen, H\. Xie, M\. Ma, Y\. Kang, X\. Gao, L\. Shi, Y\. Cao, X\. Gao, H\. Fan, M\. Wen, J\. Zeng, S\. Ghosh, X\. Zhang, C\. Zhang, Q\. Lin, S\. Rajmohan, D\. Zhang, and T\. Xu \(2024b\)Automatic root cause analysis via large language models for cloud incidents\.InProceedings of the Nineteenth European Conference on Computer Systems,pp\. 674–688\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p1.1)\.
- L\. Gao, A\. Madaan, S\. Zhou, U\. Alon, P\. Liu, Y\. Yang, J\. Callan, and G\. Neubig \(2023\)Pal: program\-aided language models\.InInternational conference on machine learning,pp\. 10764–10799\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2607.08010#A1.SS0.SSS0.Px5.p1.3),[§3\.2](https://arxiv.org/html/2607.08010#S3.SS2.p1.1)\.
- T\. Jin, Y\. Choi, Y\. Zhu, and D\. Kang \(2026\)Pervasive annotation errors break text\-to\-sql benchmarks and leaderboards\.arXiv preprint arXiv:2601\.08778\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px2.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[Appendix A](https://arxiv.org/html/2607.08010#A1.SS0.SSS0.Px2.p1.1)\.
- J\. Li, B\. Hui, G\. Qu, J\. Yang, B\. Li, B\. Li, B\. Wang, B\. Qin, R\. Cao, R\. Geng, N\. Huo, X\. Zhou, C\. Ma, G\. Li, K\. C\. C\. Chang, F\. Huang, R\. Cheng, and Y\. Li \(2023\)Can llm already serve as a database interface? a big bench for large\-scale database grounded text\-to\-sqls\.Advances in Neural Information Processing Systems36,pp\. 42330–42357\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px2.p1.1)\.
- Z\. Li, S\. Huang, J\. Wang, N\. Zhang, A\. Antoniades, W\. Hua, K\. Zhu, S\. Zeng, C\. Wang, W\. Y\. Wang, and X\. Yan \(2025\)SOPBench: evaluating language agents at following standard operating procedures and constraints\.arXiv preprint arXiv:2503\.08669\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- B\. Liu, Z\. Qian, Y\. Zhao, X\. Jiang, Z\. Liang, Y\. Ma, J\. Zhuang, B\. Chen, S\. Yang, H\. Wan, Y\. Wu, C\. Lei, and X\. Liang \(2026a\)Bian que: an agentic framework with flexible skill arrangement for online system operations\.arXiv preprint arXiv:2604\.26805\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- N\. Liu, K\. Kujanpää, Z\. Zhu, P\. A\. Sreekar, K\. Liu, C\. Sun, J\. M\. Menendez, M\. Bales, T\. Yang, S\. Alam, R\. Yu, B\. Liu, K\. Klinkner, and S\. Malmasi \(2026b\)Eluna: an agentic LLM system for automating warehouse operations with reasoning and task execution\.arXiv preprint\.Cited by:[Appendix A](https://arxiv.org/html/2607.08010#A1.SS0.SSS0.Px5.p1.3),[§3](https://arxiv.org/html/2607.08010#S3.p1.1)\.
- J\. Mao, L\. Li, Y\. Gao, Z\. Peng, S\. He, C\. Zhang, S\. Qin, S\. Khalid, Q\. Lin, S\. Rajmohan, S\. Lanka, and D\. Zhang \(2025\)StepFly: agentic troubleshooting guide automation for incident diagnosis\.arXiv preprint arXiv:2510\.10074\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p3.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- S\. Nandi, A\. Datta, R\. Nama, U\. Patel, N\. Vichare, I\. Bhattacharya, P\. Grover, S\. Asija, G\. Carenini, W\. Zhang, A\. Gupta, S\. Bhaduri, J\. Xu, H\. Raja, S\. Ray, A\. Chan, E\. X\. Fei, G\. Du, Z\. Akhtar, H\. Asnani, W\. Chan, M\. Xiong, F\. Carbone, and J\. Mirchandani \(2025\)SOP\-bench: complex industrial sops for evaluating llm agents\.arXiv preprint arXiv:2506\.08119\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p2.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- T\. X\. Olausson, J\. P\. Inala, C\. Wang, J\. Gao, and A\. Solar\-Lezama \(2024\)Is self\-repair a silver bullet for code generation?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 36545–36593\.Cited by:[§B\.2](https://arxiv.org/html/2607.08010#A2.SS2.p2.3),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px2.p1.1)\.
- S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. Gonzalez \(2024\)Gorilla: large language model connected with massive apis\.Advances in Neural Information Processing Systems37,pp\. 126544–126565\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- C\. Pei, Z\. Wang, F\. Liu, Z\. Li, Y\. Liu, X\. He, R\. Kang, T\. Zhang, J\. Chen, J\. Li, G\. Xie, and D\. Pei \(2025\)Flow\-of\-action: sop enhanced llm\-based multi\-agent system for root cause analysis\.InCompanion Proceedings of the ACM on Web Conference 2025,pp\. 422–431\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p2.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- C\. Qian, C\. Han, Y\. R\. Fung, Y\. Qin, Z\. Liu, and H\. Ji \(2023\)Creator: disentangling abstract and concrete reasonings of large language models through tool creation\.arXiv preprint arXiv:2305\.14318\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- E\. Riddell, J\. Riddell, G\. Sun, M\. Antkiewicz, and K\. Czarnecki \(2026\)Stalled, biased, and confused: uncovering reasoning failures in llms for cloud\-based root cause analysis\.arXiv preprint arXiv:2601\.22208\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p2.1)\.
- T\. Ridnik, D\. Kredo, and I\. Friedman \(2024\)Code generation with alphacodium: from prompt engineering to flow engineering\.arXiv preprint arXiv:2401\.08500\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px2.p1.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.Advances in neural information processing systems36,pp\. 68539–68551\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- B\. Wang, C\. Ren, J\. Yang, X\. Liang, J\. Bai, L\. Chai, Z\. Yan, Q\. Zhang, D\. Yin, X\. Sun, and Z\. Li \(2025a\)MAC\-sql: a multi\-agent collaborative framework for text\-to\-sql\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 540–557\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px2.p1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2023\)Voyager: an open\-ended embodied agent with large language models\.arXiv preprint arXiv:2305\.16291\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p3.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- J\. Wang, Z\. Tang, Z\. Jin, H\. Chen, Y\. Jin, P\. Ding, X\. Li, and X\. Cao \(2025b\)SOP\-maze: evaluating large language models on complicated business standard operating procedures\.arXiv preprint arXiv:2510\.08942\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p2.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- X\. Wang, Y\. Chen, L\. Yuan, Y\. Zhang, Y\. Li, H\. Peng, and H\. Ji \(2024a\)Executable code actions elicit better llm agents\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p1.1),[§1](https://arxiv.org/html/2607.08010#S1.p2.1),[§3](https://arxiv.org/html/2607.08010#S3.p1.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- Z\. Wang, Z\. Liu, Y\. Zhang, A\. Zhong, J\. Wang, F\. Yin, L\. Fan, L\. Wu, and Q\. Wen \(2024b\)Rcagent: cloud root cause analysis by autonomous agents with tool\-augmented large language models\.InProceedings of the 33rd ACM international conference on information and knowledge management,pp\. 4966–4974\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p1.1)\.
- Z\. Wang, D\. Fried, and G\. Neubig \(2024c\)Trove: inducing verifiable and efficient toolboxes for solving programmatic tasks\.arXiv preprint arXiv:2401\.12869\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p3.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- Z\. Z\. Wang, J\. Mao, D\. Fried, and G\. Neubig \(2024d\)Agent workflow memory\.arXiv preprint arXiv:2409\.07429\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p3.1)\.
- G\. Wölflein, D\. Ferber, D\. Truhn, O\. Arandjelovic, and J\. N\. Kather \(2025\)Llm agents making agent tools\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 26092–26130\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p3.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- N\. Wretblad, F\. Riseby, R\. Biswas, A\. Ahmadi, and O\. Holmström \(2024\)Understanding the effects of noise in text\-to\-sql: an examination of the bird\-bench benchmark\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 356–369\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2607.08010#S4.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2607.08010#S5.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2022\)React: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- A\. Ye, Q\. Ma, J\. Chen, M\. Li, T\. Li, F\. Liu, S\. Mai, M\. Lu, H\. Bao, and Y\. You \(2025\)Sop\-agent: empower general purpose ai agent with domain\-specific sops\.arXiv preprint arXiv:2501\.09316\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p1.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- L\. Yuan, Y\. Chen, X\. Wang, Y\. Fung, H\. Peng, and H\. Ji \(2024\)Craft: customizing llms by creating and retrieving from specialized toolsets\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 40097–40125\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p3.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- Z\. Yuan, H\. Yuan, C\. Li, G\. Dong, K\. Lu, C\. Tan, C\. Zhou, and J\. Zhou \(2023\)Scaling relationship on learning mathematical reasoning with large language models\.arXiv preprint arXiv:2308\.01825\.Cited by:[§3\.2](https://arxiv.org/html/2607.08010#S3.SS2.p1.1)\.
- A\. Zeng, X\. Lv, Z\. Hou, Z\. Du, Q\. Zheng, B\. Chen, D\. Yin, C\. Ge, C\. Huang, C\. Xie, C\. Zhu, C\. Yin, C\. Wang, G\. Pan, H\. Zeng, H\. Zhang, H\. Wang, H\. Chen, J\. Zhang, J\. Jiao, J\. Guo, J\. Wang, J\. Du, J\. Wu, K\. Wang, L\. Li, L\. Fan, L\. Zhong, M\. Liu, M\. Zhao, P\. Du, Q\. Dong, R\. Lu, Shuang\-Li, S\. Cao, S\. Liu, T\. Jiang, X\. Chen, X\. Zhang, X\. Huang, X\. Dong, Y\. Xu, Y\. Wei, Y\. An, Y\. Niu, Y\. Zhu, Y\. Wen, Y\. Cen, Y\. Bai, Z\. Qiao, Z\. Wang, Z\. Wang, Z\. Zhu, Z\. Liu, Z\. Li, B\. Wang, B\. Wen, C\. Huang, C\. Cai, C\. Yu, C\. Li, C\. Hu, C\. Zhang, D\. Zhang, D\. Lin, D\. Yang, D\. Wang, D\. Ai, E\. Zhu, F\. Yi, F\. Chen, G\. Wen, H\. Sun, H\. Zhao, H\. Hu, H\. Zhang, H\. Liu, H\. Zhang, H\. Peng, H\. Tai, H\. Zhang, H\. Liu, H\. Wang, H\. Yan, H\. Ge, H\. Liu, H\. Chu, J\. Zhao, J\. Wang, J\. Zhao, J\. Ren, J\. Wang, J\. Zhang, J\. Gui, J\. Zhao, J\. Li, J\. An, J\. Li, J\. Yuan, J\. Du, J\. Liu, J\. Zhi, J\. Duan, K\. Zhou, K\. Wei, K\. Wang, K\. Luo, L\. Zhang, L\. Sha, L\. Xu, L\. Wu, L\. Ding, L\. Chen, M\. Li, N\. Lin, P\. Ta, Q\. Zou, R\. Song, R\. Yang, S\. Tu, S\. Yang, S\. Wu, S\. Zhang, S\. Li, S\. Li, S\. Fan, W\. Qin, W\. Tian, W\. Zhang, W\. Yu, W\. Liang, X\. Kuang, X\. Cheng, X\. Li, X\. Yan, X\. Hu, X\. Ling, X\. Fan, X\. Xia, X\. Zhang, X\. Zhang, X\. Pan, X\. Zou, X\. Zhang, Y\. Liu, Y\. Wu, Y\. Li, Y\. Wang, Y\. Zhu, Y\. Tan, Y\. Zhou, Y\. Pan, Y\. Zhang, Y\. Su, Y\. Geng, Y\. Yan, Y\. Tan, Y\. Bi, Y\. Shen, Y\. Yang, Y\. Li, Y\. Liu, Y\. Wang, Y\. Li, Y\. Wu, Y\. Zhang, Y\. Duan, Y\. Zhang, Z\. Liu, Z\. Jiang, Z\. Yan, Z\. Zhang, Z\. Wei, Z\. Chen, Z\. Feng, Z\. Yao, Z\. Chai, Z\. Wang, Z\. Zhang, B\. Xu, M\. Huang, H\. Wang, J\. Li, Y\. Dong, and J\. Tang \(2026\)GLM\-5: from vibe coding to agentic engineering\.arXiv preprint arXiv:2602\.15763\.Cited by:[§4\.1](https://arxiv.org/html/2607.08010#S4.SS1.SSS0.Px2.p1.1)\.
- A\. Zeng, X\. Lv, Q\. Zheng, Z\. Hou, B\. Chen, C\. Xie, C\. Wang, D\. Yin, H\. Zeng, J\. Zhang, K\. Wang, L\. Zhong, M\. Liu, R\. Lu, S\. Cao, X\. Zhang, X\. Huang, Y\. Wei, Y\. Cheng, Y\. An, Y\. Niu, Y\. Wen, Y\. Bai, Z\. Du, Z\. Wang, Z\. Zhu, B\. Zhang, B\. Wen, B\. Wu, B\. Xu, C\. Huang, C\. Zhao, C\. Cai, C\. Yu, C\. Li, C\. Ge, C\. Huang, C\. Zhang, C\. Xu, C\. Zhu, C\. Li, C\. Yin, D\. Lin, D\. Yang, D\. Jiang, D\. Ai, E\. Zhu, F\. Wang, G\. Pan, G\. Wang, H\. Sun, H\. Li, H\. Li, H\. Hu, H\. Zhang, H\. Peng, H\. Tai, H\. Zhang, H\. Wang, H\. Yang, H\. Liu, H\. Zhao, H\. Liu, H\. Yan, H\. Liu, H\. Chen, J\. Li, J\. Zhao, J\. Ren, J\. Jiao, J\. Zhao, J\. Yan, J\. Wang, J\. Gui, J\. Zhao, J\. Liu, J\. Li, J\. Li, J\. Lu, J\. Wang, J\. Yuan, J\. Li, J\. Du, J\. Du, J\. Liu, J\. Zhi, J\. Gao, K\. Wang, L\. Yang, L\. Xu, L\. Fan, L\. Wu, L\. Ding, L\. Wang, M\. Zhang, M\. Li, M\. Xu, M\. Zhao, M\. Zhai, P\. Du, Q\. Dong, S\. Lei, S\. Tu, S\. Yang, S\. Lu, S\. Li, S\. Li, Shuang\-Li, S\. Yang, S\. Yi, T\. Yu, W\. Tian, W\. Wang, W\. Yu, W\. L\. Tam, W\. Liang, W\. Liu, X\. Wang, X\. Jia, X\. Gu, X\. Ling, X\. Wang, X\. Fan, X\. Pan, X\. Zhang, X\. Zhang, X\. Fu, X\. Zhang, Y\. Xu, Y\. Wu, Y\. Lu, Y\. Wang, Y\. Zhou, Y\. Pan, Y\. Zhang, Y\. Wang, Y\. Li, Y\. Su, Y\. Geng, Y\. Zhu, Y\. Yang, Y\. Li, Y\. Wu, Y\. Li, Y\. Liu, Y\. Wang, Y\. Li, Y\. Zhang, Z\. Liu, Z\. Yang, Z\. Zhou, Z\. Qiao, Z\. Feng, Z\. Liu, Z\. Zhang, Z\. Wang, Z\. Yao, Z\. Wang, Z\. Liu, Z\. Chai, Z\. Li, Z\. Zhao, W\. Chen, J\. Zhai, B\. Xu, M\. Huang, H\. Wang, J\. Li, Y\. Dong, and J\. Tang \(2025\)GLM\-4\.5: agentic, reasoning, and coding \(arc\) foundation models\.arXiv preprint arXiv:2508\.06471\.Cited by:[Appendix A](https://arxiv.org/html/2607.08010#A1.SS0.SSS0.Px5.p1.3),[§4\.1](https://arxiv.org/html/2607.08010#S4.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2607.08010#S5.p1.1)\.
- K\. Zhang, Z\. Li, J\. Li, G\. Li, and Z\. Jin \(2023\)Self\-edit: fault\-aware code editor for code generation\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 769–787\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px2.p1.1)\.
- Y\. Zhao, J\. Huang, J\. Hu, X\. Wang, Y\. Mao, D\. Zhang, Z\. Jiang, Z\. Wu, B\. Ai, A\. Wang, W\. Zhou, and Y\. Chen \(2024\)SWIFT:a scalable lightweight infrastructure for fine\-tuning\.External Links:2408\.05517,[Link](https://arxiv.org/abs/2408.05517)Cited by:[Appendix A](https://arxiv.org/html/2607.08010#A1.SS0.SSS0.Px5.p1.3)\.
- B\. Zheng, M\. Y\. Fatemi, X\. Jin, Z\. Z\. Wang, A\. Gandhi, Y\. Song, Y\. Gu, J\. Srinivasa, G\. Liu, G\. Neubig, and Y\. Su \(2025\)SkillWeaver: web agents can self\-improve by discovering and honing skills\.arXiv preprint arXiv:2504\.07079\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.
- L\. Zhong, Z\. Wang, and J\. Shang \(2024\)Debug like a human: a large language model debugger via verifying runtime execution step by step\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 851–870\.Cited by:[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px2.p1.1)\.
- N\. Zwerdling, D\. Boaz, E\. Rabinovich, G\. Uziel, D\. Amid, and A\. A\. Tavor \(2025\)Towards enforcing company policy adherence in agentic workflows\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,pp\. 595–606\.Cited by:[§1](https://arxiv.org/html/2607.08010#S1.p3.1),[§7](https://arxiv.org/html/2607.08010#S7.SS0.SSS0.Px1.p1.1)\.

## Appendix

## Appendix AExperimental Setup

#### Seeds\.

Per\-node pass@1 results in Section[4](https://arxiv.org/html/2607.08010#S4)use 30 independent generation runs per configuration, except the SOP column in Table[1](https://arxiv.org/html/2607.08010#S4.T1)and the augmented\-data row in Table[2](https://arxiv.org/html/2607.08010#S4.T2), which use 10\. End\-to\-end agent accuracy in Section[5](https://arxiv.org/html/2607.08010#S5)uses 3 seeds per configuration on the same 1,500 alarm scenarios\.

#### Inference framework\.

The deployed agent \(Qwen3 32B and GLM\-4\.5\-Air\) runs on vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2607.08010#bib.bib52)\)for both offline benchmarks and production\. The open\-weight tool\-makers \(GLM\-4\.7, Qwen3 235B, GLM\-4\.7 Flash\) likewise run on vLLM\. GLM\-5 and the Claude ablations are served via Amazon Bedrock\.

#### Hardware\.

Inference, both offline benchmarks and production, runs on a single 8×\\timesH100 instance\. Fine\-tuning runs on 64 H100s for Qwen3 32B and 128 H100s for GLM\-4\.5\-Air\.

#### Latency measurement\.

For competitive reasons we cannot disclose absolute production latencies\. Hence, Table[3](https://arxiv.org/html/2607.08010#S5.T3)reports figures normalized to the no\-tools sub\-agent baseline\.

#### Fine\-tuning\.

We follow the training procedure ofLiuet al\.\([2026b](https://arxiv.org/html/2607.08010#bib.bib56)\)\. We distill GLM\-4\.7\(Zenget al\.,[2025](https://arxiv.org/html/2607.08010#bib.bib26)\)into the two deployed students, Qwen3 32B and GLM\-4\.5\-Air\. A teacher trajectory is retained only when its extracted observations, root causes, and actions all match the ground\-truth labels\. This filter accepts approximately 92% of trajectories \(93% on the with\-tools slice with dropout, 4,171/4,500, and 91% on the no\-tools slice, 1,358/1,500\)\. The same filter applied to the no\-tools baseline accepts 91% of trajectories \(6,786/7,500\), so tool availability also raises the teacher’s own accuracy\. Students are fine\-tuned with LoRA\(Huet al\.,[2022](https://arxiv.org/html/2607.08010#bib.bib55)\)using MS\-SWIFT\(Zhaoet al\.,[2024](https://arxiv.org/html/2607.08010#bib.bib57)\), with rank andα\\alphaof 256 and learning rate2×10−42\{\\times\}10^\{\-4\}for Qwen3 32B and 32 and1×10−41\{\\times\}10^\{\-4\}for GLM\-4\.5\-Air\. Both train at batch size 64 with a 5% linear warmup and cosine decay to zero, on 64 and 128 H100 GPUs respectively\. The with\-tools fine\-tunes use one epoch, while the no\-tools baseline fine\-tunes use two epochs\(Liuet al\.,[2026b](https://arxiv.org/html/2607.08010#bib.bib56)\)\. After the training, we merge the LoRA adapters so inference carries no added overhead\.

#### Offline labeler\.

The ground\-truth verdict for each per\-node case \(Section[3\.1](https://arxiv.org/html/2607.08010#S3.SS1)\) is produced by an offline labeler\. Subject\-matter experts give step\-by\-step guidance on each node’s logic, which we implement and verify with them\.

## Appendix BFurther Method Ablations

### B\.1Impact of Repair Loop

The deployed pipeline’s repair loop uses single\-path reflection, where the model produces a brief diagnosis of the failures and writes a corrected tool after each test round\. This is repeated for three rounds\. We compare this strategy against two alternatives: no reflection \(the model repairs from failures alone without a separate diagnosis step\), and multi\-hypothesis branching\. In multi\-hypothesis branching, three independent diagnoses are generated per round\. Each is tried as a separate repair branch and we keep the lowest\-failure branch\.

![Refer to caption](https://arxiv.org/html/2607.08010v1/x2.png)Figure 3:pass@1 across repair rounds for each repair strategy, without the data\-collection trace \(SOP\+R, Opus\)\. No\-reflection plateaus around 92\.5%, while reflection peaks at 94\.3% by round 10 but underperforms no\-reflection at low round counts\. Multi\-hypothesis branching reaches 94\.3% by round 3 at roughly3×3\{\\times\}the per\-round cost\.Figure[3](https://arxiv.org/html/2607.08010#A2.F3)shows that reflection and branching reach similar performance at similar total cost \(556K tokens for reflection at 10 rounds, 597K for branching at 4 rounds\)\. Branching reaches the ceiling in fewer rounds because it explores three diagnoses in parallel each round\. No\-reflection plateaus around 92\.5% because additional rounds repeat the same diagnostic errors\.

### B\.2Impact of the Data\-Collection Trace

The two intermediate conditions in Table[5](https://arxiv.org/html/2607.08010#A2.T5)each prepend only a part of the trace to the tool\-maker’s prompt: the schema metadata observed by the sub\-agent \(column names, dtypes, row counts\), or the sub\-agent’s verdict graded against the labeled answer\.

Table 5:Trace\-content ablation \(Opus, three repair rounds\)\. Each component contributes a consistent gain\. Confidence intervals are in Table[10](https://arxiv.org/html/2607.08010#A6.T10)\.Schema metadata adds\+1\.8\+1\.8pp and grading the sub\-agent’s verdict adds\+1\.7\+1\.7pp, whereas the full trace adds another\+1\.2\+1\.2pp by showing the tool\-maker working code\. This is a form of execution\-grounded feedback that prior work has identified as a significant contributor to repair quality\(Olaussonet al\.,[2024](https://arxiv.org/html/2607.08010#bib.bib27)\)\.

Two concrete failure modes illustrate the gain\. Including the grading corrects semantic biases that are underspecified by the SOP\. One tool should return true on zero violations, and models consistently emit a`count \> 0`guard\. When the sub\-agent makes the same error on a sampled case, the trace shows the sub\-agent’s verdict marked incorrect, and the tool\-maker drops the guard in subsequent generations\. Including the code can help prevent boundary bugs\. Zero\-shot tool\-maker code often rounds the outcome before comparing to the threshold, whereas the sub\-agent’s code does not exhibit this failure mode\.

### B\.3Interaction Between the Repair Loop and the Trace

Table 6:Interaction between data grounding and repair strategy \(Opus, 3 rounds\)\. pass@1 %\. Confidence intervals andtt\-tests are in Table[11](https://arxiv.org/html/2607.08010#A6.T11)\.Table[6](https://arxiv.org/html/2607.08010#A2.T6)reports each repair\-loop strategy at three repair rounds, the deployed budget, with and without the data\-collection trace\. The main finding is that the trace yields the largest gain under reflection \(\+4\.0\+4\.0pp\)\. At three rounds without the trace, reflection underperforms no\-reflection \(90\.5% vs\. 91\.6%\), though reflection eventually catches up as shown in Appendix[B\.1](https://arxiv.org/html/2607.08010#A2.SS1)\. The mechanism is that the trace’s working code and graded verdict ground the diagnosis step in concrete failure modes, so reflection no longer expends repair rounds on incorrect hypotheses\. No\-reflection benefits less \(91\.6% to 92\.8%,\+1\.2\+1\.2pp\) because without an explicit diagnosis step, it does not fully exploit the information in the trace\. Branching is essentially unchanged \(94\.3% to 94\.4%\)\. With three independent diagnoses, branching explores enough alternatives to converge even without the data\-collection trace\. With the trace, reflection \(94\.5%, 395K tokens\) and branching \(94\.4%, 496K tokens\) reach the same accuracy, but reflection costs 20% fewer tokens, so the deployed pipeline uses reflection\.

### B\.4Data Efficiency

![Refer to caption](https://arxiv.org/html/2607.08010v1/x3.png)Figure 4:pass@1 vs\. training\-set size, with and without data grounding \(Opus\)\.Each new SOP node requires labeled training cases to generate its tool\. Hence, a practical pipeline must reach high pass@1 from few labels\. With grounding, pass@1 reaches 91\.8% atN=5N\{=\}5training cases per node, within 3pp of the full\-set ceiling \(Figure[4](https://arxiv.org/html/2607.08010#A2.F4)\)\. Removing labels entirely is also feasible\. We investigate a label\-free tool\-maker that uses judge\-filtered pseudo\-labels and multi\-candidate voting instead of the labeled repair\-loop test cases\. This configuration reaches 91\.6% pass@1, also within 3pp of the labeled ceiling \(Appendix[C](https://arxiv.org/html/2607.08010#A3)\)\.

### B\.5Token Scaling

We perform two follow\-ups to the main\-text GLM\-4\.7 ablation \(Figure[2](https://arxiv.org/html/2607.08010#S4.F2)\)\. First, we investigate whether GLM\-4\.7 repair\-only catches up to the deployed configuration at higher token budgets\. The second reports the same ablation on a stronger tool\-maker, Opus 4\.6\.

#### SOP\+R does not scale with tokens on GLM\-4\.7\.

Table[7](https://arxiv.org/html/2607.08010#A2.T7)compares the best SOP\+R configuration we ran against a higher\-budget SOP\+R configuration that explores three repair branches per round\. Despite spending 55% more tokens, branching reduces pass@1 from 89\.3% to 87\.0%\. The cheapest SOP\+D\+R configuration \(1 sample, 3\-round repair\) reaches 91\.2% at 596K tokens, which is fewer tokens than the best SOP\+R point\. The deployed SOP\+D\+R configuration in the main text reaches 94\.5% at 799K\. No SOP\+R configuration we ran exceeds 89\.3%\.

Table 7:GLM\-4\.7 SOP\+R configurations at higher token budgets, paired with two SOP\+D\+R configurations for reference\. \+D adds the data\-collection trace; \+R adds the test\-repair loop\. Per\-seed tokens are means over 30 seeds \(10 for the 3\-branches row\)\.
#### Opus 4\.6 component ablation\.

Figure[5](https://arxiv.org/html/2607.08010#A2.F5)repeats the Figure[2](https://arxiv.org/html/2607.08010#S4.F2)ablation on Opus 4\.6\. The repair loop alone reaches 94\.3% pass@1 at 556K tokens with 10 rounds\. The deployed configuration reaches comparable accuracy \(94\.5%\) at 395K tokens, a 29% token reduction\. The trace helps Opus reach the ceiling at lower cost, but the ceiling is unchanged\.

![Refer to caption](https://arxiv.org/html/2607.08010v1/x4.png)Figure 5:Per\-node pass@1 \(left bars\) and generation cost in tokens per seed \(right bars\) for each ablation configuration on Claude Opus 4\.6\. The deployed configuration matches the repair\-only ceiling within seed variance at 29% fewer tokens\.

## Appendix CLabel\-Free Generation

We investigate how to replace the labeled repair loop with a pipeline that combines voting and LLM judges\. For each tool,KKindependent sub\-agents each investigate one sampled unlabeled case\. An LLM judge filters out traces that are not consistent with the SOP\. From the surviving traces, we generateNNindependent candidates, and execute each candidate against theKKsampled cases\. We then assign a pseudo\-label for each case by majority vote across the surviving traces and theNNcandidate outputs, and iterate a repair loop forRRrounds\. Each round, every candidate is executed on theKKinvestigated cases\. Candidates that agree with all current pseudo\-labels and pass a code\-review judge \(an LLM that inspects the candidate tool’s code for consistency with the SOP\) are carried forward unchanged\. The remaining candidates see their disagreement cases as failure feedback, write a short diagnosis, and regenerate\. After allNNcandidates have either regenerated or been carried forward, the pseudo\-labels for each case are recomputed by majority vote over the updated candidate outputs and the judge\-passing sub\-agent determinations\. The final candidate is selected by counting how many of theKKinvestigated cases each candidate matches the pseudo\-label on\. Ties are broken first by the code\-review judge, then by code length\.

The headline configuration is GLM\-4\.7 as the tool\-maker, Claude Sonnet 4\.5 as the trace judge, and Claude Opus 4\.7 as the reflector agent, withK=3K\{=\}3,N=10N\{=\}10, andR=5R\{=\}5\. The reflector here is Opus 4\.7 due to model availability at the time of this experiment\. We do not attribute the result to the newer model\. This reaches 91\.6% pass@1 \(cf\. the 94\.5% labeled ceiling\)\. The most significant component is voting acrossNNcandidates:N=1N\{=\}1loses 8\.6pp\. Other ablations \(removing the repair loop, reducing toK=1K\{=\}1, skipping the code\-review judge\) each cost 2\.1–3\.4pp\. Once we clarify the specification as in Section[4\.3](https://arxiv.org/html/2607.08010#S4.SS3), Opus 4\.7 without labels reaches 99\.1% pass@1\.

## Appendix DThe Repair Loop Unlearns Correct Code

The fourth residual node in Section[4\.3](https://arxiv.org/html/2607.08010#S4.SS3)is a precondition for a downstaffing action: downstaffing only proceeds if the collector belt is sufficiently empty\. The SOP requires, for each quadrant, a 5\-minute average leading up to that quadrant’s most recent timestamp, with no rows older than 10 minutes included\. The round\-0 candidate below is mostly correct, but uses\>\>instead of the SOP’s≥\\geqat the start boundary, which produces 6 test failures in round 1\. The round\-2 candidate replaces the 5\-minute averaging logic with a simple average over the full 10\-minute window, which produces fewer test failures, but no longer matches the SOP\.

fresh=df\[

\(df\[’five\_minute\_interval’\]\>=start\_time\)&

\(df\[’five\_minute\_interval’\]<=current\_time\)

\]

forq,colinquadrants\.items\(\):

q\_data=fresh\[fresh\[col\]\.notna\(\)\]

analysis\_time=q\_data\[’five\_minute\_interval’\]\.max\(\)

five\_min\_start=max\(analysis\_time\-timedelta\(minutes=5\),

start\_time\)

subset=q\_data\[

\(q\_data\[’five\_minute\_interval’\]\>five\_min\_start\)&

\(q\_data\[’five\_minute\_interval’\]<=analysis\_time\)

\]

avgs\[q\]=subset\[col\]\.mean\(\)

fresh=df\[

\(df\[’five\_minute\_interval’\]\>=start\_time\)&

\(df\[’five\_minute\_interval’\]<=current\_time\)

\]

forq,colinquadrants\.items\(\):

avgs\[q\]=fresh\[col\]\.dropna\(\)\.mean\(\)

north\_avg=\(avgs\[’NW’\]\+avgs\[’NE’\]\)/2

south\_avg=\(avgs\[’SW’\]\+avgs\[’SE’\]\)/2

holds=\(north\_avg<=north\_thresholdand

south\_avg<=south\_threshold\)

## Appendix EPer\-Component Output\-Token Breakdown

Section[5](https://arxiv.org/html/2607.08010#S5)reports a 58% reduction in per\-alarm output tokens with tools\. Table[8](https://arxiv.org/html/2607.08010#A5.T8)gives a breakdown by agent role \(main vs\. sub\-agent\) and by content type \(reasoning, prose text, code blocks\)\. The main contributor to the reduction is sub\-agent code generation \(−80%\-80\\%\), which is expected when per\-invocation code generation is replaced with tool calls\. The sub\-agent also needs to reason less, as calling the tool is simpler than writing the code\. The main agent’s tokens also fall \(−16%\-16\\%\), as reading a structured verdict is cheaper than parsing the sub\-agent’s free\-text answer\. The 20% of sub\-agent code tokens that remain consist mostly of the tool calls themselves, which appear as code blocks under the CodeAct\-style execution model, and the agent’s programmatic todo list\.

ComponentWith toolsNo toolsΔ\\DeltaMain: thinking1,1181,349−17%\-17\\%Main: text391436−10%\-10\\%Main: code202259−22%\-22\\%Main total1,7112,044−𝟏𝟔%\-\\mathbf\{16\\%\}Sub: thinking2,0134,144−51%\-51\\%Sub: text1,0351,358−24%\-24\\%Sub: code1,4667,452−80%\-80\\%Sub total4,51412,954−𝟔𝟓%\-\\mathbf\{65\\%\}Total6,22514,998−𝟓𝟖%\-\\mathbf\{58\\%\}Table 8:Per\-trace output\-token breakdown by agent role and content type, per\-alarm production means \(Qwen3 32B\)\.
## Appendix FConfidence Intervals and Significance Tests

Point estimates reported elsewhere in the paper are repeated here with 95% confidence intervals \(mean±1\.96×\\pm 1\.96\\timesSE over per\-seed pass@1 scores\) and corresponding significance tests\.

Table 9:Component ablations of the deployed pipeline with 95% CIs \(pass@1, %\), the confidence intervals for Table[1](https://arxiv.org/html/2607.08010#S4.T1)\. \+D adds the data\-collection trace \(Section[3\.1](https://arxiv.org/html/2607.08010#S3.SS1)\); \+R adds the test\-repair loop\. SOP\+D\+R is the deployed configuration\. Differences between SOP\+D\+R and the SOP\+D and SOP\+R ablations are statistically significant on GLM\-4\.7, GLM\-5, and GLM\-4\.7 Flash \(significance tests in Table[15](https://arxiv.org/html/2607.08010#A6.T15)\)\.Table 10:Trace\-content ablation with CIs \(Table[5](https://arxiv.org/html/2607.08010#A2.T5)\)\. Opus, three repair rounds\.Table 11:Data grounding and repair strategy with CIs and pairedtt\-tests \(Table[6](https://arxiv.org/html/2607.08010#A2.T6)\)\. Opus, three repair rounds, 30 seeds\.#### Per\-seed pass@1 with confidence intervals\.

Tables[9](https://arxiv.org/html/2607.08010#A6.T9),[10](https://arxiv.org/html/2607.08010#A6.T10), and[11](https://arxiv.org/html/2607.08010#A6.T11)give the build\-time pipeline results from Section[4](https://arxiv.org/html/2607.08010#S4)and Appendix[B](https://arxiv.org/html/2607.08010#A2)with 95% CIs and pairedtt\-tests over 30 seeds \(10 for the SOP column\)\.

Table 12:Qwen3 32B, with vs\. without tools \(%\)\. 95% CIs andpp\-values are bootstrap\-by\-task \(nboot=5,000n\_\{\\text\{boot\}\}=5\{,\}000\)\.Table 13:GLM\-4\.5\-Air, with vs\. without tools \(%\)\. 95% CIs andpp\-values are bootstrap\-by\-task \(nboot=5,000n\_\{\\text\{boot\}\}=5\{,\}000\)\.Table 14:GLM\-4\.5\-Air with main\-agent direct tool calls vs\. without tools \(%\)\. 95% CIs andpp\-values are bootstrap\-by\-task \(nboot=5,000n\_\{\\text\{boot\}\}=5\{,\}000\)\.
#### Significance of end\-to\-end correctness \(Table[4](https://arxiv.org/html/2607.08010#S5.T4)\)\.

For each model, we run three seeds with tools and three seeds without tools, and we evaluate each seed on the same 1,500 tasks\. To compute confidence intervals andpp\-values, we aggregate at the task level because we cannot treat the 4,500 \(task, seed\) pairs as independent\. We average the three seed\-level results per task, and we resample tasks with replacement\. The reported 95% CIs are the 2\.5th and 97\.5th percentiles of the resampled accuracy difference \(with tools minus without tools\) overnboot=5,000n\_\{\\text\{boot\}\}=5\{,\}000bootstrap iterations\. The reportedpp\-value is the fraction of bootstrap iterations in which the resampled accuracy difference was zero or had the opposite sign to the observed delta, multiplied by two for the two\-sided test\.

Improvements in overall accuracy are significant for both agent models in the sub\-agent architecture \(Tables[12](https://arxiv.org/html/2607.08010#A6.T12)and[13](https://arxiv.org/html/2607.08010#A6.T13)\) and persist when the main agent calls tools directly \(Table[14](https://arxiv.org/html/2607.08010#A6.T14)\)\. Qwen3 32B’s gains come primarily from improved root\-cause and action accuracy, whereas GLM\-4\.5\-Air analyzes observations more accurately\.

Table 15:Significance of Table[9](https://arxiv.org/html/2607.08010#A6.T9)ablation differences\. \+D adds the data\-collection trace; \+R adds the test\-repair loop\. Pairedtt\-tests over 30 seeds\.Δ\\Deltain pp\.
#### Significance of repair loop and trace contributions\.

For the two best models, GLM\-4\.7 and GLM\-5, all three pairwise comparisons are significant \(p<10−4p<10^\{\-4\}\)\. The deployed configuration \(SOP\+D\+R\) is significantly better than either single\-component ablation, and both components contribute\. On Qwen3 235B, SOP\+D\+R is not significantly better than SOP\+R alone\. Manual inspection of regressions shows that Qwen3 235B mis\-infers schema details from the data\-collection trace \(wrong tool selection, wrong column types, missing columns\), producing tools that crash or return "No Data" on inputs the SOP\-only configuration handles correctly\. On GLM\-4\.7 Flash, the smaller tool\-maker fails to productively use the repair loop without the trace to ground the repairs\. The performance of SOP\+D\+R exceeds that of SOP\+R and SOP\+D, and neither component alone matches the gain of the combined configuration\.

Table 16:Significance of the specification ceiling interventions \(Table[2](https://arxiv.org/html/2607.08010#S4.T2)\)\. Two\-sample Welch’stt\-tests over per\-seed pass@1\.Δ\\Deltain pp\.nx,nyn\_\{x\},n\_\{y\}are the per\-condition seed counts\.
#### Significance of specification interventions\.

Both interventions, clarifying the SOP and augmenting the training data, are individually significant \(Table[16](https://arxiv.org/html/2607.08010#A6.T16)\)\.