AutoGym：蓝图优先的可验证智能体训练环境生成

# AutoGym: Blueprint-First Generation of Verifiable Agent Gyms
Source: [https://arxiv.org/html/2609.22592](https://arxiv.org/html/2609.22592)
\\workshoptitle

Who Verifies the Agents?

Kavya RavikumarAffiliation:Amazon AGICarly Xiaoyu LinAffiliation:Amazon AGI

###### Abstract

Training agents with reinforcement learning requires a*gym*, comprising a task, an executable environment in which the task can be attempted, and a verifier that reliably distinguishes success from failure\. Constructing such gyms remains manual, expensive, and static\. Task sets saturate as models improve and are increasingly exposed to contamination\. Synthetic generation offers scale, but single\-pass synthesis produces tasks whose difficulty is largely cosmetic\. Models comparable in capability solve them despite convoluted phrasing, and correctness must be adjudicated post\-hoc by unreliable LLM judges\. We present AutoGym, a framework that generates complete gyms \(tasks, executable environments, and verifiers\) from a minimal domain seed or prior model trajectories\. AutoGym introduces three mechanisms\. \(1\)*Blueprint\-first generation*specifies the valid solution space, environment requirements, and verification criteria*before*the environment is materialized, making solvability a construction prerequisite rather than a property verified after the fact\. \(2\)*Explicit generation parameters*control task topology, interaction depth, capability axes, question obfuscation, and distractor composition, enabling fine\-grained difficulty steering\. \(3\)*Active curriculum synthesis*uses performance\-informed calibration to adjust the distribution over these parameters as model capabilities evolve\. Across productivity and temporal\-reasoning settings, AutoGym generates gyms spanning the capability spectrum, including instances that challenge frontier models\.

## 1Introduction

Reinforcement learning has become the dominant paradigm for teaching language models to act as agents, enabling them to plan over long horizons, orchestrate tools, and recover from imperfect environments\([Ouyang et al\., 2022](https://arxiv.org/html/2609.22592#bib.bib39);[DeepSeek\-AI, 2025](https://arxiv.org/html/2609.22592#bib.bib41);[Lambert et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib40)\)\. Yet progress is increasingly bottlenecked not by the learning algorithm but by the*gym*\. An agent gym comprises three components\. A task specification defines what the agent must accomplish\. An executable environment provides the world in which the task can be attempted\. A verifier reliably distinguishes success from failure\. Historically, large and carefully constructed evaluation resources have driven rapid progress in NLP\([Rajpurkar et al\., 2016](https://arxiv.org/html/2609.22592#bib.bib1);[Wang et al\., 2019](https://arxiv.org/html/2609.22592#bib.bib2)\), and the same pattern holds for agents, where executable environments with verifiable rewards translate directly into measurable capability gains\([Pan et al\., 2025](https://arxiv.org/html/2609.22592#bib.bib43);[Qi et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib42)\)\. But the environments that enable this progress remain expensive to build, static once built, and increasingly exposed to contamination as models improve\([Golchin and Surdeanu, 2023](https://arxiv.org/html/2609.22592#bib.bib44);[Sainz et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib45)\)\.

A natural response is to synthesize tasks with an LLM\. However, single\-pass generation creates two problems: controlling genuine task difficulty and ensuring reliable verification\. Tasks generated in a single pass are often solvable by models of comparable capability despite convoluted surface phrasing, making their difficulty largely cosmetic\. Without an explicit solution specification, the correct outcome must be inferred after generation, either through costly human annotation or through LLM\-based adjudication that introduces position bias, verbosity bias, and inconsistent scoring\([Zheng et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib36);[Wang et al\., 2024a](https://arxiv.org/html/2609.22592#bib.bib37)\)\. Existing environment\-generation systems address executability\([Hu et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib28);[Song et al\., 2025](https://arxiv.org/html/2609.22592#bib.bib34);[Chen et al\., 2025](https://arxiv.org/html/2609.22592#bib.bib33);[Wang et al\., 2026](https://arxiv.org/html/2609.22592#bib.bib31)\), but they typically generate the task before establishing whether the environment contains a valid and verifiable solution path\.

We present AutoGym, a framework that reverses the conventional generation order\. Rather than first posing a question and then searching for a valid answer, AutoGym begins with a blueprint that specifies the valid solution space, environment requirements, and verification criteria before the environment is instantiated\. This makes solvability and verifiability design constraints rather than post\-hoc checks\. Explicit generation parameters control task topology, interaction depth, capability requirements, obfuscation, and distractor composition independently of the domain or tool interface, enabling systematic variation in difficulty within a fixed setting\. A closed\-loop curriculum further uses model performance to adapt these parameters as capabilities evolve\. Across two evaluation settings, AutoGym systematically produces gyms spanning a broad difficulty range, separates models by capability, and generates instances that remain challenging for frontier models \(§[4](https://arxiv.org/html/2609.22592#S4)\)\.

## 2Related Work

### 2\.1Manually Authored Agent Environments

The dominant approach to evaluating tool\-using agents is careful manual construction\. WebArena and VisualWebArena provide realistic self\-hosted web environments with human\-authored tasks\([Zhou et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib3);[Koh et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib4)\)\. OSWorld extends this to full desktop operating systems\([Xie et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib5)\)\. WorkArena and TheAgentCompany target knowledge work on enterprise platforms\([Drouin et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib13);[Xu et al\., 2024b](https://arxiv.org/html/2609.22592#bib.bib12)\)\. AppWorld couples a simulated application ecosystem with programmatic evaluation\([Trivedi et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib11)\), andτ\\tau\-bench andτ2\\tau^\{2\}\-bench add simulated users and domain policies\([Yao et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib8);[Barres et al\., 2025](https://arxiv.org/html/2609.22592#bib.bib9)\)\. Broader suites span coding, browsing, and interactive tool use\([Liu et al\., 2024a](https://arxiv.org/html/2609.22592#bib.bib6);[Mialon et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib7);[Deng et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib14);[Yang et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib15);[Wang et al\., 2024b](https://arxiv.org/html/2609.22592#bib.bib19)\), SWE\-bench derives verifiable tasks from real GitHub issues\([Jimenez et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib10)\), and function\-calling resources evaluate API invocation at scale\([Patil et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib16);[Qin et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib17);[Li et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib18)\)\. These environments set the standard for realism and programmatic verification\. However, their task distributions are fixed at authoring time\. They saturate as models improve\([Golchin and Surdeanu, 2023](https://arxiv.org/html/2609.22592#bib.bib44);[Sainz et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib45)\), are expensive to extend, and cannot be steered toward the weaknesses of a particular model\.

### 2\.2Synthetic Task and Environment Generation

Self\-Instruct demonstrated that models can bootstrap their own training instructions\([Wang et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib20)\), and Evol\-Instruct that iterative rewriting can increase surface complexity\([Xu et al\., 2024a](https://arxiv.org/html/2609.22592#bib.bib21)\)\. Agent\-oriented pipelines synthesize tool\-use data and trajectories\([Tang et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib25);[Zeng et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib26);[Chen et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib27);[Mitra et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib22)\), with APIGen and APIGen\-MT adding multi\-stage verification of format, execution, and semantics\([Liu et al\., 2024b](https://arxiv.org/html/2609.22592#bib.bib23);[Prabhakar et al\., 2025](https://arxiv.org/html/2609.22592#bib.bib24)\)\. In these pipelines, difficulty and answerability are properties of the generated text rather than of an executable world\. Tasks synthesized in a single pass are generally solvable by same\-class models, and correctness falls back on LLM judgment\.

A newer line of work generates the environment itself\. AgentGen synthesizes planning environments of graded difficulty\([Hu et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib28)\)\. OMNI\-EPIC programs open\-ended environments in code\([Faldor et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib30)\)\. AgentGym and AgentSynth scale task collections across platforms\([Xi et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib29);[Xie et al\., 2025](https://arxiv.org/html/2609.22592#bib.bib35)\)\. Recent systems synthesize tool\-interactive worlds for agentic RL\([Song et al\., 2025](https://arxiv.org/html/2609.22592#bib.bib34);[Chen et al\., 2025](https://arxiv.org/html/2609.22592#bib.bib33);[Zhang et al\., 2025](https://arxiv.org/html/2609.22592#bib.bib32)\), including the Agent World Model framework\([Wang et al\., 2026](https://arxiv.org/html/2609.22592#bib.bib31)\), which decomposes gym creation into scenario, task, data, and verification stages and directly inspired our initial prototype\. SWE\-Gym and WebRL demonstrate that executable environments with verifiable rewards translate into measurable agent improvement\([Pan et al\., 2025](https://arxiv.org/html/2609.22592#bib.bib43);[Qi et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib42)\), consistent with the broader success of RL on verifiable rewards\([Lambert et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib40);[DeepSeek\-AI, 2025](https://arxiv.org/html/2609.22592#bib.bib41)\)\. These systems improve executability and scale, but they typically generate the task before establishing whether the environment contains a valid and verifiable solution path\.

### 2\.3Verification and Reward Design

LLM judges are scalable but exhibit position, verbosity, and self\-preference biases\([Zheng et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib36);[Wang et al\., 2024a](https://arxiv.org/html/2609.22592#bib.bib37)\)\. Process supervision improves reliability by evaluating intermediate reasoning steps\([Lightman et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib38)\), while execution\-based checking grounds correctness in observable outcomes\([Jimenez et al\., 2024](https://arxiv.org/html/2609.22592#bib.bib10);[Yang et al\., 2023](https://arxiv.org/html/2609.22592#bib.bib15);[Liu et al\., 2024b](https://arxiv.org/html/2609.22592#bib.bib23)\)\.τ2\\tau^\{2\}\-bench introduces compositional task generators with programmatic assertion functions\([Barres et al\., 2025](https://arxiv.org/html/2609.22592#bib.bib9)\), demonstrating that tasks can be both generated at scale and verified deterministically when the solution space is defined at construction time\.

### 2\.4Positioning

Prior work addresses individual components of the gym construction problem\. Manually authored benchmarks achieve realism and programmatic verification but cannot adapt to evolving model capabilities\. Synthetic pipelines achieve task generation at scale but often leave solvability, verification, and difficulty control as downstream or implicit concerns\. AutoGym addresses these limitations by jointly constructing the task, executable environment, and verifiers from an explicit solution blueprint\. It further exposes generation parameters for controllable difficulty and performance\-adaptive curriculum synthesis\.

Figure 1:Overview of the AutoGym pipeline\. A blueprint specifying the task context, valid solution space, environment requirements, and verification criteria is constructed first, then materialized into an executable environment with controlled distractors, and finally paired with verifiers derived from the blueprint\.

## 3Methodology

AutoGym constructs agent gyms from the inside out\. It first defines a*blueprint*containing the operational context, valid solution space, environment requirements, and verification criteria\. The blueprint is then materialized as an executable environment containing task\-supporting resources and controlled distractors\. This ordering makes solvability and verifiability construction requirements while keeping difficulty under explicit parametric control\. Figure[1](https://arxiv.org/html/2609.22592#S2.F1)gives an overview of the pipeline\. We trace a running example from the procurement domain through each stage \(the boxed*Running Example*panels\)\. To show the pipeline generalizes, a complete worked task from a*different*domain \(K–12 education\) is given in Appendix[D](https://arxiv.org/html/2609.22592#A4), and prompt templates occupy Appendix[H](https://arxiv.org/html/2609.22592#A8)\.

### 3\.1Generation inputs

The domain seed specifies the target platform and application setting, while the tool specification defines the operations and schemas available to the agent\. Together, these inputs constrain generated tasks to a coherent domain and an executable action space\. Prior trajectories, when available, provide performance signals for active curriculum synthesis \(Section[3\.6](https://arxiv.org/html/2609.22592#S3.SS6)\)\. Generated entities are grounded against domain\-specific corpora and retrieval sources so that environment content reflects real\-world distributions\. We illustrate the pipeline with a task generated from the procurement domain\.

Running Example: Generation Inputs

### 3\.2Blueprint initialization

A*scenario*and*persona*establish the operational context in which a task is instantiated, including available resources, user role, and access constraints\. AutoGym additionally samples*generation parameters*that determine the intended task structure before task\-specific content is produced\. These parameters control task topology \(Appendix[A\.1](https://arxiv.org/html/2609.22592#A1.SS1)\), interaction depth, capability axes \(Appendix[A\.2](https://arxiv.org/html/2609.22592#A1.SS2)\), controlled obfuscation \(Appendix[A\.3](https://arxiv.org/html/2609.22592#A1.SS3)\), and distractor composition \(Appendix[A\.4](https://arxiv.org/html/2609.22592#A1.SS4)\)\.

Separating these controls from the domain allows AutoGym to vary structural difficulty while holding the platform and tool interface fixed\. The same pipeline can therefore support shallow retrieval tasks or tasks requiring cross\-source reconciliation, deeper interaction, and stronger distractor resistance\.

Running Example: Blueprint Initializationgeneration parameters

### 3\.3Blueprint construction

The blueprint specifies the information structure that makes a task solvable\. Its*solution space*identifies the entities, attributes, relationships, and conditions required to reach a correct outcome, while its*verification criteria*define the outcome constraints\. A dependency structure may encode intermediate information requirements without prescribing a unique agent trajectory\. Controlled obfuscation \(Appendix[A\.3](https://arxiv.org/html/2609.22592#A1.SS3)\) determines which solution\-relevant values appear in the question and which must instead be recovered from the environment\. Increasing obfuscation changes the interaction required to solve the task without changing the underlying solution\.

The blueprint also defines distractors \(Appendix[A\.4](https://arxiv.org/html/2609.22592#A1.SS4)\) such as stale records, conflicting versions, misleading resources, access constraints, and tool\-level complications\. These may increase navigation and reasoning difficulty but must preserve at least one valid route to success\. A reconciliation check enforces this requirement before materialization by ensuring that all solution\-relevant entities are specified, withheld information remains discoverable, and the tool interface supports the intended solution\.

Running Example: Generated BlueprintSolution path:Marcus filters the delivery log to his APAC metals suppliers for Q3\-2026, computes their on\-time delivery rate from theon\_time\_flagcolumn, retrieves the category\-specific SLA threshold from the procurement policies table, and compares the two only to determine that some of his vendors are not meeting the standard\. Answer\-relevant entities:The solution draws on two tables: \(i\)supplier\_delivery\_log\(4 of 13 columns relevant\):commodity\_group,vendor\_region,fiscal\_quarter,on\_time\_flag; and \(ii\)procurement\_policies\(3 of 9 columns relevant\):policy\_type,commodity\_group,threshold\_pct\. Obfuscated constraints:The question states none of the filter values explicitly\. “APAC vendors” implies both the commodity group and region but names neither field nor value; “this quarter” requires identifying Q3\-2026 as the current fiscal period; “hitting the mark” refers to a delivery SLA threshold the agent must locate inprocurement\_policies; and the relevant metric \(on\_time\_flag\) is never mentioned, leaving the agent to determine which of 13 columns encodes delivery performance\. Distractors: Region reclassification:8 records havevendor\_region= ‘APAC’ butregion\_override= EMEA \(mid\-quarter transfer\)\. Near\-match commodity:12 records use “Precision Metals” \(retired label\) instead of the canonical name\. Competing threshold:POL\-089 sets 85% for a different metric \(fill rate, not on\-time delivery\)\. Pagination boundary:Log returns 50 rows per page; 96 relevant records span two pages\. Verification criteria:Reports on\-time delivery rate; identifies the applicable SLA threshold; concludes whether vendors meet it\. Question:“Something’s off with what was reported last week about our APAC vendors being fine\. Can you check whether they’re actually hitting the mark this quarter?”

### 3\.4Maze materialization

The reconciled blueprint induces an executable environment containing the resources, relationships, permissions, and distractors required by the task\. Information omitted from the request is instantiated within the environment so that it can be recovered through agent interaction\. The resulting resources are exposed through the supplied tool interface\.

This construction is substrate\-independent\. Our primary experiments use a relational store accessed through tool wrappers, but the same blueprint\-first ordering applies to stateful or service\-backed environments \(Appendix[E](https://arxiv.org/html/2609.22592#A5)\)\.

Running Example: Maze MaterializationEnvironment:supplier\_delivery\_log\(800 rows, 13 columns\) andprocurement\_policies\(200 rows, 9 columns\), including distractor records \(reclassified regions, retired labels, competing thresholds\)\. Tool Interface:list\_topics,get\_dataset\_columns,execute\_codewith pagination where needed\.

### 3\.5Ground truth, verification, and repair

Every generated gym must satisfy an*answerability invariant*: the task and the environment accessible to the agent must jointly contain sufficient information to satisfy the verification criteria\. Difficulty may therefore arise from finding, filtering, or reconciling information, but not from inaccessible hidden state\. Ground truth is computed from the*materialized environment*by executing the blueprint’s solution derivation, rather than by trusting values proposed during generation\. A separate validation pass checks that the derivation uses authoritative resources, remains unaffected by distractors, and does not admit another equally defensible outcome\.

The verifier is then derived from the resulting ground truth and blueprint criteria\. Deterministic outcomes are checked programmatically against expected values, state changes, or entity sets\. Additional checks can enforce grounding, distractor resilience, required intermediate subgoals, and persona constraints\. Persona verifiers are only applied when the blueprint specifies persona preferences\. Open\-ended components are evaluated only against blueprint\-defined evidence and factual constraints\. If validation reveals a missing resource, unreachable criterion, or inconsistent derivation, the responsible component is repaired and re\-materialized\. Agent failures on otherwise valid gyms do not trigger repair\.

Running Example: Ground Truth Verification and RepairGround truth\(computed by walking the blueprint’s solution logic against the materialized environment\): Verifier: Outcome correctness:•Reports on\-time delivery rate within±\\pm1\.0 of 81\.25%•Identifies the correct threshold as 90\.0% \(from POL\-027, not POL\-089\)•Concludes that APAC metals vendors are not meeting the SLADistractor resilience:•Does not use the 85\.0% threshold from POL\-089 \(wrong metric type\)•Does not include the 8 records with region\_override = EMEA•Does not include “Precision Metals” records \(retired category label\)•Paginates past the first 50 results to capture all 96 relevant deliveriesProcess quality:•Applies all three filters \(commodity group, region, fiscal quarter\) simultaneously•Looks up the SLA threshold from the policies table rather than assuming a value•Selects the correct policy type \(on\_time\_delivery, not delivery\_performance\)

### 3\.6Active curriculum synthesis

A fixed generation distribution provides coverage but does not remain calibrated as model capabilities change\. AutoGym therefore uses model performance to adapt the distribution over generation parameters\. When prior model trajectories are available as the initial input, AutoGym can begin directly from failure analysis rather than requiring a seed\-driven first round\.

The target model is evaluated through multiple rollouts, and performance is aggregated by capability dimension\. Trajectory analysis identifies recurring failure patterns while separately checking for evaluation defects\. AutoGym then adjusts the sampling weights for topology, capability axes, obfuscation, and distractor composition\. Dimensions below0\.300\.30success are treated as too difficult and relaxed; those above0\.800\.80are strengthened\. Updates are bounded to reduce oscillation\. The updated parameters define the next generation round; the process continues until performance stabilizes or the iteration limit is reached\.

Running Example: Curriculum AdaptationObserved failures:Agents select POL\-089 \(85%, fill rate\) instead of POL\-027 \(90%, on\-time delivery\) when both apply to the same commodity group\. Separately, agents retrieve only the first page of results and compute OTD from 50 records rather than the full 96\. Parameter update:Raise sampling weight for implicit\-criteria dimension; increase distractor budget for competing\-threshold and pagination\-boundary types\. Effect on next round:Generated tasks more frequently require disambiguating between multiple plausible policy entries and paginating to retrieve complete result sets\.

## 4Experiments and Results

We evaluate AutoGym against three research questions\.

1. Q1\.Difficulty:Do generated tasks occupy a useful difficulty range for a frontier model, and can generation parameters shift this distribution relative to an expert\-authored baseline?
2. Q2\.Cross\-model discrimination:Does performance on generated tasks distinguish models of different capability levels, and can failure\-driven generation target an identified capability gap?
3. Q3\.Gym quality:Are the generated gyms grounded, realistic, internally consistent, and reliably verifiable under a structured certification audit?

We study these three questions in two detailed evaluation settings: a*seed\-driven productivity gym*\(two configurations of 350 candidates each, yielding 297 and 280 tasks after repair; Section[4\.1](https://arxiv.org/html/2609.22592#S4.SS1)\), which tests controllable generation from a minimal domain seed; and a*failure\-driven temporal\-reasoning gym*\(100 candidates, 92 retained; Section[4\.2](https://arxiv.org/html/2609.22592#S4.SS2)\), which tests active curriculum synthesis from prior model trajectories\.

Both use the complete AutoGym pipeline, including iterative repair\. Generation uses Claude Sonnet 4\.6\. Evaluated agents receive only the user\-facing task and access to the materialized environment\. Blueprints, ground truth, and verifier criteria remain hidden\. Claude Opus 4\.6 serves as the frontier reference model, with gpt\-oss\-120b for cross\-model comparison\. We ran eight independent rollouts per task\. A rollout succeeds if its verifier score is 1, and tasks are classified by mean score as*hard*\(<0\.30<0\.30\),*medium*\(0\.300\.30–0\.800\.80\), or*easy*\(≥0\.80\\geq 0\.80\)\.

Two further studies appear in the appendices\. AutoGym generated a six\-domain suite of 600 tasks in under ten hours demonstrating scalability and cost efficiency\. Generation cost was approximately $100–200 per 50\-task batch with 20\-way parallelism, and 86% of tasks were retained after repair \(Appendix[F](https://arxiv.org/html/2609.22592#A6)\)\. Separately, we built a 600\-task instruction\-following gym and ran GRPO on an 8B model, producing a clear learning signal \(Appendix[G](https://arxiv.org/html/2609.22592#A7)\)\.

### 4\.1Seed\-driven Productivity gym

We evaluate seed\-driven generation in a productivity environment with seven tools supporting document and spreadsheet reading, contact lookup, and task tracking\. AutoGym independently generated 350 candidate tasks each under 2 configurations C1 and C2\. We additionally compare against a human expert\-authored suite and Agent World Model \(AWM\)\([Wang et al\., 2026](https://arxiv.org/html/2609.22592#bib.bib31)\), both using the same tool interface\. Of 300 generated AWM tasks, 217 were retained after manual review for verifier correctness and environment consistency\.

##### Difficulty \(Q1\):

Table[1](https://arxiv.org/html/2609.22592#S4.T1)compares difficulty distributions\. The Agent World Model baseline is predominantly easy for Claude Opus 4\.6; while 21\.5% of the human expert\-authored suite is hard\. Under configuration C1, where generation parameters sit at the lower ends of their ranges, AutoGym produces 10% hard, 34\.7% medium, and 55\.3% easy tasks\. Under configuration C2, where the same parameters are pushed toward the mid\-to\-hard range, 39% of generated tasks fall in the hard band\. This is a substantially greater concentration near the model’s capability frontier\.

The two configurations differ only in topology tier weights, obfuscation gradient, and distractor budgets\. Full settings appear in Appendix[A\.5](https://arxiv.org/html/2609.22592#A1.SS5)\. This demonstrates that the generation parameters can shift the difficulty distribution while preserving the domain and tool interface\. The distinction matters for training\. A gym that is uniformly hard yields sparse reward, and one that is uniformly easy yields no learning signal\.

Table 1:Difficulty distributions for the productivity gym \(eight\-rollout protocol\)\. Tasks are categorized by mean verifier score: hard \(<0\.30<0\.30\), medium \(0\.300\.30–0\.800\.80\), easy \(≥0\.80\\geq 0\.80\)\. All conditions share the same 7\-tool interface\. C1/C2 settings in Appendix[A\.5](https://arxiv.org/html/2609.22592#A1.SS5)\.
##### Cross\-model discrimination \(Q2\):

Low mean verifier score alone does not establish that tasks are well\-designed, since broken or underspecified tasks also produce low scores\. We therefore evaluate the same C2 gym with gpt\-oss\-120b under identical conditions\. For Claude Opus 4\.6, 61% of tasks fall in the medium or easy bands\. For gpt\-oss\-120b, only 30% do, with 70% classified as hard\. This separation is consistent with the generated tasks measuring genuine capability demands \(multi\-step reasoning, evidence integration, tool orchestration\) rather than producing uniform evaluation noise\.

##### Gym quality \(Q3\):

After three repair iterations, 297 of 350 C1 candidates \(84\.9% yield\) and 280 of 350 C2 candidates \(80\.0% yield\) were retained for evaluation\. Tasks that remained inconsistent or unverifiable were excluded from evaluation\. Table[2](https://arxiv.org/html/2609.22592#S4.T2)compares the difficulty distribution before and after repair for C2, showing how validation removes cases whose apparent difficulty arises from environment, ground\-truth, or verifier defects rather than agent capability\.

Table 2:Difficulty distribution before and after three repair iterations on gym generated with C2 configuration \(productivity gym, Claude Opus 4\.6\)\.All retained tasks were independently validated by human inspectors who verified that \(1\) the ground truth is reachable from the materialized environment under the persona’s access scope, \(2\) no distractor blocks all valid solution routes, and \(3\) verifier criteria match the intended task semantics\.

Complementing per\-task human validation, an independent LLM certification pipeline evaluated the generated gym under the C2 configuration against a fixed seven\-dimension rubric in Appendix[B](https://arxiv.org/html/2609.22592#A2)\. Task\-level dimensions assess individual task and environment quality, while suite\-level dimensions such as scalability and diversity are assessed across the gym as a whole\. The gym scores highest on scope compliance \(92\.7\) and groundedness \(91\.4\)\. The high groundedness reflects the pipeline’s access to an internal domain corpus and web retrieval during generation, anchoring entities in real sources\. The weakest dimension is interface realism \(78\.0\), a consequence of the tool\-mocking layer requiring departures from the deployed interface \(e\.g\., per\-task databases\)\. The lower environment\-realism score \(86\.0\) was primarily attributable to distractors that were unnaturally integrated\.

Table 3:Certification scores for the productivity gym \(0–100 scale\)\.

### 4\.2Failure\-driven temporal\-reasoning gym

Table 4:Difficulty distributions for the failure\-driven temporal\-reasoning gym \(eight\-rollout protocol\)\. Tasks are categorized by mean verifier score: hard \(<0\.30<0\.30\), medium \(0\.300\.30–0\.800\.80\), easy \(≥0\.80\\geq 0\.80\)\.The second setting evaluates active curriculum synthesis \(Section[3\.6](https://arxiv.org/html/2609.22592#S3.SS6)\)\. Failure analysis of trajectories from gpt\-oss\-120b identified temporal reasoning \(relative dates, recurring schedules, time zones, and ordering constraints\) as a recurring failure mode\. AutoGym calibrated capability weights and distractor patterns toward this gap, producing a 92\-task gym concentrated on temporal reasoning\.

##### Difficulty \(Q1\):

Table[4](https://arxiv.org/html/2609.22592#S4.T4)shows that the gym remains within Claude Opus 4\.6’s capability range \(89\.1% easy\), while 41\.4% of tasks are hard or medium for gpt\-oss\-120b\. The curriculum thus produces targeted difficulty\. The gym challenges the model whose failures drove generation without being uniformly hard for stronger models\.

##### Cross\-model discrimination \(Q2\):

The task\-level ordering is strictly one\-sided\. All 10 tasks outside Opus 4\.6’s easy band also fall in gpt\-oss\-120b’s hard band, and 9 additional tasks are hard for gpt\-oss\-120b alone\. No task is harder for the frontier model than for the weaker one\. This one\-sided asymmetry is consistent with targeted generation toward the identified capability gap, rather than difficulty that affects both models uniformly\.

##### Gym quality \(Q3\):

The repair pipeline retained 92 of 100 candidates \(92% yield\)\. A frontier model solves 89\.1% of tasks\. This strongly signals that ground truth is reachable across nearly all retained instances\. For the 10 tasks outside Opus 4\.6’s easy band, we performed manual inspection and confirmed that each is well\-formed\. Difficulty on these tasks arises from genuine temporal\-reasoning demands \(multi\-step date arithmetic, timezone conversions, schedule\-overlap detection\), not from environment or verifier defects\. Table[5](https://arxiv.org/html/2609.22592#S4.T5)reports certification scores; the higher scores relative to the productivity gym reflect the simpler environment structure of a single\-axis gym \(fewer entities, fewer cross\-source dependencies\)\.

Table 5:Certification scores for the temporal reasoning gym \(0–100 scale\)\.

## 5Limitations

Interface and environment realism:Certification identifies interface realism as the weakest dimension \(78\.0\), a consequence of the tool\-mocking layer and its departures from the deployed interface \(per\-task databases, simplified schemas\)\. The reduced environment\-realism score \(86\.0\) reflects distractors that were unnaturally integrated\. Both point toward tighter coupling with production APIs rather than fundamental framework constraints\.

Verification coverage:While the answerability invariant and multi\-signal verifier provide strong guarantees for deterministic tasks, open\-ended tasks still rely partly on LLM\-based semantic judgment\. The process quality signal mitigates this by independently evaluating the trajectory, but eliminating LLM adjudication entirely for free\-form tasks remains an open problem\.

Downstream training:Our RL experiment \(Appendix[G](https://arxiv.org/html/2609.22592#A7)\) is limited to a single 8B model over 500 GRPO steps\. Full\-scale training across model sizes, longer horizons, and multiple gym domains remains future work\.

Substrate evaluation:The construction generalizes to a live cloud\-emulator substrate \(Appendix[E](https://arxiv.org/html/2609.22592#A5)\), but empirical evaluation of difficulty and discrimination on it is future work\.

## 6Conclusion

Across two evaluation settings, AutoGym produces gyms that challenge frontier models, discriminate between models of different strength, and pass independent quality certification\. These results suggest that gym construction, which has long been the manual bottleneck for agent evaluation, can itself be automated, steered, and kept current\.

## Acknowledgments and Disclosure of Funding

We thank Nehal Belgamwar, Arpit Gupta, Rajiv Reddy and Yang Liu for feedback and guidance, and Amazon AGI for resources supporting this work\.

## References

- Barreset al\.\(2025\)V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhanτ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.arXiv preprint arXiv:2506\.07982\.Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2609.22592#S2.SS3.p1.1)\.
- Chenet al\.\(2023\)B\. Chen, C\. Shu, E\. Shareghi, N\. Collier, K\. Narasimhan, and S\. YaoFireAct: toward language agent fine\-tuning\.arXiv preprint arXiv:2310\.05915\.Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p1.1)\.
- Chenet al\.\(2025\)Z\. Chen, Z\. Zhao, K\. Zhang, B\. Liu, Q\. Qi, Y\. Wu, T\. Kalluri, S\. Cao, Y\. Xiong, H\. Tong, H\. Yao, H\. Li, J\. Zhu, X\. Li, D\. Song, B\. Li, J\. Weston, and D\. HuynhScaling agent learning via experience synthesis\.arXiv preprint arXiv:2511\.03773\.Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-AIDeepSeek\-r1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1)\.
- Denget al\.\(2023\)X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. SuMind2Web: towards a generalist agent for the web\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Note:arXiv:2306\.06070Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Drouinet al\.\(2024\)A\. Drouin, M\. Gasse, M\. Caccia, I\. H\. Laradji, M\. Del Verme, T\. Marty, L\. Boisvert, M\. Thakkar, Q\. Cappart, D\. Vazquez,et al\.WorkArena: how capable are web agents at solving common knowledge work tasks?\.InInternational Conference on Machine Learning \(ICML\),Note:arXiv:2403\.07718Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Faldoret al\.\(2024\)M\. Faldor, J\. Zhang, A\. Cully, and J\. CluneOMNI\-EPIC: open\-endedness via models of human notions of interestingness with environments programmed in code\.arXiv preprint arXiv:2405\.15568\.Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1)\.
- Golchin and Surdeanu \(2023\)S\. Golchin and M\. SurdeanuTime travel in LLMs: tracing data contamination in large language models\.arXiv preprint arXiv:2308\.08493\.Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Huet al\.\(2024\)M\. Hu, P\. Zhao, C\. Xu, Q\. Sun, J\. Lou, Q\. Lin, P\. Luo, S\. Rajmohan, and D\. ZhangAgentGen: enhancing planning abilities for large language model based agents via environment and task generation\.arXiv preprint arXiv:2408\.00764\.Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSWE\-bench: can language models resolve real\-world GitHub issues?\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2310\.06770Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2609.22592#S2.SS3.p1.1)\.
- Kohet al\.\(2024\)J\. Y\. Koh, R\. Lo, L\. Jang, V\. Duvvur, M\. C\. Lim, P\. Huang, G\. Neubig, S\. Zhou, R\. Salakhutdinov, and D\. FriedVisualWebArena: evaluating multimodal agents on realistic visually grounded web tasks\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2401\.13649Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Lambertet al\.\(2024\)N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu,et al\.Tülu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1)\.
- Liet al\.\(2023\)M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. LiAPI\-bank: a comprehensive benchmark for tool\-augmented LLMs\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Note:arXiv:2304\.08244Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2305\.20050Cited by:[§2\.3](https://arxiv.org/html/2609.22592#S2.SS3.p1.1)\.
- Liuet al\.\(2024a\)X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang,et al\.AgentBench: evaluating LLMs as agents\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2308\.03688Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Liuet al\.\(2024b\)Z\. Liu, T\. Hoang, J\. Zhang, M\. Zhu, T\. Lan, S\. Kokane, J\. Tan, W\. Yao, Z\. Liu, Y\. Feng,et al\.APIGen: automated pipeline for generating verifiable and diverse function\-calling datasets\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2406\.18518Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p1.1),[§2\.3](https://arxiv.org/html/2609.22592#S2.SS3.p1.1)\.
- Mialonet al\.\(2023\)G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. ScialomGAIA: a benchmark for general AI assistants\.arXiv preprint arXiv:2311\.12983\.Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Mitraet al\.\(2024\)A\. Mitra, L\. Del Corro, G\. Zheng, S\. Mahajan, D\. Rouhana, A\. Codas, Y\. Lu, W\. Chen, O\. Vrousgos, C\. Rosset,et al\.AgentInstruct: toward generative teaching with agentic flows\.arXiv preprint arXiv:2407\.03502\.Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2203\.02155Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p1.1)\.
- Panet al\.\(2025\)J\. Pan, X\. Wang, G\. Neubig, N\. Jaitly, H\. Ji, A\. Suhr, and Y\. ZhangTraining software engineering agents and verifiers with SWE\-gym\.InInternational Conference on Machine Learning \(ICML\),Note:arXiv:2412\.21139Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1)\.
- Patilet al\.\(2023\)S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. GonzalezGorilla: large language model connected with massive APIs\.arXiv preprint arXiv:2305\.15334\.Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Prabhakaret al\.\(2025\)A\. Prabhakar, Z\. Liu, M\. Zhu, J\. Zhang, T\. Awalgaonkar, S\. Wang, Z\. Liu, H\. Chen, T\. Hoang, J\. C\. Niebles, S\. Heinecke, W\. Yao, H\. Wang, S\. Savarese, and C\. XiongAPIGen\-MT: agentic pipeline for multi\-turn data generation via simulated agent\-human interplay\.arXiv preprint arXiv:2504\.03601\.Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p1.1)\.
- Qiet al\.\(2024\)Z\. Qi, X\. Liu, I\. L\. Iong, H\. Lai, X\. Sun, X\. Yang, J\. Sun, Y\. Yang, S\. Yao, T\. Zhang,et al\.WebRL: training LLM web agents via self\-evolving online curriculum reinforcement learning\.arXiv preprint arXiv:2411\.02337\.Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1)\.
- Qinet al\.\(2024\)Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.ToolLLM: facilitating large language models to master 16000\+ real\-world APIs\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2307\.16789Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Rajpurkaret al\.\(2016\)P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. LiangSQuAD: 100,000\+ questions for machine comprehension of text\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 2383–2392\.Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p1.1)\.
- Sainzet al\.\(2023\)O\. Sainz, J\. A\. Campos, I\. García\-Ferrero, J\. Etxaniz, O\. L\. de Lacalle, and E\. AgirreNLP evaluation in trouble: on the need to measure LLM data contamination for each benchmark\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Note:arXiv:2310\.18018Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Songet al\.\(2025\)X\. Song, H\. Chang, G\. Dong, Y\. Zhu, J\. Wen, and Z\. DouEnvScaler: scaling tool\-interactive environments for LLM agents via programmatic synthesis\.arXiv preprint arXiv:2601\.05808\.Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1)\.
- Tanget al\.\(2023\)Q\. Tang, Z\. Deng, H\. Lin, X\. Han, Q\. Liang, B\. Cao, and L\. SunToolAlpaca: generalized tool learning for language models with 3000 simulated cases\.arXiv preprint arXiv:2306\.05301\.Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p1.1)\.
- Trivediet al\.\(2024\)H\. Trivedi, T\. Khot, M\. Hartmann, R\. Manku, V\. Dong, E\. Li, S\. Gupta, A\. Sabharwal, and N\. BalasubramanianAppWorld: a controllable world of apps and people for benchmarking interactive coding agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2407\.18901Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Wanget al\.\(2019\)A\. Wang, Y\. Pruksachatkun, N\. Nangia, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. BowmanSuperGLUE: a stickier benchmark for general\-purpose language understanding systems\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p1.1)\.
- Wanget al\.\(2024a\)P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. SuiLarge language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2305\.17926Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.22592#S2.SS3.p1.1)\.
- Wanget al\.\(2024b\)X\. Wang, Z\. Wang, J\. Liu, Y\. Chen, L\. Yuan, H\. Peng, and H\. JiMINT: evaluating LLMs in multi\-turn interaction with tools and language feedback\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2309\.10691Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Wanget al\.\(2023\)Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. HajishirziSelf\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2212\.10560Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p1.1)\.
- Wanget al\.\(2026\)Z\. Wang, C\. Xu, B\. Liu, Y\. Wang, S\. Han, Z\. Yao, H\. Yao, and Y\. HeAgent world model: infinity synthetic environments for agentic reinforcement learning\.arXiv preprint arXiv:2602\.10090\.Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2609.22592#S4.SS1.p1.1)\.
- Xiet al\.\(2024\)Z\. Xi, Y\. Ding, W\. Chen, B\. Hong, H\. Guo, J\. Wang, D\. Yang, C\. Liao, X\. Guo, W\. He,et al\.AgentGym: evolving large language model\-based agents across diverse environments\.arXiv preprint arXiv:2406\.04151\.Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1)\.
- Xieet al\.\(2025\)J\. Xie, D\. Xu, X\. Zhao, and D\. SongAgentSynth: scalable task generation for generalist computer\-use agents\.arXiv preprint arXiv:2506\.14205\.Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1)\.
- Xieet al\.\(2024\)T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei,et al\.OSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.arXiv preprint arXiv:2404\.07972\.Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Xuet al\.\(2024a\)C\. Xu, Q\. Sun, K\. Zheng, X\. Geng, P\. Zhao, J\. Feng, C\. Tao, and D\. JiangWizardLM: empowering large language models to follow complex instructions\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2304\.12244Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p1.1)\.
- Xuet al\.\(2024b\)F\. F\. Xu, Y\. Song, B\. Li, Y\. Tang, K\. Jain, M\. Bao, Z\. Z\. Wang, X\. Zhou, Z\. Guo, M\. Cao,et al\.TheAgentCompany: benchmarking LLM agents on consequential real world tasks\.arXiv preprint arXiv:2412\.14161\.Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Yanget al\.\(2023\)J\. Yang, A\. Prabhakar, K\. Narasimhan, and S\. YaoInterCode: standardizing and benchmarking interactive coding with execution feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Note:arXiv:2306\.14898Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2609.22592#S2.SS3.p1.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.
- Zenget al\.\(2023\)A\. Zeng, M\. Liu, R\. Lu, B\. Wang, X\. Liu, Y\. Dong, and J\. TangAgentTuning: enabling generalized agent abilities for LLMs\.arXiv preprint arXiv:2310\.12823\.Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p1.1)\.
- Zhanget al\.\(2025\)J\. Zhang, Y\. Peng, F\. Kong, C\. Yang, Y\. Wu, Z\. Yu, J\. Xiang, J\. Ruan, J\. Wang, M\. Song, H\. Liu, X\. Tang, B\. Liu, C\. Wu, and Y\. LuoAutoEnv: automated environments for measuring cross\-environment agent learning\.arXiv preprint arXiv:2511\.19304\.Cited by:[§2\.2](https://arxiv.org/html/2609.22592#S2.SS2.p2.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Note:arXiv:2306\.05685Cited by:[§1](https://arxiv.org/html/2609.22592#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.22592#S2.SS3.p1.1)\.
- Zhouet al\.\(2024\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. NeubigWebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2307\.13854Cited by:[§2\.1](https://arxiv.org/html/2609.22592#S2.SS1.p1.1)\.

## Appendix AGeneration Parameters

AutoGym exposes several categories of generation parameters that jointly determine the structure, difficulty, and content of generated tasks\. This appendix defines each category and reports the configurations used in our experiments\.

### A\.1Task Topology

Task topology defines the cognitive structure of a task, independent of any particular tool interface or domain\. We arrived at this taxonomy by surveying tasks across different domains, observing that the reasoning patterns that recur cluster into a small set of structural types at three levels of compositional complexity\. Each task is assigned a*tier*and a*pattern*within that tier\. The distribution over tiers is configured as sampling weights\. Within a tier, the pattern is sampled uniformly\. For Tier 2 and Tier 3 tasks, a supporting pattern from a lower tier may be added with configurable probability, introducing a second reasoning phase\.

This taxonomy is not fixed\. When failure analysis \(Section[3\.6](https://arxiv.org/html/2609.22592#S3.SS6)\) identifies a recurring topology that does not map to an existing entry, it is added as a new pattern at the appropriate tier with its own configurable range\.

Table[6](https://arxiv.org/html/2609.22592#A1.T6)defines all patterns\.

Table 6:Task topology patterns\. Each pattern defines a structural type of reasoning\. The configurable range controls within\-pattern difficulty\.TierPatternDescriptionConfigurableDefault Range1Direct RetrievalLocate one entity, extract a value or subset––2Multi\-hopOutput of query A is the filter for query BNo of hops2–42Multi\-constraintMultiple filter criteria applied simultaneouslyNo of constraints2–52List AggregationCollect all matching items exhaustivelyNo of list elements3–203Cross\-source ComparisonSame metric from 2\+ independent sources, compare or rankNo of sources2–43Parallel SynthesisN entities, different dimensions of same subject, combine into unified answerNo of entities to merge3–53Multi\-hop \+ MergeDiscover criteria via lookup, then apply across multiple entitiesNo of hops×\\timesentities2–3×\\times2–33Complex ComparisonCompute a derived metric per group, then rank or flag outliersNo of groups, metrics2–5, 1–3Examples:

Tier 1, Direct Retrieval:“What is the current status of the Horizon project?” The agent retrieves a single project record and extracts one field\.

Tier 2, Multi\-hop:“Who is the manager of the person who filed the most expense reports last quarter?” The agent identifies the top filer \(hop 1\), then looks up their manager \(hop 2\)\.

Tier 2, Multi\-constraint:“Find all contracts expiring within 30 days that exceed $50K and are not yet renewed\.” The agent applies three simultaneous filters \(date, amount, renewal status\)\.

Tier 2, List Aggregation:“List every unresolved support ticket assigned to the EMEA team\.” The agent must paginate through all results and collect the complete set\.

Tier 3, Cross\-source Comparison:“Compare the on\-time delivery rates reported in the vendor scorecard versus the logistics dashboard for our top 3 suppliers\.” The agent retrieves the same metric from two independent sources and reconciles discrepancies\.

Tier 3, Parallel Synthesis:“Prepare a briefing on Project Atlas covering timeline, budget status, risk register, and staffing\.” The agent pulls from four different entities and synthesizes into one answer\.

Tier 3, Multi\-hop \+ Merge:“Which team lead owns the most overdue deliverables, and what are those deliverables?” The agent first discovers which deliverables are overdue \(hop 1\), groups by owner \(hop 2\), then retrieves the specific items for the top lead\.

Tier 3, Complex Comparison:“Which vendor had the largest week\-over\-week drop in on\-time delivery, and what purchase orders drove it?” The agent computes a derived metric per vendor \(WoW change\), ranks to find the worst, then cross\-references purchase order data to attribute cause\.

### A\.2Capability Axes

Capability axes define the cognitive skills a task exercises, orthogonal to its topology\. A task’s topology determines its structural shape \(e\.g\., multi\-hop\), while capability axes determine what kind of reasoning fills that shape\. Each axis can be sampled independently and composed with any topology pattern\. Multiple axes may be composed within a single task\.

The specific axes available depend on the domain and grounding sources \(web search, document retrieval, database access, etc\.\)\. The following are examples from the productivity setting used in our experiments\.

Capability axes are sampled after topology\. Once a tier and pattern are selected, the number of axes is determined by the tier \(1 for Tier 1, 2 for Tier 2, 2\+ for Tier 3\)\. Axes are then drawn from a weighted distribution over the available set\. All sampled axes must be reflected in the blueprint’s state space, answer derivation, and verifier constraints\. Axis weights are configurable to skew generation toward specific capabilities\.

Topology and capability axes are orthogonal by construction: the tier and pattern fix thestructural shapeof the task \(its dependency graph and step count\), while the capability axes fixwhat reasoning fills that shape\. When a Tier 2 or Tier 3 task draws a supporting pattern from a lower tier, that supporting pattern introduces an additional reasoning phase but does not add capability axes; the primary tier continues to govern how many axes the task exercises\. Each sampled axis carries avalidation predicatethat must hold in the materialized task for the axis to count as genuinely exercised rather than nominally labeled\. Exhaustive aggregation, for example, requires that the number of matching records exceed the tool’s page size, so that completeness cannot be achieved without paginating; implicit criteria requires that the governing threshold be absent from the request and present only in a reference resource; negative verification requires that the correct answer be an absence\. These predicates are checked during reconciliation, and a task that fails to satisfy the predicate for a sampled axis is revised or the axis is resampled\.

Examples:

Quantitative computation:“What’s the completion rate for this cohort?” The agent must compute COUNT\(completed\) / COUNT\(\*\)×\\times100, not simply extract a stored value\.

Temporal reasoning:“How are we looking compared to last cycle?” The agent must scope to two distinct time windows, compute the metric for each, and compare\.

Implicit criteria:“Are we compliant?” The agent must discover the 85% threshold from a policy document in the environment before it can evaluate the question\.

Negative verification:“Is there a backup policy for this service?” The correct answer is “no” and the agent must report absence rather than hallucinate a policy\.

Cross\-source reconciliation:“What’s the headcount?” The HR report says 102 and the project tracker says 105\. The agent must determine which is authoritative and explain the discrepancy\.

When failure analysis identifies a recurring capability gap \(Section[3\.6](https://arxiv.org/html/2609.22592#S3.SS6)\), it surfaces a new axis\. The temporal\-reasoning gym of Section[4\.2](https://arxiv.org/html/2609.22592#S4.SS2)was generated after failure analysis surfaced this axis as a dominant failure mode in evaluation trajectories\. In other domains \(e\.g\., customer support with web access\), different axes emerge naturally from the available grounding sources\.

### A\.3Obfuscation Gradient

The obfuscation gradient \(Section[3\.3](https://arxiv.org/html/2609.22592#S3.SS3)\) controls how much task\-relevant information is withheld from the question and must instead be discovered through environment interaction\. It is configured as a continuous value from 0\.0 to 1\.0\.

Example:

This is the same underlying task about attrition rates for an accreditation response\.

0\.2 \(Moderate\):“How are we looking on the attrition numbers for the Fall 2025 ADN cohort ahead of the ACEN follow\-up?” Omitted: compliance threshold \(15%, from accreditation policy doc\), data source \(student retention tracker\), computation \(withdrawn / enrolled×\\times100\)\.

0\.6 \(Vague\):“How are we looking on the attrition numbers for the accreditation response?” Additionally omitted: specific cohort \(Fall 2025 ADN\), accreditor \(ACEN\), event type \(follow\-up vs\. initial\)\.

0\.9 \(Very vague\):“Are we ready for the ACEN follow\-up?” Additionally omitted: the metric itself \(attrition\), the domain \(student retention\)\. Agent must discover what “ready” means in this context\.

At all gradient levels, the request must not contain tool names, column or table names, specific thresholds or numeric criteria, computation formulas, or hints about which data sources to consult\. Every value removed from the request is recorded with the literal value, its type, and the environment entity where the agent can discover it\. This record is used during validation via an explicit test to confirm that all omitted values remain discoverable in the materialized environment\.

### A\.4Distractor Composition

Distractors are controlled through a budget that specifies the number and type of adversarial conditions instantiated in the environment\. The budget is configured per\-task and drawn from three channels\.

- •Persona distractors:Permission boundaries, privacy obligations, or role\-based access restrictions that the agent must respect\.
- •Content distractors:Near\-matching records, stale values, conflicting versions, or irrelevant entities that resemble task\-relevant data\.
- •Tool distractors:Pagination, truncated responses, transient failures, rate limits, or parameter restrictions in tool behavior\.

Higher budgets increase the number of misleading branches the agent must navigate without invalidating the correct solution path\.

Examples:

Persona distractor:The agent has access to a shared drive but cannot read files owned by HR due to role\-based permissions\. A relevant\-looking HR document exists but returns an access\-denied error\.

Content distractor:A spreadsheet contains a “Draft” column with last month’s figures alongside the current “Final” column\. The agent must use the correct version\.

Tool distractor:A search query returns 150 results but the tool returns only 50 per page\. The agent must paginate to collect the full set rather than stopping at the first page\.

### A\.5Experimental Configurations

Table[7](https://arxiv.org/html/2609.22592#A1.T7)reports the generation parameter settings for configurations C1 and C2 used in the productivity experiments\. Both use the same 7\-tool interface and domain seed\.

Table 7:Generation parameter settings for configurations C1 and C2\. Both use the same 7\-tool interface and domain seed\. Capability axis weights are held at defaults across both configurations\. The configurable ranges within individual task topology patterns are also constant across C1 and C2\. Only topology tier weights, obfuscation, and distractor budgets differ\.

## Appendix BCertification Rubric Dimensions

The certification rubric comprises seven dimensions scored on a 0–100 scale by an LLM judge blind to the generation process\. Dimensions are evaluated at the task or gym level as appropriate\.

- •Scope compliance:Tasks stay within the defined domain and tool capabilities\.
- •Groundedness:Entities, references, and domain details are anchored in realistic sources\.
- •Scalability:The generation process produces diverse tasks without manual intervention per\-task\.
- •Task realism:Tasks reflect plausible user objectives in the target domain\.
- •Environment realism:Materialized resources, permissions, and behaviors are consistent with a real platform\.
- •Diversity:Tasks span varied scenarios, personas, reasoning types, and difficulty levels\.
- •Interface realism:Tool behavior matches the conventions of the target platform’s API\.

## Appendix CVerifier Implementation

The four verifier signals \(Section[3\.5](https://arxiv.org/html/2609.22592#S3.SS5)\) are realized by a combination of programmatic checks and a staged judge pipeline\. Deterministic outcomes \(expected values, classifications, selected entity sets, and state changes\) and distractor\-compliance conditions are scored by rule\. Open\-ended outcome correctness and process quality are scored by a three\-stage pipeline applied per trajectory\.

Aprocess judgeevaluates the approach independently of the final answer, scoring task understanding, evidence gathering, alignment of tool use with the task’s information dependencies, reasoning, and failure recovery, and separately flagging indicators of environment failure so that infrastructure faults are not charged to the agent\. Anoutcome judgecompares the final response against the computed ground truth component by component, admitting semantic equivalence \(numeric tolerance, set membership where order is immaterial\) rather than requiring exact string match\. Ameta\-judgethen synthesizes the two into a scalar reward in\[0,1\]\[0,1\]and a root\-cause classification \(pass, environment failure, ambiguous answer, or agent error\), following a fixed policy in which strong process with correct outcome scores highest, correct outcome with weak process is discounted, and correct process with incorrect outcome receives partial credit\. Decoupling the judges limits the context each must reason over and separates "was the answer right" from "was the method sound," which the reward policy then combines explicitly\.

## Appendix DTask Sample

The following is a complete generated task from the productivity gym with education as the seed at obfuscation gradient 0\.7\.

#### Scenario

K\-12 Special Education IEP & FERPA Compliance Workspace:A district\-level triennial evaluation tracking platform used by case managers, school psychologists, and compliance coordinators to monitor IDEA\-mandated re\-evaluation timelines across 14 schools \(7 elementary, 4 middle, 3 high school\), approximately 2,400 students with IEPs, and 45 active case managers in a suburban Illinois district\.

#### Persona

Marcus Bell, School Psychologist, age 29, Aurora IL, master’s degree\. Early\-career NASP\-certified school psychologist rotating across four elementary buildings conducting psychoeducational evaluations and contributing to IEP eligibility determinations\. Methodical, data\-driven, frequently navigates shared compliance folders\. Motivational frames: calibration, discrepancy, delegation\.

#### Blueprint

Reasoning tierTier 2 \(Multi\-constraint\) \+ Tier 1 \(Direct Retrieval\)Capability axesImplicit criteria, ranking/comparisonObfuscation gradient0\.7Motivational frameDiscrepancy
Operational context:Marcus Bell is preparing for Friday’s quarterly calibration meeting across the four elementary buildings he serves\. Earlier in the week, the Senior Case Manager forwarded him a district “triennial evaluation readiness” summary, but the building\-level numbers Marcus has been tracking in his own caseload do not agree with what he saw in her message\. Before the meeting he wants to identify which elementary buildings in the district are underperforming on triennial evaluation completion relative to the district’s internal target, using only the official compliance\-rate sheet and the accompanying policy guidance that defines what “on\-track” actually means \(the threshold is not printed on the summary sheet itself; it lives in the written policy that the Compliance Coordinator circulates\)\. He specifically cares about elementary buildings, completed\-on\-time triennials only, and the current evaluation cycle; he does not want last year’s closed\-out numbers polluting the comparison, and he wants to be able to tell the team which buildings fall short and by how much\.

Question \(as posed to agent\):“Something in Nadia’s triennial readiness summary isn’t matching what I’ve been tracking; can you tell me which of our elementary buildings are actually falling short on the current cycle and by how much, using the district compliance rate sheet and whatever the policy says counts as on\-track?”

#### Distractors

#### Environment

Sample rows \(building\_compliance\_rate\):

Sample rows \(compliance\_policy\_doc\):

#### Verifier and Ground Truth

Ground truth\(computed by executing the solution path against the materialized environment\):

on\_track\_threshold85\.0% \(from POL\-TE\-001, policy\_area=triennial\_evaluation, is\_current=1\)
Underperforming elementary buildings \(ranked by shortfall\):

Verifier:

Outcome correctness:

- •Identifies the correct threshold as 85\.0% \(from POL\-TE\-001, not the superseded 90\.0% or draft 80\.0%\)
- •Reports all 5 underperforming elementary buildings in correct rank order by shortfall
- •Reports shortfall values within±\\pm0\.5pp of ground truth

Distractor resilience:

- •Does not use the mirror summary sheet’s conflicting percentages \(e\.g\., 88\.0% for Greenbrier\)
- •Does not include prior\-cycle rows \(cycle\_label=“Prior”\)
- •Uses the current policy threshold \(85\.0%\), not the superseded one \(90\.0%\) or the draft \(80\.0%\)
- •Does not rely on the narrative document or briefing deck for quantitative data

Process quality:

- •Retrieves the threshold from the policy document rather than assuming a value
- •Filters to elementary buildings and current cycle simultaneously
- •Ranks results by shortfall magnitude \(descending\)

The agent must discover that “on\-track” means the 85\.0% threshold from the compliance policy \(navigating a superseded 90\.0% clause and a draft 80\.0% in the briefing deck\), filter to current\-cycle elementary buildings only \(ignoring prior\-cycle rows and a mirror sheet with conflicting values\), and rank the 5 underperforming buildings by shortfall\.

## Appendix ECloud\-API Substrate

The blueprint\-first construction generalizes beyond relational stores to stateful, side\-effecting substrates\. To demonstrate, we instantiate an operational cloud domain in which the environment is a live cloud\-API emulator and each tool wraps a real service call rather than a database query\.

A single declarative state definition plays the role the database plays elsewhere\. Each resource records both the properties needed to provision it and its intended runtime state, so the same artifact is materialized into an executable environment and read to compute ground truth\. Cross\-resource dependencies are expressed as references and resolved by topological ordering, so that resources are created in dependency order and identifiers are substituted once known\. Runtime conditions that a creation call cannot express \(encryption settings, alarm states, key\-rotation status, seeded metric history\) are applied in a second pass\. Adversarial conditions carry over directly: the emulated tools exhibit pagination, transient throttling, and eventual consistency as first\-class behaviors, and content distractors take the form of near\-matching resources, stale configuration, and conflicting cross\-service signals\.

Because ground truth is derived from the state definition rather than from a live query, verification does not require the environment to be running; an optional executable pass re\-confirms the derived answer against the materialized environment when it is available\. Empirical evaluation of difficulty and discrimination on this substrate is future work\.

##### Task sample:

The following is a complete generated task from this cloud\-operations domain\. Unlike the relational task sample of Appendix[D](https://arxiv.org/html/2609.22592#A4), the environment is a set of provisioned cloud resources\. The task is at Tier 1, gradient 0\.56, and is a deliberate red herring: the correct conclusion is that no resource violates the policy, exercising negative verification\.

### Scenario and Persona

Meridian settlement platform \(financial services\):A payments company operating a settlement batch\-processing platform on managed cloud infrastructure, preparing evidence for a SOC2 audit\. The persona is an infrastructure engineer who must verify the backup\-retention posture of the platform’s databases against an internally defined policy\.

### Blueprint

Reasoning tierTier 1 \(Direct Retrieval\) with quantitative aggregationCapability axesImplicit criteria, quantitative computation, negative verificationServicesRDS, SSM Parameter Store, CloudWatchMotivational frameCompliance audit
Question \(as posed to agent\):*“I’m prepping evidence for our SOC2 audit next week and need to verify the backup retention posture of the RDS instances supporting our settlement batch processing platform\. Our infrastructure policy defines minimum retention requirements in Parameter Store\. Can you confirm whether all settlement database instances meet the policy threshold, and compute the average backup retention across the fleet?”*

The retention threshold is not stated in the request; the agent must retrieve it from a Parameter Store entry \(implicit criteria\), read the retention period of each database instance, compare against the threshold, and compute the fleet average \(quantitative computation\)\.

### State definition \(excerpt\)

The environment is specified declaratively\. Each resource records thepropertiesused to provision it, an annotation\_ground\_truthcapturing its intended runtime state, and optionalpost\_createsteps applying state that a creation call cannot express\. Cross\-resource references \(absent in this compact example\) are writtenref:<id\>and resolved by topological ordering during provisioning\.

\{

"type":"ssm:parameter",

"id":"ssm\_param\_backup\_policy",

"properties":\{

"Name":"/meridian/platform/rds\-backup\-retention\-policy",

"Value":"\{\\"minimum\_retention\_days\\":7,\\"recommended\_retention\_days\\":14,\.\.\.\}",

"Type":"String"

\},

"\_ground\_truth":\{"minimum\_retention\_days":7,"recommended\_retention\_days":14\}

\},

\{

"type":"rds:db\_instance",

"id":"rds\_instance\_batch\_ledger",

"properties":\{

"DBInstanceIdentifier":"batch\-ledger\-db",

"Engine":"postgres","DBInstanceClass":"db\.r6g\.xlarge",

"BackupRetentionPeriod":21,"MultiAZ":true,"StorageEncrypted":true

\},

"post\_create":\[

\{"service":"rds","method":"add\_tags\_to\_resource",

"params":\{"ResourceName":"$arn",

"Tags":\[\{"Key":"environment","Value":"production"\}\]\}\}

\],

"\_ground\_truth":\{"BackupRetentionPeriod":21,"DBInstanceStatus":"available"\}

\}

The full environment comprises four RDS instances \(retention 14, 14, 21, 7\), the policy parameter, and two CloudWatch alarms\. The alarms are answer\-irrelevant and held in theOKstate, corroborating a healthy posture without contributing to the decision\.

### Distractors

### Ground truth and verifier

Ground truth is derived from the state definition without a running environment, by reading each instance’sBackupRetentionPeriodfrom its\_ground\_truthannotation and applying the policy threshold retrieved from the parameter’s value\.

Verifier:

*Outcome correctness:*

- •Reports the fleet average as 14 days
- •Concludes all four instances meet the 7\-day minimum
- •Reports no violations

*Distractor resilience:*

- •Uses each instance’s configuredBackupRetentionPeriod, not thebackup\-retention\-targettag onsettlement\-archive
- •Does not conclude compliance solely from theOKalarm states

*Process quality:*

- •Retrieves the threshold from Parameter Store rather than assuming a value
- •Reads retention for all four instances before aggregating

When available, an executable verification pass re\-confirms the derived answer against the provisioned environment; because ground truth is computed from the state definition, the derived and executed answers agree by construction\.

## Appendix FGeneration Cost and Timing

The pipeline is model\-agnostic\. We report costs at mid\-tier model pricing \($2\-3 per million input tokens, $10\-15 per million output tokens\)\. Expert authoring costs are estimated at $500–1000 per task including environment construction, verifier design, and quality assurance\. The human validation reported in Section[4\.1](https://arxiv.org/html/2609.22592#S4.SS1)was a one\-time study and is not included in per\-task generation cost\. AutoGym achieves two orders of magnitude cost reduction and roughly 100×\\timeswall\-clock speedup\.

Table 8:Per\-task costTable 9:Gym\-level cost \(50 tasks, 20 parallelism\)
## Appendix GInstruction\-Following Gym

Failure analysis identified instruction\-following issues such as format compliance, constraint adherence, and verbosity as recurring failure modes\. The curriculum loop generated 600 targeted tasks from these signals\. RL training on the 8B checkpoint using the generated gym shows consistent improvement over 500 steps \(Figure[2](https://arxiv.org/html/2609.22592#A7.F2)\), providing evidence that the generated tasks produce a strong and non\-degenerate training signal\.

![Refer to caption](https://arxiv.org/html/2609.22592v1/rl.png)Figure 2:Training reward over 500 RL steps on the instruction\-following gym\. The steady improvement confirms the generated tasks provide learnable signal\.
## Appendix HPrompt Templates

This appendix collects the prompt templates used in AutoGym’s generation and evaluation pipelines\. The voice fitting and reconciliation templates govern task generation\. The verifier templates instantiate the staged judge pipeline described in Appendix[C](https://arxiv.org/html/2609.22592#A3)\. Fields in double braces are populated per task or trajectory at evaluation time\.

Coreprinciple:DifficultycomesfromtheGAPbetweenthevaguequestionandthedetailedoperational\_context\.ThequestiongivestheMINIMUMinformationneededtoidentifywhatthepersonawants\-\-theagentmustfigureouteverythingelsefromtheenvironment\.

WhatthequestionMUSTNOTcontain:

\-Toolnames\(anynamefromthetoolsschema\)

\-Fieldnames,resourceidentifiers,ordatasourcenamesfromtheenvironmentspecification

\-Specificthresholdsornumericcriteria\(theseliveinoperational\_context\)

\-Computationformulasorstep\-by\-stepinstructions

\-Hintsaboutwhichdatasourcestolookat

Obfuscationrules\(preventleakinganswercriteria\):

\-Ifoperational\_contextspecifiesadate\(e\.g\.,"Q32025"\),thequestionmustuserelative/fuzzyterms\("lastquarter","themostrecentperiod","thiscycle"\)

\-Ifoperational\_contextspecifiesathreshold\(e\.g\.,"below40\\%"\),thequestionshouldaskabout"meetingourtargets"or"compliancestatus"\-\-neverthenumber

\-Ifoperational\_contextnamesaspecificdocument\(e\.g\.,"theStudentProgressionTracker"\),thequestionshouldreferenceitgenerically\("ourenrollmentdata","thestudentrecords"\)

\-Ifoperational\_contextnamesaspecificregulation\(e\.g\.,"ACENStandard4\.7"\),thequestioncanreferencetheregulationbyname\(personaswouldknowthis\)butNOTthespecificmetricorthresholdfromit

WhatthequestionSHOULDsoundlike:

Avagueaskfromabusyprofessionaltotheirassistant:

\-TOOSPECIFIC:"CalculatetheattritionratefortheFall2025ADNcohortbydividingwithdrawnstudentsbytotalenrolled,thenclassifyagainstthe40\\%ACENthreshold"

\-RIGHTLEVEL:"Howarewelookingontheattritionnumbersfortheaccreditationresponse?"

\-EVENHARDER:"ArewereadyfortheACENfollow\-up?"

Translatetoolactionsintobusinessintent:

\-BAD:"Use\[tool\_name\]tofindthetracker"\-\>GOOD:"Pullupthecurrentfigures"

\-BAD:"Use\[tool\_name\]toreadthepolicy"\-\>GOOD:"Whatdoesourpolicysayaboutthis?"

\-BAD:"Lookupallrecordswherestatusiswithdrawnandcountthem"\-\>GOOD:"Howmanyhavewelost?"

GradientScore\(0\.0to1\.0\):

Eachtaskisassignedagradient\_scorecontrollingquestionvagueness:

0\.0\-0\.3Moderate:Questionmaynamethegeneraldomainareaandtimescope\.Mayreferenceregulatoryframeworksbyname\.

0\.4\-0\.6Vague:Questionstatesonlythegoal\.Noscopehints,notimereferences,nospecificentities\.

0\.7\-0\.9VeryVague:Questionis1\-2sentencesmax\.Usesonlybusinessintentlanguage\.

1\.0MaximallyVague:QuestioncontainsZEROnounsfromstate\_spaceentitynames\.Agentmustdiscoverdatasourcesentirelyonitsown\.

Calibrationtest:

IfsomeonecansolvethetaskbyreadingONLYthequestion\(withouttheoperational\_contextandanswer\_derivation\),thequestionistoodetailed\.Stripmoreinformation\.Theanswer\_derivation\.formulamustNOTbederivablefromthequestiontextalone\.Itshouldonlybecomeclearafterreadingtheoperational\_context\.

Obfuscationtracking:

Duringvoicefitting,trackeveryspecificvalueremovedfromthequestion:

"obfuscated\_values":\[

\{"value":"85","type":"threshold","context":"attendanceratethresholdfrom504protocol","entity":"compliance\_policies"\},

\{"value":"2025\-07\-01","type":"date","context":"fiscalyeardeadline","entity":"academic\_calendar"\},

\{"value":"Active","type":"enum","context":"planstatusfilter","entity":"enrollment\_config"\},

\{"value":"504frameworkdocument","type":"entity\_name","context":"policydocumentreference","entity":"policy\_documents"\}

\]

Eachentryhas:value\(theliteralremoved\),type\(threshold/date/enum/entity\_name/identifier\),context\(whatitmeantintheoperational\_context\),entity\(thestate\_spaceentitywheretheagentcandiscoverthisvalue\)\.

Atgradient0\.2\(moderate\),fewvaluesareobfuscated\-\>shortlist\.Atgradient1\.0\(maximallyvague\),mostvaluesareobfuscated\-\>longlist\.

Gradientleakcheck\(mandatoryforgradient\>=0\.9\):

Atgradient0\.9\-1\.0,verifythequestioncontainsNONEoftheliteralstringsfromobfuscated\_values\[\]\.value\.Ifanyobfuscatedvalueappearsinthequestiontext\(evenasasynonymorpartialmatch\),rephrasethequestiontoremoveit\.Thisisthefinalcheckbeforewritingthequestion\.

Questionformat:

\-Tier1:1\-2sentences\.Tier2:2\-3sentences\.Tier3:1\-3sentences\(shortandvague,NOTlonganddetailed\)\.

\-DoNOTinclude"Answeras:\.\.\."\-\-theagentmustfigureoutwhattodeliver,notbetoldtheoutputshape\.

DATA\-QUESTIONRECONCILIATION\(postvoicefitting\)

Usestheobfuscated\_valueslistfromvoicefittingtoensuretheenvironmentcontainseverythingtheagentneedstodiscoverviatoolcalls\.Alsosetsdiscoverableflagsonentitiestoavoidmaterializingunnecessaryones\.

Coreinvariant\(gym\-agnostic\):Allvaluesreferencedinthesolutionpathmustbeeitherstatedinthequestionorpresentinadiscoverableentity\.

\-Marksolution\-pathentitiesasrequired:Everyentityreferencedinanswer\_derivation\[\]\.entity\-\>setdiscoverable:true\.Thesecontainthedatatheagentcomputesanswersfrom\.

\-Markrelationshipentitiesasdiscoverable:Foranydiscoverableentity,ifitsrelationshipsdictreferencesasource\_entity,markthatsourceentityasdiscoverable:trueaswell\.Referencedentitiesmustbereachableforrelationshiptraversaltowork\.

\-Markobfuscated\-valueentitiesasdiscoverable:Foreachentryinobfuscated\_values,lookuptheentityfield\.Ifthatentityexistsinstate\_space\-\>setdiscoverable:true\.IfitdoesNOTexist\-\>addittostate\_spacewithdiscoverable:true\.Thenewentityshouldbeappropriatefortheplatform\(apolicydocumentfordocplatforms,aconfigrecordforcloudplatforms,asettingsentityforSaaSplatforms\)andmustcontaintheobfuscatedvalueinatool\-readablefield\.

\-Markremainingentities:Anyentitynotflaggedbytheabovethreerules\-\>setdiscoverable:false\.

Example\(gradient=0\.8\):obfuscated\_valuesincludes\{"value":"85","type":"threshold",\.\.\.\}\-\>areferenceentitymustexistwheretheagentcandiscover"85\\%"viatoolcalls\-\>discoverable:true\.Atgradient=0\.2,thequestionalreadysays"below85\\%"sothereferenceentitystaysdiscoverable:false\.

\-\-\-\-

ValidateTask\(afterreconciliation\):

\-Derivation:everyitemhaskeys\{component,entity,formula\},entityexistsinstate\_space,allfieldsinformulaexistinthatentity’sattributes

\-Verifierconstraints:atleast1programmaticconstraintexists\.Programmatictypemustbeoneof:numeric\_match,set\_match,ranking\_match,contains,format,boolean\_match,count\_match\.Semantictypemustbeoneof:completeness,tone,no\_hallucination,explanation\_quality\.

\-Dataspec:answer\_relevant\_records<=0\.6\*total\_records\(\>=40\\%distractors\),everyentityreachablebyatoolinaccessed\_by\_tools

\-Reconciliation:everyobfuscated\_values\[\]\.entitymustexistinstate\_spacewithdiscoverable:true\.Everyanswer\_derivation\[\]\.entitymustbediscoverable:true\.Everysource\_entityreferencedbyadiscoverableentitymustalsobediscoverable:true\.Noentityoutsidethesethreesetsshouldbediscoverable:true\.

\-Voicefitting:questioncontainsnotoolnames,fieldnames,datasourcenames,specificthresholds,orformulafragments;questionisshorterthanoperational\_context\.Atgradient\>=0\.9,questioncontainsNONEoftheobfuscated\_values\[\]\.valuestrings\.

Ifanycheckfails,fixandrevalidate\.

YouareastrictevaluationjudgeassessingwhetheranAIagentfollowedareasonable,validapproachtosolvingatask\.

Evaluateprocessquality,NOTanswercorrectness\.ButyouMUSTdetectseriousprocessflaws:incorrectcalculations,invalidtransformations,unsupportedassumptions,contradictoryreasoning,misuseofevidence\.

YouMUSTalsodetectenvironmentfailures\-\-problemscausedbybrokeninfrastructureratherthanagentmistakes\(servererrors,missingdata,emptysearches,inaccessibleentities\)\.

\-\-\-\-

Inputs

UserTask:

\{\{user\_task\}\}

ToolDAG\(expectedworkflow\):

\{\{expected\_tools\}\}

AgentExecutionTrajectory:

\{\{trajectory\}\}

\-\-\-\-

Rubric\(scoreeach0\-2\):

1\.TaskUnderstanding:Didtheagentcorrectlyinterpretwhatwasasked?

2\.EvidenceGathering:Diditaccesssufficient,relevantdata?

3\.ToolUsage/SubgoalAlignment:Appropriatetools?ToolDAGsubgoalscovered?

4\.ReasoningCoherence:Logicallyvalid?Nocomputation/transformationerrors?

5\.WorkflowPlausibility:Reliableapproachthatwouldproducecorrectresults?

6\.FailureSensitivity:Anywarningsigns\(errorsignored,contradictions,wrongentities\)?

EnvironmentFailureDetection:

Iftheagentfailedduetoservererrors,emptysearches,missingentities,brokendata,orpermissionover\-filtering\-\-theseareNOTtheagent’sfault\.Notethemin‘environment\_failure\_indicators‘\.DoNOTpenalizeprocessscoresforinfrabugs\.

\-\-\-\-

ReturnvalidJSONonly:

\{

"task\_understanding\_score":0,

"evidence\_gathering\_score":0,

"tool\_usage\_alignment\_score":0,

"reasoning\_score":0,

"workflow\_plausibility\_score":0,

"failure\_sensitivity\_score":0,

"expected\_tools\_used":\[\],

"expected\_tools\_missing":\[\],

"subgoals\_addressed":\[\],

"subgoals\_missed":\[\],

"approach\_reasonable":true,

"process\_quality\_label":"Strong\|Acceptable\|Weak",

"environment\_failure\_indicators":\[\],

"critical\_process\_failures":\[\],

"suspected\_transformation\_errors":\[\],

"confidence":0\.0,

"reasoning":""

\}

Youareastrictoutcomeevaluatordeterminingwhethertheagent’sfinalanswermatchesthegroundtruth\.

Performa\*\*semanticcomparison\*\*oftheagent’sansweragainsteachgroundtruthcomponent\.Foreachcomponent,determinewhethertheagent’sanswerconveysthesamefactualcontentasthecomputed‘result‘\.

\-\-\-\-

Inputs

UserTask:

\{\{user\_task\}\}

AgentExecutionTrajectory:

\{\{trajectory\}\}

GroundTruth:

\{\{ground\_truth\}\}

Groundtruthcontains‘correctness\.components\[\]‘,eachwith:

\-‘component‘:nameoftheanswerpiece

\-‘sql‘:thequerythatwasexecuted

\-‘result‘:thecomputedcorrectvalue

Foreachcomponent,semanticallycheck:doestheagent’sfinalanswercontaininformationconsistentwiththisresult?Anumericresultlike‘67\.4‘matchesiftheagentsays"67\.4\\%","approximately67\\%","67\.4percent",etc\.Alistresultmatchesiftheagentmentionsalltheitems\.Anarrativeresultmatchesiftheagentconveysthesamekeyfacts\.

If‘solvability\.solvable‘isfalse,thecorrectanswerisrecognizingthepermissionrestriction\.

\-\-\-\-

Rubric\(scoreeach0\-2\):

1\.AnswerGrounding:Isthefinalanswersupportedbytrajectoryevidence?

2\.EvidenceConsistency:Doestheanswermatchtooloutputsandretrieveddata?

3\.TaskCompletion:Doestheanswerfullyaddressthequestion?

4\.Derived\-ValueCorrectness:Arecomputations/conversionsvalid?

5\.ContradictionHandling:Wereconflictingsignalshandledproperly?

Foreachcomponent:extracttherelevantpartoftheagent’sanswer,comparesemanticallyagainsttheGT‘result‘\.

Iftheagent’sanswerdiffersfromGTbutusesadefensibleinterpretationofanambiguousquestion,markas‘ambiguous\_acceptable‘\.

\-\-\-\-

ReturnvalidJSONonly:

\{

"component\_results":\[

\{"component":"\.\.\.","gt\_result":"\.\.\.","agent\_value":"\.\.\.","result":"match\|mismatch\|ambiguous\_acceptable\|not\_attempted"\}

\],

"all\_components\_match":true\|false,

"match\_count":0,

"mismatch\_count":0,

"answer\_grounding\_score":0,

"derived\_value\_correctness\_score":0,

"contradiction\_handling\_score":0,

"final\_verdict":"Correct\|PartiallyCorrect\|Ambiguous\|Incorrect\|NotAttempted",

"defensible\_alternative\_interpretation":null,

"material\_errors\_detected":\[\],

"confidence":0\.0,

"reasoning":""

\}

Youareameta\-evaluatorcombiningprocessandoutcomejudgeresultsintoafinalrewardscoreANDclassification\.

\-\-\-\-

Inputs

UserTask:

\{\{user\_task\}\}

ProcessJudgeOutput:

\{\{process\_judge\_outputs\}\}

OutcomeJudgeOutput:

\{\{outcome\_judge\_outputs\}\}

\-\-\-\-

RewardGuidance:

1\.HighProcess\+HighOutcome\(correctanswer,goodreasoning\)\-\>0\.9\-1\.0

2\.HighProcess\+LowOutcome\(goodreasoning,wronganswer\)\-\>0\.3\-0\.5

3\.LowProcess\+HighOutcome\(correctanswer,weakreasoning\)\-\>0\.5\-0\.7

4\.LowProcess\+LowOutcome\-\>0\.0\-0\.3

ClassificationGuidance:

Basedonprocessandoutcomejudgeevidence,determinetherootcauseofanyfailure:

\-\*\*PASS\*\*\-\-outcomejudgeconfirmssemanticmatchonallGTcomponents,processwasreasonable

\-\*\*ENVIRONMENT\_FAILURE\*\*\-\-processjudgefoundtoolerrors,emptyresults,servercrashes,orunreachabledatathatblockedtheagent\.Theagentcouldnothavesucceededregardlessofreasoningquality\.

\-\*\*AMBIGUOUS\_ANSWER\*\*\-\-outcomejudgefoundadefensiblealternativeinterpretation\.Agent’sanswerisreasonablebutdiffersfromGT\.

\-\*\*SOLVER\_ERROR\*\*\-\-environmentwasfunctional\(processjudgeshowstoolsworked,datawasaccessible\),butagentgotthewronganswerthroughitsownreasoningmistakes

If‘environment\_failure\_indicators‘arepresentintheprocessjudgeoutput,theagentwasblockedbyinfra\-\-rewardshouldreflecttheagent’squalityuptothefailurepoint,notpenalizeforenvbugs\.ClassificationshouldbeENVIRONMENT\_FAILURE\.

\-\-\-\-

ReturnvalidJSONonly:

\{

"average\_process\_score":0\.0,

"average\_outcome\_score":0\.0,

"final\_label":"StrongSuccess\|Success\|PartialSuccess\|Failure\|Uncertain",

"classification":"PASS\|ENVIRONMENT\_FAILURE\|AMBIGUOUS\_ANSWER\|SOLVER\_ERROR",

"reward\_score":0\.0,

"confidence":0\.0,

"key\_success\_factors":\[\],

"key\_failure\_factors":\[\],

"reasoning":""

\}