面向长时程工具使用的自调优基准的演化方法

# Evolving Self-tuning Benchmarks for Long-Horizon Tool Use
Source: [https://arxiv.org/html/2610.27307](https://arxiv.org/html/2610.27307)
Michael DennisAffiliation:Google DeepmindNatasha JaquesMinsuk ChangAffiliation:Stanford UniversityAffiliation:University of WashingtonCorrespondence:[hjnam@stanford\.edu](mailto:[email protected])

###### Abstract

For AI assistants to move beyond simple chat interfaces and reliably assist users with applications like Gmail, Venmo, and Slack, we need to improve their tool\-use capabilities\. However, existing tool use benchmarks are still limited and difficult to scale because they rely on human supervision or are too domain\-specific\. To make data generation scalable, we propose,TURTLE\(ToolUseRelatedTasks withLong\-HorizonEvolution\), an Unsupervised Environment Design \(UED\) approach to evolve existing tasks using a proposer, a solver, and a feedback generator\. The proposer is rewarded for*difficult yet solvable*tasks based on the solver’s empirical performance, and the feedback generator summarizes the solver’s execution traces for the proposer to exploit\.TURTLEproduces evolving, self\-tuning test suites adaptable to different environment requirements, from supporting user simulators and dense document retrieval to interacting across multiple apps\. Empirical results on AppWorld,τ\\tau\-Bench, and AutomationBench using Gemma 4 31B\-it show that the evolved tasks target the desired difficulty range \(e\.g\., a 165% increase in the percentage of hard tasks on AppWorld\), increase task diversity \(in terms of reasoning and simultaneous tool interactions\), and elicit a broader range of failure modes than the original tasks\.

††footnotetext:†Equal advising\.‡Work done during internship at Google DeepMind\.## 1Introduction

Figure 1:Even with the same underlying toolset, task difficulty can change depending on the instruction and the amount of reasoning and multi\-step interaction required\. Rather than building new benchmarks from scratch,TURTLEevolves existing tasks using a task proposer, a solver, and a feedback generator LLM*without human supervision\.*During the evolutionary search, the proposer modifies the parent task, generating increasingly challenging but solvable tasks, while trivial or infeasible tasks are pruned based on their scores\.As Large Language Models \(LLMs\) become more pervasive and integrated into our everyday lives and workflows, evaluating models across a wide range of scenarios is crucial for safe deployment\. Especially for nascent applications like tool\-use, evaluating models across diverse task suites is important for understanding their capabilities and vulnerabilities\. However, existing tool use benchmarks are still limited and difficult to scale\. Many existing benchmarks are domain\-specific\([Merzouki et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib28);[Shepard and Salimans, 2026](https://arxiv.org/html/2610.27307#bib.bib8);[Yang et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib27)\), or they rely on humans in the loop for task design and refinement\([Trivedi et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib41);[Shi et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib7);[Barres et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib6)\)\. Furthermore, many new datasets are built from scratch and show little to no compatibility with the existing harnesses and testing setups of prior benchmarks, creating engineering overhead when incorporating new tests into an existing evaluation pipeline\. Therefore, we need a scalable way to generate new tasks that test the robustness of models without adding to the engineering burden\. Our key insight is that we*do not need to reinvent the wheel*\. Instead, we can reuse components of existing tasks to build new tasks\. Existing tasks give us primitives and steps that we can recombine to produce a vast space of possible new tasks\. By searching this space to target gaps in the solver’s performance, we can still generate new tasks efficiently while ensuring task robustness\.

We proposeTURTLE\(ToolUseRelatedTasks withLong\-HorizonEvolution\), an Unsupervised Environment Design \(UED\) approach\([Dennis et al\., 2021](https://arxiv.org/html/2610.27307#bib.bib33);[Wei et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib31);[Rutherford et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib32)\)based on evolutionary search\([Bäck and Schwefel, 1993](https://arxiv.org/html/2610.27307#bib.bib39);[Mouret and Clune, 2015](https://arxiv.org/html/2610.27307#bib.bib9)\)\.TURTLEconsists of a proposer, a solver, and a feedback generator, each responsible for new task generation, reward evaluation, and task reflection\. Similar to UED and compositional environment design\([Gur et al\., 2022](https://arxiv.org/html/2610.27307#bib.bib26)\), we use a Proposer agent to automatically generate new tasks that target the solver’s current level\. We implement the proposer using an advanced evolutionary search operator, AlphaEvolve\([Novikov et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib42)\)\. This allows our generation pipeline to incorporate highly capable models as the proposer—even closed\-source ones—without requiring access to their weights, since evolutionary search hanldes gradient\-free optimization\. This is important given the large capability gap and varied behaviors observed across models of different sizes and release dates\([Marchand et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib44)\)\. Evolutionary approaches have shown success in environment generation\([Faldor et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib4)\), persona generation\([Paglieri et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib3);[Chopra et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib11)\), strategy discovery\([Wang et al\., 2026a](https://arxiv.org/html/2610.27307#bib.bib34)\), and red\-teaming\([Samvelyan et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib45)\), but they remain under\-explored for tool\-use task generation\.

The proposer receives an initial task and a toolset from open\-source datasets and suggests mutations to the task based on the sovler’s prior execution traces and failure reflections\. The modified task is then attempted by the solverNNindependent times\. To avoid tasks that are either too difficult or too easy, we design rewards based on the solver’s completion rate, targeting a 50% average completion rate, as in*sampling for learnability*\([Rutherford et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib32)\), which maximizes the variance of the solver’s success rate in 2D grid navigation environments\. As a result,TURTLEproduces evolving, self\-tuning benchmarks tailored to the current solver\.

When evaluating the evolved tasks, we focus on three aspects: \(1\) the solver’s empirical performance, \(2\) task diversity, and \(3\) elicited failure modes\. Our empirical results span open\-source benchmarks with different requirements, featuring user simulators\([Barres et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib6);[Shi et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib7)\), documents for knowledge retrieval\([Shi et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib7)\), 47\+ simulated SaaS applications\([Shepard and Salimans, 2026](https://arxiv.org/html/2610.27307#bib.bib8)\), and extensive user databases and product catalogs modeled after real\-world software applications like Amazon and Spotify\([Trivedi et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib41)\)\. Our main results are reported with Gemma 4 31B\-it\([Team et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib15)\)as the solver, but we also include ablations with other models, such as Gemini 3\.6 Flash\([Doshi and Google Gemini Team, 2026](https://arxiv.org/html/2610.27307#bib.bib40)\)and Gemini 3\.1 Flash Lite\([Google Gemini Team, 2026](https://arxiv.org/html/2610.27307#bib.bib38)\), to show thatTURTLEadapts to different solver types\. In summary,TURTLEis a multi\-agent evolutionary data\-generation approach for eliciting long\-horizon tool\-use capabilities\. It:

- •Increases task difficulty when the original tasks are too easy \(e\.g\., a 165% increase in the proportion of hard tasks compared to the original AppWorld tasks for Gemma 4 31B\-it\)
- •Increases the frequency of desirable task features, such as logical filtering and reasoning, by 29–212% on AppWorld, and makes both semantic and functional changes to the task instructions;
- •Can be used to diagnose solver vulnerabilities, broadening the coverage of \(task feature×\\timeselicited error type\) pairs from 49 to 54\.

## 2Related Work

#### Tool\-use benchmarks for LLMs\.

Frontier models still struggle on long\-context benchmarks requiring multi\-step reasoning, such as FactoryBench\([Merzouki et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib28)\), AutomationBench\([Shepard and Salimans, 2026](https://arxiv.org/html/2610.27307#bib.bib8)\), ProgramBench\([Yang et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib27)\), andτ\\tau\-bench \(banking\)\([Shi et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib7)\)\. Despite their usefulness, these benchmarks are difficult to scale because they mainly provide static environments and tasks\. Therefore, once models learn to solve problems in these datasets, the benchmarks cannot be reused to discover new tasks or model failures\.

Methods have been developed to generate new tasks\. For example, Tool\-R0\([Acikgoz et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib30)\)and GenEnv\([Guo et al\., 2025b](https://arxiv.org/html/2610.27307#bib.bib14)\)operate only in text space to generate and verify tasks\. However, LLM\-simulated environments may suffer from hallucinations and inconsistencies\([Wang et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib20)\)\. Alternatively,[Lu et al\. \(2025\)](https://arxiv.org/html/2610.27307#bib.bib23);[Barres et al\. \(2025\)](https://arxiv.org/html/2610.27307#bib.bib6);[Yao et al\. \(2024\)](https://arxiv.org/html/2610.27307#bib.bib21);[Shi et al\. \(2026\)](https://arxiv.org/html/2610.27307#bib.bib7);[Trivedi et al\. \(2024\)](https://arxiv.org/html/2610.27307#bib.bib41);[Esfandiarpoor et al\. \(2025\)](https://arxiv.org/html/2610.27307#bib.bib22)build tool\-interactive sandboxes and database states that more closely resemble real\-world applications\. However, these are challenging to scale, as they require humans to refine tasks or evaluation environments for robustness\. Furthermore, they are often tailored to specific domains \(e\.g\.,τ\\tau\-bench for travel and retail\([Yao et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib21)\), banking\([Shi et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib7)\), and telecom\([Barres et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib6)\)are all separate benchmarks\)\. Other methods rely more heavily on prompting for environment design and task discovery, such as EnvScaler\([Song et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib18)\)and Agent World Model\([Wang et al\., 2026b](https://arxiv.org/html/2610.27307#bib.bib25)\)\. Agent\-World\([Dong et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib17)\)uses prompts to adjust task difficulty with hand\-designed task features, such as the number of reasoning steps and the number of unique tools used per task\. Validating task feasibility remains challenging in automated task generation\. To guarantee solvability, ToolGrad\([Zhou et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib16)\)reverses the task\-generation process by first constructing the ground\-truth tool\-call sequence, then writing a matching natural language instruction for the given sequence\.TURTLEis different from prior approaches in two ways: \(1\) evolutionary: using gradient\-free evolutionary optimization to incorporate advanced models, and \(2\) self\-tuning: minimizing human oversight in task design and adapting to particular solvers\.

Our work shares the motivation of[Cai et al\. \(2025\)](https://arxiv.org/html/2610.27307#bib.bib24);[Fang et al\. \(2025\)](https://arxiv.org/html/2610.27307#bib.bib19), which use existing tool\-use datasets to automate task generation\. However, they rely on state\-dependency graphs and random walks, which can limit the flexibility of task generation, and building graphs become challenging as the number of tools increases and their inter\-dependencies grow complex\.TURTLEcombines the best of both approaches by leveraging existing benchmarks and verified API implementations, while evolving them to adjust their difficulty based on the solver’s performance\.

#### LLM\-powered evolutionary search\.

Evolutionary algorithms are gradient\-free optimization inspired by biological evolution\([Bäck and Schwefel, 1993](https://arxiv.org/html/2610.27307#bib.bib39);[Mouret and Clune, 2015](https://arxiv.org/html/2610.27307#bib.bib9)\)\. They iteratively mutate a population of programs and select candidates based on their fitness scores, while allowing fitter individuals to reproduce more often than less fit ones\. Incorporating large language models into evolutionary search has proven successful in scientific and mathematical optimization\([Novikov et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib42);[Assumpção et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib2)\), red\-teaming\([Yuan et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib12)\), and social applications using conversational assistants, such as generating diverse user personas\([Chopra et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib11);[Paglieri et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib3)\)and strategy discovery\([Wang et al\., 2026a](https://arxiv.org/html/2610.27307#bib.bib34)\)\. AlphaEvolve\([Novikov et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib42)\)and OpenEvolve\([Rudes et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib5)\)are among the popular frameworks for orchestrating evolutionary search using LLMs\.

## 3Method

TURTLEconsists of three key LLM agents: \(1\) aproposerimplemented with AlphaEvolve; \(2\) asolver, which attempts the new task yielding fitness scores; and \(3\) afeedback generator, which identifies the main reason for failure based on the solver’s execution traces and the ground\-truth solution\. We leverage multi\-agent dynamics to propose, evaluate, and refine task mutations according to the solver’s problem\-solving competence\.

After the evolutionary search is run for a fixed number of steps, we add programs with the highest completion rate variance \(including ties\) per parent task from the original dataset to the final dataset𝒟TURTLE\\mathcal\{D\}\_\{\\text\{\{\\includegraphics\[page=\{705\},height=6\.83331pt\]\{all\-twemojis\.pdf\}\}\{\\color\[rgb\]\{0,0\.88,0\}TURTLE\} \}\}\. We then apply a filtering step with an LLM judge \(Gemini 3\.6 Flash\) to remove tasks with trivial solver failures, such as formatting mismatches or paraphrases of the same content, which can be attributed to ambiguous or underspecified instructions \(see judge prompt in Appendix Fig\.[9](https://arxiv.org/html/2610.27307#A1.F9)\)\.

### 3\.1Evolutionary Task Proposer

Task structure\.A task is presented as a Python program, and the proposer is instructed to act as a programmer\. In particular, on AppWorld, we provide minimal instructions to explore how AlphaEvolve discovers task features on its own \(proposer instruction prompts are included in Appendix[A](https://arxiv.org/html/2610.27307#A1)\)\.

Depending on the features of the initial task, the proposer is given different items specifying the task\. For example, in AppWorld, a task is defined by:\(1\) the initial database state, which specifies the user’s account information, friends list, message history, Amazon product catalogs etc\.;\(2\) the original task instruction in natural language\(e\.g\., “*Give me a comma\-separated list of the six most\-played EDM songs from my Spotify playlist\.*"\);\(3\) the ground\-truth solution, which shows the sequence of API calls and their arguments; and\(4\) optionally, evaluation criteria\. If a task changes only part of the database state, the evaluation criteria are set automatically to the oracle final state obtained by executing the ground\-truth solution in a simulated environment, so the proposer does not need to specify them\. If enumerating all state changes is too expensive, the proposer additionally specifies which parts of the database must be inspected, and the solver is evaluated only on the changes made to those parts\.

Task mutation\.During each evolutionary step, the proposer is given a parent task and five additional tasks sampled from the archive, stochastically chosen between the best global and local tasks \(highest\-scoring per metric\)\. Each task is represented by its instruction text, a corresponding score, and a summary by the feedback generator\. The proposer uses this information to propose modifications to the parent task that can maximize the fitness function\. Unlike prior work that uses RL post\-training to update the proposer\([Acikgoz et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib30);[Guo et al\., 2025b](https://arxiv.org/html/2610.27307#bib.bib14)\), we frame task generation as a search problem seeded with a parent task and use AlphaEvolve to discover variants of the original task\.

1deftask\(\):

2return\{

3db\_changes:

4full\_description:

5instruction:

6ground\_truth\_calls:

7\}

Listing 1:Appworld task as a program\.Task verification\.Before sending a new task to the solver, we execute the proposed ground\-truth solution in the simulated environment to verify there are no system errors\. This step filters out tasks with inconsistent state dependencies \(e\.g\., asking to send a message to a friend who is not in the contact list\)\. If the proposer specifies evaluation criteria, we verify that the ground\-truth solution achieves a perfect score under those criteria\.

### 3\.2Solver \(Task Difficulty Evaluator\)

A new task is evaluated based on the solver’s empirical performance\. In particular, we reward tasks that are solvable but not consistently solved perfectly\. Each task is attemptedNNtimes, and an attempt scores 1 if the solver fully completes the task; otherwise 0 \(partial credits are allowed in AutomationBench\([Shepard and Salimans, 2026](https://arxiv.org/html/2610.27307#bib.bib8)\)to avoid reward sparsity on harder domains\)\. To avoid penalizing trivial string mismatches, we additionally prompt the LLM judge to evaluate semantic state matching using the original task instruction and the observed versus expected final states\.

How to avoid reward hacking\.Some works on multi\-agent task generation leverage zero\-sum adversarial dynamics, where the proposer attempts to minimize the solver’s performance, in order to ensure robustness\([Liu et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib37)\)\. However, in the case of tool\-use task generation this objective is badly posed\. Adversarial dynamic between the proposer and the solver can lead to reward hacking, since the proposer may be incentivized to generating ambiguous or partially solvable instructions\. Therefore, to ensure the tasks are feasible, we check that they are solved perfectly in at least one of the solver attempts\.

Fitness function\.Our main objective is to generate long\-horizon tasks that can reveal different model failures\. To this end, we construct a multi\-objective reward function based on the solver’s completion rate\. We defineccas the solver’s average completion rate acrossNNindependent attempts, andLLas the average solver’s trajectory length\. We propose the following six metrics:

- •Target difficulty reward:exp⁡\{−4​ln⁡\(10\)​\(c−0\.5\)2\}\\exp\\\{\-4\\ln\(10\)\(c\-0\.5\)^\{2\}\\\}how closeccis to the target value of 0\.5; a bell\-curve with a maximum of 1 atc=0\.5c=0\.5and a minimum of 0\.1 at the endpoints;
- •Sampling for learnability\([Rutherford et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib32)\):c⁡\(1−c\)c\(1\-c\);
- •Trajectory length bonus×\\timesrelaxed difficulty threshold:max\{1,L\}∗I\{0\.3≤c≤0\.7\}\\max\\\{1,L\\\}\*I\\\{0\.3\\leq c\\leq 0\.7\\\};
- •Trajectory length bonus×\\timesstrict difficulty threshold:max\{1,L\}∗I\{0\.4≤c≤0\.6\}\\max\\\{1,L\\\}\*I\\\{0\.4\\leq c\\leq 0\.6\\\};
- •Trajectory length bonus×\\timessampling for learnability:max⁡\{1,L\}∗c⁡\(1−c\)\\max\\\{1,L\\\}\*c\(1\-c\);
- •Trajectory length bonus×\\timestarget difficulty:max⁡\{1,L\}∗exp⁡\{−4​ln⁡\(10\)​\(c−0\.5\)2\}\\max\\\{1,L\\\}\*\\exp\\\{\-4\\ln\(10\)\(c\-0\.5\)^\{2\}\\\}\.

To encourage more complex tasks, we reward the proposer for generating longer tasks that the solver is still able to complete\. To prevent the proposer from trivially extending the ground\-truth task length with unnecessary tool calls that do nothing \(a reward hacking behavior we initially observed\), we base these rewards on the length of the solver’s successful trajectory\.

During the evolutionary search, the proposer stochastically chooses between global optimality \(the highest\-scoring task on average across six metrics\) and local optimality \(the highest\-scoring task for one of the six metrics\) to balance exploration and exploitation\. When suggesting modifications, the proposer reviews the parent task, along with five sampled in\-context examples, and their corresponding textual and numeric performance summaries, and is instructed to optimize the average fitness score based on the individual score report\.

### 3\.3Feedback Generator

One key advantage of LLM\-powered evolutionary search is that the task proposer can use natural\-language feedback, in addition to fitness scores, to guide its search for better performing mutations\. To leverage this strength, we incorporate a feedback generator LLM into the evolutionary search\. For tasks with unsuccessful solver attempts, the feedback generator outputs a short explanation of what went wrong by comparing the solver’s outcome against the ground\-truth solution and the provided task instruction\. This feedback is stored in the task archive alongside the solver’s execution traces and provided to the proposer during mutation\.

## 4Experiment

TURTLEis applied to four different benchmarks: AppWorld\([Trivedi et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib41)\),τ\\tau\-telecom\([Barres et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib6)\),τ\\tau\-banking\([Shi et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib7)\), and AutomationBench\([Shepard and Salimans, 2026](https://arxiv.org/html/2610.27307#bib.bib8)\)\. More details of each environment in Appendix[B](https://arxiv.org/html/2610.27307#A2)\.

AppWorldis built on 4,571 executable APIs across widely used, everyday software applications\. Each task is associated with a particular user and their personal data \(e\.g\., login credentials, transaction history, and product search history\), and the agent completes tasks as the user’s personal assistant\.

τ\\tau\-bench \(banking & telecom\)requires the model to operate as a customer service agent in a dual\-control, multi\-turn dialogue setting with a \(LLM\) simulated user\. Certain tool calls can be executed only by the user, so the agent must guide the user through the process step by step\.τ\\tau\-banking is designed to test the agent’s document retrieval and long\-context reasoning \(consisting of 698 documents spanning 21 product categories and financial topics\)\. We test the solver with two of the retrieval conditions presented in[Shi et al\. \(2026\)](https://arxiv.org/html/2610.27307#bib.bib7): BM25 and golden retrieval\.

AutomationBenchuses 47 simulated SaaS applications to model realistic workflows involving tools such as Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira, and HubSpot\. The agent needs to navigate through business objectives and guardrails to solve tasks spanning domains such as Finance, HR, Marketing, Operations, Sales, and Customer Support\.

Evolutionary search is implemented with AlphaEvolve[Novikov et al\. \(2025\)](https://arxiv.org/html/2610.27307#bib.bib42)\. Tasks are evolved for 1,000 steps for AppWorld and 500 steps for the other datasets, and every new task is attempted 10 times\.TURTLEis equally applicable to both closed\- and open\-weight models\.

Main empirical results are reported using Gemma 4 31B\-it\([Team et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib15)\)as both the solver and the feedback generator\. We also conduct ablations with two proprietary models, Gemini 3\.6 Flash\([Doshi and Google Gemini Team, 2026](https://arxiv.org/html/2610.27307#bib.bib40)\)and Gemini 3\.1 Flash Lite\([Google Gemini Team, 2026](https://arxiv.org/html/2610.27307#bib.bib38)\)\.

## 5Results

### 5\.1TURTLEadjusts difficulty to solver

Data scaling can be done trivially if tasks are padded with unnecessary or repeated tool calls\. To show the usefulness of evolving, self\-tuning benchmarks, we compare the difficulty of the original \(n=106n=106\) and new tasks \(n=735n=735\) evolved from AppWorld\. We assess difficulty in two ways: \(1\) the average completion rate; and \(2\) the LLM\-judge evaluation using a task difficulty rubric from[Trivedi et al\. \(2024\)](https://arxiv.org/html/2610.27307#bib.bib41)\. In their work, humans rate tasks on a scale of 1 \(easy\) to 3 \(hard\) based on the amount of interaction and reasoning required\. We adapt their prompt to enable a LLM\-judge using Gemini 3\.6 Flash \(prompt in Appendix Fig\.[12](https://arxiv.org/html/2610.27307#A2.F12)\)\.

Figure 2:Distribution of average completion rates on the original AppWorld \(orange\) vsTURTLE\(blue\) tasks\.Figure 3:Distribution of LLM\-evaluated difficulty levels on the original AppWorld vsTURTLEtasks\.Solver’s empirical performance\.Fig\.[2](https://arxiv.org/html/2610.27307#S5.F2)shows that the initial AppWorld task difficulty distribution, as measured by completion rate, is largely bimodal, with 69% of tasks achieving perfect completion and 12% always failing\. AfterTURTLE, nearly all tasks lie in the 40–60% average completion rate range\. Ablations with different solver models and initial task datasets are reported in Appendix Figs\.[16](https://arxiv.org/html/2610.27307#A2.F16)and[17](https://arxiv.org/html/2610.27307#A2.F17)††Inτ\\tau\-telecom, we obtain 299 tasks from 84 parent tasks; inτ\\tau\-banking \(BM25\), we obtain 82 from 17; inτ\\tau\-banking \(golden retrieval\), we obtain 178 from 24; and on AutomationBench, we 23 from 16\. Gemini 3\.1 Flash Lite yields 141 tasks from 64 seeds in AppWorld, while Gemini 3\.6 Flash yields 22 from 9\.\. Across different setups,TURTLEconsistently generates tasks that largely fall into the 40\-60% average completion rate, regardless of the original task difficulty\. This shows that the self\-evolving benchmarks target the solver’s learning frontier, i\.e\., identifying solvable problems that the solver sometimes trips over\.

LLM\-judge evaluation\.The original AppWorld tasks are often too easy for the Gemma 4 31B\-it solver, and the LLM\-judge evaluations in Fig\.[3](https://arxiv.org/html/2610.27307#S5.F3)further confirm that 80% of the original tasks are considered either easy or medium\. In contrast,TURTLE\-generated tasks are evaluated as 44% medium and 53% hard, which is a 165% increase in the proportion of hard tasks compared to the original dataset \(i\.e\., 20%\)\.

### 5\.2Emergent features from self\-tuning benchmarks

When evolving tasks in AppWorld, we give minimal instruction to the task proposer on how to mutate tasks\. The only signals the proposer receives are the fitness scores based on the solver’s performance and summaries by the feedback generator\. Despite minimal guidance,TURTLEdiscovers new task features that can make the tasks harder\.

Hand\-picked examples of evolved task features\# Applying constraints & filtering logic
Original task:*Download all the songs from my Spotify album library that I have liked\.*
New task:*Download all the songs from my Spotify album library that I have liked and were released in 2022 or later\.*\# Introducing multi\-step tool calls
Original task:*Download all the songs from my Spotify album library that I have liked\.*
New task:*Download all the songs from my Spotify album library that I have liked\. For each of these songs, create a note in my Simple Note account with the song’s title as the title and the string ’Genre: , Duration: s’ as the content\. Tell me the total number of songs downloaded and the total number of notes created\.*\# Hiding variables behind other tool calls
Original task:*The last Venmo payment request I sent to Cory was an accident and they approved it\. Send them the money back\.*
New task:*Find my roommate’s email and refund them for the ’movies’ payment they made\. Use ’Accidental Request Refund’ as the description\.*Figure 4:Features discovered duringTURTLE\.Figure 5:Frequency of task features in the original AppWorld \(orange\) vsTURTLE\(blue\) datasets\.We build a taxonomy of task features: \(1\) specification, \(2\) reasoning, \(3\) distractors, \(4\) intermediate tool calls, and \(5\) constraints and filters \(see Appendix Table[1](https://arxiv.org/html/2610.27307#A2.T1)for definitions and examples\) and use an LLM judge \(Gemini 3\.6 Flash\) to annotate each task with the observed feature for comparison in Fig\.[5](https://arxiv.org/html/2610.27307#S5.F5)\. Intermediate tool calls are always present in both the original and evolved task sets, but the percentage of all other task features increase as large as 212% afterTURTLE\.

![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/semantic_embedding_gemma_appworld_v2.png)Figure 6:t\-SNE visualization of task instruction embeddings in orange \(original\) and blue \(TURTLE\)\. The arrows show the top 10 original\-evolved task pairs with the largest cosine distances\.TURTLEgenerates new tasks that are semantically different from the original tasks, expanding the search space\.
### 5\.3TURTLEenhances diversity

Another desideratum for task evolution is whether the tasks are modified in semantically and functionally meaningful ways beyond making natural\-language or numeric perturbations to the original task instructions\.

Text embeddings of task instructions\.To examine whetherTURTLEmakes semantically meaningful changes to the original dataset, we embed the original and generated task instructions using Gemini Embedding 2\([Shanbhogue et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib36)\)\(with 768 embedding size\) and present a t\-SNE visualization\([van der Maaten and Hinton, 2008](https://arxiv.org/html/2610.27307#bib.bib35)\)of the embeddings in Fig\.[6](https://arxiv.org/html/2610.27307#S5.F6)\(This figure shows results from experiments with the Gemma solver on AppWorld; more examples in Appendix Fig\.[20](https://arxiv.org/html/2610.27307#A2.F20)\)\. The top 10 largest cosine distances range from 0\.30 to 0\.49, with the largest change between the original instruction:Send $68 privately on Venmo to the person with this phone number 2114992604, and the evolved instruction, involving multi\-step tool calls and reasoning about state dependencies:Find my birth month from my profile\. Then, find all my coworkers who were also born in that month and live in Seattle\. Send $75 to each of them privately on Venmo\. For the one who is oldest, use the note ‘To the most senior birthday buddy\!’\. For all the others, use ‘Happy birthday \[First Name\] buddy\!’, replacing \[First Name\] with their actual first name \(e\.g\., ‘Happy birthday Sherry buddy\!’\)\. Finally, send an email to the oldest coworker with the subject ‘Happy Birthday\!’ and body ‘Hope you have a great one\!’\.In this example, the evolved instruction requires the solver to first find each user’s birth month, send messages to all users who satisfy the condition, and customize each message based on the user’s attributes, which are missing from the original instruction\. We also provide examples of a task lineage in Appendix Fig\.[13](https://arxiv.org/html/2610.27307#A2.F13),[14](https://arxiv.org/html/2610.27307#A2.F14),[15](https://arxiv.org/html/2610.27307#A2.F15), showing how the proposer gradually adds new steps and variables to increase the task difficulty\.

Diversity of simultaneous tool calls\.We also observe changes in task requirements \(i\.e\., the number and types of software applications used\) after applyingTURTLEto AutomationBench\. Initially, all 16 original tasks require four apps, but after evolution, 52% of the 23 evolved tasks require five apps and 13% require six\. The changes are not limited to the number of apps used\. Fig\.[7](https://arxiv.org/html/2610.27307#S5.F7)shows the frequency of app co\-occurrences in the original and evolved tasks\. The number of unique app pairs required by each task also increases from 15 to 30 \(out of 36 possible unordered pairs\), and the evolved tasks now require using new apps like Banboohr, Google Drive, and Stripe\.TURTLEmakes substantive changes to the underlying tool interactions, rather than modifying instructional styles or function parameters in a templated way\.

![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/co_occurence_plot_automation_bench.png)Figure 7:Co\-occurrence of apps between original \(left\) andTURTLE\(right\) tasks in AutomationBench\. Diagonal entries indicate the percentage of tasks that use that particular app rather than co\-occurrence of two apps\.
### 5\.4TURTLEfor failure diagnosis

TURTLEcan elicit new model failure modes, whereas the original tasks are often too easy to highlight solver vulnerabilities\. For example, in the AppWorld datasets, we observe that Gemini 3\.1 Flash Lite and Gemini 3\.6 Flash sometimes struggle with ambiguous instructions or responses and fail at non\-tool\-calling related reasoning steps, while Gemma 4 31B\-it often fails to infer the user’s intent when following complex instructions\.

- •Original task:*How many people follow the artist of the currently playing song on Spotify?*\(solved perfectly\)
- •New task:*Check the currently playing song on Spotify\. I need a passcode\. The passcode is the song’s duration minus the song\_id, multiplied by the number of artists, and then concatenated with the last word of the song’s title in uppercase\. For example, if duration is 300, song\_id is 100, there are 2 artists, and the title is ’Hello World’, the passcode is ’400WORLD’\. What is the passcode?*
- •Gemini 3\.1 Flash Liteinterprets the artist’s name, “Seraphina Dawn", as two artists and multiplies incorrectly\. This shows the model lacks the commonsense understanding needed to resolve ambiguity in the text response\.
- •Original task:*Send the following phone message to my siblings and roommates, who do not have a venmo account, “Get on venmo please\!"\.*\(solved perfectly\)
- •New task:*I need to contact people I live with and my brothers, but not my coworkers\. Find out if they use Venmo\. For those who don’t, text them ‘Get on venmo please\!’\. For people I live with that are on Venmo, send them a reminder to ‘Pay me back\!’\. Don’t message my brothers if they’re on Venmo\.*
- •Gemma 4 31B\-itsends a Venmo payment request with a hallucinated amount of $1 instead of sending a text reminder, as requested\.

We also build a taxonomy of solver failure modes \(Appendix Table[2](https://arxiv.org/html/2610.27307#A2.T2)\) and use Gemini 3\.6 Flash as the judge to identify the primary failure reason for each task with any observed failure\. For both the original andTURTLE\-generated datasets, we count how often a specific failure reason co\-occurs with a given task feature, as shown in Fig\.[8](https://arxiv.org/html/2610.27307#S5.F8)\(note that each task can have multiple feature labels but only one failure reason\), so each cell in the heatmap shows the fraction of tasks that exhibit featureiiand elicit failure modejj, divided by the dataset size\.TURTLE\-generated tasks cover more \(task feature, failure mode\) combinations than the original tasks \(54 vs\.49 pairs\)\. Broader coverage is desirable, since a robust test suite ideally elicits a variety of failure modes across different task features, rather than exploiting a single model vulnerability or a source of task difficulty\. In the original task heatmap, 11 pairs are missing, whereas in the evolved heatmap, 6 pairs are missing\.

To further increase the diversity of the tasks \(and elicited failures\), one could turn this taxonomy into a reward bonus or into guidelines for the task Proposer, target each failure mode during the evolutionary search, and repeat generation for each seed task and desired failure combination\. Despite not using this taxonomy a priori,TURTLEalready produces tasks that cover diverse failure categories and feature types, compared to the original benchmarks\.

![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/failure_matrix.png)Figure 8:Task features versus error categories for the original \(left\) and evolved \(right\) tasks\. Darker cells indicate higher frequency and cells with 0 errors are highlighted in red\. Tasks with 100% completion rates are excluded from the numerator since they do not elicit any failure, but they remain in the denominator\.

## 6Discussion & Conclusion

As LLM capabilities advance, we need a scalable task generation approach to understand their current challenges and ways to improve them\. We proposeTURTLE, a multi\-agent evolutionary approach for generating*solvable yet difficult*tasks\.TURTLEshows that hard tasks do not always require building new testing environments or APIs from scratch\. Instead, existing tasks can be evolved and expanded by modifying instructions and recombining tool call sequences\.

TURTLEhas two clear advantages over existing data generation methods\. \(1\) Evolving: Tasks evolve from the parent seed to target the solver’s competence, rather than converging on a one\-size\-fits\-all static evaluation set\. It also uses a gradient\-free evolutionary search algorithm, which allows task generation to incorporate closed, frontier models for higher\-quality generation\. \(2\) Self\-tuning: Unlike prior works that rely on humans for task design and refinement,TURTLEis fully automated, with self\-verification in a Dockerized environment and tuning based on the solver’s completion rate to ensure task reliability\.

We applyTURTLEto popular benchmarks, including AppWorld,τ\\tau\-bench, and AutomationBench, and evaluate the difficulty of the generated tasks relative to the original datasets\. When the original tasks are too easy,TURTLEyields tasks that are more difficult, leading to a broader range of solver failures\.

While we mainly useTURTLEfor evaluation, it can also be integrated into model training by scaling up training data generation with minimal human supervision\. In particular,[Rutherford et al\. \(2024\)](https://arxiv.org/html/2610.27307#bib.bib32)show that generating tasks with maximum outcome variance is ideal for learning, since they target the solver’s learning frontier\. We can first generate a new curriculum withTURTLEand train the solver using reinforcement learning with verifiable rewards\([Lambert et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib43);[Guo et al\., 2025a](https://arxiv.org/html/2610.27307#bib.bib1)\)or apply filtered behavioral cloning to successful trajectories\([Hong et al\., 2023](https://arxiv.org/html/2610.27307#bib.bib10)\)\. This process of generating new tasks and training the solver can be repeated until the solver’s learning plateaus, without collecting new human data or manually adjusting task difficulty\. This can help close the model\-improvement loop, from synthetic data generation for evaluation to training\.

## Limitations

TURTLEuses gradient\-free evolutionary search, so it can be applied to both open\-weight and closed\-weight models through API calls\. However, the inference cost of proprietary models can be a limiting factor\. We ran some ablations with different solver models \(e\.g\., Gemini 3\.6 Flash and Gemini 3\.1 Flash Lite\) on AppWorld using only a small subset of initial tasks\. Our main objective was to show thatTURTLEapplies to different solver types and is not overfitted to the Gemma solver, and to demonstrate different ways the generated tasks may be used\. To make the datasets larger and more useful for future use, the initial task sets would need to be seeded with more parent tasks\.

#### Ethical Consideration\.

As LLMs become more pervasive and integrated into our everyday lives and workflows, evaluating their failure modes is crucial for safe deployment and user control over AI systems\. Our work contributes to safety by proposing new ways of generating task data to assess model capabilities, while minimizing human supervision, reducing the engineering and data\-generation burden\. However, overreliance on AI systems based on their benchmark performance can be harmful, since benchmarks are designed to assess only certain aspects of a model’s competence and can become saturated as models are continuously optimized against publicly released datasets\.

#### AI Use Disclosure\.

We used AI coding assistants to extend AlphaEvolve to different tool\-use benchmarks and to debug implementations\. We also used AI to polish the paper \(both the writing and the formatting in Overleaf\) and to polish the prompts used in our experiments\. When LLMs are used as the evolutionary search orchestrator or as judges \(e\.g\., evaluating task difficulty, labeling task features, and identifying failure reasons\), their uses are described in the Methods section\.

## References

- Acikgozet al\.\(2026\)E\. C\. Acikgoz, C\. Qian, J\. Hübotter, H\. Ji, D\. Hakkani\-Tür, and G\. TurTool\-r0: self\-evolving llm agents for tool\-learning from zero data\.External Links:2602\.21320,[Link](https://arxiv.org/abs/2602.21320)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1),[§3\.1](https://arxiv.org/html/2610.27307#S3.SS1.p3.1)\.
- Assumpçãoet al\.\(2026\)H\. Assumpção, D\. Ferreira, L\. Campos, and F\. MuraiCodeEvolve: an open source evolutionary coding agent for algorithmic discovery and optimization\.External Links:2510\.14150,[Link](https://arxiv.org/abs/2510.14150)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px2.p1.1)\.
- Bäck and Schwefel \(1993\)T\. Bäck and H\. SchwefelAn overview of evolutionary algorithms for parameter optimization\.Evolutionary Computation1\(1\),pp\. 1–23\.External Links:[Document](https://dx.doi.org/10.1162/evco.1993.1.1.1)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px2.p1.1)\.
- Barreset al\.\(2025\)V\. Barres, H\. Dong, S\. Ray, X\. Si, and K\. Narasimhanτ2\\tau^\{2\}\-Bench: evaluating conversational agents in a dual\-control environment\.External Links:2506\.07982,[Link](https://arxiv.org/abs/2506.07982)Cited by:[Appendix B](https://arxiv.org/html/2610.27307#A2.p3.1),[Appendix B](https://arxiv.org/html/2610.27307#A2.p4.1),[Appendix B](https://arxiv.org/html/2610.27307#A2.p5.1),[§1](https://arxiv.org/html/2610.27307#S1.p1.1),[§1](https://arxiv.org/html/2610.27307#S1.p4.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1),[§4](https://arxiv.org/html/2610.27307#S4.p1.1)\.
- Caiet al\.\(2025\)S\. Cai, R\. Fang, J\. Wu, B\. Li, X\. Wang, Y\. Jiang, L\. Su, L\. Zhang, W\. Yin, Z\. Zhang, F\. Feng, P\. Xie, and X\. WangAutoForge: automated environment synthesis for agentic reinforcement learning\.External Links:2512\.22857,[Link](https://arxiv.org/abs/2512.22857)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p3.1)\.
- Chopraet al\.\(2026\)H\. Chopra, K\. Ghate, A\. Caliskan, T\. Kohno, C\. Shah, and N\. JaquesBeyond cooperative simulators: generating realistic user personas for robust evaluation of llm agents\.External Links:2605\.12894,[Link](https://arxiv.org/abs/2605.12894)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px2.p1.1)\.
- Denniset al\.\(2021\)M\. Dennis, N\. Jaques, E\. Vinitsky, A\. Bayen, S\. Russell, A\. Critch, and S\. LevineEmergent complexity and zero\-shot transfer via unsupervised environment design\.External Links:2012\.02096,[Link](https://arxiv.org/abs/2012.02096)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1)\.
- Donget al\.\(2026\)G\. Dong, J\. Lu, J\. Huang, W\. Zhong, L\. Liu, S\. Huang, Z\. Li, Y\. Zhao, X\. Song, X\. Li, J\. Jin, Y\. Zhu, H\. Wang, F\. Lei, Q\. Luo, M\. Chen, Z\. Chen, J\. Feng, J\. Wen, and Z\. DouAgent\-world: scaling real\-world environment synthesis for evolving general agent intelligence\.External Links:2604\.18292,[Link](https://arxiv.org/abs/2604.18292)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1)\.
- Doshi and Google Gemini Team \(2026\)T\. Doshi and Google Gemini TeamIntroducing gemini 3\.6 flash, 3\.5 flash\-lite, and 3\.5 flash cyber\.Note:Google Blog\. Accessed: 2026\-09\-30External Links:[Link](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-6-flash-3-5-flash-lite-3-5-flash-cyber/)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p4.1),[§4](https://arxiv.org/html/2610.27307#S4.p6.1)\.
- Esfandiarpooret al\.\(2025\)R\. Esfandiarpoor, V\. Suryanarayanan, S\. H\. Bach, V\. Chowdhary, and A\. AueTheMCPCompany: creating general\-purpose agents with task\-specific tools\.External Links:2510\.19286,[Link](https://arxiv.org/abs/2510.19286)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1)\.
- Faldoret al\.\(2025\)M\. Faldor, J\. Zhang, A\. Cully, and J\. CluneOMNI\-epic: open\-endedness via models of human notions of interestingness with environments programmed in code\.External Links:2405\.15568,[Link](https://arxiv.org/abs/2405.15568)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1)\.
- Fanget al\.\(2025\)R\. Fang, S\. Cai, B\. Li, J\. Wu, G\. Li, W\. Yin, X\. Wang, X\. Wang, L\. Su, Z\. Zhang, S\. Wu, Z\. Tao, Y\. Jiang, P\. Xie, F\. Huang, and J\. ZhouTowards general agentic intelligence via environment scaling\.External Links:2509\.13311,[Link](https://arxiv.org/abs/2509.13311)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p3.1)\.
- Google Gemini Team \(2026\)Google Gemini TeamGemini 3\.1 flash\-lite: built for intelligence at scale\.Note:Google Blog\. Accessed: 2026\-09\-30External Links:[Link](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-flash-lite/)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p4.1),[§4](https://arxiv.org/html/2610.27307#S4.p6.1)\.
- Guoet al\.\(2025a\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Ding, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Chen, J\. Yuan, J\. Tu, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. You, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. ZhangDeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§6](https://arxiv.org/html/2610.27307#S6.p4.1)\.
- Guoet al\.\(2025b\)J\. Guo, L\. Yang, P\. Chen, Q\. Xiao, Y\. Wang, X\. Juan, J\. Qiu, K\. Shen, and M\. WangGenEnv: difficulty\-aligned co\-evolution between llm agents and environment simulators\.External Links:2512\.19682,[Link](https://arxiv.org/abs/2512.19682)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1),[§3\.1](https://arxiv.org/html/2610.27307#S3.SS1.p3.1)\.
- Guret al\.\(2022\)I\. Gur, N\. Jaques, Y\. Miao, J\. Choi, M\. Tiwari, H\. Lee, and A\. FaustEnvironment generation for zero\-shot compositional reinforcement learning\.External Links:2201\.08896,[Link](https://arxiv.org/abs/2201.08896)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1)\.
- Honget al\.\(2023\)J\. Hong, S\. Levine, and A\. DraganZero\-shot goal\-directed dialogue via rl on imagined conversations\.External Links:2311\.05584,[Link](https://arxiv.org/abs/2311.05584)Cited by:[§6](https://arxiv.org/html/2610.27307#S6.p4.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSWE\-bench: can language models resolve real\-world github issues?\.External Links:2310\.06770,[Link](https://arxiv.org/abs/2310.06770)Cited by:[Table 1](https://arxiv.org/html/2610.27307#A2.T1),[Table 2](https://arxiv.org/html/2610.27307#A2.T2)\.
- Lambertet al\.\(2025\)N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu, Y\. Gu, S\. Malik, V\. Graf, J\. D\. Hwang, J\. Yang, R\. L\. Bras, O\. Tafjord, C\. Wilhelm, L\. Soldaini, N\. A\. Smith, Y\. Wang, P\. Dasigi, and H\. HajishirziTulu 3: pushing frontiers in open language model post\-training\.External Links:2411\.15124,[Link](https://arxiv.org/abs/2411.15124)Cited by:[§6](https://arxiv.org/html/2610.27307#S6.p4.1)\.
- Liuet al\.\(2026\)M\. Liu, L\. Jiang, Y\. Liang, S\. S\. Du, Y\. Choi, T\. Althoff, and N\. JaquesChasing moving targets with online self\-play reinforcement learning for safer language models\.External Links:2506\.07468,[Link](https://arxiv.org/abs/2506.07468)Cited by:[§3\.2](https://arxiv.org/html/2610.27307#S3.SS2.p2.1)\.
- Luet al\.\(2025\)J\. Lu, T\. Holleis, Y\. Zhang, B\. Aumayer, F\. Nan, F\. Bai, S\. Ma, S\. Ma, M\. Li, G\. Yin, Z\. Wang, and R\. PangToolSandbox: a stateful, conversational, interactive evaluation benchmark for llm tool use capabilities\.External Links:2408\.04682,[Link](https://arxiv.org/abs/2408.04682)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1)\.
- Marchandet al\.\(2026\)R\. Marchand, A\. O\. Cathain, J\. Wynne, P\. M\. Giavridis, S\. Jennings, F\. Tuxworth, T\. H\. Dur, S\. Deverett, J\. Wilkinson, J\. Gwartz, and H\. CoppockQuantifying frontier llm capabilities for container sandbox escape\.External Links:2603\.02277,[Link](https://arxiv.org/abs/2603.02277)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1)\.
- Merzoukiet al\.\(2026\)Y\. Merzouki, C\. Izquierdo, M\. Ignuta\-Ciuncanu, M\. Gomez\-Bracamonte, R\. Maggioni, A\. Lombardi, C\. Mazzoleni, F\. Martelli, B\. Gunther, J\. Petersen, and P\. PetersenFactoryBench: evaluating industrial machine understanding\.External Links:2605\.07675,[Link](https://arxiv.org/abs/2605.07675)Cited by:[Table 1](https://arxiv.org/html/2610.27307#A2.T1),[Table 2](https://arxiv.org/html/2610.27307#A2.T2),[§1](https://arxiv.org/html/2610.27307#S1.p1.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p1.1)\.
- Mouret and Clune \(2015\)J\. Mouret and J\. CluneIlluminating search spaces by mapping elites\.External Links:1504\.04909,[Link](https://arxiv.org/abs/1504.04909)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px2.p1.1)\.
- Novikovet al\.\(2025\)A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. R\. Ruiz, A\. Mehrabian, M\. P\. Kumar, A\. See, S\. Chaudhuri, G\. Holland, A\. Davies, S\. Nowozin, P\. Kohli, and M\. BalogAlphaEvolve: a coding agent for scientific and algorithmic discovery\.External Links:2506\.13131,[Link](https://arxiv.org/abs/2506.13131)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2610.27307#S4.p5.1)\.
- Paglieriet al\.\(2026\)D\. Paglieri, L\. Cross, W\. A\. Cunningham, J\. Z\. Leibo, and A\. S\. VezhnevetsPersona generators: generating diverse synthetic personas for arbitrary contexts\.External Links:2602\.03545,[Link](https://arxiv.org/abs/2602.03545)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px2.p1.1)\.
- Patwardhanet al\.\(2025\)T\. Patwardhan, R\. Dias, E\. Proehl, G\. Kim, M\. Wang, O\. Watkins, S\. P\. Fishman, M\. Aljubeh, P\. Thacker, L\. Fauconnet, N\. S\. Kim, P\. Chao, S\. Miserendino, G\. Chabot, D\. Li, M\. Sharman, A\. Barr, A\. Glaese, and J\. TworekGDPval: evaluating ai model performance on real\-world economically valuable tasks\.External Links:2510\.04374,[Link](https://arxiv.org/abs/2510.04374)Cited by:[Table 1](https://arxiv.org/html/2610.27307#A2.T1),[Table 2](https://arxiv.org/html/2610.27307#A2.T2)\.
- Rudeset al\.\(2025\)OpenEvolve: codebase\-scale evolutionary algorithms for code optimization, powered by llmsNote:[https://github\.com/ryanrudes/openevolve](https://github.com/ryanrudes/openevolve)Open\-source evolutionary program synthesis framework\. Based on original code licensed under Apache License 2\.0 by the FunSearch project\.Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px2.p1.1)\.
- Rutherfordet al\.\(2024\)A\. Rutherford, M\. Beukman, T\. Willi, B\. Lacerda, N\. Hawes, and J\. FoersterNo regrets: investigating and improving regret approximations for curriculum discovery\.External Links:2408\.15099,[Link](https://arxiv.org/abs/2408.15099)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1),[§1](https://arxiv.org/html/2610.27307#S1.p3.1),[2nd item](https://arxiv.org/html/2610.27307#S3.I1.i2.p1.1),[§6](https://arxiv.org/html/2610.27307#S6.p4.1)\.
- Samvelyanet al\.\(2024\)M\. Samvelyan, S\. C\. Raparthy, A\. Lupu, E\. Hambro, A\. H\. Markosyan, M\. Bhatt, Y\. Mao, M\. Jiang, J\. Parker\-Holder, J\. Foerster, T\. Rocktäschel, and R\. RaileanuRainbow teaming: open\-ended generation of diverse adversarial prompts\.External Links:2402\.16822,[Link](https://arxiv.org/abs/2402.16822)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1)\.
- Shanbhogueet al\.\(2026\)M\. Shanbhogue, Z\. Li, S\. Zhang, G\. H\. Ábrego, S\. Huang, A\. Jain, D\. Salz, S\. Goenka, C\. Hegde, J\. Ma, F\. Chen, J\. Wu, T\. Dabral, B\. Samari, K\. Poulet, D\. Cer, K\. Chen, P\. Suganathan, H\. Hui, J\. Andonov, P\. Schlattner, J\. Han, I\. Naim, W\. Lowe, V\. Pchelin, A\. Yang, Y\. Chen, Z\. Ding, G\. Zhang, G\. Heigold, Y\. Chen, A\. Reveillon, B\. Mccloskey, W\. Zhou, D\. Kim, R\. Meng, E\. Wang, J\. Zheng, H\. Fede, Z\. Yang, K\. Mosley, B\. Potetz, S\. Dua, H\. S\. Vera, S\. Gao, H\. Zhang, A\. Hess, H\. Ying, A\. Montes, K\. Gill, M\. Choi, S\. Russo, A\. Hauth, J\. Lee, M\. Boratko, M\. Barnes, V\. Rao, C\. Musat, C\. Allauzen, E\. Variani, S\. Kumar, T\. Bagby, J\. Jiao, Y\. Gu, T\. Li, A\. Agrawal, R\. Santana, D\. Nath, S\. Karukas, S\. Han, L\. Loher, A\. Twu, N\. Vyas, S\. Bhai, F\. P\. Gomez, W\. Zhang, C\. Liu, J\. Yang, S\. Qiu, S\. Zhang, S\. Kulkarni, S\. Rothe, S\. Nakamoto, R\. Hoffmann, Z\. Gleicher, Y\. Sung, Q\. Yin, T\. Duerig, and M\. SeyedhosseiniGemini embedding 2: a native multimodal embedding model from gemini\.External Links:2605\.27295,[Link](https://arxiv.org/abs/2605.27295)Cited by:[§5\.3](https://arxiv.org/html/2610.27307#S5.SS3.p2.1)\.
- Shepard and Salimans \(2026\)D\. Shepard and R\. SalimansAutomationBench\.External Links:2604\.18934,[Link](https://arxiv.org/abs/2604.18934)Cited by:[Table 1](https://arxiv.org/html/2610.27307#A2.T1),[Table 2](https://arxiv.org/html/2610.27307#A2.T2),[Appendix B](https://arxiv.org/html/2610.27307#A2.p9.1),[§1](https://arxiv.org/html/2610.27307#S1.p1.1),[§1](https://arxiv.org/html/2610.27307#S1.p4.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2610.27307#S3.SS2.p1.1),[§4](https://arxiv.org/html/2610.27307#S4.p1.1)\.
- Shiet al\.\(2026\)Q\. Shi, A\. Zytek, P\. Razavi, K\. Narasimhan, and V\. Barresτ\\tau\-Knowledge: evaluating conversational agents over unstructured knowledge\.External Links:2603\.04370,[Link](https://arxiv.org/abs/2603.04370)Cited by:[Table 1](https://arxiv.org/html/2610.27307#A2.T1),[Table 2](https://arxiv.org/html/2610.27307#A2.T2),[Appendix B](https://arxiv.org/html/2610.27307#A2.p3.1),[Appendix B](https://arxiv.org/html/2610.27307#A2.p6.1),[Appendix B](https://arxiv.org/html/2610.27307#A2.p7.1),[§1](https://arxiv.org/html/2610.27307#S1.p1.1),[§1](https://arxiv.org/html/2610.27307#S1.p4.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1),[§4](https://arxiv.org/html/2610.27307#S4.p1.1),[§4](https://arxiv.org/html/2610.27307#S4.p3.1)\.
- Songet al\.\(2026\)X\. Song, H\. Chang, G\. Dong, Y\. Zhu, J\. Wen, and Z\. DouEnvScaler: scaling tool\-interactive environments for llm agent via programmatic synthesis\.External Links:2601\.05808,[Link](https://arxiv.org/abs/2601.05808)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1)\.
- Teamet al\.\(2026\)G\. Team, S\. E\. Abd, V\. Aggarwal, R\. Algayres, A\. Andreev, O\. Bachem, I\. Ballantyne, C\. Brick, V\. Cărbune, M\. Casbon, M\. Chaturvedi, A\. Chawla, V\. Cotruta, A\. Coucke, P\. Culliton, R\. Dadashi, L\. Dixon, M\. Elhawaty, U\. Evci, C\. Farabet, J\. Ferret, F\. Galgani, S\. Girgin, J\. Grill, M\. Grootendorst, J\. Guo, C\. Hardin, Y\. He, S\. M\. Hernandez, O\. Homburger, L\. Hussenot, J\. Ji, A\. Joulin, A\. Kamath, P\. Kassraie, O\. Lacombe, P\. Lahoti, G\. Liu, G\. Martins, L\. Martins, T\. Matejovicova, R\. Merhej, N\. Momchev, S\. Mondal, R\. Mullins, S\. R\. Panyam, S\. Pathak, S\. Perrin, A\. Pinto, E\. Pot, A\. Pouget, A\. Ramé, S\. Ramos, D\. Reid, D\. Rim, M\. Rivière, K\. Roth, L\. Rouillard, O\. Sanseviero, P\. G\. Sessa, S\. Settle, D\. Sinopalnikov, S\. Smoot, P\. Stanczyk, A\. Steiner, L\. Stewart, I\. Tolstikhin, M\. Tschannen, A\. Tsitsulin, N\. Vieillard, R\. Wu, P\. Xu, H\. Yang, E\. Yvinec, B\. Zhang, L\. Zhang, J\. Zou, N\. Aagnes, A\. Abdelhamed, J\. Adamek, S\. Agrawal, S\. Agrawal, I\. Alabdulmohsin, J\. B\. Alayrac, U\. Alon, C\. Amarnath, A\. Anand, C\. Anastasiou, S\. Ariafar, F\. Aubet, K\. Axiotis, F\. Barbero, J\. Barral, A\. Bendebury, U\. Bergmann, S\. Bileschi, K\. Black, M\. Blondel, S\. Borgeaud, A\. Bražinskas, R\. Burnell, R\. Busa\-Fekete, M\. Cai, D\. Calandriello, G\. Cameron, C\. Caucheteux, R\. Chaabouni, G\. Chadha, J\. Chan, B\. J\. Chen, J\. Chen, L\. Chen, X\. Chen, D\. Cheng, T\. Chien, N\. Chinaev, Y\. Chou, Z\. Chu, B\. Coleman, P\. Consul, S\. Conway\-Rahman, S\. Crowell, D\. Cutler, V\. Dani, S\. Daruki, A\. Das, D\. Deutsch, N\. Dikkala, L\. Ding, Q\. Ding, S\. Dodhia, K\. Donhauser, T\. Doshi, A\. Dragan, A\. Druinsky, S\. Dua, Z\. Egyed, D\. Eisenbud, D\. Eppens, C\. Fan, B\. Fatemi, Y\. Fathullah, V\. Feinberg, M\. Ferev, S\. Flennerhag, T\. Fujimoto, J\. G\. Oliveira, I\. Galatzer\-Levy, J\. Gante, S\. Geisler, S\. Ghosal, A\. M\. Girgis, T\. von Glehn, A\. Go, A\. Gokhale, A\. Grills, Y\. Gu, M\. Gupta, P\. Gupta, G\. Guruganesh, R\. Hadsell, H\. Harkous, J\. Harlalka, D\. Hassabis, A\. Hauth, J\. Heyward, A\. Hosseini, C\. Hsia, I\. Hsu, X\. Huang, Y\. Huang, K\. Hui, A\. Hutter, T\. I, F\. Iliopoulos, A\. Jain, G\. Jawahar, Z\. Ji, Q\. Jin, M\. Johnson, K\. Joshi, A\. Kandoor, W\. Kang, K\. Kavukcuoglu, M\. Kazemi, K\. Kenealy, A\. Khalifa, P\. Kirk, I\. Korotkov, S\. Kothawade, V\. Kovalev, N\. Kovelamudi, A\. Kraft, R\. Kumar, V\. Kumar, H\. Kuppam, J\. Lannin, C\. Lee, S\. Lee, D\. Lepikhin, A\. Levkovitch, D\. Li, Q\. Li, V\. Liévin, E\. Lin, Z\. Lin, C\. Liu, T\. Liu, T\. Liu, X\. Liu, I\. Lobov, M\. Lunayach, M\. Ma, G\. Madan, A\. Maksai, E\. Malmi, M\. Matuszak, D\. McDuff, G\. Menghani, M\. Mikuła, D\. Mirylenka, K\. Misiunas, V\. Misra, A\. Mitran, K\. Mohamed, M\. Mukha, E\. Noland, J\. O’Donnell, B\. O’Donoghue, K\. Olszewska, B\. Orlando, W\. Pan, R\. Panigrahy, U\. Parekh, N\. Perez\-Nieves, C\. Park, E\. Paskie, L\. Peng, B\. Petrini, S\. Petrov, J\. Pfeiffer, B\. Piot, M\. Plomecka, S\. Poder, O\. Ponce, A\. Pramanik, D\. Racz, A\. Rajan, M\. Ramanovich, A\. Rao, M\. Ritter, V\. Rodrigues, E\. Rosen, M\. Rybiński, N\. Sachdeva, M\. E\. Sander, R\. Sathyanarayana, S\. Savla, S\. Schmidgall, T\. Schuster, G\. Scrivener, B\. Seguin, A\. Sellergren, A\. Severyn, I\. Shafran, D\. Shah, B\. Shahriari, Y\. Shangguan, A\. Shenoy, P\. Shenoy, R\. Shivanna, P\. Sho, L\. Spangher, W\. Stokowiec, T\. Strother, Y\. Su, Y\. Sun, M\. Sundararajan, A\. Tacchetti, M\. H\. Taege, P\. Tafti, J\. Tarbouriech, C\. Tekur, S\. Thakoor, R\. Thapa, M\. Traverse, L\. Treven, T\. Tu, C\. T\. Tung, Çağlar Ünlü, P\. Veličković, M\. P\. Venkat, S\. G\. Venkatesh, V\. Venkiteswaran, F\. Visin, A\. Vitvitskyi, K\. Vodrahalli, W\. Wang, X\. Wang, T\. Warkentin, J\. Wassenberg, J\. Wieting, C\. Wu, L\. Xiao, H\. Xu, Y\. Xu, F\. Xue, A\. Yadav, J\. Yan, A\. Yang, L\. Yang, M\. Yang, Z\. Ying, J\. H\. Yoo, M\. Zadimoghaddam, S\. Zafar, F\. Zhang, J\. Zhang, J\. Zhang, X\. Zhang, C\. Zhao, D\. Zhou, and C\. ZouGemma 4 technical report\.External Links:2607\.02770,[Link](https://arxiv.org/abs/2607.02770)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p4.1),[§4](https://arxiv.org/html/2610.27307#S4.p6.1)\.
- Trivediet al\.\(2024\)H\. Trivedi, T\. Khot, M\. Hartmann, R\. Manku, V\. Dong, E\. Li, S\. Gupta, A\. Sabharwal, and N\. BalasubramanianAppWorld: a controllable world of apps and people for benchmarking interactive coding agents\.External Links:2407\.18901,[Link](https://arxiv.org/abs/2407.18901)Cited by:[Figure 12](https://arxiv.org/html/2610.27307#A2.F12),[Table 1](https://arxiv.org/html/2610.27307#A2.T1),[Table 2](https://arxiv.org/html/2610.27307#A2.T2),[Appendix B](https://arxiv.org/html/2610.27307#A2.p1.1),[§1](https://arxiv.org/html/2610.27307#S1.p1.1),[§1](https://arxiv.org/html/2610.27307#S1.p4.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1),[§4](https://arxiv.org/html/2610.27307#S4.p1.1),[§5\.1](https://arxiv.org/html/2610.27307#S5.SS1.p1.1)\.
- van der Maaten and Hinton \(2008\)L\. van der Maaten and G\. HintonVisualizing data using t\-sne\.Journal of Machine Learning Research9\(86\),pp\. 2579–2605\.External Links:[Link](http://jmlr.org/papers/v9/vandermaaten08a.html)Cited by:[§5\.3](https://arxiv.org/html/2610.27307#S5.SS3.p2.1)\.
- Wanget al\.\(2026a\)C\. Wang, D\. Kasenberg, K\. Stachenfeld, and P\. S\. CastroDiscovering differences in strategic behavior between humans and llms\.External Links:2602\.10324,[Link](https://arxiv.org/abs/2602.10324)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024\)R\. Wang, G\. Todd, Z\. Xiao, X\. Yuan, M\. Côté, P\. Clark, and P\. JansenCan language models serve as text\-based world simulators?\.External Links:2406\.06485,[Link](https://arxiv.org/abs/2406.06485)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1)\.
- Wanget al\.\(2026b\)Z\. Wang, C\. Xu, B\. Liu, Y\. Wang, S\. Han, Z\. Yao, H\. Yao, and Y\. HeAgent world model: infinity synthetic environments for agentic reinforcement learning\.External Links:2602\.10090,[Link](https://arxiv.org/abs/2602.10090)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1)\.
- Weiet al\.\(2026\)Y\. Wei, Z\. Sun, E\. McMilin, J\. Gehring, D\. Zhang, G\. Synnaeve, D\. Fried, L\. Zhang, and S\. WangToward training superintelligent software agents through self\-play swe\-rl\.External Links:2512\.18552,[Link](https://arxiv.org/abs/2512.18552)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p2.1)\.
- Yanget al\.\(2026\)J\. Yang, K\. Lieret, J\. Ma, P\. Thakkar, D\. Pedchenko, S\. Sootla, E\. McMilin, P\. Yin, R\. Hou, G\. Synnaeve, D\. Yang, and O\. PressProgramBench: can language models rebuild programs from scratch?\.External Links:2605\.03546,[Link](https://arxiv.org/abs/2605.03546)Cited by:[§1](https://arxiv.org/html/2610.27307#S1.p1.1),[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.External Links:2406\.12045,[Link](https://arxiv.org/abs/2406.12045)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1)\.
- Yuanet al\.\(2026\)J\. Yuan, J\. Nöther, N\. Jaques, and G\. RadanovićAgenticRed: evolving agentic systems for red\-teaming\.External Links:2601\.13518,[Link](https://arxiv.org/abs/2601.13518)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px2.p1.1)\.
- Zhouet al\.\(2026\)Z\. Zhou, K\. Uehara, H\. Zhang, J\. Zhou, L\. Gu, R\. Du, Z\. Xu, and T\. HaradaToolGrad: efficient tool\-use dataset generation with textual "gradients"\.External Links:2508\.04086,[Link](https://arxiv.org/abs/2508.04086)Cited by:[§2](https://arxiv.org/html/2610.27307#S2.SS0.SSS0.Px1.p2.1)\.

## Appendix AAppendix: Task Proposer Prompts

1

2

3

4

5Actasanexpertsoftwaredeveloper\.Yourtaskistoiterativelyimprovetheprovidedcodebase\.Theprimarygoalistoincreasethescoresontheprovidedevaluationmetrics,wherelargervaluesarebetter\.

6

7AlwaysadheretobestpracticesinPythoncoding\.

8

9

10Whenevolvingthe‘get\_task‘functionin‘appworld\_task\_composer\.py‘forAppWorld:

11\-ReturnaPythondictionarycontainingthetaskstructure\.

12\-\*\*IMPORTANT\*\*:MakesuretousePythonconstants\(e\.g\.,‘True‘,‘False‘,‘None‘\)\.

13\-Keepthedictionarywithinthe‘EVOLVE\-BLOCK\-START‘and‘EVOLVE\-BLOCK\-END‘\.

Task Proposer Prompt for AppWorld1

2

3Actasanexpertsoftwaredeveloper\.Yourtaskistoiterativelyimprovetheprovidedcodebase\.Theprimarygoalistoincreasethescoresontheprovidedevaluationmetrics,wherelargervaluesarebetter\.

4

5AlwaysadheretobestpracticesinPythoncoding\.

6

7

8Theoverarchinggoaloftaskdesignisto\*\*stress\-testthesolveragenttodetectitsblindspots,failuremodes,andreasoningvulnerabilities\*\*:

9\-\*\*IsolateProblems&FailureModes\*\*:Designscenariosthatisolatespecificreasoningortool\-usedifficulties\(e\.g\.,multi\-constrainttrade\-offs,distractoroptions,subtlepolicystipulations,ambiguousvs\.preciseuserintents,stricteligibilityprerequisites\)\.

10\-\*\*CreativeVulnerabilityElicitation\*\*:Becreativeinhowvulnerabilitiesareelicitedthroughtheconversationalflow\.Forinstance,craftrealisticuserpersonasthatdiscloseconstraintsincrementally,presentconflictingpreferencesthatrequirecarefulKBverification,ortestthesolver’ssusceptibilitytoprematureconclusionswithoutproperverification\.

11

12\#\#TaskGenerationGuidelines

13Whenevolvingthe‘get\_task‘functionin‘tau\_banking\_task\_composer\.py‘forTauBanking:

14\-Evolvethe‘task‘Pythondictionarydefiningthe‘user\_scenario‘,‘initial\_state‘,‘evaluation\_criteria‘,and‘user\_tools‘\.

15\-\*\*IMPORTANT\*\*:MakesuretousevalidPythontypes\(e\.g\.,‘True‘,‘False‘,‘None‘\)andfollowthePydanticschemaforTau2Tasks\.DoNOTuseJSONtypes\(e\.g\.,‘true‘,‘false‘,‘null‘\)\.

16\-Keepyourchangesstrictlywithinthe‘EVOLVE\-BLOCK\-START‘and‘EVOLVE\-BLOCK\-END‘comments\.

17

18\#\#CreatingNewTasks\(Important\)

19Unlikestandaloneagents,Tau2involvesaconversationalinteraction\.ThesolveragentstartscompletelyblindanddoesNOTreceivetaskinstructionsdirectly\.Instead,thetaskinstructionisconveyedtothesolverdynamicallythrougha\*\*UserSimulator\*\*\.

20\-Togivethesolveranewtask,youmustmodifythe‘user\_scenario\.instructions‘fieldtoprovideanewroleplayscript,persona,andconversationflowfortheUserSimulatortoactout\.

21\-Youmustalsoupdatethe‘evaluation\_criteria\.actions‘toreflectthenewexpectedsequenceofgroundtruthAPIcallsthesolvershouldusetosolvethisnewuserscenario\.

22\-\*\*DoNOTmodifyorredefine‘required\_documents‘\*\*:The‘required\_documents‘fieldisintentionallyplaced\*\*outside\*\*the‘EVOLVE\-BLOCK‘attheendof‘get\_task‘andcontainsthestaticKnowledgeBasedocumentsavailableforthistask\.YoumustNOTadd,remove,ormodify‘required\_documents‘inyourevolvedcode\.Instead,usetheprovideddocumentsoutsidetheevolveblockasfixedreferencematerialtogroundyour‘user\_scenario‘and‘evaluation\_criteria‘\.’

Task Proposer Prompt forτ\\tau\-bench banking1

2

3Actasanexpertsoftwaredeveloper\.Yourtaskistoiterativelyimprovetheprovidedcodebase\.Theprimarygoalistoincreasethescoresontheprovidedevaluationmetrics,wherelargervaluesarebetter\.

4

5AlwaysadheretobestpracticesinPythoncoding\.

6

7

8Theoverarchinggoaloftaskdesignisto\*\*stress\-testthesolveragenttodetectitsblindspots,failuremodes,andreasoningvulnerabilities\*\*:

9\-\*\*IsolateProblems&FailureModes\*\*:Designscenariosthatisolatespecificdiagnosticsortroubleshootingchallenges\(e\.g\.,compoundingnetworkfaults,multi\-stepdependencyresolution,hiddendevicesettingmisconfigurations,edge\-caseroaming/datarules\)\.

10\-\*\*CreativeVulnerabilityElicitation\*\*:Becreativeinhowvulnerabilitiesareelicitedthroughtheconversationalusersimulation\.Forinstance,craftpersonaswithincompleteinitialproblemdescriptions,misleadingsymptomdescriptionsthatrequirerigoroussystematicdiagnosis,orstep\-dependentdevicetogglesthatpunishguessingwithoutchecking\.

11

12

13Whenevolvingthe‘get\_task‘functioninthe‘tau\_telecom\_task\_composer\.py‘fileforTauTelecom:

14\-ReturnaPythondictionaryrepresentingthenewormodifiedTau2Taskstructure\.

15\-\*\*IMPORTANT\*\*:MakesuretousevalidPythontypes\(e\.g\.,‘True‘,‘False‘,‘None‘\)andfollowthePydanticschemaforTau2Tasks\.DoNOTuseJSONtypes\(e\.g\.,‘true‘,‘false‘,‘null‘\)\.

16\-Keepthegenerateddictionarywithinthe‘EVOLVE\-BLOCK\-START‘and‘EVOLVE\-BLOCK\-END‘comments\.

17

18

19Unlikestandaloneagents,Tau2involvesaconversationalinteraction\.ThesolveragentstartscompletelyblindanddoesNOTreceivetaskinstructionsdirectly\.Instead,thetaskinstructionisconveyedtothesolverdynamicallythrougha\*\*UserSimulator\*\*\.

20\-Togivethesolveranewtask,youmustmodify‘user\_scenario\.instructions‘\(‘reason\_for\_call‘,‘known\_info‘,‘task\_instructions‘\)and‘user\_scenario\.persona‘toprovidetheroleplayscriptandconversationflowfortheUserSimulator\.

21\-Youmustset‘initial\_state\.initialization\_actions‘toconfiguretheinitialdevice/networkstate\(e\.g\.‘set\_network\_mode\_preference‘,‘break\_vpn‘,‘turn\_airplane\_mode\_on‘,‘set\_user\_location‘,‘turn\_roaming\_off‘,‘disable\_roaming‘,etc\.\)\.

22\-Youmustupdate‘evaluation\_criteria\.actions‘toreflecttheexpectedsequenceofgroundtruthAPI/deviceactions,and‘evaluation\_criteria\.env\_assertions‘toverifythattheissueisgenuinelyresolved\(e\.g\.‘assert\_mobile\_data\_status‘,‘assert\_internet\_speed‘,‘assert\_can\_send\_mms‘,‘assert\_service\_status‘\)\.

Task Proposer Prompt forτ\\tau\-bench telecom1

2

3Actasanexpertsoftwaredeveloperandbenchmarktaskdesigner\.Yourtaskistoiterativelyimprovetheprovidedcodebase\.Theprimarygoalistoincreasethescoresontheprovidedevaluationmetrics,wherelargervaluesarebetter\.

4

5AlwaysadheretobestpracticesinPythoncoding\.

6

7

8Whenevolvingthe‘get\_task‘functionin‘automation\_bench\_task\_composer\.py‘forAutomationBench:

9\-ReturnaPythondictionarycontainingthefulltaskdefinitionforAutomationBench\.

10\-\*\*IMPORTANT\*\*:MakesuretousevalidPythontypes\(e\.g\.,‘True‘,‘False‘,‘None‘,lists,dicts,strings,ints,floats\)\.DoNOTuserawJSONliterals\(e\.g\.‘true‘,‘false‘,‘null‘\)\.

11\-Keepthereturneddictionarystrictlywithinthe‘

12\-DoNOTchangethefunctionsignature‘defget\_task\(task\_name:str=None\)\-\>dict:‘\.

13

14

15Thedictionaryreturnedby‘get\_task‘mustcontain:

161\.‘example\_id‘:AnintegerID\(e\.g\.,4001,4002,etc\.\)\.

172\.‘task\_id‘:Auniquesnake\_casestringidentifierreflectingthedomainandworkflow\(e\.g\.,‘"finance\_disputed\_invoice\_escalation\_4001"‘,‘"support\_sla\_breach\_escalation\_4002"‘,‘"sales\_high\_value\_lead\_routing\_4003"‘\)\.

183\.‘domain‘:Thebusinessdomain\(e\.g\.,‘"finance"‘,‘"support"‘,‘"sales"‘,‘"crm"‘,‘"devops"‘,‘"hr"‘,‘"marketing"‘,‘"operations"‘\)\.

194\.‘name‘:Aconcisehuman\-readabletitledescribingthebusinessworkflow\.

205\.‘description‘:Adictionarycontaining:

21\-‘purpose‘:High\-levelexplanationoftheworkflowacrossmultipleSaaSapplications\.

22\-‘difficulty\_tier‘:‘"easy"‘,‘"medium"‘,or‘"hard"‘\.

23\-‘apps\_used‘:Listofapplicationnamesinvolvedinthescenario\(e\.g\.,‘\["google\_sheets","gmail","crm","slack"\]‘\)\.

246\.‘prompt‘:Therealisticuserrequest/triggerinstructionsprovidedtothesolveragent\.Itmustcontainthenecessarycontext,identifiers,andbusinessconstraintswithoutleakingtheexactsequenceoftoolcalls\.

257\.‘initial\_state‘:Themocksandboxdatabasespopulatedbeforeevaluationstarts\(supports47\+SaaSappsacrosscategories\):

26\-\*\*GoogleWorkspace\*\*:

27\-‘google\_sheets‘:‘\{"spreadsheets":\{"<sheet\_id\>":\{"title":"\.\.\.","rows":\[\{"invoice\_id":"\.\.\.","client":"\.\.\.","amount":123\.45,"status":"\.\.\.","due\_date":"\.\.\.","notes":"\.\.\."\},\.\.\.\]\}\}\}‘

28\-‘gmail‘:‘\{"inbox":\[\{"message\_id":"\.\.\.","from":"\.\.\.","to":"\.\.\.","subject":"\.\.\.","body":"\.\.\."\},\.\.\.\]\}‘

29\-‘google\_drive‘:‘\{"files":\[\{"file\_id":"\.\.\.","name":"\.\.\.","mime\_type":"\.\.\.","content":"\.\.\."\},\.\.\.\]\}‘

30\-‘google\_docs‘:‘\{"documents":\[\{"document\_id":"\.\.\.","title":"\.\.\.","body":"\.\.\."\},\.\.\.\]\}‘

31\-‘google\_calendar‘:‘\{"events":\[\{"event\_id":"\.\.\.","summary":"\.\.\.","start\_time":"\.\.\.","end\_time":"\.\.\.","attendees":\[\.\.\.\]\},\.\.\.\]\}‘

32\-‘google\_workspace‘:‘\{"users":\[\{"email":"\.\.\.","first\_name":"\.\.\.","last\_name":"\.\.\.","org\_unit":"\.\.\."\},\.\.\.\]\}‘

33\-\*\*CRM&Sales\*\*:

34\-‘crm‘/‘hubspot‘/‘salesforce‘:‘\{"accounts":\{"<account\_id\>":\{"account\_name":"\.\.\.","status":"\.\.\.","primary\_contact":"\.\.\.","assigned\_ae":"\.\.\.","support\_tier":"\.\.\.","tam\_email":"\.\.\.","region":"\.\.\.","lifecycle\_stage":"\.\.\."\},\.\.\.\]\}\}‘

35\-\*\*Communications\*\*:

36\-‘slack‘:‘\{"channels":\{"<channel\_id\>":\{"name":"finance\-ops","members":\[\.\.\.\]\}\},"messages":\[\{"channel":"\.\.\.","text":"\.\.\."\}\]\}‘

37\-\*\*Support&Helpdesk\*\*:

38\-‘zendesk‘:‘\{"tickets":\{"<ticket\_id\>":\{"account\_name":"\.\.\.","subject":"\.\.\.","status":"\.\.\.","priority":"\.\.\.","description":"\.\.\."\},\.\.\.\]\}\}‘

39\-‘freshdesk‘:‘\{"tickets":\{"<ticket\_id\>":\{"subject":"\.\.\.","description":"\.\.\.","email":"\.\.\."\},\.\.\.\]\}\}‘

40\-\*\*ProjectManagement\*\*:

41\-‘jira‘:‘\{"issues":\[\{"issue\_key":"\.\.\.","project":"\.\.\.","summary":"\.\.\.","description":"\.\.\.","priority":"\.\.\.","status":"\.\.\."\},\.\.\.\]\}‘

42\-‘linear‘:‘\{"issues":\[\{"id":"\.\.\.","title":"\.\.\.","team\_id":"\.\.\.","description":"\.\.\."\},\.\.\.\]\}‘

43\-‘asana‘:‘\{"tasks":\[\{"task\_id":"\.\.\.","name":"\.\.\.","projects":\[\.\.\.\],"notes":"\.\.\."\},\.\.\.\]\}‘

44\-‘notion‘:‘\{"pages":\[\{"page\_id":"\.\.\.","database\_id":"\.\.\.","title":"\.\.\.","properties":\{\.\.\.\}\},\.\.\.\]\}‘

45\-\*\*HR&Marketing\*\*:

46\-‘airtable‘:‘\{"bases":\{"<base\_id\>":\{"<table\_name\>":\[\{"contact\_id":"\.\.\.","name":"\.\.\.","email":"\.\.\.","company":"\.\.\.","minutes\_attended":45,"questions\_asked":2,"marketing\_opt\_in":True\},\.\.\.\]\}\}\}‘

47\-‘bamboohr‘:‘\{"employees":\[\{"id":"\.\.\.","name":"\.\.\.","status":"\.\.\.","role":"\.\.\.","department":"\.\.\.","manager":"\.\.\.","salary":"\.\.\.","home\_address":"\.\.\."\},\.\.\.\]\}‘

48\-\*\*Finance&Billing\*\*:

49\-‘stripe‘:‘\{"invoices":\[\{"invoice\_id":"\.\.\.","customer":"\.\.\.","amount\_due":123\.45,"status":"\.\.\."\},\.\.\.\],"customers":\[\{"customer\_id":"\.\.\.","email":"\.\.\.","name":"\.\.\."\},\.\.\.\]\}‘

50\-\*\*DevPlatforms\*\*:

51\-‘github‘:‘\{"issues":\[\{"id":"\.\.\.","owner":"\.\.\.","repo":"\.\.\.","title":"\.\.\.","body":"\.\.\."\},\.\.\.\],"invites":\[\{"email":"\.\.\.","role":"\.\.\.","team":"\.\.\."\},\.\.\.\]\}‘

528\.‘info‘:

53\-‘assertions‘:Listoftargetgoalassertionsevaluatedafterthesolverexecution:

54\-‘crm\_account\_field\_equals‘:parameters‘\{"account\_id":"\.\.\.","field":"\.\.\.","expected\_value":"\.\.\."\}‘

55\-‘gmail\_message\_sent\_to\_with\_body\_contains‘:parameters‘\{"to":"\.\.\.","subject\_contains":"\.\.\.","body\_contains":\["\.\.\.","\.\.\."\]\}‘

56\-‘slack\_message\_posted\_to\_channel‘:parameters‘\{"channel":"\.\.\.","text\_contains":\["\.\.\.","\.\.\."\]\}‘

57\-‘google\_sheets\_row\_exists‘:parameters‘\{"spreadsheet\_id":"\.\.\.","cell\_contains":\{"<col\>":"<val\>"\}\}‘

58\-‘zendesk\_ticket\_field\_equals‘:parameters‘\{"ticket\_id":"\.\.\.","field":"status"\|"priority","expected\_value":"\.\.\."\}‘

59\-‘jira\_issue\_exists\_in\_project‘:parameters‘\{"project":"\.\.\.","summary\_contains":"\.\.\."\}‘

60\-‘hubspot\_contact\_has\_tag‘:parameters‘\{"email":"\.\.\.","expected\_tag":"\.\.\."\}‘

61\-‘google\_workspace\_user\_exists‘:parameters‘\{"primary\_email":"\.\.\."\}‘

62\-‘github\_invite\_sent‘:parameters‘\{"email":"\.\.\.","expected\_role":"\.\.\."\}‘

63\-‘guardrails‘:NegativepolicyconstraintsthatmustNOTbetriggered:

64\-‘slack\_message\_not\_posted\_to\_channel‘:parameters‘\{"channel":"general"\}‘

65\-‘gmail\_message\_not\_sent\_to‘:parameters‘\{"to":"unauthorized@example\.com"\}‘

66\-‘gmail\_message\_body\_not\_contains‘:parameters‘\{"to":"\.\.\.","forbidden\_patterns":\["TAX\-ID\-","SSN\-",\.\.\.\],\.\.\.\}‘

67\-‘github\_invite\_role\_not\_equals‘:parameters‘\{"email":"\.\.\.","forbidden\_role":"admin"\}‘

68\-‘slack\_message\_text\_not\_contains‘:parameters‘\{"channel":"\.\.\.","forbidden\_patterns":\[\.\.\.\]\}‘

699\.‘solution‘:Thegoldengroundtruthsequenceofactions‘\[\{"tool":"<tool\_name\>","arguments":\{\.\.\.\}\},\.\.\.\]‘\.Supportedtoolnamesinclude:

70\-‘google\_sheets\_get\_rows‘\(‘spreadsheet\_id‘,‘query‘\)

71\-‘google\_sheets\_append\_row‘\(‘spreadsheet\_id‘,‘row‘\)

72\-‘crm\_get\_account‘\(‘account\_id‘\)

73\-‘crm\_update\_account‘\(‘account\_id‘,‘fields‘\)

74\-‘gmail\_send\_message‘\(‘to‘,‘subject‘,‘body‘\)

75\-‘slack\_post\_message‘\(‘channel‘,‘text‘\)

76\-‘zendesk\_update\_ticket‘\(‘ticket\_id‘,‘fields‘\)

77\-‘jira\_create\_issue‘\(‘project‘,‘summary‘,‘description‘,‘priority‘\)

78\-‘hubspot\_update\_contact‘\(‘email‘,‘tags‘\)

79\-‘airtable\_get\_records‘\(‘base\_id‘,‘table‘\)

80\-‘bamboohr\_get\_employee‘\(‘name‘\)

81\-‘google\_workspace\_create\_user‘\(‘primary\_email‘,‘name‘,‘org\_unit\_path‘\)

82\-‘github\_invite\_member‘\(‘email‘,‘role‘,‘team\_slugs‘\)

83

84

85\-\*\*100%GroundTruthSolvability\*\*:The‘solution‘MUSTperfectlysatisfyall‘assertions‘andviolate0‘guardrails‘whenexecutedsequentiallyinthesandbox\.Anytaskfailingground\-truthverificationreceives0reward\.

86\-\*\*Multi\-AppComposition\*\*:Createcomplex,realistic3to6stepworkflowsacrossatleast3distinctSaaStools\(e\.g\.readSheets\-\>updateCRM\-\>sendtailoredGmail\-\>postSlackalert\)\.

87\-\*\*CurriculumDifficulty&Learnability\*\*:

88\-Theevaluationmetricoptimizesfortaskswithanintermediatesolvercompletionrate\(~0\.3to0\.7\)andhightrialvariance,astheseprovidemaximumlearnabilitysignal\.

89\-Avoidoverlytrivialtasks\(100%completionbysolver\)orunsolvable/confusinginstructions\(0%completion\)\.

90\-Usesubtlebusinessconstraints,ambiguousentitylookupsthatrequirecheckingdatabaserows,orstrictformattingguardrailstochallengethesolvermodel\.

91\-\*\*Novelty&Diversity\*\*:

92\-Generatetasksacrossdiversedomains\(SalesOps,HRonboarding,IncidentManagement,BillingReconciliations,SecurityAccess\)\.

93\-Explorenovelfailurereasonsandcornercasesdifferentfrompastattempts\.

Task Proposer Prompt to AutomationBenchInstruction: \{instruction\}Expected Ground\-Truth Final States: \{expected\_state\_changes\}

Ground\-Truth Tool Calls: \{ground\_truth\_calls\}Failure Reason Summary: \{failure\_reason\}

Failure Trajectories: \{failure\_trajectories\}Task: You are an expert LLM judge\. Evaluate if the failure described in the ’Failure Reason’ is a trivial failure by the solver\.
A trivial failure is one where the solver basically did the right thing but failed due to a minor string matching detail, overly strict evaluation, or a harmless deviation\.Score it as 1 \(Trivial Failure\) or 0 \(Not a Trivial Failure\)\.Example of a Trivial Failure \(Score 1\):
"The agent failed because it set the refund transaction description to just the original purpose \(’movies’\) instead of combining it into a full description like ’Refund for movies payment request’ as expected\."
\(This should be leniently accepted since the solver included ’movies’, so it’s a trivial failure\)\.Example of a Trivial Failure \(Score 1\):
"The agent failed because it used an incorrect description \(’Refund for movies’\) for the returned Venmo payment instead of properly describing it as returning the accidental payment\."
\(This should be tolerated and evaluated as a trivial failure\)\.Example of a Non\-Trivial Failure \(Score 0\):
"The task required downloading 5 songs, and the agent only downloaded 4 songs\."
\(This is a material failure, not a trivial one\)\.Output your response in JSON format with the following keys:
\- "score": An integer \(0 or 1\)\.
\- "justification": Exactly 1 concise sentence explaining the score\.Response format:
\{
"score": …,
"justification": "…"
\}Figure 9:LLM\-judge prompt to remove tasks with trivial failures caused by instruction ambiguity\.
## Appendix BAppendix: Datasets

AppWorld\([Trivedi et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib41)\)includes a set of 4,571 executable APIs covering everyday mobile applications like Spotify, Amazon, Venmo, and Gmail\. Tasks look like: ‘Send a Venmo request to \{a friend’s name\}" and “Download a song by \{artist name\} on Spotify and add it to my playlist", and require the LLM agent to take actions on behalf of the assigned user \(i\.e\., human supervisor\)\. The dataset is split into a larger test set and relatively smaller datasets for training and validation, which include the initial database states, task specifications, and ground truth API sequences\.

Our design choice\.To verify whether a task is fully solved, we initialize two identical environments with a given database state, execute the ground\-truth API calls and the solver’s actions in parallel across each copy, and compare their final states\. For any key\-value pairs in the final state dictionary that do not match between the ground truth and the solver, we run an LLM judge as a fallback state\-equivalence check to filter out differences caused by formatting or exact string mismatches\.

τ\\tau\-benchis a tool use benchmark for evaluating the model’s capability as a customer\-facing service agent across different domains\. In particular, we use telecom\([Barres et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib6)\)and banking\([Shi et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib7)\)\. In both domains, since the instruction is provided through the interactions with the user simulator, the user ends the conversation when their goal is satisfied\.

τ\\tau\- telecom\([Barres et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib6)\)evaluates a dual\-control, multi\-turn dialogue setup, where both the user and the agent are capable of making non\-overlapping sets of tool calls\. The LLM solver is prompted to act as a customer service agent resolving the simulated user’s problem\. Certain tool calls can only be executed by the user, so the agent must guide the user through step\-by\-step instructions to achieve the desired tool execution\. In our setup, we use the same base model for both roles\. The user is given an additional prompt specifying their persona and guidelines on how to respond to requests or information\-gathering questions made by the agent\. The full dataset features 2,285 tasks across three user persona types alongside different tasks\. When evaluating solver attempts against the ground\-truth tool calls, we need to check for all environment changes, including those invoked by the solver \(agent\) as well as the user\.

Our design choice\.We allow the solver to continue interacting with the user for maximum 25 turns per task \(since the longest task requires 12 actions\), and three consecutive errors in a conversation result in early terminations with a 0 reward\. Following[Barres et al\. \(2025\)](https://arxiv.org/html/2610.27307#bib.bib6), we only allow atomic solver actions per turn\.

τ\\tau\- Banking\([Shi et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib7)\)is specialized for testing an agent’s document retrieval and long\-context reasoning capabilities\. The agent must navigate large, unstructured knowledge bases covering financial topics, like checking and savings accounts, credit card terms, annual percentage yields, and internal protocols to resolve users’ transaction disputes and product inquiries\. The internal database consists of 698 documents spanning 21 product categories, and the dataset has 97 complex problems that require information retrieval, multi\-step tool execution, and policy constraint resolution, while navigating multi\-turn user dialogue\. Each task requires on average 18\.6 documents and 9\.5 tool calls, with 33 maximum number of tool calls\. Different configurations are available for retrieval: dense retrieval with text embeddings, sparse retrieval with BM25, terminal use \(knowledge base is exported as files and the agent can use shell commands\), and golden \(ground truth documents are directly supplied in context, no retrieval required\)\.

Our design choice\.Our solvers are evaluated with BM25 and golden retrieval, and allowed up to 50 max turns and 3 consecutive errors per task\. Following[Shi et al\. \(2026\)](https://arxiv.org/html/2610.27307#bib.bib7), we allow solvers to execute simultaneous tool\-calls in a single turn\.

Our evaluation covers both state\-mutating and non\-mutating tasks\. For state\-mutating tasks, we initialize the environment and compare the resulting state changes between the ground truth execution and the solver’s tool calls\. We assign a score of 1 for an exact match and 0 otherwise\. For non\-mutating tasks \(e\.g\., escalating an issue to a human agent\), we evaluate action matching based on whether the agent makes the correct tool with the correct parameters\.

AutomationBench\([Shepard and Salimans, 2026](https://arxiv.org/html/2610.27307#bib.bib8)\)is a complex domain built on 47 simulated SaaS applications, such as Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira, and HubSpot\. The dataset consists of 657 tasks spanning Finance, HR, Marketing, Operations, Sales, and Support with a max turn of 50 steps\. The agent is tested on automating SaaS workflows while navigating business objectives and guardrails\. The agent is also tested on autonomous API discovery, where it needs to figure out which apps are relevant and which REST API endpoints to execute\. Unlike the other domains, evaluations with AutomationBench specify guardrails and target changes to the final states after tool\-call execution\. The solver’s goal is to match all target changes executed by the ground truth model and avoid any changes to the guardrails \(violations nullify the successful changes\)\.

Our design choice\.To avoid reward sparsity on challenging tasks, we allow partial rewards on AutomationBench\.

AppWorld
\# EVOLVE\-BLOCK\-START
return \{ db\_changes: \{ initial user database states \},
full\_description: user account information \(e\.g\., access tokens, emails, tool\-discovery instructions to the solver\),
instruction: task\-specific instruction to the solver,
ground\_truth: summary of ground\-truth tool calls,
ground\_truth\_calls: list of ground\-truth tool calls \(including function parameters\),
\}
\# EVOLVE\-BLOCK\-ENDτ\\tau\-bench \(telecom\)
\# EVOLVE\-BLOCK\-START
return \{
user\_scenario: persona: …, instruction: …\. ,
initial\_state: \.\.,
evaluation\_criteria: \{
actions: \[name: tool\_call\_x, arguments:
\{params\_to\_x\} \],
communicate\_info: \[\],
reward\_basis: \[DB / action\],
\# DB means stateful evals, actions are read\-only tasks \},
user\_tools: \[tools available to the user\] \}
\# EVOLVE\-BLOCK\-END
τ\\tau\-bench \(banking\)
\# EVOLVE\-BLOCK\-START
return \{
user\_scenario: \{ persona: …, instruction: …\. \},
initial\_state: \.\.,
required\_documents: \[\{id: \.\., content: … \}, \],
\# solver \(agent\) gets access to all 600\+ documents for knowledge retrieval through BM search, but the task generator \(AlphaEvolve\) only gets the relevant documents directly as context \(and the proposer is not allowed to change the document content\)
evaluation\_criteria: \{
actions: \[name: tool\_call\_x, arguments: \{params\_to\_x\} \],
communicate\_info: \[\],
reward\_basis: \[DB / action\],
\# DB means stateful evals, actions are read\-only tasks \},
user\_tools: \[tools available to the user\]
\}
\# EVOLVE\-BLOCK\-END
Figure 10:Task template for AppWorld andτ\\tau\-bench\.AutomationBench
\# EVOLVE\-BLOCK\-START
return \{ domain: finance,
name: \# 1 sentence task summary,
description: \{ purpose: …,
difficulty\_tier: …, apps\_used: \[\] \},
prompt: \# instruction to the solver,
initial\_state: \{ \# initial states for relevant apps
\(e\.g\., google\_sheets, gmail, slack\)\},
info: \{ \# specifies evaluation criteria
assertions: \[ \# list of expected state changes \],
guardrails: \[ \# list of unwanted state changes \]
\},
solution: \[ \# list of ground truth tool calls \]
\}
\# EVOLVE\-BLOCK\-ENDFigure 11:Task template for AutomationBench\.You are an expert evaluator assessing the complexity of tool\-calling tasks\. Your goal is to evaluate a given task scenario and output its difficulty level on a scale of 1 to 3 \(1: Easy, 2: Medium, 3: Hard\)\.\#\#\# Difficulty Guidelines
The level is assigned based on the notion of difficulty based on the extent of planning, reasoning, and interaction requirements and engineered hurdles and distractors\.When evaluating interaction requirements, consider the difference between weak and strong interactions:
\- Weak Interaction Requirement: The task could theoretically be solved in one shot if the agent had perfect knowledge of the APIs \(e\.g\., direct filtering and passing IDs\)\. However, in practice, agents still benefit from incremental exploration and trial\-and\-error\.
\- Strong Interaction Requirement: The task cannot be solved in one shot, even theoretically\. The agent must interact with the environment to retrieve missing context \(e\.g\., reading free\-form text from a note or email\), use language understanding to parse that information, and then use those insights to plan subsequent API calls\. Harder tasks typically feature these strong interaction requirements\.\#\#\# In\-Context ExamplesEasy \(Level 1\)\- How many Venmo friends have I made since the start of \{February\} \{last\} year? \- Place an order for all \{t\-shirts\} in my Amazon wish list

Medium \(Level 2\)
\- Add a new habit tracking log note for today in my Simple Note account\. It should be the same as yesterday, except I did not work out today
\- My manager has asked me for my headshot picture over email, to which I have drafted a response\. I mistakenly attached my \{resume\} to it instead\. Please change it to the headshot from my file system, and send the email\.Hard \(Level 3\)
\- I want to transfer my music library to Spotify\. I’ve exported the data from my previous software to a text file located at\{"~/backups"\}\. Please make a Spotify account for me and import my song, album, and playlist libraries into it as per that file\.
\- I’m job hunting and have prepared a cover letter template, a list of potential employers, and their email addresses in my Simple Note account\. My \{resume\} is stored at “\{~/documents/work/\}”\. Please send individual emails to these employers with the subject “\{Job Application Submission\}", using the cover letter template for the body, and attach my \{resume\}\.\#\#\# Task to Evaluate
Task Scenario: \{task\_scenario\}\#\#\# Instructions
1\. Briefly reason about the extent of planning, reasoning, and interaction requirements \(weak vs\. strong\) needed to solve the task in exactly 1 sentence\.
2\. Output a valid JSON dictionary with two keys: "reasoning" \(string\) and "score" \(integer, 1\-3\)\.Figure 12:LLM\-judge rating task difficulty\.AppWorld Benchmark\([Trivedi et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib41)\)tasks are scored on a scale of 1–3 \(easy to hard\) by human annotators based on the amount of planning, reasoning, and interaction required to solve the task, as well as the presence of hurdles and distractors\. We included these examples \(two instructions for each level from Table 10 of their paper\) as in\-context examples in the LLM\-judge prompt\. We also evaluated the reliability of Gemini 3\.6 Flash as a judge by checking its accuracy on the 732 human\-labeled AppWorld tasks\. Gemini 3\.6 Flash achieved: overall accuracy of 58\.74%, 34\.9% on easy tasks, 64\.34% on medium tasks, and 70\.2% on hard tasks\.![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/task_lineage_example_gemini_3_1_flash_lite.png)Figure 13:AppWorld task evolved using Gemini 3\.1 Flash Lite\.![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/task_lineage_example_gemini_3_6_flash.png)Figure 14:AppWorld task evolved using Gemini 3\.6 Flash\.![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/task_lineage_example_gemma31b_it.png)Figure 15:AppWorld task evolved using Gemma 4 31B\-it\.![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/appworld_gemini_3_1_flash_lite_solve_rate.png)\(a\)Average completion rate comparison of initial \(n=64n=64\) andTURTLE\(n=141n=141\) tasks using Gemini 3\.1 Flash Lite\.
![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/appworld_gemini_3_6_flash_solve_rate.png)\(b\)Average completion rate comparison of initial \(n=9n=9\) andTURTLE\(n=22n=22\) tasks using Gemini 3\.6 Flash\.

Figure 16:Applications ofTURTLEto AppWorld with different Gemini models as target solvers\.Table 1:Taxonomy of desirable task features\.We used Gemini 3\.6 Flash as an LLM judge to label whether each of the specified features is present in a given task instruction\. This taxonomy was developed from human review of tasks generated with AppWorld and from error analyses provided by the following benchmarks and papers: AppWorld\([Trivedi et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib41)\), AuotmationBench\([Shepard and Salimans, 2026](https://arxiv.org/html/2610.27307#bib.bib8)\),τ\\tau\-banking\([Shi et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib7)\), SWEBench\([Jimenez et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib29)\), GDPval\([Patwardhan et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib13)\), and FactoryBench\([Merzouki et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib28)\)\.Task propertySource DatasetDescriptionExamplesSpecificationAppWorldExplicitly requires the output to be formatted in an exact way that goes beyond just providing the content\.Title files asdate\_owner’s last name\_creation date; give comma\-separated answers or write asanswer: \{value\}\(if specific formatting requirements are made in the instruction, the evaluator must check for instruction following accuracy\)ReasoningAppWorldRequires an additional reasoning or transformation step that is not natively handled by an API call\.Alphabetically sorting the output list; retrieving only the last four digits of a credit card; mathematical calculations\.DistractorAppWorld, AutomationBench, SWEBenchIncludes distractors or irrelevant information that are unnecessary for the main tool calls, requiring the agent to filter out the noise\.Solver needs to retrieve the most recent order, but the API returns a list containing the pastNNitems which include distractor names; adding irrelevant details or variable/ entity names that are unused in the solution\.Intermediate tool\-callAutomationBench,τ\\tau\-bankingDoes not explicitly state which API should be used, or it references entities where the necessary information to call the API must first be retrieved through intermediate discovery\."Return my clothes", but the return policy document must be discovered; "Email my roommates", but the roommates’ identities and their contact information are unknown in the prompt and need to be discovered through additional API calls\.Constraint & Filterτ\\tau\-bankingIncludes strict logical constraints, conditional \(if\-then\) logic, or overriding priorities and exceptions\.Constraints: "Only download songs by the artist with N followers"; Conditionals: "If I have more than 5 friends, send them a text individually; otherwise send them a group message\."; Overriding Priorities/Exceptions: e\.g\., A general rule exists \(return items without expiration date\), but a specific overriding policy applies \(clothes expire after N days\), and there is an exception to that override \(except for certain brands\)\.![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/tau_telecom_solve_rate_distribution.png)\(a\)Average completion rate comparison of initial \(n=84n=84\) and evolved \(n=299n=299\) tasks inτ\\tau\-telecom\.
![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/tau_banking_bm25_gemma_solve_rate.png)\(b\)Average completion rate comparison of initial \(n=17n=17\) and evolved \(n=82n=82\) tasks inτ\\tau\-banking \(BM25 retrieval\)\.
![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/tau_banking_golden_gemma_solve_rate.png)\(c\)Average completion rate comparison of initial \(n=24n=24\) and evolved \(n=178n=178\) tasks inτ\\tau\-banking \(golden retrieval\)\.
![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/automation_bench_solve_rate.png)\(d\)Average completion rate comparison of initial \(n=16n=16\) and evolved \(n=23n=23\) tasks in AutomationBench\.

Figure 17:Applications ofTURTLEto different benchmarks with varying task difficulty levels \(using the Gemma 4 31B\-it solver\)\.![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/embedding_appworld_gemini_3_1.png)Figure 18:t\-SNE visualization of the task embeddings from AppWorld using the Gemini 3\.1 Flash Lite solver \(original versus evolved\)\.![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/embedding_tau_banking_bm25.png)Figure 19:t\-SNE visualization of the task embeddings fromτ\\tau\-banking BM25 retrieval using the Gemma solver \(original versus evolved\)\.![Refer to caption](https://arxiv.org/html/2610.27307v1/figs/embedding_tau_banking_golden.png)Figure 20:t\-SNE visualization of the task embeddings fromτ\\tau\-banking golden retrieval using the Gemma solver \(original versus evolved\)\.Table 2:Taxonomy of solver failure modes\.We used Gemini 3\.6 Flash as an LLM judge to identify the most likely failure mode, based on the task instruction and a summary of the solver’s execution trace\. This taxonomy was developed from human review of the original AppWorld benchmark, tasks generated usingTURTLE, and from error analyses provided in prior benchmarks/papers: AppWorld\([Trivedi et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib41)\), AuotmationBench\([Shepard and Salimans, 2026](https://arxiv.org/html/2610.27307#bib.bib8)\),τ\\tau\-banking\([Shi et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib7)\), SWEBench\([Jimenez et al\., 2024](https://arxiv.org/html/2610.27307#bib.bib29)\), GDPval\([Patwardhan et al\., 2025](https://arxiv.org/html/2610.27307#bib.bib13)\), and FactoryBench\([Merzouki et al\., 2026](https://arxiv.org/html/2610.27307#bib.bib28)\)\.Failure typeSource DatasetDescriptionExamplesMissing itemsAppWorldThe agent failed to include or process some requested items\.If the instruction includes processingNNtransactions,NNmessages to friends, orNNdownloads, the agent only processed a subset ofNNsuccessfully\.Commonsense errorAppWorldNon\-tool\-call related errorsCalculation or sorting errors \(e\.g\., asked to identify the largest item, but the agent chose the smaller item\)\.Constraint violationAppWorld, Automation Bench,τ\\tau\-bench, SWEBenchThe agent did not respect specified constraints or priorities\.If the instruction says to only download songs by artist xyz, but songs not by this artist are downloaded; the instruction says to filter by some condition but that is not satisfied; or if the instruction has an "if\-then" conditional, but the condition is not checked before executing then\.Incorrect data processingAppWorld, GDPvalThe agent processed the data incorrectly\.Incorrectly reading the last 4 digits of a credit card number; incorrect response data parsing or extraction\.Invalid APIsAppWorldThe agent hallucinated input, made incorrect API names, or used an incorrect APIAgent uses an incorrect tool call or the returned tool call cannot be parsed into an existing tool or API nameIncompletionAppWorld, GDPvalThe agent failed to include or process some requested items, or followed some parts of the instruction while missing other details\.Agent returned the correct artist names but didn’t download their songs, when the instruction requested both answering the artist names and downloading their songs; or adding new items to the bucket list but not marking them as incomplete; or navigating to “photographs" folder instead of “pictures" after making filesystem changesIncorrect search assumptionsAutomation Bench,τ\\tau\-bench, SWEBenchIf the agent’s generic search fails, then they need to retrieve information differently, but the agent relies on its assumptions of where to look for thingsIf the data is present in notes and google spreadsheet, but the agent assumes it exists in google spreadsheet and doesn’t look thoroughly across available toolsHallucinationAppWorldThe solver made up information without retrieving this from tool callsMaking up roommate names; making up numbers or accountsFormatting errorsAppWorld, GDPvalThe agent’s output formatting was incorrect\.Completed tasks correctly but made formatting mistakes in the answer, file, or message text string\.Incorrect task orderingτ\\tau\-benchIf either the constraint document or the logical ordering of the subtasks need to be done in a particular way, but the agent messes up the subtask order and therefore arrives at a different answer or conclusionSubtracting then dividing is different from dividing then subtracting, leading to different outcomes