面向数据分析湖编码智能体的无监督工具进化

# Unsupervised Tool Evolution for Coding Agents in Data-Lake Analytics
Source: [https://arxiv.org/html/2610.27327](https://arxiv.org/html/2610.27327)
\\uselogo

Yurong LiuAffiliation:\\thepaAffiliation:New York UniversityDeqing FuAffiliation:University of Southern CaliforniaAffiliation:\\thepaManav GargAffiliation:\\thepaSravan Babu BodapatiAffiliation:\\thepaFatma OzcanAffiliation:\\thepaAlon Y\. HalevyAffiliation:\\thepaJiani ZhangAffiliation:\\thepa

###### Abstract

Coding agents can answer analytical questions over a data lake by discovering, cleaning, and joining the relevant data\. However, they discard this work after each question and redo it from scratch for the next one\. Tool evolution, which automatically builds and refines reusable tools across tasks, is a natural fit for this repetition\. Since a ground\-truth label for a data\-lake question requires an expert to solve it first, we consider the unsupervised setting\. We proposeTEAL\(ToolEvolution forAnalytics over dataLakes\), which evolves alibrary of tools and usage skillsfor a given Worker agent without labels\. Two agents propose tools from complementary evidence, an Explorer from the data lake itself and a Critic from Worker trajectories\. A proposed change is kept only if the Worker adopts the tool on a rerun without extra steps and any changed answer is supported by trajectory evidence\. We also introduceTEAL\-Q, which generates synthetic questions for tool evolution when real ones are unavailable\. We evaluate on KramaBench and LakeQA with four Worker LLMs under step budgets, which cap the number of steps the Worker may take per question\. With a tight budget of 10 or 15 steps,TEALimproves every Worker on both benchmarks, lifting Sonnet\-5 at 15 steps by 9\.3 points on KramaBench and 21\.7 points on LakeQA\. A tighter budget also yields better feedback for evolution\. For example, Sonnet\-5’s LakeQA library evolved at 20 steps and deployed unchanged at 30 steps scores 68\.3, above both the baseline \(59\.6\) and a library evolved directly at 30 steps \(64\.2\)\. Additionally, a library evolved on synthetic questions performs comparably to one evolved on real questions\. These results show that useful tools can be evolved without labels, and hopefully inspire future work on what to evolve and how to obtain feedback in unsupervised agent evolution\.

## 1Introduction

A data lake\([Nargesian et al\., 2019](https://arxiv.org/html/2610.27327#bib.bib32);[Hai et al\., 2023](https://arxiv.org/html/2610.27327#bib.bib17)\)stores a large, heterogeneous collection of tables, documents, and raw files\. Answering analytical questions over a data lake requires discovering relevant data\([Fernandez et al\., 2018](https://arxiv.org/html/2610.27327#bib.bib33);[Castelo et al\., 2021](https://arxiv.org/html/2610.27327#bib.bib34)\), transforming messy records into structured representations\([Kandel et al\., 2011](https://arxiv.org/html/2610.27327#bib.bib36);[Li et al\., 2023](https://arxiv.org/html/2610.27327#bib.bib37)\), and integrating data across sources\([Freire et al\., 2025](https://arxiv.org/html/2610.27327#bib.bib22);[Zhang et al\., 2026b](https://arxiv.org/html/2610.27327#bib.bib21)\)\. Coding agents equipped with a bash shell and a file editor\([Wang et al\., 2024a](https://arxiv.org/html/2610.27327#bib.bib11);[Wang et al\., 2025](https://arxiv.org/html/2610.27327#bib.bib18)\)can automate this workflow\([Narayan et al\., 2022](https://arxiv.org/html/2610.27327#bib.bib38);[Fernandez et al\., 2023](https://arxiv.org/html/2610.27327#bib.bib39)\)\. However, they typically treat each question over the same data lake in isolation and discard every parser, schema check, and join script as soon as the question ends\.

Consider an analyst asking which metropolitan area accounts for the largest proportion of its state’s fraud reports\([Lai et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib12)\)\. The answer lies in 52 per\-state CSV files of metropolitan\-area counts with irregular headers, hidden among the data lake’s thousands of files next to similar national and state summaries\. Without a catalog or index, table granularity cannot be inferred from filenames, so a coding agent may pick a national summary table that lacks state breakdowns\. Even when the agent locates the 52 state files, opening and cleaning them one by one exhausts a 30\-step budget and fails sooner under a 15\-step budget\. When the next question comes, the agent repeats the same manual exploration and cleaning from scratch\.

Database systems avoid repeated scans of raw data by maintaining catalogs\([Halevy et al\., 2016](https://arxiv.org/html/2610.27327#bib.bib27)\), indexes\([Graefe, 2024](https://arxiv.org/html/2610.27327#bib.bib30)\), and views\([Halevy, 2001](https://arxiv.org/html/2610.27327#bib.bib28)\)\. Coding agents can follow the same principle with reusable*tools*that replace repeated step\-by\-step exploration with a single call\. Some of these tools are backed by precomputed*data artifacts*, such as catalogs and indexes, and*usage skills*tell the agent when and how to invoke each tool\.

However, existing methods cannot automatically generate such tools, for two reasons\. First, catalog and ontology formats\([Zhang et al\., 2026a](https://arxiv.org/html/2610.27327#bib.bib40)\)store descriptive metadata rather than executable tools for cleaning and querying data, whereas code tool generators\([Wang et al\., 2024b](https://arxiv.org/html/2610.27327#bib.bib15);[Li et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib1)\)build tools for the task at hand, not artifacts that pay off only across many questions\. Second, supervised tool generators\([Cai et al\., 2024](https://arxiv.org/html/2610.27327#bib.bib13)\)and self\-evolving ontologies\([Chong et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib31)\)rely on ground\-truth validation labels, which are costly to obtain in data\-lake analytics because verifying even one answer requires an expert to manually discover, clean, and join the underlying data sets first\. Without ground\-truth labels, keeping every tool that runs without a runtime error\([Li et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib1)\)is unreliable because a tool can execute cleanly while reading a table at the wrong granularity, dropping rows during parsing, or joining tables on unaligned keys\.

To address these challenges, we introduceTEAL\(ToolEvolution forAnalytics over dataLakes\), an unsupervised framework that evolves a*library*for a fixed Worker agent\. The library consists of executable tools, some backed by precomputed data artifacts, and usage skills\. At test time, the library is frozen and provided to the Worker to answer new questions\.

### 1\.1Contributions

##### Tool evolution from data exploration and Worker trajectories\.

TEALbuilds the library offline from two complementary channels, which differ in the evidence they draw on\. An*Explorer*agent inspects the data lake without seeing any question or trajectory and builds tools that a domain expert would prepare for the data lake, for example search and lookup tools backed by precomputed catalogs or cleaned tables\. A*Critic*agent examines the Worker’s trajectories on training questions to find where the Worker wastes steps or makes errors, and creates or revises tools accordingly\. A*Skill Writer*agent then updates the usage skills to reflect every implemented change\.

##### Tool verification without ground\-truth answers\.

Because a newly proposed tool can still contain logic bugs or be ignored by the Worker, we introduce a rerun verification stage that accepts or rejects each change without answer labels\.TEALreruns the Worker with the updated library on the questions that motivated the change\. The change is kept only if the Worker uses the changed tool, still finishes without extra steps, and, whenever its answer changes, the trajectory evidence supports the new answer over the old one\.

##### Synthetic question generation for tool evolution\.

When real user questions are scarce for a data lake, we introduceTEAL\-Qto generate questions that can be used byTEAL, using a small set of seed questions as demonstrations\.TEAL\-Qfirst drafts a question from a cluster of related files, then hardens it over several revisions by adding data processing and reasoning demands, such as multi\-table joins, while ensuring it is answerable\.

##### Empirical evaluation across models and step budgets\.

We evaluateTEALon KramaBench\([Lai et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib12)\)and LakeQA\([Wang et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib4)\)with four Worker backbones \(Gemini\-3\.8\-Flash, Gemini\-3\.1\-Pro, Sonnet\-5, and Opus\-4\.8\) under varying step budgets, i\.e\., the maximum Worker steps per question\. We make three main observations\. First, under tight 10\- and 15\-step budgets,TEALimproves accuracy for every Worker on both benchmarks\. With Sonnet\-5 at 15 steps, it gains 9\.3 points on KramaBench and 21\.7 points on LakeQA, and also outperforms the execution\-only tool evolution method\([Li et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib1)\)\. Second, libraries evolved under a tighter budget transfer to looser budgets\. Sonnet\-5’s LakeQA library evolved at 20 steps scores 68\.3 at 30 steps, above both the baseline \(59\.6\) and a library evolved at 30 steps \(64\.2\), suggesting that tighter budgets give better feedback for evolution\. Third, evolving on 50 synthetic questions fromTEAL\-Qperforms comparably to evolving on real questions, and our unsupervised rerun\-based verification matches oracle ground\-truth verification\. The library evolved in the OpenHands harness also transfers unchanged to Google ADK, Codex CLI, and Prime Agent, improving accuracy by up to 16\.3 points\.

## 2Related Work

##### Tools and Agents for Data Analytics\.

Data analytics over heterogeneous sources involves discovering relevant files\([Paton et al\., 2023](https://arxiv.org/html/2610.27327#bib.bib20)\), preparing and integrating their contents\([Liu et al\., 2024](https://arxiv.org/html/2610.27327#bib.bib35)\), and performing analysis\([Nam et al\., 2025](https://arxiv.org/html/2610.27327#bib.bib19);[Yang et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib43)\)\. Prior work supports this with resources that humans prepare in advance, including data artifacts such as metadata catalogs and relationship indexes\([Halevy et al\., 2016](https://arxiv.org/html/2610.27327#bib.bib27);[Fernandez et al\., 2018](https://arxiv.org/html/2610.27327#bib.bib33)\)or materialized views and indexes\([Agrawal et al\., 2000](https://arxiv.org/html/2610.27327#bib.bib29)\), and tools such as user\-defined plugins\([Qiao et al\., 2023](https://arxiv.org/html/2610.27327#bib.bib41)\)or a fixed toolset for searching and inspecting the lake\([Wang et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib4)\)\. Recent work lets agents build such resources themselves\. In EvoOntology\([Chong et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib31)\), a builder agent constructs and refines an ontology layer that data agents query through tools, but a refinement is kept only if the score on a held\-out labeled split improves\. Closest to our setting, DataCOPE\([Qiu et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib42)\)discovers skills without labels by scoring sampled trajectories through answer agreement, but its skills are procedural guidance, not callable tools backed by persistent artifacts\. We study fully automatic tool evolution, where no human writes tools, curates artifacts, or labels answers, and agents decide from data exploration and unlabeled trajectories which tools to build and how to guide their use with skills\.

##### Tool Evolution for LLM Agents\.

Automatic tool creation for LLMs began with math and science questions that a single program can answer\. A model writes a helper function, verified against labeled examples\([Cai et al\., 2024](https://arxiv.org/html/2610.27327#bib.bib13)\)or by execution tests\([Lu et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib6);[Shen et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib7)\), and reuses it across questions of the same type\. Agent systems extended this to tools built during long\-horizon tasks\([Xia et al\., 2025](https://arxiv.org/html/2610.27327#bib.bib9)\)and usually stored for later use\. These agents create a tool when a task cannot proceed without one, such as a transcript downloader or a web page fetcher, so the tool is exercised immediately, and acceptance reduces to whether it executes\([Li et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib1)\), whether the environment reflects completion\([Wang et al\., 2023](https://arxiv.org/html/2610.27327#bib.bib14)\), or whether the agent judges its output plausible\([Qiu et al\., 2025](https://arxiv.org/html/2610.27327#bib.bib5);[Ding et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib8)\)\. In this work, we study unsupervised tool evolution for data\-lake analytics, where neither assumption holds\. Here tools rarely fill a gap, since the Worker is not blocked without them, and they cannot be credited for a correct answer without labels\.TEALtherefore verifies tools by the Worker’s behavior on a rerun, and we study how to make unsupervised tool evolution more effective, in particular by evolving under tighter step budgets\.

## 3Method

### 3\.1Problem Formulation

##### Coding agents and tools\.

We consider coding agents equipped with a shell and a file editor, which enable them to read and modify files and execute code\. Beyond these basic capabilities, we define a*tool*as a named, callable program added to the agent’s action space and retained across tasks\. Once implemented and debugged, a tool can be invoked in a single action without reconstructing its procedure\. A tool may also rely on persistent artifacts, such as indexes or cached metadata, that are built once and reused across tasks\.

##### Tool evolution for data\-lake analytics\.

Given a natural\-language question over a data lake, a coding agent, which we call the*Worker*, autonomously discovers the relevant files and performs the data preparation, integration, analysis, and reasoning needed to answer the question under a step budget\. Without access to ground\-truth answers, a tool evolution system observes the Worker’s trajectories on training questions and produces a*library*ℒ=\(𝒯,S\)\\mathcal\{L\}=\(\\mathcal\{T\},S\), consisting of a set of tools𝒯\\mathcal\{T\}and a skill fileSS, a natural\-language document that tells the Worker when and how to use the tools\. We then freeze the library and measure the Worker’s accuracy with it on a disjoint set of test questions over the same data lake\.

Figure 1:TheTEALevolution loop\. The Worker first solves the questions with the current library \(Stage 0\)\. The Explorer proposes tools for the data lake as a whole \(Stage 1\), and the Critic proposes targeted fixes from the Worker’s trajectories \(Stage 2\)\. The Skill Writer then updates the usage guidance \(Stage 3\), and a rerun of the Worker decides which changes to keep \(Stage 4\)\. Robot icons mark LLM roles\. No stage uses ground\-truth answers\.

### 3\.2Unsupervised Tool Evolution viaTEAL

Alongside the*Worker*that answers questions over the data lake,TEALcoordinates five LLM agents to evolveℒ\\mathcal\{L\}\. The*Explorer*inspects the data lake to propose data\-specific tools and precomputed data artifacts\. The*Critic*diagnoses execution bottlenecks and errors in the Worker trajectories to propose targeted tool changes\. The*Toolsmith*implements proposals from the Explorer and Critic as executable code and validates them with unit tests\. The*Skill Writer*maintainsSSthat guides when and how to invoke each tool in𝒯\\mathcal\{T\}, and the*Analyst*compares original and rerun trajectories during verification whenever their final answers differ\.

#### 3\.2\.1The tool evolution pipeline

During offline tool evolution, which we refer to as training even though no model weights are updated,TEALprocesses the training questions in batches and runs the five stages in Figure[1](https://arxiv.org/html/2610.27327#S3.F1)on each batch to iteratively updateℒ\\mathcal\{L\}\.

InStage 0: Trajectory Collection, the Worker executes a batch of questions with the current libraryℒ\\mathcal\{L\}\. Each trajectory is recorded together with a summary of its tool usage as evidence for later stages\. The summary covers which tools were used, which were unused, and where the Worker took many consecutive steps without any tool call\. For example, on a question over a large directory or an Excel workbook, the summary flags that the Worker spent seven consecutive bash steps running file searches without calling any library tool and exhausted its step budget\.

InStage 1: Expert Exploration, an Explorer agent inspects the data lake andℒ\\mathcal\{L\}without access to task questions or Worker trajectories\. It*acts as a domain expert*preparing the tools for the data lake as a whole rather than for any specific task, and proposesadd toolandupdate tooloperations\. It can also create tools backed by persistent artifacts that precompute information for reuse across tasks\. A Toolsmith agent implements these proposals and checks them with executable unit tests\. Unlike the other stages, this stage runs only once everyEEbatches\. For example, after surveying thousands of raw CSV and Excel files, the Explorer proposes a dataset search tool backed by a precomputed SQLite catalog of file formats, column headers, and sample rows\.

InStage 2: Trajectory Criticism, a Critic agent reviews the evidence collected in Stage 0 and the current tool set𝒯\\mathcal\{T\}, including those just added or updated by the Explorer, to proposeupdate tooloperations for defects in used tools,promote tooloperations for unused tools that could have helped, andadd tooloperations for capability gaps that the current tools do not cover\. Theadd toolandupdate toolproposals are implemented by the Toolsmith, and thepromote toolproposals are passed to the next skill update stage\. For instance, when trajectories show the Worker failing on Excel sheets whose headers do not start on the first row, the Critic proposesadd toolfor a header\-aware workbook loader, or proposesupdate toolwhen an existing catalog tool misses yearly editions of a table\.

InStage 3: Skill Update, a Skill Writer agent updates the skill fileSSby integrating the implemented changes from the Explorer and the Critic and thepromote toolproposals\.SScontains two parts, the overall cross\-tool guidance and the per\-tool entries\. For example, the Skill Writer adds a cross\-tool workflow rule instructing the Worker to query the catalog tool before running shell searches, and documents when to invoke the workbook loader on multi\-sheet Excel files\.

InStage 4: Verification, the Worker reruns the questions with the updated library\. Each rerun is compared with the original run on three checks, whether the changed tool was used, whether the task still finished, and whether any changed answer is better supported by the trajectory \(Section[3\.2\.2](https://arxiv.org/html/2610.27327#S3.SS2.SSS2)\)\. A change is kept if a rerun passes all three checks and reverted otherwise, after at most one round of repair\. The resulting libraryℒ\\mathcal\{L\}carries forward to the next batch and is frozen after offline training for evaluation\.

##### Two complementary channels for tool evolution\.

TEALcombines two complementary channels, the Critic and the Explorer, because neither suffices alone\. Prior tool\-evolution methods rely only on trajectory analysis similar to our Critic\([Xia et al\., 2025](https://arxiv.org/html/2610.27327#bib.bib9);[Li et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib1);[Ding et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib8)\)\. Trajectory analysis is effective for fixing specific execution errors, such as misread offset headers in Excel sheets or bugs in called tools\. However, because the Critic only sees a small batch of trajectories at a time, it focuses on immediate fixes for those questions rather than building data lake\-wide tools\. The Explorer addresses this limitation\. Without seeing any question or trajectory, it is instructed to act as a domain expert preparing tools for the whole data lake\. It scans the lake’s directory structures, file formats, and schemas and proposes tools that serve many questions, such as search and lookup tools, some backed by precomputed data artifacts such as metadata catalogs and value indexes\. These tools let the Worker locate relevant data in a single tool call\. In short, the Critic asks*what should change given how the Worker actually solved these tasks?*, while the Explorer asks*what would a domain expert build for this data lake?*

#### 3\.2\.2Verification without ground\-truth answers

A proposed tool change can pass its unit tests and still fail to help the Worker\. The tool may contain a logic error, such as reading a table at the wrong granularity, or the Worker may simply ignore it\.*Verification*is the stage that decides whether to keep or revert each change\. Because ground\-truth answers are unavailable,TEALcannot judge a change by whether it makes answers correct\. Instead, it reruns the Worker with the updated library on the questions that motivated the change and compares each rerun with the original trajectory on three checks\.

The first check confirms that the Worker actually calls the changed tool during the rerun, since a tool that is never called cannot improve any answer\. When the Worker ignores the tool, the reason is usually unclear guidance rather than a broken tool, soTEALasks the Skill Writer to revise the tool’s usage instructions and reruns the question before rejecting the change\.

The second check confirms that the change does not make the Worker worse at finishing the question\. A rerun fails this check if it produces no answer where the original run produced one, or if it uses more thanmmsteps beyond thennsteps of the original run\.

The third check applies only when the rerun produces a different answer from the original run\. In this case, an Analyst agent compares the two trajectories and judges which answer is better supported by the execution evidence, for example whether the final value appears in a tool output or is merely asserted by the Worker\. The Analyst does not assume that the original answer is correct\. The rerun fails this check only if the evidence favors the original answer, and passes if the evidence favors the new answer or is inconclusive\.

TEALkeeps a change if at least one rerun passes all three checks and reverts it otherwise\. In this way, verification goes beyond checking whether a tool runs without errors and uses the Worker’s own trajectories as evidence of whether the tool actually helped\.

Figure 2:Synthetic question generation withTEAL\-Q\. Given the data lake and a small set of seed questions as demonstrations,TEAL\-Qgenerates each question in four stages\.

### 3\.3Synthetic Question Generation viaTEAL\-Q

Questions written by users are scarce and not always accessible\. Existing data lake benchmarks contain only around a hundred questions each, and a newly deployed data lake may have none\. BecauseTEALdoes not need ground\-truth answers, evolution needs only questions\. We therefore proposeTEAL\-Q, which generates synthetic questions from the data lake, using a small set of seed questions as demonstrations of style and difficulty\. As shown in Figure[2](https://arxiv.org/html/2610.27327#S3.F2),TEAL\-Qgenerates each question in four stages\.

①Explore and Draft\.TEAL\-Qinspects a cluster of relevant data files and drafts a question with an answer, computed by running code on the data\.② Harden\.In up to three rounds,TEAL\-Qmakes the question harder, for example by adding a join across tables or a hard\-to\-parse file, and recomputes the answer after each round\.③ Build Pipeline\.TEAL\-Qturns the final solution into an executable pipeline and checks that it reproduces the computed answer\. This confirms that the question is answerable from the data\.④ Rewrite\.TEAL\-Qrewrites the question in natural language, preserving essential definitions that are needed to make it unambiguous but omitting details that the Worker can discover itself\.

TEAL\-Qpasses only the questions toTEAL\. The answers and solution pipelines serve only to keep each question well posed and answerable\. As a result,TEAL\-Qdoes not need to verify that its answers are correct, which is a major challenge in synthetic data generation\([Ramrakhya et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib3);[Gureja et al\., 2025](https://arxiv.org/html/2610.27327#bib.bib2)\)\.

## 4Experiments

Our experiments address two research questions \(RQs\)\.RQ1: Are agent\-evolved tools useful?We ask whether a library evolved without labels improves a coding agent’s accuracy on data\-lake analytics, across Workers, datasets, and step budgets\.RQ2: How can feedback be used effectively without ground\-truth answers?We ask which signals, from step budget pressure to the proposal channels and rerun\-based verification, make unsupervised evolution work\.

### 4\.1Experimental Settings

##### Datasets\.

For KramaBench\([Lai et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib12)\), we merge the data lakes of all six domains \(archaeology, astronomy, biomedical, environment, legal, and wildfire\) into a single unified data lake of 1,742 files and use stratified sampling to select 51 training and 53 test tasks\.

For LakeQA\([Wang et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib4)\), we use its tables\-only variant, which keeps the 130 questions whose final reasoning step relies on tables and folds any fact from removed non\-table sources into the question text, preserving the original answers\. We randomly split them into 50 training and 80 test tasks\. Each requires one to five reasoning hops over a lake of 4,180 tables\.

We report LakeQA accuracy under its deterministic scorer and the official KramaBench score, which is exact match for most tasks and partial credit \(F1 for lists, relative error for approximate numerics, an LLM judge for free\-text answers\) for the rest\. We refer to both as accuracy\.

Figure 3:Accuracy versus Worker step budget in the real\-question scenario\. Rows correspond to benchmarks \(top: KramaBench, bottom: LakeQA\) and columns to Worker models\. The horizontal axis is the step budgetkkat test time, from 10 to 30 steps\. The grey line is the baseline Worker without evolved tools\. The blue line isTEALwith a library evolved and tested at the same budgetkk\. The orange dashed line is a frozen library: it is evolved once at the ringed budget and then tested unchanged at each looser budget\.
##### Setup\.

We use OpenHands111[https://github\.com/OpenHands/software\-agent\-sdk](https://github.com/OpenHands/software-agent-sdk)\.as the harness for all agents\. To assess generalization across Worker backends, we evaluate Gemini\-3\.8\-Flash\([Google DeepMind, 2026b](https://arxiv.org/html/2610.27327#bib.bib26)\), Gemini\-3\.1\-Pro\([Google DeepMind, 2026a](https://arxiv.org/html/2610.27327#bib.bib25)\), Sonnet\-5\([Anthropic, 2026b](https://arxiv.org/html/2610.27327#bib.bib23)\), and Opus\-4\.8\([Anthropic, 2026a](https://arxiv.org/html/2610.27327#bib.bib24)\), while using Opus\-4\.8 for all evolution roles\. Each Worker episode has a step budget ofkk: up tok−3k\-3steps for the task, plus at most 3 steps to write an answer if none has been written by then\. A step budget is the practical way to bound the cost and latency of answering a question, and how many steps a deployment can afford varies\. We therefore treat it as an experimental variable\. In all experiments, we use batches of 5 questions, run the Explorer on the first batch and everyE=4E=4batches after, and setm=max⁡\(2,0\.2​n\)m=\\max\(2,\\,0\.2n\)\.

### 4\.2Main Results

Given the real training questions from the datasets, we sweepkkover 10, 15, 20, 25, and 30 steps\. Figure[3](https://arxiv.org/html/2610.27327#S4.F3)plots accuracy againstkkfor the baseline without the evolved library and forTEAL, reporting the mean and standard deviation over three test repeats\. We have the following findings\.

Under a tight step budget,TEALimproves accuracy substantially\.With Sonnet\-5 as the Worker,TEALimproves accuracy by 22\.6 points on KramaBench and 30\.4 points on LakeQA at a budget of 10 steps, and by 9\.3 and 21\.7 points at 15 steps\. The gain holds for all four Workers on both lakes at 10 and 15 steps\. In five of the eight panels,TEALat 15 steps is on par with or above the baseline at 30 steps\. We attribute the improvement to the fact that a single tool call replaces a multi\-step procedure, such as locating, loading, and joining the relevant files, so the Worker can finish the task within a budget that the baseline exhausts on file discovery\. Appendix[A](https://arxiv.org/html/2610.27327#A1)illustrates this with two test tasks, and Appendix[B](https://arxiv.org/html/2610.27327#A2)lists the most used tools\.

Under a loose budget, a library evolved at that budget does not always help, but one evolved under a tighter budget does\.At 20 to 30 steps, the library evolved at the deployment budget \(blue line\) sometimes falls below the baseline \(grey line\), whereas the library evolved under a tighter budget and deployed unchanged \(orange line\) stays above the baseline at every looser budget and is usually the best of the three\. The difference lies in the feedback that evolution receives\. Under a tight budget, the Worker fails in visible ways, through empty answers, truncated pipelines, and repeated processing work, which give the Critic concrete targets\. A saved step then changes the rerun outcome, so verification can tell a useful tool from a useless one, and the short trajectories are also easier to analyze\. Under a loose budget, the Worker solves most tasks inline, so a tool rarely changes the outcome, and the long trajectories make proposals harder to ground and reruns harder to compare, which leaves the quality of tools and skills unchecked\. Step budget pressure thus acts as a training signal in place of gold labels, which makes evolving under a tighter budget and deploying under a looser one a practical recipe\.

Appendix[C](https://arxiv.org/html/2610.27327#A3)further reports the number of steps each Worker actually uses and the resulting cost per episode, which reflect the practical value ofTEALin deployment\.

### 4\.3Comparison with Existing Methods

We also compareTEALwith the agentic methods evaluated in KramaBench and LakeQA and with an existing tool evolution method\.DS\-Guru\([Lai et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib12)\)writes a complete Python script from a data preview and repairs it with error feedback\.Smolagents DeepResearch\([Roucher et al\., 2025](https://arxiv.org/html/2610.27327#bib.bib16);[Lai et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib12)\)is a ReAct agent\([Yao et al\., 2023](https://arxiv.org/html/2610.27327#bib.bib10)\)that executes Python step by step with file inspection tools\.LakeQA Agent\([Wang et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib4)\)is the reference agent released with LakeQA, which calls human\-designed tools to search, inspect, and run Python, andLakeQA Agent \+ RAG\([Wang et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib4)\)adds a retrieval tool backed by a pre\-built index of the lake’s tables\.ISE\([Li et al\., 2026](https://arxiv.org/html/2610.27327#bib.bib1)\)evolves tools without gold labels and keeps every tool that runs without errors, and we reproduce it from the released code and replace only its Worker with that ofTEAL\.

Table 1:Accuracy \(%\) of existing methods andTEAL, Sonnet\-5 as the Worker\.†Library evolved at 15 \(KramaBench\) or 20 \(LakeQA\) steps, deployed unchanged \(orange lines in Figure[3](https://arxiv.org/html/2610.27327#S4.F3)\)\.For each existing method, we use the hyperparameters from its original evaluation on the corresponding dataset, including a 20\-step budget for Smolagents and a 25\-step budget for LakeQA Agent, so its result is the same in both columns\. Table[1](https://arxiv.org/html/2610.27327#S4.T1)leads to two observations\.

First, our baseline Worker is already a strong coding agent\. On KramaBench, it outperforms DS\-Guru and Smolagents by a wide margin even at 15 steps, fewer than the 20 steps Smolagents is allowed\. On LakeQA, it surpasses LakeQA Agent at 30 steps, with or without RAG, even though LakeQA Agent comes with human\-designed tools and a pre\-built retrieval index\.TEALimproves on this strong baseline on both datasets and at both budgets\.

Second, we compareTEALwith ISE, another unsupervised tool evolution method whose released code allows a faithful reproduction\. ISE retains a tool once it executes successfully\. Under the same Worker and the same 15\-step budget, ISE underperformsTEALon both datasets \(by 5\.7 points on KramaBench\), indicating that a tool that executes successfully does not necessarily help with the task\. This issue is more pronounced on LakeQA, where ISE falls 10\.0 points below the baseline\. For instance, ISE evolvedsearch\_file\_content, which scans the lake line by line\. The tool was retained after a single successful run on a simple search, yet 121 of its 126 calls timed out at test time\. The steps consumed by these calls prevented the Worker from reaching an answer within the step budget\. Under execution\-only verification, such a tool therefore does more harm than good\.

### 4\.4Results in the synthetic\-question scenario

Table 2:Accuracy \(%\) ofTEALevolved on real or synthetic questions with Sonnet\-5 as the Worker\. All configurations use a 15\-step budget and the same held\-out test sets\.In the synthetic\-question scenario,TEAL\-Qgenerates 50 questions from the lake of each dataset\. Evolution sees only these questions, without answers, and the resulting library is evaluated on the same held\-out test tasks as in the real\-question scenario\. In Table[2](https://arxiv.org/html/2610.27327#S4.T2), the library evolved on synthetic questions improves Sonnet\-5 over the baseline on both datasets and performs comparably to the library evolved on real questions\. The same holds for all four Workers \(Table[7](https://arxiv.org/html/2610.27327#A4.T7)in Appendix[D](https://arxiv.org/html/2610.27327#A4)\)\. Although the generated questions are never verified by a human, they exercise enough of the lake’s preparation and discovery work forTEALto build a library that transfers to real questions, so tool evolution can begin before any user question has been asked\.

### 4\.5Ablation and Generalization

Table 3:Accuracy \(%\) of ablations and reference configurations with Sonnet\-5 as the Worker at a 15\-step budget, in the real\-question scenario\.Table[3](https://arxiv.org/html/2610.27327#S4.T3)isolates the contribution of each component with Sonnet\-5 as the Worker at a budget of 15 steps\. Removing the Explorer leaves only the Critic channel, which still improves over the baseline but falls slightly short of the fullTEAL\. Evolution without the Explorer also no longer produces the artifact\-backed tools shown in Appendix[A](https://arxiv.org/html/2610.27327#A1), consistent with the complementary roles of the two channels \(Section[3\.2\.1](https://arxiv.org/html/2610.27327#S3.SS2.SSS1.Px1)\)\. Removing the evolved skills causes a larger drop, because the Worker no longer knows how to use the tools\. A good tool alone therefore does not guarantee a gain, and the tools and their usage skills must be evolved together\.

We also report two oracle variants that giveTEALground\-truth labels during evolution, which our unsupervised setting does not have\. Given only the ground\-truth answers,TEALperforms comparably to its label\-free version, so our verification checks already obtain the feedback that the answers would provide\. Given the full ground\-truth solutions as well, the system knows how to solve the tasks and creates more targeted tools, such asmerge\_entity\_tables, which joins tables on both state and metropolitan area and matches area names that differ only in punctuation, whereas the label\-free library only standardizes state names withnormalize\_join\_state\_keys\.

To test generalization across harnesses, we take the library evolved under OpenHands and deploy it unchanged under three other harnesses, Google ADK222[https://github\.com/google/adk\-python](https://github.com/google/adk-python)\., Prime Agent333[https://github\.com/PrimeIntellect\-ai/prime\-agent](https://github.com/PrimeIntellect-ai/prime-agent)\., and Codex CLI444[https://github\.com/openai/codex](https://github.com/openai/codex)\., with Sonnet\-5 as the Worker in each\. The library improves the Worker over its baseline under every harness, so the evolved tools and skills generalize beyond the harness they were evolved in\.

## 5Conclusion

In this paper, we proposedTEAL, a framework that evolves tools and usage skills for a coding agent over a data lake without ground\-truth labels, andTEAL\-Q, which generates synthetic questions for evolution\.TEALproposes tools from both the data lake and the Worker’s trajectories, and verifies each change by checking the Worker’s tool use and trajectory evidence on a rerun, which performs comparably to verification with ground\-truth answers\. Our results show that unsupervised tool evolution is most effective under a tight step budget, which reveals where the Worker needs tools\. Libraries evolved in this way improve all four Workers under tight budgets and remain effective under looser budgets, and libraries evolved on synthetic questions perform comparably to those evolved on real ones\.

##### Limitations and future work\.

The evolved library of tools and skills introduces additional overhead in two forms\. First, it lengthens the prompt that the Worker reads at every step, which increases the input tokens\. Second, the Worker tends to reason more at each step, plausibly about which tools to call \(Appendix[C](https://arxiv.org/html/2610.27327#A3)\)\. Future work may reduce the input overhead by improving cache reuse of repeated prompt content, and the reasoning overhead by internalizing the skills into the Worker\.

## References

- Agrawalet al\.\(2000\)S\. Agrawal, S\. Chaudhuri, and V\. R\. NarasayyaAutomated selection of materialized views and indexes in sql databases\.InProceedings of the 26th International Conference on Very Large Data Bases,VLDB ’00,San Francisco, CA, USA,pp\. 496–505\.External Links:ISBN 1558607153Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px1.p1.1)\.
- Anthropic \(2026a\)AnthropicSystem card: claude opus 4\.8\.Note:[https://www\.anthropic\.com/claude\-opus\-4\-8\-system\-card](https://www.anthropic.com/claude-opus-4-8-system-card)Accessed: 2026\-09\-15Cited by:[§4\.1](https://arxiv.org/html/2610.27327#S4.SS1.SSS0.Px2.p1.1)\.
- Anthropic \(2026b\)AnthropicSystem card: claude sonnet 5\.Note:[https://www\.anthropic\.com/claude\-sonnet\-5\-system\-card](https://www.anthropic.com/claude-sonnet-5-system-card)Accessed: 2026\-09\-15Cited by:[§4\.1](https://arxiv.org/html/2610.27327#S4.SS1.SSS0.Px2.p1.1)\.
- Caiet al\.\(2024\)T\. Cai, X\. Wang, T\. Ma, X\. Chen, and D\. ZhouLarge language models as tool makers\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54067–54089\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p4.1),[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px2.p1.1)\.
- Casteloet al\.\(2021\)S\. Castelo, R\. Rampin, A\. Santos, A\. Bessa, F\. Chirigati, and J\. FreireAuctus: a dataset search engine for data augmentation\.arXiv preprint arXiv:2102\.05716\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1)\.
- Chonget al\.\(2026\)M\. Chong, S\. Zhang, J\. Fan, and X\. DuEvoOntology: a self\-evolving ontology layer for data agents\.arXiv preprint arXiv:2609\.15779\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p4.1),[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px1.p1.1)\.
- Dinget al\.\(2026\)H\. Ding, Y\. Xie, Z\. Wei, Y\. Li, and B\. DingFrom atomic actions to standard operating procedures: iterative tool optimization for self\-evolving llm agents\.arXiv preprint arXiv:2607\.07321\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px2.p1.1),[§3\.2\.1](https://arxiv.org/html/2610.27327#S3.SS2.SSS1.Px1.p1.1)\.
- Fernandezet al\.\(2018\)R\. C\. Fernandez, Z\. Abedjan, F\. Koko, G\. Yuan, S\. Madden, and M\. StonebrakerAurum: a data discovery system\.In2018 IEEE 34th International conference on data engineering \(ICDE\),pp\. 1001–1012\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1),[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px1.p1.1)\.
- Fernandezet al\.\(2023\)R\. C\. Fernandez, A\. J\. Elmore, M\. J\. Franklin, S\. Krishnan, and C\. TanHow large language models will disrupt data management\.Proceedings of the VLDB Endowment16\(11\),pp\. 3302–3309\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1)\.
- Freireet al\.\(2025\)J\. Freire, G\. Fan, B\. Feuer, C\. Koutras, Y\. Liu, E\. Peña, A\. Santos, C\. T\. Silva, and E\. WuLarge language models for data discovery and integration: challenges and opportunities\.IEEE Data Engineering Bulletin\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1)\.
- Google DeepMind \(2026a\)Google DeepMindGemini 3\.1 pro model card\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-1\-pro/](https://deepmind.google/models/model-cards/gemini-3-1-pro/)Accessed: 2026\-09\-15Cited by:[§4\.1](https://arxiv.org/html/2610.27327#S4.SS1.SSS0.Px2.p1.1)\.
- Google DeepMind \(2026b\)Google DeepMindGemini 3\.8 flash model card\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-8\-flash/](https://deepmind.google/models/model-cards/gemini-3-8-flash/)Accessed: 2026\-09\-15Cited by:[§4\.1](https://arxiv.org/html/2610.27327#S4.SS1.SSS0.Px2.p1.1)\.
- Graefe \(2024\)G\. GraefeMore modern b\-tree techniques\.Foundations and Trends in Databases13\(3\),pp\. 169–249\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p3.1)\.
- Gurejaet al\.\(2025\)S\. Gureja, E\. Tommasone, J\. He, S\. Hooker, M\. Gallé, and M\. FadaeeVerification limits code llm training\.arXiv preprint arXiv:2509\.20837\.Cited by:[§3\.3](https://arxiv.org/html/2610.27327#S3.SS3.p3.1)\.
- Haiet al\.\(2023\)R\. Hai, C\. Koutras, C\. Quix, and M\. JarkeData lakes: a survey of functions and systems\.IEEE Transactions on Knowledge and Data Engineering35\(12\),pp\. 12571–12590\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1)\.
- Halevyet al\.\(2016\)A\. Halevy, F\. Korn, N\. F\. Noy, C\. Olston, N\. Polyzotis, S\. Roy, and S\. E\. WhangGoods: organizing google’s datasets\.InProceedings of the 2016 International Conference on Management of Data,pp\. 795–806\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p3.1),[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px1.p1.1)\.
- Halevy \(2001\)A\. Y\. HalevyAnswering queries using views: a survey\.The VLDB Journal10\(4\),pp\. 270–294\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p3.1)\.
- Kandelet al\.\(2011\)S\. Kandel, A\. Paepcke, J\. Hellerstein, and J\. HeerWrangler: interactive visual specification of data transformation scripts\.InProceedings of the sigchi conference on human factors in computing systems,pp\. 3363–3372\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1)\.
- Laiet al\.\(2026\)E\. Lai, G\. Vitagliano, Z\. Zhang, O\. Chabra, S\. Sudhir, A\. Zeng, A\. Zabreyko, C\. Li, F\. Kossmann, J\. Ding,et al\.Kramabench: a benchmark for ai systems on data\-to\-insight pipelines over data lakes\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 142883–142912\.Cited by:[§1\.1](https://arxiv.org/html/2610.27327#S1.SS1.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.27327#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.27327#S4.SS1.SSS0.Px1.p1.1),[§4\.3](https://arxiv.org/html/2610.27327#S4.SS3.p1.1)\.
- Liet al\.\(2026\)H\. Li, S\. Yang, W\. Qi, S\. Zhao, R\. Hua, M\. Song, X\. Yang, and C\. PengYunjue agent tech report: a fully reproducible, zero\-start in\-situ self\-evolving agent system for open\-ended tasks\.arXiv preprint arXiv:2601\.18226\.Cited by:[§1\.1](https://arxiv.org/html/2610.27327#S1.SS1.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.27327#S1.p4.1),[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px2.p1.1),[§3\.2\.1](https://arxiv.org/html/2610.27327#S3.SS2.SSS1.Px1.p1.1),[§4\.3](https://arxiv.org/html/2610.27327#S4.SS3.p1.1)\.
- Liet al\.\(2023\)P\. Li, Y\. He, C\. Yan, Y\. Wang, and S\. ChaudhuriAuto\-tables: synthesizing multi\-step transformations to relationalize tables without using examples\.Proc\. VLDB Endow\.16\(11\),pp\. 3391–3403\.External Links:ISSN 2150\-8097,[Link](https://doi.org/10.14778/3611479.3611534),[Document](https://dx.doi.org/10.14778/3611479.3611534)Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1)\.
- Liuet al\.\(2024\)Y\. Liu, E\. Pena, A\. Santos, E\. Wu, and J\. FreireMagneto: combining small and large language models for schema matching\.arXiv preprint arXiv:2412\.08194\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px1.p1.1)\.
- Luet al\.\(2026\)J\. Lu, Z\. Kong, Y\. Wang, R\. Fu, H\. Wan, C\. Yang, W\. Lou, H\. Sun, L\. Wang, Y\. Jiang, X\. Wang, X\. Sun, and D\. ZhouBeyond static tools: test\-time tool evolution for scientific reasoning\.arXiv preprint arXiv:2601\.07641\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px2.p1.1)\.
- Namet al\.\(2025\)J\. Nam, J\. Yoon, J\. Chen, R\. Sinha, J\. Shin, and T\. PfisterDS\-star: data science agent for solving diverse tasks across heterogeneous formats and open\-ended queries\.arXiv preprint arXiv:2509\.21825\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px1.p1.1)\.
- Narayanet al\.\(2022\)A\. Narayan, I\. Chami, L\. Orr, S\. Arora, and C\. RéCan foundation models wrangle your data?\.arXiv preprint arXiv:2205\.09911\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1)\.
- Nargesianet al\.\(2019\)F\. Nargesian, E\. Zhu, R\. J\. Miller, K\. Q\. Pu, and P\. C\. ArocenaData lake management: challenges and opportunities\.Proceedings of the VLDB Endowment12\(12\),pp\. 1986–1989\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1)\.
- Patonet al\.\(2023\)N\. W\. Paton, J\. Chen, and Z\. WuDataset discovery and exploration: a survey\.ACM Computing Surveys56\(4\),pp\. 1–37\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px1.p1.1)\.
- Qiaoet al\.\(2023\)B\. Qiao, L\. Li, X\. Zhang, S\. He, Y\. Kang, C\. Zhang, F\. Yang, H\. Dong, J\. Zhang, L\. Wang,et al\.Taskweaver: a code\-first agent framework\.arXiv preprint arXiv:2311\.17541\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px1.p1.1)\.
- Qiuet al\.\(2025\)J\. Qiu, X\. Qi, T\. Zhang, X\. Juan, J\. Guo, Y\. Lu, Y\. Wang, Z\. Yao, Q\. Ren, X\. Jiang,et al\.Alita: generalist agent enabling scalable agentic reasoning with minimal predefinition and maximal self\-evolution\.arXiv preprint arXiv:2505\.20286\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px2.p1.1)\.
- Qiuet al\.\(2026\)Z\. Qiu, K\. Song, S\. Tang, S\. Qiao, L\. Liang, H\. Chen, and S\. DengUnsupervised skill discovery for agentic data analysis\.arXiv preprint arXiv:2606\.06416\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px1.p1.1)\.
- Ramrakhyaet al\.\(2026\)R\. Ramrakhya, A\. Szot, O\. Attia, B\. Mazoure, A\. Nguyen, Y\. Yang, Z\. Gan, H\. Agrawal, and A\. ToshevScaling synthetic task generation for agents via exploration\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 123630–123660\.Cited by:[§3\.3](https://arxiv.org/html/2610.27327#S3.SS3.p3.1)\.
- Roucheret al\.\(2025\)A\. Roucher, A\. V\. del Moral, T\. Wolf, L\. von Werra, and E\. KaunismäkiSmolagents: a smol library to build great agentic systems\.Hugging Face\.Cited by:[§4\.3](https://arxiv.org/html/2610.27327#S4.SS3.p1.1)\.
- Shenet al\.\(2026\)X\. Shen, J\. Chen, L\. Zheng, H\. Ma, T\. Wei, and K\. ZhanEvolving from tool user to creator via training\-free experience reuse in multimodal reasoning\.arXiv preprint arXiv:2602\.01983\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2023\)G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. AnandkumarVoyager: an open\-ended embodied agent with large language models\.arXiv preprint arXiv:2305\.16291\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026\)H\. Wang, J\. Liu, Y\. Liu, A\. S\. Wijaya, T\. Zhou, E\. Wu, Y\. Chen, W\. You, R\. Vir, D\. Pinto,et al\.LakeQA: an exploratory qa benchmark over a million\-scale data lake\.arXiv preprint arXiv:2606\.10460\.Cited by:[§1\.1](https://arxiv.org/html/2610.27327#S1.SS1.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.27327#S4.SS1.SSS0.Px1.p2.1),[§4\.3](https://arxiv.org/html/2610.27327#S4.SS3.p1.1)\.
- Wanget al\.\(2024a\)X\. Wang, Y\. Chen, L\. Yuan, Y\. Zhang, Y\. Li, H\. Peng, and H\. JiExecutable Code Actions Elicit Better LLM Agents\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2402.01030)Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1)\.
- Wanget al\.\(2025\)X\. Wang, B\. Li, Y\. Song, F\. F\. Xu, X\. Tang, M\. Zhuge, J\. Pan, Y\. Song, B\. Li, J\. Singh,et al\.Openhands: an open platform for ai software developers as generalist agents\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 65882–65919\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1)\.
- Wanget al\.\(2024b\)Z\. Wang, D\. Fried, and G\. NeubigTrove: inducing verifiable and efficient toolboxes for solving programmatic tasks\.arXiv preprint arXiv:2401\.12869\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p4.1)\.
- Xiaet al\.\(2025\)C\. S\. Xia, Z\. Wang, Y\. Yang, Y\. Wei, and L\. ZhangLive\-swe\-agent: can software engineering agents self\-evolve on the fly?\.arXiv preprint arXiv:2511\.13646\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px2.p1.1),[§3\.2\.1](https://arxiv.org/html/2610.27327#S3.SS2.SSS1.Px1.p1.1)\.
- Yanget al\.\(2026\)Y\. Yang, Q\. Zhu, Z\. Han, B\. Han, Z\. Shen, S\. Wang, V\. N\. Ioannidis, and H\. RangwalaWhen llms read tables carelessly: measuring and reducing data referencing errors\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 16734–16752\.Cited by:[§2](https://arxiv.org/html/2610.27327#S2.SS0.SSS0.Px1.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: Synergizing Reasoning and Acting in Language Models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2210.03629)Cited by:[§4\.3](https://arxiv.org/html/2610.27327#S4.SS3.p1.1)\.
- Zhanget al\.\(2026a\)H\. Zhang, Y\. Liu, A\. Santos, W\. \(\. Hung, and J\. FreireAutoDDG: automated dataset description generation using large language models\.Proc\. ACM Manag\. Data4\(1\)\.External Links:[Link](https://doi.org/10.1145/3786626),[Document](https://dx.doi.org/10.1145/3786626)Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p4.1)\.
- Zhanget al\.\(2026b\)J\. Zhang, S\. O\. Arik, C\. Arad, F\. Ozcan, and A\. HalevyAn agentic approach to metadata reasoning\.arXiv preprint arXiv:2604\.20144\.Cited by:[§1](https://arxiv.org/html/2610.27327#S1.p1.1)\.

## Appendix AQualitative Analysis

##### Tools with build artifacts\.

As introduced in Section[3](https://arxiv.org/html/2610.27327#S3), the Explorer may create a tool backed by a persistent artifact that is constructed once, stored alongside the tool, and consulted at call time\. We illustrate this mechanism withsearch\_dataset\_catalog, the discovery tool of the Sonnet\-5 LakeQA library evolved under a budget of 10 steps\.

Its artifact is a catalog of the lake, built in a single pass over every directory that infers the true format of each file from its bytes, records the header, an estimated row count, per\-column samples, and data\-quality flags, and embeds the name of each directory\. The pass takes approximately four minutes of wall\-clock time and roughly forty batched embedding calls, and the resulting 93 MB SQLite catalog is stored beside the tool and never rebuilt inside a task\. The tool itself is a ranked search over this catalog\. It takes a natural\-language description of the data sought and returns the best\-matching directories\. The tool was revised fourteen times, each revision traceable to a Worker trajectory\. For instance, after a Worker had to fall back tols \| grepto collect the yearly editions of a table, the tool was revised to return all editions of a matched table together\.

Across all experiments, most artifacts are catalogs, which answer which file is relevant, or value indexes, which answer which file contains a given value\. What exactly is built varies from one evolution run to another\. For example, one value index inverts the cell values themselves and reaches 6 GB, whereas another is a full\-text index over names, headers, and sampled cells\. Other artifacts also occur, such as manifests that record which files belong together and in what order, and decoded copies of archives that were mis\-stored as text\.

##### How tools help\.

First, tools save steps, which is whyTEALis particularly effective under a tight step budget\. At test time the Worker invokedsearch\_dataset\_catalogin all 80 held\-out tasks\. On the task requesting the 2018 violent crime rate of Albany County, everyTEALrepeat finished within four steps\. One catalog query located the county\-level crime\-rate table, one SQL call through the companionquery\_tabular\_filetool returned 359\.8, and the Worker submitted the answer\. Without such a tool, the baseline used nine to ten steps in every repeat, most of them onfindandgrepover the lake\. The additional steps did not translate into accuracy, since in two of the three repeats the search selected an incorrect crime table and the baseline did not answer correctly\. Figure[6](https://arxiv.org/html/2610.27327#A4.F6)shows one repeat of each\.

Second, tools make accuracy more robust, because a tool’s output is consistent across repeats while the Worker’s manual work is not\. On the task asking for the calcium reading at the smallest dust\-proxy value in a paleoclimate workbook,inspect\_excel\_workbookdetected the header on the sixth row and returned the columns under their real names, so everyTEALrepeat filtered the two relevant columns with short scripts and answered correctly\. The baseline, with nine steps in every repeat but without reusable tools, did not reach a correct answer\. In one repeat it parsed the workbook manually and misaligned the header, and in another it located the file but failed to parse the header\. Figure[7](https://arxiv.org/html/2610.27327#A4.F7)shows one repeat of each\.

## Appendix BTool Statistics

Tables[4](https://arxiv.org/html/2610.27327#A2.T4)and[5](https://arxiv.org/html/2610.27327#A2.T5)list the five most used tools in the libraries evolved for each lake with Gemini\-3\.1\-Pro as the Worker\. We run each test task three times and call each run an episode\. The coverage of a tool is the percentage of test episodes in which the Worker called that tool at least once\. On both lakes the most used tools answer where the data lives\.

As shown in Table[4](https://arxiv.org/html/2610.27327#A2.T4), three of the KramaBench tools help the Worker find data\. One maps a topic to the files that hold it, one searches for a column name or a value across all files, including Excel workbooks thatgrepcannot read, and one lists the files in the lake\. The other two load files that standard readers parse incorrectly, namely CSVs that stack several tables in one file and Excel sheets whose header is not on the first row\. The LakeQA lake is larger, with over four thousand files\. As shown in Table[5](https://arxiv.org/html/2610.27327#A2.T5), its library therefore indexes the lake ahead of time\. One tool looks up a pre\-built catalog that records where each dataset is and which columns and format it has, and another searches a pre\-built full\-text index of dataset names and column headers\. Its compute tool runs a filter or an aggregate over a large file through DuckDB, which streams the file instead of loading it into memory\. The Worker calls the catalog tool in nearly every LakeQA episode, so almost every solution starts by querying the catalog rather than walking the directory tree\.

Table 4:The five most used tools in the KramaBench library evolved with Gemini\-3\.1\-Pro as the Worker under a 10\-step budget\. Coverage \(%\) is the share of the 159 test episodes \(53 tasks, three repeats\) that call the tool at least once\.Table 5:The five most used tools in the LakeQA library evolved with Gemini\-3\.1\-Pro as the Worker under a 20\-step budget\. Coverage \(%\) is the share of the 240 test episodes \(80 tasks, three repeats\) that call the tool at least once\.
## Appendix CSteps and Cost

### C\.1Step counting

The number of steps a Worker actually uses can differ from its step budget\. Figure[4](https://arxiv.org/html/2610.27327#A3.F4)therefore plots the accuracy ofTEALand the baseline against the steps they actually use under each step budget\. The horizontal axis is the mean number of steps per episode over three test repeats, and the vertical axis is the accuracy, so a curve closer to the upper left offers a better trade\-off between accuracy and steps\. As expected, the evolved tools let the Worker reach higher accuracy with fewer steps\.

Figure 4:Accuracy versus mean Worker steps per test episode\.
### C\.2Cost accounting

A step budget is a practical way to bound the cost and latency of an agent in deployment\. A budget stated directly in tokens or in dollars is less practical, since the agent would spend extra tokens tracking how many it has already used\. However, a step budget does not reflect the actual cost of an episode\. We therefore also plot accuracy against the estimated cost per test episode in Figure[5](https://arxiv.org/html/2610.27327#A3.F5)\.

Table 6:Vertex AI list prices \(USD per million tokens\) used to estimate the cost of a test episode\.We compute the cost of each test episode from three token counts\. The*uncached input*is the number of tokens written to the prompt cache for the first time, the*cached input*is the number of tokens read back from the cache on later turns, and the*output*is the number of completion tokens, including hidden reasoning, which both providers bill as output\. We assume an optimal caching policy\. The prefix, consisting of the system prompt, the tool schemas, and the skills, stays the same across all test episodes with the same method, Worker, lake, and step budget, so it is always read from the cache\. Each turn then writes only the tokens added since the previous turn and reads the rest from the cache\. The cost of an episode is then

C=nu​pu\+nc​pc\+no​po,C=n\_\{\\mathrm\{u\}\}\\,p\_\{\\mathrm\{u\}\}\+n\_\{\\mathrm\{c\}\}\\,p\_\{\\mathrm\{c\}\}\+n\_\{\\mathrm\{o\}\}\\,p\_\{\\mathrm\{o\}\},\(1\)wherenun\_\{\\mathrm\{u\}\},ncn\_\{\\mathrm\{c\}\}, andnon\_\{\\mathrm\{o\}\}are the uncached input, cached input, and output token counts of the episode, andpup\_\{\\mathrm\{u\}\},pcp\_\{\\mathrm\{c\}\}, andpop\_\{\\mathrm\{o\}\}are the corresponding Vertex AI list prices of the Worker, given in Table[6](https://arxiv.org/html/2610.27327#A3.T6)\. Since it assumes no cache misses, this estimate is a lower bound\. We report this estimate instead of the billed cost because prompt caching behaved differently across Workers in our runs, so the billed cost mostly reflects the caching configuration of the client, whereas the estimate applies the same accounting to every Worker and method\.

Figure 5:Accuracy versus estimated inference cost per test episode\.Figure[5](https://arxiv.org/html/2610.27327#A3.F5)is read in the same way as Figure[4](https://arxiv.org/html/2610.27327#A3.F4), except that the horizontal axis is the mean cost per episode, so a curve closer to the upper left again offers a better trade\-off\. With Sonnet\-5, Gemini\-3\.1\-Pro, and Gemini\-3\.8\-Flash as the Worker,TEALgenerally reaches higher accuracy at the same cost, or the same accuracy at a lower cost\. Opus\-4\.8 has the most expensive input tokens, which limits the cost advantage ofTEAL, butTEALremains the better choice when the goal is higher accuracy\.

Three factors shape the cost ofTEAL\. First,TEALadds several tools and their skills to the prompt, so its fixed prefix is roughly two to three times as long as the baseline’s, and the Worker reads this prefix again at every step\. Prompt caching absorbs much of this overhead, so we recommend enabling it when deployingTEAL, and we hope that future work on more effective caching will further reduce the cost of such repeated input\. Second, perhaps surprisingly, whenTEALshortens an episode, the saving appears in uncached input rather than in output\. The result of a tool or shell command is billed as new input at the next step, and an evolved tool returns a compact result in place of the raw intermediate output that the baseline passes through the shell\. On KramaBench at 15 steps, for example,TEALcuts the uncached input per episode by 37% to 63% for Sonnet\-5, Gemini\-3\.1\-Pro, and Gemini\-3\.8\-Flash\. Third, the output shrinks much less than the number of steps, since the output per step grows for most Workers\. We attribute this to the Worker reasoning more at each step, plausibly about which of the additional tools to call\.

## Appendix DSynthetic\-Question Scenario Across Workers

Table[7](https://arxiv.org/html/2610.27327#A4.T7)extends Table[2](https://arxiv.org/html/2610.27327#S4.T2)to all four Workers\.

Table 7:Accuracy \(%\) ofTEALevolved on real or synthetic questions, compared with the same Worker without evolved tools\. All configurations use a 15\-step budget and the same held\-out test tasks\.\[step\]steptoolevolved toolinputWorker inputThoughtWorker thinking\-\> outputtool or shell output

TEAL✓

\[step 1\]search\_dataset\_catalogquery: "violent crime rate by county New York 2018"\-\> \{"best\_match":"index\-violent\-property\-and\-firearm\-rates\-by\-county\-beginning\-1990","dataset\_geography\_sample": \[\{"column": "County", "value": "Albany"\}, \.\.\.\],"score": 10\.104, \.\.\.\}\[step 2\]query\_tabular\_fileThought: This is exactly the dataset\. Buffalo \(1832\) is earlier thanRochester \(1834\), so Buffalo is the terminus city; the other terminus isAlbany, in Albany County\.path: "data/datagov/index\-violent\-property\-and\-firearm\-rates\-by\-county\-beginning\-1990/files/rows\.txt"mode: "query"sql: "SELECT County, Year, "Violent Rate" FROM \{src\}WHERE County=’Albany’ AND Year=2018"\-\> \{"rows": \[\{"County": "Albany", "Year": 2018, "Violent Rate": 359\.8\}\],"columns": \["County", "Year", "Violent Rate"\], "n\_returned": 1, \.\.\.\}\[step 3\]file\_editor create \./answer\.txt "359\.8"

Baseline✗, where it wasted steps on data discovery\.

\[step 1\]$ find \./data \-iname "\*crime\*"; find \./data \-iname "\*albany\*"\-\> \(no output\)\[step 2\]$ find \./data \-type f \| head \-100\-\> \(no output\)\[step 3\]$ pwd; ls \-la; find \. \-maxdepth 3 \-type d; find \. \-type f \| wc \-l; \.\.\.\-\> lrwxrwxrwx data \-\> /\.\.\./lake/lakeqa\_tables/unified\[step 4\]$ find \-L \./data \-type f \| wc \-l; find \-L \./data \-iname "\*crime\*"; \\find \-L \./data \-iname "\*albany\*"\-\> 4180\./data/datagov/crime\-incidents\-in\-2010\[\.\.\. 20 more datasets \.\.\.\]\./data/datagov/violent\-crime\-property\-crime\-by\-county\-1975\-to\-present\[\.\.\. 16 more datasets \.\.\.\]\./data/datagov/city\-of\-albany\-five\-year\-capital\-plan\-beginning\-2013\[step 5\]$ find \-L \\\./data/datagov/violent\-crime\-property\-crime\-by\-county\-1975\-to\-present \-type f\-\> \./data/datagov/violent\-crime\-property\-crime\-by\-county\-1975\-to\-present/files/rows\.txt\[step 6\]$ head \-1 "\.\.\./rows\.txt"; \\grep \-i "ˆ\\"Albany\\"\\\|,Albany," "\.\.\./rows\.txt" \| grep ",2018,"\-\> JURISDICTION,YEAR,POPULATION,MURDER,RAPE,ROBBERY,AGG\. ASSAULT,B & E,LARCENY THEFT,M/V THEFT,GRAND TOTAL,PERCENT CHANGE,VIOLENT CRIME TOTAL,\[\.\.\.\]\(grep matched nothing, exit code 1\)\[step 7\]$ grep \-i "Albany" "\.\.\./rows\.txt" \| head \-20\-\> \(no output\)\[warning\]All 7 task steps are used\. Please write your answer now\.\[step 8\]file\_editor create \./answer\.txt "421\.8"

Figure 6:Trajectories on the Albany County task from Section[A](https://arxiv.org/html/2610.27327#A1), both with Sonnet\-5 as the Worker under a 10\-step budget\. Tool outputs and shell observations are abridged, with elisions marked by\[\.\.\.\]\. The finalfinishstep is omitted\.\[step\]steptoolevolved toolinputWorker inputThoughtWorker thinking\-\> outputtool or shell output

TEAL✓

\[step 1\]resolve\_concept\_to\_datasetsconcept: "dust proxy OR wet\-dry index OR calcium"\-\> \{"candidates": \[\{"path": "\./data/climateMeasurements\.xlsx", "relevance": 0\.4,"kind": "data\_table", "ext": "xlsx", "recommended\_loader": "inspect\_excel\_workbook","why": \{"content\_evidence": \{"sheet": "Sheet1", "column\_index": 0, "column\_name":"Data Supplement to: \\"A 3 million year index for North African humidity/aridity\[\.\.\.\]\\"", \.\.\.\}\}\}, \[\.\.\. 7 more candidates \.\.\.\]\], "n\_files\_walked": 1742, \.\.\.\}\[step 2\]inspect\_excel\_workbookpath: "\./data/climateMeasurements\.xlsx" sheet: "Sheet1"\-\> \{"artifact\_path": "\.\.\./inspect\_excel\_workbook/cache/climateMeasurements\_\_Sheet1\.parquet","artifact\_format": "parquet", "artifact\_rows": 8360, "artifact\_cols": 30,"headerless": false, "header\_index": 5, "header\_depth": 1, "score": 1\.0,"columns": \["Site", "Hole", "Core", "Section", "Interval\_cm", "Depth below seafloor\_m","Splice depth\_m", "Age\_ky", "Al", "Si", "S", "K", "Ca", "Ti", "Mn", "Fe", "Rb", "Sr","Zr", "Ba", "col\_20", "Age\_ky\.1", "PC1", "PC2", "col\_24", "Age\_ky\.2","ODP 967 Dust proxy", "col\_27", "Age\_ky\.3", "ODP 967 wet\-dry index"\],"dtypes": \{"Site": "str", "Hole": "str", "Core": "float64", \.\.\., "Ca": "float64", \.\.\.\},"preview": \[\.\.\.\]\}\[step 3\]run\_python\_scriptdf = pd\.read\_parquet\("\.\.\./climateMeasurements\_\_Sheet1\.parquet"\)min\_dust = df\[’ODP 967 Dust proxy’\]\.min\(\)subset = df\[df\[’ODP 967 Dust proxy’\] == min\_dust\]min\_wet\_dry = subset\[’ODP 967 wet\-dry index’\]\.min\(\)final\_subset = subset\[subset\[’ODP 967 wet\-dry index’\] == min\_wet\_dry\]print\(final\_subset\[’Ca’\]\.iloc\[0\]\)\-\> 207878\.877358207\[step 4\]run\_python\_scriptsorted\_df = df\.sort\_values\(by=\[’ODP 967 Dust proxy’, ’ODP 967 wet\-dry index’\]\)print\(sorted\_df\[\[’ODP 967 Dust proxy’, ’ODP 967 wet\-dry index’, ’Ca’\]\]\.head\(10\)\)\-\> ODP 967 Dust proxy ODP 967 wet\-dry index Ca5370 0\.001 0\.059866 207878\.8773585369 0\.001 0\.396778 197141\.431580\[\.\.\.\]\[step 5\]$ echo "207878\.8774" \> \./answer\.txt

Baseline✗, where it failed to parse the header\.

\[step 1\]$ find \./data \-type f\-\> \(no output\)\[step 2\]$ ls \-la\-\> lrwxrwxrwx data \-\> /\.\.\./lake/kb\_all/unified\[step 3\]$ grep \-rilE "calcium\|dust proxy\|wet\-dry" \./data\-\> \(no output, exit code 1\)\[step 4\]$ grep \-RiL "calcium" \./data \|\| grep \-Ril "calcium" \./data\-\> \./data/noaa\_wildfires\_variabledescrip\.csv\./data/environmental\-justice\-populations\.csv\[\.\.\. 393 more files, climateMeasurements\.xlsx among them \.\.\.\]\[step 5\]$ cat << ’EOF’ \> search\.pyimport osimport pandas as pdfor root, \_, files in os\.walk\(’\./data’\):for f in files:if f\.endswith\(’\.csv’\):df = pd\.read\_csv\(path, nrows=5\) \[\.\.\.\]elif f\.endswith\(’\.xlsx’\):df = pd\.read\_excel\(path, sheet\_name=sheet, nrows=5\)cols = \[str\(c\)\.lower\(\) for c in df\.columns\]if any\(’calcium’ in c for c in cols\) or any\(’dust’ in c \.\.\.\):print\("Found in:", path, "Sheet:", sheet\) \[\.\.\.\]EOFpython3 search\.py\-\> \[An error occurred during execution\.\] Cannot execute multiple commands at once\.\[step 6\]$ cat << ’EOF’ \> search\.py && python3 search\.py\[\.\.\. same script, os\.walk\(’\./data’, followlinks=True\) \.\.\.\]\-\> \(no "Found in:" line, exit code 0\)\[step 7\]$ grep \-ril "dust proxy" \./data\-\> \(no output, exit code 1\)\[warning\]All 7 task steps are used\. Please write your answer now\.\[step 8\]$ echo ’0\.0000’ \> \./answer\.txt

Figure 7:Trajectories on the paleoclimate workbook task from Section[A](https://arxiv.org/html/2610.27327#A1), both with Gemini\-3\.1\-Pro as the Worker under a 10\-step budget\. Tool outputs and shell observations are abridged, with elisions marked by\[\.\.\.\]\. The finalfinishstep is omitted\.