Can Generalist Agents Automate Data Curation?

# Can Generalist Agents Automate Data Curation?
Source: [https://arxiv.org/abs/2606.04261](https://arxiv.org/abs/2606.04261)
[View PDF](https://arxiv.org/pdf/2606.04261)

> Abstract:Curating training data is among the most consequential yet labor\-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data policies against noisy benchmark feedback\. We ask whether generalist coding agents can automate this data\-curation loop\. We introduce \*Curation\-Bench\*, an agent\-centric benchmark that fixes the model, training recipe, and evaluation suite while giving agents command\-line access to inspect data, implement policies, submit them to a fixed training/evaluation pipeline, and revise\. In a vision\-language instruction\-tuning instantiation, out\-of\-the\-box agents reach strong published data\-selection baselines within ten iterations\. However, trajectory analysis reveals a persistent \*execution\-research gap\*: agents mainly tune local policy variants rather than explore new policy families, even when given strategy guides and paper references\. Scaffolds requiring each iteration to cite, instantiate, and adapt a prior method shift agents toward method\-guided exploration\. The scaffolded agent autonomously composes \-\- without human design input \-\- a data\-selection policy that outperforms strong published baselines at one\-tenth their data budget\. Overall, current agents can run the curation loop, but reliable data research requires scaffolded method adaptation, not open\-ended prompting alone\. Code and benchmark are open\-sourced\.

## Submission history

From: Feiyang Kang \[[view email](https://arxiv.org/show-email/2c993ff5/2606.04261)\] **\[v1\]**Tue, 2 Jun 2026 22:26:53 UTC \(2,150 KB\)