Paper page - Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes

Source: https://huggingface.co/papers/2605.05724

Abstract

Auto research operates as an empirical loop where agents iteratively refine code based on evaluation feedback, achieving improved performance across multiple tasks without human intervention.

We study auto research as a closedempirical loopdriven by external measurement. Each submitted trial carries a hypothesis, an executable code edit, anevaluator-owned outcome, and feedback that shapes the next proposal. The output is not a generated paper or a single model checkpoint, but an auditable trajectory of proposals, code diffs, experiments, scores, and failure labels. We instantiate this loop withspecialist agentsthat partitionrecipe surfacesand share measured lineage across trials. The central empirical finding is thatlineage feedbacklets agents turn evaluator outcomes, including crashes, budget overruns, size failures, and accuracy-gate misses, into laterprogram-level recipe editsrather than one-shot suggestions. Across 1,197 headline-run trials plus 600 Parameter Golf control trials after one-time setup and launch, humans did not choose proposals, edit recipes, override scores, or repair failed trials during the search. In the three headline runs, the same submitted-trial loop reduces Parameter Golf validation bpb by 0.81%, raises NanoChat-D12 CORE by 38.7%, and reduces CIFAR-10 Airbench96 wallclock by 4.59%, with each task measured by its own external evaluator and legality checks. The trace includes a strictarchitecture-domain auditof 157 headline-run submissions andprogram rewritessuch as a NanoChat attention-kernel path change. Within this scope the loop autonomously writes code, submits experiments, absorbs feedback, applies and combines known techniques inside each environment, and improves public starting recipes.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2605\.05724

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.05724 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.05724 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.05724 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.