Harvard’s latest AutoScientists is worth reading. It makes me think that AI doing research might not lead to “a single super AI scientist handling the entire process,” but rather an AI lab that organizes itself.

The core of this article is: having multiple agents share experimental state, automatically form teams around different hypotheses, review each other’s research plans before consuming real experimental resources, and identify problems in hypotheses, experimental designs, and validation paths early on; after experiments, both successes and failures are written back to shared memory.

This is very similar to a real lab: not one person rushing through alone, but a group of people continuously proposing ideas, doing experiments, questioning each other, recording dead ends, and then reallocating tasks based on results.

What I find most inspiring is that the key to AI handling complex tasks in the future may not just be how smart a single agent is, but whether these agents can form an effective organizational structure.

What truly matters is not “one-click paper generation,” but the ability to explore long-term, avoid repeating mistakes, and turn failure experiences into part of the system.

The future AI Scientist might be more like an agent team that holds its own meetings, divides work, and reviews progress.

Paper: https://arxiv.org/abs/2605.28655

#AutoScientists #MultiAgent #AIScientist #AIResearch #AIPapers #AutomatedResearch

Self-Organizing Agent Teams for Long-Running Scientific Experimentation

Source: https://arxiv.org/html/2605.28655 Shanghua Gao Harvard University [email protected] &Ada Fang11footnotemark:1 Harvard University [email protected] Marinka Zitnik Harvard University [email protected]

Abstract

Scientific research proceeds through iterative cycles of hypothesis generation, experiment design, execution, and revision. AI agents can automate parts of this process, but existing approaches typically follow a single research trajectory or coordinate through a central planner with fixed objectives. As a result, they struggle to sustain parallel exploration, adapt as experimental evidence changes, or preserve knowledge of failed directions over long-running experiments. We introduceAutoScientists, a decentralized team of AI agents for long-running computational scientific experimentation. Agents interpret a shared experimental state, self-organize into teams around promising hypotheses, critique proposals before using experimental compute, and share successes and failures to reduce redundant exploration. Under matched experimental budgets,AutoScientistsimproves over prior AI agents across biomedical machine learning, language-model training optimization, and protein fitness prediction. On BioML-Bench, spanning biomedical imaging, protein engineering, single-cell omics, and drug discovery,AutoScientistsachieves a mean leaderboard percentile of 74.4% across 24 tasks, improving over the strongest AI agent by +8.33%. On GPT training optimization,AutoScientistsreaches a target validation bits-per-byte 1.9×\timesfaster than Autoresearch and continues discovering improvements from a starting champion where the single-agent approach finds none (7 vs. 0 accepted improvements). On ProteinGym fitness prediction,AutoScientistsdiscovers a method for ACE2-Spike binding that improves over the current state-of-the-art model by +12.5% in Spearman correlation. Applied without modification across all 217 ProteinGym assays, the same method improves over the prior state of the art by +6.5% (Spearman correlation).

1Introduction

AI agents for science are beginning to move beyond answering questions and running predefined workflows toward proposing and executing research steps[1 (https://arxiv.org/html/2605.28655#bib.bib1)], from protein engineering in biology to language model optimization in machine learning[2 (https://arxiv.org/html/2605.28655#bib.bib2),3 (https://arxiv.org/html/2605.28655#bib.bib3)]. Agents can generate hypotheses, synthesize literature, design computational experiments, write and execute code, and refine models from experimental feedback[4 (https://arxiv.org/html/2605.28655#bib.bib4),5 (https://arxiv.org/html/2605.28655#bib.bib5),6 (https://arxiv.org/html/2605.28655#bib.bib6),7 (https://arxiv.org/html/2605.28655#bib.bib7),8 (https://arxiv.org/html/2605.28655#bib.bib8),9 (https://arxiv.org/html/2605.28655#bib.bib9),10 (https://arxiv.org/html/2605.28655#bib.bib10)]. However, most current approaches remain limited to short-horizon optimization or fixed pipelines. They typically follow a single reasoning thread or use a search-space decomposition set at the start of the run. This assumption breaks down in long-running scientific experimentation, where research directions are not known in advance and change over time.

Existing AI agents can run experiments, but long-running science requires more: maintaining competing hypotheses, updating them as evidence changes, and using failures to redirect the search. Single-agent systems such as AIDE[11 (https://arxiv.org/html/2605.28655#bib.bib11)]and Autoresearch[3 (https://arxiv.org/html/2605.28655#bib.bib3)]iteratively refine proposals but follow a single search trajectory, limiting their ability to explore competing hypotheses in parallel. Multi-agent systems[12 (https://arxiv.org/html/2605.28655#bib.bib12),13 (https://arxiv.org/html/2605.28655#bib.bib13),14 (https://arxiv.org/html/2605.28655#bib.bib14)]distribute work across agents, but still coordinate through a central structure: a planner decomposes the problem, a search algorithm ranks proposals, or agents converge through discussion or voting[15 (https://arxiv.org/html/2605.28655#bib.bib15),16 (https://arxiv.org/html/2605.28655#bib.bib16)]. These approaches assume that the search space can be partitioned into stable directions at the start of the run.In long-running experimentation, however, productive directions shift as evidence accumulates. Some hypotheses stop yielding improvements, failed directions must be tracked to avoid repeated exploration, and new hypotheses often emerge only after earlier experiments are analyzed.

Present Work.We introduceAutoScientists, a self-organizing agent team for long-running scientific experimentation that coordinates without a central orchestrator agent (Figure1 (https://arxiv.org/html/2605.28655#S1.F1)). Rather than receiving assignments from a planner, agents act on a shared state that records proposals, experiments, results, failures, and the current champion. Teams form dynamically through agent interaction rather than user-specified decomposition. Agents post experiment proposals to a shared forum, where peers critique them before execution, filtering weak ideas before compute is committed. As results accumulate, agents reorganize around productive directions, retire exhausted directions, and share successes and failures across teams to reduce redundant exploration. We applyAutoScientiststo research tasks spanning imaging, drug discovery, single-cell omics, protein engineering, protein fitness prediction, and language model training optimization.

Refer to captionFigure 1:Self-organizing agent teams for long-running experimentation.Overview ofAutoScientists. Agents identify promising research directions, organize into teams, and execute experiments in parallel.Across benchmarks,AutoScientistsimproves over existing AI agents. On BioML-Bench[2 (https://arxiv.org/html/2605.28655#bib.bib2)],AutoScientistsachieves the highest average leaderboard percentile among the evaluated agents, reaching74.40%74.40\%across 24 biomedical ML tasks compared with66.07%66.07\%for Autoresearch under the same task interface, model backend, and hardware budget. The performance improvements are largest in drug discovery, whereAutoScientistsimproves from46.16%46.16\%to64.52%64.52\%. On GPT nanochat training optimization[3 (https://arxiv.org/html/2605.28655#bib.bib3)],AutoScientistsreaches the same intermediate validation loss in 34 experiments that Autoresearch reaches in 65 experiments, and when continuing from aAutoScientistschampion reaches a validation bits-per-byte (bpb) of 0.9730 while Autoresearch finds no accepted improvements over 100 experiments. On ProteinGym supervised substitution fitness prediction[17 (https://arxiv.org/html/2605.28655#bib.bib17)],AutoScientistsstarts from Kermut and discovers a Kermut extension that improves ACE2–Spike binding Spearman’sρ\rhofrom0.7470.747to0.8400.840. Furthermore, the frozen recipe transfers across the full 217-assay ProteinGym supervised substitution benchmark, improving the official average Spearman’sρ\rhofrom0.6570.657to0.7000.700. Below we summarize our contributions:

• •A self-organizing agent team for long-horizon scientific experimentation. Unlike prior systems that rely on central coordinators, consensus-based discussion, or fixed decompositions of the search space,AutoScientistsallows agents to independently interpret a shared experimental state and decide which hypotheses to pursue. Agents post proposals to a shared forum where peers critique and filter them before experiments run, allowing teams and experimental directions to emerge through interaction rather than external assignment.
• •State-of-the-art performance across scientific domains, with sustained improvement during long-running experimental search.AutoScientistsimproves over prior agents on biomedical ML, protein fitness prediction, and language-model training optimization, and continues identifying productive modifications after single-agent baselines stop improving.

2Related Work

AI Agents for Scientific Research.AI agents are increasingly being developed to automate scientific workflows, including literature review, hypothesis generation, tool use, code execution, experimental design, benchmarking, and manuscript drafting[18 (https://arxiv.org/html/2605.28655#bib.bib18),19 (https://arxiv.org/html/2605.28655#bib.bib19),20 (https://arxiv.org/html/2605.28655#bib.bib20),21 (https://arxiv.org/html/2605.28655#bib.bib21),22 (https://arxiv.org/html/2605.28655#bib.bib22)]. Biomedical agents combine multi-step reasoning with biomedical tools, literature grounding, omics analysis, code execution, and evidence reconciliation[10 (https://arxiv.org/html/2605.28655#bib.bib10),6 (https://arxiv.org/html/2605.28655#bib.bib6),9 (https://arxiv.org/html/2605.28655#bib.bib9),23 (https://arxiv.org/html/2605.28655#bib.bib23),24 (https://arxiv.org/html/2605.28655#bib.bib24),25 (https://arxiv.org/html/2605.28655#bib.bib25)]. Other systems push toward longer-horizon discovery through repeated cycles of literature search, hypothesis generation, debate, refinement, tool integration, optimization, equation discovery, self-directed exploration, and skill accumulation[4 (https://arxiv.org/html/2605.28655#bib.bib4),7 (https://arxiv.org/html/2605.28655#bib.bib7),26 (https://arxiv.org/html/2605.28655#bib.bib26),27 (https://arxiv.org/html/2605.28655#bib.bib27),28 (https://arxiv.org/html/2605.28655#bib.bib28),29 (https://arxiv.org/html/2605.28655#bib.bib29),30 (https://arxiv.org/html/2605.28655#bib.bib30),31 (https://arxiv.org/html/2605.28655#bib.bib31),12 (https://arxiv.org/html/2605.28655#bib.bib12),32 (https://arxiv.org/html/2605.28655#bib.bib32)]. Several scientific-agent systems rely on role-specialized architectures, such as PI–scientist–critic organizations or Manager–Developer–Critic–Tool Creation pipelines[14 (https://arxiv.org/html/2605.28655#bib.bib14),5 (https://arxiv.org/html/2605.28655#bib.bib5)]. In AI research, related systems have also generated research papers or evolved algorithms through iterative code modification and experimentation[33 (https://arxiv.org/html/2605.28655#bib.bib33),3 (https://arxiv.org/html/2605.28655#bib.bib3),34 (https://arxiv.org/html/2605.28655#bib.bib34)]. Our system differs in that agents collectively determine research directions through discussion and coordinate through shared forums rather than fixed pipelines or a central orchestrator that directs others. Unlike debate frameworks that use discussion to converge on a shared hypothesis[15 (https://arxiv.org/html/2605.28655#bib.bib15),16 (https://arxiv.org/html/2605.28655#bib.bib16)],AutoScientistsuses discussion to filter out weak proposals before any experiment runs, while allowing agents to continue pursuing different research directions in parallel.

Coordination of Multi-Agent Systems.Beyond scientific applications, multi-agent performance depends strongly on collaboration structure and agent composition[35 (https://arxiv.org/html/2605.28655#bib.bib35),36 (https://arxiv.org/html/2605.28655#bib.bib36),37 (https://arxiv.org/html/2605.28655#bib.bib37),38 (https://arxiv.org/html/2605.28655#bib.bib38),39 (https://arxiv.org/html/2605.28655#bib.bib39),40 (https://arxiv.org/html/2605.28655#bib.bib40)]. Interaction is not automatically beneficial. For example, multi-agent systems have underperformed their best individual member on tasks[41 (https://arxiv.org/html/2605.28655#bib.bib41),40 (https://arxiv.org/html/2605.28655#bib.bib40)], and recent benchmarks analyse how collaboration and competition affect collective performance[42 (https://arxiv.org/html/2605.28655#bib.bib42)]. These findings motivate our ablation studies and comparison to single-agent baselines like Autoresearch. Human scientific teams provide a complementary perspective as they benefit from diversity and flatter structures, but excessive diversity can introduce coordination costs[43 (https://arxiv.org/html/2605.28655#bib.bib43),44 (https://arxiv.org/html/2605.28655#bib.bib44),45 (https://arxiv.org/html/2605.28655#bib.bib45),46 (https://arxiv.org/html/2605.28655#bib.bib46)]. Recent work also emphasizes context management, memory, and reusable skills for sustained collaboration[12 (https://arxiv.org/html/2605.28655#bib.bib12),47 (https://arxiv.org/html/2605.28655#bib.bib47),32 (https://arxiv.org/html/2605.28655#bib.bib32)]. Our system draws on these findings by organizing agents as teams focused on complementary research directions and uses shared forums to support conference-style knowledge sharing and collective intelligence[48 (https://arxiv.org/html/2605.28655#bib.bib48)].

3AutoScientists: Long-Running Self-Organizing Agent Teams

We proceed by formalizing long-running scientific experimentation as an iterative search process and introduceAutoScientists. We first define the optimization setting and then describe how agents organize into teams, propose and execute experiments, exchange experimental evidence through a shared state, and reorganize as search trajectories evolve over time.

3.1Problem Formulation

We are given a task description, optionally accompanied by an initial program (e.g., a training script)p0p_{0}, together with a datasetD\mathcal{D}and an evaluation metricl\ell.

The datasetD\mathcal{D}consists of a training setDtrain\mathcal{D}_{\text{train}}and an evaluation protocol. The evaluation protocol may take one of the following forms: a validation setDval\mathcal{D}_{\text{val}}, or a cross-validation (CV) scheme overDtrain\mathcal{D}_{\text{train}}. We denote byleval(p;D)\ell_{\text{eval}}(p;\mathcal{D})the evaluation metric computed under this protocol.

A system ofnnlong-running LLM agentsA={a1,…,an}\mathcal{A}=\{a_{1},\ldots,a_{n}\}iteratively proposes and generates new programs. Long-running agentsaia_{i}persist over the course of the search process, maintaining internal state and updating their behavior based on accumulated experience. This contrasts with one-shot agents that generate a solution in a single forward pass. Each proposed programppis trained onDtrain\mathcal{D}_{\text{train}}and evaluated usingleval\ell_{\text{eval}}. The goal is to identify a program

p∗=arg⁡maxp∈P⁡leval(p;D),p^{*}=\arg\max_{p\in\mathcal{P}}\ \ell_{\text{eval}}(p;\mathcal{D}),whereP\mathcal{P}denotes the space of programs explored by the agents during the search process, optionally initialized fromp0p_{0}. We assume without loss of generality thatl\ellis oriented so that higher values correspond to better performance (e.g., by negating metrics that are typically minimized such as loss).

At the end of the search process, performance is reported usingltest\ell_{\text{test}}if a held-out test set is available, otherwise,leval\ell_{\text{eval}}is used (e.g., validation or CV performance).

3.2AutoScientistsApproach

Overview.AutoScientistsdeploysnnlong-running agents that maintain state across the run, adap