SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery

# SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery
Source: [https://arxiv.org/html/2607.02807](https://arxiv.org/html/2607.02807)
Yuvraj Virk Zack Edds Chunqiu Steven Xia Lingming Zhang University of Illinois Urbana\-Champaign![[Uncaptioned image]](https://arxiv.org/html/2607.02807v1/resources/uiuc.png) \{yvirk2, zedds2, chunqiu2, lingming\}@illinois\.edu

###### Abstract

Long\-running coding agents such as autoresearch can persistently discover optimizations for open\-ended problems\. However, they tend to converge onto a single high\-level approach, then proceed with low\-level edits while missing other superior approaches to the problem\. We hypothesize two harness\-level design choices contribute to this behavior: accumulating context in a single long\-running agent and only exposing a single program state to edit\. We introduceSwarmResearch, an orchestrator\-subagent harness in which a Shepherd Agent usesglobal contextto steer a population of Search Agents, each operating withlocal contextin their respective git branch\. On open\-ended optimization tasks,SwarmResearchdiscovers better or comparable solutions to state\-of\-the\-art LLM\-guided evolution and multi\-agent techniques on 13/15 tasks, driven by higher\-level exploration\. Compared with fixed scaling of serial and parallel agents,SwarmResearch’s orchestrator\-guided scaling discovers better\-performing solutions by adapting parallelism at different search depths\. Code and details available at[https://github\.com/SwarmResearch/SwarmResearch](https://github.com/SwarmResearch/SwarmResearch)\.

![Refer to caption](https://arxiv.org/html/2607.02807v1/x1.png)Figure 1:SwarmResearchdesigns speculative decoding implementations\. Nodes represent spawned search agents and their solutions\. Edges indicate the lower node builds on the solution above\. The top node value shows spawn order; the bottom value shows speedup over vanilla decoding\.SwarmResearchexplores diverse approaches over∼\\sim11 hours without converging onto any one approach\. On 100 held\-out reasoning\-intensive tasks \(averaged across 5 seeds\),SwarmResearch’s achieved mean tok/s is 4\.58×\\timesfaster than naive vanilla decoding\. After∼\\sim12 hours,vanilla autoresearch’s\(Karpathy,[2026](https://arxiv.org/html/2607.02807#bib.bib16)\)speedup is only 1\.80×\\timeswhileCORAL’s\(Qu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib33)\)is 2\.26×\\times\. All methods use Opus 4\.8 Claude Code\. Details are in Section[4](https://arxiv.org/html/2607.02807#S4)\.## 1Introduction

Open\-ended research and engineering problems often have vast solution spaces and no known optimum\. An example is designing fast decoding algorithms for LLM inference: researchers pursue many different directions, from KV\-cache management\(Kwon et al\.,[2023](https://arxiv.org/html/2607.02807#bib.bib18)\)to improved speculative decoding schemes\(Leviathan et al\.,[2023](https://arxiv.org/html/2607.02807#bib.bib20)\)\. One way they evaluate progress is by end\-to\-end algorithm speed\. These directions can be viewed as distinct regions of an optimization landscape, where some regions correspond to weak approaches which contain poor local optima, while others contain substantially faster solutions\. Making steady progress therefore requires continuous exploration of diverse, high\-level approaches instead of immediately committing to a single neighborhood of the search space\.

Test\-time scaling\(Brown et al\.,[2024](https://arxiv.org/html/2607.02807#bib.bib4)\)has shown promise in eliciting LLMs’ ability to search open\-ended solution spaces\. LLM\-guided evolution iteratively samples an LLM to generate a new solution given select context on solutions generated so far; heuristic algorithms control exact context to guide search behavior\(Novikov et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib30); Sharma,[2025](https://arxiv.org/html/2607.02807#bib.bib39); Liu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib23); Cemri et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib6); Lange et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib19)\)\. However,long\-running coding agentsmay support more adaptive and complex search by better expressing frontier model capabilities than heuristic LLM\-guided evolution\. These agents can iteratively implement and test complex code, decide when to refine an existing approach versus pivot to a new direction, and use external tools such as web search\. Autoresearch\(Karpathy,[2026](https://arxiv.org/html/2607.02807#bib.bib16)\)is a popular example: it runs off\-the\-shelf coding agents such as Claude Code for hours or days in an experimental loop\. One iteration implements changes, evaluates them, commits improvements, and reverts regressions\. Autoresearch has inspired projects across domains with impressive results in systems optimization\(Lütke,[2026](https://arxiv.org/html/2607.02807#bib.bib26)\), kernel optimization\(RightNow\-AI,[2026](https://arxiv.org/html/2607.02807#bib.bib36)\), and prompt injection\(Panfilov et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib31)\)\.

Yet, long\-running coding agents tend to converge quickly on a single high\-level approach, then spend hours making small edits and micro\-optimizations\. Rather than exploring diverse directions, they can become trapped in a suboptimal region of the search space\. We view this as a harness\-engineering problem: context management and program versioning bottleneck divergent discovery\. As agents accumulate long conversation histories filled with incremental refinements on one approach, substantial shifts in approach become unlikely\. By committing all improvements to a single editable program, coding agent experimental loops do not preserve alternative directions for further exploration, effectively turning the process into greedy local search\.

Instead,SwarmResearchuses an orchestrator to delegate experiment ideation and execution to subagents with fresh or controlled context windows, and it assigns all major edits to separate git branches\.SwarmResearchis implemented as three skills compatible with popular coding agent tools such as Claude Code\. AShepherd Agentspawns and steers a population ofSearch Agents\. Each Search Agent proposes and executes new ideas given minimal context, while the Shepherd Agent maintains global context over all Search Agent activity and strategically steers the population toward productive, diverse exploration\.

We compareSwarmResearchon state\-of\-the\-art LLM\-guided evolution\(Liu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib23)\)and multi\-agent techniques\(Qu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib33)\)on 15 open\-ended optimization tasks across mathematics, heuristics, and competitive heuristic engineering\.SwarmResearchexceeds or matches these techniques on 13/15 tasks\. It typically makes 1\.7×\\timesto 3\.2×\\timestimes larger code changes, reflecting higher\-level exploration of approaches\. We analyzeSwarmResearchfrom a test\-time scaling perspective\. By using an orchestrator to autonomously vary the number of parallel agents at different search depths, it exceeds the performance of optimal fixed scaling of parallel and serial agents on 4/5 tasks\. Finally, we present a detailed case study on speculative decoding to illustrateSwarmResearch’s capabilities and limitations\.

![Refer to caption](https://arxiv.org/html/2607.02807v1/x2.png)Figure 2:Overview ofSwarmResearch\.Implements an orchestrator\-subagent loop:\(1\)Shepherd Agentspawns concurrent search agents with 3 per\-agent steering mechanisms: selecting a parent solution by setting up its git branch, writing a prompt to share minimal context, and selecting the Search Agent type\.\(2\)Search Agentsideate, implement, evaluate, and iterate on a solution\.Explorer Search Agentshave fresh context windows so they’re unanchored to prior work, whileOptimizer Search Agentsfork their parent’s conversation history to continue along its detailed history of attempts\. The Shepherd Agent receives summaries back from all Search Agents to strategically steer new ones\.
## 2Approach

We proposeSwarmResearch, an orchestrator\-subagent harness in which a Shepherd Agent uses global context to steer a population of Search Agents, each operating in their respective git branch with fresh or controlled local context\. Figure[2](https://arxiv.org/html/2607.02807#S1.F2)describes an overview of the agent architecture\. Our design is centered around two problems in long\-running coding agents that bottleneck divergent solution discovery: accumulating context around one approach and only editing one program state\.

First, early exploration by long\-running coding agents often only covers a small number of high\-level approaches\. Once one approach appears promising, the agent proceeds with a long\-tail of low\-level edits and micro\-optimizations on it\. Experimenting with new high\-level approaches requires several rounds of large program rewrites, which are unlikely when the agent is conditioned on a long history of incremental refinements to the current approach\. As shown in Figure[2](https://arxiv.org/html/2607.02807#S1.F2), to reduce this bias, the Shepherd Agent delegates experiment ideation and execution to Search Agents with fresh or controlled context windows\.

Second, long\-running coding agents edit a single program state: all improvements are committed, regressions are reverted, and future search proceeds from the current best implementation\. This loop design makes maintaining competing high\-level directions difficult and encourages greedy local search\. Ideally, an agent harness should let search branch naturally, so that multiple different versions of a program can evolve independently when they instantiate different ideas\. As shown in Figure[2](https://arxiv.org/html/2607.02807#S1.F2), the Shepherd Agent assigns a new git branch for every Search Agent so all major edits occupy different versions\. This way, promising yet not top\-performing solutions are not prematurely discarded and can be refined\. We describe the harness design, context management, and git management strategies in detail below\. See Appendix[C](https://arxiv.org/html/2607.02807#A3)for the full skills\.

### 2\.1Architecture

TheShepherd Agentis an orchestrator responsible for spawning and steering waves of concurrent search agents, whileSearch Agentsare subagents responsible for ideating a new approach, implementing, evaluating, and iterating on it\. The Shepherd Agent is given a Search Agent budget to manage\. Unlike classic orchestrator\-subagent architectures, which partition complex tasks such as large\-scale code generation across subagents, the Shepherd Agent does not decompose the open\-ended problem for handoff\. Instead, it steers population\-level search behavior: our skills emphasize initializing and maintaining a diverse population of ideas, prioritizing search agent budget on promising ideas, and breaking out of plateaus\. The Shepherd Agent accomplishes these goals using 3 steering mechanisms \(Figure[2](https://arxiv.org/html/2607.02807#S1.F2); Step 1\) when spawning new Search Agents: parent selection, search agent type, and prompts\.

Parent Selection\.For every new Search Agent, the Shepherd Agent creates a new git branch and worktree from either a task’s baseline solution or a completed Search Agent’s commit\. Since each Search Agent begins by reading and modifying thisparent solutionin its worktree, that parent solution determines the neighborhood of solutions the agent explores\. Parent selection lets the Shepherd Agent steer population search without prescribing specific ideas\. Branches from the initial baseline will encourage new from\-scratch exploration, while branches from completed agents will direct effort toward promising or underdeveloped directions\.

In traditional evolutionary methods\(Novikov et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib30); Sharma,[2025](https://arxiv.org/html/2607.02807#bib.bib39); Lange et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib19)\), parent selection is also used to direct exploration\-exploitation behavior but is completed by a heuristic algorithm rather than an LLM\. In existing single and multi\-agent approaches\(Karpathy,[2026](https://arxiv.org/html/2607.02807#bib.bib16); Qu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib33)\), parent selection is implicit\. Agentsindependentlyreflect on their conversation history and shared memory to decide which solution to expand\. In contrast,SwarmResearchuses an orchestrator to distribute parents strategically rather than relying on uncoordinated agent decisions\.

Search agent type\.Explorer Search Agentshave fresh context windows and use the content in their worktree as context\. Their goal is to explore new approaches\.Optimizer Search Agentsfork their parent agent’s conversation history, in addition to using worktree content as context\. Their goal is to make a few refinements to the parent solution\.

Prompts\.Since Search Agents only have local context of their parent, the Shepherd Agent can supplement their context through prompts that share minimal, relevant context or add directions\. For example, if Explorer Agents repeat similar ideas, prompts can provide a summary of already explored ideas and instruct them to explore new ideas\. A prompt could also include complementary findings by other Search Agents\. On plateaus, the prompt could include a briefing of strengths and weaknesses of the best approaches, with a suggestion to identify novel techniques targeting limitations\. We found that restricting the Shepherd Agent from prescribing specific ideas in prompts helped maintain idea diversity; otherwise, the Shepherd Agent itself could get stuck inside a basin of ideas\.

### 2\.2Context Management

![Refer to caption](https://arxiv.org/html/2607.02807v1/x3.png)Figure 3:SwarmResearchcontext management for search agents\.SwarmResearchseparates context into two tiers\. Search Agents operate withlocalcontext: the contents of their worktree and, for Optimizer Agents, the conversation history of their direct parent\. The Shepherd Agent operates withglobalcontext: summaries of all Search Agent attempts, including their approach and evaluated score\. This separation is designed to preserve diversity among Search Agents while still giving the Shepherd Agent enough information to strategically steer the population\.

A Search Agent’s persistent state is primarily captured by its worktree\. As shown in Figure[3](https://arxiv.org/html/2607.02807#S2.F3), Explorer Agents start with fresh context windows, so their only context is the contents of the worktree they are assigned\. Fresh context makes each agent less anchored to previous work and better suited for trying higher\-level changes in approach\. Optimizer Agents inherit their parent’s conversation history in addition to the worktree, giving them access to a detailed log of successes and failures to continue long serial refinement when an approach appears promising\. Before committing their changes, each Search Agent appends a short summary of its approach and evaluation results to afindings\.mdfile\. It serves as a lineage\-local log of attempts and outcomes, giving future Search Agents lightweight access to their ancestors’ attempts without anchoring them in a long conversation history\.

Our orchestrator\-subagent design addresses two failure modes of context accumulation\. First, standalone coding agents tend to accumulate long conversation histories that repeatedly refine a small number of high\-level approaches\. These histories can make larger pivots less likely, because the agent is repeatedly conditioned on the current approach and its incremental refinements\.SwarmResearchavoids this diversity loss by spawning Explorer Agents with fresh context windows, while still supporting long\-tail improvement through Optimizer Agents that inherit detailed conversation history when serial refinement is useful\.

Second, multi\-agent systems that self\-organize through a shared memory e\.g\. CORAL\(Qu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib33)\), rather than an orchestrator, also converge onto a single high\-level approach\. When all agents see that another agent found a stronger solution, they decide to improve the best solution and abandon independent directions\.SwarmResearcheliminates this possibility by restricting Search Agent context to their direct lineage, rather than exposing them to all lineages\. At the same time, global information remains useful for strategy, by revealing which approaches are promising, which fall behind, and where the population lacks coverage\.SwarmResearchincorporates this information through the Shepherd Agent, which uses global context to distribute Search Agents across parents and write prompts that selectively pass relevant information to new Search Agents\.

![Refer to caption](https://arxiv.org/html/2607.02807v1/x4.png)Figure 4:Git management for Search Agents\.
### 2\.3Git Management

The Shepherd Agent assigns every Search Agent to a new git branch and worktree\. Themainbranch contains only the initial task setup: aprompt\.mdfile defining the problem and a minimal working baseline solution\. As shown in Figure[4](https://arxiv.org/html/2607.02807#S2.F4), branches can start frommainto try a new from\-scratch approach, from a completed Search Agent’s commit to iterate on a prior solution, or merge multiple parent branches to combine different solutions\. Branching allows the Shepherd Agent to express a wide range of search patterns\. For example, it can spawn multiple concurrent agents from the same parent to fan out around a promising solution, or spawn a serial chain of explorers and optimizers to build on a lineage’s successes and failures\. Since every Search Agent is versioned, fan\-out and serial iteration can be flexibly interleaved across time and distant regions of the search space\.

The branching approach inSwarmResearchcontrasts with existing long\-running coding agents, which usually make successive edits to a single program on a single branch\. Since improvements are kept and regressions are reverted, the agent’s search trajectory becomes a greedy sequence of local refinements\.SwarmResearch’s branch\-based versioning instead preserves competing approaches and avoids collapsing the search back into the current best implementation, making it easier to revisit, extend, combine, or abandon approaches\.

## 3Evaluation

### 3\.1Experimental Setup

Benchmark\.Our benchmark consists of open\-ended optimization problems with no known optimum\. They fall in 3 domains: math, systems, and heuristic algorithms\. Most mathematical optimization tasks require writing optimization programs to generate constructions demonstrating new bounds on a problem\. We use 5 math tasks also used by prior work\(Liu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib23); Qu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib33); Novikov et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib30)\)\. Systems tasks are from the ADRS benchmark\(Cheng et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib8)\), which is based on real\-world systems research\. Our math and systems tasks match the benchmark suite used by our multi\-agent baseline CORAL\(Qu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib33)\)\(excluding MMD\-16\-2 for cost because of its similarity to MMD\-14\-3\)\. Heuristics tasks are from ALE\-Bench\-Lite\(Imajuku et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib14)\), which pulls problems from the AtCoder Heuristics Contest\. These competitive heuristic algorithm problems are inspired by optimization challenges from industry\. Our 5 problem subset was selected by top\-level solution categories that reward high\-level algorithm exploration over tuning of generic optimization patterns\. We use ALE\-Bench’s suggested performance metric which typically ranges from 0 to 3500\. This metric is derived from the relative rankings of competition participants for a problem\. All tasks include a minimal working baseline solution\.

Baselines and configurations\.We compareSwarmResearchto the top\-performing evolutionary baseline EvoX\(Liu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib23)\)and the state\-of\-the\-art multi\-agent system CORAL\(Qu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib33)\)\. EvoX adapts search behavior by generating parent selection programs on\-the\-fly using an LLM\. In CORAL, concurrent coding agents run experimental loops and use a shared filesystem\-based memory to exchange insights and build on one another’s successes and failures\. Periodic prompts instruct agents to reflect, pivot on plateaus, and consolidate findings into memory\. We configure CORAL to use 4 agents, as in their paper\. Both CORAL andSwarmResearchare ran until a $50 budget is exhausted\. EvoX is executed for 100 iterations as configured in their paper, after which it has diminishing returns, with average task cost $23\.50\. Both CORAL and Shepherd use Claude Code as the agent runtime\. Note that CORAL andSwarmResearch’s solution performances do not converge within the budget but reach a point of diminishing returns\. CORAL settles on a top approach and subsequent gains would result from long\-tail polishing\. No approach has internet access\. CORAL andSwarmResearchonly have access to evaluators as APIs so their internals are hidden\.

We note that EvoX achieves poorer performance with Opus 4\.6 on some problems than presented in their original paper which used GPT\-5\. Small\-scale experiments indicate that EvoX performs significantly better using GPT models, so its ranking may change if the latest GPT models are used\. We only report Opus 4\.6 results due to cost limitations and since model dependency is a real limitation of the approach\. Additionally, our results obtained by reproducing CORAL with their publicly available implementation is worse than their originally reported results\. However, we include their originally reported results as the state\-of\-the\-art where applicable\. Finally, we run every technique once per task\. Although one run does not give a stable estimate of every technique’s per\-task performance, obtaining stable estimates are unrealistically expensive\. Given realistic budget constraints and the stochastic nature of LLM\-based discovery techniques, one long run per task achieves the mostrepresentativeresults as per suggested use by each baseline\. For a more detailed discussion of between\-run variance, see Appendix[B](https://arxiv.org/html/2607.02807#A2)\.

Granularity of Code Changes\.Low\-level changes appear as targeted edits to a parent program\. High\-level changes in approach appear as rewrites of a parent program and thus associated with a higher number of lines of code changes\. ForSwarmResearchand CORAL, we compare the lines changed between every commit and its parent commit\. EvoX only logs programs that improved over their parents, so we only take the changed lines of code for those programs and their parent\. For each task, we compute the mean lines of code changed per attempt by a given method\. Then, we compare the median of these task\-level means\.

Comparison to Optimal Fixed Scaling\.Fixed scaling is our naive test\-time scaling baseline\. By fixed scaling, we mean executingnnconcurrent agents forkkserial iterations\. An iteration consists of one\-shot generating/editing a solution and evaluating it\. On the other hand,SwarmResearchuses orchestrator\-guided scaling, where an orchestrator dynamically decides when to launch a serial agent off a parent or a parallel agent off another parent\. Does orchestrator\-guided scaling exceed the performance of fixed scaling? Instead of completely independent agents, we find sharing a git history between them performs better\. Shared git history enables parallel agents to switch to refining another agent’s better solution\. Here, an iteration consists of exploring the shared git history, generating a solution, evaluating it, and making a descriptive commit back to the shared git history\. This way, separate agents can switch to better approaches discovered by other agents\. Concurrent agents are synchronized at each iteration so they make consistent use of the shared git history\. Fixed scaling is tested using the minimal coding agent harness Pi\(Zechner,[2025](https://arxiv.org/html/2607.02807#bib.bib46)\)with Minimax\-M2\.5\. We test 5 configurations of\(n,k\)\(n,k\)within a 60 total iteration budget: \(5, 12\), \(10, 6\), \(15, 4\), \(20, 3\), \(30, 2\)\. Results are averaged over 3 runs and shown for the best\-performing configuration\.

To isolate the effect of using an orchestrator\-subagent architecture, we use a minimalSwarmResearchharness in Pi\. The orchestrator has a subagent tool that only accepts abranchfield for parent selection\. The orchestrator can not write prompts for subagents and Explorer Agents are the only subagent type\. Spawning a subagent runs one serial iteration off the parent without access to a shared git history\. Subagents use Minimax\-M2\.5 while the orchestrator uses Claude Sonnet\-4\.6; Minimax\-M2\.5 struggled to perform orchestration restricted to branch selection\. We note again that the orchestrator does not contribute any ideas for subagents to pursue\. Its subagent tool only gives a branch field for selecting parent solutions\. Therefore, the model configuration faithfully tests whether using a sufficiently powerful orchestrator to guide scaling exceeds fixed scaling\. The orchestrator also has a 60 iteration budget and results are averaged over 3 runs\.

### 3\.2Comparison to Baselines

As shown in Table[1](https://arxiv.org/html/2607.02807#S3.T1),SwarmResearchexceeds the evolutionary baseline EvoX on 13/15 tasks and matches it on 1 task\.SwarmResearchstrictly exceeds the multi\-agent baseline CORAL on 10/15 tasks and matches it on 2 tasks\. However, differences between CORAL andSwarmResearchon the mathematical optimization tasks Circle Packing, Erdos Min Overlap, and MMD\-14\-3 are small\. They could be reduced by further solution polishing under higher budgets\.SwarmResearchsucceeds by exploring diverse high\-level approaches while CORAL experiments with diverse but low\-level changes on fewer high\-level approaches\. For example, in Figure[5](https://arxiv.org/html/2607.02807#S3.F5),SwarmResearch’s median sized diff involves trying a new type of signal processing algorithm while CORAL makes a targeted change to how signal corrections are computed\. As another example, in AHC016, a median sized diff from CORAL involves tuning thresholds while a median size diff fromSwarmResearchredesigns the solver into different modes applied on different conditions\. On most tasks, higher\-level experiments meanSwarmResearchdiscovers better solutions\. However, within the budget, CORAL better exploited strong approaches and discovered meaningful low\-level changes on the systems task EPLB and heuristics task AHC026\.

Table 1:Comparison ofSwarmResearchto state\-of\-the\-art Evolutionary and Multi\-Agent Baselines\.*We run each approach once with Opus 4\.6 and publicly available implementations*\.SwarmResearchand multi\-agent baseline CORAL use Claude Code for $50 per task\. EvoX runs for 100 iterations \($23\.50 per task on average\)\.Boldedmeans top score among these methods\. Highlighted ingreenmeans score matches or exceeds state\-of\-the\-art set by AI systems or humans\.![Refer to caption](https://arxiv.org/html/2607.02807v1/x5.png)

![Refer to caption](https://arxiv.org/html/2607.02807v1/x6.png)

Figure 5:Examples of median\-sizedSwarmResearchand CORAL diffs forSignal Processing\.State\-of\-the\-art results on the math and systems tasks are all claimed by AI systems, though many are difficult to verify because their solutions are not publicly released\. On the other hand, the state\-of\-the\-art results on heuristics tasks are all human\-authored, and AI systems still lag behind\. While math and systems tasks are more widely studied and more standard solution patterns are successful, heuristic tasks are competition problems whose best solutions have been intensively optimized by expert humans using creative, problem\-specific approaches\. We also reiterate that heuristic task scores indicate competition skill level: a score of 3000 versus 1500 does not imply a 2×\\timeshigher objective metric, but corresponds to a higher Elo\-like performance level where a 3000\-rated competitor would outperform a 1500\-rated competitor with high probability\.

SwarmResearch’s performance exceeds or matches evolutionary and multi\-agent baselines on 13/15 open\-ended optimization tasks, indicating more effective search of their solution spaces\.SwarmResearchexceeds EvoX on 13 tasks while matching it on 1 task\.SwarmResearchexceeds CORAL on 8 tasks while performing similarly on 5 tasks\.

### 3\.3Granularity of Code Changes\.

![Refer to caption](https://arxiv.org/html/2607.02807v1/x7.png)Figure 6:Median lines of code changed per attempt bySwarmResearchand baselines\.As shown in Figure[6](https://arxiv.org/html/2607.02807#S3.F6),SwarmResearchtypically makes larger code changes than the baselines\. This difference is most prominent in heuristics tasks and some systems tasks like Transaction Scheduling\. These tasks have large baseline solutions, and implementing new approaches require more code than other tasks\. On mathematical optimization tasks like Circle Packing, experiments involve smaller changes like hyperparameter tuning or optimizer tweaks\. Since low\-level changes appear as targeted edits while high\-level changes in approach appear as larger code changes, this result confirms our context and git management strategies increase the share of experiments that test high\-level changes\. Although high\-level experiments are productive for most tasks, they can be inefficient when they lead nowhere and inference is better spent on optimizing strong approaches\.

SwarmResearch’s experiments test more high\-level changes than the baselines\.For the average attempt and median task,SwarmResearchchanges 3\.2×\\timesmore lines than CORAL and 1\.7×\\timesmore than EvoX\.

### 3\.4Comparison to Fixed Scaling

Table 2:Comparison of Orchestrator\-guided Scaling to Fixed Scaling\. The depth of orchestrator\-guided scaling is the longest chain of serially spawned search agents, and its width is the average number of parallel agents spawned at each depth\.Table[2](https://arxiv.org/html/2607.02807#S3.T2)compares orchestrator\-guided scaling to optimal fixed scaling\. Experiments uses Minimax\-M2\.5 as the primary model and only 5 tasks since searching for the optimal scaling configuration is costly \(details are in section[3\.1](https://arxiv.org/html/2607.02807#S3.SS1)\)\. Table[2](https://arxiv.org/html/2607.02807#S3.T2)shows that optimal fixed scaling is wider than deeper for 4/5 tasks\. However, the most wide configuration\(30,2\)\(30,2\)is never optimal\. Overall, we find that shorter parallel runs reach better performance than longer serial runs, up to a point where higher parallelism is not worth it\. In longer serial runs, agents successfully refine an approach but stay within the same neighborhood of solutions and miss stronger neighborhoods discovered by highly parallel runs\. Additionally, the optimal number of parallel agents and serial iterations under fixed budget varies with task\.

Orchestrator\-guided scaling, which requires no scaling hyperparameter selection, exceeds optimal fixed scaling on 4/5 tasks\. In fixed scaling, the maximum number of parallel and serial iterations on top of any solution is fixed\. However, orchestrator\-guided scaling is more flexible\. The orchestrator can decide to allocate the iteration budget to wide initial parallel exploration\. Then, it can selectively apply deep serial iteration to the most promising solutions\. In effect, the agent can express varying width at different depths\. For example, orchestrator\-guided scaling’s max depth is much higher than fixed scaling forSignal Processingwhich allows it to iterate on the best solution for longer\. The 60 subagents dominate inference cost, so the orchestrator only increases total output tokens by 7\.7% over subagent\-only\.

Orchestrator\-guided scaling exceed optimal fixed scaling\.By varying parallelism at different depths, orchestrator\-guided scaling achieves better performance than optimal fixed scaling on 2/3 mathematical optimization tasks and 2/2 systems optimization tasks\.

### 3\.5Shepherd Agent Behavior

With minimal prompting, typical Claude Opus 4\.6 Shepherd Agent behavior involves launching waves of∼\\sim4\-8 Search Agents\. Waves mostly consist of Explorer Agents, while Optimizers Agents are spawned when dominant solutions emerge\. Its default search behavior is near\-greedy\. The Shepherd Agent begins by spawning concurrent explorers and keeps a few high\-level approaches open\. Then, it assigns concurrent search agents to iterate on the top\-few solutions\. Eventually, the Shepherd Agent concentrates Search Agents onto the single, top approach\. It very rarely merges agent branches\. Its prompts to Search Agents typically prescribe specific ideas to pursue\. On plateaus, the Shepherd Agent spawns more concurrent Explorer Agents from the best solution but also from the baseline, with context on the top\-performing approach\.

We prompt out negative Shepherd Agent behaviors like assigning specific ideas to Search Agents and collapsing onto one approach, while reinforcing positive behaviors like focusing Search Agents on discovering new approaches from the baseline on plateaus\. However, even with thorough prompting, the Shepherd Agent’s ability to strategically steer Search Agents without prescribing exact ideas is limited\. For example, it struggles to synthesize limitations into prompts or pose specific problems to different Search Agents based on current bottlenecks\. Improving an agent’s ability to plan strategic experiments is an open problem that would advance discovery ability\.

## 4Case Study: Approximate Speculative Decoding

We use speculative decoding as a detailed case study to illustrateSwarmResearch’s capabilities and limitations\. Figure[1](https://arxiv.org/html/2607.02807#S0.F1)visualizesSwarmResearch’s search trajectory for the task\.

Task & Setup\.Speculative decoding techniques are commonly lossless\(Leviathan et al\.,[2023](https://arxiv.org/html/2607.02807#bib.bib20); Hu et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib12); Li et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib21)\), where the target model verifies every draft token to ensure the output exactly matches what the target model would have generated\. However, strict per\-token verification may not be necessary for preserving downstream task accuracy, and trading some accuracy for higher speed may be desirable\(Kim et al\.,[2023](https://arxiv.org/html/2607.02807#bib.bib17); Huang et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib13); Lu et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib25)\)\. To explore this direction withSwarmResearch, our fast evaluator includes 30 total reasoning\-intensive tasks from LiveCodeBench v6, AIME 2026, GPQA Diamond, and HLE\-MCQ\(Jain et al\.,[2024](https://arxiv.org/html/2607.02807#bib.bib15); MathArena,[2026](https://arxiv.org/html/2607.02807#bib.bib27); Rein et al\.,[2023](https://arxiv.org/html/2607.02807#bib.bib35); Phan et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib32)\)\. The objective ofSwarmResearchis to maximize token throughput while preserving benchmark accuracy\. The baseline is a basic single\-file, lossless speculative decoding implementation using rejection sampling \(1\.68× faster than vanilla decoding\)\.SwarmResearchuse Claude Code Opus 4\.8\. Before solution generation, an initialSwarmResearchrun analyses the baseline from different angles without generating solutions and its findings are included in the solution generation run worktrees\. See details in Appendix[A](https://arxiv.org/html/2607.02807#A1)\.

SwarmResearch’s Solutions\.On 100 held\-out tasks, averaged across 5 seeds,SwarmResearchachieves a mean tok/s that is 4\.58×\\timesfaster than naive vanilla decoding, with 60\.6% accuracy, while vanilla decoding’s baseline accuracy is 65\.8%\. Vanilla autoresearch\(Karpathy,[2026](https://arxiv.org/html/2607.02807#bib.bib16)\)achieves 1\.80×\\timesspeedup and CORAL\(Qu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib33)\)achieves 2\.26×\\timesspeedup, with accuracies of 65\.8% and 58\.4% respectively\. These methods generated solutions for∼\\sim12 hours\. ForSwarmResearch, accuracy loss is due to more responses exceeding our evaluator’s 16,384\-token limit, despite some reasoning chains including the correct answer\.

The largest throughput gains with minimal accuracy loss come from systems optimizations\. The best\-performing optimization is batching the target forward pass across multiple benchmark sequences\. To preserve accuracy for the evaluated sampling seed,SwarmResearchuses independent RNG streams across batches for sampling independence\. Autoresearch and CORAL’s solutions do not use batching\. Figure[1](https://arxiv.org/html/2607.02807#S0.F1)showsSwarmResearch’s 23rd completed Search Agent applied batching on a fresh branch from main, even after significant serial exploration on approaches like relaxed acceptance conditions\. Additionally,SwarmResearch’s best solution optimizes the construction of token distributions for rejection sampling by using a persistent thread pool, top\-kkbounded distributions, and parallel distribution construction during batching\. The best solution additionally adapts the number of target\-model experts\. The baseline usesk=8k=8experts throughout, butSwarmResearchfinds thatk=8k=8is only necessary during prefill\. For target\-token generation,SwarmResearchusesk=4k=4for most tokens and increases tok=5k=5when the previous verification round’s mean top\-1 probability is less than 0\.7\. Unlike batching and distribution\-construction optimizations, adaptive MoE expert count is not strictly lossless with respect to the original target model\.

Existing inference engines\(Zheng et al\.,[2024](https://arxiv.org/html/2607.02807#bib.bib47)\)already implement batching for speculative decoding and optimized token\-distribution processing, while adaptive expert count has been explored outside speculative decoding\(Balsamo,[2026](https://arxiv.org/html/2607.02807#bib.bib3)\)\. Other than systems optimizations,SwarmResearchexplores decoding techniques that trade speed and accuracy, including relaxed draft acceptance and skipping verification for confident tokens\. Prior work explores related techniques\(Kim et al\.,[2023](https://arxiv.org/html/2607.02807#bib.bib17); Huang et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib13); Lu et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib25)\)\. Novel attempts, not in existing decoding literature as far as we know, perform poorly \(see risk and challenges for details\)\.

In one relaxed\-acceptance variant, a draft token is accepted directly if: \(1\) it is also the target model’s top\-1 token, and \(2\) either the target is low \(< 1\.098\), or the target top\-1 probability exceeds the target top\-2 probability by a large margin \(≥\\geq0\.30\)\. Condition 1 essentially implements greedy speculative decoding\. This does not preserve the intent of rejection sampling, but it increases the draft\-token acceptance rate\. Condition 2 is intended to identify tokens that are less likely to be critical for accuracy, at least based on single\-seed observations\. This approach is not used by the final solution because it causes accuracy loss when combined with batching\. When combined with the final solution and evaluated on the 30\-task benchmark across 5 seeds, this approach increases mean tok/s by 9%, and increases the draft acceptance rate from 76\.8% to 84\.8% while reducing accuracy from 91\.3% to 83\.3%\. Most accuracy loss is due to hitting the benchmark’s token cap\.

Risks and ChallengesSwarmResearch’s novel attempts, not in existing decoding literature as far as we know, performed poorly and we did not find them reasonable\. For example, one attempt was "comonotone sampling": during rejection sampling for a draft span, it sampled a single shared acceptance threshold for all tokens in the span, rather than sampling independent thresholds per token as is typical\. The model produced a run with improved performance and reasoned the change increased the acceptance rate\. However, after we generated acceptance rate data and evaluated additional seeds, we found its claims were unsubstantiated\.SwarmResearchgenerates a high throughput of ideas and experiments and careful review requires high effort\. Without careful review, users may be convinced by low\-quality approaches, where proposals and justifications seem attractive at first glance despite being incorrect\.

## 5Related Work

LLM\-Guided Evolutionary Discovery\.A growing number of work are developing LLM\-based search and discovery approaches\(Romera\-Paredes et al\.,[2024](https://arxiv.org/html/2607.02807#bib.bib38)\)\. Initially, the focus has been on using LLM as a mutation operator and combining it with heuristics as part of an evaluator\-guided evolutionary loop\(Novikov et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib30); Sharma,[2025](https://arxiv.org/html/2607.02807#bib.bib39); Lange et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib19)\)\. These LLM\-guided evolutionary algorithms follow a pipeline process of selecting parents using predefined heuristics based on evaluation feedback and then use an LLM to generate the next round of mutation\. AlphaEvolve\(Novikov et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib30)\)selects new candidates for future mutation using the MAP\-Elites algorithm\(Mouret & Clune,[2015](https://arxiv.org/html/2607.02807#bib.bib28)\)\. Follow up work improves search algorithms with techniques like adaptive sampling\(Lange et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib19); Sharma,[2025](https://arxiv.org/html/2607.02807#bib.bib39)\), pareto frontiers\(Agrawal et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib1)\), island\-based architectures\(Assumpção et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib2)\), and momentum\-based backtracking\(Yan et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib43)\)\. Compare to using fixed candidate selection and variation mechanisms, AdaEvolve\(Cemri et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib6)\)and EvoX\(Liu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib23)\)also evolve the search strategy itself\. However, these works also use LLMs inside a fixed pipeline rather than leveraging their capabilities as open\-ended agents\. Additionally, other work focus on training LLMs to perform better discovery\. For example ThetaEvolve\(Wang et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib41)\), TTT\-Discover\(Yuksekgonul et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib45)\), and FLEX\(Cai et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib5)\)use reinforcement learning to iteratively improve the LLM during the evolutionary loop at test\-time\.

Agentic Solutions for Open\-ended Discovery\.In addition to LLM\-guided evolutionary approaches, other work uses LLM agents to directly carry out open\-ended tasks without constraining the LLM to specific operations\. AI Scientist\(Lu et al\.,[2024](https://arxiv.org/html/2607.02807#bib.bib24)\)and AI Co\-Scientist\(Gottweis et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib11)\), developed by Sakana AI and Google respectively, are examples of agentic systems that assign LLM agents specialized roles, such as brainstormer and reviewer, to perform scientific discovery\. More recently, autoresearch\(Karpathy,[2026](https://arxiv.org/html/2607.02807#bib.bib16)\)has spurred growing interest in autonomous research agents\. It uses a general\-purpose coding agent, such as Codex or Claude Code, to iteratively modify research code, run experiments, evaluate results, and keep or discard changes\. Researchers and industry practitioners have developed more advanced solutions with better memory management\(Chen et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib7)\), shared skills\(rllm org,[2026](https://arxiv.org/html/2607.02807#bib.bib37)\), and multi\-agent collaboration\(Qu et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib33)\)\. Yet, these approaches still use one or more self\-organized agents, each in long\-running experimental loops with one editable program state\. Instead,SwarmResearch’s orchestrator\-subagent architecture avoids context accumulation in a few agents and idea convergence from shared memory \(see Section[2\.2](https://arxiv.org/html/2607.02807#S2.SS2)for details\), while its branch\-based versioning preserves competing approaches to avoid greedy local search \(see Section[2\.3](https://arxiv.org/html/2607.02807#S2.SS3)for details\)\. Additionally, unlike existing multi\-agent approaches, which involve complex orchestration such as heartbeat events and require manually configuring the number of agentsSwarmResearchis fully implemented as skills and requires no hyperparameters since the orchestrator adapts the number of parallel and serial agents autonomously\.

Test\-time Scaling\.Using more LLM inference\-time computation for better task performance is known as test\-time scaling\(Snell et al\.,[2024](https://arxiv.org/html/2607.02807#bib.bib40); Brown et al\.,[2024](https://arxiv.org/html/2607.02807#bib.bib4)\)\. A straightforward way to perform test\-time scaling is to dobest\-of\-Nsampling: sample multiple outputs in parallel from an LLM and select the one that obtains the highest score\(Cobbe et al\.,[2021](https://arxiv.org/html/2607.02807#bib.bib9); Lightman et al\.,[2023](https://arxiv.org/html/2607.02807#bib.bib22)\)\. This is also known as parallel test\-time scaling\. On the other hand, sequential test\-time scaling iteratively prompts the LLM to revise or update their previous output, sometimes based on evaluator feedback\(Qu et al\.,[2024](https://arxiv.org/html/2607.02807#bib.bib34)\)\. Test\-time scaling has been demonstrated to be very effective for challenging domains like software engineering\(Muennighoff et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib29); Ehrlich et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib10); Xia et al\.,[2025](https://arxiv.org/html/2607.02807#bib.bib42)\)\. A recent method, SimpleTES\(Ye et al\.,[2026](https://arxiv.org/html/2607.02807#bib.bib44)\), achieves impressive results on open\-ended discovery by scaling an evaluation\-driven workflow across a fixed number of parallel independent trajectories, sequential refinement steps, and best\-of\-KKcandidate selection between refinement steps\. While existing work uses heuristics or pre\-configures scaling,SwarmResearchdemonstrates how an LLM can autonomously and effectively adapt the degree of parallel and serial scaling on the fly\.

## 6Conclusion

SwarmResearchelicits high\-level exploration from coding agents through simple harness design choices: improved context management through an orchestrator\-subagent architecture and improved program versioning through git branching\. Our minimal implementation of these choices as skills makes the approach interoperable with popular coding agent tools\. We demonstrate these choices produce a harness that discovers stronger solutions than state\-of\-the\-art evolutionary and multi\-agent baselines through higher\-level experiments\. However, major challenges still remain\. Our harness does not enhance the basic creativity of LLMs, so approaches it explores still tend towards variations of known approaches\. Additionally, complex Shepherd Agent behavior is prompt\-reliant\. Models we test do not exhibit sophisticated Search Agent steering behavior, limiting their ability to meet the full potential of theSwarmResearcharchitecture\. Our intention is that our minimal harness is usable by practitioners, serves as a base for future harnesses and training on open\-ended discovery, and inspires future work on using orchestrators to steer Search Agents\.

## References

- Agrawal et al\. \(2025\)Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl\-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al\.Gepa: Reflective prompt evolution can outperform reinforcement learning\.*arXiv preprint arXiv:2507\.19457*, 2025\.
- Assumpção et al\. \(2025\)Henrique Assumpção, Diego Ferreira, Leandro Campos, and Fabricio Murai\.Codeevolve: An open source evolutionary coding agent for algorithm discovery and optimization\.2025\.URL[https://arxiv\.org/abs/2510\.14150](https://arxiv.org/abs/2510.14150)\.
- Balsamo \(2026\)Gabriele Balsamo\.Entropy\-guided dynamic expert selection in mixture\-of\-experts models, January 2026\.URL[https://adaptive\-k\.vercel\.app/paper\.html](https://adaptive-k.vercel.app/paper.html)\.Preprint, under review\.
- Brown et al\. \(2024\)Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini\.Large language monkeys: Scaling inference compute with repeated sampling\.*arXiv preprint arXiv:2407\.21787*, 2024\.
- Cai et al\. \(2025\)Zhicheng Cai, Xinyuan Guo, Yu Pei, Jiangtao Feng, Jinsong Su, Jiangjie Chen, Ya\-Qin Zhang, Wei\-Ying Ma, Mingxuan Wang, and Hao Zhou\.Flex: Continuous agent evolution via forward learning from experience\.*arXiv preprint arXiv:2511\.06449*, 2025\.
- Cemri et al\. \(2026\)Mert Cemri, Shubham Agrawal, Akshat Gupta, Shu Liu, Audrey Cheng, Qiuyang Mang, Ashwin Naren, Lutfi Eren Erdogan, Koushik Sen, Matei Zaharia, et al\.Adaevolve: Adaptive llm driven zeroth\-order optimization\.*arXiv preprint arXiv:2602\.20133*, 2026\.
- Chen et al\. \(2026\)Terry Chen, Zhifan Ye, Bing Xu, Zihao Ye, Timmy Liu, Ali Hassani, Tianqi Chen, Andrew Kerr, Haicheng Wu, Yang Xu, et al\.Avo: Agentic variation operators for autonomous evolutionary search\.*arXiv preprint arXiv:2603\.24517*, 2026\.
- Cheng et al\. \(2025\)Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Bowen Wang, Alex Krentsel, Tian Xia, Mert Cemri, Jongseok Park, Shuo Yang, Jeff Chen, Aditya Desai, Jiarong Xing, Koushik Sen, Matei Zaharia, and Ion Stoica\.Barbarians at the gate: How ai is upending systems research\.*arXiv preprint arXiv:2510\.06189*, 2025\.
- Cobbe et al\. \(2021\)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*, 2021\.
- Ehrlich et al\. \(2025\)Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Ré, and Azalia Mirhoseini\.Codemonkeys: Scaling test\-time compute for software engineering\.*arXiv preprint arXiv:2501\.14723*, 2025\.
- Gottweis et al\. \(2025\)Juraj Gottweis, Wei\-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al\.Towards an ai co\-scientist\.*arXiv preprint arXiv:2502\.18864*, 2025\.
- Hu et al\. \(2025\)Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, and Jing Zhang\.Sam decoding: Speculative decoding via suffix automaton\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics*, 2025\.
- Huang et al\. \(2025\)Kaixuan Huang, Xudong Guo, and Mengdi Wang\.SpecDec\+\+: Boosting speculative decoding via adaptive candidate lengths\.In*Proceedings of the Second Conference on Language Modeling*, 2025\.URL[https://openreview\.net/forum?id=NnExMNiTHw](https://openreview.net/forum?id=NnExMNiTHw)\.
- Imajuku et al\. \(2025\)Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba\.Ale\-bench: A benchmark for long\-horizon objective\-driven algorithm engineering\.In*Advances in Neural Information Processing Systems*, 2025\.
- Jain et al\. \(2024\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\.Livecodebench: Holistic and contamination free evaluation of large language models for code\.*arXiv preprint arXiv:2403\.07974*, 2024\.URL[https://arxiv\.org/abs/2403\.07974](https://arxiv.org/abs/2403.07974)\.
- Karpathy \(2026\)Andrej Karpathy\.autoresearch, 2026\.URL[https://github\.com/karpathy/autoresearch](https://github.com/karpathy/autoresearch)\.
- Kim et al\. \(2023\)Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W\. Mahoney, Amir Gholami, and Kurt Keutzer\.Speculative decoding with big little decoder\.In*Advances in Neural Information Processing Systems*, volume 36, pp\. 39236–39256\. Curran Associates, Inc\., 2023\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2023/hash/7b97adeafa1c51cf65263459ca9d0d7c\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2023/hash/7b97adeafa1c51cf65263459ca9d0d7c-Abstract-Conference.html)\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E\. Gonzalez, Hao Zhang, and Ion Stoica\.Efficient memory management for large language model serving with pagedattention\.In*Proceedings of the 29th Symposium on Operating Systems Principles*, 2023\.
- Lange et al\. \(2025\)Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin\.Shinkaevolve: Towards open\-ended and sample\-efficient program evolution\.*arXiv preprint arXiv:2509\.19349*, 2025\.
- Leviathan et al\. \(2023\)Yaniv Leviathan, Matan Kalman, and Yossi Matias\.Fast inference from transformers via speculative decoding\.In*Proceedings of the 40th International Conference on Machine Learning*, 2023\.
- Li et al\. \(2025\)Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang\.Eagle\-3: Scaling up inference acceleration of large language models via training\-time test\.*arXiv preprint arXiv:2503\.01840*, 2025\.URL[https://arxiv\.org/abs/2503\.01840](https://arxiv.org/abs/2503.01840)\.
- Lightman et al\. \(2023\)Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\.Let’s verify step by step\.In*The twelfth international conference on learning representations*, 2023\.
- Liu et al\. \(2026\)Shu Liu, Shubham Agarwal, Monishwaran Maheswaran, Mert Cemri, Zhifei Li, Qiuyang Mang, Ashwin Naren, Ethan Boneh, Audrey Cheng, Melissa Z Pan, et al\.Evox: Meta\-evolution for automated discovery\.*arXiv preprint arXiv:2602\.23413*, 2026\.
- Lu et al\. \(2024\)Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha\.The ai scientist: Towards fully automated open\-ended scientific discovery\.*arXiv preprint arXiv:2408\.06292*, 2024\.
- Lu et al\. \(2025\)Kuan\-Wei Lu, Ding\-Yong Hong, Pangfeng Liu, and Jan\-Jan Wu\.AdaSD: Adaptive speculative decoding for efficient language model inference, 2025\.URL[https://arxiv\.org/abs/2512\.11280](https://arxiv.org/abs/2512.11280)\.
- Lütke \(2026\)Tobias Lütke\.Performance: 53% faster parse\+render, 61% fewer allocations\.[https://github\.com/Shopify/liquid/pull/2056](https://github.com/Shopify/liquid/pull/2056), mar 2026\.Pull request \#2056 to Shopify Liquid\.
- MathArena \(2026\)MathArena\.AIME 2026\.Hugging Face dataset, 2026\.URL[https://huggingface\.co/datasets/MathArena/aime\_2026](https://huggingface.co/datasets/MathArena/aime_2026)\.Dataset of problems from the 2026 American Invitational Mathematics Examination\.
- Mouret & Clune \(2015\)Jean\-Baptiste Mouret and Jeff Clune\.Illuminating search spaces by mapping elites\.*arXiv preprint arXiv:1504\.04909*, 2015\.
- Muennighoff et al\. \(2025\)Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei\-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori B Hashimoto\.s1: Simple test\-time scaling\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pp\. 20286–20332, 2025\.
- Novikov et al\. \(2025\)Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po\-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al\.Alphaevolve: A coding agent for scientific and algorithmic discovery\.*arXiv preprint arXiv:2506\.13131*, 2025\.
- Panfilov et al\. \(2026\)Alexander Panfilov, Peter Romov, Igor Shilov, Yves\-Alexandre de Montjoye, Jonas Geiping, and Maksym Andriushchenko\.Claudini: Autoresearch discovers state\-of\-the\-art adversarial attack algorithms for llms\.*arXiv preprint arXiv:2603\.24511*, 2026\.
- Phan et al\. \(2025\)Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sarah Zhang, Michael Chen, Michael Ong, Aarohi Srivastava, et al\.Humanity’s last exam\.*arXiv preprint arXiv:2501\.14249*, 2025\.URL[https://arxiv\.org/abs/2501\.14249](https://arxiv.org/abs/2501.14249)\.
- Qu et al\. \(2026\)Ao Qu, Han Zheng, Zijian Zhou, Yihao Yan, Yihong Tang, Shao Yong Ong, Fenglu Hong, Kaichen Zhou, Chonghe Jiang, Minwei Kong, Jiacheng Zhu, Xuan Jiang, Sirui Li, Cathy Wu, Bryan Kian Hsiang Low, Jinhua Zhao, and Paul Pu Liang\.Coral: Towards autonomous multi\-agent evolution for open\-ended discovery\.*arXiv preprint arXiv:2604\.01658*, 2026\.URL[https://arxiv\.org/pdf/2604\.01658](https://arxiv.org/pdf/2604.01658)\.
- Qu et al\. \(2024\)Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar\.Recursive introspection: Teaching language model agents how to self\-improve\.*Advances in Neural Information Processing Systems*, 37:55249–55285, 2024\.
- Rein et al\. \(2023\)David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\.Gpqa: A graduate\-level google\-proof q&a benchmark\.*arXiv preprint arXiv:2311\.12022*, 2023\.URL[https://arxiv\.org/abs/2311\.12022](https://arxiv.org/abs/2311.12022)\.
- RightNow\-AI \(2026\)RightNow\-AI\.Autokernel\.[https://github\.com/RightNow\-AI/autokernel](https://github.com/RightNow-AI/autokernel), 2026\.Open\-source autoresearch system for GPU kernel optimization\.
- rllm org \(2026\)rllm org\.Hive, 2026\.URL[https://github\.com/rllm\-org/hive](https://github.com/rllm-org/hive)\.
- Romera\-Paredes et al\. \(2024\)Bernardino Romera\-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al\.Mathematical discoveries from program search with large language models\.*Nature*, 625\(7995\):468–475, 2024\.
- Sharma \(2025\)Asankhaya Sharma\.Openevolve: an open\-source evolutionary coding agent, 2025\.URL[https://github\.com/algorithmicsuperintelligence/openevolve](https://github.com/algorithmicsuperintelligence/openevolve)\.
- Snell et al\. \(2024\)Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar\.Scaling llm test\-time compute optimally can be more effective than scaling model parameters\.*arXiv preprint arXiv:2408\.03314*, 2024\.
- Wang et al\. \(2025\)Yiping Wang, Shao\-Rong Su, Zhiyuan Zeng, Eva Xu, Liliang Ren, Xinyu Yang, Zeyi Huang, Xuehai He, Luyao Ma, Baolin Peng, et al\.Thetaevolve: Test\-time learning on open problems\.*arXiv preprint arXiv:2511\.23473*, 2025\.
- Xia et al\. \(2025\)Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang\.Agentless: Demystifying llm\-based software engineering agents\.*Proceedings of the ACM on Software Engineering*, 2\(FSE\):801–824, 2025\.doi:10\.1145/3715754\.
- Yan et al\. \(2026\)Minghao Yan, Bo Peng, Benjamin Coleman, Ziqi Chen, Zhouhang Xie, Shuo Chen, Zhankui He, Noveen Sachdeva, Isabella Ye, Weili Wang, et al\.Pacevolve: Enabling long\-horizon progress\-aware consistent evolution\.*arXiv preprint arXiv:2601\.10657*, 2026\.
- Ye et al\. \(2026\)Haotian Ye, Haowei Lin, Jingyi Tang, Yizhen Luo, Caiyin Yang, Chang Su, Rahul Thapa, Rui Yang, Ruihua Liu, Zeyu Li, Chong Gao, Dachao Ding, Guangrong He, Miaolei Zhang, Lina Sun, Wenyang Wang, Yuchen Zhong, Zhuohao Shen, Di He, Jianzhu Ma, Stefano Ermon, Tongyang Li, Xiaowen Chu, James Zou, and Yuzhi Xu\.Evaluation\-driven scaling for scientific discovery\.*arXiv preprint arXiv:2604\.19341*, 2026\.
- Yuksekgonul et al\. \(2026\)Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, Jed McCaleb, Xiaolong Wang, Jan Kautz, Yejin Choi, James Zou, Carlos Guestrin, et al\.Learning to discover at test time\.*arXiv preprint arXiv:2601\.16175*, 2026\.
- Zechner \(2025\)Mario Zechner\.Pi coding agent\.[https://github\.com/earendil\-works/pi](https://github.com/earendil-works/pi), 2025\.Open\-source coding agent CLI\.
- Zheng et al\. \(2024\)Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E\. Gonzalez, Clark Barrett, and Ying Sheng\.SGLang: Efficient execution of structured language model programs\.In*Advances in Neural Information Processing Systems*, volume 37, pp\. 62557–62583\. Curran Associates, Inc\., 2024\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2024/hash/724be4472168f31ba1c9ac630f15dec8\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2024/hash/724be4472168f31ba1c9ac630f15dec8-Abstract-Conference.html)\.

Appendix

## Appendix ASpeculative Decoding Details

Evaluation Details\.The target model is gemma\-4\-26B\-A4B\-it, and the draft model is its corresponding gemma\-assistant MTP head\. All methods have access to and are evaluated on 8xA6000s\. Temperature 1\.0 is used\.

Analysis Phase\.We do 2 sequentialSwarmResearchruns: an∼\\sim11\-hour analysis phase and an∼\\sim11\-hour solution generation phase\. The goal of analysis it to collect data on the baseline from different angles without generating solutions\. For example, Explorer Search Agents profile the baseline and measure target vs\. draft confidences\. The Shepherd Agent’s role is to maintain a diverse population of analysis while going deep on promising directions\. The analysis phase’s findings are organized into a directory accessible to the solution generation phase’s Search Agents\. This helps structure Search Agent idea generation, so its ideas are more novel but motivated by specific observations\. As an example, analysis finds that most of the target model’s generation consists of low\-entropy tokens that the draft predicts well, but there are few high\-entropy tokens that the draft model does not predict well and throws of the reasoning trace if incorrect\.

## Appendix BBetween\-run Variance

Between\-run variance is a special challenge for discovery technique reproducibility\. Discovery techniques rely on high\-temperature sampling over uncertain LLM token distributions to open\-ended queries, which drives the apparent creativity of LLMs\. Given that LLM variance forms a basis of many discovery techniques, some variance between independent runs can be hard to avoid\. To achieve a stable estimate of some discovery techniques e\.g\. vanilla autoresearch\(Karpathy,[2026](https://arxiv.org/html/2607.02807#bib.bib16)\), a researcher might have to run it over 5 times for some tasks\. Even more so, we have have observed that the number of independent runs needed to achieve a stable estimate can be task\-specific\. However, running discovery techniques multiple times is infeasibly expensive when using frontier models, which are important to use since they discover better solutions than cheaper models and they exhibit unique behaviors a technique might rely on\. For instance, just one run of all three approaches on our 15 task benchmark cost∼\\sim$1700 worth of Opus 4\.6 credits\.

In light of the costliness of these techniques, the number of independent runs should be part of the recommended use of a discovery technique\. This recommendation is crucial for both users and researchers to know how to allocate their budget to independent runs when using and reproducing a technique\. For example, given a $50 budget of frontier model credits per task, a technique should inform users whether splitting it into 3 runs of $17 or doing 1 long $50 run will achieve the most representative results\. Exact reproducibility can be challenging for some discovery techniques, but these guidelines inform users and researchers on how to best achieve arepresentativeresult for both real\-world use and benchmarking\.

For our approach, we recommend one run since independent search agents are a form of parallel scaling that improve between\-run stability\. We also run the baseline approaches once since independent runs are not in the baselines’ recommendation for users\. This way, under realistic budget constraints, we can reproduce the most representative result for the baselines\. Although this means that our reported baseline results might not reflect a stable estimate across many independent runs, we view this as the best option given their stochastic and costly nature\.

## Appendix CSkills

### C\.1Shepherd

``

`C\.2 Explorer C\.2\.1 Optimizer`