Compute Allocation in Evolutionary Search: From Depth-Breadth to Multi-Armed Bandits

# Compute Allocation in Evolutionary Search: From Depth–Breadth to Multi-Armed Bandits
Source: [https://arxiv.org/html/2605.29268](https://arxiv.org/html/2605.29268)
Sixue Xing1Haoyu He††footnotemark:2Kerui Wu††footnotemark:3 Zhuo Yang4Haozheng Luo5Tianfan Fu6, 7Aarthy Nagarajan1

1University of Notre Dame2Northeastern University3University of Massachusetts Amherst 4Southeast University5Northwestern University6Nanjing University 7Shanghai Artificial Intelligence Laboratory

###### Abstract

LLM\-guided evolutionary search \(Evolve systems\) has reached state\-of\-the\-art results on mathematical and combinatorial tasks, yet most existing systems report only the best of many runs and leave the run\-to\-run distribution undocumented\. We ask how a fixed budget of LLM calls should be allocated, and how reliably a single run reaches the reported numbers\. Sweeping the depth–breadth grid over five models and three tasks, we identify two empirical regularities: a fitness–compute envelope along which capability ordering largely collapses on effective FLOPs, and a bilinear depth–breadth fit with task\-specific interaction; both are gated by model–task capability\. Motivated by these regularities, we propose BaSE \(Bandit\-based Self\-Evolving\), a multi\-armed bandit that allocates LLM calls across parallel trajectories\. Without changing the model, prompt, or evaluator, BaSE improves mean fitness by 12\.3% over the strongest island\-protocol baseline across 8 \(model, task\) cells, with the largest gains on high\-variance settings: a reliability gain from allocation alone\. The code is available at:[https://github\.com/keruiwu/self\-evolving\-allocation](https://github.com/keruiwu/self-evolving-allocation)\.

Compute Allocation in Evolutionary Search: From Depth–Breadth to Multi\-Armed Bandits

## 1Introduction

Large language models are increasingly used as mutation engines for evolutionary search: given a candidate program, a frozen LLM proposes variants; a deterministic evaluator scores them; the best variant seeds the next round\(Romera\-Paredeset al\.,[2024](https://arxiv.org/html/2605.29268#bib.bib3); Novikovet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib4)\)\. The resulting*Evolve*systems\(Langeet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib5); Wanget al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib6); Assumpçãoet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib47); Cemriet al\.,[2026](https://arxiv.org/html/2605.29268#bib.bib11)\)have produced state\-of\-the\-art results across mathematical discovery, combinatorial optimization, and algorithm design\. However, the headline numbers*\*Evolve*systems report are systematically incomparable: FunSearch reports a 4\-of\-140 hit rate\(Romera\-Paredeset al\.,[2024](https://arxiv.org/html/2605.29268#bib.bib3)\); CodeEvolve displays “only the best”\(Assumpçãoet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib47)\); AlphaEvolve reports a single number on then=26n\{=\}26Circle Packing benchmark\(Novikovet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib4)\)\. Reported per\-run cost spans more than two orders of magnitude, from∼\\sim150 LLM calls in ShinkaEvolve\(Langeet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib5)\)to the 204,800 candidates ThetaEvolve processes in a single run\(Wanget al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib6)\)\.[Figure˜1](https://arxiv.org/html/2605.29268#S1.F1)plots these reports on a common axis: each point is a single hand\-picked configuration, and the dominant convention is to report only the best of an unspecified number of runs\. Few systems report the run\-to\-run distribution behind these headline numbers\. Combined with the order\-of\-magnitude variation in reported computational cost, this makes the numbers an unreliable guide to single\-run performance under realistic deployment settings: existing reports characterize what is achievable on a favorable run, not what a practitioner should expect at a finite computational cost\.

![Refer to caption](https://arxiv.org/html/2605.29268v1/x1.png)Figure 1:Cost–performance frontier onn=26n\{=\}26Circle Packing \(CP\)\.The expected performance is shaped by multiple design choices\. A stronger base model can generate better mutations; a more informative prompt can guide the search toward more useful edits; and, importantly, allocation determines how the evolutionary process balances exploration and exploitation\. This allocation effect is orthogonal to model and prompt quality, and remains meaningful once the model, prompt, and evaluator are fixed\. Better allocation can improve expected outcomes by avoiding both premature commitment to weak trajectories and excessive breadth without refinement\.

While*\*Evolve*systems are often described as iterative loops that continue until progress saturates, practical deployments must operate under explicit resource constraints, such as a fixed number of model calls, a limited compute allocation, or a bounded experimental campaign\. This motivates a controlled study of expected performance and reliability in LLM\-guided evolutionary search\. We adopt the fixed\-budget perspective\(Jansen and Zarges,[2012](https://arxiv.org/html/2605.29268#bib.bib2)\)from classical evolutionary computation and bring it to LLM\-guided evolution\. To this end, we conduct the first empirical study of cost allocation in LLM\-guided evolutionary search, characterizing how a fixed number of LLM call should be spent, and use the resulting picture to ask whether allocation can be precomputed offline or must be discovered online\. In summary, our contributions are listed below:

1. 1\.Proposed the first systematic empirical measurement of fixed\-budget allocation between exploration and exploitation in LLM\-guided evolutionary search\.
2. 2\.Identified two regularities:*\(i\)*a performance–compute envelope of attainable fitness against effective FLOPs, along which capability ordering largely collapses among capable models; and*\(ii\)*a parametric depth–breadth regularity characterizing each model\-task cell\.
3. 3\.Designed an adaptive bandit allocator \(BaSE\) over parallel evolutionary trajectories, improving best mean fitness by12\.3%on average over the strongest island\-protocol baseline across 8 \(model, task\) cells — this is not a model improvement, nor a prompt improvement, but an allocation improvement\.

## 2Related Work

LLM\-Guided Evolutionary Search\.FunSearch\(Romera\-Paredeset al\.,[2024](https://arxiv.org/html/2605.29268#bib.bib3)\)introduced LLMs as mutation operators in evolutionary program synthesis, and AlphaEvolve\(Novikovet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib4)\)scaled the paradigm to state\-of\-the\-art results\.*\*Evolve*variants vary individual axes — sample efficiency\(Langeet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib5)\), test\-time RL\(Wanget al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib6)\), open\-source islands — with adjacent work on code, architectures, and prompts\(Lehmanet al\.,[2022](https://arxiv.org/html/2605.29268#bib.bib7); Chenet al\.,[2023](https://arxiv.org/html/2605.29268#bib.bib8); Yanget al\.,[2023](https://arxiv.org/html/2605.29268#bib.bib9)\); all leave population size and generation count hand\-set\. ThetaEvolve only notes qualitatively that small databases progress faster early but large ones win at scale\(Wanget al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib6)\), and Population\-Evolve sweeps population size without holding total budget fixed\(Zhanget al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib10)\)\. Adjacent allocation work routes compute between islands\(Cemriet al\.,[2026](https://arxiv.org/html/2605.29268#bib.bib11)\), compares evolution to best\-of\-N and sequential revision\(Leeet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib53)\), or focuses on parent sampling\(Novikovet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib4); Langeet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib5)\)and combination operators\(Langeet al\.,[2023](https://arxiv.org/html/2605.29268#bib.bib32); Meyersonet al\.,[2024](https://arxiv.org/html/2605.29268#bib.bib33)\)\. The closest precedent, ShinkaEvolve\(Langeet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib5)\), uses a bandit to ensemble*models*at the final phase; we instead target the within\-run depth–breadth split\.

Compute Allocation\.Inference\-scaling work allocates test\-time compute\(Snellet al\.,[2024](https://arxiv.org/html/2605.29268#bib.bib12); Brownet al\.,[2024](https://arxiv.org/html/2605.29268#bib.bib13); Wuet al\.,[2024](https://arxiv.org/html/2605.29268#bib.bib14); Chenet al\.,[2024](https://arxiv.org/html/2605.29268#bib.bib15)\), with breadth\-vs\-depth studies on single\-query reasoning and tree search\(Sharma and Chopra,[2025](https://arxiv.org/html/2605.29268#bib.bib16); Wenet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib17); Inoueet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib45); Miyamotoet al\.,[2026](https://arxiv.org/html/2605.29268#bib.bib19)\)reporting empirical, regime\-dependent curves; none targets population\-based evolution with parent\-conditioned mutation\. Our online allocator draws on multi\-armed bandits\(Aueret al\.,[2002a](https://arxiv.org/html/2605.29268#bib.bib37); Lattimore and Szepesvári,[2020](https://arxiv.org/html/2605.29268#bib.bib38)\)and fixed\-budget best\-arm identification\(Audibert and Bubeck,[2010](https://arxiv.org/html/2605.29268#bib.bib40); Karninet al\.,[2013](https://arxiv.org/html/2605.29268#bib.bib41)\), with precedents in hyperparameter search\(Liet al\.,[2018](https://arxiv.org/html/2605.29268#bib.bib42)\), prompt selection\(Shiet al\.,[2024](https://arxiv.org/html/2605.29268#bib.bib43)\), MCTS expansion\(Inoueet al\.,[2025](https://arxiv.org/html/2605.29268#bib.bib45)\), island routing\(Cemriet al\.,[2026](https://arxiv.org/html/2605.29268#bib.bib11)\), and code repair\(Tanget al\.,[2024](https://arxiv.org/html/2605.29268#bib.bib20)\), all of which*assume*adaptive allocation helps\. Classical EA contributes optimal\-population\-under\-fixed\-compute analyses\(Nakanoet al\.,[1994](https://arxiv.org/html/2605.29268#bib.bib21); Brieschet al\.,[2023](https://arxiv.org/html/2605.29268#bib.bib22)\), offspring\-size theory\(Jansenet al\.,[2005](https://arxiv.org/html/2605.29268#bib.bib23); Doerr and Künnemann,[2015](https://arxiv.org/html/2605.29268#bib.bib24); Gießen and Witt,[2017](https://arxiv.org/html/2605.29268#bib.bib25); Badkobehet al\.,[2014](https://arxiv.org/html/2605.29268#bib.bib26)\), and the fixed\-budget framing\(Jansen and Zarges,[2012](https://arxiv.org/html/2605.29268#bib.bib2),[2014](https://arxiv.org/html/2605.29268#bib.bib27)\); all assume random mutation, whereas LLM mutation carries code priors and discrete attractors that break i\.i\.d\. mean\-field assumptions, so we characterize the surface empirically\.

## 3Problem Formulation

Self\-Evolving\.In high level, we formulate self\-evolution as an iterative optimization process under a fixed inference budget ofCCLLM calls\. Specifically, as described in[Algorithm˜1](https://arxiv.org/html/2605.29268#alg1), a Self\-Evolve system repeatedly generates a candidate response, evaluates its quality, and refines the proposed solution based on historical response\-evaluation pairs\.

Algorithm 1Self\-Evolving Process1:Task

qq, base LLM

ff, prompt generator

hh, budget \(number of LLM calls\)

CC
2:for

c=1,…,Cc=1,\\dots,Cdo

3:

prompt←h​\(q,\(response\(j\),score\(j\)\)j∈\[c\]\)\\mathrm\{prompt\}\\leftarrow h\\left\(q,\\left\(\\mathrm\{response\}^\{\(j\)\},\\mathrm\{score\}^\{\(j\)\}\\right\)\_\{j\\in\[c\]\}\\right\)
4:

response\(c\+1\)←f​\(prompt\)\\mathrm\{response\}^\{\(c\+1\)\}\\leftarrow f\(\\mathrm\{prompt\}\)
5:

score\(c\+1\)←Eval​\(q,response\(c\+1\)\)\\mathrm\{score\}^\{\(c\+1\)\}\\leftarrow\\mathrm\{Eval\}\\left\(q,\\mathrm\{response\}^\{\(c\+1\)\}\\right\)
6:endfor

7:

c∗←arg⁡maxc∈\[C\]⁡score\(c\)c^\{\*\}\\leftarrow\\arg\\max\_\{c\\in\[C\]\}\\mathrm\{score\}^\{\(c\)\}
8:return

response\(c∗\)\\mathrm\{response\}^\{\(c^\{\*\}\)\}

The key design question in this process is the*parent sampling protocol*described in[˜3](https://arxiv.org/html/2605.29268#alg1.l3)of[Algorithm˜1](https://arxiv.org/html/2605.29268#alg1): how should the system select and reuse historical responses when constructing the next prompt? This choice determines how the evolutionary process balances exploitation of high\-performing solutions with exploration of diverse alternatives\. In this work, we explored two representative protocols:*greedy*and*island*:

Greedy Protocol\.At each generation, greedy protocol selects the current best\-scoring response as the parent and generatesNNchildren from it in parallel\. Equivalently, it spendsNNLLM calls per generation refining the best solution found so far\.

Island Protocol\.The island protocol originates in classical evolutionary computationTanese \([1989](https://arxiv.org/html/2605.29268#bib.bib48)\)and was brought into LLM\-guided evolutionary search by FunSearchRomera\-Paredeset al\.\([2024](https://arxiv.org/html/2605.29268#bib.bib3)\), later adopted by many Evolve\-style systems; it maintains a population database partitioned into multiple islands\. At each generation, it first selects an island according to a MAP\-Elites\-style coverage ruleMouret and Clune \([2015](https://arxiv.org/html/2605.29268#bib.bib46)\), and then samples a parent uniformly from that island\.

The Depth–Breadth Allocation Problem\.Under the greedy protocol, a run of budgetCCis fully specified by how that budget is split:TTgenerations ofNNchildren each, so that:

The two ends of this split are familiar limits:T=1T\{=\}1spends the whole budget on a single generation of parallel samples \(best\-of\-NN\), whereasT=CT\{=\}Crefines one trajectory forCCsequential steps\. LetV​\(C,T\)V\(C,T\)be the expected best fitness of a run at budgetCCand depthTT\. Holding the evaluator, prompt template, base model, and initial program fixed, the split is the only remaining degree of freedom, and we seek the*compute\-optimal depth*

T∗​\(C\)=arg⁡maxT⁡V​\(C,T\),Vmax​\(C\)=maxT⁡V​\(C,T\)\.\\begin\{gathered\}T^\{\*\}\(C\)=\\arg\\max\_\{T\}V\(C,T\),\\\\ V\_\{\\max\}\(C\)=\\max\_\{T\}V\(C,T\)\.\\end\{gathered\}\(1\)
the question we take up in[Section˜5](https://arxiv.org/html/2605.29268#S5)is how V\(C,T\) varies with allocation\.

## 4Experiment Setup

Tasks\.We evaluate three geometric optimization tasks drawn from AlphaEvolve and shipped with the OpenEvolve example suite:Circle Packing\(CP,n=26n\{=\}26\),MinMaxDist\(MMD,n=16n\{=\}16\), andHeilbronn Triangle\(HT,n=11n\{=\}11\)\. For each task we use the OpenEvolve evaluator and initial\-program files verbatim, and normalize raw objectives by the best published construction so thatfitness=1\.0\\textsc\{fitness\}\{=\}1\.0matches the state\-of\-the\-art\. Full problem statements and normalizers are in[Appendix˜B](https://arxiv.org/html/2605.29268#A2)\.

Models\.We sweep the open\-weight Qwen3 family at four sizes: 1\.7B, 4B, 8B, and 14B, all with thinking mode enabled and Llama\-3\.1\-8B \(Llama\), temperature0\.60\.6, top\-pp0\.950\.95\. Inference is served by vLLM v0\.18 with\-\-quantization fp8,\-\-max\-model\-len=40960\\texttt\{\-\-max\-model\-len\}=40960,\-\-max\-num\-seqs=16\\texttt\{\-\-max\-num\-seqs\}=16\. The vllm server runs with up to 16 in\-flight parallel LLM calls on H100 GPU\.

Bandits Algorithms\.In[Section˜6](https://arxiv.org/html/2605.29268#S6), we explore the impact of different runs against fitness score performance and employ Multi\-Armed Bandits \(MAB\) for adaptive trajectory allocation\. We implement three classic MAB algorithms, namely, Upper Confidence Bound \(UCB\)Aueret al\.\([2002b](https://arxiv.org/html/2605.29268#bib.bib34)\), Exponential\-weight algorithm for Exploration and Exploitation with high Probability \(EXP3\.P\)Aueret al\.\([2002b](https://arxiv.org/html/2605.29268#bib.bib34)\), and Thompson Sampling \(Thompson\)Thompson \([1933](https://arxiv.org/html/2605.29268#bib.bib35)\); Agrawal and Goyal \([2012](https://arxiv.org/html/2605.29268#bib.bib36)\), as well as one naive random baseline111We emphasize that random is a MAB algorithm baseline that chooses which evolving trajectory to pull instead of choosing parent inside the evolving process\., sampling parent from evolve trajectories\.

Parameters\.We sweep greedy on Qwen3 8B/14B and Llama atC∈\{8,…,512\}C\\in\\\{8,\\dots,512\\\}withT∈\{1,2,4,…,C\}T\\in\\\{1,2,4,\\dots,C\\\}\(smaller Qwen3 1\.7B/4B capped atC=128C\{=\}128\), 10 seeds per cell;T=1T\{=\}1recovers best\-of\-NN,T=CT\{=\}Cis pure sequential\. Island protocols \(OpenEvolve, CodeEvolve, ShinkaEvolve\) run atC=512C\{=\}512on the same three models\. For each \(model, task\) cell, Bandit and all protocols run end\-to\-end 10 times with independent LLM seeds\. We report stratified bootstrap standard errors and 95% CIs over the 1000 resamples, following therliableevaluation protocol ofAgarwalet al\.\([2022](https://arxiv.org/html/2605.29268#bib.bib55)\)\.

Post\-hoc FLOPs accounting\.Counting LLM calls \(CC\) is not a fair cost axis across models or protocols: different model sizes and different prefix\-cache hit rates \(greedy reuses one parent prefix acrossNNsiblings; island rewrites the prompt per call\) consume substantially different FLOPs at the sameCC\. FollowingHoffmannet al\.\([2022](https://arxiv.org/html/2605.29268#bib.bib1)\), we charge each LLM call:

FLOPscall=2​Pactive​\(pprompt−pcached\+pout\),\\textsc\{FLOPs\}\_\{\\text\{call\}\}=2\\,P\_\{\\text\{active\}\}\\,\(p\_\{\\text\{prompt\}\}\-p\_\{\\text\{cached\}\}\+p\_\{\\text\{out\}\}\),wherePactiveP\_\{\\text\{active\}\}is the active parameter count andpprompt,pcached,poutp\_\{\\text\{prompt\}\},p\_\{\\text\{cached\}\},p\_\{\\text\{out\}\}are the prompt, prefix\-cached, and completion token counts\. Per\-run FLOPs sum over all calls\. See[Appendix˜A](https://arxiv.org/html/2605.29268#A1)for full definitions\. FLOPs/CCvaries by under 15% acrossTTat fixedCCwithin each model \(R2≥0\.94R^\{2\}\\geq 0\.94, pooled linear fit;[Figure˜3](https://arxiv.org/html/2605.29268#S5.F3), top row\), so we useCCwithin models and effective FLOPs across models\.

## 5Depth–Breadth Allocation

We do greedy runs, whose budget decomposes cleanly asC=T⋅NC\{=\}T\\cdot N, making depth–breadth allocation the only remaining degree of freedom\. We analyze the full\(C,T\)\(C,T\)sweep in two steps:[Section˜5\.1](https://arxiv.org/html/2605.29268#S5.SS1)asks*how much*fitness a budget buys, on a cost axis comparable across model sizes\.[Section˜5\.2](https://arxiv.org/html/2605.29268#S5.SS2)asks*how*the budget should be split, and fits a parametric regularity to the depth–breadth surface\.

### 5\.1The Fitness\-Compute Envelope

![Refer to caption](https://arxiv.org/html/2605.29268v1/x2.png)Figure 2:Fitness versus effective FLOPs\. Solid lines reportVmaxV\_\{\\max\}over sweeped depth\-breadth allocations; dashed lines report the pure best\-of\-NNbaseline \(T=1T=1\)\. Vertical gap between solid and dashed curves is the gain of ’evolving’ from using evaluator feedback across generations rather than spending the full budget on one generation of parallel samples\.At each budget we summarize the sweep by the best fitnessVmax​\(C\)V\_\{\\max\}\(C\), the highest fitness achieved across all tested depth–breadth allocations at that compute level, following the compute\-performance envelope convention ofHoffmannet al\.\([2022](https://arxiv.org/html/2605.29268#bib.bib1)\), and study how it scales with effective FLOPs\. We further measure compute in effective FLOPs for comparing this envelope across models \([Section˜4](https://arxiv.org/html/2605.29268#S4)\) and studyVmaxV\_\{\\max\}\(best in\-sweep allocation\) on that axis\. Figure[2](https://arxiv.org/html/2605.29268#S5.F2)reportsVmaxV\_\{\\max\}and the best\-of\-NNbaseline against this axis\. For every modelVmaxV\_\{\\max\}rises smoothly and monotonically with compute compute envelope\. This is a clean envelope with no phase transitions or reversals\.

#### Capability ordering collapsing\.

At equal LLM call count, larger models lead, but the lead is not free: a larger model spends proportionally more FLOPs per call\. Re\-priced in effective FLOPs \([Figure˜2](https://arxiv.org/html/2605.29268#S5.F2)\), the ordering largely dissolves: on MMD the 4B, 8B and 14B envelopes nearly coincide \(R2=0\.94R^\{2\}=0\.94\); on CP: 8B and 14B coincide over the sub\-ceiling range \(R2=0\.93R^\{2\}=0\.93\)\. This collapse itself is gated by model\-task capability \([Appendix˜C](https://arxiv.org/html/2605.29268#A3)\)\.

#### Allocation improves fitness\.

The vertical gap between the solid envelope and the dashed best\-of\-NNbaseline in[Figure˜2](https://arxiv.org/html/2605.29268#S5.F2)is the gain from multi\-generation refinement\. AtC=128C\{=\}128it reaches up to a tenth of the normalized fitness range \(\+0\.119\+0\.119on CP 8B,\+0\.102\+0\.102on MMD 8B\); it persists on unsaturated tasks \(\+0\.115\+0\.115on MMD 14B atC=512C\{=\}512\) but diminishes where the task saturates \(\+0\.016\+0\.016on MMD 8B\)\. A selection\-free test confirms a real onset on MMD but no sharp threshold on CP\. Allocation thus yields real but task\-dependent gains, whose structure we characterize in[Section˜5\.2](https://arxiv.org/html/2605.29268#S5.SS2)\.

### 5\.2The Depth–Breadth Regularity

Table 1:Posthoc depth–breadth gap model on sub\-ceiling cells of the C=512 sweeps\. Coefficients are for[Equation˜2](https://arxiv.org/html/2605.29268#S5.E2)with natural logs\. MMD has a clearly negative interaction \(interior ridge\) on both capable models; CP and HT sit near the corner limit \(\|c\|<0\.03\|c\|<0\.03\)\.![Refer to caption](https://arxiv.org/html/2605.29268v1/x3.png)Figure 3:Full\(C,T\)\(C,T\)surfaces for the C=512 sweeps\. Top: effective FLOPs; bottom row: mean fitness\. The FLOPs surfaces are nearly flat inTTat fixedCC, so changes along the depth axis mostly reflect allocation effects rather than hidden cost differences\. The score surfaces reveal different allocation geometry: Circle Packing is depth\-favored with broad plateaus, whereas MinMaxDist has an interior ridge\.The depth–breadth split determines where on the fitness landscape a run lands\. Here we characterize the landscape itself\.

#### The fitness landscape\.

For a practitioner with fixed budgetCC, choosingTTis equivalent to selecting a point on the fitness surface along the budget slice ofCC\. As[Figure˜3](https://arxiv.org/html/2605.29268#S5.F3)shows, the effective\-FLOPs surface \(top\) is approximately linear inTTand remains weakly dependent onTTat fixedCC, so fitness differences along the depth axis reflect allocation choices rather than hidden cost variation\. The fitness surface \(bottom\) reveals different geometries across tasks: CP shows a broad plateau where many depth allocations achieve near\-equivalent fitness, whereas MMD shows an interior ridge where only a balanced depth allocation achieves the highest in\-sweep fitness\. The same protocol and budget range thus produce qualitatively different optimization landscapes, raising the question of what underlying task properties drive the difference\.

An empirical bilinear form\.We fit a parametric model to the sub\-ceiling cells \(V<0\.97V<0\.97\), with log fitness gap as the response and depth and breadth as predictors:

log⁡\(1−V\)=β0\+a​log⁡T\+b​log⁡N\+c​log⁡T​log⁡N\.\\log\(1\-V\)=\\beta\_\{0\}\+a\\log T\+b\\log N\+c\\log T\\log N\.\(2\)A budget\-only model \(c=0c\{=\}0,a=ba\{=\}b\) reachesR2≈0\.74R^\{2\}\\\!\\approx\\\!0\.74–0\.780\.78; the bilinear form reachesR2∈\[0\.75,0\.92\]R^\{2\}\\in\[0\.75,0\.92\]across all three tasks \([Table˜1](https://arxiv.org/html/2605.29268#S5.T1)\)\. Allocation, with budget, carries the signal\. The coefficientcccharacterizes the regime \(interior at large negative\|c\|\|c\|, corner near\|c\|≈0\|c\|\\approx 0; full boundary analysis in[Section˜F\.3](https://arxiv.org/html/2605.29268#A6.SS3)\) in a way prior inference\-time scaling work reports descriptively but does not pin downSnellet al\.\([2024](https://arxiv.org/html/2605.29268#bib.bib12)\); Inoueet al\.\([2025](https://arxiv.org/html/2605.29268#bib.bib45)\); higher\-order terms \(log2⁡T\\log^\{2\}T,log2⁡N\\log^\{2\}N\) do not materially improve the fit \([Appendix˜F](https://arxiv.org/html/2605.29268#A6)\)\.

Among the four coefficients,ccalone controls geometry: small\|c\|\|c\|leaves the budget slice near\-separable with the optimum at the all\-depth corner, while large negative\|c\|\|c\|bends it inward to a balanced interior optimum, with plateau half\-width∝1/\|c\|\\propto 1/\\sqrt\{\|c\|\}\([Section˜F\.2](https://arxiv.org/html/2605.29268#A6.SS2)\)\. MMD sits in the latter limit on both capable models; CP and HT sit in the former \([Table˜1](https://arxiv.org/html/2605.29268#S5.T1), with the near\-zero rows discussed in[Section˜F\.3](https://arxiv.org/html/2605.29268#A6.SS3)\)\. The case studies \([Appendix˜D](https://arxiv.org/html/2605.29268#A4)\) are consistent with a mechanism we call*asymmetric proposal mass*: each task admits one high\-fitness algorithmic family, but the LLM’s base rate on it appears differ across tasks, so where the good family is rare, breadth raises the probability that at least one parallel trajectory anchors on it before depth can refine within it\.

## 6Modeling Evolving Trajectory through Multi\-Armed Bandits

![Refer to caption](https://arxiv.org/html/2605.29268v1/x4.png)Figure 4:Per\-run greedy fitness trajectories atT=512T\{=\}512,N=1N\{=\}1on Qwen3\-8B\. Greedy reliably reaches high fitness in CP, while several seeds in MMD and HT stagnate at suboptimal values\.Evolve Trajectory Analysis\.[Section˜5](https://arxiv.org/html/2605.29268#S5)characterized within\-run depth\-breadth allocation on capable model and task\. However, the self\-evolving process itself remains highly stochastic\. Specifically, we run the experiments multiple times under the same allocation configuration \(sequential:T=512,N=1T=512,N=1\) and analyze each*greedy*run’s performance by tracing its fitness score trajectory throughout the game in[Figure˜4](https://arxiv.org/html/2605.29268#S6.F4)\. Evolving may dramatically fail from time to time by converging at a low score without further improvement even under the*same configuration*, producing a distribution \([Appendix˜E](https://arxiv.org/html/2605.29268#A5)\) rather than a single value \([Appendix˜D](https://arxiv.org/html/2605.29268#A4)\), a spread that within\-run allocation cannot remove\. A second allocation lever is therefore needed, one that operates between trajectories rather than within them\.

We instantiate the cross\-trajectory lever as a multi\-armed bandit \(MAB\) overKKparallel runs\. Each arm corresponds to one trajectory initialized from the same seed program; each pull spends one LLM call extending that trajectory and reveals its new fitness\. The policy’s decision after every call, i\.e\., which arm to pull next, routes budget away from stagnating runs and toward more promising ones\.

### 6\.1Bandits Adaptation

MAB provides a framework for sequential decision\-making in uncertain environments\. In the standard setting, the player chooses an action to take from a finite action set\[K\]\[K\]at each roundc∈\[C\]:=\{1,2,…,C\}c\\in\[C\]:=\\\{1,2,\\dots,C\\\}and observes the associated reward, which may immediately factor into the decision in the next round\. Deploying Improving BanditsHeidariet al\.\([2016](https://arxiv.org/html/2605.29268#bib.bib49)\)as the backbone, we propose our adaptive self\-evolving algorithm, namely, Bandits\-based Self\-Evolving \(BaSE\), in[Algorithm˜2](https://arxiv.org/html/2605.29268#alg2)\.

Algorithm 2Bandits\-based Self\-Evolving \(BaSE\)1:Task

qq, base LLM

ff, prompt generator

hh, budget \(number of LLM calls\)

CC, number of runs

KK
2:Initialize

ci←1c\_\{i\}\\leftarrow 1for all

i∈\[K\]i\\in\[K\]
3:for all

i∈\[K\]i\\in\[K\]in paralleldo

4:

prompt←h​\(q\)\\mathrm\{prompt\}\\leftarrow h\\left\(q\\right\)
5:

responsei\(ci\)←f​\(prompt\)\\mathrm\{response\}\_\{i\}^\{\(c\_\{i\}\)\}\\leftarrow f\(\\mathrm\{prompt\}\)
6:

scorei←Eval​\(q,responsei\(ci\)\)\\mathrm\{score\}\_\{i\}\\leftarrow\\mathrm\{Eval\}\\left\(q,\\mathrm\{response\}\_\{i\}^\{\(c\_\{i\}\)\}\\right\)
7:endfor

8:for

c=K\+1,…,Cc=K\+1,\\dots,Cdo

9:

i←MAB​\(1,…,K\)i\\leftarrow\\mathrm\{MAB\}\(1,\\dots,K\)
10:

prompt←h​\(q,\(responsei\(j\),scorei\(j\)\)j∈\[ci\]\)\\mathrm\{prompt\}\\leftarrow h\\left\(q,\\left\(\\mathrm\{response\}\_\{i\}^\{\(j\)\},\\mathrm\{score\}\_\{i\}^\{\(j\)\}\\right\)\_\{j\\in\[c\_\{i\}\]\}\\right\)
11:

responsei\(ci\+1\)←f​\(prompt\)\\mathrm\{response\}\_\{i\}^\{\(c\_\{i\}\+1\)\}\\leftarrow f\(\\mathrm\{prompt\}\)
12:

scorei\(ci\+1\)←Eval​\(q,responsei\(ci\+1\)\)\\mathrm\{score\}\_\{i\}^\{\(c\_\{i\}\+1\)\}\\leftarrow\\mathrm\{Eval\}\\left\(q,\\mathrm\{response\}\_\{i\}^\{\(c\_\{i\}\+1\)\}\\right\)
13:

ci←ci\+1c\_\{i\}\\leftarrow c\_\{i\}\+1
14:endfor

15:

i⋆←arg⁡maxi∈\[K\],j∈\[ci\]⁡scorei\(ci\)i^\{\\star\}\\leftarrow\\arg\\max\_\{i\\in\[K\],j\\in\[c\_\{i\}\]\}\\mathrm\{score\}\_\{i\}^\{\(c\_\{i\}\)\}
16:return

responsei⋆\(ci⋆\)\\mathrm\{response\}\_\{i^\{\\star\}\}^\{\\left\(c\_\{i^\{\\star\}\}\\right\)\}

Intuitively,[Algorithm˜2](https://arxiv.org/html/2605.29268#alg2)treats self\-evolution as a*cross\-trajectory allocation*problem\. Under the same fixed LLM\-call budget, a single\-trajectory method spends all generations on one evolving path, whereasBaSEallocates the budget acrossKKparallel trajectories and adaptively decides which trajectory should receive the next call\. Each arm corresponds to one evolving trajectory initialized from the same seed program, and the bandit policy only decides which trajectory should reveal its next point in an online manner\. It does not alter the local refinement rule or change how a selected trajectory evolves\. In this sense,BaSEimproves search by reallocating computation across heterogeneous trajectories rather than by modifying the trajectory dynamics themselves\.

Table 2:Fitness scores \(mean±\\pmSE\) across models, tasks, and evolution strategies at 512 LLM calls \(C=512C\{=\}512\)\. The maximum fitness score in each \(model, task\) isbolded\.Table 3:Minimum generation \(Gen\.\) and cumulative FLOPs \(×1015\\times 10^\{15\}\) required for≥90%\\geq 90\\%samples to reach thresholds \(≥τ\\geq\\tau\) with Qwen3\-8B as the base LLM\. Unreached thresholds are denoted as “—”\. The minimum generation and FLOPs for each threshold arebolded\.Task𝝉\\boldsymbol\{\\tau\}BaselinesBaSEGreedyOpenEvolveCodeEvolveShinkaEvolveRandUCBEXP3\.PThompsonGen\.FLOPsGen\.FLOPsGen\.FLOPsGen\.FLOPsGen\.FLOPsGen\.FLOPsGen\.FLOPsGen\.FLOPsCP0\.95152182\.12————4557\.905666\.581618\.19139173\.571618\.190\.99359411\.93————7390\.15138170\.8885100\.79207256\.1285101\.220\.999——————————————327395\.47MMD0\.80212318\.46119181\.48——71112\.002632\.34812\.414052\.72812\.410\.90296436\.55——————119179\.0792117\.6395142\.59105144\.750\.95485656\.81——————132199\.9592117\.63114172\.74106146\.75HT0\.50——————201189\.289299\.906066\.90102108\.677479\.450\.70————————367381\.23125130\.55209213\.22101104\.53

### 6\.2Our Results

Fitness Score\.As shown in[Table˜2](https://arxiv.org/html/2605.29268#S6.T2), we report the average fitness score across three tasks, model families, and self\-evolving strategies atC=512C=512generations\.BaSEachieves the best or near\-best fitness across different model families and scales\. For example, Thompson reaches0\.87360\.8736on Qwen3\-8B HT, clearly above greedy \(0\.67800\.6780\) and ShinkaEvolve \(0\.73790\.7379\); on Llama HT, it obtains0\.43870\.4387, outperforming all baselines by a large margin; and on Qwen3\-14B CP, it attains the highest average fitness score of1\.00031\.0003\.

The improvement reflects the value of allocating the fixed budget across multiple evolving trajectories, rather than modifying the refinement dynamics within a trajectory, while keeping the total LLM\-call budget fixed\. Indeed, under the same fixed computation \(generation\) budget, trajectory diversification itself helps avoid committing the full budget to a low\-potential run but adaptively allocates budget to more promising trajectories\.

Although BaSE attains the strongest mean across all reported model\-task pairs, we note that the magnitude of the improvement varies with task difficulty and available headroom\. On saturated tasks such as CP, several baselines already approach the score ceiling; for example, Qwen3\-8B greedy and ShinkaEvolve achieve0\.99850\.9985and0\.99860\.9986, while BaSE\-Thompson reaches1\.00031\.0003with CI\[0\.9980,1\.0004\]\[0\.9980,1\.0004\]\. Hence, the CP gains are necessarily small\. By contrast, BaSE shows clearer benefits on harder settings such as HT, where Qwen3\-8B BaSE\-Thompson reaches0\.87360\.8736compared with the best non\-BaSE score of0\.73790\.7379, and Llama BaSE\-Thompson reaches0\.43870\.4387while all non\-BaSE methods remain below0\.25380\.2538\.

Cross\-Trajectory Allocation with Parent Sampling Protocols\.BaSEis orthogonal to parent\-sampling strategies:BaSEdecides which trajectory receives the next LLM call, while parent sampling decides which historical response is used to refine the selected trajectory\. Thus,[˜10](https://arxiv.org/html/2605.29268#alg2.l10)in[Algorithm˜2](https://arxiv.org/html/2605.29268#alg2)can replace the vanilla prompt generator with existing parent\-sampling methods\. In[Section˜G\.1](https://arxiv.org/html/2605.29268#A7.SS1), we evaluate this combination by pairingBaSEwith different parent\-sampling baselines and comparing against greedy variants under different breadth\-depth allocations\. As shown in[Table˜8](https://arxiv.org/html/2605.29268#A6.T8), these strategies can be naturally integrated intoBaSE, often improving their best fitness scores, especially in unstable settings with high trajectory variance\.

Sample\-Efficient Threshold Reaching\.Beyond final fitness, we evaluate how efficiently each method reaches a target fitness thresholdτ\\tauunder a fixed budget\. Following the time\-to\-threshold metric ofTayloret al\.\([2007](https://arxiv.org/html/2605.29268#bib.bib51)\), we report the earliest generationGGand corresponding cumulative FLOPs at which90%90\\%of bootstrap samples reachτ\\tau\. As shown in[Table˜3](https://arxiv.org/html/2605.29268#S6.T3),BaSEreaches most thresholds with fewer generations and FLOPs, especially on MMD and HT\. For example, on MMD, UCB reachesτ=0\.95\\tau=0\.95at9292generations, while greedy requires485485; on HT, UCB reachesτ=0\.70\\tau=0\.70within6060generations, whereas greedy, OpenEvolve, and CodeEvolve never reach it\. Compared with random allocation, Thompson reaches the same thresholds with∼\\sim40% fewer generations on average across the seven reached cells\. Full results across three models are deferred to[Section˜G\.2](https://arxiv.org/html/2605.29268#A7.SS2)\.

Ablation on Arm\-Pool Size\.We ablate the number of parallel runsKK\(number of arms\) used byBaSEand defer the full results to[Section˜G\.3](https://arxiv.org/html/2605.29268#A7.SS3)\([Table˜10](https://arxiv.org/html/2605.29268#A7.T10)\)\. The results show that moderate arm pools usually provide the best trade\-off between trajectory diversity and per\-run refinement depth\. In particular,K∈\{5,10,20\}K\\in\\\{5,10,20\\\}often achieves the strongest performance, while very small pools can lack sufficient exploration and overly large pools such asK=50K=50may dilute the refinement budget across too many shallow trajectories\.

## 7Discussion

#### Allocation wins, not prompt engineering\.

The CP results in[Table˜2](https://arxiv.org/html/2605.29268#S6.T2)appear to make ShinkaEvolve a rather strong baseline \(0\.99860\.9986on Qwen3\-8B\), while inspection of the prompts in[Appendix˜H](https://arxiv.org/html/2605.29268#A8)shows this advantage could be prompt\-induced: ShinkaEvolve’s CP system prompt explicitly suggestsscipy\.optimize\. While our method use exactly the same prompt as OpenEvolve, steering away from quick\-win hints\. Our greedy baseline reaches0\.99850\.9985on the same task, which is a\+0\.17\+0\.17gap over OpenEvolve\. As[Appendix˜D](https://arxiv.org/html/2605.29268#A4)documents, this gap is the LLM eventually discoveringscipydespite the prompt discouraging it, not on top of a privileged prompt\.

#### Model capability and prompt set the ceiling that allocation can reach\.

Allocation amplifies an existing signal but cannot create one\. On Llama HT \([Table˜2](https://arxiv.org/html/2605.29268#S6.T2)\), OpenEvolve and CodeEvolve collapse to0\.00\.0\(no run produces a valid configuration in 512 calls\) and even Thompson only recovers to0\.43870\.4387, an order of magnitude below Qwen3\-8B on the same task\.[Appendix˜C](https://arxiv.org/html/2605.29268#A3)shows the same pattern across the full sweep: when a model fails to cross the capability threshold on relatively difficult task, depth gains are statistically indistinguishable from selection noise\. The prompt sets a separate ceiling: ShinkaEvolve’s HT prompt \([Appendix˜H](https://arxiv.org/html/2605.29268#A8)\) contains a “CRITICAL — degeneracy warning” about collinear\-triplet failures that no other baseline carries, and this directly addresses the failure mode diagnosed in[Appendix˜D](https://arxiv.org/html/2605.29268#A4)\(HT run 5\)\. Capability and prompt thus jointly determine the achievable ceiling; allocation governs how efficiently a method approaches it\.

#### BaSEis complementary to within\-run mechanisms\.

BaSEoperates at trajectory granularity, while the parent\-sampling protocols in \*Evolve systems operate within a single run; the two compose\. The improvement ofBaSEover a single\-trajectory baseline admits a clean two\-step decomposition:*Greedy/Island*→\\to*Random*isolates the pool effect \(drawingKKindependent trajectories and returning the best\), and*Random*→\\toBaSEisolates the allocation effect \(adaptively routing compute toward promising trajectories\)\. The pool effect is licensed by the per\-run heterogeneity documented in[Figure˜4](https://arxiv.org/html/2605.29268#S6.F4)and[Appendix˜D](https://arxiv.org/html/2605.29268#A4)\. The allocation effect is then isolated by the Random vs\.BaSEgap in[Table˜3](https://arxiv.org/html/2605.29268#S6.T3)\. Both effects survive whenBaSEreplaces the prompt generator of OpenEvolve, CodeEvolve, or ShinkaEvolve \([Table˜8](https://arxiv.org/html/2605.29268#A6.T8)in[Section˜G\.1](https://arxiv.org/html/2605.29268#A7.SS1)\), with the largest gains where the underlying protocol leaves trajectory variance unexploited and the smallest where it has already saturated the task\.

## 8Conclusion

In conclusion, we presented the first fixed\-budget characterization of depth–breadth allocation in LLM\-guided evolutionary search, revealing structured, task\-dependent fitness surfaces and an empirical bilinear regularity\. We then showed that fixed allocations still leave substantial cross\-trajectory heterogeneity\. Motivated by this observation, we introducedBaSE, a bandit\-based allocator that improves expected fitness and threshold\-reaching efficiency over strong*\*Evolve*baselines without changing the base model, evaluator, or prompt design\. This highlights compute allocation as a key lever for reliable finite\-budget search\.

## References

- R\. Agarwal, M\. Schwarzer, P\. S\. Castro, A\. Courville, and M\. G\. Bellemare \(2022\)Deep reinforcement learning at the edge of the statistical precipice\.External Links:2108\.13264,[Link](https://arxiv.org/abs/2108.13264)Cited by:[§4](https://arxiv.org/html/2605.29268#S4.p4.7)\.
- S\. Agrawal and N\. Goyal \(2012\)Analysis of thompson sampling for the multi\-armed bandit problem\.InConference on learning theory,pp\. 39–1\.Cited by:[§4](https://arxiv.org/html/2605.29268#S4.p3.1)\.
- H\. Assumpção, D\. Ferreira, L\. Campos, and F\. Murai \(2025\)CodeEvolve: an open source evolutionary coding agent for algorithmic discovery and optimization\.arXiv preprint arXiv:2510\.14150\.External Links:[Link](https://arxiv.org/abs/2510.14150)Cited by:[§1](https://arxiv.org/html/2605.29268#S1.p1.2)\.
- J\. Audibert and S\. Bubeck \(2010\)Best arm identification in multi\-armed bandits\.InCOLT\-23th Conference on learning theory\-2010,pp\. 13–p\.Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- P\. Auer, N\. Cesa\-Bianchi, and P\. Fischer \(2002a\)Finite\-time analysis of the multiarmed bandit problem\.Machine Learning47\(2\),pp\. 235–256\.Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- P\. Auer, N\. Cesa\-Bianchi, Y\. Freund, and R\. E\. Schapire \(2002b\)The nonstochastic multiarmed bandit problem\.SIAM journal on computing32\(1\),pp\. 48–77\.Cited by:[§4](https://arxiv.org/html/2605.29268#S4.p3.1)\.
- G\. Badkobeh, P\. K\. Lehre, and D\. Sudholt \(2014\)Unbiased black\-box complexity of parallel search\.InParallel Problem Solving from Nature – PPSN XIII,pp\. 892–901\.External Links:[Document](https://dx.doi.org/10.1007/978-3-319-10762-2%5F88)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- M\. Briesch, D\. Sobania, and F\. Rothlauf \(2023\)On the trade\-off between population size and number of generations in GP for program synthesis\.InProceedings of the Companion Conference on Genetic and Evolutionary Computation \(GECCO Companion\),pp\. 535–538\.External Links:[Document](https://dx.doi.org/10.1145/3583133.3590681)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- B\. Brown, J\. Juravsky, R\. Ehrlich, R\. Clark, Q\. V\. Le, C\. Ré, and A\. Mirhoseini \(2024\)Large language monkeys: scaling inference compute with repeated sampling\.arXiv preprint arXiv:2407\.21787\.External Links:[Link](https://arxiv.org/abs/2407.21787)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- M\. Cemri, S\. Agrawal, A\. Gupta, S\. Liu, A\. Cheng, Q\. Mang, A\. Naren, L\. E\. Erdogan, K\. Sen, M\. Zaharia, A\. Dimakis, and I\. Stoica \(2026\)AdaEvolve: adaptive LLM driven zeroth\-order optimization\.arXiv preprint arXiv:2602\.20133\.External Links:[Link](https://arxiv.org/abs/2602.20133)Cited by:[§1](https://arxiv.org/html/2605.29268#S1.p1.2),[§2](https://arxiv.org/html/2605.29268#S2.p1.1),[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- A\. Chen, D\. M\. Dohan, and D\. R\. So \(2023\)EvoPrompting: language models for code\-level neural architecture search\.arXiv preprint arXiv:2302\.14838\.External Links:[Link](https://arxiv.org/abs/2302.14838)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p1.1)\.
- Y\. Chen, X\. Pan, Y\. Li, B\. Ding, and J\. Zhou \(2024\)Provable scaling laws for the test\-time compute of large language models\.arXiv preprint arXiv:2411\.19477\.Note:NeurIPS 2025External Links:[Link](https://arxiv.org/abs/2411.19477)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- B\. Doerr and M\. Künnemann \(2015\)Optimizing linear functions with the\(1\+λ\)\(1\+\\lambda\)evolutionary algorithm—different asymptotic runtimes for different instances\.Theoretical Computer Science561,pp\. 3–23\.External Links:[Document](https://dx.doi.org/10.1016/j.tcs.2014.03.015)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- C\. Gießen and C\. Witt \(2017\)The interplay of population size and mutation probability in the\(1\+λ\)\(1\+\\lambda\)EA on OneMax\.Algorithmica78\(2\),pp\. 587–609\.External Links:[Document](https://dx.doi.org/10.1007/s00453-016-0214-z)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- H\. Heidari, M\. Kearns, and A\. Roth \(2016\)Tight policy regret bounds for improving and decaying bandits\.InProceedings of the Twenty\-Fifth International Joint Conference on Artificial Intelligence,IJCAI’16,pp\. 1562–1570\.External Links:ISBN 9781577357704Cited by:[§6\.1](https://arxiv.org/html/2605.29268#S6.SS1.p1.2)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. Sifre \(2022\)Training compute\-optimal large language models\.External Links:2203\.15556,[Link](https://arxiv.org/abs/2203.15556)Cited by:[Appendix A](https://arxiv.org/html/2605.29268#A1.SS0.SSS0.Px1.p1.6),[§4](https://arxiv.org/html/2605.29268#S4.p5.3),[§5\.1](https://arxiv.org/html/2605.29268#S5.SS1.p1.5)\.
- Y\. Inoue, K\. Misaki, Y\. Imajuku, S\. Kuroki, T\. Nakamura, and T\. Akiba \(2025\)Wider or deeper? scaling llm inference\-time compute with adaptive branching tree search\.External Links:2503\.04412,[Link](https://arxiv.org/abs/2503.04412)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1),[§5\.2](https://arxiv.org/html/2605.29268#S5.SS2.SSS0.Px1.p2.11)\.
- T\. Jansen, K\. A\. De Jong, and I\. Wegener \(2005\)On the choice of the offspring population size in evolutionary algorithms\.Evolutionary Computation13\(4\),pp\. 413–440\.Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- T\. Jansen and C\. Zarges \(2012\)Fixed budget computations: a different perspective on run time analysis\.InProceedings of the 14th Annual Conference on Genetic and Evolutionary Computation \(GECCO\),pp\. 1325–1332\.External Links:[Document](https://dx.doi.org/10.1145/2330163.2330347)Cited by:[§1](https://arxiv.org/html/2605.29268#S1.p3.1),[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- T\. Jansen and C\. Zarges \(2014\)Performance analysis of randomised search heuristics operating with a fixed budget\.Theoretical Computer Science545,pp\. 39–58\.Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- Z\. Karnin, T\. Koren, and O\. Somekh \(2013\)Almost optimal exploration in multi\-armed bandits\.InProceedings of the 30th International Conference on Machine Learning,S\. Dasgupta and D\. McAllester \(Eds\.\),Proceedings of Machine Learning Research, Vol\.28,Atlanta, Georgia, USA,pp\. 1238–1246\.External Links:[Link](https://proceedings.mlr.press/v28/karnin13.html)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- R\. Lange, T\. Schaul, Y\. Chen, T\. Zahavy, V\. Dalibard, C\. Lu, S\. Singh, and S\. Flennerhag \(2023\)Discovering evolution strategies via meta\-black\-box optimization\.InProceedings of the companion conference on genetic and evolutionary computation,pp\. 29–30\.Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p1.1)\.
- R\. T\. Lange, Y\. Imajuku, and E\. Cetin \(2025\)ShinkaEvolve: towards open\-ended and sample\-efficient program evolution\.arXiv preprint arXiv:2509\.19349\.External Links:[Link](https://arxiv.org/abs/2509.19349)Cited by:[§1](https://arxiv.org/html/2605.29268#S1.p1.2),[§2](https://arxiv.org/html/2605.29268#S2.p1.1)\.
- T\. Lattimore and C\. Szepesvári \(2020\)Bandit algorithms\.Cambridge University Press\.Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- K\. Lee, I\. Fischer, Y\. Wu, D\. Marwood, S\. Baluja, D\. Schuurmans, and X\. Chen \(2025\)Evolving deeper llm thinking\.External Links:2501\.09891,[Link](https://arxiv.org/abs/2501.09891)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p1.1)\.
- J\. Lehman, J\. Gordon, S\. Jain, K\. Ndousse, C\. Yeh, and K\. O\. Stanley \(2022\)Evolution through large models\.arXiv preprint arXiv:2206\.08896\.External Links:[Link](https://arxiv.org/abs/2206.08896)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p1.1)\.
- L\. Li, K\. Jamieson, G\. DeSalvo, A\. Rostamizadeh, and A\. Talwalkar \(2018\)Hyperband: a novel bandit\-based approach to hyperparameter optimization\.Journal of Machine Learning Research18\(185\),pp\. 1–52\.Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- E\. Meyerson, M\. J\. Nelson, H\. Bradley, A\. Gaier, A\. Moradi, A\. K\. Hoover, and J\. Lehman \(2024\)Language model crossover: variation through few\-shot prompting\.ACM Transactions on Evolutionary Learning4\(4\),pp\. 1–40\.Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p1.1)\.
- S\. Miyamoto, D\. Oba, and N\. Okazaki \(2026\)Aligning tree\-search policies with fixed token budgets in test\-time scaling of LLMs\.arXiv preprint arXiv:2602\.09574\.External Links:[Link](https://arxiv.org/abs/2602.09574)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- J\. Mouret and J\. Clune \(2015\)Illuminating search spaces by mapping elites\.arXiv preprint arXiv:1504\.04909\.Cited by:[§3](https://arxiv.org/html/2605.29268#S3.p4.1)\.
- R\. Nakano, Y\. Davidor, and T\. Yamada \(1994\)Optimal population size under constant computation cost\.InParallel Problem Solving from Nature – PPSN III,pp\. 130–138\.External Links:[Document](https://dx.doi.org/10.1007/3-540-58484-6%5F257)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. R\. Ruiz, A\. Mehrabian, M\. P\. Kumar, A\. See, S\. Chaudhuri, G\. Holland, A\. Davies, S\. Nowozin, P\. Kohli, and M\. Balog \(2025\)AlphaEvolve: a coding agent for scientific and algorithmic discovery\.arXiv preprint arXiv:2506\.13131\.External Links:[Link](https://arxiv.org/abs/2506.13131)Cited by:[§1](https://arxiv.org/html/2605.29268#S1.p1.2),[§2](https://arxiv.org/html/2605.29268#S2.p1.1)\.
- B\. Romera\-Paredes, M\. Barekatain, A\. Novikov, M\. Balog, M\. P\. Kumar, E\. Dupont, F\. J\. R\. Ruiz, J\. S\. Ellenberg, P\. Wang, O\. Fawzi, P\. Kohli, and A\. Fawzi \(2024\)Mathematical discoveries from program search with large language models\.Nature625\(7995\),pp\. 468–475\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06924-6)Cited by:[§1](https://arxiv.org/html/2605.29268#S1.p1.2),[§2](https://arxiv.org/html/2605.29268#S2.p1.1),[§3](https://arxiv.org/html/2605.29268#S3.p4.1)\.
- A\. Sharma and P\. Chopra \(2025\)The sequential edge: inverse\-entropy voting beats parallel self\-consistency at matched compute\.arXiv preprint arXiv:2511\.02309\.External Links:[Link](https://arxiv.org/abs/2511.02309)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- C\. Shi, K\. Yang, Z\. Chen, J\. Li, J\. Yang, and C\. Shen \(2024\)Efficient prompt optimization through the lens of best arm identification\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2402\.09723Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- C\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2024\)Scaling LLM test\-time compute optimally can be more effective than scaling model parameters\.arXiv preprint arXiv:2408\.03314\.External Links:[Link](https://arxiv.org/abs/2408.03314)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1),[§5\.2](https://arxiv.org/html/2605.29268#S5.SS2.SSS0.Px1.p2.11)\.
- R\. Tanese \(1989\)Distributed genetic algorithms for function optimization\.Ph\.D\. Thesis,University of Michigan,Ann Arbor, MI\.Note:ProQuest Dissertations & Theses, 9001722External Links:[Link](https://deepblue.lib.umich.edu/handle/2027.42/162372)Cited by:[§3](https://arxiv.org/html/2605.29268#S3.p4.1)\.
- H\. Tang, K\. Hu, J\. P\. Zhou, S\. Zhong, W\. Zheng, X\. Si, and K\. Ellis \(2024\)Code repair with LLMs gives an exploration\-exploitation tradeoff\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2405.17503)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- M\. E\. Taylor, P\. Stone, and Y\. Liu \(2007\)Transfer learning via inter\-task mappings for temporal difference learning\.Journal of Machine Learning Research8\(73\),pp\. 2125–2167\.External Links:[Link](http://jmlr.org/papers/v8/taylor07a.html)Cited by:[§6\.2](https://arxiv.org/html/2605.29268#S6.SS2.p5.10)\.
- W\. R\. Thompson \(1933\)On the likelihood that one unknown probability exceeds another in view of the evidence of two samples\.Biometrika25\(3/4\),pp\. 285–294\.Cited by:[§4](https://arxiv.org/html/2605.29268#S4.p3.1)\.
- Y\. Wang, S\. Su, Z\. Zeng, E\. Xu, L\. Ren, X\. Yang, Z\. Huang, X\. He, L\. Ma, B\. Peng, H\. Cheng, P\. He, W\. Chen, S\. Wang, S\. S\. Du, and Y\. Shen \(2025\)ThetaEvolve: test\-time learning on open problems\.arXiv preprint arXiv:2511\.23473\.External Links:[Link](https://arxiv.org/abs/2511.23473)Cited by:[§1](https://arxiv.org/html/2605.29268#S1.p1.2),[§2](https://arxiv.org/html/2605.29268#S2.p1.1)\.
- H\. Wen, Y\. Su, F\. Zhang, Y\. Liu, Y\. Liu, Y\. Zhang, and Y\. Li \(2025\)ParaThinker: native parallel thinking as a new paradigm to scale LLM test\-time compute\.arXiv preprint arXiv:2509\.04475\.External Links:[Link](https://arxiv.org/abs/2509.04475)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- Y\. Wu, Z\. Sun, S\. Li, S\. Welleck, and Y\. Yang \(2024\)Inference scaling laws: an empirical analysis of compute\-optimal inference for problem\-solving with language models\.arXiv preprint arXiv:2408\.00724\.External Links:[Link](https://arxiv.org/abs/2408.00724)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p2.1)\.
- C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. Chen \(2023\)Large language models as optimizers\.arXiv preprint arXiv:2309\.03409\.External Links:[Link](https://arxiv.org/abs/2309.03409)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p1.1)\.
- Y\. Zhang, Y\. Duan, Z\. Zhang, J\. He, and S\. Zheng \(2025\)Population\-Evolve: a parallel sampling and evolutionary method for LLM math reasoning\.arXiv preprint arXiv:2512\.19081\.External Links:[Link](https://arxiv.org/abs/2512.19081)Cited by:[§2](https://arxiv.org/html/2605.29268#S2.p1.1)\.

## Appendix Contents

## Appendix AEffective FLOPs

The protocols differ in cache behavior as a structural consequence of their prompt construction, where greedy reuses a single parent prefix across batched siblings in a generation, while island rewrites the parent \+ inspirations per call\.

#### Formula\.

We charge each completed LLM call

FLOPscall=2​Pactive​\(puncached\+pout\),\\textsc\{FLOPs\}\_\{\\text\{call\}\}=2\\,P\_\{\\text\{active\}\}\\,\(p\_\{\\text\{uncached\}\}\+p\_\{\\text\{out\}\}\),wherePactiveP\_\{\\text\{active\}\}is the active parameter count,puncached=pprompt−pcachedp\_\{\\text\{uncached\}\}=p\_\{\\text\{prompt\}\}\-p\_\{\\text\{cached\}\}, andpoutp\_\{\\text\{out\}\}is the completion token count\. Per\-run FLOPs sums over all attempts, including retries\. Token counts and cache statistics are read directly from vLLM’s per\-requestusagerecords\. The factor2​Pactive2\\,P\_\{\\text\{active\}\}per token is the forward\-pass cost in theC≈6​N​DC\\approx 6NDin the spirit of ChinchillaHoffmannet al\.\([2022](https://arxiv.org/html/2605.29268#bib.bib1)\), using only the forward2​N2Ncomponent as no backward pass occurs in LLM\-guided evolution\. As in Chinchilla, this convention omits sub\-leading terms \(attention score computation, softmax, layer norms, embeddings, sampling\); Chinchilla’s reports agreement with detailed accounting to within 10% over two orders of magnitude\.

#### Active parameters\.

All models in our sweep \(Qwen3 1\.7B/4B/8B/14B and Llama\-3\.1\-8B\) are dense, soPactiveP\_\{\\text\{active\}\}equals the total parameter count\. The notation is retained for compatibility with MoE models\.

#### Prefix\-cached tokens\.

Cached prompt tokens are subtracted because vLLM’s PagedAttention reuses their KV states without recomputing the corresponding forward pass\. Cache hit rates in our sweeps range from 94% to 98%; omitting this subtraction over\-reports FLOPs by 6–9%\. The residual attention computation over cached tokens is sub\-leading and is omitted, consistent with the convention above\.

#### Non\-embedding variant\.

Some scaling\-law conventions use non\-embedding parameter counts in place ofPactiveP\_\{\\text\{active\}\}\. Substituting would multiply every reported FLOPs value by a fixed per\-model factorPnon\-embed/PactiveP\_\{\\text\{non\-embed\}\}/P\_\{\\text\{active\}\}, preserving all cross\-model ratios and orderings used in our analysis\.

## Appendix BTask Details

For all three tasks, raw objectives are normalized by the best published construction so thatfitness=1\.0\\textsc\{fitness\}\{=\}1\.0matches the state of the art:

Circle Packing \(CP,n=26n\{=\}26\)\.Pack2626non\-overlapping circles into the unit square to maximize the sum of their radii\. Fitness is normalized to the AlphaEvolve reference of2\.6352\.635forn=26n=26, so normalizer:fitness=sumradii/2\.635\\textsc\{fitness\}=\\textsc\{sum\}\_\{\\text\{radii\}\}/2\.635\.

MinMaxDist \(MMD,n=16,d=2n\{=\}16,d\{=\}2\)\.Place1616points in the plane to maximize the ratio of minimum to maximum pairwise distance,dmin/dmaxd\_\{\\min\}/d\_\{\\max\}\. Fitness is the squared ratio normalized to the benchmark of1/12\.889266112≈0\.27861/\\sqrt\{12\.889266112\}\\approx 0\.2786, normalizer:fitness=\(dmin/dmax\)2⋅12\.889266112\\textsc\{fitness\}=\(d\_\{\\min\}/d\_\{\\max\}\)^\{2\}\\cdot 12\.889266112; The configuration space contains a sharp deterministic attractor at fitness≈0\.9603\\approx 0\.9603corresponding to a5\+115\{\+\}11construction\.

Heilbronn Triangle \(HT,n=11n\{=\}11\)\.Place1111points on or inside the equilateral triangle with vertices\(0,0\)\(0,0\),\(1,0\)\(1,0\), and\(0\.5,3/2\)\(0\.5,\\sqrt\{3\}/2\)\. The evaluator enumerates all\(113\)\\binom\{11\}\{3\}triplets and computes the smallest triangle area, normalized by the area of the containing equilateral triangle\. The final score divides this normalized minimum area by the referencerAE=0\.036529889880030156r\_\{\\mathrm\{AE\}\}=0\.036529889880030156\.

Note that subsequent \*Evolve runs have marginally surpassed the2\.6352\.635AlphaEvolve CP reference, sofitnessvalues slightly above1\.01\.0reflect this margin rather than a novel construction; we retain2\.6352\.635as the normalizer for harness consistency with prior work\.

## Appendix CThe Capability Gate of the Collapse

The fitness–FLOPs envelope of[Section˜5\.1](https://arxiv.org/html/2605.29268#S5.SS1)collapses across model size only partially: capable models lie on one curve, weaker ones fall below it\. To characterise the boundary we compute, per model×\\timestask cell, the best\-of\-NNfitnessVT=1V\_\{T\{=\}1\}, the best mean fitness over all tested depthsVmaxV\_\{\\max\}, the depth gainpenBoN=Vmax−VT=1\\mathrm\{penBoN\}=V\_\{\\max\}\-V\_\{T\{=\}1\}, and a permutationpp\-value for that gain \([Table˜4](https://arxiv.org/html/2605.29268#A3.T4)\)\.

Under the null that depth has no effect, we shuffle per\-seed fitness acrossTT, recomputepenBoN\\mathrm\{penBoN\}on each shuffle, and letppbe the fraction of20,00020\{,\}000shuffles meeting or exceeding the observed value\. A bootstrap CI is uninformative becauseVmaxV\_\{\\max\}already takes a max over∼10\\sim\\\!10depth cells, forcingpenBoN≥0\\mathrm\{penBoN\}\\geq 0on every resample; the permutation null re\-maxes too and so absorbs that selection inflation\.

Table 4:Capability gate per model×\\timestask cell\.VT=1V\_\{T\{=\}1\}is the best\-of\-NNfitness,VmaxV\_\{\\max\}the best over all depths,penBoN=Vmax−VT=1\\mathrm\{penBoN\}=V\_\{\\max\}\-V\_\{T\{=\}1\}, andppthe depth permutation test\. 1\.7B/4B evaluated atC=128C\{=\}128, all others atC=512C\{=\}512\.ModelTaskVT=1V\_\{T\{=\}1\}VmaxV\_\{\\max\}penBoNpp1\.7BCP0\.5820\.615\+0\.033\+0\.0330\.484BCP0\.7380\.790\+0\.052\+0\.0520\.308BCP0\.9890\.999\+0\.009\+0\.0090\.1314BCP0\.9260\.980\+0\.054\+0\.0540\.07LlamaCP0\.8430\.843\+0\.000\+0\.0001\.001\.7BMMD0\.7160\.748\+0\.031\+0\.0310\.164BMMD0\.6940\.906\+0\.213\+0\.2130\.0028BMMD0\.9440\.960\+0\.016\+0\.0160\.6414BMMD0\.8670\.981\+0\.115\+0\.115<10−3\{<\}10^\{\-3\}LlamaMMD0\.6730\.843\+0\.170\+0\.1700\.238BHT0\.3390\.672\+0\.333\+0\.3330\.002LlamaHT0\.0470\.253\+0\.206\+0\.2060\.013

The depth signal turns on with scale within Qwen3\. At 1\.7BpenBoN\\mathrm\{penBoN\}is small and not significant on either task \(p≥0\.16p\\geq 0\.16\)\. At 4B it appears on MMD \(\+0\.213\+0\.213,p=0\.002p=0\.002\) but not on CP \(p=0\.30p=0\.30\)\. On CP and MMD, 8B and 14B reachVmax≥0\.96V\_\{\\max\}\\geq 0\.96on every tested cell, near the task ceiling, and depth gains compress to small residuals, while the exception is 14B/MMD,\+0\.115\+0\.115atp<10−3p<10^\{\-3\}\. On the unsaturated HT task, 8B/HT instead shows the largest clean depth gain in the table:VVrises from0\.3390\.339to0\.6720\.672\(\+0\.333\+0\.333,p=0\.002p=0\.002\)\. The depth signal therefore turns on whenever a capable model has room above best\-of\-NN\. The threshold is per \(model, task\), not per model: 4B sits on either side depending on the task\.

Most striking is Llama\-3\.1\-8B\. On CP best\-of\-NNalready reaches0\.8430\.843and depth does not improve it \(penBoN=0\\mathrm\{penBoN\}=0,p=1\.00p=1\.00\)\. On MMD the nominal\+0\.170\+0\.170depth gain is consistent with max\-selection noise \(p=0\.23p=0\.23\)\. On HT depth does compound \(VVrises from0\.0470\.047to0\.2530\.253atp=0\.013p=0\.013\), but the absolute level stays at0\.250\.25, far below the0\.670\.67that Qwen3\-8B reaches on the same task\. Despite its 8B parameters, Llama\-3\.1\-8B never achieves both a significant depth gain and a non\-trivial absolute level on any tested task\. Crossing the threshold is thus not a matter of parameter count\.

## Appendix DCase Studies: Why Runs Stagnate

The main text \([Figure˜4](https://arxiv.org/html/2605.29268#S6.F4)\) shows that ten greedy runs of an identical configuration produce very different final fitness: some climb smoothly, others stop improving early and sit at a low score for the rest of theC=512C\{=\}512budget\. This appendix opens up that variability and asks, at the level of the actual Python code the LLM produced,*why*certain runs get stuck\.

Each run is a chain of accepted programs\. At every generation, greedy keeps the single best\-scoring program seen so far and asks the LLM to mutate it\. A child is accepted only if it strictly beats this running best\. So a run gets “stuck” when, for many generations in a row, the LLM proposes mutations that the evaluator either rejects \(returns a worse score\) or cannot score at all \(invalid output\)\. To understand a stuck run we therefore have to look at two things: \(i\) what program the run is anchored on, and \(ii\) why no proposed mutation beats it\.

[Figure˜5](https://arxiv.org/html/2605.29268#A4.F5)shows one healthy and one stuck run per task, using the actual point/circle configurations the anchored program outputs\. The rest of this appendix walks through each task in turn\.

![Refer to caption](https://arxiv.org/html/2605.29268v1/x5.png)Figure 5:One healthy \(top\) and one stuck \(bottom\) greedy run per task at GreedyT=512T\{=\}512,N=1N\{=\}1on Qwen3\-8B\. Each title gives the run number \(out of 10\), the generation at which the displayed program was the running best, the normalized fitness, and the task’s raw metric\.CP: the unit square is filled with 26 non\-overlapping circles;MMD: 16 points; the red bar is the closest pair \(dmind\_\{\\min\}\) and the blue dashed bar the farthest pair \(dmaxd\_\{\\max\}\);HT: 11 points inside the equilateral triangle; the red shaded triplet is the smallest of the\(113\)=165\\binom\{11\}\{3\}\{=\}165triangles, which sets the score\.### D\.1Circle Packing:scipyacquisition is the gating event

#### The task in one sentence\.

Pack2626non\-overlapping circles into the unit square so that the sum of their radii is as large as possible; the AlphaEvolve reference is∑iri=2\.635\\sum\_\{i\}r\_\{i\}=2\.635, accordingfitness=1\.0\\text\{fitness\}\{=\}1\.0base on this sum of radii matches the state of the art\.

#### Two regimes the LLM oscillates between\.

On CP the LLM writes programs that fall into two clearly different families:

1. \(a\)*Hand\-coded ring constructors:*the program lays2626circles down in geometric rows \(e\.g\.6\+5\+6\+5\+46\{\+\}5\{\+\}6\{\+\}5\{\+\}4hexagonal rows\) and assigns each circle the same fixed radius based on the row spacing\. No optimization is performed\. The best score this family can produce in our runs is∼0\.89\\sim\\\!0\.89, because uniform radii waste space in the corners and at the row endings\.
2. \(b\)scipy\.optimize\.minimize\-based programs: the same row layout is used as an initial guess, but the program then calls SLSQP or COBYLA with non\-overlap and box\-containment constraints, letting both centers and radii float\. These programs reach∼1\.0\\sim\\\!1\.0\.

Acquiringscipyis therefore the gating event for high fitness on CP: every run that finishes above0\.990\.99is anchored on ascipyprogram; the one run that does not is the one where the LLM took longest to write ascipycall\.

#### Healthy example — run 10\.

At generation 13, the LLM proposes the first child that importsscipyand wraps the ring layout in a constrained SLSQP call\. The accepted score jumps from0\.55840\.5584to0\.83960\.8396in a single generation, a\+0\.28\+0\.28improvement that no later generation matches\. Within another threescipy\-family accepts the run is at0\.99810\.9981\(gen2424\); few more refinements over the next generations push it to∑iri=2\.636\\sum\_\{i\}r\_\{i\}=2\.636\([Figure˜5](https://arxiv.org/html/2605.29268#A4.F5)\)\.

#### Stuck example — run 6\.

The LLM proposes hand\-coded ring constructors for the first437437generations of this run\. Within that family, the anchored program slowly improves through55accepts from0\.360\.36\(initial\) to0\.670\.67\(gen8888\), then sits at0\.670\.67for233233generations, then jumps to0\.790\.79\(gen321321\) and0\.820\.82\(gen327327\) on two more ring\-rewrites with non\-hex vertical spacing, and freezes at0\.820\.82for another110110generations\. The displayed gen\-437437best \(∑iri=2\.167\\sum\_\{i\}r\_\{i\}=2\.167, panel “CP run 6 gen 437” in[Figure˜5](https://arxiv.org/html/2605.29268#A4.F5)\) is the entire visible cost of this stretch: hundreds of attempts to improve a constructor whose intrinsic ceiling is below0\.890\.89\. The firstscipychild finally arrives at generation438438, producing a\+0\.16\+0\.16jump in one step \(from0\.8220\.822to0\.9830\.983\)\.

### D\.2MinMaxDist: the analytic attractor wins;scipy*poisons*the run

#### The task in one sentence\.

Place1616points in the plane to maximize\(dmin/dmax\)2\(d\_\{\\min\}/d\_\{\\max\}\)^\{2\}— the squared ratio of the closest pair to the farthest pair — normalized so thatfitness=1\.0\\text\{fitness\}\{=\}1\.0matches the AlphaEvolve reference\.

#### An analytic attractor exists\.

For this problem, the5\+115\{\+\}11regular\-polygon construction —55points uniformly on an inner circle of radiusrr,1111points uniformly on an outer circle of radiusR=r​\(1\+2​sin⁡\(π/5\)\)R=r\\,\(1\+2\\sin\(\\pi/5\)\)— reaches the mathematical optimum for this split at fitness0\.96030\.9603\. It involves no optimizer at all: justnp\.linspaceandnp\.cos/sincalls in roughly fifteen lines of code\.

#### scipyis the*bad*family on MMD\.

Counter\-intuitively to the CP story, on MMD the LLM’sscipy\-based programs are exactly the ones that get stuck\. Across the ten runs:

- •3 Runs finish above the attractor \(0\.96030\.9603\), allscipy\-free: pure analytic5\+115\{\+\}11polygons\.
- •6 Runs finish below the attractor \(0\.580\.58–0\.950\.95\)\. All six final programs call eitherdifferential\_evolutionorminimize, on an objective of the form−\(dmin/dmax\)2\-\(d\_\{\\min\}/d\_\{\\max\}\)^\{2\}\.
- •Run 5 finishes at0\.860\.86withoutscipybut also without the polygon construction\.

The reason thescipyprograms fail is that the objective−\(dmin/dmax\)2\-\(d\_\{\\min\}/d\_\{\\max\}\)^\{2\}is highly non\-convex; numerical optimizers from a generic initial guess \(4×\\times4 grid, 16\-gon, etc\.\) converge to local optima where one pair of points is much closer than the rest, draggingdmind\_\{\\min\}down\. The first such program greedy accepts becomes the run’s permanent anchor, because future proposed mutations have to beat its specific stochastic local optimum in a single re\-evaluation\.

#### Healthy Attractor example — run 1\.

The run climbs in44accepts from0\.020\.02\(initial:1616random Gaussian points\) through intermediates \(0\.720\.72,0\.860\.86,0\.890\.89\) to0\.96030\.9603at gen1919, where the LLM writes the closed\-form polygon construction\. The displayed gen\-1919best \([Figure˜5](https://arxiv.org/html/2605.29268#A4.F5)\) shows two clean concentric rings with all inner\-to\-outer distances equal; the closest pair sits between adjacent outer points and the farthest pair spans a diameter\.

#### Stuck example — run 10\.

The run accepts five different scipy\-based programs \(gens 2, 3, 5, 7, 8\), cycling through scipy\.minimize from a 16\-gon, then concentric squares, then differential evolution from concentric circles, finally locking onto a differential evolution call from a 4×4 grid with strategy rand1bin at gen 8\. By gen88the run is anchored at0\.67460\.6746\. For the remaining504504generations greedy accepts nothing\. The displayed gen\-512512best \([Figure˜5](https://arxiv.org/html/2605.29268#A4.F5)\) shows the consequence:1414of the1616points are reasonably spread, but theDElocal optimum happens to leave one pair noticeably closer than the rest — the red bar in the figure — and that single tight pair fixesdmind\_\{\\min\}at roughly20%20\\%ofdmaxd\_\{\\max\}\.

On CP, scipy\-based accepted programs score on average\+0\.35\+0\.35higher than non\-scipy ones \(0\.990\.99vs\.0\.640\.64\); on MMD the sign flips, where scipy programs average−0\.14\-0\.14lower \(0\.590\.59vs\.0\.730\.73\)\. Both tasks have a clear “correct” algorithmic family:scipyfor CP, analytic polygon for MMD\. And both have a wrong family that imposes a hard ceiling\. Whether greedy escapes depends entirely on which family it locks into first\.

### D\.3Heilbronn Triangle: the same mechanism, with stricter feasibility

#### The task in one sentence\.

Place1111points on or inside the equilateral triangle with vertices\(0,0\)\(0,0\),\(1,0\)\(1,0\),\(0\.5,3/2\)\(0\.5,\\sqrt\{3\}/2\)to maximize the area of the smallest triangle formed by any three of them\.

HT differs from CP and MMD in two ways\. First, the initial program returns1111zeros and scores0\.00\.0, so every run begins with a long stretch of invalid output — the LLM has to write*something*that puts1111points inside the triangle before any fitness signal can be received\. Second, there is no analytic attractor: every accepted improvement past the initial valid configuration is a numerical refinement, and the score is set by a single bottleneck triplet whose area must be enlarged without destroying the area of any of the other164164triplets\.

#### Healthy example — run 6\.

After4242generations of invalid output, the run accepts its first valid configuration \(score0\.24450\.2445\) and then climbs through1313further small accepts to0\.95100\.9510at generation470470\. The anchor program at gen470470usesscipy\.optimize\.minimizewith the SLSQP method, called from100100randomly perturbed initial guesses, with*hard inequality constraints*for the three triangle edges \(one constraint per point per edge\) and a log\-sum\-exp soft\-min \(1k​log​∑ie−k​ai\\frac\{1\}\{k\}\\log\\sum\_\{i\}e^\{\-ka\_\{i\}\},k=1000k\{=\}1000\) over the165165triplet areas as the objective\. The three structural choices — hard constraints, multi\-start, smooth soft\-min — are what distinguish this program from every stuck\-HT program\.

#### Stuck example — run 5\.

At gen 46 the run anchors on adifferential\_evolutionprogram with bounds\[0,1\]22\[0,1\]^\{22\}\(the unit square, which strictly contains the triangle\) and a soft\+1000\+1000penalty per outside\-triangle violator; anchor score is0\.26410\.2641\. The run then freezes for466466further generations, during which465465children evaluate to fitness0\.00\.0— DE returns at least one out\-of\-triangle point and the evaluator raises aValueError\. The soft penalty is the structural culprit: a flat constant above a sharp feasibility boundary, with no gradient telling DE which way to pull a violator back in\. The single positive\-fit child \(0\.25770\.2577\) is still below the anchor\. The displayed gen\-512512best \([Figure˜5](https://arxiv.org/html/2605.29268#A4.F5)\) shows the pathology: three of the eleven points lie on a near\-straight diagonal, and the small triangle they form is the bottleneck\.

#### HT in summary\.

The mechanism is the same as MMD — greedy commits to one algorithmic template early, and the template itself imposes the ceiling — but with one extra failure mode specific to HT: when the chosen template uses soft penalties on a hard feasibility constraint, the vast majority of mutations fail the constraint outright and are silently rejected at score0\.00\.0\.

#### Implication for depth\-breadth\.

Across all three tasks, outcomes are gated by which algorithmic family the run anchors on in the initial generations\. Breadth and depth play complementary roles here: breadth gives more parallel attempts at finding a viable family, while depth refines within the family once anchored\. This explains why tasks with multiple near\-viable families benefit from balanced allocation, whereas tasks with a single dominant family tolerate either extreme\.

![Refer to caption](https://arxiv.org/html/2605.29268v1/x6.png)Figure 6:Cross\-model allocation and performance scaling\. Top: near\-optimal depthT∗​\(C\)T^\{\*\}\(C\)as a function of total LLM call budget\. Bottom: best attainable mean fitnessVmax​\(C\)V\_\{\\max\}\(C\)at the same budgets\. The depth curves do not form a single transferable law, whileVmax​\(C\)V\_\{\\max\}\(C\)is substantially smoother and more capability\-ordered\.![Refer to caption](https://arxiv.org/html/2605.29268v1/x7.png)Figure 7:Final fitness of repeated runs under an identical configuration\.Each dot is one run; horizontal bars mark the mean \(dashed\), and IQM \(solid\)\. For every configuration, repeated runs spread over a range of final fitness rather than collapsing to a single value\. Grey dashed line: initial\-program fitness\.![Refer to caption](https://arxiv.org/html/2605.29268v1/x8.png)Figure 8:Fitness of children from a single mutation step\.Distribution of child fitness atT=1T=1, where all children share the initial program as parent \(log count axis\)\. One application of the LLM mutation operator already spreads child fitness broadly across\[0,1\]\[0,1\]instead of concentrating them near the parent; this step\-level spread is the source of the run\-to\-run spread in Figure[7](https://arxiv.org/html/2605.29268#A4.F7)\.

## Appendix EDistribution

In the main text we model each self\-evolution run as one sample from a latent distribution determined by the allocation configuration\(model,task,C,T\)\(\\text\{model\},\\text\{task\},C,T\)\. This appendix gives the empirical support for that view\. We run every configuration with1010independent runs and look at the fitness these repeated runs produce\.

Figure[7](https://arxiv.org/html/2605.29268#A4.F7)shows the final fitness of every run\. Repeating the same configuration does not return the same result: the different runs spread over a range of final finess\. This is precisely the “heterogeneous final fitness score” described in the main text—a single run is one draw, and which draw one obtains varies from run to run, which is why we treat a run as a sample from a latent distribution rather than as a fixed quantity\.

Figure[8](https://arxiv.org/html/2605.29268#A4.F8)traces this spread to its source\. AtT=1T=1all children are generated from the same parent \(the initial program\), so their fitness scores show the effect of a single mutation step\. Even one step already spreads the child fitness broadly across\[0,1\]\[0,1\]instead of concentrating them near the parent\. The run\-to\-run spread in Figure[7](https://arxiv.org/html/2605.29268#A4.F7)is the accumulation of this step\-level spread overTTgenerations\.

Because each configuration yields a spread rather than a single value, we summarise it by the interquartile mean \(IQM\), the mean of the central half of the runs\. As Figure[7](https://arxiv.org/html/2605.29268#A4.F7)shows, the mean, median, and IQM of each configuration are close, so the reported results are not sensitive to this choice; we use IQM for robustness to occasional unusually high or low trajectories\.

## Appendix FDepth–Breadth Regularity

### F\.1Robustness to Richer Specifications

[Section˜5\.2](https://arxiv.org/html/2605.29268#S5.SS2)claims that adding higher\-order terms to the bilinear form \([Equation˜2](https://arxiv.org/html/2605.29268#S5.E2)\) does not materially improve the fit\.[Table˜5](https://arxiv.org/html/2605.29268#A6.T5)reports theR2R^\{2\}of five nested specifications on the same sub\-ceiling cells \(V<0\.97V<0\.97\) as[Table˜7](https://arxiv.org/html/2605.29268#A6.T7), plus an F\-test of each richer specification against the bilinear \(M1\) baseline\.

Table 5:Nested model comparison on the five main\-experiment cells\. M0: budget\-only \(c=0c\{=\}0,a=ba\{=\}b\)\. M1: bilinear \([Equation˜2](https://arxiv.org/html/2605.29268#S5.E2)\)\. M2/M3/M4: M1 pluslog2⁡T\\log^\{2\}T, pluslog2⁡N\\log^\{2\}N, or plus both\.pFp\_\{\\\!F\}columns give the F\-testpp\-value for the richer model vs\. M1\.R2R^\{2\}F\-test vs\. M1TaskModelM0M1M2M3M4pF​\(M​2\)p\_\{\\\!F\}\(M2\)pF​\(M​3\)p\_\{\\\!F\}\(M3\)pF​\(M​4\)p\_\{\\\!F\}\(M4\)CP8B0\.7800\.8110\.8160\.8120\.8190\.440\.860\.60CP14B0\.7650\.7850\.8300\.8060\.8330\.0020\.040\.005MMD8B0\.7440\.9160\.9170\.9170\.9170\.620\.500\.78MMD14B0\.7540\.8690\.8760\.8700\.8770\.130\.810\.29HT8B0\.5260\.5260\.8530\.8530\.9000\.9000\.8560\.8560\.9030\.9030\.0000\.360\.360\.000The bilinear form is preferred on three of the five cells \(CP 8B, MMD 8B, MMD 14B\): F\-tests against M2, M3, and M4 are allp\>0\.1p\>0\.1and theR2R^\{2\}gain from any richer specification is at most\+0\.008\+0\.008\. CP 14B and HT 8B are the exceptions — on both, addinglog2⁡T\\log^\{2\}Tis significant \(p≤0\.002p\\leq 0\.002\) and yields aΔ​R2\\Delta R^\{2\}of\+0\.045\+0\.045and\+0\.047\+0\.047respectively\. On those two rows a richer specification fits noticeably better, though the fittedccremains small in magnitude in both bilinear and richer forms\.

The qualitative regime classification throughcc, however, is robust to spec choice on all five cells\.[Table˜6](https://arxiv.org/html/2605.29268#A6.T6)reports the interaction coefficient under each spec\.

Table 6:Interaction coefficientccunder each specification\. Magnitudes shift but the regime \(corner\-favoring vs\. interior\-favoring\) does not: CP and HT rows stay near zero, MMD rows stay clearly negative\.On the MMD rowsccis essentially unchanged across specs \(within±0\.02\\pm 0\.02of the bilinear estimate\), so the interior\-optimum classification is unambiguous\. On the CP and HT rowsccvaries more in magnitude and even flips sign across specs, but every estimate has\|c\|<0\.12\|c\|<0\.12and these rows are already not statistically distinguishable from zero under M1 \([Table˜7](https://arxiv.org/html/2605.29268#A6.T7)\); the flip reflects noise around thec≈0c\\approx 0limit rather than a genuine regime change, consistent with the discussion in[Section˜F\.3](https://arxiv.org/html/2605.29268#A6.SS3)\. We therefore retain the bilinear form for the narrative in[Section˜5\.2](https://arxiv.org/html/2605.29268#S5.SS2): it is the simplest specification that captures the regime structure, and the structure itself does not change under the richer specifications\.

### F\.2Plateau Width and Online Search

The derivation below is an algebraic consequence of the bilinear ansatz of[Equation˜2](https://arxiv.org/html/2605.29268#S5.E2), not an independently fitted scaling law\. The closed\-formT∗​\(C\)∝CT^\{\*\}\(C\)\\propto\\sqrt\{C\}that appears en route to the plateau\-width result inherits its12\\tfrac\{1\}\{2\}exponent from the symmetry of thelog⁡T​log⁡N\\log T\\log Ncross\-term; the empiricalT∗​\(C\)T^\{\*\}\(C\)curves in[Figure˜6](https://arxiv.org/html/2605.29268#A4.F6)are noisy and non\-monotone across models and do not pin this exponent down independently\. We therefore use the closed form only as a stepping stone to the plateau\-width result, which is the substantive claim\.

Near the optimumT∗​\(C\)T^\{\*\}\(C\), we expandlog⁡\(1−V\)\\log\(1\-V\)as a quadratic inlog⁡T\\log Talong the budget sliceC=T​NC=TN\. Substitutinglog⁡N=log⁡C−log⁡T\\log N=\\log C\-\\log Tinto[Equation˜2](https://arxiv.org/html/2605.29268#S5.E2)gives

log⁡\(1−V\)=\(β0\+b​log⁡C\)\+\(a−b\+c​log⁡C\)​log⁡T−c​\(log⁡T\)2,\\begin\{split\}\\log\(1\{\-\}V\)&=\(\\beta\_\{0\}\+b\\log C\)\+\(a\{\-\}b\{\+\}c\\log C\)\\log T\\\\ &\\quad\-c\(\\log T\)^\{2\},\\end\{split\}\(3\)which is a quadratic inlog⁡T\\log Twith curvature−2​c\-2cand vertex at

log⁡T∗​\(C\)=a−b\+c​log⁡C2​c=a−b2​c\+log⁡C2\.\\log T^\{\*\}\(C\)=\\frac\{a\-b\+c\\log C\}\{2c\}=\\frac\{a\-b\}\{2c\}\+\\frac\{\\log C\}\{2\}\.\(4\)The fitness gap at distanceδ=log⁡T−log⁡T∗\\delta=\\log T\-\\log T^\{\*\}from the optimum satisfies

log⁡\(1−V\)−log⁡\(1−V∗\)=−c​δ2,\\log\(1\-V\)\-\\log\(1\-V^\{\*\}\)=\-c\\,\\delta^\{2\},\(5\)so the plateau half\-width withinΔ\\Deltaof the optimal log fitness gap is

\|δ\|≤Δ\|c\|\.\|\\delta\|\\leq\\sqrt\{\\frac\{\\Delta\}\{\|c\|\}\}\.\(6\)When\|c\|\|c\|is small the plateau is wide and missingT∗T^\{\*\}costs little fitness; when\|c\|\|c\|is large the ridge is sharp but the stronger curvature signal makesT∗T^\{\*\}easier to identify from online feedback\. In either case a cheap online searcher achieves near\-maximum in\-sweep fitness, justifying the approach of[Section˜6](https://arxiv.org/html/2605.29268#S6)\.[Figure˜6](https://arxiv.org/html/2605.29268#A4.F6)shows this empirically: the estimatedT∗​\(C\)T^\{\*\}\(C\)curves are noisy and non\-monotone across the four Qwen3 sizes, yetVmax​\(C\)V\_\{\\max\}\(C\)on the same budget axis is smooth and capability\-ordered\. The fitness*value*at the optimum is therefore stable even when its precise*location*is not — the surface is searchable in the sense that what online search achieves does not depend sensitively on identifyingT∗T^\{\*\}exactly\.

### F\.3Boundary of the Regime

The CP row sign\-flips between 8B \(c=−0\.027c=\-0\.027,p=0\.49p=0\.49\) and 14B \(c=\+0\.007c=\+0\.007,p=0\.77p=0\.77\)\. Both estimates are statistically consistent with zero, so the flip is noise around thec≈0c\\approx 0limit rather than a regime change; it is the continuous\-\|c\|\|c\|framing of[Section˜5\.2](https://arxiv.org/html/2605.29268#S5.SS2)that is informative on CP, not the sign ofccitself\.

The lower rows of[Table˜1](https://arxiv.org/html/2605.29268#S5.T1)mark where the regularity stops applying\. Well\-fit small\-model rows \(CP 4BR2=0\.87R^\{2\}\{=\}0\.87, CP 1\.7BR2=0\.72R^\{2\}\{=\}0\.72\) have\|c\|≤0\.024\|c\|\\leq 0\.024, exhibiting the same geometry in attenuated form\. Poorly\-fit rows \(CP LlamaR2=0\.33R^\{2\}\{=\}0\.33, MMD 1\.7BR2=0\.36R^\{2\}\{=\}0\.36\) never enter the sub\-ceiling regime: weak models do not produce mutations structured enough for the depth–breadth tradeoff to become active, the same capability bottleneck identified in[Section˜5\.1](https://arxiv.org/html/2605.29268#S5.SS1)\.

One nuance: the form\-mismatch cells are not strictly capability failures\.[Figure˜9](https://arxiv.org/html/2605.29268#A6.F9)plotsRC2R^\{2\}\_\{C\}againstVmaxV\_\{\\max\}across the eleven \(model, task\) cells we fit; the cells whereRC2<0\.7R^\{2\}\_\{C\}<0\.7areCP/Llama, HT/Llama, andMMD/1\.7B\. MMD/1\.7B is a classic capability\-floor failure \(Vmax=0\.748V\_\{\\max\}=0\.748, only marginally above best\-of\-NN\), but CP/Llama sits atVmax=0\.843V\_\{\\max\}=0\.843which is a non\-trivial absolute level and still fails the bilinear fit \(RC2=0\.33R^\{2\}\_\{C\}=0\.33\)\. The shared signature is model family, not raw fitness: Llama’s proposals lack the depth–breadth interaction structure the bilinear form detects even when Llama itself reaches competitive fitness\. The bilinear regularity is therefore best read as a property of the Qwen3 family’s mutation distribution, with the lower boundary of its domain being more model\-family\-specific than purely capability\-driven\. The Spearman correlations betweenVmaxV\_\{\\max\},RC2R^\{2\}\_\{C\}, and−log10⁡p\-\\log\_\{10\}pacross all cells are individually weak \(ρ≤0\.45\\rho\\leq 0\.45\), consistent with this: the three statistics do not collapse to one capability axis\.

![Refer to caption](https://arxiv.org/html/2605.29268v1/x9.png)Figure 9:Bilinear\-fit qualityRC2R^\{2\}\_\{C\}versus best fitnessVmaxV\_\{\\max\}\(atC=512C\{=\}512\) across the eleven \(model, task\) cells\. Qwen3 cells \(blue ramp\) cluster in the upper\-right; Llama\-3\.1\-8B cells \(purple, black outline\) sit below theR2=0\.7R^\{2\}=0\.7threshold even whenVmaxV\_\{\\max\}is high \(CP/Llama atVmax=0\.843V\_\{\\max\}=0\.843butRC2=0\.33R^\{2\}\_\{C\}=0\.33\)\. The bilinear regularity’s lower boundary is therefore not strictly a capability floor\.Table 7:Bilinear fitness\-gap model \([Equation˜2](https://arxiv.org/html/2605.29268#S5.E2)\) fit on sub\-ceiling cells of theC=512C=512sweeps\. Permutationpp\-values testc≠0c\\neq 0\(10,00010\{,\}000shuffles\)\.p∗⁣∗∗<0\.001\{\}^\{\*\*\*\}p<0\.001,p∗∗<0\.01\{\}^\{\*\*\}p<0\.01,p∗<0\.05\{\}^\{\*\}p<0\.05\.Top block \(main\-experiment models\):bilinear form fits cleanly across all three tasks \(R2∈\[0\.75,0\.92\]R^\{2\}\\in\[0\.75,0\.92\]\), with task\-dependentcc— CP has no significant interaction \(broad plateau\), while MMD and HT have significant negative interactions \(interior ridge\)\.Bottom block \(smaller\-model regime\):well\-fit rows \(R2≥0\.72R^\{2\}\\geq 0\.72\) all satisfy\|c\|≤0\.024\|c\|\\leq 0\.024, consistent with a flattened version of the same surface; weak\-model configurations \(HT Llama, MMD 1\.7B, CP Llama\) fail to enter the sub\-ceiling regime and the form does not describe them \(R2≤0\.36R^\{2\}\\leq 0\.36\); significance stars for these rows should be disregarded\.TaskModelβ0\\beta\_\{0\}aabbccp​\(c\)p\(c\)R2R^\{2\}CP8B\-0\.020\-0\.602\-0\.496−0\.027\-0\.0270\.490\.81CP14B\-0\.561\-0\.442\-0\.373\+0\.007\+0\.0070\.770\.79MMD8B\-0\.590\-0\.208\-0\.290−0\.106∗⁣∗∗\-0\.106^\{\*\*\*\}1\.2×10−111\.2\\\!\\times\\\!10^\{\-11\}0\.92MMD14B\-0\.641\-0\.342\-0\.238−0\.057∗⁣∗∗\-0\.057^\{\*\*\*\}5\.0×10−45\.0\\\!\\times\\\!10^\{\-4\}0\.87HT8B\+0\.042\-0\.154\-0\.081−0\.034∗∗\-0\.034^\{\*\*\}0\.0020\.75CP1\.7B\-0\.498\-0\.076\-0\.088\+0\.001\+0\.0010\.780\.72CP4B\-0\.524\-0\.190\-0\.157−0\.017∗\-0\.017^\{\*\}0\.0450\.87CPLlama\-0\.917\-0\.017\-0\.101−0\.010\-0\.0100\.530\.33†MMD1\.7B\-1\.131\-0\.035\-0\.030\+0\.000\+0\.0000\.980\.36†MMD4B\-0\.223\-0\.418\-0\.248\+0\.024\+0\.0240\.300\.73HTLlama\+0\.013\-0\.026\-0\.006−0\.009\-0\.0095\.0×10−45\.0\\\!\\times\\\!10^\{\-4\}0\.66†Table 8:Pairwise best\-fitness comparisons: each block adapts one parent sampling protocol \(*greedy protocol*with different breadthNNor*island protocols*including OpenEvolve, CodeEvolve, and ShinkaEvolve\) into a baseline self\-evolving process as described in[Algorithm˜1](https://arxiv.org/html/2605.29268#alg1)andBaSEas described in[Algorithm˜2](https://arxiv.org/html/2605.29268#alg2), whereBaSEincludes three bandit algorithms \(UCB, EXP3\.P, and Thompson sampling\)\. The best mean fitness score in each block isbolded\.Table 9:Minimum generation \(Gen\.\) and cumulative FLOPs \(×1015\\times 10^\{15\}\) required for≥90%\\geq 90\\%of replicates to reach thresholds \(≥τ\\geq\\tau\) with three LLM models\. Unreached thresholds are denoted as “—”\. The minimum generation and FLOPs for each threshold arebolded\.ModelTask𝝉\\boldsymbol\{\\tau\}BaselinesBaSEGreedyOpenEvol\.CodeEvol\.ShinkaRand\.UCBEXP3Thomps\.Gen\.FLOPsGen\.FLOPsGen\.FLOPsGen\.FLOPsGen\.FLOPsGen\.FLOPsGen\.FLOPsGen\.FLOPsQwen3\-8BCP0\.90136163\.88————4557\.903947\.371618\.19139102\.081618\.190\.95152182\.12————4557\.903947\.371618\.19139102\.081618\.190\.99359411\.93————7390\.15125155\.4085100\.79207254\.6585101\.220\.999——————————————336403\.30MMD0\.80212318\.46119181\.48——71112\.002631\.65812\.414052\.01812\.410\.90296436\.55——————113164\.3192117\.6395139\.93105143\.720\.95485656\.81——————124185\.1492117\.63129200\.66106145\.72HT0\.50——————201189\.289299\.986066\.90102108\.857681\.220\.70————————367380\.48125130\.55209213\.78103106\.79Qwen3\-14BCP0\.904650\.19————89\.416880\.336072\.665867\.442830\.280\.95186134\.13————3943\.71120142\.526072\.66122139\.764951\.430\.99——————146114\.73137163\.876072\.66137155\.09295284\.780\.999——————————————295284\.78MMD0\.80257289\.208092\.46——3440\.025954\.39119\.957570\.48119\.950\.90372406\.99146168\.19——165189\.18175171\.16146148\.08226229\.18166160\.660\.95418451\.65259291\.33——210239\.02226224\.90146148\.08328331\.98178171\.35LlamaCP0\.6026255\.30————305\.67182\.5550\.87121\.7850\.870\.8026255\.30——————182\.5550\.87121\.7850\.87MMD0\.6021558\.25——————41734\.7532118\.9932127\.6125720\.550\.7021558\.25——————45139\.1332419\.4432429\.2026020\.880\.80——————————37122\.7443552\.9732333\.490\.90——————————40038\.13————HT0\.60————————————————

## Appendix GDeferredBaSEExperiments

### G\.1Pairwise Fitness Comparisons

[Table˜8](https://arxiv.org/html/2605.29268#A6.T8)reports pairwise comparisons under the same LLM\-call budgetCC\. For each parent sampling protocol, we compare its original self\-evolving version with aBaSEvariant that uses the same prompt generator, i\.e\., the same parent sampling protocol with identical configuration, so as to focus on the effect of adaptive trajectory allocation\.

Overall,BaSEimproves the best fitness score in most settings, especially when the original protocol has substantial remaining headroom or high variance across trajectories\. Under the greedyT=512T\{=\}512protocol,BaSEimproves Qwen3\-8B MMD from0\.81970\.8197to0\.96030\.9603and HT from0\.64410\.6441to0\.81040\.8104, while also improving Llama CP from0\.70340\.7034to0\.95330\.9533and MMD from0\.63880\.6388to0\.79780\.7978\. Similar gains appear under greedyT=256T\{=\}256, where Qwen3\-8B HT increases from0\.67800\.6780to0\.87360\.8736, and Llama MMD increases from0\.51820\.5182to0\.87770\.8777\.

The improvement is smaller when the original protocol already saturates the task\. For example, on CP, several methods already achieve near\-ceiling fitness scores, leaving little room for further gains\. This is visible for Qwen3\-8B ShinkaEvolve on CP, where the original protocol reaches0\.99860\.9986andBaSEvariants remain at a similar level\. In contrast,BaSEyields clearer gains in harder or more unstable settings by allocating more computation to trajectories that appear more promising\. For instance, when using the CodeEvolve prompt generator,BaSEimproves Qwen3\-8B HT from0\.51680\.5168to0\.71640\.7164, Llama MMD from0\.23150\.2315to0\.49060\.4906, and Qwen3\-14B CP from0\.87680\.8768to0\.96860\.9686\.

In some cases, especially when the original breadth\-depth allocation is already strong, or when all candidate trajectories have limited potential,BaSEcan underperform the original protocol\. For example, Qwen3\-8B greedy withT=128,N=4T\{=\}128,N\{=\}4performs better thanBaSEon MMD and HT, and Llama HT remains difficult across several parent sampling protocols\. These results suggest thatBaSEis most useful when the candidate runs contain heterogeneous trajectory quality that can be exploited by adaptive allocation\.

### G\.2Full Threshold Comparison

[Table˜9](https://arxiv.org/html/2605.29268#A6.T9)reports the complete threshold\-reaching results across three backbone models\. Overall, the same pattern observed in the main table remains consistent:BaSEvariants, especially UCB and Thompson sampling, reach target fitness levels with fewer generations and lower cumulative FLOPs in most settings where the thresholds are attainable\. On Qwen3\-8B,BaSEdominates MMD and HT, with UCB/Thompson reaching MMD thresholds up toτ=0\.95\\tau=0\.95substantially earlier than greedy, while several evolution baselines fail to reach these targets\. On CP, ShinkaEvolve is competitive for intermediate thresholds, but Thompson is the only method that reaches the stringentτ=0\.999\\tau=0\.999threshold\. Similar trends hold for Qwen3\-14B:BaSE\-UCB is strongest on MMD and reaches CPτ=0\.99\\tau=0\.99earlier than all baselines, while Thompson is again the only method reaching CPτ=0\.999\\tau=0\.999\. For Llama, absolute performance is lower and many thresholds remain unreached, especially on HT; nevertheless,BaSEstill provides the most reliable threshold\-reaching behavior, achieving the best or lowest\-FLOP results on most reachable CP and MMD thresholds\. These results suggest that adaptive trajectory allocation improves sample efficiency across model scales, while its gains are most pronounced on tasks where evolutionary trajectories exhibit substantial variance and early allocation decisions matter\.

### G\.3Ablation on Bandit Arm\-Pool Size

To study how the number of parallel trajectories affects online budget allocation, we ablate the arm\-pool sizeK∈\{2,5,10,20,50\}K\\in\\\{2,5,10,20,50\\\}forBaSEunder the same fixed generation budget ofC=512C=512with configurationT=512,N=1T=512,N=1\. As shown in[Table˜10](https://arxiv.org/html/2605.29268#A7.T10), performance is generally strongest with a moderate number of arms rather than the largest pool\. This reflects a natural cross\-trajectory level breadth–depth trade\-off: increasingKKprovides more trajectory diversity, but also reduces the number of refinement steps that can be allocated to each trajectory within the fixed budget\. For Qwen3\-8B, MMD reaches its best score withK=5K=5–2020, while HT performs best with small or moderate arm pools, indicating that excessive breadth can leave promising trajectories under\-refined\. Similarly, Qwen3\-14B achieves strong CP performance across severalKKvalues under Thompson sampling, while MMD peaks atK=5K=5\. For Llama, moderate arm pools again perform best on CP and MMD, whereas all methods fail on HT, suggesting that allocation alone cannot compensate when the underlying model rarely produces viable improvements\.

Table 10:BaSErunning\-max fitness atT=512T=512versus bandit arm\-pool sizeK∈\{2,5,10,20,50\}K\\in\\\{2,5,10,20,50\\\}\. The largest mean within each \(Model, Task\) block isbolded\.

## Appendix HPrompts

This appendix lists, verbatim, the system prompts used by the three*\*Evolve*frameworks compared in the paper — OpenEvolve, CodeEvolve, and ShinkaEvolve — on the three tasks we study: Circle Packing \(CP,n=26n=26in the unit square\), Min/Max Distance \(MMD,n=16n=16in 2D\), and Heilbronn Triangle \(HT,n=11n=11in an equilateral triangle\)\. Each task is specified by a fixed objective, a benchmark constant from AlphaEvolve, and a constructor function with a fixed name and return signature; these are shared across all three frameworks\. What differs between frameworks is the*scaffold*around the task: the chat format, what context from the archive is surfaced to the LLM, and the output protocol the LLM is asked to follow \(full rewrite vs\. SEARCH/REPLACE edit, free\-form vs\. tagged response\)\. The subsections below give each framework’s prompts in turn\.

### H\.1OpenEvolve \(also used by Greedy andBaSE\)

Our Greedy baseline and the proposedBaSEallocator use the OpenEvolve prompts*verbatim*\(the same system message, user\-message template, and per\-task instructions reproduced below\) so that any fitness difference between Greedy,BaSE, and OpenEvolve reflects*allocation*of the LLM call budget rather than prompt engineering\. OpenEvolve composes each call as one system message and one user message\. We use the full\-rewrite mode with three top programs and two diverse programs from the archive surfaced in each call\. The user message wraps the per\-task system message below together with the current program, its fitness, MAP\-Elites feature coordinates, and the top and diverse program archive\.

`OpenEvolve: user\-message template OpenEvolve: CP system message OpenEvolve: MMD system message OpenEvolve: HT system message`

`H\.2 CodeEvolve CodeEvolve uses a multi\-turn chat format that replays the program’s parent lineage as alternating user and assistant turns\. The system message is assembled as: the per\-task system block below, followed by a computational\-budget block, followed by one of four task templates \(exploitation or exploration, each with or without inspirations\)\. All four templates require the LLM to emit changes in strict <<<<<<< SEARCH \.\.\. ======= \.\.\. \>\>\>\>\>\>\> REPLACE form inside designated edit regions; full rewrites are not supported\. CodeEvolve: exploitation task template \(no inspirations\) The exploration variant retitles the task as code exploration and diversification, asking for novel strategies and distinct algorithmic pathways rather than incremental refinement\. The with\-inspirations variants additionally require the LLM to analyse the supplied inspiration programs and synthesise their differences before producing the SEARCH/REPLACE block\. CodeEvolve: CP system block CodeEvolve: MMD system block CodeEvolve: HT system block H\.3 ShinkaEvolve ShinkaEvolve concatenates the per\-task system prompt below with one of five full\-rewrite format variants sampled uniformly per call \(default, different\-algorithm, context\-motivated, structural\-redesign, and parametric\-design\), and pairs it with an iteration message that carries the current program and its metrics\. ShinkaEvolve: full\-rewrite format suffix ShinkaEvolve: iteration message ShinkaEvolve: CP system prompt ShinkaEvolve: MMD system prompt ShinkaEvolve: HT system prompt Appendix I Use of AI Assistants We used large language models \(Anthropic’s Claude\) to assist with paper revising\. Appendix J Artifact Licenses We use the following artifacts under their respective licenses, consistent with their intended research use: • OpenEvolve \(Apache\-2\.0\): we use the example evaluators, initial programs, and prompt templates for Circle Packing, MinMaxDist, and Heilbronn Triangle verbatim\. • Qwen3 1\.7B / 4B / 8B / 14B \(Apache\-2\.0\): used as the mutation engine in inference mode only; no weights modified\. • Llama\-3\.1\-8B \(Llama 3\.1 Community License\): used as the mutation engine in inference mode only; no weights modified\. • vLLM \(Apache\-2\.0\): used as the inference server\. Appendix K Ethics Statement This work uses publicly available open\-weight LLMs \(Qwen3, Llama\-3\.1\) and public geometric\-optimization benchmarks\. It involves no human subjects, no personal data, and no deployment in safety\-critical settings\. We foresee no direct ethical risks\. Appendix L Potential Risks LLM\-generated programs may contain correctness or security flaws that fitness\-based evaluators do not catch, and adaptive allocation can concentrate compute on superficially promising trajectories\.`