EliteTable：用于可执行控制策略的自我改进程序搜索

# Self-Improving Program Searchfor Executable Control Policies
Source: [https://arxiv.org/html/2610.12536](https://arxiv.org/html/2610.12536)
## EliteTable: Self\-Improving Program Search for Executable Control Policies

Abstract—EliteTable is a self\-improving program\-search algorithm guided by large language models \(LLMs\) that uses a fixed\-capacity table of high\-scoring solutions to guide new proposals, edits of existing solutions, and remixes of several solutions\. Performance was evaluated across six Gymnasium control tasks, each with ten generations and fifty candidate slots per generation\. Each task was evaluated across ten independently initialized searches\. Candidate\-score medians rose between the first and final generations in all six tasks, and selected\-winner search\-score medians rose in five\. InCartPole\-v1, the selected winner reached the search\-score ceiling in the first generation of all ten searches\. All ten paired held\-out endpoints improved in four of the six tasks\.BipedalWalker\-v3andCarRacing\-v3improved without reaching their reporting targets\. Six of tenAcrobot\-v1searches declined on held\-out episodes, and the median paired change was negative\. The results document improvement during program search and the transfer of selected\-winner gains to new episodes in some tasks\. Comparisons and ablations are needed to establish EliteTable’s relative performance and the contributions of its components\.

## 1 Introduction

This work was motivated by the broader study of recursive self\-improvement and LLM\-guided program search: how these algorithms are structured, which tasks they address, how well their solutions generalize, and which mechanisms support improvement\.

The study examines one part of that motivation: whether a program\-search algorithm inspired by prior work can also show improvement through repeated evaluation and revision\.

In program search, one part of the improvement loop proposes a solution, often source code or an executable policy\. Another part, the evaluator, provides a score or feedback on how that solution performs on the task\.

Optimization by PROmpting \(OPRO\)illustrates this feedback pattern: a language model receives a task description and previously evaluated solutions, then uses that feedback to propose further solutions\[[1](https://arxiv.org/html/2610.12536#bib.bib1)\]\.

FunSearchmakes this feedback loop concrete for executable programs\. It uses an island\-based database of solutions, maintaining multiple populations to support diversity\. Its applications include cap\-set constructions and online bin\-packing heuristics\[[2](https://arxiv.org/html/2610.12536#bib.bib2)\]\.

AlphaEvolveextends evolutionary code modification to scientific and computational problems, using evaluator feedback to guide revisions\[[3](https://arxiv.org/html/2610.12536#bib.bib3)\]\. In these systems, evaluation gives program search an external measure of whether a proposed change is useful\.

These algorithms differ in what they retain and how they generate candidates\.ShinkaEvolvecombines parent sampling, code\-novelty rejection, and bandit\-based selection among language models\[[4](https://arxiv.org/html/2610.12536#bib.bib4)\]\.

Evolution of Heuristicsevolves descriptions of heuristic ideas together with their executable implementations, giving the language model both conceptual and code\-level information about the solutions it explores\[[5](https://arxiv.org/html/2610.12536#bib.bib5)\]\.

EliteTable draws on these approaches through a fixed\-capacity table of high\-scoring programs\. New candidates compete with retained solutions after each generation, while three operators generate fresh proposals, edits, and remixes\. The design aims to provide a simple feedback loop for repeated program improvement\.

Keeping incumbents eligible guarantees that the best recorded search score cannot decrease, but does not guarantee that newly generated solutions improve\. A search may retain elites based on similar heuristics and become confined to a local optimum\[[2](https://arxiv.org/html/2610.12536#bib.bib2)\]\. Selected solutions may also fail to generalize to new episodes, so held\-out evaluation is necessary\[[10](https://arxiv.org/html/2610.12536#bib.bib10)\]\.

The study asks two questions\. First, do in\-sample winner scores and candidate\-score distributions improve across generations? Second, do selected\-winner gains transfer to held\-out episodes? I examine these questions across six Gymnasium control tasks\[[7](https://arxiv.org/html/2610.12536#bib.bib7)\]\.

Records were kept for every candidate across the sixty searches\. Between generations one and ten, pooled candidate\-score medians increased in all six tasks and median selected\-winner search scores increased in five\.CartPole\-v1winners were already at the ceiling in generation one\.

InCarRacing\-v3, the candidate population showed a bimodal score distribution: a higher\-return concentration moved toward stronger scores across generations while a low\-return group persisted\.

Held\-out evaluation addressed the second question\. All ten paired endpoint changes were positive in four tasks\.Acrobot\-v1’s median paired held\-out change was negative\.BipedalWalker\-v3andCarRacing\-v3improved between the first and final evaluations, but neither reached its reporting target\.

Held\-out gains were not necessarily monotonic: intermediate performance sometimes declined\. Longer searches might produce further gains, but the tested budget does not establish whetherBipedalWalker\-v3orCarRacing\-v3would eventually reach their targets\.

## 2 EliteTable Program Search

Figure[1](https://arxiv.org/html/2610.12536#S2.F1)shows the search loop, summarized inAlgorithm[1](https://arxiv.org/html/2610.12536#alg1)\. The retained table supplies context for subsequent proposals and preserves previously discovered high\-scoring programs\.

Frozen elite tableSource, scores, descriptionsSelect operationand parent programsLanguage modelNew, edit, or remixValidate and repairAt most five repairsEvaluate candidatesTen search episodesRetain top tenCandidates \+ incumbentsArchive each generation’s search\-selected winner\.Evaluate held\-out episodes only after search finishes\.Figure 1:EliteTable’s evaluation–revision loop\. All candidate slots finish before the table is updated\. Held\-out scores do not enter the loop\.Table 1:Task suite, settings and descriptive reporting targets\. Other constructor settings follow the recorded environment profiles\. RGB denotes red, green, and blue image channels\.### 2\.1 Generation and repair

EliteTable uses three operators: new, edit, and remix\. Generation one contains only fresh proposals; the topKKevaluated solutions enter the elite table\. Subsequent generations use the allocation inTable[2](https://arxiv.org/html/2610.12536#S3.T2)\. Fresh proposals receive elite descriptions and scores, edits receive one parent’s source, and remixes combine information from up to three parents\. Parents are sampled uniformly from the table frozen at the start of the generation\.

Before evaluation, proposed solutions are checked for schema compliance, valid source edits, and parseable Python syntax using the abstract syntax tree \(AST\)\.

Solutions must satisfy the required policy interface and pass duplicate checks\. Duplicate detection compares AST structures rather than source text\.

Solutions are then passed to the evaluator\. Validation or execution failures, such as access to an undefined variable, return diagnostics to a repair loop so the model can revise the proposal\.

If the revised solution still fails, it returns to the repair loop, up to five repairs per candidate slot\.

Candidate timeouts are treated as failures, not zero\-reward episodes or successful truncations\. Infrastructure failures interrupt execution rather than enter the candidate\-repair loop\.

Algorithm 1EliteTable search1:Task, fixed LLM, search panel;

T=10T=10,

N=50N=50,

K=10K=10
2:

ℰ0←∅\\mathcal\{E\}\_\{0\}\\leftarrow\\emptyset
3:for

t=1,…,Tt=1,\\ldots,Tdo

4:Freeze parent table at

ℰt−1\\mathcal\{E\}\_\{t\-1\}
5:Allocate new/edit/remix slots \(Table[2](https://arxiv.org/html/2610.12536#S3.T2)\)

6:Apply empty/single\-elite fallbacks; shuffle slots

7:Sample parents uniformly; remix parents are distinct

8:

𝒞t←∅\\mathcal\{C\}\_\{t\}\\leftarrow\\emptyset
9:for allcandidate slotsdo

10:Generate program; validate, evaluate, and repair

11:Add successful evaluated program to

𝒞t\\mathcal\{C\}\_\{t\}
12:endfor

13:Wait for every slot to finish or be discarded

14:

ℰt←TopK⁡\(ℰt−1∪𝒞t\)\\mathcal\{E\}\_\{t\}\\leftarrow\\operatorname\{TopK\}\(\\mathcal\{E\}\_\{t\-1\}\\cup\\mathcal\{C\}\_\{t\}\)
15:Archive the search\-selected winner in

ℰt\\mathcal\{E\}\_\{t\}
16:endfor

17:Evaluate archived winners on held\-out episodes

### 2\.2 Elite selection

As briefly mentioned, EliteTable retains the highest\-scoring programs across completed generations\. For elite capacityKK, the update rule is

ℰt=TopK⁡\(ℰt−1∪𝒞t\)\.\\mathcal\{E\}\_\{t\}=\\operatorname\{TopK\}\\left\(\\mathcal\{E\}\_\{t\-1\}\\cup\\mathcal\{C\}\_\{t\}\\right\)\.\(1\)
Hereℰt\\mathcal\{E\}\_\{t\}is the retained elite set after generationtt,𝒞t\\mathcal\{C\}\_\{t\}contains valid new candidates from that generation, andTopK\\operatorname\{TopK\}selects up toKKprograms\. Candidates are ranked by descending mean search reward\.

For equal scores, edits take priority over remixes, followed by incumbent elites and then fresh proposals\. Any remaining ties are resolved by ascending organism identifier\.

This persistent elitist mechanism ensures that the highest recorded*search\-panel*reward cannot decrease between completed generations, assuming previously recorded evaluations remain unchanged\. This monotonicity does not imply monotonic held\-out performance\.

## 3 Experimental Design

### 3\.1 Study structure

Six control environments were selected from the Gymnasium reinforcement learning library\[[7](https://arxiv.org/html/2610.12536#bib.bib7)\]\. Their task profiles are listed inTable[1](https://arxiv.org/html/2610.12536#S2.T1)\.

The tasks were selected for their perceived differences in difficulty and their variety of observation and action spaces\.

Across the sixty searches, hyperparameters were chosen to balance computational cost against the opportunity for improvement\.

The subsequent operator split applies when at least two elites are available\. Held\-out episodes are evaluated per distinct generation winner within each search\.

Table 2:Shared search and evaluation settings\.
Each candidate slot includes its initial proposal and any repair attempts; it either produces an evaluated policy or is discarded\. Slot counts therefore differ from counts of valid programs, model requests, and completed episode evaluations\.

The study records the configuration and execution history of each search\. Initialization seeds control operator scheduling and parent sampling, but do not make hosted model responses deterministic\. Reproduction can therefore follow the recorded procedure without guaranteeing identical search trajectories\.

### 3\.2 Environments and policy interface

Generated solutions must implementSolution\(Policy\), a Python class with an asynchronous action method\. Policy objects receive the current observation, observation and action spaces, and task instructions\.

The policy interacts with the environment through an observe–act loop\. Its runtime interface does not expose reward values or environment internals\.

A policy may retain state within an episode, provided it satisfies the required interface\. That state must be reset at the start of each episode\.

Generation and repair prompts include task documentation, resolved environment settings, the observation/action contract, the objective, and execution constraints\. This documented task knowledge is distinct from the information exposed to a policy during execution\.

ForBipedalWalker\-v3, the prompts also include explicit observation indexing, normalization, motor ordering, and reward details\. Prompt construction is configurable by task\.

The study therefore evaluates program search with documented task knowledge, rather than learning solely from interaction\.

### 3\.3 Model configuration

I used the requested model routeopenai/gpt\-oss\-120b:nitrothrough OpenRouter for proposal generation and repair\.

Responses were capped at 16,384 tokens per request\.

The runner does not explicitly set temperature or top\-pp, so the study does not specify fixed values for those parameters\. Returned model, provider, and usage metadata are retained when available\. The requested route does not guarantee identical provider execution across requests\.

Because fresh proposals receive elite summaries, they cannot serve as a history\-independent sampling baseline\.

### 3\.4 Fitness and held\-out evaluation

For policyπ\\pi, its episode return under seedssis the undiscounted sum

R⁡\(π,s\)=∑t=0Ts−1rt,R\(\\pi,s\)=\\sum\_\{t=0\}^\{T\_\{s\}\-1\}r\_\{t\},\(2\)
whereTsT\_\{s\}is the number of steps before environment termination or the configured step cap\.

Candidate fitness is the arithmetic mean over the ten fixed search seedsS=\{0,…,9\}S=\\\{0,\\ldots,9\\\}:

F⁡\(π\)=110​∑s∈SR⁡\(π,s\)\.F\(\\pi\)=\\frac\{1\}\{10\}\\sum\_\{s\\in S\}R\(\\pi,s\)\.\(3\)
EliteTable ranks candidates by higher fitness scores\. Tasks formulated as minimization problems would require an appropriate score transformation\.

All candidate slots must finish evaluation or be discarded before the elite table is updated\. The topKKprograms are then selected from the new evaluated candidates and existing elites, using the tie\-breaking order described above\.

After search finishes, the fixed historical winner from every generation is evaluated on 100 held\-out episode seeds,

H=\{1000,…,1099\},H=\\\{1000,\\ldots,1099\\\},which are disjoint from both the search panel and the previously inspected pilot panel 100–199\.

Each distinct selected policy is evaluated once per held\-out seed within a search\. Repeated generation winners within that search reuse cached measurements\. Held\-out outcomes do not influence generation, repair, promotion, stopping, or winner selection\.

For environmentee, searchii, and generationgg, held\-out performance is

Ye,i,g=1100​∑s∈HR⁡\(πe,i,g∗,s\),Y\_\{e,i,g\}=\\frac\{1\}\{100\}\\sum\_\{s\\in H\}R\(\\pi^\{\*\}\_\{e,i,g\},s\),\(4\)
whereπe,i,g∗\\pi^\{\*\}\_\{e,i,g\}is the winner selected by search fitness\.

A valid held\-out mean requires finite returns on all 100 specified seeds and no recorded evaluation failures\. Partial means are excluded\. Missing evaluations remain missing; they are not replaced with zero or rerun\.

The paired held\-out change isΔe,i=Ye,i,10−Ye,i,1\\Delta\_\{e,i\}=Y\_\{e,i,10\}\-Y\_\{e,i,1\}\. I report the median of these ten within\-search changes for each task, rather than the difference between the first\- and final\-generation medians\.

Descriptive 95% percentile intervals use 10,000 bootstrap resamples of the ten whole\-search paired records, with replacement\. Each resample preserves the within\-search pairing and recomputes the median change\. These intervals describe variation conditional on the fixed episode panels and tested configuration; they do not establish robustness across new panels or models\.

As a retrospective exploratory analysis, I applied two\-sided exact paired sign tests\[[11](https://arxiv.org/html/2610.12536#bib.bib11)\]to the ten endpoint changes per task\. The null isPr⁡\(Δ\>0∣Δ≠0\)=1/2\\Pr\(\\Delta\>0\\mid\\Delta\\neq 0\)=1/2, not equality of mean returns\. Exact zero changes are excluded from sign counts\. Holm adjustment\[[12](https://arxiv.org/html/2610.12536#bib.bib12)\]covers all six tasks, retaining an all\-tied task asp=1p=1for correction only\. These tests assume independent searches and condition on the fixed episode panels and tested configuration\. They do not establish comparative superiority or causal operator effects\.

Figure 2:Candidate search\-return distributions across generations, pooled over ten searches per task\. Solid blue lines show pooled candidate medians; faint lines show medians within individual searches\. Violin widths show smoothed density, not operator membership or independent replication\.
### 3\.5 Execution and reproducibility

Sixty searches were completed: ten initialization seeds across six tasks\. Each began at generation zero with no existing solutions\. Evaluation ran on an AMD64 host using eight worker processes, with up to 25 concurrent model operations for generation and repair\. The configured deadlines were 45 seconds per episode, ten seconds per policy action, and 120 seconds per model\-operation attempt\.

The episode and action deadlines exclude Docker startup, queueing, and final cleanup\. Generated policies ran in Docker containers with networking disabled, a read\-only root filesystem, and resource limits\.

Core reproduction code and recorded evidence are available at[https://github\.com/StuartFarmer/elitetable](https://github.com/StuartFarmer/elitetable)\. The accompanying reproduction guide identifies the pinned revision and offline reconstruction commands\. The retrospective sign\-test extension and additional inputs and scripts for supplemental analyses accompany this manuscript locally; they are not included in that pinned public release\. Reconstructing recorded results and running new stochastic searches are distinct procedures\.

Figure 3:Search and held\-out returns over ten generations\. Dashed blue lines show median search return; solid orange lines show median held\-out return, with interquartile ranges \(IQRs\) shaded and individual search trajectories drawn faintly\. Dotted lines indicate the reporting threshold\. Each environment contains ten eligible searches\.MountainCar\-v0generation 6 has nine valid held\-out measurements; all other environment\-generation combinations have ten\. Shading represents between\-search dispersion rather than a confidence interval\.Table 3:Held\-out performance across ten searches per task\. First and Final are medians of the generation\-one and generation\-ten mean returns\. Change is the median paired change with a descriptive 95% bootstrap confidence interval \(CI\)\. Improved counts positive paired changes; Target counts final winners meeting the reporting threshold\. Exploratory two\-sided sign\-testpp\-values are Holm\-adjusted across all six tasks\. A dash denotes ten ties and no informative test\.

## 4 Experimental Results

### 4\.1 Completion and improvement during search

All sixty searches completed\. Held\-out measurements were available for 599 of 600 generation\-winner panels\. The missingMountainCar\-v0search\-seed\-8 panel was at generation six and did not affect any endpoint pair\.

Median selected\-winner search scores increased between the first and final generations in five environments\.CartPole\-v1winners were already at the score ceiling in generation one\.

Newly evaluated candidate medians rose between generations one and ten in all six tasks, includingCartPole\-v1\.Figure[2](https://arxiv.org/html/2610.12536#S3.F2)shows these population changes, extending the evidence beyond retention of strong initial winners\.

These distributions contain newly evaluated candidates, not retained elites or held\-out winners\. Each score averages ten search episodes; failed or discarded candidates without scores are excluded\. TheLunarLander\-v3display is restricted to returns between−1000\-1000and350350: 198 scores outside this range remain in the calculations\.

Candidate\-score distributions show bimodality in some tasks\. This motivated an exploratory analysis of whether fresh proposals, edits, and remixes contribute differently to the score mixture\. The operator analyses appear in Supplemental Analyses\.

### 4\.2 Performance on held\-out episodes

Table[3](https://arxiv.org/html/2610.12536#S3.T3)reports paired endpoints and target counts\. All ten searches improved in each of four environments:MountainCar\-v0,LunarLander\-v3,BipedalWalker\-v3, andCarRacing\-v3\.CartPole\-v1remained at the ceiling, while six of tenAcrobot\-v1searches declined\.BipedalWalker\-v3andCarRacing\-v3improved without a final winner reaching the reporting target\.

Each of the four tasks with ten positive changes has rawp=0\.00195p=0\.00195and Holm\-adjustedp=0\.01172p=0\.01172\. ForAcrobot\-v1, rawp=0\.75391p=0\.75391and adjustedp=1p=1, so the observed negative median does not establish a systematic decline\. The tenCartPole\-v1ties provide no informative sign test and do not establish equivalence\.

Figure[3](https://arxiv.org/html/2610.12536#S3.F3)shows how outcomes develop across generations\. Held\-out scores sometimes decline even when the endpoint improves\.Acrobot\-v1’s positive search gains coexist with a negative median paired held\-out change, consistent with possible selection\-panel specialization\. The study shows transfer of endpoint gains to new episodes in four tasks, but does not establish robustness across task families or rule out overfitting\. Broader evaluation and comparisons with other search and reinforcement\-learning methods are needed\.

### 4\.3 Candidate reliability and resource use

Of 30,000 candidate slots, 29,959 reached evaluated status \(99\.9%\) and 41 were discarded\. The logs contain 38,289 model calls, including 8,287 repair calls, or 21\.6% of all calls\. Repair calls amount to 27\.6% of the 30,000 initial proposal calls; because a slot may undergo multiple repairs, this is not the percentage of proposals needing repair\. The completion rate describes the combined procedure and does not isolate the repair loop’s contribution\.

## 5 Discussion and Conclusion

The rising candidate\-score medians show progress beyond retaining a single strong initial solution\. Newly generated populations also shift toward higher returns\. In four tasks, every final selected winner improved over its first\-generation counterpart on the fixed held\-out episode panel\. These endpoint gains do not by themselves rule out overfitting to the selection panel\.

Acrobot\-v1exposes a limitation: search scores rose while the median paired held\-out change was negative\. Further examination is needed to identify the cause\. Held\-out evaluation covers each generation’s selected winner, not every retained elite, so it does not describe the full elite population\. Search used ten episode seeds, compared with 100 held\-out seeds\. Future studies could test whether broader selection panels reduce specialization; the current design does not isolate that effect\.

BipedalWalker\-v3andCarRacing\-v3improved without reaching their reporting targets\. The tested budget does not establish whether additional generations would eventually solve these tasks\. Their gains should therefore be distinguished from task attainment\.

### 5\.1 Program search and neural\-policy training

EliteTable revises policy source code rather than updating neural\-policy weights through policy gradients\. Source edits can change numerical constants as well as program structure\. Policy\-gradient methods instead update parameterized policies, often neural networks, using estimates of the gradient of expected return with respect to policy parameters\[[8](https://arxiv.org/html/2610.12536#bib.bib8)\]\.

The selected tasks are drawn from Gymnasium’s standard reinforcement\-learning environments\[[7](https://arxiv.org/html/2610.12536#bib.bib7)\]\. Matched comparisons are needed to assess the relative strengths of program search and neural\-policy training on these tasks\.

Source revisions can alter control structures, numerical constants, state, and execution flow\. Programmatic reinforcement learning can also combine program search with guidance from a trained neural policy\[[9](https://arxiv.org/html/2610.12536#bib.bib9)\]\. These are different update mechanisms; the present study provides no matched comparison of their learning efficiency\.

### 5\.2 Limitations and next steps

Understanding EliteTable’s components requires ablations and comparisons with other systems\. The observed improvements cannot yet be attributed specifically to reuse, editing, remixing, retention, or repair\. Ten searches per environment quantify variation under fixed episode panels, but do not establish robustness across episode distributions, language models, prompts, or search configurations\. The tasks are well known, so the model may have encountered descriptions or potential solutions during training\.

Time\-based deadlines can produce different outcomes on different hardware\. Hosted\-provider variation can also change model outputs and search trajectories, even when the same model route is requested\.

The final solutions are Python programs available for inspection, but source availability alone does not establish human interpretability\.

I would prioritize budget\-matched comparisons with other program\-search algorithms, including a history\-independent generation baseline, followed by complete\-search ablations and comparisons across language models\. Longer budgets, new episode panels, and other task families would test the limits of the observations\. The present results provide development evidence for those designs rather than confirmatory evidence of comparative superiority\.

### 5\.3 Conclusion

EliteTable contributes to the study of self\-improving LLM\-assisted program search\. Across sixty searches, candidate populations improved on selection episodes, while held\-out evaluation distinguished transferable gains, saturation, and regression\. This work supports further investigation of repeated program improvement as part of the broader agenda toward recursive self\-improvement\. It does not test a system that modifies its own optimizer or language model\.

## References

- \[1\]Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V\. Le, Denny Zhou, and Xinyun Chen\.Large Language Models as Optimizers\.International Conference on Learning Representations, 2024\.[Paper](https://arxiv.org/abs/2309.03409v3)\.
- \[2\]Bernardino Romera\-Paredes et al\.Mathematical discoveries from program search with large language models\.Nature, 625:468–475, 2024\.[Article](https://doi.org/10.1038/s41586-023-06924-6)\.
- \[3\]Alexander Novikov et al\.AlphaEvolve: A coding agent for scientific and algorithmic discovery\.arXiv:2506\.13131, 2025\.[Paper](https://arxiv.org/abs/2506.13131v1)\.
- \[4\]Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin\.ShinkaEvolve: Towards Open\-Ended And Sample\-Efficient Program Evolution\.arXiv:2509\.19349, 2025\.[Paper](https://arxiv.org/abs/2509.19349v1)\.
- \[5\]Fei Liu et al\.Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model\.Proceedings of the 41st International Conference on Machine Learning, PMLR 235:32201–32223, 2024\.[Conference paper](https://proceedings.mlr.press/v235/liu24bs.html)\.
- \[6\]Jacky Liang et al\.Code as Policies: Language Model Programs for Embodied Control\.arXiv:2209\.07753, 2022; revised 2023\.[Paper](https://arxiv.org/abs/2209.07753v4)\.
- \[7\]Mark Towers et al\.Gymnasium: A Standard Interface for Reinforcement Learning Environments\.arXiv:2407\.17032, 2024; revised 2025\.[Paper](https://arxiv.org/abs/2407.17032v4)\.
- \[8\]Richard S\. Sutton, David A\. McAllester, Satinder P\. Singh, and Yishay Mansour\.Policy Gradient Methods for Reinforcement Learning with Function Approximation\.Advances in Neural Information Processing Systems 12, 1999\.[Conference paper](https://proceedings.nips.cc/paper_files/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html)\.
- \[9\]Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri\.Programmatically Interpretable Reinforcement Learning\.Proceedings of the 35th International Conference on Machine Learning, PMLR 80:5045–5054, 2018\.[Conference paper](https://proceedings.mlr.press/v80/verma18a.html)\.
- \[10\]Gavin C\. Cawley and Nicola L\. C\. Talbot\.On Over\-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation\.Journal of Machine Learning Research, 11:2079–2107, 2010\.[Article](https://www.jmlr.org/papers/v11/cawley10a.html)\.
- \[11\]National Institute of Standards and Technology\.Sign Test\.Dataplot Reference Manual, updated 2023\.[Technical reference](https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/signtest.htm)\.
- \[12\]Sture Holm\.A Simple Sequentially Rejective Multiple Test Procedure\.Scandinavian Journal of Statistics, 6\(2\):65–70, 1979\.[Article](https://www.jstor.org/stable/4615733)\.

## Supplemental Analyses

All analyses use the same sixty completed searches\. Core reproduction code is available in the public GitHub repository\. Prompts, policy sources, and additional analysis inputs accompany this manuscript locally; the reproduction guide identifies additions outside the pinned public release\.

## S1 Candidate populations during search

### S1\.1 Population and winner endpoints

I first compared candidate and selected\-winner scores in the first and final generations to distinguish progress in newly generated programs from retention of strong incumbents\. For each task, I pooled evaluated candidates across ten searches and calculated their median search score\. Separately, I calculated the median score of the ten selected generation winners\. Scores average returns on ten fixed search episodes; discarded candidates without scores are excluded\.

Table S1:Candidate and selected\-winner medians at generations one and ten\.
New\-candidate medians increased in all six environments\. Selected\-winner medians increased in five, asCartPole\-v1winners were already at the ceiling\. These results show progress in the programs being generated beyond retention of a strong early winner\.

Generation one contains only fresh proposals, whereas later generations mix fresh proposals, edits, and remixes\. The population changes include this composition change\. This analysis does not isolate an operator effect or establish held\-out improvement of the full candidate population\.

### S1\.2CarRacing\-v3distribution across generations

I also examined the fullCarRacing\-v3candidate\-score distribution to determine whether rising medians reflected a broad shift or the disappearance of weak candidates\. For each generation, I pooled evaluated scores across ten searches and estimated a Gaussian kernel density with a common bandwidth of 25 return units\. Multiplying each density by its evaluated\-candidate count gives a band whose area equals that count\.

The ten generations are stacked from bottom to top\. A colored band’s*thickness*gives that generation’s count density; its cumulative upper boundary is not the density\. All 4,992 evaluated scores are included, with 498–500 candidates per generation\. Eight discarded slots have no score\. Scores average ten selection episodes, and smoothing extends beyond the observed score range\.

Figure S1:CarRacing\-v3candidate distributions over ten generations\.
Higher\-return concentrations emerge and move toward stronger scores as generations advance\. Weak candidates persist, but the distribution shifts toward higher returns\. These are in\-sample changes under a changing operator mixture, not evidence of population\-wide held\-out gains\.

## S2 Held\-out outcomes and sensitivity

### S2\.1 Paired search endpoints

I evaluated each search’s first and final selected winner on the same 100 held\-out episodes\. Winners were selected exclusively by search scores; held\-out returns did not affect selection\. Pairing within each search makes improvement and regression visible without pooling reward scales across tasks\.

Hollow circles mark the first generation and filled points mark the final generation\. Rightward movement indicates improvement; dashed lines mark reporting targets\. Panel headings give the numbers of improved searches and final target\-reaching winners\. Each task retains its own reward scale\.

Figure S2:First and final held\-out performance for all sixty searches\.
All ten endpoint pairs improved inMountainCar\-v0,LunarLander\-v3,BipedalWalker\-v3, andCarRacing\-v3\.CartPole\-v1remained at the ceiling, while six of tenAcrobot\-v1searches declined\.BipedalWalker\-v3andCarRacing\-v3improved without a final winner reaching the reporting target\.

All sixty endpoint pairs are available\. The missingMountainCar\-v0search\-seed\-8 panel at generation six affects only the intermediate trajectory\.

### S2\.2 Sensitivity to individual searches

For each task, I computed the ten paired held\-out changes from generation one to generation ten\. I omitted one search at a time and recomputed the median of the remaining nine changes\. The minimum and maximum of these ten medians measure sensitivity to an individual search; they are not confidence intervals\.

CP, MC and AC denoteCartPole\-v1,MountainCar\-v0andAcrobot\-v1; LL, BW and CR denoteLunarLander\-v3,BipedalWalker\-v3andCarRacing\-v3\. The final rows give the full\-sample median and leave\-one\-search\-out \(LOO\) range\.

\(a\) Classic\-control tasks

\(b\) Box2D tasks

Table S2:Paired held\-out changes and leave\-one\-search\-out medians\.
Omitting any one search leaves the direction of every task’s median change unchanged\. This check concerns variation among the ten searches under common episode panels\. It does not establish robustness to new episode panels, prompts, or models\.

Endpoint gains also need not be monotonic\. At least one adjacent\-generation decline occurred in oneCartPole\-v1, fiveMountainCar\-v0, tenAcrobot\-v1, tenLunarLander\-v3, sevenBipedalWalker\-v3and nineCarRacing\-v3searches\.LunarLander\-v3’s median fell from 179\.38 at generation six to 168\.29 at generation seven before reaching 214\.04\.CarRacing\-v3fell from 683\.81 at generation seven to 658\.00 at generation eight before reaching 691\.81\.

## S3 Resource accounting

I aggregated model\-call logs, candidate repair counters, discarded slots, and recorded attempt durations across ten searches per task\. All sixty searches began at generation zero and finished in one recorded attempt; none resumed a previous search\.

For reward\-scale context, I also evaluated a random\-action policy on the 100\-episode held\-out panel and report the median of its run\-level mean returns\. This is not a budget\-matched program\-search baseline\. Calls, repairs, discarded slots \(Disc\.\), and hours are summed across searches\. Hours include reporting as well as search\.

Table S3:Random\-action returns and recorded resources by task\.
The logs contain 38,289 model calls, including 8,287 repair calls\. Repairs account for approximately 21\.6% of logged calls, so candidate\-slot counts understate model usage\. Of 30,000 slots, 29,959 \(99\.9%\) reached evaluated status and 41 were discarded\.

Recorded attempt durations total 56\.88 hours, including 29\.76 hours forCarRacing\-v3\. These are neither CPU\-hours nor elapsed wall time for the whole study\. Two logged calls lack usage records, so complete token and monetary totals remain unknown\.

## S4 Operator distributions and elite admission

### S4\.1CarRacing\-v3candidate distributions

I groupedCarRacing\-v3candidates by generation operator to examine how fresh proposals, edits, and remixes contributed to the score mixture\. Generations two, six, and ten represent early, middle, and final post\-initialization search, pooling ten searches at each generation\.

For each operator and generation, I estimated a Gaussian kernel density with bandwidth 25 return units and multiplied it by the evaluated\-candidate count\. Band thickness gives count density; band area gives that operator’s candidate count\. Shared axes and bandwidth support comparison\. Counts appear above each panel; the discarded slot at generation six is excluded\. Smoothing extends beyond observed returns\.

Figure S3:CarRacing\-v3score distributions by generation operator\.
All three operators contribute to the early low\-score concentration\. By generation ten, fresh proposals remain concentrated at low scores while edits and remixes supply most of the higher\-score population\. Unequal allocation, parent selection, and inherited information prevent attributing these differences solely to operator choice\.

### S4\.2 Admission to the elite table

I extended the analysis by measuring how often an evaluated candidate entered the elite table in its birth generation, pooling generations 2–10 across ten searches per task\. I grouped admissions by the operator used to generate the candidate\. Dividing admissions by each operator’s evaluated\-candidate count accounts for unequal allocations\. This measures entry into the table, not the duration of later retention\.

Points give admission percentages, with admitted and evaluated counts noted alongside\. Discarded slots are excluded from the rate denominators\.

Figure S4:Birth\-generation elite\-admission rates by task and operator\.
Only 55 of 5,394 evaluated fresh proposals entered the elite table, approximately 1%\. Fresh proposals had lower observed admission rates under the tested configuration\. They receive elite summaries but no parent source, so they are not a history\-independent sampling baseline\.

Incumbent competition and deterministic tie\-breaking also affect admission\.CartPole\-v1’s equal\-score priority for edits is especially visible\. The rates do not identify causal operator effects, and candidates are dependent observations within searches\.

### S4\.3 Complete operator counts and returns

I also summarized candidates by operator within each task over generations 2–10\. The table records slots, evaluated candidates \(Eval\.\), discarded slots \(Disc\.\), median search returns, and birth\-generation elite admissions \(Adm\.\)\. Each search return averages ten selection episodes\. Score medians exclude discarded slots; slot totals include them\.

Table S4:Operator counts, median search returns and elite admissions\.
Fresh proposals had lower median returns than both edits and remixes in every environment\. Admissions count elite membership at the candidate’s birth generation, not later retention\. These are descriptive associations: incumbent competition, parent selection, and tie\-breaking prevent attributing the differences solely to operator choice\.