Component-Aware Feedback for Self-Evolving Programs

# Component-Aware Feedback for Self-Evolving Programs
Source: [https://arxiv.org/html/2609.38639](https://arxiv.org/html/2609.38639)
Jinming NianAffiliation:Santa Clara UniversityEmail:[jnian@scu\.edu](mailto:)Yi FangAffiliation:Santa Clara UniversityEmail:[yfang@scu\.edu](mailto:)

###### Abstract

LLM\-guided evolutionary search can discover complex programs, but existing methods mostly only save candidate programs and fitness scores while discarding which component edits produced which fitness metric changes\. Existing methods force the mutator LLM to infer the effect of prior edits from cluttered histories, making program search slow and unstable\. This is especially true for locally servable LLMs to evolve multi\-component systems\. We introduce component\-aware feedback, which compares each evaluated program with its parent, identifies the components that changed, and logs them with the associated metric differences into an attribution memory that later mutations read\. The memory keeps each change in two reference frames, local against the parent it came from and global against the seed program, which shows both the immediate effect of a change and the cumulative progress made since the seed\. We study this on LLM reranking, a multi\-objective optimization problem where a multi\-stage pipeline must balance quality against serving cost\. Across twelveBrightdatasets, our method reaches the strongest baseline’s final quality after a median of one third of the search budget and ends 7\.2% higher in held\-out nDCG@10, and under a cost\-aware objective it finds pipelines that are on average more accurate while using 11% fewer tokens per query, showing component\-aware feedback to be a promising direction for more efficient self\-evolving systems\.

## 1Introduction

Evolutionary search guided by large language models \(LLMs\) improves algorithms by proposing changes to executable programs, evaluating the resulting programs, and using the outcomes to guide further proposals\. FunSearch and AlphaEvolve demonstrate the potential of this approach for mathematical and algorithmic discovery\([Romera\-Paredes et al\., 2024](https://arxiv.org/html/2609.38639#bib.bib8);[Novikov et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib9)\)\. Since then, a growing line of systems has improved the loop in many ways: which parent to mutate, which population to grow, and to break out of local minima\([Lange et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib6);[Cemri et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib1)\)\. The other half of the loop has received much less attention\. Every iteration ends with an evaluation, and evaluations are the only signal the search learns from\. As the evolved programs grow larger, one question becomes increasingly important:What information should each evaluation retain to best guide the mutator LLM in proposing the next change?Today the answer is mostly a program and its score\. The mutation prompt shows the parent, a few high scoring or diverse programs, their fitness, and the score changes of recent children\([Novikov et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib9);[Lange et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib6);[Cemri et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib1)\)\. For a program with several interacting components, the full pipeline code accompanied by an overall fitness score leaves substantial interpretation challenge to the mutator LLM\. To tell which earlier edits helped, the mutator must diff full programs implicitly, decide which of the changed functions and constants mattered, and connect them to whichever metrics moved\. It must do all of this from a context that grows longer and more cluttered as the search proceeds\. LLMs are known to use long contexts unevenly and to lose track of accumulated multi\-turn histories\([Liu et al\., 2024b](https://arxiv.org/html/2609.38639#bib.bib32);[Laban et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib33)\)\. This burden matters most for the open, locally servable models that make evolutionary search affordable, whereas AlphaEvolve, ShinkaEvolve, and AdaEvolve all report their results with frontier models\([Novikov et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib9);[Lange et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib6);[Cemri et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib1)\)\. Reflection\-based search and semantic\-delta memories show that representing earlier modifications helps\([Ye et al\., 2024](https://arxiv.org/html/2609.38639#bib.bib10);[Jiang et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib11)\), but they ask an LLM to describe what an edit did, which is both expensive and hard to trust\. The changes in component and the corresponding metric change can be programatically extracted\. We argue that the mutator is best guided by these component\-aware feedback\.

We propose component\-aware feedback for self\-evolving programs, which turns each evaluation into a set of such records\. After every evaluation, an*observe*step decomposes the parent and the child into named components, namely their functions and module\-level constants\. It identifies which components were added, removed, or modified, and associates this edit set with the vector of differences in every metric the evaluator reports\. The records accumulate in an*attribution memory*that the mutation prompt reads, and each record is kept in two reference frames\. When evolution is making steady progress, the*local*frame compares a child with its parent and tells the mutator whether its last step helped\. When search stagnates, the*global*frame compares with seed and shows what has been attempted and changed so far\. The records are built entirely from evaluations the search already performs, and we add them to AdaEvolve\([Cemri et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib1)\)without changing its search controller\. We study this approach on LLM reranking, which is an important problem on its own, and contains multiple components by nature\. Reranking decides what a downstream model reads in search and retrieval\-augmented systems\. It remains an active research area, spanning prompting and log\-likelihood methods\([Sun et al\., 2023](https://arxiv.org/html/2609.38639#bib.bib3);[Qin et al\., 2024](https://arxiv.org/html/2609.38639#bib.bib12);[Chen et al\., 2025b](https://arxiv.org/html/2609.38639#bib.bib13)\)and rerankers trained with reinforcement learning\([Zhuang et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib23);[Liu et al\., 2026b](https://arxiv.org/html/2609.38639#bib.bib24)\), many of which are evaluated on the reasoning\-intensiveBrightbenchmark\([SU et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib2)\)\. LLM reranker is a pipeline whose components constrain one another\. Retrieval sets the ceiling on achievable quality, and the reranking strategy decides how much of that ceiling is achieved, and at what computation cost\. Evolving LLM reranking is often a trade off problem between quality and cost, where a good edit of one component may negatively impact the other\. We therefore evolve the complete pipeline, and study a quality\-only objective and a cost\-aware one\.

Figure 1:Local and global component\-aware feedback\.A standard LLM\-guided search \(top left\) rebuilds its context from the database at every iteration and keeps nothing between iterations\. We add an observe step \(1\), which diffs a child against its parent, and an attribution memory, which records each edit and is read by the context step \(top right\)\. Local and global entries \(bottom\) differ only in their reference point\. A local entry \(2\) compares a child with its parent, and a global entry \(3\) compares it with the seed\.We evaluate on all twelveBrightdatasets, with one open\-weight 35B model with 3B active parameters serving as both mutator and reranker\([Qwen Team, 2026](https://arxiv.org/html/2609.38639#bib.bib5)\)\. With component\-aware feedback, the search is faster, more consistent, and finds better programs\. It raises held\-out nDCG@10 by7\.2%7\.2\\%over AdaEvolve and leads on nine out of twelve datasets\. On ten datasets, it reaches AdaEvolve’s final search score after a median of 31 out of 100 iterations, and its final scores vary less across runs\. Under the cost\-aware objective, it finds pipelines that are more accurate and use11%11\\%fewer tokens per query compared to AdaEvolve\. Its programs also transfer better\. When run unchanged on other domains, they retain90%90\\%of the score obtained by evolving on the target directly, against79%79\\%for AdaEvolve\. An ablation shows that the gains come from both attribution and from exposing more metrics\. On circle packing and signal processing, our method is comparable to existing evolve methods, and works better on tasks where objectives compete with each other\. These results suggest that when a program has many interacting components, telling the mutator which component changes moved which metrics is as important as deciding which programs to mutate\.

## 2Methodology

### 2\.1Problem definition

LLM\-guided evolutionary search can be fundamentally broken down into a simple loop\. A user initializes the search by providing a seed programp0p\_\{0\}and an evaluator with a fitness functionFF, and the system repeats five steps until a budget ofTTiterations is reached:

1. 1\.Selection:sample a parent programppfrom the databaseDD, together with a small set of inspiration programs\.
2. 2\.Context:assemble a prompt from the parent and inspiration programs\.
3. 3\.Mutation:generate a child programp′p^\{\\prime\}from a mutation modelMMgiven that prompt\.
4. 4\.Evaluation:run the evaluator on the child program, obtaining a fitness scoref′=F⁡\(p′\)f^\{\\prime\}=F\(p^\{\\prime\}\)\.
5. 5\.Update:addp′p^\{\\prime\}into the database and update the controller that guides future selection\.

The goal of this loop is to maximize the fitness functionF:𝒫→ℝF:\\mathcal\{P\}\\rightarrow\\mathbb\{R\}over the space of executable programs𝒫\\mathcal\{P\}by repeatedly generating and evaluating new programspip\_\{i\}over a fixed budget ofTTiterations\. Most systems maintain not one database butKKdatabasesD=\{D1,D2​…,DK\}D=\\\{D\_\{1\},D\_\{2\}\\dots,D\_\{K\}\\\}that evolve semi\-independently and periodically exchange programs, which keeps the design space from collapsing into a single lineage\. Much of the effort in designing these systems goes into how to control the search to most efficiently utilize the fixed budget \(e\.g\. which parent to sample, which database to spend the next iteration on, when to abandon a direction\)\.

### 2\.2Adaptive search control

Prior works controlled the search with hand\-designed parameters, fixing exploration rates, database counts and stagnation criteria ahead of a run and re\-tuning them per problem\. AdaEvolve\([Cemri et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib1)\)removes this friction by deriving those parameters from the search itself\. For each databasekkit maintains an accumulated improvement signal, built from the normalized fitness gainδt\(k\)\\delta^\{\(k\)\}\_\{t\}that iterationttproduced,

Gt\(k\)=ρ​Gt−1\(k\)\+\(1−ρ\)​\(δt\(k\)\)2G^\{\(k\)\}\_\{t\}\\;=\\;\\rho\\,G^\{\(k\)\}\_\{t\-1\}\\;\+\\;\(1\-\\rho\)\\big\(\\delta^\{\(k\)\}\_\{t\}\\big\)^\{2\}\(1\)
with decayρ∈\[0,1\)\\rho\\in\[0,1\)\. Because each value is built from its predecessor,Gt\(k\)G^\{\(k\)\}\_\{t\}summarizes how productive that database has recently been\. The search parameters then become functions of this signal rather than constants\.

Despite these improvements, the content of the mutation prompt is not a function of any such signal\. It is a fixed template of meta\-instructions containing the full source code of the parent and a few selected programs as inspiration, the fitness scores of each, and a list of score deltas for the parent’s earlier children\. The improvement signal is used only to decide which parent is sampled, and the inspiration programs are selected by fitness scores and by how textually different they are from that parent\. Everything in the template is therefore an information dump of source code and fitness scores, and none of it indicates the state the search is in the way the improvement signal does for the search parameters\. That signal describes how the search is doing rather than what it has tried, so it cannot attribute which changes produced which effects\. Thus in its current state, the mutation model is left to infer both the state of the search and the effect of every prior edit from a cluttered history\.

### 2\.3Component\-aware feedback

Component\-aware feedback removes this inference\. For every program the search evaluates, it records which components the mutation changed and how each metric responded, and carries that record forward into the prompts that follow\. The mutator can then see what has already been tried and what each attempt did, so that later mutations build on those outcomes instead of rediscovering them\. We add an*observe*step, which runs after each evaluation, and an*attribution memory*that the context step reads \(Figure[1](https://arxiv.org/html/2609.38639#S1.F1)\)\. The context is then no longer an undifferentiated dump of programs and scores, but rather a record of which changes were made and what each one produced\.

We decompose each program into named components, its functions and its module\-level constants\. A program is then a map from component names to source, wherep⁡\[n\]p\[n\]is the source code of a component namednnand∅\\emptysetifpphas no component with that name\. Observe diffs the parent’s map against the child’s to identify which components the mutation touched, and attributes the resulting change in every metric to that set,

𝒞t=\{n\|p⁡\[n\]≠p′​\[n\]\},Δt=μ⁡\(p′\)−μ⁡\(p\),\\mathcal\{C\}\_\{t\}\\;=\\;\\big\\\{\\,n\\;\\big\|\\;p\[n\]\\neq p^\{\\prime\}\[n\]\\,\\big\\\},\\qquad\\Delta\_\{t\}\\;=\\;\\mu\(p^\{\\prime\}\)\-\\mu\(p\),\(2\)
where𝒞t\\mathcal\{C\}\_\{t\}is the set of namesnnfor which the parent and child hold different source, andμ⁡\(⋅\)\\mu\(\\cdot\)is the vector of metrics the evaluator returns\. Three cases fall out of that single condition\. A name in the child but not the parent is a component the mutation created, a name in the parent but not the child is one it removed, and a name in both with different source is one it modified\. The pairmt=\(𝒞t,Δt\)m\_\{t\}=\(\\mathcal\{C\}\_\{t\},\\Delta\_\{t\}\)is an*edit*, and it is the unit the attribution memory stores\.

Edits accumulate in the attribution memory𝒜\\mathcal\{A\}, which the context step reads alongside the database,

promptt=C⁡\(p,Dt,𝒜t\),𝒜t=𝒜t−1∪\{mt\},\\text\{prompt\}\_\{t\}\\;=\\;C\\big\(p,\\,D\_\{t\},\\,\\mathcal\{A\}\_\{t\}\\big\),\\qquad\\mathcal\{A\}\_\{t\}\\;=\\;\\mathcal\{A\}\_\{t\-1\}\\cup\\\{m\_\{t\}\\\},\(3\)
so the context depends on the whole trajectory of the search rather than only the programs that happened to be surfaced inDtD\_\{t\}\. The context becomes a function of the search state, as the search parameters already were in Eq\.[1](https://arxiv.org/html/2609.38639#S2.E1)\. As with the improvement signal, one memory is kept per database and the two are merged whenever the databases exchange programs, so an edit made in one lineage becomes available to the other at migration rather than immediately\.

### 2\.4Local and global reference frames

An edit is only meaningful relative to a reference point\. Applying Eq\.[2](https://arxiv.org/html/2609.38639#S2.E2)against the parent and against the seed gives two views that together show where the search stands,

mtloc=\(𝒞tloc,Δtloc\),mtglob=\(𝒞tglob,Δtglob\),m^\{\\,\\text\{loc\}\}\_\{t\}\\;=\\;\\big\(\\,\\mathcal\{C\}^\{\\,\\text\{loc\}\}\_\{t\},\\;\\Delta^\{\\,\\text\{loc\}\}\_\{t\}\\,\\big\),\\qquad m^\{\\,\\text\{glob\}\}\_\{t\}\\;=\\;\\big\(\\,\\mathcal\{C\}^\{\\,\\text\{glob\}\}\_\{t\},\\;\\Delta^\{\\,\\text\{glob\}\}\_\{t\}\\,\\big\),\(4\)
where𝒞tloc=\{n∣p⁡\[n\]≠p′​\[n\]\}\\mathcal\{C\}^\{\\,\\text\{loc\}\}\_\{t\}=\\\{\\,n\\mid p\[n\]\\neq p^\{\\prime\}\[n\]\\,\\\}andΔtloc=μ⁡\(p′\)−μ⁡\(p\)\\Delta^\{\\,\\text\{loc\}\}\_\{t\}=\\mu\(p^\{\\prime\}\)\-\\mu\(p\)measure the child against its parent, and𝒞tglob=\{n∣p0​\[n\]≠p′​\[n\]\}\\mathcal\{C\}^\{\\,\\text\{glob\}\}\_\{t\}=\\\{\\,n\\mid p\_\{0\}\[n\]\\neq p^\{\\prime\}\[n\]\\,\\\}andΔtglob=μ⁡\(p′\)−μ⁡\(p0\)\\Delta^\{\\,\\text\{glob\}\}\_\{t\}=\\mu\(p^\{\\prime\}\)\-\\mu\(p\_\{0\}\)measure it against the seed\.

The local frame measures an edit against its parent and answers whether that change helped\. Local entries go into the mutation prompt as a sliding window of the ten most recent edits\.

The global frame replaces the parent with the seed, so it describes a whole lineage rather than a single step\. A program reached by a dozen mutations is compared directly againstp0p\_\{0\}, which says which components that entire chain has introduced or discarded and what the chain bought\. Global entries are built only when the search stagnates, for the prompt that proposes a new strategy, where the shared origin makes those chains comparable and gives a direction time to pay off\.

The two frames can disagree, and an edit that lost ground against a strong parent is negative inΔloc\\Delta^\{\\,\\text\{loc\}\}while its lineage is still ahead inΔglob\\Delta^\{\\,\\text\{glob\}\}, so the direction is worth continuing even though the last step was not\. Figure[1](https://arxiv.org/html/2609.38639#S1.F1)shows an example entry in each frame\. Appendix[A](https://arxiv.org/html/2609.38639#A1)shows full prompts with each frame\.

### 2\.5Application to LLM reranking

We study this on LLM reranking\. A programppmaps a queryqqto a rankingσ\\sigmaover a corpus𝒟\\mathcal\{D\}, retrieving its own candidates from𝒟\\mathcal\{D\}and then reordering them\. The fitness score is ranking quality averaged over a query setQQ,

p:\(q,𝒟\)⟼σ,F⁡\(p\)=1\|Q\|​∑q∈QnDCG​@​10​\(p⁡\(q,𝒟\)\)\.p:\(q,\\mathcal\{D\}\)\\longmapsto\\sigma,\\qquad F\(p\)\\;=\\;\\frac\{1\}\{\|Q\|\}\\sum\_\{q\\in Q\}\\text\{nDCG\}@10\\big\(p\(q,\\mathcal\{D\}\)\\big\)\.\(5\)
Beyond nDCG@10 the evaluator reports the recall of the retrieved set, the nDCG an oracle reranker would achieve on it, the fraction of that ceiling the program achieves, and the cost of a query in LLM calls and tokens\. Together these form the metric vectorμ\\muof Eq\.[2](https://arxiv.org/html/2609.38639#S2.E2)\.

A fitness score of nDCG@10 alone credits a program for any gain in ranking quality and never charges it for the LLM calls that produced the gain\. We therefore also optimize a cost\-aware objective that trades ranking quality against efficiency\. We measure efficiency in LLM tokens per query\. Every input or output token is one more the model must process or generate, so token usage is closely aligned with both the serving cost and the latency of a program\. It measures neither directly, but it is a good proxy for how efficient the programs our search proposes are\. We do not use wall\-clock time because it depends on serving conditions such as resource contention from other processes, which vary over a run and would score the same program differently at different points in it\. The objective combines a quality termaaand an efficiency termbb, each normalized against the seed program,

s=1\+min⁡\(a,b\)\+λ⁡\(a\+b\),a=nDCG−nDCG01−nDCG0,b=1−ττ0,s\\;=\\;1\+\\min\(a,b\)\+\\lambda\\,\(a\+b\),\\qquad a=\\frac\{\\text\{nDCG\}\-\\text\{nDCG\}\_\{0\}\}\{1\-\\text\{nDCG\}\_\{0\}\},\\qquad b=1\-\\frac\{\\tau\}\{\\tau\_\{0\}\},\(6\)
whereτ\\tauis tokens per query, subscript00denotes the seed, andλ=0\.05\\lambda=0\.05\. The quality termaais the gain in nDCG@10 over the seed as a fraction of the largest gain possible from it, and the efficiency termbbis the fraction of the seed’s tokens per query the program saves\. Both are zero at the seed and one at their ceiling, and the constant places the seed at a score of 1\. We clip each term to\[−1,1\]\[\-1,1\], since a program that uses far more tokens than the seed would otherwise have an unbounded efficiency term\. The min term directs the search toward whichever of quality and efficiency is weaker\. Without theλ\\lambdaterm, the search could spend its whole budget fixated on either quality or efficiency, when the goal is the improve both\. Theλ\\lambdaterm adds a small bonus for gains in both, so progress on either objective is rewarded\.

## 3Experiments

### 3\.1Experimental Setup

We evaluate onBright\([SU et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib2)\), twelve retrieval datasets spanning StackExchange domains, coding, and theorem\-based questions\. Each dataset is split 60/40 into train and test\. The search optimizes a fitness score evaluated on the train split, either nDCG@10 or the cost\-aware score of Eq\.[6](https://arxiv.org/html/2609.38639#S2.E6)\. We report held\-out test nDCG@10 of the final program, mean, and standard deviation over three runs per dataset\.

The seed program is BM25\([Robertson and Zaragoza, 2009](https://arxiv.org/html/2609.38639#bib.bib4)\)over the corpus followed by a sliding\-window listwise reranker\([Sun et al\., 2023](https://arxiv.org/html/2609.38639#bib.bib3)\)\. All methods share this seed, evaluator, budget ofT=100T=100iterations, model for both mutation and reranking,K=2K=2databases with migration every 10 iterations, and four context programs per prompt\. The mutator samples at temperature0\.70\.7with up to16,38416\{,\}384output tokens, and the reranking calls inside evolved programs decode greedily\. The main results use Qwen3\.6\-35B\-A3B\([Qwen Team, 2026](https://arxiv.org/html/2609.38639#bib.bib5)\)\. We compare against BM25, the seed program, and AdaEvolve\([Cemri et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib1)\)\. Appendix[B](https://arxiv.org/html/2609.38639#A2)shows the seed and examples of evolved programs under both objectives\.

### 3\.2Experimental Results

Ranking quality and cost\.Table[1](https://arxiv.org/html/2609.38639#S3.T1)reports held\-out nDCG@10 under the quality\-only objective\. With component\-aware feedback, the search returns programs that average0\.3460\.346against AdaEvolve’s0\.3230\.323, a relative gain of7\.2%7\.2\\%, and leads on nine of the twelve datasets\. Both searches far exceed the seed at0\.2570\.257and BM25 at0\.1400\.140\.

Under the quality\-only objective this improvement comes at a cost\. These programs use71\.871\.8k LLM tokens per query averaged over the twelve datasets against AdaEvolve’s53\.753\.7k\. A component that widens the candidate pool is credited with higher ranking quality and is never penalized for adding more LLM calls or tokens per query\. Thus, an efficient pipeline and an expensive one are indistinguishable under a quality\-only objective\. Component\-aware feedback finds the better\-ranking program at the expense of cost and latency because nothing in the objective promotes efficiency\. Exploring the trade\-off between ranking quality and efficiency therefore requires a fitness score that accounts for both\.

Table 1:Ranking performance onBright\.nDCG@10 on the train and held\-out test splits, split 60/40 per dataset\. Best is the best single run on test\.Δ\\Delta% is our relative gain over AdaEvolve on test\. Mean±\\pmstd over 3 runs\.BM25seed programAdaEvolveOursdatasettraintesttraintesttraintestbesttraintestbestΔ\\Delta%StackExchangebio\.0\.1840\.1670\.3720\.372±\\pm\.0060\.3620\.5720\.572±\\pm\.0270\.5810\.581±\\pm\.0440\.6190\.602\\mathbf\{0\.602\}±\\pm\.0050\.613\\mathbf\{0\.613\}±\\pm\.0210\.633\+5\.6\+5\.6earth\.0\.2380\.3630\.4040\.404±\\pm\.0120\.5090\.4920\.492±\\pm\.0160\.5590\.559±\\pm\.0140\.5750\.529\\mathbf\{0\.529\}±\\pm\.0200\.573\\mathbf\{0\.573\}±\\pm\.0350\.606\+2\.4\+2\.4econ\.0\.1800\.1180\.2990\.299±\\pm\.0090\.2010\.4070\.407±\\pm\.0240\.2530\.253±\\pm\.0300\.2780\.412\\mathbf\{0\.412\}±\\pm\.0180\.274\\mathbf\{0\.274\}±\\pm\.0230\.299\+8\.1\+8\.1psy\.0\.0890\.2280\.3070\.307±\\pm\.0060\.5010\.4340\.434±\\pm\.0230\.540\\mathbf\{0\.540\}±\\pm\.0350\.5630\.448\\mathbf\{0\.448\}±\\pm\.0080\.5330\.533±\\pm\.0660\.604−1\.3\-1\.3rob\.0\.1540\.0750\.3370\.337±\\pm\.0060\.2390\.3760\.376±\\pm\.0140\.2540\.254±\\pm\.0150\.2680\.404\\mathbf\{0\.404\}±\\pm\.0250\.275\\mathbf\{0\.275\}±\\pm\.0050\.281\+8\.4\+8\.4stack\.0\.1990\.1420\.3220\.322±\\pm\.0160\.2190\.4060\.406±\\pm\.0290\.2110\.211±\\pm\.0440\.2520\.424\\mathbf\{0\.424\}±\\pm\.0110\.219\\mathbf\{0\.219\}±\\pm\.0400\.260\+3\.9\+3\.9sus\.0\.1720\.1370\.2670\.267±\\pm\.0090\.3000\.3140\.314±\\pm\.0290\.3020\.302±\\pm\.0600\.3630\.353\\mathbf\{0\.353\}±\\pm\.0260\.321\\mathbf\{0\.321\}±\\pm\.0530\.377\+6\.3\+6\.3Codingleet\.0\.3030\.1960\.2890\.289±\\pm\.0040\.1900\.3600\.360±\\pm\.0160\.2070\.207±\\pm\.0140\.2200\.381\\mathbf\{0\.381\}±\\pm\.0310\.247\\mathbf\{0\.247\}±\\pm\.0170\.264\+19\.1\+19\.1pony0\.0520\.0470\.1650\.165±\\pm\.0110\.1500\.267\\mathbf\{0\.267\}±\\pm\.0110\.220\\mathbf\{0\.220\}±\\pm\.0320\.2540\.2480\.248±\\pm\.0030\.2040\.204±\\pm\.0010\.205−7\.0\-7\.0Theorem\-basedaops0\.0370\.0980\.0570\.057±\\pm\.0040\.1300\.081\\mathbf\{0\.081\}±\\pm\.0130\.119\\mathbf\{0\.119\}±\\pm\.0280\.1450\.0800\.080±\\pm\.0190\.0870\.087±\\pm\.0250\.115−26\.4\-26\.4theoQ\.0\.0860\.0710\.1290\.129±\\pm\.0060\.1240\.3140\.314±\\pm\.0600\.2860\.286±\\pm\.0480\.3250\.381\\mathbf\{0\.381\}±\\pm\.0370\.348\\mathbf\{0\.348\}±\\pm\.0540\.380\+21\.7\+21\.7theoT\.0\.0040\.0370\.0630\.063±\\pm\.0020\.1570\.4210\.421±\\pm\.1310\.3440\.344±\\pm\.1300\.4940\.536\\mathbf\{0\.536\}±\\pm\.0220\.459\\mathbf\{0\.459\}±\\pm\.0330\.496\+33\.5\+33\.5mean0\.1410\.1400\.2510\.2570\.3700\.3230\.3630\.4000\.3460\.377\+7\.2\+7\.2

Table 2:Cost\-aware search onBright\.Test nDCG@10 and LLM tokens per query of the final program\. Both searches optimize the cost\-aware objective rather than nDCG alone\.Δ\\Deltais the change in test nDCG@10 against Table[1](https://arxiv.org/html/2609.38639#S3.T1)\. Cheaper is the quality\-only program’s tokens per query divided by this program’s\. Mean±\\pmstd over 3 runs\.The cost\-aware objective of Eq\.[6](https://arxiv.org/html/2609.38639#S2.E6)scores ranking quality and token usage together\. Under it, both searches return programs six to seven times cheaper than under quality alone \(Table[2](https://arxiv.org/html/2609.38639#S3.T2)\)\. Component\-aware feedback leads on both terms, reaching0\.3050\.305nDCG@10 at12,67612\{,\}676tokens per query against AdaEvolve’s0\.2960\.296at14,24314\{,\}243, an11%11\\%saving at a higher score\.

Convergence speed\.If the mutator no longer has to infer the effect of each prior edit from a cluttered history, the search should reach a given score in fewer iterations\. Figures[2](https://arxiv.org/html/2609.38639#S3.F2)and[3](https://arxiv.org/html/2609.38639#S3.F3)plot the best\-so\-far train score against iteration, one panel per dataset\. Under the quality\-only objective our mean reaches AdaEvolve’s final score on ten of the twelve datasets, at a median of3131iterations out of100100and a mean of3232\. Under the cost\-aware objective it does so on eleven of twelve, at a median of4242and a mean of4848\. Our method’s curve also stays above AdaEvolve’s for most of the run on eleven of the twelve datasets under both objectives\.

Figure 2:Convergence onBright\.Best\-so\-far nDCG@10 on the train split, one panel per dataset\. Solid lines are the mean over 3 runs, faint lines the individual runs\. The marker gives the iteration at which our mean first reaches AdaEvolve’s final score\.Figure 3:Convergence under the cost\-aware objective\.Best\-so\-far objective value on the train split, one panel per dataset\. The seed program scores 1\. Solid lines are the mean over 3 runs, faint lines the individual runs\. The marker gives the iteration at which our mean first reaches AdaEvolve’s final value\.Both searches plateau well before the budget runs out, improving by a median of under1%1\\%over the final quarter under the quality\-only objective, so most of what a search will find it finds early\. Each iteration pairs a mutation call with an evaluation, and in most applications of evolutionary search the evaluation dominates the cost, whether it runs a pipeline over a benchmark, trains a model, or executes a simulation\. A search that reaches a given score in a third of the iterations therefore does so for roughly a third of the compute, and under a fixed compute budget spends more of that budget searching from better programs\.

Cross\-domain generalization\.To test whether the programs component\-aware feedback finds are specific to the dataset they were evolved on, we take the three programs each method optimized on biology under the quality\-only objective and run them unchanged on four other datasets\. The four datasets differ in how closely they resemble biology\. Earth science is a neighboring natural science, robotics shares biology’s StackExchange source, and leetcode and theoremqa come from the coding and theorem\-based sections of theBrightbenchmark\. Each is scored on the held\-out split\.

Component\-aware feedback finds programs that generalize better across datasets\. Both methods lose ranking quality when we applied their biology programs to a new dataset, but ours lose less on every dataset \(Table[3](https://arxiv.org/html/2609.38639#S3.T3)\) and keep90%90\\%of their in\-domain score on average compared to79%79\\%for AdaEvolve, even matching the in\-domain programs on robotics\.

Table 3:Cross\-domain generalization onBright\.Programs optimized on biology scored on other datasets’ held\-out splits \(nDCG@10\)\. Seed and optimized on that dataset are the unevolved program and each method’s search run on that dataset respectively, all from Table[1](https://arxiv.org/html/2609.38639#S3.T1)\. Retained is the transferred program’s score as a percentage of the same method’s score when optimized on that dataset\. Mean±\\pmstd over 3 runs\.Beyond retrieval\.Beyond retrieval, we evaluate component\-aware feedback on two mathematical optimization benchmarks \(Figure[4](https://arxiv.org/html/2609.38639#S3.F4)\)\. Circle packing places2626circles in a unit square to maximize the sum of their radii\. Signal processing evolves a filter that removes noise from a changing signal while tracking it closely, scored on a combination of several measures of smoothness, accuracy and noise removal\. Alongside AdaEvolve we compare against ShinkaEvolve\([Lange et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib6)\), another LLM\-driven evolutionary search, and CORAL\([Qu et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib7)\), a multi\-agent system of autonomous coding agents\. CORAL has no notion of an iteration, so we give it a budget equal to the average cost of one of our runs\.

Figure 4:Mathematical optimization\.Best\-so\-far objective against the fraction of the search budget spent, one panel per benchmark, higher is better\. Solid lines are the mean over 3 runs, faint lines the individual runs\.On circle packing, component\-aware feedback matches AdaEvolve,2\.6172\.617against2\.6192\.619, with the same small spread acorss runs, and both lead CORAL at2\.5222\.522and ShinkaEvolve at2\.3222\.322\. Circle packing has a single score, so attribution adds little to the score change AdaEvolve already reports\. On signal processing, component\-aware feedback is more consistent than AdaEvolve and CORAL\. Its three runs stay between0\.6620\.662and0\.7200\.720, while AdaEvolve and CORAL each have a run that drops to0\.5210\.521and0\.5680\.568, which gives component\-aware feedback the highest mean,0\.6980\.698versus0\.6730\.673for CORAL,0\.6510\.651for AdaEvolve and0\.5600\.560for ShinkaEvolve\. Unlike circle packing, signal processing has competing goals, since a filter that removes more noise also responds more slowly to real changes\. Attribution shows the mutator which edits moved which measure, which is what balancing these competing objectives requires, as with quality and cost in reranking\.

Ablations\.Component\-aware feedback has two parts, the evaluator’s decomposed metrics and their attribution to the components that changed\. We isolate each part to test its value on biology, leetcode, and theoremqa, one dataset from each section of theBrightbenchmark\. We compare against AdaEvolve and only evolving the reranking prompt\. Evolving the whole program raises test nDCG@10 from0\.2450\.245to0\.3580\.358\. Decomposed metrics raise it to0\.3820\.382and attribution to0\.4030\.403, so each part accounts for about half of our gain over AdaEvolve\. Attribution adds little on train,0\.4540\.454versus0\.4480\.448, but narrows the drop from train to test from0\.0660\.066to0\.0510\.051\.

Table 4:Component ablation onBright\.nDCG@10 on one dataset perBrightsection\. Prompt evolution evolves the reranking prompt, and the other rows evolve the program\. Mean±\\pmstd over 3 runs\.

## 4Related Work

LLM\-guided program evolution\.FunSearch and AlphaEvolve established the basic loop of LLM mutation, program evaluation, and a program database\([Romera\-Paredes et al\., 2024](https://arxiv.org/html/2609.38639#bib.bib8);[Novikov et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib9)\)\. EoH and ReEvo extended the loop to heuristic design\([Liu et al\., 2024a](https://arxiv.org/html/2609.38639#bib.bib14);[Ye et al\., 2024](https://arxiv.org/html/2609.38639#bib.bib10)\), and OpenEvolve provides an open implementation of AlphaEvolve\([Sharma, 2025](https://arxiv.org/html/2609.38639#bib.bib28)\)\. Most later work improves search control\. ShinkaEvolve does so through parent sampling, novelty rejection, and model selection\([Lange et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib6)\), AdaEvolve through adaptive exploration and budget allocation\([Cemri et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib1)\), and EvoX by evolving the search strategy itself\([Liu et al\., 2026a](https://arxiv.org/html/2609.38639#bib.bib15)\)\. CORAL and SwarmResearch instead hand the loop to autonomous coding agents\([Qu et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib7);[Virk et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib16)\)\. Our work is orthogonal to search control as we focus on the feedback provided to the mutator\.

Feedback for LLM optimizersOpenEvolve and AdaEvolve shows past solutions and their scores in the prompt\. ReEvo and ShinkaEvolve summarize past attempts in LLM\-written reflections\([Ye et al\., 2024](https://arxiv.org/html/2609.38639#bib.bib10);[Lange et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib6)\)\. For compound AI systems, MIPRO, TextGrad, Trace, and GEPA assign credit to individual modules using surrogate models, textual gradients, execution traces, and reflection\([Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2609.38639#bib.bib29);[Yuksekgonul et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib30);[Cheng et al\., 2024](https://arxiv.org/html/2609.38639#bib.bib31);[Agrawal et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib17)\)\. DeltaEvolve describes each parent to child change in text and pairs it with a qualitative signal of improvement\([Jiang et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib11)\)\. Component\-aware feedback instead reads directly from the source code diff and the evaluator, so we do not need LLM summaries, extra evaluations, or learned models\.

LLM reranking\.Zero\-shot LLM rerankers differ in how they elicit relevance\. Pointwise methods use the likelihood of relevance labels or of the query\([Nogueira et al\., 2020](https://arxiv.org/html/2609.38639#bib.bib18);[Zhuang et al\., 2024a](https://arxiv.org/html/2609.38639#bib.bib34);[Sachan et al\., 2022](https://arxiv.org/html/2609.38639#bib.bib19)\), or attention patterns\([Chen et al\., 2025a](https://arxiv.org/html/2609.38639#bib.bib35)\)\. Comparative methods use pairwise, setwise, listwise, or tournament prompting\([Qin et al\., 2024](https://arxiv.org/html/2609.38639#bib.bib12);[Zhuang et al\., 2024b](https://arxiv.org/html/2609.38639#bib.bib20);[Sun et al\., 2023](https://arxiv.org/html/2609.38639#bib.bib3);[Gangi Reddy et al\., 2024](https://arxiv.org/html/2609.38639#bib.bib21);[Chen et al\., 2025b](https://arxiv.org/html/2609.38639#bib.bib13)\)\. Another line of work trains rerankers, through distillation\([Pradeep et al\., 2023](https://arxiv.org/html/2609.38639#bib.bib22);[Weller et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib27)\)or reinforcement learning\([Zhuang et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib23);[Zhang et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib37);[Liu et al\., 2026b](https://arxiv.org/html/2609.38639#bib.bib24);[Cai et al\., 2026](https://arxiv.org/html/2609.38639#bib.bib25)\), and these rerankers are often evaluated onBright\([SU et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib2)\)\. JudgeRank and DIVER combine several reasoning or retrieval stages into one pipeline\([Niu et al\., 2024](https://arxiv.org/html/2609.38639#bib.bib36);[Sun et al\., 2025](https://arxiv.org/html/2609.38639#bib.bib26)\)\. Each of these methods is a hand\-designed point in a space of retrieval and reranking choices\. We instead search that space with a fixed, locally served LLM\. Because all methods we compare share one seed, one model, and one evaluation protocol, our results measure the search itself and are not meant to compete with publishedBrightnumbers\.

## 5Conclusion

Most work on LLM\-guided evolutionary search refines how the search is controlled\. We asked instead what each evaluation should retain to guide the mutator\. Our answer is component\-aware feedback, a record of which components each edit changed and their associated outcomes\. With it, a locally served model reaches the strongest baseline’s final score in a third of the iterations and evolves reranking pipelines that score higher and generalize better to new datasets\. More broadly, it helps the search better navigate trade\-offs between competing objectives, such as quality against cost in reranking\. Component\-aware feedback provides a practical way to make self\-evolving systems more efficient, complementary to improving the search itself\.

## References

- Agrawalet al\.\(2026\)L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang, C\. Potts, K\. Sen, A\. G\. Dimakis, I\. Stoica, D\. Klein, M\. Zaharia, and O\. KhattabGEPA: reflective prompt evolution can outperform reinforcement learning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=RQm2KQTM5r)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p2.1)\.
- Caiet al\.\(2026\)Y\. Cai, Y\. Zhang, D\. Long, M\. Li, P\. Xie, and W\. ZhengERank: fusing supervised fine\-tuning and reinforcement learning for effective and efficient text reranking\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 30121–30129\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i36.40261),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/40261)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Cemriet al\.\(2026\)M\. Cemri, S\. Agrawal, A\. Gupta, S\. Liu, A\. Cheng, Q\. Mang, A\. Naren, L\. E\. Erdogan, K\. Sen, M\. Zaharia, A\. Dimakis, and I\. StoicaAdaEvolve: adaptive llm driven zeroth\-order optimization\.External Links:2602\.20133,[Link](https://arxiv.org/abs/2602.20133)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p1.1),[§1](https://arxiv.org/html/2609.38639#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.38639#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2609.38639#S3.SS1.p2.1),[§4](https://arxiv.org/html/2609.38639#S4.p1.1)\.
- Chenet al\.\(2025a\)S\. Chen, B\. Jiménez Gutiérrez, and Y\. SuAttention in large language models yields efficient zero\-shot re\-rankers\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=yzloNYH3QN)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Chenet al\.\(2025b\)Y\. Chen, Q\. Liu, Y\. Zhang, W\. Sun, X\. Ma, W\. Yang, D\. Shi, J\. Mao, and D\. YinTourRank: utilizing large language models for documents ranking with a tournament\-inspired strategy\.InProceedings of the ACM Web Conference 2025,External Links:[Document](https://dx.doi.org/10.1145/3696410.3714863),[Link](https://arxiv.org/abs/2406.11678)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p2.1),[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Chenget al\.\(2024\)C\. Cheng, A\. Nie, and A\. SwaminathanTrace is the next AutoDiff: generative optimization with rich feedback, execution traces, and LLMs\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/83ba7056bce2c3c3c27e17397cf3e1f0-Abstract-Conference.html)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p2.1)\.
- Gangi Reddyet al\.\(2024\)R\. Gangi Reddy, J\. Doo, Y\. Xu, M\. A\. Sultan, D\. Swain, A\. Sil, and H\. JiFIRST: faster improved listwise reranking with single token decoding\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8642–8652\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.491),[Link](https://aclanthology.org/2024.emnlp-main.491/)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Jianget al\.\(2026\)J\. Jiang, T\. Ding, and Z\. ZhuDeltaEvolve: accelerating scientific discovery through momentum\-driven evolution\.External Links:2602\.02919,[Link](https://arxiv.org/abs/2602.02919)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p1.1),[§4](https://arxiv.org/html/2609.38639#S4.p2.1)\.
- Labanet al\.\(2026\)P\. Laban, H\. Hayashi, Y\. Zhou, and J\. NevilleLLMs get lost in multi\-turn conversation\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2505.06120)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p1.1)\.
- Langeet al\.\(2025\)R\. T\. Lange, Y\. Imajuku, and E\. CetinShinkaEvolve: towards open\-ended and sample\-efficient program evolution\.arXiv preprint arXiv:2509\.19349\.Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p1.1),[§3\.2](https://arxiv.org/html/2609.38639#S3.SS2.p8.1),[§4](https://arxiv.org/html/2609.38639#S4.p1.1),[§4](https://arxiv.org/html/2609.38639#S4.p2.1)\.
- Liuet al\.\(2024a\)F\. Liu, T\. Xialiang, M\. Yuan, X\. Lin, F\. Luo, Z\. Wang, Z\. Lu, and Q\. ZhangEvolution of heuristics: towards efficient automatic algorithm design using large language model\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 32201–32223\.External Links:[Link](https://proceedings.mlr.press/v235/liu24bs.html)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p1.1)\.
- Liuet al\.\(2024b\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p1.1)\.
- Liuet al\.\(2026a\)S\. Liu, S\. Agarwal, M\. Maheswaran, M\. Cemri, Z\. Li, Q\. Mang, A\. Naren, E\. Boneh, A\. Cheng, M\. Z\. Pan, A\. Du, K\. Keutzer, A\. Cheung, A\. G\. Dimakis, K\. Sen, M\. Zaharia, and I\. StoicaEvoX: meta\-evolution for automated discovery\.External Links:2602\.23413,[Link](https://arxiv.org/abs/2602.23413)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p1.1)\.
- Liuet al\.\(2026b\)W\. Liu, X\. Ma, W\. Sun, Y\. Zhu, Y\. Li, D\. Yin, and Z\. DouReasonRank: empowering passage ranking with strong reasoning ability\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 22007–22031\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1005),[Link](https://aclanthology.org/2026.acl-long.1005/)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p2.1),[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Niuet al\.\(2024\)T\. Niu, S\. Joty, Y\. Liu, C\. Xiong, Y\. Zhou, and S\. YavuzJudgeRank: leveraging large language models for reasoning\-intensive reranking\.External Links:2411\.00142,[Link](https://arxiv.org/abs/2411.00142)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Nogueiraet al\.\(2020\)R\. Nogueira, Z\. Jiang, R\. Pradeep, and J\. LinDocument ranking with a pretrained sequence\-to\-sequence model\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 708–718\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.63),[Link](https://aclanthology.org/2020.findings-emnlp.63/)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Novikovet al\.\(2025\)A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. R\. Ruiz, A\. Mehrabian, M\. P\. Kumar, A\. See, S\. Chaudhuri, G\. Holland, A\. Davies, S\. Nowozin, P\. Kohli, and M\. BalogAlphaEvolve: a coding agent for scientific and algorithmic discovery\.External Links:2506\.13131,[Link](https://arxiv.org/abs/2506.13131)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p1.1),[§4](https://arxiv.org/html/2609.38639#S4.p1.1)\.
- Opsahl\-Onget al\.\(2024\)K\. Opsahl\-Ong, M\. J\. Ryan, J\. Purtell, D\. Broman, C\. Potts, M\. Zaharia, and O\. KhattabOptimizing instructions and demonstrations for multi\-stage language model programs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 9340–9366\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.525),[Link](https://aclanthology.org/2024.emnlp-main.525/)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p2.1)\.
- Pradeepet al\.\(2023\)R\. Pradeep, S\. Sharifymoghaddam, and J\. LinRankZephyr: effective and robust zero\-shot listwise reranking is a breeze\!\.External Links:2312\.02724,[Link](https://arxiv.org/abs/2312.02724)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Qinet al\.\(2024\)Z\. Qin, R\. Jagerman, K\. Hui, H\. Zhuang, J\. Wu, L\. Yan, J\. Shen, T\. Liu, J\. Liu, D\. Metzler, X\. Wang, and M\. BenderskyLarge language models are effective text rankers with pairwise ranking prompting\.InFindings of the Association for Computational Linguistics: NAACL 2024,pp\. 1504–1518\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.97),[Link](https://aclanthology.org/2024.findings-naacl.97/)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p2.1),[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Quet al\.\(2026\)A\. Qu, H\. Zheng, Z\. Zhou, Y\. Yan, Y\. Tang, S\. Y\. Ong, F\. Hong, K\. Zhou, C\. Jiang, M\. Kong, J\. Zhu, X\. Jiang, S\. Li, C\. Wu, B\. K\. H\. Low, J\. Zhao, and P\. P\. LiangCORAL: towards autonomous multi\-agent evolution for open\-ended discovery\.InConference on Language Modeling \(COLM\),Cited by:[§3\.2](https://arxiv.org/html/2609.38639#S3.SS2.p8.1),[§4](https://arxiv.org/html/2609.38639#S4.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.6\-35B\-A3B: agentic coding power, now open to all\.External Links:[Link](https://qwen.ai/blog?id=qwen3.6-35b-a3b)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p3.1),[§3\.1](https://arxiv.org/html/2609.38639#S3.SS1.p2.1)\.
- Robertson and Zaragoza \(2009\)S\. Robertson and H\. ZaragozaThe probabilistic relevance framework: bm25 and beyond\.Found\. Trends Inf\. Retr\.3\(4\),pp\. 333–389\.External Links:ISSN 1554\-0669,[Link](https://doi.org/10.1561/1500000019),[Document](https://dx.doi.org/10.1561/1500000019)Cited by:[§3\.1](https://arxiv.org/html/2609.38639#S3.SS1.p2.1)\.
- Romera\-Paredeset al\.\(2024\)B\. Romera\-Paredes, M\. Barekatain, A\. Novikov, M\. Balog, M\. P\. Kumar, E\. Dupont, F\. J\. R\. Ruiz, J\. S\. Ellenberg, P\. Wang, O\. Fawzi, P\. Kohli, and A\. FawziMathematical discoveries from program search with large language models\.Nature625,pp\. 468–475\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06924-6),[Link](https://www.nature.com/articles/s41586-023-06924-6)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p1.1),[§4](https://arxiv.org/html/2609.38639#S4.p1.1)\.
- Sachanet al\.\(2022\)D\. Sachan, M\. Lewis, M\. Joshi, A\. Aghajanyan, W\. Yih, J\. Pineau, and L\. ZettlemoyerImproving passage retrieval with zero\-shot question generation\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 3781–3797\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.249),[Link](https://aclanthology.org/2022.emnlp-main.249/)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Sharma \(2025\)A\. SharmaOpenEvolve: an open\-source implementation of AlphaEvolve\.Note:[https://github\.com/codelion/openevolve](https://github.com/codelion/openevolve)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p1.1)\.
- SUet al\.\(2025\)H\. SU, H\. Yen, M\. Xia, W\. Shi, N\. Muennighoff, H\. Wang, L\. Haisu, Q\. Shi, Z\. S\. Siegel, M\. Tang, R\. Sun, J\. Yoon, S\. O\. Arik, D\. Chen, and T\. YuBRIGHT: a realistic and challenging benchmark for reasoning\-intensive retrieval\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ykuc5q381b)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p2.1),[§3\.1](https://arxiv.org/html/2609.38639#S3.SS1.p1.1),[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Sunet al\.\(2025\)D\. Sun, M\. Long, D\. Yang, J\. Wang, Y\. Luo, Y\. Shen, J\. Wang, H\. Zhou, C\. Guo, P\. Wei, J\. Wang, and J\. GuDIVER: a multi\-stage approach for reasoning\-intensive information retrieval\.External Links:2508\.07995,[Link](https://arxiv.org/abs/2508.07995)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Sunet al\.\(2023\)W\. Sun, L\. Yan, X\. Ma, S\. Wang, P\. Ren, Z\. Chen, D\. Yin, and Z\. RenIs ChatGPT good at search? investigating large language models as re\-ranking agents\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 14918–14937\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.923/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.923)Cited by:[§B\.1](https://arxiv.org/html/2609.38639#A2.SS1.p1.1),[§1](https://arxiv.org/html/2609.38639#S1.p2.1),[§3\.1](https://arxiv.org/html/2609.38639#S3.SS1.p2.1),[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Virket al\.\(2026\)Y\. Virk, Z\. Edds, C\. S\. Xia, and L\. ZhangSwarmResearch: orchestrating coding agents for open\-ended discovery\.External Links:2607\.02807,[Link](https://arxiv.org/abs/2607.02807)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p1.1)\.
- Welleret al\.\(2025\)O\. Weller, K\. Ricci, E\. Yang, A\. Yates, D\. Lawrie, and B\. Van DurmeRank1: test\-time compute for reranking in information retrieval\.InSecond Conference on Language Modeling,External Links:[Link](https://arxiv.org/abs/2502.18418)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Yeet al\.\(2024\)H\. Ye, J\. Wang, Z\. Cao, F\. Berto, C\. Hua, H\. Kim, J\. Park, and G\. SongReEvo: large language models as hyper\-heuristics with reflective evolution\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/4ced59d480e07d290b6f29fc8798f195-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p1.1),[§4](https://arxiv.org/html/2609.38639#S4.p1.1),[§4](https://arxiv.org/html/2609.38639#S4.p2.1)\.
- Yuksekgonulet al\.\(2025\)M\. Yuksekgonul, F\. Bianchi, J\. Boen, S\. Liu, P\. Lu, Z\. Huang, C\. Guestrin, and J\. ZouOptimizing generative AI by backpropagating language model feedback\.Nature639\(8055\),pp\. 609–616\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-08661-4)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p2.1)\.
- Zhanget al\.\(2025\)L\. Zhang, B\. Wang, X\. Qiu, S\. Reddy, and A\. AgrawalREARANK: reasoning re\-ranking agent via reinforcement learning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 2458–2471\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.125),[Link](https://aclanthology.org/2025.emnlp-main.125/)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Zhuanget al\.\(2024a\)H\. Zhuang, Z\. Qin, K\. Hui, J\. Wu, L\. Yan, X\. Wang, and M\. BenderskyBeyond yes and no: improving zero\-shot LLM rankers via scoring fine\-grained relevance labels\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),pp\. 358–370\.External Links:[Link](https://aclanthology.org/2024.naacl-short.31/)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Zhuanget al\.\(2026\)S\. Zhuang, X\. Ma, Z\. Yao, S\. Wang, B\. Koopman, J\. Lin, and G\. ZucconRank\-R1: enhancing reasoning in LLM\-based document rerankers via reinforcement learning\.InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval,External Links:[Document](https://dx.doi.org/10.1145/3805712.3809961),[Link](https://cs.uwaterloo.ca/~jimmylin/publications/Zhuang_etal_SIGIR2026_RankR1.pdf)Cited by:[§1](https://arxiv.org/html/2609.38639#S1.p2.1),[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.
- Zhuanget al\.\(2024b\)S\. Zhuang, H\. Zhuang, B\. Koopman, and G\. ZucconA setwise approach for effective and highly efficient zero\-shot ranking with large language models\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,External Links:[Document](https://dx.doi.org/10.1145/3626772.3657813),[Link](https://arxiv.org/abs/2310.09497)Cited by:[§4](https://arxiv.org/html/2609.38639#S4.p3.1)\.

## Appendix APrompts

The search makes two kinds of LLM calls\. Every iteration, a mutation call edits a parent program\. When the search stagnates, a strategy call proposes new directions for the mutation calls to implement\. Both searches use the same system prompt \(Prompt\) and the same exploration and exploitation blocks \(Promptsand\)\. All examples come from biology runs\.

### A\.1System Prompt

The system prompt states the task, the function the program must implement, and the tools the program can call\. Only the dataset name changes between datasets\.

Youareanexperttaskedwithiterativelyimprovingasolution\.

YourgoalistomaximizetheCOMBINEDSCOREwhileexploringdiverseapproaches\.

Thesystemmaintainsacollectionofdiversesolutions\-bothhighcombinedscoreANDdiversityarevaluable\.

Thetask:givenabiologysearchquery,returndocumentidsrankedbest\-first\.Scoringis

nDCG@10againsthiddengoldlabels\.Readinggoldlabelsorimportingtheevaluatoror

datasetisprohibited\.

CONTRACT\-\-\-theevaluatorcallsexactlythisfunction\(keepitsnameandsignature\):

rank\(query:str,tools\)\-\>list\[str\]\#doc\_ids,most\-\>leastrelevant

TOOLS\(passedinas\`tools\`;donotimportanythingtogetthese\):

tools\.bm25\(query\_text:str,k:int\)\-\>list\[str\]\#BM25top\-koverthecorpus;callablerepeatedly

tools\.doc\(doc\_id:str\)\-\>str\#fulldocumenttext

tools\.llm\(messages:list\[dict\],max\_tokens:int=1024\)\-\>str\#onechatcompletion;greedy,token\-capped

### A\.2Mutation prompt

Every iteration makes one mutation call\. Depending on whether the search is exploring or exploiting, the mutation prompt can include a exploration block \(Prompt\) or an exploitation block \(Prompt\)\.

\#\#PARENTSELECTIONCONTEXT

Thisparentwasselectedthroughdiversity\-drivensamplingtoexploredifferentregions\.

\#\#\#EXPLORATIONGUIDANCE

\-Consideralternativealgorithmicapproaches

\-Don'tbeconstrainedbytheparent'sapproach

\-Lookforfundamentallydifferentalgorithmsornoveltechniques

\-Balancecreativitywithcorrectness

Yourgoal:Discovernewapproachesthatmightoutperformcurrentsolutions\.

\#\#PARENTSELECTIONCONTEXT

Thisparentwasselectedfromthearchiveoftop\-performingprograms\.

\#\#\#OPTIMIZATIONGUIDANCE

\-Thissolutionworkswell,butmeaningfulimprovementsarestillpossible

\-YoumayrefinetheexistingapproachORintroducebetteralgorithms

\-Consider:algorithmicimprovements,betterdatastructures,efficientlibraries

\-Ensurecorrectnessismaintained

Yourgoal:Improveuponthissolution\.

Promptsandare examples of full mutation prompts from AdaEvolve and component\-aware feedback\.

\#CurrentSolutionInformation

\-MainMetrics:

\-combined\_score:0\.4569

Metrics:

\-ndcg\_at\_10:0\.4569

\-num\_queries:62

\-Focusareas:\-Considersimplifying\-solutionlengthexceeds500characters

\#ProgramGenerationHistory

\#\#PreviousAttempts

Nopreviousattemptsyet\.

\#\#OtherContextSolutions

Theseprogramsrepresentdiverseapproachesandcreativesolutionsthatmayberelevanttothecurrenttask:

\#\#\#Program1\(combined\_score:0\.3837\)

Scorebreakdown:\-ndcg\_at\_10:0\.3837\-num\_queries:62

\`\`\`python

\[\.\.\.36linesofprogramsource\.\.\.\]

\`\`\`

\#\#\#Program2\(combined\_score:0\.4389\)

Scorebreakdown:\-ndcg\_at\_10:0\.4389\-num\_queries:62

\`\`\`python

\[\.\.\.66linesofprogramsource\.\.\.\]

\`\`\`

\#\#\#Program3\(combined\_score:0\.4569\)

Scorebreakdown:\-ndcg\_at\_10:0\.4569\-num\_queries:62

\`\`\`python

\[\.\.\.84linesofprogramsource\.\.\.\]

\`\`\`

\#\#\#Program4\(combined\_score:0\.4569\)

Scorebreakdown:\-ndcg\_at\_10:0\.4569\-num\_queries:62

\`\`\`python

\[\.\.\.84linesofprogramsource\.\.\.\]

\`\`\`

\#CurrentSolution

\#\#PARENTSELECTIONCONTEXT

Thisparentwasselectedfromthearchiveoftop\-performingprograms\.

\#\#\#OPTIMIZATIONGUIDANCE

\-Thissolutionworkswell,butmeaningfulimprovementsarestillpossible

\-YoumayrefinetheexistingapproachORintroducebetteralgorithms

\-Consider:algorithmicimprovements,betterdatastructures,efficientlibraries

\-Ensurecorrectnessismaintained

Yourgoal:Improveuponthissolution\.

\#\#ProgramInformation

combined\_score:0\.4569

Scorebreakdown:

\-ndcg\_at\_10:0\.4569

\-num\_queries:62

\`\`\`python

\[\.\.\.84linesofprogramsource\.\.\.\]

\`\`\`

\#\#BREAKTHROUGHIDEA\-IMPLEMENTTHIS

Thesearchhasstagnatedglobally\.YouMUSTimplementthisbreakthroughidea:

\*\*IDEA:\*\*ExplicitRelevanceScoringwithLLM

\*\*HOWTOIMPLEMENT:\*\*

Replacethesliding\-windowlistwisererankerwithanexplicitrelevancescoringapproach\.Retrievetop50candidatesviaBM25andqueryexpansion,thenprompttheLLMtoassignanumericrelevancescore\(1\-10\)toeachdocumentindividually\.Parsethesescoresandusethesortedfunctiontoordercandidatesbyscoredescending,returningthetop10\.ThisdirectlyoptimizesfornDCG@10byfocusingtheLLM'sreasoningonabsoluterelevanceratherthanrelativeorderingofmanyitems,whichreducestokenconsumptionandeliminatesorderingdriftinlower\-rankedpositions\.Byscoringeachdocindependently,youavoidcontextwindowlimitationsandensurethemostrelevantdocumentsbubbletothetopbasedonexplicitevidence\.

\*\*TARGETMETRIC:\*\*nDCG@10precisionandtop\-10recall

\*\*CAUTIONS:\*\*Ensurethepromptstrictlyrequestsastructurednumericoutputtoallowreliableparsingwithregex\.HandlecaseswheretheLLMreturnsnon\-numericvaluesbydefaultingtoalowscore\.Keepthecandidatepoolsmall\(30\-50\)tostaywithintokenlimitswhilemaintainingsufficientrecall\.

\*\*APPROACHTYPE:\*\*sorted

\*\*CRITICAL:\*\*

\-YouMUSTimplementthebreakthroughidea

\-Ensuretheparadigmisactuallyusedinyourimplementation\(notjustmentionedincomments\)

\-Correctnessisessential\-yourimplementationmustbecorrectandfunctional

\-Verifyoutputformatmatchesevaluatorrequirements

\-Makepurposefulchangesthatimplementtheidea

\-Testyourimplementationlogiccarefully

\#\#PREVIOUSATTEMPTSONTHISPARENT

Summary:0improved,0unchanged,1regressed

1\.0\.4569\-\>0\.2710\(\-0\.1859\)\[REGRESSED\]

Avoidrepeatingapproachesthatdidn'twork\.

\#Task

SuggestimprovementstotheprogramthatwillimproveitsCOMBINED\_SCORE\.

Thesystemmaintainsdiversityacrossthesedimensions:score,complexity\.

Differentsolutionswithsimilarcombined\_scorebutdifferentfeaturesarevaluable\.

YouMUSTusetheexactSEARCH/REPLACEdiffformatshownbelowtoindicatechanges:

<<<<<<<SEARCH

\#Originalcodetofindandreplace\(mustmatchexactly\)

=======

\#Newreplacementcode

\>\>\>\>\>\>\>REPLACE

Exampleofvaliddiffformat:

<<<<<<<SEARCH

foriinrange\(m\):

forjinrange\(p\):

forkinrange\(n\):

C\[i,j\]\+=A\[i,k\]\*B\[k,j\]

=======

\#Reorderloopsforbettermemoryaccesspattern

foriinrange\(m\):

forkinrange\(n\):

forjinrange\(p\):

C\[i,j\]\+=A\[i,k\]\*B\[k,j\]

\>\>\>\>\>\>\>REPLACE

\*\*CRITICAL\*\*:Youcansuggestmultiplechanges\.EachSEARCHsectionmustEXACTLYmatchcodein"\#CurrentSolution"\-copyitcharacter\-for\-character,preservingallwhitespaceandindentation\.DoNOTparaphraseorreformat\.

Bethoughtfulaboutyourchangesandexplainyourreasoningthoroughly\.

Includeaconcisedocstringatthestartoffunctionsdescribingtheexactapproachtaken\.

IMPORTANT:Ifaninstructionheaderof"\#\#IMPORTANT:\.\.\."isgivenbelowthe"\#CurrentSolution",youMUSTfollowit\.Otherwise,

focusontargetedimprovementsoftheprogram\.

\-Timelimit:Programsshouldcompleteexecutionwithin1500seconds;otherwise,theywilltimeout\.

\#CurrentSolutionInformation

\-MainMetrics:

\-combined\_score:0\.3942

Metrics:

\-ndcg\_at\_10:0\.3942

\-num\_queries:62

\-retrieval\_recall:0\.4274

\-oracle\_ndcg:0\.4813

\-scoring\_realization:0\.8190

\-llm\_calls\_per\_query:18\.0000

\-llm\_tokens\_per\_query:53770\.3710

\-Focusareas:\-Considersimplifying\-solutionlengthexceeds500characters

\#ProgramGenerationHistory

\#\#Whatthissearchhasalreadytried

\#\#\#Bestsofar\(0\.3903\)

\-madeatit8:~\_parse\_ranking,~\_rerank\_window,~STRIDE10\-\>5,~WINDOW20\-\>15combined\_score0\.3801→\\rightarrow0\.3903\(Δ\\Delta\+0\.0103\)

llm\_calls\_per\_query9\.0000→\\rightarrow18\.0000llm\_tokens\_per\_query34707\.4032→\\rightarrow60074\.0968oracle\_ndcg0\.4813→\\rightarrow0\.4813retrieval\_recall0\.4274→\\rightarrow0\.4274scoring\_realization0\.7896→\\rightarrow0\.8110

itsparts:

\_parse\_ranking:

\[\.\.\.52linesofsource\.\.\.\]

\_rerank\_window:

\[\.\.\.29linesofsource\.\.\.\]

rank:

\[\.\.\.16linesofsource\.\.\.\]

\#\#\#Otherprogramsinthearchive\(2\)

\-program\(0\.3868\)

madeatit9:\+\_rerank\_window,~\_parse\_rankingcombined\_score0\.2781→\\rightarrow0\.3868\(Δ\\Delta\+0\.1087\)\(\+rewired\`rank\`\)

llm\_calls\_per\_query1\.0000→\\rightarrow18\.0000llm\_tokens\_per\_query4213\.5968→\\rightarrow53108\.7742oracle\_ndcg0\.3200→\\rightarrow0\.4813retrieval\_recall0\.2737→\\rightarrow0\.4274scoring\_realization0\.8692→\\rightarrow0\.8036

itsparts:

\_parse\_ranking:

\[\.\.\.19linesofsource\.\.\.\]

\_rerank\_window:

\[\.\.\.21linesofsource\.\.\.\]

rank:

\[\.\.\.13linesofsource\.\.\.\]

\-program\(0\.3801\)

itsparts:

\_parse\_ranking:

\[\.\.\.11linesofsource\.\.\.\]

\_rerank\_window:

\[\.\.\.24linesofsource\.\.\.\]

rank:

\[\.\.\.16linesofsource\.\.\.\]

\#\#\#Changes,andwhateachonemoved\(mostrecent10\)

\[\.\.\.7earlieredits\.\.\.\]

\-it10:~\_parse\_ranking,~\_rerank\_window,\+STRIDE=5,~WINDOW25\-\>15combined\_score0\.2943→\\rightarrow0\.3834\(Δ\\Delta\+0\.0891\)\(\+rewired\`rank\`\)

llm\_calls\_per\_query1\.0000→\\rightarrow18\.0000llm\_tokens\_per\_query4984\.4032→\\rightarrow60007\.6452oracle\_ndcg0\.4813→\\rightarrow0\.4813retrieval\_recall0\.4274→\\rightarrow0\.4274scoring\_realization0\.6114→\\rightarrow0\.7964

\_parse\_ranking:

ParseLLMrankingoutputinto0\-basedindices\.

Triesbracketformat,numberedlists,comma\-separated;fillsmissingwithoriginalorder\.

\_rerank\_window:

Reorderwindow\_idsbyrelevanceusingLLMlistwiseranking\.

Usesstructuredpromptwithreasoninginstruction\.Fallsbacktoinputorderonfailure\.

\-it11:\-STRIDE,~DEPTH100\-\>50,~WINDOW15\-\>20combined\_score0\.3834→\\rightarrow0\.2847\(Δ\\Delta\-0\.0987\)\(\+rewired\`rank\`\)

llm\_calls\_per\_query18\.0000→\\rightarrow1\.0000llm\_tokens\_per\_query60007\.6452→\\rightarrow4470\.6613oracle\_ndcg0\.4813→\\rightarrow0\.3996retrieval\_recall0\.4274→\\rightarrow0\.3505scoring\_realization0\.7964→\\rightarrow0\.7124

\-it15:~\_parse\_ranking,~\_rerank\_windowcombined\_score0\.3868→\\rightarrow0\.3779\(Δ\\Delta\-0\.0089\)\(\+rewired\`rank\`\)

llm\_calls\_per\_query18\.0000→\\rightarrow18\.0000llm\_tokens\_per\_query53108\.7742→\\rightarrow55624\.3226oracle\_ndcg0\.4813→\\rightarrow0\.4813retrieval\_recall0\.4274→\\rightarrow0\.4274scoring\_realization0\.8036→\\rightarrow0\.7852

\_parse\_ranking:

ParseLLMrankingoutputintoavalidpermutationof0\.\.k\-1\.

Prioritizesexplicitbracketformat\[a\]\>\[b\]\>\[c\],thenfallsbackto

numberedlists\.Alwaysreturnsavalidpermutationbyfillinggapswith

originalorder\.

\_rerank\_window:

Reorderwindow\_idsbyrelevanceusingLLMlistwiseranking\.

Usesasystemmessageforbetterinstructionfollowingandastructured

promptwithexplicitformatconstraints\.Fallsbacktoinputorderonfailure\.

\#CurrentSolution

\#\#PARENTSELECTIONCONTEXT

Thisparentwasselectedfromthearchiveoftop\-performingprograms\.

\#\#\#OPTIMIZATIONGUIDANCE

\-Thissolutionworkswell,butmeaningfulimprovementsarestillpossible

\-YoumayrefinetheexistingapproachORintroducebetteralgorithms

\-Consider:algorithmicimprovements,betterdatastructures,efficientlibraries

\-Ensurecorrectnessismaintained

Yourgoal:Improveuponthissolution\.

\#\#ProgramInformation

combined\_score:0\.3942

Scorebreakdown:

\-ndcg\_at\_10:0\.3942

\-num\_queries:62

\-retrieval\_recall:0\.4274

\-oracle\_ndcg:0\.4813

\-scoring\_realization:0\.8190

\-llm\_calls\_per\_query:18\.0000

\-llm\_tokens\_per\_query:53770\.3710

\`\`\`python

\[\.\.\.58linesofprogramsource\.\.\.\]

\`\`\`

\#\#BREAKTHROUGHIDEA\-IMPLEMENTTHIS

Thesearchhasstagnatedglobally\.YouMUSTimplementthisbreakthroughidea:

\*\*IDEA:\*\*ReciprocalRankFusion\(RRF\)forMulti\-QueryRetrieval

\*\*HOWTOIMPLEMENT:\*\*

Biologyqueriesoftenmissrelevantdocumentsduetoterminologygaps\.UsetheLLMtogenerate3\-5semanticallyrelatedqueries\.Runtools\.bm25fortheoriginalandeachexpandedquery\.MergeallcandidatepoolsusingReciprocalRankFusion\(RRF\),scoringeachdocumentasthesumof1/\(k\+rank\)acrossallqueries\(withk=60\)\.Thissinglefusionstepcaptureslexicalandsemanticdiversityinonepass,directlyboostingretrievalrecallandoracleNDCGwithoutcomplexpipelines\.

\*\*TARGETMETRIC:\*\*retrieval\_recall,oracle\_ndcg

\*\*CAUTIONS:\*\*Keepexpandedqueriesconcisetostaywithintokenlimits\.UseafixedRRFconstant\.Ensurestrictdeduplicationwhilepreservingrankpositions\.HandlecaseswhereBM25returnsfewerthankresultsgracefully\.

\*\*APPROACHTYPE:\*\*custom\_rrf\_fusion

\*\*CRITICAL:\*\*

\-YouMUSTimplementthebreakthroughidea

\-Ensuretheparadigmisactuallyusedinyourimplementation\(notjustmentionedincomments\)

\-Correctnessisessential\-yourimplementationmustbecorrectandfunctional

\-Verifyoutputformatmatchesevaluatorrequirements

\-Makepurposefulchangesthatimplementtheidea

\-Testyourimplementationlogiccarefully

\#Task

\[\.\.\.identicaltotheTasksectionofPrompt4\.\.\.\]

### A\.3Strategy prompt

When the search stagnates, a strategy call proposes three new strategies\. Later mutation calls implement them on the best program\. AdaEvolve’s strategy prompt shows the best program and a list of strategies already tried\. Component\-aware feedback adds the seed as a reference point\.

\#\#ProblemObjective

Youareanexperttaskedwithiterativelyimprovingasolution\.

YourgoalistomaximizetheCOMBINEDSCOREwhileexploringdiverseapproaches\.

Thesystemmaintainsacollectionofdiversesolutions\-bothhighcombinedscoreANDdiversityarevaluable\.

Thetask:givenabiologysearchquery,returndocumentidsrankedbest\-first\.Scoringis

nDCG@10againsthiddengoldlabels\.Readinggoldlabelsorimportingtheevaluatoror

datasetisprohibited\.

CONTRACT\-\-\-theevaluatorcallsexactlythisfunction\(keepitsnameandsignature\):

rank\(query:str,tools\)\-\>list\[str\]\#doc\_ids,most\-\>leastrelevant

TOOLS\(passedinas\`tools\`;donotimportanythingtogetthese\):

tools\.bm25\(query\_text:str,k:int\)\-\>list\[str\]\#BM25top\-koverthecorpus;callablerepeatedly

tools\.doc\(doc\_id:str\)\-\>str\#fulldocumenttext

tools\.llm\(messages:list\[dict\],max\_tokens:int=1024\)\-\>str\#onechatcompletion;greedy,token\-capped

\#\#OptimizationTargets

Optimizetheprimaryscalarscoredefinedbytheevaluator\.

\#\#EvaluatorCode\(showshowsolutionsarescored\)

\`\`\`python

\[\.\.\.evaluatorsource\.\.\.\]

\`\`\`

\#\#CurrentBestProgram\(score:0\.483734\)

\`\`\`python

\[\.\.\.90linesofprogramsource\.\.\.\]

\`\`\`

\*\*CRITICAL:\*\*Analyzethecurrentprogramfirst\.Whatalgorithmdoesituse?

Whatareitsstrengthsandweaknesses?Howcanyouimproveuponit?

\*\*CRITICAL:ANALYZETHECURRENTPROGRAMFIRST\*\*

Beforesuggestingnewideas,carefullyanalyzetheCurrentProgramabove:

\-Whatalgorithm/approachdoesituse?\(Thisiswhat'sWORKING\-score0\.483734\)

\-Whatareitsstrengths?\(Whydoesitachievethisscore?\)

\-Whatareitsweaknesses?\(Whatlimitsfurtherimprovement?\)

\-Howcanyouimproveit?\(Howtobeatit?\)

\*\*IMPORTANT:\*\*TheprogramaboveistheCURRENTprogramthatneedstobeimproved\.Startbyunderstandingwhatworks,thensuggestbreakthroughideasthatbuildonorimproveit\.

\#\#AnalysisFramework\-CompleteBeforeGeneratingIdeas

\*\*STEP0:UnderstandtheTASK\(MOSTIMPORTANT\-DOTHISFIRST\)\*\*

\-Whatistheproblemaskingyoutodo?

\-Whatisthegoalorobjective?\(maximize,minimize,optimize\)

\-Whataretheinputsandoutputs?

\-Whatneedstobeimproved?\(variables/decisionsthataffectthegoal\)

\-Whatconstraintsexist?

\*\*STEP1:AnalyzetheEvaluatorCode\*\*

\-Howaresolutionsscored?

\-Whatmetricsarecomputed?

\-Whatcausesfailuresorpenalties?

\*\*STEP2:IdentifyMetrics\*\*

\-WhatistheprimarymetricorParetoobjectiveset?

\-Howisitcalculated?

\-Whatsecondarymetricsexist?

\-Ifvariance/stdispenalized,theprogramneedsconsistencyacrossscenarios

\*\*STEP3:IdentifyConstraints\*\*

\-Whatconditionsmustbesatisfied?

\-Whatvalidationhappens?

\-Whatcausesscorepenalties?

\*\*STEP4:IdentifyProblemStructure\*\*

\-Isprocessingsequentialorglobal?

\-Aredecisionvariablesdiscreteorcontinuous?

\-Whatdependenciesexistbetweendecisions?

\-\*\*CRITICAL:\*\*Whatdatadoesyourprogramreceivevswhattheevaluatoruses?

\-\*\*CRITICAL:\*\*Howaremetricscomputedacrosscomponents?Independentlythenaggregated,orjointly?

\*\*STEP5:DetermineAppropriateApproach\*\*

\-Matchapproachtoproblemstructure

\-Considerwhathasworkedvsfailedbefore

\-Identifypromisinglibrary/techniquecombinations

\*\*STEP6:IdentifyImprovementOpportunities\*\*

\-Whatwouldincreaseeachmetric?

\-Whatwouldsatisfyconstraintsbetter?

\-Whatfundamentallydifferentapproachescouldwork?

Currentbestscoreis0\.483734\.Yourideasmustimprovetheconfiguredoptimizationtargetsand,inmultiobjectivemode,explicitlyreasonaboutobjectivetrade\-offs\.

\#\#PreviouslyTriedIdeas\-CHECKTHISFIRST

\*\*CRITICAL:\*\*Reviewwhatwasalreadytried\.DoNOTsuggestideasthatuse

thesamelibraries,functions,orapproachesasFAILEDattempts\.

\-FAILED:sorted\-ExplicitRelevanceScoringwithLLM\(improvement:\+0\.0000\)

\-FAILED:set\.union\-Multi\-QueryEntityExpansionforRecall\(improvement:\+0\.0000\)

\-FAILED:itertools\.combinations\-PairwiseKnockoutTournamentforTop\-10Selection\(improvement:\+0\.0000\)

\-SUCCESS:scipy\.ndimage\.generic\_filter\-Multi\-stageretrievalfunnelwithexplicittop\-10optimization\(improvement:\+0\.0268\)

\-SUCCESS:numpy\.fft\.fft\-QueryunderstandingandadaptiveretrievalusingLLM\-generatedsearchterms\(improvement:\+0\.0268\)

\-SUCCESS:scipy\.cluster\.hierarchy\.linkage\-Iterativeretrievalwithgapanalysisandcomplementaryquerygeneration\(improvement:\+0\.0268\)

\*\*STRICTPROHIBITION:\*\*DoNOTkeepsuggestingapproachesthathavealreadyfailed\.

Ifanapproachfailed,understandWHYbeforesuggestingsimilartechniques\.

Prioritizeapproachesthatarefundamentallydifferentfromfailedattempts\.

\*\*LearningfromFailures\-UnderstandRootCauses:\*\*

Whenatechniquefailsbadly\(scoredecreasedsignificantly\),understandWHYbeforesuggestingalternatives:

\-\*\*Fundamentalmismatch:\*\*Wrongproblemtype\(e\.g\.,continuousoptimizerondiscreteproblem\)\-\>avoidthatentireclassofapproaches

\-\*\*Structuralmismatch:\*\*Wrongapproachforproblemstructure\(e\.g\.,linearproxyfornon\-linearobjective\)\-\>useapproachesthatmatchtheactualstructure

\-\*\*Implementationissues:\*\*Ifthesamelibraryfailedmultipletimesorverybadly\(\>10%decrease\),itlikelyindicatesafundamentalmismatch\-suggestadifferentclassofapproaches

\#\#TechniqueGuidance

\*\*Note:\*\*Standardscientificlibraries\(scipy,numpy,etc\.\)areavailable\.PyTorchandTensorFlowarenotavailable\.

\*\*ForContinuousOptimizationwithConstraints:\*\*

\-scipy\.optimize\.minimizewithconstrainthandling\(SLSQP,trust\-constr\)

\-Multipleinitialguessesforglobaloptimization

\-Geometricapproaches\(Voronoi,convexhull\)

\*\*ForDiscrete/CombinatorialProblems:\*\*

\-Greedyheuristicswithgoodordering

\-Localsearch\(swaps,moves\)

\-scipy\.optimize\.linear\_sum\_assignmentforassignmentproblems

\-scipy\.optimize\.linprogforlinearconstraints

\*\*ForGraph/NetworkProblems:\*\*

\-NetworkXalgorithms\(shortestpath,minspanningtree,flow\)

\-Spectralmethods\(eigenvalue\-basedordering\)

\*\*ForRepair/Reconstruction:\*\*

\-Heuristic\-baseddetectionandcorrection

\-Structuralconstraintexploitation

\-Averaging/interpolationforconsistency

\*\*ForRobustFiltering/NoiseReduction:\*\*

\-scipy\.signal\(medfilt,savgol\_filter,wiener\)fordirectfiltering

\-Usemethodsthathandleoutliersbetterthanmean\-based\(median,percentile\)

\-DoNOTusescipy\.optimize\.minimizetotunefilterparameters

\-Usefilteringfunctionsdirectly,notmulti\-stageoptimization

\*\*GeneralPrinciples:\*\*

\-Prefersingle\-functionlibrarycallsovermulti\-stagepipelines

\-Matchalgorithmtoproblemstructure

\-Simpleapproacheswithgoodheuristicsoftenbeatcomplexmethods

\#\#ANTI\-PATTERNS\-CriticalrulesaboutwhatNOTtodo

1\.\*\*DoNOTusemulti\-stageoptimization\*\*:DoNOTcallonefunctionthenoptimizeitsoutput\.Deterministicsetupcodefollowedbyasingleoptimizationcallisallowed\.

2\.\*\*DoNOTusescipy\.optimize\.minimizeforhyperparametertuning\*\*:Useminimizetosolvetheproblemdirectly,NOTtotuneparametersforanotherfunction\.

3\.\*\*DoNOTusescipy\.optimize\.minimizefordiscreteproblems\*\*:Continuousoptimizerscannothandlediscreteconstraintviolationsproperly\.

4\.\*\*EachideaMUSTbeasingle\-functionlibrarycall\*\*:DoNOTsuggestmulti\-stageprocessing\(e\.g\.,"callAthencallB"\)\.

\*\*AVOID:\*\*DEAP,geneticalgorithmlibraries,domain\-specificcomplexlibraries,customresearchalgorithms,oranylibraryrequiringadditional\`pipinstall\`

\*\*LearningfromSuccess:\*\*

Whenanapproachsucceeds,think:whatprinciplemadeitwork?Learnandthinkofbetterideas,don'tjustaddcomplexity\.Ifbreakthroughpatternsareknown,prioritizeapproachesthatmatchthem\.

\#\#DIVERSITYREQUIREMENTS

Beforegeneratingideas,explicitlythink:

\-Idea1:\[TypeA\-e\.g\.,algorithmicrefinementorlibrary\-basedapproach\]

\-Idea2:\[TypeB\-e\.g\.,structuralchangeorprocessingpattern\-DIFFERENTfromA\]

\-Idea3:\[TypeC\-e\.g\.,differenttechniqueoroptimizationmethod\-DIFFERENTfromAandB\]

\*\*Verify:\*\*AretheseDIFFERENTtypes?NOTvariationsofthesameapproach\.

Eachideamust:

\-UseDIFFERENTlibraries/techniquesthanfailedattempts

\-TargetDIFFERENTmetrics/aspectsfromtheevaluator

\-Beindependentlyimplementable

\-Preferclearimplementations\(different\!=morecomplex\)

\#\#\#BeSpecificandActionable

Notvague:"Tryoptimization"

Specific:"Usescipy\.optimize\.minimizewithSLSQPmethod"

\-Includeexactlibrarynames,functionnames,methods,parameters

\-Providestep\-by\-stepimplementationguide

\-Focusoncorelogicthatimplementstheideacorrectly

\-Handleedgecasesandavoiderrors/warnings

\-Foroptimization:usemultipleinitializations,appropriateiterationcountsandconvergencecriteria\(evaluationtimeout:\{self\.eval\_timeout\}s\)

\#\#OutputFormat

\*\*IMPORTANT:\*\*RespondwithaJSONobjectcontainingexactly3ideaobjectsunderthe"ideas"key\.

Donotincludecodepatchesordiffs\-\-\-describestrategiesinnaturallanguage\.

Generate3breakthroughideasofDIFFERENTtypes\.

EachideamustbeaJSONobjectwiththesefields:

\-"idea":Clear,directdescriptionwithlibrary/techniquename

\-"description":Detailedimplementationguide\(5\-10sentences\)

\-"what\_to\_optimize":Whatmetrics/areastofocuson

\-"cautions":Importantimplementationdetailstowatchfor

\-"approach\_type":Exact"library\.function"format\(e\.g\.,"scipy\.optimize\.minimize"\)

\*\*DiversityRequirement:\*\*EachideamustuseaDIFFERENTapproachtype\.

Donotgeneratevariationsofthesametechnique\.

ReturnONLYaJSONobjectinthisshape:\{"ideas":\[\.\.\.\]\}with3paradigmobjects\.Noothertext\.

Example:

\`\`\`json

\{

"ideas":\[

\{

"idea":"Usescipy\.optimize\.minimizewithSLSQP",

"description":"Applyscipy\.optimize\.minimizedirectlytooptimizeallvariablestogether\.\.\.",

"what\_to\_optimize":"primaryevaluatorscore",

"cautions":"Ensureconstraintsareproperlyformulated,usemultiplestartingpoints",

"approach\_type":"scipy\.optimize\.minimize"

\}

\]

\}

\`\`\`

\#\#ProblemObjective

Youareanexperttaskedwithiterativelyimprovingasolution\.

YourgoalistomaximizetheCOMBINEDSCOREwhileexploringdiverseapproaches\.

Thesystemmaintainsacollectionofdiversesolutions\-bothhighcombinedscoreANDdiversityarevaluable\.

Thetask:givenabiologysearchquery,returndocumentidsrankedbest\-first\.Scoringis

nDCG@10againsthiddengoldlabels\.Readinggoldlabelsorimportingtheevaluatoror

datasetisprohibited\.

CONTRACT\-\-\-theevaluatorcallsexactlythisfunction\(keepitsnameandsignature\):

rank\(query:str,tools\)\-\>list\[str\]\#doc\_ids,most\-\>leastrelevant

TOOLS\(passedinas\`tools\`;donotimportanythingtogetthese\):

tools\.bm25\(query\_text:str,k:int\)\-\>list\[str\]\#BM25top\-koverthecorpus;callablerepeatedly

tools\.doc\(doc\_id:str\)\-\>str\#fulldocumenttext

tools\.llm\(messages:list\[dict\],max\_tokens:int=1024\)\-\>str\#onechatcompletion;greedy,token\-capped

\#\#OptimizationTargets

Optimizetheprimaryscalarscoredefinedbytheevaluator\.

\#\#EvaluatorCode\(showshowsolutionsarescored\)

\`\`\`python

\[\.\.\.383linesofprogramsource\.\.\.\]

\`\`\`

\#\#TheSeedProgram,wherethesearchstarts\(score:0\.380053\)

\`\`\`python

\[\.\.\.67linesofprogramsource\.\.\.\]

\`\`\`

\*\*NOTE:\*\*ThisseedistheREFERENCEFRAME,nottheprogramtoimprove\.Every

numberinthesearchhistorybelowisstatedasadeltafromthisprogram\.

TheprogramyouareimprovingistheCurrentBestProgramshownnext\.

\#\#TheCurrentBestProgram\(score:0\.607106,foundatiteration20\)

\*\*Thisistheprogramyourideawillbeappliedto\.\*\*Itiswhattheseedabovebecameafter

thesearch;readitbeforeproposinganything,becauseanideaitalreadyimplementsisa

wastediteration\.

\`\`\`python

\[\.\.\.120linesofprogramsource\.\.\.\]

\`\`\`

\*\*CRITICAL:ANALYZETHECURRENTPROGRAMFIRST\*\*

Beforesuggestingnewideas,carefullyanalyzetheCurrentProgramabove:

\-Whatalgorithm/approachdoesituse?\(Thisiswhat'sWORKING\-score0\.607106\)

\-Whatareitsstrengths?\(Whydoesitachievethisscore?\)

\-Whatareitsweaknesses?\(Whatlimitsfurtherimprovement?\)

\-Howcanyouimproveit?\(Howtobeatit?\)

\*\*IMPORTANT:\*\*TheprogramaboveistheCURRENTprogramthatneedstobeimproved\.Startbyunderstandingwhatworks,thensuggestbreakthroughideasthatbuildonorimproveit\.

\#\#AnalysisFramework\-CompleteBeforeGeneratingIdeas

\*\*STEP0:UnderstandtheTASK\(MOSTIMPORTANT\-DOTHISFIRST\)\*\*

\-Whatistheproblemaskingyoutodo?

\-Whatisthegoalorobjective?\(maximize,minimize,optimize\)

\-Whataretheinputsandoutputs?

\-Whatneedstobeimproved?\(variables/decisionsthataffectthegoal\)

\-Whatconstraintsexist?

\*\*STEP1:AnalyzetheEvaluatorCode\*\*

\-Howaresolutionsscored?

\-Whatmetricsarecomputed?

\-Whatcausesfailuresorpenalties?

\*\*STEP2:IdentifyMetrics\*\*

\-WhatistheprimarymetricorParetoobjectiveset?

\-Howisitcalculated?

\-Whatsecondarymetricsexist?

\-Ifvariance/stdispenalized,theprogramneedsconsistencyacrossscenarios

\*\*STEP3:IdentifyConstraints\*\*

\-Whatconditionsmustbesatisfied?

\-Whatvalidationhappens?

\-Whatcausesscorepenalties?

\*\*STEP4:IdentifyProblemStructure\*\*

\-Isprocessingsequentialorglobal?

\-Aredecisionvariablesdiscreteorcontinuous?

\-Whatdependenciesexistbetweendecisions?

\-\*\*CRITICAL:\*\*Whatdatadoesyourprogramreceivevswhattheevaluatoruses?

\-\*\*CRITICAL:\*\*Howaremetricscomputedacrosscomponents?Independentlythenaggregated,orjointly?

\*\*STEP5:DetermineAppropriateApproach\*\*

\-Matchapproachtoproblemstructure

\-Considerwhathasworkedvsfailedbefore

\-Identifypromisinglibrary/techniquecombinations

\*\*STEP6:IdentifyImprovementOpportunities\*\*

\-Whatwouldincreaseeachmetric?

\-Whatwouldsatisfyconstraintsbetter?

\-Whatfundamentallydifferentapproachescouldwork?

Currentbestscoreis0\.380053\.Yourideasmustimprovetheconfiguredoptimizationtargetsand,inmultiobjectivemode,explicitlyreasonaboutobjectivetrade\-offs\.

\#\#Whatthesearchhasexplored,measuredfromtheseed

seed:combined\_score0\.3801llm\_calls\_per\_query9\.0000llm\_tokens\_per\_query34707\.4032oracle\_ndcg0\.4813retrieval\_recall0\.4274scoring\_realization0\.7896

Anamed\[Strategy\]inalineageisaparadigm;anunnamedoneisordinary

mutation,whichishowthesearchstarts\.Everyfigureisagainsttheseed\.

CURRENTBEST\(combined\_score0\.6071atit20\)\-\-anewideagetsappliedtothisprogram:

\-seed→\\rightarrow3mut→\\rightarrow\[ReciprocalRankFusion\(RRF\)forMulti\-QueryRetrieval\]\(it17\)→\\rightarrow\[ReciprocalRankFusion\(RRF\)forMulti\-QueryRetrieval\]\(it20\)

combined\_score0\.3801→\\rightarrow0\.6071\(Δ\\Delta\+0\.2271fromseed\)

llm\_calls\_per\_query9\.0000→\\rightarrow63\.2742llm\_tokens\_per\_query34707\.4032→\\rightarrow146786\.0806oracle\_ndcg0\.4813→\\rightarrow0\.8036retrieval\_recall0\.4274→\\rightarrow0\.7664scoring\_realization0\.7896→\\rightarrow0\.7555

netchangefromseed:\+\_generate\_queries,\+\_rrf\_fuse,~\_parse\_ranking,~\_rerank\_window,\-DEPTH,\-STRIDE,\-WINDOW\(\+rewired\`rank\`\)

\_generate\_queries:

Generate3semanticallyrelatedqueriesusingLLMformulti\-queryretrieval\.

UsesLLMtoexpandtheoriginalqueryintorelatedformulationsthatcapture

differentterminology,improvingretrievalrecallacrossdiversebiologyconcepts\.

\_rrf\_fuse:

FusemultipleBM25resultlistsusingReciprocalRankFusion\.

Scoreseachdocumentassumof1/\(k\+rank\)acrossallqueries\.

Higherscoresindicatebetteroverallrankingacrossqueries\.

\_parse\_ranking:

ExtractrankingorderfromLLMoutput,prioritizing\[n\]brackets\.

\_rerank\_window:

RerankawindowofcandidatesusingLLMlistwisecomparison\.

The10mostrecentlineages,oldestfirst:

\-seed→\\rightarrow3mut→\\rightarrow\[ReciprocalRankFusion\(RRF\)forMulti\-QueryRetrieval\]\(it17\)→\\rightarrow\[ReciprocalRankFusion\(RRF\)forMulti\-QueryRetrieval\]\(it20\)→\\rightarrow\[Tournament\-StylePairwiseEliminationforTop\-10Ranking\]\(it31\)

combined\_score0\.3801→\\rightarrow0\.3810\(Δ\\Delta\+0\.0010fromseed\)

llm\_calls\_per\_query9\.0000→\\rightarrow45\.0000llm\_tokens\_per\_query34707\.4032→\\rightarrow24599\.2581oracle\_ndcg0\.4813→\\rightarrow0\.5824retrieval\_recall0\.4274→\\rightarrow0\.4986scoring\_realization0\.7896→\\rightarrow0\.6543

netchangefromseed:\+\_generate\_queries,\+\_listwise\_rank,\+\_llm\_pairwise,\+\_pairwise\_tournament,\+\_rrf\_fuse,\-\_rerank\_window,~\_parse\_ranking,\-DEPTH,\-STRIDE,\-WINDOW\(\+rewired\`rank\`\)

\_listwise\_rank:

ListwiserankcandidatesusingasingleLLMcomparisoncall\.

Takesasmallsetofcandidates\(~10\)andaskstheLLMtoorderthem

frommosttoleastrelevantusingbracketnotation\.

\_llm\_pairwise:

Determinewhichoftwodocumentsismorerelevanttothequery\.

UsesLLMfordirectpairwisecomparison\.Tiesarebrokendeterministically

bypreferringDocumentA\(whichappearsearlierinthepre\-rankedlist,

servingasaBM25/RRFranktiebreaker\)\.

\_pairwise\_tournament:

Tournament\-stylepairwiseeliminationfortop\-10ranking\.

Splitscandidatesintopairs,usesLLMtodeterminerelevance,

repeatsknockoutroundsuntil<=10candidatesremain,thenlistwiseranksthem\.

ThislogarithmicapproachminimizesLLMcallswhilemaximizingtop\-10precision\.

\-seed→\\rightarrow3mut→\\rightarrow\[ReciprocalRankFusion\(RRF\)forMulti\-QueryRetrieval\]\(it17\)→\\rightarrow\[ReciprocalRankFusion\(RRF\)forMulti\-QueryRetrieval\]\(it20\)→\\rightarrow\[DeterministicQueryExpansionwithLogarithmicRRFFusion\]\(it32\)

combined\_score0\.3801→\\rightarrow0\.3837\(Δ\\Delta\+0\.0036fromseed\)

llm\_calls\_per\_query9\.0000→\\rightarrow32\.5161llm\_tokens\_per\_query34707\.4032→\\rightarrow85195\.2581oracle\_ndcg0\.4813→\\rightarrow0\.4917retrieval\_recall0\.4274→\\rightarrow0\.4373scoring\_realization0\.7896→\\rightarrow0\.7803

netchangefromseed:\+\_deterministic\_expand,\+\_log\_rrf\_fuse,~\_parse\_ranking,~\_rerank\_window,\-DEPTH,\-STRIDE,\-WINDOW\(\+rewired\`rank\`\)

\_deterministic\_expand:

Expandquerydeterministicallybyextractingkeytermsandhandlingnegations\.

ReplacesLLM\-basedquerygenerationwithalightweight,deterministicstrategy

thatimprovesretrievalrecallandoracleNDCGwithoutLLMoverhead\.

\_log\_rrf\_fuse:

FusemultipleBM25resultlistsusinglogarithmicRRFwithoriginalqueryweighting\.

Heavilyweightsdocumentsfromtheoriginalquerywhileboostingexpandedmatches\.

Penalizesdocumentsappearingonlyinexpandedqueriestomaintainprecision\.

\-seed→\\rightarrow3mut→\\rightarrow\[ReciprocalRankFusion\(RRF\)forMulti\-QueryRetrieval\]\(it17\)→\\rightarrow\[ReciprocalRankFusion\(RRF\)forMulti\-QueryRetrieval\]\(it20\)→\\rightarrow\[BatchedPointwiseLLMRelevanceScoringwithStructuredJSON\]\(it33\)

combined\_score0\.3801→\\rightarrow0\.3911\(Δ\\Delta\+0\.0110fromseed\)

llm\_calls\_per\_query9\.0000→\\rightarrow2\.0000llm\_tokens\_per\_query34707\.4032→\\rightarrow8821\.9194oracle\_ndcg0\.4813→\\rightarrow0\.6531retrieval\_recall0\.4274→\\rightarrow0\.6003scoring\_realization0\.7896→\\rightarrow0\.5988

netchangefromseed:\+\_batch\_score\_candidates,\+\_generate\_queries,\+\_rrf\_fuse,\-\_parse\_ranking,\-\_rerank\_window,\-DEPTH,\-STRIDE,\-WINDOW\(\+rewired\`rank\`\)

\_batch\_score\_candidates:

ScorecandidatesbyrelevanceusingasinglebatchedLLMcallwithJSONoutput\.

Evaluateseachcandidateona0\-100relevancescaleusingbiology\-specific

criteria\(accuracy,directrelevance,completeness,specificity\)\.Returns

candidatessortedbyscoredescending\.SingleLLMcalleliminateserror

propagationfromwindowoverlapsandreducescallsfrom~60to1\.

\[\.\.\.7morelineages\.\.\.\]

\#\#TechniqueGuidance

\*\*Note:\*\*Standardscientificlibraries\(scipy,numpy,etc\.\)areavailable\.PyTorchandTensorFlowarenotavailable\.

\*\*ForContinuousOptimizationwithConstraints:\*\*

\-scipy\.optimize\.minimizewithconstrainthandling\(SLSQP,trust\-constr\)

\-Multipleinitialguessesforglobaloptimization

\-Geometricapproaches\(Voronoi,convexhull\)

\*\*ForDiscrete/CombinatorialProblems:\*\*

\-Greedyheuristicswithgoodordering

\-Localsearch\(swaps,moves\)

\-scipy\.optimize\.linear\_sum\_assignmentforassignmentproblems

\-scipy\.optimize\.linprogforlinearconstraints

\*\*ForGraph/NetworkProblems:\*\*

\-NetworkXalgorithms\(shortestpath,minspanningtree,flow\)

\-Spectralmethods\(eigenvalue\-basedordering\)

\*\*ForRepair/Reconstruction:\*\*

\-Heuristic\-baseddetectionandcorrection

\-Structuralconstraintexploitation

\-Averaging/interpolationforconsistency

\*\*ForRobustFiltering/NoiseReduction:\*\*

\-scipy\.signal\(medfilt,savgol\_filter,wiener\)fordirectfiltering

\-Usemethodsthathandleoutliersbetterthanmean\-based\(median,percentile\)

\-DoNOTusescipy\.optimize\.minimizetotunefilterparameters

\-Usefilteringfunctionsdirectly,notmulti\-stageoptimization

\*\*GeneralPrinciples:\*\*

\-Prefersingle\-functionlibrarycallsovermulti\-stagepipelines

\-Matchalgorithmtoproblemstructure

\-Simpleapproacheswithgoodheuristicsoftenbeatcomplexmethods

\#\#ANTI\-PATTERNS\-CriticalrulesaboutwhatNOTtodo

1\.\*\*DoNOTusemulti\-stageoptimization\*\*:DoNOTcallonefunctionthenoptimizeitsoutput\.Deterministicsetupcodefollowedbyasingleoptimizationcallisallowed\.

2\.\*\*DoNOTusescipy\.optimize\.minimizeforhyperparametertuning\*\*:Useminimizetosolvetheproblemdirectly,NOTtotuneparametersforanotherfunction\.

3\.\*\*DoNOTusescipy\.optimize\.minimizefordiscreteproblems\*\*:Continuousoptimizerscannothandlediscreteconstraintviolationsproperly\.

4\.\*\*EachideaMUSTbeasingle\-functionlibrarycall\*\*:DoNOTsuggestmulti\-stageprocessing\(e\.g\.,"callAthencallB"\)\.

\*\*AVOID:\*\*DEAP,geneticalgorithmlibraries,domain\-specificcomplexlibraries,customresearchalgorithms,oranylibraryrequiringadditional\`pipinstall\`

\*\*LearningfromSuccess:\*\*

Whenanapproachsucceeds,think:whatprinciplemadeitwork?Learnandthinkofbetterideas,don'tjustaddcomplexity\.Ifbreakthroughpatternsareknown,prioritizeapproachesthatmatchthem\.

\#\#DIVERSITYREQUIREMENTS

Beforegeneratingideas,explicitlythink:

\-Idea1:\[TypeA\-e\.g\.,algorithmicrefinementorlibrary\-basedapproach\]

\-Idea2:\[TypeB\-e\.g\.,structuralchangeorprocessingpattern\-DIFFERENTfromA\]

\-Idea3:\[TypeC\-e\.g\.,differenttechniqueoroptimizationmethod\-DIFFERENTfromAandB\]

\*\*Verify:\*\*AretheseDIFFERENTtypes?NOTvariationsofthesameapproach\.

Eachideamust:

\-UseDIFFERENTlibraries/techniquesthanfailedattempts

\-TargetDIFFERENTmetrics/aspectsfromtheevaluator

\-Beindependentlyimplementable

\-Preferclearimplementations\(different\!=morecomplex\)

\#\#\#BeSpecificandActionable

Notvague:"Tryoptimization"

Specific:"Usescipy\.optimize\.minimizewithSLSQPmethod"

\-Includeexactlibrarynames,functionnames,methods,parameters

\-Providestep\-by\-stepimplementationguide

\-Focusoncorelogicthatimplementstheideacorrectly

\-Handleedgecasesandavoiderrors/warnings

\-Foroptimization:usemultipleinitializations,appropriateiterationcountsandconvergencecriteria\(evaluationtimeout:\{self\.eval\_timeout\}s\)

\#\#OutputFormat

\*\*IMPORTANT:\*\*RespondwithaJSONobjectcontainingexactly3ideaobjectsunderthe"ideas"key\.

Donotincludecodepatchesordiffs\-\-\-describestrategiesinnaturallanguage\.

Generate3breakthroughideasofDIFFERENTtypes\.

EachideamustbeaJSONobjectwiththesefields:

\-"idea":Clear,directdescriptionwithlibrary/techniquename

\-"description":Detailedimplementationguide\(5\-10sentences\)

\-"what\_to\_optimize":Whatmetrics/areastofocuson

\-"cautions":Importantimplementationdetailstowatchfor

\-"approach\_type":Exact"library\.function"format\(e\.g\.,"scipy\.optimize\.minimize"\)

\*\*DiversityRequirement:\*\*EachideamustuseaDIFFERENTapproachtype\.

Donotgeneratevariationsofthesametechnique\.

ReturnONLYaJSONobjectinthisshape:\{"ideas":\[\.\.\.\]\}with3paradigmobjects\.Noothertext\.

Example:

\`\`\`json

\{

"ideas":\[

\{

"idea":"Usescipy\.optimize\.minimizewithSLSQP",

"description":"Applyscipy\.optimize\.minimizedirectlytooptimizeallvariablestogether\.\.\.",

"what\_to\_optimize":"primaryevaluatorscore",

"cautions":"Ensureconstraintsareproperlyformulated,usemultiplestartingpoints",

"approach\_type":"scipy\.optimize\.minimize"

\}

\]

\}

\`\`\`

## Appendix BPrograms

### B\.1Seed program

Both searches initialize every run with this program\. It retrieves the top 100 BM25 candidates and reranks them with a sliding\-window listwise reranker\([Sun et al\., 2023](https://arxiv.org/html/2609.38639#bib.bib3)\), using windows of 20 with a stride of 10\.

"""InitialBRIGHTreranker\-\-\-BM25top\-100\+RankGPTsliding\-windowbaseline\."""

importre

\#Tunables\-\-\-evolutionmaychangetheseorremovethementirely\.

DEPTH=100\#BM25candidatestorerank

WINDOW=20\#sliding\-windowsize

STRIDE=10\#sliding\-windowstep

def\_parse\_ranking\(text,k\):

"""Parse'\[3\]\>\[1\]\>\.\.\.'into0\-basedindices:deduped,restrictedto0\.\.k\-1,with

anyomittedidentifiersappendedinoriginalorder\(alwaysafullpermutation\)\."""

order,seen=\[\],set\(\)

forminre\.findall\(r"\\\[\(\\d\+\)\\\]",textor""\):

idx=int\(m\)\-1

if0<=idx<kandidxnotinseen:

seen\.add\(idx\)

order\.append\(idx\)

order\.extend\(iforiinrange\(k\)ifinotinseen\)

returnorder

def\_rerank\_window\(query,window\_ids,tools\):

"""OnelistwiseLLMcall:reorderwindow\_idsbyrelevance\.Fallsbacktoinputorder

onanyfailure,sothewindowisalwaysavalidpermutationofitsinputs\."""

passages="\\n\\n"\.join\(

f"\[\{i\+1\}\]\{tools\.doc\(cid\)\}"fori,cidinenumerate\(window\_ids\)

\)

system="Rankthepassages\."

user=\(

f"Query:\\n\{query\}\\n\\n"

f"Candidatepassages:\\n\{passages\}\\n\\n"

f"RankALL\{len\(window\_ids\)\}passagesfrommosttoleastrelevanttothequery\."

f"RespondONLYwiththerankingasidentifiersintheformat"

f"\[a\]\>\[b\]\>\[c\]\>\.\.\.usingeveryidentifierfrom1to\{len\(window\_ids\)\}exactlyonce\."

\)

try:

text=tools\.llm\(

\[\{"role":"system","content":system\},

\{"role":"user","content":user\}\],

max\_tokens=1024,

\)

order=\_parse\_ranking\(text,len\(window\_ids\)\)

exceptException:

order=list\(range\(len\(window\_ids\)\)\)

return\[window\_ids\[i\]foriinorder\]

defrank\(query,tools\):

"""Returndoc\_idsrankedmost\-\>leastrelevantfor\`query\`\."""

\#Firststage:BM25candidatepool\.

ranked=tools\.bm25\(query,DEPTH\)

\#Secondstage:RankGPTlistwiseslidingwindow,bottom\-\>topsorelevantdocs

\#bubbleupwardthroughtheoverlap\.

window=min\(WINDOW,len\(ranked\)\)

end=len\(ranked\)

whilewindowandend\>0:

start=max\(0,end\-window\)

ranked\[start:end\]=\_rerank\_window\(query,ranked\[start:end\],tools\)

ifstart==0:

break

end\-=STRIDE

returnranked

### B\.2Evolved programs

For each objective, we show one of the final programs from biology with the highest train score\. The docstrings and comments are written and maintained by the mutation model\.

#### Quality\-only objective\.

Query expansion with reciprocal rank fusion and a denser sliding window, reaching0\.6330\.633nDCG@10 on test at151151k tokens per query\.

"""BM25retrievalwithmulti\-queryRRFfusion\+slidingwindowLLMreranker\."""

importre

def\_generate\_queries\(query,tools\):

"""Generate3semanticallyrelatedqueriesusingLLMformulti\-queryretrieval\.

UsesLLMtoexpandtheoriginalqueryintorelatedformulationsthatcapture

differentterminology,improvingretrievalrecallacrossdiversebiologyconcepts\.

"""

prompt=\(

f"Originalquery:\{query\}\\n\\n"

f"Generateexactly3relatedsearchqueriesforbiologyresearch\.\\n"

f"Keepeachqueryconcise\(3\-8words\)\.Outputoneperline\.\\n"

f"Format:1\.query1\\n2\.query2\\n3\.query3\\n"

f"Donotincludeanyothertext\."

\)

try:

text=tools\.llm\(

\[\{"role":"system","content":"Youareahelpfulassistant\."\},

\{"role":"user","content":prompt\}\],

max\_tokens=256

\)

\#Extractnumberedqueries

lines=text\.strip\(\)\.split\('\\n'\)

expanded=\[\]

forlineinlines:

line=line\.strip\(\)

\#Removenumbering

m=re\.match\(r"\\d\+\[\\\.\\\)\]\\s\*\(\.\+\)",line\)

ifm:

expanded\.append\(m\.group\(1\)\.strip\(\)\)

eliflineandnotline\.startswith\('\#'\):

expanded\.append\(line\)

\#Keeponlyfirst3

return\[query\]\+expanded\[:3\]

exceptException:

return\[query\]

def\_rrf\_fuse\(all\_results,k=60\):

"""FusemultipleBM25resultlistsusingReciprocalRankFusion\.

Scoreseachdocumentassumof1/\(k\+rank\)acrossallqueries\.

Higherscoresindicatebetteroverallrankingacrossqueries\.

"""

scores=\{\}

forrank\_listinall\_results:

forrank,doc\_idinenumerate\(rank\_list,1\):

scores\[doc\_id\]=scores\.get\(doc\_id,0\)\+1\.0/\(k\+rank\)

\#Sortbyscoredescending

fused=sorted\(scores\.keys\(\),key=lambdax:\-scores\[x\]\)

returnfused

def\_parse\_ranking\(text,k\):

"""ExtractrankingorderfromLLMoutput,prioritizing\[n\]brackets\."""

order,seen=\[\],set\(\)

forminre\.findall\(r"\\\[\(\\d\+\)\\\]",textor""\):

idx=int\(m\)\-1

if0<=idx<kandidxnotinseen:

seen\.add\(idx\)

order\.append\(idx\)

\#Fillmissingwithoriginalordertoguaranteeavalidpermutation

order\.extend\(iforiinrange\(k\)ifinotinseen\)

returnorder\[:k\]

def\_rerank\_window\(query,window\_ids,tools\):

"""RerankawindowofcandidatesusingLLMlistwisecomparison\."""

ifnotwindow\_ids:

return\[\]

passages="\\n\\n"\.join\(f"\[\{i\+1\}\]\{tools\.doc\(cid\)\}"fori,cidinenumerate\(window\_ids\)\)

prompt=\(

f"Query:\{query\}\\n\\n"

f"Candidates:\\n\{passages\}\\n\\n"

f"Rankthese\{len\(window\_ids\)\}passagesfrommosttoleastrelevant\.\\n\\n"

f"OutputONLYthefinalrankinginthisexactformat:\[a\]\>\[b\]\>\[c\]\\n"

f"Useeveryidentifierfrom1to\{len\(window\_ids\)\}exactlyonce\.\\n"

f"Donotincludeanyothertext\."

\)

try:

text=tools\.llm\(

\[\{"role":"system","content":"Youareahelpfulassistantthatranksdocuments\."\},

\{"role":"user","content":prompt\}\],

max\_tokens=1024

\)

order=\_parse\_ranking\(text,len\(window\_ids\)\)

exceptException:

order=list\(range\(len\(window\_ids\)\)\)

return\[window\_ids\[i\]foriinorder\]

defrank\(query,tools\):

"""Returndoc\_idsrankedmost\-\>leastrelevantfor\`query\`\."""

\#Multi\-queryretrievalwithRRFfusion

queries=\_generate\_queries\(query,tools\)

all\_results=\[\]

forqinqueries:

results=tools\.bm25\(q,100\)

all\_results\.append\(results\)

\#FuseusingRRFtoboostrecallandoracleNDCG

ranked=\_rrf\_fuse\(all\_results,k=60\)

ifnotranked:

return\[\]

\#SlidingwindowLLMreranker

window,stride,end=15,5,len\(ranked\)

whileend\>0:

start=max\(0,end\-window\)

ranked\[start:end\]=\_rerank\_window\(query,ranked\[start:end\],tools\)

ifstart==0:

break

end\-=stride

returnranked

#### Cost\-aware objective\.

This program replaces the sliding window with a single listwise LLM call over the top 100 candidates\. Before ranking, it asks the LLM which concepts the top ten BM25 results are missing and retrieves those concepts\. It reaches0\.5690\.569nDCG@10 on test with22LLM calls and17\.517\.5k tokens per query, about a ninth of the tokens of the quality\-only program\.

"""Single\-callLLMreranker\-\-\-BM25top\-100withcomprehensiverankinginoneLLMcall\."""

importre

def\_parse\_ranking\(text,k\):

"""Parse'\[3\]\>\[1\]\>\.\.\.'into0\-basedindices:deduped,restrictedto0\.\.k\-1,with

anyomittedidentifiersappendedinoriginalorder\(alwaysafullpermutation\)\."""

order,seen=\[\],set\(\)

forminre\.findall\(r"\\\[\(\\d\+\)\\\]",textor""\):

idx=int\(m\)\-1

if0<=idx<kandidxnotinseen:

seen\.add\(idx\)

order\.append\(idx\)

order\.extend\(iforiinrange\(k\)ifinotinseen\)

returnorder

defrank\(query,tools\):

"""Coverage\-GreedyCandidateExpansionwithConceptualGapAnalysis\."""

\#1\.InitialBM25retrieval

initial=tools\.bm25\(query,50\)

ifnotinitial:

return\[\]

\#2\.ConceptualGapAnalysis

\#Previewtop10candidatestoidentifymissingbiologicalconcepts

preview="\\n"\.join\(f"\[\{i\+1\}\]\{tools\.doc\(d\)\}"fori,dinenumerate\(initial\[:10\]\)\)

gap\_prompt=\(

f"Query:\{query\}\\n\\n"

f"Topretrieveddocscover:\\n\{preview\}\\n\\n"

f"Identifyupto3importantbiologicalconceptsmissingfromtheseresults\."

f"Returnthemasacomma\-separatedlistofshortsearchterms\."

\)

try:

gap\_resp=tools\.llm\(\[\{"role":"user","content":gap\_prompt\}\],max\_tokens=128\)

gap\_terms=\[t\.strip\(\)\.strip\('"\\''\)fortingap\_resp\.split\(","\)ift\.strip\(\)\]

exceptException:

gap\_terms=\[\]

\#3\.Targetedretrievalforgaps

gap\_hits=\[\]

fortermingap\_terms\[:3\]:

gap\_hits\.extend\(tools\.bm25\(term,15\)\)

\#4\.RRFFusion

rrf=\{\}

fori,dinenumerate\(initial\):

rrf\[d\]=rrf\.get\(d,0\)\+1\.0/\(i\+60\)

fordingap\_hits:

rrf\[d\]=rrf\.get\(d,0\)\+1\.0/60\.0

fused=sorted\(rrf\.items\(\),key=lambdax:x\[1\],reverse=True\)

candidates=\[dford,\_infused\[:100\]\]

ifnotcandidates:

return\[\]

\#5\.LLMComprehensiveRanking

passages="\\n"\.join\(f"\[\{i\+1\}\]\{tools\.doc\(d\)\}"fori,dinenumerate\(candidates\)\)

prompt=\(

f"Query:\{query\}\\n\\n"

f"RankALL\{len\(candidates\)\}passagesfrommosttoleastrelevant\.\\n"

f"Passages:\\n\{passages\}\\n\\n"

f"RespondONLYwiththerankingintheformat\[1\]\>\[2\]\>\[3\]\.\.\."

f"usingeveryidentifierfrom1to\{len\(candidates\)\}exactlyonce\."

\)

try:

text=tools\.llm\(

\[\{"role":"system","content":"Youarearelevancerankingassistant\."\},

\{"role":"user","content":prompt\}\],

max\_tokens=1024,

\)

order=\_parse\_ranking\(text,len\(candidates\)\)

exceptException:

order=list\(range\(len\(candidates\)\)\)

return\[candidates\[i\]foriinorder\]