Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombination

# Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombination
Source: [https://arxiv.org/html/2607.28947](https://arxiv.org/html/2607.28947)
Jingwen Fu\\equalcontrib1,2, Zhen Liu\\equalcontrib3, Yuhan Liu4, He Zhang1,2\\corresponding, Nanning Zheng3\\corresponding

###### Abstract

Large language models \(LLMs\) are increasingly used to solve complex problems by searching over program space, offering a general paradigm for scientific problems that can be naturally represented and solved as programs\. Despite recent progress, identifying effective optimization directions for a candidate program remains challenging\. By analogy with automatic differentiation, existing methods typically guide the search using a textual “gradient”: a first\-order update direction expressed as textual edits\. Such gradients are inferred either from previously evaluated programs or from LLM\-generated feedback on the implicit program\-score mapping\. However, these estimates become increasingly unreliable as the program–score mapping grows more complex, limiting their practical utility\. We argue that explicit gradients are not essential for effective program optimization\. Leveraging their prior knowledge, LLMs can propose plausible atomic edits directly from the current program, thereby enabling a zeroth\-order optimization strategy\. However, zeroth\-order search suffers from aweakest\-link effect: when a bundle of edits is accepted or rejected as a whole, a single harmful edit can negate the benefits of all remaining edits\. To address this issue, we introduce HERO, a program optimizer that prompts an LLM to generate diverse, non\-overlapping atomic edits and then systematically selects and composes them into coherent program improvements using evaluator scores\. We evaluate HERO across algorithmic problems, strategy games, the design of LLM\-based agentic systems, and robotic path planning\. Across these domains, HERO consistently discovers higher\-scoring programs and converges substantially faster than prior LLM\-based optimizers, while consuming fewer tokens\.

## 1Introduction

Optimization over program space provides a general framework for solving complex problems, ranging from mathematical construction to agent design\. Such problems can often be expressed naturally as programs, which are executable, automatically evaluable, and compositional\(Zhang et al\.[2024b](https://arxiv.org/html/2607.28947#bib.bib42); Yuksekgonul et al\.[2025](https://arxiv.org/html/2607.28947#bib.bib39); Ding et al\.[2025](https://arxiv.org/html/2607.28947#bib.bib6)\)\. LLMs are well suited to this setting as they can reason about program semantics, draw on broad prior knowledge, and iteratively revise candidate solutions\(Zhou et al\.[2025](https://arxiv.org/html/2607.28947#bib.bib45); Madaan et al\.[2023](https://arxiv.org/html/2607.28947#bib.bib20); Zhang et al\.[2025b](https://arxiv.org/html/2607.28947#bib.bib44); Romera\-Paredes et al\.[2024](https://arxiv.org/html/2607.28947#bib.bib29)\)\. The central difficulty, however, is how to steer the search\. At each iteration, the optimizer must determine an*optimization direction*to guide how the current program should be modified to improve its performance\(Nie et al\.[2023](https://arxiv.org/html/2607.28947#bib.bib25); Liu et al\.[2026c](https://arxiv.org/html/2607.28947#bib.bib19),[a](https://arxiv.org/html/2607.28947#bib.bib17),[b](https://arxiv.org/html/2607.28947#bib.bib18)\)\. Existing methods typically infer this direction from the observed performance of previously evaluated candidates, which we refer to as*first\-order information*\(Yuksekgonul et al\.[2025](https://arxiv.org/html/2607.28947#bib.bib39); Cheng, Nie, and Swaminathan[2024](https://arxiv.org/html/2607.28947#bib.bib5)\)\. The underlying intuition is that the evaluation history provides a local description of the objective: by identifying which past modifications helped or hurt, the optimizer can infer a useful update direction, analogous to estimating a numerical gradient\. Consequently, access to evaluation scores is often treated as essential\.

![Refer to caption](https://arxiv.org/html/2607.28947v1/x1.png)Figure 1:Workflow of HERO\. Given a task description, HERO iteratively improves the program by applying selectively recombined edits proposed by the LLM\.![Refer to caption](https://arxiv.org/html/2607.28947v1/x2.png)Figure 2:Overview of HERO\.\(A\)Working mechanism\. At each step, the LLM generates multiple non\-overlapping, heterogeneous atomic edits\. Exsiting methods apply or reject all edits by the global score, hence suffering from the weakest\-link effect\. HERO instead recombines the edits and selects the subset that maximizes performance, neutralizing the influence of harmful edits\.\(B\)Best\-subset recombination mitigates the weakest\-link effect and leads to substantial performance improvement\.We posit that this assumption becomes unreliable as the program–score mapping grows more complex\. Inferring an update from performance can be regarded as a credit\-assignment problem: the optimizer must attribute a scalar score to the many interacting components of a program and determine which components should be changed\. When the program–score mapping is simple, this attribution could be tractable\. Nevetheless, for those open scientific problems remained underexplored by LLMs, the candidate programs typically involve long execution traces, tightly coupled components, or stochastic outcomes\. As a result, the same evaluator score can be consistent with several different explanations, making the inferred attribution or ‘gradient direction’ fundamentally ambiguous\. We theoretically formulate this limitation in Appendix A, and show that a finite history of scores cannot uniquely determine the correct update direction when the decoder from scores to update labels is unconstrained\.

The limitations of score\-based reasoning raise the central question of this paper:Can an LLM\-based optimizer remain effective without reasoning about ground\-truth performance?Our starting point is a simple empirical evidence observed in prior research\(Yang et al\.[2024](https://arxiv.org/html/2607.28947#bib.bib37); Zhang et al\.[2024c](https://arxiv.org/html/2607.28947#bib.bib43); Xia et al\.[2024](https://arxiv.org/html/2607.28947#bib.bib34)\): even without performance feedback, LLMs possess substantial implicit knowledge of how a candidate solution might be improved \(Fig\.[1](https://arxiv.org/html/2607.28947#S1.F1)\)\. Given only a task description and a candidate program, they can often propose plausible, localized*atomic edits*\. The main difficulty, we argue, lies not in generating individual edits but in combining them\. As shown in Fig\.[2](https://arxiv.org/html/2607.28947#S1.F2)A, a straightforward approach is to apply all proposed edits together and accept or reject the resulting program according to its global score\. However, a batch may contain both beneficial and harmful edits since edit generation is stochastic and unguided by performance feedback\. Under this all\-or\-nothing decision rule, a single harmful edit can outweigh the gains from the others, causing the entire batch, including its useful edits, to be discarded\. We refer to this failure mode as theweakest\-link effect\. Simply instructing the model to generate only beneficial edits does not resolve the problem, because the model cannot reliably determine the effect of each edit before evaluation without ground\-truth feedback\. Harmful edits must therefore be filtered during selection, by evaluating modifications at a finer granularity than the full batch\.

Motivated by this observation, we introduceHERO\(HeterogeneousEditRecombinationOptimizer\), which separates edit generation from edit selection \(Fig\.[2](https://arxiv.org/html/2607.28947#S1.F2)A\)\. During generation, the LLM proposes optimization directions using its internal knowledge; during selection, scalar evaluations identify effective combinations of those directions\. At each iteration, HERO provides the LLM with only the task description and the current program, deliberately withholding its ground\-truth performance, and asks it to generate a set of*heterogeneous, non\-overlapping atomic edits*\. Instead of applying the entire set, HERO evaluates recombinations of these edits and retains the best\-performing subset, preventing a single harmful edit from overriding beneficial ones\. Ground\-truth performance is used only as a scalar criterion for comparing edit subsets and evaluating the resulting program; it is never interpreted by the LLM or used to infer an update direction\. In this sense, HERO performs zeroth\-order optimization while converting the LLM’s latent knowledge of plausible edits into reliable program improvements\. We evaluate HERO in four domains: Circle Packing\(Novikov et al\.[2025](https://arxiv.org/html/2607.28947#bib.bib26)\), Strategy Games\(Dor[2018](https://arxiv.org/html/2607.28947#bib.bib7)\), Agent System Design\(Zhang et al\.[2025a](https://arxiv.org/html/2607.28947#bib.bib40)\), and Path Planning\(Meng et al\.[2024](https://arxiv.org/html/2607.28947#bib.bib24); Xiao et al\.[2023](https://arxiv.org/html/2607.28947#bib.bib35)\)\. Together, these domains cover combinatorial construction, adversarial and partially observable decision making, agentic reasoning, and planning\.

Overall, this paper makes three contributions:

- •We identify a key bottleneck in LLM\-based program optimization: the challenge is not generating promising update directions, but composing heterogeneous edits without suffering from the*weakest\-link effect*induced by all\-or\-nothing acceptance\.
- •We introduce HERO, which separates edit generation from edit selection\. It generates heterogeneous, non\-overlapping atomic edits and selectively recombines them using scalar evaluations, without requiring the LLM to reason about ground\-truth performance\.
- •Across four domains, HERO achieves faster convergence, lower token consumption, and higher final performance\.

Finally, our results are not a claim that first\-order information is useless\. By charting what is achievable*without*it, we aim to recalibrate the community’s estimate of its value and to inform when, and how, it is worth the cost of methods that*do*leverage it\. The full discussion of related work is deferred to Appendix D\.

## 2Method

### 2\.1Formulation and Intuition

#### Problem setup\.

Let𝒳\\mathcal\{X\}be the solution space andf:𝒳→ℝ\+f:\\mathcal\{X\}\\to\\mathbb\{R\}^\{\+\}a target \(performance\) function\. The optimizer seeks a maximizer

x⋆∈arg​maxx∈𝒳⁡f​\(x\),x^\{\\star\}\\in\\operatorname\*\{arg\\,max\}\_\{x\\in\\mathcal\{X\}\}f\(x\),\(1\)and proceeds iteratively: at stepttit maintains an incumbentxt∈𝒳x\_\{t\}\\in\\mathcal\{X\}together with an*evaluation history*

Ht=\{\(xi,f​\(xi\)\)\}i=1t\.H\_\{t\}\\;=\\;\\bigl\\\{\(x\_\{i\},\\,f\(x\_\{i\}\)\)\\bigr\\\}\_\{i=1\}^\{t\}\.A single step proposes a*modification*α:𝒳→𝒳\\alpha:\\mathcal\{X\}\\to\\mathcal\{X\}and moves toα​\(xt\)\\alpha\(x\_\{t\}\)\. To contrast HERO with prior work precisely, we first classify the information available to such a proposal, then explain why HERO deliberately uses only part of it\.

#### First\- vs\. zeroth\-order information\.

We distinguish two sources of information a proposal mechanism may exploit\.*First\-order information*is any functional of the evaluation historyHtH\_\{t\}used to estimate the local ascent direction∇f​\(xt\)\\nabla f\(x\_\{t\}\)\(Cheng, Nie, and Swaminathan[2024](https://arxiv.org/html/2607.28947#bib.bib5); Zhang et al\.[2024a](https://arxiv.org/html/2607.28947#bib.bib41)\)\. The aim is not to reconstruct the entire gradient field but only the direction at the incumbentxtx\_\{t\}; this is presumed feasible becauseHtH\_\{t\}implicitly constrainsff\. The difficulty is that recovering∇f​\(xt\)\\nabla f\(x\_\{t\}\)fromHtH\_\{t\}is an*inverse*problem, and Appendix shows it is ill\-posed onceffis complex: a finite history is consistent with many distinct directions, so the inferred direction is unreliable precisely in the hard regime\.*Zeroth\-order information*, by contrast, is any information independent ofHtH\_\{t\}, namely that carried by the task description and the model’s internal prior\. Formally, a first\-order optimizer samples a modification from a distribution conditioned on the full context,

α∼π\(⋅∣Task,Ht,xt\),\\alpha\\sim\\pi\\bigl\(\\cdot\\mid\\mathrm\{Task\},\\,H\_\{t\},\\,x\_\{t\}\\bigr\),drawing on both sources, whereas a purely zeroth\-order optimizer conditions only on\(Task,xt\)\(\\mathrm\{Task\},\\,x\_\{t\}\)\.

#### A latent\-objective model of zeroth\-order proposals\.

As a zeroth\-order optimizer, HERO samplesα∼π\(⋅∣Task,xt\)\\alpha\\sim\\pi\\bigl\(\\cdot\\mid\\mathrm\{Task\},\\,x\_\{t\}\\bigr\)withholding all scores, so the proposed direction cannot be inferred fromHtH\_\{t\}and must originate from the model’s prior\. To analyze this prior we adopt the following modeling assumption—a tractable abstraction, not a claim about internal mechanics\. We posit a finite set of*latent objectives*g1,…,gh:𝒳→ℝg\_\{1\},\\dots,g\_\{h\}:\\mathcal\{X\}\\to\\mathbb\{R\}, each encoding a property the model associates with quality \(e\.g\., “avoid boundary waste” in packing, or “protect corners” in Othello\)\. We model each sampled modificationα\\alphaas an approximate ascent step on a*random mixture*of these objectives:

gθα=∑i=1hθiα​gi,α​\(xt\)−xt≈η​∇gθα​\(xt\),g^\{\\alpha\}\_\{\\theta\}=\\sum\_\{i=1\}^\{h\}\\theta^\{\\alpha\}\_\{i\}\\,g\_\{i\},\\qquad\\alpha\(x\_\{t\}\)\-x\_\{t\}\\;\\approx\\;\\eta\\,\\nabla g^\{\\alpha\}\_\{\\theta\}\(x\_\{t\}\),\(2\)whereη\>0\\eta\>0is a step size and the weight vectorθα=\(θ1α,…,θhα\)\\theta^\{\\alpha\}=\(\\theta^\{\\alpha\}\_\{1\},\\dots,\\theta^\{\\alpha\}\_\{h\}\)is drawn afresh for each sample, reflecting the stochasticity of generation\. Under this model a single edit is a noisy, partially correct direction: it ascends*some*objective the model believes in, but not necessarilyff\.

#### Objective: aligning the aggregate withff\.

Since the mixture weights are random, no singleα\\alphais guaranteed to increaseff\. The problem HERO solves is therefore one of*aggregation*: select a sub\-collection\{α1,…,αn\}\\\{\\alpha\_\{1\},\\dots,\\alpha\_\{n\}\\\}whose combined direction is positively aligned with the true objective nearxtx\_\{t\}, i\.e\., there existsc\>0c\>0such that

∇​∑i=1ngθαi​\(xt\)≈c​∇f​\(xt\)\.\\nabla\\\!\\sum\_\{i=1\}^\{n\}g^\{\\alpha\_\{i\}\}\_\{\\theta\}\(x\_\{t\}\)\\;\\approx\\;c\\,\\nabla f\(x\_\{t\}\)\.\(3\)Equation \([3](https://arxiv.org/html/2607.28947#S2.E3)\) formalizes the intuition that combining many partially correct edits cancels their objective\-specific \(off\-ff\) components while reinforcing the shared component that tracksff, converting diffuse prior knowledge into a usable ascent direction without ever reasoning over scores\.

#### The weakest\-link effect\.

A naive aggregation applies*all*proposed edits and accepts the result by its global score; we show this squanders the very alignment that Eq\. \([3](https://arxiv.org/html/2607.28947#S2.E3)\) targets\. Let\{α1,…,αn\}\\\{\\alpha\_\{1\},\\dots,\\alpha\_\{n\}\\\}be the proposals, write the joint updateα1:n=α1∘⋯∘αn\\alpha\_\{1:n\}=\\alpha\_\{1\}\\circ\\cdots\\circ\\alpha\_\{n\}, and letδi=f​\(αi​\(xt\)\)−f​\(xt\)\\delta\_\{i\}\\;=\\;f\\bigl\(\\alpha\_\{i\}\(x\_\{t\}\)\\bigr\)\-f\(x\_\{t\}\)denote the marginal gain of editii\. When the edits act on disjoint parts of the solution—as enforced in Section[2\.2](https://arxiv.org/html/2607.28947#S2.SS2)—their effects are approximately additive,

f​\(α1:n​\(xt\)\)−f​\(xt\)≈∑i=1nδi\.f\\bigl\(\\alpha\_\{1:n\}\(x\_\{t\}\)\\bigr\)\-f\(x\_\{t\}\)\\;\\approx\\;\\sum\_\{i=1\}^\{n\}\\delta\_\{i\}\.\(4\)*All\-or\-nothing*acceptance keeps the bundle iff this sum is positive\. Because the model receives no feedback, it cannot certify any edit, so someδj<0\\delta\_\{j\}<0is essentially unavoidable; by Eq\. \([4](https://arxiv.org/html/2607.28947#S2.E4)\), a single sufficiently harmful edit—one withδj<−∑i≠jδi\\delta\_\{j\}<\-\\sum\_\{i\\neq j\}\\delta\_\{i\}when the remaining edits are beneficial—renders the total non\-positive and triggers rejection of the*entire*bundle, discarding every beneficial edit along with the harmful one\. We call this theweakest\-link effect: the fate of the bundle is dictated by its worst component\. It explains the puzzle from Section 1—the model proposes sound edits, yet all\-or\-nothing acceptance prevents them from accumulating into gains\. The remedy is to evaluate at a finer granularity than the full bundle: the subsetS⋆=arg​maxS⊆\[n\]​∑i∈SδiS^\{\\star\}=\\operatorname\*\{arg\\,max\}\_\{S\\subseteq\[n\]\}\\sum\_\{i\\in S\}\\delta\_\{i\}retains exactly the beneficial edits\{i:δi\>0\}\\\{i:\\delta\_\{i\}\>0\\\}and attains gain∑i:δi\>0δi≥∑i=1nδi\\sum\_\{i:\\delta\_\{i\}\>0\}\\delta\_\{i\}\\geq\\sum\_\{i=1\}^\{n\}\\delta\_\{i\}\. This subset search is the core of HERO, developed next\.

### 2\.2Generation–Selection Framework

The analysis of Section[2\.1](https://arxiv.org/html/2607.28947#S2.SS1)isolates two distinct concerns:*proposing*candidate directions, which the model’s prior already supports, and*verifying*which of them to keep, which all\-or\-nothing acceptance handles poorly\. HERO decouples these into a knowledge\-driven*Generation*phase and a feedback\-driven*Selection*phase\. Starting from an initial programx0∈𝒳x\_\{0\}\\in\\mathcal\{X\}, it applies the composite update

xt=Update​\(xt−1\)=\(Select∘Generate\)​\(xt−1\),\\displaystyle x\_\{t\}=\\mathrm\{Update\}\(x\_\{t\-1\}\)=\\bigl\(\\mathrm\{Select\}\\circ\\mathrm\{Generate\}\\bigr\)\(x\_\{t\-1\}\),\(5\)and returnsxTx\_\{T\}as the final solution\. Generation supplies candidate edits from the model’s prior alone, while selection uses ground\-truth performance purely as a scalar criterion to retain the edits that help\. We specify each phase in turn\.

#### Generation phase\.

Queried with only the task description and the incumbentxt−1x\_\{t\-1\}—no scores—the model returns a set of atomic edits

A=\{α1,…,αn\},αi:𝒳→𝒳,A=\\\{\\alpha\_\{1\},\\dots,\\alpha\_\{n\}\\\},\\qquad\\alpha\_\{i\}:\\mathcal\{X\}\\to\\mathcal\{X\},whose cardinalitynnis chosen by the model itself, so that granularity adapts to how much the program admits improvement\. We require the edits to be*heterogeneous*and pairwise*commuting*: for alli,j∈\[n\]i,j\\in\[n\],

αi∘αj=αj∘αi\.\\alpha\_\{i\}\\circ\\alpha\_\{j\}=\\alpha\_\{j\}\\circ\\alpha\_\{i\}\.\(6\)Commutativity is what makes subset selection well posed\. For any index subsetS⊆\[n\]S\\subseteq\[n\], define the recombined program

AS\(x\)=\(○i∈Sαi\)\(x\),A\_\{S\}\(x\)=\\Bigl\(\\,\\bigcirc\_\{i\\in S\}\\alpha\_\{i\}\\,\\Bigr\)\(x\),the composition of the edits indexed bySS\. By Eq\. \([6](https://arxiv.org/html/2607.28947#S2.E6)\) this composition is independent of the order of application, soASA\_\{S\}is unambiguous and an instruction such as “keep edits\{i,k\}\\\{i,k\\\}but dropjj” is well defined; without commutativity the effect of a subset would depend on application order and on which other edits are present, and recombination would be ill posed\. Commutativity thus turns recombination into a clean combinatorial choice over the2n2^\{n\}subsets of disjoint, heterogeneous edits\.

#### Selection phase\.

GivenAA, selection returns the best\-performing subset at the incumbentxt−1x\_\{t\-1\}:

S⋆∈arg​maxS⊆\[n\]⁡D^S​f​\(xt−1\),D^S​f​\(xt−1\)=f​\(AS​\(xt−1\)\)−f​\(xt−1\)∥S∥\.\\begin\{gathered\}S^\{\\star\}\\in\\operatorname\*\{arg\\,max\}\_\{S\\subseteq\[n\]\}\\;\\widehat\{D\}\_\{S\}\\,f\(x\_\{t\-1\}\),\\\\ \\widehat\{D\}\_\{S\}\\,f\(x\_\{t\-1\}\)=\\frac\{f\\\!\\left\(A\_\{S\}\(x\_\{t\-1\}\)\\right\)\-f\(x\_\{t\-1\}\)\}\{\\lVert S\\rVert\}\.\\end\{gathered\}\(7\)after which we setxt=AS⋆​\(xt−1\)x\_\{t\}=A\_\{S^\{\\star\}\}\(x\_\{t\-1\}\)\. HereD^S​f​\(xt−1\)\\widehat\{D\}\_\{S\}\\,f\(x\_\{t\-1\}\)is the empirical directional improvement offfalong the recombined edit \(the finite\-sample analogue of the directional derivativeDS​fD\_\{S\}f\),ASA\_\{S\}is as defined above, and∥S∥\\lVert S\\rVertis the update magnitude, set to11throughout for simplicity\. Two observations connect this rule to Section[2\.1](https://arxiv.org/html/2607.28947#S2.SS1)\. First, because the optimum ranges over*all*subsets rather than the single full bundleS=\[n\]S=\[n\], harmful edits are excluded while beneficial ones are retained: under the additive approximation of Eq\. \([4](https://arxiv.org/html/2607.28947#S2.E4)\) with∥S∥=1\\lVert S\\rVert=1, Eq\. \([7](https://arxiv.org/html/2607.28947#S2.E7)\) reduces tomaxS⊆\[n\]​∑i∈Sδi\\max\_\{S\\subseteq\[n\]\}\\sum\_\{i\\in S\}\\delta\_\{i\}, whose maximizer is exactly\{i:δi\>0\}\\\{i:\\delta\_\{i\}\>0\\\}\. This is the precise mechanism by which HERO neutralizes the weakest\-link effect\. Second, Eq\. \([7](https://arxiv.org/html/2607.28947#S2.E7)\) usesffonly as a black\-box scalar to*compare*candidates; it never asks*why*one subset scores higher, keeping HERO strictly zeroth\-order in the sense of Section[2\.1](https://arxiv.org/html/2607.28947#S2.SS1)\.

### 2\.3Design Details

HERO follows a score\-free generation and evaluator\-based selection procedure\. Given the task description and current program, the LLM produces localized, pairwise non\-overlappingSEARCH/REPLACEedits\(Novikov et al\.[2025](https://arxiv.org/html/2607.28947#bib.bib26)\)\. The non\-overlapping constraint makes edit composition order\-independent and enables well\-defined subset recombination\. Selection first removes edits that cause execution errors, then evaluates candidate subsets and retains the highest\-scoring recombination\. We enumerate all subsets when feasible; otherwise, we use a budgeted subset pool\. For expensive evaluator, we adopt an approximately order\-preserving surrogate\. The evaluation budget therefore balances subset coverage against ranking reliability\. Evaluator scores are used only for subset comparison and are never provided to the LLM\. See more implementation details in Appendix E\.

#### Discussion\.

We close by stating precisely the role ground\-truth performance plays in HERO, since this is exactly what separates it from first\-order methods and substantiates the zeroth\-order claim made in the abstract and introduction\. During optimization the signal enters at a single point—the comparison of edit subsets in Eq\. \([7](https://arxiv.org/html/2607.28947#S2.E7)\)—and at one further point afterward, the evaluation of the returned programxTx\_\{T\}\. It is never*reasoned about*: HERO extracts no explanation of*why*one subset outscores another and infers no structural insight from the score\. All directional knowledge originates from the model’s prior \(the Generation phase\), whereas performance acts solely as a scalar comparison oracle \(the Selection phase\)\. This cleanly classifies HERO as a*zeroth\-order*optimizer in the sense of Section[2\.1](https://arxiv.org/html/2607.28947#S2.SS1), and it explains why HERO sidesteps the credit\-assignment failure formalized in Appendix A: by*ranking*candidates rather than inverting a scalar score back onto a solution’s parts, it never attempts the ill\-posed inversion that renders first\-order reasoning unreliable in the hard regime\.

## 3Experiments

![Refer to caption](https://arxiv.org/html/2607.28947v1/x3.png)Figure 3:Results on the Circle Packing task\.A\.Number of steps required to reach a configuration withsum\_radii\>2\.60\\texttt\{sum\\\_radii\}\>2\.60\.B\.Average number of tokens consumed per evolution step\.C\.Optimization trajectories of different methods during the search\.Our experiments are designed to test the paper’s central claim, not merely to report wins\. If the bottleneck in LLM\-driven optimization is composing edits rather than knowing directions, then a zeroth\-order method that recombines edits should \(1\) converge faster, \(2\) cost fewer tokens, and \(3\) reach better final solutions than first\-order baselines—and, critically, its advantage should*grow*as the solution\-to\-performance mapping becomes harder to reason about, since that is where first\-order credit assignment degrades\. We therefore study four task families—Circle Packing, Strategy Games, Agent System Design, and Path Planning—chosen to span combinatorial construction, adversarial and partially observable decision making, agentic reasoning, and planning, so that they stress both the reasoning and the optimization capabilities of the LLM\. Unless otherwise stated, we use Qwen\-Plus\(Alibaba[2025](https://arxiv.org/html/2607.28947#bib.bib1)\)as the base LLM\.

### 3\.1Circle Packing

Task Description\.The circle packing problem aims to place 26 circles within a unit square while maximizing the sum of their radii, subject to non\-overlapping constraints and the requirement that all circles remain inside the square\. The task blends discrete placement decisions with continuous radius adjustments, making it a challenging benchmark: it features numerous local optima, and naive methods often become trapped in suboptimal configurations that use space inefficiently\. This task provides a suitable testbed for our claim because good moves \(e\.g\., shifting a row, regrowing a radius\) are locally describable, yet their joint effect on the global sum is highly coupled—precisely the regime where bundling edits should hurt\.

Convergence Speed\.We measure the number of steps each method needs to reach a target performance of2\.602\.60; results are summarized in Fig\.[3](https://arxiv.org/html/2607.28947#S3.F3)A\. HERO reaches the target with substantially fewer steps than all baselines\. The reason is structural rather than incidental: by recombining edits, each step retains locally beneficial change instead of discarding a whole batch whenever one edit misfires, so progress compounds across steps rather than stalling\. This translates into strong sample efficiency, which matters most when interaction budgets are tight—a common constraint when deploying LLM\-driven optimization under time or compute limits\.

Token Consumption\.Faster convergence would be of little practical value if each step were expensive, so we next measure the average token usage per step over a 100\-step run \(Fig\.[3](https://arxiv.org/html/2607.28947#S3.F3)B\); tokens directly reflect runtime cost in LLM\-based applications\. HERO consistently requires the fewest tokens per iteration\. The saving is a direct consequence of being zeroth\-order: because HERO neither feeds in scored examples nor asks the model to reason over performance, its prompts omit the long example contexts and chain\-of\-thought analyses that first\-order methods rely on, and each update is produced in a single edit\-generation pass\. Efficiency here is thus not an engineering trick but a built\-in benefit of removing first\-order reasoning\.

Learning Curve\.To see*how*the methods reach their solutions, we compare learning curves over 100 steps \(Fig\.[3](https://arxiv.org/html/2607.28947#S3.F3)C\)\. HERO shows a pronounced early acceleration, improving far sooner than competitors, which progress slowly and incrementally\. This shape is exactly what our diagnosis predicts: the model already retrieves strong task\-relevant moves from the outset, so once selection prevents good moves from being vetoed, large gains are available immediately rather than having to be slowly inferred from accumulated score history\.

### 3\.2Strategy Games

The previous task is fully deterministic\. We now raise the difficulty of credit assignment along two axes—long\-horizon interaction and partial observability—using two games, which lets us test the prediction that HERO’s advantage widens as reasoning from scores gets harder\.

Othello\.Othello is a classic two\-player deterministic board game on an8×88\\times 8grid\. At each turn a player places a disc to flip the opponent’s pieces, and the outcome depends heavily on long\-horizon positional planning\. Solving the task requires evaluating board configurations, anticipating multi\-step consequences, and countering an adaptive opponent, making it a strong benchmark for strategic reasoning and adversarial foresight\. We use win\-rate as the performance signal and design two opponents \(details in Appendix\); we denote the task against the hard opponent as Othello\-H and against the easy one as Othello\-E\.

Battleship\.Battleship is a probabilistic strategy game in which a player must locate hidden ships on a grid through a sequence of queries\. Unlike Othello, the environment is uncertain and only partially observable: the agent must maintain and continuously update a belief over feasible ship configurations \(details in Appendix\)\. Strong performance hinges on probing strategies that maximize information gain, balancing exploration and exploitation, and revising plans as new evidence arrives\. Performance is measured as the ratio of successful hits to total shots\. We define two difficulty levels by grid size, Battleship\-H \(hard\) and Battleship\-E \(easy\)\.

Table 1:Results on Strategy Games\.Performance\.We run the optimizer for 60 steps on Othello and 20 steps on Battleship, and report final scores in Table[1](https://arxiv.org/html/2607.28947#S3.T1), comparing against EoH, Best\-of\-N, OpenEvolve, and ShinkaEvolve under identical settings\. HERO achieves the best performance on all four tasks\. More telling than the wins is their pattern\. On the easy variants the field is close \(Othello\-E:0\.970\.97vs\.0\.900\.90–0\.930\.93for the evolutionary baselines\), but on the hard variants the gap explodes: on Othello\-H HERO reaches0\.850\.85while*every*competitor collapses to at most0\.150\.15\. This is the signature our thesis predicts\. On Othello\-E the score\-to\-performance mapping is benign enough that first\-order reasoning still works, so all reasonable methods cluster together; on Othello\-H the mapping becomes deeply entangled, first\-order direction\-finding degenerates, and only a method that bypasses it—recombining the model’s prior knowledge—keeps improving\. The same trend holds on the partially observable Battleship \(0\.340\.34/0\.280\.28vs\. at most0\.200\.20/0\.140\.14\)\. Averaged over the four tasks, HERO scores0\.610\.61, well above OpenEvolve \(0\.360\.36\) and ShinkaEvolve \(0\.330\.33\); notably, the largest contribution to this margin comes from the hard settings, which is precisely where the value of avoiding first\-order reasoning should be greatest\.

![Refer to caption](https://arxiv.org/html/2607.28947v1/x4.png)Figure 4:Analysis on Strategy Games\.A\.Effect of the surrogate evaluatorE′E^\{\\prime\}with different numbers of replicate games\.B\.Average performance of HERO, ShinkaEvolve, and OpenEvolve across three LLMs\.Choice of Evaluator and Overfitting\.A possible objection is that HERO simply trades expensive reasoning for expensive evaluation, since selection \(Eq\.[7](https://arxiv.org/html/2607.28947#S2.E7)\) callsffrepeatedly\. In Othello, the faithful evaluator runs 1,000 games per candidate, which is costly, so we replace it with a cheap surrogateE′E^\{\\prime\}and ask how coarseE′E^\{\\prime\}can be before selection suffers\. Recall the method’s claim that selection needs only the correct*ranking*of subsets, not accurate absolute scores; Fig\.[4](https://arxiv.org/html/2607.28947#S3.F4)A tests this directly by sweeping the grain ofE′E^\{\\prime\}\(number of games used to estimate the win\-rate, horizontal axis\) against win\-rate \(vertical axis\)\. The training win\-rate \(blue\) stays close to1\.01\.0even with very few games, and the test win\-rate \(orange\), though lower with a handful of games, rises rapidly and plateaus, with negligible gains beyond a small budget\. The interpretation is that ranking is far more robust to estimation noise than the scores themselves: even a noisyE′E^\{\\prime\}usually orders a clearly\-good subset above a clearly\-bad one, which is all selection requires\. HERO therefore attains accurate final performance without expensive large\-scale evaluation, confirming that its reliance onffis cheap in practice rather than a hidden cost\.

Choice of LLM\.If the gains came from a quirk of one model, our claim about LLMs’ latent knowledge would be weak; we therefore repeat the comparison across three base models in Fig\.[4](https://arxiv.org/html/2607.28947#S3.F4)B: Qwen\-Plus, Deepseek\-v3\.2, and Kimi\-K2\-Instruct \(full per\-task breakdown in Appendix B\)\. HERO leads consistently—61%61\\%vs\.33%33\\%/36%36\\%on Qwen\-Plus,60%60\\%vs\.38%38\\%/41%41\\%on Deepseek\-v3\.2, and61%61\\%vs\.46%46\\%/44%44\\%on Kimi\-K2\-Instruct—and its lead persists even on the weaker base model, indicating that the relevant directional knowledge is broadly present across models and that HERO’s role is to unlock it rather than to supply it\. HERO is also more stable across runs, whereas the baselines show larger variability and lower ceilings, consistent with the brittleness that all\-or\-nothing acceptance introduces\.

### 3\.3LLM Agent System Design

The previous tasks optimize a single artifact\. We now test whether the same mechanism improves an*agent scaffold*, and whether the improvements transfer across model scales\. We follow the protocol ofLange, Imajuku, and Cetin \([2025](https://arxiv.org/html/2607.28947#bib.bib14)\)on the AIME benchmark: the optimizer is run on AIME 2025\(Mathematical Association of America[2025](https://arxiv.org/html/2607.28947#bib.bib23)\)and evaluated on held\-out AIME 2024\(Mathematical Association of America[2024](https://arxiv.org/html/2607.28947#bib.bib22)\)and AIME 2023\(Mathematical Association of America[2023](https://arxiv.org/html/2607.28947#bib.bib21)\)\. Optimization runs for 10 steps using Qwen3\-0\.6B to generate candidate scaffolds; to probe generalization, we then evaluate the optimized scaffold on the larger Qwen3\-7B model\. Alongside accuracy, we report the average number of LLM queries per method as a proxy for computational cost\.

Figure[5](https://arxiv.org/html/2607.28947#S3.F5)shows that all optimization\-based methods beat the base agent, lifting average accuracy from18\.5%18\.5\\%to over27%27\\%; HERO is best at28\.7%28\.7\\%\(33\.3%33\.3\\%on 2024,24\.1%24\.1\\%on 2023\)\. Two aspects support our thesis\. First, the optimized scaffold, discovered on AIME 2025 with a tiny 0\.6B model, still wins when transferred to a 7B model and to earlier years—improvements that are structural and model\-agnostic, as expected if HERO is selecting genuinely good design choices rather than overfitting to one evaluator\. Second, the efficiency gap is large: HERO needs only2\.152\.15LLM calls on average, versus44for ShinkaEvolve and55for OpenEvolve, because it neither replays scored exemplars nor runs reasoning chains over them\. The result is a favorable accuracy–efficiency trade\-off that scales to practical settings\.

![Refer to caption](https://arxiv.org/html/2607.28947v1/x5.png)Figure 5:Accuracy–efficiency trade\-off on AIME\. Each point reports average accuracy over AIME 2024 and 2023 together with the average number of LLM queries\. Points closer to the upper\-left corner are preferred\.
### 3\.4Path Planning

Our final task moves from open\-ended construction to a problem with a known optimum, which lets us separate*solution quality*from*search efficiency*\. Grid\-based robot path planning\(Wang et al\.[2019](https://arxiv.org/html/2607.28947#bib.bib33); Wang[2021](https://arxiv.org/html/2607.28947#bib.bib31); Jahanshahi and Sari[2018](https://arxiv.org/html/2607.28947#bib.bib12)\)guides an agent from a start cell to a target cell on a two\-dimensional lattice, where each cell is either traversable or occupied\. The objective is to find a collision\-free path of adjacent traversable cells that avoids all obstacles\. Depending on the setting, movement may be restricted to four\-connected neighborhoods \(up, down, left, right\) or extended to eight\-connected motion including diagonals\. Each step incurs a cost—uniform in simple settings, but variable when diagonal motion or terrain\-dependent constraints are present—so the challenge is to reach the goal while minimizing cumulative path cost\. Because the environment is fully discretized, the problem is naturally formulated as graph search, with grid cells as nodes and valid transitions as edges; classical algorithms such as A\* find feasible, cost\-optimal paths via heuristics and cost structures\.

Figure[6](https://arxiv.org/html/2607.28947#S3.F6)reportspath length\(solution quality\) andsearch complexity\(planning cost, measured by visited states\), comparing algorithms designed by HERO and OpenEvolve against A\* and PPO\. On path length all methods tie—length 33/34 in Canyon and 40 in Double Door—so optimality is not in question, and the comparison reduces to efficiency\. There the methods diverge sharply: in Canyon, A\* and PPO visit 264 and 256 states while HERO visits only 142, and in the harder Double Door the gap widens further—A\* and PPO need 320 and 314 while HERO needs just 132, about a third of the classical cost\. Tellingly, OpenEvolve, the other LLM optimizer, is the*least*efficient \(361 and 491\): merely using an LLM does not buy efficiency\. The advantage comes from HERO composing the model’s heuristic knowledge into a search procedure that prunes unnecessary exploration, and once again it grows with task difficulty\.

![Refer to caption](https://arxiv.org/html/2607.28947v1/x6.png)Figure 6:Path planning performance\. All methods achieve comparable path quality, while HERO substantially reduces search complexity measured by visited states\. The path lengths are reported below each environment, and the bars show the number of visited states\.

## 4Analysis of HERO

Having shown*that*HERO works, we now examine*why*, and whether its simplicity is a limitation or a feature\. We analyze two representative game benchmarks, Othello and Battleship, which differ in decision structure and feedback: Othello emphasizes local strategy composition and long\-horizon value evaluation in a sequential game, whereas Battleship emphasizes search and exploration under imperfect information\. Studying both lets us probe HERO across complementary, strategy\-driven and search\-driven regimes, and to ask whether common “add\-ons” would improve it\.

![Refer to caption](https://arxiv.org/html/2607.28947v1/x7.png)Figure 7:Ablation study on Strategy Games\. Each cell reports the performance change relative to HERO, computed as the variant score minus the full HERO score\. Negative values indicate performance drops compared with HERO, while positive values indicate improvements\.### 4\.1Ablation Study

We first remove HERO’s two core components—the*recombination mechanism*and the*best\-update mechanism*—to test whether each is necessary, and we report both alongside two common enhancements \(in\-context examples and a larger population\) for context\. All results are in Figure[7](https://arxiv.org/html/2607.28947#S4.F7)\.

Removing recombination—reverting to applying the raw bundled output—is the most direct test of the weakest\-link hypothesis\. As shown in Fig\.[7](https://arxiv.org/html/2607.28947#S4.F7), this ablation causes the largest average drop, reducing the score by0\.150\.15relative to the full HERO configuration\. The effect is especially pronounced on Othello\-H, where performance drops from0\.850\.85to0\.440\.44\. Battleship\-H also decreases from0\.280\.28to0\.160\.16, while Battleship\-E remains saturated at0\.340\.34\. This pattern supports the claim that recombination is most valuable when the generated edit bundle is likely to contain both beneficial and harmful edits\.

Removing best\-update also weakens the method, but less severely\. The average score decreases from0\.610\.61to0\.560\.56, with the main loss appearing on Othello\-H, where performance drops from0\.850\.85to0\.630\.63\. Battleship\-H remains unchanged in the best\-of\-seeds score, but this does not negate the role of best\-update; rather, it suggests that its stabilizing effect is most visible in strategy\-driven settings such as Othello\. Overall, recombination provides the main protection against weakest\-link failures, while best\-update stabilizes the trajectory around the best\-so\-far program\.

### 4\.2Optimality Verification

A natural worry is that such a minimal design leaves easy gains on the table\. We test this by adding two enhancements widely believed to help LLM optimizers—extra in\-context examples and a larger program population—and asking whether either improves on the full method \(Figure[7](https://arxiv.org/html/2607.28947#S4.F7)\)\.

The answer is mixed at the individual\-task level but clear at the average level\. Adding examples slightly improves Othello\-E from0\.970\.97to0\.980\.98, but it hurts the harder settings, reducing Othello\-H from0\.850\.85to0\.640\.64and Battleship\-H from0\.280\.28to0\.240\.24\. Its average score is therefore0\.550\.55, below the full HERO score of0\.610\.61\. This suggests that additional examples can help in benign settings, but they do not provide a robust improvement across tasks\.

Increasing the population shows a similar pattern\. It remains competitive on Othello\-H, reaching0\.800\.80, but it reduces Battleship\-H from0\.280\.28to0\.180\.18and yields an average score of0\.570\.57, still below the full method\. Thus, neither enhancement improves the overall accuracy–efficiency trade\-off\. The results support the design choice of keeping HERO minimal: its gains come primarily from selectively recombining heterogeneous edits and anchoring the search to the best\-so\-far program, rather than from adding more context or maintaining a larger population\.

## 5Conclusion

We introduced HERO, which improves programs by generating atomic edits without performance feedback and selecting their best recombination using scalar scores\. This avoids unreliable score reasoning and the weakest\-link effect\.

## References

- Alibaba \(2025\)Alibaba\. 2025\.Qwen\-Plus\.*https://qwen\-ai\.chat/models/qwen\-plus/*\.
- Bosio and Mueller \(2025\)Bosio, C\.; and Mueller, M\. W\. 2025\.Synthesizing interpretable control policies through large language model guided search\.In*2025 American Control Conference \(ACC\)*, 583–590\. IEEE\.
- Brahmachary et al\. \(2024\)Brahmachary, S\.; Joshi, S\. M\.; Panda, A\.; Koneripalli, K\.; Sagotra, A\. K\.; Patel, H\.; Sharma, A\.; Jagtap, A\. D\.; and Kalyanaraman, K\. 2024\.Large Language Model\-Based Evolutionary Optimizer: Reasoning with Elitism\.*arXiv preprint arXiv:2403\.02054*\.
- Cheng et al\. \(2023\)Cheng, C\.\-A\.; Kolobov, A\.; Misra, D\.; Nie, A\.; and Swaminathan, A\. 2023\.Llf\-bench: Benchmark for interactive learning from language feedback\.*arXiv preprint arXiv:2312\.06853*\.
- Cheng, Nie, and Swaminathan \(2024\)Cheng, C\.\-A\.; Nie, A\.; and Swaminathan, A\. 2024\.Trace is the next autodiff: Generative optimization with rich feedback, execution traces, and llms\.*Advances in Neural Information Processing Systems*, 37: 71596–71642\.
- Ding et al\. \(2025\)Ding, Z\.; Hong, J\.; Wang, J\. T\.; Lin, Z\.; Wang, Z\.; and Chen, Y\. 2025\.Scaling Textual Gradients via Sampling\-Based Momentum\.*arXiv preprint arXiv:2506\.00400*\.
- Dor \(2018\)Dor, S\. 2018\.Strategy in games or strategy games: Dictionary and encyclopaedic definitions for game studies\.*Game Studies*, 18\(1\): 43–55\.
- Fu et al\. \(2023\)Fu, J\.; Wang, B\.; Zhang, H\.; Zhang, Z\.; Chen, W\.; and Zheng, N\. 2023\.When and Why Momentum Accelerates SGD: An Empirical Study\.arXiv:2306\.09000\.
- Fu et al\. \(2024\)Fu, J\.; Zhang, X\.; Wang, Y\.; Zeng, W\.; and Zheng, N\. 2024\.Understanding Mobile GUI: From Pixel\-Words to Screen\-Sentences\.*Neurocomputing*, 601: 128200\.
- Fu, Zhu, and Li \(2019\)Fu, J\.; Zhu, X\.; and Li, Y\. 2019\.Recognition of Surface Defects on Steel Sheet Using Transfer Learning\.arXiv:1909\.03258\.
- Guo et al\. \(2024\)Guo, Q\.; Wang, R\.; Guo, J\.; Li, B\.; Song, K\.; Tan, X\.; Liu, G\.; Bian, J\.; and Yang, Y\. 2024\.Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers\.In*The Twelfth International Conference on Learning Representations*\.
- Jahanshahi and Sari \(2018\)Jahanshahi, H\.; and Sari, N\. N\. 2018\.Robot path planning algorithms: a review of theory and experiment\.*arXiv preprint arXiv:1805\.08137*\.
- Kuang et al\. \(2025\)Kuang, Z\.; Rong, R\.; Yuan, Y\.; and Nie, A\. 2025\.Learning Game\-Playing Agents with Generative Code Optimization\.*arXiv preprint arXiv:2508\.19506*\.
- Lange, Imajuku, and Cetin \(2025\)Lange, R\. T\.; Imajuku, Y\.; and Cetin, E\. 2025\.Shinkaevolve: Towards open\-ended and sample\-efficient program evolution\.*arXiv preprint arXiv:2509\.19349*\.
- Li et al\. \(2025\)Li, P\.; Wu, K\.; Fu, J\.; and Zhou, S\. 2025\.REGNav: Room Expert Guided Image\-Goal Navigation\.*Proceedings of the AAAI Conference on Artificial Intelligence*, 39\(5\): 4860–4868\.
- Liu et al\. \(2024\)Liu, F\.; Tong, X\.; Yuan, M\.; Lin, X\.; Luo, F\.; Wang, Z\.; Lu, Z\.; and Zhang, Q\. 2024\.Evolution of heuristics: Towards efficient automatic algorithm design using large language model\.*arXiv preprint arXiv:2401\.02051*\.
- Liu et al\. \(2026a\)Liu, Z\.; Liu, Y\.; Wang, J\.; Liu, J\.; Song, W\.; and Fu, J\. 2026a\.The Essence of Balance for Self\-Improving Agents in Vision\-and\-Language Navigation\.arXiv:2604\.19064\.
- Liu et al\. \(2026b\)Liu, Z\.; Liu, Y\.; Wang, J\.; Liu, J\.; Song, W\.; and Fu, J\. 2026b\.Instruction\-as\-State: Environment\-Guided and State\-Conditioned Semantic Understanding for Embodied Navigation\.arXiv:2604\.18223\.
- Liu et al\. \(2026c\)Liu, Z\.; Liu, Y\.; Wang, J\.; Song, W\.; Liu, J\.; and Fu, J\. 2026c\.Structured Progressive Knowledge Activation for LLM\-Driven Neural Architecture Search\.*arXiv preprint arXiv:2605\.04057*\.
- Madaan et al\. \(2023\)Madaan, A\.; Tandon, N\.; Gupta, P\.; Hallinan, S\.; Gao, L\.; Wiegreffe, S\.; Alon, U\.; Dziri, N\.; Prabhumoye, S\.; Yang, Y\.; et al\. 2023\.Self\-refine: Iterative refinement with self\-feedback\.*Advances in Neural Information Processing Systems*, 36: 46534–46594\.
- Mathematical Association of America \(2023\)Mathematical Association of America\. 2023\.2023 American Invitational Mathematics Examination \(AIME\) Problems and Solutions\.American Mathematics Competitions\.
- Mathematical Association of America \(2024\)Mathematical Association of America\. 2024\.2024 American Invitational Mathematics Examination \(AIME\) Problems and Solutions\.American Mathematics Competitions\.
- Mathematical Association of America \(2025\)Mathematical Association of America\. 2025\.2025 American Invitational Mathematics Examination \(AIME\) Problems and Solutions\.American Mathematics Competitions\.
- Meng et al\. \(2024\)Meng, S\.; Wang, Y\.; Yang, C\.; Peng, N\.; and Chang, K\. 2024\.LLM\-A\*: Large Language Model Enhanced Incremental Heuristic Search on Path Planning\.
- Nie et al\. \(2023\)Nie, A\.; Cheng, C\.\-A\.; Kolobov, A\.; and Swaminathan, A\. 2023\.The Importance of Directional Feedback for LLM\-Based Optimizers\.In*NeurIPS 2023 Workshop on Foundation Models for Decision Making*\.
- Novikov et al\. \(2025\)Novikov, A\.; Vũ, N\.; Eisenberger, M\.; Dupont, E\.; Huang, P\.\-S\.; Wagner, A\. Z\.; Shirobokov, S\.; Kozlovskii, B\.; Ruiz, F\. J\.; Mehrabian, A\.; et al\. 2025\.AlphaEvolve: A coding agent for scientific and algorithmic discovery\.*arXiv preprint arXiv:2506\.13131*\.
- Paszke et al\. \(2019\)Paszke, A\.; Gross, S\.; Massa, F\.; Lerer, A\.; Bradbury, J\.; Chanan, G\.; Killeen, T\.; Lin, Z\.; Gimelshein, N\.; Antiga, L\.; et al\. 2019\.Pytorch: An imperative style, high\-performance deep learning library\.*Advances in neural information processing systems*, 32\.
- Pourcel, Colas, and Oudeyer \(2025\)Pourcel, J\.; Colas, C\.; and Oudeyer, P\.\-Y\. 2025\.Self\-improving language models for evolutionary program synthesis: A case study on ARC\-AGI\.*arXiv preprint arXiv:2507\.14172*\.
- Romera\-Paredes et al\. \(2024\)Romera\-Paredes, B\.; Little, D\.; Torr, P\.; et al\. 2024\.Mathematical discoveries from program search with large language models\.*Nature*, 625\(7999\): 476–483\.
- Shen et al\. \(2023\)Shen, Y\.; Zhou, S\.; Fu, J\.; Wang, R\.; Chen, S\.; and Zheng, N\. 2023\.StructVPR: Distill Structural Knowledge with Weighting Samples for Visual Place Recognition\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 11217–11226\.
- Wang \(2021\)Wang, B\. 2021\.Path planning of mobile robot based on A\* algorithm\.In*2021 IEEE International Conference on Electronic Technology, Communication and Information \(ICETCI\)*, 524–528\. IEEE\.
- Wang et al\. \(2023\)Wang, B\.; Fu, J\.; Zhang, H\.; Zheng, N\.; and Chen, W\. 2023\.Closing the Gap Between the Upper Bound and Lower Bound of Adam’s Iteration Complexity\.In*Advances in Neural Information Processing Systems*, volume 36, 39006–39032\.
- Wang et al\. \(2019\)Wang, H\.; Hao, C\.; Zhang, P\.; Zhang, M\.; Yin, P\.; and Zhang, Y\. 2019\.Path planning of mobile robots based on A\* algorithm and artificial potential field algorithm\.*China mechanical engineering*, 30\(20\): 2489\.
- Xia et al\. \(2024\)Xia, C\. S\.; Deng, Y\.; Dunn, S\.; and Zhang, L\. 2024\.Agentless: Demystifying llm\-based software engineering agents\.*arXiv preprint arXiv:2407\.01489*\.
- Xiao et al\. \(2023\)Xiao, H\.; Wang, P\.; Yu, M\.; and Robbiani, M\. 2023\.Llm a\*: Human in the loop large language models enabled a\* search for robotics\.*arXiv preprint arXiv:2312\.01797*\.
- Yang et al\. \(2023\)Yang, C\.; Wang, X\.; Lu, Y\.; Liu, H\.; Le, Q\. V\.; Zhou, D\.; and Chen, X\. 2023\.Large language models as optimizers\.In*The Twelfth International Conference on Learning Representations*\.
- Yang et al\. \(2024\)Yang, J\.; Jimenez, C\. E\.; Wettig, A\.; Lieret, K\.; Yao, S\.; Narasimhan, K\.; and Press, O\. 2024\.Swe\-agent: Agent\-computer interfaces enable automated software engineering\.*Advances in Neural Information Processing Systems*, 37: 50528–50652\.
- Ye et al\. \(2024\)Ye, H\.; Wang, J\.; Cao, Z\.; Berto, F\.; Hua, C\.; Kim, H\.; Park, J\.; and Song, G\. 2024\.Reevo: Large language models as hyper\-heuristics with reflective evolution\.*Advances in neural information processing systems*, 37: 43571–43608\.
- Yuksekgonul et al\. \(2025\)Yuksekgonul, M\.; Bianchi, F\.; Boen, J\.; Liu, S\.; Lu, P\.; Huang, Z\.; Guestrin, C\.; and Zou, J\. 2025\.Optimizing generative ai by backpropagating language model feedback\.*Nature*, 639\(8055\): 609–616\.
- Zhang et al\. \(2025a\)Zhang, J\.; Xiang, J\.; Yu, Z\.; Teng, F\.; Chen, X\.\-H\.; Chen, J\.; Zhuge, M\.; Cheng, X\.; Hong, S\.; Wang, J\.; et al\. 2025a\.AFlow: Automating Agentic Workflow Generation\.In*The Thirteenth International Conference on Learning Representations*\.
- Zhang et al\. \(2024a\)Zhang, P\.; Jin, H\.; Hu, L\.; Li, X\.; Kang, L\.; Luo, M\.; Song, Y\.; and Wang, H\. 2024a\.HessianGrad: Optimizing AI Systems with Hessian\-Aware Textual Gradients\.ICLR 2025 conference submission\.
- Zhang et al\. \(2024b\)Zhang, P\.; Jin, H\.; Hu, L\.; Li, X\.; Kang, L\.; Luo, M\.; Song, Y\.; and Wang, H\. 2024b\.Revolve: Optimizing ai systems by tracking response evolution in textual optimization\.*arXiv preprint arXiv:2412\.03092*\.
- Zhang et al\. \(2024c\)Zhang, Y\.; Ruan, H\.; Fan, Z\.; and Roychoudhury, A\. 2024c\.Autocoderover: Autonomous program improvement\.In*Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis*, 1592–1604\.
- Zhang et al\. \(2025b\)Zhang, Y\.; Zhang, Y\.; Leach, K\.; and Huang, Y\. 2025b\.CodeGrad: Integrating Multi\-Step Verification with Gradient\-Based LLM Refinement\.*arXiv preprint arXiv:2508\.10059*\.
- Zhou et al\. \(2025\)Zhou, C\.; Shi, Z\.; Yao, Y\.; Liang, L\.; Chen, H\.; and Zhang, Q\. 2025\.RiOT: Efficient Prompt Refinement with Residual Optimization Tree\.*arXiv preprint arXiv:2506\.16389*\.

## Appendix ARelated Works: Discussion on First\-Order Information

The contribution of this paper is best understood against the assumption it questions—that reasoning from ground\-truth performance is necessary for LLM\-based optimization\. We therefore organize related work around*first\-order information*: how it is used, when it genuinely helps, and where it quietly breaks down\.

### A\.1First\-Order Information in LLM Optimizers

Following the convention of numerical optimization, we treat the ground\-truth performance of a solution as zeroth\-order information and any*reasoning*built on that performance as first\-order information\(Yuksekgonul et al\.[2025](https://arxiv.org/html/2607.28947#bib.bib39); Zhang et al\.[2024a](https://arxiv.org/html/2607.28947#bib.bib41)\)\. The idea that LLMs can act as optimizers dates back to early work showing they can iteratively improve solutions when shown past attempts and their scores\(Yang et al\.[2023](https://arxiv.org/html/2607.28947#bib.bib36)\), and first\-order information has since been viewed as the key signal for choosing the next direction\.

Two lines of work operationalize this idea, and it is useful to see that they sit on a spectrum of how*explicitly*they use first\-order information\. The first makes it explicit\. TextGrad\(Yuksekgonul et al\.[2025](https://arxiv.org/html/2607.28947#bib.bib39)\), inspired by the autodiff mechanism of PyTorch\(Paszke et al\.[2019](https://arxiv.org/html/2607.28947#bib.bib27)\), asks the model to analyze its previous outputs against performance feedback and emit improvement suggestions that act as a “textual gradient\.” The second uses it implicitly\. Evolutionary LLM frameworks\(Novikov et al\.[2025](https://arxiv.org/html/2607.28947#bib.bib26); Romera\-Paredes et al\.[2024](https://arxiv.org/html/2607.28947#bib.bib29); Liu et al\.[2024](https://arxiv.org/html/2607.28947#bib.bib16)\)maintain a population of scored candidates and sample parents from it\(Brahmachary et al\.[2024](https://arxiv.org/html/2607.28947#bib.bib3); Pourcel, Colas, and Oudeyer[2025](https://arxiv.org/html/2607.28947#bib.bib28)\), so the optimizer need not fixate on the most recent solution; first\-order reasoning still enters, however, because the model is shown several past solutions*with their scores*and asked to reason across them when proposing new ones\(Guo et al\.[2024](https://arxiv.org/html/2607.28947#bib.bib11); Ye et al\.[2024](https://arxiv.org/html/2607.28947#bib.bib38)\)\. Despite their differences, both lines share a premise: the model should look at performance and reason about it\.

### A\.2Potential and Limitation of First\-Order Information

First\-order information is genuinely useful when the link between a solution and its score is easy to reason about\(Wang et al\.[2023](https://arxiv.org/html/2607.28947#bib.bib32); Fu, Zhu, and Li[2019](https://arxiv.org/html/2607.28947#bib.bib10); Shen et al\.[2023](https://arxiv.org/html/2607.28947#bib.bib30); Fu et al\.[2023](https://arxiv.org/html/2607.28947#bib.bib8),[2024](https://arxiv.org/html/2607.28947#bib.bib9); Li et al\.[2025](https://arxiv.org/html/2607.28947#bib.bib15)\)\. Its core function is to let the model associate a candidate with its performance and adjust accordingly; when that mapping is simple and near\-decomposable, the association is reliable\. In a poem\-generation task\(Cheng et al\.[2023](https://arxiv.org/html/2607.28947#bib.bib4)\), for example, the model can directly see that a line is too long and shorten it\. And in settings where the model has little intrinsic prior about the task\(Yang et al\.[2023](https://arxiv.org/html/2607.28947#bib.bib36)\), first\-order signals may be the*only*source of direction, making them indispensable\.

The picture changes when evaluation is a complex, multi\-stage process—as in multi\-step games\(Kuang et al\.[2025](https://arxiv.org/html/2607.28947#bib.bib13); Bosio and Mueller[2025](https://arxiv.org/html/2607.28947#bib.bib2)\)\. There the final score aggregates many interacting decisions, so the causal path from any single component to the outcome is long and entangled\. Reasoning back from the score to “what to fix” becomes a credit\-assignment problem with no unique answer, and the resulting direction is unreliable\. This is the same failure our analysis in Appendix[C](https://arxiv.org/html/2607.28947#A3)formalizes, and it is, paradoxically, the regime where strong optimization is most needed\.

### A\.3Our Perspective

We draw a different conclusion from these limitations\. Rather than working harder to extract directions from first\-order information, we observe that for many complex tasks the LLM already holds substantial internal knowledge about what makes a solution good, by virtue of understanding the task itself\. When that is true, performance need not be*reasoned about*at all; it need only serve as a scalar criterion for comparing candidates—zeroth\-order information in our terminology\. The remaining obstacle is then not knowing the direction but composing the model’s many partial directions into a coherent improvement, which is exactly the weakest\-link effect that HERO addresses\.

We stress what this perspective does and does not claim\. We do not argue that first\-order information is unimportant or obsolete; in low\-prior tasks it remains essential\. Rather, by showing how far zeroth\-order information alone can go, we aim to surface an underappreciated strength—the knowledge already inside the model—and to prompt a more careful, cost\-aware account of*when*first\-order reasoning is worth its overhead and*how*it should be combined with the model’s prior\.

## Appendix BDesign Details

We instantiate the two phases with concrete choices that preserve the abstractions above while keeping the method strictly zeroth\-order\.

#### Generation\.

We realize the generation query asLLM​\(Task,p\)\\mathrm\{LLM\}\(\\mathrm\{Task\},p\): the model receives the task description and the current programpponly, with the performance ofppwithheld\. Following prior work, the task description carries the essential problem specification plus optional hints and is held*fixed*throughout optimization, so that any change in behavior is attributable to the search rather than to a shifting prompt\. Rather than regenerating the whole program, we adopt theSEARCH/REPLACEedit format ofNovikov et al\. \([2025](https://arxiv.org/html/2607.28947#bib.bib26)\): the model emits severalSEARCH/REPLACEpairs, each a small, localized modification\. We constrain theSEARCHregions to be pairwise non\-overlapping\. This is a simple sufficient condition for the commutativity requirement \([6](https://arxiv.org/html/2607.28947#S2.E6)\)—edits touching disjoint code regions cannot interfere—and it makes the resulting atomic edits genuinely heterogeneous, as the Generation phase demands\.

#### Selection\.

LetEEdenote the evaluator that returns the scalar performance used in place offf\(e\.g\., a test harness or simulator\)\. Selection proceeds in two sequential steps\.\(1\) Error filtering\.We apply each editαi\\alpha\_\{i\}individually and discard any that triggers an execution error, yielding the valid setAne⊆AA\_\{\\mathrm\{ne\}\}\\subseteq A\. This removes obviously broken directions and, crucially, prevents a single invalid edit from contaminating every subset that would contain it\.\(2\) Performance optimization\.OverAneA\_\{\\mathrm\{ne\}\}we solve Eq\. \([7](https://arxiv.org/html/2607.28947#S2.E7)\)\. Exact selection is a subset search whose cost grows exponentially in\|Ane\|\|A\_\{\\mathrm\{ne\}\}\|, so we match the budget to the regime: when\|Ane\|\|A\_\{\\mathrm\{ne\}\}\|is small orEEis cheap, we enumerate all subsets; when\|Ane\|\|A\_\{\\mathrm\{ne\}\}\|is large, we evaluate randomly sampled subsets; and whenEEis expensive, we construct a cheap surrogateE′≈EE^\{\\prime\}\\approx Eand optimize againstE′E^\{\\prime\}\. The surrogate is sound for a specific reason: by Eq\. \([7](https://arxiv.org/html/2607.28947#S2.E7)\), selection depends only on the relative*ranking*of subsets, not on accurate absolute scores, so any coarse but approximately order\-preservingE′E^\{\\prime\}suffices\. We verify this empirically in Section[3\.2](https://arxiv.org/html/2607.28947#S3.SS2)via the evaluator\-grain analysis\.

#### Selection\.

LetEEdenote the scalar evaluator used in place offf\(e\.g\., a test harness or simulator\)\. Selection uses this signal only as a comparison oracle and proceeds in two steps\.\(1\) Error filtering\.We first apply each editαi\\alpha\_\{i\}individually and discard any edit that causes an execution error, obtaining the non\-error setAne⊆AA\_\{\\mathrm\{ne\}\}\\subseteq A\. This prevents a single invalid edit from corrupting every recombined subset that contains it\.\(2\) Budgeted subset ranking\.OverAneA\_\{\\mathrm\{ne\}\}, exact selection would enumerate all subsets as in Eq\. \([7](https://arxiv.org/html/2607.28947#S2.E7)\), whose cost is exponential in\|Ane\|\|A\_\{\\mathrm\{ne\}\}\|\. We therefore solve a budgeted approximation:

S⋆∈arg⁡maxS∈𝒞B​\(Ane\)⁡E^mS​\(AS​\(xt−1\)\),∑S∈𝒞B​\(Ane\)mS≤B\.\\begin\{gathered\}S^\{\\star\}\\in\\arg\\max\_\{S\\in\\mathcal\{C\}\_\{B\}\(A\_\{\\mathrm\{ne\}\}\)\}\\widehat\{E\}\_\{m\_\{S\}\}\\\!\\left\(A\_\{S\}\(x\_\{t\-1\}\)\\right\),\\\\ \\sum\_\{S\\in\\mathcal\{C\}\_\{B\}\(A\_\{\\mathrm\{ne\}\}\)\}m\_\{S\}\\leq B\.\\end\{gathered\}\(8\)where𝒞B​\(Ane\)⊆2Ane\\mathcal\{C\}\_\{B\}\(A\_\{\\mathrm\{ne\}\}\)\\subseteq 2^\{A\_\{\\mathrm\{ne\}\}\}is the subset pool considered under budgetBB,mSm\_\{S\}is the number of evaluator calls assigned to subsetSS, andE^m​\(x\)=1m​∑r=1mE~r​\(x\)\\widehat\{E\}\_\{m\}\(x\)=\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\\widetilde\{E\}\_\{r\}\(x\)denotes the empirical score estimated either with the faithful evaluatorEEor, when evaluation is expensive, with a cheaper surrogateE′≈EE^\{\\prime\}\\approx E\. This formulation captures the practical breadth–reliability trade\-off: the budget can be spent on evaluating more subsets \(larger𝒞B\\mathcal\{C\}\_\{B\}\) or on reranking promising subsets more reliably \(largermSm\_\{S\}\)\. When the budget is sufficient,𝒞B​\(Ane\)=2Ane\\mathcal\{C\}\_\{B\}\(A\_\{\\mathrm\{ne\}\}\)=2^\{A\_\{\\mathrm\{ne\}\}\}and Eq\. \([8](https://arxiv.org/html/2607.28947#A2.E8)\) recovers exact selection; otherwise, we sample a subset pool and optionally allocate additional evaluations to top\-ranked candidates\. Because selection depends only on the relative ranking of subsets, an approximately order\-preserving surrogate is sufficient\. In all cases, the performance signal is used only for subset comparison: it is never provided to the LLM during edit generation and is never used as reasoning context\.

## Appendix CA Analysis of First\-Order Information

This appendix formalizes the claim made in the main text: once the mapping from a candidate solution to its performance is unrestricted, finitely many evaluations cannot determine the performance of an unseen candidate, so any direction inferred by reasoning over scores is unidentifiable\. We proceed from definitions and assumptions to a core lemma, the main impossibility theorem, and two corollaries, and we close with remarks on intuition, tightness, and the implication for first\-order optimization\.

### C\.1Setup and Definitions

Throughout,𝒳\\mathcal\{X\}denotes the space of candidate solutions and𝒴⊆ℝ\+\\mathcal\{Y\}\\subseteq\\mathbb\{R\}^\{\+\}the space of \(performance\) labels, with\|𝒴\|≥2\|\\mathcal\{Y\}\|\\geq 2\. We use\[k\]=\{1,…,k\}\[k\]=\\\{1,\\dots,k\\\}\.

###### Definition 1\(Score–decoder model\)\.

Fix a measurable*latent score*s:𝒳→ℝs:\\mathcal\{X\}\\to\\mathbb\{R\}\. Letℳ\\mathcal\{M\}be the set of all measurable*decoders*ϕ:ℝ→𝒴\\phi:\\mathbb\{R\}\\to\\mathcal\{Y\}\. The induced hypothesis class is

ℱs=\{ϕ∘s:ϕ∈ℳ\}\.\\mathcal\{F\}\_\{s\}\\;=\\;\\\{\\,\\phi\\circ s\\;:\\;\\phi\\in\\mathcal\{M\}\\,\\\}\.The target functionf⋆=ϕ⋆∘s∈ℱsf^\{\\star\}=\\phi^\{\\star\}\\circ s\\in\\mathcal\{F\}\_\{s\}is fixed but unknown; here the latent scoressis common to every hypothesis, and only the decoderϕ⋆\\phi^\{\\star\}is unknown\.

###### Definition 2\(Sample, observed scores, version space\)\.

A size\-kksample isSk=\{\(xi,yi\)\}i=1kS\_\{k\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{k\}withyi=f⋆​\(xi\)y\_\{i\}=f^\{\\star\}\(x\_\{i\}\), and its*observed score set*isZobs=\{s​\(xi\):i∈\[k\]\}⊂ℝZ\_\{\\mathrm\{obs\}\}=\\\{s\(x\_\{i\}\):i\\in\[k\]\\\}\\subset\\mathbb\{R\}\. A decoderϕ∈ℳ\\phi\\in\\mathcal\{M\}is*consistent*withSkS\_\{k\}ifϕ​\(s​\(xi\)\)=yi\\phi\(s\(x\_\{i\}\)\)=y\_\{i\}for alli∈\[k\]i\\in\[k\]\. The*version space*is the set of all consistent decoders,V​\(Sk\)=\{ϕ∈ℳ:ϕ​\(s​\(xi\)\)=yi​for all​i∈\[k\]\}V\(S\_\{k\}\)=\\\{\\phi\\in\\mathcal\{M\}:\\phi\(s\(x\_\{i\}\)\)=y\_\{i\}\\ \\text\{for all \}i\\in\[k\]\\\}\.

We make the following assumptions explicit\. Assumptions \(A1\)–\(A3\) underlie all results below; \(A4\) is invoked only for the probabilistic statement of Corollary[4](https://arxiv.org/html/2607.28947#Thmtheorem4)\.

- \(A1\)Unrestricted decoder\.ℳ\\mathcal\{M\}is the class of*all*measurable mapsℝ→𝒴\\mathbb\{R\}\\to\\mathcal\{Y\}: no smoothness, monotonicity, or parametric restriction is imposed onϕ\\phi\.
- \(A2\)Latent observability\.The scoressis fixed and shared acrossℱs\\mathcal\{F\}\_\{s\}, but its values are never observed directly; the learner observes only labelsy=ϕ⋆​\(s​\(x\)\)y=\\phi^\{\\star\}\(s\(x\)\)\.
- \(A3\)Finite data\.The sample sizekkis finite, soZobsZ\_\{\\mathrm\{obs\}\}is a finite subset ofℝ\\mathbb\{R\}\.
- \(A4\)Non\-atomic query scores\.When the query pointXqX\_\{q\}is random, the law ofs​\(Xq\)s\(X\_\{q\}\)is non\-atomic \(e\.g\. it admits a density\), so thatPr⁡\[s​\(Xq\)=z\]=0\\Pr\[s\(X\_\{q\}\)=z\]=0for every fixedz∈ℝz\\in\\mathbb\{R\}\.

The next lemma isolates the construction on which all subsequent results rest: outside the finitely many observed scores, a measurable decoder may be assigned freely without violating consistency\.

###### Lemma 1\(Free assignment off the observed scores\)\.

Assume\(A1\)–\(A3\)\. Letz1,…,zm∈ℝ∖Zobsz\_\{1\},\\dots,z\_\{m\}\\in\\mathbb\{R\}\\setminus Z\_\{\\mathrm\{obs\}\}be distinct, and letb1,…,bm∈𝒴b\_\{1\},\\dots,b\_\{m\}\\in\\mathcal\{Y\}be arbitrary\. Then there existsϕadv∈V​\(Sk\)\\phi\_\{\\mathrm\{adv\}\}\\in V\(S\_\{k\}\)withϕadv​\(zj\)=bj\\phi\_\{\\mathrm\{adv\}\}\(z\_\{j\}\)=b\_\{j\}for everyj∈\[m\]j\\in\[m\]\.

###### Proof\.

We constructϕadv\\phi\_\{\\mathrm\{adv\}\}directly and verify each required property in turn\.

*\(i\) Consistency on observed scores\.*Forz∈Zobsz\\in Z\_\{\\mathrm\{obs\}\}, defineϕadv​\(z\)=yi\\phi\_\{\\mathrm\{adv\}\}\(z\)=y\_\{i\}for any indexi∈\[k\]i\\in\[k\]withs​\(xi\)=zs\(x\_\{i\}\)=z\. This is well defined: ifs​\(xi\)=s​\(xi′\)s\(x\_\{i\}\)=s\(x\_\{i^\{\\prime\}\}\), then by Definition[1](https://arxiv.org/html/2607.28947#Thmdefinition1)yi=ϕ⋆​\(s​\(xi\)\)=ϕ⋆​\(s​\(xi′\)\)=yi′y\_\{i\}=\\phi^\{\\star\}\(s\(x\_\{i\}\)\)=\\phi^\{\\star\}\(s\(x\_\{i^\{\\prime\}\}\)\)=y\_\{i^\{\\prime\}\}, so the assigned value is independent of the chosen index\. By constructionϕadv​\(s​\(xi\)\)=yi\\phi\_\{\\mathrm\{adv\}\}\(s\(x\_\{i\}\)\)=y\_\{i\}for alli∈\[k\]i\\in\[k\], which is exactly the consistency condition of Definition[2](https://arxiv.org/html/2607.28947#Thmdefinition2)\.

*\(ii\) Prescribing the target values\.*By \(A3\),ZobsZ\_\{\\mathrm\{obs\}\}is finite and hence closed, and the pointsz1,…,zm∈ℝ∖Zobsz\_\{1\},\\dots,z\_\{m\}\\in\\mathbb\{R\}\\setminus Z\_\{\\mathrm\{obs\}\}are distinct\. We may therefore choose pairwise\-disjoint open intervalsI1,…,ImI\_\{1\},\\dots,I\_\{m\}withzj∈Ijz\_\{j\}\\in I\_\{j\}andIj∩Zobs=∅I\_\{j\}\\cap Z\_\{\\mathrm\{obs\}\}=\\emptysetfor everyj∈\[m\]j\\in\[m\]\. Setϕadv​\(z\)=bj\\phi\_\{\\mathrm\{adv\}\}\(z\)=b\_\{j\}for allz∈Ijz\\in I\_\{j\}\.

*\(iii\) Extension and measurability\.*Fix anyy0∈𝒴y\_\{0\}\\in\\mathcal\{Y\}and setϕadv​\(z\)=y0\\phi\_\{\\mathrm\{adv\}\}\(z\)=y\_\{0\}forz∈ℝ∖\(Zobs∪⋃j=1mIj\)z\\in\\mathbb\{R\}\\setminus\\bigl\(Z\_\{\\mathrm\{obs\}\}\\cup\\bigcup\_\{j=1\}^\{m\}I\_\{j\}\\bigr\)\. Thenϕadv\\phi\_\{\\mathrm\{adv\}\}is constant on each of finitely many measurable pieces—a finite point set, finitely many disjoint open intervals, and their complement—hence is a simple function and therefore measurable; by \(A1\),ϕadv∈ℳ\\phi\_\{\\mathrm\{adv\}\}\\in\\mathcal\{M\}\.

By \(i\) the decoder is consistent, soϕadv∈V​\(Sk\)\\phi\_\{\\mathrm\{adv\}\}\\in V\(S\_\{k\}\); by \(ii\) andzj∈Ijz\_\{j\}\\in I\_\{j\}we haveϕadv​\(zj\)=bj\\phi\_\{\\mathrm\{adv\}\}\(z\_\{j\}\)=b\_\{j\}for everyj∈\[m\]j\\in\[m\]\. ∎

### C\.2Impossibility Results

###### Theorem 2\(Pointwise unidentifiability\)\.

Assume\(A1\)–\(A3\)\. Letxq∈𝒳x\_\{q\}\\in\\mathcal\{X\}be a query point with unobserved latent score, i\.e\.s​\(xq\)∉Zobss\(x\_\{q\}\)\\notin Z\_\{\\mathrm\{obs\}\}\. Then for every target labelytarget∈𝒴y\_\{\\mathrm\{target\}\}\\in\\mathcal\{Y\}there exists a hypothesisfadv=ϕadv∘s∈ℱsf\_\{\\mathrm\{adv\}\}=\\phi\_\{\\mathrm\{adv\}\}\\circ s\\in\\mathcal\{F\}\_\{s\}that is consistent withSkS\_\{k\}and satisfiesfadv​\(xq\)=ytargetf\_\{\\mathrm\{adv\}\}\(x\_\{q\}\)=y\_\{\\mathrm\{target\}\}\. Consequentlyf⋆​\(xq\)f^\{\\star\}\(x\_\{q\}\)is not determined bySkS\_\{k\}: no algorithm that observes onlySkS\_\{k\}can exclude any label in𝒴\\mathcal\{Y\}as the value atxqx\_\{q\}\.

###### Proof\.

We argue directly using Lemma[1](https://arxiv.org/html/2607.28947#Thmtheorem1), then conclude by a two\-hypotheses \(indistinguishability\) argument\. Applying Lemma[1](https://arxiv.org/html/2607.28947#Thmtheorem1)with the single pointz1=s​\(xq\)∈ℝ∖Zobsz\_\{1\}=s\(x\_\{q\}\)\\in\\mathbb\{R\}\\setminus Z\_\{\\mathrm\{obs\}\}and valueb1=ytargetb\_\{1\}=y\_\{\\mathrm\{target\}\}yields a decoderϕadv∈V​\(Sk\)\\phi\_\{\\mathrm\{adv\}\}\\in V\(S\_\{k\}\)withϕadv​\(s​\(xq\)\)=ytarget\\phi\_\{\\mathrm\{adv\}\}\(s\(x\_\{q\}\)\)=y\_\{\\mathrm\{target\}\}\. The hypothesisfadv=ϕadv∘sf\_\{\\mathrm\{adv\}\}=\\phi\_\{\\mathrm\{adv\}\}\\circ sthen lies inℱs\\mathcal\{F\}\_\{s\}by Definition[1](https://arxiv.org/html/2607.28947#Thmdefinition1), is consistent withSkS\_\{k\}becauseϕadv∈V​\(Sk\)\\phi\_\{\\mathrm\{adv\}\}\\in V\(S\_\{k\}\)\(Definition[2](https://arxiv.org/html/2607.28947#Thmdefinition2)\), and satisfiesfadv​\(xq\)=ϕadv​\(s​\(xq\)\)=ytargetf\_\{\\mathrm\{adv\}\}\(x\_\{q\}\)=\\phi\_\{\\mathrm\{adv\}\}\(s\(x\_\{q\}\)\)=y\_\{\\mathrm\{target\}\}\.

For the final claim, suppose some algorithm, on inputSkS\_\{k\}, assertedf⋆​\(xq\)≠yf^\{\\star\}\(x\_\{q\}\)\\neq yfor a particulary∈𝒴y\\in\\mathcal\{Y\}\. Instantiating the construction withytarget=yy\_\{\\mathrm\{target\}\}=yproducesfadv∈ℱsf\_\{\\mathrm\{adv\}\}\\in\\mathcal\{F\}\_\{s\}consistent withSkS\_\{k\}and withfadv​\(xq\)=yf\_\{\\mathrm\{adv\}\}\(x\_\{q\}\)=y\. The two hypothesesf⋆f^\{\\star\}andfadvf\_\{\\mathrm\{adv\}\}agree on all ofSkS\_\{k\}and are thus indistinguishable to any procedure whose input isSkS\_\{k\}; sincefadv​\(xq\)=yf\_\{\\mathrm\{adv\}\}\(x\_\{q\}\)=yis admissible, the assertionf⋆​\(xq\)≠yf^\{\\star\}\(x\_\{q\}\)\\neq yis unjustified\. Asy∈𝒴y\\in\\mathcal\{Y\}was arbitrary, no label can be excluded\. ∎

The barrier extends from single labels to*comparisons*, which is the form first\-order reasoning actually relies on\.

###### Corollary 3\(Preference unidentifiability\)\.

Assume\(A1\)–\(A3\)\. Letx′,x′′∈𝒳x^\{\\prime\},x^\{\\prime\\prime\}\\in\\mathcal\{X\}have distinct, unobserved latent scores, i\.e\.s​\(x′\)≠s​\(x′′\)s\(x^\{\\prime\}\)\\neq s\(x^\{\\prime\\prime\}\)ands​\(x′\),s​\(x′′\)∉Zobss\(x^\{\\prime\}\),s\(x^\{\\prime\\prime\}\)\\notin Z\_\{\\mathrm\{obs\}\}\. Then both strict orderings are realizable by consistent hypotheses: there existf1,f2∈ℱsf\_\{1\},f\_\{2\}\\in\\mathcal\{F\}\_\{s\}, each consistent withSkS\_\{k\}, such thatf1​\(x′\)<f1​\(x′′\)f\_\{1\}\(x^\{\\prime\}\)<f\_\{1\}\(x^\{\\prime\\prime\}\)andf2​\(x′\)\>f2​\(x′′\)f\_\{2\}\(x^\{\\prime\}\)\>f\_\{2\}\(x^\{\\prime\\prime\}\)\. Hence no algorithm observing onlySkS\_\{k\}can determine which ofx′,x′′x^\{\\prime\},x^\{\\prime\\prime\}has the larger label\.

###### Proof\.

Since\|𝒴\|≥2\|\\mathcal\{Y\}\|\\geq 2, fixa,b∈𝒴a,b\\in\\mathcal\{Y\}witha<ba<b\. The pointsz1=s​\(x′\)z\_\{1\}=s\(x^\{\\prime\}\)andz2=s​\(x′′\)z\_\{2\}=s\(x^\{\\prime\\prime\}\)are distinct and lie inℝ∖Zobs\\mathbb\{R\}\\setminus Z\_\{\\mathrm\{obs\}\}, so Lemma[1](https://arxiv.org/html/2607.28947#Thmtheorem1)applies withm=2m=2\. Choosing\(b1,b2\)=\(a,b\)\(b\_\{1\},b\_\{2\}\)=\(a,b\)gives a consistent decoder whose hypothesisf1f\_\{1\}satisfiesf1​\(x′\)=a<b=f1​\(x′′\)f\_\{1\}\(x^\{\\prime\}\)=a<b=f\_\{1\}\(x^\{\\prime\\prime\}\); choosing\(b1,b2\)=\(b,a\)\(b\_\{1\},b\_\{2\}\)=\(b,a\)gives a consistentf2f\_\{2\}withf2​\(x′\)=b\>a=f2​\(x′′\)f\_\{2\}\(x^\{\\prime\}\)=b\>a=f\_\{2\}\(x^\{\\prime\\prime\}\)\. Both hypotheses lie inℱs\\mathcal\{F\}\_\{s\}and are consistent withSkS\_\{k\}by Lemma[1](https://arxiv.org/html/2607.28947#Thmtheorem1), so the observed data are compatible with either ordering\. ∎

Theorem[2](https://arxiv.org/html/2607.28947#Thmtheorem2)is stated for a fixed query point satisfying the genericity conditions​\(xq\)∉Zobss\(x\_\{q\}\)\\notin Z\_\{\\mathrm\{obs\}\}\. We now show that, for a randomly drawn query, this condition holds almost surely\.

###### Corollary 4\(Almost\-sure unidentifiability\)\.

Assume\(A1\)–\(A4\)and let the queryXqX\_\{q\}be drawn independently ofSkS\_\{k\}from a distribution whose pushforward underssis non\-atomic\. Thens​\(Xq\)∉Zobss\(X\_\{q\}\)\\notin Z\_\{\\mathrm\{obs\}\}with probability one, and hence the conclusion of Theorem[2](https://arxiv.org/html/2607.28947#Thmtheorem2)holds almost surely\.

###### Proof\.

By \(A3\) the setZobs=\{s​\(x1\),…,s​\(xk\)\}Z\_\{\\mathrm\{obs\}\}=\\\{s\(x\_\{1\}\),\\dots,s\(x\_\{k\}\)\\\}is finite\. By \(A4\) and a union bound,

Pr⁡\[s​\(Xq\)∈Zobs\]≤∑i=1kPr⁡\[s​\(Xq\)=s​\(xi\)\]=0\.\\Pr\\bigl\[s\(X\_\{q\}\)\\in Z\_\{\\mathrm\{obs\}\}\\bigr\]\\;\\leq\\;\\sum\_\{i=1\}^\{k\}\\Pr\\bigl\[s\(X\_\{q\}\)=s\(x\_\{i\}\)\\bigr\]\\;=\\;0\.On the complementary event\{s​\(Xq\)∉Zobs\}\\\{s\(X\_\{q\}\)\\notin Z\_\{\\mathrm\{obs\}\}\\\}, which therefore has probability one, Theorem[2](https://arxiv.org/html/2607.28947#Thmtheorem2)applies verbatim\. ∎

### C\.3Experiments

To further demonstrate that first\-order information is little used in practice, we conduct a controlled experiment\. We use OpenEvolve as the base optimizer and iteratively remove the in\-context examples that carry first\-order signal, observing how performance changes\. We find that removing these examples leaves performance largely unaffected, indicating that the optimizer relies far less on first\-order information than commonly assumed\.

## Appendix DExtra Results

This section reports the full per\-task results underlying the cross\-model study summarized in Section[3\.2](https://arxiv.org/html/2607.28947#S3.SS2)\(Fig\.[4](https://arxiv.org/html/2607.28947#S3.F4)B\)\. To confirm that HERO’s advantage is not an artifact of a single backbone, we repeat the Strategy Games protocol of Section[3\.2](https://arxiv.org/html/2607.28947#S3.SS2)on two additional base models—DeepSeek\-v3\.2 \(Table[2](https://arxiv.org/html/2607.28947#A4.T2)\) and Moonshot\-Kimi\-K2\-Instruct \(Table[3](https://arxiv.org/html/2607.28947#A4.T3)\)—comparing against the two strongest evolutionary baselines, ShinkaEvolve and OpenEvolve, under identical settings \(60 steps on Othello, 20 on Battleship\)\. Together with the Qwen\-Plus results in Table[1](https://arxiv.org/html/2607.28947#S3.T1), these span three independently developed model families\.

Table 2:Per\-task results on Strategy Games using DeepSeek\-v3\.2\. The best result in each row is inbold\.Table 3:Per\-task results on Strategy Games using Moonshot\-Kimi\-K2\-Instruct\. The best result in each row is inbold\.Two observations reinforce the conclusions of the main text\. First, HERO attains the best average win\-rate on both backbones—0\.600\.60versus0\.410\.41\(OpenEvolve\) and0\.380\.38\(ShinkaEvolve\) on DeepSeek\-v3\.2, and0\.610\.61versus0\.440\.44and0\.460\.46on Kimi\-K2\-Instruct—mirroring the0\.610\.61versus0\.360\.36/0\.330\.33margin on Qwen\-Plus \(Table[1](https://arxiv.org/html/2607.28947#S3.T1)\)\. The directional knowledge HERO exploits is therefore broadly present across model families rather than peculiar to one\. Second, the easy–hard pattern recurs within each backbone: on the easy variants \(Othello\-E, Battleship\-E\) the methods are close, and a baseline can occasionally edge ahead \(e\.g\., OpenEvolve reaches0\.960\.96on Othello\-E with DeepSeek\-v3\.2\), whereas on the hard variants the gap widens sharply—on Othello\-H, HERO scores0\.950\.95and0\.930\.93while no baseline exceeds0\.280\.28and0\.650\.65, respectively\. This is the same signature predicted by our analysis and observed on Qwen\-Plus: the benefit of bypassing first\-order reasoning is largest exactly where the score\-to\-performance mapping is hardest to invert, and it holds regardless of the underlying model\.

## Appendix EDetail of Task Design

### E\.1Othello Game

![Refer to caption](https://arxiv.org/html/2607.28947v1/x8.png)Figure 8:Examples of Battleship and Othello Games\.Othello, also known as Reversi, is a two\-player deterministic board game that combines simple operational rules with substantial strategic depth\. The game is played on a square board of fixed size, traditionally an8×88\\times 8grid, and uses a set of identical discs that are colored black on one side and white on the other\. Each disc represents ownership by one of the two players, and ownership may change during the course of the game through a well\-defined flipping mechanism\. The fundamental objective of the game is to finish with a greater number of discs of one’s own color on the board than the opponent\.

At the beginning of the game, the board is initialized with four discs placed at its center\(Figure[8](https://arxiv.org/html/2607.28947#A5.F8)B\)\. Two black discs and two white discs occupy these central squares in a diagonal configuration\. All remaining squares on the board are empty\. One player is assigned the black discs and always moves first, while the other player is assigned the white discs and moves second\. Gameplay proceeds in alternating turns\. On each turn, a player places a disc of their own color onto an empty square of the board, subject to strict legality conditions\. A move is legal if and only if the placement of the disc results in at least one of the opponent’s discs being captured \(the dashed circles in Figure[8](https://arxiv.org/html/2607.28947#A5.F8)are valid movement for the black player\.\)\. Capture occurs when a contiguous line of one or more opponent discs becomes sandwiched between the newly placed disc and another disc of the current player’s color\. This sandwiching must occur along a straight line in one of the eight possible directions defined by the board geometry, namely horizontal, vertical, or diagonal directions\. If no such line exists, the move is illegal and cannot be played\.

If a player has no legal moves available on their turn, they are required to pass\. Passing does not involve placing a disc or altering the board state, and the turn is immediately transferred to the opponent\. The game continues as long as at least one player has a legal move available\. The game terminates when the board is completely filled with discs or when both players are unable to make a legal move consecutively\. These termination conditions ensure that the game always reaches a finite conclusion\. At the end of the game, scoring is performed by counting the number of discs of each color present on the board\. Each disc contributes one point to the owning player’s score\. The player with the higher score is declared the winner\. In the case where both players have the same number of discs, the game ends in a draw\. Notably, victory is determined solely by the final configuration of the board rather than intermediate advantages accrued during play\.

In the Othello game setting, we design two types of opponents, denoted as the “easy” and “hard” agents, each implementing distinct strategies with different levels of tactical sophistication\. The LLM is tasked with developing new strategies that can consistently outperform these opponents\. This setup encourages the LLM not only to discover effective local tactics but also to synthesize higher\-level strategic patterns that generalize across different styles of adversaries\.

### E\.2Battleship

This game presents the agent with a two\-dimensional grid map on which a fixed fleet of ships is secretly deployed \(Figure[8](https://arxiv.org/html/2607.28947#A5.F8)A\)\. The agent has no prior knowledge of the ships’ exact positions, their shapes, or how many squares each ship occupies\. On every turn the agent selects a single cell to “shoot\.” Immediately after the shot is fired the environment returns a reward signal: a positive value if a ship was hit and zero otherwise\. No other information—such as the identity of the struck ship, its orientation, or how many of its remaining segments are still afloat—is revealed\.

The agent’s task is to infer, shot by shot, the most probable locations of the hidden vessels and to maximize the total number of hits \(or minimize the number of shots required to sink every ship\)\. Because the only feedback is the sparse binary reward, the agent must maintain an internal belief state—typically a probability distribution over all possible ship configurations—and update that belief using the history of past shots and their outcomes\. Efficient exploration, information gain, and probabilistic reasoning are therefore central to strong performance\. Performance is evaluated as the ratio of successful hits to the total number of shots taken\.