A Hybrid Nested Harness for Decoupling Structure and Parameters in LLM-Driven Optimization

# A Hybrid Nested Harness for Decoupling Structure and Parameters in LLM-Driven Optimization
Source: [https://arxiv.org/html/2608.08156](https://arxiv.org/html/2608.08156)
###### Abstract

In evolutionary algorithms powered by language models, the LLM acts as a single operator that simultaneously updates structural components \(like control flow\) and continuous parameters\. While LLMs can be good at the first, they are not efficient at the second, wasting tokens taking discrete jumps inside a trial and error loop\. We resolve this by formalizing a hybrid nested search, in which an outer loop has the LLM propose a structural sketch, with numeric gaps, and an inner numerical optimizer tunes the sketch\. Both the outer and inner solvers are pluggable: any text\-based optimizer can be combined with a zero\-order optimizer \(CMA\-ES\), gradient\-based routines, or MCMC samplers\. We validate our framework across three scientific domains: \(i\) meta\-optimizers on closed\-form test functions, \(ii\) code\-based policies for systems research and social dilemmas; and \(iii\) approximate Bayesian inference tasks\. Across all three, the hybrid optimizer is superior to both vanilla LLM\-driven search and pure numerical optimization baselines\. Code at:[github\.com/vicgalle/hybrid\-nested\-search](https://github.com/vicgalle/hybrid-nested-search)

## 1Introduction

We adopt the setting in which a frozen LLMℳ\\mathcal\{M\}acts as a synthesis operator inside a loop\. At each iteration the model generates a code\-based artifact \(a policy function, a probabilistic\-model reparameterization, a training script, a GPU kernel, etc\.\), a harness evaluates it under a fitness \(scoring\) functionff, and structured feedback is prompted to the next generation\. This is the architecture behind recent LLM\-based discovery systems in different scientific domains\(Romera\-Paredes et al\.,[2024](https://arxiv.org/html/2608.08156#bib.bib25); Novikov et al\.,[2025](https://arxiv.org/html/2608.08156#bib.bib23)\)\. Vanilla autoresearch letsℳ\\mathcal\{M\}propose the entire artifact at once in a single optimization turn\.

We argue that an artifact \(the optimization solution\) can be decomposed into two qualitatively different decision types\. Thestructure\(sketch\) is the program topology: control structures like branches, hard logic, auxiliary function definitions\. This is text\-based and benefits from the model’s knowledge prior\. Theparameter vectoris the numeric values embedded in that structure: thresholds, learning rates, etc\. This is a continuous \(possibly mixed\-integer\) black\-box optimization problem\. Language models are poor optimizers\(Yang et al\.,[2023](https://arxiv.org/html/2608.08156#bib.bib33)\), wasting huge amounts of compute resources whereas*traditional*numerical solvers are dramatically more efficient\. Our idea is thus to formalize this dual structure as a nested search loop, and to propose a practical algorithm that can leverage any text\-based optimizer, any numerical optimizer, and combine them by*sketching*\(see Figure[1](https://arxiv.org/html/2608.08156#S2.F1)\)\.

Our contributions are these: \(i\) we formalize hybrid nested search as a bilevel objective and give an algorithm in which a frozen LLM searches structure and an inner solver tunes parameters \(Sec\.[2](https://arxiv.org/html/2608.08156#S2)\); \(ii\) we identify*parametric aliasing*as the failure mode of joint search and prove that the inner loop de\-aliases the outer ranking \(Sec\.[2\.3](https://arxiv.org/html/2608.08156#S2.SS3)\); \(iii\) we validate this factorization on three problem families: meta\-optimizers \(Sec\.[3\.1](https://arxiv.org/html/2608.08156#S3.SS1)\), code\-based policies for systems research and social dilemmas \(Sec\.[3\.2](https://arxiv.org/html/2608.08156#S3.SS2)\), and approximate Bayesian inference tasks \(Sec\.[3\.3](https://arxiv.org/html/2608.08156#S3.SS3)\)\. The result holds across inner solvers: CMA\-ES\(Hansen & Ostermeier,[2001](https://arxiv.org/html/2608.08156#bib.bib9)\), VI\(Kucukelbir et al\.,[2017](https://arxiv.org/html/2608.08156#bib.bib15)\), NUTS\(Hoffman & Gelman,[2014](https://arxiv.org/html/2608.08156#bib.bib11)\), outer structural optimizers \(a simple LLM loop and reflective prompt evolution\(Agrawal et al\.,[2025](https://arxiv.org/html/2608.08156#bib.bib1)\)\), and different language model families\.

## 2Hybrid Nested Search

Frozen LLMℳ\\mathcal\{M\}propose structureτ\\tau\+ manifestdef policy\(env, p\):if w\(env\) \>\[θ1\\theta\_\{1\}\]:clean\(\[θ2\\theta\_\{2\}\]\*N\)manifest:\(ℓj,uj,typej,θ¯j\)\(\\ell\_\{j\},\\,u\_\{j\},\\,\\mathrm\{type\}\_\{j\},\\,\\bar\{\\theta\}\_\{j\}\)Inner optimizerTunefill the holes:θ¯→θ⋆​\(τ\)\\bar\{\\theta\}\\\!\\to\\\!\\theta^\{\\star\}\(\\tau\)θ¯\\bar\{\\theta\}θ⋆\\theta^\{\\star\}Score & promoteeval on thetunedvalueF^​\(τ\)≈maxθ⁡f​\(τ,θ\)\\widehat\{F\}\(\\tau\)\\approx\\displaystyle\\max\_\{\\theta\}f\(\\tau,\\theta\)\(1\+1\)\(1\\\!\+\\\!1\): keepτ\\tauif betterthan current incumbentτ\\tau, manifestθ⋆,F^​\(τ\)\\theta^\{\\star\},\\ \\widehat\{F\}\(\\tau\)outer loop • feedback:F^​\(τk−1\),θ⋆,\\widehat\{F\}\(\\tau\_\{k\-1\}\),\\ \\theta^\{\\star\},diagnosticspluggable inner solverCMA\-ESVINUTSDe\-aliasing\.Vanilla joint search ranks structures by the untuned guessf​\(τ,θ¯\)f\(\\tau,\\bar\{\\theta\}\); hybrid ranks them by the parametric optimumF^​\(τ\)\\widehat\{F\}\(\\tau\)\. The gapΔ​\(τ\)=F​\(τ\)−f​\(τ,θ¯\)\\Delta\(\\tau\)=F\(\\tau\)\-f\(\\tau,\\bar\{\\theta\}\)correlates with the advantage of hybrid vs\. vanilla joint search\.

Figure 1:Hybrid nested search\.A frozen LLM \(left\) proposes a text sketchτ\\tauwith numeric gaps and a manifest of bounds, types, and a suggested valueθ¯\\bar\{\\theta\}\. An inner numerical optimizer \(center\) tunes the guessθ¯\\bar\{\\theta\}to the optimumθ⋆​\(τ\)\\theta^\{\\star\}\(\\tau\), and returnsF^​\(τ\)≈maxθ⁡f​\(τ,θ\)\\widehat\{F\}\(\\tau\)\\approx\\max\_\{\\theta\}f\(\\tau,\\theta\)\. The artifact is scored and promoted on that tuned value \(right\), and feedback is provided for the outer loop’s next iteration\. Our framework is agnostic to the choice of inner and outer solvers\. Evaluating on the tuned value in place of the LLM guess removes the parametric aliasing of joint search\.Our framework is displayed in Figure[1](https://arxiv.org/html/2608.08156#S2.F1): a frozen LLM proposes a structural sketch with numeric holes, an inner optimizer fills the holes to their parametric optimum, and the structure is scored on that tuned result before the next proposal\. Note this nested factorization is agnostic to the choice of inner and outer solvers, and we leverage text\-based outer optimizers to learn from structured feedback \(not only the score, but also compiler diagnostics\)\.

### 2\.1Decomposition and the parameter manifest

###### Definition 1\(Sketch decomposition\)\.

A candidate program is a pair⟨τ,θ⟩\\langle\\tau,\\theta\\ranglewhereτ∈𝒯\\tau\\in\\mathcal\{T\}is a structure drawn from a discrete sketch space \(expressed via text\), exposingh​\(τ\)∈ℕh\(\\tau\)\\in\\mathbb\{N\}numeric*holes*, andθ∈Θ​\(τ\)⊆ℝh​\(τ\)\\theta\\in\\Theta\(\\tau\)\\subseteq\\mathbb\{R\}^\{h\(\\tau\)\}is a parameter vector\. An instantiation mapInst:\(τ,θ\)↦\\textsc\{Inst\}:\(\\tau,\\theta\)\\mapstoexecutable code fills the holes with actual values\. The parameter spaceΘ​\(τ\)\\Theta\(\\tau\)depends onτ\\tau, so while the outer loop searches over structures, the number of parameters can vary\.

Rather than parsing numeric literals out of generated code, we requireℳ\\mathcal\{M\}to output, for each gapjj, a*manifest*: a tuplem​\(τ\)=\{\(j,ℓj,uj,typej,θ¯j\)\}j=1h​\(τ\)m\(\\tau\)=\\\{\(j,\\ \\ell\_\{j\},\\ u\_\{j\},\\ \\mathrm\{type\}\_\{j\},\\ \\bar\{\\theta\}\_\{j\}\)\\\}\_\{j=1\}^\{h\(\\tau\)\}, giving a lower boundℓj\\ell\_\{j\}, upper bounduju\_\{j\}, a typetypej∈\{cont,int,log,pow2\}\\mathrm\{type\}\_\{j\}\\in\\\{\\mathrm\{cont\},\\mathrm\{int\},\\mathrm\{log\},\\mathrm\{pow2\}\\\}, and a suggested valueθ¯j\\bar\{\\theta\}\_\{j\}, the LLM’s own best guess\. The inner optimizer takes the vectorθ¯\\bar\{\\theta\}as a warm start, at each outer iteration\. The manifest is a lightweight interface that allows the LLM to delegate the numeric search to a specialized solver, making this choice explicit compared to other numeric values that do not need to be tuned \(e\.g\., dimension sizes\)\.

### 2\.2Bilevel objective and algorithm

Letf​\(τ,θ\)f\(\\tau,\\theta\)be the fitness function of the problem \(welfare for a policy, sampling efficiency for an inference task, cost savings for a systems algorithm, …\)\. Define the*structural value*F​\(τ\)F\(\\tau\)as the optimized fitness for a fixed structureτ\\tau,

F​\(τ\)=maxθ∈Θ​\(τ\)⁡f​\(τ,θ\),θ⋆​\(τ\)=arg⁡maxθ∈Θ​\(τ\)⁡f​\(τ,θ\)\.F\(\\tau\)\\;=\\;\\max\_\{\\theta\\in\\Theta\(\\tau\)\}f\(\\tau,\\theta\),\\qquad\\theta^\{\\star\}\(\\tau\)\\;=\\;\\arg\\max\_\{\\theta\\in\\Theta\(\\tau\)\}f\(\\tau,\\theta\)\.\(1\)Hence, we are interested in searching for the structure that maximizes its structural value:

τ⋆=arg⁡maxτ∈𝒯⁡F​\(τ\)=arg⁡maxτ∈𝒯⁡maxθ∈Θ​\(τ\)⁡f​\(τ,θ\)\.\\tau^\{\\star\}\\;=\\;\\arg\\max\_\{\\tau\\in\\mathcal\{T\}\}F\(\\tau\)\\;=\\;\\arg\\max\_\{\\tau\\in\\mathcal\{T\}\}\\ \\max\_\{\\theta\\in\\Theta\(\\tau\)\}f\(\\tau,\\theta\)\.\(2\)This is a bilevel optimization problem, over two different scales\. A numerical optimizer solves the inner level \([1](https://arxiv.org/html/2608.08156#S2.E1)\); the LLM searches the outer level \([2](https://arxiv.org/html/2608.08156#S2.E2)\)\. The inner solver returns an estimate of \([1](https://arxiv.org/html/2608.08156#S2.E1)\) within a budgetBinB\_\{\\mathrm\{in\}\}of fitness evaluations,F^​\(τ\)=Tune​\(f​\(τ,⋅\),Θ​\(τ\),θ¯​\(τ\),Bin\)\\widehat\{F\}\(\\tau\)=\\textsc\{Tune\}\\big\(f\(\\tau,\\cdot\),\\Theta\(\\tau\),\\bar\{\\theta\}\(\\tau\),B\_\{\\mathrm\{in\}\}\\big\), usingθ¯​\(τ\)\\bar\{\\theta\}\(\\tau\)as the warm start\.

Algorithm 1Hybrid nested structural / parametric search \(\(1\+1\) outer loop\)1:task

TT, LLM

ℳ\\mathcal\{M\}, system prompt

pp, outer budget

KK, inner budget

BinB\_\{\\mathrm\{in\}\}
2:incumbent structure

τ⋆\\tau^\{\\star\}, parameters

θ⋆\\theta^\{\\star\}, value

V⋆V^\{\\star\}
3:

\(τ0,m0\)←ℳ​\(p,“propose initial structure \+ manifest”\)\(\\tau\_\{0\},m\_\{0\}\)\\leftarrow\\mathcal\{M\}\(p,\\text\{\`\`propose initial structure \+ manifest''\}\)
4:

F^0,θ0⋆←Tune​\(f​\(τ0,⋅\),Θ​\(τ0\),θ¯​\(τ0\),Bin\)\\widehat\{F\}\_\{0\},\\theta^\{\\star\}\_\{0\}\\leftarrow\\textsc\{Tune\}\(f\(\\tau\_\{0\},\\cdot\),\\Theta\(\\tau\_\{0\}\),\\bar\{\\theta\}\(\\tau\_\{0\}\),B\_\{\\mathrm\{in\}\}\)
5:

τ⋆,θ⋆,V⋆←τ0,θ0⋆,F^0\\tau^\{\\star\},\\theta^\{\\star\},V^\{\\star\}\\leftarrow\\tau\_\{0\},\\theta^\{\\star\}\_\{0\},\\widehat\{F\}\_\{0\}
6:for

k=1,…,Kk=1,\\dots,Kdo

7:

\(τk,mk\)←ℳ​\(p,q​\(τ⋆,V⋆,θ⋆,diag\)\)\(\\tau\_\{k\},m\_\{k\}\)\\leftarrow\\mathcal\{M\}\(p,\\ q\(\\tau^\{\\star\},V^\{\\star\},\\theta^\{\\star\},\\mathrm\{diag\}\)\)⊳\\trianglerightLLM mutates structure

8:if

¬Validate​\(τk\)\\neg\\textsc\{Validate\}\(\\tau\_\{k\}\)thencontinue⊳\\trianglerightcheck sketch is correct

9:endif

10:

F^k,θk⋆←Tune​\(f​\(τk,⋅\),Θ​\(τk\),θ¯​\(τk\),Bin\)\\widehat\{F\}\_\{k\},\\theta^\{\\star\}\_\{k\}\\leftarrow\\textsc\{Tune\}\(f\(\\tau\_\{k\},\\cdot\),\\Theta\(\\tau\_\{k\}\),\\bar\{\\theta\}\(\\tau\_\{k\}\),B\_\{\\mathrm\{in\}\}\)⊳\\trianglerightinner tune

11:if

F^k\>V⋆\\widehat\{F\}\_\{k\}\>V^\{\\star\}then

12:

τ⋆,θ⋆,V⋆←τk,θk⋆,F^k\\tau^\{\\star\},\\theta^\{\\star\},V^\{\\star\}\\leftarrow\\tau\_\{k\},\\theta^\{\\star\}\_\{k\},\\widehat\{F\}\_\{k\}⊳\\trianglerightpromote on tuned value

13:endif

14:endfor

15:return

τ⋆,θ⋆,V⋆\\tau^\{\\star\},\\theta^\{\\star\},V^\{\\star\}

#### Inner loop: a pluggable parametric optimizer\.

Tuneis a routine agnostic to the LLM and chosen to refine the inner fitness\. For a non differentiable black\-box objective, we use CMA\-ES withθ¯​\(τ\)\\bar\{\\theta\}\(\\tau\)as the initial guess, projecting integer and power\-of\-2 coordinates onto their feasible lattice\. For differentiable magnitudes, the natural choice is a gradient\-based routine; and these can be replaced with MCMC methods in case we are interested in optimizing a distribution rather than a single point\. The factorization holds whichever solver fills the holes in the sketch; only the solver changes\. The fitness function should be a robust score \(e\.g\. using the mean across several samples\), so the inner loop does not overfit\.

#### Outer loop: LLM structural proposal\.

At outer stepkkthe LLM proposes a new structureτk\\tau\_\{k\}conditioned on the previous structureτk−1\\tau\_\{k\-1\}, its optimized valueF^​\(τk−1\)\\widehat\{F\}\(\\tau\_\{k\-1\}\), the optimized parameters, and other domain diagnostics, provided as text feedback\. With this feedback, we can expose the LLM to theF^​\(τk−1\)\\widehat\{F\}\(\\tau\_\{k\-1\}\)estimate, the structure’s potential at its parametric best, instead of just the score of the model’s original guess\. That is, the LLM is informed with the ground truth signal from the inner optimizer, so it is better informed for the next iteration\. The accept rule may be the usual \(1\+1\) rule \(keep only the best incumbent, Algorithm[1](https://arxiv.org/html/2608.08156#alg1)\) or population\-based; our results also hold when a more sophisticated method, such as GEPA\(Agrawal et al\.,[2025](https://arxiv.org/html/2608.08156#bib.bib1)\), optimizes the outer loop \(Sec\.[3\.2](https://arxiv.org/html/2608.08156#S3.SS2)\)\.

### 2\.3Parametric de\-aliasing

Recent work has shown that, within text\-based structural optimizers, providing the LLM operator with multiple signals can break the aliasing of different failure modes that collapse to the same scalar reward\(Cheng et al\.,[2024](https://arxiv.org/html/2608.08156#bib.bib3); Agrawal et al\.,[2025](https://arxiv.org/html/2608.08156#bib.bib1); Gallego,[2026](https://arxiv.org/html/2608.08156#bib.bib6)\)\. We generalize this aliasing notion to the hybrid setting, in which the aliased quantities are the structural quality and the quality of the parameter guesses\. Vanilla, pure text\-based search observes the LLM’s untuned scoregvan​\(τ\)=f​\(τ,θ¯​\(τ\)\)g\_\{\\mathrm\{van\}\}\(\\tau\)=f\(\\tau,\\bar\{\\theta\}\(\\tau\)\)\. Hybrid search observes insteadghyb​\(τ\)=F^​\(τ\)≈F​\(τ\)g\_\{\\mathrm\{hyb\}\}\(\\tau\)=\\widehat\{F\}\(\\tau\)\\approx F\(\\tau\)\. Let’s define the*parameter\-tuning gap*as

Δ​\(τ\)=F​\(τ\)−f​\(τ,θ¯​\(τ\)\)≥0,\\Delta\(\\tau\)\\;=\\;F\(\\tau\)\-f\(\\tau,\\bar\{\\theta\}\(\\tau\)\)\\;\\geq\\;0,\(3\)that is, the fitness a structure leaves on the table under the LLM’s guessed parameters\.

###### Proposition 1\(Parametric aliasing in LLM\-driven search\)\.

Assume the outer operator’s accept decision is monotone in the observed score \(a structure is preferred to the incumbent iff its observed score is higher\), and that the inner solver isε\\varepsilon\-accurate,\|F^​\(τ\)−F​\(τ\)\|≤ε\|\\widehat\{F\}\(\\tau\)\-F\(\\tau\)\|\\leq\\varepsilon\. Then: \(1\) vanilla and hybrid induce the same preference betweenτ,τ′\\tau,\\tau^\{\\prime\}iffΔ​\(τ\)−Δ​\(τ′\)\\Delta\(\\tau\)\-\\Delta\(\\tau^\{\\prime\}\)does not flip the sign ofF​\(τ\)−F​\(τ′\)F\(\\tau\)\-F\(\\tau^\{\\prime\}\); \(2\) vanilla can reject the structural optimumτ⋆\\tau^\{\\star\}in favor of an inferiorτ′\\tau^\{\\prime\}wheneverΔ​\(τ⋆\)−Δ​\(τ′\)\>F​\(τ⋆\)−F​\(τ′\)\\Delta\(\\tau^\{\\star\}\)\-\\Delta\(\\tau^\{\\prime\}\)\>F\(\\tau^\{\\star\}\)\-F\(\\tau^\{\\prime\}\), while hybrid never commits this error up to a margin2​ε2\\varepsilon; \(3\) under an i\.i\.d\. proposal model withF⟂ΔF\\perp\\Delta\(i\.e\., they are independent\), the expected hybrid advantage decomposes into a*selection*term, positive on the inversions of \(1\) and increasing in the cross\-structure variance ofΔ\\Delta, and a*tuning*term, present on every accepted structure and increasing in the mean𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]; it vanishes iffΔ≡0\\Delta\\equiv 0\.

We place the proof in Appendix[A](https://arxiv.org/html/2608.08156#A1)\. When the LLM’s guesses for parameters are already near optimal \(Δ≈0\\Delta\\approx 0, e\.g\. a policy that doesn’t require numerical parameters\) the two search methods coincide\. When good structures carry constants the model cannot guess \(learning rates, preconditioner scales, etc\.\), vanilla search discards them prematurely, while the inner loop from hybrid search can de\-alias the signal\. We arrive at a falsifiable prediction:

\(hybrid advantage\)∝𝔼τ∼ℳ​\[Δ​\(τ\)\]\.\(\\text\{hybrid advantage\}\)\\ \\propto\\ \\mathbb\{E\}\_\{\\tau\\sim\\mathcal\{M\}\}\\big\[\\Delta\(\\tau\)\\big\]\.\(4\)Here the*hybrid advantage*ismaxτ⁡F^​\(τ\)−maxτ⁡gvan​\(τ\)\\max\_\{\\tau\}\\widehat\{F\}\(\\tau\)\-\\max\_\{\\tau\}g\_\{\\mathrm\{van\}\}\(\\tau\), the best delivered value of each search method \(hybrid vs vanilla\)\. Throughout our experiments we will measureΔ​\(τ\)\\Delta\(\\tau\)empirically, and see that the hybrid advantage grows with it\.

### 2\.4On the cost of hybrid search

Letcllmc\_\{\\mathrm\{llm\}\}be the cost of one LLM proposal \(to generate a structureτ\\tau\), and letcevc\_\{\\mathrm\{ev\}\}be the cost of one evaluation of the fitness functionf​\(τ,θ\)f\(\\tau,\\theta\)\. OverKKouter optimization steps, the total cost of hybrid search isChyb=K​\(cllm\+Bin​cev\)C\_\{\\mathrm\{hyb\}\}=K\(c\_\{\\mathrm\{llm\}\}\+B\_\{\\mathrm\{in\}\}c\_\{\\mathrm\{ev\}\}\), againstCvan=K​cllmC\_\{\\mathrm\{van\}\}=Kc\_\{\\mathrm\{llm\}\}as the cost for vanilla, LLM\-driven search\. That is, the hybrid search costs a factor\(1\+cev/cllm​Bin\)\(1\+c\_\{\\mathrm\{ev\}\}/c\_\{\\mathrm\{llm\}\}B\_\{\\mathrm\{in\}\}\)more\. In practice, a rollout is on the order of seconds, while a high thinking budget LLM call is on the order of minutes, socev/cllm∼10−2c\_\{\\mathrm\{ev\}\}/c\_\{\\mathrm\{llm\}\}\\sim 10^\{\-2\}–10−310^\{\-3\}\. With an inner budgetBin∼100B\_\{\\mathrm\{in\}\}\\sim 100, this adds only1010–100%100\\%overhead while exhausting each expensive LLM proposal\. For the experiments here, the inner cost is negligible compared to the LLM\.

A stronger inner optimizer of the fitness score is also a stronger overfitter of it\. We therefore \(i\) make the fitness function an aggregate, so it cannot trade robustness for a single point win, and \(ii\) keep a held\-out evaluation function, evaluated once on the tuned incumbent at a sample never folded into feedback\. We exercise this design choice with simulation\-based calibration in Sec\.[3\.3](https://arxiv.org/html/2608.08156#S3.SS3)\.

## 3Experiments

We evaluate our harness over three problem families\. Each is a lightweight sketch interface on top of an existing evolutionary harness, so it is straightforward to implement hybrid search from a vanilla LLM\-based process\. Throughout the domains we compare three approaches:vanilla autoresearch\(LLM joint search, no inner loop\),pure numerical\(the inner solver over a fixed structure, no structural search\), and ourhybrid\(Algorithm[1](https://arxiv.org/html/2608.08156#alg1)\)\. Unless noted, the inner budget isBin=100B\_\{\\mathrm\{in\}\}=100and the objective function is withheld from the LLM: its prompts show only the artifact interface and a structural description, so theΔ\\Deltagap is the LLM’s genuine miscalculation measured empirically\. Table[1](https://arxiv.org/html/2608.08156#S3.T1)summarizes one representative task across all three families, the subsections below expand the analysis, and Appendix[D](https://arxiv.org/html/2608.08156#A4)shows the entire list of results\.

Table 1:Representative resultsacross the three problem families \(full results list in Appendix[D](https://arxiv.org/html/2608.08156#A4)\)\. Three search strategies:vanilla\(LLM joint search\),numerical\-only \(numerical solver on a fixed structure: an*oracle*CMA\-ES for meta\-optimizers, or a hand\-tuned baseline elsewhere\), and ourhybrid\. Inboldwe mark the better of vanilla/hybrid; “–” an strategy not run for that case\.𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]is the empirical tuning gap\.Shadingof𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]and adv\. columns is a heatmap, darker==larger: the two columns sharing a gradient is a visual check that advantage∝𝔼​\[Δ\]\\propto\\mathbb\{E\}\[\\Delta\]\(Eq\.[4](https://arxiv.org/html/2608.08156#S2.E4)\)\. Advantage units: loss difference van−\-hyb \(meta\-optimizers\); % cost or welfare achieved \(policies\); nats or decades ESS/grad \(Bayes\)\.### 3\.1Meta\-optimizers

The goal is to write the optimization algorithm \(the structure: momentum, adaptive gradients, restarts, etc\) for a hidden 2\-D objective, while CMA\-ES tunes that algorithm’s hyperparameters\. The LLM only sees a black\-box gradient oracle and a starting point, and never the function nor its name\. We use five objectives spanning different hard optimization regimes \(rosenbrock, an ill\-conditionedellipsoid, multimodalrastrigin,ackley, and the deceptiveschwefel, whose gradient points away from the global optimum\); we test three different LLMs, andN=3N=3proposals per strategy\.

![Refer to caption](https://arxiv.org/html/2608.08156v1/x1.png)Figure 2:Meta\-optimizers \(Sec\.[3\.1](https://arxiv.org/html/2608.08156#S3.SS1)\)\.\(a\) Delivered loss per function, vanilla \(∘\\circ\)→\\tohybrid \(⋆\\star\), for three proposer models \(lower is better, log scale\)\. Hybrid moves the delivered loss by up to twelve orders of magnitude onackleyandrosenbrock\. \(b\) Best\-so\-far convergence; the LLM\-proposed\-and\-tuned structure \(bold\) beats CMA\-ES applied directly to the objective \(gray\) on the multimodalrastrigin/ackley, where plain CMA\-ES has no globally\-aware structure\.#### Avoiding the aliasing trap\.

A momentum or adaptive optimizer at default parameters overshoots and diverges: the LLM’s untuned guess scores∞\\inftyor plateaus, so pure text\-based search concludes the structure is bad and discards it\. Hybrid hands the proposed structure to CMA\-ES, which finds a stable configuration: acrossackley,rastrigin, androsenbrockhybrid search improves vanilla by up to many orders of magnitude and even beats the oracle CMA\-ES on the multimodal cases \(Table[1](https://arxiv.org/html/2608.08156#S3.T1); full result list in TableLABEL:tab:master, block A\)\. The two search strategies tie onellipsoid\(the adaptive proposal already sits near the basin at its default\),schwefelis the hardest, and while no method achieves the global optimum, hybrid search is the best performer again\. See Figure[2](https://arxiv.org/html/2608.08156#S3.F2)for the convergence curves\. As can be seen, hybrid search tipycally achieves lower loss than the oracle CMA\-ES with much less function evaluations\.

### 3\.2Executable policies for systems and social dilemmas

We move to code\-based artifacts evaluated by simulators, where the objective is no longer closed\-form\. We use three benchmark tasks with the same three strategies, each minimizing economic cost or maximizing utility\.

#### Cloud infrastructure algorithms\.

Two algorithms from a cloud\-systems research benchmark\.*Can’t Be Late*\(Wu et al\.,[2024](https://arxiv.org/html/2608.08156#bib.bib32)\)is a spot/on\-demand scheduler: the LLM proposes, at every step, how the state \(remaining work, slack to deadline, restart overhead\) maps to an action, and CMA\-ES tunes the slack buffers and pressure thresholds it exposes; cost is dollars under a hard deadline\.*Cloudcast*\(Wooders et al\.,[2024](https://arxiv.org/html/2608.08156#bib.bib31)\)is a multi\-cloud broadcast router: the LLM proposes the routing topology \(shared relay trees, hub routing,kk\-shortest multipath\) and CMA\-ES tunes parameters like weights tolerances\. To save space we report the highest difficulty regime of each \(costly restarts and inter\-cloud routing\); hybrid search wins in every regime evaluated \(Figure[3](https://arxiv.org/html/2608.08156#A2.F3)a,b; Appendix[B\.2](https://arxiv.org/html/2608.08156#A2.SS2)\)\. Our harness delivers the lowest cost for both proposer LLMs\. On inter\-cloud Cloudcast, hybrid \($169, Gemini\) beats joint LLM\-driven search \($213\) and pure CMA\-ES \($317\), a1\.9×1\.9\\timesimprovement over pure CMA\-ES that tunes a hand designed topology\.

#### Sequential social dilemma \(the Cleanup game\)\.

A multi\-agent team policy for the Cleanup public goods gridworld\(Hughes et al\.,[2018](https://arxiv.org/html/2608.08156#bib.bib13)\), an instance of a sequential social dilemma\(Leibo et al\.,[2017](https://arxiv.org/html/2608.08156#bib.bib18)\): there is a tradeoff between collecting apples \(individual rewards\) and contributing to cleaning the river \(collective welfare\)\. The LLM generates a policy at each round, that maps environment variables \(like pollution level\) to the role of each agent\. These policies expose parameters that CMA\-ES tunes; fitness is utilitarian welfareUU, evaluated by self\-play\. We run three seeds of increasing difficulty and aggregate over three proposer models\. Hybrid search achieves the highest welfare in 9/9 model×\\timesseed runs \(Figure[4](https://arxiv.org/html/2608.08156#A2.F4), Appendix[B\.3](https://arxiv.org/html/2608.08156#A2.SS3)\), and its advantage over pure CMA\-ES grows monotonically with difficulty: structural discovery matters more as the hidden dynamics get harsher\. Two findings are of interest\. First, a common ceiling: all three models independently converge on the same algorithmic family \(deadzone\+\+ramp\+\+saturation cap\), optimized to the same welfare \(heavy=0\.582=0\.582for all three\)\. Hybrid search lifts policies onto a ceiling none reaches without tuning\. Second, a clear Proposition[1](https://arxiv.org/html/2608.08156#Thmproposition1)application: on heavy \(Gemini\), a piecewise ramp policy is the worst untuned proposal \(U=−0\.10U=\-0\.10\) yet the best tuned one \(U=0\.582U=0\.582\); vanilla search discards this, promotes a sigmoid policy, and peaks atU=0\.480U=0\.480, leaving∼\\sim21% to aliasing\.

#### Orthogonality to the outer optimizer\.

Our hybrid factorization is agnostic to which structural optimizer is adopted\. We re\-ran all three tasks with loop of Algorithm[1](https://arxiv.org/html/2608.08156#alg1)replaced by a reflective, Pareto\-frontier prompt evolution method \(GEPA,Agrawal et al\. \([2025](https://arxiv.org/html/2608.08156#bib.bib1)\)\) as the outer operator, keeping CMA\-ES as the innerTune\. The qualitative result prevails: hybrid search beats tuner\-free GEPA in every regime with a non\-negligibleΔ\\Delta\(6/8 regimes, the two lowest gap regimes tie, and de\-aliasing attributes those ties to structure selection rather than to the absence of a tuner\)\. A stronger text\-based optimizer recovers more of the gain on its own, so the marginal value of the inner loop shrinks from∼\\sim10–20% \(1\+1 evolutionary loop\) to∼\\sim5–10% \(GEPA\)\. Yet the hybrid parameterization advantage never reverses where tuning matters \(Appendix[B\.4](https://arxiv.org/html/2608.08156#A2.SS4)\)\.

### 3\.3Approximate Bayesian inference

This instantiation is the most distant test of whether the factorization concerns*variable types*or a specific solver\. The artifact is a*reparameterization*: a diffeomorphismθ=reparam​\(z;holes\)\\theta=\\mathrm\{reparam\}\(z;\\,\\text\{holes\}\),z∼ℝDz\\sim\\mathbb\{R\}^\{D\}, under which the host runs inference inzz\-space with the change\-of\-variables Jacobian taken by autodiff \(so the LLM writes*arbitrary*differentiable code instead of picking from a menu\)\. The structureτ\\tauis which latents to transform and the bijector family \(non\-centering, affine, nonlinear warps\); the holes are the transform constants \(scales, interpolation fractions\)\. The innerTuneis no longer CMA\-ES: it is variational inference \(Adam on the ELBO of a fixed standard\-normal guide, where the transform carries all location and scale\) or NUTS itself, whose warmup mass\-matrix adaptation serves as the inner tuner \(untuned==unit metric, tuned==adapted metric\)\. The held\-out gate is a longer multi\-chain run scoring divergences,R^\\hat\{R\}, and effective sample size\.

#### An MCMC\-specific reading of Eq\. \([4](https://arxiv.org/html/2608.08156#S2.E4)\)\.

NUTS warmup already adapts a*linear*preconditioner \(the mass matrix\), so affine reparameterizations are redundant with it \(Δ≈0\\Delta\\approx 0, a tie under NUTS\-inner\) while nonlinear ones \(a funnel’s non\-centering\) are not fixable by any mass matrix \(genuineΔ\\Delta, hybrid wins\)\. This predicts a VI\-vs\-NUTS*asymmetry on the affine controls*: an ill\-conditioned Gaussian ties under NUTS yet is a large VI gain, since a fixed standard guide cannot rescale without the transform\. TableLABEL:tab:master\(blocks F–G\) confirms both halves on a four\-model suite—under VI the affine Gaussians win \(\+37\.7/\+9\.3\+37\.7/\+9\.3nats\) yet tie under NUTS, while under NUTS the nonlinear funnel and eight\-schools win \(\+3\.1/\+2\.2\+3\.1/\+2\.2decades\) yet the affine controls tie\.

#### Reference\-free correctness, and surplus on a hard model\.

A practitioner has no gold posterior, so we replace the moment oracle with a*reference\-free*gate: simulation\-based calibration \(SBC\)\(Talts et al\.,[2018](https://arxiv.org/html/2608.08156#bib.bib27)\)\. For a fixed reparam, repeatedly drawθ⋆∼\\theta^\{\\star\}\\simprior,y∼p\(⋅∣θ⋆\)y\\sim p\(\\cdot\\mid\\theta^\{\\star\}\), sample the posterior inzz\-space, and rankθ⋆\\theta^\{\\star\}among the draws; under a correct sampler\-plus\-transform the ranks are uniform, and deviation \(a Bonferroni\-corrected per\-parameter KS statistic\) flags an invalid transform*or*a sampler that did not reach the posterior, both meaning “do not trust this\.” On a sticky horseshoe\(Carvalho et al\.,[2010](https://arxiv.org/html/2608.08156#bib.bib2)\)logistic regression \(D=18D=18\) where identity\-plus\-adaptation suffers 520 divergences \(ESS/grad6\.2×10−46\.2\\times 10^\{\-4\}\), every one of five proposer models found an SBC\-certified reparam giving≥10×\\geq\\\!10\\timesESS/grad \(up to24×24\\times\), with0/250/25live proposals SBC\-rejected \(block H of TableLABEL:tab:master\)\. The gate earns its keep on a constructed trap,θ=C​tanh⁡\(z\)\\theta=C\\tanh\(z\): a smooth, non\-singular bijection that samples easily but targets a*truncated, wrong*posterior\. SBC rejects it while ESS,R^\\hat\{R\}, and the Jacobian smoke\-test all green\-light it\.

#### The cleanest de\-aliasing in the experiments\.

Our hardest target is a curved\-ridge “banana” stack \(D=10D=10\) whose optimal transform is a nonlinear quadratic*shear*b←b−c​a2b\\leftarrow b\-c\\,a^\{2\}, outside the non\-centering/affine idiom set, with the curvaturecca single hidden nonlinear hole\. Given only the generative structure \(curvature and widths withheld\), all three proposer models*constructed the shear*\. Vanilla autoresearch proposes the same shear at a*guessed*curvature \(ELBO≈−74\\approx\-74\); hybrid tunes the one hidden constant to ELBO−13\-13\(block I of TableLABEL:tab:master\)\. The\+60\+60\-nat win comes from the inner loop tuning a single hidden parameter on an identical structure\. The affine case has a complementary asymmetry: NUTS’s linear metric cannot tune a nonlinear curvature, but once VI findscc, NUTS samples the de\-curved space at\+1\.57\+1\.57decades \(∼\\sim37×37\\times\), a NeuTra\-style\(Hoffman et al\.,[2019](https://arxiv.org/html/2608.08156#bib.bib10)\)pipeline in which the two inner solvers compose\.

### 3\.4Analysis: when is the hybrid factorization worthwhile?

Hybrid nested search cannot be an universal optimizer, by the No Free Lunch theorem\(Wolpert & Macready,[1997](https://arxiv.org/html/2608.08156#bib.bib30); Wolpert,[1996](https://arxiv.org/html/2608.08156#bib.bib29); Schumacher et al\.,[2001](https://arxiv.org/html/2608.08156#bib.bib26); Igel & Toussaint,[2004](https://arxiv.org/html/2608.08156#bib.bib14); Droste et al\.,[2002](https://arxiv.org/html/2608.08156#bib.bib4)\): every search algorithm ties when averaged over all objectives, so any claim of superiority is a claim about a particular class of objective functions the algorithm biases toward\. Three features put our method outside the strict black\-box regime: the outer operator is*informed*\(it is conditioned on rich, structured feedback, and a pretraining prior, not only queried values\); theτ×θ\\tau\\times\\thetafactorization acts as a reparameterization of the problem space; and the cost asymmetry \(Sec\.[2\.4](https://arxiv.org/html/2608.08156#S2.SS4)\) and the tuned\-vs\-untuned signal avoid NFL’s homogeneousquery accounting\. The factorization thus pays off on problems that are near decomposable \(a goodτ\\tauleaves a well\-behaved inner problem\), carry structural prior knowledge \(a near optimalτ\\tauresembles the training corpus\), expose a non\-negligible parametric gap𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]with holes that the inner solver can exploit, and are cheap to evaluate; we predict hybrid search merely ties when these fail: the low parametric gap cases we observe \(ellipsoid, intra\-cloud routing, the deceptiveschwefel\)\. The advantage vs\.𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]trend \(e\.g\. Figure[3](https://arxiv.org/html/2608.08156#A2.F3)c\) plots where this prior places its mass\.

## 4Related work

#### LLM\-driven program and algorithm discovery\.

Placing a frozen LLM in an evolutionary loop to synthesize programs has driven discoveries in mathematics and algorithms\(Romera\-Paredes et al\.,[2024](https://arxiv.org/html/2608.08156#bib.bib25); Novikov et al\.,[2025](https://arxiv.org/html/2608.08156#bib.bib23)\), with a wave of recent systems pushing sample efficiency\(Lange et al\.,[2025](https://arxiv.org/html/2608.08156#bib.bib17)\), recursive self\-improvement\(Zhang et al\.,[2025](https://arxiv.org/html/2608.08156#bib.bib35)\), automated agent design\(Hu et al\.,[2024](https://arxiv.org/html/2608.08156#bib.bib12)\), LLM\-driven heuristic evolution\(Liu et al\.,[2024a](https://arxiv.org/html/2608.08156#bib.bib19)\), and end\-to\-end autoresearch\(Lu et al\.,[2024](https://arxiv.org/html/2608.08156#bib.bib21)\)\. All treat the model as a single operator that mutates program structure and embedded numeric constants jointly in one text turn\. We instead factor that operator by variable type; our orthogonality result \(Sec\.[3\.2](https://arxiv.org/html/2608.08156#S3.SS2)\) shows the inner numerical tuner improves either a\(1\+1\)\(1\{\+\}1\)loop or a reflective prompt\-evolution outer loop\(Agrawal et al\.,[2025](https://arxiv.org/html/2608.08156#bib.bib1)\), and the sample\-efficiency goal ofLange et al\. \([2025](https://arxiv.org/html/2608.08156#bib.bib17)\)is precisely what our budget argument \(Sec\.[2\.4](https://arxiv.org/html/2608.08156#S2.SS4)\) secures, by exhausting each expensive proposal with a cheap inner solver\.

#### LLMs as optimizers, and prompt optimization\.

A parallel line casts the model itself as the optimizer over numeric or structured spaces\(Yang et al\.,[2023](https://arxiv.org/html/2608.08156#bib.bib33)\), as evolution strategies\(Lange et al\.,[2024](https://arxiv.org/html/2608.08156#bib.bib16)\), to steer Bayesian optimization\(Liu et al\.,[2024b](https://arxiv.org/html/2608.08156#bib.bib20)\), or to search reward code\(Ma et al\.,[2023](https://arxiv.org/html/2608.08156#bib.bib22)\), asking the LLM to do the continuous search it is weakest at; our de\-aliasing analysis \(Sec\.[2\.3](https://arxiv.org/html/2608.08156#S2.SS3)\) instead hands that subproblem to a matched numerical solver\. Reflective and “textual\-gradient” prompt methods\(Agrawal et al\.,[2025](https://arxiv.org/html/2608.08156#bib.bib1); Yuksekgonul et al\.,[2024](https://arxiv.org/html/2608.08156#bib.bib34); Guo et al\.,[2023](https://arxiv.org/html/2608.08156#bib.bib8)\)are, in our framework, candidate outer operators rather than competitors: we adopt one \(GEPA\) and show the inner tuner is orthogonal to and composes with it, closing the numeric\-constant gap none of them addresses\.

#### Bilevel optimization, reparameterization, and automatic inference\.

The target Eq\. \([2](https://arxiv.org/html/2608.08156#S2.E2)\) is a bilevel program, studied for hyperparameter optimization and meta\-learning\(Franceschi et al\.,[2018](https://arxiv.org/html/2608.08156#bib.bib5)\)and solved at the inner level by zero\-order search\(Hansen & Ostermeier,[2001](https://arxiv.org/html/2608.08156#bib.bib9)\); our contribution is the assignment of its two levels to operators matched by variable type\. On the inference side, non\-centering\(Papaspiliopoulos et al\.,[2007](https://arxiv.org/html/2608.08156#bib.bib24)\), normalizing\-flow transports\(Hoffman et al\.,[2019](https://arxiv.org/html/2608.08156#bib.bib10)\), and gradient\-based VI\(Kucukelbir et al\.,[2017](https://arxiv.org/html/2608.08156#bib.bib15)\)are the target class of Sec\.[3\.3](https://arxiv.org/html/2608.08156#S3.SS3); closest is automatic reparameterization\(Gorinova et al\.,[2020](https://arxiv.org/html/2608.08156#bib.bib7)\), whereas we cast the transform as an LLM structural search whose constants an inner solver tunes, certified reference\-free by SBC\(Talts et al\.,[2018](https://arxiv.org/html/2608.08156#bib.bib27)\)\.

## 5Conclusion

Factoring an LLM synthesis operator by variable type \(structure to the model, parameters to a nested numerical solver\) removes the parametric aliasing that makes joint search discard good structures with default parameters\. The advantage is predictable: it scales with the mean tuning gap and vanishes when the model already guesses well\. Our harness demonstrates strong results across the inner solver \(CMA\-ES, VI, NUTS\), the outer structural optimizer, and the proposer LLMs, across three different scientific domains\. See Appendix[C](https://arxiv.org/html/2608.08156#A3)for a gallery of discovered code artifacts\.

#### Limitations\.

The manifest must be parseable, and mixed\-integer holes need more specialized solvers\. The generalist CMA\-ES degrades past a few dozen dimensions, soh​\(τ\)h\(\\tau\)must be capped \(high\-dimensional holes call for a neural inner operator, left for further work\)\. A novelty ceiling remains, the method is best used as a human\-in\-the\-loop copilot rather than an autonomous frontier system\.

## References

- Agrawal et al\. \(2025\)L\. Agrawal et al\.GEPA: reflective prompt evolution can outperform reinforcement learning\.*arXiv:2507\.19457*, 2025\.
- Carvalho et al\. \(2010\)C\. M\. Carvalho, N\. G\. Polson, and J\. G\. Scott\.The horseshoe estimator for sparse signals\.*Biometrika*, 97\(2\):465–480, 2010\.
- Cheng et al\. \(2024\)C\.\-A\. Cheng, A\. Nie, and A\. Swaminathan\.Trace is the next AutoDiff: generative optimization with rich feedback, execution traces, and LLMs\.*arXiv:2406\.16218*, 2024\.
- Droste et al\. \(2002\)S\. Droste, T\. Jansen, and I\. Wegener\.Optimization with randomized search heuristics: the \(A\)NFL theorem, realistic scenarios, and difficult functions\.*Theoretical Computer Science*, 287\(1\):131–144, 2002\.
- Franceschi et al\. \(2018\)L\. Franceschi, P\. Frasconi, S\. Salzo, R\. Grazzi, and M\. Pontil\.Bilevel programming for hyperparameter optimization and meta\-learning\.In*ICML*, 2018\.
- Gallego \(2026\)V\. Gallego\.Beyond scalar rewards: dense feedback for LLM policy synthesis in sequential social dilemmas\.*arXiv:2603\.19453*, 2026\.
- Gorinova et al\. \(2020\)M\. I\. Gorinova, D\. Moore, and M\. D\. Hoffman\.Automatic reparameterisation of probabilistic programs\.In*ICML*, 2020\.
- Guo et al\. \(2023\)Q\. Guo, R\. Wang, J\. Guo, B\. Li, et al\.Connecting large language models with evolutionary algorithms yields powerful prompt optimizers\.*arXiv:2309\.08532*, 2023\.
- Hansen & Ostermeier \(2001\)N\. Hansen and A\. Ostermeier\.Completely derandomized self\-adaptation in evolution strategies\.*Evolutionary Computation*, 9\(2\):159–195, 2001\.
- Hoffman et al\. \(2019\)M\. Hoffman, P\. Sountsov, J\. V\. Dillon, I\. Langmore, D\. Tran, and S\. Vasudevan\.NeuTra\-lizing bad geometry in Hamiltonian Monte Carlo using neural transport\.*arXiv:1903\.03704*, 2019\.
- Hoffman & Gelman \(2014\)M\. D\. Hoffman and A\. Gelman\.The No\-U\-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo\.*Journal of Machine Learning Research*, 15:1593–1623, 2014\.
- Hu et al\. \(2024\)S\. Hu, C\. Lu, and J\. Clune\.Automated design of agentic systems\.*arXiv:2408\.08435*, 2024\.
- Hughes et al\. \(2018\)E\. Hughes, J\. Z\. Leibo, M\. Phillips, K\. Tuyls, et al\.Inequity aversion improves cooperation in intertemporal social dilemmas\.In*NeurIPS*, 2018\.
- Igel & Toussaint \(2004\)C\. Igel and M\. Toussaint\.A no\-free\-lunch theorem for non\-uniform distributions of target functions\.*Journal of Mathematical Modelling and Algorithms*, 3\(4\):313–322, 2004\.
- Kucukelbir et al\. \(2017\)A\. Kucukelbir, D\. Tran, R\. Ranganath, A\. Gelman, and D\. M\. Blei\.Automatic differentiation variational inference\.*Journal of Machine Learning Research*, 18:1–45, 2017\.
- Lange et al\. \(2024\)R\. T\. Lange, Y\. Tian, and Y\. Tang\.Large language models as evolution strategies\.*arXiv:2402\.18381*, 2024\.
- Lange et al\. \(2025\)R\. T\. Lange, Y\. Imajuku, and E\. Cetin\.ShinkaEvolve: towards open\-ended and sample\-efficient program evolution\.*arXiv:2509\.19349*, 2025\.
- Leibo et al\. \(2017\)J\. Z\. Leibo, V\. Zambaldi, M\. Lanctot, J\. Marecki, and T\. Graepel\.Multi\-agent reinforcement learning in sequential social dilemmas\.In*AAMAS*, 2017\.
- Liu et al\. \(2024a\)F\. Liu, X\. Tong, M\. Yuan, X\. Lin, et al\.Evolution of heuristics: towards efficient automatic algorithm design using large language models\.*arXiv:2401\.02051*, 2024a\.
- Liu et al\. \(2024b\)T\. Liu, N\. Astorga, N\. Seedat, and M\. van der Schaar\.Large language models to enhance Bayesian optimization\.*arXiv:2402\.03921*, 2024b\.
- Lu et al\. \(2024\)C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, et al\.The AI Scientist: towards fully automated open\-ended scientific discovery\.*arXiv:2408\.06292*, 2024\.
- Ma et al\. \(2023\)Y\. J\. Ma, W\. Liang, G\. Wang, D\.\-A\. Huang, et al\.Eureka: human\-level reward design via coding large language models\.*arXiv:2310\.12931*, 2023\.
- Novikov et al\. \(2025\)A\. Novikov et al\.AlphaEvolve: a coding agent for scientific and algorithmic discovery\.Technical report, 2025\.
- Papaspiliopoulos et al\. \(2007\)O\. Papaspiliopoulos, G\. O\. Roberts, and M\. Sköld\.A general framework for the parametrization of hierarchical models\.*Statistical Science*, 22\(1\):59–73, 2007\.
- Romera\-Paredes et al\. \(2024\)B\. Romera\-Paredes et al\.Mathematical discoveries from program search with large language models\.*Nature*, 625:468–475, 2024\.
- Schumacher et al\. \(2001\)C\. Schumacher, M\. D\. Vose, and L\. D\. Whitley\.The no free lunch and problem description length\.In*GECCO*, pp\. 565–570, 2001\.
- Talts et al\. \(2018\)S\. Talts, M\. Betancourt, D\. Simpson, A\. Vehtari, and A\. Gelman\.Validating Bayesian inference algorithms with simulation\-based calibration\.*arXiv:1804\.06788*, 2018\.
- Vehtari et al\. \(2021\)A\. Vehtari, A\. Gelman, D\. Simpson, B\. Carpenter, and P\.\-C\. Bürkner\.Rank\-normalization, folding, and localization: an improvedR^\\widehat\{R\}for assessing convergence of MCMC\.*Bayesian Analysis*, 16\(2\):667–718, 2021\.
- Wolpert \(1996\)D\. H\. Wolpert\.The lack of a priori distinctions between learning algorithms\.*Neural Computation*, 8\(7\):1341–1390, 1996\.
- Wolpert & Macready \(1997\)D\. H\. Wolpert and W\. G\. Macready\.No free lunch theorems for optimization\.*IEEE Transactions on Evolutionary Computation*, 1\(1\):67–82, 1997\.
- Wooders et al\. \(2024\)S\. Wooders, S\. Liu, P\. Jain, X\. Mo, J\. E\. Gonzalez, V\. Liu, and I\. Stoica\.Cloudcast: high\-throughput, cost\-aware overlay multicast in the cloud\.In*NSDI*, 2024\.
- Wu et al\. \(2024\)Z\. Wu, W\.\-L\. Chiang, Z\. Mao, Z\. Yang, E\. Friedman, S\. Shenker, and I\. Stoica\.Can’t be late: optimizing spot instance savings under deadlines\.In*NSDI*, 2024\.
- Yang et al\. \(2023\)C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. Chen\.Large language models as optimizers\.*arXiv:2309\.03409*, 2023\.
- Yuksekgonul et al\. \(2024\)M\. Yuksekgonul, F\. Bianchi, J\. Boen, S\. Liu, et al\.TextGrad: automatic “differentiation” via text\.*arXiv:2406\.07496*, 2024\.
- Zhang et al\. \(2025\)J\. Zhang, S\. Hu, C\. Lu, R\. Lange, and J\. Clune\.Darwin Gödel Machine: open\-ended evolution of self\-improving agents\.*arXiv:2505\.22954*, 2025\.

## Appendix AProof of Proposition[1](https://arxiv.org/html/2608.08156#Thmproposition1)

We recall the setup of §[2\.3](https://arxiv.org/html/2608.08156#S2.SS3): the outer operator accepts by a rule monotone in the observed score, the inner solver isε\\varepsilon\-accurate \(\|F^​\(τ\)−F​\(τ\)\|≤ε\|\\widehat\{F\}\(\\tau\)\-F\(\\tau\)\|\\leq\\varepsilon\), andΔ​\(τ\)=F​\(τ\)−f​\(τ,θ¯​\(τ\)\)≥0\\Delta\(\\tau\)=F\(\\tau\)\-f\(\\tau,\\bar\{\\theta\}\(\\tau\)\)\\geq 0\.

###### Proof\.

Writegvan​\(τ\)=f​\(τ,θ¯​\(τ\)\)=F​\(τ\)−Δ​\(τ\)g\_\{\\mathrm\{van\}\}\(\\tau\)=f\(\\tau,\\bar\{\\theta\}\(\\tau\)\)=F\(\\tau\)\-\\Delta\(\\tau\)andghyb​\(τ\)=F^​\(τ\)∈\[F​\(τ\)−ε,F​\(τ\)\+ε\]g\_\{\\mathrm\{hyb\}\}\(\\tau\)=\\widehat\{F\}\(\\tau\)\\in\[F\(\\tau\)\-\\varepsilon,F\(\\tau\)\+\\varepsilon\]\. By monotone acceptance, every preference is a comparison of these scores\.

*\(1\)*Fixτ,τ′\\tau,\\tau^\{\\prime\}and seta=F​\(τ\)−F​\(τ′\)a=F\(\\tau\)\-F\(\\tau^\{\\prime\}\),d=Δ​\(τ\)−Δ​\(τ′\)d=\\Delta\(\\tau\)\-\\Delta\(\\tau^\{\\prime\}\)\. The vanilla score difference isgvan​\(τ\)−gvan​\(τ′\)=a−dg\_\{\\mathrm\{van\}\}\(\\tau\)\-g\_\{\\mathrm\{van\}\}\(\\tau^\{\\prime\}\)=a\-d; when\|a\|\>2​ε\|a\|\>2\\varepsilonthe hybrid differenceghyb​\(τ\)−ghyb​\(τ′\)g\_\{\\mathrm\{hyb\}\}\(\\tau\)\-g\_\{\\mathrm\{hyb\}\}\(\\tau^\{\\prime\}\)has the sign ofaa\. The two agree iffsign⁡\(a−d\)=sign⁡\(a\)\\operatorname\{sign\}\(a\-d\)=\\operatorname\{sign\}\(a\), i\.e\. iff subtractingΔ​\(τ\)−Δ​\(τ′\)\\Delta\(\\tau\)\-\\Delta\(\\tau^\{\\prime\}\)does not flip the sign ofF​\(τ\)−F​\(τ′\)F\(\\tau\)\-F\(\\tau^\{\\prime\}\)\. They disagree \(an*inversion*\) iffsign⁡\(d\)=sign⁡\(a\)\\operatorname\{sign\}\(d\)=\\operatorname\{sign\}\(a\)and\|d\|\>\|a\|\|d\|\>\|a\|\.

*\(2\)*Letτ⋆\\tau^\{\\star\}be a structural optimum, soa=F​\(τ⋆\)−F​\(τ′\)≥0a=F\(\\tau^\{\\star\}\)\-F\(\\tau^\{\\prime\}\)\\geq 0\. Vanilla prefers the inferiorτ′\\tau^\{\\prime\}iffgvan​\(τ′\)\>gvan​\(τ⋆\)g\_\{\\mathrm\{van\}\}\(\\tau^\{\\prime\}\)\>g\_\{\\mathrm\{van\}\}\(\\tau^\{\\star\}\), i\.e\. iffΔ​\(τ⋆\)−Δ​\(τ′\)\>F​\(τ⋆\)−F​\(τ′\)\\Delta\(\\tau^\{\\star\}\)\-\\Delta\(\\tau^\{\\prime\}\)\>F\(\\tau^\{\\star\}\)\-F\(\\tau^\{\\prime\}\), the stated condition\. For hybrid,ghyb​\(τ′\)−ghyb​\(τ⋆\)≤2​ε−a≤0g\_\{\\mathrm\{hyb\}\}\(\\tau^\{\\prime\}\)\-g\_\{\\mathrm\{hyb\}\}\(\\tau^\{\\star\}\)\\leq 2\\varepsilon\-a\\leq 0whenevera\>2​εa\>2\\varepsilon, so hybrid never prefers a structure more than2​ε2\\varepsilonworse\.

*\(3\)*Model proposals as i\.i\.d\.τ1,…,τN∼ℳ\\tau\_\{1\},\\dots,\\tau\_\{N\}\\sim\\mathcal\{M\}withF⟂ΔF\\perp\\Delta\(the structural ceiling and the guess gap independent underℳ\\mathcal\{M\}\),μ=𝔼​\[Δ\]\\mu=\\mathbb\{E\}\[\\Delta\],σ2=Var​\(Δ\)\\sigma^\{2\}=\\mathrm\{Var\}\(\\Delta\)\. Vanilla deploysτ^v=arg⁡maxi⁡gvan​\(τi\)\\hat\{\\tau\}\_\{v\}=\\arg\\max\_\{i\}g\_\{\\mathrm\{van\}\}\(\\tau\_\{i\}\)at valueF​\(τ^v\)−Δ​\(τ^v\)F\(\\hat\{\\tau\}\_\{v\}\)\-\\Delta\(\\hat\{\\tau\}\_\{v\}\); hybrid deploysarg⁡maxi⁡F^​\(τi\)\\arg\\max\_\{i\}\\widehat\{F\}\(\\tau\_\{i\}\)at value≥maxi⁡F​\(τi\)−ε\\geq\\max\_\{i\}F\(\\tau\_\{i\}\)\-\\varepsilon\. Hence

AN=\(maxi⁡F​\(τi\)−F​\(τ^v\)\)\+Δ​\(τ^v\)−ε,A\_\{N\}=\\big\(\\textstyle\\max\_\{i\}F\(\\tau\_\{i\}\)\-F\(\\hat\{\\tau\}\_\{v\}\)\\big\)\+\\Delta\(\\hat\{\\tau\}\_\{v\}\)\-\\varepsilon,a sum of two non\-negative terms \(up toε\\varepsilon\)\. The*tuning*termΔ​\(τ^v\)≥0\\Delta\(\\hat\{\\tau\}\_\{v\}\)\\geq 0is deployed on every accepted structure regardless of any inversion; it is stochastically increasing in the scale ofΔ\\Deltaand vanishes iffΔ≡0\\Delta\\equiv 0, so its expectation increases withμ\\mu\. The*selection*term is positive exactly on the inversions of \(1\); writingd=σ​d0d=\\sigma d\_\{0\}for standardized symmetricd0d\_\{0\}independent ofaa,Pr⁡\[inversion∣a\]=12​Pr⁡\[\|d0\|\>\|a\|/σ\]\\Pr\[\\text\{inversion\}\\mid a\]=\\tfrac\{1\}\{2\}\\Pr\[\|d\_\{0\}\|\>\|a\|/\\sigma\]is non\-decreasing inσ\\sigma, as is the expected structural regret𝔼​\[maxi⁡F​\(τi\)−F​\(τ^v\)\]\\mathbb\{E\}\[\\max\_\{i\}F\(\\tau\_\{i\}\)\-F\(\\hat\{\\tau\}\_\{v\}\)\]\. Thus𝔼​\[AN\]\\mathbb\{E\}\[A\_\{N\}\]grows with both the mean and the cross\-structure variance ofΔ\\Delta, and equals zero whenΔ≡0\\Delta\\equiv 0\. \(The independenceF⟂ΔF\\perp\\Deltais a simplifying assumption taken for a clean two\-term decomposition; allowing dependence—better structures systematically carrying larger or smaller gaps—couples the terms but leaves the qualitative claim intact, namely that𝔼​\[AN\]\\mathbb\{E\}\[A\_\{N\}\]increases in𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]andVar​\(Δ\)\\mathrm\{Var\}\(\\Delta\), which is what the experiments test\.\) ∎

## Appendix BAdditional results

This appendix collects the per\-family configuration, interpretation, and figures behind the main\-text results; every numerical cell—all proposer models and regimes—is consolidated in the comprehensive TableLABEL:tab:master\(Appendix[D](https://arxiv.org/html/2608.08156#A4)\), to which each subsection points\.

### B\.1Meta\-optimizers

Configuration:N=3N=3proposals per arm, inner CMA\-ES budgetBin=100B\_\{\\mathrm\{in\}\}=100, direct\-CMA\-ES reference budget20002000\. “Delivered” vanilla loss is the minimum untuned loss over the arm’s proposals; delivered hybrid loss is the minimum tuned loss\. The full grid—all five objectives and three proposers, with the oracle Direct CMA\-ES reference—is block A of TableLABEL:tab:master\.

On the convexellipsoid, both arms reach machine\-precision basins \(the𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]figure conflates “bad guess” with “ambitious high\-ceiling proposal,” so it can be large even where both arms solve the problem\); the practical difference is moot\. Onschwefelthe global basin is far from the start and the gradient is deceptive: GLM and Gemini hybrid improve over vanilla and beat direct CMA\-ES, but Opus ties \(both arms trapped in the same local minimum\), and no arm reaches the global optimum\.

### B\.2Cloud\-systems policies

Configuration:N=3N=3proposals per arm,Bin=100B\_\{\\mathrm\{in\}\}=100, pure\-CMA\-ES budget120120\. Delivered cost is the minimum over each arm’s proposals \(dollars; lower is better\); pure CMA\-ES is model\-independent \(it tunes a fixed seed structure\)\. Can’t Be Late \(all three regimes\) is block B of TableLABEL:tab:masterand Cloudcast block C, where intra\-cloud is the𝔼​\[Δ\]≈0\\mathbb\{E\}\[\\Delta\]\\\!\\approx\\\!0tie control \(cost\-only shortest paths are already near\-optimal\)\.

![Refer to caption](https://arxiv.org/html/2608.08156v1/x2.png)Figure 3:Cloud\-systems policies \(§[3\.2](https://arxiv.org/html/2608.08156#S3.SS2)\)\.\(a\) Can’t Be Late and \(b\) Cloudcast delivered cost by arm and regime, two proposer models \(lower is better\)\. \(c\) The discriminating prediction Eq\. \([4](https://arxiv.org/html/2608.08156#S2.E4)\): hybrid cost advantage rises with the measured mean tuning gap𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]; intra\-cloud routing \(𝔼​\[Δ\]≈0\\mathbb\{E\}\[\\Delta\]\\\!\\approx\\\!0\) ties\.Hybrid delivers the lowest cost in every cell\. \(We note the measured𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]ordering for Can’t Be Late is not monotone in the engineered regime difficulty: the LLM’s untuned guesses happen to be poorest in the cheap\-restart regime, so the per\-regime gap and the headline cost\-advantage axes do not coincide for this domain\. Cloudcast and Cleanup carry the Eq\. \([4](https://arxiv.org/html/2608.08156#S2.E4)\) trend in Figure[3](https://arxiv.org/html/2608.08156#A2.F3)c\.\)

### B\.3Cleanup

WelfareUU\(higher is better\), best\-of\-NNselection,N=3N=3,Bin=100B\_\{\\mathrm\{in\}\}=100; pure CMA\-ES \(model\-independent\) is0\.93/0\.58/0\.260\.93/0\.58/0\.26for light/moderate/heavy, and the heavy hybrid ceiling is0\.5820\.582for all three models\. The per\-model, per\-seed welfare and the hybrid/vanilla ratios are block D of TableLABEL:tab:master\.

![Refer to caption](https://arxiv.org/html/2608.08156v1/x3.png)Figure 4:Cleanup: \(a\) delivered welfare by arm, seed, and model; \(b\) hybrid advantage vs\. measured𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]\(Eq\. \([4](https://arxiv.org/html/2608.08156#S2.E4)\)\), across models\. Weaker operators carry larger gaps and reap larger de\-aliasing gains\.A weaker operator carries a larger gap and reaps a larger de\-aliasing gain: Opus is the best untuned guesser \(highest vanilla on heavy,0\.450\.45\) and shows the most compressed advantage there \(1\.29×1\.29\\timesvs\. Gemini’s1\.70×1\.70\\times\), reading Eq\. \([4](https://arxiv.org/html/2608.08156#S2.E4)\) cleanly along the model\-strength axis\. Best\-of\-NNis the conservative rule; hybrid wins 9/9 under last\-iteration selection too, and is markedly less sensitive to the rule than vanilla \(the inner tuner repairs whatever structure is current\)\.

### B\.4GEPA orthogonality

Replacing the \(1\+1\) LLM outer loop with GEPA \(reflective Pareto\-frontier prompt evolution, frozen reflection LM gemini\-3\.5\-flash,≤30\\leq 30metric calls\) while keeping CMA\-ES as the innerTune; “GEPA\-vanilla” runs the proposed structures at the LM’s guessed constants \(no inner loop\), “GEPA\-hybrid” tunes them\. The delivered numbers \(cost for Can’t Be Late and Cloudcast, welfareUUfor Cleanup\) are block E of TableLABEL:tab:master: hybrid wins every non\-negligible\-gap regime, and the two ties are the lowest\-Δ\\Deltaregimes\.

Compared with the \(1\+1\) LLM runs \(Appendices[B\.2](https://arxiv.org/html/2608.08156#A2.SS2),[B\.3](https://arxiv.org/html/2608.08156#A2.SS3)\), the hybrid edge under GEPA points the same way at smaller magnitude: a strong reflective optimizer recovers more of the parameter gain on its own, so the*marginal*value of the explicit inner tuner shrinks\. It never reverses whereΔ\\Deltais materially nonzero, and on the two low\-Δ\\Deltaregimes where GEPA\-vanilla edges ahead on the headline metric, post\-hoc tuning of its own pick attributes the difference to structure\-selection noise rather than to removing the tuner\.

### B\.5Bayesian inference: suite and self\-test

The reparameterization suite spans the matched/unmatched classes\.𝒞\+\\mathcal\{C\}^\{\+\}\(nonlinear geometry, hybrid wins\):funnel\(D=10D\{=\}10, non\-centering\),eight\_schools\(D=10D\{=\}10, hierarchical partial pooling, the aliasing case\)\.𝒞−\\mathcal\{C\}^\{\-\}\(affine controls, predicted ties under NUTS\):gauss\_ill\(cond10410^\{4\}, axis\-aligned\),gauss\_rot\(same spectrum, rotated\)\. Models are unconstrained densities onℝD\\mathbb\{R\}^\{D\}so the identity is the centered baseline\. ESS andR^\\hat\{R\}use the rank\-normalized multi\-chain estimator\(Vehtari et al\.,[2021](https://arxiv.org/html/2608.08156#bib.bib28)\)computed in\-module\. ESS/grad is reported as the divergence\-awareeff=ESS/grad⋅exp⁡\(−40⋅div​\_​rate\)\\mathrm\{eff\}=\\mathrm\{ESS\}/\\mathrm\{grad\}\\cdot\\exp\(\-40\\cdot\\mathrm\{div\\\_rate\}\)so a high\-ESS\-but\-divergent sampler is correctly demoted\. The hybrid advantage by inner solver—VI in nats \(tuned ELBO vs\. the untuned guess\), NUTS in decades of ESS/grad \(hybrid vs\. identity\+\+full warmup adaptation\)—is blocks F and G of TableLABEL:tab:master\.

For the hard horseshoe logistic regression and the curved banana \(§[3\.3](https://arxiv.org/html/2608.08156#S3.SS3)\), SBC is the hard acceptance filter and the held\-out gate\. The SBC band is Bonferroni\-corrected across theDDparameters,band=−12​ln⁡\(αfw/\(2​D\)\)/L\\mathrm\{band\}=\\sqrt\{\-\\tfrac\{1\}\{2\}\\ln\(\\alpha\_\{\\mathrm\{fw\}\}/\(2D\)\)\}/\\sqrt\{L\}\(family\-wiseαfw=0\.01\\alpha\_\{\\mathrm\{fw\}\}=0\.01,LLreplicates\); without the correction valid non\-centering on the horseshoe is falsely rejected\. SBC is cheap insurance that rarely fires on live proposals from capable models \(which propose only valid idioms\) but is the only signal that catches a fast\-but\-wrong transform such as thetanh\\tanhtruncation trap\.

### B\.6The hardest case studies

The reparameterization suite uses textbook geometries that NUTS adaptation often already handles\. We stress\-test the loop on two harder targets, certified*reference\-free*by SBC \(§[3\.3](https://arxiv.org/html/2608.08156#S3.SS3)\), to ask whether the gain survives on models practitioners struggle with and on a transform that is*not*a retrieved idiom\. Both keep compute low \(Gaussian/logistic likelihoods,D≤18D\\leq 18\), and both replace CMA\-ES with VI\-gradient and NUTS\-warmup inner solvers\.

#### Horseshoe logistic regression \(D=18D=18\)\.

A sticky model: identity\+\+full NUTS adaptation suffers520520divergences \(ESS/grad6\.2×10−46\.2\\times 10^\{\-4\}\)\. The LLM searches for an SBC\-certified reparam; the headline is hybrid \(best calibrated proposal\) vs\. pure\-numerical \(identity\+\+adaptation\)\. Every one of the five proposer models, frontier and cheap alike, found a certified reparam giving≥10×\\geq\\\!10\\timesESS/grad \(block H of TableLABEL:tab:master\),0/250/25live proposals SBC\-rejected\. The frontier model buys∼\\sim2\.3×2\.3\\timesover the cheap ones, short of an order of magnitude, and high reasoning effort gave no benefit\. The task rewards retrieving an apt idiom over deliberation\. The five SBC\-certified reparams and their ESS/grad advantages \(decades over the6\.2×10−46\.2\\times 10^\{\-4\}baseline\) are block H of TableLABEL:tab:master\.

#### Curved\-ridge “banana” \(D=10D=10\)\.

The optimal transform is a nonlinear quadratic shearb←b−c​a2b\\\!\\leftarrow\\\!b\-c\\,a^\{2\}whose curvatureccis a single hidden hole, sitting*outside*the non\-centering/affine/sinh idiom set and forming the canonical failure case for Gaussian VI\. The vanilla\-numerical identity baseline is ELBO−174\-174, NUTS ESS/grad4\.1×10−34\.1\\times 10^\{\-3\}\. All three proposers constructed the shear; vanilla\-AR proposes it at a*guessed*curvature, hybrid tunes the one constant \(block I of TableLABEL:tab:master\)\. The de\-aliasing win is∼\\sim\+60\+60nats from tuning a single hidden parameter on an identical structure, the cleanest demonstration in the project\. The complementary asymmetry: NUTS’s linear metric cannot tune the curvature, but once VI finds it, NUTS samples the de\-curved space at\+1\.57\+1\.57decades \(∼\\sim37×37\\times\)\.

## Appendix CA gallery of discovered artifacts

The factorization makes a concrete prediction about*what*the LLM contributes: an apt algorithmic skeleton whose load\-bearing constants it cannot place\. The eight artifacts below \(two optimizers, three policies, three reparameterizations\) each exhibit the pattern: a recognizable structural idea that is mediocre or divergent at the model’s guessed constants and excellent once the inner solver finds them\. We reproduce the discovered code \(lightly elided\); the inner solver set the tuned constants, the model only the structure\.

#### An optimizer: Nesterov with cosine schedule and gradient clipping\.

GLM\-5\.2 \(hybrid arm\) discovered this one on the hiddenrosenbrockvalley \(Listing[1](https://arxiv.org/html/2608.08156#LST1)\)\. The structure is textbook: Nesterov lookahead, a warmup\-then\-cosine learning\-rate schedule,L2L\_\{2\}gradient clipping\. Its default constants stall in the curved valley \(loss1\.511\.51\)\. CMA\-ES pushes the learning rate*up*120×120\\times\(0\.05→6\.00\.05\\\!\\to\\\!6\.0\) and the clip threshold*down*88×88\\times\(1\.0→0\.0111\.0\\\!\\to\\\!0\.011\): an enormous step rate that stays stable only because every gradient is clipped to a tiny fixed norm, turning the method into a large\-step normalized crawl along the valley floor \(tuned loss2\.0×10−62\.0\\times 10^\{\-6\}\)\. This knife\-edge is the regime vanilla discards\.

Listing 1:Discovered optimizernesterov\_cosine\_clip\(GLM\-5\.2,rosenbrock\)\. Tuned:lr=6\.0,momentum=0\.886,clip=0\.0114,warmup\_frac=0\.003\.defcustom\_optimizer\(grad\_fn,x,y,steps,params\):

lr=params\[’lr’\];momentum=params\[’momentum’\];clip=params\[’clip’\]

eta\_min=params\[’eta\_min’\];warmup\_frac=params\[’warmup\_frac’\]

vx=0\.0;vy=0\.0

warmup\_steps=int\(warmup\_frac\*steps\)

foriinrange\(steps\):

ifi<warmup\_stepsandwarmup\_steps\>0:

cur\_lr=lr\*\(i\+1\)/warmup\_steps

else:

t=\(i\-warmup\_steps\)/max\(steps\-warmup\_steps\-1,1\)

cur\_lr=eta\_min\+0\.5\*\(lr\-eta\_min\)\*\(1\.0\+math\.cos\(math\.pi\*t\)\)

look\_x=x\-momentum\*vx;look\_y=y\-momentum\*vy

gx,gy=grad\_fn\(look\_x,look\_y\)

gnorm=math\.sqrt\(gx\*gx\+gy\*gy\)

ifgnorm\>clipandgnorm\>0\.0:

s=clip/gnorm;gx\*=s;gy\*=s

vx=momentum\*vx\+gx;vy=momentum\*vy\+gy

x\-=cur\_lr\*vx;y\-=cur\_lr\*vy

returnx,y

#### A second optimizer: restarts for a multimodal lattice\.

On the multimodalrastriginlattice, Opus 4\.8 wrapped a Nesterov heavy\-ball method in*periodic random restarts*\(Listing[2](https://arxiv.org/html/2608.08156#LST2)\): every few dozen steps it kicks the iterate by a random perturbation and zeroes the velocity — the one piece of*globally*aware structure that plain CMA\-ES on the objective lacks \(Figure[2](https://arxiv.org/html/2608.08156#S3.F2)b\)\. It is the textbook Proposition[1](https://arxiv.org/html/2608.08156#Thmproposition1)inversion: at its guessed constants this is the*worst*of the model’s three proposals \(untuned loss39\.639\.6, a wild kick that bounces the iterate out of every basin\), yet CMA\-ES tunes the restart period \(≈30\\approx\\\!30of10001000steps\), the kick scale, and the slow LR decay into the*best*of the three \(tuned1\.571\.57\), beating both vanilla’s best untuned pick \(10\.410\.4\) and the oracle Direct CMA\-ES \(9\.959\.95\)\. Vanilla, ranking by untuned score, discards exactly the structure that wins\.

Listing 2:Discovered optimizer with periodic random restarts \(Opus 4\.8,rastrigin\)\. Tuned:restart\_period≈\\approx30,pert=1\.98,lr=0\.104,mom=0\.838,lr\_decay=0\.9992\.defcustom\_optimizer\(grad\_fn,x,y,steps,params\):

lr=params\[’lr’\];mom=params\[’mom’\];clip=params\[’clip’\]

pert=params\[’pert’\];pert\_decay=params\[’pert\_decay’\]

restart\_period=max\(1,int\(params\[’restart\_period’\]\)\)

rng=np\.random\.RandomState\(int\(abs\(params\[’seed’\]\)\*1000\)%100000\+1\)

vx=0\.0;vy=0\.0;cur\_lr=lr

foriinrange\(steps\):

gx,gy=grad\_fn\(x\+mom\*vx,y\+mom\*vy\)

gn=math\.sqrt\(gx\*gx\+gy\*gy\)\+1e\-12

ifgn\>clip:gx\*=clip/gn;gy\*=clip/gn

vx=mom\*vx\-cur\_lr\*gx;vy=mom\*vy\-cur\_lr\*gy

x\+=vx;y\+=vy

if\(i\+1\)%restart\_period==0:

scale=pert\*\(pert\_decay\*\*\(i/restart\_period\)\)

x\+=scale\*rng\.randn\(\);y\+=scale\*rng\.randn\(\)

vx=0\.0;vy=0\.0

cur\_lr\*=params\[’lr\_decay’\]

returnx,y

#### A policy: a stateless hysteresis controller\.

The most inventive artifact in the study came from Claude Opus 4\.8 \(Cleanup, hybrid arm; Listing[3](https://arxiv.org/html/2608.08156#LST3)\)\. Every other proposal is a memoryless mapwaste​\_​ratio→cleaners\\mathrm\{waste\\\_ratio\}\\\!\\to\\\!\\mathrm\{cleaners\}; Opus instead reached for*hysteresis*: commit to a high cleaner count once pollution crosses an upper threshold and relax only below a*lower*one, to stop the team chattering cleaners on and off around a single breakpoint\. The policy API is*stateless*\(a pure function of the current observation, no shared memory\), so classic hysteresis is impossible; the model reconstructs the missing latch from the observable itself \(which side of the band midpoint the waste ratio sits on\)\. Because every agent computes the identical quantities, they all infer the same committed regime, which coordinates the team with zero communication through common knowledge\. Its value hinges on threshold placement: CMA\-ES sethi\_thr=0\.299=0\.299, right*on*the hidden apple\-death cliff at0\.300\.30that the model never sees, lifting welfare from1\.1311\.131\(untuned\) to1\.3281\.328\.

Listing 3:Discovered policyhysteresis\_two\_threshold\_band\(Opus 4\.8, Cleanup\)\. Tuned:lo\_thr=0\.073,hi\_thr=0\.299,low\_frac=0\.110,mid\_frac=0\.606,high\_frac=0\.935\.defpolicy\(env,agent\_id,params\):

wr=waste\_ratio\(env\);n=env\.n\_agents

lo\_thr=params\[’lo\_thr’\];hi\_thr=params\[’hi\_thr’\]

ifhi\_thr<lo\_thr:lo\_thr,hi\_thr=hi\_thr,lo\_thr

low\_frac=params\[’low\_frac’\];high\_frac=params\[’high\_frac’\];mid\_frac=params\[’mid\_frac’\]

mid=0\.5\*\(lo\_thr\+hi\_thr\)

ifwr\>=hi\_thr:frac=high\_frac

elifwr<=lo\_thr:frac=low\_frac

else:

ifwr\>=mid:frac=mid\_frac\+\(high\_frac\-mid\_frac\)\*\(\(wr\-mid\)/max\(hi\_thr\-mid,1e\-6\)\)

else:frac=low\_frac\+\(mid\_frac\-low\_frac\)\*\(\(wr\-lo\_thr\)/max\(mid\-lo\_thr,1e\-6\)\)

n\_cleaners=max\(0\.0,min\(float\(n\),frac\*n\)\)

roles=assign\_roles\(env,int\(round\(n\_cleaners\)\)\)

returnclean\_action\(env,agent\_id\)ifroles\[agent\_id\]=="clean"elseharvest\_action\(env,agent\_id\)

#### A policy: a risk\-aware spot/on\-demand scheduler\.

On Can’t Be Late \(§[3\.2](https://arxiv.org/html/2608.08156#S3.SS2), costly\-restart regime\), Gemini 3\.5 Flash proposed the slack\-gated scheduler of Listing[4](https://arxiv.org/html/2608.08156#LST4)\. Cheap spot capacity is preemptible, reliable on\-demand is expensive, and the artifact maps the per\-step state \(slack to deadline, restart overhead, remaining work\) to one of three actions — take spot, take on\-demand, or*wait*for spot to return — through slack thresholds that grow with both the restart overhead*and*the remaining work, plus a finish\-line lock\-in and a hysteresis bonus that suppresses costly toggling\. The model exposes those buffers as holes at cautious guesses \(untuned $127\.6127\.6\); CMA\-ES drives the spot buffer sharply negative and inflates the finish\-line and hysteresis margins, delivering $120\.5120\.5, under both vanilla \($126\.9126\.9\) and the deployable pure\-CMA\-ES baseline \($124\.4124\.4\)\.

Listing 4:Discovered schedulerCumulativeRiskAwareSlackPolicy\(Gemini 3\.5 Flash, Can’t Be Late, costly restarts\)\. Actions:0wait,11spot,22on\-demand\. Tuned \(selected\):spot\_offset=−3480\-3480,finish\_spot\_buffer=1133711337,hysteresis=67146714\.defdecide\(obs,params\):

slack=obs\["slack"\];ro=obs\["restart\_overhead"\]

rem=obs\["remaining\_task\_time"\];has\_spot=obs\["has\_spot"\]

last=int\(obs\["last\_cluster\_type"\]\)

ifobs\["remaining\_restart\_overhead"\]\>0:

iflast==2:return2

iflast==1andhas\_spot:return1

near\_done=rem<params\["finish\_coef"\]\*ro\+params\["finish\_offset"\]

spot\_thr=params\["spot\_coef"\]\*ro\+params\["spot\_task\_coef"\]\*rem\+params\["spot\_offset"\]

wait\_thr=params\["wait\_coef"\]\*ro\+params\["wait\_task\_coef"\]\*rem\+params\["wait\_offset"\]

ifnear\_done:spot\_thr\+=params\["finish\_spot\_buffer"\]

ifhas\_spot:

iflast==2:spot\_thr\+=params\["hysteresis"\]

return1ifslack\>spot\_threlse2

iflast==2:wait\_thr\+=params\["hysteresis"\]

return0ifslack\>wait\_threlse2

#### A policy: a self\-sharing Steiner broadcast tree\.

On Cloudcast inter\-cloud routing, Gemini 3\.5 Flash built the graph algorithm of Listing[5](https://arxiv.org/html/2608.08156#LST5): a greedy per\-partition Steiner\-tree approximation in which each destination is routed by shortest path and*the edges it uses are then discounted*, so later destinations in the same partition reuse them, growing a shared broadcast tree rather than independent paths; a deterministic sinusoidal perturbation diversifies the parallel partitions\. CMA\-ES tunes the cost/throughput blend \(α,β\\alpha,\\beta\), the sharing discount, and the perturbation scale\. Hybrid delivers $168\.7168\.7, against vanilla $213\.3213\.3and — the headline gap —1\.9×1\.9\\timesbelow pure CMA\-ES \($317317\) tuning a fixed topology: here structural discovery, not constant\-tuning, carries the domain\.

Listing 5:Discovered routerGreedySteinerPartitionSharing\(Gemini 3\.5 Flash, Cloudcast inter\-cloud\)\. Tuned:alpha=4\.1×10−44\.1\\\!\\times\\\!10^\{\-4\},beta=2\.06,sharing\_factor=0\.018,perturb\_scale=0\.042\.defsearch\_algorithm\(src,dsts,G,num\_partitions,params\):

idx=\{n:ifori,ninenumerate\(G\.nodes\(\)\)\}

base=\{\(u,v\):G\[u\]\[v\]\[’cost’\]\+params\[’alpha’\]/G\[u\]\[v\]\[’throughput’\]\*\*params\[’beta’\]

foru,vinG\.edges\(\)\}

dist=nx\.single\_source\_dijkstra\_path\_length\(G,src,weight=lambdau,v,d:base\[\(u,v\)\]\)

order=sorted\(dsts,key=lambdad:dist\.get\(d,1e9\),reverse=params\[’sort\_reverse’\]\>0\.5\)

bc=BroadCastTopology\(src,dsts,num\_partitions\)

forjinrange\(num\_partitions\):

w=\{e:base\[e\]\*\(1\+params\[’perturb\_scale’\]\*

math\.sin\(idx\[e\[0\]\]\*17\.23\+idx\[e\[1\]\]\*43\.19\+j\*97\.43\)\)foreinG\.edges\(\)\}

fordstinorder:

path=nx\.shortest\_path\(G,src,dst,weight=lambdau,v,d:w\[\(u,v\)\]\)

foru,vinzip\(path\[:\-1\],path\[1:\]\):

bc\.append\_dst\_partition\_path\(dst,j,\[u,v,G\[u\]\[v\]\]\)

w\[\(u,v\)\]\*=params\[’sharing\_factor’\]

returnbc

#### A reparameterization: a nonlinear quadratic shear\.

On the curved\-ridge “banana” \(§[3\.3](https://arxiv.org/html/2608.08156#S3.SS3)\), all three proposer models independently constructed the same non\-textbook transform: a nonlinear quadratic shear that straightens the ridge \(Listing[6](https://arxiv.org/html/2608.08156#LST6), the minimal three\-hole form of the family\)\. The curvaturecurvis the single hidden hole; VI tuned it to0\.9370\.937and the ridge widthb\_widthto0\.3000\.300, recovering the generative constants \(1\.01\.0and0\.30\.3\)\. Vanilla proposes the same shear at a*guessed*curvature \(ELBO≈−74\\approx\-74\); the inner loop tunes the one constant to ELBO−66\-66\(and−13\-13for a per\-coordinate variant\), against the identity baseline−174\-174\.

Listing 6:Discovered reparameterization \(curved\-ridge banana; family found by Opus 4\.8, Gemini 3\.1 Pro, Gemini 3\.5 Flash\)\. Manifest:curvguess 1\.0 \(range−5\-5to55\)\. VI\-tuned:curv=0\.937,a\_scale=0\.527,b\_width=0\.300\.defreparam\(z,params\):

K=5

a=params\[’a\_scale’\]\*z\[:K\]

b=params\[’curv’\]\*a\*\*2\+params\[’b\_width’\]\*z\[K:\]

returnjnp\.concatenate\(\[a,b\]\)

#### A reparameterization: partial non\-centering for a sparse funnel\.

On the sticky horseshoe logistic regression \(D=18D=18\), Gemini 3\.1 Pro found the most effective SBC\-certified transform \(Listing[7](https://arxiv.org/html/2608.08156#LST7)\)\. It is the cheapest structure that works: an affine map plus a single*partial\-non\-centering*knobncthat scales the coefficient block by a tunable power of the global×\\timeslocal shrinkage\. NUTS adaptation alone leaves520520divergences \(ESS/grad6\.2×10−46\.2\\times 10^\{\-4\}\); this transform reaches ESS/grad1\.48×10−21\.48\\times 10^\{\-2\}\(∼\\sim24×24\\times,7373divergences\) and passes SBC, ahead of heavier Cauchy / sinh\-arcsinh competitors\.

Listing 7:Discovered reparameterizationPartialNCP\_Affine\(Gemini 3\.1 Pro, horseshoe logistic regression\)\. Key holencinterpolates centered \(0\) to non\-centered \(11\); delivered atnc=0\.5, unit affine scales\.defreparam\(z,params\):

theta\_0=z\[0:1\]\*params\[’s\_int’\]\+params\[’m\_int’\]

theta\_1=z\[1:2\]\*params\[’s\_glob’\]\+params\[’m\_glob’\]

theta\_2=z\[2:10\]\*params\[’s\_loc’\]\+params\[’m\_loc’\]

eff\_scale=jnp\.exp\(params\[’nc’\]\*\(theta\_1\[0\]\+theta\_2\)\)

theta\_3=z\[10:18\]\*eff\_scale\*params\[’s\_coef’\]\+params\[’m\_coef’\]

returnjnp\.concatenate\(\[theta\_0,theta\_1,theta\_2,theta\_3\]\)

#### A reparameterization: a learned affine decorrelation\.

The affine controls make the complementary point to the banana’s nonlinear shear\. Ongauss\_rot, a rotated ill\-conditioned Gaussian, Gemini 3\.5 Flash proposed the dense affine map of Listing[8](https://arxiv.org/html/2608.08156#LST8): per\-coordinate shifts and log\-scales plus a rank\-1 strictly\-lower\-triangular mixing term \(a cumulative\-sum coupling\) that captures an arbitrary\-direction rotation inD=10D=10with3838holes\. At the guessed constants the transform is*worse than doing nothing*\(untuned ELBO−33\.4\-33\.4, below the identity guide’s−28\.5\-28\.5\): the aliasing trap again\. VI tunes all3838holes to ELBO\+4\.30\+4\.30, a\+37\.7\+37\.7\-nat gain over vanilla, and the headline VI\-vs\-NUTS asymmetry follows — a fixed standard guide cannot rescale or rotate without the transform \(VI gains enormously\), yet the*same*transform barely moves NUTS, whose warmup mass matrix already is a linear preconditioner\. This is Eq\. \([4](https://arxiv.org/html/2608.08156#S2.E4)\) read along the linear/nonlinear axis: redundant with the inner solver’s competence, soΔ≈0\\Delta\\\!\\approx\\\!0under NUTS; orthogonal to it, so a largeΔ\\Deltaunder VI\.

Listing 8:Discovered reparameterizationRank1LowerTriangularAffine\(Gemini 3\.5 Flash,gauss\_rot, VI\)\. The3838holes are a per\-coordinate shiftmuand log\-scales, and a rank\-1 lower\-triangular coupling\(a, b\); literal unpacking elided\.defreparam\(z,params\):

mu=jnp\.array\(\[params\[f’mu\{i\}’\]foriinrange\(10\)\]\)

s=jnp\.array\(\[params\[f’s\{i\}’\]foriinrange\(10\)\]\)

a=jnp\.array\(\[params\[f’a\{i\}’\]foriinrange\(9\)\]\)

b=jnp\.array\(\[params\[f’b\{i\}’\]foriinrange\(9\)\]\)

w=jnp\.concatenate\(\[jnp\.zeros\(1\),jnp\.cumsum\(b\*z\[:\-1\]\)\]\)

a\_pad=jnp\.concatenate\(\[jnp\.zeros\(1\),a\]\)

returnmu\+jnp\.exp\(s\)\*\(z\+a\_pad\*w\)

Across all eight, the LLM supplies the*idea*\(accelerated descent with clipping, globally aware restarts, stateful control reconstructed from common knowledge, risk\-aware spot scheduling, a self\-sharing broadcast tree, a quadratic shear, partial non\-centering, a learned affine decorrelation\) and the inner solver supplies the*constants*that decide whether the idea works\. This is the factorization made tangible\.

## Appendix DConsolidated results: every cell

TableLABEL:tab:masteris the single source for every experimental cell — all proposer models, regimes, and inner/outer solvers from Appendix[B](https://arxiv.org/html/2608.08156#A2)— in one schema:vanilla/numerical\-only /hybrid, the mean tuning gap𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\], and the hybrid advantage\.Boldmarks the better of vanilla/hybrid; “–” an arm not run for that arm\-definition \(the VI arm reports hybrid vs\. vanilla, the NUTS / horseshoe arms hybrid vs\. pure\-numerical\)\. The numerical arm is an*oracle*\(Direct CMA\-ES\) only in block A; elsewhere it is a deployable no\-structure baseline\. Shading of the𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]and Hyb\. adv\. columns is a per\-group heatmap \(darker==larger\): matching gradients down the two columns are the visual signature of advantage∝𝔼​\[Δ\]\\propto\\mathbb\{E\}\[\\Delta\]\(Eq\.[4](https://arxiv.org/html/2608.08156#S2.E4)\)\. Meta \(block A\) is shaded per model on a log scale; the policy blocks B–D per block on a linear scale — Cloudcast \(C\) and Cleanup \(D\) track cleanly, while Can’t Be Late \(B\), whose gap is non\-monotone in regime, intentionally does not align\. Advantage units are per\-block as stated in each header\.

Table 2:Consolidated experimental results \(all cells\)\.TaskModelVanillaNumericalHybrid𝔼​\[Δ\]\\mathbb\{E\}\[\\Delta\]Hyb\. adv\.Family 1 Meta\-optimizers on closed\-form objectivesA\. Meta\-optimizers⋅\\cdotouter: \(1\+1\) LLM⋅\\cdotinner: CMA\-ES⋅\\cdotmetric: final loss↓\\downarrow⋅\\cdotnumerical: Direct CMA\-ES \(*oracle*ref\.\)⋅\\cdotadv: loss reduction van−\-hybrosenbrockOpus 4\.85\.7×10−65\.7\\\!\\times\\\!10^\{\-6\}4\.2×10−134\.2\\\!\\times\\\!10^\{\-13\}2\.4×𝟏𝟎−𝟐𝟓\\mathbf\{2\.4\\\!\\times\\\!10^\{\-25\}\}0\.270\.27\+5\.7×10−6\+5\.7\\\!\\times\\\!10^\{\-6\}ellipsoidOpus 4\.89\.3×10−99\.3\\\!\\times\\\!10^\{\-9\}8\.6×10−148\.6\\\!\\times\\\!10^\{\-14\}3\.8×𝟏𝟎−𝟏𝟎𝟔\\mathbf\{3\.8\\\!\\times\\\!10^\{\-106\}\}5\.95\.9tie†rastriginOpus 4\.810\.410\.49\.959\.951\.57\\mathbf\{1\.57\}10\.810\.8\+8\.8\+8\.8ackleyOpus 4\.810\.810\.84\.4×10−124\.4\\\!\\times\\\!10^\{\-12\}4\.1×𝟏𝟎−𝟏𝟑\\mathbf\{4\.1\\\!\\times\\\!10^\{\-13\}\}12\.312\.3\+10\.8\+10\.8schwefelOpus 4\.840540535835840540520\.420\.4tierosenbrockGLM\-5\.22\.1×10−42\.1\\\!\\times\\\!10^\{\-4\}4\.2×10−134\.2\\\!\\times\\\!10^\{\-13\}1\.5×𝟏𝟎−𝟏𝟐\\mathbf\{1\.5\\\!\\times\\\!10^\{\-12\}\}0\.280\.28\+2\.1×10−4\+2\.1\\\!\\times\\\!10^\{\-4\}ellipsoidGLM\-5\.22\.3×𝟏𝟎−𝟒𝟎\\mathbf\{2\.3\\\!\\times\\\!10^\{\-40\}\}8\.6×10−148\.6\\\!\\times\\\!10^\{\-14\}3\.0×10−283\.0\\\!\\times\\\!10^\{\-28\}1\.11\.1tie†rastriginGLM\-5\.219\.219\.29\.959\.951\.22\\mathbf\{1\.22\}11\.011\.0\+18\.0\+18\.0ackleyGLM\-5\.213\.113\.14\.4×10−124\.4\\\!\\times\\\!10^\{\-12\}3\.3×𝟏𝟎−𝟗\\mathbf\{3\.3\\\!\\times\\\!10^\{\-9\}\}11\.411\.4\+13\.1\+13\.1schwefelGLM\-5\.2405405358358𝟐𝟗𝟕\\mathbf\{297\}43\.043\.0\+108\+108rosenbrockGemini 3\.5 Flash8\.6×10−68\.6\\\!\\times\\\!10^\{\-6\}4\.2×10−134\.2\\\!\\times\\\!10^\{\-13\}2\.7×𝟏𝟎−𝟐𝟐\\mathbf\{2\.7\\\!\\times\\\!10^\{\-22\}\}0\.0030\.003\+8\.6×10−6\+8\.6\\\!\\times\\\!10^\{\-6\}ellipsoidGemini 3\.5 Flash4\.5×10−384\.5\\\!\\times\\\!10^\{\-38\}8\.6×10−148\.6\\\!\\times\\\!10^\{\-14\}0\.0\\mathbf\{0\.0\}418\.6418\.6tie†rastriginGemini 3\.5 Flash19\.219\.29\.959\.950\.33\\mathbf\{0\.33\}10\.610\.6\+18\.9\+18\.9ackleyGemini 3\.5 Flash13\.913\.94\.4×10−124\.4\\\!\\times\\\!10^\{\-12\}5\.7×𝟏𝟎−𝟕\\mathbf\{5\.7\\\!\\times\\\!10^\{\-7\}\}14\.014\.0\+13\.9\+13\.9schwefelGemini 3\.5 Flash405405358358𝟐𝟐𝟕\\mathbf\{227\}92\.692\.6\+178\+178Family 2 Executable policies \(systems & social dilemmas\)B\. Can’t Be Late\(spot/on\-demand scheduling\)⋅\\cdot\(1\+1\) LLM / CMA\-ES⋅\\cdotmetric: cost $↓\\downarrow⋅\\cdotnumerical: Pure CMA\-ES⋅\\cdotadv: % vs\. vanillacheap restartGemini97\.497\.4110\.3110\.387\.6\\mathbf\{87\.6\}18\.818\.810\.1%10\.1\\%cheap restartOpus102\.6102\.6110\.3110\.387\.5\\mathbf\{87\.5\}14\.714\.714\.7%14\.7\\%moderateGemini117\.7117\.7117\.3117\.3106\.2\\mathbf\{106\.2\}11\.111\.19\.8%9\.8\\%moderateOpus115\.0115\.0117\.3117\.3108\.9\\mathbf\{108\.9\}5\.35\.35\.3%5\.3\\%costly restartGemini126\.9126\.9124\.4124\.4120\.5\\mathbf\{120\.5\}5\.55\.55\.0%5\.0\\%costly restartOpus125\.0125\.0124\.4124\.4113\.5\\mathbf\{113\.5\}1\.51\.59\.2%9\.2\\%C\. Cloudcast\(multi\-cloud broadcast routing\)⋅\\cdot\(1\+1\) LLM / CMA\-ES⋅\\cdotmetric: cost $↓\\downarrow⋅\\cdotnumerical: Pure CMA\-ES⋅\\cdotadv: % vs\. vanilla⋅\\cdotintra=𝔼​\[Δ\]≈0=\\mathbb\{E\}\[\\Delta\]\{\\approx\}0controlintra\-cloudGemini109\.0109\.0157\.1157\.1101\.7\\mathbf\{101\.7\}8\.18\.16\.7%6\.7\\%intra\-cloudOpus107\.6107\.6157\.1157\.1106\.8\\mathbf\{106\.8\}3\.83\.80\.7%0\.7\\%inter\-cloudGemini213\.3213\.3317\.4317\.4168\.7\\mathbf\{168\.7\}11\.711\.720\.9%20\.9\\%inter\-cloudOpus317\.4317\.4317\.4317\.4205\.1\\mathbf\{205\.1\}12\.512\.535\.4%35\.4\\%D\. Cleanup\(sequential social dilemma\)⋅\\cdot\(1\+1\) LLM / CMA\-ES⋅\\cdotmetric: welfareUU↑\\uparrow⋅\\cdotnumerical: Pure CMA\-ES \(light/mod/heavy=0\.93/0\.58/0\.26=0\.93/0\.58/0\.26\)⋅\\cdotadv: hyb/vanlightGemini 3\.5 Flash1\.211\.210\.930\.931\.34\\mathbf\{1\.34\}0\.170\.171\.11×1\.11\\timesmoderateGemini 3\.5 Flash0\.940\.940\.580\.581\.01\\mathbf\{1\.01\}0\.210\.211\.08×1\.08\\timesheavyGemini 3\.5 Flash0\.340\.340\.260\.260\.58\\mathbf\{0\.58\}0\.370\.371\.70×1\.70\\timeslightGLM\-5\.21\.191\.190\.930\.931\.34\\mathbf\{1\.34\}0\.200\.201\.13×1\.13\\timesmoderateGLM\-5\.20\.880\.880\.580\.581\.03\\mathbf\{1\.03\}0\.200\.201\.17×1\.17\\timesheavyGLM\-5\.20\.420\.420\.260\.260\.58\\mathbf\{0\.58\}0\.270\.271\.39×1\.39\\timeslightOpus 4\.81\.161\.160\.930\.931\.33\\mathbf\{1\.33\}0\.260\.261\.14×1\.14\\timesmoderateOpus 4\.80\.920\.920\.580\.581\.07\\mathbf\{1\.07\}0\.290\.291\.16×1\.16\\timesheavyOpus 4\.80\.450\.450\.260\.260\.58\\mathbf\{0\.58\}0\.300\.301\.29×1\.29\\timesE\. GEPA as outer operator\(orthogonality\)⋅\\cdotouter: GEPA⋅\\cdotinner: CMA\-ES⋅\\cdotLM: Gemini 3\.5 Flash⋅\\cdotnumerical: Pure CMA\-ES⋅\\cdotadv: signed % \(cost orUU\)Can’t Be Late⋅\\cdotcheap \($↓\\downarrow\)GEPA65\.9\\mathbf\{65\.9\}110\.3110\.366\.066\.0–−0\.2%\-0\.2\\%tieCan’t Be Late⋅\\cdotmoderate \($↓\\downarrow\)GEPA88\.288\.2117\.3117\.381\.5\\mathbf\{81\.5\}–\+7\.6%\+7\.6\\%Can’t Be Late⋅\\cdotcostly \($↓\\downarrow\)GEPA94\.894\.8124\.4124\.491\.7\\mathbf\{91\.7\}–\+3\.3%\+3\.3\\%Cleanup⋅\\cdotlight \(UU↑\\uparrow\)GEPA1\.211\.210\.930\.931\.34\\mathbf\{1\.34\}–\+10\.7%\+10\.7\\%Cleanup⋅\\cdotmoderate \(UU↑\\uparrow\)GEPA0\.970\.970\.580\.581\.07\\mathbf\{1\.07\}–\+10\.6%\+10\.6\\%Cleanup⋅\\cdotheavy \(UU↑\\uparrow\)GEPA0\.510\.510\.260\.260\.58\\mathbf\{0\.58\}–\+14\.4%\+14\.4\\%Cloudcast⋅\\cdotintra \($↓\\downarrow\)GEPA102\.1\\mathbf\{102\.1\}157\.1157\.1107\.6107\.6–−5\.4%\-5\.4\\%tieCloudcast⋅\\cdotinter \($↓\\downarrow\)GEPA213\.3213\.3317\.4317\.4201\.8\\mathbf\{201\.8\}–\+5\.4%\+5\.4\\%Family 3 Approximate Bayesian inference \(reparameterization\)F\. Reparam suite, VI inner⋅\\cdot\(1\+1\) LLM / VI \(Adam/ELBO\)⋅\\cdotGemini 3\.5 Flash⋅\\cdotmetric: ELBO nats↑\\uparrow⋅\\cdotvan==untuned, hyb==tuned⋅\\cdotadv: natsfunnel \(𝒞\+\\mathcal\{C\}^\{\+\}\)Gemini 3\.5 Flash0\.080\.08–0\.010\.01–−0\.07\-0\.07tieeight\_schools \(𝒞\+\\mathcal\{C\}^\{\+\}\)Gemini 3\.5 Flash2\.142\.14–7\.97\\mathbf\{7\.97\}–\+5\.8\+5\.8gauss\_ill \(𝒞−\\mathcal\{C\}^\{\-\}\)Gemini 3\.5 Flash−0\.34\-0\.34–8\.97\\mathbf\{8\.97\}–\+9\.3\+9\.3gauss\_rot \(𝒞−\\mathcal\{C\}^\{\-\}\)Gemini 3\.5 Flash−33\.4\-33\.4–4\.30\\mathbf\{4\.30\}–\+37\.7\+37\.7G\. Reparam suite, NUTS inner⋅\\cdot\(1\+1\) LLM / NUTS warmup⋅\\cdotGemini 3\.5 Flash⋅\\cdotmetric: ESS/grad↑\\uparrow⋅\\cdotnumerical: identity\+\+adapt\.⋅\\cdotadv: decadesfunnel \(𝒞\+\\mathcal\{C\}^\{\+\}\)Gemini 3\.5 Flash–8\.4×10−58\.4\\\!\\times\\\!10^\{\-5\}0\.111\\mathbf\{0\.111\}–\+3\.12\+3\.12eight\_schools \(𝒞\+\\mathcal\{C\}^\{\+\}\)Gemini 3\.5 Flash–3\.8×10−43\.8\\\!\\times\\\!10^\{\-4\}0\.056\\mathbf\{0\.056\}–\+2\.17\+2\.17gauss\_ill \(𝒞−\\mathcal\{C\}^\{\-\}\)Gemini 3\.5 Flash–0\.1490\.1490\.1510\.151–\+0\.01\+0\.01tiegauss\_rot \(𝒞−\\mathcal\{C\}^\{\-\}\)Gemini 3\.5 Flash–0\.0030\.0030\.0030\.003–−0\.02\-0\.02tieH\. Horseshoe logistic regression\(D=18D\{=\}18, hard case\)⋅\\cdot\(1\+1\) LLM / NUTS warmup⋅\\cdotmetric: ESS/grad↑\\uparrow⋅\\cdotnumerical: identity\+\+adapt\. \(6\.2×10−46\.2\\\!\\times\\\!10^\{\-4\}, 520 div\.\)⋅\\cdotadv: decadeshorseshoeGemini 3\.1 Pro–6\.2×10−46\.2\\\!\\times\\\!10^\{\-4\}1\.48×𝟏𝟎−𝟐\\mathbf\{1\.48\\\!\\times\\\!10^\{\-2\}\}–\+1\.37\+1\.37horseshoeClaude Opus 4\.8–6\.2×10−46\.2\\\!\\times\\\!10^\{\-4\}1\.29×𝟏𝟎−𝟐\\mathbf\{1\.29\\\!\\times\\\!10^\{\-2\}\}–\+1\.32\+1\.32horseshoeOpus 4\.8 \(high reas\.\)–6\.2×10−46\.2\\\!\\times\\\!10^\{\-4\}7\.4×𝟏𝟎−𝟑\\mathbf\{7\.4\\\!\\times\\\!10^\{\-3\}\}–\+1\.07\+1\.07horseshoeGLM\-5\.2–6\.2×10−46\.2\\\!\\times\\\!10^\{\-4\}6\.4×𝟏𝟎−𝟑\\mathbf\{6\.4\\\!\\times\\\!10^\{\-3\}\}–\+1\.01\+1\.01horseshoeGemini 3\.5 Flash–6\.2×10−46\.2\\\!\\times\\\!10^\{\-4\}6\.2×𝟏𝟎−𝟑\\mathbf\{6\.2\\\!\\times\\\!10^\{\-3\}\}–\+1\.00\+1\.00I\. Curved\-ridge “banana”\(D=10D\{=\}10, hard case\)⋅\\cdot\(1\+1\) LLM / VI \(then NUTS\)⋅\\cdotmetric: ELBO nats↑\\uparrow⋅\\cdotnumerical: identity \(−174\-174\)⋅\\cdotvan==guessedcc, hyb==tunedcc⋅\\cdotadv: nats; NUTS cross\-solver\+1\.57\+1\.57decbananaGemini 3\.1 Pro−72\.9\-72\.9−174\-174−13\.4\\mathbf\{\-13\.4\}–\+59\.5\+59\.5bananaGemini 3\.5 Flash−74\.1\-74\.1−174\-174−13\.2\\mathbf\{\-13\.2\}–\+60\.9\+60\.9bananaClaude Opus 4\.8−74\.1\-74\.1−174\-174−13\.4\\mathbf\{\-13\.4\}–\+60\.8\+60\.8