LLM-Driven Evolutionary Generation of Multi-Objective Bayesian Optimization Algorithms

# LLM-Driven Evolutionary Generation of Multi-Objective Bayesian Optimization Algorithms
Source: [https://arxiv.org/html/2607.08791](https://arxiv.org/html/2607.08791)
G\. LaskarisTerra Quantum AG, Kornhausstrasse 25, 9000 St\. Gallen, SwitzerlandLIACS, Leiden University, Leiden, NetherlandsN\. van SteinLIACS, Leiden University, Leiden, NetherlandsE\. RaponiLIACS, Leiden University, Leiden, NetherlandsT\. BäckTerra Quantum AG, Kornhausstrasse 25, 9000 St\. Gallen, SwitzerlandLIACS, Leiden University, Leiden, NetherlandsF\. NeukartTerra Quantum AG, Kornhausstrasse 25, 9000 St\. Gallen, SwitzerlandLIACS, Leiden University, Leiden, Netherlands

###### Abstract

Designing effective multi\-objective Bayesian optimization \(MOBO\) algorithms requires balancing many interdependent design choices whose optimal configuration is problem\-dependent and typically demands deep expertise\. We extend the LLaMEA framework to MOBO, using large language models as mutation and crossover operators within evolutionary strategies to generate complete algorithm implementations, with SMAC hyperparameter optimization integrated into the evolutionary loop\. Across nine evolutionary runs we generated approximately 900 algorithms and benchmarked them on twelve synthetic problems \(ZDT, DTLZ, WFG\) and three real\-world engineering problems \(RE\), using a BoFire qParEGO implementation as a state\-of\-the\-art Bayesian\-optimization baseline\. On the synthetic suite the strongest generated algorithm attains the highest mean normalized hypervolume \(0\.971, vs\. 0\.869 for qParEGO\) while requiring roughly 60× less wall\-clock time; a Friedman test with post\-hoc analysis places the two in a single top\-performing group, and per\-problem tests find the generated algorithm significantly better than qParEGO on 7 of the 12 problems and never worse, matching state\-of\-the\-art accuracy at an order\-of\-magnitude lower cost\. On the three unseen real\-world engineering problems a generated algorithm attains the best mean normalized hypervolume \(0\.985, vs\. 0\.971 for qParEGO\)—significantly better than qParEGO on two of the three problems—at roughly 3\.4× lower wall\-clock cost, confirming that the gains transfer beyond the synthetic regime\. LLM\-driven evolutionary search can thus discover algorithm designs that achieve Pareto\-efficient trade\-offs difficult to reach through manual design\.

## IIntroduction

Multi\-objective optimization \(MOO\) problems arise naturally across science and engineering with extensive applications to drug design\[[39](https://arxiv.org/html/2607.08791#bib.bib9),[44](https://arxiv.org/html/2607.08791#bib.bib10)\]and structural engineering\[[13](https://arxiv.org/html/2607.08791#bib.bib11),[34](https://arxiv.org/html/2607.08791#bib.bib12)\]among others\. When the objective functions are expensive to evaluate, Bayesian optimization \(BO\)\[[40](https://arxiv.org/html/2607.08791#bib.bib14),[24](https://arxiv.org/html/2607.08791#bib.bib13)\]offers a framework for Pareto\-optimal solutions with limited budget\. Designing an effective multi\-objective Bayesian optimization \(MOBO\) algorithm remains a challenge since there are numerous interdependent design choices\. The best configuration is often problem\-dependent, requiring deep expertise and extensive empirical tuning\.

Recent work on LLM\-driven algorithm generation\[[38](https://arxiv.org/html/2607.08791#bib.bib5),[31](https://arxiv.org/html/2607.08791#bib.bib61),[55](https://arxiv.org/html/2607.08791#bib.bib21),[46](https://arxiv.org/html/2607.08791#bib.bib6),[37](https://arxiv.org/html/2607.08791#bib.bib35)\]has shown that large language models, coupled with evolutionary search, can automatically design optimization algorithms competitive with human\-designed alternatives—going beyond parameter tuning to synthesize new algorithms\.

LLaMEA\-BO\[[29](https://arxiv.org/html/2607.08791#bib.bib7)\]is an automated method that uses large language models as variation operators within an evolutionary loop to design complete Bayesian optimization algorithms\. In this work, we extend it to the multi\-objective setting, with the discovered algorithms as the object of study\. We frame the work as an extension of the LLaMEA framework rather than as a new automated\-design paradigm: our contribution is the MOBO generation loop and the concrete, competitive algorithms it produces, which we benchmark against established hand\-designed MOBO methods\. Our contributions are as follows:

- •We extend the LLaMEA framework\[[46](https://arxiv.org/html/2607.08791#bib.bib6),[29](https://arxiv.org/html/2607.08791#bib.bib7)\]to the multi\-objective Bayesian optimization setting, generating and configuring complete MOBO algorithms with normalized hypervolume across a suite of synthetic and real\-world problems as the fitness signal\. Following the LLaMEA\-HPO methodology\[[48](https://arxiv.org/html/2607.08791#bib.bib8)\], we integrate SMAC\-based hyperparameter optimization directly into the evolutionary loop, implemented within our own multi\-objective pipeline\.
- •We benchmark three evolutionary\-strategy configurations within this framework—\(1\+1\)\(1\{\+\}1\)\-ES,\(4\+16\)\(4\{\+\}16\)\-ES, and\(8,16\)\(8,16\)\-ES—running Gemini\-2\.5\-flash as the LLM operator, and analyze how the selection regime affects the diversity and quality of the generated algorithms\.
- •We show that the strongest generated algorithm on the synthetic suite,MOEAD\-EI Hybrid, attains the highest mean normalized hypervolume across the twelve synthetic problems—significantly more accurate than the state\-of\-the\-art BoFireqParEGObaseline on 7 of the 12 problems \(per\-problem Welchtt\-test\) and never significantly worse—while requiring roughly60×60\\timesless wall\-clock time\. On three unseen real\-world engineering problems, the best systematically generated algorithm,Improved\-Scalarized\-EI, is the single most accurate method, significantly outperformingqParEGOon two of the three problems at≈\\approx3\.4×3\.4\\timeslower cost\.
- •We characterize the time–accuracy trade\-off across all nine benchmarked algorithms \(four LLaMEA\-generated, the state\-of\-the\-artqParEGObaseline, the surrogate\-assistedIOC\-SAMO\-COBRA, two classical evolutionary baselines, and Random Search\), showing that the generated algorithms occupy the efficient frontier: they match or exceed state\-of\-the\-art accuracy while running one to three orders of magnitude faster, and every one is significantly faster thanqParEGOon all fifteen problems \(Wilcoxon signed\-rank,p=6\.1×10−5p=6\.1\\times 10^\{\-5\}\)\.

The remainder of this paper is organized as follows\. Section[II](https://arxiv.org/html/2607.08791#S2)reviews related work in multi\-objective Bayesian optimization, LLM\-based algorithm design, and automated algorithm configuration\. Section[III](https://arxiv.org/html/2607.08791#S3)describes the LLaMEA\-BO framework as extended to the multi\-objective setting\. Section[IV](https://arxiv.org/html/2607.08791#S4)presents the experimental setup—the nine evolutionary runs, the twelve synthetic benchmark problems, and the three real\-world engineering problems used for generalization assessment\. Section[V](https://arxiv.org/html/2607.08791#S5)reports benchmarking results for each of the three phases and the time–accuracy trade\-off analysis\. Section[VI](https://arxiv.org/html/2607.08791#S6)explains why the selected algorithms perform well and outlines limitations and future directions\. Section[VII](https://arxiv.org/html/2607.08791#S7)concludes\.

## IIRelated work

### II\.1Multi\-Objective Bayesian Optimization

Bayesian optimization extends to multi\-objective settings by replacing the scalar acquisition function with one that accounts for multiple objectives\. ParEGO\[[25](https://arxiv.org/html/2607.08791#bib.bib1)\]achieves this through random scalarization: at each iteration it draws a weight vector, scalarizes the objectives via the augmented Tchebycheff decomposition, and maximizes the Expected Improvement \(EI\) on the resulting scalar surrogate\. A limitation of ParEGO is the evaluation of a single scalarization per iteration affecting the simultaneous exploration of different Pareto front regions\. qParEGO\[[9](https://arxiv.org/html/2607.08791#bib.bib15)\]addresses this with a Monte Carlo, parallel reformulation that draws a fresh random scalarization for each point in a batch, enablingqqcandidates to be selected jointly per iteration; we adopt its implementation from the BoFire framework\[[15](https://arxiv.org/html/2607.08791#bib.bib41)\]\(specifically itsQparegoStrategy, which is built on BoTorch\[[2](https://arxiv.org/html/2607.08791#bib.bib16)\]\) as a state\-of\-the\-art baseline in our experiments\.

An alternative family of methods directly targets the hypervolume indicator\. Expected Hypervolume Improvement \(EHVI\)\[[17](https://arxiv.org/html/2607.08791#bib.bib2)\]computes the expected gain in dominated hypervolume from a candidate point, providing an acquisition function that balances convergence and diversity\. The limitation whereby EHVI computations scale poorly with the number of objectives has been mitigated by Monte Carlo approximations\[[9](https://arxiv.org/html/2607.08791#bib.bib15)\]and differentiable formulations within the BoTorch framework\[[2](https://arxiv.org/html/2607.08791#bib.bib16)\]\.

Among the information\-theoretic approaches are MESMO\[[3](https://arxiv.org/html/2607.08791#bib.bib17)\]which extends max\-value entropy search to the multi\-objective setting and PESMO\[[21](https://arxiv.org/html/2607.08791#bib.bib18)\]which uses predictive entropy search to identify points that maximally reduce uncertainty\. Decomposition\-based methods such as MOEA/D\-EGO\[[57](https://arxiv.org/html/2607.08791#bib.bib19)\]combine the weight\-vector decomposition strategy from evolutionary multi\-objective optimization with Gaussian process surrogates\.

Despite the handful of approaches, each algorithm embodies specific design decisions—surrogate type, acquisition function, candidate generation, normalization—that are difficult to recombine or modify without expert knowledge\. Our work avoids this manual design process utilizing an LLM to generate MOBO algorithms from scratch\.

### II\.2LLM\-Based Algorithm Design

The use of LLMs for automated algorithm design was opened by FunSearch\[[38](https://arxiv.org/html/2607.08791#bib.bib5)\], an evolutionary procedure showing that an LLM guided by execution feedback can discover novel mathematical constructs and new solutions to the cap\-set problem\[[20](https://arxiv.org/html/2607.08791#bib.bib62)\]\. Shortly afterwards, LLaMEA\[[46](https://arxiv.org/html/2607.08791#bib.bib6)\]introduced a framework in which an LLM acts as the variation operator within an evolution strategy to generate single\-objective metaheuristics, receiving the current best algorithm together with its performance feedback and returning an improved variant\. The paradigm was later scaled to a broad class of scientific and engineering problems by AlphaEvolve\[[37](https://arxiv.org/html/2607.08791#bib.bib35)\], which couples an ensemble of frontier LLMs with an evolutionary outer loop and an automated evaluation harness, reporting discoveries such as improved matrix\-multiplication algorithms; open re\-implementations \(OpenEvolve\[[41](https://arxiv.org/html/2607.08791#bib.bib57)\]\) and related systems \(ShinkaEvolve\[[27](https://arxiv.org/html/2607.08791#bib.bib58)\]\) have followed\. A parallel line targets automatic*heuristic*design: Evolution of Heuristics \(EoH\)\[[31](https://arxiv.org/html/2607.08791#bib.bib61)\]co\-evolves natural\-language design ideas and their code, and ReEvo\[[55](https://arxiv.org/html/2607.08791#bib.bib21)\]adds reflection on past failures to steer generation, while other methods replace the evolutionary population altogether—LHNS\[[54](https://arxiv.org/html/2607.08791#bib.bib60)\]with LLM\-driven neighborhood search and MCTS\-AHD\[[58](https://arxiv.org/html/2607.08791#bib.bib59)\]with Monte\-Carlo tree search\. EvoPrompting\[[8](https://arxiv.org/html/2607.08791#bib.bib20)\]applies a comparable evolutionary LLM loop to neural\-architecture search\. These methods target different problem classes and few direct comparisons exist across the families, so we treat them as context rather than as baselines\.

Most relevant to our work, LLaMEA\-BO\[[29](https://arxiv.org/html/2607.08791#bib.bib7)\]extended LLaMEA to single\-objective Bayesian optimization, generating BO algorithms with their own surrogate models and acquisition functions, and LLaMEA\-HPO\[[48](https://arxiv.org/html/2607.08791#bib.bib8)\]integrated SMAC\-based hyperparameter optimization into the evolutionary loop so that each generated algorithm’s tunable parameters are configured automatically\.

We build on this lineage—rather than the heuristic\-design methods above—because it is purpose\-built for our setting: to our knowledge, LLaMEA\-BO is the only LLM\-driven framework that synthesizes*complete*BO algorithms\. Because its modular design decouples the problem suite and evaluator from the generation loop, extending it to the multi\-objective setting is primarily a matter of changing the evaluation signal—to normalized hypervolume over a Pareto\-front approximation—rather than redesigning the generation mechanism\.

### II\.3Automated Algorithm Configuration

Automated algorithm configuration seeks to find optimal parameter settings for a given algorithm across a set of problem instances\. SMAC\[[30](https://arxiv.org/html/2607.08791#bib.bib3)\]uses a random forest surrogate to model the relationship between configurations and performance, employing Bayesian optimization to efficiently search the configuration space\. irace\[[32](https://arxiv.org/html/2607.08791#bib.bib4)\]takes an iterated racing approach, progressively eliminating poor configurations through statistical testing\.

Our work builds directly on LLaMEA\-BO and LLaMEA\-HPO, extending them to the multi\-objective setting\. This extension is non\-trivial: multi\-objective algorithms must handle Pareto dominance, hypervolume computation, objective normalization, and decomposition strategies that have no counterpart in the single\-objective case\.

## IIIMethodology

### III\.1Framework Overview

Our multi\-objective implementation LLaMEA\-MOBO, operates as an evolutionary loop in which an LLM serves as both the mutation and crossover operator\. Figure[1](https://arxiv.org/html/2607.08791#S3.F1)illustrates the pipeline\. At each generation, the framework:

![Refer to caption](https://arxiv.org/html/2607.08791v1/x1.png)Figure 1:Overview of the LLaMEA\-MOBO framework\. An*initiation*stage compiles the start prompt from a problem specification and an example algorithm \(with its configuration space\)\. The*LLM\-driven optimization loop*then synthesizes candidate algorithms with Gemini 2\.5 Flash, filters duplicates and non\-compiling code, tunes hyper\-parameters with SMAC, evaluates each candidate on the benchmark suite, and feeds the resulting fitness \(and any error information\) back through the session history until the stopping criterion is met\.1. 1\.Prompts the LLMwith the parent algorithm’s source code, its fitness feedback \(normalized hypervolume with one standard deviation and execution time\), and any runtime errors from the previous evaluation\. For mutation, a single parent is provided with the instruction to*refine the strategy of the selected solution to improve it*; for crossover, two parents are provided with the instruction to*combine the selected solutions into a new solution and then refine it*, additionally analyzing and avoiding any runtime errors reported for the parents\.
2. 2\.Extracts the response, which must follow a structured format: a natural\-language description, a justification of design choices, the complete Python implementation, and a hyperparameter search space definition\.
3. 3\.Detects duplicatesby stripping comments and comparing against all previously generated algorithms usingdifflib\.SequenceMatcherwith a 0\.99 similarity threshold\. Duplicates reuse cached fitness values\.
4. 4\.Optimizes hyperparametersvia SMAC \(Section[III\.3](https://arxiv.org/html/2607.08791#S3.SS3)\), tuning the algorithm’s configurable parameters across multiple problem instances\.
5. 5\.Evaluates fitnessas the normalized hypervolume averaged across benchmark problems, and feeds the result back to the LLM for the next generation\.

Each LLM\-generated algorithm must conform to a fixed object\-oriented interface\. The constructor receives the evaluation budget, decision\-space dimensionality, and variable bounds together with any tunable hyperparameters, while a separate callable entry point executes the optimization loop given a black\-box objective function and returns its final approximation of the Pareto front\. Crucially, the number of objectives is not supplied as an input: algorithms must infer it at runtime from the first function evaluation, which ensures generality across bi\-objective and many\-objective problems without manual reconfiguration\.

### III\.2Evolutionary Strategies

We study several evolutionary strategy configurations\. The\(μ\+λ\)\(\\text\{$\\mu$\}\+\\text\{$\\lambda$\}\)\-ES\[[4](https://arxiv.org/html/2607.08791#bib.bib22)\]is an elitist selection strategy where off\-springs are produced through mutation \(single parent\) or crossover \(two or more parents\)\. In particular, we examine the\(1\+1\)\(1\+1\)\-ESand\(4\+16\)\(4\+16\)\-EScases\. The first, maintains a single parent and generates one offspring per generation, which leads to a sequential refinement of the generated algorithms\. The second strategy, maintains a population ofμ=4\\mu=4parents and in each generation producesλ=16\\lambda=16off\-springs\. Under elitist, the best44individuals from the combined parent\-offspring pool survive to the next generation\. This configuration favors the exploration of diverse generated algorithms, trading\-off the exploitative sequential refinement of\(1\+1\)\(1\+1\)\-ES\. We examine also a non\-elitist evolutionary strategy\(μ,λ\)\(\\text\{$\\mu$\},\\text\{$\\lambda$\}\)\-ES\[[4](https://arxiv.org/html/2607.08791#bib.bib22)\]withμ=8\\mu=8andλ=16\\lambda=16\. This strategy allows for more exploration and smaller probability of getting stuck in a local optima but also means slower convergence\. Finally, across all three configurations, parents are ranked by fitness \(mean normalized hypervolume\) and offspring are generated from this ranked set with reuse permitted: each offspring applies crossover with probability0\.60\.6and mutation otherwise, drawing two parents for crossover and a single parent for mutation\. The\(1\+1\)\(1\{\+\}1\)\-ES, which maintains a single parent, therefore relies on mutation alone\.

### III\.3Hyperparameter Optimization

Each generated algorithm defines a set of tunable hyperparameters \(e\.g\., batch size, kernel parameters, number of weight vectors\) together with a configuration space specifying their admissible ranges and types\. The LLM is instructed to emit this configuration space alongside the algorithm code, as a Python dictionary whose keys are hyperparameter names and whose values encode the parameter type and search range: a two\-tuple\(ℓ,u\)\(\\ell,u\)denotes a continuous or integer range between boundsℓ\\ellanduuwith uniform sampling; a three\-tuple\(ℓ,u,"log"\)\(\\ell,u,\\texttt\{"log"\}\)denotes a log\-uniform range, used for quantities such as learning rates or regularization strengths that span several orders of magnitude; and a list of values denotes a categorical choice\. To keep the search tractable and the configurations comparable across candidates, the LLM is constrained to declare between three and seven hyperparameters per algorithm, to use log\-scaled ranges where appropriate, and to exclude problem\-specific parameters such as budget, dimensionality, and bounds, which are fixed by the benchmark and not optimized\. The resulting dictionary is parsed into a configuration\-space object compatible with the SMAC hyperparameter optimizer used in the next step\.

Following the LLaMEA\-HPO approach\[[48](https://arxiv.org/html/2607.08791#bib.bib8)\], we use SMAC\[[30](https://arxiv.org/html/2607.08791#bib.bib3)\]with multi\-fidelity optimization to search the hyperparameter space\. As in that work, each configuration is evaluated across an evenly\-spaced subset of the benchmark suite to promote generalization rather than overfitting to a single problem\. The objective minimized by SMAC is1−h¯1\-\\bar\{h\}, whereh¯\\bar\{h\}is the mean normalized hypervolume across instances, defined as:

h¯=1\|P\|​∑p∈PHV​\(Fp,rp\)∏irp,i\\bar\{h\}=\\frac\{1\}\{\|P\|\}\\sum\_\{p\\in P\}\\frac\{\\mathrm\{HV\}\(F\_\{p\},r\_\{p\}\)\}\{\\prod\_\{i\}r\_\{p,i\}\}\(1\)whereFpF\_\{p\}is the Pareto front approximation on problempp,rpr\_\{p\}is the reference point, andHV​\(⋅\)\\mathrm\{HV\}\(\\cdot\)is the hypervolume indicator\. Dividing by∏irp,i\\prod\_\{i\}r\_\{p,i\}normalizes across problems with different objective scales, ensuring that no single problem dominates the fitness signal\.

Before running the full SMAC optimization, we test the generated algorithm with a random configuration and a reduced budget, to detect any errors and avoid running SMAC on a non\-functional code\. The incumbent configuration found by SMAC is then used for the final fitness evaluation on the complete benchmark suite\.

## IVExperimental setup

### IV\.1Algorithms

We execute a total of nine runs \(three\(1\+1\)\(1\+1\)\-ES, three\(4\+16\)\(4\+16\)\-ES, and three\(8,16\)\(8,16\)\-ES\)\. The budget is defined as the number of algorithms evaluated across all complete generations rounded up to reach at least 100\. This yields 100 evaluated algorithms for\(1\+1\)\(1\+1\)\-ESand\(4\+16\)\(4\+16\)\-ES\(4 initial parents \+ 6 generations × 16 offspring\) and 104 for\(8,16\)\(8,16\)\-ES\(8 initial parents \+ 6 generations × 16 offspring\)\. From each run we select the best\-performing generated algorithm for benchmarking\. Additionally, we compare against a pool of baselines spanning a state\-of\-the\-art Bayesian\-optimization method \(qParEGO\[[9](https://arxiv.org/html/2607.08791#bib.bib15)\]\), a surrogate\-assisted optimizer \(IOC\-SAMO\-COBRA\[[10](https://arxiv.org/html/2607.08791#bib.bib25)\]\), two evolutionary algorithms \(NSGA\-II\[[11](https://arxiv.org/html/2607.08791#bib.bib23)\],NSGA\-III\[[51](https://arxiv.org/html/2607.08791#bib.bib24)\]\), and multi\-objective Random Search\. ForqParEGOwe use the implementation provided by the BoFire framework\[[15](https://arxiv.org/html/2607.08791#bib.bib41)\]\(built on BoTorch\[[2](https://arxiv.org/html/2607.08791#bib.bib16)\]\), ensuring a faithful and maintained state\-of\-the\-art baseline\.

### IV\.2Benchmark Problems

We evaluate all algorithms \(both LLaMEA\-generated and established baselines\) across three phases\. Phase 1 ranks the nine LLaMEA\-generated algorithms against one another; Phase 2 compares the top three of them—together with the development\-found MOEAD\-EI Hybrid—against five established baselines; and Phase 3 evaluates those nine algorithms on unseen real\-world problems\. The first two phases use the synthetic suite; the third uses real\-world engineering problems\. Synthetic benchmark suites are the de facto standard in the MOO community because they offer known Pareto fronts, controllable difficulty axes \(multimodality, non\-convexity, disconnectedness, deceptiveness, bias\), and reproducible conditions across studies\[[59](https://arxiv.org/html/2607.08791#bib.bib38),[12](https://arxiv.org/html/2607.08791#bib.bib39),[22](https://arxiv.org/html/2607.08791#bib.bib40)\]\. We adopt three such suites and complement them with a curated set of real\-world engineering problems to assess generalization\.

All problems are treated as black\-box, with a fixed budget of 400 expensive function evaluations per run and 5 independent repetitions per algorithm–problem pair to account for stochasticity\. Performance is measured by the hypervolume \(HV\) indicator\[[61](https://arxiv.org/html/2607.08791#bib.bib36),[60](https://arxiv.org/html/2607.08791#bib.bib37)\], computed at each evaluation step relative to a fixed reference point\.

The synthetic suite combines twelve problems drawn from three established families, all sourced from thepymoolibrary\[[6](https://arxiv.org/html/2607.08791#bib.bib26)\]:

- •ZDT\[[59](https://arxiv.org/html/2607.08791#bib.bib38)\]: bi\-objective problems with controllable convexity, multimodality, and connectedness of the Pareto front\. We include*ZDT1*\(convex front\),*ZDT2*\(concave\),*ZDT3*\(disconnected\),*ZDT4*\(highly multimodal\), and*ZDT6*\(non\-uniform Pareto density\)\.
- •DTLZ\[[12](https://arxiv.org/html/2607.08791#bib.bib39)\]: a scalable suite designed for arbitrary numbers of objectives and decision variables, with parameterized features such as multimodality \(DTLZ1, DTLZ4\) and disconnected Pareto fronts \(DTLZ7\)\. We use the tri\-objective configurations*DTLZ1, DTLZ2, DTLZ4*, and*DTLZ7*\.
- •WFG\[[22](https://arxiv.org/html/2607.08791#bib.bib40)\]: problems featuring bias, parameter dependency, and non\-separability, providing a stiffer test than ZDT/DTLZ\. We include*WFG4*and*WFG9*\(bi\-objective\) and*WFG7*\(tri\-objective\)\.

The selected twelve problems span decision\-space dimensionalities from 5 to 30 variables and both bi\- and tri\-objective settings\. Complete dimensions, objective counts, and reference points used for the HV calculation are listed in Table[1](https://arxiv.org/html/2607.08791#S4.T1)\.

For the real\-world generalization phase, we use the three unconstrained problems of the RE benchmark suite\[[45](https://arxiv.org/html/2607.08791#bib.bib27)\]:*RE21*\(a four\-bar truss design balancing weight and deflection\),*RE34*\(a rocket\-injector design with three competing performance objectives\), and*RE37*\(a gear\-train design under three accuracy and weight criteria\), summarized in Table[1](https://arxiv.org/html/2607.08791#S4.T1)\. The RE suite is curated from peer\-reviewed engineering case studies and exhibits the irregular, non\-smooth objective geometry that synthetic benchmarks typically lack, making it a natural test for whether algorithms developed against synthetic problems transfer to genuine deployment scenarios\.

### IV\.3Benchmark Phases

In the first phase, we benchmark the best\-performing generated algorithm from each LLaMEA run against the synthetic problems listed in Table[1](https://arxiv.org/html/2607.08791#S4.T1)\. This suite is designed to be diverse, covering both low\- and high\-dimensional decision spaces, as well as bi\- and tri\-objective settings\. Importantly, only a subset of these problems—ZDT1–4, ZDT6, and DTLZ1\-2,4 problems—were used during the LLaMEA generation phase; the remaining problems are held out and unseen by the algorithms during evolution, allowing us to assess generalization\.

Table 1:Benchmark problems: twelve synthetic problems \(ZDT, DTLZ, WFG; used in Phases 1–2\) and three real\-world engineering problems \(RE suite\[[45](https://arxiv.org/html/2607.08791#bib.bib27)\]; Phase 3\)\.dd: decision\-space dimension;mm: number of objectives\. For the RE problems, reference points are set at1\.1×1\.1\\timesthe 95th percentile of10,00010\{,\}000uniform random samples per objective\.In the second phase, we gather the three best performing generated algorithms from the first phase and benchmark them against all baselines on the same problem set of phase 1\. The best algorithms of first phase come after ranking all nine generated algorithms by their mean normalized hypervolume across all phase 1 problems and repeats\. Specifically, for each problem we normalize the final\-epoch HV of each algorithm by the maximum HV achieved by any algorithm on that problem, ensuring that problems of vastly different scales \(e\.g\. ZDT4 vs\. ZDT1\) contribute equally to the ranking\. We emphasize that this is a*within\-phase*relative score: the per\-problem maximum is taken over the algorithms present in each phase and recomputed separately for Phase 1 \(the nine generated algorithms\) and Phases 2–3 \(the generated algorithms together with the baselines\)\. It is therefore distinct from the reference\-point\-based search fitness of Section[III\.3](https://arxiv.org/html/2607.08791#S3.SS3), and the normalized HV of a given algorithm is not directly comparable across phases—for instance, Improved\-Scalarized\-EI reads0\.8550\.855in Phase 1 but0\.8110\.811in Phase 2, as the strong baselines raise the per\-problem maxima\. The three best algorithms from this procedure are Improved\-Scalarized\-EI, RF\-LCB\-PBI, and RF\-ParEGO\-Batch \(see Appendix[A](https://arxiv.org/html/2607.08791#Ax1.SS1)\), achieving mean normalized HVs of 0\.855, 0\.795, and 0\.756 respectively\.

The third and final phase benchmarks the same algorithms from phase 2 but this time on real\-world unconstrained multi\-objective problems of RE benchmark suite\[[45](https://arxiv.org/html/2607.08791#bib.bib27)\]as seen in Table[1](https://arxiv.org/html/2607.08791#S4.T1)\. Reference points for RE problems are set to1\.1×1\.1\\timesthe 95th percentile of each objective, estimated from10,00010,000uniformly sampled candidate solutions\.

## VResults

### V\.1LLaMEA\-Generated Algorithms

Across the nine evolutionary runs, LLaMEA generated a total of approximately 900 candidate algorithms\. These algorithms span a broad range of surrogate models and acquisition functions, including Gaussian Process\-based surrogates with decomposition strategies\[[1](https://arxiv.org/html/2607.08791#bib.bib28),[18](https://arxiv.org/html/2607.08791#bib.bib29),[33](https://arxiv.org/html/2607.08791#bib.bib30)\], tree\-ensemble regressors \(random forests and gradient boosting\)\[[5](https://arxiv.org/html/2607.08791#bib.bib31),[26](https://arxiv.org/html/2607.08791#bib.bib32)\], windowed model updates, and hybrid evolutionary\-Bayesian acquisition schemes\[[49](https://arxiv.org/html/2607.08791#bib.bib33),[28](https://arxiv.org/html/2607.08791#bib.bib34)\]\. This diversity of strategies demonstrates that LLaMEA does not favor any particular algorithmic type but rather explores different regions of the algorithmic design space\.

![Refer to caption](https://arxiv.org/html/2607.08791v1/figures/fig3_phase1_grid.png)Figure 2:Convergence of the nine LLaMEA\-generated algorithms \(Phase 1\) on representative problems from each family\. Shaded bands denote±1\\pm 1std over five repeats\.![Refer to caption](https://arxiv.org/html/2607.08791v1/figures/fig4_phase2_grid.png)Figure 3:Convergence of all nine algorithms—the four LLaMEA\-generated \(the top\-3 plus MOEAD\-EI Hybrid\) and five baselines/SOTA—on representative Phase\-2 problems\. Shaded bands denote±1\\pm 1std over five repeats\.As described in Section[IV](https://arxiv.org/html/2607.08791#S4), the three best algorithms are selected by mean normalized HV across all Phase 1 problems\. The ranking is as follows: Improved\-Scalarized\-EI \(0\.855\), RF\-LCB\-PBI \(0\.795\), and RF\-ParEGO\-Batch \(0\.756\); we examine these designs and the mechanisms behind their performance in Section[VI\.1](https://arxiv.org/html/2607.08791#S6.SS1)\. In addition to the systematically selected algorithms, during the development of our experimentation we discovered MOEAD\-EI Hybrid, an algorithm that proved the strongest overall on the synthetic benchmark\. Given its exceptional performance, we retain it as a generated candidate throughout the benchmark comparison and analyze it alongside the selected algorithms in Section[VI\.1](https://arxiv.org/html/2607.08791#S6.SS1)\.

### V\.2Benchmarking Results

#### V\.2\.1Phase 1: Comparison Among Generated Algorithms

In Phase 1, the nine best\-performing generated algorithms are evaluated on the twelve synthetic benchmark problems listed in Table[1](https://arxiv.org/html/2607.08791#S4.T1), using a budget of 400 function evaluations and 5 independent repeats per problem\. Results are summarized in terms of HV convergence curves and final normalized HV\.

Improved\-Scalarized\-EI achieves the highest mean normalized HV \(0\.855\), winning 8 of the 12 problems: it dominates the entire ZDT family and also leads on DTLZ7, WFG4, and WFG7\. On ZDT4 — a highly multimodal problem — it reaches a mean final HV of 258\.4, the highest among the generated algorithms \(against 218–257 for the others\)\. Its advantage is not uniform, however: the random\-forest\-based RF\-LCB\-PBI and RF\-ParEGO\-Batch lead on the harder DTLZ problems \(DTLZ1, DTLZ4\), where Improved\-Scalarized\-EI is markedly less effective, reflecting complementary strengths across the generated set\.

All nine generated algorithms complete all twelve problems given an adequate per\-run time budget; the substantial differences in their computational cost are analyzed in Section[V\.2\.4](https://arxiv.org/html/2607.08791#S5.SS2.SSS4)\.

#### V\.2\.2Phase 2: Comparison Against Baselines and State\-of\-the\-Art on Synthetic Problems

In Phase 2, the best three generated algorithms — together with the development\-found MOEAD\-EI Hybrid — are benchmarked alongside five established algorithms — Multi\-objective Random Search, NSGA\-II\[[11](https://arxiv.org/html/2607.08791#bib.bib23)\], NSGA\-III\[[51](https://arxiv.org/html/2607.08791#bib.bib24)\], IOC\-SAMO\-COBRA\[[10](https://arxiv.org/html/2607.08791#bib.bib25)\], and qParEGO\[[9](https://arxiv.org/html/2607.08791#bib.bib15)\]— on the same twelve synthetic problems under identical conditions\.

Table 2:Mean normalized hypervolume \(±\\pmstd across problems\) in Phase 2 \(12 synthetic\) and Phase 3 \(3 real\-world\)\. Bold indicates the best mean per column\.†\\daggerdenotes LLaMEA\-generated algorithms\.MOEAD\-EI Hybrid achieves the highest mean normalized HV of 0\.971, beating qParEGO on 8 of the 12 problems and outperforming every classical baseline; in the per\-problem analysis it is significantly more accurate than the state\-of\-the\-art qParEGO \(0\.869\) on 7 of the 12 problems and never significantly worse \(Section[V\.3](https://arxiv.org/html/2607.08791#S5.SS3)\)\. Among the systematically selected algorithms, Improved\-Scalarized\-EI is the strongest at 0\.811 — ahead of every classical baseline \(best: NSGA\-III, 0\.765\) and behind only qParEGO \(0\.869\) and MOEAD\-EI Hybrid \(0\.971\) — whereas the random\-forest\-based RF\-LCB\-PBI \(0\.700\) and RF\-ParEGO\-Batch \(0\.693\) perform on par with the classical baselines on the synthetic suite\. NSGA\-II performs competitively at 0\.691, reflecting the well\-known effectiveness of evolutionary methods on synthetic benchmarks with moderate budget\. The classical baselines \(NSGA\-II, NSGA\-III, IOC\-SAMO\-COBRA, Random Search\) require negligible computation — under one second per repeat — but rank below the strongest generated algorithm, confirming that surrogate\-assisted search provides meaningful gains within a budget of 400 evaluations\. Random Search achieves a mean normalized HV of 0\.600, establishing the lower bound of the comparison\.

#### V\.2\.3Phase 3: Generalization to Real\-World Problems

Phase 3 evaluates all nine algorithms on three unconstrained real\-world engineering optimization problems from the RE benchmark suite\[[45](https://arxiv.org/html/2607.08791#bib.bib27)\]\(Table[1](https://arxiv.org/html/2607.08791#S4.T1)\), which were unseen during the LLaMEA generation phase and during Phase 1 and Phase 2 benchmarking\.

![Refer to caption](https://arxiv.org/html/2607.08791v1/figures/fig5_phase3_grid.png)Figure 4:Convergence on the three real\-world RE problems \(Phase 3\) for all nine algorithms\. Shaded bands denote±1\\pm 1std over five repeats\.Here the ordering changes\. Improved\-Scalarized\-EI attains the highest mean normalized HV \(0\.985\), narrowly ahead of the state\-of\-the\-art qParEGO \(0\.971\); MOEAD\-EI Hybrid follows at 0\.926—third overall, ahead of RF\-ParEGO\-Batch \(0\.902\) and every classical baseline—so two generated algorithms join qParEGO at the top of the real\-world ranking\. Notably, NSGA\-II remains competitive \(0\.849\) despite its complete absence of a surrogate model, consistent with prior findings that population diversity is particularly valuable in small\-dimensional, multi\-peaked landscapes\. As the real\-world suite contains only three problems, we report these results descriptively; the rank\-based test over it is underpowered \(Section[V\.3](https://arxiv.org/html/2607.08791#S5.SS3)\)\.

#### V\.2\.4Time–Accuracy Trade\-off

A central contribution of this work is the identification of LLaMEA\-generated algorithms that match or exceed state\-of\-the\-art accuracy at a fraction of the computational cost\. Wall\-clock time is measured as the total execution time for one complete optimization run of 400 function evaluations on a single problem instance, averaged across all problems and independent repetitions within the respective benchmark phase\. All benchmark runs were executed in parallel across1010CPU cores using a process\-pool executor; in a controlled comparison against isolated single\-process execution, parallel timings were inflated by approximately 30% due to memory\-bandwidth contention and Python global\-interpreter\-lock overhead\. We report parallel timings throughout, as they reflect the practical deployment scenario; the relative ranking between algorithms is unaffected by this inflation, since all algorithms are subject to the same contention\. Each algorithm is timed in its own pass over the problem suite, so every method has the full ten\-core pool available during its measurement and the timings are directly comparable across algorithms\.

Figure[5](https://arxiv.org/html/2607.08791#S5.F5)plots mean normalized HV against mean runtime per repeat for both Phase 2 and Phase 3, making the efficiency landscape explicit\.

![Refer to caption](https://arxiv.org/html/2607.08791v1/figures/fig6_time_accuracy.png)Figure 5:Time–accuracy trade\-off on \(a\) the twelve synthetic and \(b\) the three real\-world \(RE\) problems\. Horizontal axis: mean wall\-clock time per run \(log scale\); vertical axis: mean normalized hypervolume\. The LLaMEA\-generated algorithms occupy the efficient region\.Three tiers emerge clearly\. First, the classical baselines \(NSGA\-II,NSGA\-III,IOC\-SAMO\-COBRA,Random Search\) operate at under one second per repeat but achieve moderate HV, forming a fast\-but\-limited group\. Second,qParEGOis the most expensive method by a wide margin \- a mean of 11,304 seconds per run on synthetic problems and 1,704 seconds on real\-world problems\. The LLaMEA\-generated algorithms occupy the efficient region in between, spanning the accuracy–cost frontier\. MOEAD\-EI Hybrid matches or exceeds qParEGO’s accuracy at a small fraction of the cost: on the synthetic suite it attains the best accuracy of any method \(0\.971\) at only 188 seconds per run—a 60\.0×\\timesreduction—and on the real\-world problems 0\.926 at 79 seconds \(21\.5×\\times\)\. Improved\-Scalarized\-EI is the most accurate method on the real\-world problems \(0\.985\) at 499 seconds, a 3\.4× reduction relative to qParEGO \(and 12\.4× on synthetic\)\. At the cheap extreme, RF\-ParEGO\-Batch reaches competitive accuracy \(0\.902 on real\-world, ahead of every classical baseline\) at just 2–5 seconds per run — two\-to\-three orders of magnitude faster than qParEGO \(754× on real\-world, 2,310× on synthetic\)\.

### V\.3Statistical Significance

![Refer to caption](https://arxiv.org/html/2607.08791v1/figures/fig7_cd_diagram.png)Figure 6:Critical\-difference diagram \(Nemenyi,α=0\.05\\alpha=0\.05\) of mean ranks across the twelve synthetic problems\. Lower rank \(left\) is better; methods joined by a bar are not significantly different \(CD = 3\.47\)\. MOEAD\-EI Hybrid ranks first \(1\.83\), ahead of qParEGO \(2\.58\) and Improved\-Scalarized\-EI \(3\.92\)\.Following the per\-problem evaluation protocol of the BLADE benchmarking suite\[[47](https://arxiv.org/html/2607.08791#bib.bib47)\], we assess significance problem by problem: for each problem a two\-sample Welchtt\-test\[[50](https://arxiv.org/html/2607.08791#bib.bib48)\]over the five independent runs \(α=0\.05\\alpha=0\.05\) compares a generated algorithm against qParEGO, with no multiple\-comparison correction \(as in BLADE\)\. We focus on the two competitive generated algorithms; each comparison is additionally summarized across the synthetic suite with a Wilcoxon signed\-rank test\[[52](https://arxiv.org/html/2607.08791#bib.bib44)\]\.

On the synthetic suite,*MOEAD\-EI Hybrid*is significantly more accurate than qParEGO on 7 of the 12 problems, never significantly worse, and statistically indistinguishable on the remaining 5; across the suite the improvement is significant \(Wilcoxon signed\-rank,p=0\.043p=0\.043\)\. On the three real\-world problems it is statistically on par with qParEGO—indistinguishable on re21 and re34 and significantly worse only on re37\.

*Improved\-Scalarized\-EI*matches qParEGO on the synthetic suite: it is significantly better on 4 problems and significantly worse on 4, indistinguishable on the other 4, with no significant suite\-level difference \(Wilcoxon,p=0\.52p=0\.52\)\. On the real\-world problems, however, it is the single most accurate method, significantly outperforming qParEGO on 2 of the 3 \(re21 and re34\) and worse only on re37\.

The two random\-forest variants \(RF\-LCB\-PBI and RF\-ParEGO\-Batch\) are significantly*less*accurate than qParEGO on most problems \(10 of 12 synthetic and all three real\-world\), trading accuracy for one\-to\-three orders of magnitude lower runtime\.

The advantage in computational cost is unequivocal\. Every generated algorithm is significantly faster than qParEGO on all 15 problems \(Wilcoxon signed\-rank over the 15 problems\[[52](https://arxiv.org/html/2607.08791#bib.bib44)\],p=6\.1×10−5p=6\.1\\times 10^\{\-5\}in every case\); the per\-phase magnitudes of these reductions are reported in the time–accuracy analysis above\.

At the level of the whole suite, a Friedman test\[[19](https://arxiv.org/html/2607.08791#bib.bib43)\]on the synthetic problems confirms that the nine algorithms differ significantly \(χ2=49\.91\\chi^\{2\}=49\.91,p=4\.3×10−8p=4\.3\\times 10^\{\-8\}\) and ranks MOEAD\-EI Hybrid first \(1\.831\.83\), ahead of qParEGO \(2\.582\.58\) and Improved\-Scalarized\-EI \(3\.923\.92\)\. The corresponding Nemenyi critical\-difference diagram \(Fig\.[6](https://arxiv.org/html/2607.08791#S5.F6),CD=3\.47\\mathrm\{CD\}=3\.47\)\[[36](https://arxiv.org/html/2607.08791#bib.bib46),[14](https://arxiv.org/html/2607.08791#bib.bib42)\]separates MOEAD\-EI Hybrid from random search and the weaker baselines but not from qParEGO; as this all\-pairs test has low power on small suites\[[14](https://arxiv.org/html/2607.08791#bib.bib42)\], the per\-problem analysis above gives the more direct head\-to\-head comparison with the state of the art\.

### V\.4Effect of Hyperparameter Optimization

To isolate the contribution of the SMAC hyperparameter optimization integrated into the generation loop, we compare each of the four highlighted generated algorithms in its deployed SMAC\-tuned configuration against the hyperparameters originally proposed by the LLM \(its code defaults\)\. Table[3](https://arxiv.org/html/2607.08791#S5.T3)reports the mean normalized hypervolume under both settings\.

On the synthetic suite—the distribution SMAC tunes on—hyperparameter optimization yields consistent gains for all four algorithms \(\+0\.9%\+0\.9\\%to\+5\.2%\+5\.2\\%;\+1\.6%\+1\.6\\%for MOEAD\-EI Hybrid\), largest for the cheaper random\-forest designs\. On the held\-out real\-world problems, tuning the three systematically generated algorithms produces no improvement \(−0\.3%\-0\.3\\%to−0\.7%\-0\.7\\%, within run\-to\-run variation\), whereas for MOEAD\-EI Hybrid it yields a large gain \(\+7\.8%\+7\.8\\%,0\.859→0\.9260\.859\\to 0\.926\) on a configuration that transfers to the real\-world problems\. Hyperparameter optimization is thus algorithm\-dependent—a modest in\-distribution refinement for most designs, and a larger benefit for MOEAD\-EI Hybrid’s real\-world generalization\.

Table 3:Effect of SMAC hyperparameter optimization: mean normalized hypervolume with the LLM\-proposed default hyperparameters versus the SMAC\-tuned incumbent, for the four highlighted LLaMEA\-generated algorithms\. Normalization uses the same per\-problem\-max denominator as Table[2](https://arxiv.org/html/2607.08791#S5.T2)\.

## VIDiscussion

![Refer to caption](https://arxiv.org/html/2607.08791v1/figures/fig2_llamea_search_progress_9runs.png)Figure 7:Best\-so\-far search fitness \(mean normalized HV on the training suite\) over the first 100 evaluated algorithms, for all nine evolutionary runs \(three per evolution strategy; color denotes strategy, line style the run\)\. The\(4\+16\)\(4\+16\)\-ESruns reach the highest fitness \(≈0\.70\\approx 0\.70\) and produce the top\-ranked Improved\-Scalarized\-EI, while the\(1\+1\)\(1\+1\)\-ESand\(8,16\)\(8,16\)\-ESruns plateau lower \(≈0\.53\\approx 0\.53\)\.### VI\.1Selected Algorithms Analysis

To understand not only what the selected algorithms are but how they were discovered, we trace each one back to its origin through the parent links recorded during the search—so the lineages reported here are the actual ancestry of each design, not a post\-hoc reconstruction\. All four emerge within a single run in at most five generations, each improving markedly over its lineage root \(Fig\.[7](https://arxiv.org/html/2607.08791#S6.F7)\)—directed refinement rather than one\-shot sampling\. Despite being generated automatically, all assemble recognizable, well\-motivated building blocks, which we relate below to established components of Bayesian and decomposition\-based multi\-objective optimization\.

- •Reached by mutation over five monotonic generations \(search fitness0\.485→0\.6980\.485\\rightarrow 0\.698; EvolutionaryAcquisition → ESI → ESI\-Vec → ESI\-ImprovedEA → Improved\-Scalarized\-EI\) that specialize the acquisition and its inner optimizer step by step,*Improved\-Scalarized\-EI*pairs a Matérn\-2\.5 Gaussian\-process surrogate\[[53](https://arxiv.org/html/2607.08791#bib.bib49)\]with an Expected\-Improvement acquisition\[[24](https://arxiv.org/html/2607.08791#bib.bib13)\]applied to a scale\-invariant Tchebycheff scalarization of the objectives\[[35](https://arxiv.org/html/2607.08791#bib.bib51)\], and maximizes it with a vectorized evolutionary inner optimizer that uses tournament selection\[[16](https://arxiv.org/html/2607.08791#bib.bib56)\]\. The combination of a calibrated GP posterior and EI’s principled exploration–exploitation trade\-off is especially effective on the smooth, low\-dimensional real\-world problems, where it is the single most accurate method; the vectorized EA keeps the acquisition step inexpensive, and a fit\-failure fallback makes it robust\.
- •The best design found during development \(search fitness0\.8800\.880\),*MOEAD\-EI Hybrid*arose at generation 2 by crossover of a fast\-but\-unstable MOEA/D–GP hybrid and a strong\-but\-slow ParEGO–GP method, inheriting EI’s accuracy while resolving ParEGO’s cost\. It embeds independent GP surrogates in an MOEA/D decomposition framework\[[56](https://arxiv.org/html/2607.08791#bib.bib50)\], driving each scalar subproblem with an augmented\-Tchebycheff Expected\-Improvement acquisition\[[43](https://arxiv.org/html/2607.08791#bib.bib52),[35](https://arxiv.org/html/2607.08791#bib.bib51)\]and proposing candidates with genetic operators over a fixed set of weight vectors\. Decomposition into many scalar subproblems suits the higher\-dimensional synthetic suite—where it is the most accurate method overall—and it remains competitive on the low\-dimensional real\-world problems, ranking third in Phase 3\.
- •RF\-LCB\-PBI \(search fitness0\.489→0\.5300\.489\\rightarrow 0\.530\) swaps the surrogate fromkk\-nearest\-neighbors to a random\-forest ensemble\[[7](https://arxiv.org/html/2607.08791#bib.bib55),[23](https://arxiv.org/html/2607.08791#bib.bib54)\]and folds a lower\-confidence bound\[[42](https://arxiv.org/html/2607.08791#bib.bib53)\]into the penalty\-based boundary\-intersection \(PBI\) scalarization of MOEA/D\[[56](https://arxiv.org/html/2607.08791#bib.bib50)\]\. The ensemble variance provides cheap, robust uncertainty estimates that make the method orders of magnitude faster than its GP\-based counterparts; the price is a less smooth surrogate, which yields competitive rather than leading accuracy on the continuous synthetic problems\.
- •RF\-ParEGO\-Batch \(search fitness0\.464→0\.5160\.464\\rightarrow 0\.516\) keeps the random\-forest surrogate and evolves the acquisition from a multi\-objective LCB into a batched ParEGO scalarization\[[25](https://arxiv.org/html/2607.08791#bib.bib1)\], drawing fresh random scalarization weights each iteration and proposing candidates in batches\. This makes it the cheapest method in the study—two to three orders of magnitude faster than qParEGO—deliberately trading surrogate fidelity for throughput while remaining ahead of every classical baseline on the real\-world problems\.

Two broader observations follow\. First, the lineages are short and interpretable—every generation corresponds to a named, sensible algorithmic change, with mutation specializing a single module and crossover combining two complementary parents—so the LLM\-driven search is legible to a practitioner rather than a black box\.

Second, the strongest designs implicitly balance surrogate quality against acquisition cost—for example MOEAD\-EI Hybrid’s Tchebycheff decomposition with independent GP surrogates, or the random\-forest variants’ trade of fidelity for speed\. This balance is not explicitly optimized by the LLM; it emerges from evaluating candidate algorithms under a fixed evaluation budget, which naturally penalizes slow\-converging or computationally expensive designs and rewards those that reach high hypervolume quickly\.

### VI\.2Limitations

Several limitations of the current study should be acknowledged\. First, the surrogate\-assisted methods differ by orders of magnitude in computational cost: at a400400\-evaluation budget,qParEGOalone requires hours per run\. This wall\-clock expense—not any methodological limit—is what capped the number of repeats and benchmark problems we could include; a broader study would require substantially more compute\. Second, the evaluation budget of400400function evaluations, while standard for expensive black\-box optimization, may not fully reflect the behavior of surrogate\-assisted methods at larger scales\. Third, the real\-world evaluation in Phase 3 is limited to three unconstrained engineering problems; broader conclusions about generalization to constrained, noisy, or higher\-dimensional real\-world settings require further investigation\. Fourth, the quality and diversity of generated algorithms may be sensitive to the choice of LLM backbone and prompt design, aspects that we did not systematically vary\.

A further, more subtle hazard concerns*stochasticity*\. We did not initially anticipate that, absent an explicit instruction, an LLM may generate*deterministic*heuristics—for instance by fixing the random seed internally\. When benchmarking stochastic optimizers this is dangerous: a fixed\-seed algorithm reports zero run\-to\-run variance, which both understates uncertainty and risks overfitting to a single fortunate trajectory\. We detected this behavior for*Improved\-Scalarized\-EI*and addressed it post hoc by re\-running it across five independent seeds, restoring a fair variance estimate\. A future generation loop should include informed prompts that explicitly require stochastic behavior and proper seed handling, together with an automated validation check that flags zero\-variance candidates during the search\.

Finally, the development\-found MOEAD\-EI Hybrid required one mechanical correction before benchmarking: on the lower\-dimensional problems its generated code initialized onlyninitn\_\{\\mathrm\{init\}\}solutions yet indexed a decomposition population ofNpop\>ninitN\_\{\\mathrm\{pop\}\}\>n\_\{\\mathrm\{init\}\}subproblems, raising an out\-of\-bounds error\. We cap the population with a single line,self\.population\_size = min\(self\.population\_size, self\.n\_init\), so the algorithm runs consistently across all problem dimensions; the change is purely mechanical and leaves its surrogate and acquisition logic untouched\.

### VI\.3Future Work

Several directions remain open\. Extending LLaMEA to constrained multi\-objective problems would substantially broaden the practical applicability of the generated algorithms\. Investigating larger evaluation budgets and a wider suite of real\-world benchmarks—including noisy and mixed\-integer settings—would provide a more complete picture of where LLaMEA\-generated algorithms succeed and fail\. Furthermore, during the benchmark evaluation phases the discovered algorithms are deployed with the hyperparameter configurations produced during the LLaMEA search; running problem\-specific hyperparameter tuning at deployment time \(e\.g\. re\-invoking SMAC per target problem\) could close this gap and further improve their competitiveness on unseen problem instances\. Beyond tuning, a further shift is from generating*general\-purpose*algorithms—our present goal, where a single discovered algorithm must perform well across a suite—to generating*specialized*solvers for a particular target problem or family\. Where evaluations are cheap enough to amortize a dedicated generation run, LLaMEA could be directed at one problem and optimize an algorithm specifically for it; understanding when such specialization outperforms a strong general\-purpose solver, and which design mechanisms drive the gain, is a promising direction\. More broadly, applying the LLaMEA framework to other algorithm families—such as multi\-objective surrogate\-assisted evolutionary algorithms or meta\-heuristics—remains a promising direction\. Finally, a systematic comparison of LLaMEA against other LLM\-driven automated algorithm\-design methods, under a common benchmarking protocol, is a natural next step toward understanding the relative strengths of different generation paradigms\.

## VIIConclusion

In this work we applied LLaMEA to the automatic generation of multi\-objective Bayesian optimization algorithms, showing that large language model\-driven evolutionary search can discover competitive and computationally efficient algorithms without any hand\-crafted design\. Across nine evolutionary runs spanning three evolution strategies, LLaMEA produced approximately900900candidates covering fundamentally different surrogate and acquisition\-function designs\. The development\-foundMOEAD\-EI Hybridattained the highest mean normalized hypervolume on the twelve synthetic benchmark problems \(0\.971\)—significantly more accurate than the state\-of\-the\-artqParEGOon 7 of 12 and never significantly worse—while running on average60×60\\timesfaster; it also generalizes to the real\-world problems \(0\.926, third overall,≈\\approx21\.5×21\.5\\timesfaster\)\. On the held\-out real\-world RE problems, the systematically generatedImproved\-Scalarized\-EIwas the single most accurate method, significantly outperformingqParEGOon two of the three problems at3\.4×3\.4\\timeslower wall\-clock cost\. Together these results show that LLaMEA can identify structurally sound algorithm designs that match or exceed state\-of\-the\-art accuracy at a fraction of the computational cost—a balance difficult to achieve through manual design alone\.

## References

- \[1\]L\. Bajer, Z\. Pitra, J\. Repickỳ, and M\. Holeňa\(2019\)Gaussian process surrogate models for the cma evolution strategy\.Evolutionary computation27\(4\),pp\. 665–697\.External Links:[Link](https://direct.mit.edu/evco/article-abstract/27/4/665/94974/Gaussian-Process-Surrogate-Models-for-the-CMA)Cited by:[§V\.1](https://arxiv.org/html/2607.08791#S5.SS1.p1.1)\.
- \[2\]M\. Balandat, B\. Karrer, D\. R\. Jiang, S\. Daulton, B\. Letham, A\. G\. Wilson, and E\. Bakshy\(2020\)BoTorch: a framework for efficient Monte\-Carlo Bayesian optimization\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 21524–21538\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/f5b1b89d98b7286673128a5fb112cb9a-Abstract.html)Cited by:[§II\.1](https://arxiv.org/html/2607.08791#S2.SS1.p1.1),[§II\.1](https://arxiv.org/html/2607.08791#S2.SS1.p2.1),[§IV\.1](https://arxiv.org/html/2607.08791#S4.SS1.p1.6)\.
- \[3\]S\. Belakaria, A\. Deshwal, and J\. R\. Doppa\(2019\)Max\-value entropy search for multi\-objective Bayesian optimization\.Advances in neural information processing systems32\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/82edc5c9e21035674d481640448049f3-Abstract.html)Cited by:[§II\.1](https://arxiv.org/html/2607.08791#S2.SS1.p3.1)\.
- \[4\]H\. Beyer and D\. V\. Arnold\(2001\)Theory of evolution strategies—a tutorial\.Theoretical aspects of evolutionary computing,pp\. 109–133\.External Links:[Link](https://link.springer.com/chapter/10.1007/978-3-662-04448-3_6)Cited by:[§III\.2](https://arxiv.org/html/2607.08791#S3.SS2.p1.12)\.
- \[5\]B\. Bhattacharjee, K\. Unni, and M\. Pratap\(2024\)Bayesian\-optimized extreme gradient boosting models for classification problems: an experimental analysis of product return case\.Journal of Systems and Information Technology26\(4\),pp\. 495–527\.External Links:[Link](https://www.emerald.com/jsit/article-abstract/26/4/495/1232001/Bayesian-optimized-extreme-gradient-boosting?redirectedFrom=fulltext)Cited by:[§V\.1](https://arxiv.org/html/2607.08791#S5.SS1.p1.1)\.
- \[6\]J\. Blank and K\. Deb\(2020\)Pymoo: multi\-objective optimization in python\.IEEE Access8\(\),pp\. 89497–89509\.External Links:[Link](https://pymoo.org/index.html)Cited by:[§IV\.2](https://arxiv.org/html/2607.08791#S4.SS2.p3.1)\.
- \[7\]L\. Breiman\(2001\)Random forests\.Machine learning45\(1\),pp\. 5–32\.External Links:[Link](https://link.springer.com/article/10.1023/a:1010933404324)Cited by:[3rd item](https://arxiv.org/html/2607.08791#S6.I1.i3.p1.2)\.
- \[8\]A\. Chen, D\. Dohan, and D\. So\(2023\)EvoPrompting: language models for code\-level neural architecture search\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/184c1e18d00d7752805324da48ad25be-Abstract-Conference.html?ref=hungdu.com)Cited by:[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p1.1)\.
- \[9\]S\. Daulton, M\. Balandat, and E\. Bakshy\(2020\)Differentiable expected hypervolume improvement for parallel multi\-objective Bayesian optimization\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 9851–9864\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/6fec24eac8f18ed793f5eaad3dd7977c-Abstract.html)Cited by:[§II\.1](https://arxiv.org/html/2607.08791#S2.SS1.p1.1),[§II\.1](https://arxiv.org/html/2607.08791#S2.SS1.p2.1),[§IV\.1](https://arxiv.org/html/2607.08791#S4.SS1.p1.6),[§V\.2\.2](https://arxiv.org/html/2607.08791#S5.SS2.SSS2.p1.1)\.
- \[10\]R\. De Winter, B\. Milatz, J\. Blank, N\. Van Stein, T\. Bäck, and K\. Deb\(2024\)Parallel multi\-objective optimization for expensive and inexpensive objectives and constraints\.Swarm and Evolutionary Computation86,pp\. 101508\.External Links:[Link](https://www.sciencedirect.com/science/article/pii/S2210650224000415)Cited by:[§IV\.1](https://arxiv.org/html/2607.08791#S4.SS1.p1.6),[§V\.2\.2](https://arxiv.org/html/2607.08791#S5.SS2.SSS2.p1.1)\.
- \[11\]K\. Deb, S\. Agrawal, A\. Pratap, and T\. Meyarivan\(2000\)A fast elitist non\-dominated sorting genetic algorithm for multi\-objective optimization: NSGA\-II\.InInternational conference on parallel problem solving from nature,pp\. 849–858\.External Links:[Link](https://link.springer.com/chapter/10.1007/3-540-45356-3_83)Cited by:[§IV\.1](https://arxiv.org/html/2607.08791#S4.SS1.p1.6),[§V\.2\.2](https://arxiv.org/html/2607.08791#S5.SS2.SSS2.p1.1)\.
- \[12\]K\. Deb, L\. Thiele, M\. Laumanns, and E\. Zitzler\(2005\)Scalable test problems for evolutionary multiobjective optimization\.InEvolutionary multiobjective optimization: theoretical advances and applications,pp\. 105–145\.External Links:[Link](https://link.springer.com/chapter/10.1007/1-84628-137-7_6)Cited by:[2nd item](https://arxiv.org/html/2607.08791#S4.I1.i2.p1.1),[§IV\.2](https://arxiv.org/html/2607.08791#S4.SS2.p1.1)\.
- \[13\]K\. Deb\(2011\)Multi\-objective optimisation using evolutionary algorithms: an introduction\.InMulti\-objective evolutionary optimisation for product design and manufacturing,pp\. 3–34\.External Links:[Link](https://link.springer.com/chapter/10.1007/978-0-85729-652-8_1)Cited by:[§I](https://arxiv.org/html/2607.08791#S1.p1.1)\.
- \[14\]J\. Demšar\(2006\)Statistical comparisons of classifiers over multiple data sets\.Journal of Machine learning research7\(Jan\),pp\. 1–30\.External Links:[Link](https://www.jmlr.org/papers/volume7/demsar06a/demsar06a.pdf)Cited by:[§V\.3](https://arxiv.org/html/2607.08791#S5.SS3.p6.6)\.
- \[15\]J\. P\. Dürholt, T\. S\. Asche, J\. Kleinekorte, G\. Mancino\-Ball, B\. Schiller, S\. Sung, J\. Keupp, A\. Osburg, T\. Boyne, R\. Misener, R\. Eldred, C\. Kappatou, R\. M\. Lee, D\. Linzner, W\. S\. Costa, D\. Walz, N\. Wulkow, and B\. Shafei\(2025\)BoFire: Bayesian optimization framework intended for real experiments\.Journal of Machine Learning Research26\(204\),pp\. 1–7\.External Links:[Link](https://www.jmlr.org/papers/v26/24-1540.html),[Link](http://jmlr.org/papers/v26/24-1540.html)Cited by:[§II\.1](https://arxiv.org/html/2607.08791#S2.SS1.p1.1),[§IV\.1](https://arxiv.org/html/2607.08791#S4.SS1.p1.6)\.
- \[16\]A\. E\. Eiben and J\. E\. Smith\(2015\)Introduction to evolutionary computing\.Springer\.External Links:[Link](https://link.springer.com/book/10.1007/978-3-662-44874-8)Cited by:[1st item](https://arxiv.org/html/2607.08791#S6.I1.i1.p1.1)\.
- \[17\]M\. T\. M\. Emmerich, K\. C\. Giannakoglou, and B\. Naujoks\(2006\)Single\- and multiobjective evolutionary optimization assisted by Gaussian random field metamodels\.InIEEE Transactions on Evolutionary Computation,Vol\.10,pp\. 421–439\.External Links:[Link](https://ieeexplore.ieee.org/abstract/document/1665031)Cited by:[§II\.1](https://arxiv.org/html/2607.08791#S2.SS1.p2.1)\.
- \[18\]P\. I\. Frazier\(2018\)A tutorial on Bayesian optimization\.arXiv preprint arXiv:1807\.02811\.External Links:[Link](https://arxiv.org/abs/1807.02811)Cited by:[§V\.1](https://arxiv.org/html/2607.08791#S5.SS1.p1.1)\.
- \[19\]M\. Friedman\(1937\)The use of ranks to avoid the assumption of normality implicit in the analysis of variance\.Journal of the american statistical association32\(200\),pp\. 675–701\.External Links:[Link](https://www.tandfonline.com/doi/pdf/10.1080/01621459.1937.10503522)Cited by:[§V\.3](https://arxiv.org/html/2607.08791#S5.SS3.p6.6)\.
- \[20\]J\. Grochow\(2019\)New applications of the polynomial method: the cap set conjecture and beyond\.Bulletin of the American Mathematical Society56\(1\),pp\. 29–64\.External Links:[Link](https://pubs.ams.org/journals/bull/2019-56-01/S0273-0979-2018-01648-0)Cited by:[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p1.1)\.
- \[21\]D\. Hernández\-Lobato, J\. Hernandez\-Lobato, A\. Shah, and R\. Adams\(2016\)Predictive entropy search for multi\-objective Bayesian optimization\.InInternational Conference on Machine Learning,pp\. 1492–1501\.External Links:[Link](https://proceedings.mlr.press/v48/hernandez-lobatoa16.html)Cited by:[§II\.1](https://arxiv.org/html/2607.08791#S2.SS1.p3.1)\.
- \[22\]S\. Huband, P\. Hingston, L\. Barone, and L\. While\(2006\)A review of multiobjective test problems and a scalable test problem toolkit\.IEEE Transactions on Evolutionary Computation10\(5\),pp\. 477–506\.External Links:[Link](https://ieeexplore.ieee.org/abstract/document/1705400)Cited by:[3rd item](https://arxiv.org/html/2607.08791#S4.I1.i3.p1.1),[§IV\.2](https://arxiv.org/html/2607.08791#S4.SS2.p1.1)\.
- \[23\]F\. Hutter, H\. H\. Hoos, and K\. Leyton\-Brown\(2011\)Sequential model\-based optimization for general algorithm configuration\.InInternational conference on learning and intelligent optimization,pp\. 507–523\.External Links:[Link](https://link.springer.com/chapter/10.1007/978-3-642-25566-3_40)Cited by:[3rd item](https://arxiv.org/html/2607.08791#S6.I1.i3.p1.2)\.
- \[24\]D\. R\. Jones, M\. Schonlau, and W\. J\. Welch\(1998\)Efficient global optimization of expensive black\-box functions\.Journal of Global optimization13\(4\),pp\. 455–492\.External Links:[Link](https://link.springer.com/article/10.1023/A:1008306431147)Cited by:[§I](https://arxiv.org/html/2607.08791#S1.p1.1),[1st item](https://arxiv.org/html/2607.08791#S6.I1.i1.p1.1)\.
- \[25\]J\. Knowles\(2006\)ParEGO: a hybrid algorithm with on\-line landscape approximation for expensive multiobjective optimization problems\.IEEE Transactions on Evolutionary Computation10\(1\),pp\. 50–66\.External Links:[Link](https://ieeexplore.ieee.org/abstract/document/1583627)Cited by:[§II\.1](https://arxiv.org/html/2607.08791#S2.SS1.p1.1),[4th item](https://arxiv.org/html/2607.08791#S6.I1.i4.p1.1)\.
- \[26\]Y\. Lai, P\. Zheng, C\. Ji, C\. Qiu, T\. Wang, S\. Zhang, Z\. Wang, and Y\. Du\(2025\)CoffeeBoost: gradient boosting native conformal inference for Bayesian optimization\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 18017–18025\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/33982)Cited by:[§V\.1](https://arxiv.org/html/2607.08791#S5.SS1.p1.1)\.
- \[27\]R\. T\. Lange, Y\. Imajuku, and E\. Cetin\(2025\)Shinkaevolve: towards open\-ended and sample\-efficient program evolution\.arXiv preprint arXiv:2509\.19349\.External Links:[Link](https://arxiv.org/abs/2509.19349)Cited by:[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p1.1)\.
- \[28\]H\. Li, Y\. Jin, and T\. Chai\(2023\)Evolutionary multi\-objective Bayesian optimization based on multisource online transfer learning\.IEEE Transactions on Emerging Topics in Computational Intelligence8\(1\),pp\. 488–502\.External Links:[Link](https://ieeexplore.ieee.org/abstract/document/10238804)Cited by:[§V\.1](https://arxiv.org/html/2607.08791#S5.SS1.p1.1)\.
- \[29\]W\. Li, N\. van Stein, T\. Bäck, and E\. Raponi\(2025\)LLaMEA\-BO: a large language model evolutionary algorithm for automatically generating Bayesian optimization algorithms\.arXiv preprint arXiv:2505\.21034\.External Links:[Link](https://arxiv.org/abs/2505.21034)Cited by:[1st item](https://arxiv.org/html/2607.08791#S1.I1.i1.p1.1),[§I](https://arxiv.org/html/2607.08791#S1.p3.1),[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p2.1)\.
- \[30\]M\. Lindauer, K\. Eggensperger, M\. Feurer, A\. Biedenkapp, D\. Deng, C\. Benjamins, T\. Ruhkopf, R\. Sass, and F\. Hutter\(2022\)SMAC3: a versatile Bayesian optimization package for hyperparameter optimization\.Journal of Machine Learning Research23\(54\),pp\. 1–9\.External Links:[Link](https://www.jmlr.org/papers/v23/21-0888.html)Cited by:[§II\.3](https://arxiv.org/html/2607.08791#S2.SS3.p1.1),[§III\.3](https://arxiv.org/html/2607.08791#S3.SS3.p2.2)\.
- \[31\]F\. Liu, X\. Tong, M\. Yuan, X\. Lin, F\. Luo, Z\. Wang, Z\. Lu, and Q\. Zhang\(2024\)Evolution of heuristics: towards efficient automatic algorithm design using large language model\.InInternational Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.235,pp\. 32201–32223\.External Links:[Link](https://proceedings.mlr.press/v235/liu24bs.html)Cited by:[§I](https://arxiv.org/html/2607.08791#S1.p2.1),[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p1.1)\.
- \[32\]M\. López\-Ibáñez, J\. Dubois\-Lacoste, L\. P\. Cáceres, M\. Birattari, and T\. Stützle\(2016\)The irace package: iterated racing for automatic algorithm configuration\.Operations Research Perspectives3,pp\. 43–58\.External Links:[Link](https://www.sciencedirect.com/science/article/pii/S2214716015300270)Cited by:[§II\.3](https://arxiv.org/html/2607.08791#S2.SS3.p1.1)\.
- \[33\]Q\. Lu, K\. D\. Polyzos, B\. Li, and G\. B\. Giannakis\(2023\)Surrogate modeling for Bayesian optimization beyond a single gaussian process\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(9\),pp\. 11283–11296\.External Links:[Link](https://ieeexplore.ieee.org/abstract/document/10093035)Cited by:[§V\.1](https://arxiv.org/html/2607.08791#S5.SS1.p1.1)\.
- \[34\]R\. T\. Marler and J\. S\. Arora\(2004\)Survey of multi\-objective optimization methods for engineering\.Structural and multidisciplinary optimization26\(6\),pp\. 369–395\.External Links:[Link](https://link.springer.com/article/10.1007/s00158-003-0368-6)Cited by:[§I](https://arxiv.org/html/2607.08791#S1.p1.1)\.
- \[35\]K\. Miettinen\(1999\)Nonlinear multiobjective optimization\.Vol\.12,Springer Science & Business Media\.External Links:[Link](https://link.springer.com/book/10.1007/978-1-4615-5563-6)Cited by:[1st item](https://arxiv.org/html/2607.08791#S6.I1.i1.p1.1),[2nd item](https://arxiv.org/html/2607.08791#S6.I1.i2.p1.1)\.
- \[36\]P\. B\. Nemenyi\(1963\)Distribution\-free multiple comparisons\.\.Princeton University\.External Links:[Link](https://www.proquest.com/openview/c1f3e8829e8351e9c2a1c5e51778c6cf/1?pq-origsite=gscholar&cbl=18750&diss=y)Cited by:[§V\.3](https://arxiv.org/html/2607.08791#S5.SS3.p6.6)\.
- \[37\]A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. Ruiz, A\. Mehrabian,et al\.\(2025\)Alphaevolve: a coding agent for scientific and algorithmic discovery\.arXiv preprint arXiv:2506\.13131\.External Links:[Link](https://arxiv.org/abs/2506.13131)Cited by:[§I](https://arxiv.org/html/2607.08791#S1.p2.1),[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p1.1)\.
- \[38\]B\. Romera\-Paredes, M\. Barekatain, A\. Novikov, M\. Balog, M\. P\. Kumar, E\. Dupont, F\. J\. R\. Ruiz, J\. S\. Ellenberg, P\. Wang, O\. Fawzi,et al\.\(2024\)Mathematical discoveries from program search with large language models\.Nature625\(7995\),pp\. 468–475\.External Links:[Link](https://www.nature.com/articles/s41586-023-06924-6)Cited by:[§I](https://arxiv.org/html/2607.08791#S1.p2.1),[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p1.1)\.
- \[39\]P\. Schneider, W\. P\. Walters, A\. T\. Plowright, N\. Sieroka, J\. Listgarten, R\. A\. Goodnow Jr, J\. Fisher, J\. M\. Jansen, J\. S\. Duca, T\. S\. Rush,et al\.\(2020\)Rethinking drug design in the artificial intelligence era\.Nature reviews drug discovery19\(5\),pp\. 353–364\.External Links:[Link](https://www.nature.com/articles/s41573-019-0050-3)Cited by:[§I](https://arxiv.org/html/2607.08791#S1.p1.1)\.
- \[40\]B\. Shahriari, K\. Swersky, Z\. Wang, R\. P\. Adams, and N\. De Freitas\(2015\)Taking the human out of the loop: a review of Bayesian optimization\.Proceedings of the IEEE104\(1\),pp\. 148–175\.External Links:[Link](https://ieeexplore.ieee.org/abstract/document/7352306)Cited by:[§I](https://arxiv.org/html/2607.08791#S1.p1.1)\.
- \[41\]OpenEvolve: an open\-source evolutionary coding agentExternal Links:[Link](https://github.com/algorithmicsuperintelligence/openevolve),[Link](https://github.com/algorithmicsuperintelligence/openevolve)Cited by:[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p1.1)\.
- \[42\]N\. Srinivas, A\. Krause, S\. M\. Kakade, and M\. W\. Seeger\(2012\)Information\-theoretic regret bounds for Gaussian process optimization in the bandit setting\.IEEE Transactions on Information Theory58\(5\),pp\. 3250–3265\.External Links:[Link](https://ieeexplore.ieee.org/abstract/document/6138914)Cited by:[3rd item](https://arxiv.org/html/2607.08791#S6.I1.i3.p1.2)\.
- \[43\]R\. E\. Steuer and E\. Choo\(1983\)An interactive weighted Tchebycheff procedure for multiple objective programming\.Mathematical programming26\(3\),pp\. 326–344\.External Links:[Link](https://link.springer.com/article/10.1007/BF02591870)Cited by:[2nd item](https://arxiv.org/html/2607.08791#S6.I1.i2.p1.1)\.
- \[44\]J\. M\. Stokes, K\. Yang, K\. Swanson, W\. Jin, A\. Cubillos\-Ruiz, N\. M\. Donghia, C\. R\. MacNair, S\. French, L\. A\. Carfrae, Z\. Bloom\-Ackermann,et al\.\(2020\)A deep learning approach to antibiotic discovery\.Cell180\(4\),pp\. 688–702\.External Links:[Link](https://www.cell.com/cell/fulltext/S0092-8674(20)30102-1?fbclid=IwAR1HXccciF6clitQSUhVfcoq_3bmeBYm4vhndZabChZORx5bGHUYxuKe04U)Cited by:[§I](https://arxiv.org/html/2607.08791#S1.p1.1)\.
- \[45\]R\. Tanabe and H\. Ishibuchi\(2020\)An easy\-to\-use real\-world multi\-objective optimization problem suite\.Applied Soft Computing89,pp\. 106078\.External Links:[Link](https://www.sciencedirect.com/science/article/abs/pii/S1568494620300181)Cited by:[§IV\.2](https://arxiv.org/html/2607.08791#S4.SS2.p5.1),[§IV\.3](https://arxiv.org/html/2607.08791#S4.SS3.p3.2),[Table 1](https://arxiv.org/html/2607.08791#S4.T1),[Table 1](https://arxiv.org/html/2607.08791#S4.T1.8.4),[§V\.2\.3](https://arxiv.org/html/2607.08791#S5.SS2.SSS3.p1.1)\.
- \[46\]N\. van Stein and T\. Bäck\(2025\-04\)LLaMEA: a large language model evolutionary algorithm for automatically generating metaheuristics\.Transactions on Evolutionary Computation29\(2\),pp\. 331–345\.External Links:ISSN 1089\-778X,[Link](https://doi.org/10.1109/TEVC.2024.3497793),[Document](https://dx.doi.org/10.1109/TEVC.2024.3497793)Cited by:[1st item](https://arxiv.org/html/2607.08791#S1.I1.i1.p1.1),[§I](https://arxiv.org/html/2607.08791#S1.p2.1),[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p1.1)\.
- \[47\]N\. van Stein, A\. V\. Kononova, H\. Yin, and T\. Bäck\(2025\)BLADE: benchmark suite for LLM\-driven automated design and evolution of iterative optimisation heuristics\.InProceedings of the Genetic and Evolutionary Computation Conference Companion,pp\. 2336–2344\.External Links:[Link](https://dl.acm.org/doi/abs/10.1145/3712255.3734347)Cited by:[§V\.3](https://arxiv.org/html/2607.08791#S5.SS3.p1.2)\.
- \[48\]N\. van Stein, D\. Vermetten, and T\. Bäck\(2024\)In\-the\-loop hyper\-parameter optimization for LLM\-based automated design of heuristics\.ACM Transactions on Evolutionary Learning\.External Links:[Link](https://arxiv.org/abs/2410.16309)Cited by:[1st item](https://arxiv.org/html/2607.08791#S1.I1.i1.p1.1),[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p2.1),[§III\.3](https://arxiv.org/html/2607.08791#S3.SS3.p2.2)\.
- \[49\]X\. Wang, Y\. Jin, S\. Schmitt, and M\. Olhofer\(2023\)Recent advances in Bayesian optimization\.ACM computing surveys55\(13s\),pp\. 1–36\.External Links:[Link](https://dl.acm.org/doi/full/10.1145/3582078)Cited by:[§V\.1](https://arxiv.org/html/2607.08791#S5.SS1.p1.1)\.
- \[50\]B\. L\. Welch\(1947\)The generalization of ‘student’s’problem when several different population varlances are involved\.Biometrika34\(1\-2\),pp\. 28–35\.External Links:[Link](https://academic.oup.com/biomet/article-abstract/34/1-2/28/210174?redirectedFrom=PDF)Cited by:[§V\.3](https://arxiv.org/html/2607.08791#S5.SS3.p1.2)\.
- \[51\]S\. Wietheger and B\. Doerr\(2024\)A mathematical runtime analysis of the non\-dominated sorting genetic algorithm iii \(NSGA\-III\)\.InProceedings of the Genetic and Evolutionary Computation Conference Companion,pp\. 63–64\.External Links:[Link](https://dl.acm.org/doi/abs/10.1145/3638530.3664062)Cited by:[§IV\.1](https://arxiv.org/html/2607.08791#S4.SS1.p1.6),[§V\.2\.2](https://arxiv.org/html/2607.08791#S5.SS2.SSS2.p1.1)\.
- \[52\]F\. Wilcoxon\(1945\)Individual comparisons by ranking methods\.Biometrics bulletin1\(6\),pp\. 80–83\.External Links:[Link](https://www.jstor.org/stable/3001968)Cited by:[§V\.3](https://arxiv.org/html/2607.08791#S5.SS3.p1.2),[§V\.3](https://arxiv.org/html/2607.08791#S5.SS3.p5.1)\.
- \[53\]C\. K\. Williams and C\. E\. Rasmussen\(2006\)Gaussian processes for machine learning\.Vol\.2,MIT press Cambridge, MA\.External Links:[Link](https://pdfs.semanticscholar.org/6a52/87336aa1a2a653d6df95fac402612fd20772.pdf)Cited by:[1st item](https://arxiv.org/html/2607.08791#S6.I1.i1.p1.1)\.
- \[54\]Z\. Xie, F\. Liu, Z\. Wang, and Q\. Zhang\(2025\)LLM\-driven neighborhood search for efficient heuristic design\.\.InCEC,pp\. 1–8\.External Links:[Link](https://www.researchgate.net/profile/Fei-Liu-46/publication/392988386_LLM-Driven_Neighborhood_Search_for_Efficient_Heuristic_Design/links/68770ca3dd6b84447df8de4b/LLM-Driven-Neighborhood-Search-for-Efficient-Heuristic-Design.pdf)Cited by:[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p1.1)\.
- \[55\]H\. Ye, J\. Wang, Z\. Cao, F\. Berto, C\. Hua, H\. Kim, J\. Park, and G\. Song\(2024\)ReEvo: large language models as hyper\-heuristics with reflective evolution\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/4ced59d480e07d290b6f29fc8798f195-Abstract-Conference.html)Cited by:[§I](https://arxiv.org/html/2607.08791#S1.p2.1),[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p1.1)\.
- \[56\]Q\. Zhang and H\. Li\(2007\)MOEA/D: a multiobjective evolutionary algorithm based on decomposition\.IEEE Transactions on evolutionary computation11\(6\),pp\. 712–731\.External Links:[Link](https://ieeexplore.ieee.org/document/4358754)Cited by:[2nd item](https://arxiv.org/html/2607.08791#S6.I1.i2.p1.1),[3rd item](https://arxiv.org/html/2607.08791#S6.I1.i3.p1.2)\.
- \[57\]Q\. Zhang, W\. Liu, E\. Tsang, and B\. Virginas\(2010\)Expensive multiobjective optimization by MOEA/D with Gaussian process model\.IEEE Transactions on Evolutionary Computation14\(3\),pp\. 456–474\.External Links:[Link](https://ieeexplore.ieee.org/document/5353656)Cited by:[§II\.1](https://arxiv.org/html/2607.08791#S2.SS1.p3.1)\.
- \[58\]Z\. Zheng, Z\. Xie, Z\. Wang, and B\. Hooi\(2025\)Monte carlo tree search for comprehensive exploration in LLM\-based automatic heuristic design\.InInternational Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research\.External Links:[Link](https://icml.cc/virtual/2025/poster/45984)Cited by:[§II\.2](https://arxiv.org/html/2607.08791#S2.SS2.p1.1)\.
- \[59\]E\. Zitzler, K\. Deb, and L\. Thiele\(2000\)Comparison of multiobjective evolutionary algorithms: empirical results\.Evolutionary computation8\(2\),pp\. 173–195\.External Links:[Link](https://ieeexplore.ieee.org/abstract/document/6787994)Cited by:[1st item](https://arxiv.org/html/2607.08791#S4.I1.i1.p1.1),[§IV\.2](https://arxiv.org/html/2607.08791#S4.SS2.p1.1)\.
- \[60\]E\. Zitzler, L\. Thiele, M\. Laumanns, C\. M\. Fonseca, and V\. G\. Da Fonseca\(2003\)Performance assessment of multiobjective optimizers: an analysis and review\.IEEE Transactions on evolutionary computation7\(2\),pp\. 117–132\.External Links:[Link](https://ieeexplore.ieee.org/abstract/document/1197687)Cited by:[§IV\.2](https://arxiv.org/html/2607.08791#S4.SS2.p2.1)\.
- \[61\]E\. Zitzler and L\. Thiele\(1999\)Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach\.IEEE transactions on Evolutionary Computation3\(4\),pp\. 257–271\.External Links:[Link](https://ieeexplore.ieee.org/abstract/document/797969)Cited by:[§IV\.2](https://arxiv.org/html/2607.08791#S4.SS2.p2.1)\.

## Appendix

### ASelected LLaMEA\-Generated Algorithms

List of algorithms 1Improved\-Scalarized\-EI1:budget

BB, dimension

dd, bounds

\[𝐥,𝐮\]\[\\mathbf\{l\},\\mathbf\{u\}\], window

NwN\_\{w\}, weight count

NλN\_\{\\lambda\}, EA population

PP, EA generations

GG
2:non\-dominated archive

\(ℱ⋆,𝒳⋆\)\(\\mathcal\{F\}^\{\\star\},\\mathcal\{X\}^\{\\star\}\)
3:

𝒳←\\mathcal\{X\}\\leftarrowSobol sample of

ninitn\_\{\\mathrm\{init\}\}points;

ℱ←Evaluate​\(f,𝒳\)\\mathcal\{F\}\\leftarrow\\textsc\{Evaluate\}\(f,\\mathcal\{X\}\); infer

MM
4:pre\-generate

NλN\_\{\\lambda\}uniform weight vectors

\{𝝀k\}\\\{\\bm\{\\lambda\}\_\{k\}\\\}on the simplex \(Sobol\)

5:while

nevals<Bn\_\{\\mathrm\{evals\}\}<Bdo

6:fit one GP \(Matérn\-

2\.52\.5\) per objective on the last

NwN\_\{w\}points, with objectives normalized to

\[0,1\]\[0,1\]; on failure, refit without optimizer restarts

7:

𝝀←𝝀\(nevalsmodNλ\)\\bm\{\\lambda\}\\leftarrow\\bm\{\\lambda\}\_\{\(n\_\{\\mathrm\{evals\}\}\\bmod N\_\{\\lambda\}\)\}⊳\\trianglerightcycle through weights

8:

gmin←min𝐲∈ℱ⁡𝝀⊤​𝐲~g^\{\\min\}\\leftarrow\\min\_\{\\mathbf\{y\}\\in\\mathcal\{F\}\}\\bm\{\\lambda\}^\{\\top\}\\tilde\{\\mathbf\{y\}\}⊳\\trianglerightbest scalarized value \(normalized\)

9:initialize EA population by Sobol sampling, injecting up to

20%20\\%of the current Pareto points

10:forgeneration

=1,…,G=1,\\dots,Gdo

11:foreach individual

𝐱\\mathbf\{x\}do

12:predict

\(𝝁,𝝈\)\(\\bm\{\\mu\},\\bm\{\\sigma\}\)with the GPs; scalarize

μs=𝝀⊤​𝝁\\mu\_\{s\}=\\bm\{\\lambda\}^\{\\top\}\\bm\{\\mu\},

σs=∑mλm2​σm2\\sigma\_\{s\}=\\sqrt\{\\textstyle\\sum\_\{m\}\\lambda\_\{m\}^\{2\}\\sigma\_\{m\}^\{2\}\}
13:fitness

←EI​\(μs,σs;gmin\)\\leftarrow\\mathrm\{EI\}\(\\mu\_\{s\},\\sigma\_\{s\};g^\{\\min\}\)⊳\\trianglerightExpected Improvement, minimization

14:endfor

15:elitism \(retain best\); reproduce by tournament selection, SBX crossover and Gaussian mutation; clip to bounds

16:endfor

17:

𝐱\+←\\mathbf\{x\}^\{\+\}\\leftarrowbest individual found by the EA

18:

\(𝐱\+,𝐲\+\)←Evaluate​\(f,𝐱\+\)\(\\mathbf\{x\}^\{\+\},\\mathbf\{y\}^\{\+\}\)\\leftarrow\\textsc\{Evaluate\}\(f,\\mathbf\{x\}^\{\+\}\); append to archive; update Pareto front

19:endwhile

20:returnnon\-dominated subset of the archive

List of algorithms 2RF\-LCB\-PBI1:budget

BB, dimension

dd, bounds

\[𝐥,𝐮\]\[\\mathbf\{l\},\\mathbf\{u\}\], candidate pool

NcN\_\{c\}, weights

NwN\_\{w\}, trees

TT, window

WW, PBI penalty

θ\\theta, LCB weight

κ\\kappa, crowding weight

κc​d\\kappa\_\{cd\}
2:non\-dominated archive

\(ℱ⋆,𝒳⋆\)\(\\mathcal\{F\}^\{\\star\},\\mathcal\{X\}^\{\\star\}\)
3:

𝒳←\\mathcal\{X\}\\leftarrowSobol sample of

ninitn\_\{\\mathrm\{init\}\};

ℱ←Evaluate​\(f,𝒳\)\\mathcal\{F\}\\leftarrow\\textsc\{Evaluate\}\(f,\\mathcal\{X\}\); infer

MM
4:while

nevals<Bn\_\{\\mathrm\{evals\}\}<Bdo

5:fit one Random Forest \(

TTtrees\) per objective on the last

WWpoints

6:

𝒞←\\mathcal\{C\}\\leftarrowSobol sample of

NcN\_\{c\}candidates

7:predict mean

𝝁\\bm\{\\mu\}and ensemble std

𝝈\\bm\{\\sigma\}for

𝒞\\mathcal\{C\}; normalize with the ideal/nadir of

ℱ\\mathcal\{F\}
8:

𝐟LCB←𝝁norm−κ​𝝈norm\\mathbf\{f\}^\{\\mathrm\{LCB\}\}\\leftarrow\\bm\{\\mu\}\_\{\\mathrm\{norm\}\}\-\\kappa\\,\\bm\{\\sigma\}\_\{\\mathrm\{norm\}\}⊳\\trianglerightlower confidence bound

9:draw

NwN\_\{w\}weight vectors

\{𝐰\}\\\{\\mathbf\{w\}\\\}\(Sobol,

∑mwm=1\\sum\_\{m\}w\_\{m\}=1\)

10:foreach weight

𝐰\\mathbf\{w\}do

11:

d1←\(𝐰⊤​𝐟LCB\)/∥𝐰∥d\_\{1\}\\leftarrow\(\\mathbf\{w\}^\{\\top\}\\mathbf\{f\}^\{\\mathrm\{LCB\}\}\)/\\lVert\\mathbf\{w\}\\rVert;

d2←∥𝐟LCB−d1​𝐰^∥d\_\{2\}\\leftarrow\\lVert\\mathbf\{f\}^\{\\mathrm\{LCB\}\}\-d\_\{1\}\\hat\{\\mathbf\{w\}\}\\rVert;

PBI𝐰←d1\+θ​d2\\mathrm\{PBI\}\_\{\\mathbf\{w\}\}\\leftarrow d\_\{1\}\+\\theta d\_\{2\}
12:endfor

13:

s​\(𝐜\)←min𝐰⁡PBI𝐰​\(𝐜\)s\(\\mathbf\{c\}\)\\leftarrow\\min\_\{\\mathbf\{w\}\}\\mathrm\{PBI\}\_\{\\mathbf\{w\}\}\(\\mathbf\{c\}\)⊳\\trianglerightbest \(most optimistic\) subproblem

14:

s​\(𝐜\)←s​\(𝐜\)−κc​d​dist¯k​\(𝐜,ℱ⋆\)s\(\\mathbf\{c\}\)\\leftarrow s\(\\mathbf\{c\}\)\-\\kappa\_\{cd\}\\,\\overline\{\\mathrm\{dist\}\}\_\{k\}\(\\mathbf\{c\},\\mathcal\{F\}^\{\\star\}\)⊳\\trianglerightcrowding\-distance bonus \(k=3k\{=\}3\)

15:evaluate the candidate with the lowest

ss; append to archive; update Pareto front

16:endwhile

17:returnnon\-dominated subset of the archive

List of algorithms 3RF\-ParEGO\-Batch1:budget

BB, dimension

dd, bounds

\[𝐥,𝐮\]\[\\mathbf\{l\},\\mathbf\{u\}\], candidate pool

NcN\_\{c\}, batch size

qq, trees

TT, window

WW, exploration weight

β\\beta
2:non\-dominated archive

\(ℱ⋆,𝒳⋆\)\(\\mathcal\{F\}^\{\\star\},\\mathcal\{X\}^\{\\star\}\)
3:

𝒳←\\mathcal\{X\}\\leftarrowLHS sample of

ninitn\_\{\\mathrm\{init\}\};

ℱ←Evaluate​\(f,𝒳\)\\mathcal\{F\}\\leftarrow\\textsc\{Evaluate\}\(f,\\mathcal\{X\}\); infer

MM
4:while

nevals<Bn\_\{\\mathrm\{evals\}\}<Bdo

5:fit one Random Forest \(

TTtrees\) per objective on the last

WWpoints

6:

𝒞←\\mathcal\{C\}\\leftarrowLHS sample of

NcN\_\{c\}candidates

7:predict mean

𝝁\\bm\{\\mu\}and ensemble std

𝝈\\bm\{\\sigma\}for

𝒞\\mathcal\{C\}; normalize with the observed ideal/nadir

8:

ℬ←∅\\mathcal\{B\}\\leftarrow\\emptyset
9:for

i=1,…,qi=1,\\dots,qdo

10:draw weight vector

𝐰∼Dirichlet​\(𝟏\)\\mathbf\{w\}\\sim\\mathrm\{Dirichlet\}\(\\mathbf\{1\}\)⊳\\trianglerightfresh scalarization per batch member

11:

tμ​\(𝐜\)←maxm⁡wm​μnorm,mt\_\{\\mu\}\(\\mathbf\{c\}\)\\leftarrow\\max\_\{m\}w\_\{m\}\\,\\mu\_\{\\mathrm\{norm\},m\};

tσ​\(𝐜\)←maxm⁡wm​σnorm,mt\_\{\\sigma\}\(\\mathbf\{c\}\)\\leftarrow\\max\_\{m\}w\_\{m\}\\,\\sigma\_\{\\mathrm\{norm\},m\}⊳\\trianglerightTchebycheff

12:

a​\(𝐜\)←−\(tμ​\(𝐜\)−β​tσ​\(𝐜\)\)a\(\\mathbf\{c\}\)\\leftarrow\-\\big\(t\_\{\\mu\}\(\\mathbf\{c\}\)\-\\beta\\,t\_\{\\sigma\}\(\\mathbf\{c\}\)\\big\)⊳\\trianglerightminimize scalarized value, LCB exploration

13:add to

ℬ\\mathcal\{B\}the unselected candidate with highest

aa; mark it selected

14:endfor

15:evaluate

ℬ\\mathcal\{B\}; append to archive

16:endwhile

17:returnnon\-dominated subset of the archive

List of algorithms 4MOEAD\-EI Hybrid1:initial\-design ratio

rinitr\_\{\\mathrm\{init\}\}, evaluation budget

BB, dimension

dd, bounds

\[𝐥,𝐮\]\[\\mathbf\{l\},\\mathbf\{u\}\], population size

NpopN\_\{\\mathrm\{pop\}\}, surrogate\-offspring count

NoffN\_\{\\mathrm\{off\}\}, batch size

qq, window size

NwN\_\{w\}, augmentation constant

ρ\\rho
2:non\-dominated archive

\(ℱ⋆,𝒳⋆\)\(\\mathcal\{F\}^\{\\star\},\\mathcal\{X\}^\{\\star\}\)
3:

4:

ninit←max⁡\(min⁡\(⌊B⋅rinit⌋,2​d\+1\),1\)n\_\{\\mathrm\{init\}\}\\leftarrow\\max\\\!\\big\(\\min\(\\lfloor B\\cdot r\_\{\\mathrm\{init\}\}\\rfloor,\\,2d\{\+\}1\),\\,1\\big\)
5:

Npop←min⁡\(Npop,ninit\)N\_\{\\mathrm\{pop\}\}\\leftarrow\\min\(N\_\{\\mathrm\{pop\}\},\\,n\_\{\\mathrm\{init\}\}\)⊳\\trianglerightcap population by initial\-design size

6:

𝒳←\\mathcal\{X\}\\leftarrowLatin Hypercube sample of

ninitn\_\{\\mathrm\{init\}\}points in

\[𝐥,𝐮\]\[\\mathbf\{l\},\\mathbf\{u\}\]
7:

ℱ←Evaluate​\(f,𝒳\)\\mathcal\{F\}\\leftarrow\\textsc\{Evaluate\}\(f,\\mathcal\{X\}\); infer

MM\(number of objectives\) from first call

8:initialise archive

\(𝒳all,ℱall\)←\(𝒳,ℱ\)\(\\mathcal\{X\}\_\{\\mathrm\{all\}\},\\mathcal\{F\}\_\{\\mathrm\{all\}\}\)\\leftarrow\(\\mathcal\{X\},\\mathcal\{F\}\)and population

\(𝒳pop,ℱpop\)←\(𝒳,ℱ\)\(\\mathcal\{X\}\_\{\\mathrm\{pop\}\},\\mathcal\{F\}\_\{\\mathrm\{pop\}\}\)\\leftarrow\(\\mathcal\{X\},\\mathcal\{F\}\)
9:generate

NpopN\_\{\\mathrm\{pop\}\}weight vectors

\{𝐰j\}\\\{\\mathbf\{w\}\_\{j\}\\\}\(uniform grid if

M=2M\{=\}2, Dirichlet samples if

M\>2M\{\>\}2\)

10:update ideal point

𝐳∗\\mathbf\{z\}^\{\*\}and nadir point

𝐳nad\\mathbf\{z\}^\{\\mathrm\{nad\}\}from

ℱall\\mathcal\{F\}\_\{\\mathrm\{all\}\}
11:

12:while

nevals<Bn\_\{\\mathrm\{evals\}\}<Bdo

13:fit one GP surrogate per objective on the last

NwN\_\{w\}points of

\(𝒳all,ℱall\)\(\\mathcal\{X\}\_\{\\mathrm\{all\}\},\\mathcal\{F\}\_\{\\mathrm\{all\}\}\)
14:generate

NoffN\_\{\\mathrm\{off\}\}offspring from

𝒳pop\\mathcal\{X\}\_\{\\mathrm\{pop\}\}via SBX crossover and polynomial mutation

15:predict

\(𝝁,𝝈\)\(\\bm\{\\mu\},\\bm\{\\sigma\}\)for every offspring with the GPs

16:normalise

𝝁,𝝈\\bm\{\\mu\},\\bm\{\\sigma\}and

ℱall\\mathcal\{F\}\_\{\\mathrm\{all\}\}to

\[0,1\]\[0,1\]using

𝐳∗,𝐳nad\\mathbf\{z\}^\{\*\},\\mathbf\{z\}^\{\\mathrm\{nad\}\}
17:foreach subproblem

j=1,…,Npopj=1,\\dots,N\_\{\\mathrm\{pop\}\}do

18:

gjmin←ming^\{\\min\}\_\{j\}\\leftarrow\\minaugmented\-Tchebycheff value of

ℱall\\mathcal\{F\}\_\{\\mathrm\{all\}\}under

𝐰j\\mathbf\{w\}\_\{j\}
19:compute Expected Improvement

EIj​\(⋅\)\\mathrm\{EI\}\_\{j\}\(\\cdot\)of each offspring on the scalarised subproblem

𝐰j\\mathbf\{w\}\_\{j\}
20:endfor

21:for each subproblem pick the EI\-maximising offspring; take the union and keep the top

qqby EI

22:

\(𝒳new,ℱnew\)←Evaluate​\(f,selected batch\)\(\\mathcal\{X\}\_\{\\mathrm\{new\}\},\\mathcal\{F\}\_\{\\mathrm\{new\}\}\)\\leftarrow\\textsc\{Evaluate\}\(f,\\text\{selected batch\}\)
23:append

\(𝒳new,ℱnew\)\(\\mathcal\{X\}\_\{\\mathrm\{new\}\},\\mathcal\{F\}\_\{\\mathrm\{new\}\}\)to the archive; update

𝐳∗,𝐳nad\\mathbf\{z\}^\{\*\},\\mathbf\{z\}^\{\\mathrm\{nad\}\}
24:foreach new point

\(𝐱,𝐟\)\(\\mathbf\{x\},\\mathbf\{f\}\)and each subproblem

jjdo

25:ifaugmented\-Tchebycheff

\(𝐟,𝐰j\)<\(\\mathbf\{f\},\\mathbf\{w\}\_\{j\}\)<that of

ℱpop​\[j\]\\mathcal\{F\}\_\{\\mathrm\{pop\}\}\[j\]then

26:replace population member

jjwith

\(𝐱,𝐟\)\(\\mathbf\{x\},\\mathbf\{f\}\)
27:endif

28:endfor

29:

\(ℱ⋆,𝒳⋆\)←\(\\mathcal\{F\}^\{\\star\},\\mathcal\{X\}^\{\\star\}\)\\leftarrownon\-dominated subset of

\(ℱall,𝒳all\)\(\\mathcal\{F\}\_\{\\mathrm\{all\}\},\\mathcal\{X\}\_\{\\mathrm\{all\}\}\)
30:endwhile

31:return

\(ℱ⋆,𝒳⋆\)\(\\mathcal\{F\}^\{\\star\},\\mathcal\{X\}^\{\\star\}\)

### BComplete benchmark tables

Table 4:Synthetic per\-problem mean final HV±\\pmstd \(part 1\)\. Best per problem in bold;†\\daggermarks LLaMEA\-generated\.Table 5:Synthetic per\-problem mean final HV±\\pmstd \(part 2\)\.Table 6:Mean final hypervolume±\\pmstd across 5 repetitions on the Phase 3 real\-world RE problems\. Best per problem inbold;†\\daggermarks LLaMEA\-generated\.
### CAdditional plots

This appendix contains the complete set of convergence curves and Pareto fronts for all twelve synthetic benchmark problems used in Phases 1 and 2, supplementing the four representative panels shown in the main text \(Figs\.[2](https://arxiv.org/html/2607.08791#S5.F2)and[3](https://arxiv.org/html/2607.08791#S5.F3)\)\. Each curve is the mean over five independent repetitions, with the shaded band denoting±1​σ\\pm 1\\sigma\.

![Refer to caption](https://arxiv.org/html/2607.08791v1/figures/fig8_phase1_all.png)Figure 8:Convergence on all twelve synthetic problems \(Phase 1\) for the nine LLaMEA\-generated algorithms\. Shaded bands denote±1\\pm 1std over five repeats\.![Refer to caption](https://arxiv.org/html/2607.08791v1/figures/fig9_phase2_all.png)Figure 9:Convergence on all twelve synthetic problems \(Phase 2\) for the nine benchmarked algorithms \(four LLaMEA\-generated, the SOTA qParEGO baseline, the surrogate\-assisted IOC\-SAMO\-COBRA, two classical evolutionary baselines, and Random Search\)\. Shaded bands denote±1\\pm 1std over five repeats\.![Refer to caption](https://arxiv.org/html/2607.08791v1/figures/fig10_pareto_synth.png)Figure 10:Non\-dominated fronts on the twelve synthetic benchmark problems for the four LLaMEA\-generated algorithms and the SOTA qParEGO baseline\. Bi\-objective problems show the front directly; tri\-objective problems \(DTLZ1, DTLZ2, DTLZ4, DTLZ7, WFG7\) are shown as 3D scatters\.![Refer to caption](https://arxiv.org/html/2607.08791v1/figures/fig11_pareto_real.png)Figure 11:Non\-dominated fronts on the real\-world RE problems for the four generated algorithms and qParEGO\. RE21 is bi\-objective; RE34 and RE37 are shown as 3D scatters\.