EVOM: Agentic Meta-Evolution of Actor-Critic Architectures for Reinforcement Learning

# EVOM: Agentic Meta-Evolution of Actor-Critic Architectures for Reinforcement Learning
Source: [https://arxiv.org/html/2606.26327](https://arxiv.org/html/2606.26327)
Boyun Zhang, Chao Wang, Kai Wu Xidian University 23009300248@stu\.xidian\.edu\.cn, xiaofengxd@126\.com, kwu@xidian\.edu\.cn

###### Abstract

In actor\-critic reinforcement learning, network architectures are typically manually designed\. Automating this design is challenging because each candidate must be trained before evaluation, and the design space is open\-ended\. To address these challenges, we introduceEVOM, an agentic meta\-evolution framework for discovering high\-performance actor\-critic architectures\. We frame architecture search as a bi\-level optimization: an inner loop trains weights via the low\-fidelity proximal policy optimization \(PPO\), while an outer loop drives meta\-evolution by iteratively refining architecture programs\. Crucially, this outer loop is powered by an LLM\-based design agent that operates purely as an architecture designer, completely decoupled from policy execution and environment control\. Experiments reveal thatEVOMoutperforms the manually designed baseline, an LLM\-guided random search, and the state\-of\-the\-art LLM\-guided programmatic policy search methodMLES, delivering superior performance on Ant\-v4 and HalfCheetah\-v4\. Ablation studies validate that both the meta\-evolution loop and the LLM Design Agent are indispensable for final performance\.

## 1Introduction

The success of deep learningLeCunet al\.\([2015](https://arxiv.org/html/2606.26327#bib.bib1)\)has been driven not only by algorithmic advances but also by innovations in network architecture\. In supervised learning, this observation has motivated extensive research on neural architecture search \(NAS\)Elskenet al\.\([2019](https://arxiv.org/html/2606.26327#bib.bib13)\); Liuet al\.\([2021](https://arxiv.org/html/2606.26327#bib.bib2)\), producing architectures that are competitive with or better than hand\-designed alternatives across vision and language benchmarks\. In reinforcement learning \(RL\)Suttonet al\.\([1998](https://arxiv.org/html/2606.26327#bib.bib3)\); Liet al\.\([2024](https://arxiv.org/html/2606.26327#bib.bib4)\), however, automated architecture design has received less attention\. Within actor\-critic methods, such as proximal policy optimization \(PPO\)Schulmanet al\.\([2017](https://arxiv.org/html/2606.26327#bib.bib5)\), the networks used to instantiate the policy and value functions are typically treated as fixed implementation choices inherited from prior work\. This convention may overlook an important design factor: actor\-critic architectures can affect learning stability and final performance, motivating automated approaches to architecture discovery for actor\-critic systems\.

![Refer to caption](https://arxiv.org/html/2606.26327v1/x1.png)Figure 1:Overview ofEVOMas bi\-level optimization: Outer loop evolves architectures, and inner loop trains weights\.Automated architecture discovery for actor\-critic methods faces two main challenges\. First, each candidate architecture must be trained before it can be evaluated, making full\-budget comparison across many candidates computationally expensive\. Second, actor\-critic design is open\-ended: policy heads, value heads, shared trunks, normalization layers, activation functions, and other modules can be composed in many ways, without a canonical fixed\-topology search space\. We proposeEVOM, an agentic meta\-evolution framework that jointly addresses these challenges\. A low\-fidelity inner evaluation loop provides computationally tractable fitness estimates\. At the same time, an LLM\-based design agent generates and refines architectures as executable programs, reducing reliance on a manually predefined search space\. As illustrated in Fig\.[1](https://arxiv.org/html/2606.26327#S1.F1),EVOMoperates as a bi\-level optimization: the outer loop evolves a population of architecture programs, and the inner loop evaluates each candidate by training its weights with PPO under a reduced budget\.

EVOMalso differs from recent LLM\-guided policy search methods such asMLESHuet al\.\([2026](https://arxiv.org/html/2606.26327#bib.bib20)\), which synthesize executable controller programs that map observations directly to actions\. In contrast,EVOMgenerates trainable actor\-critic architectures: the generated programs specify network structure, while PPO learns the weights through environment interaction\. This positions the LLM as a reusable design operator within a meta\-evolution loop, rather than as a replacement for RL\. Our contributions are as follows:

- •We identify two challenges in automated actor\-critic architecture design, namely evaluation cost and the open\-ended design space, and formulate a bi\-level meta\-evolution framework that addresses them jointly\.
- •We instantiate architecture search with a low\-fidelity PPO evaluator, which provides a practical fitness signal for comparing candidate architectures during evolution\.
- •We introduce an LLM design agent as a program\-level architecture operator for initialization, mutation, and crossover within an evolutionary loop\.
- •Experiments on Ant\-v4 and HalfCheetah\-v4 show thatEVOMimproves over the manually designed baseline, an LLM\-guided random search, and anMLES\-style programmatic policy search baseline, with ablations indicating the contributions of evolutionary inheritance and LLM\-guided design\.

## 2Related Work

EVOMis related to three areas: NASElskenet al\.\([2019](https://arxiv.org/html/2606.26327#bib.bib13)\)and neuroevolutionMiikkulainen \([2025](https://arxiv.org/html/2606.26327#bib.bib15)\), LLM\-assisted automated algorithm designMaet al\.\([2026](https://arxiv.org/html/2606.26327#bib.bib16)\); Romera\-Paredeset al\.\([2024](https://arxiv.org/html/2606.26327#bib.bib17)\); Liuet al\.\([2026](https://arxiv.org/html/2606.26327#bib.bib19)\), and actor\-critic reinforcement learningSuttonet al\.\([1998](https://arxiv.org/html/2606.26327#bib.bib3)\); Grondmanet al\.\([2012](https://arxiv.org/html/2606.26327#bib.bib18)\)\. Prior work has studied how to automate architecture design, evolve executable structures, and use LLMs to generate algorithms or programs\. In contrast,EVOMfocuses on LLM\-guided evolution of actor\-critic architectures, where the generated programs define network structure and PPO remains responsible for policy learning\.

NAS and neuroevolution\.NAS automates network design through controller policies, evolutionary search, and differentiable relaxationZoph and Le \([2017](https://arxiv.org/html/2606.26327#bib.bib11)\); Realet al\.\([2019](https://arxiv.org/html/2606.26327#bib.bib12)\); Elskenet al\.\([2019](https://arxiv.org/html/2606.26327#bib.bib13)\)\. Evolutionary computation and genetic programming provide a broader framework for reproduction, evaluation, and selection over executable structuresKoza \([1992](https://arxiv.org/html/2606.26327#bib.bib21)\); Bäck \([1996](https://arxiv.org/html/2606.26327#bib.bib22)\)\. Neuroevolution applies related principles to policies, weights, and topologiesStanley and Miikkulainen \([2002](https://arxiv.org/html/2606.26327#bib.bib14)\); Salimanset al\.\([2017](https://arxiv.org/html/2606.26327#bib.bib37)\); Suchet al\.\([2017](https://arxiv.org/html/2606.26327#bib.bib38)\); Gaier and Ha \([2019](https://arxiv.org/html/2606.26327#bib.bib39)\)\.EVOMfollows this evolutionary view, but applies LLM\-guided program generation to the design of trainable actor\-critic architectures\.

LLM\-assisted automated algorithm design\.LLMs can generate programs, use feedback, and support iterative design loopsBrownet al\.\([2020](https://arxiv.org/html/2606.26327#bib.bib32)\); Ouyanget al\.\([2022](https://arxiv.org/html/2606.26327#bib.bib33)\); OpenAI \([2023](https://arxiv.org/html/2606.26327#bib.bib23)\); Chenet al\.\([2021](https://arxiv.org/html/2606.26327#bib.bib34)\); Yaoet al\.\([2023](https://arxiv.org/html/2606.26327#bib.bib25)\); Shinnet al\.\([2023](https://arxiv.org/html/2606.26327#bib.bib26)\); Wanget al\.\([2024](https://arxiv.org/html/2606.26327#bib.bib27),[2025](https://arxiv.org/html/2606.26327#bib.bib9)\)\. Recent studies use LLMs for automated algorithm design and program search, including mathematical discovery, heuristic design, self\-refinement, black\-box optimization, and reward designRomera\-Paredeset al\.\([2024](https://arxiv.org/html/2606.26327#bib.bib17)\); Liuet al\.\([2024](https://arxiv.org/html/2606.26327#bib.bib28)\); Madaanet al\.\([2023](https://arxiv.org/html/2606.26327#bib.bib35)\); Zhanget al\.\([2023](https://arxiv.org/html/2606.26327#bib.bib36)\); Maet al\.\([2024](https://arxiv.org/html/2606.26327#bib.bib29)\); Wanget al\.\([2026a](https://arxiv.org/html/2606.26327#bib.bib8),[b](https://arxiv.org/html/2606.26327#bib.bib7)\)\. Closely related to our setting,MLESevolves interpretable programmatic controllers with multimodal feedbackHuet al\.\([2026](https://arxiv.org/html/2606.26327#bib.bib20)\)\. In contrast,EVOMuses the LLM design agent to generate trainable actor\-critic architectures, leaving policy and value parameters to be learned by PPO through environment interaction\.

Actor\-critic reinforcement learning\.Actor\-critic methods, including PPOSchulmanet al\.\([2017](https://arxiv.org/html/2606.26327#bib.bib5)\), learn separate policy and value functions and are widely used for continuous\-control tasks\. In practice, these functions are instantiated by neural networks whose architecture can influence optimization stability, exploration behavior, and advantage estimation\. We use PPO from Stable\-Baselines3Raffinet al\.\([2021](https://arxiv.org/html/2606.26327#bib.bib6)\)on MuJoCo tasksBrockmanet al\.\([2016](https://arxiv.org/html/2606.26327#bib.bib24)\); Towerset al\.\([2025](https://arxiv.org/html/2606.26327#bib.bib10)\)as the inner learner for evaluating candidate architectures\. BecauseEVOMoutputs explicit programs, the resulting architectures can be inspected, reproduced, and reused rather than treated as opaque policies\.

## 3Problem Formulation

We consider episodic continuous\-control environments with observationsot∈ℝdo\_\{t\}\\in\\mathbb\{R\}^\{d\}, actionsat∈ℝma\_\{t\}\\in\\mathbb\{R\}^\{m\}, rewardsrtr\_\{t\}, and horizonHH\. Letp∈𝒜p\\in\\mathcal\{A\}denote an actor\-critic architecture program from an open\-ended architecture space𝒜\\mathcal\{A\}\. Rather than assuming a fixed topology or a closed set of hand\-specified choices,𝒜\\mathcal\{A\}is defined by an executable interface and validity constraints\. Givenpp, the actor and critic are instantiated as

μθ​\(o\)=πθ​\(o;p\),vϕ​\(o\)=Vϕ​\(o;p\),\\mu\_\{\\theta\}\(o\)=\\pi\_\{\\theta\}\(o;p\),\\qquad v\_\{\\phi\}\(o\)=V\_\{\\phi\}\(o;p\),\(1\)whereμθ​\(o\)\\mu\_\{\\theta\}\(o\)is the continuous\-action mean,vϕ​\(o\)v\_\{\\phi\}\(o\)is a scalar value estimate, and\(θ,ϕ\)\(\\theta,\\phi\)are trainable parameters\. Thus, architecture search concerns the outer choice ofpp, while PPO performs the inner optimization of\(θ,ϕ\)\(\\theta,\\phi\)for an architecture\.

Given an architectureppand a training seedss, letUB​\(p,s\)U\_\{B\}\(p,s\)denote PPO training under budgetBB\. The result ofUB​\(p,s\)U\_\{B\}\(p,s\)is a trained actor\-critic policy whose deterministic evaluation return is denoted by

J​\(UB​\(p,s\)\)=𝔼​\[∑t=0H−1rt∣UB​\(p,s\)\]\.J\(U\_\{B\}\(p,s\)\)=\\mathbb\{E\}\\left\[\\sum\_\{t=0\}^\{H\-1\}r\_\{t\}\\mid U\_\{B\}\(p,s\)\\right\]\.\(2\)
Let𝒜valid⊆𝒜\\mathcal\{A\}\_\{\\mathrm\{valid\}\}\\subseteq\\mathcal\{A\}denote the subset of programs that compile, satisfy the required actor\-critic interface, produce tensors with the expected shapes, and return finite outputs\. The full\-budget architecture search objective is

maxp∈𝒜valid⁡𝔼s​\[J​\(UBfull​\(p,s\)\)\],\\max\_\{p\\in\\mathcal\{A\}\_\{\\mathrm\{valid\}\}\}\\;\\mathbb\{E\}\_\{s\}\\left\[J\(U\_\{B\_\{\\mathrm\{full\}\}\}\(p,s\)\)\\right\],\(3\)whereBfullB\_\{\\mathrm\{full\}\}denotes the final training budget\. Directly optimizing this objective is expensive because each architecture evaluation requires an RL training run\. During search, we therefore use a lower training budgetBlow≪BfullB\_\{\\mathrm\{low\}\}\\ll B\_\{\\mathrm\{full\}\}to obtain a proxy fitness signal\. For an architecturepptrained with seedss, the low\-fidelity estimate is

J^low​\(p,s\)=1K​∑k=1KJk​\(UBlow​\(p,s\)\),\\hat\{J\}\_\{\\mathrm\{low\}\}\(p,s\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}J\_\{k\}\\left\(U\_\{B\_\{\\mathrm\{low\}\}\}\(p,s\)\\right\),\(4\)whereJkJ\_\{k\}denotes the return of thekk\-th deterministic evaluation episode\. This formulation separates architecture selection from parameter learning: the outer problem searches over architecture programs, while PPO learns the policy and value parameters for each architecture\.

## 4EVOM

### 4\.1Overview

Fig\.[1](https://arxiv.org/html/2606.26327#S1.F1)summarizes theEVOMworkflow\. Given the problem setup, the LLM design agent first performs population initialization to produce an initial populationP\(0\)=\{p1\(0\),…,pN\(0\)\}P^\{\(0\)\}=\\\{p\_\{1\}^\{\(0\)\},\\ldots,p\_\{N\}^\{\(0\)\}\\\}, whereNNis the population size\. The evolution loop then runs forGGgenerations, repeating reproduction, evaluation, ranking, and selection\. In reproduction, mutation \(edit architectures\) and crossover \(fuse architectures\) generate an offspring populationP′=\{p1′,…,pN′\}P^\{\\prime\}=\\\{p^\{\\prime\}\_\{1\},\\ldots,p^\{\\prime\}\_\{N\}\\\}\. In evaluation, the inner PPO evaluator \(low\-fidelity\) trains each offspring under the low\-budget setting and estimates its return usingKKdeterministic evaluation episodes\. Ranking then orders the union of the current elite population and the offspring population according to the evaluation signal\. Selection keeps the topNNcandidates as the next elite populationP\(t\+1\)P^\{\(t\+1\)\}\. After evolution, final evaluation retrains selected elites with full PPO training, and the output stage returns a deployable elite architecture\. Algorithm[1](https://arxiv.org/html/2606.26327#alg1)describes the workflow\. A key design choice is that the LLM design agent does not observe gradients, trained parameter values, or privileged simulator states\. It receives the*Environment Description*,*Fixed State\-Action*information,*Output Requirements*, and selected architecture designs, and returns executable actor\-critic architecture programs\. The LLM proposes architecture programs for the outer loop, while PPO training and environment interaction remain in the inner loop\.

Algorithm 1EVOM0:Environment, observation dimension

dd, action dimension

mm, population size

NN, generations

GG, low\-budget

BlowB\_\{\\mathrm\{low\}\}, full\-budget

BfullB\_\{\\mathrm\{full\}\}\.

0:Best actor\-critic architecture and final return

rfinalr\_\{\\mathrm\{final\}\}\.

1:Use the LLM design agent for population initialization and obtain

P\(0\)=\{p1\(0\),…,pN\(0\)\}P^\{\(0\)\}=\\\{p\_\{1\}^\{\(0\)\},\\ldots,p\_\{N\}^\{\(0\)\}\\\};

2:Evaluate each architecture

pi\(0\)p\_\{i\}^\{\(0\)\}with the inner PPO evaluator under budget

BlowB\_\{\\mathrm\{low\}\};

3:for

t=0,…,G−1t=0,\\ldots,G\-1do

4:Generate offspring

P′=\{p1′,…,pN′\}P^\{\\prime\}=\\\{p^\{\\prime\}\_\{1\},\\ldots,p^\{\\prime\}\_\{N\}\\\}with the LLM design agent using mutation or crossover over

P\(t\)P^\{\(t\)\};

5:foreach architecture

pi′∈P′p^\{\\prime\}\_\{i\}\\in P^\{\\prime\}do

6:Train the policy and value networks defined by

pi′p^\{\\prime\}\_\{i\}with PPO under budget

BlowB\_\{\\mathrm\{low\}\};

7:Estimate the evaluation signal

rir\_\{i\}of

pi′p^\{\\prime\}\_\{i\}over

KKdeterministic evaluation episodes;

8:endfor

9:Rank candidates in

P\(t\)∪P′P^\{\(t\)\}\\cup P^\{\\prime\}by their evaluation signals;

10:Select the top

NNcandidates as the next elite population

P\(t\+1\)P^\{\(t\+1\)\};

11:endfor

12:Retrain the best architecture in

P\(G\)P^\{\(G\)\}under budget

BfullB\_\{\\mathrm\{full\}\}, evaluate it, and return

rfinalr\_\{\\mathrm\{final\}\};

### 4\.2LLM\-Guided Evolution Loop

The evolution process begins with population initialization\. The LLM design agent receives the task description, input and output dimensions, program interface, and design constraints, and returns the initial populationP\(0\)P^\{\(0\)\}\. Each architecturepi\(0\)p\_\{i\}^\{\(0\)\}is evaluated immediately, so the initial elite population is selected according to empirical feedback rather than model preference alone\. For each generationtt, the outer loop constructs a prompt context containing the*Environment Description*,*Fixed State\-Action*information,*Output Requirements*, selected architecture designs, and an evolution directive\. The evolution directive asks the agent to perform either mutation on one program or crossover on two programs\. The LLM design agent then emits the offspring populationP′P^\{\\prime\}\. Each offspringpi′p^\{\\prime\}\_\{i\}inP′P^\{\\prime\}is compiled, trained with PPO underBlowB\_\{\\mathrm\{low\}\}, evaluated, and then ranked together with the current elite population for selection\.

The loop is agentic in a restricted sense: it maintains state, takes design actions, and observes empirical feedback\. The state is the current elite population together with historical evaluation logs, the actions are program\-generation steps for mutation or crossover, and the observation is the evaluation signal returned by the inner PPO evaluator\. Because each prompt is conditioned on the current elite population,EVOMcan reuse design patterns across generations rather than drawing independent LLM samples at each step\.

The prompt state is intentionally compact\. For mutation, the prompt records*Current Architecture*with its label and network design; for crossover, it records*Parent 1*and*Parent 2*with their labels and network designs\. This gives the LLM design agent enough context to compare design patterns without exposing raw trajectories, gradients, or training internals\. This compact state representation also supports controlled ablations: removing crossover, removing mutation, or replacing the LLM backbone changes the design operator while keeping the inner PPO evaluator and selection rule fixed\.

### 4\.3Open\-Ended Architecture Representation

Eachpi\(t\)p\_\{i\}^\{\(t\)\}is represented as an actor\-critic architecture program, which allows the search to evolve architectures beyond a fixed topology or a closed set of layer choices\. The program representation remains bounded by a common interface, making open\-ended evolution compatible with systematic evaluation\. Each program contains a generation label, a structured design description, and an executable Python program\. The description records layer types, widths, activations, normalization choices, residual structure, initialization, and actor\-critic sharing or asymmetry\. The executable program is the authoritative object used for evaluation\.

The required interface follows the actor\-critic architecture output in Fig\.[1](https://arxiv.org/html/2606.26327#S1.F1)\. Eachpi\(t\)p\_\{i\}^\{\(t\)\}must definePolicyNet\(obs\_length, act\_length\), which returns an action mean of shape\(nb,m\)\(n\_\{b\},m\), andValueNet\(obs\_length\), which returns a value tensor of shape\(nb,1\)\(n\_\{b\},1\), wherenbn\_\{b\}is the batch size\. Depth, width, activation functions, normalization, residual blocks, dropout, initialization, and actor\-critic asymmetry remain open to search\. Each program is written to an isolated file, imported by a worker, instantiated with task\-specific dimensions, and connected to a custom Stable\-Baselines3 actor\-critic policy wrapper\. Programs that fail to compile, produce non\-finite outputs, violate tensor\-shape constraints, or crash during training are assigned a penalty fitness of−1000\-1000\. This penalty safeguards PPO evaluation and makes invalid architectures unlikely to survive elitist selection or be used as parents in later generations\. This validation step is important because executable\-program search is less restricted than a hand\-built grammar, and the empirical evaluation loop requires well\-defined candidates\.

### 4\.4LLM\-Guided Design Operators

The outer loop uses an LLM as a reusable design agent rather than using it for one\-shot architecture generation\. The agent receives elite programs, an evolution directive, and an environment description, and returns a structured description and executable program for each offspring architecturepi′p^\{\\prime\}\_\{i\}\. Population initialization produces the initial set ofNNprograms\. Mutation edits one parent program through semantic changes to components such as activations, capacity, normalization, and residual blocks\. Crossover combines two parent programs into a single offspring program, rather than mechanically splicing program text\. The default schedule uses mixed reproduction, withN/2N/2offspring programs generated by mutation andN/2N/2generated by crossover\.

### 4\.5PPO Evaluation and Final Retraining

For each candidate architecture, thePolicyNetandValueNetare trained with PPO under the low\-budget settingBlowB\_\{\\mathrm\{low\}\}\. The evaluation signal is the mean return overKKdeterministic evaluation episodes\. After evolution, the best architecture is retrained from scratch under the full\-budget settingBfullB\_\{\\mathrm\{full\}\}\. Low\-fidelity evaluation guides the evolutionary search, while full\-budget PPO training determines the reported performance\.

![Refer to caption](https://arxiv.org/html/2606.26327v1/x2.png)Figure 2:Prompt templates for initialization, mutation, and crossover in theEVOMdesign agent\. Red placeholders denote fixed task/interface fields, while blue placeholders denote architecture inputs from the current prompt context\.

## 5Experiments

### 5\.1Experimental Setup and Details

We evaluateEVOMon two MuJoCo continuous\-control tasks:Ant\-v4, with 27\-dimensional observations and 8\-dimensional continuous torque actions, andHalfCheetah\-v4, with 17\-dimensional observations and 6\-dimensional continuous torque actions\. Both tasks useVecNormalize\.

ForEVOM, the evolutionary search uses population sizeN=16N=16andG=20G=20generations\. For each candidate architecture, the low\-budget settingBlowB\_\{\\mathrm\{low\}\}trains PPO for 100k timesteps and estimates the evaluation signal usingK=3K=3deterministic evaluation episodes\. The 100k\-step budget is used as a search\-time proxy to indicate whether an actor\-critic architecture can be optimized by PPO, while keeping population\-level evaluation feasible on consumer GPUs\. The full\-budget settingBfullB\_\{\\mathrm\{full\}\}trains PPO for 5M timesteps, matching the manually designed baseline budget, and we report final results over three independent runs\. The code ofEVOMcan be accessed at[https://github\.com/xiaofangxd/EVOM](https://github.com/xiaofangxd/EVOM)\.

As shown in Fig\.[2](https://arxiv.org/html/2606.26327#S4.F2),EVOMuses three prompt templates for initialization, mutation, and crossover\. Each prompt provides the environment description, fixed state\-action information, and output requirements, including network\-design JSON and executablePolicyNet/ValueNetprogram templates\. Initialization creates a new actor\-critic architecture from the environment description\. Mutation supplies the current architecture label and design, then asks the agent to modify either or both networks while producing a different design\. Crossover supplies two parent labels and designs, and asks the agent to fuse architectural features into a coherent new actor\-critic program\. Each output includes a structured architecture description and a runnable Python program\.

### 5\.2Baselines and Ablations

Manually designed baseline \(Manual PPO\)uses Stable\-Baselines3 PPO with a fixed MLP actor\-critic architectureRaffinet al\.\([2021](https://arxiv.org/html/2606.26327#bib.bib6)\)\.Random searchuses the same LLM generation, PPO evaluation, logging, program interface, and final retraining asEVOM, but samples new architectures without mutation, crossover, and selection\.MLESHuet al\.\([2026](https://arxiv.org/html/2606.26327#bib.bib20)\)evolves executable controller programs, testing direct programmatic control rather than trainable actor\-critic architecture design\. ForMLES, we use the hyperparameters reported in the original paper\. ForEVOM, we conduct ablations on the reproduction operator, population size, and LLM backbone\. These ablations compare mixed reproduction with mutation\-only and crossover\-only variants, evaluateN∈\{8,16,32\}N\\in\\\{8,16,32\\\}, and test DeepSeek V4, Claude Opus 4\.7, and Qwen3\.6 Plus\.

### 5\.3Evaluation Metrics and Visualization

For learning curves, we report the historical\-best reward\. PPO\-based methods use logged evaluation returns, whileMLESuses controller\-level evaluation logs aligned to the same 5M\-step horizontal axis\. The architecture diagrams in Figs\.[3](https://arxiv.org/html/2606.26327#S5.F3)and[4](https://arxiv.org/html/2606.26327#S5.F4)compare the manually designed architecture, the best initial\-generation actor\-critic architecture, and the final selected actor\-critic architecture, highlighting design patterns such as residual blocks, normalization placement, actor\-critic asymmetry, and output\-scale handling\.

![Refer to caption](https://arxiv.org/html/2606.26327v1/x3.png)Figure 3:Main experimental results on Ant\-v4\. The evolution loop improves the historical\-best reward and the elite\-population mean\. The final selected actor\-critic architecture uses residual and normalization components, differing from the manually designed architecture and the best initial\-generation architecture\.![Refer to caption](https://arxiv.org/html/2606.26327v1/x4.png)Figure 4:Main experimental results on HalfCheetah\-v4\. The final selected actor\-critic architecture uses a deeper LayerNorm\-SiLU MLP than the manually designed architecture, while the convergence plots summarize the evolution loop and aligned PPO learning curves under the same full\-budget scale\.

## 6Experimental Results

### 6\.1Full\-Budget Performance

Table[1](https://arxiv.org/html/2606.26327#S6.T1)reports full\-budget performance after final retraining\. On Ant\-v4,EVOMachieves a higher mean return, suggesting that the low\-budget evaluation signal can guide the evolution loop toward effective actor\-critic architectures\. On HalfCheetah\-v4,EVOMalso obtains the highest mean return\. Random search is a stronger baseline on HalfCheetah\-v4 than on Ant\-v4, but its large standard deviation indicates that independent LLM sampling can occasionally find useful architectures while remaining unstable across full\-budget retraining runs\.

### 6\.2Convergence and Architecture Analysis

Figs\.[3](https://arxiv.org/html/2606.26327#S5.F3)and[4](https://arxiv.org/html/2606.26327#S5.F4)summarize evolution\-loop convergence, PPO learning curves, and discovered architectures\. Ant\-v4 shows a clear improvement pattern: both the historical\-best reward and the elite\-population mean increase over generations, and the final selected actor\-critic architecture includes residual and normalization components that are absent from the manually designed baseline\. On HalfCheetah\-v4, the random\-search learning curve reaches high historical\-best rewards but has much larger final\-evaluation variance, whereasEVOMgives a higher and more reliable full\-budget mean\. This gap between historical\-best curves and final retraining results shows why the tables report final full\-budget evaluation rather than search\-time or best\-checkpoint rewards\.

### 6\.3Operator and Population Ablations

Table[2](https://arxiv.org/html/2606.26327#S6.T2)reports ablations on the reproduction operator and population size\. Mixed reproduction gives the best mean return on both tasks\. The crossover\-only variant remains competitive on Ant\-v4 and improves over mutation\-only on HalfCheetah\-v4, but both single\-operator variants underperform mixedEVOM\. This suggests that local editing and design\-pattern recombination are complementary\. Increasing the population size does not lead to monotonic gains, likely because larger candidate pools also increase reliance on noisy low\-budget evaluation signals\.

Table 1:Main comparison under full\-budget evaluation\. Values are mean reward±\\pmstandard deviation\.Table 2:Ablation results under full\-budget evaluation\. Values are mean reward±\\pmstandard deviation\.
### 6\.4LLM Backbone Comparison

Table[3](https://arxiv.org/html/2606.26327#S6.T3)compares different LLM backbones\. Among the tested backbones, Claude Opus 4\.7 yields the highest mean return on Ant\-v4, whereas DeepSeek V4 yields the highest mean return on HalfCheetah\-v4\. The strong but task\-dependent results of Claude Opus 4\.7 and Qwen3\.6 Plus indicate that the LLM backbone changes the distribution of generated architectures, while the PPO evaluator and selection loop remain necessary to identify which designs survive full\-budget training\.

Table 3:LLM backbone comparison for the design agent\. Values are mean reward±\\pmstandard deviation\.
### 6\.5Architecture\-Level Analysis

The selected elite architectures differ from the manual PPO architecture in several recurring design choices\. The Ant\-v4 elite uses residual blocks, normalization, SiLU activations, dropout, and output\-scale handling, while the HalfCheetah\-v4 elite uses a deeper LayerNorm\-SiLU MLP with a larger hidden width than the manually designed 64\-by\-64 MLP\. These design patterns are relevant to PPO because the policy network affects action means and exploration behavior, while the value network affects advantage estimation and policy\-update quality\.

The best initial\-generation architectures serve as controls for one\-shot LLM design\. In both tasks, the final selected architectures differ from these initial designs, especially in residual organization and normalization placement\. This suggests thatEVOMdoes more than select the best initial sample, as the evolution loop preserves and recombines design patterns across generations\. Program\-level search also provides a practical benefit: eachpi\(t\)p\_\{i\}^\{\(t\)\}is an executable actor\-critic architecture program, so the final design can be inspected, reproduced, and reused\. This contrasts with using the LLM only as a natural\-language advisor\.

## 7Discussion and Limitations

EVOMuses low\-budget PPO evaluation to make population\-level search feasible\. The resulting proxy can rank many candidate architectures efficiently, but it may not reliably predict full\-budget performance\. In particular, 100k\-step returns may favor architectures that learn quickly but plateau early, or underestimate architectures that require longer training\. We therefore treat evolution curves as search diagnostics and report final results after retraining selected elites under the 5M\-step full\-budget setting\.

The benefit of architecture evolution is consistent across the two evaluated tasks\. On Ant\-v4,EVOMsubstantially improves over both the manually designed baseline and random search, suggesting that evolutionary refinement helps discover architectures that are not easily obtained by independent sampling\. On HalfCheetah\-v4,EVOMalso outperforms the compared baselines\. These results suggest that meta\-evolution can improve architecture discovery beyond one\-shot or independent LLM\-generated candidates\.

The LLM design agent may generate programs that are invalid, unstable under PPO, or unnecessarily expensive to train\. We assign penalty fitness to failed programs, but future work should reduce such wasted evaluations through program repair or adaptive evaluation budgets\.

The current experiments are limited to two MuJoCo locomotion tasks\. A broader evaluation is needed to assess generality, especially on tasks with different observation structures, reward densities, contact dynamics, or action spaces\. Future work should also study the transfer of discovered architecture patterns across related tasks and coordination with PPO hyperparameter tuning\.

## 8Conclusion

We presentedEVOM, an agentic meta\-evolution framework for automated actor\-critic architecture search\.EVOMuses an open\-ended program representation, LLM\-guided design operators, and low\-fidelity PPO evaluation to refine candidate architectures across generations\. Experiments on Ant\-v4 and HalfCheetah\-v4 show thatEVOMdiscovers actor\-critic architectures that outperform the manually designed PPO architecture,MLES\-style direct policy search, and LLM\-guided random search under full\-budget evaluation\. These results suggest that LLMs can serve as architectural design operators within evolution loops, while policy and value parameters are learned via RL\.

## References

- T\. Bäck \(1996\)Evolutionary algorithms in theory and practice\.Oxford University Press\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p2.1)\.
- G\. Brockman, V\. Cheung, L\. Pettersson, J\. Schneider, J\. Schulman, J\. Tang, and W\. Zaremba \(2016\)Openai gym\.arXiv preprint arXiv:1606\.01540\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p4.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- T\. Elsken, J\. H\. Metzen, and F\. Hutter \(2019\)Neural architecture search: a survey\.Journal of Machine Learning Research20\(55\),pp\. 1–21\.Cited by:[§1](https://arxiv.org/html/2606.26327#S1.p1.1),[§2](https://arxiv.org/html/2606.26327#S2.p1.1),[§2](https://arxiv.org/html/2606.26327#S2.p2.1)\.
- A\. Gaier and D\. Ha \(2019\)Weight agnostic neural networks\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p2.1)\.
- I\. Grondman, L\. Busoniu, G\. A\. Lopes, and R\. Babuska \(2012\)A survey of actor\-critic reinforcement learning: standard and natural policy gradients\.IEEE Transactions on Systems, Man, and Cybernetics, part C \(applications and reviews\)42\(6\),pp\. 1291–1307\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p1.1)\.
- Q\. Hu, T\. Xialiang, M\. Yuan, F\. Liu, Z\. Lu, and Q\. Zhang \(2026\)Multimodal LLM\-assisted evolutionary search for programmatic control policies\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=OHFNJoNtjW)Cited by:[§1](https://arxiv.org/html/2606.26327#S1.p3.1),[§2](https://arxiv.org/html/2606.26327#S2.p3.1),[§5\.2](https://arxiv.org/html/2606.26327#S5.SS2.p1.1)\.
- J\. R\. Koza \(1992\)Genetic programming: on the programming of computers by means of natural selection\.MIT Press\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p2.1)\.
- Y\. LeCun, Y\. Bengio, and G\. Hinton \(2015\)Deep learning\.nature521\(7553\),pp\. 436–444\.Cited by:[§1](https://arxiv.org/html/2606.26327#S1.p1.1)\.
- P\. Li, J\. Hao, H\. Tang, X\. Fu, Y\. Zhen, and K\. Tang \(2024\)Bridging evolutionary algorithms and reinforcement learning: a comprehensive survey on hybrid algorithms\.IEEE Transactions on evolutionary computation\.Cited by:[§1](https://arxiv.org/html/2606.26327#S1.p1.1)\.
- F\. Liu, T\. Xialiang, M\. Yuan, X\. Lin, F\. Luo, Z\. Wang, Z\. Lu, and Q\. Zhang \(2024\)Evolution of heuristics: towards efficient automatic algorithm design using large language model\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=BwAkaxqiLB)Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- F\. Liu, Y\. Yao, P\. Guo, Z\. Yang, X\. Lin, Z\. Zhao, X\. Tong, K\. Mao, Z\. Lu, Z\. Wang, M\. Yuan, and Q\. Zhang \(2026\)A systematic survey on large language models for algorithm design\.ACM Comput\. Surv\.58\(8\)\.External Links:ISSN 0360\-0300,[Link](https://doi.org/10.1145/3787585),[Document](https://dx.doi.org/10.1145/3787585)Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p1.1)\.
- Y\. Liu, Y\. Sun, B\. Xue, M\. Zhang, G\. G\. Yen, and K\. C\. Tan \(2021\)A survey on evolutionary neural architecture search\.IEEE transactions on neural networks and learning systems34\(2\),pp\. 550–570\.Cited by:[§1](https://arxiv.org/html/2606.26327#S1.p1.1)\.
- Y\. J\. Ma, W\. Liang, G\. Wang, D\. Huang, O\. Bastani, D\. Jayaraman, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2024\)Eureka: human\-level reward design via coding large language models\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- Z\. Ma, H\. Guo, Y\. Gong, J\. Zhang, and K\. C\. Tan \(2026\)Toward automated algorithm design: a survey and practical guide to meta\-black\-box\-optimization\.IEEE Transactions on Evolutionary Computation30\(2\),pp\. 667–687\.External Links:[Document](https://dx.doi.org/10.1109/TEVC.2025.3568053)Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p1.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.\(2023\)Self\-refine: iterative refinement with self\-feedback\.Advances in neural information processing systems36,pp\. 46534–46594\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- R\. Miikkulainen \(2025\)Neuroevolution insights into biological neural computation\.Science387\(6735\),pp\. eadp7478\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p1.1)\.
- OpenAI \(2023\)GPT\-4 technical report\.Technical reportOpenAI\.External Links:2303\.08774Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- A\. Raffin, A\. Hill, A\. Gleave, A\. Kanervisto, M\. Ernestus, and N\. Dormann \(2021\)Stable\-baselines3: reliable reinforcement learning implementations\.Journal of Machine Learning Research22\(268\),pp\. 1–8\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p4.1),[§5\.2](https://arxiv.org/html/2606.26327#S5.SS2.p1.1)\.
- E\. Real, A\. Aggarwal, Y\. Huang, and Q\. V\. Le \(2019\)Regularized evolution for image classifier architecture search\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.33,pp\. 4780–4789\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p2.1)\.
- B\. Romera\-Paredes, M\. Barekatain, A\. Novikov, M\. Balog, M\. P\. Kumar, E\. Dupont, F\. J\. Ruiz, J\. S\. Ellenberg, P\. Wang, O\. Fawzi,et al\.\(2024\)Mathematical discoveries from program search with large language models\.Nature625\(7995\),pp\. 468–475\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p1.1),[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- T\. Salimans, J\. Ho, X\. Chen, S\. Sidor, and I\. Sutskever \(2017\)Evolution strategies as a scalable alternative to reinforcement learning\.arXiv preprint arXiv:1703\.03864\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p2.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2606.26327#S1.p1.1),[§2](https://arxiv.org/html/2606.26327#S2.p4.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- K\. O\. Stanley and R\. Miikkulainen \(2002\)Evolving neural networks through augmenting topologies\.Evolutionary Computation10\(2\),pp\. 99–127\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p2.1)\.
- F\. P\. Such, V\. Madhavan, E\. Conti, J\. Lehman, K\. O\. Stanley, and J\. Clune \(2017\)Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning\.arXiv preprint arXiv:1712\.06567\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p2.1)\.
- R\. S\. Sutton, A\. G\. Barto,et al\.\(1998\)Reinforcement learning: an introduction\.MIT press Cambridge\.Cited by:[§1](https://arxiv.org/html/2606.26327#S1.p1.1),[§2](https://arxiv.org/html/2606.26327#S2.p1.1)\.
- M\. Towers, A\. Kwiatkowski, J\. U\. Balis, G\. D\. Cola, T\. Deleu, M\. Goulão, K\. Andreas, M\. Krimmel, A\. KG, R\. D\. L\. Perez\-Vicente, J\. K\. Terry, A\. Pierré, S\. V\. Schulhoff, J\. J\. Tai, H\. Tan, and O\. G\. Younis \(2025\)Gymnasium: a standard interface for reinforcement learning environments\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=qPMLvJxtPK)Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p4.1)\.
- C\. Wang, L\. Jiao, L\. Li, J\. Zhao, G\. Wang, F\. Liu, and S\. Yang \(2026a\)Task\-free adaptive meta black\-box optimization\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=AufVSUgMUo)Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- C\. Wang, L\. Li, L\. Jiao, J\. Zhao, F\. Liu, and S\. Yang \(2026b\)Learning evolution via optimization knowledge adaptation\.IEEE Transactions on Pattern Analysis and Machine Intelligence\(\),pp\. 1–18\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2026.3686919)Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- C\. Wang, J\. Zhao, L\. Jiao, L\. Li, F\. Liu, and S\. Yang \(2025\)When large language models meet evolutionary algorithms: potential enhancements and challenges\.Research8,pp\. 0646\.Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2024\)Voyager: an open\-ended embodied agent with large language models\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=ehfRiF0R3a)Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- M\. Zhang, N\. Desai, J\. Bae, J\. Lorraine, and J\. Ba \(2023\)Using large language models for hyperparameter optimization\.InNeurIPS 2023 Foundation Models for Decision Making Workshop,External Links:[Link](https://openreview.net/forum?id=FUdZ6HEOre)Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p3.1)\.
- B\. Zoph and Q\. V\. Le \(2017\)Neural architecture search with reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2606.26327#S2.p2.1)\.