Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning

# Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
Source: [https://arxiv.org/html/2607.21971](https://arxiv.org/html/2607.21971)
Shujin Wu, Cheng Qian, Xiusi Chen, Heng Ji University of Illinois Urbana\-Champaign \{shujinwu, chengq9, xiusic, hengji\}@illinois\.edu

###### Abstract

Test\-time scaling through iterative self\-evolution with environment feedback, as demonstrated by AlphaEvolve, shows remarkable performance gains\. We hypothesize that the success of such evolution frameworks hinges on meta\-skills, such as self\-reflection with environment feedback, that enable effective multi\-round refinement, yet are largely neglected by traditional post\-training\. To bridge this gap, we presentMetaEvolve, a framework designed to develop these meta\-skills via a data synthesis pipeline, evolution\-aware reinforcement learning \(RL\), and inference\-time evolutionary search\. Concretely, we groundMetaEvolvein coding, where program execution provides natural, continuous reward signals beyond binary correctness\. Building on these signals, we synthesize evolution trajectories as training data, each containing a current program, its fitness score \(combining correctness and efficiency\), and a history of prior attempts, and train the model via RL with verifiable rewards derived from test case execution\. By training on large\-scale code data, we aim to inspire generalizable domain\-agnostic meta\-skills that can transfer broadly to open\-ended problems where such rich training signals are scarce\. Across seven coding benchmarks,MetaEvolveoutperforms the strongest baseline by 10\.01% absolute on in\-distribution tasks and 24\.12% on out\-of\-distribution tasks\. On open\-ended algorithm optimization problems entirely outside the training domain, it further achieves a 46\.9% relative improvement\. These results demonstrate that explicitly cultivating self\-evolution meta\-skills offers a principled path toward more capable and autonomously self\-evolving AI\.

## 1Introduction

Test\-time scaling has emerged as a powerful paradigm for unlocking the potential of large language models \(LLMs\)\(Muennighoffet al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib5); Zhanget al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib6)\)\. Recent frameworks such as AlphaEvolve\(Novikovet al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib7)\)show that iterative refinement with environment feedback can not only improve solution quality but also discover novel results beyond existing human knowledge\(Jianget al\.,[2026](https://arxiv.org/html/2607.21971#bib.bib8); Wanget al\.,[2025b](https://arxiv.org/html/2607.21971#bib.bib9); Acikgozet al\.,[2026](https://arxiv.org/html/2607.21971#bib.bib68)\)\. We hypothesize that this success critically depends on core meta\-skills that go beyond typical problem\-solving abilities\(Gandhiet al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib12); Songet al\.,[2026](https://arxiv.org/html/2607.21971#bib.bib13)\): reflecting on the weaknesses of current solutions, extracting actionable insights from historical attempts, and incorporating external feedback to guide meaningful improvement\.

However, conventional post\-training, whether single\-turn\(Shaoet al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib25); Caoet al\.,[2026](https://arxiv.org/html/2607.21971#bib.bib14)\)or multi\-turn\(Zhouet al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib34);[2025](https://arxiv.org/html/2607.21971#bib.bib33)\), optimizes for task completion rather than feedback\-driven self\-evolution\(Wanget al\.,[2025a](https://arxiv.org/html/2607.21971#bib.bib10)\)\. While recent work observes that standard reinforcement learning \(RL\) can incidentally elicit self\-correction behaviors\(Guoet al\.,[2025a](https://arxiv.org/html/2607.21971#bib.bib17)\), these models are never explicitly trained within the multi\-round evolution process that self\-evolving systems demand at inference time\(Luft,[2014](https://arxiv.org/html/2607.21971#bib.bib15); Wanget al\.,[2023b](https://arxiv.org/html/2607.21971#bib.bib16)\)\. As a result, these critical meta\-skills remain largely underexplored as direct optimization objectives\.

To bridge this gap, we presentMetaEvolve, which explicitly cultivates self\-evolution meta\-skills through RL by directly training models to improve solutions relative to prior attempts — framing iterative refinement as a learnable capability rather than a purely inference\-time heuristic\. We groundMetaEvolvein competitive coding, where self\-evolution has a concrete meaning: producing code that is not only correct but also faster\. Program runtime serves as a natural, fine\-grained reward signal that captures the gap betweenworking codeandbetter code— unlike domains such as mathematics where solutions are typically binary\(Tianet al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib45)\), leaving little room for incremental learning\. We leverage this property to train on large\-scale code data for multi\-round self\-improvement\(Wanget al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib44)\), with the expectation that while the reward signal is domain\-specific, the underlying meta\-skills, such as diagnosing weaknesses, learning from prior attempts, and translating feedback into improvements, are domain\-agnostic and can transfer broadly to open\-ended optimization tasks where such rich training signals are scarce\(Wanget al\.,[2023a](https://arxiv.org/html/2607.21971#bib.bib49); Siet al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib47)\)\.

MetaEvolveinvolves a data synthesis pipeline that constructs evolution\-trajectory\-aware samples, where each sample mirrors the inference\-time self\-evolution setting: the model is exposed to a contextual prompt containing its current solution, performance feedback, and a history of prior attempts, and is expected to produce a meaningfully improved program\. Built on this synthesized data, we then apply RL training guided by verifiable reward signals automatically derived from program execution, which incentivize genuine improvement without requiring any human annotation, making the entire training scalable by design\.

Experiments across seven coding benchmarks show thatMetaEvolveimproves the absolute average improvement rate by 10\.01% on in\-distribution tasks and 24\.12% on out\-of\-distribution tasks over the strongest baseline \(Table[2](https://arxiv.org/html/2607.21971#S3.T2)\)\. On AlgoTune\(Presset al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib65)\), a suite of open\-ended algorithm optimization problems beyond the training domain,MetaEvolveachieves a 46\.91% relative improvement \(Table[4](https://arxiv.org/html/2607.21971#S3.T4)\)\. Program novelty analysis reveals thatMetaEvolveproduces more structurally diverse solutions \(Table[3](https://arxiv.org/html/2607.21971#S3.T3)\), and qualitative analysis confirms emergent self\-evolution behaviors, such as targeted self\-reflection, feedback\-driven strategy refinement, and progressive solution restructuring, that are largely absent in the naive AlphaEvolve \(Figure[2](https://arxiv.org/html/2607.21971#A2.F2)\)\. These results demonstrate that explicitly cultivating self\-evolution meta\-skills offers a principled path toward autonomously self\-evolving AI\.

## 2MetaEvolve

![Refer to caption](https://arxiv.org/html/2607.21971v1/x1.png)Figure 1:Overview of the data generation pipeline ofMetaEvolve\.Step 1samples multiple responses per question and applies two\-stage diversity filtering to retain high\-quality training questions and responses\.Step 2formats each question\-response pair into a contextual prompt containing the current program, its fitness score, and a synthesized evolution history\.We propose to cultivate self\-evolution meta\-skills via RL, grounded in competitive coding where test case execution provides verifiable reward signals for both correctness and runtime efficiency\. The abundance of coding data enables training at scale, with the expectation that acquired meta\-skills transfer to open\-ended tasks where comparable data is expensive to obtain\.MetaEvolveconsists of two stages \(Figure[1](https://arxiv.org/html/2607.21971#S2.F1)\): \(1\) a data synthesis pipeline that constructs evolution\-trajectory\-aware samples mirroring inference\-time self\-evolution, and \(2\) RL training guided by execution\-based rewards that incentivize improvement over prior attempts\.

### 2\.1Data Synthesis

Our data synthesis pipeline is built on top of the coding tasks from the PRIME\-RL/Eurus\-2\-RL\-Data training dataset\(Cuiet al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib43)\), which spans four diverse benchmarks: TACO\(Liet al\.,[2023](https://arxiv.org/html/2607.21971#bib.bib42)\), APPS\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.21971#bib.bib50)\), Codeforces111[https://codeforces\.com/](https://codeforces.com/), and CodeContests\(Liet al\.,[2022](https://arxiv.org/html/2607.21971#bib.bib51)\)\. This diversity in problem sources ensures that our synthesized training data covers a wide range of coding styles, difficulty levels, and problem types, providing a rich foundation for cultivating generalizable self\-evolution meta\-skills\.

#### 2\.1\.1Response Sampling and Diversity Filtering

Given the full training set with over 25k coding questions \(Q\), we begin by sampling 10 candidate responses \(R\) for each, yielding a diverse pool of candidate programs\. To ensure the training data is highly informative, we apply a two\-stage diversity filtering process to keep only the most diverse Q\-R pairs\.

##### Question Filtering

We filter at the question level to retain problems whose solution pools exhibit sufficient diversity\. The intuition is that if all solutions to a problem are too similar, the model cannot effectively learn from the fine\-grained evolution history — it needs meaningfully different programs to compare against in order to identify what subtle changes lead to improvement\. For each pair of solutions to a given problem, we compute a diversity score that combines three complementary signals: \(1\) difference in character length, capturing size variation; \(2\) difference in line count, reflecting structural complexity; and \(3\) difference in character set, measuring lexical variation in keywords and operators\. Line count difference receives the highest weight, as it best distinguishes meaningfully different algorithmic approaches from minor syntactic edits:

Diversity​\(c1,c2\)=0\.1×Δlength\+10×Δlines\+0\.5×Δcharset\\text\{Diversity\}\(c\_\{1\},c\_\{2\}\)=0\.1\\times\\Delta\_\{\\text\{length\}\}\+10\\times\\Delta\_\{\\text\{lines\}\}\+0\.5\\times\\Delta\_\{\\text\{charset\}\}\(1\)The pairwise scores are aggregated per question to produce a single diversity score\. We retain questions whose score falls within a predefined range\[τmin,τmax\]\[\\tau\_\{\\min\},\\tau\_\{\\max\}\]determined by analyzing the score distribution over the full dataset, filtering out both overly homogeneous and overly noisy questions while retaining sufficient data for subsequent response\-level filtering\.

##### Response Filtering

We filter at the individual response level within each surviving question using a greedy selection algorithm that balances response quality and diversity\. Starting with the highest\-quality response as the seed, at each subsequent iteration we compute the average pairwise diversity of every remaining candidate against all already\-selected responses using Equation \([1](https://arxiv.org/html/2607.21971#S2.E1)\), retaining only candidates above a minimum diversity threshold\. Among the eligible candidates, we select the one with the highest combined score:

Scoreselection=0\.3×correctness\+0\.7×diversity\\text\{Score\}\_\{\\text\{selection\}\}=0\.3\\times\\text\{correctness\}\+0\.7\\times\\text\{diversity\}\(2\)We deliberately weight diversity more than twice as heavily as correctness, since correctness is already prioritized in the seed selection step and over\-emphasizing it again would bias the pool toward similar high\-scoring responses\. The algorithm terminates when the desired number of responses is reached or no remaining candidate satisfies the diversity threshold, causing the selected pool size to vary across questions\. This self\-regulating property prevents forcing low\-diversity responses into the pool simply to meet a quota, ensuring every retained response contributes distinct and meaningful training signal\.

Through this two\-stage diversity filtering process, we reduce the original 25k\+ question\-response pairs to a refined subset of approximately 6k\+ high\-quality, diverse pairs\.

#### 2\.1\.2Contextual Prompt Formatting

For each retained question\-response pair \(Q, R\), we construct a contextual prompt that serves as the training input to familiarize the model with the inference\-time setting\. As illustrated in Figure[1](https://arxiv.org/html/2607.21971#S2.F1), each prompt contains four key components: \(1\) the question description; \(2\) the current program along with its fitness score; \(3\) a synthesized evolution history of prior attempts; and \(4\) an instruction asking the model to improve its fitness score\.

Most components map naturally from existing data\. The question description is directly taken from the problem statement and the current program is the code block extracted from the model response\. For the fitness score, we evaluate each program against its test cases and compute a combined score that captures both correctness and runtime efficiency:

Score=min⁡\(1\(speed\+10−6\)×1000,1\.0\)×correctness\\text\{Score\}=\\min\\left\(\\frac\{1\}\{\(\\text\{speed\}\+10^\{\-6\}\)\\times 1000\},1\.0\\right\)\\times\\text\{correctness\}\(3\)where correctness is 1 if all test cases pass and 0 otherwise, and speed is the average runtime in seconds\. The small epsilon10−610^\{\-6\}is added to avoid division by zero and to gracefully handle extremely fast programs\. This formulation maps runtime to a normalized efficiency score in the range\(0,1\]\(0,1\], preventing exceptionally fast programs from disproportionately dominating the combined score\. Multiplying by correctness ensures that quality is always the priority and we only care about efficiency when it is functionally correct\.

##### Evolution History Synthesis

Existing code datasets consist of isolated question\-response pairs with no evolution history\. Rather than relying on expensive online multi\-turn rollouts, we construct realistic evolution trajectories from static datasets, enabling the model to learn multi\-round refinement behaviors within a single\-turn training framework — computationally efficient while still exposing the model to the core structure of iterative self\-evolution\.

For a given current responseRR, we sampleN∈\[1,3\]N\\in\[1,3\]responses from the same question’s response pool to simulate prior attempts\. IfRRis correct, we sample freely from the entire pool, allowing the history to include both incorrect and correct attempts\. IfRRis incorrect, we restrict sampling to other incorrect responses only, so the model must learn to recover from persistent failures\. Each sampled response is evaluated on test cases for correctness, pass rate, and runtime, then sorted in ascending order of performance — weakest attempts first, withRRas the most recent entry\. This gives the model a coherent improvement trajectory to reason over when generating its next refinement\.

### 2\.2Training

We train the model using Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib25)\)\. We denote the current program in the contextual prompt asP1P\_\{1\}and the newly generated program asP2P\_\{2\}\. Both are evaluated against the same test cases to obtain combined scores following Equation[3](https://arxiv.org/html/2607.21971#S2.E3), and the reward is computed as:

r=\{−1ifScore​\(P1\)≥Score​\(P2\)Score​\(P2\)−Score​\(P1\)otherwiser=\\begin\{cases\}\-1&\\text\{if \}\\text\{Score\}\(P\_\{1\}\)\\geq\\text\{Score\}\(P\_\{2\}\)\\\\ \\text\{Score\}\(P\_\{2\}\)\-\\text\{Score\}\(P\_\{1\}\)&\\text\{otherwise\}\\end\{cases\}\(4\)A reward of−1\-1is assigned when the generated solution fails to improve upon the current one, explicitly penalizing stagnation\. Otherwise, the reward is the difference in combined scores, directly incentivizing the model to produce progressively better programs\. We discuss the details of GRPO formulation and implementation in Appendix[A](https://arxiv.org/html/2607.21971#A1)\.

## 3Experiments

### 3\.1Experimental Setup

#### 3\.1\.1Benchmarks

##### Coding

We evaluateMetaEvolveon seven coding benchmarks spanning both in\-distribution and out\-of\-distribution settings\. The in\-distribution benchmarks — APPS, TACO, CodeContests, and Codeforces — correspond to the four data sources used during training\. The out\-of\-distribution benchmarks — Atcoder\(Imajukuet al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib4)\), Leetcode\(Xiaet al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib3)\), and USACO\(Shiet al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib2)\)— are held out entirely during training and serve to evaluate the generalization of the learned self\-evolution meta\-skills\. For each benchmark, we randomly sample 50 test cases to construct the evaluation set, as each problem requires multiple rounds of evolution with full program execution, making large\-scale evaluation prohibitively expensive under limited computational resources\.

##### Open\-ended Problems

To examine whether the learned meta\-skills can broadly transfer to multiple domains, we adopt 8 tasks from the AlgoTune benchmark\(Presset al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib65)\)that are included in OpenEvolve\(Sharma,[2025](https://arxiv.org/html/2607.21971#bib.bib38)\), spanning linear algebra, signal processing, and scientific computing — domains that are fundamentally different from coding\.

#### 3\.1\.2Baselines

We compare against the following test\-time scaling baselines, all implemented on Qwen3\-14B\(Yanget al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib11)\)for fair comparison:

- •Best\-of\-N\(Wanget al\.,[2023c](https://arxiv.org/html/2607.21971#bib.bib30)\): We sample multiple solutions for each problem and select the one with the highest execution\-based score\.
- •Self\-Refine\(Madaanet al\.,[2023](https://arxiv.org/html/2607.21971#bib.bib40)\): The model is presented with its previous solution and the corresponding score, and asked to generate a refined version\. We evaluate both single\-round and multi\-round refinement\.
- •Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2607.21971#bib.bib41)\): The model generates verbal self\-reflections on training problems, which are stored in memory\. At test time, the most similar training problems are retrieved along with their reflections, and the model generates solutions conditioned on this prior experience\.
- •AlphaEvolve\(Novikovet al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib7)\): The model iteratively improves solutions through an evolutionary search algorithm that applies code modifications guided by continuous evaluator feedback\.

Table 1:Ablation study on evolution hyperparameters conducted on 10 TACO problems\. The best configuration within each group is highlighted in gray and shown inbold\. Note that results are only compared within each group of three but not across groups, as cross\-group comparisons are confounded by different fixed conditions\.
#### 3\.1\.3Evolutionary Search Algorithm

##### Pipeline

For AlphaEvolve andMetaEvolve, we use the same evolutionary search algorithm implemented in OpenEvolve\(Sharma,[2025](https://arxiv.org/html/2607.21971#bib.bib38)\)to ensure fair comparison; the only difference lies in the backbone LLM used to generate solutions at each evolution step\. For each problem, we sample 10 responses from the same base model \(Qwen3\-14B\) and select the most efficient correct solution as the shared starting point\. The model then performsKKrounds of evolution: at each round, the controller samplesNNparent programs from an island\-based population, constructs contextual prompts containing each parent’s code, fitness score, and evolution history, and generates one candidate improvement per parent\. Candidates are evaluated against test cases using the combined score \(Equation[3](https://arxiv.org/html/2607.21971#S2.E3)\), and the topMMare retained to update the population, with periodic migration between islands to maintain diversity\. The final best program’s combined score is recorded as the outcome\.

##### Evolution Hyperparameters Setting

We conduct ablation studies over evolution rounds, programs sampled per round, and top\-NNretained, on 10 TACO problems \(details in Appendix[B](https://arxiv.org/html/2607.21971#A2)\)\. Since each group fixes different hyperparameters, we select the best value within each group independently rather than the globally highest\-scoring configuration, which would conflate the effects of different fixed conditions\. Based on the results in Table[1](https://arxiv.org/html/2607.21971#S3.T1), all our following experiments use 10 evolution rounds, 20 parent programs per round with one candidate generated per parent, and the top 5 candidates retained\. The population is maintained across 5 parallel islands with migration every 5 generations, using an exploration\-exploitation\-random ratio of 0\.6:0\.2:0\.2\.

Table 2:Comparison of multiple baselines andMetaEvolveacross seven coding benchmarks, spanning in\-distribution and out\-of\-distribution\.

### 3\.2Experiment Results

#### 3\.2\.1Coding Tasks

##### Performance

Table[2](https://arxiv.org/html/2607.21971#S3.T2)presents the main results across seven coding benchmarks over 10 rounds of self\-evolution\. We highlight the following key findings:

- •Naive test\-time scaling strategies are insufficient:Self\-Refine, Reflexion, and Best\-of\-N sampling yield only marginal improvements \(typically below 5%\), indicating that simply scaling inference compute or applying prompt\-based refinement is inadequate for competitive programming tasks\.
- •Significant gains on in\-distribution benchmarks:MetaEvolveachieves 59\.77% on APPS and 50\.62% on TACO — roughly 17–18 percentage points higher than AlphaEvolve, the strongest baseline\. The only exception is Codeforces\. The gap is small \(1\.29%\), and both methods achieve relatively low scores on this benchmark, suggesting that these problems are difficult enough that they require fundamental algorithmic breakthroughs rather than iterative refinement promoted byMetaEvolve\.
- •Strong generalization to out\-of\-distribution benchmarks:On Atcoder, Leetcode, and USACO, which are entirely held out during training,MetaEvolveachieves 57\.70%, 68\.87%, and 49\.03% respectively, outperforming AlphaEvolve by 30, 29, and 13 percentage points\. The consistently large margins across three diverse out\-of\-distribution sources suggest that our RL\-based training cultivates transferable self\-evolution capabilities, rather than memorizing dataset\-specific refinement patterns\.

##### Novel Program Analysis

To understand whetherMetaEvolveproduces genuinely novel solutions rather than superficial edits, we measure the structural divergence between initial and final programs after 10 rounds of self\-evolution\. We evaluate using three metrics: AST Edit Distance\(Zhang and Shasha,[1989](https://arxiv.org/html/2607.21971#bib.bib66)\), which measures structural difference by counting tree\-editing operations on the abstract syntax tree; CodeBLEU\(Renet al\.,[2020](https://arxiv.org/html/2607.21971#bib.bib67)\), which captures lexical, semantic, and syntactic similarity; and a Combined Novelty Score, defined as the average of normalized AST Edit Distance and\(1−CodeBLEU\)\(1\-\\text\{CodeBLEU\}\), which integrates both structural and lexical divergence into a single measure\. We compute each metric between the final evolved program and all 10 programs initially sampled from the base model, and average across 50 questions per benchmark\. Results in Table[3](https://arxiv.org/html/2607.21971#S3.T3)indicate thatMetaEvolveconsistently achieves higher novelty than AlphaEvolve across nearly all benchmarks and metrics\. On AST Edit Distance, it leads on 6 out of 7 benchmarks, indicating more substantial structural modifications during evolution\. The gap is even more pronounced on CodeBLEU, whereMetaEvolveachieves greater divergence on all 7 benchmarks — with reductions of up to 6 percentage points on APPS and Codeforces\. These results suggest that our RL\-trained model explores more diverse solution strategies rather than relying on local, incremental patches\. This is consistent with the meta\-skills our training encourages: by learning to reflect on prior failures and reason about alternative approaches, the model is more willing to abandon unproductive paths and restructure its programs fundamentally\.

Table 3:Program novelty comparison between AlphaEvolve andMetaEvolveacross seven benchmarks, measured by AST Edit Distance \(↑\\uparrowhigher is more novel\), CodeBLEU \(↓\\downarrowlower is more novel\), and a Combined Novelty Score \(↑\\uparrow\)\. All metrics compare the initial program to the final evolved program after 10 rounds of self\-evolution\.Table 4:Results on AlgoTune benchmark \(8 tasks\), comparing AlphaEvolve andMetaEvolveover 50 rounds of self\-evolution\. We report speedups \(×\\times\) for each of the 8 selected tasks, with an aggregated AlgoTune Score computed via harmonic mean\. Task names are abbreviated:affine=affine\_transform\_2d,convolve2d=convolve2d\_full\_fill,eigenvectors=eigenvectors\_complex,fft\_cmplx=fft\_cmplx\_scipy\_fftpack,fft\_conv=fft\_convolution,lu\_fact=lu\_factorization,poly\_real=polynomial\_real,psd\_cone=psd\_cone\_projection\.

#### 3\.2\.2General Open\-ended Problems

To evaluate whether the meta\-skills learned from competitive programming transfer to fundamentally different problem domains, we conduct evaluation on AlgoTune — a benchmark of open\-ended algorithm optimization tasks where the goal is to produce faster implementations compared to the baseline solution \(Table[4](https://arxiv.org/html/2607.21971#S3.T4)\)\. The tasks involve numerical computing primitives \(e\.g\.,FFT, LU factorization, convolution\) that are structurally distinct from the competitive programming problems seen during training\. Over 8 tasks,MetaEvolveachieves an overall AlgoTune Score of 2\.045×\\timescompared to AlphaEvolve’s 1\.392×\\times, demonstrating a clear advantage in aggregate\. On individual tasks,MetaEvolveachieves notably larger speedups on several problems — for instance, 6\.945×\\timesonaffine\_transform\_2dversus AlphaEvolve’s 1\.072×\\times, and 2\.457×\\timesonpolynomial\_realversus 1\.014×\\times\. The one exception isconvolve2d\_full\_fill, where AlphaEvolve achieves a substantially higher speedup \(291\.3×\\timesvs\. 78\.1×\\times\), likely due to discovering a particularly effective algorithmic shortcut through its evolutionary search\. Nevertheless,MetaEvolvedelivers more consistent improvements across the full task suite, winning on 7 out of 8 individual tasks\. These results provide strong evidence that the self\-reflection and strategic revision capabilities cultivated through our RL training are not confined to code correctness tasks, but generalize to broader algorithmic optimization scenarios requiring fundamentally different reasoning strategies\.

## 4Further Analysis

### 4\.1Ablation Study on Training Data Structure

A key design choice in our training data is the composition of evolution starting points: each training example presents the model with a seed program that is either correct \(requiring efficiency refinement\) or incorrect \(requiring correctness recovery\)\. The balance between these two types may significantly influence which meta\-skills the model prioritizes during training\. To investigate this, we vary the correct\-to\-incorrect seed program ratio across three settings — 50:50, 60:40, and 80:20 — while keeping the total data size fixed\. As shown in Table[5](https://arxiv.org/html/2607.21971#S4.T5), the 80:20 ratio consistently yields the best performance, achieving the highest improvement rate on 5 out of 7 benchmarks, with gains of over 10 percentage points on APPS, TACO, and USACO compared to the balanced setting\. This suggests that emphasizing correct seed programs is beneficial: the model is more frequently exposed to productive evolution trajectories — refining working solutions toward more efficient ones — which better aligns with the iterative improvement behavior expected at inference time\. The remaining incorrect seed programs play a complementary role, providing opportunities for the model to learn the critical meta\-skill of recovering from persistent failures\.

Table 5:Ablation study on the ratio of correct to incorrect responses in training data across seven benchmarks\.Boldindicates the best result per benchmark\.
### 4\.2Qualitative Analysis

To better understand how the cultivated meta\-skills manifest during self\-evolution, we present a qualitative comparison of the reasoning traces produced by AlphaEvolve andMetaEvolvein Figure[2](https://arxiv.org/html/2607.21971#A2.F2)\. The naive AlphaEvolve exhibits several failure modes: it begins with unguided exploration accompanied by self\-doubt \(e\.g\.,“let me recall”, “but I’m not sure”\) without grounding its uncertainty in concrete evidence, drifts into abstract theoretical discussion disconnected from the current code, and falls into circular reasoning that revisits the same considerations without converging on actionable improvements\. In contrast,MetaEvolvedemonstrates structured, purposeful reasoning throughout the evolution process\. It sets feedback\-driven goals grounded in specific problem constraints, validates its understanding against provided test cases, and diagnoses concrete bottlenecks in the current code rather than reasoning abstractly\. Most notably, the trained model actively consults its evolution history — explicitly comparing the current program against prior attempts, identifying why earlier approaches were less efficient, and applying these insights to derive improved solutions\. This pattern ofself\-reflection,learning from historical attempts, andincorporating external feedbackdirectly mirrors the core meta\-skills our training is designed to cultivate\.

### 4\.3Error Recovery Capability

Table 6:Error recovery rate: the percentage of problems with incorrect initial programs that are successfully corrected within 10 rounds of self\-evolution\.We further examine whether models can recover from incorrect initial solutions — cases where all 10 sampled programs from the base model fail, leaving the evolution process with no correct starting point\. As shown in Table[6](https://arxiv.org/html/2607.21971#S4.T6), we compareMetaEvolveagainst AlphaEvolve and a variant trained exclusively on incorrect seed programs \(Trained on Incorrect\)\. Surprisingly, the model trained solely on incorrect examples shows no improvement over AlphaEvolve, achieving 0% recovery on three out of four benchmarks\. In contrast,MetaEvolveachieves the highest recovery rate on 2 out of 4 benchmarks, including 10% on APPS and 18\.18% on Codeforces\. This suggests that error recovery is not effectively learned by simply exposing the model to more incorrect examples during training; rather, it emerges as a byproduct of well\-rounded meta\-skill cultivation — the ability to diagnose failures, reason about alternatives, and restructure solutions — which is better developed through training predominantly on productive evolution trajectories\.

## 5Related Work

Scaling computation at inference time has proven to be a powerful complement to scaling the training time compute and the model size\. Chain\-of\-thought \(CoT\) prompting\(Weiet al\.,[2022](https://arxiv.org/html/2607.21971#bib.bib29)\)first demonstrates that eliciting intermediate reasoning steps unlocks latent capabilities\. Self\-consistency\(Wanget al\.,[2023c](https://arxiv.org/html/2607.21971#bib.bib30)\)extends this by sampling diverse reasoning paths and selecting answers via majority voting, while Tree of Thoughts\(Yaoet al\.,[2023](https://arxiv.org/html/2607.21971#bib.bib31)\)generalizes CoT into tree\-structured deliberate search with self\-evaluation and backtracking\.Snellet al\.\([2024](https://arxiv.org/html/2607.21971#bib.bib32)\)systematically studies inference\-time compute scaling laws and shows that a compute\-optimal test\-time strategy can outperform a 14×\\timeslarger model\. These ideas culminated in OpenAI o1\(OpenAI,[2024](https://arxiv.org/html/2607.21971#bib.bib39)\), which uses large\-scale RL to train internal chains of thought, achieving strong performance on mathematical and programming competitions\. A parallel line of work explores iterative self\-refinement at test time\. Self\-Refine\(Madaanet al\.,[2023](https://arxiv.org/html/2607.21971#bib.bib40)\)implements a generate\-feedback\-refine loop using a single LLM without additional training, while Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2607.21971#bib.bib41)\)equips agents with verbal self\-reflection and episodic memory to improve across sequential attempts\. In the evolutionary paradigm, FunSearch\(Romera\-Paredeset al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib48)\)pairs LLM generation with automated evaluation in an evolutionary loop over program space, yielding genuine mathematical discoveries, and AlphaEvolve\(Novikovet al\.,[2025](https://arxiv.org/html/2607.21971#bib.bib7)\)extends this to a Gemini\-powered coding agent that iteratively refines and evaluates algorithms for scientific and infrastructure optimization\. While these approaches demonstrate the promise of test\-time scaling, they rely on fixed prompting strategies, external search procedures, or evolutionary scaffolds that are not internalized by the model\. Our work instead*trains*the model via RL on synthesized evolution trajectories, making iterative self\-improvement a learned capability rather than an external mechanism\. We also discuss related work about RL for LLMs in Appendix[C](https://arxiv.org/html/2607.21971#A3)\.

## 6Conclusion

We presentedMetaEvolve, a framework that explicitly cultivates core meta\-skills for self\-evolution through reinforcement learning on large\-scale and evolution\-aware competitive coding data\. Experiments demonstrate thatMetaEvolvesubstantially outperforms existing baselines on both in\-distribution and out\-of\-distribution coding benchmarks, and transfers effectively to open\-ended algorithm optimization tasks — confirming that meta\-skills learned from coding generalize to fundamentally different problem domains\. We hope this work inspires a broader shift in how we think about LLM self\-improvement — from scaling inference\-time compute to cultivating the underlying cognitive habits that make iterative refinement effective in the first place\.

## Ethics Statement

This work focuses on improving the self\-evolution capabilities of large language models through reinforcement learning in the competitive programming domain\. All training and evaluation data are drawn from publicly available coding benchmarks, and no private or personally identifiable information is involved\. Our approach aims to enhance models’ ability to iteratively refine solutions — a general\-purpose capability that we believe contributes positively to AI research\. We acknowledge that improved code generation capabilities could potentially be misused; however, the meta\-skills cultivated by our approach \(self\-reflection, learning from feedback, iterative refinement\) are broadly beneficial reasoning abilities rather than domain\-specific exploits\. All experiments are conducted on open\-source models and publicly available benchmarks to support reproducibility\.

## Acknowledgement

This research is based upon work supported by DARPA ITM Program No\. FA8650\-23\-C\-7316, the Office of the Director of National Intelligence \(ODNI\), Intelligence Advanced Research Projects Activity \(IARPA\), via 560000C260018, NSF Molecule Maker Lab Institute, an AI Institute for Molecular Discovery, Synthesis Strategy, and Manufacturing funded by the U\.S\. National Science Foundation under Awards No\. 2019897 and 2505932, DARPA MAGICS Program, NSF NAIRR Award, the AI Research Institutes program by National Science Foundation and the Institute of Education Sciences, U\.S\. Department of Education through Award \# 2229873 \- AI Institute for Transforming Education for Children with Speech and Language Processing Challenges, Amazon\-Illinois Center on AI for Interactive Conversational Experiences \(AICE\), CapitalOne\-Illinois Center for Generative AI Safety, Knowledge Systems, and Cybersecurity \(ASKS\), IBM\-Illinois Discovery Accelerator Institute \(IIDAI\) Center\. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, NSF, or the U\.S\. Government\. The U\.S\. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein\. This research used the Delta and DeltaAI advanced computing and data resources, which are supported by the National Science Foundation \(award OAC 2320345 and award OAC 2005572\) and the State of Illinois\. Delta and DeltaAI are joint efforts of the University of Illinois Urbana\-Champaign and its National Center for Supercomputing Applications\.

## References

- Tool\-r0: self\-evolving llm agents for tool\-learning from zero data\.arXiv preprint arXiv:2602\.21320\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p1.1)\.
- Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.\(2022\)Constitutional AI: harmlessness from AI feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[Appendix C](https://arxiv.org/html/2607.21971#A3.p1.1)\.
- R\. Cao, M\. Chen, J\. Chen, Z\. Cui, Y\. Feng, B\. Hui, Y\. Jing, K\. Li, M\. Li, J\. Lin,et al\.\(2026\)Qwen3\-coder\-next technical report\.arXiv preprint arXiv:2603\.00729\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p2.1)\.
- Z\. Chen, Y\. Deng, H\. Yuan, K\. Ji, and Q\. Gu \(2024\)Self\-play fine\-tuning converts weak language models to strong language models\.InInternational Conference on Machine Learning,Cited by:[Appendix C](https://arxiv.org/html/2607.21971#A3.p1.1)\.
- P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[Appendix C](https://arxiv.org/html/2607.21971#A3.p1.1)\.
- G\. Cui, L\. Yuan, Z\. Wang, H\. Wang, Y\. Zhang, J\. Chen, W\. Li, B\. He, Y\. Fan, T\. Yu,et al\.\(2025\)Process reinforcement through implicit rewards\.arXiv preprint arXiv:2502\.01456\.Cited by:[§2\.1](https://arxiv.org/html/2607.21971#S2.SS1.p1.1)\.
- K\. Gandhi, A\. Chakravarthy, A\. Singh, N\. Lile, and N\. D\. Goodman \(2025\)Cognitive behaviors that enable self\-improving reasoners, or, four habits of highly effective stars\.arXiv preprint arXiv:2503\.01307\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025a\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p2.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025b\)DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[Appendix C](https://arxiv.org/html/2607.21971#A3.p1.1)\.
- D\. Hendrycks, S\. Basart, S\. Kadavath, M\. Mazeika, A\. Zou, D\. Song, and J\. Steinhardt \(2021\)Measuring coding challenge competence with APPS\.InNeurIPS Datasets and Benchmarks Track,Cited by:[§2\.1](https://arxiv.org/html/2607.21971#S2.SS1.p1.1)\.
- Y\. Imajuku, K\. Horie, Y\. Iwata, K\. Aoki, N\. Takahashi, and T\. Akiba \(2025\)Ale\-bench: a benchmark for long\-horizon objective\-driven algorithm engineering\.arXiv preprint arXiv:2506\.09050\.Cited by:[§3\.1\.1](https://arxiv.org/html/2607.21971#S3.SS1.SSS1.Px1.p1.1)\.
- S\. A\. Jacobs, M\. Tanaka, C\. Zhang, M\. Zhang, S\. L\. Song, S\. Rajbhandari, and Y\. He \(2023\)Deepspeed ulysses: system optimizations for enabling training of extreme long sequence transformer models\.arXiv preprint arXiv:2309\.14509\.Cited by:[Appendix A](https://arxiv.org/html/2607.21971#A1.SS0.SSS0.Px2.p1.2)\.
- J\. Jiang, T\. Ding, and Z\. Zhu \(2026\)DeltaEvolve: accelerating scientific discovery through momentum\-driven evolution\.arXiv preprint arXiv:2602\.02919\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[Appendix A](https://arxiv.org/html/2607.21971#A1.SS0.SSS0.Px2.p1.2)\.
- R\. Li, J\. Fu, B\. Zhang, T\. Huang, Z\. Sun, C\. Lyu, G\. Liu, Z\. Jin, and G\. Li \(2023\)Taco: topics in algorithmic code generation dataset\.arXiv preprint arXiv:2312\.14852\.Cited by:[§2\.1](https://arxiv.org/html/2607.21971#S2.SS1.p1.1)\.
- Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. Dal Lago,et al\.\(2022\)Competition\-level code generation with AlphaCode\.Science378\(6624\),pp\. 1092–1097\.Cited by:[§2\.1](https://arxiv.org/html/2607.21971#S2.SS1.p1.1)\.
- C\. D\. B\. Luft \(2014\)Learning from feedback: the neural mechanisms of feedback processing facilitating better performance\.Behavioural brain research261,pp\. 356–368\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p2.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.\(2023\)Self\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[2nd item](https://arxiv.org/html/2607.21971#S3.I1.i2.p1.1),[§5](https://arxiv.org/html/2607.21971#S5.p1.1)\.
- N\. Muennighoff, Z\. Yang, W\. Shi, X\. L\. Li, L\. Fei\-Fei, H\. Hajishirzi, L\. Zettlemoyer, P\. Liang, E\. Candès, and T\. B\. Hashimoto \(2025\)S1: simple test\-time scaling\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 20286–20332\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p1.1)\.
- A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. Ruiz, A\. Mehrabian,et al\.\(2025\)Alphaevolve: a coding agent for scientific and algorithmic discovery\.arXiv preprint arXiv:2506\.13131\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p1.1),[4th item](https://arxiv.org/html/2607.21971#S3.I1.i4.p1.1),[§5](https://arxiv.org/html/2607.21971#S5.p1.1)\.
- OpenAI \(2024\)Learning to reason with LLMs\.Note:OpenAI Blog, September 2024\. System card: arXiv:2412\.16720Cited by:[§5](https://arxiv.org/html/2607.21971#S5.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[Appendix C](https://arxiv.org/html/2607.21971#A3.p1.1)\.
- O\. Press, B\. Amos, H\. Zhao, Y\. Wu, S\. K\. Ainsworth, D\. Krupke, P\. Kidger, T\. Sajed, B\. Stellato, J\. Park, N\. Bosch, E\. Meril, A\. Steppi, A\. Zharmagambetov, F\. Zhang, D\. Perez\-Pineiro, A\. Mercurio, N\. Zhan, T\. Abramovich, K\. Lieret, H\. Zhang, S\. Huang, M\. Bethge, and O\. Press \(2025\)AlgoTune: can language models speed up general\-purpose numerical programs?\.External Links:2507\.15887,[Link](https://arxiv.org/abs/2507.15887)Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p5.1),[§3\.1\.1](https://arxiv.org/html/2607.21971#S3.SS1.SSS1.Px2.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[Appendix C](https://arxiv.org/html/2607.21971#A3.p1.1)\.
- S\. Ren, D\. Guo, S\. Lu, L\. Zhou, S\. Liu, D\. Tang, N\. Sundaresan, M\. Zhou, A\. Blanco, and S\. Ma \(2020\)CodeBLEU: a method for automatic evaluation of code synthesis\.External Links:2009\.10297,[Link](https://arxiv.org/abs/2009.10297)Cited by:[§3\.2\.1](https://arxiv.org/html/2607.21971#S3.SS2.SSS1.Px2.p1.1)\.
- B\. Romera\-Paredes, M\. Barekatain, A\. Novikov, M\. Balog, M\. P\. Kumar, E\. Dupont, F\. J\. R\. Ruiz, J\. S\. Oldfield, P\. Liao, N\. d\. Freitas,et al\.\(2024\)Mathematical discoveries from program search with large language models\.Nature625\(7995\),pp\. 468–475\.Cited by:[§5](https://arxiv.org/html/2607.21971#S5.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.Cited by:[Appendix C](https://arxiv.org/html/2607.21971#A3.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu,et al\.\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Appendix A](https://arxiv.org/html/2607.21971#A1.SS0.SSS0.Px1.p1.11),[Appendix C](https://arxiv.org/html/2607.21971#A3.p1.1),[§1](https://arxiv.org/html/2607.21971#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.21971#S2.SS2.p1.2)\.
- A\. Sharma \(2025\)OpenEvolve: an open\-source evolutionary coding agentExternal Links:[Link](https://github.com/algorithmicsuperintelligence/openevolve)Cited by:[§3\.1\.1](https://arxiv.org/html/2607.21971#S3.SS1.SSS1.Px2.p1.1),[§3\.1\.3](https://arxiv.org/html/2607.21971#S3.SS1.SSS3.Px1.p1.3)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2024\)HybridFlow: a flexible and efficient rlhf framework\.arXiv preprint arXiv: 2409\.19256\.Cited by:[Appendix A](https://arxiv.org/html/2607.21971#A1.SS0.SSS0.Px2.p1.2)\.
- Q\. Shi, M\. Tang, K\. Narasimhan, and S\. Yao \(2024\)Can language models solve olympiad programming?\.arXiv preprint arXiv:2404\.10952\.Cited by:[§3\.1\.1](https://arxiv.org/html/2607.21971#S3.SS1.SSS1.Px1.p1.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[3rd item](https://arxiv.org/html/2607.21971#S3.I1.i3.p1.1),[§5](https://arxiv.org/html/2607.21971#S5.p1.1)\.
- C\. Si, D\. Yang, and T\. Hashimoto \(2024\)Can llms generate novel research ideas? a large\-scale human study with 100\+ nlp researchers\.arXiv preprint arXiv:2409\.04109\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p3.1)\.
- C\. Snell, J\. Lee, K\. Xu, and A\. Kumar \(2024\)Scaling LLM test\-time compute optimally can be more effective than scaling model parameters\.arXiv preprint arXiv:2408\.03314\.Cited by:[§5](https://arxiv.org/html/2607.21971#S5.p1.1)\.
- P\. Song, P\. Han, and N\. Goodman \(2026\)Large language model reasoning failures\.arXiv preprint arXiv:2602\.06176\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p1.1)\.
- M\. Tian, L\. Gao, S\. D\. Zhang, X\. Chen, C\. Fan, X\. Guo, R\. Haas, P\. Ji, K\. Krongchon, Y\. Li,et al\.\(2024\)Scicode: a research coding benchmark curated by scientists\.Advances in Neural Information Processing Systems37,pp\. 30624–30650\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p3.1)\.
- B\. Wang, C\. Lee, N\. Lee, S\. Lin, W\. Dai, Y\. Chen, Y\. Chen, Z\. Yang, Z\. Liu, M\. Shoeybi,et al\.\(2025a\)Nemotron\-cascade: scaling cascaded reinforcement learning for general\-purpose reasoning models\.arXiv preprint arXiv:2512\.13607\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p2.1)\.
- H\. Wang, T\. Fu, Y\. Du, W\. Gao, K\. Huang, Z\. Liu, P\. Chandak, S\. Liu, P\. Van Katwyk, A\. Deac,et al\.\(2023a\)Scientific discovery in the age of artificial intelligence\.Nature620\(7972\),pp\. 47–60\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p3.1)\.
- X\. Wang, Y\. Chen, L\. Yuan, Y\. Zhang, Y\. Li, H\. Peng, and H\. Ji \(2024\)Executable code actions elicit better llm agents\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p3.1)\.
- X\. Wang, Z\. Wang, J\. Liu, Y\. Chen, L\. Yuan, H\. Peng, and H\. Ji \(2023b\)Mint: evaluating llms in multi\-turn interaction with tools and language feedback\.arXiv preprint arXiv:2309\.10691\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p2.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023c\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations,Cited by:[1st item](https://arxiv.org/html/2607.21971#S3.I1.i1.p1.1),[§5](https://arxiv.org/html/2607.21971#S5.p1.1)\.
- Y\. Wang, S\. Su, Z\. Zeng, E\. Xu, L\. Ren, X\. Yang, Z\. Huang, X\. He, L\. Ma, B\. Peng,et al\.\(2025b\)Thetaevolve: test\-time learning on open problems\.arXiv preprint arXiv:2511\.23473\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§5](https://arxiv.org/html/2607.21971#S5.p1.1)\.
- Y\. Xia, W\. Shen, Y\. Wang, J\. K\. Liu, H\. Sun, S\. Wu, J\. Hu, and X\. Xu \(2025\)Leetcodedataset: a temporal dataset for robust evaluation and efficient training of code llms\.arXiv preprint arXiv:2504\.14655\.Cited by:[§3\.1\.1](https://arxiv.org/html/2607.21971#S3.SS1.SSS1.Px1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.1\.2](https://arxiv.org/html/2607.21971#S3.SS1.SSS2.p1.1)\.
- S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. Narasimhan \(2023\)Tree of thoughts: deliberate problem solving with large language models\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§5](https://arxiv.org/html/2607.21971#S5.p1.1)\.
- W\. Yuan, R\. Y\. Pang, K\. Cho, S\. Sukhbaatar, J\. Xu, and J\. Weston \(2024\)Self\-rewarding language models\.InInternational Conference on Machine Learning,Cited by:[Appendix C](https://arxiv.org/html/2607.21971#A3.p1.1)\.
- K\. Zhang and D\. Shasha \(1989\)Simple fast algorithms for the editing distance between trees and related problems\.SIAM J\. Comput\.18\(6\),pp\. 1245–1262\.External Links:ISSN 0097\-5397,[Link](https://doi.org/10.1137/0218082),[Document](https://dx.doi.org/10.1137/0218082)Cited by:[§3\.2\.1](https://arxiv.org/html/2607.21971#S3.SS2.SSS1.Px2.p1.1)\.
- Q\. Zhang, F\. Lyu, Z\. Sun, L\. Wang, W\. Zhang, W\. Hua, H\. Wu, Z\. Guo, Y\. Wang, N\. Muennighoff,et al\.\(2025\)A survey on test\-time scaling in large language models: what, how, where, and how well?\.arXiv preprint arXiv:2503\.24235\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p1.1)\.
- Y\. Zhou, S\. Jiang, Y\. Tian, J\. Weston, S\. Levine, S\. Sukhbaatar, and X\. Li \(2025\)Sweet\-rl: training multi\-turn llm agents on collaborative reasoning tasks\.arXiv preprint arXiv:2503\.15478\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p2.1)\.
- Y\. Zhou, A\. Zanette, J\. Pan, S\. Levine, and A\. Kumar \(2024\)Archer: training language model agents via hierarchical multi\-turn rl\.arXiv preprint arXiv:2402\.19446\.Cited by:[§1](https://arxiv.org/html/2607.21971#S1.p2.1)\.
- D\. M\. Ziegler, N\. Stiennon, J\. Wu, T\. B\. Brown, A\. Radford, D\. Amodei, P\. Christiano, and G\. Irving \(2019\)Fine\-tuning language models from human preferences\.arXiv preprint arXiv:1909\.08593\.Cited by:[Appendix C](https://arxiv.org/html/2607.21971#A3.p1.1)\.

## Appendix AGRPO Training Algorithm

##### Formulation

We perform RL using the Group Relative Policy Optimization \(GRPO\) algorithm\(Shaoet al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib25)\):

ℒGRPO​\(θ\)=−𝔼​\[∑i=1Gmin⁡\(πθ​\(oi\|x\)πθold​\(oi\|x\)​A^i,clip​\(πθ​\(oi\|x\)πθold​\(oi\|x\),1−ϵ,1\+ϵ\)​A^i\)\]−β​𝔻KL​\[πθ∥πref\]\\mathcal\{L\}\_\{\\text\{GRPO\}\}\(\\theta\)=\-\\mathbb\{E\}\\left\[\\sum\_\{i=1\}^\{G\}\\min\\left\(\\frac\{\\pi\_\{\\theta\}\(o\_\{i\}\|x\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(o\_\{i\}\|x\)\}\\hat\{A\}\_\{i\},\\ \\text\{clip\}\\left\(\\frac\{\\pi\_\{\\theta\}\(o\_\{i\}\|x\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(o\_\{i\}\|x\)\},1\-\\epsilon,1\+\\epsilon\\right\)\\hat\{A\}\_\{i\}\\right\)\\right\]\-\\beta\\mathbb\{D\}\_\{\\text\{KL\}\}\\left\[\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\\right\]

\(5\)whereπθ\\pi\_\{\\theta\}is the current policy being optimized,πθold\\pi\_\{\\theta\_\{\\text\{old\}\}\}is the policy from the previous iteration used to compute importance sampling ratios,GGis the group size,oio\_\{i\}is theii\-th sampled output,xxis the input contextual prompt,A^i\\hat\{A\}\_\{i\}is the advantage estimate computed by normalizing rewards within the group,ϵ\\epsilonis the clipping coefficient that constrains the policy update step, andβ\\betacontrols the strength of the KL penalty against the reference policyπref\\pi\_\{\\text\{ref\}\}to prevent the model from deviating too far from its initial behavior\.

##### Implementation

We implement GRPO using the verl framework\(Shenget al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib37)\)\. We set the learning rate to 4e\-6, group sizeGGto 8, batch size to 64, and train for 2 epochs\. The maximum prompt length is 18,432 tokens and the maximum response length is 40,000 tokens\. We use a temperature of 0\.7 for rollout sampling, vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2607.21971#bib.bib36)\)as the rollout engine with tensor parallelism of 4, and Ulysses sequence parallelism\(Jacobset al\.,[2023](https://arxiv.org/html/2607.21971#bib.bib35)\)of 4 for the actor\. The KL penalty coefficientβ\\betais set to 0\. We enable gradient checkpointing for memory efficiency\.

## Appendix BEvolution Ablations

We simultaneously evaluate three key hyperparameters: evolution rounds∈\{5,10,15\}\\in\\\{5,10,15\\\}, number of programs sampled per round∈\{10,20,30\}\\in\\\{10,20,30\\\}, and keep topN∈\{2,5,7\}N\\in\\\{2,5,7\\\}\. For each hyperparameter, we vary its value across three candidates while holding the other two fixed, and confirm one value at a time\. Importantly, results are only compared within each group of three — we do not select the globally highest\-scoring configuration across all nine runs, as cross\-group comparisons are confounded by different fixed conditions\. Within each group, 10 evolution rounds and 20 programs sampled per round yield the best performance \(2\.40% and 2\.76%, respectively\), and we confirm these two values first\. Since keep topNNand number to sample are proportionally correlated — intuitively, retaining more candidates is only meaningful when more programs are sampled — the initial keep topNNablation conducted under sample=10=10\(Table[1](https://arxiv.org/html/2607.21971#S3.T1), third group\) is not directly applicable once we confirm sample=20=20as optimal\. Specifically, whileN=7N=7performs best under sample=10=10\(1\.61%\), this result reflects a suboptimal sampling regime and does not transfer to the confirmed setting\. We therefore re\-validate keep topNNby fixing evolution rounds to 10 and number sampled to 20, and varyN∈\{5,10,14\}N\\in\\\{5,10,14\\\}, findingN=5N=5optimal \(2\.21%, Table[1](https://arxiv.org/html/2607.21971#S3.T1), fourth group\)\.

![Refer to caption](https://arxiv.org/html/2607.21971v1/x2.png)Figure 2:Qualitative comparison of reasoning traces produced by AlphaEvolve andMetaEvolveon the same problem\.
## Appendix CRelated Work about Reinforcement Learning for LLMs

RL has become a central paradigm in post\-training LLMs\. The RLHF pipeline—training a reward model from human preferences and optimizing the policy via proximal policy optimization \(PPO\)\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.21971#bib.bib19)\)—is established byChristianoet al\.\([2017](https://arxiv.org/html/2607.21971#bib.bib20)\)andZiegleret al\.\([2019](https://arxiv.org/html/2607.21971#bib.bib21)\), then extended to instruction following in InstructGPT\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.21971#bib.bib22)\)\. Follow\-up work aims to simplify or extend this pipeline: Constitutional AI\(Baiet al\.,[2022](https://arxiv.org/html/2607.21971#bib.bib23)\)replaces human annotators with AI\-generated feedback guided by natural\-language principles, while Direct Preference Optimization \(DPO;Rafailovet al\.,[2023](https://arxiv.org/html/2607.21971#bib.bib24)\) reparameterizes the RLHF objective into a closed\-form loss that eliminates explicit reward modeling entirely\. More recently, RL has been applied to enhance reasoning capabilities\. DeepSeekMath\(Shaoet al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib25)\)introduces Group Relative Policy Optimization \(GRPO\), which estimates baselines from group\-level reward statistics and removes the need for a critic network, substantially reducing memory overhead\. DeepSeek\-R1\(Guoet al\.,[2025b](https://arxiv.org/html/2607.21971#bib.bib26)\)demonstrates that advanced reasoning behaviors can emerge through pure RL with rule\-based and verifiable rewards\. Self\-Rewarding Language Models\(Yuanet al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib27)\)and SPIN\(Chenet al\.,[2024](https://arxiv.org/html/2607.21971#bib.bib28)\)further explore self\-play and self\-evaluation loops for iterative model improvement\. However, these methods primarily target single\-turn reasoning or alignment\. In contrast, our work uses RL to explicitly cultivate*multi\-turn meta\-skills*—the ability to reflect on prior attempts and progressively refine solutions to open problems—rather than optimizing for a single correct output\.

## Appendix DExample for the Qualitative Analysis

We show the example for the qualitative analysis in Figure[2](https://arxiv.org/html/2607.21971#A2.F2)\. AlphaEvolve exhibits circular, divergent reasoning without actionable insights, whileMetaEvolvedemonstrates structured self\-reflection, grounded optimization, and active use of evolution history\.