LLMZero: Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents

# Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents
Source: [https://arxiv.org/html/2606.18388](https://arxiv.org/html/2606.18388)
Haoyang Fang†, Wei Zhu†, Boran Han†, Alex Zhang, Zhenyu Pan∗,Shuo Yang∗,Shuai Zhang,Jiading Gai,Peng Tang, Cuixiong Hu∗,Xuan Zhu∗,Huzefa Rangwala∗,George Karypis∗,Bernie Wang† Amazon \{haoyfang, weizhuq, boranhan, yuyawang\}@amazon\.com

###### Abstract

RL post\-training strategies are dataset\-dependent and reveal a recurring empirical pattern: capacity parameters accumulate monotonically across stages, while regularization parameters predominantly oscillate in response to shifting training dynamics\. This distinction matters because fixed schedules commit all parameters to fixed trajectories and therefore cannot express the non\-stationary exploration\-exploitation tradeoffs that regularization must track; the principle provides actionable design rules for multi\-stage training\. We discover this throughLLMZero, a system where LLM agents search over training trajectories via tree search, diagnosing pathologies at each checkpoint and proposing coordinated multi\-parameter transitions\. Across 4 diverse GRPO tasks,LLMZerodiscovers strategies that improve over the base model by 9% to 140% relative and over grid search by 6% to 15% relative, consistently outperforming random search and the skill\-based agent\. The structural principle transfers across tasks, providing an explanation for why discovered strategies take qualitatively different forms yet share similar parameter dynamics\.

LLMZero: Discovering Adaptive Training Strategies for RL Post\-Training via LLM Agents

Haoyang Fang†, Wei Zhu†, Boran Han†, Alex Zhang,Zhenyu Pan∗,Shuo Yang∗,Shuai Zhang,Jiading Gai,Peng Tang,Cuixiong Hu∗,Xuan Zhu∗,Huzefa Rangwala∗,George Karypis∗,Bernie Wang†Amazon\{haoyfang, weizhuq, boranhan, yuyawang\}@amazon\.com

†††LLMZero Project Core Team\.††∗Work done at Amazon\.††This is a preprint\. Code will be open\-sourced shortly\. The experiments in this paper were conducted using an internal variation of VeRL that cannot be publicly distributed; we are actively migrating the codebase to ensure full compatibility with the latest public release of VeRL\.## 1Introduction

Fixed training schedules are suboptimal for RL post\-trainingLvet al\.\([2025](https://arxiv.org/html/2606.18388#bib.bib51)\); Wanget al\.\([2025a](https://arxiv.org/html/2606.18388#bib.bib52)\)\. In most recent works, the community has converged on a narrow set of progressive scheduling techniques with all other hyperparameters held constant, applied identically regardless of dataset, model size, or emergent training dynamics\. The dominant approach is gradually increasing response length\(Luoet al\.,[2025b](https://arxiv.org/html/2606.18388#bib.bib28); Chenet al\.,[2025a](https://arxiv.org/html/2606.18388#bib.bib29); Heet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib31); Haoet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib32); Xiaomiet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib33); Luoet al\.,[2025a](https://arxiv.org/html/2606.18388#bib.bib34); Chenet al\.,[2025b](https://arxiv.org/html/2606.18388#bib.bib35); Luoet al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib39); Jiet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib40)\)\. Others gradually increase rollouts\(Luoet al\.,[2025b](https://arxiv.org/html/2606.18388#bib.bib28); Chenet al\.,[2025a](https://arxiv.org/html/2606.18388#bib.bib29); Songet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib30); Luoet al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib39)\), stage training data by progressive difficulty\(Chenet al\.,[2025a](https://arxiv.org/html/2606.18388#bib.bib29); Songet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib30); Heet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib31); Lai and Nissim,[2026](https://arxiv.org/html/2606.18388#bib.bib36); Wanet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib38); Luoet al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib39); Jiet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib40)\), or adopt an oscillating response length schedule\(Songet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib30)\)\. This practice is motivated by training base models to produce increasingly long chains of thought, but is less well\-justified for continued training on models that already generate extended reasoning\. These guidebook\-driven schedules do not systematically specify*when*to trigger a transition,*how much*to adjust, or*which*parameters to change for a given task\. When training dynamics deviate from expectations \(KL divergence spikes, model collapse, stagnating validation\), no systematic mechanism responds \(§[4\.3](https://arxiv.org/html/2606.18388#S4.SS3)\)\.

The strategies our system discovers reveal a recurring structural asymmetry:*capacity parameters \(response length, rollouts\) accumulate monotonically across all four tasks, while regularization parameters \(learning rate, KL coefficient, temperature\) predominantly oscillate*\. Capacity parameters are information\-constructive: reducing response length or rollouts discards what prior stages built\. Regularization parameters track a non\-stationary tradeoff where the optimal exploration\-exploitation balance shifts continuously during training, making monotonic decay a poor fit in practice\. This principle manifests differently per task \(ChemCoTBench\(Liet al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib41)\)uses 5\-stage progressive stabilization with reactive KL spikes, SSMR\-Bench\(Wanget al\.,[2025b](https://arxiv.org/html/2606.18388#bib.bib43)\)benefits from LR/KL oscillation with monotonic capacity expansion, and PaperSearchQA\(Burgesset al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib42)\)uses a “tighten then loosen” pattern to escape convergence plateaus\), but the underlying asymmetry between parameter classes is consistent\.

Why use LLM agents for this search? Simple adaptive controllers \(e\.g\., proportional KL adjustment\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.18388#bib.bib49)\)\) tune one parameter based on one signal\. The strategies we discover require*coordinated*multi\-dimensional transitions, such as simultaneously raising learning rate to escape a plateau while increasing KL penalty to prevent larger steps from causing divergence\. All four best strategies include transitions that change 3\+ parameters simultaneously in coordinated combinations\. These coordinated interventions require understanding the causal relationships between parameters and training dynamics, which is what LLM reasoning provides\.

We introduceLLMZero, a system that discovers adaptive training strategies for RL post\-training\.LLMZerobuilds a tree of training trajectories where LLM agents analyze training dynamics, through textual metrics and visual plots, and then propose targeted hyperparameter transitions conditioned on the observed training state\. An agentic early stopper terminates unpromising branches in real time, focusing the search budget\. UCT \(Upper Confidence bounds applied to Trees\) search balances deepening promising branches against exploring alternatives, while checkpoint\-based composition enables multi\-stage strategies \(§[3\.2](https://arxiv.org/html/2606.18388#S3.SS2)\)\.

Across ChemCoTBench\(Liet al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib41)\), PaperSearchQA\(Burgesset al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib42)\), SSMR\-Bench\(Wanget al\.,[2025b](https://arxiv.org/html/2606.18388#bib.bib43)\), and WildSci\(Liuet al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib44)\),LLMZerodiscovers adaptive strategies that improve over the base model by 9% to 140% relative and over grid search by 6% to 15% relative, consistently outperforming random search and skill\-based LLM agents under the same iterations of refinement \(§[4](https://arxiv.org/html/2606.18388#S4)\)\. Notably,LLMZerofinds its best strategy within the first 12 iterations on 3 of 4 tasks, demonstrating high iteration efficiency\. The discovered strategies exhibit consistent structural patterns that provide actionable design principles for the community \(§[4\.3](https://arxiv.org/html/2606.18388#S4.SS3)\)\.

Beyond the system itself, our findings demonstrate that optimal strategies are dataset\-dependent, consistently exhibit non\-monotonic regularization trajectories, and cannot be prescribed by fixed guidebooks \(§[4\.3\.2](https://arxiv.org/html/2606.18388#S4.SS3.SSS2)\)\. Adaptive training viaLLMZeroconsistently improves over base configurations from 0\.6B to 8B parameters, suggesting that dynamics\-aware strategy search generalizes across model scales \(§[4\.4](https://arxiv.org/html/2606.18388#S4.SS4)\)\.

![Refer to caption](https://arxiv.org/html/2606.18388v1/figures/LLMZeroTeaser.png)Figure 1:Overview ofLLMZero\. The system builds a tree of training trajectories where each node stores a full hyperparameter configuration and resumes from a parent checkpoint, composing multi\-stage adaptive strategies via backtracking\. At each iteration, the*proposer agent*analyzes training dynamics \(rewards, KL divergence, validation scores, gradient norms\) through both text summaries and visual plots, then proposes a new configuration with a checkpoint to resume from\. During training, the*early stopper*periodically overlays the current run’s trajectory against the best completed strategy and terminates dominated runs\.
## 2Preliminary

### 2\.1Training Strategy Formalization

We formalize three paradigms of increasing complexity for RL post\-training\. LetM0M\_\{0\}denote the base model,Θ\\Thetathe hyperparameter space,ℋt=\{\(s1,r1\),…,\(st,rt\)\}\\mathcal\{H\}\_\{t\}=\\\{\(s\_\{1\},r\_\{1\}\),\\ldots,\(s\_\{t\},r\_\{t\}\)\\\}the training history up to steptt, andμ\\mua validation metric\.

###### Definition 1\(Single\-Stage Training\)\.

A single\-stage strategy selects one fixed configuration and trains to completion:

σstatic=⟨\(θ,0\)⟩,θ∗=arg⁡maxθ∈Θ⁡μ​\(𝒯​\(M0,θ\)\)\.\\sigma\_\{\\text\{static\}\}=\\langle\(\\theta,0\)\\rangle,\\quad\\theta^\{\*\}=\\arg\\max\_\{\\theta\\in\\Theta\}\\;\\mu\\\!\\left\(\\mathcal\{T\}\(M\_\{0\},\\theta\)\\right\)\.\(1\)HPO methods \(grid, random, Bayesian\) search overΘ\\Thetaby running multiple independent static trials\.

###### Definition 2\(Multi\-Stage Training\)\.

A multi\-stage strategy is a*guidebook\-driven*sequence ofL\>1L\>1phases:

σmulti\\displaystyle\\sigma\_\{\\text\{multi\}\}=⟨\(θ1,k1\),\(θ2,k2\),…,\(θL,kL\)⟩,\\displaystyle=\\langle\(\\theta\_\{1\},k\_\{1\}\),\(\\theta\_\{2\},k\_\{2\}\),\\ldots,\(\\theta\_\{L\},k\_\{L\}\)\\rangle,θℓ∈Θ,kℓ∈ℕ,\\displaystyle\\quad\\theta\_\{\\ell\}\\in\\Theta,\\;k\_\{\\ell\}\\in\\mathbb\{N\},\(2\)where phaseℓ\\elltrains with configurationθℓ\\theta\_\{\\ell\}starting from stepkℓk\_\{\\ell\}\. The schedule structure is specified before training begins and does not systematically depend on training historyℋt\\mathcal\{H\}\_\{t\}\.

###### Definition 3\(Adaptive Training\)\.

An adaptive strategy selects both the configuration and the checkpoint to resume from based on observations from prior phases\. A transition policyπ\\piselects:

\(θℓ,kℓ,jℓ\)=π​\(\{\(θi,ki,ji,ℋi\)\}i<ℓ\),\(\\theta\_\{\\ell\},\\,k\_\{\\ell\},\\,j\_\{\\ell\}\)=\\pi\\\!\\left\(\\\{\(\\theta\_\{i\},k\_\{i\},j\_\{i\},\\mathcal\{H\}\_\{i\}\)\\\}\_\{i<\\ell\}\\right\),\(3\)wherejℓ∈\{1,…,ℓ−1\}j\_\{\\ell\}\\in\\\{1,\\ldots,\\ell\{\-\}1\\\}identifies which prior phase to resume from\. The policy can backtrack to any earlier checkpoint, enabling branching\. Neither the number of phases, configurations, transition points, nor resumption targets are determined before training begins\.

RL training is inherently non\-stationary: the pace at which exploration must yield to exploitation depends on the dataset, model size, and reward structure, all of which are difficult to predict before training begins\. An adaptive strategy can respond in real time, but the space of possible transition policies is vast, motivating automated search\.

## 3LLMZero

LLMZero\(Figure[1](https://arxiv.org/html/2606.18388#S1.F1)\) builds a tree of training trajectories where each branch point represents a hyperparameter transition chosen based on observed training dynamics\. This section describes how the system discovers adaptive strategies\.

### 3\.1Problem Formulation

Given a dataset𝒟=\{\(xi,mi\)\}i=1N\\mathcal\{D\}=\\\{\(x\_\{i\},m\_\{i\}\)\\\}\_\{i=1\}^\{N\}, a base modelM0M\_\{0\}, a training procedure𝒯\\mathcal\{T\}, and a validation metricμ:𝒴×ℳ→\[0,1\]\\mu:\\mathcal\{Y\}\\times\\mathcal\{M\}\\to\[0,1\], we seek an adaptive strategyσ∗\\sigma^\{\*\}maximizing held\-out performance:

σ∗\\displaystyle\\sigma^\{\*\}=arg​maxσ⁡𝔼\(x,m\)∼𝒟val​\[μ​\(𝒯​\(M0,σ\)​\(x\),m\)\],\\displaystyle=\\operatorname\*\{arg\\,max\}\_\{\\sigma\}\\,\\mathbb\{E\}\_\{\(x,m\)\\sim\\mathcal\{D\}\_\{\\text\{val\}\}\}\\\!\\left\[\\mu\\\!\\big\(\\mathcal\{T\}\(M\_\{0\},\\sigma\)\(x\),m\\big\)\\right\]\\\!,s\.t\.\#iterations≤B,\\displaystyle\\quad\\text\{s\.t\.\}\\quad\\text\{\\\#iterations\}\\leq B,\(4\)whereσ=⟨\(θ1,k1\),…,\(θL,kL\)⟩\\sigma=\\langle\(\\theta\_\{1\},k\_\{1\}\),\\ldots,\(\\theta\_\{L\},k\_\{L\}\)\\rangleis constructed online \(§[2\.1](https://arxiv.org/html/2606.18388#S2.SS1)\) under a small budgetBB\(typically 4–16 iterations, each requiring hours of GPU time\)\.

We model the search as a tree problem\. Each node represents one training phase\. The root uses a default configuration\. Children are created by resuming from a parent checkpoint with modified hyperparameters \(*evolving*\), by fixing failed runs \(*debugging*\), or by starting fresh to maintain diversity\. Each scratch\-to\-leaf path forms a candidate multi\-stage strategy, and siblings reuse the same parent checkpoint for compute sharing\.

### 3\.2Tree Search and Subtree Pruning

LLMZeroperforms Monte Carlo Tree Search \(MCTS\) over training trajectories\. Each iteration selects a node via UCT, expands it by proposing a hyperparameter transition \(or debugging a failure\), executes the training phase, and backpropagates the validation score\. We adopt a UCT variant with scale\-invariant scoring and virtual child competition from prior work; details are reproduced in Appendix[D](https://arxiv.org/html/2606.18388#A4)for completeness\.

##### Subtree pruning\.

A node is marked*terminal*when it can no longer produce children, and terminal subtrees are excluded from selection\. When a failed run is debugged successfully, the successfully fixed descendant is reparented as a sibling of the oldest ancestor in the debug chain, and the entire debug subtree below is pruned\. Terminality propagates upward: a node becomes terminal when fully expanded with all children terminal\.

##### The search loop\.

Algorithm[1](https://arxiv.org/html/2606.18388#alg1)\(Appendix[D](https://arxiv.org/html/2606.18388#A4)\) gives the full procedure\. The key mechanisms are: \(1\) a*proposer agent*that performs multimodal analysis of training dynamics \(§[3\.4](https://arxiv.org/html/2606.18388#S3.SS4)\), \(2\) an*agentic early stopper*that terminates unpromising runs in real time \(§[3\.5](https://arxiv.org/html/2606.18388#S3.SS5)\), and \(3\) forced from\-scratch injection to maintain diversity \(§[3\.3](https://arxiv.org/html/2606.18388#S3.SS3)\)\.

### 3\.3Checkpoint\-Based Strategy Composition

When an evolve node is created, it loads its parent’s model weights at stepkkand continues training with modified hyperparameters\. Successive transitions compose into a multi\-stage strategy along the scratch\-to\-leaf path\.

We resume only themodel weights, reinitializing optimizer state and dataloader position\. This allows arbitrary configuration changes at each transition \(batch size, learning rate, optimizer type\) while avoiding inheritance of suboptimal momentum accumulators\. Because each checkpoint must persist on disk for potential future resumption, storage cost grows linearly with tree depth and training steps\. We use LoRA throughout and save only the adapter weights at each checkpoint\. This design choice means the LoRA rank is fixed across all phases of a strategy and cannot be modified at transitions\.

##### Forced from\-scratch injection\.

To prevent the search from exclusively exploiting a potentially suboptimal initial configuration, we enforce a minimum from\-scratch ratioρmin=0\.2\\rho\_\{\\min\}=0\.2\. Whennscratch/nevolve<ρminn\_\{\\text\{scratch\}\}/n\_\{\\text\{evolve\}\}<\\rho\_\{\\min\}, we select the best\-scoring noden∗n^\{\*\}, create a from\-scratch child that does not count againstn∗n^\{\*\}’s branching limit, and withhold checkpoint information from the proposer to ensure a fresh strategy\. This is the primary mechanism by whichLLMZeroescapes a poor default configuration\. Otherwise, a bad initial run would anchor all subsequent evolve nodes to a weak checkpoint\.

### 3\.4Dynamics\-Aware Transition Proposal

The proposer agent is the core adaptive component\. It receives the parent node’s training configuration, text summaries of step\-level metrics, and the best validation score\. It also performs*visual reasoning*over per\-metric training curve plots, enabling pattern recognition such as trend inflections, divergence onset, and plateau detection that textual summaries alone may miss\. Its reasoning proceeds in four stages: \(1\)*diagnose*the parent run’s training health from primary metrics and diagnostic signals; \(2\) propose*coordinated hyperparameter changes*that address the diagnosed issue, reasoning about causal dependencies between parameters \(e\.g\., raising LR to escape a plateau while increasing KL penalty to prevent divergence\); \(3\) make a*checkpoint decision*\(resume from a specific step or train from scratch\); and \(4\) compute the*epoch budget*based on the chosen batch size and checkpoint step\. Because even the latest LLMs frequently misinterpret domain\-specific GRPO/PPO metrics and hyperparameters, we inject human\-written descriptions to ground the agent’s reasoning without restricting which parameters it can modify \(Appendix[F](https://arxiv.org/html/2606.18388#A6)\)\. The full prompt template is in Appendix[G\.1](https://arxiv.org/html/2606.18388#A7.SS1)\.

### 3\.5Agentic Early Stopping

Every 900 seconds during training, the early stopper samples current metrics and generates overlay plots comparing the current run’s trajectory \(blue\) against the best completed strategy \(green\)\. It outputsCONTINUEorSTOPwith explicit reasoning about whether the current trajectory can realistically overtake the incumbent \(full prompt in Appendix[G\.2](https://arxiv.org/html/2606.18388#A7.SS2)\)\. Across all 4 tasks, the early stopper terminated 62\.1% of nodes before completion, reducing total GPU consumption by an estimated 40–60% relative to running all nodes to completion\.

### 3\.6Automated Pipeline

LLMZeroincludes an automated pipeline that handles data preparation, reward function implementation, training code generation, and job execution\. In our evaluation, data processing and reward functions are fixed across all methods to ensure a fair comparison\. The adaptive strategy search follows a fixed workflow; the essential LLM\-based components are theproposer agent\(§[3\.4](https://arxiv.org/html/2606.18388#S3.SS4)\) and theagentic early stopper\(§[3\.5](https://arxiv.org/html/2606.18388#S3.SS5)\)\. Pipeline details are in Appendix[B](https://arxiv.org/html/2606.18388#A2)\.

## 4Experiments

We evaluateLLMZeroacross 4 diverse GRPO tasks to answer:\(RQ1\)Can adaptive strategies outperform static configurations?\(RQ2\)Are discovered patterns dataset\-dependent?\(RQ3\)How does performance scale with model size?\(RQ4\)How do individual components contribute?\(RQ5\)Do strategies transfer across tasks?

Table 1:Main results: test score \(%\) at the best validation node within 16 iterations on Qwen3\-4B\.Bold: best per column \(aggregate only\)\.Underline: second best\. Subscripts show gain over the base model\. ChemCoT reports category\-averaged test accuracy across 3 task families \(mol\. optimization, 6 subtasks at 0% for all methods, is omitted\)\. SSMR shows per\-subtask test scores \(Scl=scale, Bea=beat, Cho=chord, Int=interval\)\. WildSci practitioner config scores below Qwen3\-4B on weighted test aggregate due to domain\-level trade\-offs \(see §[4\.3\.1](https://arxiv.org/html/2606.18388#S4.SS3.SSS1)\)\.ChemCoT\(Chem\)PaperSearchQASSMR\(Music\)WildSciTypeUndEditRxnAvgBioSclBeaChoIntAvgSciQwen3\-4B–40\.58\.02\.316\.931\.660\.044\.850\.450\.451\.453\.6PractitionerStatic61\.1\(\+20\.6\)23\.0\(\+15\.0\)14\.9\(\+12\.6\)33\.0\(\+16\.1\)39\.0\(\+7\.4\)76\.8\(\+16\.8\)60\.0\(\+15\.2\)65\.6\(\+15\.2\)60\.8\(\+10\.4\)65\.8\(\+14\.4\)53\.2\(−\-0\.4\)Random searchStatic66\.4\(\+25\.9\)23\.0\(\+15\.0\)28\.7\(\+26\.4\)39\.4\(\+22\.5\)37\.6\(\+6\.0\)90\.4\(\+30\.4\)77\.6\(\+32\.8\)68\.8\(\+18\.4\)60\.8\(\+10\.4\)74\.4\(\+23\.0\)55\.8\(\+2\.2\)Grid searchStatic61\.2\(\+20\.7\)23\.4\(\+15\.4\)20\.9\(\+18\.6\)35\.2\(\+18\.3\)39\.0\(\+7\.4\)87\.2\(\+27\.2\)84\.0\(\+39\.2\)69\.6\(\+19\.2\)69\.6\(\+19\.2\)77\.6\(\+26\.2\)53\.0\(−\-0\.6\)Skill\-based LLM agentAdaptive61\.8\(\+21\.3\)21\.4\(\+13\.4\)27\.7\(\+25\.4\)37\.0\(\+20\.1\)40\.2\(\+8\.6\)96\.0\(\+36\.0\)88\.0\(\+43\.2\)74\.4\(\+24\.0\)61\.6\(\+11\.2\)80\.0\(\+28\.6\)56\.6\(\+3\.0\)LLMZeroAdaptive69\.8\(\+29\.3\)33\.3\(\+25\.3\)18\.5\(\+16\.2\)40\.5\(\+23\.6\)42\.6\(\+11\.0\)94\.4\(\+34\.4\)81\.6\(\+36\.8\)77\.6\(\+27\.2\)75\.2\(\+24\.8\)82\.2\(\+30\.8\)58\.5\(\+4\.9\)

### 4\.1Setup

##### Tasks\.

We evaluate on ChemCoTBench\(Liet al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib41)\), PaperSearchQA\(Burgesset al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib42)\), SSMR\-Bench\(Wanget al\.,[2025b](https://arxiv.org/html/2606.18388#bib.bib43)\), and WildSci\(Liuet al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib44)\)\. Each dataset is uniformly subsampled to 5,000 train, 500 validation, and 500 test examples to keep per\-iteration training time tractable for search\. All tasks use GRPO via VeRL\(Shenget al\.,[2024](https://arxiv.org/html/2606.18388#bib.bib50)\)\(Appendix[B\.1](https://arxiv.org/html/2606.18388#A2.SS1)\)\.

##### Models\.

Primary evaluation usesQwen3\-4B\(LoRA, rank=64\)\. Scaling analysis spansQwen3\-0\.6Bthrough8B\. Base model evaluation: greedy decoding, max response length=8192\. Infrastructure: Ray clusters on EKS, 32–64 A100 40G GPUs\.

##### Baselines\.

We compare against: \(1\) apractitioner baseline\(fixed GRPO recipe tuned on separate tasks\), \(2\)random search\(8 trials from broad HP ranges\), \(3\)grid search\(8 trials over LR×\\timesLoRA rank, selected as the most efficient range based on internal experience\), and \(4\) askill\-based LLM agentbuilt on Claude Code as the orchestration backend with Claude Opus 4\.6 at high reasoning effort, which autonomously plans iterations and can stop/resume from any checkpoint without tree search or visual reasoning\. Because its autonomous orchestration maintains full conversation context, each task costs 44–144×\\timesmore in API thanLLMZerowith its fixed workflow, limiting it to 6–9 iterations in practice\. Full baseline configurations are in Appendix[B](https://arxiv.org/html/2606.18388#A2); budget fairness details are in Appendix[B\.3](https://arxiv.org/html/2606.18388#A2.SS3)\.

### 4\.2Main Results \(RQ1\)

Table[1](https://arxiv.org/html/2606.18388#S4.T1)presents the test score at the best validation step for each method\. The comparison isolates two questions\. First, does adaptive beat static \(LLMZerovs\. practitioner/random/grid search\)? Second, does the fixed workflow with tree search outperform a general\-purpose LLM agent \(LLMZerovs\. skill\-based LLM agent\)?LLMZerooutperforms all static baselines on every task\. Notably, the WildSci practitioner config scores below the base model \(53\.2% vs\. 53\.6%\), yetLLMZerorecovers to 58\.5%\.

##### Compute efficiency\.

Figure[2](https://arxiv.org/html/2606.18388#S4.F2)plots best\-so\-far test score against cumulative GPU\-hours for each method\.LLMZeroreaches the highest final test score on all 4 tasks under comparable total GPU compute \(4,159–10,013 GPU\-hours forLLMZerovs\. 4,543–16,846 for HPO baselines\)\. Early stopping terminates 56–70% of nodes before completion\. The forced from\-scratch injection mechanism \(§[3\.3](https://arxiv.org/html/2606.18388#S3.SS3)\), which starts new trajectories from scratch at a given ratio to maintain diversity, explains whyLLMZerouses more total training time than the skill\-based agent\.

##### API cost\.

LLMZero’s fixed workflow consumes 44–144×\\timesless API cost than the skill\-based agent \($48 vs\. $3,545 total; Table[5](https://arxiv.org/html/2606.18388#A2.T5)in Appendix[B\.4](https://arxiv.org/html/2606.18388#A2.SS4)\)\.

![Refer to caption](https://arxiv.org/html/2606.18388v1/x1.png)Figure 2:Test score at the best\-validation run so far vs\. cumulative GPU\-hours\. Dots show per\-run/node test scores; step curves track the test score of whichever run has the highest validation so far\.LLMZero\(red\) achieves the highest final test score on all 4 tasks under comparable total compute\.![Refer to caption](https://arxiv.org/html/2606.18388v1/x2.png)
![Refer to caption](https://arxiv.org/html/2606.18388v1/x3.png)
![Refer to caption](https://arxiv.org/html/2606.18388v1/x4.png)
![Refer to caption](https://arxiv.org/html/2606.18388v1/x5.png)

Figure 3:Best adaptive strategies across all four tasks\. Green solid: validation score\. Blue dashed: test score\. Each point is one phase with annotations summarizing observed training dynamics\.

### 4\.3Analysis of Discovered Strategies \(RQ2\)

We examine whether the adaptive strategiesLLMZerodiscovers are dataset\-dependent and what structural patterns emerge\. Figure[3](https://arxiv.org/html/2606.18388#S4.F3)visualizes the best strategy for each task\.

#### 4\.3\.1Per\-Dataset Strategies

##### ChemCoT \(Chemistry\)\.

The best strategy is a 5\-phase chain \(Figure[3](https://arxiv.org/html/2606.18388#S4.F3)\)\. After initial training \(val=29\.2%\), the system widens the clip range and enables advantage normalization while lowering temperature \(Phase 2, val=31\.0%\), then observes a 16×\\timesKL loss spike coinciding with response length inflation \(1630→\\to2278 tokens\) without validation improvement, responding with a 5×\\timesKL penalty increase \(Phase 3, val=33\.2%\)\. This KL divergence spike preceded validation degradation by 1–2 phases, suggesting KL as a leading indicator that practitioners should monitor proactively\. Phase 4 expands response capacity \(6144→\\to7168 tokens\) with a transient regression \(val=32\.8%\), before Phase 5 addresses low gradient norms \(<<0\.001\) and flat validation by raising LR and rollouts while reducing batch size \(val=35\.6%\)\.

##### Failure case: molecular optimization\.

Nine subtasks \(allmol\_opt\_\*,rxn\_retro,rxn\_nepp,rxn\_mechanism\) score 0% across all methods includingLLMZero\. Molecular optimization requires valid SMILES string generation, which is a learned structural capability rather than a reasoning capability\. No training strategy can elicit a skill the base model fundamentally lacks at 4B scale\. This demonstrates the boundary of adaptive scheduling: it optimizes*how*to train but cannot compensate for missing model capacity\.

##### PaperSearchQA \(Biomedical QA\)\.

The best strategy is a 4\-phase chain \(Figure[3](https://arxiv.org/html/2606.18388#S4.F3), test=42\.6%, \+11\.0 pp over the base model\)\. LR and temperature are progressively tightened through Phases 2–3 \(val stabilizes at 40\.8%\), then the proposer diagnoses stagnation \(near\-zero clip ratio, low gradient norms\) and reverses course at Phase 4: LR doubles, temperature increases, and batch size decreases to break the plateau \(val=42\.0%\)\. The KL coefficient increases monotonically throughout \(0\.001→0\.010\.001\\to 0\.01\), unlike the non\-monotonic trajectories on other tasks\.

##### SSMR\-Bench \(Music Theory\)\.

The best strategy is a 4\-phase chain \(Figure[3](https://arxiv.org/html/2606.18388#S4.F3)\)\. From Phase 1 \(val=70\.0%\), Phase 2 applies multiple conservative changes simultaneously: reducing LR to 3e\-5, doubling KL to 0\.002, reducing gradient clipping to 0\.5, enabling advantage normalization, and raising temperature to 1\.1\. This over\-constrains learning and causes regression to 67\.2%\. Phase 3 reverses the core constraints \(restoring LR to 5e\-5, relaxing KL to 0\.001, widening clip to 0\.30\) while retaining the higher temperature, driving a \+15\.8 pp recovery \(val=83\.0%\)\. Phase 4 re\-tightens for convergence \(LR=3e\-5, KL=0\.002, T=1\.0\) with expanded response length \(6144→\\to7168 tokens\), reaching val=87\.0% \(test=82\.2%\)\. LR and KL oscillate across phases while epochs and response capacity accumulate monotonically\. The agentic early stopper terminated 7 of 10 explored nodes\.

##### WildSci \(Multi\-Discipline Science\)\.

The best strategy is a 4\-phase chain \(Figure[3](https://arxiv.org/html/2606.18388#S4.F3), test=58\.5%\)\. After aggressive LR scaling in Phase 2 triggers a KL divergence spike, the system tightens constraints in Phase 3\. This intervention causes an entropy collapse\. During this collapse, the validation score increases slightly, but the hidden test score drops\. To recover entropy, Phase 4 relaxes the KL penalty, lowers the LR, and raises both the temperature and the clip ratio\. This successfully recovers the hidden test performance, though it does not fully surpass the Phase 2 peak due to the negative effects of Phase 3\. Noticeably, the validation score continued to rise even as test performance degraded during the entropy collapse, which suggests that monitoring comprehensive training dynamics is more robust than optimizing solely for the validation score\.

#### 4\.3\.2Cross\-Task Structural Patterns

Three empirical observations emerge from comparing strategies across tasks:

##### Dataset\-specific dynamics determine strategy structure\.

Each task is characterized by a different observable pattern\. ChemCoT exhibits KL divergence spikes with response length inflation, PaperSearchQA shows stagnating validation with near\-zero clip ratios, SSMR\-Bench shows validation regression under conservative hyperparameters, and WildSci shows model collapse after aggressive LR scaling\. These patterns emerge unpredictably during training \(ChemCoT’s KL spike appears only after 2 phases\) and require interventions calibrated to their severity\. The KL coefficient trajectories directly reflect these differences: reactive spikes for KL divergence, monotonic increase for progressive stabilization, symmetric oscillation for validation regression, and tighten\-then\-relax for model collapse\. No fixed schedule can anticipate which pattern will dominate or when it will manifest\.

##### Multi\-dimensional transitions are effective\.

In all 4 tasks, the highest\-gain transition changes 3\+ hyperparameters simultaneously in coordinated combinations\. For example, ChemCoT Phase 5 simultaneously raises LR and increases KL penalty, a combination the proposer reasoned would escape the plateau without causing divergence\. These coordinated interventions are unlikely to be discovered by sampling or tuning parameters independently\.

Notably, the KL coefficient is the most frequently adjusted parameter across all best strategies \(changed in 12 of 13 transitions\), yet it is held constant in all surveyed multi\-stage works \(§[1](https://arxiv.org/html/2606.18388#S1)\)\. Its non\-monotonic trajectory appears load\-bearing: on 3 of 4 tasks, the best strategies tighten KL reactively and relax it proactively \(PaperSearchQA is the exception, where KL increases monotonically while LR and temperature oscillate instead\), suggesting that a fixed KL schedule would miss these dynamics on most tasks\.

##### Capacity parameters accumulate while regularization parameters oscillate\.

Across all 4 best strategies, response length and rollout count exhibit*zero*direction reversals\. Learning rate and temperature exhibit 1–2 reversals on every task; the KL coefficient reverses on 3 of 4 tasks but increases monotonically on PaperSearchQA, where it serves as a progressively tightening constraint to stabilize noisy updates\. This provides a candidate design principle:*capacity parameters should accumulate monotonically while regularization parameters should be free to oscillate in response to shifting training dynamics\.*The asymmetry reflects that capacity parameters are information\-constructive \(reducing them truncates reasoning chains or increases gradient variance\), while regularization parameters control an exploration\-exploitation tradeoff whose optimal balance shifts continuously during training\.

##### Diagnosis quality\.

We human\-verified all stated observations in the proposer’s diagnoses against metric traces \(Tables[8](https://arxiv.org/html/2606.18388#A3.T8)–[11](https://arxiv.org/html/2606.18388#A3.T11)\): all are correct\. Of 13 non\-initial transitions across the 4 best strategies, 11 result in improved validation scores\. On 3 of 4 tasks, test scores increase monotonically along the best path \(Figure[3](https://arxiv.org/html/2606.18388#S4.F3)\); on WildSci, one transition regresses but the subsequent phase recovers\.

### 4\.4Scaling Analysis \(RQ3\)

0\.6B1\.7B4B8B0202040406060808010010032\.432\.431\.631\.651\.451\.452\.652\.653\.052\.265\.8OOM72\.472\.462\.462\.482\.282\.283\.683\.6Model SizeAverage Accuracy \(%\)Qwen3 \(base\)PractitionerLLMZeroFigure 4:Model scaling on SSMR\-Bench \(average across 4 subtasks\)\.LLMZeroconsistently outperforms baselines across all sizes\. Practitioner config failed \(OOM\) on 8B;LLMZeroautonomously found a working configuration\. Per\-subtask breakdown in Table[7](https://arxiv.org/html/2606.18388#A3.T7)\(Appendix[C\.3](https://arxiv.org/html/2606.18388#A3.SS3)\)\.Figure[4](https://arxiv.org/html/2606.18388#S4.F4)reports average accuracy on SSMR\-Bench across model sizes\.LLMZeroconsistently outperforms baselines from 0\.6B to 8B, with gains of \+30\.8 pp to \+40\.0 pp over the base model\. The practitioner config failed with OOM on 8B, whileLLMZeroautonomously discovered a working configuration \(83\.6%\)\. This illustrates a practical advantage of dynamics\-aware search: it can navigate infrastructure failures that would require manual intervention in fixed\-schedule approaches, effectively expanding the feasible configuration space at larger scales\. Per\-subtask results are in Table[7](https://arxiv.org/html/2606.18388#A3.T7)\(Appendix[C\.3](https://arxiv.org/html/2606.18388#A3.SS3)\)\.

### 4\.5Ablation Studies \(RQ4\)

Table[2](https://arxiv.org/html/2606.18388#S4.T2)ablates key components on SSMR\-Bench to distinguish which drive accuracy gains versus compute efficiency\. Removing multi\-stage composition drops accuracy by 9\.4 pp \(82\.2%→\\to72\.8%\), confirming that the ability to compose adaptive multi\-stage strategies is the primary driver of improvement; without it, the system reduces to selecting the best single\-phase configuration from the same search budget\. Removing visual reasoning or early stopping yields on\-par accuracy \(82\.4% and 82\.8%\), but at substantially worse compute efficiency \(0\.6×\\timesand 0\.29×\\timesrespectively\)\. Visual reasoning enables the early stopper to reliably judge trajectory dominance from overlay plots, while early stopping itself focuses compute on promising branches\. Together they reduce wall\-clock time without sacrificing strategy quality\.

### 4\.6Strategy Transfer \(RQ5\)

We test whether multi\-stage structure alone drives improvement by executing two fixed schedules on three held\-out tasks \(Table[3](https://arxiv.org/html/2606.18388#S4.T3)\): the discovered SSMR\-Bench 4\-phase strategy, and a Capacity Guidebook that progressively increases only response length and rollout count following the dominant community practice\(Luoet al\.,[2025b](https://arxiv.org/html/2606.18388#bib.bib28); Chenet al\.,[2025a](https://arxiv.org/html/2606.18388#bib.bib29)\)\(Table[13](https://arxiv.org/html/2606.18388#A5.T13)\), with stage durations estimated from our successful trajectories\. Both fixed schedules improve over the base model on all tasks \(\+4\.6 to \+24\.4 pp for SSMR transfer, \+4\.8 to \+19\.7 pp for the Guidebook\), confirming that multi\-stage training is broadly beneficial\. However, their gains are inconsistent: the SSMR transfer nearly matches adaptive search on WildSci \(58\.3% vs\. 58\.5%\) but underperforms by 6\.4 pp on PaperSearchQA, while the Guidebook lags the SSMR transfer by 4\.7 pp on ChemCoT\. These inconsistencies demonstrate that fixed schedules cannot always reliably generalize across tasks without adapting to observed training dynamics\. Notably, the strong transfer performance on some tasks may benefit from identical dataset sizes and the same base model, which produce similar training dynamics; adaptive search remains necessary for robustness when dataset scale or model family varies \(More analysis in Appendix[C\.5](https://arxiv.org/html/2606.18388#A3.SS5)\)\.

Table 2:Ablation on SSMR\-Bench: test accuracy \(%\) at best validation node\. Speed is relative compute efficiency normalized to the full system\.Table 3:Strategy transfer: test scores \(%\) for fixed multi\-stage schedules on held\-out tasks\. SSMR transfer applies the discovered 4\-phase strategy\. Capacity Guidebook applies only progressive capacity scaling\.

## 5Conclusion

We have identified a recurring structural asymmetry in the optimal multi\-stage reinforcement learning paradigm for LLMs: capacity parameters \(response length, rollouts, etc\.\) accumulate monotonically while regularization parameters \(learning rate, KL coefficient, temperature, etc\.\) predominantly oscillate in response to shifting training dynamics\. We discovered this throughLLMZero, a system where LLM agents reason about training dynamics at each checkpoint, proposing coordinated multi\-parameter transitions that address diagnosed pathologies\. Across 4 diverse GRPO tasks, adaptive strategies embodying this principle improve over the base model by 9% to 140% relative and over grid search by 6% to 15% relative, outperforming all baselines\. These findings suggest that the multi\-stage training paradigm’s current focus on staging one or two capacity parameters leaves substantial performance on the table, and that dynamics\-aware, multi\-dimensional adaptation is needed to realize its full potential\.

## Limitations

Our system expands one node at a time; a hybrid with population\-based training\(Jaderberget al\.,[2017](https://arxiv.org/html/2606.18388#bib.bib10)\)that maintains multiple trajectories with LLM\-guided transitions would combine broad exploration with intelligent proposals and is a natural next step\. The search uses 500 validation examples for checkpoint selection and early stopping to maintain reasonable per\-step evaluation time, with strict separation from test data; a larger validation set would yield more robust checkpoint selection at the cost of longer evaluation cycles\. All experiments use the Qwen3 family; while scaling from 0\.6B to 8B shows robustness within this family, generalization to larger models or other architectures remains unverified\. All datasets are subsampled to 5,000 training examples to keep per\-iteration search tractable; validating that discovered strategies and structural patterns hold at production data scales requires substantially more compute and is left to future work\.

## References

- Random search for hyper\-parameter optimization\.Journal of Machine Learning Research13\(10\),pp\. 281–305\.External Links:[Link](http://jmlr.org/papers/v13/bergstra12a.html)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px2.p1.1)\.
- J\. Burgess, J\. N\. Hansen, D\. Peng, Y\. Zhang, A\. Lozano, M\. W\. Sun, E\. Lundberg, and S\. Yeung\-Levy \(2026\)PaperSearchQA: learning to search and reason over scientific papers with rlvr\.External Links:2601\.18207,[Link](https://arxiv.org/abs/2601.18207)Cited by:[§1](https://arxiv.org/html/2606.18388#S1.p2.1),[§1](https://arxiv.org/html/2606.18388#S1.p5.1),[§4\.1](https://arxiv.org/html/2606.18388#S4.SS1.SSS0.Px1.p1.1)\.
- Y\. Chen, Z\. Yang, Z\. Liu, C\. Lee, P\. Xu, M\. Shoeybi, B\. Catanzaro, and W\. Ping \(2025a\)AceReason\-nemotron: advancing math and code reasoning through reinforcement learning\.External Links:2505\.16400,[Link](https://arxiv.org/abs/2505.16400)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2606.18388#S1.p1.1),[§4\.6](https://arxiv.org/html/2606.18388#S4.SS6.p1.1)\.
- Z\. Chen, Y\. Min, B\. Zhang, J\. Chen, J\. Jiang, D\. Cheng, W\. X\. Zhao, Z\. Liu, X\. Miao, Y\. Lu, L\. Fang, Z\. Wang, and J\. Wen \(2025b\)An empirical study on eliciting and improving r1\-like reasoning models\.External Links:2503\.04548,[Link](https://arxiv.org/abs/2503.04548)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- Y\. Chi, Y\. Lin, S\. Hong, D\. Pan, Y\. Fei, G\. Mei, B\. Liu, T\. Pang, J\. Kwok, C\. Zhang, B\. Liu, and C\. Wu \(2024\)SELA: tree\-search enhanced llm agents for automated machine learning\.External Links:2410\.17238,[Link](https://arxiv.org/abs/2410.17238)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px3.p1.1)\.
- Y\. Du, X\. Yang, Z\. Zhou, W\. Liu, Z\. Lei, Z\. Chen, F\. Liu, H\. Wu, Y\. Cai, Z\. Liu,et al\.\(2026\)DataMaster: towards autonomous data engineering for machine learning\.arXiv preprint arXiv:2605\.10906\.Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px3.p3.1)\.
- H\. Fang, B\. Han, N\. Erickson, X\. Zhang, S\. Zhou, A\. Dagar, J\. Zhang, A\. C\. Turkmen, C\. Hu, H\. Rangwala, Y\. N\. Wu, B\. Wang, and G\. Karypis \(2025\)MLZero: a multi\-agent system for end\-to\-end machine learning automation\.External Links:2505\.13941,[Link](https://arxiv.org/abs/2505.13941)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px3.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Ding, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Chen, J\. Yuan, J\. Tu, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. You, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. Zhang \(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4)\.
- S\. Hao, Y\. Gu, H\. Ma, J\. J\. Hong, Z\. Wang, D\. Z\. Wang, and Z\. Hu \(2023\)Reasoning with language model is planning with world model\.External Links:2305\.14992,[Link](https://arxiv.org/abs/2305.14992)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px3.p2.1)\.
- Y\. Hao, F\. Chao, Y\. Hao, Z\. Cui, H\. Bai, H\. Zhang, Y\. Liu, C\. Deng, and J\. Feng \(2025\)JT\-math: a multi\-stage framework for advanced mathematical reasoning in large language models\.External Links:2507\.19748,[Link](https://arxiv.org/abs/2507.19748)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- J\. He, J\. Liu, C\. Y\. Liu, R\. Yan, C\. Wang, P\. Cheng, X\. Zhang, F\. Zhang, J\. Xu, W\. Shen, S\. Li, L\. Zeng, T\. Wei, C\. Cheng, B\. An, Y\. Liu, and Y\. Zhou \(2025\)Skywork open reasoner 1 technical report\.External Links:2505\.22312,[Link](https://arxiv.org/abs/2505.22312)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- M\. Jaderberg, V\. Dalibard, S\. Osindero, W\. M\. Czarnecki, J\. Donahue, A\. Razavi, O\. Vinyals, T\. Green, I\. Dunning, K\. Simonyan, C\. Fernando, and K\. Kavukcuoglu \(2017\)Population based training of neural networks\.External Links:1711\.09846,[Link](https://arxiv.org/abs/1711.09846)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px2.p1.1),[Limitations](https://arxiv.org/html/2606.18388#Sx1.p1.1)\.
- Y\. Ji, S\. Zhao, X\. Tian, H\. Wang, S\. Chen, Y\. Peng, H\. Zhao, and X\. Li \(2025\)How difficulty\-aware staged reinforcement learning enhances llms’ reasoning capabilities: a preliminary experimental study\.External Links:2504\.00829,[Link](https://arxiv.org/abs/2504.00829)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- Z\. Jiang, D\. Schmidt, D\. Srikanth, D\. Xu, I\. Kaplan, D\. Jacenko, and Y\. Wu \(2025\)AIDE: ai\-driven exploration in the space of code\.External Links:2502\.13138,[Link](https://arxiv.org/abs/2502.13138)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px3.p1.1)\.
- F\. Kang, H\. Li, A\. Nguyen, M\. Dabas, J\. W\. Ma, F\. Sala, D\. Song, and R\. Jia \(2026\)Can generalist agents automate data curation?\.arXiv preprint arXiv:2606\.04261\.Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px3.p3.1)\.
- H\. Lai and M\. Nissim \(2026\)TACLer: tailored curriculum reinforcement learning for efficient reasoning\.External Links:2601\.21711,[Link](https://arxiv.org/abs/2601.21711)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- H\. Li, H\. Cao, B\. Feng, Y\. Shao, X\. Tang, Z\. Yan, L\. Yuan, Y\. Tian, and Y\. Li \(2026\)Beyond chemical qa: evaluating llm’s chemical reasoning with modular chemical operations\.External Links:2505\.21318,[Link](https://arxiv.org/abs/2505.21318)Cited by:[§1](https://arxiv.org/html/2606.18388#S1.p2.1),[§1](https://arxiv.org/html/2606.18388#S1.p5.1),[§4\.1](https://arxiv.org/html/2606.18388#S4.SS1.SSS0.Px1.p1.1)\.
- L\. Li, K\. Jamieson, G\. DeSalvo, A\. Rostamizadeh, and A\. Talwalkar \(2018\)Hyperband: a novel bandit\-based approach to hyperparameter optimization\.External Links:1603\.06560,[Link](https://arxiv.org/abs/1603.06560)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px2.p1.1)\.
- T\. Liu, D\. Nathani, Z\. Li, K\. Yang, and W\. Y\. Wang \(2026\)WildSci: advancing scientific reasoning from in\-the\-wild literature\.External Links:2601\.05567,[Link](https://arxiv.org/abs/2601.05567)Cited by:[§1](https://arxiv.org/html/2606.18388#S1.p5.1),[§4\.1](https://arxiv.org/html/2606.18388#S4.SS1.SSS0.Px1.p1.1)\.
- M\. Luo, S\. Tan, R\. Huang, A\. Patel, A\. Ariyak, Q\. Wu, X\. Shi, R\. Xin, C\. Cai, M\. Weber, C\. Zhang, L\. E\. Li, R\. A\. Popa, and I\. Stoica \(2025a\)DeepCoder: a fully open\-source 14b coder at o3\-mini level\.Note:Notion BlogCited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- M\. Luo, S\. Tan, J\. Wong, X\. Shi, W\. Y\. Tang, M\. Roongta, C\. Cai, J\. Luo, L\. E\. Li, R\. A\. Popa, and I\. Stoica \(2025b\)DeepScaleR: surpassing o1\-preview with a 1\.5b model by scaling rl\.Note:Notion BlogCited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2606.18388#S1.p1.1),[§4\.6](https://arxiv.org/html/2606.18388#S4.SS6.p1.1)\.
- Y\. Luo, F\. Wang, Q\. Cheng, F\. Yu, H\. Lei, J\. Yan, C\. Li, J\. Chen, Y\. Zhao, H\. Wan, Y\. Zhang, S\. Zheng, J\. Yao, Q\. Zhang, H\. He, W\. Zeng, L\. Sheng, C\. Xie, Y\. Zuo, Y\. Li, Y\. Wu, R\. Huang, D\. Zhou, K\. Chen, Y\. Qiao, L\. Bai, Y\. Cheng, N\. Ding, B\. Zhou, P\. Ye, and G\. Cui \(2026\)P1\-vl: bridging visual perception and scientific reasoning in physics olympiads\.External Links:2602\.09443,[Link](https://arxiv.org/abs/2602.09443)Cited by:[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- X\. Lv, Y\. Zuo, Y\. Sun, H\. Liu, Y\. Wei, Z\. Chen, X\. Zhu, K\. Zhang, B\. Wang, N\. Ding,et al\.\(2025\)Towards a unified view of large language model post\-training\.arXiv preprint arXiv:2509\.04419\.Cited by:[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. R\. Ruiz, A\. Mehrabian, M\. P\. Kumar, A\. See, S\. Chaudhuri, G\. Holland, A\. Davies, S\. Nowozin, P\. Kohli, and M\. Balog \(2025\)AlphaEvolve: a coding agent for scientific and algorithmic discovery\.External Links:2506\.13131,[Link](https://arxiv.org/abs/2506.13131)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px3.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2024\)Direct preference optimization: your language model is secretly a reward model\.External Links:2305\.18290,[Link](https://arxiv.org/abs/2305.18290)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px4.p1.1)\.
- B\. Rank, H\. Bhatnagar, A\. Prabhu, S\. Eisenberg, K\. Nguyen, M\. Bethge, and M\. Andriushchenko \(2026\)PostTrainBench: can llm agents automate llm post\-training?\.arXiv preprint arXiv:2603\.08640\.Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px3.p3.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2606.18388#S1.p3.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px4.p1.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2024\)HybridFlow: a flexible and efficient rlhf framework\.arXiv preprint arXiv: 2409\.19256\.Cited by:[§4\.1](https://arxiv.org/html/2606.18388#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Snoek, H\. Larochelle, and R\. P\. Adams \(2012\)Practical bayesian optimization of machine learning algorithms\.External Links:1206\.2944,[Link](https://arxiv.org/abs/1206.2944)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px2.p1.1)\.
- M\. Song, M\. Zheng, Z\. Li, W\. Yang, X\. Luo, Y\. Pan, and F\. Zhang \(2025\)FastCuRL: curriculum reinforcement learning with stage\-wise context scaling for efficient training r1\-like reasoning models\.External Links:2503\.17287,[Link](https://arxiv.org/abs/2503.17287)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- F\. Wan, W\. Shen, S\. Liao, Y\. Shi, C\. Li, Z\. Yang, J\. Zhang, F\. Huang, J\. Zhou, and M\. Yan \(2025\)QwenLong\-l1: towards long\-context large reasoning models with reinforcement learning\.External Links:2505\.17667,[Link](https://arxiv.org/abs/2505.17667)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- Z\. Wang, G\. Cui, Y\. Li, K\. Wan, and W\. Zhao \(2025a\)Dump: automated distribution\-level curriculum learning for rl\-based llm post\-training\.arXiv preprint arXiv:2504\.09710\.Cited by:[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- Z\. Wang, Z\. Yang, Y\. Luo, Y\. Li, X\. Qu, Z\. Qiao, H\. Zhang, R\. Zhan, D\. F\. Wong, J\. Zhou, and Y\. Cheng \(2025b\)Towards an ai musician: synthesizing sheet music problems for musical reasoning\.External Links:2509\.04059,[Link](https://arxiv.org/abs/2509.04059)Cited by:[§1](https://arxiv.org/html/2606.18388#S1.p2.1),[§1](https://arxiv.org/html/2606.18388#S1.p5.1),[§4\.1](https://arxiv.org/html/2606.18388#S4.SS1.SSS0.Px1.p1.1)\.
- H\. Wen, Y\. Bai, J\. Li, and J\. Tang \(2025\)SIRI: scaling iterative reinforcement learning with interleaved compression\.External Links:2509\.25176,[Link](https://arxiv.org/abs/2509.25176)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4)\.
- L\. Xiaomi, :, B\. Xia, B\. Shen, Cici, D\. Zhu, D\. Zhang, G\. Wang, H\. Zhang, H\. Liu, J\. Xiao, J\. Dong, L\. Zhao, P\. Li, P\. Wang, S\. Yu, S\. Chen, W\. Wang, W\. Ma, X\. Deng, Y\. Huang, Y\. Song, Z\. Jiang, B\. Ye, C\. Cai, C\. He, D\. Zhang, D\. Zhang, G\. Wang, H\. Tian, H\. Zhao, H\. Qu, H\. Xu, J\. Shi, K\. Bao, K\. Fang, K\. Zhou, K\. Zhou, L\. Li, M\. Zhu, N\. Chen, Q\. Wang, S\. Liu, S\. Li, S\. Gu, S\. Ren, S\. Liu, S\. Deng, W\. Zhuang, W\. Lv, W\. Yang, X\. Zhang, X\. Yong, X\. Zhang, X\. Song, X\. Xu, X\. Wang, Y\. Yan, Y\. Tu, Y\. Tian, Y\. Wang, Y\. Yu, Z\. Lin, Z\. Song, and Z\. Yue \(2025\)MiMo: unlocking the reasoning potential of language model – from pretraining to posttraining\.External Links:2505\.07608,[Link](https://arxiv.org/abs/2505.07608)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2606.18388#S1.p1.1)\.
- S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. Narasimhan \(2023\)Tree of thoughts: deliberate problem solving with large language models\.External Links:2305\.10601,[Link](https://arxiv.org/abs/2305.10601)Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px3.p2.1)\.
- C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang,et al\.\(2025\)Group sequence policy optimization\.arXiv preprint arXiv:2507\.18071\.Cited by:[Appendix A](https://arxiv.org/html/2606.18388#A1.SS0.SSS0.Px4.p1.1)\.

###### Contents

1. [1Introduction](https://arxiv.org/html/2606.18388#S1)
2. [2Preliminary](https://arxiv.org/html/2606.18388#S2)1. [2\.1Training Strategy Formalization](https://arxiv.org/html/2606.18388#S2.SS1)
3. [3LLMZero](https://arxiv.org/html/2606.18388#S3)1. [3\.1Problem Formulation](https://arxiv.org/html/2606.18388#S3.SS1) 2. [3\.2Tree Search and Subtree Pruning](https://arxiv.org/html/2606.18388#S3.SS2) 3. [3\.3Checkpoint\-Based Strategy Composition](https://arxiv.org/html/2606.18388#S3.SS3) 4. [3\.4Dynamics\-Aware Transition Proposal](https://arxiv.org/html/2606.18388#S3.SS4) 5. [3\.5Agentic Early Stopping](https://arxiv.org/html/2606.18388#S3.SS5) 6. [3\.6Automated Pipeline](https://arxiv.org/html/2606.18388#S3.SS6)
4. [4Experiments](https://arxiv.org/html/2606.18388#S4)1. [4\.1Setup](https://arxiv.org/html/2606.18388#S4.SS1) 2. [4\.2Main Results \(RQ1\)](https://arxiv.org/html/2606.18388#S4.SS2) 3. [4\.3Analysis of Discovered Strategies \(RQ2\)](https://arxiv.org/html/2606.18388#S4.SS3)1. [4\.3\.1Per\-Dataset Strategies](https://arxiv.org/html/2606.18388#S4.SS3.SSS1) 2. [4\.3\.2Cross\-Task Structural Patterns](https://arxiv.org/html/2606.18388#S4.SS3.SSS2) 4. [4\.4Scaling Analysis \(RQ3\)](https://arxiv.org/html/2606.18388#S4.SS4) 5. [4\.5Ablation Studies \(RQ4\)](https://arxiv.org/html/2606.18388#S4.SS5) 6. [4\.6Strategy Transfer \(RQ5\)](https://arxiv.org/html/2606.18388#S4.SS6)
5. [5Conclusion](https://arxiv.org/html/2606.18388#S5)
6. [References](https://arxiv.org/html/2606.18388#bib)
7. [ARelated Work](https://arxiv.org/html/2606.18388#A1)
8. [BExperimental Setup Details](https://arxiv.org/html/2606.18388#A2)1. [B\.1Task Details](https://arxiv.org/html/2606.18388#A2.SS1) 2. [B\.2Baseline Configurations](https://arxiv.org/html/2606.18388#A2.SS2) 3. [B\.3Compute Budget Fairness](https://arxiv.org/html/2606.18388#A2.SS3) 4. [B\.4API Cost](https://arxiv.org/html/2606.18388#A2.SS4) 5. [B\.5Model and Infrastructure](https://arxiv.org/html/2606.18388#A2.SS5) 6. [B\.6LLMZeroConfiguration](https://arxiv.org/html/2606.18388#A2.SS6) 7. [B\.7Skill\-Based LLM Agent](https://arxiv.org/html/2606.18388#A2.SS7)
9. [CAdditional Results and Analysis](https://arxiv.org/html/2606.18388#A3)1. [C\.1Search Convergence](https://arxiv.org/html/2606.18388#A3.SS1) 2. [C\.2ChemCoT Per\-Subtask Breakdown](https://arxiv.org/html/2606.18388#A3.SS2) 3. [C\.3Model Scaling Detailed Results](https://arxiv.org/html/2606.18388#A3.SS3) 4. [C\.4Best Discovered Strategy Configurations](https://arxiv.org/html/2606.18388#A3.SS4) 5. [C\.5Capacity Guidebook Analysis](https://arxiv.org/html/2606.18388#A3.SS5)
10. [DSearch Algorithm Details](https://arxiv.org/html/2606.18388#A4)1. [D\.1Search Loop Pseudocode](https://arxiv.org/html/2606.18388#A4.SS1) 2. [D\.2UCT Computation](https://arxiv.org/html/2606.18388#A4.SS2) 3. [D\.3Virtual New Child Competition](https://arxiv.org/html/2606.18388#A4.SS3)
11. [EDetailed Per\-Run Results](https://arxiv.org/html/2606.18388#A5)1. [E\.1HPO Baseline Configurations](https://arxiv.org/html/2606.18388#A5.SS1) 2. [E\.2Per\-Run Results: Random Search](https://arxiv.org/html/2606.18388#A5.SS2) 3. [E\.3Per\-Run Results: Grid Search](https://arxiv.org/html/2606.18388#A5.SS3) 4. [E\.4Per\-Iteration Results: Skill\-Based LLM Agent](https://arxiv.org/html/2606.18388#A5.SS4) 5. [E\.5Per\-Node Results:LLMZero](https://arxiv.org/html/2606.18388#A5.SS5)
12. [FHuman Knowledge Injection](https://arxiv.org/html/2606.18388#A6)1. [F\.1Metric Descriptions](https://arxiv.org/html/2606.18388#A6.SS1) 2. [F\.2Hyperparameter Descriptions](https://arxiv.org/html/2606.18388#A6.SS2)
13. [GAgent Prompts](https://arxiv.org/html/2606.18388#A7)1. [G\.1Proposer Agent Prompt](https://arxiv.org/html/2606.18388#A7.SS1) 2. [G\.2Early Stopper Agent Prompt](https://arxiv.org/html/2606.18388#A7.SS2)
14. [HEthics and Artifact Documentation](https://arxiv.org/html/2606.18388#A8)
15. [IUse of AI](https://arxiv.org/html/2606.18388#A9)

## Appendix ARelated Work

##### Multi\-stage RL post\-training\.

Progressive response length extension is the dominant pattern: DeepScaleR\(Luoet al\.,[2025b](https://arxiv.org/html/2606.18388#bib.bib28)\)introduced 3\-stage length scaling \(8K→\\to16K→\\to24K\), replicated by AceReason\-Nemotron\(Chenet al\.,[2025a](https://arxiv.org/html/2606.18388#bib.bib29)\)\(4 stages to 32K\), DeepCoder\(Luoet al\.,[2025a](https://arxiv.org/html/2606.18388#bib.bib34)\)\(code, 16K→\\to32K\), MiMo\(Xiaomiet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib33)\)\(32K→\\to48K\), and others\(Heet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib31); Haoet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib32); Chenet al\.,[2025b](https://arxiv.org/html/2606.18388#bib.bib35); Wanet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib38); Guoet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib4)\)\. Data difficulty staging\(Chenet al\.,[2025a](https://arxiv.org/html/2606.18388#bib.bib29); Lai and Nissim,[2026](https://arxiv.org/html/2606.18388#bib.bib36); Songet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib30); Jiet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib40)\)and compression\-extension cycles\(Songet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib30); Wenet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib37)\)provide complementary patterns, while most of them hold learning rate, KL coefficient, temperature, and batch size constant across stages, leaving most of the hyperparameter space unexplored at transitions\.

##### Adaptive training and HPO\.

Population\-based training \(PBT\)\(Jaderberget al\.,[2017](https://arxiv.org/html/2606.18388#bib.bib10)\)is the closest methodological ancestor: it also discovers hyperparameter schedules during training by evolving a population of configurations\. The key differences are: \(1\) PBT transitions via scalar fitness ranking and random perturbation, whileLLMZerodiagnoses specific pathologies and proposes coordinated interventions; \(2\) PBT requires a population \(typically 10–80 parallel workers\), whileLLMZerooperates sequentially with checkpoint reuse under severe budget constraints; \(3\) PBT’s output is an opaque schedule, whileLLMZero’s diagnostic analysis produces transferable design principles \(validated by our cross\-task transfer results\)\. PBT excels in GPU\-rich regimes where broad parallel exploration is cheap; even a minimal population of 8 workers would require 256 concurrent A100 GPUs \(8×\\times32 GPUs per training run\), placing it beyond our compute budget\. Random search\(Bergstra and Bengio,[2012](https://arxiv.org/html/2606.18388#bib.bib5)\), Bayesian optimization\(Snoeket al\.,[2012](https://arxiv.org/html/2606.18388#bib.bib6)\), and Hyperband\(Liet al\.,[2018](https://arxiv.org/html/2606.18388#bib.bib8)\)search over static configurations without adapting within a training run\.LLMZerosearches over configuration*sequences*conditioned on training dynamics, staging parameters the literature holds constant with non\-monotonic trajectories\.

##### Agentic ML automation\.

Several concurrent systems apply LLM agents with tree\-structured exploration to ML automation\. SELA\(Chiet al\.,[2024](https://arxiv.org/html/2606.18388#bib.bib13)\)uses MCTS for ML pipeline configuration; AIDE\(Jianget al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib12)\)applies tree search to data science competitions; MLZero\(Fanget al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib26)\)provides end\-to\-end automation across modalities; and AlphaEvolve\(Novikovet al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib17)\)applies evolutionary search to code\.LLMZerotargets a fundamentally different search space: RL post\-training trajectories where \(1\) each evaluation costs much more GPU time, imposing severe budget constraints; \(2\) nodes are not independent solutions but training*phases*that compose via checkpoint resumption into multi\-stage strategies; \(3\) the search must reason about non\-stationary training dynamics \(KL divergence spikes, model collapse, reward stagnation, etc\.\) rather than static metrics; and \(4\) the proposer must make coordinated multi\-dimensional hyperparameter changes conditioned on the observed training state\. These challenges motivate our dynamics\-aware proposal mechanism and agentic early stopping\.

Our contributions to the search process are: redefining nodes as training phases with checkpoint composition \(§[3\.3](https://arxiv.org/html/2606.18388#S3.SS3)\), dynamics\-aware multimodal proposals \(§[3\.4](https://arxiv.org/html/2606.18388#S3.SS4)\), agentic early stopping \(§[3\.5](https://arxiv.org/html/2606.18388#S3.SS5)\), and forced from\-scratch injection \(§[3\.3](https://arxiv.org/html/2606.18388#S3.SS3)\)\. The UCT value function with virtual child competition is adopted from prior work\. Tree\-of\-Thought\(Yaoet al\.,[2023](https://arxiv.org/html/2606.18388#bib.bib18)\)and RAP\(Haoet al\.,[2023](https://arxiv.org/html/2606.18388#bib.bib19)\)apply tree search at inference time where simulations are cheap;LLMZeroapplies it at training time where each node requires hours of GPU compute\.

While our approach focuses on optimizing RL training trajectories, a concurrent line of work shifts the focus of agentic ML automation toward autonomous data collection and curation\. For instance, PostTrainBench\(Ranket al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib45)\)evaluates agents on their ability to autonomously gather and curate external data to optimize the post\-training phase of base LLMs\. Similarly, the DataMaster\(Duet al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib46)\)framework isolates the data engineering process, using tree\-structured search and cumulative memory to let agents discover, clean, and compose datasets without altering the underlying learning algorithm\. Exploring a related automated curation loop, Curation\-Bench\(Kanget al\.,[2026](https://arxiv.org/html/2606.18388#bib.bib47)\)demonstrates that properly scaffolded agents can autonomously compose highly efficient data\-selection policies that outperform standard baselines\. These data\-centric approaches complement our dynamics\-aware search by targeting dataset optimization rather than training trajectory search\.

##### LLM post\-training methods\.

Current LLM pipelines utilize a variety of optimization algorithms for RL training, including PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.18388#bib.bib49)\), DPO\(Rafailovet al\.,[2024](https://arxiv.org/html/2606.18388#bib.bib2)\), GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.18388#bib.bib3)\), GSPO\(Zhenget al\.,[2025](https://arxiv.org/html/2606.18388#bib.bib48)\), etc\.LLMZerodoes not introduce a new alignment objective; instead, it automates the*training strategy*required to effectively deploy them\. We demonstrate the efficacy of our approach specifically using GRPO, leaving its application to other algorithms for future work\.

## Appendix BExperimental Setup Details

### B\.1Task Details

Table[4](https://arxiv.org/html/2606.18388#A2.T4)summarizes the four evaluation tasks\. All datasets are uniformly subsampled to 5,000 train, 500 validation, and 500 test examples\. The dynamic orchestration workflow automatically designs a task\-specific reward function for each dataset\.

Table 4:Evaluation tasks\. All tasks train with GRPO via VeRL\. For ChemCoT we report the average over subtasks in this table\.All datasets use fixed, deterministic splits with subsampled train \(≤\\leq5,000\), validation \(≤\\leq500\), and test \(≤\\leq500\) sets\. Base model performance \(0\.6B and 4B columns\) is measured via greedy decoding \(pass@1, temperature=0\) with extended thinking and max response length=8192 on 2 nodes×\\times8 A100 40G GPUs\.

### B\.2Baseline Configurations

Practitioner baseline\.A carefully tuned general\-purpose GRPO recipe for Qwen3 \(on 8x A100 40G GPUs\), developed through internal iteration on separate tasks \(e\.g\., math reasoning\) and applied without task\-specific modification\. Single static configuration \(L=1L=1\)\.

Random search\.8 trials with configurations sampled from: learning rate∼\\simLogUniform\(1e\-5, 1e\-4\), KL coefficient∼\\simLogUniform\(1e\-5, 1e\-3\), temperature∼\\simUniform\(0\.6, 1\.2\), clip ratio∼\\simUniform\(0\.15, 0\.35\), batch size∈\{\\in\\\{64, 128, 256\}\\\}, LoRA rank∈\{\\in\\\{16, 32, 64, 128\}\\\}, rollout count∈\{\\in\\\{6, 8, 12\}\\\}, epochs∈\{\\in\\\{3, 5\}\\\}\. Each trial runs to completion without early stopping\.

Skill\-based LLM agent\.An autonomous LLM agent \(Claude Opus 4\.6\) built on the same training infrastructure toLLMZero\(VeRL, reward functions, hyperparameter space\)\. UnlikeLLMZero’s fixed workflow where each stage has a deterministic prompt template, the iterative agent uses a skill\-based general workflow: the LLM autonomously plans and orchestrates each iteration, invoking skills for dataset preparation, reward design, job submission, metric diagnosis, and checkpoint resume\. This generality makes it more flexible but less token\-efficient \(44–144×\\timeshigher API cost\) since the LLM must maintain full context across all decisions\. It can stop training at any time and resume from any previous checkpoint, without explicit tree search or visual reasoning\.

### B\.3Compute Budget Fairness

Random and grid search each run 8 full\-length training runs to completion and achieve similar total GPU\-hours toLLMZero\. BothLLMZeroand the skill\-based LLM agent have a maximum of 16 iterations, but the skill\-based agent is limited to less than 700 GPU\-hours of total runtime due to its significant API cost \(Table[5](https://arxiv.org/html/2606.18388#A2.T5)\)\. This budget is sufficient for convergence: the agent does not force restarts from scratch, and its validation scores begin to plateau or decrease near the end of its time limit \(Table[17](https://arxiv.org/html/2606.18388#A5.T17), Figure[2](https://arxiv.org/html/2606.18388#S4.F2)\)\. Notably,LLMZero’s best discovered strategy on all four tasks originates from the root node \(the practitioner default configuration\), indicating that the default is a strong starting point and that forced from\-scratch injection, while necessary for robustness, was not the source of the best strategies in these experiments\.

### B\.4API Cost

Table[5](https://arxiv.org/html/2606.18388#A2.T5)compares API costs betweenLLMZeroand the skill\-based LLM agent\. The fixed workflow with structured prompts at each stage consumes 73×\\timesless total API cost than the agent’s autonomous orchestration, which must maintain full conversation context across all decisions\.

Table 5:LLM API cost comparison \(ratios from unrounded costs\)\.LLMZero’s structured pipeline uses 44–144×\\timesless API cost than the skill\-based LLM agent\.
### B\.5Model and Infrastructure

All experiments use Qwen3\-4B as the base model with LoRA\. Training uses a modified version of VeRL \(for better LoRA support, etc\.\) on Ray clusters deployed on AWS EKS with 4 to 8×\\timesA100 40G nodes\. The scaling experiment \(Table[7](https://arxiv.org/html/2606.18388#A3.T7)\) additionally evaluates Qwen3\-0\.6B, 1\.7B, and 8B\.

### B\.6LLMZeroConfiguration

C=1\.414C=1\.414,T=0\.3T=0\.3,wf=0\.5w\_\{f\}=0\.5,we=0\.3w\_\{e\}=0\.3,of=2o\_\{f\}=2, max evolve children = 2, max debug children = 3, max debug depth = 5,ρmin=0\.2\\rho\_\{\\min\}=0\.2, early stopping interval = 900s\. LLM backbone: Claude 4\.6 Sonnet for all agents\.

### B\.7Skill\-Based LLM Agent

The skill\-based LLM agent uses Claude Opus 4\.6 as an autonomous orchestrator with unrestricted tool access\. Rather than a fixed pipeline, the agent is equipped with 8*skills*, which are natural language instruction sets that the LLM invokes as needed:

1. 1\.prepare\-dataset: Download and convert data to VeRL parquet format
2. 2\.define\-reward: Analyze dataset and write acompute\_score\(\)reward function
3. 3\.validate\-run: Pre\-flight evaluation on a sample
4. 4\.generate\-config: Write a full VeRL sweep configuration
5. 5\.download\-model: Download the base model
6. 6\.submit\-training: Generate and submit SLURM jobs
7. 7\.check\-training: Monitor metrics, diagnose training health, recommend action
8. 8\.gather\-results: Parse outputs and write a report

The agent’s optimization loop proceeds as: \(1\) submit a training job, \(2\) poll metrics at 5–30 minute intervals, \(3\) perform a 14\-parameter diagnostic analysis producing a per\-parameter verdict \(KEEP/INCREASE/DECREASE\) based on observed KL divergence, gradient norms, clip ratios, and validation trends, \(4\) decide whether to continue, early\-stop, tune\-and\-resume from the best checkpoint, or restart from scratch\.

The agent can also be coupled withLLMZeroto form an end\-to\-end training experience with static workflow as search backend\.

##### Key differences fromLLMZero\.

The agent maintains full conversation context across all decisions \(leading to long token histories\), plans its own workflow \(may skip or reorder stages\), and tends to always resume from the single best checkpoint in a linear chain\. It analyzes only numerical metrics \(no visual training curves\)\. These design choices make it more flexible thanLLMZero’s fixed harness but 44–144×\\timesless token\-efficient\.

##### Budget limitation\.

Due to high API cost \($635–1,146 per task\), the agent was limited to 6–9 training iterations \(488–709 GPU\-hours\) rather than the full 16\-iteration budget allocated toLLMZero\. This limitation is inherent to the agent’s design: autonomous orchestration requires maintaining full conversation context, making each iteration 44–144×\\timesmore expensive in API cost thanLLMZero’s fixed prompts\. A budget\-matched comparison at equal GPU\-hours would require either reducing the agent to 1–2 iterations or spending $10K\+ per task, neither of which yields a meaningful evaluation\. Despite this handicap, it achieves competitive results \(Table[1](https://arxiv.org/html/2606.18388#S4.T1)\), confirming that LLM reasoning about training dynamics is broadly effective\. The advantage ofLLMZerocomes from its tree structure and fixed harness rather than from a fundamentally different reasoning capability\.

## Appendix CAdditional Results and Analysis

### C\.1Search Convergence

Figure[5](https://arxiv.org/html/2606.18388#A3.F5)shows best\-so\-far validation score vs\. search iteration for all 4 tasks\.LLMZerosurpasses the practitioner baseline early in the search and continues improving throughout the budget\.

![Refer to caption](https://arxiv.org/html/2606.18388v1/x6.png)
![Refer to caption](https://arxiv.org/html/2606.18388v1/x7.png)
![Refer to caption](https://arxiv.org/html/2606.18388v1/x8.png)
![Refer to caption](https://arxiv.org/html/2606.18388v1/x9.png)

Figure 5:Best\-so\-far validation score vs\. search iteration\.LLMZerosurpasses the practitioner baseline early in the search and continues improving throughout the budget\.
### C\.2ChemCoT Per\-Subtask Breakdown

Table[6](https://arxiv.org/html/2606.18388#A3.T6)reports per\-subtask test accuracy for ChemCoT\.

Table 6:ChemCoT per\-subtask test accuracy \(%\) for Qwen3\-4B \(pass@1, rl=8192\), the first training run \(Node 0\), and the best discovered strategy \(Node 11, 5\-phase adaptive\)\. Subtasks sorted by Node 11 accuracy\. Nine subtasks scoring 0% across all conditions are omitted \(allmol\_opt\_\*,rxn\_retro,rxn\_nepp,rxn\_mechanism\)\.Note that the average number here refers to average over subtasks which is different from the average over domains in main table\.
### C\.3Model Scaling Detailed Results

Table[7](https://arxiv.org/html/2606.18388#A3.T7)reports per\-subtask results for the scaling analysis on SSMR\-Bench\.

Table 7:Model scaling analysis on SSMR\-Bench: per\-subtask test accuracy \(%\) at the best validation node\. Subscripts show gain over the base model\.Bold: best per subtask per model\.
### C\.4Best Discovered Strategy Configurations

Tables[8](https://arxiv.org/html/2606.18388#A3.T8)–[11](https://arxiv.org/html/2606.18388#A3.T11)detail the full hyperparameter configuration at each phase of the best discovered strategy for each task\. For each phase, we report the node ID in the search tree, the training step range, the proposer’s diagnosis that triggered the transition, and the complete configuration\.

Table 8:Best discovered strategy forChemCoT\(5\-phase, Node path: 0→\\to2→\\to3→\\to5→\\to11\)\. Node 2 resumes from Node 1’s checkpoint \(Node 1 failed but produced a valid checkpoint\)\. Final val=35\.6%, test=28\.5% \(macro\-avg 19 subtasks\)\.Table 9:Best discovered strategy forPaperSearchQA\(4\-phase, Node path: 0→\\to1→\\to2→\\to5\)\. Final val=42\.0%, test=42\.6%\.Table 10:Best discovered strategy forSSMR\-Bench\(4\-phase, Node path: 0→\\to1→\\to4→\\to5\)\. Final val=87\.0%, test=82\.2%\.Table 11:Best discovered strategy forWildSci\(4\-phase, Node path: 0→\\to5→\\to13→\\to15\)\. Final val=64\.0%, test=58\.5%\.
### C\.5Capacity Guidebook Analysis

Comparing the Capacity Guidebook to the full SSMR transfer \(Table[3](https://arxiv.org/html/2606.18388#S4.T3)\) isolates the contribution of regularization oscillation versus pure capacity expansion\.

##### WildSci\.

Both schedules converge to near\-identical performance \(58\.3% vs\. 58\.4% vs\.LLMZero’s 58\.5%\), confirming that expanded reasoning capacity, rather than hyperparameter oscillation, drives improvement on this task\.

##### ChemCoT\.

The SSMR transfer \(41\.3%\) substantially outperforms the Capacity Guidebook \(36\.6%\), indicating that LR/KL oscillation provides \+4\.7 pp beyond pure capacity scaling, consistent with the KL divergence spike diagnosis in §[4\.3\.1](https://arxiv.org/html/2606.18388#S4.SS3.SSS1)\.

##### PaperSearchQA\.

Both fixed schedules underperform the static practitioner \(37\.8% and 36\.2% vs\. 39\.0%\), confirming that this task’s progressive\-tightening dynamic cannot be captured by any fixed multi\-stage recipe\.

These results validate the structural principle: capacity parameters should accumulate monotonically, but the task\-specific LR and KL trajectories thatLLMZerodiscovers are essential for realizing the full benefit of multi\-stage training\.

##### Transfer caveats\.

Both the SSMR transfer and the Capacity Guidebook benefit from all tasks sharing the same subsampled dataset size \(5,000 training examples\), which produces similar steps\-per\-epoch and possibly comparable training dynamics timelines\. Transferring fixed schedules between datasets of substantially different sizes may require recalibrating transition points, as the step at which the model saturates its current response budget depends on dataset complexity and size\. This is an additional advantage of adaptive methods: they discover appropriate transition points from observed dynamics regardless of dataset scale\.

## Appendix DSearch Algorithm Details

While the UCT computation and virtual child competition mechanisms are adopted from prior work \(reproduced here for completeness\), we introduce the failure subtree pruning \(§[3\.2](https://arxiv.org/html/2606.18388#S3.SS2)\) to improve search efficiency and forced from\-scratch injection mechanism \(§[3\.3](https://arxiv.org/html/2606.18388#S3.SS3)\) to maintain exploration diversity\. As search optimization is not the primary focus of this paper, we leave the design of more sophisticated search algorithms, particularly those tailored for budget constraints or asynchronous execution, to future work\.

### D\.1Search Loop Pseudocode

Algorithm[1](https://arxiv.org/html/2606.18388#alg1)summarizes the main loop\. At each iteration, UCT selection traverses the tree to choose a parent node for expansion\. If the selected parent failed, an error analyzer diagnoses the failure and proposes a fix; otherwise, the proposer agent analyzes the parent’s training dynamics and proposes a new configuration with a checkpoint to resume from\. The resulting child node is executed as a training job, monitored by the agentic early stopper at fixed intervals\. Upon completion \(or early termination\), the best validation score is backpropagated up the tree and terminal subtrees are pruned\. The final output is the scratch\-to\-leaf path with the highest validation score, interpreted as a multi\-stage adaptive strategy\.

Algorithm 1LLMZeroadaptive strategy discovery loop0:Dataset

𝒟\\mathcal\{D\}, base model

M0M\_\{0\}, budget

BB
1:Initialize root node

n0n\_\{0\}with default configuration

θ0\\theta\_\{0\}
2:Run initialization agents \(data perception, task descriptor, tool selector\)

3:for

t=1t=1to

BBdo

4:

nparent←n\_\{\\text\{parent\}\}\\leftarrowSelectNode\(tree\) \{UCT selection \(Appendix[D\.2](https://arxiv.org/html/2606.18388#A4.SS2)\)\}

5:if

nparentn\_\{\\text\{parent\}\}failedthen

6:

\(θnew,fix\)←\(\\theta\_\{\\text\{new\}\},\\text\{fix\}\)\\leftarrowErrorAnalyzer\(

nparentn\_\{\\text\{parent\}\}\) \{Diagnose and propose fix\}

7:else

8:

\(θnew,k\)←\(\\theta\_\{\\text\{new\}\},k\)\\leftarrowProposer\(

nparentn\_\{\\text\{parent\}\}\.metrics,

nparentn\_\{\\text\{parent\}\}\.plots\) \{Multimodal analysis\}

9:endif

10:

nnew←n\_\{\\text\{new\}\}\\leftarrowCreateChild\(

nparentn\_\{\\text\{parent\}\},

θnew\\theta\_\{\\text\{new\}\}, checkpoint step

kk\)

11:Generate and submit training code via multi\-agent pipeline \(§[3\.6](https://arxiv.org/html/2606.18388#S3.SS6)\)

12:whiletraining in progressdo

13:Sample metrics and generate comparison plots at intervals

14:ifEarlyStopper\(current metrics, best strategy plots\) =STOPthen

15:Terminate training early

16:endif

17:endwhile

18:

s←s\\leftarrowbest validation score observed during run

19:Backpropagate\(

nnewn\_\{\\text\{new\}\},

ss\);PruneTerminal\(

nnewn\_\{\\text\{new\}\}\)

20:endfor

21:returnStrategy \(scratch\-to\-leaf path\) with highest validation score

### D\.2UCT Computation

We integrate expansion into selection via a*virtual new child*that competes against existing children at every internal node\. At nodeppwith existing children\{c1,…,ck\}\\\{c\_\{1\},\\ldots,c\_\{k\}\\\}:

UCT​\(ci\)\\displaystyle\\text\{UCT\}\(c\_\{i\}\)=Q​\(ci\)\+C​ln⁡N​\(p\)N​\(ci\),\\displaystyle=Q\(c\_\{i\}\)\+C\\sqrt\{\\frac\{\\ln N\(p\)\}\{N\(c\_\{i\}\)\}\},\(5\)UCT​\(new\)\\displaystyle\\text\{UCT\}\(\\text\{new\}\)=Qprior​\(p\)\+C​ln⁡N​\(p\)Nfair,\\displaystyle=Q\_\{\\text\{prior\}\}\(p\)\+C\\sqrt\{\\frac\{\\ln N\(p\)\}\{N\_\{\\text\{fair\}\}\}\},\(6\)whereQprior​\(p\)Q\_\{\\text\{prior\}\}\(p\)is the parent’s own normalized score \(encoding the prior belief that a new child will perform similarly to its parent\) andNfair=N​\(p\)/\(k\+1\)N\_\{\\text\{fair\}\}=N\(p\)/\(k\+1\)gives the virtual child a fair share of the parent’s visit budget\.

The exploitation termQ​\(c\)Q\(c\)uses min\-max normalization followed by exponential shaping for scale invariance:

s^=s−sminsmax−smin,fT​\(s^\)=es^/T−1e1/T−1,\\displaystyle\\hat\{s\}=\\frac\{s\-s\_\{\\min\}\}\{s\_\{\\max\}\-s\_\{\\min\}\},\\qquad f\_\{T\}\(\\hat\{s\}\)=\\frac\{e^\{\\hat\{s\}/T\}\-1\}\{e^\{1/T\}\-1\},\(7\)whereTTis a temperature parameter \(default 0\.3\)\. The full exploitation term combines three signals:

Q​\(c\)\\displaystyle Q\(c\)=vvalvtotal⋅fT​\(s¯c\)\\displaystyle=\\frac\{v\_\{\\text\{val\}\}\}\{v\_\{\\text\{total\}\}\}\\cdot f\_\{T\}\(\\bar\{s\}\_\{c\}\)−wf⋅max⁡\(0,vfail−of\)vtotal−we⋅vearlyvtotal,\\displaystyle\\quad\-w\_\{f\}\\cdot\\frac\{\\max\(0,\\,v\_\{\\text\{fail\}\}\-o\_\{f\}\)\}\{v\_\{\\text\{total\}\}\}\-w\_\{e\}\\cdot\\frac\{v\_\{\\text\{early\}\}\}\{v\_\{\\text\{total\}\}\},\(8\)wherevvalv\_\{\\text\{val\}\},vfailv\_\{\\text\{fail\}\},vearlyv\_\{\\text\{early\}\},vtotalv\_\{\\text\{total\}\}are validated, failed, early\-stopped, and total visit counts;s¯c\\bar\{s\}\_\{c\}is the average normalized score of validated descendants;wf=0\.5w\_\{f\}=0\.5andwe=0\.3w\_\{e\}=0\.3are penalty weights; andofo\_\{f\}\(default 2\) forgives the firstofo\_\{f\}failures before penalizing\.

### D\.3Virtual New Child Competition

At each internal nodeppduring selection traversal:

1. 1\.Compute UCT for all existing non\-terminal children\{c1,…,ck\}\\\{c\_\{1\},\\ldots,c\_\{k\}\\\}\(Eq\.[5](https://arxiv.org/html/2606.18388#A4.E5)\)\.
2. 2\.Compute UCT for the virtual new child \(Eq\.[6](https://arxiv.org/html/2606.18388#A4.E6)\) withQprior=fT​\(s^p\)Q\_\{\\text\{prior\}\}=f\_\{T\}\(\\hat\{s\}\_\{p\}\)andNfair=N​\(p\)/\(k\+1\)N\_\{\\text\{fair\}\}=N\(p\)/\(k\+1\)\.
3. 3\.IfUCT​\(new\)\>maxi⁡UCT​\(ci\)\\text\{UCT\}\(\\text\{new\}\)\>\\max\_\{i\}\\text\{UCT\}\(c\_\{i\}\)andk<kmaxk<k\_\{\\max\}: expand \(create new child atpp\)\.
4. 4\.Otherwise: descend intoarg⁡maxi⁡UCT​\(ci\)\\arg\\max\_\{i\}\\text\{UCT\}\(c\_\{i\}\)and repeat\.

This mechanism naturally adapts breadth vs\. depth: when children underperform their parent, the virtual child’s prior wins, triggering exploration of a new transition from the same checkpoint\.

## Appendix EDetailed Per\-Run Results

This section reports the full hyperparameter configuration and performance for every run across all methods and tasks\.

### E\.1HPO Baseline Configurations

##### Practitioner baseline\.

Table[12](https://arxiv.org/html/2606.18388#A5.T12)reports the practitioner configuration, a carefully tuned general\-purpose GRPO recipe applied identically to all tasks\.

Table 12:Practitioner baseline configuration \(applied to all tasks without modification\)\.
##### Capacity Guidebook\.

Table[13](https://arxiv.org/html/2606.18388#A5.T13)reports the 3\-phase capacity\-scaling schedule\. All hyperparameters not listed remain at practitioner defaults \(Table[12](https://arxiv.org/html/2606.18388#A5.T12)\)\. Phase durations are inspired by transition points observed inLLMZero’s discovered strategies\.

Table 13:Capacity Guidebook configuration\. Progressive capacity scaling with all other hyperparameters fixed to practitioner defaults \(Table[12](https://arxiv.org/html/2606.18388#A5.T12)\)\.
##### Grid search\.

All grid runs use the practitioner defaults except for two swept parameters: LR∈\{\\in\\\{1e\-5, 3e\-5, 5e\-5, 1e\-4\}\\\}×\\timesLoRA rank∈\{\\in\\\{64, 128\}\\\}\(8 configurations per task\)\.Grid runs additionally use epochs=5 and response length=8192\.

##### Random search\.

Each run samples all hyperparameters independently\. Table[14](https://arxiv.org/html/2606.18388#A5.T14)reports the sampled configurations\.

Table 14:Random search hyperparameter configurations for all tasks\. LR=learning rate, KL=KL coefficient, T=temperature, Cl=clip ratio, BS=batch size, Rk=LoRA rank, Ro=rollouts, Ep=epochs, RL=response length\.

### E\.2Per\-Run Results: Random Search

Table[15](https://arxiv.org/html/2606.18388#A5.T15)reports results for each random search trial\.

Table 15:Random search per\-run results\. GPU\-hrs on 32×\\timesA100\. Best Val = best validation score achieved\. Test reported at best validation step\.Bold: best per task\.
### E\.3Per\-Run Results: Grid Search

Table[16](https://arxiv.org/html/2606.18388#A5.T16)reports results for each grid search trial\.

Table 16:Grid search per\-run results\. Grid varies LR∈\{\\in\\\{1e\-5, 3e\-5, 5e\-5, 1e\-4\}\\\}×\\timesLoRA rank∈\{\\in\\\{64, 128\}\\\}, while SSMR\-Bench uses LR∈\{\\in\\\{1e\-5, 3e\-5, 5e\-5, 1e\-4\}\\\}×\\timesLoRA rank∈\{\\in\\\{16, 32, 64, 128\}\\\}to explore broader search space; all other HPs at default\.Bold: best per task\.
### E\.4Per\-Iteration Results: Skill\-Based LLM Agent

Table[17](https://arxiv.org/html/2606.18388#A5.T17)reports per\-iteration results for the skill\-based LLM agent\.

Table 17:Skill\-based LLM agent per\-iteration results\. GPU\-hrs reported per iteration\. Budget\-limited due to API cost \($635–1,146 per task\)\. Best Val = best validation score at any step within the iteration\.TaskIterGPU\-hrsStateBest ValBest StepTestPaperSearchQA14\.7Failed———23\.8Failed———3165\.7Cancelled0\.384500\.3844137\.1Cancelled0\.392800\.3665167\.9Completed0\.3921200\.39468\.6Failed0\.3742000\.3867161\.5Completed0\.3962600\.40286\.0Failed———953\.0Cancelled0\.3761600\.394WildSci1136\.8Cancelled0\.591200\.561271\.7Cancelled0\.593250\.5713110\.8Cancelled0\.619450\.5384111\.8Cancelled0\.621550\.566517\.1Completed0\.597750\.570639\.8Cancelled0\.611750\.603SSMR\-Bench1328\.8Cancelled0\.828900\.820221\.2Completed0\.816810\.788348\.1Cancelled0\.846850\.800464\.2Completed0\.8241010\.820565\.4Completed0\.8141050\.796672\.2Cancelled0\.8141050\.804ChemCoT1399\.7Cancelled0\.2811250\.376223\.8Failed0\.2531200\.371355\.4Cancelled0\.2711250\.334496\.6Cancelled0\.2991300\.380546\.8Cancelled0\.2911400\.380647\.3Cancelled0\.2951400\.377739\.2Cancelled0\.2591250\.356

### E\.5Per\-Node Results:LLMZero

Tables[18](https://arxiv.org/html/2606.18388#A5.T18)–[21](https://arxiv.org/html/2606.18388#A5.T21)report per\-node results forLLMZeroon all four tasks\.

Table 18:LLMZeroper\-node results on SSMR\-Bench \(10 nodes, 64×\\timesA100\)\. Val = aggregate validation score\. Test = average of 4 subtask test scores at best validation step\. ES = early\-stopped\.NodeWall\-hrsGPU\-hrsStatusBest ValBest StepTest \(avg\)04\.5291Completed0\.700300\.65812\.5162ES0\.672330\.65822\.5162ES0\.682360\.68237\.0451ES0\.732660\.706413\.2846Completed0\.8301530\.82257\.3467ES0\.8701800\.822612\.8821ES0\.764990\.73875\.8371ES0\.558480\.52682\.5162ES0\.8581950\.83896\.7428Crashed0\.8561920\.818Total4,159

Table 19:LLMZeroper\-node results on ChemCoT \(16 nodes, 64×\\timesA100\)\. Test = avg\(und\_avg, edit\_avg, rxn\_avg\) at best validation step\.Table 20:LLMZeroper\-node results on PaperSearchQA \(16 nodes, 32×\\timesA100\)\. Test = single\-metric test score at best validation step\.NodeWall\-hrsGPU\-hrsStatusBest ValBest StepTest019\.1611Completed0\.3881140\.390115\.7501Completed0\.4081230\.410218\.8601Completed0\.4081410\.42234\.0129ES0\.3961440\.40043\.5113ES0\.3901230\.400522\.3714Completed0\.4203030\.426640\.01,281Completed0\.3902460\.37875\.5177ES0\.4021650\.39886\.0193ES0\.4143060\.424910\.0321ES0\.4001410\.396106\.3201ES0\.4021620\.3981140\.01,281Completed0\.3803000\.372124\.0129ES0\.3981470\.400133\.5113ES0\.3961620\.402144\.0129ES0\.3921650\.400154\.8153ES0\.4041710\.406Total6,646

Table 21:LLMZeroper\-node results on WildSci \(16 nodes, 32×\\timesA100\)\. Test = average of 9 domain test scores at best validation step\.NodeWall\-hrsGPU\-hrsStatusBest ValBest StepTest \(avg\)07\.3234Completed0\.608360\.548110\.3329ES0\.626510\.548216\.8537Completed0\.6241050\.55139\.3297ES0\.618540\.565414\.6466ES0\.6321590\.574514\.5464Completed0\.620960\.593630\.0960Completed0\.6361890\.586717\.5560Completed0\.6281860\.57387\.8249ES0\.6362100\.58997\.0225ES0\.6362160\.5731014\.1450ES0\.6302160\.5691140\.01,281Completed0\.6221200\.560124\.0129ES0\.6201950\.588139\.8313ES0\.6221530\.5631413\.8442ES0\.6201290\.5861513\.6436Completed0\.6401740\.585Total7,370

## Appendix FHuman Knowledge Injection

Even the latest LLMs frequently misinterpret domain\-specific training metrics and hyperparameters in GRPO/PPO\. For example, models consistently confuse PPO policy clipping with response length clipping\. To ground the LLM’s reasoning, we inject structured human\-written descriptions for metrics and hyperparameters into the prompt\. Importantly, these descriptions do*not*restrict the LLM to only modifying listed parameters; the agent can change any hyperparameter in the training configuration, including those not covered by the guide\.

### F\.1Metric Descriptions

Figure[6](https://arxiv.org/html/2606.18388#A6.F6)lists the 12 metric descriptions injected into both the proposer and early stopper prompts\. These address common LLM misinterpretations\.

Metric Descriptions \(injected into both proposer and early stopper prompts\)actor/entropyPolicy entropy measuring exploration\. Healthy: dropping with rising rewards\. Problem: both dropping \(losing generality without gains\)\.actor/kl\_lossKL divergence loss measuring policy divergence from old policy\. Should remain small and stable\.actor/ppo\_klPPO KL divergence\.Critical:if ppo\_kl spikes AND rewards drop afterward, training has collapsed\.actor/pg\_lossPolicy gradient loss \(primary training signal\)\. Should generally decrease as policy improves\.actor/pg\_clipfracPPO policy clipping fraction \(upper\)\. Ratio of updates hitting clip\_ratio\_high\. High \(\>\>0\.3\) = many updates being prevented\.Unrelated to response\_length/clip\_ratio\.actor/pg\_clipfrac\_lowerPPO policy clipping fraction \(lower\)\. Shows how often PPO constrains updates\.Unrelated to response\_length/clip\_ratio\.actor/grad\_normGradient norm magnitude\. Very high = instability; very low = vanishing gradients\.critic/rewards/meanMean predicted rewards\. Key indicator of training progress\. Should generally increase\.response\_length/meanAverage response length in tokens\. Bounded by max\_response\_length\. Interpret alongside rewards\.response\_length/clip\_ratioResponse truncation ratio \(responses hitting max length\)\.NOT related to PPO clip\_ratio\.High = responses being cut off\.timing\_s/stepTime per step\. Sudden increases may indicate memory issues or inefficient batch sizes\.perf/max\_memory\_reserved\_gbMax GPU memory reserved\. Monitors OOM risk\. Helps optimize batch sizes\.Figure 6:Human\-written metric descriptions injected into agent prompts to ground LLM reasoning about training dynamics\.
### F\.2Hyperparameter Descriptions

Figure[7](https://arxiv.org/html/2606.18388#A6.F7)provides the hyperparameter reference guide injected into the proposer prompt\. Parameters are organized by functional category\. The LLM is not restricted to modifying only these parameters; it can change any value in the training configuration\.

Hyperparameter Reference Guide \(injected into proposer prompt; LLM not restricted to this list\)Learning Rates & Optimizationweight\_decayL2 regularization\. Increase if large train/val gap; decrease if underfitting\.grad\_clipMax gradient norm\. Decrease if NaN losses or collapse; increase if gradients consistently small\.Batch Sizesppo\_micro\_batch\_size\_per\_gpuMicro\-batch for gradient accumulation\. Critical: ppo\_mini\_batch\_size must be divisible by this\.PPO/GRPO Clippingclip\_ratio\_lowPPO lower bound\. Keep at default; increasing may collapse sampling space\.clip\_ratio\_highPPO upper bound\. Higher values enhance entropy and diversity\.KL Divergence & Regularizationkl\_loss\_coefKL loss weight constraining policy changes\. Only increase if KL spikes coincide with reward collapse\.entropy\_coeffEntropy bonus weight\.Critical: must remain 0 for GRPO\.norm\_adv\_by\_std\_in\_grpoNormalize advantages by std\. Generally keep True for stability\.Rollout & Samplingrollout\.nSamples per prompt\. Higher improves advantage estimates, increases compute linearly\.temperatureSampling temperature\. Higher increases exploration and diversity\.Sequence & Memory Managementmax\_response\_lengthMax response tokens\. Increase if response\_length/clip\_ratio is high\.max\_num\_seqsMax concurrent vLLM sequences\. Decrease if OOM; increase for GPU utilization\.tensor\_model\_parallel\_sizeGPUs for model sharding\. Use minimum that fits in memory\.Training Configurationtotal\_epochsPasses through data\. Increase if validation still improving at final epoch\.Figure 7:Human\-written hyperparameter descriptions injected into the proposer prompt, organized by functional category\.

## Appendix GAgent Prompts

We provide the full proposer and early stopper prompts below\. These prompts were minimally tuned: we deliberately avoided hardcoding any numeric thresholds or task\-specific values, relying instead on the LLM’s reasoning to interpret metrics in context\. This design leaves substantial room for improvement through prompt engineering, and we expect that more carefully crafted prompts could further improve search efficiency\.

### G\.1Proposer Agent Prompt

The proposer receives the previous training configuration \(as YAML\), step\-level metric summaries \(text and/or plots\), and produces a diagnosis with hyperparameter suggestions\. The template \(Figure[8](https://arxiv.org/html/2606.18388#A7.F8)\) is instantiated with run\-specific data at each iteration; placeholders in\{braces\}are filled dynamically\.

Proposer Agent Prompt Template[⬇](data:text/plain;base64,WW91IGFyZSBhIGh5cGVycGFyYW1ldGVyIG9wdGltaXphdGlvbiBleHBlcnQuIFlvdXIgZ29hbCBpcyB0byBtYXhpbWl6ZSB0aGUgdmFsaWRhdGlvbgpzY29yZSBieSBwcm9wb3NpbmcgaHlwZXJwYXJhbWV0ZXIgY2hhbmdlcyBmb3IgdGhlIG5leHQgdHJhaW5pbmcgcnVuLgoKe3ByZXZpb3VzX3RyYWluaW5nX2NvbmZpZ30KCntlYXJseV9zdG9wX25vdGljZX0KCiMjIyBUcmFpbmluZyBTdGVwIE1ldHJpY3MgU3VtbWFyeQpUaGUgdHJhaW5pbmcgcmFuIGZvciB7dHJhaW5pbmdfc3RlcHN9IHN0ZXBzOgoKe3N0ZXBfbWV0cmljc19zdW1tYXJ5fQoKe21ldHJpY19kZXNjcmlwdGlvbnN9Cgp7aHlwZXJwYXJhbWV0ZXJfZ3VpZGV9CgojIyMgTWV0cmljIEludGVycHJldGF0aW9uCioqUFJJTUFSWToqKiBSZXdhcmRzIChjcml0aWMvcmV3YXJkcy9tZWFuKSBhbmQgdmFsaWRhdGlvbiBzY29yZXMgbWVhc3VyZSBvdmVyYWxsCnBlcmZvcm1hbmNlLiBKdWRnZSBzdWNjZXNzIGJ5IHRoZXNlIG1ldHJpY3MuCioqRElBR05PU1RJQzoqKiBhY3Rvci9lbnRyb3B5LCBhY3Rvci9rbF9sb3NzLCBhY3Rvci9ncmFkX25vcm0sIGV0Yy4gT25seSByZWxldmFudAp3aGVuIHByaW1hcnkgbWV0cmljcyBkZWdyYWRlLiBSaXNpbmcgS0wgd2l0aCBzdGFibGUgcmV3YXJkcyBpcyBhY2NlcHRhYmxlLgoKIyMjIFlvdXIgVGFzawoKRm9sbG93IHRoZXNlIHN0ZXBzICoqaW4gb3JkZXIqKi4gRG8gTk9UIHN1Z2dlc3QgYHRvdGFsX2Vwb2Noc2AgdW50aWwgU3RlcCA0LgoKKipTdGVwIDEgLSBEaWFnbm9zaXM6KiogU3VtbWFyaXplIHRoZSB0cmFpbmluZyBoZWFsdGguIFdoYXQgYXJlIHRoZSBrZXkgdHJlbmRzIGluCnByaW1hcnkgbWV0cmljcz8gSXMgdGhlIHJ1biBoZWFsdGh5LCBwbGF0ZWF1aW5nLCBvciBjb2xsYXBzaW5nPyBOb3RlIGFueSBpbmZsZWN0aW9uCnBvaW50cyBvciBhbm9tYWxpZXMuCgoqKlN0ZXAgMiAtIEh5cGVycGFyYW1ldGVyIHN1Z2dlc3Rpb25zOioqIEJhc2VkIG9uIHlvdXIgZGlhZ25vc2lzLCBwcm9wb3NlIHNwZWNpZmljCmh5cGVycGFyYW1ldGVyIGNoYW5nZXMgZm9yIHRoZSBuZXh0IHJ1bi4gRG8gTk9UIGluY2x1ZGUgYHRvdGFsX2Vwb2Noc2AgaGVyZQooY29tcHV0ZWQgaW4gU3RlcCA0KS4Ke3N1Z2dlc3Rpb25fY291bnRfZ3VpZGFuY2V9CgoqKlN0ZXAgMyAtIENoZWNrcG9pbnQgZGVjaXNpb246KioKe2NoZWNrcG9pbnRfaW5mb30KCioqU3RlcCA0IC0gRXBvY2ggY29tcHV0YXRpb246KiogTm93IHRoYXQgeW91IGhhdmUgZGVjaWRlZCBhbGwgaHlwZXJwYXJhbWV0ZXJzCihpbmNsdWRpbmcgYHRyYWluX2JhdGNoX3NpemVgKSBhbmQgdGhlIGNoZWNrcG9pbnQgc3RyYXRlZ3ksIGNvbXB1dGUgYHRvdGFsX2Vwb2Noc2A6Ci0gSWYgcmVzdW1pbmc6IG9sZF9zdGVwc19wZXJfZXBvY2ggPSB0b3RhbF90cmFpbmluZ19zdGVwcyAvIG9sZF90b3RhbF9lcG9jaHMsCiAgbmV3X3N0ZXBzX3Blcl9lcG9jaCA9IG9sZF9zdGVwc19wZXJfZXBvY2ggKiBvbGRfYmF0Y2hfc2l6ZSAvIG5ld19iYXRjaF9zaXplLAogIHRvdGFsX2Vwb2NocyA9IGNlaWwoKHJlc3VtZV9zdGVwICsgQURESVRJT05BTF9UUkFJTklOR19TVEVQUykgLyBuZXdfc3RlcHNfcGVyX2Vwb2NoKQotIElmIHRyYWluaW5nIGZyb20gc2NyYXRjaDogY2hvb3NlIGB0b3RhbF9lcG9jaHNgIGJhc2VkIG9uIHRhcmdldCB0cmFpbmluZyBzdGVwcwogIGFuZCBuZXcgYmF0Y2ggc2l6ZS4KClNob3cgdGhlIG1hdGggZXhwbGljaXRseS4=)Youareahyperparameteroptimizationexpert\.Yourgoalistomaximizethevalidationscorebyproposinghyperparameterchangesforthenexttrainingrun\.\{previous\_training\_config\}\{early\_stop\_notice\}\#\#\#TrainingStepMetricsSummaryThetrainingranfor\{training\_steps\}steps:\{step\_metrics\_summary\}\{metric\_descriptions\}\{hyperparameter\_guide\}\#\#\#MetricInterpretation\*\*PRIMARY:\*\*Rewards\(critic/rewards/mean\)andvalidationscoresmeasureoverallperformance\.Judgesuccessbythesemetrics\.\*\*DIAGNOSTIC:\*\*actor/entropy,actor/kl\_loss,actor/grad\_norm,etc\.Onlyrelevantwhenprimarymetricsdegrade\.RisingKLwithstablerewardsisacceptable\.\#\#\#YourTaskFollowthesesteps\*\*inorder\*\*\.DoNOTsuggest‘total\_epochs‘untilStep4\.\*\*Step1\-Diagnosis:\*\*Summarizethetraininghealth\.Whatarethekeytrendsinprimarymetrics?Istherunhealthy,plateauing,orcollapsing?Noteanyinflectionpointsoranomalies\.\*\*Step2\-Hyperparametersuggestions:\*\*Basedonyourdiagnosis,proposespecifichyperparameterchangesforthenextrun\.DoNOTinclude‘total\_epochs‘here\(computedinStep4\)\.\{suggestion\_count\_guidance\}\*\*Step3\-Checkpointdecision:\*\*\{checkpoint\_info\}\*\*Step4\-Epochcomputation:\*\*Nowthatyouhavedecidedallhyperparameters\(including‘train\_batch\_size‘\)andthecheckpointstrategy,compute‘total\_epochs‘:\-Ifresuming:old\_steps\_per\_epoch=total\_training\_steps/old\_total\_epochs,new\_steps\_per\_epoch=old\_steps\_per\_epoch\*old\_batch\_size/new\_batch\_size,total\_epochs=ceil\(\(resume\_step\+ADDITIONAL\_TRAINING\_STEPS\)/new\_steps\_per\_epoch\)\-Iftrainingfromscratch:choose‘total\_epochs‘basedontargettrainingstepsandnewbatchsize\.Showthemathexplicitly\.Figure 8:Proposer agent prompt template\. Placeholders are filled with run\-specific data at each search iteration\.
### G\.2Early Stopper Agent Prompt

The early stopper receives comparison plots \(current run in blue vs\. best run in green\) and decides whether to continue or stop\. The prompt template is shown in Figure[9](https://arxiv.org/html/2606.18388#A7.F9)\.

Early Stopper Agent Prompt Template[⬇](data:text/plain;base64,WW91IGFyZSBhbiBleHBlcnQgYXQgYW5hbHl6aW5nIFJMIGZvciBMTE0gcG9zdCB0cmFpbmluZyB1c2luZyB2aXN1YWwgcGxvdHMuIFlvdXIgdGFzawppcyB0byBkZWNpZGUgaWYgYSBydW4gc2hvdWxkIGNvbnRpbnVlIG9yIHN0b3AgZWFybHkuCgojIyMgT2JqZWN0aXZlCllvdXIgT05MWSBnb2FsIGlzIHRvIG1heGltaXplIHRoZSB2YWxpZGF0aW9uIHNjb3JlLiBDT05USU5VRSBpZiB0aGUgY3VycmVudCBydW4ncwp2YWxpZGF0aW9uIHNjb3JlIHRyYWplY3RvcnkgaGFzIHBvdGVudGlhbCB0byBleGNlZWQgdGhlIGJlc3QgdmFsaWRhdGlvbiBzY29yZSBzZWVuIHNvCmZhci4gU1RPUCBvbmx5IHdoZW4gdGhlcmUgaXMgbm8gcmVhbGlzdGljIGNoYW5jZSBvZiBiZWF0aW5nIGl0LgoKIyMjIEJlc3QgUnVuIEluZm9ybWF0aW9uCntiZXN0X3J1bl9zdW1tYXJ5fQoKIyMjIEN1cnJlbnQgUnVuIFByb2dyZXNzIChhdCBjaGVja3BvaW50KQp7Y3VycmVudF9ydW5fc3VtbWFyeX0KCiMjIyBWaXN1YWwgQ29tcGFyaXNvbgpUaGUgcGxvdHMgc2hvdyB0cmFpbmluZyBtZXRyaWNzIG92ZXIgdGltZS4gQmx1ZSA9IGN1cnJlbnQgcnVuLCBncmVlbiBkYXNoZWQgPSBiZXN0CnJ1biwgYm90aCBtYXkgc3RhcnQgZnJvbSBhbiBpbnRlcm1lZGlhdGUgY2hlY2twb2ludC4KCkZvY3VzIG9uIHRoZSAqKnZhbGlkYXRpb24gc2NvcmUgcGxvdCoqLiBBc2s6IGNhbiB0aGUgYmx1ZSBjdXJ2ZSdzIHBlYWsgZXZlbnR1YWxseQpleGNlZWQgdGhlIGdyZWVuIGN1cnZlJ3MgcGVhaz8KCnttZXRyaWNfZGVzY3JpcHRpb25zfQoKIyMjIERlY2lzaW9uIFJ1bGVzCioqRGVmYXVsdCBhY3Rpb24gaXMgQ09OVElOVUUuKiogV2hlbiB1bmNlcnRhaW4sIENPTlRJTlVFLiBPbmx5IFNUT1AgaWYgQUxMIG9mIHRoZXNlCmFyZSB0cnVlOgoxLiBDdXJyZW50IHN0ZXAgPj0gMTAgKHRvbyBlYXJseSB0byBqdWRnZSBiZWZvcmUgdGhhdCkKMi4gVGhlIHZhbGlkYXRpb24gc2NvcmUgdHJhamVjdG9yeSBoYXMgbm8gcmVhbGlzdGljIGNoYW5jZSBvZiBleGNlZWRpbmcgdGhlIGJlc3QKICAgdmFsaWRhdGlvbiBzY29yZSBzZWVuIHNvIGZhciwgY29uc2lkZXJpbmcgdGhlIGltcHJvdmVtZW50IHJhdGUsIG5vdCBqdXN0IHRoZQogICBjdXJyZW50IHZhbHVlLiBBIHJ1biBiZWhpbmQgdGhlIGJlc3QgY2FuIHN0aWxsIHdpbiBpZiBpdHMgdHJhamVjdG9yeSBpcyBzdGVlcGVyOwogICBhIHJ1biBhaGVhZCBjYW4gc3RpbGwgbG9zZSBpZiBpdCBpcyBwbGF0ZWF1aW5nLgoKKipPbmx5IHRoZSB2YWxpZGF0aW9uIHNjb3JlIGRldGVybWluZXMgU1RPUC9DT05USU5VRS4qKiBBbGwgb3RoZXIgbWV0cmljcyAocmV3YXJkcywKZW50cm9weSwgS0wsIGdyYWRpZW50IG5vcm1zLCBjbGlwX3JhdGlvLCBldGMuKSBwcm92aWRlIGNvbnRleHQgYnV0IE5FVkVSIGp1c3RpZnkKc3RvcHBpbmcgb24gdGhlaXIgb3duLgoKKipSZXNwb25zZSBGb3JtYXQ6KioKUkVBU09OOiBbMS0yIHNlbnRlbmNlcyBvbiB3aGV0aGVyIHRoZSB2YWxpZGF0aW9uIHNjb3JlIHRyYWplY3RvcnkgY2FuIGJlYXQgdGhlIGJlc3QKdmFsaWRhdGlvbiBzY29yZSBzZWVuIHNvIGZhcl0KREVDSVNJT046IFtDT05USU5VRSBvciBTVE9QXQ==)YouareanexpertatanalyzingRLforLLMposttrainingusingvisualplots\.Yourtaskistodecideifarunshouldcontinueorstopearly\.\#\#\#ObjectiveYourONLYgoalistomaximizethevalidationscore\.CONTINUEifthecurrentrun’svalidationscoretrajectoryhaspotentialtoexceedthebestvalidationscoreseensofar\.STOPonlywhenthereisnorealisticchanceofbeatingit\.\#\#\#BestRunInformation\{best\_run\_summary\}\#\#\#CurrentRunProgress\(atcheckpoint\)\{current\_run\_summary\}\#\#\#VisualComparisonTheplotsshowtrainingmetricsovertime\.Blue=currentrun,greendashed=bestrun,bothmaystartfromanintermediatecheckpoint\.Focusonthe\*\*validationscoreplot\*\*\.Ask:canthebluecurve’speakeventuallyexceedthegreencurve’speak?\{metric\_descriptions\}\#\#\#DecisionRules\*\*DefaultactionisCONTINUE\.\*\*Whenuncertain,CONTINUE\.OnlySTOPifALLofthesearetrue:1\.Currentstep\>=10\(tooearlytojudgebeforethat\)2\.Thevalidationscoretrajectoryhasnorealisticchanceofexceedingthebestvalidationscoreseensofar,consideringtheimprovementrate,notjustthecurrentvalue\.Arunbehindthebestcanstillwinifitstrajectoryissteeper;arunaheadcanstillloseifitisplateauing\.\*\*OnlythevalidationscoredeterminesSTOP/CONTINUE\.\*\*Allothermetrics\(rewards,entropy,KL,gradientnorms,clip\_ratio,etc\.\)providecontextbutNEVERjustifystoppingontheirown\.\*\*ResponseFormat:\*\*REASON:\[1\-2sentencesonwhetherthevalidationscoretrajectorycanbeatthebestvalidationscoreseensofar\]DECISION:\[CONTINUEorSTOP\]Figure 9:Early stopper agent prompt template\. Invoked every 900 seconds during training\.

## Appendix HEthics and Artifact Documentation

##### Potential risks\.

LLMZeroautomates hyperparameter search for RL post\-training, which could lower the barrier to fine\-tuning language models for harmful purposes\. However, the system requires substantial compute, limiting misuse to well\-resourced actors who already have access to equivalent capabilities\.

##### Artifact licenses\.

All artifacts are used under their respective open\-source licenses\.

##### Intended use consistency\.

All datasets are used for their intended purpose of evaluating language model capabilities on domain\-specific reasoning tasks\. VeRL and Ray are used for distributed RL training, consistent with their documented use cases\.

##### Artifact documentation\.

We use 4 evaluation datasets \(5,000 train / 500 val / 500 test each\), Qwen3 models \(0\.6B–8B\) with LoRA fine\-tuning, a modified version of VeRL for GRPO training, and Ray for distributed orchestration on EKS clusters with A100 GPUs\. All datasets are publicly available\. We do not release trained model weights; we release discovered strategy configurations to enable reproduction\.

## Appendix IUse of AI

We used AI\-based tools to assist with grammar and writing clarity of the paper\.