Efficient Long-Horizon Learning for Learned Optimization

# Efficient Long-Horizon Learning for Learned Optimization
Source: [https://arxiv.org/html/2607.06772](https://arxiv.org/html/2607.06772)
Xiaolong Huang1,3Benjamin Thérien1,4James Harrison2Eugene Belilovsky1,3 1Mila \- Quebec AI Institute2Google DeepMind 3Concordia University4Université de Montréal

###### Abstract

Learned optimization aims to improve upon hand\-designed optimizers \(e\.g\., Adam and Muon\) by meta\-learning small neural network optimizers over a distribution of tasks\. While recent work has greatly advanced the architectural design and inductive biases of learned optimizers \(LOs\), current meta\-training approaches still suffer from two main difficulties: \(1\) they cannot efficiently scale meta\-training to long\-horizon inner problems and \(2\) they often fail to surpass comparable hand\-designed optimizers\. To address these limitations, we propose Efficient Long\-hOrizon \(ELO\) learning, an efficient meta\-training algorithm that \(1\) reallocates redundant meta\-training compute to longer failure regimes, achieving efficient long\-horizon learning, and \(2\) enforces decoupled progressive expert supervision, providing stable meta\-learning signals that additionally improve the generalization of LOs\. Our empirical study evaluates ELO for meta\-training both element\-wise and matrix\-based LOs\. Across downstream language modeling \(GPT\-2\-124M/350M on FineWeb\) and image classification \(ViT\-B/16, ResNet\-50 on ImageNet\-1K\) tasks, ELO substantially improves the long\-unroll performance and out\-of\-distribution generalization of the base LOs\. In particular, ELO\-Celo2 consistently outperforms well\-tuned AdamW across all evaluated tasks, while remaining competitive with Muon on language modeling\.Notably, all ELO baselines require less than 7 H100 GPU\-hours for meta\-training\.

## 1Introduction

The remarkable achievements of deep neural networks are closely tied to the evolution of optimization techniques\(Sunet al\.,[2019](https://arxiv.org/html/2607.06772#bib.bib34); Sun,[2020](https://arxiv.org/html/2607.06772#bib.bib35); Abdulkadirovet al\.,[2023](https://arxiv.org/html/2607.06772#bib.bib36)\)\. While hand\-designed optimizers \(e\.g\., Adam, Muon, etc\.\) remain the dominant choice in practice, learned optimizers \(LOs\), which use a small neural network to estimate parameter updates, have demonstrated the potential to discover superior optimization rules through meta\-learning\(Andrychowiczet al\.,[2016](https://arxiv.org/html/2607.06772#bib.bib7); Metzet al\.,[2019](https://arxiv.org/html/2607.06772#bib.bib31); Chenet al\.,[2020b](https://arxiv.org/html/2607.06772#bib.bib5); Moudgilet al\.,[2025](https://arxiv.org/html/2607.06772#bib.bib65),[2026](https://arxiv.org/html/2607.06772#bib.bib63); Thérienet al\.,[2024](https://arxiv.org/html/2607.06772#bib.bib4)\)\.

![Refer to caption](https://arxiv.org/html/2607.06772v1/x1.png)Figure 1:ELO improves downstream optimization and meta\-training efficiency\.\(A\)Validation loss on GPT\-2 350M pretraining with FineWeb\. Meta\-trained only on four tiny vision\-MLP tasks, ELO\-Celo2 is able to generalize to realistic language modeling problem, outperforming AdamW and matching Muon\.\(B\)Meta\-evaluation loss during meta\-training\. ELO converges faster than other baselines over the 2K\-step unroll covered by all methods, and achieves lower loss at the longer 10K\-step horizon\.A long line of work has sought to improve learned optimization mainly through architectural innovations and carefully designed inductive biases\. Architecturally, LOs have evolved from LSTM\-based update rules\(Andrychowiczet al\.,[2016](https://arxiv.org/html/2607.06772#bib.bib7)\)and hierarchical RNNs\(Wichrowskaet al\.,[2017](https://arxiv.org/html/2607.06772#bib.bib37)\)to LSTM\-MLP hybrids\(Metzet al\.,[2020](https://arxiv.org/html/2607.06772#bib.bib67)\), per\-parameter MLPs\(Metzet al\.,[2022a](https://arxiv.org/html/2607.06772#bib.bib9)\), LSTM hypernetworks\(Metzet al\.,[2022b](https://arxiv.org/html/2607.06772#bib.bib8)\), and, more recently, per\-parameter MLPs equipped with Newton\-Schulz iterations\(Moudgilet al\.,[2026](https://arxiv.org/html/2607.06772#bib.bib63)\)\. Complementary work has improved the inductive bias of LOs by adding nominal descent terms, weight decay, and output preconditioning\(Harrisonet al\.,[2022](https://arxiv.org/html/2607.06772#bib.bib53)\), incorporating Hessian and local\-entropy flatness\-aware regularizers\(Yanget al\.,[2023](https://arxiv.org/html/2607.06772#bib.bib68)\), and placing the LOs in theμ\\muP parameterization to improve generalization\(Thérienet al\.,[2024](https://arxiv.org/html/2607.06772#bib.bib4)\)\. Despite steady progress in these directions, existing meta\-training algorithms still struggle to scale efficiently to long\-horizon optimization learning\. Moreover, the resulting LOs often underperform comparable strong hand\-designed optimizers\.

In this work, we start by providing an analysis of how traditional meta\-training pipelines lead to wasted compute\. To address this inefficiency, we use a failure\-aware resume buffer to effectively recycle compute from redundant early\-unroll learning to later regions where long\-horizon failures happen\. However, this also introduces unstable meta\-training dynamics due to compounding errors in long\-unroll optimization\. For stabilization and generalization purposes, we additionally constrain the LO with decoupled progressive expert guidance, which anchors early parameter updates to a reliable expert optimizer through separate direction and magnitude supervision, and gradually relaxes this constraint as the LO improves\. Our contributions can be summarized as follows:

- •We propose efficient long\-horizon learning \(ELO\), a novel meta\-training algorithm for learned optimization, which incorporates a failure\-aware resume buffer and decoupled progressive expert guidance\. We empirically demonstrate that ELO achieves faster meta\-training convergence than prior approaches under the same compute budget\.
- •We establish the performance of LOs meta\-trained with ELO on practical downstream image classification and language modeling tasks, where ELO baselines consistently improve over their corresponding base LOs\. Most notably, ELO\-Celo2 outperforms well\-tuned AdamW across all evaluated tasks and remains competitive with Muon on language modeling\.
- •Through extensive ablation studies, we establish the contribution of ELO’s key components, including the improvements from the failure\-aware buffer, the effect of expert optimizer choice, and the benefit of a decoupled imitation objective over a standard MSE loss\.

Our code is open\-sourced at:[https://github\.com/xiaol827/ELO](https://github.com/xiaol827/ELO)\.

## 2Background

We review the necessary background and related work on learned optimization to establish the context for our method\. For an extended review of related literature, we refer the reader to Appendix\.[B](https://arxiv.org/html/2607.06772#A2)\.

LO Architecture\.In this work, we adopt two LO architectures: the state\-of\-the\-art Celo2\(Moudgilet al\.,[2026](https://arxiv.org/html/2607.06772#bib.bib63)\)and the standard per\-parameter MLP baselinesmall\_fc\_lopt\(Metzet al\.,[2022a](https://arxiv.org/html/2607.06772#bib.bib9)\), which we refer to assmall\_fcfor brevity\.small\_fcis a three\-layer MLP with ReLU activations\. It takes as input a feature vector \(e\.g\., gradient, momentum, rms, etc\.\) for each parameterθ\\thetaof the optimizee \(the network currently being optimized\) and outputs an update direction,dd, and magnitude,mm\. That is,fϕ​\(⋅\)=\[d,m\]f\_\{\\phi\}\(\\cdot\)=\[d,m\], wherefϕf\_\{\\phi\}is the LO andϕ\\phidenotes its parameters\. Concretely, the update is applied as follows:

θn\+1=θn−Δn;Δn=λ1​d×eλ2​m,\\displaystyle\\theta\_\{n\+1\}=\\theta\_\{n\}\-\\Delta\_\{n\};\\;\\;\\;\\Delta\_\{n\}=\\lambda\_\{1\}d\\times e^\{\\lambda\_\{2\}m\},\(1\)whereλ1\\lambda\_\{1\}andλ2\\lambda\_\{2\}are hyperparameters usually fixed to0\.0010\.001\. Building on the element\-wise update rule ofsmall\_fc, Celo2 introduces tensor\-level orthonormalization as a key improvement for non\-1D parameters in the optimizee\. Specifically, for any weight matrix𝑾\{\\bm\{W\}\}in the optimizee, the update is applied as:

𝑾n\+1=𝑾n−NewtonShulz5​\(Δn\)/‖Δn‖rms\.\{\\bm\{W\}\}\_\{n\+1\}=\{\\bm\{W\}\}\_\{n\}\-\\text\{NewtonShulz5\}\(\\Delta\_\{n\}\)/\\\|\\Delta\_\{n\}\\\|\_\{\\text\{rms\}\}\.\(2\)
Meta\-training ObjectiveIn general, learning the meta\-parameters,ϕ\\phi, involves solving an optimization problem of the form:

minϕ⁡𝔼τ∼𝒟​\[∑n=1Nℒnmeta​\(θn;ϕ,τ\)\],\\min\_\{\\phi\}\\;\\mathbb\{E\}\_\{\\tau\\sim\\mathcal\{D\}\}\\Big\[\\sum\_\{n=1\}^\{N\}\\mathcal\{L\}^\{\\text\{meta\}\}\_\{n\}\(\\theta\_\{n\};\\phi,\\tau\)\\Big\],\(3\)where𝒟\\mathcal\{D\}is a distribution of tasks,τ\\tauspecifies a task \(e\.g\., optimizee initialization, dataset, objective\)\. The objective seeks to minimize the sum of per inner step losses over the training horizonNN\(Thérienet al\.,[2024](https://arxiv.org/html/2607.06772#bib.bib4)\)\.

![Refer to caption](https://arxiv.org/html/2607.06772v1/x2.png)Figure 2:Ablation of ELO sampling and expert imitation\.\(A\)ELO shifts meta\-training budget toward longer inner horizons, reaching up to17×17\\timeslarger inner steps than the standard baseline\.\(B\)Complete ELO remains stable under longer\-horizon sampling, whereas buffer\-only training without progressive expert supervision becomes unstable\.\(C\)Compared with direct MSE loss, the decoupled direction and magnitude objective yields better direction alignment and lower meta\-evaluation loss\.for*t=0,1,…,T−1t=0,1,\\ldots,T\-1*do

Sample

u∼Uniform⁡\(0,1\)u\\sim\\operatorname\{Uniform\}\(0,1\);

if*\(u<Pℬ\)∧\(ℬ≠∅\)\(u<P\_\{\\mathcal\{B\}\}\)\\wedge\(\\mathcal\{B\}\\neq\\emptyset\)*then

K,θK←ℬK,\\theta\_\{K\}\\leftarrow\\mathcal\{B\};

else

K=0,θ0←K=0,\\theta\_\{0\}\\leftarrowrandomly initialized inner state;

end if

VK←0V\_\{K\}\\leftarrow 0;

αt=tT−1\\alpha\_\{t\}=\\frac\{t\}\{T\-1\};

for*n=K,K\+1,…,K\+N−1n=K,K\+1,\\ldots,K\+N\-1*do

θn\+1ℰ←θn\+Δ​θnℰ\\theta\_\{n\+1\}^\{\\mathcal\{E\}\}\\leftarrow\\theta\_\{n\}\+\\Delta\\theta\_\{n\}^\{\\mathcal\{E\}\};

θn\+1𝒪←θn\+Δ​θn𝒪\\theta\_\{n\+1\}^\{\\mathcal\{O\}\}\\leftarrow\\theta\_\{n\}\+\\Delta\\theta\_\{n\}^\{\\mathcal\{O\}\};

θn\+1←\(1−αt\)​θn\+1ℰ\+αt​θn\+1𝒪\\theta\_\{n\+1\}\\leftarrow\(1\-\\alpha\_\{t\}\)\\theta\_\{n\+1\}^\{\\mathcal\{E\}\}\+\\alpha\_\{t\}\\theta\_\{n\+1\}^\{\\mathcal\{O\}\};

ℒndir←1−Δ​θnℰ⋅Δ​θn𝒪‖Δ​θnℰ‖2​‖Δ​θn𝒪‖2\\mathcal\{L\}\_\{n\}^\{\\mathrm\{dir\}\}\\leftarrow 1\-\\frac\{\\Delta\\theta\_\{n\}^\{\\mathcal\{E\}\}\\cdot\\Delta\\theta\_\{n\}^\{\\mathcal\{O\}\}\}\{\\left\\\|\\Delta\\theta\_\{n\}^\{\\mathcal\{E\}\}\\right\\\|\_\{2\}\\left\\\|\\Delta\\theta\_\{n\}^\{\\mathcal\{O\}\}\\right\\\|\_\{2\}\};

ℒnmag←\|‖Δ​θnℰ‖2−‖Δ​θn𝒪‖2\|\\mathcal\{L\}\_\{n\}^\{\\mathrm\{mag\}\}\\leftarrow\\left\|\\left\\\|\\Delta\\theta\_\{n\}^\{\\mathcal\{E\}\}\\right\\\|\_\{2\}\-\\left\\\|\\Delta\\theta\_\{n\}^\{\\mathcal\{O\}\}\\right\\\|\_\{2\}\\right\|;

ℒnimt←λ​ℒndir\+\(1−λ\)​ℒnmag\\mathcal\{L\}^\{\\mathrm\{imt\}\}\_\{n\}\\leftarrow\\lambda\\mathcal\{L\}^\{\\mathrm\{dir\}\}\_\{n\}\+\(1\-\\lambda\)\\mathcal\{L\}^\{\\mathrm\{mag\}\}\_\{n\};

ℒnmeta←\(1−αt\)​ℒnimt\+αt​ℒntask\\mathcal\{L\}^\{\\mathrm\{meta\}\}\_\{n\}\\leftarrow\(1\-\\alpha\_\{t\}\)\\mathcal\{L\}^\{\\mathrm\{imt\}\}\_\{n\}\+\\alpha\_\{t\}\\mathcal\{L\}^\{\\mathrm\{task\}\}\_\{n\};

Δnℓ←ℒn−1task−ℒntaskifn≠Kelse0\\Delta^\{\\ell\}\_\{n\}\\leftarrow\\mathcal\{L\}^\{\\mathrm\{task\}\}\_\{n\-1\}\-\\mathcal\{L\}^\{\\mathrm\{task\}\}\_\{n\}\\quad\\text\{if\}\\quad n\\neq K\\quad\\text\{else\}\\quad 0;

Vn\+1←max⁡\(0,Vn−Δnℓ\)V\_\{n\+1\}\\leftarrow\\max\(0,V\_\{n\}\-\\Delta^\{\\ell\}\_\{n\}\);

end for

n⋆←arg⁡max⁡Vnn^\{\\star\}\\leftarrow\\arg\\max V\_\{n\};

Kpush←max⁡\(K,n⋆−R\)K\_\{\\mathrm\{push\}\}\\leftarrow\\max\(K,n^\{\\star\}\-R\);

ℬ←sKpush\\mathcal\{B\}\\leftarrow s\_\{K\_\{\\mathrm\{push\}\}\};

gt←PES⁡\(∑n=KK\+N−1ℒnmeta\)g\_\{t\}\\leftarrow\\operatorname\{PES\}\\left\(\\sum\_\{n=K\}^\{K\+N\-1\}\\mathcal\{L\}^\{\\mathrm\{meta\}\}\_\{n\}\\right\);

ϕt\+1←Update⁡\(ϕt,gt\)\\phi\_\{t\+1\}\\leftarrow\\operatorname\{Update\}\(\\phi\_\{t\},g\_\{t\}\);

end for

Algorithm 1Efficient Long\-hOrizon Learning \(ELO\)
Input:

Resume probabilityPℬ=0\.8P\_\{\\mathcal\{B\}\}=0\.8, truncation lengthR=50R=50, direction loss weightλ=0\.7\\lambda=0\.7\.

Initialize:

Resume bufferℬ←∅\\mathcal\{B\}\\leftarrow\\emptyset\.

Notation:Red boxes designate resume buffer operations, while blue boxes highlight progressive expert supervision\. For simplicity, we assume the truncation length equals the unroll horizon in this demonstration\.

Unbiased Truncated Gradient EstimationEstimating meta\-gradients of long unrolled computation graphs with backpropagation is prone to noise due to exploding or vanishing gradients\(Metzet al\.,[2019](https://arxiv.org/html/2607.06772#bib.bib31)\)\. As such, the learned optimization community has turned towards zeroth\-order estimators to estimate meta\-gradients, including full ES\(Metzet al\.,[2019](https://arxiv.org/html/2607.06772#bib.bib31)\)and the more efficient truncated estimator PES\(Vicolet al\.,[2021](https://arxiv.org/html/2607.06772#bib.bib33)\), which we adopt in this work\. PES is an unbiased estimator of the Gaussian\-smoothed objective\(Vicolet al\.,[2021](https://arxiv.org/html/2607.06772#bib.bib33)\)\. It estimates meta\-gradients over segments ofRRinner steps, whereR≪NR\\ll NandNNdenotes the full inner\-problem horizon\. With antithetic samples, the PES estimator at truncation indexkkis given by:

𝒈^kPES=12​σ2​P​∑i=1P∑n=1Rξk\(i\)​Δ​ℒn,i,kmeta;Δ​ℒn,i,kmeta=ℒnmeta​\(ϕ\+ϵ\(i,k\)\)−ℒnmeta​\(ϕ−ϵ\(i,k\)\)\.\\hat\{\\bm\{g\}\}^\{\\mathrm\{PES\}\}\_\{k\}=\\frac\{1\}\{2\\sigma^\{2\}P\}\\sum\_\{i=1\}^\{P\}\\sum\_\{n=1\}^\{R\}\\xi^\{\(i\)\}\_\{k\}\\Delta\\mathcal\{L\}^\{\\text\{meta\}\}\_\{n,i,k\};\\;\\;\\;\\Delta\\mathcal\{L\}^\{\\text\{meta\}\}\_\{n,i,k\}=\\mathcal\{L\}^\{\\text\{meta\}\}\_\{n\}\(\\phi\+\{\\mathbf\{\\epsilon\}\}^\{\(i,k\)\}\)\-\\mathcal\{L\}^\{\\text\{meta\}\}\_\{n\}\(\\phi\-\{\\mathbf\{\\epsilon\}\}^\{\(i,k\)\}\)\.\(4\)
WherePPis the number of monte\-carlo estimates or particles,ξk\(i\)=∑j=1kϵ\(i,j\)\\xi\_\{k\}^\{\(i\)\}=\\sum^\{k\}\_\{j=1\}\{\\mathbf\{\\epsilon\}\}^\{\(i,j\)\}is an accumulator of perturbations, andϵ\(i,j\)\{\\mathbf\{\\epsilon\}\}^\{\(i,j\)\}is the current perturbations sampled from𝒩​\(0,σ2\)\{\\mathcal\{N\}\}\(0,\\sigma^\{2\}\)\. Since our failure\-aware resume buffer maintains the PES state, the meta\-gradients resulting from resumed inner problems are also unbiased \(See Appendix[A](https://arxiv.org/html/2607.06772#A1)for more details\)\.

#### Sampling the inner problem horizon\.

A crucial but seldom discussed aspect of meta\-training LOs is how to choose the horizon of the inner problems\. Existing work samples the unroll horizon mainly through log\-uniform sampling\(Metzet al\.,[2022a](https://arxiv.org/html/2607.06772#bib.bib9); Thérienet al\.,[2024](https://arxiv.org/html/2607.06772#bib.bib4); Moudgilet al\.,[2025](https://arxiv.org/html/2607.06772#bib.bib65),[2026](https://arxiv.org/html/2607.06772#bib.bib63)\)or curriculum sampling\(Chenet al\.,[2020a](https://arxiv.org/html/2607.06772#bib.bib62)\)\. In log\-uniform sampling, each inner problem is assigned a horizonNNsampled according to:

p​\(N\)=1N​log⁡\(Nmax/Nmin\),N∈\(Nmin,Nmax\]\.p\(N\)=\\frac\{1\}\{N\\log\(N\_\{\\max\}/N\_\{\\min\}\)\},\\qquad N\\in\(N\_\{\\min\},N\_\{\\max\}\]\.\(5\)In curriculum sampling, training starts from an initial horizonNinitN\_\{\\mathrm\{init\}\}and increases it over time:

Nj=min⁡\(Nmax,Ninit\+⌊jNperiod⌋​ΔN\),N\_\{j\}=\\min\\left\(N\_\{\\max\},N\_\{\\mathrm\{init\}\}\+\\left\\lfloor\\frac\{j\}\{N\_\{\\mathrm\{period\}\}\}\\right\\rfloor\\Delta\_\{N\}\\right\),\(6\)wherejjdenotes the number of inner problem resets,NperiodN\_\{\\mathrm\{period\}\}controls how often the horizon is increased, andΔN\\Delta\_\{N\}is the horizon increment\. Larger values ofNmaxN\_\{\\max\}, or more aggressive curriculum schedules, expose the LO to very long horizons, but also risk instability, similar to ELO\-small\_fc\(w/o expert\) inFigure[2](https://arxiv.org/html/2607.06772#S2.F2)\(B\)\. Therefore, in practice, relatively conservative settings are commonly used \(Figure[2](https://arxiv.org/html/2607.06772#S2.F2)\(A\)\)\.

On the other hand, these strategies constantly start an unroll from random initialization, which our analysis shows to be inefficient\. Consider a case where the LO makes an update errorϵn\\epsilon\_\{n\}at inner stepnn, then the state reached at a later stepm\>nm\>ndepends on the accumulated effect of all previous errors:

θm−θm⋆≈∑n=0m−1Cn,m​ϵn,\\theta\_\{m\}\-\\theta\_\{m\}^\{\\star\}\\approx\\sum\_\{n=0\}^\{m\-1\}C\_\{n,m\}\\epsilon\_\{n\},\(7\)whereθm⋆\\theta\_\{m\}^\{\\star\}represents the ideal optimal state at stepmm, andCn,mC\_\{n,m\}describes how an error made at stepnnpropagates to the state at stepmm\. Consequently, shorter horizons usually provide a cleaner meta\-gradient signal due to smaller accumulated error along the inner trajectory, which further allows the optimizer to more easily master the early stages of optimization\. As a result, once early\-stage behavior has been learned, random restarts continue to allocate a disproportionate fraction of the meta\-training budget to these already well\-covered regions, leaving later, failure\-prone regimes under\-trained\.

![Refer to caption](https://arxiv.org/html/2607.06772v1/x3.png)Figure 3:Ablation of expert choice and resume buffer\.Downstream validation loss on GPT\-2 124M pretraining\.\(A\)The best expert depends on the LO architecture: AdamW works best for the element\-wisesmall\_fc, while Muon works best for the matrix\-based Celo2\.\(B\)Progressive expert supervision substantially improves the generalization of ELO\-small\_fcto language modeling tasks, while the resume buffer further improves its long\-unroll optimization performance\.

## 3ELO: Efficient Long\-hOrizon learning

In this section, we describe how ELO addresses the imbalance mentioned inSec\.[2](https://arxiv.org/html/2607.06772#S2.SS0.SSS0.Px1)and how its components interact to ensure an efficient and stable long\-horizon meta\-training loop\. A pseudocode description of ELO can be found inAlgorithm[1](https://arxiv.org/html/2607.06772#algorithm1)\.

### 3\.1Failure\-Aware Resume Buffer

![Refer to caption](https://arxiv.org/html/2607.06772v1/x4.png)Figure 4:Expert supervision reduces meta\-gradient variance and improves the signal\-to\-noise ratio of theΔ\\Deltameta\-loss between antithetic PES perturbations \(SeeSec\.[2](https://arxiv.org/html/2607.06772#S2)\), leading to cleaner meta\-gradient estimates\.During meta\-training, we maintain a failure\-aware resume buffer that provides difficult restart states from ongoing trajectories for future unrolls\. Specifically, we identify these regions from the loss dynamics of the current trajectory\. Given inner lossesℓ0,ℓ1,…,ℓN\\ell\_\{0\},\\ell\_\{1\},\\ldots,\\ell\_\{N\}, define the one\-step loss improvementΔn=ℓn−1−ℓn\\Delta\_\{n\}=\\ell\_\{n\-1\}\-\\ell\_\{n\}\. Positive values indicate progress, while non\-positive values indicate stagnation or divergence\. We convert this local signal into a cumulative difficulty score,Vn=max⁡\(0,Vn−1−Δn\)V\_\{n\}=\\max\(0,V\_\{n\-1\}\-\\Delta\_\{n\}\), withV0=0V\_\{0\}=0\. Values close to0indicate consistent optimization progress, while large positive values indicate regions of poor optimization\. We choose the hardest point asn⋆=arg⁡maxn⁡Vnn^\{\\star\}=\\arg\\max\_\{n\}V\_\{n\}, and store the state one truncation window before this step,Kpush=max⁡\(0,n⋆−R\)K\_\{\\mathrm\{push\}\}=\\max\(0,n^\{\\star\}\-R\)\. This process ensures the next unroll will pass through the onset of the difficult region rather than resume after it\. At the beginning of each new unroll, we resume from the buffered state with probabilityPℬP\_\{\\mathcal\{B\}\}and otherwise start from a fresh random initialization\. This shifts a controlled fraction of meta\-training compute from already well\-covered early stages to later regimes where long\-horizon failures emerge\. We set the buffer size to one by default, both for simplicity and to avoid stale states\.

### 3\.2Progressive Expert Supervision

When setting a large resume probabilityPℬP\_\{\\mathcal\{B\}\}, we observe unstable meta\-training dynamics \(ELO\-small\_fc \(w/o buffer\) inFigure[2](https://arxiv.org/html/2607.06772#S2.F2)\(B\)\)\. This is analogous to the long\-trajectory learning difficulty observed in reinforcement learning\(Suttonet al\.,[1998](https://arxiv.org/html/2607.06772#bib.bib69); Rosset al\.,[2011](https://arxiv.org/html/2607.06772#bib.bib11)\)\. In our case, a largePℬP\_\{\\mathcal\{B\}\}can expose the LO to very long unrolls early in meta\-training, where the LO has not yet built a reliable policy\. Its update errors then compound along the inner trajectory, causing the late\-stage inner loss to grow exponentially and finally diverge\. This further makes the meta\-gradients estimated by PES dominated by high variance noise \(ELO\-small\_fc \(w/o buffer\) inFigure[4](https://arxiv.org/html/2607.06772#S3.F4)\), which eventually destabilizes meta\-training and, in severe cases, can lead to collapse\.

To stabilize meta\-training under a largePℬP\_\{\\mathcal\{B\}\}and encourage task\-agnostic generalization learning, we regularize the LO with a reliable expert optimizer during early meta\-training and progressively relax this constraint as the LO improves\.

#### Trajectory fusion\.

For any inner stepnn, we fuse the trajectory produced by the expert and the one produced by the LO in the following way:

θn\+1ℰ=θn\+Δ​θnℰ;θn\+1𝒪=θn\+Δ​θn𝒪;\\displaystyle\\theta\_\{n\+1\}^\{\\mathcal\{E\}\}=\\theta\_\{n\}\+\\Delta\\theta\_\{n\}^\{\\mathcal\{E\}\};\\;\\;\\;\\;\\theta\_\{n\+1\}^\{\\mathcal\{O\}\}=\\theta\_\{n\}\+\\Delta\\theta\_\{n\}^\{\\mathcal\{O\}\};\\;\\;\\;\\;\(8\)θn\+1=\(1−αt\)​θn\+1ℰ\+αt​θn\+1𝒪,\\theta\_\{n\+1\}=\(1\-\\alpha\_\{t\}\)\\,\\theta\_\{n\+1\}^\{\\mathcal\{E\}\}\+\\alpha\_\{t\}\\theta\_\{n\+1\}^\{\\mathcal\{O\}\},\(9\)whereℰ\\mathcal\{E\}and𝒪\\mathcal\{O\}denote the expert and the LO, respectively\. As meta\-training progresses, the LO’s updates become increasingly reliable; the scheduleαt\\alpha\_\{t\}correspondingly shifts weight from the expert to the LO, keeping the inner trajectory high\-quality throughout meta\-training\.

![Refer to caption](https://arxiv.org/html/2607.06772v1/x5.png)Figure 5:ImageNet\-1K training from scratch\.Top\-1 validation accuracy for ViT\-B/16 \(Left\) and ResNet\-50 \(Right\) trained on ImageNet\-1K at224×224224\\times 224resolution for 50K steps with batch size 2048\. ELO\-trained LOs consistently improve over their corresponding LO baselines and outperform the AdamW baseline on both architectures\. Insets zoom in on the final training stage\.
#### Decoupled Progressive Expert Supervision\.

When applying mixed trajectories, meta\-gradients at the beginning of meta\-training become uninformative, as they are estimated from pure expert trajectories \(αt=0\\alpha\_\{t\}=0\), rather than trajectories induced by the LO itself\. To address this, we leverage expert supervision to provide clean gradient signals for early meta\-training\. A natural choice is to imitate expert updates with a standard MSE loss; however, this compresses two distinct aspects of an update, its direction and its magnitude, into a single scalar objective\. We hypothesize that imitating the direction and magnitude of expert updates carries explicitly distinct importance \(verified inFigure[7](https://arxiv.org/html/2607.06772#A4.F7)\)\. We decouple the imitation objective into a weighted combination of direction supervision and magnitude supervision, yielding the following learning objective:

ℒnimt=λ​ℒndir\+\(1−λ\)​ℒnmag;ℒndir=1−Δ​θnℰ⋅Δ​θn𝒪‖Δ​θnℰ‖2​‖Δ​θn𝒪‖2;ℒnmag=\|‖Δ​θnℰ‖2−‖Δ​θn𝒪‖2\|\.\\mathcal\{L\}^\{\\mathrm\{imt\}\}\_\{n\}=\\lambda\\mathcal\{L\}\_\{n\}^\{\\mathrm\{dir\}\}\+\(1\-\\lambda\)\\mathcal\{L\}\_\{n\}^\{\\mathrm\{mag\}\};\\;\\;\\mathcal\{L\}\_\{n\}^\{\\mathrm\{dir\}\}=1\-\\frac\{\\Delta\\theta\_\{n\}^\{\\mathcal\{E\}\}\\cdot\\Delta\\theta\_\{n\}^\{\\mathcal\{O\}\}\}\{\\left\\\|\\Delta\\theta\_\{n\}^\{\\mathcal\{E\}\}\\right\\\|\_\{2\}\\left\\\|\\Delta\\theta\_\{n\}^\{\\mathcal\{O\}\}\\right\\\|\_\{2\}\};\\;\\;\\mathcal\{L\}\_\{n\}^\{\\mathrm\{mag\}\}=\\left\|\\left\\\|\\Delta\\theta\_\{n\}^\{\\mathcal\{E\}\}\\right\\\|\_\{2\}\-\\left\\\|\\Delta\\theta\_\{n\}^\{\\mathcal\{O\}\}\\right\\\|\_\{2\}\\right\|\.We define the meta\-loss asℒnmeta=\(1−αt\)​ℒnimt\+αt​ℒntask\\mathcal\{L\}^\{\\mathrm\{meta\}\}\_\{n\}=\(1\-\\alpha\_\{t\}\)\\,\\mathcal\{L\}^\{\\mathrm\{imt\}\}\_\{n\}\+\\alpha\_\{t\}\\,\\mathcal\{L\}^\{\\mathrm\{task\}\}\_\{n\}, whereαt\\alpha\_\{t\}follows the same schedule as inEq\.[8](https://arxiv.org/html/2607.06772#S3.E8)\. Whenαt=0\\alpha\_\{t\}=0, the objective reduces to pure expert supervision\. Asαt\\alpha\_\{t\}increases, the objective continuously interpolates between imitation and task optimization\. Atαt=1\\alpha\_\{t\}=1, the LO is trained only through task\-driven loss, allowing it to improve beyond the expert policy\.

Table 1:Best validation accuracy across benchmarks\.Top\-1 accuracy on ImageNet\-1K validation, ImageNet\-ReaL, and ImageNet\-V2 for ResNet\-50 and ViT\-B/16 trained from scratch\. The best value in each column is inbold\.ResNet\-50ViT\-B/16MethodIN\-1KIN\-RealIN\-V2IN\-1KIN\-RealIN\-V2Avg\.AdamW75\.8882\.1963\.4074\.1680\.8761\.3972\.98small\_fc65\.3673\.7053\.4331\.7736\.7025\.5747\.75CL\-small\_fc72\.2579\.5359\.9367\.4674\.7454\.9468\.14Celo274\.0480\.9161\.9374\.9781\.1062\.0272\.50ELO\-small\_fc\(Ours\)\{\}\_\{\(\\text\{Ours\}\)\}73\.9281\.0062\.1175\.1881\.7061\.9872\.65ELO\-Celo2\(Ours\)\{\}\_\{\(\\text\{Ours\}\)\}76\.2982\.9364\.2976\.3382\.2163\.1074\.19

## 4Empirical Evaluation

We describe the main experimental settings for evaluating ELO, with complete details provided in Appendix[D](https://arxiv.org/html/2607.06772#A4)\.

#### Baselines\.

We adopt two representative MLP\-based LOs,small\_fc\(Metzet al\.,[2022a](https://arxiv.org/html/2607.06772#bib.bib9)\)and Celo2\(Moudgilet al\.,[2026](https://arxiv.org/html/2607.06772#bib.bib63)\)\. For each architecture, we compare the ELO\-trained variant, ELO\-small\_fcand ELO\-Celo2, with its non\-ELO counterpart\. We also compare against CL\-small\_fc, a curriculum baseline with alternating offline imitation learning inspired by\(Chenet al\.,[2020a](https://arxiv.org/html/2607.06772#bib.bib62)\), and strong hand\-designed optimizers including AdamW\(Loshchilov and Hutter,[2017](https://arxiv.org/html/2607.06772#bib.bib27)\)and Muon\(Jordanet al\.,[2024](https://arxiv.org/html/2607.06772#bib.bib47)\)\.

Meta\-training\.Our experimental setup and meta\-training pipeline largely follow\(Metzet al\.,[2022a](https://arxiv.org/html/2607.06772#bib.bib9); Thérienet al\.,[2024](https://arxiv.org/html/2607.06772#bib.bib4); Moudgilet al\.,[2025](https://arxiv.org/html/2607.06772#bib.bib65)\)with tuned hyperparameters applied for each baseline\. All LOs are meta\-trained on four simple8×88\\times 8image classification tasks: MNIST, Fashion\-MNIST, CIFAR\-10, and SVHN\. The optimizee is a one\-layer MLP with width 32\. We meta\-train for 100K outer steps with batch size 64, using AdamW as the meta\-optimizer with a cosine learning rate schedule\. We search the outer learning rate over\{10−3,3×10−4,10−4\}\\\{10^\{\-3\},3\\times 10^\{\-4\},10^\{\-4\}\\\}and use a weight decay of10−410^\{\-4\}\. We estimate meta\-gradients with Persistent Evolution Strategies \(PES\), using a truncation length ofK=50K=50\. We also apply task augmentation with range of \[0\.001, 1000\] during meta\-training, following\(Moudgilet al\.,[2025](https://arxiv.org/html/2607.06772#bib.bib65)\)\.

#### Downstream Evaluation\.

We evaluate ELO on realistic vision and language model pre\-training tasks\. For vision experiments, We train on ResNet\-50\(Heet al\.,[2016](https://arxiv.org/html/2607.06772#bib.bib23)\)and ViT\-Base/16\(Dosovitskiy,[2020](https://arxiv.org/html/2607.06772#bib.bib61)\)on ImageNet\-1K\. For language modeling, we train GPT\-2 models on FineWeb\(Penedoet al\.,[2024](https://arxiv.org/html/2607.06772#bib.bib21)\), including GPT\-2\-small \(124M\) and GPT\-2\-medium \(350M\)\. Our training follows a 20 token per parameter budget\(Hoffmannet al\.,[2022](https://arxiv.org/html/2607.06772#bib.bib70)\)\. All methods use cosine learning rate schedules, and we carefully tune the learning rate and weight decay for each optimizer family\.

### 4\.1Generalization to Practical Pre\-training Tasks

Takeaway: ELO\-LOs meta\-trained at a small scale are able to generalize to models with hundreds of millions of parameters, consistently improving over their corresponding base LOs across multiple benchmarks\. Most notably, ELO\-Celo2 outperforms well\-tuned AdamW across all tasks and remains competitive with Muon on language modeling at different scales\.

All ELO baselines are meta\-trained on four simple vision tasks using a one\-layer MLP with width 32\. We evaluate whether optimizers learned in this small meta\-training setting can transfer to realistic optimization problems\. We compare ELO\-trained LOs with other LO baselines and hand\-designed optimizers on various significantly larger scale vision and language model pre\-training tasks\.

Image classification\.We train ViT\-B/16 and ResNet\-50 models on ImageNet\-1K at224×224224\\times 224resolution for 50K steps with batch size 2048\. We report top\-1 validation accuracy inFigure[5](https://arxiv.org/html/2607.06772#S3.F5)on three validation benchmarks: ImageNet\-1K validation, ImageNet\-ReaL, and ImageNet\-V2\. LOs meta\-trained with ELO achieve stronger performance than all LO baselines using the same architecture and outperform a strong AdamW baseline\. During our augmentation search, we find that LOs require substantially stronger data augmentation than AdamW\. Under the augmentation setting well tuned for AdamW, LOs tend to overfit, while increasing augmentation strength improves their validation performance\. In contrast, AdamW naturally generalizes better under a lighter augmentation, and its performance degrades under stronger augmentation\. We provide details of the augmentation sweep in Appendix[9](https://arxiv.org/html/2607.06772#A4.F9)\.

Language model pre\-training\.We evaluate ELO on realistic out\-of\-distribution language modeling tasks \(Figure[6](https://arxiv.org/html/2607.06772#S4.F6)andFigure[1](https://arxiv.org/html/2607.06772#S1.F1)A\)\. We first consider GPT\-2\-small \(124M\) trained for 2\.5B tokens, where both ELO\-small\_fc and ELO\-Celo2 substantially improve over their corresponding base LOs\. In this setting, ELO\-Celo2 achieves the best performance, with a small advantage over Muon\. We further evaluate on a larger\-scale setting with GPT\-2\-medium \(350M\) trained for 7B tokens\. ELO\-Celo2 continues to significantly outperform AdamW and performs on par with Muon\. These evidences show that ELO\-trained LOs transfer effectively to language modeling tasks that are substantially different from those used during meta\-training\.

### 4\.2ELO Design Analysis

![Refer to caption](https://arxiv.org/html/2607.06772v1/x6.png)Figure 6:GPT\-2 124M pretraining on FineWeb\.Validation loss throughout 2\.5B training tokens for element\-wise and matrix\-based optimizers\. ELO improves both LO architectures, outperforming other LO baselines and slightly exceeding their corresponding hand\-designed counterparts\.Takeaway: ELO works best when the resume buffer and progressive expert supervision are used together: the buffer improves long\-horizon coverage, while expert supervision keeps these harder unrolls stable\. The optimal expert is architecture dependent, with Adam preferred forsmall\_fcand Muon preferred for Celo2\. Decoupling direction and magnitude supervision further outperforms directℓ2\\ell\_\{2\}regression\.

In this section, we conduct ablation and comparison experiments to understand how each design of ELO contributes and how they interact\.

Ablating ELO’s main components\.ELO consists of two main components: the failure\-aware resume buffer and progressive expert supervision\. When applying only the failure\-aware resume buffer, meta\-training becomes unstable \(Figure[2](https://arxiv.org/html/2607.06772#S2.F2)B, “ELO\-small\_fc \(w/o expert\)”\)\. This instability arises because the buffer exposes the LO to long\-horizon inner problems before it has learned reliable updates, causing errors to accumulate along the inner trajectory\. In contrast, using only progressive expert supervision generalizes well to language model pretraining, but does not sufficiently improve long\-horizon optimization \(Figure[3](https://arxiv.org/html/2607.06772#S2.F3)B, “ELO\-small\_fc \(w/o buffer\)”\)\. Combining the two components gives stable meta\-training and strong long\-horizon optimization performance \(Figure[3](https://arxiv.org/html/2607.06772#S2.F3),Figure[1](https://arxiv.org/html/2607.06772#S1.F1), andFigure[5](https://arxiv.org/html/2607.06772#S3.F5)\)\.

Expert optimizer selection\.The choice of expert optimizer is important because the expert defines the update target used during imitation and is the only source of learning signal early on during meta\-training\. We search over several hand\-designed optimizers independently for different LO architectures and compare the downstream performance of resulting optimizers on language model pre\-training with GPT\-2\-124M, trained for 2\.5B tokens\. As shown inFigure[3](https://arxiv.org/html/2607.06772#S2.F3)A, Adam gives the best validation loss when used as the expert for the element\-wisesmall\_fc\. In contrast, Muon works best as the expert for the matrix\-based Celo2 optimizer\. A likely explanation is thatsmall\_fcapplies element\-wise update rules and does not explicitly model matrix operations, making it difficult to imitate Muon’s orthogonalization based update\. Celo2, by contrast, contains matrix\-based operations and can more naturally imitate Muon\-like behavior\.

Imitation objective\.We compare our decoupled imitation objective with directℓ2\\ell\_\{2\}regression between the parameter updates produced by the LO and the expert optimizer\. While directℓ2\\ell\_\{2\}regression ties update direction and scale together, we find that carefully tuning their relative weights substantially reduces meta\-evaluation loss \(Figure[2](https://arxiv.org/html/2607.06772#S2.F2)C\)\. This indicates that direction and scale provide complementary supervision signals and should be balanced separately\.

### 4\.3ELO Hyperparameter Sensitivity

Takeaway: Imitating expert’s update direction is more important than matching its magnitude\. A large buffer probability is preferred in our meta\-training settings\.

ELO introduces two critical hyperparameters: 1\) the buffer resume probabilityPℬP\_\{\\mathcal\{B\}\}, which controls how often unrolls are initialized from buffered states, and 2\) the direction loss weightλ\\lambda, which balances direction and magnitude supervision\. We tune both hyperparameters using ELO\-small\_fcand then use the selected values for the remaining experiments\.

Direction vs\. magnitude\.We setPℬ=0P\_\{\\mathcal\{B\}\}=0to isolating the imitation objective from the buffer mechanism\. We sweep the direction loss weightλ\\lambdaover\{0\.1,0\.2,…,0\.9\}\\\{0\.1,0\.2,\\ldots,0\.9\\\}and select it according to meta\-evaluation loss\. As shown in Appendix[7](https://arxiv.org/html/2607.06772#A4.F7), assigning weight0\.70\.7to the direction term and0\.30\.3to the magnitude term gives the best performance\. This suggests that matching the expert update direction is more important than exactly matching its scale, although magnitude supervision remains useful for controlling the update size\.

Buffer probability\.PℬP\_\{\\mathcal\{B\}\}controls how aggressively meta\-training compute is reallocated toward longer and harder regimes, which is critical for downstream long\-horizon tasks\. Larger values naturally increase the probability of training from later inner states, but can also make meta\-training less stable\. We searchPℬP\_\{\\mathcal\{B\}\}over\{0\.1,…,0\.9\}\\\{0\.1,\\ldots,0\.9\\\}\. To reduce the sweep cost while selecting a value that generalizes to downstream tasks, we conduct a two\-state selection\. We first select three candidates,Pℬ∈\{0\.1,0\.5,0\.8\}P\_\{\\mathcal\{B\}\}\\in\\\{0\.1,0\.5,0\.8\\\}, based on meta\-evaluation loss\. We then select the finalPℬ=0\.8P\_\{\\mathcal\{B\}\}=0\.8according to the validation loss of GPT\-2 \(124M, 2\.5B tokens\) pretraining in Appendix[8](https://arxiv.org/html/2607.06772#A4.F8)\. In practice, smallerPℬP\_\{\\mathcal\{B\}\}is safer at small meta\-training scale, while largerPℬP\_\{\\mathcal\{B\}\}becomes beneficial as meta\-training becomes more stable and can support longer continuations\.

## 5Discussion&Limitation

While we have made our best effort to provide a thorough scientific study and taken the necessary measures to ensure our results are sound, limitations naturally arise from the computational resources available in academia\. In no particular order, our empirical results do not evaluate models beyond 350M scale, LM architectures beyond standard dense transformers \(e\.g\. MoEs, efficient\-attention variants, or hybrid architectures\), or downstream tasks beyond image classification and language modeling\. In addition, while we compare against strong hand\-designed optimizers such as AdamW and Muon, we do not evaluate additional structured or second\-order\-inspired optimizers such as SOAP or Shampoo\. On the implementation side, ELO can introduce additional overhead compared with standard meta\-training recipes \(Appendix[2](https://arxiv.org/html/2607.06772#A3.T2)\) due to constraints of the JAX framework, where more data movement is required across nested functions\. This overhead is more visible on tiny meta\-training tasks, where GPU computation is small and CPU and memory operations become the main bottleneck\. As the task scale increases, however, runtime becomes dominated by GPU computation, and ELO’s relative overhead is expected to decrease substantially\.

## 6Conclusion

We proposed Efficient Long\-hOrizon learning \(ELO\), a novel meta\-training algorithm for learned optimization that combines decoupled progressive expert guidance with a failure\-aware resume buffer to achieve efficient long\-horizon learning with stability and improve downstream performance\. Across various vision and language\-model benchmarks, ELO\-small\_fc and ELO\-Celo2 outperform their corresponding learned\-optimizer baselines, with ELO\-Celo2 also outperforming AdamW and matching Muon on GPT2\-124M/350M pre\-training\. These results suggest that ELO is a promising recipe for scaling learned optimization\.

## References

- Survey of optimization algorithms in modern neural networks\.Mathematics11\(11\),pp\. 2466\.Cited by:[§1](https://arxiv.org/html/2607.06772#S1.p1.1)\.
- M\. Andrychowicz, M\. Denil, S\. Gomez, M\. W\. Hoffman, D\. Pfau, T\. Schaul, B\. Shillingford, and N\. De Freitas \(2016\)Learning to learn by gradient descent by gradient descent\.Advances in neural information processing systems29\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.06772#S1.p1.1),[§1](https://arxiv.org/html/2607.06772#S1.p2.1)\.
- P\. Buzzega, M\. Boschini, A\. Porrello, D\. Abati, and S\. Calderara \(2020\)Dark experience for general continual learning: a strong, simple baseline\.Advances in neural information processing systems33,pp\. 15920–15930\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1)\.
- A\. Chaudhry, A\. Gordo, P\. Dokania, P\. Torr, and D\. Lopez\-Paz \(2021\)Using hindsight to anchor past knowledge in continual learning\.InProceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 6993–7001\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1)\.
- T\. Chen, W\. Zhang, Z\. Jingyang, S\. Chang, S\. Liu, L\. Amini, and Z\. Wang \(2020a\)Training stronger baselines for learning to optimize\.Advances in Neural Information Processing Systems33,pp\. 7332–7343\.Cited by:[§2](https://arxiv.org/html/2607.06772#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px1.p1.1)\.
- T\. Chen, W\. Zhang, J\. Zhou, S\. Chang, S\. Liu, L\. Amini, and Z\. Wang \(2020b\)Training stronger baselines for learning to optimize\.InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6\-12, 2020, virtual,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\. Balcan, and H\. Lin \(Eds\.\),Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.06772#S1.p1.1)\.
- A\. Dosovitskiy \(2020\)An image is worth 16x16 words: transformers for image recognition at scale\.arXiv preprint arXiv:2010\.11929\.Cited by:[Appendix D](https://arxiv.org/html/2607.06772#A4.SS0.SSS0.Px2.p1.4),[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px2.p1.1)\.
- P\. Florence, C\. Lynch, A\. Zeng, O\. A\. Ramirez, A\. Wahid, L\. Downs, A\. Wong, J\. Lee, I\. Mordatch, and J\. Tompson \(2022\)Implicit behavioral cloning\.InConference on robot learning,pp\. 158–168\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px3.p1.5)\.
- J\. Harrison, L\. Metz, and J\. Sohl\-Dickstein \(2022\)A closer look at learned optimization: stability, robustness, and inductive biases\.Advances in Neural Information Processing Systems35,pp\. 3758–3773\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.06772#S1.p2.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 770–778\.Cited by:[Appendix D](https://arxiv.org/html/2607.06772#A4.SS0.SSS0.Px2.p1.4),[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px2.p1.1)\.
- T\. Hester, M\. Vecerik, O\. Pietquin, M\. Lanctot, T\. Schaul, B\. Piot, D\. Horgan, J\. Quan, A\. Sendonaris, G\. Dulac\-Arnold, I\. Osband, J\. Agapiou, J\. Z\. Leibo, and A\. Gruslys \(2018\)Deep q\-learning from demonstrations\.InProceedings of the Thirty\-Second AAAI Conference on Artificial Intelligence \(AAAI\),pp\. 3223–3230\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. Sifre \(2022\)Training compute\-optimal large language models\.External Links:2203\.15556,[Link](https://arxiv.org/abs/2203.15556)Cited by:[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px2.p1.1)\.
- K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. Bernstein \(2024\)Muon: an optimizer for hidden layers in neural networks\.Cited by:[Appendix D](https://arxiv.org/html/2607.06772#A4.SS0.SSS0.Px3.p1.3),[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px1.p1.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[Appendix D](https://arxiv.org/html/2607.06772#A4.SS0.SSS0.Px3.p1.3)\.
- O\. Li, J\. Harrison, J\. Sohl\-Dickstein, V\. Smith, and L\. Metz \(2023\)Variance\-reduced gradient estimation via noise\-reuse in online evolution strategies\.InThirty\-seventh Conference on Neural Information Processing Systems,Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1)\.
- M\. Liu, H\. Zhao, Z\. Yang, J\. Shen, W\. Zhang, L\. Zhao, and T\. Liu \(2021\)Curriculum offline imitating learning\.Advances in Neural Information Processing Systems34,pp\. 6266–6277\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px3.p1.5)\.
- R\. Liu and J\. Zou \(2018\)The effects of memory replay in reinforcement learning\.In2018 56th annual allerton conference on communication, control, and computing \(Allerton\),pp\. 478–485\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1)\.
- I\. Loshchilov and F\. Hutter \(2017\)Decoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[Appendix D](https://arxiv.org/html/2607.06772#A4.SS0.SSS0.Px2.p1.4),[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px1.p1.1)\.
- L\. Metz, C\. D\. Freeman, J\. Harrison, N\. Maheswaranathan, and J\. Sohl\-Dickstein \(2022a\)Practical tradeoffs between memory, compute, and performance in learned optimizers\.InConference on Lifelong Learning Agents \(CoLLAs\),Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1),[Appendix D](https://arxiv.org/html/2607.06772#A4.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2607.06772#S1.p2.1),[§2](https://arxiv.org/html/2607.06772#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.06772#S2.p2.6),[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px1.p2.4)\.
- L\. Metz, J\. Harrison, C\. D\. Freeman, A\. Merchant, L\. Beyer, J\. Bradbury, N\. Agrawal, B\. Poole, I\. Mordatch, A\. Roberts,et al\.\(2022b\)Velo: training versatile learned optimizers by scaling up\.arXiv preprint arXiv:2211\.09760\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.06772#S1.p2.1)\.
- L\. Metz, N\. Maheswaranathan, C\. D\. Freeman, B\. Poole, and J\. Sohl\-Dickstein \(2020\)Tasks, stability, architecture, and compute: training more effective learned optimizers, and using them to train themselves\.External Links:2009\.11243,[Link](https://arxiv.org/abs/2009.11243)Cited by:[§1](https://arxiv.org/html/2607.06772#S1.p2.1)\.
- L\. Metz, N\. Maheswaranathan, J\. Nixon, D\. Freeman, and J\. Sohl\-Dickstein \(2019\)Understanding and correcting pathologies in the training of learned optimizers\.InInternational Conference on Machine Learning,pp\. 4556–4565\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.06772#S1.p1.1),[§2](https://arxiv.org/html/2607.06772#S2.p4.4)\.
- V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski,et al\.\(2015\)Human\-level control through deep reinforcement learning\.nature518\(7540\),pp\. 529–533\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1)\.
- A\. Moudgil, B\. Knyazev, and E\. Belilovsky \(2026\)Celo2: towards learned optimization free lunch\.arXiv preprint arXiv:2602\.19142\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.06772#S1.p1.1),[§1](https://arxiv.org/html/2607.06772#S1.p2.1),[§2](https://arxiv.org/html/2607.06772#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.06772#S2.p2.6),[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px1.p1.1)\.
- A\. Moudgil, B\. Knyazev, G\. Lajoie, and E\. Belilovsky \(2025\)Celo: training versatile learned optimizers on a compute diet\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=SLqJbt4emY)Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1),[Appendix D](https://arxiv.org/html/2607.06772#A4.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2607.06772#S1.p1.1),[§2](https://arxiv.org/html/2607.06772#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px1.p2.4)\.
- A\. Nair, B\. McGrew, M\. Andrychowicz, W\. Zaremba, and P\. Abbeel \(2017\)Overcoming exploration in reinforcement learning with demonstrations\.InIEEE International Conference on Robotics and Automation \(ICRA\),pp\. 6292–6299\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1)\.
- T\. Osa, J\. Pajarinen, G\. Neumann, J\. A\. Bagnell, P\. Abbeel, J\. Peters,et al\.\(2018\)An algorithmic perspective on imitation learning\.Foundations and Trends® in Robotics7\(1\-2\),pp\. 1–179\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px3.p1.5)\.
- G\. Penedo, H\. Kydlíček, A\. Lozhkov, M\. Mitchell, C\. A\. Raffel, L\. Von Werra, T\. Wolf,et al\.\(2024\)The fineweb datasets: decanting the web for the finest text data at scale\.Advances in Neural Information Processing Systems37,pp\. 30811–30849\.Cited by:[Appendix D](https://arxiv.org/html/2607.06772#A4.SS0.SSS0.Px3.p1.3),[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px2.p1.1)\.
- D\. A\. Pomerleau \(1988\)Alvinn: an autonomous land vehicle in a neural network\.Advances in neural information processing systems1\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px3.p1.5)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[Appendix D](https://arxiv.org/html/2607.06772#A4.SS0.SSS0.Px3.p1.3)\.
- N\. Rajaraman, L\. Yang, J\. Jiao, and K\. Ramchandran \(2020\)Toward the fundamental limits of imitation learning\.Advances in Neural Information Processing Systems33,pp\. 2914–2924\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px3.p1.5)\.
- D\. Rolnick, A\. Ahuja, J\. Schwarz, T\. Lillicrap, and G\. Wayne \(2019\)Experience replay for continual learning\.Advances in neural information processing systems32\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1)\.
- S\. Ross, G\. Gordon, and D\. Bagnell \(2011\)A reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the fourteenth international conference on artificial intelligence and statistics,pp\. 627–635\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2607.06772#S3.SS2.p1.2)\.
- T\. Schaul, J\. Quan, I\. Antonoglou, and D\. Silver \(2015\)Prioritized experience replay\.arXiv preprint arXiv:1511\.05952\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1)\.
- S\. Schmitt, J\. J\. Hudson, A\. Zidek, S\. Osindero, C\. Doersch, W\. M\. Czarnecki, J\. Z\. Leibo, H\. Kuttler, A\. Zisserman, K\. Simonyan, and S\. M\. A\. Eslami \(2018\)Kickstarting deep reinforcement learning\.arXiv preprint arXiv:1803\.03835\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1)\.
- R\. Sun \(2020\)Optimization for deep learning: an overview\.Journal of the Operations Research Society of China8\(2\),pp\. 249–294\.Cited by:[§1](https://arxiv.org/html/2607.06772#S1.p1.1)\.
- S\. Sun, Z\. Cao, H\. Zhu, and J\. Zhao \(2019\)A survey of optimization methods from a machine learning perspective\.IEEE transactions on cybernetics50\(8\),pp\. 3668–3681\.Cited by:[§1](https://arxiv.org/html/2607.06772#S1.p1.1)\.
- R\. S\. Sutton, A\. G\. Barto,et al\.\(1998\)Reinforcement learning: an introduction\.Vol\.1,MIT press Cambridge\.Cited by:[§3\.2](https://arxiv.org/html/2607.06772#S3.SS2.p1.2)\.
- B\. Thérien, C\. Joseph, B\. Knyazev, E\. Oyallon, I\. Rish, and E\. Belilovsky \(2024\)μ\\mulo: Compute\-efficient meta\-generalization of learned optimizers\.External Links:2406\.00153Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1),[Appendix D](https://arxiv.org/html/2607.06772#A4.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2607.06772#S1.p1.1),[§1](https://arxiv.org/html/2607.06772#S1.p2.1),[§2](https://arxiv.org/html/2607.06772#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.06772#S2.p3.4),[§4](https://arxiv.org/html/2607.06772#S4.SS0.SSS0.Px1.p2.4)\.
- F\. Torabi, G\. Warnell, and P\. Stone \(2018\)Behavioral cloning from observation\.arXiv preprint arXiv:1805\.01954\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px3.p1.5)\.
- M\. Vecerík, T\. Hester, J\. Scholz, F\. Wang, O\. Pietquin, B\. Piot, N\. Heess, T\. Rothörl, T\. Lampe, and M\. Riedmiller \(2017\)Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards\.arXiv preprint arXiv:1707\.08817\.Note:This is often referred to as “DDPG from Demonstrations” \(DDPGfD\)Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1)\.
- P\. Vicol, L\. Metz, and J\. Sohl\-Dickstein \(2021\)Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies\.InProceedings of the 38th International Conference on Machine Learning, ICML 2021,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 10553–10563\.Cited by:[Statement A\.1](https://arxiv.org/html/2607.06772#A1.Thmtheorem1),[Appendix A](https://arxiv.org/html/2607.06772#A1.p1.16),[Appendix A](https://arxiv.org/html/2607.06772#A1.p1.20),[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.06772#S2.p4.4)\.
- P\. Vicol \(2023\)Low\-variance gradient estimation in unrolled computation graphs with es\-single\.InInternational Conference on Machine Learning,pp\. 35084–35119\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px1.p1.1)\.
- O\. Wichrowska, N\. Maheswaranathan, M\. W\. Hoffman, S\. G\. Colmenarejo, M\. Denil, N\. Freitas, and J\. Sohl\-Dickstein \(2017\)Learned optimizers that scale and generalize\.InInternational conference on machine learning,pp\. 3751–3760\.Cited by:[§1](https://arxiv.org/html/2607.06772#S1.p2.1)\.
- J\. Yang, T\. Chen, M\. Zhu, F\. He, D\. Tao, Y\. Liang, and Z\. Wang \(2023\)Learning to generalize provably in learning to optimize\.External Links:2302\.11085,[Link](https://arxiv.org/abs/2302.11085)Cited by:[§1](https://arxiv.org/html/2607.06772#S1.p2.1)\.
- S\. Zhang and R\. S\. Sutton \(2017\)A deeper look at experience replay\.arXiv preprint arXiv:1712\.01275\.Cited by:[Appendix B](https://arxiv.org/html/2607.06772#A2.SS0.SSS0.Px2.p1.1)\.

## Appendix APES gradients remain unbiased when resuming from the buffer

In the following sections, For the reader’s convenience, we will now restate background from\[[42](https://arxiv.org/html/2607.06772#bib.bib33)\]required to understand our proof\.\[[42](https://arxiv.org/html/2607.06772#bib.bib33)\]derive*Persistent Evolution Strategies*\(PES\) for unrolled computation graphs whose total loss isL​\(𝜽\)=∑t=1TLt​\(𝜽\)L\(\{\\bm\{\\theta\}\}\)=\\sum\_\{t=1\}^\{T\}L\_\{t\}\(\{\\bm\{\\theta\}\}\), where shared parameters𝜽∈ℝP\{\\bm\{\\theta\}\}\\in\\mathbb\{R\}^\{P\}appear at every timestep\. To account for the contribution of each application of𝜽\{\\bm\{\\theta\}\}, they introduce per\-timestep copies𝜽t=𝜽\{\\bm\{\\theta\}\}\_\{t\}=\{\\bm\{\\theta\}\}and stack them asΘ=\(𝜽1,…,𝜽T\)⊤\\Theta=\(\{\\bm\{\\theta\}\}\_\{1\},\\dots,\{\\bm\{\\theta\}\}\_\{T\}\)^\{\\top\}; writingLt​\(𝒔t;𝜽\)L\_\{t\}\(\\bm\{s\}\_\{t\};\{\\bm\{\\theta\}\}\)asLt​\(𝜽1,…,𝜽t\)L\_\{t\}\(\{\\bm\{\\theta\}\}\_\{1\},\\dots,\{\\bm\{\\theta\}\}\_\{t\}\)\(or simplyLt​\(Θ\)L\_\{t\}\(\\Theta\)\) impliesLt​\(⋅\)L\_\{t\}\(\\cdot\)is independent of𝜽τ\{\\bm\{\\theta\}\}\_\{\\tau\}forτ\>t\\tau\>t\. Letϵ=\(ϵ1,…,ϵT\)⊤\\bm\{\\epsilon\}=\(\\bm\{\\epsilon\}\_\{1\},\\dots,\\bm\{\\epsilon\}\_\{T\}\)^\{\\top\}be a matrix of perturbations with entries i\.i\.d\. Gaussian of varianceσ2\\sigma^\{2\}, let⊗\\otimesdenote the Kronecker product, and define𝝃t=∑τ=1tϵτ\\bm\{\\xi\}\_\{t\}=\\sum\_\{\\tau=1\}^\{t\}\\bm\{\\epsilon\}\_\{\\tau\}\. Starting from the full gradient∂L​\(Θ\)∂vec⁡\(Θ\)∈ℝP​T×1\\frac\{\\partial L\(\\Theta\)\}\{\\partial\\operatorname\{vec\}\\left\(\\Theta\\right\)\}\\in\\mathbb\{R\}^\{PT\\times 1\}and using ES,

d​L​\(𝜽\)d​𝜽=∑τ=1T∂L​\(Θ\)∂𝜽τ=\(𝐈⊗𝟏⊤\)​∂L​\(Θ\)∂vec⁡\(Θ\),\\displaystyle\\frac\{dL\(\{\\bm\{\\theta\}\}\)\}\{d\{\\bm\{\\theta\}\}\}=\\sum\_\{\\tau=1\}^\{T\}\\frac\{\\partial L\(\\Theta\)\}\{\\partial\{\\bm\{\\theta\}\}\_\{\\tau\}\}=\(\\mathbf\{I\}\\otimes\\bm\{1\}^\{\\top\}\)\\frac\{\\partial L\(\\Theta\)\}\{\\partial\\operatorname\{vec\}\\left\(\\Theta\\right\)\},\(10\)𝒈PES=\(𝐈⊗𝟏⊤\)​𝔼ϵ​\[1σ2​vec⁡\(ϵ\)​L​\(Θ\+ϵ\)\]\\displaystyle\\bm\{g\}^\{\\text\{PES\}\}=\(\\mathbf\{I\}\\otimes\\bm\{1\}^\{\\top\}\)\\,\\mathbb\{E\}\_\{\\bm\{\\epsilon\}\}\\left\[\\frac\{1\}\{\\sigma^\{2\}\}\\,\\operatorname\{vec\}\\left\(\\bm\{\\epsilon\}\\right\)\\,L\(\\Theta\+\\bm\{\\epsilon\}\)\\right\]\(11\)=1σ2​𝔼ϵ​\[\(∑τ=1Tϵτ\)​L​\(Θ\+ϵ\)\]\.\\displaystyle=\\frac\{1\}\{\\sigma^\{2\}\}\\mathbb\{E\}\_\{\\bm\{\\epsilon\}\}\\left\[\\Big\(\\sum\_\{\\tau=1\}^\{T\}\\bm\{\\epsilon\}\_\{\\tau\}\\Big\)L\(\\Theta\+\\bm\{\\epsilon\}\)\\right\]\.\(12\)This ES approximation is an unbiased estimator of the gradient of the Gaussian\-smoothed objective𝔼ϵ​\[L​\(Θ\+ϵ\)\]\\mathbb\{E\}\_\{\\bm\{\\epsilon\}\}\[L\(\\Theta\+\\bm\{\\epsilon\}\)\], and it decomposes sequentially as𝒈PES=𝔼ϵ​\[∑t=1T𝒈^t,ϵPES\]\\bm\{g\}^\{\\text\{PES\}\}=\\mathbb\{E\}\_\{\\bm\{\\epsilon\}\}\\left\[\\sum\_\{t=1\}^\{T\}\\hat\{\\bm\{g\}\}^\{\\text\{PES\}\}\_\{t,\\bm\{\\epsilon\}\}\\right\]with𝒈^t,ϵPES=1σ2​𝝃t​Lt​\(𝜽1\+ϵ1,…,𝜽t\+ϵt\)\\hat\{\\bm\{g\}\}^\{\\text\{PES\}\}\_\{t,\\bm\{\\epsilon\}\}=\\frac\{1\}\{\\sigma^\{2\}\}\\bm\{\\xi\}\_\{t\}\\,L\_\{t\}\(\{\\bm\{\\theta\}\}\_\{1\}\+\\bm\{\\epsilon\}\_\{1\},\\dots,\{\\bm\{\\theta\}\}\_\{t\}\+\\bm\{\\epsilon\}\_\{t\}\)\. The Monte Carlo PES estimator is then

𝒈^PES=1N​∑i=1N∑t=1T𝒈^t,ϵ\(i\)PES\.\\hat\{\\bm\{g\}\}^\{\\text\{PES\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\sum\_\{t=1\}^\{T\}\\hat\{\\bm\{g\}\}^\{\\text\{PES\}\}\_\{t,\\bm\{\\epsilon\}^\{\(i\)\}\}\.In practice,\[[42](https://arxiv.org/html/2607.06772#bib.bib33)\]use antithetic sampling,

𝒈^PES\-A≈12​σ2​N​∑i=1N∑t=1T𝝃t\(i\)​\(Lt​\(Θ\+ϵ\(i\)\)−Lt​\(Θ−ϵ\(i\)\)\)\.\\hat\{\\bm\{g\}\}^\{\\text\{PES\-A\}\}\\approx\\frac\{1\}\{2\\sigma^\{2\}N\}\\sum\_\{i=1\}^\{N\}\\sum\_\{t=1\}^\{T\}\\bm\{\\xi\}\_\{t\}^\{\(i\)\}\\Big\(L\_\{t\}\(\\Theta\+\\bm\{\\epsilon\}^\{\(i\)\}\)\-L\_\{t\}\(\\Theta\-\\bm\{\\epsilon\}^\{\(i\)\}\)\\Big\)\.Finally, they show PES is exactly unbiased for quadratic objectives:

###### Statement A\.1\(PES is unbiased restated from\[[42](https://arxiv.org/html/2607.06772#bib.bib33)\]\)\.

Let𝛉∈ℝP\{\\bm\{\\theta\}\}\\in\\mathbb\{R\}^\{P\}andL​\(𝛉\)=∑t=1TLt​\(𝛉\)L\(\{\\bm\{\\theta\}\}\)=\\sum\_\{t=1\}^\{T\}L\_\{t\}\(\{\\bm\{\\theta\}\}\)\. Suppose that∇𝛉L​\(𝛉\)\\nabla\_\{\{\\bm\{\\theta\}\}\}L\(\{\\bm\{\\theta\}\}\)exists, and assume thatLLis quadratic, so that it is equivalent to its second\-order Taylor series expansion:L\(Θ\+ϵ\)=L\(Θ\)\+vec\(ϵ\)⊤∇vec⁡\(Θ\)L\(Θ\)\+12vec\(ϵ\)⊤∇vec⁡\(Θ\)2L\(Θ\)vec\(ϵ\)L\(\\Theta\+\\bm\{\\epsilon\}\)=L\(\\Theta\)\+\\operatorname\{vec\}\(\\bm\{\\epsilon\}\)^\{\\top\}\\nabla\_\{\\operatorname\{vec\}\(\\Theta\)\}L\(\\Theta\)\+\\frac\{1\}\{2\}\\operatorname\{vec\}\(\\bm\{\\epsilon\}\)^\{\\top\}\\nabla^\{2\}\_\{\\operatorname\{vec\}\(\\Theta\)\}L\(\\Theta\)\\operatorname\{vec\}\(\\bm\{\\epsilon\}\)\. Then,*bias*​\(𝐠^PES\-A\)=𝔼ϵ​\[𝐠^PES\-A\]−∇𝛉L​\(𝛉\)=𝟎\\text\{\\emph\{bias\}\}\(\\hat\{\\bm\{g\}\}^\{\\text\{PES\-A\}\}\)=\\mathbb\{E\}\_\{\\bm\{\\epsilon\}\}\[\\hat\{\\bm\{g\}\}^\{\\text\{PES\-A\}\}\]\-\\nabla\_\{\{\\bm\{\\theta\}\}\}L\(\{\\bm\{\\theta\}\}\)=\\bm\{0\}\.

When meta\-training with ELO, we use an identical gradient estimator𝒈^PES\-A\-Buffer≡𝒈^PES\-A\\hat\{\{\\bm\{g\}\}\}^\{\\text\{PES\-A\-Buffer\}\}\\equiv\\hat\{\{\\bm\{g\}\}\}^\{\\text\{PES\-A\}\}, with the only difference being that some particles may have been resumed from the particle replay buffer\. This has the effect of distancing the subsequent𝜽i\{\\bm\{\\theta\}\}\_\{i\}\. However, by the definition ofΘ\\Thetathe resulting gradient is unaffected by this distancing\.

###### Corollary 4\.2\(Corollary 4\.2 restated: PES gradients from the replay buffer are unbiased\)\.

Mirroring the assumptions from Statement[A\.1](https://arxiv.org/html/2607.06772#A1.Thmtheorem1)and additionally assuming that the full PES state is preserved within the buffer and subsequently resumed frim, then*bias*​\(𝒈^PES\-A\-Buffer\)=𝔼ϵ​\[𝒈^PES\-A\-Buffer\]−∇𝜽L​\(𝜽\)=𝟎\\text\{\\emph\{bias\}\}\(\\hat\{\{\\bm\{g\}\}\}^\{\\text\{PES\-A\-Buffer\}\}\)=\\mathbb\{E\}\_\{\\bm\{\\epsilon\}\}\[\\hat\{\{\\bm\{g\}\}\}^\{\\text\{PES\-A\-Buffer\}\}\]\-\\nabla\_\{\{\\bm\{\\theta\}\}\}L\(\{\\bm\{\\theta\}\}\)=\\bm\{0\}\.

Proof\.Since Statement 4\.1 has no temporal constraint onΘ\\Theta, it directly applies to our case where particles are restarted from the buffer\.∎

## Appendix BExtended Related Work

We now review additional related works beyond those discussed in section[2](https://arxiv.org/html/2607.06772#S2)\.

#### Learned Optimization

Learned optimizers \(LOs\) aim to replace hand\-crafted optimizers with small neural networks\. In incipient work,\[[2](https://arxiv.org/html/2607.06772#bib.bib7)\]show that an LSTM can successfully be used to optimize small deep learning problems to a better loss than SGD or Adam\. Follow\-up research proposed new gradient estimators\[[22](https://arxiv.org/html/2607.06772#bib.bib31),[42](https://arxiv.org/html/2607.06772#bib.bib33),[43](https://arxiv.org/html/2607.06772#bib.bib51),[15](https://arxiv.org/html/2607.06772#bib.bib52)\], introduced new architectures and features\[[20](https://arxiv.org/html/2607.06772#bib.bib8),[19](https://arxiv.org/html/2607.06772#bib.bib9)\], proposed low cost meta\-training recipes\[[25](https://arxiv.org/html/2607.06772#bib.bib65),[24](https://arxiv.org/html/2607.06772#bib.bib63),[39](https://arxiv.org/html/2607.06772#bib.bib4)\], and introduced stabilizing mechanisms to improve genralization\[[6](https://arxiv.org/html/2607.06772#bib.bib5),[9](https://arxiv.org/html/2607.06772#bib.bib53)\]\. Of these works,\[[6](https://arxiv.org/html/2607.06772#bib.bib5)\]is the most closely related to our own as their proposed meta\-training scheme also leverages behaviour cloning\. However, unlike our work, they \(1\) behaviour clone using an alternating optimization scheme instead of adding the behavior cloning term to the meta\-loss and \(2\) they do no use a replay buffer\.

#### Replay Buffers\.

Replay buffers are widely used in reinforcement learning \(RL\)\[[33](https://arxiv.org/html/2607.06772#bib.bib11),[17](https://arxiv.org/html/2607.06772#bib.bib12),[46](https://arxiv.org/html/2607.06772#bib.bib13)\]and continual learning \(CL\)\[[32](https://arxiv.org/html/2607.06772#bib.bib29),[4](https://arxiv.org/html/2607.06772#bib.bib30)\]\. They store trajectories collected during training, enabling the learner to sample from past experiences rather than relying solely on the most recent data\. In RL, this helps break temporal correlations\[[23](https://arxiv.org/html/2607.06772#bib.bib41)\], improves sample efficiency\[[34](https://arxiv.org/html/2607.06772#bib.bib42),[35](https://arxiv.org/html/2607.06772#bib.bib59),[41](https://arxiv.org/html/2607.06772#bib.bib56),[11](https://arxiv.org/html/2607.06772#bib.bib57),[26](https://arxiv.org/html/2607.06772#bib.bib58)\], and stabilize learning, while in CL it plays a key role in alleviating catastrophic forgetting\[[32](https://arxiv.org/html/2607.06772#bib.bib29),[3](https://arxiv.org/html/2607.06772#bib.bib43)\]\. Unlike conventional replay buffers for CL or RL, which typically contain existing training examples, we leverage a particle buffer that contains inner\-problem checkpoints from previous unrolls, allowing existing particles to restart from the buffer instead of random initialization\.

#### Behavior Cloning\.

Behavior Cloning \(BC\)\[[31](https://arxiv.org/html/2607.06772#bib.bib15),[40](https://arxiv.org/html/2607.06772#bib.bib16)\]is one of the most widely used paradigms in imitation learning\[[29](https://arxiv.org/html/2607.06772#bib.bib18),[27](https://arxiv.org/html/2607.06772#bib.bib50),[16](https://arxiv.org/html/2607.06772#bib.bib17)\], where the objective is to approximate an expert policy by directly regressing from observed states to expert actions\. Formally, given a dataset of expert demonstrations𝒟=\{\(xi,ai⋆\)\}i=1N\\mathcal\{D\}=\\\{\(x\_\{i\},a\_\{i\}^\{\\star\}\)\\\}\_\{i=1\}^\{N\}, wherexi∈𝒳x\_\{i\}\\in\\mathcal\{X\}denotes the state andai⋆∈𝒜a\_\{i\}^\{\\star\}\\in\\mathcal\{A\}is the corresponding expert action, the goal is to learn a policyπθ:𝒳→𝒜\\pi\_\{\\theta\}:\\mathcal\{X\}\\to\\mathcal\{A\}, parameterized byθ\\theta, that closely approximates the expert’s behavior\. A simple version of behavior cloning solves the supervised regression problem\[[8](https://arxiv.org/html/2607.06772#bib.bib19)\]:

π^θ=arg⁡minπθ⁡1N​∑i=1N‖πθ​\(xi\)−ai⋆‖22\.\\hat\{\\pi\}\_\{\\theta\}=\\arg\\min\_\{\\pi\_\{\\theta\}\}\\;\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\\\|\\pi\_\{\\theta\}\(x\_\{i\}\)\-a\_\{i\}^\{\\star\}\\right\\\|\_\{2\}^\{2\}\.\(13\)In our work, we leverage progressive behavior cloning primarily as a stabilizer for buffer\-based meta\-training, enabling efficient long\-unroll meta\-training\.

## Appendix CWall\-clock Optimizer steps and meta\-training time

Table 2:Wall\-clock training time of different learned optimization algorithms on a single H100 GPU\.Each algorithm is meta\-trained for100,000100\{,\}000outer steps following the recipes outlined in Sec\.[4](https://arxiv.org/html/2607.06772#S4)\. We report the median per\-step time, computed as the median of consecutive\_runtimedeltas logged on W&B; total time is computed as median×\{\}\\times\{\}steps\.Δ\\Deltais the relative overhead over the same\-family baseline, eithersmall\_fcorCelo2\. Although these runs are measured on H100, our meta\-training tasks are small and the runtime is mainlylimited by CPU operations and data movement, so using more accessible GPUs such as L40S gives nearly unchanged wall\-clock time\. As the per\-step optimizee computation increases, e\.g\., in the Celo2 setting, ELO’s relative overhead becomes substantially smaller\.
## Appendix DExtended Empirical Settings

This section complements sec\.[4](https://arxiv.org/html/2607.06772#S4)from the main manuscript with additional experimental details\.

#### Meta\-training\.

Our experimental setup and meta\-training pipeline largely follow\[[19](https://arxiv.org/html/2607.06772#bib.bib9),[39](https://arxiv.org/html/2607.06772#bib.bib4),[25](https://arxiv.org/html/2607.06772#bib.bib65)\]with best hyperparameters applied for each baseline\. All learned optimizers are meta\-trained on four simple image classification tasks, MNIST, Fashion\-MNIST, CIFAR\-10, and SVHN, resized to8×88\\times 8\. The optimizee is a one\-layer MLP with width 32\. We meta\-train for 100K outer steps with batch size 64, using AdamW as the meta\-optimizer with a cosine learning rate schedule\. We search the outer learning rate over\{10−3,3×10−4,10−4\}\\\{10^\{\-3\},3\\times 10^\{\-4\},10^\{\-4\}\\\}and use a weight decay of10−410^\{\-4\}\. We estimate meta\-gradients with Persistent Evolution Strategies \(PES\), using a truncation length ofK=50K=50\. We also apply task augmentation with range of \[0\.001, 1000\] during meta\-training, following\[[25](https://arxiv.org/html/2607.06772#bib.bib65)\]\.

Forsmall\_fc\_loptand Celo2, we sample the unroll length from\[100,2000\]\[100,2000\]using log\-uniform sampling\. For CL\-small\_fc\_lopt, we use Adam as the expert optimizer\. We evaluate maximum meta\-training unroll lengths of 10K and 30K, and find that 10K is the most stable setting and gives the best downstream performance\. For ELO variants, we sample the base unroll length uniformly from\[100,2000\]\[100,2000\]and use the proposed failure\-aware resume buffer to resume from difficult regions of the current optimization trajectory\. We first evaluate learned optimizer checkpoints on ImageNet\-1K \(32×3232\\times 32\), using a three\-layer MLP with width 128 as the optimizee\. Each optimizer is evaluated for 10K inner steps with batch size 4096\. This evaluation serves as an intermediate meta\-evaluation benchmark before transferring the learned optimizers to larger downstream workloads\.

#### Image classification\.

For realistic vision pre\-training, we evaluate on ImageNet\-1K at224×224224\\times 224resolution using ResNet\-50\[[10](https://arxiv.org/html/2607.06772#bib.bib23)\]and ViT\-Base/16\[[7](https://arxiv.org/html/2607.06772#bib.bib61)\]\. We compare learned optimizers mainly against AdamW\[[18](https://arxiv.org/html/2607.06772#bib.bib27)\], which remains the dominant trend for vision tasks\. Training runs for 50K steps with batch size 2048 under a cosine learning\-rate schedule and weight decay searched from \{0\.1, 0\.01, 0\.001, 0\.0001\}\. For AdamW, we sweep seven learning rates on a logarithmic grid in\[10−2,10−5\]\[10^\{\-2\},10^\{\-5\}\]for ViT\-Base/16 and in\[10−1,10−4\]\[10^\{\-1\},10^\{\-4\}\]for ResNet\-50\. For all learned optimizers, we sweep seven learning rates logarithmically in\[10−2,10−5\]\[10^\{\-2\},10^\{\-5\}\]\. We also sweep the data augmentation strength for each method, ranging from light augmentation \(i\.g\. horizontal flip, random crop\) to stronger combinations \(i\.g\. randomly select six augmentations from a pool of more than twenty operations\)\.

#### Language model pre\-training\.

For language modeling, we compare learned optimizers with both AdamW and matrix\-based optimizers such as Muon\[[13](https://arxiv.org/html/2607.06772#bib.bib47)\], which has shown strong performance for language model pre\-training\. We train GPT\-2 models on FineWeb\[[28](https://arxiv.org/html/2607.06772#bib.bib21)\]at two scales: 124M parameters\[[30](https://arxiv.org/html/2607.06772#bib.bib22)\]for 2\.5B tokens over 19073 steps with batch size 128, and 350M parameters for 7B tokens over 26701 steps with batch size 256, both using sequence length 1024\. This token budget follows the scaling\-law\[[14](https://arxiv.org/html/2607.06772#bib.bib64)\]\. We use a cosine learning rate schedule for all methods\. We search weight decay from \{0\.1, 0\.01, 0\.001, 0\.0001\} for all methods\. For learned optimizer baselines, we sweep seven learning rates logarithmically in\[10−2,10−5\]\[10^\{\-2\},10^\{\-5\}\]\. For AdamW and Muon, we tune the learning rate separately: AdamW is swept over\[10−2,10−5\]\[10^\{\-2\},10^\{\-5\}\], while Muon is swept over\[10−1,10−4\]\[10^\{\-1\},10^\{\-4\}\], both with seven logarithmically spaced values\.

![Refer to caption](https://arxiv.org/html/2607.06772v1/x7.png)Figure 7:Effect of direction and magnitude weights in the imitation objective\. We sweep the weighting between direction loss and magnitude loss and report meta\-validation loss during training\. Direction\-dominant objectives consistently achieve lower meta\-validation loss, while overly large magnitude weights slow convergence and lead to worse final performance\. This supports decoupling update direction and scale rather than using a single regression loss on the full expert update\.![Refer to caption](https://arxiv.org/html/2607.06772v1/x8.png)Figure 8:Effect of the resume probabilityPℬP\_\{\\mathcal\{B\}\}on long\-horizon sampling and downstream performance\. Top: increasingPℬP\_\{\\mathcal\{B\}\}shifts the sampled inner steps toward later regions of the trajectory, with the maximum reached step increasing from 5,153 atPℬ=0\.1P\_\{\\mathcal\{B\}\}=0\.1to 91,221 atPℬ=0\.9P\_\{\\mathcal\{B\}\}=0\.9\. Bottom left: on the ImageNet\-1K32×3232\\times 32meta\-evaluation task, smallerPℬP\_\{\\mathcal\{B\}\}generally gives better meta\-validation loss, yet with one is achieved atPℬ=0\.1P\_\{\\mathcal\{B\}\}=0\.1\. Bottom right: on GPT\-2 pre\-training with 124M parameters on FineWeb for 2\.5B tokens, largerPℬP\_\{\\mathcal\{B\}\}improves validation loss, indicating that practical long\-horizon workloads benefit from more frequent resume\-based sampling\.![Refer to caption](https://arxiv.org/html/2607.06772v1/x9.png)Figure 9:In our setting, learned optimizers achieve their best validation accuracy under stronger augmentation, using 6 randomly selected operations from a pool of more than 20, whereas hand\-designed optimizers perform best under medium augmentation, using 4 randomly selected operations\.