OPTScientist: Multi-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining

# OPTScientist: Multi-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining
Source: [https://arxiv.org/html/2607.20486](https://arxiv.org/html/2607.20486)
Zhongzheng Li1,2,3, Tiancan Feng3, Wenhao Li3, Qingsong Ran3, Shikun Feng3, Xiaoyuan Zhang3, Yue Wang3, Xiaoguang Zhao1 1Institute of Automation, Chinese Academy of Sciences 2University of Chinese Academy of Sciences 3Zhongguancun Academy zhangxiaoyuan@bza\.edu\.cn

###### Abstract

Designing optimizers for modern deep learning remains a challenging scientific problem, requiring the joint consideration of optimization geometry, state dynamics, numerical stability, implementation constraints, and empirical generalization\. Existing automated optimizer discovery methods typically search either over unconstrained code spaces or within narrowly parameterized optimizer families\. The former is flexible but often produces invalid or uninterpretable programs, while the latter is stable but limits novelty\. We introduce OPTScientist, a theory\-guided multi\-agent framework for optimizer discovery in a typed domain\-specific language \(DSL\)\. OPTScientist formulates optimizer design as a constrained scientific search process, where candidate updates are expressed through direction, scaling, preconditioning, regularization, state, and grouping modules\. Four role agents, Theorist, Designer, Engineer, and Reviewer, collaborate within a single orchestration loop to propose hypotheses, synthesize DSL candidates, compile and evaluate optimizers, and critique results\. To overcome the limitations of a fixed search space, OPTScientist combines evolutionary search over optimizer programs with a second\-stage mechanism that proposes small DSL extensions when repeated failures reveal representational bottlenecks\. Using this framework, we discover RS\-MR, a reduced\-state matrix optimizer that improves transformer pretraining over strong baselines under our native evaluation protocol\. Our results suggest a path toward automated optimizer science grounded in theory, typed programs, compiler validation, and closed\-loop experimentation\.

## 1Introduction

Optimizers are central to modern deep learning systems\(Kingma and Ba,[2014](https://arxiv.org/html/2607.20486#bib.bib33); Loshchilov and Hutter,[2019](https://arxiv.org/html/2607.20486#bib.bib6)\)\. They determine training speed, stability, memory consumption, final validation performance, and the feasibility of scaling models under fixed compute budgets\. Despite substantial progress, optimizer design remains largely expert\-driven\(Zhaoet al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib2); Liuet al\.,[2023](https://arxiv.org/html/2607.20486#bib.bib1); Vyaset al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib3); Zhanget al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib4); Liuet al\.,[2025a](https://arxiv.org/html/2607.20486#bib.bib5)\)\. Methods such as AdamW, Lion, Muon, and Shampoo encode different assumptions about update geometry, state accumulation, scaling, preconditioning, and regularization, but these assumptions are usually introduced through a mixture of intuition, partial theory, and extensive empirical iteration\.

The difficulty is that an optimizer is not merely a formula mapping gradients to updates\. It is a coupled dynamical system involving stochastic gradients, historical states, adaptive scales, numerical safeguards, parameter grouping rules, and implementation constraints\. Small design changes can substantially alter convergence and stability\. As a result, the optimizer design space is large, but only a small subset is scientifically meaningful, computationally feasible, and robust under realistic training protocols\(Liuet al\.,[2025b](https://arxiv.org/html/2607.20486#bib.bib10); Wenet al\.,[2025](https://arxiv.org/html/2607.20486#bib.bib11)\)\.

Automated optimizer discovery offers a promising alternative, but existing approaches face a tension between expressivity and reliability\(Boikoet al\.,[2023](https://arxiv.org/html/2607.20486#bib.bib34); Baeket al\.,[2025](https://arxiv.org/html/2607.20486#bib.bib35)\)\. Unconstrained program search or language\-model\-based code generation can explore broad update rules, but often produces invalid, unstable, redundant, or difficult\-to\-interpret candidates\. Conversely, searching within a fixed optimizer family improves reliability, but restricts discovery to local variants of known methods\. What is missing is a framework that preserves expert\-like theoretical priors while making optimizer design executable, searchable, auditable, and extensible\(Ghafarollahi and Buehler,[2025](https://arxiv.org/html/2607.20486#bib.bib36); Honget al\.,[2023](https://arxiv.org/html/2607.20486#bib.bib37)\)\.

We introduceOPTScientist, a theory\-guided multi\-agent framework for discovering optimizers as typed programs\. Rather than generating arbitrary Python code,OPTScientistsearches within a typed optimizer domain\-specific language \(DSL\), where candidate updates are expressed through direction, scaling, preconditioning, regularization, state, and grouping modules\. A compiler parses each candidate, checks tensor and scalar compatibility, lowers it into an intermediate representation, and generates an executable optimizer\. This compiler\-backed representation rejects many invalid candidates before training and keeps discovered optimizers compact and inspectable\.

At the search level,OPTScientistorganizes optimizer discovery as a scientific loop\. ATheoristproposes search hypotheses and priors from shared memory, aDesignerconverts them into diverse DSL candidates, anEngineercompiles, repairs, and evaluates candidates, and aReviewerscores them for novelty, validity, performance, target\-stage behavior, and scalability\. These roles are coordinated within a single orchestration loop rather than as independent processes\. The resulting workflow turns optimizer discovery from blind mutation into an iterative process of hypothesis generation, program synthesis, execution, critique, and revision\.

A key component ofOPTScientistis two\-stage evolution\. In the first stage, an evolutionary algorithm searches for optimizer programs within the current DSL\. In the second stage, when repeated failures, target\-stage plateaus, or compiler bottlenecks reveal representational limitations, the system proposes conservative DSL extensions, such as safe macro\-like composition primitives\. Thus,OPTScientistevolves not only optimizer candidates but also, when necessary, the language used to express them\.

Using this framework, we discoverReduced\-State MAGMA\-RowNorm\(RS\-MR\), a low\-state matrix optimizer for transformer weight matrices\. RS\-MR combines row\-normalized polar directions, lightweight global damping, and soft blockwise update gating\. Unlike heavier preconditioned methods, it improves matrix update selectivity with only a compact block\-level state\. In our native transformer pretraining benchmark, RS\-MR consistently outperforms strong baselines such as Muon while remaining close to Muon’s optimizer\-state memory footprint\.

Our contributions are as follows:

- •We propose a typed\-program formulation of automated optimizer discovery, where candidate optimizers are expressed in a compiler\-backed DSL rather than generated as unrestricted Python code\.
- •We developOPTScientist, a theory\-guided multi\-agent framework that coordinates hypothesis generation, DSL candidate synthesis, compilation, evaluation, reviewer scoring, and conservative DSL evolution within an auditable scientific search loop\.
- •We present RS\-MR, a discovered reduced\-state matrix optimizer for transformer pretraining, and demonstrate that it achieves better native multi\-stage validation performance than strong baselines while preserving a favorable memory–performance tradeoff\.

## 2Related Work

### 2\.1Optimizers for Transformer Pretraining

Optimizer design has been central to large\-scale neural network training\. Classical first\-order methods such as SGD with momentum and Nesterov acceleration provide simple and robust update rules, while adaptive methods such as Adam and AdamW are widely used for transformer pretraining due to their stability and ease of tuning\(Kingma and Ba,[2014](https://arxiv.org/html/2607.20486#bib.bib33); Loshchilov and Hutter,[2019](https://arxiv.org/html/2607.20486#bib.bib6)\)\. Memory\-efficient variants such as Adafactor reduce second\-moment memory costs for large language models\(Shazeer and Stern,[2018](https://arxiv.org/html/2607.20486#bib.bib7)\), while recent optimizers such as Lion, Sophia, Shampoo\-style preconditioners, and Muon\-style matrix optimizers explore sign\-based updates, curvature approximations, and matrix\-structured normalization\(Zhaoet al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib2); Liuet al\.,[2023](https://arxiv.org/html/2607.20486#bib.bib1); Vyaset al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib3); Zhanget al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib4); Liuet al\.,[2025a](https://arxiv.org/html/2607.20486#bib.bib5)\)\.

Despite these advances, most optimizers are still manually designed and tuned through extensive empirical validation, requiring expert intuition about update geometry, state design, numerical stability, and hardware efficiency\(Shazeer and Stern,[2018](https://arxiv.org/html/2607.20486#bib.bib7); Yuanet al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib8); Pagliardiniet al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib9)\)\. This challenge is amplified in transformer pretraining by heterogeneous parameter types, long\-horizon training dynamics, and the mismatch between short\-horizon proxy performance and final training behavior\(Liuet al\.,[2025b](https://arxiv.org/html/2607.20486#bib.bib10); Wenet al\.,[2025](https://arxiv.org/html/2607.20486#bib.bib11)\)\.OPTScientistdiffers by treating optimizer design as a searchable typed\-program space and validating candidates through native training; the discovered RS\-MR optimizer shows that reduced\-state matrix mechanisms, such as row\-normalized polar directions and soft blockwise gating, can improve performance without heavy second\-order states\.

### 2\.2Automated Optimizer Discovery and Program Search

Another line of work has studied automated discovery of learning rules and optimization algorithms\. Learned optimizers parameterize update rules with neural networks and train them through meta\-learning\(Andrychowiczet al\.,[2016](https://arxiv.org/html/2607.20486#bib.bib14); Romera\-Paredeset al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib23); Novikovet al\.,[2025](https://arxiv.org/html/2607.20486#bib.bib24)\), but they are often difficult to interpret, expensive to train, and sensitive to the meta\-training distribution\. Another direction searches over symbolic or programmatic spaces, where genetic programming, evolutionary algorithms, and AutoML\-style systems discover update formulas, neural components, or complete algorithms\(Koza,[1994](https://arxiv.org/html/2607.20486#bib.bib15); Realet al\.,[2020](https://arxiv.org/html/2607.20486#bib.bib16); Belloet al\.,[2017](https://arxiv.org/html/2607.20486#bib.bib17)\); symbolic optimizer discovery has also produced practical methods such as Lion\(Chenet al\.,[2023](https://arxiv.org/html/2607.20486#bib.bib12)\)\.

Existing automated discovery methods face a tradeoff between expressivity and reliability\. Unconstrained code search is flexible but prone to invalid programs, poor auditability, and brittle implementation behavior\(Moudgilet al\.,[2025](https://arxiv.org/html/2607.20486#bib.bib18); Marfinetz,[2025](https://arxiv.org/html/2607.20486#bib.bib19)\), whereas narrowly parameterized spaces are easier to validate but restrict discovery to local variants of known mechanisms\(Yanget al\.,[2023](https://arxiv.org/html/2607.20486#bib.bib20); Liuet al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib21); Yeet al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib22)\)\.OPTScientistaddresses this tradeoff by searching inside a typed optimizer DSL that enforces explicit state types, tensor operations, and update semantics while still supporting mechanisms such as momentum, row/column statistics, polar updates, block masking, and structured preconditioning\.

### 2\.3LLM Agents for Scientific Discovery

Large language models are increasingly used as agents that reason, call tools, write code, and iteratively improve solutions\. Early agentic methods combine language\-model reasoning with external actions or feedback, including tool\-use agents, ReAct\-style reasoning, Reflexion\-style self\-improvement, and autonomous exploration systems\(Schicket al\.,[2023](https://arxiv.org/html/2607.20486#bib.bib27); Yaoet al\.,[2022](https://arxiv.org/html/2607.20486#bib.bib28); Shinnet al\.,[2023](https://arxiv.org/html/2607.20486#bib.bib29); Wanget al\.,[2023](https://arxiv.org/html/2607.20486#bib.bib30)\)\. Multi\-agent frameworks decompose complex tasks into specialized roles for planning, implementation, critique, or verification\(Wuet al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib31); Liet al\.,[2023](https://arxiv.org/html/2607.20486#bib.bib32)\), and LLM\-based scientific discovery systems have been used to generate hypotheses, propose code, search over programs, and improve mathematical or algorithmic constructions\(Romera\-Paredeset al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib23); Luet al\.,[2024](https://arxiv.org/html/2607.20486#bib.bib25); Schmidgallet al\.,[2025](https://arxiv.org/html/2607.20486#bib.bib26)\)\.

OPTScientistis related to this line of work but focuses on the constrained and empirically grounded setting of optimizer design for transformer pretraining\. Rather than allowing agents to freely edit arbitrary code, it assigns each role a specific scientific function: theTheoristproposes hypotheses, theDesignerinstantiates DSL programs, theEngineercompiles and evaluates them, and theReviewerscores candidates and decides whether the DSL should be minimally extended\. Compared with general\-purpose LLM agents,OPTScientistemphasizes compiler\-backed validity, native training evaluation, explicit search memory, and controlled DSL evolution\.

## 3Preliminaries

#### Optimizers as modular update programs\.

Modern optimizers differ substantially in their surface forms, but many of them can be understood as different ways of constructing the update tensor inθt\+1=θt−ηt​Ut\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\_\{t\}U\_\{t\}\. For example, SGD chooses the raw gradient or momentum as the update direction, Adam\-style methods add adaptive scaling through second\-moment statistics, Lion uses sign\-based directions, Muon applies matrix\-orthogonalized directions, and Shampoo\-style methods introduce structured preconditioning\. We use this observation to define a structured optimizer design space: an optimizer update can be decomposed into reusable modules for direction, scaling, preconditioning or geometry, regularization, state, and parameter grouping\. This decomposition is not intended to be a unique mathematical factorization; rather, it provides a practical coordinate system for searching over optimizer mechanisms\.

#### Optimizer DSL\.

OPTScientistrepresents each candidate optimizer as a program in a typed domain\-specific language \(DSL\)\. Instead of asking a language model to generate arbitrary Python optimizer code, the system searches over compact DSL programs composed of metadata declarations, hyperparameter declarations, state updates, intermediate expressions, and a final update expression\. At a high level, a DSL program specifies how to computeUtU\_\{t\}from the current gradient, parameter tensor, hyperparameters, and persistent optimizer states\. This design makes optimizer discovery closer to program synthesis over a structured scientific language than to unconstrained code generation\.

#### Atomic optimizer modules\.

The DSL exposes optimizer building blocks at several levels of abstraction\. Direction modules include raw gradients, momentum, sign directions, normalized directions, and matrix\-polar directions\. Scaling modules include constant scaling, RMS\-like scaling, adaptive second\-moment scaling, global damping, and norm\-based rescaling\. Geometry and preconditioning modules include diagonal transformations, row or column normalization, blockwise operations, low\-rank or sketched transformations, and matrix\-factor or polar operations\. Regularization and stabilization modules include decoupled weight decay, clipping, update sanitization, and structured masks\. The DSL also supports different state granularities, ranging from scalar and parameter\-shaped states to row\-wise, column\-wise, blockwise, and matrix\-factor states\.

#### Search space\.

The resulting search space is combinatorial: a candidate optimizer is formed by selecting state variables, choosing update modules, composing tensor operations, setting hyperparameters, and deciding the granularity at which each mechanism is applied\. This space contains many existing optimizer families as special cases, including momentum SGD, RMSProp, AdamW, Lion\-like sign optimizers, Muon\-like matrix optimizers, and Shampoo\-style structured preconditioners\. More importantly, it also contains hybrid mechanisms that combine motifs from different families, such as matrix\-normalized directions with lightweight blockwise gating\. This makes the DSL expressive enough for discovery while still more constrained and auditable than arbitrary source\-code search\.

#### Compiler\-backed validity\.

A DSL candidate is parsed and checked before it is evaluated in training\. The compiler enforces coarse tensor typing, operator constraints, and compatibility between the final update and the target parameter group\. Invalid programs are rejected before expensive training runs\. In our transformer pretraining experiments, discovered DSL optimizers are mainly applied to two\-dimensional matrix parameters, while non\-matrix parameter groups are handled by the baseline optimizer in the hybrid training stack\. This keeps the discovery target focused on matrix\-structured optimizer design, where operations such as row normalization, blockwise gating, and polar directions are most natural\.

## 4Method

![Refer to caption](https://arxiv.org/html/2607.20486v1/figures/optscientist_framework.png)Figure 1:Overview ofOPTScientist\. The framework discovers optimizer programs through a theory\-guided multi\-agent loop\. ATheoristproposes optimizer hypotheses, aDesignerinstantiates them as typed DSL programs, anEngineercompiles and evaluates candidates through transformer pretraining, and aReviewerscores candidates and updates shared memory\. The system alternates between fixed\-DSL optimizer evolution and conservative DSL evolution through safe macro extensions\.We introduceOPTScientist, a theory\-guided multi\-agent framework for discovering optimizer programs for transformer pretraining\. Instead of asking a language model to write arbitrary Python optimizer code,OPTScientistsearches over a typed optimizer DSL\. Each candidate is a compact program specifying optimizer states, tensor operations, hyperparameters, and a final update expression\. The program is compiled into an executable PyTorch optimizer, evaluated in a native training pipeline, reviewed by a role\-specialized agent, and stored in a shared search memory for later rounds\.

### 4\.1Multi\-Agent Search over Typed Optimizer Programs

Given a typed DSLℒ\\mathcal\{L\}, the goal is to find an optimizer programP∈ℒP\\in\\mathcal\{L\}that performs well under a target pretraining protocol\. The system maintains a shared memoryℋ\\mathcal\{H\}containing prior candidates, evaluation outcomes, failure modes, reviewer critiques, and accepted DSL extensions\. Each search round follows the loop*hypothesis*→\\rightarrow*DSL candidates*→\\rightarrow*compiled optimizers*→\\rightarrow*training evaluation*→\\rightarrow*review*→\\rightarrow*memory update*\.

Figure 2:OPTScientistsearch loop\.1:DSL

ℒ0\\mathcal\{L\}\_\{0\}, compiler

𝒞\\mathcal\{C\}, evaluator

ℰ\\mathcal\{E\}, memory

ℋ0\\mathcal\{H\}\_\{0\}
2:

ℒ←ℒ0\\mathcal\{L\}\\\!\\leftarrow\\\!\\mathcal\{L\}\_\{0\},

ℋ←ℋ0\\mathcal\{H\}\\\!\\leftarrow\\\!\\mathcal\{H\}\_\{0\}
3:whilebudget remainsdo

4:

z←Theorist​\(ℋ,ℒ\)z\\leftarrow\\textsc\{Theorist\}\(\\mathcal\{H\},\\mathcal\{L\}\)
5:

𝒫←Designer​\(z,ℋ,ℒ\)\\mathcal\{P\}\\leftarrow\\textsc\{Designer\}\(z,\\mathcal\{H\},\\mathcal\{L\}\)
6:for

P∈𝒫P\\in\\mathcal\{P\}do

7:

\(b,P^,ϵ\)←CompileOrRepair​\(P,𝒞\)\(b,\\widehat\{P\},\\epsilon\)\\leftarrow\\textsc\{CompileOrRepair\}\(P,\\mathcal\{C\}\)
8:

m←ℰ​\(P^\)m\\leftarrow\\mathcal\{E\}\(\\widehat\{P\}\)if

bbelse

Failure​\(ϵ\)\\textsc\{Failure\}\(\\epsilon\)
9:store

\(P,P^,m,ϵ\)\(P,\\widehat\{P\},m,\\epsilon\)
10:endfor

11:

\(S,a\)←Reviewer​\(𝒫,ℋ\)\(S,a\)\\leftarrow\\textsc\{Reviewer\}\(\\mathcal\{P\},\\mathcal\{H\}\)
12:

ℋ←UpdateMemory​\(ℋ,𝒫,S\)\\mathcal\{H\}\\leftarrow\\textsc\{UpdateMemory\}\(\\mathcal\{H\},\\mathcal\{P\},S\)
13:if

a=TriggerDSLEvolutiona=\\textsc\{TriggerDSLEvolution\}then

14:

Δ​ℒ←ProposeMacro​\(ℋ,ℒ\)\\Delta\\mathcal\{L\}\\leftarrow\\textsc\{ProposeMacro\}\(\\mathcal\{H\},\\mathcal\{L\}\)
15:

ℒ←ℒ∪Δ​ℒ\\mathcal\{L\}\\leftarrow\\mathcal\{L\}\\cup\\Delta\\mathcal\{L\}if accepted

16:endif

17:endwhile

18:returnbest final\-validation candidate in

ℋ\\mathcal\{H\}

The four agents implement different scientific roles\. TheTheoristturns prior knowledge and recent failures into optimizer hypotheses, such as using row normalization, blockwise gating, or matrix\-geometric update directions\. TheDesignerconverts these hypotheses into multiple DSL programs and enforces diversity across state choices, normalization paths, and update compositions\. TheEngineercompiles, optionally repairs, and evaluates candidates using the fixed compiler and evaluator; it does not invent execution semantics\. TheReviewerscores candidates by validity, novelty, empirical performance, target\-stage progress, scalability, and resource cost, and decides whether the search should continue within the current DSL or trigger DSL evolution\.

This role decomposition differs from ordinary evolutionary search\. Candidate variation is not an unguided mutation over source code, but a hypothesis\-conditioned synthesis step over typed optimizer programs\. Selection is also not based only on short\-horizon validation loss: the reviewer incorporates compiler validity, target\-stage reachability, cross\-scale behavior, memory overhead, and repeated failure patterns\.

### 4\.2Two\-Stage Evolution

OPTScientistalternates between two stages\. InStage 1, the DSL is fixed and the system searches over optimizer programsP∈ℒP\\in\\mathcal\{L\}\. Each round proposes several candidates, compiles and evaluates them, reviews the results, and updates the shared memory\. This stage is the main optimizer\-discovery loop\.

InStage 2, the system performs conservative DSL evolution\. When repeated failures or plateaus suggest that the current language cannot express a promising mechanism, the reviewer may trigger a small extensionΔ​ℒ\\Delta\\mathcal\{L\}\. In the current implementation, these extensions are safe macro\-like operators expanded before compilation, such as smooth mixing, gating, or stable interpolation between update branches\. The system does not rewrite the compiler or introduce arbitrary grammar changes\. This design allows the search space to grow while preserving compiler\-backed validation and reproducibility\.

### 4\.3Compiler\-Backed Candidate Realization

Every candidate proposed by theDesigneris represented as DSL text, not executable Python code\. The compiler parses the program, checks tensor kinds and operator constraints, lowers it to an intermediate representation, and generates a PyTorch optimizer\. Invalid candidates are rejected before expensive training\. For example, matrix\-only operators must be applied to matrix\-like tensors, row or block states must be compatible with two\-dimensional parameters, and the final update must match the target parameter shape\.

Compiler\-backed realization is central to the framework\. It prevents unverifiable code generation, makes candidates comparable as concise typed programs, and ensures that evaluation depends on executable optimizer behavior rather than language\-model self\-assessment\.

### 4\.4Evaluation and Selection

Candidate optimizers are evaluated through a deterministic multi\-scale transformer pretraining pipeline\. During search, candidates are promoted across progressively larger stages, with the target objective emphasizing reachability and performance at the largest stage rather than only small\-scale proxy score\. The evaluator returns validation performance, completed stages, failed stages, cross\-scale score, optimizer\-state memory, and other runtime artifacts\.

The reviewer converts these raw metrics into an objective\-aware selection score\. In simplified form,S​\(P\)S\(P\)rewards empirical performance, validity, novelty, target\-stage progress, and cross\-scale robustness, while penalizing memory overhead, runtime overhead, numerical instability, duplicate behavior, and evaluator failure\. This score guides parent selection and memory updates\.

Our training setup uses hybrid parameter grouping\. The discovered DSL optimizer is applied to two\-dimensional transformer matrix parameters, where structured matrix updates are most natural\. Embeddings, language\-model heads, vector parameters, and scalar parameters are optimized by AdamW\. Thus, the discovered object should be interpreted as a matrix\-parameter optimizer embedded in a hybrid pretraining stack, not necessarily a full\-model replacement for AdamW\.

### 4\.5Final Validation

Search\-time proxy evaluations are useful for exploration but can favor brittle or memory\-heavy candidates\. Therefore,OPTScientistdistinguishes proxy\-best candidates from final\-validation winners\. The optimizer promoted as the main artifact is selected by native long\-horizon validation rather than by proxy score alone\. This distinction is important in our experiments: the final discovered RS\-MR optimizer is not merely the short\-horizon proxy winner, but the candidate family that provides the best long\-horizon validation and resource tradeoff\.

## 5Discovered Optimizer: RS\-MR

OPTScientistsearches over many typed optimizer programs, but the strongest final design found by the fullOPTScientistdiscovery pipeline is a reduced\-state matrix optimizer\. We call itReduced\-State MAGMA\-RowNorm\(RS\-MR\)\. RS\-MR is not chosen as the best short\-horizon proxy candidate\. It is selected after the full pipeline of hypothesis generation, DSL instantiation, compiler\-backed evaluation, and reviewer\-guided final validation\. This distinction is important: the optimizer promoted in this paper is the one that performs best under native long\-horizon transformer pretraining, not merely under proxy search\.

Figure 3:RS\-MRupdate rule\.1:gradient

gtg\_\{t\}, momentum

mt−1m\_\{t\-1\}, block gate

st−1s\_\{t\-1\}
2:

mt←β1​mt−1\+\(1−β1\)​gtm\_\{t\}\\leftarrow\\beta\_\{1\}m\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)g\_\{t\}
3:

st←0\.9​st−1\+0\.1​σ​\(block​\_​cossim⁡\(mt,gt\)/τ\)s\_\{t\}\\leftarrow 0\.9s\_\{t\-1\}\+0\.1\\sigma\(\\operatorname\{block\\\_cossim\}\(m\_\{t\},g\_\{t\}\)/\\tau\)
4:

ut←polar⁡\(row​\_​norm⁡\(mt\)\)u\_\{t\}\\leftarrow\\operatorname\{polar\}\(\\operatorname\{row\\\_norm\}\(m\_\{t\}\)\)
5:

αt←1\+0\.3​σ​\(rms⁡\(mt2\)/0\.1\)\\alpha\_\{t\}\\leftarrow 1\+0\.3\\sigma\(\\operatorname\{rms\}\(m\_\{t\}^\{2\}\)/0\.1\)
6:

u~t←ut/clipγ⁡\(αt\)\\widetilde\{u\}\_\{t\}\\leftarrow u\_\{t\}/\\operatorname\{clip\}\_\{\\gamma\}\(\\alpha\_\{t\}\)
7:

at←σ​\(block​\_​cossim⁡\(u~t,gt\)/0\.15\)a\_\{t\}\\leftarrow\\sigma\(\\operatorname\{block\\\_cossim\}\(\\widetilde\{u\}\_\{t\},g\_\{t\}\)/0\.15\)
8:return

Ut←\(st⊙at\)⊙u~tU\_\{t\}\\leftarrow\(s\_\{t\}\\odot a\_\{t\}\)\\odot\\widetilde\{u\}\_\{t\}

RS\-MR maintains only two optimizer states: a momentum matrixmtm\_\{t\}and a compact blockwise gating statests\_\{t\}\. Its update rule is summarized in Algorithm[3](https://arxiv.org/html/2607.20486#S5.F3.fig1)\. The design combines three main mechanisms\. First, it uses a Muon\-like matrix direction, but applies row\-wise normalization before the polar operation\. This “normalize\-then\-orthogonalize” step reduces row\-scale imbalance before forming the matrix\-geometric update\. Second, it applies a lightweight RMS\-based damping factor to prevent excessive update magnitudes without introducing a full second\-moment tensor\. Third, it uses a MAGMA\-style soft block gate: the slow gatests\_\{t\}tracks historical blockwise alignment between momentum and gradient, while the fast gateata\_\{t\}measures instantaneous alignment between the candidate update and the current gradient\.

The resulting update is selective rather than uniform\. RS\-MR preferentially updates blocks that are both historically reliable and currently aligned, while suppressing unstable blocks\. This improves over plain Muon not by adding a heavy second\-order preconditioner, but by adding low\-cost structured update selection\. In our training protocol, RS\-MR is applied only to two\-dimensional transformer matrix parameters; embeddings, value embeddings, language\-model heads, vector parameters, and scalar parameters are optimized by AdamW\. Therefore, RS\-MR should be interpreted as a discovered matrix\-parameter optimizer inside a hybrid transformer pretraining stack\.

## 6Experiments

We evaluate whether the optimizer discovered byOPTScientisttransfers from search\-time proxy evaluation to native long\-horizon transformer pretraining\. The main question is whether RS\-MR, selected by the fullOPTScientistdiscovery pipeline in Section Method, remains effective in the realNanoChattraining environment and improves the performance–memory tradeoff over strong baselines\(Karpathy,[2025](https://arxiv.org/html/2607.20486#bib.bib38)\)\.

### 6\.1Dataset and Benchmark

All optimizer variants are trained on the FineWeb\-Edu 100B pretraining corpus, using the Karpathy reshuffled version hosted asfineweb\-edu\-100b\-shuffle\. The corpus is tokenized on the fly with theNanoChattokenizer with vocabulary size3276832768and packed using BOS\-aligned best\-fit cropping\. In our local environment, 370 shards are available: shards 0–368 are used for training and shard 369 is used for validation\. This is more than sufficient for the 3000\-step benchmark, and all optimizer variants are trained on the same data prefix, making the comparison controlled\.

Our main validation benchmark is the native d21NanoChatsetting\. The model has 21 layers, model dimension14081408, 11 attention heads with head dimension128128, context length20482048, and vocabulary size3276832768\. It contains1,099,370,3141\{,\}099\{,\}370\{,\}314total parameters, of which545,722,144545\{,\}722\{,\}144are non\-embedding scaling parameters\. Each run uses 3000 optimization steps with total batch size1,048,5761\{,\}048\{,\}576tokens, corresponding to approximately3\.153\.15B training tokens\. Validation is performed every 250 steps on a held\-out validation slice of approximately 20M tokens\. The metric is validation bits\-per\-byte \(BPB\), where lower is better\.

### 6\.2Baselines and Evaluation Protocol

We compare RS\-MR against representative first\-order, adaptive, and matrix\-oriented optimizers, including AdamW, Muon, Sophia, RMSProp, Lion, and SGD with momentum, as well as other top candidates discovered during search\. All 16 optimizer variants in the native d21 comparison share the same model, data pipeline, tokenizer, batch size, number of steps, and evaluation schedule\. Thus, performance differences primarily reflect optimizer behavior rather than changes in data or training configuration\.

RS\-MR is first discovered through theOPTScientistsearch process and then re\-evaluated in the native d21 benchmark\. This final validation is separate from proxy search\. The goal is to avoid selecting candidates that overfit short\-horizon proxy settings or rely on expensive optimizer states that do not provide a favorable long\-horizon tradeoff\.

### 6\.3Main Results

Table[1](https://arxiv.org/html/2607.20486#S6.T1)reports the main native d21 results\. RS\-MR achieves the best final validation BPB, reaching0\.7983750\.798375\. It outperforms Muon, which reaches0\.8029490\.802949, by0\.0045740\.004574BPB, a relative improvement of about0\.57%0\.57\\%\. It also substantially outperforms AdamW, which reaches0\.8984150\.898415BPB\.

Table 1:Native d21NanoChatresults \(3000 steps,1\.0991\.099B parameters,∼\\sim3\.15 B training tokens on FineWeb\-Edu 100B; lower validation BPB is better\)\. RS\-MR improves over Muon while adding only a small optimizer\-state memory overhead, and clearly beats all single\-state and standard two\-state baselines\.OptimizerFinal Val BPB↓\\downarrowGap to RS\-MRState MemoryRS\-MR \(ours\)0\.798375\\mathbf\{0\.798375\}—1913\.211913\.21MBMuon0\.8029490\.802949\+0\.004574\+0\.0045741905\.761905\.76MBSophia0\.8773530\.877353\+0\.078978\+0\.0789783811\.533811\.53MBAdamW0\.8984150\.898415\+0\.100040\+0\.1000403811\.533811\.53MBRMSProp0\.9016050\.901605\+0\.103230\+0\.1032301905\.761905\.76MBLion0\.9201010\.920101\+0\.121726\+0\.1217261905\.761905\.76MBSGD\+Momentum1\.1493981\.149398\+0\.351023\+0\.3510231905\.761905\.76MBThe memory overhead over Muon is only7\.457\.45MB, or approximately0\.39%0\.39\\%\. Therefore, the gain does not come from a substantially larger optimizer state\. Instead, RS\-MR improves the Muon\-like update by adding row\-normalized matrix directions and compact blockwise update selection\.

### 6\.4Training Dynamics and Pareto Tradeoff

Figure[4](https://arxiv.org/html/2607.20486#S6.F4)shows the validation and training curves\. RS\-MR is better than Muon at every validation checkpoint from 250 to 3000 steps, indicating that the improvement is not a final\-step artifact\. The early training curve also shows that RS\-MR improves the initial optimization trajectory in the nativeNanoChatenvironment\.

![Refer to caption](https://arxiv.org/html/2607.20486v1/figures/nanochat_d21_native_7opt_loss_vs_step.png)\(a\) Validation BPB\.
![Refer to caption](https://arxiv.org/html/2607.20486v1/figures/nanochat_d21_native_7opt_train_loss_first500_vs_step.png)\(b\) First\-500\-step training loss\.

Figure 4:Native d21NanoChattraining dynamics\. RS\-MR maintains a consistent advantage over Muon throughout the 3000\-step validation trajectory and improves early training loss\.Figure[5](https://arxiv.org/html/2607.20486#S6.F5)compares the top\-16 optimizer variants in terms of final validation BPB, optimizer\-state memory, and peak VRAM\. RS\-MR lies in the favorable Pareto region: it achieves the best final validation BPB while remaining close to Muon in both optimizer\-state memory and peak VRAM\. Some discovered candidates obtain competitive BPB, but they require substantially larger states or higher VRAM\. This confirms that RS\-MR is not only the best final\-validation optimizer, but also the most attractive performance–resource tradeoff among the evaluated candidates\.

![Refer to caption](https://arxiv.org/html/2607.20486v1/figures/nanochat_d21_native_pareto_rsmr.png)Figure 5:Top\-16 optimizer comparison on the native d21 benchmark\. RS\-MR achieves the best validation BPB while staying close to Muon in optimizer\-state memory and peak VRAM\.
### 6\.5Summary

The native d21 benchmark shows that the optimizer selected byOPTScientist’s full discovery pipeline transfers to the realNanoChatpretraining setting\. RS\-MR achieves the best final validation BPB, improves over Muon throughout training, substantially outperforms AdamW, and adds only0\.39%0\.39\\%optimizer\-state memory over Muon\. These results support the central claim that typed, compiler\-validated optimizer discovery can find practical optimizer mechanisms that improve long\-horizon transformer pretraining without relying on heavy optimizer states\.

## 7Conclusion and Limitations

We presentedOPTScientist, a theory\-guided multi\-agent framework for discovering typed optimizer programs for transformer pretraining\. By searching over a typed DSL rather than unrestricted optimizer code,OPTScientistcombines language\-model reasoning, compiler\-backed validation, native training evaluation, and reviewer\-guided refinement into an auditable optimizer discovery workflow\.

The strongest discovered design is RS\-MR, a reduced\-state matrix optimizer with row\-normalized polar directions, lightweight RMS damping, and MAGMA\-style soft block gating\. On the native 21\-layerNanoChatbenchmark, RS\-MR achieves the best final validation BPB among the evaluated optimizers, consistently improves over Muon, and adds only a small optimizer\-state memory overhead\. These results show that structured typed\-program search can discover practical optimizer mechanisms beyond short\-horizon proxy winners\.

\(Limitations\) Our current evidence is mainly from transformer pretraining, especially the native 21\-layerNanoChatbenchmark\. Broader validation on larger models, longer training horizons, more datasets, and other architectures is needed before making general claims about optimizer superiority\.

RS\-MR is also a matrix\-parameter optimizer inside a hybrid training stack: non\-matrix parameters are still optimized by AdamW\. In addition, the current DSL\-evolution stage is conservative and supports only safe macro\-like extensions rather than open\-ended compiler or language invention\. Finally, the search still depends on seed programs, prompt priors, evaluator design, and compute, and the theoretical reason why RS\-MR improves over Muon remains an important direction for future work\.

## References

- M\. Andrychowicz, M\. Denil, S\. Gomez, M\. W\. Hoffman, D\. Pfau, T\. Schaul, B\. Shillingford, and N\. De Freitas \(2016\)Learning to learn by gradient descent by gradient descent\.Advances in neural information processing systems29\.Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p1.1)\.
- Researchagent: iterative research idea generation over scientific literature with large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 6709–6738\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p3.1)\.
- I\. Bello, B\. Zoph, V\. Vasudevan, and Q\. V\. Le \(2017\)Neural optimizer search with reinforcement learning\.InInternational Conference on Machine Learning,pp\. 459–468\.Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p1.1)\.
- D\. A\. Boiko, R\. MacKnight, B\. Kline, and G\. Gomes \(2023\)Autonomous chemical research with large language models\.Nature624\(7992\),pp\. 570–578\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p3.1)\.
- X\. Chen, C\. Liang, D\. Huang, E\. Real, K\. Wang, H\. Pham, X\. Dong, T\. Luong, C\. Hsieh, Y\. Lu,et al\.\(2023\)Symbolic discovery of optimization algorithms\.Advances in neural information processing systems36,pp\. 49205–49233\.Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p1.1)\.
- A\. Ghafarollahi and M\. J\. Buehler \(2025\)SciAgents: automating scientific discovery through bioinspired multi\-agent intelligent graph reasoning\.Advanced Materials37\(22\),pp\. 2413523\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p3.1)\.
- S\. Hong, M\. Zhuge, J\. Chen, X\. Zheng, Y\. Cheng, J\. Wang, C\. Zhang, Z\. Wang, S\. K\. S\. Yau, Z\. Lin,et al\.\(2023\)MetaGPT: meta programming for a multi\-agent collaborative framework\.InThe twelfth international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p3.1)\.
- A\. Karpathy \(2025\)Nanochat: the best chatgpt that $100 can buy\.GitHub\.External Links:[Link](https://github.com/karpathy/nanochat)Cited by:[§6](https://arxiv.org/html/2607.20486#S6.p1.1)\.
- D\. P\. Kingma and J\. Ba \(2014\)Adam: a method for stochastic optimization\.arXiv preprint arXiv:1412\.6980\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p1.1)\.
- J\. R\. Koza \(1994\)Genetic programming as a means for programming computers by natural selection\.Statistics and computing4\(2\),pp\. 87–112\.Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p1.1)\.
- G\. Li, H\. Hammoud, H\. Itani, D\. Khizbullin, and B\. Ghanem \(2023\)Camel: communicative agents for" mind" exploration of large language model society\.Advances in neural information processing systems36,pp\. 51991–52008\.Cited by:[§2\.3](https://arxiv.org/html/2607.20486#S2.SS3.p1.1)\.
- F\. Liu, X\. Tong, M\. Yuan, X\. Lin, F\. Luo, Z\. Wang, Z\. Lu, and Q\. Zhang \(2024\)Evolution of heuristics: towards efficient automatic algorithm design using large language model\.arXiv preprint arXiv:2401\.02051\.Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p2.1)\.
- H\. Liu, Z\. Li, D\. Hall, P\. Liang, and T\. Ma \(2023\)Sophia: a scalable stochastic second\-order optimizer for language model pre\-training\.arXiv preprint arXiv:2305\.14342\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p1.1)\.
- J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan,et al\.\(2025a\)Muon is scalable for llm training\.arXiv preprint arXiv:2502\.16982\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p1.1)\.
- L\. Liu, Z\. Xu, Z\. Zhang, H\. Kang, Z\. Li, C\. Liang, W\. Chen, and T\. Zhao \(2025b\)Cosmos: a hybrid adaptive optimizer for memory\-efficient training of llms\.arXiv preprint arXiv:2502\.17410\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p2.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\. arxiv\.arXiv preprint arXiv:1711\.0510110\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p1.1)\.
- C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. Ha \(2024\)The ai scientist: towards fully automated open\-ended scientific discovery\.arXiv preprint arXiv:2408\.06292\.Cited by:[§2\.3](https://arxiv.org/html/2607.20486#S2.SS3.p1.1)\.
- M\. Marfinetz \(2025\)Evolving deep learning optimizers\.arXiv preprint arXiv:2512\.11853\.Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p2.1)\.
- A\. Moudgil, B\. Knyazev, G\. Lajoie, and E\. Belilovsky \(2025\)Celo: training versatile learned optimizers on a compute diet\.arXiv preprint arXiv:2501\.12670\.Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p2.1)\.
- A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. Ruiz, A\. Mehrabian,et al\.\(2025\)Alphaevolve: a coding agent for scientific and algorithmic discovery\.arXiv preprint arXiv:2506\.13131\.Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p1.1)\.
- M\. Pagliardini, P\. Ablin, and D\. Grangier \(2024\)The ademamix optimizer: better, faster, older\.arXiv preprint arXiv:2409\.03137\.Cited by:[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p2.1)\.
- E\. Real, C\. Liang, D\. So, and Q\. Le \(2020\)Automl\-zero: evolving machine learning algorithms from scratch\.InInternational conference on machine learning,pp\. 8007–8019\.Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p1.1)\.
- B\. Romera\-Paredes, M\. Barekatain, A\. Novikov, M\. Balog, M\. P\. Kumar, E\. Dupont, F\. J\. Ruiz, J\. S\. Ellenberg, P\. Wang, O\. Fawzi,et al\.\(2024\)Mathematical discoveries from program search with large language models\.Nature625\(7995\),pp\. 468–475\.Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p1.1),[§2\.3](https://arxiv.org/html/2607.20486#S2.SS3.p1.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.Advances in neural information processing systems36,pp\. 68539–68551\.Cited by:[§2\.3](https://arxiv.org/html/2607.20486#S2.SS3.p1.1)\.
- S\. Schmidgall, Y\. Su, Z\. Wang, X\. Sun, J\. Wu, X\. Yu, J\. Liu, M\. Moor, Z\. Liu, and E\. Barsoum \(2025\)Agent laboratory: using llm agents as research assistants\.Findings of the Association for Computational Linguistics: EMNLP 2025,pp\. 5977–6043\.Cited by:[§2\.3](https://arxiv.org/html/2607.20486#S2.SS3.p1.1)\.
- N\. Shazeer and M\. Stern \(2018\)Adafactor: adaptive learning rates with sublinear memory cost\.InInternational conference on machine learning,pp\. 4596–4604\.Cited by:[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p2.1)\.
- N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[§2\.3](https://arxiv.org/html/2607.20486#S2.SS3.p1.1)\.
- N\. Vyas, D\. Morwani, R\. Zhao, M\. Kwun, I\. Shapira, D\. Brandfonbrener, L\. Janson, and S\. Kakade \(2024\)Soap: improving and stabilizing shampoo using adam\.arXiv preprint arXiv:2409\.11321\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2023\)Voyager: an open\-ended embodied agent with large language models\.arXiv preprint arXiv:2305\.16291\.Cited by:[§2\.3](https://arxiv.org/html/2607.20486#S2.SS3.p1.1)\.
- K\. Wen, D\. Hall, T\. Ma, and P\. Liang \(2025\)Fantastic pretraining optimizers and where to find them\.arXiv preprint arXiv:2509\.02046\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p2.1)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu,et al\.\(2024\)Autogen: enabling next\-gen llm applications via multi\-agent conversations\.InFirst conference on language modeling,Cited by:[§2\.3](https://arxiv.org/html/2607.20486#S2.SS3.p1.1)\.
- C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. Chen \(2023\)Large language models as optimizers\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p2.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2022\)React: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§2\.3](https://arxiv.org/html/2607.20486#S2.SS3.p1.1)\.
- H\. Ye, J\. Wang, Z\. Cao, F\. Berto, C\. Hua, H\. Kim, J\. Park, and G\. Song \(2024\)Reevo: large language models as hyper\-heuristics with reflective evolution\.Advances in neural information processing systems37,pp\. 43571–43608\.Cited by:[§2\.2](https://arxiv.org/html/2607.20486#S2.SS2.p2.1)\.
- H\. Yuan, Y\. Liu, S\. Wu, X\. Zhou, and Q\. Gu \(2024\)Mars: unleashing the power of variance reduction for training large models\.arXiv preprint arXiv:2411\.10438\.Cited by:[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p2.1)\.
- Y\. Zhang, C\. Chen, Z\. Li, T\. Ding, C\. Wu, D\. P\. Kingma, Y\. Ye, Z\. Luo, and R\. Sun \(2024\)Adam\-mini: use fewer learning rates to gain more\.arXiv preprint arXiv:2406\.16793\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p1.1)\.
- J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. Tian \(2024\)Galore: memory\-efficient llm training by gradient low\-rank projection\.arXiv preprint arXiv:2403\.03507\.Cited by:[§1](https://arxiv.org/html/2607.20486#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.20486#S2.SS1.p1.1)\.

OPTScientist: Multi\-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining \(Appendix\)

## Appendix AUnified Optimizer Design Space

This appendix gives a more detailed description of the optimizer design space used to motivate our DSL\. The main paper only introduces the high\-level idea: optimizers are represented as modular update programs\. Here we describe the modules, state structures, granularity choices, and compiler\-backed DSL representation in more detail\.

### A\.1General Update Form

We consider neural network training with parametersθt\\theta\_\{t\}, stochastic gradientgt=∇θℓ​\(θt;ξt\)g\_\{t\}=\\nabla\_\{\\theta\}\\ell\(\\theta\_\{t\};\\xi\_\{t\}\), learning rateηt\\eta\_\{t\}, and optimizer update tensorUtU\_\{t\}\. The generic update is

θt\+1=θt−ηt​Ut\.\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta\_\{t\}U\_\{t\}\.\(1\)The central design question is how to constructUtU\_\{t\}from the current gradient, parameter tensor, and optimizer history\.

We view many optimizers as composing four conceptual modules:

Ut=Dt​\(gt,st\)⊙St​\(gt,st\)⊙Pt​\(gt,st\)\+Rt​\(θt\),U\_\{t\}=D\_\{t\}\(g\_\{t\},s\_\{t\}\)\\odot S\_\{t\}\(g\_\{t\},s\_\{t\}\)\\odot P\_\{t\}\(g\_\{t\},s\_\{t\}\)\+R\_\{t\}\(\\theta\_\{t\}\),\(2\)wherests\_\{t\}denotes optimizer states\. The direction moduleDtD\_\{t\}determines the descent signal, the scaling moduleStS\_\{t\}controls update magnitude, the preconditioning or geometry modulePtP\_\{t\}changes the update geometry, and the regularization moduleRtR\_\{t\}captures weight decay, clipping, masks, or constraints\. This decomposition is not unique, but it is useful for organizing the search space\.

### A\.2Direction Space

The direction module determines where the optimizer moves\. Representative choices include:

Dt\\displaystyle D\_\{t\}=gt\\displaystyle=g\_\{t\}raw gradient,\\displaystyle\\text\{raw gradient\},\(3\)mt\\displaystyle m\_\{t\}=β​mt−1\+\(1−β\)​gt,Dt=mt\\displaystyle=\\beta m\_\{t\-1\}\+\(1\-\\beta\)g\_\{t\},\\quad D\_\{t\}=m\_\{t\}momentum,\\displaystyle\\text\{momentum\},\(4\)Dt\\displaystyle D\_\{t\}=sign⁡\(mt\)\\displaystyle=\\operatorname\{sign\}\(m\_\{t\}\)sign direction,\\displaystyle\\text\{sign direction\},\(5\)Dt\\displaystyle D\_\{t\}=gt‖gt‖\+ϵ\\displaystyle=\\frac\{g\_\{t\}\}\{\\\|g\_\{t\}\\\|\+\\epsilon\}normalized gradient,\\displaystyle\\text\{normalized gradient\},\(6\)Dt\\displaystyle D\_\{t\}=gt−gt−1\\displaystyle=g\_\{t\}\-g\_\{t\-1\}gradient difference,\\displaystyle\\text\{gradient difference\},\(7\)Dt\\displaystyle D\_\{t\}=Ht−1​gt\\displaystyle=H\_\{t\}^\{\-1\}g\_\{t\}second\-order direction\.\\displaystyle\\text\{second\-order direction\}\.\(8\)Matrix optimizers introduce additional direction choices\. For a two\-dimensional matrix parameter, a momentum matrixMtM\_\{t\}may be transformed by a polar or orthogonalization operator:

Dt=polar⁡\(Mt\)\.D\_\{t\}=\\operatorname\{polar\}\(M\_\{t\}\)\.\(9\)This captures the core motif behind Muon\-like update rules\.

### A\.3Scaling Space

The scaling module determines how far to move along the chosen direction\. Common choices include constant scaling, RMS scaling, second\-moment scaling, norm\-based scaling, signal\-to\-noise\-ratio\-like scaling, and curvature\-based damping\. For example, Adam\-style methods maintain

vt=β2​vt−1\+\(1−β2\)​gt2v\_\{t\}=\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)g\_\{t\}^\{2\}\(10\)and use a scaling factor of the form

St=1vt\+ϵ\.S\_\{t\}=\\frac\{1\}\{\\sqrt\{v\_\{t\}\}\+\\epsilon\}\.\(11\)Other optimizers may use global RMS statistics, layer\-wise norms, or scalar damping factors to control the step size without maintaining a full parameter\-shaped second\-moment state\.

### A\.4Preconditioning and Geometry Space

The preconditioning module changes the geometry of the update\. Diagonal preconditioners use elementwise statistics and are the basis of Adagrad, RMSProp, and Adam\-style methods:

Pt=diag⁡\(pt\)\.P\_\{t\}=\\operatorname\{diag\}\(p\_\{t\}\)\.\(12\)Block\-diagonal and Kronecker\-factored preconditioners approximate curvature with structured matrices:

Pt=blockdiag⁡\(H1−1,H2−1,…\),H−1≈A−1⊗B−1\.P\_\{t\}=\\operatorname\{blockdiag\}\(H\_\{1\}^\{\-1\},H\_\{2\}^\{\-1\},\\ldots\),\\qquad H^\{\-1\}\\approx A^\{\-1\}\\otimes B^\{\-1\}\.\(13\)Low\-rank preconditioners approximate curvature with a small subspace:

H−1≈U​Λ−1​U⊤\.H^\{\-1\}\\approx U\\Lambda^\{\-1\}U^\{\\top\}\.\(14\)Matrix\-geometric methods instead transform the update direction through normalization or polar operations\. Examples include row normalization, column normalization, blockwise transformations, and approximate polar decomposition\.

### A\.5Regularization and Stabilization

Regularization and stabilization modules include L2 regularization, decoupled weight decay, clipping, trust\-region\-like constraints, and masks\. Classical L2 regularization adds a termRt​\(θt\)=λ​θtR\_\{t\}\(\\theta\_\{t\}\)=\\lambda\\theta\_\{t\}to the update\. Decoupled weight decay instead applies shrinkage directly:

θt\+1=\(1−ηt​λ\)​θt−ηt​Ut\.\\theta\_\{t\+1\}=\(1\-\\eta\_\{t\}\\lambda\)\\theta\_\{t\}\-\\eta\_\{t\}U\_\{t\}\.\(15\)Stabilization mechanisms such as clipping and sanitization can be viewed as constraints on the update tensor, for example enforcing‖Ut‖≤c\\\|U\_\{t\}\\\|\\leq cor replacing invalid numerical values before applying the update\. Structured masks can further suppress unreliable coordinates, rows, columns, or blocks\.

### A\.6State Space and Granularity

Optimizer states store information from previous steps\. Common states include first moments, second moments, Hessian or curvature estimates, exponential moving averages, gradient\-history buffers, and blockwise reliability statistics\. These states can have different shapes and memory costs:

- •scalar states;
- •vector or parameter\-shaped states;
- •row\-wise or column\-wise matrix states;
- •blockwise matrix states;
- •square matrix\-factor states;
- •global states shared across parameter groups\.

The update itself can also be applied at different granularities, including parameter\-wise, tensor\-wise, layer\-wise, block\-wise, or globally\. This choice strongly affects both memory cost and optimization behavior\.

## Appendix BOptimizer DSL Details

### B\.1Program Structure

The DSL used byOPTScientistis a compact line\-based language for defining optimizer programs\. A program contains metadata declarations, hyperparameter declarations, state declarations, optional intermediate assignments, and a final update expression:

```
meta kind = Mat2D
hparam beta1 = 0.95
hparam tau = 0.15

state m = beta1 * m + (1 - beta1) * grad
u = polar(row_norm(m))
update = u
```

The finalupdateexpression must produce a tensor compatible with the parameter being optimized\. The training loop applies the update; the DSL program does not directly mutate model parameters\.

### B\.2Metadata and Hyperparameters

Themetafields provide typing or backend information\. For example,meta kind = Mat2Dindicates that the program is intended for two\-dimensional matrix parameters\. Other metadata fields can control execution semantics such as masking granularity, state initialization, or numerical safeguards, depending on the backend implementation\.

Hyperparameters are declared explicitly:

```
hparam lr = 0.02
hparam beta1 = 0.95
hparam eps = 1e-8
```

The compiler records these declarations and exposes them as scalar values in the generated optimizer\.

### B\.3State Declarations

The DSL supports several persistent state declarations\. The most common declaration isstate, which creates a parameter\-shaped state:

```
state m = beta1 * m + (1 - beta1) * grad
```

This can express momentum, second moments, or other full\-tensor states\. The DSL also supports reduced or structured state types:

- •state\_row: row\-wise matrix state;
- •state\_col: column\-wise matrix state;
- •state\_block: blockwise matrix state;
- •state\_mat: left square matrix\-factor state;
- •state\_rmat: right square matrix\-factor state;
- •state\_scalar: per\-parameter scalar state;
- •global\_state\_scalar: optimizer\-wide scalar state\.

These declarations allow the search to trade off expressivity and memory\. For example, a blockwise state can encode structural update reliability at much lower cost than a full parameter\-shaped second\-moment state\.

### B\.4Operators

The DSL operator set includes arithmetic operations, reductions, elementwise nonlinearities, normalization functions, matrix operations, and blockwise functions\. Representative operators include:

- •elementwise operations such assquare,sqrt,sign,abs,sigmoid, andclip;
- •scalar reductions such asnorm,rms,max\_val, andmin\_val;
- •row and column statistics such asrow\_mean,row\_rms,col\_mean, andcol\_rms;
- •matrix operations such aspolar,newton\_schulz,transpose,diag, andouter;
- •block operations such asblock\_rms,block\_mask, andblock\_cossim;
- •structured transformations such aslow\_rank,sketch,project\_psd, andblock\_inv\_sqrt\.

This operator library is designed to cover a broad range of optimizer motifs while remaining statically checkable\.

### B\.5Typing and Validation

The compiler distinguishes scalar expressions from tensor expressions and assigns coarse tensor kinds to tensor expressions\. The main tensor kinds includeVec1D,Mat2D,MatRow,MatCol,MatLeftSquare,MatRightSquare, and a genericAnykind\. Operator constraints are checked before evaluation\. For example,polarandnewton\_schulzrequire matrix\-like tensors; row and column reductions require matrix\-compatible inputs; and the final update must be a tensor whose kind is compatible with the target parameter\.

The compiler pipeline is:

P→parseAST​\(P\)→typecheckIR​\(P\)→codegenP^,P\\xrightarrow\{\\mathrm\{parse\}\}\\mathrm\{AST\}\(P\)\\xrightarrow\{\\mathrm\{typecheck\}\}\\mathrm\{IR\}\(P\)\\xrightarrow\{\\mathrm\{codegen\}\}\\widehat\{P\},\(16\)wherePPis the DSL program andP^\\widehat\{P\}is the generated PyTorch optimizer\. This pipeline prevents invalid candidates from reaching the expensive training evaluator and makes discovered optimizers auditable as compact typed programs\.

### B\.6Known Optimizers as DSL Programs

Many known optimizers can be expressed as short DSL programs\. A simplified momentum optimizer is:

```
hparam beta = 0.9
state m = beta * m + (1 - beta) * grad
update = m
```

A simplified Adam\-like update is:

```
hparam beta1 = 0.9
hparam beta2 = 0.999
hparam eps = 1e-8

state m = beta1 * m + (1 - beta1) * grad
state v = beta2 * v + (1 - beta2) * square(grad)

update = m / (sqrt(v) + eps)
```

A simplified Muon\-like matrix update is:

```
meta kind = Mat2D
hparam beta1 = 0.95

state m = beta1 * m + (1 - beta1) * grad
update = polar(m)
```

These examples illustrate why the DSL is useful for search: known optimizers become points in the same program space, while new optimizers can be discovered by recombining modules, states, and tensor operations\.

### B\.7Hybrid Parameter Grouping

In our transformer pretraining experiments, the discovered DSL optimizer is primarily applied to two\-dimensional transformer matrix parameters\. Other parameter groups, such as token embeddings, value embeddings, language\-model heads, vector parameters, and scalar parameters, are optimized by the baseline optimizer used by the training stack\. This hybrid grouping is important for both numerical stability and fair evaluation\. The discovered object should therefore be interpreted as a matrix\-parameter optimizer embedded inside a hybrid training system, rather than as a full\-model replacement for every parameter class\.

## Appendix CAdditional Method Details

### C\.1Shared Memory

The shared memoryℋ\\mathcal\{H\}stores all information needed to make the search iterative rather than a collection of independent trials\. Each candidate record can be written as

ri=\(Pi,P^i,mi,ei,si,ai\),r\_\{i\}=\(P\_\{i\},\\widehat\{P\}\_\{i\},m\_\{i\},e\_\{i\},s\_\{i\},a\_\{i\}\),\(17\)wherePiP\_\{i\}is the DSL program,P^i\\widehat\{P\}\_\{i\}is the compiled optimizer if compilation succeeds,mim\_\{i\}denotes evaluator metrics,eie\_\{i\}denotes compiler or runtime errors,sis\_\{i\}denotes reviewer scores, andaia\_\{i\}denotes agent annotations such as design notes or critiques\.

The memory includes historical best candidates, selection\-best candidates, recent failures, target\-stage failures, plateau indicators, accepted DSL extensions, previous hypotheses, reviewer critiques, candidate lineage, evaluation metrics, stage\-completion flags, and optimizer\-state memory statistics\. TheTheoristuses this memory to identify promising mechanisms and recurring failures; theDesigneruses it to avoid redundant candidates; theEngineeruses it for cache\-aware evaluation and repair; and theRevieweruses it to detect plateaus and decide whether DSL evolution should be triggered\.

### C\.2Agent Interfaces

#### Theorist\.

TheTheoristreadsℋ\\mathcal\{H\}, the current DSLℒt\\mathcal\{L\}\_\{t\}, recent candidate outcomes, and optional paper summaries\. It outputs

ztth=\{ht,πt,ρt,qt,et\},z\_\{t\}^\{\\mathrm\{th\}\}=\\\{h\_\{t\},\\pi\_\{t\},\\rho\_\{t\},q\_\{t\},e\_\{t\}\\\},\(18\)wherehth\_\{t\}is a mechanism hypothesis,πt\\pi\_\{t\}is a set of search priors,ρt\\rho\_\{t\}is a parent\-selection strategy,qtq\_\{t\}is a DSL\-pressure estimate, andete\_\{t\}is an optional extension request\.

#### Designer\.

TheDesignermaps the hypothesis into a set of candidates𝒫t=\{Pt,1,…,Pt,K\}\\mathcal\{P\}\_\{t\}=\\\{P\_\{t,1\},\\ldots,P\_\{t,K\}\\\}, where eachPt,k∈ℒtP\_\{t,k\}\\in\\mathcal\{L\}\_\{t\}\. Candidates are emitted as DSL text\. The system encourages structural diversity across state definitions, tensor operations, normalization rules, preconditioning choices, and gating mechanisms\. If LLM generation fails, the system falls back to deterministic DSL templates\.

#### Engineer\.

TheEngineerinvokes the compiler on each candidate:

Compile​\(Pt,k\)→\(bt,kcomp,P^t,k,ϵt,kcomp\),\\mathrm\{Compile\}\(P\_\{t,k\}\)\\rightarrow\(b\_\{t,k\}^\{\\mathrm\{comp\}\},\\widehat\{P\}\_\{t,k\},\\epsilon\_\{t,k\}^\{\\mathrm\{comp\}\}\),\(19\)wherebt,kcompb\_\{t,k\}^\{\\mathrm\{comp\}\}indicates compile success,P^t,k\\widehat\{P\}\_\{t,k\}is the compiled optimizer, andϵt,kcomp\\epsilon\_\{t,k\}^\{\\mathrm\{comp\}\}is the compiler error\. If compilation fails, the Engineer may perform a bounded repair step using the compiler error\. Successful candidates are passed to the evaluator, which returns metricsmt,km\_\{t,k\}\.

#### Reviewer\.

TheReviewercomputes candidate scores

𝐬​\(P\)=\(snovel,svalid,sperf,starget,sscale\),\\mathbf\{s\}\(P\)=\(s\_\{\\mathrm\{novel\}\},s\_\{\\mathrm\{valid\}\},s\_\{\\mathrm\{perf\}\},s\_\{\\mathrm\{target\}\},s\_\{\\mathrm\{scale\}\}\),\(20\)measuring structural novelty, compiler/runtime validity, empirical performance, target\-stage progress, and cross\-scale robustness\. The reviewer also emits a stage\-2 signal when repeated failures suggest a language\-level bottleneck\.

### C\.3Selection Score

The reviewer combines metrics into an objective\-aware selection score:

S​\(P\)=\\displaystyle S\(P\)=λperf​sperf​\(P\)\+λvalid​svalid​\(P\)\+λnovel​snovel​\(P\)\\displaystyle\\lambda\_\{\\mathrm\{perf\}\}s\_\{\\mathrm\{perf\}\}\(P\)\+\\lambda\_\{\\mathrm\{valid\}\}s\_\{\\mathrm\{valid\}\}\(P\)\+\\lambda\_\{\\mathrm\{novel\}\}s\_\{\\mathrm\{novel\}\}\(P\)\(21\)\+λtarget​starget​\(P\)\+λscale​sscale​\(P\)−λcost​c​\(P\)\.\\displaystyle\+\\lambda\_\{\\mathrm\{target\}\}s\_\{\\mathrm\{target\}\}\(P\)\+\\lambda\_\{\\mathrm\{scale\}\}s\_\{\\mathrm\{scale\}\}\(P\)\-\\lambda\_\{\\mathrm\{cost\}\}c\(P\)\.Herec​\(P\)c\(P\)penalizes memory overhead, runtime overhead, numerical instability, duplicate signatures, evaluator failure, and target\-stage collapse\. The exact weighting can be implementation\-dependent, but the intended behavior is fixed: candidates should be valid, novel, stable across scales, capable of reaching the target stage, and resource\-efficient\.

### C\.4Detailed Compiler Pipeline

A DSL program contains metadata declarations, hyperparameter declarations, persistent state updates, intermediate expressions, and a final update expression\. The compiler pipeline is

P→parseAST​\(P\)→typecheckIR​\(P\)→codegenP^\.P\\xrightarrow\{\\mathrm\{parse\}\}\\mathrm\{AST\}\(P\)\\xrightarrow\{\\mathrm\{typecheck\}\}\\mathrm\{IR\}\(P\)\\xrightarrow\{\\mathrm\{codegen\}\}\\widehat\{P\}\.\(22\)The type checker distinguishes scalar expressions from tensor expressions and assigns coarse structural kinds to tensors, such as vector, matrix, row\-reduced, column\-reduced, blockwise, and matrix\-factor kinds\. Operator constraints are checked before evaluation\. For example,polarrequires matrix\-like tensors, row and column reductions require matrix\-compatible inputs, and the final update must be a tensor compatible with the target parameter group\.

This pipeline reduces wasted evaluator budget and improves auditability\. Instead of inspecting a long generated Python optimizer, one can inspect the concise DSL program and its compiler\-produced optimizer\.

### C\.5Parameter Grouping

Transformer parameters are heterogeneous\. Matrix\-oriented operations, such as polar directions, row/column normalization, structured preconditioning, and blockwise gating, are naturally suited for two\-dimensional matrices but are not always appropriate for embeddings, output heads, vector parameters, or scalar parameters\. Therefore, our evaluation uses hybrid parameter grouping\.

Letθ=θmat∪θaux\\theta=\\theta\_\{\\mathrm\{mat\}\}\\cup\\theta\_\{\\mathrm\{aux\}\}, whereθmat\\theta\_\{\\mathrm\{mat\}\}denotes two\-dimensional transformer matrix parameters andθaux\\theta\_\{\\mathrm\{aux\}\}denotes auxiliary non\-matrix parameters\. The DSL optimizer updates the matrix group as

θmatt\+1=θmatt−η​UP​\(gmatt,st\),\\theta\_\{\\mathrm\{mat\}\}^\{t\+1\}=\\theta\_\{\\mathrm\{mat\}\}^\{t\}\-\\eta U\_\{P\}\(g\_\{\\mathrm\{mat\}\}^\{t\},s^\{t\}\),\(23\)whereUPU\_\{P\}is the update compiled from DSL programPP\. The auxiliary group is updated by AdamW:

θauxt\+1=AdamW​\(θauxt,gauxt\)\.\\theta\_\{\\mathrm\{aux\}\}^\{t\+1\}=\\mathrm\{AdamW\}\(\\theta\_\{\\mathrm\{aux\}\}^\{t\},g\_\{\\mathrm\{aux\}\}^\{t\}\)\.\(24\)This avoids overstating the scope of the discovered optimizer\. The discovered program is a matrix\-parameter optimizer inside a hybrid transformer pretraining stack\.

### C\.6Multi\-Scale Search\-Time Evaluation

During search, candidates are evaluated with a multi\-scale transformer pretraining protocol\. The stages are

ℳ=\{49\.8​M,96\.0​M,201\.1​M,514\.3​M\}\.\\mathcal\{M\}=\\\{49\.8\\mathrm\{M\},96\.0\\mathrm\{M\},201\.1\\mathrm\{M\},514\.3\\mathrm\{M\}\\\}\.\(25\)Each stage uses a fixed training budget and returns validation bits\-per\-byte, stage\-completion flags, failure information, optimizer\-state memory, and other metrics\. Early\-stage scores provide useful feedback, but the target objective emphasizes stability and performance at the largest stage\. This discourages candidates that look strong at small scale but fail during promotion\.

### C\.7Proxy Search and Final Validation

The search loop may use cheaper proxy evaluations for exploration, but the main optimizer artifact is selected by final native validation\. This separation is necessary because proxy\-best candidates can be brittle, memory\-heavy, or specialized to short horizons\. Final validation filters for long\-horizon robustness, memory efficiency, and consistent training behavior\. In our experiments, this protocol selects RS\-MR as the representative discovered optimizer\.

Table 2:All 16 optimizer variants on the native d21NanoChat3000\-step benchmark, ranked by final validation BPB\. RS\-MR is the same DSL program as OE\-1 \(rank\_008\_\_oe\_20260325\_222424\)\. The top 10 OpenEvolve\-discovered DSL optimizers \(OE\-1∼\\simOE\-10\) all outperform every classical baseline\. The number of dense optimizer states is reported as recorded by the DSL compiler; small block\-level states \(e\.g\. a16×1616\{\\times\}16MAGMA mask\) are counted but contribute negligibly to memory\.OptimizerFinal Val BPB↓\\downarrowGap to RS\-MRState Mem\. \(MB\)\#StatesRS\-MR \(= OE\-1, ours\)0\.798375\\mathbf\{0\.798375\}—1913\.211913\.212OE\-20\.7983900\.798390\+0\.000015\+0\.0000151913\.211913\.212OE\-30\.7984240\.798424\+0\.000049\+0\.0000491913\.211913\.212OE\-40\.7984330\.798433\+0\.000058\+0\.0000581913\.211913\.212OE\-50\.7984380\.798438\+0\.000063\+0\.0000631913\.211913\.212OE\-60\.7989520\.798952\+0\.000577\+0\.0005775359\.945359\.943OE\-70\.7999400\.799940\+0\.001565\+0\.0015651915\.241915\.244OE\-80\.8002250\.800225\+0\.001850\+0\.0018505248\.285248\.283OE\-90\.8002690\.800269\+0\.001894\+0\.0018945248\.285248\.283OE\-100\.8015180\.801518\+0\.003143\+0\.0031431913\.211913\.212Muon0\.8029490\.802949\+0\.004574\+0\.0045741905\.761905\.761Sophia0\.8773530\.877353\+0\.078978\+0\.0789783811\.533811\.532AdamW0\.8984150\.898415\+0\.100040\+0\.1000403811\.533811\.532RMSProp0\.9016050\.901605\+0\.103230\+0\.1032301905\.761905\.761Lion0\.9201010\.920101\+0\.121726\+0\.1217261905\.761905\.761SGD\+Momentum1\.1493981\.149398\+0\.351023\+0\.3510231905\.761905\.761