Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression

# Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression
Source: [https://arxiv.org/html/2607.21855](https://arxiv.org/html/2607.21855)
11institutetext:UCI, USA
11email:edfink234@gmail\.com22institutetext:San Diego, California, USA
22email:charlieb1658@gmail\.com###### Abstract

We investigate whether symbolic regression can discover explicit neural network weight\-update rules that outperform standard hand\-designed optimizers on small symbolic regression benchmarks\. Candidate update rules are represented as fixed\-depth symbolic expressions over operands derived from common optimizers, including gradient, momentum, adaptive\-gradient, and moment\-estimate quantities\. Across 30 benchmark/neural network combinations, the symbolic regression procedure found an update rule outperforming the best hyperparameter\-tuned established optimizer in 25 cases, with an aggregate MSE reduction of 44\.47% over the improved cases\. The discovered rules do not all share a single common symbolic form, but many combine adaptive normalization, momentum\-like quantities, nonlinear transformations, and rational expressions\. These results suggest that symbolic regression can serve as a lightweight mechanism for discovering compact optimizer variants, while also highlighting the need for larger\-scale validation\.

## 1Neural Network Weight\-Update Rules

Training a feed\-forward neural network requires repeatedly updating its weights to reduce an error function\. The basic gradient\-descent update rule is:

wt\+1=wt−α​∇f​\(wt\),w\_\{t\+1\}=w\_\{t\}\-\\alpha\\nabla f\(w\_\{t\}\),\(1\)wherewtw\_\{t\}denotes the weights at iterationtt,α\>0\\alpha\>0is the learning rate, and

f=1N​‖x​\(w\)−y‖2\.f=\\frac\{1\}\{N\}\|\|x\(w\)\-y\|\|^\{2\}\.\(2\)Here,x​\(w\)x\(w\)is the prediction vector,yyis the target vector, andNNis the vector size\.

Several widely used optimizers modify \([1](https://arxiv.org/html/2607.21855#S1.E1)\) by incorporating momentum, adaptive learning rates, or moment estimates\. Heavy\-ball momentum is given by

vt\+1\\displaystyle v\_\{t\+1\}=θ​vt\+α​∇f​\(wt\)\\displaystyle=\\theta v\_\{t\}\+\\alpha\\nabla f\(w\_\{t\}\)wt\+1\\displaystyle w\_\{t\+1\}=wt−vt\+1,\\displaystyle=w\_\{t\}\-v\_\{t\+1\},\(3\)whereθ∈\[0,1\]\\theta\\in\[0,1\]controls the influence of the previous velocity\. Nesterov accelerated gradient instead evaluates the gradient at a look\-ahead point:

vt\+1\\displaystyle v\_\{t\+1\}=θ​vt\+α​∇f​\(wt−θ​vt\)\\displaystyle=\\theta v\_\{t\}\+\\alpha\\nabla f\(w\_\{t\}\-\\theta v\_\{t\}\)wt\+1\\displaystyle w\_\{t\+1\}=wt−vt\+1\.\\displaystyle=w\_\{t\}\-v\_\{t\+1\}\.\(4\)
Adaptive\-gradient methods rescale updates using information from past gradients\. AdaGrad is given by

wt\+1=wt−α​gΣ∘gt,w\_\{t\+1\}=w\_\{t\}\-\\alpha\\,g\_\{\\Sigma\}\\circ g\_\{t\},\(5\)wheregt=∇f​\(wt\)g\_\{t\}=\\nabla f\(w\_\{t\}\),∘\\circdenotes the Hadamard product, and

gΣ=\{\(ϵ\+∑i=1tgi,12\)−1/2,…,\(ϵ\+∑i=1tgi,N2\)−1/2\}\.g\_\{\\Sigma\}=\\left\\\{\\left\(\\epsilon\+\\sum\_\{i=1\}^\{t\}g\_\{i,1\}^\{2\}\\right\)^\{\-1/2\},\\ldots,\\left\(\\epsilon\+\\sum\_\{i=1\}^\{t\}g\_\{i,N\}^\{2\}\\right\)^\{\-1/2\}\\right\\\}\.\(6\)RMSProp replaces the cumulative squared\-gradient sum with an exponentially weighted running average:

wt\+1\\displaystyle w\_\{t\+1\}=wt\+Δ​wt\\displaystyle=w\_\{t\}\+\\Delta w\_\{t\}\(7\)Δ​wt\\displaystyle\\Delta w\_\{t\}=−αE​\[g2\]t\+ϵ​gt\\displaystyle=\-\\frac\{\\alpha\}\{\\sqrt\{E\[g^\{2\}\]\_\{t\}\+\\epsilon\}\}g\_\{t\}\(8\)E​\[g2\]t\\displaystyle E\[g^\{2\}\]\_\{t\}=γ​E​\[g2\]t−1\+\(1−γ\)​gt2\.\\displaystyle=\\gamma E\[g^\{2\}\]\_\{t\-1\}\+\(1\-\\gamma\)g\_\{t\}^\{2\}\.\(9\)
Adadelta applies a similar running\-average normalization to both gradients and updates:

E​\[g2\]t\\displaystyle E\\left\[g^\{2\}\\right\]\_\{t\}=γ​E​\[g2\]t−1\+\(1−γ\)​gt2\\displaystyle=\\gamma E\\left\[g^\{2\}\\right\]\_\{t\-1\}\+\(1\-\\gamma\)g\_\{t\}^\{2\}\(10\)Δ​wt\\displaystyle\\Delta w\_\{t\}=−E​\[Δ​w2\]t−1\+ϵE​\[g2\]t\+ϵ​gt\\displaystyle=\-\\sqrt\{\\frac\{E\\left\[\\Delta w^\{2\}\\right\]\_\{t\-1\}\+\\epsilon\}\{E\\left\[g^\{2\}\\right\]\_\{t\}\+\\epsilon\}\}g\_\{t\}\(11\)E​\[Δ​w2\]t\\displaystyle E\\left\[\\Delta w^\{2\}\\right\]\_\{t\}=γ​E​\[Δ​w2\]t−1\+\(1−γ\)​Δ​wt2\\displaystyle=\\gamma E\\left\[\\Delta w^\{2\}\\right\]\_\{t\-1\}\+\(1\-\\gamma\)\\Delta w\_\{t\}^\{2\}\(12\)wt\+1\\displaystyle w\_\{t\+1\}=wt\+Δ​wt\.\\displaystyle=w\_\{t\}\+\\Delta w\_\{t\}\.\(13\)
Adam combines a running average of past gradients with a running average of past squared gradients to adapt the update direction and scale:

μt\\displaystyle\\mu\_\{t\}=β1​μt−1\+\(1−β1\)​gt\\displaystyle=\\beta\_\{1\}\\mu\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)g\_\{t\}\(14\)νt\\displaystyle\\nu\_\{t\}=β2​νt−1\+\(1−β2\)​gt2\\displaystyle=\\beta\_\{2\}\\nu\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)g\_\{t\}^\{2\}\(15\)μ^t\\displaystyle\\widehat\{\\mu\}\_\{t\}=μt1−β1t\\displaystyle=\\frac\{\\mu\_\{t\}\}\{1\-\\beta\_\{1\}^\{t\}\}\(16\)ν^t\\displaystyle\\widehat\{\\nu\}\_\{t\}=νt1−β2t\\displaystyle=\\frac\{\\nu\_\{t\}\}\{1\-\\beta\_\{2\}^\{t\}\}\(17\)wt\+1\\displaystyle w\_\{t\+1\}=wt−α​μ^tν^t\+ϵ\.\\displaystyle=w\_\{t\}\-\\frac\{\\alpha\\widehat\{\\mu\}\_\{t\}\}\{\\sqrt\{\\widehat\{\\nu\}^\{\\phantom\{o\}\}\_\{t\}\+\\epsilon\}\}\.\(18\)Finally, AdamW modifies Adam by applying regularization directly to the weights while leaving the adaptive moment estimates unchanged:

wt\+1=wt−η​λ​wt−α​μ^tν^t\+ϵ\.w\_\{t\+1\}=w\_\{t\}\-\\eta\\lambda w\_\{t\}\-\\frac\{\\alpha\\widehat\{\\mu\}\_\{t\}\}\{\\sqrt\{\\widehat\{\\nu\}^\{\\phantom\{o\}\}\_\{t\}\+\\epsilon\}\}\.\(19\)
The operands appearing in these optimizers motivate the symbolic regression search space used in this work\.

## 2Applying Symbolic Regression to the Weight\-Update Problem

Given that many of the weight update rules developed and currently used in practice are*heuristically*motivated, an interesting question arises, namely, if algorithmically, one can find an empirically*better*weight\-update algorithm without significantly altering complexity and/or computational cost of existing ones\. This section outlines the procedure of this paper to probe an answer to this question\.

### 2\.1Symbolic Regression

Symbolic regression entails algorithmically searching a pre\-specified basis\-set of mathematical operators and operands for expressions which minimize a user\-defined loss\-metric,ℒ\\mathcal\{L\}\. In this paper,ℒ\\mathcal\{L\}is simply taken to be the mean\-squared error between the labelsY→\\vec\{Y\}of a dataset defined by the tuple\(X→,Y→\)\(\\vec\{X\},\\vec\{Y\}\)and the output of a feed\-forward neural networkf​\(X→,w→\)f\(\\vec\{X\},\\vec\{w\}\)whose weightsw→\\vec\{w\}shall be optimized\.

In this paper, the set of operators considered shall be the same for all tests conducted, given as follows:

- •Unary Operators:\-,𝐥𝐧\\mathbf\{ln\},𝐞𝐱𝐩\\mathbf\{exp\},𝐜𝐨𝐬\\mathbf\{cos\},𝐬𝐢𝐧\\mathbf\{sin\},𝐬𝐪𝐫𝐭\\mathbf\{sqrt\},𝐚𝐬𝐢𝐧\\mathbf\{asin\},𝐚𝐜𝐨𝐬\\mathbf\{acos\},𝐭𝐚𝐧𝐡\\mathbf\{tanh\}
- •Binary Operators:\+,\-,\*,/,∧\\bm\{\\wedge\}
- •Hyper\-parameters:𝜼\\bm\{\\eta\},𝜽\\bm\{\\theta\},ϵ\\bm\{\\epsilon\},𝜸\\bm\{\\gamma\},𝜷𝟏\\bm\{\\beta\_\{1\}\},𝜷𝟐\\bm\{\\beta\_\{2\}\}

### 2\.2Weight\-Update Problem

In section[1](https://arxiv.org/html/2607.21855#S1), various weight\-update rules were described, all of which can be represented by expression\-trees using a pre\-defined set of unary\+binary operators and operands\. Table[2](https://arxiv.org/html/2607.21855#footnote2)exemplifies this concept\.

Table 1:Weight update rules from section[1](https://arxiv.org/html/2607.21855#S1)and their corresponding minimal222meaning the simplest \(least number of nodes possible\) expression tree one can construct to represent the algebraic formulaexpression tree information\. Inputs in this table refer to unique leaf nodes in the tree\. The expression trees are constructed based on the input operands defined for this paper in Table[2](https://arxiv.org/html/2607.21855#S2.T2)\. An applet to scroll through the expression trees can be found[here](https://cb1658.github.io/Website/)\.The choice of operands, given in Table[2](https://arxiv.org/html/2607.21855#S2.T2), reflects those utilized in the weight\-update rules of section[1](https://arxiv.org/html/2607.21855#S1), with the additional simplification that the studies conducted in this paper will only consider fully\-connected feed\-forward neural networks with either sigmoid \(f​\(x\)=1/\(1\+exp⁡\(−x\)\)f\(x\)=1/\(1\+\\exp\(\-x\)\)\) and/or linear \(f​\(x\)=xf\(x\)=x\) perceptron activation functions\.

Table 2:Input operands considered for the symbolic searches conducted in this paper\. The symbolffdenotes the error function,xj=∑iyi​wj,ix\_\{j\}=\\sum\_\{i\}y\_\{i\}w\_\{j,i\}is the non\-activated input to neuronjj, whereyiy\_\{i\}are the individual outputs of the neuronsiifrom the previous layer,wj,kw\_\{j,k\}are the weights connecting neuronsjjin the “current” layer to the neuronskkin the next layer\. The hyperparameters are initialized arbitrarily as\{η←0\.5,θ←0\.01,γ←0\.9,ϵ←10−8,β1←0\.9,β2←0\.999\}\\\{\\eta\\leftarrow 0\.5,\\,\\theta\\leftarrow 0\.01,\\,\\gamma\\leftarrow 0\.9,\\,\\epsilon\\leftarrow 10^\{\-8\},\\,\\beta\_\{1\}\\leftarrow 0\.9,\\,\\beta\_\{2\}\\leftarrow 0\.999\\\}; the symbolic evolution in practice automatically forms combined constants anyway\.
### 2\.3Symbolic Regression Method

The symbolic regression method utilized in this paper is the multi\-threaded implementation of the generalized fixed\-depth postfix grammar and the accompanying Genetic Programming algorithm utilized in the previous works\[[7](https://arxiv.org/html/2607.21855#bib.bib17)\]and\[[6](https://arxiv.org/html/2607.21855#bib.bib18)\]\. The accompanying repository for this paper,[here](https://github.com/edfink234/Alpha-Zero-Symbolic-Regression/tree/NeuralNetworkWeightUpdate), includes a driver code that generates symbolic expressions and, for each weight\-update rule expression, evaluates its “score” by updating the weights of a randomly initialized feed\-forward neural network for an integer number of epochs using the generated weight\-update rule, and computing its loss thereafter\. The codes utilized for the experiments conducted in this paper are hyperlinked and described below:

- •[NeuralNetworks\_VecSR\.cpp](https://github.com/edfink234/Alpha-Zero-Symbolic-Regression/blob/cfbb9cb7f10ec1746a7ec1c33de9abb9a368f35a/NeuralNetworks_VecSR.cpp): The file that implements the genetic\-programming evolution and generation of candidate weight\-update rules\.
- •[MLP\_Vec\.h](https://github.com/edfink234/Alpha-Zero-Symbolic-Regression/blob/cfbb9cb7f10ec1746a7ec1c33de9abb9a368f35a/MLP_Vec.h)and[MLP\_Vec\.cpp](https://github.com/edfink234/Alpha-Zero-Symbolic-Regression/blob/cfbb9cb7f10ec1746a7ec1c33de9abb9a368f35a/MLP_Vec.cpp): The files that implement the multi\-layer perceptron class and weight\-update rule logic\.
- •[RunTestsNeuralNetworksVecSRPart1\.py](https://github.com/edfink234/Alpha-Zero-Symbolic-Regression/blob/cfbb9cb7f10ec1746a7ec1c33de9abb9a368f35a/RunTestsNeuralNetworksVecSRPart1.py): Driver code for conducting the hyperparameter grid\-sweep for the established weight\-update rules we benchmark against; see Table[3](https://arxiv.org/html/2607.21855#S4.T3)\.
- •[RunTestsNeuralNetworksVecSRPart2\.py](https://github.com/edfink234/Alpha-Zero-Symbolic-Regression/blob/cfbb9cb7f10ec1746a7ec1c33de9abb9a368f35a/RunTestsNeuralNetworksVecSRPart2.py): Driver code for executing the 30 different symbolic genetic evolutions for each of the 10 benchmarks and 3 feed\-forward neural network architectures considered herein\.

The primary contribution of this work is therefore an empirical study showing that symbolic regression with a grammar that produces expressions of a fixed expression\-tree depth\[[6](https://arxiv.org/html/2607.21855#bib.bib18)\]can discover compact, explicit weight\-update rules that outperform hyperparameter\-tuned standard optimizers on a collection of small neural network symbolic regression tasks\.

## 3Background

Discovering the optimal parameters of a feed\-forward neural network entails devising a method to update them\. To this end, many novel ideas have emerged over the years\. For example,\[[8](https://arxiv.org/html/2607.21855#bib.bib2)\]employs neural networks to learn weight and activation function update rules and shows the optimality of such rules to depend on problem complexity and neuron type\(s\)\. In\[[1](https://arxiv.org/html/2607.21855#bib.bib22)\], the authors employ Long\-short term memory \(LSTM\) neural networks as surrogates to the traditional gradient descent weight\-update rule and show that this approach can generalize to similar examples on which the LSTM\-updater was trained\.

Generally, the field of meta\-learning or “learning to learn” in the context of neural network optimization has been studied and theorized for quite some time\[[12](https://arxiv.org/html/2607.21855#bib.bib26)\]\[[14](https://arxiv.org/html/2607.21855#bib.bib27)\]\. For example,\[[11](https://arxiv.org/html/2607.21855#bib.bib25)\]represents a learning update rule as part of the neural network being used for prediction and found that more complicated \(depth 20\-30\) neural network weight\-updates performed better for linearly separable learning tasks\. On the other hand, genetic algorithms have also been shown to learn effective neural network updaters in the work of\[[3](https://arxiv.org/html/2607.21855#bib.bib24)\]\.

Another approach to meta\-learning was implemented in\[[2](https://arxiv.org/html/2607.21855#bib.bib30)\]; their framework uses a controller to generate strings \(which are specially designed to not contain parentheses\) that define the weight update rules\. Each update rule applies two separate unary operators on two separate operands, and combines the resulting two operands with a binary operator to form a primitive of the formΔ​w=λ⋅b​\(u1​\(o1\),u2​\(o2\)\)\\Delta w=\\lambda\\cdot b\(u\_\{1\}\(o\_\{1\}\),u\_\{2\}\(o\_\{2\}\)\)\. While the domain\-specific language \(DSL\) the authors created is not expansive enough to express all mathematical equations, they show that it is expansive enough to represent competitively\-performing optimizers on different machine\-learning tasks\.

The work of\[[2](https://arxiv.org/html/2607.21855#bib.bib30)\]points toward studies most relevant to our work; those which directly investigate discovering*explicit, symbolic weight\-update rules*from data\. Most closely related to our work, following in the spirit of Auto\-ML Zero\[[10](https://arxiv.org/html/2607.21855#bib.bib29)\], the authors of\[[4](https://arxiv.org/html/2607.21855#bib.bib28)\]strongly shift the paradigm from neural network weight\-update rules towards the symbolic discovery over the space of*algorithms*used to update the weights of a neural network\. Specifically, the authors of\[[4](https://arxiv.org/html/2607.21855#bib.bib28)\]perform a search over the space of possibletrainprogram\-functions on a variety of tasks\. When considering the performance of their resulting*Lion*\(*EvoLved Sign Momentum*\) algorithm, it is important to note that the authors populated the search space with adaptive algorithms such asAdamWand softly constrained thetrainprograms to depend on first and second non\-central moments of the gradient\-vector at iterationtt\. However, just like in\[[2](https://arxiv.org/html/2607.21855#bib.bib30)\], they enforce the weights to update as:

wj,m,t=τ=wj,m,t=τ−1−f^​\(Operands\),w\_\{j,m,t=\\tau\}=w\_\{j,m,t=\\tau\-1\}\-\\hat\{f\}\(\\texttt\{Operands\}\),\(20\)wheref^\\hat\{f\}here denotes the symbolic weight\-update rule candidate, and the set ofOperandsis included in table[2](https://arxiv.org/html/2607.21855#S2.T2)\.

In this work, we only include the operators and operands used to*form*adaptive algorithms instead of explicitly inserting the algorithms themselves into the equation population\. Additionally, we frame the problem as a search over weight\-update rules of the form333Note thatOperandscontains the previous weightwj,m,t=τ−1w\_\{j,m,t=\\tau\-1\}\.:

wj,m,t=τ=f^​\(Operands\)\.w\_\{j,m,t=\\tau\}=\\hat\{f\}\(\\texttt\{Operands\}\)\.\(21\)
In our work, we conduct a search over basis\-operands derived from mainstream weight\-update rules \(table[2](https://arxiv.org/html/2607.21855#S2.T2)\) to derive a population of explicit, compact, and interpretable weight\-update\-rule equations\. Then, we test their generalizability by seeding initial genetic populations for subsequent benchmarks with weight\-update rules found previously\.

## 4Experiments

In this section, we compare established neural\-network update rules against update rules discovered through symbolic regression on a collection of symbolic\-regression benchmark problems\.

### 4\.1Benchmarks

As discussed thoroughly in\[[5](https://arxiv.org/html/2607.21855#bib.bib19)\], there exist numerous “domain\-specific benchmarks” to quantify neural networks’ performance on specific tasks such as image/text classification, prediction, etc\. Here, we simply aim to investigate the performance of symbolically generated weight\-update rules for learning different symbolic regression benchmark\-functions from literature\[[9](https://arxiv.org/html/2607.21855#bib.bib20)\]\[[13](https://arxiv.org/html/2607.21855#bib.bib21)\]\.

In section[4\.2](https://arxiv.org/html/2607.21855#S4.SS2), we perform a grid search over a range of hyperparameters for each established weight\-update rule\. Then, in section[4\.3](https://arxiv.org/html/2607.21855#S4.SS3), we use genetic\-programming based symbolic regression for each of the 30 neural net\-benchmark combinations44410 benchmark expressions \(see tables[5](https://arxiv.org/html/2607.21855#S4.T5)and[6](https://arxiv.org/html/2607.21855#S4.T6)\)×\\times3 Neural networks \(see figure[1](https://arxiv.org/html/2607.21855#S4.F1)\) = 30 experiments\.to learn a population of weight\-update rules that perform better than the best performing established weight\-update rules\. In all cases, to make repeated evaluation of thousands of candidate update rules computationally feasible, we restrict all weight\-update rules to execute for exactly 10 epochs, and we terminate early if any of the weights become numerically undefined\.

### 4\.2Established Weight\-Update Rules Grid\-Search

For each of the ten benchmark expressions we consider in this work, denoted in tables[5](https://arxiv.org/html/2607.21855#S4.T5)and[6](https://arxiv.org/html/2607.21855#S4.T6), we test the performance of 3 simple feed\-forward neural network architectures, shown in Figure[1](https://arxiv.org/html/2607.21855#S4.F1), for each weight update rule in table[2](https://arxiv.org/html/2607.21855#footnote2), for the hyperparameter combos denoted in table[3](https://arxiv.org/html/2607.21855#S4.T3)\. The results of this hyperparameter sweep are shown in table[4](https://arxiv.org/html/2607.21855#S4.T4)\.

Neural Network 1

Input 1Input 2⋮\\vdotsInputNNSigmoidSigmoidInput 1Input 2InputNNSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidOutputSigmoidSigmoidSigmoidSigmoidSigmoidSigmoid

Neural Network 2

Input 1Input 2⋮\\vdotsInputNNSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidNoneNoneNoneNoneNoneOutput

Neural Network 3

Input 1Input 2⋮\\vdotsInputNNSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidSigmoidNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneOutput

Figure 1:The three feed\-forward neural network architectures we consider in the experiments section[4](https://arxiv.org/html/2607.21855#S4)\. “Sigmoid” denotes a neuron with sigmoid\-activation function and “None” denotes a neuron with no activation function \(linear\)\. TheNNinputs are determined by the number of inputs in the benchmark symbolic functions that these neural network architectures shall approximate\.Table 3:Hyperparameter grid searches for each weight\-update rule considered for each benchmark neural network combo\.BenchmarkNNBest Rule𝜼\\bm\{\\eta\}𝜽\\bm\{\\theta\}𝜸\\bm\{\\gamma\}ϵ\\bm\{\\epsilon\}𝜷𝟏\\bm\{\\beta\_\{1\}\}𝜷𝟐\\bm\{\\beta\_\{2\}\}𝝀\\bm\{\\lambda\}MSEFeynman 11heavy ball1\.00×10−31\.00\\times 10^\{\-3\}9\.00×10−19\.00\\times 10^\{\-1\}N/AN/AN/AN/AN/A3\.51×1003\.51\\times 10^\{0\}Feynman 12heavy ball1\.00×10−31\.00\\times 10^\{\-3\}8\.00×10−18\.00\\times 10^\{\-1\}N/AN/AN/AN/AN/A3\.54×1003\.54\\times 10^\{0\}Feynman 13AdaDeltaN/AN/A9\.95×10−19\.95\\times 10^\{\-1\}1\.00×10−41\.00\\times 10^\{\-4\}N/AN/AN/A4\.03×1004\.03\\times 10^\{0\}Feynman 21AdaDeltaN/AN/A9\.50×10−19\.50\\times 10^\{\-1\}1\.00×10−41\.00\\times 10^\{\-4\}N/AN/AN/A1\.25×1011\.25\\times 10^\{1\}Feynman 22AdaDeltaN/AN/A9\.90×10−19\.90\\times 10^\{\-1\}1\.00×10−41\.00\\times 10^\{\-4\}N/AN/AN/A1\.49×1011\.49\\times 10^\{1\}Feynman 23AdaDeltaN/AN/A9\.95×10−19\.95\\times 10^\{\-1\}1\.00×10−61\.00\\times 10^\{\-6\}N/AN/AN/A4\.83×1004\.83\\times 10^\{0\}Feynman 31heavy ball3\.00×10−43\.00\\times 10^\{\-4\}9\.50×10−19\.50\\times 10^\{\-1\}N/AN/AN/AN/AN/A2\.89×1082\.89\\times 10^\{8\}Feynman 32AdaGrad3\.00×10−33\.00\\times 10^\{\-3\}N/AN/AN/AN/AN/AN/A2\.93×1082\.93\\times 10^\{8\}Feynman 33AdaGrad1\.00×10−31\.00\\times 10^\{\-3\}N/AN/AN/AN/AN/AN/A2\.92×1082\.92\\times 10^\{8\}Feynman 41heavy ball3\.00×10−43\.00\\times 10^\{\-4\}9\.00×10−19\.00\\times 10^\{\-1\}N/AN/AN/AN/AN/A5\.41×1005\.41\\times 10^\{0\}Feynman 42basic1\.00×10−31\.00\\times 10^\{\-3\}N/AN/AN/AN/AN/AN/A7\.77×1007\.77\\times 10^\{0\}Feynman 43AdaDeltaN/AN/A9\.00×10−19\.00\\times 10^\{\-1\}1\.00×10−41\.00\\times 10^\{\-4\}N/AN/AN/A2\.09×1002\.09\\times 10^\{0\}Feynman 51AdaDeltaN/AN/A9\.50×10−19\.50\\times 10^\{\-1\}1\.00×10−41\.00\\times 10^\{\-4\}N/AN/AN/A1\.82×1011\.82\\times 10^\{1\}Feynman 52AdaDeltaN/AN/A9\.00×10−19\.00\\times 10^\{\-1\}1\.00×10−41\.00\\times 10^\{\-4\}N/AN/AN/A2\.07×1012\.07\\times 10^\{1\}Feynman 53AdaDeltaN/AN/A9\.00×10−19\.00\\times 10^\{\-1\}1\.00×10−41\.00\\times 10^\{\-4\}N/AN/AN/A1\.65×1011\.65\\times 10^\{1\}Hemberg 11NAG1\.00×10−31\.00\\times 10^\{\-3\}9\.00×10−19\.00\\times 10^\{\-1\}N/AN/AN/AN/AN/A3\.17×10−13\.17\\times 10^\{\-1\}Hemberg 12NAG1\.00×10−31\.00\\times 10^\{\-3\}7\.00×10−17\.00\\times 10^\{\-1\}N/AN/AN/AN/AN/A3\.89×10−13\.89\\times 10^\{\-1\}Hemberg 13NAG1\.00×10−41\.00\\times 10^\{\-4\}8\.00×10−18\.00\\times 10^\{\-1\}N/AN/AN/AN/AN/A3\.79×10−13\.79\\times 10^\{\-1\}Hemberg 21NAG3\.00×10−43\.00\\times 10^\{\-4\}9\.50×10−19\.50\\times 10^\{\-1\}N/AN/AN/AN/AN/A5\.47×1025\.47\\times 10^\{2\}Hemberg 22heavy ball1\.00×10−31\.00\\times 10^\{\-3\}8\.00×10−18\.00\\times 10^\{\-1\}N/AN/AN/AN/AN/A4\.91×1024\.91\\times 10^\{2\}Hemberg 23NAG1\.00×10−41\.00\\times 10^\{\-4\}7\.00×10−17\.00\\times 10^\{\-1\}N/AN/AN/AN/AN/A4\.09×1024\.09\\times 10^\{2\}Hemberg 31AdaDeltaN/AN/A9\.90×10−19\.90\\times 10^\{\-1\}1\.00×10−41\.00\\times 10^\{\-4\}N/AN/AN/A1\.49×1011\.49\\times 10^\{1\}Hemberg 32heavy ball1\.00×10−31\.00\\times 10^\{\-3\}8\.00×10−18\.00\\times 10^\{\-1\}N/AN/AN/AN/AN/A9\.05×1009\.05\\times 10^\{0\}Hemberg 33AdaDeltaN/AN/A9\.90×10−19\.90\\times 10^\{\-1\}1\.00×10−41\.00\\times 10^\{\-4\}N/AN/AN/A6\.97×1006\.97\\times 10^\{0\}Hemberg 41NAG1\.00×10−31\.00\\times 10^\{\-3\}8\.50×10−18\.50\\times 10^\{\-1\}N/AN/AN/AN/AN/A5\.56×1035\.56\\times 10^\{3\}Hemberg 42heavy ball1\.00×10−41\.00\\times 10^\{\-4\}8\.50×10−18\.50\\times 10^\{\-1\}N/AN/AN/AN/AN/A6\.01×1036\.01\\times 10^\{3\}Hemberg 43AdaDeltaN/AN/A9\.50×10−19\.50\\times 10^\{\-1\}1\.00×10−41\.00\\times 10^\{\-4\}N/AN/AN/A3\.34×1033\.34\\times 10^\{3\}Hemberg 51heavy ball3\.00×10−43\.00\\times 10^\{\-4\}9\.50×10−19\.50\\times 10^\{\-1\}N/AN/AN/AN/AN/A5\.67×1035\.67\\times 10^\{3\}Hemberg 52heavy ball1\.00×10−41\.00\\times 10^\{\-4\}8\.50×10−18\.50\\times 10^\{\-1\}N/AN/AN/AN/AN/A5\.86×1035\.86\\times 10^\{3\}Hemberg 53AdaDeltaN/AN/A9\.50×10−19\.50\\times 10^\{\-1\}1\.00×10−41\.00\\times 10^\{\-4\}N/AN/AN/A3\.77×1033\.77\\times 10^\{3\}

Table 4:Results of the hyperparameter grid sweep conducted; see table[3](https://arxiv.org/html/2607.21855#S4.T3)\. Benchmarks Feynman 1\-5 denote the corresponding equations 1\-5 in table[6](https://arxiv.org/html/2607.21855#S4.T6), and benchmarks Hemberg 1\-5 denote the corresponding equations 1\-5 in table[5](https://arxiv.org/html/2607.21855#S4.T5)\.
### 4\.3Symbolic Regression Benchmarks

Following\[[6](https://arxiv.org/html/2607.21855#bib.bib18)\], we apply symbolic regression to discover neural network update rules on benchmark expressions from\[[9](https://arxiv.org/html/2607.21855#bib.bib20)\]and\[[13](https://arxiv.org/html/2607.21855#bib.bib21)\]with varying complexity/depth of expression trees\. The ground\-truth equations, 10 in total, are shown in tables[5](https://arxiv.org/html/2607.21855#S4.T5)and[6](https://arxiv.org/html/2607.21855#S4.T6)\.

Table 5:Expressions from\[[9](https://arxiv.org/html/2607.21855#bib.bib20)\]considered in section[4\.1](https://arxiv.org/html/2607.21855#S4.SS1)\.Table 6:Expressions from\[[13](https://arxiv.org/html/2607.21855#bib.bib21)\]considered in section[4\.1](https://arxiv.org/html/2607.21855#S4.SS1)\.For each of the 30 benchmark/neural network combinations555denoted by the first 2 columns of Table[4](https://arxiv.org/html/2607.21855#S4.T4), we allow a maximum genetic population size of 100 candidate weight\-update rules split across all threads such that each thread performs an independent search, as preliminary experiments indicated that larger populations produced similar solutions at substantially greater computational cost\. The tree\-depth for each generated weight update rule is fixed to 5; thus, there are≈1\.71×1065\\approx 1\.71\\times 10^\{65\}weight\-update rules that can possibly be generated in our study666The number of unique equations for a given depth is computed asNdepth−i=NU⋅Ndepth−\(i−1\)\+NB​\(Ndepth−\(i−1\)2\+2⋅Ndepth−\(i−1\)​\(∑j=0i−2Ndepth−j\)\)N\_\{\\mathrm\{depth\-\}i\}=N\_\{U\}\\cdot N\_\{\\mathrm\{depth\-\}\(i\-1\)\}\+N\_\{B\}\\left\(N\_\{\\mathrm\{depth\}\-\(i\-1\)\}^\{2\}\+2\\cdot N\_\{\\mathrm\{depth\-\}\(i\-1\)\}\\left\(\\sum\_\{j=0\}^\{i\-2\}N\_\{\\mathrm\{depth\-\}j\}\\right\)\\right\), whereNdepth−1=NU⋅NL\+NB⋅NL2N\_\{\\mathrm\{depth\-1\}\}=N\_\{U\}\\cdot N\_\{L\}\+N\_\{B\}\\cdot N\_\{L\}^\{2\}andNdepth−0=​NLN\_\{\\mathrm\{depth\-0\}\}=\\text\{ \}N\_\{L\}, withNB=5N\_\{B\}=5,NL=22N\_\{L\}=22, andNU=9N\_\{U\}=9\. See a live code[here](https://www.sololearn.com/en/compiler-playground/cI8qd1dMjMog)\.\. For each experiment, we first attempt to seed the starting populations with the best weight\-update rules found in the previous experiment; however, if evolving this population fails to produce competitive update rules for a given benchmark, we fall back to evolving a fresh random population\. For each experiment, evolution continued until improvements in the best fitness became negligible over an extended period\. Runs were also terminated early if 100 expressions were discovered that outperformed the corresponding best hyperparameter\-tuned baseline optimizer\.

## 5Results

Table[8](https://arxiv.org/html/2607.21855#S5.T8)shows the results of our 30 experiments\. Symbolic regression discovered update rules that outperformed the best hyperparameter\-tuned baseline in 25 of 30 benchmark/neural network combinations, yielding an aggregate MSE reduction of 44\.47% across the improved cases\. The five failure cases were concentrated in the largest architecture \(Neural Network 3\), suggesting that the discovered rules may become less competitive as model complexity increases\.

Although the best discovered update equations \(available[here](https://cb1658.github.io/BenchmarkRules/)\) do not appear to share a single obvious symbolic structure, several qualitative trends emerge\. Many of the discovered rules incorporate nonlinearities such as trigonometric functions, hyperbolic tangent functions, exponentials, and rational expressions involving moving\-average statistics\. Several also resemble familiar optimizer design motifs, including momentum\-like terms, adaptive normalization, and variance scaling, despite these structures not being explicitly enforced during search\.

At the same time, the discovered expressions are not entirely unrelated\. Among the best expressions found by symbolic regression for each benchmark, four canonical expression signatures reappear across multiple benchmarks, see Table[7](https://arxiv.org/html/2607.21855#S5.T7), all of which incorporate*accumulated*gradient statistics over raw gradients\.

Table 7:Unique weight\-update expression signatures that reappear as part of the best expressions found by symbolic regression across benchmarks, whereCCrepresents a numerical\-value and the operands are denoted in table[2](https://arxiv.org/html/2607.21855#S2.T2)\.Many algebraically distinct expressions nevertheless achieved comparable predictive performance, suggesting that the search space contains large equivalence classes of effective update dynamics\. This observation may partially explain why symbolic regression was able to repeatedly discover update equations that outperform standard hand\-designed optimizers despite the overall structural diversity of the final expressions\.

Another notable observation is that the symbolic regression\-discovered rules often remained relatively compact despite the large search space\. This suggests that useful optimization dynamics may frequently admit low\-complexity symbolic representations, even when their functional forms differ substantially across tasks\.

Table 8:Results of running the symbolic regression searches delineated in sub\-section[4\.3](https://arxiv.org/html/2607.21855#S4.SS3)\. For 25/30 experiments, symbolic regression was able to find a weight\-update rule that performed better than the best hyperparameter\-tuned baseline rule in sub\-section[4\.2](https://arxiv.org/html/2607.21855#S4.SS2)\. For the 5 experiments where symbolic regression failed to find a better rule, 4 of them utilized neural network architecture 3 from Figure[1](https://arxiv.org/html/2607.21855#S4.F1)and the remaining failure case was architecture 2 on Hemberg 3\.
## 6Limitations

The experiments in this paper should be interpreted as a preliminary study rather than a complete optimizer benchmark\. The neural networks considered are small, training is restricted to 10 epochs, and performance is evaluated on symbolic regression benchmark functions rather than large\-scale machine\-learning datasets\. Moreover, the symbolic regression search is stochastic, and the discovered expressions are selected based on empirical performance rather than theoretical convergence guarantees\. Thus, while the results demonstrate that symbolic regression can discover effective, compact update rules in this setting, further experiments are needed to determine how these rules generalize to larger architectures, longer training horizons, classification tasks, and standard deep\-learning benchmarks\.

## 7Conclusion and Outlook

We presented a symbolic regression framework for searching the space of explicit feed\-forward neural network weight\-update rules\. Using a fixed set of operators and optimizer\-derived operands, the method discovered update rules that outperformed the best hyperparameter\-tuned established optimizer in 25 of 30 benchmark/neural network combinations\. Across the improved cases, the aggregate reduction in MSE was 44\.47%\.

The discovered rules were not structurally uniform, despite our strategy of seeding genetic populations with rules discovered in previous benchmarks, suggesting that the space of effective update rules may be heterogeneous and problem\-dependent\. Nevertheless, many rules combined recognizable optimizer components, including adaptive moment quantities, gradient\-normalization terms, momentum\-like variables, and nonlinear transformations\. This indicates that symbolic regression may be useful not only for optimizer discovery, but also for generating interpretable candidate update rules that can be inspected and simplified\.

Future work should evaluate these rules on larger neural networks, longer training runs, classification datasets, and repeated random seeds\. It would also be useful to add complexity penalties, stability constraints, and post\-hoc algebraic simplification to the search procedure, so that discovered optimizers are both performant and easier to analyze\.

## References

- \[1\]M\. Andrychowicz, M\. Denil, S\. Gomez, M\. W\. Hoffman, D\. Pfau, T\. Schaul, B\. Shillingford, and N\. de Freitas\(2016\)Learning to learn by gradient descent by gradient descent\.External Links:1606\.04474,[Link](https://arxiv.org/abs/1606.04474)Cited by:[§3](https://arxiv.org/html/2607.21855#S3.p1.1)\.
- \[2\]I\. Bello, B\. Zoph, V\. Vasudevan, and Q\. V\. Le\(2017\)Neural optimizer search with reinforcement learning\.External Links:1709\.07417,[Link](https://arxiv.org/abs/1709.07417)Cited by:[§3](https://arxiv.org/html/2607.21855#S3.p3.1),[§3](https://arxiv.org/html/2607.21855#S3.p4.1)\.
- \[3\]D\. J\. Chalmers\(1991\)The evolution of learning: an experiment in genetic connectionism\.InConnectionist Models,D\. S\. Touretzky, J\. L\. Elman, T\. J\. Sejnowski, and G\. E\. Hinton \(Eds\.\),pp\. 81–90\.External Links:ISBN 978\-1\-4832\-1448\-1,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/B978-1-4832-1448-1.50014-7),[Link](https://www.sciencedirect.com/science/article/pii/B9781483214481500147)Cited by:[§3](https://arxiv.org/html/2607.21855#S3.p2.1)\.
- \[4\]X\. Chen, C\. Liang, D\. Huang, E\. Real, K\. Wang, Y\. Liu, H\. Pham, X\. Dong, T\. Luong, C\. Hsieh, Y\. Lu, and Q\. V\. Le\(2023\)Symbolic discovery of optimization algorithms\.External Links:2302\.06675,[Link](https://arxiv.org/abs/2302.06675)Cited by:[§3](https://arxiv.org/html/2607.21855#S3.p4.1)\.
- \[5\]G\. E\. Dahl, F\. Schneider, Z\. Nado, N\. Agarwal, C\. S\. Sastry, P\. Hennig, S\. Medapati, R\. Eschenhagen, P\. Kasimbeg, D\. Suo, J\. Bae, J\. Gilmer, A\. L\. Peirson, B\. Khan, R\. Anil, M\. Rabbat, S\. Krishnan, D\. Snider, E\. Amid, K\. Chen, C\. J\. Maddison, R\. Vasudev, M\. Badura, A\. Garg, and P\. Mattson\(2023\)Benchmarking neural network training algorithms\.External Links:2306\.07179,[Link](https://arxiv.org/abs/2306.07179)Cited by:[§4\.1](https://arxiv.org/html/2607.21855#S4.SS1.p1.1)\.
- \[6\]E\. Finkelstein\(2024\)Generalized fixed\-depth prefix and postfix symbolic regression grammars\.External Links:2410\.08137,[Link](https://arxiv.org/abs/2410.08137)Cited by:[§2\.3](https://arxiv.org/html/2607.21855#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2607.21855#S2.SS3.p2.1),[§4\.3](https://arxiv.org/html/2607.21855#S4.SS3.p1.1)\.
- \[7\]E\. Finkelstein\(2024\)Solving the 2d advection\-diffusion equation using fixed\-depth symbolic regression and symbolic differentiation without expression trees\.External Links:2411\.00011,[Link](https://arxiv.org/abs/2411.00011)Cited by:[§2\.3](https://arxiv.org/html/2607.21855#S2.SS3.p1.1)\.
- \[8\]K\. Gregor\(2020\)Finding online neural update rules by learning to remember\.External Links:2003\.03124Cited by:[§3](https://arxiv.org/html/2607.21855#S3.p1.1)\.
- \[9\]E\. Hemberg, N\. McPhee, M\. O’Neill, and A\. Brabazon\(2008\)Pre\-, in\-and postfix grammars for symbolic regression in grammatical evolution\.InIEEE Workshop and Summer School on Evolutionary Computing,Vol\.2008,pp\. 18–22\.Cited by:[§4\.1](https://arxiv.org/html/2607.21855#S4.SS1.p1.1),[§4\.3](https://arxiv.org/html/2607.21855#S4.SS3.p1.1),[Table 5](https://arxiv.org/html/2607.21855#S4.T5),[Table 5](https://arxiv.org/html/2607.21855#S4.T5.9.2)\.
- \[10\]E\. Real, C\. Liang, D\. R\. So, and Q\. V\. Le\(2020\)AutoML\-zero: evolving machine learning algorithms from scratch\.External Links:2003\.03384,[Link](https://arxiv.org/abs/2003.03384)Cited by:[§3](https://arxiv.org/html/2607.21855#S3.p4.1)\.
- \[11\]T\.P\. Runarsson and M\.T\. Jonsson\(2000\)Evolution and design of distributed learning rules\.In2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks\. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks \(Cat\. No\.00,Vol\.,pp\. 59–63\.External Links:[Document](https://dx.doi.org/10.1109/ECNN.2000.886220)Cited by:[§3](https://arxiv.org/html/2607.21855#S3.p2.1)\.
- \[12\]J\. Schmidhuber\(1993\)A neural network that embeds its own meta\-levels\.InIEEE International Conference on Neural Networks,Vol\.,pp\. 407–412 vol\.1\.External Links:[Document](https://dx.doi.org/10.1109/ICNN.1993.298591)Cited by:[§3](https://arxiv.org/html/2607.21855#S3.p2.1)\.
- \[13\]S\. Udrescu and M\. Tegmark\(2020\)AI feynman: a physics\-inspired method for symbolic regression\.External Links:1905\.11481Cited by:[§4\.1](https://arxiv.org/html/2607.21855#S4.SS1.p1.1),[§4\.3](https://arxiv.org/html/2607.21855#S4.SS3.p1.1),[Table 6](https://arxiv.org/html/2607.21855#S4.T6),[Table 6](https://arxiv.org/html/2607.21855#S4.T6.9.2)\.
- \[14\]A\. Younger, S\. Hochreiter, and P\.R\. Conwell\(2001\-02\)Meta\-learning with backpropagation\.Vol\.3,pp\. 2001 – 2006 vol\.3\.External Links:ISBN 0\-7803\-7044\-9,[Document](https://dx.doi.org/10.1109/IJCNN.2001.938471)Cited by:[§3](https://arxiv.org/html/2607.21855#S3.p2.1)\.