EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale

# EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale
Source: [https://arxiv.org/html/2609.10980](https://arxiv.org/html/2609.10980)
Abolfazl Hashemi††thanks:The authors are with the Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA\.

###### Abstract

EGGROLL makes evolution strategies \(ES\) practical for LLMs by replacing dense Gaussian weight perturbations with low\-rank Gaussian products, often of rank one\. This choice is computationally attractive but geometrically severe: each rank\-one perturbation lies in a zero\-volume subset of the ambient matrix space, despite having identity covariance\. We characterize the mean EGGROLL update field at finite rank and nonzero perturbation radii, then analyze the error of its finite\-population estimator\. The population field is obtained by applying an explicit resolvent to the gradient of the objective smoothed by the perturbations\. We show that the resolvent can introduce a nonconservative component and can reverse the local stability of an optimum\. EGGROLL is nevertheless exact on every quadratic objective at every rank and radius\. For smooth objectives, its first local finite\-rank correction isO⁡\(σ2/r\)O\(\\sigma^\{2\}/r\), and nonasymptotic bounds control the resulting field error under smoothness assumptions\. Under a local affine model, rank\-one perturbations increase the variance of the gradient estimator by only2​\(m\+n\+1\)m​n\+1\\frac\{2\(m\+n\+1\)\}\{mn\+1\}relative to dense Gaussian ES, or0\.098%0\.098\\%for a4096×40964096\\times 4096matrix\. We then introduce LOO\-ROLL, a leave\-one\-out estimator that preserves the finite\-rank population field while replacing EGGROLL’s two antithetic evaluations per direction by one\. At equal evaluation cost, LOO\-ROLL halves estimator MSE in transformer blocks\. At matched wall time across ten post\-training settings and models up to 8B parameters, LOO\-ROLL improves seven outcomes in individual paired tests, with no significant loss\. On the GSM8K test set, accuracy increases from38\.1%38\.1\\%to63\.0%63\.0\\%at 0\.6B and from65\.9%65\.9\\%to80\.0%80\.0\\%at 8B\. Transformer measurements recover the predicted finite\-rank variance, while the rank comparisons show no reproducible reward\-based advantage for rank eight\.

## 1Introduction

Evolution strategies \(ES\)\[[37](https://arxiv.org/html/2609.10980#bib.bib38),[34](https://arxiv.org/html/2609.10980#bib.bib37),[6](https://arxiv.org/html/2609.10980#bib.bib39),[35](https://arxiv.org/html/2609.10980#bib.bib1)\]are zeroth\-order methods for optimizing a scalar function, customarily called fitness, using only perturbed function evaluations\. Because zeroth\-order methods require only forward evaluations, they have been explored for memory\-efficient language\-model fine\-tuning\[[30](https://arxiv.org/html/2609.10980#bib.bib14),[18](https://arxiv.org/html/2609.10980#bib.bib15),[53](https://arxiv.org/html/2609.10980#bib.bib41)\]\. Their parallel population structure has also motivated scalable evolution\-strategy methods for large\-model post\-training\[[36](https://arxiv.org/html/2609.10980#bib.bib2),[24](https://arxiv.org/html/2609.10980#bib.bib16),[49](https://arxiv.org/html/2609.10980#bib.bib40)\]\. The remaining difficulty is moving a large*population*of independently perturbed parameter settings through the network efficiently\. Modern accelerators obtain high throughput when arithmetic reuses data already brought into fast memory\[[48](https://arxiv.org/html/2609.10980#bib.bib7),[10](https://arxiv.org/html/2609.10980#bib.bib8)\]\. Giving every population member a different dense perturbation requires generating or reading anotherm​nmnnumbers per member, making memory traffic grow with the population\.

EGGROLL\[[36](https://arxiv.org/html/2609.10980#bib.bib2)\]avoids the use of such dense matrices by perturbing each weight matrix with a rank\-rrGaussian product:

Er=1r​A​B⊤=1r​∑s=1ras​bs⊤,as∼𝒩⁡\(0,Im\),bs∼𝒩⁡\(0,In\),E\_\{r\}=\\frac\{1\}\{\\sqrt\{r\}\}AB^\{\\top\}=\\frac\{1\}\{\\sqrt\{r\}\}\\sum\_\{s=1\}^\{r\}a\_\{s\}b\_\{s\}^\{\\top\},\\qquad a\_\{s\}\\sim\\mathcal\{N\}\(0,I\_\{m\}\),\\quad b\_\{s\}\\sim\\mathcal\{N\}\(0,I\_\{n\}\),\(1\)which permits the evaluation of very large populations at near\-inference throughput\. The normalization by1/r1/\\sqrt\{r\}gives every matrix coordinate unit variance and makes distinct coordinates uncorrelated:

𝔼⁡\[\(Er\)i​j​\(Er\)k​ℓ\]=δi​k​δj​ℓ\.\\mathbb\{E\}\\big\[\(E\_\{r\}\)\_\{ij\}\(E\_\{r\}\)\_\{k\\ell\}\\big\]=\\delta\_\{ik\}\\delta\_\{j\\ell\}\.\(2\)Hereδi​k\\delta\_\{ik\}is the Kronecker delta, equal to one wheni=ki=kand zero otherwise\. Thusvec⁡\(Er\)\\operatorname\{vec\}\(E\_\{r\}\)has identity covariance, just as a dense standard Gaussian perturbation\.

For a single activationxx, the evaluation by a single population member can be written as

\(W\+σr​A​B⊤\)​x=W​x\+σr​A​\(B⊤​x\)\.\\Bigl\(W\+\\frac\{\\sigma\}\{\\sqrt\{r\}\}AB^\{\\top\}\\Bigr\)x=Wx\+\\frac\{\\sigma\}\{\\sqrt\{r\}\}A\(B^\{\\top\}x\)\.\(3\)For a batch of population members, the large multiplication byWWcan be performed as one shared batched operation\. Each member then receives its own correction through the two thin multiplicationsB⊤​xB^\{\\top\}xandA⁡\(B⊤​x\)A\(B^\{\\top\}x\)\. These corrections requireO⁡\(r⁡\(m\+n\)\)O\(r\(m\+n\)\)additional storage and arithmetic per layer rather than theO⁡\(m​n\)O\(mn\)cost of a dense perturbation\. Batching them allows the population to share the base weights and much of the inference machinery\. Low\-rank adapters are already widely used in language\-model training\[[25](https://arxiv.org/html/2609.10980#bib.bib3),[12](https://arxiv.org/html/2609.10980#bib.bib47),[51](https://arxiv.org/html/2609.10980#bib.bib48),[28](https://arxiv.org/html/2609.10980#bib.bib49)\]because they replace a dense matrix modification by two thin factors\. EGGROLL uses the same computational advantage for a different purpose: the factors are randomly drawn search directions whose fitness is obtained from forward model evaluations, using no backpropagation\.

EGGROLL performs well in LLM\-scale training even in the seemingly prohibitive but computationally attractive regimer=1r=1\. The original analysis gives two asymptotic justifications: a high\-dimensional, vanishing\-radius result at fixed rank and convergence to dense Gaussian ES at rateO⁡\(1/r\)O\(1/r\)under regularity assumptions\[[36](https://arxiv.org/html/2609.10980#bib.bib2)\]\. These results do not identify the update field at the finite ranks and perturbation radii used in practice, the accuracy with which a finite population estimates that field, or the costs revealed by the finite\-rank analysis\. We therefore ask three guiding questions:

> *What mean field does the finite\-rank EGGROLL update follow, and why can rank one work?* *How accurately does a finite population estimate the resulting mean field?* *What weaknesses does the analysis expose, and how can they be corrected without hurting EGGROLL’s computational advantage?*

It is of note that the question of why rank one works is not settled by covariance\. Identity covariance means that repeated draws have nonzero variance along every linear direction, but every individual realization has rank at mostrr\. Whenr<min⁡\(m,n\)r<\\min\(m,n\), the set of possible realizations has dimensionr⁡\(m\+n−r\)<m​nr\(m\+n\-r\)<mnand occupies zero volume in the ambient space\[[1](https://arxiv.org/html/2609.10980#bib.bib9)\]\. EGGROLL also weights these Gaussian\-product samples using the score associated with a dense Gaussian perturbation, creating a score mismatch at every finite rank\. Both issues are most pronounced atr=1r=1, precisely the case of greatest practical interest\. A finite\-rank analysis is therefore needed in place of an appeal to covariance or a large\-rank limit\.

Figure 1:The two finite\-rank questions studied in this paper\. Batches of antithetic forward evaluations produce a finite\-population estimateg^r,σ,N\\widehat\{g\}\_\{r,\\sigma,N\}of the population fieldgr,σfg\_\{r,\\sigma\}^\{f\}\. The mean\-field branch characterizes the direction followed after averaging over perturbations and identifies both its possible departure from gradient dynamics and the quadratic regime in which that departure vanishes\. The finite\-population branch computes the sampling error around this mean\.Our answers to these questions separate three different aspects of EGGROLL\. First, the population field is a resolvent\-filtered gradient\. At a finite perturbation radius, this filtering can make the field nonconservative and can even make a local optimum repelling\. The discrepancy vanishes on quadratic objectives and remains controlled when the objective is well\-approximated by a quadratic over the perturbation scale\. Second, low rank has a much milder effect on finite\-population accuracy\. For wide matrices, the additional variance of a rank\-one product perturbation is negligible beside the variance already present in dense Gaussian ES\. Rank one can therefore estimate its own population field almost as efficiently as dense Gaussian perturbations estimate theirs, even though the two fields need not coincide\. Third,[the released EGGROLL implementation](https://github.com/ESHyperscale/HyperscaleES)spends two fitness evaluations per direction through antithetic estimators\. Our proposed algorithm, LOO\-ROLL, uses the other population members as leave\-one\-out baselines and requires only one evaluation per direction without changing the expected field\. Rank extrapolation and variance reduction techniques are mathematically natural responses to the finite\-rank bias and finite\-population variance identified above, but our experiments do not find a practical gain from their additional cost, so they are treated separately in Appendix[B](https://arxiv.org/html/2609.10980#A2)\.

##### Organization and contributions\.

We answer these questions and develop their algorithmic consequences in five parts\.

1. 1\.The finite\-rank mean field\.[Sections3](https://arxiv.org/html/2609.10980#S3)and[4](https://arxiv.org/html/2609.10980#S4)derive the expected EGGROLL update at every finite rank\. Averaging the objective over the perturbations gives a smoothed objective with its own gradient\. Because EGGROLL uses the dense\-Gaussian score with a low\-rank Gaussian\-product perturbation, it instead follows a resolvent\-filtered version of that gradient \([Theorems4\.1](https://arxiv.org/html/2609.10980#S4.Thmtheorem1)and[4\.2](https://arxiv.org/html/2609.10980#S4.Thmtheorem2)\)\. In a neural network, the objective depends jointly on many parameter matrices rather than on a single matrix in isolation\.[AppendixA](https://arxiv.org/html/2609.10980#A1)extends the result to this setting\.
2. 2\.The effect on optimization dynamics\.[Section5](https://arxiv.org/html/2609.10980#S5)determines when the expected update is the gradient of any scalar objective and gives an explicit smooth example in which a strict local maximum of the smoothed objective is unstable under rank\-one EGGROLL\.[Section6](https://arxiv.org/html/2609.10980#S6)shows that this failure disappears for quadratic objectives and remains controlled when the objective is approximately quadratic over the perturbation scale\. EGGROLL is exact on quadratic objectives, its first rank\-dependent local error isO⁡\(σ2/r\)O\(\\sigma^\{2\}/r\), and standard smoothness assumptions give nonasymptotic control of the field error \([Theorem5\.1](https://arxiv.org/html/2609.10980#S5.Thmtheorem1),[Proposition5\.3](https://arxiv.org/html/2609.10980#S5.Thmtheorem3),[Propositions6\.1](https://arxiv.org/html/2609.10980#S6.Thmtheorem1)and[6\.2](https://arxiv.org/html/2609.10980#S6.Thmtheorem2), and[Lemma6\.3](https://arxiv.org/html/2609.10980#S6.Thmtheorem3)\)\. Conventional convergence consequences under smoothness assumptions are collected in[AppendixC](https://arxiv.org/html/2609.10980#A3)\.
3. 3\.Finite\-population accuracy\.[Sections7](https://arxiv.org/html/2609.10980#S7)and[8](https://arxiv.org/html/2609.10980#S8)calculate the exact covariance and total variance of a finite\-population update\. Rank\-rrperturbations add2​\(m\+n\+1\)/r2\(m\+n\+1\)/rto the dense\-Gaussian variance factorm​n\+1mn\+1, so the relative variance increase quickly vanishes with matrix width even at rank one\. We also derive the fixed\-radius1/r1/rexpansion, while[AppendixC](https://arxiv.org/html/2609.10980#A3)records the corresponding finite\-iteration guarantees under relative population noise \([Theorems7\.1](https://arxiv.org/html/2609.10980#S7.Thmtheorem1)and[8\.1](https://arxiv.org/html/2609.10980#S8.Thmtheorem1)\)\.
4. 4\.LOO\-ROLL\.[Section9](https://arxiv.org/html/2609.10980#S9)introduces an unbiased leave\-one\-out estimator that uses one fitness evaluation per direction rather than the antithetic implementation’s two evaluations\. The construction applies the leave\-one\-out baseline to EGGROLL’s Gaussian\-product population and preserves the finite\-rank population field exactly before score standardization \([Proposition9\.1](https://arxiv.org/html/2609.10980#S9.Thmtheorem1)\)\. Rank extrapolation and a control\-variate estimator are derived and evaluated in[AppendixB](https://arxiv.org/html/2609.10980#A2)\.
5. 5\.Numerical and transformer experiments\.[Section10](https://arxiv.org/html/2609.10980#S10)first tests the exact finite\-rank and finite\-population predictions, then compares ranks at matched evaluations and wall time\. The principal algorithmic study of the section evaluates LOO\-ROLL across ten post\-training settings, three model sizes, two model families, and reward\-based objectives and next\-token prediction loss\. At matched wall time, LOO\-ROLL records seven improvements in individual paired tests, three unresolved differences, and no significant loss\. Five gains survive Holm correction across the ten comparisons \([Table9](https://arxiv.org/html/2609.10980#A5.T9)\)\.

## 2Related work

Random\-direction finite differences and simultaneous perturbation are classical gradient\-free methods\[[39](https://arxiv.org/html/2609.10980#bib.bib10),[32](https://arxiv.org/html/2609.10980#bib.bib11),[13](https://arxiv.org/html/2609.10980#bib.bib12)\]\. Their nonconvex analyses make the dependence on dimension and smoothing radius explicit\[[4](https://arxiv.org/html/2609.10980#bib.bib13)\]\. MeZO shows that an in\-place two\-forward\-pass implementation can fine\-tune large language models with inference\-level memory, and subsequent work added stochastic variance reduction\[[30](https://arxiv.org/html/2609.10980#bib.bib14),[18](https://arxiv.org/html/2609.10980#bib.bib15)\]\. Residual feedback uses a stored function value to obtain a one\-query estimator for online zeroth\-order optimization\[[52](https://arxiv.org/html/2609.10980#bib.bib42)\]\. These methods use dense or coordinatewise perturbations\. EGGROLL instead constructs each matrix perturbation as a low\-rank Gaussian product and batches the resulting population during inference, giving rise to the finite\-rank population field analyzed here\.

Baselines have long been used to reduce the variance of score\-function and policy\-gradient estimators without changing their expectation\[[47](https://arxiv.org/html/2609.10980#bib.bib45)\]\. In REINFORCE leave\-one\-out, the reward of each sampled response is compared with the average reward of the other responses generated from the same prompt\[[2](https://arxiv.org/html/2609.10980#bib.bib46)\]\. LOREN also uses REINFORCE leave\-one\-out in curvature\-aware zeroth\-order LLM tuning\[[38](https://arxiv.org/html/2609.10980#bib.bib50)\]\. LOO\-ROLL specializes the baseline to EGGROLL’s Gaussian\-product population and evaluates the resulting alternative to its antithetic training estimator\.

The original EGGROLL analysis proves a fixed\-rank, high\-dimensional vanishing\-radius guarantee and convergence of its Gaussian\-score update to dense Gaussian ES at rateO⁡\(1/r\)O\(1/r\)under regularity assumptions\[[36](https://arxiv.org/html/2609.10980#bib.bib2)\]\. Our results concern the complementary regime of finite dimension, finite radius, and finite population\. Recent work compares full Gaussian ES with gradient\-based LLM post\-training and studies its behavior in flat, linear, and quadratic geometries\[[24](https://arxiv.org/html/2609.10980#bib.bib16),[27](https://arxiv.org/html/2609.10980#bib.bib17)\]\.

Orthogonal and structured perturbations can reduce estimator variance or the cost of applying a direction, while structured control variates use additional problem information\[[8](https://arxiv.org/html/2609.10980#bib.bib18),[42](https://arxiv.org/html/2609.10980#bib.bib19)\]\. Guided and active\-subspace ES concentrate exploration in learned subspaces\[[29](https://arxiv.org/html/2609.10980#bib.bib20),[7](https://arxiv.org/html/2609.10980#bib.bib21)\]\. Natural ES and CMA\-ES adapt a Gaussian search law\[[46](https://arxiv.org/html/2609.10980#bib.bib22),[21](https://arxiv.org/html/2609.10980#bib.bib23)\]\. EGGROLL makes a different choice, where the Gaussian\-product law is fixed by the desired matrix computation\. We analyze the mean field and sampling error that follow from that choice\.

Stein operators for scalar products of independent normal variables, and for sums and broader products built from such variables, are well established\[[16](https://arxiv.org/html/2609.10980#bib.bib43),[17](https://arxiv.org/html/2609.10980#bib.bib44)\]\. These identities act on one scalar product\-normal variable\. The entries ofA​B⊤AB^\{\\top\}instead share row and column factors, so the matrix law couples different coordinates\. Our operator identity keeps this dependence, produces a matrix differential operator acting on the population field, and connects its resolvent to finite\-rank optimization dynamics\.

[Wang et al\. \[44\]](https://arxiv.org/html/2609.10980#bib.bib24)show that convergence can fail for structured zeroth\-order perturbations and proposed mixing them with isotropic Gaussian directions\. Their use of isotropy concerns an ambient Gaussian component\. EGGROLL already has identity covariance, yet its law is not spherically symmetric in the vectorized matrix space\. Our conservativity theorem isolates this stronger distributional requirement\. Exact spherical symmetry cannot be achieved by a nondegenerate law supported entirely on rank\-deficient matrices whenm,n≥2m,n\\geq 2, since a uniformly rotated nonzero matrix is full rank almost surely after reshaping\. Occasional dense directions could supply an ambient isotropic component, though those evaluations would no longer have the purely low\-rank cost of EGGROLL\.

## 3Setup

We start our inquiry by asking what finite\-rank EGGROLL follows in expectation\. We first consider one parameter matrixW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}\. Appendix[A](https://arxiv.org/html/2609.10980#A1)combines independent perturbations across the matrix blocks of a network\. Letf:ℝm×n→ℝf:\\mathbb\{R\}^\{m\\times n\}\\to\\mathbb\{R\}be a fitness to maximize, and use the Frobenius inner product⟨A,B⟩F=tr⁡\(A⊤​B\)\\langle A,\\,B\\rangle\_\{F\}=\\operatorname\{tr\}\(A^\{\\top\}B\)\. The perturbationErE\_\{r\}is defined as in[Equation1](https://arxiv.org/html/2609.10980#S1.E1)\. Sampling perturbed weights produces the smoothed objective

Fr,σ​\(W\)=𝔼⁡\[f⁡\(W\+σ​Er\)\],F\_\{r,\\sigma\}\(W\)=\\mathbb\{E\}\\big\[f\(W\+\\sigma E\_\{r\}\)\\big\],\(4\)whereσ\>0\\sigma\>0is the perturbation radius\. EGGROLL weights each sampled perturbation by its observed fitness\. Averaging this update over the perturbation law gives the mean update field, which we call the*population field*:

gr,σf​\(W\)=1σ​𝔼​\[Er​f​\(W\+σ​Er\)\]\.g\_\{r,\\sigma\}^\{f\}\(W\)=\\frac\{1\}\{\\sigma\}\\mathbb\{E\}\\big\[E\_\{r\}f\(W\+\\sigma E\_\{r\}\)\\big\]\.\(5\)This is the infinite\-population update that a finite ES population approximates\. For a perturbation densitypp, the*score*is∇log⁡p\\nabla\\log p\. A dense standard Gaussian matrixE∞E\_\{\\infty\}has score−E∞\-E\_\{\\infty\}\. Under the usual differentiability and integrability conditions, Stein’s Gaussian identity𝔼⁡\[E∞​h​\(E∞\)\]=𝔼⁡\[∇h​\(E∞\)\]\\mathbb\{E\}\[E\_\{\\infty\}h\(E\_\{\\infty\}\)\]=\\mathbb\{E\}\[\\nabla h\(E\_\{\\infty\}\)\]\[[41](https://arxiv.org/html/2609.10980#bib.bib25)\]applied toh⁡\(E\)=f⁡\(W\+σ​E\)h\(E\)=f\(W\+\\sigma E\)gives

1σ​𝔼​\[E∞​f​\(W\+σ​E∞\)\]=𝔼⁡\[∇f​\(W\+σ​E∞\)\]=∇F∞,σ​\(W\)\.\\frac\{1\}\{\\sigma\}\\mathbb\{E\}\\big\[E\_\{\\infty\}f\(W\+\\sigma E\_\{\\infty\}\)\\big\]=\\mathbb\{E\}\\big\[\\nabla f\(W\+\\sigma E\_\{\\infty\}\)\\big\]=\\nabla F\_\{\\infty,\\sigma\}\(W\)\.\(6\)The same identity does not automatically hold for the Gaussian\-product law\. Whenr<min⁡\(m,n\)r<\\min\(m,n\), the law is supported on a lower\-dimensional set and has no full\-dimensional density\. At every finite rank, including ranks for which a density exists,−Er\-E\_\{r\}is not its exact score\[[36](https://arxiv.org/html/2609.10980#bib.bib2)\]\. We first ask whether the finite\-rank population fieldgr,σfg\_\{r,\\sigma\}^\{f\}nevertheless equals the gradient∇Fr,σ\\nabla F\_\{r,\\sigma\}of the finite\-rank\-smoothed objective\. If it does not, it would be natural to next ask whether it is the gradient of some other scalar objective\.

The schematic algorithm and principal update equation in[Sarkar et al\. \[36\]](https://arxiv.org/html/2609.10980#bib.bib2)are written in the one\-sided form of[Equation5](https://arxiv.org/html/2609.10980#S3.E5)\. Their reported experiments, fitness shaping, and released LLM implementation, however, use paired antithetic perturbations\+Er\+E\_\{r\}and−Er\-E\_\{r\}:

g^r,σ,N​\(W\)=1N​∑s=1NEr\(s\)​f⁡\(W\+σ​Er\(s\)\)−f⁡\(W−σ​Er\(s\)\)2​σ\.\\widehat\{g\}\_\{r,\\sigma,N\}\(W\)=\\frac\{1\}\{N\}\\sum\_\{s=1\}^\{N\}E\_\{r\}^\{\(s\)\}\\frac\{f\(W\+\\sigma E\_\{r\}^\{\(s\)\}\)\-f\(W\-\\sigma E\_\{r\}^\{\(s\)\}\)\}\{2\\sigma\}\.\(7\)HereNNis the number of independent directions, each evaluated with both signs\. The symmetry ofErE\_\{r\}makes this an unbiased estimator of the population field\. Evaluating both signs on the same prompt batch also isolates the effect of the perturbation\. Any fitness component shared by the pair, such as the difficulty of the prompts in that batch, cancels when the two evaluations are subtracted\. For a smooth objective, a Taylor expansion aroundWWmakes the same cancellation explicit:

f⁡\(W\+σ​Er\)−f⁡\(W−σ​Er\)2​σ=⟨∇f​\(W\),Er⟩F\+O⁡\(σ2​∥Er∥F3\),\\frac\{f\(W\+\\sigma E\_\{r\}\)\-f\(W\-\\sigma E\_\{r\}\)\}\{2\\sigma\}=\\langle\\nabla f\(W\),\\,E\_\{r\}\\rangle\_\{F\}\+O\\left\(\\sigma^\{2\}\\lVert E\_\{r\}\\rVert\_\{F\}^\{3\}\\right\),\(8\)where the constant and the other even\-order terms cancel\.

The EGGROLL experiments and implementation normally center and standardize fitness values across the sampled population\. To keep the calculation explicit, our main analysis omits this score processing and studies the raw\-score field in[Equation5](https://arxiv.org/html/2609.10980#S3.E5)\. Appendix[F](https://arxiv.org/html/2609.10980#A6)treats the standardized update and quantifies its difference from the raw field\.

Both[Equations4](https://arxiv.org/html/2609.10980#S3.E4)and[5](https://arxiv.org/html/2609.10980#S3.E5)average translated copies offfagainst the Gaussian\-product perturbation law\. The smoothed objective is a convolution with this law, while the population field includes the additional weightEr/σE\_\{r\}/\\sigma\. Fourier modes diagonalize such convolution expressions, allowing both quantities to be calculated one frequency at a time\[[20](https://arxiv.org/html/2609.10980#bib.bib26)\]\. The next section performs this calculation and identifies the resulting transformation in parameter space\.

## 4Characterization of the population field

We continue our inquiry by asking:*What deterministic field does finite\-rank EGGROLL follow?*Two effects separate the optimization of the original objective from the EGGROLL population field\. Sampling from the finite\-rank perturbation law first smoothsffintoFr,σF\_\{r,\\sigma\}\. The gradient∇Fr,σ\\nabla F\_\{r,\\sigma\}is the update that gradient ascent on this smoothed objective would follow\. The problem, however, is that EGGROLL uses the dense\-Gaussian score−Er\-E\_\{r\}with Gaussian\-product samples, so its actual mean fieldgr,σfg\_\{r,\\sigma\}^\{f\}can differ from this gradient\. This section characterizes this second effect\. We determine how smoothing and score mismatch act on one Fourier frequency, then combine the frequencies to expressgr,σfg\_\{r,\\sigma\}^\{f\}as an operator applied to∇Fr,σ\\nabla F\_\{r,\\sigma\}\. The result separates the change caused by smoothing from the additional change caused by the mismatched score\.

FixT∈ℝm×nT\\in\\mathbb\{R\}^\{m\\times n\}and set the objective in[Equations4](https://arxiv.org/html/2609.10980#S3.E4)and[5](https://arxiv.org/html/2609.10980#S3.E5)to the Fourier modef⁡\(W\)=fT​\(W\):=ei​⟨T,W⟩Ff\(W\)=f\_\{T\}\(W\):=e^\{i\\langle T,\\,W\\rangle\_\{F\}\}\. LetΦr​\(T\)=𝔼⁡\[ei​⟨T,Er⟩F\]\\Phi\_\{r\}\(T\)=\\mathbb\{E\}\\big\[e^\{i\\langle T,\\,E\_\{r\}\\rangle\_\{F\}\}\\big\]denote the characteristic function ofErE\_\{r\}\. By direct substitution, we have

Fr,σ​\(W\)=ei​⟨T,W⟩F​Φr​\(σ​T\),gr,σfT​\(W\)=ei​⟨T,W⟩Fσ​𝔼​\[Er​ei​σ​⟨T,Er⟩F\]\.F\_\{r,\\sigma\}\(W\)=e^\{i\\langle T,\\,W\\rangle\_\{F\}\}\\Phi\_\{r\}\(\\sigma T\),\\qquad g\_\{r,\\sigma\}^\{f\_\{T\}\}\(W\)=\\frac\{e^\{i\\langle T,\\,W\\rangle\_\{F\}\}\}\{\\sigma\}\\mathbb\{E\}\\big\[E\_\{r\}e^\{i\\sigma\\langle T,\\,E\_\{r\}\\rangle\_\{F\}\}\\big\]\.\(9\)Differentiating the characteristic function with respect to its argument yields∇TΦr​\(T\)=i​𝔼​\[Er​ei​⟨T,Er⟩F\]\\nabla\_\{T\}\\Phi\_\{r\}\(T\)=i\\mathbb\{E\}\\big\[E\_\{r\}e^\{i\\langle T,\\,E\_\{r\}\\rangle\_\{F\}\}\\big\]\. Hence,

gr,σfT​\(W\)=Mr,σ​\(T\)​ei​⟨T,W⟩F,where​Mr,σ​\(T\):=1i​σ​∇TΦr​\(σ​T\)\.g\_\{r,\\sigma\}^\{f\_\{T\}\}\(W\)=M\_\{r,\\sigma\}\(T\)e^\{i\\langle T,\\,W\\rangle\_\{F\}\},\\text\{ where \}M\_\{r,\\sigma\}\(T\):=\\frac\{1\}\{i\\sigma\}\\nabla\_\{T\}\\Phi\_\{r\}\(\\sigma T\)\.\(10\)Thus,Mr,σ​\(T\)M\_\{r,\\sigma\}\(T\)is the coefficient by which the population field transforms the Fourier mode at frequencyTT\. The next theorem computes this coefficient from the characteristic function\.

###### Theorem 4\.1\(Exact Gaussian\-product law\)\.

ForErE\_\{r\}in[Equation1](https://arxiv.org/html/2609.10980#S1.E1), the characteristic function at frequencyT∈ℝm×nT\\in\\mathbb\{R\}^\{m\\times n\}is

Φr\(T\)=𝔼\[ei​⟨T,Er⟩F\]=det\(Im\+T​T⊤r\)−r/2\.\\Phi\_\{r\}\(T\)=\\mathbb\{E\}\\big\[e^\{i\\langle T,\\,E\_\{r\}\\rangle\_\{F\}\}\\big\]=\\det\\left\(I\_\{m\}\+\\frac\{TT^\{\\top\}\}\{r\}\\right\)^\{\-r/2\}\.\(11\)Moreover,

∇TΦr​\(T\)=−Φr​\(T\)​\(Im\+T​T⊤r\)−1​T\.\\nabla\_\{T\}\\Phi\_\{r\}\(T\)=\-\\Phi\_\{r\}\(T\)\\left\(I\_\{m\}\+\\frac\{TT^\{\\top\}\}\{r\}\\right\)^\{\-1\}T\.\(12\)

###### Proof\.

For a pair\(a,b\)∈ℝm×ℝn\(a,b\)\\in\\mathbb\{R\}^\{m\}\\times\\mathbb\{R\}^\{n\}, condition onaa:

𝔼b​\[ei​⟨T,a​b⊤⟩F/r\]=𝔼b​\[ei​tr⁡\(T⊤​a​b⊤\)/r\]=𝔼b​\[ei​b⊤​T⊤​a/r\]\.\\mathbb\{E\}\_\{b\}\\big\[e^\{i\\langle T,\\,ab^\{\\top\}\\rangle\_\{F\}/\\sqrt\{r\}\}\\big\]=\\mathbb\{E\}\_\{b\}\\big\[e^\{i\\operatorname\{tr\}\(T^\{\\top\}ab^\{\\top\}\)/\\sqrt\{r\}\}\\big\]=\\mathbb\{E\}\_\{b\}\\big\[e^\{ib^\{\\top\}T^\{\\top\}a/\\sqrt\{r\}\}\\big\]\.\(13\)Because we have conditioned onaa,c:=T⊤​a∈ℝnc:=T^\{\\top\}a\\in\\mathbb\{R\}^\{n\}is fixed\. Furthermore, sinceb∼𝒩⁡\(0,In\)b\\sim\\mathcal\{N\}\(0,I\_\{n\}\),b⊤​cb^\{\\top\}cis a one\-dimensional Gaussian, i\.e\.,

b⊤​c∼𝒩⁡\(0,∥c∥2\)\.b^\{\\top\}c\\sim\\mathcal\{N\}\(0,\\lVert c\\rVert^\{2\}\)\.\(14\)Hence,

𝔼b​\[ei​b⊤​T⊤​a/r\]=𝔼b​\[ei​b⊤​c/r\]=exp⁡\(−12​r​∥c∥2\)=exp⁡\(−12​r​∥T⊤​a∥2\),\\mathbb\{E\}\_\{b\}\\big\[e^\{ib^\{\\top\}T^\{\\top\}a/\\sqrt\{r\}\}\\big\]=\\mathbb\{E\}\_\{b\}\\big\[e^\{ib^\{\\top\}c/\\sqrt\{r\}\}\\big\]=\\exp\\Bigl\(\-\\frac\{1\}\{2r\}\\lVert c\\rVert^\{2\}\\Bigr\)=\\exp\\Bigl\(\-\\frac\{1\}\{2r\}\\lVert T^\{\\top\}a\\rVert^\{2\}\\Bigr\),\(15\)by the Gaussian characteristic function\. We have thus obtained the expectation conditional onaa\. To have the full expectation, we now evaluate

𝔼a​\[exp⁡\(−12​r​∥T⊤​a∥2\)\]=𝔼a​\[exp⁡\(−12​r​a⊤​T​T⊤​a\)\]\.\\mathbb\{E\}\_\{a\}\\Bigl\[\\exp\\Bigl\(\-\\frac\{1\}\{2r\}\\lVert T^\{\\top\}a\\rVert^\{2\}\\Bigr\)\\Bigr\]=\\mathbb\{E\}\_\{a\}\\Bigl\[\\exp\\Bigl\(\-\\frac\{1\}\{2r\}a^\{\\top\}TT^\{\\top\}a\\Bigr\)\\Bigr\]\.\(16\)By the Gaussian quadratic\-form identity\[[31](https://arxiv.org/html/2609.10980#bib.bib6)\], this is

det\(Im\+T​T⊤r\)−1/2\.\\det\\Bigl\(I\_\{m\}\+\\frac\{TT^\{\\top\}\}\{r\}\\Bigr\)^\{\-1/2\}\.\(17\)This means that one rank\-one term contributes

𝔼\[ei​⟨T,a​b⊤⟩F/r\]=det\(Im\+T​T⊤r\)−1/2\.\\mathbb\{E\}\\big\[e^\{i\\langle T,\\,ab^\{\\top\}\\rangle\_\{F\}/\\sqrt\{r\}\}\\big\]=\\det\\Bigl\(I\_\{m\}\+\\frac\{TT^\{\\top\}\}\{r\}\\Bigr\)^\{\-1/2\}\.\(18\)Now,

Er=1r​∑s=1ras​bs⊤,E\_\{r\}=\\frac\{1\}\{\\sqrt\{r\}\}\\sum\_\{s=1\}^\{r\}a\_\{s\}b\_\{s\}^\{\\top\},\(19\)so

Φr\(T\)=𝔼\[ei​⟨T,Er⟩F\]=∏s=1r𝔼\[ei​⟨T,as​bs⊤⟩F/r\]=det\(Im\+T​T⊤r\)−r/2,\\Phi\_\{r\}\(T\)=\\mathbb\{E\}\\big\[e^\{i\\langle T,\\,E\_\{r\}\\rangle\_\{F\}\}\\big\]=\\prod\_\{s=1\}^\{r\}\\mathbb\{E\}\\big\[e^\{i\\langle T,\\,a\_\{s\}b\_\{s\}^\{\\top\}\\rangle\_\{F\}/\\sqrt\{r\}\}\\big\]=\\det\\Bigl\(I\_\{m\}\+\\frac\{TT^\{\\top\}\}\{r\}\\Bigr\)^\{\-r/2\},\(20\)since allrrpairs\(as,bs\)\(a\_\{s\},b\_\{s\}\)are independent\. We finish by computing the gradient of the resulting determinant\. WriteS=Im\+T​T⊤/rS=I\_\{m\}\+TT^\{\\top\}/r\. SincelogΦr\(T\)=−\(r/2\)logdetS\\log\\Phi\_\{r\}\(T\)=\-\(r/2\)\\log\\det S, its differential is

d​log⁡Φr​\(T\)=−r2​tr⁡\(S−1​d​S\),d​S=\(d​T\)​T⊤\+T​\(d​T\)⊤r\.d\\log\\Phi\_\{r\}\(T\)=\-\\frac\{r\}\{2\}\\operatorname\{tr\}\(S^\{\-1\}dS\),\\qquad dS=\\frac\{\(dT\)T^\{\\top\}\+T\(dT\)^\{\\top\}\}\{r\}\.\(21\)Substitutingd​SdScancels the factorrrand gives

d​log⁡Φr​\(T\)=−12​tr⁡\(S−1​\(d​T\)​T⊤\+S−1​T​\(d​T\)⊤\)\.d\\log\\Phi\_\{r\}\(T\)=\-\\frac\{1\}\{2\}\\operatorname\{tr\}\\left\(S^\{\-1\}\(dT\)T^\{\\top\}\+S^\{\-1\}T\(dT\)^\{\\top\}\\right\)\.\(22\)The matrixS−1S^\{\-1\}is symmetric\. Cyclicity of the trace andtr⁡\(A​\(d​T\)⊤\)=tr⁡\(A⊤​d​T\)\\operatorname\{tr\}\(A\(dT\)^\{\\top\}\)=\\operatorname\{tr\}\(A^\{\\top\}dT\)therefore make the two terms equal, so

d​log⁡Φr​\(T\)=−tr⁡\(T⊤​S−1​d​T\)=⟨−S−1​T,d​T⟩F\.d\\log\\Phi\_\{r\}\(T\)=\-\\operatorname\{tr\}\\left\(T^\{\\top\}S^\{\-1\}dT\\right\)=\\langle\-S^\{\-1\}T,\\,dT\\rangle\_\{F\}\.\(23\)By the defining identityd​h​\(T\)=⟨∇Th​\(T\),d​T⟩Fdh\(T\)=\\langle\\nabla\_\{T\}h\(T\),\\,dT\\rangle\_\{F\}for the Frobenius gradient,∇T​log​Φr​\(T\)=−S−1​T\\nabla\_\{T\}\\log\\Phi\_\{r\}\(T\)=\-S^\{\-1\}T\. Finally,∇TΦr​\(T\)=Φr​\(T\)​∇T​log⁡Φr​\(T\)\\nabla\_\{T\}\\Phi\_\{r\}\(T\)=\\Phi\_\{r\}\(T\)\\nabla\_\{T\}\\log\\Phi\_\{r\}\(T\), which proves[Equation12](https://arxiv.org/html/2609.10980#S4.E12)\. ∎

The determinant form is familiar from matrix\-variate product distributions\[[11](https://arxiv.org/html/2609.10980#bib.bib4)\]\. The geometric consequence of the determinant form is easiest to see from the singular valuess1​\(T\),…,sq​\(T\)s\_\{1\}\(T\),\\ldots,s\_\{q\}\(T\), whereq=min⁡\{m,n\}q=\\min\\\{m,n\\\}\. Note that[Equation11](https://arxiv.org/html/2609.10980#S4.E11)can equivalently be written as

Φr\(T\)=∏j=1q\(1\+sj​\(T\)2r\)−r/2\.\\Phi\_\{r\}\(T\)=\\prod\_\{j=1\}^\{q\}\\left\(1\+\\frac\{s\_\{j\}\(T\)^\{2\}\}\{r\}\\right\)^\{\-r/2\}\.\(24\)We thus see that finite\-rank smoothing depends on each singular value separately\. Dense Gaussian smoothing instead has characteristic function

Φ∞\(T\)=exp\(−12∥T∥F2\)=exp\(−12∑j=1qsj\(T\)2\),\\Phi\_\{\\infty\}\(T\)=\\exp\\left\(\-\\frac\{1\}\{2\}\\lVert T\\rVert\_\{F\}^\{2\}\\right\)=\\exp\\left\(\-\\frac\{1\}\{2\}\\sum\_\{j=1\}^\{q\}s\_\{j\}\(T\)^\{2\}\\right\),\(25\)which depends only on the squared sum of singular values\. Consequently, dense Gaussian smoothing attenuates two Fourier modes equally whenever their Frobenius norms agree, whereas finite\-rank smoothing can distinguish modes with the same norm but different singular values, hinting at a disruption of radial symmetry\.

The finite\-rank perturbation law retains some rotational symmetry, albeit less than a dense Gaussian law\. IfU∈ℝm×mU\\in\\mathbb\{R\}^\{m\\times m\}andV∈ℝn×nV\\in\\mathbb\{R\}^\{n\\times n\}are orthogonal matrices, then

U​Er​V⊤=1r​\(U​A\)​\(V​B\)⊤UE\_\{r\}V^\{\\top\}=\\frac\{1\}\{\\sqrt\{r\}\}\(UA\)\(VB\)^\{\\top\}\(26\)has the same distribution asErE\_\{r\}\. This is becauseUUandVVseparately rotate the row and column components of every outer productas​bs⊤a\_\{s\}b\_\{s\}^\{\\top\}\. Standard Gaussian vectors are rotationally invariant, soU​asUa\_\{s\}andV​bsVb\_\{s\}have the same distributions asasa\_\{s\}andbsb\_\{s\}, respectively\. These transformations therefore preserve the product law and its matrix structure\. A dense Gaussian matrix, on the other hand, remains unchanged in distribution under any orthogonal transformation after itsm​nmnentries are stacked into a vector\. The finite\-rank product law is not invariant under arbitrary orthogonal transformations of thosem​nmncoordinates\. This explains howErE\_\{r\}can have identity covariance while[Equation11](https://arxiv.org/html/2609.10980#S4.E11)still distinguishes the singular\-value profile ofTT\.

Substituting[Equation12](https://arxiv.org/html/2609.10980#S4.E12)into[Equation10](https://arxiv.org/html/2609.10980#S4.E10)gives the decomposition

Mr,σ​\(T\)=i​Φr​\(σ​T\)​J​\(T\),J⁡\(T\):=\(Im\+σ2​T​T⊤r\)−1​T\.M\_\{r,\\sigma\}\(T\)=i\\Phi\_\{r\}\(\\sigma T\)J\(T\),\\qquad J\(T\):=\\left\(I\_\{m\}\+\\frac\{\\sigma^\{2\}TT^\{\\top\}\}\{r\}\\right\)^\{\-1\}T\.\(27\)The gradient mode of the finite\-rank\-smoothed objective has coefficienti​Φr​\(σ​T\)​Ti\\Phi\_\{r\}\(\\sigma T\)T, whereas the population field has coefficienti​Φr​\(σ​T\)​J​\(T\)i\\Phi\_\{r\}\(\\sigma T\)J\(T\)\. Thus, smoothing contributes the same scalar magnitude and phase to both fields and the source of the entire discrepancy is the replacement of the matrix directionTTbyJ⁡\(T\)J\(T\)\.[Figure2](https://arxiv.org/html/2609.10980#S4.F2)displays this transformation on the two\-dimensional subspaceT=diag⁡\(t1,t2\)T=\\operatorname\{diag\}\(t\_\{1\},t\_\{2\}\)\. Within this subspace, a matrix is identified with its two diagonal entries, so bothTTand its transformed valueJ⁡\(T\)J\(T\)can be represented as arrows in the\(t1,t2\)\(t\_\{1\},t\_\{2\}\)plane\. For the single Fourier objectivefT​\(W\)=ei​⟨T,W⟩Ff\_\{T\}\(W\)=e^\{i\\langle T,\\,W\\rangle\_\{F\}\}, these arrows give the matrix directions of the gradient mode ofFr,σF\_\{r,\\sigma\}and the EGGROLL population mode after their common scalar factors are removed\.

Figure 2:Characteristic\-function geometry on the diagonal subspaceT=diag⁡\(t1,t2\)T=\\operatorname\{diag\}\(t\_\{1\},t\_\{2\}\), represented by\(t1,t2\)\(t\_\{1\},t\_\{2\}\)\. Darker contours denote weaker attenuation\. At the same frequencyTT, the blue arrow gives the gradient\-mode direction ofFr,σF\_\{r,\\sigma\}and the orange arrow gives the EGGROLL population\-mode directionJ⁡\(T\)=\(Im\+σ2​T​T⊤/r\)−1​TJ\(T\)=\(I\_\{m\}\+\\sigma^\{2\}TT^\{\\top\}/r\)^\{\-1\}T\.A:A dense Gaussian has radial contours and does not tilt the mode\.B:The rank\-one product law is nonradial and tiltsJ⁡\(T\)J\(T\)away fromTT\.C:Decreasingσ\\sigmaat fixed rank reduces both attenuation and tilt\.D:Increasing rank at fixedσ\\sigmamoves the geometry and direction toward the dense\-Gaussian limit\.WithJ⁡\(T\)J\(T\), we have a description of how the population field transforms the Fourier mode at frequencyTT\. To express this transformation in parameter space, we seek a second\-order differential operatorℒ\\mathcal\{L\}acting on matrix vector fieldsV:ℝm×n→ℝm×nV:\\mathbb\{R\}^\{m\\times n\}\\to\\mathbb\{R\}^\{m\\times n\}whose inverse reproduces the factor\(Im\+σ2​T​T⊤/r\)−1\(I\_\{m\}\+\\sigma^\{2\}TT^\{\\top\}/r\)^\{\-1\}on every mode\. Each differentiation of a Fourier mode produces a factor ofTT, so an appropriate second\-order operator can produceT​T⊤TT^\{\\top\}\. We therefore chooseℒ\\mathcal\{L\}to act on each mode by left multiplication of its coefficient byT​T⊤TT^\{\\top\}, and define

\(ℒV\)i​j=−∑k=1m∑ℓ=1n∂i​ℓ∂k​ℓVk​j\.\(\\mathcal\{L\}V\)\_\{ij\}=\-\\sum\_\{k=1\}^\{m\}\\sum\_\{\\ell=1\}^\{n\}\\partial\_\{i\\ell\}\\partial\_\{k\\ell\}V\_\{kj\}\.\(28\)The index expression becomes transparent when it is applied to one matrix\-valued mode\. LetC∈ℂm×nC\\in\\mathbb\{C\}^\{m\\times n\}and setV⁡\(W\)=C​ei​⟨T,W⟩FV\(W\)=Ce^\{i\\langle T,\\,W\\rangle\_\{F\}\}\. Since differentiation with respect toWk​ℓW\_\{k\\ell\}multiplies the exponential byi​Tk​ℓiT\_\{k\\ell\}, direct substitution into[Equation28](https://arxiv.org/html/2609.10980#S4.E28)gives

ℒ⁡\(C​ei​⟨T,W⟩F\)=\(T​T⊤​C\)​ei​⟨T,W⟩F\.\\mathcal\{L\}\\left\(Ce^\{i\\langle T,\\,W\\rangle\_\{F\}\}\\right\)=\(TT^\{\\top\}C\)e^\{i\\langle T,\\,W\\rangle\_\{F\}\}\.\(29\)Indeed, the\(i,j\)\(i,j\)entry on the right is∑k,ℓTi​ℓ​Tk​ℓ​Ck​j​ei​⟨T,W⟩F\\sum\_\{k,\\ell\}T\_\{i\\ell\}T\_\{k\\ell\}C\_\{kj\}e^\{i\\langle T,\\,W\\rangle\_\{F\}\}, exactly the sum produced by the two derivatives in[Equation28](https://arxiv.org/html/2609.10980#S4.E28)\. ThereforeI\+σ2​ℒ/rI\+\\sigma^\{2\}\\mathcal\{L\}/racts on this mode by

C⟼\(Im\+σ2​T​T⊤r\)​C\.C\\longmapsto\\left\(I\_\{m\}\+\\frac\{\\sigma^\{2\}TT^\{\\top\}\}\{r\}\\right\)C\.\(30\)Consequently,\(I\+σ2​ℒ/r\)−1\(I\+\\sigma^\{2\}\\mathcal\{L\}/r\)^\{\-1\}acts on the coefficient of each Fourier mode by left multiplication with\(Im\+σ2​T​T⊤/r\)−1\(I\_\{m\}\+\\sigma^\{2\}TT^\{\\top\}/r\)^\{\-1\}\. In particular, it maps the smoothed\-gradient directionTTtoJ⁡\(T\)J\(T\)in[Equation27](https://arxiv.org/html/2609.10980#S4.E27)\. The operatorℒ\\mathcal\{L\}is positive semidefinite and self\-adjoint on its naturalL2L^\{2\}domain\.

The analysis up to this point has fixed one frequencyTT, and hence one Fourier modefTf\_\{T\}\. A general objective can be treated by decomposing it into such modes through its Fourier transform\. The operator identity can be stated at two levels of regularity\. At the broader level, any locally integrableffof at most polynomial growth defines a*tempered distribution*, meaning a generalized function whose growth at infinity is no faster than polynomial and whose Fourier transform is defined in the distributional sense\[[14](https://arxiv.org/html/2609.10980#bib.bib27)\]\. The moments ofErE\_\{r\}ensure that averaging the translated functionsf⁡\(W\+σ​Er\)f\(W\+\\sigma E\_\{r\}\), with or without the factorErE\_\{r\}, remains well\-defined in this sense\. Fourier transformation then applies the multiplier derived above to the Fourier transform offf, giving the resolvent identity as an equality of tempered distributions\.

For a classical pointwise formula, we use a stronger sufficient condition\. Assume that the real\-valued functionffadmits the representation

f⁡\(W\)=∫ei​⟨T,W⟩F​μf​\(𝑑T\)f\(W\)=\\int e^\{i\\langle T,\\,W\\rangle\_\{F\}\}\\,\\mu\_\{f\}\(dT\)\(31\)and satisfies the weighted Fourier moment condition

∫\(1\+∥T∥F3\)​\|μf\|​\(𝑑T\)<∞,\\int\(1\+\\lVert T\\rVert\_\{F\}^\{3\}\)\\,\|\\mu\_\{f\}\|\(dT\)<\\infty,\(32\)whereμf\\mu\_\{f\}is a finite complex Borel measure satisfyingμf​\(−A\)=μf​\(A\)¯\\mu\_\{f\}\(\-A\)=\\overline\{\\mu\_\{f\}\(A\)\}\. The moment condition permits the single\-mode identity to be integrated absolutely overTTand allows the integral to be exchanged with expectations and derivatives\. This condition is sufficient for the pointwise formulas below, but is not required for the distributional operator identity\.

Under either formulation, linearity carries the single\-mode calculation through the Fourier decomposition\. At each frequencyTT, the coefficient of the smoothed gradient is multiplied by\(Im\+σ2​T​T⊤/r\)−1\(I\_\{m\}\+\\sigma^\{2\}TT^\{\\top\}/r\)^\{\-1\}to obtain the coefficient of the population field\. Combining the transformed frequencies therefore gives the full population field\. Equivalently, finite\-rank EGGROLL applies the inverse operator

Jr,σ:=\(I\+σ2r​ℒ\)−1J\_\{r,\\sigma\}:=\\left\(I\+\\frac\{\\sigma^\{2\}\}\{r\}\\mathcal\{L\}\\right\)^\{\-1\}\(33\)to the gradient of the smoothed objective\. This inverse is the*resolvent*ofσ2​ℒ/r\\sigma^\{2\}\\mathcal\{L\}/rin the standard terminology of monotone operator theory\[[5](https://arxiv.org/html/2609.10980#bib.bib5)\]\. Sinceℒ\\mathcal\{L\}is positive semidefinite,Jr,σJ\_\{r,\\sigma\}attenuates the selected matrix\-frequency components and is firmly nonexpansive:

∥Jr,σ​u−Jr,σ​v∥L22≤⟨Jr,σ​u−Jr,σ​v,u−v⟩L2,\\lVert J\_\{r,\\sigma\}u\-J\_\{r,\\sigma\}v\\rVert\_\{L^\{2\}\}^\{2\}\\leq\\left\\langle J\_\{r,\\sigma\}u\-J\_\{r,\\sigma\}v,u\-v\\right\\rangle\_\{L^\{2\}\},\(34\)that is, it cannot increase the globalL2L^\{2\}distance between two input fields\[[5](https://arxiv.org/html/2609.10980#bib.bib5)\]\. The following theorem states the operator identity and its corresponding energy relation\.

###### Theorem 4\.2\(The finite\-rank resolvent\)\.

Letffbe locally integrable and of at most polynomial growth\. Interpretingff,Fr,σF\_\{r,\\sigma\}, andgr,σfg\_\{r,\\sigma\}^\{f\}as tempered distributions,

\(I\+σ2rℒ\)gr,σf=∇Fr,σor equivalently,gr,σf=\(I\+σ2rℒ\)−1∇Fr,σ\.\\left\(I\+\\frac\{\\sigma^\{2\}\}\{r\}\\mathcal\{L\}\\right\)g\_\{r,\\sigma\}^\{f\}=\\nabla F\_\{r,\\sigma\}\\qquad\\text\{or equivalently,\}\\qquad g\_\{r,\\sigma\}^\{f\}=\\left\(I\+\\frac\{\\sigma^\{2\}\}\{r\}\\mathcal\{L\}\\right\)^\{\-1\}\\nabla F\_\{r,\\sigma\}\.\(35\)The equality holds in the space of tempered distributions\. Ifffsatisfies the preceding Fourier moment condition, both sides have classical Fourier representations and the identity holds at everyW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}\. Ifgr,σfg\_\{r,\\sigma\}^\{f\}and∇Fr,σ\\nabla F\_\{r,\\sigma\}are square\-integrable, the resolvent is self\-adjoint, positive, contractive, and firmly nonexpansive onL2L^\{2\}\. If, in addition,gr,σfg\_\{r,\\sigma\}^\{f\}belongs to the domain ofℒ\\mathcal\{L\}, meaning that bothgr,σfg\_\{r,\\sigma\}^\{f\}andℒ​gr,σf\\mathcal\{L\}g\_\{r,\\sigma\}^\{f\}lie inL2L^\{2\}, then

⟨gr,σf,∇Fr,σ⟩L2=∥gr,σf∥L22\+σ2r​∥ℒ1/2​gr,σf∥L22\.\\left\\langle g\_\{r,\\sigma\}^\{f\},\\nabla F\_\{r,\\sigma\}\\right\\rangle\_\{L^\{2\}\}=\\lVert g\_\{r,\\sigma\}^\{f\}\\rVert\_\{L^\{2\}\}^\{2\}\+\\frac\{\\sigma^\{2\}\}\{r\}\\lVert\\mathcal\{L\}^\{1/2\}g\_\{r,\\sigma\}^\{f\}\\rVert\_\{L^\{2\}\}^\{2\}\.\(36\)

###### Proof\.

For a tempered distributionff, averaging its translations against the perturbation law gives

F^r,σ​\(T\)=Φr​\(σ​T\)​f^​\(T\),g^r,σf​\(T\)=i​Φr​\(σ​T\)​\(Im\+σ2​T​T⊤r\)−1​T​f^​\(T\)\\widehat\{F\}\_\{r,\\sigma\}\(T\)=\\Phi\_\{r\}\(\\sigma T\)\\widehat\{f\}\(T\),\\qquad\\widehat\{g\}\_\{r,\\sigma\}^\{f\}\(T\)=i\\Phi\_\{r\}\(\\sigma T\)\\left\(I\_\{m\}\+\\frac\{\\sigma^\{2\}TT^\{\\top\}\}\{r\}\\right\)^\{\-1\}T\\widehat\{f\}\(T\)\(37\)in the distributional sense\. The Fourier multiplier ofI\+\(σ2/r\)​ℒI\+\(\\sigma^\{2\}/r\)\\mathcal\{L\}is left multiplication byIm\+σ2​T​T⊤/rI\_\{m\}\+\\sigma^\{2\}TT^\{\\top\}/r\. Applying it to the second expression givesi​T​F^r,σ​\(T\)iT\\widehat\{F\}\_\{r,\\sigma\}\(T\), the Fourier transform of∇Fr,σ\\nabla F\_\{r,\\sigma\}\. This proves[Equation35](https://arxiv.org/html/2609.10980#S4.E35)as an identity of tempered distributions\.

Under the stated Fourier moment condition, substituting the Fourier representation offfinto[Equation5](https://arxiv.org/html/2609.10980#S3.E5), interchanging expectation and integration, and using[Equation12](https://arxiv.org/html/2609.10980#S4.E12)gives the classical formula

gr,σf​\(W\)=∫i​Φr​\(σ​T\)​\(Im\+σ2​T​T⊤r\)−1​T​ei​⟨T,W⟩F​μf​\(𝑑T\),g\_\{r,\\sigma\}^\{f\}\(W\)=\\int i\\Phi\_\{r\}\(\\sigma T\)\\left\(I\_\{m\}\+\\frac\{\\sigma^\{2\}TT^\{\\top\}\}\{r\}\\right\)^\{\-1\}Te^\{i\\langle T,\\,W\\rangle\_\{F\}\}\\,\\mu\_\{f\}\(dT\),\(38\)whereas

∇Fr,σ​\(W\)=∫i​Φr​\(σ​T\)​T​ei​⟨T,W⟩F​μf​\(𝑑T\)\.\\nabla F\_\{r,\\sigma\}\(W\)=\\int i\\Phi\_\{r\}\(\\sigma T\)Te^\{i\\langle T,\\,W\\rangle\_\{F\}\}\\,\\mu\_\{f\}\(dT\)\.\(39\)The moment condition justifies these operations and makes the identity pointwise\. At each frequencyTT, the multiplierIm\+σ2​T​T⊤/rI\_\{m\}\+\\sigma^\{2\}TT^\{\\top\}/ris symmetric with eigenvalues at least one, making its inverse positive, contractive, and firmly nonexpansive\. Parseval’s identity equatesL2L^\{2\}inner products of fields in parameter space with the corresponding inner products of their Fourier coefficients\[[14](https://arxiv.org/html/2609.10980#bib.bib27)\]\. Applying this identity to\(I\+\(σ2/r\)​ℒ\)​g=∇F\(I\+\(\\sigma^\{2\}/r\)\\mathcal\{L\}\)g=\\nabla Fgives[Equation36](https://arxiv.org/html/2609.10980#S4.E36)\. ∎

Equation \([35](https://arxiv.org/html/2609.10980#S4.E35)\) answers the question posed at the start of the section\. Finite\-rank sampling first formsFr,σF\_\{r,\\sigma\}, and the mismatched Gaussian score then transforms its gradient through the resolvent\(I\+σ2​ℒ/r\)−1\(I\+\\sigma^\{2\}\\mathcal\{L\}/r\)^\{\-1\}\. The action of this resolvent is explicit at every frequency\. IfT=U​diag⁡\(s1,…,sq\)​V⊤T=U\\operatorname\{diag\}\(s\_\{1\},\\ldots,s\_\{q\}\)V^\{\\top\}is a singular value decomposition ofTT, then

J⁡\(T\)=U​diag⁡\(s11\+σ2​s12/r,…,sq1\+σ2​sq2/r\)​V⊤\.J\(T\)=U\\operatorname\{diag\}\\left\(\\frac\{s\_\{1\}\}\{1\+\\sigma^\{2\}s\_\{1\}^\{2\}/r\},\\ldots,\\frac\{s\_\{q\}\}\{1\+\\sigma^\{2\}s\_\{q\}^\{2\}/r\}\\right\)V^\{\\top\}\.\(40\)Thus the smoothing factorΦr​\(σ​T\)\\Phi\_\{r\}\(\\sigma T\)is common to both fields, while the score mismatch multiplies thejjth singular direction by the attenuation factor

aj​\(sj\)=\(1\+σ2​sj2r\)−1\.a\_\{j\}\(s\_\{j\}\)=\\left\(1\+\\frac\{\\sigma^\{2\}s\_\{j\}^\{2\}\}\{r\}\\right\)^\{\-1\}\.\(41\)The factoraj​\(sj\)a\_\{j\}\(s\_\{j\}\)decreases monotonically from one assjs\_\{j\}increases, which is the sense in which the resolvent acts as an anisotropic low\-pass filter\. Attenuation begins for everysj\>0s\_\{j\}\>0and reaches one half atsj=r/σs\_\{j\}=\\sqrt\{r\}/\\sigma\. The transformed singular coefficientsj​aj​\(sj\)s\_\{j\}a\_\{j\}\(s\_\{j\}\)is not monotone: it increases up to this same threshold and decreases beyond it\. At finiterrandσ\\sigma, unequal attenuation of the singular components can tilt a mode rather than merely scale its magnitude\. Superposing such tilted modes can produce a field with a nonsymmetric Jacobian and therefore a nonconservative population field\. The question of whether the population field can be the gradient of*any*scalar objective is the subject of the next section\.

The energy identity of[Equation36](https://arxiv.org/html/2609.10980#S4.E36)follows by pairing the resolvent equation withgr,σfg\_\{r,\\sigma\}^\{f\}inL2L^\{2\}and using⟨g,ℒ​g⟩L2=∥ℒ1/2​g∥L22\\langle g,\\mathcal\{L\}g\\rangle\_\{L^\{2\}\}=\\lVert\\mathcal\{L\}^\{1/2\}g\\rVert\_\{L^\{2\}\}^\{2\}\. In particular,

⟨gr,σf,∇Fr,σ⟩L2≥∥gr,σf∥L22≥0\.\\left\\langle g\_\{r,\\sigma\}^\{f\},\\nabla F\_\{r,\\sigma\}\\right\\rangle\_\{L^\{2\}\}\\geq\\lVert g\_\{r,\\sigma\}^\{f\}\\rVert\_\{L^\{2\}\}^\{2\}\\geq 0\.\(42\)The energy identity shows that the population field is globally nonnegatively aligned with the smoothed gradient\. The first term on the right is the squaredL2L^\{2\}magnitude of the population field, while the second gives greater weight to matrix\-frequency components on which the resolvent acts more strongly\.

The single\-matrix resolvent extends independently across the matrix blocks of a network while the smoothed objective remains jointly coupled\. Appendix[A](https://arxiv.org/html/2609.10980#A1)states this generalization and the corresponding network\-level field and variance bounds\. The resolvent identity also gives a quantitative bound on the difference between the population field and the gradient of the smoothed objective\.

###### Corollary 4\.3\(Exact score\-mismatch control\)\.

LetJr,σ=\(I\+σ2​ℒ/r\)−1J\_\{r,\\sigma\}=\(I\+\\sigma^\{2\}\\mathcal\{L\}/r\)^\{\-1\}\. If∇Fr,σ\\nabla F\_\{r,\\sigma\}belongs to the domain ofℒ\\mathcal\{L\}, then

∥gr,σf−∇Fr,σ∥L2≤σ2r∥ℒ∇Fr,σ∥L2\.\\lVert g\_\{r,\\sigma\}^\{f\}\-\\nabla F\_\{r,\\sigma\}\\rVert\_\{L^\{2\}\}\\leq\\frac\{\\sigma^\{2\}\}\{r\}\\lVert\\mathcal\{L\}\\nabla F\_\{r,\\sigma\}\\rVert\_\{L^\{2\}\}\.\(43\)More exactly,gr,σf−∇Fr,σ=−\(σ2/r\)Jr,σℒ∇Fr,σg\_\{r,\\sigma\}^\{f\}\-\\nabla F\_\{r,\\sigma\}=\-\(\\sigma^\{2\}/r\)J\_\{r,\\sigma\}\\mathcal\{L\}\\nabla F\_\{r,\\sigma\}\.

###### Proof\.

Setα=σ2/r\\alpha=\\sigma^\{2\}/rand writeJ=Jr,σ=\(I\+α​ℒ\)−1J=J\_\{r,\\sigma\}=\(I\+\\alpha\\mathcal\{L\}\)^\{\-1\}\. On the domain ofℒ\\mathcal\{L\}, the resolvent identity gives

J−I=−α​J​ℒ\.J\-I=\-\\alpha J\\mathcal\{L\}\.\(44\)To see this directly, take any matrix vector fieldhhfor whichℒ​h\\mathcal\{L\}his defined\. SinceJJis the inverse ofI\+α​ℒI\+\\alpha\\mathcal\{L\},

J⁡\(h\+α​ℒ​h\)=h\.J\(h\+\\alpha\\mathcal\{L\}h\)=h\.\(45\)Linearity ofJJthen givesJ​h−h=−α​J​ℒ​hJh\-h=\-\\alpha J\\mathcal\{L\}h\. Applying this equality withh=∇Fr,σh=\\nabla F\_\{r,\\sigma\}and usinggr,σf=J∇Fr,σg\_\{r,\\sigma\}^\{f\}=J\\nabla F\_\{r,\\sigma\}yields

gr,σf−∇Fr,σ=−σ2rJr,σℒ∇Fr,σ,g\_\{r,\\sigma\}^\{f\}\-\\nabla F\_\{r,\\sigma\}=\-\\frac\{\\sigma^\{2\}\}\{r\}J\_\{r,\\sigma\}\\mathcal\{L\}\\nabla F\_\{r,\\sigma\},\(46\)which proves the exact formula\. The Fourier calculation in[Theorem4\.2](https://arxiv.org/html/2609.10980#S4.Thmtheorem2)also shows thatJJmultiplies every eigendirection at every frequency by a number in\[0,1\]\[0,1\]\. Parseval’s identity therefore gives∥J​h∥L2≤∥h∥L2\\lVert Jh\\rVert\_\{L^\{2\}\}\\leq\\lVert h\\rVert\_\{L^\{2\}\}for every square\-integrable fieldhh\. TakingL2L^\{2\}norms in the exact formula and applying this inequality toh=ℒ∇Fr,σh=\\mathcal\{L\}\\nabla F\_\{r,\\sigma\}proves[Equation43](https://arxiv.org/html/2609.10980#S4.E43)\. ∎

The bound shows that the score mismatch is small whenσ2/r\\sigma^\{2\}/ris small and the smoothed gradient has little variation in the matrix\-frequency directions measured byℒ\\mathcal\{L\}\. Thus, increasing the perturbation rank or decreasing its radius brings the population field closer to the gradient of the objective smoothed under the sampled perturbations\.

## 5Conservativity of the population field

The preceding section answers what EGGROLL follows in expectation: a resolvent\-filtered gradient of the finite\-rank\-smoothed objective\. This identity does not by itself imply that the resulting field is necessarily the gradient of some other scalar objective\. We now ask when such an objective exists and whether its absence can change optimization behavior\. Throughout this section, we vectorize the matrix parameter and writed=m​nd=mn, so the fields are defined onℝd\\mathbb\{R\}^\{d\}\. On the simply connected domainℝd\\mathbb\{R\}^\{d\}, a continuously differentiable field is the gradient of a scalar potential if and only if its Jacobian is symmetric\[[40](https://arxiv.org/html/2609.10980#bib.bib28)\]\. This property is called*conservativity*\. We first characterize the perturbation laws for which every population field is conservative, then construct a finite\-rank example in which the altered field changes the local stability of an optimum\.

The question of conservativity admits a general answer for any given perturbation distribution\. LetZ∈ℝdZ\\in\\mathbb\{R\}^\{d\}be integrable and centered\. The characteristic functionΦ⁡\(t\)=𝔼⁡\[ei​t⊤​Z\]\\Phi\(t\)=\\mathbb\{E\}\\big\[e^\{it^\{\\top\}Z\}\\big\]is continuously differentiable, with∇Φ​\(t\)=i​𝔼​\[Z​ei​t⊤​Z\]\\nabla\\Phi\(t\)=i\\mathbb\{E\}\[Ze^\{it^\{\\top\}Z\}\]\. Define

\(𝒯Z,σ​f\)​\(x\)=σ−1​𝔼​\[Z​f​\(x\+σ​Z\)\]\.\(\\mathcal\{T\}\_\{Z,\\sigma\}f\)\(x\)=\\sigma^\{\-1\}\\mathbb\{E\}\[Zf\(x\+\\sigma Z\)\]\.\(47\)This is the same population\-field construction with the finite\-rank perturbationErE\_\{r\}replaced by a general random vectorZZ\.

Real\-valued trigonometric polynomials are finite sums of sine and cosine Fourier modes\. Since𝒯Z,σ\\mathcal\{T\}\_\{Z,\\sigma\}is linear inff, conservativity can be analyzed one mode at a time, and a single nonconservative mode rules out universal conservativity\. The following theorem characterizes the perturbation laws for which𝒯Z,σ​f\\mathcal\{T\}\_\{Z,\\sigma\}fis conservative for every real\-valued trigonometric polynomial and everyσ\>0\\sigma\>0\.

###### Theorem 5\.1\(Universal conservativity if and only if spherical symmetry\)\.

Supposed≥2d\\geq 2,𝔼⁡\[Z\]=0\\mathbb\{E\}\[Z\]=0, and𝔼⁡\[∥Z∥\]<∞\\mathbb\{E\}\[\\lVert Z\\rVert\]<\\infty\. The following are equivalent:

1. \(i\)for everyσ\>0\\sigma\>0and every real\-valued trigonometric polynomialf⁡\(x\)=∑k=1K\[ak​cos⁡\(tk⊤​x\)\+bk​sin⁡\(tk⊤​x\)\]f\(x\)=\\sum\_\{k=1\}^\{K\}\[a\_\{k\}\\cos\(t\_\{k\}^\{\\top\}x\)\+b\_\{k\}\\sin\(t\_\{k\}^\{\\top\}x\)\], the continuously differentiable field𝒯Z,σ​f\\mathcal\{T\}\_\{Z,\\sigma\}fis the gradient of aC2C^\{2\}potential onℝd\\mathbb\{R\}^\{d\}\.
2. \(ii\)for everyt∈ℝdt\\in\\mathbb\{R\}^\{d\},∇Φ​\(t\)=λ​\(t\)​t\\nabla\\Phi\(t\)=\\lambda\(t\)tfor some complex scalarλ⁡\(t\)\\lambda\(t\)\.
3. \(iii\)Φ\\Phiis radial\.
4. \(iv\)the law ofZZis spherically symmetric\.

###### Proof\.

We first prove the equivalence of \(i\) and \(ii\)\. Fixt∈ℝdt\\in\\mathbb\{R\}^\{d\}and consider the complex Fourier modeft​\(x\)=ei​t⊤​xf\_\{t\}\(x\)=e^\{it^\{\\top\}x\}\. The real and imaginary parts of this mode arecos⁡\(t⊤​x\)\\cos\(t^\{\\top\}x\)andsin⁡\(t⊤​x\)\\sin\(t^\{\\top\}x\), so applying𝒯Z,σ\\mathcal\{T\}\_\{Z,\\sigma\}toftf\_\{t\}records its action on both real modes\. From the definition ofΦ\\Phi,

∇Φ​\(σ​t\)=i​𝔼​\[Z​ei​σ​t⊤​Z\]\.\\nabla\\Phi\(\\sigma t\)=i\\mathbb\{E\}\\left\[Ze^\{i\\sigma t^\{\\top\}Z\}\\right\]\.\(48\)Consequently,

\(𝒯Z,σft\)\(x\)=ei​t⊤​xσ𝔼\[Zei​σ​t⊤​Z\]=1i​σ∇Φ\(σt\)ei​t⊤​x\.\(\\mathcal\{T\}\_\{Z,\\sigma\}f\_\{t\}\)\(x\)=\\frac\{e^\{it^\{\\top\}x\}\}\{\\sigma\}\\mathbb\{E\}\\left\[Ze^\{i\\sigma t^\{\\top\}Z\}\\right\]=\\frac\{1\}\{i\\sigma\}\\nabla\\Phi\(\\sigma t\)e^\{it^\{\\top\}x\}\.\(49\)A vector field of the formc​ei​t⊤​xce^\{it^\{\\top\}x\}is a gradient field if and only if its coefficientc∈ℂdc\\in\\mathbb\{C\}^\{d\}is parallel tott\. Indeed, ifc=γ​tc=\\gamma t, then

c​ei​t⊤​x=∇\(γi​ei​t⊤​x\)\.ce^\{it^\{\\top\}x\}=\\nabla\\left\(\\frac\{\\gamma\}\{i\}e^\{it^\{\\top\}x\}\\right\)\.\(50\)Conversely, the Jacobian ofc​ei​t⊤​xce^\{it^\{\\top\}x\}isi​c​t⊤​ei​t⊤​xict^\{\\top\}e^\{it^\{\\top\}x\}\. Any continuously differentiable gradient field has a symmetric Jacobian because its Jacobian is the Hessian of its potential\. Fort≠0t\\neq 0, symmetry of the present Jacobian is equivalent toc​t⊤=t​c⊤ct^\{\\top\}=tc^\{\\top\}\. Choose an indexjjfor whichtj≠0t\_\{j\}\\neq 0\. Comparing the\(i,j\)\(i,j\)entries givesci​tj=ti​cjc\_\{i\}t\_\{j\}=t\_\{i\}c\_\{j\}for everyii, and hencec=\(cj/tj\)​tc=\(c\_\{j\}/t\_\{j\}\)t\. Thus the Jacobian is symmetric only whenccandttare parallel\. Equation \([49](https://arxiv.org/html/2609.10980#S5.E49)\) is therefore conservative precisely when∇Φ​\(σ​t\)\\nabla\\Phi\(\\sigma t\)is parallel tott\. Sinceσ\>0\\sigma\>0andttare arbitrary, this condition is equivalent to \(ii\)\. Linearity then extends the conclusion from individual sine and cosine modes to every trigonometric polynomial, proving \(i\)⇔\\Leftrightarrow\(ii\)\. The zero\-frequency mode causes no exception because∇Φ​\(0\)=i​𝔼​\[Z\]=0\\nabla\\Phi\(0\)=i\\mathbb\{E\}\[Z\]=0\.

We next prove \(ii\)⇔\\Leftrightarrow\(iii\)\. Suppose \(ii\) holds\. Ifv⊤​t=0v^\{\\top\}t=0, thenvvis tangent atttto the sphere of radius∥t∥\\lVert t\\rVert, and

v⊤∇Φ\(t\)=λ\(t\)v⊤t=0\.v^\{\\top\}\\nabla\\Phi\(t\)=\\lambda\(t\)v^\{\\top\}t=0\.\(51\)ThusΦ\\Phihas zero directional derivative along every tangent direction to each sphere\. Spheres are connected whend≥2d\\geq 2, soΦ\\Phiis constant on every sphere and hence depends only on∥t∥\\lVert t\\rVert\. This is precisely radiality\. Conversely, ifΦ\\Phiis radial, then it is constant in every direction tangent to a sphere\. The gradient is therefore orthogonal to every tangent direction and must be parallel to the radial directiontt, which gives \(ii\)\.

Finally, \(iii\)⇔\\Leftrightarrow\(iv\) is the standard characteristic\-function characterization of spherical symmetry\[[31](https://arxiv.org/html/2609.10980#bib.bib6)\]\. If the law ofZZis spherically symmetric, thenQ​ZQZandZZhave the same law for every orthogonal matrixQQ, and henceΦ⁡\(Q⊤​t\)=Φ⁡\(t\)\\Phi\(Q^\{\\top\}t\)=\\Phi\(t\)\. ThereforeΦ\\Phidepends only on∥t∥\\lVert t\\rVert\. Conversely, ifΦ\\Phiis radial, the characteristic function ofQ​ZQZis

𝔼⁡\[ei​t⊤​Q​Z\]=Φ⁡\(Q⊤​t\)=Φ⁡\(t\)\.\\mathbb\{E\}\\left\[e^\{it^\{\\top\}QZ\}\\right\]=\\Phi\(Q^\{\\top\}t\)=\\Phi\(t\)\.\(52\)Uniqueness of characteristic functions then implies thatQ​ZQZandZZhave the same law, so the law is spherically symmetric\. The assumptiond≥2d\\geq 2is necessary because in one dimension every continuous scalar field is conservative, regardless of whether the perturbation law is symmetric\. ∎

Theorem[5\.1](https://arxiv.org/html/2609.10980#S5.Thmtheorem1)shows that universal conservativity requires the perturbation law to be invariant under every rotation ofℝd\\mathbb\{R\}^\{d\}\. We have shown in[Section4](https://arxiv.org/html/2609.10980#S4)that finite\-rank EGGROLL has only part of this symmetry\. The perturbation law is unchanged under transformationsEr↦U​Er​V⊤E\_\{r\}\\mapsto UE\_\{r\}V^\{\\top\}, but not under arbitrary rotations of them​nmnvectorized matrix coordinates\. Equivalently,[Equation11](https://arxiv.org/html/2609.10980#S4.E11)depends on the complete singular\-value profile of the frequency matrix rather than only on its Frobenius norm\. Thus, whenm,n≥2m,n\\geq 2andr<∞r<\\infty, the perturbation law is not spherical, and[Theorem5\.1](https://arxiv.org/html/2609.10980#S5.Thmtheorem1)implies that there exists a trigonometric\-polynomial objective which has a nonconservative population field\. No nondegenerate spherically symmetric law can remain supported on rank\-deficient matrices in this setting: conditional on any nonzero Frobenius norm, spherical symmetry makes the vectorized direction uniform on a sphere, while the rank\-deficient matrices form a measure\-zero subset of that sphere\. Whenmin⁡\(m,n\)=1\\min\(m,n\)=1, the matrix reduces to a vector and the Gaussian\-product law is spherical, so this obstruction disappears\.

To exhibit this failure directly, we choose an objective for which the resolvent shrinks different singular directions by different amounts\. The resulting update coefficient is not parallel to the original Fourier frequency, which makes the Jacobian of the population field nonsymmetric\. The following diagonal mode is the smallest such example\.

###### Corollary 5\.2\(A bounded smooth nonconservative example\)\.

Letm=n=2m=n=2,T=diag⁡\(1,2\)T=\\operatorname\{diag\}\(1,2\), andf⁡\(W\)=cos⁡⟨T,W⟩Ff\(W\)=\\cos\\langle T,\\,W\\rangle\_\{F\}\. Then

gr,σf\(W\)=−Φr\(σT\)\(I\+σ2​T​T⊤r\)−1Tsin⟨T,W⟩F\.g\_\{r,\\sigma\}^\{f\}\(W\)=\-\\Phi\_\{r\}\(\\sigma T\)\\left\(I\+\\frac\{\\sigma^\{2\}TT^\{\\top\}\}\{r\}\\right\)^\{\-1\}T\\sin\\langle T,\\,W\\rangle\_\{F\}\.\(53\)For every finiterrandσ\>0\\sigma\>0, its Jacobian is nonsymmetric atW=0W=0and at everyWWwithcos⁡⟨T,W⟩F≠0\\cos\\langle T,\\,W\\rangle\_\{F\}\\neq 0\. Hence the field is not conservative\.

###### Proof\.

Recall thatJ⁡\(T\)=\(I\+σ2​T​T⊤/r\)−1​TJ\(T\)=\(I\+\\sigma^\{2\}TT^\{\\top\}/r\)^\{\-1\}T\. After vectorization, differentiating[Equation53](https://arxiv.org/html/2609.10980#S5.E53)gives

Dgr,σf\(W\)=−Φr\(σT\)cos⟨T,W⟩Fvec\(J\(T\)\)vec\(T\)⊤\.Dg\_\{r,\\sigma\}^\{f\}\(W\)=\-\\Phi\_\{r\}\(\\sigma T\)\\cos\\langle T,\\,W\\rangle\_\{F\}\\,\\operatorname\{vec\}\(J\(T\)\)\\operatorname\{vec\}\(T\)^\{\\top\}\.\(54\)A nonzero rank\-one matrixu​v⊤uv^\{\\top\}is symmetric only ifuuandvvare parallel, since its column and row spaces are spanned byuuandvv, respectively, and these spaces coincide for a symmetric matrix\. Here

J⁡\(T\)=diag⁡\(11\+σ2/r,21\+4​σ2/r\),J\(T\)=\\operatorname\{diag\}\\left\(\\frac\{1\}\{1\+\\sigma^\{2\}/r\},\\frac\{2\}\{1\+4\\sigma^\{2\}/r\}\\right\),\(55\)which is not a scalar multiple ofT=diag⁡\(1,2\)T=\\operatorname\{diag\}\(1,2\)for finiterrandσ\>0\\sigma\>0\. The displayed Jacobian is therefore nonsymmetric whenevercos⁡⟨T,W⟩F≠0\\cos\\langle T,\\,W\\rangle\_\{F\}\\neq 0\. ∎

The preceding example establishes that the population field need not have a potential\. Possibly a more consequential question that follows is whether the change from the smoothed gradient to the population field can alter the optimization dynamics\. The next construction shows that this can indeed be the case\. Two Fourier modes are combined so that both the original objective and its finite\-rank\-smoothed version have a strict local maximum at the origin, while the rank\-one EGGROLL field points away from that maximum along a local direction\.

###### Proposition 5\.3\(A strict local maximum of the smoothed objective can be unstable under EGGROLL\)\.

Setm=n=2m=n=2,r=σ=1r=\\sigma=1,ε=0\.01\\varepsilon=0\.01, and

T1=\(−2−2−2−1\),T2=\(−222−1\)\.T\_\{1\}=\\begin\{pmatrix\}\-2&\-2\\\\ \-2&\-1\\end\{pmatrix\},\\qquad T\_\{2\}=\\begin\{pmatrix\}\-2&2\\\\ 2&\-1\\end\{pmatrix\}\.\(56\)For

f\(W\)=cos⟨T1,W⟩F\+14cos⟨T2,W⟩F\+ε∑i,jcosWi​j,f\(W\)=\\cos\\langle T\_\{1\},\\,W\\rangle\_\{F\}\+\\frac\{1\}\{4\}\\cos\\langle T\_\{2\},\\,W\\rangle\_\{F\}\+\\varepsilon\\sum\_\{i,j\}\\cos W\_\{ij\},\(57\)W=0W=0is a strict local maximum of bothffandF1,1F\_\{1,1\}\. The Jacobian of rank\-one population EGGROLL ascent at zero has a positive eigenvalue

λ\+=1737−35144​2−ε2​2≈0\.02925\.\\lambda\_\{\+\}=\\frac\{\\sqrt\{1737\}\-35\}\{144\\sqrt\{2\}\}\-\\frac\{\\varepsilon\}\{2\\sqrt\{2\}\}\\approx 0\.02925\.\(58\)Thus the strict optimum is linearly unstable\.

###### Proof\.

Lettq=vec⁡\(Tq\)t\_\{q\}=\\operatorname\{vec\}\(T\_\{q\}\)\. The Hessian offfat zero is

−t1​t1⊤−14​t2​t2⊤−ε​I4≺0\.\-t\_\{1\}t\_\{1\}^\{\\top\}\-\\frac\{1\}\{4\}t\_\{2\}t\_\{2\}^\{\\top\}\-\\varepsilon I\_\{4\}\\prec 0\.\(59\)Smoothing multiplies the first two cosine coefficients byΦ1​\(T1\)=Φ1​\(T2\)=1/\(3​2\)\\Phi\_\{1\}\(T\_\{1\}\)=\\Phi\_\{1\}\(T\_\{2\}\)=1/\(3\\sqrt\{2\}\)and every coordinate cosine byΦ1​\(ei​j\)=1/2\\Phi\_\{1\}\(e\_\{ij\}\)=1/\\sqrt\{2\}:

∇2F1,1​\(0\)=−13​2​\(t1​t1⊤\+14​t2​t2⊤\)−ε2​I4≺0\.\\nabla^\{2\}F\_\{1,1\}\(0\)=\-\\frac\{1\}\{3\\sqrt\{2\}\}\\left\(t\_\{1\}t\_\{1\}^\{\\top\}\+\\frac\{1\}\{4\}t\_\{2\}t\_\{2\}^\{\\top\}\\right\)\-\\frac\{\\varepsilon\}\{\\sqrt\{2\}\}I\_\{4\}\\prec 0\.\(60\)Hence, the origin remains a strict local maximum after smoothing\.

We next compute the Jacobian of the population field\. For a cosine mode with frequencyTT,[Equation53](https://arxiv.org/html/2609.10980#S5.E53)gives the contribution

−Φ1​\(T\)​vec⁡\(J⁡\(T\)\)​vec​\(T\)⊤\-\\Phi\_\{1\}\(T\)\\operatorname\{vec\}\(J\(T\)\)\\operatorname\{vec\}\(T\)^\{\\top\}\(61\)at the origin\. We have

J⁡\(T1\)=\(0−1/3−1/31/6\),J⁡\(T2\)=\(01/31/31/6\)\.J\(T\_\{1\}\)=\\begin\{pmatrix\}0&\-1/3\\\\ \-1/3&1/6\\end\{pmatrix\},\\qquad J\(T\_\{2\}\)=\\begin\{pmatrix\}0&1/3\\\\ 1/3&1/6\\end\{pmatrix\}\.\(62\)Therefore, the contribution of the first two cosine modes is

A0=−13​2​\(vec⁡\(J⁡\(T1\)\)​t1⊤\+14​vec⁡\(J⁡\(T2\)\)​t2⊤\)\.A\_\{0\}=\-\\frac\{1\}\{3\\sqrt\{2\}\}\\left\(\\operatorname\{vec\}\(J\(T\_\{1\}\)\)t\_\{1\}^\{\\top\}\+\\frac\{1\}\{4\}\\operatorname\{vec\}\(J\(T\_\{2\}\)\)t\_\{2\}^\{\\top\}\\right\)\.\(63\)Substituting the displayed matrices and expanding the determinant gives

det\(λ​I4−A0\)=λ2​\(λ2\+35​2144​λ−181\)\.\\det\(\\lambda I\_\{4\}\-A\_\{0\}\)=\\lambda^\{2\}\\left\(\\lambda^\{2\}\+\\frac\{35\\sqrt\{2\}\}\{144\}\\lambda\-\\frac\{1\}\{81\}\\right\)\.\(64\)The positive root of the quadratic factor is

λ\+​\(A0\)=1737−35144​2\.\\lambda\_\{\+\}\(A\_\{0\}\)=\\frac\{\\sqrt\{1737\}\-35\}\{144\\sqrt\{2\}\}\.\(65\)For a coordinate modeT=ei​jT=e\_\{ij\}, we haveJ⁡\(ei​j\)=ei​j/2J\(e\_\{ij\}\)=e\_\{ij\}/2andΦ1​\(ei​j\)=1/2\\Phi\_\{1\}\(e\_\{ij\}\)=1/\\sqrt\{2\}\. The corresponding Jacobian contribution is−εvec\(ei​j\)vec\(ei​j\)⊤/\(22\)\-\\varepsilon\\operatorname\{vec\}\(e\_\{ij\}\)\\operatorname\{vec\}\(e\_\{ij\}\)^\{\\top\}/\(2\\sqrt\{2\}\)\. Summing over the four coordinates gives−εI4/\(22\)\-\\varepsilon I\_\{4\}/\(2\\sqrt\{2\}\)\. This scalar shift subtractsε/\(2​2\)\\varepsilon/\(2\\sqrt\{2\}\)from every eigenvalue ofA0A\_\{0\}, so the larger eigenvalue of the full Jacobian is

1737−35144​2−ε2​2,\\frac\{\\sqrt\{1737\}\-35\}\{144\\sqrt\{2\}\}\-\\frac\{\\varepsilon\}\{2\\sqrt\{2\}\},\(66\)which is positive atε=0\.01\\varepsilon=0\.01\. ∎

The proposition highlights the contrast between gradient ascent on the smoothed objective and the mean EGGROLL dynamics at the stated choicer=σ=1r=\\sigma=1\. Ordinary ascent onF1,1F\_\{1,1\}follows

Wk\+1=Wk\+η∇F1,1\(Wk\),W\_\{k\+1\}=W\_\{k\}\+\\eta\\nabla F\_\{1,1\}\(W\_\{k\}\),\(67\)whereas the deterministic rank\-one EGGROLL population update at radius one is

Wk\+1=Wk\+η​g1,1f​\(Wk\)\.W\_\{k\+1\}=W\_\{k\}\+\\eta g\_\{1,1\}^\{f\}\(W\_\{k\}\)\.\(68\)Both fields vanish at the origin\. Because∇2F1,1​\(0\)\\nabla^\{2\}F\_\{1,1\}\(0\)is negative definite, sufficiently small gradient\-ascent steps on the smoothed objective contract toward zero\. By contrast, we show thatD​g1,1f​\(0\)Dg\_\{1,1\}^\{f\}\(0\)has the positive eigenvalueλ\+\\lambda\_\{\+\}in[Equation58](https://arxiv.org/html/2609.10980#S5.E58)\. The linearized EGGROLL update therefore has eigenvalue1\+η​λ\+\>11\+\\eta\\lambda\_\{\+\}\>1in the corresponding direction and repels nearby points from the same fixed point\.

Figure 3:Local linearized dynamics for the bounded cosine objective in[Equation57](https://arxiv.org/html/2609.10980#S5.E57)\. For each field, letAAdenote its Jacobian at the origin\. The curves solveδ˙=A​δ\\dot\{\\delta\}=A\\deltaand are projected onto the same two\-dimensional plane, spanned by the eigenvectors of the rank\-one Jacobian with the largest and smallest real eigenvalues\. Along an eigendirection, a displacement scales aseλ​te^\{\\lambda t\}, so positive and negative real parts correspond to local repulsion and attraction and agree with the stability of sufficiently small discrete mean\-update steps\. Dots mark initial displacements, arrowheads show increasing time, and the star marks the origin, which is a strict local maximum of both the objective and its finite\-rank\-smoothed version\. Rank\-one EGGROLL repels along one direction, whereas rank two and the rank extrapolation through Richardson combination2​g2​r,σ−gr,σ2g\_\{2r,\\sigma\}\-g\_\{r,\\sigma\}\(elaborated upon in[CorollaryB\.1](https://arxiv.org/html/2609.10980#A2.Thmtheorem1)\) are locally attracting\.
## 6Finite\-rank dynamics near quadratic objectives

The preceding section shows that finite\-rank EGGROLL dynamics can reverse the local stability of an optimum\. We now ask what property of the objective, if any, prevents this reversal\. Quadratic objectives provide a useful dividing line\. EGGROLL is exact on this entire class at every rank and perturbation radius\. When an objective is only approximately quadratic over the scale of the sampled perturbations, a local expansion identifies the first error caused by ordinary smoothing and the additional error caused by finite rank\. Standard smoothness assumptions then give nonasymptotic field\-error bounds without imposing global Fourier regularity\. Routine stationarity consequences are collected in[AppendixC](https://arxiv.org/html/2609.10980#A3)\.

We start by observing that for any quadraticffand any matrix directionEE, the antithetic difference is exactly equal to the directional derivative offfatWWalongEE:

f⁡\(W\+σ​E\)−f⁡\(W−σ​E\)2​σ=⟨∇f​\(W\),E⟩F\.\\frac\{f\(W\+\\sigma E\)\-f\(W\-\\sigma E\)\}\{2\\sigma\}=\\langle\\nabla f\(W\),\\,E\\rangle\_\{F\}\.\(69\)Substituting the random EGGROLL directionE=ErE=E\_\{r\}, multiplying byErE\_\{r\}, and taking expectations recovers∇f​\(W\)\\nabla f\(W\)by identity covariance\. Consequently, for any quadratic objective, including one with arbitrary linear coefficients and cross\-coordinate terms in its Hessian, the EGGROLL population field equals the true gradient at every finite rank\.

###### Proposition 6\.1\(Exactness on quadratic objectives\)\.

Letf⁡\(W\)=12​⟨W,ℋ​W⟩F\+⟨C,W⟩F\+cf\(W\)=\\frac\{1\}\{2\}\\langle W,\\,\\mathcal\{H\}W\\rangle\_\{F\}\+\\langle C,\\,W\\rangle\_\{F\}\+c, whereℋ\\mathcal\{H\}is self\-adjoint\. Then

gr,σf​\(W\)=∇f​\(W\)g\_\{r,\\sigma\}^\{f\}\(W\)=\\nabla f\(W\)\(70\)for everyr≥1r\\geq 1andσ\>0\\sigma\>0\.

###### Proof\.

The preceding antithetic identity gives

gr,σf​\(W\)=𝔼⁡\[Er​⟨∇f​\(W\),Er⟩F\]=∇f​\(W\),g\_\{r,\\sigma\}^\{f\}\(W\)=\\mathbb\{E\}\\left\[E\_\{r\}\\langle\\nabla f\(W\),\\,E\_\{r\}\\rangle\_\{F\}\\right\]=\\nabla f\(W\),\(71\)where the last equality follows from𝔼⁡\[vec⁡\(Er\)​vec⁡\(Er\)⊤\]=I\\mathbb\{E\}\[\\operatorname\{vec\}\(E\_\{r\}\)\\operatorname\{vec\}\(E\_\{r\}\)^\{\\top\}\]=I\. ∎

This result gives us an idea of why the optimum\-repelling example of[Proposition5\.3](https://arxiv.org/html/2609.10980#S5.Thmtheorem3)fails: It falls outside this quadratic class over the distances sampled by its perturbations\. For either dominant termcos⁡⟨Tj,W⟩F\\cos\\langle T\_\{j\},\\,W\\rangle\_\{F\}, perturbing the origin atσ=1\\sigma=1produces the scalar argument

Xj=⟨Tj,E1⟩F\.X\_\{j\}=\\langle T\_\{j\},\\,E\_\{1\}\\rangle\_\{F\}\.\(72\)By identity covariance,𝔼⁡\[Xj2\]=∥Tj∥F2=13\\mathbb\{E\}\[X\_\{j\}^\{2\}\]=\\lVert T\_\{j\}\\rVert\_\{F\}^\{2\}=13\. Consequently, the expected quadratic approximation and the expected cosine are

𝔼⁡\[1−Xj22\]=−112,𝔼⁡\[cos⁡Xj\]=Φ1​\(Tj\)=13​2,\\mathbb\{E\}\\left\[1\-\\frac\{X\_\{j\}^\{2\}\}\{2\}\\right\]=\-\\frac\{11\}\{2\},\\qquad\\mathbb\{E\}\[\\cos X\_\{j\}\]=\\Phi\_\{1\}\(T\_\{j\}\)=\\frac\{1\}\{3\\sqrt\{2\}\},\(73\)so the quadratic approximation does a relatively poor job of estimating the objective over this perturbation distribution\. This discrepancy is large enough to account for the stability reversal\. Letfquadf\_\{\\mathrm\{quad\}\}be the quadratic Taylor approximation of the complete objective at the origin\. The functionsfquadf\_\{\\mathrm\{quad\}\}andffhave the same value, gradient, and Hessian there, but quadratic exactness gives

D​g1,1fquad​\(0\)=∇2f​\(0\)⪯−ε​I4,Dg\_\{1,1\}^\{f\_\{\\mathrm\{quad\}\}\}\(0\)=\\nabla^\{2\}f\(0\)\\preceq\-\\varepsilon I\_\{4\},\(74\)whereasD​g1,1f​\(0\)Dg\_\{1,1\}^\{f\}\(0\)has the positive eigenvalue in[Equation58](https://arxiv.org/html/2609.10980#S5.E58)\. The two objectives have identical local curvature at zero\. Their different EGGROLL dynamics arise because the perturbed evaluations reach points where the cosine is no longer well approximated by its quadratic Taylor polynomial\.

For a general smooth objective, exactness is lost in two stages\. Even before the effect of the low\-rank approximation and the score mismatch enters, dense Gaussian smoothing changes the gradient\. IfE∞E\_\{\\infty\}is a standard Gaussian matrix, then under the regularity conditions below,

∇F∞,σ\(W\)=∇f\(W\)\+σ22∇Δf\(W\)\+O\(σ4𝔼\[∥E∞∥F4\]\)\.\\nabla F\_\{\\infty,\\sigma\}\(W\)=\\nabla f\(W\)\+\\frac\{\\sigma^\{2\}\}\{2\}\\nabla\\Delta f\(W\)\+O\\left\(\\sigma^\{4\}\\mathbb\{E\}\\big\[\\lVert E\_\{\\infty\}\\rVert\_\{F\}^\{4\}\\big\]\\right\)\.\(75\)The term\(σ2/2\)∇Δf\(\\sigma^\{2\}/2\)\\nabla\\Delta fis therefore the leading change caused by dense Gaussian smoothing, before finite rank enters\. The next proposition shows that the finite\-rank Gaussian\-product law retains this term and adds a second correction proportional to1/r1/r\. We note∥⋅∥op\\lVert\\cdot\\rVert\_\{\\mathrm\{op\}\}for the multilinear operator norm induced by the Frobenius norm on each matrix argument\.

###### Proposition 6\.2\(Finite\-rank expansion for small perturbation radius\)\.

Fixm,nm,nandrr\. Iff∈C5f\\in C^\{5\}andM5:=supW∥D5​f​\(W\)∥op<∞M\_\{5\}:=\\sup\_\{W\}\\lVert D^\{5\}f\(W\)\\rVert\_\{\\mathrm\{op\}\}<\\infty, then, uniformly inWWasσ→0\\sigma\\to 0,

gr,σf=∇f\+σ22∇Δf\+σ2r𝒟f\+Rr,σ,supW∥Rr,σ\(W\)∥F≤σ4​M5120𝔼\[∥Er∥F6\],g\_\{r,\\sigma\}^\{f\}=\\nabla f\+\\frac\{\\sigma^\{2\}\}\{2\}\\nabla\\Delta f\+\\frac\{\\sigma^\{2\}\}\{r\}\\mathcal\{D\}f\+R\_\{r,\\sigma\},\\qquad\\sup\_\{W\}\\lVert R\_\{r,\\sigma\}\(W\)\\rVert\_\{F\}\\leq\\frac\{\\sigma^\{4\}M\_\{5\}\}\{120\}\\mathbb\{E\}\[\\lVert E\_\{r\}\\rVert\_\{F\}^\{6\}\],\(76\)HereΔ​f\\Delta fis the Laplacian offfover them​nmnmatrix coordinates,

Δ​f​\(W\)=∑k=1m∑ℓ=1n∂2f⁡\(W\)∂Wk​ℓ2,\\Delta f\(W\)=\\sum\_\{k=1\}^\{m\}\\sum\_\{\\ell=1\}^\{n\}\\frac\{\\partial^\{2\}f\(W\)\}\{\\partial W\_\{k\\ell\}^\{2\}\},\(77\)The notation∂a​b\\partial\_\{ab\}denotes differentiation with respect to the matrix coordinateWa​bW\_\{ab\}, and

\(𝒟​f\)i​j=∑k=1m∑ℓ=1n∂i​ℓ∂k​ℓ∂k​jf\.\(\\mathcal\{D\}f\)\_\{ij\}=\\sum\_\{k=1\}^\{m\}\\sum\_\{\\ell=1\}^\{n\}\\partial\_\{i\\ell\}\\partial\_\{k\\ell\}\\partial\_\{kj\}f\.\(78\)The first correction is shared with dense Gaussian smoothing\. The additional term\(σ2/r\)​𝒟​f\(\\sigma^\{2\}/r\)\\mathcal\{D\}fis specific to the Gaussian\-product perturbation law, vanishes asr→∞r\\to\\infty, and need not be a gradient field\.

###### Proof\.

Taylor expansion of the antithetic update gives

gr,σf​\(W\)=∇f​\(W\)\+σ26​𝔼​\[Er​D3​f​\(W\)​\[Er,Er,Er\]\]\+Rr,σ​\(W\),g\_\{r,\\sigma\}^\{f\}\(W\)=\\nabla f\(W\)\+\\frac\{\\sigma^\{2\}\}\{6\}\\mathbb\{E\}\\left\[E\_\{r\}D^\{3\}f\(W\)\[E\_\{r\},E\_\{r\},E\_\{r\}\]\\right\]\+R\_\{r,\\sigma\}\(W\),\(79\)where identity covariance was used for the first term\. The order\-σ2\\sigma^\{2\}expectation is

𝔼\[ErD3f\(W\)\[Er,Er,Er\]\]=3∇Δf\(W\)\+6r𝒟f\(W\)\.\\mathbb\{E\}\\left\[E\_\{r\}D^\{3\}f\(W\)\[E\_\{r\},E\_\{r\},E\_\{r\}\]\\right\]=3\\nabla\\Delta f\(W\)\+\\frac\{6\}\{r\}\\mathcal\{D\}f\(W\)\.\(80\)This identity follows by writingEr=r−1/2∑q=1raqbq⊤E\_\{r\}=r^\{\-1/2\}\\sum\_\{q=1\}^\{r\}a\_\{q\}b\_\{q\}^\{\\top\}and applying the Gaussian fourth\-moment formula\[[26](https://arxiv.org/html/2609.10980#bib.bib29)\]separately to the independent row and column factors\. The terms that also occur for a dense Gaussian matrix give3∇Δf3\\nabla\\Delta f\. The remaining product\-law contribution gives6​𝒟​f/r6\\mathcal\{D\}f/r\. Substitution into[Equation79](https://arxiv.org/html/2609.10980#S6.E79)yields the two order\-σ2\\sigma^\{2\}terms in[Equation76](https://arxiv.org/html/2609.10980#S6.E76)\. Appendix[D](https://arxiv.org/html/2609.10980#A4)provides a more in\-depth version of this calculation, including the provenance of the coefficient1/r1/r\.

Finally, letM5=supW∥D5​f​\(W\)∥opM\_\{5\}=\\sup\_\{W\}\\lVert D^\{5\}f\(W\)\\rVert\_\{\\mathrm\{op\}\}\. Taylor’s theorem bounds the matrix\-valued remainder in[Equation79](https://arxiv.org/html/2609.10980#S6.E79)by

∥Rr,σ​\(W\)∥F≤σ4​M5120​𝔼​\[∥Er∥F6\]\.\\lVert R\_\{r,\\sigma\}\(W\)\\rVert\_\{F\}\\leq\\frac\{\\sigma^\{4\}M\_\{5\}\}\{120\}\\mathbb\{E\}\[\\lVert E\_\{r\}\\rVert\_\{F\}^\{6\}\]\.\(81\)∎

The expansion now separates the two costs\. Smoothing by the sampled perturbation law changes∇f\\nabla finto∇Fr,σ\\nabla F\_\{r,\\sigma\}, with the same leading correction as dense Gaussian smoothing\. Finite\-rank score mismatch then changes∇Fr,σ\\nabla F\_\{r,\\sigma\}intogr,σfg\_\{r,\\sigma\}^\{f\}\. For fixedm,nm,n, andrr, the preceding calculation and the corresponding expansion of the smoothed gradient give, asσ→0\\sigma\\to 0,

∇Fr,σ−∇f=σ22∇Δf\+O\(σ4\),gr,σf−∇Fr,σ=σ2r​𝒟​f\+O⁡\(σ4\)\.\\begin\{gathered\}\\nabla F\_\{r,\\sigma\}\-\\nabla f=\\frac\{\\sigma^\{2\}\}\{2\}\\nabla\\Delta f\+O\(\\sigma^\{4\}\),\\\\ g\_\{r,\\sigma\}^\{f\}\-\\nabla F\_\{r,\\sigma\}=\\frac\{\\sigma^\{2\}\}\{r\}\\mathcal\{D\}f\+O\(\\sigma^\{4\}\)\.\\end\{gathered\}\(82\)The first line is the leading change caused by smoothing and is shared with dense Gaussian ES\. The second isolates the additional effect of using the dense\-Gaussian score with finite\-rank perturbations, whose leading term depends on third derivatives, is proportional toσ2/r\\sigma^\{2\}/r, vanishes on quadratics, and need not itself be a gradient field\. The constants hidden in the two remainders may depend on the dimensions, rank, and derivative bounds\.

It is also of note to clarify what exactly constitutes a “small perturbation radius,” within which the local expansion remains valid, in a high\-dimensional matrix space\. The expansion is local in the complete matrix displacementσ​Er\\sigma E\_\{r\}\. Since𝔼⁡\[∥Er∥F2\]=d\\mathbb\{E\}\[\\lVert E\_\{r\}\\rVert\_\{F\}^\{2\}\]=d, a typical displacement has Frobenius norm of orderσ​d\\sigma\\sqrt\{d\}\. The accuracy of the expansion also depends onM5M\_\{5\}and on𝔼⁡\[∥Er∥F6\]\\mathbb\{E\}\[\\lVert E\_\{r\}\\rVert\_\{F\}^\{6\}\]\. A numerically smallσ\\sigmamay therefore be insufficient in high dimension or in a region with large higher\-order derivatives\. The relevant regime is one in which the sampled perturbations remain within a region where the low\-order Taylor approximation is accurate\.

The preceding analysis identifies where finite\-rank error first enters, but its fifth\-order remainder is more detailed than necessary for a general field\-error bound\. Classical analyses of Gaussian smoothing instead assume a Lipschitz gradient, or a Lipschitz Hessian for a sharper radius dependence\[[32](https://arxiv.org/html/2609.10980#bib.bib11),[19](https://arxiv.org/html/2609.10980#bib.bib30)\]\. We now follow that route while retaining the exact fourth moment of the Gaussian\-product perturbation\.

### 6\.1Field error under standard smoothness assumptions

We switch to minimization in this subsection by writingh:=−fh:=\-fandgr,σh=−gr,σfg\_\{r,\\sigma\}^\{h\}=\-g\_\{r,\\sigma\}^\{f\}\. The differencegr,σh−∇hg\_\{r,\\sigma\}^\{h\}\-\\nabla hdepends on the fourth moment of the perturbation norm\. This moment can be computed exactly\. Conditional onAA, thenncolumns ofEr=A​B⊤/rE\_\{r\}=AB^\{\\top\}/\\sqrt\{r\}are independent Gaussian vectors with covarianceC=A​A⊤/rC=AA^\{\\top\}/r, and hence

𝔼B​\[∥Er∥F4∣A\]=n2​\(tr⁡C\)2\+2​n​tr⁡\(C2\)\.\\mathbb\{E\}\_\{B\}\[\\lVert E\_\{r\}\\rVert\_\{F\}^\{4\}\\mid A\]=n^\{2\}\(\\operatorname\{tr\}C\)^\{2\}\+2n\\operatorname\{tr\}\(C^\{2\}\)\.\(83\)For GaussianA∈ℝm×rA\\in\\mathbb\{R\}^\{m\\times r\},

𝔼⁡\[\(tr⁡C\)2\]=m2\+2​mr,𝔼⁡\[tr⁡\(C2\)\]=m⁡\(m\+r\+1\)r\.\\mathbb\{E\}\[\(\\operatorname\{tr\}C\)^\{2\}\]=m^\{2\}\+\\frac\{2m\}\{r\},\\qquad\\mathbb\{E\}\[\\operatorname\{tr\}\(C^\{2\}\)\]=\\frac\{m\(m\+r\+1\)\}\{r\}\.\(84\)Therefore, ford=m​nd=mn,

μ4​\(r\):=𝔼⁡\[∥Er∥F4\]=d⁡\(d\+2\)\+2​d​\(m\+n\+1\)r\.\\mu\_\{4\}\(r\):=\\mathbb\{E\}\[\\lVert E\_\{r\}\\rVert\_\{F\}^\{4\}\]=d\(d\+2\)\+\\frac\{2d\(m\+n\+1\)\}\{r\}\.\(85\)The corresponding fourth moment for independently perturbed network blocks is given in Appendix[A](https://arxiv.org/html/2609.10980#A1)\. The following lemma turns these moments into two nonasymptotic field\-error bounds\. The first assumes only that the gradient is Lipschitz\. The second uses a Lipschitz Hessian and gains an additional power ofσ\\sigmafrom the cancellation in the antithetic difference\.

###### Lemma 6\.3\(Field error under standard smoothness\)\.

SupposehhisLL\-smooth\. Then, for everyWW,

∥gr,σh​\(W\)−∇h​\(W\)∥F≤L​σ2​μ4​\(r\)3/4\.\\lVert g\_\{r,\\sigma\}^\{h\}\(W\)\-\\nabla h\(W\)\\rVert\_\{F\}\\leq\\frac\{L\\sigma\}\{2\}\\mu\_\{4\}\(r\)^\{3/4\}\.\(86\)If, in addition, the Hessian isMM\-Lipschitz as an operator, meaning that∥∇2h​\(W\)−∇2h​\(V\)∥op≤M​∥W−V∥F\\lVert\\nabla^\{2\}h\(W\)\-\\nabla^\{2\}h\(V\)\\rVert\_\{\\mathrm\{op\}\}\\leq M\\lVert W\-V\\rVert\_\{F\}for allW,VW,V, then

∥gr,σh​\(W\)−∇h​\(W\)∥F≤M​σ26​μ4​\(r\)\.\\lVert g\_\{r,\\sigma\}^\{h\}\(W\)\-\\nabla h\(W\)\\rVert\_\{F\}\\leq\\frac\{M\\sigma^\{2\}\}\{6\}\\mu\_\{4\}\(r\)\.\(87\)

###### Proof\.

For any matrixEE,LL\-smoothness gives

\|h⁡\(W\+σ​E\)−h⁡\(W−σ​E\)2​σ−⟨∇h​\(W\),E⟩F\|≤L​σ2​∥E∥F2\.\\left\|\\frac\{h\(W\+\\sigma E\)\-h\(W\-\\sigma E\)\}\{2\\sigma\}\-\\langle\\nabla h\(W\),\\,E\\rangle\_\{F\}\\right\|\\leq\\frac\{L\\sigma\}\{2\}\\lVert E\\rVert\_\{F\}^\{2\}\.\(88\)Multiplying byE=ErE=E\_\{r\}, taking expectations, and using identity covariance yields[Equation86](https://arxiv.org/html/2609.10980#S6.E86), since𝔼⁡\[∥Er∥F3\]≤μ4​\(r\)3/4\\mathbb\{E\}\[\\lVert E\_\{r\}\\rVert\_\{F\}^\{3\}\]\\leq\\mu\_\{4\}\(r\)^\{3/4\}\. If the Hessian isMM\-Lipschitz, the second\-order Taylor remainders atW±σ​EW\\pm\\sigma Eare bounded byM​σ3​∥E∥F3/6M\\sigma^\{3\}\\lVert E\\rVert\_\{F\}^\{3\}/6\. The quadratic terms cancel in the antithetic difference, leaving

\|h⁡\(W\+σ​E\)−h⁡\(W−σ​E\)2​σ−⟨∇h​\(W\),E⟩F\|≤M​σ26​∥E∥F3\.\\left\|\\frac\{h\(W\+\\sigma E\)\-h\(W\-\\sigma E\)\}\{2\\sigma\}\-\\langle\\nabla h\(W\),\\,E\\rangle\_\{F\}\\right\|\\leq\\frac\{M\\sigma^\{2\}\}\{6\}\\lVert E\\rVert\_\{F\}^\{3\}\.\(89\)Averaging after multiplication byErE\_\{r\}gives[Equation87](https://arxiv.org/html/2609.10980#S6.E87)\. ∎

The Lipschitz\-Hessian bound uses the exact fourth moment in[Equation85](https://arxiv.org/html/2609.10980#S6.E85)\. UnderLL\-smoothness alone, the same proof gives the slightly sharper coefficient\(L​σ/2\)​𝔼​\[∥Er∥F3\]\(L\\sigma/2\)\\mathbb\{E\}\[\\lVert E\_\{r\}\\rVert\_\{F\}^\{3\}\]\. The displayed form replaces this third moment byμ4​\(r\)3/4\\mu\_\{4\}\(r\)^\{3/4\}to obtain an explicit expression without any additional assumption\.

The same argument applies to the full collection of independently perturbed blocks\. Appendix[A](https://arxiv.org/html/2609.10980#A1)records the resulting network field\-error bound\. The section has supplied complementary answers to the instability exhibited in[Section5](https://arxiv.org/html/2609.10980#S5)\. The Taylor expansion identifies the dominant finite\-rank correction term\. The standard smoothness bounds control the complete field error without global Fourier assumptions\. Quadratic objectives require no approximation and are exact at every rank and radius\. The conventional descent consequences of the field bounds under smoothness conditions are recorded in[AppendixC](https://arxiv.org/html/2609.10980#A3)\.

All of the conclusions of this section concern the population mean\. A finite EGGROLL population introduces sampling error around that mean even when the antithetic difference has no truncation bias on a quadratic objective\. The next section computes this sampling error exactly\.

## 7Finite\-population variance in the local linear model

The preceding sections determine the update obtained after averaging over the perturbation law\. An actual EGGROLL step replaces this idealized expectation with an average over finitely many sampled directions\. We now ask how accurately that finite population estimates its mean\. We would expect this question to be particularly important at rank one\. Since every rank\-one direction occupies only a small part of the ambient matrix space, it is natural to worry that many more directions might be needed to obtain an accurate update\. Such a population increase could potentially erase the computational advantage that motivated low rank in the first place\.

We isolate the sampling variability of the perturbation law by taking the fitnessf⁡\(W\)f\(W\)to be deterministic111In many applications, including LLM post\-training, the evaluation procedure can also be stochastic because prompts, generated trajectories, or environment transitions are sampled\. The present calculation conditions on such randomness in order to isolate the variability due to the perturbations\.and replacing its behavior near the current weights by the first\-order approximation

f⁡\(W\+σ​E\)=f⁡\(W\)\+σ​⟨G,E⟩F\+O⁡\(σ2\),G=∇f​\(W\)\.f\(W\+\\sigma E\)=f\(W\)\+\\sigma\\langle G,\\,E\\rangle\_\{F\}\+O\(\\sigma^\{2\}\),\\qquad G=\\nabla f\(W\)\.\(90\)This is the*affine local model*, which removes curvature and stochastic fitness\-evaluation noise and leaves only the randomness of the sampled directions\. Under this model, one directionErE\_\{r\}gives the following estimator of the local gradientGG:

X=Er​⟨Er,G⟩F\.X=E\_\{r\}\\langle E\_\{r\},\\,G\\rangle\_\{F\}\.\(91\)This is an unbiased estimator becauseErE\_\{r\}has identity covariance\. Indeed, for everyi∈\[m\]i\\in\[m\]andj∈\[n\]j\\in\[n\],

𝔼⁡\[Xi​j\]=∑k=1m∑ℓ=1nGk​ℓ​𝔼​\[\(Er\)i​j​\(Er\)k​ℓ\]=∑k=1m∑ℓ=1nGk​ℓ​δi​k​δj​ℓ=Gi​j\.\\mathbb\{E\}\[X\_\{ij\}\]=\\sum\_\{k=1\}^\{m\}\\sum\_\{\\ell=1\}^\{n\}G\_\{k\\ell\}\\mathbb\{E\}\[\(E\_\{r\}\)\_\{ij\}\(E\_\{r\}\)\_\{k\\ell\}\]=\\sum\_\{k=1\}^\{m\}\\sum\_\{\\ell=1\}^\{n\}G\_\{k\\ell\}\\delta\_\{ik\}\\delta\_\{j\\ell\}=G\_\{ij\}\.\(92\)Thus𝔼⁡\[X\]=G\\mathbb\{E\}\[X\]=G\.

By variance of a matrix\-valued estimator, we mean its*total variance*, the sum of its entrywise variances\. Since𝔼⁡\[X\]=G\\mathbb\{E\}\[X\]=G, this equals its mean\-squared error \(MSE\):tr⁡Cov⁡\(X\)=𝔼⁡\[∥X−G∥F2\]\\operatorname\{tr\}\\operatorname\{Cov\}\(X\)=\\mathbb\{E\}\[\\lVert X\-G\\rVert\_\{F\}^\{2\}\]\. The following theorem computes the complete covariance ofXX\. Taking its trace gives the variance of one direction and, after division byNN, the variance of a population average\. Comparing this expression with the dense\-Gaussian result will be informative in deducing whether low rank requires a larger population\.

###### Theorem 7\.1\(Exact finite\-rank covariance\)\.

For all indicesi,k∈\[m\]i,k\\in\[m\]andj,ℓ∈\[n\]j,\\ell\\in\[n\],

Cov⁡\(Xi​j,Xk​ℓ\)=∥G∥F2​δi​k​δj​ℓ\+Gi​j​Gk​ℓ\+2r​\[δi​k​\(G⊤​G\)j​ℓ\+δj​ℓ​\(G​G⊤\)i​k\+Gi​ℓ​Gk​j\]\.\\operatorname\{Cov\}\(X\_\{ij\},X\_\{k\\ell\}\)=\\lVert G\\rVert\_\{F\}^\{2\}\\delta\_\{ik\}\\delta\_\{j\\ell\}\+G\_\{ij\}G\_\{k\\ell\}\+\\frac\{2\}\{r\}\\left\[\\delta\_\{ik\}\(G^\{\\top\}G\)\_\{j\\ell\}\+\\delta\_\{j\\ell\}\(GG^\{\\top\}\)\_\{ik\}\+G\_\{i\\ell\}G\_\{kj\}\\right\]\.\(93\)Define the variance factor

κr:=m​n\+1\+2​\(m\+n\+1\)r\.\\kappa\_\{r\}:=mn\+1\+\\frac\{2\(m\+n\+1\)\}\{r\}\.\(94\)Then

𝔼⁡\[∥X−G∥F2\]=κr​∥G∥F2\.\\mathbb\{E\}\[\\lVert X\-G\\rVert\_\{F\}^\{2\}\]=\\kappa\_\{r\}\\lVert G\\rVert\_\{F\}^\{2\}\.\(95\)For independent copiesX1,…,XNX\_\{1\},\\ldots,X\_\{N\}and their averageG^N=1N​∑s=1NXs\\widehat\{G\}\_\{N\}=\\frac\{1\}\{N\}\\sum\_\{s=1\}^\{N\}X\_\{s\},

𝔼⁡\[∥G^N−G∥F2\]=1N​𝔼​\[∥X−G∥F2\]=κrN​∥G∥F2\.\\mathbb\{E\}\[\\lVert\\widehat\{G\}\_\{N\}\-G\\rVert\_\{F\}^\{2\}\]=\\frac\{1\}\{N\}\\mathbb\{E\}\[\\lVert X\-G\\rVert\_\{F\}^\{2\}\]=\\frac\{\\kappa\_\{r\}\}\{N\}\\lVert G\\rVert\_\{F\}^\{2\}\.\(96\)

###### Proof\.

SubstituteEr=r−1/2∑q=1raqbq⊤E\_\{r\}=r^\{\-1/2\}\\sum\_\{q=1\}^\{r\}a\_\{q\}b\_\{q\}^\{\\top\}into𝔼⁡\[Xi​j​Xk​ℓ\]\\mathbb\{E\}\[X\_\{ij\}X\_\{k\\ell\}\]and apply the Gaussian fourth\-moment identity to the independent factorsaqa\_\{q\}andbqb\_\{q\}\. Summing the resulting fourth\-moment terms against the entries ofGGgives

𝔼⁡\[Xi​j​Xk​ℓ\]=∥G∥F2​δi​k​δj​ℓ\+2​Gi​j​Gk​ℓ\+2r​\[δi​k​\(G⊤​G\)j​ℓ\+δj​ℓ​\(G​G⊤\)i​k\+Gi​ℓ​Gk​j\]\.\\mathbb\{E\}\[X\_\{ij\}X\_\{k\\ell\}\]=\\lVert G\\rVert\_\{F\}^\{2\}\\delta\_\{ik\}\\delta\_\{j\\ell\}\+2G\_\{ij\}G\_\{k\\ell\}\+\\frac\{2\}\{r\}\\left\[\\delta\_\{ik\}\(G^\{\\top\}G\)\_\{j\\ell\}\+\\delta\_\{j\\ell\}\(GG^\{\\top\}\)\_\{ik\}\+G\_\{i\\ell\}G\_\{kj\}\\right\]\.\(97\)Since𝔼⁡\[X\]=G\\mathbb\{E\}\[X\]=G, subtractingGi​j​Gk​ℓG\_\{ij\}G\_\{k\\ell\}proves[Equation93](https://arxiv.org/html/2609.10980#S7.E93)\. Takingi=ki=k,j=ℓj=\\ell, and summing yields

tr⁡Cov⁡\(X\)=m​n​∥G∥F2\+∥G∥F2\+2r​\[m​∥G∥F2\+n​∥G∥F2\+∥G∥F2\],\\operatorname\{tr\}\\operatorname\{Cov\}\(X\)=mn\\lVert G\\rVert\_\{F\}^\{2\}\+\\lVert G\\rVert\_\{F\}^\{2\}\+\\frac\{2\}\{r\}\\left\[m\\lVert G\\rVert\_\{F\}^\{2\}\+n\\lVert G\\rVert\_\{F\}^\{2\}\+\\lVert G\\rVert\_\{F\}^\{2\}\\right\],\(98\)which is[Equation95](https://arxiv.org/html/2609.10980#S7.E95)\. Independence gives the factor1/N1/Nin[Equation96](https://arxiv.org/html/2609.10980#S7.E96)\. ∎

Dense Gaussian directions and Gaussian\-product directions share the variance factorm​n\+1mn\+1\. Finite rank adds only the product\-law correction2​\(m\+n\+1\)/r2\(m\+n\+1\)/r\. The relative variance increase of finite\-rank EGGROLL over dense Gaussian ES in the affine model is therefore

ρm,n,r=2​\(m\+n\+1\)r⁡\(m​n\+1\)\.\\rho\_\{m,n,r\}=\\frac\{2\(m\+n\+1\)\}\{r\(mn\+1\)\}\.\(99\)For a4096×40964096\\times 4096weight matrix, as found, for instance, in Qwen3\-8B,ρm,n,1≈9\.77×10−4\\rho\_\{m,n,1\}\\approx 9\.77\\times 10^\{\-4\}\. Thus a rank\-one law can be singular with respect to a dense Gaussian law and still have nearly the same local\-linear variance\. This is because these two statements concern different properties of the distribution\. Singularity concerns where the perturbations can lie, whereas estimator variance depends on a particular weighted sum of their fourth moments\. The rank\-dependent part grows asm\+nm\+n, while the common dense\-Gaussian contribution grows asm​nmn\.

For a network, independence confines the additional finite\-rank variance to within\-block terms\. Appendix[A](https://arxiv.org/html/2609.10980#A1)gives the exact formula and shows how each block is weighted by its local gradient energy\.

This completes the finite\-population comparison in the affine local model\. Rank\-one EGGROLL and dense Gaussian ES have nearly the same sampling error for wide matrices, even though their mean fields can differ at finite radius\. The next section returns to that difference and studies how the finite\-rank mean field approaches dense Gaussian ES asrrincreases\.

## 8Approach to dense Gaussian ES at fixed radius

The analysis up to now has treated the rankrras fixed and used a small perturbation radiusσ\\sigmato control the error of the local approximation\. EGGROLL offers two adjustable quantities, however:σ\\sigmacontrols the scale of the perturbations, whilerroffers a handle on how closely their distribution resembles a dense Gaussian\. We therefore now explore the alternative approach of holdingσ\\sigmafixed and asking how the population field changes asrrincreases\. The analysis has already hinted that asrrincreases, the Gaussian\-product perturbation approaches a dense Gaussian, and the EGGROLL mean field approaches the gradient followed by dense Gaussian ES at the same radius\. The purpose of this section is to identify the leading difference between these two fields\. Appendix[C](https://arxiv.org/html/2609.10980#A3)combines the small\-radius and fixed\-radius bounds with the finite\-population variance from[Section7](https://arxiv.org/html/2609.10980#S7)\.

For a fixed frequencyTT, expanding the exact characteristic function asr→∞r\\to\\inftyusing the matrix\-logarithm series\[[22](https://arxiv.org/html/2609.10980#bib.bib31)\]gives

log⁡Φr​\(T\)=−12​∥T∥F2\+14​r​tr⁡\(\(T​T⊤\)2\)−16​r2​tr⁡\(\(T​T⊤\)3\)\+O⁡\(r−3\)\.\\log\\Phi\_\{r\}\(T\)=\-\\frac\{1\}\{2\}\\lVert T\\rVert\_\{F\}^\{2\}\+\\frac\{1\}\{4r\}\\operatorname\{tr\}\\left\(\(TT^\{\\top\}\)^\{2\}\\right\)\-\\frac\{1\}\{6r^\{2\}\}\\operatorname\{tr\}\\left\(\(TT^\{\\top\}\)^\{3\}\\right\)\+O\(r^\{\-3\}\)\.\(100\)The leading term is the logarithm of the characteristic function of a dense Gaussian and depends only on∥T∥F\\lVert T\\rVert\_\{F\}\. The first finite\-rank correction also depends on how this norm is distributed across singular directions throughtr⁡\(\(T​T⊤\)2\)\\operatorname\{tr\}\(\(TT^\{\\top\}\)^\{2\}\)\. Thus the same anisotropy that produced the resolvent in[Section4](https://arxiv.org/html/2609.10980#S4)appears as the leading1/r1/rerror at fixed radius\.

As in[Section4](https://arxiv.org/html/2609.10980#S4), we now want to obtain a statement about the complete population field from the calculation at a single frequency\. The distributional Fourier argument in[Section4](https://arxiv.org/html/2609.10980#S4)gives an exact operator identity, but does not by itself bound the remainder of a rank expansion uniformly inWW\. We therefore expand the score\-weighted perturbation law that acts onff\. Bounding its remainder in a polynomially weighted integral norm will allow us to treat objectives of polynomial growth without assuming a Fourier density forff\. For a matrix displacementZZ, define the matrix\-valued measure

νr,σ​\(d​Z\):=Zσ2​ℙ​\(σ​Er∈d​Z\)\.\\nu\_\{r,\\sigma\}\(dZ\):=\\frac\{Z\}\{\\sigma^\{2\}\}\\,\\mathbb\{P\}\(\\sigma E\_\{r\}\\in dZ\)\.\(101\)The population field can then be written as

gr,σf​\(W\)=∫f⁡\(W\+Z\)​νr,σ​\(𝑑Z\)\.g\_\{r,\\sigma\}^\{f\}\(W\)=\\int f\(W\+Z\)\\,\\nu\_\{r,\\sigma\}\(dZ\)\.\(102\)Thus an expansion ofνr,σ\\nu\_\{r,\\sigma\}immediately becomes an expansion of the update field\. For these kernels, writeν^​\(T\)=∫ei​⟨T,Z⟩F​ν​\(𝑑Z\)\\widehat\{\\nu\}\(T\)=\\int e^\{i\\langle T,\\,Z\\rangle\_\{F\}\}\\nu\(dZ\), with a positive sign because the field evaluatesf⁡\(W\+Z\)f\(W\+Z\)\. The transform of the update measure is then the multiplier derived in[Section4](https://arxiv.org/html/2609.10980#S4):

ν^r,σ​\(T\)=i​Φr​\(σ​T\)​\(Im\+σ2r​T​T⊤\)−1​T\.\\widehat\{\\nu\}\_\{r,\\sigma\}\(T\)=i\\Phi\_\{r\}\(\\sigma T\)\\left\(I\_\{m\}\+\\frac\{\\sigma^\{2\}\}\{r\}TT^\{\\top\}\\right\)^\{\-1\}T\.\(103\)Recall from[Section4](https://arxiv.org/html/2609.10980#S4)thatΦr\(σT\)=det\(Im\+σ2TT⊤/r\)−r/2\\Phi\_\{r\}\(\\sigma T\)=\\det\(I\_\{m\}\+\\sigma^\{2\}TT^\{\\top\}/r\)^\{\-r/2\}\. We expand both rank\-dependent factors in powers of1/r1/r:

Φr\(σT\)=e−σ2∥T∥F2/2\[1\+σ44​rtr\(\(TT⊤\)2\)\+O\(r−2\)\],\(Im\+σ2rTT⊤\)−1=Im−σ2rTT⊤\+O\(r−2\)\.\\Phi\_\{r\}\(\\sigma T\)=e^\{\-\\sigma^\{2\}\\lVert T\\rVert\_\{F\}^\{2\}/2\}\\left\[1\+\\frac\{\\sigma^\{4\}\}\{4r\}\\operatorname\{tr\}\\left\(\(TT^\{\\top\}\)^\{2\}\\right\)\+O\(r^\{\-2\}\)\\right\],\\quad\\left\(I\_\{m\}\+\\frac\{\\sigma^\{2\}\}\{r\}TT^\{\\top\}\\right\)^\{\-1\}=I\_\{m\}\-\\frac\{\\sigma^\{2\}\}\{r\}TT^\{\\top\}\+O\(r^\{\-2\}\)\.\(104\)Consequently,

ν^r,σ\(T\)=ie−σ2∥T∥F2/2T\+1rβ^σ\(T\)\+O\(r−2\),\\widehat\{\\nu\}\_\{r,\\sigma\}\(T\)=ie^\{\-\\sigma^\{2\}\\lVert T\\rVert\_\{F\}^\{2\}/2\}T\+\\frac\{1\}\{r\}\\widehat\{\\beta\}\_\{\\sigma\}\(T\)\+O\(r^\{\-2\}\),\(105\)where the matrix\-valued correction kernelβσ\\beta\_\{\\sigma\}is defined by

β^σ\(T\)=ie−σ2∥T∥F2/2\[σ44tr\(\(TT⊤\)2\)T−σ2TT⊤T\],\\widehat\{\\beta\}\_\{\\sigma\}\(T\)=ie^\{\-\\sigma^\{2\}\\lVert T\\rVert\_\{F\}^\{2\}/2\}\\left\[\\frac\{\\sigma^\{4\}\}\{4\}\\operatorname\{tr\}\\left\(\(TT^\{\\top\}\)^\{2\}\\right\)T\-\\sigma^\{2\}TT^\{\\top\}T\\right\],\(106\)and its operation on the objective is denoted by

ℬσ​f​\(W\):=∫f⁡\(W\+Z\)​βσ​\(𝑑Z\)\.\\mathcal\{B\}\_\{\\sigma\}f\(W\):=\\int f\(W\+Z\)\\,\\beta\_\{\\sigma\}\(dZ\)\.\(107\)The two terms in[Equation106](https://arxiv.org/html/2609.10980#S8.E106)have distinct origins\. The term containingtr⁡\(\(T​T⊤\)2\)​T\\operatorname\{tr\}\(\(TT^\{\\top\}\)^\{2\}\)Trecords the difference between finite\-rank and dense\-Gaussian smoothing, while the term containingT​T⊤​TTT^\{\\top\}Trecords the score mismatch identified in[Section4](https://arxiv.org/html/2609.10980#S4)\.

To turn the kernel expansion into a uniform bound on the update field, we must account for the permitted growth of the objective\. Iff⁡\(W\)f\(W\)grows polynomially, then the field and its approximation error need not be bounded uniformly over the entire parameter space\. We therefore measure the error after normalizing by the same polynomial growth\. For a matrix vector fielduuand a nonnegative integerpp, define

∥u∥∞,p:=supW∥u⁡\(W\)∥F\(1\+∥W∥F\)p\.\\lVert u\\rVert\_\{\\infty,p\}:=\\sup\_\{W\}\\frac\{\\lVert u\(W\)\\rVert\_\{F\}\}\{\(1\+\\lVert W\\rVert\_\{F\}\)^\{p\}\}\.\(108\)Whenp=0p=0, this is the ordinary uniform norm\.

The following theorem makes the rank dependence precise\. Dense Gaussian ES is the zeroth\-order term,ℬσ​f/r\\mathcal\{B\}\_\{\\sigma\}f/ris the leading finite\-rank correction, and the rest of the terms are of order1/r21/r^\{2\}\.

###### Theorem 8\.1\(Fixed\-radius rank expansion\)\.

Letf:ℝm×n→ℝf:\\mathbb\{R\}^\{m\\times n\}\\to\\mathbb\{R\}be Borel measurable and suppose that, for some nonnegative integerppandCf<∞C\_\{f\}<\\infty,

\|f⁡\(W\)\|≤Cf​\(1\+∥W∥Fp\)\.\|f\(W\)\|\\leq C\_\{f\}\(1\+\\lVert W\\rVert\_\{F\}^\{p\}\)\.\(109\)For every fixedσ\>0\\sigma\>0, there existRσ,p<∞R\_\{\\sigma,p\}<\\inftyandr0r\_\{0\}such that, for everyr≥r0r\\geq r\_\{0\},

∥gr,σf−g∞,σf−1r​ℬσ​f∥∞,p≤Rσ,pr2\.\\lVert g\_\{r,\\sigma\}^\{f\}\-g\_\{\\infty,\\sigma\}^\{f\}\-\\frac\{1\}\{r\}\\mathcal\{B\}\_\{\\sigma\}f\\rVert\_\{\\infty,p\}\\leq\\frac\{R\_\{\\sigma,p\}\}\{r^\{2\}\}\.\(110\)Herer0r\_\{0\}depends onm,n,pm,n,p, and the remainder constant may also depend onσ\\sigmaandCfC\_\{f\}\. Consequently,∥gr,σf−g∞,σf∥∞,p=O⁡\(r−1\)\\lVert g\_\{r,\\sigma\}^\{f\}\-g\_\{\\infty,\\sigma\}^\{f\}\\rVert\_\{\\infty,p\}=O\(r^\{\-1\}\)\. Ifℬσ​f\\mathcal\{B\}\_\{\\sigma\}fis nonzero, this difference is noto⁡\(r−1\)o\(r^\{\-1\}\)in the same norm\.

###### Proof\.

We expand the update kernel in the reciprocal rankt=1/rt=1/rand bound the remainder before integrating againstff\. First setσ=1\\sigma=1and writeD=m​nD=mn\. Fort\>0t\>0, define

ϕt\(T\)=det\(Im\+tTT⊤\)−1/\(2t\),At\(T\)=\(Im\+tTT⊤\)−1,mt\(T\)=iϕt\(T\)At\(T\)T\.\\phi\_\{t\}\(T\)=\\det\(I\_\{m\}\+tTT^\{\\top\}\)^\{\-1/\(2t\)\},\\qquad A\_\{t\}\(T\)=\(I\_\{m\}\+tTT^\{\\top\}\)^\{\-1\},\\qquad m\_\{t\}\(T\)=i\\phi\_\{t\}\(T\)A\_\{t\}\(T\)T\.\(111\)Att=0t=0, setϕ0\(T\)=e−∥T∥F2/2\\phi\_\{0\}\(T\)=e^\{\-\\lVert T\\rVert\_\{F\}^\{2\}/2\}andA0​\(T\)=ImA\_\{0\}\(T\)=I\_\{m\}\. Thenm1/r=ν^r,1m\_\{1/r\}=\\widehat\{\\nu\}\_\{r,1\},m0=ν^∞,1m\_\{0\}=\\widehat\{\\nu\}\_\{\\infty,1\}, and∂tm0=β^1\\partial\_\{t\}m\_\{0\}=\\widehat\{\\beta\}\_\{1\}\. These functions are smooth att=0t=0because

ℓt\(T\):=logϕt\(T\)=−12∫01tr\[TT⊤\(Im\+utTT⊤\)−1\]du\.\\ell\_\{t\}\(T\):=\\log\\phi\_\{t\}\(T\)=\-\\frac\{1\}\{2\}\\int\_\{0\}^\{1\}\\operatorname\{tr\}\\left\[TT^\{\\top\}\(I\_\{m\}\+utTT^\{\\top\}\)^\{\-1\}\\right\]du\.\(112\)In particular,∂tℓ0=tr⁡\(\(T​T⊤\)2\)/4\\partial\_\{t\}\\ell\_\{0\}=\\operatorname\{tr\}\(\(TT^\{\\top\}\)^\{2\}\)/4and∂tA0=−T​T⊤\\partial\_\{t\}A\_\{0\}=\-TT^\{\\top\}, recovering the two terms in[Equation106](https://arxiv.org/html/2609.10980#S8.E106)\.

To control the remainder, letα\\alphabe a multi\-index for theDDentries ofTT, so that∂Tα\\partial\_\{T\}^\{\\alpha\}has total derivative order\|α\|\|\\alpha\|\. For every fixed orderkk, there are constantsCkC\_\{k\}and an integerKkK\_\{k\}, independent of0≤t≤10\\leq t\\leq 1, such that

∥∂Tα∂t2mt​\(T\)∥F≤Ck​\(1\+∥T∥F\)Kk​ϕt​\(T\),\|α\|≤k\.\\lVert\\partial\_\{T\}^\{\\alpha\}\\partial\_\{t\}^\{2\}m\_\{t\}\(T\)\\rVert\_\{F\}\\leq C\_\{k\}\(1\+\\lVert T\\rVert\_\{F\}\)^\{K\_\{k\}\}\\phi\_\{t\}\(T\),\\qquad\|\\alpha\|\\leq k\.\(113\)To verify this bound, differentiate[Equation112](https://arxiv.org/html/2609.10980#S8.E112)under the integral\. Every derivative of an inverse matrix is obtained from∂B−1=−B−1​\(∂B\)​B−1\\partial B^\{\-1\}=\-B^\{\-1\}\(\\partial B\)B^\{\-1\}\. All inverse factors here have operator norm at most one, while derivatives ofT​T⊤TT^\{\\top\}are polynomial inTT\. Thus every fixed mixed derivative ofℓt\\ell\_\{t\}andAtA\_\{t\}has a polynomial bound, uniformly for0≤t≤10\\leq t\\leq 1\. Repeated differentiation ofeℓte^\{\\ell\_\{t\}\}leaves the factorϕt\\phi\_\{t\}multiplied by products of these bounded derivatives\. Applying the product rule tomt=i​ϕt​At​Tm\_\{t\}=i\\phi\_\{t\}A\_\{t\}Tproves[Equation113](https://arxiv.org/html/2609.10980#S8.E113)\.

The determinant supplies a common decay bound\. For0<t≤t0≤10<t\\leq t\_\{0\}\\leq 1,

ϕt\(T\)≤\(1\+t∥T∥F2\)−1/\(2t\)≤\(1\+t0∥T∥F2\)−1/\(2t0\)\.\\phi\_\{t\}\(T\)\\leq\(1\+t\\lVert T\\rVert\_\{F\}^\{2\}\)^\{\-1/\(2t\)\}\\leq\(1\+t\_\{0\}\\lVert T\\rVert\_\{F\}^\{2\}\)^\{\-1/\(2t\_\{0\}\)\}\.\(114\)The first inequality follows by expanding the product of1\+t​sj21\+ts\_\{j\}^\{2\}, wheresjs\_\{j\}are the singular values ofTT\. The second follows becauselog⁡\(1\+t​a\)/t\\log\(1\+ta\)/tdecreases withttfora≥0a\\geq 0\. The bound also holds att=0t=0by continuity\. Chooset0t\_\{0\}so that1/t0\>Kk\+D/21/t\_\{0\}\>K\_\{k\}\+D/2\. The right\-hand side of[Equation113](https://arxiv.org/html/2609.10980#S8.E113), with[Equation114](https://arxiv.org/html/2609.10980#S8.E114), is then square\-integrable inTT, uniformly over0≤t≤t00\\leq t\\leq t\_\{0\}\.

Taylor’s formula with an integral remainder therefore gives

∂Tα\(mt−m0−t​∂tm0\)=t2​∫01\(1−v\)​∂Tα∂t2mv​t​𝑑v,∥∂Tα\(mt−m0−t​∂tm0\)∥L2≤Ck′​t2\.\\partial\_\{T\}^\{\\alpha\}\(m\_\{t\}\-m\_\{0\}\-t\\partial\_\{t\}m\_\{0\}\)=t^\{2\}\\int\_\{0\}^\{1\}\(1\-v\)\\partial\_\{T\}^\{\\alpha\}\\partial\_\{t\}^\{2\}m\_\{vt\}\\,dv,\\qquad\\lVert\\partial\_\{T\}^\{\\alpha\}\(m\_\{t\}\-m\_\{0\}\-t\\partial\_\{t\}m\_\{0\}\)\\rVert\_\{L^\{2\}\}\\leq C\_\{k\}^\{\\prime\}t^\{2\}\.\(115\)For general fixedσ\>0\\sigma\>0, the update multiplier isσ−1​mt​\(σ​T\)\\sigma^\{\-1\}m\_\{t\}\(\\sigma T\)\. RescalingTTand takingt=1/rt=1/rtherefore proves, forr≥r0:=⌈1/t0⌉r\\geq r\_\{0\}:=\\lceil 1/t\_\{0\}\\rceil,

‖∂Tα\(ν^r,σ−ν^∞,σ−1r​β^σ\)‖L2≤Cσ,kr2,\|α\|≤k\.\\left\\\|\\partial\_\{T\}^\{\\alpha\}\\left\(\\widehat\{\\nu\}\_\{r,\\sigma\}\-\\widehat\{\\nu\}\_\{\\infty,\\sigma\}\-\\frac\{1\}\{r\}\\widehat\{\\beta\}\_\{\\sigma\}\\right\)\\right\\\|\_\{L^\{2\}\}\\leq\\frac\{C\_\{\\sigma,k\}\}\{r^\{2\}\},\\qquad\|\\alpha\|\\leq k\.\(116\)
Finally choose an integerq\>D/2q\>D/2and takek=p\+qk=p\+q\. Letρr,σ\\rho\_\{r,\\sigma\}be the inverse transform of the remainder, using the positive\-sign kernel convention of[Equation103](https://arxiv.org/html/2609.10980#S8.E103)\. Cauchy–Schwarz and Plancherel’s identity\[[14](https://arxiv.org/html/2609.10980#bib.bib27)\]imply

∫\(1\+∥Z∥F\)p​∥ρr,σ​\(Z\)∥F​𝑑Z≤CD,p,q​∑\|α\|≤p\+q∥∂Tαρ^r,σ∥L2≤Cσ,pr2\.\\int\(1\+\\lVert Z\\rVert\_\{F\}\)^\{p\}\\lVert\\rho\_\{r,\\sigma\}\(Z\)\\rVert\_\{F\}\\,dZ\\leq C\_\{D,p,q\}\\sum\_\{\|\\alpha\|\\leq p\+q\}\\lVert\\partial\_\{T\}^\{\\alpha\}\\widehat\{\\rho\}\_\{r,\\sigma\}\\rVert\_\{L^\{2\}\}\\leq\\frac\{C\_\{\\sigma,p\}\}\{r^\{2\}\}\.\(117\)Indeed, apply Cauchy–Schwarz with the square\-integrable weight\(1\+∥Z∥F\)−q\(1\+\\lVert Z\\rVert\_\{F\}\)^\{\-q\}, then bound the weightedL2L^\{2\}norm by the finitely many monomialsZα​ρr,σZ^\{\\alpha\}\\rho\_\{r,\\sigma\}of degree at mostp\+qp\+q\. Plancherel converts those monomials into Fourier derivatives\. The correction kernelβσ\\beta\_\{\\sigma\}has a Schwartz density because its transform is a polynomial times a Gaussian\. Uniqueness of Fourier transforms of finite measures identifiesρr,σ​\(Z\)​d​Z\\rho\_\{r,\\sigma\}\(Z\)\\,dZwithνr,σ−ν∞,σ−βσ/r\\nu\_\{r,\\sigma\}\-\\nu\_\{\\infty,\\sigma\}\-\\beta\_\{\\sigma\}/r\.

Since\|f⁡\(W\+Z\)\|≤2​Cf​\(1\+∥W∥F\)p​\(1\+∥Z∥F\)p\|f\(W\+Z\)\|\\leq 2C\_\{f\}\(1\+\\lVert W\\rVert\_\{F\}\)^\{p\}\(1\+\\lVert Z\\rVert\_\{F\}\)^\{p\}, integration against this remainder gives

∥gr,σf​\(W\)−g∞,σf​\(W\)−1r​ℬσ​f​\(W\)∥F≤2​Cf​Cσ,pr2​\(1\+∥W∥F\)p,\\lVert g\_\{r,\\sigma\}^\{f\}\(W\)\-g\_\{\\infty,\\sigma\}^\{f\}\(W\)\-\\frac\{1\}\{r\}\\mathcal\{B\}\_\{\\sigma\}f\(W\)\\rVert\_\{F\}\\leq\\frac\{2C\_\{f\}C\_\{\\sigma,p\}\}\{r^\{2\}\}\(1\+\\lVert W\\rVert\_\{F\}\)^\{p\},\(118\)which proves[Equation110](https://arxiv.org/html/2609.10980#S8.E110)\. Multiplying the expansion byrrshows thatr⁡\(gr,σf−g∞,σf\)r\(g\_\{r,\\sigma\}^\{f\}\-g\_\{\\infty,\\sigma\}^\{f\}\)converges toℬσ​f\\mathcal\{B\}\_\{\\sigma\}fin∥⋅∥∞,p\\lVert\\cdot\\rVert\_\{\\infty,p\}\. If this limit is nonzero, the difference cannot decay faster than1/r1/r\. ∎

This result explicitly identifies not only the rate of convergence, but also the asymptotic\-limit fieldg∞,σfg\_\{\\infty,\\sigma\}^\{f\}of the dense Gaussian ES, and the direction and magnitude of the leading errorℬσ​f/r\\mathcal\{B\}\_\{\\sigma\}f/rintroduced by finite rank\. The two terms in[Equation106](https://arxiv.org/html/2609.10980#S8.E106)show that this error combines a change in the smoothing distribution with the score mismatch\. The remaining discrepancy is smaller by another factor of1/r1/r\.

Sinceg∞,σf=∇F∞,σg\_\{\\infty,\\sigma\}^\{f\}=\\nabla F\_\{\\infty,\\sigma\}, both fields in this comparison use the same fixed perturbation radius\. Their difference therefore isolates the effect of finite rank rather than the smoothing shared with dense Gaussian ES\. In particular,

∥gr,σf−∇F∞,σ∥∞,p≤Kσ,pr\+Rσ,pr2,Kσ,p:=∥ℬσ​f∥∞,p\.\\lVert g\_\{r,\\sigma\}^\{f\}\-\\nabla F\_\{\\infty,\\sigma\}\\rVert\_\{\\infty,p\}\\leq\\frac\{K\_\{\\sigma,p\}\}\{r\}\+\\frac\{R\_\{\\sigma,p\}\}\{r^\{2\}\},\\qquad K\_\{\\sigma,p\}:=\\lVert\\mathcal\{B\}\_\{\\sigma\}f\\rVert\_\{\\infty,p\}\.\(119\)For boundedff, this is an ordinary uniform bound\. For polynomially growingff, the denominator in∥⋅∥∞,p\\lVert\\cdot\\rVert\_\{\\infty,p\}allows the field error to grow at the same polynomial rate withWW\. Whenℬσ​f\\mathcal\{B\}\_\{\\sigma\}fis nonzero, the leading term does not vanish, so the1/r1/rrate cannot be improved in this norm\. This strengthens theO⁡\(1/r\)O\(1/r\)comparison of[Sarkar et al\. \[36\]](https://arxiv.org/html/2609.10980#bib.bib2): their Edgeworth argument establishes the rate, whereas the kernel expansion identifies its coefficient and leaves anO⁡\(r−2\)O\(r^\{\-2\}\)residual\. Appendix[C](https://arxiv.org/html/2609.10980#A3)combines the field\-error bounds with the finite\-population variance from[Section7](https://arxiv.org/html/2609.10980#S7)to derive finite\-iteration guarantees for approaching stationary points of the original objective, and, at fixed perturbation radius, of the dense\-Gaussian\-smoothed objective\.

## 9EGGROLL leads to LOO\-ROLL

The preceding analysis identifies three distinct costs\. Finite rank changes the population field through bias, finite sampling leaves the large directional variance already present in Gaussian zeroth\-order optimization, and the antithetic implementation evaluates both signs of every direction\. The first two observations suggest rank extrapolation and a reference\-point control variate\. We derive both in Appendix[B](https://arxiv.org/html/2609.10980#A2), but small empirical studies find that their additional evaluations and state do not improve on EGGROLL in the tested regime\. The third cost admits a simpler correction that does produce a sizeable practical gain\.

The method description and principal update equation of[Sarkar et al\. \[36\]](https://arxiv.org/html/2609.10980#bib.bib2)use the one\-sided quantityE​f​\(W\+σ​E\)Ef\(W\+\\sigma E\)\. Their reported LLM experiments and implementation instead use antithetic pairs, evaluating bothf⁡\(W\+σ​E\)f\(W\+\\sigma E\)andf⁡\(W−σ​E\)f\(W\-\\sigma E\)for every independent direction\. Practical EGGROLL therefore spends two forward evaluations per direction\. LOO\-ROLL reduces this cost to one by using the rest of the sampled population to construct a leave\-one\-out baseline\.

### 9\.1LOO\-ROLL

Score\-function estimators may subtract any baseline independent of the sampled variable without changing their expectation, a device used throughout policy\-gradient optimization\[[47](https://arxiv.org/html/2609.10980#bib.bib45)\]and evolution strategies\[[46](https://arxiv.org/html/2609.10980#bib.bib22)\]\. When several samples share the same conditioning information, a leave\-one\-out average supplies such a baseline for every sample\. REINFORCE leave\-one\-out applies this principle to groups of responses in language\-model alignment\[[2](https://arxiv.org/html/2609.10980#bib.bib46)\]\. EGGROLL provides an analogous group across parameter perturbations rather than responses\.

At iterationtt, we sampleN≥2N\\geq 2independent directionsEt,1,…,Et,NE\_\{t,1\},\\ldots,E\_\{t,N\}and evaluate each direction only once\. WriteFt,s=f⁡\(Wt\+σ​Et,s\)F\_\{t,s\}=f\(W\_\{t\}\+\\sigma E\_\{t,s\}\)and letF¯t,−s=\(N−1\)−1​∑j≠sFt,j\\overline\{F\}\_\{t,\-s\}=\(N\-1\)^\{\-1\}\\sum\_\{j\\neq s\}F\_\{t,j\}\. We define

g^tLOO:=1N​σ​∑s=1NEt,s​\(Ft,s−F¯t,−s\)\.\\widehat\{g\}\_\{t\}^\{\\mathrm\{LOO\}\}:=\\frac\{1\}\{N\\sigma\}\\sum\_\{s=1\}^\{N\}E\_\{t,s\}\(F\_\{t,s\}\-\\overline\{F\}\_\{t,\-s\}\)\.\(120\)The same estimator can alternatively be computed from the population meanF¯t=N−1​∑sFt,s\\overline\{F\}\_\{t\}=N^\{\-1\}\\sum\_\{s\}F\_\{t,s\}, sinceFt,s−F¯t,−s=N⁡\(Ft,s−F¯t\)/\(N−1\)F\_\{t,s\}\-\\overline\{F\}\_\{t,\-s\}=N\(F\_\{t,s\}\-\\overline\{F\}\_\{t\}\)/\(N\-1\)\. Thus, the leave\-one\-out baselines require no additional fitness evaluations and only a population reduction already natural in distributed ES\. We call the resulting method*LOO\-ROLL*\. The following proposition shows that the leave\-one\-out baseline preserves the finite\-rank population field exactly before score standardization\.

###### Proposition 9\.1\(Unbiasedness of LOO\-ROLL\)\.

Letℱt\\mathcal\{F\}\_\{t\}denote the sigma\-field generated by the optimization history before iterationtt\. Conditional onℱt\\mathcal\{F\}\_\{t\}, suppose thatN≥2N\\geq 2andEt,1,…,Et,NE\_\{t,1\},\\ldots,E\_\{t,N\}are independent, centered, and have distributionLaw⁡\(Er\)\\operatorname\{Law\}\(E\_\{r\}\)\. Assume thatFt,sF\_\{t,s\}and∥Et,s∥F​Ft,s\\lVert E\_\{t,s\}\\rVert\_\{F\}F\_\{t,s\}are conditionally integrable\. Then,

𝔼⁡\[g^tLOO∣ℱt\]=gr,σf​\(Wt\)\.\\mathbb\{E\}\\left\[\\widehat\{g\}\_\{t\}^\{\\mathrm\{LOO\}\}\\mid\\mathcal\{F\}\_\{t\}\\right\]=g\_\{r,\\sigma\}^\{f\}\(W\_\{t\}\)\.\(121\)

###### Proof\.

The termEt,s​Ft,s/σE\_\{t,s\}F\_\{t,s\}/\\sigmahas conditional expectationgr,σf​\(Wt\)g\_\{r,\\sigma\}^\{f\}\(W\_\{t\}\)\. The baselineF¯t,−s\\overline\{F\}\_\{t,\-s\}depends only on the other directions and is therefore conditionally independent ofEt,sE\_\{t,s\}\. Centering gives𝔼⁡\[Et,s​F¯t,−s∣ℱt\]=0\\mathbb\{E\}\[E\_\{t,s\}\\overline\{F\}\_\{t,\-s\}\\mid\\mathcal\{F\}\_\{t\}\]=0\. Averaging overssproves the claim\. ∎

Before score standardization, LOO\-ROLL therefore targets exactly the same population field analyzed in[Section4](https://arxiv.org/html/2609.10980#S4), while replacing two forward evaluations per independent direction by one\. The saving can fund twice as many directions at a fixed evaluation budget or additional optimization steps at a fixed wall\-time budget\. To analyze the variance of these estimators, we approximate the fitness locally by an affine function, as in[Section7](https://arxiv.org/html/2609.10980#S7), with gradientG=∇f​\(W\)G=\\nabla f\(W\)\. At fixedWW,rr, andσ\\sigma, writeg^LOO​\(W,N\)\\widehat\{g\}^\{\\mathrm\{LOO\}\}\(W;N\)for the estimator in[Equation120](https://arxiv.org/html/2609.10980#S9.E120)formed fromNNindependent directions, andg^anti​\(W,N\):=g^r,σ,N​\(W\)\\widehat\{g\}^\{\\mathrm\{anti\}\}\(W;N\):=\\widehat\{g\}\_\{r,\\sigma,N\}\(W\)for the antithetic estimator in[Equation7](https://arxiv.org/html/2609.10980#S3.E7)\.

###### Proposition 9\.2\(Variance of LOO\-ROLL in the affine model\)\.

Supposef⁡\(W\+σ​E\)=f⁡\(W\)\+σ​⟨G,E⟩Ff\(W\+\\sigma E\)=f\(W\)\+\\sigma\\langle G,\\,E\\rangle\_\{F\}and writed=m​nd=mn, withκr\\kappa\_\{r\}as defined in[Equation94](https://arxiv.org/html/2609.10980#S7.E94)\. ForN≥2N\\geq 2independent directions, the LOO\-ROLL estimator satisfies

𝔼⁡\[∥g^LOO​\(W,N\)−G∥F2\]=\[κrN\+d\+1N⁡\(N−1\)\]​∥G∥F2\.\\mathbb\{E\}\\left\[\\lVert\\widehat\{g\}^\{\\mathrm\{LOO\}\}\(W;N\)\-G\\rVert\_\{F\}^\{2\}\\right\]=\\left\[\\frac\{\\kappa\_\{r\}\}\{N\}\+\\frac\{d\+1\}\{N\(N\-1\)\}\\right\]\\lVert G\\rVert\_\{F\}^\{2\}\.\(122\)ForG≠0G\\neq 0, comparing the variance of2​N2NLOO\-ROLL directions withNNantithetic directions at equal evaluation cost gives

𝔼⁡\[∥g^LOO​\(W,2​N\)−G∥F2\]𝔼⁡\[∥g^anti​\(W,N\)−G∥F2\]=12\+d\+12​κr​\(2​N−1\)\.\\frac\{\\mathbb\{E\}\[\\lVert\\widehat\{g\}^\{\\mathrm\{LOO\}\}\(W;2N\)\-G\\rVert\_\{F\}^\{2\}\]\}\{\\mathbb\{E\}\[\\lVert\\widehat\{g\}^\{\\mathrm\{anti\}\}\(W;N\)\-G\\rVert\_\{F\}^\{2\}\]\}=\\frac\{1\}\{2\}\+\\frac\{d\+1\}\{2\\kappa\_\{r\}\(2N\-1\)\}\.\(123\)

###### Proof\.

In the affine model, the constant fitness cancels from every centered score\. Writezi=vec⁡\(Ei\)z\_\{i\}=\\operatorname\{vec\}\(E\_\{i\}\),v=vec⁡\(G\)v=\\operatorname\{vec\}\(G\), andyi=zi⊤​vy\_\{i\}=z\_\{i\}^\{\\top\}v\. The vectorsziz\_\{i\}are independent, centered, and have identity covariance\. Withz¯=N−1​∑izi\\bar\{z\}=N^\{\-1\}\\sum\_\{i\}z\_\{i\}andy¯=N−1​∑iyi\\bar\{y\}=N^\{\-1\}\\sum\_\{i\}y\_\{i\}, the vectorized estimator is the sample covariance applied tovv:

vec⁡\(g^LOO​\(W,N\)\)=1N−1​∑i=1N\(zi−z¯\)​\(yi−y¯\)\.\\operatorname\{vec\}\(\\widehat\{g\}^\{\\mathrm\{LOO\}\}\(W;N\)\)=\\frac\{1\}\{N\-1\}\\sum\_\{i=1\}^\{N\}\(z\_\{i\}\-\\bar\{z\}\)\(y\_\{i\}\-\\bar\{y\}\)\.\(124\)Expanding the two means gives the following centered decomposition:

vec⁡\(g^LOO​\(W,N\)\)−v=1N​∑i\(zi​yi−v\)−1N⁡\(N−1\)​∑i<j\(zi​yj\+zj​yi\)\.\\operatorname\{vec\}\(\\widehat\{g\}^\{\\mathrm\{LOO\}\}\(W;N\)\)\-v=\\frac\{1\}\{N\}\\sum\_\{i\}\(z\_\{i\}y\_\{i\}\-v\)\-\\frac\{1\}\{N\(N\-1\)\}\\sum\_\{i<j\}\(z\_\{i\}y\_\{j\}\+z\_\{j\}y\_\{i\}\)\.\(125\)Every cross\-pair summand has conditional mean zero given either of its two directions\. Consequently, these summands are uncorrelated with the first sum and with one another, including when two pairs share an index\. By[Theorem7\.1](https://arxiv.org/html/2609.10980#S7.Thmtheorem1),𝔼⁡\[∥zi​yi−v∥2\]=κr​∥v∥2\\mathbb\{E\}\[\\lVert z\_\{i\}y\_\{i\}\-v\\rVert^\{2\}\]=\\kappa\_\{r\}\\lVert v\\rVert^\{2\}\. Independence and identity covariance also give

𝔼⁡\[∥zi​yj\+zj​yi∥2\]=2​𝔼​\[∥zi∥2\]​𝔼​\[yj2\]\+2​∥𝔼⁡\[zi​yi\]∥2=2​\(d\+1\)​∥v∥2,i≠j\.\\mathbb\{E\}\[\\lVert z\_\{i\}y\_\{j\}\+z\_\{j\}y\_\{i\}\\rVert^\{2\}\]=2\\mathbb\{E\}\[\\lVert z\_\{i\}\\rVert^\{2\}\]\\mathbb\{E\}\[y\_\{j\}^\{2\}\]\+2\\lVert\\mathbb\{E\}\[z\_\{i\}y\_\{i\}\]\\rVert^\{2\}=2\(d\+1\)\\lVert v\\rVert^\{2\},\\qquad i\\neq j\.\(126\)There areN⁡\(N−1\)/2N\(N\-1\)/2unordered pairs\. Adding their variances therefore proves[Equation122](https://arxiv.org/html/2609.10980#S9.E122)\. Substituting2​N2Nfor the LOO\-ROLL population and dividing by the antithetic varianceκr​∥G∥F2/N\\kappa\_\{r\}\\lVert G\\rVert\_\{F\}^\{2\}/Nproves[Equation123](https://arxiv.org/html/2609.10980#S9.E123)\. ∎

Antithetic sampling and baseline subtraction are both established ways to reduce the variance of score\-function estimators\[[35](https://arxiv.org/html/2609.10980#bib.bib1),[47](https://arxiv.org/html/2609.10980#bib.bib45),[46](https://arxiv.org/html/2609.10980#bib.bib22)\]\. EGGROLL spends two evaluations on each independent direction and subtracts their fitnesses, canceling reward components shared by the pair and all even\-order terms in the local expansion\. LOO\-ROLL instead removes the population\-level reward offset with a leave\-one\-out baseline, allowing every evaluation to use a new independent direction\. At fixedNN, estimating this baseline adds only an order\-N−2N^\{\-2\}term to the order\-N−1N^\{\-1\}directional variance\. Consequently, with the same2​N2Nfitness evaluations, LOO\-ROLL uses2​N2Nindependent directions rather thanNNantithetic directions, and[Equation123](https://arxiv.org/html/2609.10980#S9.E123)shows that its variance in the affine model approaches one half of EGGROLL’s asNNgrows\.[Section10\.3](https://arxiv.org/html/2609.10980#S10.SS3)tests whether this advantage persists in transformer blocks and complete post\-training runs\. To use LOO\-ROLL in practical LLM post\-training, we retain the score processing used by EGGROLL’s LLM implementation\[[36](https://arxiv.org/html/2609.10980#bib.bib2)\]\. EGGROLL centers fitness values separately for each prompt, averages them over prompts, and divides the resulting population scores by their standard deviation\. Centering removes reward offsets shared by the population, while standardization makes the update magnitude less sensitive to the scale of the fitness values\. Keeping this procedure in LOO\-ROLL lets us compare the two estimators without also changing how their scores are scaled\. The leave\-one\-out scores differ from population\-centered scores only by the common factorN/\(N−1\)N/\(N\-1\), which cancels upon standardization, apart from the numerical stabilizer\. We can therefore implement LOO\-ROLL using the same centering and scaling operations, but with independently sampled one\-sided perturbations in place of antithetic pairs\. The propositions above describe the estimator before this scaling\. Appendix[F](https://arxiv.org/html/2609.10980#A6)analyzes how estimating the scale from the same population affects its expectation\.

We finally remark that LOO\-ROLL preserves the computational structure that makes EGGROLL efficient\[[36](https://arxiv.org/html/2609.10980#bib.bib2)\]\. Each evaluation still uses a low\-rank perturbation, allowing the population to share the base weights and use the same batched forward computations\. The implementation retains seed\-based perturbation reconstruction, layerwise weight updates, and the existing weight\-synchronization procedure\. The leave\-one\-out baseline is computed from the population mean already used for score centering, so it requires neither additional fitness evaluations nor additional model\-sized state or communication rounds\. LOO\-ROLL therefore retains EGGROLL’s low\-rank memory and batching advantages while allowing the evaluations previously spent on antithetic partners to fund more directions or further optimization steps\.

## 10Experiments: from finite\-rank theory to LLM post\-training

The preceding sections prescribe exact predictions for the finite\-rank population field, its possible departure from gradient dynamics, and the sampling error of a finite population\. We first test these predictions both in controlled synthetic experiments and later within actual transformer blocks\. We then examine whether increasing rank changes LLM post\-training performance when evaluations and wall time are matched\. Finally, we evaluate the practical consequence of the analysis developed in[Section9](https://arxiv.org/html/2609.10980#S9): LOO\-ROLL replaces each antithetic pair by a single evaluation\. Across ten matched\-wall\-time settings and models up to 8B parameters, LOO\-ROLL produces seven improvements in individual paired tests, three unresolved differences, and no significant loss\. Five gains survive Holm correction across the ten comparisons \([Table9](https://arxiv.org/html/2609.10980#A5.T9)\)\.

### 10\.1Validation of the theoretical predictions

#### 10\.1\.1Controlled tests of the theoretical predictions

We first test the predictions on controlled objectives, where the population field and sampling variance can be evaluated independently of the theoretical formulas\. The main findings are as follows\.

##### The sampling variance matches the exact prediction\.

Monte Carlo estimates agree with[Equation95](https://arxiv.org/html/2609.10980#S7.E95)to within0\.9%0\.9\\%at every tested rank\. The agreement validates the magnitude of the additional finite\-rank variance and its decrease as rank increases\. This supports the decomposition into a common dense\-Gaussian contribution and a surcharge proportional to1/r1/r\. Appendix[E\.3](https://arxiv.org/html/2609.10980#A5.SS3)reports the affine experiment and[Table15](https://arxiv.org/html/2609.10980#A5.T15)gives the numerical comparison\.

##### Finite rank can change the mean dynamics\.

Independent calculations reproduce the identity covariance and frequency\-dependent resolvent in[Section4](https://arxiv.org/html/2609.10980#S4)\. In the counterexample of[Proposition5\.3](https://arxiv.org/html/2609.10980#S5.Thmtheorem3), the largest real part of the population\-field Jacobian eigenvalues is0\.029250\.02925at rank one and−0\.00444\-0\.00444at rank two\. The calculation therefore recovers the predicted change from repulsion to local stability shown in[Figure3](https://arxiv.org/html/2609.10980#S5.F3), despite identity covariance\. The operator and Hessian calculations are described in Appendix[E\.3](https://arxiv.org/html/2609.10980#A5.SS3)\.

##### The radius and rank expansions recover their leading corrections\.

A cubic objective reproduces the third\-derivative correction in[Proposition6\.2](https://arxiv.org/html/2609.10980#S6.Thmtheorem2), including itsσ2/r\\sigma^\{2\}/rdependence\. A separate calculation recovers the leading1/r1/rcorrection at fixed radius in[Theorem8\.1](https://arxiv.org/html/2609.10980#S8.Thmtheorem1)\. Together, the calculations support the two distinct comparisons developed in the theory: departure from the original gradient at small radius, and departure from dense Gaussian ES at increasing rank\. Appendix[E\.3](https://arxiv.org/html/2609.10980#A5.SS3)describes them in detail\.

##### Population noise produces the predicted stability transition\.

Combining quadratic exactness with the1/N1/Npopulation\-variance formula gives the exact finite\-population dynamics in[Equation169](https://arxiv.org/html/2609.10980#A5.E169)\. For the identity quadratic used here, this expression gives the expected loss relative to its initial value\. The prediction agrees with both the one\-step measurements and the 25\-step Monte Carlo trajectories\. At step size0\.30\.3andN=16N=16, the expected loss grows with rank\-one perturbations but decreases with rank\-eight and dense Gaussian perturbations, reproducing the predicted stability transition\. Unlike the nonquadratic counterexample above, this transition is entirely due to sampling variance, since all three methods have the same exact mean field\.[SectionE\.3](https://arxiv.org/html/2609.10980#A5.SS3)derives the identity, gives the full protocol, and compares the predicted and measured trajectories in[Figure6](https://arxiv.org/html/2609.10980#A5.F6)\.

These controlled tests establish numerical agreement with the theoretical predictions\. We next ask whether the same effects are measurable inside a transformer, before evaluating their consequences for complete LLM post\-training runs and the equal\-time performance of LOO\-ROLL\.

#### 10\.1\.2Is the rank\-one surcharge visible in transformer blocks?

The affine calculation predicts that finite rank should add a tangible MSE surcharge in the case of a small matrix, but an almost negligible one at full transformer width\. We test both the variance and mean\-direction predictions inside Qwen3\-0\.6B\. The audit perturbs a16×1616\\times 16sub\-block of one attention matrix or one MLP matrix while evaluating next\-token prediction loss on a fixed minibatch\. Backpropagation supplies the reference gradient\. We compare ranksr∈\{1,2,4,8\}r\\in\\\{1,2,4,8\\\}with dense Gaussian ES over four perturbation radii, using8,1928,192directions and three independent perturbation seeds\. Appendix[E](https://arxiv.org/html/2609.10980#A5)provides the full experimental protocol and numerical aggregates\.

Figure 4:Finite\-rank audit of two Qwen3\-0\.6B blocks\.Left:single\-direction MSE relative to dense Gaussian ES\. The dashed curve is the exact affine prediction\[257\+66/r\]/257\[257\+66/r\]/257for a16×1616\\times 16block\.Right:the mean of 8,192 antithetic directions remains aligned with the backpropagated block gradient\. Points average four radii within each seed, and bars are standard errors over three seeds\.##### The finite\-rank MSE surcharge follows the affine prediction\.

In the left\-hand side plot of[Figure4](https://arxiv.org/html/2609.10980#S10.F4), we observe that the attention\-block ratios at ranks1,2,4,81,2,4,8are1\.264,1\.127,1\.089,1\.0281\.264,1\.127,1\.089,1\.028, very close to the theory\-predicted values1\.257,1\.128,1\.064,1\.0321\.257,1\.128,1\.064,1\.032\. The MLP block shows the same decline toward dense Gaussian ES\. The surcharge is therefore visible at16×1616\\times 16, while[Equation99](https://arxiv.org/html/2609.10980#S7.E99)predicts that it falls below0\.1%0\.1\\%for a square width\-40964096matrix\.

##### The mean direction does not change detectably with rank\.

The right\-hand side plot of[Figure4](https://arxiv.org/html/2609.10980#S10.F4)shows cosine similarity between0\.9800\.980and0\.9840\.984with the backpropagated gradient across ranks and radii, while adjacent\-rank differences remain at the Monte Carlo scale\. The experiment therefore resolves the variance effect predicted by[Theorem7\.1](https://arxiv.org/html/2609.10980#S7.Thmtheorem1), but not a systematic finite\-rank change in the mean field\.

##### Score standardization has negligible directional effect at the training population\.

The theory above concerns unstandardized scores, while the complete training runs retain EGGROLL’s score standardization\. AtN=128N=128, the standardized and unstandardized rank\-one estimates have a cosine similarity of0\.999700\.99970in[Table16](https://arxiv.org/html/2609.10980#A6.T16)\. Appendix[F](https://arxiv.org/html/2609.10980#A6)develops the corresponding bias bound and reports the full audit\.

### 10\.2Consequences for LLM post\-training

#### 10\.2\.1Does higher rank improve training at matched cost?

Increasing rank can reduce both the finite\-radius field error and the finite\-population variance, but also makes each update computationally more expensive\. We test whether that tradeoff favors rank eight over rank one in complete post\-training runs on Countdown\[[15](https://arxiv.org/html/2609.10980#bib.bib52),[33](https://arxiv.org/html/2609.10980#bib.bib53)\], GSM8K\[[9](https://arxiv.org/html/2609.10980#bib.bib34)\], and next\-token prediction\. The study spans Qwen3 models from 0\.6B to 8B parameters\[[50](https://arxiv.org/html/2609.10980#bib.bib35)\]and SmolLM2\-1\.7B\[[3](https://arxiv.org/html/2609.10980#bib.bib36)\]\. Every principal comparison uses five paired seeds and a fixed validation set\. Matched\-evaluation runs hold the number of iterations, population sizes, and model evaluations constant\. In matched\-wall\-time runs, rank one gets to spend the time saved on additional updates\.[AppendixE](https://arxiv.org/html/2609.10980#A5)contains the results measured during training, the experiments at the larger perturbation radius, and the complete protocol\.

##### Higher rank does not yield a reproducible reward advantage\.

On fixed Countdown and GSM8K evaluations, every rank\-eight minus rank\-one confidence interval in[Table1](https://arxiv.org/html/2609.10980#S10.T1)contains zero\. The intermediate ranks tested at 0\.6B show no monotone trend\. Rank eight also raises mean iteration time by1212–25%25\\%across the broader reward\-based study\. The predicted finite\-rank improvements are therefore too small to produce a reliable reward gain at the tested radius and populations\.

Table 1:Fixed\-validation changes in held\-out task reward from rank one to rank eight, in percentage points, over five matched seeds\. Positive values favor rank eight and negative values favor rank one\. The evaluation set contains 256 examples and is shared across ranks and budgets\. “Matched evaluations” uses the same ES iterations and model evaluations\. “Matched wall time” extends rank\-one training to the measured rank\-eight runtime\. Brackets give paired 95% Student\-ttintervals\.ModelTaskMatched evaluationsMatched wall timeQwen3\-0\.6BCountdown−\.60%​\[−3\.28,2\.07\]\-\.60\\%\\;\[\-3\.28,2\.07\]−1\.57%​\[−3\.29,\.15\]\-1\.57\\%\\;\[\-3\.29,\.15\]Qwen3\-0\.6BGSM8K\+2\.50%​\[−\.82,5\.82\]\+2\.50\\%\\;\[\-\.82,5\.82\]−5\.70%​\[−14\.57,3\.16\]\-5\.70\\%\\;\[\-14\.57,3\.16\]Qwen3\-1\.7BCountdown\+1\.50%​\[−\.29,3\.30\]\+1\.50\\%\\;\[\-\.29,3\.30\]\+1\.88%​\[−\.25,4\.01\]\+1\.88\\%\\;\[\-\.25,4\.01\]Qwen3\-1\.7BGSM8K\+\.94%​\[−\.76,2\.64\]\+\.94\\%\\;\[\-\.76,2\.64\]−1\.33%​\[−2\.88,\.23\]\-1\.33\\%\\;\[\-2\.88,\.23\]
##### Higher rank improves progress per update on next\-token prediction, but usually not progress per unit time\.

We repeat the comparison in[Table2](https://arxiv.org/html/2609.10980#S10.T2)on 256 fixed held\-out sequences at each Qwen3 model size, using five paired seeds\. With the same iterations and evaluations, rank eight achieves lower loss at all four sizes\. Once rank one is allowed to train for the same wall time, the advantage disappears at 0\.6B and reverses at 4B and 8B\. Rank eight retains an advantage only at 1\.7B\. Thus higher rank can improve each update, but its additional wall\-time cost usually outweighs the improvement it leads to\.

Table 2:Percentage change in next\-token prediction loss from rank one to rank eight over five paired seeds and 256 fixed held\-out sequences\. Negative values indicate lower loss at rank eight\. Brackets give paired 95% Student\-ttintervals\. Bold entries exclude zero\.ModelMatched evaluationsMatched wall timeQwen3\-0\.6B−\.58%​\[−\.86,−\.30\]\\bm\{\-\.58\\%\\;\[\-\.86,\-\.30\]\}−\.11%​\[−\.29,\.07\]\-\.11\\%\\;\[\-\.29,\.07\]Qwen3\-1\.7B−2\.23%​\[−2\.53,−1\.92\]\\bm\{\-2\.23\\%\\;\[\-2\.53,\-1\.92\]\}−\.93%​\[−1\.34,−\.53\]\\bm\{\-\.93\\%\\;\[\-1\.34,\-\.53\]\}Qwen3\-4B−1\.76%​\[−1\.95,−1\.57\]\\bm\{\-1\.76\\%\\;\[\-1\.95,\-1\.57\]\}\+1\.92%​\[1\.48,2\.35\]\\bm\{\+1\.92\\%\\;\[1\.48,2\.35\]\}Qwen3\-8B−2\.18%​\[−2\.60,−1\.77\]\\bm\{\-2\.18\\%\\;\[\-2\.60,\-1\.77\]\}\+2\.14%​\[1\.95,2\.33\]\\bm\{\+2\.14\\%\\;\[1\.95,2\.33\]\}
##### Performance is substantially more sensitive to radius than to rank\.

Raisingσ\\sigmafrom10−310^\{\-3\}to4×10−34\\times 10^\{\-3\}reduces mean reward by7070–100%100\\%across the 0\.6B and 1\.7B studies, whereas useful\-radius rank differences are small \([Figures5](https://arxiv.org/html/2609.10980#A5.F5)and[10](https://arxiv.org/html/2609.10980#A5.T10)\)\. The kind of adversarial instability we can observe in[Proposition5\.3](https://arxiv.org/html/2609.10980#S5.Thmtheorem3)therefore does not seem to appear in the tested transformer settings\. Appendix[E](https://arxiv.org/html/2609.10980#A5)reports the runs with larger perturbation radius, expanded rank comparisons, and transformer\-block measurements\. The score\-standardization audit is reported separately in[AppendixF](https://arxiv.org/html/2609.10980#A6)\.

### 10\.3LOO\-ROLL halves evaluation cost and improves equal\-time performance

The principal practical result is that LOO\-ROLL improves post\-training performance at matched wall\-clock training time\. On the full GSM8K test set, the mean accuracy gains over antithetic EGGROLL are24\.9024\.90,2\.852\.85, and14\.1314\.13percentage points at 0\.6B, 1\.7B, and 8B parameters, respectively\. We examine this result through three comparisons: the error of an individual update, performance at the same evaluation budget, and performance at the same wall time\. Together, these distinguish the statistical effect of replacing antithetic pairs from the benefit of spending the saved evaluations on further training\.

##### At equal evaluation cost, update MSE is approximately halved\.

Holding the number of directions fixed isolates the statistical price of dropping the negative evaluation\. Holding the number of fitness evaluations fixed instead allows LOO\-ROLL to sample twice as many directions as antithetic EGGROLL\.[Table3](https://arxiv.org/html/2609.10980#S10.T3)averages over two transformer blocks, two perturbation radii, and three seeds\. At the same number of directions, the MSE increase is at most5\.2%5\.2\\%and falls to0\.8%0\.8\\%byN=128N=128\. At the same evaluation cost, doubling the directions reduces MSE by approximately one half at every population size\.

Table 3:Signed percentage change in LOO\-ROLL MSE relative to antithetic EGGROLL\. Positive values mean higher MSE and negative values mean lower MSE\. The equal\-direction comparison usesNNdirections for both estimators, while the equal\-evaluation comparison uses2​N2NLOO\-ROLL directions againstNNantithetic directions\. Values average twelve block–radius–seed settings\.Antithetic directionsNNEqual directionsEqual evaluations16\+5\.2%\+5\.2\\%−49\.1%\\bm\{\-49\.1\\%\}32\+2\.5%\+2\.5\\%−49\.0%\\bm\{\-49\.0\\%\}64\+1\.2%\+1\.2\\%−50\.1%\\bm\{\-50\.1\\%\}128\+0\.8%\+0\.8\\%−49\.5%\\bm\{\-49\.5\\%\}
##### LOO\-ROLL substantially improves performance at equal wall time\.

[Tables4](https://arxiv.org/html/2609.10980#S10.T4)and[5](https://arxiv.org/html/2609.10980#S10.T5)compare complete post\-training runs over five paired seeds\. Equal\-iteration runs use the same number of directions per update and ask what is lost by omitting the negative evaluation\. Equal\-evaluation runs give LOO\-ROLL twice as many directions\.[Table7](https://arxiv.org/html/2609.10980#A5.T7)specifies the direction counts, training batches, update budgets, and measured runtimes\. The principal comparison gives LOO\-ROLL its original population size and continues training until the measured wall time matches the paired EGGROLL run\. Across the original 45 pairs, the mean LOO\-ROLL\-to\-EGGROLL wall\-time ratio is1\.0051\.005\. Across the individual model–task settings, LOO\-ROLL’s mean runtime is between0\.3%0\.3\\%shorter and2\.7%2\.7\\%longer than EGGROLL’s, so the matched\-wall\-time results compare performance under nearly equal computational budgets\. At this budget, individual paired tests resolve improvements in seven of the ten settings and no loss, with five gains surviving Holm correction across all ten tests \([Table9](https://arxiv.org/html/2609.10980#A5.T9)\)\.

All GSM8K comparisons use the full set of 1,319 test questions\. Countdown and the next\-token prediction experiments use 256 shared held\-out examples\. Tables[4](https://arxiv.org/html/2609.10980#S10.T4)and[5](https://arxiv.org/html/2609.10980#S10.T5)report mean changes and sample standard deviations across the five paired training seeds\. These describe seed variability on the fixed evaluation set\. Appendix[E\.1](https://arxiv.org/html/2609.10980#A5.SS1)gives paired tests and a multiple\-comparison robustness analysis\.

Table 4:LOO\-ROLL minus antithetic EGGROLL reward\. GSM8K results use all 1,319 test questions, while Countdown results use 256 shared held\-out examples\. Values are mean±\\pmsample standard deviation of five paired training\-seed differences, in percentage points\. Bold marks positive mean improvements\. The equal\-wall gains correspond to relative improvements of\+65\.3%\+65\.3\\%on Qwen3\-0\.6B GSM8K,\+9\.1%\+9\.1\\%on Qwen3\-1\.7B Countdown,\+21\.5%\+21\.5\\%on Qwen3\-8B GSM8K, and\+22\.6%\+22\.6\\%on SmolLM2\-1\.7B GSM8K\.ModelTaskEqual iterationsEqual evaluationsEqual wall timeSmolLM2\-1\.7BCountdown−0\.20%±0\.22\-0\.20\\%\\pm 0\.22−0\.08%±0\.34\-0\.08\\%\\pm 0\.34−0\.10%±0\.46\-0\.10\\%\\pm 0\.46SmolLM2\-1\.7BGSM8K\+0\.50%±0\.64\\bm\{\+0\.50\\%\\pm 0\.64\}\+0\.23%±0\.74\\bm\{\+0\.23\\%\\pm 0\.74\}\+4\.18%±1\.54\\bm\{\+4\.18\\%\\pm 1\.54\}Qwen3\-0\.6BCountdown−4\.09%±3\.97\-4\.09\\%\\pm 3\.97−1\.05%±1\.49\-1\.05\\%\\pm 1\.49−0\.62%±3\.46\-0\.62\\%\\pm 3\.46Qwen3\-0\.6BGSM8K\+5\.35%±1\.76\\bm\{\+5\.35\\%\\pm 1\.76\}−4\.82%±4\.87\-4\.82\\%\\pm 4\.87\+24\.90%±2\.38\\bm\{\+24\.90\\%\\pm 2\.38\}Qwen3\-1\.7BCountdown−3\.89%±1\.22\-3\.89\\%\\pm 1\.22−0\.31%±1\.56\-0\.31\\%\\pm 1\.56\+1\.68%±1\.03\\bm\{\+1\.68\\%\\pm 1\.03\}Qwen3\-1\.7BGSM8K0\.00%±1\.010\.00\\%\\pm 1\.01−0\.26%±0\.81\-0\.26\\%\\pm 0\.81\+2\.85%±0\.83\\bm\{\+2\.85\\%\\pm 0\.83\}Qwen3\-8BGSM8K\+0\.85%±0\.66\\bm\{\+0\.85\\%\\pm 0\.66\}−1\.08%±1\.21\-1\.08\\%\\pm 1\.21\+14\.13%±0\.88\\bm\{\+14\.13\\%\\pm 0\.88\}Table 5:Percentage reduction in fixed\-validation negative log likelihood for LOO\-ROLL relative to antithetic EGGROLL\. Values are mean±\\pmsample standard deviation of five paired loss differences, each divided by the mean EGGROLL loss for that model and multiplied by 100\. Bold marks positive mean improvements\.ModelEqual iterationsEqual evaluationsEqual wall timeQwen3\-0\.6B−0\.20%±0\.44\-0\.20\\%\\pm 0\.44−0\.11%±0\.30\-0\.11\\%\\pm 0\.30\+0\.55%±0\.46\\bm\{\+0\.55\\%\\pm 0\.46\}Qwen3\-1\.7B−0\.21%±0\.58\-0\.21\\%\\pm 0\.58\+0\.10%±0\.34\\bm\{\+0\.10\\%\\pm 0\.34\}\+1\.20%±0\.62\\bm\{\+1\.20\\%\\pm 0\.62\}Qwen3\-8B−0\.53%±0\.57\-0\.53\\%\\pm 0\.57−0\.31%±0\.49\-0\.31\\%\\pm 0\.49\+2\.16%±0\.81\\bm\{\+2\.16\\%\\pm 0\.81\}At equal evaluations, the mean effects are small relative to seed variability, and the paired tests in[Table8](https://arxiv.org/html/2609.10980#A5.T8)do not resolve a difference in any of the ten settings\. This does not establish equivalence\. The clear advantage appears at matched wall time, where LOO\-ROLL spends its savings over evaluations on further updates\. At equal iterations, seven comparisons are unresolved, Qwen3\-0\.6B and Qwen3\-8B GSM8K favor LOO\-ROLL, and Qwen3\-1\.7B Countdown favors antithetic EGGROLL\.

##### Additional updates yield consistent equal\-time gains on GSM8K\.

Full\-test evaluation in[Table4](https://arxiv.org/html/2609.10980#S10.T4)gives38\.10%±2\.0338\.10\\%\\pm 2\.03accuracy for EGGROLL and63\.00%±0\.8063\.00\\%\\pm 0\.80for LOO\-ROLL at 0\.6B\. At 1\.7B, accuracy rises from80\.80%±0\.7380\.80\\%\\pm 0\.73to83\.65%±0\.5283\.65\\%\\pm 0\.52\. At 8B, accuracy rises from65\.87%±0\.7865\.87\\%\\pm 0\.78to80\.00%±0\.7880\.00\\%\\pm 0\.78\. The paired gains are positive for every training seed at all three sizes\. All three Qwen3 budget comparisons use their previously saved checkpoints and the same decoding settings on the full test set, so the expanded evaluation changes neither training nor the cost comparisons\. On SmolLM2\-1\.7B, accuracy rises from18\.54%±0\.5118\.54\\%\\pm 0\.51to22\.73%±1\.1522\.73\\%\\pm 1\.15, a relative gain of\+22\.6%\+22\.6\\%with improvements in all five seeds\. Next\-token prediction loss in[Table5](https://arxiv.org/html/2609.10980#S10.T5)also falls significantly at 1\.7B and 8B, by1\.20%1\.20\\%and2\.16%2\.16\\%relative to their EGGROLL baselines\. Across the seven gains resolved by individual equal\-wall tests, all 35 paired\-seed differences favor LOO\-ROLL\. No setting significantly favors antithetic EGGROLL\.

Population size marks the boundary behind this result\. On Qwen3\-0\.6B with next\-token prediction loss, comparing the same number of directions makes the LOO\-ROLL validation gap fall monotonically from0\.2840\.284at 8 directions to0\.0420\.042at 64 directions\. LOO\-ROLL is therefore not preferable at every small fixed population\. The evaluation saving becomes useful when it funds either more directions or more optimization steps\. The transformer\-block audit and complete runs together show both effects: twice as many directions halve update MSE at equal evaluation cost, while additional steps yield large and consistent gains at equal wall time\.

## 11Conclusion

Finite\-rank EGGROLL raises two distinct questions: which direction the method follows on average, and how accurately a finite population estimates that direction\. The resolvent characterization shows how low rank can alter the mean dynamics, including the stability of an optimum\. EGGROLL nevertheless recovers the gradient exactly on quadratic objectives, while the small\-radius expansion identifies the first departure through third derivatives\. The variance formula gives a complementary result: the additional rank\-one contribution grows in the order ofm\+nm\+n, compared with the common dense\-Gaussian contribution of orderm​nmn\. These results explain how rank one can retain nearly all of the sampling efficiency of dense Gaussian ES even when their mean dynamics differ\.

The experiments connect this analysis to LLM post\-training\. Transformer blocks exhibit the predicted decline in sampling error with rank, while the complete training runs show that improvements per update must be weighed against their computational cost\. LOO\-ROLL uses that budget more effectively by replacing antithetic pairs with independent directions and leave\-one\-out baselines\. Before score standardization, the estimator preserves the population field\. LOO\-ROLL retains EGGROLL’s low\-rank memory, batching, and communication machinery\. At equal evaluation cost, the additional directions approximately halve estimator MSE in transformer blocks\. At matched wall time, LOO\-ROLL improves seven of ten tested settings in individual paired tests, with no significant loss, and five gains survive Holm correction\. On the full GSM8K test set, accuracy rises from38\.1%38\.1\\%to63\.0%63\.0\\%at 0\.6B and from65\.9%65\.9\\%to80\.0%80\.0\\%at 8B, with gains in every paired seed at all three tested sizes\. The finite\-rank analysis thus leads to a practical improvement that preserves EGGROLL’s computational advantages and uses its evaluations more efficiently\.

The exact variance comparison uses a local affine model, and the experiments cover a finite collection of objectives and models up to 8B parameters\. Appendix[F](https://arxiv.org/html/2609.10980#A6)treats the score standardization used in practice, while extending the empirical conclusions to longer training horizons and larger model scales remains an important direction for future work\.

## References

- \[1\]P\. Absil, R\. Mahony, and R\. Sepulchre\(2008\)Optimization algorithms on matrix manifolds\.Princeton University Press\.Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p4.3)\.
- \[2\]A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin, A\. Üstün, and S\. Hooker\(2024\)Back to basics: revisiting REINFORCE\-style optimization for learning from human feedback in LLMs\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12248–12267\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p2.1),[§9\.1](https://arxiv.org/html/2609.10980#S9.SS1.p1.1)\.
- \[3\]L\. B\. Allal, A\. Lozhkov, E\. Bakouch, G\. M\. Blázquez, G\. Penedo, L\. Tunstall, A\. Marafioti, H\. Kydlíček, A\. P\. Lajarín, V\. Srivastav,et al\.\(2025\)SmolLM2: when smol goes big–data\-centric training of a small language model\.arXiv preprint arXiv:2502\.02737\.Cited by:[§10\.2\.1](https://arxiv.org/html/2609.10980#S10.SS2.SSS1.p1.1)\.
- \[4\]K\. Balasubramanian and S\. Ghadimi\(2022\)Zeroth\-order nonconvex stochastic optimization: handling constraints, high dimensionality, and saddle points\.Foundations of Computational Mathematics22\(1\),pp\. 35–76\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p1.1)\.
- \[5\]H\. H\. Bauschke and P\. L\. Combettes\(2020\)Correction to: convex analysis and monotone operator theory in Hilbert spaces\.InConvex analysis and monotone operator theory in Hilbert spaces,pp\. C1–C4\.Cited by:[§4](https://arxiv.org/html/2609.10980#S4.p10.2),[§4](https://arxiv.org/html/2609.10980#S4.p10.3)\.
- \[6\]H\. Beyer and H\. Schwefel\(2002\)Evolution strategies–a comprehensive introduction\.Natural computing1\(1\),pp\. 3–52\.Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p1.1)\.
- \[7\]K\. M\. Choromanski, A\. Pacchiano, J\. Parker\-Holder, Y\. Tang, and V\. Sindhwani\(2019\)From complexity to simplicity: adaptive ES\-active subspaces for blackbox optimization\.Advances in Neural Information Processing Systems32\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p4.1)\.
- \[8\]K\. Choromanski, M\. Rowland, V\. Sindhwani, R\. Turner, and A\. Weller\(2018\)Structured evolution with compact architectures for scalable policy optimization\.InInternational Conference on Machine Learning,pp\. 970–978\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p4.1)\.
- \[9\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§10\.2\.1](https://arxiv.org/html/2609.10980#S10.SS2.SSS1.p1.1)\.
- \[10\]T\. Dao, D\. Fu, S\. Ermon, A\. Rudra, and C\. Ré\(2022\)Flashattention: fast and memory\-efficient exact attention with IO\-awareness\.Advances in neural information processing systems35,pp\. 16344–16359\.Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p1.1)\.
- \[11\]A\. P\. Dawid\(1981\)Some matrix\-variate distribution theory: notational considerations and a Bayesian application\.Biometrika68\(1\),pp\. 265–274\.Cited by:[§4](https://arxiv.org/html/2609.10980#S4.p4.1)\.
- \[12\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)QLoRA: efficient finetuning of quantized LLMs\.Advances in neural information processing systems36,pp\. 10088–10115\.Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p3.2)\.
- \[13\]J\. C\. Duchi, M\. I\. Jordan, M\. J\. Wainwright, and A\. Wibisono\(2015\)Optimal rates for zero\-order convex optimization: the power of two function evaluations\.IEEE Transactions on Information Theory61\(5\),pp\. 2788–2806\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p1.1)\.
- \[14\]G\. B\. Folland\(1999\)Real analysis: modern techniques and their applications\.John Wiley & Sons\.Cited by:[§4](https://arxiv.org/html/2609.10980#S4.p12.3.1),[§4](https://arxiv.org/html/2609.10980#S4.p8.1),[§8](https://arxiv.org/html/2609.10980#S8.p10.1.1)\.
- \[15\]K\. Gandhi, D\. H\. J\. Lee, G\. Grand, M\. Liu, W\. Cheng, A\. Sharma, and N\. Goodman\(2024\)Stream of search \(SoS\): learning to search in language\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=2cop2jmQVL)Cited by:[§10\.2\.1](https://arxiv.org/html/2609.10980#S10.SS2.SSS1.p1.1)\.
- \[16\]R\. E\. Gaunt\(2018\)On Stein’s method for products of normal random variables and zero bias couplings\.Bernoulli23\(4B\),pp\. 3311–3345\.External Links:ISSN 13507265,[Link](http://www.jstor.org/stable/26491760)Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p5.1)\.
- \[17\]R\. Gaunt\(2018\)Products of normal, beta and gamma random variables: Stein operators and distributional theory\.Brazilian Journal of Probability and Statistics32\(2\),pp\. 437–466\(English\)\.External Links:[Document](https://dx.doi.org/10.1214/16-BJPS349)Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p5.1)\.
- \[18\]T\. Gautam, Y\. Park, H\. Zhou, P\. Raman, and W\. Ha\(2024\)Variance\-reduced zeroth\-order methods for fine\-tuning language models\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 15180–15208\.External Links:[Link](https://proceedings.mlr.press/v235/gautam24a.html)Cited by:[§B\.2](https://arxiv.org/html/2609.10980#A2.SS2.p1.1),[§1](https://arxiv.org/html/2609.10980#S1.p1.1),[§2](https://arxiv.org/html/2609.10980#S2.p1.1)\.
- \[19\]S\. Ghadimi and G\. Lan\(2013\)Stochastic first\-and zeroth\-order methods for nonconvex stochastic programming\.SIAM journal on optimization23\(4\),pp\. 2341–2368\.Cited by:[§6](https://arxiv.org/html/2609.10980#S6.p10.1)\.
- \[20\]L\. Grafakoset al\.\(2008\)Classical Fourier analysis\.Vol\.2,Springer\.Cited by:[§3](https://arxiv.org/html/2609.10980#S3.p4.1)\.
- \[21\]N\. Hansen\(2016\)The CMA evolution strategy: a tutorial\.arXiv preprint arXiv:1604\.00772\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p4.1)\.
- \[22\]N\. J\. Higham\(2008\)Functions of matrices: theory and computation\.SIAM\.Cited by:[§8](https://arxiv.org/html/2609.10980#S8.p2.1)\.
- \[23\]S\. Holm\(1979\)A simple sequentially rejective multiple test procedure\.Scandinavian journal of statistics,pp\. 65–70\.Cited by:[§E\.1](https://arxiv.org/html/2609.10980#A5.SS1.SSS0.Px3.p5.1)\.
- \[24\]W\. Hoy, B\. Wang, and X\. Pan\(2026\)Matching accuracy, different geometry: evolution strategies vs GRPO in LLM post\-training\.InThird Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=HsyvUQdeGS)Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p1.1),[§2](https://arxiv.org/html/2609.10980#S2.p3.1)\.
- \[25\]E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p3.2)\.
- \[26\]L\. Isserlis\(1918\)On a formula for the product\-moment coefficient of any order of a normal frequency distribution in any number of variables\.Biometrika12\(1/2\),pp\. 134–139\.Cited by:[§6](https://arxiv.org/html/2609.10980#S6.p6.3.1)\.
- \[27\]J\. Levi, S\. Nabarro, and M\. van der Wilk\(2026\)On the scalability of forward gradients, evolution strategies, and control variates\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=s6g8yZimHE)Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p3.1)\.
- \[28\]S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen\(2024\)DoRA: weight\-decomposed low\-rank adaptation\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 32100–32121\.External Links:[Link](https://proceedings.mlr.press/v235/liu24bn.html)Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p3.2)\.
- \[29\]N\. Maheswaranathan, L\. Metz, G\. Tucker, D\. Choi, and J\. Sohl\-Dickstein\(2019\)Guided evolutionary strategies: augmenting random search with surrogate gradients\.InInternational Conference on Machine Learning,pp\. 4264–4273\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p4.1)\.
- \[30\]S\. Malladi, T\. Gao, E\. Nichani, A\. Damian, J\. D\. Lee, D\. Chen, and S\. Arora\(2023\)Fine\-tuning language models with just forward passes\.Advances in Neural Information Processing Systems36,pp\. 53038–53075\.Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p1.1),[§2](https://arxiv.org/html/2609.10980#S2.p1.1)\.
- \[31\]R\. J\. Muirhead\(2009\)Aspects of multivariate statistical theory\.John Wiley & Sons\.Cited by:[§4](https://arxiv.org/html/2609.10980#S4.p3.5.1),[§5](https://arxiv.org/html/2609.10980#S5.p6.1.1)\.
- \[32\]Y\. Nesterov and V\. Spokoiny\(2017\)Random gradient\-free minimization of convex functions\.Foundations of Computational Mathematics17\(2\),pp\. 527–566\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p1.1),[§6](https://arxiv.org/html/2609.10980#S6.p10.1)\.
- \[33\]J\. Pan, J\. Zhang, X\. Wang, L\. Yuan, H\. Peng, and A\. Suhr\(2025\)TinyZero\.Note:https://github\.com/Jiayi\-Pan/TinyZeroAccessed: 2025\-01\-24Cited by:[§10\.2\.1](https://arxiv.org/html/2609.10980#S10.SS2.SSS1.p1.1)\.
- \[34\]I\. Rechenberg\(1978\)Evolutionsstrategien\.InSimulationsmethoden in der Medizin und Biologie: Workshop, Hannover, 29\. Sept\.–1\. Okt\. 1977,pp\. 83–114\.Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p1.1)\.
- \[35\]T\. Salimans, J\. Ho, X\. Chen, S\. Sidor, and I\. Sutskever\(2017\)Evolution strategies as a scalable alternative to reinforcement learning\.arXiv preprint arXiv:1703\.03864\.Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p1.1),[§9\.1](https://arxiv.org/html/2609.10980#S9.SS1.p6.1)\.
- \[36\]B\. Sarkar, M\. Fellows, J\. A\. Duque, A\. Letcher, A\. L\. Villares, A\. Sims, C\. Wibault, D\. Samsonov, D\. Cope, J\. L\. Liesen, K\. Li, L\. Seier, T\. Wolf, U\. Berdica, V\. Mohl, A\. D\. Goldie, A\. Courville, K\. Sevegnani, S\. Whiteson, and J\. N\. Foerster\(2026\)Evolution strategies at the hyperscale\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=bfVJ4GsHrO)Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p1.1),[§1](https://arxiv.org/html/2609.10980#S1.p2.1),[§1](https://arxiv.org/html/2609.10980#S1.p4.1),[§2](https://arxiv.org/html/2609.10980#S2.p3.1),[§3](https://arxiv.org/html/2609.10980#S3.p1.4),[§3](https://arxiv.org/html/2609.10980#S3.p2.1),[§8](https://arxiv.org/html/2609.10980#S8.p13.2),[§9\.1](https://arxiv.org/html/2609.10980#S9.SS1.p6.1),[§9\.1](https://arxiv.org/html/2609.10980#S9.SS1.p7.1),[§9](https://arxiv.org/html/2609.10980#S9.p2.1)\.
- \[37\]H\. Schwefel\(1977\)Numerische optimierung von computer\-modellen mittels der evolutionsstrategie: mit einer vergleichenden einführung in die hill\-climbing\-und zufallsstrategie\.Vol\.57,Springer\.Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p1.1)\.
- \[38\]H\. Seung, J\. Lee, and H\. Ko\(2026\)Low\-rank curvature for zeroth\-order optimization in LLM fine\-tuning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 25235–25242\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p2.1)\.
- \[39\]J\. C\. Spall\(1992\)Multivariate stochastic approximation using a simultaneous perturbation gradient approximation\.IEEE transactions on automatic control37\(3\),pp\. 332–341\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p1.1)\.
- \[40\]M\. Spivak\(1965\)Calculus on manifolds: a modern approach to classical theorems of advanced calculus\.Westview Press\.Cited by:[§5](https://arxiv.org/html/2609.10980#S5.p1.1)\.
- \[41\]C\. M\. Stein\(1981\)Estimation of the mean of a multivariate normal distribution\.The annals of Statistics,pp\. 1135–1151\.Cited by:[§3](https://arxiv.org/html/2609.10980#S3.p1.3)\.
- \[42\]Y\. Tang, K\. Choromanski, and A\. Kucukelbir\(2020\)Variance reduction for evolution strategies via structured control variates\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 646–656\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p4.1)\.
- \[43\]S\. Vaswani, F\. Bach, and M\. Schmidt\(2019\)Fast and faster convergence of SGD for over\-parameterized models and an accelerated perceptron\.InThe 22nd international conference on artificial intelligence and statistics,pp\. 1195–1204\.Cited by:[§C\.1](https://arxiv.org/html/2609.10980#A3.SS1.p11.1)\.
- \[44\]J\. Wang, X\. Li, and P\. Li\(2020\)Zeroth order optimization by a mixture of evolution strategies\.External Links:[Link](https://openreview.net/forum?id=SklE_CNFPr)Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p6.1)\.
- \[45\]Z\. Wang, K\. Balasubramanian, S\. Ma, and M\. Razaviyayn\(2023\)Zeroth\-order algorithms for nonconvex–strongly\-concave minimax problems with improved complexities\.Journal of Global Optimization87\(2\),pp\. 709–740\.Cited by:[§C\.1](https://arxiv.org/html/2609.10980#A3.SS1.p11.1)\.
- \[46\]D\. Wierstra, T\. Schaul, T\. Glasmachers, Y\. Sun, J\. Peters, and J\. Schmidhuber\(2014\)Natural evolution strategies\.The Journal of Machine Learning Research15\(1\),pp\. 949–980\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p4.1),[§9\.1](https://arxiv.org/html/2609.10980#S9.SS1.p1.1),[§9\.1](https://arxiv.org/html/2609.10980#S9.SS1.p6.1)\.
- \[47\]R\. J\. Williams\(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine learning8\(3\),pp\. 229–256\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p2.1),[§9\.1](https://arxiv.org/html/2609.10980#S9.SS1.p1.1),[§9\.1](https://arxiv.org/html/2609.10980#S9.SS1.p6.1)\.
- \[48\]S\. Williams, A\. Waterman, and D\. Patterson\(2009\)Roofline: an insightful visual performance model for multicore architectures\.Communications of the ACM52\(4\),pp\. 65–76\.Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p1.1)\.
- \[49\]Y\. Xu, K\. Schweighofer, R\. Miikkulainen, and X\. Qiu\(2026\)Quantized evolution strategies: high\-precision fine\-tuning of quantized LLMs at low\-precision cost\.arXiv preprint arXiv:2602\.03120\.Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p1.1)\.
- \[50\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§10\.2\.1](https://arxiv.org/html/2609.10980#S10.SS2.SSS1.p1.1)\.
- \[51\]Q\. Zhang, M\. Chen, A\. Bukharin, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao\(2023\)Adaptive budget allocation for parameter\-efficient fine\-tuning\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=lq62uWRJjiY)Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p3.2)\.
- \[52\]Y\. Zhang, Y\. Zhou, K\. Ji, Y\. Shen, and M\. M\. Zavlanos\(2024\)Boosting one\-point derivative\-free online optimization via residual feedback\.IEEE Transactions on Automatic Control69\(9\),pp\. 6309–6316\.Cited by:[§2](https://arxiv.org/html/2609.10980#S2.p1.1)\.
- \[53\]Y\. Zhao, S\. Dang, H\. Ye, G\. Dai, Y\. Qian, and I\. Tsang\(2025\)Second\-order fine\-tuning without pain for LLMs: a Hessian informed zeroth\-order optimizer\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 43496–43520\.Cited by:[§1](https://arxiv.org/html/2609.10980#S1.p1.1)\.

## Appendix AGeneralization to network parameters

The main results are stated for one matrix so that the finite\-rank mechanism remains visible\. A network objective couples many parameter matrices, but EGGROLL samples their perturbations independently\. This section gives the corresponding joint resolvent, field\-error, and variance statements\.

###### Corollary A\.1\(Network resolvent\)\.

LetW=\(W1,…,WQ\)W=\(W\_\{1\},\\ldots,W\_\{Q\}\), whereWq∈ℝmq×nqW\_\{q\}\\in\\mathbb\{R\}^\{m\_\{q\}\\times n\_\{q\}\}, and let the blocksErq\(q\)E\_\{r\_\{q\}\}^\{\(q\)\}be independent Gaussian\-product perturbations\. Define

F𝒓,σ​\(W\)=𝔼⁡\[f⁡\(W1\+σ​Er1\(1\),…,WQ\+σ​ErQ\(Q\)\)\],gq​\(W\)=1σ​𝔼​\[Erq\(q\)​f​\(W1\+σ​Er1\(1\),…,WQ\+σ​ErQ\(Q\)\)\]\.\\begin\{gathered\}F\_\{\\bm\{r\},\\sigma\}\(W\)=\\mathbb\{E\}\\big\[f\(W\_\{1\}\+\\sigma E\_\{r\_\{1\}\}^\{\(1\)\},\\ldots,W\_\{Q\}\+\\sigma E\_\{r\_\{Q\}\}^\{\(Q\)\}\)\\big\],\\\\ g\_\{q\}\(W\)=\\frac\{1\}\{\\sigma\}\\mathbb\{E\}\\left\[E\_\{r\_\{q\}\}^\{\(q\)\}f\(W\_\{1\}\+\\sigma E\_\{r\_\{1\}\}^\{\(1\)\},\\ldots,W\_\{Q\}\+\\sigma E\_\{r\_\{Q\}\}^\{\(Q\)\}\)\\right\]\.\\end\{gathered\}\(127\)Letℒq\\mathcal\{L\}\_\{q\}be the operator in[Equation28](https://arxiv.org/html/2609.10980#S4.E28), acting on theqqth vector\-field component and differentiating with respect toWqW\_\{q\}\. Ifffadmits the analogous Fourier representation and moment condition on the product parameter space, then

\(I\+σ2rq​ℒq\)​gq=∇WqF𝒓,σfor every​q\.\\left\(I\+\\frac\{\\sigma^\{2\}\}\{r\_\{q\}\}\\mathcal\{L\}\_\{q\}\\right\)g\_\{q\}=\\nabla\_\{W\_\{q\}\}F\_\{\\bm\{r\},\\sigma\}\\quad\\text\{for every \}q\.\(128\)Equivalently,g=J𝐫,σ∇F𝐫,σg=J\_\{\\bm\{r\},\\sigma\}\\nabla F\_\{\\bm\{r\},\\sigma\}forJ𝐫,σ=diagq⁡\(I\+σ2​ℒq/rq\)−1J\_\{\\bm\{r\},\\sigma\}=\\operatorname\{diag\}\_\{q\}\(I\+\\sigma^\{2\}\\mathcal\{L\}\_\{q\}/r\_\{q\}\)^\{\-1\}\. The operatorJ𝐫,σJ\_\{\\bm\{r\},\\sigma\}is self\-adjoint, positive, contractive, and firmly nonexpansive on the direct\-sumL2L^\{2\}space\.

###### Proof\.

Independence gives the joint characteristic function

Φ𝒓​\(T1,…,TQ\)=∏q=1QΦrq​\(Tq\)\.\\Phi\_\{\\bm\{r\}\}\(T\_\{1\},\\ldots,T\_\{Q\}\)=\\prod\_\{q=1\}^\{Q\}\\Phi\_\{r\_\{q\}\}\(T\_\{q\}\)\.\(129\)Substitution of the Fourier representation offf, followed by differentiation of theqqth factor and application of[Equation12](https://arxiv.org/html/2609.10980#S4.E12), gives the multiplier

i​Φ𝒓​\(σ​𝑻\)​\(Imq\+σ2​Tq​Tq⊤rq\)−1​Tq\.i\\Phi\_\{\\bm\{r\}\}\(\\sigma\\bm\{T\}\)\\left\(I\_\{m\_\{q\}\}\+\\frac\{\\sigma^\{2\}T\_\{q\}T\_\{q\}^\{\\top\}\}\{r\_\{q\}\}\\right\)^\{\-1\}T\_\{q\}\.\(130\)The multiplier of∇WqF𝒓,σ\\nabla\_\{W\_\{q\}\}F\_\{\\bm\{r\},\\sigma\}isi​Tq​Φ𝒓​\(σ​𝑻\)iT\_\{q\}\\Phi\_\{\\bm\{r\}\}\(\\sigma\\bm\{T\}\)\. The blockwise identity follows, and the remaining properties follow from[Theorem4\.2](https://arxiv.org/html/2609.10980#S4.Thmtheorem2)on each diagonal block\. ∎

For the smoothness bounds, writedq=mq​nqd\_\{q\}=m\_\{q\}n\_\{q\},D=∑qdqD=\\sum\_\{q\}d\_\{q\}, and𝐄=\(Er1\(1\),…,ErQ\(Q\)\)\\mathbf\{E\}=\(E\_\{r\_\{1\}\}^\{\(1\)\},\\ldots,E\_\{r\_\{Q\}\}^\{\(Q\)\}\)\. Equip this product space with the direct\-sum Frobenius norm∥𝐄∥2=∑q∥Erq\(q\)∥F2\\lVert\\mathbf\{E\}\\rVert^\{2\}=\\sum\_\{q\}\\lVert E\_\{r\_\{q\}\}^\{\(q\)\}\\rVert\_\{F\}^\{2\}\. Independence across blocks gives

μ4​\(𝒓\):=𝔼⁡\[∥𝐄∥4\]=D⁡\(D\+2\)\+∑q=1Q2​dq​\(mq\+nq\+1\)rq\.\\mu\_\{4\}\(\\bm\{r\}\):=\\mathbb\{E\}\[\\lVert\\mathbf\{E\}\\rVert^\{4\}\]=D\(D\+2\)\+\\sum\_\{q=1\}^\{Q\}\\frac\{2d\_\{q\}\(m\_\{q\}\+n\_\{q\}\+1\)\}\{r\_\{q\}\}\.\(131\)
###### Corollary A\.2\(Network field error under standard smoothness\)\.

Lethhbe a loss on the product of the block spaces, and defineg𝐫,σhg\_\{\\bm\{r\},\\sigma\}^\{h\}using the joint perturbation𝐄\\mathbf\{E\}\. IfhhisLL\-smooth in the direct\-sum norm, then

∥g𝒓,σh​\(𝐖\)−∇h​\(𝐖\)∥≤L​σ2​μ4​\(𝒓\)3/4\.\\lVert g\_\{\\bm\{r\},\\sigma\}^\{h\}\(\\mathbf\{W\}\)\-\\nabla h\(\\mathbf\{W\}\)\\rVert\\leq\\frac\{L\\sigma\}\{2\}\\mu\_\{4\}\(\\bm\{r\}\)^\{3/4\}\.\(132\)If the Hessian isMM\-Lipschitz in the same norm, then

∥g𝒓,σh​\(𝐖\)−∇h​\(𝐖\)∥≤M​σ26​μ4​\(𝒓\)\.\\lVert g\_\{\\bm\{r\},\\sigma\}^\{h\}\(\\mathbf\{W\}\)\-\\nabla h\(\\mathbf\{W\}\)\\rVert\\leq\\frac\{M\\sigma^\{2\}\}\{6\}\\mu\_\{4\}\(\\bm\{r\}\)\.\(133\)

###### Proof\.

The smoothness inequalities in[Lemma6\.3](https://arxiv.org/html/2609.10980#S6.Thmtheorem3)depend only on the norm of the full perturbation\. Identity covariance supplies the first\-order term, and[Equation131](https://arxiv.org/html/2609.10980#A1.E131)supplies the required fourth moment\. ∎

###### Corollary A\.3\(Network\-level variance\)\.

Let blockq=1,…,Qq=1,\\ldots,Qhave shapemq×nqm\_\{q\}\\times n\_\{q\}, rankrqr\_\{q\}, independent perturbation, and gradientGqG\_\{q\}\. WriteD=∑qmq​nqD=\\sum\_\{q\}m\_\{q\}n\_\{q\}and∥G∥2=∑q∥Gq∥F2\\lVert G\\rVert^\{2\}=\\sum\_\{q\}\\lVert G\_\{q\}\\rVert\_\{F\}^\{2\}\. The concatenated single\-direction estimator satisfies

𝔼⁡\[∥X−G∥2\]=\(D\+1\)​∥G∥2\+∑q=1Q2​\(mq\+nq\+1\)rq​∥Gq∥F2\.\\mathbb\{E\}\[\\lVert X\-G\\rVert^\{2\}\]=\(D\+1\)\\lVert G\\rVert^\{2\}\+\\sum\_\{q=1\}^\{Q\}\\frac\{2\(m\_\{q\}\+n\_\{q\}\+1\)\}\{r\_\{q\}\}\\lVert G\_\{q\}\\rVert\_\{F\}^\{2\}\.\(134\)

###### Proof\.

Concatenate the matrix blocks into one vector\. The Gaussian part of the fourth moment produces\(D\+1\)​∥G∥2\(D\+1\)\\lVert G\\rVert^\{2\}\. Independence makes the cross\-block fourth cumulants vanish, and[Theorem7\.1](https://arxiv.org/html/2609.10980#S7.Thmtheorem1)gives the remaining within\-block terms\. ∎

The additional variance from blockqqis weighted by∥Gq∥F2\\lVert G\_\{q\}\\rVert\_\{F\}^\{2\}\. Its contribution therefore depends on both the dimensions of the block and the amount of gradient energy concentrated there\. The same formula supplies a network\-level relative\-noise constant for the convergence results in Appendix[C](https://arxiv.org/html/2609.10980#A3)\. One valid choice is

C2=D\+1\+maxq⁡2​\(mq\+nq\+1\)rq\.C^\{2\}=D\+1\+\\max\_\{q\}\\frac\{2\(m\_\{q\}\+n\_\{q\}\+1\)\}\{r\_\{q\}\}\.\(135\)

## Appendix BSecondary corrections suggested by the analysis

The fixed\-radius expansion and variance calculation suggest two further modifications\. Although both are theoretically valid estimators, neither improved on ordinary EGGROLL in our model\-scale studies\. We record the results in this appendix rather than presenting them as viable practical contributions\.

### B\.1Rank extrapolation

The coefficient in[Theorem8\.1](https://arxiv.org/html/2609.10980#S8.Thmtheorem1)can be cancelled by combining two ranks\. Fors\>rs\>r, define

gr,s,σRE:=s​gs,σf−r​gr,σfs−r\.g\_\{r,s,\\sigma\}^\{\\mathrm\{RE\}\}:=\\frac\{sg\_\{s,\\sigma\}^\{f\}\-rg\_\{r,\\sigma\}^\{f\}\}\{s\-r\}\.\(136\)The following corollary quantifies the resulting bias for a chosen rank ratio\.

###### Corollary B\.1\(Rank extrapolation at a prescribed tolerance\)\.

Under the assumptions of[Theorem8\.1](https://arxiv.org/html/2609.10980#S8.Thmtheorem1), lets=α​rs=\\alpha rfor someα\>1\\alpha\>1such thatssis an integer\. For all sufficiently largerr,

∥gr,s,σRE−g∞,σf∥∞,p≤Rσ,pr2​α\+1α⁡\(α−1\)\.\\lVert g\_\{r,s,\\sigma\}^\{\\mathrm\{RE\}\}\-g\_\{\\infty,\\sigma\}^\{f\}\\rVert\_\{\\infty,p\}\\leq\\frac\{R\_\{\\sigma,p\}\}\{r^\{2\}\}\\frac\{\\alpha\+1\}\{\\alpha\(\\alpha\-1\)\}\.\(137\)Ifδ=ε​r2/Rσ,p\\delta=\\varepsilon r^\{2\}/R\_\{\\sigma,p\}, it is sufficient to choose

α≥δ\+1\+\(δ\+1\)2\+4​δ2​δ\.\\alpha\\geq\\frac\{\\delta\+1\+\\sqrt\{\(\\delta\+1\)^\{2\}\+4\\delta\}\}\{2\\delta\}\.\(138\)

###### Proof\.

Writegk,σf=g∞,σf\+ℬσ​f/k\+Rkg\_\{k,\\sigma\}^\{f\}=g\_\{\\infty,\\sigma\}^\{f\}\+\\mathcal\{B\}\_\{\\sigma\}f/k\+R\_\{k\}with∥Rk∥∞,p≤Rσ,p/k2\\lVert R\_\{k\}\\rVert\_\{\\infty,p\}\\leq R\_\{\\sigma,p\}/k^\{2\}\. Substitution into[Equation136](https://arxiv.org/html/2609.10980#A2.E136)cancelsℬσ​f\\mathcal\{B\}\_\{\\sigma\}fand gives[Equation137](https://arxiv.org/html/2609.10980#A2.E137)\. Solving\(α\+1\)/\(α⁡\(α−1\)\)≤δ\(\\alpha\+1\)/\(\\alpha\(\\alpha\-1\)\)\\leq\\deltagives[Equation138](https://arxiv.org/html/2609.10980#A2.E138)\. ∎

Cancellation of the mean\-field term comes at the cost of estimating two fields, and the extrapolation weights amplify their sampling errors\. If the two estimates are independent and useNrN\_\{r\}andNsN\_\{s\}directions with single\-direction variancesVrV\_\{r\}andVsV\_\{s\}, then

𝔼⁡\[∥g^r,s,σRE−gr,s,σRE∥F2\]=s2​Vs/Ns\+r2​Vr/Nr\(s−r\)2\.\\mathbb\{E\}\\left\[\\lVert\\widehat\{g\}\_\{r,s,\\sigma\}^\{\\mathrm\{RE\}\}\-g\_\{r,s,\\sigma\}^\{\\mathrm\{RE\}\}\\rVert\_\{F\}^\{2\}\\right\]=\\frac\{s^\{2\}V\_\{s\}/N\_\{s\}\+r^\{2\}V\_\{r\}/N\_\{r\}\}\{\(s\-r\)^\{2\}\}\.\(139\)At a fixed budget of 256 fitness evaluations, transformer\-block experiments compare ordinary rank\-one EGGROLL with extrapolation using ranks\(1,2\)\(1,2\)and\(1,4\)\(1,4\)\. Across two blocks and three radii, the extrapolated estimator has respectively7\.637\.63–8\.678\.67and2\.352\.35–2\.622\.62times the MSE of ordinary EGGROLL\. The leading bias is below the sampling resolution in these settings, so cancelling it does not repay the variance introduced by dividing the population between two ranks\.

### B\.2Variance reduction with a reference point

The dominant term in[Equation95](https://arxiv.org/html/2609.10980#S7.E95)is shared with dense Gaussian ES\. Following zeroth\-order variance reduction for language\-model fine\-tuning\[[18](https://arxiv.org/html/2609.10980#bib.bib15)\], letξ\\xidenote the sampled data and define

Yr,σ​\(W,ξ,E\):=E​ℓ⁡\(W\+σ​E,ξ\)−ℓ⁡\(W−σ​E,ξ\)2​σ,gr,σ​\(W\):=𝔼⁡\[Yr,σ​\(W,ξ,E\)\]\.Y\_\{r,\\sigma\}\(W;\\xi,E\):=E\\frac\{\\ell\(W\+\\sigma E;\\xi\)\-\\ell\(W\-\\sigma E;\\xi\)\}\{2\\sigma\},\\qquad g\_\{r,\\sigma\}\(W\):=\\mathbb\{E\}\\left\[Y\_\{r,\\sigma\}\(W;\\xi,E\)\\right\]\.\(140\)At a reference pointW~\\widetilde\{W\}, letg¯B​\(W~\)\\overline\{g\}\_\{B\}\(\\widetilde\{W\}\)be an independent unbiased average ofBBupdates and set

YVR​\(W,W~\):=Yr,σ​\(W,ξ,E\)−Yr,σ​\(W~,ξ,E\)\+g¯B​\(W~\)\.Y^\{\\mathrm\{VR\}\}\(W,\\widetilde\{W\}\):=Y\_\{r,\\sigma\}\(W;\\xi,E\)\-Y\_\{r,\\sigma\}\(\\widetilde\{W\};\\xi,E\)\+\\overline\{g\}\_\{B\}\(\\widetilde\{W\}\)\.\(141\)The following proposition gives the standard control\-variate guarantee for Gaussian\-product directions\.

###### Proposition B\.2\(Low\-rank variance reduction\)\.

The estimator in[Equation141](https://arxiv.org/html/2609.10980#A2.E141)is unbiased forgr,σ​\(W\)g\_\{r,\\sigma\}\(W\)\. If

𝔼⁡\[∥Yr,σ​\(W,ξ,E\)−Yr,σ​\(V,ξ,E\)∥F2\]≤LY2​∥W−V∥F2\\mathbb\{E\}\\left\[\\lVert Y\_\{r,\\sigma\}\(W;\\xi,E\)\-Y\_\{r,\\sigma\}\(V;\\xi,E\)\\rVert\_\{F\}^\{2\}\\right\]\\leq L\_\{Y\}^\{2\}\\lVert W\-V\\rVert\_\{F\}^\{2\}\(142\)and𝔼⁡\[∥g¯B​\(W~\)−gr,σ​\(W~\)∥F2\]≤Vref/B\\mathbb\{E\}\[\\lVert\\overline\{g\}\_\{B\}\(\\widetilde\{W\}\)\-g\_\{r,\\sigma\}\(\\widetilde\{W\}\)\\rVert\_\{F\}^\{2\}\]\\leq V\_\{\\mathrm\{ref\}\}/B, then

𝔼⁡\[∥YVR​\(W,W~\)−gr,σ​\(W\)∥F2\]≤LY2​∥W−W~∥F2\+VrefB\.\\mathbb\{E\}\\left\[\\lVert Y^\{\\mathrm\{VR\}\}\(W,\\widetilde\{W\}\)\-g\_\{r,\\sigma\}\(W\)\\rVert\_\{F\}^\{2\}\\right\]\\leq L\_\{Y\}^\{2\}\\lVert W\-\\widetilde\{W\}\\rVert\_\{F\}^\{2\}\+\\frac\{V\_\{\\mathrm\{ref\}\}\}\{B\}\.\(143\)

###### Proof\.

Taking expectations in[Equation141](https://arxiv.org/html/2609.10980#A2.E141)cancels the field at the reference point and leavesgr,σ​\(W\)g\_\{r,\\sigma\}\(W\)\. Independence of the reference estimate makes the centered errors orthogonal, and the two assumed bounds give[Equation143](https://arxiv.org/html/2609.10980#A2.E143)\. ∎

We tested this estimator on Qwen3\-0\.6B and Qwen3\-1\.7B with five paired seeds using next\-token prediction loss and GSM8K rewards\.[Table6](https://arxiv.org/html/2609.10980#A2.T6)reports held\-out reward relative to matched EGGROLL checkpoints\. All four means favor ordinary EGGROLL, with the 1\.7B next\-token prediction loss difference excluding zero\. These results do not establish a practical benefit from maintaining the reference estimate and its additional evaluations\.

Table 6:SVRG improvement over EGGROLL over five paired seeds\. For next\-token prediction, entries are the reduction in held\-out loss, computed as EGGROLL loss minus SVRG loss\. For GSM8K, entries are SVRG reward minus EGGROLL reward in percentage points\. Positive values favor SVRG and negative values favor EGGROLL\. Brackets give paired 95% Student\-ttintervals\.ModelObjectiveDifferenceQwen3\-0\.6BNext\-token prediction loss−\.0161​\[−\.0488,\.0166\]\-\.0161\\;\[\-\.0488,\.0166\]Qwen3\-0\.6BGSM8K−6\.17%​\[−17\.43,5\.08\]\-6\.17\\%\\;\[\-17\.43,5\.08\]Qwen3\-1\.7BNext\-token prediction loss−\.0197​\[−\.0321,−\.0072\]\\bm\{\-\.0197\\;\[\-\.0321,\-\.0072\]\}Qwen3\-1\.7BGSM8K−1\.25%​\[−2\.98,\.48\]\-1\.25\\%\\;\[\-2\.98,\.48\]

## Appendix CConvergence consequences of the field bounds

The main text establishes the bias of the population field and the exact finite\-population variance pertaining to EGGROLL\. The results below translate those bounds into standard mean\-field and finite\-population stationarity guarantees under smoothness\.

###### Lemma C\.1\(Descent with accumulated field error\)\.

SupposehhisLL\-smooth and bounded below byh⋆h\_\{\\star\}\. Letg=∇h\+bg=\\nabla h\+bandxt\+1=xt−η​g​\(xt\)x\_\{t\+1\}=x\_\{t\}\-\\eta g\(x\_\{t\}\)\. If0<η≤1/L0<\\eta\\leq 1/L, then

1T​∑t=0T−1∥∇h​\(xt\)∥2≤2​\(h⁡\(x0\)−h⋆\)η​T\+1T​∑t=0T−1∥b⁡\(xt\)∥2\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\lVert\\nabla h\(x\_\{t\}\)\\rVert^\{2\}\\leq\\frac\{2\(h\(x\_\{0\}\)\-h\_\{\\star\}\)\}\{\\eta T\}\+\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\lVert b\(x\_\{t\}\)\\rVert^\{2\}\.\(144\)

###### Proof\.

Writeat=∇h​\(xt\)a\_\{t\}=\\nabla h\(x\_\{t\}\)andbt=b⁡\(xt\)b\_\{t\}=b\(x\_\{t\}\)\. Smoothness and the identity−⟨at,at\+bt⟩=−∥at∥2/2−∥at\+bt∥2/2\+∥bt∥2/2\-\\langle a\_\{t\},\\,a\_\{t\}\+b\_\{t\}\\rangle=\-\\lVert a\_\{t\}\\rVert^\{2\}/2\-\\lVert a\_\{t\}\+b\_\{t\}\\rVert^\{2\}/2\+\\lVert b\_\{t\}\\rVert^\{2\}/2give

h⁡\(xt\+1\)≤h⁡\(xt\)−η2​∥at∥2−η2​\(1−L​η\)​∥at\+bt∥2\+η2​∥bt∥2\.h\(x\_\{t\+1\}\)\\leq h\(x\_\{t\}\)\-\\frac\{\\eta\}\{2\}\\lVert a\_\{t\}\\rVert^\{2\}\-\\frac\{\\eta\}\{2\}\(1\-L\\eta\)\\lVert a\_\{t\}\+b\_\{t\}\\rVert^\{2\}\+\\frac\{\\eta\}\{2\}\\lVert b\_\{t\}\\rVert^\{2\}\.\(145\)Summation proves the claim\. ∎

Define

Br,σ\(L\):=L​σ2​μ4​\(r\)3/4,Br,σ\(M\):=M​σ26​μ4​\(r\)\.B\_\{r,\\sigma\}^\{\(L\)\}:=\\frac\{L\\sigma\}\{2\}\\mu\_\{4\}\(r\)^\{3/4\},\\qquad B\_\{r,\\sigma\}^\{\(M\)\}:=\\frac\{M\\sigma^\{2\}\}\{6\}\\mu\_\{4\}\(r\)\.\(146\)Combining[LemmaC\.1](https://arxiv.org/html/2609.10980#A3.Thmtheorem1)with[Lemma6\.3](https://arxiv.org/html/2609.10980#S6.Thmtheorem3)gives the following direct consequence\.

###### Theorem C\.2\(Mean\-field convergence under standard smoothness\)\.

SupposehhisLL\-smooth and bounded below byh⋆h\_\{\\star\}\. Fixε\>0\\varepsilon\>0, useη=1/L\\eta=1/L, and setH0=h⁡\(x0\)−h⋆H\_\{0\}=h\(x\_\{0\}\)\-h\_\{\\star\}\. Either of the sufficient choices

0<σ≤σL​\(r,ε\):=2​εL​μ4​\(r\)3/4,0<\\sigma\\leq\\sigma\_\{L\}\(r,\\varepsilon\):=\\frac\{\\sqrt\{2\}\\,\\varepsilon\}\{L\\mu\_\{4\}\(r\)^\{3/4\}\},\(147\)or, when the Hessian isMM\-Lipschitz,

0<σ≤σM​\(r,ε\):=\(3​2​εM​μ4​\(r\)\)1/20<\\sigma\\leq\\sigma\_\{M\}\(r,\\varepsilon\):=\\left\(\\frac\{3\\sqrt\{2\}\\,\\varepsilon\}\{M\\mu\_\{4\}\(r\)\}\\right\)^\{1/2\}\(148\)ensuresT−1​∑t=0T−1∥∇h​\(xt\)∥2≤ε2T^\{\-1\}\\sum\_\{t=0\}^\{T\-1\}\\lVert\\nabla h\(x\_\{t\}\)\\rVert^\{2\}\\leq\\varepsilon^\{2\}forT≥⌈4​L​H0/ε2⌉T\\geq\\lceil 4LH\_\{0\}/\\varepsilon^\{2\}\\rceil\.

###### Proof\.

The applicable radius makes the corresponding field\-error bound at mostε/2\\varepsilon/\\sqrt\{2\}\. Substitution into[Equation144](https://arxiv.org/html/2609.10980#A3.E144)completes the proof\. ∎

### C\.1Finite\-population convergence

An actual EGGROLL update differs from a gradient step for two reasons\. The finite population fluctuates around the population field, and the population field can itself differ from the gradient chosen as the reference\. We have already bounded both quantities\.[Section7](https://arxiv.org/html/2609.10980#S7)gives the sampling variance,[Section6](https://arxiv.org/html/2609.10980#S6)compares the mean field with the original loss, while[Section8](https://arxiv.org/html/2609.10980#S8)compares it with dense Gaussian ES at the same radius\. We now combine these bounds into finite\-iteration guarantees\.

As in[Section6\.1](https://arxiv.org/html/2609.10980#S6.SS1), we minimize a losshhthroughxt\+1=xt−η​g^tx\_\{t\+1\}=x\_\{t\}\-\\eta\\widehat\{g\}\_\{t\}\. The variance assumption contains a term proportional to the squared mean update, as predicted by[Theorems7\.1](https://arxiv.org/html/2609.10980#S7.Thmtheorem1)and[A\.3](https://arxiv.org/html/2609.10980#A1.Thmtheorem3), and an additive term for evaluation noise that may remain even when the mean update vanishes\. The following theorem separates optimization progress, mean\-field error, and finite\-population noise\. We keep the mean\-field term explicit so that either comparison from the preceding sections can be substituted without hiding its dependence onrrandσ\\sigma\.

###### Theorem C\.3\(Nonconvex convergence under relative population noise\)\.

SupposehhisLL\-smooth and bounded below byh⋆h\_\{\\star\}\. Let\(ℱt\)t≥0\(\\mathcal\{F\}\_\{t\}\)\_\{t\\geq 0\}be the filtration generated byx0x\_\{0\}and all perturbation and evaluation randomness revealed before iterationtt, so thatxtx\_\{t\}isℱt\\mathcal\{F\}\_\{t\}\-measurable\. At every iterate, assume

𝔼⁡\[g^t∣ℱt\]\\displaystyle\\mathbb\{E\}\[\\widehat\{g\}\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\]=g⁡\(xt\),\\displaystyle=g\(x\_\{t\}\),\(149\)𝔼⁡\[∥g^t−g⁡\(xt\)∥2∣ℱt\]\\displaystyle\\mathbb\{E\}\[\\lVert\\widehat\{g\}\_\{t\}\-g\(x\_\{t\}\)\\rVert^\{2\}\\mid\\mathcal\{F\}\_\{t\}\]≤G2\+C2​∥g⁡\(xt\)∥2N\\displaystyle\\leq\\frac\{G^\{2\}\+C^\{2\}\\lVert g\(x\_\{t\}\)\\rVert^\{2\}\}\{N\}\(150\)forC,G≥0C,G\\geq 0\. If0<η≤\[L⁡\(1\+C2/N\)\]−10<\\eta\\leq\[L\(1\+C^\{2\}/N\)\]^\{\-1\}, then

1T​∑t=0T−1𝔼⁡\[∥∇h​\(xt\)∥2\]≤2​\(h⁡\(x0\)−h⋆\)η​T\+1T​∑t=0T−1𝔼⁡\[∥g⁡\(xt\)−∇h​\(xt\)∥2\]\+L​η​G2N\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\[\\lVert\\nabla h\(x\_\{t\}\)\\rVert^\{2\}\]\\leq\\frac\{2\(h\(x\_\{0\}\)\-h\_\{\\star\}\)\}\{\\eta T\}\+\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\[\\lVert g\(x\_\{t\}\)\-\\nabla h\(x\_\{t\}\)\\rVert^\{2\}\]\+\\frac\{L\\eta G^\{2\}\}\{N\}\.\(151\)

###### Proof\.

Smoothness and conditional expectation give

𝔼t​\[h⁡\(xt\+1\)\]≤h⁡\(xt\)−η⁡⟨∇h​\(xt\),g⁡\(xt\)⟩\+L​η22​\[\(1\+C2N\)​∥g⁡\(xt\)∥2\+G2N\]\.\\mathbb\{E\}\_\{t\}\[h\(x\_\{t\+1\}\)\]\\leq h\(x\_\{t\}\)\-\\eta\\langle\\nabla h\(x\_\{t\}\),\\,g\(x\_\{t\}\)\\rangle\+\\frac\{L\\eta^\{2\}\}\{2\}\\left\[\\left\(1\+\\frac\{C^\{2\}\}\{N\}\\right\)\\lVert g\(x\_\{t\}\)\\rVert^\{2\}\+\\frac\{G^\{2\}\}\{N\}\\right\]\.\(152\)Writeg=∇h\+bg=\\nabla h\+band use

−⟨∇h,g⟩=−12​∥∇h∥2−12​∥g∥2\+12​∥b∥2\.\-\\langle\\nabla h,\\,g\\rangle=\-\\frac\{1\}\{2\}\\lVert\\nabla h\\rVert^\{2\}\-\\frac\{1\}\{2\}\\lVert g\\rVert^\{2\}\+\\frac\{1\}\{2\}\\lVert b\\rVert^\{2\}\.\(153\)The step\-size condition makes the total coefficient of∥g∥2\\lVert g\\rVert^\{2\}nonpositive\. Taking full expectation and summing overttgives[Equation151](https://arxiv.org/html/2609.10980#A3.E151)\. ∎

We first measure stationarity with respect to the original losshh\. In this comparison, the field error includes both smoothing and finite\-rank score mismatch, and the standard smoothness bounds from[Section6](https://arxiv.org/html/2609.10980#S6)control them together\.

###### Corollary C\.4\(Convergence relative to the original loss\)\.

Suppose thathhisLL\-smooth and bounded below byh⋆h\_\{\\star\}, that the finite\-population estimator satisfies[Equations149](https://arxiv.org/html/2609.10980#A3.E149)and[150](https://arxiv.org/html/2609.10980#A3.E150)withg=gr,σhg=g\_\{r,\\sigma\}^\{h\}, and set0<η≤\[L⁡\(1\+C2/N\)\]−10<\\eta\\leq\[L\(1\+C^\{2\}/N\)\]^\{\-1\}\. DefineBr,σstd=Br,σ\(L\)B\_\{r,\\sigma\}^\{\\mathrm\{std\}\}=B\_\{r,\\sigma\}^\{\(L\)\}\. If the Hessian isMM\-Lipschitz, one may instead takeBr,σstd=Br,σ\(M\)B\_\{r,\\sigma\}^\{\\mathrm\{std\}\}=B\_\{r,\\sigma\}^\{\(M\)\}\. Then

1T​∑t=0T−1𝔼⁡\[∥∇h​\(xt\)∥2\]≤2​\(h⁡\(x0\)−h⋆\)η​T\+\(Br,σstd\)2\+L​η​G2N\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\[\\lVert\\nabla h\(x\_\{t\}\)\\rVert^\{2\}\]\\leq\\frac\{2\(h\(x\_\{0\}\)\-h\_\{\\star\}\)\}\{\\eta T\}\+\\left\(B\_\{r,\\sigma\}^\{\\mathrm\{std\}\}\\right\)^\{2\}\+\\frac\{L\\eta G^\{2\}\}\{N\}\.\(154\)Ifσ≤σL​\(r,ε\)\\sigma\\leq\\sigma\_\{L\}\(r,\\varepsilon\), orσ≤σM​\(r,ε\)\\sigma\\leq\\sigma\_\{M\}\(r,\\varepsilon\)under the Lipschitz\-Hessian assumption, then the middle term is at mostε2/2\\varepsilon^\{2\}/2\.

###### Proof\.

Apply[Lemma6\.3](https://arxiv.org/html/2609.10980#S6.Thmtheorem3)to the field\-error term in[Equation151](https://arxiv.org/html/2609.10980#A3.E151)\. ∎

To turn the bound into an iteration complexity, we divide the target tolerance among the mean\-field error, the finite optimization horizon, and irreducible population noise\. The following corollary gives one explicit allocation for a single matrix\. Replacingμ4​\(r\)\\mu\_\{4\}\(r\)byμ4​\(𝒓\)\\mu\_\{4\}\(\\bm\{r\}\)from[Equation131](https://arxiv.org/html/2609.10980#A1.E131)gives the corresponding network statement\.

###### Corollary C\.5\(Explicit finite\-population complexity\)\.

Under the assumptions of[CorollaryC\.4](https://arxiv.org/html/2609.10980#A3.Thmtheorem4), fixε\>0\\varepsilon\>0and letΔ=h⁡\(x0\)−h⋆\\Delta=h\(x\_\{0\}\)\-h\_\{\\star\}\. Setη=1/\(2​L\)\\eta=1/\(2L\)and choose

0<σ≤2​ε3​L​μ4​\(r\)3/4\.0<\\sigma\\leq\\frac\{2\\varepsilon\}\{\\sqrt\{3\}\\,L\\mu\_\{4\}\(r\)^\{3/4\}\}\.\(155\)If the Hessian isMM\-Lipschitz, the alternative choice

0<σ≤\(2​3​εM​μ4​\(r\)\)1/20<\\sigma\\leq\\left\(\\frac\{2\\sqrt\{3\}\\,\\varepsilon\}\{M\\mu\_\{4\}\(r\)\}\\right\)^\{1/2\}\(156\)is sufficient\. If

N≥max⁡\{C2,3​G22​ε2\}N\\geq\\max\\left\\\{C^\{2\},\\frac\{3G^\{2\}\}\{2\\varepsilon^\{2\}\}\\right\\\}\(157\)and

T≥⌈12​L​Δε2⌉,T\\geq\\left\\lceil\\frac\{12L\\Delta\}\{\\varepsilon^\{2\}\}\\right\\rceil,\(158\)then

1T​∑t=0T−1𝔼⁡\[∥∇h​\(xt\)∥2\]≤ε2\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[\\lVert\\nabla h\(x\_\{t\}\)\\rVert^\{2\}\\right\]\\leq\\varepsilon^\{2\}\.\(159\)

###### Proof\.

The radius choices make the field\-error bound at mostε/3\\varepsilon/\\sqrt\{3\}\. The requirementN≥C2N\\geq C^\{2\}givesη≤\[L⁡\(1\+C2/N\)\]−1\\eta\\leq\[L\(1\+C^\{2\}/N\)\]^\{\-1\}, while the second part of[Equation157](https://arxiv.org/html/2609.10980#A3.E157)bounds the irreducible term byG2/\(2​N\)≤ε2/3G^\{2\}/\(2N\)\\leq\\varepsilon^\{2\}/3\. Finally,[Equation158](https://arxiv.org/html/2609.10980#A3.E158)bounds the optimization term by4​L​Δ/T≤ε2/34L\\Delta/T\\leq\\varepsilon^\{2\}/3\. Substitution into[Equation154](https://arxiv.org/html/2609.10980#A3.E154)proves the claim\. ∎

The preceding guarantees concern the original loss\. A different question is whether finite\-rank EGGROLL follows dense Gaussian ES at the same radius\. Dense Gaussian ES follows the gradient ofH∞,σ​\(x\):=𝔼⁡\[h⁡\(x\+σ​Z\)\]H\_\{\\infty,\\sigma\}\(x\):=\\mathbb\{E\}\[h\(x\+\\sigma Z\)\], whereZ∼𝒩⁡\(0,I\)Z\\sim\\mathcal\{N\}\(0,I\)\. Taking this smoothed loss as the reference removes the smoothing error shared by both methods and leaves only the finite\-rank discrepancy derived in[Section8](https://arxiv.org/html/2609.10980#S8)\.

###### Corollary C\.6\(Convergence relative to dense Gaussian ES\)\.

Supposehhsatisfies the assumptions of[Theorem8\.1](https://arxiv.org/html/2609.10980#S8.Thmtheorem1),H∞,σH\_\{\\infty,\\sigma\}isLσL\_\{\\sigma\}\-smooth and bounded below byH∞,σ⋆H\_\{\\infty,\\sigma\}^\{\\star\}, and the finite\-population estimator satisfies[Equations149](https://arxiv.org/html/2609.10980#A3.E149)and[150](https://arxiv.org/html/2609.10980#A3.E150)withg=gr,σhg=g\_\{r,\\sigma\}^\{h\}\. For the constantsKσ,p,Rσ,p,r0K\_\{\\sigma,p\},R\_\{\\sigma,p\},r\_\{0\}in[Equation119](https://arxiv.org/html/2609.10980#S8.E119), everyr≥r0r\\geq r\_\{0\}and0<η≤\[Lσ​\(1\+C2/N\)\]−10<\\eta\\leq\[L\_\{\\sigma\}\(1\+C^\{2\}/N\)\]^\{\-1\}satisfy

1T​∑t=0T−1𝔼⁡\[∥∇H∞,σ​\(xt\)∥2\]≤2​\(H∞,σ​\(x0\)−H∞,σ⋆\)η​T\+\(Kσ,pr\+Rσ,pr2\)2​1T​∑t=0T−1𝔼⁡\[\(1\+∥xt∥\)2​p\]\+Lσ​η​G2N\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[\\lVert\\nabla H\_\{\\infty,\\sigma\}\(x\_\{t\}\)\\rVert^\{2\}\\right\]\\leq\\frac\{2\(H\_\{\\infty,\\sigma\}\(x\_\{0\}\)\-H\_\{\\infty,\\sigma\}^\{\\star\}\)\}\{\\eta T\}\+\\left\(\\frac\{K\_\{\\sigma,p\}\}\{r\}\+\\frac\{R\_\{\\sigma,p\}\}\{r^\{2\}\}\\right\)^\{2\}\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[\(1\+\\lVert x\_\{t\}\\rVert\)^\{2p\}\\right\]\+\\frac\{L\_\{\\sigma\}\\eta G^\{2\}\}\{N\}\.\(160\)If the displayed parameter moments are uniformly bounded along the trajectory, the mean\-field contribution of finite rank to the squared\-stationarity guarantee isO⁡\(r−2\)O\(r^\{\-2\}\)whenever the leading coefficient is nonzero\. For bounded fitness,p=0p=0and no trajectory\-moment condition is needed\.

###### Proof\.

Equation \([119](https://arxiv.org/html/2609.10980#S8.E119)\) gives

∥gr,σh​\(xt\)−∇H∞,σ​\(xt\)∥≤\(Kσ,pr\+Rσ,pr2\)​\(1\+∥xt∥\)p\.\\lVert g\_\{r,\\sigma\}^\{h\}\(x\_\{t\}\)\-\\nabla H\_\{\\infty,\\sigma\}\(x\_\{t\}\)\\rVert\\leq\\left\(\\frac\{K\_\{\\sigma,p\}\}\{r\}\+\\frac\{R\_\{\\sigma,p\}\}\{r^\{2\}\}\\right\)\(1\+\\lVert x\_\{t\}\\rVert\)^\{p\}\.\(161\)Square this bound and apply[TheoremC\.3](https://arxiv.org/html/2609.10980#A3.Thmtheorem3)withH∞,σH\_\{\\infty,\\sigma\}as the reference loss\. ∎

The two reference objectives should not be conflated\. Relative tohh, the error contains smoothing and finite\-rank effects and must be controlled through bothσ\\sigmaandrr\. Relative to dense Gaussian ES at the sameσ\\sigma, the shared smoothing disappears and the mean\-field difference isO⁡\(1/r\)O\(1/r\)\. The finite\-population noise is the same in either comparison\.

The constants in the variance assumption can be connected to quantities that are calculable or measurable for EGGROLL\. The constantG2G^\{2\}allows noise to remain when the mean update vanishes\. The termC2​∥g⁡\(x\)∥2C^\{2\}\\lVert g\(x\)\\rVert^\{2\}is relative or multiplicative noise, also called a strong\-growth component\[[43](https://arxiv.org/html/2609.10980#bib.bib32),[45](https://arxiv.org/html/2609.10980#bib.bib33)\]\. For one matrix in the exact affine model,[Theorem7\.1](https://arxiv.org/html/2609.10980#S7.Thmtheorem1)givesC2=m​n\+1\+2​\(m\+n\+1\)/rC^\{2\}=mn\+1\+2\(m\+n\+1\)/r\. Appendix[A](https://arxiv.org/html/2609.10980#A1)gives the corresponding network expression\. Relative noise restricts the stable step size, but it vanishes with the mean field and creates no additional error floor\. The additive componentG2G^\{2\}produces the final term in[Equation151](https://arxiv.org/html/2609.10980#A3.E151)\. SettingC=0C=0recovers the usual bounded\-variance assumption, while the exact affine EGGROLL model hasG=0G=0\.

## Appendix DFourth moments of the Gaussian\-product law

This appendix supplies the fourth\-moment calculation used in[Proposition6\.2](https://arxiv.org/html/2609.10980#S6.Thmtheorem2)\. The coefficient in[Equation80](https://arxiv.org/html/2609.10980#S6.E80)follows from the fourth moment of four entries ofErE\_\{r\}\. At rank one,\(E1\)i​j=ai​bj\(E\_\{1\}\)\_\{ij\}=a\_\{i\}b\_\{j\}, whereaaandbbare independent standard Gaussian vectors\. For arbitrary row indicesi1,…,i4i\_\{1\},\\ldots,i\_\{4\}and column indicesj1,…,j4j\_\{1\},\\ldots,j\_\{4\},

𝔼⁡\[∏s=14\(E1\)is​js\]=\(δi1​i2​δi3​i4\+δi1​i3​δi2​i4\+δi1​i4​δi2​i3\)​\(δj1​j2​δj3​j4\+δj1​j3​δj2​j4\+δj1​j4​δj2​j3\)\.\\mathbb\{E\}\\left\[\\prod\_\{s=1\}^\{4\}\(E\_\{1\}\)\_\{i\_\{s\}j\_\{s\}\}\\right\]=\\left\(\\delta\_\{i\_\{1\}i\_\{2\}\}\\delta\_\{i\_\{3\}i\_\{4\}\}\+\\delta\_\{i\_\{1\}i\_\{3\}\}\\delta\_\{i\_\{2\}i\_\{4\}\}\+\\delta\_\{i\_\{1\}i\_\{4\}\}\\delta\_\{i\_\{2\}i\_\{3\}\}\\right\)\\left\(\\delta\_\{j\_\{1\}j\_\{2\}\}\\delta\_\{j\_\{3\}j\_\{4\}\}\+\\delta\_\{j\_\{1\}j\_\{3\}\}\\delta\_\{j\_\{2\}j\_\{4\}\}\+\\delta\_\{j\_\{1\}j\_\{4\}\}\\delta\_\{j\_\{2\}j\_\{3\}\}\\right\)\.\(162\)Each parenthesis contains the three possible pairings of four Gaussian factors\. Expanding their product gives nine terms\. In three of them, the row and column indices use the same pairing\. For example, pairing the first entry with the second and the third with the fourth in both factors gives

δi1​i2​δj1​j2​δi3​i4​δj3​j4\.\\delta\_\{i\_\{1\}i\_\{2\}\}\\delta\_\{j\_\{1\}j\_\{2\}\}\\delta\_\{i\_\{3\}i\_\{4\}\}\\delta\_\{j\_\{3\}j\_\{4\}\}\.\(163\)These three terms are precisely the fourth moment of a dense Gaussian matrix with independent entries\. The remaining six terms use different pairings for the row and column indices, such as

δi1​i2​δi3​i4​δj1​j3​δj2​j4,\\delta\_\{i\_\{1\}i\_\{2\}\}\\delta\_\{i\_\{3\}i\_\{4\}\}\\delta\_\{j\_\{1\}j\_\{3\}\}\\delta\_\{j\_\{2\}j\_\{4\}\},\(164\)and arise from the rank\-one factorization\.

For general rank, writeEr=r−1/2∑q=1raqbq⊤E\_\{r\}=r^\{\-1/2\}\\sum\_\{q=1\}^\{r\}a\_\{q\}b\_\{q\}^\{\\top\}\. A nonzero fourth\-moment term either draws all four factors from one summand or draws two factors from each of two summands\. The three dense\-Gaussian terms receive both kinds of contributions, with total coefficient

rr2\+r⁡\(r−1\)r2=1\.\\frac\{r\}\{r^\{2\}\}\+\\frac\{r\(r\-1\)\}\{r^\{2\}\}=1\.\(165\)Each of the six additional terms requires all four factors to come from the same rank\-one summand and therefore has coefficientr/r2=1/rr/r^\{2\}=1/r\. Consequently, the fourth moment ofErE\_\{r\}consists of the three dense\-Gaussian terms plus the six additional terms scaled by1/r1/r\. To apply this identity in[Equation79](https://arxiv.org/html/2609.10980#S6.E79), write the\(i0,j0\)\(i\_\{0\},j\_\{0\}\)entry as

∑i1,j1,i2,j2,i3,j3∂i1​j1∂i2​j2∂i3​j3f⁡\(W\)​𝔼​\[\(Er\)i0​j0​\(Er\)i1​j1​\(Er\)i2​j2​\(Er\)i3​j3\]\.\\sum\_\{i\_\{1\},j\_\{1\},i\_\{2\},j\_\{2\},i\_\{3\},j\_\{3\}\}\\partial\_\{i\_\{1\}j\_\{1\}\}\\partial\_\{i\_\{2\}j\_\{2\}\}\\partial\_\{i\_\{3\}j\_\{3\}\}f\(W\)\\,\\mathbb\{E\}\\left\[\(E\_\{r\}\)\_\{i\_\{0\}j\_\{0\}\}\(E\_\{r\}\)\_\{i\_\{1\}j\_\{1\}\}\(E\_\{r\}\)\_\{i\_\{2\}j\_\{2\}\}\(E\_\{r\}\)\_\{i\_\{3\}j\_\{3\}\}\\right\]\.\(166\)The three dense\-Gaussian terms give the same quantity,∂i0​j0Δ​f​\(W\)\\partial\_\{i\_\{0\}j\_\{0\}\}\\Delta f\(W\), after relabeling indices\. The six additional terms give the same quantity,\(𝒟​f​\(W\)\)i0​j0\(\\mathcal\{D\}f\(W\)\)\_\{i\_\{0\}j\_\{0\}\}, by symmetry of the third derivative\. Summing them proves

𝔼\[ErD3f\(W\)\[Er,Er,Er\]\]=3∇Δf\(W\)\+6r𝒟f\(W\)\.\\mathbb\{E\}\\left\[E\_\{r\}D^\{3\}f\(W\)\[E\_\{r\},E\_\{r\},E\_\{r\}\]\\right\]=3\\nabla\\Delta f\(W\)\+\\frac\{6\}\{r\}\\mathcal\{D\}f\(W\)\.\(167\)

## Appendix EComplete empirical aggregates

### E\.1LOO\-ROLL comparison protocol and simultaneous inference

##### Training budgets and starting models\.

Each run uses one NVIDIA H100 80GB GPU and 14 CPU cores\. Both methods start from the same pretrained model within each pair, using the Qwen3 and SmolLM2 checkpoints listed in[Table7](https://arxiv.org/html/2609.10980#A5.T7)\. The table reports the EGGROLL budget against which LOO\-ROLL is compared\. Equal\-iteration and equal\-evaluation runs use the EGGROLL update count\. Equal\-time runs continue LOO\-ROLL to the paired EGGROLL runtime\. Timing covers the training process, including model loading, updates, and checkpoint writing\. For resumed runs, elapsed times are accumulated across the constituent training processes\. The separate final test evaluation is excluded\. Checkpoints are selected by update count or the stopping budget, without choosing the best test score\. The full GSM8K evaluations reuse these selected checkpoints\.

Table 7:Training budgets for the ten LOO\-ROLL comparisons\.NNcounts independent EGGROLL directions andBBis the number of prompts or sequences per update\. Per update, EGGROLL uses2​N2Nevaluations\. LOO\-ROLL usesNNevaluations for equal iterations and equal time, or2​N2Nfor equal evaluations\.TET\_\{E\}is the EGGROLL update count andTLT\_\{L\}is the mean number of LOO\-ROLL updates completed at equal time\. Hours are mean EGGROLL runtimes over five seeds\. The last column averages the paired LOO\-ROLL/EGGROLL time ratios\. NTP denotes next\-token prediction\.ModelTaskNNBBTET\_\{E\}TLT\_\{L\}HoursTime ratioQwen3\-0\.6BCountdown1288150374\.42\.6680\.9970Qwen3\-0\.6BGSM8K1284150368\.44\.1450\.9978Qwen3\-0\.6BNTP32480131\.80\.1511\.0088Qwen3\-1\.7BCountdown1288200496\.04\.1870\.9980Qwen3\-1\.7BGSM8K1284200485\.66\.1900\.9985Qwen3\-1\.7BNTP324160268\.00\.2931\.0199Qwen3\-8BGSM8K1284120303\.68\.9291\.0009Qwen3\-8BNTP324120203\.40\.3631\.0271SmolLM2\-1\.7BCountdown1288200485\.04\.5210\.9981SmolLM2\-1\.7BGSM8K1284200444\.83\.2500\.9981
##### Data and paired seeds\.

GSM8K training uses the training split ofopenai/gsm8k, with one deterministic response per perturbation and prompt\. Next\-token prediction useswikitext, configurationwikitext\-2\-raw\-v1, with separate training and validation splits\. We take the first 4,096 rows at 0\.6B and 8B or 6,144 rows at 1\.7B, shuffle them using the training seed, tokenize each row to at most 128 tokens, and retain sequences with at least eight tokens\. Training cycles through batches of four sequences\. Reward\-based Qwen3\-0\.6B comparisons use seeds0,1,2,404,5050,1,2,404,505, while Qwen3\-1\.7B and 8B use101,202,303,404,505101,202,303,404,505\. SmolLM2 uses27109,27211,27329,27409,2750927109,27211,27329,27409,27509\. Next\-token prediction uses16111,16229,16333,16447,1656116111,16229,16333,16447,16561at 0\.6B and9103,9209,9311,9413,95119103,9209,9311,9413,9511at 1\.7B and 8B\.

##### Implementation\.

The implementation extends the[EGGROLL vLLM repository](https://github.com/ESHyperscale/eggroll-vllm), starting from commitbcc215e8784f5f44d24985145c0a71e74283cf1f\. Our implementation replaces the antithetic adapter construction and update with one\-sided directions and leave\-one\-out scores\.

The equal\-time comparison uses ten model–task settings with five paired training seeds each\. The configured learning rate is2×10−42\\times 10^\{\-4\}, except for Qwen3\-8B with next\-token prediction loss, which uses4×10−44\\times 10^\{\-4\}for both methods\. The reward\-based comparisons use rank one,σ=10−3\\sigma=10^\{\-3\}, 128 independent antithetic directions for EGGROLL, and 128 one\-sided directions for LOO\-ROLL\. The next\-token prediction experiments use 32 independent directions for each method andσ=10−2\\sigma=10^\{\-2\}at 0\.6B and 1\.7B or3×10−33\\times 10^\{\-3\}at 8B\. Equal\-evaluation comparisons double the LOO\-ROLL direction count\. Score standardization is applied for both methods\. These comparisons retain the corresponding EGGROLL learning rate rather than separately tuning each estimator\.

All GSM8K results in the main tables evaluate the saved checkpoints on the full set of 1,319 test questions, without selecting new checkpoints or changing hyperparameters\. The ten\-setting tests below use the full\-test GSM8K differences and the original Countdown and next\-token prediction loss differences\. The SmolLM2\-1\.7B comparisons use five paired seeds, the model’s chat template, and a 1,024\-token generation limit\. For SmolLM2 GSM8K, EGGROLL trains for 200 updates with 128 independent antithetic directions, or 256 signed population members\. The implementation’s population\-size parameter counts signed members for EGGROLL and independent directions for LOO\-ROLL\. LOO\-ROLL uses 128 directions for the equal\-iteration and equal\-wall\-time comparisons and 256 directions for the equal\-evaluation comparison\. The mean LOO\-ROLL\-to\-EGGROLL time ratio in the equal\-wall comparison is0\.99810\.9981, with a maximum per\-pair mismatch of0\.34%0\.34\\%\.

GSM8K decoding is deterministic, with a generation cap of 768 tokens at 0\.6B and 1024 tokens at 1\.7B and 8B\. Countdown uses a cap of 256 tokens\. Validation uses the same held\-out examples and decoding settings for paired methods\. The equal\-time budget is determined from the completed EGGROLL run\. The mean time ratio in the original nine settings is1\.0051\.005, and the largest absolute per\-pair mismatch is5\.94%5\.94\\%\. No interpolation of validation scores is used to enforce an exact time match\.

For inference, we form the five paired differences within each setting and test a zero mean using a two\-sided Student\-ttstatistic with four degrees of freedom\. Next\-token prediction differences are expressed as reductions in negative log likelihood, so positive differences favor LOO\-ROLL for every task\. The main LOO\-ROLL tables report standard deviations rather than confidence intervals\.[Table9](https://arxiv.org/html/2609.10980#A5.T9)applies Holm’s step\-down correction to the ten equal\-timepp\-values\[[23](https://arxiv.org/html/2609.10980#bib.bib51)\], where five gains survive correction\.

Table 8:Two\-sided paired tests at equal evaluation cost, with five paired seeds and full\-test GSM8K evaluation\. No comparison hasp<0\.05p<0\.05\.ModelTaskIndividualppQwen3\-0\.6BCountdown0\.19198Qwen3\-0\.6BGSM8K0\.09112Qwen3\-0\.6BNext\-token prediction0\.46333Qwen3\-1\.7BCountdown0\.68251Qwen3\-1\.7BGSM8K0\.51541Qwen3\-1\.7BNext\-token prediction0\.53072Qwen3\-8BGSM8K0\.11829Qwen3\-8BNext\-token prediction0\.23088SmolLM2\-1\.7BCountdown0\.64704SmolLM2\-1\.7BGSM8K0\.53016Table 9:Two\-sided paired tests for the ten equal\-time LOO\-ROLL comparisons\. Holm adjustment treats all ten settings as one family\. Bold adjusted values are below0\.050\.05, and all five corresponding effects favor LOO\-ROLL\.ModelTaskIndividualppHolm\-adjustedppQwen3\-0\.6BGSM8K0\.0000200\.0000200\.00018\\bm\{0\.00018\}Qwen3\-1\.7BGSM8K0\.0015320\.0015320\.0123\\bm\{0\.0123\}Qwen3\-8BGSM8K0\.0000040\.0000040\.00004\\bm\{0\.00004\}SmolLM2\-1\.7BCountdown0\.6545840\.6545841\.0000001\.000000SmolLM2\-1\.7BGSM8K0\.0037080\.0037080\.0260\\bm\{0\.0260\}Qwen3\-0\.6BCountdown0\.7105830\.7105831\.0000001\.000000Qwen3\-1\.7BCountdown0\.0222380\.0222380\.0889530\.088953Qwen3\-0\.6BNext\-token prediction loss0\.0557550\.0557550\.1672660\.167266Qwen3\-1\.7BNext\-token prediction loss0\.0121900\.0121900\.0609510\.060951Qwen3\-8BNext\-token prediction loss0\.0039850\.0039850\.0260\\bm\{0\.0260\}

### E\.2Rank comparisons and transformer audits

The following tables report every run summarized in[Figure5](https://arxiv.org/html/2609.10980#A5.F5), including the stress\-radius runs omitted from the main fixed\-validation comparison\. Rank columns give mean final training fitness±\\pmstandard error over three independent seeds\. Higher fitness is better\. The experiments use commitbcc215eof the[eggroll\-vLLM](https://github.com/ESHyperscale/eggroll-vllm)repository, the latest available at the time of the writing of this manuscript, with the LOO\-ROLL modifications described in[Section9](https://arxiv.org/html/2609.10980#S9)\.

Figure 5:Mean final training fitness for the 0\.6B and 1\.7B rank studies\. Bars are standard errors over three seeds\. Atσ=10−3\\sigma=10^\{\-3\}, rank differences are small relative to seed variation\. Increasing the radius toσ=4×10−3\\sigma=4\\times 10^\{\-3\}produces a much larger decline in fitness across ranks\.Table 10:Expanded useful\-radius comparison over five matched seeds\. Reward changes are rank eight minus rank one in percentage points, so positive values favor rank eight\. Wall overhead is the percentage increase in mean iteration time from rank one to rank eight\. Brackets give paired 95% Student\-ttintervals\. The bold entry excludes zero\.ModelTaskReward changeRank\-8 wall overheadQwen3\-0\.6BCountdown−\.69%​\[−2\.73,1\.34\]\-\.69\\%\\;\[\-2\.73,1\.34\]\+12%\+12\\%Qwen3\-0\.6BGSM8K\+2\.09%​\[−\.23,4\.41\]\+2\.09\\%\\;\[\-\.23,4\.41\]\+12%\+12\\%Qwen3\-1\.7BCountdown\+\.28%​\[−1\.61,2\.17\]\+\.28\\%\\;\[\-1\.61,2\.17\]\+22%\+22\\%Qwen3\-1\.7BGSM8K\+\.76%​\[\.07,1\.46\]\\bm\{\+\.76\\%\\;\[\.07,1\.46\]\}\+12%\+12\\%Qwen3\-4BGSM8K\+\.21%​\[−1\.30,1\.73\]\+\.21\\%\\;\[\-1\.30,1\.73\]\+25%\+25\\%Qwen3\-8BGSM8K−\.70%​\[−1\.58,\.18\]\-\.70\\%\\;\[\-1\.58,\.18\]\+19%\+19\\%Table 11:Complete Qwen3\-0\.6B rank study\. Entries are mean final training fitness±\\pmstandard error over three seeds\. Higher values are better\.Taskσ\\sigmaRank 1Rank 2Rank 4Rank 8Countdown\.001\.001\.195±\.047\.195\\pm\.047\.200±\.053\.200\\pm\.053\.202±\.049\.202\\pm\.049\.190±\.048\.190\\pm\.048Countdown\.004\.004\.0081±\.0007\.0081\\pm\.0007\.0082±\.0007\.0082\\pm\.0007\.0069±\.0007\.0069\\pm\.0007\.0086±\.0004\.0086\\pm\.0004GSM8K\.001\.001\.268±\.006\.268\\pm\.006\.264±\.020\.264\\pm\.020\.272±\.021\.272\\pm\.021\.278±\.013\.278\\pm\.013GSM8K\.004\.004\.0003±\.0003\.0003\\pm\.0003\.0000±\.0000\.0000\\pm\.0000\.0000±\.0000\.0000\\pm\.0000\.0000±\.0000\.0000\\pm\.0000Table 12:Paired rank\-eight minus rank\-one reward changes over three matched seeds, in percentage points\. Positive values favor rank eight\. Brackets give two\-sided 95% Student\-ttintervals, and bold entries exclude zero\.ModelTaskσ=\.001\\sigma=\.001σ=\.004\\sigma=\.004Qwen3\-0\.6BCountdown−\.51%​\[−5\.83,4\.82\]\-\.51\\%\\;\[\-5\.83,4\.82\]\+\.04%​\[−\.42,\.51\]\+\.04\\%\\;\[\-\.42,\.51\]Qwen3\-0\.6BGSM8K\+1\.04%​\[−5\.42,7\.50\]\+1\.04\\%\\;\[\-5\.42,7\.50\]−\.03%​\[−\.17,\.11\]\-\.03\\%\\;\[\-\.17,\.11\]Qwen3\-1\.7BCountdown\+1\.75%​\[−\.15,3\.66\]\+1\.75\\%\\;\[\-\.15,3\.66\]−3\.06%​\[−5\.18,−\.94\]\\bm\{\-3\.06\\%\\;\[\-5\.18,\-\.94\]\}Qwen3\-1\.7BGSM8K\+\.39%​\[−1\.50,2\.29\]\+\.39\\%\\;\[\-1\.50,2\.29\]−4\.07%​\[−4\.44,−3\.70\]\\bm\{\-4\.07\\%\\;\[\-4\.44,\-3\.70\]\}Table 13:Complete Qwen3\-1\.7B rank study\. Rank columns give mean final training fitness±\\pmstandard error over three seeds\. Higher fitness is better\. The final column gives the paired rank\-eight minus rank\-one reward change in percentage points\.Taskσ\\sigmaRank 1Rank 8Reward changeCountdown\.001\.001\.539±\.014\.539\\pm\.014\.556±\.017\.556\\pm\.017\+1\.75%\+1\.75\\%Countdown\.004\.004\.073±\.005\.073\\pm\.005\.042±\.004\.042\\pm\.004−3\.06%\-3\.06\\%GSM8K\.001\.001\.962±\.013\.962\\pm\.013\.966±\.011\.966\\pm\.011\+\.39%\+\.39\\%GSM8K\.004\.004\.283±\.025\.283\\pm\.025\.243±\.025\.243\\pm\.025−4\.07%\-4\.07\\%The complete training results above combine the mean update, sampling variability, and optimization trajectory\. The transformer\-block audit separates the first two effects at a fixed model state\.[Table14](https://arxiv.org/html/2609.10980#A5.T14)compares each finite\-rank estimator with dense Gaussian ES through its single\-direction MSE and compares its estimated population mean with the backpropagated gradient through cosine similarity\. The rank\-one MSE is2626–32%32\\%above the dense baseline and falls to33–4%4\\%above it at rank eight, while the population mean retains a cosine near0\.980\.98at every rank\. The finite\-rank variance is therefore clearly visible in these small blocks, whereas the mean direction changes little over the tested radii\.

Table 14:Comparison of finite\-rank and dense\-Gaussian ES updates in two Qwen3\-0\.6B transformer blocks\. Relative MSE is the single\-direction squared error about the backpropagated block gradient, divided by the corresponding dense\-Gaussian MSE, so11denotes equal sampling error and lower values are better\. Mean\-field cosine is the cosine similarity between the average of 8,192 antithetic updates and the backpropagated block gradient, so11denotes perfect directional agreement\. Values average four perturbation radii within each of three independent perturbation seeds and are reported as means±\\pmstandard errors over seeds\.QuantityBlockRank 1Rank 2Rank 4Rank 8Relative MSEMLP1\.319±\.0361\.319\\pm\.0361\.168±\.0091\.168\\pm\.0091\.090±\.0091\.090\\pm\.0091\.040±\.0021\.040\\pm\.002Relative MSEAttention1\.264±\.0181\.264\\pm\.0181\.127±\.0181\.127\\pm\.0181\.089±\.0251\.089\\pm\.0251\.028±\.0151\.028\\pm\.015Mean\-field cosineMLP\.9817±\.0013\.9817\\pm\.0013\.9830±\.0011\.9830\\pm\.0011\.9837±\.0003\.9837\\pm\.0003\.9827±\.0007\.9827\\pm\.0007Mean\-field cosineAttention\.9800±\.0011\.9800\\pm\.0011\.9832±\.0009\.9832\\pm\.0009\.9836±\.0006\.9836\\pm\.0006\.9842±\.0012\.9842\\pm\.0012
### E\.3Controlled numerical experiments

These experiments examine the parts of the theory that have exact numerical answers\. Computing the covariance directly from sampled perturbation entries reproduces the tensor predicted by[Equation93](https://arxiv.org/html/2609.10980#S7.E93), without evaluating the formula itself\. Independent calculations also reproduce the resolvent multiplier at individual frequencies, the two Hessians in[Proposition5\.3](https://arxiv.org/html/2609.10980#S5.Thmtheorem3), the cubic identity in[Proposition6\.2](https://arxiv.org/html/2609.10980#S6.Thmtheorem2), and the1/r1/rrank expansion in[Theorem8\.1](https://arxiv.org/html/2609.10980#S8.Thmtheorem1)\. Together, these calculations verify the algebra before any model or optimization effects enter\.

The finite\-sampling experiment asks whether Monte Carlo estimates reproduce those exact quantities\.[Table15](https://arxiv.org/html/2609.10980#A5.T15)uses100,000100\{,\}000directions for a3×53\\times 5affine problem\. The empirical single\-direction variance is within0\.9%0\.9\\%of[Equation95](https://arxiv.org/html/2609.10980#S7.E95)at every rank\. The same calculation recovers the stability transition in[Figure3](https://arxiv.org/html/2609.10980#S5.F3): the largest Jacobian real part is0\.029250\.02925at rank one and−0\.00444\-0\.00444at rank two\.

Table 15:Monte Carlo verification of the exact single\-direction variance in[Equation95](https://arxiv.org/html/2609.10980#S7.E95)\. Results use100,000100\{,\}000independent sampled directions for a normalized3×53\\times 5affine gradient\.RankrrExact varianceMonte Carlo varianceRelative error134\.0034\.0034\.2434\.240\.70%0\.70\\%225\.0025\.0025\.2125\.210\.86%0\.86\\%420\.5020\.5020\.6220\.620\.57%0\.57\\%818\.2518\.2518\.1918\.190\.35%0\.35\\%The quadratic case provides an exact test of the finite\-population dynamics\. Leth⁡\(W\)=∥W∥F2/2h\(W\)=\\lVert W\\rVert\_\{F\}^\{2\}/2, so that the exact population field isWWby[Proposition6\.1](https://arxiv.org/html/2609.10980#S6.Thmtheorem1)\. IfG^N\\widehat\{G\}\_\{N\}averagesNNdirections, unbiasedness and[Equation96](https://arxiv.org/html/2609.10980#S7.E96)give

𝔼⁡\[∥W−η​G^N∥F2∣W\]=\[1−2​η\+η2​\(1\+κrN\)\]​∥W∥F2\.\\mathbb\{E\}\\left\[\\lVert W\-\\eta\\widehat\{G\}\_\{N\}\\rVert\_\{F\}^\{2\}\\mid W\\right\]=\\left\[1\-2\\eta\+\\eta^\{2\}\\left\(1\+\\frac\{\\kappa\_\{r\}\}\{N\}\\right\)\\right\]\\lVert W\\rVert\_\{F\}^\{2\}\.\(168\)Drawing a fresh independent population at every step and iterating this identity yields

𝔼⁡\[∥WT∥F2\]∥W0∥F2=\[1−2​η\+η2​\(1\+κrN\)\]T\.\\frac\{\\mathbb\{E\}\[\\lVert W\_\{T\}\\rVert\_\{F\}^\{2\}\]\}\{\\lVert W\_\{0\}\\rVert\_\{F\}^\{2\}\}=\\left\[1\-2\\eta\+\\eta^\{2\}\\left\(1\+\\frac\{\\kappa\_\{r\}\}\{N\}\\right\)\\right\]^\{T\}\.\(169\)We test an8×88\\times 8problem atη=0\.3\\eta=0\.3using 4,000 independent one\-step trials per rank–population pair and 5,000 independent 25\-step trajectories\.

Figure 6:Finite\-sample validation of[Equation169](https://arxiv.org/html/2609.10980#A5.E169)\.Left:Monte Carlo one\-step ratios agree with the exact curves across populations and ranks\. The horizontal line marks stability\.Right:atN=16N=16, the formula predicts the full mean\-square trajectory\. Rank one is unstable, whereas rank eight and dense Gaussian ES contract\. Marked curves are Monte Carlo means and dashed curves are exact\.The synthetic result makes the finite\-population prediction visible in a controlled setting\. With few directions, the termκr/N\\kappa\_\{r\}/Ncan move the iteration across the stability boundary\. IncreasingNNremoves this difference, while increasing the matrix width makes the rank\-one contribution small relative to the commonm​n\+1mn\+1term\. The transformer experiments in[Section10](https://arxiv.org/html/2609.10980#S10)examine these predictions in a nonlinear model\.

## Appendix FEffect of population centering and standardization

The operator analysis concerns the raw antithetic estimator, while the end\-to\-end experiments retain the population centering and standardization specified by EGGROLL\. We therefore distinguish the direction of one realized update from the expectation of the normalized estimator\. The proposition shows that prompt\-wise centering and one common scale do not rotate a realized update\. The corollary shows why a scale computed from the same random population can nevertheless change the expected field\.

###### Proposition F\.1\(Population centering and common standardization preserve each realized direction\)\.

Let pairjjuse perturbations\+Ej\+E\_\{j\}and−Ej\-E\_\{j\}, with prompt\-level fitnessesYj,\+,pY\_\{j,\+,p\}andYj,−,pY\_\{j,\-,p\}\. For arbitrary prompt offsetscpc\_\{p\}, define

Zj,±=1P​∑p=1P\(Yj,±,p−cp\)\.Z\_\{j,\\pm\}=\\frac\{1\}\{P\}\\sum\_\{p=1\}^\{P\}\(Y\_\{j,\\pm,p\}\-c\_\{p\}\)\.\(170\)ThenZj,\+−Zj,−=P−1​∑p\(Yj,\+,p−Yj,−,p\)Z\_\{j,\+\}\-Z\_\{j,\-\}=P^\{\-1\}\\sum\_\{p\}\(Y\_\{j,\+,p\}\-Y\_\{j,\-,p\}\)exactly\. If allZj,±Z\_\{j,\\pm\}are subsequently divided by the same random scalarS\>0S\>0, the ES update is exactlyS−1S^\{\-1\}times the unstandardized update\. Consequently, per\-prompt population centering has no effect on the update, and standardization changes only its realized magnitude\.

###### Proof\.

The offsets cancel algebraically in every pair difference\. The ES update is linear in those differences, so dividing every score by the same positiveSSmultiplies their weighted sum byS−1S^\{\-1\}and cannot change its realized direction\. ∎

###### Corollary F\.2\(Bias induced by a same\-population normalizer\)\.

At a fixed iterate, letg^=g\+ξ\\widehat\{g\}=g\+\\xiwith deterministicggand𝔼⁡\[ξ\]=0\\mathbb\{E\}\[\\xi\]=0, and letα=S−1\\alpha=S^\{\-1\}have finite second moment\. Then

‖𝔼⁡\[α​g^\]−𝔼⁡\[α\]​g‖≤Var⁡\(α\)​𝔼​\[∥ξ∥2\]\.\\left\\\|\\mathbb\{E\}\[\\alpha\\widehat\{g\}\]\-\\mathbb\{E\}\[\\alpha\]g\\right\\\|\\leq\\sqrt\{\\operatorname\{Var\}\(\\alpha\)\\mathbb\{E\}\[\\lVert\\xi\\rVert^\{2\}\]\}\.\(171\)Ifα\\alphais measurable with respect to a sigma\-field𝒢\\mathcal\{G\}and𝔼⁡\[ξ∣𝒢\]=0\\mathbb\{E\}\[\\xi\\mid\\mathcal\{G\}\]=0, the left side is zero\. Independence ofα\\alphaandξ\\xiis sufficient\. A same\-population normalizer can otherwise change the expected field\.

###### Proof\.

Since𝔼⁡\[ξ\]=0\\mathbb\{E\}\[\\xi\]=0,

𝔼⁡\[α​g^\]−𝔼⁡\[α\]​g=𝔼⁡\[\(α−𝔼⁡\[α\]\)​ξ\]\.\\mathbb\{E\}\[\\alpha\\widehat\{g\}\]\-\\mathbb\{E\}\[\\alpha\]g=\\mathbb\{E\}\[\(\\alpha\-\\mathbb\{E\}\[\\alpha\]\)\\xi\]\.\(172\)Cauchy–Schwarz gives[Equation171](https://arxiv.org/html/2609.10980#A6.E171)\. Ifα\\alphais𝒢\\mathcal\{G\}\-measurable and𝔼⁡\[ξ∣𝒢\]=0\\mathbb\{E\}\[\\xi\\mid\\mathcal\{G\}\]=0, then𝔼⁡\[α​ξ\]=𝔼⁡\[α​𝔼​\[ξ∣𝒢\]\]=0\\mathbb\{E\}\[\\alpha\\xi\]=\\mathbb\{E\}\[\\alpha\\mathbb\{E\}\[\\xi\\mid\\mathcal\{G\}\]\]=0\. ∎

The preceding bound becomes explicit in the population size once the update mean and the empirical scale satisfy their usual concentration rates\.

###### Proposition F\.3\(NN\-dependent standardization bias\)\.

For each population sizeNN, writeg^N=g\+ξN\\widehat\{g\}\_\{N\}=g\+\\xi\_\{N\}andαN=SN−1\\alpha\_\{N\}=S\_\{N\}^\{\-1\}\. Suppose that, for constantsV,A<∞V,A<\\inftyindependent ofNN,

𝔼⁡\[∥ξN∥2\]≤VN,Var⁡\(αN\)≤AN\.\\mathbb\{E\}\[\\lVert\\xi\_\{N\}\\rVert^\{2\}\]\\leq\\frac\{V\}\{N\},\\qquad\\operatorname\{Var\}\(\\alpha\_\{N\}\)\\leq\\frac\{A\}\{N\}\.\(173\)Then

‖𝔼⁡\[αN​g^N\]−𝔼⁡\[αN\]​g‖≤A​VN\.\\left\\\|\\mathbb\{E\}\[\\alpha\_\{N\}\\widehat\{g\}\_\{N\}\]\-\\mathbb\{E\}\[\\alpha\_\{N\}\]g\\right\\\|\\leq\\frac\{\\sqrt\{AV\}\}\{N\}\.\(174\)Consequently, same\-population standardization changes the raw population field by anO⁡\(N−1\)O\(N^\{\-1\}\)term, apart from the scalar factor𝔼⁡\[αN\]\\mathbb\{E\}\[\\alpha\_\{N\}\]\.

###### Proof\.

Apply[Equation171](https://arxiv.org/html/2609.10980#A6.E171)and substitute the two bounds in[Equation173](https://arxiv.org/html/2609.10980#A6.E173)\. ∎

The first condition in[Equation173](https://arxiv.org/html/2609.10980#A6.E173)is the ordinaryN−1N^\{\-1\}variance reduction from averaging independent member updates\. The second follows, for example, when the empirical scale concentrates at rateN−1/2N^\{\-1/2\}around a positive population scale and its inverse has uniformly controlled second moments\. A deterministic floor on the empirical scale supplies a direct sufficient condition\. More precisely, ifSN≥s0\>0S\_\{N\}\\geq s\_\{0\}\>0almost surely, the population scale iss\>0s\>0, and𝔼⁡\[\(SN−s\)2\]≤CS/N\\mathbb\{E\}\[\(S\_\{N\}\-s\)^\{2\}\]\\leq C\_\{S\}/N, then

Var⁡\(SN−1\)≤𝔼⁡\[\(SN−1−s−1\)2\]≤CSs02​s2​N,\\operatorname\{Var\}\(S\_\{N\}^\{\-1\}\)\\leq\\mathbb\{E\}\[\(S\_\{N\}^\{\-1\}\-s^\{\-1\}\)^\{2\}\]\\leq\\frac\{C\_\{S\}\}\{s\_\{0\}^\{2\}s^\{2\}N\},\(175\)so[Equation174](https://arxiv.org/html/2609.10980#A6.E174)holds withA=CS/\(s02​s2\)A=C\_\{S\}/\(s\_\{0\}^\{2\}s^\{2\}\)\. Thus the raw resolvent field remains the leading expected direction as the population grows\. The normalization audit in[Table16](https://arxiv.org/html/2609.10980#A6.T16)measures the remaining discrepancy at the population sizes used here\. AtN=128N=128, the raw and standardized means have cosine similarities of0\.999900\.99990for dense perturbations and0\.999700\.99970for rank one\. The directional residual also decreases with population size, showing that standardization acts predominantly as a common rescaling in this regime\.

### F\.1Transformer\-block audit

The official EGGROLL implementation divides all member scores by one standard deviation computed from the same population\. We test the resulting finite\-population effect with 4,096 coupled directions for each of two Qwen3\-0\.6B blocks, four radii from10−310^\{\-3\}to8×10−38\\times 10^\{\-3\}, and three seeds\. For each block, radius, and seed, the directions are partitioned into independent populations of sizeNN\. After fitting the best scalar multiple of the raw field, the directional residual is the norm of the remaining orthogonal component divided by the norm of the standardized field\.

Table 16:Comparison of mean updates before and after population score standardization, averaged over two transformer blocks, four perturbation radii, and three seeds\. Cosine measures directional agreement between the raw and standardized means, with11indicating identical directions\. Residual is the component of the standardized mean that cannot be explained by rescaling the raw mean, expressed as a percentage of the standardized mean’s norm\. Higher cosine and lower residual indicate a smaller directional effect\.NNDense cosineDense residualRank\-one cosineRank\-one residual32\.99954\.999543\.03%3\.03\\%\.99858\.998585\.28%5\.28\\%64\.99979\.999792\.04%2\.04\\%\.99924\.999243\.87%3\.87\\%128\.99990\.999901\.44%1\.44\\%\.99970\.999702\.42%2\.42\\%The normalized and raw fields remain closely aligned, and their difference decreases with population size\. At the training valueN=128N=128, the rank\-one cosine is0\.999700\.99970with a2\.42%2\.42\\%directional residual\. The dense control gives0\.999900\.99990and1\.44%1\.44\\%\. Fits across ranks1,2,4,81,2,4,8do not produce a coefficient whose direction and magnitude remain stable under the predictedσ2\\sigma^\{2\}scaling, leaving that correction below the resolution of this audit\.