当预处理指数转为负数：学习率耦合与跨环境泛化

# When the Preconditioning Exponent Turns Negative: Learning-Rate Coupling and Cross-Environment Generalization
Source: [https://arxiv.org/html/2609.30271](https://arxiv.org/html/2609.30271)
###### Abstract

Adaptive optimizers are commonly parameterized by a fixed power of the second\-moment estimate\. Existing partially adaptive methods study exponents between momentum\-like updates and the standard Adam square root, while the interaction between this exponent and the global learning rate is less understood\. We perform a controlled cross\-environment study using a paired four\-environment classification problem with stable sparse features, environment\-dependent spurious sparse features, dense features, and high\-dimensional noise\. Across 420 source\-training runs covering 21 preconditioning exponentsp∈\[−0\.5,0\.5\]p\\in\[\-0\.5,0\.5\]and five learning ratesη∈\[10−4,10−2\]\\eta\\in\[10^\{\-4\},10^\{\-2\}\], we find that the exponent maximizing cross\-environment accuracy decreases almost linearly withlog10⁡η\\log\_\{10\}\\eta\. The fitted slopes range from−0\.270\-0\.270to−0\.300\-0\.300, withR2R^\{2\}between0\.9720\.972and0\.9960\.996\. Atη=10−2\\eta=10^\{\-2\}, source\-validation selection still prefers positive exponents in all four environments, whereas cross\-environment and worst\-environment criteria prefer negative exponents\. Checkpoint decomposition shows that lowerppreduces the learned spurious\-to\-stable and noise\-to\-stable weight ratios; under reversed correlation, it also reduces the magnitude of the harmful spurious margin\. Negativeppis therefore not a universally optimal setting\. It is a high\-step\-size allocation regime produced by the joint action of learning rate and preconditioning\. The study also exposes a model\-selection conflict: source\-domain validation systematically selects a different preconditioning regime from the one that maximizes robustness to environmental change\. The results are a single\-seed, finite\-budget mechanism study rather than a broad benchmark claim\.

## 1Introduction

Adaptive gradient methods change the geometry of optimization by assigning a different effective gain to each coordinate\. Adam, for example, divides a bias\-corrected first moment by the square root of a bias\-corrected second moment\(Kingma and Ba,[2015](https://arxiv.org/html/2609.30271#bib.bib5)\)\. This coordinate\-wise rescaling often accelerates optimization, but it can also select solutions that differ substantially from those reached by gradient descent\(Wilson et al\.,[2017](https://arxiv.org/html/2609.30271#bib.bib11)\)\. A natural way to expose the strength of adaptivity is to replace the fixed square root with a continuous exponent,

Δ​θt,i=−η​m^t,i​\(v^t,i\+ϵ\)−p\.\\Delta\\theta\_\{t,i\}=\-\\eta\\,\\widehat\{m\}\_\{t,i\}\(\\widehat\{v\}\_\{t,i\}\+\\epsilon\)^\{\-p\}\.\(1\)The casesp=0p=0andp=1/2p=1/2recover a momentum\-like direction and the usual Adam denominator, respectively\. Partially adaptive methods have usedp∈\(0,1/2\]p\\in\(0,1/2\]to interpolate between these behaviors\(Chen et al\.,[2020](https://arxiv.org/html/2609.30271#bib.bib2)\)\. More recent work has also questioned whether the square root is intrinsically necessary\(Lin et al\.,[2024](https://arxiv.org/html/2609.30271#bib.bib7)\)\.

Most discussions treatppas an optimizer\-family parameter and the learning rateη\\etaas a separate scale parameter\. This separation is exact only for a fixed update field\. In multi\-step training,η\\etachanges the trajectory and therefore changes future gradients and moment estimates\. Even before this feedback is considered, the instantaneous effective gain in[Equation1](https://arxiv.org/html/2609.30271#S1.E1)contains the product

η​\(v^t,i\+ϵ\)−p\.\\eta\(\\widehat\{v\}\_\{t,i\}\+\\epsilon\)^\{\-p\}\.\(2\)The same nominal exponent can therefore correspond to very different effective coordinate gains at different learning rates\. Conversely, a change inppcan partly compensate for a change inη\\eta\.

This interaction is especially relevant under distribution shift\. A model may fit both stable and spurious features on its source distribution, yet only the stable features remain predictive when the environment changes\. Domain generalization and robust optimization usually address this problem through objectives, environment labels, or group information\(Arjovsky et al\.,[2019](https://arxiv.org/html/2609.30271#bib.bib1); Sagawa et al\.,[2019](https://arxiv.org/html/2609.30271#bib.bib10); Gulrajani and Lopez\-Paz,[2021](https://arxiv.org/html/2609.30271#bib.bib4)\)\. Here we ask a narrower question:

> How does the optimizer’s preconditioning exponent interact with learning rate to allocate a linear classifier across stable, spurious, and noise features, and where does the cross\-environment optimum lie?

We answer this question in a paired four\-environment extension of the controlled construction used byWilson et al\. \([2017](https://arxiv.org/html/2609.30271#bib.bib11)\)\. Every environment shares the same labels, dense features, stable sparse features, sparse masks, amplitude noise, and pure\-noise coordinates\. Only the sign correlation of one sparse block with the label changes\. A model is trained on one environment, its checkpoint is selected only by source validation loss, and the frozen checkpoint is evaluated on all four test environments\. This design isolates optimizer\-dependent feature allocation from unrelated dataset variation\.

The central empirical result is a learning\-rate\-dependent reversal of the preferred exponent\. For each source environment, the exponent maximizing mean cross\-environment accuracy decreases nearly linearly withlog10⁡η\\log\_\{10\}\\eta\. At small learning rates, the optimum remains positive\. At large learning rates, the optimum enters the negative region\. Source validation follows the same broad downward trend but remains systematically higher than the cross\-environment optimum\. Thus, negativeppshould not be interpreted as a universally better optimizer\. It is a regime that appears when a large global step is combined with a preconditioner that suppresses low\-second\-moment coordinates rather than amplifying them\.

The contributions are:

1. 1\.We conduct a completep×ηp\\times\\etascan over four paired source environments, with target environments excluded from checkpoint selection\.
2. 2\.We identify a consistent empirical law: the cross\-environment\-optimal exponent decreases approximately linearly withlog10⁡η\\log\_\{10\}\\eta, with closely matched slopes across environments\.
3. 3\.We show that source\-validation\-optimal and cross\-environment\-optimal exponents are systematically separated, revealing an optimizer model\-selection conflict under distribution shift\.
4. 4\.By decomposing the learned linear weights and signed margins, we show that lower and negativeppreduce relative reliance on spurious and noise coordinates in the high\-learning\-rate regime\.

The scope is deliberately limited\. The study uses one random seed, one affine classifier, and a fixed training budget\. Its purpose is to establish and explain a controlled phenomenon, not to claim a universal performance improvement on real\-world domain generalization benchmarks\.

## 2Related Work

#### Adaptive optimization and implicit bias\.

AdaGrad and Adam use historical gradient magnitudes to construct coordinate\-wise gains\(Duchi et al\.,[2011](https://arxiv.org/html/2609.30271#bib.bib3); Kingma and Ba,[2015](https://arxiv.org/html/2609.30271#bib.bib5)\)\.Wilson et al\. \([2017](https://arxiv.org/html/2609.30271#bib.bib11)\)showed that adaptive methods can converge to solutions with different generalization properties from gradient descent, even on simple linearly separable problems\. Padam introduced a continuous partial\-adaptivity exponent between SGD\-like and Adam\-like updates and argued that excessive adaptivity can harm generalization\(Chen et al\.,[2020](https://arxiv.org/html/2609.30271#bib.bib2)\)\.Lin et al\. \([2024](https://arxiv.org/html/2609.30271#bib.bib7)\)studied the opposite extension by removing the square root and strengthening the preconditioner\. Our study differs in two ways\. First, it scans the exponent symmetrically into the negative region\. Second, it treatsppand the global learning rate as a coupled two\-dimensional control space rather than evaluatingppat a fixed learning\-rate regime\.

#### Learning rate and feature learning\.

Large learning rates can change which patterns are learned and in what order\.Li et al\. \([2019](https://arxiv.org/html/2609.30271#bib.bib6)\)connected large initial learning rates to delayed memorization of easy but poorly generalizing patterns\.Lu et al\. \([2024](https://arxiv.org/html/2609.30271#bib.bib8)\)showed that large\-learning\-rate oscillation can promote weak\-feature learning in a controlled feature\-noise model\. These works motivate examining learning rate as more than a uniform multiplier\. We complement them by showing that the exponent of adaptive preconditioning shifts systematically with learning rate, and that the robust regime can cross through zero into negativepp\.

#### Spurious correlations and domain generalization\.

Invariant risk minimization, group distributionally robust optimization, and DomainBed study learning under environmental or group shifts\(Arjovsky et al\.,[2019](https://arxiv.org/html/2609.30271#bib.bib1); Sagawa et al\.,[2019](https://arxiv.org/html/2609.30271#bib.bib10); Gulrajani and Lopez\-Paz,[2021](https://arxiv.org/html/2609.30271#bib.bib4)\)\. Recent controlled analyses emphasize that core and spurious features have distinct learning dynamics and that their relative complexity matters\(Qiu et al\.,[2024](https://arxiv.org/html/2609.30271#bib.bib9)\)\. Our experiment does not propose a new domain generalization objective\. Instead, it isolates how a standard source\-only optimizer reallocates a fixed linear model across known stable, spurious, and noise blocks\. This makes the optimizer mechanism directly observable\.

#### Information allocation\.

The view that optimizer bias can be studied through relative signal allocation across parameter pathways has been developed in recent work\(Zhang et al\.,[2026](https://arxiv.org/html/2609.30271#bib.bib12)\)\. The present paper applies the same general perspective to feature blocks under distribution shift: the preconditioning exponent changes which coordinates receive effective update mass, and this allocation has environment\-dependent consequences\.

## 3Learning\-Rate–Exponent Coupling

### 3\.1Generalized preconditioning

Letgt,ig\_\{t,i\}denote the stochastic gradient of coordinateii\. We use Adam\-style exponential moving averages

mt,i\\displaystyle m\_\{t,i\}=β1​mt−1,i\+\(1−β1\)​gt,i,\\displaystyle=\\beta\_\{1\}m\_\{t\-1,i\}\+\(1\-\\beta\_\{1\}\)g\_\{t,i\},\(3\)vt,i\\displaystyle v\_\{t,i\}=β2​vt−1,i\+\(1−β2\)​gt,i2,\\displaystyle=\\beta\_\{2\}v\_\{t\-1,i\}\+\(1\-\\beta\_\{2\}\)g\_\{t,i\}^\{2\},\(4\)with standard bias correctionsm^t,i\\widehat\{m\}\_\{t,i\}andv^t,i\\widehat\{v\}\_\{t,i\}\. The generalized update is given by[Equation1](https://arxiv.org/html/2609.30271#S1.E1)\. For two coordinatesiiandjj, the instantaneous update ratio is

\|Δ​θt,i\|\|Δ​θt,j\|=\|m^t,i\|\|m^t,j\|​\(v^t,j\+ϵv^t,i\+ϵ\)p\.\\frac\{\|\\Delta\\theta\_\{t,i\}\|\}\{\|\\Delta\\theta\_\{t,j\}\|\}=\\frac\{\|\\widehat\{m\}\_\{t,i\}\|\}\{\|\\widehat\{m\}\_\{t,j\}\|\}\\left\(\\frac\{\\widehat\{v\}\_\{t,j\}\+\\epsilon\}\{\\widehat\{v\}\_\{t,i\}\+\\epsilon\}\\right\)^\{p\}\.\(5\)The learning rate cancels from this same\-step ratio, whereasppdirectly changes coordinate allocation\. Across steps, however,η\\etachanges the parameters reached by the optimizer and therefore changes subsequent gradients, moments, and ratios\.

Forp<0p<0, the preconditioner becomes

\(v^t,i\+ϵ\)−p=\(v^t,i\+ϵ\)\|p\|\.\(\\widehat\{v\}\_\{t,i\}\+\\epsilon\)^\{\-p\}=\(\\widehat\{v\}\_\{t,i\}\+\\epsilon\)^\{\|p\|\}\.\(6\)Coordinates with very small second moments are then suppressed rather than amplified\. This does not imply that every high\-vvcoordinate is useful\. It only reverses the usual Adam ordering of low\- and high\-second\-moment coordinates\.

### 3\.2A local compensation relation

To understand the joint trend, consider a representative coordinate scales=v^\+ϵs=\\widehat\{v\}\+\\epsilon\. The log effective gain is

log⁡G​\(η,p;s\)=log⁡η−p​log⁡s\.\\log G\(\\eta,p;s\)=\\log\\eta\-p\\log s\.\(7\)If comparable training behavior requires this gain to remain near a constantCC, then

p≈log⁡η−Clog⁡s\.p\\approx\\frac\{\\log\\eta\-C\}\{\\log s\}\.\(8\)When0<s<10<s<1,log⁡s<0\\log s<0, so

∂p∂log⁡η=1log⁡s<0\.\\frac\{\\partial p\}\{\\partial\\log\\eta\}=\\frac\{1\}\{\\log s\}<0\.\(9\)This simple relation predicts the direction observed in the experiments: increasingη\\etashould reduce the exponent required to reach a comparable effective\-update regime\. It is not a full dynamical theory, becausessitself changes along the trajectory and differs across coordinates\. It nevertheless supplies a testable sign prediction and explains why the optimum may pass below zero\.

### 3\.3Source and cross\-environment optima

For each source environmentese\_\{s\}, learning rateη\\eta, and exponentpp, letθ⋆​\(es,η,p\)\\theta^\{\\star\}\(e\_\{s\},\\eta,p\)be the checkpoint selected by source validation loss\. We report three exponents:

pID⋆​\(η\)\\displaystyle p\_\{\\mathrm\{ID\}\}^\{\\star\}\(\\eta\)=arg⁡minp⁡Lvales​\(θ⋆\),\\displaystyle=\\arg\\min\_\{p\}L\_\{\\mathrm\{val\}\}^\{e\_\{s\}\}\(\\theta^\{\\star\}\),\(10\)pcross⋆​\(η\)\\displaystyle p\_\{\\mathrm\{cross\}\}^\{\\star\}\(\\eta\)=arg⁡maxp⁡1\|ℰ\|−1​∑e≠esAteste​\(θ⋆\),\\displaystyle=\\arg\\max\_\{p\}\\frac\{1\}\{\|\\mathcal\{E\}\|\-1\}\\sum\_\{e\\neq e\_\{s\}\}A\_\{\\mathrm\{test\}\}^\{e\}\(\\theta^\{\\star\}\),\(11\)pworst⋆​\(η\)\\displaystyle p\_\{\\mathrm\{worst\}\}^\{\\star\}\(\\eta\)=arg⁡maxp⁡mine∈ℰ⁡Ateste​\(θ⋆\)\.\\displaystyle=\\arg\\max\_\{p\}\\min\_\{e\\in\\mathcal\{E\}\}A\_\{\\mathrm\{test\}\}^\{e\}\(\\theta^\{\\star\}\)\.\(12\)The latter two are*oracle analysis criteria*: target test environments are used only after source\-based checkpoint selection\. They are not deployable source\-only model\-selection rules\. This distinction is essential because a major result of the paper is precisely that source selection and cross\-environment selection disagree\.

## 4Controlled Cross\-Environment Experiment

### 4\.1Paired environments

Labelsy∈\{0,1\}y\\in\\\{0,1\\\}are sampled withPr⁡\(y=1\)=0\.6\\Pr\(y=1\)=0\.6, and we writey~=2​y−1∈\{−1,\+1\}\\widetilde\{y\}=2y\-1\\in\\\{\-1,\+1\\\}\. Each input is a concatenation

x=\[xdense,xstable,xspur,xnoise\]\.x=\[x\_\{\\mathrm\{dense\}\},x\_\{\\mathrm\{stable\}\},x\_\{\\mathrm\{spur\}\},x\_\{\\mathrm\{noise\}\}\]\.\(13\)The four blocks are:

- •Dense block:four coordinates, each generated as0\.7​y~\+𝒩​\(0,32\)0\.7\\widetilde\{y\}\+\\mathcal\{N\}\(0,3^\{2\}\)\.
- •Stable sparse block:60 coordinates divided equally among activation probabilitiesπ∈\{0\.005,0\.01,0\.02,0\.05,0\.1,0\.2\}\\pi\\in\\\{0\.005,0\.01,0\.02,0\.05,0\.1,0\.2\\\}\. Active coordinates have signal magnitude3\.03\.0, additive noise standard deviation0\.050\.05, and label correlation1\.01\.0\.
- •Spurious sparse block:the same dimensions, activation probabilities, signal magnitude, and noise as the stable block, but with environment\-dependent label correlation\.
- •Noise block:3000 independent standard Gaussian coordinates\.

The spurious correlations are0\.90\.9,0\.50\.5,0, and−0\.5\-0\.5, defining strong\-positive, weak\-positive, neutral, and reversed environments\. For a fixed split, all random draws are paired across environments\. Labels, dense features, stable sparse features, sparse masks, amplitude noise, and pure\-noise coordinates are identical\. Only the sign correlation of the spurious block changes\. This pairing makes cross\-environment differences attributable to the intended shift rather than to resampling noise\.

Each environment contains 8192 training, 2048 validation, and 8192 test examples\. The input dimension is 3124\.

### 4\.2Model and optimization grid

The model is a single affine binary classifier

fθ​\(x\)=x⊤​w\+b,f\_\{\\theta\}\(x\)=x^\{\\top\}w\+b,\(14\)trained with binary cross entropy\. The optimizer uses[Equation1](https://arxiv.org/html/2609.30271#S1.E1)withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,ϵ=10−8\\epsilon=10^\{\-8\}, and zero weight decay\. We scan

p∈\{−0\.50,−0\.45,…,0\.45,0\.50\}p\\in\\\{\-0\.50,\-0\.45,\\ldots,0\.45,0\.50\\\}\(15\)and

η∈\{10−4,3×10−4,10−3,3×10−3,10−2\}\.\\eta\\in\\\{10^\{\-4\},3\\times 10^\{\-4\},10^\{\-3\},3\\times 10^\{\-3\},10^\{\-2\}\\\}\.\(16\)For each of the four source environments, this gives21×5=10521\\times 5=105configurations and 420 runs in total\. Every run uses seed 42, batch size 128, and 50 epochs\. The best checkpoint is chosen only by source validation loss\. The selected checkpoint is then evaluated on all four test environments, yielding 1680 cross\-evaluations\.

### 4\.3Feature\-allocation diagnostics

Because the classifier is linear and the feature blocks are known, the learned allocation is directly measurable\. For blockBB, we report

RB=RMS⁡\(wB\),R\_\{B\}=\\operatorname\{RMS\}\(w\_\{B\}\),\(17\)and especially

Rspur/stable=RMS⁡\(wspur\)RMS⁡\(wstable\),Rnoise/stable=RMS⁡\(wnoise\)RMS⁡\(wstable\)\.R\_\{\\mathrm\{spur/stable\}\}=\\frac\{\\operatorname\{RMS\}\(w\_\{\\mathrm\{spur\}\}\)\}\{\\operatorname\{RMS\}\(w\_\{\\mathrm\{stable\}\}\)\},\\qquad R\_\{\\mathrm\{noise/stable\}\}=\\frac\{\\operatorname\{RMS\}\(w\_\{\\mathrm\{noise\}\}\)\}\{\\operatorname\{RMS\}\(w\_\{\\mathrm\{stable\}\}\)\}\.\(18\)We also decompose the mean signed test margin by block:

MBe=𝔼\(x,y\)∼e​\[y~​xB⊤​wB\]\.M\_\{B\}^\{e\}=\\mathbb\{E\}\_\{\(x,y\)\\sim e\}\[\\widetilde\{y\}\\,x\_\{B\}^\{\\top\}w\_\{B\}\]\.\(19\)A negativeMspureM\_\{\\mathrm\{spur\}\}^\{e\}means that the learned spurious component actively opposes the correct label in environmentee\.

## 5Results

### 5\.1The cross\-environment\-optimal exponent decreases with learning rate

[Figure1](https://arxiv.org/html/2609.30271#S5.F1)shows mean cross\-environment accuracy over the fullp×ηp\\times\\etagrid\. The white path markspcross⋆p\_\{\\mathrm\{cross\}\}^\{\\star\}at each learning rate\. All four environments exhibit the same qualitative motion: the optimum shifts from the positive side at smallη\\etatoward zero and then into negativeppat largeη\\eta\.

![Refer to caption](https://arxiv.org/html/2609.30271v1/x1.png)Figure 1:Mean cross\-environment accuracy over the completep×ηp\\times\\etagrid\. Each panel uses one source environment; the source test environment is excluded from the mean\. White markers show the cross\-environment\-optimal exponent at each learning rate\. The optimum moves monotonically toward lowerppas the learning rate increases\.Fitting

pcross⋆=a​log10⁡η\+bp\_\{\\mathrm\{cross\}\}^\{\\star\}=a\\log\_\{10\}\\eta\+b\(20\)produces slopes between−0\.270\-0\.270and−0\.300\-0\.300\. The fit is unusually consistent across the four shifts, withR2R^\{2\}values from0\.9720\.972to0\.9960\.996\([Table1](https://arxiv.org/html/2609.30271#S5.T1)\)\. The result supports the sign prediction in[Equation9](https://arxiv.org/html/2609.30271#S3.E9)\. It does not imply an exact universal linear law, but it does show that the observed reversal is structured rather than an isolated grid artifact\.

Table 1:Linear fits ofpcross⋆p\_\{\\mathrm\{cross\}\}^\{\\star\}againstlog10⁡η\\log\_\{10\}\\eta\.![Refer to caption](https://arxiv.org/html/2609.30271v1/x2.png)Figure 2:Exponent selected by source validation, mean cross\-environment accuracy, and worst\-environment accuracy\. All three optima generally decline with learning rate, but the robust criteria are systematically lower and enter the negative region earlier\.
### 5\.2Source validation and robust selection disagree

[Figure2](https://arxiv.org/html/2609.30271#S5.F2)comparespID⋆p\_\{\\mathrm\{ID\}\}^\{\\star\},pcross⋆p\_\{\\mathrm\{cross\}\}^\{\\star\}, andpworst⋆p\_\{\\mathrm\{worst\}\}^\{\\star\}\. The source\-validation optimum declines with learning rate but stays above the robust optima\. Atη=10−2\\eta=10^\{\-2\}, source validation selects positiveppin every environment, while both robust criteria select negativepp\([Table2](https://arxiv.org/html/2609.30271#S5.T2)\)\. The ID–cross gap ranges from0\.200\.20to0\.400\.40\.

Table 2:Selected exponents atη=10−2\\eta=10^\{\-2\}\. Robust optima are evaluated only for analysis after source\-validation checkpoint selection\.This gap has a direct methodological implication\. Tuning the optimizer solely on source validation loss can favor a preconditioning regime that fits the source more aggressively but is less robust to a change in spurious correlation\. Domain\-generalization work has emphasized that model selection is part of the problem\(Gulrajani and Lopez\-Paz,[2021](https://arxiv.org/html/2609.30271#bib.bib4)\); here, even a one\-dimensional optimizer exponent exhibits this conflict\.

### 5\.3Negativeppis a high\-learning\-rate regime, not a universal optimum

[Figure3](https://arxiv.org/html/2609.30271#S5.F3)shows the fullppcurves atη=10−2\\eta=10^\{\-2\}\. Negativeppdoes not uniformly improve every metric\. Source accuracy is usually maximized at a higher exponent than cross or worst\-environment accuracy\. The robust curves peak at modestly negative values, while excessively negativeppeventually underfits useful structure\. This produces a finite negative optimum rather than a monotonic preference for the smallestpp\.

![Refer to caption](https://arxiv.org/html/2609.30271v1/x3.png)Figure 3:Source, mean cross\-environment, and worst\-environment accuracy atη=10−2\\eta=10^\{\-2\}\. Negativeppimproves robustness relative to the best nonnegative setting in this high\-step\-size regime, but the optimum is finite and environment\-dependent\.Restricting the comparison to the best negative and best nonnegative exponent atη=10−2\\eta=10^\{\-2\}, the cross\-environment gains are small for strong\-positive and neutral sources, but reach 0\.35 percentage points for the weak\-positive source and 0\.87 points for the reversed source \([Table3](https://arxiv.org/html/2609.30271#S5.T3)\)\. The worst\-environment gain for the reversed source is 2\.48 points\. These are single\-seed differences and should be read as mechanism magnitudes, not confidence intervals\.

Table 3:Best negative versus best nonnegative exponent for mean cross\-environment accuracy atη=10−2\\eta=10^\{\-2\}\.
### 5\.4Lowerppreduces relative spurious and noise allocation

The checkpoint weights explain why the robust optimum moves left\.[Figure4](https://arxiv.org/html/2609.30271#S5.F4)plots the spurious\-to\-stable and noise\-to\-stable RMS ratios atη=10−2\\eta=10^\{\-2\}\. For the weak\-positive and reversed sources, both ratios increase substantially withpp\. At the weak\-positive cross optimump=−0\.20p=\-0\.20, the spurious\-to\-stable ratio is 0\.381, compared with 0\.435 at the best nonnegative cross settingp=0p=0\. The noise\-to\-stable ratio falls from 0\.066 to 0\.054\. For the reversed source, the corresponding spurious\-to\-stable ratio falls from 0\.447 to 0\.368\.

![Refer to caption](https://arxiv.org/html/2609.30271v1/x4.png)Figure 4:Learned feature allocation atη=10−2\\eta=10^\{\-2\}\. Lowerppgenerally reduces the RMS weight assigned to the spurious and pure\-noise blocks relative to the stable sparse block\. The neutral environment has little spurious signal and therefore shows a smaller, non\-monotone ratio\.The strong\-positive source is a useful boundary case\. Its spurious feature is highly predictive in the source and therefore remains heavily weighted across the grid\. Negativeppyields only a small cross\-environment improvement there\. This confirms that the effect is not a generic shrinkage of all spurious coordinates\. It depends on the competition between stable evidence, source correlation, training budget, and optimizer scaling\.

### 5\.5Margin decomposition under correlation reversal

Weight norms do not by themselves show whether a feature block helps or harms a target environment\. We therefore evaluate the signed margin contribution in[Equation19](https://arxiv.org/html/2609.30271#S4.E19)\.[Figure5](https://arxiv.org/html/2609.30271#S5.F5)considers models trained on positive\-correlation environments and tested on the reversed environment\. The stable sparse block contributes a positive margin, while the spurious block contributes a negative margin\. Asppincreases, the model learns more of both\. The stable contribution grows, but so does the harmful reversed spurious contribution\. The total margin is therefore maximized in a lower\-ppregion than source fitting alone would select\.

![Refer to caption](https://arxiv.org/html/2609.30271v1/x5.png)Figure 5:Mean signed margin decomposition on the reversed test environment for models trained on strong\- and weak\-positive sources atη=10−2\\eta=10^\{\-2\}\. The stable sparse contribution is positive and the spurious contribution is negative\. Lowerppreduces the harmful spurious margin, explaining why the cross\-environment optimum lies below the source\-validation optimum\.This decomposition also clarifies why negativeppdoes not simply mean “ignore sparse features\.” Both the stable and spurious blocks are sparse and use the same activation probabilities and signal magnitudes\. Their difference is persistence across environments\. The optimizer is not given this semantic distinction\. Yet under the high\-learning\-rate trajectory, lowerppproduces a solution with a smaller harmful spurious component relative to the stable component\.

### 5\.6Finite\-budget character of the result

In 92\.38% of the 420 runs, the best source\-validation checkpoint occurs at epoch 50, the end of the training budget\. The experiment therefore primarily characterizes*finite\-budget feature allocation*, not the unique asymptotic solution of every configuration\. This matters because changingppcan alter both the direction and speed with which different feature blocks are learned\. A longer training horizon may narrow some differences or move the optimum\. We retain this limitation explicitly because finite\-budget training is also the practical regime in which optimizer choice matters\.

## 6Discussion

### 6\.1Why can the robust optimum become negative?

For ordinary Adam\-likep\>0p\>0, coordinates with smallv^i\\widehat\{v\}\_\{i\}receive larger gains\. In a high\-dimensional sparse\-noise setting, many weakly activated or inconsistent coordinates can have small second moments\. A large global learning rate already supplies substantial step amplitude\. Continuing to amplify low\-vvcoordinates can then over\-allocate updates to weak, noisy, or source\-specific directions\. Decreasingppcounteracts this effect\. Oncep<0p<0, low\-vvcoordinates are suppressed and coordinates with more sustained second\-moment statistics receive relatively larger updates\.

This interpretation is consistent with the checkpoint decomposition, but it is not a universal theorem that stable features always have largervv\. The mapping depends on feature frequency, signal magnitude, label correlation, batch noise, and the evolving residual\. The correct conclusion is conditional:

> In this controlled sparse\-feature regime, the learning\-rate increase shifts the robust allocation toward lowerpp, and the shift crosses zero at sufficiently largeη\\eta\.

### 6\.2Learning rate andppare not interchangeable

The approximate compensation relation in[Equation8](https://arxiv.org/html/2609.30271#S3.E8)explains why the optimum follows a diagonal path in the grid\. However,η\\etaandppremain distinct controls\. The learning rate scales the entire current update and changes the next parameter state\. The exponent changes coordinate ratios immediately through[Equation5](https://arxiv.org/html/2609.30271#S3.E5)\. Two configurations can have similar aggregate performance while assigning different weights to stable, spurious, and noise blocks\. Therefore,ppcannot be reduced to a reparameterized learning rate\.

### 6\.3Optimizer model selection under distribution shift

The source\-validation optimum remains positive even when the robust optimum is negative\. This finding creates a practical obstacle: target environments are unavailable in genuine domain generalization\. The current experiment usespcross⋆p\_\{\\mathrm\{cross\}\}^\{\\star\}andpworst⋆p\_\{\\mathrm\{worst\}\}^\{\\star\}only as diagnostic oracles\. A deployable method would require a source\-observable proxy for harmful allocation, perhaps based on gradient stability, moment concentration, feature perturbations, or multiple source splits\. Developing such a criterion is outside the present study\.

### 6\.4Relation to partially adaptive and root\-free methods

Padam restrictsppto the interval between momentum\-like and Adam\-like updates and uses it to moderate adaptivity\(Chen et al\.,[2020](https://arxiv.org/html/2609.30271#bib.bib2)\)\. Root\-free methods move in the opposite positive direction and can improve optimization and generalization in other architectures\(Lin et al\.,[2024](https://arxiv.org/html/2609.30271#bib.bib7)\)\. Our result does not contradict either line\. It shows that the useful exponent is task\-, learning\-rate\-, and criterion\-dependent\. The negative regime emerges here because the objective is cross\-environment robustness under a large global step and a sparse high\-dimensional feature competition\.

## 7Limitations

The main limitations are:

1. 1\.Single seed\.The paired construction removes substantial sampling variation, but it does not replace a multi\-seed estimate of optimizer variability\.
2. 2\.Linear model and synthetic data\.The feature blocks are directly interpretable, which is essential for mechanism analysis, but the result is not yet a claim about deep representations or natural datasets\.
3. 3\.Finite training horizon\.Most checkpoints are selected at the final epoch\. Additional long\-horizon and matched\-source\-loss experiments are needed to separate transient allocation from asymptotic implicit bias\.
4. 4\.Oracle robust selection\.Cross and worst\-environment criteria use target test environments only for retrospective analysis\. No source\-only rule for selecting negativeppis proposed\.
5. 5\.One optimizer family\.The experiment uses Adam\-style first and second moments with a variable exponent\. Momentum, moment decay, clipping, weight decay, and schedules may change the coupling\.

These limitations define a direct validation path: repeat a reduced grid across several seeds, extend representative runs to convergence, match source loss across configurations, and test whether source\-observable allocation statistics predict the robust exponent\.

## 8Conclusion

A continuous preconditioning exponent and a global learning rate form a coupled optimization system\. In a controlled four\-environment sparse\-feature problem, the exponent that maximizes cross\-environment generalization decreases nearly linearly with the logarithm of the learning rate\. At high learning rates, the robust optimum becomes negative even though source validation continues to prefer positive exponents\. Negativeppreduces relative allocation to spurious and noise features and lowers the harmful spurious margin under correlation reversal\. The main lesson is not that negative preconditioning is universally superior\. It is that the sign and magnitude of the useful exponent depend on the step\-size regime and on whether optimization is judged by source fit or environmental robustness\.

## Appendix AComplete Optimal\-Exponent Table

Table 4:Complete source\-validation, mean\-cross, and worst\-environment optima for every learning rate\. Accuracy values correspond to the criterion\-specific robust optimum\.
## Appendix BSource\-Selected Cross\-Environment Matrix

For completeness,[Table5](https://arxiv.org/html/2609.30271#A2.T5)reports test accuracy when bothppandη\\etaare selected only by source validation loss\. Rows are source environments and columns are test environments\. This table is not the oracle robust result; it shows the performance of the deployable source\-selection protocol used for checkpoint and hyperparameter selection in the original experiment\.

Table 5:Cross\-environment accuracy \(%\) after selectingppandη\\etaby source validation loss\.
## Appendix CReproducibility and Data Files

The accompanying source package contains:

- •the original training script;
- •the data\-preparation and checkpoint\-decomposition script;
- •the complete manifest of 420 runs;
- •all 1680 cross\-environment evaluations;
- •enriched run\-level metrics with feature\-block weight statistics and margin decompositions;
- •compact CSV files used for each figure and table;
- •vector PDF figures generated from those CSV files\.

The exact experiment defaults are stored indata/fixed\_config\.json\. The main analysis files areoptimal\_p\_by\_learning\_rate\.csv,optimal\_p\_linear\_fits\.csv,high\_lr\_p\_curves\.csv, andrun\_metrics\_with\_feature\_allocation\.csv\.

## References

- Arjovsky et al\. \[2019\]Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez\-Paz\.Invariant risk minimization\.*arXiv preprint arXiv:1907\.02893*, 2019\.
- Chen et al\. \[2020\]Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu\.Closing the generalization gap of adaptive gradient methods in training deep neural networks\.*Proceedings of the Twenty\-Ninth International Joint Conference on Artificial Intelligence*, pages 3267–3275, 2020\.
- Duchi et al\. \[2011\]John Duchi, Elad Hazan, and Yoram Singer\.Adaptive subgradient methods for online learning and stochastic optimization\.In*Journal of Machine Learning Research*, volume 12, pages 2121–2159, 2011\.
- Gulrajani and Lopez\-Paz \[2021\]Ishaan Gulrajani and David Lopez\-Paz\.In search of lost domain generalization\.In*International Conference on Learning Representations*, 2021\.
- Kingma and Ba \[2015\]Diederik P\. Kingma and Jimmy Ba\.Adam: A method for stochastic optimization\.*International Conference on Learning Representations*, 2015\.
- Li et al\. \[2019\]Yuanzhi Li, Colin Wei, and Tengyu Ma\.Towards explaining the regularization effect of initial large learning rate in training neural networks\.In*Advances in Neural Information Processing Systems*, volume 32, 2019\.
- Lin et al\. \[2024\]Wu Lin, Felix Dangel, Runa Eschenhagen, Juhan Bae, Richard E\. Turner, and Alireza Makhzani\.Can we remove the square\-root in adaptive gradient methods? a second\-order perspective\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235, pages 29949–29973, 2024\.
- Lu et al\. \[2024\]Miao Lu, Beining Wu, Xiaodong Yang, and Difan Zou\.Benign oscillation of stochastic gradient descent with large learning rate\.In*International Conference on Learning Representations*, 2024\.
- Qiu et al\. \[2024\]Guanwen Qiu, Da Kuang, and Surbhi Goel\.Complexity matters: Feature learning in the presence of spurious correlations\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235, pages 41658–41697, 2024\.
- Sagawa et al\. \[2019\]Shiori Sagawa, Pang Wei Koh, Tatsunori B\. Hashimoto, and Percy Liang\.Distributionally robust neural networks for group shifts: On the importance of regularization for worst\-case generalization\.*arXiv preprint arXiv:1911\.08731*, 2019\.
- Wilson et al\. \[2017\]Ashia C\. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht\.The marginal value of adaptive gradient methods in machine learning\.In*Advances in Neural Information Processing Systems*, volume 30, 2017\.
- Zhang et al\. \[2026\]Gongyue Zhang, Donghan Liu, Weihong Ren, Yixuan Sheng, Zhiyong Wang, and Honghai Liu\.Information allocation dynamics in neural network optimization\.*arXiv preprint arXiv:2607\.07156*, 2026\.