LionVote: Per-Layer Learning Rate Adaptation for Lion

# Per-Layer Learning Rate Adaptation for Lion
Source: [https://arxiv.org/html/2607.09266](https://arxiv.org/html/2607.09266)
###### Abstract

Per\-layer diagnostics reveal that, at the prescribed learning rate, Lion’s effective scale is2\.62\.6–2\.8×2\.8\{\\times\}too high for attention and MLP parameters and∼2×\{\\sim\}2\{\\times\}too high for normalisation layers on ViT\-Tiny/CIFAR\-100; this32%32\\%cross\-layer\-type disparity cannot be reproduced by a single global rate\. The measurement comes from LionVote, a per\-layer learning rate mechanism in which each parameter tensor maintains a compound level, a persistent integer updated everyccepochs by two diagnostics \(gradient direction stability and momentum health\) resolved by a validation loss tiebreaker\. Voting thresholds derive from geometric identities, the EMA time constant, and a noise\-floor estimate; cadence is bounded structurally and selected by ablation\. On ViT\-Tiny/CIFAR\-100, LionVote achieves69\.7%69\.7\\%top\-1 accuracy vs\. Lion’s69\.0%69\.0\\%\(p<0\.02p<0\.02, Welch’stt\-test\) and AdamW’s68\.8%68\.8\\%\. Per\-layer adaptation value depends on both architectural heterogeneity and task; on uniform CNN architectures tuned SGD with cosine annealing remains dominant, and on ViT architectures gains are task\-dependent\.

## 1Introduction

Zhao et al\.\(Zhao et al\.,[2025](https://arxiv.org/html/2607.09266#bib.bib25)\)show that applying per\-layer adaptive preconditioning only to the last layer and LayerNorm parameters recovers most of Adam’s advantage over SGD on autoregressive language models; this is evidence that layer types have different optimisation characteristics\. Their analysis does not prescribe how much each type should diverge from a base rate, nor propose a mechanism to determine this during training\.

Existing adaptive methods operate per\-coordinate or per\-layer but statelessly, and schedule\-free approaches\(Defazio et al\.,[2024](https://arxiv.org/html/2607.09266#bib.bib6)\)maintain a single global rate\. No existing method combines stateful layer\-level adaptation with data\-driven rate adjustment \(§[2\.4](https://arxiv.org/html/2607.09266#S2.SS4)\)\.

LionVote is a per\-layer learning rate mechanism for Lion\(Chen et al\.,[2023](https://arxiv.org/html/2607.09266#bib.bib3)\)\. Lion’s sign operation discards gradient magnitude, preventing directionally unstable layers from compensating through larger updates; this makes per\-layer miscalibration more consequential for sign\-based methods than for second\-moment methods like Adam \(§[5\.1](https://arxiv.org/html/2607.09266#S5.SS1)\)\. Each parameter tensor maintains a compound level \(a persistent integer, §[3\.1](https://arxiv.org/html/2607.09266#S3.SS1)\) that modulates the base learning rate exponentially\. Everyccepochs, two per\-layer diagnostics \(gradient direction stability and momentum health, with derived thresholds; Appendix[A\.1](https://arxiv.org/html/2607.09266#A1.SS1)–[A\.2](https://arxiv.org/html/2607.09266#A1.SS2)\) vote on whether to increase, decrease, or maintain each layer’s rate\. When the votes conflict, a validation loss tiebreaker resolves the decision\. The mechanism adds one extra accumulation per parameter per batch; voting runs once everyccepochs\. With all votes disabled, compound levels decay to zero and LionVote reduces to standard Lion\.

We evaluate LionVote on WideResNet and ViT\-Tiny across CIFAR\-10 and CIFAR\-100, with 8 seeds per configuration\. The contributions are:

1. 1\.A per\-layer adaptive mechanism for a sign\-based optimizer whose voting thresholds derive from geometric identities and the EMA time constant; cadence and structural parameters \(the exponent divisord=2d\{=\}2, maximum levelL=4L\{=\}4\) are bounded rather than uniquely determined \(Appendix[A\.1](https://arxiv.org/html/2607.09266#A1.SS1)–[A\.6](https://arxiv.org/html/2607.09266#A1.SS6)\)\. The threshold derivation methodology is reusable for principled design of per\-layer mechanisms in other optimizers\.
2. 2\.A quantified finding about Lion: its effective scale is2\.62\.6–2\.8×2\.8\{\\times\}too high for attention and MLP parameters and∼2×\{\\sim\}2\{\\times\}too high for normalisation parameters on ViT\-Tiny/CIFAR\-100, measured via compound level trajectories across 8 seeds\. Normalisation layers receive32%32\\%higher effective scale than attention layers \(§[5\.1](https://arxiv.org/html/2607.09266#S5.SS1)\)\. Because the compound multiplier scales both the sign update and decoupled weight decay, this measures joint LR\+WD miscalibration \(§[5\.1](https://arxiv.org/html/2607.09266#S5.SS1)\)\.
3. 3\.Evidence that per\-layer adaptation value depends on both architectural heterogeneity and task\. On ViT\-Tiny, compound level spread between layer types is2\.8×2\.8\{\\times\}larger than on WideResNet, and LionVote at cadence 8 achieves the best accuracy on CIFAR\-100 \(69\.7%69\.7\\%\)\. The same architecture and comparable spread on CIFAR\-10 yields a significant loss vs\. Lion \(−0\.88\-0\.88pp,p<0\.001p<0\.001\)\. On WideResNet, tuned SGD with cosine annealing remains dominant \(§[5\.2](https://arxiv.org/html/2607.09266#S5.SS2)\)\.

## 2Related Work

Deep networks are not monolithic\. Residual networks\(He et al\.,[2016](https://arxiv.org/html/2607.09266#bib.bib9)\), wide residual networks\(Zagoruyko & Komodakis,[2016](https://arxiv.org/html/2607.09266#bib.bib24)\), and vision transformers\(Dosovitskiy et al\.,[2021](https://arxiv.org/html/2607.09266#bib.bib7)\)differ qualitatively in how gradients flow and how representations evolve across depth\. Standard schedules such as cosine annealing\(Loshchilov & Hutter,[2017](https://arxiv.org/html/2607.09266#bib.bib15)\)and step decay apply a single learning rate trajectory to every parameter, regardless of each layer’s convergence state\.

### 2\.1Per\-Parameter Adaptation

Per\-coordinate adaptation progressed from AdaGrad’s\(Duchi et al\.,[2011](https://arxiv.org/html/2607.09266#bib.bib8)\)accumulated squared gradients through Adam\(Kingma & Ba,[2015](https://arxiv.org/html/2607.09266#bib.bib13)\)and decoupled weight decay\(Loshchilov & Hutter,[2019](https://arxiv.org/html/2607.09266#bib.bib16)\); all adapt per\-coordinate, not per\-layer\. Sign\-based methods \(signSGD\(Bernstein et al\.,[2018](https://arxiv.org/html/2607.09266#bib.bib2)\), Lion\(Chen et al\.,[2023](https://arxiv.org/html/2607.09266#bib.bib3)\)\) reduce the update to a sign of a momentum–gradient interpolation, matching Adam at lower memory cost under a global schedule\.

Cautious Optimizers\(Liang et al\.,[2026](https://arxiv.org/html/2607.09266#bib.bib14)\)mask update coordinates by sign agreement, operating per\-coordinate per\-batch without per\-layer history\.

### 2\.2Per\-Layer Adaptation

Howard and Ruder\(Howard & Ruder,[2018](https://arxiv.org/html/2607.09266#bib.bib11)\)show that assigning progressively smaller learning rates to lower layers outperforms a single global rate when adapting pretrained language models to new tasks\. LARS\(You et al\.,[2017](https://arxiv.org/html/2607.09266#bib.bib22)\)introduced per\-layer rate scaling by multiplying each layer’s learning rate by the ratio of its weight norm to its gradient norm, working well in large\-batch settings\. LAMB\(You et al\.,[2020](https://arxiv.org/html/2607.09266#bib.bib23)\)applies the same trust\-ratio idea to Adam, enabling large\-batch training where standard Adam degrades\. LARS and LAMB are stateless: a layer that has been stable for many epochs is treated identically to one emerging from a noisy phase\.

Gradient direction stability—whether a layer’s gradient aligns consistently across epochs—is a complementary signal to the weight\-to\-gradient ratio\. To our knowledge, using alignment between successive gradient estimates to modulate step size has not been applied at the layer level\.

Muon\(Jordan et al\.,[2024](https://arxiv.org/html/2607.09266#bib.bib12)\)applies orthogonalised Newton–Schulz updates to hidden layers but hardcodes which layers receive which treatment\.

Zhao et al\.\(Zhao et al\.,[2025](https://arxiv.org/html/2607.09266#bib.bib25)\)show that restricting per\-layer adaptive preconditioning to the last layer and LayerNorm parameters suffices to capture most of the accuracy benefit on autoregressive language models, without quantifying per\-type divergence or proposing a discovery mechanism\.

Hao et al\.\(Hao et al\.,[2025](https://arxiv.org/html/2607.09266#bib.bib10)\)\(LANTON\) assign noise\-adaptive per\-layer learning rates to geometry\-aware optimizers such as Muon, estimating gradient variance in the dual norm per layer at each step\. The adaptation is instantaneous: it does not accumulate cross\-epoch evidence or maintain persistent per\-layer state\.

### 2\.3Data\-Driven Schedule Replacement

AutoDrop\(Wang et al\.,[2024](https://arxiv.org/html/2607.09266#bib.bib21)\)monitors angular velocity to trigger rate reductions automatically\. D\-Adaptation\(Defazio & Mishchenko,[2023](https://arxiv.org/html/2607.09266#bib.bib5)\)derives the learning rate from an online lower bound on the distance to a minimizer\. Orvieto and Xiao\(Orvieto & Xiao,[2024](https://arxiv.org/html/2607.09266#bib.bib18)\)exploit a Gauss\-Newton reformulation to derive stepsizes that warm up, peak, and decay without a schedule\. Defazio et al\.\(Defazio et al\.,[2024](https://arxiv.org/html/2607.09266#bib.bib6)\)replace momentum with iterate interpolation and averaging, matching hand\-tuned cosine schedules with no stopping time\.

All of these methods maintain a single global rate for the entire parameter vector\.

### 2\.4The Gap

Schedule\-free methods establish that data\-driven adaptation can replace hand\-designed global schedules; layer\-type\-aware methods establish that layers benefit from heterogeneous treatment\. No existing method bridges the two\. LARS and LAMB adapt per\-layer but statelessly \(§[2\.2](https://arxiv.org/html/2607.09266#S2.SS2)\)\. Schedule\-free and D\-Adaptation methods accumulate history but apply a single rate globally\. Muon differentiates layer types but hardcodes which layers receive which treatment\. LANTON adapts per\-layer rates instantaneously \(§[2\.2](https://arxiv.org/html/2607.09266#S2.SS2)\) but does not accumulate cross\-epoch evidence\. Meta\-learned optimizers\(Andrychowicz et al\.,[2016](https://arxiv.org/html/2607.09266#bib.bib1)\)can in principle discover per\-layer policies, but sacrifice interpretability and require expensive meta\-training on proxy tasks\.

## 3Proposed Approach

LionVote is a per\-layer learning rate mechanism for the Lion optimizer\(Chen et al\.,[2023](https://arxiv.org/html/2607.09266#bib.bib3)\), implemented as a PyTorchOptimizersubclass\. It makes no structural modification to Lion’s update rule\. Each parameter tensor receives an effective learning rateαi\\alpha\_\{i\}adjusted by a persistent integer*compound level*that is updated everyccepochs through a two\-vote system with a global tiebreaker\.

### 3\.1Per\-Layer Learning Rate

Each parameteriihas a compound levelsi∈\{−L,…,0,…,\+L\}s\_\{i\}\\in\\\{\-L,\\ldots,0,\\ldots,\+L\\\}\(defaultL=4L=4\) that modulates the base learning rate:

αi=lr⋅exp⁡\(si⋅β1/2\),\\alpha\_\{i\}\\;=\\;\\mathrm\{lr\}\\cdot\\exp\\\!\\bigl\(s\_\{i\}\\cdot\\beta\_\{1\}/2\\bigr\),\(1\)wherelr\\mathrm\{lr\}is the global base learning rate \(which may be updated by an external scheduler\) andβ1\\beta\_\{1\}is Lion’s sign\-interpolation coefficient\. All compound levels are initialised to zero\. At the defaultβ1=0\.9\\beta\_\{1\}=0\.9, each unit step insis\_\{i\}corresponds to a×1\.57\{\\times\}1\.57multiplier, and the full range\[−4,\+4\]\[\-4,\+4\]spans from×0\.165\{\\times\}0\.165to×6\.05\{\\times\}6\.05relative to the base rate\. The derivation of the exponentβ1/2\\beta\_\{1\}/2and the boundL=4L=4are given in Appendix[A\.4](https://arxiv.org/html/2607.09266#A1.SS4)and[A\.5](https://arxiv.org/html/2607.09266#A1.SS5)\.

### 3\.2Batch Update

On every training batch, LionVote performs the standard Lion update withαi\\alpha\_\{i\}in place of the global learning rate: decoupled weight decay, followed by a sign updatesign⁡\(β1​mi\+\(1−β1\)​gi\)\\operatorname\{sign\}\(\\beta\_\{1\}m\_\{i\}\+\(1\-\\beta\_\{1\}\)g\_\{i\}\), followed by aβ2\\beta\_\{2\}\-EMA momentum update \(Algorithm[1](https://arxiv.org/html/2607.09266#alg1)\)\. The raw gradient is accumulated for epoch\-level voting\.

### 3\.3Epoch Step and Voting

At the end of every epoch, LionVote computes the epoch\-mean gradientg¯i=ai/N\\bar\{g\}\_\{i\}=a\_\{i\}/Nfor each parameter, resets the accumulators, and recomputesαi\\alpha\_\{i\}from the currentlr\\mathrm\{lr\}so that per\-layer rates track any external scheduler\. On*voting epochs*—epochs whereepochmodc=0\\mathrm\{epoch\}\\bmod c=0, withccthe voting cadence—the system additionally evaluates two per\-layer diagnostics and one global tiebreaker\.

##### Vote 1: Gradient Direction Stability\.

alignmenti=g¯i\(curr\)⋅g¯i\(prev\)‖g¯i\(curr\)‖​‖g¯i\(prev\)‖\+ϵ\.\\mathrm\{alignment\}\_\{i\}\\;=\\;\\frac\{\\bar\{g\}\_\{i\}^\{\(\\mathrm\{curr\}\)\}\\cdot\\bar\{g\}\_\{i\}^\{\(\\mathrm\{prev\}\)\}\}\{\\\|\\bar\{g\}\_\{i\}^\{\(\\mathrm\{curr\}\)\}\\\|\\,\\\|\\bar\{g\}\_\{i\}^\{\(\\mathrm\{prev\}\)\}\\\|\+\\epsilon\}\.The vote is\+1\+1ifalignmenti\>0\.5\\mathrm\{alignment\}\_\{i\}\>0\.5,−1\-1ifalignmenti<0\\mathrm\{alignment\}\_\{i\}<0, and0otherwise \(ϵ=10−8\\epsilon=10^\{\-8\}throughout\)\. The lower threshold follows directly from the definition of cosine \(negative iff angle exceedsπ/2\\pi/2\); the upper threshold corresponds to a two\-thirds supermajority of coordinate sign pairs agreeing under coordinate isotropy \(Appendix[A\.1](https://arxiv.org/html/2607.09266#A1.SS1)\)\. Vote 1 uses epoch\-mean gradients rather than momentum to avoid self\-contamination from the EMA\.

##### Vote 2: Momentum Health\.

ri=‖mi‖‖g¯i\(curr\)‖\+ϵ\.r\_\{i\}\\;=\\;\\frac\{\\\|m\_\{i\}\\\|\}\{\\\|\\bar\{g\}\_\{i\}^\{\(\\mathrm\{curr\}\)\}\\\|\+\\epsilon\}\.The vote is−1\-1ifri\>er\_\{i\}\>e\(momentum norm exceeds gradient norm by more than a factor ofee\),\+1\+1ifri<1/er\_\{i\}<1/e\(gradient norm exceeds momentum norm by more than a factor ofee\), and0otherwise\. Both thresholds derive from the EMA’s intrinsic time constant:β2τ=e−1\\beta\_\{2\}^\{\\tau\}=e^\{\-1\}for anyβ2∈\(0,1\)\\beta\_\{2\}\\in\(0,1\)\(Appendix[A\.2](https://arxiv.org/html/2607.09266#A1.SS2)\)\.

##### Tiebreaker: Validation Loss\.

The tiebreaker is a global scalar that fires only when the two local votes sum to zero \(both abstain or conflict\):

δ=L¯prev−L¯curr\|L¯prev\|\+ϵ\.\\delta\\;=\\;\\frac\{\\bar\{L\}\_\{\\mathrm\{prev\}\}\-\\bar\{L\}\_\{\\mathrm\{curr\}\}\}\{\|\\bar\{L\}\_\{\\mathrm\{prev\}\}\|\+\\epsilon\}\.It returns\+1\+1ifδ\>0\.01\\delta\>0\.01,−1\-1ifδ<−0\.01\\delta<\-0\.01, and0otherwise \(ϵ=10−8\\epsilon=10^\{\-8\}as in Votes 1 and 2\)\. The1%1\\%threshold matches the noise floor of cross\-entropy loss estimation for typical validation set sizes \(Appendix[A\.3](https://arxiv.org/html/2607.09266#A1.SS3)\)\. Under Lion on ViT\-Tiny/CIFAR\-100, the tiebreaker condition is satisfied for∼59%\{\\sim\}59\\%of per\-parameter voting decisions \(Appendix[B\.6](https://arxiv.org/html/2607.09266#A2.SS6)\), so the tiebreaker is not a tie\-breaking edge case but the dominant signal path; Vote 1 or Vote 2 alone determine the outcome for the remaining∼41%\{\\sim\}41\\%\.

##### Vote Resolution\.

Vote 1 monitors directional stability; Vote 2 monitors the momentum\-gradient magnitude ratio\. The final vote is resolved per the table in Algorithm[1](https://arxiv.org/html/2607.09266#alg1)\.

### 3\.4Compound Level Update

The compound levelsis\_\{i\}is updated via the*asymmetric*rule: vote=\+1=\+1resetssis\_\{i\}to0if negative, else increments \(capped at\+L\+L\); vote=−1=\-1resets to0if positive, else decrements \(capped at−L\-L\); vote=0=0decays toward zero\. A single opposing vote revokes the accumulated level entirely\.

### 3\.5Algorithm

Algorithm 1LionVote0:

θ\\theta, lr,

β1,β2,λ\\beta\_\{1\},\\beta\_\{2\},\\lambda, cadence

cc, max level

LL
1:Init

mi,ai,si←0m\_\{i\},a\_\{i\},s\_\{i\}\\leftarrow 0;

αi←lr\\alpha\_\{i\}\\leftarrow\\mathrm\{lr\};

N←0N\\leftarrow 0
2:forepoch

=1,2,…=1,2,\\ldotsdo

3:foreach batch with gradient

gig\_\{i\}do

4:

θi←θi−αi​λ​θi\\theta\_\{i\}\\leftarrow\\theta\_\{i\}\-\\alpha\_\{i\}\\lambda\\theta\_\{i\}⊳\\trianglerightdecay

5:

θi←θi−αi​sign⁡\(β1​mi\+\(1−β1\)​gi\)\\theta\_\{i\}\\leftarrow\\theta\_\{i\}\-\\alpha\_\{i\}\\operatorname\{sign\}\(\\beta\_\{1\}m\_\{i\}\+\(1\{\-\}\\beta\_\{1\}\)g\_\{i\}\)⊳\\trianglerightLion

6:

mi←β2​mi\+\(1−β2\)​gim\_\{i\}\\leftarrow\\beta\_\{2\}m\_\{i\}\+\(1\{\-\}\\beta\_\{2\}\)g\_\{i\}
7:

ai←ai\+gia\_\{i\}\\leftarrow a\_\{i\}\+g\_\{i\};

N←N\+1N\\leftarrow N\{\+\}1
8:endfor

9:

g¯i←ai/N\\bar\{g\}\_\{i\}\\leftarrow a\_\{i\}/N
10:ifep

modc=0\\bmod c=0g¯\(prev\)\\bar\{g\}^\{\(\\mathrm\{prev\}\)\}existsthen

11:Tiebreaker from

L¯curr,L¯prev\\bar\{L\}\_\{\\mathrm\{curr\}\},\\bar\{L\}\_\{\\mathrm\{prev\}\}
12:foreach parameter

iido

13:

v1←v\_\{1\}\\leftarrowVote 1;

v2←v\_\{2\}\\leftarrowVote 2

14:Resolve

→v\\rightarrow v; update

sis\_\{i\}
15:endfor

16:endif

17:ifep

modc=0\\bmod c=0orep

=1=1then

18:Snapshot

g¯i\(prev\),L¯prev\\bar\{g\}\_\{i\}^\{\(\\mathrm\{prev\}\)\},\\bar\{L\}\_\{\\mathrm\{prev\}\}
19:endif

20:

αi←lr⋅exp⁡\(si⋅β1/2\)\\alpha\_\{i\}\\leftarrow\\mathrm\{lr\}\\cdot\\exp\(s\_\{i\}\\cdot\\beta\_\{1\}/2\)
21:

ai,N←0a\_\{i\},N\\leftarrow 0
22:endfor

Vote Resolution V1V2Result±1\\pm 1sameagree±1\\pm 10V10±1\\pm 1V2elsetiebreaker

Computational cost is detailed in Appendix[B\.18](https://arxiv.org/html/2607.09266#A2.SS18)\.

## 4Experiments

We evaluate LionVote against AdamW, SGD with Nesterov momentum, and Lion on four model–dataset configurations\. All experiments use 8 random seeds\. We report top\-1 validation accuracy \(mean±\\pmstd\) and epochs to reach intermediate accuracy thresholds\.

### 4\.1Experimental Setup

##### Models and datasets\.

WideResNet\-28\-10\(Zagoruyko & Komodakis,[2016](https://arxiv.org/html/2607.09266#bib.bib24)\)on CIFAR\-10 \(∼36\.5\{\\sim\}36\.5M parameters, 200 epochs, batch size 128\) and WideResNet\-40\-10 on CIFAR\-100 \(∼55\.8\{\\sim\}55\.8M parameters, 200 epochs, batch size 128\)\. ViT\-Tiny\(Dosovitskiy et al\.,[2021](https://arxiv.org/html/2607.09266#bib.bib7)\)\(patch size 4, embedding dimension 192, 12 blocks, 3 heads,∼5\.7\{\\sim\}5\.7M parameters\) on CIFAR\-10 and CIFAR\-100 \(300 epochs, batch size 128\)\. WideResNet uses pre\-activation blocks with BatchNorm and dropout 0\.3\. ViT\-Tiny uses learnable positional embeddings, stochastic depth \(linearly increasing to 0\.1\), and a CLS token classification head\.

##### Augmentation\.

CNN configurations: RandomCrop, HorizontalFlip, and AutoAugment \(CIFAR\-10 policy\) for CIFAR\-100\. ViT configurations: RandAugment\(2,9\)\(2,9\), RandomErasing\(p=0\.25\)\(p\{=\}0\.25\), Mixup\(α=0\.8\)\(\\alpha\{=\}0\.8\)/CutMix\(α=1\.0\)\(\\alpha\{=\}1\.0\)\(50/50 per batch\), and label smoothing 0\.1 with soft cross\-entropy\.

##### Optimizers\.

AdamW \(lr=10−3\{=\}10^\{\-3\}\), SGD \(lr=0\.1\{=\}0\.1, momentum 0\.9, Nesterov\), Lion \(lr=10−4\{=\}10^\{\-4\},β1=0\.9\\beta\_\{1\}\{=\}0\.9,β2=0\.99\\beta\_\{2\}\{=\}0\.99\), and LionVote \(same as Lion, plus voting\)\. Weight decay follows the Lion paper\(Chen et al\.,[2023](https://arxiv.org/html/2607.09266#bib.bib3)\): Lion and LionVote use33–10×10\{\\times\}larger WD than AdamW \(e\.g\.5×10−15\{\\times\}10^\{\-1\}vs\.5×10−25\{\\times\}10^\{\-2\}for ViT/CIFAR\-10\)\. Full weight decay values per configuration are in Appendix[B](https://arxiv.org/html/2607.09266#A2)\. LARS and LAMB are omitted: both are stateless norm\-ratio methods designed for large\-batch distributed training \(§[2\.2](https://arxiv.org/html/2607.09266#S2.SS2)\), a different regime from the single\-GPU setting studied here\.

##### Schedulers\.

WRN configurations: cosine annealing\. ViT configurations: 5\-epoch linear warmup followed by cosine annealing\. LionVote is additionally tested without any schedule\.

##### Voting hyperparameters\.

Cadencec∈\{4,6,8\}c\\in\\\{4,6,8\\\}, max levelL=4L\{=\}4, asymmetric update rule\. All voting thresholds are derived \(Appendix[A\.1](https://arxiv.org/html/2607.09266#A1.SS1)–[A\.3](https://arxiv.org/html/2607.09266#A1.SS3)\); cadence is selected by ablation within a structurally bounded range \(Appendix[A\.6](https://arxiv.org/html/2607.09266#A1.SS6)\)\.

### 4\.2Main Results

Table 1:Best top\-1 validation accuracy \(%, mean±\\pmstd, 8 seeds\)\. WRN configurations use cosine scheduling; ViT configurations use cosine with 5\-epoch warmup\. Best result per row inbold\.SGD with cosine annealing dominates the WRN configurations \(Table[1](https://arxiv.org/html/2607.09266#S4.T1)\), outperforming the next\-best method by2\.32\.3pp on CIFAR\-10 and5\.35\.3pp on CIFAR\-100, consistent with the established difficulty of beating tuned SGD on uniform CNN architectures\(Schmidt et al\.,[2021](https://arxiv.org/html/2607.09266#bib.bib19)\)\. On ViTs, Lion outperforms both SGD and AdamW\. LionVote at cadence 8 achieves the best result on ViT\-Tiny/CIFAR\-100 \(69\.71%69\.71\\%\) and ties AdamW on ViT\-Tiny/CIFAR\-10 \(91\.52%91\.52\\%\)\. The ViT\-Tiny/CIFAR\-100 improvement over Lion is statistically significant \(Welch’stt\-test,p=0\.017p=0\.017, 8 seeds\); cadence is selected by ablation fromc∈\{4,6,8\}c\\in\\\{4,6,8\\\}\(§[5\.3](https://arxiv.org/html/2607.09266#S5.SS3)\)\. On ViT\-Tiny/CIFAR\-10, Lion’s advantage over LionVote at cadence 8 \(92\.40%92\.40\\%vs\.91\.52%91\.52\\%\) is also significant \(p<0\.001p<0\.001\)\. At cadence 8, the differences between Lion and LionVote are not significant on either WRN configuration \(p\>0\.15p\>0\.15\)\. At cadence 4, LionVote significantly underperforms Lion on WRN\-28\-10/CIFAR\-10 \(−0\.57\-0\.57pp,p=0\.011p=0\.011\)\.

![Refer to caption](https://arxiv.org/html/2607.09266v1/fig/fig1_convergence.png)Figure 1:Validation accuracy vs\. epoch for all four configurations \(mean±1\\pm 1std over 8 seeds\)\. Panels \(left to right, top to bottom\): WRN\-28\-10/C10, WRN\-40\-10/C100, ViT\-Tiny/C10, ViT\-Tiny/C100\. LionVote at cadence 8 converges fastest on ViT\-Tiny/CIFAR\-100, reaching the68%68\\%threshold∼80\{\\sim\}80epochs before Lion\. On WRN configurations \(top row\), SGD with cosine annealing achieves the highest final accuracy despite slower initial convergence\.LionVote at cadence 8 reaches the68%68\\%threshold∼80\{\\sim\}80epochs before Lion on ViT\-Tiny/CIFAR\-100 \(Figure[1](https://arxiv.org/html/2607.09266#S4.F1); full convergence thresholds in Appendix, Table[8](https://arxiv.org/html/2607.09266#A2.T8)\)\. Validation loss curves and generalisation gap analysis are in Appendix[B\.13](https://arxiv.org/html/2607.09266#A2.SS13)and[B\.14](https://arxiv.org/html/2607.09266#A2.SS14)\.

### 4\.3Ablation Study

Table 2:Ablation on WRN\-28\-10/CIFAR\-10 and ViT\-Tiny/CIFAR\-100 \(%, mean±\\pmstd, 8 seeds\)\. Lion \(no voting\) included for reference\. Best numerical result per column inbold; the asymmetric update rule \(c8\) is the default configuration\. Cadence and max\-level sensitivity in Appendix[B\.3](https://arxiv.org/html/2607.09266#A2.SS3)\.*Vote 2 monitors momentum health and finds it stable\.*The v2only variant matches Lion on both configurations \(93\.46%93\.46\\%WRN/C10,68\.93%68\.93\\%ViT/C100\); the momentum\-to\-gradient ratio rarely exits the\[1/e,e\]\[1/e,e\]dead zone \(Appendix[B\.5](https://arxiv.org/html/2607.09266#A2.SS5)\)\.

*The dominant component is configuration\-dependent\.*On WRN/C10, the v1only variant \(which disables both Vote 2 and the tiebreaker\) degrades to92\.22%92\.22\\%, a0\.920\.92pp gap from the full system \(93\.14%93\.14\\%\)\. Of this gap,0\.590\.59pp is attributable to the tiebreaker \(Welchp=0\.006p=0\.006\); the remaining0\.330\.33pp attributed to Vote 2 is not significant \(p=0\.23p=0\.23\), consistent with its near\-zero firing rate\. On ViT/C100, the pattern reverses: v1only achieves68\.57%±0\.4968\.57\\%\\pm 0\.49,*below*Lion \(−0\.38\-0\.38pp,p=0\.12p=0\.12\); notie achieves68\.64%±0\.5268\.64\\%\\pm 0\.52\(−0\.31\-0\.31pp vs\. Lion,p=0\.22p=0\.22\), while the full system at cadence 8 reaches69\.71%69\.71\\%\(\+0\.76\+0\.76pp,p=0\.017p=0\.017\)\. The entire gain on ViT/C100 is attributable to the tiebreaker: LionVote c8 exceeds notie by\+1\.07\+1\.07pp \(p=0\.003p=0\.003\)\. Vote 1 alone and Vote 1\+2 without the tiebreaker both fall below Lion \(neither difference significant\)\. LionVote thus combines per\-layer gradient diagnostics with a global validation loss signal: Vote 1 provides differentiated per\-layer votes \(negative56%56\\%of the time for attention vs\.30%30\\%for normalisation\), while the tiebreaker applies the same direction to all undecided layers whenever Vote 1 abstains \(approximately43%43\\%of voting epochs for attention/MLP and69%69\\%for normalisation, estimated from standard Lion gradient statistics; Appendix[B\.16](https://arxiv.org/html/2607.09266#A2.SS16)\)\. The per\-layer mechanism creates divergent compound level trajectories \(Figure[2](https://arxiv.org/html/2607.09266#S5.F2)\), but the global tiebreaker signal is necessary to convert this differentiation into an accuracy gain on ViT/C100\.

The voting mechanism works with either update rule\. On the two configurations tested with symmetric updates, the difference from asymmetric is not significant:−0\.26\-0\.26pp on WRN/C10 \(p=0\.49p=0\.49\) and\+0\.83\+0\.83pp on ViT/C100 \(p=0\.060p=0\.060\)\. The symmetric variant achieves the highest single\-configuration accuracy in the study:70\.54%±0\.9470\.54\\%\\pm 0\.94on ViT/C100, significantly above Lion \(\+1\.59\+1\.59pp,p=0\.002p=0\.002\), though with higher cross\-seed variance than the asymmetric default \(std0\.940\.94vs\.0\.600\.60\)\. The choice of update rule is a design decision, not a derivation \(§[5\.3](https://arxiv.org/html/2607.09266#S5.SS3)\)\.

## 5Analysis

### 5\.1Layer\-Type Differentiation

Table 3:Mean compound level by layer type, ViT\-Tiny/CIFAR\-100, LionVote \+ cosine warmup, cadence 4 \(8 seeds, 12 blocks\)\. The effective LR multiplier at epoch 300 isexp⁡\(s⋅0\.9/2\)\\exp\(s\\cdot 0\.9/2\); SDseedis the standard deviation of the multiplier across 8 seeds \(embed\+patch and head\+final\_norm each contain too few tensors per seed for meaningful per\-type SDs\)\.By epoch 300 \(Table[3](https://arxiv.org/html/2607.09266#S5.T3)\), Lion’s effective scale is2\.6×2\.6\{\\times\}too high for attention parameters,2\.8×2\.8\{\\times\}too high for MLP, and2\.0×2\.0\{\\times\}too high for normalisation\. Normalisation layers receive32%32\\%higher effective scale than attention layers \(0\.50/0\.38=1\.320\.50/0\.38=1\.32\); this ratio is consistent at cadence 8 \(1\.331\.33; Appendix[B\.7](https://arxiv.org/html/2607.09266#A2.SS7)\)\. Because the compound multiplierαi\\alpha\_\{i\}scales both the sign update and decoupled weight decay, the absolute reductions measure joint LR\+WD miscalibration, and a global rate reduction cannot reproduce the per\-layer differentiation regardless of the LR/WD decomposition\.

![Refer to caption](https://arxiv.org/html/2607.09266v1/fig/fig2_spaghetti.png)Figure 2:Compound level trajectories for all 152 parameters of ViT\-Tiny on CIFAR\-100 \(cadence 8, 8\-seed average\)\. Thin lines show individual parameters; thick lines show layer\-type means\. All layer types drift negative over training, but normalisation layers \(green\) and the classification head \(brown\) are penalised less than attention \(blue\) and MLP \(orange\) parameters, reflecting a∼33%\{\\sim\}33\\%difference in effective scale by epoch 296\. Within\-type spread \(∼4\{\\sim\}4levels\) substantially exceeds between\-type mean differences \(0\.60\.6levels\)\.All levels are negative \(Figure[2](https://arxiv.org/html/2607.09266#S5.F2)\), but the32%32\\%effective\-scale disparity between normalisation and attention layers, consistent across cadence 4 \(ratio1\.321\.32\) and cadence 8 \(ratio1\.331\.33; Appendix[B\.7](https://arxiv.org/html/2607.09266#A2.SS7)\), confirms that a uniform rate reduction cannot substitute for per\-layer differentiation\.

The compound trajectories quantify the disparity that Zhao et al\.\(Zhao et al\.,[2025](https://arxiv.org/html/2607.09266#bib.bib25)\)identify qualitatively on autoregressive language models \(§[2\.2](https://arxiv.org/html/2607.09266#S2.SS2)\): the layers they flag as needing distinct treatment \(normalisation and the classification head\) are the same layers that the voting system assigns the highest effective rates\.

The differential treatment has a structural basis\. LayerNorm parameters are elementwise scalars whose gradient distributions differ qualitatively from the large projection matrices in attention and MLP layers\. The voting system detects this through gradient alignment statistics \(Vote 1\) without requiring architectural knowledge; standard Lion training without voting produces the same layer\-type ordering \(Appendix[B\.16](https://arxiv.org/html/2607.09266#A2.SS16)\)\. Cross\-optimizer diagnostics confirm the pattern: the attn<<norm ordering holds under both Lion and AdamW, but Lion’ssign⁡\(⋅\)\\operatorname\{sign\}\(\\cdot\)amplifies the gap \(1\.88×1\.88\{\\times\}larger norm–attn cosine difference;0\.1090\.109vs\.0\.0580\.058\)\. AdamW’s second moment smooths layer\-type differences; Lion’s sign operation preserves raw gradient structure, making per\-layer adaptation more consequential \(Appendix[B\.16](https://arxiv.org/html/2607.09266#A2.SS16)\)\.

Proposition[A\.6](https://arxiv.org/html/2607.09266#A1.Thmtheorem6)predicts the observed operating regime: the convergence bound degradation at compound levelssgrows ascosh⁡\(s⋅β1/2\)\\cosh\(s\\cdot\\beta\_\{1\}/2\), reaching\+43%\+43\\%at level−2\-2where most layer types settle, and\+106%\+106\\%at level−3\-3which appears only transiently \(Appendix[A\.4](https://arxiv.org/html/2607.09266#A1.SS4)\)\.

##### What transfers beyond Lion\.

Running LionVote’s diagnostics against a paired AdamW trajectory on the same workload \(Appendix[B\.16](https://arxiv.org/html/2607.09266#A2.SS16)\) separates the optimizer\-agnostic and Lion\-specific components of the mechanism\. Vote 1’s input signal \(cosine of epoch\-mean gradients\) and firing rates are near\-invariant between Lion and AdamW \(V1=−1V\_\{1\}=\-1rate37\.6%37\.6\\%vs\.35\.4%35\.4\\%\); its upper threshold’s derivation uses Lion’s sign structure, but the2/32/3supermajority convention is optimizer\-agnostic\. Vote 2’s thresholds, derived from Lion’sβ2\\beta\_\{2\}\-EMA time constant, misfire under AdamW’s different momentum mechanics \(V2=−1V\_\{2\}=\-1fires on99\.98%99\.98\\%of AdamW observations vs\.2\.43%2\.43\\%under Lion\); these thresholds are Lion\-specific and would need re\-derivation for transfer\. The tiebreaker, compound\-level state machine, and cadence/max\-level hyperparameters are optimizer\-agnostic\. A port to another momentum\-based optimizer would preserve the voting architecture, recalibrate Vote 2 against that optimizer’s effective momentum time constant, and re\-examine the LR exponent\. A full breakdown is in Table[18](https://arxiv.org/html/2607.09266#A2.T18)\.

### 5\.2Architecture Dependence

On WRN\-28\-10/CIFAR\-10, the three residual groups \(layer1, layer2, layer3\) settle to compound levels of−1\.68\-1\.68,−1\.62\-1\.62, and−1\.84\-1\.84at epoch 200, a spread of0\.220\.22levels between the most and least reduced groups \(Appendix[B\.7](https://arxiv.org/html/2607.09266#A2.SS7)\)\. On ViT\-Tiny, the spread between attention \(−2\.15\-2\.15\) and normalisation \(−1\.54\-1\.54\) is0\.610\.61levels,2\.8×2\.8\{\\times\}larger\.

The difference reflects architectural heterogeneity\. WRN groups are structurally uniform: each is a sequence of conv\+BN blocks with the same connectivity pattern at different spatial resolutions\. ViT blocks contain three qualitatively different layer types, each with distinct gradient dynamics\. The voting system finds more to differentiate in the ViT\.

However, heterogeneity is necessary but not sufficient\. At cadence 8, the ViT attn\-norm spread is0\.640\.64levels on CIFAR\-10 and0\.630\.63levels on CIFAR\-100 \(nearly identical\), yet LionVote significantly helps on CIFAR\-100 \(\+0\.76\+0\.76pp,p=0\.012p=0\.012\) and significantly hurts on CIFAR\-10 \(−0\.88\-0\.88pp,p<0\.001p<0\.001\), where all cadences degrade performance\. What distinguishes CIFAR\-100 from CIFAR\-10 is not architectural spread but a task\-level factor; two tasks do not suffice to identify it\. Additional architectures and tasks are needed to map the boundary of per\-layer adaptation value\.

LionVote without a schedule matches its scheduled variant on WRN/C10 but falls short on ViT/C100 \(Appendix[B\.17](https://arxiv.org/html/2607.09266#A2.SS17)\)\.

### 5\.3Limitations

Experiments are limited to CIFAR\-10 and CIFAR\-100 with WideResNet and ViT\-Tiny\. The scope is deliberate: every configuration is run with 8 random seeds to eliminate variance\-driven conclusions, a level of replication that is incompatible with large\-scale experiments under limited compute\. Whether the layer\-type differentiation observed here \(attention/MLP penalised relative to normalisation\) holds at larger scale and on different architectures is an open question\.

##### Testing hierarchy\.

Cadence is selected by ablation fromc∈\{4,6,8\}c\\in\\\{4,6,8\\\}, making the LionVote c8 vs\. Lion comparison partially post\-hoc\. Four control experiments on ViT\-Tiny/CIFAR\-100 \(8 seeds each; Appendix[B\.15](https://arxiv.org/html/2607.09266#A2.SS15)\) test the per\-layer adaptation claim\. Lion at a uniformly lower LR \(4×10−54\{\\times\}10^\{\-5\}, matching LionVote’s average effective rate\) is indistinguishable from standard Lion \(p=0\.30p=0\.30\) and significantly below LionVote c8 \(p=0\.003p=0\.003, survives Bonferroni correction\)\. Static per\-layer\-type multipliers \(epoch\-300 values from Table[3](https://arxiv.org/html/2607.09266#S5.T3)\) perform worse than standard Lion \(p=0\.024p=0\.024\), confirming that the non\-monotonic trajectory cannot be replicated by static scheduling\. A weight decay sweep \(wd∈\{0\.25,0\.5,0\.75\}\\mathrm\{wd\}\\in\\\{0\.25,0\.5,0\.75\\\}\) does not close the gap \(p≥0\.075p\\geq 0\.075\)\. Thec=8c\{=\}8vs\. Lion comparison \(p=0\.017p=0\.017\) is marginal after Holm–Bonferroni correction; the primary evidence rests on the control comparison\. The study is underpowered below Cohen’sd≈0\.8d\\approx 0\.8\.

##### Practitioner guidance\.

LionVote at cadence 8 is most likely to help on architectures with qualitatively distinct layer types \(ViT\-style\)\. On uniform CNNs or tasks where Lion is near its ceiling, it provides no consistent advantage\.

##### Additional limitations\.

The tiebreaker uses validation loss during training \(∼37\{\\sim\}37decisions at cadence 8\); future work should confirm on a held\-out split\. Vote 2 rarely fires at Lion’s recommended rate \(§[4\.3](https://arxiv.org/html/2607.09266#S4.SS3)\)\. LionVote exhibits higher cross\-seed variance than Lion \(e\.g\. std0\.820\.82vs\.0\.400\.40on WRN\-28\-10/C10\)\. The symmetric variant \(70\.54%70\.54\\%on ViT/C100,p=0\.002p=0\.002\) was tested on only 2 of 4 configurations\.

## 6Conclusion

LionVote is a per\-layer learning rate mechanism for Lion whose voting thresholds are derived from mathematical structure rather than tuned\. Three findings are useful beyond the specific method\.

First, the threshold derivation methodology \(Appendix[A\.1](https://arxiv.org/html/2607.09266#A1.SS1)–[A\.6](https://arxiv.org/html/2607.09266#A1.SS6)\)—Sheppard’s sign\-agreement formula, the EMA time constant, and the descent lemma for sign\-based updates—provides reusable building blocks for per\-layer mechanisms in other optimizers\.

Second, Lion’s effective scale is systematically miscalibrated across layer types on ViT\-Tiny, with a32%32\\%disparity between normalisation and attention layers \(§[5\.1](https://arxiv.org/html/2607.09266#S5.SS1)\)\. Because the compound multiplier scales both the sign update and decoupled weight decay, this is a joint LR\+WD finding\. A weight decay sweep \(WD=0\.25\{=\}0\.25, WD=0\.75\{=\}0\.75; §[5\.3](https://arxiv.org/html/2607.09266#S5.SS3)\) confirms that neither WD adjustment closes the LionVote gap, ruling out simple global WD miscalibration as the explanation and supporting the per\-layer adaptation claim\. This complementsZhao et al\. \([2025](https://arxiv.org/html/2607.09266#bib.bib25)\)by quantifying the disparity under Lion on vision architectures\. Future work on Lion should consider layer\-type\-aware scaling even without the full voting mechanism\. The voting mechanism is not tied to the asymmetric update rule: a symmetric variant achieves70\.54%70\.54\\%on ViT/C100 \(p=0\.002p=0\.002vs\. Lion, tested on 2 of 4 configurations\), the study’s best single result\.

Third, per\-layer adaptation provides more value as architectural heterogeneity increases\. The compound level spread between layer types is2\.8×2\.8\{\\times\}larger on ViT\-Tiny than on WideResNet, and the accuracy benefit tracks this spread\. This suggests that architectures with diverse layer types are the natural setting for per\-layer methods\.

Open questions include whether the voting mechanism generalises to other sign\-based or adaptive optimizers, whether cadence can be adapted during training rather than fixed, and whether training regimes exist—such as higher base learning rates or differentβ2\\beta\_\{2\}values—in which Vote 2 \(momentum health\) becomes a primary driver of compound level movement\.

## References

- Andrychowicz et al\. \[2016\]Andrychowicz, M\., Denil, M\., Gomez, S\., Hoffman, M\. W\., Pfau, D\., Schaul, T\., Shillingford, B\., and de Freitas, N\.Learning to learn by gradient descent by gradient descent\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 29, 2016\.
- Bernstein et al\. \[2018\]Bernstein, J\., Wang, Y\.\-X\., Azizzadenesheli, K\., and Anandkumar, A\.signSGD: Compressed optimisation for non\-convex problems\.In*Proceedings of the 35th International Conference on Machine Learning \(ICML\)*, volume 80, pages 560–569\. PMLR, 2018\.
- Chen et al\. \[2023\]Chen, X\., Liang, C\., Huang, D\., Real, E\., Wang, K\., Liu, Y\., Pham, H\., Dong, X\., Luong, T\., Hsieh, C\.\-J\., Lu, Y\., and Le, Q\. V\.Symbolic discovery of optimization algorithms\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 36, 2023\.
- Cohen et al\. \[2021\]Cohen, J\., Kaur, S\., Li, Y\., Kolter, J\. Z\., and Talwalkar, A\.Gradient descent on neural networks typically occurs at the edge of stability\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.
- Defazio & Mishchenko \[2023\]Defazio, A\. and Mishchenko, K\.Learning\-rate\-free learning by D\-Adaptation\.In*Proceedings of the 40th International Conference on Machine Learning \(ICML\)*, volume 202, pages 7449–7479\. PMLR, 2023\.
- Defazio et al\. \[2024\]Defazio, A\., Yang, X\., Mehta, H\., Mishchenko, K\., Khaled, A\., and Cutkosky, A\.The road less scheduled\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 37, 2024\.
- Dosovitskiy et al\. \[2021\]Dosovitskiy, A\., Beyer, L\., Kolesnikov, A\., Weissenborn, D\., Zhai, X\., Unterthiner, T\., Dehghani, M\., Minderer, M\., Heigold, G\., Gelly, S\., Uszkoreit, J\., and Houlsby, N\.An image is worth 16x16 words: Transformers for image recognition at scale\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.
- Duchi et al\. \[2011\]Duchi, J\., Hazan, E\., and Singer, Y\.Adaptive subgradient methods for online learning and stochastic optimization\.*Journal of Machine Learning Research*, 12:2121–2159, 2011\.
- He et al\. \[2016\]He, K\., Zhang, X\., Ren, S\., and Sun, J\.Deep residual learning for image recognition\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 770–778, 2016\.
- Hao et al\. \[2025\]Hao, J\., Gong, X\., Xu, J\., Wang, Z\., and Liu, M\.Noise\-adaptive layerwise learning rates: Accelerating geometry\-aware optimization for deep neural network training\.*arXiv preprint arXiv:2510\.14009*, 2025\.
- Howard & Ruder \[2018\]Howard, J\. and Ruder, S\.Universal language model fine\-tuning for text classification\.In*Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pages 328–339, 2018\.
- Jordan et al\. \[2024\]Jordan, K\., Jin, Y\., Boza, V\., You, J\., Cesista, F\., Newhouse, L\., and Bernstein, J\.Muon: An optimizer for hidden layers in neural networks\.Blog post, 2024\.
- Kingma & Ba \[2015\]Kingma, D\. P\. and Ba, J\.Adam: A method for stochastic optimization\.In*International Conference on Learning Representations \(ICLR\)*, 2015\.
- Liang et al\. \[2026\]Liang, K\., Chen, L\., Liu, B\., and Liu, Q\.Cautious optimizers: Improving training with one line of code\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.
- Loshchilov & Hutter \[2017\]Loshchilov, I\. and Hutter, F\.SGDR: Stochastic gradient descent with warm restarts\.In*International Conference on Learning Representations \(ICLR\)*, 2017\.
- Loshchilov & Hutter \[2019\]Loshchilov, I\. and Hutter, F\.Decoupled weight decay regularization\.In*International Conference on Learning Representations \(ICLR\)*, 2019\.
- Paszke et al\. \[2019\]Paszke, A\., Gross, S\., Massa, F\., Lerer, A\., Bradbury, J\., Chanan, G\., Killeen, T\., Lin, Z\., Gimelshein, N\., Antiga, L\., Desmaison, A\., Köpf, A\., Yang, E\., DeVito, Z\., Raison, M\., Tejani, A\., Chilamkurthy, S\., Steiner, B\., Fang, L\., Bai, J\., and Chintala, S\.PyTorch: An imperative style, high\-performance deep learning library\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 32, 2019\.
- Orvieto & Xiao \[2024\]Orvieto, A\. and Xiao, L\.An adaptive stochastic gradient method with non\-negative Gauss\-Newton stepsizes\.*arXiv preprint arXiv:2407\.04358*, 2024\.
- Schmidt et al\. \[2021\]Schmidt, R\. M\., Schneider, F\., and Hennig, P\.Descending through a crowded valley — benchmarking deep learning optimizers\.In*Proceedings of the 38th International Conference on Machine Learning \(ICML\)*, volume 139, pages 9367–9376\. PMLR, 2021\.
- Vaswani et al\. \[2017\]Vaswani, A\., Shazeer, N\., Parmar, N\., Uszkoreit, J\., Jones, L\., Gomez, A\. N\., Kaiser, L\., and Polosukhin, I\.Attention is all you need\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 30, 2017\.
- Wang et al\. \[2024\]Wang, J\., Teng, Y\., and Choromanska, A\.AutoDrop: Training deep learning models with automatic learning rate drop\.In*Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence \(UAI\)*, volume 244, pages 3603–3629\. PMLR, 2024\.
- You et al\. \[2017\]You, Y\., Gitman, I\., and Ginsburg, B\.Large batch training of convolutional networks\.*arXiv preprint arXiv:1708\.03888*, 2017\.
- You et al\. \[2020\]You, Y\., Li, J\., Reddi, S\., Hseu, J\., Kumar, S\., Bhojanapalli, S\., Song, X\., Demmel, J\., Keutzer, K\., and Hsieh, C\.\-J\.Large batch optimization for deep learning: Training BERT in 76 minutes\.In*International Conference on Learning Representations \(ICLR\)*, 2020\.
- Zagoruyko & Komodakis \[2016\]Zagoruyko, S\. and Komodakis, N\.Wide residual networks\.In*Proceedings of the British Machine Vision Conference \(BMVC\)*, 2016\.
- Zhao et al\. \[2025\]Zhao, R\., Morwani, D\., Brandfonbrener, D\., Vyas, N\., and Kakade, S\.Deconstructing what makes a good optimizer for autoregressive language models\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.

## Appendix AHyperparameters

LionVote’s voting thresholds are not tuned\. Vote 1’s lower boundary is a geometric identity; its upper boundary connects a classical supermajority convention to a unique cosine threshold via the sign structure of Lion’s update rule\. Vote 2’s boundaries follow from the intrinsic timescale of theβ2\\beta\_\{2\}exponential moving average\. The tiebreaker threshold is the noise floor of cross\-entropy loss estimation for standard validation set sizes; unlike Votes 1 and 2, it is determined to order of magnitude rather than uniquely\. The per\-layer LR exponentβ1/2\\beta\_\{1\}/2couples to Lion’s momentum\-gradient balance throughβ1\\beta\_\{1\}and to the curvature exponent ofLL\-smooth objectives through the divisor22\.

### A\.1Vote 1: Gradient Direction Stability \(Thresholds 0 and 0\.5\)

Vote 1 computes

c=cos⁡\(g¯curr,g¯prev\),c\\;=\\;\\cos\(\\bar\{g\}\_\{\\mathrm\{curr\}\},\\,\\bar\{g\}\_\{\\mathrm\{prev\}\}\),the cosine similarity between the current and previous epoch\-mean gradient snapshots, and returns\+1\+1ifc\>0\.5c\>0\.5,−1\-1ifc<0c<0, and0otherwise\. The two thresholds partition\[−1,1\]\[\-1,1\]into a*reversal zone*\(c<0c<0\), a*dead band*\(c∈\[0,0\.5\]c\\in\[0,0\.5\]\), and a*stability zone*\(c\>0\.5c\>0\.5\)\. We justify each boundary in turn, then characterise the dead band\.

#### Lower boundary:c=0c=0

The cosine similarity satisfiesc<0c<0if and only if the angle betweeng¯curr\\bar\{g\}\_\{\\mathrm\{curr\}\}andg¯prev\\bar\{g\}\_\{\\mathrm\{prev\}\}exceedsπ/2\\pi/2: the epoch\-mean gradient has reversed direction across the voting window\. This is a complete, parameter\-free geometric criterion\.

#### Upper boundary:c=0\.5c=0\.5

The lower boundary establishesc=0c=0as the onset of positive alignment\. The upper threshold determines where alignment is strong enough to warrant a positive vote\. Lion appliessign⁡\(⋅\)\\operatorname\{sign\}\(\\cdot\)coordinate\-wise; the operationally relevant quantity for a sign\-based update is the fraction of gradient coordinates that maintain their sign between voting epochs\. We show thatc=0\.5c=0\.5is the unique threshold corresponding to exactly a two\-thirds supermajority of coordinate sign pairs agreeing\. This is a consequence of the exact trigonometric identityarccos⁡\(1/2\)=π/3\\arccos\(1/2\)=\\pi/3\.

###### Lemma A\.1\(Sheppard’s sign\-agreement formula\)\.

For\(X,Y\)\(X,Y\)jointly Gaussian with zero means, equal variances, and correlationρ∈\(−1,1\)\\rho\\in\(\-1,1\),

P​\(sign⁡\(X\)=sign⁡\(Y\)\)=1−arccos⁡\(ρ\)π\.P\\\!\\bigl\(\\operatorname\{sign\}\(X\)=\\operatorname\{sign\}\(Y\)\\bigr\)\\;=\\;1\-\\frac\{\\arccos\(\\rho\)\}\{\\pi\}\.

###### Proof\.

P​\(X​Y\>0\)\\displaystyle P\(XY\>0\)=2​P​\(X\>0,Y\>0\)=2​\[14\+arcsin⁡\(ρ\)2​π\]=12\+arcsin⁡\(ρ\)π=1−arccos⁡\(ρ\)π,\\displaystyle=2\\,P\(X\>0,\\,Y\>0\)=2\\\!\\left\[\\frac\{1\}\{4\}\+\\frac\{\\arcsin\(\\rho\)\}\{2\\pi\}\\right\]=\\frac\{1\}\{2\}\+\\frac\{\\arcsin\(\\rho\)\}\{\\pi\}=1\-\\frac\{\\arccos\(\\rho\)\}\{\\pi\},usingarcsin⁡\(ρ\)\+arccos⁡\(ρ\)=π/2\\arcsin\(\\rho\)\+\\arccos\(\\rho\)=\\pi/2\. ∎

###### Theorem A\.2\(Sign\-agreement at the upper threshold\)\.

Assume coordinate isotropy: for each coordinateii, the pair\(g¯curr,i,g¯prev,i\)\(\\bar\{g\}\_\{\\mathrm\{curr\},i\},\\,\\bar\{g\}\_\{\\mathrm\{prev\},i\}\)is drawn from a bivariate zero\-mean Gaussian with equal variances and correlationcc\. Atc=0\.5c=0\.5, the probability that a coordinate pair agrees in sign is exactly2/32/3\. Moreover,c=0\.5c=0\.5is the unique value in\(−1,1\)\(\-1,1\)for which this probability equals2/32/3\.

###### Proof\.

Sincearccos⁡\(1/2\)=π/3\\arccos\(1/2\)=\\pi/3exactly, Lemma[A\.1](https://arxiv.org/html/2607.09266#A1.Thmtheorem1)givesP=1−\(π/3\)/π=2/3P=1\-\(\\pi/3\)/\\pi=2/3\. Uniqueness follows from the bijectivity ofarccos\\arccoson\[−1,1\]\[\-1,1\]: the equation1−arccos⁡\(ρ\)/π=2/31\-\\arccos\(\\rho\)/\\pi=2/3has the unique solutionρ=1/2\\rho=1/2\. ∎

The thresholdc=0\.5c=0\.5is therefore the unique value in\(−1,1\)\(\-1,1\)at which a two\-thirds supermajority of gradient coordinate sign pairs are stable across voting epochs\.

##### Empirical validation\.

On ViT\-Tiny/CIFAR\-100 \(seed 2, cadence 8\), the per\-coordinate sign\-agreement fraction was measured at each voting epoch for all parameters whose global cosine similarity falls within a given bin\. Table[4](https://arxiv.org/html/2607.09266#A1.T4)compares the Sheppard prediction \(Lemma[A\.1](https://arxiv.org/html/2607.09266#A1.Thmtheorem1)\) against the observed fraction\.

Table 4:Predicted vs\. observed coordinate sign\-agreement fraction by cosine similarity bin, ViT\-Tiny/CIFAR\-100 \(seed 2, cadence 8, all voting epochs\)\. “Predicted” is1−arccos⁡\(cmid\)/π1\-\\arccos\(c\_\{\\mathrm\{mid\}\}\)/\\pifrom Lemma[A\.1](https://arxiv.org/html/2607.09266#A1.Thmtheorem1)\.Atc∈\[0\.45,0\.55\]c\\in\[0\.45,0\.55\]\(n=59n=59measurements\), the mean observed sign agreement is0\.6840\.684, within2\.6%2\.6\\%of the predicted2/32/3\. The prediction tracks the data across the full rangec∈\[0\.3,0\.7\]c\\in\[0\.3,0\.7\], with a consistent positive bias of11–7%7\\%\(increasing with cosine; upper bins haven<10n<10\): real ViT\-Tiny gradients exhibit slightly more sign agreement than coordinate isotropy predicts\. The bias makes the0\.50\.5threshold conservative—the true two\-thirds boundary lies slightly below0\.50\.5\. This is a single\-seed measurement; the direction of the bias is consistent across all eight cosine bins\. The2/32/3supermajority convention is itself a design choice; Appendix[B\.4](https://arxiv.org/html/2607.09266#A2.SS4)sweeps the resulting upper threshold across\[0\.3,0\.7\]\[0\.3,0\.7\]on recorded Lion diagnostics and shows the vote composition does not sit near a phase transition in this range\.

#### Dead band

The two boundaries together define the dead bandc∈\[0,0\.5\]c\\in\[0,0\.5\]\. Within it, the gradient direction has not reversed, but under the coordinate isotropy model strictly fewer than two\-thirds of coordinate sign pairs are stable\. Vote 1 abstains, treating this as an ambiguous regime where neither direction is warranted\.

### A\.2Vote 2: Momentum Health \(Thresholds1/e1/eandee\)

Vote 2 computes

r=‖m‖‖g¯curr‖\+ϵ,r\\;=\\;\\frac\{\\\|m\\\|\}\{\\\|\\bar\{g\}\_\{\\mathrm\{curr\}\}\\\|\+\\epsilon\},the ratio of the momentum EMA norm to the epoch\-mean gradient norm, and returns−1\-1ifr\>er\>e,\+1\+1ifr<1/er<1/e, and0otherwise\. Both thresholds derive from the EMA’s intrinsic timescale as a single symmetric pair\.

###### Lemma A\.3\(EMA time\-constant identity\)\.

The EMAmt=β2​mt−1\+\(1−β2\)​gtm\_\{t\}=\\beta\_\{2\}m\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)g\_\{t\}has time constantτ=−1/ln⁡β2\\tau=\-1/\\ln\\beta\_\{2\}\. For anyβ2∈\(0,1\)\\beta\_\{2\}\\in\(0,1\),

β2τ=e−1\.\\beta\_\{2\}^\{\\,\\tau\}\\;=\\;e^\{\-1\}\.

###### Proof\.

Consider the unit\-step response: ifgtg\_\{t\}switches from0to11att=0t=0, unrolling the recurrence gives

mt=1−β2t\.m\_\{t\}\\;=\\;1\-\\beta\_\{2\}^\{\\,t\}\.In standard signal\-processing usage, the*time constant*τ\\tauof such a first\-order system is the time at which the step response reaches1−1/e≈63\.2%1\-1/e\\approx 63\.2\\,\\%of its final value\. Settingmτ=1−1/em\_\{\\tau\}=1\-1/e:

1−β2τ=1−1e⟹β2τ=e−1\.1\-\\beta\_\{2\}^\{\\,\\tau\}\\;=\\;1\-\\frac\{1\}\{e\}\\qquad\\Longrightarrow\\qquad\\beta\_\{2\}^\{\\,\\tau\}\\;=\\;e^\{\-1\}\.Taking logarithms of both sides,

τ​ln⁡β2=−1⟹τ=−1ln⁡β2\.\\tau\\ln\\beta\_\{2\}\\;=\\;\-1\\qquad\\Longrightarrow\\qquad\\tau\\;=\\;\-\\frac\{1\}\{\\ln\\beta\_\{2\}\}\.No free parameter enters: the identityβ2τ=e−1\\beta\_\{2\}^\{\\,\\tau\}=e^\{\-1\}holds for everyβ2∈\(0,1\)\\beta\_\{2\}\\in\(0,1\)\. ∎

Under stationarity, the EMA converges in mean to the gradient mean \(𝔼​\[mt\]=𝔼​\[gt\]\\mathbb\{E\}\[m\_\{t\}\]=\\mathbb\{E\}\[g\_\{t\}\]\), sorrconcentrates near11\. Deviations signal non\-stationarity\.

###### Proposition A\.4\(Ratio dynamics after a gradient magnitude change\)\.

Suppose the EMA has converged tom0=A​u^m\_\{0\}=A\\hat\{u\}and the gradient magnitude changes instantaneously fromAAtoBBwith direction preserved \(u^\\hat\{u\}denotes the common unit direction\)\. Then fort≥0t\\geq 0,

r​\(t\)=‖mt‖B=1\+β2t​\(AB−1\)\.r\(t\)\\;=\\;\\frac\{\\\|m\_\{t\}\\\|\}\{B\}\\;=\\;1\+\\beta\_\{2\}^\{t\}\\\!\\left\(\\frac\{A\}\{B\}\-1\\right\)\.In particular,r​\(0\)=A/Br\(0\)=A/B\.

###### Proof\.

Direction is preserved, so‖mt‖=β2t​A\+\(1−β2t\)​B\\\|m\_\{t\}\\\|=\\beta\_\{2\}^\{t\}A\+\(1\-\\beta\_\{2\}^\{t\}\)B\. Dividing byBBgives the result\. ∎

###### Corollary A\.5\(The\(1/e,e\)\(1/e,\\,e\)threshold pair\)\.

The pair\(1/e,e\)\(1/e,\\,e\)is the unique symmetric pair\(a,1/a\)\(a,\\,1/a\)witha∈\(0,1\)a\\in\(0,1\)satisfyinga=β2τa=\\beta\_\{2\}^\{\\tau\}\. The interval\[1/e,e\]\[1/e,\\,e\]has log half\-width exactly 1, the natural unit of the EMA’s exponential decayβ2t=e−t/τ\\beta\_\{2\}^\{t\}=e^\{\-t/\\tau\}, and is the only log\-symmetric interval\[e−κ,eκ\]\[e^\{\-\\kappa\},\\,e^\{\\kappa\}\]centred at11whose boundary coincides withβ2τ\\beta\_\{2\}^\{\\tau\}\.

###### Proof\.

By Lemma[A\.3](https://arxiv.org/html/2607.09266#A1.Thmtheorem3),β2τ=1/e\\beta\_\{2\}^\{\\tau\}=1/efor allβ2∈\(0,1\)\\beta\_\{2\}\\in\(0,1\)\. Settinga=β2τa=\\beta\_\{2\}^\{\\tau\}givesa=1/ea=1/eand1/a=e1/a=euniquely\. The log half\-width of the interval isln⁡e=1\\ln e=1\. Any alternative pair\(e−κ,eκ\)\(e^\{\-\\kappa\},\\,e^\{\\kappa\}\)withκ≠1\\kappa\\neq 1would satisfye−κ≠β2τe^\{\-\\kappa\}\\neq\\beta\_\{2\}^\{\\tau\}and would require an independent choice ofκ\\kappa\. ∎

From Proposition[A\.4](https://arxiv.org/html/2607.09266#A1.Thmtheorem4),r​\(0\)=A/B=er\(0\)=A/B=ecorresponds to an instantaneous gradient magnitude drop by a factor of exactlyee\. Corollary[A\.5](https://arxiv.org/html/2607.09266#A1.Thmtheorem5)identifieseeas the EMA’s fundamental unit of magnitude change, sor=er=emarks a one\-time\-constant excess of the EMA over the current gradient: the upper threshold fires when the EMA retains more than one time\-constant’s worth of surplus signal\. Symmetrically,r=1/er=1/emarks a one\-time\-constant deficit, where the gradient has surged by more than oneee\-fold relative to the EMA\. The dead zoner∈\[1/e,e\]r\\in\[1/e,\\,e\]spans exactly two units in log scale\.

### A\.3Tiebreaker: Validation Loss \(Threshold±1%\\pm 1\\%\)

Unlike Votes 1 and 2, whose thresholds derive from mathematical identities independent of any experimental parameter, the tiebreaker threshold depends on the validation set sizenn\. We show that forn≈104n\\approx 10^\{4\}, the±1%\\pm 1\\%relative threshold matches the intrinsic noise floor of cross\-entropy loss estimation\.

The tiebreaker computes the relative loss improvement

δ=L¯prev−L¯curr\|L¯prev\|\+ϵ,\\delta\\;=\\;\\frac\{\\bar\{L\}\_\{\\mathrm\{prev\}\}\-\\bar\{L\}\_\{\\mathrm\{curr\}\}\}\{\|\\bar\{L\}\_\{\\mathrm\{prev\}\}\|\+\\epsilon\},whereL¯=1n​∑i=1nℓi\\bar\{L\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\ell\_\{i\}is the mean validation loss withℓi=−log⁡pcorrect,i\\ell\_\{i\}=\-\\log p\_\{\\mathrm\{correct\},i\}, and returns\+1\+1ifδ\>0\.01\\delta\>0\.01,−1\-1ifδ<−0\.01\\delta<\-0\.01, and0otherwise\.

By the central limit theorem, the standard error ofL¯\\bar\{L\}isσℓ/n\\sigma\_\{\\ell\}/\\sqrt\{n\}\. Dividing byL¯\\bar\{L\}gives the relative standard error

RSE=σℓL¯​n=CVℓn,\\mathrm\{RSE\}\\;=\\;\\frac\{\\sigma\_\{\\ell\}\}\{\\bar\{L\}\\,\\sqrt\{n\}\}\\;=\\;\\frac\{\\mathrm\{CV\}\_\{\\ell\}\}\{\\sqrt\{n\}\},whereCVℓ=σℓ/L¯\\mathrm\{CV\}\_\{\\ell\}=\\sigma\_\{\\ell\}/\\bar\{L\}is the coefficient of variation of the per\-sample loss\.

The threshold therefore reduces to a question aboutCVℓ\\mathrm\{CV\}\_\{\\ell\}\. For cross\-entropy loss on a trained but imperfect classifier, the lossℓi=−log⁡pcorrect,i\\ell\_\{i\}=\-\\log p\_\{\\mathrm\{correct\},i\}is supported on\[0,∞\)\[0,\\infty\)and right\-skewed: most predictions have high confidence and low loss, while a minority of hard examples have low confidence and high loss\. BothL¯\\bar\{L\}andσℓ\\sigma\_\{\\ell\}are dominated by this same right tail, so their ratioCVℓ\\mathrm\{CV\}\_\{\\ell\}staysΘ​\(1\)\\Theta\(1\)across models, datasets, and training stages\.

Atn≈104n\\approx 10^\{4\}this givesRSE≈CVℓ/n=Θ​\(1%\)\\mathrm\{RSE\}\\approx\\mathrm\{CV\}\_\{\\ell\}/\\sqrt\{n\}=\\Theta\(1\\%\)\. The1%1\\%threshold is therefore the order of magnitude of the noise floor at this validation set size\. It is not uniquely determined: nearby values would be comparably justified\. It serves as a round threshold that separates genuine loss movement from estimation noise\.

### A\.4LR Exponent:β1/2\\beta\_\{1\}/2

The per\-layer learning rate formula

αi=lr⋅exp⁡\(si⋅β1/2\)\\alpha\_\{i\}\\;=\\;\\mathrm\{lr\}\\cdot\\exp\\\!\\bigl\(s\_\{i\}\\cdot\\beta\_\{1\}/2\\bigr\)involves two choices:β1\\beta\_\{1\}as the numerator and22as the divisor\. The numerator follows from Lion’s momentum\-gradient structure\. The divisor coincides with the curvature exponent universal toLL\-smooth objectives\.

#### Numerator:β1\\beta\_\{1\}

The boundary behaviours ofβ1\\beta\_\{1\}pin down the numerator\. Atβ1=0\\beta\_\{1\}=0,exp⁡\(si⋅0/2\)=1\\exp\(s\_\{i\}\\cdot 0/2\)=1for allsis\_\{i\}, soαi=lr\\alpha\_\{i\}=\\mathrm\{lr\}uniformly and per\-layer modulation vanishes\. The modulation rangeα\+L/α−L=exp⁡\(L⋅β1\)\\alpha\_\{\+L\}/\\alpha\_\{\-L\}=\\exp\(L\\cdot\\beta\_\{1\}\)is strictly increasing inβ1\\beta\_\{1\}\(sinceL\>0L\>0\), and reaches its maximumexp⁡\(L\)\\exp\(L\)atβ1=1\\beta\_\{1\}=1\.

These boundaries reflect the optimizer’s structure\. Whenβ1=0\\beta\_\{1\}=0, Lion’s update rule reduces tosign⁡\(g\)\\operatorname\{sign\}\(g\): the current gradient alone, with no dependence on momentum history\[Chen et al\.,[2023](https://arxiv.org/html/2607.09266#bib.bib3)\]\. Every layer receives the same quality of learning signal regardless of convergence state, so per\-layer modulation is unnecessary and the formula correctly produces none\. Asβ1→1\\beta\_\{1\}\\to 1, the update approachessign⁡\(m\)\\operatorname\{sign\}\(m\): pure momentum\. Layers that have been learning in a stable direction accumulate useful momentum, while layers in noisy or transitioning regimes accumulate stale or contradictory momentum\.

Making the exponent proportional toβ1\\beta\_\{1\}is the simplest form that \(a\) vanishes when momentum is absent, \(b\) grows monotonically with the momentum fraction, and \(c\) introduces no free parameter beyond those already present in the optimizer\.

#### Divisor:22

The divisor determines the per\-level granularity of learning rate adaptation\. We derive a quantitative criterion from the convergence bound of sign\-based methods onLL\-smooth objectives\.

Under coordinate\-wiseLL\-smoothness, one step of a sign\-based updatexk\+1=xk−α​sign⁡\(vk\)x\_\{k\+1\}=x\_\{k\}\-\\alpha\\,\\operatorname\{sign\}\(v\_\{k\}\)satisfies

f​\(xk\+1\)≤f​\(xk\)−α​⟨∇f​\(xk\),sign⁡\(vk\)⟩\+α22​‖L~‖1,f\(x\_\{k\+1\}\)\\;\\leq\\;f\(x\_\{k\}\)\-\\alpha\\,\\langle\\nabla\\\!f\(x\_\{k\}\),\\,\\operatorname\{sign\}\(v\_\{k\}\)\\rangle\+\\frac\{\\alpha^\{2\}\}\{2\}\\,\\\|\\tilde\{L\}\\\|\_\{1\},where‖L~‖1=∑iLi\\\|\\tilde\{L\}\\\|\_\{1\}=\\sum\_\{i\}L\_\{i\}\. This is a consequence of the coordinate\-wise smoothness framework ofBernstein et al\. \[[2018](https://arxiv.org/html/2607.09266#bib.bib2)\]\(Theorem 1 and Lemma E\.1\)\. The key simplification relative to standard gradient descent is that\|sign⁡\(⋅\)\|2=1\|\\operatorname\{sign\}\(\\cdot\)\|^\{2\}=1renders the curvature penalty\(α2/2\)​‖L~‖1\(\\alpha^\{2\}/2\)\\\|\\tilde\{L\}\\\|\_\{1\}independent of the gradient magnitude\.

Telescoping overKKiterations and dividing byα​K\\alpha Kyields the convergence bound

B​\(α\)=Aα\+C​α,A=Δ​fK,C=‖L~‖12,B\(\\alpha\)\\;=\\;\\frac\{A\}\{\\alpha\}\+C\\alpha,\\qquad A=\\frac\{\\Delta f\}\{K\},\\quad C=\\frac\{\\\|\\tilde\{L\}\\\|\_\{1\}\}\{2\},where the first term captures progress \(decreasing inα\\alpha\) and the second captures curvature cost \(increasing inα\\alpha\)\.

###### Proposition A\.6\(Log\-LR perturbation cost\)\.

The optimal learning rate isα∗=A/C\\alpha^\{\*\}=\\sqrt\{A/C\}\. At a perturbed rateα∗​eε\\alpha^\{\*\}e^\{\\varepsilon\}:

B​\(α∗​eε\)B​\(α∗\)=cosh⁡\(ε\)\.\\frac\{B\(\\alpha^\{\*\}\\,e^\{\\varepsilon\}\)\}\{B\(\\alpha^\{\*\}\)\}\\;=\\;\\cosh\(\\varepsilon\)\.The relative degradation depends only on the log\-LR perturbationε\\varepsilon, not onΔ​f\\Delta f,‖L~‖1\\\|\\tilde\{L\}\\\|\_\{1\}, the dimension, or the number of steps\.

###### Proof\.

At the optimum,A/α∗=C​α∗A/\\alpha^\{\*\}=C\\alpha^\{\*\}, soB​\(α∗\)=2​A​CB\(\\alpha^\{\*\}\)=2\\sqrt\{AC\}\. Then

B​\(α∗​eε\)\\displaystyle B\(\\alpha^\{\*\}e^\{\\varepsilon\}\)=Aα∗​eε\+C​α∗​eε=A​C​\(e−ε\+eε\)=2​A​C​cosh⁡\(ε\)\.\\displaystyle=\\frac\{A\}\{\\alpha^\{\*\}e^\{\\varepsilon\}\}\+C\\,\\alpha^\{\*\}e^\{\\varepsilon\}=\\sqrt\{AC\}\\,\(e^\{\-\\varepsilon\}\+e^\{\\varepsilon\}\)=2\\sqrt\{AC\}\\,\\cosh\(\\varepsilon\)\.Dividing byB​\(α∗\)=2​A​CB\(\\alpha^\{\*\}\)=2\\sqrt\{AC\}givescosh⁡\(ε\)\\cosh\(\\varepsilon\)\. ∎

Each compound level shifts the log\-LR byε=β1/d\\varepsilon=\\beta\_\{1\}/d, whereddis the divisor\. The per\-level convergence cost iscosh⁡\(β1/d\)\\cosh\(\\beta\_\{1\}/d\), and the worst case at±L\\pm Llevels iscosh⁡\(L⋅β1/d\)\\cosh\(L\\cdot\\beta\_\{1\}/d\)\. Forβ1=0\.9\\beta\_\{1\}=0\.9andL=4L=4:

Table 5:Per\-level and worst\-case convergence bound degradation by divisor \(β1=0\.9\\beta\_\{1\}=0\.9,L=4L=4\)\.Atd=1d=1, the worst\-case degradation of18×18\\timesrenders the outer levels unusable: a layer at level\+4\+4pays an order\-of\-magnitude convergence penalty, defeating the purpose of a graded system\. Atd=3d=3, the per\-level LR factorexp\(0\.30\)≈1\.35×\\exp\(0\.30\)\\approx 1\.35\\timesprovides insufficient differentiation per vote, making the voting system sluggish\.

Atd=2d=2, each compound level incurs∼10%\{\\sim\}10\\%convergence bound degradation, and the full±4\\pm 4range spans from∼0\.17×\{\\sim\}0\.17\\timesto∼6×\{\\sim\}6\\timesthe base rate\. This permits meaningful per\-layer adaptation without excessive worst\-case cost\.

#### Consistency check: edge of stability

At the edge of stability\[Cohen et al\.,[2021](https://arxiv.org/html/2607.09266#bib.bib4)\], training dynamics self\-organise so that the loss landscape sharpness hovers near the optimizer’s stability threshold\. When the learning rate increases, the threshold drops: the loss landscape must flatten for training to remain stable\. A one\-level LR increase ofexp\(β1/2\)≈1\.57×\\exp\(\\beta\_\{1\}/2\)\\approx 1\.57\\timesis a moderate perturbation; the network has multiple epochs before the next vote to re\-equilibrate\. Usingβ1\\beta\_\{1\}directly as the exponent \(d=1d=1\) givesexp\(0\.9\)≈2\.46×\\exp\(0\.9\)\\approx 2\.46\\timesper level, a substantially larger perturbation that demands more aggressive landscape rearrangement within the same time window\.

### A\.5Maximum Compound Level

The compound level is bounded by±L\\pm L, whereL=max\_levelL=\\texttt\{max\\\_level\}\. The choice ofLLinvolves a trade\-off between expressivity \(the range of per\-layer learning rates\) and worst\-case convergence cost\.

At compound levelsswith divisord=2d=2, the learning rate multiplier isexp⁡\(s⋅β1/2\)\\exp\(s\\cdot\\beta\_\{1\}/2\)and the convergence bound degradation iscosh⁡\(s⋅β1/2\)\\cosh\(s\\cdot\\beta\_\{1\}/2\)\(Proposition[A\.6](https://arxiv.org/html/2607.09266#A1.Thmtheorem6)\)\. Forβ1=0\.9\\beta\_\{1\}=0\.9:

Table 6:LR range and worst\-case convergence bound degradation by maximum compound level \(β1=0\.9\\beta\_\{1\}=0\.9,d=2d=2\)\.Two constraints boundLLfrom above and below\.

*Lower bound \(expressivity\)\.*Compound levels are earned through consecutive same\-sign votes\. AtL=2L=2, the maximum multiplier is×2\.46\{\\times\}2\.46, comparable to a single step of a typical schedule drop, and the total LR range is×6\.0\{\\times\}6\.0\. A system with this range has limited ability to differentiate layers with substantially different convergence needs\. AtL=3L=3, the range grows to×14\.9\{\\times\}14\.9and the maximum multiplier to×3\.86\{\\times\}3\.86: enough to express meaningful differentiation, but still less than one and a half orders of magnitude\.

*Upper bound \(worst\-case cost\)\.*The worst\-case convergence bound degradationcosh⁡\(L⋅β1/2\)\\cosh\(L\\cdot\\beta\_\{1\}/2\)grows withLL\. In the table above, the growth accelerates: fromL=2L=2toL=3L=3the worst case increases by44%44\\%\(1\.43→2\.061\.43\\to 2\.06\); fromL=3L=3toL=4L=4by51%51\\%\(2\.06→3\.112\.06\\to 3\.11\); fromL=4L=4toL=5L=5by54%54\\%\(3\.11→4\.803\.11\\to 4\.80\)\. This reflects the transition ofcosh\\coshfrom its quadratic regime \(cosh⁡\(x\)≈1\+x2/2\\cosh\(x\)\\approx 1\+x^\{2\}/2\) to its exponential regime \(cosh⁡\(x\)≈ex/2\\cosh\(x\)\\approx e^\{x\}/2\)\. AtL=4L=4, the argument tocosh\\coshis1\.81\.8and the quadratic approximation1\+1\.82/2=2\.621\+1\.8^\{2\}/2=2\.62underestimates the true value3\.1073\.107by16%16\\%\. AtL=5L=5, the argument is2\.252\.25and the quadratic approximation gives3\.533\.53versus the true4\.7974\.797, a26%26\\%underestimate\. The quadratic regime, wherecosh\\coshgrows gently, is ending\.

L=4L=4is the operating point where the LR range first exceeds one and a half orders of magnitude \(×36\.6\{\\times\}36\.6\) while the worst\-case degradation remains below×3\.2\{\\times\}3\.2, moderate enough that a layer at the boundary is not catastrophically penalised\.

### A\.6Voting Cadence

The voting cadenceccis the number of epochs between consecutive voting events\. Unlike the preceding hyperparameters, cadence cannot be derived from a single mathematical identity: it depends on the interaction between the voting system, the compound level update rule, and the training dynamics\. We characterise three structural constraints that bound the useful range ofcc\.

#### Lower bound: EMA convergence

Vote 2 uses the momentum EMAmt=β2​mt−1\+\(1−β2\)​gtm\_\{t\}=\\beta\_\{2\}m\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)g\_\{t\}\. After a learning rate change alters the gradient distribution, the EMA requiresτ=−1/ln⁡β2\\tau=\-1/\\\!\\ln\\beta\_\{2\}steps to track the new mean \(Lemma[A\.3](https://arxiv.org/html/2607.09266#A1.Thmtheorem3)\)\. Until the EMA has converged, the momentum\-to\-gradient ratiorr\(Vote 2’s input\) reflects the old regime rather than the new one\. The voting window must be long enough that the EMA statistics used for voting have equilibrated after any level change from the previous vote\.

#### Lower bound: recovery time

Under the asymmetric update rule, an opposing vote resets the compound level to zero\. After a reset, the system requires up toLLconsecutive same\-sign votes, i\.e\.L⋅cL\\cdot cepochs, to return to the boundary\. Ifccis too small, the system votes before the gradient statistics have had time to reflect the learning rate change caused by the previous vote\. The votes then respond to stale information, producing oscillatory level dynamics\.

#### Upper bound: adaptivity

The system has at most⌊E/c⌋\\lfloor E/c\\rfloorvoting opportunities overEEtraining epochs, minus one for the cold\-start reference\. The number of voting opportunities must comfortably exceed2​L2Lto allow both upward and downward traversal of the full\[−L,\+L\]\[\-L,\+L\]range over the course of training\. Atc=16c=16overE=200E=200epochs, only1212votes occur; withL=4L=4, reaching the boundary and returning to zero would consume88of them, leaving little room for adaptation to changing training dynamics\.

#### Selection within the range

The constraints above place the useful range at roughlyc∈\[4,10\]c\\in\[4,10\]for typical training lengths \(E≥200E\\geq 200epochs in this work\)\. Within this range, smaller cadence gives more frequent adaptation at the cost of noisier votes and less recovery time\. Larger cadence gives cleaner votes at the cost of fewer total voting opportunities\.

### A\.7Configuring LionVote

This subsection consolidates the tunable and non\-tunable knobs, their defaults, and guidance for non\-default choices\.

Table 7:LionVote configuration summary\. “Default” columns give the value used in all experiments in this paper\. “Derivation” cites the appendix section that pins the default\.##### Per\-iteration and memory cost\.

See Appendix[B\.18](https://arxiv.org/html/2607.09266#A2.SS18): one additional in\-place accumulation per parameter per batch,22tensors plus22scalars of extra state per parameter \(3×3\{\\times\}Lion’s optimizer\-state memory\), and152152dot products and norms once everyccepochs for ViT\-Tiny\. For billion\-parameter models the3×3\{\\times\}state increase should be weighed against Lion’s optimizer\-state footprint \(∼2×\{\\sim\}2\{\\times\}the model parameters\)\.

##### When to expect benefit\.

LionVote at cadence88is most likely to help on architectures with qualitatively distinct layer types \(ViT\-style\)\. On uniform CNNs or tasks where Lion is near its ceiling, it provides no consistent advantage; see practitioner guidance in §[5\.3](https://arxiv.org/html/2607.09266#S5.SS3)\. Per\-layer adaptation value increases with architectural heterogeneity \(§[5\.2](https://arxiv.org/html/2607.09266#S5.SS2)\)\.

##### Diagnostics\-only deployment\.

Running LionVote with all votes disabled \(and compound levels decaying to zero\) is equivalent to plain Lion\. This makes diagnostics collection non\-invasive: one can deploy LionVote with the voting thresholds set to±∞\\pm\\infty\(or using a flag\) purely to collect the per\-layer gradient\-alignment and momentum\-ratio statistics, then decide whether to enable voting\.

## Appendix BExperimental Details

### B\.1Weight Decay by Configuration

Per\-config, per\-optimizer weight decay values\. Lion and LionVote use identical weight decay; all ablation variants inherit LionVote’s values\.

Lion’s weight decay values follow the recommendation of Chen et al\.\[Chen et al\.,[2023](https://arxiv.org/html/2607.09266#bib.bib3)\]\(Section 5\):33–10×10\{\\times\}larger WD than AdamW, compensating for Lion’s33–10×10\{\\times\}smaller learning rate\. AdamW uses PyTorch’s built\-in decoupled weight decay; SGD uses L2 regularisation; Lion and LionVote apply decoupled weight decay manually in the update step\.

##### Augmentation note\.

CNN configurations on CIFAR\-100 use the CIFAR\-10 AutoAugment policy\. PyTorch’sAutoAugmentPolicy\.CIFAR10is the only CIFAR policy provided intorchvision; no separate CIFAR\-100 policy exists\. Using the CIFAR\-10 policy for both datasets is standard practice in the CIFAR benchmark literature\.

### B\.2Full Convergence Threshold Tables

Table[8](https://arxiv.org/html/2607.09266#A2.T8)provides selected convergence thresholds for all configurations\.

Table 8:Epochs to reach accuracy thresholds \(mean over seeds that reached the threshold; only entries where all 8 seeds reached are shown\)\. “—” = not reliably reached\.Expanded per\-configuration convergence tables follow \(mean over 8 seeds; “—” = fewer than 8 seeds reached the threshold\)\.

##### WRN\-28\-10 / CIFAR\-10\.

##### WRN\-40\-10 / CIFAR\-100\.

##### ViT\-Tiny / CIFAR\-10\.

Setup→\\to87%→\\to88%→\\to89%→\\to90%→\\to91%→\\to92%Lion\+cos\+warm105121145173211257AdamW\+cos\+warm137161180209249—LionVote\+cos\+warm \(c8\)102114136173233—LionVote\+cos\+warm \(c6\)95117145179——LionVote\+cos\+warm101120155208——SGD\+cos\+warm——————

##### ViT\-Tiny / CIFAR\-100\.

Setup→\\to63%→\\to64%→\\to65%→\\to66%→\\to67%→\\to68%LionVote\+cos\+warm \(c8\)7686100110133160LionVote\+cos\+warm \(c6\)768898114135—Lion\+cos\+warm85104130158199239LionVote\+cos\+warm718098114146193AdamW\+cos\+warm121140158183214250SGD\+cos\+warm——————

### B\.3Cadence and Maximum Level Sensitivity

Best top\-1 accuracy \(%\) on WRN\-28\-10/CIFAR\-10 with cosine schedule, varying cadenceccand maximum levelLL\(8 seeds each\)\.

c=2c=2is consistently worst, producing noisy votes before gradient statistics equilibrate\. Within thec∈\[4,8\]c\\in\[4,8\]range, accuracy is relatively stable\. The best single entry \(93\.47%93\.47\\%atc=8c\{=\}8,L=2L\{=\}2\) suggests that a small level range with clean votes can outperform a large range with noisier ones\. Cadence 8 is preferred when total training epochs \(≥200\\geq 200\) provide sufficient voting opportunities\.

##### ViT\-Tiny/CIFAR\-100 cadence sensitivity\.

Best top\-1 accuracy \(%\) on ViT\-Tiny/CIFAR\-100 with cosine warmup schedule,L=4L=4\(8 seeds each\)\.

The cadence effect is larger on ViT\-Tiny/CIFAR\-100 than on WRN\-28\-10/CIFAR\-10: the gap between cadence 4 and cadence 8 is0\.810\.81pp \(vs\.0\.160\.16pp on WRN atL=4L\{=\}4\)\. This is consistent with the analysis in §[5\.2](https://arxiv.org/html/2607.09266#S5.SS2): ViT’s greater layer\-type heterogeneity makes Vote 1’s gradient alignment estimates more consequential\.

### B\.4Threshold Robustness \(Counterfactual Voting\)

The cadenceccand maximum levelLLare swept empirically in Appendix[B\.3](https://arxiv.org/html/2607.09266#A2.SS3)\. The*inner*thresholds \(Vote 1 upper0\.50\.5, Vote 2 factoree, divisor22, tiebreaker0\.010\.01\) are derived from the conventions in Appendix[A\.1](https://arxiv.org/html/2607.09266#A1.SS1)–[A\.3](https://arxiv.org/html/2607.09266#A1.SS3); Vote 1’s lower boundary0is the parameter\-free geometric identity and is not tunable\. This subsection reports post\-hoc*counterfactual*sensitivity: how the vote composition would change under alternative thresholds, computed by replaying LionVote’s voting logic against the recorded per\-parameter gradient statistics from a plain\-Lion training run \(Appendix[B\.16](https://arxiv.org/html/2607.09266#A2.SS16), seed 2, 36 measurement epochs, 152 parameters,n=5472n=5472\)\. The replay characterises the threshold’s*operating regime*but does not predict accuracy under retraining at a different threshold; that would require fresh training runs, which are outside the scope of this study\.

##### Vote 1 upper threshold\.

Table[9](https://arxiv.org/html/2607.09266#A2.T9)reports the vote composition as the upper threshold sweeps across\[0\.30,0\.70\]\[0\.30,0\.70\]with the lower threshold fixed at0\.

Table 9:Vote 1 composition under alternative upper thresholds \(ViT\-Tiny/CIFAR\-100, Lion baseline, seed 2,n=5472n=5472\)\.The−1\-1rate is fixed by the lower threshold0; perturbing the upper threshold redistributes mass between0and\+1\+1\. At the default0\.500\.50, the\+1\+1rate is already a minority behaviour on Lion \(2\.47%2\.47\\%\); a±0\.1\\pm 0\.1perturbation changes the\+1\+1rate by roughly a factor of two but leaves the dominant−1\-1and abstain rates within22percentage points of baseline\. The method does not sit on a phase transition in this regime\.

##### Vote 2 factor\.

Table[10](https://arxiv.org/html/2607.09266#A2.T10)sweeps the symmetric factorkk\(vote\+1\+1ifr<1/kr<1/k,−1\-1ifr\>kr\>k\) from1\.251\.25to4\.04\.0; the default isk=e≈2\.718k=e\\approx 2\.718\.

Table 10:Vote 2 composition under alternative symmetric factorskk\(same data as Table[9](https://arxiv.org/html/2607.09266#A2.T9)\)\.Three observations\. First,\+1\+1rate is zero for everykktested: the momentum norm never falls below the gradient norm by a factorkkon this workload\. The asymmetry is a property of Lion’s training dynamics on ViT\-Tiny, not of the threshold\. Second, Vote 2 sits on a steep transition betweenk=1\.5k=1\.5\(fires−1\-1on87%87\\%of observations\) andk=ek=e\(fires on2\.4%2\.4\\%\)\. The defaultk=ek=elands in a near\-inactive regime, which is consistent with the v2only ablation \(Appendix[B\.5](https://arxiv.org/html/2607.09266#A2.SS5)\)\. A choice ofkkbetween2\.02\.0andeewould make Vote 2 a materially more active contributor; whether this improves accuracy is not determined by counterfactual replay\. Third, the EMA time\-constant derivation \(Appendix[A\.2](https://arxiv.org/html/2607.09266#A1.SS2), Lemma[A\.3](https://arxiv.org/html/2607.09266#A1.Thmtheorem3)\) pinskktoeeviaβ2τ=e−1\\beta\_\{2\}^\{\\tau\}=e^\{\-1\}; any alternate choice would need a corresponding derivation\. At the derived value, Vote 2 behaves conservatively, consistent with its role as a second diagnostic rather than a primary driver\.

##### Divisor\.

The existing Table[5](https://arxiv.org/html/2607.09266#A1.T5)\(Appendix[A\.4](https://arxiv.org/html/2607.09266#A1.SS4)\) gives the*worst\-case*per\-level convergence degradationcosh⁡\(4​β1/d\)\\cosh\(4\\beta\_\{1\}/d\)abstractly\. We complement this with the*empirically realised mean*degradation𝔼s​\[cosh⁡\(s​β1/d\)\]\\mathbb\{E\}\_\{s\}\[\\cosh\(s\\beta\_\{1\}/d\)\], averaged over the compound level distribution observed across all 8 seeds of cadence\-8 LionVote on ViT\-Tiny/CIFAR\-100 \(n=44992n=44992per\-voting\-epoch observations, Table[12](https://arxiv.org/html/2607.09266#A2.T12)\)\.

Table 11:Realised mean convergence\-bound degradation by divisor \(β1=0\.9\\beta\_\{1\}=0\.9, compound levels from 8\-seed pooled cadence\-8 run\)\. The realised mean is a weighted average ofcosh⁡\(s​β1/d\)\\cosh\(s\\beta\_\{1\}/d\)using the empirical distribution overss\(Table[12](https://arxiv.org/html/2607.09266#A2.T12)\)\.Table 12:Empirical distribution of compound levelssacross all 8 seeds of cadence\-8 LionVote on ViT\-Tiny/CIFAR\-100 \(n=44992n=44992per\-voting\-epoch observations\)\.Atd=1\.5d=1\.5the realised mean degradation is∼1\.3×\{\\sim\}1\.3\{\\times\}that atd=2d=2; this quantifies, on realised rather than worst\-case levels, the claim in Appendix[A\.4](https://arxiv.org/html/2607.09266#A1.SS4)thatd=2d=2sits near the centre of a defensible interval\. Atd=3d=3, the per\-level step is small enough \(1\.35×1\.35\{\\times\}per level\) that the mechanism has limited expressivity in the observed level range\. The\|s\|=4\|s\|=4saturation rate of11\.7%11\.7\\%indicates that the defaultL=4L=4is actively used, and lowering toL=3L=3would truncate approximately12%12\\%of per\-voting\-epoch decisions\.

##### Tiebreaker threshold\.

The1%1\\%threshold is calibrated to the per\-validation\-sample coefficient of variation at typical validation set sizes \(Appendix[A\.3](https://arxiv.org/html/2607.09266#A1.SS3)\), and is explicitly described there as “determined to order of magnitude rather than uniquely”\. We do not sweep this threshold because the tiebreaker produces a single global scalar per voting epoch \(3737decisions at cadence 8\), so the counterfactual\-vote\-composition exercise used above for Votes 1 and 2 would not be informative at the per\-parameter level\.

##### What this does and does not show\.

Counterfactual vote composition characterises the threshold’s*operating regime*– whether the method sits near a phase transition, and which vote components are active versus dormant\. It does*not*predict accuracy under retraining at alternate thresholds\. Vote 1 appears robust near its default; Vote 2 at the derivedk=ek=eis near\-dormant on this workload \(consistent with the v2only ablation\); the divisord=2d=2is defensible on both worst\-case and realised\-mean grounds\.

### B\.5Vote 2 Only: Compound Level Analysis

The v2only variant disables Vote 1 and the tiebreaker, leaving only the momentum health vote\. On ViT\-Tiny/CIFAR\-100 \(seed 2, cadence 4\), compound level statistics at each voting epoch show that Vote 2 rarely fires:

At every epoch,≥89%\{\\geq\}89\\%of parameters remain at level 0\. No parameter ever reaches a positive level\. The momentum\-to\-gradient ratior=‖m‖/‖g¯‖r=\\\|m\\\|/\\\|\\bar\{g\}\\\|stays within the dead zone\[1/e,e\]\[1/e,e\]for the vast majority of parameters, consistent with the EMA operating near stationarity\. v2only approximates Lion because Vote 2 rarely fires under these training dynamics \(§[4\.3](https://arxiv.org/html/2607.09266#S4.SS3)\)\.

### B\.6Vote Dynamics Over Training

Appendix[B\.5](https://arxiv.org/html/2607.09266#A2.SS5)established that Vote 2 rarely fires\. This subsection characterises the joint firing behaviour of Votes 1 and 2 across training, including the resulting tiebreaker activation rate\. Data are from the plain\-Lion diagnostic run used in Appendix[B\.16](https://arxiv.org/html/2607.09266#A2.SS16)\(n=5472n=5472:3636voting epochs×\\times152152parameters, ViT\-Tiny/CIFAR\-100, seed 2, cadence 8\)\.

##### Joint vote distribution\.

Applying the paper’s default thresholds \(cos\>0\.5\\cos\>0\.5,r\>er\>eorr<1/er<1/e\) to the recorded diagnostics yields the joint distribution in Table[13](https://arxiv.org/html/2607.09266#A2.T13)\.

Table 13:Joint distribution of Vote 1 and Vote 2 outcomes under default thresholds \(Lion baseline, ViT\-Tiny/CIFAR\-100, seed 2,n=5472n=5472\)\.Three facts stand out\. First,V2=\+1V\_\{2\}=\+1never fires \(momentum norm never falls below gradient norm by a factor ofeeon any observation\)\. Second,V1V\_\{1\}andV2V\_\{2\}never take strictly opposite non\-zero values in the dominant sense \(\(V1,V2\)=\(−1,\+1\)\(V\_\{1\},V\_\{2\}\)=\(\-1,\+1\)is0%0\\%, and\(V1,V2\)=\(\+1,−1\)\(V\_\{1\},V\_\{2\}\)=\(\+1,\-1\)is0\.04%0\.04\\%– two observations\)\. Third, and most consequentially,V1\+V2=0V\_\{1\}\+V\_\{2\}=0holds for58\.68%58\.68\\%of per\-parameter decisions – overwhelmingly via the\(0,0\)\(0,0\)cell \(58\.64%58\.64\\%\), with active disagreement contributing only0\.04%0\.04\\%\. On this workload, the tiebreaker is therefore eligible for a majority of per\-parameter voting decisions, and Vote 1 alone determines the outcome for the remaining41%41\\%\(Vote 2 is decisive in isolation only whenV1V\_\{1\}abstains andV2V\_\{2\}fires, which is1\.33%1\.33\\%\)\.

##### Per\-voting\-epoch firing\.

Table[14](https://arxiv.org/html/2607.09266#A2.T14)sweepsV1V\_\{1\}andV2V\_\{2\}firing rates across training at nine representative epochs \(full3636\-epoch series available in supplementary CSV\)\.

Table 14:Per\-voting\-epoch firing rates, default thresholds \(Lion baseline, seed 2, ViT\-Tiny/CIFAR\-100;152152parameter tensors per row\)\. Early epochs:V1=0V\_\{1\}=0dominates; late epochs:V1=−1V\_\{1\}=\-1dominates\.V2V\_\{2\}fires sporadically throughout with no monotonic trend\.V1=−1V\_\{1\}=\-1\(reversal detection\) rises nearly monotonically from20%20\\%in early training to56%56\\%by epoch296296: as training progresses, epoch\-to\-epoch gradient direction stability decreases for an increasing fraction of layers\.V1=\+1V\_\{1\}=\+1\(strong stability\) correspondingly decays from5%5\\%to1%1\\%\.V2V\_\{2\}shows no monotonic trend; its firings appear episodically at single\-digit rates\. This profile is what drives the downward level migration visible in the stacked area plot \(Figure[4](https://arxiv.org/html/2607.09266#A2.F4)\)\.

##### Relation to level changes\.

Combined with the compound level dynamics \(Figure[2](https://arxiv.org/html/2607.09266#S5.F2), Table[3](https://arxiv.org/html/2607.09266#S5.T3)\), the picture is: the dominant driver of negative compound level accumulation is Vote 1’s growing reversal detection rate, moderated by the tiebreaker whenever both local votes abstain \(58\.6%58\.6\\%of decisions\) – typically resolving in favour of rate maintenance or reduction as validation loss ceases to improve monotonically\.

### B\.7Additional Compound Level Trajectories

##### ViT\-Tiny/CIFAR\-100, cadence 8\.

At cadence 8, the early\-training trajectory differs: levels are near zero or slightly positive at epoch 64, indicating the system initially finds no strong signal for rate reduction with the longer voting window\. By mid\-training \(epoch 120\), differentiation emerges, and by epoch 296 the attention\-vs\-normalisation spread \(0\.630\.63levels, LR ratio1\.331\.33\) is comparable to cadence 4 \(0\.610\.61levels, ratio1\.321\.32\)\.

##### WRN\-28\-10/CIFAR\-10, cadence 4\.

The three repeated block groups \(layer1, layer2, layer3\) show a spread of0\.220\.22levels at epoch 200, compared to0\.610\.61levels between attention and normalisation on ViT\-Tiny\. The smaller spread reflects the architectural uniformity of WideResNet’s residual groups\.

### B\.8Effective Learning Rate Fingerprint

![Refer to caption](https://arxiv.org/html/2607.09266v1/fig/fig_radar.png)Figure 3:Effective LR\+WD fingerprint for LionVote \(cadence 8\) on ViT\-Tiny/CIFAR\-100 at epoch 296 \(8\-seed average\)\. Each axis shows the compound multiplierαi/lr\\alpha\_\{i\}/\\mathrm\{lr\}as a fraction of the base rate\. The dashed circle marks the base rate \(1×1\{\\times\}\)\. Head and final normalisation layers retain the highest multiplier \(0\.59×0\.59\{\\times\}, from compound level−1\.19\-1\.19, Table[B\.7](https://arxiv.org/html/2607.09266#A2.SS7.SSS0.Px1)\); attention, MLP, and embedding layers are reduced to∼0\.40×\{\\sim\}0\.40\{\\times\}\.
### B\.9Compound Level Mass Flow

![Refer to caption](https://arxiv.org/html/2607.09266v1/fig/fig_level_flow.png)Figure 4:Stacked area plot showing the fraction of parameters at each compound level over training \(ViT\-Tiny/CIFAR\-100, cadence 8, seed 2\)\. Green shades indicate positive levels; red shades indicate negative levels; grey is level 0\. The mass migrates from positive/neutral levels early in training toward increasingly negative levels, with most parameters settling between levels−1\-1and−3\-3by epoch 250\.
### B\.10Cadence Effect on Layer\-Type Differentiation

![Refer to caption](https://arxiv.org/html/2607.09266v1/fig/fig_cadence_comparison.png)Figure 5:Mean compound level for attention \(solid/dashed\) and LayerNorm \(dotted\) parameters across cadencesc∈\{4,6,8\}c\\in\\\{4,6,8\\\}on ViT\-Tiny/CIFAR\-100 \(8\-seed average\)\. Lower cadence produces faster initial differentiation but noisier long\-term trajectories\. All cadences converge to similar attention–normalisation spreads by epoch 250, thoughc=4c\{=\}4overshoots early in training\.
### B\.11Effective Learning Rate Dynamics

![Refer to caption](https://arxiv.org/html/2607.09266v1/fig/fig_effective_lr.png)Figure 6:Effective learning rate \(log scale\) for all configurations and optimizers\. Bands for LionVote show±1\\pm 1std across layers within each seed, reflecting per\-layer LR spread; baselines have zero spread \(single global rate\)\. LionVote’s per\-layer rates span approximately one order of magnitude, with the spread widening as training progresses\.
### B\.12Training Loss Curves

![Refer to caption](https://arxiv.org/html/2607.09266v1/fig/fig_train_loss.png)Figure 7:Training loss for all configurations \(mean±1\\pm 1std, 8 seeds\)\. Adaptive optimizers \(AdamW, Lion, LionVote\) converge substantially faster than SGD on ViT\-Tiny\. On WRN configurations, SGD achieves the lowest final training loss despite slower initial descent\.
### B\.13Validation Loss Curves

![Refer to caption](https://arxiv.org/html/2607.09266v1/fig/fig_val_loss.png)Figure 8:Validation loss for all configurations \(mean±1\\pm 1std, 8 seeds\)\. On WRN configurations, SGD achieves the lowest validation loss\. On ViT\-Tiny, all adaptive methods converge to similar validation loss, with LionVote showing slightly faster early descent on CIFAR\-100\.
### B\.14Generalisation Gap

![Refer to caption](https://arxiv.org/html/2607.09266v1/fig/fig_generalization_gap.png)Figure 9:Generalisation gap \(validation loss−\-training loss\) for all configurations \(mean±1\\pm 1std, 8 seeds\)\. On WRN, SGD maintains the smallest gap, indicating less overfitting\. On ViT\-Tiny, all methods show negative gaps \(validation loss<<training loss\), a consequence of Mixup, CutMix, and label smoothing augmentations that make the training objective harder than the clean validation objective\.
### B\.15Control Experiments: Static Multipliers and Lower Base Rate

Four control baselines on ViT\-Tiny/CIFAR\-100 \(8 seeds each, cosine warmup, 300 epochs\) test whether the per\-layer differentiation discovered by LionVote can be replicated by static means or by adjusting global hyperparameters\.

##### Lion\-Typed\.

Lion with fixed per\-layer\-type LR multipliers from epoch 1, using the exact epoch\-300 values from Table[3](https://arxiv.org/html/2607.09266#S5.T3)\(cadence 4\): attention×0\.38\{\\times\}0\.38, MLP×0\.36\{\\times\}0\.36, normalisation×0\.50\{\\times\}0\.50, embedding×0\.40\{\\times\}0\.40\. Baselr=10−4\\mathrm\{lr\}\{=\}10^\{\-4\},wd=0\.5\\mathrm\{wd\}\{=\}0\.5\.

##### Lion \(𝐥𝐫=𝟒×𝟏𝟎−𝟓\\mathbf\{lr\{=\}4\{\\times\}10^\{\-5\}\}\)\.

Plain Lion at∼2\.5×\{\\sim\}2\.5\{\\times\}lower base rate, matching LionVote’s average effective rate at convergence\.wd=0\.5\\mathrm\{wd\}\{=\}0\.5\.

##### Weight decay sweep\.

Lion atwd=0\.25\\mathrm\{wd\}\{=\}0\.25\(0\.5×0\.5\{\\times\}default\) andwd=0\.75\\mathrm\{wd\}\{=\}0\.75\(1\.5×1\.5\{\\times\}default\), both withlr=10−4\\mathrm\{lr\}\{=\}10^\{\-4\}\.

Allpp\-values are Welch two\-samplett\-tests \(n=8n=8per group\)\.

Lion\-Typed is significantly worse than standard Lion \(−0\.48\-0\.48pp,p=0\.024p=0\.024\) but not significantly worse than AdamW \(−0\.28\-0\.28pp,p=0\.15p=0\.15\)\. The lower\-LR control is statistically indistinguishable from Lion \(p=0\.30p=0\.30\) and from AdamW \(p=0\.89p=0\.89\)\. The two controls are indistinguishable from each other \(p=0\.23p=0\.23\), and both are significantly below LionVote at cadence 8 \(p=0\.003p=0\.003andp<0\.001p<0\.001respectively\)\. Neither weight decay variant significantly differs from the default:wd=0\.25\\mathrm\{wd\}\{=\}0\.25is the best static variant but remains0\.590\.59pp below LionVote at cadence 8 \(p=0\.075p=0\.075\)\.

##### Per\-seed accuracy\.

No Lion\-Typed seed exceeds69%69\\%; one lower\-LR seed \(of eight\) exceeds69%69\\%; seven of eight LionVote seeds exceed69%69\\%\.

##### Convergence speed\.

Both controls converge faster than Lion \(10−410^\{\-4\}\) and AdamW to early thresholds \(6363–67%67\\%\) but plateau lower\. LionVote at cadence 8 reaches68%68\\%at epoch160160, a full8080epochs before the lower\-LR control\.

##### Per\-seed accuracy: ablation variants and WD sweep \(ViT/C100, cadence 8\)\.

### B\.16Raw Lion Gradient Diagnostics \(No Voting Mechanism\)

To test whether the layer\-type differentiation observed in §[5\.1](https://arxiv.org/html/2607.09266#S5.SS1)is an artefact of LionVote’s specific voting mechanism, the same per\-layer diagnostics were measured during standard Lion training on ViT\-Tiny/CIFAR\-100 \(seed 2, cadence 8 measurement interval, cosine warmup, 300 epochs\)\. No voting, no compound levels, no per\-layer LR adjustment—only the raw diagnostic signals\.

##### Gradient direction stability\.

Table[15](https://arxiv.org/html/2607.09266#A2.T15)reports the mean epoch\-mean cosine alignmentci=cos⁡\(g¯i\(curr\),g¯i\(prev\)\)c\_\{i\}=\\cos\(\\bar\{g\}\_\{i\}^\{\(\\mathrm\{curr\}\)\},\\bar\{g\}\_\{i\}^\{\(\\mathrm\{prev\}\)\}\)by layer type in late training \(epoch≥200\\geq 200\)\.

Table 15:Raw per\-layer gradient diagnostics during standard Lion training \(ViT\-Tiny/CIFAR\-100, seed 2, epoch≥200\\geq 200\)\. “Would\-vote” columns show the fraction of measurement epochs at which the raw cosine alignment would trigger a Vote 1 outcome under the paper’s thresholds\.Under standard Lion, attention and MLP layers have mean cosine alignment near zero: their epoch\-mean gradients are essentially uncorrelated across consecutive measurement windows\. Normalisation layers have mean cosine\+0\.104\+0\.104—substantially higher, indicating more directionally stable gradients\. Attention and MLP parameters would receive a negative vote \(cosine<0<0\) on56%56\\%of measurement epochs, compared to30%30\\%for normalisation—a1\.9×1\.9\{\\times\}ratio\. This asymmetry in raw gradient statistics is the signal that LionVote’s Vote 1 detects; it exists in standard Lion training independent of the voting mechanism\.

##### Momentum health\.

The momentum\-to\-gradient norm ratior=‖m‖/‖g¯‖r=\\\|m\\\|/\\\|\\bar\{g\}\\\|stays within the Vote 2 dead zone\[1/e,e\]\[1/e,e\]for9898–100%100\\%of measurements across all layer types, confirming that Vote 2’s low firing rate \(§[4\.3](https://arxiv.org/html/2607.09266#S4.SS3)\) reflects the training dynamics, not the threshold placement\.

##### Cross\-optimizer comparison \(Lion vs\. AdamW\)\.

Running the same diagnostics under AdamW \(ViT\-Tiny/CIFAR\-100, cosine warmup, lr=10−3\{=\}10^\{\-3\}, wd=0\.05\{=\}0\.05, single seed\) reveals that the attn<<norm ordering is optimizer\-general, but Lion amplifies the gap\.

Table 16:Cross\-optimizer gradient diagnostics \(ViT\-Tiny/CIFAR\-100, seed 2, late training epoch≥200\\geq 200, 13 measurement epochs×\\times152 parameters\)\. Negative\-vote rate: fraction of \(param, epoch\) pairs with cosine<0<~0\.The attn<<norm ordering holds for both optimizers \(100%100\\%of late epochs under Lion,92%92\\%under AdamW\), confirming the heterogeneity is architectural\. Lion’ssign⁡\(⋅\)\\operatorname\{sign\}\(\\cdot\)amplifies the norm–attn cosine gap1\.88×1\.88\{\\times\}relative to AdamW \(0\.1090\.109vs\.0\.0580\.058\): because sign discards gradient magnitude, directionally unstable layers cannot compensate through large gradients\. AdamW’s second moment \(1/v^1/\\sqrt\{\\hat\{v\}\}\) implicitly down\-weights noisy coordinates, partially absorbing the layer\-type difference\. This explains why per\-layer correction has the largest marginal value for sign\-based optimizers: no within\-update self\-correction mechanism exists\.

##### Vote 2 under AdamW: thresholds are Lion\-specific\.

The cosine\-alignment comparison above concerns only Vote 1’s input signal\. The analogous comparison for Vote 2 – the momentum\-to\-gradient norm ratior=‖m‖/‖g¯‖r=\\\|m\\\|/\\\|\\bar\{g\}\\\|– reveals a qualitative difference\. Under Lion,rrhas geometric mean∼1\.8\{\\sim\}1\.8across layer types and sits within the dead zone\[1/e,e\]\[1/e,e\]for∼97\.6%\{\\sim\}97\.6\\%of observations \(Table[17](https://arxiv.org/html/2607.09266#A2.T17)\)\. Under AdamW with the same parameters monitored,rrhas geometric mean∼8\{\\sim\}8–1010and lies aboveeefor∼100%\{\\sim\}100\\%of observations\.

Table 17:Vote 2 input statistics under Lion vs\. AdamW \(ViT\-Tiny/CIFAR\-100, seed 2, all3636voting epochs,n=5472n=5472per optimizer\)\. Geometric meanr=‖m‖/‖g¯‖r=\\\|m\\\|/\\\|\\bar\{g\}\\\|is reported per layer type; overall “r\>er\>e” is theV2=−1V\_\{2\}=\-1firing rate\.The ratio difference is a consequence of the different momentum mechanics\. Lion’sβ2\\beta\_\{2\}outer EMA \(β2=0\.99\\beta\_\{2\}=0\.99default\) smooths the update buffer over roughly100100steps, keeping its norm comparable to the current gradient\. AdamW’s first\-moment EMA withβ1=0\.9\\beta\_\{1\}=0\.9accumulates raw gradients on a much longer effective horizon when unnormalised by the second moment \(which is applied separately by division inside the update itself rather than to the buffer we monitor\), so the buffer norm significantly exceeds the current\-gradient norm on this workload\. Consequently, Vote 2’s symmetric\[1/e,e\]\[1/e,e\]dead zone – which is derived from the EMA time\-constant identityβ2τ=e−1\\beta\_\{2\}^\{\\tau\}=e^\{\-1\}– is calibrated against Lion’sβ2\\beta\_\{2\}\-EMA and produces a near\-constantV2=−1V\_\{2\}=\-1signal under AdamW, making it useless for per\-layer modulation without recalibration\.

##### Optimizer\-agnostic versus Lion\-specific components\.

Combining the Vote 1 comparison \(near\-invariant between Lion and AdamW:V1=−1V\_\{1\}=\-1rate37\.6%37\.6\\%vs\.35\.4%35\.4\\%;V1=\+1V\_\{1\}=\+1rate2\.5%2\.5\\%vs\.1\.1%1\.1\\%\) with the Vote 2 comparison \(optimizer\-dependent: dead\-zone occupancy97\.6%97\.6\\%vs\.0\.02%0\.02\\%\), the LionVote mechanism decomposes as follows:

Table 18:LionVote component portability to other momentum\-based optimizers, based on the Lion/AdamW diagnostic comparison \(this subsection and Table[16](https://arxiv.org/html/2607.09266#A2.T16)\)\.Two components are essentially Lion\-specific: Vote 2’s thresholds \(empirically demonstrated above\) and the LR exponent’s numeratorβ1\\beta\_\{1\}\(by Lion’s specific use ofβ1\\beta\_\{1\}as the momentum\-gradient interpolation coefficient\)\. Three components are optimizer\-agnostic in input and require only Lion’s sign\-update structure for one of the two derivations on their thresholds: Vote 1, the tiebreaker, and the compound\-level state machine\. The divisor22principle transfers but the coefficient may not\. A port to another momentum\-based optimizer would therefore preserve the architecture of the voting mechanism, recalibrate Vote 2 using that optimizer’s effective momentum time constant, and re\-examine the LR exponent\. We do not carry out such a port; this is left to future work\.

##### Single\-seed limitation\.

These measurements \(Vote 1, Vote 2, and Lion/AdamW cross\-optimizer comparisons above\) are from a single seed\. The layer\-type ordering \(norm more stable than attn/mlp\) is consistent with the 8\-seed compound level trajectories in Table[3](https://arxiv.org/html/2607.09266#S5.T3)and the structural argument in §[5\.1](https://arxiv.org/html/2607.09266#S5.SS1)\. The Vote 2 dead\-zone discrepancy between Lion and AdamW reflects a single\-seed observation of an effect \(∼5×\{\\sim\}5\{\\times\}ratio difference\) whose magnitude far exceeds plausible seed\-to\-seed variation and is therefore unlikely to reverse under replication, though confirming this is left to future work\.

### B\.17Partial Schedule Replacement

LionVote without any global schedule outperforms the scheduled variant on WRN/C10 at cadence 8 by0\.180\.18pp \(93\.48%93\.48\\%vs\.93\.30%93\.30\\%\), though this difference is not statistically significant given the standard deviations \(±0\.39\\pm 0\.39vs\.±0\.82\\pm 0\.82\)\. It falls short by0\.820\.82pp on ViT/C100 \(68\.89%68\.89\\%vs\.69\.71%69\.71\\%; Table[2](https://arxiv.org/html/2607.09266#S4.T2)\)\. The universally negative compound levels implement a form of learning rate decay that overlaps in function with cosine annealing\. On architectures where this overlap is sufficient \(WRN\), the schedule becomes unnecessary; on ViTs, the schedule provides additional value\.

### B\.18Computational Cost Details

LionVote adds one in\-place gradient accumulation per parameter per batch and linear voting cost \(one cosine similarity, one norm ratio, one integer comparison per parameter\) everyccepochs\. No second\-order information, meta\-gradients, or additional forward/backward passes are required\. Memory overhead beyond Lion is two tensors per parameter \(the gradient accumulator and the previous epoch\-mean gradient snapshot\) plus two scalars \(the compound level and the effective learning rate\)\. The per\-batch accumulation is a single in\-place addition per parameter \(the same operation count as one line of Lion’s momentum update\) and is negligible relative to the forward/backward pass\. Voting fires once everyccepochs; for ViT\-Tiny \(152152parameter tensors\), this is152152dot products and norms\. The dominant overhead is memory: the two extra tensors triple Lion’s per\-parameter optimizer state \(∼23\{\\sim\}23MB for Lion,∼69\{\\sim\}69MB for LionVote on ViT\-Tiny\)\. For billion\-parameter models the3×3\{\\times\}increase is a deployment concern\.

##### Hardware and total compute\.

All experiments were run on a single NVIDIA GPU \(A100, T4, or L40S depending on availability\)\. The 605 runs reported in this paper required∼901\{\\sim\}901GPU\-hours in total:

No preliminary or failed experiments beyond those reported were conducted\.