PsiLogic: Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark

# Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark
Source: [https://arxiv.org/html/2607.16268](https://arxiv.org/html/2607.16268)
###### Abstract

Adaptive optimizers such as Adam and AdamW apply the same update rule regardless of whether training is in a chaotic early phase or near convergence\. We introducePsiLogic\(Ψ\\PsiLogic\), an optimizer that augments Adam with a*dynamic Active Cancellation Term*gated by a dual exponential moving average \(EMA\) of scale\-normalized gradient norms\. The resulting*chaos detector*strengthens damping when gradient statistics are unstable and fades to zero as training stabilizes, providing an implicit warmup without a hand\-tuned schedule\.

We evaluate PsiLogic against Adam, AdamW, and Lion usingFairBench—a reproducible benchmark protocol with per\-optimizer learning\-rate sweeps, identical initialization per seed, and Welchtt\-tests\. On an NVIDIA H100 80GB reference run \(4 arenas, 3 seeds, 2000 steps, bf16 AMP\), PsiLogic achieves the best validation metric inthree of four arenas: NLP perplexity7\.79±0\.187\.79\\pm 0\.18vs\.8\.17±0\.088\.17\\pm 0\.08\(AdamW,p=0\.049p=0\.049\), ViT top\-1 accuracy0\.244±0\.0060\.244\\pm 0\.006vs\.0\.223±0\.0020\.223\\pm 0\.002\(AdamW,p=0\.015p=0\.015\), and ResNet top\-1 accuracy0\.222±0\.0010\.222\\pm 0\.001vs\.0\.172±0\.0040\.172\\pm 0\.004\(Adam,p=0\.001p=0\.001\)\. On diffusion, validation MSE is statistically tied with Adam/AdamW \(p=0\.49p=0\.49\)\. ResNet accuracy vs\. AdamW is a numerical tie without significance at three seeds \(p=0\.44p=0\.44\)\. Peak GPU memory is comparable across optimizers; PsiLogic incurs1\.2–1\.8×\\timeswall\-clock overhead on transformer\-heavy arenas \(implementation\-bound; Section[6](https://arxiv.org/html/2607.16268#S6.SS0.SSS0.Px4)\)\.

We release an open\-source PyTorch implementation, the full FairBench harness, and all raw CSV outputs to support independent verification\.

Keywords:optimization, Adam, adaptive learning rate, deep learning, reproducibility

## 1Introduction

The choice of optimizer affects convergence speed, generalization, and training stability in deep learning\. Adam\[[7](https://arxiv.org/html/2607.16268#bib.bib7)\]and AdamW\[[9](https://arxiv.org/html/2607.16268#bib.bib9)\]dominate practice, yet their corrective signal does not adapt to*how confused the model currently is*\. At initialization, gradients are large and noisy; near convergence, they are small and stable\. Standard Adam treats both regimes with structurally similar updates\.

We proposePsiLogic, which adds a chaos\-conditioned damping term to the Adam update\. The term is strongest when a dual EMA of normalized gradient norms signals instability, and vanishes automatically as training settles\. PsiLogic is designed as a drop\-in replacement fortorch\.optim\.Adamwith optional task presets \(PsiLogicNLP,PsiLogicGPT,PsiLogicViT\)\.

#### Contributions\.

1. 1\.PsiLogic optimizer—chaos\-gated Active Cancellation on top of Adam, with unified decay, optional gradient centralization \(GC\), and adaptive gradient clipping \(AGC\)\.
2. 2\.FairBench—a bias\-mitigated evaluation protocol: per\-optimizer LR sweep, identical weights per seed, multi\-arena tasks, and Welchtt\-tests\.
3. 3\.Reference H100 benchmark—reproducible CSVs and learning\-curve plots committed atbenchmark/results/full/, showing competitive or superior quality on NLP, ViT, and ResNet with explicit reporting of non\-significant and negative results\.

We donotclaim universal dominance over AdamW or Lion\. We report limitations—including step\-time overhead and ties on diffusion and ResNet\-vs\-AdamW—explicitly\.

## 2Related Work

#### Adaptive first\-order methods\.

Adam\[[7](https://arxiv.org/html/2607.16268#bib.bib7)\]maintains bias\-corrected first\- and second\-moment estimates for per\-parameter adaptive rates\. AdamW\[[9](https://arxiv.org/html/2607.16268#bib.bib9)\]decouples weight decay from the gradient step and is the de facto standard for Transformers\. AdaFactor\[[11](https://arxiv.org/html/2607.16268#bib.bib11)\]reduces memory via factored second\-moment estimates\. Lion\[[3](https://arxiv.org/html/2607.16268#bib.bib3)\]uses sign\-based updates with coupled weight decay; it can be memory\-efficient but often requires careful LR tuning\.

#### Large\-batch and layer\-wise scaling\.

LARS\[[13](https://arxiv.org/html/2607.16268#bib.bib13)\]and LAMB\[[14](https://arxiv.org/html/2607.16268#bib.bib14)\]rescale updates using the ratio of parameter norm to gradient norm, stabilizing very large minibatch training\. These methods address scale mismatch across layers but do not gate damping on online gradient*volatility*the way PsiLogic’s chaos detector does\.

#### Second\-order and curvature\-aware methods\.

Shampoo\[[6](https://arxiv.org/html/2607.16268#bib.bib6)\]and Sophia\[[8](https://arxiv.org/html/2607.16268#bib.bib8)\]incorporate richer curvature or Hessian information for faster convergence, at higher per\-step cost\. PsiLogic stays in the first\-order Adam family and adds only scalar chaos statistics shared across parameters\.

#### Automatic learning\-rate and warmup\.

Manual LR warmup\[[5](https://arxiv.org/html/2607.16268#bib.bib5)\]is standard for large\-batch SGD and Transformers\. Hypergradient descent\[[1](https://arxiv.org/html/2607.16268#bib.bib1)\]differentiates through the optimizer to adapt the LR online\. Recent*parameter\-free*methods such as D\-Adaptation\[[4](https://arxiv.org/html/2607.16268#bib.bib4)\]and Prodigy\[[10](https://arxiv.org/html/2607.16268#bib.bib10)\]estimate a suitable global step size from observed gradients\. PsiLogic offers a complementary, chaos\-driven*implicit warmup*: effective damping rises when gradient statistics are unstable and fades without an external schedule\.

#### Stability mechanisms\.

Gradient centralization\[[12](https://arxiv.org/html/2607.16268#bib.bib12)\]and adaptive gradient clipping\[[2](https://arxiv.org/html/2607.16268#bib.bib2)\]improve training stability\. PsiLogic optionally integrates both\. Its active\-cancellation term is orthogonal: it shrinks weights when chaos is detected, rather than only rescaling or clipping gradients\.

#### Optimizer evaluation\.

Fair comparison requires matched tuning budgets\. FairBench gives each optimizer its own LR search rather than a single shared LR, reducing tuning bias that has historically confounded optimizer comparisons\.

## 3Method

### 3\.1Notation and Per\-Group Hyperparameters

PsiLogic operates on parameter groups indexed bykk, each with learning rateη\\eta, AdamW weight decayλ\\lambda, chaos gainγ\\gamma, and a group\-specificchaos amplification factorPk≥0P\_\{k\}\\geq 0\(implementation namep\_ext, default1\.01\.0\)\.PkP\_\{k\}lets presets assign stronger cancellation to sensitive groups \(e\.g\., embeddings\) and weaker damping to others, without changing the global chaos signal\. We write𝐠t=∇θℒ\\mathbf\{g\}\_\{t\}=\\nabla\_\{\\theta\}\\mathcal\{L\}for the gradient at stepttandwt∈\[0,1\]w\_\{t\}\\in\[0,1\]for the*chaos warmup weight*\(Section[3\.5](https://arxiv.org/html/2607.16268#S3.SS5)\); this avoids overloading𝐠t\\mathbf\{g\}\_\{t\}with scalar gains\.

### 3\.2Update Rule

Each step first applies unified multiplicative decay, then the bias\-corrected Adam gradient step\. Composing the two operations yields

θt\+1=θt⋅\(1−δt\)−η⋅m^tv^t\+ε,\\theta\_\{t\+1\}=\\theta\_\{t\}\\cdot\(1\-\\delta\_\{t\}\)\-\\eta\\cdot\\frac\{\\hat\{m\}\_\{t\}\}\{\\sqrt\{\\hat\{v\}\_\{t\}\}\+\\varepsilon\}\\,,\(1\)wherem^t\\hat\{m\}\_\{t\}andv^t\\hat\{v\}\_\{t\}are the usual*bias\-corrected*Adam moments \(we use⋅^\\hat\{\\cdot\}consistently below and in Listing[1](https://arxiv.org/html/2607.16268#LST1)\) andδt\\delta\_\{t\}is the scalar unified\-decay coefficient \(Section[3\.4](https://arxiv.org/html/2607.16268#S3.SS4)\)\. Listing[1](https://arxiv.org/html/2607.16268#LST1)spells out the same sequence procedurally\.

### 3\.3Chaos Detector

Letgnt=‖𝐠t‖2/numel\\mathrm\{gn\}\_\{t\}=\\\|\\mathbf\{g\}\_\{t\}\\\|\_\{2\}/\\sqrt\{\\mathrm\{numel\}\}be the scale\-normalized gradient norm\. We maintain:

fastt\\displaystyle\\mathrm\{fast\}\_\{t\}=0\.90⋅fastt−1\+0\.10⋅gnt\\displaystyle=0\.90\\cdot\\mathrm\{fast\}\_\{t\-1\}\+0\.10\\cdot\\mathrm\{gn\}\_\{t\}\(2\)slowt\\displaystyle\\mathrm\{slow\}\_\{t\}=0\.99⋅slowt−1\+0\.01⋅gnt\\displaystyle=0\.99\\cdot\\mathrm\{slow\}\_\{t\-1\}\+0\.01\\cdot\\mathrm\{gn\}\_\{t\}\(3\)ratiot\\displaystyle\\mathrm\{ratio\}\_\{t\}=fastt/\(slowt\+ε\)\\displaystyle=\\mathrm\{fast\}\_\{t\}/\(\\mathrm\{slow\}\_\{t\}\+\\varepsilon\)\(4\)chaost\\displaystyle\\mathrm\{chaos\}\_\{t\}=tanh⁡\(slowt\)⋅\(1\+0\.5⋅tanh⁡\(relu⁡\(ratiot−1\)\)\)\\displaystyle=\\tanh\(\\mathrm\{slow\}\_\{t\}\)\\cdot\\bigl\(1\+0\.5\\cdot\\tanh\(\\operatorname\{relu\}\(\\mathrm\{ratio\}\_\{t\}\-1\)\)\\bigr\)\(5\)The fast and slow EMAs correspond to effective horizons of roughly1010and100100steps, respectively\. In adaptive mode \(default\), cancellation activates whenfastt\>τscale⋅slowt\\mathrm\{fast\}\_\{t\}\>\\tau\_\{\\mathrm\{scale\}\}\\cdot\\mathrm\{slow\}\_\{t\}\(τscale=2\.0\\tau\_\{\\mathrm\{scale\}\}=2\.0\), detecting relative spikes in gradient chaos\. Asslowt→0\\mathrm\{slow\}\_\{t\}\\to 0at convergence,chaost→0\\mathrm\{chaos\}\_\{t\}\\to 0and PsiLogic reduces toward AdamW\-like behavior\.

### 3\.4Unified Decay

Naively applying weight decay and active cancellation as separate multiplicative factors, e\.g\.θ​\(1−η​λ\)\\theta\(1\-\\eta\\lambda\)followed byθ​\(1−ct\)\\theta\(1\-c\_\{t\}\), would shrink parameters by\(1−η​λ\)​\(1−ct\)≈1−η​λ−ct\(1\-\\eta\\lambda\)\(1\-c\_\{t\}\)\\approx 1\-\\eta\\lambda\-c\_\{t\}only to first order; at large early\-step rates the cross term−η​λ​ct\-\\eta\\lambda c\_\{t\}over\-dampens weights\. PsiLogic instead computes one combined coefficient per step\.

Define the chaos warmup weightwt∈\[0,1\]w\_\{t\}\\in\[0,1\]\(ramps from 0 during an initial warmup window; Section[3\.5](https://arxiv.org/html/2607.16268#S3.SS5)\) and the per\-group spike maskst∈\{0,1\}s\_\{t\}\\in\\\{0,1\\\}from the chaos gate\. The raw cancellation fraction before clamping is

c~t=st⋅chaost⋅η⋅γ⋅Pk\.\\tilde\{c\}\_\{t\}=s\_\{t\}\\cdot\\mathrm\{chaos\}\_\{t\}\\cdot\\eta\\cdot\\gamma\\cdot P\_\{k\}\.\(6\)We clamp and add weight decay:

δt=η​λ\+wt⋅min⁡\(c~t,cmax\),cmax=max\_cancel\\delta\_\{t\}=\\eta\\lambda\+w\_\{t\}\\cdot\\min\\\!\\bigl\(\\tilde\{c\}\_\{t\},\\;c\_\{\\max\}\\bigr\),\\qquad c\_\{\\max\}=\\texttt\{max\\\_cancel\}\(7\)which is theδt\\delta\_\{t\}used in Eq\. \([1](https://arxiv.org/html/2607.16268#S3.E1)\)\. Intuitively,η​λ\\eta\\lambdais the AdamW decay contribution;wt⋅min⁡\(c~t,cmax\)w\_\{t\}\\cdot\\min\(\\tilde\{c\}\_\{t\},c\_\{\\max\}\)is chaos\-gated active cancellation scaled byPkP\_\{k\};cmaxc\_\{\\max\}\(default 0\.05\) caps per\-step shrinkage during volatile initialization\. Optional cosine schedules onγ\\gammaare supported viagamma\_T\_max\.

#### Optional quantum decay\.

Letq0≥0q\_\{0\}\\geq 0denote thequantum\_decayhyperparameter \(defaultq0=0q\_\{0\}=0disables the feature\)\. Whenq0\>0q\_\{0\}\>0, an effective rateqtq\_\{t\}is cosine\-scheduled over training alongsideγ\\gammaviagamma\_T\_max\(the same schedule helper used forγ\\gamma\)\. After computingδt\\delta\_\{t\}but before the Adam subtraction in Listing[1](https://arxiv.org/html/2607.16268#LST1), each coordinate is multiplied by

ρt,i=1−η​qt​wt​\(1−st\)​tanh⁡\(\|gt,i\|\),\\rho\_\{t,i\}=1\-\\eta\\,q\_\{t\}\\,w\_\{t\}\\,\(1\-s\_\{t\}\)\\,\\tanh\\\!\\bigl\(\|g\_\{t,i\}\|\\bigr\),\(8\)but only whenst=0s\_\{t\}=0, so auxiliary gradient\-dependent regularization does not stack with active cancellation on spike steps\. FairBench presets useq0=0q\_\{0\}=0unless noted\.

### 3\.5Chaos Warmup

Forchaos\_warmup=−1=\-1, the warmup horizon auto\-scales asmax⁡\(500,T/20\)\\max\(500,\\,T/20\)overTTtraining steps\. Whilet≤twarmt\\leq t\_\{\\mathrm\{warm\}\},wt=0w\_\{t\}=0; thenwtw\_\{t\}ramps linearly to11overtwarm/4t\_\{\\mathrm\{warm\}\}/4steps\. This prevents the chaos term from firing into raw from\-scratch gradient noise\.

### 3\.6Algorithm

Listing 1:PsiLogic \(simplified; matches Eqs\.[7](https://arxiv.org/html/2607.16268#S3.E7),[1](https://arxiv.org/html/2607.16268#S3.E1)\)\.fort=1\.\.\.T:

grad<\-nablaL\(theta\);optionallyapplyAGCandgradientcentralization

updateAdammomentsm,v;updatefast\_t,slow\_tfrom\|\|grad\|\|\_2

s\_t<\-spikemaskfromchaosgate

c\_t<\-min\(s\_t\*chaos\_t\*eta\*gamma\*P\_k,max\_cancel\)

delta<\-eta\*lambda\+w\_t\*c\_t

theta<\-theta\*\(1\-delta\)

theta<\-theta\*\(1\-eta\*q\_t\*w\_t\*\(1\-s\_t\)\*tanh\(abs\(grad\)\)\)

theta<\-theta\-eta\*m\_hat/\(sqrt\(v\_hat\)\+eps\)

### 3\.7Comparison with Baselines

Table 1:Feature comparison of optimizers\.

## 4FairBench Evaluation

### 4\.1Protocol and Arenas

All headline numbers come from one reference run onNVIDIA H100 80GB HBM3\(PyTorch 2\.4\.1\+cu124, CUDA 12\.4\), configuration frozen inbenchmark/results/full/config\.json\. FairBench uses a two\-stage protocol: \(1\) per\-optimizer LR sweep over 7 log\-spaced rates from10−510^\{\-5\}to10−210^\{\-2\}\(500 steps each\); \(2\) evaluation at the best LR for 2000 steps with seeds\{0,1,2\}\\\{0,1,2\\\}and identical initialization\. Shared settings: batch 64, bf16 AMP, grad clip 1\.0, cosine LR, 100\-step warmup\. Four arenas cover NLP \(Small GPT / TinyStories\), ViT\-Tiny on CIFAR\-100, ResNet\-18 on Tiny ImageNet, and DDPM on CelebA64264^\{2\}\. Full protocol and arena tables are in Appendix[A](https://arxiv.org/html/2607.16268#A1)\. PsiLogic uses fixed per\-arena presets;only LR is tuned, as for all baselines\.

### 4\.2Main Results

Table 2:Main FairBench results \(mean±\\pmstd over 3 seeds\)\. Best per row in bold\.Selected LRs\.NLP—all3\.16×10−43\.16\\times 10^\{\-4\}; ViT—Adam3\.16×10−53\.16\\times 10^\{\-5\}, AdamW/PsiLogic3\.16×10−43\.16\\times 10^\{\-4\}, Lion10−410^\{\-4\}; ResNet—Adam/Lion10−410^\{\-4\}, AdamW/PsiLogic3\.16×10−43\.16\\times 10^\{\-4\}; Diffusion— Adam/AdamW/PsiLogic10−310^\{\-3\}, Lion10−410^\{\-4\}\. Welchtt\-tests, compute costs, and per\-seed breakdowns are reported in Appendix[B](https://arxiv.org/html/2607.16268#A2)–[D](https://arxiv.org/html/2607.16268#A4)\.

### 4\.3Learning Curves and Overhead

![Refer to caption](https://arxiv.org/html/2607.16268v1/figures/vit_val_val_acc.png)\(a\)ViT val\. accuracy\.
![Refer to caption](https://arxiv.org/html/2607.16268v1/figures/nlp_val_perplexity.png)\(b\)NLP perplexity\.
![Refer to caption](https://arxiv.org/html/2607.16268v1/figures/resnet_val_val_acc.png)\(c\)ResNet top\-1 acc\.
![Refer to caption](https://arxiv.org/html/2607.16268v1/figures/vit_train_step_time_s.png)\(d\)ViT step time\.

Figure 1:FairBench learning curves \(mean±\\pmstd\) and ViT per\-step wall\-time overhead on H100\.PsiLogic reports the best validation metric on NLP, ViT, and ResNet in Table[2](https://arxiv.org/html/2607.16268#S4.T2)\. Against Adam, all three gains are statistically significant \(Appendix[B](https://arxiv.org/html/2607.16268#A2)\); against AdamW, ViT and NLP perplexity are significant, while ResNet is a numerical edge \(0\.2220\.222vs\.0\.2190\.219\) without significance at three seeds \(p=0\.44p=0\.44\), and diffusion remains tied\. Step\-time overhead reaches 1\.79×\\timeson ViT \(Figure[1](https://arxiv.org/html/2607.16268#S4.F1), panel d\); this gap is implementation\-bound rather than inherent to the chaos statistic \(Section[6](https://arxiv.org/html/2607.16268#S6.SS0.SSS0.Px4)\)\.

## 5Ablations and Component Analysis

Prior ablations on a synthetic MLP task \(v0\.3\.x\) showed that gradient centralization and adaptive gradient clipping each independently improve stability when combined with the chaos term\. A*mirror ablation*demonstrated that dynamically mirroring PsiLogic’s cancellation magnitude as AdamW weight decay does not fully reproduce PsiLogic’s per\-parameter behavior, indicating the chaos signal is not equivalent to a single global weight\-decay schedule\.

These ablations predate FairBench; component tests are maintained intests/\. Extended FairBench ablations \(γ\\gamma,max\_cancel,chaos\_warmup\) are planned\.

## 6Discussion

#### Why chaos damping helps\.

Large early gains on ViT \(0\.2440\.244vs\.0\.0790\.079Adam\) suggest the chaos term suppresses destructive early updates when gradient statistics are volatile\. Under fair LR tuning, NLP perplexity still favors PsiLogic over AdamW\.

#### Implicit warmup\.

The cancellation term reduces effective step size during chaotic phases, similar in spirit to LR warmup but driven by online gradient statistics rather than a fixed schedule—complementary to hypergradient and parameter\-free LR methods cited in Section[2](https://arxiv.org/html/2607.16268#S2)\.

#### Reproducibility\.

ResNet shows the lowest cross\-seed standard deviation among optimizers \(±0\.001\\pm 0\.001on accuracy\), which may matter for production training pipelines\.

#### Limitations \(stated explicitly\)\.

1. 1\.Small seed count—3 seeds; some comparisons \(ResNet vs\. AdamW, diffusion vs\. AdamW\) are not statistically significant\.
2. 2\.Short training budget—2000 steps per arena; not ImageNet\- or LLM\-scale\.
3. 3\.Step\-time overhead—up to 1\.79×\\timesvs\. AdamW on ViT\. The chaos detector tracks only scalar gradient statistics; the measured wall\-clock gap is*implementation\-bound*—sequential element\-wise PyTorch ops and the lack of fusedforeachCUDA kernels in the reference build—not an inherent theoretical cost\. Future releases can reduce overhead via kernel fusion \(e\.g\., ATen or Triton\)\.
4. 4\.Diffusion—no quality win over Adam/AdamW at this budget\.
5. 5\.No convergence proof—empirical stability only\.
6. 6\.Independent evaluation—results have not yet been replicated by external groups\.

## 7Reproducibility Statement

gitclonehttps://github\.com/Troxter222/psilogic

cdpsilogic&&pipinstall\-e"\.\[benchmark\]"&&pipinstall\-rbenchmark/requirements\.txt

cdbenchmark

python\-mfairbench\.download\-\-data\-root\./data

python\-mfairbench\-\-data\-root\./data\-\-output\-dirresults/full

Reference outputs:benchmark/results/full/\{aggregate,summary,significance\}\.csv Software DOI:10\.5281/zenodo\.18739857PyPI:pip install psilogic

## 8Conclusion

PsiLogic augments Adam with a chaos\-gated Active Cancellation term that is strong during unstable training and vanishes at convergence\. Under FairBench on NVIDIA H100, it achieves the best validation metric in three of four cross\-domain arenas, with honest reporting of ties and overhead\. Future work: reduce step\-time cost, increase seed count and training length, and seek independent replication at scale\.

## References

- \[1\]A\. G\. Baydin, R\. Cornish, M\. Rubinstein, and D\. M\. Wood\. Online learning rate adaptation with hypergradient descent\.*ICLR*, 2018\.
- \[2\]A\. Brock, et al\. High\-performance large\-scale image recognition without normalization\.*ICML*, 2021\.
- \[3\]X\. Chen, C\. Liang, D\. Huang, E\. Real, K\. Wang, Y\. Liu, et al\. Symbolic discovery of optimization algorithms\.*NeurIPS*, 2023\.
- \[4\]M\. Defazio and K\. Mishchenko\. Learning\-rate\-free learning by D\-Adaptation\.*ICML*, 2023\.
- \[5\]P\. Goyal, et al\. Accurate, large minibatch SGD\.*arXiv preprint*[arXiv:1706\.02677](https://arxiv.org/abs/1706.02677), 2017\.
- \[6\]V\. Gupta, T\. Koren, and Y\. Singer\. Shampoo: Preconditioned stochastic tensor optimization\.*ICML*, 2018\.
- \[7\]D\. P\. Kingma and J\. Ba\. Adam: A method for stochastic optimization\.*ICLR*, 2015\.
- \[8\]H\. Liu, Z\. Shen, Y\. Li, S\. Lin, K\. Wang, and L\. Ma\. Sophia: A scalable stochastic second\-order optimizer\.*ICLR*, 2024\.
- \[9\]I\. Loshchilov and F\. Hutter\. Decoupled weight decay regularization\.*ICLR*, 2019\.
- \[10\]K\. Mishchenko and M\. Defazio\. Prodigy: An expeditiously adaptive parameter\-free learner\.*arXiv preprint*[arXiv:2306\.06169](https://arxiv.org/abs/2306.06169), 2023\.
- \[11\]N\. Shazeer and M\. Stern\. Adafactor: Adaptive learning rates with sublinear memory cost\.*ICML*, 2018\.
- \[12\]H\. Yong, J\. Huang, X\. Hua, and L\. Zhang\. Gradient centralization\.*ECCV*, 2020\.
- \[13\]Y\. You, I\. Gitman, and B\. Ginsburg\. Large batch training of convolutional networks with layer\-wise adaptive rate scaling\.*arXiv preprint*[arXiv:1708\.03888](https://arxiv.org/abs/1708.03888), 2017\.
- \[14\]Y\. You, J\. Li, S\. Reddi, J\. Hseu, S\. Kumar, S\. Bhojanapalli, X\. Song, J\. Demmel, and C\.\-J\. Hsieh\. Large batch optimization for deep learning: Training BERT in 76 minutes\.*ICLR*, 2020\.

## Appendix AFairBench Protocol and Arenas

Table 3:FairBench protocol stages\.Table 4:FairBench arenas\.
## Appendix BStatistical Significance

Table 5:Welchtt\-test: PsiLogic vs\. baseline\.p∗<0\.05\{\}^\{\*\}p<0\.05,p∗∗<0\.01\{\}^\{\*\*\}p<0\.01,p∗⁣∗∗<0\.001\{\}^\{\*\*\*\}p<0\.001; n\.s\. = not significant\.
## Appendix CCompute Cost

Table 6:Compute cost on H100\. A/W/L/P = Adam/AdamW/Lion/PsiLogic\.VRAM differences are≤3%\\leq 3\\%except Lion on ResNet/NLP \(lower\)\.

## Appendix DPer\-Seed Results

Table 7:Per\-seed ViT validation accuracy\.Full per\-seed tables for all arenas:benchmark/results/full/summary\.csv\.

## Appendix EArchived Experiments

Pre\-FairBench results \(CIFAR\-10 A40, BERT, AG News, etc\.\) are archived inOLD\_RESULTS\.mdand arenotused for claims in this preprint\.