Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws
Summary
This paper shows that different optimizers, particularly Muon versus AdamW, induce distinct spectral scaling behaviors in Transformer models, with Muon achieving significantly better utilization of representation capacity, and suggests that optimizer choice should be a first-class axis in scaling laws.
View Cached Full Text
Cached at: 05/22/26, 02:20 AM
Paper page - Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws
Source: https://huggingface.co/papers/2605.21803
Abstract
Different optimizers produce distinct spectral scaling behaviors in Transformer models, with Muon achieving superior scaling efficiency compared to AdamW in representation capacity utilization.
Scaling laws have made language-model performance predictable from model size, data, and compute, but they typically treat theoptimizeras a fixed training detail. We show that this assumption misses a fundamental axis of representation scaling: how effectively theoptimizerconverts added FFN width into utilized spectral capacity. Usingeigenspectraoffeed-forward networkrepresentations, measured through soft andhard spectral-ranks, we find that the sameTransformer architecturerealizes markedly differentspectral scaling lawswhen trained with differentoptimizers. Holding architecture and width schedule fixed,AdamWexhibits weak hard-rank scaling (β=0.44) on rare-token (TAIL) representations where learning is known to be hardest, whereasMuonachieves linear scaling (β=1.02) in the same regimes, a 2.3times increase in the scaling exponent. This difference is not reducible to validation loss:AdamWconfigurations can match low-rank Dion variants in perplexity, under extended training, while exhibiting sharply different spectral geometry, demonstrating that matched loss does not imply matched representation structure. Hard--soft rank asymmetry further reveals thatoptimizers differ not only in how much capacity is realized, but also in how that capacity is structured across eigenmodes. To disentangleoptimizereffects from architectural ones, we compare against architectural interventions (e.g.,attention rankandpositional encoding), and find thatoptimizer-induced spectral shifts often exceed the architectural effects. These results suggest optimization as a first-class axis of representation scaling, motivatingoptimizer--architecture co-design.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2605\.21803
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.21803 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.21803 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.21803 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Microsoft tested a coding-agent memory plugin. It cost more at first, then paid for itself.
Microsoft Research and UC Berkeley found Claude Code repeated 30% of tool calls after five tasks in a repo, prompting them to release TokenEconomizer, an open-source plugin that carries lessons between sessions and trims available tools. It raised initial costs but dropped per-task cost 18.4-30.9% from the sixth task onward, improving success in four of five agent-model combinations.
Hyperscale data centers use 2% of US electricity
A Nature Sustainability study finds US hyperscale data centers consumed roughly 1-2% of the country's electricity, emitting 21-31 million metric tons of CO2 annually, with fossil fuels dominating their energy mix and on-site generation growing rapidly.
US data centers now burn more electricity than a midsized country
A Nature Sustainability study finds 403 US hyperscale data centers consumed about 82 terawatt-hours of electricity in a year — more than 148 countries individually — with roughly 54% coming from fossil fuels, producing 25.7 million metric tons of CO2.
@omarsar0: Recommend reading following up on the idea of weight-space programming. Bigger models (Opus 5.5) tend to do better and …
A recommendation post highlighting Microsoft and colleagues' research on weight-space programming, where Claude Opus 5.5 writes transformer weights directly in code with no gradient training, with bigger models performing better and more efficiently.
@alacheng: Few teams document the real decision-making process behind large-scale training this clearly — why each component was adopted or rejected, how risks were managed, how much efficiency improved, all backed by data and charts. Highly valuable for anyone actually training large models. Marin 535B-A23B is a 535B total parameter, 23B active M…
Commentary on the Marin 535B-A23B technical report: this 535B total / 23B active MoE model was trained to completion on 18T tokens. The report clearly documents the real decision-making process behind large-scale training, component trade-offs, and risk management. Through small-scale experiments and a scaling ladder, loss was predicted in advance, and the final run outperformed predictions while remaining stable throughout.