Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws

Hugging Face Daily Papers Papers

Summary

This paper shows that different optimizers, particularly Muon versus AdamW, induce distinct spectral scaling behaviors in Transformer models, with Muon achieving significantly better utilization of representation capacity, and suggests that optimizer choice should be a first-class axis in scaling laws.

Scaling laws have made language-model performance predictable from model size, data, and compute, but they typically treat the optimizer as a fixed training detail. We show that this assumption misses a fundamental axis of representation scaling: how effectively the optimizer converts added FFN width into utilized spectral capacity. Using eigenspectra of feed-forward network representations, measured through soft and hard spectral-ranks, we find that the same Transformer architecture realizes markedly different spectral scaling laws when trained with different optimizers. Holding architecture and width schedule fixed, AdamW exhibits weak hard-rank scaling (β=0.44) on rare-token (TAIL) representations where learning is known to be hardest, whereas Muon achieves linear scaling (β=1.02) in the same regimes, a 2.3times increase in the scaling exponent. This difference is not reducible to validation loss: AdamW configurations can match low-rank Dion variants in perplexity, under extended training, while exhibiting sharply different spectral geometry, demonstrating that matched loss does not imply matched representation structure. Hard--soft rank asymmetry further reveals that optimizers differ not only in how much capacity is realized, but also in how that capacity is structured across eigenmodes. To disentangle optimizer effects from architectural ones, we compare against architectural interventions (e.g., attention rank and positional encoding), and find that optimizer-induced spectral shifts often exceed the architectural effects. These results suggest optimization as a first-class axis of representation scaling, motivating optimizer--architecture co-design.
Original Article
View Cached Full Text

Cached at: 05/22/26, 02:20 AM

Paper page - Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws

Source: https://huggingface.co/papers/2605.21803

Abstract

Different optimizers produce distinct spectral scaling behaviors in Transformer models, with Muon achieving superior scaling efficiency compared to AdamW in representation capacity utilization.

Scaling laws have made language-model performance predictable from model size, data, and compute, but they typically treat theoptimizeras a fixed training detail. We show that this assumption misses a fundamental axis of representation scaling: how effectively theoptimizerconverts added FFN width into utilized spectral capacity. Usingeigenspectraoffeed-forward networkrepresentations, measured through soft andhard spectral-ranks, we find that the sameTransformer architecturerealizes markedly differentspectral scaling lawswhen trained with differentoptimizers. Holding architecture and width schedule fixed,AdamWexhibits weak hard-rank scaling (β=0.44) on rare-token (TAIL) representations where learning is known to be hardest, whereasMuonachieves linear scaling (β=1.02) in the same regimes, a 2.3times increase in the scaling exponent. This difference is not reducible to validation loss:AdamWconfigurations can match low-rank Dion variants in perplexity, under extended training, while exhibiting sharply different spectral geometry, demonstrating that matched loss does not imply matched representation structure. Hard--soft rank asymmetry further reveals thatoptimizers differ not only in how much capacity is realized, but also in how that capacity is structured across eigenmodes. To disentangleoptimizereffects from architectural ones, we compare against architectural interventions (e.g.,attention rankandpositional encoding), and find thatoptimizer-induced spectral shifts often exceed the architectural effects. These results suggest optimization as a first-class axis of representation scaling, motivatingoptimizer--architecture co-design.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2605\.21803

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.21803 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.21803 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.21803 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Microsoft tested a coding-agent memory plugin. It cost more at first, then paid for itself.

Reddit r/artificial

Microsoft Research and UC Berkeley found Claude Code repeated 30% of tool calls after five tasks in a repo, prompting them to release TokenEconomizer, an open-source plugin that carries lessons between sessions and trims available tools. It raised initial costs but dropped per-task cost 18.4-30.9% from the sixth task onward, improving success in four of five agent-model combinations.

Hyperscale data centers use 2% of US electricity

Ars Technica

A Nature Sustainability study finds US hyperscale data centers consumed roughly 1-2% of the country's electricity, emitting 21-31 million metric tons of CO2 annually, with fossil fuels dominating their energy mix and on-site generation growing rapidly.

US data centers now burn more electricity than a midsized country

The Verge

A Nature Sustainability study finds 403 US hyperscale data centers consumed about 82 terawatt-hours of electricity in a year — more than 148 countries individually — with roughly 54% coming from fossil fuels, producing 25.7 million metric tons of CO2.

@alacheng: Few teams document the real decision-making process behind large-scale training this clearly — why each component was adopted or rejected, how risks were managed, how much efficiency improved, all backed by data and charts. Highly valuable for anyone actually training large models. Marin 535B-A23B is a 535B total parameter, 23B active M…

X AI KOLs Timeline

Commentary on the Marin 535B-A23B technical report: this 535B total / 23B active MoE model was trained to completion on 18T tokens. The report clearly documents the real decision-making process behind large-scale training, component trade-offs, and risk management. Through small-scale experiments and a scaling ladder, loss was predicted in advance, and the final run outperformed predictions while remaining stable throughout.