@RichardYRLi: Muon erases spectral anisotropy, ISO inherits it - in current LLM-RLVR, inherit wins if RLVR mostly elicits from pretra…
Summary
The thread introduces ISO, an isospectral optimizer that preserves spectral anisotropy, contrasting with Muon which erases it, and discusses its implications for LLM-RLVR training and continual learning.
View Cached Full Text
Cached at: 07/24/26, 07:17 PM
Muon erases spectral anisotropy, ISO inherits it - in current LLM-RLVR, inherit wins if RLVR mostly elicits from pretraining. Open: an optimizer for controlled spectral motion – slow spectrum, fast frames – that writes new capability stably. necessary for “continual learning”?
great insight for structural awareness in optimizer design @zhu_hanqing666
Similar Articles
ISO: An RLVR-Native Optimization Stack
This paper studies the optimization layer for reinforcement learning with verifiable rewards (RLVR), proposing Isospectral Optimization (ISO) — a fixed-spectrum framework that reuses base model weight spectra while optimizing input/output singular frames. ISO-Merger and ISO-AdamW achieve strong performance with fewer training steps on reasoning and coding tasks.
Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR
This paper introduces Pion, a new optimizer that replaces Muon's spectral whitening with a high-pass NS iteration to stabilize training in low-rank and low-SNR regimes, achieving improved performance in VLA and RLVR tasks.
Muon is Not That Special: Random or Inverted Spectra Work Just as Well
This paper challenges the geometric justification for the Muon optimizer, arguing that precise structure is less important than step-size optimality. It introduces Freon and Kaon optimizers to demonstrate that random or inverted spectra can perform as well as Muon.
Spectral Scaling Laws of Muon
This paper presents the first systematic study of singular value spectral behavior in Muon optimizer momentum matrices during LLM training, discovering clean power-law scaling relationships across model sizes (77M–2.8B parameters). The findings provide practitioners with principled, layer-aware guidelines for configuring Newton–Schulz iterations to maintain orthonormalization quality at frontier scale without unnecessary computation.
MuCon: Clipped Muon Updates for LLM Training
This paper introduces MuCon, a clipped-Muon optimizer for LLM training that applies singular-value clipping instead of full polarization, preserving smaller singular values while clipping only the largest ones. It explores approximations to avoid full SVD, including polar/absolute-value formulas and rational Newton filters, noting numerical challenges near the threshold.