DynMuon: A Dynamic Spectral Shaping View of Muon
Summary
This paper introduces DynMuon, a dynamic spectral shaping optimizer that schedules the update parameter p from positive to mildly negative during training, consistently achieving lower validation loss and requiring 10.6-26.5% fewer steps than the standard Muon optimizer.
View Cached Full Text
Cached at: 05/22/26, 02:36 AM
Paper page - DynMuon: A Dynamic Spectral Shaping View of Muon
Source: https://huggingface.co/papers/2605.17109
Abstract
Muon optimizer’s spectral-shaping approach dynamically adjusts update parameters during training to improve convergence, achieving better validation loss with fewer training steps.
In recent years,Muonhas emerged as the dominant method for training large language models, and transformers more broadly. The essential difference, when compared to standardgradient descentmethods, is to replace the usualupdate matrixM=UΣV^top with itspolar factorUV^top. In this work, we consider a class ofMuon-like updates, where we replace the update M with UΣ^p V^top for some parameter p. We call this a “spectral-shaping” operation, and develop a theory of how to pick p which depends on (a) local curvature of theloss function, (b) noise stemming fromstochastic gradientsand label noise, and (c)training stage. Our theory and experimentation reveal a previously overlooked behavior: positive p helps early by emphasizing high-curvature directions and accelerating signal contraction, while mildly negative p helps later by reallocating update strength toward low-curvature directions that still contain useful training signals. Building on the insight, we proposeDynMuon, an efficient dynamic spectral shaping method that schedules p from positive to mildly negative over training. Extensive experiments across model sizes, architectures, and training settings show thatDynMuonconsistently achieves lower validation loss thanMuon, while requiring 10.6-26.5% fewer steps to reach the same target loss.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.17109 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.17109 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.17109 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
DE-Venus: A Data-Efficient RLVR Framework for Large Language Models
DE-Venus is a unified framework for data-efficient Reinforcement Learning with Verifiable Rewards in large language models, reducing annotation and training costs while maintaining performance.
Language-encoded network topology enables large language models to reason about complex networks
BioGlyph is a method that translates network topology into interpretable language roles to enhance large language models' ability to reason about complex networks, demonstrating significant performance improvements across various domains.
Frontier LLMs are effective batch optimizers: Assessing reasoning models in continuous and discrete settings
This research paper evaluates frontier LLMs as batch optimizers in both continuous and discrete settings, finding them competitive in numerical tasks but more effective in semantically rich environments compared to classical methods.
STAIR (STructure Aware Information Retriever): A novel dataset and LLM based retriever for document structure augmentation
STAIR introduces a structure-aware retrieval system that leverages Table of Contents for enhanced document retrieval in Large Language Models, achieving high performance on the new SearchTome benchmark dataset.
Analysis of Prompt Engineering for Drug Toxicity Prediction
This paper investigates the impact of prompt engineering on drug toxicity prediction using large language models, finding that natural variance in LLM outputs outweighs prompt fine-tuning, while chemoinformatic feature extraction improves model performance.