Can Muon Fine-tune Adam-Pretrained Models?
Summary
Research paper investigating performance degradation when using the Muon optimizer instead of Adam for fine-tuning pretrained models, demonstrating that parameter-efficient methods like LoRA effectively mitigate this optimizer mismatch across language and vision tasks.
View Cached Full Text
Cached at: 05/12/26, 10:52 AM
Paper page - Can Muon Fine-tune Adam-Pretrained Models?
Source: https://huggingface.co/papers/2605.10468
Abstract
Optimizer mismatch between Adam and Muon during fine-tuning degrades performance due to differing implicit biases, but this can be mitigated using parameter-efficient fine-tuning methods like LoRA.
Muonhas emerged as an efficient alternative toAdamfor pretraining, yet remains underused forfine-tuning. A key obstacle is that most open models are pretrained withAdam, and naively switching toMuonforfine-tuningleads to degraded performance due to anoptimizer mismatch. We investigate this mismatch through controlled experiments and relate it to the distinctimplicit biasesofAdamandMuon. We provide evidence that the mismatch disrupts pretrained knowledge, and that this disruption scales with update strength. This leads us to hypothesize that constraining updates should mitigate the mismatch. We validate this withLoRA: across language and vision tasks,LoRAreduces the performance gap betweenAdamandMuonobserved under fullfine-tuning. Studies onLoRArank,catastrophic forgetting, andLoRAvariants further confirm that mismatch severity correlates with update strength. These results shed light on howoptimizer mismatchaffectsfine-tuningand how it can be mitigated. Our code is available at https://github.com/XingyuQu/muon-finetune.
View arXiv pageView PDFGitHub2Add to collection
Get this paper in your agent:
hf papers read 2605\.10468
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.10468 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.10468 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.10468 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Why Muon Outperforms Adam: A Curvature Perspective
This paper investigates why the Muon optimizer outperforms Adam in large language model training, showing from a curvature perspective that Muon incurs a smaller curvature penalty due to lower normalized directional sharpness, with advantages amplified by data imbalance.
When Does Muon Help Agentic Reinforcement Learning?
This paper investigates the use of the Muon optimizer in reinforcement learning post-training, finding that applying Muon to hidden weight matrices significantly improves success rates on ALFRED tasks compared to AdamW, with results dependent on the advantage estimator and learning rate.
Reassessing Muon for Matrix Factorization
This paper evaluates the Muon optimizer on low-rank matrix factorization, finding it does not consistently outperform AdamW, challenging earlier claims about its advantages in large-scale deep learning.
@burny_tech: some updates on the optimizer magic
A new NVIDIA paper proposes higher-order optimizers like Muon and SOAP as more efficient alternatives to AdamW for large-scale LLM pretraining.
Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR
This paper introduces Pion, a new optimizer that replaces Muon's spectral whitening with a high-pass NS iteration to stabilize training in low-rank and low-SNR regimes, achieving improved performance in VLA and RLVR tasks.