When Does Muon Help Agentic Reinforcement Learning?

Hugging Face Daily Papers Papers
reinforcement-learning muon optimizer agentic-rl group-in-group-policy-optimization sparse-reward arxiv

Summary

This paper investigates the use of the Muon optimizer in reinforcement learning post-training, finding that applying Muon to hidden weight matrices significantly improves success rates on ALFRED tasks compared to AdamW, with results dependent on the advantage estimator and learning rate.

Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly. Multi-seed and cross-task validation remain open.
Original Article
View Cached Full Text

Cached at: 07/20/26, 09:39 AM

Paper page - When Does Muon Help Agentic Reinforcement Learning?

Source: https://huggingface.co/papers/2607.16169

Abstract

MuoniscompetitivewithAdamWinlarge-scalepre-training,butitsvalueforreinforcement-learning(RL)post-trainingremainsunclear.WestudyvanillaMuoninsparse-rewardagenticRLthroughmatchedsingle-seedcomparisonswithAdamWonALFWorldusingQwen2.5-0.5B-Instruct.UnderGroup-in-GroupPolicyOptimization(GiGPO),applyingMuononlytohiddenweightmatricesraisesfinal-windowvalidationsuccessfrom0.290to0.546(+88%);high-rateAdamWcontrolsretainnopost-updatesuccess.Theeffectdependsontheadvantageestimatorandlearningrate.At3e-5,MuonimprovesGRPOfrom0.161to0.268,whereasGraphGPO’slate-windowgapnarrowsnearsaturation.At1e-5,GraphGPOMuonreaches0.901,raisesnormalizedvalidationAUCfrom0.399to0.556,andreaches0.5and0.75success30and60updatesearlier,respectively.TheseexploratoryresultsshowthatMuoncanbenefitagenticRLandmotivatestudyingthepolicyoptimizer,advantageestimator,andlearningratejointly.Multi-seedandcross-taskvalidationremainopen.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.16169

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.16169 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.16169 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.16169 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Reassessing Muon for Matrix Factorization

arXiv cs.LG

This paper evaluates the Muon optimizer on low-rank matrix factorization, finding it does not consistently outperform AdamW, challenging earlier claims about its advantages in large-scale deep learning.

Can Muon Fine-tune Adam-Pretrained Models?

Hugging Face Daily Papers

Research paper investigating performance degradation when using the Muon optimizer instead of Adam for fine-tuning pretrained models, demonstrating that parameter-efficient methods like LoRA effectively mitigate this optimizer mismatch across language and vision tasks.

Why Muon Outperforms Adam: A Curvature Perspective

Hugging Face Daily Papers

This paper investigates why the Muon optimizer outperforms Adam in large language model training, showing from a curvature perspective that Muon incurs a smaller curvature penalty due to lower normalized directional sharpness, with advantages amplified by data imbalance.

How Much Orthogonalization Does Muon Need?

arXiv cs.LG

This paper studies how much orthogonalization the Muon optimizer requires, proposing a five-step cubic Newton-Schulz schedule that reduces computational cost while achieving training quality similar to more expensive methods across GPT-2 Small and hybrid MoE/Mamba models.

SignMuon: Communication-Efficient Distributed Muon Optimization

arXiv cs.LG

SignMuon is a 1-bit, matrix-aware optimizer for distributed training that combines signSGD's majority-vote sign aggregation with Muon's polar-step framework, achieving 32x bandwidth reduction over float32 while maintaining strong convergence and performance on benchmarks like CIFAR-10/ResNet-50 and nanoGPT.