muon-optimizer

Tag

Cards List
#muon-optimizer

@alacheng: Few teams document the real decision-making process behind large-scale training this clearly — why each component was adopted or rejected, how risks were managed, how much efficiency improved, all backed by data and charts. Highly valuable for anyone actually training large models. Marin 535B-A23B is a 535B total parameter, 23B active M…

X AI KOLs Timeline ↗ · 14h ago Cached

Commentary on the Marin 535B-A23B technical report: this 535B total / 23B active MoE model was trained to completion on 18T tokens. The report clearly documents the real decision-making process behind large-scale training, component trade-offs, and risk management. Through small-scale experiments and a scaling ladder, loss was predicted in advance, and the final run outperformed predictions while remaining stable throughout.

0 favorites 0 likes
#muon-optimizer

Delayed Effects of Temporary Weight Decay in Muon Pretraining

arXiv cs.CL ↗ · 18h ago Cached

Meta and Mila researchers show that temporary weight decay during Muon-pretrained Transformers can appear harmful at removal yet improve final loss, demonstrating delayed effects driven by persistent weight-norm deficits that should be evaluated at the full training budget.

0 favorites 0 likes
#muon-optimizer

When and How Does Muon Precondition? The Role of Structured Gradient Noise

arXiv cs.LG ↗ · 2026-10-27 Cached

This paper analyzes when and how the Muon optimizer provides preconditioning benefits, showing it offers no acceleration under isotropic noise but can reduce task-frequency dependence from linear to square-root under structured mini-batch gradient noise, while avoiding SignSGD's rotation-induced negative transfer.

0 favorites 0 likes
#muon-optimizer

How Does Orthogonalization Adapt to the Neural Network Hessian Structure? A Gradient Self Outer-Product Analysis at Initialization

arXiv cs.LG ↗ · 2026-10-22 Cached

The paper provides an asymptotic theory explaining why Muon's orthogonalization preconditioner matches the neural network Hessian structure at initialization, showing that overparameterization drives diagonal dominance of E[GG^T] across four canonical models.

0 favorites 0 likes
#muon-optimizer

Long-Horizon RL Needs Muon

Hugging Face Daily Papers ↗ · 2026-10-14 Cached

A paper examining a single-rollout asynchronous RL recipe combining the Muon optimizer, token-averaged loss, truncated importance sampling, and auxiliary penalties for training long-horizon repository-level issue resolution agents, showing improved training success over baselines across two model families.

0 favorites 0 likes
#muon-optimizer

Temperon: Full-Time SAM Quality at a Third Less Wall-Clock

arXiv cs.LG ↗ · 2026-09-17 Cached

Temperon is a training method that uses Sharpness-aware minimization (SAM) only in the final phase of training to achieve full SAM quality with a third less wall-clock time, validated on vision and language tasks.

0 favorites 0 likes
#muon-optimizer

FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon

arXiv cs.LG ↗ · 2026-09-10 Cached

FedSubMuon is a communication-efficient federated fine-tuning method for LLMs that optimizes compact coefficient matrices within structured subspaces to reduce upload costs while maintaining strong performance.

0 favorites 0 likes
#muon-optimizer

LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes

Hugging Face Daily Papers ↗ · 2026-09-03 Cached

LLaDA-Image presents a unified framework that combines a 6B diffusion transformer with a frozen vision-language module for generating photorealistic images with precise editing, achieving state-of-the-art results among open-source models through efficient training and fast inference.

0 favorites 0 likes
#muon-optimizer

Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimization

arXiv cs.LG ↗ · 2026-08-28 Cached

This paper analyzes how finite Newton-Schulz iterations in the Muon optimizer benefit nonsmooth nonconvex optimization by smoothing the polar map, providing convergence guarantees that match best-known bounds.

0 favorites 0 likes
#muon-optimizer

Scaling Muon for Diffusion Transformers

arXiv cs.LG ↗ · 2026-08-24 Cached

This paper scales the Muon optimizer for Diffusion Transformers from 1.3B to 15B parameters, introducing Periodic Row-wise Muon to reduce computational overhead while preserving generative quality improvements over AdamW.

0 favorites 0 likes
#muon-optimizer

Federated Compositional Muon Optimizer for Matrix-Wise Models

arXiv cs.LG ↗ · 2026-08-14 Cached

This paper proposes FedCoMuon, a federated compositional Muon optimizer for matrix-wise models, along with a variance-reduced variant (FedCoMuon-VR). The authors provide convergence analysis under non-i.i.d. and non-convex settings, showing improved sample complexity over existing FedMuon algorithms, and demonstrate competitive performance on robust federated learning and task-distributed risk-sensitive meta learning.

0 favorites 0 likes
#muon-optimizer

The Active Ingredient in Muon's Grokking

arXiv cs.LG ↗ · 2026-07-24 Cached

This paper ablates the Muon optimizer to find that orthogonalization (Newton-Schulz iteration), not spectral scaling, is the key ingredient behind its faster grokking on modular arithmetic, and introduces a stability-aware metric for measuring grokking speed.

0 favorites 0 likes
#muon-optimizer

How Much Orthogonalization Does Muon Need?

arXiv cs.LG ↗ · 2026-06-02 Cached

This paper studies how much orthogonalization the Muon optimizer requires, proposing a five-step cubic Newton-Schulz schedule that reduces computational cost while achieving training quality similar to more expensive methods across GPT-2 Small and hybrid MoE/Mamba models.

0 favorites 0 likes
#muon-optimizer

@zhaoran_wang: for me, the coolest finding is that you can connect/interpolate all softmax/linear variants and give a promising direct…

X AI KOLs Timeline ↗ · 2026-05-30 Cached

Discussion of a finding that all softmax/linear attention variants can be interpolated, and that the Muon optimizer is crucial for Parallax to move beyond Softmax Attention. Includes link to paper and code.

0 favorites 0 likes
#muon-optimizer

SignMuon: Communication-Efficient Distributed Muon Optimization

arXiv cs.LG ↗ · 2026-05-19 Cached

SignMuon is a 1-bit, matrix-aware optimizer for distributed training that combines signSGD's majority-vote sign aggregation with Muon's polar-step framework, achieving 32x bandwidth reduction over float32 while maintaining strong convergence and performance on benchmarks like CIFAR-10/ResNet-50 and nanoGPT.

0 favorites 0 likes
← Back to home

Submit Feedback