@burny_tech: some updates on the optimizer magic
Summary
A new NVIDIA paper proposes higher-order optimizers like Muon and SOAP as more efficient alternatives to AdamW for large-scale LLM pretraining.
View Cached Full Text
Cached at: 07/25/26, 10:04 AM
some updates on the optimizer magic
alphaXiv (@askalphaxiv): “SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales”
This new NVIDIA paper suggests that AdamW may not be sufficient for frontier LLM pretraining anymore!
In the paper, they show that higher-order optimizers like Muon and SOAP can train large LLMs more efficiently at
Similar Articles
SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
This paper improves higher-order optimizers SOAP and Muon for large-scale LLM pretraining, addressing instabilities at large batch sizes and introducing a layer-wise distributed optimizer compatible with Megatron-LM. Experiments show they consistently outperform AdamW at billion-parameter scales.
Can Muon Fine-tune Adam-Pretrained Models?
Research paper investigating performance degradation when using the Muon optimizer instead of Adam for fine-tuning pretrained models, demonstrating that parameter-efficient methods like LoRA effectively mitigate this optimizer mismatch across language and vision tasks.
Optimistic Dual Averaging Unifies Modern Optimizers
This paper introduces SODA, a generalization of Optimistic Dual Averaging that unifies various modern optimizers like Muon and Lion. It proposes a practical wrapper that improves performance across different scales without requiring additional hyperparameter tuning for weight decay.
@omarsar0: Banger compression paper from NVIDIA. (bookmark it) Bigger MoE models keep winning on quality, but serving them at inte…
NVIDIA's paper introduces Puzzle-75B-A9B, a compressed hybrid MoE model that doubles server throughput while preserving quality, enabling cost-effective deployment of large language models.
Why Muon Outperforms Adam: A Curvature Perspective
This paper investigates why the Muon optimizer outperforms Adam in large language model training, showing from a curvature perspective that Muon incurs a smaller curvature penalty due to lower normalized directional sharpness, with advantages amplified by data imbalance.