Tag
The paper proposes Muon-RT, a momentum-free variant of the Muon optimizer that reuses the EMA teacher-student weight displacement in LLM self-distillation, eliminating separate momentum buffers while improving accuracy and reducing persistent training memory on Qwen2.5-7B and Olmo-3-7B.
FARO is a function-space adaptive optimizer that assigns each parameter block a geometry based on its counterfactual return and reallocates the per-block update budget via cross-block equal risk contribution, matching hand-designed recipes like NorMuonAndAdam while providing robust small loss gains on GPT-style training.
A developer shares ApexW, an experimental PyTorch optimizer using compact optimizer states and a Triton-based execution path, claiming ~1.5x faster optimizer steps and halved optimizer-state memory on large-tensor workloads — while openly noting regressions on many-small-tensor models and minimal end-to-end gains.
RSI-Opt 通过元智能体在多次训练运行间累积并进化优化器策略（提示、流程、代码库与决策节奏），显著提升 LLM 优化器智能体在模型训练与科学机器学习任务上的性能，并支持跨模型策略迁移。
Tauon is a new optimizer using polynomial and orthogonalization techniques that outperforms Muon and AdamW in initial benchmarks on a small GPT-Mini model, showing lower loss and faster step time.
DSPy 3.4.0 has been released, featuring native support for Jev and System One models along with a new optimizer, ReAnchor, for calibrating outputs with confidence.
DSPy 3.4.0 is released with native support for Jev and System one models and a new optimizer, ReAnchor, for calibrating outputs with confidence.
This paper introduces QHAdamW, a modified optimizer for artificial neural networks, applied to air quality forecasting in the Philippines, showing improved convergence and performance in predicting PM2.5 and PM10 levels.
The paper proposes DeltaMomentum, a key-value based momentum update rule that adapts forgetting rates based on input frequency, demonstrating faster convergence in neural network training across various scales.
Dion3 is a revised Muon optimizer that reduces computational and communication overhead via Gram Newton-Schulz, symmetric GEMM kernels, and megabatching, achieving up to 6x faster optimizer steps while matching or improving loss.
This paper introduces MESH, a memory-efficient Sinkhorn-based optimizer for Mixture-of-Experts (MoE) training that restores temporal momentum without storing full optimizer state, reducing memory by 62.5% while maintaining competitive evaluation loss compared to AdamW.
This paper proposes Joint Affine Spectral Shaping (JRI), which extends weight-only spectral optimizers like Muon to jointly update weight and bias in affine layers, showing small but consistent accuracy improvements on a BERT-mini IMDb classification task.
A new NVIDIA paper proposes higher-order optimizers like Muon and SOAP as more efficient alternatives to AdamW for large-scale LLM pretraining.
This paper introduces CvAdamW, an AdamW variant that monitors attention specific heat to detect grokking phase transitions and dynamically scales weight decay, achieving grokking on modular arithmetic where baseline fails.
SkewAdam is a tiered optimizer that reduces memory usage for MoE state by 97%, enabling a 6.7B MoE model to fit on a single 40GB GPU.
Introduces PsiLogic, a chaos-aware optimizer that augments Adam with a dynamic damping term based on gradient instability, and proposes FairBench for reproducible evaluation. Shows competitive or superior results on NLP, ViT, and ResNet tasks with full transparency on limitations.
SkewAdam is a novel optimizer for mixture-of-experts models that tier allocates optimizer state across backbone, experts, and router, reducing memory footprint to 2.6% of AdamW while achieving better validation perplexity in controlled comparisons.
This paper investigates the use of the Muon optimizer in reinforcement learning post-training, finding that applying Muon to hidden weight matrices significantly improves success rates on ALFRED tasks compared to AdamW, with results dependent on the advantage estimator and learning rate.
This paper evaluates the Muon optimizer on low-rank matrix factorization, finding it does not consistently outperform AdamW, challenging earlier claims about its advantages in large-scale deep learning.
Introduces M+Adam, a novel optimizer combining additive and multiplicative updates to enable effective low-precision training of LLMs using BF16, FP8, or FP4 master weights, avoiding failure modes of standard optimizers.