optimizers

A research post introducing 'reactive optimizers': an agent watches training dynamics and rewrites the optimizer's source code during training, matching or beating the best hand-tuned recipes on image and language tasks while autonomously guarding against loss spikes and overfitting. The method is released via the open-source OPAL package.

该论文研究了分布式LLM训练中内部优化器如何影响副本扩展（replica scaling），识别出伪梯度协方差结构与噪声尺度两个关键因素，并从理论和实证上证明基于白化的预条件优化器能实现最强的worker扩展性能。

论文研究了LLM在可验证奖励强化学习（RLVR）下的优化几何特性，发现梯度能量会逐渐集中到不断收缩的有效子空间中，进而提出自适应秩优化器Ares，根据梯度谱能量分布动态调整更新秩，在多个模型规模和RLVR任务上取得优于主流优化器的表现。

论文提出 chained LMOs 框架，将 Muon 系优化器统一为线性最小化 oracle 的组合，并提出新优化器 TensorChain，在 Qwen3 0.6B/1.7B 预训练中相比 Muon 平均节省 9.6% 的 token。

该论文提出了 Spectrally Targeted Muon，通过仅对超过或低于阈值的奇异值进行正交化，在 SGD 和 Muon 之间插值，并在 CIFAR-10 与 NanoGPT speedrun 上分析了 Muon 成功的真正原因：放大过小的奇异方向而非抑制大方向。

This paper shows that no principled batch-size scaling rule for the Muon optimizer works consistently across training settings, and that the best optimizer for language model pretraining changes with batch size even after extensive tuning, arguing that optimizer design and benchmarking must account for batch size.

The article reports initial results from OpenRSI's Marin-Scaling-Ladder experiment, where AI agents autonomously propose, implement, and evaluate new optimizers across model scales from 550M to 2.5B, discovering PSPR with Codex (GPT-5.6).

The PyTorch Conference North America is announced, featuring sessions on new optimizers like Muon and TorchJD, tools like fastsafetensors and LMCache, with speakers from major tech companies and institutions, happening in San Jose on October 20-21.

An announcement for a talk at PyTorch Conference North America covering practical implementations of Muon, Dion, and Dion3 optimizers in training stacks.

This paper benchmarks various optimizers for solving inverse problems with differentiable physics simulators, comparing their performance across different physics domains and problem types to provide insights for optimizer selection and design.

A project that visualizes gradient descent for different optimization algorithms, useful for understanding how optimizers work in machine learning.

Expresses the opinion that too much effort is spent on making optimizers marginally faster, and the real need is for hyperparameter-free optimizers.

This paper introduces SODA, a generalization of Optimistic Dual Averaging that unifies various modern optimizers like Muon and Lion. It proposes a practical wrapper that improves performance across different scales without requiring additional hyperparameter tuning for weight decay.

Tilde Research introduces Aurora, a new optimizer designed to prevent neuron death in MLP layers while maintaining orthogonality, achieving state-of-the-art results on nanoGPT benchmarks and 100x data efficiency on 1B models.