momentum

该论文提出了一种统一的 actor-critic 方法，将岭正则化的路径最小二乘 LSTD critic 与重要性校正动量 (ICM) actor 相结合，并首次给出了 ICM actor-critic 在单条 Markov 轨迹上归一化与非归一化步长下的有限时间收敛界，同时将结果特化到 Muon、带动量的 SGD 及 AdaNorm 等优化器进行比较。

This paper analyzes when and how the Muon optimizer provides preconditioning benefits, showing it offers no acceleration under isotropic noise but can reduce task-frequency dependence from linear to square-root under structured mini-batch gradient noise, while avoiding SignSGD's rotation-induced negative transfer.

The article explores how first principles thinking helps senior engineers adapt to changes, particularly in the agentic era of AI, by focusing on momentum and understanding core problems through small learning loops.

Velocity Radar is a tool that measures true GitHub momentum across public and private repos, with features for pace, breakdown, and stamina, designed for developers and AI agent commanders.

The tweet discusses how a PhD can serve as a valuable training ground for building momentum, managing failed experiments, and fostering collaboration within a research lab.

Sam Altman explains that burnout stems from failure rather than overworking, and emphasizes the energizing effect of momentum for startup founders.

This paper introduces an ADMM-inspired momentum framework (AIM) and a new optimizer RADAR, offering a theoretical interpretation of momentum as a residual-driven multiplier correction, with experiments showing consistent gains over strong adaptive baselines.

This paper provides the first comprehensive convergence analysis of vanilla SGD with momentum under heavy-tailed noise without gradient clipping or normalization, revealing inferior rates compared to clipped variants and supported by experiments on synthetic functions.

The paper proposes FedCGNM, a client-side optimizer that groups classes and normalizes momentum per group to address class imbalance in federated learning, along with FedHOO for efficient hyperparameter exploration. Empirical results show consistent improvements over baselines.

A University of Johannesburg working paper proposes a Kalman-filtered, momentum-extended Anticor algorithm (K-ACM) that, in a backtest from 1962–1984, produces an extreme return of $1 to $36 billion, though acknowledged as a backtest artifact due to unrealistic assumptions.

Proposes MGUP, a momentum-gradient alignment update policy for selective intra-layer parameter updates in stochastic optimization, which integrates with optimizers like AdamW, Lion, and Muon, and provides theoretical convergence guarantees along with superior performance on large-scale model training tasks.

DP-MacAdam combines adaptive clipping and adaptive momentum to improve differentially private SGD, achieving better model utility without manual tuning of the clipping threshold.