Tag
该论文指出权重归一化（WN）训练的收益主要源于其诱导出的有效学习率（ELR）轨迹，而非归一化本身，并据此提出 AIR（仿射平方根倒数衰减）调度策略。在 128B token 的 MoE 预训练实验中，AIR 在训练损失、下游性能和超参数鲁棒性上均优于 cosine 等调度方案。
OpenAI presents weight normalization, a reparameterization technique that decouples weight vector length from direction to improve neural network training convergence and computational efficiency without introducing minibatch dependencies, making it suitable for RNNs and noise-sensitive applications.