whitening

该论文研究了分布式LLM训练中内部优化器如何影响副本扩展（replica scaling），识别出伪梯度协方差结构与噪声尺度两个关键因素，并从理论和实证上证明基于白化的预条件优化器能实现最强的worker扩展性能。

Zeta proposes a dual whitening optimizer that applies coordinate whitening before spectral whitening to resolve scale heterogeneity in momentum matrices, reducing orthogonalization error and improving convergence and generalization in large-scale neural network training.