learning-rate-schedule

This paper provably shows that whether learning-rate cooldown helps in WSD schedules depends on the structure of gradient noise and whether the optimizer normalizes its update, explaining why cooldown can be ineffective for SGD but necessary for normalized methods.