Tag
This paper provably shows that whether learning-rate cooldown helps in WSD schedules depends on the structure of gradient noise and whether the optimizer normalizes its update, explaining why cooldown can be ineffective for SGD but necessary for normalized methods.
A detailed explanation of Perlin noise and its implementation in ClojureScript, including code snippets and a visualization of moving water filaments.
OpenAI researchers discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training across a wide range of tasks. They found that more complex tasks and more powerful models tolerate larger batch sizes, suggesting future AI systems can scale further through increased parallelization.