learning-rate-scaling

This paper demonstrates that optimal learning rate scaling in deep scalar linear networks is inherently data-dependent, contradicting prior data-agnostic scaling rules. It shows that with data-dependent scaling, convergence becomes depth-independent, including at infinite depth.