linear-networks

This paper derives the eigenvalues of the Hessian for linear neural networks of arbitrary width and depth, showing that sharpness relates to maximum class proportion for classification tasks with MSE loss, and empirically validates the predictions.

This paper demonstrates that optimal learning rate scaling in deep scalar linear networks is inherently data-dependent, contradicting prior data-agnostic scaling rules. It shows that with data-dependent scaling, convergence becomes depth-independent, including at infinite depth.

OpenAI research explores how nonlinear computation can emerge in deep linear networks, presenting theoretical and empirical analysis with code examples using TensorFlow.