residual-neural-networks

This paper defines a first-order criterion to determine if a trained residual neural network has sufficient depth, proving that the absence of a strict local decrease from insertion candidates characterizes depth saturation, and validates it empirically on ResNets, GPT-2-style models, and Pythia checkpoints.