Tag
This paper proves that the minimax risk for deep variation-norm ReLU regression has quadratic dependence on depth, using local packing arguments and approximation theorems.
A theoretical study shows that in overparameterized ReLU networks, a positive-homogeneous scaling gauge hidden in the initial parameters can deterministically control which duplicate neuron learns a teacher feature, affecting specialization time and pruning trajectories.
This paper improves theoretical bounds on the depth of ReLU networks needed to represent the maximum function, showing exact two-hidden-layer representations for up to 10 inputs and improved depth for larger n via exact linear algebra techniques.
This paper proves that every finite affine iterate of vector-valued affine refinement operators admits an exact fixed-width ReLU realization with depth O(n) for M>=3, using a residual memory controller and offset frames. The result extends to arbitrary compactly supported continuous piecewise linear forcing terms.
This paper proves a conjectured law of robustness for two-layer neural networks with unbounded weights, showing that a network fitting noisy data must have a Lipschitz constant at least of order sqrt(n/m), up to a logarithmic factor, for continuous piecewise-linear activations like ReLU.