relu

This paper presents an exponential depth hierarchy for ReLU networks in terms of L2 approximation error, demonstrating that deeper networks offer exponentially improved representational power for function approximation.

This theoretical paper proves that ReLU-based message-passing GNNs are strictly more expressive than GNNs using any eventually constant activation functions (e.g., truncated ReLU) with respect to Boolean queries, even on Boolean-featured graphs.

This paper proposes efficient search methods to locate verdict boundaries in Branch and Bound (BaB) neural network verification, leveraging path monotonicity to skip irrelevant subproblems and improve verification efficiency.

This paper formally proves that training neural networks with asymmetric activation functions like ReLU, GELU, or SiLU causes weights to drift negative, leading to up to 90% activation sparsity. It also shows that squared activations like ReLU² improve performance but cause activation spikes, which can be fixed by clipping, with GELU² achieving the best validation loss.

This paper proposes a novel transformer verification approach that uses ReLU to represent precise but non-linear bounds for dot products, enabling precise and efficient verification. The method outperforms state-of-the-art baselines on sentiment analysis models.