Tag
This theoretical paper proves that ReLU-based message-passing GNNs are strictly more expressive than GNNs using any eventually constant activation functions (e.g., truncated ReLU) with respect to Boolean queries, even on Boolean-featured graphs.
This paper proposes Threshold Gating (TG) as a unified primitive for neural nonlinearity, showing that standard activation functions can be expressed as instances of TG. The authors validate their approach by converting pretrained models across various architectures without retraining and discuss hardware benefits.
A tweet promoting TensorTonic, a platform that allows users to practice implementing nine common activation functions (Sigmoid, ReLU, Tanh, Softmax, Leaky ReLU, GELU, Swish, ELU, SELU) from scratch, including forward pass and gradient computation.
The article explores the Universal Approximation Theorem in deep learning, analyzing the representation capacity of individual neurons and neural network layers using ReLU activation functions.