relu

本文提出了一种关于ReLU网络的指数型深度层次结构，基于L2近似误差，证明更深的网络在函数近似上提供了指数级的表征能力提升。

This theoretical paper proves that ReLU-based message-passing GNNs are strictly more expressive than GNNs using any eventually constant activation functions (e.g., truncated ReLU) with respect to Boolean queries, even on Boolean-featured graphs.

This paper proposes efficient search methods to locate verdict boundaries in Branch and Bound (BaB) neural network verification, leveraging path monotonicity to skip irrelevant subproblems and improve verification efficiency.

本文正式证明了使用非对称激活函数（如ReLU、GELU或SiLU）训练神经网络会导致权重向负方向漂移，进而使激活稀疏性高达90%。同时，研究表明平方激活函数（如ReLU²）能提升性能，但会导致激活尖峰，这一问题可通过裁剪解决，其中GELU²达到了最低验证损失。

本文提出了一种新颖的Transformer验证方法，利用ReLU表示点积的精确但非线性的边界，从而实现精确且高效的验证。该方法在情感分析模型上优于现有最先进的基线方法。