@linghaokong76: Can networks perform better without adding non-zero weights? Our ICML 2026 paper says yes: spreading the same active we…
Summary
A new ICML 2026 paper shows that spreading the same active weights across more neurons reduces collisions and improves accuracy in neural networks, suggesting networks can perform better without adding non-zero weights.
View Cached Full Text
Cached at: 07/07/26, 01:22 AM
Can networks perform better without adding non-zero weights?
Our ICML 2026 paper says yes: spreading the same active weights across more neurons reduces collisions and improves accuracy.
Presenting today at ICML, 2pm!
W/ @inimai_s @yonashav @micahadler @DAlistarh & Nir Shavit https://t.co/hUxBOoIIrS
Similar Articles
Bug or Feature^2: Weight Drift, Activation Sparsity, and Spikes
This paper formally proves that training neural networks with asymmetric activation functions like ReLU, GELU, or SiLU causes weights to drift negative, leading to up to 90% activation sparsity. It also shows that squared activations like ReLU² improve performance but cause activation spikes, which can be fixed by clipping, with GELU² achieving the best validation loss.
Understanding neural networks through sparse circuits
OpenAI researchers present methods for training sparse neural networks that are easier to interpret by forcing most weights to zero, enabling the discovery of small, disentangled circuits that can explain model behavior while maintaining performance. This work aims to advance mechanistic interpretability as a complement to post-hoc analysis of dense networks and support AI safety goals.
Weight normalization: A simple reparameterization to accelerate training of deep neural networks
OpenAI presents weight normalization, a reparameterization technique that decouples weight vector length from direction to improve neural network training convergence and computational efficiency without introducing minibatch dependencies, making it suitable for RNNs and noise-sensitive applications.
@rohanpaul_ai: New paper from @NaceAI shows a possible way to add large bodies of knowledge without rewriting the language model’s cor…
A new paper from NaceAI proposes a hypernetwork-based method for injecting knowledge into large language models without modifying their core parameters, potentially enabling efficient continual learning. The approach uses generated low-rank adapters to encode new facts while keeping the base model frozen.
Learning sparse neural networks through L₀ regularization
OpenAI proposes a practical L₀ regularization method for neural networks that encourages weights to become exactly zero during training, enabling network pruning for improved speed and generalization. The method uses stochastic gates and introduces the hard concrete distribution to make the non-differentiable L₀ norm optimization tractable via gradient descent.