Rethinking Neural Nonlinearity as Gating

arXiv cs.LG Papers
activation-functions threshold-gating neural-networks model-compression deep-learning hardware-implementation

Summary

This paper proposes Threshold Gating (TG) as a unified primitive for neural nonlinearity, showing that standard activation functions can be expressed as instances of TG. The authors validate their approach by converting pretrained models across various architectures without retraining and discuss hardware benefits.

arXiv:2607.03148v1 Announce Type: new Abstract: Activation functions are considered an essential primitive for neural nonlinearity, i.e., they enable neural networks to serve as universal approximators. In this paper, we show that this nonlinearity can also be achieved by input-conditioned threshold gating through branches as a universal primitive. We demonstrate that standard activations -- whether piecewise-linear (ReLU, PReLU, Hardtanh) or smooth (SiLU, Sigmoid, Tanh, GELU) -- are in fact instances of a single Threshold Gating (TG) primitive. For softmax, we show that it admits an exact TG conversion via its equivalent per-element Sigmoid form. We then validate these equivalences by converting pretrained networks across CNNs, transformer-based models, and recurrent architectures, preserving model performance without requiring retraining. Threshold Gating also enables training from scratch that goes beyond replacing existing activations, enabling gains in model compression, performance, and shorter training. We also propose a 'Minimal Branch Theorem' which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks. In terms of hardware implementation, TG maps to a unified implementation in the case of analog in-memory systems, addressing the bottleneck of analog-to-digital and digital-to-analog converters (ADC/DAC) that is known to significantly impact power consumption and on-chip area.
Original Article
View Cached Full Text

Cached at: 07/07/26, 04:41 AM 

# Rethinking Neural Nonlinearity as Gating
Source: [https://arxiv.org/abs/2607.03148](https://arxiv.org/abs/2607.03148)
[View PDF](https://arxiv.org/pdf/2607.03148)

> Abstract:Activation functions are considered an essential primitive for neural nonlinearity, i\.e\., they enable neural networks to serve as universal approximators\. In this paper, we show that this nonlinearity can also be achieved by input\-conditioned threshold gating through branches as a universal primitive\. We demonstrate that standard activations \-\- whether piecewise\-linear \(ReLU, PReLU, Hardtanh\) or smooth \(SiLU, Sigmoid, Tanh, GELU\) \-\- are in fact instances of a single Threshold Gating \(TG\) primitive\. For softmax, we show that it admits an exact TG conversion via its equivalent per\-element Sigmoid form\. We then validate these equivalences by converting pretrained networks across CNNs, transformer\-based models, and recurrent architectures, preserving model performance without requiring retraining\. Threshold Gating also enables training from scratch that goes beyond replacing existing activations, enabling gains in model compression, performance, and shorter training\. We also propose a 'Minimal Branch Theorem' which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks\. In terms of hardware implementation, TG maps to a unified implementation in the case of analog in\-memory systems, addressing the bottleneck of analog\-to\-digital and digital\-to\-analog converters \(ADC/DAC\) that is known to significantly impact power consumption and on\-chip area\.

## Submission history

From: Muhammad Sabih \[[view email](https://arxiv.org/show-email/e7765490/2607.03148)\] **\[v1\]**Fri, 3 Jul 2026 09:38:16 UTC \(44 KB\)

Similar Articles

Generalized Neurons

ML at Berkeley

The article explores the Universal Approximation Theorem in deep learning, analyzing the representation capacity of individual neurons and neural network layers using ReLU activation functions.

@AnimaAnandkumar: Neural operators – Convert popular neural networks into neural operators for scientific modeling Extending neural netwo…

X AI KOLs Timeline

This paper presents principled approaches for converting popular neural network architectures (CNNs, GNNs, transformers) into neural operators that learn mappings between infinite-dimensional function spaces, enabling consistent predictions across different discretizations for scientific modeling. Published in Nature Machine Intelligence.

Extensions and limitations of the neural GPU

OpenAI Blog

This paper explores extensions and limitations of the Neural GPU model, demonstrating improvements through curriculum design and scaling, enabling it to learn arithmetic operations on decimal numbers and long expressions while identifying failure modes on symmetric inputs analogous to adversarial examples.

Reinterpreting Safety Thresholds as Neuron Spiking Thresholds

arXiv cs.AI

This paper proposes a biologically inspired reinterpretation of surrogate safety measure thresholds using spiking neural networks, aligning with human braking behavior to bridge objective and subjective safety perception in automated driving.