Nonlinear computation in deep linear networks

OpenAI Blog Papers
deep-learning linear-networks neural-networks theory nonlinear-computation openai-research

Summary

OpenAI research explores how nonlinear computation can emerge in deep linear networks, presenting theoretical and empirical analysis with code examples using TensorFlow.

No content available
Original Article
View Cached Full Text

Cached at: 04/20/26, 02:56 PM

# Nonlinear computation in deep linear networks Source: [https://openai.com/index/nonlinear-computation-in-deep-linear-networks/](https://openai.com/index/nonlinear-computation-in-deep-linear-networks/) `` ``` 1x = tf.placeholder(dtype=tf.float32, shape=[batch_size,784])2y = tf.placeholder(dtype=tf.float32, shape=[batch_size,10])34w1 = tf.Variable(np.random.normal(scale=np.sqrt(2./784),size=[784,512]).astype(np.float32))5b1 = tf.Variable(np.zeros(512,dtype=np.float32))6w2 = tf.Variable(np.random.normal(scale=np.sqrt(2./512),size=[512,512]).astype(np.float32))7b2 = tf.Variable(np.zeros(512,dtype=np.float32))8w3 = tf.Variable(np.random.normal(scale=np.sqrt(2./512),size=[512,10]).astype(np.float32))9b3 = tf.Variable(np.zeros(10,dtype=np.float32))1011params = [w1,b1,w2,b2,w3,b3]12nr_params = sum([np.prod(p.get_shape().as_list()) for p in params])13scaling = 2**1251415def get_logits(par):16 h1 = tf.nn.bias_add(tf.matmul(x , par[0]), par[1]) / scaling17 h2 = tf.nn.bias_add(tf.matmul(h1, par[2]) , par[3] / scaling) 18 o = tf.nn.bias_add(tf.matmul(h2, par[4]), par[5]/ scaling)*scaling19return o ```

Similar Articles

How are linear representations learned? Exact solutions to the dynamics of abstraction

arXiv cs.LG

This paper develops a framework to study how linear concept representations emerge during neural network training, providing exact solutions in linear networks and analyzing abstraction dynamics in nonlinear networks. The results reveal key principles governing abstraction and offer implications for interpretability and control.

@AnimaAnandkumar: Neural operators – Convert popular neural networks into neural operators for scientific modeling Extending neural netwo…

X AI KOLs Timeline

This paper presents principled approaches for converting popular neural network architectures (CNNs, GNNs, transformers) into neural operators that learn mappings between infinite-dimensional function spaces, enabling consistent predictions across different discretizations for scientific modeling. Published in Nature Machine Intelligence.

@AnimaAnandkumar: This is something I have been emphasizing since we started our work on Neural Operators. We very quickly went from simp…

X AI KOLs Following

Anima Anandkumar highlights that neural operators, despite simple benchmarks, have achieved massive speedups (10,000–million times) in hard real-world problems like high-resolution AI weather modeling (FourCastNet) and nuclear fusion turbulence, referencing a new paper showing learned solvers become more cost-effective as PDE tasks get harder.

Rethinking Neural Nonlinearity as Gating

arXiv cs.LG

This paper proposes Threshold Gating (TG) as a unified primitive for neural nonlinearity, showing that standard activation functions can be expressed as instances of TG. The authors validate their approach by converting pretrained models across various architectures without retraining and discuss hardware benefits.