Deep double descent

OpenAI Blog Papers
deep-learning generalization double-descent overparameterization implicit-bias

Summary

OpenAI research reveals the 'double descent' phenomenon where test error exhibits a non-monotonic pattern as both model size and training steps increase, challenging traditional understanding of the bias-variance tradeoff in deep learning.

We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don’t yet fully understand why it happens, and view further study of this phenomenon as an important research direction.
Original Article
View Cached Full Text

Cached at: 04/20/26, 02:43 PM

# Deep double descent Source: [https://openai.com/index/deep-double-descent/](https://openai.com/index/deep-double-descent/) The charts above show test and train error as a function of both model size and number of optimization steps\. For a given number of optimization steps \(fixed y\-coordinate\), test and train error exhibit model\-size double descent\. For a given model size \(fixed x\-coordinate\), as training proceeds, test and train error decreases, increases, and decreases again; we call this phenomenon epoch\-wise double descent\. *In general, the peak of test error appears systematically when models are just barely able to fit the train set\.* Our intuition is that, for models at the interpolation threshold, there is effectively only one model that fits the train data, and forcing it to fit even slightly noisy or misspecified labels will destroy its global structure\. That is, there are no “good models” which both interpolate the train set and perform well on the test set\. However, in the over\-parameterized regime, there are many models that fit the train set and there exist such good models\. Moreover, the implicit bias of stochastic gradient descent \(SGD\) leads it to such good models, for reasons we don’t yet understand\. We leave fully understanding the mechanisms behind double descent in deep neural networks as an important open question\.

Similar Articles

The Implicit Bias of Depth: From Neural Collapse to Softmax Codes

arXiv cs.LG

This paper studies how depth alone induces an implicit low-rank bias in deep unconstrained feature models trained without regularization, shifting the optimal solution from neural collapse to softmax codes, and provides the first asymptotic and dynamic characterization of this bias under gradient descent with cross-entropy loss.

Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway

arXiv cs.LG

This paper shows that discrete Gradient Descent with large step sizes restores symmetry in multi-pathway Deep Linear Networks, countering the symmetry-breaking predicted by Gradient Flow, and leads to signal re-balancing across pathways. The authors theoretically prove that balanced solutions are flatter (less sharp) than sparse ones, and large learning rates drive the network toward stable, balanced configurations.

How AI training scales

OpenAI Blog

OpenAI researchers discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training across a wide range of tasks. They found that more complex tasks and more powerful models tolerate larger batch sizes, suggesting future AI systems can scale further through increased parallelization.