Neuron Populations Exhibit Divergent Selectivity with Scale [R]

Reddit r/MachineLearning 06/18/26, 07:40 PM Papers

rosetta-neurons scaling-laws monosemanticity specialization universal-neurons neural-networks

Summary

This paper introduces 'Rosetta Neurons'—universal neurons across diverse neural networks—and shows they scale as a sublinear power law, becoming more selective and monosemantic with scale, enabling data filtering that nearly matches oracle performance.

Hi! We just released a paper where we study “Rosetta Neurons”: universal neurons across different neural networks, and their relationship to scaling laws, specialization, and monosemanticity. Would love to kick off a discussion and get the community's thoughts. Main Findings: We find that the universal Rosetta Neurons scale as a sublinear power law: larger models have more of them, but they occupy a shrinking fraction of all neurons. They also become more selective/monosemantic and more specialized with scale. We can use a single Rosetta Neuron to filter data for continued pretraining and nearly match oracle data filtering. Paper: https://arxiv.org/abs/2606.03990 Summary thread: https://x.com/_AmilDravid/status/2062959617941074069?s=20 Code: https://github.com/avdravid/rosetta-neuron-scaling Project page: https://avdravid.github.io/rosetta-neuron-scaling/ https://preview.redd.it/sus4wqc9g38h1.png?width=1806&format=png&auto=webp&s=4aac2b2209779cb05e1c73cdaadac860318f0162

Original Article

Similar Articles

Revenge of Monosemanticity: Specialized Neurons Improve Data Efficiency in MLPs

arXiv cs.LG

The paper shows that multilayer perceptrons naturally develop monosemantic specialized neurons that improve data efficiency by learning local low-dimensional representations instead of a global one in regression problems with clustered data.

Generalized Neurons

ML at Berkeley

The article explores the Universal Approximation Theorem in deep learning, analyzing the representation capacity of individual neurons and neural network layers using ReLU activation functions.

Mechanistic interpretability: a first paper on disentangling a convolutional neuron [R]

Reddit r/MachineLearning

This paper introduces a technique to disentangle a single convolutional neuron in Inceptionv1 by analyzing Hadamard products, revealing clean monosemantic clusters (cars, cats, dogs) and also low-valued clusters (letters, faces) with distributed weights as evidence of gradient descent behavior.

My toy spiking network completely flunked NARMA-10, but a simple neuroscience trick unlocked a 15x compute bargain. [D]

Reddit r/MachineLearning

The author describes building a spiking neural network engine that initially failed the NARMA-10 benchmark, but by applying heterogeneous wire delays from neuroscience, it achieved usable memory depth and a 15x computational efficiency advantage over continuous nets on a recognition task.

Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos