The spectral neuron - an ML primitive for scalable and interpretable models [R]

Reddit r/MachineLearning Papers
machine-learning interpretable-models scalable-ml spectral-neuron preprint mathematics neural-networks

Summary

The article presents a new ML primitive called the Spectral Neuron, offering a simple, scalable, and interpretable model with a mathematical foundation for training and initialization.

Worked some time ago on one of the ad teams at Yahoo, and this grew out of a question I kept returning to while there are there "simple" models that are both simple, scalable, interpretable, and controllable at the same time? Decided to explore it, first in a blog (starting here), then in a new preprint "The Spectral Neuron", built by distilling latest blog-posts into a manuscript, I study models of the form: 𝑓(𝒙) = 𝛌ₖ(𝐀₀ + 𝚺ᵢ 𝑥ᵢ𝐀ᵢ). Manuscript: https://arxiv.org/abs/2608.08003 Code: https://github.com/alexshtf/spectral_neuron_paper Looks like a simple on-liner, but many interesting aspects hide there. How expressive does the model become as the matrices grow? What can we read directly from the learned matrices? Which shapes can be guaranteed by construction? I develop the mathematics, give a practical initialization and training recipe, and test the model in scaling experiments on synthetic and real data. AI disclaimer: manuscript written by yours truly, AI assisted in looking up canonical references and related work for literature review. In contrast, the code was heavily AI written and reviewed by yours truly.
Original Article

Similar Articles

Neural networks for spectral optimization

arXiv cs.LG

The paper proposes two neural network surrogate models that learn the spectrum of the Laplace operator directly from domain geometry, enabling fast differentiable shape optimization of eigenvalue functionals. A Fourier-coefficient MLP reaches 0.2% precision on star-shaped domains, while a landscape-function model achieves 1% mean relative error on the first ten eigenvalues, outperforming FNO baselines and recovering classical spectral optima.

Spectral Scaling Laws of Muon

arXiv cs.LG

This paper presents the first systematic study of singular value spectral behavior in Muon optimizer momentum matrices during LLM training, discovering clean power-law scaling relationships across model sizes (77M–2.8B parameters). The findings provide practitioners with principled, layer-aware guidelines for configuring Newton–Schulz iterations to maintain orthonormalization quality at frontier scale without unnecessary computation.