Neural Networks Provably Learn Spectral Representations for Group Composition
Summary
This paper theoretically demonstrates that two-layer neural networks trained on group composition tasks learn spectral representations, with neurons converging to irreducible representations and achieving rotational rank-one alignment, providing a representation-theoretic account of feature learning.
View Cached Full Text
Cached at: 06/03/26, 09:41 AM
# Neural Networks Provably Learn Spectral Representations for Group Composition Source: [https://arxiv.org/abs/2606.02993](https://arxiv.org/abs/2606.02993) [View PDF](https://arxiv.org/pdf/2606.02993) > Abstract:Understanding how structured internal structure emerges during neural network training is central to the study of deep learning\. We investigate this phenomenon through the group composition task, where a two\-layer neural network is trained to predict $g\_1 \\star g\_2$ for elements of a finite group $G$\. By lifting the projected gradient flow to the Fourier domain, we demonstrate that the training dynamics are governed by a Riemannian gradient ascent on a representation\-theoretic energy functional\. We prove that, under random initialization, this flow drives each neuron to converge almost surely toward a single irreducible representation, while the cross\-layer Fourier coefficients achieve a rotational rank\-one alignment\. This framework provides a representation\-theoretic account of feature learning and characterizes a novel low\-rank compression phenomenon for matrix\-valued group representations\. Moreover, for Abelian groups, we provide a complete population\-level description: random initialization promotes uniform diversification across nontrivial representations and induces Haar\-uniform phases, jointly approximating the indicator via a majority\-vote mechanism\. We further prove that both phase alignment and representation competition emerge with exponential convergence rates\. ## Submission history From: Jianliang He \[[view email](https://arxiv.org/show-email/5cd88b73/2606.02993)\] **\[v1\]**Tue, 2 Jun 2026 01:04:21 UTC \(6,164 KB\)
Similar Articles
Neural Networks Provably Learn Spectral Representations for Group Composition
This paper provides a theoretical analysis of how neural networks learn structured representations during group composition tasks, proving that training dynamics drive neurons to converge to irreducible group representations with exponential convergence rates. The work establishes a representation-theoretic account of feature learning and characterizes a low-rank compression phenomenon for matrix-valued group representations.
Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking
This empirical study validates theoretical findings on feature repulsion and spectral lock-in during the grokking phenomenon in two-layer neural networks, demonstrating how activation functions influence the transition from memorization to generalization.
The Rank the Task Demands: A Causal Rank Law for Matrix Memories Trained on Group Composition
This paper provides causal evidence that gradient descent in matrix memories trained on group composition tasks recruits exactly the rank required by the task's algebra, correlating with the minimal faithful representation dimensions of finite groups.
Group-Equivariant Poincar\'e Convolutional Networks
This paper proposes Equivariant Poincaré ResNets, combining hyperbolic geometry with discrete symmetry groups to improve efficiency in learning visual representations by treating rotated features as symmetric rather than distinct hierarchical concepts.
The spectral neuron - an ML primitive for scalable and interpretable models [R]
The article presents a new ML primitive called the Spectral Neuron, offering a simple, scalable, and interpretable model with a mathematical foundation for training and initialization.