Neural Networks Provably Learn Spectral Representations for Group Composition

arXiv cs.LG Papers
neural-networks group-theory representation-learning spectral-methods deep-learning-theory feature-learning

Summary

This paper theoretically demonstrates that two-layer neural networks trained on group composition tasks learn spectral representations, with neurons converging to irreducible representations and achieving rotational rank-one alignment, providing a representation-theoretic account of feature learning.

arXiv:2606.02993v1 Announce Type: new Abstract: Understanding how structured internal structure emerges during neural network training is central to the study of deep learning. We investigate this phenomenon through the group composition task, where a two-layer neural network is trained to predict $g_1 \star g_2$ for elements of a finite group $G$. By lifting the projected gradient flow to the Fourier domain, we demonstrate that the training dynamics are governed by a Riemannian gradient ascent on a representation-theoretic energy functional. We prove that, under random initialization, this flow drives each neuron to converge almost surely toward a single irreducible representation, while the cross-layer Fourier coefficients achieve a rotational rank-one alignment. This framework provides a representation-theoretic account of feature learning and characterizes a novel low-rank compression phenomenon for matrix-valued group representations. Moreover, for Abelian groups, we provide a complete population-level description: random initialization promotes uniform diversification across nontrivial representations and induces Haar-uniform phases, jointly approximating the indicator via a majority-vote mechanism. We further prove that both phase alignment and representation competition emerge with exponential convergence rates.
Original Article
View Cached Full Text

Cached at: 06/03/26, 09:41 AM 

# Neural Networks Provably Learn Spectral Representations for Group Composition
Source: [https://arxiv.org/abs/2606.02993](https://arxiv.org/abs/2606.02993)
[View PDF](https://arxiv.org/pdf/2606.02993)

> Abstract:Understanding how structured internal structure emerges during neural network training is central to the study of deep learning\. We investigate this phenomenon through the group composition task, where a two\-layer neural network is trained to predict $g\_1 \\star g\_2$ for elements of a finite group $G$\. By lifting the projected gradient flow to the Fourier domain, we demonstrate that the training dynamics are governed by a Riemannian gradient ascent on a representation\-theoretic energy functional\. We prove that, under random initialization, this flow drives each neuron to converge almost surely toward a single irreducible representation, while the cross\-layer Fourier coefficients achieve a rotational rank\-one alignment\. This framework provides a representation\-theoretic account of feature learning and characterizes a novel low\-rank compression phenomenon for matrix\-valued group representations\. Moreover, for Abelian groups, we provide a complete population\-level description: random initialization promotes uniform diversification across nontrivial representations and induces Haar\-uniform phases, jointly approximating the indicator via a majority\-vote mechanism\. We further prove that both phase alignment and representation competition emerge with exponential convergence rates\.

## Submission history

From: Jianliang He \[[view email](https://arxiv.org/show-email/5cd88b73/2606.02993)\] **\[v1\]**Tue, 2 Jun 2026 01:04:21 UTC \(6,164 KB\)

Similar Articles

Neural Networks Provably Learn Spectral Representations for Group Composition

Hugging Face Daily Papers

This paper provides a theoretical analysis of how neural networks learn structured representations during group composition tasks, proving that training dynamics drive neurons to converge to irreducible group representations with exponential convergence rates. The work establishes a representation-theoretic account of feature learning and characterizes a low-rank compression phenomenon for matrix-valued group representations.

Structural Instability of Feature Composition

arXiv cs.LG

This paper presents a geometric framework to analyze the instability of feature composition in Sparse Autoencoders, revealing that non-linearities cause a ratchet effect leading to compositional collapse beyond a critical density.

Learning Coherent Representations: A Topological Approach to Interpretability

arXiv cs.LG

This paper introduces coherence, a geometric constraint for neural representations inspired by grid cells and head direction cells in the brain. Coherence ensures that features respond to geometrically connected regions of the data manifold, improving interpretability; the authors propose a differentiable objective (Coh) and validate it on synthetic data, rotated MNIST, and BERT token embeddings.