Emergence of Fibrations, Compression, and Symmetry Breaking in Artificial Neural Networks

# Emergence of Fibrations, Compression, and Symmetry Breaking in Artificial Neural Networks
Source: [https://arxiv.org/html/2609.01768](https://arxiv.org/html/2609.01768)
Osvaldo M\. VelardeEmail:[ovelarde@ccny\.cuny\.edu](mailto:[email protected])Affiliation:Levich Institute and Physics Department, City College of New York, 160 Convent Ave, NY, 10031, NY, USALucas C\. ParraEmail:[parra@ccny\.cuny\.edu](mailto:[email protected])Affiliation:Biomedical Engineering Department, City College of New York, 160 Convent Ave, NY, 610101, NY, USAAlireza HashemiEmail:[ahashemi@ccny\.cuny\.edu](mailto:[email protected])Affiliation:Levich Institute and Physics Department, City College of New York, 160 Convent Ave, NY, 10031, NY, USAHernán A\. MakseEmail:[hmakse@ccny\.cuny\.edu](mailto:[email protected])Affiliation:Levich Institute and Physics Department, City College of New York, 160 Convent Ave, NY, 10031, NY, USA

###### Abstract

Artificial neural networks are often regarded as powerful yet opaque “black boxes”\. Here, we demonstrate that learning in deep neural networks generates local symmetries known in graph theory as fibrations and coverings\. We prove that covering symmetries are stable attractors of stochastic gradient descent\. Consistent with this theory, we report the emergence of covering symmetries across major network architectures, including multi\-layer, convolutional, recurrent, and transformer networks\. Exploiting these symmetries enables drastic model compression — reducing networks to 17% of their original size without sacrificing performance\. Furthermore, controlled breaking of covering symmetry overcomes the loss of plasticity, achieving state\-of\-the\-art performance in continual learning\. The theoretical results provide a new foundation for AI systems based on symmetries that convert black boxes into interpretable “colored graphs” and enable more efficient inference and lifelong learning\.

###### keywords

graph fibrations, network compression, continual learning, symmetry breaking

## 1Main

Despite the development of increasingly powerful neural network architectures, our understanding of their internal structure remains limited\. They are often regarded as black boxes, unable to provide explanations for how their network connections capture the regularity of training data\. In the absence of a theoretically grounded understanding of learning, the recent surge in artificial intelligence \(AI\) has been driven predominantly by empirical scaling laws\[[1](https://arxiv.org/html/2609.01768#bib.bib1),[2](https://arxiv.org/html/2609.01768#bib.bib2)\], resulting in ever increasing model sizes\.

The reliance on scaling as a substitute for understanding creates significant challenges\. Training with single large runs has massive cost and energy demands, and when trained sequentially, networks can suffer from a loss of plasticity over time\[[3](https://arxiv.org/html/2609.01768#bib.bib3),[4](https://arxiv.org/html/2609.01768#bib.bib4),[5](https://arxiv.org/html/2609.01768#bib.bib5)\]\. Additionally, training large models from scratch is data\-inefficient, requiring exceedingly large datasets\. Without principled guidelines, architecture design becomes a costly trial\-and\-error process\. Finally, when models are excessively over\-parameterized, paradoxically, they still perform well in practice\[[6](https://arxiv.org/html/2609.01768#bib.bib6)\], a subject of ongoing theoretical debate\[[7](https://arxiv.org/html/2609.01768#bib.bib7),[8](https://arxiv.org/html/2609.01768#bib.bib8)\]\.

To address these challenges, we propose a new theoretical formalism that analyzes the internal structure of deep neural networks through the lens of graph symmetries\. While global symmetry groups are fundamental to geometric deep learning \(GDL\)\[[9](https://arxiv.org/html/2609.01768#bib.bib9)\]and our understanding of theoretical physics\[[9](https://arxiv.org/html/2609.01768#bib.bib9),[10](https://arxiv.org/html/2609.01768#bib.bib10),[11](https://arxiv.org/html/2609.01768#bib.bib11)\], they are too rigid to capture the diversity observed in both artificial and biological neural systems\[[12](https://arxiv.org/html/2609.01768#bib.bib12),[13](https://arxiv.org/html/2609.01768#bib.bib13),[14](https://arxiv.org/html/2609.01768#bib.bib14),[15](https://arxiv.org/html/2609.01768#bib.bib15)\]\. For instance, existing machine learning applications rely heavily on strict global symmetries, such as shift equivariance in convolutional neural networks \(CNNs\)\[[16](https://arxiv.org/html/2609.01768#bib.bib16),[17](https://arxiv.org/html/2609.01768#bib.bib17)\]or permutation equivariance\[[18](https://arxiv.org/html/2609.01768#bib.bib18),[19](https://arxiv.org/html/2609.01768#bib.bib19),[20](https://arxiv.org/html/2609.01768#bib.bib20)\]in graph neural networks \(GNNs\)\[[9](https://arxiv.org/html/2609.01768#bib.bib9)\]\. Instead, we identify less restrictive local symmetries that naturally emerge in the input and output trees of computational graphs\. These local symmetries are known in graph theory as fibrations, opfibrations, and coverings\[[21](https://arxiv.org/html/2609.01768#bib.bib21),[12](https://arxiv.org/html/2609.01768#bib.bib12)\]\. Originally introduced by Grothendieck as maps in category theory\[[22](https://arxiv.org/html/2609.01768#bib.bib22)\], they were later adapted for graphs\[[21](https://arxiv.org/html/2609.01768#bib.bib21),[23](https://arxiv.org/html/2609.01768#bib.bib23)\]\.

Here, we provide a unified mathematical framework demonstrating that stochastic gradient descent \(SGD\) acts as a mechanism for local symmetry formation\. We prove mathematically that covering symmetries emerge in SGD due to the ”synchronized learning” of weight updates\. This synchronized learning induces fibration symmetries, providing a formal explanation for the emergence of neural activity synchronization, which has often been reported in deep networks\[[15](https://arxiv.org/html/2609.01768#bib.bib15),[24](https://arxiv.org/html/2609.01768#bib.bib24),[25](https://arxiv.org/html/2609.01768#bib.bib25)\]\. Furthermore, we develop a ”balance coloring” algorithm\[[26](https://arxiv.org/html/2609.01768#bib.bib26)\]to identify these structures in the weights of deep networks, and use the resulting fibrations for substantial compression of trained models with minimal impact on performance\. We validate the emergence of these local symmetries across a wide range of architectures, including multilayer perceptrons \(MLP\), CNNs, recurrent networks \(e\.g\., long short\-term memory, LSTM\), and Transformers in both supervised and reinforcement learning \(RL\) settings\. Finally, by carefully breaking these emergent symmetries, we expand the capacity of the network to overcome loss of plasticity\[[4](https://arxiv.org/html/2609.01768#bib.bib4),[5](https://arxiv.org/html/2609.01768#bib.bib5)\], demonstrating state\-of\-the\-art performance in continual learning\[[3](https://arxiv.org/html/2609.01768#bib.bib3)\]\.

### An hierarchy of symmetries in computational graphs

The symmetries we will discuss form a hierarchy and emerge in a variety of network structures\. For an easy visualization, consider a layered feedforward network with binary weights \(where the connections are 1 or 0\), as shown in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)\. In the figure, nodes are colored to indicate their class according to various symmetries\. We will generalize these concepts to continuous\-valued weights in the next section\.

![Refer to caption](https://arxiv.org/html/2609.01768v1/Fig_1.png)Figure 1:Hierarchy of symmetries in a feedforward network\.For simplicity, weights are binary, and connections with zero weight are not plotted\. Conventional automorphism permutation is exemplified in the case of a graph in panel\(d\)\. It identifies three pairs of symmetric nodes by a permutationπ\\pi\. The proposed framework is based on the generalization to local symmetries, such as \(op\)fibrations and coverings \(a\-c\); see text for detail\. These symmetries have less strict conditions that can be satisfied by a larger number of nodes resulting in fewer symmetry classes \(colors\) \(blank nodes should be interpreted as distinct colors\)\. The reduced degrees of freedom increases the inductive bias\.\(e\)Breaking of symmetry for continual learning consists of two steps\. Compress to the covering base \(middle\) to preserve the learned task\. Randomize or zero out redundant weights \(right\) to provide new degrees of freedom to continue learning\.Fibration Symmetries:Nodes possess a fibration symmetry and are said to belong to the samefiberif they have isomorphic input trees\[[12](https://arxiv.org/html/2609.01768#bib.bib12),[21](https://arxiv.org/html/2609.01768#bib.bib21),[23](https://arxiv.org/html/2609.01768#bib.bib23)\]\. This means that the entire structure of connections from the graph’s input to the nodes is isomorphic \(see Methods[4\.1](https://arxiv.org/html/2609.01768#S4.SS1)\)\. Even though the input tree spans all the network, it represents the local ’view’ of the node at its root\[[21](https://arxiv.org/html/2609.01768#bib.bib21)\]\. In practice, fibers are identified by a balanced coloring algorithm from graph theory\[[12](https://arxiv.org/html/2609.01768#bib.bib12),[26](https://arxiv.org/html/2609.01768#bib.bib26)\]\. This algorithm partitions the network into balanced coloring classes of nodes \(fiber partition\) by iteratively assigning the same color to nodes that receive the same set of input colors\[[27](https://arxiv.org/html/2609.01768#bib.bib27)\], hence the termbalanced coloring\. An example is shown in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)a, where nodes in the same fiber are colored the same\. The input tree for a red node is shown on the left\.

Fibration symmetries allow the graphGGto be compressed into a smallerbasegraphBfibB\_\{\\rm fib\}\. This compressionφfib:G→Bfib\\varphi\_\{\\rm fib\}:G\\to B\_\{\\rm fib\}works by merging all nodes that share the same color \(i\.e\., belong to the same fiber\) while conserving the input trees \(Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)a, Methods[4\.2](https://arxiv.org/html/2609.01768#S4.SS2)\)\. We will show that the resulting base graphBfibB\_\{\\rm fib\}performs the identical forward computation as the original graphGG\. In deep neural networks \(DNNs\), it means that we can compress the network without losing performance\. The inverse of this compression is calledlifting\[[21](https://arxiv.org/html/2609.01768#bib.bib21)\], an operation that restores the network’s original dimensionality without altering its functional output \(explained in detail in Methods[4\.3](https://arxiv.org/html/2609.01768#S4.SS3)\.

Opfibration Symmetries:A similar principle applies when learning the parameters of the network\. During error backpropagation — the canonical learning algorithm in modern AI — the error at the output is propagated backward through the output tree\. An opfibration symmetry occurs when nodes share isomorphic output trees — the structure of connections leading from them to the output layer \(Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)b\)\.

Covering Symmetries:When multiple nodes have isomorphic inputandoutput trees, they form a covering symmetry and belong to acover\(Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)c\)\. The corresponding coloring is the intersection of the fibration and opfibration coloring partitions; i\.e\., a finer partition of the graph\. Clearly, a covering is a more stringent symmetry than fibration or opfibration\.

The most strict symmetry is theautomorphism\. This is a global permutation of the network’s nodes that leaves the entire graph’s connectivity unchanged\. An example of nodes with automorphism symmetry \(i\.e\. they are in the sameorbit\) is shown in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)d along with the permutation of node labels\. Although automorphisms are the cornerstone of GDL\[[9](https://arxiv.org/html/2609.01768#bib.bib9)\]and theoretical physics\[[10](https://arxiv.org/html/2609.01768#bib.bib10),[11](https://arxiv.org/html/2609.01768#bib.bib11),[28](https://arxiv.org/html/2609.01768#bib.bib28)\], this symmetry is so restrictive that we have never observed it in our trained networks\.

In summary, these symmetries form a hierarchy of increasing strictness: from fibrations and opfibrations, to coverings, and finally to automorphisms\. As conditions become more stringent, there are more distinct color classes and fewer nodes within each class \(meaning fewer symmetries\)\. Less strict local symmetries, such as fibrations, are more common and allow greater compression into a more compact base\. This compression results in a model with fewer effective degrees of freedom, which corresponds to a stronger inductive bias\. For a more formal discussion on symmetries of a graph, see Methods[4\.1](https://arxiv.org/html/2609.01768#S4.SS1)\.

## 2Results

### 2\.1Theoretical results: the mathematical foundation of emergent symmetry

Here we establish that deep learning is not merely a process of parameter\-tuning, but one of symmetry formation\. Learning organizes networks into local symmetries, consisting of fibrations, opfibrations, and together, forming coverings\. In the following, we generalize this hierarchy to weighted computational graphs\.

#### Computational graphs and synchronization

The hierarchical local symmetries explained above emerge in standard architectures like MLPs, CNNs, RNNs and Transformers \(see Methods[4\.12](https://arxiv.org/html/2609.01768#S4.SS12)\), but they are most easily understood in the canonical MLP\. Here, weightWi​k\(ℓ\)W^\{\(\\ell\)\}\_\{ik\}connects nodekkin layerℓ−1\\ell\-1to nodeiiin layerℓ\\ell\. This creates a directed weighted acyclic computational graph\. The activityhi\(ℓ\)h\_\{i\}^\{\(\\ell\)\}propagates forward from inputx=h\(0\)x=h^\{\(0\)\}to outputyythrough a weighted sum followed by a nonlinear activation functionσ\\sigma,

hi\(ℓ\)=σ⁡\(∑kWi​k\(ℓ\)​hk\(ℓ−1\)\)\.h\_\{i\}^\{\(\\ell\)\}=\\sigma\\left\(\\sum\_\{k\}W^\{\(\\ell\)\}\_\{ik\}h\_\{k\}^\{\(\\ell\-1\)\}\\right\)\.\(1\)
The network generates an errorL=ℒ⁡\(y^,y\)L=\\mathcal\{L\}\(\\hat\{y\},y\)with respect to the desired targety^\\hat\{y\}\. This error propagates backward\[[29](https://arxiv.org/html/2609.01768#bib.bib29)\]starting from the output withδi\(N\)=∂L/∂hi\(N\)\\delta\_\{i\}^\{\(N\)\}=\\partial L/\\partial h\_\{i\}^\{\(N\)\}, coupling the error with the forward activity by the derivativeσ′\\sigma^\{\\prime\}of the activation function:

δi\(ℓ\)=σi′\(ℓ\)​∑kWk​i\(ℓ\+1\)​δk\(ℓ\+1\)\.\\delta\_\{i\}^\{\(\\ell\)\}=\\sigma^\{\\prime\(\\ell\)\}\_\{i\}\\sum\_\{k\}W^\{\(\\ell\+1\)\}\_\{ki\}\\delta\_\{k\}^\{\(\\ell\+1\)\}\\,\.\(2\)

#### Fibrations synchronize activity, opfibrations synchronize error

In a weighted graph, two nodesiiandjjin layerℓ\\ellare in a fibration symmetry \(i​∼fib​ji\\underset\{\\rm fib\}\{\\sim\}j\) when they have isomorphic input trees, which mean that they receive the same summed input from the fibration colors of the previous layer\. In other words, they satisfy an equal\-sum criterion of their incoming weights \(see Eq\.[14](https://arxiv.org/html/2609.01768#S4.E14)\)\. Because real\-world networks use continuous weights, this equality constraint is verified up to a thresholdεfib\\varepsilon\_\{\\rm fib\}

\|∑k∈c\(Wi​k\(ℓ\)−Wj​k\(ℓ\)\)\|≤εfib,\\bigm\|\\sum\_\{k\\in c\}\(W^\{\(\\ell\)\}\_\{ik\}\-W^\{\(\\ell\)\}\_\{jk\}\)\\bigm\|\\leq\\varepsilon\_\{\\rm fib\}\\,,\(3\)where the sum is over all fiber colorsccof the previous layer\. In the limitεfib=0\\varepsilon\_\{\\rm fib\}=0, we obtain an exact fibration symmetry \(i​∼fib​ji\\underset\{\\rm fib\}\{\\sim\}j\)\. For non\-zeroεfib\\varepsilon\_\{\\rm fib\}, we refer to these asquasi\-fibersfollowing\[[30](https://arxiv.org/html/2609.01768#bib.bib30)\]\. Our first theoretical observation is that two nodes in the same fiber synchronize their activity \(hi=hjh\_\{i\}=h\_\{j\}\) foranyinputxxto the network \(Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)a\)\. This theorem is formalized in Methods[4\.4](https://arxiv.org/html/2609.01768#S4.SS4)and proven in Methods[4\.6](https://arxiv.org/html/2609.01768#S4.SS6)\. For quasi\-fibers, the accuracy of the synchronization is bounded by a factor proportional toεfib\\varepsilon\_\{\\rm fib\}\(Methods[4\.7](https://arxiv.org/html/2609.01768#S4.SS7)\)\. Therefore, structural symmetry induces*activity synchronization*\.

In the backward pass, an opfibration symmetry \(i​∼op​ji\\underset\{\\rm op\}\{\\sim\}j\) occurs when nodes share isomorphic output trees\. Because error backpropagation involves a nonlinear coupling between the error signal and the forward activation slope throughσi′\(ℓ\)\\sigma^\{\\prime\(\\ell\)\}\_\{i\}in Eq\. \([2](https://arxiv.org/html/2609.01768#S2.E2)\), we introduceflavored connectionsto represent these dependencies \(colored arrows in Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)b; see Methods[4\.5](https://arxiv.org/html/2609.01768#S4.SS5)and[4\.9](https://arxiv.org/html/2609.01768#S4.SS9)\)\. Our second theoretical result is that flavored opfibration symmetry ensures*error synchronization*: the backpropagated error signals in an opfiber \(green right sides in Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)a\) become identical \(δi=δj\\delta\_\{i\}=\\delta\_\{j\}\) for any inputxxand outputyy\(theorem proven in Methods[4\.6](https://arxiv.org/html/2609.01768#S4.SS6)\)\. In practice, we will use quasi\-opfibers with thresholdεop\\varepsilon\_\{\\rm op\}similar to the definition presented for quasi\-fibers\.

![Refer to caption](https://arxiv.org/html/2609.01768v1/Fig_2.png)Figure 2:Summary of theoretical results\.\(a\)The gradient descent learning rule updates a connection’s weightWm​i\(ℓ\)W^\{\(\\ell\)\}\_\{mi\}based on the product of the activityhi\(ℓ−1\)h^\{\(\\ell\-1\)\}\_\{i\}and the error signalδm\(ℓ\)\\delta^\{\(\\ell\)\}\_\{m\}\. Nodes in the same fiber \(red left sides\) synchronize their activities \- Eq\. \([15](https://arxiv.org/html/2609.01768#S4.E15)\)\. Nodes in the same opfibers \(green right sides\) synchronize their error signals \- Eq\. \([17](https://arxiv.org/html/2609.01768#S4.E17)\)\. Weights connecting nodes in a fiber with nodes in an opfiber will have the same updatesΔ​Wm​i\(ℓ\)\\Delta W^\{\(\\ell\)\}\_\{mi\}\- Eq\. \([5](https://arxiv.org/html/2609.01768#S2.E5)\)\.\(b\)Stability rules used to prove the Cover Coarse\-Graining Theorem\. Connection colors indicate “flavor” for the opfibers\. Black connections for fibrations are not flavored\. This introduces an asymmetry, with connection colors defined at the output but not at the input of nodes\.\(c\)Example of fibration compression according to[6](https://arxiv.org/html/2609.01768#S2.E6)\. Connections with zero weight are not plotted\.W^\\hat\{W\}is calculated based onWWusing Eq\. \([6](https://arxiv.org/html/2609.01768#S2.E6)\)\.\(d\)Number of fibers \(left\) and opfibers \(right\) in distinct layers across learning epochs\. As nodes group into fibers and opfibers, the number of distinct classes decreases, and they grow in size\. Later layers in the networks \(black arrow\) cluster into fibers more quickly\. For opfibers, initial layers cluster more quickly\.
#### Synchronized learning

During learning, the gradient of the lossLLwith respect to the weightsWi​k\(ℓ\)W^\{\(\\ell\)\}\_\{ik\}depends locally on the activity of the node and the error signals\[[29](https://arxiv.org/html/2609.01768#bib.bib29)\]:

ΔW\(ℓ\)m​i=−α∂L∂Wm​i\(ℓ\)=−αδ\(ℓ\)m⋅h\(ℓ−1\)i,\\Delta W^\{\(\\ell\)\}\_\{mi\}=\-\\alpha\\frac\{\\partial L\}\{\\partial W^\{\(\\ell\)\}\_\{mi\}\}=\-\\alpha\\delta^\{\(\\ell\)\}\_\{m\}\\cdot h^\{\(\\ell\-1\)\}\_\{i\}\\,,\(4\)whereα\\alphais a learning rate \(see Methods[4\.9\.2](https://arxiv.org/html/2609.01768#S4.SS9.SSS2)\)\. When the input nodes in a layer are in a fiber and the output nodes are in a flavored opfiber, then activity and error are synchronized\. Consequently, the corresponding weight updates under gradient descent \(GD\) become identical \(Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)a right\):

Δ​Wm​i\(ℓ\)=Δ​Wm​j\(ℓ\)=Δ​Wn​i\(ℓ\)=Δ​Wn​j\(ℓ\)\.\\Delta W^\{\(\\ell\)\}\_\{mi\}=\\Delta W^\{\(\\ell\)\}\_\{mj\}=\\Delta W^\{\(\\ell\)\}\_\{ni\}=\\Delta W^\{\(\\ell\)\}\_\{nj\}\\,\.\(5\)We refer to this assynchronized learning\. This theorem is proven in Methods[4\.9\.3](https://arxiv.org/html/2609.01768#S4.SS9.SSS3)\)\.

#### Covers and the emergence of symmetry

We now turn to the central finding of this work, which is that these symmetries emerge naturally during learning with stochastic gradient decent \(SGD\)\. When a neural network is initialized, its weights are set to random values\. Large random graphs of this kind rarely have significant global automorphism symmetries as they are forbidden by Erdős\-Rényi asymmetry theorem\[[31](https://arxiv.org/html/2609.01768#bib.bib31)\]\. However, the training process itself appears to create structure\. For example, studies have observed that SGD causes different nodes to develop similar input and output weights\[[32](https://arxiv.org/html/2609.01768#bib.bib32)\]and redundancies\[[24](https://arxiv.org/html/2609.01768#bib.bib24),[25](https://arxiv.org/html/2609.01768#bib.bib25)\]\. We have also previously observed the emergence ofsynchronizednodes during training\[[15](https://arxiv.org/html/2609.01768#bib.bib15)\], suggesting fibration symmetries\.

To understand the emergence of these symmetries, we have to consider*covering symmetry*\(i​∼cov​ji\\underset\{\\rm cov\}\{\\sim\}j\), which occurs when the nodes are in both fibration and opfibration symmetry\. As with fibers and opfibers, covers partition the nodes in a layer into groups with the same colors, e\.g\. Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)c\. The main theoretical result is that these covering partitions𝒞cov\\mathcal\{C\}^\{\\rm cov\}are preserved under GD, which is formalized in the following Theorem:

###### Theorem 1\.

Cover Coarse\-Graining Theorem\.Under the dynamic of GD, the covering partition𝒞ℓcov​\(t\+1\)\\mathcal\{C\}^\{\\rm cov\}\_\{\\ell\}\(t\+1\)is a coarsening of𝒞ℓcov​\(t\)\\mathcal\{C\}^\{\\rm cov\}\_\{\\ell\}\(t\)\.

This theorem \(proven in Methods[4\.11](https://arxiv.org/html/2609.01768#S4.SS11)\) dictates that distinct covers can merge to form larger \(coarser\) covers, but once multiple nodes join a cover, they cannot exit\. Therefore, the weights of covers constitute an invariant set\. Any parameter set that is invariant under GD becomes a stable attractor under stochastic GD for sufficiently large learning rates\[[32](https://arxiv.org/html/2609.01768#bib.bib32)\]\. We therefore conclude that covers are stable attractors of the SGD dynamics\.

The theoretical results presented so far readily generalize to other computational graphs, including hypergraphs, as we discuss in the Methods[4\.12](https://arxiv.org/html/2609.01768#S4.SS12)\. This includes operations such as convolutions \(in CNN\), multiplicative gating \(in most RNN and LSTM in particular\), residual connections, and attention gates, which is the core innovation of transformer networks\.

#### Compression rule preserves activity and loss

Each symmetry induces a more compact base graph, whereby all nodes of the same color are merged\. We propose the followingcompression rulefor the weights in the base graph, where the connection from colorccto colorc′c^\{\\prime\}is the mean across nodes inc′c^\{\\prime\}and the sum across nodes incc\(exemplified in Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)c\):

W^c′​c\(ℓ\)=1\|c′\|​∑i∈c′∑k∈cWi​k\(ℓ\)\.\\hat\{W\}\_\{c^\{\\prime\}c\}^\{\(\\ell\)\}=\\frac\{1\}\{\|c^\{\\prime\}\|\}\\sum\_\{i\\in c^\{\\prime\}\}\\sum\_\{k\\in c\}W\_\{ik\}^\{\(\\ell\)\}\.\(6\)When colors are obtained using fibration symmetry, we will refer to Eq\.[6](https://arxiv.org/html/2609.01768#S2.E6)as thefibration compression rule; when colors are obtained using covering symmetry, we will refer to it as thecovering compression rule\.

This compression rule strictly maintains the identical forward computation \(Methods[4\.13\.1](https://arxiv.org/html/2609.01768#S4.SS13.SSS1)\), preserving the network’s exact loss function\. In the case of quasi\-fibers and quasi\-covers, the activity and loss are preserved only approximately, bounded by the precision and a data\-dependent positive coefficientKc\(ℓ\)K^\{\(\\ell\)\}\_\{c\}:

Δ​L\(ℓ\)≲Kc\(ℓ\)​εop​εfib\.\\displaystyle\\Delta L^\{\(\\ell\)\}\\lesssim K^\{\(\\ell\)\}\_\{c\}\\varepsilon\_\{\\rm op\}\\varepsilon\_\{\\rm fib\}\.\(7\)
The compression rule \([6](https://arxiv.org/html/2609.01768#S2.E6)\) can be applied in different network architectures \(Methods[4\.12](https://arxiv.org/html/2609.01768#S4.SS12)\)\. In CNNs, features are treated as nodes and can be compressed accordingly \(Methods[4\.13\.2](https://arxiv.org/html/2609.01768#S4.SS13.SSS2), Fig\.[6](https://arxiv.org/html/2609.01768#S4.F6)a\)\. In RNNs, such as the LSTM, cover symmetries emerge in the gating mechanisms, with the weight matrices of the forget and input gates playing the role of fibers and opfibers \(Methods[4\.13\.3](https://arxiv.org/html/2609.01768#S4.SS13.SSS3), Fig\.[6](https://arxiv.org/html/2609.01768#S4.F6)b\)\. In transformer networks, symmetries emerge in the weight matrices of the query and key \(Methods[4\.13\.3](https://arxiv.org/html/2609.01768#S4.SS13.SSS3), Fig\.[6](https://arxiv.org/html/2609.01768#S4.F6)c\), while the value matrix can be treated the same as the weight matrices in an MLP\.

### 2\.2Empirical results: emergence, compression and breaking of symmetries

In the following sections, we will empirically demonstrate the emergence of cover symmetries during stochastic gradient descent, and argue that this symmetry formation captures regularity in the data\. We also show that these symmetries allow substantial compression of networks with negligible change in performance, while outperforming existing pruning methods\. Finally, we argue that the emergence of covering symmetry results in the loss of plasticity in deep networks and show empirically how fiber symmetry breaking can achieve a new state\-of\-the\-art in continual learning\.

#### Emergence of symmetries during learning

First, we provide empirical evidence for the emergence of fibers, opfibers and covers \(Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)d\) during SGD for an MLP trained in the MNIST classification problem \(Methods[4\.14](https://arxiv.org/html/2609.01768#S4.SS14)\)\. Since weights are initialized with random values, at the start of learning, all nodes form trivial \(single\-node\) fibers\. As learning progresses, the nodes start to partition into multi\-node covers, forming fibers, so that the number of unique fibers and opfibers decreases\. We observe larger fibers in later layers of the network \(Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)d, left\), which is explained in our theory by the increasingly more relaxed conditions for the fibers: Each layer removes an additional degree of freedom whenever two nodes satisfy the fiber equation \([14](https://arxiv.org/html/2609.01768#S4.E14)\)\. As we move across layers, the subspace of solutions increases and the fibers grow in size\. Similar reasoning explains why opfibers are larger in the initial layers \(Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)d, right\)\.

The temporal dynamic of cover and fiber formation is illustrated in the accompanying[video S1](https://github.com/makselab/fibrations_in_dnns/blob/main/MNIST_Symmetries/animation.mp4)\. The video shows that \(quasi\-\) opfibers first appear in the output layers and their emergence propagates backwards across layers, while \(quasi\-\) fibers first emerge in the first layer and propagate to the output\. This is consistent with the stability rules \(Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)b\), that are the basis of the covering theorem \(Methods[4\.11](https://arxiv.org/html/2609.01768#S4.SS11)\)\. The rules dictate that fibers in one layer preserve fibers in the next, while opfibers preserve opfibers in the preceding layer\. Thus, opfiber formation tends to propagate backward from the output layer, while fiber formation tends to propagate forward from the input as learning progresses\. We confirm this empirically for quasi\-fibers and quasi\-covers in an MLP in Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)a\.

We also demonstrate the emergence of symmetry in a CNN trained in ImageNet classification and in an LSTM trained with reinforcement learning to play the Atari game Beam Rider \(Methods[4\.14](https://arxiv.org/html/2609.01768#S4.SS14)\)\. In the CNN, quasi\-covers grow in both convolutional and dense layers as training progresses \(i\.e\. number of unique covers decreases\), and classification accuracy improves \(Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)c\)\. For the LSTM, quasi\-covers first grow in size, i\.e\., unique nodes decrease, but these approximate symmetries break upon further training \(Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)e\) due to exploration incentives in the reinforcement learning algorithms \(Methods[4\.14](https://arxiv.org/html/2609.01768#S4.SS14)\)\.

#### Synchronization captures regularity in the data

The activity synchronization theorem, Eq\. \([15](https://arxiv.org/html/2609.01768#S4.E15)\), establishes that the nodes of a fiber have the same activity for all inputs\. We ask how node synchronization changes when we limit the input space and, in particular, if we draw inputs from individual classes in a classification task\. We find that the correlation matrices for node activity with data drawn from individual classes form large clusters of synchronized nodes \(Fig\.[9](https://arxiv.org/html/2609.01768#S4.F9)\)\. These class\-conditional clusters are a coarsening of the partitioning into fibers \(see Methods[4\.21](https://arxiv.org/html/2609.01768#S4.SS21)\- Fig\.[10](https://arxiv.org/html/2609.01768#S4.F10)\)\. Thus, each class\-conditional synchrony cluster can be decomposed into multiple fibers, forming hierarchical clusters of nodes\. The increasing size of fibers across layers is paralleled by the observed node synchronization \(Fig\.[9](https://arxiv.org/html/2609.01768#S4.F9)\), which is more prevalent in later layers of the network\.

One can interpret the class\-conditional synchronization clusters in terms of features of the input: It is well accepted that node activity in hidden layers captures a hierarchy of features in the stimulus, that are shared across classes; e\.g\. simple examples for 2D CNN are edges, junctions, and corners\[[33](https://arxiv.org/html/2609.01768#bib.bib33)\]\. The intersection of all class\-conditional clusters, capturing all the shared features, is embodied in the fibers\. The increasing size of fibers across layers captures the hierarchical organization of features\. Therefore, fibers capture the hierarchical regularity present in all training data\. In the course of training, these clusters of synchronization emerge in a hierarchical process \(Fig\.[11](https://arxiv.org/html/2609.01768#S4.F11)a,b\) consistent with the Coarse\-Graining Theorem \(Methods[4\.22](https://arxiv.org/html/2609.01768#S4.SS22)\)\.

#### Fibration and covering compression versus performance

Once fibers have formed, we can compress the network\. As we relax the thresholdsεfib\\varepsilon\_\{\\rm fib\}andεop\\varepsilon\_\{\\rm op\}a larger number of nodes can be grouped into quasi\-fiber and \-opfiber, and we achieve stronger network compression\. In practice, we have found that compression with the same valueε\\varepsilonproduces different loss changesΔ​L\\Delta Lin different layers, consistent with the dependence ofKc\(ℓ\)K^\{\(\\ell\)\}\_\{c\}in Eq\. \([7](https://arxiv.org/html/2609.01768#S2.E7)\)\. We therefore adopted layer\-dependent thresholds and developed efficient search algorithms to find the optimal set ofεfib\(ℓ\),εop\(ℓ\)\\varepsilon\_\{\\rm fib\}^\{\(\\ell\)\},\\varepsilon\_\{\\rm op\}^\{\(\\ell\)\}to achieve maximum compression with a fixed budgetΔ​L=εLoss\\Delta L=\\varepsilon\_\{\\rm Loss\}\(see Methods[4\.16](https://arxiv.org/html/2609.01768#S4.SS16)\)\.

![Refer to caption](https://arxiv.org/html/2609.01768v1/Fig_3.png)Figure 3:Empirical results\.\(a\)MLP trained on MNIST dataset\. Classification accuracy increases across learning epochs \(green\), while the number of trivial \(single\-node\) quasi\-covers \(εop=1\\varepsilon\_\{\\rm op\}=1\) decreases in favor of larger multi\-node quasi\-covers \(black dashed curves\)\.\(b\)Accuracy of the compressed network depends on the level of compression with fibration compression \(olive\) and covering compression \(red\)\. Alternative pruning methods at different levels of compression \(blue and pink curves\)\. Dashed curves indicate the change in loss\.\(c\)Same as \(a\) but for CNN trained on ImageNet\.\(d\)Same as \(b\) for CNN\. Red and olive curves correspond to optimal covering compression with different values ofεop\\varepsilon\_\{\\text\{op\}\}\.\(e\)Agent with LSTM trained via reinforcement learning to play the Atari game Beam Rider\. Returns increase with learning episode \(green curve\)\. Black curve shows the proportion of quasi\-covers \(εfib=0\.5,εop=1\.0\\varepsilon\_\{\\rm fib\}=0\.5,\\varepsilon\_\{\\rm op\}=1\.0\) as a function of training episode\. In panels \(a\)\-\(c\)\-\(e\) arrows point to the axis associated with each curve\.\(f\)Compression of a sequence\-to\-sequence Transformer trained for German\-English translation\. Optimalε→fib∗\\vec\{\\varepsilon\}^\{\*\}\_\{\\rm fib\}obtained for each bock in the network architecture\. Red indicated stronger compression, green, no compression\. ToleranceεLoss=0\\varepsilon\_\{\\rm Loss\}=0accepts no change in loss \(left\) or a threshold valueεLoss=0\.45\\varepsilon\_\{\\rm Loss\}=0\.45that will not affect translation task performance \(right\) \(see Fig\.[7](https://arxiv.org/html/2609.01768#S4.F7)c, right\)\.\(g\)Continual learning with sequential ImageNet binary classification tasks, using conventional SGD \(blue\), Continual Backpropagation \(brown\), and the proposed FSB \(red\)\.Compression in MLPs:For a three\-layer MLP trained in MNIST, the optimal covering compression \(Fig\.[7](https://arxiv.org/html/2609.01768#S4.F7)a, right, red curve\) achieves substantial compression of down to 17% of the original network size with no change in loss or accuracy \(Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)b, red dashed and solid curves, respectively\)\. We obtain similar results when we limit the search to fibration compression \(i\.e\. we search for three optimalεfib\(ℓ\)\\varepsilon\_\{\\rm fib\}^\{\(\\ell\)\}while keepingεop=1\\varepsilon\_\{\\rm op\}=1, Fig\.[7](https://arxiv.org/html/2609.01768#S4.F7), right, olive\)\. We find that covering compression \([3](https://arxiv.org/html/2609.01768#S2.F3)b, red\) tends to outperform fibration compression \(Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)b, olive\)\. We compared this with alternative compression methods\[[34](https://arxiv.org/html/2609.01768#bib.bib34)\]such as randomly pruning nodes \(blue curve\) and pruning nodes with small weights \(L2\-norm, pink curve\)\. Fibration and covering compression identify and merge nodes that play identical functional roles and thus have no impact on loss\. In contrast, these conventional pruning methods remove potentially unique contributions and therefore immediately impact the loss\.

Compression in CNNs:Similar results are observed for a CNN trained on ImageNet \(Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)d\)\. The red curve represents covering compression with allεop=0\.75\\varepsilon\_\{\\rm op\}=0\.75\. The olive curve, in turn, corresponds toεop=1\.0\\varepsilon\_\{\\rm op\}=1\.0, which is equivalent to fibration compression\. Again, we find that covering compression outperformed fibration compression as well as conventional pruning methods\. Here we found the optimal fibration thresholds at a given opfiber threshold \(see Fig\.[7](https://arxiv.org/html/2609.01768#S4.F7), right\)\.

Compression in Transformers:We also tested compression in a sequence\-to\-sequence Transformer trained for German\-English translation \(Methods[4\.14](https://arxiv.org/html/2609.01768#S4.SS14)\) using the same data and architecture as in previous work\[[35](https://arxiv.org/html/2609.01768#bib.bib35)\]\. With three encoder and three decoder layers, the network has 41 different blocks, each with its ownεfib\\varepsilon\_\{\\rm fib\}\([3](https://arxiv.org/html/2609.01768#S2.F3)f\)\. We only tested fibration compression \(with fixedεop=1\\varepsilon\_\{\\rm op\}=1\)\. The optimal values ofεfib∗\{\\varepsilon\}^\{\*\}\_\{\\rm fib\}are shown in the matrices of Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)f\. Note thatεfib=1\\varepsilon\_\{\\rm fib\}=1implies strong compression, while values close to 0 imply weak compression\. Despite a strict limit to the change in loss \(εLoss=0\\varepsilon\_\{\\rm Loss\}=0\), we can compress the network to70%70\\%of its size \(Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)f, left\)\. However, we can tolerate some increase in loss without affecting translation performance \(Fig\.[7](https://arxiv.org/html/2609.01768#S4.F7)c, left\)\. At this tolerance \(εLoss=0\.45\\varepsilon\_\{\\rm Loss\}=0\.45\) the transformer can be compressed to18%18\\%of its original size \(Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)f, right\)\.

#### Fibration symmetry breaking for continual learning

Deep networks have been found to lose their ability to learn when trained sequentially on multiple new tasks\[[4](https://arxiv.org/html/2609.01768#bib.bib4),[5](https://arxiv.org/html/2609.01768#bib.bib5)\]\. We evaluate this on the continuous ImageNet task\[[36](https://arxiv.org/html/2609.01768#bib.bib36)\]\(Methods[4\.19](https://arxiv.org/html/2609.01768#S4.SS19)\)\. First, we observe that conventional SGD stops learning after approximately 2,000 tasks \(Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)g, blue curve\), reproducing the results of\[[3](https://arxiv.org/html/2609.01768#bib.bib3)\]\. We find that this stagnation is accompanied by cover formations \(Fig\.[8](https://arxiv.org/html/2609.01768#S4.F8)\)\. The emergence of symmetry induces parameter tying, which reduces the degrees of freedom in the parameter space\. This constraint directly explains the observed loss of network plasticity: with fewer independent directions for optimization, learning stagnates\.

Fibration symmetries that emerge during SGD reduce the network’s degrees of freedom and manifest as redundant, synchronized activity that limits the repertoire of learnable features\. While several ad\-hoc heuristics attempt to promote diversity \(see Discussion\), they operate indiscriminately by preventing or breaking the fibers that naturally emerge from the data with SGD\.

In contrast, we proposeFibration Symmetry Breaking\(FSB\), a principled and targeted approach to symmetry breaking\[[37](https://arxiv.org/html/2609.01768#bib.bib37)\]by first compressing covers and then randomly choosing the remaining weights as follows \(Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)e\):

1. 1\.Prune: Identify covers and reduce the graph to the covering base by applying the fibration compression rule Eq\. \([6](https://arxiv.org/html/2609.01768#S2.E6)\) to ensure that the learned function of the network and its performance are preserved\.
2. 2\.Sprout: Add naive nodes that restore the original size\. Each new node is initialized with random input and output weights \(with zero outgoing weights to the nodes in the base\)\. This restores degrees of freedom for further learning without interfering with the learned base\.

We compared performance with the state\-of\-the\-art “Continual Backpropagation”\[[3](https://arxiv.org/html/2609.01768#bib.bib3)\], which has been shown to outperform existing heuristics such as shrink\-and\-perturb\[[38](https://arxiv.org/html/2609.01768#bib.bib38)\]\. Continual backpropagation resets the nodes that have become inactive\. In Methods[4\.20](https://arxiv.org/html/2609.01768#S4.SS20), we show that this technique is a special case of our symmetry\-breaking mechanism\. In the sequential ImageNet task \(Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)g\) continual backpropagation \(green\) and our method FSB \(red\) are comparable up to task 1,500\. However, beyond that point, our method outperforms continual backpropagation, cutting the remaining deficit in accuracy by half \(from 90% to 95% by task 5,000\)\. This indicates that the FSB continues to capture useful features that generalize between classes, outperforming the state of the art\.

## 3Discussion

Our work frames deep learning not as a process of brute\-force parameter\-tuning but as a fundamental mechanism of structure formation\. Through this lens, optimization naturally aligns a network’s internal topology to the latent geometry of the data, isolating a compact set of features to describe the data\. This framework provides a conclusive, mathematically rigorous explanation for long\-observed empirical phenomena that have previously lacked a unified theoretical basis\. For instance, the spontaneous emergence of fibration symmetry explains “node collapse” and the generation of redundant, synchronized neural activity, particularly within the deeper layers of over\-parameterized models\[[24](https://arxiv.org/html/2609.01768#bib.bib24),[25](https://arxiv.org/html/2609.01768#bib.bib25),[15](https://arxiv.org/html/2609.01768#bib.bib15)\]\. We show that stochastic gradient descent actively stabilizes these covering symmetries, acting similarly to noise\-induced stabilization in complex dynamical systems\[[39](https://arxiv.org/html/2609.01768#bib.bib39)\]\. We propose that fibration symmetries emerge from a hierarchical loss landscape \(Methods[4\.22](https://arxiv.org/html/2609.01768#S4.SS22)\)\. By demonstrating that networks navigate toward increasingly coarse symmetry structures during optimization, this geometric perspective resolves the classic paradox of over\-parameterization: gradient descent finds structured symmetric solutions where the emergent model is dramatically simpler than the raw parameter counts suggest\.

Beyond explaining the learning dynamics, this framework unifies a wide array of seemingly disparate ad\-hoc AI heuristics under a single theoretical view\. Hard\-coded inductive biases, such as weight\-tying in CNNs, can now be understood as simply pre\-defined graph symmetries; our results demonstrate how such constraints can instead emerge organically through learning\. Crucially, our theory explains why a variety of popular empirical techniques successfully enhance network performance by actively breaking or preventing these symmetries\. For example, dropout\[[40](https://arxiv.org/html/2609.01768#bib.bib40)\]breaks symmetries by updating only a fraction of nodes selected at random, while residual connections\[[41](https://arxiv.org/html/2609.01768#bib.bib41)\]break symmetries that would otherwise develop within the skipped layers \(Methods[4\.12](https://arxiv.org/html/2609.01768#S4.SS12)\)\. Similarly, methods designed to explicitly avoid representation redundancy\[[42](https://arxiv.org/html/2609.01768#bib.bib42)\]directly prevent fiber formation; this includes structural redundancy reduction algorithms\[[43](https://arxiv.org/html/2609.01768#bib.bib43)\]\(Methods[4\.18](https://arxiv.org/html/2609.01768#S4.SS18)\), feature recombination\[[44](https://arxiv.org/html/2609.01768#bib.bib44)\], or minimization for cross\-correlation\[[43](https://arxiv.org/html/2609.01768#bib.bib43)\]\(Methods[4\.18](https://arxiv.org/html/2609.01768#S4.SS18)\)\. While these existing approaches operate blindly, our symmetry\-driven pruning offers an exact, theory\-motivated methodology for model compression\. Because fibration compression merges functionally redundant nodes rather than pruning unique computations, it preserves the exact function of the model, compressing models to 17\-18% or their original size\. This structural compression complements downstream techniques such as weight quantization\[[45](https://arxiv.org/html/2609.01768#bib.bib45),[46](https://arxiv.org/html/2609.01768#bib.bib46)\]or linear low\-rank matrix decompositions \(Methods[4\.23](https://arxiv.org/html/2609.01768#S4.SS23)\)\.

Importantly, our framework directly resolves the critical trade\-off between stability and flexibility in lifelong learning systems\. The emergence of covers under standard SGD induces parameter tying, which restricts optimization and directly causes the well\-documented loss of network plasticity over time\[[5](https://arxiv.org/html/2609.01768#bib.bib5)\]\. Rather than applying indiscriminate resets to final layers or specific nodes, which often fail to fully overcome this stagnation, our Fibration Symmetry Breaking protocol offers a surgical alternative: it compresses the emergent covers to safeguard past knowledge and injects randomized, naive nodes to reclaim lost degrees of freedom without disrupting the learned base\. The resulting 50% reduction in the performance gap on sequential ImageNet benchmarks compared to state\-of\-the\-art continuous backpropagation\[[3](https://arxiv.org/html/2609.01768#bib.bib3)\]establishes FSB as a highly plausible foundation for true lifelong learning systems\.

Several theoretical and practical horizons remain open for investigation\. First, establishing the optimal sequencing to combine fibration compression with low\-rank and quantization pipelines represents an immediate engineering frontier\. Second, while our current approach relies on static model compression after training, developing algorithms for dynamic compression during training may allow stronger compression factors, or conversely, continual learning with larger datasets\. Finally, while our framework outlines how optimization trajectories navigate downhill through a tree of increasingly coarse\-grained symmetry quotients, the relation of this to the hierarchical structures in the data itself remains to be more strictly formalized\.

In conclusion, our work frames learning not as parameter\-tuning, but as structure formation\. The process transforms the opaque black box into an interpretable colored graph, where emergent symmetries reveal the data’s learned regularities\. This theory\-driven perspective offers a path to designing future AIs in which mathematical principles, not brute\-force scaling, drive performance, generalization, and interpretability\.

## 4Methods

### 4\.1Global and local symmetries in graphs: coverings, fibrations, and opfibrations

We first discuss symmetries in a binary \(unweighted\) graph where connections are restricted to binary values \(0 or 1\) to introduce the main concepts didactically, following Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)\. Graph fibrations were first introduced for binary graphs by Boldi and Vigna\[[21](https://arxiv.org/html/2609.01768#bib.bib21)\]based on the categorical definition of Grothendieck fibration from\[[22](https://arxiv.org/html/2609.01768#bib.bib22)\]\. Methods[4\.5](https://arxiv.org/html/2609.01768#S4.SS5)generalizes the definition to weighted graphs and hypergraphs for applications to DNNs\.

A directed graphG=\(NG,AG\)G=\(N\_\{G\},A\_\{G\}\)consists of a setNGN\_\{G\}of nodes and a setAGA\_\{G\}of connections\. Each connection is associated with a source and a target node, and one way to represent the full structure of the graph is its adjacency matrix\. For every nodeu∈NGu\\in N\_\{G\}, there is a corresponding input treeTuT\_\{u\}that represents the set of all paths ofGGending inuu\. Similarly, there exists a corresponding output treeT^u\\hat\{T\}\_\{u\}that represents the set of all paths ofGGstarting fromuu\. We say that two input treesTuT\_\{u\}andTvT\_\{v\}are isomorphic \(Tu∼TvT\_\{u\}\\sim T\_\{v\}\) when there is a bijective mapτu→v:Tu→Tv\\tau\_\{u\\to v\}:T\_\{u\}\\rightarrow T\_\{v\}, which maps the nodes and connections ofTuT\_\{u\}one\-to\-one to the nodes and connections ofTvT\_\{v\}\. The same definition and notation apply to output trees with a bijective map denoted byτ^u→v:T^u→T^v\\hat\{\\tau\}\_\{u\\to v\}:\\hat\{T\}\_\{u\}\\rightarrow\\hat\{T\}\_\{v\}\.

For graphGG,

1. 1\.Afibration symmetryofGGis a surjective homomorphismφfib:G→B\\varphi\_\{\\rm fib\}:G\\to Bthat preserves the input tree: ∀u,v∈NG:φfib​\(u\)=φfib​\(v\)∈NB⇔Tu∼Tv\.\\forall u,v\\in N\_\{G\}:\\varphi\_\{\\rm fib\}\(u\)=\\varphi\_\{\\rm fib\}\(v\)\\in N\_\{B\}\\iff T\_\{u\}\\sim T\_\{v\}\.\(8\)
2. 2\.Anopfibration symmetryofGGis a surjective homomorphismφop:G→B\\varphi\_\{\\rm op\}:G\\to Bthat preserves the output tree: ∀u,v∈NG:φop​\(u\)=φop​\(v\)∈NB⇔T^u∼T^v\.\\forall u,v\\in N\_\{G\}:\\varphi\_\{\\rm op\}\(u\)=\\varphi\_\{\\rm op\}\(v\)\\in N\_\{B\}\\iff\\hat\{T\}\_\{u\}\\sim\\hat\{T\}\_\{v\}\.\(9\)
3. 3\.Acovering symmetryofGGis a surjective homomorphismφcov:G→B\\varphi\_\{\\rm cov\}:G\\to Bthat preserves the input and output trees: ∀u,v∈NG:φcov​\(u\)=φcov​\(v\)∈NB⇔Tu∼Tv&T^u∼T^v\.\\forall u,v\\in N\_\{G\}:\\varphi\_\{\\rm cov\}\(u\)=\\varphi\_\{\\rm cov\}\(v\)\\in N\_\{B\}\\iff T\_\{u\}\\sim T\_\{v\}\\qquad\\&\\qquad\\hat\{T\}\_\{u\}\\sim\\hat\{T\}\_\{v\}\.\(10\)
4. 4\.Anautomorphismof a graphGGis a bijective mapπa​u​t​o:G→G\\pi\_\{auto\}:G\\to G, such that the pair of nodesuuandvvforms an connection\(u,v\)\(u,v\)if and only if \(πa​u​t​o​\(u\),πa​u​t​o​\(v\)\\pi\_\{auto\}\(u\),\\pi\_\{auto\}\(v\)\) also forms an edge\. Automorphisms are also called permutation symmetries\.

Each symmetry, fibration, opfibration, covering, and automorphism, gives rise to partitions called \(respectively\): fiber𝒞fib\\mathcal\{C\}^\{\\rm fib\}, opfiber𝒞op\\mathcal\{C\}^\{\\rm op\}, cover𝒞cov\\mathcal\{C\}^\{\\rm cov\}, and orbit partitions𝒞auto\\mathcal\{C\}^\{\\rm auto\}, which are hierarchical in the sense of coarsening:𝒞fib≥𝒞cov≥𝒞auto\\mathcal\{C\}^\{\\rm fib\}\\geq\\mathcal\{C\}^\{\\rm cov\}\\geq\\mathcal\{C\}^\{\\rm auto\}and𝒞op≥𝒞cov≥𝒞auto\\mathcal\{C\}^\{\\rm op\}\\geq\\mathcal\{C\}^\{\\rm cov\}\\geq\\mathcal\{C\}^\{\\rm auto\}\.

Automorphisms form symmetry groups\. That is, the set of automorphisms of a graph satisfies the composition law, associativity, and has an inverse and identity\. They are global symmetry transformations of the graph, as they apply a global permutation of all nodes that preserves the global adjacency matrix; that is, the connections between all nodes remain the same before and after the permutation\.

Two nodes are in an orbit iff:

u​∼auto​v⇔∃πa​u​t​o∈Aut​\(G\):πa​u​t​o​\(u\)=v,u\\underset\{\\rm auto\}\{\\sim\}v\\iff\\exists\\,\\pi\_\{auto\}\\in\\text\{Aut\}\(G\):\\pi\_\{auto\}\(u\)=v,\(11\)whereAut​\(G\)\\text\{Aut\}\(G\)is the symmetry group ofGG\. The orbits form the orbital partition of the graph\.

A partition𝒞\\mathcal\{C\}of a graphGGisfinerthan \(also calleda refinement of\) a partition𝒞′\\mathcal\{C\}^\{\\prime\}if every element of𝒞\\mathcal\{C\}is a subset of some element of𝒞′\\mathcal\{C\}^\{\\prime\}\. That is,𝒞\\mathcal\{C\}is a further fragmentation of𝒞′\\mathcal\{C\}^\{\\prime\}\. We denote this case as𝒞≤𝒞′\\mathcal\{C\}\\leq\\mathcal\{C\}^\{\\prime\}\. In this example,𝒞′\\mathcal\{C\}^\{\\prime\}iscoarserthan𝒞\\mathcal\{C\}\(𝒞′≥𝒞\\mathcal\{C\}^\{\\prime\}\\geq\\mathcal\{C\}\) and𝒞′\\mathcal\{C\}^\{\\prime\}is acoarsening \(merger\) of elements of𝒞\\mathcal\{C\}\. The coarsest possible partition of a graph is one in which all nodes belong to a single partition\. This is a graph with maximal symmetry\. The finest possible partition is the \(trivial\) partition into singletons, which is the partition of a graph with no symmetry at all \(or with only trivial symmetry\)\.

As seen in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1), for a directed graph, the fiber and the opfiber partitions are two different coarsenings of the cover partition, and the cover partition is a coarsening of the orbit partition\. In principle, there is no relation \(finer or coarser\) between the fiber and the opfiber partition\. However, when the graph is undirected, the fiber, opfiber, and cover partitions coincide\. Yet, they are still a coarsening of the orbital partition\. Furthermore, as shown in Fig\.[11](https://arxiv.org/html/2609.01768#S4.F11)b, the cluster synchronization partition \(obtained dynamically by clustering activity synchronization in the forward pass, averaging over samples\) is a coarsening of the fiber partition \(but not of the opfiber partition\)\.

All four partitions, fibers, opfibers, covers, and orbits, are different balanced colorings of the graph\[[27](https://arxiv.org/html/2609.01768#bib.bib27)\]: fibers are in\-balanced colorings, opfibers are out\-balanced colorings, and covers are in\- and out\-balanced colorings\. Orbits are also in\- and out\-balanced colorings\. However, orbits have an extra condition compared to covers: two nodes in an orbit must be obtainable from one another through the application of a permutation symmetry \(automorphism\) of the graph\.

For any partition, we will use the notationccto indicate some color and\|c\|\|c\|the cardinality of the color, i\.e\., the number of nodes with the same colorcc\.

Due to the refinement relations, the number of balanced colors increases from the \(op\)fiber partition to cover to orbit partition\. This is reflected in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)with 9 colors for fibers and 9 colors for opfibers as compared to 10 colors for covers and 12 colors for orbits \(consider that the blank nodes in the figure have all different colors\)\. This also reflects the increase in symmetry from automorphisms to covers to \(op\)fibers\.

We show an example of an automorphism that maps 1 to 2, 2 to 1, 3 to 4, 4 to 3, 5 to 6, 6 to 5, and all other nodes map to themselves in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)d\. It is denoted in cycle notation by:

π=\(1 2\)​\(3 4\)​\(5 6\)\.\\pi=\(1\\,2\)\(3\\,4\)\(5\\,6\)\.\(12\)The other nodes are trivially permuted\.

Fibrations, opfibrations, and coverings strictly generalize automorphisms by relaxing the global constraints to impose only a local preservation of structure, such as the input and/or output trees\. Examples of these symmetries in a graph are shown in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)\. These symmetries do not form groups; they form categories\. The color\-preserving isomorphisms between \(output\)input trees form groupoids \(i\.e\., groups without a composition law\)\[[27](https://arxiv.org/html/2609.01768#bib.bib27)\]\. The fibration framework is therefore also known as ’the groupoid formalism’ in the work of Golubitsky and Stewart\[[27](https://arxiv.org/html/2609.01768#bib.bib27)\]\.

Remark 1\. One might wonder why coverings are not automorphisms, since both preserve inputs and outputs\. The key distinction is scope: coverings preserve the input and output structure of the nodes within each cover, while automorphisms preserve the input and output structure globally across the entire graph\.

To exemplify the meaning of input and output trees and its relation with automorphism, we consider the graphGGin Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)c and call the remaining nodes as: first hidden layer: 7 \(top blank\) and 8 \(bottom blank\), second hidden layer: 9 \(top red\), 10 \(bottom red\), third hidden layer: 11 \(top blue\), 12 \(bottom blue\)\. We also call 13 and 14 the top and bottom inputs, and 15 the graph’s output\. Red nodes 9 and 10 form a cover since they have isomorphic input and output trees\. That is, there is an input\-isomorphism:τ9→10=\(9,7,13,14\)→\(10,8,13,14\)\\tau\_\{9\\to 10\}=\(9,7,13,14\)\\rightarrow\(10,8,13,14\)and an output\-isomorphism:τ^9→10=\(9,11,12,15\)→\(10,11,12,15\)\\hat\{\\tau\}\_\{9\\to 10\}=\(9,11,12,15\)\\rightarrow\(10,11,12,15\)from the input and output tree of node 9 to 10\. There is also an isomorphism between connections, but since all connections are the same \(binary graph\), there is no need to specify it\.

One may wonder if nodes 9 and 10 might form an orbit, i\.e\., if they are also symmetric under a permutation symmetry\. If we permute 9 and 10, then 7 and 8 need to be permuted as well to preserve the inputs of 9 and 10 \(11 and 12 do not need to be permuted, as the outputs of 9 and 10 are preserved by the permutation\)\. Thus, the permutationπ=\(9 10\)​\(7 8\)\\pi=\(9\\,10\)\(7\\,8\), in principle, preserves the adjacency of 9 and 10\. However, node 8 is now connected to 3 and 4, while node 7 is not, and there is no other permutation that can restore the original adjacency \(for instance, permuting 3 and 4 will not fix it\)\. Thus, whileπ\\piis a valid permutation of the graph, it is not a permutation symmetry \(automorphism\) of the graph\. The violation occurs not in the input or output trees of 9 and 10\. In fact, one can check that the input and output trees of nodes 9 and 10 are still isomorphic in the permuted graphπ⁡\(G\)\\pi\(G\)\. The violation occurs in other parts of the graph: the inputs of nodes 3 and 4\. This simple example illustrates the ’local’ versus ’global’ preservation condition for \(op\)fibrations/coverings versus automorphisms, an important ingredient of the framework presented here\.

Remark 2\. Note that ’local’ gauge symmetries in physics have a different meaning\[[28](https://arxiv.org/html/2609.01768#bib.bib28)\]\. They refer to spacetime locality captured by the fiber bundle, not the fibration\-theoretic locality\. Fibrations are more general than fiber bundles since they only require relaxed fibers with local symmetry, rather than the symmetry groups required by the fibers in a fiber bundle\. Fiber bundles are widespread in theoretical physics but not in biology or AI \(see\[[12](https://arxiv.org/html/2609.01768#bib.bib12)\]for further details\)\.

Remark 3\. In our formulation, nodes with no inputs \(for instance, all nodes in the input layer of an MLP\) are assigned different balanced colors, ie, each node belongs to a different fiber\. Same for the output layer and opfibers\. This differs from the original definition of graph fibrations in\[[21](https://arxiv.org/html/2609.01768#bib.bib21)\]\.

Remark 4\. WhenGGis a graph with flavored connections\[[47](https://arxiv.org/html/2609.01768#bib.bib47)\], the previous symmetry definitions remain valid using colored input and output trees\. Two flavored trees \(TuT\_\{u\}with flavorquq\_\{u\}andTvT\_\{v\}with flavorqvq\_\{v\}\) are considered isomorphic if there exists a bijectionτ:Tu→Tv\\tau:T\_\{u\}\\rightarrow T\_\{v\}that is a one\-to\-one mapping of both nodes and connections fromTuT\_\{u\}toTvT\_\{v\}and preserves flavors of the connections, i\.e\.∀a∈ATu:qu​\(a\)=qv∘τ⁡\(a\)\\forall a\\in A\_\{T\_\{u\}\}:q\_\{u\}\(a\)=q\_\{v\}\\circ\\tau\(a\)\. When all connections have the same flavor, the initial definitions are recovered as a special case\.

### 4\.2Lifting property by fibration symmetry: Construction of fibration base

So far, we have not provided a complete definition of a baseBBfor any symmetry because Eqs\. \([8](https://arxiv.org/html/2609.01768#S4.E8),[9](https://arxiv.org/html/2609.01768#S4.E9),[10](https://arxiv.org/html/2609.01768#S4.E10)\) only give us information aboutNBN\_\{B\}but not about the connectionsABA\_\{B\}\. The connections for base fibrations and opfibrations are defined usinglifting property\. In the case of fibration symmetry, thislifting propertyis based on the structure of the in\-neighborhoods of the graphGG\. The connections in the case of binary connections are selected as follows:

1. 1\.Calculate the partition𝒞fib\\mathcal\{C\}^\{\\rm fib\}\(see colors in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)a\)\.
2. 2\.Identify the in\-neighborhood for all nodes ofGG\.
3. 3\.Replace the node IDs in the in\-neighborhoods with its color to obtain the in\-neighborhoods of the nodes in the fibration baseBfibB\_\{\\rm fib\}\.
4. 4\.Connect the nodes ofBfibB\_\{\\rm fib\}using information from the neighborhoods\. Note that each node in the baseBBcan receive more than one connection from another \(see the base in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)a\)\.

Multiple connections indicate that the input is the addition of single\-connection inputs\. For networks with weighted connections, this addition is reflected in the sum of Eq\. \([6](https://arxiv.org/html/2609.01768#S2.E6)\) derived in Methods[4\.13\.1](https://arxiv.org/html/2609.01768#S4.SS13.SSS1)\.

For opfibration symmetry, the lifting property is analogous but differs in that it is based on the structure of the out\-neighborhoods of the graphGG\. The lifting property ensures that the input tree \(resp\., output\) of a node in the fibration \(opfibration\) base is isomorphic to the trees of the nodes in that fiber \(opfiber\)\.

For a covering symmetry – since it is simultaneously a fibration and an opfibration symmetry – the base can be constructed using either the fibration or the opfibration lifting property after calculating the partition𝒞cov\\mathcal\{C\}^\{\\rm cov\}\. Both cases are shown in Fig\.[4](https://arxiv.org/html/2609.01768#S4.F4)\. In general, the results will differ\. Therefore, one has to select either to follow a fibration compression \(preserving the inputs, as we have done in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)e\) or an opfibration compression\.

In this work, we will use the construction of the cover base via the fibration lifting property, as it preserves the forward computation of the deep networks\. This should not be confused with the proper fibration symmetry that compresses the fibers using the fibration map shown in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)a\.

Figure 4:Compression to the cover base:One can preserve the input tree or output tree, but generally not both unless all covers have the same cardinality which is not satisfied in this case, since 5 covers have cardinality 2 and 5 covers have cardinality 1 \(trivial\)\. Note that our notation uses white for trivial covers\. A more strict representation should assign a different color to every one of the trivial covers, but this representation would be harder to grasp with so many colors\.
### 4\.3Fibration lifting operation for weighted network

Here, we elaborate on the concept of fibration lifting for a weighted network\. The origin of the name “lifting” is the inverse operation of compression in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1), where the base is “lifted” to a full graph\. Since compression is not an injective function, the lifting is not unique\. For example, the following weightsWW\(full graph\) are transformed into the same weightsW~\\tilde\{W\}\(base\),

Wi​j\(ℓ\)=1\|c⁡\(j\)\|​W^c′​\(i\)​c​\(j\)\(ℓ\)\+Δi​jW^\{\(\\ell\)\}\_\{ij\}=\\frac\{1\}\{\|c\(j\)\|\}\\hat\{W\}^\{\(\\ell\)\}\_\{c^\{\\prime\}\(i\)c\(j\)\}\+\\Delta\_\{ij\}\(13\)where∑k∈c′m∈cΔk​m=0\\sum\_\{\\begin\{subarray\}\{c\}k\\in c^\{\\prime\}\\\\ m\\in c\\end\{subarray\}\}\\Delta\_\{km\}=0\.

In Fig\.[5](https://arxiv.org/html/2609.01768#S4.F5), we compress using Eq\. \([6](https://arxiv.org/html/2609.01768#S2.E6)\) fromG1G\_\{1\}to the baseB1′B^\{\\prime\}\_\{1\}; then,B1′B^\{\\prime\}\_\{1\}to the minimal baseBB\.BBis a base ofG1G\_\{1\}that has two fibers \(blue and red\)\. InB1′B^\{\\prime\}\_\{1\}, two nodes of the blue fiber are compressed; while in the minimal baseBB, the three nodes of the blue fiber are compressed\. Two networks in different configurations can have the same base\. For example,G1G\_\{1\}andG2G\_\{2\}can be compressed toB1′B^\{\\prime\}\_\{1\}, whileG3G\_\{3\}andG4G\_\{4\}can be compressed toB2′B^\{\\prime\}\_\{2\}\. In the end, these four graphs have the same minimal baseBB\. In this work, we address the case of weighted graphs\. With binary connection weights \(representing simply the connection or its absence\), the scenario is reduced to that described in\[[21](https://arxiv.org/html/2609.01768#bib.bib21)\]\.

All networksG1,G2,G3,G4G\_\{1\},G\_\{2\},G\_\{3\},G\_\{4\}, basesB1,B2B\_\{1\},B\_\{2\}, and the minimal baseBBshare the same forward computation\. That is, for every inputxx, the outputy^\\hat\{y\}is identical across all networks\. Moreover, for every pair\(x,y\)\(x,y\)in the dataset, the loss values are the same for all networks\. This implies that the loss functionℒ⁡\(w\)\\mathcal\{L\}\(w\)is the same for all aforementioned networks \(a degeneracy of the loss function\)\. More precisely, if two networksGGandG′G^\{\\prime\}have the same minimal baseBB, thenℒ⁡\(wG\)=ℒ⁡\(wG′\)\\mathcal\{L\}\(w\_\{G\}\)=\\mathcal\{L\}\(w\_\{G^\{\\prime\}\}\)wherewGw\_\{G\}andwG′w\_\{G^\{\\prime\}\}denote the parameters ofGGyG′G^\{\\prime\}, resp\. This result has important implications for the loss landscape as discussed in Methods[4\.22](https://arxiv.org/html/2609.01768#S4.SS22)and Fig\.[11](https://arxiv.org/html/2609.01768#S4.F11)d\.

![Refer to caption](https://arxiv.org/html/2609.01768v1/Fig_Lifting.png)Figure 5:Examples of compression and lifting operations as reduction and expansion of dimensions, respectively\. Colors represent fibers\.
### 4\.4Fibration symmetry implies activity synchronization

Here we formally generalize the concept of fibration symmetry from binary weights to continuous weights\. Two nodesiiandjjin layerℓ\\ellbelong to the same fiber \(denotedi​∼fib​ji\\underset\{\\rm fib\}\{\\sim\}j\) if and only if \(iff\):

- •Fibration symmetry: i​∼fib​j⇔∀c∈𝒞ℓ−1fib:∑k∈cWi​k\(ℓ\)=∑k∈cWj​k\(ℓ\)i\\underset\{\\rm fib\}\{\\sim\}j\\iff\\forall c\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}:\\quad\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{ik\}=\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{jk\}\(14\)

This criterion partitions the nodes in the layerℓ\\ellinto a coloring𝒞ℓfib\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\}based on the input weightsWi​k\(ℓ\)W^\{\(\\ell\)\}\_\{ik\}and the color partitioning of the previous layer𝒞ℓ−1fib\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}\. In words, this definition states that two nodes are in the same fiber iff for all colors from the previous layer the sums of weights from these colors are the same\. Or, more simply, the total input from each color is the same, which also ensures that their activity is the same\. This recursive definition starts in the input layer,ℓ=0\\ell=0, with all nodes \(d0d\_\{0\}nodes\) having distinct colors𝒞0fib=\{1,2,…,d0\}\\mathcal\{C\}^\{\\rm fib\}\_\{0\}=\\\{1,2,\.\.\.,d\_\{0\}\\\}, i\.e\., trivial fibers\. The definition of fibers in Eq\. \([14](https://arxiv.org/html/2609.01768#S4.E14)\) based on the sum of weights is the correct generalization from binary graphs \(Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)\) to weighted graphs, as we show in Methods[4\.5](https://arxiv.org/html/2609.01768#S4.SS5)\.

Similarly to symmetry\-induced invariance theorems in physics\[[10](https://arxiv.org/html/2609.01768#bib.bib10)\], fibration symmetry induces activity synchronization during forward inference\. This is the subject of the following theorem:

- •Activity synchronization:Given two nodesiiandjjin layerℓ\\ell, the following hold in networks with inputsxx: i​∼fib​j⟹hi\(ℓ\)=hj\(ℓ\)for any network input​xi\\underset\{\\rm fib\}\{\\sim\}j\\implies h^\{\(\\ell\)\}\_\{i\}=h^\{\(\\ell\)\}\_\{j\}\\qquad\\text\{for any network input \}x\(15\)

The proof of Eq\. \([15](https://arxiv.org/html/2609.01768#S4.E15)\) is in Methods[4\.6](https://arxiv.org/html/2609.01768#S4.SS6)\. The relation emerges naturally from the recursive dependence of neuronal activityhi\(ℓ\)h\_\{i\}^\{\(\\ell\)\}on activity in the previous layer through input weightsWi​k\(ℓ\)W^\{\(\\ell\)\}\_\{ik\}\. Similar proofs exist for dynamical systems in\[[12](https://arxiv.org/html/2609.01768#bib.bib12),[48](https://arxiv.org/html/2609.01768#bib.bib48),[49](https://arxiv.org/html/2609.01768#bib.bib49)\]\.

Equation \([1](https://arxiv.org/html/2609.01768#S2.E1)\) establishes the relationship between the network structure captured by its weights and the network function captured by its activity\. Therefore, the input tree structure governs the synchronization patterns\.

### 4\.5Input and output trees in weighted feedforward networks

In weighted feedforward networks \(e\.g\. Multi\-Layer Perceptrons\), an automorphism symmetry between two nodes would require that they are in the same layer and that they have identical weight vectors at the input and output, so that permuting the two nodes keeps the entire network unchanged\. Fibration and opfibration symmetries are less restrictive than automorphisms\. As we shall see, they specify a weaker constraint on the weights\.

In order to define fibers in weighted layered graphs, we first discuss isomorphic input trees\. The input treeTiT\_\{i\}of a nodeiiin layerℓ\\ellconsists of all paths from the input layer to nodeii\. For example, for a nodeiiin the first layer \(ℓ=1\\ell=1\), its input tree is the vector

Ti\(1\)=\[Wi​1\(1\),Wi​2\(1\),…,Wi​d0\(1\)\]\.T^\{\(1\)\}\_\{i\}=\[W^\{\(1\)\}\_\{i1\},W^\{\(1\)\}\_\{i2\},\.\.\.,W^\{\(1\)\}\_\{id\_\{0\}\}\]\.Then, two nodes have isomorphic input trees ifTi\(1\)=Tj\(1\)T^\{\(1\)\}\_\{i\}=T^\{\(1\)\}\_\{j\}\. Using this criterion, we can define fibers in the first layer\. For layerℓ=2\\ell=2, the input treeTi\(2\)T^\{\(2\)\}\_\{i\}of nodeiican be expressed as the concatenation of the input treesTm\(1\)T^\{\(1\)\}\_\{m\}of nodesmmin layerℓ=1\\ell=1, and their corresponding connection weightsWi​m\(2\)W^\{\(2\)\}\_\{im\}, i\.e\.

Ti\(2\)=\[T1\(1\),Wi​1\(1\),T2\(1\),Wi​2\(1\),…,Td1\(1\),Wi​d1\(1\)\]\.T^\{\(2\)\}\_\{i\}=\[T^\{\(1\)\}\_\{1\},W^\{\(1\)\}\_\{i1\},T^\{\(1\)\}\_\{2\},W^\{\(1\)\}\_\{i2\},\.\.\.,T^\{\(1\)\}\_\{d\_\{1\}\},W^\{\(1\)\}\_\{id\_\{1\}\}\]\.If the fibersccin the first layer are already known,Ti\(2\)T^\{\(2\)\}\_\{i\}can be represented more compactly as the concatenation of the input trees of the fibersTc\(1\)T^\{\(1\)\}\_\{c\}\(because they are all the same within a fiber\), but we do have to aggregate the weights as∑k∈cWi​c\\sum\_\{k\\in c\}W\_\{ic\}to replicate their sum of effects in layerℓ=1\\ell=1\. Therefore, two nodesiiandjjof the layerℓ=2\\ell=2have isomorphic input trees,Ti\(2\)=Tj\(2\)T^\{\(2\)\}\_\{i\}=T^\{\(2\)\}\_\{j\}, iff∀c:∑k∈cWj​k=∑k∈cWi​k\\forall c:\\sum\_\{k\\in c\}W\_\{jk\}=\\sum\_\{k\\in c\}W\_\{ik\}\. This criterion therefore defines the fiber for layerℓ=2\\ell=2\. This argument can be extended to the next layersℓ=3,…,N\\ell=3,\.\.\.,Nthus recursively defining fibers\. This leads to the equal\-sum criterion in the definition \([14](https://arxiv.org/html/2609.01768#S4.E14)\) of fiber in weighted graphs\. Similar formulations have been used in the context of dynamical system synchronization\[[50](https://arxiv.org/html/2609.01768#bib.bib50),[51](https://arxiv.org/html/2609.01768#bib.bib51)\]\.

In order to define flavored opfibers, we will now discuss isomorphic flavored output trees\. \(They are referred to as colored weights in\[[47](https://arxiv.org/html/2609.01768#bib.bib47)\], but here we use the word “flavor” to not confuse with node colors\)\. For a nodeiiin layerℓ=N−1\\ell=N\-1, its output tree is the concatenation of the output weightswwofiiwith its corresponding flavorsq⁡\(w\)q\(w\); i\.e\.,

T^i\(N−1\)=\[\(W1​i\(N\),q⁡\(W1​i\(N\)\)\),\(W2​i\(N\),q⁡\(W2​i\(N\)\)\),…,\(WdN​i\(N\),q⁡\(WdN​i\(N\)\)\)\]\.\\hat\{T\}^\{\(N\-1\)\}\_\{i\}=\[\(W^\{\(N\)\}\_\{1i\},q\(W^\{\(N\)\}\_\{1i\}\)\),\(W^\{\(N\)\}\_\{2i\},q\(W^\{\(N\)\}\_\{2i\}\)\),\.\.\.,\(W^\{\(N\)\}\_\{d\_\{N\}i\},q\(W^\{\(N\)\}\_\{d\_\{N\}i\}\)\)\]\\,\.We are interested in the case where the flavors of the connectionsWk​iW\_\{ki\}depend only onii\(i\.e\.q⁡\(Wk​i\)=q⁡\(i\)q\(W\_\{ki\}\)=q\(i\)\)\. That means

T^i\(N−1\)=\[\(W1​i\(N\),q⁡\(i\)\),\(W2​i\(N\),q⁡\(i\)\),…,\(WdN​i\(N\),q⁡\(i\)\)\]==\(\[W1​i\(N\),W\(N\),…,WdN​i\(N\)\],q⁡\(i\)\)\.\\begin\{split\}\\hat\{T\}^\{\(N\-1\)\}\_\{i\}&=\[\(W^\{\(N\)\}\_\{1i\},q\(i\)\),\(W^\{\(N\)\}\_\{2i\},q\(i\)\),\.\.\.,\(W^\{\(N\)\}\_\{d\_\{N\}i\},q\(i\)\)\]=\\\\ &=\(\[W^\{\(N\)\}\_\{1i\},W^\{\(N\)\},\.\.\.,W^\{\(N\)\}\_\{d\_\{N\}i\}\],q\(i\)\)\\,\\,\.\\end\{split\}Then, two nodes have isomorphic flavored output trees ifq⁡\(i\)=q⁡\(j\)q\(i\)=q\(j\)and∀k:Wk​i\(N\)=Wk​j\(N\)\\forall k:W^\{\(N\)\}\_\{ki\}=W^\{\(N\)\}\_\{kj\}\. Using this criterion, we can define flavored opfibers in layerℓ=N−1\\ell=N\-1\. For layerℓ=N−2\\ell=N\-2, the output treeT^i\(N−2\)\\hat\{T\}^\{\(N\-2\)\}\_\{i\}of nodeiican be expressed as the concatenation of the output treesT^m\(N−1\)\\hat\{T\}^\{\(N\-1\)\}\_\{m\}of nodesmmin layerℓ=N−1\\ell=N\-1, and their corresponding connection weightsWm​i\(N−1\)W^\{\(N\-1\)\}\_\{mi\}, i\.e\.,

T^i\(N−2\)=\(\[T^1\(N−1\),W1​i\(N−1\),T^2\(N−1\),W2​i\(N−1\),…,T^dN−1\(N−1\),WdN−1​i\(N−1\)\],q⁡\(i\)\)\.\\hat\{T\}^\{\(N\-2\)\}\_\{i\}=\(\[\\hat\{T\}^\{\(N\-1\)\}\_\{1\},W^\{\(N\-1\)\}\_\{1i\},\\hat\{T\}^\{\(N\-1\)\}\_\{2\},W^\{\(N\-1\)\}\_\{2i\},\.\.\.,\\hat\{T\}^\{\(N\-1\)\}\_\{d\_\{N\-1\}\},W^\{\(N\-1\)\}\_\{d\_\{N\-1\}i\}\],q\(i\)\)\.If the opfibersc^\\hat\{c\}in layerℓ=N−1\\ell=N\-1are already known,T^i\(N−2\)\\hat\{T\}^\{\(N\-2\)\}\_\{i\}can be more compactly represented as the concatenation of the output trees of the opfibersT^c^\(N−1\)\\hat\{T\}^\{\(N\-1\)\}\_\{\\hat\{c\}\}\(because they are all the same within an opfiber\), but we do have to aggregate the weights∑k∈c^Wk​i\\sum\_\{k\\in\\hat\{c\}\}W\_\{ki\}to replicate their summed effect from layerℓ=N−1\\ell=N\-1\. Then, two nodesiiandjjof the layerℓ=N−2\\ell=N\-2have isomorphic output trees iff∀c^:∑k∈c^Wk​j=∑k∈c^Wk​i\\forall\\hat\{c\}:\\sum\_\{k\\in\\hat\{c\}\}W\_\{kj\}=\\sum\_\{k\\in\\hat\{c\}\}W\_\{ki\}andq⁡\(i\)=q⁡\(j\)q\(i\)=q\(j\)to preserve flavors\. This argument can be extended to the next layersℓ=N−3,…,1\\ell=N\-3,\.\.\.,1\. This leads to the equal\-sum criterion in definition \([16](https://arxiv.org/html/2609.01768#S4.E16)\) of opfiber in weighted graphs along with the flavor\-preserving condition\.

To capture the coupling of forward pass and backpropagation of Eq\.[2](https://arxiv.org/html/2609.01768#S2.E2), we flavor the weights with the coloring of the fibration symmetries \(i\.e\., whenq⁡\(i\)=c⁡\(i\)∈𝒞fibq\(i\)=c\(i\)\\in\\mathcal\{C\}^\{\\rm fib\}\)\. This ensures that nodes in a flavored opfiber have the sameσ′\\sigma^\{\\prime\}, not just in the current layer but for the entire tree propagating the error from the output to each node of the network\. When the network is linear,q=1q=1for all connections and flavored opfibers simplify to regular opfibers, uncoupling forward from backward coloring\.

### 4\.6Proof of synchronization in Eqs\. \([15](https://arxiv.org/html/2609.01768#S4.E15)\) and \([17](https://arxiv.org/html/2609.01768#S4.E17)\)

We will prove both equations by induction\. Suppose that the synchronization of activity holds for the layerℓ−1\\ell\-1\. Leti,ji,jbe two nodes in layerℓ\\ellsuch asi​∼fib​ji\\underset\{\\rm fib\}\{\\sim\}j, then:

hi\(ℓ\)=σ⁡\(∑kWi​k\(ℓ\)​hk\(ℓ−1\)\)=σ⁡\(∑c∈𝒞ℓ−1fib∑k∈cWi​k\(ℓ\)​hk\(ℓ−1\)\)=σ\(∑c∈𝒞ℓ−1fibhc\(ℓ−1\)∑k∈cW\(ℓ\)i​k\)activity synchronization inℓ−1=σ\(∑c∈𝒞ℓ−1fibhc\(ℓ−1\)∑k∈cW\(ℓ\)j​k\)i∼fibj=hj\(ℓ\)\.\\begin\{split\}h\_\{i\}^\{\(\\ell\)\}&=\\sigma\\left\(\\sum\_\{k\}W^\{\(\\ell\)\}\_\{ik\}h\_\{k\}^\{\(\\ell\-1\)\}\\right\)\\\\ &=\\sigma\\left\(\\sum\_\{c\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}\}\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{ik\}h\_\{k\}^\{\(\\ell\-1\)\}\\right\)\\\\ &=\\sigma\\left\(\\sum\_\{c\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}\}h\_\{c\}^\{\(\\ell\-1\)\}\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{ik\}\\right\)\\qquad\\text\{activity synchronization in $\\ell\-1$\}\\\\ &=\\sigma\\left\(\\sum\_\{c\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}\}h\_\{c\}^\{\(\\ell\-1\)\}\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{jk\}\\right\)\\qquad i\\underset\{\\rm fib\}\{\\sim\}j\\\\ &=h\_\{j\}^\{\(\\ell\)\}\.\\end\{split\}
Therefore, the synchronization holds for the layerℓ\\ell\. Now, let us prove the base case of the induction \(ℓ=1\\ell=1\)\.

i​∼fib​j⟹Wi​k\(1\)=Wj​k\(1\)⟹σ⁡\(∑kWi​k\(1\)​xk\)=σ⁡\(∑kWj​k\(1\)​xk\)⟹hi\(1\)=hj\(1\)\.i\\underset\{\\rm fib\}\{\\sim\}j\\implies W^\{\(1\)\}\_\{ik\}=W^\{\(1\)\}\_\{jk\}\\implies\\sigma\\left\(\\sum\_\{k\}W^\{\(1\)\}\_\{ik\}x\_\{k\}\\right\)=\\sigma\\left\(\\sum\_\{k\}W^\{\(1\)\}\_\{jk\}x\_\{k\}\\right\)\\implies h\_\{i\}^\{\(1\)\}=h\_\{j\}^\{\(1\)\}\.
With these two steps, we have proven Eq\. \([15](https://arxiv.org/html/2609.01768#S4.E15)\)\. In Methods[4\.8](https://arxiv.org/html/2609.01768#S4.SS8), we show that this relationship is invertible in the case of linear activiation functions\. In that case, if we observe synchronization of two nodes for all inputs, then we can infer that they are in a fiber\.

Now, let’s prove Eq\. \([17](https://arxiv.org/html/2609.01768#S4.E17)\), which is based on the backpropagation rule–Eq\. \([2](https://arxiv.org/html/2609.01768#S2.E2)\)\. First, note that the non\-linear case of colored opfibration:i​∼op​j⟹i​∼fib​j⟹σi′=σj′i\\underset\{\\rm op\}\{\\sim\}j\\implies i\\underset\{\\rm fib\}\{\\sim\}j\\implies\\sigma^\{\\prime\}\_\{i\}=\\sigma^\{\\prime\}\_\{j\}\. In the linear case,∀i:σi′=1\\forall i:\\sigma^\{\\prime\}\_\{i\}=1, buti,ji,jare not necessarily in the same fiber\. Suppose that the error is synchronized in layerℓ\+1\\ell\+1\. Leti,ji,jbe two nodes in layerℓ\\ellsuch asi​∼op​ji\\underset\{\\rm op\}\{\\sim\}j\.

δi\(ℓ\)=σi′\(ℓ\)​∑kWk​i\(ℓ\+1\)​δk\(ℓ\+1\)=σi′\(ℓ\)​∑c∈𝒞ℓ\+1op∑k∈cWk​i\(ℓ\+1\)​δk\(ℓ\+1\)=σ′\(ℓ\)i∑c∈𝒞ℓ\+1opδc\(ℓ\+1\)∑k∈cW\(ℓ\+1\)k​ierror synchronization inℓ\+1=σ′\(ℓ\)j∑c∈𝒞ℓ\+1opδc\(ℓ\+1\)∑k∈cW\(ℓ\+1\)k​ji∼opj=δj\(ℓ\)\\begin\{split\}\\delta\_\{i\}^\{\(\\ell\)\}&=\\sigma^\{\\prime\(\\ell\)\}\_\{i\}\\sum\_\{k\}W^\{\(\\ell\+1\)\}\_\{ki\}\\delta\_\{k\}^\{\(\\ell\+1\)\}\\\\ &=\\sigma^\{\\prime\(\\ell\)\}\_\{i\}\\sum\_\{c\\in\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\}\\sum\_\{k\\in c\}W^\{\(\\ell\+1\)\}\_\{ki\}\\delta\_\{k\}^\{\(\\ell\+1\)\}\\\\ &=\\sigma^\{\\prime\(\\ell\)\}\_\{i\}\\sum\_\{c\\in\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\}\\delta\_\{c\}^\{\(\\ell\+1\)\}\\sum\_\{k\\in c\}W^\{\(\\ell\+1\)\}\_\{ki\}\\qquad\\text\{error synchronization in $\\ell\+1$\}\\\\ &=\\sigma^\{\\prime\(\\ell\)\}\_\{j\}\\sum\_\{c\\in\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\}\\delta\_\{c\}^\{\(\\ell\+1\)\}\\sum\_\{k\\in c\}W^\{\(\\ell\+1\)\}\_\{kj\}\\qquad i\\underset\{\\rm op\}\{\\sim\}j\\\\ &=\\delta\_\{j\}^\{\(\\ell\)\}\\end\{split\}
Then, the error synchronization holds for layerℓ\\ell\. Now, let’s prove the base case of the induction \(ℓ=N−1\\ell=N\-1\)\.

i​∼op​j⟹W\(N\)k​i=W\(N\)k​j&σ′\(N−1\)i=σ′\(N−1\)j⟹∑kW\(N\)k​iδk=∑kW\(N\)k​jδk&σ′\(N−1\)i=σ′\(N−1\)j⟹δi\(N−1\)=δj\(N−1\)\.\\begin\{split\}i\\underset\{\\rm op\}\{\\sim\}j&\\implies W^\{\(N\)\}\_\{ki\}=W^\{\(N\)\}\_\{kj\}\\quad\\&\\quad\\sigma^\{\\prime\(N\-1\)\}\_\{i\}=\\sigma^\{\\prime\(N\-1\)\}\_\{j\}\\\\ &\\implies\\sum\_\{k\}W^\{\(N\)\}\_\{ki\}\\delta\_\{k\}=\\sum\_\{k\}W^\{\(N\)\}\_\{kj\}\\delta\_\{k\}\\quad\\&\\quad\\sigma^\{\\prime\(N\-1\)\}\_\{i\}=\\sigma^\{\\prime\(N\-1\)\}\_\{j\}\\\\ &\\implies\\delta\_\{i\}^\{\(N\-1\)\}=\\delta\_\{j\}^\{\(N\-1\)\}\.\\end\{split\}
We have proven Eq\.[17](https://arxiv.org/html/2609.01768#S4.E17)\.

### 4\.7Quasi\-synchronization of activity and error in quasi\-symmetries

Although synchronization theorems are not valid for quasi\-symmetries, an upper bound for the variation of activity and error within quasi\-fibers and quasi\-opfibers can be found\. Ifσ\\sigmais a Lipschitz function with coefficientKσK\_\{\\sigma\}, two nodesi,ji,jin the same quasi\-fiber with thresholdεfib\\varepsilon\_\{\\rm fib\}, we obtain:

\|hi\(ℓ\)−hj\(ℓ\)\|\\displaystyle\|h^\{\(\\ell\)\}\_\{i\}\-h^\{\(\\ell\)\}\_\{j\}\|=\\displaystyle=\|σ⁡\(∑kWi​k\(ℓ\)​hk\(ℓ−1\)\)−σ⁡\(∑kWj​k\(ℓ\)​hk\(ℓ−1\)\)\|\\displaystyle\|\\sigma\\left\(\\sum\_\{k\}W^\{\(\\ell\)\}\_\{ik\}h\_\{k\}^\{\(\\ell\-1\)\}\\right\)\-\\sigma\\left\(\\sum\_\{k\}W^\{\(\\ell\)\}\_\{jk\}h\_\{k\}^\{\(\\ell\-1\)\}\\right\)\|≤\\displaystyle\\leqKσ​\|∑k\(Wi​k\(ℓ\)−Wj​k\(ℓ\)\)​hk\(ℓ−1\)\|\\displaystyle K\_\{\\sigma\}\|\\sum\_\{k\}\(W^\{\(\\ell\)\}\_\{ik\}\-W^\{\(\\ell\)\}\_\{jk\}\)h\_\{k\}^\{\(\\ell\-1\)\}\|≈\\displaystyle\\approxKσ​\|∑c,k∈c\(Wi​k\(ℓ\)−Wj​k\(ℓ\)\)​h^c\(ℓ−1\)\|for smallεfibfor layerℓ−1\\displaystyle K\_\{\\sigma\}\|\\sum\_\{c,k\\in c\}\(W^\{\(\\ell\)\}\_\{ik\}\-W^\{\(\\ell\)\}\_\{jk\}\)\\hat\{h\}\_\{c\}^\{\(\\ell\-1\)\}\|\\quad\\text\{for small $\\varepsilon\_\{\\rm fib\}$ for layer $\\ell\-1$\}≤\\displaystyle\\leqKσ​∑c\|h^c\(ℓ−1\)\|​\|∑k∈c\(Wi​k\(ℓ\)−Wj​k\(ℓ\)\)\|\\displaystyle K\_\{\\sigma\}\\sum\_\{c\}\|\\hat\{h\}\_\{c\}^\{\(\\ell\-1\)\}\|\|\\sum\_\{k\\in c\}\(W^\{\(\\ell\)\}\_\{ik\}\-W^\{\(\\ell\)\}\_\{jk\}\)\|≤\\displaystyle\\leqKσ​\|Cℓ−1fib\|​maxc​\|h^c\(ℓ−1\)\|​εfib\\displaystyle K\_\{\\sigma\}\|C^\{\\rm fib\}\_\{\\ell\-1\}\|\\max\_\{c\}\|\\hat\{h\}\_\{c\}^\{\(\\ell\-1\)\}\|\\varepsilon\_\{\\rm fib\}Here the approximation reflects the fact that in the input layerℓ−1\\ell\-1activations are approximately the mean value over the activations in the same quasi\-fibercc:h^c\(ℓ−1\)≈1\|c\|​∑k∈chk\(ℓ−1\)\\hat\{h\}\_\{c\}^\{\(\\ell\-1\)\}\\approx\\frac\{1\}\{\|c\|\}\\sum\_\{k\\in c\}h\_\{k\}^\{\(\\ell\-1\)\}, i\.e\. a quasi\-fiber condition on the preceding layer\.

For two nodesi,ji,jin the same quasi\-opfiber with thresholdεop\\varepsilon\_\{\\rm op\}, we obtain:

\|δi\(ℓ\)−δj\(ℓ\)\|\\displaystyle\|\\delta^\{\(\\ell\)\}\_\{i\}\-\\delta^\{\(\\ell\)\}\_\{j\}\|=\\displaystyle=\|σi′\(ℓ\)​\(∑kWk​i\(ℓ\)​δk\(ℓ\+1\)\)−σj′\(ℓ\)​\(∑kWk​j\(ℓ\)​δk\(ℓ\+1\)\)\|\\displaystyle\|\\sigma^\{\\prime\(\\ell\)\}\_\{i\}\\left\(\\sum\_\{k\}W^\{\(\\ell\)\}\_\{ki\}\\delta\_\{k\}^\{\(\\ell\+1\)\}\\right\)\-\\sigma^\{\\prime\(\\ell\)\}\_\{j\}\\left\(\\sum\_\{k\}W^\{\(\\ell\)\}\_\{kj\}\\delta\_\{k\}^\{\(\\ell\+1\)\}\\right\)\|≈\\displaystyle\\approx\|σ^c′\(ℓ\)\|​\|∑k\(Wk​i\(ℓ\)−Wk​j\(ℓ\)\)​δk\(ℓ\+1\)\|for smallεfibfor layerℓ\\displaystyle\|\\hat\{\\sigma\}^\{\\prime\(\\ell\)\}\_\{c\}\|\|\\sum\_\{k\}\\left\(W^\{\(\\ell\)\}\_\{ki\}\-W^\{\(\\ell\)\}\_\{kj\}\\right\)\\delta\_\{k\}^\{\(\\ell\+1\)\}\|\\quad\\text\{for small $\\varepsilon\_\{\\rm fib\}$ for layer $\\ell$\}≈\\displaystyle\\approx\|σ^c′\(ℓ\)​‖δ^c′\(ℓ\+1\)‖​∑c∑k∈c\(Wk​i\(ℓ\)−Wk​j\(ℓ\)\)\|for smallεopfor layerℓ\+1\\displaystyle\|\\hat\{\\sigma\}^\{\\prime\(\\ell\)\}\_\{c\}\|\|\\hat\{\\delta\}^\{\(\\ell\+1\)\}\_\{c^\{\\prime\}\}\|\|\\sum\_\{c\}\\sum\_\{k\\in c\}\\left\(W^\{\(\\ell\)\}\_\{ki\}\-W^\{\(\\ell\)\}\_\{kj\}\\right\)\|\\quad\\text\{for small $\\varepsilon\_\{\\rm op\}$ for layer $\\ell\+1$\}≤\\displaystyle\\leqKσ​\|Cℓ\+1op\|​maxc′​\|δ^c′\(ℓ\+1\)\|​εop\\displaystyle K\_\{\\sigma\}\|C^\{\\rm op\}\_\{\\ell\+1\}\|\\max\_\{c^\{\\prime\}\}\|\\hat\{\\delta\}^\{\(\\ell\+1\)\}\_\{c^\{\\prime\}\}\|\\varepsilon\_\{\\rm op\}Here the same approximation is used, replacing the error in layerℓ\+1\\ell\+1with their mean value in a quasi\-opfiber,δ^c\(ℓ\+1\)\\hat\{\\delta\}\_\{c\}^\{\(\\ell\+1\)\}, i\.e\. an quasi\-opfiber condition for the subsequent layer\. We also used the fact thatKσK\_\{\\sigma\}is an upper bound for\|σ′\|\|\\sigma^\{\\prime\}\|\.

### 4\.8Equivalence between fibration and synchronization for linear activation

Here we show that synchronization of activity for all inputs implies fibration symmetry, provided node activations are linear, i\.e\. two nodes that are synchronized are in the same fiber\. Because in the linear case, the error backpropagation is entirely analogous to the forward propagation of activity, we also conclude that error synchronization implies opfibration: two nodes with synchronized errors are in the same opfiber\. Therefore, structure not only defines function, but conversely, function defines structure\. This is an instance of the structure\-function relation, which here appears only in the linear case where the forward pass is decoupled from the backward pass\.

We will prove this by induction\. First, forℓ=1\\ell=1:

∀x:hi\(1\)=hj\(1\)⟹∀x:\[W\(1\)​x\]i=\[W\(1\)​x\]j⟹Wi​k\(1\)=Wj​k\(1\)⟹i​∼fib​j\.\\begin\{split\}\\forall x:h^\{\(1\)\}\_\{i\}=h^\{\(1\)\}\_\{j\}&\\implies\\forall x:\[W^\{\(1\)\}x\]\_\{i\}=\[W^\{\(1\)\}x\]\_\{j\}\\\\ &\\implies W^\{\(1\)\}\_\{ik\}=W^\{\(1\)\}\_\{jk\}\\\\ &\\implies i\\underset\{\\rm fib\}\{\\sim\}j\.\\end\{split\}
Now, suppose that this is valid forℓ−1\\ell\-1, then:

∀x:hi\(ℓ\)=hj\(ℓ\)⟹∀x:∑kWi​k\(ℓ\)​hk\(ℓ−1\)=∑kWj​k\(ℓ\)​hk\(ℓ−1\)⟹∀x:∑c∈𝒞ℓfibh^\(ℓ−1\)c∑k∈cWi​k\(ℓ\)=∑c∈𝒞ℓfibh^\(ℓ−1\)c∑k∈cW\(ℓ\)j​kHyp of induction⟹∑k∈cW\(ℓ\)i​k=∑k∈cW\(ℓ\)j​kbecause is valid for allh^c⟹i​∼fib​j\.\\begin\{split\}\\forall x:h^\{\(\\ell\)\}\_\{i\}=h^\{\(\\ell\)\}\_\{j\}&\\implies\\forall x:\\sum\_\{k\}W\_\{ik\}^\{\(\\ell\)\}h^\{\(\\ell\-1\)\}\_\{k\}=\\sum\_\{k\}W^\{\(\\ell\)\}\_\{jk\}h^\{\(\\ell\-1\)\}\_\{k\}\\\\ &\\implies\\forall x:\\sum\_\{c\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\}\}\\hat\{h\}^\{\(\\ell\-1\)\}\_\{c\}\\sum\_\{k\\in c\}W\_\{ik\}^\{\(\\ell\)\}=\\sum\_\{c\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\}\}\\hat\{h\}^\{\(\\ell\-1\)\}\_\{c\}\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{jk\}\\qquad\\text\{Hyp of induction\}\\\\ &\\implies\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{ik\}=\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{jk\}\\qquad\\text\{because is valid for all \}\\hat\{h\}\_\{c\}\\\\ &\\implies i\\underset\{\\rm fib\}\{\\sim\}j\.\\end\{split\}
For opfibration symmetry and error synchronization, we have an analogous proof\.

### 4\.9Flavored opfibration implies error synchronization

The backpropagation of errors in Eq\. \([2](https://arxiv.org/html/2609.01768#S2.E2)\) exhibits greater complexity due to the intrinsic coupling between error gradients and forward activations mediated by the slope of the nonlinear activationσ′\(ℓ\)\\sigma^\{\\prime\(\\ell\)\}\. Specifically, the errorδi\(ℓ\)\\delta\_\{i\}^\{\(\\ell\)\}for nodeiidepends not only on the error signals from the next layer \(mediated by the output weightsWi​k\(ℓ\)W^\{\(\\ell\)\}\_\{ik\}\) but also on the activation valuehi\(ℓ\)h\_\{i\}^\{\(\\ell\)\}at that node, which introduces a nonlinear coupling betweenδi\(ℓ\)\\delta\_\{i\}^\{\(\\ell\)\}andhi\(ℓ\)h\_\{i\}^\{\(\\ell\)\}, absent in forward propagation\.

This coupling can be systematically absorbed into an output tree that governs error propagation analogously to how the input tree governs activity propagation\. The key observation is that the slope — which is a function ofhi\(ℓ\)h\_\{i\}^\{\(\\ell\)\}— couples the in\-balanced color assignments from the forward fiber structure \(see Eq\. \([14](https://arxiv.org/html/2609.01768#S4.E14)\)\) to the connections that define the error flow in Eq\. \([2](https://arxiv.org/html/2609.01768#S2.E2)\)\.

We formalize this by introducingconnection flavorsinto the output tree, where the flavors are inherited from the node colors of the forward computation\. We take advantage of the concept offlavor\-preservingopfibration symmetry\[[47](https://arxiv.org/html/2609.01768#bib.bib47)\], which preserves flavor assignments of connections\. Therefore, two nodes belong to the same flavored opfiber if:

- •Flavor\-preserving opfibration symmetry: i∼opj⟹∀c^∈𝒞ℓ\+1op:∑k∈c^Wk​i\(ℓ\+1\)=∑k∈c^Wk​j\(ℓ\+1\)&q\(i\)=q\(j\)i\\underset\{\\rm op\}\{\\sim\}j\\implies\\quad\\forall\\hat\{c\}\\in\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}:\\quad\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{ki\}=\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{kj\}\\quad\\&\\quad q\(i\)=q\(j\)\(16\)

with flavors given by

q⁡\(i\)=\{1linear activationc⁡\(i\)∈𝒞ℓfibnon\-linear activationq\(i\)=\\left\\\{\\begin\{array\}\[\]\{ll\}1\\qquad\\qquad\\qquad\\text\{linear activation\}\\\\ c\(i\)\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\}\\qquad\\text\{non\-linear activation\}\\end\{array\}\\right\.
Equation \([16](https://arxiv.org/html/2609.01768#S4.E16)\) defines the opfiber partition𝒞ℓop\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\}\. In the linear activation case \(σ′\\sigma^\{\\prime\}= 1\), all weights have the same flavor \(q=1q=1\), and the forward and backward coloring decouple and can be treated independently\. In the non\-linear case, the flavors must first be calculated using the fiber coloring𝒞ℓfib\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\}computed forward through the entire network as above\. Then, the sum\-of\-weights condition of Eq\. \([16](https://arxiv.org/html/2609.01768#S4.E16)\) is evaluated for each flavor separately, recursively starting at the output of the network, with all nodes in different opfibers𝒞Nop=\{1,2,…,dN\}\\mathcal\{C\}^\{\\rm op\}\_\{N\}=\\\{1,2,\.\.\.,d\_\{N\}\\\}\.

Using this definition, we arrive at the following theorem\. The proof is given in Methods[4\.6](https://arxiv.org/html/2609.01768#S4.SS6)\.

- •Errors synchronization:Given two nodesiiandjjin layerℓ\\ell, the following hold in a network with inputsxxand targetsyy: i​∼op​j⟹δi\(ℓ\)=δj\(ℓ\)for any network inputx, targety\.i\\underset\{\\rm op\}\{\\sim\}j\\implies\\delta^\{\(\\ell\)\}\_\{i\}=\\delta^\{\(\\ell\)\}\_\{j\}\\qquad\\text\{for any network input $x$, target $y$\}\\,\.\(17\)

We note that for linear activations \(σ′\\sigma^\{\\prime\}= 1\), the relationship between synchronization and \(op\)fibration becomes bidirectional \(⇔\\iff\) in both Eqs\. \([15](https://arxiv.org/html/2609.01768#S4.E15)\) and \([17](https://arxiv.org/html/2609.01768#S4.E17), as we prove in Methods[4\.8](https://arxiv.org/html/2609.01768#S4.SS8)\. This means that synchronized nodes necessarily form \(op\)fibers, and conversely, nodes in a \(op\)fiber will synchronize\. This represents a strict one\-to\-one structure\-function equivalence that is difficult to demonstrate for more general non\-linear dynamical systems\[[52](https://arxiv.org/html/2609.01768#bib.bib52)\]\.

Alternative definitions for opfibration that capture the coupling with the forward pass are treated in Methods[4\.9\.1](https://arxiv.org/html/2609.01768#S4.SS9.SSS1)\. In the theory of gases, the linear case is analogous to non\-interacting particles in an ideal gas, whereas the nonlinear case corresponds to particle interactions in a real gas, and flavor plays the role of a compression factor\.

#### 4\.9\.1Alternative definitions to capture forward\-backward coupling in nonlinear Feedforward Networks

There are a few alternatives to our approach of using flavor\-preserving opfibrations\. The proof of the synchrony theorems and stability rules can be rewritten for these alternative definitions of opfibers and covers, with minor modifications\. Instead of requiring flavors, one can directly require that the sum equality hold, including the slopes for all inputsxx

- •Option 1: Hybrid opfiber definition i​∼op​j⇔∀x,∀c^∈𝒞ℓ\+1op:σi′\(ℓ\)​∑k∈c^Wk​i\(ℓ\+1\)=σj′\(ℓ\)​∑k∈c^Wk​j\(ℓ\+1\)\.i\\underset\{\\rm op\}\{\\sim\}j\\iff\\forall x,\\forall\\hat\{c\}\\in\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}:\\quad\\sigma^\{\\prime\(\\ell\)\}\_\{i\}\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{ki\}=\\sigma^\{\\prime\(\\ell\)\}\_\{j\}\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{kj\}\\,\.

These conditions are satisfied if the sum of the weight conditions is met and the slopes are synchronized between all possible inputsxx\. In the case of linear activationsσ′=1\\sigma^\{\\prime\}=1, the definition simplifies to a sum constraint on the output weight alone\.

The proofs of the stability rules and the theorem actually only require error synchronization, so we can alternatively make that explicit in the definition of a cover:

- •Option 2: Hybrid cover definition i​∼cov​j⇔i​∼fib​j&∀x:δi\(ℓ\)=δj\(ℓ\),i\\underset\{\\rm cov\}\{\\sim\}j\\iff i\\underset\{\\rm fib\}\{\\sim\}j\\quad\\&\\quad\\forall x:\\delta^\{\(\\ell\)\}\_\{i\}=\\delta^\{\(\\ell\)\}\_\{j\}\\,,

and require a covering symmetry, where we currently require a flavor\-preserving opfibration\. This alternative formulation of the stability rules and the cover coarsening theorem holds for all cover definitions in this work\. Indeed, note that our definition of a flavored opfibration implies a fiber in the current layer\. Therefore, it automatically satisfies the cover condition[18](https://arxiv.org/html/2609.01768#S4.E18)\. In other words, the flavor\-preserving opfibration as we have defined it happens to be a cover\. In the linear case, the error synchronization condition can be relaxed to the \(unflavored\) opfibration condition\.

The two alternatives above swap the structural condition based on the weights, with the explicit but weaker requirement of activity/error synchronization \(hence the label “Hybrid”\)\. Indeed, the proofs can be derived entirely from activity and error synchronization\. However, we can also use a purely structural definition based solely on weights, without requiring flavored trees or synchronization, as follows\.

- •Option 3: Recursive cover definition i∼covj⇔i∼fibj&∀c^∈𝒞covℓ\+1:∑k∈c^W\(ℓ\+1\)k​i=∑k∈c^W\(ℓ\+1\)k​j\.i\\underset\{\\rm cov\}\{\\sim\}j\\iff\\quad i\\underset\{\\rm fib\}\{\\sim\}j\\quad\\&\\quad\\forall\\hat\{c\}\\in\\mathcal\{C\}^\{\\rm cov\}\_\{\\ell\+1\}:\\quad\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{ki\}=\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{kj\}\\,\.

This last definition leverages the knowledge that the structural requirement of fibers ensures activity synchronization and, therefore, slope synchronizationσ′\(ℓ\)\\sigma^\{\\prime\(\\ell\)\}\. It essentially couples the definition of an opfiber with the definition of the cover, which is decoupled in the linear case\.

The first two alternatives above do not have an obvious coloring algorithm based solely on weights, since the definitions depend on the activity\. The last definition has the same coloring algorithm as for the flavored opfibration: One has to perform a forward coloring based on fibers, and use this to establish the cover coloring starting at the output, where in each layer, the cover is an intersection of the fibers partition with the partition due to the weight constraints\.

#### 4\.9\.2Gradient Descent Rule \- Eq\. \([4](https://arxiv.org/html/2609.01768#S2.E4)\)

For a dataset𝒟=\{\(x,y\)\}\\mathcal\{D\}=\\\{\(x,y\)\\\}, we denote the network’s input ash\(0\)=xh^\{\(0\)\}=xor𝐇\(0\)=x\\mathbf\{H\}^\{\(0\)\}=x\. The output of the network is a predictiony^\\hat\{y\}to the desiredyy\. The error of this prediction isL=ℒ⁡\(y^,y\)L=\\mathcal\{L\}\(\\hat\{y\},y\), whereℒ\\mathcal\{L\}is calledloss function\. The weightsW\(ℓ\)W^\{\(\\ell\)\}and𝐖\(ℓ\)\\mathbf\{W\}^\{\(\\ell\)\}can then be adjusted based on corrections that minimize the errorLL\. Using gradient descent, the change in the weight matrix/tensor is

W\(ℓ\)​\(t\)=W\(ℓ\)​\(t−1\)−α​∂L∂W\(ℓ\)​\(t−1\)W^\{\(\\ell\)\}\(t\)=W^\{\(\\ell\)\}\(t\-1\)\-\\alpha\\frac\{\\partial L\}\{\\partial W^\{\(\\ell\)\}\}\(t\-1\)whereα\\alphais called the learning rate\. Calculating the derivative we find:

MLP:\\displaystyle\\text\{MLP\}:∂L∂W\(ℓ\)=δ\(ℓ\)⋅\(h\(ℓ−1\)\)T,\\displaystyle\\frac\{\\partial L\}\{\\partial W^\{\(\\ell\)\}\}=\\delta^\{\(\\ell\)\}\\cdot\\left\(h^\{\(\\ell\-1\)\}\\right\)^\{T\},CNN:\\displaystyle\\text\{CNN\}:∂L∂𝐖\(ℓ\)=δ\(ℓ\)∗𝐇\(ℓ−1\)\\displaystyle\\frac\{\\partial L\}\{\\partial\\mathbf\{W\}^\{\(\\ell\)\}\}=\\delta^\{\(\\ell\)\}\\ast\\mathbf\{H\}^\{\(\\ell\-1\)\}whereδℓ\\delta^\{\\ell\}is the error signal for the nodes in layerℓ\\ell\.

#### 4\.9\.3Proof of Synchronized Learning \- Eq\. \([5](https://arxiv.org/html/2609.01768#S2.E5)\)

Consider two nodesiiandjjin the layerℓ−1\\ell\-1in the same fiber and two nodesmmandnnin the layerℓ\\ellin the same opfiber \(see red and green nodes in Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)a\)\. Using Eqs\. \([15](https://arxiv.org/html/2609.01768#S4.E15)\)\-\([17](https://arxiv.org/html/2609.01768#S4.E17)\), we obtainhi\(ℓ−1\)=hj\(ℓ−1\)h^\{\(\\ell\-1\)\}\_\{i\}=h^\{\(\\ell\-1\)\}\_\{j\}andδm\(ℓ\)=δn\(ℓ\)\\delta^\{\(\\ell\)\}\_\{m\}=\\delta^\{\(\\ell\)\}\_\{n\}\.

Then,

Δ​Wm​i\(ℓ\)=−α​∂L∂Wm​i\(ℓ\)=−α​δm\(ℓ\)​hi\(ℓ−1\)=−α​δm\(ℓ\)​hj\(ℓ−1\)=Δ​Wm​j\(ℓ\)=−α​δn\(ℓ\)​hi\(ℓ−1\)=Δ​Wn​i\(ℓ\)=−α​δn\(ℓ\)​hj\(ℓ−1\)=Δ​Wn​j\(ℓ\)\\begin\{split\}\\Delta W^\{\(\\ell\)\}\_\{mi\}=\-\\alpha\\frac\{\\partial L\}\{\\partial W\_\{mi\}^\{\(\\ell\)\}\}&=\-\\alpha\\delta^\{\(\\ell\)\}\_\{m\}h^\{\(\\ell\-1\)\}\_\{i\}\\\\ &=\-\\alpha\\delta^\{\(\\ell\)\}\_\{m\}h^\{\(\\ell\-1\)\}\_\{j\}=\\Delta W^\{\(\\ell\)\}\_\{mj\}\\\\ &=\-\\alpha\\delta^\{\(\\ell\)\}\_\{n\}h^\{\(\\ell\-1\)\}\_\{i\}=\\Delta W^\{\(\\ell\)\}\_\{ni\}\\\\ &=\-\\alpha\\delta^\{\(\\ell\)\}\_\{n\}h^\{\(\\ell\-1\)\}\_\{j\}=\\Delta W^\{\(\\ell\)\}\_\{nj\}\\end\{split\}

### 4\.10Covering symmetries emerge during stochastic gradient descent

The final symmetry that we introduce in DNNs is the covering symmetry\. Two nodes belong to the same cover iff they are in the same fiber and the same flavored opfiber:

- •Covering symmetry: i​∼cov​j⇔i​∼fib​j&i​∼op​j\.i\\underset\{\\rm cov\}\{\\sim\}j\\iff i\\underset\{\\rm fib\}\{\\sim\}j\\quad\\&\\quad i\\underset\{\\rm op\}\{\\sim\}j\\,\.\(18\)

This condition defines a covering partition of the network𝒞ℓcov\\mathcal\{C\}^\{\\rm cov\}\_\{\\ell\}, which is preserved under GD\. Specifically, as a consequence ofsynchronized learning, once two nodes are in a single covering at learning time steptt, the weight update with the gradient descent algorithm keeps the nodes in the covering at timet\+1t\+1\. This is formalized in theCover Coarse\-Graining Theorem[1](https://arxiv.org/html/2609.01768#Thmtheorem1)\(proof in Methods[4\.11](https://arxiv.org/html/2609.01768#S4.SS11)\) \. We proof this theoretical result in Methods[4\.11](https://arxiv.org/html/2609.01768#S4.SS11)\. The theorem implies that once multiple nodes are in the same cover, they cannot exit that cover, and thus constitute an invariant set\. However, distinct covers can merge to form larger \(coarser\) covers\. The proof of this Cover Coarse\-Graining Theorem is based on stability rules \(see Methods[4\.11](https://arxiv.org/html/2609.01768#S4.SS11)\)\. These provide an intuition for how fibration and opfibration symmetries develop during learning\. The stability rules state that the fibers in the layerℓ\\ellpreserve the fibers in the layerℓ\+1\\ell\+1, while the opfibers preserve the opfibers in the layerℓ−1\\ell\-1, as shown in Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)b\. Thus, opfiber formation tends to propagate backward from the output layer, while fiber formation tends to propagate forward from the input as learning progresses\. Note that for the non\-linear case, the opfibers have a flavoring of the connections to capture their dependence on the forward pass\. This introduces an asymmetry in the stability rules, reflected in Fig[2](https://arxiv.org/html/2609.01768#S2.F2)b, with connection colors defined at the output, but not at the input\.

### 4\.11Proof of Theorem[1](https://arxiv.org/html/2609.01768#Thmtheorem1): Cover Coarse\-Graining Theorem

First we will proof stability rules for fibers and opfibers, which are a direct consequence of synchronized learning under gradient descent\. Then we use these rules for a recursive proof of the Cover Coarse\-Graining Theorem\. Then we briefly discuss stability of covers under stochastic gradient descent\.

#### 4\.11\.1Opfiber, Fiber and Cover Stability Rules

Here we will proof three stability rules used to proof the cover coarsening theorem\. The opfiber stability rule states that nodes in the same cover \(blue nodes in Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)b\) at learning timettwill belong to the same opfiber after one GD learning stept\+1t\+1, if their output nodes are on the same opfiber\. The fiber stability rule states that nodes in the same cover at timettwill belong to the same fiber the next timet\+1t\+1, if their input nodes are on the same fiber\. Both rules are combined in the cover stability rule, which states that nodes in the same covering at timettremain in the same covering in the subsequent timet\+1t\+1\.

During the training process, the network weightsWW, the symmetries, and partitions of the nodes𝒞\\mathcal\{C\}will depend on the timett\.

During gradient descent training of an FNN,learning synchronizationensures that:

1. 1\.Nodes in the same coveri​∼c​o​v,ℓ,t​ji\\underset\{cov,\\ell,t\}\{\\sim\}jwill belong to the same opfiber at the next learning times stepi​∼o​p,ℓ,t\+1​ji\\underset\{op,\\ell,t\+1\}\{\\sim\}jif𝒞ℓ\+1op​\(t\+1\)\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\(t\+1\)is coarser than𝒞ℓ\+1op​\(t\)\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\(t\)\.
2. 2\.Nodes in the same coveri​∼c​o​v,ℓ,t​ji\\underset\{cov,\\ell,t\}\{\\sim\}jwill belong to the same fiber at the next learning time stepi​∼f​i​b,ℓ,t\+1​ji\\underset\{fib,\\ell,t\+1\}\{\\sim\}jif𝒞ℓ−1fib​\(t\+1\)\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}\(t\+1\)is coarser than𝒞ℓ−1fib​\(t\)\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}\(t\)\.
3. 3\.Nodes in the same coveri​∼c​o​v,ℓ,t​ji\\underset\{cov,\\ell,t\}\{\\sim\}jwill belong to the same cover at the next learning time stepi​∼c​o​v,ℓ,t\+1​ji\\underset\{cov,\\ell,t\+1\}\{\\sim\}jif𝒞ℓ−1fib​\(t\+1\)\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}\(t\+1\)is coarser than𝒞ℓ−1fib​\(t\)\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}\(t\)and𝒞ℓ\+1op​\(t\+1\)\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\(t\+1\)is coarser than𝒞ℓ\+1op​\(t\)\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\(t\)\.

Proof\(1\) Leti​∼c​o​v,ℓ,t​ji\\underset\{cov,\\ell,t\}\{\\sim\}jbe nodes in layerℓ\\ell\. For learning time stept\+1t\+1, for∀c^∈𝒞ℓ\+1op​\(t\+1\)\\forall\\hat\{c\}\\in\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\(t\+1\):

∑k∈c^Wk​i\(ℓ\+1\)​\(t\+1\)=∑k∈c^Wk​i\(ℓ\+1\)​\(t\)\+Δ​Wk​i\(ℓ\+1\)​\(t\)=∑k∈c^Wk​i\(ℓ\+1\)​\(t\)\+Δ​Wk​j\(ℓ\+1\)​\(t\)\.\\begin\{split\}\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{ki\}\(t\+1\)&=\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{ki\}\(t\)\+\\Delta W^\{\(\\ell\+1\)\}\_\{ki\}\(t\)\\\\ &=\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{ki\}\(t\)\+\\Delta W^\{\(\\ell\+1\)\}\_\{kj\}\(t\)\.\\end\{split\}
The last equality is valid due toi​∼f​i​b,ℓ,t​ji\\underset\{fib,\\ell,t\}\{\\sim\}jand activity synchronization\. Now, note that

∑k∈c^Wk​i\(ℓ\+1\)​\(t\)=∑r∈𝒞ℓ\+1op​\(t\)∑k∈c^∩rWk​i\(ℓ\+1\)​\(t\)=∑r∈𝒞ℓ\+1op​\(t\)Θ\(r⊂c^\)∑k∈rW\(ℓ\+1\)k​i\(t\)Hyp\)𝒞ℓ\+1op​\(t\+1\)is coarser than𝒞ℓ\+1op​\(t\)=∑r∈𝒞ℓ\+1op​\(t\)Θ\(r⊂c^\)∑k∈rW\(ℓ\+1\)k​j\(t\)Hyp\)i​∼o​p,ℓ,t​j=∑k∈c^Wk​j\(ℓ\+1\)​\(t\)\.\\begin\{split\}\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{ki\}\(t\)&=\\sum\_\{r\\in\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\(t\)\}\\sum\_\{k\\in\\hat\{c\}\\cap r\}W^\{\(\\ell\+1\)\}\_\{ki\}\(t\)\\\\ &=\\sum\_\{r\\in\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\(t\)\}\\Theta\(r\\subset\\hat\{c\}\)\\sum\_\{k\\in r\}W^\{\(\\ell\+1\)\}\_\{ki\}\(t\)\\qquad\\textit\{Hyp\) $\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\(t\+1\)$ is coarser than $\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\(t\)$\}\\\\ &=\\sum\_\{r\\in\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\+1\}\(t\)\}\\Theta\(r\\subset\\hat\{c\}\)\\sum\_\{k\\in r\}W^\{\(\\ell\+1\)\}\_\{kj\}\(t\)\\qquad\\textit\{Hyp\) $i\\underset\{op,\\ell,t\}\{\\sim\}j$\}\\\\ &=\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{kj\}\(t\)\.\\end\{split\}
Then, we obtain∑k∈c^Wk​i\(ℓ\+1\)​\(t\+1\)=∑k∈c^Wk​j\(ℓ\+1\)​\(t\+1\)\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{ki\}\(t\+1\)=\\sum\_\{k\\in\\hat\{c\}\}W^\{\(\\ell\+1\)\}\_\{kj\}\(t\+1\)\. That meansi​∼o​p,ℓ,t\+1​ji\\underset\{op,\\ell,t\+1\}\{\\sim\}j\.

The proof of \(2\) is similar\. The proof of \(3\) is the consequence of \(1\) and \(2\)\.

#### 4\.11\.2Recursive proof of cover coarsening theorem

The cover coarsening theorem indicates nodes in the same coveri​∼c​o​v,ℓ,t​ji\\underset\{cov,\\ell,t\}\{\\sim\}jwill belong to the same cover at the next timei​∼c​o​v,ℓ,t\+1​ji\\underset\{cov,\\ell,t\+1\}\{\\sim\}j\. The theorem relaxes the conditions of the cover stability rule\.

Forℓ=0\\ell=0andℓ=N\\ell=N,𝒞0fib​\(t\)=\{\[1\],\[2\],…,\[d0\]\}\\mathcal\{C\}^\{\\rm fib\}\_\{0\}\(t\)=\\\{\[1\],\[2\],\.\.\.,\[d\_\{0\}\]\\\}, and𝒞Nop​\(t\)=\{\[1\],\[2\],…,\[dN\]\}\\mathcal\{C\}^\{\\rm op\}\_\{N\}\(t\)=\\\{\[1\],\[2\],\.\.\.,\[d\_\{N\}\]\\\}that remain constant over timett\.

We apply fiber stability rule for layerℓ=1\\ell=1\. Nodes in the same coveri​∼c​o​v,ℓ=1,t​ji\\underset\{cov,\\ell=1,t\}\{\\sim\}jwill belong to the same fiber at the next timei​∼f​i​b,ℓ=1,t\+1​ji\\underset\{fib,\\ell=1,t\+1\}\{\\sim\}jbecause𝒞0fib​\(t\+1\)\\mathcal\{C\}^\{\\rm fib\}\_\{0\}\(t\+1\)is coarser than𝒞0fib​\(t\)\\mathcal\{C\}^\{\\rm fib\}\_\{0\}\(t\)\. This means𝒞1fib​\(t\+1\)\\mathcal\{C\}^\{\\rm fib\}\_\{1\}\(t\+1\)is coarser than𝒞1fib​\(t\)\\mathcal\{C\}^\{\\rm fib\}\_\{1\}\(t\)\.

With the same argument, we apply the fiber stability rule for layerℓ=2\\ell=2, and we obtain𝒞2fib​\(t\+1\)\\mathcal\{C\}^\{\\rm fib\}\_\{2\}\(t\+1\)is coarser than𝒞2fib​\(t\)\\mathcal\{C\}^\{\\rm fib\}\_\{2\}\(t\)\.

By iterating over all layers, we obtain∀ℓ:i​∼c​o​v,ℓ,t​j⟹i​∼f​i​b,ℓ,t\+1​j\\forall\\ell:i\\underset\{cov,\\ell,t\}\{\\sim\}j\\implies i\\underset\{fib,\\ell,t\+1\}\{\\sim\}j\.

Applying the same reasoning with the opfiber stability rule from layerℓ=N\\ell=N, then∀ℓ:i​∼c​o​v,ℓ,t​j⟹i​∼o​p,ℓ,t\+1​j\\forall\\ell:i\\underset\{cov,\\ell,t\}\{\\sim\}j\\implies i\\underset\{op,\\ell,t\+1\}\{\\sim\}j\.

Using both results, we obtain:∀ℓ:i​∼c​o​v,ℓ,t​j⟹i​∼c​o​v,ℓ,t\+1​j\\forall\\ell:i\\underset\{cov,\\ell,t\}\{\\sim\}j\\implies i\\underset\{cov,\\ell,t\+1\}\{\\sim\}j\.

The nodes in the same cover at learning time steptt, will belong to the same cover in the next time step\. Equivalently,𝒞ℓcov​\(t\+1\)\\mathcal\{C\}^\{\\rm cov\}\_\{\\ell\}\(t\+1\)is coarser than𝒞ℓcov​\(t\)\\mathcal\{C\}^\{\\rm cov\}\_\{\\ell\}\(t\)\.

#### 4\.11\.3Stability of covers

Theorem[1](https://arxiv.org/html/2609.01768#Thmtheorem1)can be interpreted as follows:If network parametersθt\\theta\_\{t\}at timettsatisfyi​∼cov​ji\\underset\{\\rm cov\}\{\\sim\}j; theni​∼cov​ji\\underset\{\\rm cov\}\{\\sim\}jstill holds forθt\+1\\theta\_\{t\+1\}\. That is, the parameter set satisfyingi​∼cov​ji\\underset\{\\rm cov\}\{\\sim\}jis invariant under the weight update in GD\.

Chenet al\.\[[32](https://arxiv.org/html/2609.01768#bib.bib32)\]have shown that any invariant parameter set of GD is a stable attractor in the SGD algorithm, provided that there is a sufficiently large learning constant\. The intuition is that stochastic fluctuations transverse to a cover are damped, drawing the parameters toward the symmetric set, while fluctuations along it vanish due to synchronized learning\. As a result, the cover theorem implies that the covering symmetries are stable attractors of SGD\.

### 4\.12Fibers coloring for sums, convolutions, gates and concatenations

We have discussed fibers for the canonical MLP architecture\. here we explain how they apply to the most common architectural motives found in modern deep networks\. Many of these, such as sums and gating, can be understood as hypergraphs, and therefore we are defining fibration rules here for hypergraphs\.

- •Sums:The most ubiquitous motive are nodes that are added together\. They can be a simple as the sum of a bias term,𝐖𝐡\+𝐛\{\\bf W\}\{\\bf h\}\+\{\\bf b\}or the sum of two sets of inputs with their corresponding weights,𝐖h​𝐡\+𝐖x​𝐱\{\\bf W\}\_\{h\}\{\\bf h\}\+\{\\bf W\}\_\{x\}\{\\bf x\}, or residual connections,𝐱\+𝐖h​𝐡\{\\bf x\}\+\{\\bf W\}\_\{h\}\{\\bf h\}\. They can be simply treated by concatenating weights and inputs\. For example:𝐖h​𝐡\+𝐖x​𝐱=\[𝐖h\|𝐖x\]​\[𝐡T\|𝐱T\]T=𝐖^​𝐡^\{\\bf W\}\_\{h\}\{\\bf h\}\+\{\\bf W\}\_\{x\}\{\\bf x\}=\[\{\\bf W\}\_\{h\}\|\{\\bf W\}\_\{x\}\]\[\{\\bf h\}^\{T\}\|\{\\bf x\}^\{T\}\]^\{T\}=\\widehat\{\\bf W\}\\widehat\{\\bf h\}\. The fiber equations are then simply applied to𝐖^\\widehat\{\\bf W\}, and the nodes in inputs𝐱\{\\bf x\}and𝐡\{\\bf h\}contribute their own colors\. In the case of residual connections, an identity matrix is used when concatenating weights\. The orthogonality of this identity makes the formation of fibers unlikely unless they were already in the input𝐱\{\\bf x\}\. In other words, residual connections will tend to break any fibers that would otherwise have formed in a layer\.
- •Convolutions:Most prominent in CNNs, convolutions,𝐖∗𝐇\\mathbf\{W\}\*\\mathbf\{H\}, where a matrix of ”kernels” of filters𝐖\\mathbf\{W\}mixes feature channels𝐇\\mathbf\{H\}, while each kernel mixes time samples, pixels, or tokens, depending on the application\. We can simply ignore the time/pixel/token dimensions and treat each filter as a scalar weights\. For instance, the fibration condition is simply: ∑k∈c𝐖^\[:,:,i,k\]=∑k∈c𝐖\(ℓ\)\[:,:,j,k\]\.\\sum\_\{k\\in c\}\\hat\{\\mathbf\{W\}\}\[:,:,i,k\]=\\sum\_\{k\\in c\}\\mathbf\{W\}^\{\(\\ell\)\}\[:,:,j,k\]\. See example in Fig\.[6](https://arxiv.org/html/2609.01768#S4.F6)a\. The usual pooling operates only along the time/pixel/toke dimensions and does not affect colors only given to feature channels\.
- •Concatenations:When two vectors of nodes are concatenated, then each contributes their own color, similar to the way sums where handled above\.
- •Gates:In RNNs such as the LSTM it is common to use gates of the form,𝐟⊙𝐱\{\\bf f\}\\odot\{\\bf x\}, where in this case an input vector𝐱\{\\bf x\}is multiplied element\-wise with a forget gate𝐟\{\\bf f\}\. This represents a multiplicative two\-node interaction\. For a fiber to emerge in this product, two nodes must be a fiber in the corresponding nodes of𝐱\{\\bf x\}andin a fiber of nodes of𝐟\{\\bf f\}, as exemplified with the colors in Fig\.[6](https://arxiv.org/html/2609.01768#S4.F6)b\. Two nodes in the product have the same colors, if they had the same colors in both factors — this color combination rule is the sameandoperation as we used for covers, but here it applies to the factors of the product\. A similar element\-wise product occurs in transformers when the key and query vectors are multiplied,𝐤⊙𝐪\{\\bf k\}\\odot\{\\bf q\}\. The coloring can be treated the same as for gates, but the two structures differ for the fibration compression rule as shown in Fig\.[6](https://arxiv.org/html/2609.01768#S4.F6)c and explained in Methods[4\.13\.3](https://arxiv.org/html/2609.01768#S4.SS13.SSS3)\. As with convolutional networks, the toke dimensions are not colored, and thus the attention matrix itself, square in the number of tokens, does not affect colors\.

Identifying symmetries in the time/pixel/token dimensions will be left to future work\.

![Refer to caption](https://arxiv.org/html/2609.01768v1/Fig_Compression.png)Figure 6:\(a\)Example of fibration compression of a convolution layer\. It reduces the number of channels \(3 to 2 and 5 to 3\)\.\(b\)Example of fibration compression of gates in LSTM\. The base has an identical number of nodes across all gates \(here 2 nodes per gate\)\.\(c\)The fibration compression for attention modules preserves the count of interactions \(3 multiplications\) even when the number of fibers of thekkandqqdiffer\.
### 4\.13Fibration compression rules

Now we want to derive the fibration compression rule for the parameters of a base networkBfibB\_\{\\rm fib\}, so that the activity of the nodes remains unchanged, i\.e\., the base network maintains forward computation of the original networkGG\. We will do this first for the MLP, and then discuss compression for other architectures\.

#### 4\.13\.1Fibration compression for a feedforward network

Leth^c′\\hat\{h\}\_\{c^\{\\prime\}\}be the activity of a fiberc′∈𝒞ℓfibc^\{\\prime\}\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\}, then it follows that:

h^c′\(ℓ\)=h\(ℓ\)i∀i∈c′\(activity synchronization\)=1\|c′\|∑i∈c′h\(ℓ\)ionly≈forεfib\>0=1\|c′\|​∑i∈c′\[Wi​k\(ℓ\)​hk\(ℓ\)\+bi\(ℓ\)\]=1\|c′\|​∑i∈c′∑c∈𝒞ℓ−1fib∑k∈cWi​k\(ℓ\)​hk\(ℓ\)\+1\|c′\|​∑i∈c′bi\(ℓ\)=∑c∈𝒞ℓ−1fib\[1\|c′\|​∑i∈c′∑k∈cWi​k\(ℓ\)\]​h^c\(ℓ\)\+\[1\|c′\|​∑i∈c′bi\(ℓ\)\]=∑c∈𝒞ℓ−1fibW^c′​c​h^c\(ℓ\)\+b^c\(ℓ\),\\begin\{split\}\\hat\{h\}^\{\(\\ell\)\}\_\{c^\{\\prime\}\}&=h^\{\(\\ell\)\}\_\{i\}\\qquad\\forall i\\in c^\{\\prime\}\\qquad\\text\{\(activity synchronization\)\}\\\\ &=\\frac\{1\}\{\|c^\{\\prime\}\|\}\\sum\_\{i\\in c^\{\\prime\}\}h^\{\(\\ell\)\}\_\{i\}\\qquad\\text\{only $\\approx$ for $\\varepsilon\_\{\\rm fib\}\>0$\}\\\\ &=\\frac\{1\}\{\|c^\{\\prime\}\|\}\\sum\_\{i\\in c^\{\\prime\}\}\\left\[W^\{\(\\ell\)\}\_\{ik\}h^\{\(\\ell\)\}\_\{k\}\+b^\{\(\\ell\)\}\_\{i\}\\right\]\\\\ &=\\frac\{1\}\{\|c^\{\\prime\}\|\}\\sum\_\{i\\in c^\{\\prime\}\}\\sum\_\{c\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}\}\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{ik\}h^\{\(\\ell\)\}\_\{k\}\+\\frac\{1\}\{\|c^\{\\prime\}\|\}\\sum\_\{i\\in c^\{\\prime\}\}b^\{\(\\ell\)\}\_\{i\}\\\\ &=\\sum\_\{c\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}\}\\left\[\\frac\{1\}\{\|c^\{\\prime\}\|\}\\sum\_\{i\\in c^\{\\prime\}\}\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{ik\}\\right\]\\hat\{h\}^\{\(\\ell\)\}\_\{c\}\+\\left\[\\frac\{1\}\{\|c^\{\\prime\}\|\}\\sum\_\{i\\in c^\{\\prime\}\}b^\{\(\\ell\)\}\_\{i\}\\right\]\\\\ &=\\sum\_\{c\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\-1\}\}\\hat\{W\}\_\{c^\{\\prime\}c\}\\hat\{h\}^\{\(\\ell\)\}\_\{c\}\+\\hat\{b\}^\{\(\\ell\)\}\_\{c\}\\,,\\\\ \\end\{split\}\(19\)whereW^c′​c\(ℓ\)=1\|c′\|​∑i∈c′∑k∈cWi​k\(ℓ\)\\hat\{W\}\_\{c^\{\\prime\}c\}^\{\(\\ell\)\}=\\frac\{1\}\{\|c^\{\\prime\}\|\}\\sum\_\{i\\in c^\{\\prime\}\}\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{ik\}andb^c′\(ℓ\)=1\|c′\|​∑i∈c′bi\(ℓ\)\\hat\{b\}^\{\(\\ell\)\}\_\{c^\{\\prime\}\}=\\frac\{1\}\{\|c^\{\\prime\}\|\}\\sum\_\{i\\in c^\{\\prime\}\}b^\{\(\\ell\)\}\_\{i\}\. These matricesW^c′​c\\hat\{W\}\_\{c^\{\\prime\}c\}and vectorsb^c′\(ℓ\)\\hat\{b\}^\{\(\\ell\)\}\_\{c^\{\\prime\}\}are the parameters of the baseBfibB\_\{\\rm fib\}\. They are obtained by adding the weights with the same input colors and averaging the weights with the same output colors \(Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)c\)

A similar equation can be derived for a base networkBopB\_\{\\rm op\}whose nodes are the opfibers of the original networkGGand the backward computation of the error is preserved\.

Then compression to a covering base can be chosen to preserve the input or output trees\. This is represented pictorially for the same covering base in a network with binary weights in Fig[4](https://arxiv.org/html/2609.01768#S4.F4)\. Compressing a graph into its covering base where both forward and backward computations are preserved is only possible when all covers have the same cardinality\|c′\|\|c^\{\\prime\}\|\. This is too strong of a requirement and has not been explored in more detail here\.

#### 4\.13\.2Fibration compression for a convolutional layer in an CNN

A similar line of reasoning works for CNNs where the nodes of the computational graph are the channels in the convolutions\. The definition of fibers and lifting for MLPs \(Eqs\.[14](https://arxiv.org/html/2609.01768#S4.E14),[6](https://arxiv.org/html/2609.01768#S2.E6)\) can be generalized to CNNs\. These equations are applied to the channelsi,ki,kin a layerℓ\\ell; whereWi​k\(ℓ\)W^\{\(\\ell\)\}\_\{ik\}is aD2D^\{2\}\-dimensional vector that represents the kernel between the channelsiiandjj\(see Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)a\)\.

More precisely,

𝐖^\[:,:,c,c′\]=1\|c′\|∑i∈c′∑k∈c𝐖\(ℓ\)\[:,:,k,i\],𝐛^c′\(ℓ\)=∑i∈c′𝐛i\(ℓ\)\.\\begin\{split\}\\hat\{\\mathbf\{W\}\}\[:,:,c,c^\{\\prime\}\]&=\\frac\{1\}\{\|c^\{\\prime\}\|\}\\sum\_\{i\\in c^\{\\prime\}\}\\sum\_\{k\\in c\}\\mathbf\{W\}^\{\(\\ell\)\}\[:,:,k,i\],\\\\ \\hat\{\\mathbf\{b\}\}^\{\(\\ell\)\}\_\{c^\{\\prime\}\}&=\\sum\_\{i\\in c^\{\\prime\}\}\\mathbf\{b\}^\{\(\\ell\)\}\_\{i\}\.\\end\{split\}\(20\)

#### 4\.13\.3Fibration compression for an Hypergraph

Let us generalize the concepts of fibrations and lifting to any computational hypergraphs, where two or more nodes interact before acting on the subsequent node\. The most common interaction is a multiplicative modulation or “gating” of activity\.

Suppose that there are two triplets of nodes\(p,i,j\)\(p,i,j\)and\(p′,i′,j′\)\(p^\{\\prime\},i^\{\\prime\},j^\{\\prime\}\)such ashp=hi\.hnh\_\{p\}=h\_\{i\}\.h\_\{n\}andhp′=hi′\.hn′h\_\{p^\{\\prime\}\}=h\_\{i^\{\\prime\}\}\.h\_\{n^\{\\prime\}\}wherehhis the activity of the node\. We will sayp∼p′p\\sim p^\{\\prime\}if and only ifi∼i′i\\sim i^\{\\prime\}andj∼j′j\\sim j^\{\\prime\}\. Notep∼p′⟹hp=hp′p\\sim p^\{\\prime\}\\implies h\_\{p\}=h\_\{p^\{\\prime\}\}\.

Many times, this modulation appears between node layersz=x⊙yz=x\\odot y, wherex,y,z∈ℝdx,y,z\\in\\mathbb\{R\}^\{d\}\(product between forget and input gate in LSTM or product between key and query in the attention mechanism\)\. The activity inzziszi=xi​yi=x^c​y^c′z\_\{i\}=x\_\{i\}y\_\{i\}=\\hat\{x\}\_\{c\}\\hat\{y\}\_\{c^\{\\prime\}\}wheniiis in the fiberccof the layerxxandiiis in the fiberc′c^\{\\prime\}of the layeryy\.

- •LSTM\. An important property of fibration in the gates in an LSTM is that they have the same fiber distribution \(see the forget gate and the input gate in Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)b\)\. This implies that all layers \(x,y,z∈ℝdx,y,z\\in\\mathbb\{R\}^\{d\}\) will be compressed to the same number \(x^,y^,z^∈ℝd′\\hat\{x\},\\hat\{y\},\\hat\{z\}\\in\\mathbb\{R\}^\{d^\{\\prime\}\}\) and the dynamics isz^c=x^c​y^c\\hat\{z\}\_\{c\}=\\hat\{x\}\_\{c\}\\hat\{y\}\_\{c\}whereccis a fiber\. Therefore, if nodes are in the same fiber for the forget gate \(see red nodes in Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)b\), their corresponding nodes in the input gate \(see blue nodes\), output gate, cell input and hidden state, will also be in the same fiber\. Consequently, the base has an identical number of nodes across all gates \(2 nodes per gate in Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)b\)\.
- •Attention Mechanism\. Within an attention module of Transformers, the number of fibers may vary for the query \(q\), key \(k\), and value \(V\) projections \(see Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)c\)\. Then we can reduce the number of nodes inqqandkkfromddtodq′d^\{\\prime\}\_\{q\}anddk′d^\{\\prime\}\_\{k\}, respectively; but we cannot reduce the number of nodes inz=q⊙kz=q\\odot k, i\.e\. fibration compression preserves the number of interactions \(see Fig\.[2](https://arxiv.org/html/2609.01768#S2.F2)c\)\.

### 4\.14Datasets and tasks used in empirical tests

Table[1](https://arxiv.org/html/2609.01768#S4.T1)summarizes the architecture and training hyperparameters for experimental setups: MNIST and ImageNet classification\. Ten independent training runs were conducted for each experiment\. The average training accuracy over epochs is shown in Figs\.[3](https://arxiv.org/html/2609.01768#S2.F3)a and c\.

Table 1:Training configurations for MLP\-MNIST and CNN\-ImageNet experiments\.Table[2](https://arxiv.org/html/2609.01768#S4.T2)lists the hyperparameters used to train the Proximal Policy Optimization \(PPO\) agent: a deep neural network consisting of a sequential stack of three Conv2d layers, a linear layer, and a LSTM\. The agent runs across 65 parallel environments to maximize sample diversity, accumulating experience in batches of 30,720 transitions before each gradient update\. A short BPTT horizon of 2 is adopted to limit the computational cost of backpropagating through the LSTM while still allowing the recurrent state to capture short\-term temporal dependencies\.

Symmetry formation during LSTM training with Proximal Policy Optimization \(PPO\) is not monotonic\. We attribute this to the entropy term in the PPO, which encourages random exploration and prevents premature collapse of the policy toward deterministic actions\[[53](https://arxiv.org/html/2609.01768#bib.bib53)\]\. We hypothesize that the entropy term acts as a symmetry\-regulating mechanism whose effect depends on the concentration of the policy\. Early in training, when the policy is nearly uniform, the curvature of the entropy term is positive, reinforcing existing quasi\-covers and promoting redundancy across nodes\. As the policy focuses on a smaller set of actions, this curvature changes sign, destabilizing symmetric configurations, and allowing nodes to specialize\. This dual role provides a mechanistic account of the non\-monotonic evolution of quasi\-covers observed in the black curve in Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)e, where the proportion of quasi\-covers first decreases, then gradually increases as specialization breaks them\.

Table 2:Training hyperparameters for the PPO agent on Atari Beam Rider\.Table[3](https://arxiv.org/html/2609.01768#S4.T3)details the architecture and training configuration of the sequence\-to\-sequence transformer\. The model uses a symmetric encoder–decoder stack of three layers each, with eight attention heads per layer\. The asymmetry between source \(10,837 tokens\) and target \(19,214 tokens\) vocabulary sizes reflects the morphological richness of German relative to English\. Training is regularized via early stopping with a patience of 20 epochs\.

Table 3:Architecture and training hyperparameters for the sequence\-to\-sequence transformer on the Multi30k German\-to\-English translation task\.
### 4\.15Refinement algorithm for balanced colorings in feedforward networks

Acoloringccof a graphGGis a mapc:NG→𝒞c:N\_\{G\}\\rightarrow\\mathcal\{C\}wherec⁡\(u\)c\(u\)is called the color of the nodeuuand𝒞\\mathcal\{C\}is called the set ofcolors\. The coloringccisin\-balancedifc⁡\(u\)=c⁡\(v\)c\(u\)=c\(v\)implies thatTuT\_\{u\}andTvT\_\{v\}are color isomorphic\. The minimal in\-balanced coloring is a in\-balanced coloring of a graph with the minimal number of colors\. In terms of synchronization, nodes inside the same subset of the in\-balanced coloring partition \(i\.e\. nodes with the same color\) can synchronize its activities, since they receive the same color inputs from the same synchronized nodes\.

A refinement algorithm ofGGstarts with an initial coloringc0c\_\{0\}and produces a new coloringctc\_\{t\}in each iterationtt, based on some criteria, until a desired property is achieved \(e\.g\. in\-balanced coloring\)\.

In a graph without weighted connections and without labeled nodes, the method can be summarized as follows\.

1. 1\.c0c\_\{0\}assigns a trivial color to each vertexvv\(e\.g\.c0​\(v\)c\_\{0\}\(v\)=1\)\.
2. 2\.ci\+1​\(v\)=\(ci​\(v\),\{\{ci​\(w\)∣w​is a neighbor of​v\}\}\)c\_\{i\+1\}\(v\)=\\left\(c\_\{i\}\(v\),\\\{\\\{c\_\{i\}\(w\)\\mid w\{\\text\{ is a neighbor of \}\}v\\\}\\\}\\right\)

At some point, it stabilizesct\+1​\(u\)=ct​\(u\)c\_\{t\+1\}\(u\)=c\_\{t\}\(u\)∀u∈NG\\forall u\\in N\_\{G\},t\>Tt\>T\. We designed a new method for a layered feedforward network\. First, let us reformulate Eq\. \([14](https://arxiv.org/html/2609.01768#S4.E14)\)

i​∼fib​j⇔∀r:∑k\|c⁡\(k\)=rWj​k\(ℓ\)=∑k\|c⁡\(k\)=rWi​k\(ℓ\)⇔∀r:W^i​r\(ℓ\)=W^j​r\(ℓ\)⇔W^\(ℓ\)i,:=W^\(ℓ\)j,:⇔d\(W^\(ℓ\)i,:,W^\(ℓ\)j,:\)=0⇔1−W^\(ℓ\)i,:\.W^\(ℓ\)j,:=0ifW^is L2\-normalized\\begin\{split\}i\\underset\{\\rm fib\}\{\\sim\}j\\iff\\forall r:\\sum\_\{k\|c\(k\)=r\}W^\{\(\\ell\)\}\_\{jk\}=\\sum\_\{k\|c\(k\)=r\}W^\{\(\\ell\)\}\_\{ik\}&\\iff\\forall r:\\hat\{W\}^\{\(\\ell\)\}\_\{ir\}=\\hat\{W\}^\{\(\\ell\)\}\_\{jr\}\\\\ &\\iff\\hat\{W\}^\{\(\\ell\)\}\_\{i,:\}=\\hat\{W\}^\{\(\\ell\)\}\_\{j,:\}\\\\ &\\iff d\(\\hat\{W\}^\{\(\\ell\)\}\_\{i,:\},\\hat\{W\}^\{\(\\ell\)\}\_\{j,:\}\)=0\\\\ &\\iff 1\-\\hat\{W\}^\{\(\\ell\)\}\_\{i,:\}\.\\hat\{W\}^\{\(\\ell\)\}\_\{j,:\}=0\\qquad\\text\{if $\\hat\{W\}$ is L2\-normalized\}\\end\{split\}\(21\)whereW^i​r\(ℓ\)=∑k\|c⁡\(k\)=rWi​k\(ℓ\)\\hat\{W\}^\{\(\\ell\)\}\_\{ir\}=\\sum\_\{k\|c\(k\)=r\}W^\{\(\\ell\)\}\_\{ik\}\.

Given that these conditions are impossible to satisfy exactly for continuous weights, we transform Eq\. \([21](https://arxiv.org/html/2609.01768#S4.E21)\) into a set of inequality constraints parameterized by a toleranceε\\varepsilon\. More precisely, we define the distance matrixDW\(ℓ\)=1−W^\(ℓ\)​W^\(ℓ\)​TD^\{\(\\ell\)\}\_\{W\}=1\-\\hat\{W\}^\{\(\\ell\)\}\\hat\{W\}^\{\(\\ell\)T\}and we use it to calculate the colors/clusters \(i​∼fib​ji\\underset\{\\rm fib\}\{\\sim\}j\) via agglomerative clustering with distance thresholdε\\varepsilon:

1. 1\.c\(ℓ=0\)∈𝒞0fibc^\{\(\\ell=0\)\}\\in\\mathcal\{C\}^\{\\rm fib\}\_\{0\}assigns unique colors to each of the input features \(ℓ=0\\ell=0\)\.
2. 2\.Forℓ=1,…,N\\ell=1,\.\.\.,N: 1. \[label\*=0\.\] 2. \(a\)Calculation ofW^i​r\(ℓ\)=∑k\|c⁡\(k\)=rWi​k\(ℓ\)\\hat\{W\}^\{\(\\ell\)\}\_\{ir\}=\\sum\_\{k\|c\(k\)=r\}W^\{\(\\ell\)\}\_\{ik\}∀r∈𝒞\(ℓ\)\\forall r\\in\\mathcal\{C\}^\{\(\\ell\)\} 3. \(b\)Normalization ofW^\(ℓ\)\\hat\{W\}^\{\(\\ell\)\}\. 4. \(c\)Calculation of the matrixDW\(ℓ\)=I−W^\(ℓ\)​W^\(ℓ\)​TD^\{\(\\ell\)\}\_\{W\}=I\-\\hat\{W\}^\{\(\\ell\)\}\\hat\{W\}^\{\(\\ell\)T\} 5. \(d\)Agglomerative Clustering of the nodes inℓ\\ellbased on the distance matrixDW\(ℓ\)D^\{\(\\ell\)\}\_\{W\}\(see Alg\.[1](https://arxiv.org/html/2609.01768#alg1)\)\. We use a distance thresholdε∈\(0,2\)\\varepsilon\\in\(0,2\)\. 6. \(e\)Fibersc\(ℓ\)∈𝒞ℓfibc^\{\(\\ell\)\}\\in\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\}are defined as the clusters\.

Algorithm 1Agglomerative Clustering1:Initialize clusters

𝒞=1,2,…,n\\mathcal\{C\}=\{\{1\},\{2\},\.\.\.,\{n\}\}
2:while

\|𝒞\|\>1\|\\mathcal\{C\}\|\>1do

3:

∀c,c′∈𝒞:dcomplete​\(c,c′\)=maxi∈c,j∈c′​DW​\(i,j\)\\forall c,c^\{\\prime\}\\in\\mathcal\{C\}:d\_\{\\rm complete\}\(c,c^\{\\prime\}\)=\\underset\{i\\in c,j\\in c^\{\\prime\}\}\{\\max\}D\_\{W\}\(i,j\)
4:

dm​i​n=minc,c′∈𝒞​dcomplete​\(c,c′\)d\_\{min\}=\\underset\{c,c^\{\\prime\}\\in\\mathcal\{C\}\}\{\\min\}d\_\{\\rm complete\}\(c,c^\{\\prime\}\)
5:If

dm​i​n\>εd\_\{min\}\>\\varepsilon: stop

6:

𝒞←\\mathcal\{C\}\\leftarrowMerge the clusters whose distance is less than

dm​i​nd\_\{min\}\.

7:endwhile

8:return

𝒞\\mathcal\{C\}

Note that agglomerative clustering employs a distance thresholdε\\varepsilonto determine cluster merging \(see Alg\.[1](https://arxiv.org/html/2609.01768#alg1)\)\. According to Eq\. \([21](https://arxiv.org/html/2609.01768#S4.E21)\), this is equivalent to treating the equalities∑k∈cWi​k\(ℓ\)=∑k∈cWj​k\(ℓ\)\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{ik\}=\\sum\_\{k\\in c\}W^\{\(\\ell\)\}\_\{jk\}with a tolerance parameterε\\varepsilon, which defines the tolerance to a weak fibration breaking \(see Step 2\.4\)\. In other words, all approximate fibration symmetries should be regarded as quasi\-fibrations\[[30](https://arxiv.org/html/2609.01768#bib.bib30)\]or quasi\-balanced colorings\[[51](https://arxiv.org/html/2609.01768#bib.bib51)\]\.

The coloring generated in the feedforward pass generates𝒞ℓfib\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\}for all layers\. The same algorithm run backwards starting at the output generates𝒞ℓop\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\}\. The number of operations to execute this coloring algorithm in a layered feed\-forward graph scales linearly with the parameter size in each layer and linearly with the number of layersNN\.

For recurrent networks, the fibration and opfibration definitions apply similarly, except that partitioning𝒞ℓfib\\mathcal\{C\}^\{\\rm fib\}\_\{\\ell\}and𝒞ℓop\\mathcal\{C\}^\{\\rm op\}\_\{\\ell\}must be iterated recursively forward and backward, respectively, until convergence\. Convergence requires, at most, the times of the longest loop in the recurrence\[[54](https://arxiv.org/html/2609.01768#bib.bib54)\]\. The code is available at[github\.com/MakseLab/fibrations\_in\_dnns](https://github.com/MakseLab/fibrations_in_dnns)\.

### 4\.16Optimal quasi\-cover thresholds for a given loss threshold

One can use the sameε\\varepsilonfor all layers, or select layer\-specific quasi\-fiber and quasi\-opfiber thresholds\. To find the optimalε\\varepsilonefficiently, it is helpful to have an efficient estimate of thisΔ​L\\Delta L\. We estimate the loss using gradientsGi​k=∂L∂Wi​kG\_\{ik\}=\\frac\{\\partial L\}\{\\partial W\_\{ik\}\}that have already been computed during training\. The weight updateΔ​𝐖\\Delta\{\\bf W\}by compression changes the loss by approximately:

Δ​L≈∑i,k∂L∂Wi​k​Δ​Wi​k=tr​\(𝐆T​Δ​𝐖\)\.\\Delta L\\approx\\sum\_\{i,k\}\\frac\{\\partial L\}\{\\partial W\_\{ik\}\}\\Delta W\_\{ik\}=\\text\{tr\}\(\{\\bf G\}^\{T\}\\Delta\{\\bf W\}\)\.Note thatΔ​W\\Delta Wdepends onε\\varepsilon, whileGGdoes not, since it is computed before compression\. The compression of the entire network, each layerℓ\\ellcontributes a termΔ​L\(ℓ\)\\Delta L^\{\(\\ell\)\}; i\.e\.,

Δ​L​\(ε→\)=∑lΔ​L\(ℓ\)​\(ε→\),\\Delta L\(\\vec\{\\varepsilon\}\)=\\sum\_\{l\}\\Delta L^\{\(\\ell\)\}\(\\vec\{\\varepsilon\}\),\(22\)whereε→=\(εfib\(1\),εfib\(2\),…,εfib\(N\),εop\(1\),εop\(2\),…,εop\(N\)\)\\vec\{\\varepsilon\}=\(\\varepsilon^\{\(1\)\}\_\{\\rm fib\},\\varepsilon^\{\(2\)\}\_\{\\rm fib\},\.\.\.,\\varepsilon^\{\(N\)\}\_\{\\rm fib\},\\varepsilon^\{\(1\)\}\_\{\\rm op\},\\varepsilon^\{\(2\)\}\_\{\\rm op\},\.\.\.,\\varepsilon^\{\(N\)\}\_\{\\rm op\}\)\. On the other hand, for eachε→\\vec\{\\varepsilon\}, the collapsed network hasP⁡\(ε→\)P\(\\vec\{\\varepsilon\}\)parameters\.PP, as a function of each dimension ofε→\\vec\{\\varepsilon\}, is decreasing\. We define a threshold for the variation of the loss functionεL​o​s​s\\varepsilon\_\{Loss\}and want to solve the following constrained optimization problem:

minε→⁡P⁡\(ε→\)withΔ​L​\(ε→\)≤εLoss\.\\min\_\{\\vec\{\\varepsilon\}\}P\(\\vec\{\\varepsilon\}\)\\quad\\text\{with \}\\quad\\Delta L\(\\vec\{\\varepsilon\}\)\\leq\\varepsilon\_\{\\rm Loss\}\.\(23\)
With this formulation, we seek the maximum possible compression, provided that the collapsed network performs similarly to the original network \(in terms of loss function\)\.εLoss\\varepsilon\_\{\\text\{Loss\}\}must be defined a priori as a fixed value\. Unlikeε\\varepsilon, which we limit to≤1\\leq 1in the search by normalizing the weight matrix in the coloring algorithm \(see Eq\. \([21](https://arxiv.org/html/2609.01768#S4.E21)\),εLoss\\varepsilon\_\{\\text\{Loss\}\}is not confined to a bounded domain\. A natural question is how to determine a reasonable value forεLoss\\varepsilon\_\{\\text\{Loss\}\}\. Each network is designed to perform a specific task, whose performance is measured by a metricAA\(e\.g\., accuracy\)\. Although there is no exact functional relationship betweenAAandLL, it is generally expected thatAAincreases asLLdecreases\. Moreover, beyond a certain point during training,AAstops increasing and stabilizes, whileLLcontinues to decrease \(see Fig\.[7](https://arxiv.org/html/2609.01768#S4.F7)\)\. The value ofLLbeyond which further reductions in loss no longer produce improvements inAAdefines ourεLoss\\varepsilon\_\{\\text\{Loss\}\}\(dotted line in Fig\.[7](https://arxiv.org/html/2609.01768#S4.F7)\)\. Note that both theAAvs\.LLrelationship and the value ofεL​o​s​s\\varepsilon\_\{Loss\}depend strongly on the architecture and the task to be performed\.

Figure 7:\(Left panel\) Relationship between accuracy and loss function\. There exists an interval of the loss function where the network does not lose performance in terms of accuracy\. This interval defines the acceptableεLoss\\varepsilon\_\{\\rm Loss\}\. \(Right panel\) For those acceptableεLoss\\varepsilon\_\{\\rm Loss\}, we determine the optimalεfib\\varepsilon\_\{\\rm fib\}andεop\\varepsilon\_\{\\rm op\}using various search method\. Thus, eachεLoss\\varepsilon\_\{\\rm Loss\}determines an optimal compressed size\.\(a\)For the MLP we used exhaustive grid search \(red\) in 6 dimensions\(εfib\(1\),εop\(1\),εfib\(2\),εop\(2\),εfib\(3\),εop\(3\)\)\(\\varepsilon^\{\(1\)\}\_\{\\rm fib\},\\varepsilon^\{\(1\)\}\_\{\\rm op\},\\varepsilon^\{\(2\)\}\_\{\\rm fib\},\\varepsilon^\{\(2\)\}\_\{\\rm op\},\\varepsilon^\{\(3\)\}\_\{\\rm fib\},\\varepsilon^\{\(3\)\}\_\{\\rm op\}\)and exhaustive search in Partition Space in 3 dimensions\(εfib\(1\),εfib\(2\),εfib\(3\)\)\(\\varepsilon^\{\(1\)\}\_\{\\rm fib\},\\varepsilon^\{\(2\)\}\_\{\\rm fib\},\\varepsilon^\{\(3\)\}\_\{\\rm fib\}\)for fixed values ofεop\\varepsilon\_\{\\rm op\}\(brown, green, magenta\)\.\(b\)For the CNN we did the same exhaustive search in Partition Space in 3 dimensions\.\(c\)For the transformer we did Coordinate Descent in 41 dimensional space ofε→fib\\vec\{\\varepsilon\}\_\{\\rm fib\}and fixed values ofεop\\varepsilon\_\{\\rm op\}\. Note thatεop=1\\varepsilon\_\{\\rm op\}=1corresponds to enforcing quasi\-fiber conditions only\. BLEU \(Bilingual Evaluation Understudy\) is a standar metric that measures the similarity between a machine\-generated text and human\-written reference translations\.Note that Eq\.[22](https://arxiv.org/html/2609.01768#S4.E22)is an efficient estimation ofΔ​L\\Delta Lbecause we reduce the computational cost of computingLL, which is usually high for large datasets\. In contrast,𝐆\\bf Gis computed automatically during training of the original network and does not depend onε→\\vec\{\\varepsilon\}\. OnlyΔ​W\\Delta Wdependent onε→\\vec\{\\varepsilon\}, as they determine the coloring of the nodes\.

### 4\.17Redundant feature pruning is a special case of fibration compression

A previous work\[[56](https://arxiv.org/html/2609.01768#bib.bib56)\]proposed to identify nodes that have the same input and output weights, and compress networks by removing these duplicate nodes to accelerate forward inference\. Here we show that this is a special case of covering symmetry, namely, a trivial permutation of nodes in a single layer, i\.e\., Eq\. \([14](https://arxiv.org/html/2609.01768#S4.E14)\) and \([16](https://arxiv.org/html/2609.01768#S4.E16)\) without the sums\. Their method organizes weights as follows:

𝐖\(ℓ\)↦\[vect\(W\[:,:,1,1\]\)\.\.\.vect\(W\[:,:,1,dL\]\)vect\(W\[:,:,2,1\]\)\.\.\.vect\(W\[:,:,2,dL\]\)\.\.\.\.\.\.\.\.\.vect\(W\[:,:,dℓ−1,1\]\)\.\.\.vect\(W\[:,:,dL−1,dL\]\)\]=\[w1\.\.\.wdL\]\\mathbf\{W\}^\{\(\\ell\)\}\\mapsto\\begin\{bmatrix\}\\text\{vect\}\(W\[:,:,1,1\]\)&\.\.\.&\\text\{vect\}\(W\[:,:,1,d\_\{L\}\]\)\\\\ \\text\{vect\}\(W\[:,:,2,1\]\)&\.\.\.&\\text\{vect\}\(W\[:,:,2,d\_\{L\}\]\)\\\\ \.\.\.&\.\.\.&\.\.\.\\\\ \\text\{vect\}\(W\[:,:,d\_\{\\ell\-1\},1\]\)&\.\.\.&\\text\{vect\}\(W\[:,:,d\_\{L\-1\},d\_\{L\}\]\)\\\\ \\end\{bmatrix\}=\\begin\{bmatrix\}w\_\{1\}&\.\.\.&w\_\{d\_\{L\}\}\\end\{bmatrix\}and uses the following similarity metric for agglomerative clustering

Sim​\(Ca,Cb\)=1\|Ca\|​\|Cb\|​∑i∈Ca,j∈Cbwi⋅wj\\text\{Sim\}\(C\_\{a\},C\_\{b\}\)=\\frac\{1\}\{\|C\_\{a\}\|\|C\_\{b\}\|\}\\sum\_\{i\\in C\_\{a\},j\\in C\_\{b\}\}w\_\{i\}\\cdot w\_\{j\}\(24\)
In the strictest case \(sim = 1\),i,ji,jare in the same cluster ifwi=wjw\_\{i\}=w\_\{j\}\. That means,∀n=1,…,dℓ−1:Wi​n=Wj​n\\forall n=1,\.\.\.,d\_\{\\ell\-1\}:W\_\{in\}=W\_\{jn\}, which corresponds to trivial permutation symmetry\.

### 4\.18Orthogonal regularization is a particular case of symmetry breaking

“Node collapse” sometimes refers to nodes in an artificial neural network that become correlated in their activity, representing the same features of the data\. Note that this phenomenon can signal the emergence of fibration symmetry due to the synchronization theorem\. It was reported in\[[24](https://arxiv.org/html/2609.01768#bib.bib24),[25](https://arxiv.org/html/2609.01768#bib.bib25)\]\. Several ad hoc ways to avoid correlations exist\. For example, orthogonal regularization prevents node collapse, forcing weight matrices to be orthogonal\[[57](https://arxiv.org/html/2609.01768#bib.bib57)\]\. Networks are initialized with random connections that are generally orthogonal, and this method ensures that the weight updates maintain this orthogonality\.

Assume a linear networky=W​x∈ℝd1y=Wx\\in\\mathbb\{R\}^\{d\_\{1\}\}withx∈ℝd0x\\in\\mathbb\{R\}^\{d\_\{0\}\}yW∈ℝd1×d0W\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{0\}\}\. For simplicity, assume that each column ofWWis normalized\. Orthogonal regularization aims to reduce‖I−WT​W‖=‖I−1\+1−WT​W‖=‖I−1\+DW‖\|\|I\-W^\{T\}W\|\|=\|\|I\-1\+1\-W^\{T\}W\|\|=\|\|I\-1\+D\_\{W\}\|\|\(see the definition ofDWD\_\{W\}in[4\.15](https://arxiv.org/html/2609.01768#S4.SS15)\)\. That is, orthogonal regularization expectsDWD\_\{W\}to have zeros on the diagonal and ones everywhere else\. This implies that if a clustering algorithm is applied toDWD\_\{W\}, all dimensions will be in distinct clusters\. In other words, the fibers in the output layer will all be trivial\. Consequently, if the weights for each output node are orthogonal, no fibers exist\. Then, orthogonality is a way to prevent fiber formation and avoid node synchronization

### 4\.19Continual learning

We explore the Fibration Symmetry Breaking algorithm to resolve the problem of loss of plasticity encountered in sequential learning scenarios\. Our investigation focuses on the Continual ImageNet task, comprising 5,000 binary classification tasks formed by pairing distinct classes from ImageNet\. A deep neural network is trained on images from two classes per task and evaluated on a corresponding test set\. ImageNet, with 1,000 classes and approximately 700 images per class, provides the dataset\. Each class is divided into 600 training images and 100 test images\. Tasks are presented sequentially, with a new task sampling a new pair of classes\. Training occurs over 250 epochs per task\. Performance, assessed by the accuracy of the test set, highlights a loss of plasticity if the accuracy decreases\.

We utilize a CNN with three convolutional layers and three fully connected layers, with the final layer consisting of two nodes for the current task’s classes\. The output layer is reinitialized for each new task, following common practice\. We evaluate standard gradient descent, deep continual learning, and symmetry\-breaking gradient descent, all minimizing cross\-entropy loss\. Each method undergoes 10 independent runs with the same sequence of class pairs, and the network weights are initialized once before the first task\.

The results, illustrated in Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)g, are representative of a feed\-forward convolutional network using unmodified backpropagation\. Initially, networks achieved up to 88% accuracy on early tasks but lost plasticity by the 2,000th task\. This pattern was consistent across architectures, parameters, and optimizers, indicating standard deep\-learning methods struggle with continual\-learning tasks\.

The emergence of covers during learning is exemplified in Fig\.[8](https://arxiv.org/html/2609.01768#S4.F8)for the sequential ImageNet task\. At the start \(Epoch 0\) all 512 nodes in the layer constitute trivial fibers \(of size 1\)\. After some training epochs, large covers emerge\. Continual backpropagation and even more so symmetry breaking keep the cover size in check and therefore have a larger number of them, across the 250 epochs of the first task \(first row\) and even more clearly after 900 tasks \(second row\)\.

To decide when and how many nodes to reinitialize, we have to select a particular schedule\. For easier comparison, we use the same re\-initialization schedule as in\[[3](https://arxiv.org/html/2609.01768#bib.bib3)\]\. Namely, nodes are re\-initialized at a fixed replacement rate \(which is the number of node per gradient step\), taken from covers only if they reach a certain size maturity threshold\. We trained the network with a momentum term\. The results in Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)g are for a learning rate of 0\.01, a replacement rate of 3e\-4, a maturity threshold of 100 and a decay rate of 0\.99\. Minibatch sizes of 100, a momentum of 0\.9\.

![Refer to caption](https://arxiv.org/html/2609.01768v1/Fig_Covers_DCL-eps-converted-to.png)Figure 8:Distribution of covers, fibers and opfibers during continual learning on the Sequential ImageNet Task\.Evolution of the size and number of covers across leaning epochs \(columns\) and task number \(rows\) for different training methods: \(blue\) SGD \(red\) continual backpropagation \(yellow\) FSB\. This is the distribution of sizes for the second dense layer\.
### 4\.20Deep Continual Learning is a particular case of Fibration Symmetry Breaking

In Deep Continual Learning \(DCL\) algorithm proposed in\[[3](https://arxiv.org/html/2609.01768#bib.bib3)\], the contribution utility is

ui\(ℓ\)​\(t\+1\)\\displaystyle u^\{\(\\ell\)\}\_\{i\}\(t\+1\)=η\.ui\(ℓ\)​\(t\)\+\(1−η\)​Fi\(ℓ\)​\(t\),\\displaystyle=\\eta\.u^\{\(\\ell\)\}\_\{i\}\(t\)\+\(1\-\\eta\)F^\{\(\\ell\)\}\_\{i\}\(t\),Fi\(ℓ\)​\(t\)\\displaystyle F^\{\(\\ell\)\}\_\{i\}\(t\)=hi\(ℓ\)​\(t\)\.∑k=1dℓ\+1Wk​i\(ℓ\+1\)​\(t\)\\displaystyle=h^\{\(\\ell\)\}\_\{i\}\(t\)\.\\sum\_\{k=1\}^\{d\_\{\\ell\+1\}\}W^\{\(\\ell\+1\)\}\_\{ki\}\(t\)withui\(ℓ\)​\(0\)=0u^\{\(\\ell\)\}\_\{i\}\(0\)=0\.

If there are two nodesi​∼c​o​v,ℓ,t​ji\\underset\{cov,\\ell,t\}\{\\sim\}j, thenFi\(ℓ\)​\(t\)=Fj\(ℓ\)​\(t\)F^\{\(\\ell\)\}\_\{i\}\(t\)=F^\{\(\\ell\)\}\_\{j\}\(t\)becausehi\(ℓ\)=hj\(ℓ\)h^\{\(\\ell\)\}\_\{i\}=h^\{\(\\ell\)\}\_\{j\}for fibration symmetry and∑k=1dℓ\+1Wk​i\(ℓ\+1\)​\(t\)=∑k=1dℓ\+1Wk​j\(ℓ\+1\)​\(t\)\\sum\_\{k=1\}^\{d\_\{\\ell\+1\}\}W^\{\(\\ell\+1\)\}\_\{ki\}\(t\)=\\sum\_\{k=1\}^\{d\_\{\\ell\+1\}\}W^\{\(\\ell\+1\)\}\_\{kj\}\(t\)for opfibration symmetry\.

If there are two nodesi​∼c​o​v,ℓ,t​ji\\underset\{cov,\\ell,t\}\{\\sim\}jwith identical utilitiesui\(ℓ\)​\(t\)=uj\(ℓ\)​\(t\)u^\{\(\\ell\)\}\_\{i\}\(t\)=u^\{\(\\ell\)\}\_\{j\}\(t\), thenui\(ℓ\)​\(t\+1\)=uj\(ℓ\)​\(t\+1\)u^\{\(\\ell\)\}\_\{i\}\(t\+1\)=u^\{\(\\ell\)\}\_\{j\}\(t\+1\)\(for definition ofuu\) andi​∼c​o​v,ℓ,t\+1​ji\\underset\{cov,\\ell,t\+1\}\{\\sim\}j\. Therefore, we obtainui​\(s\)=uj​\(s\)∀t≤su\_\{i\}\(s\)=u\_\{j\}\(s\)\\quad\\forall t\\leq s\. That means thatcovering symmetry implies utility synchronization\.

Deep Continual Learning resets nodes belonging to the same cover when their utility reaches zero\. For these nodes, input weights are reinitialized randomly, and output weights are fixed at zero\. This modifies the node’s activityhhwhile avoiding immediate disruption of the learned network behavior\. Altering both input and output weights necessarily modifies the input and output trees of the nodes, thus breaking their associated covers\. In particular, zero\-utility nodes typically exhibit null activity \(h=0h=0\) and belong to the same fiber\. Thus, DCL inherently targets the breaking of inactive covers\. The proposed fibration symmetry\-breaking protocol, on the other hand, targets all covers, not only the inactive ones\. This explains why the fibration breaking approach overcomes DCL in continual learning in Fig\.[3](https://arxiv.org/html/2609.01768#S2.F3)g\.

### 4\.21Fibers are a refinement of clustering partitions

Consider a classification problem where each inputxxhas to be classified by outputyyinto one of theNNclasses\. For each classcc, we definePc\(ℓ\)P^\{\(\\ell\)\}\_\{c\}as the partition obtained by the correlation matrix of activity with the examples drawn from classcc\. Each setA∈Pc\(ℓ\)A\\in P^\{\(\\ell\)\}\_\{c\}is a cluster of nodes in layerℓ\\ellwhere the activity is synchronized for all inputs in classcc\(i\.e\.hi​\(x\)=hj​\(x\)∀x∈ch\_\{i\}\(x\)=h\_\{j\}\(x\)\\quad\\forall x\\in c\)\. We refer to these as class\-conditional synchronization clusters\. Examples are shown in Fig\.[9](https://arxiv.org/html/2609.01768#S4.F9)\.

![Refer to caption](https://arxiv.org/html/2609.01768v1/Fig_Num_Clusters_exp_01.png)Figure 9:Class\-conditional synchrony clusters for class 9 in multiples layers of the MLP trained on MNIST\. On the right, we show the number of clusters for layer 3 as function of learning epochs for all 10 classes\.Compute the intersection of clusters from different classes

I=A1∩A2∩…∩ANI=A\_\{1\}\\cap A\_\{2\}\\cap\.\.\.\\cap A\_\{N\}
Two nodes inIIare synchronized for all inputs in the data set; however, they may not be synchronized for an input that is not part of the data set\. That is, a fiber is always contained within a set likeII\. In other words, the partition formed by the fibers is a refinement of the partition formed by sets of the formII\(i\.e\. intersection of clusters across classes in Fig\.[11](https://arxiv.org/html/2609.01768#S4.F11)b\)\.

To test this theory numerically, we threshold the correlation coefficients for the synchrony clusters as well as the fibration condition \- Eq\. \([14](https://arxiv.org/html/2609.01768#S4.E14)\), and then measure a refinement score between these two partitions of nodes \(Fig\.[10](https://arxiv.org/html/2609.01768#S4.F10)b\)\. Therefinement score, counts the number of fibers that have a unique cluster label normalized by the total number of fibers\. For small thresholds, the theoretical result is validated \(refinement score=1\) while for larger cluster thresholds the relationship breaks down \(refinement score=0\), as expected\. This observation is more generally true for any subset of all possible inputs, including inputs generated at random\.

The synchronization theorem states that fibers imply synchronization for any arbitrary input, but the reverse is not necessarily true\. We test the reverse of this statement empirically by comparing the fiber partition with synchronization clusters for random inputs\. We measure how well the two partitions match using asmatching score\. For this we use the “clustering accuracy”, which is the maximized sum of the diagonal of the confusion matrix divided by the total number of items\. The maximum assignment of partition labels is found with the linear sum assignment algorithm\. For small thresholds, the fiber partition and synchronization clusters are a perfect match \(matching score=1, Fig\.[10](https://arxiv.org/html/2609.01768#S4.F10)a\), showing that fibers can also be identified simply by correlating nodes under random inputs\.

![Refer to caption](https://arxiv.org/html/2609.01768v1/Fig_Scores.png)Figure 10:Matching score and refinement score compare the these clusters with fibers for different thresholds of identifying the clusters \(horizontal\) and fibers \(vertical\)
### 4\.22Hierarchical organization of the loss landscape under SGD dynamics

Cover formation is a hierarchical coarse\-graining under the SGD dynamic in the parameter and node space\. This is exemplified in Fig\.[11](https://arxiv.org/html/2609.01768#S4.F11)b\-c, where we show the state of the network in three different training stages \(T1,T2,T3T\_\{1\},T\_\{2\},T\_\{3\}\), their transitionsT1→2T\_\{1\\to 2\}andT2→3T\_\{2\\to 3\}, and the evolution of three nodes\(i,j,k\)\(i,j,k\)as covers merge\.

Covers are a refinement of fibers\. Fibers are a refinement of synchronization clusters, and these in turn are a refinement of class\-conditional synchrony clusters\. This is exemplified in Fig\.[11](https://arxiv.org/html/2609.01768#S4.F11)b, along with the development of covers and class\-conditional clusters during learning\. The red box groups the class\-conditional clusters for the red class \(and similarly for the blue class\) at timeT1T\_\{1\}\. Covers in green are a refinement of both red and blue clusters\. As learning progresses, the covers merge at transitionsT1→2T\_\{1\\to 2\}andT2→3T\_\{2\\to 3\}according to the Cover Coarse\-Graining Theorem\.

Figure[11](https://arxiv.org/html/2609.01768#S4.F11)c is a symbolic representation of a parameter space \(weights\)\. The potential learning trajectories \(black curves\) merge the covers containing nodesi,ji,jandkk\. During the learning phaseT1T\_\{1\}, the three nodesi,j,i,j,andkkbelong to distinct covers\. At a later point in the training,iiandjjmerge into the same cover\. According to the Cover Coarse\-Graining Theorem, once this occurs, they cannot separate again\. In other words, the first timeiiandjjshare a cover defines an irreversible transition \(green dashed lineT1→2T\_\{1\\to 2\}\)\. A similar transition occurs when the cover containing nodekkmerges into the cover containingiiandjj, defining a second transition lineT2→3T\_\{2\\to 3\}\. The colored zones in this space are regions where the network has the same cover baseBB\(with identical nodes and weights\)\.

The loss landscapeℒ⁡\(w\)\\mathcal\{L\}\(w\), withw∈ℝdw\\in\\mathbb\{R\}^\{d\}, is characterized by a hierarchy that can be formally represented as a tree \(Fig\.[11](https://arxiv.org/html/2609.01768#S4.F11)d\)\. All network configurations in thisdd\-dimensional parameter space that have the same minimal baseB′′B^\{\\prime\\prime\}will have the same loss value\. The root of the tree is this base inℝd′′\\mathbb\{R\}^\{d^\{\\prime\\prime\}\}; while the leaf nodes are all the possible lifting of the base inℝd\\mathbb\{R\}^\{d\}\. The tree contains multiple leaves becauseφfib\\varphi\_\{\\rm fib\}can reduce different graphsGGinto the same minimal baseB′′B^\{\\prime\\prime\}\(This is evident in Eq\. \([6](https://arxiv.org/html/2609.01768#S2.E6)\), as a linear subset of all possibleW\(ℓ\)W^\{\(\\ell\)\}can generate the sameW^\(ℓ\)\\hat\{W\}^\{\(\\ell\)\}\)\. The inverse lifting operation can transform a representation fromℝd′′\\mathbb\{R\}^\{d^\{\\prime\\prime\}\}\(minimal baseB′′B^\{\\prime\\prime\}\) toℝd\\mathbb\{R\}^\{d\}\(original networkGG\) through an intermediate lifting toℝd′\\mathbb\{R\}^\{d^\{\\prime\}\}whered′′<d′<dd^\{\\prime\\prime\}<d^\{\\prime\}<d\. These intermediate steps create the hierarchy of the trees \(see Methods[4\.3](https://arxiv.org/html/2609.01768#S4.SS3)\- Fig\.[5](https://arxiv.org/html/2609.01768#S4.F5)\)\.

The specific shape of the cost function in this hierarchy can appear as disjoint local minima in the full parameter space ofGG\(orange curves, point 3\.1\) corresponding to a single point the parameter space of the baseB′′B^\{\\prime\\prime\}\. In other cases, the minimum can be flat valleys in the loss landscape \(purple surface, point 2\.1\)\. Elsewhere, the structure of the loss function is replicated, creating more complex patterns \(green curves, point 1\)\.

The learning trajectory navigates downhill in this loss landscape \(solid black line, in Figs\.[11](https://arxiv.org/html/2609.01768#S4.F11)c\-d\) following the hierarchical coarsening of the covers under the SGD learning dynamics, where the system progressively explores increasingly coarse\-grained symmetry quotients during optimization\. This structure has analogies in the ultrametric organization of associative memory in neural networks\[[58](https://arxiv.org/html/2609.01768#bib.bib58)\]and the energy landscape in spin glasses\[[59](https://arxiv.org/html/2609.01768#bib.bib59)\]\.

![Refer to caption](https://arxiv.org/html/2609.01768v1/Fig_LossLanscape.png)Figure 11:Synchronization, hierarchical clustering and loss landscape\.\(a\)Class\-conditional correlation matrix of activity with network inputs drawn from individual classes\. Here they are shown for the last layer of an MLP trained on MNIST in early epochs \(1\), in middle of the training \(2\.1\) and after training \(3\.1\)\. Clusters in this matrix are referred to as synchrony clusters\.\(b\)Temporal evolution of covers in one layer\. In one of the early epochs, \(1\), we identify synchrony clusters \(red and blue\) and covers \(green\)\. Covers are located within cluster intersections\. In later epochs, \(2\.1\) and \(3\.1\), covers merged to form coarser partitions \(purple and orange\)\. Nodesii,jjandkkstart in different covers \(green\), but merge into the same cover \(purple and orange\)\.\(c\)A network is a point in parameter space, e\.g\. at time point \(1\)\. Multiple networks can have the same basisBB\(green region\)\. Learning trajectories \(black curves\) merge nodesiiandjjinto the same cover at an irreversible transition boundaryT1−2T\_\{1\-2\}\. MultipleBBmay haveiiandjjin the same cover \(e\.g\., network \(2\.1\) and \(2\.2\)\)\. SGD selects one of these possibilities\. Oncekkmerges into the same cover, another irreversible transition occursT2−3T\_\{2\-3\}\.\(d\)Hierarchical loss landscape \(see text for details\)\.
### 4\.23Low\-rank weight compression

An existing heuristic for network compression identifies input weights that are \(nearly\) collinear using a singular value decomposition \(SVD\) of the weight matrix\. The weight matrix in any layer of a network can be factorized, removing null or low\-variance dimensions, without substantially altering the function of the network\. The factorization then has a smaller parameter count compared to the original network\. Several methods were proposed simultaneously following this idea, e\.g\.\[[60](https://arxiv.org/html/2609.01768#bib.bib60),[61](https://arxiv.org/html/2609.01768#bib.bib61),[62](https://arxiv.org/html/2609.01768#bib.bib62)\]\. In modern LLMs, the removal of low\-variance dimensions of weight matrices can even improve performance without further tuning\[[45](https://arxiv.org/html/2609.01768#bib.bib45)\], and can achieve compression factors of≈\\approx3x\[[46](https://arxiv.org/html/2609.01768#bib.bib46)\]\.

In principle, compression with SVD followed by fibration compression can be easily combined\. SVD factors a matrix asW≈W\(2\)​W\(1\)W\\approx W^\{\(2\)\}W^\{\(1\)\}whereW\(2\)W^\{\(2\)\}haskkorthonormal columns andW\(1\)W^\{\(1\)\}haskkorthogonal rows\. Essentially, this is equivalent to adding an intermediate linear layer withkknodes\. The approximation is exact ifkkis the rank ofWW\. Withdout​dind\_\{\\rm out\}d\_\{\\rm in\}parameters forWW, the factorization hask⁡\(dout\+dim\)k\(d\_\{\\rm out\}\+d\_\{\\rm im\}\)parameters, which is more compact if one selectsk<dout​din/\(dout\+din\)k<d\_\{\\rm out\}d\_\{\\rm in\}/\(d\_\{\\rm out\}\+d\_\{\\rm in\}\)\. To make this effective, the dimensionkkof the subspace must be selected small, at the risk of incurring an error, which is typically corrected by the subsequent fine\-tuning of the matricesW\(1\)W^\{\(1\)\}andW\(2\)W^\{\(2\)\}\. For instance, in a square matrixdin=dout=1,000d\_\{\\rm in\}=d\_\{\\rm out\}=1,000the rankkkneeds to be smaller than500500for a compression to be effective\. Then, a parameter compression factor of 4x would require a rank compression of 8x\. Applying a fibration compression to this simply reduces the size of the new individual factors matrices:

W^c′​c=1\|c′\|​∑i∈c′∑j∈c∑kWi​k\(2\)​Wk​j\(1\)=∑kW^i​k\(2\)​W^k​j\(1\)\.\\hat\{W\}\_\{c^\{\\prime\}c\}=\\frac\{1\}\{\|c^\{\\prime\}\|\}\\sum\_\{i\\in c^\{\\prime\}\}\\sum\_\{j\\in c\}\\sum\_\{k\}W\_\{ik\}^\{\(2\)\}W\_\{kj\}^\{\(1\)\}=\\sum\_\{k\}\\hat\{W\}\_\{ik\}^\{\(2\)\}\\hat\{W\}\_\{kj\}^\{\(1\)\}\.For this representation to still be smaller,kkhas to be now smaller thank<d^out​d^in/\(d^out\+d^in\)k<\\hat\{d\}\_\{\\rm out\}\\hat\{d\}\_\{\\rm in\}/\(\\hat\{d\}\_\{\\rm out\}\+\\hat\{d\}\_\{\\rm in\}\), i\.e\. with the smaller node dimensions after fibration compression\. So SVD first followed by fibration compression can be readily combined, but the compression efficiency of SVD is potentially lost\. An SVD compression after fibration compression may potentially find additional co\-linear nodes that are not identified by fibration compression\.

From our current perspective, we can prove that if the input of any layer lies in a linear subspace, then the input weight matrix of that layer will be attracted by SGD into that subspace\. Specifically, ifWWis in the linear subspace of the inputXX, then so is the gradient∂L/∂W=Δ​XT\\partial L/\\partial W=\\Delta X^\{T\}\(bothXXandΔ\\Deltaare here matrices of dimensions×\\timesexemplars\)\. Therefore, the linear subspace defined by the data becomes an invariant set for the weight matrix, and will be an stable attractor for SGD, following\[[32](https://arxiv.org/html/2609.01768#bib.bib32)\]\. For high\-dimensional representation layers, it is quite possible that the input is rank\-deficient, which would explain why low\-rank weight matrices are found empirically and are amenable to compression\.

SVD of a weight matrixWWreveals a symmetry of algebraic nature: it decomposes the transformation into pure rotationsUU,VV, and scalingsΣ\\Sigma\. However, this type of symmetry is not part of the global\-local hierarchy of graph symmetries shown in Fig\.[1](https://arxiv.org/html/2609.01768#S1.F1)\. Graph symmetries define a partition of nodes, where nodes in the same class are structurally indistinguishable under one criterion \(e\.g\., isomorphic input trees in fibration symmetries\)\. Although graph symmetries identify structural redundancies, SVD uncovers the underlying geometric simplicity of the mappingW=U​Σ​VTW=U\\Sigma V^\{T\}\.

## Acknowledgements

Partial support for this work was provided by the National Institutes of Health through grants R01CA247910 \(OMV, LCP, HAM\) and R01EB028157 \(HAM\), as well as the Army Research Office with grant WF911\-NF\-24\-1\-0031 \(OMV, LCP\)\. We thank Matteo Serafino for his help and discussions\.

## Declarations

##### Funding:

Partial support for this work was provided by the National Institutes of Health through grants R01CA247910 \(OMV, LCP, HAM\) and R01EB028157 \(HAM\), as well as the Army Research Office with grant WF911\-NF\-24\-1\-0031 \(OMV, LCP\)\.

##### Conflict of interest/Competing interests:

There are no competing interests to declare\.

##### Ethics approval and consent to participate:

N/A

##### Consent for publication:

N/A

##### Data, materials, and code availability:

##### Author contributions:

HAM conceived the project\. OMV, HAM, and LCP designed the experiments, contributed to the mathematical proofs, and wrote the paper\. All authors contributed to the analysis of data\. OMV and AH developed the analysis code\.

## References

- \(1\)Kaplan, J\.*et al\.*Scaling laws for neural language models\.*arXiv*\(2020\)\.URL[https://arxiv\.org/abs/2001\.08361](https://arxiv.org/abs/2001.08361)\.
- \(2\)Hoffmann, J\.*et al\.*Training compute\-optimal large language models\.*arXiv*\(2022\)\.URL[https://arxiv\.org/abs/2203\.15556](https://arxiv.org/abs/2203.15556)\.
- \(3\)Dohare, S\.*et al\.*Loss of plasticity in deep continual learning\.*Nature*632, 768–774 \(2024\)\.
- \(4\)Lyle, C\.*et al\.**Understanding plasticity in neural networks*, 23190–23211 \(PMLR, 2023\)\.
- \(5\)Lyle, C\.*et al\.*Disentangling the causes of plasticity loss in neural networks\.*arXiv*\(2024\)\.URL[https://arxiv\.org/abs/2402\.18762](https://arxiv.org/abs/2402.18762)\.
- \(6\)Sejnowski, T\. J\.The unreasonable effectiveness of deep learning in artificial intelligence\.*Proc\. Nat\. Acad\. Sci\. USA*117, 30033–30038 \(2020\)\.
- \(7\)Belkin, M\., Hsu, D\., Ma, S\. & Mandal, S\.Reconciling modern machine\-learning practice and the classical bias–variance trade\-off\.*Proc\. Nat\. Acad\. Sci\. USA*116, 15849–15854 \(2019\)\.
- \(8\)Nakkiran, P\.*et al\.**Deep double descent: Where bigger models and more data hurt*\(2020\)\.
- \(9\)Bronstein, M\. M\., Bruna, J\., Cohen, T\. & Veličković, P\.Geometric deep learning: Grids, groups, graphs, geodesics, and gauges\.*arXiv*\(2021\)\.URL[https://arxiv\.org/abs/2104\.13478](https://arxiv.org/abs/2104.13478)\.
- \(10\)Wilczek, F\.*A Beautiful Question: Finding Nature’s Deep Design*\(Penguin Press, 2016\)\.
- \(11\)Weinberg, S\.*The Quantum Theory of Fields*Vol\. 2 \(Cambridge University Press, 1995\)\.
- \(12\)Makse, H\. A\., Boldi, P\., Sorrentino, F\. & Stewart, I\.*Symmetries of Living and Artificial Intelligence Systems: Fibrations and Synchronization in Networks*\(Cambridge University Press, forthcoming, 2027\)\.URL[https://arxiv\.org/pdf/2502\.18713](https://arxiv.org/pdf/2502.18713)\.
- \(13\)Gili, T\.*et al\.*Fibration symmetry\-breaking supports functional transitions in a brain network engaged in language\.*arXiv*\(2025\)\.URL[https://arxiv\.org/abs/2409\.02674](https://arxiv.org/abs/2409.02674)\.
- \(14\)Avila, B\.*et al\.*Symmetries and synchronization from whole\-neural activity inc\. elegansconnectome: Integration of functional and structural networks\.*Proc\. Natl\. Acad\. Sci\. USA*122, e2417850122 \(2025\)\.
- \(15\)Velarde, O\., Parra, L\. C\., Boldi, P\. & Makse, H\. A\.The role of fibration symmetries in geometric deep learning\.*Proc\. Natl\. Acad\. Sci\. USA*123, e2416552123 \(2026\)\.
- \(16\)Lecun, Y\., Bottou, L\., Bengio, Y\. & Haffner, P\.Gradient\-based learning applied to document recognition\.*Proceedings of the IEEE*86, 2278–2324 \(1998\)\.
- \(17\)Cohen, T\. & Welling, M\.Balcan, M\. F\. & Weinberger, K\. Q\. \(eds\)*Group equivariant convolutional networks*\.\(eds Balcan, M\. F\. & Weinberger, K\. Q\.\)*Proceedings of The 33rd International Conference on Machine Learning*, Vol\. 48 of*Proceedings of Machine Learning Research*, 2990–2999 \(PMLR, New York, New York, USA, 2016\)\.
- \(18\)Kipf, T\. N\. & Welling, M\.Semi\-supervised classification with graph convolutional networks\.*arXiv*\(2016\)\.URL[https://arxiv\.org/abs/1609\.02907](https://arxiv.org/abs/1609.02907)\.
- \(19\)Satorras, V\. G\., Hoogeboom, E\. & Welling, M\.*E\(n\) equivariant graph neural networks*\(2021\)\.
- \(20\)Makse, H\. A\. & Perazzo, R\. P\. J\.The thermodynamics of dyslexic learning\.*Inter\. J\. Neural Sys\.*3, 351–360 \(1992\)\.
- \(21\)Boldi, P\. & Vigna, S\.Fibrations of graphs\.*Discrete Math\.*243, 21–66 \(2002\)\.
- \(22\)Grothendieck, A\.Technique de descente et théorémes d’existence en géométrie algébrique, I\. Généralités\. Descente par morphismes fidélement plats\.*Seminaire Bourbaki*190\(1959\-1960\)\.
- \(23\)Morone, F\., Leifer, I\. & Makse, H\. A\.Fibration symmetries uncover the building blocks of biological networks\.*Proc\. Nat\. Acad\. Sci\. USA*117, 8306–8314 \(2020\)\.
- \(24\)Papyan, V\., Han, X\. & Donoho, D\. L\.Prevalence of neural collapse during the terminal phase of deep learning training\.*Proc\. Natl\. Acad\. Sci\. USA*117, 24652–24663 \(2020\)\.
- \(25\)Doimo, D\., Glielmo, A\., Goldt, S\. & Laio, A\.Koyejo, S\.*et al\.*\(eds\)*Redundant representations help generalization in wide neural networks*\.\(eds Koyejo, S\.*et al\.*\)*Advances in Neural Information Processing Systems*, Vol\. 35, 19659–19672 \(Curran Associates, Inc\., 2022\)\.
- \(26\)Kamei, H\. & Cock, P\. J\. A\.Computation of balanced equivalence relations and their lattice for a coupled cell network\.*SIAM J\. Appl\. Dyn\. Syst\.*12, 352–382 \(2013\)\.
- \(27\)Golubitsky, M\. & Stewart, I\.Nonlinear dynamics of networks: the groupoid formalism\.*Bulletin of the American Meteorological Society*43, 305–364 \(2006\)\.
- \(28\)Georgi, H\.*Lie Algebras in Particle Physics: From Isospin to Unified Theories*\(Taylor & Francis, 2000\)\.
- \(29\)Rumelhart, D\. E\., Hinton, G\. E\. & Williams, R\. J\.Learning representations by back\-propagating errors\.*Nature*323, 533–536 \(1986\)\.
- \(30\)Boldi, P\., Leifer, I\. & Makse, H\. A\.Quasifibrations of graphs to find symmetries and reconstruct biological networks\.*J\. Stat\. Mech\.: Theor\. Exp\.*2022, 113401 \(2022\)\.
- \(31\)Erdős, P\. & Rényi, A\.Asymmetric graphs\.*Acta Mathematica Academiae Scientiarum Hungaricae*14, 295–315 \(1963\)\.
- \(32\)Chen, F\., Kunin, D\., Yamamura, A\. & Ganguli, S\.Stochastic collapse: How gradient noise attracts SGD dynamics towards simpler subnetworks\.*Advances in Neural Information Processing Systems*36, 35027–35063 \(2023\)\.
- \(33\)Zeiler, M\. D\. & Fergus, R\.*Visualizing and understanding convolutional networks*, 818–833 \(Springer, 2014\)\.
- \(34\)Meng, F\.*et al\.**Pruning filter in filter*, Vol\. 33, 17629–17640 \(Curran Associates, Inc\., 2020\)\.
- \(35\)Vaswani, A\.*et al\.*Attention is all you need\.*arXiv*\(2023\)\.URL[https://arxiv\.org/abs/1706\.03762](https://arxiv.org/abs/1706.03762)\.
- \(36\)van de Ven, G\. M\., Tuytelaars, T\. & Tolias, A\. S\.Three types of incremental learning\.*Nature Machine Intelligence*4, 1185–1197 \(2022\)\.
- \(37\)Anderson, P\. W\.More is different: Broken symmetry and the nature of the hierarchical structure of science\.*Science*177, 393–396 \(1972\)\.
- \(38\)Chebykin, A\., Dushatskiy, A\., Alderliesten, T\. & Bosman, P\. A\. N\.Shrink\-Perturb improves architecture mixing during population based training for neural Architecture Search\.*arXiv*\(2023\)\.URL[https://arxiv\.org/abs/2307\.15621](https://arxiv.org/abs/2307.15621)\.
- \(39\)Herzog, D\. & Mattingly, J\.Noise\-induced stabilization of planar flows I\.*Electronic Journal of Probability*20, 1–43 \(2015\)\.
- \(40\)Srivastava, N\., Hinton, G\., Krizhevsky, A\., Sutskever, I\. & Salakhutdinov, R\.Dropout: A simple way to prevent neural Networks from Overfitting\.*Journal of Machine Learning Research*15, 1929–1958 \(2014\)\.
- \(41\)He, K\., Zhang, X\., Ren, S\. & Sun, J\.*Deep residual learning for image recognition*\(2016\)\.
- \(42\)Wang, Z\., Li, C\. & Wang, X\.*Convolutional neural network pruning with structural redundancy reduction*, 14913–14922 \(2021\)\.
- \(43\)Zbontar, J\., Jing, L\., Misra, I\., LeCun, Y\. & Deny, S\.*Barlow twins: Self\-supervised learning via redundancy reduction*, 12310–12320 \(PMLR, 2021\)\.
- \(44\)Qiu, J\., Chen, C\., Liu, S\., Zhang, H\. Y\. & Zeng, B\.Slimconv: Reducing channel redundancy in convolutional neural networks by features recombining\.*IEEE Transactions on Image Processing*30, 6434–6445 \(2021\)\.
- \(45\)Sharma, P\., Ash, J\. T\. & Misra, D\.The truth is in there: Improving reasoning in language models with layer\-selective rank reduction\.*arXiv preprint arXiv:2312\.13558*\(2023\)\.
- \(46\)Tomut, A\.*et al\.*Compactifai: extreme compression of large language models using quantum\-inspired tensor networks\.*arXiv preprint arXiv:2401\.14109*\(2024\)\.
- \(47\)Boldi, P\., Lonati, V\., Santini, M\. & Vigna, S\.Graph fibrations, graph isomorphism, and pagerank\.*RAIRO\. Theoretical Informatics and Applications*40, 227–253 \(2006\)\.
- \(48\)DeVille, L\. & Lerman, E\.Modular dynamical systems on networks\.*J\. Eur\. Math\. Soc\.*17, 2977–3013 \(2015\)\.
- \(49\)Nijholt, E\., Rink, B\. & Sanders, J\.Graph fibrations and symmetries of network dynamics\.*Diff\. Equ\.*261, 4861–4896 \(2016\)\.
- \(50\)Aguiar, M\. A\., Dias, A\. P\. S\. & Ferreira, F\.Patterns of synchrony for feed\-forward and auto\-regulation feed\-forward neural networks\.*Chaos: An Interdisciplinary Journal of Nonlinear Science*27\(2017\)\.
- \(51\)Leifer, I\., Phillips, D\., Sorrentino, F\. & Makse, H\. A\.Symmetry\-driven network reconstruction through pseudobalanced coloring optimization\.*J\. Stat\. Mech\.: Theor\. Exp\.*2022, 073403 \(2022\)\.
- \(52\)Park, H\. & Friston, K\.Structural and functional brain networks: from connections to cognition\.*Science*342, 1238411 \(2013\)\.
- \(53\)Schulman, J\., Wolski, F\., Dhariwal, P\., Radford, A\. & Klimov, O\.Proximal policy optimization algorithms\.*arXiv*\(2017\)\.URL[https://arxiv\.org/abs/1707\.06347](https://arxiv.org/abs/1707.06347)\.
- \(54\)Norris, N\.Universal covers of graphs: Isomorphism to depth n\-1 implies isomorphism to all depths\.*Discrete Applied Mathematics*56, 61–74 \(1995\)\.
- \(55\)Wright, S\. J\.Coordinate descent algorithms\.*Mathematical Programming*151, 3–34 \(2015\)\.
- \(56\)Ayinde, B\. O\., Inanc, T\. & Zurada, J\. M\.Redundant feature pruning for accelerated inference in deep neural networks\.*Neural Networks*118, 148–158 \(2019\)\.
- \(57\)Liu, K\., Suganuma, M\. & Okatani, T\.Bridging the gap from asymmetry tricks to decorrelation principles in non\-contrastive self\-supervised learning\.*Advances in neural information processing systems*35, 19824–19835 \(2022\)\.
- \(58\)Parga, N\. & Virasoro, M\. A\.The ultrametric organization of memories in a neural network\.*J\. Phys\. \(Paris\)*47, 1857–1864 \(1986\)\.
- \(59\)Mézard, M\., Parisi, G\. & Virasoro, M\. A\.*Spin Glass Theory and Beyond*Vol\. 9 \(World Scientific Lecture Notes in Physics, 1987\)\.
- \(60\)Jaderberg, M\., Vedaldi, A\. & Zisserman, A\.Speeding up convolutional neural networks with low rank expansions\.*arXiv preprint arXiv:1405\.3866*\(2014\)\.
- \(61\)Denton, E\. L\., Zaremba, W\., Bruna, J\., LeCun, Y\. & Fergus, R\.Exploiting linear structure within convolutional networks for efficient evaluation\.*Advances in neural information processing systems*27\(2014\)\.
- \(62\)Novikov, A\., Podoprikhin, D\., Osokin, A\. & Vetrov, D\. P\.Tensorizing neural networks\.*Advances in neural information processing systems*28\(2015\)\.
- \(63\)Rokh, B\., Azarpeyvand, A\. & Khanteymoori, A\.A comprehensive survey on model quantization for deep neural networks in image classification\.*ACM Transactions on Intelligent Systems and Technology*14, 1–50 \(2023\)\.