Observable- and Positional-Encoding-Dependent Symmetry Readout from Neural Network Weights

# Observable- and Positional-Encoding-Dependent Symmetry Readout from Neural Network Weights
Source: [https://arxiv.org/html/2607.03108](https://arxiv.org/html/2607.03108)
Naoya Chiba D3 Center, The Unversity of Osaka chiba@nchiba\.net &Satoshi Sugiyama D3 Center, The Unversity of Osaka sugiyama\.satoshi\.work@gmail\.com &Yuki Uranishi D3 Center, The Unversity of Osaka yuki\.uranishi\.cmc@osaka\-u\.ac\.jp

###### Abstract

Post\-hoc analysis of trained neural network weights often seeks to recover geometric structure directly from the parameters\. We show that, for positional\-encoding\-equipped neural fields, the symmetry visible from weights is not the true symmetry group itself, but an observable symmetry set determined by the trained parameters, the positional encoding \(PE\), and readout observable\. We formulate this dependence through an exact observability hierarchy,Gobsexact⊆Gliftexact​\(ϕ\)∩GtrueG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\\subseteq G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\(\\phi\)\\cap G\_\{\\mathrm\{true\}\}, whereGliftexact​\(ϕ\)G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\(\\phi\)is the set of input transformations that the PE can exactly lift to the feature space\. The hierarchy implies that even when a target function has a geometric symmetry, that symmetry may be structurally invisible to weight\-level observables if the PE does not represent the corresponding transformation\. We test this prediction using MLPs trained on two\-dimensional signed distance functions with multiple shape symmetry groups, positional encodings, and Gram\-based observables\. The results show a consistent PE\-dependent pattern: DyadicAxisPE supportsD4D\_\{4\}\-sensitive readout but structurally suppressesD3D\_\{3\}rotations, TriAxisPE yields lowerD3D\_\{3\}/D6D\_\{6\}readout scores under the tested Gram observables by replacing coordinate axes with three 120\-degree\-separated axes, and random Fourier features mainly exhibit aπ\\pi\-rotation response under these readouts\. These findings show that PE design affects not only approximation behavior but also which structures are accessible to post\-hoc weight\-level readouts\. This provides a basis for a principled observable\-dependent symmetry readout\.

## 1Introduction

Equivariant neural networks\[[3](https://arxiv.org/html/2607.03108#bib.bib15),[2](https://arxiv.org/html/2607.03108#bib.bib50)\]have been successful in embedding symmetry groups into the architecture a priori\. This paper studies the reverse question: can geometric symmetries be read out post hoc from the weights of a model that was not built to be equivariant? When tackling this question, this expectation fails for a structural reason: the naive expectation “the true symmetry groupGtrueG\_\{\\mathrm\{true\}\}can be recovered from the weights” does not hold\. What is visible from the weights is theobservable symmetry setGobs​\(θ;ϕ,Φ\)G\_\{\\mathrm\{obs\}\}\(\\theta;\\phi,\\Phi\), and what can be seen is determined not only by the true symmetry but also by the combination of the input representation \(PE,ϕ\\phi\) and observable \(Φ\\Phi\)\. For example, axis\-separable PE can exactly lift up toD4D\_\{4\}but does not provide an exact structural readout forD3D\_\{3\}, whereas generic Random Fourier Features \(RFF\) exactly lift only theπ\\pi\-rotation among the tested rotations, under the general\-position assumption\. That is,*not all symmetries are equally observable from weight\-level structural readouts*, and the exact\-liftability bound of the PE constrains the upper limit of the symmetry classes that can be exposed by the tested structural observables\.

Crucially, this work does not aim to discover unknown groups freely; instead, it is a post\-hoc probing \(transform\-conditioned readout\) that provides candidate transform families \(D4D\_\{4\}, rotation sweep\) and evaluates the response of each observable to every transform\. Building on this recognition, we propose a framework for observable\-dependent symmetry readout centered on the two\-factor\(ϕ,Φ\)\(\\phi,\\Phi\)structure ofGobsG\_\{\\mathrm\{obs\}\}and establish Proposition 1 \(Exact Observability Hierarchy:Gobsexact⊆Gliftexact∩GtrueG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\\subseteq G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\\cap G\_\{\\mathrm\{true\}\}\) as an idealized exact\-regime upper bound on post\-hoc structural detection\. We subsequently conduct empirical tests to test whether the empirical readout profiles are consistent with this hierarchy, specifically examining how the selection of PE andΦ\\Phiinfluences which elements ofGtrueG\_\{\\mathrm\{true\}\}are mirrored inGobsG\_\{\\mathrm\{obs\}\}\. This is achieved through systematic experiments involving PE\-equipped MLPs trained on 2D SDFs, encompassing various shape groups, multiple PEs, and different observables\.

## 2Related Work

This work relates to equivariant architectures, symmetry discovery, and implicit neural representations, but focuses on what is observable from trained weights under a given PE and observable readout\.

##### Equivariant architectures and weight\-space models\.

Equivariant neural networks embed group structure into the architecture\[[3](https://arxiv.org/html/2607.03108#bib.bib15),[21](https://arxiv.org/html/2607.03108#bib.bib38),[16](https://arxiv.org/html/2607.03108#bib.bib18)\]and have been systematized as geometric deep learning\[[2](https://arxiv.org/html/2607.03108#bib.bib50)\]\. These methods impose symmetry a priori, whereas our goal is to read out symmetry post hoc from models that do not explicitly encode the target groups\. DWSNets\[[13](https://arxiv.org/html/2607.03108#bib.bib16)\]are closest to our setting because they operate directly on neural network weights and predict properties of implicit neural representations from weight space\. However, they target downstream prediction, such as classification or regression, rather than which geometric symmetry groups are structurally observable from trained weights\. Our work instead studies observable\-dependent symmetry readout from the trained weights of PE\-equipped MLPs\.

##### Symmetry discovery from data and models\.

LieGG\[[12](https://arxiv.org/html/2607.03108#bib.bib35)\]extracts Lie algebra generators from gradient information, LieSD\[[8](https://arxiv.org/html/2607.03108#bib.bib52)\]extends related ideas to symmetry discovery and scoring, and L\-conv\[[5](https://arxiv.org/html/2607.03108#bib.bib29)\]embeds Lie algebra structure directly into the network\. These methods share a post\-hoc or discovery\-oriented motivation with our work, but typically rely on data, gradients, or model responses\. In contrast, our main observable, the weight\-prefix Gram, provides a data\-free weight\-level readout\. They also tend to treat symmetry as a property of the data or learned function, whereas our framework emphasizes observable dependence: even for the same trained model, the visible symmetry can change with the choice of observableΦ\\Phiand PEϕ\\phi\. This leads to the observable symmetry setGobs​\(θ;ϕ,Φ\)G\_\{\\mathrm\{obs\}\}\(\\theta;\\phi,\\Phi\)rather than a single recovered group\.

##### Parameter\-space and weight\-space symmetry\.

Neural network weights contain parameter\-space symmetries, such as neuron permutations and positive rescalings in ReLU networks\[[7](https://arxiv.org/html/2607.03108#bib.bib30),[23](https://arxiv.org/html/2607.03108#bib.bib49),[24](https://arxiv.org/html/2607.03108#bib.bib54)\], that do not change the represented function\. This motivates observables that suppress irrelevant gauge\-like degrees of freedom while retaining geometric information relevant to the input\-space transformations under consideration\. The Gram\-type observables used here follow this principle, sinceW⊤​WW^\{\\top\}Wis invariant to neuron permutations, though not to all residual scaling degrees of freedom\.

##### INR, positional encodings, and weight representations\.

Implicit neural representations commonly use Fourier features\[[20](https://arxiv.org/html/2607.03108#bib.bib37)\]or periodic activations\[[19](https://arxiv.org/html/2607.03108#bib.bib36)\]to represent high\-frequency signals\. GRAPE\[[22](https://arxiv.org/html/2607.03108#bib.bib12)\]provides a representation\-theoretic classification of positional encodings, identifying which group actions can be represented by a PE\. Our exact\-liftability bound uses this PE\-level classification as an upper\-bound condition for post\-hoc weight\-level readout: a symmetry must first be liftable by the PE before it can appear as an exact structural symmetry of weight observables\. Separately, Functa\[[6](https://arxiv.org/html/2607.03108#bib.bib21)\]and inr2vec\[[4](https://arxiv.org/html/2607.03108#bib.bib3)\]show that useful information can be extracted from weights or latent representations of implicit neural fields\. We share the premise that neural field parameters encode structure, but focus specifically on which geometric symmetries are observable from these parameters and how this depends on the pair\(ϕ,Φ\)\(\\phi,\\Phi\)\.

## 3Theory and Methods

### 3\.1Problem Setting

This work introduces a framework for observable\-dependent symmetry readout, systematically characterizing which geometric symmetries areobservablefrom a trained neural network\. LetX=ℝ2X=\\mathbb\{R\}^\{2\}be the input space,H=ℝdH=\\mathbb\{R\}^\{d\}\(whereddis the PE output dimension; e\.g\.,d=48d=48for all structured PEs \(DyadicAxisPE:4​K=484K\{=\}48withK=12K\{=\}12; TriAxisPE:6​K=486K\{=\}48withK=8K\{=\}8; RFF:2​n=482n\{=\}48withn=24n\{=\}24\)\) be the output space \(feature space\) of the positional encoding \(PE\),ϕ:X→H\\phi:X\\to Hbe the PE, andFθ:H→ℝF\_\{\\theta\}:H\\to\\mathbb\{R\}be the subsequent neural network\. The overall model is given by

fθ​\(x\)=Fθ​\(ϕ​\(x\)\)f\_\{\\theta\}\(x\)=F\_\{\\theta\}\(\\phi\(x\)\)whereθ\\thetadenotes all the trained parameters\.

The geometric symmetries of interest are described by a transformation groupGGthat acts on the input spaceXX\. Eachg∈Gg\\in Gacts on a pointx∈Xx\\in X, and on a functionffvia

\(g⋅f\)​\(x\)=f​\(g−1​x\)\.\(g\\cdot f\)\(x\)=f\(g^\{\-1\}x\)\.An object is said to be symmetric with respect to a groupGtrueG\_\{\\mathrm\{true\}\}when the corresponding true functionf∗f\_\{\*\}satisfies

f∗​\(g−1​x\)=f∗​\(x\)for all​g∈Gtrue\.f\_\{\*\}\(g^\{\-1\}\\,x\)=f\_\{\*\}\(x\)\\quad\\text\{for all \}g\\in G\_\{\\mathrm\{true\}\}\.
Naively, one would like to directly recoverGtrueG\_\{\\mathrm\{true\}\}from the trained weightsθ\\theta\. In practice, however, the weights themselves do not directly retain geometric groups in the input space; symmetry manifests indirectly through positional encoding, the network’s internal representations, and the choice of observable\. Therefore, this work first provides a theoretical formulation of what symmetry is visible from the weights and then organizes what is recoverable and what is in principle difficult to observe\.

### 3\.2Positional Encoding as a Representation

Positional encoding \(PE\) is a mapping that projects low\-dimensional input coordinates into a higher\-dimensional feature space using trigonometric functions or similar bases, and is widely used in implicit neural representations \(INRs\) such as Neural Radiance Fields \(NeRF\)\[[11](https://arxiv.org/html/2607.03108#bib.bib41),[20](https://arxiv.org/html/2607.03108#bib.bib37),[19](https://arxiv.org/html/2607.03108#bib.bib36)\]\. Plain MLPs tend to preferentially learn low\-frequency functions \(spectral bias\[[14](https://arxiv.org/html/2607.03108#bib.bib17)\]\), and PE mitigates this bias by lifting inputs into a higher\-dimensional space, enabling the learning of functions with high\-frequency content\.

In this work, we view PEϕ\\phinot merely as preprocessing, but as a feature map that lifts geometric transformations to the feature space\[[22](https://arxiv.org/html/2607.03108#bib.bib12)\]\. When a geometric transformationgg\(rotation, reflection, translation, etc\.\) is applied to an inputxx, passingg​xgxthrough the PE yieldsϕ​\(g​x\)\\phi\(gx\)\. A natural question is whetherϕ​\(g​x\)\\phi\(gx\)can be computed directly fromϕ​\(x\)\\phi\(x\), that is, whether the geometric transformation can be represented as an operation in the feature space\.

We focus on the case whereϕ​\(g​x\)\\phi\(gx\)is expressible as a linear transformation ofϕ​\(x\)\\phi\(x\)\. That is, there exists a linear mapρ​\(g\)\\rho\(g\)111In general, PEs with bias terms may induce affine actions, but for all PEs considered in this paper \(DyadicAxisPE, TriAxisPE, and RFF\),ρ​\(g\)\\rho\(g\)is realized as a linear map\.onHHsuch that

ϕ​\(g​x\)=ρ​\(g\)​ϕ​\(x\),\\phi\(g\\,x\)=\\rho\(g\)\\,\\phi\(x\),ϕ\\phiis said to beGG\-equivariant with respect toρ\\rho\. Here,ρ\\rhois the group representation that lifts the geometric action on the input space to the feature space\. The linearity of this lift is essential because each MLP layer is a composition of a linear transformation \(weight matrix\) and nonlinear activation\. A linear lift on the PE allows the effect of geometric transformations to be tracked through algebraic operations on weight matrices, providing a foundation for symmetry readout from weights\.

From this perspective, the positional encoding determines which input transformations can be tracked exactly as linear feature\-space actions\. That is, ifϕ\\phiisGG\-equivariant for a given transformation group, the group action can be explicitly tracked in feature space\. Conversely, for transformations for which no such lift exists, even if the symmetry is natural in the input space, it does not appear as an exact representation within the model\.

Therefore, the problem of reading out symmetry from trained weights depends first on how well the positional encoding retains the group actions\. This is the theoretical starting point of this work\.

We define the PEs used in this work and consolidate their exact liftability in Lemma[1](https://arxiv.org/html/2607.03108#Thmlemma1)\.

##### DyadicAxisPE\.

An axis\-separated PE with dyadic frequen,ρ​\(g\)\\rho\(g\)is realized as a linear map\. cies\[[20](https://arxiv.org/html/2607.03108#bib.bib37)\], with an output dimension4​K4Kfor an octave countKKis given by

ϕDyadic​\(x\)=\[\{sin⁡\(2k​π​x1\),cos⁡\(2k​π​x1\),sin⁡\(2k​π​x2\),cos⁡\(2k​π​x2\)\}k=0K−1\]\.\\phi\_\{\\mathrm\{Dyadic\}\}\(x\)=\\bigl\[\\\{\\sin\(2^\{k\}\\pi x\_\{1\}\),\\,\\cos\(2^\{k\}\\pi x\_\{1\}\),\\,\\sin\(2^\{k\}\\pi x\_\{2\}\),\\,\\cos\(2^\{k\}\\pi x\_\{2\}\)\\\}\_\{k=0\}^\{K\-1\}\\bigr\]\.

##### Random Fourier Features \(RFF\)\.

An isotropic PE using randomly sampled frequency vectorsωi∈ℝ2\\omega\_\{i\}\\in\\mathbb\{R\}^\{2\}and biasesbi∈ℝb\_\{i\}\\in\\mathbb\{R\}\(i=1,…,ni=1,\\dots,n\)\[[15](https://arxiv.org/html/2607.03108#bib.bib25),[20](https://arxiv.org/html/2607.03108#bib.bib37)\]with an output dimension2​n2nis defined as

ϕRFF​\(x\)=\[sin⁡\(ω1⊤​x\+b1\),cos⁡\(ω1⊤​x\+b1\),…,sin⁡\(ωn⊤​x\+bn\),cos⁡\(ωn⊤​x\+bn\)\]\.\\phi\_\{\\mathrm\{RFF\}\}\(x\)=\\bigl\[\\sin\(\\omega\_\{1\}^\{\\top\}x\+b\_\{1\}\),\\,\\cos\(\\omega\_\{1\}^\{\\top\}x\+b\_\{1\}\),\\,\\dots,\\,\\sin\(\\omega\_\{n\}^\{\\top\}x\+b\_\{n\}\),\\,\\cos\(\\omega\_\{n\}^\{\\top\}x\+b\_\{n\}\)\\bigr\]\.

##### Tri\-Axis Dyadic PE \(TriAxisPE\)\.

Extension of DyadicAxisPE by replacing the two coordinate axes with three axes separated by2​π/32\\pi/3\. Letv0=\(1,0\)⊤v\_\{0\}=\(1,0\)^\{\\top\},v1=R2​π/3​v0v\_\{1\}=R\_\{2\\pi/3\}\\,v\_\{0\},v2=R4​π/3​v0v\_\{2\}=R\_\{4\\pi/3\}\\,v\_\{0\}\. The encoding is:

ϕtri​\(x\)=\[\{sin⁡\(2k​π​vj⊤​x\),cos⁡\(2k​π​vj⊤​x\)\}j=0,…,2;k=0,…,K−1\],\\phi\_\{\\mathrm\{tri\}\}\(x\)=\\bigl\[\\\{\\sin\(2^\{k\}\\pi\\,v\_\{j\}^\{\\top\}x\),\\,\\cos\(2^\{k\}\\pi\\,v\_\{j\}^\{\\top\}x\)\\\}\_\{j=0,\\ldots,2;\\,k=0,\\ldots,K\-1\}\\bigr\],with an output dimension6​K6K\.

###### Lemma 1\(Exact rotation/reflection liftability of structured PEs\)\.

For the three PEs defined above \( assuming the RFF frequency set\{ωi\}\\\{\\omega\_\{i\}\\\}is in general position, i\.e\., it is not closed under the tested rotations or reflections except for sign reversal\):

1. \(i\)ϕDyadic\\phi\_\{\\mathrm\{Dyadic\}\}exactly liftsD4D\_\{4\}\(generated byπ/2\\pi/2rotation and axis reflections\); transformations outsideD4D\_\{4\}\(in particularπ/3\\pi/3and2​π/32\\pi/3rotations\) are not exactly lifted\.
2. \(ii\)ϕtri\\phi\_\{\\mathrm\{tri\}\}exactly liftsD6D\_\{6\}\(containingD3D\_\{3\}\); transformations outsideD6D\_\{6\}\(in particularπ/2\\pi/2rotation\) are not exactly lifted\.
3. \(iii\)ϕRFF\\phi\_\{\\mathrm\{RFF\}\}exactly liftsZ2Z\_\{2\}\(π\\pirotation\); transformations outsideZ2Z\_\{2\}\(rotations at other angles and reflections\) are not exactly lifted\.

The proof is provided in Appendix[A](https://arxiv.org/html/2607.03108#A1)\.

### 3\.3Functional Symmetry and Observable Symmetry

The true symmetry of an object is inherently defined as the invariance of the functionf∗f\_\{\*\}:f∗​\(g​x\)=f∗​\(x\)f\_\{\*\}\(gx\)=f\_\{\*\}\(x\)\. The most direct way to verify this with a trained modelfθf\_\{\\theta\}is dense forward computation, which evaluatesfθ​\(g​x\)≈fθ​\(x\)f\_\{\\theta\}\(gx\)\\approx f\_\{\\theta\}\(x\)at many input points\. However, this is an input–output\-level verification of how wellfθf\_\{\\theta\}approximatesf∗f\_\{\*\}and does not reveal how symmetry is encoded inside the model\. The goal of this work is not such input–output verification, but rather reading out symmetry from the weights and internal representations of the trained model \(Figure[1](https://arxiv.org/html/2607.03108#S3.F1)\)\.

Accordingly, we write the observable computed from the trained model asΦ​\(θ\)\\Phi\(\\theta\)\.Φ\\Phimay be a quantity in the weight space or may use internal representations or limited activation information as needed\. The key point is thatΦ\\Phiserves as a window for summarizing the internal structure of the model and measuring its stability under geometric transformations\.

For a transformationg∈Gg\\in G, we define the corresponding actionTgT\_\{g\}on the model’s side\. Thegg\-transformed modelTg​θT\_\{g\}\\,\\thetais the parameter representing the shapeg⋅Sg\\cdot Sobtained by applyingggto the shapeSSthatθ\\thetarepresents, i\.e\.,

fTg​θ​\(x\)=fθ​\(g−1​x\)\.f\_\{T\_\{g\}\\,\\theta\}\(x\)=f\_\{\\theta\}\(g^\{\-1\}x\)\.Wheng∈Gliftexactg\\in G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}, we haveϕ​\(g−1​x\)=ρ​\(g−1\)​ϕ​\(x\)\\phi\(g^\{\-1\}x\)=\\rho\(g^\{\-1\}\)\\,\\phi\(x\), so

fTg​θ​\(x\)=Fθ​\(ρ​\(g−1\)​ϕ​\(x\)\)\.f\_\{T\_\{g\}\\,\\theta\}\(x\)=F\_\{\\theta\}\\\!\\bigl\(\\rho\(g^\{\-1\}\)\\,\\phi\(x\)\\bigr\)\.This definition makesTTa genuine left action on the parameter space \(Tg​h=Tg​ThT\_\{gh\}=T\_\{g\}T\_\{h\}\), consistent with the definition ofρ\\rho\.

Sinceρ\\rhoacts only on PE outputs,Tg​θT\_\{g\}\\thetais realized simply by modifying the first\-layer weight \(which receives the PE output\) asW0→W0​ρ​\(g−1\)W\_\{0\}\\to W\_\{0\}\\,\\rho\(g^\{\-1\}\), leaving subsequent layers and all biases unchanged; this realization holds exactly regardless of activation functions\. Consequently, any quantity built fromW0W\_\{0\}inherits theρ\\rho\-action\. As the simplest example, taking the Gram matrixG0=W0⊤​W0G\_\{0\}=W\_\{0\}^\{\\top\}W\_\{0\}at the PE output as the observable gives

G0​\(Tg​θ\)=ρ​\(g−1\)⊤​G0​ρ​\(g−1\)G\_\{0\}\(T\_\{g\}\\,\\theta\)=\\rho\(g^\{\-1\}\)^\{\\top\}\\,G\_\{0\}\\,\\rho\(g^\{\-1\}\)222In experiments, we use the homogeneous extensionG~0=\[G0W0⊤​b0b0⊤​W0‖b0‖2\]\\tilde\{G\}\_\{0\}=\\bigl\[\\begin\{smallmatrix\}G\_\{0\}&W\_\{0\}^\{\\top\}b\_\{0\}\\\\ b\_\{0\}^\{\\top\}W\_\{0\}&\\\|b\_\{0\}\\\|^\{2\}\\end\{smallmatrix\}\\bigr\]with the correspondingΠ~g=\[ρ​\(g−1\)001\]\\tilde\{\\Pi\}\_\{g\}=\\bigl\[\\begin\{smallmatrix\}\\rho\(g^\{\-1\}\)&0\\\\ 0&1\\end\{smallmatrix\}\\bigr\]to account for bias terms \(Appendix[E](https://arxiv.org/html/2607.03108#A5)\)\.

\. This is precisely the prefix GramL0L\_\{0\}used in our main experiments\. Deeper prefixes \(Pl=Wl​⋯​W0P\_\{l\}=W\_\{l\}\\cdots W\_\{0\}with prefix GramGl=Pl⊤​PlG\_\{l\}=P\_\{l\}^\{\\top\}P\_\{l\}\) and the effective weightWeff=WL​⋯​W0W\_\{\\mathrm\{eff\}\}=W\_\{L\}\\cdots W\_\{0\}\(the end\-to\-end linear map when activations are ignored, a quantity extensively studied in deep linear network analysis\[[17](https://arxiv.org/html/2607.03108#bib.bib4),[1](https://arxiv.org/html/2607.03108#bib.bib26)\]\) satisfyPl→Pl​ρ​\(g−1\)P\_\{l\}\\to P\_\{l\}\\,\\rho\(g^\{\-1\}\)andWeff→Weff​ρ​\(g−1\)W\_\{\\mathrm\{eff\}\}\\to W\_\{\\mathrm\{eff\}\}\\,\\rho\(g^\{\-1\}\), inducing the sameρ\\rho\-sandwich structure as the shallow prefix\. A similar structure is induced viaρ\\rhofor general observable quantities\. SinceGGis closed under inversion, the detection setGobs=\{g∣Φ​\(Tg​θ\)=Φ​\(θ\)\}G\_\{\\mathrm\{obs\}\}=\\\{g\\mid\\Phi\(T\_\{g\}\\theta\)=\\Phi\(\\theta\)\\\}is invariant underg↔g−1g\\leftrightarrow g^\{\-1\}\.

In this framework, the group action is handled in the feature space viaρ\\rho, which differs from dense forward \(fθ​\(g−1​x\)≈fθ​\(x\)f\_\{\\theta\}\(g^\{\-1\}x\)\\approx f\_\{\\theta\}\(x\)\) that acts on the input space in the space on which the group action is applied\. The universality of MLPs allows training to realize function\-level symmetry \(functional symmetry\) independently of PE liftability, whereas the invariance of observables viaρ\\rho\(structural symmetry\) is constrained by the PE’s algebraic structure\. The subsequentGobs​\(θ;ϕ,Φ,𝒯\)=\{g∈𝒯:Φ​\(Tg​θ\)≈Φ​\(θ\)\}G\_\{\\mathrm\{obs\}\}\(\\theta;\\phi,\\Phi,\\mathcal\{T\}\)=\\\{g\\in\\mathcal\{T\}:\\Phi\(T\_\{g\}\\theta\)\\approx\\Phi\(\\theta\)\\\}and hierarchy are described at the structural level\. The gap between the two notions and the dual role of the PE in training and observation are detailed in Appendix[B](https://arxiv.org/html/2607.03108#A2)\.

###### Definition 1\(Observable Symmetry Set\)\.

The observable symmetry set with respect to trained parametersθ\\theta, PEϕ\\phi, and observableΦ\\Phiis defined as

Gobs​\(θ;ϕ,Φ\)=\{g∈G∣Φ​\(Tg​θ\)≈Φ​\(θ\)\}\.G\_\{\\mathrm\{obs\}\}\(\\theta;\\phi,\\Phi\)=\\\{g\\in G\\mid\\Phi\(T\_\{g\}\\theta\)\\approx\\Phi\(\\theta\)\\\}\.Here,≈\\approxdenotes approximate equality that accounts for numerical error, approximate representations, and training error\.

The above definition serves as a generic placeholder for observable symmetry sets\. In the following, we first define the exact versionGobsexactG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}restricted tog∈Gliftexactg\\in G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}, and then introduce the operational extensionDopD\_\{\\mathrm\{op\}\}that allows approximate lifts forg∉Gliftexactg\\notin G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\. The operational quantification of the approximate equality≈\\approxis formalized as the symmetry scoreS​\(g;θ,Φ\)S\(g;\\,\\theta,\\Phi\)in §3\.5\.

The key point of this definition is to capture symmetry not as “an absolute property that the model possesses,” but as an invariant structure that appears stable through a given observable\. Therefore, in this work, symmetry is generally not a property ofθ\\thetaalone, but a quantity that depends on the triple\(θ,ϕ,Φ\)\(\\theta,\\phi,\\Phi\)\.

![Refer to caption](https://arxiv.org/html/2607.03108v1/x1.png)Figure 1:Overview of the observable\-symmetry readout framework\. Weight\-level readout yields an observable symmetry setGobs​\(θ;ϕ,Φ\)G\_\{\\mathrm\{obs\}\}\(\\theta;\\phi,\\Phi\), notGtrueG\_\{\\mathrm\{true\}\}itself\. The Venn diagram shows the exact\-regime hierarchyGobsexact⊆Gliftexact​\(ϕ\)∩GtrueG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\\subseteq G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\(\\phi\)\\cap G\_\{\\mathrm\{true\}\}; the dashed region indicates the operational regime based on approximate lifts\.
### 3\.4Hierarchy of Observable Symmetry

Observable symmetry has three levels \(Figure[1](https://arxiv.org/html/2607.03108#S3.F1)\)\. \(1\) The true symmetry groupGtrueG\_\{\\mathrm\{true\}\}of the object; \(2\) The PE\-liftable groupGliftexact​\(ϕ\)=\{g∈G∣ϕ​\(g​x\)=ρ​\(g\)​ϕ​\(x\)\}G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\(\\phi\)=\\\{g\\in G\\mid\\phi\(gx\)=\\rho\(g\)\\,\\phi\(x\)\\\}is determined solely by PE’s algebraic properties of PE\. \(3\) The observable symmetry setGobsexact​\(θ;ϕ,Φ\)⊆Gliftexact∩GtrueG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\(\\theta;\\phi,\\Phi\)\\subseteq G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\\cap G\_\{\\mathrm\{true\}\}, which is the symmetry that is visible through the chosen observableΦ\\Phi\. This nested structure forms the theoretical backbone of the study\.

###### Definition 2\(Symmetry\-sufficient observable\)\.

An observableΦ\\Phiis said to besymmetry\-sufficientif for anyg∈Gliftexactg\\in G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\},

Φ​\(Tg​θ\)=Φ​\(θ\)⟹fθ​\(g−1​x\)=fθ​\(x\)for all​x∈X\.\\Phi\(T\_\{g\}\\,\\theta\)=\\Phi\(\\theta\)\\;\\Longrightarrow\\;f\_\{\\theta\}\(g^\{\-1\}\\,x\)=f\_\{\\theta\}\(x\)\\quad\\text\{for all \}x\\in X\.Invariance at the observable level implies invariance at the functional level\.

###### Proposition 1\(Exact Observability Hierarchy\)\.

Iffθf\_\{\\theta\}has sufficiently converged tof∗f\_\{\*\}\(supx∈X\|fθ​\(x\)−f∗​\(x\)\|≤δ,δ→0\\sup\_\{x\\in X\}\|f\_\{\\theta\}\(x\)\-f\_\{\*\}\(x\)\|\\leq\\delta,\\;\\delta\\to 0\), andΦ\\Phiis compatible with the inducedTgT\_\{g\}action and symmetry\-sufficient \(Definition[2](https://arxiv.org/html/2607.03108#Thmdefinition2)\), then

Gobsexact​\(θ;ϕ,Φ\)=\{g∈Gliftexact∣Φ​\(Tg​θ\)=Φ​\(θ\)\}G\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\(\\theta;\\,\\phi,\\Phi\)\\;=\\;\\\{g\\in G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\\mid\\Phi\(T\_\{g\}\\,\\theta\)=\\Phi\(\\theta\)\\\}satisfiesGobsexact​\(θ;ϕ,Φ\)⊆Gliftexact​\(ϕ\)∩GtrueG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\(\\theta;\\phi,\\Phi\)\\subseteq G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\(\\phi\)\\cap G\_\{\\mathrm\{true\}\}\.

The proof is provided in Appendix[A](https://arxiv.org/html/2607.03108#A1)\. The experiments test the consistency of this hierarchy\.

This hierarchy cleanly separates three cases: symmetry does not exist, symmetry exists in the representation but is not observed, and symmetry is not lifted into the representation\. Training convergence also affectsGobsG\_\{\\mathrm\{obs\}\}\(Appendix[F](https://arxiv.org/html/2607.03108#A6)\); the main results isolate the PE andΦ\\Phieffects while holding this factor fixed\.

### 3\.5Quantification of Symmetry: Exact Identification and Operational Detection

Thesymmetry scoreof transformationggwith respect to observableΦ\\Phiis

S​\(g;θ,Φ\)=‖Φ​\(Tg​θ\)−Φ​\(θ\)‖F‖Φ​\(θ\)‖FS\(g;\\,\\theta,\\Phi\)=\\frac\{\\\|\\Phi\(T\_\{g\}\\,\\theta\)\-\\Phi\(\\theta\)\\\|\_\{F\}\}\{\\\|\\Phi\(\\theta\)\\\|\_\{F\}\}where∥⋅∥F\\\|\\cdot\\\|\_\{F\}denotes the Frobenius norm\.S=0S=0signifies a perfect symmetry\. In the experiments, instead of testingS=0S=0directly, we used the thresholded setGobsε=\{g∈Gliftexact∣S​\(g;θ,Φ\)<ε\}G\_\{\\mathrm\{obs\}\}^\{\\varepsilon\}=\\\{g\\in G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\\mid S\(g;\\,\\theta,\\Phi\)<\\varepsilon\\\}\(forε\>0\\varepsilon\>0,Gobsε⊇GobsexactG\_\{\\mathrm\{obs\}\}^\{\\varepsilon\}\\supseteq G\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\)\.

##### PE\-level Procrustes residual\.

Whether PEϕ\\phiadmits an exact lift for transformationggis determined by the orthogonal Procrustes residual\[[18](https://arxiv.org/html/2607.03108#bib.bib48)\]

rP​\(g\)=minQ∈O​\(d\)⁡‖ϕ​\(g​X\)−Q​ϕ​\(X\)‖F‖ϕ​\(g​X\)‖F\.r\_\{P\}\(g\)=\\min\_\{Q\\in O\(d\)\}\\frac\{\\\|\\phi\(gX\)\-Q\\,\\phi\(X\)\\\|\_\{F\}\}\{\\\|\\phi\(gX\)\\\|\_\{F\}\}\.For all PEs in this work, exact lifts were realized as orthogonal transformations \(permutation matrices or block rotation matrices\), making orthogonal Procrustes the appropriate estimator\.rP≈0r\_\{P\}\\approx 0\(machine precision∼10−14\\sim 10^\{\-14\}\) confirms the existence of an exact lift,rP\>0r\_\{P\}\>0confirms its absence\.

For transformsggoutsideGliftexactG\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}, the exact liftρ​\(g\)\\rho\(g\)does not exist, soΦ​\(Tg​θ\)≈Φ​\(θ\)\\Phi\(T\_\{g\}\\theta\)\\approx\\Phi\(\\theta\)in Definition[1](https://arxiv.org/html/2607.03108#Thmdefinition1)cannot be evaluated as is\. We define the best orthogonal approximation as the solution to the above Procrustes problem as follows:

ρ^​\(g\)=argminQ∈O​\(d\)‖ϕ​\(g​X\)−Q​ϕ​\(X\)‖F\\hat\{\\rho\}\(g\)=\\operatorname\*\{argmin\}\_\{Q\\in O\(d\)\}\\\|\\phi\(gX\)\-Q\\,\\phi\(X\)\\\|\_\{F\}\(numerical procedure in Appendix[G](https://arxiv.org/html/2607.03108#A7)\)\. Sinceρ^​\(g\)=ρ​\(g\)\\hat\{\\rho\}\(g\)=\\rho\(g\)wheng∈Gliftexactg\\in G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}, replacingρ\\rhobyρ^\\hat\{\\rho\}defines an operational approximation outside the exact regime\. The operational scoreSop​\(g;θ,Φ\)S\_\{\\mathrm\{op\}\}\(g;\\,\\theta,\\Phi\)is defined as the symmetry score withρ\\rhoreplaced byρ^\\hat\{\\rho\}in the computation ofΦ​\(Tg​θ\)\\Phi\(T\_\{g\}\\,\\theta\), and the operational detection set isDop​\(θ;Φ,ε\)=\{g∈𝒯∣Sop​\(g;θ,Φ\)<ε\}D\_\{\\mathrm\{op\}\}\(\\theta;\\,\\Phi,\\,\\varepsilon\)=\\\{g\\in\\mathcal\{T\}\\mid S\_\{\\mathrm\{op\}\}\(g;\\,\\theta,\\Phi\)<\\varepsilon\\\}where𝒯\\mathcal\{T\}is the sampled transform family\. The theory section uses the exact regime \(GobsexactG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\), whereas the experiments use the operational regime \(DopD\_\{\\mathrm\{op\}\}\)\. Because the approximation error in the learned weights makesS​\(g\)\>0S\(g\)\>0at the observable level even for exact\-lift transforms \(e\.g\.,S≈0\.2S\\approx 0\.2forD4D\_\{4\}transforms under DyadicAxisPE\+\+prefix GramL0L\_\{0\}\),ε\\varepsilonis used in experiments for basic validation \(absence of false positives\), whereas the primary analysis relies on relative score profiles rather than hard\-threshold detection\.

From the above organization, the experiments address two primary questions: \(i\) how the PE constrains the upper bound of observable symmetry, and \(ii\) within weight\-only observables, how dominant is the choice of PE in determining the readout sensitivity\.

## 4Experiments and Results

### 4\.1Experimental Setup

We extracted geometric symmetries from the post\-training weights and internal representations of MLPs \(depth 5, width 128, ReLU\) trained on 2D signed distance functions \(SDFs\)\. Training protocol: 10,000 points sampled from\[−1,1\]2\[\-1,1\]^\{2\}\(50% uniform, 50% boundary\-enriched\), batch size 256, MSE loss, Adam\[[9](https://arxiv.org/html/2607.03108#bib.bib1)\]\(lr=10−3\\mathrm\{lr\}=10^\{\-3\}\), CosineAnnealingLR\[[10](https://arxiv.org/html/2607.03108#bib.bib2)\], 2,000 epochs, rotation sweep at 72 angles \(5∘5^\{\\circ\}intervals\) with 512 sample points for Procrustes estimation, 5 seeds per condition\. The main text demonstrates representative cases using six shapes; the Appendices assess robustness and variation across all 16 shapes, the full PE×\\timesobservable grid, noise sensitivity, learning dynamics, and translational symmetry\.

##### Target shapes\.

The main experiments used six shapes \(Figure[2](https://arxiv.org/html/2607.03108#S4.F2)\): one representative per discrete symmetry group \(O​\(2\)O\(2\): circle,D6D\_\{6\}: hexagon,D4D\_\{4\}: square,D3D\_\{3\}: equilateral\_triangle,D2D\_\{2\}: ellipse\) plus a trivial\-group baseline random\_noise, covering the partial ordersO​\(2\)⊃D4⊃D2O\(2\)\\supset D\_\{4\}\\supset D\_\{2\}andO​\(2\)⊃D6⊃D3O\(2\)\\supset D\_\{6\}\\supset D\_\{3\}\. Each condition was independently trained using five seeds\. Translation\-symmetric shapes \(p​1​mp1m; stripe\) and within\-group second representatives, totaling 10 additional shapes, are discussed in Appendices[C](https://arxiv.org/html/2607.03108#A3)and[E](https://arxiv.org/html/2607.03108#A5)\(Figure[A8](https://arxiv.org/html/2607.03108#A5.F8)\)\.

![Refer to caption](https://arxiv.org/html/2607.03108v1/x2.png)Figure 2:Zero level sets of the SDFs for the 6 shapes used in the main text \(red: boundary; dark: interiorf∗<0f\_\{\*\}<0\)\.
##### Positional encodings\.

The main comparisons use the three structured PEs:DyadicAxisPE\(K=12K=12,4​K=484K=48dims; exact\-liftability boundD4D\_\{4\}\),TriAxisPE\(K=8K=8,6​K=486K=48dims; liftsD6D\_\{6\}/D3D\_\{3\}but notD4D\_\{4\}\), andRFF\(n=24n=24,2​n=482n=48dims; exact\-liftability boundZ2Z\_\{2\}\)\. All three were trained independently under the same MLP architecture and training setup and shared the same 48\-dimensional representation, enabling a dimension\-controlled comparison of the encoding structure effects\. The RFF choicen=24n=24lies safely within the reconstruction saturation regime \(n≥4n\\geq 4\)\.

##### Symmetry score\.

Using the symmetry scoreS​\(g;θ,Φ\)S\(g;\\,\\theta,\\Phi\)and thresholdε\\varepsilon\. We report hard detection,S<εS<\\varepsilon, only for false\-positive analysis; the main figures analyze raw and relative score profiles\. Varyingε\\varepsilonfrom0\.020\.02to0\.100\.10preserves the primary detection/non\-detection patterns \(sensitivity analysis in Appendix[F](https://arxiv.org/html/2607.03108#A6)\)\. At the PE level, Procrustes residuals reaching machine precision \(∼10−14\\sim 10^\{\-14\}\) confirm the existence of exact lifts, whereas the observable\-levelS​\(g\)S\(g\)attained depends onΦ\\Phi\. In what follows,S​\(g\)S\(g\)for the case whereΦ\\Phiis a Gram matrix is abbreviated as the Gram distanceΔ\\Delta, with the observable specified \(e\.g\., prefix GramL0L\_\{0\}Δ\\Delta,WeffW\_\{\\mathrm\{eff\}\}GramΔ\\Delta\)\.

We note that no false positives \(DopD\_\{\\mathrm\{op\}\}exceedingGtrueG\_\{\\mathrm\{true\}\}\) were observed under the conditions of this work; this is empirically verified in Appendix[F](https://arxiv.org/html/2607.03108#A6)\. Below, we verify the exact liftability bound as our primary empirical result\.

### 4\.2PE Exact Liftability and Its Relation to Weight\-Level Readout Responses

The true symmetryGtrueG\_\{\\mathrm\{true\}\}does not appear directly in the weights; it is first restricted to the PE\-representableGliftexactG\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}, and then the observable\-dependentGobsG\_\{\\mathrm\{obs\}\}is obtained\. We fix the observable to prefix GramL0L\_\{0\}and examine how the PE’s exact liftability constrains this upper bound\. prefix GramL0L\_\{0\}is the shallowest layer of the weight\-prefix GramGl=Pl⊤​PlG\_\{l\}=P\_\{l\}^\{\\top\}P\_\{l\}\(Pl=Wl​⋯​W0P\_\{l\}=W\_\{l\}\\cdots W\_\{0\}\); it contains only the PE’s linear transformation and therefore most directly preserves the symmetric structure of PE space \(chosen for mechanistic clarity; systematic comparison withL0L\_\{0\}–L4L\_\{4\}andWeffW\_\{\\mathrm\{eff\}\}Gram, and with other observable choices, is deferred to Appendix[D](https://arxiv.org/html/2607.03108#A4),[E](https://arxiv.org/html/2607.03108#A5)\)\.

#### 4\.2\.1PE\-Level Exact Lift and Its Consequences

![Refer to caption](https://arxiv.org/html/2607.03108v1/x3.png)Figure 3:Prefix GramL0L\_\{0\}scoreΔ\\Deltaover rotation angles for 3 shapes×\\times3 PEs\. DyadicAxisPE showsD4D\_\{4\}\-aligned dips, TriAxis gives lower scores on theD3D\_\{3\}shape, and RFF mainly shows aπ\\pidip\. Shading:±1\\pm 1std over 5 seeds\. See Figure[A3](https://arxiv.org/html/2607.03108#A4.F3)and Appendix[D](https://arxiv.org/html/2607.03108#A4)for extended comparisons\.Table[1](https://arxiv.org/html/2607.03108#S4.T1)reports the Procrustes residualsrP​\(g\)r\_\{P\}\(g\)for the three PEs\. Within the exact\-liftable ranges of Lemma[1](https://arxiv.org/html/2607.03108#Thmlemma1)\(DyadicAxisPE’sD4D\_\{4\}, TriAxisPE’sD6⊃D3D\_\{6\}\\supset D\_\{3\}, RFF’sZ2Z\_\{2\}\),rPr\_\{P\}reaches near\-zero numerical residuals provide evidence for exact lifts, whereas outside these ranges,rP∼10−1r\_\{P\}\\sim 10^\{\-1\}–10010^\{0\}, orders of magnitude larger, consistent with the theoretical liftability bounds\. The translation residuals are reported in Appendix[C](https://arxiv.org/html/2607.03108#A3)\.

Table 1:PE\-level Procrustes residualsrP​\(g\)r\_\{P\}\(g\)\. Values are averaged over five input samples for deterministic PEs and over 25 trials for RFF\. Near\-zero residuals indicate exact lifts; large residuals indicate non\-liftability\.Figure[3](https://arxiv.org/html/2607.03108#S4.F3)shows the angular response of prefix GramL0L\_\{0\}Δ\\Delta\(distinct from the Procrustes residual, it measures invariance of the*trained*shallowest\-layer Gram\)\. DyadicAxisPE and TriAxisPE exhibit complementary dominance onD4D\_\{4\}shapes andD3D\_\{3\}shapes, respectively, while RFF shows a dip atπ\\pibut lacks periodic structure\. The pattern is consistent with our basic position that “the PE’s exact\-liftable group type fixes the visible structure, and shape modulates its strength\.” A six\-group systematic comparison, including shapes such as hexagons \(D6D\_\{6\}\), where both PEs compete via the common subgroupD4∩D6=D2D\_\{4\}\\cap D\_\{6\}=D\_\{2\}, is shown in Figure[A3](https://arxiv.org/html/2607.03108#A4.F3)\. Under the tested prefix GramL0L\_\{0\}, no systematic valley formation was observed outside the exact liftability bound\.

#### 4\.2\.2Suppression ofD3D\_\{3\}Rotation Responses Under DyadicAxisPE

![Refer to caption](https://arxiv.org/html/2607.03108v1/x4.png)\(a\)π/2\\pi/2rotation \(relative\-response anchor\)
![Refer to caption](https://arxiv.org/html/2607.03108v1/x5.png)\(b\)2​π/32\\pi/3rotation \(structural limitation\)

Figure 4:Prefix GramL0L\_\{0\}scoreΔ\\Deltafor 3 PEs×\\times6 shapes\. DyadicAxisPE gives the lowestπ/2\\pi/2scores onD4D\_\{4\}shapes, whereas TriAxis gives the lowest2​π/32\\pi/3scores onD3D\_\{3\}shapes, though still aboveε=0\.05\\varepsilon=0\.05\. Results are mean±1​σ\\pm 1\\sigmaover 5 seeds\.Below, we examine a positive example \(DyadicAxisPE’sπ/2\\pi/2\) and a structural limitation \(D3D\_\{3\}non\-detection\) in contrast\. Since this subsection addresses group\-level detection forD3D\_\{3\}/D4D\_\{4\}/D6D\_\{6\}, Figure[4](https://arxiv.org/html/2607.03108#S4.F4)uses two shapes per group \(a main\-text representative plus an Appendix shape;D3D\_\{3\}: equilateral\_triangle \+ star3,D4D\_\{4\}: square \+ star4,D6D\_\{6\}: hexagon \+ star6\) to confirm within\-group reproducibility\. Among the six main\-text shapes,O​\(2\)O\(2\)\(circle\),D2D\_\{2\}\(ellipse\), and the trivial group \(random\_noise\) are outside the scope of this subsection and are deferred to §4\.2\.1 and the Appendix\. Both panels share the same y\-axis for a direct comparison\.

Positive example \(Figure[4\(a\)](https://arxiv.org/html/2607.03108#S4.F4.sf1)\): onlyD4D\_\{4\}shapes drop substantially under the DyadicAxisPE, consistent withπ/2∈D4\\pi/2\\in D\_\{4\}; TriAxis and RFF remain high across all shapes \(neither liftsπ/2\\pi/2\)\. Structural limitation \(Figure[4\(b\)](https://arxiv.org/html/2607.03108#S4.F4.sf2)\): under DyadicAxisPE,D3D\_\{3\}shapes score systematically higher thanD4D\_\{4\}/D6D\_\{6\}shapes, reflecting thatπ/3\\pi/3and2​π/32\\pi/3are contained in neither DyadicAxisPE’sD4D\_\{4\}bound nor RFF’sZ2Z\_\{2\}bound\. In contrast, the TriAxis panel shows a substantial drop inD3D\_\{3\}shapes \(equilateral\_triangle:0\.480\.48, star3:0\.350\.35\), yielding aD3D\_\{3\}\-sensitive relative response under the tested Gram readout via a PE that exactly liftsD3D\_\{3\}\. An analogous complementary pattern holds forD6D\_\{6\}shapes \(hexagon, star6\): atπ/3\\pi/3, TriAxisPE drops to0\.330\.33–0\.390\.39while DyadicAxisPE remains at0\.710\.71–0\.810\.81\(six\-group comparison: Figure[A3](https://arxiv.org/html/2607.03108#A4.F3)\)\.

These results show that the exact liftability of PE constrains the bound of observable symmetry consistently across the three PEs\. Within\-group second representatives and the extension to 6 symmetry groups, the null\-PE limit, and the complementary PE\-over\-observable dominance observation are documented in Appendix[D](https://arxiv.org/html/2607.03108#A4),[E\.1](https://arxiv.org/html/2607.03108#A5.SS1), and[E](https://arxiv.org/html/2607.03108#A5)respectively\.

## 5Conclusion

In the tested PE\-MLP setting, trained neural network weights do not expose geometric symmetry as an absolute property of the target function, but through an observable symmetry set determined by the positional encoding, the observable, and training\. This work formalized this dependence via the two\-factor\(ϕ,Φ\)\(\\phi,\\Phi\)structure ofGobsG\_\{\\mathrm\{obs\}\}and showed that the PE’s exact liftability imposes a structural upper bound on the exact observable symmetry\. The same principle explains both positive and negative cases: transformations inside the PE’s exact\-liftable group tend to yield lower operational scores under the tested readouts \(D4D\_\{4\}\-aligned responses under DyadicAxisPE,D3D\_\{3\}/D6D\_\{6\}\-aligned lower scores under TriAxisPE, and aπ\\pi\-rotation response under RFF\)\. However, non\-liftable transformations are structurally suppressed\.

These results suggest that PE selection should be viewed not only as a choice affecting approximation quality, but also as a design parameter for post\-hoc symmetry readout\. DyadicAxisPE is suitable for lattice\-like symmetries up toD4D\_\{4\}, while TriAxisPE shifts the exact\-liftability bound toD6D\_\{6\}by replacing coordinate axes with three120∘120^\{\\circ\}\-separated axes, thereby producing lowerD3D\_\{3\}scores under the tested operational readout, which is suppressed under the DyadicAxisPE\. At the same time, exact liftability is necessary but not sufficient: training residuals, observable choice, and feature\-space redundancy also affect whether symmetry becomes visible at the weight level\.

The broader message is that not only what has been learned, but also what is in principle observable, is constrained by the combination of representation and readout observable\. This “limits of observability” perspective may extend beyond symmetry detection to other attempts to infer internal structure from the trained model weights\.

##### Limitations and future directions\.

This work is limited to 2D SDF MLPs, and extending the framework to 3D data or other architectures such as CNNs and Transformers requires identifying analogous structural constraints\. The automatic choice of optimal observables and a theory of approximate discriminative power outside the exact\-liftability regime remain open\. Future applications include PE design guided by representation\-theoretic classification, post\-hoc selection of equivariant architectures, and quality verification of the trained neural fields\.

## References

- \[1\]\(2019\)Implicit regularization in deep matrix factorization\.InAnnual Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§3\.3](https://arxiv.org/html/2607.03108#S3.SS3.p4.17)\.
- \[2\]M\. M\. Bronstein, J\. Bruna, T\. Cohen, and P\. Veličković\(2021\)Geometric deep learning: grids, groups, graphs, geodesics, and gauges\.arXiv preprint arXiv:2104\.13478\.Cited by:[§1](https://arxiv.org/html/2607.03108#S1.p1.7),[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px1.p1.1)\.
- \[3\]T\. Cohen and M\. Welling\(2016\)Group equivariant convolutional networks\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2607.03108#S1.p1.7),[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px1.p1.1)\.
- \[4\]L\. De Luigi, A\. Cardace, R\. Spezialetti, P\. Zama Ramirez, S\. Salti, and L\. Di Stefano\(2023\)Deep learning on implicit neural representations of shapes\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px4.p1.1)\.
- \[5\]N\. Dehmamy, R\. Walters, Y\. Liu, D\. Wang, and R\. Yu\(2021\)Automatic symmetry discovery with lie algebra convolutional network\.InAnnual Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px2.p1.3)\.
- \[6\]E\. Dupont, H\. Kim, S\. M\. A\. Eslami, D\. J\. Rezende, and D\. Rosenbaum\(2022\)From data to functa: your data point is a function and you can treat it like one\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px4.p1.1)\.
- \[7\]C\. Godfrey, D\. Brown, T\. Emerson, and H\. Kvinge\(2022\)On the symmetries of deep learning models and their internal representations\.InAnnual Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px3.p1.1)\.
- \[8\]L\. Hu, Y\. Li, and Z\. Lin\(2024\)Symmetry discovery for different data types\.arXiv preprint arXiv:2410\.09841\.Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px2.p1.3)\.
- \[9\]D\. P\. Kingma and J\. Ba\(2015\)Adam: a method for stochastic optimization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4\.1](https://arxiv.org/html/2607.03108#S4.SS1.p1.4)\.
- \[10\]I\. Loshchilov and F\. Hutter\(2017\)SGDR: stochastic gradient descent with warm restarts\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4\.1](https://arxiv.org/html/2607.03108#S4.SS1.p1.4)\.
- \[11\]B\. Mildenhall, P\. P\. Srinivasan, M\. Tancik, J\. T\. Barron, R\. Ramamoorthi, and R\. Ng\(2020\)NeRF: representing scenes as neural radiance fields for view synthesis\.InEuropean Conference on Computer Vision \(ECCV\),Cited by:[§3\.2](https://arxiv.org/html/2607.03108#S3.SS2.p1.1)\.
- \[12\]A\. Moskalev, A\. Sepliarskaia, I\. Sosnovik, and A\. Smeulders\(2022\)LieGG: studying learned lie group generators\.InAnnual Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px2.p1.3)\.
- \[13\]A\. Navon, A\. Shamsian, I\. Achituve, E\. Fetaya, G\. Chechik, and H\. Maron\(2023\)Equivariant architectures for learning in deep weight spaces\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px1.p1.1)\.
- \[14\]N\. Rahaman, A\. Baratin, D\. Arpit, F\. Draxler, M\. Lin, F\. A\. Hamprecht, Y\. Bengio, and A\. Courville\(2019\)On the spectral bias of neural networks\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§3\.2](https://arxiv.org/html/2607.03108#S3.SS2.p1.1)\.
- \[15\]A\. Rahimi and B\. Recht\(2007\)Random features for large\-scale kernel machines\.InAnnual Conference on Neural Information Processing Systems \(NIPS\),Cited by:[§3\.2](https://arxiv.org/html/2607.03108#S3.SS2.SSS0.Px2.p1.4)\.
- \[16\]V\. G\. Satorras, E\. Hoogeboom, and M\. Welling\(2021\)E\(n\) equivariant graph neural networks\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px1.p1.1)\.
- \[17\]A\. M\. Saxe, J\. L\. McClelland, and S\. Ganguli\(2014\)Exact solutions to the nonlinear dynamics of learning in deep linear neural networks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§3\.3](https://arxiv.org/html/2607.03108#S3.SS3.p4.17)\.
- \[18\]P\. H\. Schönemann\(1966\)A generalized solution of the orthogonal procrustes problem\.Psychometrika31\(1\),pp\. 1–10\.Cited by:[Appendix G](https://arxiv.org/html/2607.03108#A7.p1.11),[§3\.5](https://arxiv.org/html/2607.03108#S3.SS5.SSS0.Px1.p1.2)\.
- \[19\]V\. Sitzmann, J\. N\. P\. Martel, A\. W\. Bergman, D\. B\. Lindell, and G\. Wetzstein\(2020\)Implicit neural representations with periodic activation functions\.InAnnual Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px4.p1.1),[§3\.2](https://arxiv.org/html/2607.03108#S3.SS2.p1.1)\.
- \[20\]M\. Tancik, P\. P\. Srinivasan, B\. Mildenhall, S\. Fridovich\-Keil, N\. Raghavan, U\. Singhal, R\. Ramamoorthi, J\. T\. Barron, and R\. Ng\(2020\)Fourier features let networks learn high frequency functions in low dimensional domains\.InAnnual Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px4.p1.1),[§3\.2](https://arxiv.org/html/2607.03108#S3.SS2.SSS0.Px1.p1.3),[§3\.2](https://arxiv.org/html/2607.03108#S3.SS2.SSS0.Px2.p1.4),[§3\.2](https://arxiv.org/html/2607.03108#S3.SS2.p1.1)\.
- \[21\]M\. Weiler and G\. Cesa\(2019\)General e\(2\)\-equivariant steerable cnns\.InAnnual Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px1.p1.1)\.
- \[22\]Y\. Zhang, Z\. Chen, Y\. Liu, Z\. Qin, H\. Yuan, K\. Xu, Y\. Yuan, Q\. Gu, and A\. C\. Yao\(2026\)Group representational position embedding\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px4.p1.1),[§3\.2](https://arxiv.org/html/2607.03108#S3.SS2.p2.7)\.
- \[23\]B\. Zhao, R\. Walters, and R\. Yu\(2026\)Symmetry in neural network parameter spaces\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px3.p1.1)\.
- \[24\]L\. Ziyin, Y\. Xu, T\. Poggio, and I\. Chuang\(2025\)Parameter symmetry potentially unifies deep learning theory\.arXiv preprint arXiv:2502\.05300\.Cited by:[§2](https://arxiv.org/html/2607.03108#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AProofs of Lemma[1](https://arxiv.org/html/2607.03108#Thmlemma1)and Proposition[1](https://arxiv.org/html/2607.03108#Thmproposition1)

###### Proof of Lemma[1](https://arxiv.org/html/2607.03108#Thmlemma1)\(Exact liftability of structured PEs\)\.

We verify each claim in this section\. For each PE, we construct \(or show non\-existence of\) a linear mapρ​\(g\)\\rho\(g\)such thatϕ​\(g​x\)=ρ​\(g\)​ϕ​\(x\)\\phi\(g\\,x\)=\\rho\(g\)\\,\\phi\(x\)\.

\(i\) DyadicAxisPE:D4D\_\{4\}exact lift\.Each component of the DyadicAxisPE is independent per coordinate with frequenciesωk=2k​π\\omega\_\{k\}=2^\{k\}\\pi\(k=0,…,K−1k=0,\\dots,K\{\-\}1\)\.*π/2\\pi/2rotation\.*Rπ/2R\_\{\\pi/2\}acts as\(x1,x2\)↦\(−x2,x1\)\(x\_\{1\},x\_\{2\}\)\\mapsto\(\-x\_\{2\},x\_\{1\}\)\. For each frequencyωk\\omega\_\{k\},sin⁡\(ωk​\(−x2\)\)=−sin⁡\(ωk​x2\)\\sin\(\\omega\_\{k\}\(\-x\_\{2\}\)\)=\-\\sin\(\\omega\_\{k\}x\_\{2\}\)andcos⁡\(ωk​\(−x2\)\)=cos⁡\(ωk​x2\)\\cos\(\\omega\_\{k\}\(\-x\_\{2\}\)\)=\\cos\(\\omega\_\{k\}x\_\{2\}\), soRπ/2R\_\{\\pi/2\}is represented as a permutation\-and\-sign matrixPπ/2\(k\)P\_\{\\pi/2\}^\{\(k\)\}that swaps thex1x\_\{1\}andx2x\_\{2\}components with appropriate sign adjustments at eachkk\. Overall,ϕ​\(Rπ/2​x\)=Pπ/2​ϕ​\(x\)\\phi\(R\_\{\\pi/2\}\\,x\)=P\_\{\\pi/2\}\\,\\phi\(x\)\. BecauseRπ/22=RπR\_\{\\pi/2\}^\{2\}=R\_\{\\pi\}andRπ/24=idR\_\{\\pi/2\}^\{4\}=\\mathrm\{id\}are constructed analogously, the entire rotation subgroup⟨Rπ/2⟩\\langle R\_\{\\pi/2\}\\rangleofD4D\_\{4\}is exactly lifted\.*Axis reflections\.*The reflectionσx1:\(x1,x2\)↦\(x1,−x2\)\\sigma\_\{x\_\{1\}\}:\(x\_\{1\},x\_\{2\}\)\\mapsto\(x\_\{1\},\-x\_\{2\}\)acts as a diagonal sign matrixDσD\_\{\\sigma\}flipping onlysin⁡\(ωk​x2\)\\sin\(\\omega\_\{k\}x\_\{2\}\)\(by the oddness ofsin\\sinand evenness ofcos\\cos\)\. The reflectionσx2\\sigma\_\{x\_\{2\}\}is analogous to this\. Since⟨Rπ/2,σx1⟩=D4\\langle R\_\{\\pi/2\},\\sigma\_\{x\_\{1\}\}\\rangle=D\_\{4\}, the fullD4D\_\{4\}group is exactly lifted\.

\(i’\) DyadicAxisPE: non\-liftability ofπ/3\\pi/3and2​π/32\\pi/3\.Theπ/3\\pi/3rotationRπ/3R\_\{\\pi/3\}acts as\(x1,x2\)↦\(12​x1−32​x2,32​x1\+12​x2\)\(x\_\{1\},x\_\{2\}\)\\mapsto\(\\tfrac\{1\}\{2\}x\_\{1\}\-\\tfrac\{\\sqrt\{3\}\}\{2\}x\_\{2\},\\,\\tfrac\{\\sqrt\{3\}\}\{2\}x\_\{1\}\+\\tfrac\{1\}\{2\}x\_\{2\}\)\. The components ofϕ​\(Rπ/3​x\)\\phi\(R\_\{\\pi/3\}\\,x\)include terms such as

sin⁡\(2k​π​\(12​x1−32​x2\)\)=sin⁡\(2k​π⋅12​x1\)​cos⁡\(2k​π⋅32​x2\)−cos⁡\(2k​π⋅12​x1\)​sin⁡\(2k​π⋅32​x2\)\.\\sin\\\!\\bigl\(2^\{k\}\\pi\(\\tfrac\{1\}\{2\}x\_\{1\}\-\\tfrac\{\\sqrt\{3\}\}\{2\}x\_\{2\}\)\\bigr\)=\\sin\(2^\{k\}\\pi\\cdot\\tfrac\{1\}\{2\}\\,x\_\{1\}\)\\cos\(2^\{k\}\\pi\\cdot\\tfrac\{\\sqrt\{3\}\}\{2\}\\,x\_\{2\}\)\-\\cos\(2^\{k\}\\pi\\cdot\\tfrac\{1\}\{2\}\\,x\_\{1\}\)\\sin\(2^\{k\}\\pi\\cdot\\tfrac\{\\sqrt\{3\}\}\{2\}\\,x\_\{2\}\)\.Every component of DyadicAxisPE is a univariate function of eitherx1x\_\{1\}orx2x\_\{2\}alone, so any linear combination takes the additively separable formf​\(x1\)\+g​\(x2\)f\(x\_\{1\}\)\+g\(x\_\{2\}\)\. The term above is a product of functions ofx1x\_\{1\}andx2x\_\{2\}and hence is not additively separable; indeed,∂2∂x1​∂x2​\[sin⁡\(2k​π​\(12​x1−32​x2\)\)\]≠0\\frac\{\\partial^\{2\}\}\{\\partial x\_\{1\}\\,\\partial x\_\{2\}\}\\bigl\[\\sin\(2^\{k\}\\pi\(\\frac\{1\}\{2\}x\_\{1\}\-\\frac\{\\sqrt\{3\}\}\{2\}x\_\{2\}\)\)\\bigr\]\\neq 0, whereas∂2∂x1​∂x2​\[f​\(x1\)\+g​\(x2\)\]=0\\frac\{\\partial^\{2\}\}\{\\partial x\_\{1\}\\,\\partial x\_\{2\}\}\[f\(x\_\{1\}\)\+g\(x\_\{2\}\)\]=0identically\. Therefore, no linear mapρ​\(Rπ/3\)\\rho\(R\_\{\\pi/3\}\)satisfyingϕ​\(Rπ/3​x\)=ρ​\(Rπ/3\)​ϕ​\(x\)\\phi\(R\_\{\\pi/3\}\\,x\)=\\rho\(R\_\{\\pi/3\}\)\\,\\phi\(x\)exists\. The same argument applies toR2​π/3=Rπ/32R\_\{2\\pi/3\}=R\_\{\\pi/3\}^\{2\}\.

\(ii\) TriAxisPE:D6D\_\{6\}exact lift\.The2​π/32\\pi/3rotationR2​π/3R\_\{2\\pi/3\}satisfiesvj⊤​\(R2​π/3​x\)=vj−1⊤​xv\_\{j\}^\{\\top\}\(R\_\{2\\pi/3\}\\,x\)=v\_\{j\-1\}^\{\\top\}x\(indices mod 3\), and thus acts as a cyclic permutation of the three directional blocks\. Theπ/3\\pi/3rotation mapsvj↦−vj\+1v\_\{j\}\\mapsto\-v\_\{j\+1\}, which is realized as a cyclic permutation composed of sign flips on the sine components\. Thex1x\_\{1\}\-axis reflection mapsv0↦v0v\_\{0\}\\mapsto v\_\{0\}andv1↔v2v\_\{1\}\\leftrightarrow v\_\{2\}, again a linear action on the vertices\. Since⟨Rπ/3,σx1⟩=D6⊃D3\\langle R\_\{\\pi/3\},\\sigma\_\{x\_\{1\}\}\\rangle=D\_\{6\}\\supset D\_\{3\}, the fullD6D\_\{6\}symmetry is exactly lifted\.

\(iii\) RFF:Z2Z\_\{2\}exact lift and non\-liftability of general angles\.Writeθi:=ωi⊤​x\+bi\\theta\_\{i\}:=\\omega\_\{i\}^\{\\top\}x\+b\_\{i\}\.*π\\pirotation\.*ForRπ​x=−xR\_\{\\pi\}\\,x=\-x, we haveθi′:=ωi⊤​\(−x\)\+bi=−θi\+2​bi\\theta\_\{i\}^\{\\prime\}:=\\omega\_\{i\}^\{\\top\}\(\-x\)\+b\_\{i\}=\-\\theta\_\{i\}\+2b\_\{i\}\. By the addition formula,

\(sin⁡\(θi′\)cos⁡\(θi′\)\)=\(−cos⁡\(2​bi\)sin⁡\(2​bi\)sin⁡\(2​bi\)cos⁡\(2​bi\)\)​\(sin⁡\(θi\)cos⁡\(θi\)\)\\begin\{pmatrix\}\\sin\(\\theta\_\{i\}^\{\\prime\}\)\\\\ \\cos\(\\theta\_\{i\}^\{\\prime\}\)\\end\{pmatrix\}=\\begin\{pmatrix\}\-\\cos\(2b\_\{i\}\)&\\sin\(2b\_\{i\}\)\\\\ \\phantom\{\-\}\\sin\(2b\_\{i\}\)&\\cos\(2b\_\{i\}\)\\end\{pmatrix\}\\begin\{pmatrix\}\\sin\(\\theta\_\{i\}\)\\\\ \\cos\(\\theta\_\{i\}\)\\end\{pmatrix\}holds for eachii\. Therefore,ϕ​\(Rπ​x\)=ρ​\(Rπ\)​ϕ​\(x\)\\phi\(R\_\{\\pi\}\\,x\)=\\rho\(R\_\{\\pi\}\)\\,\\phi\(x\)with the block diagonal matrixρ​\(Rπ\)=diag​\(M1,…,Mn\)\\rho\(R\_\{\\pi\}\)=\\mathrm\{diag\}\(M\_\{1\},\\dots,M\_\{n\}\)\.*Non\-liftability of general angles\.*For a rotationRαR\_\{\\alpha\}withα≠0,π\\alpha\\neq 0,\\pi, we haveωi⊤​Rα​x=\(Rα⊤​ωi\)⊤​x\\omega\_\{i\}^\{\\top\}R\_\{\\alpha\}\\,x=\(R\_\{\\alpha\}^\{\\top\}\\omega\_\{i\}\)^\{\\top\}x\. Under the general position assumption,Rα⊤​ωi∉\{±ωj∣j=1,…,n\}R\_\{\\alpha\}^\{\\top\}\\omega\_\{i\}\\notin\\\{\\pm\\omega\_\{j\}\\mid j=1,\\dots,n\\\}for allii\. Consequently, the components ofϕ​\(Rα​x\)\\phi\(R\_\{\\alpha\}\\,x\)involve frequency directions that are absent from those ofϕ​\(x\)\\phi\(x\)\. Because plane wavesei​ω⊤​xe^\{i\\omega^\{\\top\}x\}andei​ω′⁣⊤​xe^\{i\\omega^\{\\prime\\top\}x\}withω≠±ω′\\omega\\neq\\pm\\omega^\{\\prime\}are linearly independent inL2​\(ℝ2\)L^\{2\}\(\\mathbb\{R\}^\{2\}\),ϕ​\(Rα​x\)\\phi\(R\_\{\\alpha\}\\,x\)cannot be expressed as a linear combination of the components ofϕ​\(x\)\\phi\(x\)\. ∎

###### Proof of Proposition[1](https://arxiv.org/html/2607.03108#Thmproposition1)\(Exact Observability Hierarchy\)\.

We show the two types of containment separately\.

Gobsexact⊆GliftexactG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\\subseteq G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\.By definition, the elements ofGobsexactG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}are restricted toGliftexactG\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\. Hence, containment follows immediately\.

Gobsexact⊆GtrueG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\\subseteq G\_\{\\mathrm\{true\}\}\.Letg∈Gobsexactg\\in G\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\. Then,Φ​\(Tg​θ\)=Φ​\(θ\)\\Phi\(T\_\{g\}\\,\\theta\)=\\Phi\(\\theta\)\. SinceΦ\\Phiis symmetry\-sufficient \(Definition[2](https://arxiv.org/html/2607.03108#Thmdefinition2)\),

fθ​\(g−1​x\)=fθ​\(x\)for all​x∈X\.f\_\{\\theta\}\(g^\{\-1\}\\,x\)=f\_\{\\theta\}\(x\)\\quad\\text\{for all \}x\\in X\.By the convergence assumptionsupx∈X\|fθ​\(x\)−f∗​\(x\)\|≤δ\\sup\_\{x\\in X\}\|f\_\{\\theta\}\(x\)\-f\_\{\*\}\(x\)\|\\leq\\delta, the triangle inequality gives for allxx,

\|f∗​\(g−1​x\)−f∗​\(x\)\|≤\|f∗​\(g−1​x\)−fθ​\(g−1​x\)\|\+\|fθ​\(g−1​x\)−fθ​\(x\)\|\+\|fθ​\(x\)−f∗​\(x\)\|≤2​δ\.\|f\_\{\*\}\(g^\{\-1\}\\,x\)\-f\_\{\*\}\(x\)\|\\leq\|f\_\{\*\}\(g^\{\-1\}\\,x\)\-f\_\{\\theta\}\(g^\{\-1\}\\,x\)\|\+\|f\_\{\\theta\}\(g^\{\-1\}\\,x\)\-f\_\{\\theta\}\(x\)\|\+\|f\_\{\\theta\}\(x\)\-f\_\{\*\}\(x\)\|\\leq 2\\delta\.Becausefθ​\(g−1​x\)=fθ​\(x\)f\_\{\\theta\}\(g^\{\-1\}\\,x\)=f\_\{\\theta\}\(x\), the middle term vanishes\. Under sufficient convergence \(δ→0\\delta\\to 0\),f∗​\(g−1​x\)=f∗​\(x\)f\_\{\*\}\(g^\{\-1\}\\,x\)=f\_\{\*\}\(x\)holds for allxx, and henceg∈Gtrueg\\in G\_\{\\mathrm\{true\}\}\.

Combining both containments,Gobsexact⊆Gliftexact∩GtrueG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}\\subseteq G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}\\cap G\_\{\\mathrm\{true\}\}\. ∎

## Appendix BFunctional vs Structural Symmetry: Contrast with Dense Forward

This appendix elaborates on how the symmetry readout via the observableΦ\\Phidiffers essentially from a naive dense\-forward checkfθ​\(g−1​x\)≈fθ​\(x\)f\_\{\\theta\}\(g^\{\-1\}x\)\\approx f\_\{\\theta\}\(x\)\.

##### The space on which the group action acts\.

The essential distinction between dense forward and observables in this framework lies inthe space in which the group action is performed\. Dense forward uses the input\-space actiong:x↦g​xg:x\\mapsto gxdirectly, comparingfθ​\(g−1​x\)f\_\{\\theta\}\(g^\{\-1\}x\)andfθ​\(x\)f\_\{\\theta\}\(x\)at many sample points, whereas the observableΦ\\Phiis defined as the response to a structural transformationTg​θT\_\{g\}\\thetamediated by the feature\-space liftρ​\(g\)\\rho\(g\)\. Forg∈Gliftexactg\\in G\_\{\\mathrm\{lift\}\}^\{\\mathrm\{exact\}\}, the two coincide viaϕ​\(g​x\)=ρ​\(g\)​ϕ​\(x\)\\phi\(gx\)=\\rho\(g\)\\phi\(x\); outside this set, they diverge that dense forward uses the actualϕ​\(g−1​x\)\\phi\(g^\{\-1\}x\)directly, whereas the observable handlesggonly through its feature\-space lift\.

##### Functional vs structural symmetry\.

This distinction arises because PE plays a dual role in both training and observation: on the one hand, during training, PE serves as the feature mapϕ\\phithat determines how the MLP sees inputs; on the other hand, at observation time, it provides the structural representationρ\\rhoof the group action\. Because MLPs are universal approximators, sufficient training can realize function\-level symmetryfθ​\(g−1​x\)≈fθ​\(x\)f\_\{\\theta\}\(g^\{\-1\}x\)\\approx f\_\{\\theta\}\(x\)independently of PE liftability \(functional symmetry\)\. Invariance of observables viaρ\\rho\(structural symmetry\), in contrast, is constrained by the PE’s algebraic structure\. Consequently, situations can arise in which a model is functionally symmetric yet structurally undetectable, a constraint intrinsic to structural readout that is unavoidable onceρ\\rhois used, and precisely the constraint this work characterizes\. The main textGobsexactG\_\{\\mathrm\{obs\}\}^\{\\mathrm\{exact\}\}and hierarchy are described at the structural rather than functional level of symmetry\. A concrete instance of this gap is presented for translational symmetry in Appendix[C](https://arxiv.org/html/2607.03108#A3)\.

## Appendix CTranslation Symmetry Readout

The main text focuses on rotational and reflective symmetries, and translational symmetry fits naturally into the same framework\. This appendix \(i\) consolidates the exact lift of translation for all three PEs as a single lemma, \(ii\) reports PE\-level verification via Procrustes residuals , and \(iii\) demonstrates translation detection using gram observables on trained weights\.

### C\.1Exact Lift of Translation under the PEs

Lemma[1](https://arxiv.org/html/2607.03108#Thmlemma1)addressed rotations and reflections\. Translation admits a uniform statement across all three PEs, which we formulate separately\.

###### Lemma 2\(Exact lift of translation\)\.

For each of DyadicAxisPE, TriAxisPE, and RFF, there exists a PE\-dependent block\-diagonal linear mapM​\(t\)M\(t\)such that the translationx↦x\+tx\\mapsto x\+tis exactly lifted as

ϕ​\(x\+t\)=M​\(t\)​ϕ​\(x\)\.\\phi\(x\+t\)=M\(t\)\\,\\phi\(x\)\.

###### Proof\.

DyadicAxisPE\.By the addition formula, for each frequencyωk=2k​π\\omega\_\{k\}=2^\{k\}\\piand coordinatexix\_\{i\},

\(sin⁡\(ωk​\(xi\+ti\)\)cos⁡\(ωk​\(xi\+ti\)\)\)=\(cos⁡\(ωk​ti\)sin⁡\(ωk​ti\)−sin⁡\(ωk​ti\)cos⁡\(ωk​ti\)\)​\(sin⁡\(ωk​xi\)cos⁡\(ωk​xi\)\)\.\\begin\{pmatrix\}\\sin\(\\omega\_\{k\}\(x\_\{i\}\+t\_\{i\}\)\)\\\\ \\cos\(\\omega\_\{k\}\(x\_\{i\}\+t\_\{i\}\)\)\\end\{pmatrix\}=\\begin\{pmatrix\}\\cos\(\\omega\_\{k\}t\_\{i\}\)&\\sin\(\\omega\_\{k\}t\_\{i\}\)\\\\ \-\\sin\(\\omega\_\{k\}t\_\{i\}\)&\\cos\(\\omega\_\{k\}t\_\{i\}\)\\end\{pmatrix\}\\begin\{pmatrix\}\\sin\(\\omega\_\{k\}x\_\{i\}\)\\\\ \\cos\(\\omega\_\{k\}x\_\{i\}\)\\end\{pmatrix\}\.Therefore,ϕDyadic​\(x\+t\)=M​\(t\)​ϕDyadic​\(x\)\\phi\_\{\\mathrm\{Dyadic\}\}\(x\+t\)=M\(t\)\\,\\phi\_\{\\mathrm\{Dyadic\}\}\(x\)with the block\-diagonal matrixM​\(t\)=diagk=0K−1​\(Rω0​t1,Rω0​t2,…,RωK−1​t2\)M\(t\)=\\mathrm\{diag\}\_\{k=0\}^\{K\-1\}\(R\_\{\\omega\_\{0\}t\_\{1\}\},R\_\{\\omega\_\{0\}t\_\{2\}\},\\dots,R\_\{\\omega\_\{K\-1\}t\_\{2\}\}\), where each block is a2×22\\times 2rotation matrix\.

TriAxisPE\.Apply the same addition formula argument to each directional projectionuj:=vj⊤​xu\_\{j\}:=v\_\{j\}^\{\\top\}x\(j=0,1,2j=0,1,2\)\. The shiftvj⊤​tv\_\{j\}^\{\\top\}tinuju\_\{j\}acts as a2×22\\times 2rotation at eachjjand frequencyωk=2k​π\\omega\_\{k\}=2^\{k\}\\pi, soϕtri​\(x\+t\)=M​\(t\)​ϕtri​\(x\)\\phi\_\{\\mathrm\{tri\}\}\(x\+t\)=M\(t\)\\,\\phi\_\{\\mathrm\{tri\}\}\(x\)holds with a block diagonalM​\(t\)M\(t\)\.

RFF\.Withθi:=ωi⊤​x\+bi\\theta\_\{i\}:=\\omega\_\{i\}^\{\\top\}x\+b\_\{i\}, we haveθi′:=ωi⊤​\(x\+t\)\+bi=θi\+ωi⊤​t\\theta\_\{i\}^\{\\prime\}:=\\omega\_\{i\}^\{\\top\}\(x\+t\)\+b\_\{i\}=\\theta\_\{i\}\+\\omega\_\{i\}^\{\\top\}t\. The addition formula gives

\(sin⁡\(θi′\)cos⁡\(θi′\)\)=\(cos⁡\(ωi⊤​t\)sin⁡\(ωi⊤​t\)−sin⁡\(ωi⊤​t\)cos⁡\(ωi⊤​t\)\)​\(sin⁡\(θi\)cos⁡\(θi\)\)\\begin\{pmatrix\}\\sin\(\\theta\_\{i\}^\{\\prime\}\)\\\\ \\cos\(\\theta\_\{i\}^\{\\prime\}\)\\end\{pmatrix\}=\\begin\{pmatrix\}\\cos\(\\omega\_\{i\}^\{\\top\}t\)&\\sin\(\\omega\_\{i\}^\{\\top\}t\)\\\\ \-\\sin\(\\omega\_\{i\}^\{\\top\}t\)&\\cos\(\\omega\_\{i\}^\{\\top\}t\)\\end\{pmatrix\}\\begin\{pmatrix\}\\sin\(\\theta\_\{i\}\)\\\\ \\cos\(\\theta\_\{i\}\)\\end\{pmatrix\}for eachii\. Therefore,ϕRFF​\(x\+t\)=M​\(t\)​ϕRFF​\(x\)\\phi\_\{\\mathrm\{RFF\}\}\(x\+t\)=M\(t\)\\,\\phi\_\{\\mathrm\{RFF\}\}\(x\)withM​\(t\)=diag​\(Rω1⊤​t,…,Rωn⊤​t\)M\(t\)=\\mathrm\{diag\}\(R\_\{\\omega\_\{1\}^\{\\top\}t\},\\dots,R\_\{\\omega\_\{n\}^\{\\top\}t\}\)\. ∎

### C\.2PE\-level Procrustes residuals

Complementing the Procrustes residuals \(Table[1](https://arxiv.org/html/2607.03108#S4.T1)\), the residuals for translation\(0\.5,0\)\(0\.5,0\)are DyadicAxisPE5\.4×10−55\.4\\times 10^\{\-5\}, TriAxisPE5\.4×10−65\.4\\times 10^\{\-6\}, and RFF1\.5×10−71\.5\\times 10^\{\-7\}\. All three PEs yieldrP≤10−5r\_\{P\}\\leq 10^\{\-5\}, consistent with the exact lift established in Lemma[2](https://arxiv.org/html/2607.03108#Thmlemma2): translation is exactly representable at the PE level for all three PEs\.

### C\.3Detection of translational symmetry via prefix GramL0L\_\{0\}

![Refer to caption](https://arxiv.org/html/2607.03108v1/x6.png)Figure A1:Detection of translational symmetry: translation landscape of prefix GramL0L\_\{0\}Δ\\Delta\(DyadicAxisPE: left column, TriAxis: middle column, RFF: right column\)\. The gray dashed vertical lines indicate the true translational periods\. Stripe \(top row\): DyadicAxisPE and TriAxis exhibit periodic sharp dips, whereas RFF shows only a broad valley att=0t=0\. Circle \(bottom row, translationally asymmetric\): at=0t=0valley across all three PEs\. All three PEs exactly lift translation \(Lemma[2](https://arxiv.org/html/2607.03108#Thmlemma2)\), but the sufficient conditionM​\(t\)=IM\(t\)=IforG0​\(Tt​θ\)=M​\(t\)⊤​G0​M​\(t\)G\_\{0\}\(T\_\{t\}\\theta\)=M\(t\)^\{\\top\}G\_\{0\}M\(t\)to equalG0G\_\{0\}\(i\.e\.,ωi⊤​t≡0\(mod2​π\)\\omega\_\{i\}^\{\\top\}t\\equiv 0\\pmod\{2\\pi\}simultaneously for everyωi\\omega\_\{i\}\) is met at integer multiples of the stripe period for the discrete dyadic frequencies \(DyadicAxisPE/TriAxis\) but fails with probability zero att≠0t\\neq 0for the random frequencies \(RFF\); see the derivation below\. A full\-shape, full\-layer comparison including the 2D lattice pattern \(checker\) is provided in Figure[A2](https://arxiv.org/html/2607.03108#A3.F2)\.Using prefix GramL0L\_\{0\}constructed from the trained weights as the observable, Figure[A1](https://arxiv.org/html/2607.03108#A3.F1)compares the translation responses for the stripe \(p​1​mp1m\) and circle \(translationally asymmetric\)\.

By Lemma[2](https://arxiv.org/html/2607.03108#Thmlemma2), translation is exactly lifted for all three PEs \(ϕ​\(x\+t\)=M​\(t\)​ϕ​\(x\)\\phi\(x\+t\)=M\(t\)\\,\\phi\(x\)with block\-diagonalM​\(t\)M\(t\)of2×22\\times 2rotation blocks\), and for prefix GramL0L\_\{0\}G0=W0⊤​W0G\_\{0\}=W\_\{0\}^\{\\top\}W\_\{0\}we have

G0​\(Tt​θ\)=M​\(t\)⊤​G0​M​\(t\)G\_\{0\}\(T\_\{t\}\\theta\)=M\(t\)^\{\\top\}\\,G\_\{0\}\\,M\(t\)exactly\. A sufficient condition forΔ≈0\\Delta\\approx 0\(a dip\) at a giventtisM​\(t\)=IM\(t\)=I, i\.e\.,ωi⊤​t≡0\(mod2​π\)\\omega\_\{i\}^\{\\top\}t\\equiv 0\\pmod\{2\\pi\}simultaneously for every PE frequencyωi\\omega\_\{i\}\(each blockRωi⊤​tR\_\{\\omega\_\{i\}^\{\\top\}t\}ofM​\(t\)M\(t\)reduces to the identity\)\. Geometrically, each condition defines a family of parallel lines in thett\-plane with normalωi\\omega\_\{i\},Li=\{t:ωi⊤​t∈2​π​ℤ\}L\_\{i\}=\\\{t:\\omega\_\{i\}^\{\\top\}t\\in 2\\pi\\mathbb\{Z\}\\\}, andM​\(t\)=IM\(t\)=Irequirest∈⋂iLit\\in\\bigcap\_\{i\}L\_\{i\}\. In the 2\-dimensionaltt\-plane, the behavior is determined by thenumber of independent directionsamong\{ωi\}\\\{\\omega\_\{i\}\\\}:

- •DyadicAxisPE\(ωk=2k​π\\omega\_\{k\}=2^\{k\}\\pialong two coordinate axes\) andTriAxisPE\(the same dyadic frequencies along three axes separated by2​π/32\\pi/3\): frequencies sharing a direction are mutually dependent, so the2​K2Kconditions \(respectively3​K3K\) effectively reduce to two independent directions\. Two independent constraints on 2Dttleave a non\-trivial 2D lattice as the common solution set, which meets integer multiples of the stripe period⇒\\Rightarrowperiodic dips\.
- •RFF\(ωi\\omega\_\{i\}are generic\-position random 2D frequencies\): then=24n=24frequencies are generically non\-commensurate; more than two independent constraints overdetermine the two\-dimensional translation variable, so the 24 conditions form an over\-determined system on 2Dtt\(three or more independent constraints on a 2D space generically leave only\{0\}\\\{0\\\}as the common solution\)\. The probability of simultaneous satisfaction at anyt≠0t\\neq 0is zero⇒\\Rightarrowno dip outsidet=0t=0\.

Thus, although all three PEs exactly lift translation, periodic dips in prefix GramL0L\_\{0\}appear only when the PE frequency set is discrete and compatible with the translational period of the shape\. This is a concrete instance of the functional–structural gap discussed in Appendix[B](https://arxiv.org/html/2607.03108#A2): the stripe is functionally represented by the RFF\-MLP, but the structural invariance ofG0G\_\{0\}under the sandwich action requires a discrete solution ofM​\(t\)=IM\(t\)=I, which is absent \(except att=0t=0\) for random\-frequency PEs\. For the translationally asymmetric circle, all three PEs show a valley only att=0t=0\.

Note that while translation is exactly lifted at the PE representation\-theory level, the estimator used in our implementation \(Procrustes approximation\) leaves residuals of0\.010\.01–0\.150\.15\.333This residual is inherent to the Procrustes estimator; analytic estimators exploiting the block\-diagonal structure of translation lifts could reduce it, which we leave to future work\.The PE\-level residuals in C\.2 arerP≤10−5r\_\{P\}\\leq 10^\{\-5\}\. The larger 0\.01–0\.15 values refer to downstream operational Gram\-score discrepancies under the estimator, not to PE liftability\. This distinction that representation\-level exactness versus estimator\-level approximation is important for clarifying the consistency between theory and experiments\.

### C\.4Full\-layer comparison

The layer dependence of the translation detection is shown in Figure[A2](https://arxiv.org/html/2607.03108#A3.F2)\.

![Refer to caption](https://arxiv.org/html/2607.03108v1/x7.png)Figure A2:Full\-layer comparison of translation detection \(3 shapes×\\times3 PEs×\\times6 observables\)\. Rows correspond to shape / PE; columns to observables \(prefix\_L0L\_\{0\}–L4L\_\{4\},WeffW\_\{\\mathrm\{eff\}\}\)\.Qualitative patterns are preserved across all observables \(stripe: periodic dips under DyadicAxisPE/TriAxis, a single broad valley att=0t=0under RFF; circle: only at=0t=0valley under all three PEs\)\. The dips are sharpest atL0L\_\{0\}, consistent with the adoption of prefix GramL0L\_\{0\}\.

## Appendix DLayer Dependence of prefix Gram and Extension to Six Symmetry Groups

In the main text, we used prefix GramL0L\_\{0\}\(G0=W0⊤​W0G\_\{0\}=W\_\{0\}^\{\\top\}W\_\{0\}\) as the weight\-level observable for six representative shapes\. This appendix \(i\) extends the analysis to six symmetry groups with additional shapes via a representative\-angle probe atL0L\_\{0\}, and \(ii\) supports the use ofL0L\_\{0\}as the clearest readout among the tested observables by comparing all layersL0L\_\{0\}–L4L\_\{4\}andWeffW\_\{\\mathrm\{eff\}\}Gram\.

##### Representative\-angle probe across 6 groups\.

This is a representative angle detection probe that evaluates the readout sensitivity at a single group\-specific angle rather than summarizing the full group\-integrated observability\. The exact liftability bound patterns demonstrated with the six shapes are qualitatively maintained when extended to six symmetry groups \(O​\(2\)O\(2\),D4D\_\{4\},D6D\_\{6\},D3D\_\{3\},D2D\_\{2\}, trivial\) including additional shapes \(Figure[A3](https://arxiv.org/html/2607.03108#A4.F3)\)\. Each group is probed at a single group\-specific representative angle \(D4D\_\{4\}:π/2\\pi/2,D3D\_\{3\}:2​π/32\\pi/3, and so on\)\. The structure whereby lattice groups at or belowD4D\_\{4\}are stably detected with DyadicAxisPE, whileD3D\_\{3\}remains non\-detectable with DyadicAxisPE and RFF, is reproduced across all groups\. Comparing across three PEs with TriAxisPE, DyadicAxisPE is superior forD4D\_\{4\}whereas TriAxis is superior forD3D\_\{3\}, confirming the complementary pattern corresponding to each PE’s exact\-liftable group\. However, within\-group variation exists across shapes; per\-shape prefix Gram results are available in Appendix[E](https://arxiv.org/html/2607.03108#A5)\.

![Refer to caption](https://arxiv.org/html/2607.03108v1/x8.png)Figure A3:Representative\-angle detection probe \(single\-angle probe; not a full group\-integrated score\)\. Three PEs \(DyadicAxisPE, TriAxis, RFF\)×\\times6 symmetry groups \(O​\(2\)O\(2\),D4D\_\{4\},D6D\_\{6\},D3D\_\{3\},D2D\_\{2\}, trivial\), prefix GramL0L\_\{0\}Δ\\Deltaat a single group\-specific angle \(D4D\_\{4\}:90∘90^\{\\circ\},D6D\_\{6\}:60∘60^\{\\circ\},D3D\_\{3\}:120∘120^\{\\circ\}, etc\.\)\. Each cell shows the Gram distanceΔ​\(Π\)\\Delta\(\\Pi\)averaged within each group \(five seeds\);Δ≈0\\Delta\\approx 0indicates a low invariant\-response score under the probed transform,Δ≈2\\Delta\\approx\\sqrt\{2\}indicates failure\. DyadicAxisPE achieves the best score forD4D\_\{4\}whereas TriAxis achieves the best score forD3D\_\{3\}/D6D\_\{6\}, clearly showing the complementary pattern shaped by each PE’s exact\-liftable group\. Within\-group variation: Appendix[E](https://arxiv.org/html/2607.03108#A5)\. Full\-layer comparison: Figure[A7](https://arxiv.org/html/2607.03108#A4.F7)\.
##### Rotation response curves \(full\-layer version of Figure[3](https://arxiv.org/html/2607.03108#S4.F3)\)\.

Figure[A4](https://arxiv.org/html/2607.03108#A4.F4)shows the rotation\-angle response across all observables for 3 representative shapes\. Shallower layers \(L0L\_\{0\}\) produce sharper dips with smaller seed\-to\-seed variance, whereas the readout sensitivity decreases toward deeper layers\.WeffW\_\{\\mathrm\{eff\}\}gram showed the lowest readout sensitivity\.

![Refer to caption](https://arxiv.org/html/2607.03108v1/x9.png)Figure A4:Full\-layer comparison of rotation response curves \(L0L\_\{0\}–L4L\_\{4\}\+WeffW\_\{\\mathrm\{eff\}\}\)\. Each row corresponds to a different observable, and each column corresponds to a representative shape\. Shallower layers produce sharper dips\.
##### D3D\_\{3\}non\-detectability \(full\-layer version of Figure[4\(b\)](https://arxiv.org/html/2607.03108#S4.F4.sf2)\)\.

The non\-detectability ofD3D\_\{3\}is common across all layers, although the score levels differ by layer \(Figure[A5](https://arxiv.org/html/2607.03108#A4.F5)\)\.

![Refer to caption](https://arxiv.org/html/2607.03108v1/x10.png)Figure A5:Full\-layer comparison ofD3D\_\{3\}non\-detectability \(3 PEs: DyadicAxisPE, TriAxis, RFF\)\.
##### D3D\_\{3\}recovery \(full\-layer version at group\-specific angles\)\.

TheD3D\_\{3\}response recovery pattern via TriAxisPE \(TriAxis panel of main\-text Figure[4\(b\)](https://arxiv.org/html/2607.03108#S4.F4.sf2)\) is qualitatively maintained at group\-specific angles across all layers \(Figure[A6](https://arxiv.org/html/2607.03108#A4.F6)\)\.

![Refer to caption](https://arxiv.org/html/2607.03108v1/x11.png)Figure A6:Full\-layer comparison ofD3D\_\{3\}response recovery \(6 observables×\\times3 PEs\)\. Arranged as a 2\-column×\\times3\-row grid\.
##### Observability map \(full\-layer version of Figure[A3](https://arxiv.org/html/2607.03108#A4.F3)\)\.

The observability maps for each layer are shown in Figure[A7](https://arxiv.org/html/2607.03108#A4.F7)\.

![Refer to caption](https://arxiv.org/html/2607.03108v1/x12.png)Figure A7:Full\-layer comparison of observability maps \(3 PEs: DyadicAxisPE, TriAxis, RFF; 6 observables\)\. Arranged as a 2\-column×\\times3\-row grid\.Table[A1](https://arxiv.org/html/2607.03108#A4.T1)provides a quantitative summary of the results\.L0L\_\{0\}achieves the best scores with the smallest seed\-to\-seed variance across all PEs, supporting theL0L\_\{0\}choice\. The full\-layer dependence of the translation detection is presented in Appendix[C\.4](https://arxiv.org/html/2607.03108#A3.SS4)\(Figure[A2](https://arxiv.org/html/2607.03108#A3.F2)\)\.

Table A1:Quantitative summary by observable\. Mean score across all nine symmetry groups from the full 16 shapes \(O​\(2\)O\(2\),D6D\_\{6\},D5D\_\{5\},D4D\_\{4\},D3D\_\{3\},D2D\_\{2\},p​4​mp4m,p​1​mp1m, trivial\) in the observability map \(within\-group shape mean, then across\-group mean; lower is better\) and mean seed\-to\-seed standard deviation of rotation response \(lower is more stable\)\.L0L\_\{0\}is best on both metrics for all PEs\.

## Appendix EComplete Results of the PE×\\timesObservable Grid

![Refer to caption](https://arxiv.org/html/2607.03108v1/x13.png)Figure A8:Additional 10 shapes used in the Appendix \(totaling 16 shapes with the 6 in the main text\)\. This includes second representatives of each symmetry group \(ring, star6, cross, star3, rectangle\), additional groups \(star4, star5\), the translational patternp​1​mp1m\(stripe\), and 2D translation patterns \(checker, circle\_lattice\)\. Stripe, checker, and circle\_lattice are used for the translation analysis in Appendix[C](https://arxiv.org/html/2607.03108#A3)\.This appendix presents a systematic evaluation of four PEs×\\timesup to 11 observables\. In particular, by comparing the effect sizes of the PE axis and the observable \(gram layer/form\) axis within weight\-only observables, we document the complementary observation thatwithin weight\-only readout, the PE axis dominates the observable axis, supplementing the main text analysis\. For completeness, we also include activation\-level and function\-level \(output\) observables; these lie outside the weight\-only readout focus of this paper and are not used in our main claims\.

##### Experimental design\.

The PE conditions were IdentityPE \(Din=2D\_\{\\mathrm\{in\}\}=2\), DyadicAxisPE K=12 \(48 dimensions\), TriAxis K=8 \(48 dimensions\), and RFF \(n=24n=24, 48 dimensions\), totaling four types\. The observables fall into three categories, totaling up to 11 types, defined as follows:

\(i\) Weight Gram\(1 type\): From the effective weight matrixWeff=WL​⋯​W0W\_\{\\mathrm\{eff\}\}=W\_\{L\}\\cdots W\_\{0\}\(product of all layer weights\), we construct the extended Gram matrixA~\\tilde\{A\}and compute, for each of the 8D4D\_\{4\}group elements with homogeneous orthogonal matrixΠ~g=diag​\(ρ​\(g−1\),1\)\\tilde\{\\Pi\}\_\{g\}=\\mathrm\{diag\}\(\\rho\(g^\{\-1\}\),1\),

Δ​\(Πg\)=‖A~−Π~g⊤​A~​Π~g‖F‖A~‖F\.\\Delta\(\\Pi\_\{g\}\)=\\frac\{\\\|\\tilde\{A\}\-\\tilde\{\\Pi\}\_\{g\}^\{\\top\}\\tilde\{A\}\\,\\tilde\{\\Pi\}\_\{g\}\\\|\_\{F\}\}\{\\\|\\tilde\{A\}\\\|\_\{F\}\}\.The score is the meanΔ\\Deltaover the seven non\-identity transforms \(Δ≈0\\Delta\\approx 0: symmetry preserved;Δ≈2\\Delta\\approx\\sqrt\{2\}: no symmetry\)\. In this grid, DyadicAxisPE provides analyticΠ~g\\tilde\{\\Pi\}\_\{g\}derived from the frequency matrixKK, and TriAxis uses the Procrustes\-estimatedΠ~g\\tilde\{\\Pi\}\_\{g\}\. SinceD4D\_\{4\}is closed under inversion, the alternative conventionΠ~g=diag​\(ρ​\(g\),1\)\\tilde\{\\Pi\}\_\{g\}=\\mathrm\{diag\}\(\\rho\(g\),1\)yields the same set ofΔ\\Deltavalues and is implementation equivalent\.

\(ii\) Weight\-prefix Gram\(L0L\_\{0\}–L4L\_\{4\}, 5 types\): For each layerll, the prefix weight productPl=Wl​⋯​W0P\_\{l\}=W\_\{l\}\\cdots W\_\{0\}yields the Gram matrixPl⊤​PlP\_\{l\}^\{\\top\}P\_\{l\}, which is extended and scored with the sameD4D\_\{4\}Δ\\Deltaas above\. Unlike the effective\-weight Gram, this preserves the layer\-wise structure and is thus more likely to be symmetry\-sufficient\. In this grid, DyadicAxisPE admits analyticΠ~g\\tilde\{\\Pi\}\_\{g\}and TriAxis uses Procrustes estimation; numerical estimation is possible for RFF but is not included here\.

\(iii\) Activation score\(L0L\_\{0\}–L3L\_\{3\}, 4 types\): Using post\-ReLU activationshl​\(x\)h\_\{l\}\(x\)at grid points\{xi\}\\\{x\_\{i\}\\\}, we compute the normalized MSE over the 7 non\-identity elements ofD4D\_\{4\},Gtest=\{Rπ/2,Rπ,R3​π/2,σx1,σx2,σx1=x2,σx1=−x2\}G\_\{\\mathrm\{test\}\}=\\\{R\_\{\\pi/2\},R\_\{\\pi\},R\_\{3\\pi/2\},\\sigma\_\{x\_\{1\}\},\\sigma\_\{x\_\{2\}\},\\sigma\_\{x\_\{1\}=x\_\{2\}\},\\sigma\_\{x\_\{1\}=\-x\_\{2\}\}\\\}\(with\|Gtest\|=7\|G\_\{\\mathrm\{test\}\}\|=7\):

act​\_​scorel=1\|Gtest\|​∑g∈Gtest‖hl​\(x\)−hl​\(g​x\)‖2Var​\(hl\)\.\\mathrm\{act\\\_score\}\_\{l\}=\\frac\{1\}\{\|G\_\{\\mathrm\{test\}\}\|\}\\sum\_\{g\\in G\_\{\\mathrm\{test\}\}\}\\frac\{\\\|h\_\{l\}\(x\)\-h\_\{l\}\(gx\)\\\|^\{2\}\}\{\\mathrm\{Var\}\(h\_\{l\}\)\}\.Lower values indicate greater activation space invariance undergg\. Computable for all PE conditions\.

\(iv\) Output score\(1 type\): The normalized MSE‖fθ​\(x\)−fθ​\(g​x\)‖2/Var​\(fθ\)\\\|f\_\{\\theta\}\(x\)\-f\_\{\\theta\}\(gx\)\\\|^\{2\}/\\mathrm\{Var\}\(f\_\{\\theta\}\)of the network outputfθ​\(x\)f\_\{\\theta\}\(x\), computed analogously to activation score\. This directly measures the function\-level symmetry and serves as a PE\-independent upper\-bound indicator\. Computable for all PE conditions\.

For IdentityPE and RFF, weight Gram and weight\-prefix Gram are not included in this grid: IdentityPE is treated separately as a low\-dimensional null condition, whereas RFF lacks a closed\-form finite transform action for generic rotations and is therefore handled through operational Procrustes analyses elsewhere\. TriAxis uses Procrustes\-estimated PE action matrices for these observable variables\. For IdentityPE and RFF, the grid produces only the activation and output scores\. Their weight\-side behavior is characterized separately in the IdentityPE\-null analysis \(see §3\.5 for the exact/operational regime distinction\)\. Among the 16 shapes, 6 shapes \(circle, ring, square, cross, star4, and circle\_lattice; henceforthD4D\_\{4\}\-compatible 6 shapes\) carry all 8 elements of the origin\-centeredD4D\_\{4\}as true symmetries, and we pool median scores for each grid cell over these 6 shapes×\\times5 seeds\. Because theD4D\_\{4\}transforms testD4D\_\{4\}symmetry, low scores for these shapes directly indicate strongerD4D\_\{4\}\-compatible readout responses\.

##### Main results\.

Figure[A9](https://arxiv.org/html/2607.03108#A5.F9)shows the median detection scores across the PE×\\timesobservable grid\. Within the weight\-only rows \(weight Gram, weight\-prefix GramL0L\_\{0\}–L4L\_\{4\}\), the spread across observables at a fixed PE is limited \(∼0\.07\\sim 0\.07: DyadicAxisPE0\.160\.16–0\.260\.26, TriAxis0\.570\.57–0\.640\.64\), whereas at a fixed observable, the PE\-to\-PE gap is large \(e\.g\., prefixL0L\_\{0\}: DyadicAxisPE0\.190\.19vs TriAxis0\.610\.61, a∼0\.42\\sim 0\.42difference\), so the PE axis dominates the observable axis within the weight\-only readout\. In the activation rows, scores compress to0\.0080\.008–0\.0980\.098across all PEs, and the PE\-to\-PE gap shrinks, reflecting a regime change from weight\-only to activation\-based readout, distinct from the within\-weight\-only PE dominance\. Weight Gram and weight\-prefix gram are restricted to DyadicAxisPE \(analytic\) and TriAxis \(Procrustes\) because analytic PE action matrices are unavailable for IdentityPE and RFF \(gray cells\)\. The overall ordering across observable categories isWeffW\_\{\\mathrm\{eff\}\}Gram \(∼0\.25\\sim 0\.25\)\>\>weight\-prefix Gram \(0\.160\.16–0\.260\.26\)\>\>activation \(actL3L\_\{3\}at∼0\.008\\sim 0\.008\)\>\>output \(∼10−3\\sim 10^\{\-3\}–10−610^\{\-6\}\)\.

##### Comparison across PEs\.

DyadicAxisPE K=12 is the best in the weight\-prefix gram category \(L1L\_\{1\}:0\.1580\.158\), substantially improving overWeffW\_\{\\mathrm\{eff\}\}gram \(0\.2530\.253\)\. TriAxis K=8 has Procrustes\-estimated weight\-prefix Gram available \(prefixL0L\_\{0\}median0\.6100\.610, weight Gram0\.5920\.592\); its higher scores relative to DyadicAxisPE are consistent with the absence ofD4D\_\{4\}exact lift\. RFF lacks analytic weight\-prefix gram in this grid, but achieves an activation score \(L3L\_\{3\}:0\.0240\.024\) comparable to DyadicAxisPE\. IdentityPE has no weight\-side observables in this grid \(gray cells in the figure\), and even actL3L\_\{3\}shows a score of0\.0980\.098, approximately8×8\\timeshigher than the other PEs\.

### E\.1IdentityPE: a null condition with insufficient redundancy

IdentityPE \(ϕ​\(x\)=x\\phi\(x\)=x; output 2 dimensions, equivalent to no PE\) served as a null condition with no feature expansion\. Exact liftability formally holds sinceϕ​\(g​x\)=g​ϕ​\(x\)\\phi\(gx\)=g\\,\\phi\(x\), but the projection to two dimensions lacks the redundancy needed to encode the geometric structure in the weight space\. Under the IdentityPE condition, symmetry group separation from weights was not reliably achieved with the tested weight\-level observables \(silhouette score: IdentityPE=−0\.133=\-0\.133vs\. DyadicAxisPE=\+0\.285=\+0\.285, TriAxis=\+0\.248=\+0\.248, RFF=\+0\.027=\+0\.027\)\. Meanwhile, output\-level symmetry remains high \(invariance ranges from0\.590\.59to1\.001\.00depending on the shape\)\. The problem is not a failure of training, but rather thatwithout feature expansion by PE, reliable post\-hoc recovery is difficult, at least with the weight\-level observables used in this paper\. UnlikeD3D\_\{3\}under DyadicAxisPE, whose limitation stems from the absence of an exact lift, IdentityPE’s limitation is considered to arise from insufficient PE dimensionality \(Din=2D\_\{\\mathrm\{in\}\}=2\) to provide sufficient redundancy for a reliable weight\-level readout, despite the formal existence of an exact lift\. In other words, exact liftability is necessary but insufficient for post\-hoc detection\. This is clearly an exploratory interpretation; disentangling the interaction with the estimator and observable choice, as well as additional controlled experiments \(e\.g\., learned linear lift to higher dimensions\), remains a task for future work\.

##### TriAxisPE representative\-angle probe results\.

The numerical details behind the TriAxisPE representative\-angle probe reported in Figure[A3](https://arxiv.org/html/2607.03108#A4.F3)\(Appendix[D](https://arxiv.org/html/2607.03108#A4)\) are as follows \(prefix GramL0L\_\{0\}Δ\\Delta, mean over five seeds\)\. ForD3D\_\{3\}shapes, TriAxisPE substantially outperformed DyadicAxisPE \(star3: DyadicAxisPE0\.940\.94→\\toTriAxis0\.350\.35; equilateral\_triangle:0\.940\.94→\\to0\.480\.48\)\. TriAxisPE is similarly superior forD6D\_\{6\}shapes \(hexagon: DyadicAxisPE0\.710\.71vs\. TriAxis0\.330\.33; star6: DyadicAxisPE0\.810\.81vs\. TriAxis0\.390\.39\)\. Conversely, forD4D\_\{4\}shapes, DyadicAxisPE is superior \(square atπ/2\\pi/2: DyadicAxisPE0\.220\.22vs\. TriAxis0\.750\.75\)\. All structured PEs shared the same 48\-dimensional representation \(4×12=6×8=2×24=484\\times 12=6\\times 8=2\\times 24=48\), so the score differences reflected the encoding structure rather than dimensionality\.

##### Cost–detection trade\-off\.

Weight\-prefix Gram \(L1L\_\{1\}–L3L\_\{3\}\) achieves median scores of0\.1580\.158–0\.2030\.203\(DyadicAxisPE K=12\) without activations, substantially improving overWeffW\_\{\\mathrm\{eff\}\}Gram \(0\.2530\.253\) and offering a favorable trade\-off between computational cost and readout sensitivity\.

![Refer to caption](https://arxiv.org/html/2607.03108v1/x14.png)Figure A9:Median scores over the 6D4D\_\{4\}\-compatible shapes \(circle, ring, square, cross, star4, circle\_lattice\) across the PE×\\timesobservable grid \(4 PEs: IdentityPE, DyadicAxisPE, TriAxis, RFF; log scale; lower is better\)\. Within the weight\-only rows \(prefixL0L\_\{0\}–L4L\_\{4\},WeffW\_\{\\mathrm\{eff\}\}gram\), the PE\-to\-PE gap is large, while the observable\-to\-observable spread within a fixed PE is small; switching to the activation rows compresses scores across all PEs\. Gray cells indicate PE\-observable combinations for which PE action matrices are unavailable\. DyadicAxisPE uses analytic PE action matrices, whereas TriAxis uses Procrustes estimation for weight Gram and weight\-prefix Gram\. IdentityPE and RFF weight\-side observables are gray cells\.

## Appendix FLearning Dynamics ofDopD\_\{\\mathrm\{op\}\}and Threshold Sensitivity

##### Absence of false\-positive symmetry detection and formation by training\.

A false positive occurs whenDop⊈GtrueD\_\{\\mathrm\{op\}\}\\not\\subseteq G\_\{\\mathrm\{true\}\}; we therefore check whetherDop⊆GtrueD\_\{\\mathrm\{op\}\}\\subseteq G\_\{\\mathrm\{true\}\}holds in the tested grid\.Dop⊈GtrueD\_\{\\mathrm\{op\}\}\\not\\subseteq G\_\{\\mathrm\{true\}\}\) is a prerequisite for the framework’s utility; therefore, we first verify this empirically\. Under DyadicAxisPE / Weight Gram with all eight elements ofD4D\_\{4\}as the transform family,DopD\_\{\\mathrm\{op\}\}was evaluated across the 6 main\-text shapes×\\times3 seeds×\\times10 checkpoints \(epoch 0–2000\), yielding 0 false\-positive detections in all 180 conditions \(Figure[A10](https://arxiv.org/html/2607.03108#A6.F10)\)\. No false positives were observed, even at random initialization \(epoch 0\)\. Training progression selectively lowered the scores for transforms inGtrueG\_\{\\mathrm\{true\}\}, while theD3D\_\{3\}shape \(equilateral\_triangle\) maintained high scores across all epochs, confirming that the structural non\-detectability is a stable pattern independent of training\. Prefix GramL0L\_\{0\}also showed 0 false positives under the same conditions\. This verification is limited to these specific conditions and is not a theoretical guarantee of general PE–observable combinations\.

![Refer to caption](https://arxiv.org/html/2607.03108v1/x15.png)Figure A10:Comparison of symmetry scores between random initialization \(epoch 0\) and after training \(epoch 2000\) \(DyadicAxisPE K=12, Weight Gram,D4D\_\{4\}transforms\)\. At initialization, scores are high for all transformations \(no symmetry detected\), and after training, the reductions are larger for transformations inGtrueG\_\{\\mathrm\{true\}\}, with the low\-score detections concentrated on the true\-symmetry side\. This is consistent with the detected symmetries being formed by training rather than PE bias\.
##### Rationale forε=0\.05\\varepsilon=0\.05and threshold sensitivity\.

The Gram distanceΔ\\Deltais a normalized Frobenius distance‖A−Π⊤​A​Π‖F/‖A‖F\\\|A\-\\Pi^\{\\top\}A\\Pi\\\|\_\{F\}/\\\|A\\\|\_\{F\}; it equals 0 for exact symmetry and isO​\(1\)O\(1\)for unrelated transformations\. Being a ratio,ε=0\.05\\varepsilon=0\.05corresponds to a 5% relative Frobenius error, which is a natural threshold for normalized scores\. In ourD4D\_\{4\}transform experiments \(DyadicAxisPE K=12 / Weight Gram, epoch 2000\), non\-symmetry scores are all≥0\.24\\geq 0\.24, soε=0\.05\\varepsilon=0\.05provides a wide margin against false positives\. Table[A2](https://arxiv.org/html/2607.03108#A6.T2)compares detection counts acrossε=0\.02\\varepsilon=0\.02–0\.100\.10: false positives remain 0 forε≤0\.05\\varepsilon\\leq 0\.05, while false positives first appear atε≥0\.07\\varepsilon\\geq 0\.07\. Because different observables produce raw scores at different scales, caution is required when applying a common threshold; however, the Gram distanceΔ\\Deltaused in this work is a normalized score, andε=0\.05\\varepsilon=0\.05provides a robust operating point\.

Table A2:Detection results at different thresholdsε\\varepsilon\(DyadicAxisPE K=12 / Weight Gram /D4D\_\{4\}transforms, epoch 2000, 6 shapes×\\times3 seeds\)\.

## Appendix GNumerical Procrustes Estimation for RFF

For the RFF, analytic PE action matrices are unavailable; therefore,ρ^​\(g\)\\hat\{\\rho\}\(g\)is constructed via numerical Procrustes estimation\[[18](https://arxiv.org/html/2607.03108#bib.bib48)\]\. The procedure is as follows: \(i\) sample input points\{xi\}i=1N\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}uniformly from\[−1,1\]2\[\-1,1\]^\{2\}, \(ii\) compute transformed inputs\{g​xi\}\\\{gx\_\{i\}\\\}, \(iii\) obtain PE outputsX=\[ϕ​\(xi\)\]⊤∈ℝN×DX=\[\\phi\(x\_\{i\}\)\]^\{\\top\}\\in\\mathbb\{R\}^\{N\\times D\}andY=\[ϕ​\(g​xi\)\]⊤∈ℝN×DY=\[\\phi\(gx\_\{i\}\)\]^\{\\top\}\\in\\mathbb\{R\}^\{N\\times D\}, \(iv\) compute the SVD ofM=Y⊤​XM=Y^\{\\top\}XasM=U​Σ​V⊤M=U\\Sigma V^\{\\top\}and setρ^​\(g\)=U​V⊤∈O​\(D\)\\hat\{\\rho\}\(g\)=UV^\{\\top\}\\in O\(D\)\. The Procrustes residualrP​\(g\)=‖Y−X​ρ^​\(g\)⊤‖F/‖Y‖Fr\_\{P\}\(g\)=\\\|Y\-X\\hat\{\\rho\}\(g\)^\{\\top\}\\\|\_\{F\}/\\\|Y\\\|\_\{F\}quantifies the estimation quality \(Table[1](https://arxiv.org/html/2607.03108#S4.T1)\)\. In our experiments,N=512N=512was used as default\.