Orthogonal Dendritic Intrinsic Networks: An Architecture for Significance-Ordered, Orthogonal Latent Spaces

# An Architecture for Significance-Ordered, Orthogonal Latent Spaces
Source: [https://arxiv.org/html/2607.05653](https://arxiv.org/html/2607.05653)
## Orthogonal Dendritic Intrinsic Networks: An Architecture for Significance\-Ordered, Orthogonal Latent Spaces

\\nameJeanie Schreiber\\emailjschrei@gmu\.edu \\addrDepartment of Mathematical Sciences George Mason University Fairfax, VA 22030, USA\\nameTyrus Berry\\emailtberry@gmu\.edu \\addrDepartment of Mathematical Sciences George Mason University Fairfax, VA 22030, USA\\nameZeeshan Ahmed\\emailzeeshan\.ahmed@nist\.gov \\addrSensor Science Division, Physical Measurement Laboratory NIST Gaithersburg, MD 20899, USA

###### Abstract

Principal Component Analysis or PCA\-like properties \(orthogonality, variance ranking\) are seldom realized in deep autoencoder architectures\. In this work, we present ODIN \(Orthogonal Dendritic Intrinsic Network\), a novel autoencoder architecture that recovers PCA\-like latent structure in a fully non\-linear regime\. By incorporating a set of geometric constraints directly into the training objective, ODIN encourages latent dimensions to be mutually orthogonal and ordered by explained variance, mirroring the interpretable decomposition of PCA while retaining the expressive power of deep networks\. We provide theoretical grounding for these constraints and demonstrate their compatibility with standard encoder\-decoder frameworks\. We also establish empirical results for both synthetic and real world datasets, establishing a principled path toward interpretable, structured feature learning and dimensionality reduction\.

Keywords:Autoencoder, PCA, Non\-linear PCA, Dimensionality Reduction, Feature Learning, interpretable ML

## 1Introduction

Traditional autoencoders consisting of an encoder and a decoder compress data into a latent space of lower dimensionality before the decoder attempts to reconstruct the original input from the latent representation\. The loss function, usually based on reconstruction error, drives the model to learn efficient latent embeddings encoding important features of the data\. While effective, the standard autoencoder architecture places no structural constraints on the latent space, often utilizing the entire latent representation for inference and understanding\. With no notion of ordering or separability, the resulting features are uniformly smeared across the latent space, making it challenging to interpret which latent dimensions correspond to meaningful input characteristics\.

The inherent ambiguity of latent space organization presents a common challenge in applying autoencoders for dimensionality reduction\. The traditional autoencoder architecture produces latent representations with entangled, non\-orthogonal directions that vary unpredictably across training runs\. This instability complicates both reproducibility and interpretation\.

The Orthogonal Dendritic Intrinsic Network \(ODIN\) architecture addresses these limitations through a key innovation inspired by biology\. First, ODIN employs a dendritic structure that enforces hierarchical importance ranking of latent dimensions\. By restricting decoder access to cumulative subsets of latent variables \(e\.g\., only the firstkkcomponents\), the network learns to prioritize directions that maximize reconstruction fidelity when progressively accrued\. Additionally, strict latent space orthogonality is enforced through geometric constraints in the learning function, ensuring consistent axis alignment across training sessions\. Together, these mechanisms enforce orthogonal and importance\-ordered latent dimensions, mirroring PCA’s eigenvalue ranking while maintaining neural networks’ non\-linear expressive power\.

Critically, ODIN achieves this disentanglement in a completely unsupervised manner, unlike methods requiring labeled data or predefined feature hierarchies\. The orthogonalized latent directions separate intrinsic attributes of the input data corresponding to statistically independent variation modes, enabling researchers to:

1. 1\.Identify dominant variation modes through variance\-ranked latent dimensions\.
2. 2\.Perform stable feature ablation studies by systematically removing less important axes\.
3. 3\.Achieve reproducible latent assignments across independent training runs, enabling reliable cross\-experiment comparisons using consistent latent space coordinate systems\.

For AI explainability, ODIN’s architecture creates an interpretable bridge between raw data and latent representations\. The orthogonalized dimensions permit linear decomposition analyses typically reserved for PCA, while the non\-linear encoding preserves complex relationships that linear methods might miss\. This dual capability makes ODIN particularly valuable for domains requiring both model transparency and high\-dimensional pattern recognition, such as biomedical signal processing or materials science characterization\.

## 2Background

Autoencoders are a powerful tool for dimensionality reduction, offering flexibility in capturing non\-linear patterns that traditional PCA cannot\. A comprehensive review of autoencoder architectures and their variants byLiet al\.\([2023](https://arxiv.org/html/2607.05653#bib.bib13)\)highlights the breadth of advancements in this field\. However, the review mentions a notable gap: most autoencoder designs do not attempt to replicate key features of PCA, such as orthogonality and variance\-based sorting of latent space components\. These properties are crucial for interpretability and systematic feature extraction, yet they remain largely unaddressed in autoencoder research\.

The fundamental connection between linear autoencoders and PCA was rigorously established byPlaut \([2018](https://arxiv.org/html/2607.05653#bib.bib5)\), who demonstrated that single\-hidden\-layer linear autoencoders span principal subspaces equivalent to PCA loading vectors\. However, just as PCA solutions are unique only up to orthogonal transformations, gradient descent can converge to multiple equivalent subspaces that all achieve the same minimal reconstruction error\. Moreover, this equivalence breaks completely when introducing non\-linear activations, leaving open the question of how to preserve PCA\-like interpretability in deep architectures\.

Kernel PCA may be regarded as the most natural extension of PCA to non\-linear data by way of kernel transformations rather than activation functions\. Kernel PCA projects data into higher\-dimensional spaces where non\-linear patterns become linear, enabling variance\-based sorting akin to PCA while accommodating complex relationshipsPei and others \([2018](https://arxiv.org/html/2607.05653#bib.bib14)\); Majumdar \([2021](https://arxiv.org/html/2607.05653#bib.bib15)\); Pei \([2017](https://arxiv.org/html/2607.05653#bib.bib16)\)\. Despite its effectiveness in handling non\-linear data, Kernel PCA lacks the generative capabilities of autoencoders and requires explicit kernel selection, limiting its adaptability across diverse datasets\.

Recent advances in neural implementations of PCA\-like dimensionality reduction reveal persistent challenges in combining non\-linear expressivity with structured latent spaces\.Renet al\.\([2024](https://arxiv.org/html/2607.05653#bib.bib3)\)developed an autoencoder architecture using Cayley transforms to enforce latent space orthogonality, achieving non\-linear feature extraction comparable to Kernel\-PCA\. Parallel efforts byWanget al\.\([2019](https://arxiv.org/html/2607.05653#bib.bib4)\)introduced orthogonal constraints through loss function regularization, minimizing the discrepancy between latent vector correlations and the identity matrix\. While these methods successfully separate intrinsic data attributes into distinct latent components and can be effective for clustering tasks, they lack mechanisms for importance\-based feature ordering\.

Recent work byPhamet al\.\([2022](https://arxiv.org/html/2607.05653#bib.bib9)\)directly addresses latent component independence and importance ordering through their PCA\-Autoencoder framework\. Their method introduces two innovations:

1. 1\.Covariance minimization: A loss term penalizing off\-diagonal elements of the latent space covariance matrix to enforce statistical independence\.
2. 2\.Sequential latent expansion: Training a progression of autoencoders where each iteration adds new latent dimensions while freezing previously learned components\.

While this approach successfully separates latent features on datasets like CelebA and synthetic ellipse images, it suffers from critical limitations\. The sequential training protocol requires carefully scheduled training stages and can be sensitive to transition timing, potentially limiting global optimization\. Because each stage depends on the previous one, the method is not fully end\-to\-end differentiable in the traditional sense and can introduce inefficiencies or overfitting if stage boundaries are not tuned precisely\. More fundamentally, the implicit assumption that frozen components retain maximal variance contributions breaks down in practice, as later training stages often reallocate explanatory power to newly added dimensions\.

Perozoet al\.\([2024](https://arxiv.org/html/2607.05653#bib.bib11)\)recent introduction of POLCA Net represents a significant advancement in neural implementations of PCA\-like dimensionality reduction\. This architecture makes use of a multi\-objective loss function that includes orthogonality loss and variance regularization terms\. Similar to the work of Pham and Ladjal, the orthogonality loss term penalizes off\-diagonal elements of the latent space covariance matrix to enforce independence while maintaining non\-linearity\. However, ordering of latent space components is achieved through statistical estimates and indirect variance maximization\. In this respect, POLCA Net shares a fundamental characteristic with other variance\-based regularization approaches such as the Autoencoder with Ordered Variance \(AEO\)Augustineet al\.\([2024](https://arxiv.org/html/2607.05653#bib.bib27)\), which modifies the reconstruction loss with a variance regularization term that explicitly enforces monotonically decreasing variance across latent dimensions\. Through the lens of model regularization, ODIN’s dendritic structure may be viewed as an implicit structural prior111See alsoChen and Fuge \([2024](https://arxiv.org/html/2607.05653#bib.bib28)\), while its orthogonality penalty serves as an explicit constraint on latent geometry\. We argue in the next section that this coupling yields latent representations that are both more generalizable and more interpretable in the non\-linear setting than those produced by variance\-based approaches\.

Another approach for constraining neural network weights involves Riemannian gradient descent, which leverages Riemannian geometry and optimization techniques on matrix manifolds to generalize standard gradient descent methodsHarandi and Fernando \([2016](https://arxiv.org/html/2607.05653#bib.bib22)\); Feiet al\.\([2025](https://arxiv.org/html/2607.05653#bib.bib26)\)\. This framework enables the intrinsic enforcement of desirable structural properties such as orthogonality, positive definiteness, or fixed\-rank constraints by performing optimization directly on the constrained manifold rather than in unconstrained Euclidean space\. For example,Harandi and Fernando \([2016](https://arxiv.org/html/2607.05653#bib.bib22)\)enforce orthogonality in neural network weight matrices by optimizing on the Stiefel manifold \(the set of matrices with orthonormal columns\) using retraction operators such as QR decomposition to maintain the constraint after each gradient step\. While such Riemannian optimization methods can strictly enforce orthogonality during training and have demonstrated convergence guarantees in both convex and nonconvex settings, for the purposes of comparison to more standard backpropagation\-based autoencoders, here we opt for simpler loss function penalties to enforce orthogonality\.

An alternative school of thought for implementing PCA\-like operations involves Hebbian learning, a biologically plausible network strategy that requires only pre\- and post\-synaptic activity correlations\. The fundamental update rule followsΔ​w∝y​⟨xi⟩\\Delta w\\propto y\\langle x\_\{i\}\\rangle, where synaptic weights increase proportional to correlations between pre\-synaptic\(xi\)\(x\_\{i\}\)and post\-synaptic\(y\)\(y\)activitiesOlshausen \([1998](https://arxiv.org/html/2607.05653#bib.bib21)\)\. Extensions such as Sanger’s rule generalize this framework to extract multiple principal components through a hierarchical deflation procedure, sequentially learning components ordered by eigenvalue magnitude\. In contrast to autoencoder approaches which impose orthogonality as explicit constraints \(*e\.g\.*cosine similarity or Stiefel layers in Riemannian Optimization\), Hebbian methods learn through local correlation\-based rules with exponential convergence\. However, Hebbian networks are consistently outperformed by standard backpropagation\-trained counterparts and remain insufficient for the demands of modern deep learning pipelinesAmatoet al\.\([2019](https://arxiv.org/html/2607.05653#bib.bib25)\); Miconi \([2021](https://arxiv.org/html/2607.05653#bib.bib24)\); Kierkegaard and Morfeldt Gadler \([2023](https://arxiv.org/html/2607.05653#bib.bib23)\)\.

### Variational Autoencoders

Variational Autoencoders \(VAEs\) are widely recognized as versatile generative models and may be considered as a natural point of comparison for ODIN, primarily because of their ability to map high\-dimensional data into a structured latent space\. VAEs operate by transforming latent variables drawn from a simple prior distribution, typically a multivariate Gaussian, into data\-like samples through a learned generative network\. During training, an encoder approximates the posterior distribution over latent variables, while a decoder learns to reconstruct input data from these latent representations\. These probabilistic constraints generally lead to what are called “disentangled” representations, in which a single latent variable is sensitive to changes in a single generative factor and is relatively invariant to changes in other factorsBengioet al\.\([2013](https://arxiv.org/html/2607.05653#bib.bib19)\)\. In particular, recent findings, such as byIchikawa and Hukushima \([2024](https://arxiv.org/html/2607.05653#bib.bib20)\), demonstrate that VAEs begin learning entangled representations and gradually acquire disentangled latent factors during training\. In addition, contemporary work byGhojoghet al\.\([2021](https://arxiv.org/html/2607.05653#bib.bib7)\)revealed theoretical connections between PCA, factor analysis, and variational autoencoders \(VAEs\)\. See alsoRolineket al\.\([2019](https://arxiv.org/html/2607.05653#bib.bib8)\)who demonstrate that the diagonal posterior approximation in VAEs induces local orthogonality patterns resembling non\-linear PCA\.

VAEs model the data generation process asp​\(x\)=∫p​\(x\|z\)​p​\(z\)​𝑑zp\(x\)=\\int p\(x\|z\)p\(z\)dz, where the latent variablezzis drawn from a prior distributionp​\(z\)p\(z\), typically𝒩​\(0,I\)\\mathcal\{N\}\(0,I\)\. During training, exact inference of the posteriorp​\(z\|x\)p\(z\|x\)is intractable, necessitating approximation through a variational distributionq​\(z\|x\)q\(z\|x\)\. The encoder network parameterizes this approximate posterior by outputting distribution parameters \(specifically,μ​\(x\)\\mu\(x\)andσ2​\(x\)\\sigma^\{2\}\(x\)for the Gaussian case\) such thatq​\(z\|x\)=𝒩​\(z∣μ​\(x\),σ2​\(x\)\)q\(z\|x\)=\\mathcal\{N\}\(z\\mid\\mu\(x\),\\sigma^\{2\}\(x\)\)\. To enable gradient\-based optimization while maintaining stochasticity, the reparameterization trick expresses the latent code asz=μ​\(x\)\+σ​\(x\)⊙ϵz=\\mu\(x\)\+\\sigma\(x\)\\odot\\epsilon, whereϵ∼𝒩​\(0,I\)\\epsilon\\sim\\mathcal\{N\}\(0,I\)represents external randomness independent of the network parameters\. This formulation allows differentiation through the sampling operation, as gradients can flow throughμ​\(x\)\\mu\(x\)andσ​\(x\)\\sigma\(x\)while treatingϵ\\epsilonas constant\. The training objective maximizes the Evidence Lower Bound \(ELBO\),

ELBO=𝔼q​\(z\|x\)\[logp\(x\|z\)\]−KL\(q\(z\|x\)\|\|p\(z\)\)\\text\{ELBO\}=\\mathbb\{E\}\_\{q\(z\|x\)\}\[\\log p\(x\|z\)\]\-\\text\{KL\}\(q\(z\|x\)\|\|p\(z\)\)where the first term maximizes data log\-likelihood and the second term \(called the Kullback–Leibler \(KL\) divergence\) penalizes deviation of the approximate posteriorq​\(z\|x\)q\(z\|x\)from the priorp​\(z\)p\(z\)\. This stochastic framework is essential for the generative capacity of VAEs, enabling both principled uncertainty quantification and the ability to sample novel data points from the learned latent spaceKingmaet al\.\([2014](https://arxiv.org/html/2607.05653#bib.bib31)\); Liuet al\.\([2025](https://arxiv.org/html/2607.05653#bib.bib30)\)\.

In practice, VAEs are often optimized using theβ\\beta\-VAE objective, which introduces a tunable parameterβ\\betato the KL divergence term in the ELBO loss to balance the two competing objectivesHigginset al\.\([2016](https://arxiv.org/html/2607.05653#bib.bib32),[2017](https://arxiv.org/html/2607.05653#bib.bib2)\)\. This adjustment allows for controlled trade\-offs between error fidelity and latent space regularization\.β\\beta\-VAE’s are capable of providing significant improvements over traditional VAE’s on tasks like image generation, and are today widely considered as state\-of\-the\-art models for representation learningHigginset al\.\([2017](https://arxiv.org/html/2607.05653#bib.bib2)\); Locatelloet al\.\([2019](https://arxiv.org/html/2607.05653#bib.bib38)\)\.

However, this flexibility comes with a well\-documented drawback: asβ\\betaincreases, the variational posterior tends to converge toward the prior during optimization, effectively rendering the latent variables uninformative of the input data\. This detrimental effect, known as “posterior collapse”, undermines the expressive power of the latent space and limits the effectiveness of VAEs in representation learning tasksHeet al\.\([2019](https://arxiv.org/html/2607.05653#bib.bib34)\)\. VAEs struggle particularly with capturing low\-variance directions, as these dimensions are most susceptible to collapse\. Identifying an appropriateβ\\betavalue remains a nontrivial challenge, as this hyperparameter must be carefully tuned to achieve an effective compromise between two competing objectivesFilet al\.\([2021](https://arxiv.org/html/2607.05653#bib.bib33)\)\.

In addition, while implicit probabilistic constraints enable partial disentanglement, they do not enforce any explicit ordering of latent variables\. The learned latent space exhibits permutation symmetry, meaning dimensions can be arbitrarily reordered without affecting reconstruction quality\. Empirically, this manifests as inconsistent feature assignments across training runs\. For example, dimension 3 in one instantiation might capture a high\-importance feature, while in another run this same information migrates to dimension 5\. This inconsistency severely limits utility for comparing results across experiments or performing systematic ablation studies\.

Despite their prominence in generative modeling and representation learning, VAEs are not natively designed for supervised downstream tasks such as regression, and their application in these settings remains comparatively under\-exploredZhaoet al\.\([2019](https://arxiv.org/html/2607.05653#bib.bib35)\)\. A key contributing factor is that the latent representations learned by VAEs are not guaranteed to be discriminative or predictively informative; uninformative latent features are a well\-documented failure mode that arises when the balance between reconstruction and regularization is not carefully managedZhaoet al\.\([2017](https://arxiv.org/html/2607.05653#bib.bib36)\)\. Benchmarking studies have also demonstrated that the large number of hyperparameters requiring tuning in VAE\-based models \(includingβ\\beta\) directly impacts downstream predictive performance, making consistent and reliable results difficult to achieve across varied datasets and tasksEltageret al\.\([2023](https://arxiv.org/html/2607.05653#bib.bib37)\)\.

ODIN, by contrast, avoids excessive hyperparameter tuning and achieves latent space disentanglement through architectural constraints rather than probabilistic regularization\. Moreover, the dendritic decoder structure provides explicit variance\-based hierarchical ranking of latent features, eliminating permutation symmetry and allowing for consistent, reproducible latent mode assignments across independent training runs\.

## 3ODIN

The architecture introduced in ODIN modifies the conventional autoencoder setup by incorporating multiple ‘dendritic’ decodings alongside the standard full decoding\. Each dendritic decoding is restricted to a cumulative subset of the full latent representation\. The first decoding reconstructs the data using only the first latent component, the second decoding uses the first two, and so on until all components are included \( Figure[1](https://arxiv.org/html/2607.05653#S3.F1)\)\. Specifically, given a datasetX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}consider an encoding into akk\-dimensional latent space, yielding latent representationZ=\[z1,z2,…​zk\]∈ℝN×kZ=\[z\_\{1\},z\_\{2\},\\dots z\_\{k\}\]\\in\\mathbb\{R\}^\{N\\times k\}\. For eachj∈\{1,…,k\}j\\in\\\{1,\\dots,k\\\}, letZ\[j\]=Z\[:,1:j\]=\[z1,…,zj\]Z\_\{\[j\]\}=Z\[:,1:j\]=\[z\_\{1\},\\dots,z\_\{j\}\]denote the firstjjcolumns ofZZ\. IfD​e​cDecdenotes a decoder mapping from the latent space to the data space, then thejt​hj^\{th\}reconstructionX^j\\hat\{X\}\_\{j\}produced by the decoder is defined as:

X^j=D​e​c​\(Z\[j\]\)\.\\hat\{X\}\_\{j\}=Dec\(Z\_\{\[j\]\}\)\.\(1\)In other words, at decoding stepjj, the decoder is presented with only the firstjjlearned features so thatX^j\\hat\{X\}\_\{j\}approximatesXXusing only the firstjjlatent components\.

![Refer to caption](https://arxiv.org/html/2607.05653v1/x1.png)Figure 1:Architecture of ODIN\. The encoder maps input data to a latent representationZ=\[z1,…,zk\]Z=\[z\_\{1\},\\dots,z\_\{k\}\], from which each dendritic branch produces a reconstruction using cumulatively larger number of latent variables\. The total loss combines each reconstruction term with an orthogonality penalty that encourages de\-correlation among latent dimensions\.Each dendritic decoding in the ODIN network produces output reconstructions of the same size as the original input data, regardless of how many latent dimensions it accesses\. During training, the objective error \(according to the loss function\) is calculated for each dendritic decoding individually, and these errors are then aggregated into a combined loss\. Specifically, given reconstructionsX^j\\hat\{X\}\_\{j\}from subsetsZ\[j\]Z\_\{\[j\]\}of the latent space, the mean squared error aggregate is

ℒDend=∑j=1kLj=∑j=1kℒMSE​\(X,X^j\)\\mathcal\{L\}\_\{\\text\{Dend\}\}=\\sum\_\{j=1\}^\{k\}L\_\{j\}=\\sum\_\{j=1\}^\{k\}\\mathcal\{L\}\_\{\\text\{MSE\}\}\(X,\\hat\{X\}\_\{j\}\)\(2\)This total loss guides the backpropagation process, forcing each decoding to maximize reconstruction quality given its available features\. Through this hierarchical setup, the network learns to allocate representational capacity such that earlier decodings capture the most meaningful variation, enforcing an implicit ordering of latent importance such that the earliest are most critical for faithful reconstruction\. In effect, this approach mimics the PCA variance sorting in which the principal components give a sorted set of directions with maximal variance\.

Although the totalℒDend\\mathcal\{L\}\_\{\\mathrm\{Dend\}\}error used during backpropagation increases with the introduction of dendritic loss terms, bounding it away from zero \(see appendix[B](https://arxiv.org/html/2607.05653#A2)\), faithful reconstructions remain attainable by relying on the output of the final \(full\) decoding\. When given access to all latent variables \(*i\.e\.*, all degrees of variation\) the full decoding can, in principle, reconstruct the entire data distribution with a theoretical lower bound of zero reconstruction error\. This ensures that, despite the added regularization, the ODIN model continues to produce reconstructions that are both highly accurate and visually consistent with the original data\.

To implement dendritic decodings in practice, there are generally two approaches\. The first uses an ensemble ofkkindependent decodersD​e​c1,D​e​c2,…,D​e​ck\{Dec\_\{1\},Dec\_\{2\},\\dots,Dec\_\{k\}\}, where decoderD​e​cjDec\_\{j\}receives a fixedjj\-dimensional input and produces reconstructionX^j=D​e​cj​\(Z\[j\]\)\\hat\{X\}\_\{j\}=Dec\_\{j\}\(Z\_\{\[j\]\}\)\. The second approach employs a single shared decoderD​e​cDeccalledkktimes on cumulatively zero\-masked versions of the full latent representation,*i\.e\.*X^j=D​e​c​\(Z\[j\]⊕𝟎j\+1:k\)\\hat\{X\}\_\{j\}=Dec\(Z\_\{\[j\]\}\\oplus\\mathbf\{0\}\_\{j\+1:k\}\), where⊕\\oplusdenotes concatenation with the zero\-padding matrix𝟎j\+1:k∈ℝN×\(k−j\)\\mathbf\{0\}\_\{j\+1:k\}\\in\\mathbb\{R\}^\{N\\times\(k\-j\)\}\.

In informal comparisons across both linear and non\-linear experiments, we observe empirically that the single\-decoder formulation achieves similar reconstruction performance to the independent decoder ensemble while reducing the total number of trainable parameters by a factor ofkk\. This parameter efficiency improves training stability, accelerates convergence, and facilitates scaling to higher latent dimensionalities without excessive memory overhead\.

To further emulate PCA principle components, the cumulative structure of dendritic decoding works in concert with orthogonality constraints to enforce non\-overlapping features and further stabilize axis alignment\. FollowingPerozoet al\.\([2024](https://arxiv.org/html/2607.05653#bib.bib11)\), ODIN introduces an explicit orthogonality loss term:

ℒorth=∑i<jSi​j2\\mathcal\{L\}\_\{\\text\{orth\}\}=\\sum\_\{i<j\}S\_\{ij\}^\{2\}\(3\)WhereSi​jS\_\{ij\}is the\(i,j\)t​h\(i,j\)^\{th\}component ofS=ZT​ZS=Z^\{T\}Z\. This term penalizes off\-diagonal entries in the latent similarity matrix, thereby enforcing feature disentanglement\. In addition, we incorporate a moving average into this process to account for batching of large datasets\. This geometric approach ensures that latent dimensions remain statistically independent without relying on probabilistic approximations\.

The total training objective is thus defined as a multi\-objective loss,

ℒ​\(W,X\)=ℒDend\+λorth⋅ℒorth\\mathcal\{L\}\(W,X\)=\\mathcal\{L\}\_\{\\text\{Dend\}\}\+\\lambda\_\{\\text\{orth\}\}\\cdot\\mathcal\{L\}\_\{\\text\{orth\}\}\(4\)whereλorth\\lambda\_\{\\text\{orth\}\}weights the orthogonality constraint\. The result is a latent space whose dimensions are both disentangled and importance\-ranked, facilitating downstream tasks such as ablation studies and robust explainability analyses\.

In terms of latent space ordering, ODIN adopts architectural restrictions similar toPhamet al\.\([2022](https://arxiv.org/html/2607.05653#bib.bib9)\)sequential training approach but eliminates the need for training multiple networks or freezing previously learned components\. ODIN achieves significance\-based ordering within a single, jointly trained network where structural restrictions on decoder access encourage the model to prioritize the most informative latent dimensions\.

Compared to theβ\\beta\-VAE’s method of enforcing independence probabilistically, ODIN’s approach withλorth\\lambda\_\{\\text\{orth\}\}directly penalizes correlations in the latent space via measurable covariance properties\. This makes it less prone to failures like posterior collapse and more robust to changes in the parameter\. In practice, we found tuning the parameterλorth\\lambda\_\{\\text\{orth\}\}to be less sensitive compared toβ\\betainβ\\beta\-VAE’s, making it easier to adjust\. Sinceλorth\\lambda\_\{\\text\{orth\}\}directly scales the orthogonality loss, we often start with a moderate value≈1\\approx 1and then adjust it based on the relative importance of orthogonality versus reconstruction accuracy\. Larger batch sizes and datasets with higher intrinsic correlations usually require increasingλorth\\lambda\_\{\\text\{orth\}\}to maintain effective decorrelation in the latent space\.

We also remark that it is possible to augment a VAE with dendritic\-style decoding to induce an importance ordering over latent factors\. However, we found this approach to be less robust in practice than ODIN\. Moreover, we were motivated to find a method that provably generalized PCA in the sense that it recovered PCA when applied to an affine network\. We were able to obtain these theoretical results for ODIN as shown below\.

Exploring a dendritic approach to VAEs remains an interesting direction for further research\. On one hand, the VAE framework offers well\-established theoretical grounding for latent space regularization and has been shown to be effective at disentangling latent dimensions in its own rightHigginset al\.\([2017](https://arxiv.org/html/2607.05653#bib.bib2)\); Rolineket al\.\([2019](https://arxiv.org/html/2607.05653#bib.bib8)\)\. On the other hand, a direct cosine\-similarity regularizer offers a transparent mechanism for enforcing orthogonality among latent directions, without requiring careful selection of theβ\\betaparameter across datasets and regimes\. In practice, we found that combining VAE\-style regularization with dendritic decoding leads to a significant degradation in the orthogonality otherwise present in either VAE or ODIN separately\. As such, for practitioners who simultaneously desire both importance ordering and latent disentanglement, ODIN represents a more natural and robust choice under the current framework\.

### 3\.1The True Loss Function for PCA

The standard linear algebra view of PCA takes the form:

maxW⁡Tr​\(W⊤​X⊤​X​W\),subject to​W⊤​W=Ik×k\.\\max\_\{W\}\\text\{Tr\}\(W^\{\\top\}X^\{\\top\}XW\),\\quad\\text\{subject to \}W^\{\\top\}W=I\_\{k\\times k\}\.\(5\)Note thatTr​\(WT​XT​X​W\)=‖X​W‖F2\\text\{Tr\}\(W^\{T\}X^\{T\}XW\)=\|\|XW\|\|\_\{F\}^\{2\}is the variance of the projection into the latent space\. For dataX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}, PCA finds an orthogonal matrixW=\[w1,w2,…,wk\]∈ℝn×kW=\[w\_\{1\},w\_\{2\},\\dots,w\_\{k\}\]\\in\\mathbb\{R\}^\{n\\times k\}whose columns span the directions of maximal projected variance\. The following standard result characterizes the solution set of \([5](https://arxiv.org/html/2607.05653#S3.E5)\) being any projection onto the span of the firstkkright singular vectors ofXX\(for completeness the proof is included in Appendix[A](https://arxiv.org/html/2607.05653#A1)\)\.

###### Theorem 1

LetX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}, with singular value decompositionX=U​S​V⊤X=USV^\{\\top\}, then the solution to equation[5](https://arxiv.org/html/2607.05653#S3.E5)isW=V\[k\]​Q=V​In×k​QW=V\_\{\[k\]\}Q=VI\_\{n\\times k\}QwhereQ∈ℝk×kQ\\in\\mathbb\{R\}^\{k\\times k\}is any orthogonal matrix\.

While Theorem[1](https://arxiv.org/html/2607.05653#Thmtheorem1)is typically taken as the theoretical backbone of PCA, it leaves an important degree of freedom unresolved: the solutionWWis only unique up to an arbitrary orthogonal rotationQQ\. In this framework, the individual columns ofWWneed not align with the principal componentsv1,…,vkv\_\{1\},\\dots,v\_\{k\}, nor are they ordered by importance\. Here we address both issues by shifting to a reconstruction\-based objective rather than variance in the projected latent space\. We show that minimizing this new loss not only recovers the PCA subspace but also guarantees ordering within the projection by importance\. While achieving ordering in the PCA algorithm is trivial, it is critical that the loss function capture this ordering so that we can use the loss function to generalize PCA to the nonlinear setting\.

As a starting point, we recall the well\-known equivalence between the maximum\-variance and minimum\-reconstruction formulations of PCA \(the proof is in appendix[A](https://arxiv.org/html/2607.05653#A1)\):

###### Proposition 2

For anyX∈ℝN×n,W∈ℝn×kX\\in\\mathbb\{R\}^\{N\\times n\},W\\in\\mathbb\{R\}^\{n\\times k\}we have the following equivalence:

maxW⊤​W=I⁡Tr​\(W⊤​X⊤​X​W\)=minW⁡‖X⊤−W​W⊤​X⊤‖F2\\max\_\{W^\{\\top\}W=I\}\\text\{Tr\}\(W^\{\\top\}X^\{\\top\}XW\)=\\min\_\{W\}\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}

In other words, the maximum trace PCA objective is equivalent to minimizing the squared residuals of the orthogonal projections of the data∑i=1N‖xi−pi‖2\\sum\_\{i=1\}^\{N\}\\\|x\_\{i\}\-p\_\{i\}\\\|^\{2\}, wherepi=W​W⊤​xip\_\{i\}=WW^\{\\top\}x\_\{i\}is the projection ofii’th data pointxix\_\{i\}onto thekk\-dimensional subspace spanned by the columns ofWW\. Both problems share the same solution set, recovering the subspace spanned by the topkkprincipal components but, as noted above, without pinning down the orientation within that subspace\.

To resolve this ambiguity and recover the principal components in their natural order, we break the rotational symmetry of the solution space by introducing a sequence of nested reconstruction terms\. Specifically, we augment the objective with terms involvingX​W\[i\]XW\_\{\[i\]\}fori=1,…,ki=1,\\dots,k, whereW\[i\]W\_\{\[i\]\}denotes the submatrix formed by the firstiicolumns ofWW\. Each such term measures how well the firstiidirections reconstructXX, penalizing solutions that fail to prioritize higher\-variance directions first\. This is exactly the dendritic idea in the linear setting; since eachX​W\[i\]XW\_\{\[i\]\}represents the signal available to theii\-th dendrite for reconstructingXX\. The resulting objective becomes,

maxW⊤​W=Ik×k​∑i=1k‖X​W\[i\]‖F2=minW​∑i=1k‖X⊤−W\[i\]​W\[i\]⊤​X⊤‖F2\.\\max\_\{W^\{\\top\}W=I\_\{k\\times k\}\}\\sum\_\{i=1\}^\{k\}\|\|XW\_\{\[i\]\}\|\|\_\{F\}^\{2\}=\\min\_\{W\}\\sum\_\{i=1\}^\{k\}\\\|X^\{\\top\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}\.
Under the assumption that the nonzero singular values ofXXare distinct, the difference between optimizing over the fullkk\-dimensional subspace compared to optimizing over each individual subspace simultaneously becomes explicit\. Whens1\>s2\>⋯\>sk\>0s\_\{1\}\>s\_\{2\}\>\\cdots\>s\_\{k\}\>0, each optimalii\-dimensional principal subspace is uniquely determined asspan​\(V\[i\]\)\\mathrm\{span\}\(V\_\{\[i\]\}\)\. Since there is no rotational freedom within any degenerate right singular space, the only way to simultaneously satisfy

span​\(W\[i\]\)=span​\(V\[i\]\)for all​i=1,…,k,\\mathrm\{span\}\(W\_\{\[i\]\}\)=\\mathrm\{span\}\(V\_\{\[i\]\}\)\\quad\\text\{for all \}i=1,\\ldots,k,\(6\)is for the columns ofWWto align exactly withv1,…,vkv\_\{1\},\\ldots,v\_\{k\}up to individual sign changes\. That is, the distinct spectrum forces the rotational degrees of freedom to collapse entirely, leaving sign flips as the only residual freedom in the joint minimizer\.

The following theorem characterizes the minimizer as the correctly ordered PCA variables, with proof provided in appendix[A](https://arxiv.org/html/2607.05653#A1):

###### Theorem 3

LetX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}, with singular value decompositionX=U​S​V⊤X=USV^\{\\top\}and distinct singular valuess1\>s2\>⋯\>sk\>0s\_\{1\}\>s\_\{2\}\>\\cdots\>s\_\{k\}\>0, then

k​s12\+\(k−1\)​s22\+⋯\+2​sk−12\+sk2=minW∈ℝn×k​∑i=1k‖X⊤−W\[i\]​W\[i\]⊤​X⊤‖F2\.ks\_\{1\}^\{2\}\+\(k\-1\)s\_\{2\}^\{2\}\+\\cdots\+2s\_\{k\-1\}^\{2\}\+s\_\{k\}^\{2\}=\\min\_\{W\\in\\mathbb\{R\}^\{n\\times k\}\}\\sum\_\{i=1\}^\{k\}\\\|X^\{\\top\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}\.and the minimum is achieved whenW=V\[k\]​Σ=V​In×k​ΣW=V\_\{\[k\]\}\\Sigma=VI\_\{n\\times k\}\\Sigma, whereΣ=diag​\(ε1,…,εk\)\\Sigma=\\mathrm\{diag\}\(\\varepsilon\_\{1\},\\dots,\\varepsilon\_\{k\}\), withεj=±1\\varepsilon\_\{j\}=\\pm 1\.

### 3\.2Equivalence to PCA

Having established the reconstruction\-based formulation of PCA and its ordered solution, we now turn to the central theoretical result of this work\. That is, in the linear regime, ODIN is provably equivalent to PCA\. Specifically, we show that when the encoder and decoder are constrained to linear maps, the ODIN training objective reduces exactly to the nested reconstruction loss derived above, and the resulting weight matrices converge to the principal components of the data in their natural order\.

Consider that, for a purely linear autoencoder, the*encoder*of the network can be represented asZ=X​W\+bencZ=XW\+b^\{\\text\{enc\}\}, whereW=\[w1,w2,…,wk\]∈ℝn×kW=\[w\_\{1\},w\_\{2\},\\dots,w\_\{k\}\]\\in\\mathbb\{R\}^\{n\\times k\}is the weight matrix of the encoder, with bias termbenc∈ℝkb^\{\\text\{enc\}\}\\in\\mathbb\{R\}^\{k\}\. Likewise the*decoder*reconstructs the input asX^=D​e​c​\(Z\)=Z​W⊤\+bdec\\hat\{X\}=Dec\(Z\)=ZW^\{\\top\}\+b^\{\\text\{dec\}\}, whereX^∈ℝN×n\\hat\{X\}\\in\\mathbb\{R\}^\{N\\times n\}is the reconstructed data andW⊤∈ℝk×nW^\{\\top\}\\in\\mathbb\{R\}^\{k\\times n\}is the weight matrix of the decoder\.

To speed up convergence we mean\-center our data prior to encoding, rendering the bias terms identically zero so that we may omit them without loss of generalityPlaut \([2018](https://arxiv.org/html/2607.05653#bib.bib5)\)\. The mappings simplify to:

Z=X​W,X^=Z​W⊤=X​W​W⊤\.Z=XW,\\quad\\hat\{X\}=ZW^\{\\top\}=XWW^\{\\top\}\.Using mean squared error objective, the loss function then becomes:

ℒMSE​\(X,X^\)\\displaystyle\\mathcal\{L\}\_\{\\text\{MSE\}\}\(X,\\hat\{X\}\)=minW⁡‖X−X​W​W⊤‖2\\displaystyle=\\min\_\{W\}\\\|X\-XWW^\{\\top\}\\\|^\{2\}=minW⁡‖X⊤−W​W⊤​X⊤‖2\.\\displaystyle=\\min\_\{W\}\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|^\{2\}\.
As outlined above, ODIN augments the standard autoencoder loss function with dendritic decodingsX^i=D​e​c​\(Z\[i\]\)\\hat\{X\}\_\{i\}=Dec\(Z\_\{\[i\]\}\)produced by zeroing out columns ofZZ\. That is, we haveZ\[i\]=\[X​W\]\[i\]=X​\[W\]\[i\]Z\_\{\[i\]\}=\[XW\]\_\{\[i\]\}=X\{\[W\]\_\{\[i\]\}\}, and thus:

X^i=X​W\[i\]​W⊤=X​∑s=1iws​ws⊤=X​W\[i\]​W\[i\]⊤\.\\hat\{X\}\_\{i\}=XW\_\{\[i\]\}W^\{\\top\}=X\\sum\_\{s=1\}^\{i\}w\_\{s\}w\_\{s\}^\{\\top\}=XW\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}\.The total reconstruction loss is then:

ℒDend=∑i=1kℒMSE​\(X,X^i\)\\displaystyle\\mathcal\{L\}\_\{\\text\{Dend\}\}=\\sum\_\{i=1\}^\{k\}\\mathcal\{L\}\_\{\\text\{MSE\}\}\(X,\\hat\{X\}\_\{i\}\)=minW​∑i=1k‖X−X​W\[i\]​W\[i\]⊤‖2\\displaystyle=\\min\_\{W\}\\sum\_\{i=1\}^\{k\}\\\|X\-XW\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}\\\|^\{2\}=minW​∑i=1k‖X⊤−W\[i\]​W\[i\]⊤​X⊤‖F2\\displaystyle=\\min\_\{W\}\\sum\_\{i=1\}^\{k\}\\\|X^\{\\top\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}
Thus we have shown that the loss objective of a dendritic autoencoder \(with no orthogonality constraints\) is equivalent to ordered PCA and recovers the true PCA solution in the purely linear case\.

It is interesting to note that, in place of dendritic decoding, an alternative way to recover principal components in the linear case is to augment standard reconstruction minimization with an orthogonality penalty,*i\.e\.*

ℒ=ℒMSE​\(X,X^\)\+ℒorth​\(Z\)\.\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{MSE\}\}\(X,\\hat\{X\}\)\+\\mathcal\{L\}\_\{\\text\{orth\}\}\(Z\)\.This approach also recovers the columns ofVV, but without any particular ordering\. See appendix[B](https://arxiv.org/html/2607.05653#A2)for a proof\.

The linear analysis reveals that the goals of orthogonality regularization and dendritic reconstruction are closely related in the linear regime\. Both recover the principal components, but only the dendritic objective guarantees that components are ordered by importance\. In fact, the combined objective incorporating both dendritic reconstruction and orthogonality regularization,*i\.e\.*

ℒODIN≡ℒDend\+ℒorth\\displaystyle\\mathcal\{L\}\_\{\\text\{ODIN\}\}\\equiv\\mathcal\{L\}\_\{\\text\{Dend\}\}\+\\mathcal\{L\}\_\{\\text\{orth\}\}=minW⁡\[∑i=1k‖XT−W\[i\]​W\[i\]T​XT‖F2\+∑i<j\(WT​XT​X​W\)i​j2\]\\displaystyle=\\min\_\{W\}\\left\[\\sum\_\{i=1\}^\{k\}\\\|X^\{T\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{T\}X^\{T\}\\\|\_\{F\}^\{2\}\+\\sum\_\{i<j\}\(W^\{T\}X^\{T\}XW\)\_\{ij\}^\{2\}\\right\]=minW⁡\[∑i=1k‖XT−W\[i\]​W\[i\]T​XT‖F2\+12​‖WT​XT​X​W−diag​\(WT​XT​X​W\)‖F2\],\\displaystyle=\\min\_\{W\}\\left\[\\sum\_\{i=1\}^\{k\}\\\|X^\{T\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{T\}X^\{T\}\\\|\_\{F\}^\{2\}\+\\frac\{1\}\{2\}\\\|W^\{T\}X^\{T\}XW\-\\text\{diag\}\(W^\{T\}X^\{T\}XW\)\\\|\_\{F\}^\{2\}\\right\],shares the same minimizer as dendritic reconstruction alone in the linear regime \(see the proof in appendix[B](https://arxiv.org/html/2607.05653#A2)\):

###### Theorem 4

LetX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}, with singular value decompositionX=U​S​V⊤X=USV^\{\\top\}and distinct singular valuess1\>s2\>⋯\>sk\>0s\_\{1\}\>s\_\{2\}\>\\cdots\>s\_\{k\}\>0, then

k​s12\+\(k−1\)​s22\+⋯\+2​sk−12\+sk2=minW∈ℝn×k⁡ℒDend\+ℒorth,ks\_\{1\}^\{2\}\+\(k\-1\)s\_\{2\}^\{2\}\+\\cdots\+2s\_\{k\-1\}^\{2\}\+s\_\{k\}^\{2\}=\\min\_\{W\\in\\mathbb\{R\}^\{n\\times k\}\}\\mathcal\{L\}\_\{\\text\{Dend\}\}\+\\mathcal\{L\}\_\{\\text\{orth\}\},and the minimum is achieved whenW=V\[k\]​Σ=V​In×k​ΣW=V\_\{\[k\]\}\\Sigma=VI\_\{n\\times k\}\\Sigma, whereΣ=diag​\(ε1,…,εk\)\\Sigma=\\mathrm\{diag\}\(\\varepsilon\_\{1\},\\dots,\\varepsilon\_\{k\}\), withεj=±1\\varepsilon\_\{j\}=\\pm 1\.

In the non\-linear regime, however, this similarity is no longer guaranteed, and the two objectives could potentially be considered as independent design choices serving distinct purposes\. The dendritic structure enforces concentration of information across the latent dimensions, while the orthogonality penalty promotes decorrelated representations\. By default, we adopt the joint loss function as a principled extension of the linear theory in all non\-linear variants of ODIN, but the two can in principle be decoupled\. Dendrites alone suffice when ordered concentration is the primary goal, and the orthogonality term should be added only when decorrelated latent variables are explicitly desired\.

### 3\.3Relationship between ODIN and POLCA/AEO

Our analysis of ODIN is grounded in the equivalence between PCA’s maximum variance criterion and minimum reconstruction loss for linear networks, a result we exploit to derive several properties of ODIN’s loss minimizer\. These results establish that ODIN’s linear special case is not merely PCA\-like but is provably optimal in the same sense that PCA is optimal\.

The critical question then is how the approach generalizes when linearity is relaxed\. Variance\-based methods such as AEOAugustineet al\.\([2024](https://arxiv.org/html/2607.05653#bib.bib27)\)and POLCA\-NetPerozoet al\.\([2024](https://arxiv.org/html/2607.05653#bib.bib11)\)carry their ordering principle explicitly into the non\-linear setting by retaining direct penalties on activation variance or covariance statistics in the latent space\. This is a natural extension in form, but not in function: Variance, as a second\-order statistic defined in the ambient Euclidean space, does not in general correspond to intrinsic geometric importance along directions of a non\-linear manifoldAbuqrais and Pigoli \([2026](https://arxiv.org/html/2607.05653#bib.bib29)\)\. As the degree of non\-linearity increases, the variance of latent activations becomes an increasingly indirect proxy for the information content of each latent dimension, and the ordering it induces loses its connection to any well\-defined optimality criterion analogous to PCA\. This problem is further compounded by the fact that a sufficiently expressive decoder can rescale latent dimensions arbitrarily \(a dimension assigned low variance by the encoder may carry high reconstructive weight, and vice versa\), rendering the measured variance in the latent space a gauge\-dependent quantity with no intrinsic meaning\. The regularization term remains well\-defined as a scalar penalty, but its geometric interpretation, and therefore the interpretability of the resulting latent space, degrades\.

ODIN takes a different route\. Rather than extending the variance criterion directly, it retains the*reconstruction*route to importance ordering and instead relies on the dendritic architecture to break the symmetry of the solution space in a manner that recovers ordering as a structural consequence of the model class\. This architectural prior imposes an ordering that is intrinsic to the model rather than appended to it through a statistical penalty, and it does so in a way that remains geometrically meaningful regardless of the degree of non\-linearity\. The orthogonality penalty then operates on a solution space that is already structured, enforcing independence between the components that the dendritic architecture has already separated\. We argue that this division of labor \(architecture for ordering, loss terms for independence\) is both more principled and more interpretable than asking potentially numerous scalar loss penalties to accomplish similar objectives in a regime where geometric meaning is no longer guaranteed\.

## 4Experiments

The goal of this section is to systematically evaluate the ODIN architecture in comparison to both standard and variational autoencoder models, with direct reference to Principal Component Analysis \(PCA\) where appropriate\. To establish a performance baseline, in section 4\.1 we begin with experiments on synthetic 3D Gaussian point cloud data, a setting in which PCA is known to provide the optimal solution for linear dimensionality reduction\. We verify that the ODIN architecture is capable of recovering the principal components given by PCA, confirming its correct behavior in simple, well\-understood scenarios\.

Having validated ODIN’s ability to recover principal components in the controlled setting of linear transformations, we then examine its performance on the more complex dataset of MNIST handwritten digits\. The objective in this experimental setting is not to exactly replicate PCA’s solution, as doing so would constrain the model to linear relationships and forfeit the expressive advantages of deep architectures\. Instead, we seek to demonstrate that ODIN achieves PCA\-like properties—specifically, orthogonal latent dimensions with explicit importance ordering—while simultaneously leveraging non\-linear expressivity to capture complex features inherent in human handwriting\. This benchmark dataset provides an ideal testbed for evaluating ODIN’s capacity to capture non\-linear structure while maintaining interpretability and consistency\.

Finally, in section 4\.3 ODIN is applied to a real\-world scientific task of analyzing NV diamond’s photoluminescence \(PL\) spectroscopy data\. In this domain, interpretability and explainability of latent representations are especially crucial, as the ability to resolve distinct latent modes has direct implications for ability to develop data\-driven strategies for material characterization and development of novel sensors\. Comparisons in this section emphasize ODIN’s unique strengths in exposing latent dimensions that correspond to meaningful physical changes e\.g\. changes in lattice spacing, outperforming traditional networks in terms of scientific insight and practical utility\.

### Latent Modes

Throughout this work, we refer to latent modes as the characteristic directions of variation captured by individual latent dimensions\. Mathematically, these modes can be extracted as

whereZ∈ℝN×kZ\\in\\mathbb\{R\}^\{N\\times k\}is thekk\-dimensional latent representation forNNdata points, andX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}is the matrix of original input data withnnfeatures for each sample\. Each row ofRRrepresents the projection pattern that a specific latent dimension encodes\. In ODIN, these modes are both independent and importance\-ranked, enabling researchers to identify which patterns account for the most significant data variations\.

### 4\.1Point Cloud Data

In the first set of experiments, we investigate ODIN’s ability to recover principal components using a three\-dimensional Gaussian point cloud as a controlled, synthetic dataset\. The data for this experiment are generated by sampling10,00010,000points from a standard three\-dimensional normal distribution, which is then linearly transformed by applying rotation and anisotropic scaling\. This procedure produces an ellipsoidal point cloud in three\-dimensional space, where each principal axis corresponds to a distinct variance direction dictated by the stretching operation\.

PCA is provably optimal for such datasets because it identifies orthogonal directions in the data that maximize variance, corresponding exactly to the axes of the ellipsoid after transformation\. For any dataset produced by a linear transformation of Gaussian noise, PCA provides the best linear dimensionality reduction and recovers the original transformation up to orthogonal rotation\. As such, this synthetic scenario allows for direct and unambiguous benchmarking by comparing against the ground truth solution given by the principal axes computed by PCA\.

For this experiment, the base ‘standard’ autoencoder architecture is constructed as a purely linear network, consisting of a linear encoder followed by a linear decoder, and explicitly omitting any non\-linear activation functions\. In this setting, the latent variables and output reconstructions are just linear functions of the input, which ensures that the network operates within the space of linear transformations\. This is desirable as the underlying structure of the ellipsoidal point cloud is itself defined completely by linear transformations\.

To implement ODIN, the standard network is modified by introducing dendritic decodings, each producing reconstructions using different cumulative subsets of the latent variables\. We also introduce the orthogonality loss term in equation[4](https://arxiv.org/html/2607.05653#S3.E4)to enforce independence of latent dimensions, although it is not necessary in the linear setting\. Indeed we observe experimentally that dendritic decoding alone was sufficient to achieve feature disentanglement and hierarchical importance ordering in the linear regime, though often taking longer to converge to the global minimum when including the orthogonality loss term\.

The evaluation of each model is made possible by comparing their extracted latent modes to the principal directions obtained from PCA\. To assess alignment with the principal component axes, we utilize a three\-dimensional singular value decomposition \(SVD\) representation:

whereU∈ℝ3×3U\\in\\mathbb\{R\}^\{3\\times 3\}contains the principal components \(directions of maximal variance\), facilitating a direct comparison with the extracted latent modes\. The cross\-correlation matrixMMbetween the computed latent modesRRand the PCA principal component matrixUUis given by:

We measure how far this cross\-correlation matrix deviates from the identity, quantifying the error as:

e=‖M−I‖Fe=\|\|M\-I\|\|\_\{F\}\(10\)whereIIis the3×33\\times 3identity matrix and\|\|⋅\|\|F\|\|\\cdot\|\|\_\{F\}denotes the Frobenius norm\.

#### Results

For the standard linear autoencoder, results reveal a significant lack of consistency in the orientation of the learned latent modes across different training runs\. Empirically, the first latent mode aligns reasonably well with the first principal direction given by PCA in two out of seven runs; however, the second and third latent modes are frequently miscalculated, as lower\-variance directions are often subsumed within higher\-variance axes\. This can be attributed to the comparatively small variance in the third principal direction and the unordered, entangled nature of the learned latent modes\. This instability is evident when examining the cross\-correlation of the learned latent directions with the PCA basis: the resulting alignment varies markedly between model initializations\. See Figure[2](https://arxiv.org/html/2607.05653#S4.F2)\.

In contrast, the ODIN model demonstrates remarkable stability and reproducibility in the orientation of its latent modes across all training runs\. For every model initialization and training instance, the extracted latent directions are nearly identical and exhibit precise alignment with the principal component axes prescribed by PCA\. The cross\-correlation error between ODIN’s latent modes and the PCA basis is consistently close to the identity matrix, yielding minimal Frobenius norm error\. See Figure[3](https://arxiv.org/html/2607.05653#S4.F3)\. Thus we can say with confidence that ODIN matches PCA’s optimal solution in this linear regime\.

In parallel experiments, variational autoencoders \(VAEs\) were evaluated as a leading competitor to ODIN in terms of latent space structure\. In our simple synthetic 3D Gaussian experiments, we found that VAE’s consistently struggled to capture directions of low variance due to posterior collapse\. This limitation manifests as a pronounced flattening of dimensions with subtle variation, effectively disregarding them\. Lowering theβ\\betaparameter in the ELBO loss can partially mitigate this issue, allowing some recovery of low variance directions at the cost of reduced overall disentanglement\. In our experiments, we found that after tuning theβ\\betaparameter it was possible to get reconstructions on par with those from ODIN; however, the latent representations still suffered from variability across training runs\. While the probabilistic constraints enforcing disentanglement led to the emergence of three distinct latent modes that qualitatively corresponded to the principal component directions, these modes lacked any inherent ordering or hierarchy\. Each latent dimension contributed equally to the representation, resulting in frequent permutation of modes between training instances\. See figure[2](https://arxiv.org/html/2607.05653#S4.F2)\. This frequent interchange of latent dimensions is especially problematic when recalibrating models performing downstream tasks using only a subset of the latent space\.

Another interesting experiment was conducted using non\-linear point cloud data, presenting a more challenging scenario for latent space analysis\. In this setting, both ODIN and VAE architectures were augmented with activation layers222We opted to use leaky ReLU activation functions in place of standard ReLU for the VAE network to address the issue of suppressed gradients in low\-variance dimensions, which can lead to the complete zeroing out of subtle latent features\.to enhance their non\-linear expressivity and better accommodate the data’s underlying structure\. Despite these modifications, VAEs consistently failed to fully disentangle latent factors, regardless of the degree of variation in the data or the level of randomness introduced in the model\. The latent variables in VAE’s remained entangled, and reconstructions often appeared distorted or failed to reflect the true manifold of the original data\. In contrast, ODIN demonstrated robust disentanglement and successfully sorted the latent variables according to their importance, all while maintaining faithful and interpretable reconstructions\. These results highlight ODIN’s unique ability to manage both non\-linearity and interpretability in its latent representations, outperforming VAEs in this more complex regime\.

![Refer to caption](https://arxiv.org/html/2607.05653v1/img/PCA_vs_AE_latents.png)
Figure 2:Four different autoencoder architectures are trained and evaluated seven times on a three\-dimensional point cloud dataset\. Cross\-sectional views of raw point\-cloud data are shown alongside PCA principal component directions in red\. The corresponding latent modes as resolved by each network are shown alongside the PCA vectors, demonstrating high variability across training runs for both standard AE and VAE approaches\. POLCA comes much closer to recovering the first two latent modes but has some variability and misses the third, lowest variance, latent mode\. In contrast, the latent directions learned by ODIN consistently align with the ground truth directions given by PCA\.![Refer to caption](https://arxiv.org/html/2607.05653v1/x2.png)Figure 3:The error in the cross\-correlation matrix as defined in equation[10](https://arxiv.org/html/2607.05653#S4.E10)for four different models across seven independent training runs\. In each instance, the ODIN cross\-correlation matrix is near\-identity resulting in Frobenius norm error close to zero\. This confirms that ODIN recovers the principal components in their correct importance order regardless of random initialization\.

### 4\.2MNIST Data

The next dataset we evaluate is the MNIST set of handwritten digits, restricted to digits11and22for our primary experiments\. In this setting, all three architectures under comparison—the standard autoencoder, theβ\\beta\-VAE, and the ODIN architecture—employ convolutional neural network structures, which represent the standard approach for processing image data\. The encoder consists of two convolutional layers with3232and6464filters respectively, each followed by ReLU activations\. The resulting feature maps are flattened and passed through a fully connected layer that projects to ak=7k=7\-dimensional latent space\.

The decoder architectures mirror this structure in reverse\. For the standard conv\-AE and conv\-β\\beta\-VAE models, a single decoder maps from the full77\-dimensional latent space through a fully connected layer to reshape into appropriate dimensions for transposed convolution \(deconvolution\) layers\. These layers progressively upsample and reduce channel depth to reconstruct the original28×2828\\times 28pixel images\. The ODIN architecture modifies this structure by incorporating77dendritic decodings, where each decodingX^j\\hat\{X\}\_\{j\}receives only the firstjjlatent dimensionsZ\[j\]Z\_\{\[j\]\}and produces a full28×2828\\times 28reconstruction\.

Training was conducted using the Adam optimizer with stepped learning rate for 2500 epochs on batches of 480 samples\. For theβ\\beta\-VAE, we experimented withβ\\betavalues ranging from0to11, ultimately selectingβ=1\\beta=1as providing the best balance between reconstruction quality and disentanglement\. The ODIN orthogonality weight was set toλ=1\\lambda=1after preliminary experiments showed this value effectively enforced latent orthogonality without overwhelming the reconstruction objective\. Each architecture was trained1010times with different random initializations to assess consistency and reproducibility\.

Unlike the point cloud experiments where ground truth principal directions provided an absolute reference, MNIST evaluation focuses on three complementary criteria: \(1\) consistency of latent modes across independent training runs, quantified by computing pairwise cross\-correlations between modes extracted from different model instantiations; \(2\) disentanglement, assessed through targeted latent traversals where individual dimensions are ignored while others remain fixed; and \(3\) interpretability, evaluated by whether learned modes correspond to recognizable visual features that domain experts can meaningfully describe\. We visualize latent modes both as images and through targeted reconstructions that isolate the contribution of individual dimensions to the overall data representation\.

#### Standard Convolutional Autoencoder Results

The baseline convolutional autoencoder architecture, despite achieving low reconstruction error, exhibits severe deficiencies in latent space organization\. Across the independent training runs, latent modes demonstrate high variability with no consistent correspondence between dimension indices and interpretable features\. Computing the cross\-correlation\|⟨Ri\(a\),Rj\(b\)⟩\|\|\\langle R\_\{i\}^\{\(a\)\},R\_\{j\}^\{\(b\)\}\\rangle\|for modes extracted from different runs\(a\)\(a\)and\(b\)\(b\), we find that while dimensions occasionally align by chance, there is no systematic pattern ensuring that, for example, dimension33in one training run captures similar features to dimension33in another run\. Visualizations of the learned modes as images show that they uniformly exhibit distributed, non\-localized patterns that fail to isolate coherent visual features\.

Ablation experiments further confirm the absence of hierarchical structure\. When reconstructing test images using only the firstkklatent dimensions \(by zeroing out dimensionsk\+1k\+1through 7\), reconstruction quality does not improve monotonically or predictably askkincreases\. Instead, we observe irregular performance curves where omitting certain seemingly arbitrary dimensions causes larger degradation than omitting others, and this pattern varies across training runs \(Fig[4](https://arxiv.org/html/2607.05653#S4.F4)\)\. This behavior indicates that the network has not learned to prioritize information allocation; all dimensions contribute roughly equally to reconstruction, precluding interpretable importance ranking\.

![Refer to caption](https://arxiv.org/html/2607.05653v1/x3.png)

![Refer to caption](https://arxiv.org/html/2607.05653v1/x4.png)

![Refer to caption](https://arxiv.org/html/2607.05653v1/x5.png)

![Refer to caption](https://arxiv.org/html/2607.05653v1/x6.png)

Figure 4:Reconstruction loss as a function of the number of latent dimensions \(dendrites\) across 10 independent runs for the VAE, Standard\-AE, POLCA and ODIN architectures\. Each curve corresponds to an independent training run\. While Standard and VAE networks exhibit non\-monotonic, run\-dependent loss trajectories, ODIN produces nearly identical, monotonically decreasing curves across all runs, demonstrating substantially greater stability and predictability in latent space utilization\. POLCA is able to concentrate more information into earlier latent variables due to its center of mass loss term, however, the reconstruction and classification accuracy is slightly worse when using all latent features compared to the other three models\.
#### β\\beta\-Variational Autoencoder Results

Theβ\\beta\-VAE architecture demonstrates marked improvement in latent space structure compared to the standard autoencoder, successfully achieving feature disentanglement through its probabilistic regularization framework\. Withβ=1\\beta=1, the model learns latent dimensions that respond selectively to distinct generative factors\. Systematic traversals along individual dimensions produce interpretable variations in reconstruction\. For example, varying dimension 2 while holding others constant might systematically alter the slant of digit strokes, while dimension 3 controls stroke thickness\. This disentanglement represents a significant advance for interpretability, as researchers can now identify which latent variables encode which semantic attributes\.

However, while dimensions are successfully disentangled, they lack any inherent ordering or hierarchy\. The learned latent space exhibits permutation symmetry—dimensions can be arbitrarily reordered without affecting reconstruction quality\. Empirically, this manifests as frequent permutations across training runs\. For example, dimension33in one instantiation might capture digit identity \(a high\-importance feature\), while in another run, this same information migrates to dimension55\. Cross\-correlation analysis of modes extracted from independent training runs reveals that although each run produces well\-disentangled representations, the assignment of features to specific dimension indices varies unpredictably\. This inconsistency severely limits the model’s utility for comparing results across experiments or performing systematic ablation studies, as one cannot reliably identify the “third most important feature” when this designation changes with each training run\.

Ablation experiments withβ\\beta\-VAE confirm the absence of importance ordering\. When plotting reconstruction accuracy using progressive latent dimensions, the improvement curve is neither smooth nor consistent across training runs\. See figure[4](https://arxiv.org/html/2607.05653#S4.F4)\. More critically, the subset of dimensions that proves most informative varies between model instantiations, confirming that the architecture provides no mechanism for consistently prioritizing high\-variance directions into early latent dimensions\.

#### ODIN Architecture Results

The ODIN architecture demonstrates exceptional performance across all evaluation criteria, successfully combining the disentanglement advantages ofβ\\beta\-VAE with robust importance ordering and cross\-run consistency that neither baseline achieves\. Across all 10 independent training runs, ODIN produces latent modes that are both remarkably stable in their assignments and interpretably organized in hierarchical order of importance\. The cross\-correlation analysis shows that for any two independently trained ODIN models\(a\)\(a\)and\(b\)\(b\), the mode alignment satisfies\|⟨Ri\(a\),Ri\(b\)⟩\|\>0\.999\|\\langle R\_\{i\}^\{\(a\)\},R\_\{i\}^\{\(b\)\}\\rangle\|\>0\.999for alli∈\{1,…,10\}i\\in\\\{1,\.\.\.,10\\\}, indicating that dimensioniicaptures essentially identical features across all training runs\. This consistency enables reproducible science as researchers can meaningfully compare “dimension33” across different experiments with confidence that this dimension represents the same underlying data variation pattern\.

The importance ordering emerges naturally from ODIN’s dendritic decoder architecture\. Reconstruction error curves for the dendritic outputsD​e​c​\(Z\[1\]\),…,D​e​c​\(Z\[7\]\)Dec\(Z\_\{\[1\]\}\),\\dots,Dec\(Z\_\{\[7\]\}\), show smooth, monotonic improvement as additional latent dimensions are incorporated, with the steepest error reduction occurring when the first few dimensions are added\. See figure[4](https://arxiv.org/html/2607.05653#S4.F4)\. Specifically, the first latent dimension alone achieves mean squared error of0\.0360\.036, incorporating the second reduces this to0\.0270\.027, the third to0\.0210\.021, and by the fifth dimension, the error has dropped to0\.0150\.015, nearly matching the full model’s performance\. This curve validates that ODIN successfully and reliably allocates high\-variance, reconstruction\-critical information to early dimensions, with later dimensions contributing progressively finer details\.

#### Explainability Through Hierarchical Feature Analysis

The evaluation framework extends the latent mode analysis methodology from Section 4\.1 to the image domain\. For each trained model, we compute latent modesR∈ℝk×nR\\in\\mathbb\{R\}^\{k\\times n\}via the matrix operationR=ZT​XR=Z^\{T\}X, whereZ∈ℝN×kZ\\in\\mathbb\{R\}^\{N\\times k\}represents the encoded latent representations forNNtraining samples andX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}is the matrix of flattened image vectors withn=784n=784pixels\(28×28\)\(28\\times 28\)\. Each row ofRRcorresponds to a latent dimension and can be reshaped into a28×2828\\times 28image representing that dimension’s characteristic pattern or “mode” of variation\.

In the case of ODIN restricted to digits11and22only, the hierarchical feature structure becomes particularly transparent and reveals crisp, interpretable patterns \(see Figure[5](https://arxiv.org/html/2607.05653#S4.F5)\)\. The first mode captures the dominant source of variation in the dataset—the fundamental differences between digit classes\. When visualized as a28×2828\\times 28image, this mode shows clear digit\-like structure, representing the average contrast between high\-variance digit pairs\. Subsequent modes isolate progressively more subtle features such as orientation angles, stroke widths, loop curvatures, and positional offsets\.

![Refer to caption](https://arxiv.org/html/2607.05653v1/img/mnist_modes.png)Figure 5:Panels display the first seven latent modesR=Z⊤​XR=Z^\{\\top\}X\(reshaped as28×2828\\times 28images\) on representative training runs for each method\. ODIN modes demonstrate remarkable stability, closely aligning with PCA principal components in early dimensions while revealing hierarchical non\-linear structure in later dimensions, with consistent assignment across all runs\. Standard autoencoder modes spread dominant digit class separations diffusely across latent features, exhibiting high inconsistency across runs with smeared patterns that fail to isolate coherent digit characteristics\.β\\beta\-VAE modes achieve disentanglement with recognizable independent features, but suffer from inconsistent feature assignments across initializations\.Visualizing the first latent mode by computing the average images for samples withz1\>0z\_\{1\}\>0versusz1<0z\_\{1\}<0reveals clear separation, with positive values corresponding to digit22\(with its characteristic curved top and bottom\), and negative values corresponding to digit11\(vertical stroke structure\)\. To quantify this, we evaluate the discriminative capacity ofz1z\_\{1\}using the area under the receiver operating characteristic curve \(AUC\-ROC\), treatingz1z\_\{1\}directly as a ranking score without any learned classifier\. This metric measures the probability that a randomly selected digit 2 sample receives a lower score than a randomly selected digit 1 sample, and yields an average AUC of0\.99420\.9942, confirming near\-perfect linear separability along this single dimension \(see table[1](https://arxiv.org/html/2607.05653#S4.T1)\)\. Reconstructions using onlyD​e​c​\(Z\[1\]\)Dec\(Z\_\{\[1\]\}\)produce crude but recognizable digit shapes that capture the basic structural templates—vertical for11, curved for22—while omitting stylistic details\.

ODINPCAStandard\-AEVAEPOLCA0\.99420\.97940\.96380\.70120\.8595Table 1:AUC\-ROC scores achieved by the first latent dimensionz1z\_\{1\}as a linear separator for digits11and22on held out test data, averaged across multiple runs of each unsupervised network\. An AUC near1\.01\.0indicates that the first latent dimension captures nearly all class\-discriminative structure\.Subsequent latent dimensions, having captured the primary inter\-class variation in dimension 1, are free to encode the dominant intra\-class variation\. For digit11, dimension22manifests as angular orientation or slant\. By selecting samples of digit11and sorting them byz2z\_\{2\}values, we observe a smooth progression from left\-leaning strokes\(z2<0\)\(z\_\{2\}<0\)through vertical orientations\(z2≈0\)\(z\_\{2\}\\approx 0\)to right\-leaning slants\(z2\>0\)\(z\_\{2\}\>0\)\. For digit22, dimension33distinguishes between “curly” versus “flat” variants,*i\.e\.*samples withz3<0z\_\{3\}<0exhibit tightly curved bottom loops that extend below the baseline, whilez3\>0z\_\{3\}\>0corresponds to flatter, more angular bottom segments that terminate at the baseline\. Higher\-order dimensions continue this hierarchical decomposition into progressively finer features\.

### 4\.3Nitrogen Vacancy Diamond Photoluminescence Data

To evaluate ODIN’s performance on real\-world scientific data, we applied both standard autoencoder and ODIN architectures to nitrogen\-vacancy \(NV\) center spectroscopy measurements from diamond samples\. The experimental setup and physical interpretation of NV photoluminescence \(PL\) spectra is discussed in detail by Rajpal et alShraddha Rajpal \([2026](https://arxiv.org/html/2607.05653#bib.bib40)\)in a forthcoming manuscript\. The authors carefully evaluated the suitability of ML models for mapping observed spectra onto temperature\. They note that while all ML models outperform expert knowledge derived linear models, autoencoders performed worst amongst the ML models\. This underperformance was reasoned to arise from the inability of autoencoder models to discriminate against stochastic “dark variables” that are not correlated with temperature\. Here we test, Rajpal et al’sShraddha Rajpal \([2026](https://arxiv.org/html/2607.05653#bib.bib40)\)dark variable hypothesis using ODIN\.

#### Data pre\-processing and Model

Briefly, the dataset consisted of photoluminescence spectra collected over a time series during which the sample temperature was continuously increased from 243 K to 343 K\. Each spectrum was acquired at 2 nm resolution over the 613\-800 nm wavelength range, resulting in 1340\-dimensional input vectors\. The dataset comprised 144,000 consecutive measurements, with temperature recorded simultaneously via an independent thermister\. The dataset was split into80%80\\%training and20%20\\%testing for all models\.

Both autoencoder architectures employed identical network configurations with a1340→1350→650→350→175→56→20→51340\\to 1350\\to 650\\to 350\\to 175\\to 56\\to 20\\to 5encoder structure and symmetric decoder, with ReLU \(Rectified Linear Unit\) activation functions\. The 5\-dimensional latent space was chosen to provide sufficient capacity for capturing spectral variations while maintaining meaningful dimensionality reduction\. Networks were trained using the Adam optimizer with stepped learning rate with plateau stopping criteria\. For ODIN, the orthogonality regularization weightλorth\\lambda\_\{\\text\{orth\}\}in[4](https://arxiv.org/html/2607.05653#S3.E4)was varied from0−3\.00\-3\.0to compare latent correlations\. To assess consistency, we performed 10 independent training runs for each architecture with different random weight initializations\.

To evaluate the degree to which each network encodes temperature information in its latent representation, we perform a post\-training linear regression analysis using the independently measured temperatures as ground truth labels\. Critically, these labels are never provided to either network during training; both architectures are trained purely to minimize spectral reconstruction error, with no supervised signal of any kind\. The encoder is therefore used solely to project each spectrum onto its corresponding latent representation, and only after training is complete do we introduce the temperature labels for the purpose of evaluation\. Linear regression models with different subsets of the latent space, from individual latent variables up to all dimensions simultaneously, are fit to assess how much temperature information each variable carries independently, as well as how many dimensions are collectively required to achieve accurate temperature prediction\.

#### Results

For standard autoencoders, the distribution of temperature\-related information varied substantially between runs\. The analysis revealed that temperature information was often scattered across latent dimensions, with the specific dimensions varying unpredictably across initializations \(see Figure[9](https://arxiv.org/html/2607.05653#S4.F9)\)\. In some runs, dimensions 1, 3, and 5 jointly explained90%90\\%of temperature variance; in others, dimensions 2 and 4 were required to achieve comparable performance\.

ODIN, in contrast, consistently encoded temperature information in the second latent dimension across all 10 training runs, see Figure[7](https://arxiv.org/html/2607.05653#S4.F7)\. ODIN consistently achievedR2\>0\.80R^\{2\}\>0\.80using only the first two latent variable andR2\>0\.90R^\{2\}\>0\.90using the first three latent variables in 9 of 10 runs\. Standard autoencoders required an average of 3\.2 latent variables to reachR2\>0\.80R^\{2\}\>0\.80and 4\.5 variables forR2\>0\.90R^\{2\}\>0\.90\.

Despite the stark difference in latent space organization, both architectures achieve comparable temperature predictions on testing data \(see Figure[6](https://arxiv.org/html/2607.05653#S4.F6)\)\. This might initially seem to undercut the case for ODIN, but it is precisely what one would expect: if both networks successfully capture temperature\-relevant information, a sufficiently flexible regression should recover it regardless of how that information is distributed\.333We remark that incorporating pairwise interaction \(cross\) terms between latent variables reduces the prediction error to within approximately half a degree, with ODIN achieving marginally better performance than the standard autoencoder\. While the inclusion of cross terms somewhat complicates direct interpretation, it is consistent with the physical expectation that mean spectral intensity and temperature\-dependent shifts interact non\-linearly\.The distinction lies not in raw information content but in the accessibility and reliability with which that information can be extracted and interpreted\.

![Refer to caption](https://arxiv.org/html/2607.05653v1/x7.png)

![Refer to caption](https://arxiv.org/html/2607.05653v1/x8.png)

Figure 6:Linear regression error \(in Kelvin\) on held\-out test data as a function of the number of latent dimensions included in the regression, for ODIN \(left\) and standard autoencoder \(right\) networks\. Each curve corresponds to an independent training run\. ODIN consistently achieves a sharp reduction in error at the second latent dimension, reflecting its stable assignment of temperature\-dependent spectral variation to dimension two across all runs\. The standard autoencoder exhibits substantial run\-to\-run variability, with temperature information distributed inconsistently across dimensions depending on initialization\.
#### Physical Interpretation of Latent Organization

A closer examination of the learned latent representations reveals a striking contrast in how the two architectures resolve spectral features\. Across all independent training runs, ODIN consistently assigns its first latent dimension to encode changes that are strongly correlated with the mean of the spectrum observations over time\. We refer to this mode as the “mean latent mode”\. The mean latent mode represents the single largest source of variance in the dataset*in terms of reconstruction*\. We hypothesize that this mode is capturing overall intensity changes in the overall PL profile\. The second latent dimension is strongly correlated with temperature changes \(see Figure[7](https://arxiv.org/html/2607.05653#S4.F7)\)\. Temperature\-dependent shift in the zero\-phonon line \(ZPL\) and broadening of the ZPL and first two phonon modes of the broader phonon side band captured by the second dimension, while physically significant, constitute a comparatively smaller perturbation\. ODIN’s hierarchy faithfully respects this structure\.

![Refer to caption](https://arxiv.org/html/2607.05653v1/x9.png)

![Refer to caption](https://arxiv.org/html/2607.05653v1/x10.png)

Figure 7:Bar plots showing the linear fit correlation of individual ODIN latent dimensions with the dataset mean spectrum \(top\) and measured temperature \(bottom\), for each of several independent training runs \(distinguished by color\)\. Dimension one is consistently and strongly correlated with the mean spectrum across all runs, while dimension two consistently carries the dominant correlation with temperature\. Higher\-order dimensions show weak and variable correlations with both quantities\.Visualization of the latent space confirmed this interpretation: projecting the data onto ODIN latent dimensions revealed a clear temperature gradient along dimension two, while dimension one captured measurement\-to\-measurement intensity variations, reflecting the variation in total collected photoluminescence \(see Figure[8](https://arxiv.org/html/2607.05653#S4.F8)\), although it does over short temperature intervals appear to be correlated with temperature\.

![Refer to caption](https://arxiv.org/html/2607.05653v1/x11.png)

![Refer to caption](https://arxiv.org/html/2607.05653v1/x12.png)

Figure 8:The first two latent dimensions learned by ODIN, shown for a representative training run, alongside the dataset true mean spectrum \(top\) and temperature \(bottom\) for comparison\. Latent dimension one \(top\) is strongly correlated with the true mean spectrum, capturing the baseline photoluminescence profile common to all measurements\. Latent dimension two \(bottom\) is strongly correlated with temperature, reflecting the thermally driven shift and broadening of the zero\-phonon line and phonon sideband\. Both latent modes are displayed after affine rescaling according to the coefficients obtained by regressing each mode onto its respective physical quantity\. This organization was consistent across all independent training runs\.Standard autoencoders, by contrast, do not exhibit this clean separation\. Across training runs, most \(and in some cases, all\) latent dimensions show substantial correlation with the mean spectrum, with additional dimensions appearing to encode corrective factors on top of this dominant signal \(see Figure[9](https://arxiv.org/html/2607.05653#S4.F9)\)\. Rather than isolating independent sources of variation, the standard network spreads mean latent mode\-related information diffusely across the latent space, with temperature information entangled within this distributed representation\. This behavior is consistent with the broader paradigm of unconstrained autoencoders: without an explicit mechanism to overcome the symmetry of the solution space, the network finds no reason to avoid redundant encodings, and the resulting latent space reflects arbitrary superpositions of the true underlying factors rather than a clean decomposition\.

![Refer to caption](https://arxiv.org/html/2607.05653v1/x13.png)

![Refer to caption](https://arxiv.org/html/2607.05653v1/x14.png)

Figure 9:Linear fit correlation of standard\-AE latent dimensions with the dataset mean spectrum \(top\) and with temperature \(bottom\), shown for multiple independent training runs \(colored bars\)\. In contrast to ODIN, mean\-spectrum information is spread across multiple dimensions and varies substantially from run to run, reflecting the absence of any ordering in the learned latent space\. Temperature correlation similarly fails to localize to a consistent dimension, appearing in different positions depending on the random initialization\.The relationship between the mean spectrum and temperature prediction deserves elaboration\. When the mean of each observation is subtracted from the dataset prior to training, ODIN correctly promotes temperature to its first latent dimension\. However, this mean\-centered configuration yields slightly worse regression performance onto temperature \(while maintaining comparable reconstruction quality\), suggesting that the mean spectrum carries some temperature\-predictive information\. This is physically plausible: in fiber\-coupled NV sensing configurations, the measured photoluminescence spectrum is a composite signal mainly dominated by NV\-center fluorescence, but also containing a nontrivial background contribution from the optical fiber itself and the epoxy and other materials used in fiber sensor assembly\. Both the NV fluorescence and fiber contributions are proportional to the incident laser power\. Hence random fluctuations or long\-term drift in laser power can lead to spectrum\-to\-spectrum variations observed for mean latent mode projections that have temporal correlations but no temperature correlation\. The correlation between the latent mean mode and temperature likely arises due to temperature dependent changes in the fiber’s \(and associated material’s\) background florescence intensity and fiber assembly’s transmission properties\. Other likely, though weaker, contributions could arise from temperature dependent changes in NV\-to NV0charge state conversion

In a recent study,Johanssonet al\.\([2026](https://arxiv.org/html/2607.05653#bib.bib39)\)have carefully examined the contribution of various types of optical fiber’s background florescence \(sometime referenced as autoflorescence\) in NV sensing application\. The authors demonstrate that optical fibers show a broad fluorescence band above600600nm centered near644​–​645644–645nm region that is associated with non\-bridging oxygen hole center \(NBOHC\) defects in fused\-silica glass\. It significantly overlaps with the NV florescence spectra\. Crucially, the NBOHC fluorescence intensity is itself temperature\-dependent due to phonon\-electron coupling of the NBOHC center to the silica matrix\. This suggests that the overall spectral weight and baseline intensity \(what ODIN captures as its first latent mode\) is not a temperature\-independent nuisance variable, but rather a partially informative proxy for thermal state of the sensor assembly\. Higher\-order ODIN dimensions, those beyond the first two, capture structured residual variation that is neither mean\-like nor purely temperature\-dependent\. The fact that these dimensions do not contribute substantially to improved regression suggests they are not as critical for temperature prediction, but may nonetheless encode subtle experimental covariates or higher\-order effects that are physically meaningful to understand the evolution of PL spectra over time\. These effects may include slight changes in the fiber bundle’s light coupling and transmission properties due to either temperature history, laser power exposure strain due to bending, changes in optical properties of the epoxy used in sensor and or thermo\-mechanical changes in the epoxy/fiber sensor assembly\.

When such degradation accumulates over time, as such it constitutes an archetypal dark variable; structured, physically motivated, but uncontrolled\. Because the dominant signals have been cleanly extracted into the leading modes, these residual dimensions are far easier to interrogate than the analogous dimensions of a standard autoencoder, where mean and temperature information has not been fully separated from whatever else the network has learned\. In this sense, ODIN’s ordered decomposition transforms latent space exploration from an arbitrary search into a principled, progressive investigation of the data’s underlying structure\. For the NV spectroscopy application, this provides a stable coordinate system for analyzing how spectral features evolve with thermal excitation\. Given the evidence presented inJohanssonet al\.\([2026](https://arxiv.org/html/2607.05653#bib.bib39)\), a natural first hypothesis to test is whether slow drift in the fiber’s autofluorescence background manifests as structured variation in the higher\-order ODIN dimensions\. This could be directly probed by acquiring long\-duration fiber emission spectra under controlled, temperature\-stable conditions and examining whether the resulting drift signature projects preferentially onto the residual latent modes\.

#### Cross Encoder Validation

![Refer to caption](https://arxiv.org/html/2607.05653v1/x15.png)

![Refer to caption](https://arxiv.org/html/2607.05653v1/x16.png)

Figure 10:\(Top\) Latent modes of the laser\-trained encoder layer of ODIN for in\-distribution laser power data: one dominant mean\-like mode, with higher\-order modes flat and near\-zero\. \(Bottom\) Cross\-passed temperature data through the laser encoder of ODIN: mean mode active, higher\-order modes remain inert\. The dominance of the mean mode across all conditions confirms that laser power variation occupies a single intensity\-scaling dimension in spectral space\.A central claim in the preceding analysis is that ODIN’s first latent mode encodes overall spectral weight rather than a temperature\-specific signal\. Under this hypothesis, any physical perturbation that modulates total photon flux should drive the first latent mode, regardless of whether it also modulates spectral shape\. To test this directly, we collected laser power variation data under controlled, fixed\-temperature conditions and examined the response of both the temperature\-trained encoder and the laser power\-trained encoder when presented with either dataset\.

Training ODIN on laser power variation data at fixed temperature produces a particularly clean latent structure\. One dominant mode emerges, corresponding to the row mean of the data, which is expected given that laser power and total emission intensity are approximately proportional and the spectral shape is largely invariant to laser power under fixed\-temperature conditions\. The higher\-order modes learned by the laser encoder are notably flat, carrying negligible residual structure regardless of whether laser data or temperature data are passed through them \(see Figure[10](https://arxiv.org/html/2607.05653#S4.F10)\)\. This flatness indicates that laser power variation, at fixed temperature, exhausts its spectral degrees of freedom in a single dominant direction, with no meaningful higher\-order content remaining once the mean scaling is accounted for\. The same flatness persists when temperature data are passed through the laser encoder, confirming that the laser encoder has no coordinate axes onto which the richer spectral structure of temperature variation can project\. In both directions, the shared signal between the two datasets is exactly and only the row mean\.

Passing laser power data through the temperature encoder predominantly activates the mean\-related mode in a manner consistent with the known proportionality between incident laser power and total emission intensity, while the temperature\-sensitive mode and higher\-order dimensions exhibit only weak, mean\-like residual activations\. This is a strong indicator that the first latent mode of the temperature encoder is not a specialist temperature coordinate, but a general intensity coordinate\. Higher order modes carry information that is genuinely absent from fixed temperature, variable laser\-power observations\.

The activation of the mean mode across both encoder\-dataset combinations identifies it as the unique latent coordinate shared by all physical perturbations\. The mean\-like residual activations produced when laser power data are passed through the temperature encoder suggests that the temperature encoder’s higher\-order modes are not completely orthogonal to intensity variation in the cross\-passed setting, but that the residual response they do produce carries no structured spectral content beyond a broad scaling\. This is distinct from the structured phonon\-sideband and ZPL character seen in those modes during temperature data encoding, and it is consistent with the interpretation that those modes are specialist coordinates for spectral shape variation that simply find no matching structure in the laser dataset, leaving only a weak, unstructured mean\-like residual\.

This interpretation has practical consequences for preprocessing choices\. Mean\-subtracting each spectrum prior to training removes the thermal information carried by the mean mode along with the laser\-power\-driven intensity variance\. When the goal is temperature inference, this tradeoff is unfavorable, and the experimental results reported here are consistent with the observation that training on un\-mean\-subtracted spectra yields marginally improved temperature regression performance\.

The present results, obtained entirely without labels, raise a natural question about what could be achieved with access to even partial supervision\. The current ODIN encoders are trained on single experimental conditions and learn modes that reflect the dominant sources of variance within each condition\. A supervised or semi\-supervised extension of ODIN, in which known physical labels such as laser power and temperature are incorporated as auxiliary targets on selected latent dimensions, could in principle further factorize the latent distribution\. Supervision onto known physical variables anchors those dimensions explicitly, leaving the residual modes free to absorb any remaining structured variation\. Subsequent targeted data collection under controlled conditions would then provide the labeled data needed to either confirm or rule out candidate dark\-variable hypotheses, and potentially to introduce those variables as additional supervisory targets\. Exploring this direction, and in particular whether ODIN’s orthogonality structure generalizes reliably across experimental conditions when extended to joint training on multiple specialist datasets, remains an important avenue for future work\.

## 5Conclusions

In this work, we introduced ODIN, a novel autoencoder architecture that addresses the rotational ambiguity and feature entanglement inherent to standard autoencoder optimization\. By enforcing orthogonality and ordered variance decomposition through dendritic decoding, ODIN produces latent representations that are statistically independent, sorted by explained variance, and reproducible across experiments\. Unlikeβ\\beta\-VAEs, which require careful, data\-dependent tuning of a balancing hyperparameter, ODIN achieves disentangled representations and faithful reconstructions simultaneously, without delicate trade\-offs\.

A central theoretical contribution of this work is the result that, in the linear regime, ODIN provably recovers PCA via the reconstruction pathway, establishing a rigorous foundation that connects the architecture to a well\-understood and optimal solution\. Rather than indirectly encouraging ordering structure through auxiliary losses or hyperparameter schedules, ODIN inherits its organizational principles directly from its architecture, making the nonlinear generalization both natural and theoretically motivated\. Each latent component maintains a stable semantic correspondence to a direction of data variation, empowering researchers to identify what the model has learned, in what order of priority, and with what degree of statistical independence\.

This transparency supports an iterative workflow that bridges machine learning and the traditional scientific method\. Researchers can train ODIN on initial data, interrogate the latent structure to surface dominant factors and unexpected patterns, and design targeted follow\-up experiments guided by those findings\. For the broader goal of integrating machine learning into scientific workflows, ODIN exemplifies the importance of model interpretability as a prerequisite for scientific utility\. By transforming the autoencoder from a purely predictive tool into an instrument for hypothesis testing and exploratory analysis, ODIN provides a principled and flexible foundation for dimensionality reduction in complex, high\-dimensional domains\. We hope it serves as a useful framework for researchers who require not just accurate representations, but ones they can understand, trust, and build upon\.

Acknowledgments and Disclosure of Funding

Tyrus Berry and Jeanie Schreiber acknowledge support from NIST Award 70NANB24H232 and Zeeshan Ahmed acknowledges support from AFMETCAL Award R24\-685\-0005\.

## References

- A riemannian covariance for manifold\-valued data\.Journal of Mathematical Imaging and Vision68\(2\),pp\. 7\.Cited by:[§3\.3](https://arxiv.org/html/2607.05653#S3.SS3.p2.1)\.
- G\. Amato, F\. Carrara, F\. Falchi, C\. Gennaro, and G\. Lagani \(2019\)Hebbian learning meets deep convolutional neural networks\.InInternational Conference on Image Analysis and Processing,pp\. 324–334\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p9.3)\.
- M\. T\. Augustine, P\. Patil, M\. Bhushan, and S\. Bhartiya \(2024\)Autoencoder with ordered variance for nonlinear model identification\.arXiv preprint arXiv:2402\.14031\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p7.1),[§3\.3](https://arxiv.org/html/2607.05653#S3.SS3.p2.1)\.
- Y\. Bengio, A\. Courville, and P\. Vincent \(2013\)Representation learning: a review and new perspectives\.IEEE transactions on pattern analysis and machine intelligence35\(8\),pp\. 1798–1828\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p1.1)\.
- Q\. Chen and M\. Fuge \(2024\)Compressing latent space via least volume\.InThe Twelfth International Conference on Learning Representations,Cited by:[footnote 1](https://arxiv.org/html/2607.05653#footnote1)\.
- M\. Eltager, T\. Abdelaal, M\. Charrout, A\. Mahfouz, M\. J\. Reinders, and S\. Makrodimitris \(2023\)Benchmarking variational autoencoders on cancer transcriptomics data\.PLoS One18\(10\),pp\. e0292126\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p6.1)\.
- Y\. Fei, Y\. Liu, C\. Jia, Z\. Li, X\. Wei, and M\. Chen \(2025\)A survey of geometric optimization for deep learning: from euclidean space to riemannian manifold\.ACM Computing Surveys57\(5\),pp\. 1–37\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p8.1)\.
- M\. Fil, M\. Mesinovic, M\. Morris, and J\. Wildberger \(2021\)Beta\-vae reproducibility: challenges and extensions\.arXiv preprint arXiv:2112\.14278\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p4.2)\.
- B\. Ghojogh, A\. Ghodsi, F\. Karray, and M\. Crowley \(2021\)Factor analysis, probabilistic principal component analysis, variational inference, and variational autoencoder: tutorial and survey\.arXiv preprint arXiv:2101\.00734\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p1.1)\.
- M\. Harandi and B\. Fernando \(2016\)Generalized backpropagation,\\\\backslash’\{\\\{e\}\\\}tude de cas: orthogonality\.arXiv preprint arXiv:1611\.05927\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p8.1)\.
- J\. He, D\. M\. Spokoyny, G\. Neubig, and T\. Berg\-Kirkpatrick \(2019\)Lagging inference networks and posterior collapse in variational autoencoders\.ArXivabs/1901\.05534\.External Links:[Link](https://api.semanticscholar.org/CorpusID:58014132)Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p4.2)\.
- I\. Higgins, L\. Matthey, A\. Pal, C\. P\. Burgess, X\. Glorot, M\. M\. Botvinick, S\. Mohamed, and A\. Lerchner \(2017\)Beta\-vae: learning basic visual concepts with a constrained variational framework\.\.ICLR \(Poster\)3\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p3.3),[§3](https://arxiv.org/html/2607.05653#S3.p11.1)\.
- I\. Higgins, L\. Matthey, A\. Pal, C\. P\. Burgess, X\. Glorot, M\. M\. Botvinick, S\. Mohamed, and A\. Lerchner \(2016\)Beta\-vae: learning basic visual concepts with a constrained variational framework\.InInternational Conference on Learning Representations,External Links:[Link](https://api.semanticscholar.org/CorpusID:46798026)Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p3.3)\.
- Y\. Ichikawa and K\. Hukushima \(2024\)Learning dynamics in linear vae: posterior collapse threshold, superfluous latent space pitfalls, and speedup with kl annealing\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 1936–1944\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p1.1)\.
- S\. Johansson, A\. Bukschat, D\. Lönard, A\. Erlenbach, J\. Gutsche, and A\. Widera \(2026\)Characterization of autofluorescence in optical fibers for nv\-based sensing applications\.arXiv preprint arXiv:2602\.07536\.Cited by:[§4\.3](https://arxiv.org/html/2607.05653#S4.SS3.SSSx3.p5.2),[§4\.3](https://arxiv.org/html/2607.05653#S4.SS3.SSSx3.p6.1)\.
- A\. Kierkegaard and T\. Morfeldt Gadler \(2023\)Comparison of hebbian learning and backpropagation for image classification in convolutional neural networks\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p9.3)\.
- D\. P\. Kingma, D\. J\. Rezende, S\. Mohamed, and M\. Welling \(2014\)Semi\-supervised learning with deep generative models\.Advances in neural information processing systems27\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p2.16)\.
- P\. Li, Y\. Pei, and J\. Li \(2023\)A comprehensive survey on design and application of autoencoder in deep learning\.Applied Soft Computing138,pp\. 110176\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p1.1)\.
- Y\. Liu, Z\. Ke, C\. Yin, C\. Qiu, Z\. Rong, X\. Ma, X\. Yang, B\. Zhang, X\. Ren, Y\. Su, and A\. Weng \(2025\)Efficient uncertainty quantification for 3\-d mt inversion via variational inference\.IEEE Transactions on Geoscience and Remote Sensing63\(\),pp\. 1–15\.External Links:[Document](https://dx.doi.org/10.1109/TGRS.2025.3569960)Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p2.16)\.
- F\. Locatello, S\. Bauer, M\. Lucic, G\. Raetsch, S\. Gelly, B\. Schölkopf, and O\. Bachem \(2019\)Challenging common assumptions in the unsupervised learning of disentangled representations\.Ininternational conference on machine learning,pp\. 4114–4124\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p3.3)\.
- A\. Majumdar \(2021\)Kernelized linear autoencoder\.Neural Processing Letters53\(2\),pp\. 1597–1614\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p3.1)\.
- T\. Miconi \(2021\)Hebbian learning with gradients: hebbian convolutional neural networks with modern deep learning frameworks\.arXiv preprint arXiv:2107\.01729\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p9.3)\.
- B\. A\. Olshausen \(1998\)Linear hebbian learning and pca\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p9.3)\.
- Y\. Peiet al\.\(2018\)A study on feature extraction of handwriting data using kernel method\-based autoencoder\.In2018 9th International Conference on Awareness Science and Technology \(iCAST\),pp\. 1–6\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p3.1)\.
- Y\. Pei \(2017\)Autoencoder using kernel methoc\.In2017 IEEE International Conference on Systems, Man, and Cybernetics \(SMC\),pp\. 322–327\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p3.1)\.
- F\. Perozo, M\. Lopez,et al\.\(2024\)Principal orthogonal latent components analysis \(polca net\)\.arXiv preprint arXiv:2410\.07289\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p7.1),[§3\.3](https://arxiv.org/html/2607.05653#S3.SS3.p2.1),[§3](https://arxiv.org/html/2607.05653#S3.p6.4)\.
- C\. Pham, S\. Ladjal, and A\. Newson \(2022\)Pca\-ae: principal component analysis autoencoder for organising the latent space of generative networks\.Journal of Mathematical Imaging and Vision64\(5\),pp\. 569–585\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p5.1),[§3](https://arxiv.org/html/2607.05653#S3.p8.1)\.
- E\. Plaut \(2018\)From principal subspaces to principal components with linear autoencoders\.arXiv preprint arXiv:1804\.10253\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p2.1),[§3\.2](https://arxiv.org/html/2607.05653#S3.SS2.p3.1)\.
- Z\. Ren, Y\. Jiang, X\. Yang, Y\. Tang, and W\. Zhang \(2024\)Learnable faster kernel\-pca for nonlinear fault detection: deep autoencoder\-based realization\.Journal of Industrial Information Integration40,pp\. 100622\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p4.1)\.
- M\. Rolinek, D\. Zietlow, and G\. Martius \(2019\)Variational autoencoders pursue pca directions \(by accident\)\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 12406–12415\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p1.1),[§3](https://arxiv.org/html/2607.05653#S3.p11.1)\.
- T\. B\. Shraddha Rajpal \(2026\)15 mk\-level accuracy in all\-optical nv diamond thermometry via interpretable deep learning\.manuscript in review\.Cited by:[§4\.3](https://arxiv.org/html/2607.05653#S4.SS3.p1.1)\.
- W\. Wang, D\. Yang, F\. Chen, Y\. Pang, S\. Huang, and Y\. Ge \(2019\)Clustering with orthogonal autoencoder\.IEEE Access7,pp\. 62421–62432\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.p4.1)\.
- Q\. Zhao, E\. Adeli, N\. Honnorat, T\. Leng, and K\. M\. Pohl \(2019\)Variational autoencoder for regression: application to brain aging analysis\.InInternational conference on medical image computing and computer\-assisted intervention,pp\. 823–831\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p6.1)\.
- S\. Zhao, J\. Song, and S\. Ermon \(2017\)Towards deeper understanding of variational autoencoding models\.arXiv preprint arXiv:1702\.08658\.Cited by:[§2](https://arxiv.org/html/2607.05653#S2.SSx1.p6.1)\.

## Appendix A

This appendix collects proofs of the key results about PCA stated in the main text\. We begin in section[A\.1](https://arxiv.org/html/2607.05653#A1.SS1)by characterizing the solution set of the standard \(unordered\) PCA objective, showing that any projection onto the subspace spanned by the topkkright singular vectors is a maximizer\. We then show in section[A\.2](https://arxiv.org/html/2607.05653#A1.SS2)that this rotational ambiguity is resolved by the ordered objective whose minimizer recovers the principal components in their natural order of importance\.

### A\.1Review of Standard PCA Results

The PCA projection is typically characterized by the following maximum variance projection problem,

maxW⁡Tr​\(WT​XT​X​W\),subject to​WT​W=Ik×k\\max\_\{W\}\\text\{Tr\}\(W^\{T\}X^\{T\}XW\),\\quad\\text\{subject to \}W^\{T\}W=I\_\{k\\times k\}\(11\)whereTr​\(WT​XT​X​W\)=‖X​W‖F2\\text\{Tr\}\(W^\{T\}X^\{T\}XW\)=\|\|XW\|\|\_\{F\}^\{2\}is the variance of the projection into the latent space\. The orthogonality constraint isWT​W−I=0W^\{T\}W\-I=0, so we have the following Lagrangian formulation:

ℒ​\(W,Λ\)=Tr​\(WT​XT​X​W\)−Tr​\(Λ​\(WT​W−I\)\)\.\\mathcal\{L\}\(W,\\Lambda\)=\\text\{Tr\}\(W^\{T\}X^\{T\}XW\)\-\\text\{Tr\}\(\\Lambda\(W^\{T\}W\-I\)\)\.To characterize solutions of the above maximization, first note the following:

###### Lemma 5

Only the symmetric part ofΛ\\Lambdaaffects the Lagrangianℒ​\(W,Λ\)\\mathcal\{L\}\(W,\\Lambda\)\. The Lagrangian is also invariant to orthogonal transformations applied to the right side ofWW\.

ProofAny square matrixΛ\\Lambdacan be decomposed uniquely asΛ=Λs\+Λa\\Lambda=\\Lambda\_\{s\}\+\\Lambda\_\{a\}, whereΛs=12​\(Λ\+ΛT\)\\Lambda\_\{s\}=\\frac\{1\}\{2\}\(\\Lambda\+\\Lambda^\{T\}\)andΛa=12​\(Λ−ΛT\)\\Lambda\_\{a\}=\\frac\{1\}\{2\}\(\\Lambda\-\\Lambda^\{T\}\), withΛsT=Λs\\Lambda\_\{s\}^\{T\}=\\Lambda\_\{s\}\(symmetric\) andΛaT=−Λa\\Lambda\_\{a\}^\{T\}=\-\\Lambda\_\{a\}\(skew\-symmetric\)\. SinceS=WT​W−IS=W^\{T\}W\-Iis symmetric andΛa\\Lambda\_\{a\}is skew symmetric, we have

Tr​\(Λa​\(WT​W−I\)\)\\displaystyle\\text\{Tr\}\(\\Lambda\_\{a\}\(W^\{T\}W\-I\)\)=Tr​\(\(Λa​S\)T\)\\displaystyle=\\text\{Tr\}\(\(\\Lambda\_\{a\}S\)^\{T\}\)=Tr​\(ST​ΛaT\)\\displaystyle=\\text\{Tr\}\(S^\{T\}\\Lambda\_\{a\}^\{T\}\)=Tr​\(S​\(−Λa\)\)\\displaystyle=\\text\{Tr\}\(S\(\-\\Lambda\_\{a\}\)\)=−Tr​\(Λa​S\),\\displaystyle=\-\\text\{Tr\}\(\\Lambda\_\{a\}S\),so the trace must be zero\. Thus the Lagrangian only depends onΛs\\Lambda\_\{s\}\.

Next, consider an orthogonal transformationQQ\. For anyWWsatisfyingWT​W=IW^\{T\}W=I, we have thatW~=W​Q\\tilde\{W\}=WQ, also satisfiesW~T​W~=I\\tilde\{W\}^\{T\}\\tilde\{W\}=I\. Moreover, the PCA objective \([11](https://arxiv.org/html/2607.05653#A1.E11)\) satisfies

Tr​\(WT​XT​X​W\)=Tr​\(W~T​XT​X​W~\)\.\\text\{Tr\}\(W^\{T\}X^\{T\}XW\)=\\text\{Tr\}\(\\tilde\{W\}^\{T\}X^\{T\}X\\tilde\{W\}\)\.Thus PCA as characterized by \([11](https://arxiv.org/html/2607.05653#A1.E11)\) is right\-orthogonally invariant and solutions are only unique up to rotations and reflections\.

Now we can derive the following Lagrangian formulation of the PCA subspace defined up to orthogonal transformation:

###### Theorem 6

LetX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}, with singular value decompositionX=U​S​V⊤X=USV^\{\\top\}, then

∑i=1ksi2=maxW∈ℝn×k⁡Tr​\(W⊤​X⊤​X​W\)\+Tr​\(Λ​\(W⊤​W−I\)\)\\sum\_\{i=1\}^\{k\}s\_\{i\}^\{2\}=\\max\_\{W\\in\\mathbb\{R\}^\{n\\times k\}\}\{\\rm Tr\}\(W^\{\\top\}X^\{\\top\}XW\)\+\{\\rm Tr\}\(\\Lambda\(W^\{\\top\}W\-I\)\)wheresis\_\{i\}are the singular values ofXXin decreasing order, and the maximum is achieved whenW=V\[k\]​Q=V​In×k​QW=V\_\{\[k\]\}Q=VI\_\{n\\times k\}QwhereQ∈ℝk×kQ\\in\\mathbb\{R\}^\{k\\times k\}is any orthogonal matrix\.

ProofTaking the derivative of the Lagrangian we have:

∂ℒ∂W=2​XT​X​W−2​W​Λ=0⟹XT​X​W=W​Λ\.\\frac\{\\partial\\mathcal\{L\}\}\{\\partial W\}=2X^\{T\}XW\-2W\\Lambda=0\\implies X^\{T\}XW=W\\Lambda\.By the previous lemma we may assumeΛ\\Lambdais symmetric, in particular it can be diagonalized asΛ=Q​D​QT\\Lambda=QDQ^\{T\}\(DDdiagonal\), and we may freely choose the basis whereΛ\\Lambdais diagonal\. In that case, the stationarity condition becomes:

XT​X​W~\\displaystyle X^\{T\}X\\tilde\{W\}=XT​X​\(W​Q\)\\displaystyle=X^\{T\}X\(WQ\)=\(XT​X​W\)​Q\\displaystyle=\(X^\{T\}XW\)Q=\(W​Λ\)​Q\\displaystyle=\(W\\Lambda\)Q=W​\(Q​D​QT\)​Q\\displaystyle=W\(QDQ^\{T\}\)Q=\(W​Q\)​D\\displaystyle=\(WQ\)D=W~​D\\displaystyle=\\tilde\{W\}Dwhich is an eigenproblem where each column ofW~\\tilde\{W\}is an eigenvector ofXT​XX^\{T\}X,*i\.e\.*the right singular vectors ofXX\. In order to maximize the trace objective, the columns ofW~∈ℝn×k\\tilde\{W\}\\in\\mathbb\{R\}^\{n\\times k\}should contain the leadingkksingular vectors ofXX, which is the classical result from PCA\. In that case we have

W~=W​Q=V\[k\]⟹W=V\[k\]​Q⊤=V​In×k​Q⊤\\tilde\{W\}=WQ=V\_\{\[k\]\}\\implies W=V\_\{\[k\]\}Q^\{\\top\}=VI\_\{n\\times k\}Q^\{\\top\}and

maxW⊤​W=Ik×k​‖X​W‖F2=‖X​V\[k\]​Q⊤‖F2=‖U​S​In×k​Q⊤‖F2=∑i=1ksi2\\max\_\{W^\{\\top\}W=I\_\{k\\times k\}\}\|\|XW\|\|\_\{F\}^\{2\}=\\\|XV\_\{\[k\]\}Q^\{\\top\}\\\|\_\{F\}^\{2\}=\\\|USI\_\{n\\times k\}Q^\{\\top\}\\\|\_\{F\}^\{2\}=\\sum\_\{i=1\}^\{k\}s\_\{i\}^\{2\}Note that the above Lagrangian gives a projectionWWinto the PCA subspace spanned by the leadingkkright singular vectors\. However, this optimization problem only recovers the PCA subspace \(sinceWWis only defined up to an orthogonal transformationQQ\), and does not order the the intrinsic coordinates in the latent space\.

### A\.2PCA With Ordering

To recover the principal components in their natural order, we augment the reconstruction objective with a sequence of nested terms indexed by the columns ofWW\. The resulting objective breaks the rotational symmetry of the unordered problem and, as we show below, admits a minimizer equal to the ordered principal componentsv1,v2,…,vkv\_\{1\},v\_\{2\},\\dots,v\_\{k\}up to sign change\. Consider first the following important lemma:

###### Lemma 7

IfW∈ℝn×kW\\in\\mathbb\{R\}^\{n\\times k\}is a global minimizer of the function

F​\(W\)=‖X⊤−W​W⊤​X⊤‖F2F\(W\)=\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}thenW​W⊤WW^\{\\top\}is the orthogonal projector ontorange​\(W\)\\mathrm\{range\}\(W\)\. If additionallyWWis full rank, then it’s columns are orthonormal\.

ProofNote that‖X⊤−W​W⊤​X⊤‖F2=‖\(I−W​W⊤\)​X⊤‖F2\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}=\\\|\(I\-WW^\{\\top\}\)X^\{\\top\}\\\|\_\{F\}^\{2\}\. Using the identity‖M‖F2=tr​\(M​M⊤\)\\\|M\\\|\_\{F\}^\{2\}=\\mathrm\{tr\}\(MM^\{\\top\}\)withM:=\(I−W​W⊤\)​X⊤,M:=\(I\-WW^\{\\top\}\)X^\{\\top\},then

‖\(I−W​W⊤\)​X⊤‖F2\\displaystyle\\\|\(I\-WW^\{\\top\}\)X^\{\\top\}\\\|\_\{F\}^\{2\}=tr​\(M​M⊤\)\\displaystyle=\\mathrm\{tr\}\(MM^\{\\top\}\)=tr​\(\(I−W​W⊤\)​X⊤​\(\(I−W​W⊤\)​X⊤\)⊤\)\.\\displaystyle=\\mathrm\{tr\}\(\(I\-WW^\{\\top\}\)X^\{\\top\}\\,\(\(I\-WW^\{\\top\}\)X^\{\\top\}\)^\{\\top\}\)\.Compute the transpose:

\(\(I−W​W⊤\)​X⊤\)⊤=X​\(I−W​W⊤\)⊤=X​\(I−W​W⊤\),\(\(I\-WW^\{\\top\}\)X^\{\\top\}\)^\{\\top\}=X\(I\-WW^\{\\top\}\)^\{\\top\}=X\(I\-WW^\{\\top\}\),sinceI−W​W⊤I\-WW^\{\\top\}is symmetric\. LettingS:=X⊤​X∈ℝn×nS:=X^\{\\top\}X\\in\\mathbb\{R\}^\{n\\times n\}, we have

‖\(I−W​W⊤\)​X⊤‖F2\\displaystyle\\\|\(I\-WW^\{\\top\}\)X^\{\\top\}\\\|\_\{F\}^\{2\}=tr​\(\(I−W​W⊤\)​X⊤​X​\(I−W​W⊤\)\)\\displaystyle=\\mathrm\{tr\}\(\(I\-WW^\{\\top\}\)X^\{\\top\}X\(I\-WW^\{\\top\}\)\)=tr​\(\(I−W​W⊤\)​S​\(I−W​W⊤\)\)\.\\displaystyle=\\mathrm\{tr\}\(\(I\-WW^\{\\top\}\)S\(I\-WW^\{\\top\}\)\)\.Then use cyclicity of the trace to move the left factor to the right:

tr​\(\(I−W​W⊤\)​S​\(I−W​W⊤\)\)=tr​\(\(I−W​W⊤\)​\(I−W​W⊤\)​S\)=tr​\(\(I−W​W⊤\)2​S\)\.\\mathrm\{tr\}\(\(I\-WW^\{\\top\}\)S\(I\-WW^\{\\top\}\)\)=\\mathrm\{tr\}\(\(I\-WW^\{\\top\}\)\(I\-WW^\{\\top\}\)S\)=\\mathrm\{tr\}\(\(I\-WW^\{\\top\}\)^\{2\}S\)\.
Thus

‖X⊤−W​W⊤​X⊤‖F2\\displaystyle\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}=‖\(I−W​W⊤\)​X⊤‖F2\\displaystyle=\\\|\(I\-WW^\{\\top\}\)X^\{\\top\}\\\|\_\{F\}^\{2\}=tr​\(\(I−W​W⊤\)2​S\)\.\\displaystyle=\\mathrm\{tr\}\(\(I\-WW^\{\\top\}\)^\{2\}\\,S\)\.
For any solutionWWconsider𝒰:=range⁡\(W\)\\mathcal\{U\}:=\\operatorname\{range\}\(W\)\. LetP=P𝒰P=P\_\{\\mathcal\{U\}\}be the orthogonal projector onto𝒰\\mathcal\{U\},*i\.e\.*P2=PP^\{2\}=PandP​x=xPx=xfor allx∈𝒰x\\in\\mathcal\{U\}\. SinceA:=W​W⊤A:=WW^\{\\top\}maps into𝒰\\mathcal\{U\}, we have\(I−P\)​A=0\(I\-P\)A=0, and sinceAAis symmetric we also haveA​\(I−P\)=0A\(I\-P\)=0\. Hence

I−A=\(I−P\)\+\(P−A\)with\(I−P\)​\(P−A\)=0=\(P−A\)​\(I−P\)\.I\-A=\(I\-P\)\+\(P\-A\)\\quad\\text\{with\}\\quad\(I\-P\)\(P\-A\)=0=\(P\-A\)\(I\-P\)\.So the cross terms in\(I−A\)2\(I\-A\)^\{2\}vanish and we have

\(I−A\)2=\(I−P\)2\+\(P−A\)2=\(I−P\)\+\(P−A\)2,\(I\-A\)^\{2\}=\(I\-P\)^\{2\}\+\(P\-A\)^\{2\}=\(I\-P\)\+\(P\-A\)^\{2\},and so

tr​\(\(I−A\)2​S\)=tr​\(\(I−P\)​S\)\+tr​\(\(P−A\)2​S\)\.\\mathrm\{tr\}\(\(I\-A\)^\{2\}S\)=\\mathrm\{tr\}\(\(I\-P\)S\)\+\\mathrm\{tr\}\(\(P\-A\)^\{2\}S\)\.
It is not hard to show thattr​\(M​N\)≥0\\mathrm\{tr\}\(MN\)\\geq 0for two symmetric PSD matricesM,NM,N\(sincetr​\(M​N\)=tr​\(M1/2​N​M1/2\)\\mathrm\{tr\}\(MN\)=\\mathrm\{tr\}\(M^\{1/2\}NM^\{1/2\}\)andx⊤​\(M1/2​N​M1/2\)​x=\(M1/2​x\)⊤​N​\(M1/2​x\)≥0x^\{\\top\}\(M^\{1/2\}NM^\{1/2\}\)x=\(M^\{1/2\}x\)^\{\\top\}N\(M^\{1/2\}x\)\\geq 0\)\. Then sinceS=XT​XS=X^\{T\}Xis symmetric PSD and\(P−A\)2\(P\-A\)^\{2\}is symmetric PSD \(sincex⊤​\(P−A\)2​x=\(\(P−A\)​x\)⊤​\(\(P−A\)​x\)=‖\(P−A\)​x‖2≥0x^\{\\top\}\(P\-A\)^\{2\}x=\(\(P\-A\)x\)^\{\\top\}\(\(P\-A\)x\)=\\\|\(P\-A\)x\\\|^\{2\}\\geq 0\), we have thattr​\(\(P−A\)2​S\)≥0\\mathrm\{tr\}\(\(P\-A\)^\{2\}S\)\\geq 0and hence

tr​\(\(I−A\)2​S\)\\displaystyle\\mathrm\{tr\}\(\(I\-A\)^\{2\}S\)=tr​\(\(I−P\)​S\)\+tr​\(\(P−A\)2​S\)\\displaystyle=\\mathrm\{tr\}\(\(I\-P\)S\)\+\\mathrm\{tr\}\(\(P\-A\)^\{2\}S\)≥tr​\(\(I−P\)​S\)\.\\displaystyle\\geq\\mathrm\{tr\}\(\(I\-P\)S\)\.Equality holds ifftr​\(\(P−A\)2​S\)=0\\mathrm\{tr\}\(\(P\-A\)^\{2\}S\)=0\. In particular, ifSSis positive definite on𝒰\\mathcal\{U\}this forcesA=PA=P\. In other words, for a fixed subspace𝒰\\mathcal\{U\}, the first term is minimized whenW​W⊤WW^\{\\top\}is exactly the orthogonal projectorPP\.

Now consider the columns ofWW\. Each columnwj∈𝒰w\_\{j\}\\in\\mathcal\{U\}is fixed byW​W⊤=P𝒰WW^\{\\top\}=P\_\{\\mathcal\{U\}\},*i\.e\.*W​W⊤​wj=wjWW^\{\\top\}w\_\{j\}=w\_\{j\}for eachjj\. Collecting these identities column\-wise gives

W​W⊤​W=W⟹W​\(W⊤​W−Ik\)=0\.WW^\{\\top\}W=W\\implies W\(W^\{\\top\}W\-I\_\{k\}\)=0\.IfWWis full rank, it’s null space is trivial and we obtainW⊤​W−Ik=0W^\{\\top\}W\-I\_\{k\}=0, and thusW⊤​W=IkW^\{\\top\}W=I\_\{k\}\.

With lemma[7](https://arxiv.org/html/2607.05653#Thmtheorem7)in hand, the equivalence between PCA maximum trace and minimum reconstruction objectives is straightforward:

###### Theorem 8

For anyX∈ℝN×n,W∈ℝn×kX\\in\\mathbb\{R\}^\{N\\times n\},W\\in\\mathbb\{R\}^\{n\\times k\}we have the following equivalence:

maxW⊤​W=I⁡Tr​\(W⊤​X⊤​X​W\)=minW⁡‖X⊤−W​W⊤​X⊤‖F2\\max\_\{W^\{\\top\}W=I\}\\text\{Tr\}\(W^\{\\top\}X^\{\\top\}XW\)=\\min\_\{W\}\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}

ProofConsider the Frobenius norm ofX⊤−W​W⊤​X⊤X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\. By lemma[7](https://arxiv.org/html/2607.05653#Thmtheorem7), this is minimized whenW​W⊤WW^\{\\top\}is the orthogonal projector onto the range ofWW,*i\.e\.*\(W​W⊤\)2=W​W⊤\(WW^\{\\top\}\)^\{2\}=WW^\{\\top\}\. Hence

‖X⊤−W​W⊤​X⊤‖F2\\displaystyle\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}=Tr​\[\(X⊤−W​W⊤​X⊤\)⊤​\(X⊤−W​W⊤​X⊤\)\]\\displaystyle=\\text\{Tr\}\\left\[\\left\(X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\right\)^\{\\top\}\\left\(X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\right\)\\right\]=Tr​\[\(X−X​W​W⊤\)​\(X⊤−W​W⊤​X⊤\)\]\\displaystyle=\\text\{Tr\}\\left\[\\left\(X\-XWW^\{\\top\}\\right\)\\left\(X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\right\)\\right\]=Tr​\[X​X⊤−X​W​W⊤​X⊤−X​W​W⊤​X⊤\+X​\(W​W⊤\)2​X⊤\]\\displaystyle=\\text\{Tr\}\\left\[XX^\{\\top\}\-XWW^\{\\top\}X^\{\\top\}\-XWW^\{\\top\}X^\{\\top\}\+X\(WW^\{\\top\}\)^\{\\cancel\{2\}\}X^\{\\top\}\\right\]=Tr​\(XT​X\)−2​Tr​\(W⊤​X⊤​X​W\)\+Tr​\(W⊤​X⊤​X​W\)\\displaystyle=\\text\{Tr\}\(X^\{T\}X\)\-2\\text\{Tr\}\(W^\{\\top\}X^\{\\top\}XW\)\+\\text\{Tr\}\(W^\{\\top\}X^\{\\top\}XW\)=‖X‖F2−Tr​\(W⊤​X⊤​X​W\)\\displaystyle=\\\|X\\\|\_\{F\}^\{2\}\-\\text\{Tr\}\(W^\{\\top\}X^\{\\top\}XW\)Using the fact that the trace of a product of matrices is invariant under cyclic permutations\. Since‖X‖F2\\\|X\\\|\_\{F\}^\{2\}is constant with respect toWW, we thus have

minW⁡‖X⊤−W​W⊤​X⊤‖F2⟹maxW⊤​W=I⁡Tr​\(W⊤​X⊤​X​W\)\.\\min\_\{W\}\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}\\implies\\max\_\{W^\{\\top\}W=I\}\\text\{Tr\}\(W^\{\\top\}X^\{\\top\}XW\)\.Starting frommaxW⊤​W=I⁡Tr​\(W⊤​X⊤​X​W\)\\max\_\{W^\{\\top\}W=I\}\\text\{Tr\}\(W^\{\\top\}X^\{\\top\}XW\), a similar argument follows in reverse usingW⊤​W=IW^\{\\top\}W=I\. Thus we have

minW⁡‖X⊤−W​W⊤​X⊤‖F2⇔maxW⊤​W=I⁡Tr​\(W⊤​X⊤​X​W\)\.\\min\_\{W\}\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}\\iff\\max\_\{W^\{\\top\}W=I\}\\text\{Tr\}\(W^\{\\top\}X^\{\\top\}XW\)\.
Now we are ready to prove the main theorem for ordered PCA:

###### Theorem 9

LetX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}, with singular value decompositionX=U​S​V⊤X=USV^\{\\top\}and distinct singular valuess1\>s2\>⋯\>sk\>0s\_\{1\}\>s\_\{2\}\>\\cdots\>s\_\{k\}\>0, then

k​s12\+\(k−1\)​s22\+⋯\+2​sk−12\+sk2=minW∈ℝn×k​∑i=1k‖X⊤−W\[i\]​W\[i\]⊤​X⊤‖F2,ks\_\{1\}^\{2\}\+\(k\-1\)s\_\{2\}^\{2\}\+\\cdots\+2s\_\{k\-1\}^\{2\}\+s\_\{k\}^\{2\}=\\min\_\{W\\in\\mathbb\{R\}^\{n\\times k\}\}\\sum\_\{i=1\}^\{k\}\\\|X^\{\\top\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\},and the minimum is achieved whenW=V\[k\]​Σ=V​In×k​ΣW=V\_\{\[k\]\}\\Sigma=VI\_\{n\\times k\}\\Sigma, whereΣ=diag​\(ε1,…,εk\)\\Sigma=\\mathrm\{diag\}\(\\varepsilon\_\{1\},\\dots,\\varepsilon\_\{k\}\), withεj=±1\\varepsilon\_\{j\}=\\pm 1\.

ProofLetJ​\(W\):=∑i=1k‖X⊤−W\[i\]​W\[i\]⊤​X⊤‖F2J\(W\):=\\sum\_\{i=1\}^\{k\}\\\|X^\{\\top\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}be the error of the full summed prefix objective andmi:=minW\[i\]​‖X⊤−W\[i\]​W\[i\]⊤​X⊤‖F2m\_\{i\}:=\\mathrm\{min\}\_\{W\_\{\[i\]\}\}\\\|X^\{\\top\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}the minimum of individualii\-dimensional PCA objectives\. For each fixedii, we know thatmim\_\{i\}is attainted whenW\[i\]=V\[i\]​QiW\_\{\[i\]\}=V\_\{\[i\]\}Q\_\{i\}, however, this does not guarantee the optimality of any earlier prefixes\.

Instead of optimizing over eachii\-dimensional subspace independently, the minimizer ofJ​\(W\)J\(W\)is a single shared matrixWWwhose prefixes must simultaneously define optimal subspaces of dimensions1,…,k1,\\ldots,k\. This means that instead of choosing the rotationsQ1,…,QkQ\_\{1\},\\ldots,Q\_\{k\}independently, they must be selected simultaneously so that optimality is preserved at every prefix level\.

In generalJ​\(W\)≥∑i=1kmiJ\(W\)\\geq\\sum\_\{i=1\}^\{k\}m\_\{i\}, and this coupling may initially appear to severely restrict the set of viable solutions\. However, we already know that such a solution exists, namelyW=V\[k\]W=V\_\{\[k\]\}\. In that case,W\[i\]=V\[i\]for all​i=1,…,k,W\_\{\[i\]\}=V\_\{\[i\]\}\\quad\\text\{for all \}i=1,\\ldots,k,and sincespan​\(V\[i\]\)\\mathrm\{span\}\(V\_\{\[i\]\}\)is the unique optimalii\-dimensional principal subspace under the distinct singular value assumption, each prefixW\[i\]W\_\{\[i\]\}attainsmim\_\{i\}individually\. Therefore

J​\(V\[k\]\)=∑i=1kmi,J\(V\_\{\[k\]\}\)=\\sum\_\{i=1\}^\{k\}m\_\{i\},and combined with the universal lower boundJ​\(W\)≥∑i=1kmiJ\(W\)\\geq\\sum\_\{i=1\}^\{k\}m\_\{i\}, this establishes thatV\[k\]V\_\{\[k\]\}is a global minimizer ofJJ\.

Thus, any global minimizerW∗W^\{\*\}of the fullJJmust satisfy that every prefixW\[i\]∗W^\{\*\}\_\{\[i\]\}solves theii\-dimensional PCA problem\. Sinces1\>⋯\>sk\>0s\_\{1\}\>\\dots\>s\_\{k\}\>0, the topkkright singular subspaces are distinct and we obtainspan​\(W\[i\]∗\)=span​\(V\[i\]\)\\mathrm\{span\}\(W^\{\*\}\_\{\[i\]\}\)=\\mathrm\{span\}\(V\_\{\[i\]\}\)for alli=1,…,ki=1,\\dots,k\. Moreover, eachW\[i\]∗W^\{\*\}\_\{\[i\]\}is a global minimizer of theii\-th term, so by lemma[7](https://arxiv.org/html/2607.05653#Thmtheorem7)W\[i\]∗​\(W\[i\]∗\)⊤W^\{\*\}\_\{\[i\]\}\(W^\{\*\}\_\{\[i\]\}\)^\{\\top\}is an orthogonal projector ontorange​\(W\[i\]∗\)\\mathrm\{range\}\(W^\{\*\}\_\{\[i\]\}\); in particular,W\[i\]∗W^\{\*\}\_\{\[i\]\}has orthonormal columns\.

To show exactness of solutionsV\[k\]​ΣV\_\{\[k\]\}\\Sigma, we argue inductively onii\. Fori=1i=1,span​\(w1∗\)=span​\(v1\)\\mathrm\{span\}\(w\_\{1\}^\{\*\}\)=\\mathrm\{span\}\(v\_\{1\}\), hencew1∗=±v1w\_\{1\}^\{\*\}=\\pm v\_\{1\}\. Assumewj∗=±vjw\_\{j\}^\{\*\}=\\pm v\_\{j\}forj<ij<i\. Sincespan​\(W\[i\]∗\)=span​\(V\[i\]\)\\mathrm\{span\}\(W^\{\*\}\_\{\[i\]\}\)=\\mathrm\{span\}\(V\_\{\[i\]\}\)andspan​\(W\[i−1\]∗\)=span​\(V\[i−1\]\)\\mathrm\{span\}\(W^\{\*\}\_\{\[i\-1\]\}\)=\\mathrm\{span\}\(V\_\{\[i\-1\]\}\), orthonormality implies

wi∗∈span​\(V\[i\]\)∩span​\(V\[i−1\]\)⟂=span​\(vi\),w\_\{i\}^\{\*\}\\in\\mathrm\{span\(V\_\{\[i\]\}\)\\cap\\mathrm\{span\(V\_\{\[i\-1\]\}\)^\{\\perp\}\}\}=\\mathrm\{span\}\(v\_\{i\}\),sowi∗=±viw\_\{i\}^\{\*\}=\\pm v\_\{i\}\. ThusW∗=V\[k\]​ΣW^\{\*\}=V\_\{\[k\]\}\\Sigma, whereΣ=diag​\(ε1,…,εk\)\\Sigma=\\mathrm\{diag\}\(\\varepsilon\_\{1\},\\dots,\\varepsilon\_\{k\}\), withεj=±1\\varepsilon\_\{j\}=\\pm 1\.

## Appendix B

This appendix establishes the theoretical behavior of the ODIN loss functions in the linear regime\. In general we have two different ways to proceed: either we can augment standard reconstruction minimization with an orthogonality term to recover an unordered version ofVV, or we can break the symmetry with dendrites to recoverVVwith the correct ordering\.

Section[B\.1](https://arxiv.org/html/2607.05653#A2.SS1)treats the baseline case of plain reconstruction combined with an orthogonality penalty, showing that this recovers the principal components but without any guaranteed ordering of components\. Section[B\.2](https://arxiv.org/html/2607.05653#A2.SS2)then shows that replacing the reconstruction term with a dendritic reconstruction \(even without any orthogonality constraint\) is sufficient to recover the principal components in order of importance\. Finally, section[B\.3](https://arxiv.org/html/2607.05653#A2.SS3)considers the full ODIN objective combining both dendritic reconstruction and orthogonality regularization, confirming that ordered recovery is preserved under the complete loss\. See table[2](https://arxiv.org/html/2607.05653#A2.T2)for an overview of closed\-form optimal solution sets in the linear regime\.

Table 2:Optimal solutions for considered loss objectives in the linear regime\. Solutions progress from under\-constrained to fully identified: standard PCA recovers any orthogonal rotation ofV\[k\]V\_\{\[k\]\}, adding an orthogonality penalty narrows this to any column permutation ofV\[k\]V\_\{\[k\]\}, the dendrite loss alone \(and combined with orthogonality\) uniquely identifiesV\[k\]V\_\{\[k\]\}up to sign\.### B\.1Plain MSE with Orthogonality

We begin with the simpler linear setup consisting of a single reconstruction term paired with an orthogonality penalty on latent correlations\. This combination is the most natural starting point, as the orthogonality constraint alone might be expected to pin down the solution\. However, as we show below, the resulting objective recovers only the principal components and does not enforce any ordering among solutions\.

In this setting, the loss function takes the form:

ℒMSE\+ℒorth=minW⁡‖X⊤−W​W⊤​X⊤‖F2\+∑i<j\(W⊤​X⊤​X​W\)i​j2\\mathcal\{L\}\_\{\\text\{MSE\}\}\+\\mathcal\{L\}\_\{\\text\{orth\}\}=\\min\_\{W\}\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}\+\\sum\_\{i<j\}\(W^\{\\top\}X^\{\\top\}XW\)\_\{ij\}^\{2\}\(12\)and is minimized by some permutation ofVV\.

First we show that this combined loss obtained by adding the orthogonality term is also minimized by an ‘orthogonal projector’\.

###### Lemma 10

IfW∈ℝn×kW\\in\\mathbb\{R\}^\{n\\times k\}is a global minimizer of the function

F​\(W\)=‖X⊤−W​W⊤​X⊤‖F2\+∑i<j\(W⊤​X⊤​X​W\)i​j2F\(W\)=\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}\+\\sum\_\{i<j\}\(W^\{\\top\}X^\{\\top\}XW\)\_\{ij\}^\{2\}thenW​W⊤WW^\{\\top\}is the orthogonal projector ontorange​\(W\)\\mathrm\{range\}\(W\)\.

Proof

As in lemma[7](https://arxiv.org/html/2607.05653#Thmtheorem7), chooseW∈ℝn×kW\\in\\mathbb\{R\}^\{n\\times k\}with orthonormal columns spanning𝒰=range​\(W\)\\mathcal\{U\}=\\mathrm\{range\}\(W\)to minimize the first term\. Now, for the second term, setB:=W⊤​S​WB:=W^\{\\top\}SW, which is symmetric\. By the spectral theorem, there exists orthogonalQQsuch that

Q⊤​B​Q=Λis diagonal\.Q^\{\\top\}BQ=\\Lambda\\quad\\text\{is diagonal\}\.Then we have

\(Q⊤​W⊤\)​S​\(W​Q\)=W′⁣⊤​S​W′=Λ,\(Q^\{\\top\}W^\{\\top\}\)S\(WQ\)=W^\{\\prime\\top\}SW^\{\\prime\}=\\Lambda,so all off\-diagonal entries vanish and the second term becomes0, i\.e\.

∑i<j\(W′⁣⊤​S​W′\)i​j2=0\.\\sum\_\{i<j\}\(W^\{\\prime\\top\}SW^\{\\prime\}\)\_\{ij\}^\{2\}=0\.Moreover, the orthogonal transformationW↦W′=W​QW\\mapsto W^\{\\prime\}=WQdoes not affect the first term since, for anyQ∈ℝk×kQ\\in\\mathbb\{R\}^\{k\\times k\}orthogonal, we have‖XT−W​WT​XT‖F2=‖X⊤−\(W​Q\)​\(W​Q\)⊤​X⊤‖F2\\\|X^\{T\}\-WW^\{T\}X^\{T\}\\\|\_\{F\}^\{2\}=\\\|X^\{\\top\}\-\(WQ\)\(WQ\)^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}\. Thus, the “orthogonal projector” does not increase the first term, and the subsequent orthogonal rotationW↦W′=W​QW\\mapsto W^\{\\prime\}=WQkeeps the first term fixed while making the second term as small as possible \(indeed0\)\.

The following lemma will be useful in showing that the solution to \([12](https://arxiv.org/html/2607.05653#A2.E12)\) is any permutation ofVV:

###### Lemma 11

LetS=Diag​\(s1,…,sn\)S=\\text\{\{Diag\}\}\(s\_\{1\},\\dots,s\_\{n\}\)be a diagonal matrix with distinctsi≠0s\_\{i\}\\neq 0and suppose thatQQis any orthogonal matrix satisfyingQ⊤​S​Q=DQ^\{\\top\}SQ=D, whereDDis any full rank diagonal matrix\. ThenQQis a permutation matrix\.

ProofThe hypothesisQ⊤​S​Q=DQ^\{\\top\}SQ=Dis exactly the condition

qi⊤​S​qj=∑k=1nsk​qk​i​qk​j=0,∀i≠j\.q\_\{i\}^\{\\top\}Sq\_\{j\}=\\sum\_\{k=1\}^\{n\}s\_\{k\}q\_\{ki\}q\_\{kj\}=0,\\quad\\forall i\\neq j\.IfQQis orthogonal, we have∑kqk​i​qk​j=0\\sum\_\{k\}q\_\{ki\}q\_\{kj\}=0fori≠ji\\neq j\. Multiplying bys1s\_\{1\}and subtracting from the above equation we get

∑k=2n\(sk−s1\)​qk​i​qk​j=0\.\\sum\_\{k=2\}^\{n\}\(s\_\{k\}\-s\_\{1\}\)q\_\{ki\}q\_\{kj\}=0\.We can repeat this process and since allsis\_\{i\}are distinct, all\(sk−s1\),\(sk−s2\),…\(s\_\{k\}\-s\_\{1\}\),\(s\_\{k\}\-s\_\{2\}\),\\dotsare non\-zero\. This forcesqk​i​qk​j=0q\_\{ki\}q\_\{kj\}=0for allkkandi≠ji\\neq j\. This means that for each rowkkofQQ, at most one column has a nonzero entry in that row\. If we additionally assume thatDDis full rank, then it is invertible andQQmust be invertible also\. Since invertible matrices cannot have all columns/rows completely zero, we conclude thatQQhas exactly one non\-zero entry per\-column per\-row\. SinceQQis orthogonal this non\-zero entry must be equal to±1\\pm 1, giving the form of a permutation matrix\.

We are now ready to prove the result:

###### Theorem 12

LetX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}, with singular value decompositionX=U​S​V⊤X=USV^\{\\top\}and distinct singular valuess1\>s2\>⋯\>sk\>0s\_\{1\}\>s\_\{2\}\>\\cdots\>s\_\{k\}\>0, then

∑i=1ksi2=minW∈ℝn×k⁡‖X⊤−W​W⊤​X⊤‖\+∑i<j\(W⊤​X⊤​X​W\)i​j2\\sum\_\{i=1\}^\{k\}s\_\{i\}^\{2\}=\\min\_\{W\\in\\mathbb\{R\}^\{n\\times k\}\}\\\|X^\{\\top\}\-WW^\{\\top\}X^\{\\top\}\\\|\+\\sum\_\{i<j\}\(W^\{\\top\}X^\{\\top\}XW\)\_\{ij\}^\{2\}wheresis\_\{i\}are the singular values ofXXin decreasing order, and the minimum is achieved whenW=V\[k\]​AW=V\_\{\[k\]\}Afor some permutation matrixAA\.

ProofNote that the second term is minimized whenW⊤​X⊤​X​WW^\{\\top\}X^\{\\top\}XWis diagonal,*i\.e\.*

W⊤​X⊤​X​W=W⊤​V​S2​V⊤​W=D,W^\{\\top\}X^\{\\top\}XW=W^\{\\top\}VS^\{2\}V^\{\\top\}W=D,for some diagonal matrixDD\. By lemma[11](https://arxiv.org/html/2607.05653#Thmtheorem11),A=V⊤​WA=V^\{\\top\}Wis a permutation matrix precisely whenWWis orthogonal\. However, by lemma[10](https://arxiv.org/html/2607.05653#Thmtheorem10), we may assume that any global minimizerWWmay take the form that yields the orthogonal projectorW​W⊤=P𝒰WW^\{\\top\}=P\_\{\\mathcal\{U\}\}, with𝒰=range​\(W\)\\mathcal\{U\}=\\mathrm\{range\}\(W\)\. In particular, ifWWis full rank thenW​W⊤=P𝒰WW^\{\\top\}=P\_\{\\mathcal\{U\}\}necessarily implies that it’s columns are an orthonormal basis for𝒰\\mathcal\{U\}\. ThusW=V\[k\]​AW=V\_\{\[k\]\}A, whereAAis a permutation matrix\. This is the PCA projection onto the firstkkprincipal components, though not in any particular order\.

### B\.2Dendrites with no Orthogonality

Having seen that orthogonality alone is insufficient to order the principal components, we now show that the dendritic reconstruction objective achieves this ordering on its own, even in the absence of any orthogonality constraint\. Recall:

ℒDend=∑i=1kℒMSE​\(X,X^i\)=minW​∑i=1k‖X⊤−W\[i\]​W\[i\]⊤​X⊤‖F2\.\\mathcal\{L\}\_\{\\text\{Dend\}\}=\\sum\_\{i=1\}^\{k\}\\mathcal\{L\}\_\{\\text\{MSE\}\}\(X,\\hat\{X\}\_\{i\}\)=\\min\_\{W\}\\sum\_\{i=1\}^\{k\}\\\|X^\{\\top\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}\.
This is the same loss function as in theorem[9](https://arxiv.org/html/2607.05653#Thmtheorem9), which was shown to recover the true PCA solution\. Unlike the unordered objective considered above, the nested structure of the dendritic terms breaks the rotational symmetry of the solution space, and the minimizer in the linear regime coincides withV\[k\]V\_\{\[k\]\}\.

#### B\.2\.1Lower Bound for Dendritic MSE

One practical consequence of a neural network implementation of dendritic structure is the behavior of the loss during training\. The nested structure of the reconstruction terms imposes a sequence of lower bounds on the achievable error at each stage, bounding the total aggregate MSE away from zero\.

LetX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}with covariance matrixΣX\\Sigma\_\{X\}and eigenvalue decomposition

ΣX=P​Λ​P⊤,\\Sigma\_\{X\}=P\\Lambda P^\{\\top\},whereΛ=diag⁡\(λ1,λ2,…,λn\)\\Lambda=\\operatorname\{diag\}\(\\lambda\_\{1\},\\lambda\_\{2\},\\ldots,\\lambda\_\{n\}\)andλ1≥λ2≥⋯≥λn≥0\\lambda\_\{1\}\\geq\\lambda\_\{2\}\\geq\\cdots\\geq\\lambda\_\{n\}\\geq 0denote the variances along the principal components ofXX\. Since the principal components contribute additively to total variance, anykk\-dimensional reconstructionX^k\\hat\{X\}\_\{k\}satisfies

MSE​\(X,X^k\)≥∑i=k\+1nλi\.\\mathrm\{MSE\}\(X,\\hat\{X\}\_\{k\}\)\\geq\\sum\_\{i=k\+1\}^\{n\}\\lambda\_\{i\}\.Unless the data lie entirely within the span of somekk\-dimensional subspace, this bound is strictly positive\. Otherwise, any method that reconstructsXXusing a reduced latent representation of dimensionk<nk<nis inevitably unable to capture the residual variance contained in the remaining components\.

Applying this reasoning to the ODIN architecture, the partial reconstruction errorsL1,L2,…,LnL\_\{1\},L\_\{2\},\\ldots,L\_\{n\}satisfy analogous lower bounds\. Specifically,Li≥∑j=i\+1nλjL\_\{i\}\\geq\\sum\_\{j=i\+1\}^\{n\}\\lambda\_\{j\}, and the total dendritic loss is therefore bounded below by:

ℒDend=∑i=1nLi≥∑i=1n∑j=i\+1nλj,\\mathcal\{L\}\_\{\\text\{Dend\}\}=\\sum\_\{i=1\}^\{n\}L\_\{i\}\\geq\\sum\_\{i=1\}^\{n\}\\sum\_\{j=i\+1\}^\{n\}\\lambda\_\{j\},reflecting the diminishing contribution of residual variance as successive latent dimensions are incorporated\.

### B\.3Combined Loss Objective

We now analyze the full ODIN objective, combining dendritic reconstruction with orthogonality regularization\. The results of the previous two sections suggest these terms play similar roles, and we confirm that their combination does not interfere with ordered recovery\. We show that the minimizer of the full loss in the linear regime is againW=V\[k\]W=V\_\{\[k\]\}with potential sign changes\.

###### Theorem 13

LetX∈ℝN×nX\\in\\mathbb\{R\}^\{N\\times n\}, with singular value decompositionX=U​S​V⊤X=USV^\{\\top\}and distinct singular valuess1\>s2\>⋯\>sk\>0s\_\{1\}\>s\_\{2\}\>\\cdots\>s\_\{k\}\>0, then

k​s12\+\(k−1\)​s22\+⋯\+2​sk−12\+sk2=minW∈ℝn×k⁡ℒDend\+ℒorth,ks\_\{1\}^\{2\}\+\(k\-1\)s\_\{2\}^\{2\}\+\\cdots\+2s\_\{k\-1\}^\{2\}\+s\_\{k\}^\{2\}=\\min\_\{W\\in\\mathbb\{R\}^\{n\\times k\}\}\\mathcal\{L\}\_\{\\text\{Dend\}\}\+\\mathcal\{L\}\_\{\\text\{orth\}\},and the minimum is achieved whenW=V\[k\]​Σ=V​In×k​ΣW=V\_\{\[k\]\}\\Sigma=VI\_\{n\\times k\}\\Sigma, whereΣ=diag​\(ε1,…,εk\)\\Sigma=\\mathrm\{diag\}\(\\varepsilon\_\{1\},\\dots,\\varepsilon\_\{k\}\), withεj=±1\\varepsilon\_\{j\}=\\pm 1\.

ProofNote that the second term is minimized whenW⊤​X⊤​X​WW^\{\\top\}X^\{\\top\}XWis diagonal,*i\.e\.*

W⊤​X⊤​X​W=W⊤​V​S2​V⊤​W=D,W^\{\\top\}X^\{\\top\}XW=W^\{\\top\}VS^\{2\}V^\{\\top\}W=D,for some diagonal matrixDD\. By lemma[11](https://arxiv.org/html/2607.05653#Thmtheorem11),A=V⊤​WA=V^\{\\top\}Wis a permutation matrix precisely whenWWis orthogonal\. However, by lemma[7](https://arxiv.org/html/2607.05653#Thmtheorem7), we may assume that any global minimizerWWmay take the form that yields the orthogonal projectorW​W⊤=P𝒰WW^\{\\top\}=P\_\{\\mathcal\{U\}\}, with𝒰=range​\(W\)\\mathcal\{U\}=\\mathrm\{range\}\(W\)\. In particular, sinceWWis full rank thenW​W⊤=P𝒰WW^\{\\top\}=P\_\{\\mathcal\{U\}\}necessarily implies that it’s columns are orthonormal\.

So we may assume thatAAis a permutation matrix and additionally that, for fixedii,W\[i\]​W\[i\]⊤W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}is the orthogonal projector ontospan​\{V​a1,…​V​ai\}\\text\{span\}\\\{Va\_\{1\},\\dots Va\_\{i\}\\\}\. Thus

‖X⊤−W\[i\]​W\[i\]⊤​X⊤‖F2=‖X⊤‖F2−‖W\[i\]​W\[i\]⊤​X⊤‖F2\\\|X^\{\\top\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}=\\\|X^\{\\top\}\\\|\_\{F\}^\{2\}\-\\\|W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}And sinceW\[i\]⊤​X⊤=A\[i\]⊤​V⊤​V​S​U⊤=A\[i\]⊤​S​U⊤W\_\{\[i\]\}^\{\\top\}X^\{\\top\}=A\_\{\[i\]\}^\{\\top\}V^\{\\top\}VSU^\{\\top\}=A\_\{\[i\]\}^\{\\top\}SU^\{\\top\}, we have

‖W\[i\]​W\[i\]⊤​X⊤‖F2\\displaystyle\\\|W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}=Tr​\(X​W\[i\]​W\[i\]⊤​W\[i\]​W\[i\]⊤​X⊤\)\\displaystyle=\\text\{Tr\}\(XW\_\{\[i\]\}\\cancel\{W\_\{\[i\]\}^\{\\top\}W\_\{\[i\]\}\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\)=Tr​\(X​W\[i\]​W\[i\]⊤​X⊤\)\\displaystyle=\\text\{Tr\}\(XW\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\)=Tr​\(A\[i\]⊤​S2​A\[i\]\)\\displaystyle=\\text\{Tr\}\(A\_\{\[i\]\}^\{\\top\}S^\{2\}A\_\{\[i\]\}\)Thus

‖X⊤−W\[i\]​W\[i\]⊤​X⊤‖F2\\displaystyle\\\|X^\{\\top\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}=Tr​\(X⊤​X\)−Tr​\(A\[i\]⊤​S2​A\[i\]\)\\displaystyle=\\text\{Tr\}\(X^\{\\top\}X\)\-\\text\{Tr\}\(A\_\{\[i\]\}^\{\\top\}S^\{2\}A\_\{\[i\]\}\)=∑j=1nsj2−∑ℓ=1iaℓ⊤​S2​aℓ\\displaystyle=\\sum\_\{j=1\}^\{n\}s\_\{j\}^\{2\}\-\\sum\_\{\\ell=1\}^\{i\}a\_\{\\ell\}^\{\\top\}S^\{2\}a\_\{\\ell\}Summing overiiwe have

∑i=1k‖X⊤−W\[i\]​W\[i\]⊤​X⊤‖F2=k​∑j=1nsj2−∑ℓ=1k\(k−ℓ\+1\)​aℓ⊤​S2​aℓ\.\\sum\_\{i=1\}^\{k\}\\\|X^\{\\top\}\-W\_\{\[i\]\}W\_\{\[i\]\}^\{\\top\}X^\{\\top\}\\\|\_\{F\}^\{2\}=k\\sum\_\{j=1\}^\{n\}s\_\{j\}^\{2\}\-\\sum\_\{\\ell=1\}^\{k\}\(k\-\\ell\+1\)a\_\{\\ell\}^\{\\top\}S^\{2\}a\_\{\\ell\}\.So minimizing the reconstruction loss is equivalent to maximizing∑ℓ=1k\(k−ℓ\+1\)​aℓ⊤​S2​aℓ\\sum\_\{\\ell=1\}^\{k\}\(k\-\\ell\+1\)a\_\{\\ell\}^\{\\top\}S^\{2\}a\_\{\\ell\}\. SinceS2S^\{2\}has distinct eigenvalues, then by Lemma[11](https://arxiv.org/html/2607.05653#Thmtheorem11)we haveA=P​ΣA=P\\Sigma, wherePPis a permutation matrix andΣ\\Sigmais a sign matrix\. Since the weights are decreasingk\>k−1\>⋯\>1k\>k\-1\>\\dots\>1, the sum is maximized when theaℓa\_\{\\ell\}vectors are those which pick the largest singular values \(*i\.e\.*A=In×k​Σ⟹W=V​In×k​Σ=V\[k\]​ΣA=I\_\{n\\times k\}\\Sigma\\implies W=VI\_\{n\\times k\}\\Sigma=V\_\{\[k\]\}\\Sigma\)