The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

# The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning
Source: [https://arxiv.org/html/2606.04280](https://arxiv.org/html/2606.04280)
###### Abstract

Contrastive learning has become a leading paradigm for self\-supervised representation learning, yet the conditions under which it recovers meaningful latent geometry remain incompletely understood\. We develop a measure\-theoretic framework formalizing the diversity condition, a support requirement on positive\-pair sampling that is necessary for isometric latent recovery\. We show that the standard full\-support von Mises\-Fisher setting implies the satisfaction of the diversity condition and as a consequence global contrastive loss minimizers recover latent geometry up to orthogonal transformation, while restricted conditionals can make non\-orthogonal maps attain strictly lower asymptotic contrastive loss\. We introduce a support\-corrected Information Noise Contrastive Estimation \(InfoNCE\) variant as a theoretical fix: this correction makes orthogonal latent space recovery achievable but does not uniquely select it\. Experiments on synthetic benchmarks validate the identifiability predictions, and CIFAR\-10 experiments are consistent with the qualitative prediction that architectural inductive bias becomes more important when sampling diversity is limited\. Together, our results clarify how sampling mechanisms and encoder inductive bias interact in contrastive representation learning\.

Machine Learning, ICML

## 1Introduction

The machine learning community has long envisioned methods that turn vast amounts of unlabeled data into dense, robust, and reusable representations useful for many different downstream tasks such as classification, regression, and search\. Contrastive learning \(CL\) has emerged as a successful technique for achieving this goal, which in recent years has led to advances in language\(Jaiswalet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib6)\), vision\(Chenet al\.,[2020](https://arxiv.org/html/2606.04280#bib.bib1)\), video\(Zhaoet al\.,[2024](https://arxiv.org/html/2606.04280#bib.bib21)\), and multimodal\(Radfordet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib7)\)domains\.

The vast range of applications in scientific fields like Biology\(Richteret al\.,[2025](https://arxiv.org/html/2606.04280#bib.bib18); Bahramiet al\.,[2025](https://arxiv.org/html/2606.04280#bib.bib42)\), Physics\(Cyet al\.,[2023](https://arxiv.org/html/2606.04280#bib.bib43); Wilkinsonet al\.,[2025](https://arxiv.org/html/2606.04280#bib.bib44)\), and Climate Science\(Ballard,[2022](https://arxiv.org/html/2606.04280#bib.bib45); Liuet al\.,[2026](https://arxiv.org/html/2606.04280#bib.bib46)\)has made CL one of the most widely adopted unsupervised learning methods\(Uelweret al\.,[2023](https://arxiv.org/html/2606.04280#bib.bib22)\)\. However, despite its empirical success, the precise mechanisms driving contrastive learning remain only partially understood\. This gap in theoretical understanding results in heuristic\-driven development, inefficient use of computational resources, and design choices that may not fully exploit the method’s potential\. In this work, we seek to move beyond intuitive understanding of CL and provide a rigorous framework to reason about its regimes of success and failure\.

One approach to explain the learning mechanisms posits that CL induces data representations invariant to nuisance factors\(Dangovskiet al\.,[2022](https://arxiv.org/html/2606.04280#bib.bib2); Liuet al\.,[2025](https://arxiv.org/html/2606.04280#bib.bib3); Poudelet al\.,[2022](https://arxiv.org/html/2606.04280#bib.bib4)\)\. However, this framework does not address which factors in the data are nuisance, nor how the choice of data augmentations implicitly determine this partition\. Moreover, the choice of nuisance factors, often referred to as the style\-content decomposition\(vonkügelgen2022selfsupervisedlearningdataaugmentations\), can be detrimental to downstream tasks: depending on the intended use of the learned representations, factors deemed “nuisance” by the contrastive objective may carry discriminative information necessary for a specific downstream task\. We refer to this approach as theInvariance Explanation\.

![Refer to caption](https://arxiv.org/html/2606.04280v1/images/concept_figure.png)Figure 1:Overview of contrastive learning and the role of sampling diversity and inductive bias\. The generative processggmaps latent variables to observations, and the encoderfflearns to recover the latent structure\. Heref1f\_\{1\}denotes a low inductive bias encoder \(e\.g\., MLP\) andf2f\_\{2\}a high inductive bias encoder \(e\.g\., a model of the inverse process\)\. Orange dot indicates the anchor point; green dots are co\-occurring \(positive\) samples\. Border colors on images match their latent positions\. \(a\) Diversity holds:f1f\_\{1\}recovers geometry\. \(b\) Diversity violated \(blue band\):f1f\_\{1\}fails\. \(c\) Diversity violated:f2f\_\{2\}recovers the latent structure despite restricted sampling diversity\.An alternative direction reasons that CL recovers the “true” generating factors of the data\(Jiet al\.,[2023](https://arxiv.org/html/2606.04280#bib.bib9); Kirchhofet al\.,[2023](https://arxiv.org/html/2606.04280#bib.bib11); Sandilyaet al\.,[2025](https://arxiv.org/html/2606.04280#bib.bib12)\)\. This approach assumes that data lies on a high\-dimensional manifold but possesses a low\-dimensional latent structure, and that a generative process maps this low\-dimensional representation to the observed high\-dimensional data\. In this direction, CL recovers the latent structure\. Figure[1](https://arxiv.org/html/2606.04280#S1.F1)illustrates this setup\. We refer to this approach as theRecovery Explanation\.

In this work, we argue that the Recovery Explanation provides a more epistemologically complete account of the mechanisms underlying contrastive learning\. We introduce a constraint on the conditional law of the latent space,PZ~\|zP\_\{\\tilde\{Z\}\|z\}, which we term thediversity condition, and show that it is necessary for recovering the latent space up to an isometric \(distance\-preserving\) transformation\. Building on the notion of latent space recovery introduced by\(Zimmermannet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib13)\), we separate the classical full\-support vMF setting from the practically relevant setting where sampling is restricted\. Our proof of the full\-support case uses a probabilistic Mazur\-Ulam argument\(Zaliaduonis and Gatidis,[2026](https://arxiv.org/html/2606.04280#bib.bib40)\)and does not require differentiability of the recovery map\. Our main result shows that violated diversity can make geometry\-distorting solutions preferable, and that correcting the support mismatch restores, but does not uniquely select, geometry\-preserving minimizers\.

The real\-world sampling mechanisms often violate the diversity condition and the latent structure is typically unknown a priori\. We propose a generalized InfoNCE objective to address this gap and prove that this adjustment allows geometry\-preserving latent spaces to be among the optimal solutions of the objective\.

We empirically study the qualitative implications of the theory on CIFAR\-10\(Krizhevsky,[2009](https://arxiv.org/html/2606.04280#bib.bib14)\)\. Here, we examine how architectural inductive bias affects representation quality under different augmentation regimes\. In summary, our contributions are:

- •Formalize a measure\-theoretic diversity condition on latent space sampling, distinguish it from the full\-support vMF assumption used in prior identifiability results, and analyze what fails when this condition is violated\.
- •Prove that violated diversity can make non\-orthogonal recovery maps attain strictly lower asymptotic contrastive loss than any orthogonal map, and show that a support\-corrected InfoNCE objective makes isometric embeddings achievable\.
- •Empirically validate theoretical predictions on synthetic datasets and examine their qualitative implications on CIFAR\-10, demonstrating how sampling strategies and encoder inductive bias jointly determine representation quality\.
- •Distill design guidance for choosing augmentation pipelines and encoder architectures from the theory\-experiment alignment\.

## 2Related Work

#### Identifiability in Contrastive Learning\.

The question of identifiability in unsupervised learning has a long history, beginning with classical results in Independent Component Analysis \(ICA\)\. Early work established that linear mixtures of non\-Gaussian independent sources can be identified up to permutation and scaling, laying the theoretical foundation for blind source separation\(Comon,[1994](https://arxiv.org/html/2606.04280#bib.bib23)\)\. Subsequent developments provided practical algorithms and characterized the fundamental limits of linear ICA\(Hyvärinen and Oja,[2000](https://arxiv.org/html/2606.04280#bib.bib24)\)\. The broader goal of learning disentangled representations that capture meaningful factors of variation was later articulated\(Bengioet al\.,[2013](https://arxiv.org/html/2606.04280#bib.bib25)\), though it was subsequently proved that unsupervised disentanglement is impossible without inductive biases on both the model and data\(Locatelloet al\.,[2019](https://arxiv.org/html/2606.04280#bib.bib26)\)\.

In the context of contrastive learning, a formal definition of latent space identifiability was introduced, with proofs that InfoNCE recovers ground\-truth factors up to orthogonal transformation\(Zimmermannet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib13)\)\. This analysis employs von Mises\-Fisher conditionals and cross\-entropy asymptotics on spherical manifolds\. Connections between contrastive objectives and nonlinear ICA have also been established, demonstrating identifiability through temporal structure\(Hyvärinen and Morioka,[2016](https://arxiv.org/html/2606.04280#bib.bib27)\)\.

#### Augmentations and Invariance\.

A complementary line of work examines how data augmentations shape learned representations by defining which factors should be preserved versus discarded\. A content\-style framework formalizes this intuition, proving that augmentation\-based contrastive learning achieves block\-identifiability of the invariant content partition under a latent variable model with nontrivial statistical and causal dependencies\(vonkügelgen2022selfsupervisedlearningdataaugmentations\)\. The InfoMin principle proposes that optimal views for contrastive learning should share minimal mutual information while retaining task\-relevant information, thereby discarding nuisance factors\(Tianet al\.,[2020](https://arxiv.org/html/2606.04280#bib.bib30)\)\. A causal interpretation shows that data augmentations can be viewed as interventions on style variables, motivating an explicit invariance regularizer that enforces invariant prediction of proxy targets across augmentations\(Mitrovicet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib31)\)\.Wen and Li \([2021](https://arxiv.org/html/2606.04280#bib.bib10)\)study a complementary feature\-learning regime: they analyze gradient\-based learning in finite ReLU networks and show how suitable augmentations can decouple desired sparse features from nuisance dense features\. Our analysis asks a different question, namely whether the asymptotic contrastive objective recovers latent geometry up to isometry, and how architectural inductive bias compensates when sampling diversity is insufficient\.

#### Architectural Bias and Extensions\.

The role of encoder architecture in contrastive learning has received increasing attention\. Recent work analyzes how architectural constraints influence the geometry of learned representations\(HaoChen and Ma,[2023](https://arxiv.org/html/2606.04280#bib.bib32)\), while theoretical frameworks have been extended to multimodal settings\(Tschannenet al\.,[2023](https://arxiv.org/html/2606.04280#bib.bib33)\)\. Other contributions address gaps between theoretical assumptions and practical implementations\(Rusaket al\.,[2025](https://arxiv.org/html/2606.04280#bib.bib35)\)\.

#### Empirical Methods\.

Our theoretical analysis builds upon empirically successful methods including SimCLR\(Chenet al\.,[2020](https://arxiv.org/html/2606.04280#bib.bib1)\), Contrastive Predictive Coding \(CPC\)\(van den Oordet al\.,[2019](https://arxiv.org/html/2606.04280#bib.bib17)\), and VICReg\(Bardeset al\.,[2022](https://arxiv.org/html/2606.04280#bib.bib8)\)\. These frameworks demonstrate the practical efficacy of contrastive objectives across vision, language, and multimodal domains\.

## 3Preliminaries

### 3\.1Contrastive Learning Framework

Following\(Zimmermannet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib13)\), we analyze contrastive learning using tools from Nonlinear Independent Component Analysis \(ICA\)\(Hyvärinenet al\.,[2019](https://arxiv.org/html/2606.04280#bib.bib15)\)\. We consider an encoderf:𝒳→𝒵f:\\mathcal\{X\}\\to\\mathcal\{Z\}mapping observations from the data space𝒳⊂ℝn\\mathcal\{X\}\\subset\\mathbb\{R\}^\{n\}to a latent representation space𝒵⊂ℝk\\mathcal\{Z\}\\subset\\mathbb\{R\}^\{k\}, wherek<nk<n\. We model the data as generated by an injective mappingg:𝒵→𝒳g:\\mathcal\{Z\}\\to\\mathcal\{X\}from a lower\-dimensional latent manifold whose coordinates represent statistically independent factors \(Figure[1](https://arxiv.org/html/2606.04280#S1.F1)\)\.

Contrastive learning aims to learn encoder parametersθ\\thetaby discriminating between co\-occurring signalsx,x~∈𝒳x,\\tilde\{x\}\\in\\mathcal\{X\}\(Chenet al\.,[2020](https://arxiv.org/html/2606.04280#bib.bib1)\)\. These signals may arise from natural mechanisms \(e\.g\., different modalities of the same scene\) or synthetic transformations \(e\.g\., image augmentations\)\. We define a view\-generating functionτ:𝒳→𝒳\\tau:\\mathcal\{X\}\\to\\mathcal\{X\}that produces related viewsx~=τ​\(x\)\\tilde\{x\}=\\tau\(x\), and aim to learn an encoder such that the underlying latent factors are identified\.

### 3\.2Sampling Mechanism

Since the data is generated via the injective mappinggg, observations satisfyx=g​\(z\)x=g\(z\)andx~=τ​\(g​\(z\)\)=g​\(z~\)\\tilde\{x\}=\\tau\(g\(z\)\)=g\(\\tilde\{z\}\)\. As view generation is typically non\-deterministic, we treat it stochastically through the conditional lawPX~\|xP\_\{\\tilde\{X\}\|x\}\. This formulation connects to the latent dynamicsPZ~\|zP\_\{\\tilde\{Z\}\|z\}via the pushforward measure:

PX~\|x=g∗​PZ~\|zP\_\{\\tilde\{X\}\|x\}=g\_\{\*\}P\_\{\\tilde\{Z\}\|z\}\(1\)whereg∗g\_\{\*\}is the pushforward operator\. Our analysis focuses on howPZ~\|zP\_\{\\tilde\{Z\}\|z\}affects the recovery maph:=f∘g:𝒵→𝒵h:=f\\circ g:\\mathcal\{Z\}\\to\\mathcal\{Z\}\. The sampling mechanism need not be an image augmentation; it may also arise from temporal proximity, spatial crops, multimodal co\-occurrence, or any other rule for drawing related observations\. Independent views mean conditionally independent draws fromPZ~\|zP\_\{\\tilde\{Z\}\|z\}for a fixed anchorzz, not semantic independence of the resulting observations\. Notation: uppercaseP,QP,Qdenotes probability laws, while lowercasep,qp,qdenotes probability measure densities with respect to the stated reference measure\.

### 3\.3Diversity Condition

We formalize the diversity condition and argue for its necessity in distance\-preserving latent space reconstruction\. The condition is motivated by comparing the theoretical ideal of full\-support positive\-pair sampling with practical sampling mechanisms\. In the ideal vMF setting used by\(Zimmermannet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib13)\), positive pairs can in principle cover the entire latent space around an anchor, which provides enough information to recover pairwise geometry\. Practical mechanisms often restrict this support by keeping some latent coordinates fixed while changing others\. The diversity condition identifies the weakest support requirement needed for recovery: every latent region that has nonzero marginal probability must also be reachable by the conditional positive\-pair distribution\.

###### Definition 3\.1\(Diversity Condition\)\.

For a latent measurable space𝒵\\mathcal\{Z\}with marginal probability measurePZP\_\{Z\}and conditional probability measurePZ~\|zP\_\{\\tilde\{Z\}\|z\}, the diversity condition holds if, forPZP\_\{Z\}\-almost everyz∈𝒵z\\in\\mathcal\{Z\},

PZ≪PZ~\|z\.P\_\{Z\}\\ll P\_\{\\tilde\{Z\}\|z\}\.\(2\)Equivalently, for every measurable setA⊆𝒵A\\subseteq\\mathcal\{Z\},PZ~\|z​\(A\)=0P\_\{\\tilde\{Z\}\|z\}\(A\)=0impliesPZ​\(A\)=0P\_\{Z\}\(A\)=0forPZP\_\{Z\}\-almost every anchorzz\.

Intuitively, the diversity condition requires thatPZ~\|zP\_\{\\tilde\{Z\}\|z\}has sufficiently large support to cover any region wherePZP\_\{Z\}assigns nonzero probability\. This implies that the view\-generating process must perturb all latent features\. If some generative features remain constant, the encoder cannot distinguish them along fixed dimensions, failing to invertggfor those components\.

Finally, we relate the condition to practical sampling mechanisms\. The diversity condition is stated at the level of the induced latent sampling mechanismPZ~\|zP\_\{\\tilde\{Z\}\|z\}, rather than in terms of application\-specific categories of transformations\. Descriptions such as appearance, structure, or semantic content are therefore only informal indicators of which latent directions a sampling mechanism may vary\. In practice, an augmentation or view\-generation rule is useful for recovery only to the extent that it gives the conditional distribution support along the latent factors present underPZP\_\{Z\}\. Mechanisms that vary only a restricted subset of these factors may violate the condition, even if they produce visually distinct observations\.

### 3\.4InfoNCE as Cross\-Entropy Minimization

Information Noise Contrastive Estimation \(InfoNCE\)\(van den Oordet al\.,[2019](https://arxiv.org/html/2606.04280#bib.bib17)\)is the standard contrastive objective used to train representations from one positive view and a set of negative samples\. For each anchor, it increases similarity to the positive sample while decreasing similarity to negatives, making it a probabilistic discrimination loss over co\-occurring and non\-co\-occurring samples\. The InfoNCE objective quantifies encoder performance on this discrimination task\.

###### Definition 3\.2\(InfoNCE Loss\(van den Oordet al\.,[2019](https://arxiv.org/html/2606.04280#bib.bib17)\)\)\.

Given a recovery maph:𝒵→𝒵h:\\mathcal\{Z\}\\to\\mathcal\{Z\}, positive pairs\(z,z~\)∼Ppos\(z,\\tilde\{z\}\)\\sim P\_\{\\mathrm\{pos\}\}, and negative samples\{zi−\}i=1M​∼i\.i\.d\.​P−\\\{z\_\{i\}^\{\-\}\\\}\_\{i=1\}^\{M\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}P^\{\-\}:

ℒCL​\(h;τ,M\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{CL\}\}\(h;\\tau,M\)=𝔼\(z,z~\)∼Ppos\{zi−\}i=1M​∼i\.i\.d\.​P−\[−log⁡eh​\(z\)⊤​h​\(z~\)/τD​\(z,z~\)\]\\displaystyle=\\mathop\{\\mathbb\{E\}\}\_\{\\begin\{subarray\}\{c\}\(z,\\tilde\{z\}\)\\sim P\_\{\\mathrm\{pos\}\}\\\\ \\\{z\_\{i\}^\{\-\}\\\}\_\{i=1\}^\{M\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}P^\{\-\}\\end\{subarray\}\}\\left\[\-\\log\\frac\{e^\{h\(z\)^\{\\top\}h\(\\tilde\{z\}\)/\\tau\}\}\{D\(z,\\tilde\{z\}\)\}\\right\]D​\(z,z~\)\\displaystyle D\(z,\\tilde\{z\}\)=eh​\(z\)⊤​h​\(z~\)/τ\+∑i=1Meh​\(z\)⊤​h​\(zi−\)/τ\\displaystyle=e^\{h\(z\)^\{\\top\}h\(\\tilde\{z\}\)/\\tau\}\+\\sum\_\{i=1\}^\{M\}e^\{h\(z\)^\{\\top\}h\(z\_\{i\}^\{\-\}\)/\\tau\}whereτ\>0\\tau\>0is a temperature parameter andMMis the number of negative samples\.

Following\(Zimmermannet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib13)\), we take the latent space to be a hypersphere𝕊k−1\\mathbb\{S\}^\{k\-1\}, motivated by the practical convention ofL2L^\{2\}\-normalizing contrastive representations\(Chenet al\.,[2020](https://arxiv.org/html/2606.04280#bib.bib1); Haaset al\.,[2024](https://arxiv.org/html/2606.04280#bib.bib36)\)\. We assume the conditional follows a von Mises\-Fisher \(vMF\) distribution with concentrationκ\>0\\kappa\>0, where sampling frequency is inversely proportional to latent distance\.

###### Theorem 3\.3\(Asymptotics ofℒCL\\mathcal\{L\}\_\{\\text\{CL\}\}\(Zimmermannet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib13)\)\)\.

Given a spherical latent space𝒵=𝕊k−1\\mathcal\{Z\}=\\mathbb\{S\}^\{k\-1\}, a uniform marginal lawPZ=U​\(𝒵\)P\_\{Z\}=U\(\\mathcal\{Z\}\), a von Mises\-Fisher conditional measurePZ~\|zP\_\{\\tilde\{Z\}\|z\}with density

p​\(z~\|z\)=eκ​z~⊤​z∫𝒵eκ​z′⁣⊤​z​𝑑σ​\(z′\)p\(\\tilde\{z\}\|z\)=\\frac\{e^\{\\kappa\\tilde\{z\}^\{\\top\}z\}\}\{\\int\_\{\\mathcal\{Z\}\}e^\{\\kappa z^\{\\prime\\top\}z\}\\,d\\sigma\(z^\{\\prime\}\)\}\(3\)whereσ\\sigmadenotes spherical surface measure on𝒵\\mathcal\{Z\}andκ\>0\\kappa\>0is the concentration parameter,

and a model conditional measureQh,zQ\_\{h,z\}with density

qh​\(z~\|z\)=eh​\(z~\)⊤​h​\(z\)/τ∫𝒵eh​\(z′\)⊤​h​\(z\)/τ​𝑑σ​\(z′\)q\_\{h\}\(\\tilde\{z\}\|z\)=\\frac\{e^\{h\(\\tilde\{z\}\)^\{\\top\}h\(z\)/\\tau\}\}\{\\int\_\{\\mathcal\{Z\}\}e^\{h\(z^\{\\prime\}\)^\{\\top\}h\(z\)/\\tau\}\\,d\\sigma\(z^\{\\prime\}\)\}\(4\)
For fixedτ\>0\\tau\>0, as the number of negative samplesM→∞M\\to\\infty, the \(normalized\) contrastive loss converges to

limM→∞ℒCL​\(h;τ,M\)−log⁡M\+log⁡\|𝒵\|=𝔼z∼PZ​\[H​\(PZ~\|z,Qh,z\)\]\\lim\_\{M\\to\\infty\}\\mathcal\{L\}\_\{\\text\{CL\}\}\(h;\\tau,M\)\-\\log M\+\\log\|\\mathcal\{Z\}\|=\\\\ \\mathbb\{E\}\_\{z\\sim P\_\{Z\}\}\[H\(P\_\{\\tilde\{Z\}\|z\},Q\_\{h,z\}\)\]\(5\)
whereH​\(PZ~\|z,Qh,z\)H\(P\_\{\\tilde\{Z\}\|z\},Q\_\{h,z\}\)denotes cross\-entropy, using densities with respect toσ\\sigmain the vMF setting\.

This interpretation allows us to analyze contrastive learning through the lens of distribution matching between the sampling mechanismPZ~\|zP\_\{\\tilde\{Z\}\|z\}and the model measureQh,zQ\_\{h,z\}\.

### 3\.5Inductive Bias

Inductive bias refers to the structural assumptions that constrain a learning algorithm’s hypothesis space, enabling generalization\(Vapnik,[1999](https://arxiv.org/html/2606.04280#bib.bib39)\)\. In contrastive learning, these assumptions arise through model architecture \(e\.g\., translation equivariance in CNNs\(LeCunet al\.,[1998](https://arxiv.org/html/2606.04280#bib.bib37)\), attention in Transformers\(Dosovitskiyet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib38)\)\) and geometric constraints on the embedding space\. In our framework, an encoder is geometry\-preserving when the recovery maph=f∘gh=f\\circ grecovers the latent structure up to an orthogonal transformation, i\.e\.,h​\(z\)≈A​zh\(z\)\\approx Azfor someA∈O​\(k\)A\\in O\(k\)\. Equivalently, the encoder approximatesf≈A∘g−1f\\approx A\\circ g^\{\-1\}on the data manifold\. An effective inductive bias therefore makes approximate inverses of the data\-generating process easier to represent and optimize, while restricting arbitrary non\-geometry\-preserving maps that can also satisfy the contrastive loss\.

When the diversity condition is violated, the contrastive objective alone cannot uniquely determine the latent geometry\. Inductive bias then acts as a compensatory mechanism, restricting admissible solutions to those consistent with architectural priors\. In Section[4\.3](https://arxiv.org/html/2606.04280#S4.SS3), we show that such biases are necessary for linearly identifiable reconstruction when diversity is violated\. Crucially, we demonstrate experimentally \(Section[5](https://arxiv.org/html/2606.04280#S5)\) that this necessity persists even asymptotically: the contrastive objective fails to recover latent geometry regardless of data quantity unless structural constraints are imposed\.

## 4Theoretical Results

We present the core theoretical results under the common assumption ofL2L^\{2\}\-normalized representations\(Grillet al\.,[2020](https://arxiv.org/html/2606.04280#bib.bib19); Haaset al\.,[2023](https://arxiv.org/html/2606.04280#bib.bib20)\)\. Unless stated otherwise, the results use the following standing assumptions:

- \(A1\)𝒵=𝕊k−1\\mathcal\{Z\}=\\mathbb\{S\}^\{k\-1\}is a unit hypersphere;
- \(A2\)the marginalPZP\_\{Z\}is uniform on𝒵\\mathcal\{Z\};
- \(A3\)the conditionalPZ~\|zP\_\{\\tilde\{Z\}\|z\}is vMF with concentrationκ\>0\\kappa\>0before diversity violation is introduced;
- \(A4\)the encoder has sufficient capacity to realize any measurable recovery map considered below;
- \(A5\)representations areL2L^\{2\}\-normalized, soh:𝒵→𝒵h:\\mathcal\{Z\}\\to\\mathcal\{Z\}\.

The full\-support result below is closest to prior hypersphere identifiability results, but the proof uses a probabilistic Mazur\-Ulam theorem and requires no differentiability ofhh\. The subsequent results are the main extension: they characterize the asymptotic global optima when sampling diversity is violated\. We introduce the following definitions to formalize our analysis\.

###### Definition 4\.1\(Isometry Almost Everywhere\)\.

Let\(𝒵,δ\)\(\\mathcal\{Z\},\\delta\)be a metric space with measureμ\\mu\. A measurable mappingh:𝒵→𝒵h:\\mathcal\{Z\}\\to\\mathcal\{Z\}is anisometry almost everywhereif there exists an isometrye:𝒵→𝒵e:\\mathcal\{Z\}\\to\\mathcal\{Z\}such thath​\(z\)=e​\(z\)h\(z\)=e\(z\)forμ\\mu\-almost allz∈𝒵z\\in\\mathcal\{Z\}\.

###### Definition 4\.2\(Equivalent Recovery Maps\)\.

Two mappingsh1,h2:𝒵→𝒵h\_\{1\},h\_\{2\}:\\mathcal\{Z\}\\to\\mathcal\{Z\}are equivalent with respect to the contrastive loss ifℒCL​\(h1;τ,M\)=ℒCL​\(h2;τ,M\)\\mathcal\{L\}\_\{\\text\{CL\}\}\(h\_\{1\};\\tau,M\)=\\mathcal\{L\}\_\{\\text\{CL\}\}\(h\_\{2\};\\tau,M\)for fixedτ\\tauandMM\.

### 4\.1Reconstruction Under Full\-Support Sampling

###### Lemma 4\.3\(Full\-Support vMF Implies Diversity\)\.

LetPZP\_\{Z\}be uniform on𝒵=𝕊k−1\\mathcal\{Z\}=\\mathbb\{S\}^\{k\-1\}, and letPZ~\|zP\_\{\\tilde\{Z\}\|z\}have vMF density proportional toexp⁡\(κ​z⊤​z~\)\\exp\(\\kappa z^\{\\top\}\\tilde\{z\}\)withκ\>0\\kappa\>0with respect to spherical surface measure\. Then the diversity condition holds\.

###### Proof\.

The vMF density is strictly positive on the whole sphere\. Therefore, any measurable set with zeroPZ~\|zP\_\{\\tilde\{Z\}\|z\}measure also has zero spherical surface measure, and hence zero uniform marginal measure\. ∎

Thus in the vMF setting the diversity condition is a consequence of full support, not an additional independent hypothesis\. Contrastive learning then recovers the latent space up to orthogonal transformation\.

###### Theorem 4\.4\(Linear Identifiability Under Full Diversity\)\.

Under a uniform marginal on𝒵=𝕊k−1\\mathcal\{Z\}=\\mathbb\{S\}^\{k\-1\}and full\-support vMF conditional, any recovery maph:𝒵→𝒵h:\\mathcal\{Z\}\\to\\mathcal\{Z\}that globally minimizes the asymptotic contrastive objective is an isometry almost everywhere:

h​\(z\)=A​zfor​μ​\-almost all​z∈𝒵,h\(z\)=Az\\quad\\text\{for \}\\mu\\text\{\-almost all \}z\\in\\mathcal\{Z\},whereA∈O​\(k\)A\\in O\(k\)is an orthogonal matrix\.

###### Proof\.

See Appendix[A\.1](https://arxiv.org/html/2606.04280#A1.SS1)\. ∎

*Proof sketch\.*The asymptotic InfoNCE objective reduces to conditional cross\-entropy\. Under the full\-support vMF conditional, any minimizer must match conditionals and therefore preserve inner products almost surely\. On the sphere, this gives distance preservation, and the probabilistic Mazur\-Ulam theorem extends this almost\-sure isometry to a global orthogonal map\.

This represents an ideal scenario: when the sampling mechanism perturbs all latent factors, the learned representations preserve pairwise distances, yielding a separable latent space suitable for downstream tasks\. However, real\-world augmentation pipelines rarely have full support, motivating the analysis in the following section\.

### 4\.2Reconstruction Under Violated Diversity

In practice, the diversity condition is rarely satisfied\. Most augmentation pipelines preserve certain semantic features while perturbing others\. We model this by decomposing the latent vectorz=\(u,v\)z=\(u,v\)into an invariant componentu∈ℝmu\\in\\mathbb\{R\}^\{m\}and a varying componentv∈ℝℓv\\in\\mathbb\{R\}^\{\\ell\}, withm\+ℓ=km\+\\ell=kandm,ℓ\>0m,\\ell\>0\. Let

K​\(z\)\\displaystyle K\(z\):=\{z~=\(u~,v~\)∈𝕊k−1:u~=u\},\\displaystyle=\\\{\\tilde\{z\}=\(\\tilde\{u\},\\tilde\{v\}\)\\in\\mathbb\{S\}^\{k\-1\}:\\tilde\{u\}=u\\\},K​\(z\)\\displaystyle K\(z\)≅𝕊r​\(z\)ℓ−1,r​\(z\):=1−‖u‖2\.\\displaystyle\\cong\\mathbb\{S\}^\{\\ell\-1\}\_\{r\(z\)\},\\qquad r\(z\)=\\sqrt\{1\-\\\|u\\\|^\{2\}\}\.Forr​\(z\)\>0r\(z\)\>0, letσK​\(z\)\\sigma\_\{K\(z\)\}denote the intrinsic surface measure onK​\(z\)K\(z\)\. The constrained positive\-pair conditional is the probability measurePZ~\|zKP^\{K\}\_\{\\tilde\{Z\}\|z\}supported onK​\(z\)K\(z\)with Radon\-Nikodym density

d​PZ~\|zKd​σK​\(z\)​\(z~\)=eκ​z⊤​z~∫K​\(z\)eκ​z⊤​z′​𝑑σK​\(z\)​\(z′\)\.\\frac\{dP^\{K\}\_\{\\tilde\{Z\}\|z\}\}\{d\\sigma\_\{K\(z\)\}\}\(\\tilde\{z\}\)=\\frac\{e^\{\\kappa z^\{\\top\}\\tilde\{z\}\}\}\{\\int\_\{K\(z\)\}e^\{\\kappa z^\{\\top\}z^\{\\prime\}\}\\,d\\sigma\_\{K\(z\)\}\(z^\{\\prime\}\)\}\.\(6\)This measure is singular with respect to ambient spherical measure on𝕊k−1\\mathbb\{S\}^\{k\-1\}, but it is a well\-defined probability law on the lower\-dimensional submanifoldK​\(z\)K\(z\)\.

This collapses the sampling support from𝕊k−1\\mathbb\{S\}^\{k\-1\}onto a lower\-dimensional submanifoldK​\(z\)K\(z\), mirroring the content\-style framework of\(vonkügelgen2022selfsupervisedlearningdataaugmentations\)\.

###### Theorem 4\.5\(Loss of Identifiability Under Violated Diversity\)\.

For the asymptotic contrastive objective obtained asM→∞M\\to\\inftyunder the constrained conditional \(Equation[6](https://arxiv.org/html/2606.04280#S4.E6)\) and standard full\-sphere negatives, orthogonal recovery is not globally optimal\. There exists a recovery maph:𝒵→𝒵h:\\mathcal\{Z\}\\to\\mathcal\{Z\}that is not induced by any orthogonal transformation such that

ℒCL​\(h\)<ℒCL​\(h~\),∀h~∈O​\(k\)\.\\mathcal\{L\}\_\{\\mathrm\{CL\}\}\(h\)<\\mathcal\{L\}\_\{\\mathrm\{CL\}\}\(\\tilde\{h\}\),\\qquad\\forall\\,\\tilde\{h\}\\in O\(k\)\.

This is a statement about global values of the limiting objective, not about finite\-sample optimization dynamics\.

###### Proof\.

See Appendix[A\.2](https://arxiv.org/html/2606.04280#A1.SS2)\. ∎

*Proof sketch\.*The constrained conditional fixesuuand only variesvv, while the standard model conditional still normalizes over the full sphere\. We construct an explicit non\-orthogonal maphλ​\(u,v\)=\(u,λ​v\)/‖\(u,λ​v\)‖h\_\{\\lambda\}\(u,v\)=\(u,\\lambda v\)/\\\|\(u,\\lambda v\)\\\|withλ<1\\lambda<1close to one\. Because positive pairs shareuu, this map improves the alignment term to first order, while the uniformity term is stationary at the identity, yielding lower asymptotic loss than any orthogonal map\.

The consequence is concrete: arbitrarily expressive encoders are actively disincentivized from learning orthogonal recovery maps when the diversity condition is violated\. The contrastive objective rewards geometry\-distorting solutions, leading to poorly structured latent spaces where semantic relationships are not preserved\. Consequently, downstream tasks that rely on meaningful distance relationships in the representation space face a fundamental bottleneck that cannot be overcome by increasing encoder capacity alone\.

### 4\.3Correcting the Model

The support mismatch betweenPZ~\|zKP^\{K\}\_\{\\tilde\{Z\}\|z\}and the standard full\-sphere model conditional prevents the InfoNCE objective from favoring isometric solutions\. We address this by constraining the model conditional to the same submanifoldK​\(z\)K\(z\):

qh,zK​\(z~\)=eh​\(z\)⊤​h​\(z~\)/τ∫K​\(z\)eh​\(z\)⊤​h​\(z′\)/τ​𝑑σK​\(z\)​\(z′\),z~∈K​\(z\)\.q^\{K\}\_\{h,z\}\(\\tilde\{z\}\)=\\frac\{e^\{h\(z\)^\{\\top\}h\(\\tilde\{z\}\)/\\tau\}\}\{\\int\_\{K\(z\)\}e^\{h\(z\)^\{\\top\}h\(z^\{\\prime\}\)/\\tau\}\\,d\\sigma\_\{K\(z\)\}\(z^\{\\prime\}\)\},\\qquad\\tilde\{z\}\\in K\(z\)\.\(7\)
Equivalently, for a positive pair\(z,z~\)\(z,\\tilde\{z\}\)and negativesz1−,…,zM−​∼i\.i\.d\.​U​\(K​\(z\)\)z\_\{1\}^\{\-\},\\ldots,z\_\{M\}^\{\-\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}U\(K\(z\)\), the per\-anchor adapted loss is

ℓadapt\\displaystyle\\ell\_\{\\mathrm\{adapt\}\}=−log⁡exp⁡\(h​\(z\)⊤​h​\(z~\)/τ\)Da​\(z,z~\),\\displaystyle=\-\\log\\frac\{\\exp\(h\(z\)^\{\\top\}h\(\\tilde\{z\}\)/\\tau\)\}\{D\_\{a\}\(z,\\tilde\{z\}\)\},\(8\)Da​\(z,z~\)\\displaystyle D\_\{a\}\(z,\\tilde\{z\}\)=exp⁡\(h​\(z\)⊤​h​\(z~\)/τ\)\\displaystyle=\\exp\(h\(z\)^\{\\top\}h\(\\tilde\{z\}\)/\\tau\)\+∑j=1Mexp⁡\(h​\(z\)⊤​h​\(zj−\)/τ\)\.\\displaystyle\\quad\+\\sum\_\{j=1\}^\{M\}\\exp\(h\(z\)^\{\\top\}h\(z\_\{j\}^\{\-\}\)/\\tau\)\.This corresponds to drawing negative samples fromK​\(z\)K\(z\)rather than the full latent space\. Exact sampling requires access to the invariant componentuu, which is generally unavailable in real data\. In practice, same\-anchor augmentations provide a proxy: independent transformations of the same anchor preserve the components fixed by the augmentation mechanism while varying the remaining components\. Appendix[C](https://arxiv.org/html/2606.04280#A3)gives the resulting training step, but the adapted loss should be read primarily as a theoretical diagnostic rather than a fully practical replacement for standard InfoNCE\.

###### Theorem 4\.6\(Orthogonal Mappings as Minimizers\)\.

Under the corrected model \(Equation[7](https://arxiv.org/html/2606.04280#S4.E7)\), any orthogonal transformationh∈O​\(k\)h\\in O\(k\)minimizes the asymptotic contrastive loss\.

###### Proof\.

See Appendix[A\.3](https://arxiv.org/html/2606.04280#A1.SS3)\. ∎

*Proof sketch\.*Once the model conditional is restricted to the same supportK​\(z\)K\(z\)as the true conditional, any orthogonal map preserves the inner products that define the vMF density on that support\. Thus the model conditional matches the true conditional whenκ=1/τ\\kappa=1/\\tau, minimizing cross\-entropy\. This proves achievability of orthogonal solutions, but not uniqueness\.

Although the corrected objective admits orthogonal solutions, it does not guarantee that all minimizers are orthogonal\. Thus, the adapted objective should not be interpreted as a fully general solution: it removes the support mismatch and makes geometry\-preserving recovery achievable, but it does not make such recovery unique\. The sampling mechanism determines which solutions are achievable, while inductive bias influences which solution is selected during optimization\. This non\-uniqueness highlights the necessity of inductive bias for selecting geometry\-preserving solutions\.

## 5Experimental Results

### 5\.1Synthetic Dataset Experiments

We validate the theoretical predictions from Section[4](https://arxiv.org/html/2606.04280#S4)using a controlled synthetic setup with a spherical latent space𝒵=𝕊2\\mathcal\{Z\}=\\mathbb\{S\}^\{2\}\.111Code is available at[https://github\.com/BosonicJustin/CLTheory](https://github.com/BosonicJustin/CLTheory)\.Following the assumptions of our theoretical framework, the marginal law is uniform over the sphere, and positive pairs are sampled according to a von Mises\-Fisher \(vMF\) conditional lawPZ~\|zP\_\{\\tilde\{Z\}\|z\}with densityp​\(z~\|z\)=vMF​\(z,κ\)p\(\\tilde\{z\}\|z\)=\\text\{vMF\}\(z,\\kappa\)and concentration parameterκ=1/τ\\kappa=1/\\tau\(see Equation[17](https://arxiv.org/html/2606.04280#A1.E17)\), whereτ\\tauis the temperature in the InfoNCE loss\. The generative processes used in our experiments are illustrated in Figure[2](https://arxiv.org/html/2606.04280#S5.F2)\. Details of the sampling procedures are provided in Appendix[C](https://arxiv.org/html/2606.04280#A3)\.

![Refer to caption](https://arxiv.org/html/2606.04280v1/images/generative_processes_visualization.png)Figure 2:Generative processes mapping the unit sphere to observation space\. Colors encode input coordinates \(RGB = xyz\), illustrating how each transformation warps the latent space: \(a\) identity preserves the sphere, \(b\) linear maps to an ellipsoid, \(c\) spiral twists points around the vertical axis, \(d\) patches applies piecewise rotations creating discontinuities, and \(e\) invertible MLP produces smooth nonlinear deformations\.#### Experimental Setup\.

We evaluate five generative processesg:𝒵→𝒳g:\\mathcal\{Z\}\\to\\mathcal\{X\}of varying complexity: Identity, injective Linear map \(𝕊2→ℝ7\\mathbb\{S\}^\{2\}\\to\\mathbb\{R\}^\{7\}\), Spiral rotation, Patches, and invertible MLP\(Hyvärinen and Morioka,[2016](https://arxiv.org/html/2606.04280#bib.bib27)\)\. Detailed definitions are provided in Appendix[B\.1](https://arxiv.org/html/2606.04280#A2.SS1)\. For each generative process, we compare two encoder architectures: an MLP encoder with hidden dimensions\[128,256,256,256,128\]\[128,256,256,256,128\]representing low inductive bias, and an inverse encoder designed to invert the corresponding generative process, representing high inductive bias\. The invertible MLP is an exception, as it lacks a strict analytic inverse; we therefore evaluate only the MLP encoder for this generative process\. All experiments use InfoNCE loss with temperatureτ=0\.3\\tau=0\.3, Adam optimizer\(Kingma and Ba,[2015](https://arxiv.org/html/2606.04280#bib.bib28)\)\(lr =10−310^\{\-3\}\), batch size 2000, and 5000 iterations\. The generative processggremains frozen throughout training, and each configuration is evaluated over 5 independent runs\.

#### Evaluation Metrics\.

We assess reconstruction quality using linear identifiability \(R2R^\{2\}\), which measures recovery up to affine transformation as in\(Hyvärinen and Morioka,[2016](https://arxiv.org/html/2606.04280#bib.bib27)\)\. Additional metrics \(Mean Correlation Coefficient and Angular Preservation Error\) are provided in Appendix[D](https://arxiv.org/html/2606.04280#A4)\.

#### Results\.

We summarize the experimental results in Table[1](https://arxiv.org/html/2606.04280#S5.T1)and Figure[5](https://arxiv.org/html/2606.04280#A5.F5)\.

#### Diversity Condition Holds\.

When the diversity condition is satisfied, the MLP encoder achieves near\-perfect linear identifiability \(R2≥0\.99R^\{2\}\\geq 0\.99\) across all five generative processes, confirming the theoretical prediction of Theorem[4\.4](https://arxiv.org/html/2606.04280#S4.Thmtheorem4)\. This demonstrates that a sufficiently expressive encoder can recover a representation linearly identifiable with the ground\-truth latent space when the diversity condition holds, across varying degrees of injective generative processes\.

#### Diversity Condition Violated\.

When the diversity condition is violated by fixing the first latent dimension during positive pair sampling, MLP performance collapses dramatically\. Linear identifiability drops toR2∈\[0\.05,0\.13\]R^\{2\}\\in\[0\.05,0\.13\]for Identity, MLP, and Linear generative processes, andR2=0\.25R^\{2\}=0\.25for Patches\. The Spiral process is an exception \(R2=0\.72R^\{2\}=0\.72\), maintaining poor but not catastrophic performance with high variance across runs\. This validates Theorem[4\.5](https://arxiv.org/html/2606.04280#S4.Thmtheorem5): the contrastive objective alone no longer incentivizes geometry\-preserving solutions when the conditional sampling support is restricted\.

#### Inductive Bias Compensation\.

Under violated diversity, incorporating inductive bias through inverse encoders restores near\-perfect recovery \(R2≥0\.88R^\{2\}\\geq 0\.88\), with Identity, Linear, and Spiral processes achievingR2≥0\.99R^\{2\}\\geq 0\.99\. The inverse encoders, designed to mirror the structure of each generative process, successfully recover the latent geometry even when the sampling mechanism provides insufficient information\. This demonstrates that architectural constraints can compensate for deficiencies in the sampling regime, highlighting the complementary roles of data augmentation and model design in contrastive learning\.

#### Adapted InfoNCE Loss\.

The adapted InfoNCE loss partially recovers MLP performance under violated diversity \(R2≈0\.60​–​0\.65R^\{2\}\\approx 0\.60\\text\{\-\-\}0\.65\), representing a substantial improvement over standard InfoNCE \(R2≈0\.05​–​0\.25R^\{2\}\\approx 0\.05\\text\{\-\-\}0\.25for most processes\)\. However, this falls short of the performance achieved with appropriate inductive bias \(R2≥0\.88R^\{2\}\\geq 0\.88\)\. This confirms Theorem[4\.6](https://arxiv.org/html/2606.04280#S4.Thmtheorem6): correcting the model conditional makes isometric solutions achievable but does not guarantee them\. From a practical standpoint, these results suggest that while loss modifications can mitigate the effects of violated diversity, incorporating architectural priors remains the more effective strategy, aligning with the empirical success of high inductive bias architectures such as CNNs and Vision Transformers in contrastive learning pipelines\.

Table 1:Linear identifiability \(R2R^\{2\}\) across generative processes under different experimental conditions\. Results reported as mean±\\pmstd across 5 random seeds\. The Invertible MLP process lacks a closed\-form inverse, so no inverse encoder is available \(indicated by N/A\)\.![[Uncaptioned image]](https://arxiv.org/html/2606.04280v1/images/sphere_examples/identity_sphere.png)![[Uncaptioned image]](https://arxiv.org/html/2606.04280v1/images/sphere_examples/mlp_linear_diversity_holds_sphere.png)![[Uncaptioned image]](https://arxiv.org/html/2606.04280v1/images/sphere_examples/mlp_linear_diversity_violated_sphere.png)![[Uncaptioned image]](https://arxiv.org/html/2606.04280v1/images/sphere_examples/mlp_linear_diversity_violated_info_nce_adjusted.png)![[Uncaptioned image]](https://arxiv.org/html/2606.04280v1/images/sphere_examples/linear_linear_diversity_violated_sphere.png)Diversity HoldsDiversity ViolatedGenerative ProcessInfoNCEInfoNCEInfoNCE AdaptedInfoNCE \+ Ind\. BiasIdentity1\.00±0\.001\.00\\pm 0\.000\.06±0\.030\.06\\pm 0\.030\.63±0\.000\.63\\pm 0\.000\.99±0\.000\.99\\pm 0\.00Invertible MLP1\.00±0\.001\.00\\pm 0\.000\.13±0\.120\.13\\pm 0\.120\.60±0\.010\.60\\pm 0\.01N/ALinear1\.00±0\.001\.00\\pm 0\.000\.05±0\.060\.05\\pm 0\.060\.65±0\.030\.65\\pm 0\.030\.99±0\.000\.99\\pm 0\.00Patches0\.99±0\.000\.99\\pm 0\.000\.25±0\.030\.25\\pm 0\.030\.63±0\.020\.63\\pm 0\.020\.88±0\.010\.88\\pm 0\.01Spiral1\.00±0\.001\.00\\pm 0\.000\.72±0\.330\.72\\pm 0\.330\.62±0\.010\.62\\pm 0\.011\.00±0\.001\.00\\pm 0\.00

### 5\.2CIFAR\-10 Experiments

We test the qualitative implications of our theory on CIFAR\-10 using SimCLR\(Chenet al\.,[2020](https://arxiv.org/html/2606.04280#bib.bib1)\)with three encoder architectures of comparable size \(∼\\sim11M parameters each\)\. We rate inductive bias by how closely each architecture’s structural priors align with the spatial generative structure of natural images\. Equivalently, this measures how strongly the architecture favors approximate inverses of image formation\. ResNet\-18\(Heet al\.,[2016](https://arxiv.org/html/2606.04280#bib.bib34)\)has high inductive bias because convolutional layers encode spatial locality and translation equivariance\. Vision Transformer \(ViT\)\(Dosovitskiyet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib38)\)with4×44\\times 4patches has medium inductive bias because self\-attention can learn global spatial structure, but does not hardcode locality to the same extent\. The MLP has low inductive bias because it treats each image as a flat vector and imposes little domain\-specific restriction on the hypothesis space\. All encoders project to 512\-dimensionalL2L^\{2\}\-normalized embeddings and are initialized with random weights\. Because CIFAR\-10 does not provide ground\-truth latent coordinates, linear probe accuracy is not a direct test of identifiability\. We use it as a real\-data sanity check for the qualitative predictions of the theory: richer sampling mechanisms and better aligned architectural inductive bias should improve downstream representation quality\.

#### Augmentation Regimes\.

We design three augmentation regimes to vary the degree to which the diversity condition is approximated \(Figure[4](https://arxiv.org/html/2606.04280#S5.F4)\): \(1\)All: color jitter, random crop, horizontal flip, grayscale, blur, and cutout\(DeVries and Taylor,[2017](https://arxiv.org/html/2606.04280#bib.bib29)\), perturbing as many features as possible; \(2\)Crop Only: Perturbing all features slightly, but with a lesser degree than All augmentations; \(3\)All w/o crop: all augmentations except crop, varying the features in a more fixed manner\. Training uses InfoNCE withτ=0\.5\\tau=0\.5, Adam optimizer\(Kingma and Ba,[2015](https://arxiv.org/html/2606.04280#bib.bib28)\)\(lr =3×10−43\\times 10^\{\-4\}\), batch size 2000, for 200 epochs\. We evaluate via linear probing over 5 runs per configuration\.

#### Results\.

The results \(Figure[3](https://arxiv.org/html/2606.04280#S5.F3)\) are consistent with the qualitative predictions of the theory\. First, the “All” augmentation regime yields the highest accuracy across all architectures, consistent with the expectation that broader sampling support improves representation quality\. Second, higher inductive bias consistently improves performance: ResNet\-18 outperforms ViT, which outperforms MLP, across all augmentation regimes\. Third, and most importantly, the performance gap between architectures widens as the diversity condition is increasingly violated\. Under the “All” regime, the gap between ResNet\-18 and MLP is moderate; under “All w/o crop”, this gap increases substantially\. This interaction effect supports the compensatory role of inductive bias: when sampling diversity is insufficient, architectural priors become critical for recovering useful representations\. Notably, for MLP the “All” and “Crop” regimes yield nearly identical performance, suggesting that without appropriate inductive bias, the encoder cannot exploit the additional augmentations, since cropping provides the majority of the meaningful signal\.

![Refer to caption](https://arxiv.org/html/2606.04280v1/images/cifar_10_experiments/cifar_10_by_architecture.png)Figure 3:Linear probe accuracy on CIFAR\-10 by architecture and augmentation regime\. Individual runs shown as points; bars indicate mean±1\\pm 1std\. The “All” regime best approximates the diversity condition and yields highest accuracy across all architectures\.![Refer to caption](https://arxiv.org/html/2606.04280v1/images/cifar_augmentation_examples/1_original.png)

\(a\) Original

![Refer to caption](https://arxiv.org/html/2606.04280v1/images/cifar_augmentation_examples/2_cropped.png)

\(b\) Crop

![Refer to caption](https://arxiv.org/html/2606.04280v1/images/cifar_augmentation_examples/3_all_no_crop.png)

\(c\) All w/o crop

![Refer to caption](https://arxiv.org/html/2606.04280v1/images/cifar_augmentation_examples/4_crop_then_all.png)

\(d\) All

Figure 4:CIFAR\-10 augmentation regimes\. \(a\) Original image\. \(b\) Crop Only: random resized crop altering spatial extent\. \(c\) All without crop: color jitter, horizontal flip, rotation, and blur\. \(d\) All augmentations combined\. Cropping changes visible spatial extent and local statistics, while color and blur transformations mainly alter appearance in this example\.

## 6Conclusion

We have presented a theoretical framework for understanding when contrastive learning recovers meaningful latent representations\. Our central contribution is thediversity condition\(Definition[3\.1](https://arxiv.org/html/2606.04280#S3.Thmtheorem1)\), a requirement onPZ~\|zP\_\{\\tilde\{Z\}\|z\}that is necessary for isometric latent recovery\. When it holds, sufficiently expressive encoders recover the latent space up to an orthogonal transformation; when it is violated, the contrastive objective can actively disincentivize geometry\-preserving solutions\. The adapted InfoNCE objective makes isometric solutions achievable under violated diversity, but does not guarantee their selection\. Our synthetic and CIFAR\-10 experiments show that sampling diversity and architectural inductive bias jointly determine representation quality\.

#### Limitations and future work\.

Motivated by practicalL2L^\{2\}normalization, our analysis uses a spherical latent space and vMF conditionals; future work should relax these assumptions and estimate diversity violation from data\. Adapted InfoNCE is a theoretical diagnostic rather than a scalable objective: anchor\-specific negatives requireO​\(N​\(M\+1\)\)O\(N\(M\+1\)\)memory, and support correction restores achievability but not selection\.

## Impact Statement

This work advances theoretical understanding of contrastive learning, a foundational technique for self\-supervised representation learning\. Our contributions are primarily theoretical, providing formal conditions \(the diversity condition\) under which contrastive methods succeed or fail at recovering meaningful latent structure\.

The practical implications are indirect but potentially significant\. By clarifying the interplay between sampling mechanisms and architectural inductive bias, our framework may guide more principled design of augmentation pipelines and encoder architectures, potentially reducing computational waste from trial\-and\-error experimentation\. This could lower the environmental cost of training large\-scale representation learning systems\.

Contrastive learning underlies many deployed systems in vision, language, and multimodal domains\. Improved theoretical understanding may help practitioners anticipate failure modes before deployment, particularly in high\-stakes applications such as medical imaging or scientific discovery where representation quality directly affects downstream reliability\.

We do not foresee direct negative societal consequences from this theoretical work\. However, as with any advance in representation learning, improved methods could enhance both beneficial applications \(e\.g\., drug discovery, climate modeling\) and potentially harmful ones \(e\.g\., surveillance\)\. We encourage practitioners to consider the ethical implications of specific downstream applications enabled by better representation learning\.

## References

- M\. Bahrami, A\. Tejada\-Lapuerta, S\. Becker, F\. S\. Hashemi G\., and F\. J\. Theis \(2025\)scConcept: contrastive pretraining for technology\-agnostic single\-cell representations beyond reconstruction\.bioRxiv\.External Links:[Document](https://dx.doi.org/10.1101/2025.10.14.682419),[Link](https://www.biorxiv.org/content/early/2025/10/15/2025.10.14.682419),https://www\.biorxiv\.org/content/early/2025/10/15/2025\.10\.14\.682419\.full\.pdfCited by:[§1](https://arxiv.org/html/2606.04280#S1.p2.1)\.
- T\. Ballard \(2022\)Contrastive learning for climate model bias correction and super\-resolution\.InAAAI 2022 Fall Symposium: The Role of AI in Responding to Climate Challenges,External Links:[Link](https://www.climatechange.ai/papers/aaaifss2022/10)Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p2.1)\.
- A\. Bardes, J\. Ponce, and Y\. LeCun \(2022\)VICReg: variance\-invariance\-covariance regularization for self\-supervised learning\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Bengio, A\. Courville, and P\. Vincent \(2013\)Representation learning: a review and new perspectives\.IEEE Transactions on Pattern Analysis and Machine Intelligence35\(8\),pp\. 1798–1828\.Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Chen, S\. Kornblith, M\. Norouzi, and G\. Hinton \(2020\)A simple framework for contrastive learning of visual representations\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 1597–1607\.Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p1.1),[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px4.p1.1),[§3\.1](https://arxiv.org/html/2606.04280#S3.SS1.p2.4),[§3\.4](https://arxiv.org/html/2606.04280#S3.SS4.p2.3),[§5\.2](https://arxiv.org/html/2606.04280#S5.SS2.p1.3)\.
- P\. Comon \(1994\)Independent component analysis, a new concept?\.Signal Processing36\(3\),pp\. 287–314\.Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Cy, A\. Chemparathy, M\. Han, R\. Dangovski, P\. Y\. Lu, and M\. Soljacic \(2023\)Studying phase transitions in contrastive learning With physics\-inspired datasets\.InICLR 2023 Workshop Physics4ML,Note:Physics4ML posterExternal Links:[Link](https://openreview.net/forum?id=djssHWljSA)Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p2.1)\.
- R\. Dangovski, L\. Jing, C\. Loh, S\. Han, A\. Srivastava, B\. Cheung, P\. Agrawal, and M\. Soljačić \(2022\)Equivariant contrastive learning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p3.1)\.
- T\. DeVries and G\. W\. Taylor \(2017\)Improved regularization of convolutional neural networks with cutout\.arXiv preprint arXiv:1708\.04552\.External Links:[Link](https://arxiv.org/abs/1708.04552)Cited by:[§5\.2](https://arxiv.org/html/2606.04280#S5.SS2.SSS0.Px1.p1.2)\.
- A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby \(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.InInternational Conference on Learning Representations,Cited by:[§3\.5](https://arxiv.org/html/2606.04280#S3.SS5.p1.4),[§5\.2](https://arxiv.org/html/2606.04280#S5.SS2.p1.3)\.
- J\. Grill, F\. Strub, F\. Altché, C\. Tallec, P\. H\. Richemond, E\. Buchatskaya, C\. Doersch, B\. Avila Pires, Z\. D\. Guo, M\. Gheshlaghi Azar, B\. Piot, K\. Kavukcuoglu, R\. Munos, and M\. Valko \(2020\)Bootstrap your own latent: a new approach to self\-supervised learning\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 21271–21284\.Cited by:[§4](https://arxiv.org/html/2606.04280#S4.p1.1)\.
- J\. Haas, W\. Yolland, and B\. Rabus \(2023\)Linking neural collapse and L2 normalization with improved out\-of\-distribution detection in deep neural networks\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=fjkN5Ur2d6)Cited by:[§4](https://arxiv.org/html/2606.04280#S4.p1.1)\.
- J\. Haas, W\. Yolland, and B\. Rabus \(2024\)Exploring simple, high quality out\-of\-distribution detection with L2 normalization\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=daX2UkLMS0)Cited by:[§3\.4](https://arxiv.org/html/2606.04280#S3.SS4.p2.3)\.
- J\. Z\. HaoChen and T\. Ma \(2023\)A theoretical study of inductive biases in contrastive learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=AuEgNlEAmed)Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px3.p1.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 770–778\.Cited by:[§5\.2](https://arxiv.org/html/2606.04280#S5.SS2.p1.3)\.
- A\. Hyvärinen and H\. Morioka \(2016\)Unsupervised feature extraction by time\-contrastive learning and nonlinear ICA\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix D](https://arxiv.org/html/2606.04280#A4.p1.1),[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px1.p2.1),[§5\.1](https://arxiv.org/html/2606.04280#S5.SS1.SSS0.Px1.p1.6),[§5\.1](https://arxiv.org/html/2606.04280#S5.SS1.SSS0.Px2.p1.1),[Algorithm 5](https://arxiv.org/html/2606.04280#alg5)\.
- A\. Hyvärinen and E\. Oja \(2000\)Independent component analysis: algorithms and applications\.Neural Networks13\(4\-5\),pp\. 411–430\.Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Hyvärinen, H\. Sasaki, and R\. E\. Turner \(2019\)Nonlinear ICA using auxiliary variables and generalized contrastive learning\.InProceedings of the 22nd International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.89,pp\. 859–868\.Cited by:[§3\.1](https://arxiv.org/html/2606.04280#S3.SS1.p1.5)\.
- A\. Jaiswal, A\. R\. Babu, M\. Z\. Zadeh, D\. Banerjee, and F\. Makedon \(2021\)A survey on contrastive self\-supervised learning\.Technologies9\(1\),pp\. 2\.External Links:[Document](https://dx.doi.org/10.3390/technologies9010002)Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p1.1)\.
- W\. Ji, Z\. Deng, R\. Nakada, J\. Zou, and L\. Zhang \(2023\)The power of contrast for feature learning: a theoretical analysis\.Journal of Machine Learning Research24\(330\),pp\. 1–78\.External Links:[Link](https://jmlr.org/papers/v24/21-1501.html)Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p4.1)\.
- D\. P\. Kingma and J\. Ba \(2015\)Adam: a method for stochastic optimization\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1412.6980)Cited by:[§5\.1](https://arxiv.org/html/2606.04280#S5.SS1.SSS0.Px1.p1.6),[§5\.2](https://arxiv.org/html/2606.04280#S5.SS2.SSS0.Px1.p1.2)\.
- M\. Kirchhof, E\. Kasneci, and S\. J\. Oh \(2023\)Probabilistic contrastive learning recovers the correct aleatoric uncertainty of ambiguous inputs\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 17085–17104\.Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p4.1)\.
- A\. Krizhevsky \(2009\)Learning multiple layers of features from tiny images\.Technical reportTechnical ReportTR\-2009,University of Toronto\.Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p7.1)\.
- Y\. LeCun, L\. Bottou, Y\. Bengio, and P\. Haffner \(1998\)Gradient\-based learning applied to document recognition\.Proceedings of the IEEE86\(11\),pp\. 2278–2324\.External Links:[Document](https://dx.doi.org/10.1109/5.726791)Cited by:[§3\.5](https://arxiv.org/html/2606.04280#S3.SS5.p1.4)\.
- S\. Liu, L\. Zhao, and D\. Chen \(2025\)CLICv2: image complexity representation via content invariance contrastive learning\.External Links:2503\.06641,[Link](https://arxiv.org/abs/2503.06641)Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p3.1)\.
- Z\. Liu, K\. Hu, J\. Zhang, X\. Ren, and X\. Wang \(2026\)Long\-term air quality data filling based on contrastive learning\.Information17\(2\)\.External Links:[Link](https://www.mdpi.com/2078-2489/17/2/121),ISSN 2078\-2489,[Document](https://dx.doi.org/10.3390/info17020121)Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p2.1)\.
- F\. Locatello, S\. Bauer, M\. Lucic, G\. Raetsch, S\. Gelly, B\. Schölkopf, and O\. Bachem \(2019\)Challenging common assumptions in the unsupervised learning of disentangled representations\.InInternational Conference on Machine Learning,pp\. 4114–4124\.Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Mitrovic, B\. McWilliams, J\. Walker, L\. Buesing, and C\. Blundell \(2021\)Representation learning via invariant causal mechanisms\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px2.p1.1)\.
- R\. P\. K\. Poudel, H\. Pandya, and R\. Cipolla \(2022\)Contrastive unsupervised learning of world model with invariant causal features\.External Links:2209\.14932,[Link](https://arxiv.org/abs/2209.14932)Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p3.1)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever \(2021\)Learning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 8748–8763\.Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p1.1)\.
- T\. Richter, M\. Bahrami, Y\. Xia, D\. S\. Fischer, and F\. J\. Theis \(2025\)Delineating the effective use of self\-supervised learning in single\-cell genomics\.Nature Machine Intelligence7\(1\),pp\. 68–78\.Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p2.1)\.
- E\. Rusak, P\. Reizinger, A\. Juhos, O\. Bringmann, R\. S\. Zimmermann, and W\. Brendel \(2025\)InfoNCE: identifying the gap between theory and practice\.InProceedings of The 28th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.258,pp\. 4159–4167\.Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px3.p1.1)\.
- R\. Sandilya, S\. Perez, C\. Lynch, L\. Victoria, B\. Zebley, D\. M\. Buchanan, M\. T\. Bhati, N\. Williams, T\. J\. Spellman, F\. M\. Gunning, C\. Liston, and L\. Grosenick \(2025\)Contrastive diffusion alignment: learning structured latents for controllable generation\.External Links:2510\.14190,[Link](https://arxiv.org/abs/2510.14190)Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p4.1)\.
- Y\. Tian, C\. Sun, B\. Poole, D\. Krishnan, C\. Schmid, and P\. Isola \(2020\)What makes for good views for contrastive learning?\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Tschannen, B\. Mustafa, and N\. Houlsby \(2023\)CLIPPO: image\-and\-language understanding from pixels only\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 11006–11017\.Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Uelwer, J\. Robine, S\. S\. Wagner, M\. Höftmann, E\. Upschulte, S\. Konietzny, M\. Behrendt, and S\. Harmeling \(2023\)A survey on self\-supervised representation learning\.External Links:2308\.11455,[Link](https://arxiv.org/abs/2308.11455)Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p2.1)\.
- A\. van den Oord, Y\. Li, and O\. Vinyals \(2019\)Representation learning with contrastive predictive coding\.External Links:1807\.03748,[Link](https://arxiv.org/abs/1807.03748)Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px4.p1.1),[§3\.4](https://arxiv.org/html/2606.04280#S3.SS4.p1.1),[Definition 3\.2](https://arxiv.org/html/2606.04280#S3.Thmtheorem2)\.
- V\. Vapnik \(1999\)The nature of statistical learning theory\.Springer: New York\.Cited by:[§3\.5](https://arxiv.org/html/2606.04280#S3.SS5.p1.4)\.
- T\. Wang and P\. Isola \(2020\)Understanding contrastive representation learning through alignment and uniformity on the hypersphere\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 9929–9939\.Cited by:[§A\.2](https://arxiv.org/html/2606.04280#A1.SS2.1.p1.1)\.
- Z\. Wen and Y\. Li \(2021\)Toward understanding the feature learning process of self\-supervised contrastive learning\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 11112–11122\.External Links:[Link](https://proceedings.mlr.press/v139/wen21c.html)Cited by:[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Wilkinson, R\. Radev, and S\. Alonso\-Monsalve \(2025\)Contrastive learning for robust representations of neutrino data\.Physical Review D111\(9\)\.External Links:ISSN 2470\-0029,[Link](http://dx.doi.org/10.1103/PhysRevD.111.092011),[Document](https://dx.doi.org/10.1103/physrevd.111.092011)Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p2.1)\.
- A\. T\. A\. Wood \(1994\)Simulation of the von Mises–Fisher distribution\.Communications in Statistics\-Simulation and Computation23\(1\),pp\. 157–164\.External Links:[Document](https://dx.doi.org/10.1080/03610919408813161)Cited by:[Algorithm 11](https://arxiv.org/html/2606.04280#alg11)\.
- J\. Zaliaduonis and S\. Gatidis \(2026\)A probabilistic generalization of the Mazur\-Ulam theorem\.External Links:2601\.03900,[Link](https://arxiv.org/abs/2601.03900)Cited by:[§A\.1](https://arxiv.org/html/2606.04280#A1.SS1.18.p3.7),[§A\.1](https://arxiv.org/html/2606.04280#A1.SS1.p3.1),[§1](https://arxiv.org/html/2606.04280#S1.p5.1)\.
- L\. Zhao, N\. B\. Gundavarapu, L\. Yuan, H\. Zhou, S\. Yan, J\. J\. Sun, L\. Friedman, R\. Qian, T\. Weyand, Y\. Zhao, R\. Hornung, F\. Schroff, M\. Yang, D\. A\. Ross, H\. Wang, H\. Adam, M\. Sirotenko, T\. Liu, and B\. Gong \(2024\)VideoPrism: a foundational visual encoder for video understanding\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 60785–60811\.Cited by:[§1](https://arxiv.org/html/2606.04280#S1.p1.1)\.
- R\. S\. Zimmermann, Y\. Sharma, S\. Schneider, M\. Bethge, and W\. Brendel \(2021\)Contrastive learning inverts the data generating process\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 12979–12990\.Cited by:[§A\.3](https://arxiv.org/html/2606.04280#A1.SS3.p3.1),[Appendix D](https://arxiv.org/html/2606.04280#A4.p1.1),[§1](https://arxiv.org/html/2606.04280#S1.p5.1),[§2](https://arxiv.org/html/2606.04280#S2.SS0.SSS0.Px1.p2.1),[§3\.1](https://arxiv.org/html/2606.04280#S3.SS1.p1.5),[§3\.3](https://arxiv.org/html/2606.04280#S3.SS3.p1.1),[§3\.4](https://arxiv.org/html/2606.04280#S3.SS4.p2.3),[Theorem 3\.3](https://arxiv.org/html/2606.04280#S3.Thmtheorem3)\.

## Appendix ATheoretical Results: Proofs

This appendix provides complete proofs for the theoretical results presented in Section[4](https://arxiv.org/html/2606.04280#S4)\. We organize the material into three subsections corresponding to the three scenarios analyzed: full diversity \(Section[A\.1](https://arxiv.org/html/2606.04280#A1.SS1)\), violated diversity \(Section[A\.2](https://arxiv.org/html/2606.04280#A1.SS2)\), and the corrected model \(Section[A\.3](https://arxiv.org/html/2606.04280#A1.SS3)\)\.

### A\.1Proofs for Reconstruction Under Full Diversity

We first establish that cross\-entropy minimizers preserve inner products almost surely, which forms the foundation for proving linear identifiability\.

###### Theorem A\.1\(Cross\-Entropy Minimizers Preserve Inner Products\)\.

Let𝒵=𝕊k−1\\mathcal\{Z\}=\\mathbb\{S\}^\{k\-1\},κ\>0\\kappa\>0, and letPZ~\|zP\_\{\\tilde\{Z\}\|z\}have density

p​\(z~\|z\)=Cp−1​exp⁡\(κ​z~⊤​z\),p\(\\tilde\{z\}\|z\)=C\_\{p\}^\{\-1\}\\exp\(\\kappa\\tilde\{z\}^\{\\top\}z\),whereCpC\_\{p\}is the normalizing constant with respect toσ\\sigma\. LetQh,zQ\_\{h,z\}have densityqh\(⋅\|z\)q\_\{h\}\(\\cdot\|z\)with respect toσ\\sigma\. Leth:𝒵→𝒵h:\\mathcal\{Z\}\\to\\mathcal\{Z\}be a recovery map\. Ifhhminimizes the cross\-entropy

ℒh:=𝔼z∼PZ​\[H​\(PZ~\|z,Qh,z\)\],\\mathcal\{L\}\_\{h\}:=\\mathbb\{E\}\_\{z\\sim P\_\{Z\}\}\[H\(P\_\{\\tilde\{Z\}\|z\},Q\_\{h,z\}\)\],thenp​\(z~\|z\)=qh​\(z~\|z\)p\(\\tilde\{z\}\|z\)=q\_\{h\}\(\\tilde\{z\}\|z\)PZ~\|zP\_\{\\tilde\{Z\}\|z\}\-a\.s\., andz⊤​z~=h​\(z\)⊤​h​\(z~\)z^\{\\top\}\\tilde\{z\}=h\(z\)^\{\\top\}h\(\\tilde\{z\}\)PZ,Z~P\_\{Z,\\tilde\{Z\}\}\-a\.s\.

###### Proof\.

Sincehhminimizesℒh\\mathcal\{L\}\_\{h\}, it minimizesH​\(PZ~\|z,Qh,z\)H\(P\_\{\\tilde\{Z\}\|z\},Q\_\{h,z\}\)almost surely with respect toPZP\_\{Z\}\. The cross\-entropyHHis minimized when

p\(⋅\|z\)=qh\(⋅\|z\)PZ~\|z\-a\.s\.p\(\\cdot\|z\)=q\_\{h\}\(\\cdot\|z\)\\quad P\_\{\\tilde\{Z\}\|z\}\\text\{\-a\.s\.\}
For a fixedzz, this implies:

eκ​z~⊤​zCp=eh​\(z~\)⊤​h​\(z\)/τD​\(z\)PZ~\|z​\-a\.s\.\\frac\{e^\{\\kappa\\tilde\{z\}^\{\\top\}z\}\}\{C\_\{p\}\}=\\frac\{e^\{h\(\\tilde\{z\}\)^\{\\top\}h\(z\)/\\tau\}\}\{D\(z\)\}\\quad P\_\{\\tilde\{Z\}\|z\}\\text\{\-a\.s\.\}\(9\)whereD​\(z\)D\(z\)andCpC\_\{p\}are the normalizing constants for the respective von Mises\-Fisher distributions\.

Taking the logarithm yields:

log⁡\(D​\(z\)Cp\)​τ\+τ​κ​z~⊤​z=h​\(z~\)⊤​h​\(z\)PZ~\|z​\-a\.s\.\\log\\left\(\\frac\{D\(z\)\}\{C\_\{p\}\}\\right\)\\tau\+\\tau\\kappa\\tilde\{z\}^\{\\top\}z=h\(\\tilde\{z\}\)^\{\\top\}h\(z\)\\quad P\_\{\\tilde\{Z\}\|z\}\\text\{\-a\.s\.\}\(10\)
Sincehhmaps onto the unit sphere,

−1≤h​\(z~\)⊤​h​\(z\)≤1PZ~\|z​\-a\.s\.\-1\\leq h\(\\tilde\{z\}\)^\{\\top\}h\(z\)\\leq 1\\quad P\_\{\\tilde\{Z\}\|z\}\\text\{\-a\.s\.\}\(11\)
Rearranging gives:

−1≤log⁡\(D​\(z\)Cp\)​τ\+τ​κ​z~⊤​z≤1PZ~\|z​\-a\.s\.\-1\\leq\\log\\left\(\\frac\{D\(z\)\}\{C\_\{p\}\}\\right\)\\tau\+\\tau\\kappa\\tilde\{z\}^\{\\top\}z\\leq 1\\quad P\_\{\\tilde\{Z\}\|z\}\\text\{\-a\.s\.\}\(12\)
This simplifies to:

−1−log⁡\(D​\(z\)Cp\)​ττ​κ≤z~⊤​z≤1−log⁡\(D​\(z\)Cp\)​ττ​κ\\frac\{\-1\-\\log\\left\(\\frac\{D\(z\)\}\{C\_\{p\}\}\\right\)\\tau\}\{\\tau\\kappa\}\\leq\\tilde\{z\}^\{\\top\}z\\leq\\frac\{1\-\\log\\left\(\\frac\{D\(z\)\}\{C\_\{p\}\}\\right\)\\tau\}\{\\tau\\kappa\}\(13\)
SincePZ~\|zP\_\{\\tilde\{Z\}\|z\}is a von Mises\-Fisher distribution, the support ofz~⊤​z\\tilde\{z\}^\{\\top\}zunder this measure includes regions arbitrarily close to both 1 and−1\-1\. Specifically, for anyc∈\(−1,1\)c\\in\(\-1,1\):

μ​\(\{z~∈𝒵:z~⊤​z≥c\}\)=β​∫0cos−1⁡\(c\)eκ​cos⁡θ​sink−2⁡θ​d​θ\>0\\mu\\left\(\\\{\\tilde\{z\}\\in\\mathcal\{Z\}:\\tilde\{z\}^\{\\top\}z\\geq c\\\}\\right\)=\\beta\\int\_\{0\}^\{\\cos^\{\-1\}\(c\)\}e^\{\\kappa\\cos\\theta\}\\sin^\{k\-2\}\\theta\\,d\\theta\>0\(14\)whereβ\>0\\beta\>0is the spherical\-measure normalization constant\.

To satisfy the inequality almost surely with respect toPZ~\|zP\_\{\\tilde\{Z\}\|z\}, the bounds must equal the extremes:

1−log⁡\(D​\(z\)Cp\)​ττ​κ=1and−1−log⁡\(D​\(z\)Cp\)​ττ​κ=−1\\frac\{1\-\\log\\left\(\\frac\{D\(z\)\}\{C\_\{p\}\}\\right\)\\tau\}\{\\tau\\kappa\}=1\\quad\\text\{and\}\\quad\\frac\{\-1\-\\log\\left\(\\frac\{D\(z\)\}\{C\_\{p\}\}\\right\)\\tau\}\{\\tau\\kappa\}=\-1\(15\)
Both equations yield:

log⁡\(D​\(z\)Cp\)=0⟹D​\(z\)=Cp\\log\\left\(\\frac\{D\(z\)\}\{C\_\{p\}\}\\right\)=0\\implies D\(z\)=C\_\{p\}\(16\)and

Substituting back:

z~⊤​z=h​\(z~\)⊤​h​\(z\)PZ~\|z​\-a\.s\.\\tilde\{z\}^\{\\top\}z=h\(\\tilde\{z\}\)^\{\\top\}h\(z\)\\quad P\_\{\\tilde\{Z\}\|z\}\\text\{\-a\.s\.\}\(18\)
Sincehhminimizesℒh\\mathcal\{L\}\_\{h\}, this holds for almost allzzwith respect toPZP\_\{Z\}, implying:

z~⊤​z=h​\(z~\)⊤​h​\(z\)PZ,Z~​\-a\.s\.\\tilde\{z\}^\{\\top\}z=h\(\\tilde\{z\}\)^\{\\top\}h\(z\)\\quad P\_\{Z,\\tilde\{Z\}\}\\text\{\-a\.s\.\}\(19\)∎

The following corollary translates inner product preservation to distance preservation, which is the key geometric property needed for identifiability\.

###### Corollary A\.2\(Cross\-Entropy Minimizers Preserve Distances Almost Everywhere\)\.

Ifhhminimizes the expected cross\-entropy loss𝔼z∼PZ​\[H​\(PZ~\|z,Qh,z\)\]\\mathbb\{E\}\_\{z\\sim P\_\{Z\}\}\[H\(P\_\{\\tilde\{Z\}\|z\},Q\_\{h,z\}\)\], then

‖h​\(z\)−h​\(z~\)‖2=‖z−z~‖2PZ,Z~​\-a\.s\.\\\|h\(z\)\-h\(\\tilde\{z\}\)\\\|^\{2\}=\\\|z\-\\tilde\{z\}\\\|^\{2\}\\quad P\_\{Z,\\tilde\{Z\}\}\\text\{\-a\.s\.\}

###### Proof\.

Since bothz,z~∈𝒵=𝕊k−1z,\\tilde\{z\}\\in\\mathcal\{Z\}=\\mathbb\{S\}^\{k\-1\}andh​\(z\),h​\(z~\)∈𝒵′=𝕊k−1h\(z\),h\(\\tilde\{z\}\)\\in\\mathcal\{Z\}^\{\\prime\}=\\mathbb\{S\}^\{k\-1\}, we have‖z‖2=‖z~‖2=‖h​\(z\)‖2=‖h​\(z~\)‖2=1\\\|z\\\|^\{2\}=\\\|\\tilde\{z\}\\\|^\{2\}=\\\|h\(z\)\\\|^\{2\}=\\\|h\(\\tilde\{z\}\)\\\|^\{2\}=1\.

Expanding the squared Euclidean distance:

‖z−z~‖2\\displaystyle\\\|z\-\\tilde\{z\}\\\|^\{2\}=‖z‖2−2​z⊤​z~\+‖z~‖2=2−2​z⊤​z~\\displaystyle=\\\|z\\\|^\{2\}\-2z^\{\\top\}\\tilde\{z\}\+\\\|\\tilde\{z\}\\\|^\{2\}=2\-2z^\{\\top\}\\tilde\{z\}\(20\)
Similarly, for the mapped points:

‖h​\(z\)−h​\(z~\)‖2\\displaystyle\\\|h\(z\)\-h\(\\tilde\{z\}\)\\\|^\{2\}=‖h​\(z\)‖2−2​h​\(z\)⊤​h​\(z~\)\+‖h​\(z~\)‖2=2−2​h​\(z\)⊤​h​\(z~\)\\displaystyle=\\\|h\(z\)\\\|^\{2\}\-2h\(z\)^\{\\top\}h\(\\tilde\{z\}\)\+\\\|h\(\\tilde\{z\}\)\\\|^\{2\}=2\-2h\(z\)^\{\\top\}h\(\\tilde\{z\}\)\(21\)
By Theorem[A\.1](https://arxiv.org/html/2606.04280#A1.Thmtheorem1), we havez⊤​z~=h​\(z\)⊤​h​\(z~\)z^\{\\top\}\\tilde\{z\}=h\(z\)^\{\\top\}h\(\\tilde\{z\}\)PZ,Z~P\_\{Z,\\tilde\{Z\}\}\-a\.s\. Therefore:

‖h​\(z\)−h​\(z~\)‖2=2−2​h​\(z\)⊤​h​\(z~\)=2−2​z⊤​z~=‖z−z~‖2\\\|h\(z\)\-h\(\\tilde\{z\}\)\\\|^\{2\}=2\-2h\(z\)^\{\\top\}h\(\\tilde\{z\}\)=2\-2z^\{\\top\}\\tilde\{z\}=\\\|z\-\\tilde\{z\}\\\|^\{2\}\(22\)holdsPZ,Z~P\_\{Z,\\tilde\{Z\}\}\-almost surely\. ∎

The distance preservation property established above holds only almost everywhere with respect to the joint distribution\. To conclude that the optimal recovery map is a global orthogonal transformation, we apply the probabilistic generalization of the Mazur\-Ulam theorem\(Zaliaduonis and Gatidis,[2026](https://arxiv.org/html/2606.04280#bib.bib40)\), which shows that isometries holding almost everywhere on probability spaces can be extended to global isometries on the entire space\.

###### Proof of Theorem[4\.4](https://arxiv.org/html/2606.04280#S4.Thmtheorem4)\.

By Corollary[A\.2](https://arxiv.org/html/2606.04280#A1.Thmtheorem2), the optimal recovery maph:𝒵→𝒵h:\\mathcal\{Z\}\\to\\mathcal\{Z\}preserves distances almost everywhere with respect to the joint distributionPZ,Z~P\_\{Z,\\tilde\{Z\}\}\. Specifically, there exists a setN⊂𝒵×𝒵N\\subset\\mathcal\{Z\}\\times\\mathcal\{Z\}withPZ,Z~​\(N\)=0P\_\{Z,\\tilde\{Z\}\}\(N\)=0such that

‖h​\(z\)−h​\(z~\)‖=‖z−z~‖for all​\(z,z~\)∈\(𝒵×𝒵\)∖N\.\\\|h\(z\)\-h\(\\tilde\{z\}\)\\\|=\\\|z\-\\tilde\{z\}\\\|\\quad\\text\{for all \}\(z,\\tilde\{z\}\)\\in\(\\mathcal\{Z\}\\times\\mathcal\{Z\}\)\\setminus N\.
Since the marginal lawPZP\_\{Z\}is uniform on the sphere and has full support,hhpreserves pairwise distances on a set of fullPZP\_\{Z\}\-measure\.

Applying the probabilistic Mazur\-Ulam theorem\(Zaliaduonis and Gatidis,[2026](https://arxiv.org/html/2606.04280#bib.bib40)\), which extends almost\-everywhere isometries on probability spaces to global isometries, there existsH:ℝk→ℝkH:\\mathbb\{R\}^\{k\}\\to\\mathbb\{R\}^\{k\}of the formH​\(x\)=A​x\+bH\(x\)=Ax\+bwithA∈O​\(k\)A\\in O\(k\)andb∈ℝkb\\in\\mathbb\{R\}^\{k\}, such thath​\(z\)=H​\(z\)h\(z\)=H\(z\)forPZP\_\{Z\}\-almost allz∈𝒵z\\in\\mathcal\{Z\}\.

Since both𝒵\\mathcal\{Z\}and𝒵′\\mathcal\{Z\}^\{\\prime\}are unit spheres centered at the origin, andHHagrees almost everywhere withh:𝒵→𝒵′h:\\mathcal\{Z\}\\to\\mathcal\{Z\}^\{\\prime\}, the translation component must vanish \(b=0b=0\) and the linear part must preserve the unit sphere\. This implies thatAAis an orthogonal matrix, i\.e\.,A⊤​A=IA^\{\\top\}A=I\.

Therefore, the optimal recovery maphhcoincides with an orthogonal transformation almost everywhere:

h​\(z\)=A​zfor​μ​\-almost all​z∈𝒵,h\(z\)=Az\\quad\\text\{for \}\\mu\\text\{\-almost all \}z\\in\\mathcal\{Z\},whereA∈O​\(k\)A\\in O\(k\)is an orthogonal matrix\. ∎

### A\.2Proofs for Reconstruction Under Violated Diversity

When the diversity condition is violated, the conditional law is constrained to a lower\-dimensional submanifold\. We decompose each latent vectorz∈𝒵=𝕊k−1z\\in\\mathcal\{Z\}=\\mathbb\{S\}^\{k\-1\}into two components:

z=\(u,v\)⊤∈ℝm×ℝℓ,where​m\+ℓ=k​and​m,ℓ\>0\.z=\(u,v\)^\{\\top\}\\in\\mathbb\{R\}^\{m\}\\times\\mathbb\{R\}^\{\\ell\},\\quad\\text\{where \}m\+\\ell=k\\text\{ and \}m,\\ell\>0\.Here,u∈ℝmu\\in\\mathbb\{R\}^\{m\}represents latent dimensions that remain fixed under conditional sampling, whilev∈ℝℓv\\in\\mathbb\{R\}^\{\\ell\}represents dimensions that can vary\. LetK​\(z\)=\{z~∈𝕊k−1:u~=u\}≅𝕊r​\(z\)ℓ−1K\(z\)=\\\{\\tilde\{z\}\\in\\mathbb\{S\}^\{k\-1\}:\\tilde\{u\}=u\\\}\\cong\\mathbb\{S\}^\{\\ell\-1\}\_\{r\(z\)\}, wherer​\(z\)=1−‖u‖2r\(z\)=\\sqrt\{1\-\\\|u\\\|^\{2\}\}\. LetσK​\(z\)\\sigma\_\{K\(z\)\}denote intrinsic surface measure onK​\(z\)K\(z\)\. The constrained conditional is the probability measurePZ~\|zKP^\{K\}\_\{\\tilde\{Z\}\|z\}with density

d​PZ~\|zKd​σK​\(z\)​\(z~\)=eκ​z~⊤​z∫K​\(z\)eκ​z⊤​z′​𝑑σK​\(z\)​\(z′\)\.\\frac\{dP^\{K\}\_\{\\tilde\{Z\}\|z\}\}\{d\\sigma\_\{K\(z\)\}\}\(\\tilde\{z\}\)=\\frac\{e^\{\\kappa\\tilde\{z\}^\{\\top\}z\}\}\{\\int\_\{K\(z\)\}e^\{\\kappa z^\{\\top\}z^\{\\prime\}\}d\\sigma\_\{K\(z\)\}\(z^\{\\prime\}\)\}\.\(23\)It is singular with respect to ambient spherical measure on𝕊k−1\\mathbb\{S\}^\{k\-1\}, but absolutely continuous with respect to the intrinsic measure onK​\(z\)K\(z\)\.

Despite this modification, the asymptotic relationship between the contrastive loss and cross\-entropy minimization remains valid\.

###### Theorem A\.3\(Asymptotic Equivalence Under Violated Diversity\)\.

Given the constrained conditional probability measure defined above, the marginal lawPZP\_\{Z\}uniform on𝒵=𝕊k−1\\mathcal\{Z\}=\\mathbb\{S\}^\{k\-1\}, temperatureτ\>0\\tau\>0, and number of negative samplesM\>0M\>0, asM→∞M\\to\\infty:

limM→∞ℒCL​\(h;τ,M\)−log⁡M\+log⁡\|𝒵\|=𝔼z∼PZ​\[H​\(PZ~\|zK,Qh,z\)\]\\lim\_\{M\\to\\infty\}\\mathcal\{L\}\_\{\\text\{CL\}\}\(h;\\tau,M\)\-\\log M\+\\log\|\\mathcal\{Z\}\|=\\mathbb\{E\}\_\{z\\sim P\_\{Z\}\}\[H\(P^\{K\}\_\{\\tilde\{Z\}\|z\},Q\_\{h,z\}\)\]

###### Proof\.

The proof follows identically to Theorem[3\.3](https://arxiv.org/html/2606.04280#S3.Thmtheorem3), as the asymptotic analysis relies only on the law of large numbers and properties of the logarithm, not on the specific structure of the ground\-truth conditional law\. See\(Wang and Isola,[2020](https://arxiv.org/html/2606.04280#bib.bib16)\)for the full argument\. ∎

The critical distinction arises when comparing global values of the limiting objective\. Under violated diversity, orthogonal recovery maps are not global minimizers\.

###### Proof of Theorem[4\.5](https://arxiv.org/html/2606.04280#S4.Thmtheorem5)\.

Letσ\\sigmadenote the normalized spherical measure on𝒵=𝕊k−1\\mathcal\{Z\}=\\mathbb\{S\}^\{k\-1\}\. By Theorem[A\.3](https://arxiv.org/html/2606.04280#A1.Thmtheorem3), it suffices to compare the asymptotic contrastive loss up to constants independent ofhh:

ℒ​\(h\)=−1τ​𝔼\(z,z~\)∼Ppos​\[h​\(z\)⊤​h​\(z~\)\]\+𝔼z∼σ​log​∫𝕊k−1exp⁡\(h​\(z\)⊤​h​\(z′\)τ\)​𝑑σ​\(z′\)\.\\mathcal\{L\}\(h\)=\-\\frac\{1\}\{\\tau\}\\mathbb\{E\}\_\{\(z,\\tilde\{z\}\)\\sim P\_\{\\mathrm\{pos\}\}\}\\\!\\left\[h\(z\)^\{\\top\}h\(\\tilde\{z\}\)\\right\]\+\\mathbb\{E\}\_\{z\\sim\\sigma\}\\log\\int\_\{\\mathbb\{S\}^\{k\-1\}\}\\exp\\\!\\left\(\\frac\{h\(z\)^\{\\top\}h\(z^\{\\prime\}\)\}\{\\tau\}\\right\)d\\sigma\(z^\{\\prime\}\)\.Forλ\>0\\lambda\>0, define

hλ​\(u,v\):=\(u,λ​v\)‖u‖2\+λ2​‖v‖2\.h\_\{\\lambda\}\(u,v\):=\\frac\{\(u,\\lambda v\)\}\{\\sqrt\{\\\|u\\\|^\{2\}\+\\lambda^\{2\}\\\|v\\\|^\{2\}\}\}\.Atλ=1\\lambda=1,hλh\_\{\\lambda\}is the identity\. Forλ≠1\\lambda\\neq 1,hλh\_\{\\lambda\}is not orthogonal because it changes inner products between points with different relative invariant and variant components\.

Write the alignment and uniformity terms as

A​\(λ\)\\displaystyle A\(\\lambda\):=𝔼\(z,z~\)∼Ppos​\[hλ​\(z\)⊤​hλ​\(z~\)\],\\displaystyle:=\\mathbb\{E\}\_\{\(z,\\tilde\{z\}\)\\sim P\_\{\\mathrm\{pos\}\}\}\\\!\\left\[h\_\{\\lambda\}\(z\)^\{\\top\}h\_\{\\lambda\}\(\\tilde\{z\}\)\\right\],U​\(λ\)\\displaystyle U\(\\lambda\):=𝔼z∼σ​log​∫𝕊k−1exp⁡\(hλ​\(z\)⊤​hλ​\(z′\)τ\)​𝑑σ​\(z′\)\.\\displaystyle:=\\mathbb\{E\}\_\{z\\sim\\sigma\}\\log\\int\_\{\\mathbb\{S\}^\{k\-1\}\}\\exp\\\!\\left\(\\frac\{h\_\{\\lambda\}\(z\)^\{\\top\}h\_\{\\lambda\}\(z^\{\\prime\}\)\}\{\\tau\}\\right\)d\\sigma\(z^\{\\prime\}\)\.Thenℒ​\(hλ\)=−τ−1​A​\(λ\)\+U​\(λ\)\\mathcal\{L\}\(h\_\{\\lambda\}\)=\-\\tau^\{\-1\}A\(\\lambda\)\+U\(\\lambda\)\.

We first show that shrinking the variant component improves alignment to first order\. Under the constrained positive\-pair distribution,u~=u\\tilde\{u\}=u\. Sincez,z~∈𝕊k−1z,\\tilde\{z\}\\in\\mathbb\{S\}^\{k\-1\}, this implies‖v‖=‖v~‖\\\|v\\\|=\\\|\\tilde\{v\}\\\|\. Therefore

hλ​\(z\)⊤​hλ​\(z~\)=‖u‖2\+λ2​v⊤​v~‖u‖2\+λ2​‖v‖2\.h\_\{\\lambda\}\(z\)^\{\\top\}h\_\{\\lambda\}\(\\tilde\{z\}\)=\\frac\{\\\|u\\\|^\{2\}\+\\lambda^\{2\}v^\{\\top\}\\tilde\{v\}\}\{\\\|u\\\|^\{2\}\+\\lambda^\{2\}\\\|v\\\|^\{2\}\}\.Witht=λ2t=\\lambda^\{2\}and

f​\(t\)=‖u‖2\+t​v⊤​v~‖u‖2\+t​‖v‖2,f\(t\)=\\frac\{\\\|u\\\|^\{2\}\+t\\,v^\{\\top\}\\tilde\{v\}\}\{\\\|u\\\|^\{2\}\+t\\\|v\\\|^\{2\}\},we have

f′​\(t\)=‖u‖2​\(v⊤​v~−‖v‖2\)\(‖u‖2\+t​‖v‖2\)2\.f^\{\\prime\}\(t\)=\\frac\{\\\|u\\\|^\{2\}\(v^\{\\top\}\\tilde\{v\}\-\\\|v\\\|^\{2\}\)\}\{\(\\\|u\\\|^\{2\}\+t\\\|v\\\|^\{2\}\)^\{2\}\}\.By Cauchy\-Schwarz,v⊤​v~≤‖v‖2v^\{\\top\}\\tilde\{v\}\\leq\\\|v\\\|^\{2\}, with strict inequality wheneverv≠v~v\\neq\\tilde\{v\}\. The constrained vMF conditional is non\-degenerate forκ\>0\\kappa\>0, sov≠v~v\\neq\\tilde\{v\}on a set of positivePposP\_\{\\mathrm\{pos\}\}\-measure, and‖u‖\>0\\\|u\\\|\>0forσ\\sigma\-almost everyzzwhenm\>0m\>0\. HenceA′​\(1\)<0A^\{\\prime\}\(1\)<0, and for somec\>0c\>0,

A​\(λ\)−A​\(1\)=c​\(1−λ\)\+o​\(1−λ\)as​λ↑1\.A\(\\lambda\)\-A\(1\)=c\(1\-\\lambda\)\+o\(1\-\\lambda\)\\qquad\\text\{as \}\\lambda\\uparrow 1\.
We next show that the uniformity term has zero first derivative at the identity\. Let

g​\(z\):=dd​λ​hλ​\(z\)\|λ=1=\(−‖v‖2​u,‖u‖2​v\),g\(z\):=\\left\.\\frac\{d\}\{d\\lambda\}h\_\{\\lambda\}\(z\)\\right\|\_\{\\lambda=1\}=\(\-\\\|v\\\|^\{2\}u,\\,\\\|u\\\|^\{2\}v\),soz⊤​g​\(z\)=0z^\{\\top\}g\(z\)=0\. Atλ=1\\lambda=1, the inner integral

C:=∫𝕊k−1exp⁡\(z⊤​z′/τ\)​𝑑σ​\(z′\)C:=\\int\_\{\\mathbb\{S\}^\{k\-1\}\}\\exp\(z^\{\\top\}z^\{\\prime\}/\\tau\)d\\sigma\(z^\{\\prime\}\)is independent ofzzby rotational symmetry\. Differentiating under the integral, which is justified by smoothness and compactness of the sphere,

U′​\(1\)=1τ​C​∫∫exp⁡\(z⊤​z′/τ\)​\(g​\(z\)⊤​z′\+z⊤​g​\(z′\)\)​𝑑σ​\(z′\)​𝑑σ​\(z\)\.U^\{\\prime\}\(1\)=\\frac\{1\}\{\\tau C\}\\int\\\!\\\!\\int\\exp\(z^\{\\top\}z^\{\\prime\}/\\tau\)\\left\(g\(z\)^\{\\top\}z^\{\\prime\}\+z^\{\\top\}g\(z^\{\\prime\}\)\\right\)d\\sigma\(z^\{\\prime\}\)d\\sigma\(z\)\.For any fixeda∈𝕊k−1a\\in\\mathbb\{S\}^\{k\-1\}, rotational symmetry gives

∫𝕊k−1z​exp⁡\(a⊤​z/τ\)​𝑑σ​\(z\)=α​a\\int\_\{\\mathbb\{S\}^\{k\-1\}\}z\\exp\(a^\{\\top\}z/\\tau\)d\\sigma\(z\)=\\alpha afor some scalarα\\alpha\. Applying this identity to each of the two terms above and usingz⊤​g​\(z\)=0z^\{\\top\}g\(z\)=0yieldsU′​\(1\)=0U^\{\\prime\}\(1\)=0\. Thus

U​\(λ\)−U​\(1\)=o​\(1−λ\)as​λ↑1\.U\(\\lambda\)\-U\(1\)=o\(1\-\\lambda\)\\qquad\\text\{as \}\\lambda\\uparrow 1\.
Combining the two expansions,

ℒ​\(hλ\)−ℒ​\(I\)=−1τ​\(A​\(λ\)−A​\(1\)\)\+\(U​\(λ\)−U​\(1\)\)=−cτ​\(1−λ\)\+o​\(1−λ\)<0\\mathcal\{L\}\(h\_\{\\lambda\}\)\-\\mathcal\{L\}\(I\)=\-\\frac\{1\}\{\\tau\}\\bigl\(A\(\\lambda\)\-A\(1\)\\bigr\)\+\\bigl\(U\(\\lambda\)\-U\(1\)\\bigr\)=\-\\frac\{c\}\{\\tau\}\(1\-\\lambda\)\+o\(1\-\\lambda\)<0for allλ<1\\lambda<1sufficiently close to11\.

Finally, every orthogonal maph~∈O​\(k\)\\tilde\{h\}\\in O\(k\)preserves inner products and the uniform spherical measure, soℒ​\(h~\)=ℒ​\(I\)\\mathcal\{L\}\(\\tilde\{h\}\)=\\mathcal\{L\}\(I\)\. Takingh=hλh=h\_\{\\lambda\}for anyλ<1\\lambda<1sufficiently close to11gives

ℒCL​\(h\)<ℒCL​\(h~\),∀h~∈O​\(k\),\\mathcal\{L\}\_\{\\mathrm\{CL\}\}\(h\)<\\mathcal\{L\}\_\{\\mathrm\{CL\}\}\(\\tilde\{h\}\),\\qquad\\forall\\,\\tilde\{h\}\\in O\(k\),in the asymptotic regimeM→∞M\\to\\infty\. ∎

### A\.3Proofs for the Corrected Model

To address the support mismatch, we modify the model conditional to incorporate the same constraint structure:

qh,zK​\(z~\)=eh​\(z\)⊤​h​\(z~\)/τ∫K​\(z\)eh​\(z\)⊤​h​\(z′\)/τ​𝑑σK​\(z\)​\(z′\),z~∈K​\(z\)\.q^\{K\}\_\{h,z\}\(\\tilde\{z\}\)=\\frac\{e^\{h\(z\)^\{\\top\}h\(\\tilde\{z\}\)/\\tau\}\}\{\\int\_\{K\(z\)\}e^\{h\(z\)^\{\\top\}h\(z^\{\\prime\}\)/\\tau\}d\\sigma\_\{K\(z\)\}\(z^\{\\prime\}\)\},\\qquad\\tilde\{z\}\\in K\(z\)\.\(24\)
This modification ensures thatsupp​\(Qh,zK\)=supp​\(PZ~\|zK\)\\text\{supp\}\(Q^\{K\}\_\{h,z\}\)=\\text\{supp\}\(P^\{K\}\_\{\\tilde\{Z\}\|z\}\), eliminating the support mismatch\.

###### Theorem A\.4\(Asymptotic Form of Modified Contrastive Loss\)\.

Under the corrected model conditional, where negative samples are drawn uniformly from the constrained manifoldK​\(z\)K\(z\)rather than from the full sphere𝒵\\mathcal\{Z\}, the asymptotic contrastive loss takes the form:

limM→∞ℒ​\(h,τ,M\)−log⁡\(M\)=𝔼z∼PZ​\[H​\(PZ~\|zK,Qh,zK\)\]−𝔼z∼PZ​\[log⁡\(\|K​\(z\)\|\)\]\\lim\_\{M\\to\\infty\}\\mathcal\{L\}\(h,\\tau,M\)\-\\log\(M\)=\\mathbb\{E\}\_\{z\\sim P\_\{Z\}\}\[H\(P^\{K\}\_\{\\tilde\{Z\}\|z\},Q^\{K\}\_\{h,z\}\)\]\-\\mathbb\{E\}\_\{z\\sim P\_\{Z\}\}\[\\log\(\|K\(z\)\|\)\]

The proof follows the same steps as in\(Zimmermannet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib13)\), but with conditionals defined onK​\(z\)K\(z\)\.

###### Proof\.

Step 1: Cross\-entropy decomposition\.The cross\-entropy between the true and model conditionals, both defined onK​\(z\)K\(z\), is:

H​\(PZ~\|zK,Qh,zK\)\\displaystyle H\(P^\{K\}\_\{\\tilde\{Z\}\|z\},Q^\{K\}\_\{h,z\}\)=−𝔼z~∼PZ~\|zK​\[log⁡qh,zK​\(z~\)\]\\displaystyle=\-\\mathbb\{E\}\_\{\\tilde\{z\}\\sim P^\{K\}\_\{\\tilde\{Z\}\|z\}\}\[\\log q^\{K\}\_\{h,z\}\(\\tilde\{z\}\)\]\(25\)=−𝔼z~∼PZ~\|zK​\[log⁡\(eh​\(z\)⊤​h​\(z~\)/τ∫K​\(z\)eh​\(z\)⊤​h​\(z′\)/τ​𝑑σK​\(z\)​\(z′\)\)\]\.\\displaystyle=\-\\mathbb\{E\}\_\{\\tilde\{z\}\\sim P^\{K\}\_\{\\tilde\{Z\}\|z\}\}\\left\[\\log\\left\(\\frac\{e^\{h\(z\)^\{\\top\}h\(\\tilde\{z\}\)/\\tau\}\}\{\\int\_\{K\(z\)\}e^\{h\(z\)^\{\\top\}h\(z^\{\\prime\}\)/\\tau\}d\\sigma\_\{K\(z\)\}\(z^\{\\prime\}\)\}\\right\)\\right\]\.\(26\)
H​\(PZ~\|zK,Qh,zK\)\\displaystyle H\(P^\{K\}\_\{\\tilde\{Z\}\|z\},Q^\{K\}\_\{h,z\}\)=−𝔼z~∼PZ~\|zK​\[1τ​h​\(z\)⊤​h​\(z~\)−log⁡Ch​\(z\)\]\\displaystyle=\-\\mathbb\{E\}\_\{\\tilde\{z\}\\sim P^\{K\}\_\{\\tilde\{Z\}\|z\}\}\\left\[\\frac\{1\}\{\\tau\}h\(z\)^\{\\top\}h\(\\tilde\{z\}\)\-\\log C\_\{h\}\(z\)\\right\]\(27\)=−1τ​𝔼z~∼PZ~\|zK​\[h​\(z\)⊤​h​\(z~\)\]\+log⁡Ch​\(z\)\\displaystyle=\-\\frac\{1\}\{\\tau\}\\mathbb\{E\}\_\{\\tilde\{z\}\\sim P^\{K\}\_\{\\tilde\{Z\}\|z\}\}\[h\(z\)^\{\\top\}h\(\\tilde\{z\}\)\]\+\\log C\_\{h\}\(z\)\(28\)whereCh​\(z\)=∫K​\(z\)eh​\(z\)⊤​h​\(z′\)/τ​𝑑σK​\(z\)​\(z′\)C\_\{h\}\(z\)=\\int\_\{K\(z\)\}e^\{h\(z\)^\{\\top\}h\(z^\{\\prime\}\)/\\tau\}d\\sigma\_\{K\(z\)\}\(z^\{\\prime\}\)is the normalizing constant over the constrained manifold\.

Step 2: Normalizing constant estimation\.Using the fact that the uniform distribution onK​\(z\)K\(z\)has density1/\|K​\(z\)\|1/\|K\(z\)\|:

Ch​\(z\)\\displaystyle C\_\{h\}\(z\)=∫K​\(z\)eh​\(z\)⊤​h​\(z′\)/τ​𝑑σK​\(z\)​\(z′\)=\|K​\(z\)\|⋅𝔼z′∼U​\(K​\(z\)\)​\[eh​\(z\)⊤​h​\(z′\)/τ\]\\displaystyle=\\int\_\{K\(z\)\}e^\{h\(z\)^\{\\top\}h\(z^\{\\prime\}\)/\\tau\}d\\sigma\_\{K\(z\)\}\(z^\{\\prime\}\)=\|K\(z\)\|\\cdot\\mathbb\{E\}\_\{z^\{\\prime\}\\sim\\text\{U\}\(K\(z\)\)\}\\left\[e^\{h\(z\)^\{\\top\}h\(z^\{\\prime\}\)/\\tau\}\\right\]\(29\)
Step 3: Final form\.Substituting the estimate ofCh​\(z\)C\_\{h\}\(z\)back into the cross\-entropy expression and splitting the logarithm:

H​\(PZ~\|zK,Qh,zK\)\\displaystyle H\(P^\{K\}\_\{\\tilde\{Z\}\|z\},Q^\{K\}\_\{h,z\}\)=−1τ​𝔼z~∼PZ~\|zK​\[h​\(z\)⊤​h​\(z~\)\]\\displaystyle=\-\\frac\{1\}\{\\tau\}\\mathbb\{E\}\_\{\\tilde\{z\}\\sim P^\{K\}\_\{\\tilde\{Z\}\|z\}\}\[h\(z\)^\{\\top\}h\(\\tilde\{z\}\)\]\(30\)\+log⁡𝔼z′∼U​\(K​\(z\)\)​\[eh​\(z\)⊤​h​\(z′\)/τ\]\+log⁡\|K​\(z\)\|\\displaystyle\\quad\+\\log\\mathbb\{E\}\_\{z^\{\\prime\}\\sim\\text\{U\}\(K\(z\)\)\}\\left\[e^\{h\(z\)^\{\\top\}h\(z^\{\\prime\}\)/\\tau\}\\right\]\+\\log\|K\(z\)\|\(31\)
Taking expectations overz∼PZz\\sim P\_\{Z\}yields the stated result\. ∎

###### Proof of Theorem[4\.6](https://arxiv.org/html/2606.04280#S4.Thmtheorem6)\.

With the corrected model conditional, the supports ofPZ~\|zKP^\{K\}\_\{\\tilde\{Z\}\|z\}andQh,zKQ^\{K\}\_\{h,z\}now match: both are restricted to the submanifoldK​\(z\)K\(z\)\. Within this constrained setting, the analysis proceeds analogously to Theorem[A\.1](https://arxiv.org/html/2606.04280#A1.Thmtheorem1)\.

For anyh∈O​\(k\)h\\in O\(k\), we haveh​\(z\)⊤​h​\(z~\)=z⊤​z~h\(z\)^\{\\top\}h\(\\tilde\{z\}\)=z^\{\\top\}\\tilde\{z\}for allz,z~∈𝒵z,\\tilde\{z\}\\in\\mathcal\{Z\}\. This means the model conditionalQh,zKQ^\{K\}\_\{h,z\}exactly matches the true conditionalPZ~\|zKP^\{K\}\_\{\\tilde\{Z\}\|z\}on the constrained manifoldK​\(z\)K\(z\):

qh,zK​\(z~\)=ez⊤​z~/τ∫K​\(z\)ez⊤​z′/τ​𝑑σK​\(z\)​\(z′\)=d​PZ~\|zKd​σK​\(z\)​\(z~\)q^\{K\}\_\{h,z\}\(\\tilde\{z\}\)=\\frac\{e^\{z^\{\\top\}\\tilde\{z\}/\\tau\}\}\{\\int\_\{K\(z\)\}e^\{z^\{\\top\}z^\{\\prime\}/\\tau\}d\\sigma\_\{K\(z\)\}\(z^\{\\prime\}\)\}=\\frac\{dP^\{K\}\_\{\\tilde\{Z\}\|z\}\}\{d\\sigma\_\{K\(z\)\}\}\(\\tilde\{z\}\)whenκ=1/τ\\kappa=1/\\tau\.

Since matching distributions achieves zero KL divergence and hence minimal cross\-entropy, any orthogonal transformation minimizes the asymptotic contrastive loss\. ∎

Although the corrected objective admits orthogonal solutions, it does not guarantee uniqueness\. The following theorem shows that multiple equivalent solutions exist\.

###### Theorem A\.5\(Equivalence of Feature Extractors Under Constrained Sampling\)\.

Given a data\-generating processg:𝒵→𝒳g:\\mathcal\{Z\}\\to\\mathcal\{X\}, uniform marginal lawPZP\_\{Z\}, ground\-truth conditional lawPZ~\|zP\_\{\\tilde\{Z\}\|z\}with densityp​\(z~\|z\)p\(\\tilde\{z\}\|z\), and model conditional lawQh,zQ\_\{h,z\}with densityqh​\(z~\|z\)q\_\{h\}\(\\tilde\{z\}\|z\)that define the expected cross\-entropy loss

ℒh=𝔼z∼PZ​\[H​\(PZ~\|z,Qh,z\)\],\\mathcal\{L\}\_\{h\}=\\mathbb\{E\}\_\{z\\sim P\_\{Z\}\}\[H\(P\_\{\\tilde\{Z\}\|z\},Q\_\{h,z\}\)\],letm:𝒵→𝒵m:\\mathcal\{Z\}\\to\\mathcal\{Z\}be an invertible mapping that preserves the marginal law and the ground\-truth conditional density:

m\#​PZ=PZ,p​\(m​\(z~\)\|m​\(z\)\)=p​\(z~\|z\)∀z,z~∈𝒵\.m\_\{\\\#\}P\_\{Z\}=P\_\{Z\},\\qquad p\(m\(\\tilde\{z\}\)\|m\(z\)\)=p\(\\tilde\{z\}\|z\)\\quad\\forall z,\\tilde\{z\}\\in\\mathcal\{Z\}\.Then any two mappingsh1:=f1∘gh\_\{1\}:=f\_\{1\}\\circ gandh2:=f2∘gh\_\{2\}:=f\_\{2\}\\circ gwithh2​\(z\):=h1​\(m​\(z\)\)h\_\{2\}\(z\):=h\_\{1\}\(m\(z\)\)are equivalent, i\.e\.,ℒh1=ℒh2\\mathcal\{L\}\_\{h\_\{1\}\}=\\mathcal\{L\}\_\{h\_\{2\}\}\.

###### Proof\.

Starting with the cross\-entropy loss forh2h\_\{2\}:

ℒh2=𝔼z∼PZ​\[𝔼z~∼PZ~\|z​\[−log⁡qh2​\(z~\|z\)\]\]\\mathcal\{L\}\_\{h\_\{2\}\}=\\mathbb\{E\}\_\{z\\sim P\_\{Z\}\}\\left\[\\mathbb\{E\}\_\{\\tilde\{z\}\\sim P\_\{\\tilde\{Z\}\|z\}\}\[\-\\log q\_\{h\_\{2\}\}\(\\tilde\{z\}\|z\)\]\\right\]\(32\)
Using the definition ofh2h\_\{2\}:

=𝔼z∼PZ​\[𝔼z~∼PZ~\|z​\[−log⁡qh1​\(m​\(z~\)\|m​\(z\)\)\]\]=\\mathbb\{E\}\_\{z\\sim P\_\{Z\}\}\\left\[\\mathbb\{E\}\_\{\\tilde\{z\}\\sim P\_\{\\tilde\{Z\}\|z\}\}\[\-\\log q\_\{h\_\{1\}\}\(m\(\\tilde\{z\}\)\|m\(z\)\)\]\\right\]\(33\)
Sincemmpreserves the conditional density:

=𝔼z∼PZ​\[𝔼m​\(z~\)∼PZ~\|m​\(z\)​\[−log⁡qh1​\(m​\(z~\)\|m​\(z\)\)\]\]=\\mathbb\{E\}\_\{z\\sim P\_\{Z\}\}\\left\[\\mathbb\{E\}\_\{m\(\\tilde\{z\}\)\\sim P\_\{\\tilde\{Z\}\|m\(z\)\}\}\[\-\\log q\_\{h\_\{1\}\}\(m\(\\tilde\{z\}\)\|m\(z\)\)\]\\right\]\(34\)
Sincem\#​PZ=PZm\_\{\\\#\}P\_\{Z\}=P\_\{Z\}, we can change variables in the outer expectation:

=𝔼m​\(z\)∼PZ​\[𝔼m​\(z~\)∼PZ~\|m​\(z\)​\[−log⁡qh1​\(m​\(z~\)\|m​\(z\)\)\]\]=\\mathbb\{E\}\_\{m\(z\)\\sim P\_\{Z\}\}\\left\[\\mathbb\{E\}\_\{m\(\\tilde\{z\}\)\\sim P\_\{\\tilde\{Z\}\|m\(z\)\}\}\[\-\\log q\_\{h\_\{1\}\}\(m\(\\tilde\{z\}\)\|m\(z\)\)\]\\right\]\(35\)
Finally, becausemmis invertible:

=𝔼z∼PZ​\[𝔼z~∼PZ~\|z​\[−log⁡qh1​\(z~\|z\)\]\]=ℒh1=\\mathbb\{E\}\_\{z\\sim P\_\{Z\}\}\\left\[\\mathbb\{E\}\_\{\\tilde\{z\}\\sim P\_\{\\tilde\{Z\}\|z\}\}\[\-\\log q\_\{h\_\{1\}\}\(\\tilde\{z\}\|z\)\]\\right\]=\\mathcal\{L\}\_\{h\_\{1\}\}\(36\)∎

This theorem reveals a fundamental non\-uniqueness in the solution space of the modified contrastive learning objective\. Any conditional\-preserving transformation of the latent space yields identical loss values, meaning that without additional constraints through inductive bias, the objective cannot distinguish between semantically meaningful representations and arbitrary rearrangements that preserve only local structure within constrained manifolds\.

## Appendix BGenerative Processes and Encoder Architectures

This section describes the generative processes and encoder architectures used in our synthetic experiments\.

### B\.1Generative Processes

We employ five generative processesg:𝕊d−1→ℝDg:\\mathbb\{S\}^\{d\-1\}\\to\\mathbb\{R\}^\{D\}of varying complexity to test our theoretical predictions across different data\-generating mechanisms\.

Algorithm 1Identity Process0:

z∈𝕊d−1z\\in\\mathbb\{S\}^\{d\-1\}
1:

x←zx\\leftarrow z
2:return

x∈ℝdx\\in\\mathbb\{R\}^\{d\}

Algorithm 2Linear Process0:

z∈𝕊d−1z\\in\\mathbb\{S\}^\{d\-1\}, weight matrix

W∈ℝD×dW\\in\\mathbb\{R\}^\{D\\times d\}with

rank​\(W\)=d\\text\{rank\}\(W\)=d
1:

x←W​zx\\leftarrow Wz
2:return

x∈ℝDx\\in\\mathbb\{R\}^\{D\}

Algorithm 3Spiral Rotation Process0:

z=\(z1,z2,z3\)∈𝕊2z=\(z\_\{1\},z\_\{2\},z\_\{3\}\)\\in\\mathbb\{S\}^\{2\}, period

nn
1:Compute rotation angle:

θ←n​π​z3\\theta\\leftarrow n\\pi z\_\{3\}
2:Apply 2D rotation to first two coordinates:

3:

x1←cos⁡\(θ\)​z1−sin⁡\(θ\)​z2x\_\{1\}\\leftarrow\\cos\(\\theta\)z\_\{1\}\-\\sin\(\\theta\)z\_\{2\}
4:

x2←sin⁡\(θ\)​z1\+cos⁡\(θ\)​z2x\_\{2\}\\leftarrow\\sin\(\\theta\)z\_\{1\}\+\\cos\(\\theta\)z\_\{2\}
5:

x3←z3x\_\{3\}\\leftarrow z\_\{3\}
6:return

x=\(x1,x2,x3\)∈ℝ3x=\(x\_\{1\},x\_\{2\},x\_\{3\}\)\\in\\mathbb\{R\}^\{3\}

Algorithm 4Patches Process0:

z=\(z1,z2,z3\)∈𝕊2z=\(z\_\{1\},z\_\{2\},z\_\{3\}\)\\in\\mathbb\{S\}^\{2\}, number of slices

KK
1:Step 1:Apply piecewise rotation based on

z3z\_\{3\}
2:Determine bucket

k←⌊\(z3\+1\)⋅K/2⌋k\\leftarrow\\lfloor\(z\_\{3\}\+1\)\\cdot K/2\\rfloor
3:Compute angle

θk←−π/max⁡\(1,K−k\)\\theta\_\{k\}\\leftarrow\-\\pi/\\max\(1,K\-k\)
4:

z′←Rx​y​\(θk\)⋅zz^\{\\prime\}\\leftarrow R\_\{xy\}\(\\theta\_\{k\}\)\\cdot z\{Rotate in

\(x,y\)\(x,y\)plane\}

5:Step 2:Apply 3D rotation \(pitch

=π/2=\\pi/2\)

6:

z′′←Ry​\(π/2\)⋅z′z^\{\\prime\\prime\}\\leftarrow R\_\{y\}\(\\pi/2\)\\cdot z^\{\\prime\}
7:Step 3:Apply second piecewise rotation

8:Determine new bucket, apply rotation as in Step 1

9:return

x∈ℝ3x\\in\\mathbb\{R\}^\{3\}

Algorithm 5Invertible MLP Process\(Hyvärinen and Morioka,[2016](https://arxiv.org/html/2606.04280#bib.bib27)\)0:

z∈𝕊d−1z\\in\\mathbb\{S\}^\{d\-1\}, MLP layers

\{Wi,bi\}i=1L\\\{W\_\{i\},b\_\{i\}\\\}\_\{i=1\}^\{L\}with conditioning

1:

h0←zh\_\{0\}\\leftarrow z
2:for

i=1i=1to

LLdo

3:

hi←σ​\(Wi​hi−1\+bi\)h\_\{i\}\\leftarrow\\sigma\(W\_\{i\}h\_\{i\-1\}\+b\_\{i\}\)\{

σ\\sigma= LeakyReLU\}

4:endfor

5:return

x=hL∈ℝDx=h\_\{L\}\\in\\mathbb\{R\}^\{D\}

### B\.2Encoder Architectures

We compare two classes of encoders: a generic MLP encoder representing low inductive bias, and inverse encoders designed to mirror the structure of each generative process, representing high inductive bias\.

Algorithm 6MLP Encoder \(Low Inductive Bias\)0:

x∈ℝDx\\in\\mathbb\{R\}^\{D\}, hidden dims

\[128,256,256,256,128\]\[128,256,256,256,128\]
1:

h0←xh\_\{0\}\\leftarrow x
2:for

i=1i=1to

L−1L\-1do

3:

hi←ReLU​\(BatchNorm​\(Wi​hi−1\+bi\)\)h\_\{i\}\\leftarrow\\text\{ReLU\}\(\\text\{BatchNorm\}\(W\_\{i\}h\_\{i\-1\}\+b\_\{i\}\)\)
4:endfor

5:

z′←WL​hL−1\+bLz^\{\\prime\}\\leftarrow W\_\{L\}h\_\{L\-1\}\+b\_\{L\}
6:

z←z′/‖z′‖2z\\leftarrow z^\{\\prime\}/\\\|z^\{\\prime\}\\\|\_\{2\}\{Project to sphere\}

7:return

z∈𝕊d−1z\\in\\mathbb\{S\}^\{d\-1\}

Algorithm 7Inverse Linear Encoder \(High Inductive Bias\)0:

x∈ℝDx\\in\\mathbb\{R\}^\{D\}, learnable

W∈ℝd×DW\\in\\mathbb\{R\}^\{d\\times D\},

b∈ℝdb\\in\\mathbb\{R\}^\{d\}
1:

z′←W​x\+bz^\{\\prime\}\\leftarrow Wx\+b
2:

z←z′/‖z′‖2z\\leftarrow z^\{\\prime\}/\\\|z^\{\\prime\}\\\|\_\{2\}
3:return

z∈𝕊d−1z\\in\\mathbb\{S\}^\{d\-1\}

Algorithm 8Inverse Spiral Encoder \(High Inductive Bias\)0:

x∈ℝ3x\\in\\mathbb\{R\}^\{3\}, period

nn
1:Predict rotation control:

c←MLProt​\(x\)c\\leftarrow\\text\{MLP\}\_\{\\text\{rot\}\}\(x\)\{3\-layer MLP\}

2:Extract spatial components:

\(x1,x2\)←x1:2\(x\_\{1\},x\_\{2\}\)\\leftarrow x\_\{1:2\}
3:Compute inverse rotation:

θ←−n​π​c\\theta\\leftarrow\-n\\pi c
4:Apply inverse rotation:

5:

z1←cos⁡\(θ\)​x1−sin⁡\(θ\)​x2z\_\{1\}\\leftarrow\\cos\(\\theta\)x\_\{1\}\-\\sin\(\\theta\)x\_\{2\}
6:

z2←sin⁡\(θ\)​x1\+cos⁡\(θ\)​x2z\_\{2\}\\leftarrow\\sin\(\\theta\)x\_\{1\}\+\\cos\(\\theta\)x\_\{2\}
7:

z′←\(z1,z2,c\)z^\{\\prime\}\\leftarrow\(z\_\{1\},z\_\{2\},c\)
8:

z←z′/‖z′‖2z\\leftarrow z^\{\\prime\}/\\\|z^\{\\prime\}\\\|\_\{2\}
9:return

z∈𝕊2z\\in\\mathbb\{S\}^\{2\}

Algorithm 9Inverse Patches Encoder \(High Inductive Bias\)0:

x∈ℝ3x\\in\\mathbb\{R\}^\{3\}, number of slices

KK
1:Predict original

zz\-coordinate:

zpred←tanh⁡\(MLPz​\(x\)\)z\_\{\\text\{pred\}\}\\leftarrow\\tanh\(\\text\{MLP\}\_\{z\}\(x\)\)
2:Predict bucket probabilities:

w←softmax​\(MLPbucket​\(x\)\)w\\leftarrow\\text\{softmax\}\(\\text\{MLP\}\_\{\\text\{bucket\}\}\(x\)\)
3:Step 1:Inverse second piecewise rotation

4:

x1←∑k=0K−1wk⋅Rx​y​\(−θk\)⋅xx\_\{1\}\\leftarrow\\sum\_\{k=0\}^\{K\-1\}w\_\{k\}\\cdot R\_\{xy\}\(\-\\theta\_\{k\}\)\\cdot x\{Soft inverse\}

5:Step 2:Inverse 3D rotation

6:

x2←Ry​\(−π/2\)⋅x1x\_\{2\}\\leftarrow R\_\{y\}\(\-\\pi/2\)\\cdot x\_\{1\}
7:Step 3:Inverse first piecewise rotation

8:

x3←∑k=0K−1wk⋅Rx​y​\(−θk\)⋅x2x\_\{3\}\\leftarrow\\sum\_\{k=0\}^\{K\-1\}w\_\{k\}\\cdot R\_\{xy\}\(\-\\theta\_\{k\}\)\\cdot x\_\{2\}
9:Replace

zz\-coordinate:

xrec←\(x3,1,x3,2,zpred\)x\_\{\\text\{rec\}\}\\leftarrow\(x\_\{3,1\},x\_\{3,2\},z\_\{\\text\{pred\}\}\)
10:

z←xrec/‖xrec‖2z\\leftarrow x\_\{\\text\{rec\}\}/\\\|x\_\{\\text\{rec\}\}\\\|\_\{2\}
11:return

z∈𝕊2z\\in\\mathbb\{S\}^\{2\}

## Appendix CSampling Procedures

We use standard techniques for sampling from the uniform distribution on the sphere and the von Mises\-Fisher distribution\.

Algorithm 10Uniform Sampling from𝕊d−1\\mathbb\{S\}^\{d\-1\}0:Dimension

d∈ℕd\\in\\mathbb\{N\}
1:Draw

zi∼𝒩​\(0,1\)z\_\{i\}\\sim\\mathcal\{N\}\(0,1\)independently for

i=1,…,di=1,\\ldots,d
2:

z←\(z1,…,zd\)⊤z\\leftarrow\(z\_\{1\},\\ldots,z\_\{d\}\)^\{\\top\}
3:

v←z/‖z‖2v\\leftarrow z/\\\|z\\\|\_\{2\}
4:return

v∈𝕊d−1v\\in\\mathbb\{S\}^\{d\-1\}

Algorithm 11von Mises\-Fisher Sampling\(Wood,[1994](https://arxiv.org/html/2606.04280#bib.bib41)\)0:Mean direction

μ∈𝕊d−1\\mu\\in\\mathbb\{S\}^\{d\-1\}, concentration

κ\>0\\kappa\>0
1:

p←d−1p\\leftarrow d\-1
2:

b←p/\(4​κ2\+p2\+2​κ\)b\\leftarrow p/\(\\sqrt\{4\\kappa^\{2\}\+p^\{2\}\}\+2\\kappa\)
3:

x←\(1−b\)/\(1\+b\)x\\leftarrow\(1\-b\)/\(1\+b\)
4:

c←κ​x\+p​log⁡\(1−x2\)c\\leftarrow\\kappa x\+p\\log\(1\-x^\{2\}\)
5:repeat

6:Sample

t∼Beta​\(p/2,p/2\)t\\sim\\text\{Beta\}\(p/2,p/2\)
7:

w←\(1−\(1\+b\)​t\)/\(1−\(1−b\)​t\)w\\leftarrow\(1\-\(1\+b\)t\)/\(1\-\(1\-b\)t\)
8:Sample

u∼Uniform​\(0,1\)u\\sim\\text\{Uniform\}\(0,1\)
9:until

log⁡\(u\)≤κ​w\+p​log⁡\(1−x​w\)−c\\log\(u\)\\leq\\kappa w\+p\\log\(1\-xw\)\-c
10:Sample

g∼𝒩​\(0,Id\)g\\sim\\mathcal\{N\}\(0,I\_\{d\}\)
11:

v←g−\(g⊤​μ\)​μv\\leftarrow g\-\(g^\{\\top\}\\mu\)\\mu\{Project out

μ\\mucomponent\}

12:

v←v/‖v‖2v\\leftarrow v/\\\|v\\\|\_\{2\}
13:

s←1−w2⋅v\+w⋅μs\\leftarrow\\sqrt\{1\-w^\{2\}\}\\cdot v\+w\\cdot\\mu
14:return

s∼vMF​\(μ,κ\)s\\sim\\text\{vMF\}\(\\mu,\\kappa\)

Algorithm 12Conditional Sampling: Diversity Holds0:Anchor

z∈𝕊d−1z\\in\\mathbb\{S\}^\{d\-1\}, concentration

κ\\kappa
1:

z~←vMF​\(z,κ\)\\tilde\{z\}\\leftarrow\\text\{vMF\}\(z,\\kappa\)\{Sample positive using Algorithm[11](https://arxiv.org/html/2606.04280#alg11)\}

2:return

z~∈𝕊d−1\\tilde\{z\}\\in\\mathbb\{S\}^\{d\-1\}

Algorithm 13Conditional Sampling: Diversity Violated0:Anchor

z=\(u,v\)∈𝕊d−1z=\(u,v\)\\in\\mathbb\{S\}^\{d\-1\}, fixed dimensions

dfixedd\_\{\\text\{fixed\}\}, concentration

κ\\kappa
1:

u←z1:dfixedu\\leftarrow z\_\{1:d\_\{\\text\{fixed\}\}\}\{Fixed component\}

2:

v←zdfixed\+1:dv\\leftarrow z\_\{d\_\{\\text\{fixed\}\}\+1:d\}\{Varying component\}

3:

r←‖v‖2r\\leftarrow\\\|v\\\|\_\{2\}\{Radius of sub\-sphere\}

4:if

r\>0r\>0then

5:

v^←v/r\\hat\{v\}\\leftarrow v/r\{Normalize to sub\-sphere\}

6:

v~←vMF​\(v^,κ\)\\tilde\{v\}\\leftarrow\\text\{vMF\}\(\\hat\{v\},\\kappa\)\{Sample on sub\-sphere\}

7:

v~←r⋅v~\\tilde\{v\}\\leftarrow r\\cdot\\tilde\{v\}\{Scale back\}

8:else

9:

v~←v\\tilde\{v\}\\leftarrow v
10:endif

11:

z~←\(u,v~\)\\tilde\{z\}\\leftarrow\(u,\\tilde\{v\}\)\{Concatenate fixed and sampled\}

12:return

z~∈𝕊d−1\\tilde\{z\}\\in\\mathbb\{S\}^\{d\-1\}

Algorithm 14Adapted InfoNCE with Same\-anchor Negatives0:Encoder

hh, batch

\{x1,…,xN\}\\\{x\_\{1\},\\ldots,x\_\{N\}\\\}, stochastic augmentation

𝒯\\mathcal\{T\}, temperature

τ\\tau, negatives per anchor

MM
1:foreach anchor

xix\_\{i\}do

2:Draw positive view

x~i←𝒯​\(xi\)\\tilde\{x\}\_\{i\}\\leftarrow\\mathcal\{T\}\(x\_\{i\}\)
3:for

j=1j=1to

MMdo

4:Draw same\-anchor negative

xi,j−←𝒯​\(xi\)x^\{\-\}\_\{i,j\}\\leftarrow\\mathcal\{T\}\(x\_\{i\}\)independently

5:endfor

6:Compute

ℒi=−log⁡exp⁡\(h​\(xi\)⊤​h​\(x~i\)/τ\)exp⁡\(h​\(xi\)⊤​h​\(x~i\)/τ\)\+∑j=1Mexp⁡\(h​\(xi\)⊤​h​\(xi,j−\)/τ\)\.\\mathcal\{L\}\_\{i\}=\-\\log\\frac\{\\exp\(h\(x\_\{i\}\)^\{\\top\}h\(\\tilde\{x\}\_\{i\}\)/\\tau\)\}\{\\exp\(h\(x\_\{i\}\)^\{\\top\}h\(\\tilde\{x\}\_\{i\}\)/\\tau\)\+\\sum\_\{j=1\}^\{M\}\\exp\(h\(x\_\{i\}\)^\{\\top\}h\(x^\{\-\}\_\{i,j\}\)/\\tau\)\}\.
7:endfor

8:return

ℒ=1N​∑i=1Nℒi\\mathcal\{L\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathcal\{L\}\_\{i\}

Algorithm[14](https://arxiv.org/html/2606.04280#alg14)approximates sampling negatives fromK​\(z\)K\(z\)in observation space\. Each call to𝒯\\mathcal\{T\}uses independent randomness conditional on the same anchorxix\_\{i\}, which is the observation\-space analogue of drawing conditionally independent samples fromPZ~\|ziP\_\{\\tilde\{Z\}\|z\_\{i\}\}\. The same sampling family is applied repeatedly to the same anchor, so all views preserve the invariant component encoded by the mechanism while varying the remaining components\. This changes the negative\-sampling task relative to standard InfoNCE, which samples negatives from other instances\. In our framework this is consistent with the goal of recovering latent distance structure rather than discriminating instance identity, but it can trade cross\-instance discrimination signal for better support matching\. The approximation also samples from the augmentation\-induced distribution onK​\(z\)K\(z\)rather than uniformly fromK​\(z\)K\(z\)\.

## Appendix DEvaluation Metrics

We evaluate latent space reconstruction quality using three complementary metrics, following standard practice in the identifiability literature\(Hyvärinen and Morioka,[2016](https://arxiv.org/html/2606.04280#bib.bib27); Zimmermannet al\.,[2021](https://arxiv.org/html/2606.04280#bib.bib13)\)\.

Algorithm 15Linear Identifiability \(R2R^\{2\}\)0:Ground\-truth latents

Z=\{zi\}i=1nZ=\\\{z\_\{i\}\\\}\_\{i=1\}^\{n\}, recovered latents

Z^=\{z^i\}i=1n\\hat\{Z\}=\\\{\\hat\{z\}\_\{i\}\\\}\_\{i=1\}^\{n\}
1:Fit linear regression:

z^=A​z\+b\\hat\{z\}=Az\+bminimizing

∑i‖zi−\(A​z^i\+b\)‖2\\sum\_\{i\}\\\|z\_\{i\}\-\(A\\hat\{z\}\_\{i\}\+b\)\\\|^\{2\}
2:Compute predictions:

z~i←A​z^i\+b\\tilde\{z\}\_\{i\}\\leftarrow A\\hat\{z\}\_\{i\}\+b
3:Compute total variance:

SStot←∑i=1n‖zi−z¯‖2\\text\{SS\}\_\{\\text\{tot\}\}\\leftarrow\\sum\_\{i=1\}^\{n\}\\\|z\_\{i\}\-\\bar\{z\}\\\|^\{2\}
4:Compute residual variance:

SSres←∑i=1n‖zi−z~i‖2\\text\{SS\}\_\{\\text\{res\}\}\\leftarrow\\sum\_\{i=1\}^\{n\}\\\|z\_\{i\}\-\\tilde\{z\}\_\{i\}\\\|^\{2\}
5:

R2←1−SSres/SStotR^\{2\}\\leftarrow 1\-\\text\{SS\}\_\{\\text\{res\}\}/\\text\{SS\}\_\{\\text\{tot\}\}
6:return

R2∈\(−∞,1\]R^\{2\}\\in\(\-\\infty,1\]\{

1\.01\.0= perfect linear recovery\}

Algorithm 16Mean Correlation Coefficient \(MCC\)0:Ground\-truth latents

Z∈ℝn×dZ\\in\\mathbb\{R\}^\{n\\times d\}, recovered latents

Z^∈ℝn×d\\hat\{Z\}\\in\\mathbb\{R\}^\{n\\times d\}
1:Compute correlation matrix

C∈ℝd×dC\\in\\mathbb\{R\}^\{d\\times d\}:

2:

Ci​j←\|corr​\(Z:,i,Z^:,j\)\|C\_\{ij\}\\leftarrow\|\\text\{corr\}\(Z\_\{:,i\},\\hat\{Z\}\_\{:,j\}\)\|
3:Find optimal assignment via Munkres algorithm:

4:

π∗←arg⁡maxπ​∑i=1dCi,π​\(i\)\\pi^\{\*\}\\leftarrow\\arg\\max\_\{\\pi\}\\sum\_\{i=1\}^\{d\}C\_\{i,\\pi\(i\)\}
5:

MCC←1d​∑i=1dCi,π∗​\(i\)\\text\{MCC\}\\leftarrow\\frac\{1\}\{d\}\\sum\_\{i=1\}^\{d\}C\_\{i,\\pi^\{\*\}\(i\)\}
6:return

MCC∈\[0,1\]\\text\{MCC\}\\in\[0,1\]\{

1\.01\.0= perfect factor alignment\}

Algorithm 17Angular Preservation Error \(APE\)0:Ground\-truth latents

Z=\{zi\}i=1nZ=\\\{z\_\{i\}\\\}\_\{i=1\}^\{n\}, recovered latents

Z^=\{z^i\}i=1n\\hat\{Z\}=\\\{\\hat\{z\}\_\{i\}\\\}\_\{i=1\}^\{n\}
1:Initialize

APE←0\\text\{APE\}\\leftarrow 0
2:for

i=1i=1to

nndo

3:for

j=1j=1to

nn,

j≠ij\\neq ido

4:

APE←APE\+\|zi⊤​zj−z^i⊤​z^j\|\\text\{APE\}\\leftarrow\\text\{APE\}\+\|z\_\{i\}^\{\\top\}z\_\{j\}\-\\hat\{z\}\_\{i\}^\{\\top\}\\hat\{z\}\_\{j\}\|
5:endfor

6:endfor

7:

APE←APE/\(n​\(n−1\)\)\\text\{APE\}\\leftarrow\\text\{APE\}/\(n\(n\-1\)\)
8:return

APE∈\[0,2\]\\text\{APE\}\\in\[0,2\]\{

0\.00\.0= perfect isometry\}

Table 2:Summary of evaluation metrics for latent space reconstruction\.
## Appendix EAdditional Experimental Results

### E\.1Diversity Condition Holds

Table[3](https://arxiv.org/html/2606.04280#A5.T3)reports MCC, APE, and final loss when the diversity condition is satisfied\.

Table 3:Evaluation metrics when diversity condition holds\. Results reported as mean±\\pmstd across 5 random seeds using MLP encoder\. Lower APE is better\.
### E\.2Diversity Condition Violated

Tables[4](https://arxiv.org/html/2606.04280#A5.T4)\-[6](https://arxiv.org/html/2606.04280#A5.T6)compare three approaches when diversity is violated: standard InfoNCE, adapted InfoNCE \(Section[4\.3](https://arxiv.org/html/2606.04280#S4.SS3)\), and InfoNCE with inductive bias\.

Table 4:Mean Correlation Coefficient \(MCC\) when diversity condition is violated\. Results reported as mean±\\pmstd across 5 random seeds\.Table 5:Angular Preservation Error \(APE\) when diversity condition is violated\. Results reported as mean±\\pmstd across 5 random seeds\. Lower is better\.Table 6:Final InfoNCE loss when diversity condition is violated\. Results reported as mean±\\pmstd across 5 random seeds\.
### E\.3Progressive Diversity Violation

Table[7](https://arxiv.org/html/2606.04280#A5.T7)and Figure[5\(c\)](https://arxiv.org/html/2606.04280#A5.F5.sf3)examine how performance degrades as the diversity condition is progressively violated in a 10\-dimensional latent space\. The parameterdfixedd\_\{\\text\{fixed\}\}denotes the number of dimensions held constant during positive pair sampling\.

Table 7:Linear identifiability \(R2R^\{2\}\) as a function of diversity violation severity for 10D latent space\.dfixedd\_\{\\text\{fixed\}\}denotes the number of dimensions held constant during positive pair sampling\.
### E\.4Constraint Ratio Experiments

Tables[8](https://arxiv.org/html/2606.04280#A5.T8)\-[11](https://arxiv.org/html/2606.04280#A5.T11)show identifiability metrics as a function of the constraint ratioρ\\rho, which interpolates between standard InfoNCE \(ρ=0\\rho=0\) and the fully corrected objective \(ρ=1\\rho=1\)\. The parameterρ\\rhocontrols the fraction of negative samples drawn from the constrained manifoldK​\(z\)K\(z\)versus the full sphere𝒵\\mathcal\{Z\}\.

Table 8:Linear identifiability \(R2R^\{2\}\) across all generative processes and encoder types as a function of constraint ratioρ\\rho\. Results reported as mean across 5 seeds\.Table 9:Mean Correlation Coefficient \(MCC\) across all generative processes and encoder types as a function of constraint ratioρ\\rho\. Results reported as mean across 5 seeds\.Table 10:Angular Preservation Error \(APE\) across all generative processes and encoder types as a function of constraint ratioρ\\rho\. Results reported as mean across 5 seeds\. Lower is better\.Table 11:Final InfoNCE loss across all generative processes and encoder types as a function of constraint ratioρ\\rho\. Results reported as mean across 5 seeds\.
### E\.5Synthetic Validation Figures

Figure[5](https://arxiv.org/html/2606.04280#A5.F5)provides visual summaries comparing MLP and inverse encoder performance across conditions\.

![Refer to caption](https://arxiv.org/html/2606.04280v1/images/diversity_comparison_bar_chart.png)\(a\)
![Refer to caption](https://arxiv.org/html/2606.04280v1/images/adjusted_info_nce.png)\(b\)
![Refer to caption](https://arxiv.org/html/2606.04280v1/images/diversity_collapse.png)\(c\)

Figure 5:Synthetic validation of theoretical predictions\. \(a\) Linear identifiability for MLP and inverse encoders\. When the diversity condition holds, MLP achievesR2=1\.00R^\{2\}=1\.00; when violated, it collapses toR2=0\.25R^\{2\}=0\.25while inverse encoders remain robust \(R2=0\.83R^\{2\}=0\.83\)\. \(b\) Angular preservation error vs\. constraint ratioρ\\rho\. Atρ=1\\rho=1\(corrected InfoNCE\), MLP converges to inverse encoder performance\. \(c\) Linear identifiability on𝕊9\\mathbb\{S\}^\{9\}vs\. incremental violation of diversity condition; evendfixed=1d\_\{\\text\{fixed\}\}=1causes catastrophic failure\. Results averaged across 5 generative processes; shaded regions indicate±1\\pm 1std\.